2024-08-28T19:04:06.885 INFO:root:teuthology version: 1.2.1.dev4+gcf5021f 2024-08-28T19:04:06.913 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-28T19:04:07.045 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752 branch: wip-mchangir-testing-20240828.085847-main-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: mchangir@redhat.com first_in_suite: false job_id: '7877752' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-20240828.085847-main-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug cephsqlite: 20 mgr/snap_schedule/allow_m_granularity: true mgr/snap_schedule/dump_on_update: true mgr/snap_schedule/log_level: debug mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: global: ms die on skipped message: false mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request mgr-modules: - snap_schedule sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-20240828.085847-main-debug sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 owner: scheduled_mchangir@teuthology priority: 500 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7797 sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 sleep_before_teardown: 0 subset: 139/512 suite: fs suite_branch: wip-mchangir-testing-20240828.085847-main-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHF6MfFF5p40SG5T7bZWjdbxVqi4f6QB0RMLaSKu5SqMwrK50sUPRkhdRjg8pBGmHBlJOfhuzu9Zo49uBMbwJJc= smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4hdSk0H9BDu0l0b2evLVaEKjtC177wz9JFBY5GNKI2gVHF/UcNdqCSlPwDZZyQ5h/YcJifKS+dc2r90izWuuU= smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH10zppIILtk42EmjUP+BzZ9NUblaxWorqyIKW4IpBXHv3POMxiVQgWF4EQiMaYKqdE03V9Vp4ne+l8O4yhxFw4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: cf5021f85c4b0bf435c74a1183036b3d19af44b5 timestamp: 2024-08-28_12:36:35 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-28T19:04:07.045 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa; will attempt to use it 2024-08-28T19:04:07.046 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks 2024-08-28T19:04:07.046 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-28T19:04:07.048 INFO:teuthology.task.internal:Checking packages... 2024-08-28T19:04:07.069 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd19e6118f7015f0047264006cb0c873528eb3ae6' 2024-08-28T19:04:07.069 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-28T19:04:07.069 INFO:teuthology.packaging:ref: None 2024-08-28T19:04:07.069 INFO:teuthology.packaging:tag: None 2024-08-28T19:04:07.069 INFO:teuthology.packaging:branch: wip-mchangir-testing-20240828.085847-main-debug 2024-08-28T19:04:07.069 INFO:teuthology.packaging:sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:04:07.069 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-20240828.085847-main-debug 2024-08-28T19:04:07.227 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4560.gd19e6118 2024-08-28T19:04:07.229 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-28T19:04:07.235 INFO:teuthology.task.internal:no buildpackages task found 2024-08-28T19:04:07.235 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-28T19:04:07.251 INFO:teuthology.task.internal:Saving configuration 2024-08-28T19:04:07.272 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-28T19:04:07.280 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-28T19:04:07.312 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-28 18:58:59.906656', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHF6MfFF5p40SG5T7bZWjdbxVqi4f6QB0RMLaSKu5SqMwrK50sUPRkhdRjg8pBGmHBlJOfhuzu9Zo49uBMbwJJc='} 2024-08-28T19:04:07.345 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-28 18:58:59.908636', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4hdSk0H9BDu0l0b2evLVaEKjtC177wz9JFBY5GNKI2gVHF/UcNdqCSlPwDZZyQ5h/YcJifKS+dc2r90izWuuU='} 2024-08-28T19:04:07.377 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-28 18:58:59.910433', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH10zppIILtk42EmjUP+BzZ9NUblaxWorqyIKW4IpBXHv3POMxiVQgWF4EQiMaYKqdE03V9Vp4ne+l8O4yhxFw4='} 2024-08-28T19:04:07.377 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-28T19:04:07.384 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-08-28T19:04:07.384 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-08-28T19:04:07.384 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-08-28T19:04:07.384 INFO:teuthology.run_tasks:Running task console_log... 2024-08-28T19:04:07.515 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fee9760bac0>, signals=[15]) 2024-08-28T19:04:07.515 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-28T19:04:07.521 INFO:teuthology.task.internal:Opening connections... 2024-08-28T19:04:07.521 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-08-28T19:04:07.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-28T19:04:07.636 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-08-28T19:04:07.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-28T19:04:07.737 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2024-08-28T19:04:07.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-28T19:04:07.831 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-28T19:04:07.838 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-08-28T19:04:07.857 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-08-28T19:04:07.858 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-08-28T19:04:07.916 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-28T19:04:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-28T19:04:07.918 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-08-28T19:04:07.954 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-08-28T19:04:07.973 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-08-28T19:04:07.973 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-08-28T19:04:08.031 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-08-28T19:04:08.031 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-08-28T19:04:08.031 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-08-28T19:04:08.031 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-08-28T19:04:08.032 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-08-28T19:04:08.032 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-08-28T19:04:08.032 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-28T19:04:08.032 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-08-28T19:04:08.032 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-08-28T19:04:08.032 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-28T19:04:08.032 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-08-28T19:04:08.032 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-28T19:04:08.032 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-28T19:04:08.032 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-28T19:04:08.033 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-08-28T19:04:08.062 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2024-08-28T19:04:08.080 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2024-08-28T19:04:08.081 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2024-08-28T19:04:08.139 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2024-08-28T19:04:08.139 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="9" 2024-08-28T19:04:08.139 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2024-08-28T19:04:08.139 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2024-08-28T19:04:08.139 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="9" 2024-08-28T19:04:08.139 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el9" 2024-08-28T19:04:08.139 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-28T19:04:08.139 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2024-08-28T19:04:08.140 INFO:teuthology.orchestra.run.smithi115.stdout:LOGO="fedora-logo-icon" 2024-08-28T19:04:08.140 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-28T19:04:08.140 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2024-08-28T19:04:08.140 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-28T19:04:08.140 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-28T19:04:08.140 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-28T19:04:08.140 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2024-08-28T19:04:08.171 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-28T19:04:08.181 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-28T19:04:08.189 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-28T19:04:08.189 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-08-28T19:04:08.192 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-08-28T19:04:08.194 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2024-08-28T19:04:08.212 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-28T19:04:08.218 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-28T19:04:08.219 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-08-28T19:04:08.250 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-08-28T19:04:08.252 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2024-08-28T19:04:08.360 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-28T19:04:08.466 INFO:teuthology.run_tasks:Running task kernel... 2024-08-28T19:04:08.480 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-08-28T19:04:08.480 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-08-28T19:04:08.481 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-08-28T19:04:08.481 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-28T19:04:08.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T19:04:08.498 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-08-28T19:04:08.557 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-496.el9.x86_64 2024-08-28T19:04:08.557 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-496.el9.x86_64 2024-08-28T19:04:08.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-08-28T19:04:11.264 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 9.0 MB/s | 8.2 MB 00:00 2024-08-28T19:04:19.781 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 20 MB 00:07 2024-08-28T19:04:24.023 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 18 MB/s | 6.6 MB 00:00 2024-08-28T19:04:25.841 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 18 kB 00:00 2024-08-28T19:04:26.695 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 31 MB/s | 22 MB 00:00 2024-08-28T19:04:32.309 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-08-28T19:04:34.249 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-28T19:04:34.249 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-496.el9.x86_64 is already installed. 2024-08-28T19:04:34.348 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-08-28T19:04:34.349 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-08-28T19:04:34.349 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-08-28T19:04:34.472 DEBUG:teuthology.orchestra.run.smithi016:> echo no | sudo yum reinstall kernel || true 2024-08-28T19:04:35.029 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:03 ago on Wed 28 Aug 2024 07:04:32 PM UTC. 2024-08-28T19:04:35.176 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-08-28T19:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-08-28T19:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-08-28T19:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-08-28T19:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalling: 2024-08-28T19:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-496.el9 baseos 1.8 M 2024-08-28T19:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-08-28T19:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-08-28T19:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 1.8 M 2024-08-28T19:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 0 2024-08-28T19:04:35.183 INFO:teuthology.orchestra.run.smithi016.stderr:Operation aborted. 2024-08-28T19:04:35.236 INFO:teuthology.orchestra.run.smithi016.stdout:Is this ok [y/N]: 2024-08-28T19:04:35.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum reinstall -y kernel || true 2024-08-28T19:04:35.794 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:03 ago on Wed 28 Aug 2024 07:04:32 PM UTC. 2024-08-28T19:04:35.922 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-08-28T19:04:35.923 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-08-28T19:04:35.923 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-08-28T19:04:35.923 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-08-28T19:04:35.923 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalling: 2024-08-28T19:04:35.923 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-496.el9 baseos 1.8 M 2024-08-28T19:04:35.924 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:04:35.924 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-08-28T19:04:35.924 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-08-28T19:04:35.924 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:04:35.924 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 1.8 M 2024-08-28T19:04:35.924 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 0 2024-08-28T19:04:35.924 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-08-28T19:04:37.103 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-496.el9.x86_64.rpm 1.7 MB/s | 1.8 MB 00:01 2024-08-28T19:04:37.103 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-08-28T19:04:37.104 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.5 MB/s | 1.8 MB 00:01 2024-08-28T19:04:37.134 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-08-28T19:04:37.408 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-08-28T19:04:37.409 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-08-28T19:04:37.439 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-08-28T19:04:37.439 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-08-28T19:04:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-08-28T19:04:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: Reinstalling : kernel-5.14.0-496.el9.x86_64 1/2 2024-08-28T19:04:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : kernel-5.14.0-496.el9.x86_64 2/2 2024-08-28T19:04:38.422 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-496.el9.x86_64 2/2 2024-08-28T19:04:38.422 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-496.el9.x86_64 1/2 2024-08-28T19:04:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-496.el9.x86_64 2/2 2024-08-28T19:04:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:04:39.171 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalled: 2024-08-28T19:04:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-496.el9.x86_64 2024-08-28T19:04:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:04:39.171 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-08-28T19:04:39.366 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-08-28T19:04:39.464 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-496.el9.x86_64 2024-08-28T19:04:39.465 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-496.el9.x86_64 2024-08-28T19:04:39.465 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-28T19:04:39.465 DEBUG:teuthology.parallel:result is None 2024-08-28T19:04:39.465 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-28T19:04:39.472 INFO:teuthology.task.internal:Creating test directory... 2024-08-28T19:04:39.473 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-28T19:04:39.475 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-28T19:04:39.478 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-28T19:04:39.496 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-28T19:04:39.573 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-28T19:04:39.581 INFO:teuthology.task.internal:Creating archive directory... 2024-08-28T19:04:39.581 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-28T19:04:39.583 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-28T19:04:39.586 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-28T19:04:39.691 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-28T19:04:39.697 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-28T19:04:39.697 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-28T19:04:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T19:04:39.716 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-28T19:04:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T19:04:39.734 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-28T19:04:39.736 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-28T19:04:39.758 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-28T19:04:39.791 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-28T19:04:39.807 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-28T19:04:39.816 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-28T19:04:39.845 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-28T19:04:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-28T19:04:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-28T19:04:39.905 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-28T19:04:39.912 INFO:teuthology.task.internal:Configuring sudo... 2024-08-28T19:04:39.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-28T19:04:39.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-28T19:04:39.918 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-28T19:04:39.946 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-28T19:04:39.955 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-28T19:04:39.955 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-28T19:04:39.981 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-28T19:04:39.987 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-28T19:04:40.003 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-28T19:04:40.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-28T19:04:40.130 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-28T19:04:40.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-28T19:04:40.252 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:04:40.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-28T19:04:40.316 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-28T19:04:40.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-28T19:04:40.413 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-28T19:04:40.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-28T19:04:40.542 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-08-28T19:04:40.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-28T19:04:40.611 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-28T19:04:40.664 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-28T19:04:40.707 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-28T19:04:40.800 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-28T19:04:40.838 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-08-28T19:04:40.838 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-28T19:04:40.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-08-28T19:04:40.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-08-28T19:04:40.911 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2024-08-28T19:04:40.947 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-28T19:04:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-28T19:04:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-28T19:04:41.403 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-28T19:04:41.410 INFO:teuthology.task.internal:Starting timer... 2024-08-28T19:04:41.410 INFO:teuthology.run_tasks:Running task pcp... 2024-08-28T19:04:41.423 INFO:teuthology.run_tasks:Running task selinux... 2024-08-28T19:04:41.432 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-08-28T19:04:41.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-08-28T19:04:41.486 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-08-28T19:04:41.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-08-28T19:04:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-08-28T19:04:41.541 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2024-08-28T19:04:41.618 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2024-08-28T19:04:41.620 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-28T19:04:41.620 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-08-28T19:04:41.642 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-08-28T19:04:41.643 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-08-28T19:04:41.681 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-08-28T19:04:41.681 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2024-08-28T19:04:41.712 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2024-08-28T19:04:41.712 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi115.front.sepia.ceph.com': 'permissive'} 2024-08-28T19:04:41.713 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-28T19:04:41.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T19:04:41.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-28T19:04:41.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T19:04:41.767 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-28T19:04:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T19:04:41.796 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-28T19:04:41.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-08-28T19:04:41.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-08-28T19:04:41.847 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2024-08-28T19:04:41.881 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-28T19:04:41.891 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-28T19:04:41.902 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-28T19:04:41.903 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi028.front.sepia.ceph.com,smithi115.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-28T19:06:45.351 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi115.front.sepia.ceph.com')] 2024-08-28T19:06:45.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-08-28T19:06:45.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-28T19:06:45.442 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-08-28T19:06:45.528 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-08-28T19:06:45.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-08-28T19:06:45.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-28T19:06:45.627 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-08-28T19:06:45.705 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-08-28T19:06:45.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-08-28T19:06:45.705 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-28T19:06:45.784 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-08-28T19:06:45.868 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-08-28T19:06:45.869 INFO:teuthology.run_tasks:Running task clock... 2024-08-28T19:06:45.879 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-28T19:06:45.879 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-28T19:06:45.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-28T19:06:45.882 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-28T19:06:45.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-28T19:06:45.884 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-28T19:06:45.884 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-28T19:06:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-28T19:06:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-28T19:06:45.925 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-28T19:06:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-28T19:06:45.939 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-28T19:06:45.951 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-08-28T19:06:45.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-08-28T19:06:45.954 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-28T19:06:45.962 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-08-28T19:06:45.965 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-08-28T19:06:45.977 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-28T19:06:45.978 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2024-08-28T19:06:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-28T19:06:45.989 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2024-08-28T19:06:45.991 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-28T19:06:45.993 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-28T19:06:46.003 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-28T19:06:46.017 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-28T19:06:46.049 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-08-28T19:06:46.053 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-28T19:06:46.053 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-08-28T19:06:46.053 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.053 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.053 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.053 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-08-28T19:06:46.059 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-28T19:06:46.059 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-08-28T19:06:46.059 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.059 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.059 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.059 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.080 INFO:teuthology.orchestra.run.smithi115.stderr:bash: line 1: ntpq: command not found 2024-08-28T19:06:46.083 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-28T19:06:46.083 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-08-28T19:06:46.083 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.083 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.083 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.083 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T19:06:46.084 INFO:teuthology.run_tasks:Running task install... 2024-08-28T19:06:46.093 DEBUG:teuthology.task.install:project ceph 2024-08-28T19:06:46.093 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6'}} 2024-08-28T19:06:46.093 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6'} 2024-08-28T19:06:46.093 INFO:teuthology.task.install:Using flavor: default 2024-08-28T19:06:46.098 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-28T19:06:46.098 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-08-28T19:06:46.099 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6', 'tag': None, 'wait_for_package': False} 2024-08-28T19:06:46.099 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:06:46.101 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6', 'tag': None, 'wait_for_package': False} 2024-08-28T19:06:46.101 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:06:46.102 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6', 'tag': None, 'wait_for_package': False} 2024-08-28T19:06:46.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:06:46.253 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/ 2024-08-28T19:06:46.253 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4560.gd19e6118 2024-08-28T19:06:46.267 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/ 2024-08-28T19:06:46.267 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4560.gd19e6118 2024-08-28T19:06:46.278 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/ 2024-08-28T19:06:46.279 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4560.gd19e6118 2024-08-28T19:06:46.408 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-28T19:06:46.409 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:06:46.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-28T19:06:46.415 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-28T19:06:46.415 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-08-28T19:06:46.415 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-28T19:06:46.422 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-28T19:06:46.422 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-08-28T19:06:46.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-28T19:06:46.434 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-08-28T19:06:46.434 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d19e6118f7015f0047264006cb0c873528eb3ae6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-28T19:06:46.441 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-08-28T19:06:46.441 DEBUG:teuthology.orchestra.run.smithi115:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d19e6118f7015f0047264006cb0c873528eb3ae6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-28T19:06:46.449 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-08-28T19:06:46.449 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d19e6118f7015f0047264006cb0c873528eb3ae6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-28T19:06:46.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-28T19:06:46.508 DEBUG:teuthology.orchestra.run.smithi115:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-28T19:06:46.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-28T19:06:46.575 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-28T19:06:46.584 DEBUG:teuthology.orchestra.run.smithi115:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-28T19:06:46.590 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-28T19:06:46.639 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-08-28T19:06:46.641 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-08-28T19:06:46.648 INFO:teuthology.orchestra.run.smithi115.stdout:check_obsoletes = 1 2024-08-28T19:06:46.649 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum clean all 2024-08-28T19:06:46.654 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-08-28T19:06:46.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-08-28T19:06:46.871 INFO:teuthology.orchestra.run.smithi016.stdout:45 files removed 2024-08-28T19:06:46.871 INFO:teuthology.orchestra.run.smithi115.stdout:45 files removed 2024-08-28T19:06:46.887 INFO:teuthology.orchestra.run.smithi028.stdout:45 files removed 2024-08-28T19:06:46.898 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-08-28T19:06:46.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-08-28T19:06:46.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-08-28T19:06:47.749 INFO:teuthology.orchestra.run.smithi115.stdout:ceph packages for x86_64 146 kB/s | 84 kB 00:00 2024-08-28T19:06:47.755 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 147 kB/s | 84 kB 00:00 2024-08-28T19:06:47.794 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 141 kB/s | 84 kB 00:00 2024-08-28T19:06:48.358 INFO:teuthology.orchestra.run.smithi115.stdout:ceph noarch packages 42 kB/s | 24 kB 00:00 2024-08-28T19:06:48.398 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 40 kB/s | 24 kB 00:00 2024-08-28T19:06:48.403 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 43 kB/s | 24 kB 00:00 2024-08-28T19:06:49.042 INFO:teuthology.orchestra.run.smithi115.stdout:ceph source packages 3.0 kB/s | 1.9 kB 00:00 2024-08-28T19:06:49.065 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-08-28T19:06:49.070 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-08-28T19:06:49.682 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-08-28T19:06:50.608 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.2 MB 00:01 2024-08-28T19:06:50.970 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-08-28T19:06:52.634 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-08-28T19:06:55.063 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 6.8 MB/s | 20 MB 00:02 2024-08-28T19:06:56.008 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 20 MB 00:05 2024-08-28T19:06:57.893 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.6 MB 00:01 2024-08-28T19:06:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - Extras packages 90 kB/s | 18 kB 00:00 2024-08-28T19:06:59.522 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.6 MB 00:00 2024-08-28T19:07:00.173 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 31 MB/s | 22 MB 00:00 2024-08-28T19:07:00.500 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.6 MB 00:00 2024-08-28T19:07:01.164 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 18 kB 00:00 2024-08-28T19:07:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 34 MB/s | 22 MB 00:00 2024-08-28T19:07:02.368 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 18 kB 00:00 2024-08-28T19:07:03.247 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 30 MB/s | 22 MB 00:00 2024-08-28T19:07:05.603 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-08-28T19:07:07.566 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-08-28T19:07:07.774 INFO:teuthology.orchestra.run.smithi115.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-28T19:07:07.775 INFO:teuthology.orchestra.run.smithi115.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-28T19:07:07.788 INFO:teuthology.orchestra.run.smithi115.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-08-28T19:07:07.789 INFO:teuthology.orchestra.run.smithi115.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-08-28T19:07:07.790 INFO:teuthology.orchestra.run.smithi115.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-08-28T19:07:07.791 INFO:teuthology.orchestra.run.smithi115.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-08-28T19:07:07.792 INFO:teuthology.orchestra.run.smithi115.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-08-28T19:07:07.795 INFO:teuthology.orchestra.run.smithi115.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-08-28T19:07:07.796 INFO:teuthology.orchestra.run.smithi115.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-08-28T19:07:07.856 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-08-28T19:07:07.879 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================= 2024-08-28T19:07:07.879 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-08-28T19:07:07.879 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================= 2024-08-28T19:07:07.879 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-08-28T19:07:07.879 INFO:teuthology.orchestra.run.smithi115.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 6.6 k 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 5.5 M 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 874 k 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 148 k 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 1.5 M 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 154 k 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 14 M 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 7.4 M 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 49 k 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 10 M 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 44 M 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 285 k 2024-08-28T19:07:07.880 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 781 k 2024-08-28T19:07:07.881 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-mirror x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 226 k 2024-08-28T19:07:07.881 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-shell x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 27 k 2024-08-28T19:07:07.881 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-top noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 18 k 2024-08-28T19:07:07.881 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-08-28T19:07:07.881 INFO:teuthology.orchestra.run.smithi115.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-08-28T19:07:07.881 INFO:teuthology.orchestra.run.smithi115.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-08-28T19:07:07.881 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 32 k 2024-08-28T19:07:07.881 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 766 k 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 127 k 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 161 k 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 321 k 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 305 k 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 99 k 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 86 k 2024-08-28T19:07:07.882 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.1 M 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 170 k 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.4 M 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.1 M 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 20 M 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 34 k 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 2.3 M 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 290 k 2024-08-28T19:07:07.883 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 4.9 M 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 18 M 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 16 k 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 25 k 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-28T19:07:07.884 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 159 k 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 506 k 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 5.3 M 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-28T19:07:07.885 INFO:teuthology.orchestra.run.smithi115.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: openblas x86_64 0.3.26-1.el9 appstream 38 k 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-openmp x86_64 0.3.26-1.el9 appstream 4.9 M 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-28T19:07:07.886 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 45 k 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 141 k 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-devel x86_64 3.9.19-7.el9 appstream 245 k 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-28T19:07:07.887 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-28T19:07:07.888 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-28T19:07:07.889 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-28T19:07:07.890 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout:Installing weak dependencies: 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================= 2024-08-28T19:07:07.891 INFO:teuthology.orchestra.run.smithi115.stdout:Install 148 Packages 2024-08-28T19:07:07.892 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 2 Packages 2024-08-28T19:07:07.892 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:07:07.892 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 224 M 2024-08-28T19:07:07.892 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-08-28T19:07:08.421 INFO:teuthology.orchestra.run.smithi115.stdout:(1/150): ceph-19.3.0-4560.gd19e6118.el9.x86_64. 43 kB/s | 6.6 kB 00:00 2024-08-28T19:07:08.622 INFO:teuthology.orchestra.run.smithi115.stdout:(2/150): ceph-fuse-19.3.0-4560.gd19e6118.el9.x8 4.3 MB/s | 874 kB 00:00 2024-08-28T19:07:08.672 INFO:teuthology.orchestra.run.smithi115.stdout:(3/150): ceph-immutable-object-cache-19.3.0-456 2.9 MB/s | 148 kB 00:00 2024-08-28T19:07:08.828 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-28T19:07:08.872 INFO:teuthology.orchestra.run.smithi115.stdout:(4/150): ceph-base-19.3.0-4560.gd19e6118.el9.x8 9.1 MB/s | 5.5 MB 00:00 2024-08-28T19:07:08.922 INFO:teuthology.orchestra.run.smithi115.stdout:(5/150): ceph-mds-19.3.0-4560.gd19e6118.el9.x86 9.1 MB/s | 2.3 MB 00:00 2024-08-28T19:07:09.014 INFO:teuthology.orchestra.run.smithi115.stdout:(6/150): ceph-mgr-19.3.0-4560.gd19e6118.el9.x86 11 MB/s | 1.5 MB 00:00 2024-08-28T19:07:09.365 INFO:teuthology.orchestra.run.smithi115.stdout:(7/150): ceph-mon-19.3.0-4560.gd19e6118.el9.x86 11 MB/s | 4.9 MB 00:00 2024-08-28T19:07:09.756 INFO:teuthology.orchestra.run.smithi115.stdout:(8/150): ceph-common-19.3.0-4560.gd19e6118.el9. 13 MB/s | 20 MB 00:01 2024-08-28T19:07:09.786 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-28T19:07:09.787 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-28T19:07:09.798 INFO:teuthology.orchestra.run.smithi115.stdout:(9/150): ceph-selinux-19.3.0-4560.gd19e6118.el9 607 kB/s | 25 kB 00:00 2024-08-28T19:07:09.801 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-08-28T19:07:09.802 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-08-28T19:07:09.803 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-08-28T19:07:09.804 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-08-28T19:07:09.806 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-08-28T19:07:09.809 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-08-28T19:07:09.810 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-08-28T19:07:09.876 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-08-28T19:07:09.901 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-08-28T19:07:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-08-28T19:07:09.901 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-08-28T19:07:09.901 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-08-28T19:07:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-08-28T19:07:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 6.6 k 2024-08-28T19:07:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 5.5 M 2024-08-28T19:07:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 874 k 2024-08-28T19:07:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 148 k 2024-08-28T19:07:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 1.5 M 2024-08-28T19:07:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 154 k 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 14 M 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 7.4 M 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 49 k 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 10 M 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 44 M 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 285 k 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 781 k 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 226 k 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 27 k 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 18 k 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-08-28T19:07:09.902 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 32 k 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 766 k 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 127 k 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 161 k 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 321 k 2024-08-28T19:07:09.903 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 305 k 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 99 k 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 86 k 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.1 M 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 170 k 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.4 M 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.1 M 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-28T19:07:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 20 M 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 34 k 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 2.3 M 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 290 k 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 4.9 M 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 18 M 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 16 k 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 25 k 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-28T19:07:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 159 k 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 506 k 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 5.3 M 2024-08-28T19:07:09.906 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.26-1.el9 appstream 38 k 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.26-1.el9 appstream 4.9 M 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-28T19:07:09.907 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 45 k 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 141 k 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-08-28T19:07:09.908 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.19-7.el9 appstream 245 k 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-28T19:07:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-28T19:07:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-28T19:07:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-08-28T19:07:09.912 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout:Install 148 Packages 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 2 Packages 2024-08-28T19:07:09.913 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:07:09.914 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 224 M 2024-08-28T19:07:09.914 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-08-28T19:07:10.123 INFO:teuthology.orchestra.run.smithi115.stdout:(10/150): ceph-osd-19.3.0-4560.gd19e6118.el9.x8 16 MB/s | 18 MB 00:01 2024-08-28T19:07:10.240 INFO:teuthology.orchestra.run.smithi115.stdout:(11/150): ceph-radosgw-19.3.0-4560.gd19e6118.el 12 MB/s | 10 MB 00:00 2024-08-28T19:07:10.266 INFO:teuthology.orchestra.run.smithi115.stdout:(12/150): cephfs-mirror-19.3.0-4560.gd19e6118.e 1.6 MB/s | 226 kB 00:00 2024-08-28T19:07:10.284 INFO:teuthology.orchestra.run.smithi115.stdout:(13/150): cephfs-shell-19.3.0-4560.gd19e6118.el 645 kB/s | 27 kB 00:00 2024-08-28T19:07:10.300 INFO:teuthology.orchestra.run.smithi115.stdout:(14/150): libcephfs-devel-19.3.0-4560.gd19e6118 944 kB/s | 32 kB 00:00 2024-08-28T19:07:10.350 INFO:teuthology.orchestra.run.smithi115.stdout:(15/150): libcephfs2-19.3.0-4560.gd19e6118.el9. 11 MB/s | 766 kB 00:00 2024-08-28T19:07:10.375 INFO:teuthology.orchestra.run.smithi115.stdout:(16/150): libcephsqlite-19.3.0-4560.gd19e6118.e 2.1 MB/s | 159 kB 00:00 2024-08-28T19:07:10.409 INFO:teuthology.orchestra.run.smithi115.stdout:(17/150): librados-devel-19.3.0-4560.gd19e6118. 2.1 MB/s | 127 kB 00:00 2024-08-28T19:07:10.442 INFO:teuthology.orchestra.run.smithi115.stdout:(18/150): libradosstriper1-19.3.0-4560.gd19e611 7.4 MB/s | 506 kB 00:00 2024-08-28T19:07:10.509 INFO:teuthology.orchestra.run.smithi115.stdout:(19/150): python3-ceph-argparse-19.3.0-4560.gd1 678 kB/s | 45 kB 00:00 2024-08-28T19:07:10.593 INFO:teuthology.orchestra.run.smithi115.stdout:(20/150): python3-ceph-common-19.3.0-4560.gd19e 1.6 MB/s | 141 kB 00:00 2024-08-28T19:07:10.677 INFO:teuthology.orchestra.run.smithi115.stdout:(21/150): python3-cephfs-19.3.0-4560.gd19e6118. 1.9 MB/s | 161 kB 00:00 2024-08-28T19:07:10.761 INFO:teuthology.orchestra.run.smithi115.stdout:(22/150): librgw2-19.3.0-4560.gd19e6118.el9.x86 15 MB/s | 5.3 MB 00:00 2024-08-28T19:07:10.785 INFO:teuthology.orchestra.run.smithi115.stdout:(23/150): python3-rados-19.3.0-4560.gd19e6118.e 2.9 MB/s | 321 kB 00:00 2024-08-28T19:07:10.792 INFO:teuthology.orchestra.run.smithi028.stdout:(1/150): ceph-19.3.0-4560.gd19e6118.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-08-28T19:07:10.811 INFO:teuthology.orchestra.run.smithi115.stdout:(24/150): python3-rbd-19.3.0-4560.gd19e6118.el9 6.0 MB/s | 305 kB 00:00 2024-08-28T19:07:10.836 INFO:teuthology.orchestra.run.smithi115.stdout:(25/150): python3-rgw-19.3.0-4560.gd19e6118.el9 1.9 MB/s | 99 kB 00:00 2024-08-28T19:07:10.861 INFO:teuthology.orchestra.run.smithi115.stdout:(26/150): rbd-fuse-19.3.0-4560.gd19e6118.el9.x8 1.7 MB/s | 86 kB 00:00 2024-08-28T19:07:10.920 INFO:teuthology.orchestra.run.smithi115.stdout:(27/150): rbd-nbd-19.3.0-4560.gd19e6118.el9.x86 2.8 MB/s | 170 kB 00:00 2024-08-28T19:07:10.996 INFO:teuthology.orchestra.run.smithi115.stdout:(28/150): rbd-mirror-19.3.0-4560.gd19e6118.el9. 19 MB/s | 3.1 MB 00:00 2024-08-28T19:07:11.021 INFO:teuthology.orchestra.run.smithi115.stdout:(29/150): ceph-grafana-dashboards-19.3.0-4560.g 342 kB/s | 34 kB 00:00 2024-08-28T19:07:11.054 INFO:teuthology.orchestra.run.smithi115.stdout:(30/150): ceph-mgr-cephadm-19.3.0-4560.gd19e611 2.6 MB/s | 154 kB 00:00 2024-08-28T19:07:11.075 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-28T19:07:11.076 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-28T19:07:11.090 INFO:teuthology.orchestra.run.smithi016.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-08-28T19:07:11.091 INFO:teuthology.orchestra.run.smithi016.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-08-28T19:07:11.092 INFO:teuthology.orchestra.run.smithi016.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-08-28T19:07:11.093 INFO:teuthology.orchestra.run.smithi016.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-08-28T19:07:11.095 INFO:teuthology.orchestra.run.smithi016.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-08-28T19:07:11.098 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-08-28T19:07:11.099 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-08-28T19:07:11.164 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-08-28T19:07:11.173 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-08-28T19:07:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-08-28T19:07:11.173 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-08-28T19:07:11.173 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-08-28T19:07:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-08-28T19:07:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 6.6 k 2024-08-28T19:07:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 5.5 M 2024-08-28T19:07:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 874 k 2024-08-28T19:07:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 148 k 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 1.5 M 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 154 k 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 14 M 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 7.4 M 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 49 k 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 10 M 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 44 M 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 285 k 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 781 k 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 226 k 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 27 k 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 18 k 2024-08-28T19:07:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 32 k 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 766 k 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 127 k 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 161 k 2024-08-28T19:07:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 321 k 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 305 k 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 99 k 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 86 k 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.1 M 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 170 k 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.4 M 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.1 M 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-08-28T19:07:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 20 M 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 34 k 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 2.3 M 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 290 k 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 4.9 M 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 18 M 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 16 k 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 25 k 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-28T19:07:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 159 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 506 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 5.3 M 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-08-28T19:07:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.26-1.el9 appstream 38 k 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.26-1.el9 appstream 4.9 M 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-28T19:07:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 45 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 141 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.19-7.el9 appstream 245 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-28T19:07:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-28T19:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-28T19:07:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-28T19:07:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-08-28T19:07:11.184 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-08-28T19:07:11.185 INFO:teuthology.orchestra.run.smithi016.stdout:Install 148 Packages 2024-08-28T19:07:11.185 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 2 Packages 2024-08-28T19:07:11.185 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:07:11.185 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 224 M 2024-08-28T19:07:11.185 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-08-28T19:07:11.201 INFO:teuthology.orchestra.run.smithi028.stdout:(2/150): ceph-fuse-19.3.0-4560.gd19e6118.el9.x8 2.1 MB/s | 874 kB 00:00 2024-08-28T19:07:11.293 INFO:teuthology.orchestra.run.smithi028.stdout:(3/150): ceph-immutable-object-cache-19.3.0-456 1.6 MB/s | 148 kB 00:00 2024-08-28T19:07:11.605 INFO:teuthology.orchestra.run.smithi115.stdout:(31/150): ceph-mgr-diskprediction-local-19.3.0- 13 MB/s | 7.4 MB 00:00 2024-08-28T19:07:11.688 INFO:teuthology.orchestra.run.smithi115.stdout:(32/150): ceph-mgr-modules-core-19.3.0-4560.gd1 3.4 MB/s | 290 kB 00:00 2024-08-28T19:07:11.763 INFO:teuthology.orchestra.run.smithi115.stdout:(33/150): ceph-mgr-rook-19.3.0-4560.gd19e6118.e 659 kB/s | 49 kB 00:00 2024-08-28T19:07:11.860 INFO:teuthology.orchestra.run.smithi028.stdout:(4/150): ceph-mds-19.3.0-4560.gd19e6118.el9.x86 4.0 MB/s | 2.3 MB 00:00 2024-08-28T19:07:11.871 INFO:teuthology.orchestra.run.smithi016.stdout:(1/150): ceph-19.3.0-4560.gd19e6118.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-08-28T19:07:11.960 INFO:teuthology.orchestra.run.smithi028.stdout:(5/150): ceph-base-19.3.0-4560.gd19e6118.el9.x8 4.1 MB/s | 5.5 MB 00:01 2024-08-28T19:07:12.110 INFO:teuthology.orchestra.run.smithi028.stdout:(6/150): ceph-mgr-19.3.0-4560.gd19e6118.el9.x86 6.1 MB/s | 1.5 MB 00:00 2024-08-28T19:07:12.155 INFO:teuthology.orchestra.run.smithi115.stdout:(34/150): ceph-test-19.3.0-4560.gd19e6118.el9.x 19 MB/s | 44 MB 00:02 2024-08-28T19:07:12.172 INFO:teuthology.orchestra.run.smithi115.stdout:(35/150): ceph-prometheus-alerts-19.3.0-4560.gd 40 kB/s | 16 kB 00:00 2024-08-28T19:07:12.196 INFO:teuthology.orchestra.run.smithi016.stdout:(2/150): ceph-fuse-19.3.0-4560.gd19e6118.el9.x8 2.6 MB/s | 874 kB 00:00 2024-08-28T19:07:12.280 INFO:teuthology.orchestra.run.smithi016.stdout:(3/150): ceph-immutable-object-cache-19.3.0-456 1.7 MB/s | 148 kB 00:00 2024-08-28T19:07:12.314 INFO:teuthology.orchestra.run.smithi115.stdout:(36/150): ceph-mgr-dashboard-19.3.0-4560.gd19e6 11 MB/s | 14 MB 00:01 2024-08-28T19:07:12.339 INFO:teuthology.orchestra.run.smithi115.stdout:(37/150): ceph-volume-19.3.0-4560.gd19e6118.el9 1.5 MB/s | 285 kB 00:00 2024-08-28T19:07:12.356 INFO:teuthology.orchestra.run.smithi115.stdout:(38/150): cephfs-top-19.3.0-4560.gd19e6118.el9. 441 kB/s | 18 kB 00:00 2024-08-28T19:07:12.372 INFO:teuthology.orchestra.run.smithi016.stdout:(4/150): ceph-base-19.3.0-4560.gd19e6118.el9.x8 8.2 MB/s | 5.5 MB 00:00 2024-08-28T19:07:12.390 INFO:teuthology.orchestra.run.smithi115.stdout:(39/150): cephadm-19.3.0-4560.gd19e6118.el9.noa 3.5 MB/s | 781 kB 00:00 2024-08-28T19:07:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:(7/150): ceph-mon-19.3.0-4560.gd19e6118.el9.x86 11 MB/s | 4.9 MB 00:00 2024-08-28T19:07:12.497 INFO:teuthology.orchestra.run.smithi016.stdout:(5/150): ceph-mgr-19.3.0-4560.gd19e6118.el9.x86 12 MB/s | 1.5 MB 00:00 2024-08-28T19:07:12.524 INFO:teuthology.orchestra.run.smithi115.stdout:(40/150): fuse-2.9.9-16.el9.x86_64.rpm 433 kB/s | 80 kB 00:00 2024-08-28T19:07:12.607 INFO:teuthology.orchestra.run.smithi115.stdout:(41/150): libquadmath-11.5.0-2.el9.x86_64.rpm 881 kB/s | 191 kB 00:00 2024-08-28T19:07:12.666 INFO:teuthology.orchestra.run.smithi115.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-08-28T19:07:12.700 INFO:teuthology.orchestra.run.smithi115.stdout:(43/150): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-08-28T19:07:12.717 INFO:teuthology.orchestra.run.smithi115.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-08-28T19:07:12.750 INFO:teuthology.orchestra.run.smithi115.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-08-28T19:07:12.776 INFO:teuthology.orchestra.run.smithi115.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-08-28T19:07:12.801 INFO:teuthology.orchestra.run.smithi115.stdout:(47/150): python3-urllib3-1.26.5-6.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-08-28T19:07:12.885 INFO:teuthology.orchestra.run.smithi115.stdout:(48/150): python3-cryptography-36.0.1-4.el9.x86 4.4 MB/s | 1.2 MB 00:00 2024-08-28T19:07:12.910 INFO:teuthology.orchestra.run.smithi115.stdout:(49/150): automake-1.16.2-8.el9.noarch.rpm 5.0 MB/s | 689 kB 00:00 2024-08-28T19:07:12.956 INFO:teuthology.orchestra.run.smithi016.stdout:(6/150): ceph-mon-19.3.0-4560.gd19e6118.el9.x86 11 MB/s | 4.9 MB 00:00 2024-08-28T19:07:12.960 INFO:teuthology.orchestra.run.smithi115.stdout:(50/150): elfutils-libelf-devel-0.191-4.el9.x86 434 kB/s | 22 kB 00:00 2024-08-28T19:07:12.977 INFO:teuthology.orchestra.run.smithi115.stdout:(51/150): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-08-28T19:07:13.006 INFO:teuthology.orchestra.run.smithi016.stdout:(7/150): ceph-mds-19.3.0-4560.gd19e6118.el9.x86 3.1 MB/s | 2.3 MB 00:00 2024-08-28T19:07:13.019 INFO:teuthology.orchestra.run.smithi115.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-08-28T19:07:13.044 INFO:teuthology.orchestra.run.smithi115.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 488 kB/s | 33 kB 00:00 2024-08-28T19:07:13.086 INFO:teuthology.orchestra.run.smithi115.stdout:(54/150): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 943 kB 00:00 2024-08-28T19:07:13.111 INFO:teuthology.orchestra.run.smithi115.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 272 kB/s | 18 kB 00:00 2024-08-28T19:07:13.195 INFO:teuthology.orchestra.run.smithi115.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 103 kB/s | 11 kB 00:00 2024-08-28T19:07:13.229 INFO:teuthology.orchestra.run.smithi115.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-08-28T19:07:13.254 INFO:teuthology.orchestra.run.smithi115.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 772 kB/s | 45 kB 00:00 2024-08-28T19:07:13.310 INFO:teuthology.orchestra.run.smithi028.stdout:(8/150): ceph-radosgw-19.3.0-4560.gd19e6118.el9 12 MB/s | 10 MB 00:00 2024-08-28T19:07:13.360 INFO:teuthology.orchestra.run.smithi028.stdout:(9/150): ceph-selinux-19.3.0-4560.gd19e6118.el9 506 kB/s | 25 kB 00:00 2024-08-28T19:07:13.362 INFO:teuthology.orchestra.run.smithi115.stdout:(59/150): libtool-2.4.6-46.el9.x86_64.rpm 5.2 MB/s | 581 kB 00:00 2024-08-28T19:07:13.365 INFO:teuthology.orchestra.run.smithi016.stdout:(8/150): ceph-common-19.3.0-4560.gd19e6118.el9. 12 MB/s | 20 MB 00:01 2024-08-28T19:07:13.396 INFO:teuthology.orchestra.run.smithi115.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-08-28T19:07:13.430 INFO:teuthology.orchestra.run.smithi115.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-08-28T19:07:13.448 INFO:teuthology.orchestra.run.smithi016.stdout:(9/150): ceph-selinux-19.3.0-4560.gd19e6118.el9 303 kB/s | 25 kB 00:00 2024-08-28T19:07:13.472 INFO:teuthology.orchestra.run.smithi115.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-08-28T19:07:13.505 INFO:teuthology.orchestra.run.smithi115.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-08-28T19:07:13.556 INFO:teuthology.orchestra.run.smithi115.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-08-28T19:07:13.580 INFO:teuthology.orchestra.run.smithi115.stdout:(65/150): openblas-0.3.26-1.el9.x86_64.rpm 510 kB/s | 38 kB 00:00 2024-08-28T19:07:13.764 INFO:teuthology.orchestra.run.smithi115.stdout:(66/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.0 MB/s | 3.0 MB 00:00 2024-08-28T19:07:13.919 INFO:teuthology.orchestra.run.smithi028.stdout:(10/150): ceph-osd-19.3.0-4560.gd19e6118.el9.x8 9.9 MB/s | 18 MB 00:01 2024-08-28T19:07:13.986 INFO:teuthology.orchestra.run.smithi028.stdout:(11/150): cephfs-mirror-19.3.0-4560.gd19e6118.e 3.3 MB/s | 226 kB 00:00 2024-08-28T19:07:14.045 INFO:teuthology.orchestra.run.smithi028.stdout:(12/150): cephfs-shell-19.3.0-4560.gd19e6118.el 463 kB/s | 27 kB 00:00 2024-08-28T19:07:14.112 INFO:teuthology.orchestra.run.smithi028.stdout:(13/150): libcephfs-devel-19.3.0-4560.gd19e6118 477 kB/s | 32 kB 00:00 2024-08-28T19:07:14.182 INFO:teuthology.orchestra.run.smithi016.stdout:(10/150): ceph-osd-19.3.0-4560.gd19e6118.el9.x8 15 MB/s | 18 MB 00:01 2024-08-28T19:07:14.228 INFO:teuthology.orchestra.run.smithi028.stdout:(14/150): libcephfs2-19.3.0-4560.gd19e6118.el9. 6.4 MB/s | 766 kB 00:00 2024-08-28T19:07:14.240 INFO:teuthology.orchestra.run.smithi016.stdout:(11/150): cephfs-mirror-19.3.0-4560.gd19e6118.e 3.8 MB/s | 226 kB 00:00 2024-08-28T19:07:14.278 INFO:teuthology.orchestra.run.smithi028.stdout:(15/150): libcephsqlite-19.3.0-4560.gd19e6118.e 3.1 MB/s | 159 kB 00:00 2024-08-28T19:07:14.282 INFO:teuthology.orchestra.run.smithi016.stdout:(12/150): cephfs-shell-19.3.0-4560.gd19e6118.el 648 kB/s | 27 kB 00:00 2024-08-28T19:07:14.289 INFO:teuthology.orchestra.run.smithi115.stdout:(67/150): postgresql-13.16-1.el9.x86_64.rpm 3.0 MB/s | 1.6 MB 00:00 2024-08-28T19:07:14.337 INFO:teuthology.orchestra.run.smithi028.stdout:(16/150): librados-devel-19.3.0-4560.gd19e6118. 2.1 MB/s | 127 kB 00:00 2024-08-28T19:07:14.341 INFO:teuthology.orchestra.run.smithi016.stdout:(13/150): libcephfs-devel-19.3.0-4560.gd19e6118 544 kB/s | 32 kB 00:00 2024-08-28T19:07:14.425 INFO:teuthology.orchestra.run.smithi016.stdout:(14/150): libcephfs2-19.3.0-4560.gd19e6118.el9. 8.9 MB/s | 766 kB 00:00 2024-08-28T19:07:14.440 INFO:teuthology.orchestra.run.smithi115.stdout:(68/150): postgresql-contrib-13.16-1.el9.x86_64 5.9 MB/s | 910 kB 00:00 2024-08-28T19:07:14.475 INFO:teuthology.orchestra.run.smithi016.stdout:(15/150): libcephsqlite-19.3.0-4560.gd19e6118.e 3.1 MB/s | 159 kB 00:00 2024-08-28T19:07:14.482 INFO:teuthology.orchestra.run.smithi115.stdout:(69/150): postgresql-private-libs-13.16-1.el9.x 3.2 MB/s | 137 kB 00:00 2024-08-28T19:07:14.525 INFO:teuthology.orchestra.run.smithi016.stdout:(16/150): librados-devel-19.3.0-4560.gd19e6118. 2.5 MB/s | 127 kB 00:00 2024-08-28T19:07:14.541 INFO:teuthology.orchestra.run.smithi115.stdout:(70/150): openssl-devel-3.2.2-2.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-08-28T19:07:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:(17/150): ceph-common-19.3.0-4560.gd19e6118.el9 5.1 MB/s | 20 MB 00:03 2024-08-28T19:07:14.576 INFO:teuthology.orchestra.run.smithi016.stdout:(17/150): libradosstriper1-19.3.0-4560.gd19e611 9.9 MB/s | 506 kB 00:00 2024-08-28T19:07:14.579 INFO:teuthology.orchestra.run.smithi028.stdout:(18/150): libradosstriper1-19.3.0-4560.gd19e611 2.0 MB/s | 506 kB 00:00 2024-08-28T19:07:14.655 INFO:teuthology.orchestra.run.smithi028.stdout:(19/150): python3-ceph-argparse-19.3.0-4560.gd1 602 kB/s | 45 kB 00:00 2024-08-28T19:07:14.666 INFO:teuthology.orchestra.run.smithi115.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-08-28T19:07:14.716 INFO:teuthology.orchestra.run.smithi115.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-08-28T19:07:14.730 INFO:teuthology.orchestra.run.smithi028.stdout:(20/150): python3-ceph-common-19.3.0-4560.gd19e 1.8 MB/s | 141 kB 00:00 2024-08-28T19:07:14.814 INFO:teuthology.orchestra.run.smithi028.stdout:(21/150): python3-cephfs-19.3.0-4560.gd19e6118. 1.9 MB/s | 161 kB 00:00 2024-08-28T19:07:14.866 INFO:teuthology.orchestra.run.smithi115.stdout:(73/150): postgresql-server-13.16-1.el9.x86_64. 15 MB/s | 5.8 MB 00:00 2024-08-28T19:07:14.889 INFO:teuthology.orchestra.run.smithi028.stdout:(22/150): python3-rados-19.3.0-4560.gd19e6118.e 4.2 MB/s | 321 kB 00:00 2024-08-28T19:07:14.900 INFO:teuthology.orchestra.run.smithi115.stdout:(74/150): python3-colorama-0.4.6-3.el9.noarch.r 1.8 MB/s | 61 kB 00:00 2024-08-28T19:07:14.918 INFO:teuthology.orchestra.run.smithi016.stdout:(18/150): librgw2-19.3.0-4560.gd19e6118.el9.x86 15 MB/s | 5.3 MB 00:00 2024-08-28T19:07:14.956 INFO:teuthology.orchestra.run.smithi028.stdout:(23/150): python3-rbd-19.3.0-4560.gd19e6118.el9 4.5 MB/s | 305 kB 00:00 2024-08-28T19:07:15.008 INFO:teuthology.orchestra.run.smithi115.stdout:(75/150): python3-devel-3.9.19-7.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-08-28T19:07:15.023 INFO:teuthology.orchestra.run.smithi028.stdout:(24/150): python3-rgw-19.3.0-4560.gd19e6118.el9 1.5 MB/s | 99 kB 00:00 2024-08-28T19:07:15.051 INFO:teuthology.orchestra.run.smithi016.stdout:(19/150): ceph-radosgw-19.3.0-4560.gd19e6118.el 5.1 MB/s | 10 MB 00:02 2024-08-28T19:07:15.077 INFO:teuthology.orchestra.run.smithi016.stdout:(20/150): python3-ceph-argparse-19.3.0-4560.gd1 285 kB/s | 45 kB 00:00 2024-08-28T19:07:15.090 INFO:teuthology.orchestra.run.smithi028.stdout:(25/150): rbd-fuse-19.3.0-4560.gd19e6118.el9.x8 1.3 MB/s | 86 kB 00:00 2024-08-28T19:07:15.110 INFO:teuthology.orchestra.run.smithi016.stdout:(21/150): python3-ceph-common-19.3.0-4560.gd19e 2.4 MB/s | 141 kB 00:00 2024-08-28T19:07:15.127 INFO:teuthology.orchestra.run.smithi016.stdout:(22/150): python3-cephfs-19.3.0-4560.gd19e6118. 3.1 MB/s | 161 kB 00:00 2024-08-28T19:07:15.159 INFO:teuthology.orchestra.run.smithi115.stdout:(76/150): openblas-openmp-0.3.26-1.el9.x86_64.r 3.1 MB/s | 4.9 MB 00:01 2024-08-28T19:07:15.184 INFO:teuthology.orchestra.run.smithi115.stdout:(77/150): python3-jinja2-2.11.3-6.el9.noarch.rp 1.4 MB/s | 248 kB 00:00 2024-08-28T19:07:15.201 INFO:teuthology.orchestra.run.smithi115.stdout:(78/150): python3-jsonpatch-1.21-16.el9.noarch. 610 kB/s | 26 kB 00:00 2024-08-28T19:07:15.202 INFO:teuthology.orchestra.run.smithi016.stdout:(23/150): python3-rbd-19.3.0-4560.gd19e6118.el9 4.0 MB/s | 305 kB 00:00 2024-08-28T19:07:15.218 INFO:teuthology.orchestra.run.smithi115.stdout:(79/150): python3-jsonpointer-2.0-4.el9.noarch. 568 kB/s | 19 kB 00:00 2024-08-28T19:07:15.228 INFO:teuthology.orchestra.run.smithi016.stdout:(24/150): python3-rados-19.3.0-4560.gd19e6118.e 2.7 MB/s | 321 kB 00:00 2024-08-28T19:07:15.270 INFO:teuthology.orchestra.run.smithi016.stdout:(25/150): python3-rgw-19.3.0-4560.gd19e6118.el9 1.5 MB/s | 99 kB 00:00 2024-08-28T19:07:15.293 INFO:teuthology.orchestra.run.smithi115.stdout:(80/150): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-08-28T19:07:15.295 INFO:teuthology.orchestra.run.smithi016.stdout:(26/150): rbd-fuse-19.3.0-4560.gd19e6118.el9.x8 1.3 MB/s | 86 kB 00:00 2024-08-28T19:07:15.319 INFO:teuthology.orchestra.run.smithi115.stdout:(81/150): python3-markupsafe-1.1.1-12.el9.x86_6 346 kB/s | 35 kB 00:00 2024-08-28T19:07:15.344 INFO:teuthology.orchestra.run.smithi115.stdout:(82/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-08-28T19:07:15.369 INFO:teuthology.orchestra.run.smithi115.stdout:(83/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-08-28T19:07:15.370 INFO:teuthology.orchestra.run.smithi016.stdout:(27/150): rbd-nbd-19.3.0-4560.gd19e6118.el9.x86 2.2 MB/s | 170 kB 00:00 2024-08-28T19:07:15.402 INFO:teuthology.orchestra.run.smithi115.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 2.3 MB/s | 77 kB 00:00 2024-08-28T19:07:15.407 INFO:teuthology.orchestra.run.smithi028.stdout:(26/150): librgw2-19.3.0-4560.gd19e6118.el9.x86 6.2 MB/s | 5.3 MB 00:00 2024-08-28T19:07:15.429 INFO:teuthology.orchestra.run.smithi016.stdout:(28/150): ceph-grafana-dashboards-19.3.0-4560.g 586 kB/s | 34 kB 00:00 2024-08-28T19:07:15.453 INFO:teuthology.orchestra.run.smithi115.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 42 kB 00:00 2024-08-28T19:07:15.465 INFO:teuthology.orchestra.run.smithi028.stdout:(27/150): rbd-mirror-19.3.0-4560.gd19e6118.el9. 8.2 MB/s | 3.1 MB 00:00 2024-08-28T19:07:15.478 INFO:teuthology.orchestra.run.smithi115.stdout:(86/150): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-08-28T19:07:15.479 INFO:teuthology.orchestra.run.smithi016.stdout:(29/150): rbd-mirror-19.3.0-4560.gd19e6118.el9. 15 MB/s | 3.1 MB 00:00 2024-08-28T19:07:15.491 INFO:teuthology.orchestra.run.smithi028.stdout:(28/150): rbd-nbd-19.3.0-4560.gd19e6118.el9.x86 2.0 MB/s | 170 kB 00:00 2024-08-28T19:07:15.504 INFO:teuthology.orchestra.run.smithi016.stdout:(30/150): ceph-mgr-cephadm-19.3.0-4560.gd19e611 2.0 MB/s | 154 kB 00:00 2024-08-28T19:07:15.508 INFO:teuthology.orchestra.run.smithi028.stdout:(29/150): ceph-grafana-dashboards-19.3.0-4560.g 819 kB/s | 34 kB 00:00 2024-08-28T19:07:15.541 INFO:teuthology.orchestra.run.smithi028.stdout:(30/150): ceph-mgr-cephadm-19.3.0-4560.gd19e611 3.0 MB/s | 154 kB 00:00 2024-08-28T19:07:15.562 INFO:teuthology.orchestra.run.smithi115.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-08-28T19:07:15.587 INFO:teuthology.orchestra.run.smithi115.stdout:(88/150): python3-protobuf-3.14.0-13.el9.noarch 2.0 MB/s | 269 kB 00:00 2024-08-28T19:07:15.629 INFO:teuthology.orchestra.run.smithi115.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-08-28T19:07:15.712 INFO:teuthology.orchestra.run.smithi115.stdout:(90/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-08-28T19:07:15.738 INFO:teuthology.orchestra.run.smithi115.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 493 kB/s | 54 kB 00:00 2024-08-28T19:07:15.763 INFO:teuthology.orchestra.run.smithi115.stdout:(92/150): python3-pyasn1-modules-0.4.8-6.el9.no 1.4 MB/s | 279 kB 00:00 2024-08-28T19:07:15.788 INFO:teuthology.orchestra.run.smithi115.stdout:(93/150): python3-toml-0.10.2-6.el9.noarch.rpm 831 kB/s | 42 kB 00:00 2024-08-28T19:07:15.822 INFO:teuthology.orchestra.run.smithi115.stdout:(94/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 756 kB/s | 44 kB 00:00 2024-08-28T19:07:15.839 INFO:teuthology.orchestra.run.smithi115.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-08-28T19:07:15.872 INFO:teuthology.orchestra.run.smithi115.stdout:(96/150): qatlib-service-24.02.0-1.el9.x86_64.r 713 kB/s | 36 kB 00:00 2024-08-28T19:07:15.898 INFO:teuthology.orchestra.run.smithi115.stdout:(97/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 807 kB/s | 47 kB 00:00 2024-08-28T19:07:15.948 INFO:teuthology.orchestra.run.smithi115.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-08-28T19:07:16.015 INFO:teuthology.orchestra.run.smithi115.stdout:(99/150): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-08-28T19:07:16.032 INFO:teuthology.orchestra.run.smithi115.stdout:(100/150): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.1 MB/s | 96 kB 00:00 2024-08-28T19:07:16.050 INFO:teuthology.orchestra.run.smithi028.stdout:(31/150): ceph-test-19.3.0-4560.gd19e6118.el9.x 16 MB/s | 44 MB 00:02 2024-08-28T19:07:16.083 INFO:teuthology.orchestra.run.smithi115.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 899 kB/s | 45 kB 00:00 2024-08-28T19:07:16.108 INFO:teuthology.orchestra.run.smithi115.stdout:(102/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 690 kB/s | 64 kB 00:00 2024-08-28T19:07:16.175 INFO:teuthology.orchestra.run.smithi028.stdout:(32/150): ceph-mgr-modules-core-19.3.0-4560.gd1 2.3 MB/s | 290 kB 00:00 2024-08-28T19:07:16.200 INFO:teuthology.orchestra.run.smithi115.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-08-28T19:07:16.234 INFO:teuthology.orchestra.run.smithi028.stdout:(33/150): ceph-mgr-rook-19.3.0-4560.gd19e6118.e 848 kB/s | 49 kB 00:00 2024-08-28T19:07:16.250 INFO:teuthology.orchestra.run.smithi115.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-08-28T19:07:16.275 INFO:teuthology.orchestra.run.smithi115.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 12 MB/s | 309 kB 00:00 2024-08-28T19:07:16.280 INFO:teuthology.orchestra.run.smithi016.stdout:(31/150): ceph-mgr-dashboard-19.3.0-4560.gd19e6 18 MB/s | 14 MB 00:00 2024-08-28T19:07:16.284 INFO:teuthology.orchestra.run.smithi028.stdout:(34/150): ceph-prometheus-alerts-19.3.0-4560.gd 331 kB/s | 16 kB 00:00 2024-08-28T19:07:16.301 INFO:teuthology.orchestra.run.smithi115.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 783 kB/s | 19 kB 00:00 2024-08-28T19:07:16.330 INFO:teuthology.orchestra.run.smithi016.stdout:(32/150): ceph-mgr-modules-core-19.3.0-4560.gd1 5.7 MB/s | 290 kB 00:00 2024-08-28T19:07:16.334 INFO:teuthology.orchestra.run.smithi115.stdout:(107/150): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-08-28T19:07:16.351 INFO:teuthology.orchestra.run.smithi028.stdout:(35/150): ceph-volume-19.3.0-4560.gd19e6118.el9 4.2 MB/s | 285 kB 00:00 2024-08-28T19:07:16.376 INFO:teuthology.orchestra.run.smithi115.stdout:(108/150): libarrow-doc-9.0.0-11.el9.noarch.rpm 624 kB/s | 26 kB 00:00 2024-08-28T19:07:16.389 INFO:teuthology.orchestra.run.smithi016.stdout:(33/150): ceph-mgr-rook-19.3.0-4560.gd19e6118.e 844 kB/s | 49 kB 00:00 2024-08-28T19:07:16.401 INFO:teuthology.orchestra.run.smithi115.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-08-28T19:07:16.419 INFO:teuthology.orchestra.run.smithi115.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 3.9 MB/s | 67 kB 00:00 2024-08-28T19:07:16.444 INFO:teuthology.orchestra.run.smithi115.stdout:(111/150): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-08-28T19:07:16.464 INFO:teuthology.orchestra.run.smithi016.stdout:(34/150): ceph-prometheus-alerts-19.3.0-4560.gd 219 kB/s | 16 kB 00:00 2024-08-28T19:07:16.468 INFO:teuthology.orchestra.run.smithi028.stdout:(36/150): cephadm-19.3.0-4560.gd19e6118.el9.noa 6.5 MB/s | 781 kB 00:00 2024-08-28T19:07:16.506 INFO:teuthology.orchestra.run.smithi016.stdout:(35/150): ceph-volume-19.3.0-4560.gd19e6118.el9 6.7 MB/s | 285 kB 00:00 2024-08-28T19:07:16.511 INFO:teuthology.orchestra.run.smithi115.stdout:(112/150): libarrow-9.0.0-11.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-08-28T19:07:16.535 INFO:teuthology.orchestra.run.smithi028.stdout:(37/150): cephfs-top-19.3.0-4560.gd19e6118.el9. 274 kB/s | 18 kB 00:00 2024-08-28T19:07:16.544 INFO:teuthology.orchestra.run.smithi115.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-08-28T19:07:16.570 INFO:teuthology.orchestra.run.smithi115.stdout:(114/150): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-08-28T19:07:16.587 INFO:teuthology.orchestra.run.smithi115.stdout:(115/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-08-28T19:07:16.590 INFO:teuthology.orchestra.run.smithi016.stdout:(36/150): cephadm-19.3.0-4560.gd19e6118.el9.noa 9.2 MB/s | 781 kB 00:00 2024-08-28T19:07:16.604 INFO:teuthology.orchestra.run.smithi115.stdout:(116/150): python3-cachetools-4.2.4-1.el9.noarc 951 kB/s | 32 kB 00:00 2024-08-28T19:07:16.620 INFO:teuthology.orchestra.run.smithi115.stdout:(117/150): python3-certifi-2023.05.07-4.el9.noa 421 kB/s | 14 kB 00:00 2024-08-28T19:07:16.640 INFO:teuthology.orchestra.run.smithi016.stdout:(37/150): cephfs-top-19.3.0-4560.gd19e6118.el9. 366 kB/s | 18 kB 00:00 2024-08-28T19:07:16.646 INFO:teuthology.orchestra.run.smithi115.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-08-28T19:07:16.680 INFO:teuthology.orchestra.run.smithi115.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-08-28T19:07:16.705 INFO:teuthology.orchestra.run.smithi115.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-08-28T19:07:16.730 INFO:teuthology.orchestra.run.smithi115.stdout:(121/150): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-08-28T19:07:16.744 INFO:teuthology.orchestra.run.smithi028.stdout:(38/150): ceph-mgr-dashboard-19.3.0-4560.gd19e6 11 MB/s | 14 MB 00:01 2024-08-28T19:07:16.755 INFO:teuthology.orchestra.run.smithi115.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 5.6 MB/s | 144 kB 00:00 2024-08-28T19:07:16.781 INFO:teuthology.orchestra.run.smithi115.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-08-28T19:07:16.799 INFO:teuthology.orchestra.run.smithi016.stdout:(38/150): ceph-mgr-diskprediction-local-19.3.0- 5.7 MB/s | 7.4 MB 00:01 2024-08-28T19:07:16.806 INFO:teuthology.orchestra.run.smithi115.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 713 kB/s | 18 kB 00:00 2024-08-28T19:07:16.824 INFO:teuthology.orchestra.run.smithi016.stdout:(39/150): fuse-2.9.9-16.el9.x86_64.rpm 434 kB/s | 80 kB 00:00 2024-08-28T19:07:16.831 INFO:teuthology.orchestra.run.smithi115.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 927 kB/s | 23 kB 00:00 2024-08-28T19:07:16.857 INFO:teuthology.orchestra.run.smithi115.stdout:(126/150): python3-jaraco-functools-3.5.0-2.el9 764 kB/s | 19 kB 00:00 2024-08-28T19:07:16.869 INFO:teuthology.orchestra.run.smithi028.stdout:(39/150): ceph-mgr-diskprediction-local-19.3.0- 5.6 MB/s | 7.4 MB 00:01 2024-08-28T19:07:16.882 INFO:teuthology.orchestra.run.smithi115.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 793 kB/s | 20 kB 00:00 2024-08-28T19:07:16.924 INFO:teuthology.orchestra.run.smithi016.stdout:(40/150): libquadmath-11.5.0-2.el9.x86_64.rpm 1.9 MB/s | 191 kB 00:00 2024-08-28T19:07:16.991 INFO:teuthology.orchestra.run.smithi016.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-08-28T19:07:17.025 INFO:teuthology.orchestra.run.smithi016.stdout:(42/150): libgfortran-11.5.0-2.el9.x86_64.rpm 3.5 MB/s | 801 kB 00:00 2024-08-28T19:07:17.049 INFO:teuthology.orchestra.run.smithi115.stdout:(128/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 12 MB/s | 16 MB 00:01 2024-08-28T19:07:17.067 INFO:teuthology.orchestra.run.smithi016.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-08-28T19:07:17.101 INFO:teuthology.orchestra.run.smithi016.stdout:(44/150): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-08-28T19:07:17.108 INFO:teuthology.orchestra.run.smithi115.stdout:(129/150): python3-grpcio-1.46.7-10.el9.x86_64. 5.1 MB/s | 2.0 MB 00:00 2024-08-28T19:07:17.126 INFO:teuthology.orchestra.run.smithi016.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-08-28T19:07:17.150 INFO:teuthology.orchestra.run.smithi115.stdout:(130/150): python3-kubernetes-26.1.0-2.el9.noar 3.8 MB/s | 1.0 MB 00:00 2024-08-28T19:07:17.151 INFO:teuthology.orchestra.run.smithi016.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-08-28T19:07:17.167 INFO:teuthology.orchestra.run.smithi115.stdout:(131/150): python3-logutils-0.3.5-21.el9.noarch 396 kB/s | 46 kB 00:00 2024-08-28T19:07:17.177 INFO:teuthology.orchestra.run.smithi016.stdout:(47/150): python3-urllib3-1.26.5-6.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-08-28T19:07:17.183 INFO:teuthology.orchestra.run.smithi115.stdout:(132/150): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-08-28T19:07:17.200 INFO:teuthology.orchestra.run.smithi115.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-08-28T19:07:17.217 INFO:teuthology.orchestra.run.smithi115.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 492 kB/s | 16 kB 00:00 2024-08-28T19:07:17.234 INFO:teuthology.orchestra.run.smithi115.stdout:(135/150): python3-pyperclip-1.8.2-2.el9.noarch 1.6 MB/s | 28 kB 00:00 2024-08-28T19:07:17.260 INFO:teuthology.orchestra.run.smithi115.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-08-28T19:07:17.277 INFO:teuthology.orchestra.run.smithi115.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 733 kB/s | 31 kB 00:00 2024-08-28T19:07:17.293 INFO:teuthology.orchestra.run.smithi115.stdout:(138/150): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-08-28T19:07:17.319 INFO:teuthology.orchestra.run.smithi115.stdout:(139/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-08-28T19:07:17.344 INFO:teuthology.orchestra.run.smithi115.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-08-28T19:07:17.369 INFO:teuthology.orchestra.run.smithi115.stdout:(141/150): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-08-28T19:07:17.395 INFO:teuthology.orchestra.run.smithi115.stdout:(142/150): python3-typing-extensions-4.12.2-2.e 986 kB/s | 74 kB 00:00 2024-08-28T19:07:17.411 INFO:teuthology.orchestra.run.smithi028.stdout:(40/150): fuse-2.9.9-16.el9.x86_64.rpm 91 kB/s | 80 kB 00:00 2024-08-28T19:07:17.411 INFO:teuthology.orchestra.run.smithi115.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-08-28T19:07:17.437 INFO:teuthology.orchestra.run.smithi115.stdout:(144/150): python3-zc-lockfile-2.0-10.el9.noarc 796 kB/s | 20 kB 00:00 2024-08-28T19:07:17.470 INFO:teuthology.orchestra.run.smithi115.stdout:(145/150): python3-webob-1.8.8-2.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-08-28T19:07:17.488 INFO:teuthology.orchestra.run.smithi115.stdout:(146/150): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-08-28T19:07:17.513 INFO:teuthology.orchestra.run.smithi115.stdout:(147/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-08-28T19:07:17.563 INFO:teuthology.orchestra.run.smithi115.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-08-28T19:07:17.585 INFO:teuthology.orchestra.run.smithi016.stdout:(48/150): ceph-test-19.3.0-4560.gd19e6118.el9.x 11 MB/s | 44 MB 00:04 2024-08-28T19:07:17.603 INFO:teuthology.orchestra.run.smithi028.stdout:(41/150): libquadmath-11.5.0-2.el9.x86_64.rpm 260 kB/s | 191 kB 00:00 2024-08-28T19:07:17.738 INFO:teuthology.orchestra.run.smithi115.stdout:(149/150): librados2-19.3.0-4560.gd19e6118.el9. 14 MB/s | 3.4 MB 00:00 2024-08-28T19:07:17.761 INFO:teuthology.orchestra.run.smithi028.stdout:(42/150): libgfortran-11.5.0-2.el9.x86_64.rpm 788 kB/s | 801 kB 00:01 2024-08-28T19:07:17.777 INFO:teuthology.orchestra.run.smithi016.stdout:(49/150): automake-1.16.2-8.el9.noarch.rpm 1.1 MB/s | 689 kB 00:00 2024-08-28T19:07:17.802 INFO:teuthology.orchestra.run.smithi016.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 493 kB/s | 107 kB 00:00 2024-08-28T19:07:17.805 INFO:teuthology.orchestra.run.smithi115.stdout:(150/150): librbd1-19.3.0-4560.gd19e6118.el9.x8 11 MB/s | 3.1 MB 00:00 2024-08-28T19:07:17.810 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-08-28T19:07:17.810 INFO:teuthology.orchestra.run.smithi115.stdout:Total 23 MB/s | 224 MB 00:09 2024-08-28T19:07:17.836 INFO:teuthology.orchestra.run.smithi016.stdout:(51/150): elfutils-libelf-devel-0.191-4.el9.x86 371 kB/s | 22 kB 00:00 2024-08-28T19:07:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:(52/150): bison-3.7.4-5.el9.x86_64.rpm 1.3 MB/s | 943 kB 00:00 2024-08-28T19:07:17.895 INFO:teuthology.orchestra.run.smithi016.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 557 kB/s | 33 kB 00:00 2024-08-28T19:07:17.929 INFO:teuthology.orchestra.run.smithi016.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 313 kB 00:00 2024-08-28T19:07:17.954 INFO:teuthology.orchestra.run.smithi016.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 310 kB/s | 18 kB 00:00 2024-08-28T19:07:18.021 INFO:teuthology.orchestra.run.smithi016.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-08-28T19:07:18.046 INFO:teuthology.orchestra.run.smithi016.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-08-28T19:07:18.080 INFO:teuthology.orchestra.run.smithi016.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 775 kB/s | 45 kB 00:00 2024-08-28T19:07:18.164 INFO:teuthology.orchestra.run.smithi016.stdout:(59/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 10 MB/s | 3.0 MB 00:00 2024-08-28T19:07:18.189 INFO:teuthology.orchestra.run.smithi016.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-08-28T19:07:18.214 INFO:teuthology.orchestra.run.smithi016.stdout:(61/150): libtool-2.4.6-46.el9.x86_64.rpm 4.2 MB/s | 581 kB 00:00 2024-08-28T19:07:18.239 INFO:teuthology.orchestra.run.smithi016.stdout:(62/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 337 kB/s | 25 kB 00:00 2024-08-28T19:07:18.253 INFO:teuthology.orchestra.run.smithi028.stdout:(43/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 301 kB/s | 253 kB 00:00 2024-08-28T19:07:18.265 INFO:teuthology.orchestra.run.smithi016.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-08-28T19:07:18.290 INFO:teuthology.orchestra.run.smithi016.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-08-28T19:07:18.315 INFO:teuthology.orchestra.run.smithi016.stdout:(65/150): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-08-28T19:07:18.332 INFO:teuthology.orchestra.run.smithi016.stdout:(66/150): openblas-0.3.26-1.el9.x86_64.rpm 572 kB/s | 38 kB 00:00 2024-08-28T19:07:18.520 INFO:teuthology.orchestra.run.smithi028.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 140 kB/s | 106 kB 00:00 2024-08-28T19:07:18.557 INFO:teuthology.orchestra.run.smithi016.stdout:(67/150): postgresql-13.16-1.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-08-28T19:07:18.708 INFO:teuthology.orchestra.run.smithi016.stdout:(68/150): openssl-devel-3.2.2-2.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-08-28T19:07:18.750 INFO:teuthology.orchestra.run.smithi016.stdout:(69/150): postgresql-contrib-13.16-1.el9.x86_64 4.6 MB/s | 910 kB 00:00 2024-08-28T19:07:18.767 INFO:teuthology.orchestra.run.smithi016.stdout:(70/150): postgresql-private-libs-13.16-1.el9.x 2.3 MB/s | 137 kB 00:00 2024-08-28T19:07:18.862 INFO:teuthology.orchestra.run.smithi028.stdout:(45/150): python3-cryptography-36.0.1-4.el9.x86 997 kB/s | 1.2 MB 00:01 2024-08-28T19:07:18.898 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-08-28T19:07:18.900 INFO:teuthology.orchestra.run.smithi016.stdout:(71/150): openblas-openmp-0.3.26-1.el9.x86_64.r 8.1 MB/s | 4.9 MB 00:00 2024-08-28T19:07:18.942 INFO:teuthology.orchestra.run.smithi016.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-08-28T19:07:18.967 INFO:teuthology.orchestra.run.smithi016.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-08-28T19:07:18.970 INFO:teuthology.orchestra.run.smithi028.stdout:(46/150): python3-pycparser-2.20-6.el9.noarch.r 188 kB/s | 135 kB 00:00 2024-08-28T19:07:19.022 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-08-28T19:07:19.022 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-08-28T19:07:19.043 INFO:teuthology.orchestra.run.smithi016.stdout:(74/150): python3-colorama-0.4.6-3.el9.noarch.r 817 kB/s | 61 kB 00:00 2024-08-28T19:07:19.118 INFO:teuthology.orchestra.run.smithi016.stdout:(75/150): python3-devel-3.9.19-7.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-08-28T19:07:19.193 INFO:teuthology.orchestra.run.smithi016.stdout:(76/150): python3-jinja2-2.11.3-6.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-08-28T19:07:19.277 INFO:teuthology.orchestra.run.smithi016.stdout:(77/150): postgresql-server-13.16-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-08-28T19:07:19.302 INFO:teuthology.orchestra.run.smithi016.stdout:(78/150): python3-jsonpatch-1.21-16.el9.noarch. 235 kB/s | 26 kB 00:00 2024-08-28T19:07:19.304 INFO:teuthology.orchestra.run.smithi028.stdout:(47/150): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 689 kB 00:00 2024-08-28T19:07:19.338 INFO:teuthology.orchestra.run.smithi028.stdout:(48/150): python3-requests-2.25.1-8.el9.noarch. 153 kB/s | 125 kB 00:00 2024-08-28T19:07:19.386 INFO:teuthology.orchestra.run.smithi016.stdout:(79/150): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-08-28T19:07:19.411 INFO:teuthology.orchestra.run.smithi016.stdout:(80/150): python3-jsonpointer-2.0-4.el9.noarch. 143 kB/s | 19 kB 00:00 2024-08-28T19:07:19.445 INFO:teuthology.orchestra.run.smithi016.stdout:(81/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-08-28T19:07:19.470 INFO:teuthology.orchestra.run.smithi016.stdout:(82/150): python3-markupsafe-1.1.1-12.el9.x86_6 415 kB/s | 35 kB 00:00 2024-08-28T19:07:19.479 INFO:teuthology.orchestra.run.smithi028.stdout:(49/150): bison-3.7.4-5.el9.x86_64.rpm 5.3 MB/s | 943 kB 00:00 2024-08-28T19:07:19.512 INFO:teuthology.orchestra.run.smithi016.stdout:(83/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-08-28T19:07:19.537 INFO:teuthology.orchestra.run.smithi016.stdout:(84/150): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-08-28T19:07:19.547 INFO:teuthology.orchestra.run.smithi028.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 513 kB/s | 107 kB 00:00 2024-08-28T19:07:19.579 INFO:teuthology.orchestra.run.smithi016.stdout:(85/150): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-08-28T19:07:19.604 INFO:teuthology.orchestra.run.smithi016.stdout:(86/150): python3-prettytable-0.7.2-27.el9.noar 630 kB/s | 42 kB 00:00 2024-08-28T19:07:19.655 INFO:teuthology.orchestra.run.smithi016.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-08-28T19:07:19.664 INFO:teuthology.orchestra.run.smithi028.stdout:(51/150): elfutils-libelf-devel-0.191-4.el9.x86 118 kB/s | 22 kB 00:00 2024-08-28T19:07:19.680 INFO:teuthology.orchestra.run.smithi016.stdout:(88/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-08-28T19:07:19.697 INFO:teuthology.orchestra.run.smithi028.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 2.0 MB/s | 313 kB 00:00 2024-08-28T19:07:19.722 INFO:teuthology.orchestra.run.smithi016.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-08-28T19:07:19.748 INFO:teuthology.orchestra.run.smithi028.stdout:(53/150): python3-urllib3-1.26.5-6.el9.noarch.r 243 kB/s | 215 kB 00:00 2024-08-28T19:07:19.764 INFO:teuthology.orchestra.run.smithi028.stdout:(54/150): flexiblas-3.0.4-8.el9.x86_64.rpm 325 kB/s | 33 kB 00:00 2024-08-28T19:07:19.789 INFO:teuthology.orchestra.run.smithi016.stdout:(90/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-08-28T19:07:19.798 INFO:teuthology.orchestra.run.smithi028.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-08-28T19:07:19.814 INFO:teuthology.orchestra.run.smithi016.stdout:(91/150): python3-pytz-2021.1-5.el9.noarch.rpm 380 kB/s | 51 kB 00:00 2024-08-28T19:07:19.848 INFO:teuthology.orchestra.run.smithi016.stdout:(92/150): python3-requests-oauthlib-1.3.0-12.el 427 kB/s | 54 kB 00:00 2024-08-28T19:07:19.865 INFO:teuthology.orchestra.run.smithi028.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-08-28T19:07:19.882 INFO:teuthology.orchestra.run.smithi016.stdout:(93/150): python3-toml-0.10.2-6.el9.noarch.rpm 622 kB/s | 42 kB 00:00 2024-08-28T19:07:19.899 INFO:teuthology.orchestra.run.smithi028.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-08-28T19:07:19.932 INFO:teuthology.orchestra.run.smithi016.stdout:(94/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 528 kB/s | 44 kB 00:00 2024-08-28T19:07:19.957 INFO:teuthology.orchestra.run.smithi016.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-08-28T19:07:19.982 INFO:teuthology.orchestra.run.smithi028.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-08-28T19:07:19.999 INFO:teuthology.orchestra.run.smithi016.stdout:(96/150): qatlib-service-24.02.0-1.el9.x86_64.r 536 kB/s | 36 kB 00:00 2024-08-28T19:07:20.025 INFO:teuthology.orchestra.run.smithi016.stdout:(97/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 706 kB/s | 47 kB 00:00 2024-08-28T19:07:20.067 INFO:teuthology.orchestra.run.smithi016.stdout:(98/150): socat-1.7.4.1-6.el9.x86_64.rpm 4.4 MB/s | 304 kB 00:00 2024-08-28T19:07:20.101 INFO:teuthology.orchestra.run.smithi016.stdout:(99/150): uuid-1.6.2-55.el9.x86_64.rpm 766 kB/s | 58 kB 00:00 2024-08-28T19:07:20.150 INFO:teuthology.orchestra.run.smithi016.stdout:(100/150): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.1 MB/s | 96 kB 00:00 2024-08-28T19:07:20.174 INFO:teuthology.orchestra.run.smithi028.stdout:(59/150): libtool-2.4.6-46.el9.x86_64.rpm 3.0 MB/s | 581 kB 00:00 2024-08-28T19:07:20.176 INFO:teuthology.orchestra.run.smithi016.stdout:(101/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 845 kB/s | 64 kB 00:00 2024-08-28T19:07:20.216 INFO:teuthology.orchestra.run.smithi028.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 2.0 MB/s | 662 kB 00:00 2024-08-28T19:07:20.218 INFO:teuthology.orchestra.run.smithi016.stdout:(102/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 669 kB/s | 45 kB 00:00 2024-08-28T19:07:20.233 INFO:teuthology.orchestra.run.smithi028.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 434 kB/s | 25 kB 00:00 2024-08-28T19:07:20.275 INFO:teuthology.orchestra.run.smithi028.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-08-28T19:07:20.276 INFO:teuthology.orchestra.run.smithi016.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-08-28T19:07:20.327 INFO:teuthology.orchestra.run.smithi016.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-08-28T19:07:20.358 INFO:teuthology.orchestra.run.smithi028.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-08-28T19:07:20.360 INFO:teuthology.orchestra.run.smithi016.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 9.1 MB/s | 309 kB 00:00 2024-08-28T19:07:20.377 INFO:teuthology.orchestra.run.smithi016.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 1.2 MB/s | 19 kB 00:00 2024-08-28T19:07:20.384 INFO:teuthology.orchestra.run.smithi028.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-08-28T19:07:20.426 INFO:teuthology.orchestra.run.smithi028.stdout:(65/150): openblas-0.3.26-1.el9.x86_64.rpm 572 kB/s | 38 kB 00:00 2024-08-28T19:07:20.478 INFO:teuthology.orchestra.run.smithi016.stdout:(107/150): libarrow-9.0.0-11.el9.x86_64.rpm 44 MB/s | 4.4 MB 00:00 2024-08-28T19:07:20.495 INFO:teuthology.orchestra.run.smithi016.stdout:(108/150): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.5 MB/s | 26 kB 00:00 2024-08-28T19:07:20.512 INFO:teuthology.orchestra.run.smithi016.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 2.8 MB/s | 49 kB 00:00 2024-08-28T19:07:20.626 INFO:teuthology.orchestra.run.smithi028.stdout:(66/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.2 MB/s | 3.0 MB 00:00 2024-08-28T19:07:20.670 INFO:teuthology.orchestra.run.smithi016.stdout:(110/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-08-28T19:07:20.695 INFO:teuthology.orchestra.run.smithi016.stdout:(111/150): libunwind-1.6.2-1.el9.x86_64.rpm 367 kB/s | 67 kB 00:00 2024-08-28T19:07:20.729 INFO:teuthology.orchestra.run.smithi016.stdout:(112/150): luarocks-3.9.2-5.el9.noarch.rpm 4.5 MB/s | 151 kB 00:00 2024-08-28T19:07:20.763 INFO:teuthology.orchestra.run.smithi016.stdout:(113/150): protobuf-compiler-3.14.0-13.el9.x86_ 1.5 MB/s | 863 kB 00:00 2024-08-28T19:07:20.764 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-08-28T19:07:20.764 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-08-28T19:07:20.805 INFO:teuthology.orchestra.run.smithi016.stdout:(114/150): python3-asyncssh-2.13.2-5.el9.noarch 7.3 MB/s | 548 kB 00:00 2024-08-28T19:07:20.830 INFO:teuthology.orchestra.run.smithi016.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 1.4 MB/s | 32 kB 00:00 2024-08-28T19:07:20.855 INFO:teuthology.orchestra.run.smithi016.stdout:(116/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-08-28T19:07:20.889 INFO:teuthology.orchestra.run.smithi016.stdout:(117/150): python3-certifi-2023.05.07-4.el9.noa 240 kB/s | 14 kB 00:00 2024-08-28T19:07:20.922 INFO:teuthology.orchestra.run.smithi016.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-08-28T19:07:20.948 INFO:teuthology.orchestra.run.smithi016.stdout:(119/150): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.3 MB/s | 839 kB 00:00 2024-08-28T19:07:20.973 INFO:teuthology.orchestra.run.smithi016.stdout:(120/150): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-08-28T19:07:21.015 INFO:teuthology.orchestra.run.smithi016.stdout:(121/150): python3-google-auth-2.34.0-1.el9.noa 3.2 MB/s | 220 kB 00:00 2024-08-28T19:07:21.018 INFO:teuthology.orchestra.run.smithi028.stdout:(67/150): postgresql-13.16-1.el9.x86_64.rpm 4.1 MB/s | 1.6 MB 00:00 2024-08-28T19:07:21.040 INFO:teuthology.orchestra.run.smithi016.stdout:(122/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-08-28T19:07:21.066 INFO:teuthology.orchestra.run.smithi016.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 430 kB/s | 11 kB 00:00 2024-08-28T19:07:21.091 INFO:teuthology.orchestra.run.smithi016.stdout:(124/150): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-08-28T19:07:21.108 INFO:teuthology.orchestra.run.smithi016.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 423 kB/s | 18 kB 00:00 2024-08-28T19:07:21.133 INFO:teuthology.orchestra.run.smithi016.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 924 kB/s | 23 kB 00:00 2024-08-28T19:07:21.158 INFO:teuthology.orchestra.run.smithi016.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-08-28T19:07:21.184 INFO:teuthology.orchestra.run.smithi016.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-08-28T19:07:21.209 INFO:teuthology.orchestra.run.smithi016.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-08-28T19:07:21.226 INFO:teuthology.orchestra.run.smithi016.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 4.7 MB/s | 79 kB 00:00 2024-08-28T19:07:21.251 INFO:teuthology.orchestra.run.smithi016.stdout:(131/150): python3-natsort-7.1.1-5.el9.noarch.r 2.2 MB/s | 58 kB 00:00 2024-08-28T19:07:21.252 INFO:teuthology.orchestra.run.smithi028.stdout:(68/150): postgresql-contrib-13.16-1.el9.x86_64 3.8 MB/s | 910 kB 00:00 2024-08-28T19:07:21.285 INFO:teuthology.orchestra.run.smithi016.stdout:(132/150): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-08-28T19:07:21.319 INFO:teuthology.orchestra.run.smithi028.stdout:(69/150): openssl-devel-3.2.2-2.el9.x86_64.rpm 5.0 MB/s | 4.4 MB 00:00 2024-08-28T19:07:21.319 INFO:teuthology.orchestra.run.smithi016.stdout:(133/150): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-08-28T19:07:21.344 INFO:teuthology.orchestra.run.smithi028.stdout:(70/150): postgresql-private-libs-13.16-1.el9.x 1.5 MB/s | 137 kB 00:00 2024-08-28T19:07:21.344 INFO:teuthology.orchestra.run.smithi016.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-08-28T19:07:21.369 INFO:teuthology.orchestra.run.smithi016.stdout:(135/150): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-08-28T19:07:21.395 INFO:teuthology.orchestra.run.smithi016.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-08-28T19:07:21.420 INFO:teuthology.orchestra.run.smithi016.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-08-28T19:07:21.445 INFO:teuthology.orchestra.run.smithi016.stdout:(138/150): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-08-28T19:07:21.470 INFO:teuthology.orchestra.run.smithi016.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-08-28T19:07:21.496 INFO:teuthology.orchestra.run.smithi016.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-08-28T19:07:21.521 INFO:teuthology.orchestra.run.smithi016.stdout:(141/150): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-08-28T19:07:21.538 INFO:teuthology.orchestra.run.smithi016.stdout:(142/150): python3-websocket-client-1.2.3-2.el9 5.3 MB/s | 90 kB 00:00 2024-08-28T19:07:21.588 INFO:teuthology.orchestra.run.smithi016.stdout:(143/150): python3-grpcio-1.46.7-10.el9.x86_64. 3.3 MB/s | 2.0 MB 00:00 2024-08-28T19:07:21.622 INFO:teuthology.orchestra.run.smithi016.stdout:(144/150): python3-webob-1.8.8-2.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-08-28T19:07:21.636 INFO:teuthology.orchestra.run.smithi028.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-08-28T19:07:21.647 INFO:teuthology.orchestra.run.smithi016.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-08-28T19:07:21.673 INFO:teuthology.orchestra.run.smithi016.stdout:(146/150): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-08-28T19:07:21.698 INFO:teuthology.orchestra.run.smithi016.stdout:(147/150): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-08-28T19:07:21.711 INFO:teuthology.orchestra.run.smithi028.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-08-28T19:07:21.748 INFO:teuthology.orchestra.run.smithi016.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-28T19:07:21.911 INFO:teuthology.orchestra.run.smithi028.stdout:(73/150): openblas-openmp-0.3.26-1.el9.x86_64.r 3.2 MB/s | 4.9 MB 00:01 2024-08-28T19:07:21.948 INFO:teuthology.orchestra.run.smithi016.stdout:(149/150): librbd1-19.3.0-4560.gd19e6118.el9.x8 13 MB/s | 3.1 MB 00:00 2024-08-28T19:07:21.995 INFO:teuthology.orchestra.run.smithi028.stdout:(74/150): python3-colorama-0.4.6-3.el9.noarch.r 735 kB/s | 61 kB 00:00 2024-08-28T19:07:22.090 INFO:teuthology.orchestra.run.smithi016.stdout:(150/150): librados2-19.3.0-4560.gd19e6118.el9. 8.1 MB/s | 3.4 MB 00:00 2024-08-28T19:07:22.096 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-08-28T19:07:22.096 INFO:teuthology.orchestra.run.smithi016.stdout:Total 20 MB/s | 224 MB 00:10 2024-08-28T19:07:22.137 INFO:teuthology.orchestra.run.smithi028.stdout:(75/150): python3-devel-3.9.19-7.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-08-28T19:07:22.295 INFO:teuthology.orchestra.run.smithi028.stdout:(76/150): python3-jinja2-2.11.3-6.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-08-28T19:07:22.379 INFO:teuthology.orchestra.run.smithi028.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 308 kB/s | 26 kB 00:00 2024-08-28T19:07:22.454 INFO:teuthology.orchestra.run.smithi028.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 255 kB/s | 19 kB 00:00 2024-08-28T19:07:22.529 INFO:teuthology.orchestra.run.smithi028.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-08-28T19:07:22.597 INFO:teuthology.orchestra.run.smithi028.stdout:(80/150): postgresql-server-13.16-1.el9.x86_64. 4.5 MB/s | 5.8 MB 00:01 2024-08-28T19:07:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:(81/150): python3-markupsafe-1.1.1-12.el9.x86_6 415 kB/s | 35 kB 00:00 2024-08-28T19:07:22.747 INFO:teuthology.orchestra.run.smithi028.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.8 MB/s | 250 kB 00:00 2024-08-28T19:07:23.039 INFO:teuthology.orchestra.run.smithi028.stdout:(83/150): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 6.0 MB 00:01 2024-08-28T19:07:23.115 INFO:teuthology.orchestra.run.smithi028.stdout:(84/150): python3-oauthlib-3.1.1-5.el9.noarch.r 605 kB/s | 222 kB 00:00 2024-08-28T19:07:23.140 INFO:teuthology.orchestra.run.smithi028.stdout:(85/150): python3-packaging-20.9-5.el9.noarch.r 770 kB/s | 77 kB 00:00 2024-08-28T19:07:23.173 INFO:teuthology.orchestra.run.smithi028.stdout:(86/150): python3-prettytable-0.7.2-27.el9.noar 724 kB/s | 42 kB 00:00 2024-08-28T19:07:23.207 INFO:teuthology.orchestra.run.smithi028.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 3.9 MB/s | 269 kB 00:00 2024-08-28T19:07:23.217 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-08-28T19:07:23.240 INFO:teuthology.orchestra.run.smithi028.stdout:(88/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-08-28T19:07:23.283 INFO:teuthology.orchestra.run.smithi028.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-08-28T19:07:23.299 INFO:teuthology.orchestra.run.smithi028.stdout:(90/150): python3-pytz-2021.1-5.el9.noarch.rpm 870 kB/s | 51 kB 00:00 2024-08-28T19:07:23.356 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-08-28T19:07:23.357 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-08-28T19:07:23.366 INFO:teuthology.orchestra.run.smithi028.stdout:(91/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.6 MB/s | 5.1 MB 00:00 2024-08-28T19:07:23.391 INFO:teuthology.orchestra.run.smithi028.stdout:(92/150): python3-requests-oauthlib-1.3.0-12.el 494 kB/s | 54 kB 00:00 2024-08-28T19:07:23.425 INFO:teuthology.orchestra.run.smithi028.stdout:(93/150): python3-toml-0.10.2-6.el9.noarch.rpm 712 kB/s | 42 kB 00:00 2024-08-28T19:07:23.459 INFO:teuthology.orchestra.run.smithi028.stdout:(94/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 660 kB/s | 44 kB 00:00 2024-08-28T19:07:23.500 INFO:teuthology.orchestra.run.smithi028.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-08-28T19:07:23.534 INFO:teuthology.orchestra.run.smithi028.stdout:(96/150): qatlib-service-24.02.0-1.el9.x86_64.r 478 kB/s | 36 kB 00:00 2024-08-28T19:07:23.568 INFO:teuthology.orchestra.run.smithi028.stdout:(97/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 708 kB/s | 47 kB 00:00 2024-08-28T19:07:23.610 INFO:teuthology.orchestra.run.smithi028.stdout:(98/150): socat-1.7.4.1-6.el9.x86_64.rpm 3.9 MB/s | 304 kB 00:00 2024-08-28T19:07:23.635 INFO:teuthology.orchestra.run.smithi028.stdout:(99/150): uuid-1.6.2-55.el9.x86_64.rpm 862 kB/s | 58 kB 00:00 2024-08-28T19:07:23.677 INFO:teuthology.orchestra.run.smithi028.stdout:(100/150): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.4 MB/s | 96 kB 00:00 2024-08-28T19:07:23.702 INFO:teuthology.orchestra.run.smithi028.stdout:(101/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 951 kB/s | 64 kB 00:00 2024-08-28T19:07:23.744 INFO:teuthology.orchestra.run.smithi028.stdout:(102/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 669 kB/s | 45 kB 00:00 2024-08-28T19:07:23.869 INFO:teuthology.orchestra.run.smithi028.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 134 kB/s | 22 kB 00:00 2024-08-28T19:07:23.928 INFO:teuthology.orchestra.run.smithi028.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-08-28T19:07:23.962 INFO:teuthology.orchestra.run.smithi028.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 9.0 MB/s | 309 kB 00:00 2024-08-28T19:07:23.987 INFO:teuthology.orchestra.run.smithi028.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 780 kB/s | 19 kB 00:00 2024-08-28T19:07:24.104 INFO:teuthology.orchestra.run.smithi028.stdout:(107/150): libarrow-9.0.0-11.el9.x86_64.rpm 38 MB/s | 4.4 MB 00:00 2024-08-28T19:07:24.129 INFO:teuthology.orchestra.run.smithi028.stdout:(108/150): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-08-28T19:07:24.154 INFO:teuthology.orchestra.run.smithi028.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-08-28T19:07:24.188 INFO:teuthology.orchestra.run.smithi028.stdout:(110/150): protobuf-compiler-3.14.0-13.el9.x86_ 1.9 MB/s | 863 kB 00:00 2024-08-28T19:07:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:(111/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-08-28T19:07:24.247 INFO:teuthology.orchestra.run.smithi028.stdout:(112/150): luarocks-3.9.2-5.el9.noarch.rpm 4.5 MB/s | 151 kB 00:00 2024-08-28T19:07:24.289 INFO:teuthology.orchestra.run.smithi028.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-08-28T19:07:24.314 INFO:teuthology.orchestra.run.smithi028.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-08-28T19:07:24.340 INFO:teuthology.orchestra.run.smithi028.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-08-28T19:07:24.365 INFO:teuthology.orchestra.run.smithi028.stdout:(116/150): python3-certifi-2023.05.07-4.el9.noa 569 kB/s | 14 kB 00:00 2024-08-28T19:07:24.399 INFO:teuthology.orchestra.run.smithi028.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 5.0 MB/s | 172 kB 00:00 2024-08-28T19:07:24.432 INFO:teuthology.orchestra.run.smithi028.stdout:(118/150): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.8 MB/s | 839 kB 00:00 2024-08-28T19:07:24.457 INFO:teuthology.orchestra.run.smithi028.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-08-28T19:07:24.483 INFO:teuthology.orchestra.run.smithi028.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-08-28T19:07:24.508 INFO:teuthology.orchestra.run.smithi028.stdout:(121/150): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-08-28T19:07:24.533 INFO:teuthology.orchestra.run.smithi028.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 5.7 MB/s | 144 kB 00:00 2024-08-28T19:07:24.550 INFO:teuthology.orchestra.run.smithi028.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 640 kB/s | 11 kB 00:00 2024-08-28T19:07:24.567 INFO:teuthology.orchestra.run.smithi028.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 1.0 MB/s | 18 kB 00:00 2024-08-28T19:07:24.584 INFO:teuthology.orchestra.run.smithi028.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 1.4 MB/s | 23 kB 00:00 2024-08-28T19:07:24.601 INFO:teuthology.orchestra.run.smithi028.stdout:(126/150): python3-jaraco-functools-3.5.0-2.el9 1.1 MB/s | 19 kB 00:00 2024-08-28T19:07:24.618 INFO:teuthology.orchestra.run.smithi028.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 1.2 MB/s | 20 kB 00:00 2024-08-28T19:07:24.668 INFO:teuthology.orchestra.run.smithi028.stdout:(128/150): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-08-28T19:07:24.685 INFO:teuthology.orchestra.run.smithi028.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 2.7 MB/s | 46 kB 00:00 2024-08-28T19:07:24.711 INFO:teuthology.orchestra.run.smithi028.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-08-28T19:07:24.744 INFO:teuthology.orchestra.run.smithi028.stdout:(131/150): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-08-28T19:07:24.761 INFO:teuthology.orchestra.run.smithi028.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-08-28T19:07:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:(133/150): python3-portend-3.1.0-2.el9.noarch.r 995 kB/s | 16 kB 00:00 2024-08-28T19:07:24.803 INFO:teuthology.orchestra.run.smithi028.stdout:(134/150): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-08-28T19:07:24.829 INFO:teuthology.orchestra.run.smithi028.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-08-28T19:07:24.845 INFO:teuthology.orchestra.run.smithi028.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 676 kB/s | 28 kB 00:00 2024-08-28T19:07:24.863 INFO:teuthology.orchestra.run.smithi028.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 917 kB/s | 31 kB 00:00 2024-08-28T19:07:24.888 INFO:teuthology.orchestra.run.smithi028.stdout:(138/150): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-08-28T19:07:24.913 INFO:teuthology.orchestra.run.smithi028.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 2.7 MB/s | 188 kB 00:00 2024-08-28T19:07:25.072 INFO:teuthology.orchestra.run.smithi028.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 194 kB/s | 36 kB 00:00 2024-08-28T19:07:25.097 INFO:teuthology.orchestra.run.smithi028.stdout:(141/150): python3-typing-extensions-4.12.2-2.e 2.9 MB/s | 74 kB 00:00 2024-08-28T19:07:25.122 INFO:teuthology.orchestra.run.smithi028.stdout:(142/150): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-08-28T19:07:25.148 INFO:teuthology.orchestra.run.smithi028.stdout:(143/150): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-08-28T19:07:25.173 INFO:teuthology.orchestra.run.smithi028.stdout:(144/150): python3-zc-lockfile-2.0-10.el9.noarc 809 kB/s | 20 kB 00:00 2024-08-28T19:07:25.198 INFO:teuthology.orchestra.run.smithi028.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-08-28T19:07:25.207 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-08-28T19:07:25.209 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-08-28T19:07:25.232 INFO:teuthology.orchestra.run.smithi028.stdout:(146/150): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-08-28T19:07:25.282 INFO:teuthology.orchestra.run.smithi028.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-08-28T19:07:25.491 INFO:teuthology.orchestra.run.smithi028.stdout:(148/150): librados2-19.3.0-4560.gd19e6118.el9. 13 MB/s | 3.4 MB 00:00 2024-08-28T19:07:25.566 INFO:teuthology.orchestra.run.smithi028.stdout:(149/150): librbd1-19.3.0-4560.gd19e6118.el9.x8 11 MB/s | 3.1 MB 00:00 2024-08-28T19:07:26.830 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-08-28T19:07:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-08-28T19:07:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-08-28T19:07:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-08-28T19:07:27.316 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-08-28T19:07:27.442 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 5/152 2024-08-28T19:07:27.568 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 5/152 2024-08-28T19:07:27.601 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-28T19:07:27.693 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-28T19:07:27.734 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 7/152 2024-08-28T19:07:27.741 INFO:teuthology.orchestra.run.smithi028.stdout:(150/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.7 MB/s | 16 MB 00:04 2024-08-28T19:07:27.747 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-08-28T19:07:27.747 INFO:teuthology.orchestra.run.smithi028.stdout:Total 13 MB/s | 224 MB 00:17 2024-08-28T19:07:27.763 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-08-28T19:07:27.814 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-08-28T19:07:27.966 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 10/152 2024-08-28T19:07:28.020 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-13.16-1.el9.x86_64 11/152 2024-08-28T19:07:28.048 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-28T19:07:28.155 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-28T19:07:28.182 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 13/152 2024-08-28T19:07:28.245 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 13/152 2024-08-28T19:07:28.258 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-28T19:07:28.277 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-28T19:07:28.574 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-28T19:07:28.608 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-28T19:07:28.710 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/152 2024-08-28T19:07:28.739 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/152 2024-08-28T19:07:28.774 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/152 2024-08-28T19:07:28.825 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-08-28T19:07:28.848 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/152 2024-08-28T19:07:28.893 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 20/152 2024-08-28T19:07:28.942 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-packaging-20.9-5.el9.noarch 21/152 2024-08-28T19:07:28.961 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-08-28T19:07:28.961 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-08-28T19:07:28.985 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-08-28T19:07:29.017 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-08-28T19:07:29.045 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-08-28T19:07:29.116 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-08-28T19:07:29.254 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 26/152 2024-08-28T19:07:29.290 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-08-28T19:07:29.334 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 28/152 2024-08-28T19:07:29.412 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 29/152 2024-08-28T19:07:29.470 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-common-2:19.3.0-4560.gd19e6118.el9. 30/152 2024-08-28T19:07:29.505 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el 31/152 2024-08-28T19:07:29.525 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_6 32/152 2024-08-28T19:07:29.605 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-08-28T19:07:29.657 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : luarocks-3.9.2-5.el9.noarch 34/152 2024-08-28T19:07:29.732 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-08-28T19:07:29.870 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-08-28T19:07:29.926 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-08-28T19:07:30.469 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-08-28T19:07:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-08-28T19:07:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 40/152 2024-08-28T19:07:30.637 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-08-28T19:07:30.674 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_6 42/152 2024-08-28T19:07:30.708 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-08-28T19:07:30.800 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-08-28T19:07:30.812 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-08-28T19:07:30.812 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-08-28T19:07:30.864 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 45/152 2024-08-28T19:07:30.937 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 46/152 2024-08-28T19:07:30.968 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 47/152 2024-08-28T19:07:31.004 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 48/152 2024-08-28T19:07:31.051 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 49/152 2024-08-28T19:07:31.087 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 50/152 2024-08-28T19:07:31.130 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 51/152 2024-08-28T19:07:31.170 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-08-28T19:07:31.172 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 52/152 2024-08-28T19:07:31.252 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-08-28T19:07:31.277 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-08-28T19:07:31.309 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 53/152 2024-08-28T19:07:31.320 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-08-28T19:07:31.357 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 54/152 2024-08-28T19:07:31.398 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 55/152 2024-08-28T19:07:31.502 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-08-28T19:07:31.532 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 57/152 2024-08-28T19:07:31.572 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-08-28T19:07:31.597 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-08-28T19:07:31.622 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 60/152 2024-08-28T19:07:31.718 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-08-28T19:07:31.724 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-08-28T19:07:31.803 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 5/152 2024-08-28T19:07:31.849 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-08-28T19:07:31.888 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-08-28T19:07:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 5/152 2024-08-28T19:07:31.915 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-08-28T19:07:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-28T19:07:31.950 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-08-28T19:07:31.994 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-08-28T19:07:32.013 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-28T19:07:32.031 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 67/152 2024-08-28T19:07:32.055 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 7/152 2024-08-28T19:07:32.063 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-08-28T19:07:32.083 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-08-28T19:07:32.085 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-08-28T19:07:32.109 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-08-28T19:07:32.111 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-08-28T19:07:32.144 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-08-28T19:07:32.186 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-08-28T19:07:32.228 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-08-28T19:07:32.271 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 10/152 2024-08-28T19:07:32.357 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-13.16-1.el9.x86_64 11/152 2024-08-28T19:07:32.378 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-28T19:07:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-28T19:07:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 13/152 2024-08-28T19:07:32.517 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-08-28T19:07:32.534 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 13/152 2024-08-28T19:07:32.547 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-28T19:07:32.568 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-08-28T19:07:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-28T19:07:32.598 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-08-28T19:07:32.704 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-08-28T19:07:32.758 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 78/152 2024-08-28T19:07:32.849 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-28T19:07:32.880 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-28T19:07:32.984 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/152 2024-08-28T19:07:33.011 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/152 2024-08-28T19:07:33.048 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/152 2024-08-28T19:07:33.123 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/152 2024-08-28T19:07:33.174 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 20/152 2024-08-28T19:07:33.213 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 21/152 2024-08-28T19:07:33.265 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-08-28T19:07:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-08-28T19:07:33.325 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-08-28T19:07:33.389 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-08-28T19:07:33.449 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 26/152 2024-08-28T19:07:33.478 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-08-28T19:07:33.489 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/152 2024-08-28T19:07:33.514 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 28/152 2024-08-28T19:07:33.529 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 80/152 2024-08-28T19:07:33.563 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 81/152 2024-08-28T19:07:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 29/152 2024-08-28T19:07:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.3.0-4560.gd19e6118.el9. 30/152 2024-08-28T19:07:33.664 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 82/152 2024-08-28T19:07:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el 31/152 2024-08-28T19:07:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_6 32/152 2024-08-28T19:07:33.718 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-devel-3.9.19-7.el9.x86_64 83/152 2024-08-28T19:07:33.763 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 84/152 2024-08-28T19:07:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-08-28T19:07:33.829 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-5.el9.noarch 34/152 2024-08-28T19:07:33.851 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-08-28T19:07:33.899 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-08-28T19:07:33.907 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-08-28T19:07:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-08-28T19:07:34.083 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-0.3.26-1.el9.x86_64 87/152 2024-08-28T19:07:34.105 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-openmp-0.3.26-1.el9.x86_64 88/152 2024-08-28T19:07:34.108 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-08-28T19:07:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-08-28T19:07:34.679 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-08-28T19:07:34.771 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-08-28T19:07:34.806 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 40/152 2024-08-28T19:07:34.830 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-08-28T19:07:34.849 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-08-28T19:07:34.887 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_6 42/152 2024-08-28T19:07:34.920 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-08-28T19:07:34.946 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-08-28T19:07:35.016 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-08-28T19:07:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 45/152 2024-08-28T19:07:35.152 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 46/152 2024-08-28T19:07:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 47/152 2024-08-28T19:07:35.225 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 48/152 2024-08-28T19:07:35.273 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 49/152 2024-08-28T19:07:35.308 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 50/152 2024-08-28T19:07:35.352 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 51/152 2024-08-28T19:07:35.394 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 52/152 2024-08-28T19:07:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 53/152 2024-08-28T19:07:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 54/152 2024-08-28T19:07:35.621 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 55/152 2024-08-28T19:07:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-08-28T19:07:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 57/152 2024-08-28T19:07:35.802 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-08-28T19:07:35.827 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-08-28T19:07:35.852 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 60/152 2024-08-28T19:07:35.930 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-08-28T19:07:36.068 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-08-28T19:07:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-08-28T19:07:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-08-28T19:07:36.180 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-08-28T19:07:36.197 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-08-28T19:07:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-08-28T19:07:36.262 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 67/152 2024-08-28T19:07:36.269 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-28T19:07:36.296 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-08-28T19:07:36.302 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-08-28T19:07:36.329 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-08-28T19:07:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-08-28T19:07:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-08-28T19:07:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-08-28T19:07:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-08-28T19:07:36.476 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-08-28T19:07:36.558 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-08-28T19:07:36.607 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-08-28T19:07:36.637 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-08-28T19:07:36.658 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-08-28T19:07:36.701 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 97/152 2024-08-28T19:07:36.746 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-08-28T19:07:36.792 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 97/152 2024-08-28T19:07:36.822 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 98/152 2024-08-28T19:07:36.858 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-08-28T19:07:37.009 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 78/152 2024-08-28T19:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-08-28T19:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-08-28T19:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-08-28T19:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-08-28T19:07:37.357 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 100/152 2024-08-28T19:07:37.387 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 101/152 2024-08-28T19:07:37.458 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 101/152 2024-08-28T19:07:37.480 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 102/152 2024-08-28T19:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-08-28T19:07:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/152 2024-08-28T19:07:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 80/152 2024-08-28T19:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 5/152 2024-08-28T19:07:37.861 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 81/152 2024-08-28T19:07:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 82/152 2024-08-28T19:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 5/152 2024-08-28T19:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-28T19:07:38.017 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.19-7.el9.x86_64 83/152 2024-08-28T19:07:38.063 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 84/152 2024-08-28T19:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-28T19:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 7/152 2024-08-28T19:07:38.154 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-08-28T19:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-08-28T19:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-08-28T19:07:38.189 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-08-28T19:07:38.345 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 10/152 2024-08-28T19:07:38.372 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.26-1.el9.x86_64 87/152 2024-08-28T19:07:38.399 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.16-1.el9.x86_64 11/152 2024-08-28T19:07:38.403 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.26-1.el9.x86_64 88/152 2024-08-28T19:07:38.442 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-28T19:07:38.483 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-08-28T19:07:38.550 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-28T19:07:38.576 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 13/152 2024-08-28T19:07:38.639 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 13/152 2024-08-28T19:07:38.661 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-28T19:07:38.684 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-28T19:07:38.980 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-28T19:07:39.010 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-28T19:07:39.122 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/152 2024-08-28T19:07:39.158 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/152 2024-08-28T19:07:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-08-28T19:07:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/152 2024-08-28T19:07:39.270 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/152 2024-08-28T19:07:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-08-28T19:07:39.388 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 20/152 2024-08-28T19:07:39.427 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 21/152 2024-08-28T19:07:39.471 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-08-28T19:07:39.512 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-08-28T19:07:39.548 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-08-28T19:07:39.654 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-08-28T19:07:39.713 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 26/152 2024-08-28T19:07:39.725 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-28T19:07:39.742 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-08-28T19:07:39.795 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 28/152 2024-08-28T19:07:39.818 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-28T19:07:39.874 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 29/152 2024-08-28T19:07:39.913 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.3.0-4560.gd19e6118.el9. 30/152 2024-08-28T19:07:39.958 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el 31/152 2024-08-28T19:07:39.985 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_6 32/152 2024-08-28T19:07:40.061 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-08-28T19:07:40.109 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-5.el9.noarch 34/152 2024-08-28T19:07:40.187 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-08-28T19:07:40.391 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-28T19:07:40.438 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 104/152 2024-08-28T19:07:40.450 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-08-28T19:07:40.518 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 104/152 2024-08-28T19:07:40.518 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-28T19:07:40.518 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-28T19:07:40.519 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:07:40.524 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-08-28T19:07:40.547 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 105/152 2024-08-28T19:07:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-08-28T19:07:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-28T19:07:40.720 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-08-28T19:07:40.744 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-08-28T19:07:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-08-28T19:07:41.095 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-08-28T19:07:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 97/152 2024-08-28T19:07:41.179 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-08-28T19:07:41.212 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 40/152 2024-08-28T19:07:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 97/152 2024-08-28T19:07:41.246 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 98/152 2024-08-28T19:07:41.255 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-08-28T19:07:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-08-28T19:07:41.293 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_6 42/152 2024-08-28T19:07:41.326 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-08-28T19:07:41.424 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-08-28T19:07:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 45/152 2024-08-28T19:07:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 46/152 2024-08-28T19:07:41.603 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 47/152 2024-08-28T19:07:41.639 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 48/152 2024-08-28T19:07:41.687 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 49/152 2024-08-28T19:07:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 50/152 2024-08-28T19:07:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 51/152 2024-08-28T19:07:41.807 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 100/152 2024-08-28T19:07:41.808 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 52/152 2024-08-28T19:07:41.835 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 101/152 2024-08-28T19:07:41.890 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 101/152 2024-08-28T19:07:41.920 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 102/152 2024-08-28T19:07:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 53/152 2024-08-28T19:07:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 54/152 2024-08-28T19:07:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 55/152 2024-08-28T19:07:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-08-28T19:07:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 57/152 2024-08-28T19:07:42.425 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-08-28T19:07:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-08-28T19:07:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 60/152 2024-08-28T19:07:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-08-28T19:07:42.707 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-08-28T19:07:42.740 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-08-28T19:07:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-08-28T19:07:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-08-28T19:07:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-08-28T19:07:42.893 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 67/152 2024-08-28T19:07:42.932 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-08-28T19:07:42.962 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-08-28T19:07:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-08-28T19:07:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-08-28T19:07:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-08-28T19:07:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-08-28T19:07:43.205 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-08-28T19:07:43.254 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-08-28T19:07:43.284 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-08-28T19:07:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-08-28T19:07:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 78/152 2024-08-28T19:07:44.234 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/152 2024-08-28T19:07:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 80/152 2024-08-28T19:07:44.296 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-28T19:07:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 81/152 2024-08-28T19:07:44.334 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-28T19:07:44.423 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 82/152 2024-08-28T19:07:44.472 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.19-7.el9.x86_64 83/152 2024-08-28T19:07:44.526 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 84/152 2024-08-28T19:07:44.627 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-08-28T19:07:44.677 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-08-28T19:07:44.871 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.26-1.el9.x86_64 87/152 2024-08-28T19:07:44.899 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.26-1.el9.x86_64 88/152 2024-08-28T19:07:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-28T19:07:44.970 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 104/152 2024-08-28T19:07:44.980 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-08-28T19:07:45.063 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 104/152 2024-08-28T19:07:45.063 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-28T19:07:45.063 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-28T19:07:45.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:07:45.105 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 105/152 2024-08-28T19:07:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-08-28T19:07:45.793 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-08-28T19:07:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-08-28T19:07:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-28T19:07:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-08-28T19:07:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-08-28T19:07:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-08-28T19:07:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 97/152 2024-08-28T19:07:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 97/152 2024-08-28T19:07:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 98/152 2024-08-28T19:07:47.800 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-08-28T19:07:48.327 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 100/152 2024-08-28T19:07:48.361 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 101/152 2024-08-28T19:07:48.424 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 101/152 2024-08-28T19:07:48.463 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 102/152 2024-08-28T19:07:50.810 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-28T19:07:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-28T19:07:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-28T19:07:51.496 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 104/152 2024-08-28T19:07:51.579 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 104/152 2024-08-28T19:07:51.580 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-28T19:07:51.580 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-28T19:07:51.580 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:07:51.622 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 105/152 2024-08-28T19:07:52.081 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 105/152 2024-08-28T19:07:52.081 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /sys 2024-08-28T19:07:52.082 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /proc 2024-08-28T19:07:52.082 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /mnt 2024-08-28T19:07:52.082 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /var/tmp 2024-08-28T19:07:52.082 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /home 2024-08-28T19:07:52.082 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /root 2024-08-28T19:07:52.082 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /tmp 2024-08-28T19:07:52.082 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:07:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 106/152 2024-08-28T19:07:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 106/152 2024-08-28T19:07:52.268 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:07:52.268 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-28T19:07:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:07:53.855 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 107/152 2024-08-28T19:07:53.881 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 107/152 2024-08-28T19:07:53.882 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:07:53.882 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-28T19:07:53.882 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-28T19:07:53.882 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-28T19:07:53.882 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:07:54.105 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 108/152 2024-08-28T19:07:54.128 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 108/152 2024-08-28T19:07:54.128 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:07:54.129 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-28T19:07:54.129 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-28T19:07:54.129 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-28T19:07:54.129 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:07:54.589 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 109/152 2024-08-28T19:07:54.613 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 109/152 2024-08-28T19:07:54.613 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:07:54.613 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-28T19:07:54.613 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-28T19:07:54.613 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-28T19:07:54.613 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:07:54.719 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-08-28T19:07:54.776 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-08-28T19:07:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-08-28T19:07:55.015 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-08-28T19:07:55.082 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-08-28T19:07:55.151 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-08-28T19:07:55.239 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-08-28T19:07:55.377 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-08-28T19:07:55.415 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-08-28T19:07:55.447 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-08-28T19:07:55.475 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fuse-2.9.9-16.el9.x86_64 120/152 2024-08-28T19:07:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 121/152 2024-08-28T19:07:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 121/152 2024-08-28T19:07:55.972 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.e 122/152 2024-08-28T19:07:56.054 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118. 123/152 2024-08-28T19:07:56.080 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 124/152 2024-08-28T19:07:57.090 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 105/152 2024-08-28T19:07:57.090 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-08-28T19:07:57.090 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-08-28T19:07:57.090 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-08-28T19:07:57.090 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-08-28T19:07:57.090 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-08-28T19:07:57.090 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-08-28T19:07:57.091 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-08-28T19:07:57.091 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:07:57.278 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 106/152 2024-08-28T19:07:57.297 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 106/152 2024-08-28T19:07:57.297 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:07:57.297 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-28T19:07:57.297 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 107/152 2024-08-28T19:07:58.966 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 107/152 2024-08-28T19:07:58.966 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:07:58.966 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-28T19:07:58.966 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-28T19:07:58.966 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-28T19:07:58.966 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:07:59.198 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 108/152 2024-08-28T19:07:59.222 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 108/152 2024-08-28T19:07:59.222 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:07:59.222 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-28T19:07:59.222 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-28T19:07:59.222 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-28T19:07:59.222 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:07:59.233 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 124/152 2024-08-28T19:07:59.286 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 125/152 2024-08-28T19:07:59.707 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 109/152 2024-08-28T19:07:59.730 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 109/152 2024-08-28T19:07:59.731 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:07:59.731 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-28T19:07:59.731 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-28T19:07:59.731 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-28T19:07:59.731 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:07:59.847 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-08-28T19:07:59.905 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-08-28T19:07:59.968 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-08-28T19:08:00.162 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-08-28T19:08:00.209 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-08-28T19:08:00.247 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 125/152 2024-08-28T19:08:00.277 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 126/152 2024-08-28T19:08:00.290 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-08-28T19:08:00.386 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-08-28T19:08:00.394 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 126/152 2024-08-28T19:08:00.528 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-08-28T19:08:00.561 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el 127/152 2024-08-28T19:08:00.567 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-08-28T19:08:00.588 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 128/152 2024-08-28T19:08:00.599 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-08-28T19:08:00.612 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 128/152 2024-08-28T19:08:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-28T19:08:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-28T19:08:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-28T19:08:00.613 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:08:00.626 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-16.el9.x86_64 120/152 2024-08-28T19:08:00.664 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 129/152 2024-08-28T19:08:00.676 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 129/152 2024-08-28T19:08:00.775 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 130/152 2024-08-28T19:08:00.820 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 131/152 2024-08-28T19:08:01.125 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 121/152 2024-08-28T19:08:01.150 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 121/152 2024-08-28T19:08:01.183 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.e 122/152 2024-08-28T19:08:01.251 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118. 123/152 2024-08-28T19:08:01.290 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 124/152 2024-08-28T19:08:01.784 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libtool-2.4.6-46.el9.x86_64 132/152 2024-08-28T19:08:01.825 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 133/152 2024-08-28T19:08:01.849 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 133/152 2024-08-28T19:08:01.849 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:01.849 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-28T19:08:01.849 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-28T19:08:01.849 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-28T19:08:01.849 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:08:01.892 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 134/152 2024-08-28T19:08:01.915 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 134/152 2024-08-28T19:08:01.915 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:01.915 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-28T19:08:01.915 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:08:01.967 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 135/152 2024-08-28T19:08:01.991 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 135/152 2024-08-28T19:08:01.991 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:01.991 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-08-28T19:08:01.991 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-28T19:08:01.991 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-28T19:08:01.991 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:08:02.318 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 136/152 2024-08-28T19:08:02.341 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 136/152 2024-08-28T19:08:02.342 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:02.342 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-28T19:08:02.342 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-28T19:08:02.342 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-28T19:08:02.342 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:08:03.428 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 105/152 2024-08-28T19:08:03.428 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-08-28T19:08:03.428 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-08-28T19:08:03.428 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-08-28T19:08:03.428 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-08-28T19:08:03.428 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-08-28T19:08:03.428 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-08-28T19:08:03.429 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-08-28T19:08:03.429 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 106/152 2024-08-28T19:08:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 106/152 2024-08-28T19:08:03.618 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:03.618 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-28T19:08:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:04.640 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 124/152 2024-08-28T19:08:04.688 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 125/152 2024-08-28T19:08:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 107/152 2024-08-28T19:08:05.290 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 107/152 2024-08-28T19:08:05.290 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:05.290 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-28T19:08:05.291 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-28T19:08:05.291 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-28T19:08:05.291 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 108/152 2024-08-28T19:08:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 108/152 2024-08-28T19:08:05.546 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:05.546 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-28T19:08:05.546 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-28T19:08:05.546 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-28T19:08:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:05.699 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 125/152 2024-08-28T19:08:05.730 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 126/152 2024-08-28T19:08:05.848 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 126/152 2024-08-28T19:08:06.021 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el 127/152 2024-08-28T19:08:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 109/152 2024-08-28T19:08:06.049 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 128/152 2024-08-28T19:08:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 109/152 2024-08-28T19:08:06.056 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:06.056 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-28T19:08:06.056 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-28T19:08:06.056 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-28T19:08:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:06.072 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 128/152 2024-08-28T19:08:06.073 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:06.073 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-28T19:08:06.073 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-28T19:08:06.073 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-28T19:08:06.073 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:08:06.124 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 129/152 2024-08-28T19:08:06.136 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 129/152 2024-08-28T19:08:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-08-28T19:08:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-08-28T19:08:06.239 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 130/152 2024-08-28T19:08:06.291 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 131/152 2024-08-28T19:08:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-08-28T19:08:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-08-28T19:08:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-08-28T19:08:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-08-28T19:08:06.680 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 137/152 2024-08-28T19:08:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-08-28T19:08:06.730 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 138/152 2024-08-28T19:08:06.770 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 139/152 2024-08-28T19:08:06.802 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 140/152 2024-08-28T19:08:06.829 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 141/152 2024-08-28T19:08:06.864 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-08-28T19:08:06.901 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-08-28T19:08:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-08-28T19:08:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-16.el9.x86_64 120/152 2024-08-28T19:08:07.009 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 142/152 2024-08-28T19:08:07.059 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 143/152 2024-08-28T19:08:07.089 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_ 144/152 2024-08-28T19:08:07.302 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libtool-2.4.6-46.el9.x86_64 132/152 2024-08-28T19:08:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 121/152 2024-08-28T19:08:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 121/152 2024-08-28T19:08:07.335 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 133/152 2024-08-28T19:08:07.359 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 133/152 2024-08-28T19:08:07.359 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:07.359 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-28T19:08:07.359 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-28T19:08:07.359 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-28T19:08:07.360 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:08:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.e 122/152 2024-08-28T19:08:07.403 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 134/152 2024-08-28T19:08:07.426 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 134/152 2024-08-28T19:08:07.426 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:07.426 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-28T19:08:07.426 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:08:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118. 123/152 2024-08-28T19:08:07.473 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 124/152 2024-08-28T19:08:07.479 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 135/152 2024-08-28T19:08:07.501 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 135/152 2024-08-28T19:08:07.502 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:07.502 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-08-28T19:08:07.502 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-28T19:08:07.502 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-28T19:08:07.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:08:07.661 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-28T19:08:07.704 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-28T19:08:07.724 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-28T19:08:07.838 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 136/152 2024-08-28T19:08:07.861 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 136/152 2024-08-28T19:08:07.861 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:07.861 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-28T19:08:07.861 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-28T19:08:07.861 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-28T19:08:07.861 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:08:08.552 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 146/152 2024-08-28T19:08:08.642 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 147/152 2024-08-28T19:08:08.683 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-08-28T19:08:08.764 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-08-28T19:08:08.796 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-08-28T19:08:08.796 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-28T19:08:08.880 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-28T19:08:08.880 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-28T19:08:10.834 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 124/152 2024-08-28T19:08:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 125/152 2024-08-28T19:08:11.911 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 125/152 2024-08-28T19:08:11.955 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 126/152 2024-08-28T19:08:12.073 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 126/152 2024-08-28T19:08:12.247 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el 127/152 2024-08-28T19:08:12.273 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 128/152 2024-08-28T19:08:12.297 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 128/152 2024-08-28T19:08:12.297 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:12.297 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-28T19:08:12.297 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-28T19:08:12.297 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-28T19:08:12.297 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:12.349 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 129/152 2024-08-28T19:08:12.361 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 129/152 2024-08-28T19:08:12.442 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 137/152 2024-08-28T19:08:12.464 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 130/152 2024-08-28T19:08:12.484 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 138/152 2024-08-28T19:08:12.515 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 131/152 2024-08-28T19:08:12.524 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 139/152 2024-08-28T19:08:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 140/152 2024-08-28T19:08:12.584 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 141/152 2024-08-28T19:08:12.777 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 142/152 2024-08-28T19:08:12.821 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 143/152 2024-08-28T19:08:12.851 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_ 144/152 2024-08-28T19:08:13.385 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-28T19:08:13.423 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-28T19:08:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-28T19:08:13.529 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-46.el9.x86_64 132/152 2024-08-28T19:08:13.561 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 133/152 2024-08-28T19:08:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 133/152 2024-08-28T19:08:13.584 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:13.584 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-28T19:08:13.585 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-28T19:08:13.585 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-28T19:08:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 134/152 2024-08-28T19:08:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 134/152 2024-08-28T19:08:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-28T19:08:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:13.703 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 135/152 2024-08-28T19:08:13.726 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 135/152 2024-08-28T19:08:13.726 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:13.726 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-08-28T19:08:13.726 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-28T19:08:13.726 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-28T19:08:13.727 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:14.071 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 136/152 2024-08-28T19:08:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 136/152 2024-08-28T19:08:14.093 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-28T19:08:14.094 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-28T19:08:14.094 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-28T19:08:14.094 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-28T19:08:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:14.337 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 146/152 2024-08-28T19:08:14.429 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 147/152 2024-08-28T19:08:14.463 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-08-28T19:08:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-08-28T19:08:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-08-28T19:08:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-28T19:08:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-28T19:08:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-28T19:08:15.447 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-28T19:08:15.447 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 1/152 2024-08-28T19:08:15.447 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 2/152 2024-08-28T19:08:15.448 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 3/152 2024-08-28T19:08:15.448 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 4/152 2024-08-28T19:08:15.448 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 5/152 2024-08-28T19:08:15.448 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-28T19:08:15.448 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 7/152 2024-08-28T19:08:15.448 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 8/152 2024-08-28T19:08:15.448 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 9/152 2024-08-28T19:08:15.448 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 10/152 2024-08-28T19:08:15.449 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 11/152 2024-08-28T19:08:15.450 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-28T19:08:15.450 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 13/152 2024-08-28T19:08:15.450 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 14/152 2024-08-28T19:08:15.450 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_ 15/152 2024-08-28T19:08:15.450 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 16/152 2024-08-28T19:08:15.450 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 17/152 2024-08-28T19:08:15.451 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_6 18/152 2024-08-28T19:08:15.451 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 19/152 2024-08-28T19:08:15.452 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 20/152 2024-08-28T19:08:15.452 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el 21/152 2024-08-28T19:08:15.452 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-common-2:19.3.0-4560.gd19e6118.el9. 22/152 2024-08-28T19:08:15.452 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_6 23/152 2024-08-28T19:08:15.452 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 24/152 2024-08-28T19:08:15.452 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 25/152 2024-08-28T19:08:15.452 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 26/152 2024-08-28T19:08:15.452 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 27/152 2024-08-28T19:08:15.452 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 28/152 2024-08-28T19:08:15.453 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 29/152 2024-08-28T19:08:15.463 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118. 30/152 2024-08-28T19:08:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 31/152 2024-08-28T19:08:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 32/152 2024-08-28T19:08:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 33/152 2024-08-28T19:08:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el 34/152 2024-08-28T19:08:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 35/152 2024-08-28T19:08:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.e 36/152 2024-08-28T19:08:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 37/152 2024-08-28T19:08:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 38/152 2024-08-28T19:08:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 39/152 2024-08-28T19:08:15.464 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/152 2024-08-28T19:08:15.465 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/152 2024-08-28T19:08:15.465 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/152 2024-08-28T19:08:15.466 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-08-28T19:08:15.466 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-08-28T19:08:15.466 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-08-28T19:08:15.466 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-08-28T19:08:15.466 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-08-28T19:08:15.466 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/152 2024-08-28T19:08:15.466 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-08-28T19:08:15.466 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-08-28T19:08:15.466 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-08-28T19:08:15.466 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-08-28T19:08:15.467 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-08-28T19:08:15.468 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-0.3.26-1.el9.x86_64 66/152 2024-08-28T19:08:15.468 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-openmp-0.3.26-1.el9.x86_64 67/152 2024-08-28T19:08:15.468 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/152 2024-08-28T19:08:15.468 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/152 2024-08-28T19:08:15.468 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/152 2024-08-28T19:08:15.468 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/152 2024-08-28T19:08:15.468 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/152 2024-08-28T19:08:15.468 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-08-28T19:08:15.468 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-08-28T19:08:15.468 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-08-28T19:08:15.469 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/152 2024-08-28T19:08:15.469 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-devel-3.9.19-7.el9.x86_64 77/152 2024-08-28T19:08:15.470 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/152 2024-08-28T19:08:15.470 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/152 2024-08-28T19:08:15.470 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/152 2024-08-28T19:08:15.470 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/152 2024-08-28T19:08:15.470 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/152 2024-08-28T19:08:15.470 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/152 2024-08-28T19:08:15.470 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/152 2024-08-28T19:08:15.470 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/152 2024-08-28T19:08:15.470 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/152 2024-08-28T19:08:15.470 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/152 2024-08-28T19:08:15.471 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/152 2024-08-28T19:08:15.472 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-08-28T19:08:15.473 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-08-28T19:08:15.474 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 149/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-08-28T19:08:15.475 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 151/152 2024-08-28T19:08:17.820 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-08-28T19:08:17.820 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:08:17.820 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: automake-1.16.2-8.el9.noarch 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: bison-3.7.4-5.el9.x86_64 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.821 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:17.822 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: flex-2.6.4-9.el9.x86_64 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: fuse-2.9.9-16.el9.x86_64 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-08-28T19:08:17.823 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-08-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi115.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi115.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-08-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: libtool-2.4.6-46.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: lua-5.4.4-4.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-0.3.26-1.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-openmp-0.3.26-1.el9.x86_64 2024-08-28T19:08:17.825 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-13.16-1.el9.x86_64 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-server-13.16-1.el9.x86_64 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.826 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-devel-3.9.19-7.el9.x86_64 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ply-3.11-14.el9.noarch 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-28T19:08:17.829 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi115.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob-1.8.8-2.el9.noarch 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: re2-1:20211101-20.el9.x86_64 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi115.stdout: uuid-1.6.2-55.el9.x86_64 2024-08-28T19:08:17.832 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-08-28T19:08:17.832 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-28T19:08:17.832 INFO:teuthology.orchestra.run.smithi115.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-08-28T19:08:17.832 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:08:17.832 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-08-28T19:08:18.167 DEBUG:teuthology.parallel:result is None 2024-08-28T19:08:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 137/152 2024-08-28T19:08:18.742 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 138/152 2024-08-28T19:08:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 139/152 2024-08-28T19:08:18.814 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 140/152 2024-08-28T19:08:18.842 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 141/152 2024-08-28T19:08:19.035 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 142/152 2024-08-28T19:08:19.080 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 143/152 2024-08-28T19:08:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_ 144/152 2024-08-28T19:08:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-28T19:08:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-28T19:08:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-28T19:08:20.635 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 146/152 2024-08-28T19:08:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 147/152 2024-08-28T19:08:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-08-28T19:08:20.880 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-08-28T19:08:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-08-28T19:08:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-28T19:08:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-28T19:08:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-28T19:08:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-28T19:08:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 1/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 2/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 3/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 4/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 5/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 7/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 8/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 9/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 10/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 11/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-28T19:08:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 13/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 14/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_ 15/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 16/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 17/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_6 18/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 19/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 20/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el 21/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.3.0-4560.gd19e6118.el9. 22/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_6 23/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 24/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 25/152 2024-08-28T19:08:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 26/152 2024-08-28T19:08:21.371 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 27/152 2024-08-28T19:08:21.371 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 28/152 2024-08-28T19:08:21.381 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 29/152 2024-08-28T19:08:21.381 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118. 30/152 2024-08-28T19:08:21.381 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 31/152 2024-08-28T19:08:21.381 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 32/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 33/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el 34/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 35/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.e 36/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 37/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 38/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 39/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-08-28T19:08:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-08-28T19:08:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.26-1.el9.x86_64 66/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.26-1.el9.x86_64 67/152 2024-08-28T19:08:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.19-7.el9.x86_64 77/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/152 2024-08-28T19:08:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/152 2024-08-28T19:08:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/152 2024-08-28T19:08:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/152 2024-08-28T19:08:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-08-28T19:08:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/152 2024-08-28T19:08:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/152 2024-08-28T19:08:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-08-28T19:08:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-08-28T19:08:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-08-28T19:08:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/152 2024-08-28T19:08:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-08-28T19:08:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 149/152 2024-08-28T19:08:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-08-28T19:08:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 151/152 2024-08-28T19:08:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-08-28T19:08:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:08:23.750 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-08-28T19:08:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.750 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-08-28T19:08:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-28T19:08:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: automake-1.16.2-8.el9.noarch 2024-08-28T19:08:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: bison-3.7.4-5.el9.x86_64 2024-08-28T19:08:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-08-28T19:08:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: flex-2.6.4-9.el9.x86_64 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.9-16.el9.x86_64 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: libtool-2.4.6-46.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-28T19:08:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.26-1.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.26-1.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-13.16-1.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server-13.16-1.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-28T19:08:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-28T19:08:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.19-7.el9.x86_64 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-28T19:08:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-28T19:08:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-28T19:08:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.8-2.el9.noarch 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-28T19:08:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-20.el9.x86_64 2024-08-28T19:08:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-28T19:08:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-28T19:08:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: uuid-1.6.2-55.el9.x86_64 2024-08-28T19:08:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-08-28T19:08:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-28T19:08:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-08-28T19:08:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:08:23.762 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-08-28T19:08:24.089 DEBUG:teuthology.parallel:result is None 2024-08-28T19:08:27.754 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-28T19:08:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 1/152 2024-08-28T19:08:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 2/152 2024-08-28T19:08:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 3/152 2024-08-28T19:08:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 4/152 2024-08-28T19:08:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 5/152 2024-08-28T19:08:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-28T19:08:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 7/152 2024-08-28T19:08:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 8/152 2024-08-28T19:08:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 9/152 2024-08-28T19:08:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 10/152 2024-08-28T19:08:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 11/152 2024-08-28T19:08:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-28T19:08:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 13/152 2024-08-28T19:08:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 14/152 2024-08-28T19:08:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_ 15/152 2024-08-28T19:08:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 16/152 2024-08-28T19:08:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 17/152 2024-08-28T19:08:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_6 18/152 2024-08-28T19:08:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 19/152 2024-08-28T19:08:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 20/152 2024-08-28T19:08:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el 21/152 2024-08-28T19:08:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.3.0-4560.gd19e6118.el9. 22/152 2024-08-28T19:08:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_6 23/152 2024-08-28T19:08:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 24/152 2024-08-28T19:08:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 25/152 2024-08-28T19:08:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 26/152 2024-08-28T19:08:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 27/152 2024-08-28T19:08:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 28/152 2024-08-28T19:08:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 29/152 2024-08-28T19:08:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118. 30/152 2024-08-28T19:08:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 31/152 2024-08-28T19:08:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 32/152 2024-08-28T19:08:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 33/152 2024-08-28T19:08:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el 34/152 2024-08-28T19:08:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 35/152 2024-08-28T19:08:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.e 36/152 2024-08-28T19:08:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 37/152 2024-08-28T19:08:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 38/152 2024-08-28T19:08:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 39/152 2024-08-28T19:08:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/152 2024-08-28T19:08:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/152 2024-08-28T19:08:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/152 2024-08-28T19:08:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-08-28T19:08:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-08-28T19:08:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-08-28T19:08:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-08-28T19:08:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.26-1.el9.x86_64 66/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.26-1.el9.x86_64 67/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/152 2024-08-28T19:08:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/152 2024-08-28T19:08:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-08-28T19:08:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-08-28T19:08:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-08-28T19:08:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/152 2024-08-28T19:08:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.19-7.el9.x86_64 77/152 2024-08-28T19:08:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/152 2024-08-28T19:08:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/152 2024-08-28T19:08:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/152 2024-08-28T19:08:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/152 2024-08-28T19:08:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/152 2024-08-28T19:08:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/152 2024-08-28T19:08:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/152 2024-08-28T19:08:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/152 2024-08-28T19:08:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/152 2024-08-28T19:08:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/152 2024-08-28T19:08:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/152 2024-08-28T19:08:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/152 2024-08-28T19:08:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/152 2024-08-28T19:08:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/152 2024-08-28T19:08:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/152 2024-08-28T19:08:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/152 2024-08-28T19:08:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/152 2024-08-28T19:08:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/152 2024-08-28T19:08:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-08-28T19:08:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/152 2024-08-28T19:08:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-08-28T19:08:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-08-28T19:08:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-08-28T19:08:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/152 2024-08-28T19:08:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-08-28T19:08:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 149/152 2024-08-28T19:08:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-08-28T19:08:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 151/152 2024-08-28T19:08:30.178 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-08-28T19:08:30.178 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:30.178 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-08-28T19:08:30.178 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.178 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.179 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.180 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-16.el9.x86_64 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-28T19:08:30.181 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.182 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-46.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.26-1.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.26-1.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-08-28T19:08:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.16-1.el9.x86_64 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.16-1.el9.x86_64 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.184 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.19-7.el9.x86_64 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-28T19:08:30.185 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-28T19:08:30.186 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-08-28T19:08:30.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-28T19:08:30.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.8-2.el9.noarch 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-20.el9.x86_64 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-28T19:08:30.189 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-08-28T19:08:30.190 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-08-28T19:08:30.190 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-28T19:08:30.190 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-08-28T19:08:30.190 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:08:30.190 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-08-28T19:08:30.522 DEBUG:teuthology.parallel:result is None 2024-08-28T19:08:30.523 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:08:30.672 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-28T19:08:30.698 INFO:teuthology.orchestra.run.smithi016.stdout:19.3.0-4560.gd19e6118.el9 2024-08-28T19:08:30.699 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4560.gd19e6118.el9 2024-08-28T19:08:30.699 INFO:teuthology.task.install:The correct ceph version 19.3.0-4560.gd19e6118 is installed. 2024-08-28T19:08:30.701 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:08:30.852 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-28T19:08:30.873 INFO:teuthology.orchestra.run.smithi028.stdout:19.3.0-4560.gd19e6118.el9 2024-08-28T19:08:30.874 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4560.gd19e6118.el9 2024-08-28T19:08:30.874 INFO:teuthology.task.install:The correct ceph version 19.3.0-4560.gd19e6118 is installed. 2024-08-28T19:08:30.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:08:31.022 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-28T19:08:31.048 INFO:teuthology.orchestra.run.smithi115.stdout:19.3.0-4560.gd19e6118.el9 2024-08-28T19:08:31.049 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4560.gd19e6118.el9 2024-08-28T19:08:31.049 INFO:teuthology.task.install:The correct ceph version 19.3.0-4560.gd19e6118 is installed. 2024-08-28T19:08:31.051 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-28T19:08:31.051 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:08:31.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-28T19:08:31.083 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-08-28T19:08:31.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-28T19:08:31.111 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-08-28T19:08:31.111 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-28T19:08:31.142 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-28T19:08:31.142 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:08:31.142 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-08-28T19:08:31.171 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-28T19:08:31.238 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-08-28T19:08:31.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-08-28T19:08:31.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-28T19:08:31.332 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-08-28T19:08:31.332 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2024-08-28T19:08:31.361 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-28T19:08:31.429 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-28T19:08:31.430 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:08:31.430 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-28T19:08:31.459 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-28T19:08:31.527 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-08-28T19:08:31.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-28T19:08:31.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-28T19:08:31.616 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-08-28T19:08:31.616 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-28T19:08:31.641 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-28T19:08:31.706 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-28T19:08:31.707 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:08:31.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-08-28T19:08:31.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-28T19:08:31.805 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-08-28T19:08:31.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-08-28T19:08:31.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-28T19:08:31.893 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-08-28T19:08:31.893 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/stdin-killer 2024-08-28T19:08:31.918 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-28T19:08:31.983 INFO:teuthology.run_tasks:Running task cephadm... 2024-08-28T19:08:32.071 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug cephsqlite': 20, 'mgr/snap_schedule/allow_m_granularity': True, 'mgr/snap_schedule/dump_on_update': True, 'mgr/snap_schedule/log_level': 'debug'}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'global': {'ms die on skipped message': False}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'mgr-modules': ['snap_schedule'], 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-08-28T19:08:32.072 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:08:32.072 INFO:tasks.cephadm:Cluster fsid is eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:08:32.072 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-08-28T19:08:32.072 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.16', 'mon.b': '172.21.15.28', 'mon.c': '172.21.15.115'} 2024-08-28T19:08:32.072 INFO:tasks.cephadm:First mon is mon.a on smithi016 2024-08-28T19:08:32.073 INFO:tasks.cephadm:First mgr is x 2024-08-28T19:08:32.073 INFO:tasks.cephadm:Normalizing hostnames... 2024-08-28T19:08:32.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-08-28T19:08:32.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-08-28T19:08:32.128 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname $(hostname -s) 2024-08-28T19:08:32.152 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-08-28T19:08:32.153 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:08:32.302 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/82586/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4560-gd19e6118', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4560.gd19e6118'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-08-28 11:15:32.801765', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-20240828.085847-main-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-08-28T19:08:32.447 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-20240828.085847-main-debug, sha1 d19e6118f7015f0047264006cb0c873528eb3ae6 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:08:32.448 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/x86_64/flavors/default/cephadm 2024-08-28T19:08:32.448 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/x86_64/flavors/default/cephadm 2024-08-28T19:08:32.448 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-28T19:08:32.822 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r--. 1 ubuntu ubuntu 801010 Aug 28 19:08 /home/ubuntu/cephtest/cephadm 2024-08-28T19:08:32.822 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-28T19:08:33.179 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 801010 Aug 28 19:08 /home/ubuntu/cephtest/cephadm 2024-08-28T19:08:33.179 DEBUG:teuthology.orchestra.run.smithi115:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-28T19:08:33.542 INFO:teuthology.orchestra.run.smithi115.stdout:-rw-r--r--. 1 ubuntu ubuntu 801010 Aug 28 19:08 /home/ubuntu/cephtest/cephadm 2024-08-28T19:08:33.543 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-28T19:08:33.562 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-28T19:08:33.579 DEBUG:teuthology.orchestra.run.smithi115:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-28T19:08:33.613 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 on all hosts... 2024-08-28T19:08:33.613 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 pull 2024-08-28T19:08:33.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 pull 2024-08-28T19:08:33.621 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 pull 2024-08-28T19:08:35.224 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6... 2024-08-28T19:08:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6... 2024-08-28T19:08:35.330 INFO:teuthology.orchestra.run.smithi115.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6... 2024-08-28T19:09:36.989 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:09:36.990 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.3.0-4560-gd19e6118 (d19e6118f7015f0047264006cb0c873528eb3ae6) squid (dev)", 2024-08-28T19:09:36.990 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0", 2024-08-28T19:09:36.990 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-08-28T19:09:36.990 INFO:teuthology.orchestra.run.smithi016.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61" 2024-08-28T19:09:36.990 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:09:36.990 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:09:53.299 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-08-28T19:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.3.0-4560-gd19e6118 (d19e6118f7015f0047264006cb0c873528eb3ae6) squid (dev)", 2024-08-28T19:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0", 2024-08-28T19:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-08-28T19:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61" 2024-08-28T19:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-08-28T19:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-08-28T19:09:59.956 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-08-28T19:09:59.957 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_version": "ceph version 19.3.0-4560-gd19e6118 (d19e6118f7015f0047264006cb0c873528eb3ae6) squid (dev)", 2024-08-28T19:09:59.957 INFO:teuthology.orchestra.run.smithi115.stdout: "image_id": "79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0", 2024-08-28T19:09:59.957 INFO:teuthology.orchestra.run.smithi115.stdout: "repo_digests": [ 2024-08-28T19:09:59.957 INFO:teuthology.orchestra.run.smithi115.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61" 2024-08-28T19:09:59.957 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-08-28T19:09:59.957 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-08-28T19:09:59.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-08-28T19:10:00.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-08-28T19:10:00.049 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph 2024-08-28T19:10:00.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-08-28T19:10:00.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-08-28T19:10:00.132 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /etc/ceph 2024-08-28T19:10:00.157 INFO:tasks.cephadm:Writing seed config... 2024-08-28T19:10:00.158 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-08-28T19:10:00.158 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-08-28T19:10:00.158 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-08-28T19:10:00.158 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-08-28T19:10:00.158 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-08-28T19:10:00.159 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-08-28T19:10:00.160 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-08-28T19:10:00.160 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-08-28T19:10:00.160 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-08-28T19:10:00.160 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-08-28T19:10:00.160 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-08-28T19:10:00.160 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:10:00.160 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-08-28T19:10:00.178 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eabc438a-6570-11ef-bcd4-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-08-28T19:10:00.178 DEBUG:teuthology.orchestra.run.smithi016:mon.a> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.a.service 2024-08-28T19:10:00.221 DEBUG:teuthology.orchestra.run.smithi016:mgr.x> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.x.service 2024-08-28T19:10:00.264 INFO:tasks.cephadm:Bootstrapping... 2024-08-28T19:10:00.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 -v bootstrap --fsid eabc438a-6570-11ef-bcd4-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-08-28T19:10:00.481 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-08-28T19:10:00.481 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', '-v', 'bootstrap', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.16', '--skip-admin-label'] 2024-08-28T19:10:00.481 INFO:teuthology.orchestra.run.smithi016.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-08-28T19:10:00.482 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying podman|docker is present... 2024-08-28T19:10:00.506 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.2.1 2024-08-28T19:10:00.507 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying lvm2 is present... 2024-08-28T19:10:00.507 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying time synchronization is in place... 2024-08-28T19:10:00.514 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-28T19:10:00.514 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-28T19:10:00.521 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-28T19:10:00.521 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-08-28T19:10:00.527 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-08-28T19:10:00.534 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-08-28T19:10:00.534 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-08-28T19:10:00.534 INFO:teuthology.orchestra.run.smithi016.stdout:Repeating the final host check... 2024-08-28T19:10:00.557 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.2.1 2024-08-28T19:10:00.557 INFO:teuthology.orchestra.run.smithi016.stdout:podman (/bin/podman) version 5.2.1 is present 2024-08-28T19:10:00.557 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl is present 2024-08-28T19:10:00.557 INFO:teuthology.orchestra.run.smithi016.stdout:lvcreate is present 2024-08-28T19:10:00.563 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-28T19:10:00.564 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-28T19:10:00.570 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-28T19:10:00.570 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-08-28T19:10:00.576 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-08-28T19:10:00.583 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-08-28T19:10:00.583 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-08-28T19:10:00.583 INFO:teuthology.orchestra.run.smithi016.stdout:Host looks OK 2024-08-28T19:10:00.583 INFO:teuthology.orchestra.run.smithi016.stdout:Cluster fsid: eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:10:00.583 INFO:teuthology.orchestra.run.smithi016.stdout:Acquiring lock 140379876148272 on /run/cephadm/eabc438a-6570-11ef-bcd4-c7b262605968.lock 2024-08-28T19:10:00.583 INFO:teuthology.orchestra.run.smithi016.stdout:Lock 140379876148272 acquired on /run/cephadm/eabc438a-6570-11ef-bcd4-c7b262605968.lock 2024-08-28T19:10:00.584 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 3300 ... 2024-08-28T19:10:00.584 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 6789 ... 2024-08-28T19:10:00.584 INFO:teuthology.orchestra.run.smithi016.stdout:Base mon IP(s) is [172.21.15.16:3300, 172.21.15.16:6789], mon addrv is [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-08-28T19:10:00.587 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.16 metric 100 2024-08-28T19:10:00.587 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-08-28T19:10:00.594 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-08-28T19:10:00.594 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-08-28T19:10:00.594 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-08-28T19:10:00.594 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-08-28T19:10:00.594 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-08-28T19:10:00.594 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-28T19:10:00.594 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-08-28T19:10:00.594 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 fe80::1876:14a0:73d4:8f28/64 scope link noprefixroute 2024-08-28T19:10:00.594 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-28T19:10:00.595 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-08-28T19:10:00.595 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-08-28T19:10:00.595 INFO:teuthology.orchestra.run.smithi016.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-08-28T19:10:00.595 INFO:teuthology.orchestra.run.smithi016.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-08-28T19:10:00.595 INFO:teuthology.orchestra.run.smithi016.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6... 2024-08-28T19:10:01.065 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 2024-08-28T19:10:01.065 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6... 2024-08-28T19:10:01.065 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Getting image source signatures 2024-08-28T19:10:01.065 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:1c7f48e91f1eb9e77df623f13bab8a42aaf12d95954655c07fb62a1a85ca19b2 2024-08-28T19:10:01.066 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-08-28T19:10:01.066 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying config sha256:79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 2024-08-28T19:10:01.066 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Writing manifest to image destination 2024-08-28T19:10:05.458 INFO:teuthology.orchestra.run.smithi016.stdout:ceph: stdout ceph version 19.3.0-4560-gd19e6118 (d19e6118f7015f0047264006cb0c873528eb3ae6) squid (dev) 2024-08-28T19:10:05.458 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph version: ceph version 19.3.0-4560-gd19e6118 (d19e6118f7015f0047264006cb0c873528eb3ae6) squid (dev) 2024-08-28T19:10:05.458 INFO:teuthology.orchestra.run.smithi016.stdout:Extracting ceph user uid/gid from container image... 2024-08-28T19:10:14.431 INFO:teuthology.orchestra.run.smithi016.stdout:stat: stdout 167 167 2024-08-28T19:10:14.431 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial keys... 2024-08-28T19:10:15.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQAWds9mgY4xLhAAOq4BWxuxBZLrUzpaFlZeHg== 2024-08-28T19:10:16.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQAXds9mru1hJxAA+sRjxTIhCoL9FN9U0rMMVg== 2024-08-28T19:10:16.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQAYds9mmhPTHRAA+hJmwrPjro6V+UwPMFi0dw== 2024-08-28T19:10:16.991 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial monmap... 2024-08-28T19:10:17.788 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-28T19:10:17.788 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-08-28T19:10:17.788 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:10:17.788 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-28T19:10:17.788 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool for a [v2:172.21.15.16:3300,v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-28T19:10:17.789 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = quincy 2024-08-28T19:10:17.789 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: set fsid to eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:10:17.789 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-28T19:10:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:10:17.789 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mon... 2024-08-28T19:10:18.738 INFO:teuthology.orchestra.run.smithi016.stdout:create mon.a on 2024-08-28T19:10:18.946 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-08-28T19:10:19.082 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-08-28T19:10:19.264 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eabc438a-6570-11ef-bcd4-c7b262605968.target → /etc/systemd/system/ceph-eabc438a-6570-11ef-bcd4-c7b262605968.target. 2024-08-28T19:10:19.264 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-eabc438a-6570-11ef-bcd4-c7b262605968.target → /etc/systemd/system/ceph-eabc438a-6570-11ef-bcd4-c7b262605968.target. 2024-08-28T19:10:19.508 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.a 2024-08-28T19:10:19.508 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.a.service: Unit ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.a.service not loaded. 2024-08-28T19:10:19.747 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eabc438a-6570-11ef-bcd4-c7b262605968.target.wants/ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.a.service → /etc/systemd/system/ceph-eabc438a-6570-11ef-bcd4-c7b262605968@.service. 2024-08-28T19:10:20.286 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-28T19:10:20.286 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-08-28T19:10:20.294 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-28T19:10:20.294 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-08-28T19:10:20.295 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-08-28T19:10:20.295 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-08-28T19:10:20.295 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon to start... 2024-08-28T19:10:20.295 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon... 2024-08-28T19:10:21.320 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:21 smithi016 ceph-mon[18188]: from='client.? 172.21.15.16:0/975223845' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-08-28T19:10:21.323 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout cluster: 2024-08-28T19:10:21.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout id: eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:10:21.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-08-28T19:10:21.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:21.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout services: 2024-08-28T19:10:21.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.521177s) 2024-08-28T19:10:21.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-08-28T19:10:21.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-08-28T19:10:21.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:21.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout data: 2024-08-28T19:10:21.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-08-28T19:10:21.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-08-28T19:10:21.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-08-28T19:10:21.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pgs: 2024-08-28T19:10:21.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:21.328 INFO:teuthology.orchestra.run.smithi016.stdout:mon is available 2024-08-28T19:10:21.328 INFO:teuthology.orchestra.run.smithi016.stdout:Assimilating anything we can from ceph.conf... 2024-08-28T19:10:22.373 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:22.373 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-08-28T19:10:22.373 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:10:22.373 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-08-28T19:10:22.373 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-08-28T19:10:22.373 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-28T19:10:22.373 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-28T19:10:22.373 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-28T19:10:22.373 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-28T19:10:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-28T19:10:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-28T19:10:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-08-28T19:10:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-08-28T19:10:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-08-28T19:10:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-08-28T19:10:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-28T19:10:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-28T19:10:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:Generating new minimal ceph.conf... 2024-08-28T19:10:22.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:22 smithi016 ceph-mon[18188]: from='client.? 172.21.15.16:0/2196228130' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-08-28T19:10:22.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:22 smithi016 ceph-mon[18188]: from='client.? 172.21.15.16:0/2196228130' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-08-28T19:10:23.436 INFO:teuthology.orchestra.run.smithi016.stdout:Restarting the monitor... 2024-08-28T19:10:23.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:23 smithi016 ceph-mon[18188]: from='client.? 172.21.15.16:0/1906814812' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:10:23.704 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:23 smithi016 systemd[1]: Stopping Ceph mon.a for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:10:23.987 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a[18184]: 2024-08-28T19:10:23.701+0000 7f8959610640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:10:23.987 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a[18184]: 2024-08-28T19:10:23.701+0000 7f8959610640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-08-28T19:10:23.987 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:23 smithi016 podman[18433]: 2024-08-28 19:10:23.76385901 +0000 UTC m=+0.165324849 container died 1f7e33231200fa8ee2864745d97b8ccc9cf9c174e059b32b7cf8f3f4af9a1af9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-08-28T19:10:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 podman[18433]: 2024-08-28 19:10:24.077056104 +0000 UTC m=+0.478521949 container cleanup 1f7e33231200fa8ee2864745d97b8ccc9cf9c174e059b32b7cf8f3f4af9a1af9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, CEPH_POINT_RELEASE=) 2024-08-28T19:10:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 bash[18433]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a 2024-08-28T19:10:24.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 podman[18445]: 2024-08-28 19:10:24.253081539 +0000 UTC m=+0.486578508 container remove 1f7e33231200fa8ee2864745d97b8ccc9cf9c174e059b32b7cf8f3f4af9a1af9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-08-28T19:10:24.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 systemd[1]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.a.service: Deactivated successfully. 2024-08-28T19:10:24.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 systemd[1]: Stopped Ceph mon.a for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:10:24.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 systemd[1]: Starting Ceph mon.a for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:10:24.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 podman[18537]: 2024-08-28 19:10:24.687107905 +0000 UTC m=+0.097560415 container create 1408e30d070d47f0671111208cd536ed1611b528dc95b5824ce4b923cdb1f80a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-28T19:10:24.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 podman[18537]: 2024-08-28 19:10:24.605440547 +0000 UTC m=+0.015893075 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 podman[18537]: 2024-08-28 19:10:24.897905943 +0000 UTC m=+0.308358453 container init 1408e30d070d47f0671111208cd536ed1611b528dc95b5824ce4b923cdb1f80a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 podman[18537]: 2024-08-28 19:10:24.901086034 +0000 UTC m=+0.311538545 container start 1408e30d070d47f0671111208cd536ed1611b528dc95b5824ce4b923cdb1f80a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS) 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: set uid:gid to 167:167 (ceph:ceph) 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: ceph version 19.3.0-4560-gd19e6118 (d19e6118f7015f0047264006cb0c873528eb3ae6) squid (dev), process ceph-mon, pid 2 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: pidfile_write: ignore empty --pid-file 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: load: jerasure load: lrc 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: RocksDB version: 7.9.2 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Git sha 0 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Compile date 2024-08-28 10:19:06 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: DB SUMMARY 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: DB Session ID: IMFIEOEOCSZNJXRCLC72 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: CURRENT file: CURRENT 2024-08-28T19:10:24.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: IDENTITY file: IDENTITY 2024-08-28T19:10:24.940 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-08-28T19:10:24.940 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-08-28T19:10:24.940 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 93863 ; 2024-08-28T19:10:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.error_if_exists: 0 2024-08-28T19:10:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.create_if_missing: 0 2024-08-28T19:10:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.paranoid_checks: 1 2024-08-28T19:10:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-28T19:10:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-28T19:10:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-28T19:10:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.env: 0x564173bbd5a0 2024-08-28T19:10:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.fs: PosixFileSystem 2024-08-28T19:10:24.941 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.info_log: 0x56417968b4e0 2024-08-28T19:10:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_file_opening_threads: 16 2024-08-28T19:10:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.statistics: (nil) 2024-08-28T19:10:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.use_fsync: 0 2024-08-28T19:10:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_log_file_size: 0 2024-08-28T19:10:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-28T19:10:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-28T19:10:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.keep_log_file_num: 1000 2024-08-28T19:10:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.recycle_log_file_num: 0 2024-08-28T19:10:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.allow_fallocate: 1 2024-08-28T19:10:24.942 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.allow_mmap_reads: 0 2024-08-28T19:10:24.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.allow_mmap_writes: 0 2024-08-28T19:10:24.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.use_direct_reads: 0 2024-08-28T19:10:24.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-28T19:10:24.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.create_missing_column_families: 0 2024-08-28T19:10:24.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.db_log_dir: 2024-08-28T19:10:24.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.wal_dir: 2024-08-28T19:10:24.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-28T19:10:24.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-28T19:10:24.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-28T19:10:24.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-28T19:10:24.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.advise_random_on_open: 1 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.db_write_buffer_size: 0 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.write_buffer_manager: 0x56417968f5e0 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.rate_limiter: (nil) 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.wal_recovery_mode: 2 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.enable_thread_tracking: 0 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.enable_pipelined_write: 0 2024-08-28T19:10:24.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.unordered_write: 0 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.row_cache: None 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.wal_filter: None 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.allow_ingest_behind: 0 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.two_write_queues: 0 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.manual_wal_flush: 0 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.wal_compression: 0 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.atomic_flush: 0 2024-08-28T19:10:24.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.log_readahead_size: 0 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.best_efforts_recovery: 0 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.allow_data_in_errors: 0 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.db_host_id: __hostname__ 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_background_jobs: 2 2024-08-28T19:10:24.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_background_compactions: -1 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_subcompactions: 1 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_total_wal_size: 0 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_open_files: -1 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bytes_per_sync: 0 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-28T19:10:24.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_readahead_size: 0 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_background_flushes: -1 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Compression algorithms supported: 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: kZSTD supported: 0 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: kXpressCompression supported: 0 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: kBZip2Compression supported: 0 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: kLZ4Compression supported: 1 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: kZlibCompression supported: 1 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: kLZ4HCCompression supported: 1 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: kSnappyCompression supported: 1 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-28T19:10:24.948 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.merge_operator: 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_filter: None 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_filter_factory: None 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.sst_partitioner_factory: None 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56417968b100) 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-28T19:10:24.949 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 4 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x56417564e850 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-08-28T19:10:24.950 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: bloomfilter 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-08-28T19:10:24.951 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 5 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout: max_auto_readahead_size: 262144 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout: prepopulate_block_cache: 0 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout: initial_auto_readahead_size: 8192 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.write_buffer_size: 33554432 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_write_buffer_number: 2 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compression: NoCompression 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bottommost_compression: Disabled 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.prefix_extractor: nullptr 2024-08-28T19:10:24.952 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.num_levels: 7 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-28T19:10:24.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compression_opts.level: 32767 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compression_opts.strategy: 0 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compression_opts.enabled: false 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-28T19:10:24.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.target_file_size_base: 67108864 2024-08-28T19:10:24.955 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-28T19:10:24.955 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-28T19:10:24.955 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-28T19:10:24.955 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-28T19:10:24.955 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-28T19:10:24.955 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-28T19:10:24.995 INFO:teuthology.orchestra.run.smithi016.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-08-28T19:10:25.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-28T19:10:25.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-28T19:10:25.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-28T19:10:25.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-28T19:10:25.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-28T19:10:25.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-28T19:10:25.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-28T19:10:25.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-28T19:10:25.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.arena_block_size: 1048576 2024-08-28T19:10:25.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.disable_auto_compactions: 0 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-28T19:10:25.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.inplace_update_support: 0 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.bloom_locality: 0 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.max_successive_merges: 0 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.paranoid_file_checks: 0 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.force_consistency_checks: 1 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.report_bg_io_stats: 0 2024-08-28T19:10:25.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.ttl: 2592000 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.enable_blob_files: false 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.min_blob_size: 0 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.blob_file_size: 268435456 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.blob_file_starting_level: 0 2024-08-28T19:10:25.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 499f2590-437e-4e16-a1a8-f65a95d1d049 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724872224936704, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724872224937825, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90063, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 278, "table_properties": {"data_size": 88133, "index_size": 250, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11355, "raw_average_key_size": 47, "raw_value_size": 81780, "raw_average_value_size": 345, "num_data_blocks": 11, "num_entries": 237, "num_filter_entries": 237, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1724872224, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "499f2590-437e-4e16-a1a8-f65a95d1d049", "db_session_id": "IMFIEOEOCSZNJXRCLC72", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724872224937928, "job": 1, "event": "recovery_finished"} 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5641796aea80 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: DB pointer 0x56417a450000 2024-08-28T19:10:25.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-28T19:10:25.238 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 2/0 89.81 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 89.81 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 123.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-28T19:10:25.239 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 21.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 21.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache BinnedLRUCache@0x56417564e850#2 capacity: 512.00 MB usage: 1.28 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-28T19:10:25.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: starting mon.a rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: mon.a@-1(???) e1 preinit fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: mon.a@-1(???).mds e1 new map 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: mon.a@-1(???).mds e1 print_map 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout: btime 2024-08-28T19:10:20:239202+0000 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-28T19:10:25.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 bash[18537]: 1408e30d070d47f0671111208cd536ed1611b528dc95b5824ce4b923cdb1f80a 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 systemd[1]: Started Ceph mon.a for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: monmap epoch 1 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: last_changed 2024-08-28T19:10:17.308213+0000 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: created 2024-08-28T19:10:17.308213+0000 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: min_mon_release 19 (squid) 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: election_strategy: 1 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: fsmap 2024-08-28T19:10:25.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: osdmap e1: 0 total, 0 up, 0 in 2024-08-28T19:10:25.243 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:24 smithi016 ceph-mon[18556]: mgrmap e1: no daemons active 2024-08-28T19:10:26.000 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote config to /etc/ceph/ceph.conf 2024-08-28T19:10:26.021 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-08-28T19:10:26.021 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mgr... 2024-08-28T19:10:26.022 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:9283 ... 2024-08-28T19:10:26.022 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:8765 ... 2024-08-28T19:10:26.266 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.x 2024-08-28T19:10:26.266 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.x.service: Unit ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.x.service not loaded. 2024-08-28T19:10:26.413 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eabc438a-6570-11ef-bcd4-c7b262605968.target.wants/ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-eabc438a-6570-11ef-bcd4-c7b262605968@.service. 2024-08-28T19:10:26.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:26 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/169426945' entity='client.admin' 2024-08-28T19:10:26.926 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:26 smithi016 podman[18769]: 2024-08-28 19:10:26.845134482 +0000 UTC m=+0.312409105 container start 63620dafa962f73710b89df2bde3866d6dc59c690bec0ec7673a72ac9c492d45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-08-28T19:10:26.927 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:26 smithi016 bash[18769]: 63620dafa962f73710b89df2bde3866d6dc59c690bec0ec7673a72ac9c492d45 2024-08-28T19:10:26.941 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-28T19:10:26.941 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-08-28T19:10:26.950 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-28T19:10:26.950 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-08-28T19:10:26.950 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-08-28T19:10:26.950 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-08-28T19:10:26.958 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-28T19:10:26.958 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-08-28T19:10:26.966 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-28T19:10:26.966 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-08-28T19:10:26.966 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-08-28T19:10:26.966 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-08-28T19:10:26.967 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr to start... 2024-08-28T19:10:26.967 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr... 2024-08-28T19:10:27.196 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:26 smithi016 systemd[1]: Started Ceph mgr.x for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:10:27.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:27 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/624219284' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-08-28T19:10:28.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:28.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-08-28T19:10:28.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:10:28.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-08-28T19:10:28.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-28T19:10:28.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-28T19:10:28.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-28T19:10:28.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:28.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-28T19:10:28.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-28T19:10:28.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-08-28T19:10:28.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-08-28T19:10:28.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-28T19:10:28.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-08-28T19:10:28.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-08-28T19:10:28.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-08-28T19:10:28.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-28T19:10:28.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:28.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-28T19:10:28.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-28T19:10:28.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:28.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-28T19:10:28.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:28.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-28T19:10:28.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-28T19:10:28.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-28T19:10:28.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-28T19:10:28.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-28T19:10:28.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-28T19:10:28.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:28.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:28.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "btime": "2024-08-28T19:10:20:239202+0000", 2024-08-28T19:10:28.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-28T19:10:28.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-28T19:10:28.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:28.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-28T19:10:28.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-28T19:10:28.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-28T19:10:28.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-28T19:10:28.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-28T19:10:28.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-28T19:10:28.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-08-28T19:10:28.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-08-28T19:10:28.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-28T19:10:28.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:28.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-28T19:10:28.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:28.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-08-28T19:10:20.239982+0000", 2024-08-28T19:10:28.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-28T19:10:28.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:28.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-28T19:10:28.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-08-28T19:10:28.082 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (1/15)... 2024-08-28T19:10:30.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:30 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3943636530' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-08-28T19:10:31.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:31.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-08-28T19:10:31.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:10:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-08-28T19:10:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-28T19:10:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-28T19:10:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-28T19:10:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-28T19:10:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-28T19:10:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-08-28T19:10:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-08-28T19:10:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-28T19:10:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-08-28T19:10:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-08-28T19:10:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-08-28T19:10:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-28T19:10:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-28T19:10:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-28T19:10:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-28T19:10:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-28T19:10:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-28T19:10:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-28T19:10:31.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-28T19:10:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-28T19:10:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-28T19:10:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-28T19:10:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-28T19:10:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-28T19:10:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-28T19:10:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-28T19:10:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-28T19:10:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-28T19:10:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "btime": "2024-08-28T19:10:20:239202+0000", 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-28T19:10:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-08-28T19:10:20.239982+0000", 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-28T19:10:31.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-08-28T19:10:31.201 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (2/15)... 2024-08-28T19:10:32.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: Activating manager daemon x 2024-08-28T19:10:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: mgrmap e2: x(active, starting, since 0.00334061s) 2024-08-28T19:10:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: from='mgr.14100 172.21.15.16:0/206266317' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-08-28T19:10:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: from='mgr.14100 172.21.15.16:0/206266317' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-08-28T19:10:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: from='mgr.14100 172.21.15.16:0/206266317' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-08-28T19:10:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: from='mgr.14100 172.21.15.16:0/206266317' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:10:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: from='mgr.14100 172.21.15.16:0/206266317' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-28T19:10:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: Manager daemon x is now available 2024-08-28T19:10:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: from='mgr.14100 172.21.15.16:0/206266317' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-08-28T19:10:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: from='mgr.14100 172.21.15.16:0/206266317' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-08-28T19:10:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: from='mgr.14100 172.21.15.16:0/206266317' entity='mgr.x' 2024-08-28T19:10:32.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: from='mgr.14100 172.21.15.16:0/206266317' entity='mgr.x' 2024-08-28T19:10:32.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:32 smithi016 ceph-mon[18556]: from='mgr.14100 172.21.15.16:0/206266317' entity='mgr.x' 2024-08-28T19:10:34.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:34.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-08-28T19:10:34.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-08-28T19:10:34.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-08-28T19:10:34.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-28T19:10:34.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:34.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-28T19:10:34.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-28T19:10:34.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:34.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-28T19:10:34.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:34.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-28T19:10:34.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-28T19:10:34.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-28T19:10:34.365 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "btime": "2024-08-28T19:10:20:239202+0000", 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-28T19:10:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-08-28T19:10:20.239982+0000", 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-08-28T19:10:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:mgr is available 2024-08-28T19:10:34.485 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:34 smithi016 ceph-mon[18556]: mgrmap e3: x(active, since 1.00854s) 2024-08-28T19:10:34.485 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:34 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/600410938' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-08-28T19:10:35.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:35 smithi016 ceph-mon[18556]: mgrmap e4: x(active, since 2s) 2024-08-28T19:10:35.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:35 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3839289249' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-08-28T19:10:35.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:35.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-08-28T19:10:35.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-08-28T19:10:35.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-28T19:10:35.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-28T19:10:35.500 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling cephadm module... 2024-08-28T19:10:36.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:36 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/879821138' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-08-28T19:10:36.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:36 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: ignoring --setuser ceph since I am not root 2024-08-28T19:10:36.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:36 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: ignoring --setgroup ceph since I am not root 2024-08-28T19:10:37.243 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:37 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/879821138' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-08-28T19:10:37.243 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:37 smithi016 ceph-mon[18556]: mgrmap e5: x(active, since 4s) 2024-08-28T19:10:38.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-08-28T19:10:38.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-08-28T19:10:38.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-28T19:10:38.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-08-28T19:10:38.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-28T19:10:38.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-08-28T19:10:38.219 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-08-28T19:10:38.220 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 5... 2024-08-28T19:10:38.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:38 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2828221602' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-08-28T19:10:41.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:41 smithi016 ceph-mon[18556]: Active manager daemon x restarted 2024-08-28T19:10:41.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:41 smithi016 ceph-mon[18556]: Activating manager daemon x 2024-08-28T19:10:41.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:41 smithi016 ceph-mon[18556]: osdmap e2: 0 total, 0 up, 0 in 2024-08-28T19:10:41.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:41 smithi016 ceph-mon[18556]: mgrmap e6: x(active, starting, since 0.00344565s) 2024-08-28T19:10:41.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:41 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:10:41.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:41 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-28T19:10:41.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:41 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-08-28T19:10:41.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:41 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-08-28T19:10:41.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:41 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-08-28T19:10:41.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:41 smithi016 ceph-mon[18556]: Manager daemon x is now available 2024-08-28T19:10:42.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-08-28T19:10:42.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-08-28T19:10:42.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-28T19:10:42.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-08-28T19:10:42.945 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 5 is available 2024-08-28T19:10:42.946 INFO:teuthology.orchestra.run.smithi016.stdout:Setting orchestrator backend to cephadm... 2024-08-28T19:10:43.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:43 smithi016 ceph-mon[18556]: mgrmap e7: x(active, since 1.00588s) 2024-08-28T19:10:43.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:43 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:43.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:43 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:43.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:43 smithi016 ceph-mon[18556]: Found migration_current of "None". Setting to last migration. 2024-08-28T19:10:43.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:43 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:43.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:43 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:43.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:43 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:10:43.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:43 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:10:43.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:43 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-08-28T19:10:43.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:43 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-08-28T19:10:44.701 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-mon[18556]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:10:44.702 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:44.702 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:10:44.702 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-mon[18556]: mgrmap e8: x(active, since 2s) 2024-08-28T19:10:45.001 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:10:44.797+0000 7f31fd6db640 -1 log_channel(cephadm) log [ERR] : [28/Aug/2024:19:10:44] ENGINE Error in HTTPServer.serve 2024-08-28T19:10:45.001 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: Traceback (most recent call last): 2024-08-28T19:10:45.001 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-28T19:10:45.001 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self._connections.run(self.expiration_interval) 2024-08-28T19:10:45.001 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-28T19:10:45.001 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self._run(expiration_interval) 2024-08-28T19:10:45.001 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: new_conn = self._from_server_socket(self.server.socket) 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: s = self.context.wrap_socket( 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: return self.sslsocket_class._create( 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self.do_handshake() 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self._sslobj.do_handshake() 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-28T19:10:45.002 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:10:45.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout value unchanged 2024-08-28T19:10:45.196 INFO:teuthology.orchestra.run.smithi016.stdout:Generating ssh key... 2024-08-28T19:10:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-mon[18556]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:10:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-mon[18556]: [28/Aug/2024:19:10:44] ENGINE Bus STARTING 2024-08-28T19:10:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-mon[18556]: [28/Aug/2024:19:10:44] ENGINE Serving on https://172.21.15.16:7150 2024-08-28T19:10:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-mon[18556]: [28/Aug/2024:19:10:44] ENGINE Error in HTTPServer.serve 2024-08-28T19:10:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-08-28T19:10:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-28T19:10:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-08-28T19:10:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-28T19:10:45.914 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-08-28T19:10:45.915 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-28T19:10:45.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-mon[18556]: [28/Aug/2024:19:10:44] ENGINE Serving on http://172.21.15.16:8765 2024-08-28T19:10:45.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-mon[18556]: [28/Aug/2024:19:10:44] ENGINE Bus STARTED 2024-08-28T19:10:45.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:10:46.226 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: Generating public/private rsa key pair. 2024-08-28T19:10:46.226 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: Your identification has been saved in /tmp/tmpn157w_ff/key 2024-08-28T19:10:46.226 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: Your public key has been saved in /tmp/tmpn157w_ff/key.pub 2024-08-28T19:10:46.226 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: The key fingerprint is: 2024-08-28T19:10:46.226 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: SHA256:gg1355iQQ+R5lAkTW+clqcBaIDIuvaL+PyeZUrTBPtU ceph-eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:10:46.226 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: The key's randomart image is: 2024-08-28T19:10:46.226 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: +---[RSA 3072]----+ 2024-08-28T19:10:46.226 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: |o . .+*oooo.. | 2024-08-28T19:10:46.226 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: |.+ . o+Boo.o | 2024-08-28T19:10:46.226 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: |... ooO.+.o | 2024-08-28T19:10:46.227 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: |. ..O *.E | 2024-08-28T19:10:46.227 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: |. . + * S . | 2024-08-28T19:10:46.227 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: |.. = . | 2024-08-28T19:10:46.227 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: |. . + | 2024-08-28T19:10:46.227 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: |. . = . | 2024-08-28T19:10:46.227 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: | ...o.+ | 2024-08-28T19:10:46.227 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:10:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: +----[SHA256]-----+ 2024-08-28T19:10:47.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:46 smithi016 ceph-mon[18556]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:10:47.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:46 smithi016 ceph-mon[18556]: Generating ssh key... 2024-08-28T19:10:47.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:46 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:47.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:46 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:47.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChWQqEQchcg9jxc+eh5tZ9LLiz1zqttDYZAoV3d+hbgv7I3pGFGS0JUozhHK299btvTPgSr/MJsG9zCFCrlKAEovNPtqEywkvfNo6TG92GqBDWVJoo4IvNrVHI41zfGeqnu8rtgM5R8PcR9HcJI+2ESte72SkaUqe+5/49qBKq5IZCePcfUi0BCkr5QmrhMWmhwCCB1yidmfCE7YHMw4GXhSp59//cxlCU6e9At8TRoX9QbstetYqHGboPvZeEpkcGRSnGLwuCKJhRHckKBOJyRI/OfQOa4Uj22o6s6r060qa9eWRv/VYqN+62908k2k87JG0sHz4ews05hCcw/3gZ3DVUIYG3OWn9hhKWCOJl9+r+PYOpixw3s4vb0DNoHfQGoSRk/ysxKW9EEPW+sumr1Ic6I9ozEKH/AY5QcypnGB5xqc5gCScq0/qnSzAYzLIY8X92AklgP5aEWF4kOcCkWPaw9h+fy4Y/ushHjDgwINS7Cplpp57+nFz8jKBLMCk= ceph-eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:10:47.594 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-08-28T19:10:47.594 INFO:teuthology.orchestra.run.smithi016.stdout:Adding key to root@localhost authorized_keys... 2024-08-28T19:10:47.594 INFO:teuthology.orchestra.run.smithi016.stdout:Adding host smithi016... 2024-08-28T19:10:47.985 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:47 smithi016 ceph-mon[18556]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:10:49.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:48 smithi016 ceph-mon[18556]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:10:50.466 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:50 smithi016 ceph-mon[18556]: Deploying cephadm binary to smithi016 2024-08-28T19:10:52.694 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Added host 'smithi016' with addr '172.21.15.16' 2024-08-28T19:10:52.694 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mon service... 2024-08-28T19:10:53.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:53 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:53.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:53 smithi016 ceph-mon[18556]: Added host smithi016 2024-08-28T19:10:53.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:53 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:10:54.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-08-28T19:10:54.190 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mgr service... 2024-08-28T19:10:54.481 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:54 smithi016 ceph-mon[18556]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:10:54.482 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:54 smithi016 ceph-mon[18556]: Saving service mon spec with placement count:5 2024-08-28T19:10:54.482 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:54 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:55.330 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-08-28T19:10:56.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:55 smithi016 ceph-mon[18556]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:10:56.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:55 smithi016 ceph-mon[18556]: Saving service mgr spec with placement count:2 2024-08-28T19:10:56.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:55 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:56.820 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:56 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:56.820 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:56 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:10:56.820 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:56 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2003014843' entity='client.admin' 2024-08-28T19:10:58.550 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling the dashboard module... 2024-08-28T19:10:58.550 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:58 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3997193282' entity='client.admin' 2024-08-28T19:10:58.550 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:58 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:11:00.174 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:59 smithi016 ceph-mon[18556]: from='mgr.14118 172.21.15.16:0/3928269225' entity='mgr.x' 2024-08-28T19:11:00.175 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:10:59 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1274746785' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-08-28T19:11:00.175 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:00 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: ignoring --setuser ceph since I am not root 2024-08-28T19:11:00.175 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:00 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: ignoring --setgroup ceph since I am not root 2024-08-28T19:11:01.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:01 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1274746785' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-08-28T19:11:01.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:01 smithi016 ceph-mon[18556]: mgrmap e9: x(active, since 18s) 2024-08-28T19:11:01.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-08-28T19:11:01.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-08-28T19:11:01.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-28T19:11:01.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-08-28T19:11:01.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-28T19:11:01.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-08-28T19:11:01.988 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-08-28T19:11:01.989 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 9... 2024-08-28T19:11:02.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:02 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1082146104' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-08-28T19:11:05.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:05 smithi016 ceph-mon[18556]: Active manager daemon x restarted 2024-08-28T19:11:05.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:05 smithi016 ceph-mon[18556]: Activating manager daemon x 2024-08-28T19:11:05.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:05 smithi016 ceph-mon[18556]: osdmap e3: 0 total, 0 up, 0 in 2024-08-28T19:11:05.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:05 smithi016 ceph-mon[18556]: mgrmap e10: x(active, starting, since 0.00394722s) 2024-08-28T19:11:05.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:05 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:11:05.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:05 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-28T19:11:05.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:05 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-08-28T19:11:05.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:05 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-08-28T19:11:05.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:05 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-08-28T19:11:05.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:05 smithi016 ceph-mon[18556]: Manager daemon x is now available 2024-08-28T19:11:06.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:06 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:11:06.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:06 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-08-28T19:11:06.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:06 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-08-28T19:11:06.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:06 smithi016 ceph-mon[18556]: mgrmap e11: x(active, since 1.00682s) 2024-08-28T19:11:06.814 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-08-28T19:11:06.814 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-08-28T19:11:06.814 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-28T19:11:06.814 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-08-28T19:11:06.814 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 9 is available 2024-08-28T19:11:06.815 INFO:teuthology.orchestra.run.smithi016.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-08-28T19:11:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-mon[18556]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-28T19:11:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-mon[18556]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-28T19:11:07.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:11:07.231+0000 7f9ba5e9d640 -1 log_channel(cephadm) log [ERR] : [28/Aug/2024:19:11:07] ENGINE Error in HTTPServer.serve 2024-08-28T19:11:07.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: Traceback (most recent call last): 2024-08-28T19:11:07.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-28T19:11:07.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self._connections.run(self.expiration_interval) 2024-08-28T19:11:07.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-28T19:11:07.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self._run(expiration_interval) 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: new_conn = self._from_server_socket(self.server.socket) 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: s = self.context.wrap_socket( 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: return self.sslsocket_class._create( 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self.do_handshake() 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self._sslobj.do_handshake() 2024-08-28T19:11:07.502 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-28T19:11:07.503 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:07 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:11:08.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:08 smithi016 ceph-mon[18556]: [28/Aug/2024:19:11:07] ENGINE Bus STARTING 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:08 smithi016 ceph-mon[18556]: [28/Aug/2024:19:11:07] ENGINE Serving on https://172.21.15.16:7150 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:08 smithi016 ceph-mon[18556]: [28/Aug/2024:19:11:07] ENGINE Error in HTTPServer.serve 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-28T19:11:08.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:08 smithi016 ceph-mon[18556]: [28/Aug/2024:19:11:07] ENGINE Serving on http://172.21.15.16:8765 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:08 smithi016 ceph-mon[18556]: [28/Aug/2024:19:11:07] ENGINE Bus STARTED 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:08 smithi016 ceph-mon[18556]: mgrmap e12: x(active, since 2s) 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:08 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:08.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:08 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:09.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:09 smithi016 ceph-mon[18556]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:11:10.122 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE+zCCAuOgAwIBAgIUcBK6tu2rn/V7DtdTJtGbTbacmKIwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDgyODE5MTEwOVoXDTM0\nMDgyOTE5MTEwOVowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjE2MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEA3yh7oKB8agOxBqWgaYuQ1WkCNc9guWzs2FR/\ns51n7N1NYms5BG8TWuOsU0Yb6uG2TdM+BL6zqU+P8FVakfyhe5+0lgc5cB+EppAx\nze6VbMdwZw4dPqxzc2/+lSwTPZI3uljbvxUsyIldW6SIGlCt9lmRMaL0DDyLQLwA\nPyKxmo61rZ1aUcnIFb+fqNejaMHy6SO0EYG2F3gWDnspP5EcGgDyDyNNxpUxdv2C\ncmxpk1h+a+XkOxcYqUNMq7NnAzN339/TeNuQAblNkat82Tcs/rqJ3YbyPXywU9xj\n9/dO/fmX9h7GIVIywBRIATw2G/IWhAOrhpT7cVsj3xjY99Yv0v6IE+QrmenofK/8\n1joPPpqTuV8dhjpNkGXmJD5j5y5PHrxIcGdEorgaTkQOscT37kDWoCkIjZ+SF9HR\nCZ+8gfDOrX9cVL4DdvCeB6VEG2Q+Ddamsu/IytFBOnr4BySgjQhyV1/rLKFrBS7K\nZqi+sdNFn3HjFcN9oDR619nNSyDNe8QaZZKiDofehxACRZAPmlUchbdLYxsw1hdS\nmckCrcPcGUYl1EyVs+qGa/5glm3KZbmUdaVB2Mcy+X79ArwdRv5ywWI/SKoHNv6H\naJdr8s84Taje4ycrFjOBx872X8KJj4m+wqrXE1skNk8EaPGC7u3F1Axe0PihVMZh\nio5/eYkCAwEAAaM/MD0wLQYDVR0RBCYwJIIJc21pdGhpMDE2ghFkYXNoYm9hcmRf\nc2VydmVyc4cErBUPEDAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQBq\n7q9bOiE+Ys372OYzwOih4Lcyun2QdXXrfjzDYbDsir0VY+fdQufKxRHOVj29yvQ/\nURHB1kCUZDluFbbAAMS1ENnujXqi35v1CVGpEGgVpfxfxrKXtZSp1zkTS2bZNJUL\nbSgg3nMURkDdtHqeZRcJuielb0xUAt4ghgS7LtP5xykgHJDu/z5xaIN5bFHha1U2\n9hRAJ+U2Dh0Q8tQTbz5u1R1EPcnzHES6qbaceWd3T7fFIz7ywH5Gv8fCbjxrCQUS\nMHkCeDuJR2Ekfs+PTRb6FicbIm2v8zH5m2nBYZkcui9txwKHNLAcVyOFb4X1Huoh\nZR87nGeLfPYoWUy92kkmBDC9QD4SVwCbxI9UAbYQfOZtdLzsGHKY18+9IqIgEQHC\nfi6hO0zTJZUb//q7lENnqdL0eZ6OdIfQBIb47CdZBxfN6G8bv/Sl45neV+K1+qoY\nH5Uq6w1mNMot3avUuirbWB+MYNEK/XehnelL85sNjCkYPE7Tgd6FRFGC7/mSJdZD\n/w7F/BZTt4jttbZEzZdRTUIK/AvtSac2boDo9qo4t9VqEoc2TMr8lcgPfYppAo70\n2OufFuUl/KYrtzV80MICk+pS27fxAGLAVBDaFTk+kjazXyjDt1R3mCL2KU8cML46\nT+ie3NnG1LMuQEvkR3S16IVgfAgL+u/PSYOIpOG5fQ==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKAIBAAKCAgEA3yh7oKB8agOxBqWgaYuQ1WkCNc9guWzs2FR/s51n7N1NYms5\nBG8TWuOsU0Yb6uG2TdM+BL6zqU+P8FVakfyhe5+0lgc5cB+EppAxze6VbMdwZw4d\nPqxzc2/+lSwTPZI3uljbvxUsyIldW6SIGlCt9lmRMaL0DDyLQLwAPyKxmo61rZ1a\nUcnIFb+fqNejaMHy6SO0EYG2F3gWDnspP5EcGgDyDyNNxpUxdv2Ccmxpk1h+a+Xk\nOxcYqUNMq7NnAzN339/TeNuQAblNkat82Tcs/rqJ3YbyPXywU9xj9/dO/fmX9h7G\nIVIywBRIATw2G/IWhAOrhpT7cVsj3xjY99Yv0v6IE+QrmenofK/81joPPpqTuV8d\nhjpNkGXmJD5j5y5PHrxIcGdEorgaTkQOscT37kDWoCkIjZ+SF9HRCZ+8gfDOrX9c\nVL4DdvCeB6VEG2Q+Ddamsu/IytFBOnr4BySgjQhyV1/rLKFrBS7KZqi+sdNFn3Hj\nFcN9oDR619nNSyDNe8QaZZKiDofehxACRZAPmlUchbdLYxsw1hdSmckCrcPcGUYl\n1EyVs+qGa/5glm3KZbmUdaVB2Mcy+X79ArwdRv5ywWI/SKoHNv6HaJdr8s84Taje\n4ycrFjOBx872X8KJj4m+wqrXE1skNk8EaPGC7u3F1Axe0PihVMZhio5/eYkCAwEA\nAQKCAgAZQC/N9v7g8ZI9Q+485x6QMVm/TDv/+0Xo/BW3lZzp8cgU9fg9itKENy0b\nfpw/1YKu93NF7uyuIKszOPs+t9jvX7y7ua1LhTt5q+kAyel1RbNvq8qOGg4VvrNO\nPo6eSc5duvqBeoRG3Dk4uosufBjVHmEmHk/SVyL0JBm8iyDQ0etXje2ngccuw9At\ndxn9y21gqUoxrW1U5kELqi66i0FEZX0+ToK5UM4oLksNhrwjALFMZPbQY16GX4+8\nQXhrZa5foghgHWsD8Otoq2B9BSflmmapbwC3yaU7lf1nVF/nL8/8uNJ3OIF218ym\nfCFMw/W9ghtNZwv1EYijcMCzOXSPa6YybxStPRVpjeHMIQygzkXTuxooC/sLV7UA\nvWcuvLeRyYNiknPlCTRvxMUXIadyeQ6cfUMLsjXuOEja67f5/1eWoKSLm47Ksanu\nca6KgA/BatRh1GgKym0r1Ab29xit+B14WWgLDsLR/pgia9Kw4aD18HceC7jml9a7\nOFLnLFPcDaeWY6d6jplhyGkkK+evsaht84TyVSevrBb/7NdqJg55YWEZh5xNSKYI\nKUBbExQ2uWAVUUjdY61jDskNF8fvVK7pTvg4CAjmUU2Xj7Zi+tdBFZU6r+PIUGb7\ne29EpDZdd7/R94GUoRTdBx9vQtTwb0CMNK6UFEtjBQDiOf6l4QKCAQEA8oBZIE4b\nB/NQIPqUmshOQCsw1xlUHy5cb9tRBlqP+Jmk6y5FHA/otkEoOOKwiYeJVMojmZAS\nZvaKd/oQSOrnmGt8egXRH3nhYYUgziopngDVBB8WpIULp4+hBWZMXLkR85nBaxzF\nmGcenDzy5lSuE8A03PInNaxIpq24DCEa0QMZarqS+8JRbYLr6AgvKgQ2Bbk8uGFM\npmE8en8b7N5sMdRz9dBhvAJTiO2a4HjiK8VRiXmXq83oOC0y0pDNDHrLveTmQSMh\nsIcW3R0GJ+MrAgw8DI2Qd0JgjcUwSimwwSm0kfiuerzwxOfHGP0Wc8xiGMd5NQMe\nIkWRHBJRmYCV4QKCAQEA65R+ROhoCrfjkutQ99IZ0ksWEGlFJyL40RZ7d8z97EH4\nbBt7eMIs40AAq8wwXPYSIVfTAu+WQ6v5U9AoUPmmNbCwf0jB2YeliRGjynZtFRUL\nMvJ3ij2viMSQjNTCETFv3D7YN2Bev4XSqj9LZXpSyrLduihruOX6us+IztIOCETj\nd+q15dWAAWeycr2W9cw7IHSkN4DZXvZdgU5wrrQiYY2HBHbiiqTOEkOIgyBfiiFK\nNhetAXFTyRmgZngmWNMY218MlNshbe+Z5zTayzDzXChC07FOOs/9QrfcsK5Rs9fA\naq6nGdsEIPmIKplRiokJeYHiTQCuha6yOP/nvp2IqQKCAQAf7eBOTZpniJTTUssA\nOkF7V3R9zdrypBMNs03OzVvJeslpIdV9pWaZoKigHNDrusnNPTREwuB5aBrBQ06j\nmzJVMknYcy2evFh9k57HJmwGXsutWD6N+OqY7GNiJ58EUmXXsjXyWX5225T/M1M0\nGpA1D4imUsfw3DdQouzpw+4Y+RR+pgQ2W9aG2ccf7Pvwx08pn8Mr77u86G6jHltZ\n68tJIFUIV6CWtaO0Erzis2fsJXXf6P+cYmciBml7qg8lhoGPBjV+/BYyfQb4pBqs\nY8/E3EGH0Zfk6/t0jzMVgLnCJTCeS6UFih0VgAMJ+kDbjFyaMKF35mowTwKVNKBD\n2tdhAoIBAFDTKKNRrWiIAM9GZwgvkGZjFpXY2D3+NKnf8cp/jVsCo03L/AfJbwck\nzVYj/y/VyRChhicKtl5PYYKM91gY3xqqnVN+TevpKzUk1TJAgVprVuVK5RKL6W6e\nKg33bQHjGULSWPJe4QMK0eVRQ/f0CPW8enBFL/hHB0gSrhhvXxsdL4HUCEZrgrh4\nsHPHckSwplSTUEgur+Vx11Dm7agy61zW5sz0UPNKad1ASFTcxYqps0FtzBCjpiBn\n521gqZ9uBag2KyXjLMwAGaiGrM5r5ALpWUwyF+n7tCOw20nO9P8j9cNkWrZ3Oypp\nEnuzEJPcQmDq3A6ZPkdlUYZW3la6y8kCggEBANyPWCejoxL7ISH/GR8oME2owuv7\nHdax0b6b42qV4YXsx9DDKCy8c7W0BzF9NQfjUZMrk2QEyNcjY17K8D7Fl/1Pgi76\noBlExm7kmDUMPK8KgPku2Lj1OIloZNdinskLH0wlkU8dVt7RpK68gNLLPqQGjjQJ\n/m565HpFBrrDq7MV7s+eBkTudDNh1Nab63KaW1nhFZDXkv9LBACd0bltrDyvJOa9\nDQO2ByWvWy6KVD2SqNnqC37yRQ0LeRYJ+PQlOdoLELMoTRcahOAYWq3/KeUldS/q\nSAdPt4vMwQmm/eMpDP1oYtUGX1P5mpxaRqZPe6bJ6aaEo7Y74jS7OSTCoRM=\n-----END RSA PRIVATE KEY-----\n"} 2024-08-28T19:11:11.245 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-08-28T19:11:11.940 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:11 smithi016 ceph-mon[18556]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:11:11.940 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:11 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:12.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-08-28T19:11:12.410 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial admin user... 2024-08-28T19:11:13.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:12 smithi016 ceph-mon[18556]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:11:13.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:12 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:13.835 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$bd3H3M2OKqGPXRCo6S7OueAwlpz0F0XtmHy7dXyAMHRbO9wel5.7y", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1724872273, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-08-28T19:11:13.835 INFO:teuthology.orchestra.run.smithi016.stdout:Fetching dashboard port number... 2024-08-28T19:11:14.134 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:13 smithi016 ceph-mon[18556]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:11:14.134 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:13 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:14.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 8443 2024-08-28T19:11:14.959 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-28T19:11:14.959 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-08-28T19:11:14.967 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-28T19:11:14.967 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-08-28T19:11:14.967 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-08-28T19:11:14.967 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-08-28T19:11:14.969 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph Dashboard is now available at: 2024-08-28T19:11:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: URL: https://smithi016:8443/ 2024-08-28T19:11:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: User: admin 2024-08-28T19:11:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: Password: byf76ju19e 2024-08-28T19:11:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:14.970 INFO:teuthology.orchestra.run.smithi016.stdout:Saving cluster configuration to /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config directory 2024-08-28T19:11:15.249 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:14 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/267282122' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-08-28T19:11:16.122 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout:Or, if you are only running a single cluster on this host: 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout:Please consider enabling telemetry to help improve Ceph: 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: ceph telemetry on 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout:For more information see: 2024-08-28T19:11:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:16.124 INFO:teuthology.orchestra.run.smithi016.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-08-28T19:11:16.124 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:16.124 INFO:teuthology.orchestra.run.smithi016.stdout:Bootstrap complete. 2024-08-28T19:11:16.169 INFO:tasks.cephadm:Fetching config... 2024-08-28T19:11:16.169 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:11:16.169 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-08-28T19:11:16.185 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-08-28T19:11:16.185 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:11:16.185 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-08-28T19:11:16.240 INFO:tasks.cephadm:Fetching mon keyring... 2024-08-28T19:11:16.240 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:11:16.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/keyring of=/dev/stdout 2024-08-28T19:11:16.305 INFO:tasks.cephadm:Fetching pub ssh key... 2024-08-28T19:11:16.306 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:11:16.306 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-08-28T19:11:16.360 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-08-28T19:11:16.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChWQqEQchcg9jxc+eh5tZ9LLiz1zqttDYZAoV3d+hbgv7I3pGFGS0JUozhHK299btvTPgSr/MJsG9zCFCrlKAEovNPtqEywkvfNo6TG92GqBDWVJoo4IvNrVHI41zfGeqnu8rtgM5R8PcR9HcJI+2ESte72SkaUqe+5/49qBKq5IZCePcfUi0BCkr5QmrhMWmhwCCB1yidmfCE7YHMw4GXhSp59//cxlCU6e9At8TRoX9QbstetYqHGboPvZeEpkcGRSnGLwuCKJhRHckKBOJyRI/OfQOa4Uj22o6s6r060qa9eWRv/VYqN+62908k2k87JG0sHz4ews05hCcw/3gZ3DVUIYG3OWn9hhKWCOJl9+r+PYOpixw3s4vb0DNoHfQGoSRk/ysxKW9EEPW+sumr1Ic6I9ozEKH/AY5QcypnGB5xqc5gCScq0/qnSzAYzLIY8X92AklgP5aEWF4kOcCkWPaw9h+fy4Y/ushHjDgwINS7Cplpp57+nFz8jKBLMCk= ceph-eabc438a-6570-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-28T19:11:16.434 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChWQqEQchcg9jxc+eh5tZ9LLiz1zqttDYZAoV3d+hbgv7I3pGFGS0JUozhHK299btvTPgSr/MJsG9zCFCrlKAEovNPtqEywkvfNo6TG92GqBDWVJoo4IvNrVHI41zfGeqnu8rtgM5R8PcR9HcJI+2ESte72SkaUqe+5/49qBKq5IZCePcfUi0BCkr5QmrhMWmhwCCB1yidmfCE7YHMw4GXhSp59//cxlCU6e9At8TRoX9QbstetYqHGboPvZeEpkcGRSnGLwuCKJhRHckKBOJyRI/OfQOa4Uj22o6s6r060qa9eWRv/VYqN+62908k2k87JG0sHz4ews05hCcw/3gZ3DVUIYG3OWn9hhKWCOJl9+r+PYOpixw3s4vb0DNoHfQGoSRk/ysxKW9EEPW+sumr1Ic6I9ozEKH/AY5QcypnGB5xqc5gCScq0/qnSzAYzLIY8X92AklgP5aEWF4kOcCkWPaw9h+fy4Y/ushHjDgwINS7Cplpp57+nFz8jKBLMCk= ceph-eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:16.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChWQqEQchcg9jxc+eh5tZ9LLiz1zqttDYZAoV3d+hbgv7I3pGFGS0JUozhHK299btvTPgSr/MJsG9zCFCrlKAEovNPtqEywkvfNo6TG92GqBDWVJoo4IvNrVHI41zfGeqnu8rtgM5R8PcR9HcJI+2ESte72SkaUqe+5/49qBKq5IZCePcfUi0BCkr5QmrhMWmhwCCB1yidmfCE7YHMw4GXhSp59//cxlCU6e9At8TRoX9QbstetYqHGboPvZeEpkcGRSnGLwuCKJhRHckKBOJyRI/OfQOa4Uj22o6s6r060qa9eWRv/VYqN+62908k2k87JG0sHz4ews05hCcw/3gZ3DVUIYG3OWn9hhKWCOJl9+r+PYOpixw3s4vb0DNoHfQGoSRk/ysxKW9EEPW+sumr1Ic6I9ozEKH/AY5QcypnGB5xqc5gCScq0/qnSzAYzLIY8X92AklgP5aEWF4kOcCkWPaw9h+fy4Y/ushHjDgwINS7Cplpp57+nFz8jKBLMCk= ceph-eabc438a-6570-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-28T19:11:16.494 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChWQqEQchcg9jxc+eh5tZ9LLiz1zqttDYZAoV3d+hbgv7I3pGFGS0JUozhHK299btvTPgSr/MJsG9zCFCrlKAEovNPtqEywkvfNo6TG92GqBDWVJoo4IvNrVHI41zfGeqnu8rtgM5R8PcR9HcJI+2ESte72SkaUqe+5/49qBKq5IZCePcfUi0BCkr5QmrhMWmhwCCB1yidmfCE7YHMw4GXhSp59//cxlCU6e9At8TRoX9QbstetYqHGboPvZeEpkcGRSnGLwuCKJhRHckKBOJyRI/OfQOa4Uj22o6s6r060qa9eWRv/VYqN+62908k2k87JG0sHz4ews05hCcw/3gZ3DVUIYG3OWn9hhKWCOJl9+r+PYOpixw3s4vb0DNoHfQGoSRk/ysxKW9EEPW+sumr1Ic6I9ozEKH/AY5QcypnGB5xqc5gCScq0/qnSzAYzLIY8X92AklgP5aEWF4kOcCkWPaw9h+fy4Y/ushHjDgwINS7Cplpp57+nFz8jKBLMCk= ceph-eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:16.506 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChWQqEQchcg9jxc+eh5tZ9LLiz1zqttDYZAoV3d+hbgv7I3pGFGS0JUozhHK299btvTPgSr/MJsG9zCFCrlKAEovNPtqEywkvfNo6TG92GqBDWVJoo4IvNrVHI41zfGeqnu8rtgM5R8PcR9HcJI+2ESte72SkaUqe+5/49qBKq5IZCePcfUi0BCkr5QmrhMWmhwCCB1yidmfCE7YHMw4GXhSp59//cxlCU6e9At8TRoX9QbstetYqHGboPvZeEpkcGRSnGLwuCKJhRHckKBOJyRI/OfQOa4Uj22o6s6r060qa9eWRv/VYqN+62908k2k87JG0sHz4ews05hCcw/3gZ3DVUIYG3OWn9hhKWCOJl9+r+PYOpixw3s4vb0DNoHfQGoSRk/ysxKW9EEPW+sumr1Ic6I9ozEKH/AY5QcypnGB5xqc5gCScq0/qnSzAYzLIY8X92AklgP5aEWF4kOcCkWPaw9h+fy4Y/ushHjDgwINS7Cplpp57+nFz8jKBLMCk= ceph-eabc438a-6570-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-28T19:11:16.547 INFO:teuthology.orchestra.run.smithi115.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChWQqEQchcg9jxc+eh5tZ9LLiz1zqttDYZAoV3d+hbgv7I3pGFGS0JUozhHK299btvTPgSr/MJsG9zCFCrlKAEovNPtqEywkvfNo6TG92GqBDWVJoo4IvNrVHI41zfGeqnu8rtgM5R8PcR9HcJI+2ESte72SkaUqe+5/49qBKq5IZCePcfUi0BCkr5QmrhMWmhwCCB1yidmfCE7YHMw4GXhSp59//cxlCU6e9At8TRoX9QbstetYqHGboPvZeEpkcGRSnGLwuCKJhRHckKBOJyRI/OfQOa4Uj22o6s6r060qa9eWRv/VYqN+62908k2k87JG0sHz4ews05hCcw/3gZ3DVUIYG3OWn9hhKWCOJl9+r+PYOpixw3s4vb0DNoHfQGoSRk/ysxKW9EEPW+sumr1Ic6I9ozEKH/AY5QcypnGB5xqc5gCScq0/qnSzAYzLIY8X92AklgP5aEWF4kOcCkWPaw9h+fy4Y/ushHjDgwINS7Cplpp57+nFz8jKBLMCk= ceph-eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:16.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-08-28T19:11:16.818 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:11:16.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:16 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1201768473' entity='client.admin' 2024-08-28T19:11:18.118 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-08-28T19:11:18.120 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-08-28T19:11:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:11:18.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:18 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2083006369' entity='client.admin' 2024-08-28T19:11:18.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:18 smithi016 ceph-mon[18556]: mgrmap e13: x(active, since 12s) 2024-08-28T19:11:19.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:19 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:19.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:19 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:19.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:19 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"} : dispatch 2024-08-28T19:11:19.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:19 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:19.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:19 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:11:19.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:19 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:20.744 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi028 2024-08-28T19:11:20.745 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-08-28T19:11:20.745 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/etc/ceph/ceph.conf 2024-08-28T19:11:20.764 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-08-28T19:11:20.764 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:11:20.822 INFO:tasks.cephadm:Adding host smithi028 to orchestrator... 2024-08-28T19:11:20.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch host add smithi028 2024-08-28T19:11:21.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:20 smithi016 ceph-mon[18556]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:11:21.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:20 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:21.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:20 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:11:21.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:20 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:21.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:20 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:11:21.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:20 smithi016 ceph-mon[18556]: Updating smithi016:/etc/ceph/ceph.conf 2024-08-28T19:11:21.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:20 smithi016 ceph-mon[18556]: Updating smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:21.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:20 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:21.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:20 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:21.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:20 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:11:22.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:21 smithi016 ceph-mon[18556]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:11:22.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:21 smithi016 ceph-mon[18556]: Updating smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:11:23.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:22 smithi016 ceph-mon[18556]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:11:24.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:24 smithi016 ceph-mon[18556]: Deploying cephadm binary to smithi028 2024-08-28T19:11:25.930 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi028' with addr '172.21.15.28' 2024-08-28T19:11:26.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch host ls --format=json 2024-08-28T19:11:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:11:27.131 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:26 smithi016 ceph-mon[18556]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:27.131 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:26 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:27.131 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:26 smithi016 ceph-mon[18556]: Added host smithi028 2024-08-28T19:11:27.131 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:26 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:11:27.131 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:26 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:27.131 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:26 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:27.321 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}] 2024-08-28T19:11:27.905 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi115 2024-08-28T19:11:27.905 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-08-28T19:11:27.905 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.conf 2024-08-28T19:11:27.922 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-08-28T19:11:27.922 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:11:27.977 INFO:tasks.cephadm:Adding host smithi115 to orchestrator... 2024-08-28T19:11:27.977 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch host add smithi115 2024-08-28T19:11:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:11:28.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:28 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:28.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:28 smithi016 ceph-mon[18556]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:28.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:28 smithi016 ceph-mon[18556]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:11:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:29 smithi016 ceph-mon[18556]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi115", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:11:31.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:31.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: Deploying cephadm binary to smithi115 2024-08-28T19:11:31.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:31.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:31.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:31.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:31.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"} : dispatch 2024-08-28T19:11:31.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:31.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:11:31.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: Updating smithi028:/etc/ceph/ceph.conf 2024-08-28T19:11:31.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:31.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:31.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:31 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:32 smithi016 ceph-mon[18556]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:32 smithi016 ceph-mon[18556]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:11:32.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:32 smithi016 ceph-mon[18556]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:11:32.743 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi115' with addr '172.21.15.115' 2024-08-28T19:11:33.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:33 smithi016 ceph-mon[18556]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:33.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:33 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:33.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:33 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:11:33.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:33 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:33.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch host ls --format=json 2024-08-28T19:11:33.613 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:11:34.259 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:34 smithi016 ceph-mon[18556]: Added host smithi115 2024-08-28T19:11:34.259 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:34 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:34.260 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:11:34.260 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}] 2024-08-28T19:11:34.795 INFO:tasks.cephadm:Setting crush tunables to default 2024-08-28T19:11:34.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd crush tunables default 2024-08-28T19:11:35.041 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:11:35.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:35 smithi016 ceph-mon[18556]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:35.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:35 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:36.095 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-08-28T19:11:36.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:36 smithi016 ceph-mon[18556]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:11:36.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:36 smithi016 ceph-mon[18556]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:36.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:36 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/475863753' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-08-28T19:11:36.610 INFO:tasks.cephadm:Adding mon.a on smithi016 2024-08-28T19:11:36.610 INFO:tasks.cephadm:Adding mon.b on smithi028 2024-08-28T19:11:36.610 INFO:tasks.cephadm:Adding mon.c on smithi115 2024-08-28T19:11:36.611 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch apply mon '3;smithi016:172.21.15.16=a;smithi028:172.21.15.28=b;smithi115:172.21.15.115=c' 2024-08-28T19:11:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /etc/ceph/ceph.conf 2024-08-28T19:11:36.893 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /etc/ceph/ceph.conf 2024-08-28T19:11:37.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:37 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/475863753' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-28T19:11:37.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:37 smithi016 ceph-mon[18556]: osdmap e4: 0 total, 0 up, 0 in 2024-08-28T19:11:37.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:37 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:37 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:37 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:37 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:37 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"} : dispatch 2024-08-28T19:11:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:37 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:37 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:11:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:37 smithi016 ceph-mon[18556]: Updating smithi115:/etc/ceph/ceph.conf 2024-08-28T19:11:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:37 smithi016 ceph-mon[18556]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:37.577 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled mon update... 2024-08-28T19:11:38.227 DEBUG:teuthology.orchestra.run.smithi028:mon.b> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.b.service 2024-08-28T19:11:38.230 DEBUG:teuthology.orchestra.run.smithi115:mon.c> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.c.service 2024-08-28T19:11:38.232 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-28T19:11:38.232 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-08-28T19:11:39.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:11:39.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:39.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:11:39.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi028:172.21.15.28=b;smithi115:172.21.15.115=c", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:11:39.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi028:172.21.15.28=b;smithi115:172.21.15.115=c;count:3 2024-08-28T19:11:39.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:39.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:39.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:39.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:39.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:11:39.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:39.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:38 smithi016 ceph-mon[18556]: Deploying daemon mon.c on smithi115 2024-08-28T19:11:39.265 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:40.224 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:11:40.224 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","modified":"2024-08-28T19:10:17.308213Z","created":"2024-08-28T19:10:17.308213Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-08-28T19:11:40.224 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-08-28T19:11:41.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:40 smithi016 ceph-mon[18556]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:41.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:40 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/1014417415' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 podman[22165]: 2024-08-28 19:11:42.093633696 +0000 UTC m=+0.345223283 container start 703bcb8f9d936ede9320098d38098fe9d864a2437a5465cebaca12c164d19f92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-08-28T19:11:42.294 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-28T19:11:42.294 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-08-28T19:11:42.543 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:11:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: set uid:gid to 167:167 (ceph:ceph) 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: ceph version 19.3.0-4560-gd19e6118 (d19e6118f7015f0047264006cb0c873528eb3ae6) squid (dev), process ceph-mon, pid 2 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: pidfile_write: ignore empty --pid-file 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: load: jerasure load: lrc 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: RocksDB version: 7.9.2 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Git sha 0 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Compile date 2024-08-28 10:19:06 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: DB SUMMARY 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: DB Session ID: DP3S5DA7TA31XDSSAEQR 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: CURRENT file: CURRENT 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: IDENTITY file: IDENTITY 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-08-28T19:11:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.error_if_exists: 0 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.create_if_missing: 0 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.paranoid_checks: 1 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.env: 0x559ed05ab5a0 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.fs: PosixFileSystem 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.info_log: 0x559ed6b854e0 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_file_opening_threads: 16 2024-08-28T19:11:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.statistics: (nil) 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.use_fsync: 0 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_log_file_size: 0 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.keep_log_file_num: 1000 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.recycle_log_file_num: 0 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.allow_fallocate: 1 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.allow_mmap_reads: 0 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.allow_mmap_writes: 0 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.use_direct_reads: 0 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-28T19:11:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.create_missing_column_families: 0 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.db_log_dir: 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.wal_dir: 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.advise_random_on_open: 1 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.db_write_buffer_size: 0 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.write_buffer_manager: 0x559ed6b895e0 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-28T19:11:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.rate_limiter: (nil) 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.wal_recovery_mode: 2 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.enable_thread_tracking: 0 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.enable_pipelined_write: 0 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.unordered_write: 0 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.row_cache: None 2024-08-28T19:11:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.wal_filter: None 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.allow_ingest_behind: 0 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.two_write_queues: 0 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.manual_wal_flush: 0 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.wal_compression: 0 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.atomic_flush: 0 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.log_readahead_size: 0 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.best_efforts_recovery: 0 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-28T19:11:42.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.allow_data_in_errors: 0 2024-08-28T19:11:42.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.db_host_id: __hostname__ 2024-08-28T19:11:42.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-28T19:11:42.629 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_background_jobs: 2 2024-08-28T19:11:42.629 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_background_compactions: -1 2024-08-28T19:11:42.629 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_subcompactions: 1 2024-08-28T19:11:42.629 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_total_wal_size: 0 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_open_files: -1 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bytes_per_sync: 0 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_readahead_size: 0 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_background_flushes: -1 2024-08-28T19:11:42.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Compression algorithms supported: 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: kZSTD supported: 0 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: kXpressCompression supported: 0 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: kBZip2Compression supported: 0 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: kLZ4Compression supported: 1 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: kZlibCompression supported: 1 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: kLZ4HCCompression supported: 1 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: kSnappyCompression supported: 1 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-28T19:11:42.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.merge_operator: 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_filter: None 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_filter_factory: None 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.sst_partitioner_factory: None 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559ed6b85100) 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: cache_index_and_filter_blocks: 1 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: pin_top_level_index_and_filter: 1 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: index_type: 0 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: data_block_index_type: 0 2024-08-28T19:11:42.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: index_shortening: 1 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: checksum: 4 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: no_block_cache: 0 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache: 0x559ed2b48850 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache_name: BinnedLRUCache 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache_options: 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: capacity : 536870912 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: num_shard_bits : 4 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: strict_capacity_limit : 0 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: high_pri_pool_ratio: 0.000 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache_compressed: (nil) 2024-08-28T19:11:42.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: persistent_cache: (nil) 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_size: 4096 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_size_deviation: 10 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_restart_interval: 16 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: index_block_restart_interval: 1 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: metadata_block_size: 4096 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: partition_filters: 0 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: use_delta_encoding: 1 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: filter_policy: bloomfilter 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: whole_key_filtering: 1 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: verify_compression: 0 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: read_amp_bytes_per_bit: 0 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: format_version: 5 2024-08-28T19:11:42.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: enable_index_compression: 1 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_align: 0 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: max_auto_readahead_size: 262144 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: prepopulate_block_cache: 0 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: initial_auto_readahead_size: 8192 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: num_file_reads_for_auto_readahead: 2 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.write_buffer_size: 33554432 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_write_buffer_number: 2 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compression: NoCompression 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bottommost_compression: Disabled 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.prefix_extractor: nullptr 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.num_levels: 7 2024-08-28T19:11:42.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-28T19:11:42.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-28T19:11:42.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-28T19:11:42.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-28T19:11:42.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-28T19:11:42.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-28T19:11:42.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-28T19:11:42.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-28T19:11:42.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compression_opts.level: 32767 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compression_opts.strategy: 0 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compression_opts.enabled: false 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-28T19:11:42.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-28T19:11:42.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-28T19:11:42.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.target_file_size_base: 67108864 2024-08-28T19:11:42.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-28T19:11:42.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-28T19:11:42.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-28T19:11:42.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-28T19:11:42.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-28T19:11:42.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-28T19:11:42.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-28T19:11:42.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.arena_block_size: 1048576 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.disable_auto_compactions: 0 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-28T19:11:42.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.inplace_update_support: 0 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-28T19:11:42.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.bloom_locality: 0 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.max_successive_merges: 0 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.paranoid_file_checks: 0 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.force_consistency_checks: 1 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.report_bg_io_stats: 0 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.ttl: 2592000 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.enable_blob_files: false 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.min_blob_size: 0 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.blob_file_size: 268435456 2024-08-28T19:11:42.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.blob_file_starting_level: 0 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 396eaac7-c4c1-4379-9a7b-b18153d814cc 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724872302131967, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-08-28T19:11:42.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724872302132387, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1724872302, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "396eaac7-c4c1-4379-9a7b-b18153d814cc", "db_session_id": "DP3S5DA7TA31XDSSAEQR", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724872302132481, "job": 1, "event": "recovery_finished"} 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559ed6ba8a80 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: DB pointer 0x559ed794a000 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** DB Stats ** 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-28T19:11:42.643 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** Compaction Stats [default] ** 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** Compaction Stats [default] ** 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-28T19:11:42.644 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-28T19:11:42.645 INFO:journalctl@ceph.mon.c.smithi115.stdout: Block cache BinnedLRUCache@0x559ed2b48850#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: using public_addr v2:172.21.15.115:0/0 -> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: starting mon.c rank -1 at public addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] at bind addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(???) e0 preinit fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).mds e1 new map 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).mds e1 print_map 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout: e1 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout: btime 2024-08-28T19:10:20:239202+0000 2024-08-28T19:11:42.646 INFO:journalctl@ceph.mon.c.smithi115.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout: legacy client fscid: -1 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout: No filesystems configured 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-28T19:11:42.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: Added host smithi115 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:11:42.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/475863753' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/475863753' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: osdmap e4: 0 total, 0 up, 0 in 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"} : dispatch 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: Updating smithi115:/etc/ceph/ceph.conf 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:11:42.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi028:172.21.15.28=b;smithi115:172.21.15.115=c", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi028:172.21.15.28=b;smithi115:172.21.15.115=c;count:3 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: Deploying daemon mon.c on smithi115 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/1014417415' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 ceph-mon[22184]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 bash[22165]: 703bcb8f9d936ede9320098d38098fe9d864a2437a5465cebaca12c164d19f92 2024-08-28T19:11:42.650 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:42 smithi115 systemd[1]: Started Ceph mon.c for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:11:47.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:11:47.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: mon.a calling monitor election 2024-08-28T19:11:47.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:47.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: mon.c calling monitor election 2024-08-28T19:11:47.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: monmap epoch 2 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: last_changed 2024-08-28T19:11:42.159139+0000 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: created 2024-08-28T19:10:17.308213+0000 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: min_mon_release 19 (squid) 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: election_strategy: 1 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: 1: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.c 2024-08-28T19:11:47.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: fsmap 2024-08-28T19:11:47.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: osdmap e4: 0 total, 0 up, 0 in 2024-08-28T19:11:47.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: mgrmap e13: x(active, since 41s) 2024-08-28T19:11:47.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: overall HEALTH_OK 2024-08-28T19:11:47.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:47.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:47.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:47.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:11:47.503 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:47 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:11:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: mon.a calling monitor election 2024-08-28T19:11:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: mon.c calling monitor election 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: monmap epoch 2 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: last_changed 2024-08-28T19:11:42.159139+0000 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: created 2024-08-28T19:10:17.308213+0000 2024-08-28T19:11:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: min_mon_release 19 (squid) 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: election_strategy: 1 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: 1: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.c 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: fsmap 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: osdmap e4: 0 total, 0 up, 0 in 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: mgrmap e13: x(active, since 41s) 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: overall HEALTH_OK 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:11:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:47 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:47.631 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:11:47.631 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":2,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","modified":"2024-08-28T19:11:42.159139Z","created":"2024-08-28T19:10:17.308213Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-28T19:11:47.631 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 2 2024-08-28T19:11:48.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:48 smithi016 ceph-mon[18556]: Deploying daemon mon.b on smithi028 2024-08-28T19:11:48.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:48 smithi016 ceph-mon[18556]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:48.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:48 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/550269324' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:48.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:48 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:48.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:48 smithi115 ceph-mon[22184]: Deploying daemon mon.b on smithi028 2024-08-28T19:11:48.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:48 smithi115 ceph-mon[22184]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:48.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:48 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/550269324' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:48.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:48 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:49.242 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-28T19:11:49.242 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-08-28T19:11:49.496 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:11:50.194 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:11:50.194 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":2,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","modified":"2024-08-28T19:11:42.159139Z","created":"2024-08-28T19:10:17.308213Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-28T19:11:50.194 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 2 2024-08-28T19:11:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:50 smithi115 ceph-mon[22184]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:50.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:50 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/1281851604' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:50.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:50 smithi016 ceph-mon[18556]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:50.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:50 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/1281851604' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:51.247 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-08-28T19:11:51.247 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-08-28T19:11:51.247 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-08-28T19:11:51.247 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-08-28T19:11:51.247 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-08-28T19:11:51.247 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-08-28T19:11:51.247 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-28T19:11:51.247 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-28T19:11:51.247 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-28T19:11:51.247 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-28T19:11:51.247 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: Added host smithi115 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/475863753' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-08-28T19:11:51.248 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/475863753' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-28T19:11:51.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: osdmap e4: 0 total, 0 up, 0 in 2024-08-28T19:11:51.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"} : dispatch 2024-08-28T19:11:51.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: Updating smithi115:/etc/ceph/ceph.conf 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi028:172.21.15.28=b;smithi115:172.21.15.115=c", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi028:172.21.15.28=b;smithi115:172.21.15.115=c;count:3 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: Deploying daemon mon.c on smithi115 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/1014417415' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.a calling monitor election 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.c calling monitor election 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:51.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: monmap epoch 2 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: last_changed 2024-08-28T19:11:42.159139+0000 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: created 2024-08-28T19:10:17.308213+0000 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: min_mon_release 19 (squid) 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: election_strategy: 1 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: 1: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.c 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: fsmap 2024-08-28T19:11:51.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: osdmap e4: 0 total, 0 up, 0 in 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mgrmap e13: x(active, since 41s) 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: overall HEALTH_OK 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: Deploying daemon mon.b on smithi028 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/550269324' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/1281851604' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 ceph-mon[21769]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-08-28T19:11:51.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 bash[21749]: 684e32f5e71d61a34699519cc1f5717518e193d85d1148677366366feb218de5 2024-08-28T19:11:51.565 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:51 smithi028 systemd[1]: Started Ceph mon.b for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:11:51.722 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-28T19:11:51.723 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-08-28T19:11:51.963 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:11:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:11:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: mon.a calling monitor election 2024-08-28T19:11:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: mon.c calling monitor election 2024-08-28T19:11:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: mon.b calling monitor election 2024-08-28T19:11:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: monmap epoch 3 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: last_changed 2024-08-28T19:11:51.251133+0000 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: created 2024-08-28T19:10:17.308213+0000 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: min_mon_release 19 (squid) 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: election_strategy: 1 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: 1: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.c 2024-08-28T19:11:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: 2: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.b 2024-08-28T19:11:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: fsmap 2024-08-28T19:11:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: osdmap e4: 0 total, 0 up, 0 in 2024-08-28T19:11:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: mgrmap e13: x(active, since 50s) 2024-08-28T19:11:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: overall HEALTH_OK 2024-08-28T19:11:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:56 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:56.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:11:56.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: mon.a calling monitor election 2024-08-28T19:11:56.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: mon.c calling monitor election 2024-08-28T19:11:56.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:56.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:56.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: mon.b calling monitor election 2024-08-28T19:11:56.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: monmap epoch 3 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: last_changed 2024-08-28T19:11:51.251133+0000 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: created 2024-08-28T19:10:17.308213+0000 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: min_mon_release 19 (squid) 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: election_strategy: 1 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: 1: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.c 2024-08-28T19:11:56.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: 2: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.b 2024-08-28T19:11:56.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: fsmap 2024-08-28T19:11:56.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: osdmap e4: 0 total, 0 up, 0 in 2024-08-28T19:11:56.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: mgrmap e13: x(active, since 50s) 2024-08-28T19:11:56.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: overall HEALTH_OK 2024-08-28T19:11:56.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:56.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:56.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:56.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:56 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:11:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: mon.a calling monitor election 2024-08-28T19:11:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: mon.c calling monitor election 2024-08-28T19:11:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: mon.b calling monitor election 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: monmap epoch 3 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: last_changed 2024-08-28T19:11:51.251133+0000 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: created 2024-08-28T19:10:17.308213+0000 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: min_mon_release 19 (squid) 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: election_strategy: 1 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: 1: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.c 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: 2: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.b 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: fsmap 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: osdmap e4: 0 total, 0 up, 0 in 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: mgrmap e13: x(active, since 50s) 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: overall HEALTH_OK 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:56.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:56 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:57.038 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:11:57.039 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":3,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","modified":"2024-08-28T19:11:51.251133Z","created":"2024-08-28T19:10:17.308213Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-08-28T19:11:57.039 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 3 2024-08-28T19:11:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:57 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:11:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:57 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/96369700' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:57 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:57.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:57 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:11:57.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:57 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/96369700' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:57.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:57 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:57 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:57 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/96369700' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-28T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:57 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:11:58.053 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-08-28T19:11:58.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config generate-minimal-conf 2024-08-28T19:11:58.310 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:11:58.427 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:11:58.250+0000 7f9bd98c4640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-28T19:11:58.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:58.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:58.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:11:58.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: Updating smithi016:/etc/ceph/ceph.conf 2024-08-28T19:11:58.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: Updating smithi028:/etc/ceph/ceph.conf 2024-08-28T19:11:58.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: Updating smithi115:/etc/ceph/ceph.conf 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: Updating smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.696 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:11:58.696 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:11:58.696 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:58 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: Updating smithi016:/etc/ceph/ceph.conf 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: Updating smithi028:/etc/ceph/ceph.conf 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: Updating smithi115:/etc/ceph/ceph.conf 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:58.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: Updating smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:11:58.812 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:11:58.812 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:58 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:58.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:11:58.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:58.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: Updating smithi016:/etc/ceph/ceph.conf 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: Updating smithi028:/etc/ceph/ceph.conf 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: Updating smithi115:/etc/ceph/ceph.conf 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: Updating smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:11:58.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:11:58.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:11:58.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:58 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:58.984 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:58.984 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-08-28T19:11:58.984 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:11:58.984 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2024-08-28T19:11:59.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:59 smithi016 ceph-mon[18556]: Reconfiguring mon.a (unknown last config time)... 2024-08-28T19:11:59.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:59 smithi016 ceph-mon[18556]: Reconfiguring daemon mon.a on smithi016 2024-08-28T19:11:59.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:11:59 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2898393072' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:59.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:59 smithi028 ceph-mon[21769]: Reconfiguring mon.a (unknown last config time)... 2024-08-28T19:11:59.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:59 smithi028 ceph-mon[21769]: Reconfiguring daemon mon.a on smithi016 2024-08-28T19:11:59.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:11:59 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2898393072' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:11:59.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:59 smithi115 ceph-mon[22184]: Reconfiguring mon.a (unknown last config time)... 2024-08-28T19:11:59.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:59 smithi115 ceph-mon[22184]: Reconfiguring daemon mon.a on smithi016 2024-08-28T19:11:59.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:11:59 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2898393072' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:00.355 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-08-28T19:12:00.355 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:12:00.355 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-08-28T19:12:00.382 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:12:00.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:00.446 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-08-28T19:12:00.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-08-28T19:12:00.472 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-08-28T19:12:00.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:00.537 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-08-28T19:12:00.537 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.conf 2024-08-28T19:12:00.569 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-08-28T19:12:00.569 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:00.638 INFO:tasks.cephadm:enabling module snap_schedule 2024-08-28T19:12:00.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- sudo ceph --cluster ceph mgr module enable snap_schedule 2024-08-28T19:12:00.881 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:00 smithi016 ceph-mon[18556]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:00 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:00 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:00 smithi016 ceph-mon[18556]: Reconfiguring mon.b (monmap changed)... 2024-08-28T19:12:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:00 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:12:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:00 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:12:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:00 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:00 smithi016 ceph-mon[18556]: Reconfiguring daemon mon.b on smithi028 2024-08-28T19:12:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:00 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:00 smithi028 ceph-mon[21769]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:00 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:00 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:01.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:00 smithi028 ceph-mon[21769]: Reconfiguring mon.b (monmap changed)... 2024-08-28T19:12:01.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:00 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:12:01.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:00 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:12:01.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:00 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:01.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:00 smithi028 ceph-mon[21769]: Reconfiguring daemon mon.b on smithi028 2024-08-28T19:12:01.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:00 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:01.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:00 smithi115 ceph-mon[22184]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:01.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:00 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:01.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:00 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:01.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:00 smithi115 ceph-mon[22184]: Reconfiguring mon.b (monmap changed)... 2024-08-28T19:12:01.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:00 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:12:01.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:00 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:12:01.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:00 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:01.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:00 smithi115 ceph-mon[22184]: Reconfiguring daemon mon.b on smithi028 2024-08-28T19:12:01.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:00 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:02.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:02 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:02.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:02 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:02.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:02 smithi115 ceph-mon[22184]: Reconfiguring mon.c (monmap changed)... 2024-08-28T19:12:02.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:02 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:12:02.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:02 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:12:02.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:02 smithi115 ceph-mon[22184]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:02.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:02 smithi115 ceph-mon[22184]: Reconfiguring daemon mon.c on smithi115 2024-08-28T19:12:02.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:02 smithi115 ceph-mon[22184]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:02.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:02 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3590867912' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-08-28T19:12:02.500 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:02 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: ignoring --setuser ceph since I am not root 2024-08-28T19:12:02.500 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:02 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: ignoring --setgroup ceph since I am not root 2024-08-28T19:12:02.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:02 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:02.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:02 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:02.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:02 smithi016 ceph-mon[18556]: Reconfiguring mon.c (monmap changed)... 2024-08-28T19:12:02.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:02 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:12:02.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:02 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:12:02.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:02 smithi016 ceph-mon[18556]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:02.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:02 smithi016 ceph-mon[18556]: Reconfiguring daemon mon.c on smithi115 2024-08-28T19:12:02.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:02 smithi016 ceph-mon[18556]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:02.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:02 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3590867912' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-08-28T19:12:02.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:02 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:02.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:02 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' 2024-08-28T19:12:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:02 smithi028 ceph-mon[21769]: Reconfiguring mon.c (monmap changed)... 2024-08-28T19:12:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:02 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:12:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:02 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:12:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:02 smithi028 ceph-mon[21769]: from='mgr.14150 172.21.15.16:0/4205188515' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:02 smithi028 ceph-mon[21769]: Reconfiguring daemon mon.c on smithi115 2024-08-28T19:12:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:02 smithi028 ceph-mon[21769]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:02 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3590867912' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-08-28T19:12:02.892 INFO:tasks.cephadm:Adding mgr.x on smithi016 2024-08-28T19:12:02.892 INFO:tasks.cephadm:Adding mgr.y on smithi028 2024-08-28T19:12:02.892 INFO:tasks.cephadm:Adding mgr.z on smithi115 2024-08-28T19:12:02.892 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch apply mgr '3;smithi016=x;smithi028=y;smithi115=z' 2024-08-28T19:12:03.133 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:12:03.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:03 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3590867912' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-08-28T19:12:03.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:03 smithi115 ceph-mon[22184]: mgrmap e14: x(active, since 56s) 2024-08-28T19:12:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:03 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3590867912' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-08-28T19:12:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:03 smithi016 ceph-mon[18556]: mgrmap e14: x(active, since 56s) 2024-08-28T19:12:03.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:03 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3590867912' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-08-28T19:12:03.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:03 smithi028 ceph-mon[21769]: mgrmap e14: x(active, since 56s) 2024-08-28T19:12:07.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: Active manager daemon x restarted 2024-08-28T19:12:07.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: Activating manager daemon x 2024-08-28T19:12:07.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: osdmap e5: 0 total, 0 up, 0 in 2024-08-28T19:12:07.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: mgrmap e15: x(active, starting, since 0.0044983s) 2024-08-28T19:12:07.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:12:07.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:12:07.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:12:07.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-28T19:12:07.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-08-28T19:12:07.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-08-28T19:12:07.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-08-28T19:12:07.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:07 smithi016 ceph-mon[18556]: Manager daemon x is now available 2024-08-28T19:12:07.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: Active manager daemon x restarted 2024-08-28T19:12:07.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: Activating manager daemon x 2024-08-28T19:12:07.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: osdmap e5: 0 total, 0 up, 0 in 2024-08-28T19:12:07.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: mgrmap e15: x(active, starting, since 0.0044983s) 2024-08-28T19:12:07.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:12:07.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:12:07.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:12:07.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-28T19:12:07.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-08-28T19:12:07.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-08-28T19:12:07.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-08-28T19:12:07.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:07 smithi028 ceph-mon[21769]: Manager daemon x is now available 2024-08-28T19:12:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: Active manager daemon x restarted 2024-08-28T19:12:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: Activating manager daemon x 2024-08-28T19:12:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: osdmap e5: 0 total, 0 up, 0 in 2024-08-28T19:12:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: mgrmap e15: x(active, starting, since 0.0044983s) 2024-08-28T19:12:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:12:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:12:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:12:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-28T19:12:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-08-28T19:12:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-08-28T19:12:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-08-28T19:12:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:07 smithi115 ceph-mon[22184]: Manager daemon x is now available 2024-08-28T19:12:08.400 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled mgr update... 2024-08-28T19:12:08.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:08.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-08-28T19:12:08.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-08-28T19:12:08.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:08 smithi115 ceph-mon[22184]: mgrmap e16: x(active, since 1.00964s) 2024-08-28T19:12:08.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:08.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:08.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-08-28T19:12:08.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-08-28T19:12:08.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:08 smithi016 ceph-mon[18556]: mgrmap e16: x(active, since 1.00964s) 2024-08-28T19:12:08.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:08.795 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:08.795 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-08-28T19:12:08.795 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-08-28T19:12:08.795 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:08 smithi028 ceph-mon[21769]: mgrmap e16: x(active, since 1.00964s) 2024-08-28T19:12:08.795 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:08.995 DEBUG:teuthology.orchestra.run.smithi028:mgr.y> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.y.service 2024-08-28T19:12:08.998 DEBUG:teuthology.orchestra.run.smithi115:mgr.z> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.z.service 2024-08-28T19:12:09.001 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-08-28T19:12:09.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set client client_mount_timeout 600 2024-08-28T19:12:09.003 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-08-28T19:12:09.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set client debug_client 20 2024-08-28T19:12:09.046 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-08-28T19:12:09.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set client debug_ms 1 2024-08-28T19:12:09.089 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-08-28T19:12:09.089 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set client rados_mon_op_timeout 900 2024-08-28T19:12:09.132 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-08-28T19:12:09.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set client rados_osd_op_timeout 900 2024-08-28T19:12:09.175 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-08-28T19:12:09.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds debug_mds 20 2024-08-28T19:12:09.218 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-08-28T19:12:09.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds debug_mds_balancer 20 2024-08-28T19:12:09.263 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-08-28T19:12:09.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds debug_ms 2 2024-08-28T19:12:09.308 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-08-28T19:12:09.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-08-28T19:12:09.352 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-08-28T19:12:09.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds mds_bal_merge_size 5 2024-08-28T19:12:09.404 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-08-28T19:12:09.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds mds_bal_split_bits 3 2024-08-28T19:12:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:09.465 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-08-28T19:12:09.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds mds_bal_split_size 100 2024-08-28T19:12:09.511 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-08-28T19:12:09.512 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds mds_debug_frag True 2024-08-28T19:12:09.565 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-08-28T19:12:09.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds mds_debug_scatterstat True 2024-08-28T19:12:09.617 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-08-28T19:12:09.618 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds mds_op_complaint_time 180 2024-08-28T19:12:09.661 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-08-28T19:12:09.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds mds_verify_scatter True 2024-08-28T19:12:09.718 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-08-28T19:12:09.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds osd_op_complaint_time 180 2024-08-28T19:12:09.770 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-08-28T19:12:09.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds rados_mon_op_timeout 900 2024-08-28T19:12:09.815 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-08-28T19:12:09.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mds rados_osd_op_timeout 900 2024-08-28T19:12:09.860 INFO:tasks.cephadm:setting config [mgr] debug_cephsqlite = 20 2024-08-28T19:12:09.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mgr debug_cephsqlite 20 2024-08-28T19:12:09.895 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:12:09.646+0000 7f991599f640 -1 log_channel(cephadm) log [ERR] : [28/Aug/2024:19:12:09] ENGINE Error in HTTPServer.serve 2024-08-28T19:12:09.895 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: Traceback (most recent call last): 2024-08-28T19:12:09.895 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self._connections.run(self.expiration_interval) 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self._run(expiration_interval) 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: new_conn = self._from_server_socket(self.server.socket) 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: s = self.context.wrap_socket( 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: return self.sslsocket_class._create( 2024-08-28T19:12:09.896 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-28T19:12:09.897 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self.do_handshake() 2024-08-28T19:12:09.897 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-28T19:12:09.897 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: self._sslobj.do_handshake() 2024-08-28T19:12:09.897 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-28T19:12:09.897 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:12:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:12:09.909 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/allow_m_granularity = True 2024-08-28T19:12:09.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-08-28T19:12:09.961 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/dump_on_update = True 2024-08-28T19:12:09.961 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-08-28T19:12:10.012 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/log_level = debug 2024-08-28T19:12:10.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mgr mgr/snap_schedule/log_level debug 2024-08-28T19:12:10.066 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-08-28T19:12:10.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mon mon_op_complaint_time 120 2024-08-28T19:12:10.070 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:10.145 INFO:tasks.cephadm:setting config [mon] mon_warn_on_pool_no_app = False 2024-08-28T19:12:10.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set mon mon_warn_on_pool_no_app False 2024-08-28T19:12:10.151 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:10 smithi016 ceph-mon[18556]: Saving service mgr spec with placement smithi016=x;smithi028=y;smithi115=z;count:3 2024-08-28T19:12:10.151 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.151 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.151 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"} : dispatch 2024-08-28T19:12:10.155 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-08-28T19:12:10.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config set osd osd_op_complaint_time 180 2024-08-28T19:12:10.225 DEBUG:tasks.cephadm:set 26 configs 2024-08-28T19:12:10.225 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi016') 2024-08-28T19:12:10.364 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:10.374 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:10.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:10 smithi028 ceph-mon[21769]: Saving service mgr spec with placement smithi016=x;smithi028=y;smithi115=z;count:3 2024-08-28T19:12:10.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"} : dispatch 2024-08-28T19:12:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:10 smithi115 ceph-mon[22184]: Saving service mgr spec with placement smithi016=x;smithi028=y;smithi115=z;count:3 2024-08-28T19:12:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"} : dispatch 2024-08-28T19:12:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:11.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: [28/Aug/2024:19:12:09] ENGINE Bus STARTING 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: [28/Aug/2024:19:12:09] ENGINE Serving on https://172.21.15.16:7150 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: [28/Aug/2024:19:12:09] ENGINE Error in HTTPServer.serve 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-28T19:12:11.404 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: [28/Aug/2024:19:12:09] ENGINE Serving on http://172.21.15.16:8765 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: [28/Aug/2024:19:12:09] ENGINE Bus STARTED 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: mgrmap e17: x(active, since 2s) 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.405 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.406 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"} : dispatch 2024-08-28T19:12:11.406 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2958740331' entity='client.admin' 2024-08-28T19:12:11.406 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.406 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.406 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"} : dispatch 2024-08-28T19:12:11.406 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:11.406 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:11.426 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi016') 2024-08-28T19:12:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: [28/Aug/2024:19:12:09] ENGINE Bus STARTING 2024-08-28T19:12:11.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: [28/Aug/2024:19:12:09] ENGINE Serving on https://172.21.15.16:7150 2024-08-28T19:12:11.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: [28/Aug/2024:19:12:09] ENGINE Error in HTTPServer.serve 2024-08-28T19:12:11.560 INFO:journalctl@ceph.mon.b.smithi028.stdout: Traceback (most recent call last): 2024-08-28T19:12:11.560 INFO:journalctl@ceph.mon.b.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-28T19:12:11.561 INFO:journalctl@ceph.mon.b.smithi028.stdout: self._connections.run(self.expiration_interval) 2024-08-28T19:12:11.561 INFO:journalctl@ceph.mon.b.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-28T19:12:11.561 INFO:journalctl@ceph.mon.b.smithi028.stdout: self._run(expiration_interval) 2024-08-28T19:12:11.561 INFO:journalctl@ceph.mon.b.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-28T19:12:11.561 INFO:journalctl@ceph.mon.b.smithi028.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-28T19:12:11.561 INFO:journalctl@ceph.mon.b.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-28T19:12:11.561 INFO:journalctl@ceph.mon.b.smithi028.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-28T19:12:11.561 INFO:journalctl@ceph.mon.b.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-28T19:12:11.561 INFO:journalctl@ceph.mon.b.smithi028.stdout: s = self.context.wrap_socket( 2024-08-28T19:12:11.561 INFO:journalctl@ceph.mon.b.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-28T19:12:11.561 INFO:journalctl@ceph.mon.b.smithi028.stdout: return self.sslsocket_class._create( 2024-08-28T19:12:11.562 INFO:journalctl@ceph.mon.b.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-28T19:12:11.562 INFO:journalctl@ceph.mon.b.smithi028.stdout: self.do_handshake() 2024-08-28T19:12:11.562 INFO:journalctl@ceph.mon.b.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-28T19:12:11.562 INFO:journalctl@ceph.mon.b.smithi028.stdout: self._sslobj.do_handshake() 2024-08-28T19:12:11.562 INFO:journalctl@ceph.mon.b.smithi028.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-28T19:12:11.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: [28/Aug/2024:19:12:09] ENGINE Serving on http://172.21.15.16:8765 2024-08-28T19:12:11.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: [28/Aug/2024:19:12:09] ENGINE Bus STARTED 2024-08-28T19:12:11.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: mgrmap e17: x(active, since 2s) 2024-08-28T19:12:11.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"} : dispatch 2024-08-28T19:12:11.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2958740331' entity='client.admin' 2024-08-28T19:12:11.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"} : dispatch 2024-08-28T19:12:11.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:11.564 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: [28/Aug/2024:19:12:09] ENGINE Bus STARTING 2024-08-28T19:12:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: [28/Aug/2024:19:12:09] ENGINE Serving on https://172.21.15.16:7150 2024-08-28T19:12:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: [28/Aug/2024:19:12:09] ENGINE Error in HTTPServer.serve 2024-08-28T19:12:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2024-08-28T19:12:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-28T19:12:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._connections.run(self.expiration_interval) 2024-08-28T19:12:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-28T19:12:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._run(expiration_interval) 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: s = self.context.wrap_socket( 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.sslsocket_class._create( 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.do_handshake() 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._sslobj.do_handshake() 2024-08-28T19:12:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: [28/Aug/2024:19:12:09] ENGINE Serving on http://172.21.15.16:8765 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: [28/Aug/2024:19:12:09] ENGINE Bus STARTED 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: mgrmap e17: x(active, since 2s) 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"} : dispatch 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2958740331' entity='client.admin' 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"} : dispatch 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:12.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: Updating smithi016:/etc/ceph/ceph.conf 2024-08-28T19:12:12.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: Updating smithi028:/etc/ceph/ceph.conf 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: Updating smithi115:/etc/ceph/ceph.conf 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/4202516708' entity='client.admin' 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-28T19:12:12.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-08-28T19:12:12.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:12 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:12.301 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:12.302 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:12.436 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: Updating smithi016:/etc/ceph/ceph.conf 2024-08-28T19:12:12.436 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: Updating smithi028:/etc/ceph/ceph.conf 2024-08-28T19:12:12.436 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: Updating smithi115:/etc/ceph/ceph.conf 2024-08-28T19:12:12.436 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/4202516708' entity='client.admin' 2024-08-28T19:12:12.436 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.436 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.436 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.436 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.437 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.437 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.437 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.437 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-28T19:12:12.437 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-28T19:12:12.437 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-08-28T19:12:12.437 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:12 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:12.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: Updating smithi016:/etc/ceph/ceph.conf 2024-08-28T19:12:12.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: Updating smithi028:/etc/ceph/ceph.conf 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: Updating smithi115:/etc/ceph/ceph.conf 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/4202516708' entity='client.admin' 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-08-28T19:12:12.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:12 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:13.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: Updating smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:12:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:12:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:12:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: Updating smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:12:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:12:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:12:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: Deploying daemon mgr.z on smithi115 2024-08-28T19:12:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3906205285' entity='client.admin' 2024-08-28T19:12:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:13 smithi016 ceph-mon[18556]: from='client.? ' entity='client.admin' 2024-08-28T19:12:13.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: Updating smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:12:13.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:12:13.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:12:13.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:13.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:13.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:13.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: Updating smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:12:13.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:12:13.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:12:13.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: Deploying daemon mgr.z on smithi115 2024-08-28T19:12:13.509 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3906205285' entity='client.admin' 2024-08-28T19:12:13.509 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:13 smithi115 ceph-mon[22184]: from='client.? ' entity='client.admin' 2024-08-28T19:12:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: Updating smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:12:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:12:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-28T19:12:13.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:13.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:13.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-08-28T19:12:13.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: Updating smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:12:13.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: Updating smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:12:13.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: Updating smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-28T19:12:13.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: Deploying daemon mgr.z on smithi115 2024-08-28T19:12:13.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3906205285' entity='client.admin' 2024-08-28T19:12:13.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:13 smithi028 ceph-mon[21769]: from='client.? ' entity='client.admin' 2024-08-28T19:12:13.871 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:12:13 smithi115 systemd[1]: Starting Ceph mgr.z for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:12:14.144 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:12:13 smithi115 podman[24928]: 2024-08-28 19:12:13.780719752 +0000 UTC m=+0.015739187 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:12:14.144 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:12:13 smithi115 podman[24928]: 2024-08-28 19:12:13.887169151 +0000 UTC m=+0.122188613 container create b6a0a772a01ca7d7a2d0999f76cd7bced3b48b1f0c16088276b3585b614eddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-z, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-08-28T19:12:14.145 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:12:14 smithi115 podman[24928]: 2024-08-28 19:12:14.106196028 +0000 UTC m=+0.341215453 container init b6a0a772a01ca7d7a2d0999f76cd7bced3b48b1f0c16088276b3585b614eddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-28T19:12:14.145 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:12:14 smithi115 podman[24928]: 2024-08-28 19:12:14.108980347 +0000 UTC m=+0.343999775 container start b6a0a772a01ca7d7a2d0999f76cd7bced3b48b1f0c16088276b3585b614eddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-z, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-28T19:12:14.145 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:12:14 smithi115 ceph-mgr[24947]: -- 172.21.15.115:0/4169543561 <== mon.1 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56153bf8f0e0 con 0x561540d2e480 2024-08-28T19:12:14.192 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi016') 2024-08-28T19:12:14.621 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:12:14 smithi115 bash[24928]: b6a0a772a01ca7d7a2d0999f76cd7bced3b48b1f0c16088276b3585b614eddfa 2024-08-28T19:12:14.621 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:12:14 smithi115 systemd[1]: Started Ceph mgr.z for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:12:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:14 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1925405163' entity='client.admin' 2024-08-28T19:12:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:14.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-28T19:12:14.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-28T19:12:14.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-08-28T19:12:14.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:14.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:14 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3704513654' entity='client.admin' 2024-08-28T19:12:14.710 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:14 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1925405163' entity='client.admin' 2024-08-28T19:12:14.710 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:14.710 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:14.710 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:14.710 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-28T19:12:14.711 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-28T19:12:14.711 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-08-28T19:12:14.711 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:14.711 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:14 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3704513654' entity='client.admin' 2024-08-28T19:12:14.731 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:14 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1925405163' entity='client.admin' 2024-08-28T19:12:14.731 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:14.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:14.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:14.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-28T19:12:14.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-28T19:12:14.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-08-28T19:12:14.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:14.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:14 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3704513654' entity='client.admin' 2024-08-28T19:12:15.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:15 smithi016 ceph-mon[18556]: Deploying daemon mgr.y on smithi028 2024-08-28T19:12:15.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:15 smithi115 ceph-mon[22184]: Deploying daemon mgr.y on smithi028 2024-08-28T19:12:15.615 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:15 smithi028 ceph-mon[21769]: Deploying daemon mgr.y on smithi028 2024-08-28T19:12:15.808 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:15.809 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:16.262 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:12:16 smithi028 systemd[1]: Starting Ceph mgr.y for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:12:16.540 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:12:16 smithi028 podman[24067]: 2024-08-28 19:12:16.163922618 +0000 UTC m=+0.015896989 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:12:16.540 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:12:16 smithi028 podman[24067]: 2024-08-28 19:12:16.283065689 +0000 UTC m=+0.135040086 container create fed856f47993fabfbb1f2a3a9a9e6339a9a463b91b4b51ec508dd2be09c557c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-28T19:12:16.540 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:12:16 smithi028 podman[24067]: 2024-08-28 19:12:16.502128997 +0000 UTC m=+0.354103355 container init fed856f47993fabfbb1f2a3a9a9e6339a9a463b91b4b51ec508dd2be09c557c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.1, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-28T19:12:16.540 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:12:16 smithi028 podman[24067]: 2024-08-28 19:12:16.504770226 +0000 UTC m=+0.356744584 container start fed856f47993fabfbb1f2a3a9a9e6339a9a463b91b4b51ec508dd2be09c557c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-y, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-08-28T19:12:16.809 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:12:16 smithi028 ceph-mgr[24087]: -- 172.21.15.28:0/658067456 <== mon.2 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d2bccd90e0 con 0x55d2c1a78480 2024-08-28T19:12:16.810 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:12:16 smithi028 bash[24067]: fed856f47993fabfbb1f2a3a9a9e6339a9a463b91b4b51ec508dd2be09c557c1 2024-08-28T19:12:16.810 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:12:16 smithi028 systemd[1]: Started Ceph mgr.y for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:12:17.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:16 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/16382086' entity='client.admin' 2024-08-28T19:12:17.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:16 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:16 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:16 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:16 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:16 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:17.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:16 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/16382086' entity='client.admin' 2024-08-28T19:12:17.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:16 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:16 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:16 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:16 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:16 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:16 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/16382086' entity='client.admin' 2024-08-28T19:12:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:16 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:16 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:16 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:16 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:16 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:17.668 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:17.671 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:17.672 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:18.106 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:18.214 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:18.215 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:18.825 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:18.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:18.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:18.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:18.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:18.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:18.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:18.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:18.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: Reconfiguring mgr.x (unknown last config time)... 2024-08-28T19:12:18.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-28T19:12:18.827 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-08-28T19:12:18.827 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:18.827 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: Reconfiguring daemon mgr.x on smithi016 2024-08-28T19:12:18.827 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:18 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1369945892' entity='client.admin' 2024-08-28T19:12:19.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:19.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:19.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: Reconfiguring mgr.x (unknown last config time)... 2024-08-28T19:12:19.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-28T19:12:19.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-08-28T19:12:19.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:19.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: Reconfiguring daemon mgr.x on smithi016 2024-08-28T19:12:19.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:18 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1369945892' entity='client.admin' 2024-08-28T19:12:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: Reconfiguring mgr.x (unknown last config time)... 2024-08-28T19:12:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-28T19:12:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-08-28T19:12:19.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:19.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: Reconfiguring daemon mgr.x on smithi016 2024-08-28T19:12:19.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:18 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1369945892' entity='client.admin' 2024-08-28T19:12:20.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:19 smithi016 ceph-mon[18556]: Standby manager daemon z started 2024-08-28T19:12:20.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:19 smithi016 ceph-mon[18556]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-08-28T19:12:20.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:19 smithi016 ceph-mon[18556]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-28T19:12:20.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:19 smithi016 ceph-mon[18556]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-08-28T19:12:20.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:19 smithi016 ceph-mon[18556]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-28T19:12:20.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:19 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/743793585' entity='client.admin' 2024-08-28T19:12:20.036 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:19 smithi028 ceph-mon[21769]: Standby manager daemon z started 2024-08-28T19:12:20.036 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:19 smithi028 ceph-mon[21769]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-08-28T19:12:20.036 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:19 smithi028 ceph-mon[21769]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-28T19:12:20.036 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:19 smithi028 ceph-mon[21769]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-08-28T19:12:20.036 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:19 smithi028 ceph-mon[21769]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-28T19:12:20.036 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:19 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/743793585' entity='client.admin' 2024-08-28T19:12:20.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:19 smithi115 ceph-mon[22184]: Standby manager daemon z started 2024-08-28T19:12:20.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:19 smithi115 ceph-mon[22184]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-08-28T19:12:20.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:19 smithi115 ceph-mon[22184]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-28T19:12:20.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:19 smithi115 ceph-mon[22184]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-08-28T19:12:20.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:19 smithi115 ceph-mon[22184]: from='mgr.? 172.21.15.115:0/1655255919' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-28T19:12:20.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:19 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/743793585' entity='client.admin' 2024-08-28T19:12:21.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:20 smithi016 ceph-mon[18556]: mgrmap e18: x(active, since 12s), standbys: z 2024-08-28T19:12:21.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:20 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-08-28T19:12:21.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:20 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3216671927' entity='client.admin' 2024-08-28T19:12:21.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:20 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1439452837' entity='client.admin' 2024-08-28T19:12:21.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:20 smithi028 ceph-mon[21769]: mgrmap e18: x(active, since 12s), standbys: z 2024-08-28T19:12:21.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:20 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-08-28T19:12:21.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:20 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3216671927' entity='client.admin' 2024-08-28T19:12:21.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:20 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1439452837' entity='client.admin' 2024-08-28T19:12:21.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:20 smithi115 ceph-mon[22184]: mgrmap e18: x(active, since 12s), standbys: z 2024-08-28T19:12:21.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:20 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-08-28T19:12:21.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:20 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3216671927' entity='client.admin' 2024-08-28T19:12:21.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:20 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1439452837' entity='client.admin' 2024-08-28T19:12:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 -- 172.21.15.16:0/3585061615 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9fec079f10 msgr2=0x7f9fec07c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 --2- 172.21.15.16:0/3585061615 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9fec079f10 0x7f9fec07c420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400abe0 tx=0x7f9ff400a2d0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 -- 172.21.15.16:0/3585061615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa004109240 msgr2=0x7fa00419a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 --2- 172.21.15.16:0/3585061615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa004109240 0x7fa00419a7d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9ff802a910 tx=0x7f9ff8003b90 comp rx=0 tx=0).stop 2024-08-28T19:12:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 -- 172.21.15.16:0/3585061615 shutdown_connections 2024-08-28T19:12:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 --2- 172.21.15.16:0/3585061615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa004109b90 0x7fa00419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 --2- 172.21.15.16:0/3585061615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa004109240 0x7fa00419a7d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 --2- 172.21.15.16:0/3585061615 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9fec079f10 0x7f9fec07c420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 --2- 172.21.15.16:0/3585061615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa004103900 0x7fa00419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 -- 172.21.15.16:0/3585061615 >> 172.21.15.16:0/3585061615 conn(0x7fa00407b5f0 msgr2=0x7fa004107390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 -- 172.21.15.16:0/3585061615 shutdown_connections 2024-08-28T19:12:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.504+0000 7fa0097ce640 1 -- 172.21.15.16:0/3585061615 wait complete. 2024-08-28T19:12:21.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.656+0000 7fcf5d711640 1 -- 172.21.15.16:0/2172354543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fcf581a6cc0 con 0x7fcf58104660 2024-08-28T19:12:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.660+0000 7fcf3ffff640 1 -- 172.21.15.16:0/2172354543 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v22)=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7fcf48068a30 con 0x7fcf58104660 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 -- 172.21.15.16:0/2172354543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf40079ee0 msgr2=0x7fcf4007c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 --2- 172.21.15.16:0/2172354543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf40079ee0 0x7fcf4007c3f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcf5810f950 tx=0x7fcf4c005f90 comp rx=0 tx=0).stop 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 -- 172.21.15.16:0/2172354543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf58104660 msgr2=0x7fcf5819a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 --2- 172.21.15.16:0/2172354543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf58104660 0x7fcf5819a130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800ea60 tx=0x7fcf4800ef20 comp rx=0 tx=0).stop 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 -- 172.21.15.16:0/2172354543 shutdown_connections 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 --2- 172.21.15.16:0/2172354543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf40079ee0 0x7fcf4007c3f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 --2- 172.21.15.16:0/2172354543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf5810dc10 0x7fcf5819f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 --2- 172.21.15.16:0/2172354543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf58105040 0x7fcf5819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 --2- 172.21.15.16:0/2172354543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf58104660 0x7fcf5819a130 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 -- 172.21.15.16:0/2172354543 >> 172.21.15.16:0/2172354543 conn(0x7fcf58079530 msgr2=0x7fcf58107760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 -- 172.21.15.16:0/2172354543 shutdown_connections 2024-08-28T19:12:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.662+0000 7fcf5d711640 1 -- 172.21.15.16:0/2172354543 wait complete. 2024-08-28T19:12:21.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.986+0000 7f394313c640 1 -- 172.21.15.16:0/1628682319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393c108140 msgr2=0x7f393c108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.986+0000 7f394313c640 1 --2- 172.21.15.16:0/1628682319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393c108140 0x7f393c108550 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3928009900 tx=0x7f392802a410 comp rx=0 tx=0).stop 2024-08-28T19:12:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.986+0000 7f394313c640 1 -- 172.21.15.16:0/1628682319 shutdown_connections 2024-08-28T19:12:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.986+0000 7f394313c640 1 --2- 172.21.15.16:0/1628682319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c108a90 0x7f393c10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.986+0000 7f394313c640 1 --2- 172.21.15.16:0/1628682319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393c108140 0x7f393c108550 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.986+0000 7f394313c640 1 --2- 172.21.15.16:0/1628682319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f393c102800 0x7f393c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.986+0000 7f394313c640 1 -- 172.21.15.16:0/1628682319 >> 172.21.15.16:0/1628682319 conn(0x7f393c0794d0 msgr2=0x7f393c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.986+0000 7f394313c640 1 -- 172.21.15.16:0/1628682319 shutdown_connections 2024-08-28T19:12:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.986+0000 7f394313c640 1 -- 172.21.15.16:0/1628682319 wait complete. 2024-08-28T19:12:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f394313c640 1 Processor -- start 2024-08-28T19:12:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f394313c640 1 -- start start 2024-08-28T19:12:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f394313c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f393c102800 0x7f393c19c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f394313c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c108140 0x7f393c19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f394313c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393c108a90 0x7f393c1a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f394313c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f393c1126e0 con 0x7f393c108a90 2024-08-28T19:12:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f394313c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f393c112560 con 0x7f393c108140 2024-08-28T19:12:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f394313c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f393c112860 con 0x7f393c102800 2024-08-28T19:12:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f39416d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393c108a90 0x7f393c1a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f39416d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393c108a90 0x7f393c1a1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49962/0 (socket says 172.21.15.16:49962) 2024-08-28T19:12:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f39416d9640 1 -- 172.21.15.16:0/1767109280 learned_addr learned my addr 172.21.15.16:0/1767109280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f393bfff640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c108140 0x7f393c19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f3940ed8640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f393c102800 0x7f393c19c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f39416d9640 1 -- 172.21.15.16:0/1767109280 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f393c102800 msgr2=0x7f393c19c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f39416d9640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f393c102800 0x7f393c19c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f39416d9640 1 -- 172.21.15.16:0/1767109280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c108140 msgr2=0x7f393c19ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f39416d9640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c108140 0x7f393c19ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.987+0000 7f39416d9640 1 -- 172.21.15.16:0/1767109280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393c1a2430 con 0x7f393c108a90 2024-08-28T19:12:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.988+0000 7f393bfff640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c108140 0x7f393c19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:12:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.988+0000 7f39416d9640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393c108a90 0x7f393c1a1d00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f393400eb10 tx=0x7f393400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.988+0000 7f3939ffb640 1 -- 172.21.15.16:0/1767109280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934004b20 con 0x7f393c108a90 2024-08-28T19:12:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.988+0000 7f3939ffb640 1 -- 172.21.15.16:0/1767109280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3934004cc0 con 0x7f393c108a90 2024-08-28T19:12:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.988+0000 7f394313c640 1 -- 172.21.15.16:0/1767109280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f393c1a2720 con 0x7f393c108a90 2024-08-28T19:12:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.988+0000 7f3939ffb640 1 -- 172.21.15.16:0/1767109280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934010670 con 0x7f393c108a90 2024-08-28T19:12:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.988+0000 7f394313c640 1 -- 172.21.15.16:0/1767109280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f393c1a8e80 con 0x7f393c108a90 2024-08-28T19:12:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.989+0000 7f394313c640 1 -- 172.21.15.16:0/1767109280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f393c102c10 con 0x7f393c108a90 2024-08-28T19:12:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.990+0000 7f3939ffb640 1 -- 172.21.15.16:0/1767109280 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 101904+0+0 (secure 0 0 0) 0x7f3934005020 con 0x7f393c108a90 2024-08-28T19:12:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.990+0000 7f3939ffb640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f392c079ee0 0x7f392c07c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.990+0000 7f3940ed8640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f392c079ee0 0x7f392c07c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.990+0000 7f3939ffb640 1 -- 172.21.15.16:0/1767109280 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f393409bd20 con 0x7f393c108a90 2024-08-28T19:12:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.991+0000 7f3940ed8640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f392c079ee0 0x7f392c07c3f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3930005fd0 tx=0x7f3930005e20 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:21.995+0000 7f3939ffb640 1 -- 172.21.15.16:0/1767109280 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f3934064b70 con 0x7f393c108a90 2024-08-28T19:12:22.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.183+0000 7f394313c640 1 -- 172.21.15.16:0/1767109280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f393c100a00 con 0x7f393c108a90 2024-08-28T19:12:22.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.187+0000 7f3939ffb640 1 -- 172.21.15.16:0/1767109280 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v23)=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f3934064470 con 0x7f393c108a90 2024-08-28T19:12:22.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 -- 172.21.15.16:0/1767109280 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f392c079ee0 msgr2=0x7f392c07c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:22.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f392c079ee0 0x7f392c07c3f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3930005fd0 tx=0x7f3930005e20 comp rx=0 tx=0).stop 2024-08-28T19:12:22.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 -- 172.21.15.16:0/1767109280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393c108a90 msgr2=0x7f393c1a1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:22.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393c108a90 0x7f393c1a1d00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f393400eb10 tx=0x7f393400c2d0 comp rx=0 tx=0).stop 2024-08-28T19:12:22.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 -- 172.21.15.16:0/1767109280 shutdown_connections 2024-08-28T19:12:22.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393c108a90 0x7f393c1a1d00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:22.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f392c079ee0 0x7f392c07c3f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:22.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c108140 0x7f393c19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:22.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 --2- 172.21.15.16:0/1767109280 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f393c102800 0x7f393c19c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:22.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 -- 172.21.15.16:0/1767109280 >> 172.21.15.16:0/1767109280 conn(0x7f393c0794d0 msgr2=0x7f393c1052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:22.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 -- 172.21.15.16:0/1767109280 shutdown_connections 2024-08-28T19:12:22.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.189+0000 7f394313c640 1 -- 172.21.15.16:0/1767109280 wait complete. 2024-08-28T19:12:22.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:22 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3585061615' entity='client.admin' 2024-08-28T19:12:22.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:22 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2172354543' entity='client.admin' 2024-08-28T19:12:22.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:22 smithi016 ceph-mon[18556]: Standby manager daemon y started 2024-08-28T19:12:22.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:22 smithi016 ceph-mon[18556]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-08-28T19:12:22.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:22 smithi016 ceph-mon[18556]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-28T19:12:22.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:22 smithi016 ceph-mon[18556]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-08-28T19:12:22.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:22 smithi016 ceph-mon[18556]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-28T19:12:22.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:22 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1767109280' entity='client.admin' 2024-08-28T19:12:22.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:22 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3585061615' entity='client.admin' 2024-08-28T19:12:22.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:22 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2172354543' entity='client.admin' 2024-08-28T19:12:22.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:22 smithi028 ceph-mon[21769]: Standby manager daemon y started 2024-08-28T19:12:22.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:22 smithi028 ceph-mon[21769]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-08-28T19:12:22.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:22 smithi028 ceph-mon[21769]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-28T19:12:22.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:22 smithi028 ceph-mon[21769]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-08-28T19:12:22.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:22 smithi028 ceph-mon[21769]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-28T19:12:22.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:22 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1767109280' entity='client.admin' 2024-08-28T19:12:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:22 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3585061615' entity='client.admin' 2024-08-28T19:12:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:22 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2172354543' entity='client.admin' 2024-08-28T19:12:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:22 smithi115 ceph-mon[22184]: Standby manager daemon y started 2024-08-28T19:12:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:22 smithi115 ceph-mon[22184]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-08-28T19:12:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:22 smithi115 ceph-mon[22184]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-28T19:12:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:22 smithi115 ceph-mon[22184]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-08-28T19:12:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:22 smithi115 ceph-mon[22184]: from='mgr.? 172.21.15.28:0/1070391218' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-28T19:12:22.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:22 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1767109280' entity='client.admin' 2024-08-28T19:12:23.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.736+0000 7f6b73fd7640 1 -- 172.21.15.16:0/527591843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6c105050 msgr2=0x7f6b6c10d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:23.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.737+0000 7f6b73fd7640 1 --2- 172.21.15.16:0/527591843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6c105050 0x7f6b6c10d690 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009900 tx=0x7f6b5c02a410 comp rx=0 tx=0).stop 2024-08-28T19:12:23.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.737+0000 7f6b73fd7640 1 -- 172.21.15.16:0/527591843 shutdown_connections 2024-08-28T19:12:23.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.737+0000 7f6b73fd7640 1 --2- 172.21.15.16:0/527591843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b6c10dbd0 0x7f6b6c1100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.737+0000 7f6b73fd7640 1 --2- 172.21.15.16:0/527591843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6c105050 0x7f6b6c10d690 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.737+0000 7f6b73fd7640 1 --2- 172.21.15.16:0/527591843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c104620 0x7f6b6c104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.737+0000 7f6b73fd7640 1 -- 172.21.15.16:0/527591843 >> 172.21.15.16:0/527591843 conn(0x7f6b6c0794d0 msgr2=0x7f6b6c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:23.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.737+0000 7f6b73fd7640 1 -- 172.21.15.16:0/527591843 shutdown_connections 2024-08-28T19:12:23.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.737+0000 7f6b73fd7640 1 -- 172.21.15.16:0/527591843 wait complete. 2024-08-28T19:12:23.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.737+0000 7f6b73fd7640 1 Processor -- start 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.737+0000 7f6b73fd7640 1 -- start start 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b73fd7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c104620 0x7f6b6c102530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b73fd7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6c105050 0x7f6b6c102a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b73fd7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b6c10dbd0 0x7f6b6c102fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b73fd7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c113820 con 0x7f6b6c105050 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b73fd7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c1136a0 con 0x7f6b6c104620 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b73fd7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c1139a0 con 0x7f6b6c10dbd0 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b71572640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6c105050 0x7f6b6c102a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b71d73640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c104620 0x7f6b6c102530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b71572640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6c105050 0x7f6b6c102a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50018/0 (socket says 172.21.15.16:50018) 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b71572640 1 -- 172.21.15.16:0/2112811350 learned_addr learned my addr 172.21.15.16:0/2112811350 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:23.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b72574640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b6c10dbd0 0x7f6b6c102fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b71d73640 1 -- 172.21.15.16:0/2112811350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b6c10dbd0 msgr2=0x7f6b6c102fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b71d73640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b6c10dbd0 0x7f6b6c102fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b71d73640 1 -- 172.21.15.16:0/2112811350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6c105050 msgr2=0x7f6b6c102a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b71d73640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6c105050 0x7f6b6c102a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b71d73640 1 -- 172.21.15.16:0/2112811350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6c1a8590 con 0x7f6b6c104620 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b71572640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6c105050 0x7f6b6c102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.738+0000 7f6b71d73640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c104620 0x7f6b6c102530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c06b960 tx=0x7f6b6800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.739+0000 7f6b62ffd640 1 -- 172.21.15.16:0/2112811350 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b68017070 con 0x7f6b6c104620 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.739+0000 7f6b62ffd640 1 -- 172.21.15.16:0/2112811350 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b68009c30 con 0x7f6b6c104620 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.739+0000 7f6b73fd7640 1 -- 172.21.15.16:0/2112811350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6c1a8880 con 0x7f6b6c104620 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.739+0000 7f6b62ffd640 1 -- 172.21.15.16:0/2112811350 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6800f620 con 0x7f6b6c104620 2024-08-28T19:12:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.739+0000 7f6b73fd7640 1 -- 172.21.15.16:0/2112811350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b6c1a8de0 con 0x7f6b6c104620 2024-08-28T19:12:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.740+0000 7f6b73fd7640 1 -- 172.21.15.16:0/2112811350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b6c104a30 con 0x7f6b6c104620 2024-08-28T19:12:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.741+0000 7f6b62ffd640 1 -- 172.21.15.16:0/2112811350 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7f6b6801a070 con 0x7f6b6c104620 2024-08-28T19:12:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.742+0000 7f6b62ffd640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b580b4b70 0x7f6b580b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.742+0000 7f6b71572640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b580b4b70 0x7f6b580b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.742+0000 7f6b62ffd640 1 -- 172.21.15.16:0/2112811350 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f6b680e2ef0 con 0x7f6b6c104620 2024-08-28T19:12:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.742+0000 7f6b71572640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b580b4b70 0x7f6b580b7080 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c000c00 tx=0x7f6b5c005a20 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.746+0000 7f6b62ffd640 1 -- 172.21.15.16:0/2112811350 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f6b68009de0 con 0x7f6b6c104620 2024-08-28T19:12:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.938+0000 7f6b73fd7640 1 -- 172.21.15.16:0/2112811350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f6b6c1a91a0 con 0x7f6b6c104620 2024-08-28T19:12:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.944+0000 7f6b62ffd640 1 -- 172.21.15.16:0/2112811350 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v25)=0 v25) ==== 120+0+0 (secure 0 0 0) 0x7f6b680ab530 con 0x7f6b6c104620 2024-08-28T19:12:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 -- 172.21.15.16:0/2112811350 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b580b4b70 msgr2=0x7f6b580b7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b580b4b70 0x7f6b580b7080 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c000c00 tx=0x7f6b5c005a20 comp rx=0 tx=0).stop 2024-08-28T19:12:23.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 -- 172.21.15.16:0/2112811350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c104620 msgr2=0x7f6b6c102530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:23.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c104620 0x7f6b6c102530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c06b960 tx=0x7f6b6800dde0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 -- 172.21.15.16:0/2112811350 shutdown_connections 2024-08-28T19:12:23.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b6c10dbd0 0x7f6b6c102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6c105050 0x7f6b6c102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b580b4b70 0x7f6b580b7080 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 --2- 172.21.15.16:0/2112811350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c104620 0x7f6b6c102530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 -- 172.21.15.16:0/2112811350 >> 172.21.15.16:0/2112811350 conn(0x7f6b6c0794d0 msgr2=0x7f6b6c106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:23.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 -- 172.21.15.16:0/2112811350 shutdown_connections 2024-08-28T19:12:23.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.946+0000 7f6b73fd7640 1 -- 172.21.15.16:0/2112811350 wait complete. 2024-08-28T19:12:23.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:23 smithi016 ceph-mon[18556]: mgrmap e19: x(active, since 15s), standbys: z, y 2024-08-28T19:12:23.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:23 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-28T19:12:23.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:23 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2849242206' entity='client.admin' 2024-08-28T19:12:23.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:23 smithi016 ceph-mon[18556]: from='client.? ' entity='client.admin' 2024-08-28T19:12:23.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.649+0000 7f63155d9640 1 -- 172.21.15.16:0/2675296463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6310103960 msgr2=0x7f6310103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:23.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.649+0000 7f63155d9640 1 --2- 172.21.15.16:0/2675296463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6310103960 0x7f6310103d70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6300009900 tx=0x7f630002a410 comp rx=0 tx=0).stop 2024-08-28T19:12:23.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.649+0000 7f63155d9640 1 -- 172.21.15.16:0/2675296463 shutdown_connections 2024-08-28T19:12:23.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.649+0000 7f63155d9640 1 --2- 172.21.15.16:0/2675296463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6310109ba0 0x7f63101103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.649+0000 7f63155d9640 1 --2- 172.21.15.16:0/2675296463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6310109250 0x7f6310109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.649+0000 7f63155d9640 1 --2- 172.21.15.16:0/2675296463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6310103960 0x7f6310103d70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.649+0000 7f63155d9640 1 -- 172.21.15.16:0/2675296463 >> 172.21.15.16:0/2675296463 conn(0x7f631007b6b0 msgr2=0x7f631007bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:23.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.649+0000 7f63155d9640 1 -- 172.21.15.16:0/2675296463 shutdown_connections 2024-08-28T19:12:23.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.649+0000 7f63155d9640 1 -- 172.21.15.16:0/2675296463 wait complete. 2024-08-28T19:12:23.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.649+0000 7f63155d9640 1 Processor -- start 2024-08-28T19:12:23.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f63155d9640 1 -- start start 2024-08-28T19:12:23.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f63155d9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6310103960 0x7f631019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:23.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f63155d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6310109250 0x7f631019a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:23.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f63155d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6310109ba0 0x7f631019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:23.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f63155d9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6310113860 con 0x7f6310109250 2024-08-28T19:12:23.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f63155d9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63101136e0 con 0x7f6310109ba0 2024-08-28T19:12:23.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f63155d9640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f63101139e0 con 0x7f6310103960 2024-08-28T19:12:23.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f630e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6310109250 0x7f631019a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:23.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f630e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6310109250 0x7f631019a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49990/0 (socket says 172.21.15.16:49990) 2024-08-28T19:12:23.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f630e7fc640 1 -- 172.21.15.16:0/2849242206 learned_addr learned my addr 172.21.15.16:0/2849242206 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:23.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f630effd640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6310103960 0x7f631019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:23.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f630f7fe640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6310109ba0 0x7f631019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:23.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f630e7fc640 1 -- 172.21.15.16:0/2849242206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6310103960 msgr2=0x7f631019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:23.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f630e7fc640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6310103960 0x7f631019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f630e7fc640 1 -- 172.21.15.16:0/2849242206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6310109ba0 msgr2=0x7f631019fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:23.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f630e7fc640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6310109ba0 0x7f631019fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.650+0000 7f630e7fc640 1 -- 172.21.15.16:0/2849242206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63101a01e0 con 0x7f6310109250 2024-08-28T19:12:23.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.651+0000 7f630e7fc640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6310109250 0x7f631019a810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f630400d950 tx=0x7f630400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:23.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.651+0000 7f62f7fff640 1 -- 172.21.15.16:0/2849242206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6304017070 con 0x7f6310109250 2024-08-28T19:12:23.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.651+0000 7f63155d9640 1 -- 172.21.15.16:0/2849242206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63101a04d0 con 0x7f6310109250 2024-08-28T19:12:23.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.651+0000 7f62f7fff640 1 -- 172.21.15.16:0/2849242206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6304009c30 con 0x7f6310109250 2024-08-28T19:12:23.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.651+0000 7f62f7fff640 1 -- 172.21.15.16:0/2849242206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63040047a0 con 0x7f6310109250 2024-08-28T19:12:23.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.651+0000 7f63155d9640 1 -- 172.21.15.16:0/2849242206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6310075390 con 0x7f6310109250 2024-08-28T19:12:23.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.652+0000 7f63155d9640 1 -- 172.21.15.16:0/2849242206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6310103d70 con 0x7f6310109250 2024-08-28T19:12:23.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.653+0000 7f62f7fff640 1 -- 172.21.15.16:0/2849242206 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 101904+0+0 (secure 0 0 0) 0x7f6304004020 con 0x7f6310109250 2024-08-28T19:12:23.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.653+0000 7f62f7fff640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62f8079ee0 0x7f62f807c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:23.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.653+0000 7f62f7fff640 1 -- 172.21.15.16:0/2849242206 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f630409b410 con 0x7f6310109250 2024-08-28T19:12:23.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.653+0000 7f630effd640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62f8079ee0 0x7f62f807c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:23.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.654+0000 7f630effd640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62f8079ee0 0x7f62f807c3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6300031440 tx=0x7f6300009560 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:23.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.658+0000 7f62f7fff640 1 -- 172.21.15.16:0/2849242206 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f6304020020 con 0x7f6310109250 2024-08-28T19:12:23.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.668+0000 7f62f7fff640 1 -- 172.21.15.16:0/2849242206 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7f630405fdf0 con 0x7f6310109250 2024-08-28T19:12:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.851+0000 7f63155d9640 1 -- 172.21.15.16:0/2849242206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f631019b3a0 con 0x7f6310109250 2024-08-28T19:12:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.856+0000 7f62f7fff640 1 -- 172.21.15.16:0/2849242206 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v24)=0 v24) ==== 125+0+0 (secure 0 0 0) 0x7f6304064260 con 0x7f6310109250 2024-08-28T19:12:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.858+0000 7f63155d9640 1 -- 172.21.15.16:0/2849242206 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62f8079ee0 msgr2=0x7f62f807c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.858+0000 7f63155d9640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62f8079ee0 0x7f62f807c3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6300031440 tx=0x7f6300009560 comp rx=0 tx=0).stop 2024-08-28T19:12:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.859+0000 7f63155d9640 1 -- 172.21.15.16:0/2849242206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6310109250 msgr2=0x7f631019a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.859+0000 7f63155d9640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6310109250 0x7f631019a810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f630400d950 tx=0x7f630400de10 comp rx=0 tx=0).stop 2024-08-28T19:12:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.859+0000 7f63155d9640 1 -- 172.21.15.16:0/2849242206 shutdown_connections 2024-08-28T19:12:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.859+0000 7f63155d9640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62f8079ee0 0x7f62f807c3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.859+0000 7f63155d9640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6310109ba0 0x7f631019fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.859+0000 7f63155d9640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6310109250 0x7f631019a810 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.859+0000 7f63155d9640 1 --2- 172.21.15.16:0/2849242206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6310103960 0x7f631019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.859+0000 7f63155d9640 1 -- 172.21.15.16:0/2849242206 >> 172.21.15.16:0/2849242206 conn(0x7f631007b6b0 msgr2=0x7f6310107300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.859+0000 7f63155d9640 1 -- 172.21.15.16:0/2849242206 shutdown_connections 2024-08-28T19:12:23.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:22.859+0000 7f63155d9640 1 -- 172.21.15.16:0/2849242206 wait complete. 2024-08-28T19:12:24.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:23 smithi028 ceph-mon[21769]: mgrmap e19: x(active, since 15s), standbys: z, y 2024-08-28T19:12:24.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:23 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-28T19:12:24.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:23 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2849242206' entity='client.admin' 2024-08-28T19:12:24.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:23 smithi028 ceph-mon[21769]: from='client.? ' entity='client.admin' 2024-08-28T19:12:24.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:23 smithi115 ceph-mon[22184]: mgrmap e19: x(active, since 15s), standbys: z, y 2024-08-28T19:12:24.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:23 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-28T19:12:24.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:23 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2849242206' entity='client.admin' 2024-08-28T19:12:24.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:23 smithi115 ceph-mon[22184]: from='client.? ' entity='client.admin' 2024-08-28T19:12:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.458+0000 7f8b200ae640 1 -- 172.21.15.16:0/3359733700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b18108b70 msgr2=0x7f8b1810f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.458+0000 7f8b200ae640 1 --2- 172.21.15.16:0/3359733700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b18108b70 0x7f8b1810f3a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8b10009ca0 tx=0x7f8b1002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:12:24.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.458+0000 7f8b200ae640 1 -- 172.21.15.16:0/3359733700 shutdown_connections 2024-08-28T19:12:24.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.458+0000 7f8b200ae640 1 --2- 172.21.15.16:0/3359733700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b18108b70 0x7f8b1810f3a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:24.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.458+0000 7f8b200ae640 1 --2- 172.21.15.16:0/3359733700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b18108220 0x7f8b18108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:24.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.458+0000 7f8b200ae640 1 --2- 172.21.15.16:0/3359733700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b181028e0 0x7f8b18102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:24.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.458+0000 7f8b200ae640 1 -- 172.21.15.16:0/3359733700 >> 172.21.15.16:0/3359733700 conn(0x7f8b180794f0 msgr2=0x7f8b18079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:24.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.458+0000 7f8b200ae640 1 -- 172.21.15.16:0/3359733700 shutdown_connections 2024-08-28T19:12:24.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.458+0000 7f8b200ae640 1 -- 172.21.15.16:0/3359733700 wait complete. 2024-08-28T19:12:24.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.459+0000 7f8b200ae640 1 Processor -- start 2024-08-28T19:12:24.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.459+0000 7f8b200ae640 1 -- start start 2024-08-28T19:12:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.459+0000 7f8b200ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b181028e0 0x7f8b1819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.459+0000 7f8b200ae640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b18108220 0x7f8b1819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.459+0000 7f8b200ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b18108b70 0x7f8b1819fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.459+0000 7f8b200ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b18113860 con 0x7f8b18108b70 2024-08-28T19:12:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.459+0000 7f8b200ae640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181136e0 con 0x7f8b181028e0 2024-08-28T19:12:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.459+0000 7f8b200ae640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181139e0 con 0x7f8b18108220 2024-08-28T19:12:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.459+0000 7f8b1e64b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b18108b70 0x7f8b1819fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b1e64b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b18108b70 0x7f8b1819fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39824/0 (socket says 172.21.15.16:39824) 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b1e64b640 1 -- 172.21.15.16:0/2935181646 learned_addr learned my addr 172.21.15.16:0/2935181646 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b1de4a640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b181028e0 0x7f8b1819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b1d649640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b18108220 0x7f8b1819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b1e64b640 1 -- 172.21.15.16:0/2935181646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b18108220 msgr2=0x7f8b1819a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b1e64b640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b18108220 0x7f8b1819a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b1e64b640 1 -- 172.21.15.16:0/2935181646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b181028e0 msgr2=0x7f8b1819a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b1e64b640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b181028e0 0x7f8b1819a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b1e64b640 1 -- 172.21.15.16:0/2935181646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b181a0080 con 0x7f8b18108b70 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b1e64b640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b18108b70 0x7f8b1819fa10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8b10005cc0 tx=0x7f8b1002f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b0effd640 1 -- 172.21.15.16:0/2935181646 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b10003cf0 con 0x7f8b18108b70 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b0effd640 1 -- 172.21.15.16:0/2935181646 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b100044f0 con 0x7f8b18108b70 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b200ae640 1 -- 172.21.15.16:0/2935181646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b181a0310 con 0x7f8b18108b70 2024-08-28T19:12:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b0effd640 1 -- 172.21.15.16:0/2935181646 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1002fc30 con 0x7f8b18108b70 2024-08-28T19:12:24.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.460+0000 7f8b200ae640 1 -- 172.21.15.16:0/2935181646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b181a6b90 con 0x7f8b18108b70 2024-08-28T19:12:24.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.461+0000 7f8b200ae640 1 -- 172.21.15.16:0/2935181646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b18103b60 con 0x7f8b18108b70 2024-08-28T19:12:24.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.463+0000 7f8b0effd640 1 -- 172.21.15.16:0/2935181646 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7f8b100046a0 con 0x7f8b18108b70 2024-08-28T19:12:24.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.464+0000 7f8b0effd640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b040b4ba0 0x7f8b040b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:24.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.464+0000 7f8b1de4a640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b040b4ba0 0x7f8b040b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:24.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.464+0000 7f8b0effd640 1 -- 172.21.15.16:0/2935181646 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8b10104af0 con 0x7f8b18108b70 2024-08-28T19:12:24.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.464+0000 7f8b1de4a640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b040b4ba0 0x7f8b040b70b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8b140058c0 tx=0x7f8b1400a250 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:24.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.467+0000 7f8b0effd640 1 -- 172.21.15.16:0/2935181646 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8b100cd8b0 con 0x7f8b18108b70 2024-08-28T19:12:24.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.654+0000 7f8b200ae640 1 -- 172.21.15.16:0/2935181646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f8b18102cf0 con 0x7f8b18108b70 2024-08-28T19:12:24.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.658+0000 7f8b0effd640 1 -- 172.21.15.16:0/2935181646 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v26)=0 v26) ==== 116+0+0 (secure 0 0 0) 0x7f8b18102cf0 con 0x7f8b18108b70 2024-08-28T19:12:24.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.660+0000 7f8b200ae640 1 -- 172.21.15.16:0/2935181646 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b040b4ba0 msgr2=0x7f8b040b70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.660+0000 7f8b200ae640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b040b4ba0 0x7f8b040b70b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8b140058c0 tx=0x7f8b1400a250 comp rx=0 tx=0).stop 2024-08-28T19:12:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.660+0000 7f8b200ae640 1 -- 172.21.15.16:0/2935181646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b18108b70 msgr2=0x7f8b1819fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.660+0000 7f8b200ae640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b18108b70 0x7f8b1819fa10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8b10005cc0 tx=0x7f8b1002f470 comp rx=0 tx=0).stop 2024-08-28T19:12:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.661+0000 7f8b200ae640 1 -- 172.21.15.16:0/2935181646 shutdown_connections 2024-08-28T19:12:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.661+0000 7f8b200ae640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b18108b70 0x7f8b1819fa10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.661+0000 7f8b200ae640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b18108220 0x7f8b1819a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.661+0000 7f8b200ae640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b040b4ba0 0x7f8b040b70b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.661+0000 7f8b200ae640 1 --2- 172.21.15.16:0/2935181646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b181028e0 0x7f8b1819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.661+0000 7f8b200ae640 1 -- 172.21.15.16:0/2935181646 >> 172.21.15.16:0/2935181646 conn(0x7f8b180794f0 msgr2=0x7f8b18104250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.661+0000 7f8b200ae640 1 -- 172.21.15.16:0/2935181646 shutdown_connections 2024-08-28T19:12:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:24.661+0000 7f8b200ae640 1 -- 172.21.15.16:0/2935181646 wait complete. 2024-08-28T19:12:24.943 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi016') 2024-08-28T19:12:24.944 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi016') 2024-08-28T19:12:24.944 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi016') 2024-08-28T19:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.449+0000 7fad50d73640 1 -- 172.21.15.16:0/2037717244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad4c1163e0 msgr2=0x7fad4c1188f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.449+0000 7fad50d73640 1 --2- 172.21.15.16:0/2037717244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad4c1163e0 0x7fad4c1188f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fad44009ca0 tx=0x7fad4402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:12:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.449+0000 7fad50d73640 1 -- 172.21.15.16:0/2037717244 shutdown_connections 2024-08-28T19:12:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.449+0000 7fad50d73640 1 --2- 172.21.15.16:0/2037717244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad4c1163e0 0x7fad4c1188f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.449+0000 7fad50d73640 1 --2- 172.21.15.16:0/2037717244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad4c10ac80 0x7fad4c115d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.449+0000 7fad50d73640 1 --2- 172.21.15.16:0/2037717244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4c10a330 0x7fad4c10a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.449+0000 7fad50d73640 1 -- 172.21.15.16:0/2037717244 >> 172.21.15.16:0/2037717244 conn(0x7fad4c070c60 msgr2=0x7fad4c071070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.449+0000 7fad50d73640 1 -- 172.21.15.16:0/2037717244 shutdown_connections 2024-08-28T19:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.449+0000 7fad50d73640 1 -- 172.21.15.16:0/2037717244 wait complete. 2024-08-28T19:12:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.450+0000 7fad50d73640 1 Processor -- start 2024-08-28T19:12:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad50d73640 1 -- start start 2024-08-28T19:12:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad50d73640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad4c10a330 0x7fad4c1a2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad50d73640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4c10ac80 0x7fad4c1a3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad50d73640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad4c1163e0 0x7fad4c1a82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad50d73640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad4c11c060 con 0x7fad4c10a330 2024-08-28T19:12:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad50d73640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad4c11bee0 con 0x7fad4c10ac80 2024-08-28T19:12:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad50d73640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fad4c11c1e0 con 0x7fad4c1163e0 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad4a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad4c10a330 0x7fad4c1a2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad4a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad4c10a330 0x7fad4c1a2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39844/0 (socket says 172.21.15.16:39844) 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad4a59c640 1 -- 172.21.15.16:0/827525922 learned_addr learned my addr 172.21.15.16:0/827525922 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad49d9b640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4c10ac80 0x7fad4c1a3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad4ad9d640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad4c1163e0 0x7fad4c1a82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.451+0000 7fad4a59c640 1 -- 172.21.15.16:0/827525922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad4c1163e0 msgr2=0x7fad4c1a82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.452+0000 7fad4a59c640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad4c1163e0 0x7fad4c1a82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.452+0000 7fad4a59c640 1 -- 172.21.15.16:0/827525922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4c10ac80 msgr2=0x7fad4c1a3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.452+0000 7fad4a59c640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4c10ac80 0x7fad4c1a3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.452+0000 7fad4a59c640 1 -- 172.21.15.16:0/827525922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4c1a8a00 con 0x7fad4c10a330 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.452+0000 7fad4a59c640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad4c10a330 0x7fad4c1a2a90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00d950 tx=0x7fad3c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.452+0000 7fad3b7fe640 1 -- 172.21.15.16:0/827525922 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c017070 con 0x7fad4c10a330 2024-08-28T19:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.452+0000 7fad3b7fe640 1 -- 172.21.15.16:0/827525922 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad3c009c30 con 0x7fad4c10a330 2024-08-28T19:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.452+0000 7fad50d73640 1 -- 172.21.15.16:0/827525922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad4c1a8cf0 con 0x7fad4c10a330 2024-08-28T19:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.452+0000 7fad3b7fe640 1 -- 172.21.15.16:0/827525922 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c00f620 con 0x7fad4c10a330 2024-08-28T19:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.452+0000 7fad50d73640 1 -- 172.21.15.16:0/827525922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad4c1af450 con 0x7fad4c10a330 2024-08-28T19:12:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.453+0000 7fad50d73640 1 -- 172.21.15.16:0/827525922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad4c10a740 con 0x7fad4c10a330 2024-08-28T19:12:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.454+0000 7fad3b7fe640 1 -- 172.21.15.16:0/827525922 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7fad3c010460 con 0x7fad4c10a330 2024-08-28T19:12:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.455+0000 7fad3b7fe640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad340b4af0 0x7fad340b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:25.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.455+0000 7fad49d9b640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad340b4af0 0x7fad340b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:25.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.455+0000 7fad3b7fe640 1 -- 172.21.15.16:0/827525922 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fad3c0e3110 con 0x7fad4c10a330 2024-08-28T19:12:25.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.456+0000 7fad49d9b640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad340b4af0 0x7fad340b7000 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fad4c118010 tx=0x7fad4000a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.460+0000 7fad3b7fe640 1 -- 172.21.15.16:0/827525922 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fad3c009de0 con 0x7fad4c10a330 2024-08-28T19:12:25.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.652+0000 7fad50d73640 1 -- 172.21.15.16:0/827525922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7fad4c1a3bc0 con 0x7fad4c10a330 2024-08-28T19:12:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.657+0000 7fad3b7fe640 1 -- 172.21.15.16:0/827525922 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v27)=0 v27) ==== 108+0+0 (secure 0 0 0) 0x7fad3c0b4380 con 0x7fad4c10a330 2024-08-28T19:12:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 -- 172.21.15.16:0/827525922 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad340b4af0 msgr2=0x7fad340b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad340b4af0 0x7fad340b7000 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fad4c118010 tx=0x7fad4000a2d0 comp rx=0 tx=0).stop 2024-08-28T19:12:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 -- 172.21.15.16:0/827525922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad4c10a330 msgr2=0x7fad4c1a2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad4c10a330 0x7fad4c1a2a90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00d950 tx=0x7fad3c00de10 comp rx=0 tx=0).stop 2024-08-28T19:12:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 -- 172.21.15.16:0/827525922 shutdown_connections 2024-08-28T19:12:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad4c1163e0 0x7fad4c1a82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad340b4af0 0x7fad340b7000 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4c10ac80 0x7fad4c1a3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 --2- 172.21.15.16:0/827525922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad4c10a330 0x7fad4c1a2a90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 -- 172.21.15.16:0/827525922 >> 172.21.15.16:0/827525922 conn(0x7fad4c070c60 msgr2=0x7fad4c10ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 -- 172.21.15.16:0/827525922 shutdown_connections 2024-08-28T19:12:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.660+0000 7fad50d73640 1 -- 172.21.15.16:0/827525922 wait complete. 2024-08-28T19:12:25.892 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:25 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2935181646' entity='client.admin' 2024-08-28T19:12:25.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:25 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2935181646' entity='client.admin' 2024-08-28T19:12:26.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:25 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2935181646' entity='client.admin' 2024-08-28T19:12:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.632+0000 7f5d67603640 1 -- 172.21.15.16:0/3823472130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d601050a0 msgr2=0x7f5d6010d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.632+0000 7f5d67603640 1 --2- 172.21.15.16:0/3823472130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d601050a0 0x7f5d6010d5f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c009900 tx=0x7f5d5c02a410 comp rx=0 tx=0).stop 2024-08-28T19:12:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.632+0000 7f5d67603640 1 -- 172.21.15.16:0/3823472130 shutdown_connections 2024-08-28T19:12:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.632+0000 7f5d67603640 1 --2- 172.21.15.16:0/3823472130 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d6010dbb0 0x7f5d601100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.632+0000 7f5d67603640 1 --2- 172.21.15.16:0/3823472130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d601050a0 0x7f5d6010d5f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.632+0000 7f5d67603640 1 --2- 172.21.15.16:0/3823472130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d60104750 0x7f5d60104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.632+0000 7f5d67603640 1 -- 172.21.15.16:0/3823472130 >> 172.21.15.16:0/3823472130 conn(0x7f5d600794d0 msgr2=0x7f5d600798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.632+0000 7f5d67603640 1 -- 172.21.15.16:0/3823472130 shutdown_connections 2024-08-28T19:12:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.632+0000 7f5d67603640 1 -- 172.21.15.16:0/3823472130 wait complete. 2024-08-28T19:12:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.633+0000 7f5d67603640 1 Processor -- start 2024-08-28T19:12:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.633+0000 7f5d67603640 1 -- start start 2024-08-28T19:12:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d67603640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d60104750 0x7f5d6019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d67603640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d601050a0 0x7f5d6019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d67603640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d6010dbb0 0x7f5d6019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d67603640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d60113840 con 0x7f5d601050a0 2024-08-28T19:12:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d67603640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d601136c0 con 0x7f5d60104750 2024-08-28T19:12:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d67603640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5d601139c0 con 0x7f5d6010dbb0 2024-08-28T19:12:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d64b9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d601050a0 0x7f5d6019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d64b9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d601050a0 0x7f5d6019a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39870/0 (socket says 172.21.15.16:39870) 2024-08-28T19:12:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d64b9e640 1 -- 172.21.15.16:0/3013226038 learned_addr learned my addr 172.21.15.16:0/3013226038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d65ba0640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d6010dbb0 0x7f5d6019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d6539f640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d60104750 0x7f5d6019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d64b9e640 1 -- 172.21.15.16:0/3013226038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d6010dbb0 msgr2=0x7f5d6019faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d64b9e640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d6010dbb0 0x7f5d6019faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d64b9e640 1 -- 172.21.15.16:0/3013226038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d60104750 msgr2=0x7f5d6019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d64b9e640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d60104750 0x7f5d6019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d64b9e640 1 -- 172.21.15.16:0/3013226038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d601a0220 con 0x7f5d601050a0 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.634+0000 7f5d64b9e640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d601050a0 0x7f5d6019a850 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c005ab0 tx=0x7f5d5c003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.635+0000 7f5d567fc640 1 -- 172.21.15.16:0/3013226038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c03f070 con 0x7f5d601050a0 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.635+0000 7f5d67603640 1 -- 172.21.15.16:0/3013226038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d601a04b0 con 0x7f5d601050a0 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.635+0000 7f5d567fc640 1 -- 172.21.15.16:0/3013226038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d5c031d60 con 0x7f5d601050a0 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.635+0000 7f5d67603640 1 -- 172.21.15.16:0/3013226038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d601a6c70 con 0x7f5d601050a0 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.635+0000 7f5d567fc640 1 -- 172.21.15.16:0/3013226038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c035030 con 0x7f5d601050a0 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.636+0000 7f5d67603640 1 -- 172.21.15.16:0/3013226038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d60104b60 con 0x7f5d601050a0 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.638+0000 7f5d567fc640 1 -- 172.21.15.16:0/3013226038 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7f5d5c042070 con 0x7f5d601050a0 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.639+0000 7f5d567fc640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5d4c0b4af0 0x7f5d4c0b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.639+0000 7f5d6539f640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5d4c0b4af0 0x7f5d4c0b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.639+0000 7f5d567fc640 1 -- 172.21.15.16:0/3013226038 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f5d5c103fa0 con 0x7f5d601050a0 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.639+0000 7f5d6539f640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5d4c0b4af0 0x7f5d4c0b7000 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5d6006ba00 tx=0x7f5d5000b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.641+0000 7f5d567fc640 1 -- 172.21.15.16:0/3013226038 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f5d5c0cc5e0 con 0x7f5d601050a0 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.839+0000 7f5d67603640 1 -- 172.21.15.16:0/3013226038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f5d6019b3e0 con 0x7f5d601050a0 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.843+0000 7f5d567fc640 1 -- 172.21.15.16:0/3013226038 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v28)=0 v28) ==== 115+0+0 (secure 0 0 0) 0x7f5d5c0d0da0 con 0x7f5d601050a0 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 -- 172.21.15.16:0/3013226038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5d4c0b4af0 msgr2=0x7f5d4c0b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5d4c0b4af0 0x7f5d4c0b7000 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5d6006ba00 tx=0x7f5d5000b390 comp rx=0 tx=0).stop 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 -- 172.21.15.16:0/3013226038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d601050a0 msgr2=0x7f5d6019a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d601050a0 0x7f5d6019a850 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c005ab0 tx=0x7f5d5c003cd0 comp rx=0 tx=0).stop 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 -- 172.21.15.16:0/3013226038 shutdown_connections 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d6010dbb0 0x7f5d6019faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d601050a0 0x7f5d6019a850 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5d4c0b4af0 0x7f5d4c0b7000 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:26.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 --2- 172.21.15.16:0/3013226038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d60104750 0x7f5d6019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:26.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 -- 172.21.15.16:0/3013226038 >> 172.21.15.16:0/3013226038 conn(0x7f5d600794d0 msgr2=0x7f5d60100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:26.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 -- 172.21.15.16:0/3013226038 shutdown_connections 2024-08-28T19:12:26.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:25.845+0000 7f5d67603640 1 -- 172.21.15.16:0/3013226038 wait complete. 2024-08-28T19:12:26.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:26 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/827525922' entity='client.admin' 2024-08-28T19:12:26.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:26 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3013226038' entity='client.admin' 2024-08-28T19:12:26.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:26 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/827525922' entity='client.admin' 2024-08-28T19:12:26.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:26 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3013226038' entity='client.admin' 2024-08-28T19:12:27.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:26 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/827525922' entity='client.admin' 2024-08-28T19:12:27.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:26 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3013226038' entity='client.admin' 2024-08-28T19:12:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:27.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.805+0000 7f8a95f85640 1 -- 172.21.15.16:0/1484734646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90108ac0 msgr2=0x7f8a9010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:27.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.805+0000 7f8a95f85640 1 --2- 172.21.15.16:0/1484734646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90108ac0 0x7f8a9010f2f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8a88009ca0 tx=0x7f8a8802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:12:27.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.805+0000 7f8a95f85640 1 -- 172.21.15.16:0/1484734646 shutdown_connections 2024-08-28T19:12:27.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.805+0000 7f8a95f85640 1 --2- 172.21.15.16:0/1484734646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90108ac0 0x7f8a9010f2f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:27.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.805+0000 7f8a95f85640 1 --2- 172.21.15.16:0/1484734646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a90108170 0x7f8a90108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:27.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.805+0000 7f8a95f85640 1 --2- 172.21.15.16:0/1484734646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a90102880 0x7f8a90102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:27.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.805+0000 7f8a95f85640 1 -- 172.21.15.16:0/1484734646 >> 172.21.15.16:0/1484734646 conn(0x7f8a90079530 msgr2=0x7f8a90079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:27.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.805+0000 7f8a95f85640 1 -- 172.21.15.16:0/1484734646 shutdown_connections 2024-08-28T19:12:27.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.805+0000 7f8a95f85640 1 -- 172.21.15.16:0/1484734646 wait complete. 2024-08-28T19:12:27.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.806+0000 7f8a95f85640 1 Processor -- start 2024-08-28T19:12:27.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.806+0000 7f8a95f85640 1 -- start start 2024-08-28T19:12:27.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.806+0000 7f8a95f85640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a90102880 0x7f8a900ff410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:27.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.806+0000 7f8a95f85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a90108170 0x7f8a900ff950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:27.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.806+0000 7f8a95f85640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90108ac0 0x7f8a900ffe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:27.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.806+0000 7f8a95f85640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a90112790 con 0x7f8a90108ac0 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.806+0000 7f8a95f85640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a90112610 con 0x7f8a90108170 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.806+0000 7f8a95f85640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8a90112910 con 0x7f8a90102880 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90108ac0 0x7f8a900ffe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90108ac0 0x7f8a900ffe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39898/0 (socket says 172.21.15.16:39898) 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8ffff640 1 -- 172.21.15.16:0/2643292038 learned_addr learned my addr 172.21.15.16:0/2643292038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8effd640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a90108170 0x7f8a900ff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8f7fe640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a90102880 0x7f8a900ff410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8ffff640 1 -- 172.21.15.16:0/2643292038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a90102880 msgr2=0x7f8a900ff410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8ffff640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a90102880 0x7f8a900ff410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8ffff640 1 -- 172.21.15.16:0/2643292038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a90108170 msgr2=0x7f8a900ff950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8ffff640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a90108170 0x7f8a900ff950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8ffff640 1 -- 172.21.15.16:0/2643292038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a901a8550 con 0x7f8a90108ac0 2024-08-28T19:12:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8effd640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a90108170 0x7f8a900ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:12:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8ffff640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90108ac0 0x7f8a900ffe90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a880098d0 tx=0x7f8a880317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.807+0000 7f8a8cff9640 1 -- 172.21.15.16:0/2643292038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88003b70 con 0x7f8a90108ac0 2024-08-28T19:12:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.808+0000 7f8a8cff9640 1 -- 172.21.15.16:0/2643292038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a88003d10 con 0x7f8a90108ac0 2024-08-28T19:12:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.808+0000 7f8a95f85640 1 -- 172.21.15.16:0/2643292038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a901a87e0 con 0x7f8a90108ac0 2024-08-28T19:12:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.808+0000 7f8a8cff9640 1 -- 172.21.15.16:0/2643292038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8802fa20 con 0x7f8a90108ac0 2024-08-28T19:12:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.808+0000 7f8a95f85640 1 -- 172.21.15.16:0/2643292038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a901a8d20 con 0x7f8a90108ac0 2024-08-28T19:12:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.809+0000 7f8a95f85640 1 -- 172.21.15.16:0/2643292038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a90102c90 con 0x7f8a90108ac0 2024-08-28T19:12:27.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.811+0000 7f8a8cff9640 1 -- 172.21.15.16:0/2643292038 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7f8a8802fbe0 con 0x7f8a90108ac0 2024-08-28T19:12:27.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.812+0000 7f8a8cff9640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a780b4af0 0x7f8a780b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:27.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.812+0000 7f8a8f7fe640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a780b4af0 0x7f8a780b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:27.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.812+0000 7f8a8cff9640 1 -- 172.21.15.16:0/2643292038 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8a88104950 con 0x7f8a90108ac0 2024-08-28T19:12:27.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.813+0000 7f8a8f7fe640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a780b4af0 0x7f8a780b7000 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8a90100e60 tx=0x7f8a8000b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:27.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:27.816+0000 7f8a8cff9640 1 -- 172.21.15.16:0/2643292038 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8a880cd680 con 0x7f8a90108ac0 2024-08-28T19:12:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.002+0000 7f8a95f85640 1 -- 172.21.15.16:0/2643292038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7f8a9002e430 con 0x7f8a90108ac0 2024-08-28T19:12:28.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.006+0000 7f8a8cff9640 1 -- 172.21.15.16:0/2643292038 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v29)=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7f8a880ccf80 con 0x7f8a90108ac0 2024-08-28T19:12:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 -- 172.21.15.16:0/2643292038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a780b4af0 msgr2=0x7f8a780b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a780b4af0 0x7f8a780b7000 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8a90100e60 tx=0x7f8a8000b390 comp rx=0 tx=0).stop 2024-08-28T19:12:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 -- 172.21.15.16:0/2643292038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90108ac0 msgr2=0x7f8a900ffe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90108ac0 0x7f8a900ffe90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a880098d0 tx=0x7f8a880317b0 comp rx=0 tx=0).stop 2024-08-28T19:12:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 -- 172.21.15.16:0/2643292038 shutdown_connections 2024-08-28T19:12:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90108ac0 0x7f8a900ffe90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a780b4af0 0x7f8a780b7000 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a90108170 0x7f8a900ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:28.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 --2- 172.21.15.16:0/2643292038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a90102880 0x7f8a900ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:28.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 -- 172.21.15.16:0/2643292038 >> 172.21.15.16:0/2643292038 conn(0x7f8a90079530 msgr2=0x7f8a90101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:28.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 -- 172.21.15.16:0/2643292038 shutdown_connections 2024-08-28T19:12:28.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:28.008+0000 7f8a95f85640 1 -- 172.21.15.16:0/2643292038 wait complete. 2024-08-28T19:12:28.175 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi016') 2024-08-28T19:12:28.175 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi016') 2024-08-28T19:12:28.175 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi016') 2024-08-28T19:12:29.061 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:29 smithi016 ceph-mon[18556]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:29.061 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:29 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2643292038' entity='client.admin' 2024-08-28T19:12:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.211+0000 7fa7b27fc640 1 -- 172.21.15.16:0/105513260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a80318a0 con 0x7fa7b41050b0 2024-08-28T19:12:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.211+0000 7fa7ba227640 1 -- 172.21.15.16:0/105513260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b41050b0 msgr2=0x7fa7b410d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.211+0000 7fa7ba227640 1 --2- 172.21.15.16:0/105513260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b41050b0 0x7fa7b410d600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8009900 tx=0x7fa7a802a410 comp rx=0 tx=0).stop 2024-08-28T19:12:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.211+0000 7fa7ba227640 1 -- 172.21.15.16:0/105513260 shutdown_connections 2024-08-28T19:12:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.211+0000 7fa7ba227640 1 --2- 172.21.15.16:0/105513260 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b410dbc0 0x7fa7b41100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.211+0000 7fa7ba227640 1 --2- 172.21.15.16:0/105513260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b41050b0 0x7fa7b410d600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.211+0000 7fa7ba227640 1 --2- 172.21.15.16:0/105513260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b4104760 0x7fa7b4104b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.211+0000 7fa7ba227640 1 -- 172.21.15.16:0/105513260 >> 172.21.15.16:0/105513260 conn(0x7fa7b40794b0 msgr2=0x7fa7b40798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.212+0000 7fa7ba227640 1 -- 172.21.15.16:0/105513260 shutdown_connections 2024-08-28T19:12:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.212+0000 7fa7ba227640 1 -- 172.21.15.16:0/105513260 wait complete. 2024-08-28T19:12:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.212+0000 7fa7ba227640 1 Processor -- start 2024-08-28T19:12:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7ba227640 1 -- start start 2024-08-28T19:12:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7ba227640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b4104760 0x7fa7b419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7ba227640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b41050b0 0x7fa7b419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7ba227640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b410dbc0 0x7fa7b419fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7ba227640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b41137e0 con 0x7fa7b410dbc0 2024-08-28T19:12:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7ba227640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b4113660 con 0x7fa7b4104760 2024-08-28T19:12:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7ba227640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b4113960 con 0x7fa7b41050b0 2024-08-28T19:12:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b410dbc0 0x7fa7b419fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b410dbc0 0x7fa7b419fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39908/0 (socket says 172.21.15.16:39908) 2024-08-28T19:12:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7b3fff640 1 -- 172.21.15.16:0/2185495006 learned_addr learned my addr 172.21.15.16:0/2185495006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7b37fe640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b4104760 0x7fa7b419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7b2ffd640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b41050b0 0x7fa7b419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7b3fff640 1 -- 172.21.15.16:0/2185495006 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b41050b0 msgr2=0x7fa7b419a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7b3fff640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b41050b0 0x7fa7b419a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7b3fff640 1 -- 172.21.15.16:0/2185495006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b4104760 msgr2=0x7fa7b419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7b3fff640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b4104760 0x7fa7b419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7b3fff640 1 -- 172.21.15.16:0/2185495006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b41a0180 con 0x7fa7b410dbc0 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.213+0000 7fa7b2ffd640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b41050b0 0x7fa7b419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.214+0000 7fa7b3fff640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b410dbc0 0x7fa7b419fa50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00eaa0 tx=0x7fa7ac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.214+0000 7fa7b0ff9640 1 -- 172.21.15.16:0/2185495006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac00cdb0 con 0x7fa7b410dbc0 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.214+0000 7fa7ba227640 1 -- 172.21.15.16:0/2185495006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b41a0470 con 0x7fa7b410dbc0 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.214+0000 7fa7b0ff9640 1 -- 172.21.15.16:0/2185495006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7ac005020 con 0x7fa7b410dbc0 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.214+0000 7fa7b0ff9640 1 -- 172.21.15.16:0/2185495006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac0116d0 con 0x7fa7b410dbc0 2024-08-28T19:12:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.214+0000 7fa7ba227640 1 -- 172.21.15.16:0/2185495006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b41a6bd0 con 0x7fa7b410dbc0 2024-08-28T19:12:29.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.215+0000 7fa7ba227640 1 -- 172.21.15.16:0/2185495006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b406c020 con 0x7fa7b410dbc0 2024-08-28T19:12:29.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.217+0000 7fa7b0ff9640 1 -- 172.21.15.16:0/2185495006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7fa7ac0051d0 con 0x7fa7b410dbc0 2024-08-28T19:12:29.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.217+0000 7fa7b0ff9640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa79c0b4af0 0x7fa79c0b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:29.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.217+0000 7fa7b37fe640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa79c0b4af0 0x7fa79c0b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:29.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.217+0000 7fa7b0ff9640 1 -- 172.21.15.16:0/2185495006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa7ac0e3b40 con 0x7fa7b410dbc0 2024-08-28T19:12:29.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.218+0000 7fa7b37fe640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa79c0b4af0 0x7fa79c0b7000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40058c0 tx=0x7fa7a400a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:29.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.221+0000 7fa7b0ff9640 1 -- 172.21.15.16:0/2185495006 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fa7ac016030 con 0x7fa7b410dbc0 2024-08-28T19:12:29.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:29 smithi028 ceph-mon[21769]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:29.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:29 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2643292038' entity='client.admin' 2024-08-28T19:12:29.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:29 smithi115 ceph-mon[22184]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:29.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:29 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2643292038' entity='client.admin' 2024-08-28T19:12:29.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.413+0000 7fa7ba227640 1 -- 172.21.15.16:0/2185495006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fa7b4104b70 con 0x7fa7b410dbc0 2024-08-28T19:12:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.417+0000 7fa7b0ff9640 1 -- 172.21.15.16:0/2185495006 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7fa7b4104b70 con 0x7fa7b410dbc0 2024-08-28T19:12:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 -- 172.21.15.16:0/2185495006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa79c0b4af0 msgr2=0x7fa79c0b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa79c0b4af0 0x7fa79c0b7000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40058c0 tx=0x7fa7a400a2d0 comp rx=0 tx=0).stop 2024-08-28T19:12:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 -- 172.21.15.16:0/2185495006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b410dbc0 msgr2=0x7fa7b419fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b410dbc0 0x7fa7b419fa50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00eaa0 tx=0x7fa7ac00ef60 comp rx=0 tx=0).stop 2024-08-28T19:12:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 -- 172.21.15.16:0/2185495006 shutdown_connections 2024-08-28T19:12:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b410dbc0 0x7fa7b419fa50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b41050b0 0x7fa7b419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa79c0b4af0 0x7fa79c0b7000 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 --2- 172.21.15.16:0/2185495006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b4104760 0x7fa7b419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 -- 172.21.15.16:0/2185495006 >> 172.21.15.16:0/2185495006 conn(0x7fa7b40794b0 msgr2=0x7fa7b4107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 -- 172.21.15.16:0/2185495006 shutdown_connections 2024-08-28T19:12:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:29.420+0000 7fa7ba227640 1 -- 172.21.15.16:0/2185495006 wait complete. 2024-08-28T19:12:30.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:30 smithi016 ceph-mon[18556]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:30.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:30 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2185495006' entity='client.admin' 2024-08-28T19:12:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:30 smithi028 ceph-mon[21769]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:30 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2185495006' entity='client.admin' 2024-08-28T19:12:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:30 smithi115 ceph-mon[22184]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:30 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2185495006' entity='client.admin' 2024-08-28T19:12:31.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.087+0000 7f613d085640 1 -- 172.21.15.16:0/3555640803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6138108140 msgr2=0x7f6138108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:31.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.087+0000 7f613d085640 1 --2- 172.21.15.16:0/3555640803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6138108140 0x7f6138108550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f612c009900 tx=0x7f612c02a410 comp rx=0 tx=0).stop 2024-08-28T19:12:31.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.087+0000 7f613d085640 1 -- 172.21.15.16:0/3555640803 shutdown_connections 2024-08-28T19:12:31.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.087+0000 7f613d085640 1 --2- 172.21.15.16:0/3555640803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138108a90 0x7f613810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:31.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.087+0000 7f613d085640 1 --2- 172.21.15.16:0/3555640803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6138108140 0x7f6138108550 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:31.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.087+0000 7f613d085640 1 --2- 172.21.15.16:0/3555640803 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6138102800 0x7f6138102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:31.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.087+0000 7f613d085640 1 -- 172.21.15.16:0/3555640803 >> 172.21.15.16:0/3555640803 conn(0x7f61380794d0 msgr2=0x7f61380798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:31.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.087+0000 7f613d085640 1 -- 172.21.15.16:0/3555640803 shutdown_connections 2024-08-28T19:12:31.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.087+0000 7f613d085640 1 -- 172.21.15.16:0/3555640803 wait complete. 2024-08-28T19:12:31.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.087+0000 7f613d085640 1 Processor -- start 2024-08-28T19:12:31.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f613d085640 1 -- start start 2024-08-28T19:12:31.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f613d085640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6138102800 0x7f613819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:31.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f613d085640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138108140 0x7f613819a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:31.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f613d085640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6138108a90 0x7f613819fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f613d085640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6138113820 con 0x7f6138102800 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f613d085640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61381136a0 con 0x7f6138108140 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f613d085640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f61381139a0 con 0x7f6138108a90 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f6136d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6138102800 0x7f613819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f6136d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6138102800 0x7f613819a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39932/0 (socket says 172.21.15.16:39932) 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f6136d9d640 1 -- 172.21.15.16:0/1250177172 learned_addr learned my addr 172.21.15.16:0/1250177172 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f613759e640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6138108a90 0x7f613819fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f613659c640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138108140 0x7f613819a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f6136d9d640 1 -- 172.21.15.16:0/1250177172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6138108a90 msgr2=0x7f613819fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f6136d9d640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6138108a90 0x7f613819fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f6136d9d640 1 -- 172.21.15.16:0/1250177172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138108140 msgr2=0x7f613819a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f6136d9d640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138108140 0x7f613819a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:31.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f6136d9d640 1 -- 172.21.15.16:0/1250177172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61381a0240 con 0x7f6138102800 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.088+0000 7f613659c640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138108140 0x7f613819a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.089+0000 7f6136d9d640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6138102800 0x7f613819a2d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f612800e9f0 tx=0x7f612800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.089+0000 7f611ffff640 1 -- 172.21.15.16:0/1250177172 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612800cd80 con 0x7f6138102800 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.089+0000 7f613d085640 1 -- 172.21.15.16:0/1250177172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61381a0530 con 0x7f6138102800 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.089+0000 7f611ffff640 1 -- 172.21.15.16:0/1250177172 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61280054d0 con 0x7f6138102800 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.089+0000 7f611ffff640 1 -- 172.21.15.16:0/1250177172 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128010690 con 0x7f6138102800 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.089+0000 7f613d085640 1 -- 172.21.15.16:0/1250177172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61381a6c90 con 0x7f6138102800 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.090+0000 7f613d085640 1 -- 172.21.15.16:0/1250177172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6138102c10 con 0x7f6138102800 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.091+0000 7f611ffff640 1 -- 172.21.15.16:0/1250177172 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7f6128004e30 con 0x7f6138102800 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.092+0000 7f611ffff640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61200b4af0 0x7f61200b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.092+0000 7f613659c640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61200b4af0 0x7f61200b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.092+0000 7f611ffff640 1 -- 172.21.15.16:0/1250177172 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f61280e3bd0 con 0x7f6138102800 2024-08-28T19:12:31.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.093+0000 7f613659c640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61200b4af0 0x7f61200b7000 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f612c0096c0 tx=0x7f612c031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:31.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.095+0000 7f611ffff640 1 -- 172.21.15.16:0/1250177172 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f6128056130 con 0x7f6138102800 2024-08-28T19:12:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.281+0000 7f613d085640 1 -- 172.21.15.16:0/1250177172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=debug_cephsqlite}] v 0) -- 0x7f613819b400 con 0x7f6138102800 2024-08-28T19:12:31.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.285+0000 7f611ffff640 1 -- 172.21.15.16:0/1250177172 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_cephsqlite}]=0 v31)=0 v31) ==== 115+0+0 (secure 0 0 0) 0x7f613819b400 con 0x7f6138102800 2024-08-28T19:12:31.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 -- 172.21.15.16:0/1250177172 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61200b4af0 msgr2=0x7f61200b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61200b4af0 0x7f61200b7000 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f612c0096c0 tx=0x7f612c031400 comp rx=0 tx=0).stop 2024-08-28T19:12:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 -- 172.21.15.16:0/1250177172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6138102800 msgr2=0x7f613819a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6138102800 0x7f613819a2d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f612800e9f0 tx=0x7f612800edf0 comp rx=0 tx=0).stop 2024-08-28T19:12:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 -- 172.21.15.16:0/1250177172 shutdown_connections 2024-08-28T19:12:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6138108a90 0x7f613819fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61200b4af0 0x7f61200b7000 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138108140 0x7f613819a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 --2- 172.21.15.16:0/1250177172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6138102800 0x7f613819a2d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 -- 172.21.15.16:0/1250177172 >> 172.21.15.16:0/1250177172 conn(0x7f61380794d0 msgr2=0x7f61381042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 -- 172.21.15.16:0/1250177172 shutdown_connections 2024-08-28T19:12:31.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:31.288+0000 7f613d085640 1 -- 172.21.15.16:0/1250177172 wait complete. 2024-08-28T19:12:31.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:31 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:31.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:31 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:31.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:31 smithi016 ceph-mon[18556]: Reconfiguring mon.c (monmap changed)... 2024-08-28T19:12:31.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:31 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:12:31.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:31 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:12:31.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:31 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:31.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:31 smithi016 ceph-mon[18556]: Reconfiguring daemon mon.c on smithi115 2024-08-28T19:12:31.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:31 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1250177172' entity='client.admin' 2024-08-28T19:12:31.812 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:31 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:31.812 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:31 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:31.813 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:31 smithi028 ceph-mon[21769]: Reconfiguring mon.c (monmap changed)... 2024-08-28T19:12:31.813 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:31 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:12:31.813 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:31 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:12:31.813 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:31 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:31.813 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:31 smithi028 ceph-mon[21769]: Reconfiguring daemon mon.c on smithi115 2024-08-28T19:12:31.813 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:31 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1250177172' entity='client.admin' 2024-08-28T19:12:31.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:31 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:31.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:31 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:31.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:31 smithi115 ceph-mon[22184]: Reconfiguring mon.c (monmap changed)... 2024-08-28T19:12:31.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:31 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-28T19:12:31.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:31 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-28T19:12:31.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:31 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:31.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:31 smithi115 ceph-mon[22184]: Reconfiguring daemon mon.c on smithi115 2024-08-28T19:12:31.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:31 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1250177172' entity='client.admin' 2024-08-28T19:12:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.138+0000 7fd8d03d0640 1 -- 172.21.15.16:0/1795144383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c8108140 msgr2=0x7fd8c8108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.138+0000 7fd8d03d0640 1 --2- 172.21.15.16:0/1795144383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c8108140 0x7fd8c8108550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009900 tx=0x7fd8b802a410 comp rx=0 tx=0).stop 2024-08-28T19:12:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.138+0000 7fd8d03d0640 1 -- 172.21.15.16:0/1795144383 shutdown_connections 2024-08-28T19:12:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.138+0000 7fd8d03d0640 1 --2- 172.21.15.16:0/1795144383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c8108a90 0x7fd8c810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.138+0000 7fd8d03d0640 1 --2- 172.21.15.16:0/1795144383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c8108140 0x7fd8c8108550 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.138+0000 7fd8d03d0640 1 --2- 172.21.15.16:0/1795144383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c8102800 0x7fd8c8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.138+0000 7fd8d03d0640 1 -- 172.21.15.16:0/1795144383 >> 172.21.15.16:0/1795144383 conn(0x7fd8c80794d0 msgr2=0x7fd8c80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.138+0000 7fd8d03d0640 1 -- 172.21.15.16:0/1795144383 shutdown_connections 2024-08-28T19:12:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.138+0000 7fd8d03d0640 1 -- 172.21.15.16:0/1795144383 wait complete. 2024-08-28T19:12:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.138+0000 7fd8d03d0640 1 Processor -- start 2024-08-28T19:12:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8d03d0640 1 -- start start 2024-08-28T19:12:32.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8d03d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c8102800 0x7fd8c819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:32.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8d03d0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c8108140 0x7fd8c819a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8d03d0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c8108a90 0x7fd8c819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8d03d0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c8112730 con 0x7fd8c8108a90 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8d03d0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c81125b0 con 0x7fd8c8102800 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8d03d0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c81128b0 con 0x7fd8c8108140 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8ce96d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c8108a90 0x7fd8c819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8ce96d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c8108a90 0x7fd8c819faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39948/0 (socket says 172.21.15.16:39948) 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8ce96d640 1 -- 172.21.15.16:0/2989530219 learned_addr learned my addr 172.21.15.16:0/2989530219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8ce16c640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c8102800 0x7fd8c819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8cd96b640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c8108140 0x7fd8c819a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8ce96d640 1 -- 172.21.15.16:0/2989530219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c8108140 msgr2=0x7fd8c819a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8ce96d640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c8108140 0x7fd8c819a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8ce96d640 1 -- 172.21.15.16:0/2989530219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c8102800 msgr2=0x7fd8c819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8ce96d640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c8102800 0x7fd8c819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.139+0000 7fd8ce96d640 1 -- 172.21.15.16:0/2989530219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c81a0220 con 0x7fd8c8108a90 2024-08-28T19:12:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.140+0000 7fd8ce96d640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c8108a90 0x7fd8c819faf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd8c000eaa0 tx=0x7fd8c000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.140+0000 7fd8bf7fe640 1 -- 172.21.15.16:0/2989530219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c000cdb0 con 0x7fd8c8108a90 2024-08-28T19:12:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.140+0000 7fd8d03d0640 1 -- 172.21.15.16:0/2989530219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c81a0510 con 0x7fd8c8108a90 2024-08-28T19:12:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.140+0000 7fd8bf7fe640 1 -- 172.21.15.16:0/2989530219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8c0005020 con 0x7fd8c8108a90 2024-08-28T19:12:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.140+0000 7fd8bf7fe640 1 -- 172.21.15.16:0/2989530219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c00116d0 con 0x7fd8c8108a90 2024-08-28T19:12:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.140+0000 7fd8d03d0640 1 -- 172.21.15.16:0/2989530219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c81a6c70 con 0x7fd8c8108a90 2024-08-28T19:12:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.141+0000 7fd8d03d0640 1 -- 172.21.15.16:0/2989530219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c8102c10 con 0x7fd8c8108a90 2024-08-28T19:12:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.143+0000 7fd8bf7fe640 1 -- 172.21.15.16:0/2989530219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7fd8c00054a0 con 0x7fd8c8108a90 2024-08-28T19:12:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.144+0000 7fd8bf7fe640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8b40b4af0 0x7fd8b40b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.144+0000 7fd8ce16c640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8b40b4af0 0x7fd8b40b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.144+0000 7fd8bf7fe640 1 -- 172.21.15.16:0/2989530219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fd8c00e3a20 con 0x7fd8c8108a90 2024-08-28T19:12:32.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.144+0000 7fd8ce16c640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8b40b4af0 0x7fd8b40b7000 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd8c40058c0 tx=0x7fd8c400a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:32.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.147+0000 7fd8bf7fe640 1 -- 172.21.15.16:0/2989530219 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fd8c0016030 con 0x7fd8c8108a90 2024-08-28T19:12:32.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.335+0000 7fd8d03d0640 1 -- 172.21.15.16:0/2989530219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7fd8c819b3e0 con 0x7fd8c8108a90 2024-08-28T19:12:32.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.339+0000 7fd8bf7fe640 1 -- 172.21.15.16:0/2989530219 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v32)=0 v32) ==== 129+0+0 (secure 0 0 0) 0x7fd8c819b3e0 con 0x7fd8c8108a90 2024-08-28T19:12:32.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.341+0000 7fd8d03d0640 1 -- 172.21.15.16:0/2989530219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8b40b4af0 msgr2=0x7fd8b40b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:32.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.341+0000 7fd8d03d0640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8b40b4af0 0x7fd8b40b7000 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd8c40058c0 tx=0x7fd8c400a2d0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.341+0000 7fd8d03d0640 1 -- 172.21.15.16:0/2989530219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c8108a90 msgr2=0x7fd8c819faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:32.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.341+0000 7fd8d03d0640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c8108a90 0x7fd8c819faf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd8c000eaa0 tx=0x7fd8c000ef60 comp rx=0 tx=0).stop 2024-08-28T19:12:32.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.342+0000 7fd8d03d0640 1 -- 172.21.15.16:0/2989530219 shutdown_connections 2024-08-28T19:12:32.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.342+0000 7fd8d03d0640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c8108a90 0x7fd8c819faf0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.342+0000 7fd8d03d0640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c8108140 0x7fd8c819a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.342+0000 7fd8d03d0640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8b40b4af0 0x7fd8b40b7000 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.342+0000 7fd8d03d0640 1 --2- 172.21.15.16:0/2989530219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c8102800 0x7fd8c819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.342+0000 7fd8d03d0640 1 -- 172.21.15.16:0/2989530219 >> 172.21.15.16:0/2989530219 conn(0x7fd8c80794d0 msgr2=0x7fd8c80ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:32.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.342+0000 7fd8d03d0640 1 -- 172.21.15.16:0/2989530219 shutdown_connections 2024-08-28T19:12:32.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.342+0000 7fd8d03d0640 1 -- 172.21.15.16:0/2989530219 wait complete. 2024-08-28T19:12:32.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.849+0000 7efebbae5640 1 -- 172.21.15.16:0/2528141693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeb4102880 msgr2=0x7efeb4102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:32.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.849+0000 7efebbae5640 1 --2- 172.21.15.16:0/2528141693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeb4102880 0x7efeb4102c90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009900 tx=0x7efeb002a410 comp rx=0 tx=0).stop 2024-08-28T19:12:32.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.849+0000 7efebbae5640 1 -- 172.21.15.16:0/2528141693 shutdown_connections 2024-08-28T19:12:32.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.849+0000 7efebbae5640 1 --2- 172.21.15.16:0/2528141693 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeb4108b10 0x7efeb410f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.849+0000 7efebbae5640 1 --2- 172.21.15.16:0/2528141693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeb41081c0 0x7efeb41085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.849+0000 7efebbae5640 1 --2- 172.21.15.16:0/2528141693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeb4102880 0x7efeb4102c90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.849+0000 7efebbae5640 1 -- 172.21.15.16:0/2528141693 >> 172.21.15.16:0/2528141693 conn(0x7efeb4079510 msgr2=0x7efeb4079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:32.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.849+0000 7efebbae5640 1 -- 172.21.15.16:0/2528141693 shutdown_connections 2024-08-28T19:12:32.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.850+0000 7efebbae5640 1 -- 172.21.15.16:0/2528141693 wait complete. 2024-08-28T19:12:32.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.850+0000 7efebbae5640 1 Processor -- start 2024-08-28T19:12:32.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.850+0000 7efebbae5640 1 -- start start 2024-08-28T19:12:32.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.850+0000 7efebbae5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeb4102880 0x7efeb419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:32.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.850+0000 7efebbae5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeb41081c0 0x7efeb419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:32.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efebbae5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeb4108b10 0x7efeb419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:32.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efebbae5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41127c0 con 0x7efeb4102880 2024-08-28T19:12:32.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efebbae5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efeb4112640 con 0x7efeb4108b10 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efebbae5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efeb4112940 con 0x7efeb41081c0 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeb9881640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeb4102880 0x7efeb419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeb9881640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeb4102880 0x7efeb419a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39966/0 (socket says 172.21.15.16:39966) 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeb9881640 1 -- 172.21.15.16:0/2014143304 learned_addr learned my addr 172.21.15.16:0/2014143304 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeba082640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeb4108b10 0x7efeb419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeb9080640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeb41081c0 0x7efeb419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeb9881640 1 -- 172.21.15.16:0/2014143304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeb41081c0 msgr2=0x7efeb419a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeb9881640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeb41081c0 0x7efeb419a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeb9881640 1 -- 172.21.15.16:0/2014143304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeb4108b10 msgr2=0x7efeb419fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeb9881640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeb4108b10 0x7efeb419fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeb9881640 1 -- 172.21.15.16:0/2014143304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb41a0170 con 0x7efeb4102880 2024-08-28T19:12:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeba082640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeb4108b10 0x7efeb419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:12:32.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeb9080640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeb41081c0 0x7efeb419a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:12:32.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeb9881640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeb4102880 0x7efeb419a2c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efeb0005940 tx=0x7efeb00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:32.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeaaffd640 1 -- 172.21.15.16:0/2014143304 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb003f030 con 0x7efeb4102880 2024-08-28T19:12:32.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efebbae5640 1 -- 172.21.15.16:0/2014143304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb41a0400 con 0x7efeb4102880 2024-08-28T19:12:32.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeaaffd640 1 -- 172.21.15.16:0/2014143304 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efeb0003cb0 con 0x7efeb4102880 2024-08-28T19:12:32.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efeaaffd640 1 -- 172.21.15.16:0/2014143304 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb0035070 con 0x7efeb4102880 2024-08-28T19:12:32.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.851+0000 7efebbae5640 1 -- 172.21.15.16:0/2014143304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb41a08e0 con 0x7efeb4102880 2024-08-28T19:12:32.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.853+0000 7efebbae5640 1 -- 172.21.15.16:0/2014143304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeb4103b00 con 0x7efeb4102880 2024-08-28T19:12:32.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.854+0000 7efeaaffd640 1 -- 172.21.15.16:0/2014143304 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7efeb0042030 con 0x7efeb4102880 2024-08-28T19:12:32.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.855+0000 7efeaaffd640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efea00b4b20 0x7efea00b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:32.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.855+0000 7efeb9080640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efea00b4b20 0x7efea00b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:32.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.855+0000 7efeaaffd640 1 -- 172.21.15.16:0/2014143304 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7efeb0103b50 con 0x7efeb4102880 2024-08-28T19:12:32.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.855+0000 7efeb9080640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efea00b4b20 0x7efea00b7030 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efea4005d00 tx=0x7efea4005ed0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:32.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:32.859+0000 7efeaaffd640 1 -- 172.21.15.16:0/2014143304 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7efeb00cc880 con 0x7efeb4102880 2024-08-28T19:12:33.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:33.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:33.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:33.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:33.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2989530219' entity='client.admin' 2024-08-28T19:12:33.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:33.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:33.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:33.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.053+0000 7efebbae5640 1 -- 172.21.15.16:0/2014143304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7efeb4102c90 con 0x7efeb4102880 2024-08-28T19:12:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.057+0000 7efeaaffd640 1 -- 172.21.15.16:0/2014143304 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v33)=0 v33) ==== 120+0+0 (secure 0 0 0) 0x7efeb00cc180 con 0x7efeb4102880 2024-08-28T19:12:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.060+0000 7efebbae5640 1 -- 172.21.15.16:0/2014143304 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efea00b4b20 msgr2=0x7efea00b7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.060+0000 7efebbae5640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efea00b4b20 0x7efea00b7030 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efea4005d00 tx=0x7efea4005ed0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.060+0000 7efebbae5640 1 -- 172.21.15.16:0/2014143304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeb4102880 msgr2=0x7efeb419a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.060+0000 7efebbae5640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeb4102880 0x7efeb419a2c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efeb0005940 tx=0x7efeb00042f0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.060+0000 7efebbae5640 1 -- 172.21.15.16:0/2014143304 shutdown_connections 2024-08-28T19:12:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.060+0000 7efebbae5640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efea00b4b20 0x7efea00b7030 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.060+0000 7efebbae5640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeb4108b10 0x7efeb419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.060+0000 7efebbae5640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeb41081c0 0x7efeb419a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.060+0000 7efebbae5640 1 --2- 172.21.15.16:0/2014143304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeb4102880 0x7efeb419a2c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.061+0000 7efebbae5640 1 -- 172.21.15.16:0/2014143304 >> 172.21.15.16:0/2014143304 conn(0x7efeb4079510 msgr2=0x7efeb41018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.061+0000 7efebbae5640 1 -- 172.21.15.16:0/2014143304 shutdown_connections 2024-08-28T19:12:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.061+0000 7efebbae5640 1 -- 172.21.15.16:0/2014143304 wait complete. 2024-08-28T19:12:33.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.260+0000 7f6524c4f640 1 -- 172.21.15.16:0/2749090606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6520107900 msgr2=0x7f6520107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:33.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.260+0000 7f6524c4f640 1 --2- 172.21.15.16:0/2749090606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6520107900 0x7f6520107d10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6514009900 tx=0x7f651402a410 comp rx=0 tx=0).stop 2024-08-28T19:12:33.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.260+0000 7f6524c4f640 1 -- 172.21.15.16:0/2749090606 shutdown_connections 2024-08-28T19:12:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.260+0000 7f6524c4f640 1 --2- 172.21.15.16:0/2749090606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6520108250 0x7f652010ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.260+0000 7f6524c4f640 1 --2- 172.21.15.16:0/2749090606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6520107900 0x7f6520107d10 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.260+0000 7f6524c4f640 1 --2- 172.21.15.16:0/2749090606 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6520101fc0 0x7f65201023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.260+0000 7f6524c4f640 1 -- 172.21.15.16:0/2749090606 >> 172.21.15.16:0/2749090606 conn(0x7f6520079bd0 msgr2=0x7f6520079fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.261+0000 7f6524c4f640 1 -- 172.21.15.16:0/2749090606 shutdown_connections 2024-08-28T19:12:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.261+0000 7f6524c4f640 1 -- 172.21.15.16:0/2749090606 wait complete. 2024-08-28T19:12:33.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.261+0000 7f6524c4f640 1 Processor -- start 2024-08-28T19:12:33.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.261+0000 7f6524c4f640 1 -- start start 2024-08-28T19:12:33.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f6524c4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6520101fc0 0x7f6520110370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:33.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f6524c4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6520107900 0x7f65201108b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:33.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f6524c4f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6520108250 0x7f6520110df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:33.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f6524c4f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6520113880 con 0x7f6520107900 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f6524c4f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6520113700 con 0x7f6520101fc0 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f6524c4f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6520113a00 con 0x7f6520108250 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f651dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6520107900 0x7f65201108b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f651dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6520107900 0x7f65201108b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39992/0 (socket says 172.21.15.16:39992) 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f651dd9b640 1 -- 172.21.15.16:0/3342208718 learned_addr learned my addr 172.21.15.16:0/3342208718 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f651e59c640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6520101fc0 0x7f6520110370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f651ed9d640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6520108250 0x7f6520110df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f651dd9b640 1 -- 172.21.15.16:0/3342208718 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6520108250 msgr2=0x7f6520110df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f651dd9b640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6520108250 0x7f6520110df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f651dd9b640 1 -- 172.21.15.16:0/3342208718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6520101fc0 msgr2=0x7f6520110370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f651dd9b640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6520101fc0 0x7f6520110370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f651dd9b640 1 -- 172.21.15.16:0/3342208718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65201a8530 con 0x7f6520107900 2024-08-28T19:12:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.262+0000 7f651dd9b640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6520107900 0x7f65201108b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f651402a930 tx=0x7f6514004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:33.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.263+0000 7f650f7fe640 1 -- 172.21.15.16:0/3342208718 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651403f030 con 0x7f6520107900 2024-08-28T19:12:33.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.263+0000 7f6524c4f640 1 -- 172.21.15.16:0/3342208718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65201a87c0 con 0x7f6520107900 2024-08-28T19:12:33.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.263+0000 7f650f7fe640 1 -- 172.21.15.16:0/3342208718 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6514003aa0 con 0x7f6520107900 2024-08-28T19:12:33.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.263+0000 7f650f7fe640 1 -- 172.21.15.16:0/3342208718 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514035070 con 0x7f6520107900 2024-08-28T19:12:33.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.263+0000 7f6524c4f640 1 -- 172.21.15.16:0/3342208718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65201a8ca0 con 0x7f6520107900 2024-08-28T19:12:33.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.264+0000 7f6524c4f640 1 -- 172.21.15.16:0/3342208718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65201023d0 con 0x7f6520107900 2024-08-28T19:12:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.265+0000 7f650f7fe640 1 -- 172.21.15.16:0/3342208718 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7f6514042030 con 0x7f6520107900 2024-08-28T19:12:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.266+0000 7f650f7fe640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65080b4af0 0x7f65080b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.266+0000 7f651e59c640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65080b4af0 0x7f65080b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.266+0000 7f650f7fe640 1 -- 172.21.15.16:0/3342208718 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f6514104130 con 0x7f6520107900 2024-08-28T19:12:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.267+0000 7f651e59c640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65080b4af0 0x7f65080b7000 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6520111dc0 tx=0x7f651000b410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.269+0000 7f650f7fe640 1 -- 172.21.15.16:0/3342208718 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f65140ccdc0 con 0x7f6520107900 2024-08-28T19:12:33.280 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2989530219' entity='client.admin' 2024-08-28T19:12:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:33.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:33.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2989530219' entity='client.admin' 2024-08-28T19:12:33.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:33.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:33.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:33.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.462+0000 7f6524c4f640 1 -- 172.21.15.16:0/3342208718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f6520111670 con 0x7f6520107900 2024-08-28T19:12:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.466+0000 7f650f7fe640 1 -- 172.21.15.16:0/3342208718 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v34)=0 v34) ==== 133+0+0 (secure 0 0 0) 0x7f65140cc6c0 con 0x7f6520107900 2024-08-28T19:12:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 -- 172.21.15.16:0/3342208718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65080b4af0 msgr2=0x7f65080b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65080b4af0 0x7f65080b7000 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6520111dc0 tx=0x7f651000b410 comp rx=0 tx=0).stop 2024-08-28T19:12:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 -- 172.21.15.16:0/3342208718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6520107900 msgr2=0x7f65201108b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6520107900 0x7f65201108b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f651402a930 tx=0x7f6514004270 comp rx=0 tx=0).stop 2024-08-28T19:12:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 -- 172.21.15.16:0/3342208718 shutdown_connections 2024-08-28T19:12:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6520108250 0x7f6520110df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6520107900 0x7f65201108b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65080b4af0 0x7f65080b7000 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 --2- 172.21.15.16:0/3342208718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6520101fc0 0x7f6520110370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:33.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 -- 172.21.15.16:0/3342208718 >> 172.21.15.16:0/3342208718 conn(0x7f6520079bd0 msgr2=0x7f65201059b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:33.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 -- 172.21.15.16:0/3342208718 shutdown_connections 2024-08-28T19:12:33.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:33.469+0000 7f6524c4f640 1 -- 172.21.15.16:0/3342208718 wait complete. 2024-08-28T19:12:33.715 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi016') 2024-08-28T19:12:33.715 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi016') 2024-08-28T19:12:33.715 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi016') 2024-08-28T19:12:33.715 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi016') 2024-08-28T19:12:33.716 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi016') 2024-08-28T19:12:33.716 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi016') 2024-08-28T19:12:33.716 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi016') 2024-08-28T19:12:34.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:34 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2014143304' entity='client.admin' 2024-08-28T19:12:34.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:34 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3342208718' entity='client.admin' 2024-08-28T19:12:34.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:34 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:34.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:34 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:34.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:34 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:34.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:34 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:34 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2014143304' entity='client.admin' 2024-08-28T19:12:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:34 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3342208718' entity='client.admin' 2024-08-28T19:12:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:34 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:34 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:34 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:34 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:34.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:34 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2014143304' entity='client.admin' 2024-08-28T19:12:34.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:34 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3342208718' entity='client.admin' 2024-08-28T19:12:34.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:34 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:34.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:34 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:34.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:34 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:12:34.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:34 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:35.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:35 smithi115 ceph-mon[22184]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:35.475 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:35 smithi016 ceph-mon[18556]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:35 smithi028 ceph-mon[21769]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:36.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.006+0000 7f2e5a81b640 1 -- 172.21.15.16:0/1540693492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e5410ffa0 msgr2=0x7f2e541123c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:36.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.006+0000 7f2e5a81b640 1 --2- 172.21.15.16:0/1540693492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e5410ffa0 0x7f2e541123c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009ca0 tx=0x7f2e4c02a950 comp rx=0 tx=0).stop 2024-08-28T19:12:36.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.006+0000 7f2e5a81b640 1 -- 172.21.15.16:0/1540693492 shutdown_connections 2024-08-28T19:12:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.006+0000 7f2e5a81b640 1 --2- 172.21.15.16:0/1540693492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e5410ffa0 0x7f2e541123c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.006+0000 7f2e5a81b640 1 --2- 172.21.15.16:0/1540693492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e54102d40 0x7f2e5410fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.006+0000 7f2e5a81b640 1 --2- 172.21.15.16:0/1540693492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e541023f0 0x7f2e54102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.006+0000 7f2e5a81b640 1 -- 172.21.15.16:0/1540693492 >> 172.21.15.16:0/1540693492 conn(0x7f2e54079510 msgr2=0x7f2e54079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.006+0000 7f2e5a81b640 1 -- 172.21.15.16:0/1540693492 shutdown_connections 2024-08-28T19:12:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.006+0000 7f2e5a81b640 1 -- 172.21.15.16:0/1540693492 wait complete. 2024-08-28T19:12:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.007+0000 7f2e5a81b640 1 Processor -- start 2024-08-28T19:12:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.007+0000 7f2e5a81b640 1 -- start start 2024-08-28T19:12:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.007+0000 7f2e5a81b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e541023f0 0x7f2e5419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.007+0000 7f2e5a81b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e54102d40 0x7f2e5419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.007+0000 7f2e5a81b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e5410ffa0 0x7f2e541a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.007+0000 7f2e5a81b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54115a90 con 0x7f2e5410ffa0 2024-08-28T19:12:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.007+0000 7f2e5a81b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54115910 con 0x7f2e54102d40 2024-08-28T19:12:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.007+0000 7f2e5a81b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54115c10 con 0x7f2e541023f0 2024-08-28T19:12:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e58db8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e5410ffa0 0x7f2e541a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e54102d40 0x7f2e5419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e58db8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e5410ffa0 0x7f2e541a1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46878/0 (socket says 172.21.15.16:46878) 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e58db8640 1 -- 172.21.15.16:0/223500792 learned_addr learned my addr 172.21.15.16:0/223500792 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e53fff640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e541023f0 0x7f2e5419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e58db8640 1 -- 172.21.15.16:0/223500792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e541023f0 msgr2=0x7f2e5419c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e58db8640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e541023f0 0x7f2e5419c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e58db8640 1 -- 172.21.15.16:0/223500792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e54102d40 msgr2=0x7f2e5419c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e58db8640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e54102d40 0x7f2e5419c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e58db8640 1 -- 172.21.15.16:0/223500792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e541a22d0 con 0x7f2e5410ffa0 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e537fe640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e54102d40 0x7f2e5419c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e58db8640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e5410ffa0 0x7f2e541a1c60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c02ae70 tx=0x7f2e4c02f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e517fa640 1 -- 172.21.15.16:0/223500792 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c034040 con 0x7f2e5410ffa0 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e517fa640 1 -- 172.21.15.16:0/223500792 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e4c0308f0 con 0x7f2e5410ffa0 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.008+0000 7f2e5a81b640 1 -- 172.21.15.16:0/223500792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e541a2560 con 0x7f2e5410ffa0 2024-08-28T19:12:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.009+0000 7f2e517fa640 1 -- 172.21.15.16:0/223500792 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c004720 con 0x7f2e5410ffa0 2024-08-28T19:12:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.009+0000 7f2e5a81b640 1 -- 172.21.15.16:0/223500792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e541a2a40 con 0x7f2e5410ffa0 2024-08-28T19:12:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.010+0000 7f2e5a81b640 1 -- 172.21.15.16:0/223500792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e5406c000 con 0x7f2e5410ffa0 2024-08-28T19:12:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.011+0000 7f2e517fa640 1 -- 172.21.15.16:0/223500792 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7f2e4c037020 con 0x7f2e5410ffa0 2024-08-28T19:12:36.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.012+0000 7f2e517fa640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2e3c0b4b20 0x7f2e3c0b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:36.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.012+0000 7f2e53fff640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2e3c0b4b20 0x7f2e3c0b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:36.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.012+0000 7f2e517fa640 1 -- 172.21.15.16:0/223500792 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f2e4c104350 con 0x7f2e5410ffa0 2024-08-28T19:12:36.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.012+0000 7f2e53fff640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2e3c0b4b20 0x7f2e3c0b7030 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400bbe0 tx=0x7f2e4400b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.015+0000 7f2e517fa640 1 -- 172.21.15.16:0/223500792 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f2e4c0cd110 con 0x7f2e5410ffa0 2024-08-28T19:12:36.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.200+0000 7f2e5a81b640 1 -- 172.21.15.16:0/223500792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f2e54102800 con 0x7f2e5410ffa0 2024-08-28T19:12:36.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.205+0000 7f2e517fa640 1 -- 172.21.15.16:0/223500792 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v35)=0 v35) ==== 119+0+0 (secure 0 0 0) 0x7f2e4c0cca10 con 0x7f2e5410ffa0 2024-08-28T19:12:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 -- 172.21.15.16:0/223500792 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2e3c0b4b20 msgr2=0x7f2e3c0b7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2e3c0b4b20 0x7f2e3c0b7030 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400bbe0 tx=0x7f2e4400b390 comp rx=0 tx=0).stop 2024-08-28T19:12:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 -- 172.21.15.16:0/223500792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e5410ffa0 msgr2=0x7f2e541a1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e5410ffa0 0x7f2e541a1c60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c02ae70 tx=0x7f2e4c02f5d0 comp rx=0 tx=0).stop 2024-08-28T19:12:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 -- 172.21.15.16:0/223500792 shutdown_connections 2024-08-28T19:12:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e5410ffa0 0x7f2e541a1c60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2e3c0b4b20 0x7f2e3c0b7030 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e54102d40 0x7f2e5419c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 --2- 172.21.15.16:0/223500792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e541023f0 0x7f2e5419c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 -- 172.21.15.16:0/223500792 >> 172.21.15.16:0/223500792 conn(0x7f2e54079510 msgr2=0x7f2e54109ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 -- 172.21.15.16:0/223500792 shutdown_connections 2024-08-28T19:12:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:36.207+0000 7f2e5a81b640 1 -- 172.21.15.16:0/223500792 wait complete. 2024-08-28T19:12:36.751 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi016') 2024-08-28T19:12:36.751 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi016') 2024-08-28T19:12:36.751 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi016') 2024-08-28T19:12:36.751 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'debug_cephsqlite', '20'], check_status=True, hostname='smithi016') 2024-08-28T19:12:36.751 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/allow_m_granularity', 'True'], check_status=True, hostname='smithi016') 2024-08-28T19:12:36.751 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/dump_on_update', 'True'], check_status=True, hostname='smithi016') 2024-08-28T19:12:36.752 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/log_level', 'debug'], check_status=True, hostname='smithi016') 2024-08-28T19:12:36.752 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi016') 2024-08-28T19:12:36.752 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi016') 2024-08-28T19:12:36.752 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'eabc438a-6570-11ef-bcd4-c7b262605968', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi016') 2024-08-28T19:12:36.752 INFO:tasks.cephadm:Deploying OSDs... 2024-08-28T19:12:36.753 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:12:36.753 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-08-28T19:12:36.769 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-28T19:12:36.769 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-08-28T19:12:36.825 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-28T19:12:36.825 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:36.825 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-08-28T19:12:36.825 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:36.825 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:36.825 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-08-28 19:11:19.640321987 +0000 2024-08-28T19:12:36.825 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-08-28 19:06:08.048262696 +0000 2024-08-28T19:12:36.825 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-08-28 19:06:08.048262696 +0000 2024-08-28T19:12:36.825 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-08-28 19:06:08.048262696 +0000 2024-08-28T19:12:36.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-28T19:12:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-08-28T19:12:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-08-28T19:12:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000107541 s, 4.8 MB/s 2024-08-28T19:12:36.892 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-28T19:12:36.948 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-08-28T19:12:37.003 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-28T19:12:37.003 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:37.003 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-08-28T19:12:37.003 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:37.004 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:37.004 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-08-28 19:11:19.668321530 +0000 2024-08-28T19:12:37.004 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-08-28 19:06:08.380257303 +0000 2024-08-28T19:12:37.004 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-08-28 19:06:08.380257303 +0000 2024-08-28T19:12:37.004 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-08-28 19:06:08.380257303 +0000 2024-08-28T19:12:37.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-28T19:12:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-08-28T19:12:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-08-28T19:12:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 8.41e-05 s, 6.1 MB/s 2024-08-28T19:12:37.068 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-28T19:12:37.124 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-08-28T19:12:37.180 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-28T19:12:37.180 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:37.180 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-08-28T19:12:37.180 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:37.180 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:37.180 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-08-28 19:11:19.695321090 +0000 2024-08-28T19:12:37.180 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-08-28 19:06:08.708251974 +0000 2024-08-28T19:12:37.180 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-08-28 19:06:08.708251974 +0000 2024-08-28T19:12:37.180 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-08-28 19:06:08.708251974 +0000 2024-08-28T19:12:37.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-28T19:12:37.243 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-08-28T19:12:37.243 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-08-28T19:12:37.243 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000107712 s, 4.8 MB/s 2024-08-28T19:12:37.244 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-28T19:12:37.300 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-08-28T19:12:37.355 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-28T19:12:37.355 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:37.355 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-08-28T19:12:37.355 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:37.355 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:37.355 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-08-28 19:11:19.723320633 +0000 2024-08-28T19:12:37.355 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-08-28 19:06:09.037246629 +0000 2024-08-28T19:12:37.355 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-08-28 19:06:09.037246629 +0000 2024-08-28T19:12:37.355 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-08-28 19:06:09.037246629 +0000 2024-08-28T19:12:37.356 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-28T19:12:37.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:37 smithi115 ceph-mon[22184]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:37.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:37 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/223500792' entity='client.admin' 2024-08-28T19:12:37.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:37 smithi016 ceph-mon[18556]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:37.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:37 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/223500792' entity='client.admin' 2024-08-28T19:12:37.418 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-08-28T19:12:37.418 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-08-28T19:12:37.418 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000101829 s, 5.0 MB/s 2024-08-28T19:12:37.419 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-28T19:12:37.476 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-08-28T19:12:37.476 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-08-28T19:12:37.495 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-28T19:12:37.496 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-08-28T19:12:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-28T19:12:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:37.555 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-08-28T19:12:37.555 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:37.555 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:37.555 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-08-28 19:12:15.993157908 +0000 2024-08-28T19:12:37.555 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-08-28 19:06:26.440774873 +0000 2024-08-28T19:12:37.555 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-08-28 19:06:26.440774873 +0000 2024-08-28T19:12:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-08-28 19:06:26.440774873 +0000 2024-08-28T19:12:37.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-28T19:12:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:37 smithi028 ceph-mon[21769]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:37 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/223500792' entity='client.admin' 2024-08-28T19:12:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-08-28T19:12:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-08-28T19:12:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000123798 s, 4.1 MB/s 2024-08-28T19:12:37.589 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-28T19:12:37.646 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-08-28T19:12:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-28T19:12:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:37.704 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-08-28T19:12:37.704 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:37.704 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:37.704 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-08-28 19:12:15.994157879 +0000 2024-08-28T19:12:37.704 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-08-28 19:06:26.751765546 +0000 2024-08-28T19:12:37.704 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-08-28 19:06:26.751765546 +0000 2024-08-28T19:12:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-08-28 19:06:26.751765546 +0000 2024-08-28T19:12:37.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-28T19:12:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-08-28T19:12:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-08-28T19:12:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000121987 s, 4.2 MB/s 2024-08-28T19:12:37.771 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-28T19:12:37.828 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-08-28T19:12:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-28T19:12:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:37.884 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-08-28T19:12:37.884 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:37.884 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:37.884 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-08-28 19:12:15.994157879 +0000 2024-08-28T19:12:37.884 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-08-28 19:06:27.056756400 +0000 2024-08-28T19:12:37.884 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-08-28 19:06:27.056756400 +0000 2024-08-28T19:12:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-08-28 19:06:27.056756400 +0000 2024-08-28T19:12:37.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-28T19:12:37.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-08-28T19:12:37.948 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-08-28T19:12:37.948 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000104394 s, 4.9 MB/s 2024-08-28T19:12:37.949 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-28T19:12:38.005 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-08-28T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-28T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-08-28T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-08-28 19:12:15.994157879 +0000 2024-08-28T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-08-28 19:06:27.367747073 +0000 2024-08-28T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-08-28 19:06:27.367747073 +0000 2024-08-28T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-08-28 19:06:27.367747073 +0000 2024-08-28T19:12:38.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-28T19:12:38.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-08-28T19:12:38.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-08-28T19:12:38.125 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000102398 s, 5.0 MB/s 2024-08-28T19:12:38.126 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-28T19:12:38.182 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-08-28T19:12:38.183 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2024-08-28T19:12:38.202 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-28T19:12:38.203 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2024-08-28T19:12:38.260 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-28T19:12:38.260 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:38.260 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-08-28T19:12:38.260 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:38.261 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:38.261 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-08-28 19:12:13.605344635 +0000 2024-08-28T19:12:38.261 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-08-28 19:06:26.212546444 +0000 2024-08-28T19:12:38.261 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-08-28 19:06:26.212546444 +0000 2024-08-28T19:12:38.261 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-08-28 19:06:26.212546444 +0000 2024-08-28T19:12:38.261 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-28T19:12:38.330 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-08-28T19:12:38.330 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-08-28T19:12:38.330 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000262751 s, 1.9 MB/s 2024-08-28T19:12:38.331 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-28T19:12:38.389 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2024-08-28T19:12:38.446 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-28T19:12:38.446 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:38.446 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-08-28T19:12:38.446 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:38.446 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:38.446 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-08-28 19:12:13.606344615 +0000 2024-08-28T19:12:38.446 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-08-28 19:06:26.546539637 +0000 2024-08-28T19:12:38.446 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-08-28 19:06:26.546539637 +0000 2024-08-28T19:12:38.447 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-08-28 19:06:26.546539637 +0000 2024-08-28T19:12:38.447 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-28T19:12:38.511 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-08-28T19:12:38.511 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-08-28T19:12:38.511 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000202314 s, 2.5 MB/s 2024-08-28T19:12:38.512 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-28T19:12:38.569 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2024-08-28T19:12:38.625 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-28T19:12:38.625 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:38.625 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-08-28T19:12:38.625 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:38.625 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:38.625 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-08-28 19:12:13.606344615 +0000 2024-08-28T19:12:38.625 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-08-28 19:06:26.876532911 +0000 2024-08-28T19:12:38.626 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-08-28 19:06:26.876532911 +0000 2024-08-28T19:12:38.626 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-08-28 19:06:26.876532911 +0000 2024-08-28T19:12:38.626 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-28T19:12:38.689 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-08-28T19:12:38.689 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-08-28T19:12:38.689 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.00020639 s, 2.5 MB/s 2024-08-28T19:12:38.690 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-28T19:12:38.746 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2024-08-28T19:12:38.801 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-28T19:12:38.802 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-28T19:12:38.802 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-08-28T19:12:38.802 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:12:38.802 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-08-28T19:12:38.802 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-08-28 19:12:13.606344615 +0000 2024-08-28T19:12:38.802 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-08-28 19:06:27.209526124 +0000 2024-08-28T19:12:38.802 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-08-28 19:06:27.209526124 +0000 2024-08-28T19:12:38.802 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-08-28 19:06:27.209526124 +0000 2024-08-28T19:12:38.802 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-28T19:12:38.864 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-08-28T19:12:38.864 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-08-28T19:12:38.864 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000207747 s, 2.5 MB/s 2024-08-28T19:12:38.865 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-28T19:12:38.922 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-08-28T19:12:38.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-28T19:12:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:39.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:39 smithi016 ceph-mon[18556]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:39 smithi115 ceph-mon[22184]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:39 smithi028 ceph-mon[21769]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:41.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:41 smithi016 ceph-mon[18556]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:41 smithi115 ceph-mon[22184]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:41.524 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:12:41.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-08-28T19:12:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:41 smithi028 ceph-mon[21769]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:12:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.240+0000 7f1569064640 1 -- 172.21.15.16:0/1785640690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564109240 msgr2=0x7f1564109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.240+0000 7f1569064640 1 --2- 172.21.15.16:0/1785640690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564109240 0x7f1564109650 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1558009900 tx=0x7f155802a3f0 comp rx=0 tx=0).stop 2024-08-28T19:12:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.240+0000 7f1569064640 1 -- 172.21.15.16:0/1785640690 shutdown_connections 2024-08-28T19:12:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.240+0000 7f1569064640 1 --2- 172.21.15.16:0/1785640690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1564109b90 0x7f15641103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.240+0000 7f1569064640 1 --2- 172.21.15.16:0/1785640690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564109240 0x7f1564109650 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.240+0000 7f1569064640 1 --2- 172.21.15.16:0/1785640690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564103900 0x7f1564103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.240+0000 7f1569064640 1 -- 172.21.15.16:0/1785640690 >> 172.21.15.16:0/1785640690 conn(0x7f156407b5f0 msgr2=0x7f156407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:12:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.240+0000 7f1569064640 1 -- 172.21.15.16:0/1785640690 shutdown_connections 2024-08-28T19:12:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.240+0000 7f1569064640 1 -- 172.21.15.16:0/1785640690 wait complete. 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.241+0000 7f1569064640 1 Processor -- start 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1569064640 1 -- start start 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1569064640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564103900 0x7f156419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1569064640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564109240 0x7f156419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1569064640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1564109b90 0x7f156419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1569064640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15641138a0 con 0x7f1564103900 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1569064640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1564113720 con 0x7f1564109240 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1569064640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1564113a20 con 0x7f1564109b90 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1562d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564103900 0x7f156419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1562d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564103900 0x7f156419a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46898/0 (socket says 172.21.15.16:46898) 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1562d9d640 1 -- 172.21.15.16:0/3441931379 learned_addr learned my addr 172.21.15.16:0/3441931379 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f156259c640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564109240 0x7f156419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f156359e640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1564109b90 0x7f156419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1562d9d640 1 -- 172.21.15.16:0/3441931379 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1564109b90 msgr2=0x7f156419fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1562d9d640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1564109b90 0x7f156419fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1562d9d640 1 -- 172.21.15.16:0/3441931379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564109240 msgr2=0x7f156419a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1562d9d640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564109240 0x7f156419a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f1562d9d640 1 -- 172.21.15.16:0/3441931379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15641a0120 con 0x7f1564103900 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.242+0000 7f156359e640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1564109b90 0x7f156419fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.243+0000 7f1562d9d640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564103900 0x7f156419a270 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f155400c920 tx=0x7f155400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.243+0000 7f154bfff640 1 -- 172.21.15.16:0/3441931379 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554007c70 con 0x7f1564103900 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.243+0000 7f154bfff640 1 -- 172.21.15.16:0/3441931379 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1554007e10 con 0x7f1564103900 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.243+0000 7f1569064640 1 -- 172.21.15.16:0/3441931379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15641a0410 con 0x7f1564103900 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.243+0000 7f154bfff640 1 -- 172.21.15.16:0/3441931379 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554018820 con 0x7f1564103900 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.243+0000 7f1569064640 1 -- 172.21.15.16:0/3441931379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15641a6c30 con 0x7f1564103900 2024-08-28T19:12:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.244+0000 7f1569064640 1 -- 172.21.15.16:0/3441931379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1564103d10 con 0x7f1564103900 2024-08-28T19:12:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.246+0000 7f154bfff640 1 -- 172.21.15.16:0/3441931379 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7f155400f450 con 0x7f1564103900 2024-08-28T19:12:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.246+0000 7f154bfff640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f154c0b4b20 0x7f154c0b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:12:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.246+0000 7f156259c640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f154c0b4b20 0x7f154c0b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:12:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.246+0000 7f154bfff640 1 -- 172.21.15.16:0/3441931379 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f15540e3930 con 0x7f1564103900 2024-08-28T19:12:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.247+0000 7f156259c640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f154c0b4b20 0x7f154c0b7030 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f155802a9e0 tx=0x7f1558005a20 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:12:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.249+0000 7f154bfff640 1 -- 172.21.15.16:0/3441931379 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f1554014030 con 0x7f1564103900 2024-08-28T19:12:42.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:12:42.445+0000 7f1569064640 1 -- 172.21.15.16:0/3441931379 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f15640635a0 con 0x7f154c0b4b20 2024-08-28T19:12:43.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:43 smithi016 ceph-mon[18556]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:43.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:43 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:12:43.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:43 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:12:43.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:43 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:43.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:43 smithi115 ceph-mon[22184]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:43.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:43 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:12:43.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:43 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:12:43.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:43 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:43 smithi028 ceph-mon[21769]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:43 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:12:43.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:43 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:12:43.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:43 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:44.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:44 smithi016 ceph-mon[18556]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:12:44.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:44 smithi115 ceph-mon[22184]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:12:44.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:44 smithi028 ceph-mon[21769]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:12:45.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:45 smithi016 ceph-mon[18556]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:45.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:45 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2948694136' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7a5e8a2d-958b-4c1f-b2be-90743c629505"} : dispatch 2024-08-28T19:12:45.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:45 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2948694136' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a5e8a2d-958b-4c1f-b2be-90743c629505"}]': finished 2024-08-28T19:12:45.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:45 smithi016 ceph-mon[18556]: osdmap e6: 1 total, 0 up, 1 in 2024-08-28T19:12:45.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:12:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:45 smithi115 ceph-mon[22184]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:45 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2948694136' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7a5e8a2d-958b-4c1f-b2be-90743c629505"} : dispatch 2024-08-28T19:12:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:45 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2948694136' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a5e8a2d-958b-4c1f-b2be-90743c629505"}]': finished 2024-08-28T19:12:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:45 smithi115 ceph-mon[22184]: osdmap e6: 1 total, 0 up, 1 in 2024-08-28T19:12:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:12:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:45 smithi028 ceph-mon[21769]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:45 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2948694136' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7a5e8a2d-958b-4c1f-b2be-90743c629505"} : dispatch 2024-08-28T19:12:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:45 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2948694136' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a5e8a2d-958b-4c1f-b2be-90743c629505"}]': finished 2024-08-28T19:12:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:45 smithi028 ceph-mon[21769]: osdmap e6: 1 total, 0 up, 1 in 2024-08-28T19:12:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:12:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:46 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1178693217' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:12:46.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:46 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1178693217' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:12:46.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:46 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1178693217' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:12:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:47 smithi115 ceph-mon[22184]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:47.460 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:47 smithi028 ceph-mon[21769]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:47.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:47 smithi016 ceph-mon[18556]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:49.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:49 smithi115 ceph-mon[22184]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:49.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:49 smithi016 ceph-mon[18556]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:49.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:49 smithi028 ceph-mon[21769]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:51.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:51 smithi016 ceph-mon[18556]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:51.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:51 smithi115 ceph-mon[22184]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:51.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:51 smithi028 ceph-mon[21769]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:53 smithi016 ceph-mon[18556]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:53.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-28T19:12:53.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:53 smithi115 ceph-mon[22184]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-28T19:12:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:53.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:53 smithi028 ceph-mon[21769]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:53.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-28T19:12:53.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:12:54.313 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:54 smithi016 ceph-mon[18556]: Deploying daemon osd.0 on smithi016 2024-08-28T19:12:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:54 smithi115 ceph-mon[22184]: Deploying daemon osd.0 on smithi016 2024-08-28T19:12:54.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:54 smithi028 ceph-mon[21769]: Deploying daemon osd.0 on smithi016 2024-08-28T19:12:55.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:55 smithi016 ceph-mon[18556]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:55 smithi115 ceph-mon[22184]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:55 smithi028 ceph-mon[21769]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:57.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:57 smithi016 ceph-mon[18556]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:57.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:57 smithi115 ceph-mon[22184]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:57.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:57 smithi028 ceph-mon[21769]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:58.229 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:58 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:58.229 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:58 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:58.229 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:58 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:58 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:58 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:58 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:58.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:58 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:12:58.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:58 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:58.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:58 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:12:59.246 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:12:59 smithi016 ceph-mon[18556]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:59.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:12:59 smithi115 ceph-mon[22184]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:12:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:12:59 smithi028 ceph-mon[21769]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:00 smithi115 ceph-mon[22184]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:01.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:00 smithi016 ceph-mon[18556]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:01.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:01.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:01.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:01.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:01.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:00 smithi028 ceph-mon[21769]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:01.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:01.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:01.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:01.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:01.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.510+0000 7f154bfff640 1 -- 172.21.15.16:0/3441931379 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f15640635a0 con 0x7f154c0b4b20 2024-08-28T19:13:01.512 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-08-28T19:13:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 -- 172.21.15.16:0/3441931379 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f154c0b4b20 msgr2=0x7f154c0b7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f154c0b4b20 0x7f154c0b7030 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f155802a9e0 tx=0x7f1558005a20 comp rx=0 tx=0).stop 2024-08-28T19:13:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 -- 172.21.15.16:0/3441931379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564103900 msgr2=0x7f156419a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564103900 0x7f156419a270 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f155400c920 tx=0x7f155400cde0 comp rx=0 tx=0).stop 2024-08-28T19:13:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 -- 172.21.15.16:0/3441931379 shutdown_connections 2024-08-28T19:13:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1564109b90 0x7f156419fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f154c0b4b20 0x7f154c0b7030 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564109240 0x7f156419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 --2- 172.21.15.16:0/3441931379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564103900 0x7f156419a270 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 -- 172.21.15.16:0/3441931379 >> 172.21.15.16:0/3441931379 conn(0x7f156407b5f0 msgr2=0x7f15641072f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:13:01.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 -- 172.21.15.16:0/3441931379 shutdown_connections 2024-08-28T19:13:01.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:01.513+0000 7f1569064640 1 -- 172.21.15.16:0/3441931379 wait complete. 2024-08-28T19:13:02.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:01 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:02.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:01 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:02.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:01 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:02.109 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.0.service 2024-08-28T19:13:02.111 INFO:tasks.cephadm:Deploying osd.1 on smithi028 with /dev/vg_nvme/lv_4... 2024-08-28T19:13:02.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-28T19:13:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:01 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:01 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:01 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:02.140 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:01 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:02.140 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:01 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:02.140 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:01 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:02.372 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:13:03.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:02 smithi016 ceph-mon[18556]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:03.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:02 smithi028 ceph-mon[21769]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:03.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:02 smithi115 ceph-mon[22184]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:04.251 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:13:03 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0[36367]: 2024-08-28T19:13:03.831+0000 7fd056ed1640 -1 osd.0 0 log_to_monitors true 2024-08-28T19:13:04.883 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:04 smithi028 ceph-mon[21769]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:04.883 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:04 smithi028 ceph-mon[21769]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-28T19:13:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:13:04.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-08-28T19:13:05.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:04 smithi115 ceph-mon[22184]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:05.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:04 smithi115 ceph-mon[22184]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-28T19:13:05.212 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:13:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:04 smithi016 ceph-mon[18556]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:04 smithi016 ceph-mon[18556]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-28T19:13:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.695+0000 7f0af8990640 1 -- 172.21.15.28:0/2621878996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af4109240 msgr2=0x7f0af4109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.695+0000 7f0af8990640 1 --2- 172.21.15.28:0/2621878996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af4109240 0x7f0af4109650 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009900 tx=0x7f0adc02a410 comp rx=0 tx=0).stop 2024-08-28T19:13:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.695+0000 7f0af8990640 1 -- 172.21.15.28:0/2621878996 shutdown_connections 2024-08-28T19:13:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.695+0000 7f0af8990640 1 --2- 172.21.15.28:0/2621878996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4109b90 0x7f0af41103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.695+0000 7f0af8990640 1 --2- 172.21.15.28:0/2621878996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af4109240 0x7f0af4109650 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.695+0000 7f0af8990640 1 --2- 172.21.15.28:0/2621878996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af4103900 0x7f0af4103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.695+0000 7f0af8990640 1 -- 172.21.15.28:0/2621878996 >> 172.21.15.28:0/2621878996 conn(0x7f0af407b5f0 msgr2=0x7f0af407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.695+0000 7f0af8990640 1 -- 172.21.15.28:0/2621878996 shutdown_connections 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.695+0000 7f0af8990640 1 -- 172.21.15.28:0/2621878996 wait complete. 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af8990640 1 Processor -- start 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af8990640 1 -- start start 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af8990640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af4103900 0x7f0af419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af8990640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4109240 0x7f0af419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af8990640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af4109b90 0x7f0af419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af8990640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0af4113890 con 0x7f0af4109240 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af8990640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0af4113710 con 0x7f0af4103900 2024-08-28T19:13:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af8990640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0af4113a10 con 0x7f0af4109b90 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af4103900 0x7f0af419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af4103900 0x7f0af419a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41488/0 (socket says 172.21.15.28:41488) 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af259c640 1 -- 172.21.15.28:0/2566007606 learned_addr learned my addr 172.21.15.28:0/2566007606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.696+0000 7f0af2d9d640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af4109b90 0x7f0af419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0af1d9b640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4109240 0x7f0af419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0af259c640 1 -- 172.21.15.28:0/2566007606 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af4109b90 msgr2=0x7f0af419fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0af259c640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af4109b90 0x7f0af419fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0af259c640 1 -- 172.21.15.28:0/2566007606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4109240 msgr2=0x7f0af419a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0af259c640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4109240 0x7f0af419a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0af259c640 1 -- 172.21.15.28:0/2566007606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af41a0290 con 0x7f0af4103900 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0af259c640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af4103900 0x7f0af419a320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400d770 tx=0x7f0ae400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:13:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0aeb7fe640 1 -- 172.21.15.28:0/2566007606 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4017070 con 0x7f0af4103900 2024-08-28T19:13:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0aeb7fe640 1 -- 172.21.15.28:0/2566007606 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ae40054d0 con 0x7f0af4103900 2024-08-28T19:13:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0af8990640 1 -- 172.21.15.28:0/2566007606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af41a0580 con 0x7f0af4103900 2024-08-28T19:13:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0aeb7fe640 1 -- 172.21.15.28:0/2566007606 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4004c30 con 0x7f0af4103900 2024-08-28T19:13:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.697+0000 7f0af8990640 1 -- 172.21.15.28:0/2566007606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af41a6ce0 con 0x7f0af4103900 2024-08-28T19:13:05.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.698+0000 7f0af8990640 1 -- 172.21.15.28:0/2566007606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0af4103d10 con 0x7f0af4103900 2024-08-28T19:13:05.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.700+0000 7f0aeb7fe640 1 -- 172.21.15.28:0/2566007606 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7f0ae4004dd0 con 0x7f0af4103900 2024-08-28T19:13:05.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.701+0000 7f0aeb7fe640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4af0 0x7f0ae00b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:05.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.701+0000 7f0af1d9b640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4af0 0x7f0ae00b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:05.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.701+0000 7f0aeb7fe640 1 -- 172.21.15.28:0/2566007606 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1576+0+0 (secure 0 0 0) 0x7f0ae40e34f0 con 0x7f0af4103900 2024-08-28T19:13:05.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.702+0000 7f0af1d9b640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4af0 0x7f0ae00b7000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009230 tx=0x7f0adc031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:13:05.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.705+0000 7f0aeb7fe640 1 -- 172.21.15.28:0/2566007606 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f0ae40ac130 con 0x7f0af4103900 2024-08-28T19:13:05.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:05.900+0000 7f0af8990640 1 -- 172.21.15.28:0/2566007606 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0af410b9e0 con 0x7f0ae00b4af0 2024-08-28T19:13:05.903 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-28T19:13:05.904 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: osdmap e7: 1 total, 0 up, 1 in 2024-08-28T19:13:05.904 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:13:05.904 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:05.904 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi016 2024-08-28T19:13:05.904 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:05.904 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:05.904 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:13:05.904 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi016 to 17092M 2024-08-28T19:13:05.904 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:05.904 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:05.905 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:05.905 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-28T19:13:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: osdmap e7: 1 total, 0 up, 1 in 2024-08-28T19:13:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:13:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi016 2024-08-28T19:13:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:13:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi016 to 17092M 2024-08-28T19:13:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:06.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-28T19:13:06.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: osdmap e7: 1 total, 0 up, 1 in 2024-08-28T19:13:06.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:13:06.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:06.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi016 2024-08-28T19:13:06.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:06.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:06.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:13:06.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi016 to 17092M 2024-08-28T19:13:06.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:06.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:06.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:06.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:06.751 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0[36367]: 2024-08-28T19:13:06.354+0000 7fd052e53640 -1 osd.0 0 waiting for initial osdmap 2024-08-28T19:13:06.751 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0[36367]: 2024-08-28T19:13:06.358+0000 7fd04ac49640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:13:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:13:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:13:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:13:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: osdmap e8: 1 total, 0 up, 1 in 2024-08-28T19:13:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:13:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:13:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:13:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:06 smithi115 ceph-mon[22184]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' 2024-08-28T19:13:07.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:13:07.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:13:07.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:07.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:13:07.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: osdmap e8: 1 total, 0 up, 1 in 2024-08-28T19:13:07.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:07.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:07.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:13:07.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:13:07.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:13:07.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:07.228 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:06 smithi028 ceph-mon[21769]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' 2024-08-28T19:13:07.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:13:07.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:13:07.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:07.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:13:07.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: osdmap e8: 1 total, 0 up, 1 in 2024-08-28T19:13:07.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:07.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:07.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:13:07.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:13:07.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:13:07.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:07.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:06 smithi016 ceph-mon[18556]: from='osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327]' entity='osd.0' 2024-08-28T19:13:08.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:08.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:07 smithi115 ceph-mon[22184]: osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327] boot 2024-08-28T19:13:08.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:07 smithi115 ceph-mon[22184]: osdmap e9: 1 total, 1 up, 1 in 2024-08-28T19:13:08.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:08.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:08.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:07 smithi016 ceph-mon[18556]: osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327] boot 2024-08-28T19:13:08.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:07 smithi016 ceph-mon[18556]: osdmap e9: 1 total, 1 up, 1 in 2024-08-28T19:13:08.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:08.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:07 smithi028 ceph-mon[21769]: osd.0 [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327] boot 2024-08-28T19:13:08.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:07 smithi028 ceph-mon[21769]: osdmap e9: 1 total, 1 up, 1 in 2024-08-28T19:13:08.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-28T19:13:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:08 smithi115 ceph-mon[22184]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:08 smithi115 ceph-mon[22184]: from='client.? 172.21.15.28:0/1535585844' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d4767044-e5ca-447f-9666-0840aa1028dd"} : dispatch 2024-08-28T19:13:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:08 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d4767044-e5ca-447f-9666-0840aa1028dd"} : dispatch 2024-08-28T19:13:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:08 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4767044-e5ca-447f-9666-0840aa1028dd"}]': finished 2024-08-28T19:13:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:08 smithi115 ceph-mon[22184]: osdmap e10: 2 total, 1 up, 2 in 2024-08-28T19:13:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:08 smithi115 ceph-mon[22184]: from='client.? 172.21.15.28:0/1502861469' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:13:09.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:08 smithi016 ceph-mon[18556]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:09.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:08 smithi016 ceph-mon[18556]: from='client.? 172.21.15.28:0/1535585844' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d4767044-e5ca-447f-9666-0840aa1028dd"} : dispatch 2024-08-28T19:13:09.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:08 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d4767044-e5ca-447f-9666-0840aa1028dd"} : dispatch 2024-08-28T19:13:09.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:08 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4767044-e5ca-447f-9666-0840aa1028dd"}]': finished 2024-08-28T19:13:09.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:08 smithi016 ceph-mon[18556]: osdmap e10: 2 total, 1 up, 2 in 2024-08-28T19:13:09.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:09.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:08 smithi016 ceph-mon[18556]: from='client.? 172.21.15.28:0/1502861469' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:13:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:08 smithi028 ceph-mon[21769]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-28T19:13:09.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:08 smithi028 ceph-mon[21769]: from='client.? 172.21.15.28:0/1535585844' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d4767044-e5ca-447f-9666-0840aa1028dd"} : dispatch 2024-08-28T19:13:09.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:08 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d4767044-e5ca-447f-9666-0840aa1028dd"} : dispatch 2024-08-28T19:13:09.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:08 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4767044-e5ca-447f-9666-0840aa1028dd"}]': finished 2024-08-28T19:13:09.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:08 smithi028 ceph-mon[21769]: osdmap e10: 2 total, 1 up, 2 in 2024-08-28T19:13:09.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:09.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:08 smithi028 ceph-mon[21769]: from='client.? 172.21.15.28:0/1502861469' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:13:11.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:10 smithi115 ceph-mon[22184]: pgmap v29: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:11.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:10 smithi016 ceph-mon[18556]: pgmap v29: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:10 smithi028 ceph-mon[21769]: pgmap v29: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:13.117 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:12 smithi028 ceph-mon[21769]: pgmap v30: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:12 smithi115 ceph-mon[22184]: pgmap v30: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:13.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:12 smithi016 ceph-mon[18556]: pgmap v30: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:15.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:14 smithi115 ceph-mon[22184]: pgmap v31: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:15.156 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:14 smithi028 ceph-mon[21769]: pgmap v31: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:15.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:14 smithi016 ceph-mon[18556]: pgmap v31: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:17.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:16 smithi028 ceph-mon[21769]: pgmap v32: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:17.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:16 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-28T19:13:17.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:16 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:16 smithi115 ceph-mon[22184]: pgmap v32: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:16 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-28T19:13:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:16 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:17.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:16 smithi016 ceph-mon[18556]: pgmap v32: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:17.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:16 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-28T19:13:17.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:16 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:18.001 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:17 smithi028 ceph-mon[21769]: Deploying daemon osd.1 on smithi028 2024-08-28T19:13:18.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:17 smithi115 ceph-mon[22184]: Deploying daemon osd.1 on smithi028 2024-08-28T19:13:18.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:17 smithi016 ceph-mon[18556]: Deploying daemon osd.1 on smithi028 2024-08-28T19:13:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:18 smithi115 ceph-mon[22184]: pgmap v33: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:19.133 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:18 smithi028 ceph-mon[21769]: pgmap v33: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:19.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:18 smithi016 ceph-mon[18556]: pgmap v33: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:21.004 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:20 smithi028 ceph-mon[21769]: pgmap v34: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:21.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:20 smithi016 ceph-mon[18556]: pgmap v34: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:20 smithi115 ceph-mon[22184]: pgmap v34: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:22.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:21 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:22.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:21 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:22.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:21 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:22.307 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:21 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:22.308 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:21 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:22.308 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:21 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:21 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:21 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:21 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:23.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:22 smithi016 ceph-mon[18556]: pgmap v35: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:23.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:22 smithi028 ceph-mon[21769]: pgmap v35: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:23.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:22 smithi115 ceph-mon[22184]: pgmap v35: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:24 smithi016 ceph-mon[18556]: pgmap v36: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:25.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:24 smithi028 ceph-mon[21769]: pgmap v36: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:25.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:25.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:25.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:25.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:25.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:24 smithi115 ceph-mon[22184]: pgmap v36: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.257+0000 7f0aeb7fe640 1 -- 172.21.15.28:0/2566007606 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0af410b9e0 con 0x7f0ae00b4af0 2024-08-28T19:13:26.259 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 1 on host 'smithi028' 2024-08-28T19:13:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.259+0000 7f0af8990640 1 -- 172.21.15.28:0/2566007606 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4af0 msgr2=0x7f0ae00b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.259+0000 7f0af8990640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4af0 0x7f0ae00b7000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009230 tx=0x7f0adc031400 comp rx=0 tx=0).stop 2024-08-28T19:13:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.260+0000 7f0af8990640 1 -- 172.21.15.28:0/2566007606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af4103900 msgr2=0x7f0af419a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.260+0000 7f0af8990640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af4103900 0x7f0af419a320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400d770 tx=0x7f0ae400dc30 comp rx=0 tx=0).stop 2024-08-28T19:13:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.260+0000 7f0af8990640 1 -- 172.21.15.28:0/2566007606 shutdown_connections 2024-08-28T19:13:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.260+0000 7f0af8990640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af4109b90 0x7f0af419fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.260+0000 7f0af8990640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4109240 0x7f0af419a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.260+0000 7f0af8990640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4af0 0x7f0ae00b7000 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.260+0000 7f0af8990640 1 --2- 172.21.15.28:0/2566007606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af4103900 0x7f0af419a320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.260+0000 7f0af8990640 1 -- 172.21.15.28:0/2566007606 >> 172.21.15.28:0/2566007606 conn(0x7f0af407b5f0 msgr2=0x7f0af41052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:13:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.260+0000 7f0af8990640 1 -- 172.21.15.28:0/2566007606 shutdown_connections 2024-08-28T19:13:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:13:26.260+0000 7f0af8990640 1 -- 172.21.15.28:0/2566007606 wait complete. 2024-08-28T19:13:26.919 DEBUG:teuthology.orchestra.run.smithi028:osd.1> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.1.service 2024-08-28T19:13:26.922 INFO:tasks.cephadm:Deploying osd.2 on smithi115 with /dev/vg_nvme/lv_4... 2024-08-28T19:13:26.922 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-28T19:13:27.169 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:13:27.219 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:26 smithi028 ceph-mon[21769]: pgmap v37: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:27.219 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:26 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:27.219 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:26 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:27.219 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:26 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:27.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:26 smithi016 ceph-mon[18556]: pgmap v37: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:27.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:26 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:27.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:26 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:27.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:26 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:27.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:26 smithi115 ceph-mon[22184]: pgmap v37: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:27.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:26 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:27.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:26 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:27.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:26 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:28.057 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:27 smithi028 ceph-mon[21769]: from='osd.1 [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-28T19:13:28.057 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:27 smithi028 ceph-mon[21769]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-28T19:13:28.058 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:13:27 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1[27359]: 2024-08-28T19:13:27.719+0000 7f417142f640 -1 osd.1 0 log_to_monitors true 2024-08-28T19:13:28.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:27 smithi115 ceph-mon[22184]: from='osd.1 [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-28T19:13:28.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:27 smithi115 ceph-mon[22184]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-28T19:13:28.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:27 smithi016 ceph-mon[18556]: from='osd.1 [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-28T19:13:28.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:27 smithi016 ceph-mon[18556]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-28T19:13:29.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:28 smithi016 ceph-mon[18556]: pgmap v38: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:29.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:28 smithi016 ceph-mon[18556]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-28T19:13:29.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:28 smithi016 ceph-mon[18556]: from='osd.1 [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:13:29.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:28 smithi016 ceph-mon[18556]: osdmap e11: 2 total, 1 up, 2 in 2024-08-28T19:13:29.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:29.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:28 smithi016 ceph-mon[18556]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:13:29.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:28 smithi115 ceph-mon[22184]: pgmap v38: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:29.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:28 smithi115 ceph-mon[22184]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-28T19:13:29.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:28 smithi115 ceph-mon[22184]: from='osd.1 [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:13:29.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:28 smithi115 ceph-mon[22184]: osdmap e11: 2 total, 1 up, 2 in 2024-08-28T19:13:29.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:29.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:28 smithi115 ceph-mon[22184]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:13:29.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:28 smithi028 ceph-mon[21769]: pgmap v38: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:29.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:28 smithi028 ceph-mon[21769]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-28T19:13:29.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:28 smithi028 ceph-mon[21769]: from='osd.1 [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:13:29.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:28 smithi028 ceph-mon[21769]: osdmap e11: 2 total, 1 up, 2 in 2024-08-28T19:13:29.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:29.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:28 smithi028 ceph-mon[21769]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:13:29.631 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:13:29.658 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi115:vg_nvme/lv_4 2024-08-28T19:13:29.784 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1[27359]: 2024-08-28T19:13:29.480+0000 7f416d3b1640 -1 osd.1 0 waiting for initial osdmap 2024-08-28T19:13:29.784 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1[27359]: 2024-08-28T19:13:29.484+0000 7f41651a7640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:13:29.906 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:13:30.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:13:30.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: osdmap e12: 2 total, 1 up, 2 in 2024-08-28T19:13:30.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:30.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:30.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: from='osd.1 ' entity='osd.1' 2024-08-28T19:13:30.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:13:30.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:30.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:30.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:13:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: osdmap e12: 2 total, 1 up, 2 in 2024-08-28T19:13:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: from='osd.1 ' entity='osd.1' 2024-08-28T19:13:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:13:30.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:30.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:30.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: osdmap e12: 2 total, 1 up, 2 in 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: from='osd.1 ' entity='osd.1' 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:30.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:30.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.405+0000 7fa57a57c640 1 -- 172.21.15.115:0/234441452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa574102820 msgr2=0x7fa574102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:30.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.405+0000 7fa57a57c640 1 --2- 172.21.15.115:0/234441452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa574102820 0x7fa574102c30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa564009900 tx=0x7fa56402a410 comp rx=0 tx=0).stop 2024-08-28T19:13:30.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.405+0000 7fa57a57c640 1 -- 172.21.15.115:0/234441452 shutdown_connections 2024-08-28T19:13:30.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.405+0000 7fa57a57c640 1 --2- 172.21.15.115:0/234441452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa574108ab0 0x7fa57410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:30.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.405+0000 7fa57a57c640 1 --2- 172.21.15.115:0/234441452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa574108160 0x7fa574108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:30.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.405+0000 7fa57a57c640 1 --2- 172.21.15.115:0/234441452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa574102820 0x7fa574102c30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:30.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.405+0000 7fa57a57c640 1 -- 172.21.15.115:0/234441452 >> 172.21.15.115:0/234441452 conn(0x7fa5740794e0 msgr2=0x7fa5740798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:13:30.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.406+0000 7fa57a57c640 1 -- 172.21.15.115:0/234441452 shutdown_connections 2024-08-28T19:13:30.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.406+0000 7fa57a57c640 1 -- 172.21.15.115:0/234441452 wait complete. 2024-08-28T19:13:30.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.406+0000 7fa57a57c640 1 Processor -- start 2024-08-28T19:13:30.409 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.406+0000 7fa57a57c640 1 -- start start 2024-08-28T19:13:30.411 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa57a57c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa574102820 0x7fa574104010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:30.411 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa57a57c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa574108160 0x7fa574104550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:30.411 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa57a57c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa574108ab0 0x7fa574104a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:30.411 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa57a57c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa574112730 con 0x7fa574108ab0 2024-08-28T19:13:30.411 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa57a57c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5741125b0 con 0x7fa574108160 2024-08-28T19:13:30.411 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa57a57c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa5741128b0 con 0x7fa574102820 2024-08-28T19:13:30.411 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa573fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa574102820 0x7fa574104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:30.411 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa573fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa574102820 0x7fa574104010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54478/0 (socket says 172.21.15.115:54478) 2024-08-28T19:13:30.411 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa573fff640 1 -- 172.21.15.115:0/2257997951 learned_addr learned my addr 172.21.15.115:0/2257997951 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-08-28T19:13:30.411 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa5737fe640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa574108160 0x7fa574104550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa578b19640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa574108ab0 0x7fa574104a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa573fff640 1 -- 172.21.15.115:0/2257997951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa574108160 msgr2=0x7fa574104550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa573fff640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa574108160 0x7fa574104550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa573fff640 1 -- 172.21.15.115:0/2257997951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa574108ab0 msgr2=0x7fa574104a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa573fff640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa574108ab0 0x7fa574104a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.407+0000 7fa573fff640 1 -- 172.21.15.115:0/2257997951 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5741a85b0 con 0x7fa574102820 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.408+0000 7fa573fff640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa574102820 0x7fa574104010 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa564005940 tx=0x7fa5640042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.408+0000 7fa5717fa640 1 -- 172.21.15.115:0/2257997951 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56403f030 con 0x7fa574102820 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.408+0000 7fa5717fa640 1 -- 172.21.15.115:0/2257997951 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa564003cb0 con 0x7fa574102820 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.408+0000 7fa57a57c640 1 -- 172.21.15.115:0/2257997951 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5741a8840 con 0x7fa574102820 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.408+0000 7fa5717fa640 1 -- 172.21.15.115:0/2257997951 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564035070 con 0x7fa574102820 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.408+0000 7fa57a57c640 1 -- 172.21.15.115:0/2257997951 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5741a8d70 con 0x7fa574102820 2024-08-28T19:13:30.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.409+0000 7fa57a57c640 1 -- 172.21.15.115:0/2257997951 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa574103aa0 con 0x7fa574102820 2024-08-28T19:13:30.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.411+0000 7fa5717fa640 1 -- 172.21.15.115:0/2257997951 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7fa564042030 con 0x7fa574102820 2024-08-28T19:13:30.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.412+0000 7fa5717fa640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa55c0b4b70 0x7fa55c0b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:30.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.412+0000 7fa5717fa640 1 -- 172.21.15.115:0/2257997951 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2244+0+0 (secure 0 0 0) 0x7fa5641044d0 con 0x7fa574102820 2024-08-28T19:13:30.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.412+0000 7fa5737fe640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa55c0b4b70 0x7fa55c0b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:30.414 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.413+0000 7fa5737fe640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa55c0b4b70 0x7fa55c0b7080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa568005fd0 tx=0x7fa568005850 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:13:30.417 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.415+0000 7fa5717fa640 1 -- 172.21.15.115:0/2257997951 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fa5640cce30 con 0x7fa574102820 2024-08-28T19:13:30.606 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:30.604+0000 7fa57a57c640 1 -- 172.21.15.115:0/2257997951 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa57406b8e0 con 0x7fa55c0b4b70 2024-08-28T19:13:31.200 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:13:31.200 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:13:31.200 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: pgmap v41: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:31.200 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi028 2024-08-28T19:13:31.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi028 to 17092M 2024-08-28T19:13:31.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:31.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: osd.1 [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235] boot 2024-08-28T19:13:31.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: osdmap e13: 2 total, 2 up, 2 in 2024-08-28T19:13:31.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:31.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:13:31.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:13:31.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:31.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:13:31.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:13:31.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: pgmap v41: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:31.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi028 2024-08-28T19:13:31.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi028 to 17092M 2024-08-28T19:13:31.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:31.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: osd.1 [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235] boot 2024-08-28T19:13:31.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: osdmap e13: 2 total, 2 up, 2 in 2024-08-28T19:13:31.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:31.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:13:31.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:13:31.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:31.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:13:31.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:13:31.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: pgmap v41: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-08-28T19:13:31.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi028 2024-08-28T19:13:31.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi028 to 17092M 2024-08-28T19:13:31.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:31.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: osd.1 [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235] boot 2024-08-28T19:13:31.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: osdmap e13: 2 total, 2 up, 2 in 2024-08-28T19:13:31.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-28T19:13:31.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:13:31.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:13:31.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:32.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:31 smithi115 ceph-mon[22184]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:13:32.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:31 smithi115 ceph-mon[22184]: osdmap e14: 2 total, 2 up, 2 in 2024-08-28T19:13:32.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:31 smithi016 ceph-mon[18556]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:13:32.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:31 smithi016 ceph-mon[18556]: osdmap e14: 2 total, 2 up, 2 in 2024-08-28T19:13:32.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:31 smithi028 ceph-mon[21769]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:13:32.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:31 smithi028 ceph-mon[21769]: osdmap e14: 2 total, 2 up, 2 in 2024-08-28T19:13:33.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:32 smithi115 ceph-mon[22184]: pgmap v43: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:33.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:32 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/3441574522' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22"} : dispatch 2024-08-28T19:13:33.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:32 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22"} : dispatch 2024-08-28T19:13:33.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:32 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22"}]': finished 2024-08-28T19:13:33.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:32 smithi115 ceph-mon[22184]: osdmap e15: 3 total, 2 up, 3 in 2024-08-28T19:13:33.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:33.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:32 smithi016 ceph-mon[18556]: pgmap v43: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:33.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:32 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/3441574522' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22"} : dispatch 2024-08-28T19:13:33.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:32 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22"} : dispatch 2024-08-28T19:13:33.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:32 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22"}]': finished 2024-08-28T19:13:33.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:32 smithi016 ceph-mon[18556]: osdmap e15: 3 total, 2 up, 3 in 2024-08-28T19:13:33.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:33.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:32 smithi028 ceph-mon[21769]: pgmap v43: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:33.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:32 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/3441574522' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22"} : dispatch 2024-08-28T19:13:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:32 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22"} : dispatch 2024-08-28T19:13:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:32 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22"}]': finished 2024-08-28T19:13:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:32 smithi028 ceph-mon[21769]: osdmap e15: 3 total, 2 up, 3 in 2024-08-28T19:13:33.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:34.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:33 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/3319288705' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:13:34.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:33 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/3319288705' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:13:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:33 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/3319288705' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:13:35.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:34 smithi016 ceph-mon[18556]: pgmap v46: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:35.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:34 smithi028 ceph-mon[21769]: pgmap v46: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:35.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:34 smithi115 ceph-mon[22184]: pgmap v46: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:37.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:36 smithi016 ceph-mon[18556]: pgmap v47: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:37.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:36 smithi028 ceph-mon[21769]: pgmap v47: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:37.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:36 smithi115 ceph-mon[22184]: pgmap v47: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:39.184 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:38 smithi115 ceph-mon[22184]: pgmap v48: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:39.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:38 smithi016 ceph-mon[18556]: pgmap v48: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:38 smithi028 ceph-mon[21769]: pgmap v48: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:41.222 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:40 smithi115 ceph-mon[22184]: pgmap v49: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:41.222 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-28T19:13:41.223 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:41.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:40 smithi016 ceph-mon[18556]: pgmap v49: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-28T19:13:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:40 smithi028 ceph-mon[21769]: pgmap v49: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-28T19:13:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:42.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:41 smithi115 ceph-mon[22184]: Deploying daemon osd.2 on smithi115 2024-08-28T19:13:42.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:41 smithi016 ceph-mon[18556]: Deploying daemon osd.2 on smithi115 2024-08-28T19:13:42.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:41 smithi028 ceph-mon[21769]: Deploying daemon osd.2 on smithi115 2024-08-28T19:13:43.232 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:42 smithi115 ceph-mon[22184]: pgmap v50: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:43.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:42 smithi016 ceph-mon[18556]: pgmap v50: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:42 smithi028 ceph-mon[21769]: pgmap v50: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:45.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:44 smithi115 ceph-mon[22184]: pgmap v51: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:45.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:44 smithi016 ceph-mon[18556]: pgmap v51: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:45.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:44 smithi028 ceph-mon[21769]: pgmap v51: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:46.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:46.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:46.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:46.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:46.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:46.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:46.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:46.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:47.212 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:46 smithi028 ceph-mon[21769]: pgmap v52: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:47.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:46 smithi016 ceph-mon[18556]: pgmap v52: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:47.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:46 smithi115 ceph-mon[22184]: pgmap v52: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:48 smithi028 ceph-mon[21769]: pgmap v53: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:49.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:49.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:49.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:49.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:49.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:48 smithi115 ceph-mon[22184]: pgmap v53: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:49.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:48 smithi016 ceph-mon[18556]: pgmap v53: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:49.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:49.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:49.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:49.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:49.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:49.916 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.915+0000 7fa5717fa640 1 -- 172.21.15.115:0/2257997951 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa57406b8e0 con 0x7fa55c0b4b70 2024-08-28T19:13:49.917 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 2 on host 'smithi115' 2024-08-28T19:13:49.919 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.917+0000 7fa57a57c640 1 -- 172.21.15.115:0/2257997951 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa55c0b4b70 msgr2=0x7fa55c0b7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:49.919 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.917+0000 7fa57a57c640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa55c0b4b70 0x7fa55c0b7080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa568005fd0 tx=0x7fa568005850 comp rx=0 tx=0).stop 2024-08-28T19:13:49.919 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.917+0000 7fa57a57c640 1 -- 172.21.15.115:0/2257997951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa574102820 msgr2=0x7fa574104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:49.919 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.917+0000 7fa57a57c640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa574102820 0x7fa574104010 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa564005940 tx=0x7fa5640042f0 comp rx=0 tx=0).stop 2024-08-28T19:13:49.919 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.917+0000 7fa57a57c640 1 -- 172.21.15.115:0/2257997951 shutdown_connections 2024-08-28T19:13:49.919 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.917+0000 7fa57a57c640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa574108ab0 0x7fa574104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:49.920 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.917+0000 7fa57a57c640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa55c0b4b70 0x7fa55c0b7080 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:49.920 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.917+0000 7fa57a57c640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa574108160 0x7fa574104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:49.920 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.917+0000 7fa57a57c640 1 --2- 172.21.15.115:0/2257997951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa574102820 0x7fa574104010 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:49.920 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.917+0000 7fa57a57c640 1 -- 172.21.15.115:0/2257997951 >> 172.21.15.115:0/2257997951 conn(0x7fa5740794e0 msgr2=0x7fa5741018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:13:49.920 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.918+0000 7fa57a57c640 1 -- 172.21.15.115:0/2257997951 shutdown_connections 2024-08-28T19:13:49.920 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:13:49.918+0000 7fa57a57c640 1 -- 172.21.15.115:0/2257997951 wait complete. 2024-08-28T19:13:50.513 DEBUG:teuthology.orchestra.run.smithi115:osd.2> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.2.service 2024-08-28T19:13:50.516 INFO:tasks.cephadm:Deploying osd.3 on smithi016 with /dev/vg_nvme/lv_3... 2024-08-28T19:13:50.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-28T19:13:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:13:50.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:50 smithi016 ceph-mon[18556]: pgmap v54: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:50.897 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:50.897 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:50.897 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:51.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:50 smithi115 ceph-mon[22184]: pgmap v54: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:51.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:51.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:51.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:51.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:50 smithi028 ceph-mon[21769]: pgmap v54: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:51.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:13:51.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:51.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:52.371 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:13:52 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2[28467]: 2024-08-28T19:13:52.116+0000 7f49f0ba7640 -1 osd.2 0 log_to_monitors true 2024-08-28T19:13:52.959 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:52 smithi115 ceph-mon[22184]: pgmap v55: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:52.959 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:52 smithi115 ceph-mon[22184]: from='osd.2 [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-28T19:13:52.959 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:52 smithi115 ceph-mon[22184]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-28T19:13:53.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:52 smithi028 ceph-mon[21769]: pgmap v55: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:53.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:52 smithi028 ceph-mon[21769]: from='osd.2 [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-28T19:13:53.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:52 smithi028 ceph-mon[21769]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-28T19:13:53.183 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:52 smithi016 ceph-mon[18556]: pgmap v55: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:53.183 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:52 smithi016 ceph-mon[18556]: from='osd.2 [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-28T19:13:53.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:52 smithi016 ceph-mon[18556]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-28T19:13:53.462 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:13:53.487 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-08-28T19:13:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:13:54.016 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-28T19:13:54.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: osdmap e16: 3 total, 2 up, 3 in 2024-08-28T19:13:54.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: from='osd.2 [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:13:54.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:54.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:13:54.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:13:54.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:54.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:54.038 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-28T19:13:54.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: osdmap e16: 3 total, 2 up, 3 in 2024-08-28T19:13:54.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: from='osd.2 [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:13:54.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:54.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:13:54.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:13:54.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:54.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:54.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-28T19:13:54.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: osdmap e16: 3 total, 2 up, 3 in 2024-08-28T19:13:54.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: from='osd.2 [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:13:54.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:54.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:13:54.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:13:54.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:54.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:13:54.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:54.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.224+0000 7f1df820d640 1 -- 172.21.15.16:0/174787256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df0108170 msgr2=0x7f1df0108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:54.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.224+0000 7f1df820d640 1 --2- 172.21.15.16:0/174787256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df0108170 0x7f1df0108580 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009900 tx=0x7f1de002a410 comp rx=0 tx=0).stop 2024-08-28T19:13:54.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.224+0000 7f1df820d640 1 -- 172.21.15.16:0/174787256 shutdown_connections 2024-08-28T19:13:54.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.224+0000 7f1df820d640 1 --2- 172.21.15.16:0/174787256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1df0108ac0 0x7f1df010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:54.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.224+0000 7f1df820d640 1 --2- 172.21.15.16:0/174787256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df0108170 0x7f1df0108580 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:54.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.224+0000 7f1df820d640 1 --2- 172.21.15.16:0/174787256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df0102880 0x7f1df0102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:54.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.224+0000 7f1df820d640 1 -- 172.21.15.16:0/174787256 >> 172.21.15.16:0/174787256 conn(0x7f1df0079500 msgr2=0x7f1df0079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:13:54.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.224+0000 7f1df820d640 1 -- 172.21.15.16:0/174787256 shutdown_connections 2024-08-28T19:13:54.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.224+0000 7f1df820d640 1 -- 172.21.15.16:0/174787256 wait complete. 2024-08-28T19:13:54.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df820d640 1 Processor -- start 2024-08-28T19:13:54.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df820d640 1 -- start start 2024-08-28T19:13:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df820d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1df0102880 0x7f1df019b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df820d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df0108170 0x7f1df019b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df820d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df0108ac0 0x7f1df019ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df820d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1df01126f0 con 0x7f1df0108ac0 2024-08-28T19:13:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df820d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1df0112570 con 0x7f1df0108170 2024-08-28T19:13:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df820d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1df0112870 con 0x7f1df0102880 2024-08-28T19:13:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df67aa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df0108ac0 0x7f1df019ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df5fa9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1df0102880 0x7f1df019b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df5fa9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1df0102880 0x7f1df019b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:49494/0 (socket says 172.21.15.16:49494) 2024-08-28T19:13:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df67aa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df0108ac0 0x7f1df019ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32966/0 (socket says 172.21.15.16:32966) 2024-08-28T19:13:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df5fa9640 1 -- 172.21.15.16:0/2378207454 learned_addr learned my addr 172.21.15.16:0/2378207454 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.225+0000 7f1df57a8640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df0108170 0x7f1df019b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1df67aa640 1 -- 172.21.15.16:0/2378207454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1df0102880 msgr2=0x7f1df019b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1df67aa640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1df0102880 0x7f1df019b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1df67aa640 1 -- 172.21.15.16:0/2378207454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df0108170 msgr2=0x7f1df019b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1df67aa640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df0108170 0x7f1df019b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1df67aa640 1 -- 172.21.15.16:0/2378207454 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df01a6310 con 0x7f1df0108ac0 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1df5fa9640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1df0102880 0x7f1df019b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1df67aa640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df0108ac0 0x7f1df019ba80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1de800da10 tx=0x7f1de800ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1de6ffd640 1 -- 172.21.15.16:0/2378207454 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de800bd70 con 0x7f1df0108ac0 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1de6ffd640 1 -- 172.21.15.16:0/2378207454 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1de8003e20 con 0x7f1df0108ac0 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1de6ffd640 1 -- 172.21.15.16:0/2378207454 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de80106c0 con 0x7f1df0108ac0 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1df820d640 1 -- 172.21.15.16:0/2378207454 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df01a6600 con 0x7f1df0108ac0 2024-08-28T19:13:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.226+0000 7f1df820d640 1 -- 172.21.15.16:0/2378207454 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df01a6ac0 con 0x7f1df0108ac0 2024-08-28T19:13:54.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.227+0000 7f1df820d640 1 -- 172.21.15.16:0/2378207454 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df0102c90 con 0x7f1df0108ac0 2024-08-28T19:13:54.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.229+0000 7f1de6ffd640 1 -- 172.21.15.16:0/2378207454 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 152538+0+0 (secure 0 0 0) 0x7f1de8004270 con 0x7f1df0108ac0 2024-08-28T19:13:54.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.229+0000 7f1de6ffd640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ddc0b4af0 0x7f1ddc0b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:13:54.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.229+0000 7f1df5fa9640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ddc0b4af0 0x7f1ddc0b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:13:54.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.229+0000 7f1de6ffd640 1 -- 172.21.15.16:0/2378207454 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 2786+0+0 (secure 0 0 0) 0x7f1de80e42e0 con 0x7f1df0108ac0 2024-08-28T19:13:54.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.230+0000 7f1df5fa9640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ddc0b4af0 0x7f1ddc0b7000 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0058c0 tx=0x7f1dec00a380 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:13:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.233+0000 7f1de6ffd640 1 -- 172.21.15.16:0/2378207454 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f1de80aca80 con 0x7f1df0108ac0 2024-08-28T19:13:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:54.430+0000 7f1df820d640 1 -- 172.21.15.16:0/2378207454 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1df00635a0 con 0x7f1ddc0b4af0 2024-08-28T19:13:54.810 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2[28467]: 2024-08-28T19:13:54.508+0000 7f49ecb29640 -1 osd.2 0 waiting for initial osdmap 2024-08-28T19:13:54.810 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2[28467]: 2024-08-28T19:13:54.514+0000 7f49e491f640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:13:55.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:54 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi115 2024-08-28T19:13:55.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:54 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi115 to 17088M 2024-08-28T19:13:55.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:54 smithi016 ceph-mon[18556]: pgmap v57: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:55.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:54 smithi016 ceph-mon[18556]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:13:55.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:54 smithi016 ceph-mon[18556]: osdmap e17: 3 total, 2 up, 3 in 2024-08-28T19:13:55.075 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:54 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:55.075 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:54 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:55.075 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:54 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:13:55.075 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:54 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:13:55.075 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:54 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:55.075 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:54 smithi016 ceph-mon[18556]: from='osd.2 ' entity='osd.2' 2024-08-28T19:13:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi115 2024-08-28T19:13:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi115 to 17088M 2024-08-28T19:13:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-mon[22184]: pgmap v57: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-mon[22184]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:13:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-mon[22184]: osdmap e17: 3 total, 2 up, 3 in 2024-08-28T19:13:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:13:55.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:13:55.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:55.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:54 smithi115 ceph-mon[22184]: from='osd.2 ' entity='osd.2' 2024-08-28T19:13:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:54 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi115 2024-08-28T19:13:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:54 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi115 to 17088M 2024-08-28T19:13:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:54 smithi028 ceph-mon[21769]: pgmap v57: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:54 smithi028 ceph-mon[21769]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:13:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:54 smithi028 ceph-mon[21769]: osdmap e17: 3 total, 2 up, 3 in 2024-08-28T19:13:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:54 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:54 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:54 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:13:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:54 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:13:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:54 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:13:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:54 smithi028 ceph-mon[21769]: from='osd.2 ' entity='osd.2' 2024-08-28T19:13:55.895 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:55 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:13:55.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:55 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:13:55.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:55 smithi016 ceph-mon[18556]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:13:55.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:55.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:55 smithi016 ceph-mon[18556]: osd.2 [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648] boot 2024-08-28T19:13:55.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:55 smithi016 ceph-mon[18556]: osdmap e18: 3 total, 3 up, 3 in 2024-08-28T19:13:55.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:55 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:13:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:55 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:13:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:55 smithi115 ceph-mon[22184]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:13:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:55 smithi115 ceph-mon[22184]: osd.2 [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648] boot 2024-08-28T19:13:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:55 smithi115 ceph-mon[22184]: osdmap e18: 3 total, 3 up, 3 in 2024-08-28T19:13:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:55 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:13:56.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:55 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:13:56.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:55 smithi028 ceph-mon[21769]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:13:56.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:56.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:55 smithi028 ceph-mon[21769]: osd.2 [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648] boot 2024-08-28T19:13:56.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:55 smithi028 ceph-mon[21769]: osdmap e18: 3 total, 3 up, 3 in 2024-08-28T19:13:56.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-28T19:13:57.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:56 smithi016 ceph-mon[18556]: pgmap v59: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:57.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:56 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-08-28T19:13:57.091 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:56 smithi028 ceph-mon[21769]: pgmap v59: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:57.091 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:56 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-08-28T19:13:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:56 smithi115 ceph-mon[22184]: pgmap v59: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-08-28T19:13:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:56 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-08-28T19:13:57.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40962]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-28T19:13:57.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40962]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40962]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40962]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:57.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40965]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-28T19:13:57.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40965]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40965]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40965]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:57.295 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40950]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-28T19:13:57.295 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40950]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.295 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40950]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.295 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40950]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:57.295 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40953]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-28T19:13:57.296 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40953]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.296 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40953]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.296 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:13:57 smithi016 sudo[40953]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:57.559 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31574]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-28T19:13:57.559 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31574]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.560 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31574]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.560 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31574]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:57.560 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31577]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-28T19:13:57.560 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31577]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.560 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31577]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.560 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31577]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:57.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31580]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-28T19:13:57.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31580]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31580]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31580]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:57.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31583]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-28T19:13:57.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31583]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31583]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 sudo[31583]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32554]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-28T19:13:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32554]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32554]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32554]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32557]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-28T19:13:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32557]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32557]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32557]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:57.621 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32548]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-28T19:13:57.622 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32548]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.622 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32548]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.622 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32548]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:57.622 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32551]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-28T19:13:57.622 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32551]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-28T19:13:57.622 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32551]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-28T19:13:57.622 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:13:57 smithi115 sudo[32551]: pam_unix(sudo:session): session closed for user root 2024-08-28T19:13:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-28T19:13:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: osdmap e19: 3 total, 3 up, 3 in 2024-08-28T19:13:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-08-28T19:13:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2573318656' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0edc63dd-6f28-4c2a-9bde-d6be083fc6c5"} : dispatch 2024-08-28T19:13:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2573318656' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0edc63dd-6f28-4c2a-9bde-d6be083fc6c5"}]': finished 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: osdmap e20: 4 total, 3 up, 4 in 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:13:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:13:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:13:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-28T19:13:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-28T19:13:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:13:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:13:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:13:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-28T19:13:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:57 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2946707605' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:13:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:13:58.222+0000 7f1de6ffd640 1 -- 172.21.15.16:0/2378207454 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f1de80a8610 con 0x7f1df0108ac0 2024-08-28T19:13:58.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-28T19:13:58.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: osdmap e19: 3 total, 3 up, 3 in 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2573318656' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0edc63dd-6f28-4c2a-9bde-d6be083fc6c5"} : dispatch 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2573318656' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0edc63dd-6f28-4c2a-9bde-d6be083fc6c5"}]': finished 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: osdmap e20: 4 total, 3 up, 4 in 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:13:58.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-28T19:13:58.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:57 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2946707605' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:13:58.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-28T19:13:58.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: osdmap e19: 3 total, 3 up, 3 in 2024-08-28T19:13:58.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-08-28T19:13:58.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2573318656' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0edc63dd-6f28-4c2a-9bde-d6be083fc6c5"} : dispatch 2024-08-28T19:13:58.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-28T19:13:58.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2573318656' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0edc63dd-6f28-4c2a-9bde-d6be083fc6c5"}]': finished 2024-08-28T19:13:58.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: osdmap e20: 4 total, 3 up, 4 in 2024-08-28T19:13:58.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:13:58.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-28T19:13:58.312 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-28T19:13:58.312 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-28T19:13:58.312 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-28T19:13:58.312 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-28T19:13:58.312 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-28T19:13:58.312 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:57 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2946707605' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:13:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:58 smithi016 ceph-mon[18556]: pgmap v63: 1 pgs: 1 unknown; 0 B data, 100 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:13:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:58 smithi016 ceph-mon[18556]: osdmap e21: 4 total, 3 up, 4 in 2024-08-28T19:13:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:58 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:13:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:13:58 smithi016 ceph-mon[18556]: mgrmap e20: x(active, since 110s), standbys: z, y 2024-08-28T19:13:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:58 smithi028 ceph-mon[21769]: pgmap v63: 1 pgs: 1 unknown; 0 B data, 100 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:13:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:58 smithi028 ceph-mon[21769]: osdmap e21: 4 total, 3 up, 4 in 2024-08-28T19:13:59.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:58 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:13:59.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:13:58 smithi028 ceph-mon[21769]: mgrmap e20: x(active, since 110s), standbys: z, y 2024-08-28T19:13:59.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:58 smithi115 ceph-mon[22184]: pgmap v63: 1 pgs: 1 unknown; 0 B data, 100 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:13:59.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:58 smithi115 ceph-mon[22184]: osdmap e21: 4 total, 3 up, 4 in 2024-08-28T19:13:59.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:58 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:13:59.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:13:58 smithi115 ceph-mon[22184]: mgrmap e20: x(active, since 110s), standbys: z, y 2024-08-28T19:14:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:00 smithi016 ceph-mon[18556]: pgmap v65: 1 pgs: 1 unknown; 0 B data, 100 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:00 smithi028 ceph-mon[21769]: pgmap v65: 1 pgs: 1 unknown; 0 B data, 100 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:01.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:00 smithi115 ceph-mon[22184]: pgmap v65: 1 pgs: 1 unknown; 0 B data, 100 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:03.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:02 smithi016 ceph-mon[18556]: pgmap v66: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:02 smithi028 ceph-mon[21769]: pgmap v66: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:02 smithi115 ceph-mon[22184]: pgmap v66: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:05.188 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:04 smithi016 ceph-mon[18556]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:04 smithi028 ceph-mon[21769]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:04 smithi115 ceph-mon[22184]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:06.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-28T19:14:06.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:06.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-28T19:14:06.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:06.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-28T19:14:06.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:07.068 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:06 smithi016 ceph-mon[18556]: Deploying daemon osd.3 on smithi016 2024-08-28T19:14:07.069 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:06 smithi016 ceph-mon[18556]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:06 smithi028 ceph-mon[21769]: Deploying daemon osd.3 on smithi016 2024-08-28T19:14:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:06 smithi028 ceph-mon[21769]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:06 smithi115 ceph-mon[22184]: Deploying daemon osd.3 on smithi016 2024-08-28T19:14:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:06 smithi115 ceph-mon[22184]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:09.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:08 smithi016 ceph-mon[18556]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:08 smithi028 ceph-mon[21769]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:08 smithi115 ceph-mon[22184]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:11.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:10 smithi016 ceph-mon[18556]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:11.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:11.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:11.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:10 smithi028 ceph-mon[21769]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:10 smithi115 ceph-mon[22184]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:13.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:12 smithi016 ceph-mon[18556]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:12 smithi028 ceph-mon[21769]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:12 smithi115 ceph-mon[22184]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:13.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.813+0000 7f1de6ffd640 1 -- 172.21.15.16:0/2378207454 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1df00635a0 con 0x7f1ddc0b4af0 2024-08-28T19:14:13.814 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 3 on host 'smithi016' 2024-08-28T19:14:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 -- 172.21.15.16:0/2378207454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ddc0b4af0 msgr2=0x7f1ddc0b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:14:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ddc0b4af0 0x7f1ddc0b7000 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0058c0 tx=0x7f1dec00a380 comp rx=0 tx=0).stop 2024-08-28T19:14:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 -- 172.21.15.16:0/2378207454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df0108ac0 msgr2=0x7f1df019ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:14:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df0108ac0 0x7f1df019ba80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1de800da10 tx=0x7f1de800ded0 comp rx=0 tx=0).stop 2024-08-28T19:14:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 -- 172.21.15.16:0/2378207454 shutdown_connections 2024-08-28T19:14:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df0108ac0 0x7f1df019ba80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ddc0b4af0 0x7f1ddc0b7000 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df0108170 0x7f1df019b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 --2- 172.21.15.16:0/2378207454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1df0102880 0x7f1df019b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 -- 172.21.15.16:0/2378207454 >> 172.21.15.16:0/2378207454 conn(0x7f1df0079500 msgr2=0x7f1df01041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:14:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 -- 172.21.15.16:0/2378207454 shutdown_connections 2024-08-28T19:14:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:14:13.816+0000 7f1df820d640 1 -- 172.21.15.16:0/2378207454 wait complete. 2024-08-28T19:14:14.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:14.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:14.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:14 smithi016 ceph-mon[18556]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:14.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:14.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.441 DEBUG:teuthology.orchestra.run.smithi016:osd.3> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.3.service 2024-08-28T19:14:14.444 INFO:tasks.cephadm:Deploying osd.4 on smithi028 with /dev/vg_nvme/lv_3... 2024-08-28T19:14:14.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-28T19:14:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:14 smithi115 ceph-mon[22184]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:14:14.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:14.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:14.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:14 smithi028 ceph-mon[21769]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:14.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:14.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:14.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:16.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:16 smithi016 ceph-mon[18556]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:16.751 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:14:16 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3[42974]: 2024-08-28T19:14:16.531+0000 7fca16341640 -1 osd.3 0 log_to_monitors true 2024-08-28T19:14:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:16 smithi028 ceph-mon[21769]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:16.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:16 smithi115 ceph-mon[22184]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:17.204 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:14:17.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-08-28T19:14:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:14:17.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:17 smithi016 ceph-mon[18556]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-28T19:14:17.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:17 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:17.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:17 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:17.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:17 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:17.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:17 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:17.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:17 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:17.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:17 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:17.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:17 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:17.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:17 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:17.777 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:17 smithi028 ceph-mon[21769]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-28T19:14:17.778 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:17 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:17.778 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:17 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:17.778 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:17 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:17.778 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:17 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:17.778 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:17 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:17.778 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:17 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:17.778 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:17 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:17.778 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:17 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:17 smithi115 ceph-mon[22184]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-28T19:14:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:17 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:17 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:17 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:17 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:17 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:17 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:17 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:17.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:17 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.005+0000 7ff95801c640 1 -- 172.21.15.28:0/4110108859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff950109bd0 msgr2=0x7ff950110400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:14:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.005+0000 7ff95801c640 1 --2- 172.21.15.28:0/4110108859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff950109bd0 0x7ff950110400 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff948009ca0 tx=0x7ff94802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:14:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.005+0000 7ff95801c640 1 -- 172.21.15.28:0/4110108859 shutdown_connections 2024-08-28T19:14:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.005+0000 7ff95801c640 1 --2- 172.21.15.28:0/4110108859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff950109bd0 0x7ff950110400 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.005+0000 7ff95801c640 1 --2- 172.21.15.28:0/4110108859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff950109280 0x7ff950109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.005+0000 7ff95801c640 1 --2- 172.21.15.28:0/4110108859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff950103940 0x7ff950103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.005+0000 7ff95801c640 1 -- 172.21.15.28:0/4110108859 >> 172.21.15.28:0/4110108859 conn(0x7ff9500ff3d0 msgr2=0x7ff950101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:14:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.005+0000 7ff95801c640 1 -- 172.21.15.28:0/4110108859 shutdown_connections 2024-08-28T19:14:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.005+0000 7ff95801c640 1 -- 172.21.15.28:0/4110108859 wait complete. 2024-08-28T19:14:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.005+0000 7ff95801c640 1 Processor -- start 2024-08-28T19:14:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.005+0000 7ff95801c640 1 -- start start 2024-08-28T19:14:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff95801c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff950103940 0x7ff9500754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:14:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff95801c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff950109280 0x7ff9500759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:14:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff95801c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff950109bd0 0x7ff950075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff95801c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff950113890 con 0x7ff950109280 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff95801c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff950113710 con 0x7ff950103940 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff95801c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff950113a10 con 0x7ff950109bd0 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff955db8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff950103940 0x7ff9500754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff955db8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff950103940 0x7ff9500754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54242/0 (socket says 172.21.15.28:54242) 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff955db8640 1 -- 172.21.15.28:0/3471434847 learned_addr learned my addr 172.21.15.28:0/3471434847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff9555b7640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff950109280 0x7ff9500759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff9565b9640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff950109bd0 0x7ff950075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff955db8640 1 -- 172.21.15.28:0/3471434847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff950109bd0 msgr2=0x7ff950075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff955db8640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff950109bd0 0x7ff950075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff955db8640 1 -- 172.21.15.28:0/3471434847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff950109280 msgr2=0x7ff9500759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:14:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff955db8640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff950109280 0x7ff9500759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff955db8640 1 -- 172.21.15.28:0/3471434847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9501a84c0 con 0x7ff950103940 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.006+0000 7ff955db8640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff950103940 0x7ff9500754b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff94c00e9f0 tx=0x7ff94c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.007+0000 7ff946ffd640 1 -- 172.21.15.28:0/3471434847 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c00cd80 con 0x7ff950103940 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.007+0000 7ff946ffd640 1 -- 172.21.15.28:0/3471434847 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff94c0108e0 con 0x7ff950103940 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.007+0000 7ff95801c640 1 -- 172.21.15.28:0/3471434847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9501a87b0 con 0x7ff950103940 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.007+0000 7ff946ffd640 1 -- 172.21.15.28:0/3471434847 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c018980 con 0x7ff950103940 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.007+0000 7ff95801c640 1 -- 172.21.15.28:0/3471434847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9501a8d40 con 0x7ff950103940 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.008+0000 7ff95801c640 1 -- 172.21.15.28:0/3471434847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff950103d50 con 0x7ff950103940 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.009+0000 7ff946ffd640 1 -- 172.21.15.28:0/3471434847 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7ff94c010430 con 0x7ff950103940 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.010+0000 7ff946ffd640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff93c0b4c00 0x7ff93c0b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.010+0000 7ff946ffd640 1 -- 172.21.15.28:0/3471434847 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 3588+0+0 (secure 0 0 0) 0x7ff94c0e4710 con 0x7ff950103940 2024-08-28T19:14:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.010+0000 7ff9555b7640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff93c0b4c00 0x7ff93c0b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:14:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.011+0000 7ff9555b7640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff93c0b4c00 0x7ff93c0b7110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff950102c20 tx=0x7ff94000b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:14:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.013+0000 7ff946ffd640 1 -- 172.21.15.28:0/3471434847 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ff94c0ac3c0 con 0x7ff950103940 2024-08-28T19:14:18.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:18.210+0000 7ff95801c640 1 -- 172.21.15.28:0/3471434847 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff9500635a0 con 0x7ff93c0b4c00 2024-08-28T19:14:18.748 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:18 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi016 2024-08-28T19:14:18.749 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:18 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi016 to 8546M 2024-08-28T19:14:18.749 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:18 smithi028 ceph-mon[21769]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:18.749 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:18 smithi028 ceph-mon[21769]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-28T19:14:18.749 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:18 smithi028 ceph-mon[21769]: osdmap e22: 4 total, 3 up, 4 in 2024-08-28T19:14:18.749 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:18 smithi028 ceph-mon[21769]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:14:18.749 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:18.749 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:18 smithi028 ceph-mon[21769]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:14:18.749 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:14:18.749 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:14:18.749 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:18 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:18.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:18 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi016 2024-08-28T19:14:18.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:18 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi016 to 8546M 2024-08-28T19:14:18.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:18 smithi016 ceph-mon[18556]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:18.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:18 smithi016 ceph-mon[18556]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-28T19:14:18.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:18 smithi016 ceph-mon[18556]: osdmap e22: 4 total, 3 up, 4 in 2024-08-28T19:14:18.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:18 smithi016 ceph-mon[18556]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:14:18.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:18.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:18 smithi016 ceph-mon[18556]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:14:18.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:14:18.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:14:18.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:18 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:18.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:18 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi016 2024-08-28T19:14:18.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:18 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi016 to 8546M 2024-08-28T19:14:18.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:18 smithi115 ceph-mon[22184]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:18.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:18 smithi115 ceph-mon[22184]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-28T19:14:18.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:18 smithi115 ceph-mon[22184]: osdmap e22: 4 total, 3 up, 4 in 2024-08-28T19:14:18.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:18 smithi115 ceph-mon[22184]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:14:18.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:18.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:18 smithi115 ceph-mon[22184]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:14:18.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:14:18.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:14:18.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:18 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:19.468 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:14:19 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3[42974]: 2024-08-28T19:14:19.018+0000 7fca122c3640 -1 osd.3 0 waiting for initial osdmap 2024-08-28T19:14:19.468 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:14:19 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3[42974]: 2024-08-28T19:14:19.024+0000 7fca0a0b9640 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:14:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:19 smithi028 ceph-mon[21769]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:14:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:19 smithi028 ceph-mon[21769]: osdmap e23: 4 total, 3 up, 4 in 2024-08-28T19:14:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:19.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:19 smithi028 ceph-mon[21769]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' 2024-08-28T19:14:19.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:19 smithi028 ceph-mon[21769]: osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569] boot 2024-08-28T19:14:19.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:19 smithi028 ceph-mon[21769]: osdmap e24: 4 total, 4 up, 4 in 2024-08-28T19:14:19.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:19 smithi016 ceph-mon[18556]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:14:19.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:19 smithi016 ceph-mon[18556]: osdmap e23: 4 total, 3 up, 4 in 2024-08-28T19:14:19.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:19.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:19.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:19 smithi016 ceph-mon[18556]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' 2024-08-28T19:14:19.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:19 smithi016 ceph-mon[18556]: osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569] boot 2024-08-28T19:14:19.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:19 smithi016 ceph-mon[18556]: osdmap e24: 4 total, 4 up, 4 in 2024-08-28T19:14:19.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:19 smithi115 ceph-mon[22184]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:14:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:19 smithi115 ceph-mon[22184]: osdmap e23: 4 total, 3 up, 4 in 2024-08-28T19:14:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:19 smithi115 ceph-mon[22184]: from='osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569]' entity='osd.3' 2024-08-28T19:14:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:19 smithi115 ceph-mon[22184]: osd.3 [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569] boot 2024-08-28T19:14:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:19 smithi115 ceph-mon[22184]: osdmap e24: 4 total, 4 up, 4 in 2024-08-28T19:14:19.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-28T19:14:20.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:20 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:14:20.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:20 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:14:20.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:20 smithi016 ceph-mon[18556]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:20.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:20 smithi016 ceph-mon[18556]: osdmap e25: 4 total, 4 up, 4 in 2024-08-28T19:14:20.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:20 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:14:20.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:20 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:14:20.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:20 smithi028 ceph-mon[21769]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:20.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:20 smithi028 ceph-mon[21769]: osdmap e25: 4 total, 4 up, 4 in 2024-08-28T19:14:20.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:20 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:14:20.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:20 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:14:20.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:20 smithi115 ceph-mon[22184]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-08-28T19:14:20.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:20 smithi115 ceph-mon[22184]: osdmap e25: 4 total, 4 up, 4 in 2024-08-28T19:14:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:21 smithi016 ceph-mon[18556]: from='client.? 172.21.15.28:0/4189521945' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fcaeb7d4-aa38-40a7-abdf-3f16cf9047df"} : dispatch 2024-08-28T19:14:21.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:21 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fcaeb7d4-aa38-40a7-abdf-3f16cf9047df"} : dispatch 2024-08-28T19:14:21.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:21 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcaeb7d4-aa38-40a7-abdf-3f16cf9047df"}]': finished 2024-08-28T19:14:21.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:21 smithi016 ceph-mon[18556]: osdmap e26: 5 total, 4 up, 5 in 2024-08-28T19:14:21.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:21 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:21.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:21 smithi016 ceph-mon[18556]: from='client.? 172.21.15.28:0/2370334742' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:14:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:21 smithi028 ceph-mon[21769]: from='client.? 172.21.15.28:0/4189521945' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fcaeb7d4-aa38-40a7-abdf-3f16cf9047df"} : dispatch 2024-08-28T19:14:21.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:21 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fcaeb7d4-aa38-40a7-abdf-3f16cf9047df"} : dispatch 2024-08-28T19:14:21.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:21 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcaeb7d4-aa38-40a7-abdf-3f16cf9047df"}]': finished 2024-08-28T19:14:21.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:21 smithi028 ceph-mon[21769]: osdmap e26: 5 total, 4 up, 5 in 2024-08-28T19:14:21.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:21 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:21.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:21 smithi028 ceph-mon[21769]: from='client.? 172.21.15.28:0/2370334742' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:14:21.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:21 smithi115 ceph-mon[22184]: from='client.? 172.21.15.28:0/4189521945' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fcaeb7d4-aa38-40a7-abdf-3f16cf9047df"} : dispatch 2024-08-28T19:14:21.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:21 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fcaeb7d4-aa38-40a7-abdf-3f16cf9047df"} : dispatch 2024-08-28T19:14:21.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:21 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcaeb7d4-aa38-40a7-abdf-3f16cf9047df"}]': finished 2024-08-28T19:14:21.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:21 smithi115 ceph-mon[22184]: osdmap e26: 5 total, 4 up, 5 in 2024-08-28T19:14:21.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:21 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:21.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:21 smithi115 ceph-mon[22184]: from='client.? 172.21.15.28:0/2370334742' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:14:23.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:22 smithi016 ceph-mon[18556]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-08-28T19:14:23.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:22 smithi028 ceph-mon[21769]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-08-28T19:14:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:22 smithi115 ceph-mon[22184]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-08-28T19:14:25.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:24 smithi016 ceph-mon[18556]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-08-28T19:14:25.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:24 smithi028 ceph-mon[21769]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-08-28T19:14:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:24 smithi115 ceph-mon[22184]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-08-28T19:14:27.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:26 smithi016 ceph-mon[18556]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-28T19:14:27.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:26 smithi028 ceph-mon[21769]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-28T19:14:27.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:26 smithi115 ceph-mon[22184]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-28T19:14:29.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:28 smithi016 ceph-mon[18556]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-08-28T19:14:29.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:28 smithi028 ceph-mon[21769]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-08-28T19:14:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:28 smithi115 ceph-mon[22184]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-08-28T19:14:30.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-28T19:14:30.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:30.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-28T19:14:30.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:30.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-28T19:14:30.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:31.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:30 smithi016 ceph-mon[18556]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-28T19:14:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:30 smithi016 ceph-mon[18556]: Deploying daemon osd.4 on smithi028 2024-08-28T19:14:31.031 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:30 smithi028 ceph-mon[21769]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-28T19:14:31.031 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:30 smithi028 ceph-mon[21769]: Deploying daemon osd.4 on smithi028 2024-08-28T19:14:31.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:30 smithi115 ceph-mon[22184]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-28T19:14:31.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:30 smithi115 ceph-mon[22184]: Deploying daemon osd.4 on smithi028 2024-08-28T19:14:33.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:32 smithi016 ceph-mon[18556]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-08-28T19:14:33.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:32 smithi028 ceph-mon[21769]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-08-28T19:14:33.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:32 smithi115 ceph-mon[22184]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-08-28T19:14:34.971 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:34 smithi028 ceph-mon[21769]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:35.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:34 smithi115 ceph-mon[22184]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:35.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:34 smithi016 ceph-mon[18556]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:35.940 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:35 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:35.940 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:35 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:35.940 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:35 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:36.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:35 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:36.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:35 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:36.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:35 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:36.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:35 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:36.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:35 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:36.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:35 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:36 smithi028 ceph-mon[21769]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:36 smithi115 ceph-mon[22184]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:37.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:36 smithi016 ceph-mon[18556]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:39.058 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:38 smithi028 ceph-mon[21769]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:38 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:38 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:38 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:38 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:38 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:39.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:38 smithi016 ceph-mon[18556]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:39.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:38 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:39.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:38 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:39.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:38 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:39.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:38 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:39.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:38 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:38 smithi115 ceph-mon[22184]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:38 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:38 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:38 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:38 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:38 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.412+0000 7ff946ffd640 1 -- 172.21.15.28:0/3471434847 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff9500635a0 con 0x7ff93c0b4c00 2024-08-28T19:14:39.414 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 4 on host 'smithi028' 2024-08-28T19:14:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 -- 172.21.15.28:0/3471434847 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff93c0b4c00 msgr2=0x7ff93c0b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:14:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff93c0b4c00 0x7ff93c0b7110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff950102c20 tx=0x7ff94000b390 comp rx=0 tx=0).stop 2024-08-28T19:14:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 -- 172.21.15.28:0/3471434847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff950103940 msgr2=0x7ff9500754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:14:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff950103940 0x7ff9500754b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff94c00e9f0 tx=0x7ff94c00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:14:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 -- 172.21.15.28:0/3471434847 shutdown_connections 2024-08-28T19:14:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff950109bd0 0x7ff950075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff950109280 0x7ff9500759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff93c0b4c00 0x7ff93c0b7110 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 --2- 172.21.15.28:0/3471434847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff950103940 0x7ff9500754b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 -- 172.21.15.28:0/3471434847 >> 172.21.15.28:0/3471434847 conn(0x7ff9500ff3d0 msgr2=0x7ff950107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:14:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 -- 172.21.15.28:0/3471434847 shutdown_connections 2024-08-28T19:14:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:14:39.415+0000 7ff95801c640 1 -- 172.21.15.28:0/3471434847 wait complete. 2024-08-28T19:14:40.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:39 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:40.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:39 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:40.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:39 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:40.164 DEBUG:teuthology.orchestra.run.smithi028:osd.4> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.4.service 2024-08-28T19:14:40.166 INFO:tasks.cephadm:Deploying osd.5 on smithi115 with /dev/vg_nvme/lv_3... 2024-08-28T19:14:40.167 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-28T19:14:40.195 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:39 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:40.195 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:39 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:40.195 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:39 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:40.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:39 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:14:40.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:39 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:40.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:39 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:40.417 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:14:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:40 smithi115 ceph-mon[22184]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:40 smithi115 ceph-mon[22184]: from='osd.4 [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-28T19:14:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:40 smithi115 ceph-mon[22184]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-28T19:14:41.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:40 smithi016 ceph-mon[18556]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:40 smithi016 ceph-mon[18556]: from='osd.4 [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-28T19:14:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:40 smithi016 ceph-mon[18556]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-28T19:14:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:40 smithi028 ceph-mon[21769]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:40 smithi028 ceph-mon[21769]: from='osd.4 [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-28T19:14:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:40 smithi028 ceph-mon[21769]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-28T19:14:41.310 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:14:40 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4[34221]: 2024-08-28T19:14:40.855+0000 7fd922a23640 -1 osd.4 0 log_to_monitors true 2024-08-28T19:14:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:41 smithi115 ceph-mon[22184]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-28T19:14:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:41 smithi115 ceph-mon[22184]: from='osd.4 [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:14:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:41 smithi115 ceph-mon[22184]: osdmap e27: 5 total, 4 up, 5 in 2024-08-28T19:14:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:41 smithi115 ceph-mon[22184]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:14:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:41 smithi115 ceph-mon[22184]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:14:42.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:41 smithi016 ceph-mon[18556]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-28T19:14:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:41 smithi016 ceph-mon[18556]: from='osd.4 [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:14:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:41 smithi016 ceph-mon[18556]: osdmap e27: 5 total, 4 up, 5 in 2024-08-28T19:14:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:41 smithi016 ceph-mon[18556]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:14:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:41 smithi016 ceph-mon[18556]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:14:42.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:41 smithi028 ceph-mon[21769]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-28T19:14:42.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:41 smithi028 ceph-mon[21769]: from='osd.4 [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:14:42.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:41 smithi028 ceph-mon[21769]: osdmap e27: 5 total, 4 up, 5 in 2024-08-28T19:14:42.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:42.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:41 smithi028 ceph-mon[21769]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:14:42.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:41 smithi028 ceph-mon[21769]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:14:42.809 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:14:42 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4[34221]: 2024-08-28T19:14:42.518+0000 7fd91e9a5640 -1 osd.4 0 waiting for initial osdmap 2024-08-28T19:14:42.809 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:14:42 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4[34221]: 2024-08-28T19:14:42.526+0000 7fd91679b640 -1 osd.4 28 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:14:43.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:42 smithi115 ceph-mon[22184]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:43.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:42 smithi115 ceph-mon[22184]: osdmap e28: 5 total, 4 up, 5 in 2024-08-28T19:14:43.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:43.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:43.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:42 smithi115 ceph-mon[22184]: from='osd.4 ' entity='osd.4' 2024-08-28T19:14:43.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:43.142 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:14:43.167 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi115:vg_nvme/lv_3 2024-08-28T19:14:43.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:42 smithi016 ceph-mon[18556]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:43.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:42 smithi016 ceph-mon[18556]: osdmap e28: 5 total, 4 up, 5 in 2024-08-28T19:14:43.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:43.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:43.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:42 smithi016 ceph-mon[18556]: from='osd.4 ' entity='osd.4' 2024-08-28T19:14:43.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:42 smithi028 ceph-mon[21769]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:42 smithi028 ceph-mon[21769]: osdmap e28: 5 total, 4 up, 5 in 2024-08-28T19:14:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:42 smithi028 ceph-mon[21769]: from='osd.4 ' entity='osd.4' 2024-08-28T19:14:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:43.456 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:14:43.993 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.990+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3945543197 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ee0108ac0 msgr2=0x7f0ee010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:14:43.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.990+0000 7f0ee57ae640 1 --2- 172.21.15.115:0/3945543197 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ee0108ac0 0x7f0ee010f2f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8009ca0 tx=0x7f0ed802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:14:43.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.990+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3945543197 shutdown_connections 2024-08-28T19:14:43.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.990+0000 7f0ee57ae640 1 --2- 172.21.15.115:0/3945543197 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ee0108ac0 0x7f0ee010f2f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:43.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.990+0000 7f0ee57ae640 1 --2- 172.21.15.115:0/3945543197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee0108170 0x7f0ee0108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:43.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.990+0000 7f0ee57ae640 1 --2- 172.21.15.115:0/3945543197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0102830 0x7f0ee0102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:43.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.990+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3945543197 >> 172.21.15.115:0/3945543197 conn(0x7f0ee00794f0 msgr2=0x7f0ee0079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:14:43.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.990+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3945543197 shutdown_connections 2024-08-28T19:14:43.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.990+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3945543197 wait complete. 2024-08-28T19:14:43.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.990+0000 7f0ee57ae640 1 Processor -- start 2024-08-28T19:14:43.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0ee57ae640 1 -- start start 2024-08-28T19:14:43.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0ee57ae640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ee0102830 0x7f0ee0074c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:14:43.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0ee57ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0108170 0x7f0ee006df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:14:43.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0ee57ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee0108ac0 0x7f0ee006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:14:43.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0ee57ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee0113850 con 0x7f0ee0108ac0 2024-08-28T19:14:43.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0ee57ae640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01136d0 con 0x7f0ee0108170 2024-08-28T19:14:43.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0ee57ae640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01139d0 con 0x7f0ee0102830 2024-08-28T19:14:43.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ee0102830 0x7f0ee0074c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:14:43.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ee0102830 0x7f0ee0074c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:47770/0 (socket says 172.21.15.115:47770) 2024-08-28T19:14:43.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0edeffd640 1 -- 172.21.15.115:0/3347419943 learned_addr learned my addr 172.21.15.115:0/3347419943 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-08-28T19:14:43.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0edf7fe640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee0108ac0 0x7f0ee006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0ede7fc640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0108170 0x7f0ee006df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0edeffd640 1 -- 172.21.15.115:0/3347419943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0108170 msgr2=0x7f0ee006df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0edeffd640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0108170 0x7f0ee006df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0edeffd640 1 -- 172.21.15.115:0/3347419943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee0108ac0 msgr2=0x7f0ee006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0edeffd640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee0108ac0 0x7f0ee006e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.991+0000 7f0edeffd640 1 -- 172.21.15.115:0/3347419943 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee006ed10 con 0x7f0ee0102830 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.992+0000 7f0ede7fc640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0108170 0x7f0ee006df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.992+0000 7f0edeffd640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ee0102830 0x7f0ee0074c50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000e9f0 tx=0x7f0ed000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.992+0000 7f0ec7fff640 1 -- 172.21.15.115:0/3347419943 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed000cd80 con 0x7f0ee0102830 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.992+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3347419943 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee00739e0 con 0x7f0ee0102830 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.992+0000 7f0ec7fff640 1 -- 172.21.15.115:0/3347419943 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ed00108e0 con 0x7f0ee0102830 2024-08-28T19:14:43.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.992+0000 7f0ec7fff640 1 -- 172.21.15.115:0/3347419943 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0018980 con 0x7f0ee0102830 2024-08-28T19:14:43.997 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.992+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3347419943 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee0073ed0 con 0x7f0ee0102830 2024-08-28T19:14:43.997 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.993+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3347419943 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee0103ab0 con 0x7f0ee0102830 2024-08-28T19:14:43.997 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.995+0000 7f0ec7fff640 1 -- 172.21.15.115:0/3347419943 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f0ed0010430 con 0x7f0ee0102830 2024-08-28T19:14:43.997 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.996+0000 7f0ec7fff640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ec80b8f90 0x7f0ec80bb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:14:43.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.997+0000 7f0ec7fff640 1 -- 172.21.15.115:0/3347419943 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4316+0+0 (secure 0 0 0) 0x7f0ed00e4b30 con 0x7f0ee0102830 2024-08-28T19:14:43.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.997+0000 7f0ede7fc640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ec80b8f90 0x7f0ec80bb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:14:43.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.997+0000 7f0ede7fc640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ec80b8f90 0x7f0ec80bb4a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4005fd0 tx=0x7f0ed4005850 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:14:44.001 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:43.999+0000 7f0ec7fff640 1 -- 172.21.15.115:0/3347419943 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f0ed00acb60 con 0x7f0ee0102830 2024-08-28T19:14:44.186 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:14:44.184+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3347419943 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0ee0074130 con 0x7f0ec80b8f90 2024-08-28T19:14:44.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:14:44.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:14:44.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:44.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi028 2024-08-28T19:14:44.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:44.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:44.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi028 to 8546M 2024-08-28T19:14:44.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:44.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:44.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: osd.4 [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969] boot 2024-08-28T19:14:44.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: osdmap e29: 5 total, 5 up, 5 in 2024-08-28T19:14:44.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:44.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:14:44.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:14:44.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:14:44.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:14:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:14:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi028 2024-08-28T19:14:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi028 to 8546M 2024-08-28T19:14:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:44.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:44.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: osd.4 [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969] boot 2024-08-28T19:14:44.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: osdmap e29: 5 total, 5 up, 5 in 2024-08-28T19:14:44.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:44.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:14:44.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:14:44.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:14:44.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:14:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:14:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 357 GiB / 358 GiB avail 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi028 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi028 to 8546M 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: osd.4 [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969] boot 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: osdmap e29: 5 total, 5 up, 5 in 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-28T19:14:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:14:44.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:14:44.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:14:44.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:45 smithi115 ceph-mon[22184]: osdmap e30: 5 total, 5 up, 5 in 2024-08-28T19:14:45.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:45 smithi028 ceph-mon[21769]: osdmap e30: 5 total, 5 up, 5 in 2024-08-28T19:14:46.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:45 smithi016 ceph-mon[18556]: osdmap e30: 5 total, 5 up, 5 in 2024-08-28T19:14:46.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:46 smithi028 ceph-mon[21769]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:46.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:46 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/1723922807' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "762a439b-bec1-471e-ab08-aa233ee83d7a"} : dispatch 2024-08-28T19:14:46.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:46 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "762a439b-bec1-471e-ab08-aa233ee83d7a"} : dispatch 2024-08-28T19:14:46.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:46 smithi115 ceph-mon[22184]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:46.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:46 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/1723922807' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "762a439b-bec1-471e-ab08-aa233ee83d7a"} : dispatch 2024-08-28T19:14:46.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:46 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "762a439b-bec1-471e-ab08-aa233ee83d7a"} : dispatch 2024-08-28T19:14:47.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:46 smithi016 ceph-mon[18556]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:47.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:46 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/1723922807' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "762a439b-bec1-471e-ab08-aa233ee83d7a"} : dispatch 2024-08-28T19:14:47.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:46 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "762a439b-bec1-471e-ab08-aa233ee83d7a"} : dispatch 2024-08-28T19:14:47.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:47 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "762a439b-bec1-471e-ab08-aa233ee83d7a"}]': finished 2024-08-28T19:14:47.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:47 smithi028 ceph-mon[21769]: osdmap e31: 6 total, 5 up, 6 in 2024-08-28T19:14:47.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:47 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:14:47.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:47 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/534102499' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:14:47.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:47 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "762a439b-bec1-471e-ab08-aa233ee83d7a"}]': finished 2024-08-28T19:14:47.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:47 smithi115 ceph-mon[22184]: osdmap e31: 6 total, 5 up, 6 in 2024-08-28T19:14:47.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:47 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:14:47.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:47 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/534102499' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:14:47.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:47 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "762a439b-bec1-471e-ab08-aa233ee83d7a"}]': finished 2024-08-28T19:14:47.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:47 smithi016 ceph-mon[18556]: osdmap e31: 6 total, 5 up, 6 in 2024-08-28T19:14:47.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:47 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:14:47.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:47 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/534102499' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:14:48.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:48 smithi016 ceph-mon[18556]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:48.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:48 smithi028 ceph-mon[21769]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:48 smithi115 ceph-mon[22184]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:50 smithi028 ceph-mon[21769]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:50 smithi115 ceph-mon[22184]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:51.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:50 smithi016 ceph-mon[18556]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:52.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:52 smithi028 ceph-mon[21769]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:52 smithi115 ceph-mon[22184]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:53.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:52 smithi016 ceph-mon[18556]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:54.661 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:54 smithi115 ceph-mon[22184]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:54.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:54 smithi028 ceph-mon[21769]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:55.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:54 smithi016 ceph-mon[18556]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:55.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-28T19:14:55.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:55.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:55 smithi028 ceph-mon[21769]: Deploying daemon osd.5 on smithi115 2024-08-28T19:14:55.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-28T19:14:55.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:55.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:55 smithi115 ceph-mon[22184]: Deploying daemon osd.5 on smithi115 2024-08-28T19:14:56.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-28T19:14:56.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:14:56.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:55 smithi016 ceph-mon[18556]: Deploying daemon osd.5 on smithi115 2024-08-28T19:14:56.671 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:56 smithi115 ceph-mon[22184]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:56 smithi028 ceph-mon[21769]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:57.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:56 smithi016 ceph-mon[18556]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:58.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:14:58 smithi115 ceph-mon[22184]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:58.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:14:58 smithi028 ceph-mon[21769]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:14:59.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:14:58 smithi016 ceph-mon[18556]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:00 smithi115 ceph-mon[22184]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:01.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:00 smithi016 ceph-mon[18556]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:01.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:01.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:01.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:00 smithi028 ceph-mon[21769]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:02.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:02 smithi115 ceph-mon[22184]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:03.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:02 smithi016 ceph-mon[18556]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:03.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:02 smithi028 ceph-mon[21769]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:03 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:03 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:03 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:03 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:03 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:03.957 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 5 on host 'smithi115' 2024-08-28T19:15:03.958 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.955+0000 7f0ec7fff640 1 -- 172.21.15.115:0/3347419943 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0ee0074130 con 0x7f0ec80b8f90 2024-08-28T19:15:03.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.957+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3347419943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ec80b8f90 msgr2=0x7f0ec80bb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:03.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.957+0000 7f0ee57ae640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ec80b8f90 0x7f0ec80bb4a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4005fd0 tx=0x7f0ed4005850 comp rx=0 tx=0).stop 2024-08-28T19:15:03.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.957+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3347419943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ee0102830 msgr2=0x7f0ee0074c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:03.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.957+0000 7f0ee57ae640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ee0102830 0x7f0ee0074c50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000e9f0 tx=0x7f0ed000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:15:03.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.957+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3347419943 shutdown_connections 2024-08-28T19:15:03.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.957+0000 7f0ee57ae640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee0108ac0 0x7f0ee006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:03.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.957+0000 7f0ee57ae640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ec80b8f90 0x7f0ec80bb4a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:03.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.957+0000 7f0ee57ae640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0108170 0x7f0ee006df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:03.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.957+0000 7f0ee57ae640 1 --2- 172.21.15.115:0/3347419943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ee0102830 0x7f0ee0074c50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:03.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.957+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3347419943 >> 172.21.15.115:0/3347419943 conn(0x7f0ee00794f0 msgr2=0x7f0ee01041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:15:03.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.958+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3347419943 shutdown_connections 2024-08-28T19:15:03.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:03.958+0000 7f0ee57ae640 1 -- 172.21.15.115:0/3347419943 wait complete. 2024-08-28T19:15:04.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:03 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:04.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:03 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:04.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:03 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:04.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:03 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:04.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:03 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:04.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:03 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:04.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:03 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:04.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:03 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:04.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:03 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:04.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:03 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:04.571 DEBUG:teuthology.orchestra.run.smithi115:osd.5> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.5.service 2024-08-28T19:15:04.574 INFO:tasks.cephadm:Deploying osd.6 on smithi016 with /dev/vg_nvme/lv_2... 2024-08-28T19:15:04.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-28T19:15:04.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:04 smithi115 ceph-mon[22184]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:04.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:04 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:04.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:04 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:04.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:04 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:04.836 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:15:04.956 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:04 smithi016 ceph-mon[18556]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:04.957 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:04 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:04.957 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:04 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:04.957 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:04 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:04 smithi028 ceph-mon[21769]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:04 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:04 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:04 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:06.372 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:15:06 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:15:06.141+0000 7f0941b58640 -1 osd.5 0 log_to_monitors true 2024-08-28T19:15:06.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:06 smithi016 ceph-mon[18556]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:06.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:06 smithi016 ceph-mon[18556]: from='osd.5 [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-28T19:15:06.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:06 smithi016 ceph-mon[18556]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-28T19:15:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:06 smithi115 ceph-mon[22184]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:06 smithi115 ceph-mon[22184]: from='osd.5 [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-28T19:15:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:06 smithi115 ceph-mon[22184]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-28T19:15:07.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:06 smithi028 ceph-mon[21769]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:07.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:06 smithi028 ceph-mon[21769]: from='osd.5 [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-28T19:15:07.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:06 smithi028 ceph-mon[21769]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-28T19:15:07.477 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:15:07.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-08-28T19:15:07.763 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:15:07.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-28T19:15:07.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: osdmap e32: 6 total, 5 up, 6 in 2024-08-28T19:15:07.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:07.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='osd.5 [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:15:07.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:15:07.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:07.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:07.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:07.850 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:07.850 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:07.850 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:07.850 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:07.850 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:07.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:15:07.624+0000 7f093dada640 -1 osd.5 0 waiting for initial osdmap 2024-08-28T19:15:07.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:15:07.630+0000 7f09398d8640 -1 osd.5 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:15:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-28T19:15:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: osdmap e32: 6 total, 5 up, 6 in 2024-08-28T19:15:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='osd.5 [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:15:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:15:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-28T19:15:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: osdmap e32: 6 total, 5 up, 6 in 2024-08-28T19:15:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='osd.5 [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:15:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:15:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:08.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:08.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:08.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.204+0000 7fd525217640 1 -- 172.21.15.16:0/572957636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd520109250 msgr2=0x7fd520109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.204+0000 7fd525217640 1 --2- 172.21.15.16:0/572957636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd520109250 0x7fd520109660 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd514009900 tx=0x7fd51402a410 comp rx=0 tx=0).stop 2024-08-28T19:15:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.204+0000 7fd525217640 1 -- 172.21.15.16:0/572957636 shutdown_connections 2024-08-28T19:15:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.204+0000 7fd525217640 1 --2- 172.21.15.16:0/572957636 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd520109ba0 0x7fd5201103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.204+0000 7fd525217640 1 --2- 172.21.15.16:0/572957636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd520109250 0x7fd520109660 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.204+0000 7fd525217640 1 --2- 172.21.15.16:0/572957636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd520101d10 0x7fd520102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.204+0000 7fd525217640 1 -- 172.21.15.16:0/572957636 >> 172.21.15.16:0/572957636 conn(0x7fd520077a00 msgr2=0x7fd520077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:15:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.204+0000 7fd525217640 1 -- 172.21.15.16:0/572957636 shutdown_connections 2024-08-28T19:15:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.204+0000 7fd525217640 1 -- 172.21.15.16:0/572957636 wait complete. 2024-08-28T19:15:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.205+0000 7fd525217640 1 Processor -- start 2024-08-28T19:15:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.205+0000 7fd525217640 1 -- start start 2024-08-28T19:15:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.205+0000 7fd525217640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd520101d10 0x7fd52019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.205+0000 7fd525217640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd520109250 0x7fd52019a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.205+0000 7fd525217640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd520109ba0 0x7fd52019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.205+0000 7fd525217640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd520113860 con 0x7fd520109250 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.205+0000 7fd525217640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5201136e0 con 0x7fd520109ba0 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.205+0000 7fd525217640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd5201139e0 con 0x7fd520101d10 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd51e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd520109250 0x7fd52019a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd51e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd520109250 0x7fd52019a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59602/0 (socket says 172.21.15.16:59602) 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd51e59c640 1 -- 172.21.15.16:0/2858333766 learned_addr learned my addr 172.21.15.16:0/2858333766 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd51ed9d640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd520101d10 0x7fd52019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd51f59e640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd520109ba0 0x7fd52019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd51e59c640 1 -- 172.21.15.16:0/2858333766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd520101d10 msgr2=0x7fd52019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd51e59c640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd520101d10 0x7fd52019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd51e59c640 1 -- 172.21.15.16:0/2858333766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd520109ba0 msgr2=0x7fd52019fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd51e59c640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd520109ba0 0x7fd52019fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd51e59c640 1 -- 172.21.15.16:0/2858333766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5201a0260 con 0x7fd520109250 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd51e59c640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd520109250 0x7fd52019a890 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd514006010 tx=0x7fd514004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd507fff640 1 -- 172.21.15.16:0/2858333766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51403f030 con 0x7fd520109250 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd525217640 1 -- 172.21.15.16:0/2858333766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5201a04f0 con 0x7fd520109250 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.206+0000 7fd525217640 1 -- 172.21.15.16:0/2858333766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5201a6cb0 con 0x7fd520109250 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.207+0000 7fd507fff640 1 -- 172.21.15.16:0/2858333766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd514003aa0 con 0x7fd520109250 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.207+0000 7fd507fff640 1 -- 172.21.15.16:0/2858333766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd514049060 con 0x7fd520109250 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.208+0000 7fd525217640 1 -- 172.21.15.16:0/2858333766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd50c0052c0 con 0x7fd520109250 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.210+0000 7fd507fff640 1 -- 172.21.15.16:0/2858333766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fd514042030 con 0x7fd520109250 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.210+0000 7fd507fff640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd5080b4b80 0x7fd5080b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.211+0000 7fd51ed9d640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd5080b4b80 0x7fd5080b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.211+0000 7fd507fff640 1 -- 172.21.15.16:0/2858333766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4468+0+0 (secure 0 0 0) 0x7fd514103ea0 con 0x7fd520109250 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.211+0000 7fd51ed9d640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd5080b4b80 0x7fd5080b7090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd510005d00 tx=0x7fd51000b410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:15:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.215+0000 7fd507fff640 1 -- 172.21.15.16:0/2858333766 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fd5140cb7e0 con 0x7fd520109250 2024-08-28T19:15:08.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:08.414+0000 7fd525217640 1 -- 172.21.15.16:0/2858333766 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fd50c000bf0 con 0x7fd5080b4b80 2024-08-28T19:15:08.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi115 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi115 to 8544M 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: osdmap e33: 6 total, 5 up, 6 in 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: osd.5 [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596] boot 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: osdmap e34: 6 total, 6 up, 6 in 2024-08-28T19:15:08.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi115 2024-08-28T19:15:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi115 to 8544M 2024-08-28T19:15:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:15:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: osdmap e33: 6 total, 5 up, 6 in 2024-08-28T19:15:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:15:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:15:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: osd.5 [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596] boot 2024-08-28T19:15:09.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: osdmap e34: 6 total, 6 up, 6 in 2024-08-28T19:15:09.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi115 2024-08-28T19:15:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi115 to 8544M 2024-08-28T19:15:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 447 GiB / 447 GiB avail 2024-08-28T19:15:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:15:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: osdmap e33: 6 total, 5 up, 6 in 2024-08-28T19:15:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:15:09.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:15:09.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:09.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: osd.5 [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596] boot 2024-08-28T19:15:09.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: osdmap e34: 6 total, 6 up, 6 in 2024-08-28T19:15:09.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-28T19:15:09.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:09 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:15:09.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:09 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:15:09.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:09 smithi016 ceph-mon[18556]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:15:09.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:09 smithi016 ceph-mon[18556]: osdmap e35: 6 total, 6 up, 6 in 2024-08-28T19:15:10.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:09 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:15:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:09 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:15:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:09 smithi028 ceph-mon[21769]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:15:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:09 smithi028 ceph-mon[21769]: osdmap e35: 6 total, 6 up, 6 in 2024-08-28T19:15:10.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:09 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:15:10.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:09 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:15:10.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:09 smithi115 ceph-mon[22184]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:15:10.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:09 smithi115 ceph-mon[22184]: osdmap e35: 6 total, 6 up, 6 in 2024-08-28T19:15:11.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:10 smithi016 ceph-mon[18556]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:11.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:10 smithi016 ceph-mon[18556]: osdmap e36: 6 total, 6 up, 6 in 2024-08-28T19:15:11.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:10 smithi028 ceph-mon[21769]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:11.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:10 smithi028 ceph-mon[21769]: osdmap e36: 6 total, 6 up, 6 in 2024-08-28T19:15:11.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:10 smithi115 ceph-mon[22184]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:11.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:10 smithi115 ceph-mon[22184]: osdmap e36: 6 total, 6 up, 6 in 2024-08-28T19:15:12.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:11 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1465269018' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33c56c57-ce44-4000-94b3-518d7ce0afd1"} : dispatch 2024-08-28T19:15:12.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:11 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1465269018' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33c56c57-ce44-4000-94b3-518d7ce0afd1"}]': finished 2024-08-28T19:15:12.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:11 smithi016 ceph-mon[18556]: osdmap e37: 7 total, 6 up, 7 in 2024-08-28T19:15:12.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:12.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:11 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/423254442' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:15:12.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:11 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1465269018' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33c56c57-ce44-4000-94b3-518d7ce0afd1"} : dispatch 2024-08-28T19:15:12.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:11 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1465269018' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33c56c57-ce44-4000-94b3-518d7ce0afd1"}]': finished 2024-08-28T19:15:12.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:11 smithi028 ceph-mon[21769]: osdmap e37: 7 total, 6 up, 7 in 2024-08-28T19:15:12.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:12.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:11 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/423254442' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:15:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:11 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1465269018' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33c56c57-ce44-4000-94b3-518d7ce0afd1"} : dispatch 2024-08-28T19:15:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:11 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1465269018' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33c56c57-ce44-4000-94b3-518d7ce0afd1"}]': finished 2024-08-28T19:15:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:11 smithi115 ceph-mon[22184]: osdmap e37: 7 total, 6 up, 7 in 2024-08-28T19:15:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:11 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/423254442' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:15:13.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:12 smithi016 ceph-mon[18556]: pgmap v117: 1 pgs: 1 activating+degraded; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-08-28T19:15:13.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:12 smithi016 ceph-mon[18556]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-08-28T19:15:13.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:12 smithi028 ceph-mon[21769]: pgmap v117: 1 pgs: 1 activating+degraded; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-08-28T19:15:13.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:12 smithi028 ceph-mon[21769]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-08-28T19:15:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:12 smithi115 ceph-mon[22184]: pgmap v117: 1 pgs: 1 activating+degraded; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-08-28T19:15:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:12 smithi115 ceph-mon[22184]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-08-28T19:15:15.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:14 smithi016 ceph-mon[18556]: pgmap v118: 1 pgs: 1 activating+degraded; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-08-28T19:15:15.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:14 smithi028 ceph-mon[21769]: pgmap v118: 1 pgs: 1 activating+degraded; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-08-28T19:15:15.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:14 smithi115 ceph-mon[22184]: pgmap v118: 1 pgs: 1 activating+degraded; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-08-28T19:15:16.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:16 smithi016 ceph-mon[18556]: pgmap v119: 1 pgs: 1 activating+degraded; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-08-28T19:15:17.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:16 smithi028 ceph-mon[21769]: pgmap v119: 1 pgs: 1 activating+degraded; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-08-28T19:15:17.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:16 smithi115 ceph-mon[22184]: pgmap v119: 1 pgs: 1 activating+degraded; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-08-28T19:15:17.960 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:17 smithi028 ceph-mon[21769]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-08-28T19:15:17.961 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:17 smithi028 ceph-mon[21769]: Cluster is now healthy 2024-08-28T19:15:18.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:17 smithi016 ceph-mon[18556]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-08-28T19:15:18.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:17 smithi016 ceph-mon[18556]: Cluster is now healthy 2024-08-28T19:15:18.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:17 smithi115 ceph-mon[22184]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-08-28T19:15:18.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:17 smithi115 ceph-mon[22184]: Cluster is now healthy 2024-08-28T19:15:19.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:18 smithi016 ceph-mon[18556]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-08-28T19:15:19.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:18 smithi028 ceph-mon[21769]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-08-28T19:15:19.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:18 smithi115 ceph-mon[22184]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-08-28T19:15:20.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:20 smithi016 ceph-mon[18556]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-28T19:15:20.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:20 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-28T19:15:20.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:20 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:20.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:20 smithi016 ceph-mon[18556]: Deploying daemon osd.6 on smithi016 2024-08-28T19:15:21.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:20 smithi028 ceph-mon[21769]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-28T19:15:21.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:20 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-28T19:15:21.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:20 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:21.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:20 smithi028 ceph-mon[21769]: Deploying daemon osd.6 on smithi016 2024-08-28T19:15:21.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:20 smithi115 ceph-mon[22184]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-28T19:15:21.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:20 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-28T19:15:21.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:20 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:21.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:20 smithi115 ceph-mon[22184]: Deploying daemon osd.6 on smithi016 2024-08-28T19:15:22.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:22 smithi016 ceph-mon[18556]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-28T19:15:23.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:22 smithi028 ceph-mon[21769]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-28T19:15:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:22 smithi115 ceph-mon[22184]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-28T19:15:24.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:24 smithi016 ceph-mon[18556]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:15:25.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:24 smithi028 ceph-mon[21769]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:15:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:24 smithi115 ceph-mon[22184]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:15:26.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:26.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:26.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:26.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:26.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:26.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:26.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:26.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:26.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:27.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:26 smithi016 ceph-mon[18556]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:15:27.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:26 smithi028 ceph-mon[21769]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:15:27.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:26 smithi115 ceph-mon[22184]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:15:29.323 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 6 on host 'smithi016' 2024-08-28T19:15:29.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.322+0000 7fd507fff640 1 -- 172.21.15.16:0/2858333766 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd50c000bf0 con 0x7fd5080b4b80 2024-08-28T19:15:29.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.324+0000 7fd525217640 1 -- 172.21.15.16:0/2858333766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd5080b4b80 msgr2=0x7fd5080b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:29.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.324+0000 7fd525217640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd5080b4b80 0x7fd5080b7090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd510005d00 tx=0x7fd51000b410 comp rx=0 tx=0).stop 2024-08-28T19:15:29.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.324+0000 7fd525217640 1 -- 172.21.15.16:0/2858333766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd520109250 msgr2=0x7fd52019a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:29.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.324+0000 7fd525217640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd520109250 0x7fd52019a890 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd514006010 tx=0x7fd514004060 comp rx=0 tx=0).stop 2024-08-28T19:15:29.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.325+0000 7fd525217640 1 -- 172.21.15.16:0/2858333766 shutdown_connections 2024-08-28T19:15:29.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.325+0000 7fd525217640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd5080b4b80 0x7fd5080b7090 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:29.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.325+0000 7fd525217640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd520109ba0 0x7fd52019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:29.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.325+0000 7fd525217640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd520109250 0x7fd52019a890 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:29.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.325+0000 7fd525217640 1 --2- 172.21.15.16:0/2858333766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd520101d10 0x7fd52019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:29.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.325+0000 7fd525217640 1 -- 172.21.15.16:0/2858333766 >> 172.21.15.16:0/2858333766 conn(0x7fd520077a00 msgr2=0x7fd520107360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:15:29.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.325+0000 7fd525217640 1 -- 172.21.15.16:0/2858333766 shutdown_connections 2024-08-28T19:15:29.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:15:29.325+0000 7fd525217640 1 -- 172.21.15.16:0/2858333766 wait complete. 2024-08-28T19:15:29.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:29 smithi016 ceph-mon[18556]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:15:29.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:29.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:29.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:29.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:29.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:29 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:29 smithi028 ceph-mon[21769]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:15:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:29.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:29.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:29.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:29 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:29 smithi115 ceph-mon[22184]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:15:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:29 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:29.994 DEBUG:teuthology.orchestra.run.smithi016:osd.6> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.6.service 2024-08-28T19:15:29.996 INFO:tasks.cephadm:Deploying osd.7 on smithi028 with /dev/vg_nvme/lv_2... 2024-08-28T19:15:29.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-28T19:15:30.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:30.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:30.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:15:30.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:30.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:30.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:30.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:30.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:30.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:31.429 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:31 smithi016 ceph-mon[18556]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:31.517 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:31 smithi028 ceph-mon[21769]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:31.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:31 smithi115 ceph-mon[22184]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:31.751 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:15:31 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:15:31.517+0000 7fb7deec7640 -1 osd.6 0 log_to_monitors true 2024-08-28T19:15:32.380 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:32 smithi028 ceph-mon[21769]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-28T19:15:32.492 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:32 smithi016 ceph-mon[18556]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-28T19:15:32.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:32 smithi115 ceph-mon[22184]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-28T19:15:33.267 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:33.267 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-28T19:15:33.267 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: osdmap e38: 7 total, 6 up, 7 in 2024-08-28T19:15:33.267 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:15:33.267 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:33.267 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:33.267 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:33.267 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:33.267 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:33.267 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:33.267 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:33.268 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:33.268 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:33.268 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:15:33.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-08-28T19:15:33.501 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:15:33.178+0000 7fb7dae49640 -1 osd.6 0 waiting for initial osdmap 2024-08-28T19:15:33.501 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:15:33.186+0000 7fb7d6c47640 -1 osd.6 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:15:33.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:33.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-28T19:15:33.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: osdmap e38: 7 total, 6 up, 7 in 2024-08-28T19:15:33.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:15:33.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:33.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:33.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:33.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:33.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:33.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:33.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:33.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:33.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:33.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:15:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-28T19:15:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: osdmap e38: 7 total, 6 up, 7 in 2024-08-28T19:15:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:15:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:34.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.191+0000 7fcf61685640 1 -- 172.21.15.28:0/3770361660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf5c101d10 msgr2=0x7fcf5c102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:34.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.191+0000 7fcf61685640 1 --2- 172.21.15.28:0/3770361660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf5c101d10 0x7fcf5c102120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c009900 tx=0x7fcf4c02a410 comp rx=0 tx=0).stop 2024-08-28T19:15:34.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.191+0000 7fcf61685640 1 -- 172.21.15.28:0/3770361660 shutdown_connections 2024-08-28T19:15:34.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.191+0000 7fcf61685640 1 --2- 172.21.15.28:0/3770361660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf5c109ba0 0x7fcf5c1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:34.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.191+0000 7fcf61685640 1 --2- 172.21.15.28:0/3770361660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf5c109250 0x7fcf5c109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.191+0000 7fcf61685640 1 --2- 172.21.15.28:0/3770361660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf5c101d10 0x7fcf5c102120 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.191+0000 7fcf61685640 1 -- 172.21.15.28:0/3770361660 >> 172.21.15.28:0/3770361660 conn(0x7fcf5c077a00 msgr2=0x7fcf5c077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.191+0000 7fcf61685640 1 -- 172.21.15.28:0/3770361660 shutdown_connections 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.191+0000 7fcf61685640 1 -- 172.21.15.28:0/3770361660 wait complete. 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.191+0000 7fcf61685640 1 Processor -- start 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf61685640 1 -- start start 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf61685640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf5c101d10 0x7fcf5c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf61685640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf5c109250 0x7fcf5c19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf61685640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf5c109ba0 0x7fcf5c19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf61685640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c1138b0 con 0x7fcf5c101d10 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf61685640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c113730 con 0x7fcf5c109250 2024-08-28T19:15:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf61685640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c113a30 con 0x7fcf5c109ba0 2024-08-28T19:15:34.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf5a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf5c109250 0x7fcf5c19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:34.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf5a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf5c109250 0x7fcf5c19a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45766/0 (socket says 172.21.15.28:45766) 2024-08-28T19:15:34.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf5a7fc640 1 -- 172.21.15.28:0/3402562812 learned_addr learned my addr 172.21.15.28:0/3402562812 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-08-28T19:15:34.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf5b7fe640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf5c109ba0 0x7fcf5c19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:34.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.192+0000 7fcf5affd640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf5c101d10 0x7fcf5c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:34.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf5a7fc640 1 -- 172.21.15.28:0/3402562812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf5c109ba0 msgr2=0x7fcf5c19faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:34.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf5a7fc640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf5c109ba0 0x7fcf5c19faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:34.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf5a7fc640 1 -- 172.21.15.28:0/3402562812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf5c101d10 msgr2=0x7fcf5c19a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:34.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf5a7fc640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf5c101d10 0x7fcf5c19a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:34.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf5a7fc640 1 -- 172.21.15.28:0/3402562812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c1a01d0 con 0x7fcf5c109250 2024-08-28T19:15:34.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf5b7fe640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf5c109ba0 0x7fcf5c19faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf5a7fc640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf5c109250 0x7fcf5c19a800 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000e9f0 tx=0x7fcf5000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf43fff640 1 -- 172.21.15.28:0/3402562812 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5000ccc0 con 0x7fcf5c109250 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf43fff640 1 -- 172.21.15.28:0/3402562812 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf500108e0 con 0x7fcf5c109250 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf61685640 1 -- 172.21.15.28:0/3402562812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5c1a04c0 con 0x7fcf5c109250 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf43fff640 1 -- 172.21.15.28:0/3402562812 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50018860 con 0x7fcf5c109250 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.193+0000 7fcf61685640 1 -- 172.21.15.28:0/3402562812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf5c1a6c20 con 0x7fcf5c109250 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.194+0000 7fcf61685640 1 -- 172.21.15.28:0/3402562812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf5c102120 con 0x7fcf5c109250 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.196+0000 7fcf43fff640 1 -- 172.21.15.28:0/3402562812 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fcf50010430 con 0x7fcf5c109250 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.196+0000 7fcf43fff640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf440b4b80 0x7fcf440b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.197+0000 7fcf43fff640 1 -- 172.21.15.28:0/3402562812 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 5180+0+0 (secure 0 0 0) 0x7fcf500e4e60 con 0x7fcf5c109250 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.197+0000 7fcf5affd640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf440b4b80 0x7fcf440b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.198+0000 7fcf5affd640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf440b4b80 0x7fcf440b7090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c000c00 tx=0x7fcf4c031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:15:34.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.200+0000 7fcf43fff640 1 -- 172.21.15.28:0/3402562812 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fcf500acbd0 con 0x7fcf5c109250 2024-08-28T19:15:34.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:34.396+0000 7fcf61685640 1 -- 172.21.15.28:0/3402562812 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fcf5c0635a0 con 0x7fcf440b4b80 2024-08-28T19:15:34.397 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:34 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi016 2024-08-28T19:15:34.398 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:34 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi016 to 5697M 2024-08-28T19:15:34.398 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:34 smithi028 ceph-mon[21769]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:15:34.398 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:34 smithi028 ceph-mon[21769]: osdmap e39: 7 total, 6 up, 7 in 2024-08-28T19:15:34.398 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:34 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:34.398 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:34 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:34.398 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:34 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:34.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:34 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi016 2024-08-28T19:15:34.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:34 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi016 to 5697M 2024-08-28T19:15:34.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:34 smithi016 ceph-mon[18556]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:15:34.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:34 smithi016 ceph-mon[18556]: osdmap e39: 7 total, 6 up, 7 in 2024-08-28T19:15:34.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:34 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:34.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:34 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:34.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:34 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:34.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:34 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi016 2024-08-28T19:15:34.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:34 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi016 to 5697M 2024-08-28T19:15:34.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:34 smithi115 ceph-mon[22184]: from='osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:15:34.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:34 smithi115 ceph-mon[22184]: osdmap e39: 7 total, 6 up, 7 in 2024-08-28T19:15:34.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:34 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:34.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:34 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:34.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:34 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:35.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:35 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:15:35.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:35 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:15:35.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:35 smithi016 ceph-mon[18556]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:35.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:35 smithi016 ceph-mon[18556]: osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622] boot 2024-08-28T19:15:35.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:35 smithi016 ceph-mon[18556]: osdmap e40: 7 total, 7 up, 7 in 2024-08-28T19:15:35.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:35 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:35.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:35 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:15:35.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:35 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:15:35.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:35 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:35.543 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:35 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:15:35.543 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:35 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:15:35.543 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:35 smithi028 ceph-mon[21769]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:35.543 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:35 smithi028 ceph-mon[21769]: osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622] boot 2024-08-28T19:15:35.543 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:35 smithi028 ceph-mon[21769]: osdmap e40: 7 total, 7 up, 7 in 2024-08-28T19:15:35.543 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:35 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:35.543 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:35 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:15:35.544 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:35 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:15:35.544 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:35 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:35 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:15:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:35 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:15:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:35 smithi115 ceph-mon[22184]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-08-28T19:15:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:35 smithi115 ceph-mon[22184]: osd.6 [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622] boot 2024-08-28T19:15:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:35 smithi115 ceph-mon[22184]: osdmap e40: 7 total, 7 up, 7 in 2024-08-28T19:15:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:35 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-28T19:15:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:35 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:15:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:35 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:15:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:35 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:36.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:36 smithi016 ceph-mon[18556]: from='client.24434 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:15:36.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:36 smithi016 ceph-mon[18556]: osdmap e41: 7 total, 7 up, 7 in 2024-08-28T19:15:36.545 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:36 smithi028 ceph-mon[21769]: from='client.24434 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:15:36.545 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:36 smithi028 ceph-mon[21769]: osdmap e41: 7 total, 7 up, 7 in 2024-08-28T19:15:36.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:36 smithi115 ceph-mon[22184]: from='client.24434 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:15:36.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:36 smithi115 ceph-mon[22184]: osdmap e41: 7 total, 7 up, 7 in 2024-08-28T19:15:37.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:37 smithi016 ceph-mon[18556]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:37 smithi016 ceph-mon[18556]: from='client.? 172.21.15.28:0/2881093163' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8ef482ae-33bd-4121-8b16-c80ad1b9178b"} : dispatch 2024-08-28T19:15:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:37 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8ef482ae-33bd-4121-8b16-c80ad1b9178b"} : dispatch 2024-08-28T19:15:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:37 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ef482ae-33bd-4121-8b16-c80ad1b9178b"}]': finished 2024-08-28T19:15:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:37 smithi016 ceph-mon[18556]: osdmap e42: 8 total, 7 up, 8 in 2024-08-28T19:15:37.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:37 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:15:37.506 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:37 smithi028 ceph-mon[21769]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:37.506 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:37 smithi028 ceph-mon[21769]: from='client.? 172.21.15.28:0/2881093163' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8ef482ae-33bd-4121-8b16-c80ad1b9178b"} : dispatch 2024-08-28T19:15:37.506 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:37 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8ef482ae-33bd-4121-8b16-c80ad1b9178b"} : dispatch 2024-08-28T19:15:37.506 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:37 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ef482ae-33bd-4121-8b16-c80ad1b9178b"}]': finished 2024-08-28T19:15:37.506 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:37 smithi028 ceph-mon[21769]: osdmap e42: 8 total, 7 up, 8 in 2024-08-28T19:15:37.506 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:37 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:15:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:37 smithi115 ceph-mon[22184]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:37 smithi115 ceph-mon[22184]: from='client.? 172.21.15.28:0/2881093163' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8ef482ae-33bd-4121-8b16-c80ad1b9178b"} : dispatch 2024-08-28T19:15:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:37 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8ef482ae-33bd-4121-8b16-c80ad1b9178b"} : dispatch 2024-08-28T19:15:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:37 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ef482ae-33bd-4121-8b16-c80ad1b9178b"}]': finished 2024-08-28T19:15:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:37 smithi115 ceph-mon[22184]: osdmap e42: 8 total, 7 up, 8 in 2024-08-28T19:15:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:37 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:15:38.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:38 smithi016 ceph-mon[18556]: from='client.? 172.21.15.28:0/3695751491' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:15:38.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:38 smithi028 ceph-mon[21769]: from='client.? 172.21.15.28:0/3695751491' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:15:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:38 smithi115 ceph-mon[22184]: from='client.? 172.21.15.28:0/3695751491' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:15:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:39 smithi016 ceph-mon[18556]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:39 smithi028 ceph-mon[21769]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:39 smithi115 ceph-mon[22184]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:41.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:41 smithi016 ceph-mon[18556]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:41 smithi028 ceph-mon[21769]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:41.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:41 smithi115 ceph-mon[22184]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:43.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:43 smithi016 ceph-mon[18556]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:43.522 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:43 smithi028 ceph-mon[21769]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:43.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:43 smithi115 ceph-mon[22184]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:45.479 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:45 smithi028 ceph-mon[21769]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:45.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:45 smithi016 ceph-mon[18556]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:45 smithi115 ceph-mon[22184]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:46.329 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:46 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-28T19:15:46.329 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:46 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:46.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:46 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-28T19:15:46.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:46 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:46 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-28T19:15:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:46 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:47.438 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:47 smithi028 ceph-mon[21769]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:47.438 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:47 smithi028 ceph-mon[21769]: Deploying daemon osd.7 on smithi028 2024-08-28T19:15:47.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:47 smithi016 ceph-mon[18556]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:47.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:47 smithi016 ceph-mon[18556]: Deploying daemon osd.7 on smithi028 2024-08-28T19:15:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:47 smithi115 ceph-mon[22184]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:47 smithi115 ceph-mon[22184]: Deploying daemon osd.7 on smithi028 2024-08-28T19:15:49.369 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:49 smithi028 ceph-mon[21769]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:49.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:49 smithi016 ceph-mon[18556]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:49 smithi115 ceph-mon[22184]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:51.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:51 smithi016 ceph-mon[18556]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:51.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:51 smithi028 ceph-mon[21769]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:51 smithi115 ceph-mon[22184]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:52.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:52.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:52.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:52.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:52.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:52.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:52.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:52.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:52.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:53.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:53 smithi016 ceph-mon[18556]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:53.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:53 smithi028 ceph-mon[21769]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:53.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:53 smithi115 ceph-mon[22184]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:55 smithi028 ceph-mon[21769]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:55.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:55.432 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 7 on host 'smithi028' 2024-08-28T19:15:55.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.430+0000 7fcf43fff640 1 -- 172.21.15.28:0/3402562812 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcf5c0635a0 con 0x7fcf440b4b80 2024-08-28T19:15:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 -- 172.21.15.28:0/3402562812 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf440b4b80 msgr2=0x7fcf440b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf440b4b80 0x7fcf440b7090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c000c00 tx=0x7fcf4c031400 comp rx=0 tx=0).stop 2024-08-28T19:15:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 -- 172.21.15.28:0/3402562812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf5c109250 msgr2=0x7fcf5c19a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf5c109250 0x7fcf5c19a800 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000e9f0 tx=0x7fcf5000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:15:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 -- 172.21.15.28:0/3402562812 shutdown_connections 2024-08-28T19:15:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf5c109ba0 0x7fcf5c19faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf440b4b80 0x7fcf440b7090 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf5c109250 0x7fcf5c19a800 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 --2- 172.21.15.28:0/3402562812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf5c101d10 0x7fcf5c19a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 -- 172.21.15.28:0/3402562812 >> 172.21.15.28:0/3402562812 conn(0x7fcf5c077a00 msgr2=0x7fcf5c107340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:15:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 -- 172.21.15.28:0/3402562812 shutdown_connections 2024-08-28T19:15:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:15:55.432+0000 7fcf61685640 1 -- 172.21.15.28:0/3402562812 wait complete. 2024-08-28T19:15:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:55 smithi016 ceph-mon[18556]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:55.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:55 smithi115 ceph-mon[22184]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:56.086 DEBUG:teuthology.orchestra.run.smithi028:osd.7> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.7.service 2024-08-28T19:15:56.089 INFO:tasks.cephadm:Deploying osd.8 on smithi115 with /dev/vg_nvme/lv_2... 2024-08-28T19:15:56.089 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-28T19:15:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:56 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:56 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:56 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:56.343 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:15:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:56 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:56 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:56 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:56.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:56 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:15:56.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:56 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:56.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:56 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:57.280 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:57 smithi028 ceph-mon[21769]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:57.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:57 smithi115 ceph-mon[22184]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:57.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:57 smithi016 ceph-mon[18556]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:57.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:15:57 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:15:57.277+0000 7faa2237d640 -1 osd.7 0 log_to_monitors true 2024-08-28T19:15:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:58 smithi115 ceph-mon[22184]: from='osd.7 [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-28T19:15:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:58 smithi115 ceph-mon[22184]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-28T19:15:58.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:58 smithi016 ceph-mon[18556]: from='osd.7 [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-28T19:15:58.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:58 smithi016 ceph-mon[18556]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-28T19:15:58.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:58 smithi028 ceph-mon[21769]: from='osd.7 [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-28T19:15:58.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:58 smithi028 ceph-mon[21769]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-28T19:15:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:15:58.854 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi115:vg_nvme/lv_2 2024-08-28T19:15:59.105 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:15:59.495 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:59.495 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='osd.7 [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: osdmap e43: 8 total, 7 up, 8 in 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:59.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:15:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-28T19:15:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='osd.7 [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:15:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: osdmap e43: 8 total, 7 up, 8 in 2024-08-28T19:15:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:15:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:15:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:15:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.527 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:15:59.527 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-28T19:15:59.527 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='osd.7 [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:15:59.527 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: osdmap e43: 8 total, 7 up, 8 in 2024-08-28T19:15:59.528 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:15:59.528 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:15:59.528 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.528 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.528 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:59.528 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:59.528 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:15:59.528 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.528 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:15:59.528 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:15:59.528 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:15:59.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.587+0000 7f316279a640 1 -- 172.21.15.115:0/2969738348 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f315c109b90 msgr2=0x7f315c1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:59.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.587+0000 7f316279a640 1 --2- 172.21.15.115:0/2969738348 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f315c109b90 0x7f315c1103c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3154009ca0 tx=0x7f315402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:15:59.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.587+0000 7f316279a640 1 -- 172.21.15.115:0/2969738348 shutdown_connections 2024-08-28T19:15:59.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.587+0000 7f316279a640 1 --2- 172.21.15.115:0/2969738348 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f315c109b90 0x7f315c1103c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:59.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.587+0000 7f316279a640 1 --2- 172.21.15.115:0/2969738348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c109240 0x7f315c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:59.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.587+0000 7f316279a640 1 --2- 172.21.15.115:0/2969738348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c103900 0x7f315c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:59.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.587+0000 7f316279a640 1 -- 172.21.15.115:0/2969738348 >> 172.21.15.115:0/2969738348 conn(0x7f315c07b5f0 msgr2=0x7f315c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:15:59.590 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.587+0000 7f316279a640 1 -- 172.21.15.115:0/2969738348 shutdown_connections 2024-08-28T19:15:59.590 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.587+0000 7f316279a640 1 -- 172.21.15.115:0/2969738348 wait complete. 2024-08-28T19:15:59.590 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.587+0000 7f316279a640 1 Processor -- start 2024-08-28T19:15:59.590 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.587+0000 7f316279a640 1 -- start start 2024-08-28T19:15:59.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f316279a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f315c103900 0x7f315c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:59.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f316279a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c109240 0x7f315c19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:59.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f316279a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c109b90 0x7f315c19fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:59.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f316279a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f315c113850 con 0x7f315c109240 2024-08-28T19:15:59.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f316279a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f315c1136d0 con 0x7f315c109b90 2024-08-28T19:15:59.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f316279a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f315c1139d0 con 0x7f315c103900 2024-08-28T19:15:59.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f315bfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f315c103900 0x7f315c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:59.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f315bfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f315c103900 0x7f315c19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51686/0 (socket says 172.21.15.115:51686) 2024-08-28T19:15:59.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f315bfff640 1 -- 172.21.15.115:0/2166101794 learned_addr learned my addr 172.21.15.115:0/2166101794 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-08-28T19:15:59.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f3160d37640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c109b90 0x7f315c19fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:59.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f315b7fe640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c109240 0x7f315c19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:59.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f315bfff640 1 -- 172.21.15.115:0/2166101794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c109b90 msgr2=0x7f315c19fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:59.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f315bfff640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c109b90 0x7f315c19fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:59.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f315bfff640 1 -- 172.21.15.115:0/2166101794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c109240 msgr2=0x7f315c19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:15:59.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f315bfff640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c109240 0x7f315c19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:15:59.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f315bfff640 1 -- 172.21.15.115:0/2166101794 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c1a0130 con 0x7f315c103900 2024-08-28T19:15:59.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.588+0000 7f315b7fe640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c109240 0x7f315c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:15:59.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.589+0000 7f315bfff640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f315c103900 0x7f315c19a280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f314c00c920 tx=0x7f314c00cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:15:59.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.589+0000 7f31597fa640 1 -- 172.21.15.115:0/2166101794 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c007c70 con 0x7f315c103900 2024-08-28T19:15:59.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.589+0000 7f31597fa640 1 -- 172.21.15.115:0/2166101794 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f314c007e10 con 0x7f315c103900 2024-08-28T19:15:59.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.589+0000 7f316279a640 1 -- 172.21.15.115:0/2166101794 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315c1a0420 con 0x7f315c103900 2024-08-28T19:15:59.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.589+0000 7f31597fa640 1 -- 172.21.15.115:0/2166101794 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c018820 con 0x7f315c103900 2024-08-28T19:15:59.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.589+0000 7f316279a640 1 -- 172.21.15.115:0/2166101794 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f315c1a6c40 con 0x7f315c103900 2024-08-28T19:15:59.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.590+0000 7f316279a640 1 -- 172.21.15.115:0/2166101794 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f315c103d10 con 0x7f315c103900 2024-08-28T19:15:59.594 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.591+0000 7f31597fa640 1 -- 172.21.15.115:0/2166101794 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f314c00f450 con 0x7f315c103900 2024-08-28T19:15:59.594 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.592+0000 7f31597fa640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31440b4bb0 0x7f31440b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:15:59.594 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.592+0000 7f31597fa640 1 -- 172.21.15.115:0/2166101794 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5332+0+0 (secure 0 0 0) 0x7f314c0e49d0 con 0x7f315c103900 2024-08-28T19:15:59.594 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.592+0000 7f315b7fe640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31440b4bb0 0x7f31440b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:15:59.594 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.593+0000 7f315b7fe640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31440b4bb0 0x7f31440b70c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f315c106200 tx=0x7f3150005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:15:59.596 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.595+0000 7f31597fa640 1 -- 172.21.15.115:0/2166101794 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f314c014030 con 0x7f315c103900 2024-08-28T19:15:59.783 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:15:59.781+0000 7f316279a640 1 -- 172.21.15.115:0/2166101794 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f315c10b9b0 con 0x7f31440b4bb0 2024-08-28T19:15:59.809 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:15:59.740+0000 7faa1eb14640 -1 osd.7 0 waiting for initial osdmap 2024-08-28T19:15:59.809 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:15:59 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:15:59.749+0000 7faa160f5640 -1 osd.7 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:16:00.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi028 2024-08-28T19:16:00.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi028 to 5697M 2024-08-28T19:16:00.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:16:00.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: osdmap e44: 8 total, 7 up, 8 in 2024-08-28T19:16:00.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:16:00.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:16:00.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: from='osd.7 ' entity='osd.7' 2024-08-28T19:16:00.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:16:00.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:16:00.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:00.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: osd.7 [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315] boot 2024-08-28T19:16:00.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: osdmap e45: 8 total, 8 up, 8 in 2024-08-28T19:16:00.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:00 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:16:00.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi028 2024-08-28T19:16:00.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi028 to 5697M 2024-08-28T19:16:00.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:16:00.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: osdmap e44: 8 total, 7 up, 8 in 2024-08-28T19:16:00.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:16:00.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:16:00.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: from='osd.7 ' entity='osd.7' 2024-08-28T19:16:00.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:16:00.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:16:00.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:00.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: osd.7 [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315] boot 2024-08-28T19:16:00.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: osdmap e45: 8 total, 8 up, 8 in 2024-08-28T19:16:00.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:00 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:16:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi028 2024-08-28T19:16:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi028 to 5697M 2024-08-28T19:16:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:16:00.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: osdmap e44: 8 total, 7 up, 8 in 2024-08-28T19:16:00.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:16:00.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:16:00.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: from='osd.7 ' entity='osd.7' 2024-08-28T19:16:00.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:16:00.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:16:00.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:00.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: osd.7 [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315] boot 2024-08-28T19:16:00.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: osdmap e45: 8 total, 8 up, 8 in 2024-08-28T19:16:00.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:00 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-28T19:16:01.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:01 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:16:01.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:01 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:16:01.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:01 smithi115 ceph-mon[22184]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:16:01.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:01 smithi115 ceph-mon[22184]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:16:01.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:01 smithi115 ceph-mon[22184]: osdmap e46: 8 total, 8 up, 8 in 2024-08-28T19:16:01.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:01 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:16:01.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:01 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:16:01.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:01 smithi016 ceph-mon[18556]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:16:01.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:01 smithi016 ceph-mon[18556]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:16:01.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:01 smithi016 ceph-mon[18556]: osdmap e46: 8 total, 8 up, 8 in 2024-08-28T19:16:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:01 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:16:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:01 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:16:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:01 smithi028 ceph-mon[21769]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 626 GiB / 626 GiB avail 2024-08-28T19:16:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:01 smithi028 ceph-mon[21769]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:16:01.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:01 smithi028 ceph-mon[21769]: osdmap e46: 8 total, 8 up, 8 in 2024-08-28T19:16:03.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:03 smithi115 ceph-mon[22184]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 440 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:03.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:03 smithi115 ceph-mon[22184]: osdmap e47: 8 total, 8 up, 8 in 2024-08-28T19:16:03.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:03 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/1582257058' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f814cd8e-30c2-4e74-aa5c-eb2d40e27586"} : dispatch 2024-08-28T19:16:03.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:03 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f814cd8e-30c2-4e74-aa5c-eb2d40e27586"} : dispatch 2024-08-28T19:16:03.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:03 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f814cd8e-30c2-4e74-aa5c-eb2d40e27586"}]': finished 2024-08-28T19:16:03.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:03 smithi115 ceph-mon[22184]: osdmap e48: 9 total, 8 up, 9 in 2024-08-28T19:16:03.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:03 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:03.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:03 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/1861007092' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:16:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:03 smithi028 ceph-mon[21769]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 440 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:03 smithi028 ceph-mon[21769]: osdmap e47: 8 total, 8 up, 8 in 2024-08-28T19:16:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:03 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/1582257058' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f814cd8e-30c2-4e74-aa5c-eb2d40e27586"} : dispatch 2024-08-28T19:16:03.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:03 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f814cd8e-30c2-4e74-aa5c-eb2d40e27586"} : dispatch 2024-08-28T19:16:03.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:03 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f814cd8e-30c2-4e74-aa5c-eb2d40e27586"}]': finished 2024-08-28T19:16:03.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:03 smithi028 ceph-mon[21769]: osdmap e48: 9 total, 8 up, 9 in 2024-08-28T19:16:03.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:03 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:03.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:03 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/1861007092' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:16:03.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:03 smithi016 ceph-mon[18556]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 440 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:03.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:03 smithi016 ceph-mon[18556]: osdmap e47: 8 total, 8 up, 8 in 2024-08-28T19:16:03.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:03 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/1582257058' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f814cd8e-30c2-4e74-aa5c-eb2d40e27586"} : dispatch 2024-08-28T19:16:03.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:03 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f814cd8e-30c2-4e74-aa5c-eb2d40e27586"} : dispatch 2024-08-28T19:16:03.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:03 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f814cd8e-30c2-4e74-aa5c-eb2d40e27586"}]': finished 2024-08-28T19:16:03.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:03 smithi016 ceph-mon[18556]: osdmap e48: 9 total, 8 up, 9 in 2024-08-28T19:16:03.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:03 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:03.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:03 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/1861007092' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:16:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:05 smithi028 ceph-mon[21769]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 440 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:05 smithi115 ceph-mon[22184]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 440 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:05 smithi016 ceph-mon[18556]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 440 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:07 smithi115 ceph-mon[22184]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-08-28T19:16:07.303 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:07 smithi016 ceph-mon[18556]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-08-28T19:16:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:07 smithi028 ceph-mon[21769]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-08-28T19:16:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:09 smithi115 ceph-mon[22184]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-08-28T19:16:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:09 smithi028 ceph-mon[21769]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-08-28T19:16:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:09 smithi016 ceph-mon[18556]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-08-28T19:16:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:11 smithi028 ceph-mon[21769]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-28T19:16:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-08-28T19:16:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:11.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:11 smithi115 ceph-mon[22184]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-28T19:16:11.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-08-28T19:16:11.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:11 smithi016 ceph-mon[18556]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-28T19:16:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-08-28T19:16:11.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:12.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:12 smithi028 ceph-mon[21769]: Deploying daemon osd.8 on smithi115 2024-08-28T19:16:12.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:12 smithi115 ceph-mon[22184]: Deploying daemon osd.8 on smithi115 2024-08-28T19:16:12.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:12 smithi016 ceph-mon[18556]: Deploying daemon osd.8 on smithi115 2024-08-28T19:16:13.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:13 smithi115 ceph-mon[22184]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-28T19:16:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:13 smithi028 ceph-mon[21769]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-28T19:16:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:13 smithi016 ceph-mon[18556]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-28T19:16:15.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:15 smithi115 ceph-mon[22184]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-08-28T19:16:15.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:15 smithi028 ceph-mon[21769]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-08-28T19:16:15.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:15 smithi016 ceph-mon[18556]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-08-28T19:16:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:17 smithi115 ceph-mon[22184]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:16:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:17 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:17 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:17 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:17 smithi028 ceph-mon[21769]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:16:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:17 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:17 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:17 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:17 smithi016 ceph-mon[18556]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:16:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:17 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:17 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:17.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:17 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:19 smithi028 ceph-mon[21769]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:19 smithi115 ceph-mon[22184]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:19 smithi016 ceph-mon[18556]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:19.893 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.892+0000 7f31597fa640 1 -- 172.21.15.115:0/2166101794 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f315c10b9b0 con 0x7f31440b4bb0 2024-08-28T19:16:19.894 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 8 on host 'smithi115' 2024-08-28T19:16:19.895 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 -- 172.21.15.115:0/2166101794 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31440b4bb0 msgr2=0x7f31440b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:16:19.895 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31440b4bb0 0x7f31440b70c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f315c106200 tx=0x7f3150005f90 comp rx=0 tx=0).stop 2024-08-28T19:16:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 -- 172.21.15.115:0/2166101794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f315c103900 msgr2=0x7f315c19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:16:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f315c103900 0x7f315c19a280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f314c00c920 tx=0x7f314c00cde0 comp rx=0 tx=0).stop 2024-08-28T19:16:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 -- 172.21.15.115:0/2166101794 shutdown_connections 2024-08-28T19:16:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31440b4bb0 0x7f31440b70c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c109b90 0x7f315c19fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c109240 0x7f315c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 --2- 172.21.15.115:0/2166101794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f315c103900 0x7f315c19a280 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 -- 172.21.15.115:0/2166101794 >> 172.21.15.115:0/2166101794 conn(0x7f315c07b5f0 msgr2=0x7f315c107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:16:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 -- 172.21.15.115:0/2166101794 shutdown_connections 2024-08-28T19:16:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:16:19.894+0000 7f316279a640 1 -- 172.21.15.115:0/2166101794 wait complete. 2024-08-28T19:16:20.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:20 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:20.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:20 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:20.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:20 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:20.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:20 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:20.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:20 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:20.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:20 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:20.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:20 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:20.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:20 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:20.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:20 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:20.524 DEBUG:teuthology.orchestra.run.smithi115:osd.8> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.8.service 2024-08-28T19:16:20.526 INFO:tasks.cephadm:Deploying osd.9 on smithi016 with /dev/vg_nvme/lv_1... 2024-08-28T19:16:20.526 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-28T19:16:20.792 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:16:21.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:21 smithi016 ceph-mon[18556]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:21.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:21 smithi028 ceph-mon[21769]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:21 smithi115 ceph-mon[22184]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:22.300 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:16:22 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:16:22.169+0000 7f5c35169640 -1 osd.8 0 log_to_monitors true 2024-08-28T19:16:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:23 smithi115 ceph-mon[22184]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:23 smithi115 ceph-mon[22184]: from='osd.8 [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-28T19:16:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:23 smithi115 ceph-mon[22184]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-28T19:16:23.301 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:23 smithi016 ceph-mon[18556]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:23.301 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:23 smithi016 ceph-mon[18556]: from='osd.8 [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-28T19:16:23.301 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:23 smithi016 ceph-mon[18556]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-28T19:16:23.304 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:16:23.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:23 smithi028 ceph-mon[21769]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:23.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:23 smithi028 ceph-mon[21769]: from='osd.8 [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-28T19:16:23.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:23 smithi028 ceph-mon[21769]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-28T19:16:23.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2024-08-28T19:16:23.593 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:16:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.086+0000 7f38fedb2640 1 -- 172.21.15.16:0/3779712676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8108140 msgr2=0x7f38f8108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:16:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.086+0000 7f38fedb2640 1 --2- 172.21.15.16:0/3779712676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8108140 0x7f38f8108550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009900 tx=0x7f38e002a410 comp rx=0 tx=0).stop 2024-08-28T19:16:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.086+0000 7f38fedb2640 1 -- 172.21.15.16:0/3779712676 shutdown_connections 2024-08-28T19:16:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.086+0000 7f38fedb2640 1 --2- 172.21.15.16:0/3779712676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38f8108a90 0x7f38f810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.086+0000 7f38fedb2640 1 --2- 172.21.15.16:0/3779712676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8108140 0x7f38f8108550 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.086+0000 7f38fedb2640 1 --2- 172.21.15.16:0/3779712676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f38f8102800 0x7f38f8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.086+0000 7f38fedb2640 1 -- 172.21.15.16:0/3779712676 >> 172.21.15.16:0/3779712676 conn(0x7f38f80794d0 msgr2=0x7f38f80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:16:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.086+0000 7f38fedb2640 1 -- 172.21.15.16:0/3779712676 shutdown_connections 2024-08-28T19:16:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.086+0000 7f38fedb2640 1 -- 172.21.15.16:0/3779712676 wait complete. 2024-08-28T19:16:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fedb2640 1 Processor -- start 2024-08-28T19:16:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fedb2640 1 -- start start 2024-08-28T19:16:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fedb2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38f8102800 0x7f38f819a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:16:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fedb2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f38f8108140 0x7f38f819a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:16:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fedb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8108a90 0x7f38f819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:16:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fedb2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38f81126e0 con 0x7f38f8108a90 2024-08-28T19:16:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fedb2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8112560 con 0x7f38f8102800 2024-08-28T19:16:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fedb2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8112860 con 0x7f38f8108140 2024-08-28T19:16:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fd34f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8108a90 0x7f38f819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:16:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fd34f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8108a90 0x7f38f819fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50520/0 (socket says 172.21.15.16:50520) 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fd34f640 1 -- 172.21.15.16:0/186434375 learned_addr learned my addr 172.21.15.16:0/186434375 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38f7fff640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f38f8108140 0x7f38f819a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.087+0000 7f38fcb4e640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38f8102800 0x7f38f819a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38f7fff640 1 -- 172.21.15.16:0/186434375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38f8102800 msgr2=0x7f38f819a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38f7fff640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38f8102800 0x7f38f819a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38f7fff640 1 -- 172.21.15.16:0/186434375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8108a90 msgr2=0x7f38f819fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38f7fff640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8108a90 0x7f38f819fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38f7fff640 1 -- 172.21.15.16:0/186434375 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f81a0230 con 0x7f38f8108140 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38fd34f640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8108a90 0x7f38f819fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38f7fff640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f38f8108140 0x7f38f819a860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f38e0006010 tx=0x7f38e0004060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38f5ffb640 1 -- 172.21.15.16:0/186434375 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e003f070 con 0x7f38f8108140 2024-08-28T19:16:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38fedb2640 1 -- 172.21.15.16:0/186434375 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f81a04c0 con 0x7f38f8108140 2024-08-28T19:16:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38f5ffb640 1 -- 172.21.15.16:0/186434375 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e0003ca0 con 0x7f38f8108140 2024-08-28T19:16:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38f5ffb640 1 -- 172.21.15.16:0/186434375 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e0035070 con 0x7f38f8108140 2024-08-28T19:16:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.088+0000 7f38fedb2640 1 -- 172.21.15.16:0/186434375 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f81a6c80 con 0x7f38f8108140 2024-08-28T19:16:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.090+0000 7f38fedb2640 1 -- 172.21.15.16:0/186434375 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38f8102c10 con 0x7f38f8108140 2024-08-28T19:16:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.091+0000 7f38f5ffb640 1 -- 172.21.15.16:0/186434375 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f38e003b460 con 0x7f38f8108140 2024-08-28T19:16:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.092+0000 7f38f5ffb640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f38e40b4b80 0x7f38e40b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:16:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.092+0000 7f38fcb4e640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f38e40b4b80 0x7f38e40b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:16:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.093+0000 7f38f5ffb640 1 -- 172.21.15.16:0/186434375 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5764+0+0 (secure 0 0 0) 0x7f38e0104ee0 con 0x7f38f8108140 2024-08-28T19:16:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.093+0000 7f38fcb4e640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f38e40b4b80 0x7f38e40b7090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009bd0 tx=0x7f38e8005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:16:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.099+0000 7f38f5ffb640 1 -- 172.21.15.16:0/186434375 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f38e00cc310 con 0x7f38f8108140 2024-08-28T19:16:24.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-28T19:16:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: osdmap e49: 9 total, 8 up, 9 in 2024-08-28T19:16:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='osd.8 [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:16:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:16:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:24.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:24.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:24.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:24.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:24.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:24.295+0000 7f38fedb2640 1 -- 172.21.15.16:0/186434375 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f38f80635a0 con 0x7f38e40b4b80 2024-08-28T19:16:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-28T19:16:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: osdmap e49: 9 total, 8 up, 9 in 2024-08-28T19:16:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='osd.8 [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:16:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:16:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:24.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:24.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:24.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:24.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-28T19:16:24.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: osdmap e49: 9 total, 8 up, 9 in 2024-08-28T19:16:24.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:24.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='osd.8 [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:16:24.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:16:24.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:24.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:24.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:24.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:24.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:24.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:24.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:24.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:24.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:25.068 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:16:24.731+0000 7f5c31900640 -1 osd.8 0 waiting for initial osdmap 2024-08-28T19:16:25.068 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:16:24 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:16:24.744+0000 7f5c28ee1640 -1 osd.8 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:16:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi115 2024-08-28T19:16:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi115 to 5696M 2024-08-28T19:16:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:16:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: osdmap e50: 9 total, 8 up, 9 in 2024-08-28T19:16:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:16:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:16:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:25.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: from='osd.8 ' entity='osd.8' 2024-08-28T19:16:25.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: osd.8 [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149] boot 2024-08-28T19:16:25.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: osdmap e51: 9 total, 9 up, 9 in 2024-08-28T19:16:25.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi115 2024-08-28T19:16:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi115 to 5696M 2024-08-28T19:16:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:16:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: osdmap e50: 9 total, 8 up, 9 in 2024-08-28T19:16:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:16:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:16:25.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:25.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: from='osd.8 ' entity='osd.8' 2024-08-28T19:16:25.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: osd.8 [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149] boot 2024-08-28T19:16:25.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: osdmap e51: 9 total, 9 up, 9 in 2024-08-28T19:16:25.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 715 GiB / 715 GiB avail 2024-08-28T19:16:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi115 2024-08-28T19:16:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi115 to 5696M 2024-08-28T19:16:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:16:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: osdmap e50: 9 total, 8 up, 9 in 2024-08-28T19:16:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:16:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:16:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: from='osd.8 ' entity='osd.8' 2024-08-28T19:16:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: osd.8 [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149] boot 2024-08-28T19:16:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: osdmap e51: 9 total, 9 up, 9 in 2024-08-28T19:16:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-28T19:16:26.243 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:26 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:16:26.243 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:26 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:16:26.243 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:26 smithi016 ceph-mon[18556]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:16:26.244 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:26 smithi016 ceph-mon[18556]: osdmap e52: 9 total, 9 up, 9 in 2024-08-28T19:16:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:26 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:16:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:26 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:16:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:26 smithi115 ceph-mon[22184]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:16:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:26 smithi115 ceph-mon[22184]: osdmap e52: 9 total, 9 up, 9 in 2024-08-28T19:16:26.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:26 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:16:26.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:26 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:16:26.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:26 smithi028 ceph-mon[21769]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:16:26.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:26 smithi028 ceph-mon[21769]: osdmap e52: 9 total, 9 up, 9 in 2024-08-28T19:16:27.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:27 smithi115 ceph-mon[22184]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:27.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:27 smithi115 ceph-mon[22184]: osdmap e53: 9 total, 9 up, 9 in 2024-08-28T19:16:27.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:27 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1410183016' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9617dc8d-0eb5-4fe1-bc88-5d37d40c065f"} : dispatch 2024-08-28T19:16:27.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:27 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1410183016' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9617dc8d-0eb5-4fe1-bc88-5d37d40c065f"}]': finished 2024-08-28T19:16:27.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:27 smithi115 ceph-mon[22184]: osdmap e54: 10 total, 9 up, 10 in 2024-08-28T19:16:27.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:27 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:27 smithi016 ceph-mon[18556]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:27 smithi016 ceph-mon[18556]: osdmap e53: 9 total, 9 up, 9 in 2024-08-28T19:16:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:27 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1410183016' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9617dc8d-0eb5-4fe1-bc88-5d37d40c065f"} : dispatch 2024-08-28T19:16:27.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:27 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1410183016' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9617dc8d-0eb5-4fe1-bc88-5d37d40c065f"}]': finished 2024-08-28T19:16:27.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:27 smithi016 ceph-mon[18556]: osdmap e54: 10 total, 9 up, 10 in 2024-08-28T19:16:27.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:27 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:27 smithi028 ceph-mon[21769]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:27 smithi028 ceph-mon[21769]: osdmap e53: 9 total, 9 up, 9 in 2024-08-28T19:16:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:27 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1410183016' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9617dc8d-0eb5-4fe1-bc88-5d37d40c065f"} : dispatch 2024-08-28T19:16:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:27 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1410183016' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9617dc8d-0eb5-4fe1-bc88-5d37d40c065f"}]': finished 2024-08-28T19:16:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:27 smithi028 ceph-mon[21769]: osdmap e54: 10 total, 9 up, 10 in 2024-08-28T19:16:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:27 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:28.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:28 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3576068724' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:16:28.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:28 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3576068724' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:16:28.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:28 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3576068724' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:16:29.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:29 smithi028 ceph-mon[21769]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:29.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:29 smithi115 ceph-mon[22184]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:29.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:29 smithi016 ceph-mon[18556]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:31 smithi115 ceph-mon[22184]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:31.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:31 smithi016 ceph-mon[18556]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:31 smithi028 ceph-mon[21769]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:33 smithi115 ceph-mon[22184]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-28T19:16:33.467 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:33 smithi016 ceph-mon[18556]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-28T19:16:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:33 smithi028 ceph-mon[21769]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-28T19:16:35.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:35 smithi115 ceph-mon[22184]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-08-28T19:16:35.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:35 smithi016 ceph-mon[18556]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-08-28T19:16:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:35 smithi028 ceph-mon[21769]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-08-28T19:16:36.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-08-28T19:16:36.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:36.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-08-28T19:16:36.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:36.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-08-28T19:16:36.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:37.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:37 smithi016 ceph-mon[18556]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-08-28T19:16:37.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:37 smithi016 ceph-mon[18556]: Deploying daemon osd.9 on smithi016 2024-08-28T19:16:37.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:37 smithi115 ceph-mon[22184]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-08-28T19:16:37.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:37 smithi115 ceph-mon[22184]: Deploying daemon osd.9 on smithi016 2024-08-28T19:16:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:37 smithi028 ceph-mon[21769]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-08-28T19:16:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:37 smithi028 ceph-mon[21769]: Deploying daemon osd.9 on smithi016 2024-08-28T19:16:39.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:39 smithi016 ceph-mon[18556]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-28T19:16:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:39 smithi115 ceph-mon[22184]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-28T19:16:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:39 smithi028 ceph-mon[21769]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-28T19:16:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:41 smithi115 ceph-mon[22184]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:16:41.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:41 smithi016 ceph-mon[18556]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:16:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:41 smithi028 ceph-mon[21769]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:16:42.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:42.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:42.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:42.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:42.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:42.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:43.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:43 smithi016 ceph-mon[18556]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:16:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:43 smithi115 ceph-mon[22184]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:16:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:43 smithi028 ceph-mon[21769]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:16:44.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.969+0000 7f38f5ffb640 1 -- 172.21.15.16:0/186434375 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f38f80635a0 con 0x7f38e40b4b80 2024-08-28T19:16:44.972 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 9 on host 'smithi016' 2024-08-28T19:16:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 -- 172.21.15.16:0/186434375 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f38e40b4b80 msgr2=0x7f38e40b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:16:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f38e40b4b80 0x7f38e40b7090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009bd0 tx=0x7f38e8005f90 comp rx=0 tx=0).stop 2024-08-28T19:16:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 -- 172.21.15.16:0/186434375 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f38f8108140 msgr2=0x7f38f819a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:16:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f38f8108140 0x7f38f819a860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f38e0006010 tx=0x7f38e0004060 comp rx=0 tx=0).stop 2024-08-28T19:16:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 -- 172.21.15.16:0/186434375 shutdown_connections 2024-08-28T19:16:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8108a90 0x7f38f819fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f38f8108140 0x7f38f819a860 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f38e40b4b80 0x7f38e40b7090 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 --2- 172.21.15.16:0/186434375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38f8102800 0x7f38f819a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 -- 172.21.15.16:0/186434375 >> 172.21.15.16:0/186434375 conn(0x7f38f80794d0 msgr2=0x7f38f806afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:16:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 -- 172.21.15.16:0/186434375 shutdown_connections 2024-08-28T19:16:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:16:44.973+0000 7f38fedb2640 1 -- 172.21.15.16:0/186434375 wait complete. 2024-08-28T19:16:45.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:45 smithi016 ceph-mon[18556]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:45.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:45 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:45 smithi028 ceph-mon[21769]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:45 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:45 smithi115 ceph-mon[22184]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:16:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:45 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:45.648 DEBUG:teuthology.orchestra.run.smithi016:osd.9> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.9.service 2024-08-28T19:16:45.651 INFO:tasks.cephadm:Deploying osd.10 on smithi028 with /dev/vg_nvme/lv_1... 2024-08-28T19:16:45.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-28T19:16:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:16:47.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:47 smithi016 ceph-mon[18556]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:47.280 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:16:47 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:16:47.267+0000 7f1ab310a640 -1 osd.9 0 log_to_monitors true 2024-08-28T19:16:47.313 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:47 smithi028 ceph-mon[21769]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:47 smithi115 ceph-mon[22184]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:48.297 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:48 smithi028 ceph-mon[21769]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-28T19:16:48.469 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:48 smithi016 ceph-mon[18556]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-28T19:16:48.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:48 smithi115 ceph-mon[22184]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-28T19:16:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-08-28T19:16:49.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-08-28T19:16:49.343 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:16:49.463 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:49.463 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-28T19:16:49.463 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: osdmap e55: 10 total, 9 up, 10 in 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:49.464 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:49.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-28T19:16:49.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: osdmap e55: 10 total, 9 up, 10 in 2024-08-28T19:16:49.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:16:49.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:49.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:49.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:49.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:49.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-28T19:16:49.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: osdmap e55: 10 total, 9 up, 10 in 2024-08-28T19:16:49.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-08-28T19:16:49.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:49.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-28T19:16:49.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:49.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:16:49.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:16:49.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.869+0000 7f513c766640 1 -- 172.21.15.28:0/2147580644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5134109b90 msgr2=0x7f51341103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:16:49.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.869+0000 7f513c766640 1 --2- 172.21.15.28:0/2147580644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5134109b90 0x7f51341103c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f512c009ca0 tx=0x7f512c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:16:49.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.870+0000 7f513c766640 1 -- 172.21.15.28:0/2147580644 shutdown_connections 2024-08-28T19:16:49.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.870+0000 7f513c766640 1 --2- 172.21.15.28:0/2147580644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5134109b90 0x7f51341103c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:49.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.870+0000 7f513c766640 1 --2- 172.21.15.28:0/2147580644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5134109240 0x7f5134109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:49.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.870+0000 7f513c766640 1 --2- 172.21.15.28:0/2147580644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134103900 0x7f5134103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:49.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.870+0000 7f513c766640 1 -- 172.21.15.28:0/2147580644 >> 172.21.15.28:0/2147580644 conn(0x7f513407b5f0 msgr2=0x7f513407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:16:49.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.870+0000 7f513c766640 1 -- 172.21.15.28:0/2147580644 shutdown_connections 2024-08-28T19:16:49.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.870+0000 7f513c766640 1 -- 172.21.15.28:0/2147580644 wait complete. 2024-08-28T19:16:49.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.870+0000 7f513c766640 1 Processor -- start 2024-08-28T19:16:49.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.870+0000 7f513c766640 1 -- start start 2024-08-28T19:16:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f513c766640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134103900 0x7f513419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:16:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f513c766640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5134109240 0x7f513419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:16:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f513c766640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5134109b90 0x7f513419fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:16:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f513c766640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5134113850 con 0x7f5134103900 2024-08-28T19:16:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f513c766640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51341136d0 con 0x7f5134109240 2024-08-28T19:16:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f513c766640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f51341139d0 con 0x7f5134109b90 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f5139d01640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5134109240 0x7f513419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f5139d01640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5134109240 0x7f513419a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41358/0 (socket says 172.21.15.28:41358) 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f5139d01640 1 -- 172.21.15.28:0/1966676311 learned_addr learned my addr 172.21.15.28:0/1966676311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f513ad03640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5134109b90 0x7f513419fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f513a502640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134103900 0x7f513419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f5139d01640 1 -- 172.21.15.28:0/1966676311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5134109b90 msgr2=0x7f513419fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f5139d01640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5134109b90 0x7f513419fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f5139d01640 1 -- 172.21.15.28:0/1966676311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134103900 msgr2=0x7f513419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f5139d01640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134103900 0x7f513419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f5139d01640 1 -- 172.21.15.28:0/1966676311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51341a01f0 con 0x7f5134109240 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.871+0000 7f5139d01640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5134109240 0x7f513419a820 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f512400c970 tx=0x7f512400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:16:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.872+0000 7f512b7fe640 1 -- 172.21.15.28:0/1966676311 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124004220 con 0x7f5134109240 2024-08-28T19:16:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.872+0000 7f513c766640 1 -- 172.21.15.28:0/1966676311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51341a04e0 con 0x7f5134109240 2024-08-28T19:16:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.872+0000 7f512b7fe640 1 -- 172.21.15.28:0/1966676311 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51240043c0 con 0x7f5134109240 2024-08-28T19:16:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.872+0000 7f512b7fe640 1 -- 172.21.15.28:0/1966676311 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124004a00 con 0x7f5134109240 2024-08-28T19:16:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.872+0000 7f513c766640 1 -- 172.21.15.28:0/1966676311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51341a6c40 con 0x7f5134109240 2024-08-28T19:16:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.873+0000 7f513c766640 1 -- 172.21.15.28:0/1966676311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5134103d10 con 0x7f5134109240 2024-08-28T19:16:49.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.874+0000 7f512b7fe640 1 -- 172.21.15.28:0/1966676311 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f51240079b0 con 0x7f5134109240 2024-08-28T19:16:49.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.875+0000 7f512b7fe640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51200b4c00 0x7f51200b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:16:49.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.875+0000 7f512b7fe640 1 -- 172.21.15.28:0/1966676311 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6196+0+0 (secure 0 0 0) 0x7f51240e49c0 con 0x7f5134109240 2024-08-28T19:16:49.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.875+0000 7f513a502640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51200b4c00 0x7f51200b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:16:49.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.876+0000 7f513a502640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51200b4c00 0x7f51200b7110 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5130009bd0 tx=0x7f5130005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:16:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:49.878+0000 7f512b7fe640 1 -- 172.21.15.28:0/1966676311 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f5124014030 con 0x7f5134109240 2024-08-28T19:16:50.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:16:49.751+0000 7f1aaf08c640 -1 osd.9 0 waiting for initial osdmap 2024-08-28T19:16:50.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:16:49 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:16:49.766+0000 7f1aa6e82640 -1 osd.9 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:16:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:16:50.074+0000 7f513c766640 1 -- 172.21.15.28:0/1966676311 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f513410b9b0 con 0x7f51200b4c00 2024-08-28T19:16:50.487 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:50 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi016 2024-08-28T19:16:50.487 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:50 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi016 to 4273M 2024-08-28T19:16:50.487 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:50 smithi028 ceph-mon[21769]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:16:50.487 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:50 smithi028 ceph-mon[21769]: osdmap e56: 10 total, 9 up, 10 in 2024-08-28T19:16:50.487 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:50.487 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:50.487 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:50 smithi028 ceph-mon[21769]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' 2024-08-28T19:16:50.487 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:16:50.488 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:16:50.488 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:50.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:50 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi016 2024-08-28T19:16:50.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:50 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi016 to 4273M 2024-08-28T19:16:50.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:50 smithi016 ceph-mon[18556]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:16:50.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:50 smithi016 ceph-mon[18556]: osdmap e56: 10 total, 9 up, 10 in 2024-08-28T19:16:50.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:50.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:50.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:50 smithi016 ceph-mon[18556]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' 2024-08-28T19:16:50.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:16:50.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:16:50.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:50 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi016 2024-08-28T19:16:50.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:50 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi016 to 4273M 2024-08-28T19:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:50 smithi115 ceph-mon[22184]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-08-28T19:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:50 smithi115 ceph-mon[22184]: osdmap e56: 10 total, 9 up, 10 in 2024-08-28T19:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:50 smithi115 ceph-mon[22184]: from='osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769]' entity='osd.9' 2024-08-28T19:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:16:50.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:16:51.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:51 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:16:51.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:51 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:16:51.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:51 smithi016 ceph-mon[18556]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:51.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:51 smithi016 ceph-mon[18556]: from='client.24500 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:16:51.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:51 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:51.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:51 smithi016 ceph-mon[18556]: osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769] boot 2024-08-28T19:16:51.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:51 smithi016 ceph-mon[18556]: osdmap e57: 10 total, 10 up, 10 in 2024-08-28T19:16:51.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:51 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:51.505 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:51 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:16:51.505 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:51 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:16:51.505 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:51 smithi028 ceph-mon[21769]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:51.505 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:51 smithi028 ceph-mon[21769]: from='client.24500 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:16:51.505 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:51 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:51.505 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:51 smithi028 ceph-mon[21769]: osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769] boot 2024-08-28T19:16:51.505 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:51 smithi028 ceph-mon[21769]: osdmap e57: 10 total, 10 up, 10 in 2024-08-28T19:16:51.505 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:51 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:51 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:16:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:51 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:16:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:51 smithi115 ceph-mon[22184]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 804 GiB / 805 GiB avail 2024-08-28T19:16:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:51 smithi115 ceph-mon[22184]: from='client.24500 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:16:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:51 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:51 smithi115 ceph-mon[22184]: osd.9 [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769] boot 2024-08-28T19:16:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:51 smithi115 ceph-mon[22184]: osdmap e57: 10 total, 10 up, 10 in 2024-08-28T19:16:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:51 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-28T19:16:53.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:53 smithi016 ceph-mon[18556]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:16:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:53 smithi016 ceph-mon[18556]: osdmap e58: 10 total, 10 up, 10 in 2024-08-28T19:16:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:53 smithi016 ceph-mon[18556]: from='client.? 172.21.15.28:0/3817168250' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dbed2c84-11b5-409d-a8e5-33c9fddc5698"} : dispatch 2024-08-28T19:16:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:53 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dbed2c84-11b5-409d-a8e5-33c9fddc5698"} : dispatch 2024-08-28T19:16:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:53 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbed2c84-11b5-409d-a8e5-33c9fddc5698"}]': finished 2024-08-28T19:16:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:53 smithi016 ceph-mon[18556]: osdmap e59: 11 total, 10 up, 11 in 2024-08-28T19:16:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:16:53.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:53 smithi028 ceph-mon[21769]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:16:53.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:53 smithi028 ceph-mon[21769]: osdmap e58: 10 total, 10 up, 10 in 2024-08-28T19:16:53.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:53 smithi028 ceph-mon[21769]: from='client.? 172.21.15.28:0/3817168250' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dbed2c84-11b5-409d-a8e5-33c9fddc5698"} : dispatch 2024-08-28T19:16:53.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:53 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dbed2c84-11b5-409d-a8e5-33c9fddc5698"} : dispatch 2024-08-28T19:16:53.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:53 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbed2c84-11b5-409d-a8e5-33c9fddc5698"}]': finished 2024-08-28T19:16:53.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:53 smithi028 ceph-mon[21769]: osdmap e59: 11 total, 10 up, 11 in 2024-08-28T19:16:53.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:16:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:53 smithi115 ceph-mon[22184]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:16:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:53 smithi115 ceph-mon[22184]: osdmap e58: 10 total, 10 up, 10 in 2024-08-28T19:16:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:53 smithi115 ceph-mon[22184]: from='client.? 172.21.15.28:0/3817168250' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dbed2c84-11b5-409d-a8e5-33c9fddc5698"} : dispatch 2024-08-28T19:16:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:53 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dbed2c84-11b5-409d-a8e5-33c9fddc5698"} : dispatch 2024-08-28T19:16:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:53 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbed2c84-11b5-409d-a8e5-33c9fddc5698"}]': finished 2024-08-28T19:16:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:53 smithi115 ceph-mon[22184]: osdmap e59: 11 total, 10 up, 11 in 2024-08-28T19:16:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:16:54.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:54 smithi016 ceph-mon[18556]: from='client.? 172.21.15.28:0/1104512535' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:16:54.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:54 smithi028 ceph-mon[21769]: from='client.? 172.21.15.28:0/1104512535' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:16:54.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:54 smithi115 ceph-mon[22184]: from='client.? 172.21.15.28:0/1104512535' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:16:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:55 smithi016 ceph-mon[18556]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:16:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:55 smithi028 ceph-mon[21769]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:16:55.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:55 smithi115 ceph-mon[22184]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:16:57.493 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:57 smithi028 ceph-mon[21769]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:16:57.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:57 smithi016 ceph-mon[18556]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:16:57.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:57 smithi115 ceph-mon[22184]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:16:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:16:59 smithi016 ceph-mon[18556]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:16:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:16:59 smithi028 ceph-mon[21769]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:16:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:16:59 smithi115 ceph-mon[22184]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:01.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:01 smithi016 ceph-mon[18556]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:01 smithi028 ceph-mon[21769]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:01 smithi115 ceph-mon[22184]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:02.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:02 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-08-28T19:17:02.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:02 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:02.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:02 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-08-28T19:17:02.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:02 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:02.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:02 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-08-28T19:17:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:02 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:03 smithi016 ceph-mon[18556]: Deploying daemon osd.10 on smithi028 2024-08-28T19:17:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:03 smithi016 ceph-mon[18556]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:03.511 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:03 smithi028 ceph-mon[21769]: Deploying daemon osd.10 on smithi028 2024-08-28T19:17:03.511 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:03 smithi028 ceph-mon[21769]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:03.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:03 smithi115 ceph-mon[22184]: Deploying daemon osd.10 on smithi028 2024-08-28T19:17:03.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:03 smithi115 ceph-mon[22184]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:05.443 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:05 smithi028 ceph-mon[21769]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:05 smithi016 ceph-mon[18556]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:05 smithi115 ceph-mon[22184]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:07 smithi016 ceph-mon[18556]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:07.520 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:07 smithi028 ceph-mon[21769]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:07.520 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:07.520 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:07.520 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:07.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:07 smithi115 ceph-mon[22184]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:07.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:07.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:07.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:09 smithi016 ceph-mon[18556]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:09 smithi028 ceph-mon[21769]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:09 smithi115 ceph-mon[22184]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:11.070 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 10 on host 'smithi028' 2024-08-28T19:17:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.068+0000 7f512b7fe640 1 -- 172.21.15.28:0/1966676311 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f513410b9b0 con 0x7f51200b4c00 2024-08-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 -- 172.21.15.28:0/1966676311 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51200b4c00 msgr2=0x7f51200b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51200b4c00 0x7f51200b7110 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5130009bd0 tx=0x7f5130005f90 comp rx=0 tx=0).stop 2024-08-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 -- 172.21.15.28:0/1966676311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5134109240 msgr2=0x7f513419a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5134109240 0x7f513419a820 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f512400c970 tx=0x7f512400ce30 comp rx=0 tx=0).stop 2024-08-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 -- 172.21.15.28:0/1966676311 shutdown_connections 2024-08-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5134109b90 0x7f513419fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51200b4c00 0x7f51200b7110 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5134109240 0x7f513419a820 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 --2- 172.21.15.28:0/1966676311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134103900 0x7f513419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 -- 172.21.15.28:0/1966676311 >> 172.21.15.28:0/1966676311 conn(0x7f513407b5f0 msgr2=0x7f5134107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 -- 172.21.15.28:0/1966676311 shutdown_connections 2024-08-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-08-28T19:17:11.070+0000 7f513c766640 1 -- 172.21.15.28:0/1966676311 wait complete. 2024-08-28T19:17:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:11 smithi028 ceph-mon[21769]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:11.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:11.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:11.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:11 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:11 smithi115 ceph-mon[22184]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:11 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.744 DEBUG:teuthology.orchestra.run.smithi028:osd.10> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.10.service 2024-08-28T19:17:11.746 INFO:tasks.cephadm:Deploying osd.11 on smithi115 with /dev/vg_nvme/lv_1... 2024-08-28T19:17:11.746 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-28T19:17:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:11 smithi016 ceph-mon[18556]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:11.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:11.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:11.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:11 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:11.999 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:17:12.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:12 smithi115 ceph-mon[22184]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:12.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:12 smithi028 ceph-mon[21769]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:12.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:12 smithi016 ceph-mon[18556]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:13.305 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:17:13 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:17:13.148+0000 7fdb9ddca640 -1 osd.10 0 log_to_monitors true 2024-08-28T19:17:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:13 smithi028 ceph-mon[21769]: from='osd.10 [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-28T19:17:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:13 smithi028 ceph-mon[21769]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-28T19:17:13.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:13 smithi115 ceph-mon[22184]: from='osd.10 [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-28T19:17:13.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:13 smithi115 ceph-mon[22184]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-28T19:17:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:13 smithi016 ceph-mon[18556]: from='osd.10 [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-28T19:17:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:13 smithi016 ceph-mon[18556]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-28T19:17:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:14 smithi115 ceph-mon[22184]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-28T19:17:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:14 smithi115 ceph-mon[22184]: osdmap e60: 11 total, 10 up, 11 in 2024-08-28T19:17:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:14 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:14 smithi115 ceph-mon[22184]: from='osd.10 [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:17:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:14 smithi115 ceph-mon[22184]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:17:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:14 smithi115 ceph-mon[22184]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:14.749 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:17:14 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:17:14.319+0000 7fdb99d4c640 -1 osd.10 0 waiting for initial osdmap 2024-08-28T19:17:14.749 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:17:14 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:17:14.331+0000 7fdb95b4a640 -1 osd.10 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:17:14.749 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:14 smithi028 ceph-mon[21769]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-28T19:17:14.750 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:14 smithi028 ceph-mon[21769]: osdmap e60: 11 total, 10 up, 11 in 2024-08-28T19:17:14.750 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:14 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:14.750 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:14 smithi028 ceph-mon[21769]: from='osd.10 [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:17:14.750 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:14 smithi028 ceph-mon[21769]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:17:14.750 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:14 smithi028 ceph-mon[21769]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:14.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:14 smithi016 ceph-mon[18556]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-28T19:17:14.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:14 smithi016 ceph-mon[18556]: osdmap e60: 11 total, 10 up, 11 in 2024-08-28T19:17:14.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:14 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:14.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:14 smithi016 ceph-mon[18556]: from='osd.10 [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:17:14.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:14 smithi016 ceph-mon[18556]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-08-28T19:17:14.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:14 smithi016 ceph-mon[18556]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 894 GiB / 894 GiB avail 2024-08-28T19:17:14.802 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-08-28T19:17:14.833 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi115:vg_nvme/lv_1 2024-08-28T19:17:15.083 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:17:15.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:17:15.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: osdmap e61: 11 total, 10 up, 11 in 2024-08-28T19:17:15.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:15.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:15.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi028 2024-08-28T19:17:15.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi028 to 4273M 2024-08-28T19:17:15.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:15.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:15.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: osd.10 [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207] boot 2024-08-28T19:17:15.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: osdmap e62: 11 total, 11 up, 11 in 2024-08-28T19:17:15.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:15.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.563+0000 7f334735c640 1 -- 172.21.15.115:0/3216139583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3340109c50 msgr2=0x7f3340110480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:15.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.563+0000 7f334735c640 1 --2- 172.21.15.115:0/3216139583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3340109c50 0x7f3340110480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3338009ca0 tx=0x7f333802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:17:15.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.563+0000 7f334735c640 1 -- 172.21.15.115:0/3216139583 shutdown_connections 2024-08-28T19:17:15.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.563+0000 7f334735c640 1 --2- 172.21.15.115:0/3216139583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3340109c50 0x7f3340110480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:15.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.563+0000 7f334735c640 1 --2- 172.21.15.115:0/3216139583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3340109300 0x7f3340109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:15.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.563+0000 7f334735c640 1 --2- 172.21.15.115:0/3216139583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3340101e20 0x7f3340102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:15.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.563+0000 7f334735c640 1 -- 172.21.15.115:0/3216139583 >> 172.21.15.115:0/3216139583 conn(0x7f3340079c60 msgr2=0x7f334007a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:15.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.564+0000 7f334735c640 1 -- 172.21.15.115:0/3216139583 shutdown_connections 2024-08-28T19:17:15.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.564+0000 7f334735c640 1 -- 172.21.15.115:0/3216139583 wait complete. 2024-08-28T19:17:15.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.564+0000 7f334735c640 1 Processor -- start 2024-08-28T19:17:15.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.564+0000 7f334735c640 1 -- start start 2024-08-28T19:17:15.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f334735c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3340101e20 0x7f334019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:15.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f334735c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3340109300 0x7f334019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:15.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f334735c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3340109c50 0x7f334019fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:15.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f334735c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3340113910 con 0x7f3340101e20 2024-08-28T19:17:15.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f334735c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3340113790 con 0x7f3340109c50 2024-08-28T19:17:15.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f334735c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3340113a90 con 0x7f3340109300 2024-08-28T19:17:15.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33448f7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3340109300 0x7f334019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:15.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33448f7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3340109300 0x7f334019a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40566/0 (socket says 172.21.15.115:40566) 2024-08-28T19:17:15.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33448f7640 1 -- 172.21.15.115:0/80456026 learned_addr learned my addr 172.21.15.115:0/80456026 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-08-28T19:17:15.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33450f8640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3340101e20 0x7f334019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:15.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33458f9640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3340109c50 0x7f334019fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:15.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33448f7640 1 -- 172.21.15.115:0/80456026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3340109c50 msgr2=0x7f334019fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:15.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33448f7640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3340109c50 0x7f334019fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:15.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33448f7640 1 -- 172.21.15.115:0/80456026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3340101e20 msgr2=0x7f334019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:15.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33448f7640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3340101e20 0x7f334019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:15.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33448f7640 1 -- 172.21.15.115:0/80456026 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33401a0070 con 0x7f3340109300 2024-08-28T19:17:15.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33450f8640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3340101e20 0x7f334019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:17:15.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33458f9640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3340109c50 0x7f334019fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:17:15.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33448f7640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3340109300 0x7f334019a760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f333000c8c0 tx=0x7f333000cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:15.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.565+0000 7f33367fc640 1 -- 172.21.15.115:0/80456026 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330007bb0 con 0x7f3340109300 2024-08-28T19:17:15.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.566+0000 7f334735c640 1 -- 172.21.15.115:0/80456026 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33401a0360 con 0x7f3340109300 2024-08-28T19:17:15.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.566+0000 7f33367fc640 1 -- 172.21.15.115:0/80456026 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3330007d50 con 0x7f3340109300 2024-08-28T19:17:15.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.566+0000 7f33367fc640 1 -- 172.21.15.115:0/80456026 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33300187c0 con 0x7f3340109300 2024-08-28T19:17:15.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.566+0000 7f334735c640 1 -- 172.21.15.115:0/80456026 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33401a6b80 con 0x7f3340109300 2024-08-28T19:17:15.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.566+0000 7f334735c640 1 -- 172.21.15.115:0/80456026 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3340102230 con 0x7f3340109300 2024-08-28T19:17:15.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.568+0000 7f33367fc640 1 -- 172.21.15.115:0/80456026 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f333000fa20 con 0x7f3340109300 2024-08-28T19:17:15.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.569+0000 7f33367fc640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f332c0b4bb0 0x7f332c0b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:15.583 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.569+0000 7f33367fc640 1 -- 172.21.15.115:0/80456026 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6942+0+0 (secure 0 0 0) 0x7f33300e5040 con 0x7f3340109300 2024-08-28T19:17:15.583 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.569+0000 7f33450f8640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f332c0b4bb0 0x7f332c0b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:15.583 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.570+0000 7f33450f8640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f332c0b4bb0 0x7f332c0b70c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f333c00baa0 tx=0x7f333c005d90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:15.583 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.572+0000 7f33367fc640 1 -- 172.21.15.115:0/80456026 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f3330014030 con 0x7f3340109300 2024-08-28T19:17:15.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:17:15.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: osdmap e61: 11 total, 10 up, 11 in 2024-08-28T19:17:15.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:15.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:15.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi028 2024-08-28T19:17:15.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi028 to 4273M 2024-08-28T19:17:15.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:15.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:15.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: osd.10 [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207] boot 2024-08-28T19:17:15.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: osdmap e62: 11 total, 11 up, 11 in 2024-08-28T19:17:15.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:15.758 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:15.756+0000 7f334735c640 1 -- 172.21.15.115:0/80456026 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3340108970 con 0x7f332c0b4bb0 2024-08-28T19:17:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-08-28T19:17:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: osdmap e61: 11 total, 10 up, 11 in 2024-08-28T19:17:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi028 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi028 to 4273M 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: osd.10 [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207] boot 2024-08-28T19:17:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: osdmap e62: 11 total, 11 up, 11 in 2024-08-28T19:17:15.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-28T19:17:16.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:16 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:17:16.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:16 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:17:16.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:16 smithi115 ceph-mon[22184]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:16.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:16 smithi115 ceph-mon[22184]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:17:16.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:16 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:17:16.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:16 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:17:16.428 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:16 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:16.428 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:16 smithi115 ceph-mon[22184]: osdmap e63: 11 total, 11 up, 11 in 2024-08-28T19:17:16.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:16 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:17:16.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:16 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:17:16.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:16 smithi016 ceph-mon[18556]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:16.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:16 smithi016 ceph-mon[18556]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:17:16.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:16 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:17:16.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:16 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:17:16.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:16 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:16.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:16 smithi016 ceph-mon[18556]: osdmap e63: 11 total, 11 up, 11 in 2024-08-28T19:17:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:16 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:17:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:16 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:17:16.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:16 smithi028 ceph-mon[21769]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:16.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:16 smithi028 ceph-mon[21769]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:17:16.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:16 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-28T19:17:16.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:16 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-28T19:17:16.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:16 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:16.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:16 smithi028 ceph-mon[21769]: osdmap e63: 11 total, 11 up, 11 in 2024-08-28T19:17:18.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:18 smithi115 ceph-mon[22184]: osdmap e64: 11 total, 11 up, 11 in 2024-08-28T19:17:18.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:18 smithi028 ceph-mon[21769]: osdmap e64: 11 total, 11 up, 11 in 2024-08-28T19:17:18.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:18 smithi016 ceph-mon[18556]: osdmap e64: 11 total, 11 up, 11 in 2024-08-28T19:17:19.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:19 smithi115 ceph-mon[22184]: pgmap v207: 1 pgs: 1 remapped+peering; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:19 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/3494600858' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3ce535d6-4278-46a0-af6d-a7c90a74aaee"} : dispatch 2024-08-28T19:17:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:19 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3ce535d6-4278-46a0-af6d-a7c90a74aaee"} : dispatch 2024-08-28T19:17:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:19 smithi115 ceph-mon[22184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ce535d6-4278-46a0-af6d-a7c90a74aaee"}]': finished 2024-08-28T19:17:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:19 smithi115 ceph-mon[22184]: osdmap e65: 12 total, 11 up, 12 in 2024-08-28T19:17:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:19 smithi115 ceph-mon[22184]: from='client.? 172.21.15.115:0/646467704' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:17:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:19 smithi028 ceph-mon[21769]: pgmap v207: 1 pgs: 1 remapped+peering; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:19 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/3494600858' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3ce535d6-4278-46a0-af6d-a7c90a74aaee"} : dispatch 2024-08-28T19:17:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:19 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3ce535d6-4278-46a0-af6d-a7c90a74aaee"} : dispatch 2024-08-28T19:17:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:19 smithi028 ceph-mon[21769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ce535d6-4278-46a0-af6d-a7c90a74aaee"}]': finished 2024-08-28T19:17:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:19 smithi028 ceph-mon[21769]: osdmap e65: 12 total, 11 up, 12 in 2024-08-28T19:17:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:19.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:19 smithi028 ceph-mon[21769]: from='client.? 172.21.15.115:0/646467704' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:17:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:19 smithi016 ceph-mon[18556]: pgmap v207: 1 pgs: 1 remapped+peering; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:19 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/3494600858' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3ce535d6-4278-46a0-af6d-a7c90a74aaee"} : dispatch 2024-08-28T19:17:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:19 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3ce535d6-4278-46a0-af6d-a7c90a74aaee"} : dispatch 2024-08-28T19:17:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:19 smithi016 ceph-mon[18556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ce535d6-4278-46a0-af6d-a7c90a74aaee"}]': finished 2024-08-28T19:17:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:19 smithi016 ceph-mon[18556]: osdmap e65: 12 total, 11 up, 12 in 2024-08-28T19:17:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:19 smithi016 ceph-mon[18556]: from='client.? 172.21.15.115:0/646467704' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-28T19:17:21.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:21 smithi028 ceph-mon[21769]: pgmap v209: 1 pgs: 1 remapped+peering; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:21 smithi115 ceph-mon[22184]: pgmap v209: 1 pgs: 1 remapped+peering; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:21.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:21 smithi016 ceph-mon[18556]: pgmap v209: 1 pgs: 1 remapped+peering; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:23.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:23 smithi028 ceph-mon[21769]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-28T19:17:23.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:23 smithi115 ceph-mon[22184]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-28T19:17:23.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:23 smithi016 ceph-mon[18556]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-28T19:17:25.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:25 smithi028 ceph-mon[21769]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-08-28T19:17:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:25 smithi115 ceph-mon[22184]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-08-28T19:17:25.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:25 smithi016 ceph-mon[18556]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-08-28T19:17:27.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:27 smithi115 ceph-mon[22184]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-08-28T19:17:27.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:27 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-08-28T19:17:27.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:27 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:27.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:27 smithi028 ceph-mon[21769]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-08-28T19:17:27.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:27 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-08-28T19:17:27.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:27 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:27 smithi016 ceph-mon[18556]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-08-28T19:17:27.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:27 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-08-28T19:17:27.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:27 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:28.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:28 smithi028 ceph-mon[21769]: Deploying daemon osd.11 on smithi115 2024-08-28T19:17:28.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:28 smithi115 ceph-mon[22184]: Deploying daemon osd.11 on smithi115 2024-08-28T19:17:28.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:28 smithi016 ceph-mon[18556]: Deploying daemon osd.11 on smithi115 2024-08-28T19:17:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:29 smithi115 ceph-mon[22184]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-28T19:17:29.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:29 smithi028 ceph-mon[21769]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-28T19:17:29.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:29 smithi016 ceph-mon[18556]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-28T19:17:31.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:31 smithi115 ceph-mon[22184]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-08-28T19:17:31.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:31 smithi028 ceph-mon[21769]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-08-28T19:17:31.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:31 smithi016 ceph-mon[18556]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-08-28T19:17:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:33 smithi115 ceph-mon[22184]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:17:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:33.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:33 smithi016 ceph-mon[18556]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:17:33.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:33.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:33.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:33 smithi028 ceph-mon[21769]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-28T19:17:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:35.232 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:35 smithi115 ceph-mon[22184]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:35.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:35 smithi016 ceph-mon[18556]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:35 smithi028 ceph-mon[21769]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:36.732 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.729+0000 7f33367fc640 1 -- 172.21.15.115:0/80456026 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3340108970 con 0x7f332c0b4bb0 2024-08-28T19:17:36.732 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 11 on host 'smithi115' 2024-08-28T19:17:36.734 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.731+0000 7f334735c640 1 -- 172.21.15.115:0/80456026 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f332c0b4bb0 msgr2=0x7f332c0b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:36.734 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.731+0000 7f334735c640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f332c0b4bb0 0x7f332c0b70c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f333c00baa0 tx=0x7f333c005d90 comp rx=0 tx=0).stop 2024-08-28T19:17:36.734 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.731+0000 7f334735c640 1 -- 172.21.15.115:0/80456026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3340109300 msgr2=0x7f334019a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:36.734 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.731+0000 7f334735c640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3340109300 0x7f334019a760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f333000c8c0 tx=0x7f333000cd80 comp rx=0 tx=0).stop 2024-08-28T19:17:36.734 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.731+0000 7f334735c640 1 -- 172.21.15.115:0/80456026 shutdown_connections 2024-08-28T19:17:36.734 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.731+0000 7f334735c640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f332c0b4bb0 0x7f332c0b70c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:36.734 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.731+0000 7f334735c640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3340109c50 0x7f334019fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:36.734 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.731+0000 7f334735c640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3340109300 0x7f334019a760 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:36.734 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.731+0000 7f334735c640 1 --2- 172.21.15.115:0/80456026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3340101e20 0x7f334019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:36.735 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.731+0000 7f334735c640 1 -- 172.21.15.115:0/80456026 >> 172.21.15.115:0/80456026 conn(0x7f3340079c60 msgr2=0x7f3340107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:36.735 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.732+0000 7f334735c640 1 -- 172.21.15.115:0/80456026 shutdown_connections 2024-08-28T19:17:36.735 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:36.732+0000 7f334735c640 1 -- 172.21.15.115:0/80456026 wait complete. 2024-08-28T19:17:37.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:36 smithi115 ceph-mon[22184]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:37.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:37.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:37.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:37.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:36 smithi016 ceph-mon[18556]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:37.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:37.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:37.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:37.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:36 smithi028 ceph-mon[21769]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:37.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:37.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:37.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:37.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:37.419 DEBUG:teuthology.orchestra.run.smithi115:osd.11> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.11.service 2024-08-28T19:17:37.422 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-08-28T19:17:37.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd stat -f json 2024-08-28T19:17:37.689 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:17:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.154+0000 7f6696a15640 1 -- 172.21.15.16:0/3547635363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6690109240 msgr2=0x7f6690109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.154+0000 7f6696a15640 1 --2- 172.21.15.16:0/3547635363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6690109240 0x7f6690109650 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6684009900 tx=0x7f668402a410 comp rx=0 tx=0).stop 2024-08-28T19:17:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.154+0000 7f6696a15640 1 -- 172.21.15.16:0/3547635363 shutdown_connections 2024-08-28T19:17:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.154+0000 7f6696a15640 1 --2- 172.21.15.16:0/3547635363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6690109b90 0x7f66901103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.154+0000 7f6696a15640 1 --2- 172.21.15.16:0/3547635363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6690109240 0x7f6690109650 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.154+0000 7f6696a15640 1 --2- 172.21.15.16:0/3547635363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6690103900 0x7f6690103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.154+0000 7f6696a15640 1 -- 172.21.15.16:0/3547635363 >> 172.21.15.16:0/3547635363 conn(0x7f669007b5f0 msgr2=0x7f669007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.154+0000 7f6696a15640 1 -- 172.21.15.16:0/3547635363 shutdown_connections 2024-08-28T19:17:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.154+0000 7f6696a15640 1 -- 172.21.15.16:0/3547635363 wait complete. 2024-08-28T19:17:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.155+0000 7f6696a15640 1 Processor -- start 2024-08-28T19:17:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.155+0000 7f6696a15640 1 -- start start 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.155+0000 7f6696a15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6690103900 0x7f66900754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.155+0000 7f6696a15640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6690109240 0x7f66900759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.155+0000 7f6696a15640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6690109b90 0x7f6690075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.155+0000 7f6696a15640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6690113850 con 0x7f6690103900 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.155+0000 7f6696a15640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66901136d0 con 0x7f6690109240 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.155+0000 7f6696a15640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f66901139d0 con 0x7f6690109b90 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.155+0000 7f668ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6690103900 0x7f66900754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6690103900 0x7f66900754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35428/0 (socket says 172.21.15.16:35428) 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668ffff640 1 -- 172.21.15.16:0/3055535941 learned_addr learned my addr 172.21.15.16:0/3055535941 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f6694fb2640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6690109b90 0x7f6690075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668f7fe640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6690109240 0x7f66900759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668ffff640 1 -- 172.21.15.16:0/3055535941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6690109b90 msgr2=0x7f6690075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668ffff640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6690109b90 0x7f6690075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668ffff640 1 -- 172.21.15.16:0/3055535941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6690109240 msgr2=0x7f66900759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668ffff640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6690109240 0x7f66900759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668ffff640 1 -- 172.21.15.16:0/3055535941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66901a8430 con 0x7f6690103900 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668f7fe640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6690109240 0x7f66900759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f6694fb2640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6690109b90 0x7f6690075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668ffff640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6690103900 0x7f66900754b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f668000c920 tx=0x7f668000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668d7fa640 1 -- 172.21.15.16:0/3055535941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6680007c70 con 0x7f6690103900 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f6696a15640 1 -- 172.21.15.16:0/3055535941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66901a8720 con 0x7f6690103900 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668d7fa640 1 -- 172.21.15.16:0/3055535941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6680007e10 con 0x7f6690103900 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f668d7fa640 1 -- 172.21.15.16:0/3055535941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6680018820 con 0x7f6690103900 2024-08-28T19:17:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.156+0000 7f6696a15640 1 -- 172.21.15.16:0/3055535941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66901a8c80 con 0x7f6690103900 2024-08-28T19:17:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.157+0000 7f6696a15640 1 -- 172.21.15.16:0/3055535941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6690103d10 con 0x7f6690103900 2024-08-28T19:17:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.160+0000 7f668d7fa640 1 -- 172.21.15.16:0/3055535941 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f668000f450 con 0x7f6690103900 2024-08-28T19:17:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.160+0000 7f668d7fa640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f66780b4c00 0x7f66780b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.161+0000 7f668f7fe640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f66780b4c00 0x7f66780b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.161+0000 7f668d7fa640 1 -- 172.21.15.16:0/3055535941 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 7024+0+0 (secure 0 0 0) 0x7f66800e5000 con 0x7f6690103900 2024-08-28T19:17:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.161+0000 7f668f7fe640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f66780b4c00 0x7f66780b7110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6684009230 tx=0x7f6684000bc0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:38.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.164+0000 7f668d7fa640 1 -- 172.21.15.16:0/3055535941 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f6680014030 con 0x7f6690103900 2024-08-28T19:17:38.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.353+0000 7f6696a15640 1 -- 172.21.15.16:0/3055535941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f66900767b0 con 0x7f6690103900 2024-08-28T19:17:38.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.354+0000 7f668d7fa640 1 -- 172.21.15.16:0/3055535941 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v65) ==== 74+0+133 (secure 0 0 0) 0x7f66800ac750 con 0x7f6690103900 2024-08-28T19:17:38.356 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:17:38.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 -- 172.21.15.16:0/3055535941 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f66780b4c00 msgr2=0x7f66780b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:38.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f66780b4c00 0x7f66780b7110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6684009230 tx=0x7f6684000bc0 comp rx=0 tx=0).stop 2024-08-28T19:17:38.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 -- 172.21.15.16:0/3055535941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6690103900 msgr2=0x7f66900754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:38.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6690103900 0x7f66900754b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f668000c920 tx=0x7f668000cde0 comp rx=0 tx=0).stop 2024-08-28T19:17:38.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 -- 172.21.15.16:0/3055535941 shutdown_connections 2024-08-28T19:17:38.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6690109b90 0x7f6690075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:38.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f66780b4c00 0x7f66780b7110 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:38.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6690109240 0x7f66900759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 --2- 172.21.15.16:0/3055535941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6690103900 0x7f66900754b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 -- 172.21.15.16:0/3055535941 >> 172.21.15.16:0/3055535941 conn(0x7f669007b5f0 msgr2=0x7f6690107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 -- 172.21.15.16:0/3055535941 shutdown_connections 2024-08-28T19:17:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:38.356+0000 7f6696a15640 1 -- 172.21.15.16:0/3055535941 wait complete. 2024-08-28T19:17:38.814 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:17:38 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:17:38.623+0000 7f0e4d627640 -1 osd.11 0 log_to_monitors true 2024-08-28T19:17:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:38 smithi115 ceph-mon[22184]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:38 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3055535941' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-28T19:17:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:38 smithi115 ceph-mon[22184]: from='osd.11 [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-28T19:17:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:38 smithi115 ceph-mon[22184]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-28T19:17:39.205 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:38 smithi016 ceph-mon[18556]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:39.205 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:38 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3055535941' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-28T19:17:39.205 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:38 smithi016 ceph-mon[18556]: from='osd.11 [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-28T19:17:39.205 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:38 smithi016 ceph-mon[18556]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-28T19:17:39.206 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":65,"num_osds":12,"num_up_osds":11,"osd_up_since":1724872635,"num_in_osds":12,"osd_in_since":1724872638,"num_remapped_pgs":0} 2024-08-28T19:17:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:38 smithi028 ceph-mon[21769]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:38 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3055535941' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-28T19:17:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:38 smithi028 ceph-mon[21769]: from='osd.11 [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-28T19:17:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:38 smithi028 ceph-mon[21769]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-28T19:17:40.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd stat -f json 2024-08-28T19:17:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:39 smithi016 ceph-mon[18556]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-28T19:17:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:39 smithi016 ceph-mon[18556]: osdmap e66: 12 total, 11 up, 12 in 2024-08-28T19:17:40.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:39 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:40.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:39 smithi016 ceph-mon[18556]: from='osd.11 [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:17:40.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:39 smithi016 ceph-mon[18556]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:17:40.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:39 smithi115 ceph-mon[22184]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-28T19:17:40.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:39 smithi115 ceph-mon[22184]: osdmap e66: 12 total, 11 up, 12 in 2024-08-28T19:17:40.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:39 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:40.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:39 smithi115 ceph-mon[22184]: from='osd.11 [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:17:40.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:39 smithi115 ceph-mon[22184]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:17:40.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:39 smithi028 ceph-mon[21769]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-28T19:17:40.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:39 smithi028 ceph-mon[21769]: osdmap e66: 12 total, 11 up, 12 in 2024-08-28T19:17:40.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:39 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:40.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:39 smithi028 ceph-mon[21769]: from='osd.11 [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:17:40.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:39 smithi028 ceph-mon[21769]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]} : dispatch 2024-08-28T19:17:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:17:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.929+0000 7fe9189f0640 1 -- 172.21.15.16:0/2868599855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914108150 msgr2=0x7fe914108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.929+0000 7fe9189f0640 1 --2- 172.21.15.16:0/2868599855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914108150 0x7fe914108560 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe908009900 tx=0x7fe90802a3f0 comp rx=0 tx=0).stop 2024-08-28T19:17:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.929+0000 7fe9189f0640 1 -- 172.21.15.16:0/2868599855 shutdown_connections 2024-08-28T19:17:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.929+0000 7fe9189f0640 1 --2- 172.21.15.16:0/2868599855 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe914108aa0 0x7fe91410f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.929+0000 7fe9189f0640 1 --2- 172.21.15.16:0/2868599855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914108150 0x7fe914108560 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.929+0000 7fe9189f0640 1 --2- 172.21.15.16:0/2868599855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe914102810 0x7fe914102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.929+0000 7fe9189f0640 1 -- 172.21.15.16:0/2868599855 >> 172.21.15.16:0/2868599855 conn(0x7fe9140794d0 msgr2=0x7fe9140798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.929+0000 7fe9189f0640 1 -- 172.21.15.16:0/2868599855 shutdown_connections 2024-08-28T19:17:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.929+0000 7fe9189f0640 1 -- 172.21.15.16:0/2868599855 wait complete. 2024-08-28T19:17:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe9189f0640 1 Processor -- start 2024-08-28T19:17:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe9189f0640 1 -- start start 2024-08-28T19:17:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe9189f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914102810 0x7fe91419b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe9189f0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe914108150 0x7fe91419b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe9189f0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe914108aa0 0x7fe91419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe9189f0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9141126d0 con 0x7fe914102810 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe9189f0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe914112550 con 0x7fe914108150 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe9189f0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe914112850 con 0x7fe914108aa0 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe91259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914102810 0x7fe91419b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe91259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914102810 0x7fe91419b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35452/0 (socket says 172.21.15.16:35452) 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe91259c640 1 -- 172.21.15.16:0/3475995913 learned_addr learned my addr 172.21.15.16:0/3475995913 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe911d9b640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe914108150 0x7fe91419b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe912d9d640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe914108aa0 0x7fe91419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe91259c640 1 -- 172.21.15.16:0/3475995913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe914108aa0 msgr2=0x7fe91419bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe91259c640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe914108aa0 0x7fe91419bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe91259c640 1 -- 172.21.15.16:0/3475995913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe914108150 msgr2=0x7fe91419b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe91259c640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe914108150 0x7fe91419b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe91259c640 1 -- 172.21.15.16:0/3475995913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9141a6320 con 0x7fe914102810 2024-08-28T19:17:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.930+0000 7fe911d9b640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe914108150 0x7fe91419b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:17:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.931+0000 7fe91259c640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914102810 0x7fe91419b0a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe90400d950 tx=0x7fe90400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.931+0000 7fe9037fe640 1 -- 172.21.15.16:0/3475995913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904017030 con 0x7fe914102810 2024-08-28T19:17:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.931+0000 7fe9189f0640 1 -- 172.21.15.16:0/3475995913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9141a6610 con 0x7fe914102810 2024-08-28T19:17:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.931+0000 7fe9037fe640 1 -- 172.21.15.16:0/3475995913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe904004020 con 0x7fe914102810 2024-08-28T19:17:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.931+0000 7fe9037fe640 1 -- 172.21.15.16:0/3475995913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9040047a0 con 0x7fe914102810 2024-08-28T19:17:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.931+0000 7fe9189f0640 1 -- 172.21.15.16:0/3475995913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9141a6b20 con 0x7fe914102810 2024-08-28T19:17:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.932+0000 7fe9189f0640 1 -- 172.21.15.16:0/3475995913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe914103a90 con 0x7fe914102810 2024-08-28T19:17:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.933+0000 7fe9037fe640 1 -- 172.21.15.16:0/3475995913 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fe904009a30 con 0x7fe914102810 2024-08-28T19:17:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.934+0000 7fe9037fe640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8fc0b4b80 0x7fe8fc0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.934+0000 7fe911d9b640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8fc0b4b80 0x7fe8fc0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.934+0000 7fe9037fe640 1 -- 172.21.15.16:0/3475995913 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 7062+0+0 (secure 0 0 0) 0x7fe9040e4800 con 0x7fe914102810 2024-08-28T19:17:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.935+0000 7fe911d9b640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8fc0b4b80 0x7fe8fc0b7090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe908000c00 tx=0x7fe908032da0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:40.939+0000 7fe9037fe640 1 -- 172.21.15.16:0/3475995913 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fe9040abe10 con 0x7fe914102810 2024-08-28T19:17:40.943 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:17:40.646+0000 7f0e495a9640 -1 osd.11 0 waiting for initial osdmap 2024-08-28T19:17:40.944 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:17:40.658+0000 7f0e4139f640 -1 osd.11 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-28T19:17:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.126+0000 7fe9189f0640 1 -- 172.21.15.16:0/3475995913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fe91406bd90 con 0x7fe914102810 2024-08-28T19:17:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.126+0000 7fe9037fe640 1 -- 172.21.15.16:0/3475995913 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v67) ==== 74+0+133 (secure 0 0 0) 0x7fe9040ab710 con 0x7fe914102810 2024-08-28T19:17:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:17:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.128+0000 7fe9189f0640 1 -- 172.21.15.16:0/3475995913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8fc0b4b80 msgr2=0x7fe8fc0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.128+0000 7fe9189f0640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8fc0b4b80 0x7fe8fc0b7090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe908000c00 tx=0x7fe908032da0 comp rx=0 tx=0).stop 2024-08-28T19:17:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.128+0000 7fe9189f0640 1 -- 172.21.15.16:0/3475995913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914102810 msgr2=0x7fe91419b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.128+0000 7fe9189f0640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914102810 0x7fe91419b0a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe90400d950 tx=0x7fe90400de10 comp rx=0 tx=0).stop 2024-08-28T19:17:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.128+0000 7fe9189f0640 1 -- 172.21.15.16:0/3475995913 shutdown_connections 2024-08-28T19:17:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.128+0000 7fe9189f0640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe914108aa0 0x7fe91419bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.128+0000 7fe9189f0640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8fc0b4b80 0x7fe8fc0b7090 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.128+0000 7fe9189f0640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe914108150 0x7fe91419b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.128+0000 7fe9189f0640 1 --2- 172.21.15.16:0/3475995913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914102810 0x7fe91419b0a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:41.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.128+0000 7fe9189f0640 1 -- 172.21.15.16:0/3475995913 >> 172.21.15.16:0/3475995913 conn(0x7fe9140794d0 msgr2=0x7fe914105100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:41.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.129+0000 7fe9189f0640 1 -- 172.21.15.16:0/3475995913 shutdown_connections 2024-08-28T19:17:41.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:41.129+0000 7fe9189f0640 1 -- 172.21.15.16:0/3475995913 wait complete. 2024-08-28T19:17:41.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:41.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:17:41.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: osdmap e67: 12 total, 11 up, 12 in 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:40 smithi016 ceph-mon[18556]: from='osd.11 ' entity='osd.11' 2024-08-28T19:17:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:17:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: osdmap e67: 12 total, 11 up, 12 in 2024-08-28T19:17:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:40 smithi028 ceph-mon[21769]: from='osd.11 ' entity='osd.11' 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: osdmap e67: 12 total, 11 up, 12 in 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-28T19:17:41.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:41.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:41.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:41.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:40 smithi115 ceph-mon[22184]: from='osd.11 ' entity='osd.11' 2024-08-28T19:17:41.864 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"num_osds":12,"num_up_osds":11,"osd_up_since":1724872635,"num_in_osds":12,"osd_in_since":1724872638,"num_remapped_pgs":0} 2024-08-28T19:17:42.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:41 smithi016 ceph-mon[18556]: purged_snaps scrub starts 2024-08-28T19:17:42.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:41 smithi016 ceph-mon[18556]: purged_snaps scrub ok 2024-08-28T19:17:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:41 smithi016 ceph-mon[18556]: Detected new or changed devices on smithi115 2024-08-28T19:17:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:41 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi115 to 4272M 2024-08-28T19:17:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:41 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3475995913' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-28T19:17:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:41 smithi016 ceph-mon[18556]: osd.11 [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111] boot 2024-08-28T19:17:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:41 smithi016 ceph-mon[18556]: osdmap e68: 12 total, 12 up, 12 in 2024-08-28T19:17:42.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:42.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:41 smithi028 ceph-mon[21769]: purged_snaps scrub starts 2024-08-28T19:17:42.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:41 smithi028 ceph-mon[21769]: purged_snaps scrub ok 2024-08-28T19:17:42.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:41 smithi028 ceph-mon[21769]: Detected new or changed devices on smithi115 2024-08-28T19:17:42.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:41 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi115 to 4272M 2024-08-28T19:17:42.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:42.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:41 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3475995913' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-28T19:17:42.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:41 smithi028 ceph-mon[21769]: osd.11 [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111] boot 2024-08-28T19:17:42.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:41 smithi028 ceph-mon[21769]: osdmap e68: 12 total, 12 up, 12 in 2024-08-28T19:17:42.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:42.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:41 smithi115 ceph-mon[22184]: purged_snaps scrub starts 2024-08-28T19:17:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:41 smithi115 ceph-mon[22184]: purged_snaps scrub ok 2024-08-28T19:17:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:41 smithi115 ceph-mon[22184]: Detected new or changed devices on smithi115 2024-08-28T19:17:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:41 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi115 to 4272M 2024-08-28T19:17:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:41 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3475995913' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-28T19:17:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:41 smithi115 ceph-mon[22184]: osd.11 [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111] boot 2024-08-28T19:17:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:41 smithi115 ceph-mon[22184]: osdmap e68: 12 total, 12 up, 12 in 2024-08-28T19:17:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-28T19:17:42.865 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd stat -f json 2024-08-28T19:17:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:17:43.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:42 smithi016 ceph-mon[18556]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:42 smithi028 ceph-mon[21769]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:43.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:42 smithi115 ceph-mon[22184]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-08-28T19:17:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.613+0000 7fd156cfb640 1 -- 172.21.15.16:0/75398369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd150108b10 msgr2=0x7fd15010f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.613+0000 7fd156cfb640 1 --2- 172.21.15.16:0/75398369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd150108b10 0x7fd15010f340 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd148009ca0 tx=0x7fd14802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:17:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.613+0000 7fd156cfb640 1 -- 172.21.15.16:0/75398369 shutdown_connections 2024-08-28T19:17:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.613+0000 7fd156cfb640 1 --2- 172.21.15.16:0/75398369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd150108b10 0x7fd15010f340 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.613+0000 7fd156cfb640 1 --2- 172.21.15.16:0/75398369 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1501081c0 0x7fd1501085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.613+0000 7fd156cfb640 1 --2- 172.21.15.16:0/75398369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd150102880 0x7fd150102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.613+0000 7fd156cfb640 1 -- 172.21.15.16:0/75398369 >> 172.21.15.16:0/75398369 conn(0x7fd150079510 msgr2=0x7fd150079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.613+0000 7fd156cfb640 1 -- 172.21.15.16:0/75398369 shutdown_connections 2024-08-28T19:17:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.613+0000 7fd156cfb640 1 -- 172.21.15.16:0/75398369 wait complete. 2024-08-28T19:17:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.613+0000 7fd156cfb640 1 Processor -- start 2024-08-28T19:17:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd156cfb640 1 -- start start 2024-08-28T19:17:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd156cfb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd150102880 0x7fd150199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd156cfb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1501081c0 0x7fd1501996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd156cfb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd150108b10 0x7fd15019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd156cfb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1501127e0 con 0x7fd1501081c0 2024-08-28T19:17:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd156cfb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd150112660 con 0x7fd150108b10 2024-08-28T19:17:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd156cfb640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd150112960 con 0x7fd150102880 2024-08-28T19:17:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd147fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1501081c0 0x7fd1501996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd147fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1501081c0 0x7fd1501996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35472/0 (socket says 172.21.15.16:35472) 2024-08-28T19:17:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd147fff640 1 -- 172.21.15.16:0/693101412 learned_addr learned my addr 172.21.15.16:0/693101412 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:17:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd154a97640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd150102880 0x7fd150199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd155298640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd150108b10 0x7fd15019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd147fff640 1 -- 172.21.15.16:0/693101412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd150102880 msgr2=0x7fd150199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd147fff640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd150102880 0x7fd150199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd147fff640 1 -- 172.21.15.16:0/693101412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd150108b10 msgr2=0x7fd15019e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd147fff640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd150108b10 0x7fd15019e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd147fff640 1 -- 172.21.15.16:0/693101412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15019f0c0 con 0x7fd1501081c0 2024-08-28T19:17:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd154a97640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd150102880 0x7fd150199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:17:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.614+0000 7fd147fff640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1501081c0 0x7fd1501996f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd14000d750 tx=0x7fd14000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.615+0000 7fd145ffb640 1 -- 172.21.15.16:0/693101412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140017070 con 0x7fd1501081c0 2024-08-28T19:17:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.615+0000 7fd145ffb640 1 -- 172.21.15.16:0/693101412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1400054d0 con 0x7fd1501081c0 2024-08-28T19:17:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.615+0000 7fd156cfb640 1 -- 172.21.15.16:0/693101412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd15019f3b0 con 0x7fd1501081c0 2024-08-28T19:17:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.615+0000 7fd145ffb640 1 -- 172.21.15.16:0/693101412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140004c30 con 0x7fd1501081c0 2024-08-28T19:17:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.615+0000 7fd156cfb640 1 -- 172.21.15.16:0/693101412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1501a5b10 con 0x7fd1501081c0 2024-08-28T19:17:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.616+0000 7fd156cfb640 1 -- 172.21.15.16:0/693101412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd150103b00 con 0x7fd1501081c0 2024-08-28T19:17:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.617+0000 7fd145ffb640 1 -- 172.21.15.16:0/693101412 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fd140004dd0 con 0x7fd1501081c0 2024-08-28T19:17:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.618+0000 7fd145ffb640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1380b4b80 0x7fd1380b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.618+0000 7fd154a97640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1380b4b80 0x7fd1380b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.618+0000 7fd145ffb640 1 -- 172.21.15.16:0/693101412 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7fd1400e4940 con 0x7fd1501081c0 2024-08-28T19:17:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.619+0000 7fd154a97640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1380b4b80 0x7fd1380b7090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009c20 tx=0x7fd14c005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.623+0000 7fd145ffb640 1 -- 172.21.15.16:0/693101412 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fd1400abda0 con 0x7fd1501081c0 2024-08-28T19:17:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.811+0000 7fd156cfb640 1 -- 172.21.15.16:0/693101412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fd15010a900 con 0x7fd1501081c0 2024-08-28T19:17:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.811+0000 7fd145ffb640 1 -- 172.21.15.16:0/693101412 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v69) ==== 74+0+133 (secure 0 0 0) 0x7fd1400ab6a0 con 0x7fd1501081c0 2024-08-28T19:17:43.813 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:17:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.813+0000 7fd156cfb640 1 -- 172.21.15.16:0/693101412 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1380b4b80 msgr2=0x7fd1380b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.813+0000 7fd156cfb640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1380b4b80 0x7fd1380b7090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009c20 tx=0x7fd14c005f90 comp rx=0 tx=0).stop 2024-08-28T19:17:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.813+0000 7fd156cfb640 1 -- 172.21.15.16:0/693101412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1501081c0 msgr2=0x7fd1501996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.813+0000 7fd156cfb640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1501081c0 0x7fd1501996f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd14000d750 tx=0x7fd14000dc10 comp rx=0 tx=0).stop 2024-08-28T19:17:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.813+0000 7fd156cfb640 1 -- 172.21.15.16:0/693101412 shutdown_connections 2024-08-28T19:17:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.813+0000 7fd156cfb640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1380b4b80 0x7fd1380b7090 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.813+0000 7fd156cfb640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd150108b10 0x7fd15019e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.813+0000 7fd156cfb640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1501081c0 0x7fd1501996f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.813+0000 7fd156cfb640 1 --2- 172.21.15.16:0/693101412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd150102880 0x7fd150199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.813+0000 7fd156cfb640 1 -- 172.21.15.16:0/693101412 >> 172.21.15.16:0/693101412 conn(0x7fd150079510 msgr2=0x7fd1501061b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.814+0000 7fd156cfb640 1 -- 172.21.15.16:0/693101412 shutdown_connections 2024-08-28T19:17:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:43.814+0000 7fd156cfb640 1 -- 172.21.15.16:0/693101412 wait complete. 2024-08-28T19:17:44.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:43 smithi016 ceph-mon[18556]: osdmap e69: 12 total, 12 up, 12 in 2024-08-28T19:17:44.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:43 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/693101412' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-28T19:17:44.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:43 smithi028 ceph-mon[21769]: osdmap e69: 12 total, 12 up, 12 in 2024-08-28T19:17:44.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:43 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/693101412' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-28T19:17:44.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:43 smithi115 ceph-mon[22184]: osdmap e69: 12 total, 12 up, 12 in 2024-08-28T19:17:44.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:43 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/693101412' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-28T19:17:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":69,"num_osds":12,"num_up_osds":12,"osd_up_since":1724872661,"num_in_osds":12,"osd_in_since":1724872638,"num_remapped_pgs":0} 2024-08-28T19:17:44.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:17:44.870 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:17:45.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:44 smithi016 ceph-mon[18556]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:45.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:44 smithi028 ceph-mon[21769]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:45.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.348+0000 7fe04772f640 1 -- 172.21.15.16:0/2494273887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040108a90 msgr2=0x7fe04010f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.348+0000 7fe04772f640 1 --2- 172.21.15.16:0/2494273887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040108a90 0x7fe04010f2c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe038009ca0 tx=0x7fe03802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.348+0000 7fe04772f640 1 -- 172.21.15.16:0/2494273887 shutdown_connections 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.348+0000 7fe04772f640 1 --2- 172.21.15.16:0/2494273887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040108a90 0x7fe04010f2c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.348+0000 7fe04772f640 1 --2- 172.21.15.16:0/2494273887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe040108140 0x7fe040108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.348+0000 7fe04772f640 1 --2- 172.21.15.16:0/2494273887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe040102800 0x7fe040102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.348+0000 7fe04772f640 1 -- 172.21.15.16:0/2494273887 >> 172.21.15.16:0/2494273887 conn(0x7fe0400794d0 msgr2=0x7fe0400798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.348+0000 7fe04772f640 1 -- 172.21.15.16:0/2494273887 shutdown_connections 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.348+0000 7fe04772f640 1 -- 172.21.15.16:0/2494273887 wait complete. 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe04772f640 1 Processor -- start 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe04772f640 1 -- start start 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe04772f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040102800 0x7fe0401991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe04772f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe040108140 0x7fe040199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe04772f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe040108a90 0x7fe04019e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe04772f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe040112780 con 0x7fe040102800 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe04772f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe040112600 con 0x7fe040108a90 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe04772f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe040112900 con 0x7fe040108140 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe0454cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040102800 0x7fe0401991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe0454cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040102800 0x7fe0401991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35596/0 (socket says 172.21.15.16:35596) 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe0454cb640 1 -- 172.21.15.16:0/3163758118 learned_addr learned my addr 172.21.15.16:0/3163758118 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.349+0000 7fe045ccc640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe040108a90 0x7fe04019e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe044cca640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe040108140 0x7fe040199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe0454cb640 1 -- 172.21.15.16:0/3163758118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe040108140 msgr2=0x7fe040199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe0454cb640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe040108140 0x7fe040199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe0454cb640 1 -- 172.21.15.16:0/3163758118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe040108a90 msgr2=0x7fe04019e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe0454cb640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe040108a90 0x7fe04019e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe0454cb640 1 -- 172.21.15.16:0/3163758118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe04019f120 con 0x7fe040102800 2024-08-28T19:17:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe0454cb640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040102800 0x7fe0401991b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe03c00c970 tx=0x7fe03c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe0327fc640 1 -- 172.21.15.16:0/3163758118 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c007bb0 con 0x7fe040102800 2024-08-28T19:17:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe0327fc640 1 -- 172.21.15.16:0/3163758118 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe03c007d50 con 0x7fe040102800 2024-08-28T19:17:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe0327fc640 1 -- 172.21.15.16:0/3163758118 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c00f450 con 0x7fe040102800 2024-08-28T19:17:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe04772f640 1 -- 172.21.15.16:0/3163758118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe04019f410 con 0x7fe040102800 2024-08-28T19:17:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.350+0000 7fe04772f640 1 -- 172.21.15.16:0/3163758118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0401a5b70 con 0x7fe040102800 2024-08-28T19:17:45.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.351+0000 7fe04772f640 1 -- 172.21.15.16:0/3163758118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe040102c10 con 0x7fe040102800 2024-08-28T19:17:45.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.353+0000 7fe0327fc640 1 -- 172.21.15.16:0/3163758118 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fe03c004f00 con 0x7fe040102800 2024-08-28T19:17:45.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.354+0000 7fe0327fc640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe02c0b4b80 0x7fe02c0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:45.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.354+0000 7fe044cca640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe02c0b4b80 0x7fe02c0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:45.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.354+0000 7fe0327fc640 1 -- 172.21.15.16:0/3163758118 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7fe03c0e4fb0 con 0x7fe040102800 2024-08-28T19:17:45.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.354+0000 7fe044cca640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe02c0b4b80 0x7fe02c0b7090 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe040107880 tx=0x7fe034008040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:45.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.357+0000 7fe0327fc640 1 -- 172.21.15.16:0/3163758118 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fe03c014030 con 0x7fe040102800 2024-08-28T19:17:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:44 smithi115 ceph-mon[22184]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.543+0000 7fe04772f640 1 -- 172.21.15.16:0/3163758118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe04019a2e0 con 0x7fe040102800 2024-08-28T19:17:45.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.544+0000 7fe0327fc640 1 -- 172.21.15.16:0/3163758118 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+18749 (secure 0 0 0) 0x7fe04019a2e0 con 0x7fe040102800 2024-08-28T19:17:45.546 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:17:45.546 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":69,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:17:42.949200+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:17:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.546+0000 7fe04772f640 1 -- 172.21.15.16:0/3163758118 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe02c0b4b80 msgr2=0x7fe02c0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.546+0000 7fe04772f640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe02c0b4b80 0x7fe02c0b7090 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe040107880 tx=0x7fe034008040 comp rx=0 tx=0).stop 2024-08-28T19:17:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.546+0000 7fe04772f640 1 -- 172.21.15.16:0/3163758118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040102800 msgr2=0x7fe0401991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.546+0000 7fe04772f640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040102800 0x7fe0401991b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe03c00c970 tx=0x7fe03c00ce30 comp rx=0 tx=0).stop 2024-08-28T19:17:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.546+0000 7fe04772f640 1 -- 172.21.15.16:0/3163758118 shutdown_connections 2024-08-28T19:17:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.546+0000 7fe04772f640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe02c0b4b80 0x7fe02c0b7090 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.546+0000 7fe04772f640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe040108a90 0x7fe04019e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.546+0000 7fe04772f640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe040108140 0x7fe040199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:45.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.546+0000 7fe04772f640 1 --2- 172.21.15.16:0/3163758118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040102800 0x7fe0401991b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:45.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.546+0000 7fe04772f640 1 -- 172.21.15.16:0/3163758118 >> 172.21.15.16:0/3163758118 conn(0x7fe0400794d0 msgr2=0x7fe0401050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:45.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.547+0000 7fe04772f640 1 -- 172.21.15.16:0/3163758118 shutdown_connections 2024-08-28T19:17:45.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:45.547+0000 7fe04772f640 1 -- 172.21.15.16:0/3163758118 wait complete. 2024-08-28T19:17:46.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:45 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3163758118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:17:46.195 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-28T19:13:55.828283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-28T19:17:46.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-08-28T19:17:46.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:45 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3163758118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:17:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:45 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3163758118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:17:46.446 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:17:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.037+0000 7f9a7a292640 1 -- 172.21.15.16:0/1543804622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a74108ac0 msgr2=0x7f9a7410f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.037+0000 7f9a7a292640 1 --2- 172.21.15.16:0/1543804622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a74108ac0 0x7f9a7410f2f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009ca0 tx=0x7f9a6c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:17:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.037+0000 7f9a7a292640 1 -- 172.21.15.16:0/1543804622 shutdown_connections 2024-08-28T19:17:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.037+0000 7f9a7a292640 1 --2- 172.21.15.16:0/1543804622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a74108ac0 0x7f9a7410f2f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.037+0000 7f9a7a292640 1 --2- 172.21.15.16:0/1543804622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a74108170 0x7f9a74108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.037+0000 7f9a7a292640 1 --2- 172.21.15.16:0/1543804622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a74102880 0x7f9a74102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.037+0000 7f9a7a292640 1 -- 172.21.15.16:0/1543804622 >> 172.21.15.16:0/1543804622 conn(0x7f9a74079550 msgr2=0x7f9a74079960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.037+0000 7f9a7a292640 1 -- 172.21.15.16:0/1543804622 shutdown_connections 2024-08-28T19:17:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.037+0000 7f9a7a292640 1 -- 172.21.15.16:0/1543804622 wait complete. 2024-08-28T19:17:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a7a292640 1 Processor -- start 2024-08-28T19:17:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a7a292640 1 -- start start 2024-08-28T19:17:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a7a292640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a74102880 0x7f9a7419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a7a292640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a74108170 0x7f9a7419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a7a292640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a74108ac0 0x7f9a7419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a7a292640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9a74112790 con 0x7f9a74108ac0 2024-08-28T19:17:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a7a292640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a74112610 con 0x7f9a74108170 2024-08-28T19:17:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a7a292640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9a74112910 con 0x7f9a74102880 2024-08-28T19:17:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a7882f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a74108ac0 0x7f9a7419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a7882f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a74108ac0 0x7f9a7419fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35614/0 (socket says 172.21.15.16:35614) 2024-08-28T19:17:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a7882f640 1 -- 172.21.15.16:0/1034429266 learned_addr learned my addr 172.21.15.16:0/1034429266 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:17:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.038+0000 7f9a737fe640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a74108170 0x7f9a7419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a73fff640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a74102880 0x7f9a7419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a737fe640 1 -- 172.21.15.16:0/1034429266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a74102880 msgr2=0x7f9a7419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a737fe640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a74102880 0x7f9a7419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a737fe640 1 -- 172.21.15.16:0/1034429266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a74108ac0 msgr2=0x7f9a7419fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a737fe640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a74108ac0 0x7f9a7419fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a737fe640 1 -- 172.21.15.16:0/1034429266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a741a01c0 con 0x7f9a74108170 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a7882f640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a74108ac0 0x7f9a7419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a737fe640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a74108170 0x7f9a7419a7f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a7406aad0 tx=0x7f9a6800cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a717fa640 1 -- 172.21.15.16:0/1034429266 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68007bb0 con 0x7f9a74108170 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a7a292640 1 -- 172.21.15.16:0/1034429266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a741a04b0 con 0x7f9a74108170 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a717fa640 1 -- 172.21.15.16:0/1034429266 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a68007d50 con 0x7f9a74108170 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a717fa640 1 -- 172.21.15.16:0/1034429266 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68004d80 con 0x7f9a74108170 2024-08-28T19:17:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.039+0000 7f9a7a292640 1 -- 172.21.15.16:0/1034429266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a741a6c10 con 0x7f9a74108170 2024-08-28T19:17:47.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:46 smithi016 ceph-mon[18556]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.041+0000 7f9a7a292640 1 -- 172.21.15.16:0/1034429266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a74102c90 con 0x7f9a74108170 2024-08-28T19:17:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.042+0000 7f9a717fa640 1 -- 172.21.15.16:0/1034429266 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f9a68005020 con 0x7f9a74108170 2024-08-28T19:17:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.043+0000 7f9a717fa640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9a5c0b4b80 0x7f9a5c0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.043+0000 7f9a73fff640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9a5c0b4b80 0x7f9a5c0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.043+0000 7f9a717fa640 1 -- 172.21.15.16:0/1034429266 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7f9a680e49c0 con 0x7f9a74108170 2024-08-28T19:17:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.043+0000 7f9a73fff640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9a5c0b4b80 0x7f9a5c0b7090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400bbe0 tx=0x7f9a6400b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.046+0000 7f9a717fa640 1 -- 172.21.15.16:0/1034429266 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f9a68014030 con 0x7f9a74108170 2024-08-28T19:17:47.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.230+0000 7f9a7a292640 1 -- 172.21.15.16:0/1034429266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9a7410b8f0 con 0x7f9a74108170 2024-08-28T19:17:47.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.230+0000 7f9a717fa640 1 -- 172.21.15.16:0/1034429266 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) ==== 93+0+10 (secure 0 0 0) 0x7f9a680ab7d0 con 0x7f9a74108170 2024-08-28T19:17:47.232 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-08-28T19:17:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.232+0000 7f9a7a292640 1 -- 172.21.15.16:0/1034429266 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9a5c0b4b80 msgr2=0x7f9a5c0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.232+0000 7f9a7a292640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9a5c0b4b80 0x7f9a5c0b7090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400bbe0 tx=0x7f9a6400b390 comp rx=0 tx=0).stop 2024-08-28T19:17:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.232+0000 7f9a7a292640 1 -- 172.21.15.16:0/1034429266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a74108170 msgr2=0x7f9a7419a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.232+0000 7f9a7a292640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a74108170 0x7f9a7419a7f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a7406aad0 tx=0x7f9a6800cdd0 comp rx=0 tx=0).stop 2024-08-28T19:17:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.232+0000 7f9a7a292640 1 -- 172.21.15.16:0/1034429266 shutdown_connections 2024-08-28T19:17:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.232+0000 7f9a7a292640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a74108ac0 0x7f9a7419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.232+0000 7f9a7a292640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9a5c0b4b80 0x7f9a5c0b7090 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.233+0000 7f9a7a292640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a74108170 0x7f9a7419a7f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.233+0000 7f9a7a292640 1 --2- 172.21.15.16:0/1034429266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a74102880 0x7f9a7419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.233+0000 7f9a7a292640 1 -- 172.21.15.16:0/1034429266 >> 172.21.15.16:0/1034429266 conn(0x7f9a74079550 msgr2=0x7f9a741017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:47.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.233+0000 7f9a7a292640 1 -- 172.21.15.16:0/1034429266 shutdown_connections 2024-08-28T19:17:47.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:47.233+0000 7f9a7a292640 1 -- 172.21.15.16:0/1034429266 wait complete. 2024-08-28T19:17:47.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:46 smithi028 ceph-mon[21769]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:46 smithi115 ceph-mon[22184]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:47.897 INFO:tasks.cephadm:Adding mds.a on smithi016 2024-08-28T19:17:47.897 INFO:tasks.cephadm:Adding mds.d on smithi016 2024-08-28T19:17:47.897 INFO:tasks.cephadm:Adding mds.g on smithi016 2024-08-28T19:17:47.897 INFO:tasks.cephadm:Adding mds.j on smithi016 2024-08-28T19:17:47.897 INFO:tasks.cephadm:Adding mds.b on smithi028 2024-08-28T19:17:47.897 INFO:tasks.cephadm:Adding mds.e on smithi028 2024-08-28T19:17:47.897 INFO:tasks.cephadm:Adding mds.h on smithi028 2024-08-28T19:17:47.897 INFO:tasks.cephadm:Adding mds.k on smithi028 2024-08-28T19:17:47.897 INFO:tasks.cephadm:Adding mds.c on smithi115 2024-08-28T19:17:47.898 INFO:tasks.cephadm:Adding mds.f on smithi115 2024-08-28T19:17:47.898 INFO:tasks.cephadm:Adding mds.i on smithi115 2024-08-28T19:17:47.898 INFO:tasks.cephadm:Adding mds.l on smithi115 2024-08-28T19:17:47.898 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph orch apply mds all '12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi028=b;smithi028=e;smithi028=h;smithi028=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l' 2024-08-28T19:17:48.154 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:17:48.229 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:47 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1034429266' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-28T19:17:48.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:47 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1034429266' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-28T19:17:48.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:47 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1034429266' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-28T19:17:48.663 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.646+0000 7f3c7faa2640 1 -- 172.21.15.115:0/3208380163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c78109240 msgr2=0x7f3c78109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:48.663 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.646+0000 7f3c7faa2640 1 --2- 172.21.15.115:0/3208380163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c78109240 0x7f3c78109650 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009900 tx=0x7f3c6802a410 comp rx=0 tx=0).stop 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.646+0000 7f3c7faa2640 1 -- 172.21.15.115:0/3208380163 shutdown_connections 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.646+0000 7f3c7faa2640 1 --2- 172.21.15.115:0/3208380163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78109b90 0x7f3c781103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.646+0000 7f3c7faa2640 1 --2- 172.21.15.115:0/3208380163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c78109240 0x7f3c78109650 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.646+0000 7f3c7faa2640 1 --2- 172.21.15.115:0/3208380163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c78103900 0x7f3c78103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.646+0000 7f3c7faa2640 1 -- 172.21.15.115:0/3208380163 >> 172.21.15.115:0/3208380163 conn(0x7f3c7807b5f0 msgr2=0x7f3c7807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.646+0000 7f3c7faa2640 1 -- 172.21.15.115:0/3208380163 shutdown_connections 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.646+0000 7f3c7faa2640 1 -- 172.21.15.115:0/3208380163 wait complete. 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.647+0000 7f3c7faa2640 1 Processor -- start 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.647+0000 7f3c7faa2640 1 -- start start 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.647+0000 7f3c7faa2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c78103900 0x7f3c7819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.647+0000 7f3c7faa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78109240 0x7f3c7819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.647+0000 7f3c7faa2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c78109b90 0x7f3c7819fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.647+0000 7f3c7faa2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78113850 con 0x7f3c78103900 2024-08-28T19:17:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.647+0000 7f3c7faa2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c781136d0 con 0x7f3c78109240 2024-08-28T19:17:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.647+0000 7f3c7faa2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3c781139d0 con 0x7f3c78109b90 2024-08-28T19:17:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7e03f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c78109b90 0x7f3c7819fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7d03d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78109240 0x7f3c7819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7e03f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c78109b90 0x7f3c7819fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60364/0 (socket says 172.21.15.115:60364) 2024-08-28T19:17:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7e03f640 1 -- 172.21.15.115:0/1042043644 learned_addr learned my addr 172.21.15.115:0/1042043644 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-08-28T19:17:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7d83e640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c78103900 0x7f3c7819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7e03f640 1 -- 172.21.15.115:0/1042043644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78109240 msgr2=0x7f3c7819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7e03f640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78109240 0x7f3c7819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7e03f640 1 -- 172.21.15.115:0/1042043644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c78103900 msgr2=0x7f3c7819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7e03f640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c78103900 0x7f3c7819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7e03f640 1 -- 172.21.15.115:0/1042043644 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c781a0170 con 0x7f3c78109b90 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7e03f640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c78109b90 0x7f3c7819fa40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000eb10 tx=0x7f3c7000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c6effd640 1 -- 172.21.15.115:0/1042043644 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c70004b20 con 0x7f3c78109b90 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c6effd640 1 -- 172.21.15.115:0/1042043644 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c70004cc0 con 0x7f3c78109b90 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c6effd640 1 -- 172.21.15.115:0/1042043644 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c70010670 con 0x7f3c78109b90 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7faa2640 1 -- 172.21.15.115:0/1042043644 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c781a0460 con 0x7f3c78109b90 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.648+0000 7f3c7faa2640 1 -- 172.21.15.115:0/1042043644 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c781a6bc0 con 0x7f3c78109b90 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.649+0000 7f3c7faa2640 1 -- 172.21.15.115:0/1042043644 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c78103d10 con 0x7f3c78109b90 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.652+0000 7f3c6effd640 1 -- 172.21.15.115:0/1042043644 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f3c70005000 con 0x7f3c78109b90 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.652+0000 7f3c6effd640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c640b4b80 0x7f3c640b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.653+0000 7f3c6effd640 1 -- 172.21.15.115:0/1042043644 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7f3c700e5600 con 0x7f3c78109b90 2024-08-28T19:17:48.667 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.653+0000 7f3c7d83e640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c640b4b80 0x7f3c640b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:48.667 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.653+0000 7f3c7d83e640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c640b4b80 0x7f3c640b7090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3c740058c0 tx=0x7f3c7400a4c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:48.667 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.656+0000 7f3c6effd640 1 -- 172.21.15.115:0/1042043644 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f3c70056130 con 0x7f3c78109b90 2024-08-28T19:17:48.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.841+0000 7f3c7faa2640 1 -- 172.21.15.115:0/1042043644 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi028=b;smithi028=e;smithi028=h;smithi028=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l", "target": ["mon-mgr", ""]}) -- 0x7f3c780635a0 con 0x7f3c640b4b80 2024-08-28T19:17:48.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.850+0000 7f3c6effd640 1 -- 172.21.15.115:0/1042043644 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f3c780635a0 con 0x7f3c640b4b80 2024-08-28T19:17:48.851 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled mds.all update... 2024-08-28T19:17:48.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 -- 172.21.15.115:0/1042043644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c640b4b80 msgr2=0x7f3c640b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:48.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c640b4b80 0x7f3c640b7090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3c740058c0 tx=0x7f3c7400a4c0 comp rx=0 tx=0).stop 2024-08-28T19:17:48.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 -- 172.21.15.115:0/1042043644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c78109b90 msgr2=0x7f3c7819fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:48.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c78109b90 0x7f3c7819fa40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000eb10 tx=0x7f3c7000c2d0 comp rx=0 tx=0).stop 2024-08-28T19:17:48.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 -- 172.21.15.115:0/1042043644 shutdown_connections 2024-08-28T19:17:48.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c78109b90 0x7f3c7819fa40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:48.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c640b4b80 0x7f3c640b7090 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:48.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78109240 0x7f3c7819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:48.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 --2- 172.21.15.115:0/1042043644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c78103900 0x7f3c7819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:48.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 -- 172.21.15.115:0/1042043644 >> 172.21.15.115:0/1042043644 conn(0x7f3c7807b5f0 msgr2=0x7f3c78105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:48.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 -- 172.21.15.115:0/1042043644 shutdown_connections 2024-08-28T19:17:48.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-08-28T19:17:48.852+0000 7f3c7faa2640 1 -- 172.21.15.115:0/1042043644 wait complete. 2024-08-28T19:17:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:48 smithi115 ceph-mon[22184]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:49.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:48 smithi016 ceph-mon[18556]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:49.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:49.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:49.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:49.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:49.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:49.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:49.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:49.254 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:48 smithi028 ceph-mon[21769]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:49.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:49.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:17:49.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:49.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:17:49.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:49.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:49.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:49.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:49.713 DEBUG:teuthology.orchestra.run.smithi016:mds.a> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.a.service 2024-08-28T19:17:49.715 DEBUG:teuthology.orchestra.run.smithi016:mds.d> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.d.service 2024-08-28T19:17:49.759 DEBUG:teuthology.orchestra.run.smithi016:mds.g> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.g.service 2024-08-28T19:17:49.802 DEBUG:teuthology.orchestra.run.smithi016:mds.j> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.j.service 2024-08-28T19:17:49.845 DEBUG:teuthology.orchestra.run.smithi028:mds.b> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.b.service 2024-08-28T19:17:49.848 DEBUG:teuthology.orchestra.run.smithi028:mds.e> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.e.service 2024-08-28T19:17:49.890 DEBUG:teuthology.orchestra.run.smithi028:mds.h> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.h.service 2024-08-28T19:17:49.934 DEBUG:teuthology.orchestra.run.smithi028:mds.k> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.k.service 2024-08-28T19:17:49.976 DEBUG:teuthology.orchestra.run.smithi115:mds.c> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.c.service 2024-08-28T19:17:49.979 DEBUG:teuthology.orchestra.run.smithi115:mds.f> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.f.service 2024-08-28T19:17:50.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:49 smithi016 ceph-mon[18556]: from='client.24571 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi028=b;smithi028=e;smithi028=h;smithi028=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:17:50.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:49 smithi016 ceph-mon[18556]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi028=b;smithi028=e;smithi028=h;smithi028=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l;count:12 2024-08-28T19:17:50.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:49 smithi016 ceph-mon[18556]: Deploying daemon mds.l on smithi115 2024-08-28T19:17:50.023 DEBUG:teuthology.orchestra.run.smithi115:mds.i> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.i.service 2024-08-28T19:17:50.066 DEBUG:teuthology.orchestra.run.smithi115:mds.l> sudo journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.l.service 2024-08-28T19:17:50.109 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-08-28T19:17:50.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:17:50.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:49 smithi115 ceph-mon[22184]: from='client.24571 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi028=b;smithi028=e;smithi028=h;smithi028=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:17:50.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:49 smithi115 ceph-mon[22184]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi028=b;smithi028=e;smithi028=h;smithi028=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l;count:12 2024-08-28T19:17:50.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:49 smithi115 ceph-mon[22184]: Deploying daemon mds.l on smithi115 2024-08-28T19:17:50.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:49 smithi028 ceph-mon[21769]: from='client.24571 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi028=b;smithi028=e;smithi028=h;smithi028=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:17:50.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:49 smithi028 ceph-mon[21769]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi028=b;smithi028=e;smithi028=h;smithi028=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l;count:12 2024-08-28T19:17:50.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:49 smithi028 ceph-mon[21769]: Deploying daemon mds.l on smithi115 2024-08-28T19:17:50.367 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:17:50.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.825+0000 7fa8702b7640 1 -- 172.21.15.16:0/2858247301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa868102800 msgr2=0x7fa868102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:50.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.825+0000 7fa8702b7640 1 --2- 172.21.15.16:0/2858247301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa868102800 0x7fa868102c10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa864009900 tx=0x7fa86402a410 comp rx=0 tx=0).stop 2024-08-28T19:17:50.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.825+0000 7fa8702b7640 1 -- 172.21.15.16:0/2858247301 shutdown_connections 2024-08-28T19:17:50.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.825+0000 7fa8702b7640 1 --2- 172.21.15.16:0/2858247301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa868108a90 0x7fa86810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:50.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.825+0000 7fa8702b7640 1 --2- 172.21.15.16:0/2858247301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa868108140 0x7fa868108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.825+0000 7fa8702b7640 1 --2- 172.21.15.16:0/2858247301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa868102800 0x7fa868102c10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.825+0000 7fa8702b7640 1 -- 172.21.15.16:0/2858247301 >> 172.21.15.16:0/2858247301 conn(0x7fa8680794d0 msgr2=0x7fa8680798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.825+0000 7fa8702b7640 1 -- 172.21.15.16:0/2858247301 shutdown_connections 2024-08-28T19:17:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.825+0000 7fa8702b7640 1 -- 172.21.15.16:0/2858247301 wait complete. 2024-08-28T19:17:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa8702b7640 1 Processor -- start 2024-08-28T19:17:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa8702b7640 1 -- start start 2024-08-28T19:17:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa8702b7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa868102800 0x7fa86819b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa8702b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa868108140 0x7fa86819b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa8702b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa868108a90 0x7fa86819bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa8702b7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa868112710 con 0x7fa868108a90 2024-08-28T19:17:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa8702b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa868112590 con 0x7fa868108140 2024-08-28T19:17:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa8702b7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa868112890 con 0x7fa868102800 2024-08-28T19:17:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa86e854640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa868108a90 0x7fa86819bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa86e854640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa868108a90 0x7fa86819bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35630/0 (socket says 172.21.15.16:35630) 2024-08-28T19:17:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa86e854640 1 -- 172.21.15.16:0/1428480864 learned_addr learned my addr 172.21.15.16:0/1428480864 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:17:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa86e053640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa868102800 0x7fa86819b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa86d852640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa868108140 0x7fa86819b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa86e854640 1 -- 172.21.15.16:0/1428480864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa868102800 msgr2=0x7fa86819b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa86e854640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa868102800 0x7fa86819b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa86e854640 1 -- 172.21.15.16:0/1428480864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa868108140 msgr2=0x7fa86819b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa86e854640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa868108140 0x7fa86819b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa86e854640 1 -- 172.21.15.16:0/1428480864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8681a6370 con 0x7fa868108a90 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.826+0000 7fa86e053640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa868102800 0x7fa86819b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.827+0000 7fa86e854640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa868108a90 0x7fa86819bb60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa860009fd0 tx=0x7fa86000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.827+0000 7fa85b7fe640 1 -- 172.21.15.16:0/1428480864 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa860017070 con 0x7fa868108a90 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.827+0000 7fa85b7fe640 1 -- 172.21.15.16:0/1428480864 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa860003e20 con 0x7fa868108a90 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.827+0000 7fa8702b7640 1 -- 172.21.15.16:0/1428480864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8681a6660 con 0x7fa868108a90 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.827+0000 7fa85b7fe640 1 -- 172.21.15.16:0/1428480864 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa860010460 con 0x7fa868108a90 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.827+0000 7fa8702b7640 1 -- 172.21.15.16:0/1428480864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8681a6b50 con 0x7fa868108a90 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.828+0000 7fa8702b7640 1 -- 172.21.15.16:0/1428480864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa868102c10 con 0x7fa868108a90 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.830+0000 7fa85b7fe640 1 -- 172.21.15.16:0/1428480864 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fa860003970 con 0x7fa868108a90 2024-08-28T19:17:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.831+0000 7fa85b7fe640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8540b4bb0 0x7fa8540b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:50.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.831+0000 7fa86e053640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8540b4bb0 0x7fa8540b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:50.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.831+0000 7fa85b7fe640 1 -- 172.21.15.16:0/1428480864 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7fa8600e4b70 con 0x7fa868108a90 2024-08-28T19:17:50.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.831+0000 7fa86e053640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8540b4bb0 0x7fa8540b70c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa864000c00 tx=0x7fa864009680 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:50.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:50.834+0000 7fa85b7fe640 1 -- 172.21.15.16:0/1428480864 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fa8600ac070 con 0x7fa868108a90 2024-08-28T19:17:51.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:50 smithi016 ceph-mon[18556]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.055+0000 7fa8702b7640 1 -- 172.21.15.16:0/1428480864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa86802e430 con 0x7fa868108a90 2024-08-28T19:17:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.056+0000 7fa85b7fe640 1 -- 172.21.15.16:0/1428480864 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7fa8600ab970 con 0x7fa868108a90 2024-08-28T19:17:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:17:51.058 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":1,"btime":"2024-08-28T19:10:20:239202+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-08-28T19:17:51.059 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 1 2024-08-28T19:17:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 -- 172.21.15.16:0/1428480864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8540b4bb0 msgr2=0x7fa8540b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8540b4bb0 0x7fa8540b70c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa864000c00 tx=0x7fa864009680 comp rx=0 tx=0).stop 2024-08-28T19:17:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 -- 172.21.15.16:0/1428480864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa868108a90 msgr2=0x7fa86819bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa868108a90 0x7fa86819bb60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa860009fd0 tx=0x7fa86000b260 comp rx=0 tx=0).stop 2024-08-28T19:17:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 -- 172.21.15.16:0/1428480864 shutdown_connections 2024-08-28T19:17:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa868108a90 0x7fa86819bb60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8540b4bb0 0x7fa8540b70c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa868108140 0x7fa86819b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 --2- 172.21.15.16:0/1428480864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa868102800 0x7fa86819b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 -- 172.21.15.16:0/1428480864 >> 172.21.15.16:0/1428480864 conn(0x7fa8680794d0 msgr2=0x7fa868101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 -- 172.21.15.16:0/1428480864 shutdown_connections 2024-08-28T19:17:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:51.058+0000 7fa8702b7640 1 -- 172.21.15.16:0/1428480864 wait complete. 2024-08-28T19:17:51.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:50 smithi028 ceph-mon[21769]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:51.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:50 smithi115 ceph-mon[22184]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:51.765 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:17:51 smithi115 systemd[1]: Starting Ceph mds.l for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:17:52.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:51 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1428480864' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:17:52.032 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:17:51 smithi115 podman[53040]: 2024-08-28 19:17:51.666600637 +0000 UTC m=+0.015707719 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:17:52.032 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:17:51 smithi115 podman[53040]: 2024-08-28 19:17:51.787125929 +0000 UTC m=+0.136233069 container create b1f104dd337314daa8f87a29d0512d4b9ed97f686e3bd2ed5a4a3b37f7be6d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611) 2024-08-28T19:17:52.032 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:17:52 smithi115 podman[53040]: 2024-08-28 19:17:52.031724259 +0000 UTC m=+0.380831340 container init b1f104dd337314daa8f87a29d0512d4b9ed97f686e3bd2ed5a4a3b37f7be6d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-08-28T19:17:52.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:51 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1428480864' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:17:52.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:51 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1428480864' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:17:52.371 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:17:52 smithi115 podman[53040]: 2024-08-28 19:17:52.034601572 +0000 UTC m=+0.383708653 container start b1f104dd337314daa8f87a29d0512d4b9ed97f686e3bd2ed5a4a3b37f7be6d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-28T19:17:52.371 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:17:52 smithi115 ceph-mds[53059]: -- 172.21.15.115:0/2650310688 <== mon.1 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55844021bc20 con 0x5584443c1180 2024-08-28T19:17:52.371 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:17:52 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l[53055]: starting mds.l at 2024-08-28T19:17:52.371 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:17:52 smithi115 bash[53040]: b1f104dd337314daa8f87a29d0512d4b9ed97f686e3bd2ed5a4a3b37f7be6d1d 2024-08-28T19:17:52.371 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:17:52 smithi115 systemd[1]: Started Ceph mds.l for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:17:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:53 smithi028 ceph-mon[21769]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:53 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.115:6832/2581346583,v1:172.21.15.115:6833/2581346583] up:boot 2024-08-28T19:17:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:53 smithi028 ceph-mon[21769]: fsmap 1 up:standby 2024-08-28T19:17:53.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-08-28T19:17:53.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:53.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:53.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:53.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:53.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:53.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:53.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:53 smithi028 ceph-mon[21769]: Deploying daemon mds.k on smithi028 2024-08-28T19:17:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:53 smithi115 ceph-mon[22184]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:53 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.115:6832/2581346583,v1:172.21.15.115:6833/2581346583] up:boot 2024-08-28T19:17:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:53 smithi115 ceph-mon[22184]: fsmap 1 up:standby 2024-08-28T19:17:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-08-28T19:17:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:53.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:53 smithi115 ceph-mon[22184]: Deploying daemon mds.k on smithi028 2024-08-28T19:17:53.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:53 smithi016 ceph-mon[18556]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:53 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.115:6832/2581346583,v1:172.21.15.115:6833/2581346583] up:boot 2024-08-28T19:17:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:53 smithi016 ceph-mon[18556]: fsmap 1 up:standby 2024-08-28T19:17:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-08-28T19:17:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:53.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:53 smithi016 ceph-mon[18556]: Deploying daemon mds.k on smithi028 2024-08-28T19:17:53.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:17:54.097 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:17:54.478 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:17:54 smithi028 systemd[1]: Starting Ceph mds.k for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:17:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.533+0000 7fecadec7640 1 -- 172.21.15.16:0/3263852360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca8109b90 msgr2=0x7feca81103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.533+0000 7fecadec7640 1 --2- 172.21.15.16:0/3263852360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca8109b90 0x7feca81103c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feca0009ca0 tx=0x7feca002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:17:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.533+0000 7fecadec7640 1 -- 172.21.15.16:0/3263852360 shutdown_connections 2024-08-28T19:17:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.533+0000 7fecadec7640 1 --2- 172.21.15.16:0/3263852360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca8109b90 0x7feca81103c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.533+0000 7fecadec7640 1 --2- 172.21.15.16:0/3263852360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca8109240 0x7feca8109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.533+0000 7fecadec7640 1 --2- 172.21.15.16:0/3263852360 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feca8103900 0x7feca8103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.533+0000 7fecadec7640 1 -- 172.21.15.16:0/3263852360 >> 172.21.15.16:0/3263852360 conn(0x7feca807b5f0 msgr2=0x7feca807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.533+0000 7fecadec7640 1 -- 172.21.15.16:0/3263852360 shutdown_connections 2024-08-28T19:17:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.533+0000 7fecadec7640 1 -- 172.21.15.16:0/3263852360 wait complete. 2024-08-28T19:17:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.534+0000 7fecadec7640 1 Processor -- start 2024-08-28T19:17:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.534+0000 7fecadec7640 1 -- start start 2024-08-28T19:17:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.534+0000 7fecadec7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca8103900 0x7feca819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.534+0000 7fecadec7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca8109240 0x7feca819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.534+0000 7fecadec7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feca8109b90 0x7feca819fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.534+0000 7fecadec7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feca8113850 con 0x7feca8109240 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.534+0000 7fecadec7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feca81136d0 con 0x7feca8103900 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.534+0000 7fecadec7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7feca81139d0 con 0x7feca8109b90 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.534+0000 7fec9effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca8109240 0x7feca819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.534+0000 7fec9effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca8109240 0x7feca819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32822/0 (socket says 172.21.15.16:32822) 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.534+0000 7fec9effd640 1 -- 172.21.15.16:0/1162191626 learned_addr learned my addr 172.21.15.16:0/1162191626 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7feca77fe640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca8103900 0x7feca819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7feca7fff640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feca8109b90 0x7feca819fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7fec9effd640 1 -- 172.21.15.16:0/1162191626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feca8109b90 msgr2=0x7feca819fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7fec9effd640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feca8109b90 0x7feca819fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7fec9effd640 1 -- 172.21.15.16:0/1162191626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca8103900 msgr2=0x7feca819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7fec9effd640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca8103900 0x7feca819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7fec9effd640 1 -- 172.21.15.16:0/1162191626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca81a0130 con 0x7feca8109240 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7feca7fff640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feca8109b90 0x7feca819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7fec9effd640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca8109240 0x7feca819a760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec9000e9f0 tx=0x7fec9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7feca57fa640 1 -- 172.21.15.16:0/1162191626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9000ccc0 con 0x7feca8109240 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7feca57fa640 1 -- 172.21.15.16:0/1162191626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec900108e0 con 0x7feca8109240 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7fecadec7640 1 -- 172.21.15.16:0/1162191626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca81a0420 con 0x7feca8109240 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7feca57fa640 1 -- 172.21.15.16:0/1162191626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec90018860 con 0x7feca8109240 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.535+0000 7fecadec7640 1 -- 172.21.15.16:0/1162191626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca81a6b80 con 0x7feca8109240 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.536+0000 7fecadec7640 1 -- 172.21.15.16:0/1162191626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feca8103d10 con 0x7feca8109240 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.538+0000 7feca57fa640 1 -- 172.21.15.16:0/1162191626 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fec90010430 con 0x7feca8109240 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.538+0000 7feca57fa640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec940b4c00 0x7fec940b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.539+0000 7feca77fe640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec940b4c00 0x7fec940b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.539+0000 7feca57fa640 1 -- 172.21.15.16:0/1162191626 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7fec900e5640 con 0x7feca8109240 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.539+0000 7feca77fe640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec940b4c00 0x7fec940b7110 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fec980058c0 tx=0x7fec9800a250 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.542+0000 7feca57fa640 1 -- 172.21.15.16:0/1162191626 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fec900acaa0 con 0x7feca8109240 2024-08-28T19:17:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.766+0000 7fecadec7640 1 -- 172.21.15.16:0/1162191626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feca819b2f0 con 0x7feca8109240 2024-08-28T19:17:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.767+0000 7feca57fa640 1 -- 172.21.15.16:0/1162191626 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7feca819b2f0 con 0x7feca8109240 2024-08-28T19:17:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:17:54.770 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":2,"btime":"2024-08-28T19:17:52:073174+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-08-28T19:17:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 2 2024-08-28T19:17:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.769+0000 7fecadec7640 1 -- 172.21.15.16:0/1162191626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec940b4c00 msgr2=0x7fec940b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.769+0000 7fecadec7640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec940b4c00 0x7fec940b7110 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fec980058c0 tx=0x7fec9800a250 comp rx=0 tx=0).stop 2024-08-28T19:17:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.769+0000 7fecadec7640 1 -- 172.21.15.16:0/1162191626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca8109240 msgr2=0x7feca819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.769+0000 7fecadec7640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca8109240 0x7feca819a760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec9000e9f0 tx=0x7fec9000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:17:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.770+0000 7fecadec7640 1 -- 172.21.15.16:0/1162191626 shutdown_connections 2024-08-28T19:17:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.770+0000 7fecadec7640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feca8109b90 0x7feca819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.770+0000 7fecadec7640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca8109240 0x7feca819a760 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.770+0000 7fecadec7640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec940b4c00 0x7fec940b7110 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.770+0000 7fecadec7640 1 --2- 172.21.15.16:0/1162191626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca8103900 0x7feca819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.770+0000 7fecadec7640 1 -- 172.21.15.16:0/1162191626 >> 172.21.15.16:0/1162191626 conn(0x7feca807b5f0 msgr2=0x7feca8107290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.770+0000 7fecadec7640 1 -- 172.21.15.16:0/1162191626 shutdown_connections 2024-08-28T19:17:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:54.770+0000 7fecadec7640 1 -- 172.21.15.16:0/1162191626 wait complete. 2024-08-28T19:17:54.795 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:17:54 smithi028 podman[51992]: 2024-08-28 19:17:54.3797859 +0000 UTC m=+0.015868222 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:17:54.795 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:17:54 smithi028 podman[51992]: 2024-08-28 19:17:54.533586809 +0000 UTC m=+0.169669118 container create 6f3c59426cde30f5529da075a4cfcaed596f8239509b83f6281da4855bb8095c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k, io.buildah.version=1.37.1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-08-28T19:17:54.795 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:17:54 smithi028 podman[51992]: 2024-08-28 19:17:54.794818662 +0000 UTC m=+0.430900976 container init 6f3c59426cde30f5529da075a4cfcaed596f8239509b83f6281da4855bb8095c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-08-28T19:17:55.060 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:17:54 smithi028 podman[51992]: 2024-08-28 19:17:54.798709077 +0000 UTC m=+0.434791382 container start 6f3c59426cde30f5529da075a4cfcaed596f8239509b83f6281da4855bb8095c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-28T19:17:55.060 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:17:54 smithi028 ceph-mds[52011]: -- 172.21.15.28:0/3329479920 <== mon.2 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5586a9619860 con 0x5586ad7bf600 2024-08-28T19:17:55.060 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:17:54 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k[52007]: starting mds.k at 2024-08-28T19:17:55.060 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:17:54 smithi028 bash[51992]: 6f3c59426cde30f5529da075a4cfcaed596f8239509b83f6281da4855bb8095c 2024-08-28T19:17:55.060 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:17:54 smithi028 systemd[1]: Started Ceph mds.k for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:17:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:55 smithi115 ceph-mon[22184]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:55 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1162191626' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:17:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:55 smithi016 ceph-mon[18556]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:55 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1162191626' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:17:55.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:55.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:55.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:55.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:55.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:55.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:55 smithi028 ceph-mon[21769]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:55 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1162191626' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:17:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:55.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:55.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:55.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:56.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:56 smithi016 ceph-mon[18556]: Deploying daemon mds.j on smithi016 2024-08-28T19:17:56.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:56 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.28:6832/798033587,v1:172.21.15.28:6833/798033587] up:boot 2024-08-28T19:17:56.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:56 smithi016 ceph-mon[18556]: fsmap 2 up:standby 2024-08-28T19:17:56.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:56 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-08-28T19:17:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:56 smithi115 ceph-mon[22184]: Deploying daemon mds.j on smithi016 2024-08-28T19:17:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:56 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.28:6832/798033587,v1:172.21.15.28:6833/798033587] up:boot 2024-08-28T19:17:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:56 smithi115 ceph-mon[22184]: fsmap 2 up:standby 2024-08-28T19:17:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:56 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-08-28T19:17:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:56 smithi028 ceph-mon[21769]: Deploying daemon mds.j on smithi016 2024-08-28T19:17:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:56 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.28:6832/798033587,v1:172.21.15.28:6833/798033587] up:boot 2024-08-28T19:17:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:56 smithi028 ceph-mon[21769]: fsmap 2 up:standby 2024-08-28T19:17:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:56 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-08-28T19:17:57.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:57 smithi028 ceph-mon[21769]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:57.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:57 smithi115 ceph-mon[22184]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:57.418 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:57 smithi016 ceph-mon[18556]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:57.419 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:17:57 smithi016 systemd[1]: Starting Ceph mds.j for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:17:57.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:17:57.683 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:17:57 smithi016 podman[61996]: 2024-08-28 19:17:57.320162611 +0000 UTC m=+0.016124273 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:17:57.683 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:17:57 smithi016 podman[61996]: 2024-08-28 19:17:57.428361889 +0000 UTC m=+0.124323537 container create 2fdb9fc7cc45bf8fab94aa1bedb1d1b8716e6d121aeb3d123fc136e4a1b28c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-08-28T19:17:57.683 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:17:57 smithi016 podman[61996]: 2024-08-28 19:17:57.681808252 +0000 UTC m=+0.377769923 container init 2fdb9fc7cc45bf8fab94aa1bedb1d1b8716e6d121aeb3d123fc136e4a1b28c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731) 2024-08-28T19:17:57.872 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:17:57.984 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:17:57 smithi016 podman[61996]: 2024-08-28 19:17:57.685732832 +0000 UTC m=+0.381694493 container start 2fdb9fc7cc45bf8fab94aa1bedb1d1b8716e6d121aeb3d123fc136e4a1b28c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.1) 2024-08-28T19:17:57.984 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:17:57 smithi016 ceph-mds[62033]: -- 172.21.15.16:0/2631350918 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559d3fbb3860 con 0x559d43d59600 2024-08-28T19:17:57.984 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:17:57 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-j[62011]: starting mds.j at 2024-08-28T19:17:57.984 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:17:57 smithi016 bash[61996]: 2fdb9fc7cc45bf8fab94aa1bedb1d1b8716e6d121aeb3d123fc136e4a1b28c13 2024-08-28T19:17:57.985 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:17:57 smithi016 systemd[1]: Started Ceph mds.j for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:17:58.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.326+0000 7fc8c7249640 1 -- 172.21.15.16:0/2129413014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c0108170 msgr2=0x7fc8c0108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:58.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.326+0000 7fc8c7249640 1 --2- 172.21.15.16:0/2129413014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c0108170 0x7fc8c0108580 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009900 tx=0x7fc8ac02a410 comp rx=0 tx=0).stop 2024-08-28T19:17:58.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.326+0000 7fc8c7249640 1 -- 172.21.15.16:0/2129413014 shutdown_connections 2024-08-28T19:17:58.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.326+0000 7fc8c7249640 1 --2- 172.21.15.16:0/2129413014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c0108ac0 0x7fc8c010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:58.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.326+0000 7fc8c7249640 1 --2- 172.21.15.16:0/2129413014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c0108170 0x7fc8c0108580 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.326+0000 7fc8c7249640 1 --2- 172.21.15.16:0/2129413014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8c0102880 0x7fc8c0102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.326+0000 7fc8c7249640 1 -- 172.21.15.16:0/2129413014 >> 172.21.15.16:0/2129413014 conn(0x7fc8c0079500 msgr2=0x7fc8c0079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.326+0000 7fc8c7249640 1 -- 172.21.15.16:0/2129413014 shutdown_connections 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.326+0000 7fc8c7249640 1 -- 172.21.15.16:0/2129413014 wait complete. 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c7249640 1 Processor -- start 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c7249640 1 -- start start 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c7249640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c0102880 0x7fc8c019afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c7249640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8c0108170 0x7fc8c019b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c7249640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c0108ac0 0x7fc8c019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c7249640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c0112710 con 0x7fc8c0108ac0 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c7249640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c0112590 con 0x7fc8c0102880 2024-08-28T19:17:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c7249640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c0112890 con 0x7fc8c0108170 2024-08-28T19:17:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c57e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c0108ac0 0x7fc8c019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c57e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c0108ac0 0x7fc8c019ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32874/0 (socket says 172.21.15.16:32874) 2024-08-28T19:17:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c57e6640 1 -- 172.21.15.16:0/2712166277 learned_addr learned my addr 172.21.15.16:0/2712166277 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:17:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8bffff640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8c0108170 0x7fc8c019b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c4fe5640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c0102880 0x7fc8c019afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c57e6640 1 -- 172.21.15.16:0/2712166277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8c0108170 msgr2=0x7fc8c019b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c57e6640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8c0108170 0x7fc8c019b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c57e6640 1 -- 172.21.15.16:0/2712166277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c0102880 msgr2=0x7fc8c019afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c57e6640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c0102880 0x7fc8c019afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.327+0000 7fc8c57e6640 1 -- 172.21.15.16:0/2712166277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c01a62d0 con 0x7fc8c0108ac0 2024-08-28T19:17:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.328+0000 7fc8c57e6640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c0108ac0 0x7fc8c019ba40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800cad0 tx=0x7fc8b800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.328+0000 7fc8bdffb640 1 -- 172.21.15.16:0/2712166277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b800ae20 con 0x7fc8c0108ac0 2024-08-28T19:17:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.328+0000 7fc8bdffb640 1 -- 172.21.15.16:0/2712166277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8b8005020 con 0x7fc8c0108ac0 2024-08-28T19:17:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.328+0000 7fc8c7249640 1 -- 172.21.15.16:0/2712166277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8c01a65c0 con 0x7fc8c0108ac0 2024-08-28T19:17:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.328+0000 7fc8bdffb640 1 -- 172.21.15.16:0/2712166277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b800f6b0 con 0x7fc8c0108ac0 2024-08-28T19:17:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.328+0000 7fc8c7249640 1 -- 172.21.15.16:0/2712166277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8c01a6a80 con 0x7fc8c0108ac0 2024-08-28T19:17:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.329+0000 7fc8c7249640 1 -- 172.21.15.16:0/2712166277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8c0103ab0 con 0x7fc8c0108ac0 2024-08-28T19:17:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.330+0000 7fc8bdffb640 1 -- 172.21.15.16:0/2712166277 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fc8b80051d0 con 0x7fc8c0108ac0 2024-08-28T19:17:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.331+0000 7fc8bdffb640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8b00b4b80 0x7fc8b00b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:17:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.331+0000 7fc8bdffb640 1 -- 172.21.15.16:0/2712166277 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7fc8b80e4d90 con 0x7fc8c0108ac0 2024-08-28T19:17:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.331+0000 7fc8c4fe5640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8b00b4b80 0x7fc8b00b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:17:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.332+0000 7fc8c4fe5640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8b00b4b80 0x7fc8b00b7090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc8c006c4a0 tx=0x7fc8b400b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:17:58.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.335+0000 7fc8bdffb640 1 -- 172.21.15.16:0/2712166277 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fc8b8014030 con 0x7fc8c0108ac0 2024-08-28T19:17:58.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.563+0000 7fc8c7249640 1 -- 172.21.15.16:0/2712166277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8c006bd50 con 0x7fc8c0108ac0 2024-08-28T19:17:58.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.564+0000 7fc8bdffb640 1 -- 172.21.15.16:0/2712166277 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3085 (secure 0 0 0) 0x7fc8c006bd50 con 0x7fc8c0108ac0 2024-08-28T19:17:58.566 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:17:58.566 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":4,"btime":"2024-08-28T19:17:58:090374+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24583,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24590,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-08-28T19:17:58.566 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 4 2024-08-28T19:17:58.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 -- 172.21.15.16:0/2712166277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8b00b4b80 msgr2=0x7fc8b00b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:58.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8b00b4b80 0x7fc8b00b7090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc8c006c4a0 tx=0x7fc8b400b390 comp rx=0 tx=0).stop 2024-08-28T19:17:58.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 -- 172.21.15.16:0/2712166277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c0108ac0 msgr2=0x7fc8c019ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:17:58.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c0108ac0 0x7fc8c019ba40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800cad0 tx=0x7fc8b800cf90 comp rx=0 tx=0).stop 2024-08-28T19:17:58.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 -- 172.21.15.16:0/2712166277 shutdown_connections 2024-08-28T19:17:58.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c0108ac0 0x7fc8c019ba40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:58.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8c0108170 0x7fc8c019b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:58.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8b00b4b80 0x7fc8b00b7090 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:58.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 --2- 172.21.15.16:0/2712166277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c0102880 0x7fc8c019afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:17:58.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 -- 172.21.15.16:0/2712166277 >> 172.21.15.16:0/2712166277 conn(0x7fc8c0079500 msgr2=0x7fc8c01050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:17:58.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 -- 172.21.15.16:0/2712166277 shutdown_connections 2024-08-28T19:17:58.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:17:58.566+0000 7fc8c7249640 1 -- 172.21.15.16:0/2712166277 wait complete. 2024-08-28T19:17:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: Deploying daemon mds.d on smithi016 2024-08-28T19:17:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.16:6834/2536532891,v1:172.21.15.16:6835/2536532891] up:boot 2024-08-28T19:17:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: fsmap 3 up:standby 2024-08-28T19:17:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-08-28T19:17:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:17:58 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2712166277' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:17:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: Deploying daemon mds.d on smithi016 2024-08-28T19:17:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.16:6834/2536532891,v1:172.21.15.16:6835/2536532891] up:boot 2024-08-28T19:17:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: fsmap 3 up:standby 2024-08-28T19:17:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-08-28T19:17:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:17:58 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2712166277' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:17:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:17:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:17:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:17:59.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:17:59.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:17:59.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: Deploying daemon mds.d on smithi016 2024-08-28T19:17:59.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.16:6834/2536532891,v1:172.21.15.16:6835/2536532891] up:boot 2024-08-28T19:17:59.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: fsmap 3 up:standby 2024-08-28T19:17:59.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-08-28T19:17:59.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:17:58 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2712166277' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:00.852 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:18:00 smithi016 systemd[1]: Starting Ceph mds.d for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:18:01.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:00 smithi016 ceph-mon[18556]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:01.115 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:18:00 smithi016 podman[62413]: 2024-08-28 19:18:00.861657079 +0000 UTC m=+0.017586519 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:18:01.115 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:18:00 smithi016 podman[62413]: 2024-08-28 19:18:00.982028819 +0000 UTC m=+0.137958263 container create 50e1601715d47d71f154ca703f86ae4cc9f0ad0585de3f991466c96644d63f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-28T19:18:01.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:00 smithi115 ceph-mon[22184]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:00 smithi028 ceph-mon[21769]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:01.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:18:01.501 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:18:01 smithi016 podman[62413]: 2024-08-28 19:18:01.209859448 +0000 UTC m=+0.365788879 container init 50e1601715d47d71f154ca703f86ae4cc9f0ad0585de3f991466c96644d63f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, io.buildah.version=1.37.1, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-28T19:18:01.501 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:18:01 smithi016 podman[62413]: 2024-08-28 19:18:01.212736145 +0000 UTC m=+0.368665576 container start 50e1601715d47d71f154ca703f86ae4cc9f0ad0585de3f991466c96644d63f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-d, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-28T19:18:01.501 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:18:01 smithi016 ceph-mds[62433]: -- 172.21.15.16:0/1192698023 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564cac67b860 con 0x564cb0821180 2024-08-28T19:18:01.501 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:18:01 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-d[62429]: starting mds.d at 2024-08-28T19:18:01.501 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:18:01 smithi016 bash[62413]: 50e1601715d47d71f154ca703f86ae4cc9f0ad0585de3f991466c96644d63f05 2024-08-28T19:18:01.501 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:18:01 smithi016 systemd[1]: Started Ceph mds.d for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:18:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.088+0000 7f99c4788640 1 -- 172.21.15.16:0/50369842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99bc103940 msgr2=0x7f99bc103d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.088+0000 7f99c4788640 1 --2- 172.21.15.16:0/50369842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99bc103940 0x7f99bc103d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f99b8009900 tx=0x7f99b802a410 comp rx=0 tx=0).stop 2024-08-28T19:18:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.088+0000 7f99c4788640 1 -- 172.21.15.16:0/50369842 shutdown_connections 2024-08-28T19:18:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.088+0000 7f99c4788640 1 --2- 172.21.15.16:0/50369842 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f99bc109bd0 0x7f99bc110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.088+0000 7f99c4788640 1 --2- 172.21.15.16:0/50369842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99bc109280 0x7f99bc109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.088+0000 7f99c4788640 1 --2- 172.21.15.16:0/50369842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99bc103940 0x7f99bc103d50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.088+0000 7f99c4788640 1 -- 172.21.15.16:0/50369842 >> 172.21.15.16:0/50369842 conn(0x7f99bc0ff3d0 msgr2=0x7f99bc101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.088+0000 7f99c4788640 1 -- 172.21.15.16:0/50369842 shutdown_connections 2024-08-28T19:18:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.088+0000 7f99c4788640 1 -- 172.21.15.16:0/50369842 wait complete. 2024-08-28T19:18:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c4788640 1 Processor -- start 2024-08-28T19:18:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c4788640 1 -- start start 2024-08-28T19:18:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c4788640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99bc103940 0x7f99bc19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c4788640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99bc109280 0x7f99bc19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c4788640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f99bc109bd0 0x7f99bc19fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c4788640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc113890 con 0x7f99bc103940 2024-08-28T19:18:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c4788640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc113710 con 0x7f99bc109280 2024-08-28T19:18:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c4788640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc113a10 con 0x7f99bc109bd0 2024-08-28T19:18:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c2524640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99bc103940 0x7f99bc19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c2524640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99bc103940 0x7f99bc19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32912/0 (socket says 172.21.15.16:32912) 2024-08-28T19:18:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c2524640 1 -- 172.21.15.16:0/391083141 learned_addr learned my addr 172.21.15.16:0/391083141 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.089+0000 7f99c1d23640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99bc109280 0x7f99bc19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99c2d25640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f99bc109bd0 0x7f99bc19fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99c2524640 1 -- 172.21.15.16:0/391083141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f99bc109bd0 msgr2=0x7f99bc19fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99c2524640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f99bc109bd0 0x7f99bc19fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99c2524640 1 -- 172.21.15.16:0/391083141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99bc109280 msgr2=0x7f99bc19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99c2524640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99bc109280 0x7f99bc19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99c2524640 1 -- 172.21.15.16:0/391083141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99bc1a01f0 con 0x7f99bc103940 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99c2d25640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f99bc109bd0 0x7f99bc19fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99c2524640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99bc103940 0x7f99bc19a280 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f99b8006010 tx=0x7f99b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99b37fe640 1 -- 172.21.15.16:0/391083141 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b803f030 con 0x7f99bc103940 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99c4788640 1 -- 172.21.15.16:0/391083141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99bc1a0480 con 0x7f99bc103940 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99b37fe640 1 -- 172.21.15.16:0/391083141 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99b8003800 con 0x7f99bc103940 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99b37fe640 1 -- 172.21.15.16:0/391083141 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8035070 con 0x7f99bc103940 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.090+0000 7f99c4788640 1 -- 172.21.15.16:0/391083141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99bc1a6c40 con 0x7f99bc103940 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.091+0000 7f99c4788640 1 -- 172.21.15.16:0/391083141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99bc103d50 con 0x7f99bc103940 2024-08-28T19:18:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.093+0000 7f99b37fe640 1 -- 172.21.15.16:0/391083141 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f99b8003cb0 con 0x7f99bc103940 2024-08-28T19:18:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.093+0000 7f99b37fe640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f99a80b4b80 0x7f99a80b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.093+0000 7f99c1d23640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f99a80b4b80 0x7f99a80b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.093+0000 7f99b37fe640 1 -- 172.21.15.16:0/391083141 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7f99b8105960 con 0x7f99bc103940 2024-08-28T19:18:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.094+0000 7f99c1d23640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f99a80b4b80 0x7f99a80b7090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f99bc108da0 tx=0x7f99ac00a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.097+0000 7f99b37fe640 1 -- 172.21.15.16:0/391083141 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f99b80cce60 con 0x7f99bc103940 2024-08-28T19:18:02.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.322+0000 7f99c4788640 1 -- 172.21.15.16:0/391083141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99bc10ba00 con 0x7f99bc103940 2024-08-28T19:18:02.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.322+0000 7f99b37fe640 1 -- 172.21.15.16:0/391083141 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3910 (secure 0 0 0) 0x7f99b80cc760 con 0x7f99bc103940 2024-08-28T19:18:02.324 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:02.324 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"btime":"2024-08-28T19:18:01:850485+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24583,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24590,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-08-28T19:18:02.325 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 5 2024-08-28T19:18:02.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.324+0000 7f99c4788640 1 -- 172.21.15.16:0/391083141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f99a80b4b80 msgr2=0x7f99a80b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:02.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.324+0000 7f99c4788640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f99a80b4b80 0x7f99a80b7090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f99bc108da0 tx=0x7f99ac00a2d0 comp rx=0 tx=0).stop 2024-08-28T19:18:02.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.324+0000 7f99c4788640 1 -- 172.21.15.16:0/391083141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99bc103940 msgr2=0x7f99bc19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.324+0000 7f99c4788640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99bc103940 0x7f99bc19a280 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f99b8006010 tx=0x7f99b80042f0 comp rx=0 tx=0).stop 2024-08-28T19:18:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.325+0000 7f99c4788640 1 -- 172.21.15.16:0/391083141 shutdown_connections 2024-08-28T19:18:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.325+0000 7f99c4788640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f99bc109bd0 0x7f99bc19fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.325+0000 7f99c4788640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f99a80b4b80 0x7f99a80b7090 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.325+0000 7f99c4788640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99bc109280 0x7f99bc19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.325+0000 7f99c4788640 1 --2- 172.21.15.16:0/391083141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99bc103940 0x7f99bc19a280 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.325+0000 7f99c4788640 1 -- 172.21.15.16:0/391083141 >> 172.21.15.16:0/391083141 conn(0x7f99bc0ff3d0 msgr2=0x7f99bc1017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.325+0000 7f99c4788640 1 -- 172.21.15.16:0/391083141 shutdown_connections 2024-08-28T19:18:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:02.325+0000 7f99c4788640 1 -- 172.21.15.16:0/391083141 wait complete. 2024-08-28T19:18:02.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:02.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:02.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:02.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: Deploying daemon mds.e on smithi028 2024-08-28T19:18:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.16:6836/2448628159,v1:172.21.15.16:6837/2448628159] up:boot 2024-08-28T19:18:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: fsmap 4 up:standby 2024-08-28T19:18:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-08-28T19:18:02.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:02 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/391083141' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:02.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: Deploying daemon mds.e on smithi028 2024-08-28T19:18:02.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:02.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.16:6836/2448628159,v1:172.21.15.16:6837/2448628159] up:boot 2024-08-28T19:18:02.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: fsmap 4 up:standby 2024-08-28T19:18:02.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-08-28T19:18:02.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:02 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/391083141' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:02.636 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:02.636 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:02.636 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:02.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:02.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:02.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:02.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: Deploying daemon mds.e on smithi028 2024-08-28T19:18:02.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:02.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.16:6836/2448628159,v1:172.21.15.16:6837/2448628159] up:boot 2024-08-28T19:18:02.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: fsmap 4 up:standby 2024-08-28T19:18:02.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-08-28T19:18:02.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:02 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/391083141' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:03.809 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:18:03 smithi028 podman[52297]: 2024-08-28 19:18:03.731087452 +0000 UTC m=+0.139489989 container create 145f4b253106d50dc672c485bc736ebc7d660f3ce0c5178e43eb8186edecdf89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-28T19:18:04.309 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:18:03 smithi028 podman[52297]: 2024-08-28 19:18:03.975543644 +0000 UTC m=+0.383946150 container init 145f4b253106d50dc672c485bc736ebc7d660f3ce0c5178e43eb8186edecdf89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-28T19:18:04.309 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:18:03 smithi028 podman[52297]: 2024-08-28 19:18:03.978377658 +0000 UTC m=+0.386780154 container start 145f4b253106d50dc672c485bc736ebc7d660f3ce0c5178e43eb8186edecdf89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, org.label-schema.license=GPLv2) 2024-08-28T19:18:04.310 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:18:04 smithi028 ceph-mds[52316]: -- 172.21.15.28:0/1582882552 <== mon.2 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e1eb9f7860 con 0x55e1efb9d180 2024-08-28T19:18:04.310 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:18:04 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e[52312]: starting mds.e at 2024-08-28T19:18:04.310 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:18:04 smithi028 bash[52297]: 145f4b253106d50dc672c485bc736ebc7d660f3ce0c5178e43eb8186edecdf89 2024-08-28T19:18:04.310 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:18:04 smithi028 systemd[1]: Started Ceph mds.e for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:18:05.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:18:05.364 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:05.484 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:05 smithi016 ceph-mon[18556]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:05.484 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:05.484 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:05.484 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:05.484 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:05.485 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:05.485 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:05.485 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:05 smithi016 ceph-mon[18556]: Deploying daemon mds.f on smithi115 2024-08-28T19:18:05.485 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:05 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.28:6834/2983279759,v1:172.21.15.28:6835/2983279759] up:boot 2024-08-28T19:18:05.485 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:05 smithi016 ceph-mon[18556]: fsmap 5 up:standby 2024-08-28T19:18:05.485 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:05 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-08-28T19:18:05.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:05 smithi028 ceph-mon[21769]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:05.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:05.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:05.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:05.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:05.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:05.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:05.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:05 smithi028 ceph-mon[21769]: Deploying daemon mds.f on smithi115 2024-08-28T19:18:05.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:05 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.28:6834/2983279759,v1:172.21.15.28:6835/2983279759] up:boot 2024-08-28T19:18:05.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:05 smithi028 ceph-mon[21769]: fsmap 5 up:standby 2024-08-28T19:18:05.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:05 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-08-28T19:18:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:05 smithi115 ceph-mon[22184]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:05.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:05 smithi115 ceph-mon[22184]: Deploying daemon mds.f on smithi115 2024-08-28T19:18:05.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:05 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.28:6834/2983279759,v1:172.21.15.28:6835/2983279759] up:boot 2024-08-28T19:18:05.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:05 smithi115 ceph-mon[22184]: fsmap 5 up:standby 2024-08-28T19:18:05.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:05 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-08-28T19:18:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.824+0000 7f130a9ed640 1 -- 172.21.15.16:0/1793476672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1304103900 msgr2=0x7f1304103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.824+0000 7f130a9ed640 1 --2- 172.21.15.16:0/1793476672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1304103900 0x7f1304103d10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f12f4009900 tx=0x7f12f402a410 comp rx=0 tx=0).stop 2024-08-28T19:18:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.824+0000 7f130a9ed640 1 -- 172.21.15.16:0/1793476672 shutdown_connections 2024-08-28T19:18:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.824+0000 7f130a9ed640 1 --2- 172.21.15.16:0/1793476672 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1304109b90 0x7f13041103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.824+0000 7f130a9ed640 1 --2- 172.21.15.16:0/1793476672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304109240 0x7f1304109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.824+0000 7f130a9ed640 1 --2- 172.21.15.16:0/1793476672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1304103900 0x7f1304103d10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.824+0000 7f130a9ed640 1 -- 172.21.15.16:0/1793476672 >> 172.21.15.16:0/1793476672 conn(0x7f130407b5f0 msgr2=0x7f130407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.824+0000 7f130a9ed640 1 -- 172.21.15.16:0/1793476672 shutdown_connections 2024-08-28T19:18:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.824+0000 7f130a9ed640 1 -- 172.21.15.16:0/1793476672 wait complete. 2024-08-28T19:18:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f130a9ed640 1 Processor -- start 2024-08-28T19:18:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f130a9ed640 1 -- start start 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f130a9ed640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1304103900 0x7f130419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f130a9ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304109240 0x7f130419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f130a9ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1304109b90 0x7f130419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f130a9ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1304113850 con 0x7f1304109b90 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f130a9ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13041136d0 con 0x7f1304109240 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f130a9ed640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f13041139d0 con 0x7f1304103900 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f1308f8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1304109b90 0x7f130419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f1308f8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1304109b90 0x7f130419fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56004/0 (socket says 172.21.15.16:56004) 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f1308f8a640 1 -- 172.21.15.16:0/2232916366 learned_addr learned my addr 172.21.15.16:0/2232916366 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f1303fff640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1304103900 0x7f130419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.825+0000 7f13037fe640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304109240 0x7f130419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.826+0000 7f1308f8a640 1 -- 172.21.15.16:0/2232916366 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1304103900 msgr2=0x7f130419a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.826+0000 7f1308f8a640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1304103900 0x7f130419a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.826+0000 7f1308f8a640 1 -- 172.21.15.16:0/2232916366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304109240 msgr2=0x7f130419a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.826+0000 7f1308f8a640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304109240 0x7f130419a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.826+0000 7f1308f8a640 1 -- 172.21.15.16:0/2232916366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13041a0290 con 0x7f1304109b90 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.826+0000 7f1308f8a640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1304109b90 0x7f130419fb60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00eaa0 tx=0x7f12fc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.826+0000 7f13017fa640 1 -- 172.21.15.16:0/2232916366 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc00cdb0 con 0x7f1304109b90 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.826+0000 7f13017fa640 1 -- 172.21.15.16:0/2232916366 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12fc005020 con 0x7f1304109b90 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.826+0000 7f130a9ed640 1 -- 172.21.15.16:0/2232916366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13041a0580 con 0x7f1304109b90 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.826+0000 7f13017fa640 1 -- 172.21.15.16:0/2232916366 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0116d0 con 0x7f1304109b90 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.826+0000 7f130a9ed640 1 -- 172.21.15.16:0/2232916366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13041a6ce0 con 0x7f1304109b90 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.827+0000 7f130a9ed640 1 -- 172.21.15.16:0/2232916366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1304103d10 con 0x7f1304109b90 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.829+0000 7f13017fa640 1 -- 172.21.15.16:0/2232916366 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f12fc0051d0 con 0x7f1304109b90 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.830+0000 7f13017fa640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f12ec0b4b80 0x7f12ec0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:05.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.830+0000 7f1303fff640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f12ec0b4b80 0x7f12ec0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:05.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.830+0000 7f13017fa640 1 -- 172.21.15.16:0/2232916366 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7f12fc0e5310 con 0x7f1304109b90 2024-08-28T19:18:05.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.831+0000 7f1303fff640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f12ec0b4b80 0x7f12ec0b7090 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f12f4031440 tx=0x7f12f4009560 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:05.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:05.833+0000 7f13017fa640 1 -- 172.21.15.16:0/2232916366 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f12fc016030 con 0x7f1304109b90 2024-08-28T19:18:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.057+0000 7f130a9ed640 1 -- 172.21.15.16:0/2232916366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f130419b450 con 0x7f1304109b90 2024-08-28T19:18:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.058+0000 7f13017fa640 1 -- 172.21.15.16:0/2232916366 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4735 (secure 0 0 0) 0x7f130419b450 con 0x7f1304109b90 2024-08-28T19:18:06.059 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:06.059 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":6,"btime":"2024-08-28T19:18:04:497646+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24583,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24590,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24605,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-08-28T19:18:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 6 2024-08-28T19:18:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 -- 172.21.15.16:0/2232916366 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f12ec0b4b80 msgr2=0x7f12ec0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f12ec0b4b80 0x7f12ec0b7090 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f12f4031440 tx=0x7f12f4009560 comp rx=0 tx=0).stop 2024-08-28T19:18:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 -- 172.21.15.16:0/2232916366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1304109b90 msgr2=0x7f130419fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1304109b90 0x7f130419fb60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00eaa0 tx=0x7f12fc00ef60 comp rx=0 tx=0).stop 2024-08-28T19:18:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 -- 172.21.15.16:0/2232916366 shutdown_connections 2024-08-28T19:18:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1304109b90 0x7f130419fb60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f12ec0b4b80 0x7f12ec0b7090 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304109240 0x7f130419a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 --2- 172.21.15.16:0/2232916366 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1304103900 0x7f130419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 -- 172.21.15.16:0/2232916366 >> 172.21.15.16:0/2232916366 conn(0x7f130407b5f0 msgr2=0x7f13041073e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 -- 172.21.15.16:0/2232916366 shutdown_connections 2024-08-28T19:18:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:06.060+0000 7f130a9ed640 1 -- 172.21.15.16:0/2232916366 wait complete. 2024-08-28T19:18:06.150 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:18:06 smithi115 systemd[1]: Starting Ceph mds.f for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:18:06.414 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:06 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2232916366' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:06.414 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:18:06 smithi115 podman[53340]: 2024-08-28 19:18:06.173840869 +0000 UTC m=+0.015576852 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:18:06.414 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:18:06 smithi115 podman[53340]: 2024-08-28 19:18:06.291158731 +0000 UTC m=+0.132894739 container create c6e9556c803c554166000fe860517f2ab58fb2568c7cfa3f1a3a63dd42daff8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f, io.buildah.version=1.37.1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-28T19:18:06.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:06 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2232916366' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:06.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:06 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2232916366' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:06.870 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:18:06 smithi115 podman[53340]: 2024-08-28 19:18:06.535344134 +0000 UTC m=+0.377080108 container init c6e9556c803c554166000fe860517f2ab58fb2568c7cfa3f1a3a63dd42daff8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-08-28T19:18:06.871 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:18:06 smithi115 podman[53340]: 2024-08-28 19:18:06.538136257 +0000 UTC m=+0.379872224 container start c6e9556c803c554166000fe860517f2ab58fb2568c7cfa3f1a3a63dd42daff8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.1, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-08-28T19:18:06.871 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:18:06 smithi115 ceph-mds[53360]: -- 172.21.15.115:0/903828664 <== mon.1 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e68d70d860 con 0x55e6918b3a80 2024-08-28T19:18:06.871 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:18:06 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f[53356]: starting mds.f at 2024-08-28T19:18:06.871 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:18:06 smithi115 bash[53340]: c6e9556c803c554166000fe860517f2ab58fb2568c7cfa3f1a3a63dd42daff8e 2024-08-28T19:18:06.871 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:18:06 smithi115 systemd[1]: Started Ceph mds.f for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:18:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:07 smithi028 ceph-mon[21769]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:07.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:07.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:07.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:07.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:07 smithi115 ceph-mon[22184]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:07 smithi016 ceph-mon[18556]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:07.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:08.290 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:08 smithi016 ceph-mon[18556]: Deploying daemon mds.c on smithi115 2024-08-28T19:18:08.290 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:08 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] up:boot 2024-08-28T19:18:08.290 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:08 smithi016 ceph-mon[18556]: fsmap 6 up:standby 2024-08-28T19:18:08.290 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-08-28T19:18:08.398 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:08 smithi115 ceph-mon[22184]: Deploying daemon mds.c on smithi115 2024-08-28T19:18:08.399 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:08 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] up:boot 2024-08-28T19:18:08.399 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:08 smithi115 ceph-mon[22184]: fsmap 6 up:standby 2024-08-28T19:18:08.399 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-08-28T19:18:08.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:08 smithi028 ceph-mon[21769]: Deploying daemon mds.c on smithi115 2024-08-28T19:18:08.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:08 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] up:boot 2024-08-28T19:18:08.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:08 smithi028 ceph-mon[21769]: fsmap 6 up:standby 2024-08-28T19:18:08.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-08-28T19:18:08.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:18:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:09.061 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:18:09 smithi115 podman[53640]: 2024-08-28 19:18:09.026988105 +0000 UTC m=+0.354122967 container start 81c2473c43d4d655cf77cca177df148de41958ff5f4c896f48296cc61a71334f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-08-28T19:18:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:09 smithi115 ceph-mon[22184]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:09.371 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:18:09 smithi115 ceph-mds[53659]: -- 172.21.15.115:0/2187912819 <== mon.1 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557a621d5860 con 0x557a6637ba80 2024-08-28T19:18:09.371 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:18:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-c[53655]: starting mds.c at 2024-08-28T19:18:09.371 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:18:09 smithi115 bash[53640]: 81c2473c43d4d655cf77cca177df148de41958ff5f4c896f48296cc61a71334f 2024-08-28T19:18:09.371 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:18:09 smithi115 systemd[1]: Started Ceph mds.c for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:18:09.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:09 smithi016 ceph-mon[18556]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.494+0000 7f9303238640 1 -- 172.21.15.16:0/2858443524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92fc103900 msgr2=0x7f92fc103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.494+0000 7f9303238640 1 --2- 172.21.15.16:0/2858443524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92fc103900 0x7f92fc103d10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009900 tx=0x7f92f802a410 comp rx=0 tx=0).stop 2024-08-28T19:18:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.494+0000 7f9303238640 1 -- 172.21.15.16:0/2858443524 shutdown_connections 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.494+0000 7f9303238640 1 --2- 172.21.15.16:0/2858443524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92fc109b90 0x7f92fc1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.494+0000 7f9303238640 1 --2- 172.21.15.16:0/2858443524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92fc109240 0x7f92fc109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.494+0000 7f9303238640 1 --2- 172.21.15.16:0/2858443524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92fc103900 0x7f92fc103d10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.494+0000 7f9303238640 1 -- 172.21.15.16:0/2858443524 >> 172.21.15.16:0/2858443524 conn(0x7f92fc07b5f0 msgr2=0x7f92fc07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.494+0000 7f9303238640 1 -- 172.21.15.16:0/2858443524 shutdown_connections 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.494+0000 7f9303238640 1 -- 172.21.15.16:0/2858443524 wait complete. 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.494+0000 7f9303238640 1 Processor -- start 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.494+0000 7f9303238640 1 -- start start 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9303238640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92fc103900 0x7f92fc19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9303238640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92fc109240 0x7f92fc19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9303238640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92fc109b90 0x7f92fc19fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9303238640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc113850 con 0x7f92fc103900 2024-08-28T19:18:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9303238640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1136d0 con 0x7f92fc109b90 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9303238640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1139d0 con 0x7f92fc109240 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9300fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92fc103900 0x7f92fc19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9300fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92fc103900 0x7f92fc19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56024/0 (socket says 172.21.15.16:56024) 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9300fd4640 1 -- 172.21.15.16:0/1372433054 learned_addr learned my addr 172.21.15.16:0/1372433054 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f93017d5640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92fc109b90 0x7f92fc19fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f92f3fff640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92fc109240 0x7f92fc19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9300fd4640 1 -- 172.21.15.16:0/1372433054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92fc109240 msgr2=0x7f92fc19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9300fd4640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92fc109240 0x7f92fc19a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9300fd4640 1 -- 172.21.15.16:0/1372433054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92fc109b90 msgr2=0x7f92fc19fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9300fd4640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92fc109b90 0x7f92fc19fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.495+0000 7f9300fd4640 1 -- 172.21.15.16:0/1372433054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92fc1a01b0 con 0x7f92fc103900 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.496+0000 7f9300fd4640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92fc103900 0x7f92fc19a240 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f92f8005940 tx=0x7f92f8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.496+0000 7f92f1ffb640 1 -- 172.21.15.16:0/1372433054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f803f030 con 0x7f92fc103900 2024-08-28T19:18:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.496+0000 7f9303238640 1 -- 172.21.15.16:0/1372433054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92fc1a0440 con 0x7f92fc103900 2024-08-28T19:18:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.496+0000 7f92f1ffb640 1 -- 172.21.15.16:0/1372433054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92f8003aa0 con 0x7f92fc103900 2024-08-28T19:18:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.496+0000 7f92f1ffb640 1 -- 172.21.15.16:0/1372433054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f8035070 con 0x7f92fc103900 2024-08-28T19:18:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.496+0000 7f9303238640 1 -- 172.21.15.16:0/1372433054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92fc1a6c00 con 0x7f92fc103900 2024-08-28T19:18:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.497+0000 7f9303238640 1 -- 172.21.15.16:0/1372433054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92fc103d10 con 0x7f92fc103900 2024-08-28T19:18:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.498+0000 7f92f1ffb640 1 -- 172.21.15.16:0/1372433054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f92f8042030 con 0x7f92fc103900 2024-08-28T19:18:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.499+0000 7f92f1ffb640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f92e40b4b80 0x7f92e40b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.499+0000 7f92f3fff640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f92e40b4b80 0x7f92e40b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.499+0000 7f92f1ffb640 1 -- 172.21.15.16:0/1372433054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7f92f8105670 con 0x7f92fc103900 2024-08-28T19:18:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.500+0000 7f92f3fff640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f92e40b4b80 0x7f92e40b7090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f92fc106040 tx=0x7f92ec005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.503+0000 7f92f1ffb640 1 -- 172.21.15.16:0/1372433054 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f92f80cc520 con 0x7f92fc103900 2024-08-28T19:18:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:09 smithi028 ceph-mon[21769]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.725+0000 7f9303238640 1 -- 172.21.15.16:0/1372433054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92fc19b370 con 0x7f92fc103900 2024-08-28T19:18:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.726+0000 7f92f1ffb640 1 -- 172.21.15.16:0/1372433054 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6391 (secure 0 0 0) 0x7f92f80d0ce0 con 0x7f92fc103900 2024-08-28T19:18:09.728 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:09.728 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"btime":"2024-08-28T19:18:09:159892+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24583,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24590,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24605,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24619,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-08-28T19:18:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-08-28T19:18:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.728+0000 7f9303238640 1 -- 172.21.15.16:0/1372433054 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f92e40b4b80 msgr2=0x7f92e40b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.728+0000 7f9303238640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f92e40b4b80 0x7f92e40b7090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f92fc106040 tx=0x7f92ec005f90 comp rx=0 tx=0).stop 2024-08-28T19:18:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.728+0000 7f9303238640 1 -- 172.21.15.16:0/1372433054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92fc103900 msgr2=0x7f92fc19a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.728+0000 7f9303238640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92fc103900 0x7f92fc19a240 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f92f8005940 tx=0x7f92f8004270 comp rx=0 tx=0).stop 2024-08-28T19:18:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.729+0000 7f9303238640 1 -- 172.21.15.16:0/1372433054 shutdown_connections 2024-08-28T19:18:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.729+0000 7f9303238640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f92e40b4b80 0x7f92e40b7090 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.729+0000 7f9303238640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92fc109b90 0x7f92fc19fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.729+0000 7f9303238640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92fc109240 0x7f92fc19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.729+0000 7f9303238640 1 --2- 172.21.15.16:0/1372433054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92fc103900 0x7f92fc19a240 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.729+0000 7f9303238640 1 -- 172.21.15.16:0/1372433054 >> 172.21.15.16:0/1372433054 conn(0x7f92fc07b5f0 msgr2=0x7f92fc1072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:09.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.729+0000 7f9303238640 1 -- 172.21.15.16:0/1372433054 shutdown_connections 2024-08-28T19:18:09.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:09.729+0000 7f9303238640 1 -- 172.21.15.16:0/1372433054 wait complete. 2024-08-28T19:18:10.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:10 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] up:boot 2024-08-28T19:18:10.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:10 smithi028 ceph-mon[21769]: fsmap 7 up:standby 2024-08-28T19:18:10.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-08-28T19:18:10.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:10.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:10.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:10.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:10.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:10.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:10.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:10 smithi028 ceph-mon[21769]: Deploying daemon mds.h on smithi028 2024-08-28T19:18:10.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:10 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1372433054' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:10.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:10 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] up:boot 2024-08-28T19:18:10.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:10 smithi016 ceph-mon[18556]: fsmap 7 up:standby 2024-08-28T19:18:10.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-08-28T19:18:10.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:10.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:10.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:10.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:10.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:10.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:10.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:10 smithi016 ceph-mon[18556]: Deploying daemon mds.h on smithi028 2024-08-28T19:18:10.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:10 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1372433054' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:10 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] up:boot 2024-08-28T19:18:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:10 smithi115 ceph-mon[22184]: fsmap 7 up:standby 2024-08-28T19:18:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-08-28T19:18:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:10 smithi115 ceph-mon[22184]: Deploying daemon mds.h on smithi028 2024-08-28T19:18:10.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:10 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1372433054' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:11 smithi028 ceph-mon[21769]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:11 smithi016 ceph-mon[18556]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:11 smithi115 ceph-mon[22184]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:12.007 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:18:11 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-h[52614]: starting mds.h at 2024-08-28T19:18:12.008 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:18:12 smithi028 bash[52598]: 777be3f361a35ed3daa20cd2745385ce01584d8558e2624806a5e2a086696212 2024-08-28T19:18:12.309 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:18:12 smithi028 systemd[1]: Started Ceph mds.h for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:18:12.586 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:18:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:13.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:13 smithi016 ceph-mon[18556]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:13.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:13 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:13.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:13 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:13.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:13 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:13 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:13 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:13 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:13 smithi016 ceph-mon[18556]: Deploying daemon mds.i on smithi115 2024-08-28T19:18:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:13 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] up:boot 2024-08-28T19:18:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:13 smithi016 ceph-mon[18556]: fsmap 8 up:standby 2024-08-28T19:18:13.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:13 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-08-28T19:18:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:13 smithi028 ceph-mon[21769]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:13 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:13 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:13 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:13 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:13 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:13 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:13.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:13 smithi028 ceph-mon[21769]: Deploying daemon mds.i on smithi115 2024-08-28T19:18:13.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:13 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] up:boot 2024-08-28T19:18:13.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:13 smithi028 ceph-mon[21769]: fsmap 8 up:standby 2024-08-28T19:18:13.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:13 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-08-28T19:18:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.362+0000 7fcb31ecf640 1 -- 172.21.15.16:0/2981722712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb2c105050 msgr2=0x7fcb2c10d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.362+0000 7fcb31ecf640 1 --2- 172.21.15.16:0/2981722712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb2c105050 0x7fcb2c10d690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009900 tx=0x7fcb2002a410 comp rx=0 tx=0).stop 2024-08-28T19:18:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.362+0000 7fcb31ecf640 1 -- 172.21.15.16:0/2981722712 shutdown_connections 2024-08-28T19:18:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.362+0000 7fcb31ecf640 1 --2- 172.21.15.16:0/2981722712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb2c10dbd0 0x7fcb2c1100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.362+0000 7fcb31ecf640 1 --2- 172.21.15.16:0/2981722712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb2c105050 0x7fcb2c10d690 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.362+0000 7fcb31ecf640 1 --2- 172.21.15.16:0/2981722712 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb2c104620 0x7fcb2c104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.362+0000 7fcb31ecf640 1 -- 172.21.15.16:0/2981722712 >> 172.21.15.16:0/2981722712 conn(0x7fcb2c0794f0 msgr2=0x7fcb2c079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.362+0000 7fcb31ecf640 1 -- 172.21.15.16:0/2981722712 shutdown_connections 2024-08-28T19:18:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.362+0000 7fcb31ecf640 1 -- 172.21.15.16:0/2981722712 wait complete. 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.362+0000 7fcb31ecf640 1 Processor -- start 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb31ecf640 1 -- start start 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb31ecf640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb2c104620 0x7fcb2c102530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb31ecf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb2c105050 0x7fcb2c102a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb31ecf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb2c10dbd0 0x7fcb2c102fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb31ecf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2c113850 con 0x7fcb2c105050 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb31ecf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2c1136d0 con 0x7fcb2c10dbd0 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb31ecf640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2c1139d0 con 0x7fcb2c104620 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb2affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb2c105050 0x7fcb2c102a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb2affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb2c105050 0x7fcb2c102a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56052/0 (socket says 172.21.15.16:56052) 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb2affd640 1 -- 172.21.15.16:0/349886162 learned_addr learned my addr 172.21.15.16:0/349886162 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb2bfff640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb2c10dbd0 0x7fcb2c102fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb2b7fe640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb2c104620 0x7fcb2c102530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb2affd640 1 -- 172.21.15.16:0/349886162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb2c104620 msgr2=0x7fcb2c102530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb2affd640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb2c104620 0x7fcb2c102530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb2affd640 1 -- 172.21.15.16:0/349886162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb2c10dbd0 msgr2=0x7fcb2c102fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb2affd640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb2c10dbd0 0x7fcb2c102fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.363+0000 7fcb2affd640 1 -- 172.21.15.16:0/349886162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb2c1a8500 con 0x7fcb2c105050 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.364+0000 7fcb2affd640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb2c105050 0x7fcb2c102a70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcb2002a930 tx=0x7fcb20004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.364+0000 7fcb28ff9640 1 -- 172.21.15.16:0/349886162 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2003f030 con 0x7fcb2c105050 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.364+0000 7fcb31ecf640 1 -- 172.21.15.16:0/349886162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb2c1a8790 con 0x7fcb2c105050 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.364+0000 7fcb28ff9640 1 -- 172.21.15.16:0/349886162 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb20003aa0 con 0x7fcb2c105050 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.364+0000 7fcb28ff9640 1 -- 172.21.15.16:0/349886162 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb20035070 con 0x7fcb2c105050 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.364+0000 7fcb31ecf640 1 -- 172.21.15.16:0/349886162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb2c1a8c70 con 0x7fcb2c105050 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.365+0000 7fcb31ecf640 1 -- 172.21.15.16:0/349886162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb2c104a30 con 0x7fcb2c105050 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.366+0000 7fcb28ff9640 1 -- 172.21.15.16:0/349886162 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fcb20042030 con 0x7fcb2c105050 2024-08-28T19:18:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.367+0000 7fcb28ff9640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcb140b4b80 0x7fcb140b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:13.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.367+0000 7fcb2b7fe640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcb140b4b80 0x7fcb140b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:13.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.367+0000 7fcb28ff9640 1 -- 172.21.15.16:0/349886162 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7fcb20105970 con 0x7fcb2c105050 2024-08-28T19:18:13.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.368+0000 7fcb2b7fe640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcb140b4b80 0x7fcb140b7090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c005fd0 tx=0x7fcb1c005850 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:13.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:13 smithi115 ceph-mon[22184]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:13.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:13 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:13.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:13 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:13.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:13 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:13 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:13 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:13 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:13 smithi115 ceph-mon[22184]: Deploying daemon mds.i on smithi115 2024-08-28T19:18:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:13 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] up:boot 2024-08-28T19:18:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:13 smithi115 ceph-mon[22184]: fsmap 8 up:standby 2024-08-28T19:18:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:13 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-08-28T19:18:13.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.370+0000 7fcb28ff9640 1 -- 172.21.15.16:0/349886162 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fcb200ccdd0 con 0x7fcb2c105050 2024-08-28T19:18:13.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.594+0000 7fcb31ecf640 1 -- 172.21.15.16:0/349886162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb2c1a8fe0 con 0x7fcb2c105050 2024-08-28T19:18:13.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.594+0000 7fcb28ff9640 1 -- 172.21.15.16:0/349886162 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7216 (secure 0 0 0) 0x7fcb200cc6d0 con 0x7fcb2c105050 2024-08-28T19:18:13.597 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:13.597 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"btime":"2024-08-28T19:18:12:204288+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24583,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24590,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24605,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24619,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24635,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-08-28T19:18:13.597 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2024-08-28T19:18:13.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.597+0000 7fcb31ecf640 1 -- 172.21.15.16:0/349886162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcb140b4b80 msgr2=0x7fcb140b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:13.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.597+0000 7fcb31ecf640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcb140b4b80 0x7fcb140b7090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c005fd0 tx=0x7fcb1c005850 comp rx=0 tx=0).stop 2024-08-28T19:18:13.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.597+0000 7fcb31ecf640 1 -- 172.21.15.16:0/349886162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb2c105050 msgr2=0x7fcb2c102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:13.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.597+0000 7fcb31ecf640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb2c105050 0x7fcb2c102a70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcb2002a930 tx=0x7fcb20004270 comp rx=0 tx=0).stop 2024-08-28T19:18:13.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.598+0000 7fcb31ecf640 1 -- 172.21.15.16:0/349886162 shutdown_connections 2024-08-28T19:18:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.598+0000 7fcb31ecf640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcb140b4b80 0x7fcb140b7090 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.598+0000 7fcb31ecf640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb2c10dbd0 0x7fcb2c102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.598+0000 7fcb31ecf640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb2c105050 0x7fcb2c102a70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.598+0000 7fcb31ecf640 1 --2- 172.21.15.16:0/349886162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb2c104620 0x7fcb2c102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.598+0000 7fcb31ecf640 1 -- 172.21.15.16:0/349886162 >> 172.21.15.16:0/349886162 conn(0x7fcb2c0794f0 msgr2=0x7fcb2c107750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.598+0000 7fcb31ecf640 1 -- 172.21.15.16:0/349886162 shutdown_connections 2024-08-28T19:18:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:13.598+0000 7fcb31ecf640 1 -- 172.21.15.16:0/349886162 wait complete. 2024-08-28T19:18:14.063 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:18:14 smithi115 systemd[1]: Starting Ceph mds.i for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:18:14.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:14 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/349886162' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:14 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/349886162' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:14.363 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:18:14 smithi115 podman[53943]: 2024-08-28 19:18:14.111086242 +0000 UTC m=+0.015749666 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:18:14.363 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:18:14 smithi115 podman[53943]: 2024-08-28 19:18:14.22615253 +0000 UTC m=+0.130815940 container create 65c78c0373ffee1c9258395ad882aae0a94f3bbbaa490f85619fc5a4f48b5523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, org.label-schema.vendor=CentOS) 2024-08-28T19:18:14.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:14 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/349886162' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:14.871 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:18:14 smithi115 podman[53943]: 2024-08-28 19:18:14.479264711 +0000 UTC m=+0.383928117 container init 65c78c0373ffee1c9258395ad882aae0a94f3bbbaa490f85619fc5a4f48b5523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-08-28T19:18:14.871 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:18:14 smithi115 podman[53943]: 2024-08-28 19:18:14.482501456 +0000 UTC m=+0.387164863 container start 65c78c0373ffee1c9258395ad882aae0a94f3bbbaa490f85619fc5a4f48b5523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i, GIT_CLEAN=True, io.buildah.version=1.37.1, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-28T19:18:14.871 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:18:14 smithi115 ceph-mds[53962]: -- 172.21.15.115:0/3762793063 <== mon.1 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558ced7d1860 con 0x558cf1977180 2024-08-28T19:18:14.871 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:18:14 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i[53958]: starting mds.i at 2024-08-28T19:18:14.871 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:18:14 smithi115 bash[53943]: 65c78c0373ffee1c9258395ad882aae0a94f3bbbaa490f85619fc5a4f48b5523 2024-08-28T19:18:14.871 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:18:14 smithi115 systemd[1]: Started Ceph mds.i for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:18:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:15 smithi115 ceph-mon[22184]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:15.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:15.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:15.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:15.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:15.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:15.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:15 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:15.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:15 smithi016 ceph-mon[18556]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:15.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:15.506 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:15.506 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:15.506 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:15.506 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:15.506 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:15 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:15 smithi028 ceph-mon[21769]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:15.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:15.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:15.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:15.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:15 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:16.255 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:18:16.312 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:16 smithi016 ceph-mon[18556]: Deploying daemon mds.g on smithi016 2024-08-28T19:18:16.312 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:16 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] up:boot 2024-08-28T19:18:16.312 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:16 smithi016 ceph-mon[18556]: fsmap 9 up:standby 2024-08-28T19:18:16.312 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:16 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-08-28T19:18:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:16 smithi115 ceph-mon[22184]: Deploying daemon mds.g on smithi016 2024-08-28T19:18:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:16 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] up:boot 2024-08-28T19:18:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:16 smithi115 ceph-mon[22184]: fsmap 9 up:standby 2024-08-28T19:18:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:16 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-08-28T19:18:16.557 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:16.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:16 smithi028 ceph-mon[21769]: Deploying daemon mds.g on smithi016 2024-08-28T19:18:16.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:16 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] up:boot 2024-08-28T19:18:16.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:16 smithi028 ceph-mon[21769]: fsmap 9 up:standby 2024-08-28T19:18:16.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:16 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-08-28T19:18:16.680 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:18:16 smithi016 systemd[1]: Starting Ceph mds.g for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:18:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.083+0000 7f24305b2640 1 -- 172.21.15.16:0/3109099994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428108140 msgr2=0x7f2428108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.083+0000 7f24305b2640 1 --2- 172.21.15.16:0/3109099994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428108140 0x7f2428108550 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2418009900 tx=0x7f241802a410 comp rx=0 tx=0).stop 2024-08-28T19:18:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.083+0000 7f24305b2640 1 -- 172.21.15.16:0/3109099994 shutdown_connections 2024-08-28T19:18:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.083+0000 7f24305b2640 1 --2- 172.21.15.16:0/3109099994 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2428108a90 0x7f242810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.083+0000 7f24305b2640 1 --2- 172.21.15.16:0/3109099994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428108140 0x7f2428108550 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.083+0000 7f24305b2640 1 --2- 172.21.15.16:0/3109099994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2428102800 0x7f2428102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.083+0000 7f24305b2640 1 -- 172.21.15.16:0/3109099994 >> 172.21.15.16:0/3109099994 conn(0x7f24280794d0 msgr2=0x7f24280798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.083+0000 7f24305b2640 1 -- 172.21.15.16:0/3109099994 shutdown_connections 2024-08-28T19:18:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.083+0000 7f24305b2640 1 -- 172.21.15.16:0/3109099994 wait complete. 2024-08-28T19:18:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.083+0000 7f24305b2640 1 Processor -- start 2024-08-28T19:18:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f24305b2640 1 -- start start 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f24305b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2428102800 0x7f242819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f24305b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428108140 0x7f242819a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f24305b2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2428108a90 0x7f242819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f24305b2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2428113820 con 0x7f2428108140 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f24305b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24281136a0 con 0x7f2428102800 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f24305b2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f24281139a0 con 0x7f2428108a90 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f242db4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428108140 0x7f242819a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f242db4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428108140 0x7f242819a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36046/0 (socket says 172.21.15.16:36046) 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f242db4d640 1 -- 172.21.15.16:0/936581854 learned_addr learned my addr 172.21.15.16:0/936581854 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f242eb4f640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2428108a90 0x7f242819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f242e34e640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2428102800 0x7f242819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f242db4d640 1 -- 172.21.15.16:0/936581854 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2428108a90 msgr2=0x7f242819faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f242db4d640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2428108a90 0x7f242819faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f242db4d640 1 -- 172.21.15.16:0/936581854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2428102800 msgr2=0x7f242819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f242db4d640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2428102800 0x7f242819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f242db4d640 1 -- 172.21.15.16:0/936581854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24281a0220 con 0x7f2428108140 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.084+0000 7f242eb4f640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2428108a90 0x7f242819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.085+0000 7f242db4d640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428108140 0x7f242819a850 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2418006010 tx=0x7f2418004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.085+0000 7f241f7fe640 1 -- 172.21.15.16:0/936581854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241803f030 con 0x7f2428108140 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.085+0000 7f24305b2640 1 -- 172.21.15.16:0/936581854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24281a04b0 con 0x7f2428108140 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.085+0000 7f241f7fe640 1 -- 172.21.15.16:0/936581854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2418003aa0 con 0x7f2428108140 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.085+0000 7f241f7fe640 1 -- 172.21.15.16:0/936581854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418035070 con 0x7f2428108140 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.085+0000 7f24305b2640 1 -- 172.21.15.16:0/936581854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24281a6c70 con 0x7f2428108140 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.086+0000 7f24305b2640 1 -- 172.21.15.16:0/936581854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2428103a80 con 0x7f2428108140 2024-08-28T19:18:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.087+0000 7f241f7fe640 1 -- 172.21.15.16:0/936581854 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f2418042030 con 0x7f2428108140 2024-08-28T19:18:17.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.088+0000 7f241f7fe640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24140b4b80 0x7f24140b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:17.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.088+0000 7f242e34e640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24140b4b80 0x7f24140b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:17.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.088+0000 7f241f7fe640 1 -- 172.21.15.16:0/936581854 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7f2418105750 con 0x7f2428108140 2024-08-28T19:18:17.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.089+0000 7f242e34e640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24140b4b80 0x7f24140b7090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2424005fd0 tx=0x7f2424005850 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:17.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.091+0000 7f241f7fe640 1 -- 172.21.15.16:0/936581854 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f24180ccc50 con 0x7f2428108140 2024-08-28T19:18:17.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:17 smithi016 ceph-mon[18556]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:17.250 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:18:16 smithi016 podman[63261]: 2024-08-28 19:18:16.859932704 +0000 UTC m=+0.018278078 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:18:17.250 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:18:17 smithi016 podman[63261]: 2024-08-28 19:18:17.192829966 +0000 UTC m=+0.351175330 container create 100d766a4ef23e040a2e73c3432b0f155b497a98c646865ce6a108aa6c8d6252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.1) 2024-08-28T19:18:17.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.316+0000 7f24305b2640 1 -- 172.21.15.16:0/936581854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2428102c10 con 0x7f2428108140 2024-08-28T19:18:17.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.317+0000 7f241f7fe640 1 -- 172.21.15.16:0/936581854 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8046 (secure 0 0 0) 0x7f2428102c10 con 0x7f2428108140 2024-08-28T19:18:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:17.319 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":10,"btime":"2024-08-28T19:18:15:061755+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24583,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24590,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24605,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24619,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24635,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24649,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-08-28T19:18:17.319 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 10 2024-08-28T19:18:17.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 -- 172.21.15.16:0/936581854 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24140b4b80 msgr2=0x7f24140b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:17.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24140b4b80 0x7f24140b7090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2424005fd0 tx=0x7f2424005850 comp rx=0 tx=0).stop 2024-08-28T19:18:17.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 -- 172.21.15.16:0/936581854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428108140 msgr2=0x7f242819a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:17.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428108140 0x7f242819a850 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2418006010 tx=0x7f2418004060 comp rx=0 tx=0).stop 2024-08-28T19:18:17.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 -- 172.21.15.16:0/936581854 shutdown_connections 2024-08-28T19:18:17.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2428108a90 0x7f242819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:17.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428108140 0x7f242819a850 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:17.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24140b4b80 0x7f24140b7090 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:17.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 --2- 172.21.15.16:0/936581854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2428102800 0x7f242819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:17.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 -- 172.21.15.16:0/936581854 >> 172.21.15.16:0/936581854 conn(0x7f24280794d0 msgr2=0x7f2428105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:17.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 -- 172.21.15.16:0/936581854 shutdown_connections 2024-08-28T19:18:17.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:17.320+0000 7f24305b2640 1 -- 172.21.15.16:0/936581854 wait complete. 2024-08-28T19:18:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:17 smithi115 ceph-mon[22184]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:17.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:17 smithi028 ceph-mon[21769]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:17.750 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:18:17 smithi016 podman[63261]: 2024-08-28 19:18:17.462936363 +0000 UTC m=+0.621281724 container init 100d766a4ef23e040a2e73c3432b0f155b497a98c646865ce6a108aa6c8d6252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-08-28T19:18:17.751 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:18:17 smithi016 podman[63261]: 2024-08-28 19:18:17.466208121 +0000 UTC m=+0.624553480 container start 100d766a4ef23e040a2e73c3432b0f155b497a98c646865ce6a108aa6c8d6252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-28T19:18:17.751 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:18:17 smithi016 ceph-mds[63313]: -- 172.21.15.16:0/3263696065 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c8ee0f9c20 con 0x55c8f229f180 2024-08-28T19:18:17.751 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:18:17 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g[63297]: starting mds.g at 2024-08-28T19:18:18.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:18 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/936581854' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:18.460 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:18 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/936581854' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:18.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:18 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/936581854' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:18.501 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:18:18 smithi016 bash[63261]: 100d766a4ef23e040a2e73c3432b0f155b497a98c646865ce6a108aa6c8d6252 2024-08-28T19:18:18.501 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:18:18 smithi016 systemd[1]: Started Ceph mds.g for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:18:19.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:19 smithi016 ceph-mon[18556]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:19.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:19 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.16:6838/3069459721,v1:172.21.15.16:6839/3069459721] up:boot 2024-08-28T19:18:19.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:19 smithi016 ceph-mon[18556]: fsmap 10 up:standby 2024-08-28T19:18:19.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-08-28T19:18:19.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:19.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:19.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:19.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:19.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:19.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:19 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:19.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:19 smithi016 ceph-mon[18556]: Deploying daemon mds.a on smithi016 2024-08-28T19:18:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:19 smithi115 ceph-mon[22184]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:19 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.16:6838/3069459721,v1:172.21.15.16:6839/3069459721] up:boot 2024-08-28T19:18:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:19 smithi115 ceph-mon[22184]: fsmap 10 up:standby 2024-08-28T19:18:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-08-28T19:18:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:19 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:19 smithi115 ceph-mon[22184]: Deploying daemon mds.a on smithi016 2024-08-28T19:18:19.476 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:19 smithi028 ceph-mon[21769]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:19.476 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:19 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.16:6838/3069459721,v1:172.21.15.16:6839/3069459721] up:boot 2024-08-28T19:18:19.476 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:19 smithi028 ceph-mon[21769]: fsmap 10 up:standby 2024-08-28T19:18:19.476 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-08-28T19:18:19.476 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:19.476 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:19.477 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:19.477 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:19.477 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:19.477 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:19 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:19.477 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:19 smithi028 ceph-mon[21769]: Deploying daemon mds.a on smithi016 2024-08-28T19:18:20.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:18:20.250 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:18:20 smithi016 systemd[1]: Starting Ceph mds.a for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:18:20.634 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:18:20 smithi016 podman[63620]: 2024-08-28 19:18:20.274478764 +0000 UTC m=+0.016753568 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:18:20.634 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:18:20 smithi016 podman[63620]: 2024-08-28 19:18:20.39670097 +0000 UTC m=+0.138975801 container create d0926d7b4badab2efed1b27b0d76be84f4c50ca225f8c3567334802dad076301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD) 2024-08-28T19:18:20.634 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:18:20 smithi016 podman[63620]: 2024-08-28 19:18:20.633027646 +0000 UTC m=+0.375302438 container init d0926d7b4badab2efed1b27b0d76be84f4c50ca225f8c3567334802dad076301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a, io.buildah.version=1.37.1, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-08-28T19:18:20.773 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:20.900 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:18:20 smithi016 podman[63620]: 2024-08-28 19:18:20.635926464 +0000 UTC m=+0.378201266 container start d0926d7b4badab2efed1b27b0d76be84f4c50ca225f8c3567334802dad076301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-08-28T19:18:20.900 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:18:20 smithi016 ceph-mds[63665]: -- 172.21.15.16:0/2920555092 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bb6e733860 con 0x55bb728d9600 2024-08-28T19:18:20.900 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:18:20 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a[63654]: starting mds.a at 2024-08-28T19:18:20.900 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:18:20 smithi016 bash[63620]: d0926d7b4badab2efed1b27b0d76be84f4c50ca225f8c3567334802dad076301 2024-08-28T19:18:20.900 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:18:20 smithi016 systemd[1]: Started Ceph mds.a for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:18:21.156 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:21 smithi016 ceph-mon[18556]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:21.156 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:21 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:21.156 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:21 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:21.156 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:21 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:21.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:21 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:21.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:21 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:21.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:21 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:21.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.246+0000 7fa636b06640 1 -- 172.21.15.16:0/2746432233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa630109b90 msgr2=0x7fa6301103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:21.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.246+0000 7fa636b06640 1 --2- 172.21.15.16:0/2746432233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa630109b90 0x7fa6301103c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa628009ca0 tx=0x7fa62802a8e0 comp rx=0 tx=0).stop 2024-08-28T19:18:21.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.246+0000 7fa636b06640 1 -- 172.21.15.16:0/2746432233 shutdown_connections 2024-08-28T19:18:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.246+0000 7fa636b06640 1 --2- 172.21.15.16:0/2746432233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa630109b90 0x7fa6301103c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.246+0000 7fa636b06640 1 --2- 172.21.15.16:0/2746432233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa630109240 0x7fa630109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.246+0000 7fa636b06640 1 --2- 172.21.15.16:0/2746432233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa630103900 0x7fa630103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.246+0000 7fa636b06640 1 -- 172.21.15.16:0/2746432233 >> 172.21.15.16:0/2746432233 conn(0x7fa63007b5f0 msgr2=0x7fa63007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.246+0000 7fa636b06640 1 -- 172.21.15.16:0/2746432233 shutdown_connections 2024-08-28T19:18:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.246+0000 7fa636b06640 1 -- 172.21.15.16:0/2746432233 wait complete. 2024-08-28T19:18:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.247+0000 7fa636b06640 1 Processor -- start 2024-08-28T19:18:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.247+0000 7fa636b06640 1 -- start start 2024-08-28T19:18:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.247+0000 7fa636b06640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa630103900 0x7fa63019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.247+0000 7fa636b06640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa630109240 0x7fa63019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.247+0000 7fa636b06640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa630109b90 0x7fa63019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.247+0000 7fa636b06640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa630113890 con 0x7fa630103900 2024-08-28T19:18:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.247+0000 7fa636b06640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa630113710 con 0x7fa630109b90 2024-08-28T19:18:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.247+0000 7fa636b06640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa630113a10 con 0x7fa630109240 2024-08-28T19:18:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.247+0000 7fa6348a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa630103900 0x7fa63019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.247+0000 7fa6348a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa630103900 0x7fa63019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36098/0 (socket says 172.21.15.16:36098) 2024-08-28T19:18:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.247+0000 7fa6348a2640 1 -- 172.21.15.16:0/3751087745 learned_addr learned my addr 172.21.15.16:0/3751087745 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa6350a3640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa630109b90 0x7fa63019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa6348a2640 1 -- 172.21.15.16:0/3751087745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa630109240 msgr2=0x7fa63019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa62ffff640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa630109240 0x7fa63019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa6348a2640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa630109240 0x7fa63019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa6348a2640 1 -- 172.21.15.16:0/3751087745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa630109b90 msgr2=0x7fa63019faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa6348a2640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa630109b90 0x7fa63019faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa6348a2640 1 -- 172.21.15.16:0/3751087745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6301a01d0 con 0x7fa630103900 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa6350a3640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa630109b90 0x7fa63019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa62ffff640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa630109240 0x7fa63019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa6348a2640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa630103900 0x7fa63019a260 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa62400c890 tx=0x7fa62400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa62dffb640 1 -- 172.21.15.16:0/3751087745 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624004220 con 0x7fa630103900 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa62dffb640 1 -- 172.21.15.16:0/3751087745 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6240043c0 con 0x7fa630103900 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa636b06640 1 -- 172.21.15.16:0/3751087745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6301a04c0 con 0x7fa630103900 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa62dffb640 1 -- 172.21.15.16:0/3751087745 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624004a00 con 0x7fa630103900 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.248+0000 7fa636b06640 1 -- 172.21.15.16:0/3751087745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6301a6c20 con 0x7fa630103900 2024-08-28T19:18:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.249+0000 7fa636b06640 1 -- 172.21.15.16:0/3751087745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa630103d10 con 0x7fa630103900 2024-08-28T19:18:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.251+0000 7fa62dffb640 1 -- 172.21.15.16:0/3751087745 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fa6240079b0 con 0x7fa630103900 2024-08-28T19:18:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.251+0000 7fa62dffb640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6200b4b80 0x7fa6200b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.252+0000 7fa62ffff640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6200b4b80 0x7fa6200b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.252+0000 7fa62dffb640 1 -- 172.21.15.16:0/3751087745 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7fa6240e50a0 con 0x7fa630103900 2024-08-28T19:18:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.252+0000 7fa62ffff640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6200b4b80 0x7fa6200b7090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa61c006fd0 tx=0x7fa61c008040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.255+0000 7fa62dffb640 1 -- 172.21.15.16:0/3751087745 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fa624014030 con 0x7fa630103900 2024-08-28T19:18:21.342 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:21 smithi028 ceph-mon[21769]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:21.342 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:21 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:21.342 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:21 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:21.342 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:21 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:21.342 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:21 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:21.343 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:21 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:21.343 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:21 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:21 smithi115 ceph-mon[22184]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:21 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:21 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:21 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:21 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-28T19:18:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:21 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-28T19:18:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:21 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:21.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.481+0000 7fa636b06640 1 -- 172.21.15.16:0/3751087745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa63019b390 con 0x7fa630103900 2024-08-28T19:18:21.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.482+0000 7fa62dffb640 1 -- 172.21.15.16:0/3751087745 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9698 (secure 0 0 0) 0x7fa6240ac500 con 0x7fa630103900 2024-08-28T19:18:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:21.484 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":12,"btime":"2024-08-28T19:18:21:082953+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14817,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24583,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24590,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24605,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24619,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24635,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24649,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-08-28T19:18:21.484 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 12 2024-08-28T19:18:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 -- 172.21.15.16:0/3751087745 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6200b4b80 msgr2=0x7fa6200b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6200b4b80 0x7fa6200b7090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa61c006fd0 tx=0x7fa61c008040 comp rx=0 tx=0).stop 2024-08-28T19:18:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 -- 172.21.15.16:0/3751087745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa630103900 msgr2=0x7fa63019a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa630103900 0x7fa63019a260 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa62400c890 tx=0x7fa62400cd50 comp rx=0 tx=0).stop 2024-08-28T19:18:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 -- 172.21.15.16:0/3751087745 shutdown_connections 2024-08-28T19:18:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6200b4b80 0x7fa6200b7090 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa630109b90 0x7fa63019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa630109240 0x7fa63019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 --2- 172.21.15.16:0/3751087745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa630103900 0x7fa63019a260 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 -- 172.21.15.16:0/3751087745 >> 172.21.15.16:0/3751087745 conn(0x7fa63007b5f0 msgr2=0x7fa630107290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 -- 172.21.15.16:0/3751087745 shutdown_connections 2024-08-28T19:18:21.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:21.484+0000 7fa636b06640 1 -- 172.21.15.16:0/3751087745 wait complete. 2024-08-28T19:18:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:22 smithi115 ceph-mon[22184]: Deploying daemon mds.b on smithi028 2024-08-28T19:18:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:22 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.16:6840/2698492790,v1:172.21.15.16:6841/2698492790] up:boot 2024-08-28T19:18:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:22 smithi115 ceph-mon[22184]: fsmap 11 up:standby 2024-08-28T19:18:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:22 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-08-28T19:18:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:22 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3751087745' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:22.374 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:22 smithi028 ceph-mon[21769]: Deploying daemon mds.b on smithi028 2024-08-28T19:18:22.375 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:22 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.16:6840/2698492790,v1:172.21.15.16:6841/2698492790] up:boot 2024-08-28T19:18:22.375 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:22 smithi028 ceph-mon[21769]: fsmap 11 up:standby 2024-08-28T19:18:22.375 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:22 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-08-28T19:18:22.375 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:22 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3751087745' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:22.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:22 smithi016 ceph-mon[18556]: Deploying daemon mds.b on smithi028 2024-08-28T19:18:22.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:22 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.16:6840/2698492790,v1:172.21.15.16:6841/2698492790] up:boot 2024-08-28T19:18:22.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:22 smithi016 ceph-mon[18556]: fsmap 11 up:standby 2024-08-28T19:18:22.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:22 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-08-28T19:18:22.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:22 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3751087745' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:23.059 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:18:22 smithi028 podman[52916]: 2024-08-28 19:18:22.968140812 +0000 UTC m=+0.121104010 container create 3427fd35e589418f9cbec98ea6f80c0cb22668af9b4257285b94e11d820126e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-08-28T19:18:23.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:23 smithi115 ceph-mon[22184]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:23.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:23 smithi016 ceph-mon[18556]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:23.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:23 smithi028 ceph-mon[21769]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:23.559 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:18:23 smithi028 podman[52916]: 2024-08-28 19:18:23.220774742 +0000 UTC m=+0.373737902 container init 3427fd35e589418f9cbec98ea6f80c0cb22668af9b4257285b94e11d820126e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b, io.buildah.version=1.37.1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-08-28T19:18:23.560 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:18:23 smithi028 podman[52916]: 2024-08-28 19:18:23.223601921 +0000 UTC m=+0.376565077 container start 3427fd35e589418f9cbec98ea6f80c0cb22668af9b4257285b94e11d820126e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b, io.buildah.version=1.37.1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-08-28T19:18:23.560 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:18:23 smithi028 ceph-mds[52935]: -- 172.21.15.28:0/139613737 <== mon.2 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c5dde07860 con 0x55c5e1fad600 2024-08-28T19:18:23.560 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:18:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b[52931]: starting mds.b at 2024-08-28T19:18:23.560 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:18:23 smithi028 bash[52916]: 3427fd35e589418f9cbec98ea6f80c0cb22668af9b4257285b94e11d820126e5 2024-08-28T19:18:23.560 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:18:23 smithi028 systemd[1]: Started Ceph mds.b for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:18:24.162 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:18:24.607 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:24.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:24 smithi016 ceph-mon[18556]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:24.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:18:24.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:24 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] up:boot 2024-08-28T19:18:24.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:24 smithi016 ceph-mon[18556]: fsmap 12 up:standby 2024-08-28T19:18:24.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:24 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-08-28T19:18:24.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:24 smithi028 ceph-mon[21769]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:24.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:18:24.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:24 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] up:boot 2024-08-28T19:18:24.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:24 smithi028 ceph-mon[21769]: fsmap 12 up:standby 2024-08-28T19:18:24.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:24 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-08-28T19:18:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:24 smithi115 ceph-mon[22184]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:18:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:24 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] up:boot 2024-08-28T19:18:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:24 smithi115 ceph-mon[22184]: fsmap 12 up:standby 2024-08-28T19:18:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:24 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-08-28T19:18:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.177+0000 7f61b792c640 1 -- 172.21.15.16:0/1974647842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61b0102840 msgr2=0x7f61b0102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.177+0000 7f61b792c640 1 --2- 172.21.15.16:0/1974647842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61b0102840 0x7f61b0102c50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009900 tx=0x7f61ac02a410 comp rx=0 tx=0).stop 2024-08-28T19:18:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.177+0000 7f61b792c640 1 -- 172.21.15.16:0/1974647842 shutdown_connections 2024-08-28T19:18:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.177+0000 7f61b792c640 1 --2- 172.21.15.16:0/1974647842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b0108ad0 0x7f61b010f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.177+0000 7f61b792c640 1 --2- 172.21.15.16:0/1974647842 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61b0108180 0x7f61b0108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.177+0000 7f61b792c640 1 --2- 172.21.15.16:0/1974647842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61b0102840 0x7f61b0102c50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.177+0000 7f61b792c640 1 -- 172.21.15.16:0/1974647842 >> 172.21.15.16:0/1974647842 conn(0x7f61b00794f0 msgr2=0x7f61b0079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.177+0000 7f61b792c640 1 -- 172.21.15.16:0/1974647842 shutdown_connections 2024-08-28T19:18:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.177+0000 7f61b792c640 1 -- 172.21.15.16:0/1974647842 wait complete. 2024-08-28T19:18:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b792c640 1 Processor -- start 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b792c640 1 -- start start 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b792c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61b0102840 0x7f61b019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b792c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b0108180 0x7f61b019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b792c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61b0108ad0 0x7f61b019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b792c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61b0112780 con 0x7f61b0102840 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b792c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61b0112600 con 0x7f61b0108180 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b792c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f61b0112900 con 0x7f61b0108ad0 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b56c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61b0102840 0x7f61b019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b56c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61b0102840 0x7f61b019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42992/0 (socket says 172.21.15.16:42992) 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b56c8640 1 -- 172.21.15.16:0/3407841397 learned_addr learned my addr 172.21.15.16:0/3407841397 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b4ec7640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b0108180 0x7f61b019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b5ec9640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61b0108ad0 0x7f61b019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b56c8640 1 -- 172.21.15.16:0/3407841397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61b0108ad0 msgr2=0x7f61b019faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b56c8640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61b0108ad0 0x7f61b019faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b56c8640 1 -- 172.21.15.16:0/3407841397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b0108180 msgr2=0x7f61b019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b56c8640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b0108180 0x7f61b019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b56c8640 1 -- 172.21.15.16:0/3407841397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b01a01d0 con 0x7f61b0102840 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.178+0000 7f61b5ec9640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61b0108ad0 0x7f61b019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.179+0000 7f61b56c8640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61b0102840 0x7f61b019a260 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f61ac005940 tx=0x7f61ac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.179+0000 7f61a67fc640 1 -- 172.21.15.16:0/3407841397 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac03f030 con 0x7f61b0102840 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.179+0000 7f61a67fc640 1 -- 172.21.15.16:0/3407841397 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61ac003cb0 con 0x7f61b0102840 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.179+0000 7f61b792c640 1 -- 172.21.15.16:0/3407841397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b01a0460 con 0x7f61b0102840 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.179+0000 7f61a67fc640 1 -- 172.21.15.16:0/3407841397 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac035070 con 0x7f61b0102840 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.179+0000 7f61b792c640 1 -- 172.21.15.16:0/3407841397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b01a6c20 con 0x7f61b0102840 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.180+0000 7f61b792c640 1 -- 172.21.15.16:0/3407841397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b0102c50 con 0x7f61b0102840 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.181+0000 7f61a67fc640 1 -- 172.21.15.16:0/3407841397 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f61ac042030 con 0x7f61b0102840 2024-08-28T19:18:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.182+0000 7f61a67fc640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f619c0b4b80 0x7f619c0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:25.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.182+0000 7f61b4ec7640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f619c0b4b80 0x7f619c0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:25.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.183+0000 7f61a67fc640 1 -- 172.21.15.16:0/3407841397 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7f61ac1056b0 con 0x7f61b0102840 2024-08-28T19:18:25.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.183+0000 7f61b4ec7640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f619c0b4b80 0x7f619c0b7090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f61b0079d70 tx=0x7f61a000b410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:25.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.187+0000 7f61a67fc640 1 -- 172.21.15.16:0/3407841397 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f61ac0ccbb0 con 0x7f61b0102840 2024-08-28T19:18:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.464+0000 7f61b792c640 1 -- 172.21.15.16:0/3407841397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61b019b390 con 0x7f61b0102840 2024-08-28T19:18:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.465+0000 7f61a67fc640 1 -- 172.21.15.16:0/3407841397 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10524 (secure 0 0 0) 0x7f61ac0d50f0 con 0x7f61b0102840 2024-08-28T19:18:25.466 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:25.467 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"btime":"2024-08-28T19:18:24:097660+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14817,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24583,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24590,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24605,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24619,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24635,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24649,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24665,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-08-28T19:18:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2024-08-28T19:18:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 -- 172.21.15.16:0/3407841397 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f619c0b4b80 msgr2=0x7f619c0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f619c0b4b80 0x7f619c0b7090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f61b0079d70 tx=0x7f61a000b410 comp rx=0 tx=0).stop 2024-08-28T19:18:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 -- 172.21.15.16:0/3407841397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61b0102840 msgr2=0x7f61b019a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61b0102840 0x7f61b019a260 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f61ac005940 tx=0x7f61ac0042f0 comp rx=0 tx=0).stop 2024-08-28T19:18:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 -- 172.21.15.16:0/3407841397 shutdown_connections 2024-08-28T19:18:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61b0108ad0 0x7f61b019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f619c0b4b80 0x7f619c0b7090 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b0108180 0x7f61b019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 --2- 172.21.15.16:0/3407841397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61b0102840 0x7f61b019a260 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 -- 172.21.15.16:0/3407841397 >> 172.21.15.16:0/3407841397 conn(0x7f61b00794f0 msgr2=0x7f61b01018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 -- 172.21.15.16:0/3407841397 shutdown_connections 2024-08-28T19:18:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:25.468+0000 7f61b792c640 1 -- 172.21.15.16:0/3407841397 wait complete. 2024-08-28T19:18:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:25 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3407841397' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:18:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-08-28T19:18:26.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:25 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3407841397' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:26.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:26.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:18:26.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:25 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3407841397' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:18:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:27.149 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:27.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:26 smithi016 ceph-mon[18556]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:27.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:26 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:18:27.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:26 smithi028 ceph-mon[21769]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:27.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:26 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:18:27.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:26 smithi115 ceph-mon[22184]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:27.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:26 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:18:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.617+0000 7f25738c8640 1 -- 172.21.15.16:0/4113935793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f256c102ee0 msgr2=0x7f256c1032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.617+0000 7f25738c8640 1 --2- 172.21.15.16:0/4113935793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f256c102ee0 0x7f256c1032f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2568009900 tx=0x7f256802a410 comp rx=0 tx=0).stop 2024-08-28T19:18:27.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.617+0000 7f25738c8640 1 -- 172.21.15.16:0/4113935793 shutdown_connections 2024-08-28T19:18:27.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.617+0000 7f25738c8640 1 --2- 172.21.15.16:0/4113935793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f256c108aa0 0x7f256c10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:27.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.617+0000 7f25738c8640 1 --2- 172.21.15.16:0/4113935793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256c108150 0x7f256c108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:27.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.617+0000 7f25738c8640 1 --2- 172.21.15.16:0/4113935793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f256c102ee0 0x7f256c1032f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:27.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.617+0000 7f25738c8640 1 -- 172.21.15.16:0/4113935793 >> 172.21.15.16:0/4113935793 conn(0x7f256c077a00 msgr2=0x7f256c077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:27.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.617+0000 7f25738c8640 1 -- 172.21.15.16:0/4113935793 shutdown_connections 2024-08-28T19:18:27.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.617+0000 7f25738c8640 1 -- 172.21.15.16:0/4113935793 wait complete. 2024-08-28T19:18:27.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f25738c8640 1 Processor -- start 2024-08-28T19:18:27.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f25738c8640 1 -- start start 2024-08-28T19:18:27.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f25738c8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f256c102ee0 0x7f256c199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:27.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f25738c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256c108150 0x7f256c1997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f25738c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f256c108aa0 0x7f256c19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f25738c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f256c112770 con 0x7f256c108aa0 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f25738c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f256c1125f0 con 0x7f256c108150 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f25738c8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f256c1128f0 con 0x7f256c102ee0 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f2571e65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f256c108aa0 0x7f256c19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f2571e65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f256c108aa0 0x7f256c19ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43006/0 (socket says 172.21.15.16:43006) 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f2571e65640 1 -- 172.21.15.16:0/627453182 learned_addr learned my addr 172.21.15.16:0/627453182 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.618+0000 7f2570e63640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256c108150 0x7f256c1997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f2571664640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f256c102ee0 0x7f256c199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f2571e65640 1 -- 172.21.15.16:0/627453182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f256c102ee0 msgr2=0x7f256c199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f2571e65640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f256c102ee0 0x7f256c199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f2571e65640 1 -- 172.21.15.16:0/627453182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256c108150 msgr2=0x7f256c1997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:27.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f2571e65640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256c108150 0x7f256c1997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f2571e65640 1 -- 172.21.15.16:0/627453182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256c19f190 con 0x7f256c108aa0 2024-08-28T19:18:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f2571e65640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f256c108aa0 0x7f256c19ea60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f256400eb10 tx=0x7f256400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f25627fc640 1 -- 172.21.15.16:0/627453182 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564004b20 con 0x7f256c108aa0 2024-08-28T19:18:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f25627fc640 1 -- 172.21.15.16:0/627453182 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2564004cc0 con 0x7f256c108aa0 2024-08-28T19:18:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f25738c8640 1 -- 172.21.15.16:0/627453182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256c19f480 con 0x7f256c108aa0 2024-08-28T19:18:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f25627fc640 1 -- 172.21.15.16:0/627453182 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564010670 con 0x7f256c108aa0 2024-08-28T19:18:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.619+0000 7f25738c8640 1 -- 172.21.15.16:0/627453182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f256c1a5be0 con 0x7f256c108aa0 2024-08-28T19:18:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.620+0000 7f25738c8640 1 -- 172.21.15.16:0/627453182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f256c1032f0 con 0x7f256c108aa0 2024-08-28T19:18:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.623+0000 7f25627fc640 1 -- 172.21.15.16:0/627453182 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f2564005000 con 0x7f256c108aa0 2024-08-28T19:18:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.623+0000 7f25627fc640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25580b4c00 0x7f25580b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.624+0000 7f2571664640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25580b4c00 0x7f25580b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.624+0000 7f25627fc640 1 -- 172.21.15.16:0/627453182 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7f25640e54e0 con 0x7f256c108aa0 2024-08-28T19:18:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.624+0000 7f2571664640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25580b4c00 0x7f25580b7110 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2568031440 tx=0x7f2568005d50 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.627+0000 7f25627fc640 1 -- 172.21.15.16:0/627453182 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f2564056130 con 0x7f256c108aa0 2024-08-28T19:18:27.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.836+0000 7f25738c8640 1 -- 172.21.15.16:0/627453182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f256c10a290 con 0x7f256c108aa0 2024-08-28T19:18:27.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.837+0000 7f25627fc640 1 -- 172.21.15.16:0/627453182 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v69) ==== 84+0+65 (secure 0 0 0) 0x7f25640ac9e0 con 0x7f256c108aa0 2024-08-28T19:18:27.838 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:27.838 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-08-28T19:18:27.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:18:27.839 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-08-28T19:18:27.839 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-08-28T19:18:27.839 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:18:27.839 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-08-28T19:18:27.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 -- 172.21.15.16:0/627453182 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25580b4c00 msgr2=0x7f25580b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:27.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25580b4c00 0x7f25580b7110 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2568031440 tx=0x7f2568005d50 comp rx=0 tx=0).stop 2024-08-28T19:18:27.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 -- 172.21.15.16:0/627453182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f256c108aa0 msgr2=0x7f256c19ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:27.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f256c108aa0 0x7f256c19ea60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f256400eb10 tx=0x7f256400c2d0 comp rx=0 tx=0).stop 2024-08-28T19:18:27.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 -- 172.21.15.16:0/627453182 shutdown_connections 2024-08-28T19:18:27.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f256c108aa0 0x7f256c19ea60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:27.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25580b4c00 0x7f25580b7110 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:27.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256c108150 0x7f256c1997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:27.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 --2- 172.21.15.16:0/627453182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f256c102ee0 0x7f256c199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:27.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 -- 172.21.15.16:0/627453182 >> 172.21.15.16:0/627453182 conn(0x7f256c077a00 msgr2=0x7f256c106290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:27.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 -- 172.21.15.16:0/627453182 shutdown_connections 2024-08-28T19:18:27.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:27.840+0000 7f25738c8640 1 -- 172.21.15.16:0/627453182 wait complete. 2024-08-28T19:18:28.510 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-08-28T19:18:28.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-08-28T19:18:28.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:28.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/627453182' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-08-28T19:18:28.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:28.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:18:28.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.760 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:28.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/627453182' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-08-28T19:18:28.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:28.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:18:28.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/627453182' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-08-28T19:18:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:18:28.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:18:28.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:18:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.222+0000 7ffa17ee4640 1 -- 172.21.15.16:0/2362481854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa10104750 msgr2=0x7ffa10104b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.222+0000 7ffa17ee4640 1 --2- 172.21.15.16:0/2362481854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa10104750 0x7ffa10104b60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009900 tx=0x7ffa0c02a410 comp rx=0 tx=0).stop 2024-08-28T19:18:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.222+0000 7ffa17ee4640 1 -- 172.21.15.16:0/2362481854 shutdown_connections 2024-08-28T19:18:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.222+0000 7ffa17ee4640 1 --2- 172.21.15.16:0/2362481854 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa1010dbb0 0x7ffa101100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.222+0000 7ffa17ee4640 1 --2- 172.21.15.16:0/2362481854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa101050a0 0x7ffa1010d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.222+0000 7ffa17ee4640 1 --2- 172.21.15.16:0/2362481854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa10104750 0x7ffa10104b60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.222+0000 7ffa17ee4640 1 -- 172.21.15.16:0/2362481854 >> 172.21.15.16:0/2362481854 conn(0x7ffa100794d0 msgr2=0x7ffa100798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.223+0000 7ffa17ee4640 1 -- 172.21.15.16:0/2362481854 shutdown_connections 2024-08-28T19:18:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.223+0000 7ffa17ee4640 1 -- 172.21.15.16:0/2362481854 wait complete. 2024-08-28T19:18:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.223+0000 7ffa17ee4640 1 Processor -- start 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.223+0000 7ffa17ee4640 1 -- start start 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.223+0000 7ffa17ee4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa10104750 0x7ffa1019a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.223+0000 7ffa17ee4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa101050a0 0x7ffa1019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.223+0000 7ffa17ee4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa1010dbb0 0x7ffa1019fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.223+0000 7ffa17ee4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa101138f0 con 0x7ffa101050a0 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.223+0000 7ffa17ee4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa10113770 con 0x7ffa10104750 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.223+0000 7ffa17ee4640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ffa10113a70 con 0x7ffa1010dbb0 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa1547f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa101050a0 0x7ffa1019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa1547f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa101050a0 0x7ffa1019a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43014/0 (socket says 172.21.15.16:43014) 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa1547f640 1 -- 172.21.15.16:0/4113582725 learned_addr learned my addr 172.21.15.16:0/4113582725 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa15c80640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa10104750 0x7ffa1019a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa16481640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa1010dbb0 0x7ffa1019fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa1547f640 1 -- 172.21.15.16:0/4113582725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa1010dbb0 msgr2=0x7ffa1019fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa1547f640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa1010dbb0 0x7ffa1019fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa1547f640 1 -- 172.21.15.16:0/4113582725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa10104750 msgr2=0x7ffa1019a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa1547f640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa10104750 0x7ffa1019a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa1547f640 1 -- 172.21.15.16:0/4113582725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa101a0330 con 0x7ffa101050a0 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa1547f640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa101050a0 0x7ffa1019a960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ffa00006fd0 tx=0x7ffa0000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa06ffd640 1 -- 172.21.15.16:0/4113582725 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00011830 con 0x7ffa101050a0 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa06ffd640 1 -- 172.21.15.16:0/4113582725 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa00016070 con 0x7ffa101050a0 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa17ee4640 1 -- 172.21.15.16:0/4113582725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa101a0620 con 0x7ffa101050a0 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa06ffd640 1 -- 172.21.15.16:0/4113582725 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00010e00 con 0x7ffa101050a0 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.224+0000 7ffa17ee4640 1 -- 172.21.15.16:0/4113582725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa101a6d80 con 0x7ffa101050a0 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.225+0000 7ffa17ee4640 1 -- 172.21.15.16:0/4113582725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa10104be0 con 0x7ffa101050a0 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.227+0000 7ffa06ffd640 1 -- 172.21.15.16:0/4113582725 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7ffa0001b460 con 0x7ffa101050a0 2024-08-28T19:18:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.228+0000 7ffa06ffd640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9fc0b4b80 0x7ff9fc0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:29.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.229+0000 7ffa15c80640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9fc0b4b80 0x7ff9fc0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:29.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.229+0000 7ffa06ffd640 1 -- 172.21.15.16:0/4113582725 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 7342+0+0 (secure 0 0 0) 0x7ffa000e4990 con 0x7ffa101050a0 2024-08-28T19:18:29.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.229+0000 7ffa15c80640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9fc0b4b80 0x7ff9fc0b7090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c000c00 tx=0x7ffa0c031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:29.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.232+0000 7ffa06ffd640 1 -- 172.21.15.16:0/4113582725 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ffa000abe90 con 0x7ffa101050a0 2024-08-28T19:18:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.418+0000 7ffa17ee4640 1 -- 172.21.15.16:0/4113582725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ffa1002e430 con 0x7ffa101050a0 2024-08-28T19:18:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.447+0000 7ffa06ffd640 1 -- 172.21.15.16:0/4113582725 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v70) ==== 138+0+0 (secure 0 0 0) 0x7ffa000ab790 con 0x7ffa101050a0 2024-08-28T19:18:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-08-28T19:18:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.449+0000 7ffa17ee4640 1 -- 172.21.15.16:0/4113582725 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9fc0b4b80 msgr2=0x7ff9fc0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.449+0000 7ffa17ee4640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9fc0b4b80 0x7ff9fc0b7090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c000c00 tx=0x7ffa0c031400 comp rx=0 tx=0).stop 2024-08-28T19:18:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.449+0000 7ffa17ee4640 1 -- 172.21.15.16:0/4113582725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa101050a0 msgr2=0x7ffa1019a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.449+0000 7ffa17ee4640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa101050a0 0x7ffa1019a960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ffa00006fd0 tx=0x7ffa0000b8e0 comp rx=0 tx=0).stop 2024-08-28T19:18:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.450+0000 7ffa17ee4640 1 -- 172.21.15.16:0/4113582725 shutdown_connections 2024-08-28T19:18:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.450+0000 7ffa17ee4640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa1010dbb0 0x7ffa1019fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.450+0000 7ffa17ee4640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa101050a0 0x7ffa1019a960 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.450+0000 7ffa17ee4640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9fc0b4b80 0x7ff9fc0b7090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.450+0000 7ffa17ee4640 1 --2- 172.21.15.16:0/4113582725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa10104750 0x7ffa1019a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.450+0000 7ffa17ee4640 1 -- 172.21.15.16:0/4113582725 >> 172.21.15.16:0/4113582725 conn(0x7ffa100794d0 msgr2=0x7ffa10100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.450+0000 7ffa17ee4640 1 -- 172.21.15.16:0/4113582725 shutdown_connections 2024-08-28T19:18:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:29.450+0000 7ffa17ee4640 1 -- 172.21.15.16:0/4113582725 wait complete. 2024-08-28T19:18:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:29 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/4113582725' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-08-28T19:18:29.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:29 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/4113582725' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-08-28T19:18:29.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:29 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/4113582725' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-08-28T19:18:30.249 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-08-28T19:18:30.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:30 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/4113582725' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-08-28T19:18:30.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:30 smithi016 ceph-mon[18556]: pgmap v249: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:30.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:30 smithi016 ceph-mon[18556]: osdmap e70: 12 total, 12 up, 12 in 2024-08-28T19:18:30.515 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:30.766 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:30 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/4113582725' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-08-28T19:18:30.767 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:30 smithi028 ceph-mon[21769]: pgmap v249: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:30.767 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:30 smithi028 ceph-mon[21769]: osdmap e70: 12 total, 12 up, 12 in 2024-08-28T19:18:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:30 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/4113582725' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-08-28T19:18:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:30 smithi115 ceph-mon[22184]: pgmap v249: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:30 smithi115 ceph-mon[22184]: osdmap e70: 12 total, 12 up, 12 in 2024-08-28T19:18:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.010+0000 7f549286c640 1 -- 172.21.15.16:0/3625533054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548c10dc00 msgr2=0x7f548c1100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.010+0000 7f549286c640 1 --2- 172.21.15.16:0/3625533054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548c10dc00 0x7f548c1100d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5484009ca0 tx=0x7f548402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.010+0000 7f549286c640 1 -- 172.21.15.16:0/3625533054 shutdown_connections 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.010+0000 7f549286c640 1 --2- 172.21.15.16:0/3625533054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548c10dc00 0x7f548c1100d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.010+0000 7f549286c640 1 --2- 172.21.15.16:0/3625533054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f548c105120 0x7f548c10d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.010+0000 7f549286c640 1 --2- 172.21.15.16:0/3625533054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f548c1047d0 0x7f548c104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.010+0000 7f549286c640 1 -- 172.21.15.16:0/3625533054 >> 172.21.15.16:0/3625533054 conn(0x7f548c079500 msgr2=0x7f548c079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.010+0000 7f549286c640 1 -- 172.21.15.16:0/3625533054 shutdown_connections 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.010+0000 7f549286c640 1 -- 172.21.15.16:0/3625533054 wait complete. 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.010+0000 7f549286c640 1 Processor -- start 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.010+0000 7f549286c640 1 -- start start 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f549286c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f548c1047d0 0x7f548c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f549286c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f548c105120 0x7f548c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f549286c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548c10dc00 0x7f548c19fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f549286c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f548c113840 con 0x7f548c10dc00 2024-08-28T19:18:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f549286c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f548c1136c0 con 0x7f548c105120 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f549286c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f548c1139c0 con 0x7f548c1047d0 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5490e09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548c10dc00 0x7f548c19fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5490e09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548c10dc00 0x7f548c19fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43034/0 (socket says 172.21.15.16:43034) 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5490e09640 1 -- 172.21.15.16:0/133859040 learned_addr learned my addr 172.21.15.16:0/133859040 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f548bfff640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f548c1047d0 0x7f548c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5483fff640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f548c105120 0x7f548c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5490e09640 1 -- 172.21.15.16:0/133859040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f548c1047d0 msgr2=0x7f548c19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5490e09640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f548c1047d0 0x7f548c19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5490e09640 1 -- 172.21.15.16:0/133859040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f548c105120 msgr2=0x7f548c19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5490e09640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f548c105120 0x7f548c19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5490e09640 1 -- 172.21.15.16:0/133859040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f548c1a0080 con 0x7f548c10dc00 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5490e09640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548c10dc00 0x7f548c19fa10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5484005cc0 tx=0x7f548402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5489ffb640 1 -- 172.21.15.16:0/133859040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5484003bf0 con 0x7f548c10dc00 2024-08-28T19:18:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f5489ffb640 1 -- 172.21.15.16:0/133859040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5484003d90 con 0x7f548c10dc00 2024-08-28T19:18:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.011+0000 7f549286c640 1 -- 172.21.15.16:0/133859040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f548c1a0310 con 0x7f548c10dc00 2024-08-28T19:18:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.012+0000 7f5489ffb640 1 -- 172.21.15.16:0/133859040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548402fb20 con 0x7f548c10dc00 2024-08-28T19:18:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.012+0000 7f549286c640 1 -- 172.21.15.16:0/133859040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f548c1a0850 con 0x7f548c10dc00 2024-08-28T19:18:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.013+0000 7f549286c640 1 -- 172.21.15.16:0/133859040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f548c06bf90 con 0x7f548c10dc00 2024-08-28T19:18:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.014+0000 7f5489ffb640 1 -- 172.21.15.16:0/133859040 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f5484004580 con 0x7f548c10dc00 2024-08-28T19:18:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.015+0000 7f5489ffb640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54780b4bb0 0x7f54780b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.015+0000 7f548bfff640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54780b4bb0 0x7f54780b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.015+0000 7f5489ffb640 1 -- 172.21.15.16:0/133859040 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7712+0+0 (secure 0 0 0) 0x7f54841063b0 con 0x7f548c10dc00 2024-08-28T19:18:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.016+0000 7f548bfff640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54780b4bb0 0x7f54780b70c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f548c06c690 tx=0x7f547c00b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.019+0000 7f5489ffb640 1 -- 172.21.15.16:0/133859040 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f54840cd740 con 0x7f548c10dc00 2024-08-28T19:18:31.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.207+0000 7f549286c640 1 -- 172.21.15.16:0/133859040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f548c1a6d00 con 0x7f548c10dc00 2024-08-28T19:18:31.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.459+0000 7f5489ffb640 1 -- 172.21.15.16:0/133859040 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v72) ==== 171+0+0 (secure 0 0 0) 0x7f54840098b0 con 0x7f548c10dc00 2024-08-28T19:18:31.460 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-08-28T19:18:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.461+0000 7f549286c640 1 -- 172.21.15.16:0/133859040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54780b4bb0 msgr2=0x7f54780b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.461+0000 7f549286c640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54780b4bb0 0x7f54780b70c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f548c06c690 tx=0x7f547c00b390 comp rx=0 tx=0).stop 2024-08-28T19:18:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.461+0000 7f549286c640 1 -- 172.21.15.16:0/133859040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548c10dc00 msgr2=0x7f548c19fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.461+0000 7f549286c640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548c10dc00 0x7f548c19fa10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5484005cc0 tx=0x7f548402f380 comp rx=0 tx=0).stop 2024-08-28T19:18:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.461+0000 7f549286c640 1 -- 172.21.15.16:0/133859040 shutdown_connections 2024-08-28T19:18:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.461+0000 7f549286c640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548c10dc00 0x7f548c19fa10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.461+0000 7f549286c640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54780b4bb0 0x7f54780b70c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.461+0000 7f549286c640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f548c105120 0x7f548c19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.461+0000 7f549286c640 1 --2- 172.21.15.16:0/133859040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f548c1047d0 0x7f548c19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.461+0000 7f549286c640 1 -- 172.21.15.16:0/133859040 >> 172.21.15.16:0/133859040 conn(0x7f548c079500 msgr2=0x7f548c1003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.462+0000 7f549286c640 1 -- 172.21.15.16:0/133859040 shutdown_connections 2024-08-28T19:18:31.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:31.462+0000 7f549286c640 1 -- 172.21.15.16:0/133859040 wait complete. 2024-08-28T19:18:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:31 smithi016 ceph-mon[18556]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-28T19:18:31.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:31 smithi016 ceph-mon[18556]: osdmap e71: 12 total, 12 up, 12 in 2024-08-28T19:18:31.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:31 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/133859040' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-08-28T19:18:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:31 smithi028 ceph-mon[21769]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-28T19:18:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:31 smithi028 ceph-mon[21769]: osdmap e71: 12 total, 12 up, 12 in 2024-08-28T19:18:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:31 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/133859040' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-08-28T19:18:31.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:31 smithi115 ceph-mon[22184]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-28T19:18:31.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:31 smithi115 ceph-mon[22184]: osdmap e71: 12 total, 12 up, 12 in 2024-08-28T19:18:31.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:31 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/133859040' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-08-28T19:18:32.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-08-28T19:18:32.457 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:32.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:32 smithi016 ceph-mon[18556]: pgmap v251: 33 pgs: 2 creating+activating, 17 creating+peering, 6 active+clean, 8 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:32.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:32 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/133859040' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-08-28T19:18:32.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:32 smithi016 ceph-mon[18556]: osdmap e72: 12 total, 12 up, 12 in 2024-08-28T19:18:32.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:32 smithi028 ceph-mon[21769]: pgmap v251: 33 pgs: 2 creating+activating, 17 creating+peering, 6 active+clean, 8 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:32.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:32 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/133859040' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-08-28T19:18:32.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:32 smithi028 ceph-mon[21769]: osdmap e72: 12 total, 12 up, 12 in 2024-08-28T19:18:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:32 smithi115 ceph-mon[22184]: pgmap v251: 33 pgs: 2 creating+activating, 17 creating+peering, 6 active+clean, 8 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:32 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/133859040' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-08-28T19:18:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:32 smithi115 ceph-mon[22184]: osdmap e72: 12 total, 12 up, 12 in 2024-08-28T19:18:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.940+0000 7f24222eb640 1 -- 172.21.15.16:0/1882574041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241c109240 msgr2=0x7f241c109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.940+0000 7f24222eb640 1 --2- 172.21.15.16:0/1882574041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241c109240 0x7f241c109650 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2410009900 tx=0x7f241002a410 comp rx=0 tx=0).stop 2024-08-28T19:18:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.940+0000 7f24222eb640 1 -- 172.21.15.16:0/1882574041 shutdown_connections 2024-08-28T19:18:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.940+0000 7f24222eb640 1 --2- 172.21.15.16:0/1882574041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f241c109b90 0x7f241c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.940+0000 7f24222eb640 1 --2- 172.21.15.16:0/1882574041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241c109240 0x7f241c109650 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.940+0000 7f24222eb640 1 --2- 172.21.15.16:0/1882574041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241c103900 0x7f241c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.940+0000 7f24222eb640 1 -- 172.21.15.16:0/1882574041 >> 172.21.15.16:0/1882574041 conn(0x7f241c07b5f0 msgr2=0x7f241c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.940+0000 7f24222eb640 1 -- 172.21.15.16:0/1882574041 shutdown_connections 2024-08-28T19:18:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.940+0000 7f24222eb640 1 -- 172.21.15.16:0/1882574041 wait complete. 2024-08-28T19:18:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.941+0000 7f24222eb640 1 Processor -- start 2024-08-28T19:18:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.941+0000 7f24222eb640 1 -- start start 2024-08-28T19:18:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.941+0000 7f24222eb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f241c103900 0x7f241c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.941+0000 7f24222eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241c109240 0x7f241c19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.941+0000 7f24222eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241c109b90 0x7f241c19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.941+0000 7f24222eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f241c113850 con 0x7f241c109240 2024-08-28T19:18:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.941+0000 7f24222eb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1136d0 con 0x7f241c109b90 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.941+0000 7f24222eb640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1139d0 con 0x7f241c103900 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f241b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241c109240 0x7f241c19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f241b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241c109240 0x7f241c19a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43048/0 (socket says 172.21.15.16:43048) 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f241b7fe640 1 -- 172.21.15.16:0/1061183660 learned_addr learned my addr 172.21.15.16:0/1061183660 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f2420888640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241c109b90 0x7f241c19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f241bfff640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f241c103900 0x7f241c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f241b7fe640 1 -- 172.21.15.16:0/1061183660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f241c103900 msgr2=0x7f241c19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f241b7fe640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f241c103900 0x7f241c19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f241b7fe640 1 -- 172.21.15.16:0/1061183660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241c109b90 msgr2=0x7f241c19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f241b7fe640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241c109b90 0x7f241c19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f241b7fe640 1 -- 172.21.15.16:0/1061183660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241c1a0190 con 0x7f241c109240 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f241bfff640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f241c103900 0x7f241c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:18:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f241b7fe640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241c109240 0x7f241c19a7c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f241002a930 tx=0x7f2410004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f24197fa640 1 -- 172.21.15.16:0/1061183660 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241003f030 con 0x7f241c109240 2024-08-28T19:18:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f24197fa640 1 -- 172.21.15.16:0/1061183660 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2410003aa0 con 0x7f241c109240 2024-08-28T19:18:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f24222eb640 1 -- 172.21.15.16:0/1061183660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f241c1a0420 con 0x7f241c109240 2024-08-28T19:18:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f24197fa640 1 -- 172.21.15.16:0/1061183660 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2410035070 con 0x7f241c109240 2024-08-28T19:18:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.942+0000 7f24222eb640 1 -- 172.21.15.16:0/1061183660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f241c1a6be0 con 0x7f241c109240 2024-08-28T19:18:32.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.944+0000 7f24222eb640 1 -- 172.21.15.16:0/1061183660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f241c103d10 con 0x7f241c109240 2024-08-28T19:18:32.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.945+0000 7f24197fa640 1 -- 172.21.15.16:0/1061183660 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f2410042030 con 0x7f241c109240 2024-08-28T19:18:32.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.946+0000 7f24197fa640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24040b4c00 0x7f24040b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:32.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.946+0000 7f241bfff640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24040b4c00 0x7f24040b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:32.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.947+0000 7f24197fa640 1 -- 172.21.15.16:0/1061183660 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8094+0+0 (secure 0 0 0) 0x7f2410105c40 con 0x7f241c109240 2024-08-28T19:18:32.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.947+0000 7f241bfff640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24040b4c00 0x7f24040b7110 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f240c009b40 tx=0x7f240c005e50 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:32.950+0000 7f24197fa640 1 -- 172.21.15.16:0/1061183660 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f24100ccdb0 con 0x7f241c109240 2024-08-28T19:18:33.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.176+0000 7f24222eb640 1 -- 172.21.15.16:0/1061183660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f241c19b350 con 0x7f241c109240 2024-08-28T19:18:33.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.472+0000 7f24197fa640 1 -- 172.21.15.16:0/1061183660 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f24100cc6b0 con 0x7f241c109240 2024-08-28T19:18:33.476 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 2 and data pool 3 2024-08-28T19:18:33.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.475+0000 7f24222eb640 1 -- 172.21.15.16:0/1061183660 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24040b4c00 msgr2=0x7f24040b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:33.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.475+0000 7f24222eb640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24040b4c00 0x7f24040b7110 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f240c009b40 tx=0x7f240c005e50 comp rx=0 tx=0).stop 2024-08-28T19:18:33.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.475+0000 7f24222eb640 1 -- 172.21.15.16:0/1061183660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241c109240 msgr2=0x7f241c19a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.475+0000 7f24222eb640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241c109240 0x7f241c19a7c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f241002a930 tx=0x7f2410004270 comp rx=0 tx=0).stop 2024-08-28T19:18:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.475+0000 7f24222eb640 1 -- 172.21.15.16:0/1061183660 shutdown_connections 2024-08-28T19:18:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.475+0000 7f24222eb640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24040b4c00 0x7f24040b7110 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.476+0000 7f24222eb640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241c109b90 0x7f241c19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.476+0000 7f24222eb640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241c109240 0x7f241c19a7c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.476+0000 7f24222eb640 1 --2- 172.21.15.16:0/1061183660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f241c103900 0x7f241c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.476+0000 7f24222eb640 1 -- 172.21.15.16:0/1061183660 >> 172.21.15.16:0/1061183660 conn(0x7f241c07b5f0 msgr2=0x7f241c107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.476+0000 7f24222eb640 1 -- 172.21.15.16:0/1061183660 shutdown_connections 2024-08-28T19:18:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:33.476+0000 7f24222eb640 1 -- 172.21.15.16:0/1061183660 wait complete. 2024-08-28T19:18:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:33 smithi016 ceph-mon[18556]: osdmap e73: 12 total, 12 up, 12 in 2024-08-28T19:18:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:33 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1061183660' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-08-28T19:18:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:33 smithi016 ceph-mon[18556]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-08-28T19:18:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:33 smithi028 ceph-mon[21769]: osdmap e73: 12 total, 12 up, 12 in 2024-08-28T19:18:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:33 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1061183660' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-08-28T19:18:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:33 smithi028 ceph-mon[21769]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-08-28T19:18:33.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:33 smithi115 ceph-mon[22184]: osdmap e73: 12 total, 12 up, 12 in 2024-08-28T19:18:33.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:33 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1061183660' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-08-28T19:18:33.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:33 smithi115 ceph-mon[22184]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-08-28T19:18:34.144 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-08-28T19:18:34.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-08-28T19:18:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:34 smithi016 ceph-mon[18556]: pgmap v254: 97 pgs: 2 creating+activating, 24 creating+peering, 14 active+clean, 57 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:34 smithi016 ceph-mon[18556]: osdmap e74: 12 total, 12 up, 12 in 2024-08-28T19:18:34.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:34 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1061183660' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-08-28T19:18:34.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:34 smithi016 ceph-mon[18556]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-08-28T19:18:34.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:34 smithi016 ceph-mon[18556]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-08-28T19:18:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:34 smithi028 ceph-mon[21769]: pgmap v254: 97 pgs: 2 creating+activating, 24 creating+peering, 14 active+clean, 57 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:34 smithi028 ceph-mon[21769]: osdmap e74: 12 total, 12 up, 12 in 2024-08-28T19:18:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:34 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1061183660' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-08-28T19:18:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:34 smithi028 ceph-mon[21769]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-08-28T19:18:34.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:34 smithi028 ceph-mon[21769]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-08-28T19:18:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:34 smithi115 ceph-mon[22184]: pgmap v254: 97 pgs: 2 creating+activating, 24 creating+peering, 14 active+clean, 57 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:34 smithi115 ceph-mon[22184]: osdmap e74: 12 total, 12 up, 12 in 2024-08-28T19:18:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:34 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1061183660' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-08-28T19:18:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:34 smithi115 ceph-mon[22184]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-08-28T19:18:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:34 smithi115 ceph-mon[22184]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-08-28T19:18:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.873+0000 7fc4e17ec640 1 -- 172.21.15.16:0/4242291913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc108140 msgr2=0x7fc4dc108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.873+0000 7fc4e17ec640 1 --2- 172.21.15.16:0/4242291913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc108140 0x7fc4dc108550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009900 tx=0x7fc4c402a410 comp rx=0 tx=0).stop 2024-08-28T19:18:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.873+0000 7fc4e17ec640 1 -- 172.21.15.16:0/4242291913 shutdown_connections 2024-08-28T19:18:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.873+0000 7fc4e17ec640 1 --2- 172.21.15.16:0/4242291913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc4dc108a90 0x7fc4dc10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.873+0000 7fc4e17ec640 1 --2- 172.21.15.16:0/4242291913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc108140 0x7fc4dc108550 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.873+0000 7fc4e17ec640 1 --2- 172.21.15.16:0/4242291913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4dc102800 0x7fc4dc102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.873+0000 7fc4e17ec640 1 -- 172.21.15.16:0/4242291913 >> 172.21.15.16:0/4242291913 conn(0x7fc4dc0794d0 msgr2=0x7fc4dc0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.873+0000 7fc4e17ec640 1 -- 172.21.15.16:0/4242291913 shutdown_connections 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.873+0000 7fc4e17ec640 1 -- 172.21.15.16:0/4242291913 wait complete. 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.874+0000 7fc4e17ec640 1 Processor -- start 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.874+0000 7fc4e17ec640 1 -- start start 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.874+0000 7fc4e17ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102800 0x7fc4dc19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.874+0000 7fc4e17ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4dc108140 0x7fc4dc19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.874+0000 7fc4e17ec640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc4dc108a90 0x7fc4dc19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.874+0000 7fc4e17ec640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc1126c0 con 0x7fc4dc102800 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.874+0000 7fc4e17ec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc112540 con 0x7fc4dc108140 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.874+0000 7fc4e17ec640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc112840 con 0x7fc4dc108a90 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102800 0x7fc4dc19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102800 0x7fc4dc19a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57064/0 (socket says 172.21.15.16:57064) 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4daffd640 1 -- 172.21.15.16:0/3078780819 learned_addr learned my addr 172.21.15.16:0/3078780819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:34.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4d27fc640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4dc108140 0x7fc4dc19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:34.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4db7fe640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc4dc108a90 0x7fc4dc19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:34.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4daffd640 1 -- 172.21.15.16:0/3078780819 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc4dc108a90 msgr2=0x7fc4dc19fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:34.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4daffd640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc4dc108a90 0x7fc4dc19fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:34.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4daffd640 1 -- 172.21.15.16:0/3078780819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4dc108140 msgr2=0x7fc4dc19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:34.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4daffd640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4dc108140 0x7fc4dc19a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:34.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4daffd640 1 -- 172.21.15.16:0/3078780819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4dc1a0290 con 0x7fc4dc102800 2024-08-28T19:18:34.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4daffd640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102800 0x7fc4dc19a320 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00e9f0 tx=0x7fc4cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:34.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4d8ff9640 1 -- 172.21.15.16:0/3078780819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc00cd80 con 0x7fc4dc102800 2024-08-28T19:18:34.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4d8ff9640 1 -- 172.21.15.16:0/3078780819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4cc0108e0 con 0x7fc4dc102800 2024-08-28T19:18:34.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4e17ec640 1 -- 172.21.15.16:0/3078780819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4dc1a0580 con 0x7fc4dc102800 2024-08-28T19:18:34.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4d8ff9640 1 -- 172.21.15.16:0/3078780819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc018980 con 0x7fc4dc102800 2024-08-28T19:18:34.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.875+0000 7fc4e17ec640 1 -- 172.21.15.16:0/3078780819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4dc1a6ce0 con 0x7fc4dc102800 2024-08-28T19:18:34.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.876+0000 7fc4e17ec640 1 -- 172.21.15.16:0/3078780819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4dc102c10 con 0x7fc4dc102800 2024-08-28T19:18:34.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.878+0000 7fc4d8ff9640 1 -- 172.21.15.16:0/3078780819 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fc4cc010430 con 0x7fc4dc102800 2024-08-28T19:18:34.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.879+0000 7fc4d8ff9640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4c80b4b80 0x7fc4c80b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:34.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.879+0000 7fc4d27fc640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4c80b4b80 0x7fc4c80b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:34.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.879+0000 7fc4d8ff9640 1 -- 172.21.15.16:0/3078780819 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 8194+0+0 (secure 0 0 0) 0x7fc4cc0e5a80 con 0x7fc4dc102800 2024-08-28T19:18:34.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.879+0000 7fc4d27fc640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4c80b4b80 0x7fc4c80b7090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4000c00 tx=0x7fc4c4031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:34.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:34.882+0000 7fc4d8ff9640 1 -- 172.21.15.16:0/3078780819 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fc4cc0acb70 con 0x7fc4dc102800 2024-08-28T19:18:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.078+0000 7fc4e17ec640 1 -- 172.21.15.16:0/3078780819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fc4dc1a6eb0 con 0x7fc4dc102800 2024-08-28T19:18:35.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.484+0000 7fc4d8ff9640 1 -- 172.21.15.16:0/3078780819 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v75) ==== 157+0+0 (secure 0 0 0) 0x7fc4cc0b50b0 con 0x7fc4dc102800 2024-08-28T19:18:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 -- 172.21.15.16:0/3078780819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4c80b4b80 msgr2=0x7fc4c80b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4c80b4b80 0x7fc4c80b7090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4000c00 tx=0x7fc4c4031400 comp rx=0 tx=0).stop 2024-08-28T19:18:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 -- 172.21.15.16:0/3078780819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102800 msgr2=0x7fc4dc19a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102800 0x7fc4dc19a320 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00e9f0 tx=0x7fc4cc00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:18:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 -- 172.21.15.16:0/3078780819 shutdown_connections 2024-08-28T19:18:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc4dc108a90 0x7fc4dc19fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4c80b4b80 0x7fc4c80b7090 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4dc108140 0x7fc4dc19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 --2- 172.21.15.16:0/3078780819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102800 0x7fc4dc19a320 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:35.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 -- 172.21.15.16:0/3078780819 >> 172.21.15.16:0/3078780819 conn(0x7fc4dc0794d0 msgr2=0x7fc4dc101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:35.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 -- 172.21.15.16:0/3078780819 shutdown_connections 2024-08-28T19:18:35.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:35.488+0000 7fc4e17ec640 1 -- 172.21.15.16:0/3078780819 wait complete. 2024-08-28T19:18:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:35 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] up:active 2024-08-28T19:18:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:35 smithi016 ceph-mon[18556]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:18:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:35 smithi016 ceph-mon[18556]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:18:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:35 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3078780819' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-08-28T19:18:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:35 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] up:active 2024-08-28T19:18:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:35 smithi028 ceph-mon[21769]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:18:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:35 smithi028 ceph-mon[21769]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:18:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:35 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3078780819' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-08-28T19:18:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:35 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] up:active 2024-08-28T19:18:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:35 smithi115 ceph-mon[22184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:18:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:35 smithi115 ceph-mon[22184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:18:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:35 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3078780819' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-08-28T19:18:36.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-08-28T19:18:36.427 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:36.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:36 smithi016 ceph-mon[18556]: pgmap v256: 97 pgs: 2 creating+activating, 7 creating+peering, 76 active+clean, 12 unknown; 578 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 7 op/s 2024-08-28T19:18:36.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:36 smithi016 ceph-mon[18556]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-08-28T19:18:36.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:36 smithi016 ceph-mon[18556]: Cluster is now healthy 2024-08-28T19:18:36.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:36 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3078780819' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-08-28T19:18:36.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:36 smithi016 ceph-mon[18556]: osdmap e75: 12 total, 12 up, 12 in 2024-08-28T19:18:36.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:36 smithi028 ceph-mon[21769]: pgmap v256: 97 pgs: 2 creating+activating, 7 creating+peering, 76 active+clean, 12 unknown; 578 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 7 op/s 2024-08-28T19:18:36.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:36 smithi028 ceph-mon[21769]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-08-28T19:18:36.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:36 smithi028 ceph-mon[21769]: Cluster is now healthy 2024-08-28T19:18:36.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:36 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3078780819' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-08-28T19:18:36.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:36 smithi028 ceph-mon[21769]: osdmap e75: 12 total, 12 up, 12 in 2024-08-28T19:18:36.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:36 smithi115 ceph-mon[22184]: pgmap v256: 97 pgs: 2 creating+activating, 7 creating+peering, 76 active+clean, 12 unknown; 578 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 7 op/s 2024-08-28T19:18:36.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:36 smithi115 ceph-mon[22184]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-08-28T19:18:36.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:36 smithi115 ceph-mon[22184]: Cluster is now healthy 2024-08-28T19:18:36.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:36 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3078780819' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-08-28T19:18:36.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:36 smithi115 ceph-mon[22184]: osdmap e75: 12 total, 12 up, 12 in 2024-08-28T19:18:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.907+0000 7f0e6257c640 1 -- 172.21.15.16:0/4280461435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5c109bd0 msgr2=0x7f0e5c110400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.907+0000 7f0e6257c640 1 --2- 172.21.15.16:0/4280461435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5c109bd0 0x7f0e5c110400 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009ca0 tx=0x7f0e5402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:18:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.907+0000 7f0e6257c640 1 -- 172.21.15.16:0/4280461435 shutdown_connections 2024-08-28T19:18:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.907+0000 7f0e6257c640 1 --2- 172.21.15.16:0/4280461435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5c109bd0 0x7f0e5c110400 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.907+0000 7f0e6257c640 1 --2- 172.21.15.16:0/4280461435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5c109280 0x7f0e5c109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.907+0000 7f0e6257c640 1 --2- 172.21.15.16:0/4280461435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e5c103940 0x7f0e5c103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.907+0000 7f0e6257c640 1 -- 172.21.15.16:0/4280461435 >> 172.21.15.16:0/4280461435 conn(0x7f0e5c0ff390 msgr2=0x7f0e5c1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.907+0000 7f0e6257c640 1 -- 172.21.15.16:0/4280461435 shutdown_connections 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.907+0000 7f0e6257c640 1 -- 172.21.15.16:0/4280461435 wait complete. 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e6257c640 1 Processor -- start 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e6257c640 1 -- start start 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e6257c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5c103940 0x7f0e5c074bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e6257c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e5c109280 0x7f0e5c06de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e6257c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5c109bd0 0x7f0e5c06e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e6257c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c1138f0 con 0x7f0e5c109bd0 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e6257c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c113770 con 0x7f0e5c103940 2024-08-28T19:18:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e6257c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c113a70 con 0x7f0e5c109280 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e60b19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5c109bd0 0x7f0e5c06e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e60b19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5c109bd0 0x7f0e5c06e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57074/0 (socket says 172.21.15.16:57074) 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e60b19640 1 -- 172.21.15.16:0/2817802749 learned_addr learned my addr 172.21.15.16:0/2817802749 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e5b7fe640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e5c109280 0x7f0e5c06de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.908+0000 7f0e5bfff640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5c103940 0x7f0e5c074bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.909+0000 7f0e60b19640 1 -- 172.21.15.16:0/2817802749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e5c109280 msgr2=0x7f0e5c06de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.909+0000 7f0e60b19640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e5c109280 0x7f0e5c06de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.909+0000 7f0e60b19640 1 -- 172.21.15.16:0/2817802749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5c103940 msgr2=0x7f0e5c074bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.909+0000 7f0e60b19640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5c103940 0x7f0e5c074bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.909+0000 7f0e60b19640 1 -- 172.21.15.16:0/2817802749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5c06ebf0 con 0x7f0e5c109bd0 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.909+0000 7f0e60b19640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5c109bd0 0x7f0e5c06e3d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0e54005cc0 tx=0x7f0e5402f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.909+0000 7f0e597fa640 1 -- 172.21.15.16:0/2817802749 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54003cf0 con 0x7f0e5c109bd0 2024-08-28T19:18:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.909+0000 7f0e6257c640 1 -- 172.21.15.16:0/2817802749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e5c073830 con 0x7f0e5c109bd0 2024-08-28T19:18:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.909+0000 7f0e597fa640 1 -- 172.21.15.16:0/2817802749 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e540044f0 con 0x7f0e5c109bd0 2024-08-28T19:18:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.909+0000 7f0e597fa640 1 -- 172.21.15.16:0/2817802749 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5402fc30 con 0x7f0e5c109bd0 2024-08-28T19:18:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.909+0000 7f0e6257c640 1 -- 172.21.15.16:0/2817802749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e5c073d70 con 0x7f0e5c109bd0 2024-08-28T19:18:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.910+0000 7f0e6257c640 1 -- 172.21.15.16:0/2817802749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e5c103d50 con 0x7f0e5c109bd0 2024-08-28T19:18:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.912+0000 7f0e597fa640 1 -- 172.21.15.16:0/2817802749 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f0e540046a0 con 0x7f0e5c109bd0 2024-08-28T19:18:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.912+0000 7f0e597fa640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e440b4b80 0x7f0e440b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.912+0000 7f0e5bfff640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e440b4b80 0x7f0e440b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.912+0000 7f0e597fa640 1 -- 172.21.15.16:0/2817802749 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8492+0+0 (secure 0 0 0) 0x7f0e54106890 con 0x7f0e5c109bd0 2024-08-28T19:18:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.913+0000 7f0e5bfff640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e440b4b80 0x7f0e440b7090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009990 tx=0x7f0e4c009110 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:36.915+0000 7f0e597fa640 1 -- 172.21.15.16:0/2817802749 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f0e540cd8c0 con 0x7f0e5c109bd0 2024-08-28T19:18:37.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:37.102+0000 7f0e6257c640 1 -- 172.21.15.16:0/2817802749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0e5c10b410 con 0x7f0e5c109bd0 2024-08-28T19:18:37.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:37 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2817802749' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-08-28T19:18:37.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:37 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2817802749' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-08-28T19:18:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:37 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2817802749' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-08-28T19:18:38.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.498+0000 7f0e597fa640 1 -- 172.21.15.16:0/2817802749 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v77) ==== 228+0+0 (secure 0 0 0) 0x7f0e540cd1c0 con 0x7f0e5c109bd0 2024-08-28T19:18:38.501 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' created 2024-08-28T19:18:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.501+0000 7f0e6257c640 1 -- 172.21.15.16:0/2817802749 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e440b4b80 msgr2=0x7f0e440b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.501+0000 7f0e6257c640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e440b4b80 0x7f0e440b7090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009990 tx=0x7f0e4c009110 comp rx=0 tx=0).stop 2024-08-28T19:18:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.501+0000 7f0e6257c640 1 -- 172.21.15.16:0/2817802749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5c109bd0 msgr2=0x7f0e5c06e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.501+0000 7f0e6257c640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5c109bd0 0x7f0e5c06e3d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0e54005cc0 tx=0x7f0e5402f470 comp rx=0 tx=0).stop 2024-08-28T19:18:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.502+0000 7f0e6257c640 1 -- 172.21.15.16:0/2817802749 shutdown_connections 2024-08-28T19:18:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.502+0000 7f0e6257c640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5c109bd0 0x7f0e5c06e3d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.502+0000 7f0e6257c640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e5c109280 0x7f0e5c06de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.502+0000 7f0e6257c640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e440b4b80 0x7f0e440b7090 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.502+0000 7f0e6257c640 1 --2- 172.21.15.16:0/2817802749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5c103940 0x7f0e5c074bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.502+0000 7f0e6257c640 1 -- 172.21.15.16:0/2817802749 >> 172.21.15.16:0/2817802749 conn(0x7f0e5c0ff390 msgr2=0x7f0e5c108890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.502+0000 7f0e6257c640 1 -- 172.21.15.16:0/2817802749 shutdown_connections 2024-08-28T19:18:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:38.502+0000 7f0e6257c640 1 -- 172.21.15.16:0/2817802749 wait complete. 2024-08-28T19:18:38.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:38 smithi016 ceph-mon[18556]: pgmap v258: 97 pgs: 97 active+clean; 579 KiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-08-28T19:18:38.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:38 smithi016 ceph-mon[18556]: osdmap e76: 12 total, 12 up, 12 in 2024-08-28T19:18:38.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:38 smithi028 ceph-mon[21769]: pgmap v258: 97 pgs: 97 active+clean; 579 KiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-08-28T19:18:38.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:38 smithi028 ceph-mon[21769]: osdmap e76: 12 total, 12 up, 12 in 2024-08-28T19:18:38.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:38 smithi115 ceph-mon[22184]: pgmap v258: 97 pgs: 97 active+clean; 579 KiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-08-28T19:18:38.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:38 smithi115 ceph-mon[22184]: osdmap e76: 12 total, 12 up, 12 in 2024-08-28T19:18:39.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-08-28T19:18:39.454 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:39.506 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:18:39 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b[52931]: 2024-08-28T19:18:39.106+0000 7fd5ccaae640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-28T19:18:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:39 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2817802749' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-08-28T19:18:39.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:39 smithi028 ceph-mon[21769]: osdmap e77: 12 total, 12 up, 12 in 2024-08-28T19:18:39.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:39 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2817802749' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-08-28T19:18:39.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:39 smithi016 ceph-mon[18556]: osdmap e77: 12 total, 12 up, 12 in 2024-08-28T19:18:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:39 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2817802749' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-08-28T19:18:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:39 smithi115 ceph-mon[22184]: osdmap e77: 12 total, 12 up, 12 in 2024-08-28T19:18:39.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.947+0000 7f8c08ba3640 1 -- 172.21.15.16:0/4042268189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c0410dc30 msgr2=0x7f8c04110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:39.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.947+0000 7f8c08ba3640 1 --2- 172.21.15.16:0/4042268189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c0410dc30 0x7f8c04110120 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc009ca0 tx=0x7f8bfc02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:18:39.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.947+0000 7f8c08ba3640 1 -- 172.21.15.16:0/4042268189 shutdown_connections 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.947+0000 7f8c08ba3640 1 --2- 172.21.15.16:0/4042268189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c0410dc30 0x7f8c04110120 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.947+0000 7f8c08ba3640 1 --2- 172.21.15.16:0/4042268189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c04105120 0x7f8c0410d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.947+0000 7f8c08ba3640 1 --2- 172.21.15.16:0/4042268189 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c041047d0 0x7f8c04104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.947+0000 7f8c08ba3640 1 -- 172.21.15.16:0/4042268189 >> 172.21.15.16:0/4042268189 conn(0x7f8c04079510 msgr2=0x7f8c04079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.947+0000 7f8c08ba3640 1 -- 172.21.15.16:0/4042268189 shutdown_connections 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.947+0000 7f8c08ba3640 1 -- 172.21.15.16:0/4042268189 wait complete. 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c08ba3640 1 Processor -- start 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c08ba3640 1 -- start start 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c08ba3640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c041047d0 0x7f8c04102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c08ba3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c04105120 0x7f8c04102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c08ba3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c0410dc30 0x7f8c04102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c08ba3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c041138b0 con 0x7f8c0410dc30 2024-08-28T19:18:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c08ba3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8c04113730 con 0x7f8c04105120 2024-08-28T19:18:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c08ba3640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8c04113a30 con 0x7f8c041047d0 2024-08-28T19:18:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c02d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c0410dc30 0x7f8c04102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c02d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c0410dc30 0x7f8c04102f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57078/0 (socket says 172.21.15.16:57078) 2024-08-28T19:18:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c02d9d640 1 -- 172.21.15.16:0/2225694852 learned_addr learned my addr 172.21.15.16:0/2225694852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c01d9b640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c04105120 0x7f8c04102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.948+0000 7f8c0259c640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c041047d0 0x7f8c04102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.949+0000 7f8c02d9d640 1 -- 172.21.15.16:0/2225694852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c041047d0 msgr2=0x7f8c04102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.949+0000 7f8c02d9d640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c041047d0 0x7f8c04102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.949+0000 7f8c02d9d640 1 -- 172.21.15.16:0/2225694852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c04105120 msgr2=0x7f8c04102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.949+0000 7f8c02d9d640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c04105120 0x7f8c04102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.949+0000 7f8c02d9d640 1 -- 172.21.15.16:0/2225694852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c041a8450 con 0x7f8c0410dc30 2024-08-28T19:18:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.949+0000 7f8c02d9d640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c0410dc30 0x7f8c04102f90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc005cc0 tx=0x7f8bfc02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.949+0000 7f8bf37fe640 1 -- 172.21.15.16:0/2225694852 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc003bf0 con 0x7f8c0410dc30 2024-08-28T19:18:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.949+0000 7f8bf37fe640 1 -- 172.21.15.16:0/2225694852 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bfc003d90 con 0x7f8c0410dc30 2024-08-28T19:18:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.949+0000 7f8c08ba3640 1 -- 172.21.15.16:0/2225694852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c041a86e0 con 0x7f8c0410dc30 2024-08-28T19:18:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.949+0000 7f8bf37fe640 1 -- 172.21.15.16:0/2225694852 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc02fb20 con 0x7f8c0410dc30 2024-08-28T19:18:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.949+0000 7f8c08ba3640 1 -- 172.21.15.16:0/2225694852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c041a8c70 con 0x7f8c0410dc30 2024-08-28T19:18:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.950+0000 7f8c08ba3640 1 -- 172.21.15.16:0/2225694852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c0406c000 con 0x7f8c0410dc30 2024-08-28T19:18:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.952+0000 7f8bf37fe640 1 -- 172.21.15.16:0/2225694852 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f8bfc004580 con 0x7f8c0410dc30 2024-08-28T19:18:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.953+0000 7f8bf37fe640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bec0b4c00 0x7f8bec0b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.953+0000 7f8c0259c640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bec0b4c00 0x7f8bec0b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.953+0000 7f8bf37fe640 1 -- 172.21.15.16:0/2225694852 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8985+0+0 (secure 0 0 0) 0x7f8bfc1068e0 con 0x7f8c0410dc30 2024-08-28T19:18:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.953+0000 7f8c0259c640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bec0b4c00 0x7f8bec0b7110 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8c04103f60 tx=0x7f8bf400b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:39.957+0000 7f8bf37fe640 1 -- 172.21.15.16:0/2225694852 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8bfc0cd720 con 0x7f8c0410dc30 2024-08-28T19:18:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.142+0000 7f8c08ba3640 1 -- 172.21.15.16:0/2225694852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f8c041a8f90 con 0x7f8c0410dc30 2024-08-28T19:18:40.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.512+0000 7f8bf37fe640 1 -- 172.21.15.16:0/2225694852 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v79) ==== 169+0+0 (secure 0 0 0) 0x7f8bfc0cd020 con 0x7f8c0410dc30 2024-08-28T19:18:40.513 INFO:teuthology.orchestra.run.smithi016.stderr:set pool 4 allow_ec_overwrites to true 2024-08-28T19:18:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.514+0000 7f8c08ba3640 1 -- 172.21.15.16:0/2225694852 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bec0b4c00 msgr2=0x7f8bec0b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.514+0000 7f8c08ba3640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bec0b4c00 0x7f8bec0b7110 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8c04103f60 tx=0x7f8bf400b390 comp rx=0 tx=0).stop 2024-08-28T19:18:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.514+0000 7f8c08ba3640 1 -- 172.21.15.16:0/2225694852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c0410dc30 msgr2=0x7f8c04102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.514+0000 7f8c08ba3640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c0410dc30 0x7f8c04102f90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc005cc0 tx=0x7f8bfc02f380 comp rx=0 tx=0).stop 2024-08-28T19:18:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.514+0000 7f8c08ba3640 1 -- 172.21.15.16:0/2225694852 shutdown_connections 2024-08-28T19:18:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.515+0000 7f8c08ba3640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c0410dc30 0x7f8c04102f90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.515+0000 7f8c08ba3640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bec0b4c00 0x7f8bec0b7110 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.515+0000 7f8c08ba3640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c04105120 0x7f8c04102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.515+0000 7f8c08ba3640 1 --2- 172.21.15.16:0/2225694852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c041047d0 0x7f8c04102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:40.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.515+0000 7f8c08ba3640 1 -- 172.21.15.16:0/2225694852 >> 172.21.15.16:0/2225694852 conn(0x7f8c04079510 msgr2=0x7f8c041003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:40.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.515+0000 7f8c08ba3640 1 -- 172.21.15.16:0/2225694852 shutdown_connections 2024-08-28T19:18:40.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:40.515+0000 7f8c08ba3640 1 -- 172.21.15.16:0/2225694852 wait complete. 2024-08-28T19:18:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:40 smithi028 ceph-mon[21769]: pgmap v261: 129 pgs: 11 creating+peering, 21 unknown, 97 active+clean; 579 KiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-08-28T19:18:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:40 smithi028 ceph-mon[21769]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-28T19:18:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:40 smithi028 ceph-mon[21769]: osdmap e78: 12 total, 12 up, 12 in 2024-08-28T19:18:40.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:40 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2225694852' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-08-28T19:18:40.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:40 smithi115 ceph-mon[22184]: pgmap v261: 129 pgs: 11 creating+peering, 21 unknown, 97 active+clean; 579 KiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-08-28T19:18:40.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:40 smithi115 ceph-mon[22184]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-28T19:18:40.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:40 smithi115 ceph-mon[22184]: osdmap e78: 12 total, 12 up, 12 in 2024-08-28T19:18:40.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:40 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2225694852' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-08-28T19:18:40.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:40 smithi016 ceph-mon[18556]: pgmap v261: 129 pgs: 11 creating+peering, 21 unknown, 97 active+clean; 579 KiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-08-28T19:18:40.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:40 smithi016 ceph-mon[18556]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-28T19:18:40.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:40 smithi016 ceph-mon[18556]: osdmap e78: 12 total, 12 up, 12 in 2024-08-28T19:18:40.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:40 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2225694852' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-08-28T19:18:41.251 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-08-28T19:18:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:41.771 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:41 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2225694852' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-08-28T19:18:41.771 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:41 smithi016 ceph-mon[18556]: osdmap e79: 12 total, 12 up, 12 in 2024-08-28T19:18:41.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:41 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2225694852' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-08-28T19:18:41.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:41 smithi028 ceph-mon[21769]: osdmap e79: 12 total, 12 up, 12 in 2024-08-28T19:18:41.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:41 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2225694852' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-08-28T19:18:41.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:41 smithi115 ceph-mon[22184]: osdmap e79: 12 total, 12 up, 12 in 2024-08-28T19:18:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.988+0000 7f6da5f98640 1 -- 172.21.15.16:0/228016561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da0108160 msgr2=0x7f6da0108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.988+0000 7f6da5f98640 1 --2- 172.21.15.16:0/228016561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da0108160 0x7f6da0108570 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009900 tx=0x7f6d8802a410 comp rx=0 tx=0).stop 2024-08-28T19:18:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.988+0000 7f6da5f98640 1 -- 172.21.15.16:0/228016561 shutdown_connections 2024-08-28T19:18:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.988+0000 7f6da5f98640 1 --2- 172.21.15.16:0/228016561 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da0108ab0 0x7f6da010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.988+0000 7f6da5f98640 1 --2- 172.21.15.16:0/228016561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da0108160 0x7f6da0108570 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.988+0000 7f6da5f98640 1 --2- 172.21.15.16:0/228016561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da0102820 0x7f6da0102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.988+0000 7f6da5f98640 1 -- 172.21.15.16:0/228016561 >> 172.21.15.16:0/228016561 conn(0x7f6da00794e0 msgr2=0x7f6da00798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.988+0000 7f6da5f98640 1 -- 172.21.15.16:0/228016561 shutdown_connections 2024-08-28T19:18:41.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.988+0000 7f6da5f98640 1 -- 172.21.15.16:0/228016561 wait complete. 2024-08-28T19:18:41.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.989+0000 7f6da5f98640 1 Processor -- start 2024-08-28T19:18:41.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.989+0000 7f6da5f98640 1 -- start start 2024-08-28T19:18:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.989+0000 7f6da5f98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da0102820 0x7f6da0199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.989+0000 7f6da5f98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da0108160 0x7f6da01998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.989+0000 7f6da5f98640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da0108ab0 0x7f6da019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.989+0000 7f6da5f98640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6da01127e0 con 0x7f6da0108160 2024-08-28T19:18:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.989+0000 7f6da5f98640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6da0112660 con 0x7f6da0102820 2024-08-28T19:18:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.989+0000 7f6da5f98640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6da0112960 con 0x7f6da0108ab0 2024-08-28T19:18:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.989+0000 7f6d96ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da0108160 0x7f6da01998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.989+0000 7f6d96ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da0108160 0x7f6da01998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57096/0 (socket says 172.21.15.16:57096) 2024-08-28T19:18:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.989+0000 7f6d96ffd640 1 -- 172.21.15.16:0/1193458345 learned_addr learned my addr 172.21.15.16:0/1193458345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d9f7fe640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da0102820 0x7f6da0199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d9ffff640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da0108ab0 0x7f6da019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d96ffd640 1 -- 172.21.15.16:0/1193458345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da0108ab0 msgr2=0x7f6da019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d96ffd640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da0108ab0 0x7f6da019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d96ffd640 1 -- 172.21.15.16:0/1193458345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da0102820 msgr2=0x7f6da0199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d96ffd640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da0102820 0x7f6da0199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d96ffd640 1 -- 172.21.15.16:0/1193458345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da019f280 con 0x7f6da0108160 2024-08-28T19:18:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d9ffff640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da0108ab0 0x7f6da019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:18:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d96ffd640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da0108160 0x7f6da01998b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6d88031a90 tx=0x7f6d88004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d9d7fa640 1 -- 172.21.15.16:0/1193458345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8803f030 con 0x7f6da0108160 2024-08-28T19:18:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6da5f98640 1 -- 172.21.15.16:0/1193458345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da019f510 con 0x7f6da0108160 2024-08-28T19:18:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d9d7fa640 1 -- 172.21.15.16:0/1193458345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d88004490 con 0x7f6da0108160 2024-08-28T19:18:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6d9d7fa640 1 -- 172.21.15.16:0/1193458345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88035070 con 0x7f6da0108160 2024-08-28T19:18:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.990+0000 7f6da5f98640 1 -- 172.21.15.16:0/1193458345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da01a5cd0 con 0x7f6da0108160 2024-08-28T19:18:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.992+0000 7f6da5f98640 1 -- 172.21.15.16:0/1193458345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da0103aa0 con 0x7f6da0108160 2024-08-28T19:18:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.993+0000 7f6d9d7fa640 1 -- 172.21.15.16:0/1193458345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f6d88042030 con 0x7f6da0108160 2024-08-28T19:18:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.994+0000 7f6d9d7fa640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d8c0b4b80 0x7f6d8c0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.994+0000 7f6d9f7fe640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d8c0b4b80 0x7f6d8c0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.994+0000 7f6d9d7fa640 1 -- 172.21.15.16:0/1193458345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8985+0+0 (secure 0 0 0) 0x7f6d88105b30 con 0x7f6da0108160 2024-08-28T19:18:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.994+0000 7f6d9f7fe640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d8c0b4b80 0x7f6d8c0b7090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6d90009990 tx=0x7f6d90009110 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:41.997+0000 7f6d9d7fa640 1 -- 172.21.15.16:0/1193458345 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f6d880cc970 con 0x7f6da0108160 2024-08-28T19:18:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.221+0000 7f6da5f98640 1 -- 172.21.15.16:0/1193458345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f6da010a8d0 con 0x7f6da0108160 2024-08-28T19:18:42.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.525+0000 7f6d9d7fa640 1 -- 172.21.15.16:0/1193458345 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f6d880cc270 con 0x7f6da0108160 2024-08-28T19:18:42.526 INFO:teuthology.orchestra.run.smithi016.stderr:added data pool 4 to fsmap 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.528+0000 7f6da5f98640 1 -- 172.21.15.16:0/1193458345 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d8c0b4b80 msgr2=0x7f6d8c0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.528+0000 7f6da5f98640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d8c0b4b80 0x7f6d8c0b7090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6d90009990 tx=0x7f6d90009110 comp rx=0 tx=0).stop 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.528+0000 7f6da5f98640 1 -- 172.21.15.16:0/1193458345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da0108160 msgr2=0x7f6da01998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.528+0000 7f6da5f98640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da0108160 0x7f6da01998b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6d88031a90 tx=0x7f6d88004270 comp rx=0 tx=0).stop 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.528+0000 7f6da5f98640 1 -- 172.21.15.16:0/1193458345 shutdown_connections 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.528+0000 7f6da5f98640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da0108ab0 0x7f6da019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.528+0000 7f6da5f98640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da0108160 0x7f6da01998b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.528+0000 7f6da5f98640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d8c0b4b80 0x7f6d8c0b7090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.528+0000 7f6da5f98640 1 --2- 172.21.15.16:0/1193458345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da0102820 0x7f6da0199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.528+0000 7f6da5f98640 1 -- 172.21.15.16:0/1193458345 >> 172.21.15.16:0/1193458345 conn(0x7f6da00794e0 msgr2=0x7f6da0105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.528+0000 7f6da5f98640 1 -- 172.21.15.16:0/1193458345 shutdown_connections 2024-08-28T19:18:42.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:42.529+0000 7f6da5f98640 1 -- 172.21.15.16:0/1193458345 wait complete. 2024-08-28T19:18:42.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:42 smithi028 ceph-mon[21769]: pgmap v264: 129 pgs: 13 creating+peering, 5 unknown, 111 active+clean; 579 KiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:42.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:42 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1193458345' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-08-28T19:18:42.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:42 smithi016 ceph-mon[18556]: pgmap v264: 129 pgs: 13 creating+peering, 5 unknown, 111 active+clean; 579 KiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:42.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:42 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1193458345' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-08-28T19:18:42.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:42 smithi115 ceph-mon[22184]: pgmap v264: 129 pgs: 13 creating+peering, 5 unknown, 111 active+clean; 579 KiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:42.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:42 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1193458345' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-08-28T19:18:43.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:18:43.456 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:43.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:43 smithi028 ceph-mon[21769]: osdmap e80: 12 total, 12 up, 12 in 2024-08-28T19:18:43.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:43 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1193458345' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-08-28T19:18:43.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:43 smithi028 ceph-mon[21769]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:18:43.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:43 smithi016 ceph-mon[18556]: osdmap e80: 12 total, 12 up, 12 in 2024-08-28T19:18:43.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:43 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1193458345' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-08-28T19:18:43.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:43 smithi016 ceph-mon[18556]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:18:43.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:43 smithi115 ceph-mon[22184]: osdmap e80: 12 total, 12 up, 12 in 2024-08-28T19:18:43.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:43 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1193458345' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-08-28T19:18:43.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:43 smithi115 ceph-mon[22184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:18:43.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.920+0000 7f1d6b01d640 1 -- 172.21.15.16:0/729187022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d641081c0 msgr2=0x7f1d641085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:43.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.920+0000 7f1d6b01d640 1 --2- 172.21.15.16:0/729187022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d641081c0 0x7f1d641085d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c009900 tx=0x7f1d4c02a410 comp rx=0 tx=0).stop 2024-08-28T19:18:43.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.920+0000 7f1d6b01d640 1 -- 172.21.15.16:0/729187022 shutdown_connections 2024-08-28T19:18:43.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.920+0000 7f1d6b01d640 1 --2- 172.21.15.16:0/729187022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d64108b10 0x7f1d6410f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:43.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.920+0000 7f1d6b01d640 1 --2- 172.21.15.16:0/729187022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d641081c0 0x7f1d641085d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:43.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.920+0000 7f1d6b01d640 1 --2- 172.21.15.16:0/729187022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d64102880 0x7f1d64102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:43.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.920+0000 7f1d6b01d640 1 -- 172.21.15.16:0/729187022 >> 172.21.15.16:0/729187022 conn(0x7f1d64079510 msgr2=0x7f1d64079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:43.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.920+0000 7f1d6b01d640 1 -- 172.21.15.16:0/729187022 shutdown_connections 2024-08-28T19:18:43.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.920+0000 7f1d6b01d640 1 -- 172.21.15.16:0/729187022 wait complete. 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d6b01d640 1 Processor -- start 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d6b01d640 1 -- start start 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d6b01d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d64102880 0x7f1d6419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d6b01d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d641081c0 0x7f1d6419b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d6b01d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d64108b10 0x7f1d6419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d6b01d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1d641127a0 con 0x7f1d64102880 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d6b01d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d64112620 con 0x7f1d64108b10 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d6b01d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1d64112920 con 0x7f1d641081c0 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d68db9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d64102880 0x7f1d6419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d68db9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d64102880 0x7f1d6419b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57114/0 (socket says 172.21.15.16:57114) 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d68db9640 1 -- 172.21.15.16:0/2550214439 learned_addr learned my addr 172.21.15.16:0/2550214439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d695ba640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d64108b10 0x7f1d6419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d63fff640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d641081c0 0x7f1d6419b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d68db9640 1 -- 172.21.15.16:0/2550214439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d641081c0 msgr2=0x7f1d6419b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d68db9640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d641081c0 0x7f1d6419b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d68db9640 1 -- 172.21.15.16:0/2550214439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d64108b10 msgr2=0x7f1d6419bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d68db9640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d64108b10 0x7f1d6419bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d68db9640 1 -- 172.21.15.16:0/2550214439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d641a6390 con 0x7f1d64102880 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d695ba640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d64108b10 0x7f1d6419bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.921+0000 7f1d63fff640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d641081c0 0x7f1d6419b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.922+0000 7f1d68db9640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d64102880 0x7f1d6419b0d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400d950 tx=0x7f1d5400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.922+0000 7f1d61ffb640 1 -- 172.21.15.16:0/2550214439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d54017070 con 0x7f1d64102880 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.922+0000 7f1d6b01d640 1 -- 172.21.15.16:0/2550214439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d641a6680 con 0x7f1d64102880 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.922+0000 7f1d61ffb640 1 -- 172.21.15.16:0/2550214439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d54009c30 con 0x7f1d64102880 2024-08-28T19:18:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.922+0000 7f1d61ffb640 1 -- 172.21.15.16:0/2550214439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5400f620 con 0x7f1d64102880 2024-08-28T19:18:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.922+0000 7f1d6b01d640 1 -- 172.21.15.16:0/2550214439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d641a6bc0 con 0x7f1d64102880 2024-08-28T19:18:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.923+0000 7f1d6b01d640 1 -- 172.21.15.16:0/2550214439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d64103b00 con 0x7f1d64102880 2024-08-28T19:18:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.925+0000 7f1d61ffb640 1 -- 172.21.15.16:0/2550214439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f1d54010460 con 0x7f1d64102880 2024-08-28T19:18:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.926+0000 7f1d61ffb640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d500b4b80 0x7f1d500b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.926+0000 7f1d63fff640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d500b4b80 0x7f1d500b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.926+0000 7f1d61ffb640 1 -- 172.21.15.16:0/2550214439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7f1d540e51f0 con 0x7f1d64102880 2024-08-28T19:18:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.927+0000 7f1d63fff640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d500b4b80 0x7f1d500b7090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c000c00 tx=0x7f1d4c031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:43.929+0000 7f1d61ffb640 1 -- 172.21.15.16:0/2550214439 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f1d540abfb0 con 0x7f1d64102880 2024-08-28T19:18:44.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.151+0000 7f1d6b01d640 1 -- 172.21.15.16:0/2550214439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d6406bd30 con 0x7f1d64102880 2024-08-28T19:18:44.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.151+0000 7f1d61ffb640 1 -- 172.21.15.16:0/2550214439 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11842 (secure 0 0 0) 0x7f1d6406bd30 con 0x7f1d64102880 2024-08-28T19:18:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:44.153 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"btime":"2024-08-28T19:18:42:519149+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14817,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24583,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24590,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24605,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24619,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24635,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24649,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:18:42.223206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24665,"qdb_cluster":[24665]},"id":1}]} 2024-08-28T19:18:44.153 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 17 2024-08-28T19:18:44.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 -- 172.21.15.16:0/2550214439 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d500b4b80 msgr2=0x7f1d500b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:44.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d500b4b80 0x7f1d500b7090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c000c00 tx=0x7f1d4c031400 comp rx=0 tx=0).stop 2024-08-28T19:18:44.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 -- 172.21.15.16:0/2550214439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d64102880 msgr2=0x7f1d6419b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:44.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d64102880 0x7f1d6419b0d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400d950 tx=0x7f1d5400de10 comp rx=0 tx=0).stop 2024-08-28T19:18:44.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 -- 172.21.15.16:0/2550214439 shutdown_connections 2024-08-28T19:18:44.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d500b4b80 0x7f1d500b7090 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:44.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d64108b10 0x7f1d6419bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:44.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d641081c0 0x7f1d6419b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:44.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 --2- 172.21.15.16:0/2550214439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d64102880 0x7f1d6419b0d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:44.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 -- 172.21.15.16:0/2550214439 >> 172.21.15.16:0/2550214439 conn(0x7f1d64079510 msgr2=0x7f1d64106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:44.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 -- 172.21.15.16:0/2550214439 shutdown_connections 2024-08-28T19:18:44.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:44.155+0000 7f1d6b01d640 1 -- 172.21.15.16:0/2550214439 wait complete. 2024-08-28T19:18:44.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:44 smithi016 ceph-mon[18556]: pgmap v266: 129 pgs: 13 creating+peering, 116 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:44.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:44 smithi016 ceph-mon[18556]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-28T19:18:44.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:44 smithi016 ceph-mon[18556]: Cluster is now healthy 2024-08-28T19:18:44.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:44 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2550214439' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:44 smithi028 ceph-mon[21769]: pgmap v266: 129 pgs: 13 creating+peering, 116 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:44 smithi028 ceph-mon[21769]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-28T19:18:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:44 smithi028 ceph-mon[21769]: Cluster is now healthy 2024-08-28T19:18:44.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:44 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2550214439' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:44.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:18:44.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:44 smithi115 ceph-mon[22184]: pgmap v266: 129 pgs: 13 creating+peering, 116 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:44 smithi115 ceph-mon[22184]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-28T19:18:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:44 smithi115 ceph-mon[22184]: Cluster is now healthy 2024-08-28T19:18:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:44 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2550214439' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.621+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/3981588905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88102880 msgr2=0x7f1a88102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.621+0000 7f1a8d3dd640 1 --2- 172.21.15.16:0/3981588905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88102880 0x7f1a88102c90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009900 tx=0x7f1a7802a410 comp rx=0 tx=0).stop 2024-08-28T19:18:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.621+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/3981588905 shutdown_connections 2024-08-28T19:18:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.621+0000 7f1a8d3dd640 1 --2- 172.21.15.16:0/3981588905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88108ac0 0x7f1a8810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.621+0000 7f1a8d3dd640 1 --2- 172.21.15.16:0/3981588905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1a88108170 0x7f1a88108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.621+0000 7f1a8d3dd640 1 --2- 172.21.15.16:0/3981588905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88102880 0x7f1a88102c90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.621+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/3981588905 >> 172.21.15.16:0/3981588905 conn(0x7f1a88079500 msgr2=0x7f1a88079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.621+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/3981588905 shutdown_connections 2024-08-28T19:18:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.621+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/3981588905 wait complete. 2024-08-28T19:18:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a8d3dd640 1 Processor -- start 2024-08-28T19:18:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a8d3dd640 1 -- start start 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a8d3dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88102880 0x7f1a8810fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a8d3dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88108170 0x7f1a881102c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a8d3dd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1a88108ac0 0x7f1a88110800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a8d3dd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1a88113850 con 0x7f1a88108170 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a8d3dd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a881136d0 con 0x7f1a88102880 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a8d3dd640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1a881139d0 con 0x7f1a88108ac0 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88108170 0x7f1a881102c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88108170 0x7f1a881102c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54654/0 (socket says 172.21.15.16:54654) 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a867fc640 1 -- 172.21.15.16:0/164528848 learned_addr learned my addr 172.21.15.16:0/164528848 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a877fe640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1a88108ac0 0x7f1a88110800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.622+0000 7f1a86ffd640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88102880 0x7f1a8810fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a867fc640 1 -- 172.21.15.16:0/164528848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1a88108ac0 msgr2=0x7f1a88110800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a867fc640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1a88108ac0 0x7f1a88110800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a867fc640 1 -- 172.21.15.16:0/164528848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88102880 msgr2=0x7f1a8810fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a867fc640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88102880 0x7f1a8810fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a867fc640 1 -- 172.21.15.16:0/164528848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a881a8560 con 0x7f1a88108170 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a86ffd640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88102880 0x7f1a8810fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a867fc640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88108170 0x7f1a881102c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c00c970 tx=0x7f1a7c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a6ffff640 1 -- 172.21.15.16:0/164528848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c007bb0 con 0x7f1a88108170 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/164528848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a881a8850 con 0x7f1a88108170 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a6ffff640 1 -- 172.21.15.16:0/164528848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a7c007d50 con 0x7f1a88108170 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a6ffff640 1 -- 172.21.15.16:0/164528848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c00f450 con 0x7f1a88108170 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.623+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/164528848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a881a8d90 con 0x7f1a88108170 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.624+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/164528848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a88103ab0 con 0x7f1a88108170 2024-08-28T19:18:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.626+0000 7f1a6ffff640 1 -- 172.21.15.16:0/164528848 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f1a7c004f20 con 0x7f1a88108170 2024-08-28T19:18:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.626+0000 7f1a6ffff640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1a700b4b80 0x7f1a700b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.626+0000 7f1a86ffd640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1a700b4b80 0x7f1a700b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.626+0000 7f1a6ffff640 1 -- 172.21.15.16:0/164528848 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7f1a7c0e5500 con 0x7f1a88108170 2024-08-28T19:18:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.627+0000 7f1a86ffd640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1a700b4b80 0x7f1a700b7090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1a78000c00 tx=0x7f1a78009680 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.630+0000 7f1a6ffff640 1 -- 172.21.15.16:0/164528848 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f1a7c014030 con 0x7f1a88108170 2024-08-28T19:18:45.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.816+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/164528848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a88102c90 con 0x7f1a88108170 2024-08-28T19:18:45.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.817+0000 7f1a6ffff640 1 -- 172.21.15.16:0/164528848 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+24493 (secure 0 0 0) 0x7f1a88102c90 con 0x7f1a88108170 2024-08-28T19:18:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:45.820 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:18:42.223057+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:18:45.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/164528848 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1a700b4b80 msgr2=0x7f1a700b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:45.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1a700b4b80 0x7f1a700b7090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1a78000c00 tx=0x7f1a78009680 comp rx=0 tx=0).stop 2024-08-28T19:18:45.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/164528848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88108170 msgr2=0x7f1a881102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:45.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88108170 0x7f1a881102c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c00c970 tx=0x7f1a7c00ce30 comp rx=0 tx=0).stop 2024-08-28T19:18:45.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/164528848 shutdown_connections 2024-08-28T19:18:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1a88108ac0 0x7f1a88110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88108170 0x7f1a881102c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1a700b4b80 0x7f1a700b7090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 --2- 172.21.15.16:0/164528848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88102880 0x7f1a8810fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/164528848 >> 172.21.15.16:0/164528848 conn(0x7f1a88079500 msgr2=0x7f1a88101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/164528848 shutdown_connections 2024-08-28T19:18:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:45.820+0000 7f1a8d3dd640 1 -- 172.21.15.16:0/164528848 wait complete. 2024-08-28T19:18:46.487 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:18:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:46.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:46 smithi016 ceph-mon[18556]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:46.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:46 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/164528848' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:18:46.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:46 smithi028 ceph-mon[21769]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:46.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:46 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/164528848' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:18:46.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:46 smithi115 ceph-mon[22184]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:46.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:46 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/164528848' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:18:47.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.223+0000 7fa0c81c8640 1 -- 172.21.15.16:0/123852133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c0108140 msgr2=0x7fa0c0108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:47.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.223+0000 7fa0c81c8640 1 --2- 172.21.15.16:0/123852133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c0108140 0x7fa0c0108550 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4009900 tx=0x7fa0b402a410 comp rx=0 tx=0).stop 2024-08-28T19:18:47.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.223+0000 7fa0c81c8640 1 -- 172.21.15.16:0/123852133 shutdown_connections 2024-08-28T19:18:47.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.223+0000 7fa0c81c8640 1 --2- 172.21.15.16:0/123852133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c0108a90 0x7fa0c010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:47.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.223+0000 7fa0c81c8640 1 --2- 172.21.15.16:0/123852133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c0108140 0x7fa0c0108550 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:47.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.223+0000 7fa0c81c8640 1 --2- 172.21.15.16:0/123852133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c0102800 0x7fa0c0102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:47.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.223+0000 7fa0c81c8640 1 -- 172.21.15.16:0/123852133 >> 172.21.15.16:0/123852133 conn(0x7fa0c00794d0 msgr2=0x7fa0c00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:47.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.223+0000 7fa0c81c8640 1 -- 172.21.15.16:0/123852133 shutdown_connections 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.223+0000 7fa0c81c8640 1 -- 172.21.15.16:0/123852133 wait complete. 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.224+0000 7fa0c81c8640 1 Processor -- start 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.224+0000 7fa0c81c8640 1 -- start start 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c81c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c0102800 0x7fa0c019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c81c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c0108140 0x7fa0c019a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c81c8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c0108a90 0x7fa0c019fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c81c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c0113820 con 0x7fa0c0102800 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c81c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c01136a0 con 0x7fa0c0108140 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c81c8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c01139a0 con 0x7fa0c0108a90 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c5f64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c0102800 0x7fa0c019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c5f64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c0102800 0x7fa0c019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54670/0 (socket says 172.21.15.16:54670) 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c5f64640 1 -- 172.21.15.16:0/749034702 learned_addr learned my addr 172.21.15.16:0/749034702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c5763640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c0108140 0x7fa0c019a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c6765640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c0108a90 0x7fa0c019fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c5f64640 1 -- 172.21.15.16:0/749034702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c0108a90 msgr2=0x7fa0c019fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c5f64640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c0108a90 0x7fa0c019fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c5f64640 1 -- 172.21.15.16:0/749034702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c0108140 msgr2=0x7fa0c019a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c5f64640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c0108140 0x7fa0c019a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c5f64640 1 -- 172.21.15.16:0/749034702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c01a00c0 con 0x7fa0c0102800 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c5763640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c0108140 0x7fa0c019a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0c5f64640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c0102800 0x7fa0c019a210 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00e9f0 tx=0x7fa0bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.225+0000 7fa0b2ffd640 1 -- 172.21.15.16:0/749034702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc00cd80 con 0x7fa0c0102800 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.226+0000 7fa0c81c8640 1 -- 172.21.15.16:0/749034702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c01a03b0 con 0x7fa0c0102800 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.226+0000 7fa0b2ffd640 1 -- 172.21.15.16:0/749034702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0bc0108e0 con 0x7fa0c0102800 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.226+0000 7fa0b2ffd640 1 -- 172.21.15.16:0/749034702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc018980 con 0x7fa0c0102800 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.226+0000 7fa0c81c8640 1 -- 172.21.15.16:0/749034702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c01a6bd0 con 0x7fa0c0102800 2024-08-28T19:18:47.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.227+0000 7fa0c81c8640 1 -- 172.21.15.16:0/749034702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c0102c10 con 0x7fa0c0102800 2024-08-28T19:18:47.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.228+0000 7fa0b2ffd640 1 -- 172.21.15.16:0/749034702 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fa0bc010430 con 0x7fa0c0102800 2024-08-28T19:18:47.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.229+0000 7fa0b2ffd640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0ac0b4bb0 0x7fa0ac0b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:47.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.229+0000 7fa0c5763640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0ac0b4bb0 0x7fa0ac0b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:47.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.229+0000 7fa0b2ffd640 1 -- 172.21.15.16:0/749034702 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7fa0bc0e5da0 con 0x7fa0c0102800 2024-08-28T19:18:47.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.229+0000 7fa0c5763640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0ac0b4bb0 0x7fa0ac0b70c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4000c00 tx=0x7fa0b4032da0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:47.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.232+0000 7fa0b2ffd640 1 -- 172.21.15.16:0/749034702 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fa0bc0acbc0 con 0x7fa0c0102800 2024-08-28T19:18:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.420+0000 7fa0c81c8640 1 -- 172.21.15.16:0/749034702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0c019b340 con 0x7fa0c0102800 2024-08-28T19:18:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.421+0000 7fa0b2ffd640 1 -- 172.21.15.16:0/749034702 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+24493 (secure 0 0 0) 0x7fa0bc0b5100 con 0x7fa0c0102800 2024-08-28T19:18:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:47.423 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:18:42.223057+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:18:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 -- 172.21.15.16:0/749034702 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0ac0b4bb0 msgr2=0x7fa0ac0b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0ac0b4bb0 0x7fa0ac0b70c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4000c00 tx=0x7fa0b4032da0 comp rx=0 tx=0).stop 2024-08-28T19:18:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 -- 172.21.15.16:0/749034702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c0102800 msgr2=0x7fa0c019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c0102800 0x7fa0c019a210 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00e9f0 tx=0x7fa0bc00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:18:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 -- 172.21.15.16:0/749034702 shutdown_connections 2024-08-28T19:18:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c0108a90 0x7fa0c019fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0ac0b4bb0 0x7fa0ac0b70c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c0108140 0x7fa0c019a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 --2- 172.21.15.16:0/749034702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c0102800 0x7fa0c019a210 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 -- 172.21.15.16:0/749034702 >> 172.21.15.16:0/749034702 conn(0x7fa0c00794d0 msgr2=0x7fa0c0104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 -- 172.21.15.16:0/749034702 shutdown_connections 2024-08-28T19:18:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:47.424+0000 7fa0c81c8640 1 -- 172.21.15.16:0/749034702 wait complete. 2024-08-28T19:18:47.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:47 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/749034702' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:18:47.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:47 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/749034702' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:18:47.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:47 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/749034702' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:18:48.086 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:18:48.339 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:48.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:48 smithi016 ceph-mon[18556]: pgmap v268: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:48.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:48 smithi028 ceph-mon[21769]: pgmap v268: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:48 smithi115 ceph-mon[22184]: pgmap v268: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.881+0000 7f860371a640 1 -- 172.21.15.16:0/1192622751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85fc109b90 msgr2=0x7f85fc1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.881+0000 7f860371a640 1 --2- 172.21.15.16:0/1192622751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85fc109b90 0x7f85fc1103c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009ca0 tx=0x7f85f402a8e0 comp rx=0 tx=0).stop 2024-08-28T19:18:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.881+0000 7f860371a640 1 -- 172.21.15.16:0/1192622751 shutdown_connections 2024-08-28T19:18:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.881+0000 7f860371a640 1 --2- 172.21.15.16:0/1192622751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85fc109b90 0x7f85fc1103c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.881+0000 7f860371a640 1 --2- 172.21.15.16:0/1192622751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85fc109240 0x7f85fc109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.881+0000 7f860371a640 1 --2- 172.21.15.16:0/1192622751 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85fc103900 0x7f85fc103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.881+0000 7f860371a640 1 -- 172.21.15.16:0/1192622751 >> 172.21.15.16:0/1192622751 conn(0x7f85fc07b5f0 msgr2=0x7f85fc07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.881+0000 7f860371a640 1 -- 172.21.15.16:0/1192622751 shutdown_connections 2024-08-28T19:18:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.881+0000 7f860371a640 1 -- 172.21.15.16:0/1192622751 wait complete. 2024-08-28T19:18:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f860371a640 1 Processor -- start 2024-08-28T19:18:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f860371a640 1 -- start start 2024-08-28T19:18:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f860371a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85fc103900 0x7f85fc0754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f860371a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85fc109240 0x7f85fc0759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f860371a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85fc109b90 0x7f85fc075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f860371a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc113910 con 0x7f85fc109b90 2024-08-28T19:18:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f860371a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc113790 con 0x7f85fc109240 2024-08-28T19:18:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f860371a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc113a90 con 0x7f85fc103900 2024-08-28T19:18:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f8601cb7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85fc109b90 0x7f85fc075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f8601cb7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85fc109b90 0x7f85fc075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54684/0 (socket says 172.21.15.16:54684) 2024-08-28T19:18:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f8601cb7640 1 -- 172.21.15.16:0/1106536575 learned_addr learned my addr 172.21.15.16:0/1106536575 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.882+0000 7f8600cb5640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85fc109240 0x7f85fc0759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f86014b6640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85fc103900 0x7f85fc0754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f8601cb7640 1 -- 172.21.15.16:0/1106536575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85fc103900 msgr2=0x7f85fc0754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f8601cb7640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85fc103900 0x7f85fc0754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f8601cb7640 1 -- 172.21.15.16:0/1106536575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85fc109240 msgr2=0x7f85fc0759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f8601cb7640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85fc109240 0x7f85fc0759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f8601cb7640 1 -- 172.21.15.16:0/1106536575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85fc1a8620 con 0x7f85fc109b90 2024-08-28T19:18:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f8601cb7640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85fc109b90 0x7f85fc075f30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f85f4005cc0 tx=0x7f85f4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f85f27fc640 1 -- 172.21.15.16:0/1106536575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4004040 con 0x7f85fc109b90 2024-08-28T19:18:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f85f27fc640 1 -- 172.21.15.16:0/1106536575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f4004690 con 0x7f85fc109b90 2024-08-28T19:18:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f860371a640 1 -- 172.21.15.16:0/1106536575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85fc1a88b0 con 0x7f85fc109b90 2024-08-28T19:18:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f85f27fc640 1 -- 172.21.15.16:0/1106536575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4030d40 con 0x7f85fc109b90 2024-08-28T19:18:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.883+0000 7f860371a640 1 -- 172.21.15.16:0/1106536575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85fc1a8ec0 con 0x7f85fc109b90 2024-08-28T19:18:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.884+0000 7f860371a640 1 -- 172.21.15.16:0/1106536575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85fc103d90 con 0x7f85fc109b90 2024-08-28T19:18:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.886+0000 7f85f27fc640 1 -- 172.21.15.16:0/1106536575 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f85f40041e0 con 0x7f85fc109b90 2024-08-28T19:18:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.886+0000 7f85f27fc640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85e80b4c00 0x7f85e80b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.887+0000 7f86014b6640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85e80b4c00 0x7f85e80b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.887+0000 7f85f27fc640 1 -- 172.21.15.16:0/1106536575 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7f85f4106920 con 0x7f85fc109b90 2024-08-28T19:18:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.887+0000 7f86014b6640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85e80b4c00 0x7f85e80b7110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f85fc076f00 tx=0x7f85f8009110 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:48.890+0000 7f85f27fc640 1 -- 172.21.15.16:0/1106536575 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f85f40cd6e0 con 0x7f85fc109b90 2024-08-28T19:18:49.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.111+0000 7f860371a640 1 -- 172.21.15.16:0/1106536575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85fc0767b0 con 0x7f85fc109b90 2024-08-28T19:18:49.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.112+0000 7f85f27fc640 1 -- 172.21.15.16:0/1106536575 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11842 (secure 0 0 0) 0x7f85fc0767b0 con 0x7f85fc109b90 2024-08-28T19:18:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:49.114 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"btime":"2024-08-28T19:18:42:519149+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14817,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24583,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24590,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24605,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24619,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24635,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24649,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:18:42.223206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24665,"qdb_cluster":[24665]},"id":1}]} 2024-08-28T19:18:49.115 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 17 2024-08-28T19:18:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 -- 172.21.15.16:0/1106536575 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85e80b4c00 msgr2=0x7f85e80b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85e80b4c00 0x7f85e80b7110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f85fc076f00 tx=0x7f85f8009110 comp rx=0 tx=0).stop 2024-08-28T19:18:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 -- 172.21.15.16:0/1106536575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85fc109b90 msgr2=0x7f85fc075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85fc109b90 0x7f85fc075f30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f85f4005cc0 tx=0x7f85f4003ba0 comp rx=0 tx=0).stop 2024-08-28T19:18:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 -- 172.21.15.16:0/1106536575 shutdown_connections 2024-08-28T19:18:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85fc109b90 0x7f85fc075f30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85e80b4c00 0x7f85e80b7110 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85fc109240 0x7f85fc0759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 --2- 172.21.15.16:0/1106536575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85fc103900 0x7f85fc0754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 -- 172.21.15.16:0/1106536575 >> 172.21.15.16:0/1106536575 conn(0x7f85fc07b5f0 msgr2=0x7f85fc1072f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 -- 172.21.15.16:0/1106536575 shutdown_connections 2024-08-28T19:18:49.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:49.114+0000 7f860371a640 1 -- 172.21.15.16:0/1106536575 wait complete. 2024-08-28T19:18:49.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:49 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1106536575' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:49.728 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:18:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:49 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1106536575' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:49 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1106536575' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:18:49.980 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.467+0000 7f760f21e640 1 -- 172.21.15.16:0/3698789084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7608102880 msgr2=0x7f7608102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.467+0000 7f760f21e640 1 --2- 172.21.15.16:0/3698789084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7608102880 0x7f7608102c90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f75f8009900 tx=0x7f75f802a410 comp rx=0 tx=0).stop 2024-08-28T19:18:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.467+0000 7f760f21e640 1 -- 172.21.15.16:0/3698789084 shutdown_connections 2024-08-28T19:18:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.467+0000 7f760f21e640 1 --2- 172.21.15.16:0/3698789084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7608108ac0 0x7f760810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.467+0000 7f760f21e640 1 --2- 172.21.15.16:0/3698789084 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7608108170 0x7f7608108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.467+0000 7f760f21e640 1 --2- 172.21.15.16:0/3698789084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7608102880 0x7f7608102c90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.467+0000 7f760f21e640 1 -- 172.21.15.16:0/3698789084 >> 172.21.15.16:0/3698789084 conn(0x7f7608079500 msgr2=0x7f7608079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.467+0000 7f760f21e640 1 -- 172.21.15.16:0/3698789084 shutdown_connections 2024-08-28T19:18:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.467+0000 7f760f21e640 1 -- 172.21.15.16:0/3698789084 wait complete. 2024-08-28T19:18:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.467+0000 7f760f21e640 1 Processor -- start 2024-08-28T19:18:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.467+0000 7f760f21e640 1 -- start start 2024-08-28T19:18:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760f21e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7608102880 0x7f760819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760f21e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7608108170 0x7f760819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760f21e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7608108ac0 0x7f760819fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760f21e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7608113850 con 0x7f7608102880 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760f21e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76081136d0 con 0x7f7608108ac0 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760f21e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f76081139d0 con 0x7f7608108170 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760cfba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7608102880 0x7f760819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760cfba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7608102880 0x7f760819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54692/0 (socket says 172.21.15.16:54692) 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760cfba640 1 -- 172.21.15.16:0/2749800958 learned_addr learned my addr 172.21.15.16:0/2749800958 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760d7bb640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7608108ac0 0x7f760819fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f7607fff640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7608108170 0x7f760819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760cfba640 1 -- 172.21.15.16:0/2749800958 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7608108170 msgr2=0x7f760819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760cfba640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7608108170 0x7f760819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760cfba640 1 -- 172.21.15.16:0/2749800958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7608108ac0 msgr2=0x7f760819fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760cfba640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7608108ac0 0x7f760819fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760cfba640 1 -- 172.21.15.16:0/2749800958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76081a0170 con 0x7f7608102880 2024-08-28T19:18:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.468+0000 7f760cfba640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7608102880 0x7f760819a200 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f75f8005940 tx=0x7f75f80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:50.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.469+0000 7f7605ffb640 1 -- 172.21.15.16:0/2749800958 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f803f030 con 0x7f7608102880 2024-08-28T19:18:50.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.469+0000 7f7605ffb640 1 -- 172.21.15.16:0/2749800958 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75f8003cb0 con 0x7f7608102880 2024-08-28T19:18:50.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.469+0000 7f760f21e640 1 -- 172.21.15.16:0/2749800958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76081a0400 con 0x7f7608102880 2024-08-28T19:18:50.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.469+0000 7f7605ffb640 1 -- 172.21.15.16:0/2749800958 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f8035070 con 0x7f7608102880 2024-08-28T19:18:50.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.469+0000 7f760f21e640 1 -- 172.21.15.16:0/2749800958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76081a6bc0 con 0x7f7608102880 2024-08-28T19:18:50.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.470+0000 7f760f21e640 1 -- 172.21.15.16:0/2749800958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7608103ab0 con 0x7f7608102880 2024-08-28T19:18:50.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.472+0000 7f7605ffb640 1 -- 172.21.15.16:0/2749800958 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f75f8042030 con 0x7f7608102880 2024-08-28T19:18:50.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.473+0000 7f7605ffb640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f75f00b4b80 0x7f75f00b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:50.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.473+0000 7f7607fff640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f75f00b4b80 0x7f75f00b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:50.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.473+0000 7f7605ffb640 1 -- 172.21.15.16:0/2749800958 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7f75f8105fe0 con 0x7f7608102880 2024-08-28T19:18:50.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.473+0000 7f7607fff640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f75f00b4b80 0x7f75f00b7090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f75fc005fd0 tx=0x7f75fc005850 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:50.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.476+0000 7f7605ffb640 1 -- 172.21.15.16:0/2749800958 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f75f80ccdc0 con 0x7f7608102880 2024-08-28T19:18:50.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.675+0000 7f760f21e640 1 -- 172.21.15.16:0/2749800958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f760819b330 con 0x7f7608102880 2024-08-28T19:18:50.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:50 smithi016 ceph-mon[18556]: pgmap v269: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:50.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.677+0000 7f7605ffb640 1 -- 172.21.15.16:0/2749800958 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+24493 (secure 0 0 0) 0x7f760819b330 con 0x7f7608102880 2024-08-28T19:18:50.679 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:50.679 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:18:42.223057+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:18:50.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 -- 172.21.15.16:0/2749800958 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f75f00b4b80 msgr2=0x7f75f00b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:50.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f75f00b4b80 0x7f75f00b7090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f75fc005fd0 tx=0x7f75fc005850 comp rx=0 tx=0).stop 2024-08-28T19:18:50.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 -- 172.21.15.16:0/2749800958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7608102880 msgr2=0x7f760819a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:50.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7608102880 0x7f760819a200 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f75f8005940 tx=0x7f75f80042f0 comp rx=0 tx=0).stop 2024-08-28T19:18:50.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 -- 172.21.15.16:0/2749800958 shutdown_connections 2024-08-28T19:18:50.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f75f00b4b80 0x7f75f00b7090 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:50.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7608108ac0 0x7f760819fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:50.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7608108170 0x7f760819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:50.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 --2- 172.21.15.16:0/2749800958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7608102880 0x7f760819a200 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:50.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 -- 172.21.15.16:0/2749800958 >> 172.21.15.16:0/2749800958 conn(0x7f7608079500 msgr2=0x7f7608101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:50.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 -- 172.21.15.16:0/2749800958 shutdown_connections 2024-08-28T19:18:50.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:50.679+0000 7f760f21e640 1 -- 172.21.15.16:0/2749800958 wait complete. 2024-08-28T19:18:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:50 smithi028 ceph-mon[21769]: pgmap v269: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:50 smithi115 ceph-mon[22184]: pgmap v269: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:51.400 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-28T19:18:33.178088+0000', 'modified': '2024-08-28T19:18:42.223206+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24665}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24665': {'gid': 24665, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6839/2518320595', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 2518320595}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 2518320595}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24665, 'qdb_cluster': [24665]} 2024-08-28T19:18:51.400 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-28T19:18:51.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.b status 2024-08-28T19:18:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:51.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:51 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2749800958' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:18:51.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:51 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2749800958' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:18:51.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:51 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2749800958' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:18:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.236+0000 7f746521f640 1 -- 172.21.15.16:0/2386589512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74601081f0 msgr2=0x7f7460108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.236+0000 7f746521f640 1 --2- 172.21.15.16:0/2386589512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74601081f0 0x7f7460108600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7448009900 tx=0x7f744802a410 comp rx=0 tx=0).stop 2024-08-28T19:18:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.236+0000 7f746521f640 1 -- 172.21.15.16:0/2386589512 shutdown_connections 2024-08-28T19:18:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.236+0000 7f746521f640 1 --2- 172.21.15.16:0/2386589512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7460108b40 0x7f746010f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.236+0000 7f746521f640 1 --2- 172.21.15.16:0/2386589512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74601081f0 0x7f7460108600 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.236+0000 7f746521f640 1 --2- 172.21.15.16:0/2386589512 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74601028b0 0x7f7460102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.236+0000 7f746521f640 1 -- 172.21.15.16:0/2386589512 >> 172.21.15.16:0/2386589512 conn(0x7f7460079470 msgr2=0x7f7460079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:52.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.236+0000 7f746521f640 1 -- 172.21.15.16:0/2386589512 shutdown_connections 2024-08-28T19:18:52.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.237+0000 7f746521f640 1 -- 172.21.15.16:0/2386589512 wait complete. 2024-08-28T19:18:52.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.237+0000 7f746521f640 1 Processor -- start 2024-08-28T19:18:52.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.237+0000 7f746521f640 1 -- start start 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f746521f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74601028b0 0x7f746019a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f746521f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74601081f0 0x7f746019a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f746521f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7460108b40 0x7f746019fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f746521f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f74601127c0 con 0x7f7460108b40 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f746521f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7460112640 con 0x7f74601081f0 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f746521f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7460112940 con 0x7f74601028b0 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7460108b40 0x7f746019fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7460108b40 0x7f746019fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54700/0 (socket says 172.21.15.16:54700) 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745f59e640 1 -- 172.21.15.16:0/1778046217 learned_addr learned my addr 172.21.15.16:0/1778046217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745ed9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74601028b0 0x7f746019a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f7457fff640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74601081f0 0x7f746019a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745f59e640 1 -- 172.21.15.16:0/1778046217 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74601028b0 msgr2=0x7f746019a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745f59e640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74601028b0 0x7f746019a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745f59e640 1 -- 172.21.15.16:0/1778046217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74601081f0 msgr2=0x7f746019a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745f59e640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74601081f0 0x7f746019a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745f59e640 1 -- 172.21.15.16:0/1778046217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74601a02b0 con 0x7f7460108b40 2024-08-28T19:18:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745ed9d640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74601028b0 0x7f746019a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:18:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745f59e640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7460108b40 0x7f746019fb80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f745800cad0 tx=0x7f745800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.238+0000 7f745cd99640 1 -- 172.21.15.16:0/1778046217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745800ae20 con 0x7f7460108b40 2024-08-28T19:18:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.239+0000 7f746521f640 1 -- 172.21.15.16:0/1778046217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74601a05a0 con 0x7f7460108b40 2024-08-28T19:18:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.239+0000 7f745cd99640 1 -- 172.21.15.16:0/1778046217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7458005020 con 0x7f7460108b40 2024-08-28T19:18:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.239+0000 7f745cd99640 1 -- 172.21.15.16:0/1778046217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745800f6b0 con 0x7f7460108b40 2024-08-28T19:18:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.239+0000 7f746521f640 1 -- 172.21.15.16:0/1778046217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74601a6d00 con 0x7f7460108b40 2024-08-28T19:18:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.241+0000 7f746b794740 1 Processor -- start 2024-08-28T19:18:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.241+0000 7f745cd99640 1 -- 172.21.15.16:0/1778046217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f74580051d0 con 0x7f7460108b40 2024-08-28T19:18:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.241+0000 7f746b794740 1 -- start start 2024-08-28T19:18:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f746b794740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f746b794740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a59cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f746b794740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a5a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f746b794740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5651e19d03f0 con 0x5651e1693280 2024-08-28T19:18:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f746b794740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5651e19bf980 con 0x5651e16b3f60 2024-08-28T19:18:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f746b794740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5651e19c3930 con 0x5651e168b0a0 2024-08-28T19:18:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f7457fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a5a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f7457fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a5a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54714/0 (socket says 172.21.15.16:54714) 2024-08-28T19:18:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f7457fff640 1 -- 172.21.15.16:0/2666057064 learned_addr learned my addr 172.21.15.16:0/2666057064 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f745ed9d640 1 --2- 172.21.15.16:0/2666057064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a59cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f7457fff640 1 --2- 172.21.15.16:0/2666057064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f745ed9d640 1 -- 172.21.15.16:0/2666057064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 msgr2=0x5651e1a597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f745ed9d640 1 --2- 172.21.15.16:0/2666057064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f745ed9d640 1 -- 172.21.15.16:0/2666057064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 msgr2=0x5651e1a5a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f745ed9d640 1 --2- 172.21.15.16:0/2666057064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a5a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f745ed9d640 1 -- 172.21.15.16:0/2666057064 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651e171f540 con 0x5651e168b0a0 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f7457fff640 1 --2- 172.21.15.16:0/2666057064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a5a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.242+0000 7f745cd99640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f744c0b4b80 0x7f744c0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.243+0000 7f7457fff640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f744c0b4b80 0x7f744c0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.243+0000 7f745ed9d640 1 --2- 172.21.15.16:0/2666057064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a59cf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7450009fd0 tx=0x7f74500118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccc0910b39e5f0b1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.243+0000 7f745cd99640 1 -- 172.21.15.16:0/1778046217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7f74580e5430 con 0x7f7460108b40 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.243+0000 7f74557fa640 1 -- 172.21.15.16:0/2666057064 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450004310 con 0x5651e168b0a0 2024-08-28T19:18:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.243+0000 7f74557fa640 1 -- 172.21.15.16:0/2666057064 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74500044b0 con 0x5651e168b0a0 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.243+0000 7f74557fa640 1 -- 172.21.15.16:0/2666057064 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74500035f0 con 0x5651e168b0a0 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.244+0000 7f746b794740 1 -- 172.21.15.16:0/2666057064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 msgr2=0x5651e1a59cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.244+0000 7f7457fff640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f744c0b4b80 0x7f744c0b7090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f74480096c0 tx=0x7f7448000bc0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.244+0000 7f746b794740 1 --2- 172.21.15.16:0/2666057064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a59cf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7450009fd0 tx=0x7f74500118d0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.244+0000 7f746b794740 1 -- 172.21.15.16:0/2666057064 shutdown_connections 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.244+0000 7f746b794740 1 --2- 172.21.15.16:0/2666057064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a5a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.244+0000 7f746b794740 1 --2- 172.21.15.16:0/2666057064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a59cf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.244+0000 7f746b794740 1 --2- 172.21.15.16:0/2666057064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.244+0000 7f746b794740 1 -- 172.21.15.16:0/2666057064 >> 172.21.15.16:0/2666057064 conn(0x5651e16a8350 msgr2=0x5651e1a576a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.244+0000 7f746b794740 1 -- 172.21.15.16:0/2666057064 shutdown_connections 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.244+0000 7f746b794740 1 -- 172.21.15.16:0/2666057064 wait complete. 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.244+0000 7f746b794740 1 -- start start 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f746b794740 1 Processor -- start 2024-08-28T19:18:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f746b794740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a64dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f746b794740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a65300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f746b794740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a65860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f746b794740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5651e18225f0 con 0x5651e1693280 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f746b794740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5651e1946600 con 0x5651e16b3f60 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f746b794740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5651e1a67f90 con 0x5651e168b0a0 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f746b794740 10 client.0 _ll_register_callbacks cb 0x5651e196caf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f7457fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a65300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f7457fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a65300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54722/0 (socket says 172.21.15.16:54722) 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f7457fff640 1 -- 172.21.15.16:0/2957669064 learned_addr learned my addr 172.21.15.16:0/2957669064 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f745ed9d640 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a64dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f745f59e640 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a65860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f7457fff640 1 -- 172.21.15.16:0/2957669064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 msgr2=0x5651e1a64dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f7457fff640 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a64dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f7457fff640 1 -- 172.21.15.16:0/2957669064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 msgr2=0x5651e1a65860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f7457fff640 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a65860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f7457fff640 1 -- 172.21.15.16:0/2957669064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651e197f7b0 con 0x5651e1693280 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.245+0000 7f745f59e640 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a65860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.246+0000 7f7457fff640 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a65300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7448009230 tx=0x7f7448031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1eab5e8694b87 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.246+0000 7f74467fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.246+0000 7f74467fc640 1 -- 172.21.15.16:0/2957669064 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7448050070 con 0x5651e1693280 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.246+0000 7f74467fc640 1 -- 172.21.15.16:0/2957669064 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7f74480035f0 con 0x5651e1693280 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.246+0000 7f746b794740 1 -- 172.21.15.16:0/2957669064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74500044b0 con 0x5651e1693280 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.246+0000 7f74467fc640 1 -- 172.21.15.16:0/2957669064 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f744805d050 con 0x5651e1693280 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.246+0000 7f746b794740 10 client.14931 fetch_fsmap learned FSMap version 17 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.246+0000 7f746b794740 1 -- 172.21.15.16:0/2957669064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5651e1a73c40 con 0x5651e1693280 2024-08-28T19:18:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.246+0000 7f74467fc640 1 -- 172.21.15.16:0/2957669064 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74500044b0 con 0x5651e1693280 2024-08-28T19:18:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.246+0000 7f74467fc640 1 -- 172.21.15.16:0/2957669064 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f744803d9b0 con 0x5651e1693280 2024-08-28T19:18:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.246+0000 7f746b794740 10 client.14931 fetch_fsmap finished waiting for FSMap version 17 2024-08-28T19:18:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.247+0000 7f746b794740 10 client.14931 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-28T19:18:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.247+0000 7f746b794740 20 client.14931 populate_metadata read hostname 'smithi016' 2024-08-28T19:18:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.247+0000 7f746b794740 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5651e16728e0 0x5651e1a77680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.247+0000 7f746b794740 4 client.14931 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:18:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.247+0000 7f746b794740 1 -- 172.21.15.16:0/2957669064 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651e19d2340 con 0x5651e16728e0 2024-08-28T19:18:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.247+0000 7f745ed9d640 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5651e16728e0 0x5651e1a77680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.247+0000 7f745ed9d640 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5651e16728e0 0x5651e1a77680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=452e0b380f0e33bf server_cookie=83568d64c1619ed0 in_seq=0 out_seq=0 2024-08-28T19:18:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.247+0000 7f74467fc640 10 client.14931 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:18:52.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.248+0000 7f74467fc640 1 -- 172.21.15.16:0/2957669064 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5651e19d2340 con 0x5651e16728e0 2024-08-28T19:18:52.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.248+0000 7f74467fc640 10 client.14931 handle_command_reply: tid=0 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.248+0000 7f746b794740 1 client.14931 shutdown 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.248+0000 7f746b794740 1 -- 172.21.15.16:0/2957669064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 msgr2=0x5651e1a65300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.248+0000 7f746b794740 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a65300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7448009230 tx=0x7f7448031400 comp rx=0 tx=0).stop 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.248+0000 7f746b794740 1 -- 172.21.15.16:0/2957669064 shutdown_connections 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.248+0000 7f746b794740 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a65860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.248+0000 7f746b794740 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a65300 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.248+0000 7f746b794740 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5651e16728e0 0x5651e1a77680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.248+0000 7f746b794740 1 --2- 172.21.15.16:0/2957669064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a64dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.248+0000 7f746b794740 1 -- 172.21.15.16:0/2957669064 >> 172.21.15.16:0/2957669064 conn(0x5651e16a8350 msgr2=0x5651e1a5c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.249+0000 7f74467fc640 0 client.14931 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.249+0000 7f746b794740 1 -- 172.21.15.16:0/2957669064 shutdown_connections 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.249+0000 7f746b794740 1 -- 172.21.15.16:0/2957669064 wait complete. 2024-08-28T19:18:52.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.249+0000 7f746b794740 20 client.14931 trim_cache size 0 max 16384 2024-08-28T19:18:52.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.267+0000 7f746b794740 1 Processor -- start 2024-08-28T19:18:52.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.267+0000 7f746b794740 1 -- start start 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.267+0000 7f746b794740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e168b0a0 0x5651e1a63640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.267+0000 7f746b794740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e1693280 0x5651e1a63b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.267+0000 7f746b794740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a56cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.267+0000 7f746b794740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5651e1a759c0 con 0x5651e168b0a0 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.267+0000 7f746b794740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5651e1a76090 con 0x5651e16b3f60 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.267+0000 7f746b794740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5651e1a76760 con 0x5651e1693280 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.267+0000 7f745ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e168b0a0 0x5651e1a63640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.267+0000 7f745ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e168b0a0 0x5651e1a63640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54736/0 (socket says 172.21.15.16:54736) 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.267+0000 7f745ed9d640 1 -- 172.21.15.16:0/3781214252 learned_addr learned my addr 172.21.15.16:0/3781214252 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f745f59e640 1 --2- 172.21.15.16:0/3781214252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a56cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f7457fff640 1 --2- 172.21.15.16:0/3781214252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e1693280 0x5651e1a63b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f745ed9d640 1 -- 172.21.15.16:0/3781214252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e1693280 msgr2=0x5651e1a63b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:18:52.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f745ed9d640 1 --2- 172.21.15.16:0/3781214252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e1693280 0x5651e1a63b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f745ed9d640 1 -- 172.21.15.16:0/3781214252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 msgr2=0x5651e1a56cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:18:52.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f745ed9d640 1 --2- 172.21.15.16:0/3781214252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a56cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f745ed9d640 1 -- 172.21.15.16:0/3781214252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651e1a67500 con 0x5651e168b0a0 2024-08-28T19:18:52.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f745f59e640 1 --2- 172.21.15.16:0/3781214252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a56cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:18:52.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f745ed9d640 1 --2- 172.21.15.16:0/3781214252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e168b0a0 0x5651e1a63640 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f74500058c0 tx=0x7f7450005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ece6dc649e4de7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:52.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f7454ff9640 1 -- 172.21.15.16:0/3781214252 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450004230 con 0x5651e168b0a0 2024-08-28T19:18:52.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f7454ff9640 1 -- 172.21.15.16:0/3781214252 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74500043d0 con 0x5651e168b0a0 2024-08-28T19:18:52.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.268+0000 7f7454ff9640 1 -- 172.21.15.16:0/3781214252 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745000f740 con 0x5651e168b0a0 2024-08-28T19:18:52.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.269+0000 7f746b794740 1 -- 172.21.15.16:0/3781214252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e168b0a0 msgr2=0x5651e1a63640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:52.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.269+0000 7f746b794740 1 --2- 172.21.15.16:0/3781214252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e168b0a0 0x5651e1a63640 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f74500058c0 tx=0x7f7450005f90 comp rx=0 tx=0).stop 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.269+0000 7f746b794740 1 -- 172.21.15.16:0/3781214252 shutdown_connections 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.269+0000 7f746b794740 1 --2- 172.21.15.16:0/3781214252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a56cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.269+0000 7f746b794740 1 --2- 172.21.15.16:0/3781214252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e1693280 0x5651e1a63b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.269+0000 7f746b794740 1 --2- 172.21.15.16:0/3781214252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e168b0a0 0x5651e1a63640 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.269+0000 7f746b794740 1 -- 172.21.15.16:0/3781214252 >> 172.21.15.16:0/3781214252 conn(0x5651e16a8350 msgr2=0x5651e1a75100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.269+0000 7f746b794740 1 -- 172.21.15.16:0/3781214252 shutdown_connections 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.269+0000 7f746b794740 1 -- 172.21.15.16:0/3781214252 wait complete. 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.269+0000 7f746b794740 1 -- start start 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f746b794740 1 Processor -- start 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f746b794740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a7fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f746b794740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a80260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f746b794740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a79290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f746b794740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5651e1a77500 con 0x5651e1693280 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f746b794740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5651e1a76e30 con 0x5651e16b3f60 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f746b794740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5651e1a77bd0 con 0x5651e168b0a0 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f746b794740 10 client.0 _ll_register_callbacks cb 0x7f7448032ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f7457fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a80260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f7457fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a80260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54752/0 (socket says 172.21.15.16:54752) 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f7457fff640 1 -- 172.21.15.16:0/318880439 learned_addr learned my addr 172.21.15.16:0/318880439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f745ed9d640 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a7fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f745f59e640 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a79290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f7457fff640 1 -- 172.21.15.16:0/318880439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 msgr2=0x5651e1a7fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f7457fff640 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a7fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f7457fff640 1 -- 172.21.15.16:0/318880439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 msgr2=0x5651e1a79290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:18:52.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f7457fff640 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a79290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.270+0000 7f7457fff640 1 -- 172.21.15.16:0/318880439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651e1a79a70 con 0x5651e1693280 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f7457fff640 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a80260 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f744802a450 tx=0x7f74480045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f18d0eab017556 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f74467fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f74467fc640 1 -- 172.21.15.16:0/318880439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7448050070 con 0x5651e1693280 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f74467fc640 1 -- 172.21.15.16:0/318880439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7f7448005500 con 0x5651e1693280 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f746b794740 1 -- 172.21.15.16:0/318880439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74500043d0 con 0x5651e1693280 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f74467fc640 1 -- 172.21.15.16:0/318880439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f7448035040 con 0x5651e1693280 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f746b794740 10 client.14937 fetch_fsmap learned FSMap version 17 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f746b794740 1 -- 172.21.15.16:0/318880439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5651e1a7f380 con 0x5651e1693280 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f74467fc640 1 -- 172.21.15.16:0/318880439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74500043d0 con 0x5651e1693280 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f74467fc640 1 -- 172.21.15.16:0/318880439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f7448060030 con 0x5651e1693280 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f746b794740 10 client.14937 fetch_fsmap finished waiting for FSMap version 17 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f746b794740 10 client.14937 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-28T19:18:52.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f746b794740 20 client.14937 populate_metadata read hostname 'smithi016' 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f746b794740 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5651e1647c90 0x5651e1ae5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f746b794740 4 client.14937 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.271+0000 7f746b794740 1 -- 172.21.15.16:0/318880439 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "status"}) -- 0x5651e19cdde0 con 0x5651e1647c90 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.272+0000 7f745ed9d640 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5651e1647c90 0x5651e1ae5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.272+0000 7f74467fc640 10 client.14937 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.272+0000 7f745ed9d640 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5651e1647c90 0x5651e1ae5e10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d1ccd39275a2bd3 server_cookie=33f51e4e735da6c9 in_seq=0 out_seq=0 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.272+0000 7f74467fc640 1 -- 172.21.15.16:0/318880439 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5651e19cdde0 con 0x5651e1647c90 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.272+0000 7f74467fc640 10 client.14937 handle_command_reply: tid=0 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.272+0000 7f746b794740 1 client.14937 shutdown 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.272+0000 7f746b794740 1 -- 172.21.15.16:0/318880439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 msgr2=0x5651e1a80260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.272+0000 7f746b794740 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a80260 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f744802a450 tx=0x7f74480045e0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746b794740 1 -- 172.21.15.16:0/318880439 shutdown_connections 2024-08-28T19:18:52.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746b794740 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651e16b3f60 0x5651e1a79290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746b794740 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651e1693280 0x5651e1a80260 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746b794740 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5651e1647c90 0x5651e1ae5e10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746b794740 1 --2- 172.21.15.16:0/318880439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5651e168b0a0 0x5651e1a7fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746b794740 1 -- 172.21.15.16:0/318880439 >> 172.21.15.16:0/318880439 conn(0x5651e16a8350 msgr2=0x5651e1a63600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f74467fc640 0 client.14937 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:18:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746b794740 1 -- 172.21.15.16:0/318880439 shutdown_connections 2024-08-28T19:18:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746b794740 1 -- 172.21.15.16:0/318880439 wait complete. 2024-08-28T19:18:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746b794740 20 client.14937 trim_cache size 0 max 16384 2024-08-28T19:18:52.281 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:18:52.281 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:18:52.281 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-08-28T19:18:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24665, 2024-08-28T19:18:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:18:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:18:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:18:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 18.799475999999999, 2024-08-28T19:18:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 17, 2024-08-28T19:18:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-08-28T19:18:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 80, 2024-08-28T19:18:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 29.014513244 2024-08-28T19:18:52.282 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:18:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746521f640 1 -- 172.21.15.16:0/1778046217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f744c0b4b80 msgr2=0x7f744c0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746521f640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f744c0b4b80 0x7f744c0b7090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f74480096c0 tx=0x7f7448000bc0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746521f640 1 -- 172.21.15.16:0/1778046217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7460108b40 msgr2=0x7f746019fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746521f640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7460108b40 0x7f746019fb80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f745800cad0 tx=0x7f745800cf90 comp rx=0 tx=0).stop 2024-08-28T19:18:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746521f640 1 -- 172.21.15.16:0/1778046217 shutdown_connections 2024-08-28T19:18:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746521f640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7460108b40 0x7f746019fb80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746521f640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f744c0b4b80 0x7f744c0b7090 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746521f640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74601081f0 0x7f746019a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746521f640 1 --2- 172.21.15.16:0/1778046217 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74601028b0 0x7f746019a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:52.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.273+0000 7f746521f640 1 -- 172.21.15.16:0/1778046217 >> 172.21.15.16:0/1778046217 conn(0x7f7460079470 msgr2=0x7f746006bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:52.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.274+0000 7f746521f640 1 -- 172.21.15.16:0/1778046217 shutdown_connections 2024-08-28T19:18:52.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:52.274+0000 7f746521f640 1 -- 172.21.15.16:0/1778046217 wait complete. 2024-08-28T19:18:52.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:52 smithi028 ceph-mon[21769]: pgmap v270: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:52.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:52 smithi115 ceph-mon[22184]: pgmap v270: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:53.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:52 smithi016 ceph-mon[18556]: pgmap v270: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:53.049 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-08-28T19:18:53.049 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-08-28T19:18:53.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs ls 2024-08-28T19:18:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.793+0000 7ff815122640 1 -- 172.21.15.16:0/4268044670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff810102820 msgr2=0x7ff810102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.793+0000 7ff815122640 1 --2- 172.21.15.16:0/4268044670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff810102820 0x7ff810102c30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff800009900 tx=0x7ff80002a410 comp rx=0 tx=0).stop 2024-08-28T19:18:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.794+0000 7ff815122640 1 -- 172.21.15.16:0/4268044670 shutdown_connections 2024-08-28T19:18:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.794+0000 7ff815122640 1 --2- 172.21.15.16:0/4268044670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff810108ab0 0x7ff81010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.794+0000 7ff815122640 1 --2- 172.21.15.16:0/4268044670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff810108160 0x7ff810108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.794+0000 7ff815122640 1 --2- 172.21.15.16:0/4268044670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff810102820 0x7ff810102c30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.794+0000 7ff815122640 1 -- 172.21.15.16:0/4268044670 >> 172.21.15.16:0/4268044670 conn(0x7ff8100794e0 msgr2=0x7ff8100798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.794+0000 7ff815122640 1 -- 172.21.15.16:0/4268044670 shutdown_connections 2024-08-28T19:18:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.794+0000 7ff815122640 1 -- 172.21.15.16:0/4268044670 wait complete. 2024-08-28T19:18:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.794+0000 7ff815122640 1 Processor -- start 2024-08-28T19:18:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff815122640 1 -- start start 2024-08-28T19:18:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff815122640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff810102820 0x7ff810199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff815122640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff810108160 0x7ff810199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff815122640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff810108ab0 0x7ff81019e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff815122640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff810112730 con 0x7ff810108ab0 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff815122640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8101125b0 con 0x7ff810108160 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff815122640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff8101128b0 con 0x7ff810102820 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff80f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff810108ab0 0x7ff81019e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff80f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff810108ab0 0x7ff81019e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54766/0 (socket says 172.21.15.16:54766) 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff80f59e640 1 -- 172.21.15.16:0/4285782460 learned_addr learned my addr 172.21.15.16:0/4285782460 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff80ed9d640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff810102820 0x7ff810199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff80e59c640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff810108160 0x7ff810199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff80f59e640 1 -- 172.21.15.16:0/4285782460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff810102820 msgr2=0x7ff810199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff80f59e640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff810102820 0x7ff810199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff80f59e640 1 -- 172.21.15.16:0/4285782460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff810108160 msgr2=0x7ff810199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff80f59e640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff810108160 0x7ff810199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.795+0000 7ff80f59e640 1 -- 172.21.15.16:0/4285782460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81019f0f0 con 0x7ff810108ab0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.796+0000 7ff80f59e640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff810108ab0 0x7ff81019e9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff80800eaa0 tx=0x7ff80800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.796+0000 7ff7f7fff640 1 -- 172.21.15.16:0/4285782460 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80800cdb0 con 0x7ff810108ab0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.796+0000 7ff815122640 1 -- 172.21.15.16:0/4285782460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81019f3e0 con 0x7ff810108ab0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.796+0000 7ff7f7fff640 1 -- 172.21.15.16:0/4285782460 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff808005020 con 0x7ff810108ab0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.796+0000 7ff7f7fff640 1 -- 172.21.15.16:0/4285782460 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8080116d0 con 0x7ff810108ab0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.796+0000 7ff815122640 1 -- 172.21.15.16:0/4285782460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8101a5b40 con 0x7ff810108ab0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.798+0000 7ff815122640 1 -- 172.21.15.16:0/4285782460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff810103aa0 con 0x7ff810108ab0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.799+0000 7ff7f7fff640 1 -- 172.21.15.16:0/4285782460 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7ff8080051d0 con 0x7ff810108ab0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.799+0000 7ff7f7fff640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7f80b4b80 0x7ff7f80b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.799+0000 7ff80ed9d640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7f80b4b80 0x7ff7f80b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.799+0000 7ff7f7fff640 1 -- 172.21.15.16:0/4285782460 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7ff8080e55f0 con 0x7ff810108ab0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.800+0000 7ff80ed9d640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7f80b4b80 0x7ff7f80b7090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff800031440 tx=0x7ff800009560 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:53.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:53.804+0000 7ff7f7fff640 1 -- 172.21.15.16:0/4285782460 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ff808016030 con 0x7ff810108ab0 2024-08-28T19:18:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.026+0000 7ff815122640 1 -- 172.21.15.16:0/4285782460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff81010a2f0 con 0x7ff810108ab0 2024-08-28T19:18:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.027+0000 7ff7f7fff640 1 -- 172.21.15.16:0/4285782460 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7ff8080abdc0 con 0x7ff810108ab0 2024-08-28T19:18:54.029 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-08-28T19:18:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 -- 172.21.15.16:0/4285782460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7f80b4b80 msgr2=0x7ff7f80b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7f80b4b80 0x7ff7f80b7090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff800031440 tx=0x7ff800009560 comp rx=0 tx=0).stop 2024-08-28T19:18:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 -- 172.21.15.16:0/4285782460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff810108ab0 msgr2=0x7ff81019e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff810108ab0 0x7ff81019e9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff80800eaa0 tx=0x7ff80800ef60 comp rx=0 tx=0).stop 2024-08-28T19:18:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 -- 172.21.15.16:0/4285782460 shutdown_connections 2024-08-28T19:18:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff810108ab0 0x7ff81019e9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7f80b4b80 0x7ff7f80b7090 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff810108160 0x7ff810199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 --2- 172.21.15.16:0/4285782460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff810102820 0x7ff810199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 -- 172.21.15.16:0/4285782460 >> 172.21.15.16:0/4285782460 conn(0x7ff8100794e0 msgr2=0x7ff810101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 -- 172.21.15.16:0/4285782460 shutdown_connections 2024-08-28T19:18:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54.029+0000 7ff815122640 1 -- 172.21.15.16:0/4285782460 wait complete. 2024-08-28T19:18:54.804 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:54 smithi016 ceph-mon[18556]: pgmap v271: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:54.805 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:54 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/4285782460' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-08-28T19:18:54.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:54 smithi028 ceph-mon[21769]: pgmap v271: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:54.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:54 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/4285782460' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-08-28T19:18:54.835 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-28T19:18:54.835 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-28T19:18:54.835 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-08-28T19:18:54.835 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-08-28T19:18:54.836 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-08-28T19:18:54.836 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-28T19:18:54.836 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-08-28T19:18:54.836 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-28T19:18:54.836 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-08-28T19:18:54.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:18:54.836 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-08-28T19:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-28T19:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-28T19:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-08-28T19:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-08-28T19:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-08-28T19:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-08-28T19:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-28T19:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42261sec preferred_lft 42261sec 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::1876:14a0:73d4:8f28/64 scope link noprefixroute 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-28T19:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-08-28T19:18:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-08-28T19:18:54.856 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-08-28T19:18:54.857 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-28T19:18:54.857 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-08-28T19:18:54.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-08-28T19:18:54.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-08-28T19:18:54.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-08-28T19:18:54.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-08-28T19:18:54.857 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-08-28T19:18:54.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:54 smithi115 ceph-mon[22184]: pgmap v271: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:54.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:54 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/4285782460' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-08-28T19:18:54.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-28T19:18:55.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:18:55.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:18:55.049 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-28T19:18:55.074 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-08-28T19:18:55.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:18:55.075 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-08-28T19:18:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-08-28T19:18:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-28T19:18:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-28T19:18:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-28T19:18:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-28T19:18:55.142 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-28T19:18:55.142 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-08-28T19:18:55.142 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-28T19:18:55.142 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-28T19:18:55.142 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-28T19:18:55.142 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-08-28T19:18:55.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-28T19:18:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:18:55.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:18:55.578 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-08-28T19:18:55.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:18:55.594 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-08-28T19:18:55.650 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-28T19:18:55.650 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-08-28T19:18:55.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-28T19:18:55.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-08-28T19:18:55.650 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-08-28T19:18:55.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-28T19:18:55.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:18:55.766 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-08-28T19:18:55.767 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-28T19:18:55.767 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-08-28T19:18:55.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-08-28T19:18:55.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-08-28T19:18:55.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-08-28T19:18:55.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-08-28T19:18:55.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-08-28T19:18:55.767 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-08-28T19:18:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-28T19:18:55.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:18:55.891 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-28T19:18:55.891 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-08-28T19:18:55.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-08-28T19:18:55.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-08-28T19:18:55.891 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-08-28T19:18:55.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-28T19:18:55.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:18:55.968 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-08-28T19:18:55.968 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-28T19:18:55.968 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-08-28T19:18:55.985 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-08-28T19:18:55.985 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-28T19:18:55.985 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-08-28T19:18:56.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-08-28T19:18:56.105 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-08-28T19:18:56.161 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-08-28T19:18:56.161 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-08-28T19:18:56.161 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-08-28T19:18:56.161 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-08-28T19:18:56.161 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-08-28T19:18:56.161 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-08-28T19:18:56.162 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/7d9c83fc2d6317d5da699596cf74400f02f2354a90ab60bed46a536b4946fced/merged 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/32964e950f6aceb9a10076a3df7192cbc0a872984abdb6d13fe0805162f48c9d/merged 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d86d671895e3bf23ebf8e834f6b76c82958f3e48f76543ede6404e3d843d0d39/merged 2024-08-28T19:18:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/0812f3c0036d6f2267c5774ac726407a6eb9ad3e4dfb809e34168f0cd72175f6/merged 2024-08-28T19:18:56.164 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/da0d1c27ead9e44de4284eda0065f2b7d3ab986d4bf63f797791b31dd23a6b96/merged 2024-08-28T19:18:56.164 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/a8c17ff7966149fe68481f58affa51cf8e348178ea593e6a72349ddaad8a3d64/merged 2024-08-28T19:18:56.164 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/73b1dab79a2d4f576799dca107de740f5dfb386cb29e58debfe262dd3e369501/merged 2024-08-28T19:18:56.164 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/36fae32b0d9c9f498ecbc15794c27a816072e714f30faa42abc0867937b761db/merged 2024-08-28T19:18:56.164 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/de3e2a0ea7356413218fd26d7edc683d1d20b42c906ae010c6052cde351dfc06/merged 2024-08-28T19:18:56.164 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/5a68ed67cbd5c93da91d349f84fb1a6321c235639f04e9a6b286a030faa9b3d6/merged 2024-08-28T19:18:56.164 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-08-28T19:18:56.164 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-28T19:18:56.164 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-28T19:18:56.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:18:56.164 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-08-28T19:18:56.216 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-08-28T19:18:56.216 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-08-28T19:18:56.258 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-08-28T19:18:56.283 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-08-28T19:18:56.340 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-08-28T19:18:56.340 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-08-28T19:18:56.340 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-08-28T19:18:56.340 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-08-28T19:18:56.340 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-08-28T19:18:56.340 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-08-28T19:18:56.340 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-08-28T19:18:56.340 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-08-28T19:18:56.340 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-08-28T19:18:56.340 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-08-28T19:18:56.340 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-08-28T19:18:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/7d9c83fc2d6317d5da699596cf74400f02f2354a90ab60bed46a536b4946fced/merged 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/32964e950f6aceb9a10076a3df7192cbc0a872984abdb6d13fe0805162f48c9d/merged 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d86d671895e3bf23ebf8e834f6b76c82958f3e48f76543ede6404e3d843d0d39/merged 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/0812f3c0036d6f2267c5774ac726407a6eb9ad3e4dfb809e34168f0cd72175f6/merged 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/da0d1c27ead9e44de4284eda0065f2b7d3ab986d4bf63f797791b31dd23a6b96/merged 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/a8c17ff7966149fe68481f58affa51cf8e348178ea593e6a72349ddaad8a3d64/merged 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/73b1dab79a2d4f576799dca107de740f5dfb386cb29e58debfe262dd3e369501/merged 2024-08-28T19:18:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/36fae32b0d9c9f498ecbc15794c27a816072e714f30faa42abc0867937b761db/merged 2024-08-28T19:18:56.343 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/de3e2a0ea7356413218fd26d7edc683d1d20b42c906ae010c6052cde351dfc06/merged 2024-08-28T19:18:56.343 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/5a68ed67cbd5c93da91d349f84fb1a6321c235639f04e9a6b286a030faa9b3d6/merged 2024-08-28T19:18:56.343 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-08-28T19:18:56.343 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-28T19:18:56.343 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-28T19:18:56.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:18:56.343 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-08-28T19:18:56.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:2024-08-28T19:18:56.389+0000 7fdee48b2480 -1 init, newargv = 0x7fded4004360 newargc=15 2024-08-28T19:18:56.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[66895]: starting ceph client 2024-08-28T19:18:56.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[66895]: starting fuse 2024-08-28T19:18:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:56 smithi028 ceph-mon[21769]: pgmap v272: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:56 smithi115 ceph-mon[22184]: pgmap v272: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:57.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:56 smithi016 ceph-mon[18556]: pgmap v272: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:57.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-08-28T19:18:57.530 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-08-28T19:18:57.589 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-08-28T19:18:57.589 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-08-28T19:18:57.589 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-08-28T19:18:57.589 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-08-28T19:18:57.589 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-08-28T19:18:57.590 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-08-28T19:18:57.590 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-08-28T19:18:57.590 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-08-28T19:18:57.590 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-08-28T19:18:57.590 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-08-28T19:18:57.590 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-08-28T19:18:57.590 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-08-28T19:18:57.590 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-08-28T19:18:57.590 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-08-28T19:18:57.590 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-08-28T19:18:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-08-28T19:18:57.592 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/7d9c83fc2d6317d5da699596cf74400f02f2354a90ab60bed46a536b4946fced/merged 2024-08-28T19:18:57.592 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/32964e950f6aceb9a10076a3df7192cbc0a872984abdb6d13fe0805162f48c9d/merged 2024-08-28T19:18:57.592 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-08-28T19:18:57.592 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-08-28T19:18:57.592 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d86d671895e3bf23ebf8e834f6b76c82958f3e48f76543ede6404e3d843d0d39/merged 2024-08-28T19:18:57.592 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/0812f3c0036d6f2267c5774ac726407a6eb9ad3e4dfb809e34168f0cd72175f6/merged 2024-08-28T19:18:57.592 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/da0d1c27ead9e44de4284eda0065f2b7d3ab986d4bf63f797791b31dd23a6b96/merged 2024-08-28T19:18:57.592 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/a8c17ff7966149fe68481f58affa51cf8e348178ea593e6a72349ddaad8a3d64/merged 2024-08-28T19:18:57.592 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/73b1dab79a2d4f576799dca107de740f5dfb386cb29e58debfe262dd3e369501/merged 2024-08-28T19:18:57.592 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/36fae32b0d9c9f498ecbc15794c27a816072e714f30faa42abc0867937b761db/merged 2024-08-28T19:18:57.593 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/de3e2a0ea7356413218fd26d7edc683d1d20b42c906ae010c6052cde351dfc06/merged 2024-08-28T19:18:57.593 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/5a68ed67cbd5c93da91d349f84fb1a6321c235639f04e9a6b286a030faa9b3d6/merged 2024-08-28T19:18:57.593 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-08-28T19:18:57.593 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-28T19:18:57.593 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-28T19:18:57.593 INFO:teuthology.orchestra.run.smithi016.stdout:/home/ubuntu/cephtest/mnt.admin 2024-08-28T19:18:57.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:18:57.593 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-08-28T19:18:57.646 INFO:teuthology.orchestra.run.smithi016.stdout:121 2024-08-28T19:18:57.647 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [121] 2024-08-28T19:18:57.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-08-28T19:18:57.647 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-08-28T19:18:57.647 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-08-28T19:18:57.647 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-08-28T19:18:57.647 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-08-28T19:18:57.647 DEBUG:teuthology.orchestra.run.smithi016:> 2024-08-28T19:18:57.647 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-08-28T19:18:57.647 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-08-28T19:18:57.648 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-08-28T19:18:57.649 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-08-28T19:18:57.649 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-08-28T19:18:57.649 DEBUG:teuthology.orchestra.run.smithi016:> 2024-08-28T19:18:57.649 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.admin")) 2024-08-28T19:18:57.649 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-08-28T19:18:57.772 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.admin.66895.asok 2024-08-28T19:18:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:18:57.783 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.66895.asok 2024-08-28T19:18:57.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:18:57.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.66895.asok status 2024-08-28T19:18:57.928 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:18:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d19e6118f7015f0047264006cb0c873528eb3ae6", 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.3.0-4560-gd19e6118 (d19e6118f7015f0047264006cb0c873528eb3ae6) squid (dev)", 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "admin", 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "66895", 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 14949, 2024-08-28T19:18:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 14949 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3710268141 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-08-28T19:18:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3710268141 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.14949 192.168.0.1:0/3710268141", 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/3710268141", 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 17, 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 80, 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-08-28T19:18:57.931 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:18:57.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:18:57.938 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-08-28T19:18:57.938 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-08-28T19:18:57.953 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-08-28T19:18:57.954 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-08-28T19:18:57.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:18:57.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-08-28T19:18:58.019 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-08-28T19:18:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-28T19:18:58.129 INFO:teuthology.orchestra.run.smithi016.stdout:# file: . 2024-08-28T19:18:58.130 INFO:teuthology.orchestra.run.smithi016.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-08-28T19:18:58.130 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:18:58.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:18:58.135 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:18:58.136 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-28T19:18:58.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd blocklist ls 2024-08-28T19:18:58.445 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:18:58.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:58 smithi028 ceph-mon[21769]: pgmap v273: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:58.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:58 smithi016 ceph-mon[18556]: pgmap v273: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:58.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:58 smithi115 ceph-mon[22184]: pgmap v273: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:18:58.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.954+0000 7f08be393640 1 -- 172.21.15.16:0/237588442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b8109be0 msgr2=0x7f08b8110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:58.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.954+0000 7f08be393640 1 --2- 172.21.15.16:0/237588442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b8109be0 0x7f08b8110410 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f08b0009ca0 tx=0x7f08b002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:18:58.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.954+0000 7f08be393640 1 -- 172.21.15.16:0/237588442 shutdown_connections 2024-08-28T19:18:58.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.954+0000 7f08be393640 1 --2- 172.21.15.16:0/237588442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b8109be0 0x7f08b8110410 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:58.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.954+0000 7f08be393640 1 --2- 172.21.15.16:0/237588442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b8109290 0x7f08b81096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.954+0000 7f08be393640 1 --2- 172.21.15.16:0/237588442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08b8103950 0x7f08b8103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.954+0000 7f08be393640 1 -- 172.21.15.16:0/237588442 >> 172.21.15.16:0/237588442 conn(0x7f08b80ff3a0 msgr2=0x7f08b81017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.954+0000 7f08be393640 1 -- 172.21.15.16:0/237588442 shutdown_connections 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.954+0000 7f08be393640 1 -- 172.21.15.16:0/237588442 wait complete. 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.955+0000 7f08be393640 1 Processor -- start 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.955+0000 7f08be393640 1 -- start start 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.955+0000 7f08be393640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08b8103950 0x7f08b819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.955+0000 7f08be393640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b8109290 0x7f08b819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.955+0000 7f08be393640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b8109be0 0x7f08b819fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.955+0000 7f08be393640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08b81138a0 con 0x7f08b8109290 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.955+0000 7f08be393640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08b8113720 con 0x7f08b8109be0 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.955+0000 7f08be393640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f08b8113a20 con 0x7f08b8103950 2024-08-28T19:18:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.955+0000 7f08b77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b8109290 0x7f08b819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.955+0000 7f08b77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b8109290 0x7f08b819a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49990/0 (socket says 172.21.15.16:49990) 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.955+0000 7f08b77fe640 1 -- 172.21.15.16:0/3626566139 learned_addr learned my addr 172.21.15.16:0/3626566139 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08bc930640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b8109be0 0x7f08b819fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08b7fff640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08b8103950 0x7f08b819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08b77fe640 1 -- 172.21.15.16:0/3626566139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08b8103950 msgr2=0x7f08b819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08b77fe640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08b8103950 0x7f08b819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08b77fe640 1 -- 172.21.15.16:0/3626566139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b8109be0 msgr2=0x7f08b819fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08b77fe640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b8109be0 0x7f08b819fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08b77fe640 1 -- 172.21.15.16:0/3626566139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b81a0140 con 0x7f08b8109290 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08b7fff640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08b8103950 0x7f08b819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08b77fe640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b8109290 0x7f08b819a830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00d740 tx=0x7f08ac00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08b57fa640 1 -- 172.21.15.16:0/3626566139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac017070 con 0x7f08b8109290 2024-08-28T19:18:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08be393640 1 -- 172.21.15.16:0/3626566139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08b81a0430 con 0x7f08b8109290 2024-08-28T19:18:58.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08b57fa640 1 -- 172.21.15.16:0/3626566139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08ac00de50 con 0x7f08b8109290 2024-08-28T19:18:58.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08b57fa640 1 -- 172.21.15.16:0/3626566139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac00f620 con 0x7f08b8109290 2024-08-28T19:18:58.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.956+0000 7f08be393640 1 -- 172.21.15.16:0/3626566139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08b81a6c50 con 0x7f08b8109290 2024-08-28T19:18:58.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.957+0000 7f08be393640 1 -- 172.21.15.16:0/3626566139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08b8103d60 con 0x7f08b8109290 2024-08-28T19:18:58.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.961+0000 7f08b57fa640 1 -- 172.21.15.16:0/3626566139 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f08ac01a070 con 0x7f08b8109290 2024-08-28T19:18:58.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.961+0000 7f08b57fa640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08a00b4bb0 0x7f08a00b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:18:58.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.961+0000 7f08b7fff640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08a00b4bb0 0x7f08a00b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:18:58.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.961+0000 7f08b57fa640 1 -- 172.21.15.16:0/3626566139 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7f08ac0e4dd0 con 0x7f08b8109290 2024-08-28T19:18:58.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.961+0000 7f08b7fff640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08a00b4bb0 0x7f08a00b70c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f08a800baa0 tx=0x7f08a8005d90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:18:58.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:58.964+0000 7f08b57fa640 1 -- 172.21.15.16:0/3626566139 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f08ac0abbf0 con 0x7f08b8109290 2024-08-28T19:18:59.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.164+0000 7f08be393640 1 -- 172.21.15.16:0/3626566139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f08b810ba30 con 0x7f08b8109290 2024-08-28T19:18:59.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.165+0000 7f08b57fa640 1 -- 172.21.15.16:0/3626566139 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v80) ==== 81+0+887 (secure 0 0 0) 0x7f08ac0b4130 con 0x7f08b8109290 2024-08-28T19:18:59.167 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4224232323 2024-08-29T19:12:07.367977+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4022283606 2024-08-29T19:12:07.367977+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3264341019 2024-08-29T19:12:07.367977+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3057984026 2024-08-29T19:10:41.419191+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3042282523 2024-08-29T19:10:41.419191+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3243085606 2024-08-29T19:10:41.419191+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1768458721 2024-08-29T19:10:41.419191+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/949818237 2024-08-29T19:11:05.284513+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3057984026 2024-08-29T19:10:41.419191+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1317220879 2024-08-29T19:11:05.284513+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2455667776 2024-08-29T19:11:05.284513+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3371171272 2024-08-29T19:11:05.284513+0000 2024-08-28T19:18:59.168 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3264341019 2024-08-29T19:12:07.367977+0000 2024-08-28T19:18:59.169 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3371171272 2024-08-29T19:11:05.284513+0000 2024-08-28T19:18:59.169 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3868803359 2024-08-29T19:12:07.367977+0000 2024-08-28T19:18:59.169 INFO:teuthology.orchestra.run.smithi016.stderr:listed 15 entries 2024-08-28T19:18:59.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 -- 172.21.15.16:0/3626566139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08a00b4bb0 msgr2=0x7f08a00b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:59.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08a00b4bb0 0x7f08a00b70c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f08a800baa0 tx=0x7f08a8005d90 comp rx=0 tx=0).stop 2024-08-28T19:18:59.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 -- 172.21.15.16:0/3626566139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b8109290 msgr2=0x7f08b819a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:18:59.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b8109290 0x7f08b819a830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00d740 tx=0x7f08ac00dc00 comp rx=0 tx=0).stop 2024-08-28T19:18:59.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 -- 172.21.15.16:0/3626566139 shutdown_connections 2024-08-28T19:18:59.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08a00b4bb0 0x7f08a00b70c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:59.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b8109be0 0x7f08b819fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:59.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b8109290 0x7f08b819a830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:59.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 --2- 172.21.15.16:0/3626566139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08b8103950 0x7f08b819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:18:59.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 -- 172.21.15.16:0/3626566139 >> 172.21.15.16:0/3626566139 conn(0x7f08b80ff3a0 msgr2=0x7f08b81017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:18:59.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 -- 172.21.15.16:0/3626566139 shutdown_connections 2024-08-28T19:18:59.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:18:59.168+0000 7f08be393640 1 -- 172.21.15.16:0/3626566139 wait complete. 2024-08-28T19:18:59.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:18:59 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3626566139' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-08-28T19:18:59.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:18:59 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3626566139' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-08-28T19:18:59.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:18:59 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3626566139' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-08-28T19:18:59.945 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:18:59.945 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-28T19:18:59.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd blocklist ls 2024-08-28T19:19:00.265 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:00.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:00 smithi016 ceph-mon[18556]: pgmap v274: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:19:00.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.756+0000 7fbe9a780640 1 -- 172.21.15.16:0/3199624153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe94108170 msgr2=0x7fbe94108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:00.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.756+0000 7fbe9a780640 1 --2- 172.21.15.16:0/3199624153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe94108170 0x7fbe94108580 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009900 tx=0x7fbe7c02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.756+0000 7fbe9a780640 1 -- 172.21.15.16:0/3199624153 shutdown_connections 2024-08-28T19:19:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.756+0000 7fbe9a780640 1 --2- 172.21.15.16:0/3199624153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe94108ac0 0x7fbe9410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.756+0000 7fbe9a780640 1 --2- 172.21.15.16:0/3199624153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe94108170 0x7fbe94108580 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.756+0000 7fbe9a780640 1 --2- 172.21.15.16:0/3199624153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe94102880 0x7fbe94102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.756+0000 7fbe9a780640 1 -- 172.21.15.16:0/3199624153 >> 172.21.15.16:0/3199624153 conn(0x7fbe94079500 msgr2=0x7fbe94079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.756+0000 7fbe9a780640 1 -- 172.21.15.16:0/3199624153 shutdown_connections 2024-08-28T19:19:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.756+0000 7fbe9a780640 1 -- 172.21.15.16:0/3199624153 wait complete. 2024-08-28T19:19:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.757+0000 7fbe9a780640 1 Processor -- start 2024-08-28T19:19:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.757+0000 7fbe9a780640 1 -- start start 2024-08-28T19:19:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.757+0000 7fbe9a780640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe94102880 0x7fbe9419afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.757+0000 7fbe9a780640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe94108170 0x7fbe9419b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.757+0000 7fbe9a780640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe94108ac0 0x7fbe9419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.757+0000 7fbe9a780640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94112710 con 0x7fbe94102880 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.757+0000 7fbe9a780640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94112590 con 0x7fbe94108ac0 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.757+0000 7fbe9a780640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94112890 con 0x7fbe94108170 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.757+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe94102880 0x7fbe9419afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.757+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe94102880 0x7fbe9419afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49998/0 (socket says 172.21.15.16:49998) 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.757+0000 7fbe93fff640 1 -- 172.21.15.16:0/1912596716 learned_addr learned my addr 172.21.15.16:0/1912596716 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe98d1d640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe94108ac0 0x7fbe9419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe8bfff640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe94108170 0x7fbe9419b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe93fff640 1 -- 172.21.15.16:0/1912596716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe94108170 msgr2=0x7fbe9419b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe93fff640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe94108170 0x7fbe9419b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:00.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe93fff640 1 -- 172.21.15.16:0/1912596716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe94108ac0 msgr2=0x7fbe9419ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe93fff640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe94108ac0 0x7fbe9419ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe93fff640 1 -- 172.21.15.16:0/1912596716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe941a62d0 con 0x7fbe94102880 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe8bfff640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe94108170 0x7fbe9419b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe93fff640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe94102880 0x7fbe9419afc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400d770 tx=0x7fbe8400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe91ffb640 1 -- 172.21.15.16:0/1912596716 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84017070 con 0x7fbe94102880 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe91ffb640 1 -- 172.21.15.16:0/1912596716 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe840054d0 con 0x7fbe94102880 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe9a780640 1 -- 172.21.15.16:0/1912596716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe941a65c0 con 0x7fbe94102880 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe91ffb640 1 -- 172.21.15.16:0/1912596716 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84004c30 con 0x7fbe94102880 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.758+0000 7fbe9a780640 1 -- 172.21.15.16:0/1912596716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe941a6ab0 con 0x7fbe94102880 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.759+0000 7fbe9a780640 1 -- 172.21.15.16:0/1912596716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe94102c90 con 0x7fbe94102880 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.761+0000 7fbe91ffb640 1 -- 172.21.15.16:0/1912596716 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fbe84004dd0 con 0x7fbe94102880 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.762+0000 7fbe91ffb640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbe800b4b80 0x7fbe800b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.762+0000 7fbe8bfff640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbe800b4b80 0x7fbe800b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:00.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.762+0000 7fbe91ffb640 1 -- 172.21.15.16:0/1912596716 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7fbe840e5300 con 0x7fbe94102880 2024-08-28T19:19:00.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.762+0000 7fbe8bfff640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbe800b4b80 0x7fbe800b7090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0096c0 tx=0x7fbe7c031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:00.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.765+0000 7fbe91ffb640 1 -- 172.21.15.16:0/1912596716 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fbe840ac1c0 con 0x7fbe94102880 2024-08-28T19:19:00.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:00 smithi115 ceph-mon[22184]: pgmap v274: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:19:00.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.967+0000 7fbe9a780640 1 -- 172.21.15.16:0/1912596716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbe9402e430 con 0x7fbe94102880 2024-08-28T19:19:00.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.967+0000 7fbe91ffb640 1 -- 172.21.15.16:0/1912596716 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v80) ==== 81+0+887 (secure 0 0 0) 0x7fbe840abac0 con 0x7fbe94102880 2024-08-28T19:19:00.969 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4224232323 2024-08-29T19:12:07.367977+0000 2024-08-28T19:19:00.969 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4022283606 2024-08-29T19:12:07.367977+0000 2024-08-28T19:19:00.969 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3264341019 2024-08-29T19:12:07.367977+0000 2024-08-28T19:19:00.969 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3057984026 2024-08-29T19:10:41.419191+0000 2024-08-28T19:19:00.969 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3042282523 2024-08-29T19:10:41.419191+0000 2024-08-28T19:19:00.969 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3243085606 2024-08-29T19:10:41.419191+0000 2024-08-28T19:19:00.969 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1768458721 2024-08-29T19:10:41.419191+0000 2024-08-28T19:19:00.969 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/949818237 2024-08-29T19:11:05.284513+0000 2024-08-28T19:19:00.969 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3057984026 2024-08-29T19:10:41.419191+0000 2024-08-28T19:19:00.969 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1317220879 2024-08-29T19:11:05.284513+0000 2024-08-28T19:19:00.969 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2455667776 2024-08-29T19:11:05.284513+0000 2024-08-28T19:19:00.970 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3371171272 2024-08-29T19:11:05.284513+0000 2024-08-28T19:19:00.970 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3264341019 2024-08-29T19:12:07.367977+0000 2024-08-28T19:19:00.970 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3371171272 2024-08-29T19:11:05.284513+0000 2024-08-28T19:19:00.970 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3868803359 2024-08-29T19:12:07.367977+0000 2024-08-28T19:19:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:listed 15 entries 2024-08-28T19:19:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.969+0000 7fbe9a780640 1 -- 172.21.15.16:0/1912596716 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbe800b4b80 msgr2=0x7fbe800b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.969+0000 7fbe9a780640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbe800b4b80 0x7fbe800b7090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0096c0 tx=0x7fbe7c031400 comp rx=0 tx=0).stop 2024-08-28T19:19:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.969+0000 7fbe9a780640 1 -- 172.21.15.16:0/1912596716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe94102880 msgr2=0x7fbe9419afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.969+0000 7fbe9a780640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe94102880 0x7fbe9419afc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400d770 tx=0x7fbe8400dc30 comp rx=0 tx=0).stop 2024-08-28T19:19:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.970+0000 7fbe9a780640 1 -- 172.21.15.16:0/1912596716 shutdown_connections 2024-08-28T19:19:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.970+0000 7fbe9a780640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbe800b4b80 0x7fbe800b7090 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.970+0000 7fbe9a780640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe94108ac0 0x7fbe9419ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.970+0000 7fbe9a780640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe94108170 0x7fbe9419b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.970+0000 7fbe9a780640 1 --2- 172.21.15.16:0/1912596716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe94102880 0x7fbe9419afc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.970+0000 7fbe9a780640 1 -- 172.21.15.16:0/1912596716 >> 172.21.15.16:0/1912596716 conn(0x7fbe94079500 msgr2=0x7fbe941041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:00.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.970+0000 7fbe9a780640 1 -- 172.21.15.16:0/1912596716 shutdown_connections 2024-08-28T19:19:00.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:00.970+0000 7fbe9a780640 1 -- 172.21.15.16:0/1912596716 wait complete. 2024-08-28T19:19:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:00 smithi028 ceph-mon[21769]: pgmap v274: 129 pgs: 129 active+clean; 579 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:19:01.669 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-08-28T19:19:01.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:19:01.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-08-28T19:19:01.696 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[66895]: fuse finished with error 0 and tester_r 0 2024-08-28T19:19:01.698 INFO:teuthology.orchestra.run:waiting for 300 2024-08-28T19:19:01.773 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-08-28T19:19:01.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:19:01.773 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-08-28T19:19:01.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:19:01.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:01 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1912596716' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-08-28T19:19:01.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:01 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1912596716' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-08-28T19:19:02.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:01 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1912596716' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-08-28T19:19:02.086 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.541+0000 7f83d9376640 1 -- 172.21.15.16:0/3859129356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d4104fc0 msgr2=0x7f83d41053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.541+0000 7f83d9376640 1 --2- 172.21.15.16:0/3859129356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d4104fc0 0x7f83d41053d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009900 tx=0x7f83bc02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.541+0000 7f83d9376640 1 -- 172.21.15.16:0/3859129356 shutdown_connections 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.541+0000 7f83d9376640 1 --2- 172.21.15.16:0/3859129356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f83d4105910 0x7f83d410f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.541+0000 7f83d9376640 1 --2- 172.21.15.16:0/3859129356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d4104fc0 0x7f83d41053d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.541+0000 7f83d9376640 1 --2- 172.21.15.16:0/3859129356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d4102fb0 0x7f83d41033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.541+0000 7f83d9376640 1 -- 172.21.15.16:0/3859129356 >> 172.21.15.16:0/3859129356 conn(0x7f83d4079c40 msgr2=0x7f83d407a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.541+0000 7f83d9376640 1 -- 172.21.15.16:0/3859129356 shutdown_connections 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.541+0000 7f83d9376640 1 -- 172.21.15.16:0/3859129356 wait complete. 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.541+0000 7f83d9376640 1 Processor -- start 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d9376640 1 -- start start 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d9376640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d4102fb0 0x7f83d419b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d9376640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f83d4104fc0 0x7f83d419b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d9376640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d4105910 0x7f83d419bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d9376640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83d41128c0 con 0x7f83d4102fb0 2024-08-28T19:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d9376640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83d4112740 con 0x7f83d4105910 2024-08-28T19:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d9376640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f83d4112a40 con 0x7f83d4104fc0 2024-08-28T19:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d4102fb0 0x7f83d419b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d4102fb0 0x7f83d419b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50004/0 (socket says 172.21.15.16:50004) 2024-08-28T19:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d2ffd640 1 -- 172.21.15.16:0/259915415 learned_addr learned my addr 172.21.15.16:0/259915415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83ca7fc640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f83d4104fc0 0x7f83d419b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d37fe640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d4105910 0x7f83d419bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d2ffd640 1 -- 172.21.15.16:0/259915415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f83d4104fc0 msgr2=0x7f83d419b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d2ffd640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f83d4104fc0 0x7f83d419b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d2ffd640 1 -- 172.21.15.16:0/259915415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d4105910 msgr2=0x7f83d419bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d2ffd640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d4105910 0x7f83d419bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.542+0000 7f83d2ffd640 1 -- 172.21.15.16:0/259915415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d41a6480 con 0x7f83d4102fb0 2024-08-28T19:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.543+0000 7f83d2ffd640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d4102fb0 0x7f83d419b1f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f83c400b450 tx=0x7f83c400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.543+0000 7f83d0ff9640 1 -- 172.21.15.16:0/259915415 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4015020 con 0x7f83d4102fb0 2024-08-28T19:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.543+0000 7f83d0ff9640 1 -- 172.21.15.16:0/259915415 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83c4011d40 con 0x7f83d4102fb0 2024-08-28T19:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.543+0000 7f83d9376640 1 -- 172.21.15.16:0/259915415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d41a6770 con 0x7f83d4102fb0 2024-08-28T19:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.543+0000 7f83d0ff9640 1 -- 172.21.15.16:0/259915415 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c400f340 con 0x7f83d4102fb0 2024-08-28T19:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.543+0000 7f83d9376640 1 -- 172.21.15.16:0/259915415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d41a6cd0 con 0x7f83d4102fb0 2024-08-28T19:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.544+0000 7f83d9376640 1 -- 172.21.15.16:0/259915415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d41033c0 con 0x7f83d4102fb0 2024-08-28T19:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.546+0000 7f83d0ff9640 1 -- 172.21.15.16:0/259915415 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f83c4018020 con 0x7f83d4102fb0 2024-08-28T19:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.547+0000 7f83d0ff9640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f83c00b4c30 0x7f83c00b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.547+0000 7f83ca7fc640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f83c00b4c30 0x7f83c00b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.547+0000 7f83d0ff9640 1 -- 172.21.15.16:0/259915415 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7f83c40e5300 con 0x7f83d4102fb0 2024-08-28T19:19:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.547+0000 7f83ca7fc640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f83c00b4c30 0x7f83c00b7140 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f83bc005940 tx=0x7f83bc0058d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.550+0000 7f83d0ff9640 1 -- 172.21.15.16:0/259915415 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f83c401f020 con 0x7f83d4102fb0 2024-08-28T19:19:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.737+0000 7f83d9376640 1 -- 172.21.15.16:0/259915415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83d402e430 con 0x7f83d4102fb0 2024-08-28T19:19:02.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.738+0000 7f83d0ff9640 1 -- 172.21.15.16:0/259915415 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+24493 (secure 0 0 0) 0x7f83c400d040 con 0x7f83d4102fb0 2024-08-28T19:19:02.740 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:19:02.740 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:18:42.223057+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:19:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.740+0000 7f83d9376640 1 -- 172.21.15.16:0/259915415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f83c00b4c30 msgr2=0x7f83c00b7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.740+0000 7f83d9376640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f83c00b4c30 0x7f83c00b7140 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f83bc005940 tx=0x7f83bc0058d0 comp rx=0 tx=0).stop 2024-08-28T19:19:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.740+0000 7f83d9376640 1 -- 172.21.15.16:0/259915415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d4102fb0 msgr2=0x7f83d419b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.740+0000 7f83d9376640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d4102fb0 0x7f83d419b1f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f83c400b450 tx=0x7f83c400b910 comp rx=0 tx=0).stop 2024-08-28T19:19:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.740+0000 7f83d9376640 1 -- 172.21.15.16:0/259915415 shutdown_connections 2024-08-28T19:19:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.740+0000 7f83d9376640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f83c00b4c30 0x7f83c00b7140 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.740+0000 7f83d9376640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d4105910 0x7f83d419bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.740+0000 7f83d9376640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f83d4104fc0 0x7f83d419b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.740+0000 7f83d9376640 1 --2- 172.21.15.16:0/259915415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d4102fb0 0x7f83d419b1f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.740+0000 7f83d9376640 1 -- 172.21.15.16:0/259915415 >> 172.21.15.16:0/259915415 conn(0x7f83d4079c40 msgr2=0x7f83d4106210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.741+0000 7f83d9376640 1 -- 172.21.15.16:0/259915415 shutdown_connections 2024-08-28T19:19:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:02.741+0000 7f83d9376640 1 -- 172.21.15.16:0/259915415 wait complete. 2024-08-28T19:19:02.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:02 smithi115 ceph-mon[22184]: pgmap v275: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-28T19:19:03.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:02 smithi016 ceph-mon[18556]: pgmap v275: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-28T19:19:03.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:02 smithi028 ceph-mon[21769]: pgmap v275: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-28T19:19:03.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:19:03.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:03 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/259915415' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:03.811 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:03 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/259915415' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:04.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:03 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/259915415' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:04.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.259+0000 7fe391db2640 1 -- 172.21.15.16:0/2977205313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe38c108ad0 msgr2=0x7fe38c10f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:04.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.259+0000 7fe391db2640 1 --2- 172.21.15.16:0/2977205313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe38c108ad0 0x7fe38c10f300 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe384009ca0 tx=0x7fe38402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:04.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.260+0000 7fe391db2640 1 -- 172.21.15.16:0/2977205313 shutdown_connections 2024-08-28T19:19:04.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.260+0000 7fe391db2640 1 --2- 172.21.15.16:0/2977205313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe38c108ad0 0x7fe38c10f300 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:04.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.260+0000 7fe391db2640 1 --2- 172.21.15.16:0/2977205313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe38c108180 0x7fe38c108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:04.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.260+0000 7fe391db2640 1 --2- 172.21.15.16:0/2977205313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe38c102840 0x7fe38c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:04.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.260+0000 7fe391db2640 1 -- 172.21.15.16:0/2977205313 >> 172.21.15.16:0/2977205313 conn(0x7fe38c0794f0 msgr2=0x7fe38c079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:04.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.260+0000 7fe391db2640 1 -- 172.21.15.16:0/2977205313 shutdown_connections 2024-08-28T19:19:04.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.260+0000 7fe391db2640 1 -- 172.21.15.16:0/2977205313 wait complete. 2024-08-28T19:19:04.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.260+0000 7fe391db2640 1 Processor -- start 2024-08-28T19:19:04.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe391db2640 1 -- start start 2024-08-28T19:19:04.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe391db2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe38c102840 0x7fe38c19afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:04.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe391db2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe38c108180 0x7fe38c19b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:04.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe391db2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe38c108ad0 0x7fe38c19ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:04.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe391db2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe38c1127a0 con 0x7fe38c108180 2024-08-28T19:19:04.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe391db2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe38c112620 con 0x7fe38c108ad0 2024-08-28T19:19:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe391db2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe38c112920 con 0x7fe38c102840 2024-08-28T19:19:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe38c108180 0x7fe38c19b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe38c108180 0x7fe38c19b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50012/0 (socket says 172.21.15.16:50012) 2024-08-28T19:19:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38affd640 1 -- 172.21.15.16:0/3656876633 learned_addr learned my addr 172.21.15.16:0/3656876633 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38b7fe640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe38c102840 0x7fe38c19afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38bfff640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe38c108ad0 0x7fe38c19ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38affd640 1 -- 172.21.15.16:0/3656876633 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe38c102840 msgr2=0x7fe38c19afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38affd640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe38c102840 0x7fe38c19afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38affd640 1 -- 172.21.15.16:0/3656876633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe38c108ad0 msgr2=0x7fe38c19ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38affd640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe38c108ad0 0x7fe38c19ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38affd640 1 -- 172.21.15.16:0/3656876633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe38c1a6290 con 0x7fe38c108180 2024-08-28T19:19:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38bfff640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe38c108ad0 0x7fe38c19ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.261+0000 7fe38b7fe640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe38c102840 0x7fe38c19afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.262+0000 7fe38affd640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe38c108180 0x7fe38c19b510 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe38000c970 tx=0x7fe38000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.262+0000 7fe388ff9640 1 -- 172.21.15.16:0/3656876633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe380007bb0 con 0x7fe38c108180 2024-08-28T19:19:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.262+0000 7fe388ff9640 1 -- 172.21.15.16:0/3656876633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe380007d50 con 0x7fe38c108180 2024-08-28T19:19:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.262+0000 7fe391db2640 1 -- 172.21.15.16:0/3656876633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe38c1a6580 con 0x7fe38c108180 2024-08-28T19:19:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.262+0000 7fe388ff9640 1 -- 172.21.15.16:0/3656876633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38000f450 con 0x7fe38c108180 2024-08-28T19:19:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.262+0000 7fe391db2640 1 -- 172.21.15.16:0/3656876633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe38c1a6ac0 con 0x7fe38c108180 2024-08-28T19:19:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.263+0000 7fe391db2640 1 -- 172.21.15.16:0/3656876633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe38c103ac0 con 0x7fe38c108180 2024-08-28T19:19:04.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.264+0000 7fe388ff9640 1 -- 172.21.15.16:0/3656876633 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7fe380004eb0 con 0x7fe38c108180 2024-08-28T19:19:04.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.265+0000 7fe388ff9640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3740b4b80 0x7fe3740b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:04.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.265+0000 7fe38b7fe640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3740b4b80 0x7fe3740b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:04.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.265+0000 7fe388ff9640 1 -- 172.21.15.16:0/3656876633 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7fe3800e5680 con 0x7fe38c108180 2024-08-28T19:19:04.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.266+0000 7fe38b7fe640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3740b4b80 0x7fe3740b7090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe38c06c460 tx=0x7fe37c005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:04.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.269+0000 7fe388ff9640 1 -- 172.21.15.16:0/3656876633 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fe380014030 con 0x7fe38c108180 2024-08-28T19:19:04.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.453+0000 7fe391db2640 1 -- 172.21.15.16:0/3656876633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe38c06bd10 con 0x7fe38c108180 2024-08-28T19:19:04.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.455+0000 7fe388ff9640 1 -- 172.21.15.16:0/3656876633 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+24493 (secure 0 0 0) 0x7fe38c06bd10 con 0x7fe38c108180 2024-08-28T19:19:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:19:04.457 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:18:42.223057+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:19:04.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 -- 172.21.15.16:0/3656876633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3740b4b80 msgr2=0x7fe3740b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3740b4b80 0x7fe3740b7090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe38c06c460 tx=0x7fe37c005f90 comp rx=0 tx=0).stop 2024-08-28T19:19:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 -- 172.21.15.16:0/3656876633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe38c108180 msgr2=0x7fe38c19b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe38c108180 0x7fe38c19b510 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe38000c970 tx=0x7fe38000ce30 comp rx=0 tx=0).stop 2024-08-28T19:19:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 -- 172.21.15.16:0/3656876633 shutdown_connections 2024-08-28T19:19:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3740b4b80 0x7fe3740b7090 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe38c108ad0 0x7fe38c19ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe38c108180 0x7fe38c19b510 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 --2- 172.21.15.16:0/3656876633 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe38c102840 0x7fe38c19afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 -- 172.21.15.16:0/3656876633 >> 172.21.15.16:0/3656876633 conn(0x7fe38c0794f0 msgr2=0x7fe38c104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 -- 172.21.15.16:0/3656876633 shutdown_connections 2024-08-28T19:19:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:04.457+0000 7fe391db2640 1 -- 172.21.15.16:0/3656876633 wait complete. 2024-08-28T19:19:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:04 smithi115 ceph-mon[22184]: pgmap v276: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-28T19:19:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:04 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3656876633' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:05.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:04 smithi016 ceph-mon[18556]: pgmap v276: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-28T19:19:05.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:04 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3656876633' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:04 smithi028 ceph-mon[21769]: pgmap v276: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-28T19:19:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:04 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3656876633' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:05.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-08-28T19:19:05.457 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.929+0000 7f5982871640 1 -- 172.21.15.16:0/1477319110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597c109b90 msgr2=0x7f597c1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.929+0000 7f5982871640 1 --2- 172.21.15.16:0/1477319110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597c109b90 0x7f597c1103c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5974009ca0 tx=0x7f597402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.929+0000 7f5982871640 1 -- 172.21.15.16:0/1477319110 shutdown_connections 2024-08-28T19:19:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.929+0000 7f5982871640 1 --2- 172.21.15.16:0/1477319110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597c109b90 0x7f597c1103c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.929+0000 7f5982871640 1 --2- 172.21.15.16:0/1477319110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c109240 0x7f597c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.929+0000 7f5982871640 1 --2- 172.21.15.16:0/1477319110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f597c103900 0x7f597c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.929+0000 7f5982871640 1 -- 172.21.15.16:0/1477319110 >> 172.21.15.16:0/1477319110 conn(0x7f597c07b5f0 msgr2=0x7f597c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.929+0000 7f5982871640 1 -- 172.21.15.16:0/1477319110 shutdown_connections 2024-08-28T19:19:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.929+0000 7f5982871640 1 -- 172.21.15.16:0/1477319110 wait complete. 2024-08-28T19:19:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.930+0000 7f5982871640 1 Processor -- start 2024-08-28T19:19:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.930+0000 7f5982871640 1 -- start start 2024-08-28T19:19:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.930+0000 7f5982871640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c103900 0x7f597c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f5982871640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597c109240 0x7f597c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f5982871640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f597c109b90 0x7f597c19f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f5982871640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f597c113850 con 0x7f597c109240 2024-08-28T19:19:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f5982871640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f597c1136d0 con 0x7f597c103900 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f5982871640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f597c1139d0 con 0x7f597c109b90 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f597b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597c109240 0x7f597c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f597b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597c109240 0x7f597c19a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48610/0 (socket says 172.21.15.16:48610) 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f597b7fe640 1 -- 172.21.15.16:0/3404232086 learned_addr learned my addr 172.21.15.16:0/3404232086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f597bfff640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c103900 0x7f597c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f5980e0e640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f597c109b90 0x7f597c19f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f597b7fe640 1 -- 172.21.15.16:0/3404232086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f597c109b90 msgr2=0x7f597c19f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f597b7fe640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f597c109b90 0x7f597c19f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f597b7fe640 1 -- 172.21.15.16:0/3404232086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c103900 msgr2=0x7f597c19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f597b7fe640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c103900 0x7f597c19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f597b7fe640 1 -- 172.21.15.16:0/3404232086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597c1a0110 con 0x7f597c109240 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f5980e0e640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f597c109b90 0x7f597c19f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f597b7fe640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597c109240 0x7f597c19a740 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f597000d740 tx=0x7f597000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f59797fa640 1 -- 172.21.15.16:0/3404232086 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5970017070 con 0x7f597c109240 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f59797fa640 1 -- 172.21.15.16:0/3404232086 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f597000de50 con 0x7f597c109240 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.931+0000 7f5982871640 1 -- 172.21.15.16:0/3404232086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f597c1a0400 con 0x7f597c109240 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.932+0000 7f59797fa640 1 -- 172.21.15.16:0/3404232086 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597000f620 con 0x7f597c109240 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.932+0000 7f5982871640 1 -- 172.21.15.16:0/3404232086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f597c1a6b60 con 0x7f597c109240 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.932+0000 7f5982871640 1 -- 172.21.15.16:0/3404232086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f597c103d10 con 0x7f597c109240 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.934+0000 7f59797fa640 1 -- 172.21.15.16:0/3404232086 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f597001a070 con 0x7f597c109240 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.935+0000 7f59797fa640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f59640b4b80 0x7f59640b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.935+0000 7f597bfff640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f59640b4b80 0x7f59640b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.935+0000 7f59797fa640 1 -- 172.21.15.16:0/3404232086 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 9017+0+0 (secure 0 0 0) 0x7f59700e4e80 con 0x7f597c109240 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.935+0000 7f597bfff640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f59640b4b80 0x7f59640b7090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f596c005d00 tx=0x7f596c00b410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:05.938+0000 7f59797fa640 1 -- 172.21.15.16:0/3404232086 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f59700abc40 con 0x7f597c109240 2024-08-28T19:19:06.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.161+0000 7f5982871640 1 -- 172.21.15.16:0/3404232086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f597c02e430 con 0x7f597c109240 2024-08-28T19:19:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.595+0000 7f59797fa640 1 -- 172.21.15.16:0/3404232086 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f59700b4180 con 0x7f597c109240 2024-08-28T19:19:06.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 -- 172.21.15.16:0/3404232086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f59640b4b80 msgr2=0x7f59640b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:06.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f59640b4b80 0x7f59640b7090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f596c005d00 tx=0x7f596c00b410 comp rx=0 tx=0).stop 2024-08-28T19:19:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 -- 172.21.15.16:0/3404232086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597c109240 msgr2=0x7f597c19a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597c109240 0x7f597c19a740 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f597000d740 tx=0x7f597000dc00 comp rx=0 tx=0).stop 2024-08-28T19:19:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 -- 172.21.15.16:0/3404232086 shutdown_connections 2024-08-28T19:19:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f597c109b90 0x7f597c19f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597c109240 0x7f597c19a740 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f59640b4b80 0x7f59640b7090 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 --2- 172.21.15.16:0/3404232086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c103900 0x7f597c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 -- 172.21.15.16:0/3404232086 >> 172.21.15.16:0/3404232086 conn(0x7f597c07b5f0 msgr2=0x7f597c1051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 -- 172.21.15.16:0/3404232086 shutdown_connections 2024-08-28T19:19:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:06.598+0000 7f5982871640 1 -- 172.21.15.16:0/3404232086 wait complete. 2024-08-28T19:19:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:06 smithi115 ceph-mon[22184]: pgmap v277: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-28T19:19:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:06 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3404232086' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-08-28T19:19:06.925 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:06 smithi016 ceph-mon[18556]: pgmap v277: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-28T19:19:06.925 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:06 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3404232086' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-08-28T19:19:07.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:06 smithi028 ceph-mon[21769]: pgmap v277: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-28T19:19:07.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:06 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3404232086' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-08-28T19:19:07.355 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-08-28T19:19:07.356 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-08-28T19:19:07.623 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:07 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3404232086' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-08-28T19:19:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:07 smithi115 ceph-mon[22184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:19:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4, 8, 11]} : dispatch 2024-08-28T19:19:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]} : dispatch 2024-08-28T19:19:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]} : dispatch 2024-08-28T19:19:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]} : dispatch 2024-08-28T19:19:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]} : dispatch 2024-08-28T19:19:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]} : dispatch 2024-08-28T19:19:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]} : dispatch 2024-08-28T19:19:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]} : dispatch 2024-08-28T19:19:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]} : dispatch 2024-08-28T19:19:07.890 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:07 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3404232086' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-08-28T19:19:07.890 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:07 smithi016 ceph-mon[18556]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:19:07.890 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4, 8, 11]} : dispatch 2024-08-28T19:19:07.890 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]} : dispatch 2024-08-28T19:19:07.890 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]} : dispatch 2024-08-28T19:19:07.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]} : dispatch 2024-08-28T19:19:07.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]} : dispatch 2024-08-28T19:19:07.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]} : dispatch 2024-08-28T19:19:07.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]} : dispatch 2024-08-28T19:19:07.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]} : dispatch 2024-08-28T19:19:07.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]} : dispatch 2024-08-28T19:19:08.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:07 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3404232086' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-08-28T19:19:08.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:07 smithi028 ceph-mon[21769]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:19:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4, 8, 11]} : dispatch 2024-08-28T19:19:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]} : dispatch 2024-08-28T19:19:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]} : dispatch 2024-08-28T19:19:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]} : dispatch 2024-08-28T19:19:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]} : dispatch 2024-08-28T19:19:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]} : dispatch 2024-08-28T19:19:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]} : dispatch 2024-08-28T19:19:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]} : dispatch 2024-08-28T19:19:08.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]} : dispatch 2024-08-28T19:19:08.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.124+0000 7f6bea1bb640 1 -- 172.21.15.16:0/1036496718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be4108140 msgr2=0x7f6be4108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:08.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.124+0000 7f6bea1bb640 1 --2- 172.21.15.16:0/1036496718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be4108140 0x7f6be4108550 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8009900 tx=0x7f6bd802a3f0 comp rx=0 tx=0).stop 2024-08-28T19:19:08.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.124+0000 7f6bea1bb640 1 -- 172.21.15.16:0/1036496718 shutdown_connections 2024-08-28T19:19:08.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.124+0000 7f6bea1bb640 1 --2- 172.21.15.16:0/1036496718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6be4108a90 0x7f6be410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:08.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.124+0000 7f6bea1bb640 1 --2- 172.21.15.16:0/1036496718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be4108140 0x7f6be4108550 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:08.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.124+0000 7f6bea1bb640 1 --2- 172.21.15.16:0/1036496718 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6be4102800 0x7f6be4102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:08.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.124+0000 7f6bea1bb640 1 -- 172.21.15.16:0/1036496718 >> 172.21.15.16:0/1036496718 conn(0x7f6be40794d0 msgr2=0x7f6be40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:08.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.124+0000 7f6bea1bb640 1 -- 172.21.15.16:0/1036496718 shutdown_connections 2024-08-28T19:19:08.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.124+0000 7f6bea1bb640 1 -- 172.21.15.16:0/1036496718 wait complete. 2024-08-28T19:19:08.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6bea1bb640 1 Processor -- start 2024-08-28T19:19:08.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6bea1bb640 1 -- start start 2024-08-28T19:19:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6bea1bb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6be4102800 0x7f6be419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6bea1bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be4108140 0x7f6be419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6bea1bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6be4108a90 0x7f6be419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6bea1bb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6be41126e0 con 0x7f6be4108140 2024-08-28T19:19:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6bea1bb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6be4112560 con 0x7f6be4108a90 2024-08-28T19:19:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6bea1bb640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6be4112860 con 0x7f6be4102800 2024-08-28T19:19:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6be2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be4108140 0x7f6be419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6be2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be4108140 0x7f6be419a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48630/0 (socket says 172.21.15.16:48630) 2024-08-28T19:19:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6be2ffd640 1 -- 172.21.15.16:0/2249003477 learned_addr learned my addr 172.21.15.16:0/2249003477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6be37fe640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6be4102800 0x7f6be419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.125+0000 7f6be3fff640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6be4108a90 0x7f6be419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.126+0000 7f6be2ffd640 1 -- 172.21.15.16:0/2249003477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6be4102800 msgr2=0x7f6be419a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.126+0000 7f6be2ffd640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6be4102800 0x7f6be419a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.126+0000 7f6be2ffd640 1 -- 172.21.15.16:0/2249003477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6be4108a90 msgr2=0x7f6be419fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.126+0000 7f6be2ffd640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6be4108a90 0x7f6be419fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.126+0000 7f6be2ffd640 1 -- 172.21.15.16:0/2249003477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be41a0170 con 0x7f6be4108140 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.126+0000 7f6be2ffd640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be4108140 0x7f6be419a860 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8031a90 tx=0x7f6bd8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.126+0000 7f6be0ff9640 1 -- 172.21.15.16:0/2249003477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd803f030 con 0x7f6be4108140 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.126+0000 7f6be0ff9640 1 -- 172.21.15.16:0/2249003477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bd8003cb0 con 0x7f6be4108140 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.126+0000 7f6bea1bb640 1 -- 172.21.15.16:0/2249003477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be41a0400 con 0x7f6be4108140 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.126+0000 7f6be0ff9640 1 -- 172.21.15.16:0/2249003477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8035070 con 0x7f6be4108140 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.126+0000 7f6bea1bb640 1 -- 172.21.15.16:0/2249003477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be41a0930 con 0x7f6be4108140 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.127+0000 7f6bea1bb640 1 -- 172.21.15.16:0/2249003477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6be4102c10 con 0x7f6be4108140 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.129+0000 7f6be0ff9640 1 -- 172.21.15.16:0/2249003477 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f6bd8042030 con 0x7f6be4108140 2024-08-28T19:19:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.130+0000 7f6be0ff9640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6bcc0b4c30 0x7f6bcc0b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.130+0000 7f6be37fe640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6bcc0b4c30 0x7f6bcc0b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.130+0000 7f6be0ff9640 1 -- 172.21.15.16:0/2249003477 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 9286+0+0 (secure 0 0 0) 0x7f6bd8105e70 con 0x7f6be4108140 2024-08-28T19:19:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.130+0000 7f6be37fe640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6bcc0b4c30 0x7f6bcc0b7140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40058c0 tx=0x7f6bd400a250 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.133+0000 7f6be0ff9640 1 -- 172.21.15.16:0/2249003477 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f6be41a0930 con 0x7f6be4108140 2024-08-28T19:19:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.358+0000 7f6bea1bb640 1 -- 172.21.15.16:0/2249003477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f6be402e430 con 0x7f6be4108140 2024-08-28T19:19:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.603+0000 7f6be0ff9640 1 -- 172.21.15.16:0/2249003477 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f6bd80cc4d0 con 0x7f6be4108140 2024-08-28T19:19:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 -- 172.21.15.16:0/2249003477 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6bcc0b4c30 msgr2=0x7f6bcc0b7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6bcc0b4c30 0x7f6bcc0b7140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40058c0 tx=0x7f6bd400a250 comp rx=0 tx=0).stop 2024-08-28T19:19:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 -- 172.21.15.16:0/2249003477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be4108140 msgr2=0x7f6be419a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be4108140 0x7f6be419a860 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8031a90 tx=0x7f6bd8004270 comp rx=0 tx=0).stop 2024-08-28T19:19:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 -- 172.21.15.16:0/2249003477 shutdown_connections 2024-08-28T19:19:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6bcc0b4c30 0x7f6bcc0b7140 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6be4108a90 0x7f6be419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be4108140 0x7f6be419a860 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 --2- 172.21.15.16:0/2249003477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6be4102800 0x7f6be419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 -- 172.21.15.16:0/2249003477 >> 172.21.15.16:0/2249003477 conn(0x7f6be40794d0 msgr2=0x7f6be406afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 -- 172.21.15.16:0/2249003477 shutdown_connections 2024-08-28T19:19:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:08.605+0000 7f6bea1bb640 1 -- 172.21.15.16:0/2249003477 wait complete. 2024-08-28T19:19:08.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: pgmap v278: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-28T19:19:08.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4, 8, 11]}]': finished 2024-08-28T19:19:08.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-08-28T19:19:08.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-08-28T19:19:08.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-08-28T19:19:08.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-08-28T19:19:08.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-08-28T19:19:08.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]}]': finished 2024-08-28T19:19:08.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-08-28T19:19:08.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-08-28T19:19:08.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: osdmap e81: 12 total, 12 up, 12 in 2024-08-28T19:19:08.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:08 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2249003477' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-08-28T19:19:08.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: pgmap v278: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-28T19:19:08.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4, 8, 11]}]': finished 2024-08-28T19:19:08.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-08-28T19:19:08.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-08-28T19:19:08.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-08-28T19:19:08.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-08-28T19:19:08.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-08-28T19:19:08.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]}]': finished 2024-08-28T19:19:08.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-08-28T19:19:08.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-08-28T19:19:08.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: osdmap e81: 12 total, 12 up, 12 in 2024-08-28T19:19:08.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:08 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2249003477' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-08-28T19:19:09.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: pgmap v278: 129 pgs: 129 active+clean; 581 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-28T19:19:09.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4, 8, 11]}]': finished 2024-08-28T19:19:09.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-08-28T19:19:09.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-08-28T19:19:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-08-28T19:19:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-08-28T19:19:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-08-28T19:19:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]}]': finished 2024-08-28T19:19:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-08-28T19:19:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-08-28T19:19:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: osdmap e81: 12 total, 12 up, 12 in 2024-08-28T19:19:09.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:08 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2249003477' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-08-28T19:19:09.349 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-08-28T19:19:09.604 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:09.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:09 smithi115 ceph-mon[22184]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-28T19:19:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:09 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2249003477' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-08-28T19:19:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:09 smithi115 ceph-mon[22184]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-08-28T19:19:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:09 smithi115 ceph-mon[22184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:19:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:09 smithi115 ceph-mon[22184]: osdmap e82: 12 total, 12 up, 12 in 2024-08-28T19:19:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:09 smithi115 ceph-mon[22184]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-08-28T19:19:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:09 smithi115 ceph-mon[22184]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-08-28T19:19:09.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:09 smithi016 ceph-mon[18556]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-28T19:19:09.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:09 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2249003477' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-08-28T19:19:09.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:09 smithi016 ceph-mon[18556]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-08-28T19:19:09.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:09 smithi016 ceph-mon[18556]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:19:09.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:09 smithi016 ceph-mon[18556]: osdmap e82: 12 total, 12 up, 12 in 2024-08-28T19:19:09.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:09 smithi016 ceph-mon[18556]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-08-28T19:19:09.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:09 smithi016 ceph-mon[18556]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-08-28T19:19:10.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:09 smithi028 ceph-mon[21769]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-28T19:19:10.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:09 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2249003477' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-08-28T19:19:10.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:09 smithi028 ceph-mon[21769]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-08-28T19:19:10.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:09 smithi028 ceph-mon[21769]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-28T19:19:10.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:09 smithi028 ceph-mon[21769]: osdmap e82: 12 total, 12 up, 12 in 2024-08-28T19:19:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:09 smithi028 ceph-mon[21769]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-08-28T19:19:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:09 smithi028 ceph-mon[21769]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-08-28T19:19:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.103+0000 7f27df13a640 1 -- 172.21.15.16:0/4022450949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d8109300 msgr2=0x7f27d8109710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.103+0000 7f27df13a640 1 --2- 172.21.15.16:0/4022450949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d8109300 0x7f27d8109710 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009900 tx=0x7f27c402a410 comp rx=0 tx=0).stop 2024-08-28T19:19:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.103+0000 7f27df13a640 1 -- 172.21.15.16:0/4022450949 shutdown_connections 2024-08-28T19:19:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.103+0000 7f27df13a640 1 --2- 172.21.15.16:0/4022450949 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8109c50 0x7f27d8110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.103+0000 7f27df13a640 1 --2- 172.21.15.16:0/4022450949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d8109300 0x7f27d8109710 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.103+0000 7f27df13a640 1 --2- 172.21.15.16:0/4022450949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27d8101e20 0x7f27d8102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.103+0000 7f27df13a640 1 -- 172.21.15.16:0/4022450949 >> 172.21.15.16:0/4022450949 conn(0x7f27d8079c60 msgr2=0x7f27d807a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.103+0000 7f27df13a640 1 -- 172.21.15.16:0/4022450949 shutdown_connections 2024-08-28T19:19:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.103+0000 7f27df13a640 1 -- 172.21.15.16:0/4022450949 wait complete. 2024-08-28T19:19:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.104+0000 7f27df13a640 1 Processor -- start 2024-08-28T19:19:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.104+0000 7f27df13a640 1 -- start start 2024-08-28T19:19:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.104+0000 7f27df13a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8101e20 0x7f27d819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.104+0000 7f27df13a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d8109300 0x7f27d819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.104+0000 7f27df13a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27d8109c50 0x7f27d819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.104+0000 7f27df13a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27d8113930 con 0x7f27d8109300 2024-08-28T19:19:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.104+0000 7f27df13a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81137b0 con 0x7f27d8109c50 2024-08-28T19:19:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.104+0000 7f27df13a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f27d8113ab0 con 0x7f27d8101e20 2024-08-28T19:19:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d8109300 0x7f27d819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d8109300 0x7f27d819a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48650/0 (socket says 172.21.15.16:48650) 2024-08-28T19:19:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d7fff640 1 -- 172.21.15.16:0/2766001018 learned_addr learned my addr 172.21.15.16:0/2766001018 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27dd6d7640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27d8109c50 0x7f27d819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27dced6640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8101e20 0x7f27d819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d7fff640 1 -- 172.21.15.16:0/2766001018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8101e20 msgr2=0x7f27d819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d7fff640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8101e20 0x7f27d819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d7fff640 1 -- 172.21.15.16:0/2766001018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27d8109c50 msgr2=0x7f27d819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d7fff640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27d8109c50 0x7f27d819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d7fff640 1 -- 172.21.15.16:0/2766001018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d81a02f0 con 0x7f27d8109300 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d7fff640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d8109300 0x7f27d819a920 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f27c402a930 tx=0x7f27c4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d5ffb640 1 -- 172.21.15.16:0/2766001018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c403f030 con 0x7f27d8109300 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d5ffb640 1 -- 172.21.15.16:0/2766001018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c4003aa0 con 0x7f27d8109300 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27df13a640 1 -- 172.21.15.16:0/2766001018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d81a0580 con 0x7f27d8109300 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.105+0000 7f27d5ffb640 1 -- 172.21.15.16:0/2766001018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4035070 con 0x7f27d8109300 2024-08-28T19:19:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.106+0000 7f27df13a640 1 -- 172.21.15.16:0/2766001018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d81a6d40 con 0x7f27d8109300 2024-08-28T19:19:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.107+0000 7f27df13a640 1 -- 172.21.15.16:0/2766001018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27d8102230 con 0x7f27d8109300 2024-08-28T19:19:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.109+0000 7f27d5ffb640 1 -- 172.21.15.16:0/2766001018 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f27c4042030 con 0x7f27d8109300 2024-08-28T19:19:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.110+0000 7f27d5ffb640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27c80b4c00 0x7f27c80b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.110+0000 7f27dced6640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27c80b4c00 0x7f27c80b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.110+0000 7f27d5ffb640 1 -- 172.21.15.16:0/2766001018 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f27c4105ff0 con 0x7f27d8109300 2024-08-28T19:19:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.111+0000 7f27dced6640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27c80b4c00 0x7f27c80b7110 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00bbe0 tx=0x7f27cc00b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.113+0000 7f27d5ffb640 1 -- 172.21.15.16:0/2766001018 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f27c40cc650 con 0x7f27d8109300 2024-08-28T19:19:10.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.344+0000 7f27df13a640 1 -- 172.21.15.16:0/2766001018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f27d819b4b0 con 0x7f27d8109300 2024-08-28T19:19:10.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.624+0000 7f27d5ffb640 1 -- 172.21.15.16:0/2766001018 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f27c40d0e10 con 0x7f27d8109300 2024-08-28T19:19:10.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 -- 172.21.15.16:0/2766001018 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27c80b4c00 msgr2=0x7f27c80b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:10.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27c80b4c00 0x7f27c80b7110 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00bbe0 tx=0x7f27cc00b390 comp rx=0 tx=0).stop 2024-08-28T19:19:10.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 -- 172.21.15.16:0/2766001018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d8109300 msgr2=0x7f27d819a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:10.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d8109300 0x7f27d819a920 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f27c402a930 tx=0x7f27c4004270 comp rx=0 tx=0).stop 2024-08-28T19:19:10.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 -- 172.21.15.16:0/2766001018 shutdown_connections 2024-08-28T19:19:10.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27c80b4c00 0x7f27c80b7110 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:10.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27d8109c50 0x7f27d819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d8109300 0x7f27d819a920 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 --2- 172.21.15.16:0/2766001018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8101e20 0x7f27d819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 -- 172.21.15.16:0/2766001018 >> 172.21.15.16:0/2766001018 conn(0x7f27d8079c60 msgr2=0x7f27d8107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 -- 172.21.15.16:0/2766001018 shutdown_connections 2024-08-28T19:19:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:10.627+0000 7f27df13a640 1 -- 172.21.15.16:0/2766001018 wait complete. 2024-08-28T19:19:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:10 smithi115 ceph-mon[22184]: pgmap v281: 129 pgs: 129 active+clean; 582 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-08-28T19:19:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:10 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] up:active 2024-08-28T19:19:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:10 smithi115 ceph-mon[22184]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-08-28T19:19:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:10 smithi115 ceph-mon[22184]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-08-28T19:19:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:10 smithi115 ceph-mon[22184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-08-28T19:19:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:10 smithi115 ceph-mon[22184]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-08-28T19:19:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:10 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2766001018' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-08-28T19:19:10.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:10 smithi016 ceph-mon[18556]: pgmap v281: 129 pgs: 129 active+clean; 582 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-08-28T19:19:10.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:10 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] up:active 2024-08-28T19:19:10.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:10 smithi016 ceph-mon[18556]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-08-28T19:19:10.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:10 smithi016 ceph-mon[18556]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-08-28T19:19:10.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:10 smithi016 ceph-mon[18556]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-08-28T19:19:10.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:10 smithi016 ceph-mon[18556]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-08-28T19:19:10.946 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:10 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2766001018' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-08-28T19:19:11.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:10 smithi028 ceph-mon[21769]: pgmap v281: 129 pgs: 129 active+clean; 582 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-08-28T19:19:11.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:10 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] up:active 2024-08-28T19:19:11.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:10 smithi028 ceph-mon[21769]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-08-28T19:19:11.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:10 smithi028 ceph-mon[21769]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-08-28T19:19:11.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:10 smithi028 ceph-mon[21769]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-08-28T19:19:11.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:10 smithi028 ceph-mon[21769]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-08-28T19:19:11.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:10 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2766001018' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-08-28T19:19:11.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-08-28T19:19:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:11 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2766001018' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-08-28T19:19:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:11 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] up:active 2024-08-28T19:19:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:11 smithi016 ceph-mon[18556]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-08-28T19:19:11.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:11 smithi016 ceph-mon[18556]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-08-28T19:19:11.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:11 smithi016 ceph-mon[18556]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-08-28T19:19:11.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:11 smithi016 ceph-mon[18556]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-08-28T19:19:12.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:11 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2766001018' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-08-28T19:19:12.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:11 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] up:active 2024-08-28T19:19:12.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:11 smithi028 ceph-mon[21769]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-08-28T19:19:12.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:11 smithi028 ceph-mon[21769]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-08-28T19:19:12.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:11 smithi028 ceph-mon[21769]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-08-28T19:19:12.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:11 smithi028 ceph-mon[21769]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-08-28T19:19:12.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:11 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2766001018' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-08-28T19:19:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:11 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] up:active 2024-08-28T19:19:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:11 smithi115 ceph-mon[22184]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-08-28T19:19:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:11 smithi115 ceph-mon[22184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-08-28T19:19:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:11 smithi115 ceph-mon[22184]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-08-28T19:19:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:11 smithi115 ceph-mon[22184]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-08-28T19:19:12.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.195+0000 7f0833bee640 1 -- 172.21.15.16:0/78970825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f082c10dbe0 msgr2=0x7f082c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:12.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.195+0000 7f0833bee640 1 --2- 172.21.15.16:0/78970825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f082c10dbe0 0x7f082c1100b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0824009ca0 tx=0x7f082402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:12.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.195+0000 7f0833bee640 1 -- 172.21.15.16:0/78970825 shutdown_connections 2024-08-28T19:19:12.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.195+0000 7f0833bee640 1 --2- 172.21.15.16:0/78970825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f082c10dbe0 0x7f082c1100b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:12.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.195+0000 7f0833bee640 1 --2- 172.21.15.16:0/78970825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c105060 0x7f082c10d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:12.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.195+0000 7f0833bee640 1 --2- 172.21.15.16:0/78970825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f082c104630 0x7f082c104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:12.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.195+0000 7f0833bee640 1 -- 172.21.15.16:0/78970825 >> 172.21.15.16:0/78970825 conn(0x7f082c0794d0 msgr2=0x7f082c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:12.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.195+0000 7f0833bee640 1 -- 172.21.15.16:0/78970825 shutdown_connections 2024-08-28T19:19:12.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.195+0000 7f0833bee640 1 -- 172.21.15.16:0/78970825 wait complete. 2024-08-28T19:19:12.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f0833bee640 1 Processor -- start 2024-08-28T19:19:12.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f0833bee640 1 -- start start 2024-08-28T19:19:12.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f0833bee640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f082c104630 0x7f082c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:12.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f0833bee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c105060 0x7f082c19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:12.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f0833bee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f082c10dbe0 0x7f082c19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:12.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f0833bee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f082c113820 con 0x7f082c10dbe0 2024-08-28T19:19:12.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f0833bee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f082c1136a0 con 0x7f082c105060 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f0833bee640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f082c1139a0 con 0x7f082c104630 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f083218b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f082c10dbe0 0x7f082c19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f083218b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f082c10dbe0 0x7f082c19fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48670/0 (socket says 172.21.15.16:48670) 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f083218b640 1 -- 172.21.15.16:0/2484470963 learned_addr learned my addr 172.21.15.16:0/2484470963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f083198a640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f082c104630 0x7f082c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.196+0000 7f0831189640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c105060 0x7f082c19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.197+0000 7f083218b640 1 -- 172.21.15.16:0/2484470963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f082c104630 msgr2=0x7f082c19a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.197+0000 7f083218b640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f082c104630 0x7f082c19a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.197+0000 7f083218b640 1 -- 172.21.15.16:0/2484470963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c105060 msgr2=0x7f082c19a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.197+0000 7f083218b640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c105060 0x7f082c19a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.197+0000 7f083218b640 1 -- 172.21.15.16:0/2484470963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082c1a00c0 con 0x7f082c10dbe0 2024-08-28T19:19:12.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.197+0000 7f083218b640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f082c10dbe0 0x7f082c19fa50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f08240098d0 tx=0x7f08240317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.197+0000 7f0822ffd640 1 -- 172.21.15.16:0/2484470963 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0824003b70 con 0x7f082c10dbe0 2024-08-28T19:19:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.197+0000 7f0822ffd640 1 -- 172.21.15.16:0/2484470963 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0824003d10 con 0x7f082c10dbe0 2024-08-28T19:19:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.197+0000 7f0833bee640 1 -- 172.21.15.16:0/2484470963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f082c1a0350 con 0x7f082c10dbe0 2024-08-28T19:19:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.197+0000 7f0822ffd640 1 -- 172.21.15.16:0/2484470963 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082402f9d0 con 0x7f082c10dbe0 2024-08-28T19:19:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.197+0000 7f0833bee640 1 -- 172.21.15.16:0/2484470963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f082c1a0890 con 0x7f082c10dbe0 2024-08-28T19:19:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.198+0000 7f0833bee640 1 -- 172.21.15.16:0/2484470963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f082c06bee0 con 0x7f082c10dbe0 2024-08-28T19:19:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.200+0000 7f0822ffd640 1 -- 172.21.15.16:0/2484470963 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f082402fb70 con 0x7f082c10dbe0 2024-08-28T19:19:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.201+0000 7f0822ffd640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08180b4bb0 0x7f08180b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.201+0000 7f0822ffd640 1 -- 172.21.15.16:0/2484470963 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f0824106930 con 0x7f082c10dbe0 2024-08-28T19:19:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.202+0000 7f083198a640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08180b4bb0 0x7f08180b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.202+0000 7f083198a640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08180b4bb0 0x7f08180b70c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f082c06c5e0 tx=0x7f0828005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:12.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.205+0000 7f0822ffd640 1 -- 172.21.15.16:0/2484470963 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f08240cd620 con 0x7f082c10dbe0 2024-08-28T19:19:12.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.428+0000 7f0833bee640 1 -- 172.21.15.16:0/2484470963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f082c19b340 con 0x7f082c10dbe0 2024-08-28T19:19:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.642+0000 7f0822ffd640 1 -- 172.21.15.16:0/2484470963 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v27) ==== 115+0+0 (secure 0 0 0) 0x7f08240b2e20 con 0x7f082c10dbe0 2024-08-28T19:19:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 -- 172.21.15.16:0/2484470963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08180b4bb0 msgr2=0x7f08180b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08180b4bb0 0x7f08180b70c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f082c06c5e0 tx=0x7f0828005f90 comp rx=0 tx=0).stop 2024-08-28T19:19:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 -- 172.21.15.16:0/2484470963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f082c10dbe0 msgr2=0x7f082c19fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f082c10dbe0 0x7f082c19fa50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f08240098d0 tx=0x7f08240317b0 comp rx=0 tx=0).stop 2024-08-28T19:19:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 -- 172.21.15.16:0/2484470963 shutdown_connections 2024-08-28T19:19:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f082c10dbe0 0x7f082c19fa50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08180b4bb0 0x7f08180b70c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c105060 0x7f082c19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 --2- 172.21.15.16:0/2484470963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f082c104630 0x7f082c19a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 -- 172.21.15.16:0/2484470963 >> 172.21.15.16:0/2484470963 conn(0x7f082c0794d0 msgr2=0x7f082c100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 -- 172.21.15.16:0/2484470963 shutdown_connections 2024-08-28T19:19:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:12.644+0000 7f0833bee640 1 -- 172.21.15.16:0/2484470963 wait complete. 2024-08-28T19:19:12.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:12 smithi016 ceph-mon[18556]: pgmap v282: 129 pgs: 129 active+clean; 584 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 10 op/s; 72 B/s, 0 objects/s recovering 2024-08-28T19:19:12.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:12 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] up:active 2024-08-28T19:19:12.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:12 smithi016 ceph-mon[18556]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-08-28T19:19:12.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:12 smithi016 ceph-mon[18556]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-28T19:19:12.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:12 smithi016 ceph-mon[18556]: Cluster is now healthy 2024-08-28T19:19:12.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:12 smithi016 ceph-mon[18556]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-08-28T19:19:12.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:12 smithi016 ceph-mon[18556]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-08-28T19:19:12.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:12 smithi016 ceph-mon[18556]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-08-28T19:19:12.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:12 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2484470963' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-08-28T19:19:13.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:12 smithi028 ceph-mon[21769]: pgmap v282: 129 pgs: 129 active+clean; 584 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 10 op/s; 72 B/s, 0 objects/s recovering 2024-08-28T19:19:13.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:12 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] up:active 2024-08-28T19:19:13.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:12 smithi028 ceph-mon[21769]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-08-28T19:19:13.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:12 smithi028 ceph-mon[21769]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-28T19:19:13.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:12 smithi028 ceph-mon[21769]: Cluster is now healthy 2024-08-28T19:19:13.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:12 smithi028 ceph-mon[21769]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-08-28T19:19:13.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:12 smithi028 ceph-mon[21769]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-08-28T19:19:13.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:12 smithi028 ceph-mon[21769]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-08-28T19:19:13.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:12 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2484470963' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-08-28T19:19:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:12 smithi115 ceph-mon[22184]: pgmap v282: 129 pgs: 129 active+clean; 584 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 10 op/s; 72 B/s, 0 objects/s recovering 2024-08-28T19:19:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:12 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] up:active 2024-08-28T19:19:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:12 smithi115 ceph-mon[22184]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-08-28T19:19:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:12 smithi115 ceph-mon[22184]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-28T19:19:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:12 smithi115 ceph-mon[22184]: Cluster is now healthy 2024-08-28T19:19:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:12 smithi115 ceph-mon[22184]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-08-28T19:19:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:12 smithi115 ceph-mon[22184]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-08-28T19:19:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:12 smithi115 ceph-mon[22184]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-08-28T19:19:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:12 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2484470963' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-08-28T19:19:13.387 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-08-28T19:19:13.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-08-28T19:19:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:13.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:13 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2484470963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-08-28T19:19:13.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:13 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] up:active 2024-08-28T19:19:13.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:13 smithi016 ceph-mon[18556]: fsmap cephfs:5 7 up:standby 5 up:active 2024-08-28T19:19:13.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:13 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:19:14.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:13 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2484470963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-08-28T19:19:14.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:13 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] up:active 2024-08-28T19:19:14.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:13 smithi028 ceph-mon[21769]: fsmap cephfs:5 7 up:standby 5 up:active 2024-08-28T19:19:14.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:13 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:19:14.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:13 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2484470963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-08-28T19:19:14.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:13 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] up:active 2024-08-28T19:19:14.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:13 smithi115 ceph-mon[22184]: fsmap cephfs:5 7 up:standby 5 up:active 2024-08-28T19:19:14.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:13 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:19:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.121+0000 7f438b457640 1 -- 172.21.15.16:0/696894266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4384108260 msgr2=0x7f438410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.121+0000 7f438b457640 1 --2- 172.21.15.16:0/696894266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4384108260 0x7f438410ea90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f437c009ca0 tx=0x7f437c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.121+0000 7f438b457640 1 -- 172.21.15.16:0/696894266 shutdown_connections 2024-08-28T19:19:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.121+0000 7f438b457640 1 --2- 172.21.15.16:0/696894266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4384108260 0x7f438410ea90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.121+0000 7f438b457640 1 --2- 172.21.15.16:0/696894266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4384107910 0x7f4384107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.121+0000 7f438b457640 1 --2- 172.21.15.16:0/696894266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384101fd0 0x7f43841023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.121+0000 7f438b457640 1 -- 172.21.15.16:0/696894266 >> 172.21.15.16:0/696894266 conn(0x7f4384079be0 msgr2=0x7f4384079ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.121+0000 7f438b457640 1 -- 172.21.15.16:0/696894266 shutdown_connections 2024-08-28T19:19:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.121+0000 7f438b457640 1 -- 172.21.15.16:0/696894266 wait complete. 2024-08-28T19:19:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.122+0000 7f438b457640 1 Processor -- start 2024-08-28T19:19:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.122+0000 7f438b457640 1 -- start start 2024-08-28T19:19:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.122+0000 7f438b457640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4384101fd0 0x7f438419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.122+0000 7f438b457640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4384107910 0x7f438419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.122+0000 7f438b457640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384108260 0x7f438419f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.122+0000 7f438b457640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4384111f20 con 0x7f4384101fd0 2024-08-28T19:19:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.122+0000 7f438b457640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4384111da0 con 0x7f4384108260 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.122+0000 7f438b457640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f43841120a0 con 0x7f4384107910 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.122+0000 7f43891f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4384101fd0 0x7f438419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.122+0000 7f43891f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4384101fd0 0x7f438419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48692/0 (socket says 172.21.15.16:48692) 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.122+0000 7f43891f3640 1 -- 172.21.15.16:0/3952976162 learned_addr learned my addr 172.21.15.16:0/3952976162 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f43899f4640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384108260 0x7f438419f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f43891f3640 1 -- 172.21.15.16:0/3952976162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4384107910 msgr2=0x7f438419a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f43891f3640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4384107910 0x7f438419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f43891f3640 1 -- 172.21.15.16:0/3952976162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384108260 msgr2=0x7f438419f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f43891f3640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384108260 0x7f438419f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f43891f3640 1 -- 172.21.15.16:0/3952976162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43841a0010 con 0x7f4384101fd0 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f43899f4640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384108260 0x7f438419f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f43891f3640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4384101fd0 0x7f438419a160 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f438000c920 tx=0x7f438000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f437a7fc640 1 -- 172.21.15.16:0/3952976162 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4380007c70 con 0x7f4384101fd0 2024-08-28T19:19:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f438b457640 1 -- 172.21.15.16:0/3952976162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43841a0300 con 0x7f4384101fd0 2024-08-28T19:19:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f437a7fc640 1 -- 172.21.15.16:0/3952976162 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4380007e10 con 0x7f4384101fd0 2024-08-28T19:19:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f437a7fc640 1 -- 172.21.15.16:0/3952976162 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4380018820 con 0x7f4384101fd0 2024-08-28T19:19:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.123+0000 7f438b457640 1 -- 172.21.15.16:0/3952976162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43841a6b20 con 0x7f4384101fd0 2024-08-28T19:19:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.124+0000 7f438b457640 1 -- 172.21.15.16:0/3952976162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43841023e0 con 0x7f4384101fd0 2024-08-28T19:19:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.126+0000 7f437a7fc640 1 -- 172.21.15.16:0/3952976162 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 152594+0+0 (secure 0 0 0) 0x7f438000f450 con 0x7f4384101fd0 2024-08-28T19:19:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.126+0000 7f437a7fc640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43700b4bb0 0x7f43700b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.126+0000 7f43889f2640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43700b4bb0 0x7f43700b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.126+0000 7f437a7fc640 1 -- 172.21.15.16:0/3952976162 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f43800e5a40 con 0x7f4384101fd0 2024-08-28T19:19:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.127+0000 7f43889f2640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43700b4bb0 0x7f43700b70c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f438406b550 tx=0x7f4374005c70 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.130+0000 7f437a7fc640 1 -- 172.21.15.16:0/3952976162 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f4380014030 con 0x7f4384101fd0 2024-08-28T19:19:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.346+0000 7f438b457640 1 -- 172.21.15.16:0/3952976162 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f43840635a0 con 0x7f43700b4bb0 2024-08-28T19:19:14.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.374+0000 7f437a7fc640 1 -- 172.21.15.16:0/3952976162 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f43840635a0 con 0x7f43700b4bb0 2024-08-28T19:19:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 -- 172.21.15.16:0/3952976162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43700b4bb0 msgr2=0x7f43700b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43700b4bb0 0x7f43700b70c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f438406b550 tx=0x7f4374005c70 comp rx=0 tx=0).stop 2024-08-28T19:19:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 -- 172.21.15.16:0/3952976162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4384101fd0 msgr2=0x7f438419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4384101fd0 0x7f438419a160 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f438000c920 tx=0x7f438000cde0 comp rx=0 tx=0).stop 2024-08-28T19:19:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 -- 172.21.15.16:0/3952976162 shutdown_connections 2024-08-28T19:19:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43700b4bb0 0x7f43700b70c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:14.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384108260 0x7f438419f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:14.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4384107910 0x7f438419a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:14.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 --2- 172.21.15.16:0/3952976162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4384101fd0 0x7f438419a160 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:14.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 -- 172.21.15.16:0/3952976162 >> 172.21.15.16:0/3952976162 conn(0x7f4384079be0 msgr2=0x7f4384108940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:14.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 -- 172.21.15.16:0/3952976162 shutdown_connections 2024-08-28T19:19:14.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:14.376+0000 7f438b457640 1 -- 172.21.15.16:0/3952976162 wait complete. 2024-08-28T19:19:14.559 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:19:14 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b[52931]: 2024-08-28T19:19:14.109+0000 7fd5ccaae640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-08-28T19:19:14.559 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:19:14 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b[52931]: 2024-08-28T19:19:14.109+0000 7fd5ccaae640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-08-28T19:19:14.560 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:19:14 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b[52931]: 2024-08-28T19:19:14.109+0000 7fd5ccaae640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-08-28T19:19:14.560 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:19:14 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b[52931]: 2024-08-28T19:19:14.109+0000 7fd5ccaae640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-08-28T19:19:14.651 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:19:14 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:19:14.351+0000 7f991d3de640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-28T19:19:14.651 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:19:14 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:19:14.351+0000 7f991d3de640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-28T19:19:14.651 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:19:14 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:19:14.351+0000 7f991d3de640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-28T19:19:14.651 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:19:14 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:19:14.351+0000 7f991d3de640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-28T19:19:14.651 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:19:14 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:19:14.351+0000 7f991d3de640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-28T19:19:15.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:14 smithi016 ceph-mon[18556]: pgmap v283: 129 pgs: 129 active+clean; 584 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 10 op/s; 72 B/s, 0 objects/s recovering 2024-08-28T19:19:15.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:14 smithi028 ceph-mon[21769]: pgmap v283: 129 pgs: 129 active+clean; 584 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 10 op/s; 72 B/s, 0 objects/s recovering 2024-08-28T19:19:15.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:14 smithi115 ceph-mon[22184]: pgmap v283: 129 pgs: 129 active+clean; 584 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 10 op/s; 72 B/s, 0 objects/s recovering 2024-08-28T19:19:15.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-08-28T19:19:15.589 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:15.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:15 smithi016 ceph-mon[18556]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:16.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.030+0000 7ff6ae331640 1 -- 172.21.15.16:0/817465048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a8103900 msgr2=0x7ff6a8103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:16.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.030+0000 7ff6ae331640 1 --2- 172.21.15.16:0/817465048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a8103900 0x7ff6a8103d10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff698009900 tx=0x7ff69802a410 comp rx=0 tx=0).stop 2024-08-28T19:19:16.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.030+0000 7ff6ae331640 1 -- 172.21.15.16:0/817465048 shutdown_connections 2024-08-28T19:19:16.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.030+0000 7ff6ae331640 1 --2- 172.21.15.16:0/817465048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a8109b90 0x7ff6a81103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:16.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.030+0000 7ff6ae331640 1 --2- 172.21.15.16:0/817465048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6a8109240 0x7ff6a8109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:16.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.030+0000 7ff6ae331640 1 --2- 172.21.15.16:0/817465048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a8103900 0x7ff6a8103d10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:16.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.030+0000 7ff6ae331640 1 -- 172.21.15.16:0/817465048 >> 172.21.15.16:0/817465048 conn(0x7ff6a807b5f0 msgr2=0x7ff6a807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:16.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.030+0000 7ff6ae331640 1 -- 172.21.15.16:0/817465048 shutdown_connections 2024-08-28T19:19:16.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.030+0000 7ff6ae331640 1 -- 172.21.15.16:0/817465048 wait complete. 2024-08-28T19:19:16.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.031+0000 7ff6ae331640 1 Processor -- start 2024-08-28T19:19:16.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.031+0000 7ff6ae331640 1 -- start start 2024-08-28T19:19:16.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.031+0000 7ff6ae331640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a8103900 0x7ff6a80754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:16.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.031+0000 7ff6ae331640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a8109240 0x7ff6a80759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.031+0000 7ff6ae331640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6a8109b90 0x7ff6a8075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.031+0000 7ff6ae331640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a8113890 con 0x7ff6a8109240 2024-08-28T19:19:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.031+0000 7ff6ae331640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a8113710 con 0x7ff6a8103900 2024-08-28T19:19:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.031+0000 7ff6ae331640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a8113a10 con 0x7ff6a8109b90 2024-08-28T19:19:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.031+0000 7ff6a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a8109240 0x7ff6a80759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.031+0000 7ff6a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a8109240 0x7ff6a80759f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60948/0 (socket says 172.21.15.16:60948) 2024-08-28T19:19:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.031+0000 7ff6a77fe640 1 -- 172.21.15.16:0/1815106837 learned_addr learned my addr 172.21.15.16:0/1815106837 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6a7fff640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a8103900 0x7ff6a80754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6ac8ce640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6a8109b90 0x7ff6a8075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6a77fe640 1 -- 172.21.15.16:0/1815106837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6a8109b90 msgr2=0x7ff6a8075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6a77fe640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6a8109b90 0x7ff6a8075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6a77fe640 1 -- 172.21.15.16:0/1815106837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a8103900 msgr2=0x7ff6a80754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6a77fe640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a8103900 0x7ff6a80754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6a77fe640 1 -- 172.21.15.16:0/1815106837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a81a84b0 con 0x7ff6a8109240 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6a77fe640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a8109240 0x7ff6a80759f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff69c006fd0 tx=0x7ff69c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6a57fa640 1 -- 172.21.15.16:0/1815106837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c011830 con 0x7ff6a8109240 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6ae331640 1 -- 172.21.15.16:0/1815106837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a81a87a0 con 0x7ff6a8109240 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6a57fa640 1 -- 172.21.15.16:0/1815106837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff69c016070 con 0x7ff6a8109240 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6a57fa640 1 -- 172.21.15.16:0/1815106837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c010e00 con 0x7ff6a8109240 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.032+0000 7ff6ae331640 1 -- 172.21.15.16:0/1815106837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a81a8d10 con 0x7ff6a8109240 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.033+0000 7ff6ae331640 1 -- 172.21.15.16:0/1815106837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a8103d10 con 0x7ff6a8109240 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.035+0000 7ff6a57fa640 1 -- 172.21.15.16:0/1815106837 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7ff69c0119d0 con 0x7ff6a8109240 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.035+0000 7ff6a57fa640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff6900b4c10 0x7ff6900b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.036+0000 7ff6a7fff640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff6900b4c10 0x7ff6900b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:16.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.036+0000 7ff6a57fa640 1 -- 172.21.15.16:0/1815106837 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7ff69c0e53d0 con 0x7ff6a8109240 2024-08-28T19:19:16.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.036+0000 7ff6a7fff640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff6900b4c10 0x7ff6900b7120 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff698000c00 tx=0x7ff698031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:16.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.039+0000 7ff6a57fa640 1 -- 172.21.15.16:0/1815106837 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ff69c0abf80 con 0x7ff6a8109240 2024-08-28T19:19:16.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:15 smithi028 ceph-mon[21769]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:15 smithi115 ceph-mon[22184]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:16.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.254+0000 7ff6ae331640 1 -- 172.21.15.16:0/1815106837 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff6a80635a0 con 0x7ff6900b4c10 2024-08-28T19:19:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.282+0000 7ff6a57fa640 1 -- 172.21.15.16:0/1815106837 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff6a80635a0 con 0x7ff6900b4c10 2024-08-28T19:19:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 -- 172.21.15.16:0/1815106837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff6900b4c10 msgr2=0x7ff6900b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff6900b4c10 0x7ff6900b7120 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff698000c00 tx=0x7ff698031400 comp rx=0 tx=0).stop 2024-08-28T19:19:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 -- 172.21.15.16:0/1815106837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a8109240 msgr2=0x7ff6a80759f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a8109240 0x7ff6a80759f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff69c006fd0 tx=0x7ff69c00b8e0 comp rx=0 tx=0).stop 2024-08-28T19:19:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 -- 172.21.15.16:0/1815106837 shutdown_connections 2024-08-28T19:19:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6a8109b90 0x7ff6a8075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a8109240 0x7ff6a80759f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff6900b4c10 0x7ff6900b7120 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 --2- 172.21.15.16:0/1815106837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a8103900 0x7ff6a80754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 -- 172.21.15.16:0/1815106837 >> 172.21.15.16:0/1815106837 conn(0x7ff6a807b5f0 msgr2=0x7ff6a81072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 -- 172.21.15.16:0/1815106837 shutdown_connections 2024-08-28T19:19:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:16.284+0000 7ff6ae331640 1 -- 172.21.15.16:0/1815106837 wait complete. 2024-08-28T19:19:17.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:16 smithi016 ceph-mon[18556]: pgmap v284: 129 pgs: 129 active+clean; 593 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.1 KiB/s wr, 25 op/s; 72 B/s, 0 objects/s recovering 2024-08-28T19:19:17.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:16 smithi016 ceph-mon[18556]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-08-28T19:19:17.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:16 smithi016 ceph-mon[18556]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:17.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:16 smithi028 ceph-mon[21769]: pgmap v284: 129 pgs: 129 active+clean; 593 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.1 KiB/s wr, 25 op/s; 72 B/s, 0 objects/s recovering 2024-08-28T19:19:17.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:16 smithi028 ceph-mon[21769]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-08-28T19:19:17.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:16 smithi028 ceph-mon[21769]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:17.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-08-28T19:19:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:16 smithi115 ceph-mon[22184]: pgmap v284: 129 pgs: 129 active+clean; 593 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.1 KiB/s wr, 25 op/s; 72 B/s, 0 objects/s recovering 2024-08-28T19:19:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:16 smithi115 ceph-mon[22184]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-08-28T19:19:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:16 smithi115 ceph-mon[22184]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:17.326 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:17.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.834+0000 7f8480fb0640 1 -- 172.21.15.16:0/349924007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f847c102880 msgr2=0x7f847c102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:17.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.834+0000 7f8480fb0640 1 --2- 172.21.15.16:0/349924007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f847c102880 0x7f847c102c90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f846c009900 tx=0x7f846c02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:17.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.834+0000 7f8480fb0640 1 -- 172.21.15.16:0/349924007 shutdown_connections 2024-08-28T19:19:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.834+0000 7f8480fb0640 1 --2- 172.21.15.16:0/349924007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f847c108b10 0x7f847c10f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.834+0000 7f8480fb0640 1 --2- 172.21.15.16:0/349924007 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f847c1081c0 0x7f847c1085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.834+0000 7f8480fb0640 1 --2- 172.21.15.16:0/349924007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f847c102880 0x7f847c102c90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.834+0000 7f8480fb0640 1 -- 172.21.15.16:0/349924007 >> 172.21.15.16:0/349924007 conn(0x7f847c079510 msgr2=0x7f847c079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.834+0000 7f8480fb0640 1 -- 172.21.15.16:0/349924007 shutdown_connections 2024-08-28T19:19:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.834+0000 7f8480fb0640 1 -- 172.21.15.16:0/349924007 wait complete. 2024-08-28T19:19:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.835+0000 7f8480fb0640 1 Processor -- start 2024-08-28T19:19:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.835+0000 7f8480fb0640 1 -- start start 2024-08-28T19:19:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.835+0000 7f8480fb0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f847c102880 0x7f847c19b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.835+0000 7f8480fb0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f847c1081c0 0x7f847c19b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.835+0000 7f8480fb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f847c108b10 0x7f847c19bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.835+0000 7f8480fb0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f847c1127c0 con 0x7f847c108b10 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.835+0000 7f8480fb0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f847c112640 con 0x7f847c102880 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.835+0000 7f8480fb0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f847c112940 con 0x7f847c1081c0 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.835+0000 7f847ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f847c108b10 0x7f847c19bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f847ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f847c108b10 0x7f847c19bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60960/0 (socket says 172.21.15.16:60960) 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f847ad9d640 1 -- 172.21.15.16:0/2689861070 learned_addr learned my addr 172.21.15.16:0/2689861070 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f8479d9b640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f847c1081c0 0x7f847c19b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f847a59c640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f847c102880 0x7f847c19b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f847ad9d640 1 -- 172.21.15.16:0/2689861070 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f847c1081c0 msgr2=0x7f847c19b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f847ad9d640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f847c1081c0 0x7f847c19b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f847ad9d640 1 -- 172.21.15.16:0/2689861070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f847c102880 msgr2=0x7f847c19b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f847ad9d640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f847c102880 0x7f847c19b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f847ad9d640 1 -- 172.21.15.16:0/2689861070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847c1a6350 con 0x7f847c108b10 2024-08-28T19:19:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f847ad9d640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f847c108b10 0x7f847c19bb10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f847400eaa0 tx=0x7f847400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f846b7fe640 1 -- 172.21.15.16:0/2689861070 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847400cdb0 con 0x7f847c108b10 2024-08-28T19:19:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f846b7fe640 1 -- 172.21.15.16:0/2689861070 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8474005020 con 0x7f847c108b10 2024-08-28T19:19:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f8480fb0640 1 -- 172.21.15.16:0/2689861070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f847c1a6640 con 0x7f847c108b10 2024-08-28T19:19:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f846b7fe640 1 -- 172.21.15.16:0/2689861070 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84740116d0 con 0x7f847c108b10 2024-08-28T19:19:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.836+0000 7f8480fb0640 1 -- 172.21.15.16:0/2689861070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f847c1a6b30 con 0x7f847c108b10 2024-08-28T19:19:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.837+0000 7f8480fb0640 1 -- 172.21.15.16:0/2689861070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f847c103b00 con 0x7f847c108b10 2024-08-28T19:19:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.839+0000 7f846b7fe640 1 -- 172.21.15.16:0/2689861070 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f84740051d0 con 0x7f847c108b10 2024-08-28T19:19:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.840+0000 7f846b7fe640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f84640b4c10 0x7f84640b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.840+0000 7f847a59c640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f84640b4c10 0x7f84640b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.840+0000 7f846b7fe640 1 -- 172.21.15.16:0/2689861070 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f84740e5c70 con 0x7f847c108b10 2024-08-28T19:19:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.840+0000 7f847a59c640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f84640b4c10 0x7f84640b7120 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f846c000c00 tx=0x7f846c031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:17.843+0000 7f846b7fe640 1 -- 172.21.15.16:0/2689861070 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8474016030 con 0x7f847c108b10 2024-08-28T19:19:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.061+0000 7f8480fb0640 1 -- 172.21.15.16:0/2689861070 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f847c10a8c0 con 0x7f84640b4c10 2024-08-28T19:19:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.066+0000 7f846b7fe640 1 -- 172.21.15.16:0/2689861070 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f847c10a8c0 con 0x7f84640b4c10 2024-08-28T19:19:18.068 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_0/ceefa1a7-04bd-4cb7-8da4-3abd442dc9f3 2024-08-28T19:19:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 -- 172.21.15.16:0/2689861070 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f84640b4c10 msgr2=0x7f84640b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f84640b4c10 0x7f84640b7120 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f846c000c00 tx=0x7f846c031400 comp rx=0 tx=0).stop 2024-08-28T19:19:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 -- 172.21.15.16:0/2689861070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f847c108b10 msgr2=0x7f847c19bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f847c108b10 0x7f847c19bb10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f847400eaa0 tx=0x7f847400ef60 comp rx=0 tx=0).stop 2024-08-28T19:19:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 -- 172.21.15.16:0/2689861070 shutdown_connections 2024-08-28T19:19:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f847c108b10 0x7f847c19bb10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f847c1081c0 0x7f847c19b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f84640b4c10 0x7f84640b7120 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 --2- 172.21.15.16:0/2689861070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f847c102880 0x7f847c19b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 -- 172.21.15.16:0/2689861070 >> 172.21.15.16:0/2689861070 conn(0x7f847c079510 msgr2=0x7f847c106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 -- 172.21.15.16:0/2689861070 shutdown_connections 2024-08-28T19:19:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:18.068+0000 7f8480fb0640 1 -- 172.21.15.16:0/2689861070 wait complete. 2024-08-28T19:19:18.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-08-28T19:19:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:19.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:18 smithi016 ceph-mon[18556]: pgmap v285: 129 pgs: 129 active+clean; 593 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 6.0 KiB/s wr, 23 op/s; 59 B/s, 0 objects/s recovering 2024-08-28T19:19:19.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:18 smithi016 ceph-mon[18556]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:19.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:18 smithi028 ceph-mon[21769]: pgmap v285: 129 pgs: 129 active+clean; 593 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 6.0 KiB/s wr, 23 op/s; 59 B/s, 0 objects/s recovering 2024-08-28T19:19:19.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:18 smithi028 ceph-mon[21769]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:19.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:18 smithi115 ceph-mon[22184]: pgmap v285: 129 pgs: 129 active+clean; 593 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 6.0 KiB/s wr, 23 op/s; 59 B/s, 0 objects/s recovering 2024-08-28T19:19:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:18 smithi115 ceph-mon[22184]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:19.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.409+0000 7f2796809640 1 -- 172.21.15.16:0/3551469702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2790107920 msgr2=0x7f2790107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.409+0000 7f2796809640 1 --2- 172.21.15.16:0/3551469702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2790107920 0x7f2790107d30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f278400aa10 tx=0x7f278402b720 comp rx=0 tx=0).stop 2024-08-28T19:19:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.409+0000 7f2796809640 1 -- 172.21.15.16:0/3551469702 shutdown_connections 2024-08-28T19:19:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.409+0000 7f2796809640 1 --2- 172.21.15.16:0/3551469702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2790108270 0x7f279010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.409+0000 7f2796809640 1 --2- 172.21.15.16:0/3551469702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2790107920 0x7f2790107d30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.409+0000 7f2796809640 1 --2- 172.21.15.16:0/3551469702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2790101fe0 0x7f27901023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.409+0000 7f2796809640 1 -- 172.21.15.16:0/3551469702 >> 172.21.15.16:0/3551469702 conn(0x7f2790079d30 msgr2=0x7f279007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.409+0000 7f2796809640 1 -- 172.21.15.16:0/3551469702 shutdown_connections 2024-08-28T19:19:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.409+0000 7f2796809640 1 -- 172.21.15.16:0/3551469702 wait complete. 2024-08-28T19:19:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.410+0000 7f2796809640 1 Processor -- start 2024-08-28T19:19:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.410+0000 7f2796809640 1 -- start start 2024-08-28T19:19:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f2796809640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2790101fe0 0x7f279019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f2796809640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2790107920 0x7f279019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f2796809640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2790108270 0x7f279019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f2796809640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27901116b0 con 0x7f2790101fe0 2024-08-28T19:19:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f2796809640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2790111530 con 0x7f2790107920 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f2796809640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2790111830 con 0x7f2790108270 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2790101fe0 0x7f279019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2790101fe0 0x7f279019a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60970/0 (socket says 172.21.15.16:60970) 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278ffff640 1 -- 172.21.15.16:0/239506301 learned_addr learned my addr 172.21.15.16:0/239506301 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278f7fe640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2790107920 0x7f279019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f2794da6640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2790108270 0x7f279019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278ffff640 1 -- 172.21.15.16:0/239506301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2790108270 msgr2=0x7f279019faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278ffff640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2790108270 0x7f279019faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278ffff640 1 -- 172.21.15.16:0/239506301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2790107920 msgr2=0x7f279019a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278ffff640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2790107920 0x7f279019a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278ffff640 1 -- 172.21.15.16:0/239506301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27901a0220 con 0x7f2790101fe0 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278f7fe640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2790107920 0x7f279019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278ffff640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2790101fe0 0x7f279019a2b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f278000e9f0 tx=0x7f278000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.411+0000 7f278d7fa640 1 -- 172.21.15.16:0/239506301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278000cd80 con 0x7f2790101fe0 2024-08-28T19:19:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.412+0000 7f2796809640 1 -- 172.21.15.16:0/239506301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27901a0510 con 0x7f2790101fe0 2024-08-28T19:19:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.412+0000 7f278d7fa640 1 -- 172.21.15.16:0/239506301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27800108e0 con 0x7f2790101fe0 2024-08-28T19:19:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.412+0000 7f278d7fa640 1 -- 172.21.15.16:0/239506301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780018980 con 0x7f2790101fe0 2024-08-28T19:19:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.412+0000 7f2796809640 1 -- 172.21.15.16:0/239506301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27901a6c70 con 0x7f2790101fe0 2024-08-28T19:19:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.413+0000 7f2796809640 1 -- 172.21.15.16:0/239506301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27901023f0 con 0x7f2790101fe0 2024-08-28T19:19:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.414+0000 7f278d7fa640 1 -- 172.21.15.16:0/239506301 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f2780010430 con 0x7f2790101fe0 2024-08-28T19:19:19.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.415+0000 7f278d7fa640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27780b4c10 0x7f27780b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:19.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.415+0000 7f278f7fe640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27780b4c10 0x7f27780b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:19.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.415+0000 7f278d7fa640 1 -- 172.21.15.16:0/239506301 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f27800e5ff0 con 0x7f2790101fe0 2024-08-28T19:19:19.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.416+0000 7f278f7fe640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27780b4c10 0x7f27780b7120 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f278400a340 tx=0x7f2784030510 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:19.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.419+0000 7f278d7fa640 1 -- 172.21.15.16:0/239506301 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f27800acba0 con 0x7f2790101fe0 2024-08-28T19:19:19.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.636+0000 7f2796809640 1 -- 172.21.15.16:0/239506301 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f27900635a0 con 0x7f27780b4c10 2024-08-28T19:19:19.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.659+0000 7f278d7fa640 1 -- 172.21.15.16:0/239506301 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f27900635a0 con 0x7f27780b4c10 2024-08-28T19:19:19.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.661+0000 7f2796809640 1 -- 172.21.15.16:0/239506301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27780b4c10 msgr2=0x7f27780b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:19.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.661+0000 7f2796809640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27780b4c10 0x7f27780b7120 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f278400a340 tx=0x7f2784030510 comp rx=0 tx=0).stop 2024-08-28T19:19:19.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.661+0000 7f2796809640 1 -- 172.21.15.16:0/239506301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2790101fe0 msgr2=0x7f279019a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:19.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.661+0000 7f2796809640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2790101fe0 0x7f279019a2b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f278000e9f0 tx=0x7f278000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:19:19.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.661+0000 7f2796809640 1 -- 172.21.15.16:0/239506301 shutdown_connections 2024-08-28T19:19:19.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.661+0000 7f2796809640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2790108270 0x7f279019faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:19.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.661+0000 7f2796809640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27780b4c10 0x7f27780b7120 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:19.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.661+0000 7f2796809640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2790107920 0x7f279019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:19.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.661+0000 7f2796809640 1 --2- 172.21.15.16:0/239506301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2790101fe0 0x7f279019a2b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:19.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.661+0000 7f2796809640 1 -- 172.21.15.16:0/239506301 >> 172.21.15.16:0/239506301 conn(0x7f2790079d30 msgr2=0x7f27901059b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:19.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.662+0000 7f2796809640 1 -- 172.21.15.16:0/239506301 shutdown_connections 2024-08-28T19:19:19.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:19.662+0000 7f2796809640 1 -- 172.21.15.16:0/239506301 wait complete. 2024-08-28T19:19:20.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-08-28T19:19:20.627 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:20.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:20 smithi016 ceph-mon[18556]: pgmap v286: 129 pgs: 129 active+clean; 594 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 5.5 KiB/s wr, 22 op/s; 53 B/s, 0 objects/s recovering 2024-08-28T19:19:20.918 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:20 smithi016 ceph-mon[18556]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:20.923 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:20 smithi115 ceph-mon[22184]: pgmap v286: 129 pgs: 129 active+clean; 594 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 5.5 KiB/s wr, 22 op/s; 53 B/s, 0 objects/s recovering 2024-08-28T19:19:20.923 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:20 smithi115 ceph-mon[22184]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:21.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:20 smithi028 ceph-mon[21769]: pgmap v286: 129 pgs: 129 active+clean; 594 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 5.5 KiB/s wr, 22 op/s; 53 B/s, 0 objects/s recovering 2024-08-28T19:19:21.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:20 smithi028 ceph-mon[21769]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.111+0000 7f105dcee640 1 -- 172.21.15.16:0/2772438942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1058107900 msgr2=0x7f1058107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.111+0000 7f105dcee640 1 --2- 172.21.15.16:0/2772438942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1058107900 0x7f1058107d10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f104c009900 tx=0x7f104c02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.111+0000 7f105dcee640 1 -- 172.21.15.16:0/2772438942 shutdown_connections 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.111+0000 7f105dcee640 1 --2- 172.21.15.16:0/2772438942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1058108250 0x7f105810ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.111+0000 7f105dcee640 1 --2- 172.21.15.16:0/2772438942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1058107900 0x7f1058107d10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.111+0000 7f105dcee640 1 --2- 172.21.15.16:0/2772438942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1058102010 0x7f1058102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.111+0000 7f105dcee640 1 -- 172.21.15.16:0/2772438942 >> 172.21.15.16:0/2772438942 conn(0x7f1058077a60 msgr2=0x7f1058077ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.111+0000 7f105dcee640 1 -- 172.21.15.16:0/2772438942 shutdown_connections 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.111+0000 7f105dcee640 1 -- 172.21.15.16:0/2772438942 wait complete. 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.112+0000 7f105dcee640 1 Processor -- start 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.112+0000 7f105dcee640 1 -- start start 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.112+0000 7f105dcee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1058102010 0x7f105819b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.112+0000 7f105dcee640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1058107900 0x7f105819b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.112+0000 7f105dcee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1058108250 0x7f105819bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.112+0000 7f105dcee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1058111fb0 con 0x7f1058108250 2024-08-28T19:19:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.112+0000 7f105dcee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1058111e30 con 0x7f1058102010 2024-08-28T19:19:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.112+0000 7f105dcee640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1058112130 con 0x7f1058107900 2024-08-28T19:19:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.112+0000 7f1057fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1058108250 0x7f105819bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.112+0000 7f1057fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1058108250 0x7f105819bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60976/0 (socket says 172.21.15.16:60976) 2024-08-28T19:19:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.112+0000 7f1057fff640 1 -- 172.21.15.16:0/3751073655 learned_addr learned my addr 172.21.15.16:0/3751073655 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f10577fe640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1058102010 0x7f105819b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f1056ffd640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1058107900 0x7f105819b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f1057fff640 1 -- 172.21.15.16:0/3751073655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1058107900 msgr2=0x7f105819b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f1057fff640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1058107900 0x7f105819b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f1057fff640 1 -- 172.21.15.16:0/3751073655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1058102010 msgr2=0x7f105819b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f1057fff640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1058102010 0x7f105819b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f1057fff640 1 -- 172.21.15.16:0/3751073655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10581a64a0 con 0x7f1058108250 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f10577fe640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1058102010 0x7f105819b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f1057fff640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1058108250 0x7f105819bc60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f105000b970 tx=0x7f105000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f1054ff9640 1 -- 172.21.15.16:0/3751073655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1050019070 con 0x7f1058108250 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f1054ff9640 1 -- 172.21.15.16:0/3751073655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1050014440 con 0x7f1058108250 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f105dcee640 1 -- 172.21.15.16:0/3751073655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10581a6790 con 0x7f1058108250 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f1054ff9640 1 -- 172.21.15.16:0/3751073655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10500075d0 con 0x7f1058108250 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.113+0000 7f105dcee640 1 -- 172.21.15.16:0/3751073655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10581a6d70 con 0x7f1058108250 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.114+0000 7f105dcee640 1 -- 172.21.15.16:0/3751073655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10581024a0 con 0x7f1058108250 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.116+0000 7f1054ff9640 1 -- 172.21.15.16:0/3751073655 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f105001c070 con 0x7f1058108250 2024-08-28T19:19:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.116+0000 7f1054ff9640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f10400b4c90 0x7f10400b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.117+0000 7f10577fe640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f10400b4c90 0x7f10400b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.117+0000 7f1054ff9640 1 -- 172.21.15.16:0/3751073655 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f10500e50b0 con 0x7f1058108250 2024-08-28T19:19:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.117+0000 7f10577fe640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f10400b4c90 0x7f10400b71a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1058106290 tx=0x7f1048008040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.120+0000 7f1054ff9640 1 -- 172.21.15.16:0/3751073655 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f10500abc60 con 0x7f1058108250 2024-08-28T19:19:21.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.335+0000 7f105dcee640 1 -- 172.21.15.16:0/3751073655 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f10581011a0 con 0x7f10400b4c90 2024-08-28T19:19:21.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.341+0000 7f1054ff9640 1 -- 172.21.15.16:0/3751073655 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f10581011a0 con 0x7f10400b4c90 2024-08-28T19:19:21.343 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_1/d7e0e328-6597-4453-9c11-805312a036d5 2024-08-28T19:19:21.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.343+0000 7f105dcee640 1 -- 172.21.15.16:0/3751073655 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f10400b4c90 msgr2=0x7f10400b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:21.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.343+0000 7f105dcee640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f10400b4c90 0x7f10400b71a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1058106290 tx=0x7f1048008040 comp rx=0 tx=0).stop 2024-08-28T19:19:21.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.343+0000 7f105dcee640 1 -- 172.21.15.16:0/3751073655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1058108250 msgr2=0x7f105819bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:21.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.343+0000 7f105dcee640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1058108250 0x7f105819bc60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f105000b970 tx=0x7f105000be30 comp rx=0 tx=0).stop 2024-08-28T19:19:21.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.344+0000 7f105dcee640 1 -- 172.21.15.16:0/3751073655 shutdown_connections 2024-08-28T19:19:21.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.344+0000 7f105dcee640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1058108250 0x7f105819bc60 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:21.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.344+0000 7f105dcee640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1058107900 0x7f105819b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:21.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.344+0000 7f105dcee640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f10400b4c90 0x7f10400b71a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:21.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.344+0000 7f105dcee640 1 --2- 172.21.15.16:0/3751073655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1058102010 0x7f105819b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:21.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.344+0000 7f105dcee640 1 -- 172.21.15.16:0/3751073655 >> 172.21.15.16:0/3751073655 conn(0x7f1058077a60 msgr2=0x7f1058100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:21.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.344+0000 7f105dcee640 1 -- 172.21.15.16:0/3751073655 shutdown_connections 2024-08-28T19:19:21.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:21.344+0000 7f105dcee640 1 -- 172.21.15.16:0/3751073655 wait complete. 2024-08-28T19:19:22.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:19:22.254 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.750+0000 7f0372323640 1 -- 172.21.15.16:0/32388186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f036c108aa0 msgr2=0x7f036c10f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.750+0000 7f0372323640 1 --2- 172.21.15.16:0/32388186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f036c108aa0 0x7f036c10f2d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0364009ca0 tx=0x7f036402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.750+0000 7f0372323640 1 -- 172.21.15.16:0/32388186 shutdown_connections 2024-08-28T19:19:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.750+0000 7f0372323640 1 --2- 172.21.15.16:0/32388186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f036c108aa0 0x7f036c10f2d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.750+0000 7f0372323640 1 --2- 172.21.15.16:0/32388186 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f036c108150 0x7f036c108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.750+0000 7f0372323640 1 --2- 172.21.15.16:0/32388186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102810 0x7f036c102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.750+0000 7f0372323640 1 -- 172.21.15.16:0/32388186 >> 172.21.15.16:0/32388186 conn(0x7f036c0794d0 msgr2=0x7f036c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.750+0000 7f0372323640 1 -- 172.21.15.16:0/32388186 shutdown_connections 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.750+0000 7f0372323640 1 -- 172.21.15.16:0/32388186 wait complete. 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0372323640 1 Processor -- start 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0372323640 1 -- start start 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0372323640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102810 0x7f036c19aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0372323640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f036c108150 0x7f036c19b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0372323640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f036c108aa0 0x7f036c19ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0372323640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f036c112700 con 0x7f036c108150 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0372323640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f036c112580 con 0x7f036c102810 2024-08-28T19:19:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0372323640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f036c112880 con 0x7f036c108aa0 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0363fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f036c108150 0x7f036c19b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0363fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f036c108150 0x7f036c19b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60990/0 (socket says 172.21.15.16:60990) 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0363fff640 1 -- 172.21.15.16:0/2494923840 learned_addr learned my addr 172.21.15.16:0/2494923840 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f03708c0640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f036c108aa0 0x7f036c19ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f036bfff640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102810 0x7f036c19aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.751+0000 7f0363fff640 1 -- 172.21.15.16:0/2494923840 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f036c108aa0 msgr2=0x7f036c19ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f0363fff640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f036c108aa0 0x7f036c19ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f0363fff640 1 -- 172.21.15.16:0/2494923840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102810 msgr2=0x7f036c19aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f0363fff640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102810 0x7f036c19aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f0363fff640 1 -- 172.21.15.16:0/2494923840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036c1a62b0 con 0x7f036c108150 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f036bfff640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102810 0x7f036c19aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f03708c0640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f036c108aa0 0x7f036c19ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f0363fff640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f036c108150 0x7f036c19b530 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f035400d770 tx=0x7f035400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f0369ffb640 1 -- 172.21.15.16:0/2494923840 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0354017070 con 0x7f036c108150 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f0369ffb640 1 -- 172.21.15.16:0/2494923840 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0354009c30 con 0x7f036c108150 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f0369ffb640 1 -- 172.21.15.16:0/2494923840 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035400f620 con 0x7f036c108150 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f0372323640 1 -- 172.21.15.16:0/2494923840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f036c1a65a0 con 0x7f036c108150 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.752+0000 7f0372323640 1 -- 172.21.15.16:0/2494923840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f036c1a6b30 con 0x7f036c108150 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.753+0000 7f0372323640 1 -- 172.21.15.16:0/2494923840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f036c103a90 con 0x7f036c108150 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.755+0000 7f0369ffb640 1 -- 172.21.15.16:0/2494923840 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f035401a070 con 0x7f036c108150 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.756+0000 7f0369ffb640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03580b4c90 0x7f03580b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.756+0000 7f036bfff640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03580b4c90 0x7f03580b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.756+0000 7f0369ffb640 1 -- 172.21.15.16:0/2494923840 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f03540e52b0 con 0x7f036c108150 2024-08-28T19:19:22.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.757+0000 7f036bfff640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03580b4c90 0x7f03580b71a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f036c06c400 tx=0x7f035c005e70 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:22.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.759+0000 7f0369ffb640 1 -- 172.21.15.16:0/2494923840 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f0354009de0 con 0x7f036c108150 2024-08-28T19:19:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.984+0000 7f0372323640 1 -- 172.21.15.16:0/2494923840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f036c06bcb0 con 0x7f036c108150 2024-08-28T19:19:22.989 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:22 smithi016 ceph-mon[18556]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:22.989 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:22 smithi016 ceph-mon[18556]: pgmap v287: 129 pgs: 129 active+clean; 626 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.5 KiB/s wr, 24 op/s; 48 B/s, 0 objects/s recovering 2024-08-28T19:19:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.985+0000 7f0369ffb640 1 -- 172.21.15.16:0/2494923840 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11977 (secure 0 0 0) 0x7f03540abee0 con 0x7f036c108150 2024-08-28T19:19:22.989 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:19:22.990 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-08-28T19:19:12:646580+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:19:12.646544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:19:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-08-28T19:19:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.987+0000 7f0372323640 1 -- 172.21.15.16:0/2494923840 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03580b4c90 msgr2=0x7f03580b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.987+0000 7f0372323640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03580b4c90 0x7f03580b71a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f036c06c400 tx=0x7f035c005e70 comp rx=0 tx=0).stop 2024-08-28T19:19:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.987+0000 7f0372323640 1 -- 172.21.15.16:0/2494923840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f036c108150 msgr2=0x7f036c19b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.987+0000 7f0372323640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f036c108150 0x7f036c19b530 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f035400d770 tx=0x7f035400dc30 comp rx=0 tx=0).stop 2024-08-28T19:19:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.988+0000 7f0372323640 1 -- 172.21.15.16:0/2494923840 shutdown_connections 2024-08-28T19:19:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.988+0000 7f0372323640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f036c108aa0 0x7f036c19ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.988+0000 7f0372323640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f036c108150 0x7f036c19b530 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.988+0000 7f0372323640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03580b4c90 0x7f03580b71a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.988+0000 7f0372323640 1 --2- 172.21.15.16:0/2494923840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102810 0x7f036c19aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.988+0000 7f0372323640 1 -- 172.21.15.16:0/2494923840 >> 172.21.15.16:0/2494923840 conn(0x7f036c0794d0 msgr2=0x7f036c105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.988+0000 7f0372323640 1 -- 172.21.15.16:0/2494923840 shutdown_connections 2024-08-28T19:19:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:22.988+0000 7f0372323640 1 -- 172.21.15.16:0/2494923840 wait complete. 2024-08-28T19:19:23.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:22 smithi028 ceph-mon[21769]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:23.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:22 smithi028 ceph-mon[21769]: pgmap v287: 129 pgs: 129 active+clean; 626 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.5 KiB/s wr, 24 op/s; 48 B/s, 0 objects/s recovering 2024-08-28T19:19:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:22 smithi115 ceph-mon[22184]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:19:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:22 smithi115 ceph-mon[22184]: pgmap v287: 129 pgs: 129 active+clean; 626 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.5 KiB/s wr, 24 op/s; 48 B/s, 0 objects/s recovering 2024-08-28T19:19:23.637 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:19:23.892 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:24.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:23 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2494923840' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:19:24.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:23 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2494923840' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:19:24.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:23 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2494923840' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:19:24.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.358+0000 7fc5c59be640 1 -- 172.21.15.16:0/198647012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c0108160 msgr2=0x7fc5c0108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:24.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.358+0000 7fc5c59be640 1 --2- 172.21.15.16:0/198647012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c0108160 0x7fc5c0108570 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009900 tx=0x7fc5b402a410 comp rx=0 tx=0).stop 2024-08-28T19:19:24.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.358+0000 7fc5c59be640 1 -- 172.21.15.16:0/198647012 shutdown_connections 2024-08-28T19:19:24.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.358+0000 7fc5c59be640 1 --2- 172.21.15.16:0/198647012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5c0108ab0 0x7fc5c010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.358+0000 7fc5c59be640 1 --2- 172.21.15.16:0/198647012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c0108160 0x7fc5c0108570 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.358+0000 7fc5c59be640 1 --2- 172.21.15.16:0/198647012 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c0102870 0x7fc5c0102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.358+0000 7fc5c59be640 1 -- 172.21.15.16:0/198647012 >> 172.21.15.16:0/198647012 conn(0x7fc5c0079470 msgr2=0x7fc5c0079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.358+0000 7fc5c59be640 1 -- 172.21.15.16:0/198647012 shutdown_connections 2024-08-28T19:19:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.358+0000 7fc5c59be640 1 -- 172.21.15.16:0/198647012 wait complete. 2024-08-28T19:19:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5c59be640 1 Processor -- start 2024-08-28T19:19:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5c59be640 1 -- start start 2024-08-28T19:19:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5c59be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5c0102870 0x7fc5c0199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5c59be640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c0108160 0x7fc5c0199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5c59be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c0108ab0 0x7fc5c019e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5c59be640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c01127e0 con 0x7fc5c0108ab0 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5c59be640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c0112660 con 0x7fc5c0102870 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5c59be640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c0112960 con 0x7fc5c0108160 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5bf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c0108ab0 0x7fc5c019e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5bf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c0108ab0 0x7fc5c019e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36118/0 (socket says 172.21.15.16:36118) 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5bf7fe640 1 -- 172.21.15.16:0/2086728546 learned_addr learned my addr 172.21.15.16:0/2086728546 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5be7fc640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c0108160 0x7fc5c0199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.359+0000 7fc5beffd640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5c0102870 0x7fc5c0199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.360+0000 7fc5bf7fe640 1 -- 172.21.15.16:0/2086728546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c0108160 msgr2=0x7fc5c0199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.360+0000 7fc5bf7fe640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c0108160 0x7fc5c0199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.360+0000 7fc5bf7fe640 1 -- 172.21.15.16:0/2086728546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5c0102870 msgr2=0x7fc5c0199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.360+0000 7fc5bf7fe640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5c0102870 0x7fc5c0199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.360+0000 7fc5bf7fe640 1 -- 172.21.15.16:0/2086728546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c019f100 con 0x7fc5c0108ab0 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.360+0000 7fc5bf7fe640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c0108ab0 0x7fc5c019e9d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800eaa0 tx=0x7fc5b800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.360+0000 7fc5c49bc640 1 -- 172.21.15.16:0/2086728546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800cdb0 con 0x7fc5c0108ab0 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.360+0000 7fc5c59be640 1 -- 172.21.15.16:0/2086728546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c019f3f0 con 0x7fc5c0108ab0 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.360+0000 7fc5c49bc640 1 -- 172.21.15.16:0/2086728546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5b8005020 con 0x7fc5c0108ab0 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.360+0000 7fc5c49bc640 1 -- 172.21.15.16:0/2086728546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b80116d0 con 0x7fc5c0108ab0 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.360+0000 7fc5c59be640 1 -- 172.21.15.16:0/2086728546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c01a5b50 con 0x7fc5c0108ab0 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.361+0000 7fc5c59be640 1 -- 172.21.15.16:0/2086728546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c0103aa0 con 0x7fc5c0108ab0 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.363+0000 7fc5c49bc640 1 -- 172.21.15.16:0/2086728546 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fc5b80051d0 con 0x7fc5c0108ab0 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.363+0000 7fc5c49bc640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5a80b4c10 0x7fc5a80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.364+0000 7fc5beffd640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5a80b4c10 0x7fc5a80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.364+0000 7fc5c49bc640 1 -- 172.21.15.16:0/2086728546 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fc5b80e5d70 con 0x7fc5c0108ab0 2024-08-28T19:19:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.364+0000 7fc5beffd640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5a80b4c10 0x7fc5a80b7120 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009890 tx=0x7fc5b0008040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:24.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.367+0000 7fc5c49bc640 1 -- 172.21.15.16:0/2086728546 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fc5b8016030 con 0x7fc5c0108ab0 2024-08-28T19:19:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.552+0000 7fc5c59be640 1 -- 172.21.15.16:0/2086728546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5c0102c80 con 0x7fc5c0108ab0 2024-08-28T19:19:24.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.553+0000 7fc5c49bc640 1 -- 172.21.15.16:0/2086728546 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+24992 (secure 0 0 0) 0x7fc5c0102c80 con 0x7fc5c0108ab0 2024-08-28T19:19:24.555 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:19:24.555 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:19:08.602647+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:19:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.555+0000 7fc5c59be640 1 -- 172.21.15.16:0/2086728546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5a80b4c10 msgr2=0x7fc5a80b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.555+0000 7fc5c59be640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5a80b4c10 0x7fc5a80b7120 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009890 tx=0x7fc5b0008040 comp rx=0 tx=0).stop 2024-08-28T19:19:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.555+0000 7fc5c59be640 1 -- 172.21.15.16:0/2086728546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c0108ab0 msgr2=0x7fc5c019e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.555+0000 7fc5c59be640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c0108ab0 0x7fc5c019e9d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800eaa0 tx=0x7fc5b800ef60 comp rx=0 tx=0).stop 2024-08-28T19:19:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.555+0000 7fc5c59be640 1 -- 172.21.15.16:0/2086728546 shutdown_connections 2024-08-28T19:19:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.555+0000 7fc5c59be640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c0108ab0 0x7fc5c019e9d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.555+0000 7fc5c59be640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c0108160 0x7fc5c0199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.555+0000 7fc5c59be640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5a80b4c10 0x7fc5a80b7120 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.555+0000 7fc5c59be640 1 --2- 172.21.15.16:0/2086728546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5c0102870 0x7fc5c0199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.555+0000 7fc5c59be640 1 -- 172.21.15.16:0/2086728546 >> 172.21.15.16:0/2086728546 conn(0x7fc5c0079470 msgr2=0x7fc5c01061b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.556+0000 7fc5c59be640 1 -- 172.21.15.16:0/2086728546 shutdown_connections 2024-08-28T19:19:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:24.556+0000 7fc5c59be640 1 -- 172.21.15.16:0/2086728546 wait complete. 2024-08-28T19:19:25.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:24 smithi016 ceph-mon[18556]: pgmap v288: 129 pgs: 129 active+clean; 626 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.2 KiB/s wr, 18 op/s; 0 B/s, 0 objects/s recovering 2024-08-28T19:19:25.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:24 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2086728546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:25.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:24 smithi028 ceph-mon[21769]: pgmap v288: 129 pgs: 129 active+clean; 626 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.2 KiB/s wr, 18 op/s; 0 B/s, 0 objects/s recovering 2024-08-28T19:19:25.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:24 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2086728546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:24 smithi115 ceph-mon[22184]: pgmap v288: 129 pgs: 129 active+clean; 626 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.2 KiB/s wr, 18 op/s; 0 B/s, 0 objects/s recovering 2024-08-28T19:19:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:24 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2086728546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:25.188 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-28T19:19:25.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-28T19:19:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.945+0000 7f2380fcd640 1 -- 172.21.15.16:0/1542447732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f237c103900 msgr2=0x7f237c103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.945+0000 7f2380fcd640 1 --2- 172.21.15.16:0/1542447732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f237c103900 0x7f237c103d10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f236c009ca0 tx=0x7f236c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.945+0000 7f2380fcd640 1 -- 172.21.15.16:0/1542447732 shutdown_connections 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.945+0000 7f2380fcd640 1 --2- 172.21.15.16:0/1542447732 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f237c109b90 0x7f237c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.945+0000 7f2380fcd640 1 --2- 172.21.15.16:0/1542447732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f237c109240 0x7f237c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.945+0000 7f2380fcd640 1 --2- 172.21.15.16:0/1542447732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f237c103900 0x7f237c103d10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.945+0000 7f2380fcd640 1 -- 172.21.15.16:0/1542447732 >> 172.21.15.16:0/1542447732 conn(0x7f237c07b5f0 msgr2=0x7f237c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.945+0000 7f2380fcd640 1 -- 172.21.15.16:0/1542447732 shutdown_connections 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.945+0000 7f2380fcd640 1 -- 172.21.15.16:0/1542447732 wait complete. 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.946+0000 7f2380fcd640 1 Processor -- start 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.946+0000 7f2380fcd640 1 -- start start 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.946+0000 7f2380fcd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f237c103900 0x7f237c0754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.946+0000 7f2380fcd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f237c109240 0x7f237c0759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.946+0000 7f2380fcd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f237c109b90 0x7f237c075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f2380fcd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f237c113850 con 0x7f237c103900 2024-08-28T19:19:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f2380fcd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f237c1136d0 con 0x7f237c109b90 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f2380fcd640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f237c1139d0 con 0x7f237c109240 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f237a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f237c103900 0x7f237c0754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f237a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f237c103900 0x7f237c0754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36146/0 (socket says 172.21.15.16:36146) 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f237a59c640 1 -- 172.21.15.16:0/2614821209 learned_addr learned my addr 172.21.15.16:0/2614821209 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f2379d9b640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f237c109240 0x7f237c0759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f237ad9d640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f237c109b90 0x7f237c075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f237a59c640 1 -- 172.21.15.16:0/2614821209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f237c109240 msgr2=0x7f237c0759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f237a59c640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f237c109240 0x7f237c0759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f237a59c640 1 -- 172.21.15.16:0/2614821209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f237c109b90 msgr2=0x7f237c075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f237a59c640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f237c109b90 0x7f237c075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f237a59c640 1 -- 172.21.15.16:0/2614821209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237c1a8500 con 0x7f237c103900 2024-08-28T19:19:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f237a59c640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f237c103900 0x7f237c0754b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f236c006010 tx=0x7f236c0319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f23737fe640 1 -- 172.21.15.16:0/2614821209 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c003bf0 con 0x7f237c103900 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f23737fe640 1 -- 172.21.15.16:0/2614821209 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f236c003d90 con 0x7f237c103900 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f23737fe640 1 -- 172.21.15.16:0/2614821209 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c02fc20 con 0x7f237c103900 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f2380fcd640 1 -- 172.21.15.16:0/2614821209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f237c1a8790 con 0x7f237c103900 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.947+0000 7f2380fcd640 1 -- 172.21.15.16:0/2614821209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f237c1a8cd0 con 0x7f237c103900 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.948+0000 7f2380fcd640 1 -- 172.21.15.16:0/2614821209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f237c103d10 con 0x7f237c103900 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.950+0000 7f23737fe640 1 -- 172.21.15.16:0/2614821209 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f236c0046a0 con 0x7f237c103900 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.950+0000 7f23737fe640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f23680b4c10 0x7f23680b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.951+0000 7f2379d9b640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f23680b4c10 0x7f23680b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.951+0000 7f23737fe640 1 -- 172.21.15.16:0/2614821209 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f236c106cf0 con 0x7f237c103900 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.951+0000 7f2379d9b640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f23680b4c10 0x7f23680b7120 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f237c1062d0 tx=0x7f236400a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:25.954+0000 7f23737fe640 1 -- 172.21.15.16:0/2614821209 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f236c0cd840 con 0x7f237c103900 2024-08-28T19:19:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.135+0000 7f2380fcd640 1 -- 172.21.15.16:0/2614821209 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f237c0635a0 con 0x7f23680b4c10 2024-08-28T19:19:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.143+0000 7f23737fe640 1 -- 172.21.15.16:0/2614821209 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340394 (secure 0 0 0) 0x7f237c0635a0 con 0x7f23680b4c10 2024-08-28T19:19:26.148 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:19:26.150 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-08-28T19:19:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 -- 172.21.15.16:0/2614821209 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f23680b4c10 msgr2=0x7f23680b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f23680b4c10 0x7f23680b7120 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f237c1062d0 tx=0x7f236400a2d0 comp rx=0 tx=0).stop 2024-08-28T19:19:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 -- 172.21.15.16:0/2614821209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f237c103900 msgr2=0x7f237c0754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f237c103900 0x7f237c0754b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f236c006010 tx=0x7f236c0319f0 comp rx=0 tx=0).stop 2024-08-28T19:19:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 -- 172.21.15.16:0/2614821209 shutdown_connections 2024-08-28T19:19:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f23680b4c10 0x7f23680b7120 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f237c109b90 0x7f237c075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f237c109240 0x7f237c0759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 --2- 172.21.15.16:0/2614821209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f237c103900 0x7f237c0754b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 -- 172.21.15.16:0/2614821209 >> 172.21.15.16:0/2614821209 conn(0x7f237c07b5f0 msgr2=0x7f237c1072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 -- 172.21.15.16:0/2614821209 shutdown_connections 2024-08-28T19:19:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:26.146+0000 7f2380fcd640 1 -- 172.21.15.16:0/2614821209 wait complete. 2024-08-28T19:19:26.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:26 smithi016 ceph-mon[18556]: pgmap v289: 129 pgs: 129 active+clean; 637 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 KiB/s wr, 21 op/s 2024-08-28T19:19:26.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:26 smithi016 ceph-mon[18556]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:19:26.830 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-08-28T19:19:25.467260+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652198,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":239,"num_read_kb":218,"num_write":362,"num_write_kb":2282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":412400,"kb_used_data":13232,"kb_used_omap":18,"kb_used_meta":398957,"kb_avail":1124529424,"statfs":{"total":1151940427776,"available":1151518130176,"internally_reserved":0,"allocated":13549568,"data_stored":7668898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":408532282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":54378,"num_objects":39,"num_object_clones":0,"num_object_copies":119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":158,"num_read_kb":149,"num_write":103,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003664"},"pg_stats":[{"pgid":"2.1f","version":"82'13","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:19.670524+0000","last_change":"2024-08-28T19:19:08.633326+0000","last_active":"2024-08-28T19:19:19.670524+0000","last_peered":"2024-08-28T19:19:19.670524+0000","last_clean":"2024-08-28T19:19:19.670524+0000","last_became_active":"2024-08-28T19:19:08.623876+0000","last_became_peered":"2024-08-28T19:19:08.623876+0000","last_unstale":"2024-08-28T19:19:19.670524+0000","last_undegraded":"2024-08-28T19:19:19.670524+0000","last_fullsized":"2024-08-28T19:19:19.670524+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:19:50.632337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632322+0000","last_change":"2024-08-28T19:18:32.492604+0000","last_active":"2024-08-28T19:19:08.632322+0000","last_peered":"2024-08-28T19:19:08.632322+0000","last_clean":"2024-08-28T19:19:08.632322+0000","last_became_active":"2024-08-28T19:18:32.489427+0000","last_became_peered":"2024-08-28T19:18:32.489427+0000","last_unstale":"2024-08-28T19:19:08.632322+0000","last_undegraded":"2024-08-28T19:19:08.632322+0000","last_fullsized":"2024-08-28T19:19:08.632322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:36:14.841315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622224+0000","last_change":"2024-08-28T19:18:39.528815+0000","last_active":"2024-08-28T19:19:08.622224+0000","last_peered":"2024-08-28T19:19:08.622224+0000","last_clean":"2024-08-28T19:19:08.622224+0000","last_became_active":"2024-08-28T19:18:39.527781+0000","last_became_peered":"2024-08-28T19:18:39.527781+0000","last_unstale":"2024-08-28T19:19:08.622224+0000","last_undegraded":"2024-08-28T19:19:08.622224+0000","last_fullsized":"2024-08-28T19:19:08.622224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:39.831426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632931+0000","last_change":"2024-08-28T19:18:31.082641+0000","last_active":"2024-08-28T19:19:08.632931+0000","last_peered":"2024-08-28T19:19:08.632931+0000","last_clean":"2024-08-28T19:19:08.632931+0000","last_became_active":"2024-08-28T19:18:31.082260+0000","last_became_peered":"2024-08-28T19:18:31.082260+0000","last_unstale":"2024-08-28T19:19:08.632931+0000","last_undegraded":"2024-08-28T19:19:08.632931+0000","last_fullsized":"2024-08-28T19:19:08.632931+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:09:55.836463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629290+0000","last_change":"2024-08-28T19:18:32.493498+0000","last_active":"2024-08-28T19:19:08.629290+0000","last_peered":"2024-08-28T19:19:08.629290+0000","last_clean":"2024-08-28T19:19:08.629290+0000","last_became_active":"2024-08-28T19:18:32.490969+0000","last_became_peered":"2024-08-28T19:18:32.490969+0000","last_unstale":"2024-08-28T19:19:08.629290+0000","last_undegraded":"2024-08-28T19:19:08.629290+0000","last_fullsized":"2024-08-28T19:19:08.629290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:34:29.537622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617571+0000","last_change":"2024-08-28T19:18:39.520922+0000","last_active":"2024-08-28T19:19:08.617571+0000","last_peered":"2024-08-28T19:19:08.617571+0000","last_clean":"2024-08-28T19:19:08.617571+0000","last_became_active":"2024-08-28T19:18:39.520362+0000","last_became_peered":"2024-08-28T19:18:39.520362+0000","last_unstale":"2024-08-28T19:19:08.617571+0000","last_undegraded":"2024-08-28T19:19:08.617571+0000","last_fullsized":"2024-08-28T19:19:08.617571+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:40.658820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.646751+0000","last_change":"2024-08-28T19:18:30.471870+0000","last_active":"2024-08-28T19:19:12.646751+0000","last_peered":"2024-08-28T19:19:12.646751+0000","last_clean":"2024-08-28T19:19:12.646751+0000","last_became_active":"2024-08-28T19:18:30.471438+0000","last_became_peered":"2024-08-28T19:18:30.471438+0000","last_unstale":"2024-08-28T19:19:12.646751+0000","last_undegraded":"2024-08-28T19:19:12.646751+0000","last_fullsized":"2024-08-28T19:19:12.646751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:44:34.137395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628034+0000","last_change":"2024-08-28T19:18:32.486485+0000","last_active":"2024-08-28T19:19:08.628034+0000","last_peered":"2024-08-28T19:19:08.628034+0000","last_clean":"2024-08-28T19:19:08.628034+0000","last_became_active":"2024-08-28T19:18:32.485888+0000","last_became_peered":"2024-08-28T19:18:32.485888+0000","last_unstale":"2024-08-28T19:19:08.628034+0000","last_undegraded":"2024-08-28T19:19:08.628034+0000","last_fullsized":"2024-08-28T19:19:08.628034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:45:48.757318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.633437+0000","last_change":"2024-08-28T19:19:08.633317+0000","last_active":"2024-08-28T19:19:08.633437+0000","last_peered":"2024-08-28T19:19:08.633437+0000","last_clean":"2024-08-28T19:19:08.633437+0000","last_became_active":"2024-08-28T19:19:08.632689+0000","last_became_peered":"2024-08-28T19:19:08.632689+0000","last_unstale":"2024-08-28T19:19:08.633437+0000","last_undegraded":"2024-08-28T19:19:08.633437+0000","last_fullsized":"2024-08-28T19:19:08.633437+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:35:08.740645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653995+0000","last_change":"2024-08-28T19:18:30.473982+0000","last_active":"2024-08-28T19:19:11.653995+0000","last_peered":"2024-08-28T19:19:11.653995+0000","last_clean":"2024-08-28T19:19:11.653995+0000","last_became_active":"2024-08-28T19:18:30.473552+0000","last_became_peered":"2024-08-28T19:18:30.473552+0000","last_unstale":"2024-08-28T19:19:11.653995+0000","last_undegraded":"2024-08-28T19:19:11.653995+0000","last_fullsized":"2024-08-28T19:19:11.653995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:22:29.435517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616754+0000","last_change":"2024-08-28T19:18:32.487881+0000","last_active":"2024-08-28T19:19:08.616754+0000","last_peered":"2024-08-28T19:19:08.616754+0000","last_clean":"2024-08-28T19:19:08.616754+0000","last_became_active":"2024-08-28T19:18:32.487039+0000","last_became_peered":"2024-08-28T19:18:32.487039+0000","last_unstale":"2024-08-28T19:19:08.616754+0000","last_undegraded":"2024-08-28T19:19:08.616754+0000","last_fullsized":"2024-08-28T19:19:08.616754+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:23:35.757811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617767+0000","last_change":"2024-08-28T19:18:39.518654+0000","last_active":"2024-08-28T19:19:08.617767+0000","last_peered":"2024-08-28T19:19:08.617767+0000","last_clean":"2024-08-28T19:19:08.617767+0000","last_became_active":"2024-08-28T19:18:39.518087+0000","last_became_peered":"2024-08-28T19:18:39.518087+0000","last_unstale":"2024-08-28T19:19:08.617767+0000","last_undegraded":"2024-08-28T19:19:08.617767+0000","last_fullsized":"2024-08-28T19:19:08.617767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:08.284865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617577+0000","last_change":"2024-08-28T19:18:32.491928+0000","last_active":"2024-08-28T19:19:08.617577+0000","last_peered":"2024-08-28T19:19:08.617577+0000","last_clean":"2024-08-28T19:19:08.617577+0000","last_became_active":"2024-08-28T19:18:32.491538+0000","last_became_peered":"2024-08-28T19:18:32.491538+0000","last_unstale":"2024-08-28T19:19:08.617577+0000","last_undegraded":"2024-08-28T19:19:08.617577+0000","last_fullsized":"2024-08-28T19:19:08.617577+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:39:06.077830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.955289+0000","last_change":"2024-08-28T19:19:08.634752+0000","last_active":"2024-08-28T19:19:11.955289+0000","last_peered":"2024-08-28T19:19:11.955289+0000","last_clean":"2024-08-28T19:19:11.955289+0000","last_became_active":"2024-08-28T19:19:08.620784+0000","last_became_peered":"2024-08-28T19:19:08.620784+0000","last_unstale":"2024-08-28T19:19:11.955289+0000","last_undegraded":"2024-08-28T19:19:11.955289+0000","last_fullsized":"2024-08-28T19:19:11.955289+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:51.640445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618419+0000","last_change":"2024-08-28T19:18:39.681413+0000","last_active":"2024-08-28T19:19:08.618419+0000","last_peered":"2024-08-28T19:19:08.618419+0000","last_clean":"2024-08-28T19:19:08.618419+0000","last_became_active":"2024-08-28T19:18:39.680793+0000","last_became_peered":"2024-08-28T19:18:39.680793+0000","last_unstale":"2024-08-28T19:19:08.618419+0000","last_undegraded":"2024-08-28T19:19:08.618419+0000","last_fullsized":"2024-08-28T19:19:08.618419+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:45:43.381019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632212+0000","last_change":"2024-08-28T19:18:32.486739+0000","last_active":"2024-08-28T19:19:08.632212+0000","last_peered":"2024-08-28T19:19:08.632212+0000","last_clean":"2024-08-28T19:19:08.632212+0000","last_became_active":"2024-08-28T19:18:32.486347+0000","last_became_peered":"2024-08-28T19:18:32.486347+0000","last_unstale":"2024-08-28T19:19:08.632212+0000","last_undegraded":"2024-08-28T19:19:08.632212+0000","last_fullsized":"2024-08-28T19:19:08.632212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:27:36.448569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.641964+0000","last_change":"2024-08-28T19:18:30.467690+0000","last_active":"2024-08-28T19:19:10.641964+0000","last_peered":"2024-08-28T19:19:10.641964+0000","last_clean":"2024-08-28T19:19:10.641964+0000","last_became_active":"2024-08-28T19:18:30.467249+0000","last_became_peered":"2024-08-28T19:18:30.467249+0000","last_unstale":"2024-08-28T19:19:10.641964+0000","last_undegraded":"2024-08-28T19:19:10.641964+0000","last_fullsized":"2024-08-28T19:19:10.641964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:37:28.876666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618227+0000","last_change":"2024-08-28T19:18:39.523889+0000","last_active":"2024-08-28T19:19:08.618227+0000","last_peered":"2024-08-28T19:19:08.618227+0000","last_clean":"2024-08-28T19:19:08.618227+0000","last_became_active":"2024-08-28T19:18:39.523384+0000","last_became_peered":"2024-08-28T19:18:39.523384+0000","last_unstale":"2024-08-28T19:19:08.618227+0000","last_undegraded":"2024-08-28T19:19:08.618227+0000","last_fullsized":"2024-08-28T19:19:08.618227+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:57:32.277680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614254+0000","last_change":"2024-08-28T19:18:32.482257+0000","last_active":"2024-08-28T19:19:08.614254+0000","last_peered":"2024-08-28T19:19:08.614254+0000","last_clean":"2024-08-28T19:19:08.614254+0000","last_became_active":"2024-08-28T19:18:32.481884+0000","last_became_peered":"2024-08-28T19:18:32.481884+0000","last_unstale":"2024-08-28T19:19:08.614254+0000","last_undegraded":"2024-08-28T19:19:08.614254+0000","last_fullsized":"2024-08-28T19:19:08.614254+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:56:28.754369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651638+0000","last_change":"2024-08-28T19:18:30.467307+0000","last_active":"2024-08-28T19:19:11.651638+0000","last_peered":"2024-08-28T19:19:11.651638+0000","last_clean":"2024-08-28T19:19:11.651638+0000","last_became_active":"2024-08-28T19:18:30.466942+0000","last_became_peered":"2024-08-28T19:18:30.466942+0000","last_unstale":"2024-08-28T19:19:11.651638+0000","last_undegraded":"2024-08-28T19:19:11.651638+0000","last_fullsized":"2024-08-28T19:19:11.651638+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:05:23.682110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614513+0000","last_change":"2024-08-28T19:18:39.531421+0000","last_active":"2024-08-28T19:19:08.614513+0000","last_peered":"2024-08-28T19:19:08.614513+0000","last_clean":"2024-08-28T19:19:08.614513+0000","last_became_active":"2024-08-28T19:18:39.531008+0000","last_became_peered":"2024-08-28T19:18:39.531008+0000","last_unstale":"2024-08-28T19:19:08.614513+0000","last_undegraded":"2024-08-28T19:19:08.614513+0000","last_fullsized":"2024-08-28T19:19:08.614513+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:46:49.580601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618946+0000","last_change":"2024-08-28T19:18:32.485204+0000","last_active":"2024-08-28T19:19:08.618946+0000","last_peered":"2024-08-28T19:19:08.618946+0000","last_clean":"2024-08-28T19:19:08.618946+0000","last_became_active":"2024-08-28T19:18:32.484830+0000","last_became_peered":"2024-08-28T19:18:32.484830+0000","last_unstale":"2024-08-28T19:19:08.618946+0000","last_undegraded":"2024-08-28T19:19:08.618946+0000","last_fullsized":"2024-08-28T19:19:08.618946+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:55:41.358718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632011+0000","last_change":"2024-08-28T19:18:31.082361+0000","last_active":"2024-08-28T19:19:08.632011+0000","last_peered":"2024-08-28T19:19:08.632011+0000","last_clean":"2024-08-28T19:19:08.632011+0000","last_became_active":"2024-08-28T19:18:31.081930+0000","last_became_peered":"2024-08-28T19:18:31.081930+0000","last_unstale":"2024-08-28T19:19:08.632011+0000","last_undegraded":"2024-08-28T19:19:08.632011+0000","last_fullsized":"2024-08-28T19:19:08.632011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:41:05.543088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618390+0000","last_change":"2024-08-28T19:18:39.682341+0000","last_active":"2024-08-28T19:19:08.618390+0000","last_peered":"2024-08-28T19:19:08.618390+0000","last_clean":"2024-08-28T19:19:08.618390+0000","last_became_active":"2024-08-28T19:18:39.681751+0000","last_became_peered":"2024-08-28T19:18:39.681751+0000","last_unstale":"2024-08-28T19:19:08.618390+0000","last_undegraded":"2024-08-28T19:19:08.618390+0000","last_fullsized":"2024-08-28T19:19:08.618390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:51:24.371270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628950+0000","last_change":"2024-08-28T19:18:32.491069+0000","last_active":"2024-08-28T19:19:08.628950+0000","last_peered":"2024-08-28T19:19:08.628950+0000","last_clean":"2024-08-28T19:19:08.628950+0000","last_became_active":"2024-08-28T19:18:32.490688+0000","last_became_peered":"2024-08-28T19:18:32.490688+0000","last_unstale":"2024-08-28T19:19:08.628950+0000","last_undegraded":"2024-08-28T19:19:08.628950+0000","last_fullsized":"2024-08-28T19:19:08.628950+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:24:31.560519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652924+0000","last_change":"2024-08-28T19:18:30.739812+0000","last_active":"2024-08-28T19:19:11.652924+0000","last_peered":"2024-08-28T19:19:11.652924+0000","last_clean":"2024-08-28T19:19:11.652924+0000","last_became_active":"2024-08-28T19:18:30.739414+0000","last_became_peered":"2024-08-28T19:18:30.739414+0000","last_unstale":"2024-08-28T19:19:11.652924+0000","last_undegraded":"2024-08-28T19:19:11.652924+0000","last_fullsized":"2024-08-28T19:19:11.652924+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:56:56.594993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622469+0000","last_change":"2024-08-28T19:18:39.528281+0000","last_active":"2024-08-28T19:19:08.622469+0000","last_peered":"2024-08-28T19:19:08.622469+0000","last_clean":"2024-08-28T19:19:08.622469+0000","last_became_active":"2024-08-28T19:18:39.527142+0000","last_became_peered":"2024-08-28T19:18:39.527142+0000","last_unstale":"2024-08-28T19:19:08.622469+0000","last_undegraded":"2024-08-28T19:19:08.622469+0000","last_fullsized":"2024-08-28T19:19:08.622469+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:44:19.123662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632709+0000","last_change":"2024-08-28T19:18:32.490430+0000","last_active":"2024-08-28T19:19:08.632709+0000","last_peered":"2024-08-28T19:19:08.632709+0000","last_clean":"2024-08-28T19:19:08.632709+0000","last_became_active":"2024-08-28T19:18:32.490020+0000","last_became_peered":"2024-08-28T19:18:32.490020+0000","last_unstale":"2024-08-28T19:19:08.632709+0000","last_undegraded":"2024-08-28T19:19:08.632709+0000","last_fullsized":"2024-08-28T19:19:08.632709+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:52:31.153012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"82'11","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:18.486678+0000","last_change":"2024-08-28T19:18:30.467060+0000","last_active":"2024-08-28T19:19:18.486678+0000","last_peered":"2024-08-28T19:19:18.486678+0000","last_clean":"2024-08-28T19:19:18.486678+0000","last_became_active":"2024-08-28T19:18:30.466689+0000","last_became_peered":"2024-08-28T19:18:30.466689+0000","last_unstale":"2024-08-28T19:19:18.486678+0000","last_undegraded":"2024-08-28T19:19:18.486678+0000","last_fullsized":"2024-08-28T19:19:18.486678+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:18:36.508131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.630939+0000","last_change":"2024-08-28T19:18:39.682874+0000","last_active":"2024-08-28T19:19:08.630939+0000","last_peered":"2024-08-28T19:19:08.630939+0000","last_clean":"2024-08-28T19:19:08.630939+0000","last_became_active":"2024-08-28T19:18:39.681500+0000","last_became_peered":"2024-08-28T19:18:39.681500+0000","last_unstale":"2024-08-28T19:19:08.630939+0000","last_undegraded":"2024-08-28T19:19:08.630939+0000","last_fullsized":"2024-08-28T19:19:08.630939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:45:41.815506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612195+0000","last_change":"2024-08-28T19:18:32.482047+0000","last_active":"2024-08-28T19:19:08.612195+0000","last_peered":"2024-08-28T19:19:08.612195+0000","last_clean":"2024-08-28T19:19:08.612195+0000","last_became_active":"2024-08-28T19:18:32.481640+0000","last_became_peered":"2024-08-28T19:18:32.481640+0000","last_unstale":"2024-08-28T19:19:08.612195+0000","last_undegraded":"2024-08-28T19:19:08.612195+0000","last_fullsized":"2024-08-28T19:19:08.612195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:34.717764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651944+0000","last_change":"2024-08-28T19:18:31.083524+0000","last_active":"2024-08-28T19:19:11.651944+0000","last_peered":"2024-08-28T19:19:11.651944+0000","last_clean":"2024-08-28T19:19:11.651944+0000","last_became_active":"2024-08-28T19:18:31.083106+0000","last_became_peered":"2024-08-28T19:18:31.083106+0000","last_unstale":"2024-08-28T19:19:11.651944+0000","last_undegraded":"2024-08-28T19:19:11.651944+0000","last_fullsized":"2024-08-28T19:19:11.651944+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:16.180754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615954+0000","last_change":"2024-08-28T19:18:39.531884+0000","last_active":"2024-08-28T19:19:08.615954+0000","last_peered":"2024-08-28T19:19:08.615954+0000","last_clean":"2024-08-28T19:19:08.615954+0000","last_became_active":"2024-08-28T19:18:39.531255+0000","last_became_peered":"2024-08-28T19:18:39.531255+0000","last_unstale":"2024-08-28T19:19:08.615954+0000","last_undegraded":"2024-08-28T19:19:08.615954+0000","last_fullsized":"2024-08-28T19:19:08.615954+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:36:55.496594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612691+0000","last_change":"2024-08-28T19:18:32.485180+0000","last_active":"2024-08-28T19:19:08.612691+0000","last_peered":"2024-08-28T19:19:08.612691+0000","last_clean":"2024-08-28T19:19:08.612691+0000","last_became_active":"2024-08-28T19:18:32.484804+0000","last_became_peered":"2024-08-28T19:18:32.484804+0000","last_unstale":"2024-08-28T19:19:08.612691+0000","last_undegraded":"2024-08-28T19:19:08.612691+0000","last_fullsized":"2024-08-28T19:19:08.612691+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:12:40.958251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615096+0000","last_change":"2024-08-28T19:18:31.082226+0000","last_active":"2024-08-28T19:19:08.615096+0000","last_peered":"2024-08-28T19:19:08.615096+0000","last_clean":"2024-08-28T19:19:08.615096+0000","last_became_active":"2024-08-28T19:18:31.081833+0000","last_became_peered":"2024-08-28T19:18:31.081833+0000","last_unstale":"2024-08-28T19:19:08.615096+0000","last_undegraded":"2024-08-28T19:19:08.615096+0000","last_fullsized":"2024-08-28T19:19:08.615096+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:33:56.298093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619013+0000","last_change":"2024-08-28T19:18:39.681288+0000","last_active":"2024-08-28T19:19:08.619013+0000","last_peered":"2024-08-28T19:19:08.619013+0000","last_clean":"2024-08-28T19:19:08.619013+0000","last_became_active":"2024-08-28T19:18:39.680719+0000","last_became_peered":"2024-08-28T19:18:39.680719+0000","last_unstale":"2024-08-28T19:19:08.619013+0000","last_undegraded":"2024-08-28T19:19:08.619013+0000","last_fullsized":"2024-08-28T19:19:08.619013+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:53:35.554181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612370+0000","last_change":"2024-08-28T19:18:32.487105+0000","last_active":"2024-08-28T19:19:08.612370+0000","last_peered":"2024-08-28T19:19:08.612370+0000","last_clean":"2024-08-28T19:19:08.612370+0000","last_became_active":"2024-08-28T19:18:32.486643+0000","last_became_peered":"2024-08-28T19:18:32.486643+0000","last_unstale":"2024-08-28T19:19:08.612370+0000","last_undegraded":"2024-08-28T19:19:08.612370+0000","last_fullsized":"2024-08-28T19:19:08.612370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:18:43.274493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'6","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:23.675365+0000","last_change":"2024-08-28T19:18:30.471447+0000","last_active":"2024-08-28T19:19:23.675365+0000","last_peered":"2024-08-28T19:19:23.675365+0000","last_clean":"2024-08-28T19:19:23.675365+0000","last_became_active":"2024-08-28T19:18:30.470798+0000","last_became_peered":"2024-08-28T19:18:30.470798+0000","last_unstale":"2024-08-28T19:19:23.675365+0000","last_undegraded":"2024-08-28T19:19:23.675365+0000","last_fullsized":"2024-08-28T19:19:23.675365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:50:13.948439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611755+0000","last_change":"2024-08-28T19:18:39.524190+0000","last_active":"2024-08-28T19:19:08.611755+0000","last_peered":"2024-08-28T19:19:08.611755+0000","last_clean":"2024-08-28T19:19:08.611755+0000","last_became_active":"2024-08-28T19:18:39.523771+0000","last_became_peered":"2024-08-28T19:18:39.523771+0000","last_unstale":"2024-08-28T19:19:08.611755+0000","last_undegraded":"2024-08-28T19:19:08.611755+0000","last_fullsized":"2024-08-28T19:19:08.611755+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:43:43.984625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615282+0000","last_change":"2024-08-28T19:18:32.487212+0000","last_active":"2024-08-28T19:19:08.615282+0000","last_peered":"2024-08-28T19:19:08.615282+0000","last_clean":"2024-08-28T19:19:08.615282+0000","last_became_active":"2024-08-28T19:18:32.485432+0000","last_became_peered":"2024-08-28T19:18:32.485432+0000","last_unstale":"2024-08-28T19:19:08.615282+0000","last_undegraded":"2024-08-28T19:19:08.615282+0000","last_fullsized":"2024-08-28T19:19:08.615282+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:52:11.672193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:23.675243+0000","last_change":"2024-08-28T19:18:31.082210+0000","last_active":"2024-08-28T19:19:23.675243+0000","last_peered":"2024-08-28T19:19:23.675243+0000","last_clean":"2024-08-28T19:19:23.675243+0000","last_became_active":"2024-08-28T19:18:31.081814+0000","last_became_peered":"2024-08-28T19:18:31.081814+0000","last_unstale":"2024-08-28T19:19:23.675243+0000","last_undegraded":"2024-08-28T19:19:23.675243+0000","last_fullsized":"2024-08-28T19:19:23.675243+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:33:32.281431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611523+0000","last_change":"2024-08-28T19:18:39.681858+0000","last_active":"2024-08-28T19:19:08.611523+0000","last_peered":"2024-08-28T19:19:08.611523+0000","last_clean":"2024-08-28T19:19:08.611523+0000","last_became_active":"2024-08-28T19:18:39.681382+0000","last_became_peered":"2024-08-28T19:18:39.681382+0000","last_unstale":"2024-08-28T19:19:08.611523+0000","last_undegraded":"2024-08-28T19:19:08.611523+0000","last_fullsized":"2024-08-28T19:19:08.611523+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:23:22.021022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611190+0000","last_change":"2024-08-28T19:18:32.486802+0000","last_active":"2024-08-28T19:19:08.611190+0000","last_peered":"2024-08-28T19:19:08.611190+0000","last_clean":"2024-08-28T19:19:08.611190+0000","last_became_active":"2024-08-28T19:18:32.486479+0000","last_became_peered":"2024-08-28T19:18:32.486479+0000","last_unstale":"2024-08-28T19:19:08.611190+0000","last_undegraded":"2024-08-28T19:19:08.611190+0000","last_fullsized":"2024-08-28T19:19:08.611190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:17:17.654757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:09.635071+0000","last_change":"2024-08-28T19:18:30.470447+0000","last_active":"2024-08-28T19:19:09.635071+0000","last_peered":"2024-08-28T19:19:09.635071+0000","last_clean":"2024-08-28T19:19:09.635071+0000","last_became_active":"2024-08-28T19:18:30.469862+0000","last_became_peered":"2024-08-28T19:18:30.469862+0000","last_unstale":"2024-08-28T19:19:09.635071+0000","last_undegraded":"2024-08-28T19:19:09.635071+0000","last_fullsized":"2024-08-28T19:19:09.635071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:27:46.660492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624752+0000","last_change":"2024-08-28T19:19:08.624635+0000","last_active":"2024-08-28T19:19:08.624752+0000","last_peered":"2024-08-28T19:19:08.624752+0000","last_clean":"2024-08-28T19:19:08.624752+0000","last_became_active":"2024-08-28T19:19:08.622845+0000","last_became_peered":"2024-08-28T19:19:08.622845+0000","last_unstale":"2024-08-28T19:19:08.624752+0000","last_undegraded":"2024-08-28T19:19:08.624752+0000","last_fullsized":"2024-08-28T19:19:08.624752+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:56:40.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631918+0000","last_change":"2024-08-28T19:18:32.487947+0000","last_active":"2024-08-28T19:19:08.631918+0000","last_peered":"2024-08-28T19:19:08.631918+0000","last_clean":"2024-08-28T19:19:08.631918+0000","last_became_active":"2024-08-28T19:18:32.487592+0000","last_became_peered":"2024-08-28T19:18:32.487592+0000","last_unstale":"2024-08-28T19:19:08.631918+0000","last_undegraded":"2024-08-28T19:19:08.631918+0000","last_fullsized":"2024-08-28T19:19:08.631918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:30:29.550644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.654410+0000","last_change":"2024-08-28T19:18:30.465044+0000","last_active":"2024-08-28T19:19:12.654410+0000","last_peered":"2024-08-28T19:19:12.654410+0000","last_clean":"2024-08-28T19:19:12.654410+0000","last_became_active":"2024-08-28T19:18:30.464580+0000","last_became_peered":"2024-08-28T19:18:30.464580+0000","last_unstale":"2024-08-28T19:19:12.654410+0000","last_undegraded":"2024-08-28T19:19:12.654410+0000","last_fullsized":"2024-08-28T19:19:12.654410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:21:17.223311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":109,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617187+0000","last_change":"2024-08-28T19:17:17.063316+0000","last_active":"2024-08-28T19:19:08.617187+0000","last_peered":"2024-08-28T19:19:08.617187+0000","last_clean":"2024-08-28T19:19:08.617187+0000","last_became_active":"2024-08-28T19:17:17.057091+0000","last_became_peered":"2024-08-28T19:17:17.057091+0000","last_unstale":"2024-08-28T19:19:08.617187+0000","last_undegraded":"2024-08-28T19:19:08.617187+0000","last_fullsized":"2024-08-28T19:19:08.617187+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_clean_scrub_stamp":"2024-08-28T19:13:56.815467+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:35:51.499703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"82'1","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622311+0000","last_change":"2024-08-28T19:19:08.621402+0000","last_active":"2024-08-28T19:19:08.622311+0000","last_peered":"2024-08-28T19:19:08.622311+0000","last_clean":"2024-08-28T19:19:08.622311+0000","last_became_active":"2024-08-28T19:19:08.620779+0000","last_became_peered":"2024-08-28T19:19:08.620779+0000","last_unstale":"2024-08-28T19:19:08.622311+0000","last_undegraded":"2024-08-28T19:19:08.622311+0000","last_fullsized":"2024-08-28T19:19:08.622311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:39.391414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619759+0000","last_change":"2024-08-28T19:18:32.490066+0000","last_active":"2024-08-28T19:19:08.619759+0000","last_peered":"2024-08-28T19:19:08.619759+0000","last_clean":"2024-08-28T19:19:08.619759+0000","last_became_active":"2024-08-28T19:18:32.489560+0000","last_became_peered":"2024-08-28T19:18:32.489560+0000","last_unstale":"2024-08-28T19:19:08.619759+0000","last_undegraded":"2024-08-28T19:19:08.619759+0000","last_fullsized":"2024-08-28T19:19:08.619759+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:22:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'5","reported_seq":51,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:19.670614+0000","last_change":"2024-08-28T19:18:31.083078+0000","last_active":"2024-08-28T19:19:19.670614+0000","last_peered":"2024-08-28T19:19:19.670614+0000","last_clean":"2024-08-28T19:19:19.670614+0000","last_became_active":"2024-08-28T19:18:31.082575+0000","last_became_peered":"2024-08-28T19:18:31.082575+0000","last_unstale":"2024-08-28T19:19:19.670614+0000","last_undegraded":"2024-08-28T19:19:19.670614+0000","last_fullsized":"2024-08-28T19:19:19.670614+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627532+0000","last_change":"2024-08-28T19:18:39.519814+0000","last_active":"2024-08-28T19:19:08.627532+0000","last_peered":"2024-08-28T19:19:08.627532+0000","last_clean":"2024-08-28T19:19:08.627532+0000","last_became_active":"2024-08-28T19:18:39.519258+0000","last_became_peered":"2024-08-28T19:18:39.519258+0000","last_unstale":"2024-08-28T19:19:08.627532+0000","last_undegraded":"2024-08-28T19:19:08.627532+0000","last_fullsized":"2024-08-28T19:19:08.627532+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:03:04.051376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618184+0000","last_change":"2024-08-28T19:18:32.495426+0000","last_active":"2024-08-28T19:19:08.618184+0000","last_peered":"2024-08-28T19:19:08.618184+0000","last_clean":"2024-08-28T19:19:08.618184+0000","last_became_active":"2024-08-28T19:18:32.495075+0000","last_became_peered":"2024-08-28T19:18:32.495075+0000","last_unstale":"2024-08-28T19:19:08.618184+0000","last_undegraded":"2024-08-28T19:19:08.618184+0000","last_fullsized":"2024-08-28T19:19:08.618184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:40:22.456666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652460+0000","last_change":"2024-08-28T19:18:30.472708+0000","last_active":"2024-08-28T19:19:11.652460+0000","last_peered":"2024-08-28T19:19:11.652460+0000","last_clean":"2024-08-28T19:19:11.652460+0000","last_became_active":"2024-08-28T19:18:30.472320+0000","last_became_peered":"2024-08-28T19:18:30.472320+0000","last_unstale":"2024-08-28T19:19:11.652460+0000","last_undegraded":"2024-08-28T19:19:11.652460+0000","last_fullsized":"2024-08-28T19:19:11.652460+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:57:12.055626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628468+0000","last_change":"2024-08-28T19:18:39.519398+0000","last_active":"2024-08-28T19:19:08.628468+0000","last_peered":"2024-08-28T19:19:08.628468+0000","last_clean":"2024-08-28T19:19:08.628468+0000","last_became_active":"2024-08-28T19:18:39.518997+0000","last_became_peered":"2024-08-28T19:18:39.518997+0000","last_unstale":"2024-08-28T19:19:08.628468+0000","last_undegraded":"2024-08-28T19:19:08.628468+0000","last_fullsized":"2024-08-28T19:19:08.628468+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:33.431608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619235+0000","last_change":"2024-08-28T19:18:32.489439+0000","last_active":"2024-08-28T19:19:08.619235+0000","last_peered":"2024-08-28T19:19:08.619235+0000","last_clean":"2024-08-28T19:19:08.619235+0000","last_became_active":"2024-08-28T19:18:32.489054+0000","last_became_peered":"2024-08-28T19:18:32.489054+0000","last_unstale":"2024-08-28T19:19:08.619235+0000","last_undegraded":"2024-08-28T19:19:08.619235+0000","last_fullsized":"2024-08-28T19:19:08.619235+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:31:32.647384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651121+0000","last_change":"2024-08-28T19:18:30.464993+0000","last_active":"2024-08-28T19:19:11.651121+0000","last_peered":"2024-08-28T19:19:11.651121+0000","last_clean":"2024-08-28T19:19:11.651121+0000","last_became_active":"2024-08-28T19:18:30.463951+0000","last_became_peered":"2024-08-28T19:18:30.463951+0000","last_unstale":"2024-08-28T19:19:11.651121+0000","last_undegraded":"2024-08-28T19:19:11.651121+0000","last_fullsized":"2024-08-28T19:19:11.651121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:06.780058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614780+0000","last_change":"2024-08-28T19:18:39.533061+0000","last_active":"2024-08-28T19:19:08.614780+0000","last_peered":"2024-08-28T19:19:08.614780+0000","last_clean":"2024-08-28T19:19:08.614780+0000","last_became_active":"2024-08-28T19:18:39.532607+0000","last_became_peered":"2024-08-28T19:18:39.532607+0000","last_unstale":"2024-08-28T19:19:08.614780+0000","last_undegraded":"2024-08-28T19:19:08.614780+0000","last_fullsized":"2024-08-28T19:19:08.614780+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:00:57.997665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615094+0000","last_change":"2024-08-28T19:18:32.487937+0000","last_active":"2024-08-28T19:19:08.615094+0000","last_peered":"2024-08-28T19:19:08.615094+0000","last_clean":"2024-08-28T19:19:08.615094+0000","last_became_active":"2024-08-28T19:18:32.487417+0000","last_became_peered":"2024-08-28T19:18:32.487417+0000","last_unstale":"2024-08-28T19:19:08.615094+0000","last_undegraded":"2024-08-28T19:19:08.615094+0000","last_fullsized":"2024-08-28T19:19:08.615094+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:42.497364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.630439+0000","last_change":"2024-08-28T19:18:30.470742+0000","last_active":"2024-08-28T19:19:10.630439+0000","last_peered":"2024-08-28T19:19:10.630439+0000","last_clean":"2024-08-28T19:19:10.630439+0000","last_became_active":"2024-08-28T19:18:30.470135+0000","last_became_peered":"2024-08-28T19:18:30.470135+0000","last_unstale":"2024-08-28T19:19:10.630439+0000","last_undegraded":"2024-08-28T19:19:10.630439+0000","last_fullsized":"2024-08-28T19:19:10.630439+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:49:39.581119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617294+0000","last_change":"2024-08-28T19:18:39.524052+0000","last_active":"2024-08-28T19:19:08.617294+0000","last_peered":"2024-08-28T19:19:08.617294+0000","last_clean":"2024-08-28T19:19:08.617294+0000","last_became_active":"2024-08-28T19:18:39.523601+0000","last_became_peered":"2024-08-28T19:18:39.523601+0000","last_unstale":"2024-08-28T19:19:08.617294+0000","last_undegraded":"2024-08-28T19:19:08.617294+0000","last_fullsized":"2024-08-28T19:19:08.617294+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:54:53.302424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617504+0000","last_change":"2024-08-28T19:18:32.491151+0000","last_active":"2024-08-28T19:19:08.617504+0000","last_peered":"2024-08-28T19:19:08.617504+0000","last_clean":"2024-08-28T19:19:08.617504+0000","last_became_active":"2024-08-28T19:18:32.490798+0000","last_became_peered":"2024-08-28T19:18:32.490798+0000","last_unstale":"2024-08-28T19:19:08.617504+0000","last_undegraded":"2024-08-28T19:19:08.617504+0000","last_fullsized":"2024-08-28T19:19:08.617504+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:41:00.771019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652641+0000","last_change":"2024-08-28T19:18:31.084029+0000","last_active":"2024-08-28T19:19:11.652641+0000","last_peered":"2024-08-28T19:19:11.652641+0000","last_clean":"2024-08-28T19:19:11.652641+0000","last_became_active":"2024-08-28T19:18:31.083507+0000","last_became_peered":"2024-08-28T19:18:31.083507+0000","last_unstale":"2024-08-28T19:19:11.652641+0000","last_undegraded":"2024-08-28T19:19:11.652641+0000","last_fullsized":"2024-08-28T19:19:11.652641+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:15:36.451702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611795+0000","last_change":"2024-08-28T19:18:39.527249+0000","last_active":"2024-08-28T19:19:08.611795+0000","last_peered":"2024-08-28T19:19:08.611795+0000","last_clean":"2024-08-28T19:19:08.611795+0000","last_became_active":"2024-08-28T19:18:39.526834+0000","last_became_peered":"2024-08-28T19:18:39.526834+0000","last_unstale":"2024-08-28T19:19:08.611795+0000","last_undegraded":"2024-08-28T19:19:08.611795+0000","last_fullsized":"2024-08-28T19:19:08.611795+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:48:02.294738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616892+0000","last_change":"2024-08-28T19:18:32.497337+0000","last_active":"2024-08-28T19:19:08.616892+0000","last_peered":"2024-08-28T19:19:08.616892+0000","last_clean":"2024-08-28T19:19:08.616892+0000","last_became_active":"2024-08-28T19:18:32.496929+0000","last_became_peered":"2024-08-28T19:18:32.496929+0000","last_unstale":"2024-08-28T19:19:08.616892+0000","last_undegraded":"2024-08-28T19:19:08.616892+0000","last_fullsized":"2024-08-28T19:19:08.616892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:14.748765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:22.674522+0000","last_change":"2024-08-28T19:18:30.471353+0000","last_active":"2024-08-28T19:19:22.674522+0000","last_peered":"2024-08-28T19:19:22.674522+0000","last_clean":"2024-08-28T19:19:22.674522+0000","last_became_active":"2024-08-28T19:18:30.470283+0000","last_became_peered":"2024-08-28T19:18:30.470283+0000","last_unstale":"2024-08-28T19:19:22.674522+0000","last_undegraded":"2024-08-28T19:19:22.674522+0000","last_fullsized":"2024-08-28T19:19:22.674522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:09:46.444864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614636+0000","last_change":"2024-08-28T19:18:39.519226+0000","last_active":"2024-08-28T19:19:08.614636+0000","last_peered":"2024-08-28T19:19:08.614636+0000","last_clean":"2024-08-28T19:19:08.614636+0000","last_became_active":"2024-08-28T19:18:39.518738+0000","last_became_peered":"2024-08-28T19:18:39.518738+0000","last_unstale":"2024-08-28T19:19:08.614636+0000","last_undegraded":"2024-08-28T19:19:08.614636+0000","last_fullsized":"2024-08-28T19:19:08.614636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:39:35.114702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617127+0000","last_change":"2024-08-28T19:18:32.493060+0000","last_active":"2024-08-28T19:19:08.617127+0000","last_peered":"2024-08-28T19:19:08.617127+0000","last_clean":"2024-08-28T19:19:08.617127+0000","last_became_active":"2024-08-28T19:18:32.492590+0000","last_became_peered":"2024-08-28T19:18:32.492590+0000","last_unstale":"2024-08-28T19:19:08.617127+0000","last_undegraded":"2024-08-28T19:19:08.617127+0000","last_fullsized":"2024-08-28T19:19:08.617127+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:02:02.165301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653316+0000","last_change":"2024-08-28T19:18:30.470674+0000","last_active":"2024-08-28T19:19:11.653316+0000","last_peered":"2024-08-28T19:19:11.653316+0000","last_clean":"2024-08-28T19:19:11.653316+0000","last_became_active":"2024-08-28T19:18:30.470275+0000","last_became_peered":"2024-08-28T19:18:30.470275+0000","last_unstale":"2024-08-28T19:19:11.653316+0000","last_undegraded":"2024-08-28T19:19:11.653316+0000","last_fullsized":"2024-08-28T19:19:11.653316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:01:48.547315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624414+0000","last_change":"2024-08-28T19:19:08.624194+0000","last_active":"2024-08-28T19:19:08.624414+0000","last_peered":"2024-08-28T19:19:08.624414+0000","last_clean":"2024-08-28T19:19:08.624414+0000","last_became_active":"2024-08-28T19:19:08.623681+0000","last_became_peered":"2024-08-28T19:19:08.623681+0000","last_unstale":"2024-08-28T19:19:08.624414+0000","last_undegraded":"2024-08-28T19:19:08.624414+0000","last_fullsized":"2024-08-28T19:19:08.624414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:21:11.099931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611866+0000","last_change":"2024-08-28T19:18:32.488210+0000","last_active":"2024-08-28T19:19:08.611866+0000","last_peered":"2024-08-28T19:19:08.611866+0000","last_clean":"2024-08-28T19:19:08.611866+0000","last_became_active":"2024-08-28T19:18:32.487853+0000","last_became_peered":"2024-08-28T19:18:32.487853+0000","last_unstale":"2024-08-28T19:19:08.611866+0000","last_undegraded":"2024-08-28T19:19:08.611866+0000","last_fullsized":"2024-08-28T19:19:08.611866+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:20:51.808958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.653792+0000","last_change":"2024-08-28T19:18:30.465306+0000","last_active":"2024-08-28T19:19:12.653792+0000","last_peered":"2024-08-28T19:19:12.653792+0000","last_clean":"2024-08-28T19:19:12.653792+0000","last_became_active":"2024-08-28T19:18:30.464850+0000","last_became_peered":"2024-08-28T19:18:30.464850+0000","last_unstale":"2024-08-28T19:19:12.653792+0000","last_undegraded":"2024-08-28T19:19:12.653792+0000","last_fullsized":"2024-08-28T19:19:12.653792+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:24:26.628999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616620+0000","last_change":"2024-08-28T19:18:39.524005+0000","last_active":"2024-08-28T19:19:08.616620+0000","last_peered":"2024-08-28T19:19:08.616620+0000","last_clean":"2024-08-28T19:19:08.616620+0000","last_became_active":"2024-08-28T19:18:39.523594+0000","last_became_peered":"2024-08-28T19:18:39.523594+0000","last_unstale":"2024-08-28T19:19:08.616620+0000","last_undegraded":"2024-08-28T19:19:08.616620+0000","last_fullsized":"2024-08-28T19:19:08.616620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:35:32.484399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619776+0000","last_change":"2024-08-28T19:18:32.489088+0000","last_active":"2024-08-28T19:19:08.619776+0000","last_peered":"2024-08-28T19:19:08.619776+0000","last_clean":"2024-08-28T19:19:08.619776+0000","last_became_active":"2024-08-28T19:18:32.488717+0000","last_became_peered":"2024-08-28T19:18:32.488717+0000","last_unstale":"2024-08-28T19:19:08.619776+0000","last_undegraded":"2024-08-28T19:19:08.619776+0000","last_fullsized":"2024-08-28T19:19:08.619776+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:25.468919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"82'32","reported_seq":100,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:24.681743+0000","last_change":"2024-08-28T19:18:30.468674+0000","last_active":"2024-08-28T19:19:24.681743+0000","last_peered":"2024-08-28T19:19:24.681743+0000","last_clean":"2024-08-28T19:19:24.681743+0000","last_became_active":"2024-08-28T19:18:30.468300+0000","last_became_peered":"2024-08-28T19:18:30.468300+0000","last_unstale":"2024-08-28T19:19:24.681743+0000","last_undegraded":"2024-08-28T19:19:24.681743+0000","last_fullsized":"2024-08-28T19:19:24.681743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:12:06.237498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54392,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":69,"num_write":34,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614424+0000","last_change":"2024-08-28T19:18:39.526131+0000","last_active":"2024-08-28T19:19:08.614424+0000","last_peered":"2024-08-28T19:19:08.614424+0000","last_clean":"2024-08-28T19:19:08.614424+0000","last_became_active":"2024-08-28T19:18:39.525708+0000","last_became_peered":"2024-08-28T19:18:39.525708+0000","last_unstale":"2024-08-28T19:19:08.614424+0000","last_undegraded":"2024-08-28T19:19:08.614424+0000","last_fullsized":"2024-08-28T19:19:08.614424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:21:52.156487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628145+0000","last_change":"2024-08-28T19:18:32.487634+0000","last_active":"2024-08-28T19:19:08.628145+0000","last_peered":"2024-08-28T19:19:08.628145+0000","last_clean":"2024-08-28T19:19:08.628145+0000","last_became_active":"2024-08-28T19:18:32.486702+0000","last_became_peered":"2024-08-28T19:18:32.486702+0000","last_unstale":"2024-08-28T19:19:08.628145+0000","last_undegraded":"2024-08-28T19:19:08.628145+0000","last_fullsized":"2024-08-28T19:19:08.628145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:08:12.606905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.643469+0000","last_change":"2024-08-28T19:18:31.083494+0000","last_active":"2024-08-28T19:19:10.643469+0000","last_peered":"2024-08-28T19:19:10.643469+0000","last_clean":"2024-08-28T19:19:10.643469+0000","last_became_active":"2024-08-28T19:18:31.083086+0000","last_became_peered":"2024-08-28T19:18:31.083086+0000","last_unstale":"2024-08-28T19:19:10.643469+0000","last_undegraded":"2024-08-28T19:19:10.643469+0000","last_fullsized":"2024-08-28T19:19:10.643469+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:14:54.388364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618783+0000","last_change":"2024-08-28T19:18:39.532155+0000","last_active":"2024-08-28T19:19:08.618783+0000","last_peered":"2024-08-28T19:19:08.618783+0000","last_clean":"2024-08-28T19:19:08.618783+0000","last_became_active":"2024-08-28T19:18:39.531565+0000","last_became_peered":"2024-08-28T19:18:39.531565+0000","last_unstale":"2024-08-28T19:19:08.618783+0000","last_undegraded":"2024-08-28T19:19:08.618783+0000","last_fullsized":"2024-08-28T19:19:08.618783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:29:48.824622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629297+0000","last_change":"2024-08-28T19:18:32.496296+0000","last_active":"2024-08-28T19:19:08.629297+0000","last_peered":"2024-08-28T19:19:08.629297+0000","last_clean":"2024-08-28T19:19:08.629297+0000","last_became_active":"2024-08-28T19:18:32.495680+0000","last_became_peered":"2024-08-28T19:18:32.495680+0000","last_unstale":"2024-08-28T19:19:08.629297+0000","last_undegraded":"2024-08-28T19:19:08.629297+0000","last_fullsized":"2024-08-28T19:19:08.629297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:27:34.544529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652615+0000","last_change":"2024-08-28T19:19:08.631758+0000","last_active":"2024-08-28T19:19:11.652615+0000","last_peered":"2024-08-28T19:19:11.652615+0000","last_clean":"2024-08-28T19:19:11.652615+0000","last_became_active":"2024-08-28T19:19:08.631202+0000","last_became_peered":"2024-08-28T19:19:08.631202+0000","last_unstale":"2024-08-28T19:19:11.652615+0000","last_undegraded":"2024-08-28T19:19:11.652615+0000","last_fullsized":"2024-08-28T19:19:11.652615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:27:43.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614142+0000","last_change":"2024-08-28T19:18:39.531157+0000","last_active":"2024-08-28T19:19:08.614142+0000","last_peered":"2024-08-28T19:19:08.614142+0000","last_clean":"2024-08-28T19:19:08.614142+0000","last_became_active":"2024-08-28T19:18:39.530675+0000","last_became_peered":"2024-08-28T19:18:39.530675+0000","last_unstale":"2024-08-28T19:19:08.614142+0000","last_undegraded":"2024-08-28T19:19:08.614142+0000","last_fullsized":"2024-08-28T19:19:08.614142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:38:40.414854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629314+0000","last_change":"2024-08-28T19:18:32.482182+0000","last_active":"2024-08-28T19:19:08.629314+0000","last_peered":"2024-08-28T19:19:08.629314+0000","last_clean":"2024-08-28T19:19:08.629314+0000","last_became_active":"2024-08-28T19:18:32.481689+0000","last_became_peered":"2024-08-28T19:18:32.481689+0000","last_unstale":"2024-08-28T19:19:08.629314+0000","last_undegraded":"2024-08-28T19:19:08.629314+0000","last_fullsized":"2024-08-28T19:19:08.629314+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:13:29.278469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"82'4","reported_seq":65,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:24.676242+0000","last_change":"2024-08-28T19:18:30.475244+0000","last_active":"2024-08-28T19:19:24.676242+0000","last_peered":"2024-08-28T19:19:24.676242+0000","last_clean":"2024-08-28T19:19:24.676242+0000","last_became_active":"2024-08-28T19:18:30.474861+0000","last_became_peered":"2024-08-28T19:18:30.474861+0000","last_unstale":"2024-08-28T19:19:24.676242+0000","last_undegraded":"2024-08-28T19:19:24.676242+0000","last_fullsized":"2024-08-28T19:19:24.676242+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:27:09.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631762+0000","last_change":"2024-08-28T19:18:39.520022+0000","last_active":"2024-08-28T19:19:08.631762+0000","last_peered":"2024-08-28T19:19:08.631762+0000","last_clean":"2024-08-28T19:19:08.631762+0000","last_became_active":"2024-08-28T19:18:39.519584+0000","last_became_peered":"2024-08-28T19:18:39.519584+0000","last_unstale":"2024-08-28T19:19:08.631762+0000","last_undegraded":"2024-08-28T19:19:08.631762+0000","last_fullsized":"2024-08-28T19:19:08.631762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:27:15.851303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628767+0000","last_change":"2024-08-28T19:18:39.522607+0000","last_active":"2024-08-28T19:19:08.628767+0000","last_peered":"2024-08-28T19:19:08.628767+0000","last_clean":"2024-08-28T19:19:08.628767+0000","last_became_active":"2024-08-28T19:18:39.522204+0000","last_became_peered":"2024-08-28T19:18:39.522204+0000","last_unstale":"2024-08-28T19:19:08.628767+0000","last_undegraded":"2024-08-28T19:19:08.628767+0000","last_fullsized":"2024-08-28T19:19:08.628767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:02:08.865294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614380+0000","last_change":"2024-08-28T19:18:32.485577+0000","last_active":"2024-08-28T19:19:08.614380+0000","last_peered":"2024-08-28T19:19:08.614380+0000","last_clean":"2024-08-28T19:19:08.614380+0000","last_became_active":"2024-08-28T19:18:32.485243+0000","last_became_peered":"2024-08-28T19:18:32.485243+0000","last_unstale":"2024-08-28T19:19:08.614380+0000","last_undegraded":"2024-08-28T19:19:08.614380+0000","last_fullsized":"2024-08-28T19:19:08.614380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:05:53.019365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653809+0000","last_change":"2024-08-28T19:18:30.470670+0000","last_active":"2024-08-28T19:19:11.653809+0000","last_peered":"2024-08-28T19:19:11.653809+0000","last_clean":"2024-08-28T19:19:11.653809+0000","last_became_active":"2024-08-28T19:18:30.470264+0000","last_became_peered":"2024-08-28T19:18:30.470264+0000","last_unstale":"2024-08-28T19:19:11.653809+0000","last_undegraded":"2024-08-28T19:19:11.653809+0000","last_fullsized":"2024-08-28T19:19:11.653809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:35.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615236+0000","last_change":"2024-08-28T19:18:39.530948+0000","last_active":"2024-08-28T19:19:08.615236+0000","last_peered":"2024-08-28T19:19:08.615236+0000","last_clean":"2024-08-28T19:19:08.615236+0000","last_became_active":"2024-08-28T19:18:39.530473+0000","last_became_peered":"2024-08-28T19:18:39.530473+0000","last_unstale":"2024-08-28T19:19:08.615236+0000","last_undegraded":"2024-08-28T19:19:08.615236+0000","last_fullsized":"2024-08-28T19:19:08.615236+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:21:57.504582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632033+0000","last_change":"2024-08-28T19:18:32.486027+0000","last_active":"2024-08-28T19:19:08.632033+0000","last_peered":"2024-08-28T19:19:08.632033+0000","last_clean":"2024-08-28T19:19:08.632033+0000","last_became_active":"2024-08-28T19:18:32.483273+0000","last_became_peered":"2024-08-28T19:18:32.483273+0000","last_unstale":"2024-08-28T19:19:08.632033+0000","last_undegraded":"2024-08-28T19:19:08.632033+0000","last_fullsized":"2024-08-28T19:19:08.632033+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:47:37.680695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:09.637193+0000","last_change":"2024-08-28T19:18:30.464811+0000","last_active":"2024-08-28T19:19:09.637193+0000","last_peered":"2024-08-28T19:19:09.637193+0000","last_clean":"2024-08-28T19:19:09.637193+0000","last_became_active":"2024-08-28T19:18:30.463986+0000","last_became_peered":"2024-08-28T19:18:30.463986+0000","last_unstale":"2024-08-28T19:19:09.637193+0000","last_undegraded":"2024-08-28T19:19:09.637193+0000","last_fullsized":"2024-08-28T19:19:09.637193+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:43:46.582993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.642972+0000","last_change":"2024-08-28T19:19:08.631382+0000","last_active":"2024-08-28T19:19:10.642972+0000","last_peered":"2024-08-28T19:19:10.642972+0000","last_clean":"2024-08-28T19:19:10.642972+0000","last_became_active":"2024-08-28T19:19:08.630869+0000","last_became_peered":"2024-08-28T19:19:08.630869+0000","last_unstale":"2024-08-28T19:19:10.642972+0000","last_undegraded":"2024-08-28T19:19:10.642972+0000","last_fullsized":"2024-08-28T19:19:10.642972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:32:54.942408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615952+0000","last_change":"2024-08-28T19:18:32.484120+0000","last_active":"2024-08-28T19:19:08.615952+0000","last_peered":"2024-08-28T19:19:08.615952+0000","last_clean":"2024-08-28T19:19:08.615952+0000","last_became_active":"2024-08-28T19:18:32.483013+0000","last_became_peered":"2024-08-28T19:18:32.483013+0000","last_unstale":"2024-08-28T19:19:08.615952+0000","last_undegraded":"2024-08-28T19:19:08.615952+0000","last_fullsized":"2024-08-28T19:19:08.615952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:59:41.195873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622328+0000","last_change":"2024-08-28T19:18:39.531046+0000","last_active":"2024-08-28T19:19:08.622328+0000","last_peered":"2024-08-28T19:19:08.622328+0000","last_clean":"2024-08-28T19:19:08.622328+0000","last_became_active":"2024-08-28T19:18:39.530407+0000","last_became_peered":"2024-08-28T19:18:39.530407+0000","last_unstale":"2024-08-28T19:19:08.622328+0000","last_undegraded":"2024-08-28T19:19:08.622328+0000","last_fullsized":"2024-08-28T19:19:08.622328+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:47:15.173021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'6","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:21.672671+0000","last_change":"2024-08-28T19:18:30.471268+0000","last_active":"2024-08-28T19:19:21.672671+0000","last_peered":"2024-08-28T19:19:21.672671+0000","last_clean":"2024-08-28T19:19:21.672671+0000","last_became_active":"2024-08-28T19:18:30.470845+0000","last_became_peered":"2024-08-28T19:18:30.470845+0000","last_unstale":"2024-08-28T19:19:21.672671+0000","last_undegraded":"2024-08-28T19:19:21.672671+0000","last_fullsized":"2024-08-28T19:19:21.672671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:18:18.086205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619195+0000","last_change":"2024-08-28T19:18:32.479728+0000","last_active":"2024-08-28T19:19:08.619195+0000","last_peered":"2024-08-28T19:19:08.619195+0000","last_clean":"2024-08-28T19:19:08.619195+0000","last_became_active":"2024-08-28T19:18:32.478623+0000","last_became_peered":"2024-08-28T19:18:32.478623+0000","last_unstale":"2024-08-28T19:19:08.619195+0000","last_undegraded":"2024-08-28T19:19:08.619195+0000","last_fullsized":"2024-08-28T19:19:08.619195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:14:49.914808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.623192+0000","last_change":"2024-08-28T19:18:39.531308+0000","last_active":"2024-08-28T19:19:08.623192+0000","last_peered":"2024-08-28T19:19:08.623192+0000","last_clean":"2024-08-28T19:19:08.623192+0000","last_became_active":"2024-08-28T19:18:39.530403+0000","last_became_peered":"2024-08-28T19:18:39.530403+0000","last_unstale":"2024-08-28T19:19:08.623192+0000","last_undegraded":"2024-08-28T19:19:08.623192+0000","last_fullsized":"2024-08-28T19:19:08.623192+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:33.139668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614114+0000","last_change":"2024-08-28T19:18:32.494621+0000","last_active":"2024-08-28T19:19:08.614114+0000","last_peered":"2024-08-28T19:19:08.614114+0000","last_clean":"2024-08-28T19:19:08.614114+0000","last_became_active":"2024-08-28T19:18:32.494217+0000","last_became_peered":"2024-08-28T19:18:32.494217+0000","last_unstale":"2024-08-28T19:19:08.614114+0000","last_undegraded":"2024-08-28T19:19:08.614114+0000","last_fullsized":"2024-08-28T19:19:08.614114+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:54:15.419470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622753+0000","last_change":"2024-08-28T19:18:32.484716+0000","last_active":"2024-08-28T19:19:08.622753+0000","last_peered":"2024-08-28T19:19:08.622753+0000","last_clean":"2024-08-28T19:19:08.622753+0000","last_became_active":"2024-08-28T19:18:32.484294+0000","last_became_peered":"2024-08-28T19:18:32.484294+0000","last_unstale":"2024-08-28T19:19:08.622753+0000","last_undegraded":"2024-08-28T19:19:08.622753+0000","last_fullsized":"2024-08-28T19:19:08.622753+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:39:45.045607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.621987+0000","last_change":"2024-08-28T19:18:32.484286+0000","last_active":"2024-08-28T19:19:08.621987+0000","last_peered":"2024-08-28T19:19:08.621987+0000","last_clean":"2024-08-28T19:19:08.621987+0000","last_became_active":"2024-08-28T19:18:32.483354+0000","last_became_peered":"2024-08-28T19:18:32.483354+0000","last_unstale":"2024-08-28T19:19:08.621987+0000","last_undegraded":"2024-08-28T19:19:08.621987+0000","last_fullsized":"2024-08-28T19:19:08.621987+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:09.546749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628505+0000","last_change":"2024-08-28T19:18:32.496985+0000","last_active":"2024-08-28T19:19:08.628505+0000","last_peered":"2024-08-28T19:19:08.628505+0000","last_clean":"2024-08-28T19:19:08.628505+0000","last_became_active":"2024-08-28T19:18:32.495874+0000","last_became_peered":"2024-08-28T19:18:32.495874+0000","last_unstale":"2024-08-28T19:19:08.628505+0000","last_undegraded":"2024-08-28T19:19:08.628505+0000","last_fullsized":"2024-08-28T19:19:08.628505+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:42.329572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612181+0000","last_change":"2024-08-28T19:18:32.485446+0000","last_active":"2024-08-28T19:19:08.612181+0000","last_peered":"2024-08-28T19:19:08.612181+0000","last_clean":"2024-08-28T19:19:08.612181+0000","last_became_active":"2024-08-28T19:18:32.484197+0000","last_became_peered":"2024-08-28T19:18:32.484197+0000","last_unstale":"2024-08-28T19:19:08.612181+0000","last_undegraded":"2024-08-28T19:19:08.612181+0000","last_fullsized":"2024-08-28T19:19:08.612181+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:47:50.527915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616380+0000","last_change":"2024-08-28T19:18:32.489452+0000","last_active":"2024-08-28T19:19:08.616380+0000","last_peered":"2024-08-28T19:19:08.616380+0000","last_clean":"2024-08-28T19:19:08.616380+0000","last_became_active":"2024-08-28T19:18:32.487482+0000","last_became_peered":"2024-08-28T19:18:32.487482+0000","last_unstale":"2024-08-28T19:19:08.616380+0000","last_undegraded":"2024-08-28T19:19:08.616380+0000","last_fullsized":"2024-08-28T19:19:08.616380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:52:08.175080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622242+0000","last_change":"2024-08-28T19:18:32.483850+0000","last_active":"2024-08-28T19:19:08.622242+0000","last_peered":"2024-08-28T19:19:08.622242+0000","last_clean":"2024-08-28T19:19:08.622242+0000","last_became_active":"2024-08-28T19:18:32.483476+0000","last_became_peered":"2024-08-28T19:18:32.483476+0000","last_unstale":"2024-08-28T19:19:08.622242+0000","last_undegraded":"2024-08-28T19:19:08.622242+0000","last_fullsized":"2024-08-28T19:19:08.622242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:32:37.465218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632706+0000","last_change":"2024-08-28T19:18:32.492252+0000","last_active":"2024-08-28T19:19:08.632706+0000","last_peered":"2024-08-28T19:19:08.632706+0000","last_clean":"2024-08-28T19:19:08.632706+0000","last_became_active":"2024-08-28T19:18:32.491659+0000","last_became_peered":"2024-08-28T19:18:32.491659+0000","last_unstale":"2024-08-28T19:19:08.632706+0000","last_undegraded":"2024-08-28T19:19:08.632706+0000","last_fullsized":"2024-08-28T19:19:08.632706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:19:28.901559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615146+0000","last_change":"2024-08-28T19:18:32.491940+0000","last_active":"2024-08-28T19:19:08.615146+0000","last_peered":"2024-08-28T19:19:08.615146+0000","last_clean":"2024-08-28T19:19:08.615146+0000","last_became_active":"2024-08-28T19:18:32.487474+0000","last_became_peered":"2024-08-28T19:18:32.487474+0000","last_unstale":"2024-08-28T19:19:08.615146+0000","last_undegraded":"2024-08-28T19:19:08.615146+0000","last_fullsized":"2024-08-28T19:19:08.615146+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:04:32.508620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631980+0000","last_change":"2024-08-28T19:18:32.485200+0000","last_active":"2024-08-28T19:19:08.631980+0000","last_peered":"2024-08-28T19:19:08.631980+0000","last_clean":"2024-08-28T19:19:08.631980+0000","last_became_active":"2024-08-28T19:18:32.484858+0000","last_became_peered":"2024-08-28T19:18:32.484858+0000","last_unstale":"2024-08-28T19:19:08.631980+0000","last_undegraded":"2024-08-28T19:19:08.631980+0000","last_fullsized":"2024-08-28T19:19:08.631980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:10:57.925659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627792+0000","last_change":"2024-08-28T19:18:32.489903+0000","last_active":"2024-08-28T19:19:08.627792+0000","last_peered":"2024-08-28T19:19:08.627792+0000","last_clean":"2024-08-28T19:19:08.627792+0000","last_became_active":"2024-08-28T19:18:32.488950+0000","last_became_peered":"2024-08-28T19:18:32.488950+0000","last_unstale":"2024-08-28T19:19:08.627792+0000","last_undegraded":"2024-08-28T19:19:08.627792+0000","last_fullsized":"2024-08-28T19:19:08.627792+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:06:42.284177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612366+0000","last_change":"2024-08-28T19:18:32.488236+0000","last_active":"2024-08-28T19:19:08.612366+0000","last_peered":"2024-08-28T19:19:08.612366+0000","last_clean":"2024-08-28T19:19:08.612366+0000","last_became_active":"2024-08-28T19:18:32.487776+0000","last_became_peered":"2024-08-28T19:18:32.487776+0000","last_unstale":"2024-08-28T19:19:08.612366+0000","last_undegraded":"2024-08-28T19:19:08.612366+0000","last_fullsized":"2024-08-28T19:19:08.612366+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:25:58.754740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616789+0000","last_change":"2024-08-28T19:18:32.485860+0000","last_active":"2024-08-28T19:19:08.616789+0000","last_peered":"2024-08-28T19:19:08.616789+0000","last_clean":"2024-08-28T19:19:08.616789+0000","last_became_active":"2024-08-28T19:18:32.485415+0000","last_became_peered":"2024-08-28T19:18:32.485415+0000","last_unstale":"2024-08-28T19:19:08.616789+0000","last_undegraded":"2024-08-28T19:19:08.616789+0000","last_fullsized":"2024-08-28T19:19:08.616789+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:45:32.868651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.623402+0000","last_change":"2024-08-28T19:18:32.485515+0000","last_active":"2024-08-28T19:19:08.623402+0000","last_peered":"2024-08-28T19:19:08.623402+0000","last_clean":"2024-08-28T19:19:08.623402+0000","last_became_active":"2024-08-28T19:18:32.485075+0000","last_became_peered":"2024-08-28T19:18:32.485075+0000","last_unstale":"2024-08-28T19:19:08.623402+0000","last_undegraded":"2024-08-28T19:19:08.623402+0000","last_fullsized":"2024-08-28T19:19:08.623402+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:01:28.035479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614972+0000","last_change":"2024-08-28T19:18:32.484685+0000","last_active":"2024-08-28T19:19:08.614972+0000","last_peered":"2024-08-28T19:19:08.614972+0000","last_clean":"2024-08-28T19:19:08.614972+0000","last_became_active":"2024-08-28T19:18:32.484095+0000","last_became_peered":"2024-08-28T19:18:32.484095+0000","last_unstale":"2024-08-28T19:19:08.614972+0000","last_undegraded":"2024-08-28T19:19:08.614972+0000","last_fullsized":"2024-08-28T19:19:08.614972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:53:27.229186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615703+0000","last_change":"2024-08-28T19:18:32.488030+0000","last_active":"2024-08-28T19:19:08.615703+0000","last_peered":"2024-08-28T19:19:08.615703+0000","last_clean":"2024-08-28T19:19:08.615703+0000","last_became_active":"2024-08-28T19:18:32.487042+0000","last_became_peered":"2024-08-28T19:18:32.487042+0000","last_unstale":"2024-08-28T19:19:08.615703+0000","last_undegraded":"2024-08-28T19:19:08.615703+0000","last_fullsized":"2024-08-28T19:19:08.615703+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:56.966475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627165+0000","last_change":"2024-08-28T19:18:32.489442+0000","last_active":"2024-08-28T19:19:08.627165+0000","last_peered":"2024-08-28T19:19:08.627165+0000","last_clean":"2024-08-28T19:19:08.627165+0000","last_became_active":"2024-08-28T19:18:32.488405+0000","last_became_peered":"2024-08-28T19:18:32.488405+0000","last_unstale":"2024-08-28T19:19:08.627165+0000","last_undegraded":"2024-08-28T19:19:08.627165+0000","last_fullsized":"2024-08-28T19:19:08.627165+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:21:03.026893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628286+0000","last_change":"2024-08-28T19:18:32.496044+0000","last_active":"2024-08-28T19:19:08.628286+0000","last_peered":"2024-08-28T19:19:08.628286+0000","last_clean":"2024-08-28T19:19:08.628286+0000","last_became_active":"2024-08-28T19:18:32.495549+0000","last_became_peered":"2024-08-28T19:18:32.495549+0000","last_unstale":"2024-08-28T19:19:08.628286+0000","last_undegraded":"2024-08-28T19:19:08.628286+0000","last_fullsized":"2024-08-28T19:19:08.628286+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:06:56.123264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628305+0000","last_change":"2024-08-28T19:18:32.492077+0000","last_active":"2024-08-28T19:19:08.628305+0000","last_peered":"2024-08-28T19:19:08.628305+0000","last_clean":"2024-08-28T19:19:08.628305+0000","last_became_active":"2024-08-28T19:18:32.491722+0000","last_became_peered":"2024-08-28T19:18:32.491722+0000","last_unstale":"2024-08-28T19:19:08.628305+0000","last_undegraded":"2024-08-28T19:19:08.628305+0000","last_fullsized":"2024-08-28T19:19:08.628305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:16:21.697045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612237+0000","last_change":"2024-08-28T19:18:32.479845+0000","last_active":"2024-08-28T19:19:08.612237+0000","last_peered":"2024-08-28T19:19:08.612237+0000","last_clean":"2024-08-28T19:19:08.612237+0000","last_became_active":"2024-08-28T19:18:32.477690+0000","last_became_peered":"2024-08-28T19:18:32.477690+0000","last_unstale":"2024-08-28T19:19:08.612237+0000","last_undegraded":"2024-08-28T19:19:08.612237+0000","last_fullsized":"2024-08-28T19:19:08.612237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:11.281415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624748+0000","last_change":"2024-08-28T19:19:08.624185+0000","last_active":"2024-08-28T19:19:08.624748+0000","last_peered":"2024-08-28T19:19:08.624748+0000","last_clean":"2024-08-28T19:19:08.624748+0000","last_became_active":"2024-08-28T19:19:08.623745+0000","last_became_peered":"2024-08-28T19:19:08.623745+0000","last_unstale":"2024-08-28T19:19:08.624748+0000","last_undegraded":"2024-08-28T19:19:08.624748+0000","last_fullsized":"2024-08-28T19:19:08.624748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:56.657173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619629+0000","last_change":"2024-08-28T19:18:32.499038+0000","last_active":"2024-08-28T19:19:08.619629+0000","last_peered":"2024-08-28T19:19:08.619629+0000","last_clean":"2024-08-28T19:19:08.619629+0000","last_became_active":"2024-08-28T19:18:32.498276+0000","last_became_peered":"2024-08-28T19:18:32.498276+0000","last_unstale":"2024-08-28T19:19:08.619629+0000","last_undegraded":"2024-08-28T19:19:08.619629+0000","last_fullsized":"2024-08-28T19:19:08.619629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:17:45.468448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612960+0000","last_change":"2024-08-28T19:18:32.492183+0000","last_active":"2024-08-28T19:19:08.612960+0000","last_peered":"2024-08-28T19:19:08.612960+0000","last_clean":"2024-08-28T19:19:08.612960+0000","last_became_active":"2024-08-28T19:18:32.491815+0000","last_became_peered":"2024-08-28T19:18:32.491815+0000","last_unstale":"2024-08-28T19:19:08.612960+0000","last_undegraded":"2024-08-28T19:19:08.612960+0000","last_fullsized":"2024-08-28T19:19:08.612960+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:13:11.183955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629556+0000","last_change":"2024-08-28T19:18:32.497399+0000","last_active":"2024-08-28T19:19:08.629556+0000","last_peered":"2024-08-28T19:19:08.629556+0000","last_clean":"2024-08-28T19:19:08.629556+0000","last_became_active":"2024-08-28T19:18:32.496901+0000","last_became_peered":"2024-08-28T19:18:32.496901+0000","last_unstale":"2024-08-28T19:19:08.629556+0000","last_undegraded":"2024-08-28T19:19:08.629556+0000","last_fullsized":"2024-08-28T19:19:08.629556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:05:57.634590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631412+0000","last_change":"2024-08-28T19:18:32.495871+0000","last_active":"2024-08-28T19:19:08.631412+0000","last_peered":"2024-08-28T19:19:08.631412+0000","last_clean":"2024-08-28T19:19:08.631412+0000","last_became_active":"2024-08-28T19:18:32.495515+0000","last_became_peered":"2024-08-28T19:18:32.495515+0000","last_unstale":"2024-08-28T19:19:08.631412+0000","last_undegraded":"2024-08-28T19:19:08.631412+0000","last_fullsized":"2024-08-28T19:19:08.631412+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:53:56.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617058+0000","last_change":"2024-08-28T19:18:32.495711+0000","last_active":"2024-08-28T19:19:08.617058+0000","last_peered":"2024-08-28T19:19:08.617058+0000","last_clean":"2024-08-28T19:19:08.617058+0000","last_became_active":"2024-08-28T19:18:32.495271+0000","last_became_peered":"2024-08-28T19:18:32.495271+0000","last_unstale":"2024-08-28T19:19:08.617058+0000","last_undegraded":"2024-08-28T19:19:08.617058+0000","last_fullsized":"2024-08-28T19:19:08.617058+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:05.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612310+0000","last_change":"2024-08-28T19:18:32.489731+0000","last_active":"2024-08-28T19:19:08.612310+0000","last_peered":"2024-08-28T19:19:08.612310+0000","last_clean":"2024-08-28T19:19:08.612310+0000","last_became_active":"2024-08-28T19:18:32.489366+0000","last_became_peered":"2024-08-28T19:18:32.489366+0000","last_unstale":"2024-08-28T19:19:08.612310+0000","last_undegraded":"2024-08-28T19:19:08.612310+0000","last_fullsized":"2024-08-28T19:19:08.612310+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:52:24.180155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617967+0000","last_change":"2024-08-28T19:18:32.490270+0000","last_active":"2024-08-28T19:19:08.617967+0000","last_peered":"2024-08-28T19:19:08.617967+0000","last_clean":"2024-08-28T19:19:08.617967+0000","last_became_active":"2024-08-28T19:18:32.489667+0000","last_became_peered":"2024-08-28T19:18:32.489667+0000","last_unstale":"2024-08-28T19:19:08.617967+0000","last_undegraded":"2024-08-28T19:19:08.617967+0000","last_fullsized":"2024-08-28T19:19:08.617967+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:18.571978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618388+0000","last_change":"2024-08-28T19:18:32.482553+0000","last_active":"2024-08-28T19:19:08.618388+0000","last_peered":"2024-08-28T19:19:08.618388+0000","last_clean":"2024-08-28T19:19:08.618388+0000","last_became_active":"2024-08-28T19:18:32.481161+0000","last_became_peered":"2024-08-28T19:18:32.481161+0000","last_unstale":"2024-08-28T19:19:08.618388+0000","last_undegraded":"2024-08-28T19:19:08.618388+0000","last_fullsized":"2024-08-28T19:19:08.618388+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:13.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628811+0000","last_change":"2024-08-28T19:18:32.492473+0000","last_active":"2024-08-28T19:19:08.628811+0000","last_peered":"2024-08-28T19:19:08.628811+0000","last_clean":"2024-08-28T19:19:08.628811+0000","last_became_active":"2024-08-28T19:18:32.491897+0000","last_became_peered":"2024-08-28T19:18:32.491897+0000","last_unstale":"2024-08-28T19:19:08.628811+0000","last_undegraded":"2024-08-28T19:19:08.628811+0000","last_fullsized":"2024-08-28T19:19:08.628811+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:24.279391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628899+0000","last_change":"2024-08-28T19:18:32.485621+0000","last_active":"2024-08-28T19:19:08.628899+0000","last_peered":"2024-08-28T19:19:08.628899+0000","last_clean":"2024-08-28T19:19:08.628899+0000","last_became_active":"2024-08-28T19:18:32.485267+0000","last_became_peered":"2024-08-28T19:18:32.485267+0000","last_unstale":"2024-08-28T19:19:08.628899+0000","last_undegraded":"2024-08-28T19:19:08.628899+0000","last_fullsized":"2024-08-28T19:19:08.628899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:30:08.932135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615578+0000","last_change":"2024-08-28T19:18:32.488457+0000","last_active":"2024-08-28T19:19:08.615578+0000","last_peered":"2024-08-28T19:19:08.615578+0000","last_clean":"2024-08-28T19:19:08.615578+0000","last_became_active":"2024-08-28T19:18:32.487555+0000","last_became_peered":"2024-08-28T19:18:32.487555+0000","last_unstale":"2024-08-28T19:19:08.615578+0000","last_undegraded":"2024-08-28T19:19:08.615578+0000","last_fullsized":"2024-08-28T19:19:08.615578+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:03:26.390481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61596,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":158,"num_read_kb":149,"num_write":237,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":528384,"data_stored":162686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":68,"seq":292057776151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":481155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":62,"seq":266287972380,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1642496,"data_stored":1125787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":57,"seq":244813135905,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":484509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":51,"seq":219043332134,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":970752,"data_stored":480791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":45,"seq":193273528363,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708284,"statfs":{"total":95995035648,"available":95957282816,"internally_reserved":0,"allocated":1560576,"data_stored":1077007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":40,"seq":171798691889,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":476308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888116,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1572864,"data_stored":1068774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051643,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":479991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215167,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40444,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704708,"statfs":{"total":95995035648,"available":95953620992,"internally_reserved":0,"allocated":962560,"data_stored":475625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":520899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":999424,"data_stored":520679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":974848,"data_stored":477373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":45484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":45704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":56128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-28T19:19:26.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-28T19:19:27.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:26 smithi028 ceph-mon[21769]: pgmap v289: 129 pgs: 129 active+clean; 637 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 KiB/s wr, 21 op/s 2024-08-28T19:19:27.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:26 smithi028 ceph-mon[21769]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:19:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:27.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:26 smithi115 ceph-mon[22184]: pgmap v289: 129 pgs: 129 active+clean; 637 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 KiB/s wr, 21 op/s 2024-08-28T19:19:27.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:26 smithi115 ceph-mon[22184]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:19:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.596+0000 7fdddf4c8640 1 -- 172.21.15.16:0/2680589897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd8108a90 msgr2=0x7fddd810f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.596+0000 7fdddf4c8640 1 --2- 172.21.15.16:0/2680589897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd8108a90 0x7fddd810f2c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009ca0 tx=0x7fddd002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.597+0000 7fdddf4c8640 1 -- 172.21.15.16:0/2680589897 shutdown_connections 2024-08-28T19:19:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.597+0000 7fdddf4c8640 1 --2- 172.21.15.16:0/2680589897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd8108a90 0x7fddd810f2c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.597+0000 7fdddf4c8640 1 --2- 172.21.15.16:0/2680589897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd8108140 0x7fddd8108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.597+0000 7fdddf4c8640 1 --2- 172.21.15.16:0/2680589897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd8102800 0x7fddd8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.597+0000 7fdddf4c8640 1 -- 172.21.15.16:0/2680589897 >> 172.21.15.16:0/2680589897 conn(0x7fddd80794d0 msgr2=0x7fddd80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.597+0000 7fdddf4c8640 1 -- 172.21.15.16:0/2680589897 shutdown_connections 2024-08-28T19:19:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.597+0000 7fdddf4c8640 1 -- 172.21.15.16:0/2680589897 wait complete. 2024-08-28T19:19:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.597+0000 7fdddf4c8640 1 Processor -- start 2024-08-28T19:19:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddf4c8640 1 -- start start 2024-08-28T19:19:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddf4c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd8102800 0x7fddd8199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddf4c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd8108140 0x7fddd8199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddf4c8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd8108a90 0x7fddd819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddf4c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fddd8112710 con 0x7fddd8102800 2024-08-28T19:19:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddf4c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fddd8112590 con 0x7fddd8108140 2024-08-28T19:19:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddf4c8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fddd8112890 con 0x7fddd8108a90 2024-08-28T19:19:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddd264640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd8102800 0x7fddd8199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddd264640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd8102800 0x7fddd8199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36158/0 (socket says 172.21.15.16:36158) 2024-08-28T19:19:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddd264640 1 -- 172.21.15.16:0/990754685 learned_addr learned my addr 172.21.15.16:0/990754685 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddda65640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd8108a90 0x7fddd819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddca63640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd8108140 0x7fddd8199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddd264640 1 -- 172.21.15.16:0/990754685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd8108a90 msgr2=0x7fddd819e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddd264640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd8108a90 0x7fddd819e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddd264640 1 -- 172.21.15.16:0/990754685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd8108140 msgr2=0x7fddd8199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddd264640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd8108140 0x7fddd8199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.598+0000 7fdddd264640 1 -- 172.21.15.16:0/990754685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd819f0d0 con 0x7fddd8102800 2024-08-28T19:19:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.599+0000 7fdddd264640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd8102800 0x7fddd8199160 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00d950 tx=0x7fddcc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.599+0000 7fddca7fc640 1 -- 172.21.15.16:0/990754685 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc017070 con 0x7fddd8102800 2024-08-28T19:19:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.599+0000 7fddca7fc640 1 -- 172.21.15.16:0/990754685 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddcc009c30 con 0x7fddd8102800 2024-08-28T19:19:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.599+0000 7fdddf4c8640 1 -- 172.21.15.16:0/990754685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd819f3c0 con 0x7fddd8102800 2024-08-28T19:19:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.599+0000 7fddca7fc640 1 -- 172.21.15.16:0/990754685 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc00f620 con 0x7fddd8102800 2024-08-28T19:19:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.599+0000 7fdddf4c8640 1 -- 172.21.15.16:0/990754685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd806ac00 con 0x7fddd8102800 2024-08-28T19:19:27.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.600+0000 7fdddf4c8640 1 -- 172.21.15.16:0/990754685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd8102c10 con 0x7fddd8102800 2024-08-28T19:19:27.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.602+0000 7fddca7fc640 1 -- 172.21.15.16:0/990754685 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fddcc010460 con 0x7fddd8102800 2024-08-28T19:19:27.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.603+0000 7fddca7fc640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc40b4c10 0x7fddc40b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:27.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.603+0000 7fdddca63640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc40b4c10 0x7fddc40b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:27.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.603+0000 7fddca7fc640 1 -- 172.21.15.16:0/990754685 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fddcc0e5380 con 0x7fddd8102800 2024-08-28T19:19:27.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.604+0000 7fdddca63640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc40b4c10 0x7fddc40b7120 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fddd4005d70 tx=0x7fddd4005d00 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.606+0000 7fddca7fc640 1 -- 172.21.15.16:0/990754685 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fddcc0abf30 con 0x7fddd8102800 2024-08-28T19:19:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.788+0000 7fdddf4c8640 1 -- 172.21.15.16:0/990754685 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fddd806af20 con 0x7fddc40b4c10 2024-08-28T19:19:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.794+0000 7fddca7fc640 1 -- 172.21.15.16:0/990754685 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340391 (secure 0 0 0) 0x7fddd806af20 con 0x7fddc40b4c10 2024-08-28T19:19:27.798 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:19:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-08-28T19:19:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 -- 172.21.15.16:0/990754685 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc40b4c10 msgr2=0x7fddc40b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc40b4c10 0x7fddc40b7120 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fddd4005d70 tx=0x7fddd4005d00 comp rx=0 tx=0).stop 2024-08-28T19:19:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 -- 172.21.15.16:0/990754685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd8102800 msgr2=0x7fddd8199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd8102800 0x7fddd8199160 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00d950 tx=0x7fddcc00de10 comp rx=0 tx=0).stop 2024-08-28T19:19:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 -- 172.21.15.16:0/990754685 shutdown_connections 2024-08-28T19:19:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd8108a90 0x7fddd819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc40b4c10 0x7fddc40b7120 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd8108140 0x7fddd8199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 --2- 172.21.15.16:0/990754685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd8102800 0x7fddd8199160 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 -- 172.21.15.16:0/990754685 >> 172.21.15.16:0/990754685 conn(0x7fddd80794d0 msgr2=0x7fddd8105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 -- 172.21.15.16:0/990754685 shutdown_connections 2024-08-28T19:19:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:27.798+0000 7fdddf4c8640 1 -- 172.21.15.16:0/990754685 wait complete. 2024-08-28T19:19:28.500 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-08-28T19:19:27.467914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652198,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":249,"num_read_kb":223,"num_write":369,"num_write_kb":2283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":412424,"kb_used_data":13256,"kb_used_omap":18,"kb_used_meta":398957,"kb_avail":1124529400,"statfs":{"total":1151940427776,"available":1151518105600,"internally_reserved":0,"allocated":13574144,"data_stored":7691000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":408532282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44665,"num_objects":5,"num_object_clones":0,"num_object_copies":17,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":120,"num_read_kb":106,"num_write":37,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003839"},"pg_stats":[{"pgid":"2.1f","version":"82'13","reported_seq":76,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:24.675959+0000","last_change":"2024-08-28T19:19:08.633326+0000","last_active":"2024-08-28T19:19:24.675959+0000","last_peered":"2024-08-28T19:19:24.675959+0000","last_clean":"2024-08-28T19:19:24.675959+0000","last_became_active":"2024-08-28T19:19:08.623876+0000","last_became_peered":"2024-08-28T19:19:08.623876+0000","last_unstale":"2024-08-28T19:19:24.675959+0000","last_undegraded":"2024-08-28T19:19:24.675959+0000","last_fullsized":"2024-08-28T19:19:24.675959+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:19:50.632337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632322+0000","last_change":"2024-08-28T19:18:32.492604+0000","last_active":"2024-08-28T19:19:08.632322+0000","last_peered":"2024-08-28T19:19:08.632322+0000","last_clean":"2024-08-28T19:19:08.632322+0000","last_became_active":"2024-08-28T19:18:32.489427+0000","last_became_peered":"2024-08-28T19:18:32.489427+0000","last_unstale":"2024-08-28T19:19:08.632322+0000","last_undegraded":"2024-08-28T19:19:08.632322+0000","last_fullsized":"2024-08-28T19:19:08.632322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:36:14.841315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622224+0000","last_change":"2024-08-28T19:18:39.528815+0000","last_active":"2024-08-28T19:19:08.622224+0000","last_peered":"2024-08-28T19:19:08.622224+0000","last_clean":"2024-08-28T19:19:08.622224+0000","last_became_active":"2024-08-28T19:18:39.527781+0000","last_became_peered":"2024-08-28T19:18:39.527781+0000","last_unstale":"2024-08-28T19:19:08.622224+0000","last_undegraded":"2024-08-28T19:19:08.622224+0000","last_fullsized":"2024-08-28T19:19:08.622224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:39.831426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632931+0000","last_change":"2024-08-28T19:18:31.082641+0000","last_active":"2024-08-28T19:19:08.632931+0000","last_peered":"2024-08-28T19:19:08.632931+0000","last_clean":"2024-08-28T19:19:08.632931+0000","last_became_active":"2024-08-28T19:18:31.082260+0000","last_became_peered":"2024-08-28T19:18:31.082260+0000","last_unstale":"2024-08-28T19:19:08.632931+0000","last_undegraded":"2024-08-28T19:19:08.632931+0000","last_fullsized":"2024-08-28T19:19:08.632931+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:09:55.836463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629290+0000","last_change":"2024-08-28T19:18:32.493498+0000","last_active":"2024-08-28T19:19:08.629290+0000","last_peered":"2024-08-28T19:19:08.629290+0000","last_clean":"2024-08-28T19:19:08.629290+0000","last_became_active":"2024-08-28T19:18:32.490969+0000","last_became_peered":"2024-08-28T19:18:32.490969+0000","last_unstale":"2024-08-28T19:19:08.629290+0000","last_undegraded":"2024-08-28T19:19:08.629290+0000","last_fullsized":"2024-08-28T19:19:08.629290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:34:29.537622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617571+0000","last_change":"2024-08-28T19:18:39.520922+0000","last_active":"2024-08-28T19:19:08.617571+0000","last_peered":"2024-08-28T19:19:08.617571+0000","last_clean":"2024-08-28T19:19:08.617571+0000","last_became_active":"2024-08-28T19:18:39.520362+0000","last_became_peered":"2024-08-28T19:18:39.520362+0000","last_unstale":"2024-08-28T19:19:08.617571+0000","last_undegraded":"2024-08-28T19:19:08.617571+0000","last_fullsized":"2024-08-28T19:19:08.617571+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:40.658820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.646751+0000","last_change":"2024-08-28T19:18:30.471870+0000","last_active":"2024-08-28T19:19:12.646751+0000","last_peered":"2024-08-28T19:19:12.646751+0000","last_clean":"2024-08-28T19:19:12.646751+0000","last_became_active":"2024-08-28T19:18:30.471438+0000","last_became_peered":"2024-08-28T19:18:30.471438+0000","last_unstale":"2024-08-28T19:19:12.646751+0000","last_undegraded":"2024-08-28T19:19:12.646751+0000","last_fullsized":"2024-08-28T19:19:12.646751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:44:34.137395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628034+0000","last_change":"2024-08-28T19:18:32.486485+0000","last_active":"2024-08-28T19:19:08.628034+0000","last_peered":"2024-08-28T19:19:08.628034+0000","last_clean":"2024-08-28T19:19:08.628034+0000","last_became_active":"2024-08-28T19:18:32.485888+0000","last_became_peered":"2024-08-28T19:18:32.485888+0000","last_unstale":"2024-08-28T19:19:08.628034+0000","last_undegraded":"2024-08-28T19:19:08.628034+0000","last_fullsized":"2024-08-28T19:19:08.628034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:45:48.757318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.633437+0000","last_change":"2024-08-28T19:19:08.633317+0000","last_active":"2024-08-28T19:19:08.633437+0000","last_peered":"2024-08-28T19:19:08.633437+0000","last_clean":"2024-08-28T19:19:08.633437+0000","last_became_active":"2024-08-28T19:19:08.632689+0000","last_became_peered":"2024-08-28T19:19:08.632689+0000","last_unstale":"2024-08-28T19:19:08.633437+0000","last_undegraded":"2024-08-28T19:19:08.633437+0000","last_fullsized":"2024-08-28T19:19:08.633437+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:35:08.740645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653995+0000","last_change":"2024-08-28T19:18:30.473982+0000","last_active":"2024-08-28T19:19:11.653995+0000","last_peered":"2024-08-28T19:19:11.653995+0000","last_clean":"2024-08-28T19:19:11.653995+0000","last_became_active":"2024-08-28T19:18:30.473552+0000","last_became_peered":"2024-08-28T19:18:30.473552+0000","last_unstale":"2024-08-28T19:19:11.653995+0000","last_undegraded":"2024-08-28T19:19:11.653995+0000","last_fullsized":"2024-08-28T19:19:11.653995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:22:29.435517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616754+0000","last_change":"2024-08-28T19:18:32.487881+0000","last_active":"2024-08-28T19:19:08.616754+0000","last_peered":"2024-08-28T19:19:08.616754+0000","last_clean":"2024-08-28T19:19:08.616754+0000","last_became_active":"2024-08-28T19:18:32.487039+0000","last_became_peered":"2024-08-28T19:18:32.487039+0000","last_unstale":"2024-08-28T19:19:08.616754+0000","last_undegraded":"2024-08-28T19:19:08.616754+0000","last_fullsized":"2024-08-28T19:19:08.616754+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:23:35.757811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617767+0000","last_change":"2024-08-28T19:18:39.518654+0000","last_active":"2024-08-28T19:19:08.617767+0000","last_peered":"2024-08-28T19:19:08.617767+0000","last_clean":"2024-08-28T19:19:08.617767+0000","last_became_active":"2024-08-28T19:18:39.518087+0000","last_became_peered":"2024-08-28T19:18:39.518087+0000","last_unstale":"2024-08-28T19:19:08.617767+0000","last_undegraded":"2024-08-28T19:19:08.617767+0000","last_fullsized":"2024-08-28T19:19:08.617767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:08.284865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617577+0000","last_change":"2024-08-28T19:18:32.491928+0000","last_active":"2024-08-28T19:19:08.617577+0000","last_peered":"2024-08-28T19:19:08.617577+0000","last_clean":"2024-08-28T19:19:08.617577+0000","last_became_active":"2024-08-28T19:18:32.491538+0000","last_became_peered":"2024-08-28T19:18:32.491538+0000","last_unstale":"2024-08-28T19:19:08.617577+0000","last_undegraded":"2024-08-28T19:19:08.617577+0000","last_fullsized":"2024-08-28T19:19:08.617577+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:39:06.077830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.955289+0000","last_change":"2024-08-28T19:19:08.634752+0000","last_active":"2024-08-28T19:19:11.955289+0000","last_peered":"2024-08-28T19:19:11.955289+0000","last_clean":"2024-08-28T19:19:11.955289+0000","last_became_active":"2024-08-28T19:19:08.620784+0000","last_became_peered":"2024-08-28T19:19:08.620784+0000","last_unstale":"2024-08-28T19:19:11.955289+0000","last_undegraded":"2024-08-28T19:19:11.955289+0000","last_fullsized":"2024-08-28T19:19:11.955289+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:51.640445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618419+0000","last_change":"2024-08-28T19:18:39.681413+0000","last_active":"2024-08-28T19:19:08.618419+0000","last_peered":"2024-08-28T19:19:08.618419+0000","last_clean":"2024-08-28T19:19:08.618419+0000","last_became_active":"2024-08-28T19:18:39.680793+0000","last_became_peered":"2024-08-28T19:18:39.680793+0000","last_unstale":"2024-08-28T19:19:08.618419+0000","last_undegraded":"2024-08-28T19:19:08.618419+0000","last_fullsized":"2024-08-28T19:19:08.618419+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:45:43.381019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632212+0000","last_change":"2024-08-28T19:18:32.486739+0000","last_active":"2024-08-28T19:19:08.632212+0000","last_peered":"2024-08-28T19:19:08.632212+0000","last_clean":"2024-08-28T19:19:08.632212+0000","last_became_active":"2024-08-28T19:18:32.486347+0000","last_became_peered":"2024-08-28T19:18:32.486347+0000","last_unstale":"2024-08-28T19:19:08.632212+0000","last_undegraded":"2024-08-28T19:19:08.632212+0000","last_fullsized":"2024-08-28T19:19:08.632212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:27:36.448569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.641964+0000","last_change":"2024-08-28T19:18:30.467690+0000","last_active":"2024-08-28T19:19:10.641964+0000","last_peered":"2024-08-28T19:19:10.641964+0000","last_clean":"2024-08-28T19:19:10.641964+0000","last_became_active":"2024-08-28T19:18:30.467249+0000","last_became_peered":"2024-08-28T19:18:30.467249+0000","last_unstale":"2024-08-28T19:19:10.641964+0000","last_undegraded":"2024-08-28T19:19:10.641964+0000","last_fullsized":"2024-08-28T19:19:10.641964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:37:28.876666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618227+0000","last_change":"2024-08-28T19:18:39.523889+0000","last_active":"2024-08-28T19:19:08.618227+0000","last_peered":"2024-08-28T19:19:08.618227+0000","last_clean":"2024-08-28T19:19:08.618227+0000","last_became_active":"2024-08-28T19:18:39.523384+0000","last_became_peered":"2024-08-28T19:18:39.523384+0000","last_unstale":"2024-08-28T19:19:08.618227+0000","last_undegraded":"2024-08-28T19:19:08.618227+0000","last_fullsized":"2024-08-28T19:19:08.618227+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:57:32.277680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614254+0000","last_change":"2024-08-28T19:18:32.482257+0000","last_active":"2024-08-28T19:19:08.614254+0000","last_peered":"2024-08-28T19:19:08.614254+0000","last_clean":"2024-08-28T19:19:08.614254+0000","last_became_active":"2024-08-28T19:18:32.481884+0000","last_became_peered":"2024-08-28T19:18:32.481884+0000","last_unstale":"2024-08-28T19:19:08.614254+0000","last_undegraded":"2024-08-28T19:19:08.614254+0000","last_fullsized":"2024-08-28T19:19:08.614254+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:56:28.754369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651638+0000","last_change":"2024-08-28T19:18:30.467307+0000","last_active":"2024-08-28T19:19:11.651638+0000","last_peered":"2024-08-28T19:19:11.651638+0000","last_clean":"2024-08-28T19:19:11.651638+0000","last_became_active":"2024-08-28T19:18:30.466942+0000","last_became_peered":"2024-08-28T19:18:30.466942+0000","last_unstale":"2024-08-28T19:19:11.651638+0000","last_undegraded":"2024-08-28T19:19:11.651638+0000","last_fullsized":"2024-08-28T19:19:11.651638+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:05:23.682110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614513+0000","last_change":"2024-08-28T19:18:39.531421+0000","last_active":"2024-08-28T19:19:08.614513+0000","last_peered":"2024-08-28T19:19:08.614513+0000","last_clean":"2024-08-28T19:19:08.614513+0000","last_became_active":"2024-08-28T19:18:39.531008+0000","last_became_peered":"2024-08-28T19:18:39.531008+0000","last_unstale":"2024-08-28T19:19:08.614513+0000","last_undegraded":"2024-08-28T19:19:08.614513+0000","last_fullsized":"2024-08-28T19:19:08.614513+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:46:49.580601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618946+0000","last_change":"2024-08-28T19:18:32.485204+0000","last_active":"2024-08-28T19:19:08.618946+0000","last_peered":"2024-08-28T19:19:08.618946+0000","last_clean":"2024-08-28T19:19:08.618946+0000","last_became_active":"2024-08-28T19:18:32.484830+0000","last_became_peered":"2024-08-28T19:18:32.484830+0000","last_unstale":"2024-08-28T19:19:08.618946+0000","last_undegraded":"2024-08-28T19:19:08.618946+0000","last_fullsized":"2024-08-28T19:19:08.618946+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:55:41.358718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632011+0000","last_change":"2024-08-28T19:18:31.082361+0000","last_active":"2024-08-28T19:19:08.632011+0000","last_peered":"2024-08-28T19:19:08.632011+0000","last_clean":"2024-08-28T19:19:08.632011+0000","last_became_active":"2024-08-28T19:18:31.081930+0000","last_became_peered":"2024-08-28T19:18:31.081930+0000","last_unstale":"2024-08-28T19:19:08.632011+0000","last_undegraded":"2024-08-28T19:19:08.632011+0000","last_fullsized":"2024-08-28T19:19:08.632011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:41:05.543088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618390+0000","last_change":"2024-08-28T19:18:39.682341+0000","last_active":"2024-08-28T19:19:08.618390+0000","last_peered":"2024-08-28T19:19:08.618390+0000","last_clean":"2024-08-28T19:19:08.618390+0000","last_became_active":"2024-08-28T19:18:39.681751+0000","last_became_peered":"2024-08-28T19:18:39.681751+0000","last_unstale":"2024-08-28T19:19:08.618390+0000","last_undegraded":"2024-08-28T19:19:08.618390+0000","last_fullsized":"2024-08-28T19:19:08.618390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:51:24.371270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628950+0000","last_change":"2024-08-28T19:18:32.491069+0000","last_active":"2024-08-28T19:19:08.628950+0000","last_peered":"2024-08-28T19:19:08.628950+0000","last_clean":"2024-08-28T19:19:08.628950+0000","last_became_active":"2024-08-28T19:18:32.490688+0000","last_became_peered":"2024-08-28T19:18:32.490688+0000","last_unstale":"2024-08-28T19:19:08.628950+0000","last_undegraded":"2024-08-28T19:19:08.628950+0000","last_fullsized":"2024-08-28T19:19:08.628950+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:24:31.560519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652924+0000","last_change":"2024-08-28T19:18:30.739812+0000","last_active":"2024-08-28T19:19:11.652924+0000","last_peered":"2024-08-28T19:19:11.652924+0000","last_clean":"2024-08-28T19:19:11.652924+0000","last_became_active":"2024-08-28T19:18:30.739414+0000","last_became_peered":"2024-08-28T19:18:30.739414+0000","last_unstale":"2024-08-28T19:19:11.652924+0000","last_undegraded":"2024-08-28T19:19:11.652924+0000","last_fullsized":"2024-08-28T19:19:11.652924+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:56:56.594993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622469+0000","last_change":"2024-08-28T19:18:39.528281+0000","last_active":"2024-08-28T19:19:08.622469+0000","last_peered":"2024-08-28T19:19:08.622469+0000","last_clean":"2024-08-28T19:19:08.622469+0000","last_became_active":"2024-08-28T19:18:39.527142+0000","last_became_peered":"2024-08-28T19:18:39.527142+0000","last_unstale":"2024-08-28T19:19:08.622469+0000","last_undegraded":"2024-08-28T19:19:08.622469+0000","last_fullsized":"2024-08-28T19:19:08.622469+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:44:19.123662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632709+0000","last_change":"2024-08-28T19:18:32.490430+0000","last_active":"2024-08-28T19:19:08.632709+0000","last_peered":"2024-08-28T19:19:08.632709+0000","last_clean":"2024-08-28T19:19:08.632709+0000","last_became_active":"2024-08-28T19:18:32.490020+0000","last_became_peered":"2024-08-28T19:18:32.490020+0000","last_unstale":"2024-08-28T19:19:08.632709+0000","last_undegraded":"2024-08-28T19:19:08.632709+0000","last_fullsized":"2024-08-28T19:19:08.632709+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:52:31.153012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"82'14","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:23.487955+0000","last_change":"2024-08-28T19:18:30.467060+0000","last_active":"2024-08-28T19:19:23.487955+0000","last_peered":"2024-08-28T19:19:23.487955+0000","last_clean":"2024-08-28T19:19:23.487955+0000","last_became_active":"2024-08-28T19:18:30.466689+0000","last_became_peered":"2024-08-28T19:18:30.466689+0000","last_unstale":"2024-08-28T19:19:23.487955+0000","last_undegraded":"2024-08-28T19:19:23.487955+0000","last_fullsized":"2024-08-28T19:19:23.487955+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:18:36.508131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.630939+0000","last_change":"2024-08-28T19:18:39.682874+0000","last_active":"2024-08-28T19:19:08.630939+0000","last_peered":"2024-08-28T19:19:08.630939+0000","last_clean":"2024-08-28T19:19:08.630939+0000","last_became_active":"2024-08-28T19:18:39.681500+0000","last_became_peered":"2024-08-28T19:18:39.681500+0000","last_unstale":"2024-08-28T19:19:08.630939+0000","last_undegraded":"2024-08-28T19:19:08.630939+0000","last_fullsized":"2024-08-28T19:19:08.630939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:45:41.815506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612195+0000","last_change":"2024-08-28T19:18:32.482047+0000","last_active":"2024-08-28T19:19:08.612195+0000","last_peered":"2024-08-28T19:19:08.612195+0000","last_clean":"2024-08-28T19:19:08.612195+0000","last_became_active":"2024-08-28T19:18:32.481640+0000","last_became_peered":"2024-08-28T19:18:32.481640+0000","last_unstale":"2024-08-28T19:19:08.612195+0000","last_undegraded":"2024-08-28T19:19:08.612195+0000","last_fullsized":"2024-08-28T19:19:08.612195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:34.717764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651944+0000","last_change":"2024-08-28T19:18:31.083524+0000","last_active":"2024-08-28T19:19:11.651944+0000","last_peered":"2024-08-28T19:19:11.651944+0000","last_clean":"2024-08-28T19:19:11.651944+0000","last_became_active":"2024-08-28T19:18:31.083106+0000","last_became_peered":"2024-08-28T19:18:31.083106+0000","last_unstale":"2024-08-28T19:19:11.651944+0000","last_undegraded":"2024-08-28T19:19:11.651944+0000","last_fullsized":"2024-08-28T19:19:11.651944+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:16.180754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615954+0000","last_change":"2024-08-28T19:18:39.531884+0000","last_active":"2024-08-28T19:19:08.615954+0000","last_peered":"2024-08-28T19:19:08.615954+0000","last_clean":"2024-08-28T19:19:08.615954+0000","last_became_active":"2024-08-28T19:18:39.531255+0000","last_became_peered":"2024-08-28T19:18:39.531255+0000","last_unstale":"2024-08-28T19:19:08.615954+0000","last_undegraded":"2024-08-28T19:19:08.615954+0000","last_fullsized":"2024-08-28T19:19:08.615954+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:36:55.496594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612691+0000","last_change":"2024-08-28T19:18:32.485180+0000","last_active":"2024-08-28T19:19:08.612691+0000","last_peered":"2024-08-28T19:19:08.612691+0000","last_clean":"2024-08-28T19:19:08.612691+0000","last_became_active":"2024-08-28T19:18:32.484804+0000","last_became_peered":"2024-08-28T19:18:32.484804+0000","last_unstale":"2024-08-28T19:19:08.612691+0000","last_undegraded":"2024-08-28T19:19:08.612691+0000","last_fullsized":"2024-08-28T19:19:08.612691+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:12:40.958251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615096+0000","last_change":"2024-08-28T19:18:31.082226+0000","last_active":"2024-08-28T19:19:08.615096+0000","last_peered":"2024-08-28T19:19:08.615096+0000","last_clean":"2024-08-28T19:19:08.615096+0000","last_became_active":"2024-08-28T19:18:31.081833+0000","last_became_peered":"2024-08-28T19:18:31.081833+0000","last_unstale":"2024-08-28T19:19:08.615096+0000","last_undegraded":"2024-08-28T19:19:08.615096+0000","last_fullsized":"2024-08-28T19:19:08.615096+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:33:56.298093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619013+0000","last_change":"2024-08-28T19:18:39.681288+0000","last_active":"2024-08-28T19:19:08.619013+0000","last_peered":"2024-08-28T19:19:08.619013+0000","last_clean":"2024-08-28T19:19:08.619013+0000","last_became_active":"2024-08-28T19:18:39.680719+0000","last_became_peered":"2024-08-28T19:18:39.680719+0000","last_unstale":"2024-08-28T19:19:08.619013+0000","last_undegraded":"2024-08-28T19:19:08.619013+0000","last_fullsized":"2024-08-28T19:19:08.619013+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:53:35.554181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612370+0000","last_change":"2024-08-28T19:18:32.487105+0000","last_active":"2024-08-28T19:19:08.612370+0000","last_peered":"2024-08-28T19:19:08.612370+0000","last_clean":"2024-08-28T19:19:08.612370+0000","last_became_active":"2024-08-28T19:18:32.486643+0000","last_became_peered":"2024-08-28T19:18:32.486643+0000","last_unstale":"2024-08-28T19:19:08.612370+0000","last_undegraded":"2024-08-28T19:19:08.612370+0000","last_fullsized":"2024-08-28T19:19:08.612370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:18:43.274493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'6","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:23.675365+0000","last_change":"2024-08-28T19:18:30.471447+0000","last_active":"2024-08-28T19:19:23.675365+0000","last_peered":"2024-08-28T19:19:23.675365+0000","last_clean":"2024-08-28T19:19:23.675365+0000","last_became_active":"2024-08-28T19:18:30.470798+0000","last_became_peered":"2024-08-28T19:18:30.470798+0000","last_unstale":"2024-08-28T19:19:23.675365+0000","last_undegraded":"2024-08-28T19:19:23.675365+0000","last_fullsized":"2024-08-28T19:19:23.675365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:50:13.948439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611755+0000","last_change":"2024-08-28T19:18:39.524190+0000","last_active":"2024-08-28T19:19:08.611755+0000","last_peered":"2024-08-28T19:19:08.611755+0000","last_clean":"2024-08-28T19:19:08.611755+0000","last_became_active":"2024-08-28T19:18:39.523771+0000","last_became_peered":"2024-08-28T19:18:39.523771+0000","last_unstale":"2024-08-28T19:19:08.611755+0000","last_undegraded":"2024-08-28T19:19:08.611755+0000","last_fullsized":"2024-08-28T19:19:08.611755+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:43:43.984625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615282+0000","last_change":"2024-08-28T19:18:32.487212+0000","last_active":"2024-08-28T19:19:08.615282+0000","last_peered":"2024-08-28T19:19:08.615282+0000","last_clean":"2024-08-28T19:19:08.615282+0000","last_became_active":"2024-08-28T19:18:32.485432+0000","last_became_peered":"2024-08-28T19:18:32.485432+0000","last_unstale":"2024-08-28T19:19:08.615282+0000","last_undegraded":"2024-08-28T19:19:08.615282+0000","last_fullsized":"2024-08-28T19:19:08.615282+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:52:11.672193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:23.675243+0000","last_change":"2024-08-28T19:18:31.082210+0000","last_active":"2024-08-28T19:19:23.675243+0000","last_peered":"2024-08-28T19:19:23.675243+0000","last_clean":"2024-08-28T19:19:23.675243+0000","last_became_active":"2024-08-28T19:18:31.081814+0000","last_became_peered":"2024-08-28T19:18:31.081814+0000","last_unstale":"2024-08-28T19:19:23.675243+0000","last_undegraded":"2024-08-28T19:19:23.675243+0000","last_fullsized":"2024-08-28T19:19:23.675243+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:33:32.281431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611523+0000","last_change":"2024-08-28T19:18:39.681858+0000","last_active":"2024-08-28T19:19:08.611523+0000","last_peered":"2024-08-28T19:19:08.611523+0000","last_clean":"2024-08-28T19:19:08.611523+0000","last_became_active":"2024-08-28T19:18:39.681382+0000","last_became_peered":"2024-08-28T19:18:39.681382+0000","last_unstale":"2024-08-28T19:19:08.611523+0000","last_undegraded":"2024-08-28T19:19:08.611523+0000","last_fullsized":"2024-08-28T19:19:08.611523+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:23:22.021022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611190+0000","last_change":"2024-08-28T19:18:32.486802+0000","last_active":"2024-08-28T19:19:08.611190+0000","last_peered":"2024-08-28T19:19:08.611190+0000","last_clean":"2024-08-28T19:19:08.611190+0000","last_became_active":"2024-08-28T19:18:32.486479+0000","last_became_peered":"2024-08-28T19:18:32.486479+0000","last_unstale":"2024-08-28T19:19:08.611190+0000","last_undegraded":"2024-08-28T19:19:08.611190+0000","last_fullsized":"2024-08-28T19:19:08.611190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:17:17.654757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:09.635071+0000","last_change":"2024-08-28T19:18:30.470447+0000","last_active":"2024-08-28T19:19:09.635071+0000","last_peered":"2024-08-28T19:19:09.635071+0000","last_clean":"2024-08-28T19:19:09.635071+0000","last_became_active":"2024-08-28T19:18:30.469862+0000","last_became_peered":"2024-08-28T19:18:30.469862+0000","last_unstale":"2024-08-28T19:19:09.635071+0000","last_undegraded":"2024-08-28T19:19:09.635071+0000","last_fullsized":"2024-08-28T19:19:09.635071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:27:46.660492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624752+0000","last_change":"2024-08-28T19:19:08.624635+0000","last_active":"2024-08-28T19:19:08.624752+0000","last_peered":"2024-08-28T19:19:08.624752+0000","last_clean":"2024-08-28T19:19:08.624752+0000","last_became_active":"2024-08-28T19:19:08.622845+0000","last_became_peered":"2024-08-28T19:19:08.622845+0000","last_unstale":"2024-08-28T19:19:08.624752+0000","last_undegraded":"2024-08-28T19:19:08.624752+0000","last_fullsized":"2024-08-28T19:19:08.624752+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:56:40.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631918+0000","last_change":"2024-08-28T19:18:32.487947+0000","last_active":"2024-08-28T19:19:08.631918+0000","last_peered":"2024-08-28T19:19:08.631918+0000","last_clean":"2024-08-28T19:19:08.631918+0000","last_became_active":"2024-08-28T19:18:32.487592+0000","last_became_peered":"2024-08-28T19:18:32.487592+0000","last_unstale":"2024-08-28T19:19:08.631918+0000","last_undegraded":"2024-08-28T19:19:08.631918+0000","last_fullsized":"2024-08-28T19:19:08.631918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:30:29.550644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.654410+0000","last_change":"2024-08-28T19:18:30.465044+0000","last_active":"2024-08-28T19:19:12.654410+0000","last_peered":"2024-08-28T19:19:12.654410+0000","last_clean":"2024-08-28T19:19:12.654410+0000","last_became_active":"2024-08-28T19:18:30.464580+0000","last_became_peered":"2024-08-28T19:18:30.464580+0000","last_unstale":"2024-08-28T19:19:12.654410+0000","last_undegraded":"2024-08-28T19:19:12.654410+0000","last_fullsized":"2024-08-28T19:19:12.654410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:21:17.223311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":109,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617187+0000","last_change":"2024-08-28T19:17:17.063316+0000","last_active":"2024-08-28T19:19:08.617187+0000","last_peered":"2024-08-28T19:19:08.617187+0000","last_clean":"2024-08-28T19:19:08.617187+0000","last_became_active":"2024-08-28T19:17:17.057091+0000","last_became_peered":"2024-08-28T19:17:17.057091+0000","last_unstale":"2024-08-28T19:19:08.617187+0000","last_undegraded":"2024-08-28T19:19:08.617187+0000","last_fullsized":"2024-08-28T19:19:08.617187+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_clean_scrub_stamp":"2024-08-28T19:13:56.815467+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:35:51.499703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"82'1","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622311+0000","last_change":"2024-08-28T19:19:08.621402+0000","last_active":"2024-08-28T19:19:08.622311+0000","last_peered":"2024-08-28T19:19:08.622311+0000","last_clean":"2024-08-28T19:19:08.622311+0000","last_became_active":"2024-08-28T19:19:08.620779+0000","last_became_peered":"2024-08-28T19:19:08.620779+0000","last_unstale":"2024-08-28T19:19:08.622311+0000","last_undegraded":"2024-08-28T19:19:08.622311+0000","last_fullsized":"2024-08-28T19:19:08.622311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:39.391414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619759+0000","last_change":"2024-08-28T19:18:32.490066+0000","last_active":"2024-08-28T19:19:08.619759+0000","last_peered":"2024-08-28T19:19:08.619759+0000","last_clean":"2024-08-28T19:19:08.619759+0000","last_became_active":"2024-08-28T19:18:32.489560+0000","last_became_peered":"2024-08-28T19:18:32.489560+0000","last_unstale":"2024-08-28T19:19:08.619759+0000","last_undegraded":"2024-08-28T19:19:08.619759+0000","last_fullsized":"2024-08-28T19:19:08.619759+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:22:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'5","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:24.677109+0000","last_change":"2024-08-28T19:18:31.083078+0000","last_active":"2024-08-28T19:19:24.677109+0000","last_peered":"2024-08-28T19:19:24.677109+0000","last_clean":"2024-08-28T19:19:24.677109+0000","last_became_active":"2024-08-28T19:18:31.082575+0000","last_became_peered":"2024-08-28T19:18:31.082575+0000","last_unstale":"2024-08-28T19:19:24.677109+0000","last_undegraded":"2024-08-28T19:19:24.677109+0000","last_fullsized":"2024-08-28T19:19:24.677109+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627532+0000","last_change":"2024-08-28T19:18:39.519814+0000","last_active":"2024-08-28T19:19:08.627532+0000","last_peered":"2024-08-28T19:19:08.627532+0000","last_clean":"2024-08-28T19:19:08.627532+0000","last_became_active":"2024-08-28T19:18:39.519258+0000","last_became_peered":"2024-08-28T19:18:39.519258+0000","last_unstale":"2024-08-28T19:19:08.627532+0000","last_undegraded":"2024-08-28T19:19:08.627532+0000","last_fullsized":"2024-08-28T19:19:08.627532+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:03:04.051376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618184+0000","last_change":"2024-08-28T19:18:32.495426+0000","last_active":"2024-08-28T19:19:08.618184+0000","last_peered":"2024-08-28T19:19:08.618184+0000","last_clean":"2024-08-28T19:19:08.618184+0000","last_became_active":"2024-08-28T19:18:32.495075+0000","last_became_peered":"2024-08-28T19:18:32.495075+0000","last_unstale":"2024-08-28T19:19:08.618184+0000","last_undegraded":"2024-08-28T19:19:08.618184+0000","last_fullsized":"2024-08-28T19:19:08.618184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:40:22.456666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652460+0000","last_change":"2024-08-28T19:18:30.472708+0000","last_active":"2024-08-28T19:19:11.652460+0000","last_peered":"2024-08-28T19:19:11.652460+0000","last_clean":"2024-08-28T19:19:11.652460+0000","last_became_active":"2024-08-28T19:18:30.472320+0000","last_became_peered":"2024-08-28T19:18:30.472320+0000","last_unstale":"2024-08-28T19:19:11.652460+0000","last_undegraded":"2024-08-28T19:19:11.652460+0000","last_fullsized":"2024-08-28T19:19:11.652460+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:57:12.055626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628468+0000","last_change":"2024-08-28T19:18:39.519398+0000","last_active":"2024-08-28T19:19:08.628468+0000","last_peered":"2024-08-28T19:19:08.628468+0000","last_clean":"2024-08-28T19:19:08.628468+0000","last_became_active":"2024-08-28T19:18:39.518997+0000","last_became_peered":"2024-08-28T19:18:39.518997+0000","last_unstale":"2024-08-28T19:19:08.628468+0000","last_undegraded":"2024-08-28T19:19:08.628468+0000","last_fullsized":"2024-08-28T19:19:08.628468+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:33.431608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619235+0000","last_change":"2024-08-28T19:18:32.489439+0000","last_active":"2024-08-28T19:19:08.619235+0000","last_peered":"2024-08-28T19:19:08.619235+0000","last_clean":"2024-08-28T19:19:08.619235+0000","last_became_active":"2024-08-28T19:18:32.489054+0000","last_became_peered":"2024-08-28T19:18:32.489054+0000","last_unstale":"2024-08-28T19:19:08.619235+0000","last_undegraded":"2024-08-28T19:19:08.619235+0000","last_fullsized":"2024-08-28T19:19:08.619235+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:31:32.647384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651121+0000","last_change":"2024-08-28T19:18:30.464993+0000","last_active":"2024-08-28T19:19:11.651121+0000","last_peered":"2024-08-28T19:19:11.651121+0000","last_clean":"2024-08-28T19:19:11.651121+0000","last_became_active":"2024-08-28T19:18:30.463951+0000","last_became_peered":"2024-08-28T19:18:30.463951+0000","last_unstale":"2024-08-28T19:19:11.651121+0000","last_undegraded":"2024-08-28T19:19:11.651121+0000","last_fullsized":"2024-08-28T19:19:11.651121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:06.780058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614780+0000","last_change":"2024-08-28T19:18:39.533061+0000","last_active":"2024-08-28T19:19:08.614780+0000","last_peered":"2024-08-28T19:19:08.614780+0000","last_clean":"2024-08-28T19:19:08.614780+0000","last_became_active":"2024-08-28T19:18:39.532607+0000","last_became_peered":"2024-08-28T19:18:39.532607+0000","last_unstale":"2024-08-28T19:19:08.614780+0000","last_undegraded":"2024-08-28T19:19:08.614780+0000","last_fullsized":"2024-08-28T19:19:08.614780+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:00:57.997665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615094+0000","last_change":"2024-08-28T19:18:32.487937+0000","last_active":"2024-08-28T19:19:08.615094+0000","last_peered":"2024-08-28T19:19:08.615094+0000","last_clean":"2024-08-28T19:19:08.615094+0000","last_became_active":"2024-08-28T19:18:32.487417+0000","last_became_peered":"2024-08-28T19:18:32.487417+0000","last_unstale":"2024-08-28T19:19:08.615094+0000","last_undegraded":"2024-08-28T19:19:08.615094+0000","last_fullsized":"2024-08-28T19:19:08.615094+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:42.497364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.630439+0000","last_change":"2024-08-28T19:18:30.470742+0000","last_active":"2024-08-28T19:19:10.630439+0000","last_peered":"2024-08-28T19:19:10.630439+0000","last_clean":"2024-08-28T19:19:10.630439+0000","last_became_active":"2024-08-28T19:18:30.470135+0000","last_became_peered":"2024-08-28T19:18:30.470135+0000","last_unstale":"2024-08-28T19:19:10.630439+0000","last_undegraded":"2024-08-28T19:19:10.630439+0000","last_fullsized":"2024-08-28T19:19:10.630439+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:49:39.581119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617294+0000","last_change":"2024-08-28T19:18:39.524052+0000","last_active":"2024-08-28T19:19:08.617294+0000","last_peered":"2024-08-28T19:19:08.617294+0000","last_clean":"2024-08-28T19:19:08.617294+0000","last_became_active":"2024-08-28T19:18:39.523601+0000","last_became_peered":"2024-08-28T19:18:39.523601+0000","last_unstale":"2024-08-28T19:19:08.617294+0000","last_undegraded":"2024-08-28T19:19:08.617294+0000","last_fullsized":"2024-08-28T19:19:08.617294+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:54:53.302424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617504+0000","last_change":"2024-08-28T19:18:32.491151+0000","last_active":"2024-08-28T19:19:08.617504+0000","last_peered":"2024-08-28T19:19:08.617504+0000","last_clean":"2024-08-28T19:19:08.617504+0000","last_became_active":"2024-08-28T19:18:32.490798+0000","last_became_peered":"2024-08-28T19:18:32.490798+0000","last_unstale":"2024-08-28T19:19:08.617504+0000","last_undegraded":"2024-08-28T19:19:08.617504+0000","last_fullsized":"2024-08-28T19:19:08.617504+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:41:00.771019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652641+0000","last_change":"2024-08-28T19:18:31.084029+0000","last_active":"2024-08-28T19:19:11.652641+0000","last_peered":"2024-08-28T19:19:11.652641+0000","last_clean":"2024-08-28T19:19:11.652641+0000","last_became_active":"2024-08-28T19:18:31.083507+0000","last_became_peered":"2024-08-28T19:18:31.083507+0000","last_unstale":"2024-08-28T19:19:11.652641+0000","last_undegraded":"2024-08-28T19:19:11.652641+0000","last_fullsized":"2024-08-28T19:19:11.652641+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:15:36.451702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611795+0000","last_change":"2024-08-28T19:18:39.527249+0000","last_active":"2024-08-28T19:19:08.611795+0000","last_peered":"2024-08-28T19:19:08.611795+0000","last_clean":"2024-08-28T19:19:08.611795+0000","last_became_active":"2024-08-28T19:18:39.526834+0000","last_became_peered":"2024-08-28T19:18:39.526834+0000","last_unstale":"2024-08-28T19:19:08.611795+0000","last_undegraded":"2024-08-28T19:19:08.611795+0000","last_fullsized":"2024-08-28T19:19:08.611795+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:48:02.294738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616892+0000","last_change":"2024-08-28T19:18:32.497337+0000","last_active":"2024-08-28T19:19:08.616892+0000","last_peered":"2024-08-28T19:19:08.616892+0000","last_clean":"2024-08-28T19:19:08.616892+0000","last_became_active":"2024-08-28T19:18:32.496929+0000","last_became_peered":"2024-08-28T19:18:32.496929+0000","last_unstale":"2024-08-28T19:19:08.616892+0000","last_undegraded":"2024-08-28T19:19:08.616892+0000","last_fullsized":"2024-08-28T19:19:08.616892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:14.748765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:22.674522+0000","last_change":"2024-08-28T19:18:30.471353+0000","last_active":"2024-08-28T19:19:22.674522+0000","last_peered":"2024-08-28T19:19:22.674522+0000","last_clean":"2024-08-28T19:19:22.674522+0000","last_became_active":"2024-08-28T19:18:30.470283+0000","last_became_peered":"2024-08-28T19:18:30.470283+0000","last_unstale":"2024-08-28T19:19:22.674522+0000","last_undegraded":"2024-08-28T19:19:22.674522+0000","last_fullsized":"2024-08-28T19:19:22.674522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:09:46.444864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614636+0000","last_change":"2024-08-28T19:18:39.519226+0000","last_active":"2024-08-28T19:19:08.614636+0000","last_peered":"2024-08-28T19:19:08.614636+0000","last_clean":"2024-08-28T19:19:08.614636+0000","last_became_active":"2024-08-28T19:18:39.518738+0000","last_became_peered":"2024-08-28T19:18:39.518738+0000","last_unstale":"2024-08-28T19:19:08.614636+0000","last_undegraded":"2024-08-28T19:19:08.614636+0000","last_fullsized":"2024-08-28T19:19:08.614636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:39:35.114702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617127+0000","last_change":"2024-08-28T19:18:32.493060+0000","last_active":"2024-08-28T19:19:08.617127+0000","last_peered":"2024-08-28T19:19:08.617127+0000","last_clean":"2024-08-28T19:19:08.617127+0000","last_became_active":"2024-08-28T19:18:32.492590+0000","last_became_peered":"2024-08-28T19:18:32.492590+0000","last_unstale":"2024-08-28T19:19:08.617127+0000","last_undegraded":"2024-08-28T19:19:08.617127+0000","last_fullsized":"2024-08-28T19:19:08.617127+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:02:02.165301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653316+0000","last_change":"2024-08-28T19:18:30.470674+0000","last_active":"2024-08-28T19:19:11.653316+0000","last_peered":"2024-08-28T19:19:11.653316+0000","last_clean":"2024-08-28T19:19:11.653316+0000","last_became_active":"2024-08-28T19:18:30.470275+0000","last_became_peered":"2024-08-28T19:18:30.470275+0000","last_unstale":"2024-08-28T19:19:11.653316+0000","last_undegraded":"2024-08-28T19:19:11.653316+0000","last_fullsized":"2024-08-28T19:19:11.653316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:01:48.547315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624414+0000","last_change":"2024-08-28T19:19:08.624194+0000","last_active":"2024-08-28T19:19:08.624414+0000","last_peered":"2024-08-28T19:19:08.624414+0000","last_clean":"2024-08-28T19:19:08.624414+0000","last_became_active":"2024-08-28T19:19:08.623681+0000","last_became_peered":"2024-08-28T19:19:08.623681+0000","last_unstale":"2024-08-28T19:19:08.624414+0000","last_undegraded":"2024-08-28T19:19:08.624414+0000","last_fullsized":"2024-08-28T19:19:08.624414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:21:11.099931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611866+0000","last_change":"2024-08-28T19:18:32.488210+0000","last_active":"2024-08-28T19:19:08.611866+0000","last_peered":"2024-08-28T19:19:08.611866+0000","last_clean":"2024-08-28T19:19:08.611866+0000","last_became_active":"2024-08-28T19:18:32.487853+0000","last_became_peered":"2024-08-28T19:18:32.487853+0000","last_unstale":"2024-08-28T19:19:08.611866+0000","last_undegraded":"2024-08-28T19:19:08.611866+0000","last_fullsized":"2024-08-28T19:19:08.611866+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:20:51.808958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.653792+0000","last_change":"2024-08-28T19:18:30.465306+0000","last_active":"2024-08-28T19:19:12.653792+0000","last_peered":"2024-08-28T19:19:12.653792+0000","last_clean":"2024-08-28T19:19:12.653792+0000","last_became_active":"2024-08-28T19:18:30.464850+0000","last_became_peered":"2024-08-28T19:18:30.464850+0000","last_unstale":"2024-08-28T19:19:12.653792+0000","last_undegraded":"2024-08-28T19:19:12.653792+0000","last_fullsized":"2024-08-28T19:19:12.653792+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:24:26.628999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616620+0000","last_change":"2024-08-28T19:18:39.524005+0000","last_active":"2024-08-28T19:19:08.616620+0000","last_peered":"2024-08-28T19:19:08.616620+0000","last_clean":"2024-08-28T19:19:08.616620+0000","last_became_active":"2024-08-28T19:18:39.523594+0000","last_became_peered":"2024-08-28T19:18:39.523594+0000","last_unstale":"2024-08-28T19:19:08.616620+0000","last_undegraded":"2024-08-28T19:19:08.616620+0000","last_fullsized":"2024-08-28T19:19:08.616620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:35:32.484399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619776+0000","last_change":"2024-08-28T19:18:32.489088+0000","last_active":"2024-08-28T19:19:08.619776+0000","last_peered":"2024-08-28T19:19:08.619776+0000","last_clean":"2024-08-28T19:19:08.619776+0000","last_became_active":"2024-08-28T19:18:32.488717+0000","last_became_peered":"2024-08-28T19:18:32.488717+0000","last_unstale":"2024-08-28T19:19:08.619776+0000","last_undegraded":"2024-08-28T19:19:08.619776+0000","last_fullsized":"2024-08-28T19:19:08.619776+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:25.468919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"82'32","reported_seq":100,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:24.681743+0000","last_change":"2024-08-28T19:18:30.468674+0000","last_active":"2024-08-28T19:19:24.681743+0000","last_peered":"2024-08-28T19:19:24.681743+0000","last_clean":"2024-08-28T19:19:24.681743+0000","last_became_active":"2024-08-28T19:18:30.468300+0000","last_became_peered":"2024-08-28T19:18:30.468300+0000","last_unstale":"2024-08-28T19:19:24.681743+0000","last_undegraded":"2024-08-28T19:19:24.681743+0000","last_fullsized":"2024-08-28T19:19:24.681743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:12:06.237498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54392,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":69,"num_write":34,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614424+0000","last_change":"2024-08-28T19:18:39.526131+0000","last_active":"2024-08-28T19:19:08.614424+0000","last_peered":"2024-08-28T19:19:08.614424+0000","last_clean":"2024-08-28T19:19:08.614424+0000","last_became_active":"2024-08-28T19:18:39.525708+0000","last_became_peered":"2024-08-28T19:18:39.525708+0000","last_unstale":"2024-08-28T19:19:08.614424+0000","last_undegraded":"2024-08-28T19:19:08.614424+0000","last_fullsized":"2024-08-28T19:19:08.614424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:21:52.156487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628145+0000","last_change":"2024-08-28T19:18:32.487634+0000","last_active":"2024-08-28T19:19:08.628145+0000","last_peered":"2024-08-28T19:19:08.628145+0000","last_clean":"2024-08-28T19:19:08.628145+0000","last_became_active":"2024-08-28T19:18:32.486702+0000","last_became_peered":"2024-08-28T19:18:32.486702+0000","last_unstale":"2024-08-28T19:19:08.628145+0000","last_undegraded":"2024-08-28T19:19:08.628145+0000","last_fullsized":"2024-08-28T19:19:08.628145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:08:12.606905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.643469+0000","last_change":"2024-08-28T19:18:31.083494+0000","last_active":"2024-08-28T19:19:10.643469+0000","last_peered":"2024-08-28T19:19:10.643469+0000","last_clean":"2024-08-28T19:19:10.643469+0000","last_became_active":"2024-08-28T19:18:31.083086+0000","last_became_peered":"2024-08-28T19:18:31.083086+0000","last_unstale":"2024-08-28T19:19:10.643469+0000","last_undegraded":"2024-08-28T19:19:10.643469+0000","last_fullsized":"2024-08-28T19:19:10.643469+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:14:54.388364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618783+0000","last_change":"2024-08-28T19:18:39.532155+0000","last_active":"2024-08-28T19:19:08.618783+0000","last_peered":"2024-08-28T19:19:08.618783+0000","last_clean":"2024-08-28T19:19:08.618783+0000","last_became_active":"2024-08-28T19:18:39.531565+0000","last_became_peered":"2024-08-28T19:18:39.531565+0000","last_unstale":"2024-08-28T19:19:08.618783+0000","last_undegraded":"2024-08-28T19:19:08.618783+0000","last_fullsized":"2024-08-28T19:19:08.618783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:29:48.824622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629297+0000","last_change":"2024-08-28T19:18:32.496296+0000","last_active":"2024-08-28T19:19:08.629297+0000","last_peered":"2024-08-28T19:19:08.629297+0000","last_clean":"2024-08-28T19:19:08.629297+0000","last_became_active":"2024-08-28T19:18:32.495680+0000","last_became_peered":"2024-08-28T19:18:32.495680+0000","last_unstale":"2024-08-28T19:19:08.629297+0000","last_undegraded":"2024-08-28T19:19:08.629297+0000","last_fullsized":"2024-08-28T19:19:08.629297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:27:34.544529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652615+0000","last_change":"2024-08-28T19:19:08.631758+0000","last_active":"2024-08-28T19:19:11.652615+0000","last_peered":"2024-08-28T19:19:11.652615+0000","last_clean":"2024-08-28T19:19:11.652615+0000","last_became_active":"2024-08-28T19:19:08.631202+0000","last_became_peered":"2024-08-28T19:19:08.631202+0000","last_unstale":"2024-08-28T19:19:11.652615+0000","last_undegraded":"2024-08-28T19:19:11.652615+0000","last_fullsized":"2024-08-28T19:19:11.652615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:27:43.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614142+0000","last_change":"2024-08-28T19:18:39.531157+0000","last_active":"2024-08-28T19:19:08.614142+0000","last_peered":"2024-08-28T19:19:08.614142+0000","last_clean":"2024-08-28T19:19:08.614142+0000","last_became_active":"2024-08-28T19:18:39.530675+0000","last_became_peered":"2024-08-28T19:18:39.530675+0000","last_unstale":"2024-08-28T19:19:08.614142+0000","last_undegraded":"2024-08-28T19:19:08.614142+0000","last_fullsized":"2024-08-28T19:19:08.614142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:38:40.414854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629314+0000","last_change":"2024-08-28T19:18:32.482182+0000","last_active":"2024-08-28T19:19:08.629314+0000","last_peered":"2024-08-28T19:19:08.629314+0000","last_clean":"2024-08-28T19:19:08.629314+0000","last_became_active":"2024-08-28T19:18:32.481689+0000","last_became_peered":"2024-08-28T19:18:32.481689+0000","last_unstale":"2024-08-28T19:19:08.629314+0000","last_undegraded":"2024-08-28T19:19:08.629314+0000","last_fullsized":"2024-08-28T19:19:08.629314+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:13:29.278469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"82'4","reported_seq":65,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:24.676242+0000","last_change":"2024-08-28T19:18:30.475244+0000","last_active":"2024-08-28T19:19:24.676242+0000","last_peered":"2024-08-28T19:19:24.676242+0000","last_clean":"2024-08-28T19:19:24.676242+0000","last_became_active":"2024-08-28T19:18:30.474861+0000","last_became_peered":"2024-08-28T19:18:30.474861+0000","last_unstale":"2024-08-28T19:19:24.676242+0000","last_undegraded":"2024-08-28T19:19:24.676242+0000","last_fullsized":"2024-08-28T19:19:24.676242+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:27:09.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631762+0000","last_change":"2024-08-28T19:18:39.520022+0000","last_active":"2024-08-28T19:19:08.631762+0000","last_peered":"2024-08-28T19:19:08.631762+0000","last_clean":"2024-08-28T19:19:08.631762+0000","last_became_active":"2024-08-28T19:18:39.519584+0000","last_became_peered":"2024-08-28T19:18:39.519584+0000","last_unstale":"2024-08-28T19:19:08.631762+0000","last_undegraded":"2024-08-28T19:19:08.631762+0000","last_fullsized":"2024-08-28T19:19:08.631762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:27:15.851303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628767+0000","last_change":"2024-08-28T19:18:39.522607+0000","last_active":"2024-08-28T19:19:08.628767+0000","last_peered":"2024-08-28T19:19:08.628767+0000","last_clean":"2024-08-28T19:19:08.628767+0000","last_became_active":"2024-08-28T19:18:39.522204+0000","last_became_peered":"2024-08-28T19:18:39.522204+0000","last_unstale":"2024-08-28T19:19:08.628767+0000","last_undegraded":"2024-08-28T19:19:08.628767+0000","last_fullsized":"2024-08-28T19:19:08.628767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:02:08.865294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614380+0000","last_change":"2024-08-28T19:18:32.485577+0000","last_active":"2024-08-28T19:19:08.614380+0000","last_peered":"2024-08-28T19:19:08.614380+0000","last_clean":"2024-08-28T19:19:08.614380+0000","last_became_active":"2024-08-28T19:18:32.485243+0000","last_became_peered":"2024-08-28T19:18:32.485243+0000","last_unstale":"2024-08-28T19:19:08.614380+0000","last_undegraded":"2024-08-28T19:19:08.614380+0000","last_fullsized":"2024-08-28T19:19:08.614380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:05:53.019365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653809+0000","last_change":"2024-08-28T19:18:30.470670+0000","last_active":"2024-08-28T19:19:11.653809+0000","last_peered":"2024-08-28T19:19:11.653809+0000","last_clean":"2024-08-28T19:19:11.653809+0000","last_became_active":"2024-08-28T19:18:30.470264+0000","last_became_peered":"2024-08-28T19:18:30.470264+0000","last_unstale":"2024-08-28T19:19:11.653809+0000","last_undegraded":"2024-08-28T19:19:11.653809+0000","last_fullsized":"2024-08-28T19:19:11.653809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:35.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615236+0000","last_change":"2024-08-28T19:18:39.530948+0000","last_active":"2024-08-28T19:19:08.615236+0000","last_peered":"2024-08-28T19:19:08.615236+0000","last_clean":"2024-08-28T19:19:08.615236+0000","last_became_active":"2024-08-28T19:18:39.530473+0000","last_became_peered":"2024-08-28T19:18:39.530473+0000","last_unstale":"2024-08-28T19:19:08.615236+0000","last_undegraded":"2024-08-28T19:19:08.615236+0000","last_fullsized":"2024-08-28T19:19:08.615236+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:21:57.504582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632033+0000","last_change":"2024-08-28T19:18:32.486027+0000","last_active":"2024-08-28T19:19:08.632033+0000","last_peered":"2024-08-28T19:19:08.632033+0000","last_clean":"2024-08-28T19:19:08.632033+0000","last_became_active":"2024-08-28T19:18:32.483273+0000","last_became_peered":"2024-08-28T19:18:32.483273+0000","last_unstale":"2024-08-28T19:19:08.632033+0000","last_undegraded":"2024-08-28T19:19:08.632033+0000","last_fullsized":"2024-08-28T19:19:08.632033+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:47:37.680695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:09.637193+0000","last_change":"2024-08-28T19:18:30.464811+0000","last_active":"2024-08-28T19:19:09.637193+0000","last_peered":"2024-08-28T19:19:09.637193+0000","last_clean":"2024-08-28T19:19:09.637193+0000","last_became_active":"2024-08-28T19:18:30.463986+0000","last_became_peered":"2024-08-28T19:18:30.463986+0000","last_unstale":"2024-08-28T19:19:09.637193+0000","last_undegraded":"2024-08-28T19:19:09.637193+0000","last_fullsized":"2024-08-28T19:19:09.637193+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:43:46.582993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.642972+0000","last_change":"2024-08-28T19:19:08.631382+0000","last_active":"2024-08-28T19:19:10.642972+0000","last_peered":"2024-08-28T19:19:10.642972+0000","last_clean":"2024-08-28T19:19:10.642972+0000","last_became_active":"2024-08-28T19:19:08.630869+0000","last_became_peered":"2024-08-28T19:19:08.630869+0000","last_unstale":"2024-08-28T19:19:10.642972+0000","last_undegraded":"2024-08-28T19:19:10.642972+0000","last_fullsized":"2024-08-28T19:19:10.642972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:32:54.942408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615952+0000","last_change":"2024-08-28T19:18:32.484120+0000","last_active":"2024-08-28T19:19:08.615952+0000","last_peered":"2024-08-28T19:19:08.615952+0000","last_clean":"2024-08-28T19:19:08.615952+0000","last_became_active":"2024-08-28T19:18:32.483013+0000","last_became_peered":"2024-08-28T19:18:32.483013+0000","last_unstale":"2024-08-28T19:19:08.615952+0000","last_undegraded":"2024-08-28T19:19:08.615952+0000","last_fullsized":"2024-08-28T19:19:08.615952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:59:41.195873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622328+0000","last_change":"2024-08-28T19:18:39.531046+0000","last_active":"2024-08-28T19:19:08.622328+0000","last_peered":"2024-08-28T19:19:08.622328+0000","last_clean":"2024-08-28T19:19:08.622328+0000","last_became_active":"2024-08-28T19:18:39.530407+0000","last_became_peered":"2024-08-28T19:18:39.530407+0000","last_unstale":"2024-08-28T19:19:08.622328+0000","last_undegraded":"2024-08-28T19:19:08.622328+0000","last_fullsized":"2024-08-28T19:19:08.622328+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:47:15.173021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'6","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:21.672671+0000","last_change":"2024-08-28T19:18:30.471268+0000","last_active":"2024-08-28T19:19:21.672671+0000","last_peered":"2024-08-28T19:19:21.672671+0000","last_clean":"2024-08-28T19:19:21.672671+0000","last_became_active":"2024-08-28T19:18:30.470845+0000","last_became_peered":"2024-08-28T19:18:30.470845+0000","last_unstale":"2024-08-28T19:19:21.672671+0000","last_undegraded":"2024-08-28T19:19:21.672671+0000","last_fullsized":"2024-08-28T19:19:21.672671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:18:18.086205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619195+0000","last_change":"2024-08-28T19:18:32.479728+0000","last_active":"2024-08-28T19:19:08.619195+0000","last_peered":"2024-08-28T19:19:08.619195+0000","last_clean":"2024-08-28T19:19:08.619195+0000","last_became_active":"2024-08-28T19:18:32.478623+0000","last_became_peered":"2024-08-28T19:18:32.478623+0000","last_unstale":"2024-08-28T19:19:08.619195+0000","last_undegraded":"2024-08-28T19:19:08.619195+0000","last_fullsized":"2024-08-28T19:19:08.619195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:14:49.914808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.623192+0000","last_change":"2024-08-28T19:18:39.531308+0000","last_active":"2024-08-28T19:19:08.623192+0000","last_peered":"2024-08-28T19:19:08.623192+0000","last_clean":"2024-08-28T19:19:08.623192+0000","last_became_active":"2024-08-28T19:18:39.530403+0000","last_became_peered":"2024-08-28T19:18:39.530403+0000","last_unstale":"2024-08-28T19:19:08.623192+0000","last_undegraded":"2024-08-28T19:19:08.623192+0000","last_fullsized":"2024-08-28T19:19:08.623192+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:33.139668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614114+0000","last_change":"2024-08-28T19:18:32.494621+0000","last_active":"2024-08-28T19:19:08.614114+0000","last_peered":"2024-08-28T19:19:08.614114+0000","last_clean":"2024-08-28T19:19:08.614114+0000","last_became_active":"2024-08-28T19:18:32.494217+0000","last_became_peered":"2024-08-28T19:18:32.494217+0000","last_unstale":"2024-08-28T19:19:08.614114+0000","last_undegraded":"2024-08-28T19:19:08.614114+0000","last_fullsized":"2024-08-28T19:19:08.614114+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:54:15.419470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622753+0000","last_change":"2024-08-28T19:18:32.484716+0000","last_active":"2024-08-28T19:19:08.622753+0000","last_peered":"2024-08-28T19:19:08.622753+0000","last_clean":"2024-08-28T19:19:08.622753+0000","last_became_active":"2024-08-28T19:18:32.484294+0000","last_became_peered":"2024-08-28T19:18:32.484294+0000","last_unstale":"2024-08-28T19:19:08.622753+0000","last_undegraded":"2024-08-28T19:19:08.622753+0000","last_fullsized":"2024-08-28T19:19:08.622753+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:39:45.045607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.621987+0000","last_change":"2024-08-28T19:18:32.484286+0000","last_active":"2024-08-28T19:19:08.621987+0000","last_peered":"2024-08-28T19:19:08.621987+0000","last_clean":"2024-08-28T19:19:08.621987+0000","last_became_active":"2024-08-28T19:18:32.483354+0000","last_became_peered":"2024-08-28T19:18:32.483354+0000","last_unstale":"2024-08-28T19:19:08.621987+0000","last_undegraded":"2024-08-28T19:19:08.621987+0000","last_fullsized":"2024-08-28T19:19:08.621987+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:09.546749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628505+0000","last_change":"2024-08-28T19:18:32.496985+0000","last_active":"2024-08-28T19:19:08.628505+0000","last_peered":"2024-08-28T19:19:08.628505+0000","last_clean":"2024-08-28T19:19:08.628505+0000","last_became_active":"2024-08-28T19:18:32.495874+0000","last_became_peered":"2024-08-28T19:18:32.495874+0000","last_unstale":"2024-08-28T19:19:08.628505+0000","last_undegraded":"2024-08-28T19:19:08.628505+0000","last_fullsized":"2024-08-28T19:19:08.628505+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:42.329572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612181+0000","last_change":"2024-08-28T19:18:32.485446+0000","last_active":"2024-08-28T19:19:08.612181+0000","last_peered":"2024-08-28T19:19:08.612181+0000","last_clean":"2024-08-28T19:19:08.612181+0000","last_became_active":"2024-08-28T19:18:32.484197+0000","last_became_peered":"2024-08-28T19:18:32.484197+0000","last_unstale":"2024-08-28T19:19:08.612181+0000","last_undegraded":"2024-08-28T19:19:08.612181+0000","last_fullsized":"2024-08-28T19:19:08.612181+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:47:50.527915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616380+0000","last_change":"2024-08-28T19:18:32.489452+0000","last_active":"2024-08-28T19:19:08.616380+0000","last_peered":"2024-08-28T19:19:08.616380+0000","last_clean":"2024-08-28T19:19:08.616380+0000","last_became_active":"2024-08-28T19:18:32.487482+0000","last_became_peered":"2024-08-28T19:18:32.487482+0000","last_unstale":"2024-08-28T19:19:08.616380+0000","last_undegraded":"2024-08-28T19:19:08.616380+0000","last_fullsized":"2024-08-28T19:19:08.616380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:52:08.175080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622242+0000","last_change":"2024-08-28T19:18:32.483850+0000","last_active":"2024-08-28T19:19:08.622242+0000","last_peered":"2024-08-28T19:19:08.622242+0000","last_clean":"2024-08-28T19:19:08.622242+0000","last_became_active":"2024-08-28T19:18:32.483476+0000","last_became_peered":"2024-08-28T19:18:32.483476+0000","last_unstale":"2024-08-28T19:19:08.622242+0000","last_undegraded":"2024-08-28T19:19:08.622242+0000","last_fullsized":"2024-08-28T19:19:08.622242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:32:37.465218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632706+0000","last_change":"2024-08-28T19:18:32.492252+0000","last_active":"2024-08-28T19:19:08.632706+0000","last_peered":"2024-08-28T19:19:08.632706+0000","last_clean":"2024-08-28T19:19:08.632706+0000","last_became_active":"2024-08-28T19:18:32.491659+0000","last_became_peered":"2024-08-28T19:18:32.491659+0000","last_unstale":"2024-08-28T19:19:08.632706+0000","last_undegraded":"2024-08-28T19:19:08.632706+0000","last_fullsized":"2024-08-28T19:19:08.632706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:19:28.901559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615146+0000","last_change":"2024-08-28T19:18:32.491940+0000","last_active":"2024-08-28T19:19:08.615146+0000","last_peered":"2024-08-28T19:19:08.615146+0000","last_clean":"2024-08-28T19:19:08.615146+0000","last_became_active":"2024-08-28T19:18:32.487474+0000","last_became_peered":"2024-08-28T19:18:32.487474+0000","last_unstale":"2024-08-28T19:19:08.615146+0000","last_undegraded":"2024-08-28T19:19:08.615146+0000","last_fullsized":"2024-08-28T19:19:08.615146+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:04:32.508620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631980+0000","last_change":"2024-08-28T19:18:32.485200+0000","last_active":"2024-08-28T19:19:08.631980+0000","last_peered":"2024-08-28T19:19:08.631980+0000","last_clean":"2024-08-28T19:19:08.631980+0000","last_became_active":"2024-08-28T19:18:32.484858+0000","last_became_peered":"2024-08-28T19:18:32.484858+0000","last_unstale":"2024-08-28T19:19:08.631980+0000","last_undegraded":"2024-08-28T19:19:08.631980+0000","last_fullsized":"2024-08-28T19:19:08.631980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:10:57.925659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627792+0000","last_change":"2024-08-28T19:18:32.489903+0000","last_active":"2024-08-28T19:19:08.627792+0000","last_peered":"2024-08-28T19:19:08.627792+0000","last_clean":"2024-08-28T19:19:08.627792+0000","last_became_active":"2024-08-28T19:18:32.488950+0000","last_became_peered":"2024-08-28T19:18:32.488950+0000","last_unstale":"2024-08-28T19:19:08.627792+0000","last_undegraded":"2024-08-28T19:19:08.627792+0000","last_fullsized":"2024-08-28T19:19:08.627792+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:06:42.284177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612366+0000","last_change":"2024-08-28T19:18:32.488236+0000","last_active":"2024-08-28T19:19:08.612366+0000","last_peered":"2024-08-28T19:19:08.612366+0000","last_clean":"2024-08-28T19:19:08.612366+0000","last_became_active":"2024-08-28T19:18:32.487776+0000","last_became_peered":"2024-08-28T19:18:32.487776+0000","last_unstale":"2024-08-28T19:19:08.612366+0000","last_undegraded":"2024-08-28T19:19:08.612366+0000","last_fullsized":"2024-08-28T19:19:08.612366+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:25:58.754740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616789+0000","last_change":"2024-08-28T19:18:32.485860+0000","last_active":"2024-08-28T19:19:08.616789+0000","last_peered":"2024-08-28T19:19:08.616789+0000","last_clean":"2024-08-28T19:19:08.616789+0000","last_became_active":"2024-08-28T19:18:32.485415+0000","last_became_peered":"2024-08-28T19:18:32.485415+0000","last_unstale":"2024-08-28T19:19:08.616789+0000","last_undegraded":"2024-08-28T19:19:08.616789+0000","last_fullsized":"2024-08-28T19:19:08.616789+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:45:32.868651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.623402+0000","last_change":"2024-08-28T19:18:32.485515+0000","last_active":"2024-08-28T19:19:08.623402+0000","last_peered":"2024-08-28T19:19:08.623402+0000","last_clean":"2024-08-28T19:19:08.623402+0000","last_became_active":"2024-08-28T19:18:32.485075+0000","last_became_peered":"2024-08-28T19:18:32.485075+0000","last_unstale":"2024-08-28T19:19:08.623402+0000","last_undegraded":"2024-08-28T19:19:08.623402+0000","last_fullsized":"2024-08-28T19:19:08.623402+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:01:28.035479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614972+0000","last_change":"2024-08-28T19:18:32.484685+0000","last_active":"2024-08-28T19:19:08.614972+0000","last_peered":"2024-08-28T19:19:08.614972+0000","last_clean":"2024-08-28T19:19:08.614972+0000","last_became_active":"2024-08-28T19:18:32.484095+0000","last_became_peered":"2024-08-28T19:18:32.484095+0000","last_unstale":"2024-08-28T19:19:08.614972+0000","last_undegraded":"2024-08-28T19:19:08.614972+0000","last_fullsized":"2024-08-28T19:19:08.614972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:53:27.229186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615703+0000","last_change":"2024-08-28T19:18:32.488030+0000","last_active":"2024-08-28T19:19:08.615703+0000","last_peered":"2024-08-28T19:19:08.615703+0000","last_clean":"2024-08-28T19:19:08.615703+0000","last_became_active":"2024-08-28T19:18:32.487042+0000","last_became_peered":"2024-08-28T19:18:32.487042+0000","last_unstale":"2024-08-28T19:19:08.615703+0000","last_undegraded":"2024-08-28T19:19:08.615703+0000","last_fullsized":"2024-08-28T19:19:08.615703+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:56.966475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627165+0000","last_change":"2024-08-28T19:18:32.489442+0000","last_active":"2024-08-28T19:19:08.627165+0000","last_peered":"2024-08-28T19:19:08.627165+0000","last_clean":"2024-08-28T19:19:08.627165+0000","last_became_active":"2024-08-28T19:18:32.488405+0000","last_became_peered":"2024-08-28T19:18:32.488405+0000","last_unstale":"2024-08-28T19:19:08.627165+0000","last_undegraded":"2024-08-28T19:19:08.627165+0000","last_fullsized":"2024-08-28T19:19:08.627165+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:21:03.026893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628286+0000","last_change":"2024-08-28T19:18:32.496044+0000","last_active":"2024-08-28T19:19:08.628286+0000","last_peered":"2024-08-28T19:19:08.628286+0000","last_clean":"2024-08-28T19:19:08.628286+0000","last_became_active":"2024-08-28T19:18:32.495549+0000","last_became_peered":"2024-08-28T19:18:32.495549+0000","last_unstale":"2024-08-28T19:19:08.628286+0000","last_undegraded":"2024-08-28T19:19:08.628286+0000","last_fullsized":"2024-08-28T19:19:08.628286+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:06:56.123264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628305+0000","last_change":"2024-08-28T19:18:32.492077+0000","last_active":"2024-08-28T19:19:08.628305+0000","last_peered":"2024-08-28T19:19:08.628305+0000","last_clean":"2024-08-28T19:19:08.628305+0000","last_became_active":"2024-08-28T19:18:32.491722+0000","last_became_peered":"2024-08-28T19:18:32.491722+0000","last_unstale":"2024-08-28T19:19:08.628305+0000","last_undegraded":"2024-08-28T19:19:08.628305+0000","last_fullsized":"2024-08-28T19:19:08.628305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:16:21.697045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612237+0000","last_change":"2024-08-28T19:18:32.479845+0000","last_active":"2024-08-28T19:19:08.612237+0000","last_peered":"2024-08-28T19:19:08.612237+0000","last_clean":"2024-08-28T19:19:08.612237+0000","last_became_active":"2024-08-28T19:18:32.477690+0000","last_became_peered":"2024-08-28T19:18:32.477690+0000","last_unstale":"2024-08-28T19:19:08.612237+0000","last_undegraded":"2024-08-28T19:19:08.612237+0000","last_fullsized":"2024-08-28T19:19:08.612237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:11.281415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624748+0000","last_change":"2024-08-28T19:19:08.624185+0000","last_active":"2024-08-28T19:19:08.624748+0000","last_peered":"2024-08-28T19:19:08.624748+0000","last_clean":"2024-08-28T19:19:08.624748+0000","last_became_active":"2024-08-28T19:19:08.623745+0000","last_became_peered":"2024-08-28T19:19:08.623745+0000","last_unstale":"2024-08-28T19:19:08.624748+0000","last_undegraded":"2024-08-28T19:19:08.624748+0000","last_fullsized":"2024-08-28T19:19:08.624748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:56.657173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619629+0000","last_change":"2024-08-28T19:18:32.499038+0000","last_active":"2024-08-28T19:19:08.619629+0000","last_peered":"2024-08-28T19:19:08.619629+0000","last_clean":"2024-08-28T19:19:08.619629+0000","last_became_active":"2024-08-28T19:18:32.498276+0000","last_became_peered":"2024-08-28T19:18:32.498276+0000","last_unstale":"2024-08-28T19:19:08.619629+0000","last_undegraded":"2024-08-28T19:19:08.619629+0000","last_fullsized":"2024-08-28T19:19:08.619629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:17:45.468448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612960+0000","last_change":"2024-08-28T19:18:32.492183+0000","last_active":"2024-08-28T19:19:08.612960+0000","last_peered":"2024-08-28T19:19:08.612960+0000","last_clean":"2024-08-28T19:19:08.612960+0000","last_became_active":"2024-08-28T19:18:32.491815+0000","last_became_peered":"2024-08-28T19:18:32.491815+0000","last_unstale":"2024-08-28T19:19:08.612960+0000","last_undegraded":"2024-08-28T19:19:08.612960+0000","last_fullsized":"2024-08-28T19:19:08.612960+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:13:11.183955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629556+0000","last_change":"2024-08-28T19:18:32.497399+0000","last_active":"2024-08-28T19:19:08.629556+0000","last_peered":"2024-08-28T19:19:08.629556+0000","last_clean":"2024-08-28T19:19:08.629556+0000","last_became_active":"2024-08-28T19:18:32.496901+0000","last_became_peered":"2024-08-28T19:18:32.496901+0000","last_unstale":"2024-08-28T19:19:08.629556+0000","last_undegraded":"2024-08-28T19:19:08.629556+0000","last_fullsized":"2024-08-28T19:19:08.629556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:05:57.634590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631412+0000","last_change":"2024-08-28T19:18:32.495871+0000","last_active":"2024-08-28T19:19:08.631412+0000","last_peered":"2024-08-28T19:19:08.631412+0000","last_clean":"2024-08-28T19:19:08.631412+0000","last_became_active":"2024-08-28T19:18:32.495515+0000","last_became_peered":"2024-08-28T19:18:32.495515+0000","last_unstale":"2024-08-28T19:19:08.631412+0000","last_undegraded":"2024-08-28T19:19:08.631412+0000","last_fullsized":"2024-08-28T19:19:08.631412+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:53:56.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617058+0000","last_change":"2024-08-28T19:18:32.495711+0000","last_active":"2024-08-28T19:19:08.617058+0000","last_peered":"2024-08-28T19:19:08.617058+0000","last_clean":"2024-08-28T19:19:08.617058+0000","last_became_active":"2024-08-28T19:18:32.495271+0000","last_became_peered":"2024-08-28T19:18:32.495271+0000","last_unstale":"2024-08-28T19:19:08.617058+0000","last_undegraded":"2024-08-28T19:19:08.617058+0000","last_fullsized":"2024-08-28T19:19:08.617058+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:05.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612310+0000","last_change":"2024-08-28T19:18:32.489731+0000","last_active":"2024-08-28T19:19:08.612310+0000","last_peered":"2024-08-28T19:19:08.612310+0000","last_clean":"2024-08-28T19:19:08.612310+0000","last_became_active":"2024-08-28T19:18:32.489366+0000","last_became_peered":"2024-08-28T19:18:32.489366+0000","last_unstale":"2024-08-28T19:19:08.612310+0000","last_undegraded":"2024-08-28T19:19:08.612310+0000","last_fullsized":"2024-08-28T19:19:08.612310+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:52:24.180155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617967+0000","last_change":"2024-08-28T19:18:32.490270+0000","last_active":"2024-08-28T19:19:08.617967+0000","last_peered":"2024-08-28T19:19:08.617967+0000","last_clean":"2024-08-28T19:19:08.617967+0000","last_became_active":"2024-08-28T19:18:32.489667+0000","last_became_peered":"2024-08-28T19:18:32.489667+0000","last_unstale":"2024-08-28T19:19:08.617967+0000","last_undegraded":"2024-08-28T19:19:08.617967+0000","last_fullsized":"2024-08-28T19:19:08.617967+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:18.571978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618388+0000","last_change":"2024-08-28T19:18:32.482553+0000","last_active":"2024-08-28T19:19:08.618388+0000","last_peered":"2024-08-28T19:19:08.618388+0000","last_clean":"2024-08-28T19:19:08.618388+0000","last_became_active":"2024-08-28T19:18:32.481161+0000","last_became_peered":"2024-08-28T19:18:32.481161+0000","last_unstale":"2024-08-28T19:19:08.618388+0000","last_undegraded":"2024-08-28T19:19:08.618388+0000","last_fullsized":"2024-08-28T19:19:08.618388+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:13.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628811+0000","last_change":"2024-08-28T19:18:32.492473+0000","last_active":"2024-08-28T19:19:08.628811+0000","last_peered":"2024-08-28T19:19:08.628811+0000","last_clean":"2024-08-28T19:19:08.628811+0000","last_became_active":"2024-08-28T19:18:32.491897+0000","last_became_peered":"2024-08-28T19:18:32.491897+0000","last_unstale":"2024-08-28T19:19:08.628811+0000","last_undegraded":"2024-08-28T19:19:08.628811+0000","last_fullsized":"2024-08-28T19:19:08.628811+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:24.279391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628899+0000","last_change":"2024-08-28T19:18:32.485621+0000","last_active":"2024-08-28T19:19:08.628899+0000","last_peered":"2024-08-28T19:19:08.628899+0000","last_clean":"2024-08-28T19:19:08.628899+0000","last_became_active":"2024-08-28T19:18:32.485267+0000","last_became_peered":"2024-08-28T19:18:32.485267+0000","last_unstale":"2024-08-28T19:19:08.628899+0000","last_undegraded":"2024-08-28T19:19:08.628899+0000","last_fullsized":"2024-08-28T19:19:08.628899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:30:08.932135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615578+0000","last_change":"2024-08-28T19:18:32.488457+0000","last_active":"2024-08-28T19:19:08.615578+0000","last_peered":"2024-08-28T19:19:08.615578+0000","last_clean":"2024-08-28T19:19:08.615578+0000","last_became_active":"2024-08-28T19:18:32.487555+0000","last_became_peered":"2024-08-28T19:18:32.487555+0000","last_unstale":"2024-08-28T19:19:08.615578+0000","last_undegraded":"2024-08-28T19:19:08.615578+0000","last_fullsized":"2024-08-28T19:19:08.615578+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:03:26.390481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61596,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":168,"num_read_kb":154,"num_write":244,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":184788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":68,"seq":292057776152,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":481155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":62,"seq":266287972380,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1642496,"data_stored":1125787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":57,"seq":244813135906,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":484509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":51,"seq":219043332134,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":970752,"data_stored":480791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":45,"seq":193273528363,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708284,"statfs":{"total":95995035648,"available":95957282816,"internally_reserved":0,"allocated":1560576,"data_stored":1077007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":40,"seq":171798691889,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":476308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888116,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1572864,"data_stored":1068774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051643,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":479991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215167,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40444,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704708,"statfs":{"total":95995035648,"available":95953620992,"internally_reserved":0,"allocated":962560,"data_stored":475625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1032192,"data_stored":531950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":531730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":974848,"data_stored":477373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":56535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":56128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-28T19:19:28.505 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-28T19:19:28.506 INFO:tasks.cephadm:Setting up client nodes... 2024-08-28T19:19:28.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-28T19:19:28.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:28 smithi016 ceph-mon[18556]: pgmap v290: 129 pgs: 129 active+clean; 637 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-08-28T19:19:28.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:28 smithi016 ceph-mon[18556]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:19:28.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:28 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:19:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:28 smithi115 ceph-mon[22184]: pgmap v290: 129 pgs: 129 active+clean; 637 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-08-28T19:19:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:28 smithi115 ceph-mon[22184]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:19:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:28 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:19:29.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:28 smithi028 ceph-mon[21769]: pgmap v290: 129 pgs: 129 active+clean; 637 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-08-28T19:19:29.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:28 smithi028 ceph-mon[21769]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:19:29.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:28 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:19:29.384 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:29.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.880+0000 7f03c90fe640 1 -- 172.21.15.16:0/3917683388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03c4108190 msgr2=0x7f03c41085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:29.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.880+0000 7f03c90fe640 1 --2- 172.21.15.16:0/3917683388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03c4108190 0x7f03c41085a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009900 tx=0x7f03ac02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:29.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.880+0000 7f03c90fe640 1 -- 172.21.15.16:0/3917683388 shutdown_connections 2024-08-28T19:19:29.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.880+0000 7f03c90fe640 1 --2- 172.21.15.16:0/3917683388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c4108ae0 0x7f03c410f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.880+0000 7f03c90fe640 1 --2- 172.21.15.16:0/3917683388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03c4108190 0x7f03c41085a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.880+0000 7f03c90fe640 1 --2- 172.21.15.16:0/3917683388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03c41028a0 0x7f03c4102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.880+0000 7f03c90fe640 1 -- 172.21.15.16:0/3917683388 >> 172.21.15.16:0/3917683388 conn(0x7f03c4079470 msgr2=0x7f03c4079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.880+0000 7f03c90fe640 1 -- 172.21.15.16:0/3917683388 shutdown_connections 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.880+0000 7f03c90fe640 1 -- 172.21.15.16:0/3917683388 wait complete. 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03c90fe640 1 Processor -- start 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03c90fe640 1 -- start start 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03c90fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03c41028a0 0x7f03c419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03c90fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03c4108190 0x7f03c419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03c90fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c4108ae0 0x7f03c419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03c90fe640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03c4112730 con 0x7f03c4108190 2024-08-28T19:19:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03c90fe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03c41125b0 con 0x7f03c4108ae0 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03c90fe640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f03c41128b0 con 0x7f03c41028a0 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03ba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03c4108190 0x7f03c419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03ba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03c4108190 0x7f03c419a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36188/0 (socket says 172.21.15.16:36188) 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03ba59c640 1 -- 172.21.15.16:0/770156615 learned_addr learned my addr 172.21.15.16:0/770156615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03c2d9d640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03c41028a0 0x7f03c419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.881+0000 7f03c359e640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c4108ae0 0x7f03c419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03ba59c640 1 -- 172.21.15.16:0/770156615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03c41028a0 msgr2=0x7f03c419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03ba59c640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03c41028a0 0x7f03c419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03ba59c640 1 -- 172.21.15.16:0/770156615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c4108ae0 msgr2=0x7f03c419fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03ba59c640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c4108ae0 0x7f03c419fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03ba59c640 1 -- 172.21.15.16:0/770156615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c41a01a0 con 0x7f03c4108190 2024-08-28T19:19:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03c359e640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c4108ae0 0x7f03c419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03c2d9d640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03c41028a0 0x7f03c419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03ba59c640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03c4108190 0x7f03c419a7d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f03ac006010 tx=0x7f03ac004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03c0d99640 1 -- 172.21.15.16:0/770156615 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac03f030 con 0x7f03c4108190 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03c90fe640 1 -- 172.21.15.16:0/770156615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03c41a0430 con 0x7f03c4108190 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03c0d99640 1 -- 172.21.15.16:0/770156615 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03ac003aa0 con 0x7f03c4108190 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03c0d99640 1 -- 172.21.15.16:0/770156615 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac035070 con 0x7f03c4108190 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.882+0000 7f03c90fe640 1 -- 172.21.15.16:0/770156615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03c41a6bf0 con 0x7f03c4108190 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.883+0000 7f03c90fe640 1 -- 172.21.15.16:0/770156615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03c4102cb0 con 0x7f03c4108190 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.884+0000 7f03c0d99640 1 -- 172.21.15.16:0/770156615 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f03ac042030 con 0x7f03c4108190 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.885+0000 7f03c0d99640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03b00b4c10 0x7f03b00b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.885+0000 7f03c0d99640 1 -- 172.21.15.16:0/770156615 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f03ac106130 con 0x7f03c4108190 2024-08-28T19:19:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.885+0000 7f03c2d9d640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03b00b4c10 0x7f03b00b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:29.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.886+0000 7f03c2d9d640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03b00b4c10 0x7f03b00b7120 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f03b4005fd0 tx=0x7f03b4005ea0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:29.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:29.888+0000 7f03c0d99640 1 -- 172.21.15.16:0/770156615 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f03ac0ccce0 con 0x7f03c4108190 2024-08-28T19:19:30.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.178+0000 7f03c90fe640 1 -- 172.21.15.16:0/770156615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f03c41a6df0 con 0x7f03c4108190 2024-08-28T19:19:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.182+0000 7f03c0d99640 1 -- 172.21.15.16:0/770156615 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f03ac0d5220 con 0x7f03c4108190 2024-08-28T19:19:30.183 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-08-28T19:19:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBCeM9mKuS7ChAAmgMf4iHBab928xRmX9eo/g== 2024-08-28T19:19:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 -- 172.21.15.16:0/770156615 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03b00b4c10 msgr2=0x7f03b00b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03b00b4c10 0x7f03b00b7120 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f03b4005fd0 tx=0x7f03b4005ea0 comp rx=0 tx=0).stop 2024-08-28T19:19:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 -- 172.21.15.16:0/770156615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03c4108190 msgr2=0x7f03c419a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03c4108190 0x7f03c419a7d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f03ac006010 tx=0x7f03ac004060 comp rx=0 tx=0).stop 2024-08-28T19:19:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 -- 172.21.15.16:0/770156615 shutdown_connections 2024-08-28T19:19:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03b00b4c10 0x7f03b00b7120 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c4108ae0 0x7f03c419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03c4108190 0x7f03c419a7d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 --2- 172.21.15.16:0/770156615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03c41028a0 0x7f03c419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 -- 172.21.15.16:0/770156615 >> 172.21.15.16:0/770156615 conn(0x7f03c4079470 msgr2=0x7f03c406bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 -- 172.21.15.16:0/770156615 shutdown_connections 2024-08-28T19:19:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:30.185+0000 7f03c90fe640 1 -- 172.21.15.16:0/770156615 wait complete. 2024-08-28T19:19:30.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:30 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:30 smithi016 ceph-mon[18556]: pgmap v291: 129 pgs: 129 active+clean; 637 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-08-28T19:19:30.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:30 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/770156615' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-28T19:19:30.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:30 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/770156615' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-28T19:19:30.795 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:19:30.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-08-28T19:19:30.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-28T19:19:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:30 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:30 smithi028 ceph-mon[21769]: pgmap v291: 129 pgs: 129 active+clean; 637 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-08-28T19:19:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:30 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/770156615' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-28T19:19:30.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:30 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/770156615' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-28T19:19:30.832 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph config log 1 --format=json 2024-08-28T19:19:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:30 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:30 smithi115 ceph-mon[22184]: pgmap v291: 129 pgs: 129 active+clean; 637 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-08-28T19:19:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:30 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/770156615' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-28T19:19:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:30 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/770156615' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-28T19:19:31.146 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:31.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.647+0000 7f2599d9e640 1 -- 172.21.15.16:0/1095562039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594109b90 msgr2=0x7f25941103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:31.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.647+0000 7f2599d9e640 1 --2- 172.21.15.16:0/1095562039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594109b90 0x7f25941103c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f258c009ca0 tx=0x7f258c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:31.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.647+0000 7f2599d9e640 1 -- 172.21.15.16:0/1095562039 shutdown_connections 2024-08-28T19:19:31.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.647+0000 7f2599d9e640 1 --2- 172.21.15.16:0/1095562039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594109b90 0x7f25941103c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:31.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.647+0000 7f2599d9e640 1 --2- 172.21.15.16:0/1095562039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2594109240 0x7f2594109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:31.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.647+0000 7f2599d9e640 1 --2- 172.21.15.16:0/1095562039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2594103900 0x7f2594103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:31.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.647+0000 7f2599d9e640 1 -- 172.21.15.16:0/1095562039 >> 172.21.15.16:0/1095562039 conn(0x7f259407b5f0 msgr2=0x7f259407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:31.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.647+0000 7f2599d9e640 1 -- 172.21.15.16:0/1095562039 shutdown_connections 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.647+0000 7f2599d9e640 1 -- 172.21.15.16:0/1095562039 wait complete. 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.647+0000 7f2599d9e640 1 Processor -- start 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2599d9e640 1 -- start start 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2599d9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2594103900 0x7f259419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2599d9e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2594109240 0x7f259419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2599d9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594109b90 0x7f259419f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2599d9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2594113850 con 0x7f2594109b90 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2599d9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25941136d0 con 0x7f2594103900 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2599d9e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f25941139d0 con 0x7f2594109240 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2593fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594109b90 0x7f259419f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2593fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594109b90 0x7f259419f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36212/0 (socket says 172.21.15.16:36212) 2024-08-28T19:19:31.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2593fff640 1 -- 172.21.15.16:0/1900024918 learned_addr learned my addr 172.21.15.16:0/1900024918 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f258bfff640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2594109240 0x7f259419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f25937fe640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2594103900 0x7f259419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2593fff640 1 -- 172.21.15.16:0/1900024918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2594109240 msgr2=0x7f259419a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2593fff640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2594109240 0x7f259419a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2593fff640 1 -- 172.21.15.16:0/1900024918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2594103900 msgr2=0x7f259419a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2593fff640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2594103900 0x7f259419a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f2593fff640 1 -- 172.21.15.16:0/1900024918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25941a0030 con 0x7f2594109b90 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.648+0000 7f25937fe640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2594103900 0x7f259419a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.649+0000 7f2593fff640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594109b90 0x7f259419f9c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f258c005cc0 tx=0x7f258c02f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.649+0000 7f25917fa640 1 -- 172.21.15.16:0/1900024918 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c003cf0 con 0x7f2594109b90 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.649+0000 7f25917fa640 1 -- 172.21.15.16:0/1900024918 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f258c0044f0 con 0x7f2594109b90 2024-08-28T19:19:31.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.649+0000 7f25917fa640 1 -- 172.21.15.16:0/1900024918 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c02fd10 con 0x7f2594109b90 2024-08-28T19:19:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.649+0000 7f2599d9e640 1 -- 172.21.15.16:0/1900024918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25941a02c0 con 0x7f2594109b90 2024-08-28T19:19:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.649+0000 7f2599d9e640 1 -- 172.21.15.16:0/1900024918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25941a0800 con 0x7f2594109b90 2024-08-28T19:19:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.650+0000 7f2599d9e640 1 -- 172.21.15.16:0/1900024918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2594103d10 con 0x7f2594109b90 2024-08-28T19:19:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.652+0000 7f25917fa640 1 -- 172.21.15.16:0/1900024918 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f258c004040 con 0x7f2594109b90 2024-08-28T19:19:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.653+0000 7f25917fa640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25800b4c40 0x7f25800b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.653+0000 7f25937fe640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25800b4c40 0x7f25800b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.653+0000 7f25917fa640 1 -- 172.21.15.16:0/1900024918 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f258c106e30 con 0x7f2594109b90 2024-08-28T19:19:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.654+0000 7f25937fe640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25800b4c40 0x7f25800b7150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f258400abe0 tx=0x7f258400a5c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.657+0000 7f25917fa640 1 -- 172.21.15.16:0/1900024918 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f258c0cd980 con 0x7f2594109b90 2024-08-28T19:19:31.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.844+0000 7f2599d9e640 1 -- 172.21.15.16:0/1900024918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f259419b2b0 con 0x7f2594109b90 2024-08-28T19:19:31.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.844+0000 7f25917fa640 1 -- 172.21.15.16:0/1900024918 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v57) ==== 86+0+133 (secure 0 0 0) 0x7f259419b2b0 con 0x7f2594109b90 2024-08-28T19:19:31.846 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:19:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.846+0000 7f2599d9e640 1 -- 172.21.15.16:0/1900024918 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25800b4c40 msgr2=0x7f25800b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.846+0000 7f2599d9e640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25800b4c40 0x7f25800b7150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f258400abe0 tx=0x7f258400a5c0 comp rx=0 tx=0).stop 2024-08-28T19:19:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.846+0000 7f2599d9e640 1 -- 172.21.15.16:0/1900024918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594109b90 msgr2=0x7f259419f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.846+0000 7f2599d9e640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594109b90 0x7f259419f9c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f258c005cc0 tx=0x7f258c02f470 comp rx=0 tx=0).stop 2024-08-28T19:19:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.847+0000 7f2599d9e640 1 -- 172.21.15.16:0/1900024918 shutdown_connections 2024-08-28T19:19:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.847+0000 7f2599d9e640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594109b90 0x7f259419f9c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.847+0000 7f2599d9e640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2594109240 0x7f259419a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.847+0000 7f2599d9e640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25800b4c40 0x7f25800b7150 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.847+0000 7f2599d9e640 1 --2- 172.21.15.16:0/1900024918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2594103900 0x7f259419a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.847+0000 7f2599d9e640 1 -- 172.21.15.16:0/1900024918 >> 172.21.15.16:0/1900024918 conn(0x7f259407b5f0 msgr2=0x7f2594107290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.847+0000 7f2599d9e640 1 -- 172.21.15.16:0/1900024918 shutdown_connections 2024-08-28T19:19:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:31.847+0000 7f2599d9e640 1 -- 172.21.15.16:0/1900024918 wait complete. 2024-08-28T19:19:32.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:31 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:32.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:31 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:32.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:31 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:19:32.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:31 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:19:32.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:31 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:32.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:31 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1900024918' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-08-28T19:19:32.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:31 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:32.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:31 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:32.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:31 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:19:32.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:31 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:19:32.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:31 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:32.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:31 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1900024918' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-08-28T19:19:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:31 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:31 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:31 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:19:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:31 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:19:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:31 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:19:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:31 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1900024918' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-08-28T19:19:32.585 INFO:teuthology.orchestra.run.smithi016.stdout:[{"version":57,"timestamp":"2024-08-28T19:18:23.384009+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-08-28T19:19:32.585 INFO:tasks.ceph_manager:config epoch is 57 2024-08-28T19:19:32.585 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-28T19:19:32.586 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-08-28T19:19:32.586 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph mgr dump --format=json 2024-08-28T19:19:32.843 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:33.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:32 smithi016 ceph-mon[18556]: pgmap v292: 129 pgs: 129 active+clean; 645 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-08-28T19:19:33.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:32 smithi028 ceph-mon[21769]: pgmap v292: 129 pgs: 129 active+clean; 645 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-08-28T19:19:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.313+0000 7f24dcd5f640 1 -- 172.21.15.16:0/2788144507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8109240 msgr2=0x7f24d8109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.313+0000 7f24dcd5f640 1 --2- 172.21.15.16:0/2788144507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8109240 0x7f24d8109650 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009900 tx=0x7f24cc02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.313+0000 7f24dcd5f640 1 -- 172.21.15.16:0/2788144507 shutdown_connections 2024-08-28T19:19:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.313+0000 7f24dcd5f640 1 --2- 172.21.15.16:0/2788144507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24d8109b90 0x7f24d81103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.313+0000 7f24dcd5f640 1 --2- 172.21.15.16:0/2788144507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8109240 0x7f24d8109650 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.313+0000 7f24dcd5f640 1 --2- 172.21.15.16:0/2788144507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d8103900 0x7f24d8103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.313+0000 7f24dcd5f640 1 -- 172.21.15.16:0/2788144507 >> 172.21.15.16:0/2788144507 conn(0x7f24d807b5f0 msgr2=0x7f24d807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.313+0000 7f24dcd5f640 1 -- 172.21.15.16:0/2788144507 shutdown_connections 2024-08-28T19:19:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.313+0000 7f24dcd5f640 1 -- 172.21.15.16:0/2788144507 wait complete. 2024-08-28T19:19:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.314+0000 7f24dcd5f640 1 Processor -- start 2024-08-28T19:19:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.314+0000 7f24dcd5f640 1 -- start start 2024-08-28T19:19:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.314+0000 7f24dcd5f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d8103900 0x7f24d819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.314+0000 7f24dcd5f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8109240 0x7f24d819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.314+0000 7f24dcd5f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24d8109b90 0x7f24d819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24dcd5f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24d8113850 con 0x7f24d8109240 2024-08-28T19:19:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24dcd5f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81136d0 con 0x7f24d8103900 2024-08-28T19:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24dcd5f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81139d0 con 0x7f24d8109b90 2024-08-28T19:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24d6d9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24d8109b90 0x7f24d819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d8103900 0x7f24d819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24d6d9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24d8109b90 0x7f24d819faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:44168/0 (socket says 172.21.15.16:44168) 2024-08-28T19:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d8103900 0x7f24d819a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:55192/0 (socket says 172.21.15.16:55192) 2024-08-28T19:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24d6d9d640 1 -- 172.21.15.16:0/486165677 learned_addr learned my addr 172.21.15.16:0/486165677 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24d5d9b640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8109240 0x7f24d819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24d659c640 1 -- 172.21.15.16:0/486165677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24d8109b90 msgr2=0x7f24d819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24d659c640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24d8109b90 0x7f24d819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24d659c640 1 -- 172.21.15.16:0/486165677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8109240 msgr2=0x7f24d819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24d659c640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8109240 0x7f24d819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.315+0000 7f24d659c640 1 -- 172.21.15.16:0/486165677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d81a01d0 con 0x7f24d8103900 2024-08-28T19:19:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.316+0000 7f24d659c640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d8103900 0x7f24d819a260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f24c800d6d0 tx=0x7f24c800db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.316+0000 7f24c77fe640 1 -- 172.21.15.16:0/486165677 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8017070 con 0x7f24d8103900 2024-08-28T19:19:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.316+0000 7f24dcd5f640 1 -- 172.21.15.16:0/486165677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d81a04c0 con 0x7f24d8103900 2024-08-28T19:19:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.316+0000 7f24c77fe640 1 -- 172.21.15.16:0/486165677 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24c8009820 con 0x7f24d8103900 2024-08-28T19:19:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.316+0000 7f24c77fe640 1 -- 172.21.15.16:0/486165677 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8004c40 con 0x7f24d8103900 2024-08-28T19:19:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.316+0000 7f24dcd5f640 1 -- 172.21.15.16:0/486165677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24d81a6c20 con 0x7f24d8103900 2024-08-28T19:19:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.317+0000 7f24dcd5f640 1 -- 172.21.15.16:0/486165677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24d8103d10 con 0x7f24d8103900 2024-08-28T19:19:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.319+0000 7f24c77fe640 1 -- 172.21.15.16:0/486165677 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f24c8005220 con 0x7f24d8103900 2024-08-28T19:19:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.319+0000 7f24c77fe640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24c00b4c10 0x7f24c00b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.319+0000 7f24d5d9b640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24c00b4c10 0x7f24c00b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.320+0000 7f24c77fe640 1 -- 172.21.15.16:0/486165677 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f24c80e5590 con 0x7f24d8103900 2024-08-28T19:19:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.320+0000 7f24d5d9b640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24c00b4c10 0x7f24c00b7120 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009230 tx=0x7f24cc031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.323+0000 7f24c77fe640 1 -- 172.21.15.16:0/486165677 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f24c80abaf0 con 0x7f24d8103900 2024-08-28T19:19:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:32 smithi115 ceph-mon[22184]: pgmap v292: 129 pgs: 129 active+clean; 645 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-08-28T19:19:33.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.558+0000 7f24dcd5f640 1 -- 172.21.15.16:0/486165677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f24d819b390 con 0x7f24d8103900 2024-08-28T19:19:33.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.564+0000 7f24c77fe640 1 -- 172.21.15.16:0/486165677 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v21) ==== 74+0+293208 (secure 0 0 0) 0x7f24c80ce070 con 0x7f24d8103900 2024-08-28T19:19:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:19:33.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 -- 172.21.15.16:0/486165677 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24c00b4c10 msgr2=0x7f24c00b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:33.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24c00b4c10 0x7f24c00b7120 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009230 tx=0x7f24cc031400 comp rx=0 tx=0).stop 2024-08-28T19:19:33.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 -- 172.21.15.16:0/486165677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d8103900 msgr2=0x7f24d819a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:33.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d8103900 0x7f24d819a260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f24c800d6d0 tx=0x7f24c800db90 comp rx=0 tx=0).stop 2024-08-28T19:19:33.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 -- 172.21.15.16:0/486165677 shutdown_connections 2024-08-28T19:19:33.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24d8109b90 0x7f24d819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:33.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8109240 0x7f24d819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:33.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24c00b4c10 0x7f24c00b7120 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:33.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 --2- 172.21.15.16:0/486165677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d8103900 0x7f24d819a260 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:33.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 -- 172.21.15.16:0/486165677 >> 172.21.15.16:0/486165677 conn(0x7f24d807b5f0 msgr2=0x7f24d8107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:33.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 -- 172.21.15.16:0/486165677 shutdown_connections 2024-08-28T19:19:33.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:33.567+0000 7f24dcd5f640 1 -- 172.21.15.16:0/486165677 wait complete. 2024-08-28T19:19:34.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:33 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/486165677' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-28T19:19:34.308 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"flags":0,"active_gid":14220,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2594037529},{"type":"v1","addr":"172.21.15.16:6801","nonce":2594037529}]},"active_addr":"172.21.15.16:6801/2594037529","active_change":"2024-08-28T19:12:07.368108+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24160,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24170,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3997760726}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2612310139}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2754080649}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3246555201}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1937023952}]}]} 2024-08-28T19:19:34.320 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-08-28T19:19:34.320 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-28T19:19:34.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:19:34.322 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:33 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/486165677' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-28T19:19:34.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:33 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/486165677' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-28T19:19:34.575 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:35.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:34 smithi016 ceph-mon[18556]: pgmap v293: 129 pgs: 129 active+clean; 645 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-08-28T19:19:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.089+0000 7f187c5ad640 1 -- 172.21.15.16:0/4174062337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1874108b40 msgr2=0x7f187410f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.089+0000 7f187c5ad640 1 --2- 172.21.15.16:0/4174062337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1874108b40 0x7f187410f370 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f186c009ca0 tx=0x7f186c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.089+0000 7f187c5ad640 1 -- 172.21.15.16:0/4174062337 shutdown_connections 2024-08-28T19:19:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.089+0000 7f187c5ad640 1 --2- 172.21.15.16:0/4174062337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1874108b40 0x7f187410f370 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.089+0000 7f187c5ad640 1 --2- 172.21.15.16:0/4174062337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18741081f0 0x7f1874108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.089+0000 7f187c5ad640 1 --2- 172.21.15.16:0/4174062337 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18741028b0 0x7f1874102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.089+0000 7f187c5ad640 1 -- 172.21.15.16:0/4174062337 >> 172.21.15.16:0/4174062337 conn(0x7f1874079470 msgr2=0x7f1874079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.089+0000 7f187c5ad640 1 -- 172.21.15.16:0/4174062337 shutdown_connections 2024-08-28T19:19:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.089+0000 7f187c5ad640 1 -- 172.21.15.16:0/4174062337 wait complete. 2024-08-28T19:19:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.090+0000 7f187c5ad640 1 Processor -- start 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.090+0000 7f187c5ad640 1 -- start start 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187c5ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18741028b0 0x7f187419c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187c5ad640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18741081f0 0x7f187419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187c5ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1874108b40 0x7f18741a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187c5ad640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1874113920 con 0x7f1874108b40 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187c5ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18741137a0 con 0x7f18741028b0 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187c5ad640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1874113aa0 con 0x7f18741081f0 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187ab4a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1874108b40 0x7f18741a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187ab4a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1874108b40 0x7f18741a1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56844/0 (socket says 172.21.15.16:56844) 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187ab4a640 1 -- 172.21.15.16:0/1978590947 learned_addr learned my addr 172.21.15.16:0/1978590947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f1879b48640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18741081f0 0x7f187419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187a349640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18741028b0 0x7f187419c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187ab4a640 1 -- 172.21.15.16:0/1978590947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18741081f0 msgr2=0x7f187419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187ab4a640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18741081f0 0x7f187419ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187ab4a640 1 -- 172.21.15.16:0/1978590947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18741028b0 msgr2=0x7f187419c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187ab4a640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18741028b0 0x7f187419c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187ab4a640 1 -- 172.21.15.16:0/1978590947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18741a2420 con 0x7f1874108b40 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187a349640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18741028b0 0x7f187419c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187ab4a640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1874108b40 0x7f18741a1cf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f186c0098d0 tx=0x7f186c0317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f186b7fe640 1 -- 172.21.15.16:0/1978590947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c003b70 con 0x7f1874108b40 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f186b7fe640 1 -- 172.21.15.16:0/1978590947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f186c003d10 con 0x7f1874108b40 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187c5ad640 1 -- 172.21.15.16:0/1978590947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18741a26b0 con 0x7f1874108b40 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f186b7fe640 1 -- 172.21.15.16:0/1978590947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c02fa20 con 0x7f1874108b40 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.091+0000 7f187c5ad640 1 -- 172.21.15.16:0/1978590947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18741a8e70 con 0x7f1874108b40 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.093+0000 7f187c5ad640 1 -- 172.21.15.16:0/1978590947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1874103b30 con 0x7f1874108b40 2024-08-28T19:19:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.094+0000 7f186b7fe640 1 -- 172.21.15.16:0/1978590947 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f186c02fbc0 con 0x7f1874108b40 2024-08-28T19:19:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.095+0000 7f186b7fe640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18600b4c10 0x7f18600b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.095+0000 7f187a349640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18600b4c10 0x7f18600b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.095+0000 7f186b7fe640 1 -- 172.21.15.16:0/1978590947 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f186c106cd0 con 0x7f1874108b40 2024-08-28T19:19:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.095+0000 7f187a349640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18600b4c10 0x7f18600b7120 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1870009c20 tx=0x7f1870005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.100+0000 7f186b7fe640 1 -- 172.21.15.16:0/1978590947 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f186c0cd820 con 0x7f1874108b40 2024-08-28T19:19:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.289+0000 7f187c5ad640 1 -- 172.21.15.16:0/1978590947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18741a9040 con 0x7f1874108b40 2024-08-28T19:19:35.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.290+0000 7f186b7fe640 1 -- 172.21.15.16:0/1978590947 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+24992 (secure 0 0 0) 0x7f186c0cd120 con 0x7f1874108b40 2024-08-28T19:19:35.292 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:19:35.292 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:19:08.602647+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:19:35.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.292+0000 7f187c5ad640 1 -- 172.21.15.16:0/1978590947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18600b4c10 msgr2=0x7f18600b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:35.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.292+0000 7f187c5ad640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18600b4c10 0x7f18600b7120 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1870009c20 tx=0x7f1870005f90 comp rx=0 tx=0).stop 2024-08-28T19:19:35.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.292+0000 7f187c5ad640 1 -- 172.21.15.16:0/1978590947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1874108b40 msgr2=0x7f18741a1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:35.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.292+0000 7f187c5ad640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1874108b40 0x7f18741a1cf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f186c0098d0 tx=0x7f186c0317b0 comp rx=0 tx=0).stop 2024-08-28T19:19:35.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.293+0000 7f187c5ad640 1 -- 172.21.15.16:0/1978590947 shutdown_connections 2024-08-28T19:19:35.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.293+0000 7f187c5ad640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1874108b40 0x7f18741a1cf0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:35.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.293+0000 7f187c5ad640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18741081f0 0x7f187419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:35.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.293+0000 7f187c5ad640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18600b4c10 0x7f18600b7120 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:35.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.293+0000 7f187c5ad640 1 --2- 172.21.15.16:0/1978590947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18741028b0 0x7f187419c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:35.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.293+0000 7f187c5ad640 1 -- 172.21.15.16:0/1978590947 >> 172.21.15.16:0/1978590947 conn(0x7f1874079470 msgr2=0x7f1874103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:35.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.293+0000 7f187c5ad640 1 -- 172.21.15.16:0/1978590947 shutdown_connections 2024-08-28T19:19:35.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:35.293+0000 7f187c5ad640 1 -- 172.21.15.16:0/1978590947 wait complete. 2024-08-28T19:19:35.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:34 smithi028 ceph-mon[21769]: pgmap v293: 129 pgs: 129 active+clean; 645 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-08-28T19:19:35.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:34 smithi115 ceph-mon[22184]: pgmap v293: 129 pgs: 129 active+clean; 645 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-08-28T19:19:35.926 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-28T19:19:35.926 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:19:36.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:35 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1978590947' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:36.175 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:36.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:35 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1978590947' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:36.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:35 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1978590947' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:36.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.657+0000 7f5116101640 1 -- 172.21.15.16:0/1337394152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5110108b40 msgr2=0x7f511010f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:36.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.657+0000 7f5116101640 1 --2- 172.21.15.16:0/1337394152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5110108b40 0x7f511010f370 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5108009ca0 tx=0x7f510802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:36.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.657+0000 7f5116101640 1 -- 172.21.15.16:0/1337394152 shutdown_connections 2024-08-28T19:19:36.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.657+0000 7f5116101640 1 --2- 172.21.15.16:0/1337394152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5110108b40 0x7f511010f370 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.657+0000 7f5116101640 1 --2- 172.21.15.16:0/1337394152 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51101081f0 0x7f5110108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.657+0000 7f5116101640 1 --2- 172.21.15.16:0/1337394152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51101028b0 0x7f5110102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.658+0000 7f5116101640 1 -- 172.21.15.16:0/1337394152 >> 172.21.15.16:0/1337394152 conn(0x7f5110079470 msgr2=0x7f5110079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.658+0000 7f5116101640 1 -- 172.21.15.16:0/1337394152 shutdown_connections 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.658+0000 7f5116101640 1 -- 172.21.15.16:0/1337394152 wait complete. 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.658+0000 7f5116101640 1 Processor -- start 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f5116101640 1 -- start start 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f5116101640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51101028b0 0x7f511019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f5116101640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51101081f0 0x7f511019a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f5116101640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5110108b40 0x7f511019fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f5116101640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51101138f0 con 0x7f51101028b0 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f5116101640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5110113770 con 0x7f51101081f0 2024-08-28T19:19:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f5116101640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5110113a70 con 0x7f5110108b40 2024-08-28T19:19:36.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f510f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51101028b0 0x7f511019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:36.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f510f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51101028b0 0x7f511019a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56866/0 (socket says 172.21.15.16:56866) 2024-08-28T19:19:36.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f510f7fe640 1 -- 172.21.15.16:0/3987656542 learned_addr learned my addr 172.21.15.16:0/3987656542 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:36.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f510effd640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51101081f0 0x7f511019a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:36.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f510ffff640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5110108b40 0x7f511019fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:36.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f510f7fe640 1 -- 172.21.15.16:0/3987656542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5110108b40 msgr2=0x7f511019fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:36.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f510f7fe640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5110108b40 0x7f511019fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:36.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f510f7fe640 1 -- 172.21.15.16:0/3987656542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51101081f0 msgr2=0x7f511019a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:36.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.659+0000 7f510f7fe640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51101081f0 0x7f511019a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:36.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.660+0000 7f510f7fe640 1 -- 172.21.15.16:0/3987656542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51101a0130 con 0x7f51101028b0 2024-08-28T19:19:36.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.660+0000 7f510f7fe640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51101028b0 0x7f511019a280 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f510000d950 tx=0x7f510000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:36.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.660+0000 7f510cff9640 1 -- 172.21.15.16:0/3987656542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5100017070 con 0x7f51101028b0 2024-08-28T19:19:36.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.660+0000 7f5116101640 1 -- 172.21.15.16:0/3987656542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51101a0420 con 0x7f51101028b0 2024-08-28T19:19:36.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.660+0000 7f510cff9640 1 -- 172.21.15.16:0/3987656542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f510000de80 con 0x7f51101028b0 2024-08-28T19:19:36.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.660+0000 7f510cff9640 1 -- 172.21.15.16:0/3987656542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510000f620 con 0x7f51101028b0 2024-08-28T19:19:36.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.660+0000 7f5116101640 1 -- 172.21.15.16:0/3987656542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51101a6c40 con 0x7f51101028b0 2024-08-28T19:19:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.661+0000 7f5116101640 1 -- 172.21.15.16:0/3987656542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5110102cc0 con 0x7f51101028b0 2024-08-28T19:19:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.663+0000 7f510cff9640 1 -- 172.21.15.16:0/3987656542 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f5100009c30 con 0x7f51101028b0 2024-08-28T19:19:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.664+0000 7f510cff9640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50f80b4c40 0x7f50f80b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.664+0000 7f510effd640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50f80b4c40 0x7f50f80b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.664+0000 7f510cff9640 1 -- 172.21.15.16:0/3987656542 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f51000e52e0 con 0x7f51101028b0 2024-08-28T19:19:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.664+0000 7f510effd640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50f80b4c40 0x7f50f80b7150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5110105da0 tx=0x7f510400a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.667+0000 7f510cff9640 1 -- 172.21.15.16:0/3987656542 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f51000abe90 con 0x7f51101028b0 2024-08-28T19:19:36.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.852+0000 7f5116101640 1 -- 172.21.15.16:0/3987656542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f511010a960 con 0x7f51101028b0 2024-08-28T19:19:36.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.854+0000 7f510cff9640 1 -- 172.21.15.16:0/3987656542 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+24992 (secure 0 0 0) 0x7f51000ab790 con 0x7f51101028b0 2024-08-28T19:19:36.855 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:19:36.856 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:19:08.602647+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.856+0000 7f5116101640 1 -- 172.21.15.16:0/3987656542 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50f80b4c40 msgr2=0x7f50f80b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.856+0000 7f5116101640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50f80b4c40 0x7f50f80b7150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5110105da0 tx=0x7f510400a2d0 comp rx=0 tx=0).stop 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.856+0000 7f5116101640 1 -- 172.21.15.16:0/3987656542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51101028b0 msgr2=0x7f511019a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.856+0000 7f5116101640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51101028b0 0x7f511019a280 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f510000d950 tx=0x7f510000dd50 comp rx=0 tx=0).stop 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.856+0000 7f5116101640 1 -- 172.21.15.16:0/3987656542 shutdown_connections 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.856+0000 7f5116101640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5110108b40 0x7f511019fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.856+0000 7f5116101640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50f80b4c40 0x7f50f80b7150 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.856+0000 7f5116101640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51101081f0 0x7f511019a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.856+0000 7f5116101640 1 --2- 172.21.15.16:0/3987656542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51101028b0 0x7f511019a280 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.856+0000 7f5116101640 1 -- 172.21.15.16:0/3987656542 >> 172.21.15.16:0/3987656542 conn(0x7f5110079470 msgr2=0x7f5110104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.856+0000 7f5116101640 1 -- 172.21.15.16:0/3987656542 shutdown_connections 2024-08-28T19:19:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:36.857+0000 7f5116101640 1 -- 172.21.15.16:0/3987656542 wait complete. 2024-08-28T19:19:37.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:36 smithi016 ceph-mon[18556]: pgmap v294: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-08-28T19:19:37.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:36 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3987656542' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:37.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:36 smithi028 ceph-mon[21769]: pgmap v294: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-08-28T19:19:37.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:36 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3987656542' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:37.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:36 smithi115 ceph-mon[22184]: pgmap v294: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-08-28T19:19:37.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:36 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3987656542' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:19:37.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-28T19:19:37.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-28T19:19:37.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-28T19:19:37.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-28T19:19:37.547 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-28T19:19:37.547 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-28T19:19:37.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-28T19:19:37.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-28T19:19:37.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-08-28T19:19:37.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-08-28T19:19:37.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-08-28T19:19:37.550 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-08-28T19:19:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.730+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2595517187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc8102f20 msgr2=0x7f1fc8103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.730+0000 7f1fcfb02640 1 --2- 172.21.15.16:0/2595517187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc8102f20 0x7f1fc8103330 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009900 tx=0x7f1fbc02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.730+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2595517187 shutdown_connections 2024-08-28T19:19:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.730+0000 7f1fcfb02640 1 --2- 172.21.15.16:0/2595517187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fc8108ae0 0x7f1fc810f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.730+0000 7f1fcfb02640 1 --2- 172.21.15.16:0/2595517187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1fc8108190 0x7f1fc81085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.730+0000 7f1fcfb02640 1 --2- 172.21.15.16:0/2595517187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc8102f20 0x7f1fc8103330 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.730+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2595517187 >> 172.21.15.16:0/2595517187 conn(0x7f1fc8079bb0 msgr2=0x7f1fc8079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.730+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2595517187 shutdown_connections 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.730+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2595517187 wait complete. 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.731+0000 7f1fcfb02640 1 Processor -- start 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.731+0000 7f1fcfb02640 1 -- start start 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.731+0000 7f1fcfb02640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1fc8102f20 0x7f1fc8199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.731+0000 7f1fcfb02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fc8108190 0x7f1fc8199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.731+0000 7f1fcfb02640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc8108ae0 0x7f1fc819e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.731+0000 7f1fcfb02640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc8112790 con 0x7f1fc8108ae0 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.731+0000 7f1fcfb02640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc8112610 con 0x7f1fc8108190 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.731+0000 7f1fcfb02640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc8112910 con 0x7f1fc8102f20 2024-08-28T19:19:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fce09f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc8108ae0 0x7f1fc819e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fce09f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc8108ae0 0x7f1fc819e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56876/0 (socket says 172.21.15.16:56876) 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fcd09d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fc8108190 0x7f1fc8199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fce09f640 1 -- 172.21.15.16:0/2172827123 learned_addr learned my addr 172.21.15.16:0/2172827123 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fcd89e640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1fc8102f20 0x7f1fc8199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fce09f640 1 -- 172.21.15.16:0/2172827123 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1fc8102f20 msgr2=0x7f1fc8199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fce09f640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1fc8102f20 0x7f1fc8199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fce09f640 1 -- 172.21.15.16:0/2172827123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fc8108190 msgr2=0x7f1fc8199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fce09f640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fc8108190 0x7f1fc8199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fce09f640 1 -- 172.21.15.16:0/2172827123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc819f030 con 0x7f1fc8108ae0 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fcd89e640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1fc8102f20 0x7f1fc8199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fce09f640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc8108ae0 0x7f1fc819e9c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0007cb0 tx=0x7f1fc000e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fbaffd640 1 -- 172.21.15.16:0/2172827123 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc000eb30 con 0x7f1fc8108ae0 2024-08-28T19:19:38.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fbaffd640 1 -- 172.21.15.16:0/2172827123 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc000ecd0 con 0x7f1fc8108ae0 2024-08-28T19:19:38.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2172827123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc819f320 con 0x7f1fc8108ae0 2024-08-28T19:19:38.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fbaffd640 1 -- 172.21.15.16:0/2172827123 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00153f0 con 0x7f1fc8108ae0 2024-08-28T19:19:38.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.732+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2172827123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc8101b80 con 0x7f1fc8108ae0 2024-08-28T19:19:38.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.733+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2172827123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1fc81020f0 con 0x7f1fc8108ae0 2024-08-28T19:19:38.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.736+0000 7f1fbaffd640 1 -- 172.21.15.16:0/2172827123 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f1fc000a440 con 0x7f1fc8108ae0 2024-08-28T19:19:38.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.736+0000 7f1fbaffd640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1fb40b4c40 0x7f1fb40b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:38.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.737+0000 7f1fcd89e640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1fb40b4c40 0x7f1fb40b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:38.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.737+0000 7f1fbaffd640 1 -- 172.21.15.16:0/2172827123 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f1fc00e7d20 con 0x7f1fc8108ae0 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.737+0000 7f1fbaffd640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969] conn(0x7f1fb40bff50 0x7f1fb40c2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.737+0000 7f1fbaffd640 1 -- 172.21.15.16:0/2172827123 --> [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fc0018ec0 con 0x7f1fb40bff50 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.737+0000 7f1fbaffd640 1 -- 172.21.15.16:0/2172827123 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f1fc00e8170 con 0x7f1fc8108ae0 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.737+0000 7f1fcd89e640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1fb40b4c40 0x7f1fb40b7150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc031440 tx=0x7f1fbc009680 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.737+0000 7f1fcd09d640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969] conn(0x7f1fb40bff50 0x7f1fb40c2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.738+0000 7f1fcd09d640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969] conn(0x7f1fb40bff50 0x7f1fb40c2440 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.738+0000 7f1fbaffd640 1 -- 172.21.15.16:0/2172827123 <== osd.4 v2:172.21.15.28:6808/302267969 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1fc0018ec0 con 0x7f1fb40bff50 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.752+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2172827123 --> [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1fc81024f0 con 0x7f1fb40bff50 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.753+0000 7f1fbaffd640 1 -- 172.21.15.16:0/2172827123 <== osd.4 v2:172.21.15.28:6808/302267969 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1fc81024f0 con 0x7f1fb40bff50 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.753+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2172827123 >> [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969] conn(0x7f1fb40bff50 msgr2=0x7f1fb40c2440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.753+0000 7f1fcfb02640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969] conn(0x7f1fb40bff50 0x7f1fb40c2440 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.753+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2172827123 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1fb40b4c40 msgr2=0x7f1fb40b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.753+0000 7f1fcfb02640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1fb40b4c40 0x7f1fb40b7150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc031440 tx=0x7f1fbc009680 comp rx=0 tx=0).stop 2024-08-28T19:19:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.753+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2172827123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc8108ae0 msgr2=0x7f1fc819e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.753+0000 7f1fcfb02640 1 --2- 172.21.15.16:0/2172827123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc8108ae0 0x7f1fc819e9c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0007cb0 tx=0x7f1fc000e420 comp rx=0 tx=0).stop 2024-08-28T19:19:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.753+0000 7f1fce09f640 1 -- 172.21.15.16:0/2172827123 reap_dead start 2024-08-28T19:19:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.754+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2172827123 shutdown_connections 2024-08-28T19:19:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.754+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2172827123 >> 172.21.15.16:0/2172827123 conn(0x7f1fc8079bb0 msgr2=0x7f1fc8106280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.754+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2172827123 shutdown_connections 2024-08-28T19:19:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:38.754+0000 7f1fcfb02640 1 -- 172.21.15.16:0/2172827123 wait complete. 2024-08-28T19:19:39.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:38 smithi016 ceph-mon[18556]: pgmap v295: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 10 op/s 2024-08-28T19:19:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:38 smithi028 ceph-mon[21769]: pgmap v295: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 10 op/s 2024-08-28T19:19:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:38 smithi115 ceph-mon[22184]: pgmap v295: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 10 op/s 2024-08-28T19:19:39.452 INFO:teuthology.orchestra.run.smithi016.stdout:124554051646 2024-08-28T19:19:39.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-28T19:19:39.463 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:39.471 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:39.482 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:39.815 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:39.816 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:40.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.370+0000 7f09bd0f2640 1 -- 172.21.15.16:0/4144846525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b81078d0 msgr2=0x7f09b8107ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:40.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.370+0000 7f09bd0f2640 1 --2- 172.21.15.16:0/4144846525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b81078d0 0x7f09b8107ce0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f09ac009900 tx=0x7f09ac02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:40.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.370+0000 7f09bd0f2640 1 -- 172.21.15.16:0/4144846525 shutdown_connections 2024-08-28T19:19:40.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.370+0000 7f09bd0f2640 1 --2- 172.21.15.16:0/4144846525 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09b8108220 0x7f09b810ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:40.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.370+0000 7f09bd0f2640 1 --2- 172.21.15.16:0/4144846525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b81078d0 0x7f09b8107ce0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:40.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.370+0000 7f09bd0f2640 1 --2- 172.21.15.16:0/4144846525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09b8101f90 0x7f09b81023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:40.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.370+0000 7f09bd0f2640 1 -- 172.21.15.16:0/4144846525 >> 172.21.15.16:0/4144846525 conn(0x7f09b8077a00 msgr2=0x7f09b8077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:40.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.370+0000 7f09bd0f2640 1 -- 172.21.15.16:0/4144846525 shutdown_connections 2024-08-28T19:19:40.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.370+0000 7f09bd0f2640 1 -- 172.21.15.16:0/4144846525 wait complete. 2024-08-28T19:19:40.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.371+0000 7f09bd0f2640 1 Processor -- start 2024-08-28T19:19:40.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.371+0000 7f09bd0f2640 1 -- start start 2024-08-28T19:19:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.371+0000 7f09bd0f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09b8101f90 0x7f09b8199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.371+0000 7f09bd0f2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09b81078d0 0x7f09b8199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.371+0000 7f09bd0f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b8108220 0x7f09b819e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.371+0000 7f09bd0f2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09b8111ee0 con 0x7f09b8108220 2024-08-28T19:19:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.371+0000 7f09bd0f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09b8111d60 con 0x7f09b8101f90 2024-08-28T19:19:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.371+0000 7f09bd0f2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f09b8112060 con 0x7f09b81078d0 2024-08-28T19:19:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b8108220 0x7f09b819e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b8108220 0x7f09b819e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56904/0 (socket says 172.21.15.16:56904) 2024-08-28T19:19:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b759e640 1 -- 172.21.15.16:0/1465338892 learned_addr learned my addr 172.21.15.16:0/1465338892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b6d9d640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09b8101f90 0x7f09b8199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b659c640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09b81078d0 0x7f09b8199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:40.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b759e640 1 -- 172.21.15.16:0/1465338892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09b81078d0 msgr2=0x7f09b8199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:40.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b759e640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09b81078d0 0x7f09b8199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:40.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b759e640 1 -- 172.21.15.16:0/1465338892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09b8101f90 msgr2=0x7f09b8199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:40.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b759e640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09b8101f90 0x7f09b8199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:40.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b759e640 1 -- 172.21.15.16:0/1465338892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b819f0f0 con 0x7f09b8108220 2024-08-28T19:19:40.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b6d9d640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09b8101f90 0x7f09b8199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:19:40.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09b759e640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b8108220 0x7f09b819e9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f09b000eaa0 tx=0x7f09b000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:40.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f099ffff640 1 -- 172.21.15.16:0/1465338892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b000cdb0 con 0x7f09b8108220 2024-08-28T19:19:40.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09bd0f2640 1 -- 172.21.15.16:0/1465338892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09b819f3e0 con 0x7f09b8108220 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f099ffff640 1 -- 172.21.15.16:0/1465338892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09b0005020 con 0x7f09b8108220 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f099ffff640 1 -- 172.21.15.16:0/1465338892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b00116d0 con 0x7f09b8108220 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.372+0000 7f09bd0f2640 1 -- 172.21.15.16:0/1465338892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09b81a5b40 con 0x7f09b8108220 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.373+0000 7f09bd0f2640 1 -- 172.21.15.16:0/1465338892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f09b81014d0 con 0x7f09b8108220 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.375+0000 7f099ffff640 1 -- 172.21.15.16:0/1465338892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f09b00051d0 con 0x7f09b8108220 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.376+0000 7f099ffff640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09a00b4c10 0x7f09a00b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.376+0000 7f09b6d9d640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09a00b4c10 0x7f09a00b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.377+0000 7f099ffff640 1 -- 172.21.15.16:0/1465338892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f09b00e5d80 con 0x7f09b8108220 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.377+0000 7f099ffff640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648] conn(0x7f09a00bff20 0x7f09a00c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.377+0000 7f099ffff640 1 -- 172.21.15.16:0/1465338892 --> [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09a00c2ae0 con 0x7f09a00bff20 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.377+0000 7f099ffff640 1 -- 172.21.15.16:0/1465338892 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f09b00e61d0 con 0x7f09b8108220 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.377+0000 7f09b6d9d640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09a00b4c10 0x7f09a00b7120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009fc0 tx=0x7f09a8005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:40.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.377+0000 7f09b659c640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648] conn(0x7f09a00bff20 0x7f09a00c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.378+0000 7f09b659c640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648] conn(0x7f09a00bff20 0x7f09a00c2410 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.379+0000 7f099ffff640 1 -- 172.21.15.16:0/1465338892 <== osd.2 v2:172.21.15.115:6800/1599247648 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f09a00c2ae0 con 0x7f09a00bff20 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.392+0000 7f09bd0f2640 1 -- 172.21.15.16:0/1465338892 --> [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f09b8103210 con 0x7f09a00bff20 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.393+0000 7f099ffff640 1 -- 172.21.15.16:0/1465338892 <== osd.2 v2:172.21.15.115:6800/1599247648 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f09b8103210 con 0x7f09a00bff20 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.393+0000 7f09bd0f2640 1 -- 172.21.15.16:0/1465338892 >> [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648] conn(0x7f09a00bff20 msgr2=0x7f09a00c2410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.393+0000 7f09bd0f2640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648] conn(0x7f09a00bff20 0x7f09a00c2410 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.393+0000 7f09bd0f2640 1 -- 172.21.15.16:0/1465338892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09a00b4c10 msgr2=0x7f09a00b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.393+0000 7f09bd0f2640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09a00b4c10 0x7f09a00b7120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009fc0 tx=0x7f09a8005f90 comp rx=0 tx=0).stop 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.393+0000 7f09bd0f2640 1 -- 172.21.15.16:0/1465338892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b8108220 msgr2=0x7f09b819e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.393+0000 7f09bd0f2640 1 --2- 172.21.15.16:0/1465338892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b8108220 0x7f09b819e9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f09b000eaa0 tx=0x7f09b000ef60 comp rx=0 tx=0).stop 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.393+0000 7f09b759e640 1 -- 172.21.15.16:0/1465338892 reap_dead start 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.393+0000 7f09bd0f2640 1 -- 172.21.15.16:0/1465338892 shutdown_connections 2024-08-28T19:19:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.393+0000 7f09bd0f2640 1 -- 172.21.15.16:0/1465338892 >> 172.21.15.16:0/1465338892 conn(0x7f09b8077a00 msgr2=0x7f09b8100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:40.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.394+0000 7f09bd0f2640 1 -- 172.21.15.16:0/1465338892 shutdown_connections 2024-08-28T19:19:40.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:40.394+0000 7f09bd0f2640 1 -- 172.21.15.16:0/1465338892 wait complete. 2024-08-28T19:19:41.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:40 smithi016 ceph-mon[18556]: pgmap v296: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 9 op/s 2024-08-28T19:19:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:40 smithi028 ceph-mon[21769]: pgmap v296: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 9 op/s 2024-08-28T19:19:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.342+0000 7f90f0cfa640 1 -- 172.21.15.16:0/519687353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ec103900 msgr2=0x7f90ec103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.342+0000 7f90f0cfa640 1 --2- 172.21.15.16:0/519687353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ec103900 0x7f90ec103d10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f90dc009900 tx=0x7f90dc02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.343+0000 7f90f0cfa640 1 -- 172.21.15.16:0/519687353 shutdown_connections 2024-08-28T19:19:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.343+0000 7f90f0cfa640 1 --2- 172.21.15.16:0/519687353 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f90ec109b90 0x7f90ec1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.343+0000 7f90f0cfa640 1 --2- 172.21.15.16:0/519687353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90ec109240 0x7f90ec109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.343+0000 7f90f0cfa640 1 --2- 172.21.15.16:0/519687353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ec103900 0x7f90ec103d10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.343+0000 7f90f0cfa640 1 -- 172.21.15.16:0/519687353 >> 172.21.15.16:0/519687353 conn(0x7f90ec07b5f0 msgr2=0x7f90ec07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.343+0000 7f90f0cfa640 1 -- 172.21.15.16:0/519687353 shutdown_connections 2024-08-28T19:19:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.343+0000 7f90f0cfa640 1 -- 172.21.15.16:0/519687353 wait complete. 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.343+0000 7f90f0cfa640 1 Processor -- start 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.343+0000 7f90f0cfa640 1 -- start start 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90f0cfa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ec103900 0x7f90ec19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90f0cfa640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f90ec109240 0x7f90ec19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90f0cfa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90ec109b90 0x7f90ec19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90f0cfa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90ec1138b0 con 0x7f90ec103900 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90f0cfa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f90ec113730 con 0x7f90ec109b90 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90f0cfa640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f90ec113a30 con 0x7f90ec109240 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90ea59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ec103900 0x7f90ec19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90ea59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ec103900 0x7f90ec19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56926/0 (socket says 172.21.15.16:56926) 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90ea59c640 1 -- 172.21.15.16:0/104461434 learned_addr learned my addr 172.21.15.16:0/104461434 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90e9d9b640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f90ec109240 0x7f90ec19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90ead9d640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90ec109b90 0x7f90ec19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90ea59c640 1 -- 172.21.15.16:0/104461434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f90ec109240 msgr2=0x7f90ec19a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90ea59c640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f90ec109240 0x7f90ec19a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90ea59c640 1 -- 172.21.15.16:0/104461434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90ec109b90 msgr2=0x7f90ec19faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90ea59c640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90ec109b90 0x7f90ec19faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90ea59c640 1 -- 172.21.15.16:0/104461434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90ec1a01d0 con 0x7f90ec103900 2024-08-28T19:19:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.344+0000 7f90ead9d640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90ec109b90 0x7f90ec19faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.345+0000 7f90ea59c640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ec103900 0x7f90ec19a260 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f90dc005940 tx=0x7f90dc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.345+0000 7f90db7fe640 1 -- 172.21.15.16:0/104461434 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc03f030 con 0x7f90ec103900 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.345+0000 7f90f0cfa640 1 -- 172.21.15.16:0/104461434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90ec1a0460 con 0x7f90ec103900 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.345+0000 7f90f0cfa640 1 -- 172.21.15.16:0/104461434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90ec1a6c20 con 0x7f90ec103900 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.345+0000 7f90db7fe640 1 -- 172.21.15.16:0/104461434 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90dc003cb0 con 0x7f90ec103900 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.345+0000 7f90db7fe640 1 -- 172.21.15.16:0/104461434 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc035070 con 0x7f90ec103900 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.346+0000 7f90f0cfa640 1 -- 172.21.15.16:0/104461434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f90ec0638b0 con 0x7f90ec103900 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.348+0000 7f90db7fe640 1 -- 172.21.15.16:0/104461434 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f90dc042030 con 0x7f90ec103900 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.348+0000 7f90db7fe640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f90d40b4c10 0x7f90d40b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.348+0000 7f90e9d9b640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f90d40b4c10 0x7f90d40b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.348+0000 7f90db7fe640 1 -- 172.21.15.16:0/104461434 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f90dc1062f0 con 0x7f90ec103900 2024-08-28T19:19:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.349+0000 7f90db7fe640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235] conn(0x7f90d40bff20 0x7f90d40c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.349+0000 7f90db7fe640 1 -- 172.21.15.16:0/104461434 --> [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90d40c2ae0 con 0x7f90d40bff20 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.349+0000 7f90db7fe640 1 -- 172.21.15.16:0/104461434 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f90dc106740 con 0x7f90ec103900 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.349+0000 7f90e9d9b640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f90d40b4c10 0x7f90d40b7120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f90ec108ff0 tx=0x7f90e000b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.349+0000 7f90ead9d640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235] conn(0x7f90d40bff20 0x7f90d40c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.350+0000 7f90ead9d640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235] conn(0x7f90d40bff20 0x7f90d40c2410 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.350+0000 7f90db7fe640 1 -- 172.21.15.16:0/104461434 <== osd.1 v2:172.21.15.28:6800/666617235 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f90d40c2ae0 con 0x7f90d40bff20 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.365+0000 7f90f0cfa640 1 -- 172.21.15.16:0/104461434 --> [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f90ec063cb0 con 0x7f90d40bff20 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.365+0000 7f90db7fe640 1 -- 172.21.15.16:0/104461434 <== osd.1 v2:172.21.15.28:6800/666617235 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f90ec063cb0 con 0x7f90d40bff20 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.365+0000 7f90f0cfa640 1 -- 172.21.15.16:0/104461434 >> [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235] conn(0x7f90d40bff20 msgr2=0x7f90d40c2410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.366+0000 7f90f0cfa640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235] conn(0x7f90d40bff20 0x7f90d40c2410 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.366+0000 7f90f0cfa640 1 -- 172.21.15.16:0/104461434 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f90d40b4c10 msgr2=0x7f90d40b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.366+0000 7f90f0cfa640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f90d40b4c10 0x7f90d40b7120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f90ec108ff0 tx=0x7f90e000b390 comp rx=0 tx=0).stop 2024-08-28T19:19:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.366+0000 7f90f0cfa640 1 -- 172.21.15.16:0/104461434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ec103900 msgr2=0x7f90ec19a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.366+0000 7f90f0cfa640 1 --2- 172.21.15.16:0/104461434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ec103900 0x7f90ec19a260 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f90dc005940 tx=0x7f90dc0042f0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.366+0000 7f90ead9d640 1 -- 172.21.15.16:0/104461434 reap_dead start 2024-08-28T19:19:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.366+0000 7f90f0cfa640 1 -- 172.21.15.16:0/104461434 shutdown_connections 2024-08-28T19:19:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.366+0000 7f90f0cfa640 1 -- 172.21.15.16:0/104461434 >> 172.21.15.16:0/104461434 conn(0x7f90ec07b5f0 msgr2=0x7f90ec1072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.366+0000 7f90f0cfa640 1 -- 172.21.15.16:0/104461434 shutdown_connections 2024-08-28T19:19:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.366+0000 7f90f0cfa640 1 -- 172.21.15.16:0/104461434 wait complete. 2024-08-28T19:19:41.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:40 smithi115 ceph-mon[22184]: pgmap v296: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 9 op/s 2024-08-28T19:19:41.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.983+0000 7fd7b76d7640 1 -- 172.21.15.16:0/4008277575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0107920 msgr2=0x7fd7b0107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:41.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.983+0000 7fd7b76d7640 1 --2- 172.21.15.16:0/4008277575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0107920 0x7fd7b0107d30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0009900 tx=0x7fd7a002a410 comp rx=0 tx=0).stop 2024-08-28T19:19:41.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.983+0000 7fd7b76d7640 1 -- 172.21.15.16:0/4008277575 shutdown_connections 2024-08-28T19:19:41.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.983+0000 7fd7b76d7640 1 --2- 172.21.15.16:0/4008277575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7b0108270 0x7fd7b010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.983+0000 7fd7b76d7640 1 --2- 172.21.15.16:0/4008277575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0107920 0x7fd7b0107d30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.983+0000 7fd7b76d7640 1 --2- 172.21.15.16:0/4008277575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0101fe0 0x7fd7b01023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.983+0000 7fd7b76d7640 1 -- 172.21.15.16:0/4008277575 >> 172.21.15.16:0/4008277575 conn(0x7fd7b0079d30 msgr2=0x7fd7b007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:41.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.983+0000 7fd7b76d7640 1 -- 172.21.15.16:0/4008277575 shutdown_connections 2024-08-28T19:19:41.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.983+0000 7fd7b76d7640 1 -- 172.21.15.16:0/4008277575 wait complete. 2024-08-28T19:19:41.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b76d7640 1 Processor -- start 2024-08-28T19:19:41.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b76d7640 1 -- start start 2024-08-28T19:19:41.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b76d7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7b0101fe0 0x7fd7b019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:41.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b76d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0107920 0x7fd7b019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:41.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b76d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108270 0x7fd7b019fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:41.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b76d7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b01127b0 con 0x7fd7b0107920 2024-08-28T19:19:41.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b76d7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b0112630 con 0x7fd7b0108270 2024-08-28T19:19:41.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b76d7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b0112930 con 0x7fd7b0101fe0 2024-08-28T19:19:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b4c72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0107920 0x7fd7b019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b4c72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0107920 0x7fd7b019a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56958/0 (socket says 172.21.15.16:56958) 2024-08-28T19:19:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b4c72640 1 -- 172.21.15.16:0/511499463 learned_addr learned my addr 172.21.15.16:0/511499463 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b5473640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7b0101fe0 0x7fd7b019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b5c74640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108270 0x7fd7b019fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b4c72640 1 -- 172.21.15.16:0/511499463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7b0101fe0 msgr2=0x7fd7b019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b4c72640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7b0101fe0 0x7fd7b019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b4c72640 1 -- 172.21.15.16:0/511499463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108270 msgr2=0x7fd7b019fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b4c72640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108270 0x7fd7b019fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b4c72640 1 -- 172.21.15.16:0/511499463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b01a0200 con 0x7fd7b0107920 2024-08-28T19:19:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.984+0000 7fd7b5c74640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108270 0x7fd7b019fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.985+0000 7fd7b4c72640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0107920 0x7fd7b019a830 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd7a002a930 tx=0x7fd7a0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.985+0000 7fd7a67fc640 1 -- 172.21.15.16:0/511499463 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a003f030 con 0x7fd7b0107920 2024-08-28T19:19:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.985+0000 7fd7b76d7640 1 -- 172.21.15.16:0/511499463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b01a0490 con 0x7fd7b0107920 2024-08-28T19:19:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.985+0000 7fd7a67fc640 1 -- 172.21.15.16:0/511499463 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7a0003aa0 con 0x7fd7b0107920 2024-08-28T19:19:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.985+0000 7fd7a67fc640 1 -- 172.21.15.16:0/511499463 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0035070 con 0x7fd7b0107920 2024-08-28T19:19:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.985+0000 7fd7b76d7640 1 -- 172.21.15.16:0/511499463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b01a6c50 con 0x7fd7b0107920 2024-08-28T19:19:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.986+0000 7fd7b76d7640 1 -- 172.21.15.16:0/511499463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd7b00638f0 con 0x7fd7b0107920 2024-08-28T19:19:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.987+0000 7fd7a67fc640 1 -- 172.21.15.16:0/511499463 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fd7a0042030 con 0x7fd7b0107920 2024-08-28T19:19:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.988+0000 7fd7a67fc640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd79c0b4c10 0x7fd79c0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.988+0000 7fd7b5473640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd79c0b4c10 0x7fd79c0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.988+0000 7fd7a67fc640 1 -- 172.21.15.16:0/511499463 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fd7a01062f0 con 0x7fd7b0107920 2024-08-28T19:19:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.989+0000 7fd7a67fc640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569] conn(0x7fd79c0bff20 0x7fd79c0c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.989+0000 7fd7a67fc640 1 -- 172.21.15.16:0/511499463 --> [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd79c0c2ae0 con 0x7fd79c0bff20 2024-08-28T19:19:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.989+0000 7fd7a67fc640 1 -- 172.21.15.16:0/511499463 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fd7a01066a0 con 0x7fd7b0107920 2024-08-28T19:19:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.989+0000 7fd7b5c74640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569] conn(0x7fd79c0bff20 0x7fd79c0c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.989+0000 7fd7b5473640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd79c0b4c10 0x7fd79c0b7120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00bbe0 tx=0x7fd7ac00b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.989+0000 7fd7b5c74640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569] conn(0x7fd79c0bff20 0x7fd79c0c2410 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.990+0000 7fd7a67fc640 1 -- 172.21.15.16:0/511499463 <== osd.3 v2:172.21.15.16:6810/3791831569 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd79c0c2ae0 con 0x7fd79c0bff20 2024-08-28T19:19:42.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.004+0000 7fd7b76d7640 1 -- 172.21.15.16:0/511499463 --> [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd7b0063d70 con 0x7fd79c0bff20 2024-08-28T19:19:42.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.004+0000 7fd7a67fc640 1 -- 172.21.15.16:0/511499463 <== osd.3 v2:172.21.15.16:6810/3791831569 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd7b0063d70 con 0x7fd79c0bff20 2024-08-28T19:19:42.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.004+0000 7fd7b76d7640 1 -- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569] conn(0x7fd79c0bff20 msgr2=0x7fd79c0c2410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.004+0000 7fd7b76d7640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569] conn(0x7fd79c0bff20 0x7fd79c0c2410 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.004+0000 7fd7b76d7640 1 -- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd79c0b4c10 msgr2=0x7fd79c0b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.004+0000 7fd7b76d7640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd79c0b4c10 0x7fd79c0b7120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00bbe0 tx=0x7fd7ac00b390 comp rx=0 tx=0).stop 2024-08-28T19:19:42.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.004+0000 7fd7b76d7640 1 -- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0107920 msgr2=0x7fd7b019a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.004+0000 7fd7b76d7640 1 --2- 172.21.15.16:0/511499463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0107920 0x7fd7b019a830 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd7a002a930 tx=0x7fd7a0004270 comp rx=0 tx=0).stop 2024-08-28T19:19:42.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.004+0000 7fd7b5c74640 1 -- 172.21.15.16:0/511499463 reap_dead start 2024-08-28T19:19:42.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.004+0000 7fd7b76d7640 1 -- 172.21.15.16:0/511499463 shutdown_connections 2024-08-28T19:19:42.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.004+0000 7fd7b76d7640 1 -- 172.21.15.16:0/511499463 >> 172.21.15.16:0/511499463 conn(0x7fd7b0079d30 msgr2=0x7fd7b0101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:42.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.005+0000 7fd7b76d7640 1 -- 172.21.15.16:0/511499463 shutdown_connections 2024-08-28T19:19:42.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.005+0000 7fd7b76d7640 1 -- 172.21.15.16:0/511499463 wait complete. 2024-08-28T19:19:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.249+0000 7fb780fa9640 1 -- 172.21.15.16:0/782319091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c109240 msgr2=0x7fb77c109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.249+0000 7fb780fa9640 1 --2- 172.21.15.16:0/782319091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c109240 0x7fb77c109650 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb764009900 tx=0x7fb76402a410 comp rx=0 tx=0).stop 2024-08-28T19:19:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.249+0000 7fb780fa9640 1 -- 172.21.15.16:0/782319091 shutdown_connections 2024-08-28T19:19:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.249+0000 7fb780fa9640 1 --2- 172.21.15.16:0/782319091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb77c109b90 0x7fb77c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.249+0000 7fb780fa9640 1 --2- 172.21.15.16:0/782319091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c109240 0x7fb77c109650 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.249+0000 7fb780fa9640 1 --2- 172.21.15.16:0/782319091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb77c103900 0x7fb77c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.249+0000 7fb780fa9640 1 -- 172.21.15.16:0/782319091 >> 172.21.15.16:0/782319091 conn(0x7fb77c07b5f0 msgr2=0x7fb77c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.249+0000 7fb780fa9640 1 -- 172.21.15.16:0/782319091 shutdown_connections 2024-08-28T19:19:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.249+0000 7fb780fa9640 1 -- 172.21.15.16:0/782319091 wait complete. 2024-08-28T19:19:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.250+0000 7fb780fa9640 1 Processor -- start 2024-08-28T19:19:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.250+0000 7fb780fa9640 1 -- start start 2024-08-28T19:19:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.250+0000 7fb780fa9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c103900 0x7fb77c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.250+0000 7fb780fa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb77c109240 0x7fb77c19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.250+0000 7fb780fa9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb77c109b90 0x7fb77c19fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.250+0000 7fb780fa9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c113850 con 0x7fb77c103900 2024-08-28T19:19:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.250+0000 7fb780fa9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c1136d0 con 0x7fb77c109240 2024-08-28T19:19:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.250+0000 7fb780fa9640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c1139d0 con 0x7fb77c109b90 2024-08-28T19:19:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb77a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c103900 0x7fb77c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb77a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c103900 0x7fb77c19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56968/0 (socket says 172.21.15.16:56968) 2024-08-28T19:19:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb77a59c640 1 -- 172.21.15.16:0/2273702435 learned_addr learned my addr 172.21.15.16:0/2273702435 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb773fff640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb77c109240 0x7fb77c19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb77ad9d640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb77c109b90 0x7fb77c19fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb77a59c640 1 -- 172.21.15.16:0/2273702435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb77c109b90 msgr2=0x7fb77c19fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb77a59c640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb77c109b90 0x7fb77c19fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb77a59c640 1 -- 172.21.15.16:0/2273702435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb77c109240 msgr2=0x7fb77c19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb77a59c640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb77c109240 0x7fb77c19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb77a59c640 1 -- 172.21.15.16:0/2273702435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c1a0130 con 0x7fb77c103900 2024-08-28T19:19:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb77a59c640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c103900 0x7fb77c19a280 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00e9f0 tx=0x7fb76c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb7737fe640 1 -- 172.21.15.16:0/2273702435 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c00cd80 con 0x7fb77c103900 2024-08-28T19:19:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb7737fe640 1 -- 172.21.15.16:0/2273702435 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb76c0054d0 con 0x7fb77c103900 2024-08-28T19:19:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb780fa9640 1 -- 172.21.15.16:0/2273702435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb77c1a0420 con 0x7fb77c103900 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb7737fe640 1 -- 172.21.15.16:0/2273702435 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c010690 con 0x7fb77c103900 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.251+0000 7fb780fa9640 1 -- 172.21.15.16:0/2273702435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb77c1a08e0 con 0x7fb77c103900 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.252+0000 7fb780fa9640 1 -- 172.21.15.16:0/2273702435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb77c0638f0 con 0x7fb77c103900 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.254+0000 7fb7737fe640 1 -- 172.21.15.16:0/2273702435 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fb76c004e30 con 0x7fb77c103900 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.255+0000 7fb7737fe640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb7680b4c40 0x7fb7680b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.255+0000 7fb773fff640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb7680b4c40 0x7fb7680b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.255+0000 7fb7737fe640 1 -- 172.21.15.16:0/2273702435 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fb76c0e5f80 con 0x7fb77c103900 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.255+0000 7fb7737fe640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327] conn(0x7fb7680bff50 0x7fb7680c2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.255+0000 7fb7737fe640 1 -- 172.21.15.16:0/2273702435 --> [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb76c025ea0 con 0x7fb7680bff50 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.255+0000 7fb7737fe640 1 -- 172.21.15.16:0/2273702435 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fb76c0e6330 con 0x7fb77c103900 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.255+0000 7fb77ad9d640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327] conn(0x7fb7680bff50 0x7fb7680c2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.255+0000 7fb773fff640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb7680b4c40 0x7fb7680b7150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb764009230 tx=0x7fb764031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.255+0000 7fb77ad9d640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327] conn(0x7fb7680bff50 0x7fb7680c2440 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.256+0000 7fb7737fe640 1 -- 172.21.15.16:0/2273702435 <== osd.0 v2:172.21.15.16:6802/1371554327 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb76c025ea0 con 0x7fb7680bff50 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.270+0000 7fb780fa9640 1 -- 172.21.15.16:0/2273702435 --> [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb77c063d70 con 0x7fb7680bff50 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb7737fe640 1 -- 172.21.15.16:0/2273702435 <== osd.0 v2:172.21.15.16:6802/1371554327 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb77c063d70 con 0x7fb7680bff50 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb780fa9640 1 -- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327] conn(0x7fb7680bff50 msgr2=0x7fb7680c2440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb780fa9640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327] conn(0x7fb7680bff50 0x7fb7680c2440 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb780fa9640 1 -- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb7680b4c40 msgr2=0x7fb7680b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb780fa9640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb7680b4c40 0x7fb7680b7150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb764009230 tx=0x7fb764031400 comp rx=0 tx=0).stop 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb780fa9640 1 -- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c103900 msgr2=0x7fb77c19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb780fa9640 1 --2- 172.21.15.16:0/2273702435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c103900 0x7fb77c19a280 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00e9f0 tx=0x7fb76c00edf0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb77ad9d640 1 -- 172.21.15.16:0/2273702435 reap_dead start 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb780fa9640 1 -- 172.21.15.16:0/2273702435 shutdown_connections 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb780fa9640 1 -- 172.21.15.16:0/2273702435 >> 172.21.15.16:0/2273702435 conn(0x7fb77c07b5f0 msgr2=0x7fb77c105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb780fa9640 1 -- 172.21.15.16:0/2273702435 shutdown_connections 2024-08-28T19:19:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:42.271+0000 7fb780fa9640 1 -- 172.21.15.16:0/2273702435 wait complete. 2024-08-28T19:19:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.763+0000 7fc57c9eb640 1 -- 172.21.15.16:0/2381424363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc578102840 msgr2=0x7fc578102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.763+0000 7fc57c9eb640 1 --2- 172.21.15.16:0/2381424363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc578102840 0x7fc578102c50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc568009900 tx=0x7fc56802a3f0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.764+0000 7fc57c9eb640 1 -- 172.21.15.16:0/2381424363 shutdown_connections 2024-08-28T19:19:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.764+0000 7fc57c9eb640 1 --2- 172.21.15.16:0/2381424363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc578108ad0 0x7fc57810f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.764+0000 7fc57c9eb640 1 --2- 172.21.15.16:0/2381424363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc578108180 0x7fc578108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.764+0000 7fc57c9eb640 1 --2- 172.21.15.16:0/2381424363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc578102840 0x7fc578102c50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.764+0000 7fc57c9eb640 1 -- 172.21.15.16:0/2381424363 >> 172.21.15.16:0/2381424363 conn(0x7fc5780794f0 msgr2=0x7fc578079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.764+0000 7fc57c9eb640 1 -- 172.21.15.16:0/2381424363 shutdown_connections 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.764+0000 7fc57c9eb640 1 -- 172.21.15.16:0/2381424363 wait complete. 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.764+0000 7fc57c9eb640 1 Processor -- start 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.764+0000 7fc57c9eb640 1 -- start start 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc57c9eb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc578102840 0x7fc57819b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc57c9eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc578108180 0x7fc57819b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc57c9eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc578108ad0 0x7fc57819bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc57c9eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc578112780 con 0x7fc578108ad0 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc57c9eb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc578112600 con 0x7fc578108180 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc57c9eb640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc578112900 con 0x7fc578102840 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc576d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc578108ad0 0x7fc57819bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc576d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc578108ad0 0x7fc57819bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56942/0 (socket says 172.21.15.16:56942) 2024-08-28T19:19:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc576d9d640 1 -- 172.21.15.16:0/3862205347 learned_addr learned my addr 172.21.15.16:0/3862205347 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc57659c640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc578102840 0x7fc57819b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc575d9b640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc578108180 0x7fc57819b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc576d9d640 1 -- 172.21.15.16:0/3862205347 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc578102840 msgr2=0x7fc57819b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc576d9d640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc578102840 0x7fc57819b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc576d9d640 1 -- 172.21.15.16:0/3862205347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc578108180 msgr2=0x7fc57819b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc576d9d640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc578108180 0x7fc57819b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc576d9d640 1 -- 172.21.15.16:0/3862205347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5781a62d0 con 0x7fc578108ad0 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc575d9b640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc578108180 0x7fc57819b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.765+0000 7fc576d9d640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc578108ad0 0x7fc57819bb50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc57000cce0 tx=0x7fc57000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.766+0000 7fc5677fe640 1 -- 172.21.15.16:0/3862205347 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57000f450 con 0x7fc578108ad0 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.766+0000 7fc57c9eb640 1 -- 172.21.15.16:0/3862205347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5781a65c0 con 0x7fc578108ad0 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.766+0000 7fc5677fe640 1 -- 172.21.15.16:0/3862205347 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc57000faa0 con 0x7fc578108ad0 2024-08-28T19:19:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.766+0000 7fc5677fe640 1 -- 172.21.15.16:0/3862205347 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570018b70 con 0x7fc578108ad0 2024-08-28T19:19:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.766+0000 7fc57c9eb640 1 -- 172.21.15.16:0/3862205347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5781a6b50 con 0x7fc578108ad0 2024-08-28T19:19:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.767+0000 7fc57c9eb640 1 -- 172.21.15.16:0/3862205347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc5780638b0 con 0x7fc578108ad0 2024-08-28T19:19:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.768+0000 7fc5677fe640 1 -- 172.21.15.16:0/3862205347 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fc570018d10 con 0x7fc578108ad0 2024-08-28T19:19:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.769+0000 7fc5677fe640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5600b4c40 0x7fc5600b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.769+0000 7fc57659c640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5600b4c40 0x7fc5600b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.769+0000 7fc5677fe640 1 -- 172.21.15.16:0/3862205347 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fc5700e5e30 con 0x7fc578108ad0 2024-08-28T19:19:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.769+0000 7fc5677fe640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769] conn(0x7fc5600bff50 0x7fc5600c2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.769+0000 7fc5677fe640 1 -- 172.21.15.16:0/3862205347 --> [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc570012ec0 con 0x7fc5600bff50 2024-08-28T19:19:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.769+0000 7fc5677fe640 1 -- 172.21.15.16:0/3862205347 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fc5700e61e0 con 0x7fc578108ad0 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.769+0000 7fc575d9b640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769] conn(0x7fc5600bff50 0x7fc5600c2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.770+0000 7fc57659c640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5600b4c40 0x7fc5600b7150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc568000c00 tx=0x7fc568031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.770+0000 7fc575d9b640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769] conn(0x7fc5600bff50 0x7fc5600c2440 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.770+0000 7fc5677fe640 1 -- 172.21.15.16:0/3862205347 <== osd.9 v2:172.21.15.16:6826/1827017769 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc570012ec0 con 0x7fc5600bff50 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.784+0000 7fc57c9eb640 1 -- 172.21.15.16:0/3862205347 --> [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc578063ce0 con 0x7fc5600bff50 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc5677fe640 1 -- 172.21.15.16:0/3862205347 <== osd.9 v2:172.21.15.16:6826/1827017769 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc578063ce0 con 0x7fc5600bff50 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc57c9eb640 1 -- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769] conn(0x7fc5600bff50 msgr2=0x7fc5600c2440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc57c9eb640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769] conn(0x7fc5600bff50 0x7fc5600c2440 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc57c9eb640 1 -- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5600b4c40 msgr2=0x7fc5600b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc57c9eb640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5600b4c40 0x7fc5600b7150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc568000c00 tx=0x7fc568031400 comp rx=0 tx=0).stop 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc57c9eb640 1 -- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc578108ad0 msgr2=0x7fc57819bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc57c9eb640 1 --2- 172.21.15.16:0/3862205347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc578108ad0 0x7fc57819bb50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc57000cce0 tx=0x7fc57000a620 comp rx=0 tx=0).stop 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc576d9d640 1 -- 172.21.15.16:0/3862205347 reap_dead start 2024-08-28T19:19:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc57c9eb640 1 -- 172.21.15.16:0/3862205347 shutdown_connections 2024-08-28T19:19:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc57c9eb640 1 -- 172.21.15.16:0/3862205347 >> 172.21.15.16:0/3862205347 conn(0x7fc5780794f0 msgr2=0x7fc578106290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc57c9eb640 1 -- 172.21.15.16:0/3862205347 shutdown_connections 2024-08-28T19:19:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:41.785+0000 7fc57c9eb640 1 -- 172.21.15.16:0/3862205347 wait complete. 2024-08-28T19:19:43.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:42 smithi016 ceph-mon[18556]: pgmap v297: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 938 B/s wr, 12 op/s 2024-08-28T19:19:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:42 smithi028 ceph-mon[21769]: pgmap v297: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 938 B/s wr, 12 op/s 2024-08-28T19:19:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:42 smithi115 ceph-mon[22184]: pgmap v297: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 938 B/s wr, 12 op/s 2024-08-28T19:19:44.164 INFO:teuthology.orchestra.run.smithi016.stdout:103079215171 2024-08-28T19:19:44.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-28T19:19:44.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.539+0000 7fc18fd51640 1 -- 172.21.15.16:0/96480576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc188102800 msgr2=0x7fc188102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:44.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.539+0000 7fc18fd51640 1 --2- 172.21.15.16:0/96480576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc188102800 0x7fc188102c10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc184009900 tx=0x7fc18402a410 comp rx=0 tx=0).stop 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.539+0000 7fc18fd51640 1 -- 172.21.15.16:0/96480576 shutdown_connections 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.539+0000 7fc18fd51640 1 --2- 172.21.15.16:0/96480576 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc188108a90 0x7fc18810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.539+0000 7fc18fd51640 1 --2- 172.21.15.16:0/96480576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc188108140 0x7fc188108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.539+0000 7fc18fd51640 1 --2- 172.21.15.16:0/96480576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc188102800 0x7fc188102c10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.539+0000 7fc18fd51640 1 -- 172.21.15.16:0/96480576 >> 172.21.15.16:0/96480576 conn(0x7fc1880794d0 msgr2=0x7fc1880798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.539+0000 7fc18fd51640 1 -- 172.21.15.16:0/96480576 shutdown_connections 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.539+0000 7fc18fd51640 1 -- 172.21.15.16:0/96480576 wait complete. 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18fd51640 1 Processor -- start 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18fd51640 1 -- start start 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18fd51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc188102800 0x7fc188103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18fd51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc188108140 0x7fc188104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18fd51640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc188108a90 0x7fc188104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:44.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18fd51640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc188112710 con 0x7fc188102800 2024-08-28T19:19:44.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18fd51640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc188112590 con 0x7fc188108140 2024-08-28T19:19:44.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18fd51640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc188112890 con 0x7fc188108a90 2024-08-28T19:19:44.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18daed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc188102800 0x7fc188103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:44.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18daed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc188102800 0x7fc188103ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56986/0 (socket says 172.21.15.16:56986) 2024-08-28T19:19:44.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18daed640 1 -- 172.21.15.16:0/873354969 learned_addr learned my addr 172.21.15.16:0/873354969 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:44.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18d2ec640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc188108140 0x7fc188104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:44.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18e2ee640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc188108a90 0x7fc188104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:44.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18daed640 1 -- 172.21.15.16:0/873354969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc188108a90 msgr2=0x7fc188104a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18daed640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc188108a90 0x7fc188104a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18daed640 1 -- 172.21.15.16:0/873354969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc188108140 msgr2=0x7fc188104530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18daed640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc188108140 0x7fc188104530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18daed640 1 -- 172.21.15.16:0/873354969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1881a8510 con 0x7fc188102800 2024-08-28T19:19:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.540+0000 7fc18e2ee640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc188108a90 0x7fc188104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.541+0000 7fc18daed640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc188102800 0x7fc188103ff0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc184005940 tx=0x7fc184004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.541+0000 7fc17effd640 1 -- 172.21.15.16:0/873354969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18403f030 con 0x7fc188102800 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.541+0000 7fc17effd640 1 -- 172.21.15.16:0/873354969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc184003aa0 con 0x7fc188102800 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.541+0000 7fc18fd51640 1 -- 172.21.15.16:0/873354969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1881a87a0 con 0x7fc188102800 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.541+0000 7fc17effd640 1 -- 172.21.15.16:0/873354969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184035070 con 0x7fc188102800 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.541+0000 7fc18fd51640 1 -- 172.21.15.16:0/873354969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1881a8c80 con 0x7fc188102800 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.542+0000 7fc18fd51640 1 -- 172.21.15.16:0/873354969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc188106410 con 0x7fc188102800 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.544+0000 7fc17effd640 1 -- 172.21.15.16:0/873354969 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fc184042030 con 0x7fc188102800 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.544+0000 7fc17effd640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc1740b4c10 0x7fc1740b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.544+0000 7fc18d2ec640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc1740b4c10 0x7fc1740b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.545+0000 7fc17effd640 1 -- 172.21.15.16:0/873354969 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fc1841065e0 con 0x7fc188102800 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.545+0000 7fc17effd640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596] conn(0x7fc1740bff20 0x7fc1740c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.545+0000 7fc17effd640 1 -- 172.21.15.16:0/873354969 --> [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc1740c2ae0 con 0x7fc1740bff20 2024-08-28T19:19:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.545+0000 7fc17effd640 1 -- 172.21.15.16:0/873354969 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fc184106a30 con 0x7fc188102800 2024-08-28T19:19:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.545+0000 7fc18d2ec640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc1740b4c10 0x7fc1740b7120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc188079990 tx=0x7fc17800b590 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.545+0000 7fc18e2ee640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596] conn(0x7fc1740bff20 0x7fc1740c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.545+0000 7fc18e2ee640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596] conn(0x7fc1740bff20 0x7fc1740c2410 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.546+0000 7fc17effd640 1 -- 172.21.15.16:0/873354969 <== osd.5 v2:172.21.15.115:6808/1119597596 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc1740c2ae0 con 0x7fc1740bff20 2024-08-28T19:19:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.560+0000 7fc18fd51640 1 -- 172.21.15.16:0/873354969 --> [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc188103a80 con 0x7fc1740bff20 2024-08-28T19:19:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc17effd640 1 -- 172.21.15.16:0/873354969 <== osd.5 v2:172.21.15.115:6808/1119597596 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc188103a80 con 0x7fc1740bff20 2024-08-28T19:19:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc18fd51640 1 -- 172.21.15.16:0/873354969 >> [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596] conn(0x7fc1740bff20 msgr2=0x7fc1740c2410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc18fd51640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596] conn(0x7fc1740bff20 0x7fc1740c2410 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc18fd51640 1 -- 172.21.15.16:0/873354969 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc1740b4c10 msgr2=0x7fc1740b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc18fd51640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc1740b4c10 0x7fc1740b7120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc188079990 tx=0x7fc17800b590 comp rx=0 tx=0).stop 2024-08-28T19:19:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc18fd51640 1 -- 172.21.15.16:0/873354969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc188102800 msgr2=0x7fc188103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc18fd51640 1 --2- 172.21.15.16:0/873354969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc188102800 0x7fc188103ff0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc184005940 tx=0x7fc184004270 comp rx=0 tx=0).stop 2024-08-28T19:19:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc18e2ee640 1 -- 172.21.15.16:0/873354969 reap_dead start 2024-08-28T19:19:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc18fd51640 1 -- 172.21.15.16:0/873354969 shutdown_connections 2024-08-28T19:19:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc18fd51640 1 -- 172.21.15.16:0/873354969 >> 172.21.15.16:0/873354969 conn(0x7fc1880794d0 msgr2=0x7fc1881018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc18fd51640 1 -- 172.21.15.16:0/873354969 shutdown_connections 2024-08-28T19:19:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:43.561+0000 7fc18fd51640 1 -- 172.21.15.16:0/873354969 wait complete. 2024-08-28T19:19:45.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:44 smithi016 ceph-mon[18556]: pgmap v298: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-08-28T19:19:45.295 INFO:teuthology.orchestra.run.smithi016.stdout:146028888120 2024-08-28T19:19:45.296 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-28T19:19:45.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:44 smithi028 ceph-mon[21769]: pgmap v298: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-08-28T19:19:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:44 smithi115 ceph-mon[22184]: pgmap v298: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-08-28T19:19:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.773+0000 7f440541b640 1 -- 172.21.15.16:0/815171545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440010dbb0 msgr2=0x7f44001100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.773+0000 7f440541b640 1 --2- 172.21.15.16:0/815171545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440010dbb0 0x7f44001100c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009ca0 tx=0x7f43f802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.773+0000 7f440541b640 1 -- 172.21.15.16:0/815171545 shutdown_connections 2024-08-28T19:19:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.773+0000 7f440541b640 1 --2- 172.21.15.16:0/815171545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440010dbb0 0x7f44001100c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.773+0000 7f440541b640 1 --2- 172.21.15.16:0/815171545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4400105030 0x7f440010d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.773+0000 7f440541b640 1 --2- 172.21.15.16:0/815171545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4400104600 0x7f4400104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.773+0000 7f440541b640 1 -- 172.21.15.16:0/815171545 >> 172.21.15.16:0/815171545 conn(0x7f44000794e0 msgr2=0x7f44000798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.773+0000 7f440541b640 1 -- 172.21.15.16:0/815171545 shutdown_connections 2024-08-28T19:19:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.773+0000 7f440541b640 1 -- 172.21.15.16:0/815171545 wait complete. 2024-08-28T19:19:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f440541b640 1 Processor -- start 2024-08-28T19:19:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f440541b640 1 -- start start 2024-08-28T19:19:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f440541b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4400104600 0x7f440019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f440541b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4400105030 0x7f440019a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f440541b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440010dbb0 0x7f440019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f440541b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44001138d0 con 0x7f4400104600 2024-08-28T19:19:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f440541b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4400113750 con 0x7f4400105030 2024-08-28T19:19:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f440541b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4400113a50 con 0x7f440010dbb0 2024-08-28T19:19:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f43feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4400104600 0x7f440019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f43feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4400104600 0x7f440019a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56126/0 (socket says 172.21.15.16:56126) 2024-08-28T19:19:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f43feffd640 1 -- 172.21.15.16:0/2990880765 learned_addr learned my addr 172.21.15.16:0/2990880765 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f43ff7fe640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440010dbb0 0x7f440019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.774+0000 7f43fe7fc640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4400105030 0x7f440019a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f43feffd640 1 -- 172.21.15.16:0/2990880765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440010dbb0 msgr2=0x7f440019fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f43feffd640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440010dbb0 0x7f440019fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f43feffd640 1 -- 172.21.15.16:0/2990880765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4400105030 msgr2=0x7f440019a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f43feffd640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4400105030 0x7f440019a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f43feffd640 1 -- 172.21.15.16:0/2990880765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44001a0270 con 0x7f4400104600 2024-08-28T19:19:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f43ff7fe640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440010dbb0 0x7f440019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f43fe7fc640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4400105030 0x7f440019a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f43feffd640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4400104600 0x7f440019a300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f43f000b450 tx=0x7f43f000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f43e7fff640 1 -- 172.21.15.16:0/2990880765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0015020 con 0x7f4400104600 2024-08-28T19:19:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f440541b640 1 -- 172.21.15.16:0/2990880765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44001a0560 con 0x7f4400104600 2024-08-28T19:19:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f43e7fff640 1 -- 172.21.15.16:0/2990880765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43f0011d40 con 0x7f4400104600 2024-08-28T19:19:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f43e7fff640 1 -- 172.21.15.16:0/2990880765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f000f340 con 0x7f4400104600 2024-08-28T19:19:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.775+0000 7f440541b640 1 -- 172.21.15.16:0/2990880765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44001a6cc0 con 0x7f4400104600 2024-08-28T19:19:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.776+0000 7f440541b640 1 -- 172.21.15.16:0/2990880765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4400100d80 con 0x7f4400104600 2024-08-28T19:19:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.777+0000 7f43e7fff640 1 -- 172.21.15.16:0/2990880765 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f43f0018020 con 0x7f4400104600 2024-08-28T19:19:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.778+0000 7f43e7fff640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43e80b4c10 0x7f43e80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.778+0000 7f43fe7fc640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43e80b4c10 0x7f43e80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.778+0000 7f43e7fff640 1 -- 172.21.15.16:0/2990880765 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f43f00e58c0 con 0x7f4400104600 2024-08-28T19:19:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.779+0000 7f43e7fff640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207] conn(0x7f43e80bff20 0x7f43e80c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:45.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.779+0000 7f43e7fff640 1 -- 172.21.15.16:0/2990880765 --> [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43f0017e50 con 0x7f43e80bff20 2024-08-28T19:19:45.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.779+0000 7f43e7fff640 1 -- 172.21.15.16:0/2990880765 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f43f00e5c70 con 0x7f4400104600 2024-08-28T19:19:45.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.779+0000 7f43fe7fc640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43e80b4c10 0x7f43e80b7120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f43f40058c0 tx=0x7f43f4005850 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:45.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.779+0000 7f43ff7fe640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207] conn(0x7f43e80bff20 0x7f43e80c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:45.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.779+0000 7f43ff7fe640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207] conn(0x7f43e80bff20 0x7f43e80c2410 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.780+0000 7f43e7fff640 1 -- 172.21.15.16:0/2990880765 <== osd.10 v2:172.21.15.28:6824/4224878207 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f43f0017e50 con 0x7f43e80bff20 2024-08-28T19:19:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.794+0000 7f440541b640 1 -- 172.21.15.16:0/2990880765 --> [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f440006b130 con 0x7f43e80bff20 2024-08-28T19:19:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f43e7fff640 1 -- 172.21.15.16:0/2990880765 <== osd.10 v2:172.21.15.28:6824/4224878207 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f440006b130 con 0x7f43e80bff20 2024-08-28T19:19:45.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f440541b640 1 -- 172.21.15.16:0/2990880765 >> [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207] conn(0x7f43e80bff20 msgr2=0x7f43e80c2410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:45.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f440541b640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207] conn(0x7f43e80bff20 0x7f43e80c2410 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:45.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f440541b640 1 -- 172.21.15.16:0/2990880765 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43e80b4c10 msgr2=0x7f43e80b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:45.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f440541b640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43e80b4c10 0x7f43e80b7120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f43f40058c0 tx=0x7f43f4005850 comp rx=0 tx=0).stop 2024-08-28T19:19:45.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f440541b640 1 -- 172.21.15.16:0/2990880765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4400104600 msgr2=0x7f440019a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:45.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f440541b640 1 --2- 172.21.15.16:0/2990880765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4400104600 0x7f440019a300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f43f000b450 tx=0x7f43f000b910 comp rx=0 tx=0).stop 2024-08-28T19:19:45.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f43ff7fe640 1 -- 172.21.15.16:0/2990880765 reap_dead start 2024-08-28T19:19:45.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f440541b640 1 -- 172.21.15.16:0/2990880765 shutdown_connections 2024-08-28T19:19:45.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f440541b640 1 -- 172.21.15.16:0/2990880765 >> 172.21.15.16:0/2990880765 conn(0x7f44000794e0 msgr2=0x7f44000ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:45.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f440541b640 1 -- 172.21.15.16:0/2990880765 shutdown_connections 2024-08-28T19:19:45.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:45.795+0000 7f440541b640 1 -- 172.21.15.16:0/2990880765 wait complete. 2024-08-28T19:19:46.566 INFO:teuthology.orchestra.run.smithi016.stdout:244813135910 2024-08-28T19:19:46.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-08-28T19:19:46.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.831+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4142791217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5c81081f0 msgr2=0x7fb5c8108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:46.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.831+0000 7fb5cdc3b640 1 --2- 172.21.15.16:0/4142791217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5c81081f0 0x7fb5c8108600 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0009900 tx=0x7fb5b002a410 comp rx=0 tx=0).stop 2024-08-28T19:19:46.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.832+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4142791217 shutdown_connections 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.832+0000 7fb5cdc3b640 1 --2- 172.21.15.16:0/4142791217 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5c8108b40 0x7fb5c810f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.832+0000 7fb5cdc3b640 1 --2- 172.21.15.16:0/4142791217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5c81081f0 0x7fb5c8108600 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.832+0000 7fb5cdc3b640 1 --2- 172.21.15.16:0/4142791217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c81028b0 0x7fb5c8102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.832+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4142791217 >> 172.21.15.16:0/4142791217 conn(0x7fb5c8079470 msgr2=0x7fb5c8079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.832+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4142791217 shutdown_connections 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.832+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4142791217 wait complete. 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.832+0000 7fb5cdc3b640 1 Processor -- start 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.832+0000 7fb5cdc3b640 1 -- start start 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5cdc3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c81028b0 0x7fb5c819a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5cdc3b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5c81081f0 0x7fb5c819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5cdc3b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5c8108b40 0x7fb5c819fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5cdc3b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c8112820 con 0x7fb5c8108b40 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5cdc3b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c81126a0 con 0x7fb5c81028b0 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5cdc3b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c81129a0 con 0x7fb5c81081f0 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5c8108b40 0x7fb5c819fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5c8108b40 0x7fb5c819fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56158/0 (socket says 172.21.15.16:56158) 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5c7fff640 1 -- 172.21.15.16:0/4287193996 learned_addr learned my addr 172.21.15.16:0/4287193996 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5c77fe640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c81028b0 0x7fb5c819a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5beffd640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5c81081f0 0x7fb5c819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5c7fff640 1 -- 172.21.15.16:0/4287193996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5c81081f0 msgr2=0x7fb5c819a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5c7fff640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5c81081f0 0x7fb5c819a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5c7fff640 1 -- 172.21.15.16:0/4287193996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c81028b0 msgr2=0x7fb5c819a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5c7fff640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c81028b0 0x7fb5c819a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5c7fff640 1 -- 172.21.15.16:0/4287193996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c81a0290 con 0x7fb5c8108b40 2024-08-28T19:19:46.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.833+0000 7fb5beffd640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5c81081f0 0x7fb5c819a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.834+0000 7fb5c7fff640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5c8108b40 0x7fb5c819fb60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb5c000b970 tx=0x7fb5c000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.834+0000 7fb5c57fa640 1 -- 172.21.15.16:0/4287193996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0019070 con 0x7fb5c8108b40 2024-08-28T19:19:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.834+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4287193996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c81a0580 con 0x7fb5c8108b40 2024-08-28T19:19:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.834+0000 7fb5c57fa640 1 -- 172.21.15.16:0/4287193996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5c0014440 con 0x7fb5c8108b40 2024-08-28T19:19:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.834+0000 7fb5c57fa640 1 -- 172.21.15.16:0/4287193996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c00075d0 con 0x7fb5c8108b40 2024-08-28T19:19:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.834+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4287193996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5c81a6ce0 con 0x7fb5c8108b40 2024-08-28T19:19:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.835+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4287193996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb5c80638f0 con 0x7fb5c8108b40 2024-08-28T19:19:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.837+0000 7fb5c57fa640 1 -- 172.21.15.16:0/4287193996 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fb5c001c070 con 0x7fb5c8108b40 2024-08-28T19:19:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.837+0000 7fb5c57fa640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb5b40b4c10 0x7fb5b40b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.837+0000 7fb5c77fe640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb5b40b4c10 0x7fb5b40b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.837+0000 7fb5c57fa640 1 -- 172.21.15.16:0/4287193996 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fb5c00e53c0 con 0x7fb5c8108b40 2024-08-28T19:19:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.838+0000 7fb5c57fa640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149] conn(0x7fb5b40bff20 0x7fb5b40c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.838+0000 7fb5c57fa640 1 -- 172.21.15.16:0/4287193996 --> [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5c0020e50 con 0x7fb5b40bff20 2024-08-28T19:19:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.838+0000 7fb5c57fa640 1 -- 172.21.15.16:0/4287193996 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fb5c00e5810 con 0x7fb5c8108b40 2024-08-28T19:19:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.838+0000 7fb5c77fe640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb5b40b4c10 0x7fb5b40b7120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800bbe0 tx=0x7fb5b800b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.838+0000 7fb5beffd640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149] conn(0x7fb5b40bff20 0x7fb5b40c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.838+0000 7fb5beffd640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149] conn(0x7fb5b40bff20 0x7fb5b40c2410 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.839+0000 7fb5c57fa640 1 -- 172.21.15.16:0/4287193996 <== osd.8 v2:172.21.15.115:6816/557022149 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5c0020e50 con 0x7fb5b40bff20 2024-08-28T19:19:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.853+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4287193996 --> [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb5c8063d70 con 0x7fb5b40bff20 2024-08-28T19:19:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5c57fa640 1 -- 172.21.15.16:0/4287193996 <== osd.8 v2:172.21.15.115:6816/557022149 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb5c8063d70 con 0x7fb5b40bff20 2024-08-28T19:19:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4287193996 >> [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149] conn(0x7fb5b40bff20 msgr2=0x7fb5b40c2410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5cdc3b640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149] conn(0x7fb5b40bff20 0x7fb5b40c2410 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4287193996 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb5b40b4c10 msgr2=0x7fb5b40b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5cdc3b640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb5b40b4c10 0x7fb5b40b7120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800bbe0 tx=0x7fb5b800b390 comp rx=0 tx=0).stop 2024-08-28T19:19:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4287193996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5c8108b40 msgr2=0x7fb5c819fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5cdc3b640 1 --2- 172.21.15.16:0/4287193996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5c8108b40 0x7fb5c819fb60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb5c000b970 tx=0x7fb5c000be30 comp rx=0 tx=0).stop 2024-08-28T19:19:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5c7fff640 1 -- 172.21.15.16:0/4287193996 reap_dead start 2024-08-28T19:19:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4287193996 shutdown_connections 2024-08-28T19:19:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4287193996 >> 172.21.15.16:0/4287193996 conn(0x7fb5c8079470 msgr2=0x7fb5c81018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4287193996 shutdown_connections 2024-08-28T19:19:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.854+0000 7fb5cdc3b640 1 -- 172.21.15.16:0/4287193996 wait complete. 2024-08-28T19:19:47.220 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:46 smithi016 ceph-mon[18556]: pgmap v299: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 11 op/s 2024-08-28T19:19:47.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:46 smithi028 ceph-mon[21769]: pgmap v299: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 11 op/s 2024-08-28T19:19:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:46 smithi115 ceph-mon[22184]: pgmap v299: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 11 op/s 2024-08-28T19:19:47.556 INFO:teuthology.orchestra.run.smithi016.stdout:219043332139 2024-08-28T19:19:47.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-08-28T19:19:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.091+0000 7fa3b81aa640 1 -- 172.21.15.16:0/97348515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b01025c0 msgr2=0x7fa3b01029d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.091+0000 7fa3b81aa640 1 --2- 172.21.15.16:0/97348515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b01025c0 0x7fa3b01029d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac009930 tx=0x7fa3ac02a420 comp rx=0 tx=0).stop 2024-08-28T19:19:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.091+0000 7fa3b81aa640 1 -- 172.21.15.16:0/97348515 shutdown_connections 2024-08-28T19:19:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.091+0000 7fa3b81aa640 1 --2- 172.21.15.16:0/97348515 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3b0107a00 0x7fa3b0107ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.091+0000 7fa3b81aa640 1 --2- 172.21.15.16:0/97348515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b0102f10 0x7fa3b0107440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.091+0000 7fa3b81aa640 1 --2- 172.21.15.16:0/97348515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b01025c0 0x7fa3b01029d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.091+0000 7fa3b81aa640 1 -- 172.21.15.16:0/97348515 >> 172.21.15.16:0/97348515 conn(0x7fa3b0079510 msgr2=0x7fa3b0079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.091+0000 7fa3b81aa640 1 -- 172.21.15.16:0/97348515 shutdown_connections 2024-08-28T19:19:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.091+0000 7fa3b81aa640 1 -- 172.21.15.16:0/97348515 wait complete. 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b81aa640 1 Processor -- start 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b81aa640 1 -- start start 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b81aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b01025c0 0x7fa3b01a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b81aa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b0102f10 0x7fa3b01a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b81aa640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3b0107a00 0x7fa3b01a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b81aa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b0119ea0 con 0x7fa3b0102f10 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b81aa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b0119d20 con 0x7fa3b01025c0 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b81aa640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b011a020 con 0x7fa3b0107a00 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b5745640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b0102f10 0x7fa3b01a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b5745640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b0102f10 0x7fa3b01a0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56182/0 (socket says 172.21.15.16:56182) 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b5745640 1 -- 172.21.15.16:0/566412599 learned_addr learned my addr 172.21.15.16:0/566412599 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b5f46640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b01025c0 0x7fa3b01a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.092+0000 7fa3b6747640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3b0107a00 0x7fa3b01a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3b5745640 1 -- 172.21.15.16:0/566412599 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3b0107a00 msgr2=0x7fa3b01a6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3b5745640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3b0107a00 0x7fa3b01a6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3b5745640 1 -- 172.21.15.16:0/566412599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b01025c0 msgr2=0x7fa3b01a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3b5745640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b01025c0 0x7fa3b01a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3b5745640 1 -- 172.21.15.16:0/566412599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b01a67a0 con 0x7fa3b0102f10 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3b5f46640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b01025c0 0x7fa3b01a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3b5745640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b0102f10 0x7fa3b01a0e90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000d740 tx=0x7fa3a000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3a6ffd640 1 -- 172.21.15.16:0/566412599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0017070 con 0x7fa3b0102f10 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3b81aa640 1 -- 172.21.15.16:0/566412599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b01a6a90 con 0x7fa3b0102f10 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3a6ffd640 1 -- 172.21.15.16:0/566412599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3a000de50 con 0x7fa3b0102f10 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3a6ffd640 1 -- 172.21.15.16:0/566412599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000f620 con 0x7fa3b0102f10 2024-08-28T19:19:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.093+0000 7fa3b81aa640 1 -- 172.21.15.16:0/566412599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b01ad2b0 con 0x7fa3b0102f10 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.094+0000 7fa3b81aa640 1 -- 172.21.15.16:0/566412599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa3b0106e00 con 0x7fa3b0102f10 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.096+0000 7fa3a6ffd640 1 -- 172.21.15.16:0/566412599 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fa3a001a070 con 0x7fa3b0102f10 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.096+0000 7fa3a6ffd640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa39c0b4c40 0x7fa39c0b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.096+0000 7fa3b5f46640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa39c0b4c40 0x7fa39c0b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.096+0000 7fa3a6ffd640 1 -- 172.21.15.16:0/566412599 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fa3a00e57d0 con 0x7fa3b0102f10 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.097+0000 7fa3a6ffd640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111] conn(0x7fa39c0bff50 0x7fa39c0c2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.097+0000 7fa3a6ffd640 1 -- 172.21.15.16:0/566412599 --> [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3a001ae50 con 0x7fa39c0bff50 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.097+0000 7fa3a6ffd640 1 -- 172.21.15.16:0/566412599 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fa3a00e5c20 con 0x7fa3b0102f10 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.097+0000 7fa3b6747640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111] conn(0x7fa39c0bff50 0x7fa39c0c2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.097+0000 7fa3b5f46640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa39c0b4c40 0x7fa39c0b7150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac000c00 tx=0x7fa3ac032da0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.097+0000 7fa3b6747640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111] conn(0x7fa39c0bff50 0x7fa39c0c2440 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.098+0000 7fa3a6ffd640 1 -- 172.21.15.16:0/566412599 <== osd.11 v2:172.21.15.115:6824/3906373111 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3a001ae50 con 0x7fa39c0bff50 2024-08-28T19:19:48.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.112+0000 7fa3b81aa640 1 -- 172.21.15.16:0/566412599 --> [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa3b006b130 con 0x7fa39c0bff50 2024-08-28T19:19:48.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3a6ffd640 1 -- 172.21.15.16:0/566412599 <== osd.11 v2:172.21.15.115:6824/3906373111 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa3b006b130 con 0x7fa39c0bff50 2024-08-28T19:19:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3b81aa640 1 -- 172.21.15.16:0/566412599 >> [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111] conn(0x7fa39c0bff50 msgr2=0x7fa39c0c2440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3b81aa640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111] conn(0x7fa39c0bff50 0x7fa39c0c2440 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3b81aa640 1 -- 172.21.15.16:0/566412599 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa39c0b4c40 msgr2=0x7fa39c0b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3b81aa640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa39c0b4c40 0x7fa39c0b7150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac000c00 tx=0x7fa3ac032da0 comp rx=0 tx=0).stop 2024-08-28T19:19:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3b81aa640 1 -- 172.21.15.16:0/566412599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b0102f10 msgr2=0x7fa3b01a0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3b81aa640 1 --2- 172.21.15.16:0/566412599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b0102f10 0x7fa3b01a0e90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000d740 tx=0x7fa3a000dc00 comp rx=0 tx=0).stop 2024-08-28T19:19:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3b6747640 1 -- 172.21.15.16:0/566412599 reap_dead start 2024-08-28T19:19:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3b81aa640 1 -- 172.21.15.16:0/566412599 shutdown_connections 2024-08-28T19:19:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3b81aa640 1 -- 172.21.15.16:0/566412599 >> 172.21.15.16:0/566412599 conn(0x7fa3b0079510 msgr2=0x7fa3b0105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3b81aa640 1 -- 172.21.15.16:0/566412599 shutdown_connections 2024-08-28T19:19:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.113+0000 7fa3b81aa640 1 -- 172.21.15.16:0/566412599 wait complete. 2024-08-28T19:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout:292057776157 2024-08-28T19:19:48.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-08-28T19:19:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.208+0000 7f17d6c05640 1 -- 172.21.15.16:0/3059334530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17d0103950 msgr2=0x7f17d0103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.208+0000 7f17d6c05640 1 --2- 172.21.15.16:0/3059334530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17d0103950 0x7f17d0103d60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f17c4009900 tx=0x7f17c402a410 comp rx=0 tx=0).stop 2024-08-28T19:19:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.208+0000 7f17d6c05640 1 -- 172.21.15.16:0/3059334530 shutdown_connections 2024-08-28T19:19:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.208+0000 7f17d6c05640 1 --2- 172.21.15.16:0/3059334530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d0109be0 0x7f17d0110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.208+0000 7f17d6c05640 1 --2- 172.21.15.16:0/3059334530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17d0109290 0x7f17d01096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.208+0000 7f17d6c05640 1 --2- 172.21.15.16:0/3059334530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17d0103950 0x7f17d0103d60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.208+0000 7f17d6c05640 1 -- 172.21.15.16:0/3059334530 >> 172.21.15.16:0/3059334530 conn(0x7f17d00ff3a0 msgr2=0x7f17d01017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.208+0000 7f17d6c05640 1 -- 172.21.15.16:0/3059334530 shutdown_connections 2024-08-28T19:19:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.208+0000 7f17d6c05640 1 -- 172.21.15.16:0/3059334530 wait complete. 2024-08-28T19:19:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17d6c05640 1 Processor -- start 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17d6c05640 1 -- start start 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17d6c05640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17d0103950 0x7f17d019c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17d6c05640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17d0109290 0x7f17d019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17d6c05640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d0109be0 0x7f17d01a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17d6c05640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f17d01138a0 con 0x7f17d0109290 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17d6c05640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0113720 con 0x7f17d0109be0 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17d6c05640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0113a20 con 0x7f17d0103950 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17d0109290 0x7f17d019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17d0109290 0x7f17d019ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56140/0 (socket says 172.21.15.16:56140) 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17cffff640 1 -- 172.21.15.16:0/3670927054 learned_addr learned my addr 172.21.15.16:0/3670927054 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17d51a2640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d0109be0 0x7f17d01a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.209+0000 7f17d49a1640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17d0103950 0x7f17d019c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17cffff640 1 -- 172.21.15.16:0/3670927054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17d0103950 msgr2=0x7f17d019c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17cffff640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17d0103950 0x7f17d019c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17cffff640 1 -- 172.21.15.16:0/3670927054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d0109be0 msgr2=0x7f17d01a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17cffff640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d0109be0 0x7f17d01a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17cffff640 1 -- 172.21.15.16:0/3670927054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d01a2440 con 0x7f17d0109290 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17d49a1640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17d0103950 0x7f17d019c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17cffff640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17d0109290 0x7f17d019ca70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00e9c0 tx=0x7f17bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17cdffb640 1 -- 172.21.15.16:0/3670927054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00ccc0 con 0x7f17d0109290 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17cdffb640 1 -- 172.21.15.16:0/3670927054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17bc0054d0 con 0x7f17d0109290 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17d6c05640 1 -- 172.21.15.16:0/3670927054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d01a2730 con 0x7f17d0109290 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17cdffb640 1 -- 172.21.15.16:0/3670927054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc010430 con 0x7f17d0109290 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.210+0000 7f17d6c05640 1 -- 172.21.15.16:0/3670927054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d01a8e90 con 0x7f17d0109290 2024-08-28T19:19:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.211+0000 7f17d6c05640 1 -- 172.21.15.16:0/3670927054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f17d0110120 con 0x7f17d0109290 2024-08-28T19:19:49.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.212+0000 7f17cdffb640 1 -- 172.21.15.16:0/3670927054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f17bc005000 con 0x7f17d0109290 2024-08-28T19:19:49.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.213+0000 7f17cdffb640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f17c00b4c90 0x7f17c00b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:49.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.213+0000 7f17d49a1640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f17c00b4c90 0x7f17c00b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:49.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.213+0000 7f17cdffb640 1 -- 172.21.15.16:0/3670927054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f17bc0e5f10 con 0x7f17d0109290 2024-08-28T19:19:49.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.213+0000 7f17cdffb640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315] conn(0x7f17c00bffa0 0x7f17c00c2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:49.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.213+0000 7f17cdffb640 1 -- 172.21.15.16:0/3670927054 --> [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17bc011e50 con 0x7f17c00bffa0 2024-08-28T19:19:49.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.213+0000 7f17cdffb640 1 -- 172.21.15.16:0/3670927054 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f17bc0e62c0 con 0x7f17d0109290 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.214+0000 7f17d49a1640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f17c00b4c90 0x7f17c00b71a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f17c40096c0 tx=0x7f17c4031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.214+0000 7f17d51a2640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315] conn(0x7f17c00bffa0 0x7f17c00c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.214+0000 7f17d51a2640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315] conn(0x7f17c00bffa0 0x7f17c00c2490 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.215+0000 7f17cdffb640 1 -- 172.21.15.16:0/3670927054 <== osd.7 v2:172.21.15.28:6816/2650596315 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f17bc011e50 con 0x7f17c00bffa0 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.230+0000 7f17d6c05640 1 -- 172.21.15.16:0/3670927054 --> [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f17d0104bd0 con 0x7f17c00bffa0 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.230+0000 7f17cdffb640 1 -- 172.21.15.16:0/3670927054 <== osd.7 v2:172.21.15.28:6816/2650596315 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f17d0104bd0 con 0x7f17c00bffa0 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.231+0000 7f17d6c05640 1 -- 172.21.15.16:0/3670927054 >> [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315] conn(0x7f17c00bffa0 msgr2=0x7f17c00c2490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.231+0000 7f17d6c05640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315] conn(0x7f17c00bffa0 0x7f17c00c2490 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.231+0000 7f17d6c05640 1 -- 172.21.15.16:0/3670927054 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f17c00b4c90 msgr2=0x7f17c00b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.231+0000 7f17d6c05640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f17c00b4c90 0x7f17c00b71a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f17c40096c0 tx=0x7f17c4031400 comp rx=0 tx=0).stop 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.231+0000 7f17d6c05640 1 -- 172.21.15.16:0/3670927054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17d0109290 msgr2=0x7f17d019ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.231+0000 7f17d6c05640 1 --2- 172.21.15.16:0/3670927054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17d0109290 0x7f17d019ca70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00e9c0 tx=0x7f17bc00ee80 comp rx=0 tx=0).stop 2024-08-28T19:19:49.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.231+0000 7f17d51a2640 1 -- 172.21.15.16:0/3670927054 reap_dead start 2024-08-28T19:19:49.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.231+0000 7f17d6c05640 1 -- 172.21.15.16:0/3670927054 shutdown_connections 2024-08-28T19:19:49.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.231+0000 7f17d6c05640 1 -- 172.21.15.16:0/3670927054 >> 172.21.15.16:0/3670927054 conn(0x7f17d00ff3a0 msgr2=0x7f17d010e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:49.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.231+0000 7f17d6c05640 1 -- 172.21.15.16:0/3670927054 shutdown_connections 2024-08-28T19:19:49.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:46.231+0000 7f17d6c05640 1 -- 172.21.15.16:0/3670927054 wait complete. 2024-08-28T19:19:49.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:48 smithi016 ceph-mon[18556]: pgmap v300: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-08-28T19:19:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:48 smithi028 ceph-mon[21769]: pgmap v300: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-08-28T19:19:49.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:48 smithi115 ceph-mon[22184]: pgmap v300: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-08-28T19:19:49.625 INFO:teuthology.orchestra.run.smithi016.stdout:193273528368 2024-08-28T19:19:49.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-28T19:19:50.143 INFO:teuthology.orchestra.run.smithi016.stdout:38654705746 2024-08-28T19:19:50.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-28T19:19:50.639 INFO:teuthology.orchestra.run.smithi016.stdout:77309411400 2024-08-28T19:19:50.640 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-28T19:19:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.941+0000 7f715880f640 1 -- 172.21.15.16:0/1416596459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7154104660 msgr2=0x7f7154104a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.941+0000 7f715880f640 1 --2- 172.21.15.16:0/1416596459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7154104660 0x7f7154104a70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7144009900 tx=0x7f714402a410 comp rx=0 tx=0).stop 2024-08-28T19:19:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.941+0000 7f715880f640 1 -- 172.21.15.16:0/1416596459 shutdown_connections 2024-08-28T19:19:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.941+0000 7f715880f640 1 --2- 172.21.15.16:0/1416596459 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f715410dc10 0x7f71541100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.941+0000 7f715880f640 1 --2- 172.21.15.16:0/1416596459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7154105040 0x7f715410d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.941+0000 7f715880f640 1 --2- 172.21.15.16:0/1416596459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7154104660 0x7f7154104a70 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.941+0000 7f715880f640 1 -- 172.21.15.16:0/1416596459 >> 172.21.15.16:0/1416596459 conn(0x7f7154079530 msgr2=0x7f7154079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.941+0000 7f715880f640 1 -- 172.21.15.16:0/1416596459 shutdown_connections 2024-08-28T19:19:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.941+0000 7f715880f640 1 -- 172.21.15.16:0/1416596459 wait complete. 2024-08-28T19:19:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.942+0000 7f715880f640 1 Processor -- start 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.942+0000 7f715880f640 1 -- start start 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.942+0000 7f715880f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7154104660 0x7f715419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.942+0000 7f715880f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7154105040 0x7f715419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.942+0000 7f715880f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f715410dc10 0x7f715419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.942+0000 7f715880f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7154113850 con 0x7f7154105040 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.942+0000 7f715880f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71541136d0 con 0x7f715410dc10 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.942+0000 7f715880f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f71541139d0 con 0x7f7154104660 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.942+0000 7f7151d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7154105040 0x7f715419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f7151d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7154105040 0x7f715419a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56196/0 (socket says 172.21.15.16:56196) 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f7151d9b640 1 -- 172.21.15.16:0/2147846554 learned_addr learned my addr 172.21.15.16:0/2147846554 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f7152d9d640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f715410dc10 0x7f715419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f715259c640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7154104660 0x7f715419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f7151d9b640 1 -- 172.21.15.16:0/2147846554 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7154104660 msgr2=0x7f715419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f7151d9b640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7154104660 0x7f715419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f7151d9b640 1 -- 172.21.15.16:0/2147846554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f715410dc10 msgr2=0x7f715419faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f7151d9b640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f715410dc10 0x7f715419faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f7151d9b640 1 -- 172.21.15.16:0/2147846554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71541a01d0 con 0x7f7154105040 2024-08-28T19:19:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f715259c640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7154104660 0x7f715419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f7151d9b640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7154105040 0x7f715419a800 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f714800e9f0 tx=0x7f714800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f71437fe640 1 -- 172.21.15.16:0/2147846554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714800ccc0 con 0x7f7154105040 2024-08-28T19:19:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f71437fe640 1 -- 172.21.15.16:0/2147846554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71480108e0 con 0x7f7154105040 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f715880f640 1 -- 172.21.15.16:0/2147846554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71541a04c0 con 0x7f7154105040 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f71437fe640 1 -- 172.21.15.16:0/2147846554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148018860 con 0x7f7154105040 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.943+0000 7f715880f640 1 -- 172.21.15.16:0/2147846554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71541a6c20 con 0x7f7154105040 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.944+0000 7f715880f640 1 -- 172.21.15.16:0/2147846554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f71541081c0 con 0x7f7154105040 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.947+0000 7f71437fe640 1 -- 172.21.15.16:0/2147846554 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f7148010430 con 0x7f7154105040 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.948+0000 7f71437fe640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f713c0b4c10 0x7f713c0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.948+0000 7f715259c640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f713c0b4c10 0x7f713c0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.948+0000 7f71437fe640 1 -- 172.21.15.16:0/2147846554 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f71480e65e0 con 0x7f7154105040 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.948+0000 7f71437fe640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622] conn(0x7f713c0bff20 0x7f713c0c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.948+0000 7f71437fe640 1 -- 172.21.15.16:0/2147846554 --> [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f713c0c2ae0 con 0x7f713c0bff20 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.948+0000 7f71437fe640 1 -- 172.21.15.16:0/2147846554 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f71480e6a30 con 0x7f7154105040 2024-08-28T19:19:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.949+0000 7f7152d9d640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622] conn(0x7f713c0bff20 0x7f713c0c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.949+0000 7f715259c640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f713c0b4c10 0x7f713c0b7120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7144031440 tx=0x7f7144009680 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.949+0000 7f7152d9d640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622] conn(0x7f713c0bff20 0x7f713c0c2410 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.950+0000 7f71437fe640 1 -- 172.21.15.16:0/2147846554 <== osd.6 v2:172.21.15.16:6818/1671078622 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f713c0c2ae0 con 0x7f713c0bff20 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.965+0000 7f715880f640 1 -- 172.21.15.16:0/2147846554 --> [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f715406bc10 con 0x7f713c0bff20 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.965+0000 7f71437fe640 1 -- 172.21.15.16:0/2147846554 <== osd.6 v2:172.21.15.16:6818/1671078622 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f715406bc10 con 0x7f713c0bff20 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.966+0000 7f715880f640 1 -- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622] conn(0x7f713c0bff20 msgr2=0x7f713c0c2410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.966+0000 7f715880f640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622] conn(0x7f713c0bff20 0x7f713c0c2410 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.966+0000 7f715880f640 1 -- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f713c0b4c10 msgr2=0x7f713c0b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.966+0000 7f715880f640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f713c0b4c10 0x7f713c0b7120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7144031440 tx=0x7f7144009680 comp rx=0 tx=0).stop 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.966+0000 7f715880f640 1 -- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7154105040 msgr2=0x7f715419a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.966+0000 7f715880f640 1 --2- 172.21.15.16:0/2147846554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7154105040 0x7f715419a800 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f714800e9f0 tx=0x7f714800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.966+0000 7f7152d9d640 1 -- 172.21.15.16:0/2147846554 reap_dead start 2024-08-28T19:19:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.966+0000 7f715880f640 1 -- 172.21.15.16:0/2147846554 shutdown_connections 2024-08-28T19:19:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.966+0000 7f715880f640 1 -- 172.21.15.16:0/2147846554 >> 172.21.15.16:0/2147846554 conn(0x7f7154079530 msgr2=0x7f7154107760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.966+0000 7f715880f640 1 -- 172.21.15.16:0/2147846554 shutdown_connections 2024-08-28T19:19:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:48.966+0000 7f715880f640 1 -- 172.21.15.16:0/2147846554 wait complete. 2024-08-28T19:19:51.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:50 smithi016 ceph-mon[18556]: pgmap v301: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-08-28T19:19:51.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:50 smithi028 ceph-mon[21769]: pgmap v301: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-08-28T19:19:51.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:50 smithi115 ceph-mon[22184]: pgmap v301: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-08-28T19:19:51.407 INFO:teuthology.orchestra.run.smithi016.stdout:171798691894 2024-08-28T19:19:51.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-28T19:19:51.832 INFO:teuthology.orchestra.run.smithi016.stdout:55834574926 2024-08-28T19:19:51.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-28T19:19:52.264 INFO:teuthology.orchestra.run.smithi016.stdout:266287972385 2024-08-28T19:19:52.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-08-28T19:19:52.336 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:52.393 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:52.462 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:52.463 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:52.463 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:52.463 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:52.469 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:52.470 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:52.473 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:52.476 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:53.175 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:52 smithi016 ceph-mon[18556]: pgmap v302: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:19:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:52 smithi028 ceph-mon[21769]: pgmap v302: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:19:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:52 smithi115 ceph-mon[22184]: pgmap v302: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:19:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.445+0000 7f4aec8cc640 1 -- 172.21.15.16:0/212515891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ae8102800 msgr2=0x7f4ae8102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.445+0000 7f4aec8cc640 1 --2- 172.21.15.16:0/212515891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ae8102800 0x7f4ae8102c10 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0009900 tx=0x7f4ae002a410 comp rx=0 tx=0).stop 2024-08-28T19:19:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.446+0000 7f4aec8cc640 1 -- 172.21.15.16:0/212515891 shutdown_connections 2024-08-28T19:19:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.446+0000 7f4aec8cc640 1 --2- 172.21.15.16:0/212515891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae8108a90 0x7f4ae810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.446+0000 7f4aec8cc640 1 --2- 172.21.15.16:0/212515891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ae8108140 0x7f4ae8108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.446+0000 7f4aec8cc640 1 --2- 172.21.15.16:0/212515891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ae8102800 0x7f4ae8102c10 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.446+0000 7f4aec8cc640 1 -- 172.21.15.16:0/212515891 >> 172.21.15.16:0/212515891 conn(0x7f4ae80794d0 msgr2=0x7f4ae80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.446+0000 7f4aec8cc640 1 -- 172.21.15.16:0/212515891 shutdown_connections 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.446+0000 7f4aec8cc640 1 -- 172.21.15.16:0/212515891 wait complete. 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4aec8cc640 1 Processor -- start 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4aec8cc640 1 -- start start 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4aec8cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae8102800 0x7f4ae819b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4aec8cc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ae8108140 0x7f4ae819b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4aec8cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ae8108a90 0x7f4ae819bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4aec8cc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae81126f0 con 0x7f4ae8108a90 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4aec8cc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae8112570 con 0x7f4ae8102800 2024-08-28T19:19:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4aec8cc640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae8112870 con 0x7f4ae8108140 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4ae6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ae8108a90 0x7f4ae819bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4ae6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ae8108a90 0x7f4ae819bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56226/0 (socket says 172.21.15.16:56226) 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4ae659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae8102800 0x7f4ae819b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.447+0000 7f4ae6d9d640 1 -- 172.21.15.16:0/1673767906 learned_addr learned my addr 172.21.15.16:0/1673767906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4ae5d9b640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ae8108140 0x7f4ae819b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4ae6d9d640 1 -- 172.21.15.16:0/1673767906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ae8108140 msgr2=0x7f4ae819b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4ae6d9d640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ae8108140 0x7f4ae819b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4ae6d9d640 1 -- 172.21.15.16:0/1673767906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae8102800 msgr2=0x7f4ae819b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4ae6d9d640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae8102800 0x7f4ae819b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4ae6d9d640 1 -- 172.21.15.16:0/1673767906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae81a6350 con 0x7f4ae8108a90 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4ae659c640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae8102800 0x7f4ae819b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4ae6d9d640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ae8108a90 0x7f4ae819bb40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4ad800eaa0 tx=0x7f4ad800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4adf7fe640 1 -- 172.21.15.16:0/1673767906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad800cdb0 con 0x7f4ae8108a90 2024-08-28T19:19:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4aec8cc640 1 -- 172.21.15.16:0/1673767906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae81a6640 con 0x7f4ae8108a90 2024-08-28T19:19:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4adf7fe640 1 -- 172.21.15.16:0/1673767906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ad8005020 con 0x7f4ae8108a90 2024-08-28T19:19:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4adf7fe640 1 -- 172.21.15.16:0/1673767906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad80116d0 con 0x7f4ae8108a90 2024-08-28T19:19:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.448+0000 7f4aec8cc640 1 -- 172.21.15.16:0/1673767906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae81a6b30 con 0x7f4ae8108a90 2024-08-28T19:19:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.449+0000 7f4aec8cc640 1 -- 172.21.15.16:0/1673767906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ae8102c10 con 0x7f4ae8108a90 2024-08-28T19:19:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.451+0000 7f4adf7fe640 1 -- 172.21.15.16:0/1673767906 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f4ad80051d0 con 0x7f4ae8108a90 2024-08-28T19:19:53.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.452+0000 7f4adf7fe640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ad40b4c40 0x7f4ad40b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:53.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.452+0000 7f4ae659c640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ad40b4c40 0x7f4ad40b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:53.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.452+0000 7f4adf7fe640 1 -- 172.21.15.16:0/1673767906 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f4ad80e5b90 con 0x7f4ae8108a90 2024-08-28T19:19:53.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.453+0000 7f4ae659c640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ad40b4c40 0x7f4ad40b7150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0000c00 tx=0x7f4ae0031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:53.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.456+0000 7f4adf7fe640 1 -- 172.21.15.16:0/1673767906 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f4ad8016030 con 0x7f4ae8108a90 2024-08-28T19:19:53.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.650+0000 7f4aec8cc640 1 -- 172.21.15.16:0/1673767906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f4ae802e430 con 0x7f4ae8108a90 2024-08-28T19:19:53.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.650+0000 7f4adf7fe640 1 -- 172.21.15.16:0/1673767906 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f4ad80ac0f0 con 0x7f4ae8108a90 2024-08-28T19:19:53.653 INFO:teuthology.orchestra.run.smithi016.stdout:292057776157 2024-08-28T19:19:53.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 -- 172.21.15.16:0/1673767906 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ad40b4c40 msgr2=0x7f4ad40b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:53.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ad40b4c40 0x7f4ad40b7150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0000c00 tx=0x7f4ae0031400 comp rx=0 tx=0).stop 2024-08-28T19:19:53.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 -- 172.21.15.16:0/1673767906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ae8108a90 msgr2=0x7f4ae819bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:53.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ae8108a90 0x7f4ae819bb40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4ad800eaa0 tx=0x7f4ad800ef60 comp rx=0 tx=0).stop 2024-08-28T19:19:53.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 -- 172.21.15.16:0/1673767906 shutdown_connections 2024-08-28T19:19:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ae8108a90 0x7f4ae819bb40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ae8108140 0x7f4ae819b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ad40b4c40 0x7f4ad40b7150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 --2- 172.21.15.16:0/1673767906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae8102800 0x7f4ae819b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 -- 172.21.15.16:0/1673767906 >> 172.21.15.16:0/1673767906 conn(0x7f4ae80794d0 msgr2=0x7f4ae81012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 -- 172.21.15.16:0/1673767906 shutdown_connections 2024-08-28T19:19:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.653+0000 7f4aec8cc640 1 -- 172.21.15.16:0/1673767906 wait complete. 2024-08-28T19:19:54.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:53 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/4005768218' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-28T19:19:54.248 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:53 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1673767906' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-28T19:19:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:53 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/4005768218' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-28T19:19:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:53 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1673767906' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-28T19:19:54.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:53 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/4005768218' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-28T19:19:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:53 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1673767906' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-28T19:19:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.122+0000 7f61f821f640 1 -- 172.21.15.16:0/2295846665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61f0104790 msgr2=0x7f61f0104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.122+0000 7f61f821f640 1 --2- 172.21.15.16:0/2295846665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61f0104790 0x7f61f0104ba0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f61ec009900 tx=0x7f61ec02a3f0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.122+0000 7f61f821f640 1 -- 172.21.15.16:0/2295846665 shutdown_connections 2024-08-28T19:19:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.122+0000 7f61f821f640 1 --2- 172.21.15.16:0/2295846665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61f010dbf0 0x7f61f01100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.122+0000 7f61f821f640 1 --2- 172.21.15.16:0/2295846665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f01050e0 0x7f61f010d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.122+0000 7f61f821f640 1 --2- 172.21.15.16:0/2295846665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61f0104790 0x7f61f0104ba0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.122+0000 7f61f821f640 1 -- 172.21.15.16:0/2295846665 >> 172.21.15.16:0/2295846665 conn(0x7f61f00794f0 msgr2=0x7f61f0079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.122+0000 7f61f821f640 1 -- 172.21.15.16:0/2295846665 shutdown_connections 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.122+0000 7f61f821f640 1 -- 172.21.15.16:0/2295846665 wait complete. 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.122+0000 7f61f821f640 1 Processor -- start 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f821f640 1 -- start start 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f821f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f0104790 0x7f61f019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f821f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61f01050e0 0x7f61f019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f821f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61f010dbf0 0x7f61f019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f821f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0113850 con 0x7f61f01050e0 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f821f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61f01136d0 con 0x7f61f0104790 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f821f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f61f01139d0 con 0x7f61f010dbf0 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f57ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61f01050e0 0x7f61f019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f57ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61f01050e0 0x7f61f019a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56240/0 (socket says 172.21.15.16:56240) 2024-08-28T19:19:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f57ba640 1 -- 172.21.15.16:0/3663733743 learned_addr learned my addr 172.21.15.16:0/3663733743 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f5fbb640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f0104790 0x7f61f019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f67bc640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61f010dbf0 0x7f61f019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f57ba640 1 -- 172.21.15.16:0/3663733743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61f010dbf0 msgr2=0x7f61f019faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f57ba640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61f010dbf0 0x7f61f019faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f57ba640 1 -- 172.21.15.16:0/3663733743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f0104790 msgr2=0x7f61f019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f57ba640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f0104790 0x7f61f019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.458 INFO:teuthology.orchestra.run.smithi016.stdout:193273528369 2024-08-28T19:19:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f57ba640 1 -- 172.21.15.16:0/3663733743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f01a0110 con 0x7f61f01050e0 2024-08-28T19:19:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f5fbb640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f0104790 0x7f61f019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f67bc640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61f010dbf0 0x7f61f019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:19:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.123+0000 7f61f57ba640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61f01050e0 0x7f61f019a800 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f61e000c8c0 tx=0x7f61e000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.124+0000 7f61e6ffd640 1 -- 172.21.15.16:0/3663733743 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e0007bb0 con 0x7f61f01050e0 2024-08-28T19:19:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.124+0000 7f61e6ffd640 1 -- 172.21.15.16:0/3663733743 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61e0007d50 con 0x7f61f01050e0 2024-08-28T19:19:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.124+0000 7f61f821f640 1 -- 172.21.15.16:0/3663733743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f01a0400 con 0x7f61f01050e0 2024-08-28T19:19:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.124+0000 7f61e6ffd640 1 -- 172.21.15.16:0/3663733743 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e00187c0 con 0x7f61f01050e0 2024-08-28T19:19:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.124+0000 7f61f821f640 1 -- 172.21.15.16:0/3663733743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f01a6c20 con 0x7f61f01050e0 2024-08-28T19:19:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.125+0000 7f61f821f640 1 -- 172.21.15.16:0/3663733743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f0104ba0 con 0x7f61f01050e0 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.126+0000 7f61e6ffd640 1 -- 172.21.15.16:0/3663733743 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f61e000fa20 con 0x7f61f01050e0 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.127+0000 7f61e6ffd640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61dc0b4c40 0x7f61dc0b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.127+0000 7f61f5fbb640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61dc0b4c40 0x7f61dc0b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.127+0000 7f61e6ffd640 1 -- 172.21.15.16:0/3663733743 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f61e00e5b50 con 0x7f61f01050e0 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.128+0000 7f61f5fbb640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61dc0b4c40 0x7f61dc0b7150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0096c0 tx=0x7f61ec031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.130+0000 7f61e6ffd640 1 -- 172.21.15.16:0/3663733743 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f61e0014030 con 0x7f61f01050e0 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.337+0000 7f61f821f640 1 -- 172.21.15.16:0/3663733743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f61f019b390 con 0x7f61f01050e0 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.337+0000 7f61e6ffd640 1 -- 172.21.15.16:0/3663733743 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f61f019b390 con 0x7f61f01050e0 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 -- 172.21.15.16:0/3663733743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61dc0b4c40 msgr2=0x7f61dc0b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61dc0b4c40 0x7f61dc0b7150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0096c0 tx=0x7f61ec031400 comp rx=0 tx=0).stop 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 -- 172.21.15.16:0/3663733743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61f01050e0 msgr2=0x7f61f019a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61f01050e0 0x7f61f019a800 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f61e000c8c0 tx=0x7f61e000cd80 comp rx=0 tx=0).stop 2024-08-28T19:19:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 -- 172.21.15.16:0/3663733743 shutdown_connections 2024-08-28T19:19:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61f010dbf0 0x7f61f019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61f01050e0 0x7f61f019a800 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61dc0b4c40 0x7f61dc0b7150 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 --2- 172.21.15.16:0/3663733743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f0104790 0x7f61f019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 -- 172.21.15.16:0/3663733743 >> 172.21.15.16:0/3663733743 conn(0x7f61f00794f0 msgr2=0x7f61f01076f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 -- 172.21.15.16:0/3663733743 shutdown_connections 2024-08-28T19:19:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:54.339+0000 7f61f821f640 1 -- 172.21.15.16:0/3663733743 wait complete. 2024-08-28T19:19:54.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.349+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4172959167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8108ab0 msgr2=0x7ff7f810f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:54.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.349+0000 7ff7fe8a9640 1 --2- 172.21.15.16:0/4172959167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8108ab0 0x7ff7f810f2e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009ca0 tx=0x7ff7f002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.350+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4172959167 shutdown_connections 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.350+0000 7ff7fe8a9640 1 --2- 172.21.15.16:0/4172959167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8108ab0 0x7ff7f810f2e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.350+0000 7ff7fe8a9640 1 --2- 172.21.15.16:0/4172959167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7f8108160 0x7ff7f8108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.350+0000 7ff7fe8a9640 1 --2- 172.21.15.16:0/4172959167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f8102820 0x7ff7f8102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.350+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4172959167 >> 172.21.15.16:0/4172959167 conn(0x7ff7f80794e0 msgr2=0x7ff7f80798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.350+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4172959167 shutdown_connections 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.350+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4172959167 wait complete. 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.350+0000 7ff7fe8a9640 1 Processor -- start 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.350+0000 7ff7fe8a9640 1 -- start start 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7fe8a9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7f8102820 0x7ff7f819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7fe8a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8108160 0x7ff7f819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7fe8a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f8108ab0 0x7ff7f819fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7fe8a9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f8112730 con 0x7ff7f8108160 2024-08-28T19:19:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7fe8a9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f81125b0 con 0x7ff7f8108ab0 2024-08-28T19:19:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7fe8a9640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f81128b0 con 0x7ff7f8102820 2024-08-28T19:19:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8108160 0x7ff7f819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8108160 0x7ff7f819a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56212/0 (socket says 172.21.15.16:56212) 2024-08-28T19:19:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7f77fe640 1 -- 172.21.15.16:0/4005768218 learned_addr learned my addr 172.21.15.16:0/4005768218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7f7fff640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7f8102820 0x7ff7f819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7fce46640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f8108ab0 0x7ff7f819fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7f77fe640 1 -- 172.21.15.16:0/4005768218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7f8102820 msgr2=0x7ff7f819a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7f77fe640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7f8102820 0x7ff7f819a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7f77fe640 1 -- 172.21.15.16:0/4005768218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f8108ab0 msgr2=0x7ff7f819fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7f77fe640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f8108ab0 0x7ff7f819fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7f77fe640 1 -- 172.21.15.16:0/4005768218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f81a0140 con 0x7ff7f8108160 2024-08-28T19:19:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.351+0000 7ff7fce46640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f8108ab0 0x7ff7f819fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.352+0000 7ff7f77fe640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8108160 0x7ff7f819a770 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00d740 tx=0x7ff7ec00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:54.895 INFO:teuthology.orchestra.run.smithi016.stdout:124554051649 2024-08-28T19:19:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.352+0000 7ff7f57fa640 1 -- 172.21.15.16:0/4005768218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec017070 con 0x7ff7f8108160 2024-08-28T19:19:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.352+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4005768218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f81a0430 con 0x7ff7f8108160 2024-08-28T19:19:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.352+0000 7ff7f57fa640 1 -- 172.21.15.16:0/4005768218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7ec00de50 con 0x7ff7f8108160 2024-08-28T19:19:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.352+0000 7ff7f57fa640 1 -- 172.21.15.16:0/4005768218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec004ce0 con 0x7ff7f8108160 2024-08-28T19:19:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.352+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4005768218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f81a6b90 con 0x7ff7f8108160 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.354+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4005768218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f8103aa0 con 0x7ff7f8108160 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.355+0000 7ff7f57fa640 1 -- 172.21.15.16:0/4005768218 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7ff7ec004e80 con 0x7ff7f8108160 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.356+0000 7ff7f57fa640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7e00b4c10 0x7ff7e00b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.356+0000 7ff7f7fff640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7e00b4c10 0x7ff7e00b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.356+0000 7ff7f57fa640 1 -- 172.21.15.16:0/4005768218 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7ff7ec0e5500 con 0x7ff7f8108160 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.357+0000 7ff7f7fff640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7e00b4c10 0x7ff7e00b7120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800bbe0 tx=0x7ff7e800b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.360+0000 7ff7f57fa640 1 -- 172.21.15.16:0/4005768218 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ff7ec0ac0b0 con 0x7ff7f8108160 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.555+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4005768218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff7f8102c30 con 0x7ff7f8108160 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.556+0000 7ff7f57fa640 1 -- 172.21.15.16:0/4005768218 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff7ec0ab9b0 con 0x7ff7f8108160 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4005768218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7e00b4c10 msgr2=0x7ff7e00b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7e00b4c10 0x7ff7e00b7120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800bbe0 tx=0x7ff7e800b390 comp rx=0 tx=0).stop 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4005768218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8108160 msgr2=0x7ff7f819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8108160 0x7ff7f819a770 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00d740 tx=0x7ff7ec00dc00 comp rx=0 tx=0).stop 2024-08-28T19:19:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4005768218 shutdown_connections 2024-08-28T19:19:54.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff7e00b4c10 0x7ff7e00b7120 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f8108ab0 0x7ff7f819fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8108160 0x7ff7f819a770 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 --2- 172.21.15.16:0/4005768218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7f8102820 0x7ff7f819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:54.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4005768218 >> 172.21.15.16:0/4005768218 conn(0x7ff7f80794e0 msgr2=0x7ff7f806bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:54.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4005768218 shutdown_connections 2024-08-28T19:19:54.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:53.558+0000 7ff7fe8a9640 1 -- 172.21.15.16:0/4005768218 wait complete. 2024-08-28T19:19:55.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:54 smithi016 ceph-mon[18556]: pgmap v303: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:19:55.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:54 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3663733743' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-28T19:19:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:54 smithi028 ceph-mon[21769]: pgmap v303: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:19:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:54 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3663733743' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-28T19:19:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:54 smithi115 ceph-mon[22184]: pgmap v303: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:19:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:54 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3663733743' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-28T19:19:55.541 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528368 got 193273528369 for osd.7 2024-08-28T19:19:55.542 DEBUG:teuthology.parallel:result is None 2024-08-28T19:19:55.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.729+0000 7fc7669e3640 1 -- 172.21.15.16:0/13442613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760103900 msgr2=0x7fc760103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.729+0000 7fc7669e3640 1 --2- 172.21.15.16:0/13442613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760103900 0x7fc760103d10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc750009900 tx=0x7fc75002a410 comp rx=0 tx=0).stop 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.730+0000 7fc7669e3640 1 -- 172.21.15.16:0/13442613 shutdown_connections 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.730+0000 7fc7669e3640 1 --2- 172.21.15.16:0/13442613 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc760109b90 0x7fc7601103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.730+0000 7fc7669e3640 1 --2- 172.21.15.16:0/13442613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc760109240 0x7fc760109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.730+0000 7fc7669e3640 1 --2- 172.21.15.16:0/13442613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760103900 0x7fc760103d10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.730+0000 7fc7669e3640 1 -- 172.21.15.16:0/13442613 >> 172.21.15.16:0/13442613 conn(0x7fc76007b5f0 msgr2=0x7fc76007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.730+0000 7fc7669e3640 1 -- 172.21.15.16:0/13442613 shutdown_connections 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.730+0000 7fc7669e3640 1 -- 172.21.15.16:0/13442613 wait complete. 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.730+0000 7fc7669e3640 1 Processor -- start 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.730+0000 7fc7669e3640 1 -- start start 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc7669e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760103900 0x7fc76019a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc7669e3640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc760109240 0x7fc76019a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc7669e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc760109b90 0x7fc76019fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc7669e3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc760113850 con 0x7fc760103900 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc7669e3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7601136d0 con 0x7fc760109b90 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc7669e3640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc7601139d0 con 0x7fc760109240 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc75ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760103900 0x7fc76019a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc75ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760103900 0x7fc76019a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58066/0 (socket says 172.21.15.16:58066) 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc75ffff640 1 -- 172.21.15.16:0/2515568416 learned_addr learned my addr 172.21.15.16:0/2515568416 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc764f80640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc760109b90 0x7fc76019fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc75f7fe640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc760109240 0x7fc76019a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc75ffff640 1 -- 172.21.15.16:0/2515568416 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc760109240 msgr2=0x7fc76019a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc75ffff640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc760109240 0x7fc76019a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc75ffff640 1 -- 172.21.15.16:0/2515568416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc760109b90 msgr2=0x7fc76019fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc75ffff640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc760109b90 0x7fc76019fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc75ffff640 1 -- 172.21.15.16:0/2515568416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7601a0230 con 0x7fc760103900 2024-08-28T19:19:55.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.731+0000 7fc764f80640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc760109b90 0x7fc76019fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:19:55.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.732+0000 7fc75ffff640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760103900 0x7fc76019a2c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc750005940 tx=0x7fc7500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:55.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.732+0000 7fc75d7fa640 1 -- 172.21.15.16:0/2515568416 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75003f030 con 0x7fc760103900 2024-08-28T19:19:55.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.732+0000 7fc75d7fa640 1 -- 172.21.15.16:0/2515568416 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc750003cb0 con 0x7fc760103900 2024-08-28T19:19:55.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.732+0000 7fc7669e3640 1 -- 172.21.15.16:0/2515568416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7601a04c0 con 0x7fc760103900 2024-08-28T19:19:55.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.732+0000 7fc75d7fa640 1 -- 172.21.15.16:0/2515568416 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc750035070 con 0x7fc760103900 2024-08-28T19:19:55.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.732+0000 7fc7669e3640 1 -- 172.21.15.16:0/2515568416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7601a6c80 con 0x7fc760103900 2024-08-28T19:19:55.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.733+0000 7fc7669e3640 1 -- 172.21.15.16:0/2515568416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc760103d10 con 0x7fc760103900 2024-08-28T19:19:55.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.734+0000 7fc75d7fa640 1 -- 172.21.15.16:0/2515568416 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fc750042030 con 0x7fc760103900 2024-08-28T19:19:55.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.735+0000 7fc75d7fa640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7480b4c10 0x7fc7480b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:55.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.735+0000 7fc75f7fe640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7480b4c10 0x7fc7480b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:55.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.735+0000 7fc75d7fa640 1 -- 172.21.15.16:0/2515568416 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fc750105c50 con 0x7fc760103900 2024-08-28T19:19:55.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.736+0000 7fc75f7fe640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7480b4c10 0x7fc7480b7120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc760108fa0 tx=0x7fc75400b410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:55.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.740+0000 7fc75d7fa640 1 -- 172.21.15.16:0/2515568416 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fc7500cc1b0 con 0x7fc760103900 2024-08-28T19:19:55.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.929+0000 7fc7669e3640 1 -- 172.21.15.16:0/2515568416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc76010b990 con 0x7fc760103900 2024-08-28T19:19:55.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.930+0000 7fc75d7fa640 1 -- 172.21.15.16:0/2515568416 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc7500d0970 con 0x7fc760103900 2024-08-28T19:19:55.933 INFO:teuthology.orchestra.run.smithi016.stdout:146028888122 2024-08-28T19:19:55.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 -- 172.21.15.16:0/2515568416 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7480b4c10 msgr2=0x7fc7480b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:55.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7480b4c10 0x7fc7480b7120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc760108fa0 tx=0x7fc75400b410 comp rx=0 tx=0).stop 2024-08-28T19:19:55.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 -- 172.21.15.16:0/2515568416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760103900 msgr2=0x7fc76019a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:55.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760103900 0x7fc76019a2c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc750005940 tx=0x7fc7500042f0 comp rx=0 tx=0).stop 2024-08-28T19:19:55.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 -- 172.21.15.16:0/2515568416 shutdown_connections 2024-08-28T19:19:55.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7480b4c10 0x7fc7480b7120 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:55.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc760109b90 0x7fc76019fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:55.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc760109240 0x7fc76019a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:55.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 --2- 172.21.15.16:0/2515568416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760103900 0x7fc76019a2c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:55.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 -- 172.21.15.16:0/2515568416 >> 172.21.15.16:0/2515568416 conn(0x7fc76007b5f0 msgr2=0x7fc760107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:55.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 -- 172.21.15.16:0/2515568416 shutdown_connections 2024-08-28T19:19:55.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:55.932+0000 7fc7669e3640 1 -- 172.21.15.16:0/2515568416 wait complete. 2024-08-28T19:19:56.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.016+0000 7f6f003cd640 1 -- 172.21.15.16:0/1114827131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ef8108140 msgr2=0x7f6ef8108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:56.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.016+0000 7f6f003cd640 1 --2- 172.21.15.16:0/1114827131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ef8108140 0x7f6ef8108550 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009900 tx=0x7f6ee802a410 comp rx=0 tx=0).stop 2024-08-28T19:19:56.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.016+0000 7f6f003cd640 1 -- 172.21.15.16:0/1114827131 shutdown_connections 2024-08-28T19:19:56.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.016+0000 7f6f003cd640 1 --2- 172.21.15.16:0/1114827131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ef8108a90 0x7f6ef810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:56.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.016+0000 7f6f003cd640 1 --2- 172.21.15.16:0/1114827131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ef8108140 0x7f6ef8108550 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:56.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.016+0000 7f6f003cd640 1 --2- 172.21.15.16:0/1114827131 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ef8102800 0x7f6ef8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:56.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.016+0000 7f6f003cd640 1 -- 172.21.15.16:0/1114827131 >> 172.21.15.16:0/1114827131 conn(0x7f6ef80794d0 msgr2=0x7f6ef80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:56.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.016+0000 7f6f003cd640 1 -- 172.21.15.16:0/1114827131 shutdown_connections 2024-08-28T19:19:56.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.016+0000 7f6f003cd640 1 -- 172.21.15.16:0/1114827131 wait complete. 2024-08-28T19:19:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6f003cd640 1 Processor -- start 2024-08-28T19:19:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6f003cd640 1 -- start start 2024-08-28T19:19:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6f003cd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ef8102800 0x7f6ef819a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6f003cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ef8108140 0x7f6ef819a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6f003cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ef8108a90 0x7f6ef819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6f003cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef81126e0 con 0x7f6ef8108140 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6f003cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef8112560 con 0x7f6ef8108a90 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6f003cd640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef8112860 con 0x7f6ef8102800 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6efd968640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ef8108140 0x7f6ef819a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6efd968640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ef8108140 0x7f6ef819a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58084/0 (socket says 172.21.15.16:58084) 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6efd968640 1 -- 172.21.15.16:0/2147073202 learned_addr learned my addr 172.21.15.16:0/2147073202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6efe169640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ef8102800 0x7f6ef819a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.017+0000 7f6efe96a640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ef8108a90 0x7f6ef819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6efd968640 1 -- 172.21.15.16:0/2147073202 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ef8102800 msgr2=0x7f6ef819a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6efd968640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ef8102800 0x7f6ef819a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6efd968640 1 -- 172.21.15.16:0/2147073202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ef8108a90 msgr2=0x7f6ef819fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6efd968640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ef8108a90 0x7f6ef819fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6efd968640 1 -- 172.21.15.16:0/2147073202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef81a0230 con 0x7f6ef8108140 2024-08-28T19:19:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6efe96a640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ef8108a90 0x7f6ef819fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6efd968640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ef8108140 0x7f6ef819a860 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8006010 tx=0x7f6ee8004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6eef7fe640 1 -- 172.21.15.16:0/2147073202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee803f030 con 0x7f6ef8108140 2024-08-28T19:19:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6eef7fe640 1 -- 172.21.15.16:0/2147073202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ee8003aa0 con 0x7f6ef8108140 2024-08-28T19:19:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6f003cd640 1 -- 172.21.15.16:0/2147073202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef81a04c0 con 0x7f6ef8108140 2024-08-28T19:19:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6eef7fe640 1 -- 172.21.15.16:0/2147073202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8035070 con 0x7f6ef8108140 2024-08-28T19:19:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.018+0000 7f6f003cd640 1 -- 172.21.15.16:0/2147073202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef81a6c80 con 0x7f6ef8108140 2024-08-28T19:19:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.019+0000 7f6f003cd640 1 -- 172.21.15.16:0/2147073202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef8102c10 con 0x7f6ef8108140 2024-08-28T19:19:56.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.022+0000 7f6eef7fe640 1 -- 172.21.15.16:0/2147073202 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f6ee8042030 con 0x7f6ef8108140 2024-08-28T19:19:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.022+0000 7f6eef7fe640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ee40b4c90 0x7f6ee40b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.023+0000 7f6efe169640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ee40b4c90 0x7f6ee40b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.023+0000 7f6eef7fe640 1 -- 172.21.15.16:0/2147073202 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f6ee8105f50 con 0x7f6ef8108140 2024-08-28T19:19:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.023+0000 7f6efe169640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ee40b4c90 0x7f6ee40b71a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4005fd0 tx=0x7f6ef400b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:56.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.026+0000 7f6eef7fe640 1 -- 172.21.15.16:0/2147073202 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f6ee80ccb00 con 0x7f6ef8108140 2024-08-28T19:19:56.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.211+0000 7f6f003cd640 1 -- 172.21.15.16:0/2147073202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f6ef810b8f0 con 0x7f6ef8108140 2024-08-28T19:19:56.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:55 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2515568416' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-28T19:19:56.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.212+0000 7f6eef7fe640 1 -- 172.21.15.16:0/2147073202 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6ee80cc400 con 0x7f6ef8108140 2024-08-28T19:19:56.215 INFO:teuthology.orchestra.run.smithi016.stdout:219043332141 2024-08-28T19:19:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 -- 172.21.15.16:0/2147073202 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ee40b4c90 msgr2=0x7f6ee40b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ee40b4c90 0x7f6ee40b71a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4005fd0 tx=0x7f6ef400b390 comp rx=0 tx=0).stop 2024-08-28T19:19:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 -- 172.21.15.16:0/2147073202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ef8108140 msgr2=0x7f6ef819a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ef8108140 0x7f6ef819a860 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8006010 tx=0x7f6ee8004060 comp rx=0 tx=0).stop 2024-08-28T19:19:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 -- 172.21.15.16:0/2147073202 shutdown_connections 2024-08-28T19:19:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ee40b4c90 0x7f6ee40b71a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ef8108a90 0x7f6ef819fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ef8108140 0x7f6ef819a860 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 --2- 172.21.15.16:0/2147073202 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ef8102800 0x7f6ef819a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 -- 172.21.15.16:0/2147073202 >> 172.21.15.16:0/2147073202 conn(0x7f6ef80794d0 msgr2=0x7f6ef806b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 -- 172.21.15.16:0/2147073202 shutdown_connections 2024-08-28T19:19:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:56.215+0000 7f6f003cd640 1 -- 172.21.15.16:0/2147073202 wait complete. 2024-08-28T19:19:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:55 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2515568416' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-28T19:19:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:55 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2515568416' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-28T19:19:57.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.067+0000 7f8e392e8640 1 -- 172.21.15.16:0/4265409374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3410dbe0 msgr2=0x7f8e341100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:57.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.067+0000 7f8e392e8640 1 --2- 172.21.15.16:0/4265409374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3410dbe0 0x7f8e341100b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c009900 tx=0x7f8e2c02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:57.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.067+0000 7f8e392e8640 1 -- 172.21.15.16:0/4265409374 shutdown_connections 2024-08-28T19:19:57.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.067+0000 7f8e392e8640 1 --2- 172.21.15.16:0/4265409374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3410dbe0 0x7f8e341100b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.067+0000 7f8e392e8640 1 --2- 172.21.15.16:0/4265409374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e34105060 0x7f8e3410d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.067+0000 7f8e392e8640 1 --2- 172.21.15.16:0/4265409374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34104630 0x7f8e34104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.067+0000 7f8e392e8640 1 -- 172.21.15.16:0/4265409374 >> 172.21.15.16:0/4265409374 conn(0x7f8e340794d0 msgr2=0x7f8e340798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:57.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.067+0000 7f8e392e8640 1 -- 172.21.15.16:0/4265409374 shutdown_connections 2024-08-28T19:19:57.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.067+0000 7f8e392e8640 1 -- 172.21.15.16:0/4265409374 wait complete. 2024-08-28T19:19:57.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e392e8640 1 Processor -- start 2024-08-28T19:19:57.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e392e8640 1 -- start start 2024-08-28T19:19:57.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e392e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34104630 0x7f8e3419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:57.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e392e8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e34105060 0x7f8e3419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:57.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e392e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3410dbe0 0x7f8e3419f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:57.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e392e8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e34113820 con 0x7f8e3410dbe0 2024-08-28T19:19:57.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e392e8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e341136a0 con 0x7f8e34104630 2024-08-28T19:19:57.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e392e8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8e341139a0 con 0x7f8e34105060 2024-08-28T19:19:57.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3410dbe0 0x7f8e3419f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:57.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3410dbe0 0x7f8e3419f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58094/0 (socket says 172.21.15.16:58094) 2024-08-28T19:19:57.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e337fe640 1 -- 172.21.15.16:0/2770559735 learned_addr learned my addr 172.21.15.16:0/2770559735 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:57.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.068+0000 7f8e32ffd640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34104630 0x7f8e3419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:57.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e327fc640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e34105060 0x7f8e3419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:57.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e337fe640 1 -- 172.21.15.16:0/2770559735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e34105060 msgr2=0x7f8e3419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e337fe640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e34105060 0x7f8e3419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e337fe640 1 -- 172.21.15.16:0/2770559735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34104630 msgr2=0x7f8e3419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e337fe640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34104630 0x7f8e3419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e337fe640 1 -- 172.21.15.16:0/2770559735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e341a0120 con 0x7f8e3410dbe0 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e327fc640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e34105060 0x7f8e3419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e337fe640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3410dbe0 0x7f8e3419f9f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c004770 tx=0x7f8e2c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e2bfff640 1 -- 172.21.15.16:0/2770559735 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c03f030 con 0x7f8e3410dbe0 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e2bfff640 1 -- 172.21.15.16:0/2770559735 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e2c003aa0 con 0x7f8e3410dbe0 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e392e8640 1 -- 172.21.15.16:0/2770559735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e341a03b0 con 0x7f8e3410dbe0 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e2bfff640 1 -- 172.21.15.16:0/2770559735 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c035070 con 0x7f8e3410dbe0 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.069+0000 7f8e392e8640 1 -- 172.21.15.16:0/2770559735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e341a6b70 con 0x7f8e3410dbe0 2024-08-28T19:19:57.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.070+0000 7f8e392e8640 1 -- 172.21.15.16:0/2770559735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e3406bee0 con 0x7f8e3410dbe0 2024-08-28T19:19:57.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.072+0000 7f8e2bfff640 1 -- 172.21.15.16:0/2770559735 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f8e2c042030 con 0x7f8e3410dbe0 2024-08-28T19:19:57.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.072+0000 7f8e2bfff640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e200b4c10 0x7f8e200b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:57.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.073+0000 7f8e32ffd640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e200b4c10 0x7f8e200b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:57.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.073+0000 7f8e2bfff640 1 -- 172.21.15.16:0/2770559735 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f8e2c106050 con 0x7f8e3410dbe0 2024-08-28T19:19:57.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.073+0000 7f8e32ffd640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e200b4c10 0x7f8e200b7120 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8e3406c5e0 tx=0x7f8e24009670 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:57.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.076+0000 7f8e2bfff640 1 -- 172.21.15.16:0/2770559735 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8e2c0ccc00 con 0x7f8e3410dbe0 2024-08-28T19:19:57.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:56 smithi016 ceph-mon[18556]: pgmap v304: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-08-28T19:19:57.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:56 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2147073202' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-28T19:19:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.265+0000 7f8e392e8640 1 -- 172.21.15.16:0/2770559735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f8e341a6d40 con 0x7f8e3410dbe0 2024-08-28T19:19:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.265+0000 7f8e2bfff640 1 -- 172.21.15.16:0/2770559735 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8e2c0cc500 con 0x7f8e3410dbe0 2024-08-28T19:19:57.268 INFO:teuthology.orchestra.run.smithi016.stdout:103079215174 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.268+0000 7f8e392e8640 1 -- 172.21.15.16:0/2770559735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e200b4c10 msgr2=0x7f8e200b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.268+0000 7f8e392e8640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e200b4c10 0x7f8e200b7120 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8e3406c5e0 tx=0x7f8e24009670 comp rx=0 tx=0).stop 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.268+0000 7f8e392e8640 1 -- 172.21.15.16:0/2770559735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3410dbe0 msgr2=0x7f8e3419f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.268+0000 7f8e392e8640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3410dbe0 0x7f8e3419f9f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c004770 tx=0x7f8e2c004270 comp rx=0 tx=0).stop 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.268+0000 7f8e392e8640 1 -- 172.21.15.16:0/2770559735 shutdown_connections 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.268+0000 7f8e392e8640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3410dbe0 0x7f8e3419f9f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.268+0000 7f8e392e8640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e34105060 0x7f8e3419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.268+0000 7f8e392e8640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e200b4c10 0x7f8e200b7120 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.268+0000 7f8e392e8640 1 --2- 172.21.15.16:0/2770559735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34104630 0x7f8e3419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.268+0000 7f8e392e8640 1 -- 172.21.15.16:0/2770559735 >> 172.21.15.16:0/2770559735 conn(0x7f8e340794d0 msgr2=0x7f8e34107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.268+0000 7f8e392e8640 1 -- 172.21.15.16:0/2770559735 shutdown_connections 2024-08-28T19:19:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.269+0000 7f8e392e8640 1 -- 172.21.15.16:0/2770559735 wait complete. 2024-08-28T19:19:57.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.269+0000 7f0e029e0640 1 -- 172.21.15.16:0/3275980330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dfc102800 msgr2=0x7f0dfc102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:57.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.269+0000 7f0e029e0640 1 --2- 172.21.15.16:0/3275980330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dfc102800 0x7f0dfc102c10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009900 tx=0x7f0dec02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:57.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.269+0000 7f0e029e0640 1 -- 172.21.15.16:0/3275980330 shutdown_connections 2024-08-28T19:19:57.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.269+0000 7f0e029e0640 1 --2- 172.21.15.16:0/3275980330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0dfc108a90 0x7f0dfc10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.269+0000 7f0e029e0640 1 --2- 172.21.15.16:0/3275980330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dfc108140 0x7f0dfc108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.269+0000 7f0e029e0640 1 --2- 172.21.15.16:0/3275980330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dfc102800 0x7f0dfc102c10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.269+0000 7f0e029e0640 1 -- 172.21.15.16:0/3275980330 >> 172.21.15.16:0/3275980330 conn(0x7f0dfc0794d0 msgr2=0x7f0dfc0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:57.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.269+0000 7f0e029e0640 1 -- 172.21.15.16:0/3275980330 shutdown_connections 2024-08-28T19:19:57.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.269+0000 7f0e029e0640 1 -- 172.21.15.16:0/3275980330 wait complete. 2024-08-28T19:19:57.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.270+0000 7f0e029e0640 1 Processor -- start 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.270+0000 7f0e029e0640 1 -- start start 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.270+0000 7f0e029e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dfc102800 0x7f0dfc19b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.270+0000 7f0e029e0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0dfc108140 0x7f0dfc19b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.270+0000 7f0e029e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dfc108a90 0x7f0dfc19ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.270+0000 7f0e029e0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc112710 con 0x7f0dfc102800 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.270+0000 7f0e029e0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc112590 con 0x7f0dfc108a90 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.270+0000 7f0e029e0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc112890 con 0x7f0dfc108140 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.270+0000 7f0dfbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dfc102800 0x7f0dfc19b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.270+0000 7f0dfbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dfc102800 0x7f0dfc19b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58114/0 (socket says 172.21.15.16:58114) 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.270+0000 7f0dfbfff640 1 -- 172.21.15.16:0/2524939912 learned_addr learned my addr 172.21.15.16:0/2524939912 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0e00f7d640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dfc108a90 0x7f0dfc19ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0dfb7fe640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0dfc108140 0x7f0dfc19b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0dfbfff640 1 -- 172.21.15.16:0/2524939912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0dfc108140 msgr2=0x7f0dfc19b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0dfbfff640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0dfc108140 0x7f0dfc19b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0dfbfff640 1 -- 172.21.15.16:0/2524939912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dfc108a90 msgr2=0x7f0dfc19ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0dfbfff640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dfc108a90 0x7f0dfc19ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0dfbfff640 1 -- 172.21.15.16:0/2524939912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc1a6290 con 0x7f0dfc102800 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0dfbfff640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dfc102800 0x7f0dfc19b000 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0dec005940 tx=0x7f0dec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0df97fa640 1 -- 172.21.15.16:0/2524939912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec03f030 con 0x7f0dfc102800 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0df97fa640 1 -- 172.21.15.16:0/2524939912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dec003cb0 con 0x7f0dfc102800 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0e029e0640 1 -- 172.21.15.16:0/2524939912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc1a6520 con 0x7f0dfc102800 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.271+0000 7f0df97fa640 1 -- 172.21.15.16:0/2524939912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec035070 con 0x7f0dfc102800 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.272+0000 7f0e029e0640 1 -- 172.21.15.16:0/2524939912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc1a6a50 con 0x7f0dfc102800 2024-08-28T19:19:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.273+0000 7f0e029e0640 1 -- 172.21.15.16:0/2524939912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dfc102c10 con 0x7f0dfc102800 2024-08-28T19:19:57.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.276+0000 7f0df97fa640 1 -- 172.21.15.16:0/2524939912 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f0dec042030 con 0x7f0dfc102800 2024-08-28T19:19:57.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.277+0000 7f0df97fa640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0de40b4cc0 0x7f0de40b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:57.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.277+0000 7f0dfb7fe640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0de40b4cc0 0x7f0de40b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:57.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.277+0000 7f0df97fa640 1 -- 172.21.15.16:0/2524939912 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f0dec105d90 con 0x7f0dfc102800 2024-08-28T19:19:57.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.278+0000 7f0dfb7fe640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0de40b4cc0 0x7f0de40b71d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc100910 tx=0x7f0df000b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:57.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.280+0000 7f0df97fa640 1 -- 172.21.15.16:0/2524939912 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f0dec0cc2f0 con 0x7f0dfc102800 2024-08-28T19:19:57.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:56 smithi028 ceph-mon[21769]: pgmap v304: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-08-28T19:19:57.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:56 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2147073202' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-28T19:19:57.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:56 smithi115 ceph-mon[22184]: pgmap v304: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-08-28T19:19:57.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:56 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2147073202' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-28T19:19:57.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.472+0000 7f0e029e0640 1 -- 172.21.15.16:0/2524939912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0dfc02e430 con 0x7f0dfc102800 2024-08-28T19:19:57.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.473+0000 7f0df97fa640 1 -- 172.21.15.16:0/2524939912 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0dec0d0ab0 con 0x7f0dfc102800 2024-08-28T19:19:57.475 INFO:teuthology.orchestra.run.smithi016.stdout:55834574928 2024-08-28T19:19:57.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 -- 172.21.15.16:0/2524939912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0de40b4cc0 msgr2=0x7f0de40b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0de40b4cc0 0x7f0de40b71d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc100910 tx=0x7f0df000b390 comp rx=0 tx=0).stop 2024-08-28T19:19:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 -- 172.21.15.16:0/2524939912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dfc102800 msgr2=0x7f0dfc19b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dfc102800 0x7f0dfc19b000 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0dec005940 tx=0x7f0dec0042f0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 -- 172.21.15.16:0/2524939912 shutdown_connections 2024-08-28T19:19:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0de40b4cc0 0x7f0de40b71d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dfc108a90 0x7f0dfc19ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0dfc108140 0x7f0dfc19b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 --2- 172.21.15.16:0/2524939912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dfc102800 0x7f0dfc19b000 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 -- 172.21.15.16:0/2524939912 >> 172.21.15.16:0/2524939912 conn(0x7f0dfc0794d0 msgr2=0x7f0dfc101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 -- 172.21.15.16:0/2524939912 shutdown_connections 2024-08-28T19:19:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:57.475+0000 7f0e029e0640 1 -- 172.21.15.16:0/2524939912 wait complete. 2024-08-28T19:19:58.078 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:57 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2770559735' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-28T19:19:58.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:57 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2524939912' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-28T19:19:58.106 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332139 got 219043332141 for osd.8 2024-08-28T19:19:58.106 DEBUG:teuthology.parallel:result is None 2024-08-28T19:19:58.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:57 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2770559735' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-28T19:19:58.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:57 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2524939912' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-28T19:19:58.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:57 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2770559735' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-28T19:19:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:57 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2524939912' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-28T19:19:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:19:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.774+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/938973263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2c8105010 msgr2=0x7fb2c810d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.774+0000 7fb2cfd9a640 1 --2- 172.21.15.16:0/938973263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2c8105010 0x7fb2c810d650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009900 tx=0x7fb2bc02a410 comp rx=0 tx=0).stop 2024-08-28T19:19:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.774+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/938973263 shutdown_connections 2024-08-28T19:19:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.774+0000 7fb2cfd9a640 1 --2- 172.21.15.16:0/938973263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2c810db90 0x7fb2c81100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.774+0000 7fb2cfd9a640 1 --2- 172.21.15.16:0/938973263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2c8105010 0x7fb2c810d650 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.774+0000 7fb2cfd9a640 1 --2- 172.21.15.16:0/938973263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c81045e0 0x7fb2c81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.774+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/938973263 >> 172.21.15.16:0/938973263 conn(0x7fb2c80794d0 msgr2=0x7fb2c80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.774+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/938973263 shutdown_connections 2024-08-28T19:19:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.774+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/938973263 wait complete. 2024-08-28T19:19:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.775+0000 7fb2cfd9a640 1 Processor -- start 2024-08-28T19:19:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.775+0000 7fb2cfd9a640 1 -- start start 2024-08-28T19:19:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.775+0000 7fb2cfd9a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2c81045e0 0x7fb2c819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.775+0000 7fb2cfd9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c8105010 0x7fb2c819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.775+0000 7fb2cfd9a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2c810db90 0x7fb2c819fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.775+0000 7fb2cfd9a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81137f0 con 0x7fb2c81045e0 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.775+0000 7fb2cfd9a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c8113670 con 0x7fb2c8105010 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.775+0000 7fb2cfd9a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c8113970 con 0x7fb2c810db90 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2cdb36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2c81045e0 0x7fb2c819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2cdb36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2c81045e0 0x7fb2c819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58144/0 (socket says 172.21.15.16:58144) 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2cdb36640 1 -- 172.21.15.16:0/3409252361 learned_addr learned my addr 172.21.15.16:0/3409252361 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2cd335640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c8105010 0x7fb2c819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2ce337640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2c810db90 0x7fb2c819fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2cdb36640 1 -- 172.21.15.16:0/3409252361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2c810db90 msgr2=0x7fb2c819fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2cdb36640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2c810db90 0x7fb2c819fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2cdb36640 1 -- 172.21.15.16:0/3409252361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c8105010 msgr2=0x7fb2c819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2cdb36640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c8105010 0x7fb2c819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2cdb36640 1 -- 172.21.15.16:0/3409252361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c81a01b0 con 0x7fb2c81045e0 2024-08-28T19:19:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2cdb36640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2c81045e0 0x7fb2c819a240 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400d770 tx=0x7fb2c400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2baffd640 1 -- 172.21.15.16:0/3409252361 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4017070 con 0x7fb2c81045e0 2024-08-28T19:19:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2baffd640 1 -- 172.21.15.16:0/3409252361 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c4009c30 con 0x7fb2c81045e0 2024-08-28T19:19:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.776+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/3409252361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c81a04a0 con 0x7fb2c81045e0 2024-08-28T19:19:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.777+0000 7fb2baffd640 1 -- 172.21.15.16:0/3409252361 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4003cc0 con 0x7fb2c81045e0 2024-08-28T19:19:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.777+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/3409252361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c81a6c00 con 0x7fb2c81045e0 2024-08-28T19:19:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.778+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/3409252361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c81049f0 con 0x7fb2c81045e0 2024-08-28T19:19:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.779+0000 7fb2baffd640 1 -- 172.21.15.16:0/3409252361 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fb2c4003e60 con 0x7fb2c81045e0 2024-08-28T19:19:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.780+0000 7fb2baffd640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2b40b4c10 0x7fb2b40b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.780+0000 7fb2cd335640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2b40b4c10 0x7fb2b40b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.780+0000 7fb2baffd640 1 -- 172.21.15.16:0/3409252361 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fb2c40e5090 con 0x7fb2c81045e0 2024-08-28T19:19:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.780+0000 7fb2cd335640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2b40b4c10 0x7fb2b40b7120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009230 tx=0x7fb2bc031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:58.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.784+0000 7fb2baffd640 1 -- 172.21.15.16:0/3409252361 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fb2c4009de0 con 0x7fb2c81045e0 2024-08-28T19:19:58.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.656+0000 7fb320bef640 1 -- 172.21.15.16:0/2408541857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb31c108aa0 msgr2=0x7fb31c10f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:58.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.656+0000 7fb320bef640 1 --2- 172.21.15.16:0/2408541857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb31c108aa0 0x7fb31c10f2d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb314009ca0 tx=0x7fb31402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.656+0000 7fb320bef640 1 -- 172.21.15.16:0/2408541857 shutdown_connections 2024-08-28T19:19:58.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.656+0000 7fb320bef640 1 --2- 172.21.15.16:0/2408541857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb31c108aa0 0x7fb31c10f2d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.656+0000 7fb320bef640 1 --2- 172.21.15.16:0/2408541857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c108150 0x7fb31c108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.656+0000 7fb320bef640 1 --2- 172.21.15.16:0/2408541857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb31c102810 0x7fb31c102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.656+0000 7fb320bef640 1 -- 172.21.15.16:0/2408541857 >> 172.21.15.16:0/2408541857 conn(0x7fb31c0794b0 msgr2=0x7fb31c0798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.656+0000 7fb320bef640 1 -- 172.21.15.16:0/2408541857 shutdown_connections 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.656+0000 7fb320bef640 1 -- 172.21.15.16:0/2408541857 wait complete. 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.657+0000 7fb320bef640 1 Processor -- start 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.657+0000 7fb320bef640 1 -- start start 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb320bef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c102810 0x7fb31c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb320bef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb31c108150 0x7fb31c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb320bef640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb31c108aa0 0x7fb31c19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb320bef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c112750 con 0x7fb31c108150 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb320bef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c1125d0 con 0x7fb31c102810 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb320bef640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c1128d0 con 0x7fb31c108aa0 2024-08-28T19:19:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb319d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb31c108150 0x7fb31c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb319d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb31c108150 0x7fb31c19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58136/0 (socket says 172.21.15.16:58136) 2024-08-28T19:19:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb319d9b640 1 -- 172.21.15.16:0/2888530254 learned_addr learned my addr 172.21.15.16:0/2888530254 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:19:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb31ad9d640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb31c108aa0 0x7fb31c19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb31a59c640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c102810 0x7fb31c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb319d9b640 1 -- 172.21.15.16:0/2888530254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb31c108aa0 msgr2=0x7fb31c19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb319d9b640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb31c108aa0 0x7fb31c19f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb319d9b640 1 -- 172.21.15.16:0/2888530254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c102810 msgr2=0x7fb31c19a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb319d9b640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c102810 0x7fb31c19a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb319d9b640 1 -- 172.21.15.16:0/2888530254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31c1a0100 con 0x7fb31c108150 2024-08-28T19:19:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb319d9b640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb31c108150 0x7fb31c19a730 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb31000d740 tx=0x7fb31000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:58.896 INFO:teuthology.orchestra.run.smithi016.stdout:171798691896 2024-08-28T19:19:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb30b7fe640 1 -- 172.21.15.16:0/2888530254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310017070 con 0x7fb31c108150 2024-08-28T19:19:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb320bef640 1 -- 172.21.15.16:0/2888530254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb31c1a03f0 con 0x7fb31c108150 2024-08-28T19:19:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.658+0000 7fb30b7fe640 1 -- 172.21.15.16:0/2888530254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb31000de50 con 0x7fb31c108150 2024-08-28T19:19:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.659+0000 7fb30b7fe640 1 -- 172.21.15.16:0/2888530254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31000f620 con 0x7fb31c108150 2024-08-28T19:19:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.659+0000 7fb320bef640 1 -- 172.21.15.16:0/2888530254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb31c1a6b50 con 0x7fb31c108150 2024-08-28T19:19:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.659+0000 7fb320bef640 1 -- 172.21.15.16:0/2888530254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb31c103a90 con 0x7fb31c108150 2024-08-28T19:19:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.661+0000 7fb30b7fe640 1 -- 172.21.15.16:0/2888530254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fb31001a070 con 0x7fb31c108150 2024-08-28T19:19:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.662+0000 7fb30b7fe640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3040b4c10 0x7fb3040b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:19:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.662+0000 7fb31a59c640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3040b4c10 0x7fb3040b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:19:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.662+0000 7fb30b7fe640 1 -- 172.21.15.16:0/2888530254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fb3100e5190 con 0x7fb31c108150 2024-08-28T19:19:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.662+0000 7fb31a59c640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3040b4c10 0x7fb3040b7120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0058c0 tx=0x7fb30c00a4c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.665+0000 7fb30b7fe640 1 -- 172.21.15.16:0/2888530254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fb3100abce0 con 0x7fb31c108150 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.856+0000 7fb320bef640 1 -- 172.21.15.16:0/2888530254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb31c102c20 con 0x7fb31c108150 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.857+0000 7fb30b7fe640 1 -- 172.21.15.16:0/2888530254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb3100b4220 con 0x7fb31c108150 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.860+0000 7fb320bef640 1 -- 172.21.15.16:0/2888530254 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3040b4c10 msgr2=0x7fb3040b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.860+0000 7fb320bef640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3040b4c10 0x7fb3040b7120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0058c0 tx=0x7fb30c00a4c0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.860+0000 7fb320bef640 1 -- 172.21.15.16:0/2888530254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb31c108150 msgr2=0x7fb31c19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.860+0000 7fb320bef640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb31c108150 0x7fb31c19a730 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb31000d740 tx=0x7fb31000dc00 comp rx=0 tx=0).stop 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.860+0000 7fb320bef640 1 -- 172.21.15.16:0/2888530254 shutdown_connections 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.860+0000 7fb320bef640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb31c108aa0 0x7fb31c19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.860+0000 7fb320bef640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb31c108150 0x7fb31c19a730 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.860+0000 7fb320bef640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3040b4c10 0x7fb3040b7120 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.860+0000 7fb320bef640 1 --2- 172.21.15.16:0/2888530254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c102810 0x7fb31c19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.860+0000 7fb320bef640 1 -- 172.21.15.16:0/2888530254 >> 172.21.15.16:0/2888530254 conn(0x7fb31c0794b0 msgr2=0x7fb31c101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.860+0000 7fb320bef640 1 -- 172.21.15.16:0/2888530254 shutdown_connections 2024-08-28T19:19:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.861+0000 7fb320bef640 1 -- 172.21.15.16:0/2888530254 wait complete. 2024-08-28T19:19:58.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.988+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/3409252361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fb2c81a6e60 con 0x7fb2c81045e0 2024-08-28T19:19:58.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.989+0000 7fb2baffd640 1 -- 172.21.15.16:0/3409252361 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb2c40ab590 con 0x7fb2c81045e0 2024-08-28T19:19:58.991 INFO:teuthology.orchestra.run.smithi016.stdout:244813135913 2024-08-28T19:19:58.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/3409252361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2b40b4c10 msgr2=0x7fb2b40b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:58.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2b40b4c10 0x7fb2b40b7120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009230 tx=0x7fb2bc031400 comp rx=0 tx=0).stop 2024-08-28T19:19:58.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/3409252361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2c81045e0 msgr2=0x7fb2c819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:19:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2c81045e0 0x7fb2c819a240 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400d770 tx=0x7fb2c400dc30 comp rx=0 tx=0).stop 2024-08-28T19:19:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/3409252361 shutdown_connections 2024-08-28T19:19:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2c810db90 0x7fb2c819fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2b40b4c10 0x7fb2b40b7120 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c8105010 0x7fb2c819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 --2- 172.21.15.16:0/3409252361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2c81045e0 0x7fb2c819a240 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:19:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/3409252361 >> 172.21.15.16:0/3409252361 conn(0x7fb2c80794d0 msgr2=0x7fb2c8107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:19:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/3409252361 shutdown_connections 2024-08-28T19:19:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:58.991+0000 7fb2cfd9a640 1 -- 172.21.15.16:0/3409252361 wait complete. 2024-08-28T19:19:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:58 smithi016 ceph-mon[18556]: pgmap v305: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-08-28T19:19:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:58 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2888530254' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-28T19:19:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:58 smithi028 ceph-mon[21769]: pgmap v305: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-08-28T19:19:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:58 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2888530254' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-28T19:19:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:58 smithi115 ceph-mon[22184]: pgmap v305: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-08-28T19:19:59.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:58 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2888530254' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-28T19:20:00.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:19:59 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3409252361' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-28T19:20:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:19:59 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3409252361' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-28T19:20:00.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:19:59 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3409252361' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-28T19:20:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.884+0000 7fc32b3cb640 1 -- 172.21.15.16:0/3988868030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc324109bf0 msgr2=0x7fc324110420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.884+0000 7fc32b3cb640 1 --2- 172.21.15.16:0/3988868030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc324109bf0 0x7fc324110420 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc31c009ca0 tx=0x7fc31c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.884+0000 7fc32b3cb640 1 -- 172.21.15.16:0/3988868030 shutdown_connections 2024-08-28T19:20:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.884+0000 7fc32b3cb640 1 --2- 172.21.15.16:0/3988868030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc324109bf0 0x7fc324110420 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.884+0000 7fc32b3cb640 1 --2- 172.21.15.16:0/3988868030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3241092a0 0x7fc3241096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.884+0000 7fc32b3cb640 1 --2- 172.21.15.16:0/3988868030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc324101d60 0x7fc324102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.884+0000 7fc32b3cb640 1 -- 172.21.15.16:0/3988868030 >> 172.21.15.16:0/3988868030 conn(0x7fc324079d30 msgr2=0x7fc32407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.884+0000 7fc32b3cb640 1 -- 172.21.15.16:0/3988868030 shutdown_connections 2024-08-28T19:20:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.884+0000 7fc32b3cb640 1 -- 172.21.15.16:0/3988868030 wait complete. 2024-08-28T19:20:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.885+0000 7fc32b3cb640 1 Processor -- start 2024-08-28T19:20:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.885+0000 7fc32b3cb640 1 -- start start 2024-08-28T19:20:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.885+0000 7fc32b3cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc324101d60 0x7fc32419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.885+0000 7fc32b3cb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3241092a0 0x7fc32419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.885+0000 7fc32b3cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc324109bf0 0x7fc32419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.885+0000 7fc32b3cb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3241138b0 con 0x7fc324109bf0 2024-08-28T19:20:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.885+0000 7fc32b3cb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc324113730 con 0x7fc324101d60 2024-08-28T19:20:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.885+0000 7fc32b3cb640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc324113a30 con 0x7fc3241092a0 2024-08-28T19:20:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.885+0000 7fc329968640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc324109bf0 0x7fc32419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc329968640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc324109bf0 0x7fc32419fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58164/0 (socket says 172.21.15.16:58164) 2024-08-28T19:20:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc329968640 1 -- 172.21.15.16:0/2102807785 learned_addr learned my addr 172.21.15.16:0/2102807785 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:00.429 INFO:teuthology.orchestra.run.smithi016.stdout:38654705750 2024-08-28T19:20:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc329167640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc324101d60 0x7fc32419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc328966640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3241092a0 0x7fc32419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc329968640 1 -- 172.21.15.16:0/2102807785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3241092a0 msgr2=0x7fc32419a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc329968640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3241092a0 0x7fc32419a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc329968640 1 -- 172.21.15.16:0/2102807785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc324101d60 msgr2=0x7fc32419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc329968640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc324101d60 0x7fc32419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc329968640 1 -- 172.21.15.16:0/2102807785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3241a0170 con 0x7fc324109bf0 2024-08-28T19:20:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc329968640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc324109bf0 0x7fc32419fa40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc31c005cc0 tx=0x7fc31c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc31a7fc640 1 -- 172.21.15.16:0/2102807785 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c003bf0 con 0x7fc324109bf0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc32b3cb640 1 -- 172.21.15.16:0/2102807785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3241a0400 con 0x7fc324109bf0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc31a7fc640 1 -- 172.21.15.16:0/2102807785 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc31c003d90 con 0x7fc324109bf0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc31a7fc640 1 -- 172.21.15.16:0/2102807785 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c02fbc0 con 0x7fc324109bf0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.886+0000 7fc32b3cb640 1 -- 172.21.15.16:0/2102807785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3241a6bc0 con 0x7fc324109bf0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.887+0000 7fc32b3cb640 1 -- 172.21.15.16:0/2102807785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc324102170 con 0x7fc324109bf0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.889+0000 7fc31a7fc640 1 -- 172.21.15.16:0/2102807785 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fc31c02fd60 con 0x7fc324109bf0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.890+0000 7fc31a7fc640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3100b4c10 0x7fc3100b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.890+0000 7fc329167640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3100b4c10 0x7fc3100b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.890+0000 7fc31a7fc640 1 -- 172.21.15.16:0/2102807785 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fc31c106d30 con 0x7fc324109bf0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.890+0000 7fc329167640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3100b4c10 0x7fc3100b7120 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc32000bbe0 tx=0x7fc32000b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:19:59.893+0000 7fc31a7fc640 1 -- 172.21.15.16:0/2102807785 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fc31c0cd8e0 con 0x7fc324109bf0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.081+0000 7fc32b3cb640 1 -- 172.21.15.16:0/2102807785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc32419b330 con 0x7fc324109bf0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.082+0000 7fc31a7fc640 1 -- 172.21.15.16:0/2102807785 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc32419b330 con 0x7fc324109bf0 2024-08-28T19:20:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 -- 172.21.15.16:0/2102807785 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3100b4c10 msgr2=0x7fc3100b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3100b4c10 0x7fc3100b7120 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc32000bbe0 tx=0x7fc32000b390 comp rx=0 tx=0).stop 2024-08-28T19:20:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 -- 172.21.15.16:0/2102807785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc324109bf0 msgr2=0x7fc32419fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc324109bf0 0x7fc32419fa40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc31c005cc0 tx=0x7fc31c02f380 comp rx=0 tx=0).stop 2024-08-28T19:20:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 -- 172.21.15.16:0/2102807785 shutdown_connections 2024-08-28T19:20:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc324109bf0 0x7fc32419fa40 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3241092a0 0x7fc32419a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3100b4c10 0x7fc3100b7120 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 --2- 172.21.15.16:0/2102807785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc324101d60 0x7fc32419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 -- 172.21.15.16:0/2102807785 >> 172.21.15.16:0/2102807785 conn(0x7fc324079d30 msgr2=0x7fc324107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 -- 172.21.15.16:0/2102807785 shutdown_connections 2024-08-28T19:20:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:00.084+0000 7fc32b3cb640 1 -- 172.21.15.16:0/2102807785 wait complete. 2024-08-28T19:20:00.809 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691894 got 171798691896 for osd.6 2024-08-28T19:20:00.809 DEBUG:teuthology.parallel:result is None 2024-08-28T19:20:01.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:00 smithi016 ceph-mon[18556]: pgmap v306: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-08-28T19:20:01.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:00 smithi016 ceph-mon[18556]: overall HEALTH_OK 2024-08-28T19:20:01.208 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:00 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2102807785' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-28T19:20:01.235 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705746 got 38654705750 for osd.0 2024-08-28T19:20:01.235 DEBUG:teuthology.parallel:result is None 2024-08-28T19:20:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:00 smithi028 ceph-mon[21769]: pgmap v306: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-08-28T19:20:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:00 smithi028 ceph-mon[21769]: overall HEALTH_OK 2024-08-28T19:20:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:00 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2102807785' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-28T19:20:01.341 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135910 got 244813135913 for osd.9 2024-08-28T19:20:01.341 DEBUG:teuthology.parallel:result is None 2024-08-28T19:20:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:00 smithi115 ceph-mon[22184]: pgmap v306: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-08-28T19:20:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:00 smithi115 ceph-mon[22184]: overall HEALTH_OK 2024-08-28T19:20:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:00 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2102807785' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-28T19:20:01.740 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574926 got 55834574928 for osd.1 2024-08-28T19:20:01.740 DEBUG:teuthology.parallel:result is None 2024-08-28T19:20:02.428 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776157 got 292057776157 for osd.11 2024-08-28T19:20:02.429 DEBUG:teuthology.parallel:result is None 2024-08-28T19:20:02.801 INFO:tasks.cephadm.ceph_manager.ceph:need seq 124554051646 got 124554051649 for osd.4 2024-08-28T19:20:02.801 DEBUG:teuthology.parallel:result is None 2024-08-28T19:20:03.208 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:02 smithi016 ceph-mon[18556]: pgmap v307: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-08-28T19:20:03.240 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888120 got 146028888122 for osd.5 2024-08-28T19:20:03.240 DEBUG:teuthology.parallel:result is None 2024-08-28T19:20:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:02 smithi028 ceph-mon[21769]: pgmap v307: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-08-28T19:20:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:02 smithi115 ceph-mon[22184]: pgmap v307: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-08-28T19:20:03.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.441+0000 7f8efe391640 1 -- 172.21.15.16:0/3614476016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef8104750 msgr2=0x7f8ef8104b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:03.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.441+0000 7f8efe391640 1 --2- 172.21.15.16:0/3614476016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef8104750 0x7f8ef8104b60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009900 tx=0x7f8ee802a410 comp rx=0 tx=0).stop 2024-08-28T19:20:03.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.441+0000 7f8efe391640 1 -- 172.21.15.16:0/3614476016 shutdown_connections 2024-08-28T19:20:03.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.441+0000 7f8efe391640 1 --2- 172.21.15.16:0/3614476016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef810dbb0 0x7f8ef81100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:03.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.441+0000 7f8efe391640 1 --2- 172.21.15.16:0/3614476016 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ef81050a0 0x7f8ef810d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:03.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.441+0000 7f8efe391640 1 --2- 172.21.15.16:0/3614476016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef8104750 0x7f8ef8104b60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:03.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.441+0000 7f8efe391640 1 -- 172.21.15.16:0/3614476016 >> 172.21.15.16:0/3614476016 conn(0x7f8ef80794d0 msgr2=0x7f8ef80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:03.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.441+0000 7f8efe391640 1 -- 172.21.15.16:0/3614476016 shutdown_connections 2024-08-28T19:20:03.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.441+0000 7f8efe391640 1 -- 172.21.15.16:0/3614476016 wait complete. 2024-08-28T19:20:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.442+0000 7f8efe391640 1 Processor -- start 2024-08-28T19:20:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.442+0000 7f8efe391640 1 -- start start 2024-08-28T19:20:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.442+0000 7f8efe391640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ef8104750 0x7f8ef819a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.442+0000 7f8efe391640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef81050a0 0x7f8ef819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.442+0000 7f8efe391640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef810dbb0 0x7f8ef819fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.442+0000 7f8efe391640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef8113810 con 0x7f8ef81050a0 2024-08-28T19:20:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.442+0000 7f8efe391640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef8113690 con 0x7f8ef810dbb0 2024-08-28T19:20:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.442+0000 7f8efe391640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef8113990 con 0x7f8ef8104750 2024-08-28T19:20:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.442+0000 7f8ef77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef81050a0 0x7f8ef819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef81050a0 0x7f8ef819a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58172/0 (socket says 172.21.15.16:58172) 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef77fe640 1 -- 172.21.15.16:0/744452736 learned_addr learned my addr 172.21.15.16:0/744452736 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef7fff640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ef8104750 0x7f8ef819a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8efc92e640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef810dbb0 0x7f8ef819fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef77fe640 1 -- 172.21.15.16:0/744452736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ef8104750 msgr2=0x7f8ef819a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef77fe640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ef8104750 0x7f8ef819a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef77fe640 1 -- 172.21.15.16:0/744452736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef810dbb0 msgr2=0x7f8ef819fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef77fe640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef810dbb0 0x7f8ef819fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef77fe640 1 -- 172.21.15.16:0/744452736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef81a01d0 con 0x7f8ef81050a0 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef77fe640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef81050a0 0x7f8ef819a8c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c8c0 tx=0x7f8eec00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef57fa640 1 -- 172.21.15.16:0/744452736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eec007bb0 con 0x7f8ef81050a0 2024-08-28T19:20:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef57fa640 1 -- 172.21.15.16:0/744452736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8eec007d50 con 0x7f8ef81050a0 2024-08-28T19:20:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8efe391640 1 -- 172.21.15.16:0/744452736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef81a04c0 con 0x7f8ef81050a0 2024-08-28T19:20:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8ef57fa640 1 -- 172.21.15.16:0/744452736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0187c0 con 0x7f8ef81050a0 2024-08-28T19:20:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.443+0000 7f8efe391640 1 -- 172.21.15.16:0/744452736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef81a6ce0 con 0x7f8ef81050a0 2024-08-28T19:20:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.445+0000 7f8efe391640 1 -- 172.21.15.16:0/744452736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ef8104b60 con 0x7f8ef81050a0 2024-08-28T19:20:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.446+0000 7f8ef57fa640 1 -- 172.21.15.16:0/744452736 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f8eec00fa20 con 0x7f8ef81050a0 2024-08-28T19:20:03.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.447+0000 7f8ef57fa640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8ee00b4c40 0x7f8ee00b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:03.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.447+0000 7f8ef57fa640 1 -- 172.21.15.16:0/744452736 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f8eec0e57d0 con 0x7f8ef81050a0 2024-08-28T19:20:03.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.447+0000 7f8ef7fff640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8ee00b4c40 0x7f8ee00b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:03.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.447+0000 7f8ef7fff640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8ee00b4c40 0x7f8ee00b7150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8000c00 tx=0x7f8ee8031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:03.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.451+0000 7f8ef57fa640 1 -- 172.21.15.16:0/744452736 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8eec014030 con 0x7f8ef81050a0 2024-08-28T19:20:03.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.640+0000 7f8efe391640 1 -- 172.21.15.16:0/744452736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8ef8108720 con 0x7f8ef81050a0 2024-08-28T19:20:03.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.640+0000 7f8ef57fa640 1 -- 172.21.15.16:0/744452736 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8eec0ac320 con 0x7f8ef81050a0 2024-08-28T19:20:03.642 INFO:teuthology.orchestra.run.smithi016.stdout:77309411405 2024-08-28T19:20:03.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.642+0000 7f8efe391640 1 -- 172.21.15.16:0/744452736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8ee00b4c40 msgr2=0x7f8ee00b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:03.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.642+0000 7f8efe391640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8ee00b4c40 0x7f8ee00b7150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8000c00 tx=0x7f8ee8031400 comp rx=0 tx=0).stop 2024-08-28T19:20:03.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.642+0000 7f8efe391640 1 -- 172.21.15.16:0/744452736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef81050a0 msgr2=0x7f8ef819a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:03.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.642+0000 7f8efe391640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef81050a0 0x7f8ef819a8c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c8c0 tx=0x7f8eec00cd80 comp rx=0 tx=0).stop 2024-08-28T19:20:03.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.643+0000 7f8efe391640 1 -- 172.21.15.16:0/744452736 shutdown_connections 2024-08-28T19:20:03.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.643+0000 7f8efe391640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8ee00b4c40 0x7f8ee00b7150 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:03.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.643+0000 7f8efe391640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef810dbb0 0x7f8ef819fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:03.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.643+0000 7f8efe391640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef81050a0 0x7f8ef819a8c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:03.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.643+0000 7f8efe391640 1 --2- 172.21.15.16:0/744452736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ef8104750 0x7f8ef819a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:03.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.643+0000 7f8efe391640 1 -- 172.21.15.16:0/744452736 >> 172.21.15.16:0/744452736 conn(0x7f8ef80794d0 msgr2=0x7f8ef81005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:03.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.643+0000 7f8efe391640 1 -- 172.21.15.16:0/744452736 shutdown_connections 2024-08-28T19:20:03.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:03.643+0000 7f8efe391640 1 -- 172.21.15.16:0/744452736 wait complete. 2024-08-28T19:20:03.845 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:03.870 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215171 got 103079215174 for osd.3 2024-08-28T19:20:03.870 DEBUG:teuthology.parallel:result is None 2024-08-28T19:20:04.141 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:03 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/744452736' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-28T19:20:04.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:03 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/744452736' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-28T19:20:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:03 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/744452736' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-28T19:20:04.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.374+0000 7f8bdcc18640 1 -- 172.21.15.16:0/489300950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd8102800 msgr2=0x7f8bd8102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:04.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.374+0000 7f8bdcc18640 1 --2- 172.21.15.16:0/489300950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd8102800 0x7f8bd8102c10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009900 tx=0x7f8bc802a410 comp rx=0 tx=0).stop 2024-08-28T19:20:04.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.374+0000 7f8bdcc18640 1 -- 172.21.15.16:0/489300950 shutdown_connections 2024-08-28T19:20:04.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.374+0000 7f8bdcc18640 1 --2- 172.21.15.16:0/489300950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd8108a90 0x7f8bd810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:04.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.374+0000 7f8bdcc18640 1 --2- 172.21.15.16:0/489300950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd8108140 0x7f8bd8108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:04.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.374+0000 7f8bdcc18640 1 --2- 172.21.15.16:0/489300950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd8102800 0x7f8bd8102c10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:04.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.374+0000 7f8bdcc18640 1 -- 172.21.15.16:0/489300950 >> 172.21.15.16:0/489300950 conn(0x7f8bd80794d0 msgr2=0x7f8bd80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:04.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.374+0000 7f8bdcc18640 1 -- 172.21.15.16:0/489300950 shutdown_connections 2024-08-28T19:20:04.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.374+0000 7f8bdcc18640 1 -- 172.21.15.16:0/489300950 wait complete. 2024-08-28T19:20:04.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bdcc18640 1 Processor -- start 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bdcc18640 1 -- start start 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bdcc18640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd8102800 0x7f8bd819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bdcc18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd8108140 0x7f8bd819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bdcc18640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd8108a90 0x7f8bd819fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bdcc18640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd81126f0 con 0x7f8bd8108a90 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bdcc18640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8112570 con 0x7f8bd8108140 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bdcc18640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8112870 con 0x7f8bd8102800 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bd6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd8108a90 0x7f8bd819fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bd6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd8108a90 0x7f8bd819fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51784/0 (socket says 172.21.15.16:51784) 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bd6d9d640 1 -- 172.21.15.16:0/3143648062 learned_addr learned my addr 172.21.15.16:0/3143648062 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bd5d9b640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd8108140 0x7f8bd819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:04.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bd659c640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd8102800 0x7f8bd819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:04.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bd6d9d640 1 -- 172.21.15.16:0/3143648062 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd8102800 msgr2=0x7f8bd819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:04.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.375+0000 7f8bd6d9d640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd8102800 0x7f8bd819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:04.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.376+0000 7f8bd6d9d640 1 -- 172.21.15.16:0/3143648062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd8108140 msgr2=0x7f8bd819a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:04.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.376+0000 7f8bd6d9d640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd8108140 0x7f8bd819a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:04.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.376+0000 7f8bd6d9d640 1 -- 172.21.15.16:0/3143648062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd81a01c0 con 0x7f8bd8108a90 2024-08-28T19:20:04.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.376+0000 7f8bd6d9d640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd8108a90 0x7f8bd819fa90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8bd000eaa0 tx=0x7f8bd000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:04.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.376+0000 7f8bc77fe640 1 -- 172.21.15.16:0/3143648062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd000cdb0 con 0x7f8bd8108a90 2024-08-28T19:20:04.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.376+0000 7f8bc77fe640 1 -- 172.21.15.16:0/3143648062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bd0005020 con 0x7f8bd8108a90 2024-08-28T19:20:04.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.376+0000 7f8bc77fe640 1 -- 172.21.15.16:0/3143648062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd00116d0 con 0x7f8bd8108a90 2024-08-28T19:20:04.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.376+0000 7f8bdcc18640 1 -- 172.21.15.16:0/3143648062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd81a04b0 con 0x7f8bd8108a90 2024-08-28T19:20:04.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.376+0000 7f8bdcc18640 1 -- 172.21.15.16:0/3143648062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd81a6c10 con 0x7f8bd8108a90 2024-08-28T19:20:04.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.377+0000 7f8bdcc18640 1 -- 172.21.15.16:0/3143648062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd8102c10 con 0x7f8bd8108a90 2024-08-28T19:20:04.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.379+0000 7f8bc77fe640 1 -- 172.21.15.16:0/3143648062 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f8bd00051d0 con 0x7f8bd8108a90 2024-08-28T19:20:04.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.380+0000 7f8bc77fe640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bc00b4c90 0x7f8bc00b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:04.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.380+0000 7f8bd659c640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bc00b4c90 0x7f8bc00b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:04.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.380+0000 7f8bc77fe640 1 -- 172.21.15.16:0/3143648062 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7f8bd00e5c40 con 0x7f8bd8108a90 2024-08-28T19:20:04.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.380+0000 7f8bd659c640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bc00b4c90 0x7f8bc00b71a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8031440 tx=0x7f8bc8005d50 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:04.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.383+0000 7f8bc77fe640 1 -- 172.21.15.16:0/3143648062 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8bd0016030 con 0x7f8bd8108a90 2024-08-28T19:20:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.568+0000 7f8bdcc18640 1 -- 172.21.15.16:0/3143648062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f8bd819b380 con 0x7f8bd8108a90 2024-08-28T19:20:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.568+0000 7f8bc77fe640 1 -- 172.21.15.16:0/3143648062 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f8bd819b380 con 0x7f8bd8108a90 2024-08-28T19:20:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:266287972388 2024-08-28T19:20:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 -- 172.21.15.16:0/3143648062 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bc00b4c90 msgr2=0x7f8bc00b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bc00b4c90 0x7f8bc00b71a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8031440 tx=0x7f8bc8005d50 comp rx=0 tx=0).stop 2024-08-28T19:20:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 -- 172.21.15.16:0/3143648062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd8108a90 msgr2=0x7f8bd819fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd8108a90 0x7f8bd819fa90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8bd000eaa0 tx=0x7f8bd000ef60 comp rx=0 tx=0).stop 2024-08-28T19:20:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 -- 172.21.15.16:0/3143648062 shutdown_connections 2024-08-28T19:20:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd8108a90 0x7f8bd819fa90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8bc00b4c90 0x7f8bc00b71a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd8108140 0x7f8bd819a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 --2- 172.21.15.16:0/3143648062 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd8102800 0x7f8bd819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 -- 172.21.15.16:0/3143648062 >> 172.21.15.16:0/3143648062 conn(0x7f8bd80794d0 msgr2=0x7f8bd81018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 -- 172.21.15.16:0/3143648062 shutdown_connections 2024-08-28T19:20:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:04.570+0000 7f8bdcc18640 1 -- 172.21.15.16:0/3143648062 wait complete. 2024-08-28T19:20:05.019 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411400 got 77309411405 for osd.2 2024-08-28T19:20:05.019 DEBUG:teuthology.parallel:result is None 2024-08-28T19:20:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:04 smithi016 ceph-mon[18556]: pgmap v308: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:04 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3143648062' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-28T19:20:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:04 smithi028 ceph-mon[21769]: pgmap v308: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:04 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3143648062' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-28T19:20:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:04 smithi115 ceph-mon[22184]: pgmap v308: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:04 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3143648062' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-28T19:20:05.778 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972385 got 266287972388 for osd.10 2024-08-28T19:20:05.778 DEBUG:teuthology.parallel:result is None 2024-08-28T19:20:05.778 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-28T19:20:05.779 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-28T19:20:06.041 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.540+0000 7fc48ad11640 1 -- 172.21.15.16:0/3856971421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc484108ab0 msgr2=0x7fc48410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.540+0000 7fc48ad11640 1 --2- 172.21.15.16:0/3856971421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc484108ab0 0x7fc48410f2e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc47c009900 tx=0x7fc47c02a410 comp rx=0 tx=0).stop 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.540+0000 7fc48ad11640 1 -- 172.21.15.16:0/3856971421 shutdown_connections 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.540+0000 7fc48ad11640 1 --2- 172.21.15.16:0/3856971421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc484108ab0 0x7fc48410f2e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.540+0000 7fc48ad11640 1 --2- 172.21.15.16:0/3856971421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc484108160 0x7fc484108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.540+0000 7fc48ad11640 1 --2- 172.21.15.16:0/3856971421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484102820 0x7fc484102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.540+0000 7fc48ad11640 1 -- 172.21.15.16:0/3856971421 >> 172.21.15.16:0/3856971421 conn(0x7fc4840794e0 msgr2=0x7fc4840798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.540+0000 7fc48ad11640 1 -- 172.21.15.16:0/3856971421 shutdown_connections 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.540+0000 7fc48ad11640 1 -- 172.21.15.16:0/3856971421 wait complete. 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.540+0000 7fc48ad11640 1 Processor -- start 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.540+0000 7fc48ad11640 1 -- start start 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc48ad11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484102820 0x7fc484199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc48ad11640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc484108160 0x7fc4841996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc48ad11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc484108ab0 0x7fc48419e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc48ad11640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc484112780 con 0x7fc484108ab0 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc48ad11640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc484112600 con 0x7fc484102820 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc48ad11640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc484112900 con 0x7fc484108160 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc4892ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc484108ab0 0x7fc48419e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc4892ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc484108ab0 0x7fc48419e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51794/0 (socket says 172.21.15.16:51794) 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc4892ae640 1 -- 172.21.15.16:0/2565118283 learned_addr learned my addr 172.21.15.16:0/2565118283 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc488aad640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484102820 0x7fc484199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc483fff640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc484108160 0x7fc4841996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc4892ae640 1 -- 172.21.15.16:0/2565118283 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc484108160 msgr2=0x7fc4841996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc4892ae640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc484108160 0x7fc4841996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc4892ae640 1 -- 172.21.15.16:0/2565118283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484102820 msgr2=0x7fc484199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc4892ae640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484102820 0x7fc484199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc4892ae640 1 -- 172.21.15.16:0/2565118283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc48419f0c0 con 0x7fc484108ab0 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.541+0000 7fc4892ae640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc484108ab0 0x7fc48419e990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc47c02a930 tx=0x7fc47c003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.542+0000 7fc481ffb640 1 -- 172.21.15.16:0/2565118283 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c03f070 con 0x7fc484108ab0 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.542+0000 7fc481ffb640 1 -- 172.21.15.16:0/2565118283 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc47c031d60 con 0x7fc484108ab0 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.542+0000 7fc48ad11640 1 -- 172.21.15.16:0/2565118283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc48419f350 con 0x7fc484108ab0 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.542+0000 7fc481ffb640 1 -- 172.21.15.16:0/2565118283 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c035030 con 0x7fc484108ab0 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.542+0000 7fc48ad11640 1 -- 172.21.15.16:0/2565118283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4841a5b10 con 0x7fc484108ab0 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.543+0000 7fc48ad11640 1 -- 172.21.15.16:0/2565118283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc484103aa0 con 0x7fc484108ab0 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.544+0000 7fc481ffb640 1 -- 172.21.15.16:0/2565118283 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fc47c042070 con 0x7fc484108ab0 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.545+0000 7fc481ffb640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4700b4c10 0x7fc4700b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.545+0000 7fc481ffb640 1 -- 172.21.15.16:0/2565118283 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9286+0+0 (secure 0 0 0) 0x7fc47c106720 con 0x7fc484108ab0 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.545+0000 7fc488aad640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4700b4c10 0x7fc4700b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.546+0000 7fc488aad640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4700b4c10 0x7fc4700b7120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc474005fd0 tx=0x7fc474009870 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:06.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.548+0000 7fc481ffb640 1 -- 172.21.15.16:0/2565118283 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fc47c0cd2d0 con 0x7fc484108ab0 2024-08-28T19:20:06.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.730+0000 7fc48ad11640 1 -- 172.21.15.16:0/2565118283 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc48410a8d0 con 0x7fc4700b4c10 2024-08-28T19:20:06.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.736+0000 7fc481ffb640 1 -- 172.21.15.16:0/2565118283 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340390 (secure 0 0 0) 0x7fc48410a8d0 con 0x7fc4700b4c10 2024-08-28T19:20:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:06.743 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-08-28T19:20:06.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 -- 172.21.15.16:0/2565118283 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4700b4c10 msgr2=0x7fc4700b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:06.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4700b4c10 0x7fc4700b7120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc474005fd0 tx=0x7fc474009870 comp rx=0 tx=0).stop 2024-08-28T19:20:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 -- 172.21.15.16:0/2565118283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc484108ab0 msgr2=0x7fc48419e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc484108ab0 0x7fc48419e990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc47c02a930 tx=0x7fc47c003cd0 comp rx=0 tx=0).stop 2024-08-28T19:20:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 -- 172.21.15.16:0/2565118283 shutdown_connections 2024-08-28T19:20:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc484108ab0 0x7fc48419e990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc484108160 0x7fc4841996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc4700b4c10 0x7fc4700b7120 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 --2- 172.21.15.16:0/2565118283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484102820 0x7fc484199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 -- 172.21.15.16:0/2565118283 >> 172.21.15.16:0/2565118283 conn(0x7fc4840794e0 msgr2=0x7fc484105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 -- 172.21.15.16:0/2565118283 shutdown_connections 2024-08-28T19:20:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:06.740+0000 7fc48ad11640 1 -- 172.21.15.16:0/2565118283 wait complete. 2024-08-28T19:20:07.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:06 smithi016 ceph-mon[18556]: pgmap v309: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:20:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:06 smithi028 ceph-mon[21769]: pgmap v309: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:20:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:06 smithi115 ceph-mon[22184]: pgmap v309: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:20:07.541 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-08-28T19:20:05.479463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661281,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":641,"num_read_kb":428,"num_write":374,"num_write_kb":2294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":308,"ondisk_log_size":308,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":412448,"kb_used_data":13280,"kb_used_omap":18,"kb_used_meta":398957,"kb_avail":1124529376,"statfs":{"total":1151940427776,"available":1151518081024,"internally_reserved":0,"allocated":13598720,"data_stored":7718249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":408532282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003988"},"pg_stats":[{"pgid":"2.1f","version":"82'13","reported_seq":111,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:59.725856+0000","last_change":"2024-08-28T19:19:08.633326+0000","last_active":"2024-08-28T19:19:59.725856+0000","last_peered":"2024-08-28T19:19:59.725856+0000","last_clean":"2024-08-28T19:19:59.725856+0000","last_became_active":"2024-08-28T19:19:08.623876+0000","last_became_peered":"2024-08-28T19:19:08.623876+0000","last_unstale":"2024-08-28T19:19:59.725856+0000","last_undegraded":"2024-08-28T19:19:59.725856+0000","last_fullsized":"2024-08-28T19:19:59.725856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:19:50.632337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632322+0000","last_change":"2024-08-28T19:18:32.492604+0000","last_active":"2024-08-28T19:19:08.632322+0000","last_peered":"2024-08-28T19:19:08.632322+0000","last_clean":"2024-08-28T19:19:08.632322+0000","last_became_active":"2024-08-28T19:18:32.489427+0000","last_became_peered":"2024-08-28T19:18:32.489427+0000","last_unstale":"2024-08-28T19:19:08.632322+0000","last_undegraded":"2024-08-28T19:19:08.632322+0000","last_fullsized":"2024-08-28T19:19:08.632322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:36:14.841315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622224+0000","last_change":"2024-08-28T19:18:39.528815+0000","last_active":"2024-08-28T19:19:08.622224+0000","last_peered":"2024-08-28T19:19:08.622224+0000","last_clean":"2024-08-28T19:19:08.622224+0000","last_became_active":"2024-08-28T19:18:39.527781+0000","last_became_peered":"2024-08-28T19:18:39.527781+0000","last_unstale":"2024-08-28T19:19:08.622224+0000","last_undegraded":"2024-08-28T19:19:08.622224+0000","last_fullsized":"2024-08-28T19:19:08.622224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:39.831426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632931+0000","last_change":"2024-08-28T19:18:31.082641+0000","last_active":"2024-08-28T19:19:08.632931+0000","last_peered":"2024-08-28T19:19:08.632931+0000","last_clean":"2024-08-28T19:19:08.632931+0000","last_became_active":"2024-08-28T19:18:31.082260+0000","last_became_peered":"2024-08-28T19:18:31.082260+0000","last_unstale":"2024-08-28T19:19:08.632931+0000","last_undegraded":"2024-08-28T19:19:08.632931+0000","last_fullsized":"2024-08-28T19:19:08.632931+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:09:55.836463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629290+0000","last_change":"2024-08-28T19:18:32.493498+0000","last_active":"2024-08-28T19:19:08.629290+0000","last_peered":"2024-08-28T19:19:08.629290+0000","last_clean":"2024-08-28T19:19:08.629290+0000","last_became_active":"2024-08-28T19:18:32.490969+0000","last_became_peered":"2024-08-28T19:18:32.490969+0000","last_unstale":"2024-08-28T19:19:08.629290+0000","last_undegraded":"2024-08-28T19:19:08.629290+0000","last_fullsized":"2024-08-28T19:19:08.629290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:34:29.537622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617571+0000","last_change":"2024-08-28T19:18:39.520922+0000","last_active":"2024-08-28T19:19:08.617571+0000","last_peered":"2024-08-28T19:19:08.617571+0000","last_clean":"2024-08-28T19:19:08.617571+0000","last_became_active":"2024-08-28T19:18:39.520362+0000","last_became_peered":"2024-08-28T19:18:39.520362+0000","last_unstale":"2024-08-28T19:19:08.617571+0000","last_undegraded":"2024-08-28T19:19:08.617571+0000","last_fullsized":"2024-08-28T19:19:08.617571+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:40.658820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.646751+0000","last_change":"2024-08-28T19:18:30.471870+0000","last_active":"2024-08-28T19:19:12.646751+0000","last_peered":"2024-08-28T19:19:12.646751+0000","last_clean":"2024-08-28T19:19:12.646751+0000","last_became_active":"2024-08-28T19:18:30.471438+0000","last_became_peered":"2024-08-28T19:18:30.471438+0000","last_unstale":"2024-08-28T19:19:12.646751+0000","last_undegraded":"2024-08-28T19:19:12.646751+0000","last_fullsized":"2024-08-28T19:19:12.646751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:44:34.137395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628034+0000","last_change":"2024-08-28T19:18:32.486485+0000","last_active":"2024-08-28T19:19:08.628034+0000","last_peered":"2024-08-28T19:19:08.628034+0000","last_clean":"2024-08-28T19:19:08.628034+0000","last_became_active":"2024-08-28T19:18:32.485888+0000","last_became_peered":"2024-08-28T19:18:32.485888+0000","last_unstale":"2024-08-28T19:19:08.628034+0000","last_undegraded":"2024-08-28T19:19:08.628034+0000","last_fullsized":"2024-08-28T19:19:08.628034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:45:48.757318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.633437+0000","last_change":"2024-08-28T19:19:08.633317+0000","last_active":"2024-08-28T19:19:08.633437+0000","last_peered":"2024-08-28T19:19:08.633437+0000","last_clean":"2024-08-28T19:19:08.633437+0000","last_became_active":"2024-08-28T19:19:08.632689+0000","last_became_peered":"2024-08-28T19:19:08.632689+0000","last_unstale":"2024-08-28T19:19:08.633437+0000","last_undegraded":"2024-08-28T19:19:08.633437+0000","last_fullsized":"2024-08-28T19:19:08.633437+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:35:08.740645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653995+0000","last_change":"2024-08-28T19:18:30.473982+0000","last_active":"2024-08-28T19:19:11.653995+0000","last_peered":"2024-08-28T19:19:11.653995+0000","last_clean":"2024-08-28T19:19:11.653995+0000","last_became_active":"2024-08-28T19:18:30.473552+0000","last_became_peered":"2024-08-28T19:18:30.473552+0000","last_unstale":"2024-08-28T19:19:11.653995+0000","last_undegraded":"2024-08-28T19:19:11.653995+0000","last_fullsized":"2024-08-28T19:19:11.653995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:22:29.435517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616754+0000","last_change":"2024-08-28T19:18:32.487881+0000","last_active":"2024-08-28T19:19:08.616754+0000","last_peered":"2024-08-28T19:19:08.616754+0000","last_clean":"2024-08-28T19:19:08.616754+0000","last_became_active":"2024-08-28T19:18:32.487039+0000","last_became_peered":"2024-08-28T19:18:32.487039+0000","last_unstale":"2024-08-28T19:19:08.616754+0000","last_undegraded":"2024-08-28T19:19:08.616754+0000","last_fullsized":"2024-08-28T19:19:08.616754+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:23:35.757811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617767+0000","last_change":"2024-08-28T19:18:39.518654+0000","last_active":"2024-08-28T19:19:08.617767+0000","last_peered":"2024-08-28T19:19:08.617767+0000","last_clean":"2024-08-28T19:19:08.617767+0000","last_became_active":"2024-08-28T19:18:39.518087+0000","last_became_peered":"2024-08-28T19:18:39.518087+0000","last_unstale":"2024-08-28T19:19:08.617767+0000","last_undegraded":"2024-08-28T19:19:08.617767+0000","last_fullsized":"2024-08-28T19:19:08.617767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:08.284865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617577+0000","last_change":"2024-08-28T19:18:32.491928+0000","last_active":"2024-08-28T19:19:08.617577+0000","last_peered":"2024-08-28T19:19:08.617577+0000","last_clean":"2024-08-28T19:19:08.617577+0000","last_became_active":"2024-08-28T19:18:32.491538+0000","last_became_peered":"2024-08-28T19:18:32.491538+0000","last_unstale":"2024-08-28T19:19:08.617577+0000","last_undegraded":"2024-08-28T19:19:08.617577+0000","last_fullsized":"2024-08-28T19:19:08.617577+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:39:06.077830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.955289+0000","last_change":"2024-08-28T19:19:08.634752+0000","last_active":"2024-08-28T19:19:11.955289+0000","last_peered":"2024-08-28T19:19:11.955289+0000","last_clean":"2024-08-28T19:19:11.955289+0000","last_became_active":"2024-08-28T19:19:08.620784+0000","last_became_peered":"2024-08-28T19:19:08.620784+0000","last_unstale":"2024-08-28T19:19:11.955289+0000","last_undegraded":"2024-08-28T19:19:11.955289+0000","last_fullsized":"2024-08-28T19:19:11.955289+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:51.640445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618419+0000","last_change":"2024-08-28T19:18:39.681413+0000","last_active":"2024-08-28T19:19:08.618419+0000","last_peered":"2024-08-28T19:19:08.618419+0000","last_clean":"2024-08-28T19:19:08.618419+0000","last_became_active":"2024-08-28T19:18:39.680793+0000","last_became_peered":"2024-08-28T19:18:39.680793+0000","last_unstale":"2024-08-28T19:19:08.618419+0000","last_undegraded":"2024-08-28T19:19:08.618419+0000","last_fullsized":"2024-08-28T19:19:08.618419+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:45:43.381019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632212+0000","last_change":"2024-08-28T19:18:32.486739+0000","last_active":"2024-08-28T19:19:08.632212+0000","last_peered":"2024-08-28T19:19:08.632212+0000","last_clean":"2024-08-28T19:19:08.632212+0000","last_became_active":"2024-08-28T19:18:32.486347+0000","last_became_peered":"2024-08-28T19:18:32.486347+0000","last_unstale":"2024-08-28T19:19:08.632212+0000","last_undegraded":"2024-08-28T19:19:08.632212+0000","last_fullsized":"2024-08-28T19:19:08.632212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:27:36.448569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.641964+0000","last_change":"2024-08-28T19:18:30.467690+0000","last_active":"2024-08-28T19:19:10.641964+0000","last_peered":"2024-08-28T19:19:10.641964+0000","last_clean":"2024-08-28T19:19:10.641964+0000","last_became_active":"2024-08-28T19:18:30.467249+0000","last_became_peered":"2024-08-28T19:18:30.467249+0000","last_unstale":"2024-08-28T19:19:10.641964+0000","last_undegraded":"2024-08-28T19:19:10.641964+0000","last_fullsized":"2024-08-28T19:19:10.641964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:37:28.876666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618227+0000","last_change":"2024-08-28T19:18:39.523889+0000","last_active":"2024-08-28T19:19:08.618227+0000","last_peered":"2024-08-28T19:19:08.618227+0000","last_clean":"2024-08-28T19:19:08.618227+0000","last_became_active":"2024-08-28T19:18:39.523384+0000","last_became_peered":"2024-08-28T19:18:39.523384+0000","last_unstale":"2024-08-28T19:19:08.618227+0000","last_undegraded":"2024-08-28T19:19:08.618227+0000","last_fullsized":"2024-08-28T19:19:08.618227+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:57:32.277680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614254+0000","last_change":"2024-08-28T19:18:32.482257+0000","last_active":"2024-08-28T19:19:08.614254+0000","last_peered":"2024-08-28T19:19:08.614254+0000","last_clean":"2024-08-28T19:19:08.614254+0000","last_became_active":"2024-08-28T19:18:32.481884+0000","last_became_peered":"2024-08-28T19:18:32.481884+0000","last_unstale":"2024-08-28T19:19:08.614254+0000","last_undegraded":"2024-08-28T19:19:08.614254+0000","last_fullsized":"2024-08-28T19:19:08.614254+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:56:28.754369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651638+0000","last_change":"2024-08-28T19:18:30.467307+0000","last_active":"2024-08-28T19:19:11.651638+0000","last_peered":"2024-08-28T19:19:11.651638+0000","last_clean":"2024-08-28T19:19:11.651638+0000","last_became_active":"2024-08-28T19:18:30.466942+0000","last_became_peered":"2024-08-28T19:18:30.466942+0000","last_unstale":"2024-08-28T19:19:11.651638+0000","last_undegraded":"2024-08-28T19:19:11.651638+0000","last_fullsized":"2024-08-28T19:19:11.651638+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:05:23.682110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614513+0000","last_change":"2024-08-28T19:18:39.531421+0000","last_active":"2024-08-28T19:19:08.614513+0000","last_peered":"2024-08-28T19:19:08.614513+0000","last_clean":"2024-08-28T19:19:08.614513+0000","last_became_active":"2024-08-28T19:18:39.531008+0000","last_became_peered":"2024-08-28T19:18:39.531008+0000","last_unstale":"2024-08-28T19:19:08.614513+0000","last_undegraded":"2024-08-28T19:19:08.614513+0000","last_fullsized":"2024-08-28T19:19:08.614513+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:46:49.580601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618946+0000","last_change":"2024-08-28T19:18:32.485204+0000","last_active":"2024-08-28T19:19:08.618946+0000","last_peered":"2024-08-28T19:19:08.618946+0000","last_clean":"2024-08-28T19:19:08.618946+0000","last_became_active":"2024-08-28T19:18:32.484830+0000","last_became_peered":"2024-08-28T19:18:32.484830+0000","last_unstale":"2024-08-28T19:19:08.618946+0000","last_undegraded":"2024-08-28T19:19:08.618946+0000","last_fullsized":"2024-08-28T19:19:08.618946+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:55:41.358718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632011+0000","last_change":"2024-08-28T19:18:31.082361+0000","last_active":"2024-08-28T19:19:08.632011+0000","last_peered":"2024-08-28T19:19:08.632011+0000","last_clean":"2024-08-28T19:19:08.632011+0000","last_became_active":"2024-08-28T19:18:31.081930+0000","last_became_peered":"2024-08-28T19:18:31.081930+0000","last_unstale":"2024-08-28T19:19:08.632011+0000","last_undegraded":"2024-08-28T19:19:08.632011+0000","last_fullsized":"2024-08-28T19:19:08.632011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:41:05.543088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618390+0000","last_change":"2024-08-28T19:18:39.682341+0000","last_active":"2024-08-28T19:19:08.618390+0000","last_peered":"2024-08-28T19:19:08.618390+0000","last_clean":"2024-08-28T19:19:08.618390+0000","last_became_active":"2024-08-28T19:18:39.681751+0000","last_became_peered":"2024-08-28T19:18:39.681751+0000","last_unstale":"2024-08-28T19:19:08.618390+0000","last_undegraded":"2024-08-28T19:19:08.618390+0000","last_fullsized":"2024-08-28T19:19:08.618390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:51:24.371270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628950+0000","last_change":"2024-08-28T19:18:32.491069+0000","last_active":"2024-08-28T19:19:08.628950+0000","last_peered":"2024-08-28T19:19:08.628950+0000","last_clean":"2024-08-28T19:19:08.628950+0000","last_became_active":"2024-08-28T19:18:32.490688+0000","last_became_peered":"2024-08-28T19:18:32.490688+0000","last_unstale":"2024-08-28T19:19:08.628950+0000","last_undegraded":"2024-08-28T19:19:08.628950+0000","last_fullsized":"2024-08-28T19:19:08.628950+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:24:31.560519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652924+0000","last_change":"2024-08-28T19:18:30.739812+0000","last_active":"2024-08-28T19:19:11.652924+0000","last_peered":"2024-08-28T19:19:11.652924+0000","last_clean":"2024-08-28T19:19:11.652924+0000","last_became_active":"2024-08-28T19:18:30.739414+0000","last_became_peered":"2024-08-28T19:18:30.739414+0000","last_unstale":"2024-08-28T19:19:11.652924+0000","last_undegraded":"2024-08-28T19:19:11.652924+0000","last_fullsized":"2024-08-28T19:19:11.652924+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:56:56.594993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622469+0000","last_change":"2024-08-28T19:18:39.528281+0000","last_active":"2024-08-28T19:19:08.622469+0000","last_peered":"2024-08-28T19:19:08.622469+0000","last_clean":"2024-08-28T19:19:08.622469+0000","last_became_active":"2024-08-28T19:18:39.527142+0000","last_became_peered":"2024-08-28T19:18:39.527142+0000","last_unstale":"2024-08-28T19:19:08.622469+0000","last_undegraded":"2024-08-28T19:19:08.622469+0000","last_fullsized":"2024-08-28T19:19:08.622469+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:44:19.123662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632709+0000","last_change":"2024-08-28T19:18:32.490430+0000","last_active":"2024-08-28T19:19:08.632709+0000","last_peered":"2024-08-28T19:19:08.632709+0000","last_clean":"2024-08-28T19:19:08.632709+0000","last_became_active":"2024-08-28T19:18:32.490020+0000","last_became_peered":"2024-08-28T19:18:32.490020+0000","last_unstale":"2024-08-28T19:19:08.632709+0000","last_undegraded":"2024-08-28T19:19:08.632709+0000","last_fullsized":"2024-08-28T19:19:08.632709+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:52:31.153012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"82'15","reported_seq":50,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:25.488292+0000","last_change":"2024-08-28T19:18:30.467060+0000","last_active":"2024-08-28T19:19:25.488292+0000","last_peered":"2024-08-28T19:19:25.488292+0000","last_clean":"2024-08-28T19:19:25.488292+0000","last_became_active":"2024-08-28T19:18:30.466689+0000","last_became_peered":"2024-08-28T19:18:30.466689+0000","last_unstale":"2024-08-28T19:19:25.488292+0000","last_undegraded":"2024-08-28T19:19:25.488292+0000","last_fullsized":"2024-08-28T19:19:25.488292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:18:36.508131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.630939+0000","last_change":"2024-08-28T19:18:39.682874+0000","last_active":"2024-08-28T19:19:08.630939+0000","last_peered":"2024-08-28T19:19:08.630939+0000","last_clean":"2024-08-28T19:19:08.630939+0000","last_became_active":"2024-08-28T19:18:39.681500+0000","last_became_peered":"2024-08-28T19:18:39.681500+0000","last_unstale":"2024-08-28T19:19:08.630939+0000","last_undegraded":"2024-08-28T19:19:08.630939+0000","last_fullsized":"2024-08-28T19:19:08.630939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:45:41.815506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612195+0000","last_change":"2024-08-28T19:18:32.482047+0000","last_active":"2024-08-28T19:19:08.612195+0000","last_peered":"2024-08-28T19:19:08.612195+0000","last_clean":"2024-08-28T19:19:08.612195+0000","last_became_active":"2024-08-28T19:18:32.481640+0000","last_became_peered":"2024-08-28T19:18:32.481640+0000","last_unstale":"2024-08-28T19:19:08.612195+0000","last_undegraded":"2024-08-28T19:19:08.612195+0000","last_fullsized":"2024-08-28T19:19:08.612195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:34.717764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651944+0000","last_change":"2024-08-28T19:18:31.083524+0000","last_active":"2024-08-28T19:19:11.651944+0000","last_peered":"2024-08-28T19:19:11.651944+0000","last_clean":"2024-08-28T19:19:11.651944+0000","last_became_active":"2024-08-28T19:18:31.083106+0000","last_became_peered":"2024-08-28T19:18:31.083106+0000","last_unstale":"2024-08-28T19:19:11.651944+0000","last_undegraded":"2024-08-28T19:19:11.651944+0000","last_fullsized":"2024-08-28T19:19:11.651944+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:16.180754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615954+0000","last_change":"2024-08-28T19:18:39.531884+0000","last_active":"2024-08-28T19:19:08.615954+0000","last_peered":"2024-08-28T19:19:08.615954+0000","last_clean":"2024-08-28T19:19:08.615954+0000","last_became_active":"2024-08-28T19:18:39.531255+0000","last_became_peered":"2024-08-28T19:18:39.531255+0000","last_unstale":"2024-08-28T19:19:08.615954+0000","last_undegraded":"2024-08-28T19:19:08.615954+0000","last_fullsized":"2024-08-28T19:19:08.615954+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:36:55.496594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612691+0000","last_change":"2024-08-28T19:18:32.485180+0000","last_active":"2024-08-28T19:19:08.612691+0000","last_peered":"2024-08-28T19:19:08.612691+0000","last_clean":"2024-08-28T19:19:08.612691+0000","last_became_active":"2024-08-28T19:18:32.484804+0000","last_became_peered":"2024-08-28T19:18:32.484804+0000","last_unstale":"2024-08-28T19:19:08.612691+0000","last_undegraded":"2024-08-28T19:19:08.612691+0000","last_fullsized":"2024-08-28T19:19:08.612691+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:12:40.958251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615096+0000","last_change":"2024-08-28T19:18:31.082226+0000","last_active":"2024-08-28T19:19:08.615096+0000","last_peered":"2024-08-28T19:19:08.615096+0000","last_clean":"2024-08-28T19:19:08.615096+0000","last_became_active":"2024-08-28T19:18:31.081833+0000","last_became_peered":"2024-08-28T19:18:31.081833+0000","last_unstale":"2024-08-28T19:19:08.615096+0000","last_undegraded":"2024-08-28T19:19:08.615096+0000","last_fullsized":"2024-08-28T19:19:08.615096+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:33:56.298093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619013+0000","last_change":"2024-08-28T19:18:39.681288+0000","last_active":"2024-08-28T19:19:08.619013+0000","last_peered":"2024-08-28T19:19:08.619013+0000","last_clean":"2024-08-28T19:19:08.619013+0000","last_became_active":"2024-08-28T19:18:39.680719+0000","last_became_peered":"2024-08-28T19:18:39.680719+0000","last_unstale":"2024-08-28T19:19:08.619013+0000","last_undegraded":"2024-08-28T19:19:08.619013+0000","last_fullsized":"2024-08-28T19:19:08.619013+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:53:35.554181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612370+0000","last_change":"2024-08-28T19:18:32.487105+0000","last_active":"2024-08-28T19:19:08.612370+0000","last_peered":"2024-08-28T19:19:08.612370+0000","last_clean":"2024-08-28T19:19:08.612370+0000","last_became_active":"2024-08-28T19:18:32.486643+0000","last_became_peered":"2024-08-28T19:18:32.486643+0000","last_unstale":"2024-08-28T19:19:08.612370+0000","last_undegraded":"2024-08-28T19:19:08.612370+0000","last_fullsized":"2024-08-28T19:19:08.612370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:18:43.274493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'6","reported_seq":96,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:03.732349+0000","last_change":"2024-08-28T19:18:30.471447+0000","last_active":"2024-08-28T19:20:03.732349+0000","last_peered":"2024-08-28T19:20:03.732349+0000","last_clean":"2024-08-28T19:20:03.732349+0000","last_became_active":"2024-08-28T19:18:30.470798+0000","last_became_peered":"2024-08-28T19:18:30.470798+0000","last_unstale":"2024-08-28T19:20:03.732349+0000","last_undegraded":"2024-08-28T19:20:03.732349+0000","last_fullsized":"2024-08-28T19:20:03.732349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:50:13.948439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611755+0000","last_change":"2024-08-28T19:18:39.524190+0000","last_active":"2024-08-28T19:19:08.611755+0000","last_peered":"2024-08-28T19:19:08.611755+0000","last_clean":"2024-08-28T19:19:08.611755+0000","last_became_active":"2024-08-28T19:18:39.523771+0000","last_became_peered":"2024-08-28T19:18:39.523771+0000","last_unstale":"2024-08-28T19:19:08.611755+0000","last_undegraded":"2024-08-28T19:19:08.611755+0000","last_fullsized":"2024-08-28T19:19:08.611755+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:43:43.984625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615282+0000","last_change":"2024-08-28T19:18:32.487212+0000","last_active":"2024-08-28T19:19:08.615282+0000","last_peered":"2024-08-28T19:19:08.615282+0000","last_clean":"2024-08-28T19:19:08.615282+0000","last_became_active":"2024-08-28T19:18:32.485432+0000","last_became_peered":"2024-08-28T19:18:32.485432+0000","last_unstale":"2024-08-28T19:19:08.615282+0000","last_undegraded":"2024-08-28T19:19:08.615282+0000","last_fullsized":"2024-08-28T19:19:08.615282+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:52:11.672193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":94,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:03.732659+0000","last_change":"2024-08-28T19:18:31.082210+0000","last_active":"2024-08-28T19:20:03.732659+0000","last_peered":"2024-08-28T19:20:03.732659+0000","last_clean":"2024-08-28T19:20:03.732659+0000","last_became_active":"2024-08-28T19:18:31.081814+0000","last_became_peered":"2024-08-28T19:18:31.081814+0000","last_unstale":"2024-08-28T19:20:03.732659+0000","last_undegraded":"2024-08-28T19:20:03.732659+0000","last_fullsized":"2024-08-28T19:20:03.732659+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:33:32.281431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":54,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611523+0000","last_change":"2024-08-28T19:18:39.681858+0000","last_active":"2024-08-28T19:19:08.611523+0000","last_peered":"2024-08-28T19:19:08.611523+0000","last_clean":"2024-08-28T19:19:08.611523+0000","last_became_active":"2024-08-28T19:18:39.681382+0000","last_became_peered":"2024-08-28T19:18:39.681382+0000","last_unstale":"2024-08-28T19:19:08.611523+0000","last_undegraded":"2024-08-28T19:19:08.611523+0000","last_fullsized":"2024-08-28T19:19:08.611523+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:23:22.021022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611190+0000","last_change":"2024-08-28T19:18:32.486802+0000","last_active":"2024-08-28T19:19:08.611190+0000","last_peered":"2024-08-28T19:19:08.611190+0000","last_clean":"2024-08-28T19:19:08.611190+0000","last_became_active":"2024-08-28T19:18:32.486479+0000","last_became_peered":"2024-08-28T19:18:32.486479+0000","last_unstale":"2024-08-28T19:19:08.611190+0000","last_undegraded":"2024-08-28T19:19:08.611190+0000","last_fullsized":"2024-08-28T19:19:08.611190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:17:17.654757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:09.635071+0000","last_change":"2024-08-28T19:18:30.470447+0000","last_active":"2024-08-28T19:19:09.635071+0000","last_peered":"2024-08-28T19:19:09.635071+0000","last_clean":"2024-08-28T19:19:09.635071+0000","last_became_active":"2024-08-28T19:18:30.469862+0000","last_became_peered":"2024-08-28T19:18:30.469862+0000","last_unstale":"2024-08-28T19:19:09.635071+0000","last_undegraded":"2024-08-28T19:19:09.635071+0000","last_fullsized":"2024-08-28T19:19:09.635071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:27:46.660492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624752+0000","last_change":"2024-08-28T19:19:08.624635+0000","last_active":"2024-08-28T19:19:08.624752+0000","last_peered":"2024-08-28T19:19:08.624752+0000","last_clean":"2024-08-28T19:19:08.624752+0000","last_became_active":"2024-08-28T19:19:08.622845+0000","last_became_peered":"2024-08-28T19:19:08.622845+0000","last_unstale":"2024-08-28T19:19:08.624752+0000","last_undegraded":"2024-08-28T19:19:08.624752+0000","last_fullsized":"2024-08-28T19:19:08.624752+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:56:40.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631918+0000","last_change":"2024-08-28T19:18:32.487947+0000","last_active":"2024-08-28T19:19:08.631918+0000","last_peered":"2024-08-28T19:19:08.631918+0000","last_clean":"2024-08-28T19:19:08.631918+0000","last_became_active":"2024-08-28T19:18:32.487592+0000","last_became_peered":"2024-08-28T19:18:32.487592+0000","last_unstale":"2024-08-28T19:19:08.631918+0000","last_undegraded":"2024-08-28T19:19:08.631918+0000","last_fullsized":"2024-08-28T19:19:08.631918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:30:29.550644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.654410+0000","last_change":"2024-08-28T19:18:30.465044+0000","last_active":"2024-08-28T19:19:12.654410+0000","last_peered":"2024-08-28T19:19:12.654410+0000","last_clean":"2024-08-28T19:19:12.654410+0000","last_became_active":"2024-08-28T19:18:30.464580+0000","last_became_peered":"2024-08-28T19:18:30.464580+0000","last_unstale":"2024-08-28T19:19:12.654410+0000","last_undegraded":"2024-08-28T19:19:12.654410+0000","last_fullsized":"2024-08-28T19:19:12.654410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:21:17.223311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":110,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617248+0000","last_change":"2024-08-28T19:17:17.063316+0000","last_active":"2024-08-28T19:19:08.617248+0000","last_peered":"2024-08-28T19:19:08.617248+0000","last_clean":"2024-08-28T19:19:08.617248+0000","last_became_active":"2024-08-28T19:17:17.057091+0000","last_became_peered":"2024-08-28T19:17:17.057091+0000","last_unstale":"2024-08-28T19:19:08.617248+0000","last_undegraded":"2024-08-28T19:19:08.617248+0000","last_fullsized":"2024-08-28T19:19:08.617248+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_clean_scrub_stamp":"2024-08-28T19:13:56.815467+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:35:51.499703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"82'1","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622311+0000","last_change":"2024-08-28T19:19:08.621402+0000","last_active":"2024-08-28T19:19:08.622311+0000","last_peered":"2024-08-28T19:19:08.622311+0000","last_clean":"2024-08-28T19:19:08.622311+0000","last_became_active":"2024-08-28T19:19:08.620779+0000","last_became_peered":"2024-08-28T19:19:08.620779+0000","last_unstale":"2024-08-28T19:19:08.622311+0000","last_undegraded":"2024-08-28T19:19:08.622311+0000","last_fullsized":"2024-08-28T19:19:08.622311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:39.391414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619759+0000","last_change":"2024-08-28T19:18:32.490066+0000","last_active":"2024-08-28T19:19:08.619759+0000","last_peered":"2024-08-28T19:19:08.619759+0000","last_clean":"2024-08-28T19:19:08.619759+0000","last_became_active":"2024-08-28T19:18:32.489560+0000","last_became_peered":"2024-08-28T19:18:32.489560+0000","last_unstale":"2024-08-28T19:19:08.619759+0000","last_undegraded":"2024-08-28T19:19:08.619759+0000","last_fullsized":"2024-08-28T19:19:08.619759+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:22:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'5","reported_seq":91,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:59.727453+0000","last_change":"2024-08-28T19:18:31.083078+0000","last_active":"2024-08-28T19:19:59.727453+0000","last_peered":"2024-08-28T19:19:59.727453+0000","last_clean":"2024-08-28T19:19:59.727453+0000","last_became_active":"2024-08-28T19:18:31.082575+0000","last_became_peered":"2024-08-28T19:18:31.082575+0000","last_unstale":"2024-08-28T19:19:59.727453+0000","last_undegraded":"2024-08-28T19:19:59.727453+0000","last_fullsized":"2024-08-28T19:19:59.727453+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627532+0000","last_change":"2024-08-28T19:18:39.519814+0000","last_active":"2024-08-28T19:19:08.627532+0000","last_peered":"2024-08-28T19:19:08.627532+0000","last_clean":"2024-08-28T19:19:08.627532+0000","last_became_active":"2024-08-28T19:18:39.519258+0000","last_became_peered":"2024-08-28T19:18:39.519258+0000","last_unstale":"2024-08-28T19:19:08.627532+0000","last_undegraded":"2024-08-28T19:19:08.627532+0000","last_fullsized":"2024-08-28T19:19:08.627532+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:03:04.051376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618184+0000","last_change":"2024-08-28T19:18:32.495426+0000","last_active":"2024-08-28T19:19:08.618184+0000","last_peered":"2024-08-28T19:19:08.618184+0000","last_clean":"2024-08-28T19:19:08.618184+0000","last_became_active":"2024-08-28T19:18:32.495075+0000","last_became_peered":"2024-08-28T19:18:32.495075+0000","last_unstale":"2024-08-28T19:19:08.618184+0000","last_undegraded":"2024-08-28T19:19:08.618184+0000","last_fullsized":"2024-08-28T19:19:08.618184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:40:22.456666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652460+0000","last_change":"2024-08-28T19:18:30.472708+0000","last_active":"2024-08-28T19:19:11.652460+0000","last_peered":"2024-08-28T19:19:11.652460+0000","last_clean":"2024-08-28T19:19:11.652460+0000","last_became_active":"2024-08-28T19:18:30.472320+0000","last_became_peered":"2024-08-28T19:18:30.472320+0000","last_unstale":"2024-08-28T19:19:11.652460+0000","last_undegraded":"2024-08-28T19:19:11.652460+0000","last_fullsized":"2024-08-28T19:19:11.652460+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:57:12.055626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628468+0000","last_change":"2024-08-28T19:18:39.519398+0000","last_active":"2024-08-28T19:19:08.628468+0000","last_peered":"2024-08-28T19:19:08.628468+0000","last_clean":"2024-08-28T19:19:08.628468+0000","last_became_active":"2024-08-28T19:18:39.518997+0000","last_became_peered":"2024-08-28T19:18:39.518997+0000","last_unstale":"2024-08-28T19:19:08.628468+0000","last_undegraded":"2024-08-28T19:19:08.628468+0000","last_fullsized":"2024-08-28T19:19:08.628468+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:33.431608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619235+0000","last_change":"2024-08-28T19:18:32.489439+0000","last_active":"2024-08-28T19:19:08.619235+0000","last_peered":"2024-08-28T19:19:08.619235+0000","last_clean":"2024-08-28T19:19:08.619235+0000","last_became_active":"2024-08-28T19:18:32.489054+0000","last_became_peered":"2024-08-28T19:18:32.489054+0000","last_unstale":"2024-08-28T19:19:08.619235+0000","last_undegraded":"2024-08-28T19:19:08.619235+0000","last_fullsized":"2024-08-28T19:19:08.619235+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:31:32.647384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651121+0000","last_change":"2024-08-28T19:18:30.464993+0000","last_active":"2024-08-28T19:19:11.651121+0000","last_peered":"2024-08-28T19:19:11.651121+0000","last_clean":"2024-08-28T19:19:11.651121+0000","last_became_active":"2024-08-28T19:18:30.463951+0000","last_became_peered":"2024-08-28T19:18:30.463951+0000","last_unstale":"2024-08-28T19:19:11.651121+0000","last_undegraded":"2024-08-28T19:19:11.651121+0000","last_fullsized":"2024-08-28T19:19:11.651121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:06.780058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614780+0000","last_change":"2024-08-28T19:18:39.533061+0000","last_active":"2024-08-28T19:19:08.614780+0000","last_peered":"2024-08-28T19:19:08.614780+0000","last_clean":"2024-08-28T19:19:08.614780+0000","last_became_active":"2024-08-28T19:18:39.532607+0000","last_became_peered":"2024-08-28T19:18:39.532607+0000","last_unstale":"2024-08-28T19:19:08.614780+0000","last_undegraded":"2024-08-28T19:19:08.614780+0000","last_fullsized":"2024-08-28T19:19:08.614780+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:00:57.997665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615094+0000","last_change":"2024-08-28T19:18:32.487937+0000","last_active":"2024-08-28T19:19:08.615094+0000","last_peered":"2024-08-28T19:19:08.615094+0000","last_clean":"2024-08-28T19:19:08.615094+0000","last_became_active":"2024-08-28T19:18:32.487417+0000","last_became_peered":"2024-08-28T19:18:32.487417+0000","last_unstale":"2024-08-28T19:19:08.615094+0000","last_undegraded":"2024-08-28T19:19:08.615094+0000","last_fullsized":"2024-08-28T19:19:08.615094+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:42.497364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.630439+0000","last_change":"2024-08-28T19:18:30.470742+0000","last_active":"2024-08-28T19:19:10.630439+0000","last_peered":"2024-08-28T19:19:10.630439+0000","last_clean":"2024-08-28T19:19:10.630439+0000","last_became_active":"2024-08-28T19:18:30.470135+0000","last_became_peered":"2024-08-28T19:18:30.470135+0000","last_unstale":"2024-08-28T19:19:10.630439+0000","last_undegraded":"2024-08-28T19:19:10.630439+0000","last_fullsized":"2024-08-28T19:19:10.630439+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:49:39.581119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617294+0000","last_change":"2024-08-28T19:18:39.524052+0000","last_active":"2024-08-28T19:19:08.617294+0000","last_peered":"2024-08-28T19:19:08.617294+0000","last_clean":"2024-08-28T19:19:08.617294+0000","last_became_active":"2024-08-28T19:18:39.523601+0000","last_became_peered":"2024-08-28T19:18:39.523601+0000","last_unstale":"2024-08-28T19:19:08.617294+0000","last_undegraded":"2024-08-28T19:19:08.617294+0000","last_fullsized":"2024-08-28T19:19:08.617294+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:54:53.302424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617504+0000","last_change":"2024-08-28T19:18:32.491151+0000","last_active":"2024-08-28T19:19:08.617504+0000","last_peered":"2024-08-28T19:19:08.617504+0000","last_clean":"2024-08-28T19:19:08.617504+0000","last_became_active":"2024-08-28T19:18:32.490798+0000","last_became_peered":"2024-08-28T19:18:32.490798+0000","last_unstale":"2024-08-28T19:19:08.617504+0000","last_undegraded":"2024-08-28T19:19:08.617504+0000","last_fullsized":"2024-08-28T19:19:08.617504+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:41:00.771019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652641+0000","last_change":"2024-08-28T19:18:31.084029+0000","last_active":"2024-08-28T19:19:11.652641+0000","last_peered":"2024-08-28T19:19:11.652641+0000","last_clean":"2024-08-28T19:19:11.652641+0000","last_became_active":"2024-08-28T19:18:31.083507+0000","last_became_peered":"2024-08-28T19:18:31.083507+0000","last_unstale":"2024-08-28T19:19:11.652641+0000","last_undegraded":"2024-08-28T19:19:11.652641+0000","last_fullsized":"2024-08-28T19:19:11.652641+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:15:36.451702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611795+0000","last_change":"2024-08-28T19:18:39.527249+0000","last_active":"2024-08-28T19:19:08.611795+0000","last_peered":"2024-08-28T19:19:08.611795+0000","last_clean":"2024-08-28T19:19:08.611795+0000","last_became_active":"2024-08-28T19:18:39.526834+0000","last_became_peered":"2024-08-28T19:18:39.526834+0000","last_unstale":"2024-08-28T19:19:08.611795+0000","last_undegraded":"2024-08-28T19:19:08.611795+0000","last_fullsized":"2024-08-28T19:19:08.611795+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:48:02.294738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616892+0000","last_change":"2024-08-28T19:18:32.497337+0000","last_active":"2024-08-28T19:19:08.616892+0000","last_peered":"2024-08-28T19:19:08.616892+0000","last_clean":"2024-08-28T19:19:08.616892+0000","last_became_active":"2024-08-28T19:18:32.496929+0000","last_became_peered":"2024-08-28T19:18:32.496929+0000","last_unstale":"2024-08-28T19:19:08.616892+0000","last_undegraded":"2024-08-28T19:19:08.616892+0000","last_fullsized":"2024-08-28T19:19:08.616892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:14.748765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":94,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:02.731817+0000","last_change":"2024-08-28T19:18:30.471353+0000","last_active":"2024-08-28T19:20:02.731817+0000","last_peered":"2024-08-28T19:20:02.731817+0000","last_clean":"2024-08-28T19:20:02.731817+0000","last_became_active":"2024-08-28T19:18:30.470283+0000","last_became_peered":"2024-08-28T19:18:30.470283+0000","last_unstale":"2024-08-28T19:20:02.731817+0000","last_undegraded":"2024-08-28T19:20:02.731817+0000","last_fullsized":"2024-08-28T19:20:02.731817+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:09:46.444864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614636+0000","last_change":"2024-08-28T19:18:39.519226+0000","last_active":"2024-08-28T19:19:08.614636+0000","last_peered":"2024-08-28T19:19:08.614636+0000","last_clean":"2024-08-28T19:19:08.614636+0000","last_became_active":"2024-08-28T19:18:39.518738+0000","last_became_peered":"2024-08-28T19:18:39.518738+0000","last_unstale":"2024-08-28T19:19:08.614636+0000","last_undegraded":"2024-08-28T19:19:08.614636+0000","last_fullsized":"2024-08-28T19:19:08.614636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:39:35.114702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617127+0000","last_change":"2024-08-28T19:18:32.493060+0000","last_active":"2024-08-28T19:19:08.617127+0000","last_peered":"2024-08-28T19:19:08.617127+0000","last_clean":"2024-08-28T19:19:08.617127+0000","last_became_active":"2024-08-28T19:18:32.492590+0000","last_became_peered":"2024-08-28T19:18:32.492590+0000","last_unstale":"2024-08-28T19:19:08.617127+0000","last_undegraded":"2024-08-28T19:19:08.617127+0000","last_fullsized":"2024-08-28T19:19:08.617127+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:02:02.165301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653316+0000","last_change":"2024-08-28T19:18:30.470674+0000","last_active":"2024-08-28T19:19:11.653316+0000","last_peered":"2024-08-28T19:19:11.653316+0000","last_clean":"2024-08-28T19:19:11.653316+0000","last_became_active":"2024-08-28T19:18:30.470275+0000","last_became_peered":"2024-08-28T19:18:30.470275+0000","last_unstale":"2024-08-28T19:19:11.653316+0000","last_undegraded":"2024-08-28T19:19:11.653316+0000","last_fullsized":"2024-08-28T19:19:11.653316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:01:48.547315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624414+0000","last_change":"2024-08-28T19:19:08.624194+0000","last_active":"2024-08-28T19:19:08.624414+0000","last_peered":"2024-08-28T19:19:08.624414+0000","last_clean":"2024-08-28T19:19:08.624414+0000","last_became_active":"2024-08-28T19:19:08.623681+0000","last_became_peered":"2024-08-28T19:19:08.623681+0000","last_unstale":"2024-08-28T19:19:08.624414+0000","last_undegraded":"2024-08-28T19:19:08.624414+0000","last_fullsized":"2024-08-28T19:19:08.624414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:21:11.099931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611866+0000","last_change":"2024-08-28T19:18:32.488210+0000","last_active":"2024-08-28T19:19:08.611866+0000","last_peered":"2024-08-28T19:19:08.611866+0000","last_clean":"2024-08-28T19:19:08.611866+0000","last_became_active":"2024-08-28T19:18:32.487853+0000","last_became_peered":"2024-08-28T19:18:32.487853+0000","last_unstale":"2024-08-28T19:19:08.611866+0000","last_undegraded":"2024-08-28T19:19:08.611866+0000","last_fullsized":"2024-08-28T19:19:08.611866+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:20:51.808958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.653792+0000","last_change":"2024-08-28T19:18:30.465306+0000","last_active":"2024-08-28T19:19:12.653792+0000","last_peered":"2024-08-28T19:19:12.653792+0000","last_clean":"2024-08-28T19:19:12.653792+0000","last_became_active":"2024-08-28T19:18:30.464850+0000","last_became_peered":"2024-08-28T19:18:30.464850+0000","last_unstale":"2024-08-28T19:19:12.653792+0000","last_undegraded":"2024-08-28T19:19:12.653792+0000","last_fullsized":"2024-08-28T19:19:12.653792+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:24:26.628999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616620+0000","last_change":"2024-08-28T19:18:39.524005+0000","last_active":"2024-08-28T19:19:08.616620+0000","last_peered":"2024-08-28T19:19:08.616620+0000","last_clean":"2024-08-28T19:19:08.616620+0000","last_became_active":"2024-08-28T19:18:39.523594+0000","last_became_peered":"2024-08-28T19:18:39.523594+0000","last_unstale":"2024-08-28T19:19:08.616620+0000","last_undegraded":"2024-08-28T19:19:08.616620+0000","last_fullsized":"2024-08-28T19:19:08.616620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:35:32.484399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619776+0000","last_change":"2024-08-28T19:18:32.489088+0000","last_active":"2024-08-28T19:19:08.619776+0000","last_peered":"2024-08-28T19:19:08.619776+0000","last_clean":"2024-08-28T19:19:08.619776+0000","last_became_active":"2024-08-28T19:18:32.488717+0000","last_became_peered":"2024-08-28T19:18:32.488717+0000","last_unstale":"2024-08-28T19:19:08.619776+0000","last_undegraded":"2024-08-28T19:19:08.619776+0000","last_fullsized":"2024-08-28T19:19:08.619776+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:25.468919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"82'35","reported_seq":185,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:04.733977+0000","last_change":"2024-08-28T19:18:30.468674+0000","last_active":"2024-08-28T19:20:04.733977+0000","last_peered":"2024-08-28T19:20:04.733977+0000","last_clean":"2024-08-28T19:20:04.733977+0000","last_became_active":"2024-08-28T19:18:30.468300+0000","last_became_peered":"2024-08-28T19:18:30.468300+0000","last_unstale":"2024-08-28T19:20:04.733977+0000","last_undegraded":"2024-08-28T19:20:04.733977+0000","last_fullsized":"2024-08-28T19:20:04.733977+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:12:06.237498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63475,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":119,"num_write":37,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614424+0000","last_change":"2024-08-28T19:18:39.526131+0000","last_active":"2024-08-28T19:19:08.614424+0000","last_peered":"2024-08-28T19:19:08.614424+0000","last_clean":"2024-08-28T19:19:08.614424+0000","last_became_active":"2024-08-28T19:18:39.525708+0000","last_became_peered":"2024-08-28T19:18:39.525708+0000","last_unstale":"2024-08-28T19:19:08.614424+0000","last_undegraded":"2024-08-28T19:19:08.614424+0000","last_fullsized":"2024-08-28T19:19:08.614424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:21:52.156487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628145+0000","last_change":"2024-08-28T19:18:32.487634+0000","last_active":"2024-08-28T19:19:08.628145+0000","last_peered":"2024-08-28T19:19:08.628145+0000","last_clean":"2024-08-28T19:19:08.628145+0000","last_became_active":"2024-08-28T19:18:32.486702+0000","last_became_peered":"2024-08-28T19:18:32.486702+0000","last_unstale":"2024-08-28T19:19:08.628145+0000","last_undegraded":"2024-08-28T19:19:08.628145+0000","last_fullsized":"2024-08-28T19:19:08.628145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:08:12.606905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.643469+0000","last_change":"2024-08-28T19:18:31.083494+0000","last_active":"2024-08-28T19:19:10.643469+0000","last_peered":"2024-08-28T19:19:10.643469+0000","last_clean":"2024-08-28T19:19:10.643469+0000","last_became_active":"2024-08-28T19:18:31.083086+0000","last_became_peered":"2024-08-28T19:18:31.083086+0000","last_unstale":"2024-08-28T19:19:10.643469+0000","last_undegraded":"2024-08-28T19:19:10.643469+0000","last_fullsized":"2024-08-28T19:19:10.643469+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:14:54.388364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618783+0000","last_change":"2024-08-28T19:18:39.532155+0000","last_active":"2024-08-28T19:19:08.618783+0000","last_peered":"2024-08-28T19:19:08.618783+0000","last_clean":"2024-08-28T19:19:08.618783+0000","last_became_active":"2024-08-28T19:18:39.531565+0000","last_became_peered":"2024-08-28T19:18:39.531565+0000","last_unstale":"2024-08-28T19:19:08.618783+0000","last_undegraded":"2024-08-28T19:19:08.618783+0000","last_fullsized":"2024-08-28T19:19:08.618783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:29:48.824622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629297+0000","last_change":"2024-08-28T19:18:32.496296+0000","last_active":"2024-08-28T19:19:08.629297+0000","last_peered":"2024-08-28T19:19:08.629297+0000","last_clean":"2024-08-28T19:19:08.629297+0000","last_became_active":"2024-08-28T19:18:32.495680+0000","last_became_peered":"2024-08-28T19:18:32.495680+0000","last_unstale":"2024-08-28T19:19:08.629297+0000","last_undegraded":"2024-08-28T19:19:08.629297+0000","last_fullsized":"2024-08-28T19:19:08.629297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:27:34.544529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652615+0000","last_change":"2024-08-28T19:19:08.631758+0000","last_active":"2024-08-28T19:19:11.652615+0000","last_peered":"2024-08-28T19:19:11.652615+0000","last_clean":"2024-08-28T19:19:11.652615+0000","last_became_active":"2024-08-28T19:19:08.631202+0000","last_became_peered":"2024-08-28T19:19:08.631202+0000","last_unstale":"2024-08-28T19:19:11.652615+0000","last_undegraded":"2024-08-28T19:19:11.652615+0000","last_fullsized":"2024-08-28T19:19:11.652615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:27:43.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614142+0000","last_change":"2024-08-28T19:18:39.531157+0000","last_active":"2024-08-28T19:19:08.614142+0000","last_peered":"2024-08-28T19:19:08.614142+0000","last_clean":"2024-08-28T19:19:08.614142+0000","last_became_active":"2024-08-28T19:18:39.530675+0000","last_became_peered":"2024-08-28T19:18:39.530675+0000","last_unstale":"2024-08-28T19:19:08.614142+0000","last_undegraded":"2024-08-28T19:19:08.614142+0000","last_fullsized":"2024-08-28T19:19:08.614142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:38:40.414854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629314+0000","last_change":"2024-08-28T19:18:32.482182+0000","last_active":"2024-08-28T19:19:08.629314+0000","last_peered":"2024-08-28T19:19:08.629314+0000","last_clean":"2024-08-28T19:19:08.629314+0000","last_became_active":"2024-08-28T19:18:32.481689+0000","last_became_peered":"2024-08-28T19:18:32.481689+0000","last_unstale":"2024-08-28T19:19:08.629314+0000","last_undegraded":"2024-08-28T19:19:08.629314+0000","last_fullsized":"2024-08-28T19:19:08.629314+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:13:29.278469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"82'4","reported_seq":145,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:04.734038+0000","last_change":"2024-08-28T19:18:30.475244+0000","last_active":"2024-08-28T19:20:04.734038+0000","last_peered":"2024-08-28T19:20:04.734038+0000","last_clean":"2024-08-28T19:20:04.734038+0000","last_became_active":"2024-08-28T19:18:30.474861+0000","last_became_peered":"2024-08-28T19:18:30.474861+0000","last_unstale":"2024-08-28T19:20:04.734038+0000","last_undegraded":"2024-08-28T19:20:04.734038+0000","last_fullsized":"2024-08-28T19:20:04.734038+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:27:09.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":106,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631762+0000","last_change":"2024-08-28T19:18:39.520022+0000","last_active":"2024-08-28T19:19:08.631762+0000","last_peered":"2024-08-28T19:19:08.631762+0000","last_clean":"2024-08-28T19:19:08.631762+0000","last_became_active":"2024-08-28T19:18:39.519584+0000","last_became_peered":"2024-08-28T19:18:39.519584+0000","last_unstale":"2024-08-28T19:19:08.631762+0000","last_undegraded":"2024-08-28T19:19:08.631762+0000","last_fullsized":"2024-08-28T19:19:08.631762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:27:15.851303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628767+0000","last_change":"2024-08-28T19:18:39.522607+0000","last_active":"2024-08-28T19:19:08.628767+0000","last_peered":"2024-08-28T19:19:08.628767+0000","last_clean":"2024-08-28T19:19:08.628767+0000","last_became_active":"2024-08-28T19:18:39.522204+0000","last_became_peered":"2024-08-28T19:18:39.522204+0000","last_unstale":"2024-08-28T19:19:08.628767+0000","last_undegraded":"2024-08-28T19:19:08.628767+0000","last_fullsized":"2024-08-28T19:19:08.628767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:02:08.865294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614380+0000","last_change":"2024-08-28T19:18:32.485577+0000","last_active":"2024-08-28T19:19:08.614380+0000","last_peered":"2024-08-28T19:19:08.614380+0000","last_clean":"2024-08-28T19:19:08.614380+0000","last_became_active":"2024-08-28T19:18:32.485243+0000","last_became_peered":"2024-08-28T19:18:32.485243+0000","last_unstale":"2024-08-28T19:19:08.614380+0000","last_undegraded":"2024-08-28T19:19:08.614380+0000","last_fullsized":"2024-08-28T19:19:08.614380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:05:53.019365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653809+0000","last_change":"2024-08-28T19:18:30.470670+0000","last_active":"2024-08-28T19:19:11.653809+0000","last_peered":"2024-08-28T19:19:11.653809+0000","last_clean":"2024-08-28T19:19:11.653809+0000","last_became_active":"2024-08-28T19:18:30.470264+0000","last_became_peered":"2024-08-28T19:18:30.470264+0000","last_unstale":"2024-08-28T19:19:11.653809+0000","last_undegraded":"2024-08-28T19:19:11.653809+0000","last_fullsized":"2024-08-28T19:19:11.653809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:35.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615236+0000","last_change":"2024-08-28T19:18:39.530948+0000","last_active":"2024-08-28T19:19:08.615236+0000","last_peered":"2024-08-28T19:19:08.615236+0000","last_clean":"2024-08-28T19:19:08.615236+0000","last_became_active":"2024-08-28T19:18:39.530473+0000","last_became_peered":"2024-08-28T19:18:39.530473+0000","last_unstale":"2024-08-28T19:19:08.615236+0000","last_undegraded":"2024-08-28T19:19:08.615236+0000","last_fullsized":"2024-08-28T19:19:08.615236+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:21:57.504582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632033+0000","last_change":"2024-08-28T19:18:32.486027+0000","last_active":"2024-08-28T19:19:08.632033+0000","last_peered":"2024-08-28T19:19:08.632033+0000","last_clean":"2024-08-28T19:19:08.632033+0000","last_became_active":"2024-08-28T19:18:32.483273+0000","last_became_peered":"2024-08-28T19:18:32.483273+0000","last_unstale":"2024-08-28T19:19:08.632033+0000","last_undegraded":"2024-08-28T19:19:08.632033+0000","last_fullsized":"2024-08-28T19:19:08.632033+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:47:37.680695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:09.637193+0000","last_change":"2024-08-28T19:18:30.464811+0000","last_active":"2024-08-28T19:19:09.637193+0000","last_peered":"2024-08-28T19:19:09.637193+0000","last_clean":"2024-08-28T19:19:09.637193+0000","last_became_active":"2024-08-28T19:18:30.463986+0000","last_became_peered":"2024-08-28T19:18:30.463986+0000","last_unstale":"2024-08-28T19:19:09.637193+0000","last_undegraded":"2024-08-28T19:19:09.637193+0000","last_fullsized":"2024-08-28T19:19:09.637193+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:43:46.582993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.642972+0000","last_change":"2024-08-28T19:19:08.631382+0000","last_active":"2024-08-28T19:19:10.642972+0000","last_peered":"2024-08-28T19:19:10.642972+0000","last_clean":"2024-08-28T19:19:10.642972+0000","last_became_active":"2024-08-28T19:19:08.630869+0000","last_became_peered":"2024-08-28T19:19:08.630869+0000","last_unstale":"2024-08-28T19:19:10.642972+0000","last_undegraded":"2024-08-28T19:19:10.642972+0000","last_fullsized":"2024-08-28T19:19:10.642972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:32:54.942408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615952+0000","last_change":"2024-08-28T19:18:32.484120+0000","last_active":"2024-08-28T19:19:08.615952+0000","last_peered":"2024-08-28T19:19:08.615952+0000","last_clean":"2024-08-28T19:19:08.615952+0000","last_became_active":"2024-08-28T19:18:32.483013+0000","last_became_peered":"2024-08-28T19:18:32.483013+0000","last_unstale":"2024-08-28T19:19:08.615952+0000","last_undegraded":"2024-08-28T19:19:08.615952+0000","last_fullsized":"2024-08-28T19:19:08.615952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:59:41.195873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622328+0000","last_change":"2024-08-28T19:18:39.531046+0000","last_active":"2024-08-28T19:19:08.622328+0000","last_peered":"2024-08-28T19:19:08.622328+0000","last_clean":"2024-08-28T19:19:08.622328+0000","last_became_active":"2024-08-28T19:18:39.530407+0000","last_became_peered":"2024-08-28T19:18:39.530407+0000","last_unstale":"2024-08-28T19:19:08.622328+0000","last_undegraded":"2024-08-28T19:19:08.622328+0000","last_fullsized":"2024-08-28T19:19:08.622328+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:47:15.173021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'6","reported_seq":95,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:01.730337+0000","last_change":"2024-08-28T19:18:30.471268+0000","last_active":"2024-08-28T19:20:01.730337+0000","last_peered":"2024-08-28T19:20:01.730337+0000","last_clean":"2024-08-28T19:20:01.730337+0000","last_became_active":"2024-08-28T19:18:30.470845+0000","last_became_peered":"2024-08-28T19:18:30.470845+0000","last_unstale":"2024-08-28T19:20:01.730337+0000","last_undegraded":"2024-08-28T19:20:01.730337+0000","last_fullsized":"2024-08-28T19:20:01.730337+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:18:18.086205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619195+0000","last_change":"2024-08-28T19:18:32.479728+0000","last_active":"2024-08-28T19:19:08.619195+0000","last_peered":"2024-08-28T19:19:08.619195+0000","last_clean":"2024-08-28T19:19:08.619195+0000","last_became_active":"2024-08-28T19:18:32.478623+0000","last_became_peered":"2024-08-28T19:18:32.478623+0000","last_unstale":"2024-08-28T19:19:08.619195+0000","last_undegraded":"2024-08-28T19:19:08.619195+0000","last_fullsized":"2024-08-28T19:19:08.619195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:14:49.914808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.623192+0000","last_change":"2024-08-28T19:18:39.531308+0000","last_active":"2024-08-28T19:19:08.623192+0000","last_peered":"2024-08-28T19:19:08.623192+0000","last_clean":"2024-08-28T19:19:08.623192+0000","last_became_active":"2024-08-28T19:18:39.530403+0000","last_became_peered":"2024-08-28T19:18:39.530403+0000","last_unstale":"2024-08-28T19:19:08.623192+0000","last_undegraded":"2024-08-28T19:19:08.623192+0000","last_fullsized":"2024-08-28T19:19:08.623192+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:33.139668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614114+0000","last_change":"2024-08-28T19:18:32.494621+0000","last_active":"2024-08-28T19:19:08.614114+0000","last_peered":"2024-08-28T19:19:08.614114+0000","last_clean":"2024-08-28T19:19:08.614114+0000","last_became_active":"2024-08-28T19:18:32.494217+0000","last_became_peered":"2024-08-28T19:18:32.494217+0000","last_unstale":"2024-08-28T19:19:08.614114+0000","last_undegraded":"2024-08-28T19:19:08.614114+0000","last_fullsized":"2024-08-28T19:19:08.614114+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:54:15.419470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622753+0000","last_change":"2024-08-28T19:18:32.484716+0000","last_active":"2024-08-28T19:19:08.622753+0000","last_peered":"2024-08-28T19:19:08.622753+0000","last_clean":"2024-08-28T19:19:08.622753+0000","last_became_active":"2024-08-28T19:18:32.484294+0000","last_became_peered":"2024-08-28T19:18:32.484294+0000","last_unstale":"2024-08-28T19:19:08.622753+0000","last_undegraded":"2024-08-28T19:19:08.622753+0000","last_fullsized":"2024-08-28T19:19:08.622753+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:39:45.045607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.621987+0000","last_change":"2024-08-28T19:18:32.484286+0000","last_active":"2024-08-28T19:19:08.621987+0000","last_peered":"2024-08-28T19:19:08.621987+0000","last_clean":"2024-08-28T19:19:08.621987+0000","last_became_active":"2024-08-28T19:18:32.483354+0000","last_became_peered":"2024-08-28T19:18:32.483354+0000","last_unstale":"2024-08-28T19:19:08.621987+0000","last_undegraded":"2024-08-28T19:19:08.621987+0000","last_fullsized":"2024-08-28T19:19:08.621987+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:09.546749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628505+0000","last_change":"2024-08-28T19:18:32.496985+0000","last_active":"2024-08-28T19:19:08.628505+0000","last_peered":"2024-08-28T19:19:08.628505+0000","last_clean":"2024-08-28T19:19:08.628505+0000","last_became_active":"2024-08-28T19:18:32.495874+0000","last_became_peered":"2024-08-28T19:18:32.495874+0000","last_unstale":"2024-08-28T19:19:08.628505+0000","last_undegraded":"2024-08-28T19:19:08.628505+0000","last_fullsized":"2024-08-28T19:19:08.628505+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:42.329572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612181+0000","last_change":"2024-08-28T19:18:32.485446+0000","last_active":"2024-08-28T19:19:08.612181+0000","last_peered":"2024-08-28T19:19:08.612181+0000","last_clean":"2024-08-28T19:19:08.612181+0000","last_became_active":"2024-08-28T19:18:32.484197+0000","last_became_peered":"2024-08-28T19:18:32.484197+0000","last_unstale":"2024-08-28T19:19:08.612181+0000","last_undegraded":"2024-08-28T19:19:08.612181+0000","last_fullsized":"2024-08-28T19:19:08.612181+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:47:50.527915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616380+0000","last_change":"2024-08-28T19:18:32.489452+0000","last_active":"2024-08-28T19:19:08.616380+0000","last_peered":"2024-08-28T19:19:08.616380+0000","last_clean":"2024-08-28T19:19:08.616380+0000","last_became_active":"2024-08-28T19:18:32.487482+0000","last_became_peered":"2024-08-28T19:18:32.487482+0000","last_unstale":"2024-08-28T19:19:08.616380+0000","last_undegraded":"2024-08-28T19:19:08.616380+0000","last_fullsized":"2024-08-28T19:19:08.616380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:52:08.175080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622242+0000","last_change":"2024-08-28T19:18:32.483850+0000","last_active":"2024-08-28T19:19:08.622242+0000","last_peered":"2024-08-28T19:19:08.622242+0000","last_clean":"2024-08-28T19:19:08.622242+0000","last_became_active":"2024-08-28T19:18:32.483476+0000","last_became_peered":"2024-08-28T19:18:32.483476+0000","last_unstale":"2024-08-28T19:19:08.622242+0000","last_undegraded":"2024-08-28T19:19:08.622242+0000","last_fullsized":"2024-08-28T19:19:08.622242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:32:37.465218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632706+0000","last_change":"2024-08-28T19:18:32.492252+0000","last_active":"2024-08-28T19:19:08.632706+0000","last_peered":"2024-08-28T19:19:08.632706+0000","last_clean":"2024-08-28T19:19:08.632706+0000","last_became_active":"2024-08-28T19:18:32.491659+0000","last_became_peered":"2024-08-28T19:18:32.491659+0000","last_unstale":"2024-08-28T19:19:08.632706+0000","last_undegraded":"2024-08-28T19:19:08.632706+0000","last_fullsized":"2024-08-28T19:19:08.632706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:19:28.901559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615146+0000","last_change":"2024-08-28T19:18:32.491940+0000","last_active":"2024-08-28T19:19:08.615146+0000","last_peered":"2024-08-28T19:19:08.615146+0000","last_clean":"2024-08-28T19:19:08.615146+0000","last_became_active":"2024-08-28T19:18:32.487474+0000","last_became_peered":"2024-08-28T19:18:32.487474+0000","last_unstale":"2024-08-28T19:19:08.615146+0000","last_undegraded":"2024-08-28T19:19:08.615146+0000","last_fullsized":"2024-08-28T19:19:08.615146+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:04:32.508620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631980+0000","last_change":"2024-08-28T19:18:32.485200+0000","last_active":"2024-08-28T19:19:08.631980+0000","last_peered":"2024-08-28T19:19:08.631980+0000","last_clean":"2024-08-28T19:19:08.631980+0000","last_became_active":"2024-08-28T19:18:32.484858+0000","last_became_peered":"2024-08-28T19:18:32.484858+0000","last_unstale":"2024-08-28T19:19:08.631980+0000","last_undegraded":"2024-08-28T19:19:08.631980+0000","last_fullsized":"2024-08-28T19:19:08.631980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:10:57.925659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627792+0000","last_change":"2024-08-28T19:18:32.489903+0000","last_active":"2024-08-28T19:19:08.627792+0000","last_peered":"2024-08-28T19:19:08.627792+0000","last_clean":"2024-08-28T19:19:08.627792+0000","last_became_active":"2024-08-28T19:18:32.488950+0000","last_became_peered":"2024-08-28T19:18:32.488950+0000","last_unstale":"2024-08-28T19:19:08.627792+0000","last_undegraded":"2024-08-28T19:19:08.627792+0000","last_fullsized":"2024-08-28T19:19:08.627792+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:06:42.284177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612366+0000","last_change":"2024-08-28T19:18:32.488236+0000","last_active":"2024-08-28T19:19:08.612366+0000","last_peered":"2024-08-28T19:19:08.612366+0000","last_clean":"2024-08-28T19:19:08.612366+0000","last_became_active":"2024-08-28T19:18:32.487776+0000","last_became_peered":"2024-08-28T19:18:32.487776+0000","last_unstale":"2024-08-28T19:19:08.612366+0000","last_undegraded":"2024-08-28T19:19:08.612366+0000","last_fullsized":"2024-08-28T19:19:08.612366+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:25:58.754740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616789+0000","last_change":"2024-08-28T19:18:32.485860+0000","last_active":"2024-08-28T19:19:08.616789+0000","last_peered":"2024-08-28T19:19:08.616789+0000","last_clean":"2024-08-28T19:19:08.616789+0000","last_became_active":"2024-08-28T19:18:32.485415+0000","last_became_peered":"2024-08-28T19:18:32.485415+0000","last_unstale":"2024-08-28T19:19:08.616789+0000","last_undegraded":"2024-08-28T19:19:08.616789+0000","last_fullsized":"2024-08-28T19:19:08.616789+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:45:32.868651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.623402+0000","last_change":"2024-08-28T19:18:32.485515+0000","last_active":"2024-08-28T19:19:08.623402+0000","last_peered":"2024-08-28T19:19:08.623402+0000","last_clean":"2024-08-28T19:19:08.623402+0000","last_became_active":"2024-08-28T19:18:32.485075+0000","last_became_peered":"2024-08-28T19:18:32.485075+0000","last_unstale":"2024-08-28T19:19:08.623402+0000","last_undegraded":"2024-08-28T19:19:08.623402+0000","last_fullsized":"2024-08-28T19:19:08.623402+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:01:28.035479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614972+0000","last_change":"2024-08-28T19:18:32.484685+0000","last_active":"2024-08-28T19:19:08.614972+0000","last_peered":"2024-08-28T19:19:08.614972+0000","last_clean":"2024-08-28T19:19:08.614972+0000","last_became_active":"2024-08-28T19:18:32.484095+0000","last_became_peered":"2024-08-28T19:18:32.484095+0000","last_unstale":"2024-08-28T19:19:08.614972+0000","last_undegraded":"2024-08-28T19:19:08.614972+0000","last_fullsized":"2024-08-28T19:19:08.614972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:53:27.229186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615703+0000","last_change":"2024-08-28T19:18:32.488030+0000","last_active":"2024-08-28T19:19:08.615703+0000","last_peered":"2024-08-28T19:19:08.615703+0000","last_clean":"2024-08-28T19:19:08.615703+0000","last_became_active":"2024-08-28T19:18:32.487042+0000","last_became_peered":"2024-08-28T19:18:32.487042+0000","last_unstale":"2024-08-28T19:19:08.615703+0000","last_undegraded":"2024-08-28T19:19:08.615703+0000","last_fullsized":"2024-08-28T19:19:08.615703+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:56.966475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627165+0000","last_change":"2024-08-28T19:18:32.489442+0000","last_active":"2024-08-28T19:19:08.627165+0000","last_peered":"2024-08-28T19:19:08.627165+0000","last_clean":"2024-08-28T19:19:08.627165+0000","last_became_active":"2024-08-28T19:18:32.488405+0000","last_became_peered":"2024-08-28T19:18:32.488405+0000","last_unstale":"2024-08-28T19:19:08.627165+0000","last_undegraded":"2024-08-28T19:19:08.627165+0000","last_fullsized":"2024-08-28T19:19:08.627165+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:21:03.026893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628286+0000","last_change":"2024-08-28T19:18:32.496044+0000","last_active":"2024-08-28T19:19:08.628286+0000","last_peered":"2024-08-28T19:19:08.628286+0000","last_clean":"2024-08-28T19:19:08.628286+0000","last_became_active":"2024-08-28T19:18:32.495549+0000","last_became_peered":"2024-08-28T19:18:32.495549+0000","last_unstale":"2024-08-28T19:19:08.628286+0000","last_undegraded":"2024-08-28T19:19:08.628286+0000","last_fullsized":"2024-08-28T19:19:08.628286+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:06:56.123264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628305+0000","last_change":"2024-08-28T19:18:32.492077+0000","last_active":"2024-08-28T19:19:08.628305+0000","last_peered":"2024-08-28T19:19:08.628305+0000","last_clean":"2024-08-28T19:19:08.628305+0000","last_became_active":"2024-08-28T19:18:32.491722+0000","last_became_peered":"2024-08-28T19:18:32.491722+0000","last_unstale":"2024-08-28T19:19:08.628305+0000","last_undegraded":"2024-08-28T19:19:08.628305+0000","last_fullsized":"2024-08-28T19:19:08.628305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:16:21.697045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612237+0000","last_change":"2024-08-28T19:18:32.479845+0000","last_active":"2024-08-28T19:19:08.612237+0000","last_peered":"2024-08-28T19:19:08.612237+0000","last_clean":"2024-08-28T19:19:08.612237+0000","last_became_active":"2024-08-28T19:18:32.477690+0000","last_became_peered":"2024-08-28T19:18:32.477690+0000","last_unstale":"2024-08-28T19:19:08.612237+0000","last_undegraded":"2024-08-28T19:19:08.612237+0000","last_fullsized":"2024-08-28T19:19:08.612237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:11.281415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624748+0000","last_change":"2024-08-28T19:19:08.624185+0000","last_active":"2024-08-28T19:19:08.624748+0000","last_peered":"2024-08-28T19:19:08.624748+0000","last_clean":"2024-08-28T19:19:08.624748+0000","last_became_active":"2024-08-28T19:19:08.623745+0000","last_became_peered":"2024-08-28T19:19:08.623745+0000","last_unstale":"2024-08-28T19:19:08.624748+0000","last_undegraded":"2024-08-28T19:19:08.624748+0000","last_fullsized":"2024-08-28T19:19:08.624748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:56.657173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619629+0000","last_change":"2024-08-28T19:18:32.499038+0000","last_active":"2024-08-28T19:19:08.619629+0000","last_peered":"2024-08-28T19:19:08.619629+0000","last_clean":"2024-08-28T19:19:08.619629+0000","last_became_active":"2024-08-28T19:18:32.498276+0000","last_became_peered":"2024-08-28T19:18:32.498276+0000","last_unstale":"2024-08-28T19:19:08.619629+0000","last_undegraded":"2024-08-28T19:19:08.619629+0000","last_fullsized":"2024-08-28T19:19:08.619629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:17:45.468448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612960+0000","last_change":"2024-08-28T19:18:32.492183+0000","last_active":"2024-08-28T19:19:08.612960+0000","last_peered":"2024-08-28T19:19:08.612960+0000","last_clean":"2024-08-28T19:19:08.612960+0000","last_became_active":"2024-08-28T19:18:32.491815+0000","last_became_peered":"2024-08-28T19:18:32.491815+0000","last_unstale":"2024-08-28T19:19:08.612960+0000","last_undegraded":"2024-08-28T19:19:08.612960+0000","last_fullsized":"2024-08-28T19:19:08.612960+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:13:11.183955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629556+0000","last_change":"2024-08-28T19:18:32.497399+0000","last_active":"2024-08-28T19:19:08.629556+0000","last_peered":"2024-08-28T19:19:08.629556+0000","last_clean":"2024-08-28T19:19:08.629556+0000","last_became_active":"2024-08-28T19:18:32.496901+0000","last_became_peered":"2024-08-28T19:18:32.496901+0000","last_unstale":"2024-08-28T19:19:08.629556+0000","last_undegraded":"2024-08-28T19:19:08.629556+0000","last_fullsized":"2024-08-28T19:19:08.629556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:05:57.634590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631412+0000","last_change":"2024-08-28T19:18:32.495871+0000","last_active":"2024-08-28T19:19:08.631412+0000","last_peered":"2024-08-28T19:19:08.631412+0000","last_clean":"2024-08-28T19:19:08.631412+0000","last_became_active":"2024-08-28T19:18:32.495515+0000","last_became_peered":"2024-08-28T19:18:32.495515+0000","last_unstale":"2024-08-28T19:19:08.631412+0000","last_undegraded":"2024-08-28T19:19:08.631412+0000","last_fullsized":"2024-08-28T19:19:08.631412+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:53:56.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617058+0000","last_change":"2024-08-28T19:18:32.495711+0000","last_active":"2024-08-28T19:19:08.617058+0000","last_peered":"2024-08-28T19:19:08.617058+0000","last_clean":"2024-08-28T19:19:08.617058+0000","last_became_active":"2024-08-28T19:18:32.495271+0000","last_became_peered":"2024-08-28T19:18:32.495271+0000","last_unstale":"2024-08-28T19:19:08.617058+0000","last_undegraded":"2024-08-28T19:19:08.617058+0000","last_fullsized":"2024-08-28T19:19:08.617058+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:05.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612310+0000","last_change":"2024-08-28T19:18:32.489731+0000","last_active":"2024-08-28T19:19:08.612310+0000","last_peered":"2024-08-28T19:19:08.612310+0000","last_clean":"2024-08-28T19:19:08.612310+0000","last_became_active":"2024-08-28T19:18:32.489366+0000","last_became_peered":"2024-08-28T19:18:32.489366+0000","last_unstale":"2024-08-28T19:19:08.612310+0000","last_undegraded":"2024-08-28T19:19:08.612310+0000","last_fullsized":"2024-08-28T19:19:08.612310+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:52:24.180155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617967+0000","last_change":"2024-08-28T19:18:32.490270+0000","last_active":"2024-08-28T19:19:08.617967+0000","last_peered":"2024-08-28T19:19:08.617967+0000","last_clean":"2024-08-28T19:19:08.617967+0000","last_became_active":"2024-08-28T19:18:32.489667+0000","last_became_peered":"2024-08-28T19:18:32.489667+0000","last_unstale":"2024-08-28T19:19:08.617967+0000","last_undegraded":"2024-08-28T19:19:08.617967+0000","last_fullsized":"2024-08-28T19:19:08.617967+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:18.571978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618388+0000","last_change":"2024-08-28T19:18:32.482553+0000","last_active":"2024-08-28T19:19:08.618388+0000","last_peered":"2024-08-28T19:19:08.618388+0000","last_clean":"2024-08-28T19:19:08.618388+0000","last_became_active":"2024-08-28T19:18:32.481161+0000","last_became_peered":"2024-08-28T19:18:32.481161+0000","last_unstale":"2024-08-28T19:19:08.618388+0000","last_undegraded":"2024-08-28T19:19:08.618388+0000","last_fullsized":"2024-08-28T19:19:08.618388+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:13.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628811+0000","last_change":"2024-08-28T19:18:32.492473+0000","last_active":"2024-08-28T19:19:08.628811+0000","last_peered":"2024-08-28T19:19:08.628811+0000","last_clean":"2024-08-28T19:19:08.628811+0000","last_became_active":"2024-08-28T19:18:32.491897+0000","last_became_peered":"2024-08-28T19:18:32.491897+0000","last_unstale":"2024-08-28T19:19:08.628811+0000","last_undegraded":"2024-08-28T19:19:08.628811+0000","last_fullsized":"2024-08-28T19:19:08.628811+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:24.279391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628899+0000","last_change":"2024-08-28T19:18:32.485621+0000","last_active":"2024-08-28T19:19:08.628899+0000","last_peered":"2024-08-28T19:19:08.628899+0000","last_clean":"2024-08-28T19:19:08.628899+0000","last_became_active":"2024-08-28T19:18:32.485267+0000","last_became_peered":"2024-08-28T19:18:32.485267+0000","last_unstale":"2024-08-28T19:19:08.628899+0000","last_undegraded":"2024-08-28T19:19:08.628899+0000","last_fullsized":"2024-08-28T19:19:08.628899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:30:08.932135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615578+0000","last_change":"2024-08-28T19:18:32.488457+0000","last_active":"2024-08-28T19:19:08.615578+0000","last_peered":"2024-08-28T19:19:08.615578+0000","last_clean":"2024-08-28T19:19:08.615578+0000","last_became_active":"2024-08-28T19:18:32.487555+0000","last_became_peered":"2024-08-28T19:18:32.487555+0000","last_unstale":"2024-08-28T19:19:08.615578+0000","last_undegraded":"2024-08-28T19:19:08.615578+0000","last_fullsized":"2024-08-28T19:19:08.615578+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:03:26.390481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70679,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":560,"num_read_kb":359,"num_write":249,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":212037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":68,"seq":292057776160,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":481155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":62,"seq":266287972389,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1650688,"data_stored":1134870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":57,"seq":244813135914,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":484509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":51,"seq":219043332143,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":970752,"data_stored":480791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":45,"seq":193273528372,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708284,"statfs":{"total":95995035648,"available":95957282816,"internally_reserved":0,"allocated":1560576,"data_stored":1077007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":40,"seq":171798691898,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":476308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888125,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1572864,"data_stored":1068774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051652,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":479991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215176,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40444,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704708,"statfs":{"total":95995035648,"available":95953620992,"internally_reserved":0,"allocated":962560,"data_stored":475625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":541033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":540813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705751,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":974848,"data_stored":477373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-28T19:20:07.547 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-28T19:20:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:08.218 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:08 smithi016 ceph-mon[18556]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:20:08.218 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]} : dispatch 2024-08-28T19:20:08.219 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]} : dispatch 2024-08-28T19:20:08.219 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]} : dispatch 2024-08-28T19:20:08.219 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]} : dispatch 2024-08-28T19:20:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:08 smithi028 ceph-mon[21769]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:20:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]} : dispatch 2024-08-28T19:20:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]} : dispatch 2024-08-28T19:20:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]} : dispatch 2024-08-28T19:20:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]} : dispatch 2024-08-28T19:20:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.308+0000 7f4413c16640 1 -- 172.21.15.16:0/775222083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440c102840 msgr2=0x7f440c102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.308+0000 7f4413c16640 1 --2- 172.21.15.16:0/775222083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440c102840 0x7f440c102c50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4408009900 tx=0x7f440802a410 comp rx=0 tx=0).stop 2024-08-28T19:20:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.308+0000 7f4413c16640 1 -- 172.21.15.16:0/775222083 shutdown_connections 2024-08-28T19:20:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.308+0000 7f4413c16640 1 --2- 172.21.15.16:0/775222083 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440c108ad0 0x7f440c10f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.308+0000 7f4413c16640 1 --2- 172.21.15.16:0/775222083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c108180 0x7f440c108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.308+0000 7f4413c16640 1 --2- 172.21.15.16:0/775222083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440c102840 0x7f440c102c50 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.308+0000 7f4413c16640 1 -- 172.21.15.16:0/775222083 >> 172.21.15.16:0/775222083 conn(0x7f440c079580 msgr2=0x7f440c079990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.308+0000 7f4413c16640 1 -- 172.21.15.16:0/775222083 shutdown_connections 2024-08-28T19:20:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.308+0000 7f4413c16640 1 -- 172.21.15.16:0/775222083 wait complete. 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f4413c16640 1 Processor -- start 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f4413c16640 1 -- start start 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f4413c16640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440c102840 0x7f440c104030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f4413c16640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440c108180 0x7f440c104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f4413c16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c108ad0 0x7f440c104ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f4413c16640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f440c112730 con 0x7f440c108180 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f4413c16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f440c1125b0 con 0x7f440c108ad0 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f4413c16640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f440c1128b0 con 0x7f440c102840 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f44111b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440c108180 0x7f440c104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f44111b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440c108180 0x7f440c104570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51808/0 (socket says 172.21.15.16:51808) 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f44111b1640 1 -- 172.21.15.16:0/1140536790 learned_addr learned my addr 172.21.15.16:0/1140536790 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.309+0000 7f44119b2640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440c102840 0x7f440c104030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f44121b3640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c108ad0 0x7f440c104ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f44111b1640 1 -- 172.21.15.16:0/1140536790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440c102840 msgr2=0x7f440c104030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f44111b1640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440c102840 0x7f440c104030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f44111b1640 1 -- 172.21.15.16:0/1140536790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c108ad0 msgr2=0x7f440c104ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f44111b1640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c108ad0 0x7f440c104ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f44111b1640 1 -- 172.21.15.16:0/1140536790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c1a8440 con 0x7f440c108180 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f44119b2640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440c102840 0x7f440c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f44121b3640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c108ad0 0x7f440c104ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f44111b1640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440c108180 0x7f440c104570 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f440000d770 tx=0x7f440000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f43feffd640 1 -- 172.21.15.16:0/1140536790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4400017070 con 0x7f440c108180 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f43feffd640 1 -- 172.21.15.16:0/1140536790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4400009c30 con 0x7f440c108180 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f4413c16640 1 -- 172.21.15.16:0/1140536790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f440c1a8730 con 0x7f440c108180 2024-08-28T19:20:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f43feffd640 1 -- 172.21.15.16:0/1140536790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440000f620 con 0x7f440c108180 2024-08-28T19:20:08.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.310+0000 7f4413c16640 1 -- 172.21.15.16:0/1140536790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440c1a8c70 con 0x7f440c108180 2024-08-28T19:20:08.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.311+0000 7f4413c16640 1 -- 172.21.15.16:0/1140536790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440c102c50 con 0x7f440c108180 2024-08-28T19:20:08.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.313+0000 7f43feffd640 1 -- 172.21.15.16:0/1140536790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f440001a070 con 0x7f440c108180 2024-08-28T19:20:08.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.314+0000 7f43feffd640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43f80b4c10 0x7f43f80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:08.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.314+0000 7f43feffd640 1 -- 172.21.15.16:0/1140536790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9402+0+0 (secure 0 0 0) 0x7f44000e5400 con 0x7f440c108180 2024-08-28T19:20:08.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.314+0000 7f44119b2640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43f80b4c10 0x7f43f80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:08.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.314+0000 7f44119b2640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43f80b4c10 0x7f43f80b7120 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4408000c00 tx=0x7f4408009680 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:08.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.317+0000 7f43feffd640 1 -- 172.21.15.16:0/1140536790 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f4400009de0 con 0x7f440c108180 2024-08-28T19:20:08.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:08 smithi115 ceph-mon[22184]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:20:08.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]} : dispatch 2024-08-28T19:20:08.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]} : dispatch 2024-08-28T19:20:08.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]} : dispatch 2024-08-28T19:20:08.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]} : dispatch 2024-08-28T19:20:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.503+0000 7f4413c16640 1 -- 172.21.15.16:0/1140536790 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f440c106120 con 0x7f43f80b4c10 2024-08-28T19:20:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.509+0000 7f43feffd640 1 -- 172.21.15.16:0/1140536790 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340390 (secure 0 0 0) 0x7f440c106120 con 0x7f43f80b4c10 2024-08-28T19:20:08.514 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-08-28T19:20:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 -- 172.21.15.16:0/1140536790 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43f80b4c10 msgr2=0x7f43f80b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43f80b4c10 0x7f43f80b7120 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4408000c00 tx=0x7f4408009680 comp rx=0 tx=0).stop 2024-08-28T19:20:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 -- 172.21.15.16:0/1140536790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440c108180 msgr2=0x7f440c104570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440c108180 0x7f440c104570 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f440000d770 tx=0x7f440000dc30 comp rx=0 tx=0).stop 2024-08-28T19:20:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 -- 172.21.15.16:0/1140536790 shutdown_connections 2024-08-28T19:20:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43f80b4c10 0x7f43f80b7120 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c108ad0 0x7f440c104ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f440c108180 0x7f440c104570 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 --2- 172.21.15.16:0/1140536790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f440c102840 0x7f440c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 -- 172.21.15.16:0/1140536790 >> 172.21.15.16:0/1140536790 conn(0x7f440c079580 msgr2=0x7f440c1017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 -- 172.21.15.16:0/1140536790 shutdown_connections 2024-08-28T19:20:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:08.513+0000 7f4413c16640 1 -- 172.21.15.16:0/1140536790 wait complete. 2024-08-28T19:20:09.190 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:09 smithi016 ceph-mon[18556]: pgmap v310: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:20:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-08-28T19:20:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-08-28T19:20:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-08-28T19:20:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-08-28T19:20:09.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:09 smithi016 ceph-mon[18556]: osdmap e83: 12 total, 12 up, 12 in 2024-08-28T19:20:09.219 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-08-28T19:20:08.003880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661281,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":651,"num_read_kb":433,"num_write":374,"num_write_kb":2294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":308,"ondisk_log_size":308,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":412448,"kb_used_data":13280,"kb_used_omap":18,"kb_used_meta":398957,"kb_avail":1124529376,"statfs":{"total":1151940427776,"available":1151518081024,"internally_reserved":0,"allocated":13598720,"data_stored":7718249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":408532282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.526905"},"pg_stats":[{"pgid":"2.1f","version":"82'13","reported_seq":116,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:04.733068+0000","last_change":"2024-08-28T19:19:08.633326+0000","last_active":"2024-08-28T19:20:04.733068+0000","last_peered":"2024-08-28T19:20:04.733068+0000","last_clean":"2024-08-28T19:20:04.733068+0000","last_became_active":"2024-08-28T19:19:08.623876+0000","last_became_peered":"2024-08-28T19:19:08.623876+0000","last_unstale":"2024-08-28T19:20:04.733068+0000","last_undegraded":"2024-08-28T19:20:04.733068+0000","last_fullsized":"2024-08-28T19:20:04.733068+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:19:50.632337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":55,"num_read_kb":55,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632322+0000","last_change":"2024-08-28T19:18:32.492604+0000","last_active":"2024-08-28T19:19:08.632322+0000","last_peered":"2024-08-28T19:19:08.632322+0000","last_clean":"2024-08-28T19:19:08.632322+0000","last_became_active":"2024-08-28T19:18:32.489427+0000","last_became_peered":"2024-08-28T19:18:32.489427+0000","last_unstale":"2024-08-28T19:19:08.632322+0000","last_undegraded":"2024-08-28T19:19:08.632322+0000","last_fullsized":"2024-08-28T19:19:08.632322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:36:14.841315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622224+0000","last_change":"2024-08-28T19:18:39.528815+0000","last_active":"2024-08-28T19:19:08.622224+0000","last_peered":"2024-08-28T19:19:08.622224+0000","last_clean":"2024-08-28T19:19:08.622224+0000","last_became_active":"2024-08-28T19:18:39.527781+0000","last_became_peered":"2024-08-28T19:18:39.527781+0000","last_unstale":"2024-08-28T19:19:08.622224+0000","last_undegraded":"2024-08-28T19:19:08.622224+0000","last_fullsized":"2024-08-28T19:19:08.622224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:39.831426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632931+0000","last_change":"2024-08-28T19:18:31.082641+0000","last_active":"2024-08-28T19:19:08.632931+0000","last_peered":"2024-08-28T19:19:08.632931+0000","last_clean":"2024-08-28T19:19:08.632931+0000","last_became_active":"2024-08-28T19:18:31.082260+0000","last_became_peered":"2024-08-28T19:18:31.082260+0000","last_unstale":"2024-08-28T19:19:08.632931+0000","last_undegraded":"2024-08-28T19:19:08.632931+0000","last_fullsized":"2024-08-28T19:19:08.632931+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:09:55.836463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629290+0000","last_change":"2024-08-28T19:18:32.493498+0000","last_active":"2024-08-28T19:19:08.629290+0000","last_peered":"2024-08-28T19:19:08.629290+0000","last_clean":"2024-08-28T19:19:08.629290+0000","last_became_active":"2024-08-28T19:18:32.490969+0000","last_became_peered":"2024-08-28T19:18:32.490969+0000","last_unstale":"2024-08-28T19:19:08.629290+0000","last_undegraded":"2024-08-28T19:19:08.629290+0000","last_fullsized":"2024-08-28T19:19:08.629290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:34:29.537622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617571+0000","last_change":"2024-08-28T19:18:39.520922+0000","last_active":"2024-08-28T19:19:08.617571+0000","last_peered":"2024-08-28T19:19:08.617571+0000","last_clean":"2024-08-28T19:19:08.617571+0000","last_became_active":"2024-08-28T19:18:39.520362+0000","last_became_peered":"2024-08-28T19:18:39.520362+0000","last_unstale":"2024-08-28T19:19:08.617571+0000","last_undegraded":"2024-08-28T19:19:08.617571+0000","last_fullsized":"2024-08-28T19:19:08.617571+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:40.658820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.646751+0000","last_change":"2024-08-28T19:18:30.471870+0000","last_active":"2024-08-28T19:19:12.646751+0000","last_peered":"2024-08-28T19:19:12.646751+0000","last_clean":"2024-08-28T19:19:12.646751+0000","last_became_active":"2024-08-28T19:18:30.471438+0000","last_became_peered":"2024-08-28T19:18:30.471438+0000","last_unstale":"2024-08-28T19:19:12.646751+0000","last_undegraded":"2024-08-28T19:19:12.646751+0000","last_fullsized":"2024-08-28T19:19:12.646751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:44:34.137395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628034+0000","last_change":"2024-08-28T19:18:32.486485+0000","last_active":"2024-08-28T19:19:08.628034+0000","last_peered":"2024-08-28T19:19:08.628034+0000","last_clean":"2024-08-28T19:19:08.628034+0000","last_became_active":"2024-08-28T19:18:32.485888+0000","last_became_peered":"2024-08-28T19:18:32.485888+0000","last_unstale":"2024-08-28T19:19:08.628034+0000","last_undegraded":"2024-08-28T19:19:08.628034+0000","last_fullsized":"2024-08-28T19:19:08.628034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:45:48.757318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.633437+0000","last_change":"2024-08-28T19:19:08.633317+0000","last_active":"2024-08-28T19:19:08.633437+0000","last_peered":"2024-08-28T19:19:08.633437+0000","last_clean":"2024-08-28T19:19:08.633437+0000","last_became_active":"2024-08-28T19:19:08.632689+0000","last_became_peered":"2024-08-28T19:19:08.632689+0000","last_unstale":"2024-08-28T19:19:08.633437+0000","last_undegraded":"2024-08-28T19:19:08.633437+0000","last_fullsized":"2024-08-28T19:19:08.633437+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:35:08.740645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653995+0000","last_change":"2024-08-28T19:18:30.473982+0000","last_active":"2024-08-28T19:19:11.653995+0000","last_peered":"2024-08-28T19:19:11.653995+0000","last_clean":"2024-08-28T19:19:11.653995+0000","last_became_active":"2024-08-28T19:18:30.473552+0000","last_became_peered":"2024-08-28T19:18:30.473552+0000","last_unstale":"2024-08-28T19:19:11.653995+0000","last_undegraded":"2024-08-28T19:19:11.653995+0000","last_fullsized":"2024-08-28T19:19:11.653995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:22:29.435517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616754+0000","last_change":"2024-08-28T19:18:32.487881+0000","last_active":"2024-08-28T19:19:08.616754+0000","last_peered":"2024-08-28T19:19:08.616754+0000","last_clean":"2024-08-28T19:19:08.616754+0000","last_became_active":"2024-08-28T19:18:32.487039+0000","last_became_peered":"2024-08-28T19:18:32.487039+0000","last_unstale":"2024-08-28T19:19:08.616754+0000","last_undegraded":"2024-08-28T19:19:08.616754+0000","last_fullsized":"2024-08-28T19:19:08.616754+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:23:35.757811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617767+0000","last_change":"2024-08-28T19:18:39.518654+0000","last_active":"2024-08-28T19:19:08.617767+0000","last_peered":"2024-08-28T19:19:08.617767+0000","last_clean":"2024-08-28T19:19:08.617767+0000","last_became_active":"2024-08-28T19:18:39.518087+0000","last_became_peered":"2024-08-28T19:18:39.518087+0000","last_unstale":"2024-08-28T19:19:08.617767+0000","last_undegraded":"2024-08-28T19:19:08.617767+0000","last_fullsized":"2024-08-28T19:19:08.617767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:08.284865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617577+0000","last_change":"2024-08-28T19:18:32.491928+0000","last_active":"2024-08-28T19:19:08.617577+0000","last_peered":"2024-08-28T19:19:08.617577+0000","last_clean":"2024-08-28T19:19:08.617577+0000","last_became_active":"2024-08-28T19:18:32.491538+0000","last_became_peered":"2024-08-28T19:18:32.491538+0000","last_unstale":"2024-08-28T19:19:08.617577+0000","last_undegraded":"2024-08-28T19:19:08.617577+0000","last_fullsized":"2024-08-28T19:19:08.617577+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:39:06.077830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.955289+0000","last_change":"2024-08-28T19:19:08.634752+0000","last_active":"2024-08-28T19:19:11.955289+0000","last_peered":"2024-08-28T19:19:11.955289+0000","last_clean":"2024-08-28T19:19:11.955289+0000","last_became_active":"2024-08-28T19:19:08.620784+0000","last_became_peered":"2024-08-28T19:19:08.620784+0000","last_unstale":"2024-08-28T19:19:11.955289+0000","last_undegraded":"2024-08-28T19:19:11.955289+0000","last_fullsized":"2024-08-28T19:19:11.955289+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:51.640445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618419+0000","last_change":"2024-08-28T19:18:39.681413+0000","last_active":"2024-08-28T19:19:08.618419+0000","last_peered":"2024-08-28T19:19:08.618419+0000","last_clean":"2024-08-28T19:19:08.618419+0000","last_became_active":"2024-08-28T19:18:39.680793+0000","last_became_peered":"2024-08-28T19:18:39.680793+0000","last_unstale":"2024-08-28T19:19:08.618419+0000","last_undegraded":"2024-08-28T19:19:08.618419+0000","last_fullsized":"2024-08-28T19:19:08.618419+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:45:43.381019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632212+0000","last_change":"2024-08-28T19:18:32.486739+0000","last_active":"2024-08-28T19:19:08.632212+0000","last_peered":"2024-08-28T19:19:08.632212+0000","last_clean":"2024-08-28T19:19:08.632212+0000","last_became_active":"2024-08-28T19:18:32.486347+0000","last_became_peered":"2024-08-28T19:18:32.486347+0000","last_unstale":"2024-08-28T19:19:08.632212+0000","last_undegraded":"2024-08-28T19:19:08.632212+0000","last_fullsized":"2024-08-28T19:19:08.632212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:27:36.448569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.641964+0000","last_change":"2024-08-28T19:18:30.467690+0000","last_active":"2024-08-28T19:19:10.641964+0000","last_peered":"2024-08-28T19:19:10.641964+0000","last_clean":"2024-08-28T19:19:10.641964+0000","last_became_active":"2024-08-28T19:18:30.467249+0000","last_became_peered":"2024-08-28T19:18:30.467249+0000","last_unstale":"2024-08-28T19:19:10.641964+0000","last_undegraded":"2024-08-28T19:19:10.641964+0000","last_fullsized":"2024-08-28T19:19:10.641964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:37:28.876666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618227+0000","last_change":"2024-08-28T19:18:39.523889+0000","last_active":"2024-08-28T19:19:08.618227+0000","last_peered":"2024-08-28T19:19:08.618227+0000","last_clean":"2024-08-28T19:19:08.618227+0000","last_became_active":"2024-08-28T19:18:39.523384+0000","last_became_peered":"2024-08-28T19:18:39.523384+0000","last_unstale":"2024-08-28T19:19:08.618227+0000","last_undegraded":"2024-08-28T19:19:08.618227+0000","last_fullsized":"2024-08-28T19:19:08.618227+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:57:32.277680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614254+0000","last_change":"2024-08-28T19:18:32.482257+0000","last_active":"2024-08-28T19:19:08.614254+0000","last_peered":"2024-08-28T19:19:08.614254+0000","last_clean":"2024-08-28T19:19:08.614254+0000","last_became_active":"2024-08-28T19:18:32.481884+0000","last_became_peered":"2024-08-28T19:18:32.481884+0000","last_unstale":"2024-08-28T19:19:08.614254+0000","last_undegraded":"2024-08-28T19:19:08.614254+0000","last_fullsized":"2024-08-28T19:19:08.614254+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:56:28.754369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651638+0000","last_change":"2024-08-28T19:18:30.467307+0000","last_active":"2024-08-28T19:19:11.651638+0000","last_peered":"2024-08-28T19:19:11.651638+0000","last_clean":"2024-08-28T19:19:11.651638+0000","last_became_active":"2024-08-28T19:18:30.466942+0000","last_became_peered":"2024-08-28T19:18:30.466942+0000","last_unstale":"2024-08-28T19:19:11.651638+0000","last_undegraded":"2024-08-28T19:19:11.651638+0000","last_fullsized":"2024-08-28T19:19:11.651638+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:05:23.682110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614513+0000","last_change":"2024-08-28T19:18:39.531421+0000","last_active":"2024-08-28T19:19:08.614513+0000","last_peered":"2024-08-28T19:19:08.614513+0000","last_clean":"2024-08-28T19:19:08.614513+0000","last_became_active":"2024-08-28T19:18:39.531008+0000","last_became_peered":"2024-08-28T19:18:39.531008+0000","last_unstale":"2024-08-28T19:19:08.614513+0000","last_undegraded":"2024-08-28T19:19:08.614513+0000","last_fullsized":"2024-08-28T19:19:08.614513+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:46:49.580601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618946+0000","last_change":"2024-08-28T19:18:32.485204+0000","last_active":"2024-08-28T19:19:08.618946+0000","last_peered":"2024-08-28T19:19:08.618946+0000","last_clean":"2024-08-28T19:19:08.618946+0000","last_became_active":"2024-08-28T19:18:32.484830+0000","last_became_peered":"2024-08-28T19:18:32.484830+0000","last_unstale":"2024-08-28T19:19:08.618946+0000","last_undegraded":"2024-08-28T19:19:08.618946+0000","last_fullsized":"2024-08-28T19:19:08.618946+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:55:41.358718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632011+0000","last_change":"2024-08-28T19:18:31.082361+0000","last_active":"2024-08-28T19:19:08.632011+0000","last_peered":"2024-08-28T19:19:08.632011+0000","last_clean":"2024-08-28T19:19:08.632011+0000","last_became_active":"2024-08-28T19:18:31.081930+0000","last_became_peered":"2024-08-28T19:18:31.081930+0000","last_unstale":"2024-08-28T19:19:08.632011+0000","last_undegraded":"2024-08-28T19:19:08.632011+0000","last_fullsized":"2024-08-28T19:19:08.632011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:41:05.543088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618390+0000","last_change":"2024-08-28T19:18:39.682341+0000","last_active":"2024-08-28T19:19:08.618390+0000","last_peered":"2024-08-28T19:19:08.618390+0000","last_clean":"2024-08-28T19:19:08.618390+0000","last_became_active":"2024-08-28T19:18:39.681751+0000","last_became_peered":"2024-08-28T19:18:39.681751+0000","last_unstale":"2024-08-28T19:19:08.618390+0000","last_undegraded":"2024-08-28T19:19:08.618390+0000","last_fullsized":"2024-08-28T19:19:08.618390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:51:24.371270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628950+0000","last_change":"2024-08-28T19:18:32.491069+0000","last_active":"2024-08-28T19:19:08.628950+0000","last_peered":"2024-08-28T19:19:08.628950+0000","last_clean":"2024-08-28T19:19:08.628950+0000","last_became_active":"2024-08-28T19:18:32.490688+0000","last_became_peered":"2024-08-28T19:18:32.490688+0000","last_unstale":"2024-08-28T19:19:08.628950+0000","last_undegraded":"2024-08-28T19:19:08.628950+0000","last_fullsized":"2024-08-28T19:19:08.628950+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:24:31.560519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652924+0000","last_change":"2024-08-28T19:18:30.739812+0000","last_active":"2024-08-28T19:19:11.652924+0000","last_peered":"2024-08-28T19:19:11.652924+0000","last_clean":"2024-08-28T19:19:11.652924+0000","last_became_active":"2024-08-28T19:18:30.739414+0000","last_became_peered":"2024-08-28T19:18:30.739414+0000","last_unstale":"2024-08-28T19:19:11.652924+0000","last_undegraded":"2024-08-28T19:19:11.652924+0000","last_fullsized":"2024-08-28T19:19:11.652924+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:56:56.594993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622469+0000","last_change":"2024-08-28T19:18:39.528281+0000","last_active":"2024-08-28T19:19:08.622469+0000","last_peered":"2024-08-28T19:19:08.622469+0000","last_clean":"2024-08-28T19:19:08.622469+0000","last_became_active":"2024-08-28T19:18:39.527142+0000","last_became_peered":"2024-08-28T19:18:39.527142+0000","last_unstale":"2024-08-28T19:19:08.622469+0000","last_undegraded":"2024-08-28T19:19:08.622469+0000","last_fullsized":"2024-08-28T19:19:08.622469+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:44:19.123662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632709+0000","last_change":"2024-08-28T19:18:32.490430+0000","last_active":"2024-08-28T19:19:08.632709+0000","last_peered":"2024-08-28T19:19:08.632709+0000","last_clean":"2024-08-28T19:19:08.632709+0000","last_became_active":"2024-08-28T19:18:32.490020+0000","last_became_peered":"2024-08-28T19:18:32.490020+0000","last_unstale":"2024-08-28T19:19:08.632709+0000","last_undegraded":"2024-08-28T19:19:08.632709+0000","last_fullsized":"2024-08-28T19:19:08.632709+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:52:31.153012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"82'15","reported_seq":50,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:25.488292+0000","last_change":"2024-08-28T19:18:30.467060+0000","last_active":"2024-08-28T19:19:25.488292+0000","last_peered":"2024-08-28T19:19:25.488292+0000","last_clean":"2024-08-28T19:19:25.488292+0000","last_became_active":"2024-08-28T19:18:30.466689+0000","last_became_peered":"2024-08-28T19:18:30.466689+0000","last_unstale":"2024-08-28T19:19:25.488292+0000","last_undegraded":"2024-08-28T19:19:25.488292+0000","last_fullsized":"2024-08-28T19:19:25.488292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:18:36.508131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.630939+0000","last_change":"2024-08-28T19:18:39.682874+0000","last_active":"2024-08-28T19:19:08.630939+0000","last_peered":"2024-08-28T19:19:08.630939+0000","last_clean":"2024-08-28T19:19:08.630939+0000","last_became_active":"2024-08-28T19:18:39.681500+0000","last_became_peered":"2024-08-28T19:18:39.681500+0000","last_unstale":"2024-08-28T19:19:08.630939+0000","last_undegraded":"2024-08-28T19:19:08.630939+0000","last_fullsized":"2024-08-28T19:19:08.630939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:45:41.815506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612195+0000","last_change":"2024-08-28T19:18:32.482047+0000","last_active":"2024-08-28T19:19:08.612195+0000","last_peered":"2024-08-28T19:19:08.612195+0000","last_clean":"2024-08-28T19:19:08.612195+0000","last_became_active":"2024-08-28T19:18:32.481640+0000","last_became_peered":"2024-08-28T19:18:32.481640+0000","last_unstale":"2024-08-28T19:19:08.612195+0000","last_undegraded":"2024-08-28T19:19:08.612195+0000","last_fullsized":"2024-08-28T19:19:08.612195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:34.717764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651944+0000","last_change":"2024-08-28T19:18:31.083524+0000","last_active":"2024-08-28T19:19:11.651944+0000","last_peered":"2024-08-28T19:19:11.651944+0000","last_clean":"2024-08-28T19:19:11.651944+0000","last_became_active":"2024-08-28T19:18:31.083106+0000","last_became_peered":"2024-08-28T19:18:31.083106+0000","last_unstale":"2024-08-28T19:19:11.651944+0000","last_undegraded":"2024-08-28T19:19:11.651944+0000","last_fullsized":"2024-08-28T19:19:11.651944+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:16.180754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615954+0000","last_change":"2024-08-28T19:18:39.531884+0000","last_active":"2024-08-28T19:19:08.615954+0000","last_peered":"2024-08-28T19:19:08.615954+0000","last_clean":"2024-08-28T19:19:08.615954+0000","last_became_active":"2024-08-28T19:18:39.531255+0000","last_became_peered":"2024-08-28T19:18:39.531255+0000","last_unstale":"2024-08-28T19:19:08.615954+0000","last_undegraded":"2024-08-28T19:19:08.615954+0000","last_fullsized":"2024-08-28T19:19:08.615954+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:36:55.496594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612691+0000","last_change":"2024-08-28T19:18:32.485180+0000","last_active":"2024-08-28T19:19:08.612691+0000","last_peered":"2024-08-28T19:19:08.612691+0000","last_clean":"2024-08-28T19:19:08.612691+0000","last_became_active":"2024-08-28T19:18:32.484804+0000","last_became_peered":"2024-08-28T19:18:32.484804+0000","last_unstale":"2024-08-28T19:19:08.612691+0000","last_undegraded":"2024-08-28T19:19:08.612691+0000","last_fullsized":"2024-08-28T19:19:08.612691+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:12:40.958251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615096+0000","last_change":"2024-08-28T19:18:31.082226+0000","last_active":"2024-08-28T19:19:08.615096+0000","last_peered":"2024-08-28T19:19:08.615096+0000","last_clean":"2024-08-28T19:19:08.615096+0000","last_became_active":"2024-08-28T19:18:31.081833+0000","last_became_peered":"2024-08-28T19:18:31.081833+0000","last_unstale":"2024-08-28T19:19:08.615096+0000","last_undegraded":"2024-08-28T19:19:08.615096+0000","last_fullsized":"2024-08-28T19:19:08.615096+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:33:56.298093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619013+0000","last_change":"2024-08-28T19:18:39.681288+0000","last_active":"2024-08-28T19:19:08.619013+0000","last_peered":"2024-08-28T19:19:08.619013+0000","last_clean":"2024-08-28T19:19:08.619013+0000","last_became_active":"2024-08-28T19:18:39.680719+0000","last_became_peered":"2024-08-28T19:18:39.680719+0000","last_unstale":"2024-08-28T19:19:08.619013+0000","last_undegraded":"2024-08-28T19:19:08.619013+0000","last_fullsized":"2024-08-28T19:19:08.619013+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:53:35.554181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612370+0000","last_change":"2024-08-28T19:18:32.487105+0000","last_active":"2024-08-28T19:19:08.612370+0000","last_peered":"2024-08-28T19:19:08.612370+0000","last_clean":"2024-08-28T19:19:08.612370+0000","last_became_active":"2024-08-28T19:18:32.486643+0000","last_became_peered":"2024-08-28T19:18:32.486643+0000","last_unstale":"2024-08-28T19:19:08.612370+0000","last_undegraded":"2024-08-28T19:19:08.612370+0000","last_fullsized":"2024-08-28T19:19:08.612370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:18:43.274493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'6","reported_seq":96,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:03.732349+0000","last_change":"2024-08-28T19:18:30.471447+0000","last_active":"2024-08-28T19:20:03.732349+0000","last_peered":"2024-08-28T19:20:03.732349+0000","last_clean":"2024-08-28T19:20:03.732349+0000","last_became_active":"2024-08-28T19:18:30.470798+0000","last_became_peered":"2024-08-28T19:18:30.470798+0000","last_unstale":"2024-08-28T19:20:03.732349+0000","last_undegraded":"2024-08-28T19:20:03.732349+0000","last_fullsized":"2024-08-28T19:20:03.732349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:50:13.948439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611755+0000","last_change":"2024-08-28T19:18:39.524190+0000","last_active":"2024-08-28T19:19:08.611755+0000","last_peered":"2024-08-28T19:19:08.611755+0000","last_clean":"2024-08-28T19:19:08.611755+0000","last_became_active":"2024-08-28T19:18:39.523771+0000","last_became_peered":"2024-08-28T19:18:39.523771+0000","last_unstale":"2024-08-28T19:19:08.611755+0000","last_undegraded":"2024-08-28T19:19:08.611755+0000","last_fullsized":"2024-08-28T19:19:08.611755+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:43:43.984625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615282+0000","last_change":"2024-08-28T19:18:32.487212+0000","last_active":"2024-08-28T19:19:08.615282+0000","last_peered":"2024-08-28T19:19:08.615282+0000","last_clean":"2024-08-28T19:19:08.615282+0000","last_became_active":"2024-08-28T19:18:32.485432+0000","last_became_peered":"2024-08-28T19:18:32.485432+0000","last_unstale":"2024-08-28T19:19:08.615282+0000","last_undegraded":"2024-08-28T19:19:08.615282+0000","last_fullsized":"2024-08-28T19:19:08.615282+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:52:11.672193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":94,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:03.732659+0000","last_change":"2024-08-28T19:18:31.082210+0000","last_active":"2024-08-28T19:20:03.732659+0000","last_peered":"2024-08-28T19:20:03.732659+0000","last_clean":"2024-08-28T19:20:03.732659+0000","last_became_active":"2024-08-28T19:18:31.081814+0000","last_became_peered":"2024-08-28T19:18:31.081814+0000","last_unstale":"2024-08-28T19:20:03.732659+0000","last_undegraded":"2024-08-28T19:20:03.732659+0000","last_fullsized":"2024-08-28T19:20:03.732659+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:33:32.281431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":54,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611523+0000","last_change":"2024-08-28T19:18:39.681858+0000","last_active":"2024-08-28T19:19:08.611523+0000","last_peered":"2024-08-28T19:19:08.611523+0000","last_clean":"2024-08-28T19:19:08.611523+0000","last_became_active":"2024-08-28T19:18:39.681382+0000","last_became_peered":"2024-08-28T19:18:39.681382+0000","last_unstale":"2024-08-28T19:19:08.611523+0000","last_undegraded":"2024-08-28T19:19:08.611523+0000","last_fullsized":"2024-08-28T19:19:08.611523+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:23:22.021022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611190+0000","last_change":"2024-08-28T19:18:32.486802+0000","last_active":"2024-08-28T19:19:08.611190+0000","last_peered":"2024-08-28T19:19:08.611190+0000","last_clean":"2024-08-28T19:19:08.611190+0000","last_became_active":"2024-08-28T19:18:32.486479+0000","last_became_peered":"2024-08-28T19:18:32.486479+0000","last_unstale":"2024-08-28T19:19:08.611190+0000","last_undegraded":"2024-08-28T19:19:08.611190+0000","last_fullsized":"2024-08-28T19:19:08.611190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:17:17.654757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:09.635071+0000","last_change":"2024-08-28T19:18:30.470447+0000","last_active":"2024-08-28T19:19:09.635071+0000","last_peered":"2024-08-28T19:19:09.635071+0000","last_clean":"2024-08-28T19:19:09.635071+0000","last_became_active":"2024-08-28T19:18:30.469862+0000","last_became_peered":"2024-08-28T19:18:30.469862+0000","last_unstale":"2024-08-28T19:19:09.635071+0000","last_undegraded":"2024-08-28T19:19:09.635071+0000","last_fullsized":"2024-08-28T19:19:09.635071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:27:46.660492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624752+0000","last_change":"2024-08-28T19:19:08.624635+0000","last_active":"2024-08-28T19:19:08.624752+0000","last_peered":"2024-08-28T19:19:08.624752+0000","last_clean":"2024-08-28T19:19:08.624752+0000","last_became_active":"2024-08-28T19:19:08.622845+0000","last_became_peered":"2024-08-28T19:19:08.622845+0000","last_unstale":"2024-08-28T19:19:08.624752+0000","last_undegraded":"2024-08-28T19:19:08.624752+0000","last_fullsized":"2024-08-28T19:19:08.624752+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:56:40.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631918+0000","last_change":"2024-08-28T19:18:32.487947+0000","last_active":"2024-08-28T19:19:08.631918+0000","last_peered":"2024-08-28T19:19:08.631918+0000","last_clean":"2024-08-28T19:19:08.631918+0000","last_became_active":"2024-08-28T19:18:32.487592+0000","last_became_peered":"2024-08-28T19:18:32.487592+0000","last_unstale":"2024-08-28T19:19:08.631918+0000","last_undegraded":"2024-08-28T19:19:08.631918+0000","last_fullsized":"2024-08-28T19:19:08.631918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:30:29.550644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.654410+0000","last_change":"2024-08-28T19:18:30.465044+0000","last_active":"2024-08-28T19:19:12.654410+0000","last_peered":"2024-08-28T19:19:12.654410+0000","last_clean":"2024-08-28T19:19:12.654410+0000","last_became_active":"2024-08-28T19:18:30.464580+0000","last_became_peered":"2024-08-28T19:18:30.464580+0000","last_unstale":"2024-08-28T19:19:12.654410+0000","last_undegraded":"2024-08-28T19:19:12.654410+0000","last_fullsized":"2024-08-28T19:19:12.654410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:21:17.223311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":110,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617248+0000","last_change":"2024-08-28T19:17:17.063316+0000","last_active":"2024-08-28T19:19:08.617248+0000","last_peered":"2024-08-28T19:19:08.617248+0000","last_clean":"2024-08-28T19:19:08.617248+0000","last_became_active":"2024-08-28T19:17:17.057091+0000","last_became_peered":"2024-08-28T19:17:17.057091+0000","last_unstale":"2024-08-28T19:19:08.617248+0000","last_undegraded":"2024-08-28T19:19:08.617248+0000","last_fullsized":"2024-08-28T19:19:08.617248+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_clean_scrub_stamp":"2024-08-28T19:13:56.815467+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:35:51.499703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"82'1","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622311+0000","last_change":"2024-08-28T19:19:08.621402+0000","last_active":"2024-08-28T19:19:08.622311+0000","last_peered":"2024-08-28T19:19:08.622311+0000","last_clean":"2024-08-28T19:19:08.622311+0000","last_became_active":"2024-08-28T19:19:08.620779+0000","last_became_peered":"2024-08-28T19:19:08.620779+0000","last_unstale":"2024-08-28T19:19:08.622311+0000","last_undegraded":"2024-08-28T19:19:08.622311+0000","last_fullsized":"2024-08-28T19:19:08.622311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:39.391414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619759+0000","last_change":"2024-08-28T19:18:32.490066+0000","last_active":"2024-08-28T19:19:08.619759+0000","last_peered":"2024-08-28T19:19:08.619759+0000","last_clean":"2024-08-28T19:19:08.619759+0000","last_became_active":"2024-08-28T19:18:32.489560+0000","last_became_peered":"2024-08-28T19:18:32.489560+0000","last_unstale":"2024-08-28T19:19:08.619759+0000","last_undegraded":"2024-08-28T19:19:08.619759+0000","last_fullsized":"2024-08-28T19:19:08.619759+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:22:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'5","reported_seq":96,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:04.734690+0000","last_change":"2024-08-28T19:18:31.083078+0000","last_active":"2024-08-28T19:20:04.734690+0000","last_peered":"2024-08-28T19:20:04.734690+0000","last_clean":"2024-08-28T19:20:04.734690+0000","last_became_active":"2024-08-28T19:18:31.082575+0000","last_became_peered":"2024-08-28T19:18:31.082575+0000","last_unstale":"2024-08-28T19:20:04.734690+0000","last_undegraded":"2024-08-28T19:20:04.734690+0000","last_fullsized":"2024-08-28T19:20:04.734690+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":56,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627532+0000","last_change":"2024-08-28T19:18:39.519814+0000","last_active":"2024-08-28T19:19:08.627532+0000","last_peered":"2024-08-28T19:19:08.627532+0000","last_clean":"2024-08-28T19:19:08.627532+0000","last_became_active":"2024-08-28T19:18:39.519258+0000","last_became_peered":"2024-08-28T19:18:39.519258+0000","last_unstale":"2024-08-28T19:19:08.627532+0000","last_undegraded":"2024-08-28T19:19:08.627532+0000","last_fullsized":"2024-08-28T19:19:08.627532+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:03:04.051376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618184+0000","last_change":"2024-08-28T19:18:32.495426+0000","last_active":"2024-08-28T19:19:08.618184+0000","last_peered":"2024-08-28T19:19:08.618184+0000","last_clean":"2024-08-28T19:19:08.618184+0000","last_became_active":"2024-08-28T19:18:32.495075+0000","last_became_peered":"2024-08-28T19:18:32.495075+0000","last_unstale":"2024-08-28T19:19:08.618184+0000","last_undegraded":"2024-08-28T19:19:08.618184+0000","last_fullsized":"2024-08-28T19:19:08.618184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:40:22.456666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652460+0000","last_change":"2024-08-28T19:18:30.472708+0000","last_active":"2024-08-28T19:19:11.652460+0000","last_peered":"2024-08-28T19:19:11.652460+0000","last_clean":"2024-08-28T19:19:11.652460+0000","last_became_active":"2024-08-28T19:18:30.472320+0000","last_became_peered":"2024-08-28T19:18:30.472320+0000","last_unstale":"2024-08-28T19:19:11.652460+0000","last_undegraded":"2024-08-28T19:19:11.652460+0000","last_fullsized":"2024-08-28T19:19:11.652460+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:57:12.055626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628468+0000","last_change":"2024-08-28T19:18:39.519398+0000","last_active":"2024-08-28T19:19:08.628468+0000","last_peered":"2024-08-28T19:19:08.628468+0000","last_clean":"2024-08-28T19:19:08.628468+0000","last_became_active":"2024-08-28T19:18:39.518997+0000","last_became_peered":"2024-08-28T19:18:39.518997+0000","last_unstale":"2024-08-28T19:19:08.628468+0000","last_undegraded":"2024-08-28T19:19:08.628468+0000","last_fullsized":"2024-08-28T19:19:08.628468+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:33.431608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619235+0000","last_change":"2024-08-28T19:18:32.489439+0000","last_active":"2024-08-28T19:19:08.619235+0000","last_peered":"2024-08-28T19:19:08.619235+0000","last_clean":"2024-08-28T19:19:08.619235+0000","last_became_active":"2024-08-28T19:18:32.489054+0000","last_became_peered":"2024-08-28T19:18:32.489054+0000","last_unstale":"2024-08-28T19:19:08.619235+0000","last_undegraded":"2024-08-28T19:19:08.619235+0000","last_fullsized":"2024-08-28T19:19:08.619235+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:31:32.647384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.651121+0000","last_change":"2024-08-28T19:18:30.464993+0000","last_active":"2024-08-28T19:19:11.651121+0000","last_peered":"2024-08-28T19:19:11.651121+0000","last_clean":"2024-08-28T19:19:11.651121+0000","last_became_active":"2024-08-28T19:18:30.463951+0000","last_became_peered":"2024-08-28T19:18:30.463951+0000","last_unstale":"2024-08-28T19:19:11.651121+0000","last_undegraded":"2024-08-28T19:19:11.651121+0000","last_fullsized":"2024-08-28T19:19:11.651121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:06.780058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614780+0000","last_change":"2024-08-28T19:18:39.533061+0000","last_active":"2024-08-28T19:19:08.614780+0000","last_peered":"2024-08-28T19:19:08.614780+0000","last_clean":"2024-08-28T19:19:08.614780+0000","last_became_active":"2024-08-28T19:18:39.532607+0000","last_became_peered":"2024-08-28T19:18:39.532607+0000","last_unstale":"2024-08-28T19:19:08.614780+0000","last_undegraded":"2024-08-28T19:19:08.614780+0000","last_fullsized":"2024-08-28T19:19:08.614780+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:00:57.997665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615094+0000","last_change":"2024-08-28T19:18:32.487937+0000","last_active":"2024-08-28T19:19:08.615094+0000","last_peered":"2024-08-28T19:19:08.615094+0000","last_clean":"2024-08-28T19:19:08.615094+0000","last_became_active":"2024-08-28T19:18:32.487417+0000","last_became_peered":"2024-08-28T19:18:32.487417+0000","last_unstale":"2024-08-28T19:19:08.615094+0000","last_undegraded":"2024-08-28T19:19:08.615094+0000","last_fullsized":"2024-08-28T19:19:08.615094+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:42.497364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.630439+0000","last_change":"2024-08-28T19:18:30.470742+0000","last_active":"2024-08-28T19:19:10.630439+0000","last_peered":"2024-08-28T19:19:10.630439+0000","last_clean":"2024-08-28T19:19:10.630439+0000","last_became_active":"2024-08-28T19:18:30.470135+0000","last_became_peered":"2024-08-28T19:18:30.470135+0000","last_unstale":"2024-08-28T19:19:10.630439+0000","last_undegraded":"2024-08-28T19:19:10.630439+0000","last_fullsized":"2024-08-28T19:19:10.630439+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:49:39.581119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617294+0000","last_change":"2024-08-28T19:18:39.524052+0000","last_active":"2024-08-28T19:19:08.617294+0000","last_peered":"2024-08-28T19:19:08.617294+0000","last_clean":"2024-08-28T19:19:08.617294+0000","last_became_active":"2024-08-28T19:18:39.523601+0000","last_became_peered":"2024-08-28T19:18:39.523601+0000","last_unstale":"2024-08-28T19:19:08.617294+0000","last_undegraded":"2024-08-28T19:19:08.617294+0000","last_fullsized":"2024-08-28T19:19:08.617294+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:54:53.302424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617504+0000","last_change":"2024-08-28T19:18:32.491151+0000","last_active":"2024-08-28T19:19:08.617504+0000","last_peered":"2024-08-28T19:19:08.617504+0000","last_clean":"2024-08-28T19:19:08.617504+0000","last_became_active":"2024-08-28T19:18:32.490798+0000","last_became_peered":"2024-08-28T19:18:32.490798+0000","last_unstale":"2024-08-28T19:19:08.617504+0000","last_undegraded":"2024-08-28T19:19:08.617504+0000","last_fullsized":"2024-08-28T19:19:08.617504+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:41:00.771019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652641+0000","last_change":"2024-08-28T19:18:31.084029+0000","last_active":"2024-08-28T19:19:11.652641+0000","last_peered":"2024-08-28T19:19:11.652641+0000","last_clean":"2024-08-28T19:19:11.652641+0000","last_became_active":"2024-08-28T19:18:31.083507+0000","last_became_peered":"2024-08-28T19:18:31.083507+0000","last_unstale":"2024-08-28T19:19:11.652641+0000","last_undegraded":"2024-08-28T19:19:11.652641+0000","last_fullsized":"2024-08-28T19:19:11.652641+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:15:36.451702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611795+0000","last_change":"2024-08-28T19:18:39.527249+0000","last_active":"2024-08-28T19:19:08.611795+0000","last_peered":"2024-08-28T19:19:08.611795+0000","last_clean":"2024-08-28T19:19:08.611795+0000","last_became_active":"2024-08-28T19:18:39.526834+0000","last_became_peered":"2024-08-28T19:18:39.526834+0000","last_unstale":"2024-08-28T19:19:08.611795+0000","last_undegraded":"2024-08-28T19:19:08.611795+0000","last_fullsized":"2024-08-28T19:19:08.611795+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:48:02.294738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616892+0000","last_change":"2024-08-28T19:18:32.497337+0000","last_active":"2024-08-28T19:19:08.616892+0000","last_peered":"2024-08-28T19:19:08.616892+0000","last_clean":"2024-08-28T19:19:08.616892+0000","last_became_active":"2024-08-28T19:18:32.496929+0000","last_became_peered":"2024-08-28T19:18:32.496929+0000","last_unstale":"2024-08-28T19:19:08.616892+0000","last_undegraded":"2024-08-28T19:19:08.616892+0000","last_fullsized":"2024-08-28T19:19:08.616892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:14.748765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":94,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:02.731817+0000","last_change":"2024-08-28T19:18:30.471353+0000","last_active":"2024-08-28T19:20:02.731817+0000","last_peered":"2024-08-28T19:20:02.731817+0000","last_clean":"2024-08-28T19:20:02.731817+0000","last_became_active":"2024-08-28T19:18:30.470283+0000","last_became_peered":"2024-08-28T19:18:30.470283+0000","last_unstale":"2024-08-28T19:20:02.731817+0000","last_undegraded":"2024-08-28T19:20:02.731817+0000","last_fullsized":"2024-08-28T19:20:02.731817+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:09:46.444864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614636+0000","last_change":"2024-08-28T19:18:39.519226+0000","last_active":"2024-08-28T19:19:08.614636+0000","last_peered":"2024-08-28T19:19:08.614636+0000","last_clean":"2024-08-28T19:19:08.614636+0000","last_became_active":"2024-08-28T19:18:39.518738+0000","last_became_peered":"2024-08-28T19:18:39.518738+0000","last_unstale":"2024-08-28T19:19:08.614636+0000","last_undegraded":"2024-08-28T19:19:08.614636+0000","last_fullsized":"2024-08-28T19:19:08.614636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:39:35.114702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617127+0000","last_change":"2024-08-28T19:18:32.493060+0000","last_active":"2024-08-28T19:19:08.617127+0000","last_peered":"2024-08-28T19:19:08.617127+0000","last_clean":"2024-08-28T19:19:08.617127+0000","last_became_active":"2024-08-28T19:18:32.492590+0000","last_became_peered":"2024-08-28T19:18:32.492590+0000","last_unstale":"2024-08-28T19:19:08.617127+0000","last_undegraded":"2024-08-28T19:19:08.617127+0000","last_fullsized":"2024-08-28T19:19:08.617127+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:02:02.165301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653316+0000","last_change":"2024-08-28T19:18:30.470674+0000","last_active":"2024-08-28T19:19:11.653316+0000","last_peered":"2024-08-28T19:19:11.653316+0000","last_clean":"2024-08-28T19:19:11.653316+0000","last_became_active":"2024-08-28T19:18:30.470275+0000","last_became_peered":"2024-08-28T19:18:30.470275+0000","last_unstale":"2024-08-28T19:19:11.653316+0000","last_undegraded":"2024-08-28T19:19:11.653316+0000","last_fullsized":"2024-08-28T19:19:11.653316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:01:48.547315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624414+0000","last_change":"2024-08-28T19:19:08.624194+0000","last_active":"2024-08-28T19:19:08.624414+0000","last_peered":"2024-08-28T19:19:08.624414+0000","last_clean":"2024-08-28T19:19:08.624414+0000","last_became_active":"2024-08-28T19:19:08.623681+0000","last_became_peered":"2024-08-28T19:19:08.623681+0000","last_unstale":"2024-08-28T19:19:08.624414+0000","last_undegraded":"2024-08-28T19:19:08.624414+0000","last_fullsized":"2024-08-28T19:19:08.624414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:21:11.099931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.611866+0000","last_change":"2024-08-28T19:18:32.488210+0000","last_active":"2024-08-28T19:19:08.611866+0000","last_peered":"2024-08-28T19:19:08.611866+0000","last_clean":"2024-08-28T19:19:08.611866+0000","last_became_active":"2024-08-28T19:18:32.487853+0000","last_became_peered":"2024-08-28T19:18:32.487853+0000","last_unstale":"2024-08-28T19:19:08.611866+0000","last_undegraded":"2024-08-28T19:19:08.611866+0000","last_fullsized":"2024-08-28T19:19:08.611866+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:20:51.808958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:12.653792+0000","last_change":"2024-08-28T19:18:30.465306+0000","last_active":"2024-08-28T19:19:12.653792+0000","last_peered":"2024-08-28T19:19:12.653792+0000","last_clean":"2024-08-28T19:19:12.653792+0000","last_became_active":"2024-08-28T19:18:30.464850+0000","last_became_peered":"2024-08-28T19:18:30.464850+0000","last_unstale":"2024-08-28T19:19:12.653792+0000","last_undegraded":"2024-08-28T19:19:12.653792+0000","last_fullsized":"2024-08-28T19:19:12.653792+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:24:26.628999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616620+0000","last_change":"2024-08-28T19:18:39.524005+0000","last_active":"2024-08-28T19:19:08.616620+0000","last_peered":"2024-08-28T19:19:08.616620+0000","last_clean":"2024-08-28T19:19:08.616620+0000","last_became_active":"2024-08-28T19:18:39.523594+0000","last_became_peered":"2024-08-28T19:18:39.523594+0000","last_unstale":"2024-08-28T19:19:08.616620+0000","last_undegraded":"2024-08-28T19:19:08.616620+0000","last_fullsized":"2024-08-28T19:19:08.616620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:35:32.484399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619776+0000","last_change":"2024-08-28T19:18:32.489088+0000","last_active":"2024-08-28T19:19:08.619776+0000","last_peered":"2024-08-28T19:19:08.619776+0000","last_clean":"2024-08-28T19:19:08.619776+0000","last_became_active":"2024-08-28T19:18:32.488717+0000","last_became_peered":"2024-08-28T19:18:32.488717+0000","last_unstale":"2024-08-28T19:19:08.619776+0000","last_undegraded":"2024-08-28T19:19:08.619776+0000","last_fullsized":"2024-08-28T19:19:08.619776+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:25.468919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"82'35","reported_seq":185,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:04.733977+0000","last_change":"2024-08-28T19:18:30.468674+0000","last_active":"2024-08-28T19:20:04.733977+0000","last_peered":"2024-08-28T19:20:04.733977+0000","last_clean":"2024-08-28T19:20:04.733977+0000","last_became_active":"2024-08-28T19:18:30.468300+0000","last_became_peered":"2024-08-28T19:18:30.468300+0000","last_unstale":"2024-08-28T19:20:04.733977+0000","last_undegraded":"2024-08-28T19:20:04.733977+0000","last_fullsized":"2024-08-28T19:20:04.733977+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:12:06.237498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63475,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":119,"num_write":37,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614424+0000","last_change":"2024-08-28T19:18:39.526131+0000","last_active":"2024-08-28T19:19:08.614424+0000","last_peered":"2024-08-28T19:19:08.614424+0000","last_clean":"2024-08-28T19:19:08.614424+0000","last_became_active":"2024-08-28T19:18:39.525708+0000","last_became_peered":"2024-08-28T19:18:39.525708+0000","last_unstale":"2024-08-28T19:19:08.614424+0000","last_undegraded":"2024-08-28T19:19:08.614424+0000","last_fullsized":"2024-08-28T19:19:08.614424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:21:52.156487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628145+0000","last_change":"2024-08-28T19:18:32.487634+0000","last_active":"2024-08-28T19:19:08.628145+0000","last_peered":"2024-08-28T19:19:08.628145+0000","last_clean":"2024-08-28T19:19:08.628145+0000","last_became_active":"2024-08-28T19:18:32.486702+0000","last_became_peered":"2024-08-28T19:18:32.486702+0000","last_unstale":"2024-08-28T19:19:08.628145+0000","last_undegraded":"2024-08-28T19:19:08.628145+0000","last_fullsized":"2024-08-28T19:19:08.628145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:08:12.606905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.643469+0000","last_change":"2024-08-28T19:18:31.083494+0000","last_active":"2024-08-28T19:19:10.643469+0000","last_peered":"2024-08-28T19:19:10.643469+0000","last_clean":"2024-08-28T19:19:10.643469+0000","last_became_active":"2024-08-28T19:18:31.083086+0000","last_became_peered":"2024-08-28T19:18:31.083086+0000","last_unstale":"2024-08-28T19:19:10.643469+0000","last_undegraded":"2024-08-28T19:19:10.643469+0000","last_fullsized":"2024-08-28T19:19:10.643469+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:14:54.388364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618783+0000","last_change":"2024-08-28T19:18:39.532155+0000","last_active":"2024-08-28T19:19:08.618783+0000","last_peered":"2024-08-28T19:19:08.618783+0000","last_clean":"2024-08-28T19:19:08.618783+0000","last_became_active":"2024-08-28T19:18:39.531565+0000","last_became_peered":"2024-08-28T19:18:39.531565+0000","last_unstale":"2024-08-28T19:19:08.618783+0000","last_undegraded":"2024-08-28T19:19:08.618783+0000","last_fullsized":"2024-08-28T19:19:08.618783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:29:48.824622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629297+0000","last_change":"2024-08-28T19:18:32.496296+0000","last_active":"2024-08-28T19:19:08.629297+0000","last_peered":"2024-08-28T19:19:08.629297+0000","last_clean":"2024-08-28T19:19:08.629297+0000","last_became_active":"2024-08-28T19:18:32.495680+0000","last_became_peered":"2024-08-28T19:18:32.495680+0000","last_unstale":"2024-08-28T19:19:08.629297+0000","last_undegraded":"2024-08-28T19:19:08.629297+0000","last_fullsized":"2024-08-28T19:19:08.629297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:27:34.544529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.652615+0000","last_change":"2024-08-28T19:19:08.631758+0000","last_active":"2024-08-28T19:19:11.652615+0000","last_peered":"2024-08-28T19:19:11.652615+0000","last_clean":"2024-08-28T19:19:11.652615+0000","last_became_active":"2024-08-28T19:19:08.631202+0000","last_became_peered":"2024-08-28T19:19:08.631202+0000","last_unstale":"2024-08-28T19:19:11.652615+0000","last_undegraded":"2024-08-28T19:19:11.652615+0000","last_fullsized":"2024-08-28T19:19:11.652615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:27:43.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614142+0000","last_change":"2024-08-28T19:18:39.531157+0000","last_active":"2024-08-28T19:19:08.614142+0000","last_peered":"2024-08-28T19:19:08.614142+0000","last_clean":"2024-08-28T19:19:08.614142+0000","last_became_active":"2024-08-28T19:18:39.530675+0000","last_became_peered":"2024-08-28T19:18:39.530675+0000","last_unstale":"2024-08-28T19:19:08.614142+0000","last_undegraded":"2024-08-28T19:19:08.614142+0000","last_fullsized":"2024-08-28T19:19:08.614142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:38:40.414854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629314+0000","last_change":"2024-08-28T19:18:32.482182+0000","last_active":"2024-08-28T19:19:08.629314+0000","last_peered":"2024-08-28T19:19:08.629314+0000","last_clean":"2024-08-28T19:19:08.629314+0000","last_became_active":"2024-08-28T19:18:32.481689+0000","last_became_peered":"2024-08-28T19:18:32.481689+0000","last_unstale":"2024-08-28T19:19:08.629314+0000","last_undegraded":"2024-08-28T19:19:08.629314+0000","last_fullsized":"2024-08-28T19:19:08.629314+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:13:29.278469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"82'4","reported_seq":145,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:04.734038+0000","last_change":"2024-08-28T19:18:30.475244+0000","last_active":"2024-08-28T19:20:04.734038+0000","last_peered":"2024-08-28T19:20:04.734038+0000","last_clean":"2024-08-28T19:20:04.734038+0000","last_became_active":"2024-08-28T19:18:30.474861+0000","last_became_peered":"2024-08-28T19:18:30.474861+0000","last_unstale":"2024-08-28T19:20:04.734038+0000","last_undegraded":"2024-08-28T19:20:04.734038+0000","last_fullsized":"2024-08-28T19:20:04.734038+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:27:09.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":106,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631762+0000","last_change":"2024-08-28T19:18:39.520022+0000","last_active":"2024-08-28T19:19:08.631762+0000","last_peered":"2024-08-28T19:19:08.631762+0000","last_clean":"2024-08-28T19:19:08.631762+0000","last_became_active":"2024-08-28T19:18:39.519584+0000","last_became_peered":"2024-08-28T19:18:39.519584+0000","last_unstale":"2024-08-28T19:19:08.631762+0000","last_undegraded":"2024-08-28T19:19:08.631762+0000","last_fullsized":"2024-08-28T19:19:08.631762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:27:15.851303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628767+0000","last_change":"2024-08-28T19:18:39.522607+0000","last_active":"2024-08-28T19:19:08.628767+0000","last_peered":"2024-08-28T19:19:08.628767+0000","last_clean":"2024-08-28T19:19:08.628767+0000","last_became_active":"2024-08-28T19:18:39.522204+0000","last_became_peered":"2024-08-28T19:18:39.522204+0000","last_unstale":"2024-08-28T19:19:08.628767+0000","last_undegraded":"2024-08-28T19:19:08.628767+0000","last_fullsized":"2024-08-28T19:19:08.628767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:02:08.865294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614380+0000","last_change":"2024-08-28T19:18:32.485577+0000","last_active":"2024-08-28T19:19:08.614380+0000","last_peered":"2024-08-28T19:19:08.614380+0000","last_clean":"2024-08-28T19:19:08.614380+0000","last_became_active":"2024-08-28T19:18:32.485243+0000","last_became_peered":"2024-08-28T19:18:32.485243+0000","last_unstale":"2024-08-28T19:19:08.614380+0000","last_undegraded":"2024-08-28T19:19:08.614380+0000","last_fullsized":"2024-08-28T19:19:08.614380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:05:53.019365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:11.653809+0000","last_change":"2024-08-28T19:18:30.470670+0000","last_active":"2024-08-28T19:19:11.653809+0000","last_peered":"2024-08-28T19:19:11.653809+0000","last_clean":"2024-08-28T19:19:11.653809+0000","last_became_active":"2024-08-28T19:18:30.470264+0000","last_became_peered":"2024-08-28T19:18:30.470264+0000","last_unstale":"2024-08-28T19:19:11.653809+0000","last_undegraded":"2024-08-28T19:19:11.653809+0000","last_fullsized":"2024-08-28T19:19:11.653809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:35.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615236+0000","last_change":"2024-08-28T19:18:39.530948+0000","last_active":"2024-08-28T19:19:08.615236+0000","last_peered":"2024-08-28T19:19:08.615236+0000","last_clean":"2024-08-28T19:19:08.615236+0000","last_became_active":"2024-08-28T19:18:39.530473+0000","last_became_peered":"2024-08-28T19:18:39.530473+0000","last_unstale":"2024-08-28T19:19:08.615236+0000","last_undegraded":"2024-08-28T19:19:08.615236+0000","last_fullsized":"2024-08-28T19:19:08.615236+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:21:57.504582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632033+0000","last_change":"2024-08-28T19:18:32.486027+0000","last_active":"2024-08-28T19:19:08.632033+0000","last_peered":"2024-08-28T19:19:08.632033+0000","last_clean":"2024-08-28T19:19:08.632033+0000","last_became_active":"2024-08-28T19:18:32.483273+0000","last_became_peered":"2024-08-28T19:18:32.483273+0000","last_unstale":"2024-08-28T19:19:08.632033+0000","last_undegraded":"2024-08-28T19:19:08.632033+0000","last_fullsized":"2024-08-28T19:19:08.632033+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:47:37.680695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:09.637193+0000","last_change":"2024-08-28T19:18:30.464811+0000","last_active":"2024-08-28T19:19:09.637193+0000","last_peered":"2024-08-28T19:19:09.637193+0000","last_clean":"2024-08-28T19:19:09.637193+0000","last_became_active":"2024-08-28T19:18:30.463986+0000","last_became_peered":"2024-08-28T19:18:30.463986+0000","last_unstale":"2024-08-28T19:19:09.637193+0000","last_undegraded":"2024-08-28T19:19:09.637193+0000","last_fullsized":"2024-08-28T19:19:09.637193+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:43:46.582993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:10.642972+0000","last_change":"2024-08-28T19:19:08.631382+0000","last_active":"2024-08-28T19:19:10.642972+0000","last_peered":"2024-08-28T19:19:10.642972+0000","last_clean":"2024-08-28T19:19:10.642972+0000","last_became_active":"2024-08-28T19:19:08.630869+0000","last_became_peered":"2024-08-28T19:19:08.630869+0000","last_unstale":"2024-08-28T19:19:10.642972+0000","last_undegraded":"2024-08-28T19:19:10.642972+0000","last_fullsized":"2024-08-28T19:19:10.642972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:32:54.942408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615952+0000","last_change":"2024-08-28T19:18:32.484120+0000","last_active":"2024-08-28T19:19:08.615952+0000","last_peered":"2024-08-28T19:19:08.615952+0000","last_clean":"2024-08-28T19:19:08.615952+0000","last_became_active":"2024-08-28T19:18:32.483013+0000","last_became_peered":"2024-08-28T19:18:32.483013+0000","last_unstale":"2024-08-28T19:19:08.615952+0000","last_undegraded":"2024-08-28T19:19:08.615952+0000","last_fullsized":"2024-08-28T19:19:08.615952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:59:41.195873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622328+0000","last_change":"2024-08-28T19:18:39.531046+0000","last_active":"2024-08-28T19:19:08.622328+0000","last_peered":"2024-08-28T19:19:08.622328+0000","last_clean":"2024-08-28T19:19:08.622328+0000","last_became_active":"2024-08-28T19:18:39.530407+0000","last_became_peered":"2024-08-28T19:18:39.530407+0000","last_unstale":"2024-08-28T19:19:08.622328+0000","last_undegraded":"2024-08-28T19:19:08.622328+0000","last_fullsized":"2024-08-28T19:19:08.622328+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:47:15.173021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'6","reported_seq":95,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:20:01.730337+0000","last_change":"2024-08-28T19:18:30.471268+0000","last_active":"2024-08-28T19:20:01.730337+0000","last_peered":"2024-08-28T19:20:01.730337+0000","last_clean":"2024-08-28T19:20:01.730337+0000","last_became_active":"2024-08-28T19:18:30.470845+0000","last_became_peered":"2024-08-28T19:18:30.470845+0000","last_unstale":"2024-08-28T19:20:01.730337+0000","last_undegraded":"2024-08-28T19:20:01.730337+0000","last_fullsized":"2024-08-28T19:20:01.730337+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:18:18.086205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619195+0000","last_change":"2024-08-28T19:18:32.479728+0000","last_active":"2024-08-28T19:19:08.619195+0000","last_peered":"2024-08-28T19:19:08.619195+0000","last_clean":"2024-08-28T19:19:08.619195+0000","last_became_active":"2024-08-28T19:18:32.478623+0000","last_became_peered":"2024-08-28T19:18:32.478623+0000","last_unstale":"2024-08-28T19:19:08.619195+0000","last_undegraded":"2024-08-28T19:19:08.619195+0000","last_fullsized":"2024-08-28T19:19:08.619195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:14:49.914808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.623192+0000","last_change":"2024-08-28T19:18:39.531308+0000","last_active":"2024-08-28T19:19:08.623192+0000","last_peered":"2024-08-28T19:19:08.623192+0000","last_clean":"2024-08-28T19:19:08.623192+0000","last_became_active":"2024-08-28T19:18:39.530403+0000","last_became_peered":"2024-08-28T19:18:39.530403+0000","last_unstale":"2024-08-28T19:19:08.623192+0000","last_undegraded":"2024-08-28T19:19:08.623192+0000","last_fullsized":"2024-08-28T19:19:08.623192+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:33.139668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614114+0000","last_change":"2024-08-28T19:18:32.494621+0000","last_active":"2024-08-28T19:19:08.614114+0000","last_peered":"2024-08-28T19:19:08.614114+0000","last_clean":"2024-08-28T19:19:08.614114+0000","last_became_active":"2024-08-28T19:18:32.494217+0000","last_became_peered":"2024-08-28T19:18:32.494217+0000","last_unstale":"2024-08-28T19:19:08.614114+0000","last_undegraded":"2024-08-28T19:19:08.614114+0000","last_fullsized":"2024-08-28T19:19:08.614114+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:54:15.419470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622753+0000","last_change":"2024-08-28T19:18:32.484716+0000","last_active":"2024-08-28T19:19:08.622753+0000","last_peered":"2024-08-28T19:19:08.622753+0000","last_clean":"2024-08-28T19:19:08.622753+0000","last_became_active":"2024-08-28T19:18:32.484294+0000","last_became_peered":"2024-08-28T19:18:32.484294+0000","last_unstale":"2024-08-28T19:19:08.622753+0000","last_undegraded":"2024-08-28T19:19:08.622753+0000","last_fullsized":"2024-08-28T19:19:08.622753+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:39:45.045607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.621987+0000","last_change":"2024-08-28T19:18:32.484286+0000","last_active":"2024-08-28T19:19:08.621987+0000","last_peered":"2024-08-28T19:19:08.621987+0000","last_clean":"2024-08-28T19:19:08.621987+0000","last_became_active":"2024-08-28T19:18:32.483354+0000","last_became_peered":"2024-08-28T19:18:32.483354+0000","last_unstale":"2024-08-28T19:19:08.621987+0000","last_undegraded":"2024-08-28T19:19:08.621987+0000","last_fullsized":"2024-08-28T19:19:08.621987+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:09.546749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628505+0000","last_change":"2024-08-28T19:18:32.496985+0000","last_active":"2024-08-28T19:19:08.628505+0000","last_peered":"2024-08-28T19:19:08.628505+0000","last_clean":"2024-08-28T19:19:08.628505+0000","last_became_active":"2024-08-28T19:18:32.495874+0000","last_became_peered":"2024-08-28T19:18:32.495874+0000","last_unstale":"2024-08-28T19:19:08.628505+0000","last_undegraded":"2024-08-28T19:19:08.628505+0000","last_fullsized":"2024-08-28T19:19:08.628505+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:42.329572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612181+0000","last_change":"2024-08-28T19:18:32.485446+0000","last_active":"2024-08-28T19:19:08.612181+0000","last_peered":"2024-08-28T19:19:08.612181+0000","last_clean":"2024-08-28T19:19:08.612181+0000","last_became_active":"2024-08-28T19:18:32.484197+0000","last_became_peered":"2024-08-28T19:18:32.484197+0000","last_unstale":"2024-08-28T19:19:08.612181+0000","last_undegraded":"2024-08-28T19:19:08.612181+0000","last_fullsized":"2024-08-28T19:19:08.612181+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:47:50.527915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616380+0000","last_change":"2024-08-28T19:18:32.489452+0000","last_active":"2024-08-28T19:19:08.616380+0000","last_peered":"2024-08-28T19:19:08.616380+0000","last_clean":"2024-08-28T19:19:08.616380+0000","last_became_active":"2024-08-28T19:18:32.487482+0000","last_became_peered":"2024-08-28T19:18:32.487482+0000","last_unstale":"2024-08-28T19:19:08.616380+0000","last_undegraded":"2024-08-28T19:19:08.616380+0000","last_fullsized":"2024-08-28T19:19:08.616380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:52:08.175080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.622242+0000","last_change":"2024-08-28T19:18:32.483850+0000","last_active":"2024-08-28T19:19:08.622242+0000","last_peered":"2024-08-28T19:19:08.622242+0000","last_clean":"2024-08-28T19:19:08.622242+0000","last_became_active":"2024-08-28T19:18:32.483476+0000","last_became_peered":"2024-08-28T19:18:32.483476+0000","last_unstale":"2024-08-28T19:19:08.622242+0000","last_undegraded":"2024-08-28T19:19:08.622242+0000","last_fullsized":"2024-08-28T19:19:08.622242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:32:37.465218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.632706+0000","last_change":"2024-08-28T19:18:32.492252+0000","last_active":"2024-08-28T19:19:08.632706+0000","last_peered":"2024-08-28T19:19:08.632706+0000","last_clean":"2024-08-28T19:19:08.632706+0000","last_became_active":"2024-08-28T19:18:32.491659+0000","last_became_peered":"2024-08-28T19:18:32.491659+0000","last_unstale":"2024-08-28T19:19:08.632706+0000","last_undegraded":"2024-08-28T19:19:08.632706+0000","last_fullsized":"2024-08-28T19:19:08.632706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:19:28.901559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615146+0000","last_change":"2024-08-28T19:18:32.491940+0000","last_active":"2024-08-28T19:19:08.615146+0000","last_peered":"2024-08-28T19:19:08.615146+0000","last_clean":"2024-08-28T19:19:08.615146+0000","last_became_active":"2024-08-28T19:18:32.487474+0000","last_became_peered":"2024-08-28T19:18:32.487474+0000","last_unstale":"2024-08-28T19:19:08.615146+0000","last_undegraded":"2024-08-28T19:19:08.615146+0000","last_fullsized":"2024-08-28T19:19:08.615146+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:04:32.508620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631980+0000","last_change":"2024-08-28T19:18:32.485200+0000","last_active":"2024-08-28T19:19:08.631980+0000","last_peered":"2024-08-28T19:19:08.631980+0000","last_clean":"2024-08-28T19:19:08.631980+0000","last_became_active":"2024-08-28T19:18:32.484858+0000","last_became_peered":"2024-08-28T19:18:32.484858+0000","last_unstale":"2024-08-28T19:19:08.631980+0000","last_undegraded":"2024-08-28T19:19:08.631980+0000","last_fullsized":"2024-08-28T19:19:08.631980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:10:57.925659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627792+0000","last_change":"2024-08-28T19:18:32.489903+0000","last_active":"2024-08-28T19:19:08.627792+0000","last_peered":"2024-08-28T19:19:08.627792+0000","last_clean":"2024-08-28T19:19:08.627792+0000","last_became_active":"2024-08-28T19:18:32.488950+0000","last_became_peered":"2024-08-28T19:18:32.488950+0000","last_unstale":"2024-08-28T19:19:08.627792+0000","last_undegraded":"2024-08-28T19:19:08.627792+0000","last_fullsized":"2024-08-28T19:19:08.627792+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:06:42.284177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612366+0000","last_change":"2024-08-28T19:18:32.488236+0000","last_active":"2024-08-28T19:19:08.612366+0000","last_peered":"2024-08-28T19:19:08.612366+0000","last_clean":"2024-08-28T19:19:08.612366+0000","last_became_active":"2024-08-28T19:18:32.487776+0000","last_became_peered":"2024-08-28T19:18:32.487776+0000","last_unstale":"2024-08-28T19:19:08.612366+0000","last_undegraded":"2024-08-28T19:19:08.612366+0000","last_fullsized":"2024-08-28T19:19:08.612366+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:25:58.754740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.616789+0000","last_change":"2024-08-28T19:18:32.485860+0000","last_active":"2024-08-28T19:19:08.616789+0000","last_peered":"2024-08-28T19:19:08.616789+0000","last_clean":"2024-08-28T19:19:08.616789+0000","last_became_active":"2024-08-28T19:18:32.485415+0000","last_became_peered":"2024-08-28T19:18:32.485415+0000","last_unstale":"2024-08-28T19:19:08.616789+0000","last_undegraded":"2024-08-28T19:19:08.616789+0000","last_fullsized":"2024-08-28T19:19:08.616789+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:45:32.868651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.623402+0000","last_change":"2024-08-28T19:18:32.485515+0000","last_active":"2024-08-28T19:19:08.623402+0000","last_peered":"2024-08-28T19:19:08.623402+0000","last_clean":"2024-08-28T19:19:08.623402+0000","last_became_active":"2024-08-28T19:18:32.485075+0000","last_became_peered":"2024-08-28T19:18:32.485075+0000","last_unstale":"2024-08-28T19:19:08.623402+0000","last_undegraded":"2024-08-28T19:19:08.623402+0000","last_fullsized":"2024-08-28T19:19:08.623402+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:01:28.035479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.614972+0000","last_change":"2024-08-28T19:18:32.484685+0000","last_active":"2024-08-28T19:19:08.614972+0000","last_peered":"2024-08-28T19:19:08.614972+0000","last_clean":"2024-08-28T19:19:08.614972+0000","last_became_active":"2024-08-28T19:18:32.484095+0000","last_became_peered":"2024-08-28T19:18:32.484095+0000","last_unstale":"2024-08-28T19:19:08.614972+0000","last_undegraded":"2024-08-28T19:19:08.614972+0000","last_fullsized":"2024-08-28T19:19:08.614972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:53:27.229186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615703+0000","last_change":"2024-08-28T19:18:32.488030+0000","last_active":"2024-08-28T19:19:08.615703+0000","last_peered":"2024-08-28T19:19:08.615703+0000","last_clean":"2024-08-28T19:19:08.615703+0000","last_became_active":"2024-08-28T19:18:32.487042+0000","last_became_peered":"2024-08-28T19:18:32.487042+0000","last_unstale":"2024-08-28T19:19:08.615703+0000","last_undegraded":"2024-08-28T19:19:08.615703+0000","last_fullsized":"2024-08-28T19:19:08.615703+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:56.966475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.627165+0000","last_change":"2024-08-28T19:18:32.489442+0000","last_active":"2024-08-28T19:19:08.627165+0000","last_peered":"2024-08-28T19:19:08.627165+0000","last_clean":"2024-08-28T19:19:08.627165+0000","last_became_active":"2024-08-28T19:18:32.488405+0000","last_became_peered":"2024-08-28T19:18:32.488405+0000","last_unstale":"2024-08-28T19:19:08.627165+0000","last_undegraded":"2024-08-28T19:19:08.627165+0000","last_fullsized":"2024-08-28T19:19:08.627165+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:21:03.026893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628286+0000","last_change":"2024-08-28T19:18:32.496044+0000","last_active":"2024-08-28T19:19:08.628286+0000","last_peered":"2024-08-28T19:19:08.628286+0000","last_clean":"2024-08-28T19:19:08.628286+0000","last_became_active":"2024-08-28T19:18:32.495549+0000","last_became_peered":"2024-08-28T19:18:32.495549+0000","last_unstale":"2024-08-28T19:19:08.628286+0000","last_undegraded":"2024-08-28T19:19:08.628286+0000","last_fullsized":"2024-08-28T19:19:08.628286+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:06:56.123264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628305+0000","last_change":"2024-08-28T19:18:32.492077+0000","last_active":"2024-08-28T19:19:08.628305+0000","last_peered":"2024-08-28T19:19:08.628305+0000","last_clean":"2024-08-28T19:19:08.628305+0000","last_became_active":"2024-08-28T19:18:32.491722+0000","last_became_peered":"2024-08-28T19:18:32.491722+0000","last_unstale":"2024-08-28T19:19:08.628305+0000","last_undegraded":"2024-08-28T19:19:08.628305+0000","last_fullsized":"2024-08-28T19:19:08.628305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:16:21.697045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612237+0000","last_change":"2024-08-28T19:18:32.479845+0000","last_active":"2024-08-28T19:19:08.612237+0000","last_peered":"2024-08-28T19:19:08.612237+0000","last_clean":"2024-08-28T19:19:08.612237+0000","last_became_active":"2024-08-28T19:18:32.477690+0000","last_became_peered":"2024-08-28T19:18:32.477690+0000","last_unstale":"2024-08-28T19:19:08.612237+0000","last_undegraded":"2024-08-28T19:19:08.612237+0000","last_fullsized":"2024-08-28T19:19:08.612237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:11.281415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.624748+0000","last_change":"2024-08-28T19:19:08.624185+0000","last_active":"2024-08-28T19:19:08.624748+0000","last_peered":"2024-08-28T19:19:08.624748+0000","last_clean":"2024-08-28T19:19:08.624748+0000","last_became_active":"2024-08-28T19:19:08.623745+0000","last_became_peered":"2024-08-28T19:19:08.623745+0000","last_unstale":"2024-08-28T19:19:08.624748+0000","last_undegraded":"2024-08-28T19:19:08.624748+0000","last_fullsized":"2024-08-28T19:19:08.624748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:56.657173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.619629+0000","last_change":"2024-08-28T19:18:32.499038+0000","last_active":"2024-08-28T19:19:08.619629+0000","last_peered":"2024-08-28T19:19:08.619629+0000","last_clean":"2024-08-28T19:19:08.619629+0000","last_became_active":"2024-08-28T19:18:32.498276+0000","last_became_peered":"2024-08-28T19:18:32.498276+0000","last_unstale":"2024-08-28T19:19:08.619629+0000","last_undegraded":"2024-08-28T19:19:08.619629+0000","last_fullsized":"2024-08-28T19:19:08.619629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:17:45.468448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612960+0000","last_change":"2024-08-28T19:18:32.492183+0000","last_active":"2024-08-28T19:19:08.612960+0000","last_peered":"2024-08-28T19:19:08.612960+0000","last_clean":"2024-08-28T19:19:08.612960+0000","last_became_active":"2024-08-28T19:18:32.491815+0000","last_became_peered":"2024-08-28T19:18:32.491815+0000","last_unstale":"2024-08-28T19:19:08.612960+0000","last_undegraded":"2024-08-28T19:19:08.612960+0000","last_fullsized":"2024-08-28T19:19:08.612960+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:13:11.183955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.629556+0000","last_change":"2024-08-28T19:18:32.497399+0000","last_active":"2024-08-28T19:19:08.629556+0000","last_peered":"2024-08-28T19:19:08.629556+0000","last_clean":"2024-08-28T19:19:08.629556+0000","last_became_active":"2024-08-28T19:18:32.496901+0000","last_became_peered":"2024-08-28T19:18:32.496901+0000","last_unstale":"2024-08-28T19:19:08.629556+0000","last_undegraded":"2024-08-28T19:19:08.629556+0000","last_fullsized":"2024-08-28T19:19:08.629556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:05:57.634590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.631412+0000","last_change":"2024-08-28T19:18:32.495871+0000","last_active":"2024-08-28T19:19:08.631412+0000","last_peered":"2024-08-28T19:19:08.631412+0000","last_clean":"2024-08-28T19:19:08.631412+0000","last_became_active":"2024-08-28T19:18:32.495515+0000","last_became_peered":"2024-08-28T19:18:32.495515+0000","last_unstale":"2024-08-28T19:19:08.631412+0000","last_undegraded":"2024-08-28T19:19:08.631412+0000","last_fullsized":"2024-08-28T19:19:08.631412+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:53:56.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617058+0000","last_change":"2024-08-28T19:18:32.495711+0000","last_active":"2024-08-28T19:19:08.617058+0000","last_peered":"2024-08-28T19:19:08.617058+0000","last_clean":"2024-08-28T19:19:08.617058+0000","last_became_active":"2024-08-28T19:18:32.495271+0000","last_became_peered":"2024-08-28T19:18:32.495271+0000","last_unstale":"2024-08-28T19:19:08.617058+0000","last_undegraded":"2024-08-28T19:19:08.617058+0000","last_fullsized":"2024-08-28T19:19:08.617058+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:05.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.612310+0000","last_change":"2024-08-28T19:18:32.489731+0000","last_active":"2024-08-28T19:19:08.612310+0000","last_peered":"2024-08-28T19:19:08.612310+0000","last_clean":"2024-08-28T19:19:08.612310+0000","last_became_active":"2024-08-28T19:18:32.489366+0000","last_became_peered":"2024-08-28T19:18:32.489366+0000","last_unstale":"2024-08-28T19:19:08.612310+0000","last_undegraded":"2024-08-28T19:19:08.612310+0000","last_fullsized":"2024-08-28T19:19:08.612310+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:52:24.180155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.617967+0000","last_change":"2024-08-28T19:18:32.490270+0000","last_active":"2024-08-28T19:19:08.617967+0000","last_peered":"2024-08-28T19:19:08.617967+0000","last_clean":"2024-08-28T19:19:08.617967+0000","last_became_active":"2024-08-28T19:18:32.489667+0000","last_became_peered":"2024-08-28T19:18:32.489667+0000","last_unstale":"2024-08-28T19:19:08.617967+0000","last_undegraded":"2024-08-28T19:19:08.617967+0000","last_fullsized":"2024-08-28T19:19:08.617967+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:18.571978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.618388+0000","last_change":"2024-08-28T19:18:32.482553+0000","last_active":"2024-08-28T19:19:08.618388+0000","last_peered":"2024-08-28T19:19:08.618388+0000","last_clean":"2024-08-28T19:19:08.618388+0000","last_became_active":"2024-08-28T19:18:32.481161+0000","last_became_peered":"2024-08-28T19:18:32.481161+0000","last_unstale":"2024-08-28T19:19:08.618388+0000","last_undegraded":"2024-08-28T19:19:08.618388+0000","last_fullsized":"2024-08-28T19:19:08.618388+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:13.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628811+0000","last_change":"2024-08-28T19:18:32.492473+0000","last_active":"2024-08-28T19:19:08.628811+0000","last_peered":"2024-08-28T19:19:08.628811+0000","last_clean":"2024-08-28T19:19:08.628811+0000","last_became_active":"2024-08-28T19:18:32.491897+0000","last_became_peered":"2024-08-28T19:18:32.491897+0000","last_unstale":"2024-08-28T19:19:08.628811+0000","last_undegraded":"2024-08-28T19:19:08.628811+0000","last_fullsized":"2024-08-28T19:19:08.628811+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:24.279391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.628899+0000","last_change":"2024-08-28T19:18:32.485621+0000","last_active":"2024-08-28T19:19:08.628899+0000","last_peered":"2024-08-28T19:19:08.628899+0000","last_clean":"2024-08-28T19:19:08.628899+0000","last_became_active":"2024-08-28T19:18:32.485267+0000","last_became_peered":"2024-08-28T19:18:32.485267+0000","last_unstale":"2024-08-28T19:19:08.628899+0000","last_undegraded":"2024-08-28T19:19:08.628899+0000","last_fullsized":"2024-08-28T19:19:08.628899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:30:08.932135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-28T19:19:08.615578+0000","last_change":"2024-08-28T19:18:32.488457+0000","last_active":"2024-08-28T19:19:08.615578+0000","last_peered":"2024-08-28T19:19:08.615578+0000","last_clean":"2024-08-28T19:19:08.615578+0000","last_became_active":"2024-08-28T19:18:32.487555+0000","last_became_peered":"2024-08-28T19:18:32.487555+0000","last_unstale":"2024-08-28T19:19:08.615578+0000","last_undegraded":"2024-08-28T19:19:08.615578+0000","last_fullsized":"2024-08-28T19:19:08.615578+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:03:26.390481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70679,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":570,"num_read_kb":364,"num_write":249,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":212037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":68,"seq":292057776161,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":481155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":62,"seq":266287972389,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1650688,"data_stored":1134870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":57,"seq":244813135915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":484509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":51,"seq":219043332143,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":970752,"data_stored":480791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":45,"seq":193273528372,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708284,"statfs":{"total":95995035648,"available":95957282816,"internally_reserved":0,"allocated":1560576,"data_stored":1077007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":40,"seq":171798691898,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":476308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888125,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1572864,"data_stored":1068774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051652,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":479991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215176,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40444,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704708,"statfs":{"total":95995035648,"available":95953620992,"internally_reserved":0,"allocated":962560,"data_stored":475625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":541033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":540813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705752,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":974848,"data_stored":477373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-28T19:20:09.224 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-28T19:20:09.224 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-28T19:20:09.224 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-08-28T19:20:09.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph health --format=json 2024-08-28T19:20:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:09 smithi028 ceph-mon[21769]: pgmap v310: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:20:09.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-08-28T19:20:09.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-08-28T19:20:09.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-08-28T19:20:09.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-08-28T19:20:09.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:09 smithi028 ceph-mon[21769]: osdmap e83: 12 total, 12 up, 12 in 2024-08-28T19:20:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:09 smithi115 ceph-mon[22184]: pgmap v310: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:20:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-08-28T19:20:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-08-28T19:20:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-08-28T19:20:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-08-28T19:20:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:09 smithi115 ceph-mon[22184]: osdmap e83: 12 total, 12 up, 12 in 2024-08-28T19:20:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.959+0000 7f87e76d4640 1 -- 172.21.15.16:0/3592802354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e010db90 msgr2=0x7f87e01100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.959+0000 7f87e76d4640 1 --2- 172.21.15.16:0/3592802354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e010db90 0x7f87e01100a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009ca0 tx=0x7f87d802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.959+0000 7f87e76d4640 1 -- 172.21.15.16:0/3592802354 shutdown_connections 2024-08-28T19:20:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.959+0000 7f87e76d4640 1 --2- 172.21.15.16:0/3592802354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e010db90 0x7f87e01100a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.959+0000 7f87e76d4640 1 --2- 172.21.15.16:0/3592802354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0105010 0x7f87e010d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.959+0000 7f87e76d4640 1 --2- 172.21.15.16:0/3592802354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e01045e0 0x7f87e01049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.959+0000 7f87e76d4640 1 -- 172.21.15.16:0/3592802354 >> 172.21.15.16:0/3592802354 conn(0x7f87e00794d0 msgr2=0x7f87e00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.959+0000 7f87e76d4640 1 -- 172.21.15.16:0/3592802354 shutdown_connections 2024-08-28T19:20:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.959+0000 7f87e76d4640 1 -- 172.21.15.16:0/3592802354 wait complete. 2024-08-28T19:20:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e76d4640 1 Processor -- start 2024-08-28T19:20:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e76d4640 1 -- start start 2024-08-28T19:20:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e76d4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e01045e0 0x7f87e019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e76d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0105010 0x7f87e019a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e76d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e010db90 0x7f87e019fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e76d4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01138b0 con 0x7f87e010db90 2024-08-28T19:20:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e76d4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87e0113730 con 0x7f87e0105010 2024-08-28T19:20:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e76d4640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f87e0113a30 con 0x7f87e01045e0 2024-08-28T19:20:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e5c71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e010db90 0x7f87e019fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e5c71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e010db90 0x7f87e019fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51834/0 (socket says 172.21.15.16:51834) 2024-08-28T19:20:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e5c71640 1 -- 172.21.15.16:0/450928543 learned_addr learned my addr 172.21.15.16:0/450928543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e5470640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e01045e0 0x7f87e019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.960+0000 7f87e4c6f640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0105010 0x7f87e019a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87e5c71640 1 -- 172.21.15.16:0/450928543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e01045e0 msgr2=0x7f87e019a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87e5c71640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e01045e0 0x7f87e019a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87e5c71640 1 -- 172.21.15.16:0/450928543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0105010 msgr2=0x7f87e019a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87e5c71640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0105010 0x7f87e019a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87e5c71640 1 -- 172.21.15.16:0/450928543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e01a0210 con 0x7f87e010db90 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87e4c6f640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0105010 0x7f87e019a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87e5c71640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e010db90 0x7f87e019fae0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f87d8005cc0 tx=0x7f87d802f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87d67fc640 1 -- 172.21.15.16:0/450928543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d8003cf0 con 0x7f87e010db90 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87d67fc640 1 -- 172.21.15.16:0/450928543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d80044f0 con 0x7f87e010db90 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87e76d4640 1 -- 172.21.15.16:0/450928543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e01a04a0 con 0x7f87e010db90 2024-08-28T19:20:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87d67fc640 1 -- 172.21.15.16:0/450928543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d802fc30 con 0x7f87e010db90 2024-08-28T19:20:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.961+0000 7f87e76d4640 1 -- 172.21.15.16:0/450928543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e01a6c60 con 0x7f87e010db90 2024-08-28T19:20:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.962+0000 7f87e76d4640 1 -- 172.21.15.16:0/450928543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87e01049f0 con 0x7f87e010db90 2024-08-28T19:20:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.964+0000 7f87d67fc640 1 -- 172.21.15.16:0/450928543 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f87d80046a0 con 0x7f87e010db90 2024-08-28T19:20:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.964+0000 7f87d67fc640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87cc0b4c10 0x7f87cc0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.965+0000 7f87d67fc640 1 -- 172.21.15.16:0/450928543 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f87d8106e70 con 0x7f87e010db90 2024-08-28T19:20:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.965+0000 7f87e5470640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87cc0b4c10 0x7f87cc0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.965+0000 7f87e5470640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87cc0b4c10 0x7f87cc0b7120 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f87e006b9a0 tx=0x7f87dc009110 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:09.968+0000 7f87d67fc640 1 -- 172.21.15.16:0/450928543 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f87d80cd9b0 con 0x7f87e010db90 2024-08-28T19:20:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.195+0000 7f87e76d4640 1 -- 172.21.15.16:0/450928543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f87e01a6e90 con 0x7f87e010db90 2024-08-28T19:20:10.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:10 smithi016 ceph-mon[18556]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:20:10.197 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:10 smithi016 ceph-mon[18556]: osdmap e84: 12 total, 12 up, 12 in 2024-08-28T19:20:10.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.195+0000 7f87d67fc640 1 -- 172.21.15.16:0/450928543 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f87d80cd2b0 con 0x7f87e010db90 2024-08-28T19:20:10.197 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:10.197 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-28T19:20:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.197+0000 7f87e76d4640 1 -- 172.21.15.16:0/450928543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87cc0b4c10 msgr2=0x7f87cc0b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.197+0000 7f87e76d4640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87cc0b4c10 0x7f87cc0b7120 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f87e006b9a0 tx=0x7f87dc009110 comp rx=0 tx=0).stop 2024-08-28T19:20:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.197+0000 7f87e76d4640 1 -- 172.21.15.16:0/450928543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e010db90 msgr2=0x7f87e019fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.197+0000 7f87e76d4640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e010db90 0x7f87e019fae0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f87d8005cc0 tx=0x7f87d802f470 comp rx=0 tx=0).stop 2024-08-28T19:20:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.197+0000 7f87e76d4640 1 -- 172.21.15.16:0/450928543 shutdown_connections 2024-08-28T19:20:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.197+0000 7f87e76d4640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e010db90 0x7f87e019fae0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.197+0000 7f87e76d4640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87cc0b4c10 0x7f87cc0b7120 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:10.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.197+0000 7f87e76d4640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0105010 0x7f87e019a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:10.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.197+0000 7f87e76d4640 1 --2- 172.21.15.16:0/450928543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e01045e0 0x7f87e019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:10.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.197+0000 7f87e76d4640 1 -- 172.21.15.16:0/450928543 >> 172.21.15.16:0/450928543 conn(0x7f87e00794d0 msgr2=0x7f87e01055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:10.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.197+0000 7f87e76d4640 1 -- 172.21.15.16:0/450928543 shutdown_connections 2024-08-28T19:20:10.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:10.198+0000 7f87e76d4640 1 -- 172.21.15.16:0/450928543 wait complete. 2024-08-28T19:20:10.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:10 smithi028 ceph-mon[21769]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:20:10.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:10 smithi028 ceph-mon[21769]: osdmap e84: 12 total, 12 up, 12 in 2024-08-28T19:20:10.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:10 smithi115 ceph-mon[22184]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:20:10.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:10 smithi115 ceph-mon[22184]: osdmap e84: 12 total, 12 up, 12 in 2024-08-28T19:20:11.094 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-08-28T19:20:11.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:20:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:11 smithi028 ceph-mon[21769]: pgmap v313: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-08-28T19:20:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:11 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/450928543' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-28T19:20:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:11 smithi115 ceph-mon[22184]: pgmap v313: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-08-28T19:20:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:11 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/450928543' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-28T19:20:11.475 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:11 smithi016 ceph-mon[18556]: pgmap v313: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-08-28T19:20:11.475 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:11 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/450928543' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-28T19:20:11.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.828+0000 7fcd2886e640 1 -- 172.21.15.16:0/1687006788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd24109250 msgr2=0x7fcd24109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:11.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.828+0000 7fcd2886e640 1 --2- 172.21.15.16:0/1687006788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd24109250 0x7fcd24109660 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009900 tx=0x7fcd1802a410 comp rx=0 tx=0).stop 2024-08-28T19:20:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.828+0000 7fcd2886e640 1 -- 172.21.15.16:0/1687006788 shutdown_connections 2024-08-28T19:20:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.828+0000 7fcd2886e640 1 --2- 172.21.15.16:0/1687006788 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd24109ba0 0x7fcd241103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.828+0000 7fcd2886e640 1 --2- 172.21.15.16:0/1687006788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd24109250 0x7fcd24109660 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.828+0000 7fcd2886e640 1 --2- 172.21.15.16:0/1687006788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24101d10 0x7fcd24102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.828+0000 7fcd2886e640 1 -- 172.21.15.16:0/1687006788 >> 172.21.15.16:0/1687006788 conn(0x7fcd24077a00 msgr2=0x7fcd24077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.828+0000 7fcd2886e640 1 -- 172.21.15.16:0/1687006788 shutdown_connections 2024-08-28T19:20:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.828+0000 7fcd2886e640 1 -- 172.21.15.16:0/1687006788 wait complete. 2024-08-28T19:20:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.829+0000 7fcd2886e640 1 Processor -- start 2024-08-28T19:20:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.829+0000 7fcd2886e640 1 -- start start 2024-08-28T19:20:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.829+0000 7fcd2886e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd24101d10 0x7fcd2419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd2886e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24109250 0x7fcd2419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd2886e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd24109ba0 0x7fcd2419fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd2886e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcd24113860 con 0x7fcd24109ba0 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd2886e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd241136e0 con 0x7fcd24109250 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd2886e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcd241139e0 con 0x7fcd24101d10 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd22d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd24109ba0 0x7fcd2419fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd22d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd24109ba0 0x7fcd2419fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51846/0 (socket says 172.21.15.16:51846) 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd22d9d640 1 -- 172.21.15.16:0/2272716703 learned_addr learned my addr 172.21.15.16:0/2272716703 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd21d9b640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24109250 0x7fcd2419a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd2259c640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd24101d10 0x7fcd2419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd22d9d640 1 -- 172.21.15.16:0/2272716703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd24101d10 msgr2=0x7fcd2419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd22d9d640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd24101d10 0x7fcd2419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd22d9d640 1 -- 172.21.15.16:0/2272716703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24109250 msgr2=0x7fcd2419a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd22d9d640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24109250 0x7fcd2419a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd22d9d640 1 -- 172.21.15.16:0/2272716703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd241a01a0 con 0x7fcd24109ba0 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd22d9d640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd24109ba0 0x7fcd2419fb30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00eaa0 tx=0x7fcd1c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd137fe640 1 -- 172.21.15.16:0/2272716703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c00cdb0 con 0x7fcd24109ba0 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd137fe640 1 -- 172.21.15.16:0/2272716703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd1c005020 con 0x7fcd24109ba0 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd137fe640 1 -- 172.21.15.16:0/2272716703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c0116d0 con 0x7fcd24109ba0 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.830+0000 7fcd2886e640 1 -- 172.21.15.16:0/2272716703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd241a0490 con 0x7fcd24109ba0 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.831+0000 7fcd2886e640 1 -- 172.21.15.16:0/2272716703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd241a0950 con 0x7fcd24109ba0 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.831+0000 7fcd2886e640 1 -- 172.21.15.16:0/2272716703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd24102120 con 0x7fcd24109ba0 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.834+0000 7fcd137fe640 1 -- 172.21.15.16:0/2272716703 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fcd1c0051d0 con 0x7fcd24109ba0 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.835+0000 7fcd137fe640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcd0c0b4c40 0x7fcd0c0b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.835+0000 7fcd2259c640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcd0c0b4c40 0x7fcd0c0b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.835+0000 7fcd137fe640 1 -- 172.21.15.16:0/2272716703 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fcd1c0e5cb0 con 0x7fcd24109ba0 2024-08-28T19:20:11.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.835+0000 7fcd2259c640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcd0c0b4c40 0x7fcd0c0b7150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcd14005e10 tx=0x7fcd14005da0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:11.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:11.838+0000 7fcd137fe640 1 -- 172.21.15.16:0/2272716703 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fcd1c016030 con 0x7fcd24109ba0 2024-08-28T19:20:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.060+0000 7fcd2886e640 1 -- 172.21.15.16:0/2272716703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd2419b420 con 0x7fcd24109ba0 2024-08-28T19:20:12.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.061+0000 7fcd137fe640 1 -- 172.21.15.16:0/2272716703 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11977 (secure 0 0 0) 0x7fcd2419b420 con 0x7fcd24109ba0 2024-08-28T19:20:12.062 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:12.062 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-08-28T19:19:12:646580+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:19:12.646544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:20:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-08-28T19:20:12.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 -- 172.21.15.16:0/2272716703 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcd0c0b4c40 msgr2=0x7fcd0c0b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:12.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcd0c0b4c40 0x7fcd0c0b7150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcd14005e10 tx=0x7fcd14005da0 comp rx=0 tx=0).stop 2024-08-28T19:20:12.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 -- 172.21.15.16:0/2272716703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd24109ba0 msgr2=0x7fcd2419fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:12.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd24109ba0 0x7fcd2419fb30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00eaa0 tx=0x7fcd1c00ef60 comp rx=0 tx=0).stop 2024-08-28T19:20:12.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 -- 172.21.15.16:0/2272716703 shutdown_connections 2024-08-28T19:20:12.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd24109ba0 0x7fcd2419fb30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:12.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcd0c0b4c40 0x7fcd0c0b7150 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:12.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24109250 0x7fcd2419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:12.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 --2- 172.21.15.16:0/2272716703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd24101d10 0x7fcd2419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:12.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 -- 172.21.15.16:0/2272716703 >> 172.21.15.16:0/2272716703 conn(0x7fcd24077a00 msgr2=0x7fcd24107360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:12.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 -- 172.21.15.16:0/2272716703 shutdown_connections 2024-08-28T19:20:12.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:12.063+0000 7fcd2886e640 1 -- 172.21.15.16:0/2272716703 wait complete. 2024-08-28T19:20:12.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:20:13.002 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:13 smithi028 ceph-mon[21769]: pgmap v314: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-08-28T19:20:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:13 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2272716703' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:13 smithi115 ceph-mon[22184]: pgmap v314: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-08-28T19:20:13.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:13 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2272716703' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:13.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:13 smithi016 ceph-mon[18556]: pgmap v314: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-08-28T19:20:13.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:13 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2272716703' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:13.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.480+0000 7f3490651640 1 -- 172.21.15.16:0/1566167710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3488102820 msgr2=0x7f3488102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:13.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.480+0000 7f3490651640 1 --2- 172.21.15.16:0/1566167710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3488102820 0x7f3488102c30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3484009900 tx=0x7f348402a410 comp rx=0 tx=0).stop 2024-08-28T19:20:13.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.480+0000 7f3490651640 1 -- 172.21.15.16:0/1566167710 shutdown_connections 2024-08-28T19:20:13.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.480+0000 7f3490651640 1 --2- 172.21.15.16:0/1566167710 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3488108ab0 0x7f348810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:13.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.480+0000 7f3490651640 1 --2- 172.21.15.16:0/1566167710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3488108160 0x7f3488108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.480+0000 7f3490651640 1 --2- 172.21.15.16:0/1566167710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3488102820 0x7f3488102c30 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.480+0000 7f3490651640 1 -- 172.21.15.16:0/1566167710 >> 172.21.15.16:0/1566167710 conn(0x7f34880794e0 msgr2=0x7f34880798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.480+0000 7f3490651640 1 -- 172.21.15.16:0/1566167710 shutdown_connections 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.480+0000 7f3490651640 1 -- 172.21.15.16:0/1566167710 wait complete. 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.480+0000 7f3490651640 1 Processor -- start 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.480+0000 7f3490651640 1 -- start start 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f3490651640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3488102820 0x7f348819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f3490651640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3488108160 0x7f348819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f3490651640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3488108ab0 0x7f348819fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f3490651640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3488113840 con 0x7f3488102820 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f3490651640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34881136c0 con 0x7f3488108ab0 2024-08-28T19:20:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f3490651640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f34881139c0 con 0x7f3488108160 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f348e3ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3488102820 0x7f348819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f348e3ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3488102820 0x7f348819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51872/0 (socket says 172.21.15.16:51872) 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f348e3ed640 1 -- 172.21.15.16:0/3992583136 learned_addr learned my addr 172.21.15.16:0/3992583136 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f348dbec640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3488108160 0x7f348819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f348ebee640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3488108ab0 0x7f348819fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f348e3ed640 1 -- 172.21.15.16:0/3992583136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3488108160 msgr2=0x7f348819a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f348e3ed640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3488108160 0x7f348819a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f348e3ed640 1 -- 172.21.15.16:0/3992583136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3488108ab0 msgr2=0x7f348819fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f348e3ed640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3488108ab0 0x7f348819fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f348e3ed640 1 -- 172.21.15.16:0/3992583136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34881a0200 con 0x7f3488102820 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.481+0000 7f348e3ed640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3488102820 0x7f348819a290 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3484005940 tx=0x7f34840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.482+0000 7f347f7fe640 1 -- 172.21.15.16:0/3992583136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348403f030 con 0x7f3488102820 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.482+0000 7f347f7fe640 1 -- 172.21.15.16:0/3992583136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3484003cb0 con 0x7f3488102820 2024-08-28T19:20:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.482+0000 7f3490651640 1 -- 172.21.15.16:0/3992583136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34881a0490 con 0x7f3488102820 2024-08-28T19:20:13.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.482+0000 7f347f7fe640 1 -- 172.21.15.16:0/3992583136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484035070 con 0x7f3488102820 2024-08-28T19:20:13.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.482+0000 7f3490651640 1 -- 172.21.15.16:0/3992583136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f348810fc50 con 0x7f3488102820 2024-08-28T19:20:13.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.483+0000 7f3490651640 1 -- 172.21.15.16:0/3992583136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3488103aa0 con 0x7f3488102820 2024-08-28T19:20:13.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.485+0000 7f347f7fe640 1 -- 172.21.15.16:0/3992583136 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f3484042030 con 0x7f3488102820 2024-08-28T19:20:13.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.486+0000 7f347f7fe640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34740b4c10 0x7f34740b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:13.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.486+0000 7f348dbec640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34740b4c10 0x7f34740b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:13.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.486+0000 7f347f7fe640 1 -- 172.21.15.16:0/3992583136 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3484106400 con 0x7f3488102820 2024-08-28T19:20:13.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.486+0000 7f348dbec640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34740b4c10 0x7f34740b7120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f34881009a0 tx=0x7f347800b590 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:13.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.489+0000 7f347f7fe640 1 -- 172.21.15.16:0/3992583136 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f34840ccf40 con 0x7f3488102820 2024-08-28T19:20:13.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.713+0000 7f3490651640 1 -- 172.21.15.16:0/3992583136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3488102c30 con 0x7f3488102820 2024-08-28T19:20:13.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.714+0000 7f347f7fe640 1 -- 172.21.15.16:0/3992583136 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11977 (secure 0 0 0) 0x7f3488102c30 con 0x7f3488102820 2024-08-28T19:20:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-08-28T19:19:12:646580+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:19:12.646544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:20:13.715 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-08-28T19:20:13.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 -- 172.21.15.16:0/3992583136 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34740b4c10 msgr2=0x7f34740b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:13.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34740b4c10 0x7f34740b7120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f34881009a0 tx=0x7f347800b590 comp rx=0 tx=0).stop 2024-08-28T19:20:13.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 -- 172.21.15.16:0/3992583136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3488102820 msgr2=0x7f348819a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:13.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3488102820 0x7f348819a290 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3484005940 tx=0x7f34840042f0 comp rx=0 tx=0).stop 2024-08-28T19:20:13.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 -- 172.21.15.16:0/3992583136 shutdown_connections 2024-08-28T19:20:13.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34740b4c10 0x7f34740b7120 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:13.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3488108ab0 0x7f348819fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:13.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3488108160 0x7f348819a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:13.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 --2- 172.21.15.16:0/3992583136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3488102820 0x7f348819a290 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:13.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 -- 172.21.15.16:0/3992583136 >> 172.21.15.16:0/3992583136 conn(0x7f34880794e0 msgr2=0x7f34881012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:13.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 -- 172.21.15.16:0/3992583136 shutdown_connections 2024-08-28T19:20:13.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:13.716+0000 7f3490651640 1 -- 172.21.15.16:0/3992583136 wait complete. 2024-08-28T19:20:14.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:14 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3992583136' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:14.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:14 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3992583136' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:14.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:14 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3992583136' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:14.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:20:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:15.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.140+0000 7ff29e04b640 1 -- 172.21.15.16:0/2985866839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff298108a90 msgr2=0x7ff29810f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:15.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.140+0000 7ff29e04b640 1 --2- 172.21.15.16:0/2985866839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff298108a90 0x7ff29810f2c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff290009ca0 tx=0x7ff29002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:15.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.140+0000 7ff29e04b640 1 -- 172.21.15.16:0/2985866839 shutdown_connections 2024-08-28T19:20:15.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.140+0000 7ff29e04b640 1 --2- 172.21.15.16:0/2985866839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff298108a90 0x7ff29810f2c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:15.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.140+0000 7ff29e04b640 1 --2- 172.21.15.16:0/2985866839 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff298108140 0x7ff298108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:15.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.140+0000 7ff29e04b640 1 --2- 172.21.15.16:0/2985866839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff298102800 0x7ff298102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:15.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.140+0000 7ff29e04b640 1 -- 172.21.15.16:0/2985866839 >> 172.21.15.16:0/2985866839 conn(0x7ff2980794d0 msgr2=0x7ff2980798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:15.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.140+0000 7ff29e04b640 1 -- 172.21.15.16:0/2985866839 shutdown_connections 2024-08-28T19:20:15.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.140+0000 7ff29e04b640 1 -- 172.21.15.16:0/2985866839 wait complete. 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.141+0000 7ff29e04b640 1 Processor -- start 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.141+0000 7ff29e04b640 1 -- start start 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.141+0000 7ff29e04b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff298102800 0x7ff29819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.141+0000 7ff29e04b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff298108140 0x7ff29819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.141+0000 7ff29e04b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff298108a90 0x7ff29819fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.141+0000 7ff29e04b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff2981126f0 con 0x7ff298108a90 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.141+0000 7ff29e04b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff298112570 con 0x7ff298108140 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.141+0000 7ff29e04b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff298112870 con 0x7ff298102800 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.141+0000 7ff297fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff298108a90 0x7ff29819fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.141+0000 7ff297fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff298108a90 0x7ff29819fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40948/0 (socket says 172.21.15.16:40948) 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.141+0000 7ff297fff640 1 -- 172.21.15.16:0/3917816016 learned_addr learned my addr 172.21.15.16:0/3917816016 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:15.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff28effd640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff298108140 0x7ff29819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff2977fe640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff298102800 0x7ff29819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff297fff640 1 -- 172.21.15.16:0/3917816016 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff298102800 msgr2=0x7ff29819a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff297fff640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff298102800 0x7ff29819a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff297fff640 1 -- 172.21.15.16:0/3917816016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff298108140 msgr2=0x7ff29819a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff297fff640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff298108140 0x7ff29819a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff297fff640 1 -- 172.21.15.16:0/3917816016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2981a01a0 con 0x7ff298108a90 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff297fff640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff298108a90 0x7ff29819fa70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff290005cc0 tx=0x7ff29002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff2957fa640 1 -- 172.21.15.16:0/3917816016 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff290003bf0 con 0x7ff298108a90 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff2957fa640 1 -- 172.21.15.16:0/3917816016 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff290003d90 con 0x7ff298108a90 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff29e04b640 1 -- 172.21.15.16:0/3917816016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2981a0430 con 0x7ff298108a90 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff2957fa640 1 -- 172.21.15.16:0/3917816016 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29002fbc0 con 0x7ff298108a90 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.142+0000 7ff29e04b640 1 -- 172.21.15.16:0/3917816016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2981a6bf0 con 0x7ff298108a90 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.143+0000 7ff29e04b640 1 -- 172.21.15.16:0/3917816016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff298102c10 con 0x7ff298108a90 2024-08-28T19:20:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.145+0000 7ff2957fa640 1 -- 172.21.15.16:0/3917816016 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7ff29002fd60 con 0x7ff298108a90 2024-08-28T19:20:15.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.146+0000 7ff2957fa640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2840b4c10 0x7ff2840b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:15.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.146+0000 7ff2977fe640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2840b4c10 0x7ff2840b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:15.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.146+0000 7ff2957fa640 1 -- 172.21.15.16:0/3917816016 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff290106f10 con 0x7ff298108a90 2024-08-28T19:20:15.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.147+0000 7ff2977fe640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2840b4c10 0x7ff2840b7120 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff2880058c0 tx=0x7ff28800a4c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:15.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.150+0000 7ff2957fa640 1 -- 172.21.15.16:0/3917816016 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ff2900098b0 con 0x7ff298108a90 2024-08-28T19:20:15.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:15 smithi028 ceph-mon[21769]: pgmap v315: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-28T19:20:15.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.335+0000 7ff29e04b640 1 -- 172.21.15.16:0/3917816016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff29819b360 con 0x7ff298108a90 2024-08-28T19:20:15.337 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:15 smithi016 ceph-mon[18556]: pgmap v315: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-28T19:20:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.337+0000 7ff2957fa640 1 -- 172.21.15.16:0/3917816016 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+25180 (secure 0 0 0) 0x7ff29819b360 con 0x7ff298108a90 2024-08-28T19:20:15.338 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:15.338 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:20:09.003260+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:20:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 -- 172.21.15.16:0/3917816016 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2840b4c10 msgr2=0x7ff2840b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2840b4c10 0x7ff2840b7120 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff2880058c0 tx=0x7ff28800a4c0 comp rx=0 tx=0).stop 2024-08-28T19:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 -- 172.21.15.16:0/3917816016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff298108a90 msgr2=0x7ff29819fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff298108a90 0x7ff29819fa70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff290005cc0 tx=0x7ff29002f380 comp rx=0 tx=0).stop 2024-08-28T19:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 -- 172.21.15.16:0/3917816016 shutdown_connections 2024-08-28T19:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff298108a90 0x7ff29819fa70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2840b4c10 0x7ff2840b7120 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff298108140 0x7ff29819a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 --2- 172.21.15.16:0/3917816016 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff298102800 0x7ff29819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 -- 172.21.15.16:0/3917816016 >> 172.21.15.16:0/3917816016 conn(0x7ff2980794d0 msgr2=0x7ff29806af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 -- 172.21.15.16:0/3917816016 shutdown_connections 2024-08-28T19:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:15.339+0000 7ff29e04b640 1 -- 172.21.15.16:0/3917816016 wait complete. 2024-08-28T19:20:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:15 smithi115 ceph-mon[22184]: pgmap v315: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-28T19:20:16.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:16 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3917816016' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:20:16.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:20:16.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:16 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3917816016' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:20:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:16 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3917816016' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:20:16.393 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:16.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.856+0000 7f70a8ad4640 1 -- 172.21.15.16:0/487722430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a4102800 msgr2=0x7f70a4102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:16.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.856+0000 7f70a8ad4640 1 --2- 172.21.15.16:0/487722430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a4102800 0x7f70a4102c10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7094009900 tx=0x7f709402a410 comp rx=0 tx=0).stop 2024-08-28T19:20:16.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.856+0000 7f70a8ad4640 1 -- 172.21.15.16:0/487722430 shutdown_connections 2024-08-28T19:20:16.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.856+0000 7f70a8ad4640 1 --2- 172.21.15.16:0/487722430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f70a4108a90 0x7f70a410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:16.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.856+0000 7f70a8ad4640 1 --2- 172.21.15.16:0/487722430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a4108140 0x7f70a4108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:16.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.856+0000 7f70a8ad4640 1 --2- 172.21.15.16:0/487722430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a4102800 0x7f70a4102c10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.856+0000 7f70a8ad4640 1 -- 172.21.15.16:0/487722430 >> 172.21.15.16:0/487722430 conn(0x7f70a40794d0 msgr2=0x7f70a40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.856+0000 7f70a8ad4640 1 -- 172.21.15.16:0/487722430 shutdown_connections 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.856+0000 7f70a8ad4640 1 -- 172.21.15.16:0/487722430 wait complete. 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.857+0000 7f70a8ad4640 1 Processor -- start 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.857+0000 7f70a8ad4640 1 -- start start 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a8ad4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a4102800 0x7f70a419b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a8ad4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f70a4108140 0x7f70a419b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a8ad4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a4108a90 0x7f70a419ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a8ad4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4112710 con 0x7f70a4108a90 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a8ad4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4112590 con 0x7f70a4102800 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a8ad4640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4112890 con 0x7f70a4108140 2024-08-28T19:20:16.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a4108a90 0x7f70a419ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a4108a90 0x7f70a419ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40964/0 (socket says 172.21.15.16:40964) 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a2d9d640 1 -- 172.21.15.16:0/608639890 learned_addr learned my addr 172.21.15.16:0/608639890 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a259c640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a4102800 0x7f70a419b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a1d9b640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f70a4108140 0x7f70a419b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a2d9d640 1 -- 172.21.15.16:0/608639890 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f70a4108140 msgr2=0x7f70a419b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a2d9d640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f70a4108140 0x7f70a419b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a2d9d640 1 -- 172.21.15.16:0/608639890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a4102800 msgr2=0x7f70a419b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a2d9d640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a4102800 0x7f70a419b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a2d9d640 1 -- 172.21.15.16:0/608639890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a41a6290 con 0x7f70a4108a90 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a1d9b640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f70a4108140 0x7f70a419b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70a2d9d640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a4108a90 0x7f70a419ba80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f709c00eaa0 tx=0x7f709c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.858+0000 7f70937fe640 1 -- 172.21.15.16:0/608639890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c00cdb0 con 0x7f70a4108a90 2024-08-28T19:20:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.859+0000 7f70937fe640 1 -- 172.21.15.16:0/608639890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f709c005020 con 0x7f70a4108a90 2024-08-28T19:20:16.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.859+0000 7f70937fe640 1 -- 172.21.15.16:0/608639890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c0116d0 con 0x7f70a4108a90 2024-08-28T19:20:16.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.859+0000 7f70a8ad4640 1 -- 172.21.15.16:0/608639890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a41a6580 con 0x7f70a4108a90 2024-08-28T19:20:16.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.859+0000 7f70a8ad4640 1 -- 172.21.15.16:0/608639890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a41a6a70 con 0x7f70a4108a90 2024-08-28T19:20:16.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.860+0000 7f70a8ad4640 1 -- 172.21.15.16:0/608639890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a4102c10 con 0x7f70a4108a90 2024-08-28T19:20:16.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.861+0000 7f70937fe640 1 -- 172.21.15.16:0/608639890 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f709c0051d0 con 0x7f70a4108a90 2024-08-28T19:20:16.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.862+0000 7f70937fe640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f708c0b4c40 0x7f708c0b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:16.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.862+0000 7f70a259c640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f708c0b4c40 0x7f708c0b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:16.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.862+0000 7f70937fe640 1 -- 172.21.15.16:0/608639890 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f709c0e5db0 con 0x7f70a4108a90 2024-08-28T19:20:16.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.862+0000 7f70a259c640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f708c0b4c40 0x7f708c0b7150 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7094000c00 tx=0x7f7094031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:16.865+0000 7f70937fe640 1 -- 172.21.15.16:0/608639890 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f709c016030 con 0x7f70a4108a90 2024-08-28T19:20:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.089+0000 7f70a8ad4640 1 -- 172.21.15.16:0/608639890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70a406b010 con 0x7f70a4108a90 2024-08-28T19:20:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.089+0000 7f70937fe640 1 -- 172.21.15.16:0/608639890 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11977 (secure 0 0 0) 0x7f70a406b010 con 0x7f70a4108a90 2024-08-28T19:20:17.091 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:17.091 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-08-28T19:19:12:646580+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:19:12.646544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:20:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-08-28T19:20:17.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 -- 172.21.15.16:0/608639890 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f708c0b4c40 msgr2=0x7f708c0b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:17.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f708c0b4c40 0x7f708c0b7150 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7094000c00 tx=0x7f7094031400 comp rx=0 tx=0).stop 2024-08-28T19:20:17.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 -- 172.21.15.16:0/608639890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a4108a90 msgr2=0x7f70a419ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:17.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a4108a90 0x7f70a419ba80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f709c00eaa0 tx=0x7f709c00ef60 comp rx=0 tx=0).stop 2024-08-28T19:20:17.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 -- 172.21.15.16:0/608639890 shutdown_connections 2024-08-28T19:20:17.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a4108a90 0x7f70a419ba80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:17.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f70a4108140 0x7f70a419b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:17.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f708c0b4c40 0x7f708c0b7150 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:17.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 --2- 172.21.15.16:0/608639890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a4102800 0x7f70a419b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:17.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 -- 172.21.15.16:0/608639890 >> 172.21.15.16:0/608639890 conn(0x7f70a40794d0 msgr2=0x7f70a41011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:17.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 -- 172.21.15.16:0/608639890 shutdown_connections 2024-08-28T19:20:17.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:17.092+0000 7f70a8ad4640 1 -- 172.21.15.16:0/608639890 wait complete. 2024-08-28T19:20:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:17 smithi028 ceph-mon[21769]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-08-28T19:20:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:17 smithi115 ceph-mon[22184]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-08-28T19:20:17.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:17 smithi016 ceph-mon[18556]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-08-28T19:20:17.901 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-08-28T19:18:33.178088+0000', 'modified': '2024-08-28T19:19:12.646544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24665, 'mds_1': 24649, 'mds_2': 24635, 'mds_3': 24625, 'mds_4': 24619}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14817': {'gid': 14817, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6839/3069459721', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 3069459721}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 3069459721}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14823': {'gid': 14823, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6841/2698492790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6840', 'nonce': 2698492790}, {'type': 'v1', 'addr': '172.21.15.16:6841', 'nonce': 2698492790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24583': {'gid': 24583, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.115:6833/2581346583', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6832', 'nonce': 2581346583}, {'type': 'v1', 'addr': '172.21.15.115:6833', 'nonce': 2581346583}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24590': {'gid': 24590, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6833/798033587', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 798033587}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 798033587}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24605': {'gid': 24605, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6835/2983279759', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2983279759}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2983279759}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24619': {'gid': 24619, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.115:6835/1297102593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6834', 'nonce': 1297102593}, {'type': 'v1', 'addr': '172.21.15.115:6835', 'nonce': 1297102593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24625': {'gid': 24625, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.115:6837/4186796507', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6836', 'nonce': 4186796507}, {'type': 'v1', 'addr': '172.21.15.115:6837', 'nonce': 4186796507}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24635': {'gid': 24635, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.28:6837/2888152030', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6836', 'nonce': 2888152030}, {'type': 'v1', 'addr': '172.21.15.28:6837', 'nonce': 2888152030}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24649': {'gid': 24649, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.115:6839/3027291414', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6838', 'nonce': 3027291414}, {'type': 'v1', 'addr': '172.21.15.115:6839', 'nonce': 3027291414}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24665': {'gid': 24665, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6839/2518320595', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 2518320595}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 2518320595}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24665, 'qdb_cluster': [24665, 24649, 24635, 24625, 24619]} 2024-08-28T19:20:17.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-08-28T19:20:17.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.f status 2024-08-28T19:20:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:18.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:18 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/608639890' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:18.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:18 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/608639890' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:18.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:18 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/608639890' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.641+0000 7f7f494ba640 1 -- 172.21.15.16:0/2178302922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f44109b90 msgr2=0x7f7f441103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.641+0000 7f7f494ba640 1 --2- 172.21.15.16:0/2178302922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f44109b90 0x7f7f441103c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009900 tx=0x7f7f3c02a410 comp rx=0 tx=0).stop 2024-08-28T19:20:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.641+0000 7f7f494ba640 1 -- 172.21.15.16:0/2178302922 shutdown_connections 2024-08-28T19:20:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.641+0000 7f7f494ba640 1 --2- 172.21.15.16:0/2178302922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f44109b90 0x7f7f441103c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.641+0000 7f7f494ba640 1 --2- 172.21.15.16:0/2178302922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f44109240 0x7f7f44109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.641+0000 7f7f494ba640 1 --2- 172.21.15.16:0/2178302922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f44103900 0x7f7f44103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.641+0000 7f7f494ba640 1 -- 172.21.15.16:0/2178302922 >> 172.21.15.16:0/2178302922 conn(0x7f7f4407b5f0 msgr2=0x7f7f4407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.642+0000 7f7f494ba640 1 -- 172.21.15.16:0/2178302922 shutdown_connections 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.642+0000 7f7f494ba640 1 -- 172.21.15.16:0/2178302922 wait complete. 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.642+0000 7f7f494ba640 1 Processor -- start 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.642+0000 7f7f494ba640 1 -- start start 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f494ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f44103900 0x7f7f440754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f494ba640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f44109240 0x7f7f440759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f494ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f44109b90 0x7f7f44075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f494ba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441138b0 con 0x7f7f44103900 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f494ba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f44113730 con 0x7f7f44109b90 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f494ba640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7f44113a30 con 0x7f7f44109240 2024-08-28T19:20:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f42ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f44103900 0x7f7f440754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f42ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f44103900 0x7f7f440754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40986/0 (socket says 172.21.15.16:40986) 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f42ffd640 1 -- 172.21.15.16:0/4089955178 learned_addr learned my addr 172.21.15.16:0/4089955178 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f427fc640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f44109240 0x7f7f440759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f437fe640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f44109b90 0x7f7f44075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f42ffd640 1 -- 172.21.15.16:0/4089955178 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f44109240 msgr2=0x7f7f440759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f44109240 0x7f7f440759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f42ffd640 1 -- 172.21.15.16:0/4089955178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f44109b90 msgr2=0x7f7f44075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f44109b90 0x7f7f44075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f42ffd640 1 -- 172.21.15.16:0/4089955178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f441a84e0 con 0x7f7f44103900 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f427fc640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f44109240 0x7f7f440759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f44103900 0x7f7f440754b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400ba50 tx=0x7f7f3400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.643+0000 7f7f3bfff640 1 -- 172.21.15.16:0/4089955178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f34019070 con 0x7f7f44103900 2024-08-28T19:20:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.644+0000 7f7f3bfff640 1 -- 172.21.15.16:0/4089955178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f3400cbe0 con 0x7f7f44103900 2024-08-28T19:20:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.644+0000 7f7f494ba640 1 -- 172.21.15.16:0/4089955178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f441a87d0 con 0x7f7f44103900 2024-08-28T19:20:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.644+0000 7f7f3bfff640 1 -- 172.21.15.16:0/4089955178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f340075d0 con 0x7f7f44103900 2024-08-28T19:20:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.644+0000 7f7f494ba640 1 -- 172.21.15.16:0/4089955178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f441a8d90 con 0x7f7f44103900 2024-08-28T19:20:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.646+0000 7f7f4fa2f740 1 Processor -- start 2024-08-28T19:20:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.646+0000 7f7f4fa2f740 1 -- start start 2024-08-28T19:20:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.646+0000 7f7f3bfff640 1 -- 172.21.15.16:0/4089955178 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f7f3401c070 con 0x7f7f44103900 2024-08-28T19:20:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d024f60 0x56057d3ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057cffc0a0 0x56057d3cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d004280 0x56057d3cb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56057d3413f0 con 0x56057d004280 2024-08-28T19:20:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56057d330980 con 0x56057d024f60 2024-08-28T19:20:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56057d334930 con 0x56057cffc0a0 2024-08-28T19:20:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f427fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d004280 0x56057d3cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f427fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d004280 0x56057d3cb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41002/0 (socket says 172.21.15.16:41002) 2024-08-28T19:20:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f427fc640 1 -- 172.21.15.16:0/29671146 learned_addr learned my addr 172.21.15.16:0/29671146 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/29671146 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057cffc0a0 0x56057d3cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f437fe640 1 --2- 172.21.15.16:0/29671146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d024f60 0x56057d3ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f427fc640 1 -- 172.21.15.16:0/29671146 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057cffc0a0 msgr2=0x56057d3cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f427fc640 1 --2- 172.21.15.16:0/29671146 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057cffc0a0 0x56057d3cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f427fc640 1 -- 172.21.15.16:0/29671146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d024f60 msgr2=0x56057d3ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f427fc640 1 --2- 172.21.15.16:0/29671146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d024f60 0x56057d3ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f427fc640 1 -- 172.21.15.16:0/29671146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56057d090540 con 0x56057d004280 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.647+0000 7f7f427fc640 1 --2- 172.21.15.16:0/29671146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d004280 0x56057d3cb230 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00bae0 tx=0x7f7f2c00d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd70bf0a146885f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.648+0000 7f7f397fa640 1 -- 172.21.15.16:0/29671146 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c017420 con 0x56057d004280 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.648+0000 7f7f3bfff640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f300b4c10 0x7f7f300b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.648+0000 7f7f397fa640 1 -- 172.21.15.16:0/29671146 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f2c017a70 con 0x56057d004280 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.648+0000 7f7f397fa640 1 -- 172.21.15.16:0/29671146 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c00fa80 con 0x56057d004280 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.648+0000 7f7f437fe640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f300b4c10 0x7f7f300b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.648+0000 7f7f3bfff640 1 -- 172.21.15.16:0/4089955178 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7f340e4d50 con 0x7f7f44103900 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/29671146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d004280 msgr2=0x56057d3cb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/29671146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d004280 0x56057d3cb230 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00bae0 tx=0x7f7f2c00d340 comp rx=0 tx=0).stop 2024-08-28T19:20:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f437fe640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f300b4c10 0x7f7f300b7120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c005ab0 tx=0x7f7f3c009640 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/29671146 shutdown_connections 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/29671146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d004280 0x56057d3cb230 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/29671146 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057cffc0a0 0x56057d3cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/29671146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d024f60 0x56057d3ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/29671146 >> 172.21.15.16:0/29671146 conn(0x56057d019350 msgr2=0x56057d3c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/29671146 shutdown_connections 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/29671146 wait complete. 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f4fa2f740 1 -- start start 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.649+0000 7f7f4fa2f740 1 Processor -- start 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057cffc0a0 0x56057d3d5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d004280 0x56057d3d62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d024f60 0x56057d3d6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56057d1935f0 con 0x56057d024f60 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56057d2b7600 con 0x56057d004280 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56057d3d8f90 con 0x56057cffc0a0 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f4fa2f740 10 client.0 _ll_register_callbacks cb 0x56057d2ddaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d024f60 0x56057d3d6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d024f60 0x56057d3d6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41016/0 (socket says 172.21.15.16:41016) 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f437fe640 1 -- 172.21.15.16:0/1201085562 learned_addr learned my addr 172.21.15.16:0/1201085562 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057cffc0a0 0x56057d3d5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f427fc640 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d004280 0x56057d3d62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f437fe640 1 -- 172.21.15.16:0/1201085562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057cffc0a0 msgr2=0x56057d3d5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f437fe640 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057cffc0a0 0x56057d3d5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f437fe640 1 -- 172.21.15.16:0/1201085562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d004280 msgr2=0x56057d3d62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f437fe640 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d004280 0x56057d3d62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f437fe640 1 -- 172.21.15.16:0/1201085562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56057d2f07b0 con 0x56057d024f60 2024-08-28T19:20:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f437fe640 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d024f60 0x56057d3d6830 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c033000 tx=0x7f7f3c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc692579649563c9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.650+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/1201085562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c04c070 con 0x56057d024f60 2024-08-28T19:20:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/1201085562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7f3c004460 con 0x56057d024f60 2024-08-28T19:20:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/1201085562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f2c017a70 con 0x56057d024f60 2024-08-28T19:20:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/1201085562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7f3c058050 con 0x56057d024f60 2024-08-28T19:20:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f4fa2f740 10 client.15285 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/1201085562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56057d3e4c10 con 0x56057d024f60 2024-08-28T19:20:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/1201085562 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f2c017a70 con 0x56057d024f60 2024-08-28T19:20:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/1201085562 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f7f3c003b80 con 0x56057d024f60 2024-08-28T19:20:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f4fa2f740 10 client.15285 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f4fa2f740 10 client.15285 resolve_mds: resolved name 'f' to daemon mds.f 2024-08-28T19:20:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f4fa2f740 20 client.15285 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56057cfe38e0 0x56057d3e86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f4fa2f740 4 client.15285 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:20:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.651+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/1201085562 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56057d343340 con 0x56057cfe38e0 2024-08-28T19:20:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.652+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56057cfe38e0 0x56057d3e86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.652+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56057cfe38e0 0x56057d3e86e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=26df545604b70e08 server_cookie=3dd657070e7ec166 in_seq=0 out_seq=0 2024-08-28T19:20:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.652+0000 7f7f2a7fc640 10 client.15285 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/1201085562 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56057d343340 con 0x56057cfe38e0 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f2a7fc640 10 client.15285 handle_command_reply: tid=0 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 1 client.15285 shutdown 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/1201085562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d024f60 msgr2=0x56057d3d6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d024f60 0x56057d3d6830 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c033000 tx=0x7f7f3c009260 comp rx=0 tx=0).stop 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/1201085562 shutdown_connections 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56057cfe38e0 0x56057d3e86e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057d024f60 0x56057d3d6830 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d004280 0x56057d3d62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/1201085562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057cffc0a0 0x56057d3d5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/1201085562 >> 172.21.15.16:0/1201085562 conn(0x56057d019350 msgr2=0x56057d3cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f2a7fc640 0 client.15285 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:20:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/1201085562 shutdown_connections 2024-08-28T19:20:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/1201085562 wait complete. 2024-08-28T19:20:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.653+0000 7f7f4fa2f740 20 client.15285 trim_cache size 0 max 16384 2024-08-28T19:20:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f4fa2f740 1 Processor -- start 2024-08-28T19:20:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f4fa2f740 1 -- start start 2024-08-28T19:20:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d004280 0x56057d3d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057d024f60 0x56057d3d14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56057d3e6a20 con 0x56057cffc0a0 2024-08-28T19:20:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56057d3e70f0 con 0x56057d004280 2024-08-28T19:20:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56057d3e77c0 con 0x56057d024f60 2024-08-28T19:20:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f42ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f42ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41022/0 (socket says 172.21.15.16:41022) 2024-08-28T19:20:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f42ffd640 1 -- 172.21.15.16:0/651132580 learned_addr learned my addr 172.21.15.16:0/651132580 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f437fe640 1 --2- 172.21.15.16:0/651132580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057d024f60 0x56057d3d14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f427fc640 1 --2- 172.21.15.16:0/651132580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d004280 0x56057d3d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.672+0000 7f7f42ffd640 1 -- 172.21.15.16:0/651132580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057d024f60 msgr2=0x56057d3d14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.673+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/651132580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057d024f60 0x56057d3d14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.673+0000 7f7f42ffd640 1 -- 172.21.15.16:0/651132580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d004280 msgr2=0x56057d3d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.673+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/651132580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d004280 0x56057d3d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.673+0000 7f7f42ffd640 1 -- 172.21.15.16:0/651132580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56057d3d6e50 con 0x56057cffc0a0 2024-08-28T19:20:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.673+0000 7f7f437fe640 1 --2- 172.21.15.16:0/651132580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057d024f60 0x56057d3d14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.673+0000 7f7f427fc640 1 --2- 172.21.15.16:0/651132580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d004280 0x56057d3d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.673+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/651132580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e6080 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400ba20 tx=0x7f7f340e72b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c3588cd8ca834a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.673+0000 7f7f38ff9640 1 -- 172.21.15.16:0/651132580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f340aaca0 con 0x56057cffc0a0 2024-08-28T19:20:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.673+0000 7f7f38ff9640 1 -- 172.21.15.16:0/651132580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f340ab890 con 0x56057cffc0a0 2024-08-28T19:20:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.673+0000 7f7f38ff9640 1 -- 172.21.15.16:0/651132580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f340a7420 con 0x56057cffc0a0 2024-08-28T19:20:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.674+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/651132580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 msgr2=0x56057d3e6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.674+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/651132580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e6080 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400ba20 tx=0x7f7f340e72b0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.674+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/651132580 shutdown_connections 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.674+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/651132580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057d024f60 0x56057d3d14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.674+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/651132580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d004280 0x56057d3d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.674+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/651132580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e6080 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.674+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/651132580 >> 172.21.15.16:0/651132580 conn(0x56057d019350 msgr2=0x56057d3c3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.674+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/651132580 shutdown_connections 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.674+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/651132580 wait complete. 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.674+0000 7f7f4fa2f740 1 -- start start 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f4fa2f740 1 Processor -- start 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057d004280 0x56057d3ea4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f4fa2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d024f60 0x56057d3ef480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56057d3e8560 con 0x56057cffc0a0 2024-08-28T19:20:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56057d3e7e90 con 0x56057d024f60 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f4fa2f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56057d3e8c30 con 0x56057d004280 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f4fa2f740 10 client.0 _ll_register_callbacks cb 0x7f7f3c031840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f42ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f42ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41024/0 (socket says 172.21.15.16:41024) 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f42ffd640 1 -- 172.21.15.16:0/2463116355 learned_addr learned my addr 172.21.15.16:0/2463116355 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f427fc640 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057d004280 0x56057d3ea4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f42ffd640 1 -- 172.21.15.16:0/2463116355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057d004280 msgr2=0x56057d3ea4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057d004280 0x56057d3ea4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f42ffd640 1 -- 172.21.15.16:0/2463116355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d024f60 msgr2=0x56057d3ef480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f437fe640 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d024f60 0x56057d3ef480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d024f60 0x56057d3ef480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f42ffd640 1 -- 172.21.15.16:0/2463116355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56057d3efac0 con 0x56057cffc0a0 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.675+0000 7f7f437fe640 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d024f60 0x56057d3ef480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f42ffd640 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e9f10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7f340a8400 tx=0x7f7f340e26f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26514dbc2e4856f2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/2463116355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f340ab190 con 0x56057cffc0a0 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/2463116355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7f340b4020 con 0x56057cffc0a0 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/2463116355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f340ab890 con 0x56057cffc0a0 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/2463116355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7f340afa00 con 0x56057cffc0a0 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f4fa2f740 10 client.15291 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/2463116355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56057d3f0d80 con 0x56057cffc0a0 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/2463116355 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f340ab890 con 0x56057cffc0a0 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/2463116355 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f7f340a9e60 con 0x56057cffc0a0 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f4fa2f740 10 client.15291 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f4fa2f740 10 client.15291 resolve_mds: resolved name 'f' to daemon mds.f 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f4fa2f740 20 client.15291 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56057cfb8c90 0x56057d456360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f4fa2f740 4 client.15291 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.676+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/2463116355 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "status"}) -- 0x56057d33ede0 con 0x56057cfb8c90 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.677+0000 7f7f427fc640 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56057cfb8c90 0x56057d456360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.677+0000 7f7f2a7fc640 10 client.15291 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.677+0000 7f7f427fc640 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56057cfb8c90 0x56057d456360 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=752bb5bb3ca33b22 server_cookie=42f67969fb61a1ed in_seq=0 out_seq=0 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.677+0000 7f7f2a7fc640 1 -- 172.21.15.16:0/2463116355 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56057d33ede0 con 0x56057cfb8c90 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.677+0000 7f7f2a7fc640 10 client.15291 handle_command_reply: tid=0 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.677+0000 7f7f4fa2f740 1 client.15291 shutdown 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/2463116355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 msgr2=0x56057d3e9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e9f10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7f340a8400 tx=0x7f7f340e26f0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/2463116355 shutdown_connections 2024-08-28T19:20:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56057cfb8c90 0x56057d456360 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56057d024f60 0x56057d3ef480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56057d004280 0x56057d3ea4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f4fa2f740 1 --2- 172.21.15.16:0/2463116355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56057cffc0a0 0x56057d3e9f10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f2a7fc640 0 client.15291 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:20:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/2463116355 >> 172.21.15.16:0/2463116355 conn(0x56057d019350 msgr2=0x56057d3c3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/2463116355 shutdown_connections 2024-08-28T19:20:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f4fa2f740 1 -- 172.21.15.16:0/2463116355 wait complete. 2024-08-28T19:20:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.678+0000 7f7f4fa2f740 20 client.15291 trim_cache size 0 max 16384 2024-08-28T19:20:18.692 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24619, 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 67.036846573000005, 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 82, 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 132.10474206800001 2024-08-28T19:20:18.693 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:20:18.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 -- 172.21.15.16:0/4089955178 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f300b4c10 msgr2=0x7f7f300b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:18.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f300b4c10 0x7f7f300b7120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c005ab0 tx=0x7f7f3c009640 comp rx=0 tx=0).stop 2024-08-28T19:20:18.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 -- 172.21.15.16:0/4089955178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f44103900 msgr2=0x7f7f440754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:18.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f44103900 0x7f7f440754b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400ba50 tx=0x7f7f3400bf10 comp rx=0 tx=0).stop 2024-08-28T19:20:18.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 -- 172.21.15.16:0/4089955178 shutdown_connections 2024-08-28T19:20:18.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f300b4c10 0x7f7f300b7120 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f44109b90 0x7f7f44075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f44109240 0x7f7f440759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 --2- 172.21.15.16:0/4089955178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f44103900 0x7f7f440754b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:18.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 -- 172.21.15.16:0/4089955178 >> 172.21.15.16:0/4089955178 conn(0x7f7f4407b5f0 msgr2=0x7f7f44107290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:18.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 -- 172.21.15.16:0/4089955178 shutdown_connections 2024-08-28T19:20:18.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:18.679+0000 7f7f494ba640 1 -- 172.21.15.16:0/4089955178 wait complete. 2024-08-28T19:20:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:19 smithi028 ceph-mon[21769]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-08-28T19:20:19.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:19 smithi016 ceph-mon[18556]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-08-28T19:20:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:19 smithi115 ceph-mon[22184]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-08-28T19:20:19.580 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.c status 2024-08-28T19:20:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:20.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.320+0000 7fa2d71dc640 1 -- 172.21.15.16:0/2504393578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d0108a90 msgr2=0x7fa2d010f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.320+0000 7fa2d71dc640 1 --2- 172.21.15.16:0/2504393578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d0108a90 0x7fa2d010f2c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009ca0 tx=0x7fa2c802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.320+0000 7fa2d71dc640 1 -- 172.21.15.16:0/2504393578 shutdown_connections 2024-08-28T19:20:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.320+0000 7fa2d71dc640 1 --2- 172.21.15.16:0/2504393578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d0108a90 0x7fa2d010f2c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.320+0000 7fa2d71dc640 1 --2- 172.21.15.16:0/2504393578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d0108140 0x7fa2d0108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.320+0000 7fa2d71dc640 1 --2- 172.21.15.16:0/2504393578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d0102800 0x7fa2d0102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.320+0000 7fa2d71dc640 1 -- 172.21.15.16:0/2504393578 >> 172.21.15.16:0/2504393578 conn(0x7fa2d00794d0 msgr2=0x7fa2d00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.320+0000 7fa2d71dc640 1 -- 172.21.15.16:0/2504393578 shutdown_connections 2024-08-28T19:20:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.320+0000 7fa2d71dc640 1 -- 172.21.15.16:0/2504393578 wait complete. 2024-08-28T19:20:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d71dc640 1 Processor -- start 2024-08-28T19:20:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d71dc640 1 -- start start 2024-08-28T19:20:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d71dc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d0102800 0x7fa2d010fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d71dc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d0108140 0x7fa2d0110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d71dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d0108a90 0x7fa2d01107d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d71dc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d0113820 con 0x7fa2d0102800 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d71dc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d01136a0 con 0x7fa2d0108a90 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d71dc640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d01139a0 con 0x7fa2d0108140 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d4f78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d0102800 0x7fa2d010fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d4f78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d0102800 0x7fa2d010fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41050/0 (socket says 172.21.15.16:41050) 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d4f78640 1 -- 172.21.15.16:0/3928579899 learned_addr learned my addr 172.21.15.16:0/3928579899 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.321+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d0108a90 0x7fa2d01107d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d0108140 0x7fa2d0110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2d4f78640 1 -- 172.21.15.16:0/3928579899 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d0108140 msgr2=0x7fa2d0110290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2d4f78640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d0108140 0x7fa2d0110290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2d4f78640 1 -- 172.21.15.16:0/3928579899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d0108a90 msgr2=0x7fa2d01107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2d4f78640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d0108a90 0x7fa2d01107d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2d4f78640 1 -- 172.21.15.16:0/3928579899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d01a8490 con 0x7fa2d0102800 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d0108140 0x7fa2d0110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2d4f78640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d0102800 0x7fa2d010fd50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa2c400c970 tx=0x7fa2c400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2cdffb640 1 -- 172.21.15.16:0/3928579899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4007bb0 con 0x7fa2d0102800 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2cdffb640 1 -- 172.21.15.16:0/3928579899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c4007d50 con 0x7fa2d0102800 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2d71dc640 1 -- 172.21.15.16:0/3928579899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d01a8780 con 0x7fa2d0102800 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2cdffb640 1 -- 172.21.15.16:0/3928579899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c400f450 con 0x7fa2d0102800 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.322+0000 7fa2d71dc640 1 -- 172.21.15.16:0/3928579899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d01a8c70 con 0x7fa2d0102800 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.325+0000 7fa2dd751740 1 Processor -- start 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2dd751740 1 -- start start 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2cdffb640 1 -- 172.21.15.16:0/3928579899 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fa2c4004f20 con 0x7fa2d0102800 2024-08-28T19:20:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x5602803087b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff3a0a0 0x560280308cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 0x560280309230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2dd751740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56028027f3f0 con 0x56027ff62f60 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2dd751740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56028026e980 con 0x56027ff3a0a0 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2dd751740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560280272930 con 0x56027ff42280 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2d5779640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x5602803087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2d5779640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x5602803087b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41052/0 (socket says 172.21.15.16:41052) 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2d5779640 1 -- 172.21.15.16:0/3388180859 learned_addr learned my addr 172.21.15.16:0/3388180859 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2d4f78640 1 --2- 172.21.15.16:0/3388180859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff3a0a0 0x560280308cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2d5779640 1 -- 172.21.15.16:0/3388180859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 msgr2=0x560280309230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3388180859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 0x560280309230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3388180859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 0x560280309230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2d5779640 1 -- 172.21.15.16:0/3388180859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff3a0a0 msgr2=0x560280308cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3388180859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff3a0a0 0x560280308cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2d5779640 1 -- 172.21.15.16:0/3388180859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56027ffce540 con 0x56027ff62f60 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3388180859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 0x560280309230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.326+0000 7fa2d4f78640 1 --2- 172.21.15.16:0/3388180859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff3a0a0 0x560280308cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.327+0000 7fa2cdffb640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c00b4c10 0x7fa2c00b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.327+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3388180859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x5602803087b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80098b0 tx=0x7fa2c80098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338b5574b3024afe server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.327+0000 7fa2d4f78640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c00b4c10 0x7fa2c00b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.327+0000 7fa2baffd640 1 -- 172.21.15.16:0/3388180859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8031e30 con 0x56027ff62f60 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.327+0000 7fa2cdffb640 1 -- 172.21.15.16:0/3928579899 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa2c40e5450 con 0x7fa2d0102800 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.327+0000 7fa2baffd640 1 -- 172.21.15.16:0/3388180859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c8036030 con 0x56027ff62f60 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.327+0000 7fa2baffd640 1 -- 172.21.15.16:0/3388180859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8004260 con 0x56027ff62f60 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2dd751740 1 -- 172.21.15.16:0/3388180859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 msgr2=0x5602803087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3388180859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x5602803087b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80098b0 tx=0x7fa2c80098e0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2d4f78640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c00b4c10 0x7fa2c00b7120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x560280307430 tx=0x7fa2c4004980 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2dd751740 1 -- 172.21.15.16:0/3388180859 shutdown_connections 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3388180859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 0x560280309230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3388180859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff3a0a0 0x560280308cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3388180859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x5602803087b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2dd751740 1 -- 172.21.15.16:0/3388180859 >> 172.21.15.16:0/3388180859 conn(0x56027ff57350 msgr2=0x5602803066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2dd751740 1 -- 172.21.15.16:0/3388180859 shutdown_connections 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2dd751740 1 -- 172.21.15.16:0/3388180859 wait complete. 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2dd751740 1 -- start start 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.328+0000 7fa2dd751740 1 Processor -- start 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff3a0a0 0x560280313e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 0x5602803143b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff62f60 0x5602803200a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2dd751740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5602800d15f0 con 0x56027ff3a0a0 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2dd751740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602801f5600 con 0x56027ff62f60 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2dd751740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560280316db0 con 0x56027ff42280 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2dd751740 10 client.0 _ll_register_callbacks cb 0x56028021baf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2d5779640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff3a0a0 0x560280313e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2d5779640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff3a0a0 0x560280313e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41064/0 (socket says 172.21.15.16:41064) 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2d5779640 1 -- 172.21.15.16:0/2905381049 learned_addr learned my addr 172.21.15.16:0/2905381049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2cffff640 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 0x5602803143b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2d5779640 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff62f60 0x5602803200a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2d5779640 1 -- 172.21.15.16:0/2905381049 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 msgr2=0x5602803143b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2d5779640 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 0x5602803143b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2d5779640 1 -- 172.21.15.16:0/2905381049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff62f60 msgr2=0x5602803200a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2d5779640 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff62f60 0x5602803200a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2d5779640 1 -- 172.21.15.16:0/2905381049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56028022e7b0 con 0x56027ff3a0a0 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2cffff640 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 0x5602803143b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2d5779640 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff3a0a0 0x560280313e70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8005c90 tx=0x7fa2c8003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fae7d80f2aab308 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.329+0000 7fa2affff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2affff640 1 -- 172.21.15.16:0/2905381049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8058060 con 0x56027ff3a0a0 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2affff640 1 -- 172.21.15.16:0/2905381049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa2c802f860 con 0x56027ff3a0a0 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2dd751740 1 -- 172.21.15.16:0/2905381049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2c8036030 con 0x56027ff3a0a0 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2affff640 1 -- 172.21.15.16:0/2905381049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa2c805f050 con 0x56027ff3a0a0 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2dd751740 10 client.15303 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2dd751740 1 -- 172.21.15.16:0/2905381049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5602803229d0 con 0x56027ff3a0a0 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2affff640 1 -- 172.21.15.16:0/2905381049 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c8036030 con 0x56027ff3a0a0 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2affff640 1 -- 172.21.15.16:0/2905381049 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa2c8043070 con 0x56027ff3a0a0 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2dd751740 10 client.15303 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2dd751740 10 client.15303 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2dd751740 20 client.15303 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2dd751740 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56027ff218e0 0x560280326100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2dd751740 4 client.15303 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.330+0000 7fa2dd751740 1 -- 172.21.15.16:0/2905381049 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560280281340 con 0x56027ff218e0 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.331+0000 7fa2cffff640 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56027ff218e0 0x560280326100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.331+0000 7fa2cffff640 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56027ff218e0 0x560280326100 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=721700e5c01db1c2 server_cookie=164e1ba5eaba4884 in_seq=0 out_seq=0 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.331+0000 7fa2affff640 10 client.15303 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2affff640 1 -- 172.21.15.16:0/2905381049 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560280281340 con 0x56027ff218e0 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2affff640 10 client.15303 handle_command_reply: tid=0 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2dd751740 1 client.15303 shutdown 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2dd751740 1 -- 172.21.15.16:0/2905381049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff3a0a0 msgr2=0x560280313e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2dd751740 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff3a0a0 0x560280313e70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8005c90 tx=0x7fa2c8003f00 comp rx=0 tx=0).stop 2024-08-28T19:20:20.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2dd751740 1 -- 172.21.15.16:0/2905381049 shutdown_connections 2024-08-28T19:20:20.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2dd751740 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56027ff218e0 0x560280326100 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2dd751740 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff62f60 0x5602803200a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2dd751740 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff42280 0x5602803143b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2dd751740 1 --2- 172.21.15.16:0/2905381049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff3a0a0 0x560280313e70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2dd751740 1 -- 172.21.15.16:0/2905381049 >> 172.21.15.16:0/2905381049 conn(0x56027ff57350 msgr2=0x56028030b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:20.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2affff640 0 client.15303 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:20:20.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2dd751740 1 -- 172.21.15.16:0/2905381049 shutdown_connections 2024-08-28T19:20:20.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.332+0000 7fa2dd751740 1 -- 172.21.15.16:0/2905381049 wait complete. 2024-08-28T19:20:20.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.333+0000 7fa2dd751740 20 client.15303 trim_cache size 0 max 16384 2024-08-28T19:20:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2dd751740 1 Processor -- start 2024-08-28T19:20:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2dd751740 1 -- start start 2024-08-28T19:20:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff3a0a0 0x56028030f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff42280 0x560280315aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x560280320010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2dd751740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560280324440 con 0x56027ff62f60 2024-08-28T19:20:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2dd751740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560280324b10 con 0x56027ff42280 2024-08-28T19:20:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2dd751740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5602803251e0 con 0x56027ff3a0a0 2024-08-28T19:20:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2d5779640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x560280320010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2d5779640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x560280320010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41068/0 (socket says 172.21.15.16:41068) 2024-08-28T19:20:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2d5779640 1 -- 172.21.15.16:0/3236469491 learned_addr learned my addr 172.21.15.16:0/3236469491 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3236469491 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff3a0a0 0x56028030f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3236469491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff42280 0x560280315aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2d5779640 1 -- 172.21.15.16:0/3236469491 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff3a0a0 msgr2=0x56028030f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3236469491 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff3a0a0 0x56028030f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2d5779640 1 -- 172.21.15.16:0/3236469491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff42280 msgr2=0x560280315aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3236469491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff42280 0x560280315aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.351+0000 7fa2d5779640 1 -- 172.21.15.16:0/3236469491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56028030ad70 con 0x56027ff62f60 2024-08-28T19:20:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.352+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3236469491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x560280320010 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa2c802ade0 tx=0x7fa2c8005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7138e781d8ec2df6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.352+0000 7fa2ba7fc640 1 -- 172.21.15.16:0/3236469491 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8031910 con 0x56027ff62f60 2024-08-28T19:20:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.352+0000 7fa2ba7fc640 1 -- 172.21.15.16:0/3236469491 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c8036070 con 0x56027ff62f60 2024-08-28T19:20:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.352+0000 7fa2ba7fc640 1 -- 172.21.15.16:0/3236469491 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c802f660 con 0x56027ff62f60 2024-08-28T19:20:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.352+0000 7fa2dd751740 1 -- 172.21.15.16:0/3236469491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 msgr2=0x560280320010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.352+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3236469491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x560280320010 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa2c802ade0 tx=0x7fa2c8005d80 comp rx=0 tx=0).stop 2024-08-28T19:20:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.353+0000 7fa2dd751740 1 -- 172.21.15.16:0/3236469491 shutdown_connections 2024-08-28T19:20:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.353+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3236469491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff62f60 0x560280320010 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.353+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3236469491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff42280 0x560280315aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.353+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3236469491 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff3a0a0 0x56028030f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.353+0000 7fa2dd751740 1 -- 172.21.15.16:0/3236469491 >> 172.21.15.16:0/3236469491 conn(0x56027ff57350 msgr2=0x560280315390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.353+0000 7fa2dd751740 1 -- 172.21.15.16:0/3236469491 shutdown_connections 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.353+0000 7fa2dd751740 1 -- 172.21.15.16:0/3236469491 wait complete. 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.353+0000 7fa2dd751740 1 -- start start 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.353+0000 7fa2dd751740 1 Processor -- start 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff3a0a0 0x56028032e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff42280 0x560280327ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2dd751740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff62f60 0x5602803283e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2dd751740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560280325f80 con 0x56027ff42280 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2dd751740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602803258b0 con 0x56027ff62f60 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2dd751740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560280326650 con 0x56027ff3a0a0 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2dd751740 10 client.0 _ll_register_callbacks cb 0x7fa2c8003ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff42280 0x560280327ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff42280 0x560280327ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41072/0 (socket says 172.21.15.16:41072) 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2cffff640 1 -- 172.21.15.16:0/3644990391 learned_addr learned my addr 172.21.15.16:0/3644990391 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff62f60 0x5602803283e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff3a0a0 0x56028032e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2cffff640 1 -- 172.21.15.16:0/3644990391 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff3a0a0 msgr2=0x56028032e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff3a0a0 0x56028032e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2cffff640 1 -- 172.21.15.16:0/3644990391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff62f60 msgr2=0x5602803283e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff62f60 0x5602803283e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2cffff640 1 -- 172.21.15.16:0/3644990391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602803289d0 con 0x56027ff42280 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff3a0a0 0x56028032e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff42280 0x560280327ea0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x5602803267d0 tx=0x7fa2bc004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a492dd9cb9e8e1a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2affff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2affff640 1 -- 172.21.15.16:0/3644990391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc005220 con 0x56027ff42280 2024-08-28T19:20:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2dd751740 1 -- 172.21.15.16:0/3644990391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2c8036070 con 0x56027ff42280 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.354+0000 7fa2affff640 1 -- 172.21.15.16:0/3644990391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa2bc0053c0 con 0x56027ff42280 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2affff640 1 -- 172.21.15.16:0/3644990391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa2bc02b050 con 0x56027ff42280 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2dd751740 10 client.15309 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2dd751740 1 -- 172.21.15.16:0/3644990391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56028032e460 con 0x56027ff42280 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2affff640 1 -- 172.21.15.16:0/3644990391 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c8036070 con 0x56027ff42280 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2affff640 1 -- 172.21.15.16:0/3644990391 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa2c8031cb0 con 0x56027ff42280 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2dd751740 10 client.15309 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2dd751740 10 client.15309 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2dd751740 20 client.15309 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56027fef6c90 0x560280394d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2dd751740 4 client.15309 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2dd751740 1 -- 172.21.15.16:0/3644990391 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "status"}) -- 0x56028027cde0 con 0x56027fef6c90 2024-08-28T19:20:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.355+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56027fef6c90 0x560280394d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2d5779640 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56027fef6c90 0x560280394d20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ed813752c6acfb1a server_cookie=ba2eb898b9fa7365 in_seq=0 out_seq=0 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2affff640 10 client.15309 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2affff640 1 -- 172.21.15.16:0/3644990391 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56028027cde0 con 0x56027fef6c90 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2affff640 10 client.15309 handle_command_reply: tid=0 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2dd751740 1 client.15309 shutdown 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2dd751740 1 -- 172.21.15.16:0/3644990391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff42280 msgr2=0x560280327ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff42280 0x560280327ea0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x5602803267d0 tx=0x7fa2bc004610 comp rx=0 tx=0).stop 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2dd751740 1 -- 172.21.15.16:0/3644990391 shutdown_connections 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56027fef6c90 0x560280394d20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56027ff62f60 0x5602803283e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56027ff42280 0x560280327ea0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2affff640 0 client.15309 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:20:20.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2dd751740 1 --2- 172.21.15.16:0/3644990391 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56027ff3a0a0 0x56028032e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.356+0000 7fa2dd751740 1 -- 172.21.15.16:0/3644990391 >> 172.21.15.16:0/3644990391 conn(0x56027ff57350 msgr2=0x560280315ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:20.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.357+0000 7fa2dd751740 1 -- 172.21.15.16:0/3644990391 shutdown_connections 2024-08-28T19:20:20.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.357+0000 7fa2dd751740 1 -- 172.21.15.16:0/3644990391 wait complete. 2024-08-28T19:20:20.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.357+0000 7fa2dd751740 20 client.15309 trim_cache size 0 max 16384 2024-08-28T19:20:20.363 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:20:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:20:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-08-28T19:20:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24625, 2024-08-28T19:20:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:20:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:20:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:20:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 69.725663229999995, 2024-08-28T19:20:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-08-28T19:20:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:20:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 82, 2024-08-28T19:20:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 131.29446306700001 2024-08-28T19:20:20.364 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:20:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.357+0000 7fa2d71dc640 1 -- 172.21.15.16:0/3928579899 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c00b4c10 msgr2=0x7fa2c00b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.357+0000 7fa2d71dc640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c00b4c10 0x7fa2c00b7120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x560280307430 tx=0x7fa2c4004980 comp rx=0 tx=0).stop 2024-08-28T19:20:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.357+0000 7fa2d71dc640 1 -- 172.21.15.16:0/3928579899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d0102800 msgr2=0x7fa2d010fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.357+0000 7fa2d71dc640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d0102800 0x7fa2d010fd50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa2c400c970 tx=0x7fa2c400ce30 comp rx=0 tx=0).stop 2024-08-28T19:20:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.358+0000 7fa2d71dc640 1 -- 172.21.15.16:0/3928579899 shutdown_connections 2024-08-28T19:20:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.358+0000 7fa2d71dc640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c00b4c10 0x7fa2c00b7120 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.358+0000 7fa2d71dc640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d0108a90 0x7fa2d01107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.358+0000 7fa2d71dc640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d0108140 0x7fa2d0110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.358+0000 7fa2d71dc640 1 --2- 172.21.15.16:0/3928579899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d0102800 0x7fa2d010fd50 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.358+0000 7fa2d71dc640 1 -- 172.21.15.16:0/3928579899 >> 172.21.15.16:0/3928579899 conn(0x7fa2d00794d0 msgr2=0x7fa2d0104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:20.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.358+0000 7fa2d71dc640 1 -- 172.21.15.16:0/3928579899 shutdown_connections 2024-08-28T19:20:20.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:20.358+0000 7fa2d71dc640 1 -- 172.21.15.16:0/3928579899 wait complete. 2024-08-28T19:20:21.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.h status 2024-08-28T19:20:21.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:21 smithi028 ceph-mon[21769]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-08-28T19:20:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:21 smithi115 ceph-mon[22184]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-08-28T19:20:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:21.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:21 smithi016 ceph-mon[18556]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-08-28T19:20:21.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.873+0000 7f5cc9a17640 1 -- 172.21.15.16:0/2232714349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cc41081c0 msgr2=0x7f5cc41085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.873+0000 7f5cc9a17640 1 --2- 172.21.15.16:0/2232714349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cc41081c0 0x7f5cc41085d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5cac009900 tx=0x7f5cac02a410 comp rx=0 tx=0).stop 2024-08-28T19:20:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.873+0000 7f5cc9a17640 1 -- 172.21.15.16:0/2232714349 shutdown_connections 2024-08-28T19:20:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.873+0000 7f5cc9a17640 1 --2- 172.21.15.16:0/2232714349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc4108b10 0x7f5cc410f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.873+0000 7f5cc9a17640 1 --2- 172.21.15.16:0/2232714349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cc41081c0 0x7f5cc41085d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.873+0000 7f5cc9a17640 1 --2- 172.21.15.16:0/2232714349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cc4102880 0x7f5cc4102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.873+0000 7f5cc9a17640 1 -- 172.21.15.16:0/2232714349 >> 172.21.15.16:0/2232714349 conn(0x7f5cc4079510 msgr2=0x7f5cc4079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.873+0000 7f5cc9a17640 1 -- 172.21.15.16:0/2232714349 shutdown_connections 2024-08-28T19:20:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.873+0000 7f5cc9a17640 1 -- 172.21.15.16:0/2232714349 wait complete. 2024-08-28T19:20:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.874+0000 7f5cc9a17640 1 Processor -- start 2024-08-28T19:20:21.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.874+0000 7f5cc9a17640 1 -- start start 2024-08-28T19:20:21.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.874+0000 7f5cc9a17640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cc4102880 0x7f5cc419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.874+0000 7f5cc9a17640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cc41081c0 0x7f5cc419b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.874+0000 7f5cc9a17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc4108b10 0x7f5cc419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.874+0000 7f5cc9a17640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc4112780 con 0x7f5cc4102880 2024-08-28T19:20:21.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.874+0000 7f5cc9a17640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc4112600 con 0x7f5cc4108b10 2024-08-28T19:20:21.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.874+0000 7f5cc9a17640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc4112900 con 0x7f5cc41081c0 2024-08-28T19:20:21.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.874+0000 7f5cc2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cc4102880 0x7f5cc419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.874+0000 7f5cc2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cc4102880 0x7f5cc419b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41094/0 (socket says 172.21.15.16:41094) 2024-08-28T19:20:21.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.874+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/3819950135 learned_addr learned my addr 172.21.15.16:0/3819950135 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:21.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc37fe640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc4108b10 0x7f5cc419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc27fc640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cc41081c0 0x7f5cc419b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/3819950135 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cc41081c0 msgr2=0x7f5cc419b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cc41081c0 0x7f5cc419b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/3819950135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc4108b10 msgr2=0x7f5cc419bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc4108b10 0x7f5cc419bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/3819950135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc41a6360 con 0x7f5cc4102880 2024-08-28T19:20:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc27fc640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cc41081c0 0x7f5cc419b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cc4102880 0x7f5cc419b0d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400eab0 tx=0x7f5cb400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc8a15640 1 -- 172.21.15.16:0/3819950135 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb400ce40 con 0x7f5cc4102880 2024-08-28T19:20:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc8a15640 1 -- 172.21.15.16:0/3819950135 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cb4011920 con 0x7f5cc4102880 2024-08-28T19:20:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc9a17640 1 -- 172.21.15.16:0/3819950135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc41a6650 con 0x7f5cc4102880 2024-08-28T19:20:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc8a15640 1 -- 172.21.15.16:0/3819950135 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4010a30 con 0x7f5cc4102880 2024-08-28T19:20:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.875+0000 7f5cc9a17640 1 -- 172.21.15.16:0/3819950135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc41a6b60 con 0x7f5cc4102880 2024-08-28T19:20:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5ccff8c740 1 Processor -- start 2024-08-28T19:20:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5cc8a15640 1 -- 172.21.15.16:0/3819950135 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f5cb4010bd0 con 0x7f5cc4102880 2024-08-28T19:20:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5ccff8c740 1 -- start start 2024-08-28T19:20:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af207b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af20cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab5a280 0x56091af21230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56091ae973f0 con 0x56091ab520a0 2024-08-28T19:20:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56091ae86980 con 0x56091ab7af60 2024-08-28T19:20:21.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56091ae8a930 con 0x56091ab5a280 2024-08-28T19:20:21.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5cc2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af20cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5cc2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af20cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41096/0 (socket says 172.21.15.16:41096) 2024-08-28T19:20:21.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.878+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/1080721536 learned_addr learned my addr 172.21.15.16:0/1080721536 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:21.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc27fc640 1 --2- 172.21.15.16:0/1080721536 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab5a280 0x56091af21230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc37fe640 1 --2- 172.21.15.16:0/1080721536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af207b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/1080721536 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab5a280 msgr2=0x56091af21230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:21.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/1080721536 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab5a280 0x56091af21230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/1080721536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 msgr2=0x56091af207b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:21.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/1080721536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af207b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/1080721536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56091abe6540 con 0x56091ab520a0 2024-08-28T19:20:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc37fe640 1 --2- 172.21.15.16:0/1080721536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af207b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc8a15640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cb00b4c40 0x7f5cb00b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc37fe640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cb00b4c40 0x7f5cb00b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/1080721536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af20cf0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40e3240 tx=0x7f5cb40ee1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45ccc3eff5b0397 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cc8a15640 1 -- 172.21.15.16:0/3819950135 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f5cb40e5de0 con 0x7f5cc4102880 2024-08-28T19:20:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cb9ffb640 1 -- 172.21.15.16:0/1080721536 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40d8e60 con 0x56091ab520a0 2024-08-28T19:20:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cb9ffb640 1 -- 172.21.15.16:0/1080721536 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cb40abd30 con 0x56091ab520a0 2024-08-28T19:20:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.879+0000 7f5cb9ffb640 1 -- 172.21.15.16:0/1080721536 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40abb70 con 0x56091ab520a0 2024-08-28T19:20:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.880+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1080721536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 msgr2=0x56091af20cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.880+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/1080721536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af20cf0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40e3240 tx=0x7f5cb40ee1a0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.880+0000 7f5cc37fe640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cb00b4c40 0x7f5cb00b7150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc013d80 tx=0x7f5cbc005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.880+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1080721536 shutdown_connections 2024-08-28T19:20:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.880+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/1080721536 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab5a280 0x56091af21230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.880+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/1080721536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af20cf0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.880+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/1080721536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af207b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.880+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1080721536 >> 172.21.15.16:0/1080721536 conn(0x56091ab6f350 msgr2=0x56091af1e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.880+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1080721536 shutdown_connections 2024-08-28T19:20:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.880+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1080721536 wait complete. 2024-08-28T19:20:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.881+0000 7f5ccff8c740 1 -- start start 2024-08-28T19:20:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.881+0000 7f5ccff8c740 1 Processor -- start 2024-08-28T19:20:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.881+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab520a0 0x56091af2bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.881+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab5a280 0x56091af2c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.881+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af382e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.881+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56091ace95f0 con 0x56091ab5a280 2024-08-28T19:20:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.881+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56091ae0d600 con 0x56091ab7af60 2024-08-28T19:20:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.881+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56091af2ef90 con 0x56091ab520a0 2024-08-28T19:20:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.881+0000 7f5ccff8c740 10 client.0 _ll_register_callbacks cb 0x56091ae33af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.881+0000 7f5cc27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab5a280 0x56091af2c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.881+0000 7f5cc27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab5a280 0x56091af2c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41102/0 (socket says 172.21.15.16:41102) 2024-08-28T19:20:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5cc27fc640 1 -- 172.21.15.16:0/1265988894 learned_addr learned my addr 172.21.15.16:0/1265988894 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5cc37fe640 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af382e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab520a0 0x56091af2bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5cc27fc640 1 -- 172.21.15.16:0/1265988894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab520a0 msgr2=0x56091af2bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5cc27fc640 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab520a0 0x56091af2bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5cc27fc640 1 -- 172.21.15.16:0/1265988894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 msgr2=0x56091af382e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5cc27fc640 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af382e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5cc27fc640 1 -- 172.21.15.16:0/1265988894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56091ae467b0 con 0x56091ab5a280 2024-08-28T19:20:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5cc27fc640 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab5a280 0x56091af2c420 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5cac000c00 tx=0x7f5cac0316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850726131c8f0e7d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5ca2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/1265988894 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cac035040 con 0x56091ab5a280 2024-08-28T19:20:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1265988894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cb40abd30 con 0x56091ab5a280 2024-08-28T19:20:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/1265988894 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f5cac004240 con 0x56091ab5a280 2024-08-28T19:20:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/1265988894 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5cac04e050 con 0x56091ab5a280 2024-08-28T19:20:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5ccff8c740 10 client.15321 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.882+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1265988894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56091af3ad60 con 0x56091ab5a280 2024-08-28T19:20:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.883+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/1265988894 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cb40abd30 con 0x56091ab5a280 2024-08-28T19:20:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.883+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/1265988894 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f5cac004990 con 0x56091ab5a280 2024-08-28T19:20:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.883+0000 7f5ccff8c740 10 client.15321 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.883+0000 7f5ccff8c740 10 client.15321 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-28T19:20:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.883+0000 7f5ccff8c740 20 client.15321 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.883+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56091ab398e0 0x56091af3e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.883+0000 7f5ccff8c740 4 client.15321 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:20:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.883+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1265988894 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56091ae99340 con 0x56091ab398e0 2024-08-28T19:20:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.883+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56091ab398e0 0x56091af3e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.884+0000 7f5ca2ffd640 10 client.15321 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:20:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.884+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56091ab398e0 0x56091af3e750 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=680d71dc75287237 server_cookie=1f09e6fb8a101a56 in_seq=0 out_seq=0 2024-08-28T19:20:21.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/1265988894 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56091ae99340 con 0x56091ab398e0 2024-08-28T19:20:21.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ca2ffd640 10 client.15321 handle_command_reply: tid=0 2024-08-28T19:20:21.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 1 client.15321 shutdown 2024-08-28T19:20:21.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1265988894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab5a280 msgr2=0x56091af2c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:21.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab5a280 0x56091af2c420 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5cac000c00 tx=0x7f5cac0316d0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1265988894 shutdown_connections 2024-08-28T19:20:21.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56091ab398e0 0x56091af3e750 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af382e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab5a280 0x56091af2c420 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/1265988894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab520a0 0x56091af2bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ca2ffd640 0 client.15321 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:20:21.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1265988894 >> 172.21.15.16:0/1265988894 conn(0x56091ab6f350 msgr2=0x56091af23940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:21.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1265988894 shutdown_connections 2024-08-28T19:20:21.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 1 -- 172.21.15.16:0/1265988894 wait complete. 2024-08-28T19:20:21.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.885+0000 7f5ccff8c740 20 client.15321 trim_cache size 0 max 16384 2024-08-28T19:20:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5ccff8c740 1 Processor -- start 2024-08-28T19:20:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5ccff8c740 1 -- start start 2024-08-28T19:20:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af3c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab5a280 0x56091af38250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af2bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56091af3ca90 con 0x56091ab520a0 2024-08-28T19:20:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56091af3d160 con 0x56091ab7af60 2024-08-28T19:20:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56091af3d830 con 0x56091ab5a280 2024-08-28T19:20:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5cc2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af3c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5cc2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af3c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41104/0 (socket says 172.21.15.16:41104) 2024-08-28T19:20:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/3704180474 learned_addr learned my addr 172.21.15.16:0/3704180474 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5cc27fc640 1 --2- 172.21.15.16:0/3704180474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab5a280 0x56091af38250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/3704180474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab5a280 msgr2=0x56091af38250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/3704180474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab5a280 0x56091af38250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/3704180474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 msgr2=0x56091af2bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/3704180474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af2bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/3704180474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56091af2e620 con 0x56091ab520a0 2024-08-28T19:20:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.904+0000 7f5cc27fc640 1 --2- 172.21.15.16:0/3704180474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab5a280 0x56091af38250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.905+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/3704180474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af3c1d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40e3380 tx=0x7f5cb40104d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37b5f94e39659e6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.905+0000 7f5cb97fa640 1 -- 172.21.15.16:0/3704180474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40e9a20 con 0x56091ab520a0 2024-08-28T19:20:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.905+0000 7f5cb97fa640 1 -- 172.21.15.16:0/3704180474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cb40e9460 con 0x56091ab520a0 2024-08-28T19:20:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.905+0000 7f5cb97fa640 1 -- 172.21.15.16:0/3704180474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40ab990 con 0x56091ab520a0 2024-08-28T19:20:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.906+0000 7f5ccff8c740 1 -- 172.21.15.16:0/3704180474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 msgr2=0x56091af3c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.906+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/3704180474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af3c1d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40e3380 tx=0x7f5cb40104d0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.906+0000 7f5ccff8c740 1 -- 172.21.15.16:0/3704180474 shutdown_connections 2024-08-28T19:20:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.906+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/3704180474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab7af60 0x56091af2bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.906+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/3704180474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab5a280 0x56091af38250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.906+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/3704180474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af3c1d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.906+0000 7f5ccff8c740 1 -- 172.21.15.16:0/3704180474 >> 172.21.15.16:0/3704180474 conn(0x56091ab6f350 msgr2=0x56091af2a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.906+0000 7f5ccff8c740 1 -- 172.21.15.16:0/3704180474 shutdown_connections 2024-08-28T19:20:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.906+0000 7f5ccff8c740 1 -- 172.21.15.16:0/3704180474 wait complete. 2024-08-28T19:20:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.906+0000 7f5ccff8c740 1 -- start start 2024-08-28T19:20:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.906+0000 7f5ccff8c740 1 Processor -- start 2024-08-28T19:20:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af40b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab5a280 0x56091af41050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5ccff8c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab7af60 0x56091af41590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56091af3e5d0 con 0x56091ab520a0 2024-08-28T19:20:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56091af3df00 con 0x56091ab5a280 2024-08-28T19:20:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5ccff8c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56091af3eca0 con 0x56091ab7af60 2024-08-28T19:20:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5ccff8c740 10 client.0 _ll_register_callbacks cb 0x7f5cac005e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af40b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af40b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41112/0 (socket says 172.21.15.16:41112) 2024-08-28T19:20:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/555135118 learned_addr learned my addr 172.21.15.16:0/555135118 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc37fe640 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab7af60 0x56091af41590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc27fc640 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab5a280 0x56091af41050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/555135118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab7af60 msgr2=0x56091af41590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab7af60 0x56091af41590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/555135118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab5a280 msgr2=0x56091af41050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab5a280 0x56091af41050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc2ffd640 1 -- 172.21.15.16:0/555135118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56091af46400 con 0x56091ab520a0 2024-08-28T19:20:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc37fe640 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab7af60 0x56091af41590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5cc2ffd640 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af40b10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4000f80 tx=0x7f5cb40abb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f36ffcf8acf7eed server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5ca2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/555135118 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40abff0 con 0x56091ab520a0 2024-08-28T19:20:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/555135118 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f5cb40e3550 con 0x56091ab520a0 2024-08-28T19:20:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.907+0000 7f5ccff8c740 1 -- 172.21.15.16:0/555135118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56091ae94de0 con 0x56091ab520a0 2024-08-28T19:20:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/555135118 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5cb40abd30 con 0x56091ab520a0 2024-08-28T19:20:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5ccff8c740 10 client.15327 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5ccff8c740 1 -- 172.21.15.16:0/555135118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56091afa9f40 con 0x56091ab520a0 2024-08-28T19:20:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/555135118 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56091ae94de0 con 0x56091ab520a0 2024-08-28T19:20:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/555135118 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f5cb400f040 con 0x56091ab520a0 2024-08-28T19:20:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5ccff8c740 10 client.15327 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5ccff8c740 10 client.15327 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-28T19:20:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5ccff8c740 20 client.15327 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56091ab0ec90 0x56091afad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5ccff8c740 4 client.15327 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:20:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5ccff8c740 1 -- 172.21.15.16:0/555135118 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "status"}) -- 0x56091ae92520 con 0x56091ab0ec90 2024-08-28T19:20:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.908+0000 7f5cc27fc640 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56091ab0ec90 0x56091afad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ca2ffd640 10 client.15327 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:20:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5cc27fc640 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56091ab0ec90 0x56091afad230 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=38ddeb95ffb498fd server_cookie=9c89e953524598c0 in_seq=0 out_seq=0 2024-08-28T19:20:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ca2ffd640 1 -- 172.21.15.16:0/555135118 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56091ae92520 con 0x56091ab0ec90 2024-08-28T19:20:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ca2ffd640 10 client.15327 handle_command_reply: tid=0 2024-08-28T19:20:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 1 client.15327 shutdown 2024-08-28T19:20:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 1 -- 172.21.15.16:0/555135118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 msgr2=0x56091af40b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af40b10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4000f80 tx=0x7f5cb40abb70 comp rx=0 tx=0).stop 2024-08-28T19:20:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 1 -- 172.21.15.16:0/555135118 shutdown_connections 2024-08-28T19:20:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56091ab0ec90 0x56091afad230 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56091ab7af60 0x56091af41590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56091ab5a280 0x56091af41050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ca2ffd640 0 client.15327 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:20:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 1 --2- 172.21.15.16:0/555135118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56091ab520a0 0x56091af40b10 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 1 -- 172.21.15.16:0/555135118 >> 172.21.15.16:0/555135118 conn(0x56091ab6f350 msgr2=0x56091af2bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 1 -- 172.21.15.16:0/555135118 shutdown_connections 2024-08-28T19:20:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 1 -- 172.21.15.16:0/555135118 wait complete. 2024-08-28T19:20:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.909+0000 7f5ccff8c740 20 client.15327 trim_cache size 0 max 16384 2024-08-28T19:20:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:20:21.918 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:20:21.918 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-08-28T19:20:21.918 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24635, 2024-08-28T19:20:21.918 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:20:21.918 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:20:21.918 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:20:21.919 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 72.284470798000001, 2024-08-28T19:20:21.919 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-08-28T19:20:21.919 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:20:21.919 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 82, 2024-08-28T19:20:21.919 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 129.964166549 2024-08-28T19:20:21.919 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:20:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 -- 172.21.15.16:0/3819950135 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cb00b4c40 msgr2=0x7f5cb00b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cb00b4c40 0x7f5cb00b7150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc013d80 tx=0x7f5cbc005f90 comp rx=0 tx=0).stop 2024-08-28T19:20:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 -- 172.21.15.16:0/3819950135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cc4102880 msgr2=0x7f5cc419b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cc4102880 0x7f5cc419b0d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400eab0 tx=0x7f5cb400ef70 comp rx=0 tx=0).stop 2024-08-28T19:20:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 -- 172.21.15.16:0/3819950135 shutdown_connections 2024-08-28T19:20:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cb00b4c40 0x7f5cb00b7150 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc4108b10 0x7f5cc419bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cc41081c0 0x7f5cc419b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 --2- 172.21.15.16:0/3819950135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cc4102880 0x7f5cc419b0d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 -- 172.21.15.16:0/3819950135 >> 172.21.15.16:0/3819950135 conn(0x7f5cc4079510 msgr2=0x7f5cc41062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 -- 172.21.15.16:0/3819950135 shutdown_connections 2024-08-28T19:20:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:21.910+0000 7f5cc9a17640 1 -- 172.21.15.16:0/3819950135 wait complete. 2024-08-28T19:20:22.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.i status 2024-08-28T19:20:22.954 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:23.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:23 smithi028 ceph-mon[21769]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:20:23.328 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:23 smithi016 ceph-mon[18556]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:20:23.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:23 smithi115 ceph-mon[22184]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:20:23.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.419+0000 7fa6e3915640 1 -- 172.21.15.16:0/822377277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6dc108ab0 msgr2=0x7fa6dc10f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:23.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.419+0000 7fa6e3915640 1 --2- 172.21.15.16:0/822377277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6dc108ab0 0x7fa6dc10f2e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009ca0 tx=0x7fa6d402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.419+0000 7fa6e3915640 1 -- 172.21.15.16:0/822377277 shutdown_connections 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.419+0000 7fa6e3915640 1 --2- 172.21.15.16:0/822377277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6dc108ab0 0x7fa6dc10f2e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.419+0000 7fa6e3915640 1 --2- 172.21.15.16:0/822377277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6dc108160 0x7fa6dc108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.419+0000 7fa6e3915640 1 --2- 172.21.15.16:0/822377277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6dc102820 0x7fa6dc102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.419+0000 7fa6e3915640 1 -- 172.21.15.16:0/822377277 >> 172.21.15.16:0/822377277 conn(0x7fa6dc0794e0 msgr2=0x7fa6dc0798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.420+0000 7fa6e3915640 1 -- 172.21.15.16:0/822377277 shutdown_connections 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.420+0000 7fa6e3915640 1 -- 172.21.15.16:0/822377277 wait complete. 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.420+0000 7fa6e3915640 1 Processor -- start 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.420+0000 7fa6e3915640 1 -- start start 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e3915640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6dc102820 0x7fa6dc10fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e3915640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6dc108160 0x7fa6dc1102b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e3915640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6dc108ab0 0x7fa6dc1107f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e3915640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc113880 con 0x7fa6dc108160 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e3915640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc113700 con 0x7fa6dc108ab0 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e3915640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc113a00 con 0x7fa6dc102820 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e0eb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6dc108160 0x7fa6dc1102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e0eb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6dc108160 0x7fa6dc1102b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41134/0 (socket says 172.21.15.16:41134) 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e0eb0640 1 -- 172.21.15.16:0/3383215485 learned_addr learned my addr 172.21.15.16:0/3383215485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6dc102820 0x7fa6dc10fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e1eb2640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6dc108ab0 0x7fa6dc1107f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e0eb0640 1 -- 172.21.15.16:0/3383215485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6dc102820 msgr2=0x7fa6dc10fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6dc102820 0x7fa6dc10fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e0eb0640 1 -- 172.21.15.16:0/3383215485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6dc108ab0 msgr2=0x7fa6dc1107f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6dc108ab0 0x7fa6dc1107f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e0eb0640 1 -- 172.21.15.16:0/3383215485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6dc1a8510 con 0x7fa6dc108160 2024-08-28T19:20:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6dc108160 0x7fa6dc1102b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc00b450 tx=0x7fa6cc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6d27fc640 1 -- 172.21.15.16:0/3383215485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc015020 con 0x7fa6dc108160 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6e3915640 1 -- 172.21.15.16:0/3383215485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6dc1a8800 con 0x7fa6dc108160 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6d27fc640 1 -- 172.21.15.16:0/3383215485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6cc011d40 con 0x7fa6dc108160 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.421+0000 7fa6d27fc640 1 -- 172.21.15.16:0/3383215485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc00f340 con 0x7fa6dc108160 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.422+0000 7fa6e3915640 1 -- 172.21.15.16:0/3383215485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6dc1a8d40 con 0x7fa6dc108160 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.424+0000 7fa6e9e8a740 1 Processor -- start 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.424+0000 7fa6e9e8a740 1 -- start start 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.424+0000 7fa6d27fc640 1 -- 172.21.15.16:0/3383215485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fa6cc018020 con 0x7fa6dc108160 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55915328df60 0x5591536337b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x559153633cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x559153634230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591535aa3f0 con 0x5591532650a0 2024-08-28T19:20:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559153599980 con 0x55915326d280 2024-08-28T19:20:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55915359d930 con 0x55915328df60 2024-08-28T19:20:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e16b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x559153633cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e16b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x559153633cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41140/0 (socket says 172.21.15.16:41140) 2024-08-28T19:20:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e16b1640 1 -- 172.21.15.16:0/1578929018 learned_addr learned my addr 172.21.15.16:0/1578929018 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e1eb2640 1 --2- 172.21.15.16:0/1578929018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55915328df60 0x5591536337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/1578929018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x559153634230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e16b1640 1 -- 172.21.15.16:0/1578929018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55915328df60 msgr2=0x5591536337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/1578929018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55915328df60 0x5591536337b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e16b1640 1 -- 172.21.15.16:0/1578929018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 msgr2=0x559153634230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/1578929018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x559153634230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e16b1640 1 -- 172.21.15.16:0/1578929018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591532f9540 con 0x5591532650a0 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/1578929018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x559153634230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.425+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/1578929018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x559153633cf0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800bd10 tx=0x7fa6d80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bc124a25b9d165 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.426+0000 7fa6c77fe640 1 -- 172.21.15.16:0/1578929018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8005220 con 0x5591532650a0 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.426+0000 7fa6d27fc640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6c80b4c10 0x7fa6c80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.426+0000 7fa6c77fe640 1 -- 172.21.15.16:0/1578929018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6d8004c50 con 0x5591532650a0 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.426+0000 7fa6c77fe640 1 -- 172.21.15.16:0/1578929018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8011660 con 0x5591532650a0 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.426+0000 7fa6e1eb2640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6c80b4c10 0x7fa6c80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.426+0000 7fa6d27fc640 1 -- 172.21.15.16:0/3383215485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa6cc0e5100 con 0x7fa6dc108160 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.426+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/1578929018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 msgr2=0x559153633cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.426+0000 7fa6e1eb2640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6c80b4c10 0x7fa6c80b7120 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4006010 tx=0x7fa6d402f310 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.426+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/1578929018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x559153633cf0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800bd10 tx=0x7fa6d80107a0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.427+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/1578929018 shutdown_connections 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.427+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/1578929018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x559153634230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.427+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/1578929018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x559153633cf0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.427+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/1578929018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55915328df60 0x5591536337b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.427+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/1578929018 >> 172.21.15.16:0/1578929018 conn(0x559153282350 msgr2=0x5591536316a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.427+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/1578929018 shutdown_connections 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.427+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/1578929018 wait complete. 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.427+0000 7fa6e9e8a740 1 -- start start 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.427+0000 7fa6e9e8a740 1 Processor -- start 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591532650a0 0x55915363ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x55915363f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915328df60 0x55915364b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591533fc5f0 con 0x55915328df60 2024-08-28T19:20:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559153520600 con 0x55915326d280 2024-08-28T19:20:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559153641f90 con 0x5591532650a0 2024-08-28T19:20:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e9e8a740 10 client.0 _ll_register_callbacks cb 0x559153546af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e1eb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915328df60 0x55915364b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e16b1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591532650a0 0x55915363ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e1eb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915328df60 0x55915364b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41142/0 (socket says 172.21.15.16:41142) 2024-08-28T19:20:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e1eb2640 1 -- 172.21.15.16:0/171240696 learned_addr learned my addr 172.21.15.16:0/171240696 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x55915363f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e16b1640 1 -- 172.21.15.16:0/171240696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 msgr2=0x55915363f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x55915363f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e16b1640 1 -- 172.21.15.16:0/171240696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915328df60 msgr2=0x55915364b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915328df60 0x55915364b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e16b1640 1 -- 172.21.15.16:0/171240696 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591535597b0 con 0x5591532650a0 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e1eb2640 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915328df60 0x55915364b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.428+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x55915363f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.429+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591532650a0 0x55915363ee80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8000c00 tx=0x7fa6d80055b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c8751e3de07b399 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.429+0000 7fa6c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.429+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/171240696 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8004a70 con 0x5591532650a0 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.429+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/171240696 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6d8004c50 con 0x5591532650a0 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.429+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/171240696 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa6d8005010 con 0x5591532650a0 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.429+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/171240696 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa6d8005210 con 0x5591532650a0 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.429+0000 7fa6e9e8a740 10 client.25009 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.429+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/171240696 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55915364dc00 con 0x5591532650a0 2024-08-28T19:20:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.429+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/171240696 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6d8004c50 con 0x5591532650a0 2024-08-28T19:20:23.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.429+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/171240696 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa6d80039c0 con 0x5591532650a0 2024-08-28T19:20:23.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.429+0000 7fa6e9e8a740 10 client.25009 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:23.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.430+0000 7fa6e9e8a740 10 client.25009 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-28T19:20:23.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.430+0000 7fa6e9e8a740 20 client.25009 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:23.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.430+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55915324c8e0 0x559153651240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.430+0000 7fa6e9e8a740 4 client.25009 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.430+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/171240696 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591535ac340 con 0x55915324c8e0 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.430+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55915324c8e0 0x559153651240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.430+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55915324c8e0 0x559153651240 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1b0550e307ede7fc server_cookie=65c831b45d4f2aad in_seq=0 out_seq=0 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.430+0000 7fa6c4ff9640 10 client.25009 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/171240696 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5591535ac340 con 0x55915324c8e0 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6c4ff9640 10 client.25009 handle_command_reply: tid=0 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6e9e8a740 1 client.25009 shutdown 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/171240696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591532650a0 msgr2=0x55915363ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591532650a0 0x55915363ee80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8000c00 tx=0x7fa6d80055b0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/171240696 shutdown_connections 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55915324c8e0 0x559153651240 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915328df60 0x55915364b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x55915363f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6c4ff9640 0 client.25009 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:20:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/171240696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591532650a0 0x55915363ee80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.431+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/171240696 >> 172.21.15.16:0/171240696 conn(0x559153282350 msgr2=0x5591536347e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.432+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/171240696 shutdown_connections 2024-08-28T19:20:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.432+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/171240696 wait complete. 2024-08-28T19:20:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.432+0000 7fa6e9e8a740 20 client.25009 trim_cache size 0 max 16384 2024-08-28T19:20:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e9e8a740 1 Processor -- start 2024-08-28T19:20:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e9e8a740 1 -- start start 2024-08-28T19:20:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55915324c8e0 0x55915364fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x55915363a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x55915364b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55915364f580 con 0x5591532650a0 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55915364fc50 con 0x55915326d280 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559153650320 con 0x55915324c8e0 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e0eb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x55915363a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e0eb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x55915363a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41154/0 (socket says 172.21.15.16:41154) 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e0eb0640 1 -- 172.21.15.16:0/3260688532 learned_addr learned my addr 172.21.15.16:0/3260688532 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.450+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/3260688532 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55915324c8e0 0x55915364fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.451+0000 7fa6e0eb0640 1 -- 172.21.15.16:0/3260688532 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55915324c8e0 msgr2=0x55915364fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.451+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/3260688532 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55915324c8e0 0x55915364fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.451+0000 7fa6e0eb0640 1 -- 172.21.15.16:0/3260688532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 msgr2=0x55915364b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.451+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/3260688532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x55915364b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.451+0000 7fa6e0eb0640 1 -- 172.21.15.16:0/3260688532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559153640140 con 0x5591532650a0 2024-08-28T19:20:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.451+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/3260688532 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55915324c8e0 0x55915364fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.451+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/3260688532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x55915363a030 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0ab4d0 tx=0x7fa6cc0e75f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615c735673c004f8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.451+0000 7fa6c6ffd640 1 -- 172.21.15.16:0/3260688532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc0afdb0 con 0x5591532650a0 2024-08-28T19:20:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.451+0000 7fa6c6ffd640 1 -- 172.21.15.16:0/3260688532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6cc0b4860 con 0x5591532650a0 2024-08-28T19:20:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.451+0000 7fa6c6ffd640 1 -- 172.21.15.16:0/3260688532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc0b4630 con 0x5591532650a0 2024-08-28T19:20:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.452+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/3260688532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 msgr2=0x55915363a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.452+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/3260688532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x55915363a030 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0ab4d0 tx=0x7fa6cc0e75f0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.452+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/3260688532 shutdown_connections 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.452+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/3260688532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915326d280 0x55915364b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.452+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/3260688532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591532650a0 0x55915363a030 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.452+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/3260688532 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55915324c8e0 0x55915364fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.452+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/3260688532 >> 172.21.15.16:0/3260688532 conn(0x559153282350 msgr2=0x55915362c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.452+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/3260688532 shutdown_connections 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.452+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/3260688532 wait complete. 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.452+0000 7fa6e9e8a740 1 -- start start 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.452+0000 7fa6e9e8a740 1 Processor -- start 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915324c8e0 0x5591536541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591532650a0 0x5591536547b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e9e8a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915326d280 0x559153658260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591536510c0 con 0x55915326d280 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591536509f0 con 0x55915324c8e0 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e9e8a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559153651790 con 0x5591532650a0 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e9e8a740 10 client.0 _ll_register_callbacks cb 0x7fa6d800bec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e1eb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915326d280 0x559153658260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e1eb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915326d280 0x559153658260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41164/0 (socket says 172.21.15.16:41164) 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e1eb2640 1 -- 172.21.15.16:0/2670302908 learned_addr learned my addr 172.21.15.16:0/2670302908 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915324c8e0 0x5591536541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e0eb0640 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591532650a0 0x5591536547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e1eb2640 1 -- 172.21.15.16:0/2670302908 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591532650a0 msgr2=0x5591536547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e1eb2640 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591532650a0 0x5591536547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e1eb2640 1 -- 172.21.15.16:0/2670302908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915324c8e0 msgr2=0x5591536541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e1eb2640 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915324c8e0 0x5591536541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.453+0000 7fa6e1eb2640 1 -- 172.21.15.16:0/2670302908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591536588d0 con 0x55915326d280 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6e1eb2640 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915326d280 0x559153658260 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4031a20 tx=0x7fa6d402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f5426f67e495c7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/2670302908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4059060 con 0x55915326d280 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/2670302908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa6d40043d0 con 0x55915326d280 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/2670302908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6cc0b4860 con 0x55915326d280 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/2670302908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa6d4060050 con 0x55915326d280 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6e9e8a740 10 client.15345 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/2670302908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559153659c90 con 0x55915326d280 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/2670302908 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6cc0b4860 con 0x55915326d280 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/2670302908 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa6d40305c0 con 0x55915326d280 2024-08-28T19:20:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6e9e8a740 10 client.15345 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6e9e8a740 10 client.15345 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6e9e8a740 20 client.15345 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55915328df60 0x5591536bf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6e9e8a740 4 client.15345 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.454+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/2670302908 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "status"}) -- 0x5591535a7de0 con 0x55915328df60 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.455+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55915328df60 0x5591536bf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.455+0000 7fa6e16b1640 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55915328df60 0x5591536bf7f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d8a0474322cf92b5 server_cookie=ce76749dcbedf3f1 in_seq=0 out_seq=0 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.455+0000 7fa6c4ff9640 10 client.15345 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6c4ff9640 1 -- 172.21.15.16:0/2670302908 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5591535a7de0 con 0x55915328df60 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6c4ff9640 10 client.15345 handle_command_reply: tid=0 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 1 client.15345 shutdown 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/2670302908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915326d280 msgr2=0x559153658260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915326d280 0x559153658260 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4031a20 tx=0x7fa6d402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/2670302908 shutdown_connections 2024-08-28T19:20:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55915328df60 0x5591536bf7f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915326d280 0x559153658260 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591532650a0 0x5591536547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 1 --2- 172.21.15.16:0/2670302908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55915324c8e0 0x5591536541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6c4ff9640 0 client.15345 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:20:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/2670302908 >> 172.21.15.16:0/2670302908 conn(0x559153282350 msgr2=0x55915362c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/2670302908 shutdown_connections 2024-08-28T19:20:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 1 -- 172.21.15.16:0/2670302908 wait complete. 2024-08-28T19:20:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.456+0000 7fa6e9e8a740 20 client.15345 trim_cache size 0 max 16384 2024-08-28T19:20:23.462 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24649, 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 74.843716853000004, 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 82, 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 128.93865065700001 2024-08-28T19:20:23.463 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:20:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 -- 172.21.15.16:0/3383215485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6c80b4c10 msgr2=0x7fa6c80b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6c80b4c10 0x7fa6c80b7120 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4006010 tx=0x7fa6d402f310 comp rx=0 tx=0).stop 2024-08-28T19:20:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 -- 172.21.15.16:0/3383215485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6dc108160 msgr2=0x7fa6dc1102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6dc108160 0x7fa6dc1102b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc00b450 tx=0x7fa6cc00b910 comp rx=0 tx=0).stop 2024-08-28T19:20:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 -- 172.21.15.16:0/3383215485 shutdown_connections 2024-08-28T19:20:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6c80b4c10 0x7fa6c80b7120 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6dc108ab0 0x7fa6dc1107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6dc108160 0x7fa6dc1102b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 --2- 172.21.15.16:0/3383215485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6dc102820 0x7fa6dc10fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 -- 172.21.15.16:0/3383215485 >> 172.21.15.16:0/3383215485 conn(0x7fa6dc0794e0 msgr2=0x7fa6dc105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 -- 172.21.15.16:0/3383215485 shutdown_connections 2024-08-28T19:20:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:23.457+0000 7fa6e3915640 1 -- 172.21.15.16:0/3383215485 wait complete. 2024-08-28T19:20:24.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.b status 2024-08-28T19:20:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.955+0000 7fd267d8c640 1 -- 172.21.15.16:0/728782922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd260102880 msgr2=0x7fd260102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.955+0000 7fd267d8c640 1 --2- 172.21.15.16:0/728782922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd260102880 0x7fd260102c90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009900 tx=0x7fd25c02a410 comp rx=0 tx=0).stop 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.955+0000 7fd267d8c640 1 -- 172.21.15.16:0/728782922 shutdown_connections 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.955+0000 7fd267d8c640 1 --2- 172.21.15.16:0/728782922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd260108b10 0x7fd26010f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.955+0000 7fd267d8c640 1 --2- 172.21.15.16:0/728782922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2601081c0 0x7fd2601085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.955+0000 7fd267d8c640 1 --2- 172.21.15.16:0/728782922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd260102880 0x7fd260102c90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.955+0000 7fd267d8c640 1 -- 172.21.15.16:0/728782922 >> 172.21.15.16:0/728782922 conn(0x7fd260079510 msgr2=0x7fd260079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.955+0000 7fd267d8c640 1 -- 172.21.15.16:0/728782922 shutdown_connections 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.955+0000 7fd267d8c640 1 -- 172.21.15.16:0/728782922 wait complete. 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.955+0000 7fd267d8c640 1 Processor -- start 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd267d8c640 1 -- start start 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd267d8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260102880 0x7fd26019b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd267d8c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2601081c0 0x7fd26019b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd267d8c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd260108b10 0x7fd26019bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd267d8c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd2601127c0 con 0x7fd2601081c0 2024-08-28T19:20:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd267d8c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd260112640 con 0x7fd260102880 2024-08-28T19:20:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd267d8c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd260112940 con 0x7fd260108b10 2024-08-28T19:20:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd265327640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2601081c0 0x7fd26019b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd265327640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2601081c0 0x7fd26019b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47740/0 (socket says 172.21.15.16:47740) 2024-08-28T19:20:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd265327640 1 -- 172.21.15.16:0/2646136374 learned_addr learned my addr 172.21.15.16:0/2646136374 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd265b28640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260102880 0x7fd26019b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd266329640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd260108b10 0x7fd26019bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd265327640 1 -- 172.21.15.16:0/2646136374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd260108b10 msgr2=0x7fd26019bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd265327640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd260108b10 0x7fd26019bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd265327640 1 -- 172.21.15.16:0/2646136374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260102880 msgr2=0x7fd26019b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd265327640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260102880 0x7fd26019b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.956+0000 7fd265327640 1 -- 172.21.15.16:0/2646136374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2601a6350 con 0x7fd2601081c0 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.957+0000 7fd265327640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2601081c0 0x7fd26019b690 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd25000c8c0 tx=0x7fd25000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.957+0000 7fd256ffd640 1 -- 172.21.15.16:0/2646136374 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250007bb0 con 0x7fd2601081c0 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.957+0000 7fd256ffd640 1 -- 172.21.15.16:0/2646136374 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd250007d50 con 0x7fd2601081c0 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.957+0000 7fd267d8c640 1 -- 172.21.15.16:0/2646136374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2601a6640 con 0x7fd2601081c0 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.957+0000 7fd256ffd640 1 -- 172.21.15.16:0/2646136374 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2500187c0 con 0x7fd2601081c0 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.957+0000 7fd267d8c640 1 -- 172.21.15.16:0/2646136374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2601a6bd0 con 0x7fd2601081c0 2024-08-28T19:20:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.959+0000 7fd26e301740 1 Processor -- start 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.959+0000 7fd26e301740 1 -- start start 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd256ffd640 1 -- 172.21.15.16:0/2646136374 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fd25000fa20 con 0x7fd2601081c0 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd26e301740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06003bf60 0x55a0603e17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd26e301740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0600130a0 0x55a0603e1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd26e301740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06001b280 0x55a0603e2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd26e301740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a0603583f0 con 0x55a06001b280 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd26e301740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a060347980 con 0x55a0600130a0 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd26e301740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a06034b930 con 0x55a06003bf60 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd265327640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06001b280 0x55a0603e2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd265327640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06001b280 0x55a0603e2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47746/0 (socket says 172.21.15.16:47746) 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd265327640 1 -- 172.21.15.16:0/1518188219 learned_addr learned my addr 172.21.15.16:0/1518188219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd265b28640 1 --2- 172.21.15.16:0/1518188219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0600130a0 0x55a0603e1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd266329640 1 --2- 172.21.15.16:0/1518188219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06003bf60 0x55a0603e17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd256ffd640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd24c0b4cc0 0x7fd24c0b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd265327640 1 -- 172.21.15.16:0/1518188219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06003bf60 msgr2=0x55a0603e17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd265327640 1 --2- 172.21.15.16:0/1518188219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06003bf60 0x55a0603e17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd265327640 1 -- 172.21.15.16:0/1518188219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0600130a0 msgr2=0x55a0603e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd265327640 1 --2- 172.21.15.16:0/1518188219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0600130a0 0x55a0603e1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd265327640 1 -- 172.21.15.16:0/1518188219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0600a7540 con 0x55a06001b280 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.960+0000 7fd265b28640 1 --2- 172.21.15.16:0/1518188219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0600130a0 0x55a0603e1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.961+0000 7fd266329640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd24c0b4cc0 0x7fd24c0b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.961+0000 7fd256ffd640 1 -- 172.21.15.16:0/2646136374 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fd2500e5760 con 0x7fd2601081c0 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.961+0000 7fd265327640 1 --2- 172.21.15.16:0/1518188219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06001b280 0x55a0603e2230 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd2500e2bc0 tx=0x7fd250070550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e521419c40338842 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.961+0000 7fd24bfff640 1 -- 172.21.15.16:0/1518188219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2500ab970 con 0x55a06001b280 2024-08-28T19:20:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.961+0000 7fd266329640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd24c0b4cc0 0x7fd24c0b71d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd25800af30 tx=0x7fd258004a40 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.961+0000 7fd24bfff640 1 -- 172.21.15.16:0/1518188219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2500ab6b0 con 0x55a06001b280 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.961+0000 7fd24bfff640 1 -- 172.21.15.16:0/1518188219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2500a7330 con 0x55a06001b280 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.962+0000 7fd26e301740 1 -- 172.21.15.16:0/1518188219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06001b280 msgr2=0x55a0603e2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.962+0000 7fd26e301740 1 --2- 172.21.15.16:0/1518188219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06001b280 0x55a0603e2230 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd2500e2bc0 tx=0x7fd250070550 comp rx=0 tx=0).stop 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.962+0000 7fd26e301740 1 -- 172.21.15.16:0/1518188219 shutdown_connections 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.962+0000 7fd26e301740 1 --2- 172.21.15.16:0/1518188219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06001b280 0x55a0603e2230 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.962+0000 7fd26e301740 1 --2- 172.21.15.16:0/1518188219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0600130a0 0x55a0603e1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.962+0000 7fd26e301740 1 --2- 172.21.15.16:0/1518188219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06003bf60 0x55a0603e17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.962+0000 7fd26e301740 1 -- 172.21.15.16:0/1518188219 >> 172.21.15.16:0/1518188219 conn(0x55a060030350 msgr2=0x55a0603df6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.962+0000 7fd26e301740 1 -- 172.21.15.16:0/1518188219 shutdown_connections 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.962+0000 7fd26e301740 1 -- 172.21.15.16:0/1518188219 wait complete. 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.962+0000 7fd26e301740 1 -- start start 2024-08-28T19:20:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.963+0000 7fd26e301740 1 Processor -- start 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.963+0000 7fd26e301740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a0603ece70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.963+0000 7fd26e301740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06001b280 0x55a0603ed3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.963+0000 7fd26e301740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06003bf60 0x55a0603f9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.963+0000 7fd26e301740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a0601aa5f0 con 0x55a0600130a0 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.963+0000 7fd26e301740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0602ce600 con 0x55a06003bf60 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.963+0000 7fd26e301740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a0603eff30 con 0x55a06001b280 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.963+0000 7fd265b28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a0603ece70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.963+0000 7fd26e301740 10 client.0 _ll_register_callbacks cb 0x55a0602f4af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.963+0000 7fd265b28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a0603ece70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47758/0 (socket says 172.21.15.16:47758) 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.963+0000 7fd265b28640 1 -- 172.21.15.16:0/3847698326 learned_addr learned my addr 172.21.15.16:0/3847698326 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd266329640 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06003bf60 0x55a0603f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd265327640 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06001b280 0x55a0603ed3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd265b28640 1 -- 172.21.15.16:0/3847698326 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06001b280 msgr2=0x55a0603ed3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd265b28640 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06001b280 0x55a0603ed3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd265b28640 1 -- 172.21.15.16:0/3847698326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06003bf60 msgr2=0x55a0603f9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd265b28640 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06003bf60 0x55a0603f9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd265b28640 1 -- 172.21.15.16:0/3847698326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0603077b0 con 0x55a0600130a0 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd266329640 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06003bf60 0x55a0603f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd265b28640 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a0603ece70 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009900 tx=0x7fd25c0316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902cd04f1b8f0344 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd2497fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd2497fa640 1 -- 172.21.15.16:0/3847698326 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25c035040 con 0x55a0600130a0 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd2497fa640 1 -- 172.21.15.16:0/3847698326 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fd25c004040 con 0x55a0600130a0 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd26e301740 1 -- 172.21.15.16:0/3847698326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2500ab6b0 con 0x55a0600130a0 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd2497fa640 1 -- 172.21.15.16:0/3847698326 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd25c04e050 con 0x55a0600130a0 2024-08-28T19:20:24.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd26e301740 10 client.15357 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:24.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.964+0000 7fd26e301740 1 -- 172.21.15.16:0/3847698326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a0603fbc40 con 0x55a0600130a0 2024-08-28T19:20:24.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.965+0000 7fd2497fa640 1 -- 172.21.15.16:0/3847698326 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2500ab6b0 con 0x55a0600130a0 2024-08-28T19:20:24.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.965+0000 7fd2497fa640 1 -- 172.21.15.16:0/3847698326 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd25c03d450 con 0x55a0600130a0 2024-08-28T19:20:24.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.965+0000 7fd26e301740 10 client.15357 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:24.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.965+0000 7fd26e301740 10 client.15357 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-28T19:20:24.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.965+0000 7fd26e301740 20 client.15357 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:24.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.965+0000 7fd26e301740 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a05fffa8e0 0x55a0603ff6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.965+0000 7fd26e301740 4 client.15357 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:20:24.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.965+0000 7fd26e301740 1 -- 172.21.15.16:0/3847698326 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a06035a340 con 0x55a05fffa8e0 2024-08-28T19:20:24.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.965+0000 7fd265327640 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a05fffa8e0 0x55a0603ff6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.966+0000 7fd265327640 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a05fffa8e0 0x55a0603ff6e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79d358b5dd7e8bd1 server_cookie=d21187f231a589eb in_seq=0 out_seq=0 2024-08-28T19:20:24.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.966+0000 7fd2497fa640 10 client.15357 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:20:24.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.966+0000 7fd2497fa640 1 -- 172.21.15.16:0/3847698326 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a06035a340 con 0x55a05fffa8e0 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.966+0000 7fd2497fa640 10 client.15357 handle_command_reply: tid=0 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 1 client.15357 shutdown 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 1 -- 172.21.15.16:0/3847698326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 msgr2=0x55a0603ece70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a0603ece70 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009900 tx=0x7fd25c0316d0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 1 -- 172.21.15.16:0/3847698326 shutdown_connections 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06003bf60 0x55a0603f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a05fffa8e0 0x55a0603ff6e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06001b280 0x55a0603ed3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 1 --2- 172.21.15.16:0/3847698326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a0603ece70 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd2497fa640 0 client.15357 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 1 -- 172.21.15.16:0/3847698326 >> 172.21.15.16:0/3847698326 conn(0x55a060030350 msgr2=0x55a0603e4940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 1 -- 172.21.15.16:0/3847698326 shutdown_connections 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 1 -- 172.21.15.16:0/3847698326 wait complete. 2024-08-28T19:20:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.967+0000 7fd26e301740 20 client.15357 trim_cache size 0 max 16384 2024-08-28T19:20:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.985+0000 7fd26e301740 1 Processor -- start 2024-08-28T19:20:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.985+0000 7fd26e301740 1 -- start start 2024-08-28T19:20:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.985+0000 7fd26e301740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0600130a0 0x55a0603f9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.985+0000 7fd26e301740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06001b280 0x55a0603eb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.985+0000 7fd26e301740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06003bf60 0x55a0603ebc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.985+0000 7fd26e301740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a0603fda20 con 0x55a06003bf60 2024-08-28T19:20:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.985+0000 7fd26e301740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0603fe0f0 con 0x55a06001b280 2024-08-28T19:20:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.985+0000 7fd26e301740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a0603fe7c0 con 0x55a0600130a0 2024-08-28T19:20:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.985+0000 7fd266329640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06003bf60 0x55a0603ebc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.985+0000 7fd266329640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06003bf60 0x55a0603ebc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47760/0 (socket says 172.21.15.16:47760) 2024-08-28T19:20:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.985+0000 7fd266329640 1 -- 172.21.15.16:0/2275697072 learned_addr learned my addr 172.21.15.16:0/2275697072 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.986+0000 7fd265327640 1 --2- 172.21.15.16:0/2275697072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06001b280 0x55a0603eb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.986+0000 7fd265b28640 1 --2- 172.21.15.16:0/2275697072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0600130a0 0x55a0603f9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.986+0000 7fd266329640 1 -- 172.21.15.16:0/2275697072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0600130a0 msgr2=0x55a0603f9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.986+0000 7fd266329640 1 --2- 172.21.15.16:0/2275697072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0600130a0 0x55a0603f9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.986+0000 7fd266329640 1 -- 172.21.15.16:0/2275697072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06001b280 msgr2=0x55a0603eb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.986+0000 7fd266329640 1 --2- 172.21.15.16:0/2275697072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06001b280 0x55a0603eb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.986+0000 7fd266329640 1 -- 172.21.15.16:0/2275697072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0603ee130 con 0x55a06003bf60 2024-08-28T19:20:24.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.986+0000 7fd266329640 1 --2- 172.21.15.16:0/2275697072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06003bf60 0x55a0603ebc30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd25800f560 tx=0x7fd258016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c3666547e2e869 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:24.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.986+0000 7fd24b7fe640 1 -- 172.21.15.16:0/2275697072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25801e650 con 0x55a06003bf60 2024-08-28T19:20:24.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.986+0000 7fd24b7fe640 1 -- 172.21.15.16:0/2275697072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd25801eca0 con 0x55a06003bf60 2024-08-28T19:20:24.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.986+0000 7fd24b7fe640 1 -- 172.21.15.16:0/2275697072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd258027e50 con 0x55a06003bf60 2024-08-28T19:20:24.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.987+0000 7fd26e301740 1 -- 172.21.15.16:0/2275697072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06003bf60 msgr2=0x55a0603ebc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:24.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.987+0000 7fd26e301740 1 --2- 172.21.15.16:0/2275697072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06003bf60 0x55a0603ebc30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd25800f560 tx=0x7fd258016000 comp rx=0 tx=0).stop 2024-08-28T19:20:24.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.987+0000 7fd26e301740 1 -- 172.21.15.16:0/2275697072 shutdown_connections 2024-08-28T19:20:24.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.987+0000 7fd26e301740 1 --2- 172.21.15.16:0/2275697072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a06003bf60 0x55a0603ebc30 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.987+0000 7fd26e301740 1 --2- 172.21.15.16:0/2275697072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06001b280 0x55a0603eb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.987+0000 7fd26e301740 1 --2- 172.21.15.16:0/2275697072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0600130a0 0x55a0603f9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.987+0000 7fd26e301740 1 -- 172.21.15.16:0/2275697072 >> 172.21.15.16:0/2275697072 conn(0x55a060030350 msgr2=0x55a0603fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:24.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.987+0000 7fd26e301740 1 -- 172.21.15.16:0/2275697072 shutdown_connections 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.987+0000 7fd26e301740 1 -- 172.21.15.16:0/2275697072 wait complete. 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd26e301740 1 -- start start 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd26e301740 1 Processor -- start 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd26e301740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a060407e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd26e301740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06001b280 0x55a0604083a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd26e301740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06003bf60 0x55a060404050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd26e301740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a0603ff560 con 0x55a0600130a0 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd26e301740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0603fee90 con 0x55a06001b280 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd26e301740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a0603ffc30 con 0x55a06003bf60 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd26e301740 10 client.0 _ll_register_callbacks cb 0x7fd25c031d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd265b28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a060407e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd265b28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a060407e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47764/0 (socket says 172.21.15.16:47764) 2024-08-28T19:20:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.988+0000 7fd265b28640 1 -- 172.21.15.16:0/2233464087 learned_addr learned my addr 172.21.15.16:0/2233464087 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd265327640 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06001b280 0x55a0604083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd266329640 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06003bf60 0x55a060404050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd265b28640 1 -- 172.21.15.16:0/2233464087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06003bf60 msgr2=0x55a060404050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd265b28640 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06003bf60 0x55a060404050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd265b28640 1 -- 172.21.15.16:0/2233464087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06001b280 msgr2=0x55a0604083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd265b28640 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06001b280 0x55a0604083a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd265b28640 1 -- 172.21.15.16:0/2233464087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0604047e0 con 0x55a0600130a0 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd265b28640 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a060407e00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd25c03d9c0 tx=0x7fd25c031970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d39315c96cbf00c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd2497fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd2497fa640 1 -- 172.21.15.16:0/2233464087 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25c035050 con 0x55a0600130a0 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd2497fa640 1 -- 172.21.15.16:0/2233464087 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fd25c04b070 con 0x55a0600130a0 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd26e301740 1 -- 172.21.15.16:0/2233464087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd25801eca0 con 0x55a0600130a0 2024-08-28T19:20:24.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd2497fa640 1 -- 172.21.15.16:0/2233464087 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd25c05f050 con 0x55a0600130a0 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd26e301740 10 client.15363 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.989+0000 7fd26e301740 1 -- 172.21.15.16:0/2233464087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a06046efc0 con 0x55a0600130a0 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.990+0000 7fd2497fa640 1 -- 172.21.15.16:0/2233464087 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd25801eca0 con 0x55a0600130a0 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.990+0000 7fd2497fa640 1 -- 172.21.15.16:0/2233464087 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd25c005120 con 0x55a0600130a0 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.990+0000 7fd26e301740 10 client.15363 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.990+0000 7fd26e301740 10 client.15363 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.990+0000 7fd26e301740 20 client.15363 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.990+0000 7fd26e301740 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a05ffcfc90 0x55a060472780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.990+0000 7fd26e301740 4 client.15363 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.990+0000 7fd26e301740 1 -- 172.21.15.16:0/2233464087 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "status"}) -- 0x55a060355de0 con 0x55a05ffcfc90 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.990+0000 7fd265327640 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a05ffcfc90 0x55a060472780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd265327640 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a05ffcfc90 0x55a060472780 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1139894d6be5b2e server_cookie=c881cb0ee070bd7f in_seq=0 out_seq=0 2024-08-28T19:20:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd2497fa640 10 client.15363 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd2497fa640 1 -- 172.21.15.16:0/2233464087 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a060355de0 con 0x55a05ffcfc90 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd2497fa640 10 client.15363 handle_command_reply: tid=0 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 1 client.15363 shutdown 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 1 -- 172.21.15.16:0/2233464087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 msgr2=0x55a060407e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a060407e00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd25c03d9c0 tx=0x7fd25c031970 comp rx=0 tx=0).stop 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 1 -- 172.21.15.16:0/2233464087 shutdown_connections 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a05ffcfc90 0x55a060472780 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a06003bf60 0x55a060404050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a06001b280 0x55a0604083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd2497fa640 0 client.15363 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 1 --2- 172.21.15.16:0/2233464087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0600130a0 0x55a060407e00 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 1 -- 172.21.15.16:0/2233464087 >> 172.21.15.16:0/2233464087 conn(0x55a060030350 msgr2=0x55a0603ecc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 1 -- 172.21.15.16:0/2233464087 shutdown_connections 2024-08-28T19:20:24.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 1 -- 172.21.15.16:0/2233464087 wait complete. 2024-08-28T19:20:24.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.991+0000 7fd26e301740 20 client.15363 trim_cache size 0 max 16384 2024-08-28T19:20:24.998 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:20:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:20:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-08-28T19:20:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24665, 2024-08-28T19:20:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:20:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:20:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:20:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 111.516921642, 2024-08-28T19:20:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-08-28T19:20:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:20:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 82, 2024-08-28T19:20:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 121.73195905599999 2024-08-28T19:20:24.999 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:20:24.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 -- 172.21.15.16:0/2646136374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd24c0b4cc0 msgr2=0x7fd24c0b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:24.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd24c0b4cc0 0x7fd24c0b71d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd25800af30 tx=0x7fd258004a40 comp rx=0 tx=0).stop 2024-08-28T19:20:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 -- 172.21.15.16:0/2646136374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2601081c0 msgr2=0x7fd26019b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2601081c0 0x7fd26019b690 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd25000c8c0 tx=0x7fd25000cd80 comp rx=0 tx=0).stop 2024-08-28T19:20:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 -- 172.21.15.16:0/2646136374 shutdown_connections 2024-08-28T19:20:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd260108b10 0x7fd26019bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2601081c0 0x7fd26019b690 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd24c0b4cc0 0x7fd24c0b71d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 --2- 172.21.15.16:0/2646136374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260102880 0x7fd26019b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 -- 172.21.15.16:0/2646136374 >> 172.21.15.16:0/2646136374 conn(0x7fd260079510 msgr2=0x7fd260106300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 -- 172.21.15.16:0/2646136374 shutdown_connections 2024-08-28T19:20:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:24.992+0000 7fd267d8c640 1 -- 172.21.15.16:0/2646136374 wait complete. 2024-08-28T19:20:25.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:25 smithi016 ceph-mon[18556]: pgmap v320: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:25.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:25 smithi028 ceph-mon[21769]: pgmap v320: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:25.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:25 smithi115 ceph-mon[22184]: pgmap v320: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:25.851 INFO:tasks.cephadm:Setup complete, yielding 2024-08-28T19:20:25.851 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-08-28T19:20:25.861 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-08-28T19:20:25.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- bash -c 'ceph orch status' 2024-08-28T19:20:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:26.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.632+0000 7f4608515640 1 -- 172.21.15.16:0/4213574765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4600108140 msgr2=0x7f4600108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:26.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.632+0000 7f4608515640 1 --2- 172.21.15.16:0/4213574765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4600108140 0x7f4600108550 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009900 tx=0x7f45f002a410 comp rx=0 tx=0).stop 2024-08-28T19:20:26.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.632+0000 7f4608515640 1 -- 172.21.15.16:0/4213574765 shutdown_connections 2024-08-28T19:20:26.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.632+0000 7f4608515640 1 --2- 172.21.15.16:0/4213574765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4600108a90 0x7f460010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:26.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.632+0000 7f4608515640 1 --2- 172.21.15.16:0/4213574765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4600108140 0x7f4600108550 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.632+0000 7f4608515640 1 --2- 172.21.15.16:0/4213574765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4600102800 0x7f4600102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.632+0000 7f4608515640 1 -- 172.21.15.16:0/4213574765 >> 172.21.15.16:0/4213574765 conn(0x7f46000794d0 msgr2=0x7f46000798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.632+0000 7f4608515640 1 -- 172.21.15.16:0/4213574765 shutdown_connections 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.632+0000 7f4608515640 1 -- 172.21.15.16:0/4213574765 wait complete. 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4608515640 1 Processor -- start 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4608515640 1 -- start start 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4608515640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4600102800 0x7f460019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4608515640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4600108140 0x7f460019a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4608515640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4600108a90 0x7f460019fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4608515640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46001126c0 con 0x7f4600108140 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4608515640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4600112540 con 0x7f4600108a90 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4608515640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4600112840 con 0x7f4600102800 2024-08-28T19:20:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4605ab0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4600108140 0x7f460019a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4605ab0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4600108140 0x7f460019a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47776/0 (socket says 172.21.15.16:47776) 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4605ab0640 1 -- 172.21.15.16:0/3364262019 learned_addr learned my addr 172.21.15.16:0/3364262019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f46062b1640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4600102800 0x7f460019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.633+0000 7f4606ab2640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4600108a90 0x7f460019fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f4605ab0640 1 -- 172.21.15.16:0/3364262019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4600102800 msgr2=0x7f460019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f4605ab0640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4600102800 0x7f460019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f4605ab0640 1 -- 172.21.15.16:0/3364262019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4600108a90 msgr2=0x7f460019fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f4605ab0640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4600108a90 0x7f460019fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f4605ab0640 1 -- 172.21.15.16:0/3364262019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46001a01f0 con 0x7f4600108140 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f4606ab2640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4600108a90 0x7f460019fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f4605ab0640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4600108140 0x7f460019a820 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f45f0032fd0 tx=0x7f45f0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f45f77fe640 1 -- 172.21.15.16:0/3364262019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f003f030 con 0x7f4600108140 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f4608515640 1 -- 172.21.15.16:0/3364262019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46001a0480 con 0x7f4600108140 2024-08-28T19:20:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f45f77fe640 1 -- 172.21.15.16:0/3364262019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45f0004490 con 0x7f4600108140 2024-08-28T19:20:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f45f77fe640 1 -- 172.21.15.16:0/3364262019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0035070 con 0x7f4600108140 2024-08-28T19:20:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.634+0000 7f4608515640 1 -- 172.21.15.16:0/3364262019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46001a6c40 con 0x7f4600108140 2024-08-28T19:20:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.635+0000 7f4608515640 1 -- 172.21.15.16:0/3364262019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4600103a80 con 0x7f4600108140 2024-08-28T19:20:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.637+0000 7f45f77fe640 1 -- 172.21.15.16:0/3364262019 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f45f0042030 con 0x7f4600108140 2024-08-28T19:20:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.638+0000 7f45f77fe640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45ec0b4c10 0x7f45ec0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.638+0000 7f46062b1640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45ec0b4c10 0x7f45ec0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.638+0000 7f45f77fe640 1 -- 172.21.15.16:0/3364262019 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f45f0106100 con 0x7f4600108140 2024-08-28T19:20:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.638+0000 7f46062b1640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45ec0b4c10 0x7f45ec0b7120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0058c0 tx=0x7f45fc00a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:26.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.641+0000 7f45f77fe640 1 -- 172.21.15.16:0/3364262019 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f45f00ccc40 con 0x7f4600108140 2024-08-28T19:20:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.844+0000 7f4608515640 1 -- 172.21.15.16:0/3364262019 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f460006abe0 con 0x7f45ec0b4c10 2024-08-28T19:20:26.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.845+0000 7f45f77fe640 1 -- 172.21.15.16:0/3364262019 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f460006abe0 con 0x7f45ec0b4c10 2024-08-28T19:20:26.846 INFO:teuthology.orchestra.run.smithi016.stdout:Backend: cephadm 2024-08-28T19:20:26.846 INFO:teuthology.orchestra.run.smithi016.stdout:Available: Yes 2024-08-28T19:20:26.846 INFO:teuthology.orchestra.run.smithi016.stdout:Paused: No 2024-08-28T19:20:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 -- 172.21.15.16:0/3364262019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45ec0b4c10 msgr2=0x7f45ec0b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45ec0b4c10 0x7f45ec0b7120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0058c0 tx=0x7f45fc00a2d0 comp rx=0 tx=0).stop 2024-08-28T19:20:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 -- 172.21.15.16:0/3364262019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4600108140 msgr2=0x7f460019a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4600108140 0x7f460019a820 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f45f0032fd0 tx=0x7f45f0004270 comp rx=0 tx=0).stop 2024-08-28T19:20:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 -- 172.21.15.16:0/3364262019 shutdown_connections 2024-08-28T19:20:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45ec0b4c10 0x7f45ec0b7120 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4600108a90 0x7f460019fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4600108140 0x7f460019a820 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 --2- 172.21.15.16:0/3364262019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4600102800 0x7f460019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 -- 172.21.15.16:0/3364262019 >> 172.21.15.16:0/3364262019 conn(0x7f46000794d0 msgr2=0x7f46000ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 -- 172.21.15.16:0/3364262019 shutdown_connections 2024-08-28T19:20:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:26.847+0000 7f4608515640 1 -- 172.21.15.16:0/3364262019 wait complete. 2024-08-28T19:20:27.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:27 smithi016 ceph-mon[18556]: pgmap v321: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:20:27.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:27 smithi115 ceph-mon[22184]: pgmap v321: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:20:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:27 smithi028 ceph-mon[21769]: pgmap v321: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:20:27.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- bash -c 'ceph orch ps' 2024-08-28T19:20:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:28.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:28 smithi016 ceph-mon[18556]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:28.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:28 smithi115 ceph-mon[22184]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:28.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.435+0000 7f2d57061640 1 -- 172.21.15.16:0/1879019167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d50102880 msgr2=0x7f2d50102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:28.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.435+0000 7f2d57061640 1 --2- 172.21.15.16:0/1879019167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d50102880 0x7f2d50102c90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009900 tx=0x7f2d4402a410 comp rx=0 tx=0).stop 2024-08-28T19:20:28.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.436+0000 7f2d57061640 1 -- 172.21.15.16:0/1879019167 shutdown_connections 2024-08-28T19:20:28.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.436+0000 7f2d57061640 1 --2- 172.21.15.16:0/1879019167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d50108ac0 0x7f2d5010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:28.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.436+0000 7f2d57061640 1 --2- 172.21.15.16:0/1879019167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d50108170 0x7f2d50108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:28.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.436+0000 7f2d57061640 1 --2- 172.21.15.16:0/1879019167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d50102880 0x7f2d50102c90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:28.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.436+0000 7f2d57061640 1 -- 172.21.15.16:0/1879019167 >> 172.21.15.16:0/1879019167 conn(0x7f2d50079500 msgr2=0x7f2d50079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.436+0000 7f2d57061640 1 -- 172.21.15.16:0/1879019167 shutdown_connections 2024-08-28T19:20:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.436+0000 7f2d57061640 1 -- 172.21.15.16:0/1879019167 wait complete. 2024-08-28T19:20:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.436+0000 7f2d57061640 1 Processor -- start 2024-08-28T19:20:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.436+0000 7f2d57061640 1 -- start start 2024-08-28T19:20:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d57061640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d50102880 0x7f2d5019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d57061640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d50108170 0x7f2d5019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d57061640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d50108ac0 0x7f2d5019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d57061640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2d501127a0 con 0x7f2d50108ac0 2024-08-28T19:20:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d57061640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d50112620 con 0x7f2d50108170 2024-08-28T19:20:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d57061640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2d50112920 con 0x7f2d50102880 2024-08-28T19:20:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d555fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d50108ac0 0x7f2d5019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d54dfd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d50102880 0x7f2d5019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d54dfd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d50102880 0x7f2d5019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:58354/0 (socket says 172.21.15.16:58354) 2024-08-28T19:20:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d54dfd640 1 -- 172.21.15.16:0/1675078913 learned_addr learned my addr 172.21.15.16:0/1675078913 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d4ffff640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d50108170 0x7f2d5019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d555fe640 1 -- 172.21.15.16:0/1675078913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d50102880 msgr2=0x7f2d5019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d555fe640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d50102880 0x7f2d5019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d555fe640 1 -- 172.21.15.16:0/1675078913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d50108170 msgr2=0x7f2d5019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d555fe640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d50108170 0x7f2d5019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d555fe640 1 -- 172.21.15.16:0/1675078913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d501a01d0 con 0x7f2d50108ac0 2024-08-28T19:20:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.437+0000 7f2d555fe640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d50108ac0 0x7f2d5019faa0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800be20 tx=0x7f2d4800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.438+0000 7f2d4dffb640 1 -- 172.21.15.16:0/1675078913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48019070 con 0x7f2d50108ac0 2024-08-28T19:20:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.438+0000 7f2d57061640 1 -- 172.21.15.16:0/1675078913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d501a04c0 con 0x7f2d50108ac0 2024-08-28T19:20:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.438+0000 7f2d4dffb640 1 -- 172.21.15.16:0/1675078913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d480148f0 con 0x7f2d50108ac0 2024-08-28T19:20:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.438+0000 7f2d4dffb640 1 -- 172.21.15.16:0/1675078913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48012840 con 0x7f2d50108ac0 2024-08-28T19:20:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.438+0000 7f2d57061640 1 -- 172.21.15.16:0/1675078913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d501a6ca0 con 0x7f2d50108ac0 2024-08-28T19:20:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.439+0000 7f2d57061640 1 -- 172.21.15.16:0/1675078913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d50103ab0 con 0x7f2d50108ac0 2024-08-28T19:20:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.440+0000 7f2d4dffb640 1 -- 172.21.15.16:0/1675078913 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f2d4801c070 con 0x7f2d50108ac0 2024-08-28T19:20:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.441+0000 7f2d4dffb640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2d400b4c10 0x7f2d400b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.441+0000 7f2d54dfd640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2d400b4c10 0x7f2d400b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.441+0000 7f2d4dffb640 1 -- 172.21.15.16:0/1675078913 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f2d480e4ea0 con 0x7f2d50108ac0 2024-08-28T19:20:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.442+0000 7f2d54dfd640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2d400b4c10 0x7f2d400b7120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2d44031440 tx=0x7f2d440058d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.446+0000 7f2d4dffb640 1 -- 172.21.15.16:0/1675078913 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f2d480ab330 con 0x7f2d50108ac0 2024-08-28T19:20:28.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:28 smithi028 ceph-mon[21769]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:28.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.643+0000 7f2d57061640 1 -- 172.21.15.16:0/1675078913 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f2d5006b8e0 con 0x7f2d400b4c10 2024-08-28T19:20:28.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.655+0000 7f2d4dffb640 1 -- 172.21.15.16:0/1675078913 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f2d5006b8e0 con 0x7f2d400b4c10 2024-08-28T19:20:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-28T19:20:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:mds.a smithi016 running (2m) 57s ago 2m 83.5M - 19.3.0-4560-gd19e6118 79f4cde5e369 d0926d7b4bad 2024-08-28T19:20:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:mds.b smithi028 running (2m) 59s ago 2m 96.2M - 19.3.0-4560-gd19e6118 79f4cde5e369 3427fd35e589 2024-08-28T19:20:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:mds.c smithi115 running (2m) 59s ago 2m 86.9M - 19.3.0-4560-gd19e6118 79f4cde5e369 81c2473c43d4 2024-08-28T19:20:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:mds.d smithi016 running (2m) 57s ago 2m 81.8M - 19.3.0-4560-gd19e6118 79f4cde5e369 50e1601715d4 2024-08-28T19:20:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:mds.e smithi028 running (2m) 59s ago 2m 86.7M - 19.3.0-4560-gd19e6118 79f4cde5e369 145f4b253106 2024-08-28T19:20:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:mds.f smithi115 running (2m) 59s ago 2m 83.9M - 19.3.0-4560-gd19e6118 79f4cde5e369 c6e9556c803c 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mds.g smithi016 running (2m) 57s ago 2m 82.6M - 19.3.0-4560-gd19e6118 79f4cde5e369 100d766a4ef2 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mds.h smithi028 running (2m) 59s ago 2m 84.0M - 19.3.0-4560-gd19e6118 79f4cde5e369 777be3f361a3 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mds.i smithi115 running (2m) 59s ago 2m 87.3M - 19.3.0-4560-gd19e6118 79f4cde5e369 65c78c0373ff 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mds.j smithi016 running (2m) 57s ago 2m 82.1M - 19.3.0-4560-gd19e6118 79f4cde5e369 2fdb9fc7cc45 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mds.k smithi028 running (2m) 59s ago 2m 85.9M - 19.3.0-4560-gd19e6118 79f4cde5e369 6f3c59426cde 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mds.l smithi115 running (2m) 59s ago 2m 86.0M - 19.3.0-4560-gd19e6118 79f4cde5e369 b1f104dd3373 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi016 *:9283,8765 running (10m) 57s ago 10m 587M - 19.3.0-4560-gd19e6118 79f4cde5e369 63620dafa962 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi028 *:8443,8765 running (8m) 59s ago 8m 522M - 19.3.0-4560-gd19e6118 79f4cde5e369 fed856f47993 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.z smithi115 *:8443,8765 running (8m) 59s ago 8m 523M - 19.3.0-4560-gd19e6118 79f4cde5e369 b6a0a772a01c 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (10m) 57s ago 10m 120M 2048M 19.3.0-4560-gd19e6118 79f4cde5e369 1408e30d070d 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi028 running (8m) 59s ago 8m 109M 2048M 19.3.0-4560-gd19e6118 79f4cde5e369 684e32f5e71d 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi115 running (8m) 59s ago 8m 112M 2048M 19.3.0-4560-gd19e6118 79f4cde5e369 703bcb8f9d93 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (7m) 57s ago 7m 121M 4273M 19.3.0-4560-gd19e6118 79f4cde5e369 e03638c66c68 2024-08-28T19:20:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi028 running (7m) 59s ago 7m 122M 4273M 19.3.0-4560-gd19e6118 79f4cde5e369 74e0fe42a0dd 2024-08-28T19:20:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi115 running (6m) 59s ago 6m 124M 4272M 19.3.0-4560-gd19e6118 79f4cde5e369 782f2e07fbd8 2024-08-28T19:20:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (6m) 57s ago 6m 120M 4273M 19.3.0-4560-gd19e6118 79f4cde5e369 9c8828eb6d76 2024-08-28T19:20:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi028 running (5m) 59s ago 5m 120M 4273M 19.3.0-4560-gd19e6118 79f4cde5e369 590ecde2c25b 2024-08-28T19:20:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi115 running (5m) 59s ago 5m 113M 4272M 19.3.0-4560-gd19e6118 79f4cde5e369 89f59934a212 2024-08-28T19:20:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi016 running (5m) 57s ago 5m 109M 4273M 19.3.0-4560-gd19e6118 79f4cde5e369 d467e691f6bd 2024-08-28T19:20:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi028 running (4m) 59s ago 4m 119M 4273M 19.3.0-4560-gd19e6118 79f4cde5e369 691b97276c76 2024-08-28T19:20:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 smithi115 running (4m) 59s ago 4m 119M 4272M 19.3.0-4560-gd19e6118 79f4cde5e369 968bb9423fca 2024-08-28T19:20:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 smithi016 running (3m) 57s ago 3m 118M 4273M 19.3.0-4560-gd19e6118 79f4cde5e369 eb84bbff5179 2024-08-28T19:20:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 smithi028 running (3m) 59s ago 3m 112M 4273M 19.3.0-4560-gd19e6118 79f4cde5e369 07b52226f45d 2024-08-28T19:20:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 smithi115 running (2m) 59s ago 2m 117M 4272M 19.3.0-4560-gd19e6118 79f4cde5e369 9965a6b9afe9 2024-08-28T19:20:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 -- 172.21.15.16:0/1675078913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2d400b4c10 msgr2=0x7f2d400b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2d400b4c10 0x7f2d400b7120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2d44031440 tx=0x7f2d440058d0 comp rx=0 tx=0).stop 2024-08-28T19:20:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 -- 172.21.15.16:0/1675078913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d50108ac0 msgr2=0x7f2d5019faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d50108ac0 0x7f2d5019faa0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800be20 tx=0x7f2d4800bf10 comp rx=0 tx=0).stop 2024-08-28T19:20:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 -- 172.21.15.16:0/1675078913 shutdown_connections 2024-08-28T19:20:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d50108ac0 0x7f2d5019faa0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2d400b4c10 0x7f2d400b7120 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d50108170 0x7f2d5019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 --2- 172.21.15.16:0/1675078913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d50102880 0x7f2d5019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 -- 172.21.15.16:0/1675078913 >> 172.21.15.16:0/1675078913 conn(0x7f2d50079500 msgr2=0x7f2d501018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 -- 172.21.15.16:0/1675078913 shutdown_connections 2024-08-28T19:20:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:28.658+0000 7f2d57061640 1 -- 172.21.15.16:0/1675078913 wait complete. 2024-08-28T19:20:29.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:29 smithi115 ceph-mon[22184]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:20:29.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:29 smithi016 ceph-mon[18556]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:20:29.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- bash -c 'ceph orch ls' 2024-08-28T19:20:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:29 smithi028 ceph-mon[21769]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:20:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.201+0000 7f049e4ce640 1 -- 172.21.15.16:0/359357005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498109b90 msgr2=0x7f04981103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.201+0000 7f049e4ce640 1 --2- 172.21.15.16:0/359357005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498109b90 0x7f04981103c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0490009ca0 tx=0x7f049002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.201+0000 7f049e4ce640 1 -- 172.21.15.16:0/359357005 shutdown_connections 2024-08-28T19:20:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.201+0000 7f049e4ce640 1 --2- 172.21.15.16:0/359357005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498109b90 0x7f04981103c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.201+0000 7f049e4ce640 1 --2- 172.21.15.16:0/359357005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0498109240 0x7f0498109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.201+0000 7f049e4ce640 1 --2- 172.21.15.16:0/359357005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0498103900 0x7f0498103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.201+0000 7f049e4ce640 1 -- 172.21.15.16:0/359357005 >> 172.21.15.16:0/359357005 conn(0x7f049807b5f0 msgr2=0x7f049807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.201+0000 7f049e4ce640 1 -- 172.21.15.16:0/359357005 shutdown_connections 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.201+0000 7f049e4ce640 1 -- 172.21.15.16:0/359357005 wait complete. 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.201+0000 7f049e4ce640 1 Processor -- start 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.201+0000 7f049e4ce640 1 -- start start 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049e4ce640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0498103900 0x7f049819c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049e4ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0498109240 0x7f049819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049e4ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498109b90 0x7f04981a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049e4ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0498113850 con 0x7f0498109b90 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049e4ce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04981136d0 con 0x7f0498109240 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049e4ce640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f04981139d0 con 0x7f0498103900 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049ca6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498109b90 0x7f04981a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049ca6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498109b90 0x7f04981a1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47818/0 (socket says 172.21.15.16:47818) 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049ca6b640 1 -- 172.21.15.16:0/2126038766 learned_addr learned my addr 172.21.15.16:0/2126038766 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f0497fff640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0498103900 0x7f049819c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f04977fe640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0498109240 0x7f049819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049ca6b640 1 -- 172.21.15.16:0/2126038766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0498103900 msgr2=0x7f049819c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049ca6b640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0498103900 0x7f049819c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049ca6b640 1 -- 172.21.15.16:0/2126038766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0498109240 msgr2=0x7f049819c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049ca6b640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0498109240 0x7f049819c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f049ca6b640 1 -- 172.21.15.16:0/2126038766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04981a2390 con 0x7f0498109b90 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.202+0000 7f04977fe640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0498109240 0x7f049819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.203+0000 7f049ca6b640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498109b90 0x7f04981a1c60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f04900098d0 tx=0x7f04900317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.203+0000 7f04957fa640 1 -- 172.21.15.16:0/2126038766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490003b70 con 0x7f0498109b90 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.203+0000 7f049e4ce640 1 -- 172.21.15.16:0/2126038766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04981a2620 con 0x7f0498109b90 2024-08-28T19:20:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.203+0000 7f04957fa640 1 -- 172.21.15.16:0/2126038766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0490003d10 con 0x7f0498109b90 2024-08-28T19:20:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.203+0000 7f04957fa640 1 -- 172.21.15.16:0/2126038766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049002fa20 con 0x7f0498109b90 2024-08-28T19:20:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.203+0000 7f049e4ce640 1 -- 172.21.15.16:0/2126038766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04981a8de0 con 0x7f0498109b90 2024-08-28T19:20:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.204+0000 7f049e4ce640 1 -- 172.21.15.16:0/2126038766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0498103d10 con 0x7f0498109b90 2024-08-28T19:20:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.206+0000 7f04957fa640 1 -- 172.21.15.16:0/2126038766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f049002fbe0 con 0x7f0498109b90 2024-08-28T19:20:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.206+0000 7f04957fa640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f04800b4c10 0x7f04800b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.206+0000 7f0497fff640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f04800b4c10 0x7f04800b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.206+0000 7f04957fa640 1 -- 172.21.15.16:0/2126038766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0490106c40 con 0x7f0498109b90 2024-08-28T19:20:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.207+0000 7f0497fff640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f04800b4c10 0x7f04800b7120 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f048800bbe0 tx=0x7f048800b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.210+0000 7f04957fa640 1 -- 172.21.15.16:0/2126038766 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f04900cd780 con 0x7f0498109b90 2024-08-28T19:20:30.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:30 smithi115 ceph-mon[22184]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.403+0000 7f049e4ce640 1 -- 172.21.15.16:0/2126038766 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f0498107760 con 0x7f04800b4c10 2024-08-28T19:20:30.405 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:30 smithi016 ceph-mon[18556]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.409+0000 7f04957fa640 1 -- 172.21.15.16:0/2126038766 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f0498107760 con 0x7f04800b4c10 2024-08-28T19:20:30.411 INFO:teuthology.orchestra.run.smithi016.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-08-28T19:20:30.411 INFO:teuthology.orchestra.run.smithi016.stdout:mds.all 12/12 60s ago 2m smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi028=b;smithi028=e;smithi028=h;smithi028=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l;count:12 2024-08-28T19:20:30.411 INFO:teuthology.orchestra.run.smithi016.stdout:mgr 3/3 60s ago 8m smithi016=x;smithi028=y;smithi115=z;count:3 2024-08-28T19:20:30.411 INFO:teuthology.orchestra.run.smithi016.stdout:mon 3/3 60s ago 8m smithi016:172.21.15.16=a;smithi028:172.21.15.28=b;smithi115:172.21.15.115=c;count:3 2024-08-28T19:20:30.411 INFO:teuthology.orchestra.run.smithi016.stdout:osd 12 60s ago - 2024-08-28T19:20:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 -- 172.21.15.16:0/2126038766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f04800b4c10 msgr2=0x7f04800b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f04800b4c10 0x7f04800b7120 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f048800bbe0 tx=0x7f048800b390 comp rx=0 tx=0).stop 2024-08-28T19:20:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 -- 172.21.15.16:0/2126038766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498109b90 msgr2=0x7f04981a1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498109b90 0x7f04981a1c60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f04900098d0 tx=0x7f04900317b0 comp rx=0 tx=0).stop 2024-08-28T19:20:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 -- 172.21.15.16:0/2126038766 shutdown_connections 2024-08-28T19:20:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498109b90 0x7f04981a1c60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f04800b4c10 0x7f04800b7120 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0498109240 0x7f049819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 --2- 172.21.15.16:0/2126038766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0498103900 0x7f049819c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 -- 172.21.15.16:0/2126038766 >> 172.21.15.16:0/2126038766 conn(0x7f049807b5f0 msgr2=0x7f0498107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 -- 172.21.15.16:0/2126038766 shutdown_connections 2024-08-28T19:20:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:30.412+0000 7f049e4ce640 1 -- 172.21.15.16:0/2126038766 wait complete. 2024-08-28T19:20:30.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:30 smithi028 ceph-mon[21769]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:31.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- bash -c 'ceph orch host ls' 2024-08-28T19:20:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:31 smithi115 ceph-mon[22184]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-08-28T19:20:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:31.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:31 smithi016 ceph-mon[18556]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-08-28T19:20:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:31 smithi028 ceph-mon[21769]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-08-28T19:20:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.000+0000 7f7a3c561640 1 -- 172.21.15.16:0/3773702712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a34104600 msgr2=0x7f7a34104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.000+0000 7f7a3c561640 1 --2- 172.21.15.16:0/3773702712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a34104600 0x7f7a34104a10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009900 tx=0x7f7a3002a410 comp rx=0 tx=0).stop 2024-08-28T19:20:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.000+0000 7f7a3c561640 1 -- 172.21.15.16:0/3773702712 shutdown_connections 2024-08-28T19:20:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.000+0000 7f7a3c561640 1 --2- 172.21.15.16:0/3773702712 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a3410dbb0 0x7f7a34110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.000+0000 7f7a3c561640 1 --2- 172.21.15.16:0/3773702712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a34105030 0x7f7a3410d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.000+0000 7f7a3c561640 1 --2- 172.21.15.16:0/3773702712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a34104600 0x7f7a34104a10 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.000+0000 7f7a3c561640 1 -- 172.21.15.16:0/3773702712 >> 172.21.15.16:0/3773702712 conn(0x7f7a340794d0 msgr2=0x7f7a340798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.000+0000 7f7a3c561640 1 -- 172.21.15.16:0/3773702712 shutdown_connections 2024-08-28T19:20:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.000+0000 7f7a3c561640 1 -- 172.21.15.16:0/3773702712 wait complete. 2024-08-28T19:20:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.001+0000 7f7a3c561640 1 Processor -- start 2024-08-28T19:20:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.001+0000 7f7a3c561640 1 -- start start 2024-08-28T19:20:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.001+0000 7f7a3c561640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a34104600 0x7f7a34102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.001+0000 7f7a3c561640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a34105030 0x7f7a34102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.001+0000 7f7a3c561640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a3410dbb0 0x7f7a34102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.001+0000 7f7a3c561640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a34113820 con 0x7f7a34105030 2024-08-28T19:20:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.001+0000 7f7a3c561640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a341136a0 con 0x7f7a3410dbb0 2024-08-28T19:20:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.001+0000 7f7a3c561640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7a341139a0 con 0x7f7a34104600 2024-08-28T19:20:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.001+0000 7f7a39afc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a34105030 0x7f7a34102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.001+0000 7f7a39afc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a34105030 0x7f7a34102a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47844/0 (socket says 172.21.15.16:47844) 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.001+0000 7f7a39afc640 1 -- 172.21.15.16:0/3225019690 learned_addr learned my addr 172.21.15.16:0/3225019690 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a3a2fd640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a34104600 0x7f7a34102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a3aafe640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a3410dbb0 0x7f7a34102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a39afc640 1 -- 172.21.15.16:0/3225019690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a34104600 msgr2=0x7f7a34102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a39afc640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a34104600 0x7f7a34102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a39afc640 1 -- 172.21.15.16:0/3225019690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a3410dbb0 msgr2=0x7f7a34102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a39afc640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a3410dbb0 0x7f7a34102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a39afc640 1 -- 172.21.15.16:0/3225019690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a341a84b0 con 0x7f7a34105030 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a39afc640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a34105030 0x7f7a34102a50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400d740 tx=0x7f7a2400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a2b7fe640 1 -- 172.21.15.16:0/3225019690 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a24017070 con 0x7f7a34105030 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a3c561640 1 -- 172.21.15.16:0/3225019690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a341a87a0 con 0x7f7a34105030 2024-08-28T19:20:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a2b7fe640 1 -- 172.21.15.16:0/3225019690 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a2400de50 con 0x7f7a34105030 2024-08-28T19:20:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a2b7fe640 1 -- 172.21.15.16:0/3225019690 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2400f620 con 0x7f7a34105030 2024-08-28T19:20:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.002+0000 7f7a3c561640 1 -- 172.21.15.16:0/3225019690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a341a8ce0 con 0x7f7a34105030 2024-08-28T19:20:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.003+0000 7f7a3c561640 1 -- 172.21.15.16:0/3225019690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a34104a10 con 0x7f7a34105030 2024-08-28T19:20:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.005+0000 7f7a2b7fe640 1 -- 172.21.15.16:0/3225019690 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f7a2401a070 con 0x7f7a34105030 2024-08-28T19:20:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.006+0000 7f7a2b7fe640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a200b4c10 0x7f7a200b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.006+0000 7f7a3a2fd640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a200b4c10 0x7f7a200b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.006+0000 7f7a2b7fe640 1 -- 172.21.15.16:0/3225019690 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7a240e5140 con 0x7f7a34105030 2024-08-28T19:20:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.006+0000 7f7a3a2fd640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a200b4c10 0x7f7a200b7120 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7a30000c00 tx=0x7f7a30031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.010+0000 7f7a2b7fe640 1 -- 172.21.15.16:0/3225019690 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f7a240ab630 con 0x7f7a34105030 2024-08-28T19:20:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.221+0000 7f7a3c561640 1 -- 172.21.15.16:0/3225019690 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f7a340635a0 con 0x7f7a200b4c10 2024-08-28T19:20:32.223 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:32 smithi016 ceph-mon[18556]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:32.223 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:32 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:20:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.223+0000 7f7a2b7fe640 1 -- 172.21.15.16:0/3225019690 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f7a340635a0 con 0x7f7a200b4c10 2024-08-28T19:20:32.224 INFO:teuthology.orchestra.run.smithi016.stdout:HOST ADDR LABELS STATUS 2024-08-28T19:20:32.224 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 172.21.15.16 2024-08-28T19:20:32.224 INFO:teuthology.orchestra.run.smithi016.stdout:smithi028 172.21.15.28 2024-08-28T19:20:32.224 INFO:teuthology.orchestra.run.smithi016.stdout:smithi115 172.21.15.115 2024-08-28T19:20:32.224 INFO:teuthology.orchestra.run.smithi016.stdout:3 hosts in cluster 2024-08-28T19:20:32.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 -- 172.21.15.16:0/3225019690 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a200b4c10 msgr2=0x7f7a200b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:32.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a200b4c10 0x7f7a200b7120 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7a30000c00 tx=0x7f7a30031400 comp rx=0 tx=0).stop 2024-08-28T19:20:32.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 -- 172.21.15.16:0/3225019690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a34105030 msgr2=0x7f7a34102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a34105030 0x7f7a34102a50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400d740 tx=0x7f7a2400dc00 comp rx=0 tx=0).stop 2024-08-28T19:20:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 -- 172.21.15.16:0/3225019690 shutdown_connections 2024-08-28T19:20:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a200b4c10 0x7f7a200b7120 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a3410dbb0 0x7f7a34102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a34105030 0x7f7a34102a50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 --2- 172.21.15.16:0/3225019690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a34104600 0x7f7a34102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 -- 172.21.15.16:0/3225019690 >> 172.21.15.16:0/3225019690 conn(0x7f7a340794d0 msgr2=0x7f7a34100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 -- 172.21.15.16:0/3225019690 shutdown_connections 2024-08-28T19:20:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:32.225+0000 7f7a3c561640 1 -- 172.21.15.16:0/3225019690 wait complete. 2024-08-28T19:20:32.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:32 smithi115 ceph-mon[22184]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:32.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:32 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:20:32.368 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:32 smithi028 ceph-mon[21769]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:32.368 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:32 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:20:33.112 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- bash -c 'ceph orch device ls' 2024-08-28T19:20:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:33 smithi115 ceph-mon[22184]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:20:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:33 smithi115 ceph-mon[22184]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:33 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:33 smithi016 ceph-mon[18556]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:20:33.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:33 smithi016 ceph-mon[18556]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:33.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:33 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:33 smithi028 ceph-mon[21769]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:20:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:33 smithi028 ceph-mon[21769]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:33 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.124+0000 7f632c7e5640 1 -- 172.21.15.16:0/307941765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6324108150 msgr2=0x7f6324108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.124+0000 7f632c7e5640 1 --2- 172.21.15.16:0/307941765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6324108150 0x7f6324108560 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6314009900 tx=0x7f631402a410 comp rx=0 tx=0).stop 2024-08-28T19:20:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.124+0000 7f632c7e5640 1 -- 172.21.15.16:0/307941765 shutdown_connections 2024-08-28T19:20:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.124+0000 7f632c7e5640 1 --2- 172.21.15.16:0/307941765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6324108aa0 0x7f632410f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.124+0000 7f632c7e5640 1 --2- 172.21.15.16:0/307941765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6324108150 0x7f6324108560 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.124+0000 7f632c7e5640 1 --2- 172.21.15.16:0/307941765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6324102810 0x7f6324102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.124+0000 7f632c7e5640 1 -- 172.21.15.16:0/307941765 >> 172.21.15.16:0/307941765 conn(0x7f63240794b0 msgr2=0x7f63240798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.124+0000 7f632c7e5640 1 -- 172.21.15.16:0/307941765 shutdown_connections 2024-08-28T19:20:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.124+0000 7f632c7e5640 1 -- 172.21.15.16:0/307941765 wait complete. 2024-08-28T19:20:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.124+0000 7f632c7e5640 1 Processor -- start 2024-08-28T19:20:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.124+0000 7f632c7e5640 1 -- start start 2024-08-28T19:20:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f632c7e5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6324102810 0x7f632410f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f632c7e5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6324108150 0x7f632410d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f632c7e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6324108aa0 0x7f632410d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f632c7e5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6324110c50 con 0x7f6324108aa0 2024-08-28T19:20:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f632c7e5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6324110ad0 con 0x7f6324102810 2024-08-28T19:20:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f632c7e5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6324110dd0 con 0x7f6324108150 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f632ad82640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6324108aa0 0x7f632410d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f6329d80640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6324108150 0x7f632410d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f632ad82640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6324108aa0 0x7f632410d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47872/0 (socket says 172.21.15.16:47872) 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f632ad82640 1 -- 172.21.15.16:0/3827042638 learned_addr learned my addr 172.21.15.16:0/3827042638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f632a581640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6324102810 0x7f632410f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f6329d80640 1 -- 172.21.15.16:0/3827042638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6324102810 msgr2=0x7f632410f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f6329d80640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6324102810 0x7f632410f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f6329d80640 1 -- 172.21.15.16:0/3827042638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6324108aa0 msgr2=0x7f632410d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f6329d80640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6324108aa0 0x7f632410d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f6329d80640 1 -- 172.21.15.16:0/3827042638 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632410e0b0 con 0x7f6324108150 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.125+0000 7f632ad82640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6324108aa0 0x7f632410d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.126+0000 7f6329d80640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6324108150 0x7f632410d350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f631402a930 tx=0x7f6314032ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.126+0000 7f631b7fe640 1 -- 172.21.15.16:0/3827042638 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631403f070 con 0x7f6324108150 2024-08-28T19:20:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.126+0000 7f631b7fe640 1 -- 172.21.15.16:0/3827042638 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6314003ca0 con 0x7f6324108150 2024-08-28T19:20:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.126+0000 7f632c7e5640 1 -- 172.21.15.16:0/3827042638 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63241accb0 con 0x7f6324108150 2024-08-28T19:20:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.126+0000 7f631b7fe640 1 -- 172.21.15.16:0/3827042638 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6314035070 con 0x7f6324108150 2024-08-28T19:20:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.126+0000 7f632c7e5640 1 -- 172.21.15.16:0/3827042638 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63241ad1b0 con 0x7f6324108150 2024-08-28T19:20:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.127+0000 7f632c7e5640 1 -- 172.21.15.16:0/3827042638 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6324103a90 con 0x7f6324108150 2024-08-28T19:20:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.128+0000 7f631b7fe640 1 -- 172.21.15.16:0/3827042638 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f631403b460 con 0x7f6324108150 2024-08-28T19:20:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.129+0000 7f631b7fe640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63100b4c80 0x7f63100b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.129+0000 7f632a581640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63100b4c80 0x7f63100b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.129+0000 7f631b7fe640 1 -- 172.21.15.16:0/3827042638 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6314105fb0 con 0x7f6324108150 2024-08-28T19:20:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.129+0000 7f632a581640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63100b4c80 0x7f63100b7190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f632410eda0 tx=0x7f6320009110 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:34.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.134+0000 7f631b7fe640 1 -- 172.21.15.16:0/3827042638 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f63140ccaf0 con 0x7f6324108150 2024-08-28T19:20:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.333+0000 7f632c7e5640 1 -- 172.21.15.16:0/3827042638 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f63240635a0 con 0x7f63100b4c80 2024-08-28T19:20:34.337 INFO:teuthology.orchestra.run.smithi016.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-08-28T19:20:34.337 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002C480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-08-28T19:20:34.337 INFO:teuthology.orchestra.run.smithi016.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-08-28T19:20:34.337 INFO:teuthology.orchestra.run.smithi016.stdout:smithi115 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UU400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-08-28T19:20:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.336+0000 7f631b7fe640 1 -- 172.21.15.16:0/3827042638 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f63240635a0 con 0x7f63100b4c80 2024-08-28T19:20:34.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 -- 172.21.15.16:0/3827042638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63100b4c80 msgr2=0x7f63100b7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63100b4c80 0x7f63100b7190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f632410eda0 tx=0x7f6320009110 comp rx=0 tx=0).stop 2024-08-28T19:20:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 -- 172.21.15.16:0/3827042638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6324108150 msgr2=0x7f632410d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6324108150 0x7f632410d350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f631402a930 tx=0x7f6314032ad0 comp rx=0 tx=0).stop 2024-08-28T19:20:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 -- 172.21.15.16:0/3827042638 shutdown_connections 2024-08-28T19:20:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6324108aa0 0x7f632410d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6324108150 0x7f632410d350 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63100b4c80 0x7f63100b7190 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 --2- 172.21.15.16:0/3827042638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6324102810 0x7f632410f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 -- 172.21.15.16:0/3827042638 >> 172.21.15.16:0/3827042638 conn(0x7f63240794b0 msgr2=0x7f6324106290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 -- 172.21.15.16:0/3827042638 shutdown_connections 2024-08-28T19:20:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:34.338+0000 7f632c7e5640 1 -- 172.21.15.16:0/3827042638 wait complete. 2024-08-28T19:20:35.041 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-08-28T19:20:35.051 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-08-28T19:20:35.052 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- bash -c 'ceph fs dump' 2024-08-28T19:20:35.325 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:35.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:35 smithi115 ceph-mon[22184]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:35.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:35 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:35.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:35 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:35.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:35 smithi016 ceph-mon[18556]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:35.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:35 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:35.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:35 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:35 smithi028 ceph-mon[21769]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:35 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:35.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:35 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:35.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.799+0000 7fb420b2d640 1 -- 172.21.15.16:0/620691361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41c108140 msgr2=0x7fb41c108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:35.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.799+0000 7fb420b2d640 1 --2- 172.21.15.16:0/620691361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41c108140 0x7fb41c108550 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb410009900 tx=0x7fb41002a410 comp rx=0 tx=0).stop 2024-08-28T19:20:35.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.799+0000 7fb420b2d640 1 -- 172.21.15.16:0/620691361 shutdown_connections 2024-08-28T19:20:35.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.799+0000 7fb420b2d640 1 --2- 172.21.15.16:0/620691361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb41c108a90 0x7fb41c10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.799+0000 7fb420b2d640 1 --2- 172.21.15.16:0/620691361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41c108140 0x7fb41c108550 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.799+0000 7fb420b2d640 1 --2- 172.21.15.16:0/620691361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb41c102800 0x7fb41c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.799+0000 7fb420b2d640 1 -- 172.21.15.16:0/620691361 >> 172.21.15.16:0/620691361 conn(0x7fb41c079500 msgr2=0x7fb41c079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.800+0000 7fb420b2d640 1 -- 172.21.15.16:0/620691361 shutdown_connections 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.800+0000 7fb420b2d640 1 -- 172.21.15.16:0/620691361 wait complete. 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.800+0000 7fb420b2d640 1 Processor -- start 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.800+0000 7fb420b2d640 1 -- start start 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb420b2d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb41c102800 0x7fb41c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb420b2d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb41c108140 0x7fb41c19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb420b2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41c108a90 0x7fb41c19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb420b2d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c112700 con 0x7fb41c108a90 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb420b2d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c112580 con 0x7fb41c102800 2024-08-28T19:20:35.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb420b2d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c112880 con 0x7fb41c108140 2024-08-28T19:20:35.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb41ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41c108a90 0x7fb41c19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:35.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb41ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41c108a90 0x7fb41c19faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44204/0 (socket says 172.21.15.16:44204) 2024-08-28T19:20:35.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb41ad9d640 1 -- 172.21.15.16:0/73126277 learned_addr learned my addr 172.21.15.16:0/73126277 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:35.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb419d9b640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb41c108140 0x7fb41c19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:35.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb41a59c640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb41c102800 0x7fb41c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:35.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb41ad9d640 1 -- 172.21.15.16:0/73126277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb41c108140 msgr2=0x7fb41c19a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:35.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb41ad9d640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb41c108140 0x7fb41c19a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:35.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb41ad9d640 1 -- 172.21.15.16:0/73126277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb41c102800 msgr2=0x7fb41c19a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:35.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb41ad9d640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb41c102800 0x7fb41c19a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:35.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb41ad9d640 1 -- 172.21.15.16:0/73126277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41c1a01d0 con 0x7fb41c108a90 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.801+0000 7fb41ad9d640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41c108a90 0x7fb41c19faa0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb41400eaa0 tx=0x7fb41400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.802+0000 7fb40b7fe640 1 -- 172.21.15.16:0/73126277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41400cdb0 con 0x7fb41c108a90 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.802+0000 7fb40b7fe640 1 -- 172.21.15.16:0/73126277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb414005020 con 0x7fb41c108a90 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.802+0000 7fb420b2d640 1 -- 172.21.15.16:0/73126277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb41c1a04c0 con 0x7fb41c108a90 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.802+0000 7fb40b7fe640 1 -- 172.21.15.16:0/73126277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4140116d0 con 0x7fb41c108a90 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.802+0000 7fb420b2d640 1 -- 172.21.15.16:0/73126277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb41c1a6c20 con 0x7fb41c108a90 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.803+0000 7fb420b2d640 1 -- 172.21.15.16:0/73126277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb41c103a80 con 0x7fb41c108a90 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.804+0000 7fb40b7fe640 1 -- 172.21.15.16:0/73126277 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7fb4140051d0 con 0x7fb41c108a90 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.805+0000 7fb40b7fe640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4040b4c10 0x7fb4040b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.805+0000 7fb41a59c640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4040b4c10 0x7fb4040b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.805+0000 7fb40b7fe640 1 -- 172.21.15.16:0/73126277 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb4140e5cd0 con 0x7fb41c108a90 2024-08-28T19:20:35.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.806+0000 7fb41a59c640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4040b4c10 0x7fb4040b7120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00bbe0 tx=0x7fb40c00b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:35.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:35.809+0000 7fb40b7fe640 1 -- 172.21.15.16:0/73126277 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fb414016030 con 0x7fb41c108a90 2024-08-28T19:20:36.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.032+0000 7fb420b2d640 1 -- 172.21.15.16:0/73126277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fb41c102c10 con 0x7fb41c108a90 2024-08-28T19:20:36.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.033+0000 7fb40b7fe640 1 -- 172.21.15.16:0/73126277 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3132 (secure 0 0 0) 0x7fb41c102c10 con 0x7fb41c108a90 2024-08-28T19:20:36.034 INFO:teuthology.orchestra.run.smithi016.stdout:e28 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:btime 2024-08-28T19:19:12:646580+0000 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 28 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-08-28T19:18:33.178088+0000 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-08-28T19:19:12.646544+0000 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-08-28T19:20:36.035 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 300 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 0 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 5 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:in 0,1,2,3,4 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24665,1=24649,2=24635,3=24625,4=24619} 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-08-28T19:20:36.036 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3,4] 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data disabled 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 0 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 24665 members: 24665,24649,24635,24625,24619 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b{0:24665} state up:active seq 4 addr [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.e{0:24605} state up:standby-replay seq 1 addr [v2:172.21.15.28:6834/2983279759,v1:172.21.15.28:6835/2983279759] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.i{1:24649} state up:active seq 15 addr [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.k{1:24590} state up:standby-replay seq 1 addr [v2:172.21.15.28:6832/798033587,v1:172.21.15.28:6833/798033587] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.h{2:24635} state up:active seq 16 addr [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.l{2:24583} state up:standby-replay seq 1 addr [v2:172.21.15.115:6832/2581346583,v1:172.21.15.115:6833/2581346583] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c{3:24625} state up:active seq 17 addr [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.037 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a{3:14823} state up:standby-replay seq 1 addr [v2:172.21.15.16:6840/2698492790,v1:172.21.15.16:6841/2698492790] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.038 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.f{4:24619} state up:active seq 18 addr [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.038 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.g{4:14817} state up:standby-replay seq 1 addr [v2:172.21.15.16:6838/3069459721,v1:172.21.15.16:6839/3069459721] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.038 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:36.038 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:36.038 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-08-28T19:20:36.038 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:36.038 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.j{-1:14751} state up:standby seq 1 addr [v2:172.21.15.16:6834/2536532891,v1:172.21.15.16:6835/2536532891] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.038 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d{-1:14763} state up:standby seq 1 addr [v2:172.21.15.16:6836/2448628159,v1:172.21.15.16:6837/2448628159] compat {c=[1],r=[1],i=[1fff]}] 2024-08-28T19:20:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-08-28T19:20:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 -- 172.21.15.16:0/73126277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4040b4c10 msgr2=0x7fb4040b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4040b4c10 0x7fb4040b7120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00bbe0 tx=0x7fb40c00b390 comp rx=0 tx=0).stop 2024-08-28T19:20:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 -- 172.21.15.16:0/73126277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41c108a90 msgr2=0x7fb41c19faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41c108a90 0x7fb41c19faa0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb41400eaa0 tx=0x7fb41400ef60 comp rx=0 tx=0).stop 2024-08-28T19:20:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 -- 172.21.15.16:0/73126277 shutdown_connections 2024-08-28T19:20:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41c108a90 0x7fb41c19faa0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb41c108140 0x7fb41c19a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4040b4c10 0x7fb4040b7120 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:36.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 --2- 172.21.15.16:0/73126277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb41c102800 0x7fb41c19a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:36.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 -- 172.21.15.16:0/73126277 >> 172.21.15.16:0/73126277 conn(0x7fb41c079500 msgr2=0x7fb41c06bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:36.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 -- 172.21.15.16:0/73126277 shutdown_connections 2024-08-28T19:20:36.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:36.036+0000 7fb420b2d640 1 -- 172.21.15.16:0/73126277 wait complete. 2024-08-28T19:20:36.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:36 smithi115 ceph-mon[22184]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:36.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:20:36.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:20:36.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:36.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:36 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/73126277' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-08-28T19:20:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:36 smithi016 ceph-mon[18556]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:20:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:20:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:36 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/73126277' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-08-28T19:20:36.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:36 smithi028 ceph-mon[21769]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:20:36.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:20:36.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:20:36.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:20:36.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:36 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/73126277' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-08-28T19:20:36.753 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- bash -c 'ceph osd dump' 2024-08-28T19:20:37.009 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:37.270 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:37 smithi016 ceph-mon[18556]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:20:37.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.516+0000 7f8cf53b9640 1 -- 172.21.15.16:0/1836110684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf0103950 msgr2=0x7f8cf0103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:37.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.516+0000 7f8cf53b9640 1 --2- 172.21.15.16:0/1836110684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf0103950 0x7f8cf0103d60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009900 tx=0x7f8ce002a410 comp rx=0 tx=0).stop 2024-08-28T19:20:37.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.517+0000 7f8cf53b9640 1 -- 172.21.15.16:0/1836110684 shutdown_connections 2024-08-28T19:20:37.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.517+0000 7f8cf53b9640 1 --2- 172.21.15.16:0/1836110684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf0109be0 0x7f8cf0110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:37.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.517+0000 7f8cf53b9640 1 --2- 172.21.15.16:0/1836110684 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8cf0109290 0x7f8cf01096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:37.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.517+0000 7f8cf53b9640 1 --2- 172.21.15.16:0/1836110684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf0103950 0x7f8cf0103d60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.517+0000 7f8cf53b9640 1 -- 172.21.15.16:0/1836110684 >> 172.21.15.16:0/1836110684 conn(0x7f8cf00ff3a0 msgr2=0x7f8cf01017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.517+0000 7f8cf53b9640 1 -- 172.21.15.16:0/1836110684 shutdown_connections 2024-08-28T19:20:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.517+0000 7f8cf53b9640 1 -- 172.21.15.16:0/1836110684 wait complete. 2024-08-28T19:20:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.517+0000 7f8cf53b9640 1 Processor -- start 2024-08-28T19:20:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.517+0000 7f8cf53b9640 1 -- start start 2024-08-28T19:20:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8cf53b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf0103950 0x7f8cf019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8cf53b9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf0109290 0x7f8cf019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8cf53b9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8cf0109be0 0x7f8cf019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8cf53b9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf01138a0 con 0x7f8cf0103950 2024-08-28T19:20:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8cf53b9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf0113720 con 0x7f8cf0109290 2024-08-28T19:20:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8cf53b9640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf0113a20 con 0x7f8cf0109be0 2024-08-28T19:20:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8ceeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf0103950 0x7f8cf019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8ceeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf0103950 0x7f8cf019a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44220/0 (socket says 172.21.15.16:44220) 2024-08-28T19:20:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8ceeffd640 1 -- 172.21.15.16:0/936154504 learned_addr learned my addr 172.21.15.16:0/936154504 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8ce67fc640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf0109290 0x7f8cf019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8cef7fe640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8cf0109be0 0x7f8cf019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8ceeffd640 1 -- 172.21.15.16:0/936154504 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8cf0109be0 msgr2=0x7f8cf019faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8ceeffd640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8cf0109be0 0x7f8cf019faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8ceeffd640 1 -- 172.21.15.16:0/936154504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf0109290 msgr2=0x7f8cf019a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8ceeffd640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf0109290 0x7f8cf019a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8ceeffd640 1 -- 172.21.15.16:0/936154504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf01a0220 con 0x7f8cf0103950 2024-08-28T19:20:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.518+0000 7f8ceeffd640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf0103950 0x7f8cf019a2b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0005940 tx=0x7f8ce00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.519+0000 7f8cecff9640 1 -- 172.21.15.16:0/936154504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce003f030 con 0x7f8cf0103950 2024-08-28T19:20:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.519+0000 7f8cecff9640 1 -- 172.21.15.16:0/936154504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ce0003cb0 con 0x7f8cf0103950 2024-08-28T19:20:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.519+0000 7f8cf53b9640 1 -- 172.21.15.16:0/936154504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf01a04b0 con 0x7f8cf0103950 2024-08-28T19:20:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.519+0000 7f8cecff9640 1 -- 172.21.15.16:0/936154504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0035070 con 0x7f8cf0103950 2024-08-28T19:20:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.519+0000 7f8cf53b9640 1 -- 172.21.15.16:0/936154504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf01a6c70 con 0x7f8cf0103950 2024-08-28T19:20:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.520+0000 7f8cf53b9640 1 -- 172.21.15.16:0/936154504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cf0103d60 con 0x7f8cf0103950 2024-08-28T19:20:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.521+0000 7f8cecff9640 1 -- 172.21.15.16:0/936154504 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f8ce0042030 con 0x7f8cf0103950 2024-08-28T19:20:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.523+0000 7f8cecff9640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8cdc0b4c10 0x7f8cdc0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.523+0000 7f8ce67fc640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8cdc0b4c10 0x7f8cdc0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.523+0000 7f8cecff9640 1 -- 172.21.15.16:0/936154504 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f8ce0106360 con 0x7f8cf0103950 2024-08-28T19:20:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.523+0000 7f8ce67fc640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8cdc0b4c10 0x7f8cdc0b7120 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8005fd0 tx=0x7f8cd8005e20 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.527+0000 7f8cecff9640 1 -- 172.21.15.16:0/936154504 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8ce00ccea0 con 0x7f8cf0103950 2024-08-28T19:20:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:37 smithi028 ceph-mon[21769]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:20:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:37 smithi115 ceph-mon[22184]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:20:37.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.717+0000 7f8cf53b9640 1 -- 172.21.15.16:0/936154504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f8cf010b9b0 con 0x7f8cf0103950 2024-08-28T19:20:37.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.718+0000 7f8cecff9640 1 -- 172.21.15.16:0/936154504 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v84) ==== 56+0+6056 (secure 0 0 0) 0x7f8ce00cc7a0 con 0x7f8cf0103950 2024-08-28T19:20:37.719 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 84 2024-08-28T19:20:37.719 INFO:teuthology.orchestra.run.smithi016.stdout:fsid eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:20:37.719 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-08-28T19:10:20.239380+0000 2024-08-28T19:20:37.720 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-08-28T19:20:09.003260+0000 2024-08-28T19:20:37.720 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-08-28T19:20:37.720 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 27 2024-08-28T19:20:37.720 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2024-08-28T19:20:37.720 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2024-08-28T19:20:37.720 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2024-08-28T19:20:37.720 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2024-08-28T19:20:37.720 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client luminous 2024-08-28T19:20:37.720 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release squid 2024-08-28T19:20:37.720 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2024-08-28T19:20:37.721 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-08-28T19:20:37.721 INFO:teuthology.orchestra.run.smithi016.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-08-28T19:20:37.721 INFO:teuthology.orchestra.run.smithi016.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-08-28T19:20:37.721 INFO:teuthology.orchestra.run.smithi016.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-08-28T19:20:37.721 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 12 2024-08-28T19:20:37.721 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 9 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6802/1371554327,v1:172.21.15.16:6803/1371554327] [v2:172.21.15.16:6804/1371554327,v1:172.21.15.16:6805/1371554327] exists,up 7a5e8a2d-958b-4c1f-b2be-90743c629505 2024-08-28T19:20:37.721 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 13 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6800/666617235,v1:172.21.15.28:6801/666617235] [v2:172.21.15.28:6802/666617235,v1:172.21.15.28:6803/666617235] exists,up d4767044-e5ca-447f-9666-0840aa1028dd 2024-08-28T19:20:37.721 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6800/1599247648,v1:172.21.15.115:6801/1599247648] [v2:172.21.15.115:6802/1599247648,v1:172.21.15.115:6803/1599247648] exists,up ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22 2024-08-28T19:20:37.721 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up in weight 1 up_from 24 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6810/3791831569,v1:172.21.15.16:6811/3791831569] [v2:172.21.15.16:6812/3791831569,v1:172.21.15.16:6813/3791831569] exists,up 0edc63dd-6f28-4c2a-9bde-d6be083fc6c5 2024-08-28T19:20:37.721 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 up in weight 1 up_from 29 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6808/302267969,v1:172.21.15.28:6809/302267969] [v2:172.21.15.28:6810/302267969,v1:172.21.15.28:6811/302267969] exists,up fcaeb7d4-aa38-40a7-abdf-3f16cf9047df 2024-08-28T19:20:37.722 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 up in weight 1 up_from 34 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6808/1119597596,v1:172.21.15.115:6809/1119597596] [v2:172.21.15.115:6810/1119597596,v1:172.21.15.115:6811/1119597596] exists,up 762a439b-bec1-471e-ab08-aa233ee83d7a 2024-08-28T19:20:37.722 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 up in weight 1 up_from 40 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6818/1671078622,v1:172.21.15.16:6819/1671078622] [v2:172.21.15.16:6820/1671078622,v1:172.21.15.16:6821/1671078622] exists,up 33c56c57-ce44-4000-94b3-518d7ce0afd1 2024-08-28T19:20:37.722 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 up in weight 1 up_from 45 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6816/2650596315,v1:172.21.15.28:6817/2650596315] [v2:172.21.15.28:6818/2650596315,v1:172.21.15.28:6819/2650596315] exists,up 8ef482ae-33bd-4121-8b16-c80ad1b9178b 2024-08-28T19:20:37.722 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 up in weight 1 up_from 51 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6816/557022149,v1:172.21.15.115:6817/557022149] [v2:172.21.15.115:6818/557022149,v1:172.21.15.115:6819/557022149] exists,up f814cd8e-30c2-4e74-aa5c-eb2d40e27586 2024-08-28T19:20:37.722 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 up in weight 1 up_from 57 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6826/1827017769,v1:172.21.15.16:6827/1827017769] [v2:172.21.15.16:6828/1827017769,v1:172.21.15.16:6829/1827017769] exists,up 9617dc8d-0eb5-4fe1-bc88-5d37d40c065f 2024-08-28T19:20:37.722 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 up in weight 1 up_from 62 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6824/4224878207,v1:172.21.15.28:6825/4224878207] [v2:172.21.15.28:6826/4224878207,v1:172.21.15.28:6827/4224878207] exists,up dbed2c84-11b5-409d-a8e5-33c9fddc5698 2024-08-28T19:20:37.722 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 up in weight 1 up_from 68 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6824/3906373111,v1:172.21.15.115:6825/3906373111] [v2:172.21.15.115:6826/3906373111,v1:172.21.15.115:6827/3906373111] exists,up 3ce535d6-4278-46a0-af6d-a7c90a74aaee 2024-08-28T19:20:37.722 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.b [2,4,8,11] 2024-08-28T19:20:37.722 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.16 [2,4] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.1a [8,1] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.1f [7,11] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.21 [5,9] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.22 [3,9] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.23 [4,8] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.31 [4,6] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.34 [4,9] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.5 [5,10] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.6 [9,8] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.14 [9,8] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.1b [9,8] 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/4224232323 expires 2024-08-29T19:12:07.367977+0000 2024-08-28T19:20:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/4022283606 expires 2024-08-29T19:12:07.367977+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/3264341019 expires 2024-08-29T19:12:07.367977+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/3057984026 expires 2024-08-29T19:10:41.419191+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3042282523 expires 2024-08-29T19:10:41.419191+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3243085606 expires 2024-08-29T19:10:41.419191+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/1768458721 expires 2024-08-29T19:10:41.419191+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/949818237 expires 2024-08-29T19:11:05.284513+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/3057984026 expires 2024-08-29T19:10:41.419191+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/1317220879 expires 2024-08-29T19:11:05.284513+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/2455667776 expires 2024-08-29T19:11:05.284513+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/3371171272 expires 2024-08-29T19:11:05.284513+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/3264341019 expires 2024-08-29T19:12:07.367977+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/3371171272 expires 2024-08-29T19:11:05.284513+0000 2024-08-28T19:20:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3868803359 expires 2024-08-29T19:12:07.367977+0000 2024-08-28T19:20:37.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 -- 172.21.15.16:0/936154504 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8cdc0b4c10 msgr2=0x7f8cdc0b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:37.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8cdc0b4c10 0x7f8cdc0b7120 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8005fd0 tx=0x7f8cd8005e20 comp rx=0 tx=0).stop 2024-08-28T19:20:37.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 -- 172.21.15.16:0/936154504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf0103950 msgr2=0x7f8cf019a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:37.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf0103950 0x7f8cf019a2b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0005940 tx=0x7f8ce00042f0 comp rx=0 tx=0).stop 2024-08-28T19:20:37.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 -- 172.21.15.16:0/936154504 shutdown_connections 2024-08-28T19:20:37.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8cf0109be0 0x7f8cf019faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:37.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8cdc0b4c10 0x7f8cdc0b7120 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:37.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf0109290 0x7f8cf019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:37.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 --2- 172.21.15.16:0/936154504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf0103950 0x7f8cf019a2b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:37.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 -- 172.21.15.16:0/936154504 >> 172.21.15.16:0/936154504 conn(0x7f8cf00ff3a0 msgr2=0x7f8cf01017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:37.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 -- 172.21.15.16:0/936154504 shutdown_connections 2024-08-28T19:20:37.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:37.721+0000 7f8cf53b9640 1 -- 172.21.15.16:0/936154504 wait complete. 2024-08-28T19:20:38.402 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:38 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/936154504' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-28T19:20:38.403 INFO:teuthology.run_tasks:Running task fs.ready... 2024-08-28T19:20:38.414 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:20:38.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:38 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/936154504' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-28T19:20:38.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:38 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/936154504' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-28T19:20:38.674 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.196+0000 7f01f423c640 1 -- 172.21.15.16:0/1163320412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec102840 msgr2=0x7f01ec102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.196+0000 7f01f423c640 1 --2- 172.21.15.16:0/1163320412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec102840 0x7f01ec102c50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009900 tx=0x7f01e802a410 comp rx=0 tx=0).stop 2024-08-28T19:20:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.196+0000 7f01f423c640 1 -- 172.21.15.16:0/1163320412 shutdown_connections 2024-08-28T19:20:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.196+0000 7f01f423c640 1 --2- 172.21.15.16:0/1163320412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ec108ad0 0x7f01ec10f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.196+0000 7f01f423c640 1 --2- 172.21.15.16:0/1163320412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ec108180 0x7f01ec108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.196+0000 7f01f423c640 1 --2- 172.21.15.16:0/1163320412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec102840 0x7f01ec102c50 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.196+0000 7f01f423c640 1 -- 172.21.15.16:0/1163320412 >> 172.21.15.16:0/1163320412 conn(0x7f01ec079580 msgr2=0x7f01ec079990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.196+0000 7f01f423c640 1 -- 172.21.15.16:0/1163320412 shutdown_connections 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.196+0000 7f01f423c640 1 -- 172.21.15.16:0/1163320412 wait complete. 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f423c640 1 Processor -- start 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f423c640 1 -- start start 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f423c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec102840 0x7f01ec19b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f423c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ec108180 0x7f01ec19b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f423c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ec108ad0 0x7f01ec19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f423c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01ec112780 con 0x7f01ec102840 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f423c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01ec112600 con 0x7f01ec108ad0 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f423c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f01ec112900 con 0x7f01ec108180 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f1fd8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec102840 0x7f01ec19b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f1fd8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec102840 0x7f01ec19b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44250/0 (socket says 172.21.15.16:44250) 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f1fd8640 1 -- 172.21.15.16:0/2462829355 learned_addr learned my addr 172.21.15.16:0/2462829355 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.197+0000 7f01f27d9640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ec108ad0 0x7f01ec19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01f17d7640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ec108180 0x7f01ec19b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01f1fd8640 1 -- 172.21.15.16:0/2462829355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ec108180 msgr2=0x7f01ec19b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01f1fd8640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ec108180 0x7f01ec19b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01f1fd8640 1 -- 172.21.15.16:0/2462829355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ec108ad0 msgr2=0x7f01ec19bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01f1fd8640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ec108ad0 0x7f01ec19bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01f1fd8640 1 -- 172.21.15.16:0/2462829355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ec1a62d0 con 0x7f01ec102840 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01f17d7640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ec108180 0x7f01ec19b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01f1fd8640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec102840 0x7f01ec19b050 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f01e8005ab0 tx=0x7f01e8003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01deffd640 1 -- 172.21.15.16:0/2462829355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e803f070 con 0x7f01ec102840 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01f423c640 1 -- 172.21.15.16:0/2462829355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01ec1a6560 con 0x7f01ec102840 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01deffd640 1 -- 172.21.15.16:0/2462829355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01e8031d60 con 0x7f01ec102840 2024-08-28T19:20:39.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01deffd640 1 -- 172.21.15.16:0/2462829355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8035030 con 0x7f01ec102840 2024-08-28T19:20:39.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.198+0000 7f01f423c640 1 -- 172.21.15.16:0/2462829355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01ec1a6a40 con 0x7f01ec102840 2024-08-28T19:20:39.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.200+0000 7f01f423c640 1 -- 172.21.15.16:0/2462829355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01ec102c50 con 0x7f01ec102840 2024-08-28T19:20:39.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.201+0000 7f01deffd640 1 -- 172.21.15.16:0/2462829355 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 152645+0+0 (secure 0 0 0) 0x7f01e8042070 con 0x7f01ec102840 2024-08-28T19:20:39.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.202+0000 7f01deffd640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01d80b4c10 0x7f01d80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:39.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.202+0000 7f01f17d7640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01d80b4c10 0x7f01d80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:39.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.203+0000 7f01deffd640 1 -- 172.21.15.16:0/2462829355 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f01e8106840 con 0x7f01ec102840 2024-08-28T19:20:39.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.203+0000 7f01f17d7640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01d80b4c10 0x7f01d80b7120 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f01ec1009f0 tx=0x7f01e0005d30 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:39.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.205+0000 7f01deffd640 1 -- 172.21.15.16:0/2462829355 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f01e80cd360 con 0x7f01ec102840 2024-08-28T19:20:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.428+0000 7f01f423c640 1 -- 172.21.15.16:0/2462829355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01ec06ba30 con 0x7f01ec102840 2024-08-28T19:20:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.429+0000 7f01deffd640 1 -- 172.21.15.16:0/2462829355 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11977 (secure 0 0 0) 0x7f01e80ccc60 con 0x7f01ec102840 2024-08-28T19:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-08-28T19:19:12:646580+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:19:12.646544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:20:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-08-28T19:20:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.431+0000 7f01f423c640 1 -- 172.21.15.16:0/2462829355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01d80b4c10 msgr2=0x7f01d80b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.431+0000 7f01f423c640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01d80b4c10 0x7f01d80b7120 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f01ec1009f0 tx=0x7f01e0005d30 comp rx=0 tx=0).stop 2024-08-28T19:20:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.431+0000 7f01f423c640 1 -- 172.21.15.16:0/2462829355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec102840 msgr2=0x7f01ec19b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.431+0000 7f01f423c640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec102840 0x7f01ec19b050 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f01e8005ab0 tx=0x7f01e8003d50 comp rx=0 tx=0).stop 2024-08-28T19:20:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.431+0000 7f01f423c640 1 -- 172.21.15.16:0/2462829355 shutdown_connections 2024-08-28T19:20:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.431+0000 7f01f423c640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01d80b4c10 0x7f01d80b7120 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.431+0000 7f01f423c640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ec108ad0 0x7f01ec19bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.431+0000 7f01f423c640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ec108180 0x7f01ec19b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.431+0000 7f01f423c640 1 --2- 172.21.15.16:0/2462829355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec102840 0x7f01ec19b050 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.431+0000 7f01f423c640 1 -- 172.21.15.16:0/2462829355 >> 172.21.15.16:0/2462829355 conn(0x7f01ec079580 msgr2=0x7f01ec101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.432+0000 7f01f423c640 1 -- 172.21.15.16:0/2462829355 shutdown_connections 2024-08-28T19:20:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:39.432+0000 7f01f423c640 1 -- 172.21.15.16:0/2462829355 wait complete. 2024-08-28T19:20:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:39 smithi016 ceph-mon[18556]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:20:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:39 smithi028 ceph-mon[21769]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:20:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:39 smithi115 ceph-mon[22184]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:20:40.160 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:20:40.421 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:40 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2462829355' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:40.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:40 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2462829355' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:40 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2462829355' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.912+0000 7ff83f4ec640 1 -- 172.21.15.16:0/2473772007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838103900 msgr2=0x7ff838103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.912+0000 7ff83f4ec640 1 --2- 172.21.15.16:0/2473772007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838103900 0x7ff838103d10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff834009900 tx=0x7ff83402a410 comp rx=0 tx=0).stop 2024-08-28T19:20:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.912+0000 7ff83f4ec640 1 -- 172.21.15.16:0/2473772007 shutdown_connections 2024-08-28T19:20:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.912+0000 7ff83f4ec640 1 --2- 172.21.15.16:0/2473772007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff838109b90 0x7ff8381103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.912+0000 7ff83f4ec640 1 --2- 172.21.15.16:0/2473772007 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff838109240 0x7ff838109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.912+0000 7ff83f4ec640 1 --2- 172.21.15.16:0/2473772007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838103900 0x7ff838103d10 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.912+0000 7ff83f4ec640 1 -- 172.21.15.16:0/2473772007 >> 172.21.15.16:0/2473772007 conn(0x7ff83807b5f0 msgr2=0x7ff83807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.912+0000 7ff83f4ec640 1 -- 172.21.15.16:0/2473772007 shutdown_connections 2024-08-28T19:20:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.912+0000 7ff83f4ec640 1 -- 172.21.15.16:0/2473772007 wait complete. 2024-08-28T19:20:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83f4ec640 1 Processor -- start 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83f4ec640 1 -- start start 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83f4ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff838103900 0x7ff83819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83f4ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838109240 0x7ff83819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83f4ec640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff838109b90 0x7ff83819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83f4ec640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff838113890 con 0x7ff838109240 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83f4ec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff838113710 con 0x7ff838103900 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83f4ec640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff838113a10 con 0x7ff838109b90 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83ca87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838109240 0x7ff83819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83ca87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838109240 0x7ff83819a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44272/0 (socket says 172.21.15.16:44272) 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83ca87640 1 -- 172.21.15.16:0/3347166010 learned_addr learned my addr 172.21.15.16:0/3347166010 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.913+0000 7ff83d288640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff838103900 0x7ff83819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff83da89640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff838109b90 0x7ff83819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff83ca87640 1 -- 172.21.15.16:0/3347166010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff838109b90 msgr2=0x7ff83819fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff83ca87640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff838109b90 0x7ff83819fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff83ca87640 1 -- 172.21.15.16:0/3347166010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff838103900 msgr2=0x7ff83819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff83ca87640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff838103900 0x7ff83819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff83ca87640 1 -- 172.21.15.16:0/3347166010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8381a0150 con 0x7ff838109240 2024-08-28T19:20:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff83ca87640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838109240 0x7ff83819a840 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff828006fd0 tx=0x7ff82800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff82e7fc640 1 -- 172.21.15.16:0/3347166010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828011830 con 0x7ff838109240 2024-08-28T19:20:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff83f4ec640 1 -- 172.21.15.16:0/3347166010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8381a0440 con 0x7ff838109240 2024-08-28T19:20:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff82e7fc640 1 -- 172.21.15.16:0/3347166010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff828016070 con 0x7ff838109240 2024-08-28T19:20:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff82e7fc640 1 -- 172.21.15.16:0/3347166010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828010e00 con 0x7ff838109240 2024-08-28T19:20:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.914+0000 7ff83f4ec640 1 -- 172.21.15.16:0/3347166010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8381a6c60 con 0x7ff838109240 2024-08-28T19:20:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.915+0000 7ff83f4ec640 1 -- 172.21.15.16:0/3347166010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff838103d10 con 0x7ff838109240 2024-08-28T19:20:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.917+0000 7ff82e7fc640 1 -- 172.21.15.16:0/3347166010 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7ff8280119d0 con 0x7ff838109240 2024-08-28T19:20:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.918+0000 7ff82e7fc640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff8240b4bb0 0x7ff8240b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.918+0000 7ff83d288640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff8240b4bb0 0x7ff8240b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.918+0000 7ff82e7fc640 1 -- 172.21.15.16:0/3347166010 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff8280e51d0 con 0x7ff838109240 2024-08-28T19:20:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.918+0000 7ff83d288640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff8240b4bb0 0x7ff8240b70c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff834031440 tx=0x7ff834005e80 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:40.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:40.921+0000 7ff82e7fc640 1 -- 172.21.15.16:0/3347166010 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ff8280abe70 con 0x7ff838109240 2024-08-28T19:20:41.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.143+0000 7ff83f4ec640 1 -- 172.21.15.16:0/3347166010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff83819b3d0 con 0x7ff838109240 2024-08-28T19:20:41.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.144+0000 7ff82e7fc640 1 -- 172.21.15.16:0/3347166010 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11977 (secure 0 0 0) 0x7ff8280ab770 con 0x7ff838109240 2024-08-28T19:20:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:41.146 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-08-28T19:19:12:646580+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:19:12.646544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:20:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-08-28T19:20:41.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 -- 172.21.15.16:0/3347166010 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff8240b4bb0 msgr2=0x7ff8240b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:41.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff8240b4bb0 0x7ff8240b70c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff834031440 tx=0x7ff834005e80 comp rx=0 tx=0).stop 2024-08-28T19:20:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 -- 172.21.15.16:0/3347166010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838109240 msgr2=0x7ff83819a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838109240 0x7ff83819a840 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff828006fd0 tx=0x7ff82800b8e0 comp rx=0 tx=0).stop 2024-08-28T19:20:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 -- 172.21.15.16:0/3347166010 shutdown_connections 2024-08-28T19:20:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff838109b90 0x7ff83819fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838109240 0x7ff83819a840 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff8240b4bb0 0x7ff8240b70c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 --2- 172.21.15.16:0/3347166010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff838103900 0x7ff83819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 -- 172.21.15.16:0/3347166010 >> 172.21.15.16:0/3347166010 conn(0x7ff83807b5f0 msgr2=0x7ff8381072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 -- 172.21.15.16:0/3347166010 shutdown_connections 2024-08-28T19:20:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:41.146+0000 7ff83f4ec640 1 -- 172.21.15.16:0/3347166010 wait complete. 2024-08-28T19:20:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:41 smithi016 ceph-mon[18556]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-08-28T19:20:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:41 smithi016 ceph-mon[18556]: mgrmap e22: x(active, since 8m), standbys: z, y 2024-08-28T19:20:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:41 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3347166010' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:41 smithi028 ceph-mon[21769]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-08-28T19:20:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:41 smithi028 ceph-mon[21769]: mgrmap e22: x(active, since 8m), standbys: z, y 2024-08-28T19:20:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:41 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3347166010' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:41 smithi115 ceph-mon[22184]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-08-28T19:20:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:41 smithi115 ceph-mon[22184]: mgrmap e22: x(active, since 8m), standbys: z, y 2024-08-28T19:20:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:41 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3347166010' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:41.883 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:20:42.137 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.614+0000 7fb965d71640 1 -- 172.21.15.16:0/2803447706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb960103900 msgr2=0x7fb960103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.614+0000 7fb965d71640 1 --2- 172.21.15.16:0/2803447706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb960103900 0x7fb960103d10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb950009900 tx=0x7fb95002a410 comp rx=0 tx=0).stop 2024-08-28T19:20:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.614+0000 7fb965d71640 1 -- 172.21.15.16:0/2803447706 shutdown_connections 2024-08-28T19:20:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.614+0000 7fb965d71640 1 --2- 172.21.15.16:0/2803447706 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb960109b90 0x7fb9601103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.614+0000 7fb965d71640 1 --2- 172.21.15.16:0/2803447706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb960109240 0x7fb960109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.614+0000 7fb965d71640 1 --2- 172.21.15.16:0/2803447706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb960103900 0x7fb960103d10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.614+0000 7fb965d71640 1 -- 172.21.15.16:0/2803447706 >> 172.21.15.16:0/2803447706 conn(0x7fb96007b5f0 msgr2=0x7fb96007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.614+0000 7fb965d71640 1 -- 172.21.15.16:0/2803447706 shutdown_connections 2024-08-28T19:20:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.614+0000 7fb965d71640 1 -- 172.21.15.16:0/2803447706 wait complete. 2024-08-28T19:20:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.615+0000 7fb965d71640 1 Processor -- start 2024-08-28T19:20:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.615+0000 7fb965d71640 1 -- start start 2024-08-28T19:20:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb965d71640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb960103900 0x7fb96019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb965d71640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb960109240 0x7fb96019a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb965d71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb960109b90 0x7fb96019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb965d71640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb960113850 con 0x7fb960109b90 2024-08-28T19:20:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb965d71640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9601136d0 con 0x7fb960103900 2024-08-28T19:20:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb965d71640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb9601139d0 con 0x7fb960109240 2024-08-28T19:20:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb960109b90 0x7fb96019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb960109b90 0x7fb96019fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44292/0 (socket says 172.21.15.16:44292) 2024-08-28T19:20:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95ffff640 1 -- 172.21.15.16:0/1273465466 learned_addr learned my addr 172.21.15.16:0/1273465466 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95f7fe640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb960103900 0x7fb96019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95effd640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb960109240 0x7fb96019a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95ffff640 1 -- 172.21.15.16:0/1273465466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb960109240 msgr2=0x7fb96019a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95ffff640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb960109240 0x7fb96019a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95ffff640 1 -- 172.21.15.16:0/1273465466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb960103900 msgr2=0x7fb96019a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95ffff640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb960103900 0x7fb96019a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95ffff640 1 -- 172.21.15.16:0/1273465466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9601a0290 con 0x7fb960109b90 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95ffff640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb960109b90 0x7fb96019fb60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb95800cad0 tx=0x7fb95800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.616+0000 7fb95cff9640 1 -- 172.21.15.16:0/1273465466 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95800ae20 con 0x7fb960109b90 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.617+0000 7fb95cff9640 1 -- 172.21.15.16:0/1273465466 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb958005020 con 0x7fb960109b90 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.617+0000 7fb965d71640 1 -- 172.21.15.16:0/1273465466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9601a0580 con 0x7fb960109b90 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.617+0000 7fb95cff9640 1 -- 172.21.15.16:0/1273465466 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95800f6b0 con 0x7fb960109b90 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.617+0000 7fb965d71640 1 -- 172.21.15.16:0/1273465466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9601a6ce0 con 0x7fb960109b90 2024-08-28T19:20:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.618+0000 7fb965d71640 1 -- 172.21.15.16:0/1273465466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb960103d10 con 0x7fb960109b90 2024-08-28T19:20:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.619+0000 7fb95cff9640 1 -- 172.21.15.16:0/1273465466 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fb9580051d0 con 0x7fb960109b90 2024-08-28T19:20:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.620+0000 7fb95cff9640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb9480b4b80 0x7fb9480b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.620+0000 7fb95f7fe640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb9480b4b80 0x7fb9480b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.620+0000 7fb95cff9640 1 -- 172.21.15.16:0/1273465466 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb9580e5980 con 0x7fb960109b90 2024-08-28T19:20:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.621+0000 7fb95f7fe640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb9480b4b80 0x7fb9480b7090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb950031440 tx=0x7fb950005e80 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.623+0000 7fb95cff9640 1 -- 172.21.15.16:0/1273465466 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fb958014030 con 0x7fb960109b90 2024-08-28T19:20:42.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.809+0000 7fb965d71640 1 -- 172.21.15.16:0/1273465466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb96019b450 con 0x7fb960109b90 2024-08-28T19:20:42.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.810+0000 7fb95cff9640 1 -- 172.21.15.16:0/1273465466 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+25180 (secure 0 0 0) 0x7fb96019b450 con 0x7fb960109b90 2024-08-28T19:20:42.812 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:42.812 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:20:09.003260+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:20:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.812+0000 7fb965d71640 1 -- 172.21.15.16:0/1273465466 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb9480b4b80 msgr2=0x7fb9480b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.812+0000 7fb965d71640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb9480b4b80 0x7fb9480b7090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb950031440 tx=0x7fb950005e80 comp rx=0 tx=0).stop 2024-08-28T19:20:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.812+0000 7fb965d71640 1 -- 172.21.15.16:0/1273465466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb960109b90 msgr2=0x7fb96019fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.812+0000 7fb965d71640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb960109b90 0x7fb96019fb60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb95800cad0 tx=0x7fb95800cf90 comp rx=0 tx=0).stop 2024-08-28T19:20:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.812+0000 7fb965d71640 1 -- 172.21.15.16:0/1273465466 shutdown_connections 2024-08-28T19:20:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.812+0000 7fb965d71640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb960109b90 0x7fb96019fb60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.813+0000 7fb965d71640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb960109240 0x7fb96019a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.813+0000 7fb965d71640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb9480b4b80 0x7fb9480b7090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.813+0000 7fb965d71640 1 --2- 172.21.15.16:0/1273465466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb960103900 0x7fb96019a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.813+0000 7fb965d71640 1 -- 172.21.15.16:0/1273465466 >> 172.21.15.16:0/1273465466 conn(0x7fb96007b5f0 msgr2=0x7fb960107390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:42.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.813+0000 7fb965d71640 1 -- 172.21.15.16:0/1273465466 shutdown_connections 2024-08-28T19:20:42.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:42.813+0000 7fb965d71640 1 -- 172.21.15.16:0/1273465466 wait complete. 2024-08-28T19:20:43.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:43 smithi016 ceph-mon[18556]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 12 op/s 2024-08-28T19:20:43.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:43 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1273465466' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:20:43.537 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-08-28T19:18:33.178088+0000', 'modified': '2024-08-28T19:19:12.646544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24665, 'mds_1': 24649, 'mds_2': 24635, 'mds_3': 24625, 'mds_4': 24619}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14817': {'gid': 14817, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6839/3069459721', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 3069459721}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 3069459721}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14823': {'gid': 14823, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6841/2698492790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6840', 'nonce': 2698492790}, {'type': 'v1', 'addr': '172.21.15.16:6841', 'nonce': 2698492790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24583': {'gid': 24583, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.115:6833/2581346583', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6832', 'nonce': 2581346583}, {'type': 'v1', 'addr': '172.21.15.115:6833', 'nonce': 2581346583}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24590': {'gid': 24590, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6833/798033587', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 798033587}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 798033587}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24605': {'gid': 24605, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6835/2983279759', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2983279759}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2983279759}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24619': {'gid': 24619, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.115:6835/1297102593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6834', 'nonce': 1297102593}, {'type': 'v1', 'addr': '172.21.15.115:6835', 'nonce': 1297102593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24625': {'gid': 24625, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.115:6837/4186796507', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6836', 'nonce': 4186796507}, {'type': 'v1', 'addr': '172.21.15.115:6837', 'nonce': 4186796507}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24635': {'gid': 24635, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.28:6837/2888152030', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6836', 'nonce': 2888152030}, {'type': 'v1', 'addr': '172.21.15.28:6837', 'nonce': 2888152030}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24649': {'gid': 24649, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.115:6839/3027291414', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6838', 'nonce': 3027291414}, {'type': 'v1', 'addr': '172.21.15.115:6839', 'nonce': 3027291414}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24665': {'gid': 24665, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6839/2518320595', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 2518320595}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 2518320595}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24665, 'qdb_cluster': [24665, 24649, 24635, 24625, 24619]} 2024-08-28T19:20:43.537 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-08-28T19:20:43.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.f status 2024-08-28T19:20:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:43 smithi028 ceph-mon[21769]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 12 op/s 2024-08-28T19:20:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:43 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1273465466' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:20:43.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:43 smithi115 ceph-mon[22184]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 12 op/s 2024-08-28T19:20:43.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:43 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1273465466' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:20:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:44.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.250+0000 7f9798821640 1 -- 172.21.15.16:0/2415709978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9794108170 msgr2=0x7f9794108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:44.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.250+0000 7f9798821640 1 --2- 172.21.15.16:0/2415709978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9794108170 0x7f9794108580 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9788009900 tx=0x7f978802a410 comp rx=0 tx=0).stop 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.250+0000 7f9798821640 1 -- 172.21.15.16:0/2415709978 shutdown_connections 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.250+0000 7f9798821640 1 --2- 172.21.15.16:0/2415709978 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9794108ac0 0x7f979410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.250+0000 7f9798821640 1 --2- 172.21.15.16:0/2415709978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9794108170 0x7f9794108580 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.250+0000 7f9798821640 1 --2- 172.21.15.16:0/2415709978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9794102880 0x7f9794102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.250+0000 7f9798821640 1 -- 172.21.15.16:0/2415709978 >> 172.21.15.16:0/2415709978 conn(0x7f9794079550 msgr2=0x7f9794079960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.250+0000 7f9798821640 1 -- 172.21.15.16:0/2415709978 shutdown_connections 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.250+0000 7f9798821640 1 -- 172.21.15.16:0/2415709978 wait complete. 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.251+0000 7f9798821640 1 Processor -- start 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.251+0000 7f9798821640 1 -- start start 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.251+0000 7f9798821640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9794102880 0x7f979419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.251+0000 7f9798821640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9794108170 0x7f979419a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.251+0000 7f9798821640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9794108ac0 0x7f979419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.251+0000 7f9798821640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97941138a0 con 0x7f9794108170 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.251+0000 7f9798821640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9794113720 con 0x7f9794102880 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.251+0000 7f9798821640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9794113a20 con 0x7f9794108ac0 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.251+0000 7f9791d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9794108170 0x7f979419a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.251+0000 7f9791d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9794108170 0x7f979419a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44316/0 (socket says 172.21.15.16:44316) 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.251+0000 7f9791d9b640 1 -- 172.21.15.16:0/2650049249 learned_addr learned my addr 172.21.15.16:0/2650049249 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f9792d9d640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9794108ac0 0x7f979419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f979259c640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9794102880 0x7f979419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f9791d9b640 1 -- 172.21.15.16:0/2650049249 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9794108ac0 msgr2=0x7f979419fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f9791d9b640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9794108ac0 0x7f979419fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f9791d9b640 1 -- 172.21.15.16:0/2650049249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9794102880 msgr2=0x7f979419a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f9791d9b640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9794102880 0x7f979419a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f9791d9b640 1 -- 172.21.15.16:0/2650049249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97941a02a0 con 0x7f9794108170 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f9791d9b640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9794108170 0x7f979419a8d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f978802a930 tx=0x7f9788004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f97837fe640 1 -- 172.21.15.16:0/2650049249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978803f030 con 0x7f9794108170 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f97837fe640 1 -- 172.21.15.16:0/2650049249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9788003aa0 con 0x7f9794108170 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f9798821640 1 -- 172.21.15.16:0/2650049249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97941a0530 con 0x7f9794108170 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f97837fe640 1 -- 172.21.15.16:0/2650049249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9788035070 con 0x7f9794108170 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.252+0000 7f9798821640 1 -- 172.21.15.16:0/2650049249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97941a6cf0 con 0x7f9794108170 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f979ed96740 1 Processor -- start 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f979ed96740 1 -- start start 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f979ed96740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d4304717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f97837fe640 1 -- 172.21.15.16:0/2650049249 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7f9788042030 con 0x7f9794108170 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f979ed96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d430471cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f979ed96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300ab280 0x55d430472230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f979ed96740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d4303e83f0 con 0x55d4300a30a0 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f979ed96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d4303d7980 con 0x55d4300ab280 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f979ed96740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d4303db930 con 0x55d4300cbf60 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f979259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d430471cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f979259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d430471cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44322/0 (socket says 172.21.15.16:44322) 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.255+0000 7f979259c640 1 -- 172.21.15.16:0/1678473909 learned_addr learned my addr 172.21.15.16:0/1678473909 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f9791d9b640 1 --2- 172.21.15.16:0/1678473909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300ab280 0x55d430472230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f9792d9d640 1 --2- 172.21.15.16:0/1678473909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d4304717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f979259c640 1 -- 172.21.15.16:0/1678473909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 msgr2=0x55d4304717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f979259c640 1 --2- 172.21.15.16:0/1678473909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d4304717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f979259c640 1 -- 172.21.15.16:0/1678473909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300ab280 msgr2=0x55d430472230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f979259c640 1 --2- 172.21.15.16:0/1678473909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300ab280 0x55d430472230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f979259c640 1 -- 172.21.15.16:0/1678473909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d430137540 con 0x55d4300a30a0 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f979259c640 1 --2- 172.21.15.16:0/1678473909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d430471cf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f978400b820 tx=0x7f978400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b31130b39222bf server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f97837fe640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f977c0b4b80 0x7f977c0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f9792d9d640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f977c0b4b80 0x7f977c0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f9780ff9640 1 -- 172.21.15.16:0/1678473909 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978400d8f0 con 0x55d4300a30a0 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f97837fe640 1 -- 172.21.15.16:0/2650049249 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f9788105d40 con 0x7f9794108170 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f9780ff9640 1 -- 172.21.15.16:0/1678473909 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9784019070 con 0x55d4300a30a0 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.256+0000 7f9780ff9640 1 -- 172.21.15.16:0/1678473909 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97840123e0 con 0x55d4300a30a0 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.257+0000 7f979ed96740 1 -- 172.21.15.16:0/1678473909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 msgr2=0x55d430471cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.257+0000 7f979ed96740 1 --2- 172.21.15.16:0/1678473909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d430471cf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f978400b820 tx=0x7f978400bce0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.257+0000 7f9792d9d640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f977c0b4b80 0x7f977c0b7090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f978c005fd0 tx=0x7f978c009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.257+0000 7f979ed96740 1 -- 172.21.15.16:0/1678473909 shutdown_connections 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.257+0000 7f979ed96740 1 --2- 172.21.15.16:0/1678473909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300ab280 0x55d430472230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.257+0000 7f979ed96740 1 --2- 172.21.15.16:0/1678473909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d430471cf0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.257+0000 7f979ed96740 1 --2- 172.21.15.16:0/1678473909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d4304717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.257+0000 7f979ed96740 1 -- 172.21.15.16:0/1678473909 >> 172.21.15.16:0/1678473909 conn(0x55d4300c0350 msgr2=0x55d43046f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.257+0000 7f979ed96740 1 -- 172.21.15.16:0/1678473909 shutdown_connections 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.257+0000 7f979ed96740 1 -- 172.21.15.16:0/1678473909 wait complete. 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f979ed96740 1 -- start start 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f979ed96740 1 Processor -- start 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f979ed96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300a30a0 0x55d430481250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f979ed96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430481790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f979ed96740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d43048d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f979ed96740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d43023a5f0 con 0x55d4300ab280 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f979ed96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d43035e600 con 0x55d4300a30a0 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f979ed96740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d4304842f0 con 0x55d4300cbf60 2024-08-28T19:20:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f979ed96740 10 client.0 _ll_register_callbacks cb 0x55d430384af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f9791d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430481790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f9791d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430481790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44334/0 (socket says 172.21.15.16:44334) 2024-08-28T19:20:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f9791d9b640 1 -- 172.21.15.16:0/59861692 learned_addr learned my addr 172.21.15.16:0/59861692 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f979259c640 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300a30a0 0x55d430481250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.258+0000 7f9792d9d640 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d43048d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9791d9b640 1 -- 172.21.15.16:0/59861692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 msgr2=0x55d43048d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9791d9b640 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d43048d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9791d9b640 1 -- 172.21.15.16:0/59861692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300a30a0 msgr2=0x55d430481250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9791d9b640 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300a30a0 0x55d430481250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9791d9b640 1 -- 172.21.15.16:0/59861692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4303977b0 con 0x55d4300ab280 2024-08-28T19:20:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9792d9d640 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d43048d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9791d9b640 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430481790 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9788031440 tx=0x7f978803b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aed546f9ad1d0c7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9779ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9779ffb640 1 -- 172.21.15.16:0/59861692 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9788049050 con 0x55d4300ab280 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f979ed96740 1 -- 172.21.15.16:0/59861692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9784019070 con 0x55d4300ab280 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9779ffb640 1 -- 172.21.15.16:0/59861692 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f978803bdc0 con 0x55d4300ab280 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9779ffb640 1 -- 172.21.15.16:0/59861692 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f97880cc9e0 con 0x55d4300ab280 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f979ed96740 10 client.15435 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f979ed96740 1 -- 172.21.15.16:0/59861692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d430490010 con 0x55d4300ab280 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.259+0000 7f9779ffb640 1 -- 172.21.15.16:0/59861692 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9784019070 con 0x55d4300ab280 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.260+0000 7f9779ffb640 1 -- 172.21.15.16:0/59861692 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f97880c8570 con 0x55d4300ab280 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.260+0000 7f979ed96740 10 client.15435 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.260+0000 7f979ed96740 10 client.15435 resolve_mds: resolved name 'f' to daemon mds.f 2024-08-28T19:20:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.260+0000 7f979ed96740 20 client.15435 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.260+0000 7f979ed96740 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d43008a8e0 0x55d430493a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.260+0000 7f979ed96740 4 client.15435 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:20:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.260+0000 7f979ed96740 1 -- 172.21.15.16:0/59861692 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4303ea340 con 0x55d43008a8e0 2024-08-28T19:20:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.260+0000 7f979259c640 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d43008a8e0 0x55d430493a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.261+0000 7f979259c640 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d43008a8e0 0x55d430493a40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ca7a772f3268ee0 server_cookie=7b2c7312d9bdc315 in_seq=0 out_seq=0 2024-08-28T19:20:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.261+0000 7f9779ffb640 10 client.15435 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:20:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.261+0000 7f9779ffb640 1 -- 172.21.15.16:0/59861692 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d4303ea340 con 0x55d43008a8e0 2024-08-28T19:20:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.261+0000 7f9779ffb640 10 client.15435 handle_command_reply: tid=0 2024-08-28T19:20:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.261+0000 7f979ed96740 1 client.15435 shutdown 2024-08-28T19:20:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.261+0000 7f979ed96740 1 -- 172.21.15.16:0/59861692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 msgr2=0x55d430481790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.261+0000 7f979ed96740 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430481790 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9788031440 tx=0x7f978803b6b0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.262+0000 7f979ed96740 1 -- 172.21.15.16:0/59861692 shutdown_connections 2024-08-28T19:20:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.262+0000 7f979ed96740 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d43008a8e0 0x55d430493a40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.262+0000 7f979ed96740 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d43048d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.262+0000 7f979ed96740 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430481790 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.262+0000 7f979ed96740 1 --2- 172.21.15.16:0/59861692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300a30a0 0x55d430481250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.262+0000 7f9779ffb640 0 client.15435 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:20:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.262+0000 7f979ed96740 1 -- 172.21.15.16:0/59861692 >> 172.21.15.16:0/59861692 conn(0x55d4300c0350 msgr2=0x55d43046dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.262+0000 7f979ed96740 1 -- 172.21.15.16:0/59861692 shutdown_connections 2024-08-28T19:20:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.262+0000 7f979ed96740 1 -- 172.21.15.16:0/59861692 wait complete. 2024-08-28T19:20:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.262+0000 7f979ed96740 20 client.15435 trim_cache size 0 max 16384 2024-08-28T19:20:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f979ed96740 1 Processor -- start 2024-08-28T19:20:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f979ed96740 1 -- start start 2024-08-28T19:20:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f979ed96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d4304811c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f979ed96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300ab280 0x55d430481700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f979ed96740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d43047c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f979ed96740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d430491d80 con 0x55d4300a30a0 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f979ed96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d430492450 con 0x55d4300ab280 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f979ed96740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d430492b20 con 0x55d4300cbf60 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f979259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d4304811c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f979259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d4304811c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33238/0 (socket says 172.21.15.16:33238) 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f979259c640 1 -- 172.21.15.16:0/3457066349 learned_addr learned my addr 172.21.15.16:0/3457066349 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.280+0000 7f9791d9b640 1 --2- 172.21.15.16:0/3457066349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300ab280 0x55d430481700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.281+0000 7f9792d9d640 1 --2- 172.21.15.16:0/3457066349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d43047c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.281+0000 7f979259c640 1 -- 172.21.15.16:0/3457066349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 msgr2=0x55d43047c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.281+0000 7f979259c640 1 --2- 172.21.15.16:0/3457066349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d43047c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.281+0000 7f979259c640 1 -- 172.21.15.16:0/3457066349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300ab280 msgr2=0x55d430481700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.281+0000 7f979259c640 1 --2- 172.21.15.16:0/3457066349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300ab280 0x55d430481700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.281+0000 7f979259c640 1 -- 172.21.15.16:0/3457066349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4304740c0 con 0x55d4300a30a0 2024-08-28T19:20:44.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.281+0000 7f979259c640 1 --2- 172.21.15.16:0/3457066349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d4304811c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x55d430470430 tx=0x7f9784008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390d8758ea2c688c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:44.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.281+0000 7f977bfff640 1 -- 172.21.15.16:0/3457066349 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784014a60 con 0x55d4300a30a0 2024-08-28T19:20:44.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.281+0000 7f977bfff640 1 -- 172.21.15.16:0/3457066349 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9784019040 con 0x55d4300a30a0 2024-08-28T19:20:44.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.281+0000 7f977bfff640 1 -- 172.21.15.16:0/3457066349 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97840275e0 con 0x55d4300a30a0 2024-08-28T19:20:44.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.282+0000 7f979ed96740 1 -- 172.21.15.16:0/3457066349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 msgr2=0x55d4304811c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:44.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.282+0000 7f979ed96740 1 --2- 172.21.15.16:0/3457066349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d4304811c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x55d430470430 tx=0x7f9784008040 comp rx=0 tx=0).stop 2024-08-28T19:20:44.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.282+0000 7f979ed96740 1 -- 172.21.15.16:0/3457066349 shutdown_connections 2024-08-28T19:20:44.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.282+0000 7f979ed96740 1 --2- 172.21.15.16:0/3457066349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300cbf60 0x55d43047c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.282+0000 7f979ed96740 1 --2- 172.21.15.16:0/3457066349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300ab280 0x55d430481700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.282+0000 7f979ed96740 1 --2- 172.21.15.16:0/3457066349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300a30a0 0x55d4304811c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.282+0000 7f979ed96740 1 -- 172.21.15.16:0/3457066349 >> 172.21.15.16:0/3457066349 conn(0x55d4300c0350 msgr2=0x55d43046a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.282+0000 7f979ed96740 1 -- 172.21.15.16:0/3457066349 shutdown_connections 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.282+0000 7f979ed96740 1 -- 172.21.15.16:0/3457066349 wait complete. 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.282+0000 7f979ed96740 1 -- start start 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.282+0000 7f979ed96740 1 Processor -- start 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f979ed96740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300a30a0 0x55d430497590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f979ed96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430497b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f979ed96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300cbf60 0x55d4304f6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f979ed96740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d4304938c0 con 0x55d4300ab280 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f979ed96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d4304931f0 con 0x55d4300cbf60 2024-08-28T19:20:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f979ed96740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d430493f90 con 0x55d4300a30a0 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f979ed96740 10 client.0 _ll_register_callbacks cb 0x7f97880c81a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f9791d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430497b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f9791d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430497b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33244/0 (socket says 172.21.15.16:33244) 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f9791d9b640 1 -- 172.21.15.16:0/3388586161 learned_addr learned my addr 172.21.15.16:0/3388586161 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f9792d9d640 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300cbf60 0x55d4304f6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f979259c640 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300a30a0 0x55d430497590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f9791d9b640 1 -- 172.21.15.16:0/3388586161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300a30a0 msgr2=0x55d430497590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f9791d9b640 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300a30a0 0x55d430497590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f9791d9b640 1 -- 172.21.15.16:0/3388586161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300cbf60 msgr2=0x55d4304f6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f9791d9b640 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300cbf60 0x55d4304f6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f9791d9b640 1 -- 172.21.15.16:0/3388586161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4304f7540 con 0x55d4300ab280 2024-08-28T19:20:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f9791d9b640 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430497b50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9788103720 tx=0x7f97880cc0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f2d361985db294 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.283+0000 7f9779ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f9779ffb640 1 -- 172.21.15.16:0/3388586161 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9788049070 con 0x55d4300ab280 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f9779ffb640 1 -- 172.21.15.16:0/3388586161 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f97880d5170 con 0x55d4300ab280 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f979ed96740 1 -- 172.21.15.16:0/3388586161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9784019040 con 0x55d4300ab280 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f9779ffb640 1 -- 172.21.15.16:0/3388586161 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f97880cc750 con 0x55d4300ab280 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f979ed96740 10 client.15441 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f979ed96740 1 -- 172.21.15.16:0/3388586161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d4304fe840 con 0x55d4300ab280 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f9779ffb640 1 -- 172.21.15.16:0/3388586161 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9784019040 con 0x55d4300ab280 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f9779ffb640 1 -- 172.21.15.16:0/3388586161 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f97880cb540 con 0x55d4300ab280 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f979ed96740 10 client.15441 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f979ed96740 10 client.15441 resolve_mds: resolved name 'f' to daemon mds.f 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f979ed96740 20 client.15441 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f979ed96740 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d43005fc90 0x55d430501ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:44.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f979ed96740 4 client.15441 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:20:44.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.284+0000 7f979ed96740 1 -- 172.21.15.16:0/3388586161 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "status"}) -- 0x55d4303e5de0 con 0x55d43005fc90 2024-08-28T19:20:44.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.285+0000 7f979259c640 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d43005fc90 0x55d430501ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:44.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.285+0000 7f9779ffb640 10 client.15441 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:20:44.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.285+0000 7f979259c640 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d43005fc90 0x55d430501ec0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=686e660f2e24e0b3 server_cookie=b7709565a3012ad7 in_seq=0 out_seq=0 2024-08-28T19:20:44.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.285+0000 7f9779ffb640 1 -- 172.21.15.16:0/3388586161 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55d4303e5de0 con 0x55d43005fc90 2024-08-28T19:20:44.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.285+0000 7f9779ffb640 10 client.15441 handle_command_reply: tid=0 2024-08-28T19:20:44.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.285+0000 7f979ed96740 1 client.15441 shutdown 2024-08-28T19:20:44.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f979ed96740 1 -- 172.21.15.16:0/3388586161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 msgr2=0x55d430497b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f979ed96740 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430497b50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9788103720 tx=0x7f97880cc0b0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f979ed96740 1 -- 172.21.15.16:0/3388586161 shutdown_connections 2024-08-28T19:20:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f979ed96740 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d43005fc90 0x55d430501ec0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f979ed96740 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4300cbf60 0x55d4304f6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f979ed96740 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4300ab280 0x55d430497b50 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f979ed96740 1 --2- 172.21.15.16:0/3388586161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4300a30a0 0x55d430497590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f9779ffb640 0 client.15441 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:20:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f979ed96740 1 -- 172.21.15.16:0/3388586161 >> 172.21.15.16:0/3388586161 conn(0x55d4300c0350 msgr2=0x55d4304811c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f979ed96740 1 -- 172.21.15.16:0/3388586161 shutdown_connections 2024-08-28T19:20:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f979ed96740 1 -- 172.21.15.16:0/3388586161 wait complete. 2024-08-28T19:20:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f979ed96740 20 client.15441 trim_cache size 0 max 16384 2024-08-28T19:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-08-28T19:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24619, 2024-08-28T19:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 92.644760925, 2024-08-28T19:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-08-28T19:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 82, 2024-08-28T19:20:44.294 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 157.71265803099999 2024-08-28T19:20:44.294 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:20:44.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f9798821640 1 -- 172.21.15.16:0/2650049249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f977c0b4b80 msgr2=0x7f977c0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:44.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f9798821640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f977c0b4b80 0x7f977c0b7090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f978c005fd0 tx=0x7f978c009600 comp rx=0 tx=0).stop 2024-08-28T19:20:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f9798821640 1 -- 172.21.15.16:0/2650049249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9794108170 msgr2=0x7f979419a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.286+0000 7f9798821640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9794108170 0x7f979419a8d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f978802a930 tx=0x7f9788004270 comp rx=0 tx=0).stop 2024-08-28T19:20:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.287+0000 7f9798821640 1 -- 172.21.15.16:0/2650049249 shutdown_connections 2024-08-28T19:20:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.287+0000 7f9798821640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9794108ac0 0x7f979419fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.287+0000 7f9798821640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9794108170 0x7f979419a8d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.287+0000 7f9798821640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f977c0b4b80 0x7f977c0b7090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.287+0000 7f9798821640 1 --2- 172.21.15.16:0/2650049249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9794102880 0x7f979419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.287+0000 7f9798821640 1 -- 172.21.15.16:0/2650049249 >> 172.21.15.16:0/2650049249 conn(0x7f9794079550 msgr2=0x7f9794106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.287+0000 7f9798821640 1 -- 172.21.15.16:0/2650049249 shutdown_connections 2024-08-28T19:20:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:44.287+0000 7f9798821640 1 -- 172.21.15.16:0/2650049249 wait complete. 2024-08-28T19:20:44.960 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.c status 2024-08-28T19:20:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:45.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:45 smithi016 ceph-mon[18556]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-08-28T19:20:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:45 smithi028 ceph-mon[21769]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-08-28T19:20:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:45 smithi115 ceph-mon[22184]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-08-28T19:20:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.718+0000 7f98bb583640 1 -- 172.21.15.16:0/3571927539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b4109b90 msgr2=0x7f98b41103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.718+0000 7f98bb583640 1 --2- 172.21.15.16:0/3571927539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b4109b90 0x7f98b41103c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009ca0 tx=0x7f98ac02a8e0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.718+0000 7f98bb583640 1 -- 172.21.15.16:0/3571927539 shutdown_connections 2024-08-28T19:20:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.719+0000 7f98bb583640 1 --2- 172.21.15.16:0/3571927539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b4109b90 0x7f98b41103c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.719+0000 7f98bb583640 1 --2- 172.21.15.16:0/3571927539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b4109240 0x7f98b4109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.719+0000 7f98bb583640 1 --2- 172.21.15.16:0/3571927539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b4103900 0x7f98b4103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.719+0000 7f98bb583640 1 -- 172.21.15.16:0/3571927539 >> 172.21.15.16:0/3571927539 conn(0x7f98b407b5f0 msgr2=0x7f98b407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.719+0000 7f98bb583640 1 -- 172.21.15.16:0/3571927539 shutdown_connections 2024-08-28T19:20:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.719+0000 7f98bb583640 1 -- 172.21.15.16:0/3571927539 wait complete. 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98bb583640 1 Processor -- start 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98bb583640 1 -- start start 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98bb583640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b4103900 0x7f98b419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98bb583640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b4109240 0x7f98b419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98bb583640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b4109b90 0x7f98b419fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98bb583640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f98b4113890 con 0x7f98b4109b90 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98bb583640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98b4113710 con 0x7f98b4103900 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98bb583640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f98b4113a10 con 0x7f98b4109240 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98b9b20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b4109b90 0x7f98b419fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98b9b20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b4109b90 0x7f98b419fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33262/0 (socket says 172.21.15.16:33262) 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98b9b20640 1 -- 172.21.15.16:0/1876303628 learned_addr learned my addr 172.21.15.16:0/1876303628 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98b931f640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b4103900 0x7f98b419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98b8b1e640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b4109240 0x7f98b419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98b9b20640 1 -- 172.21.15.16:0/1876303628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b4109240 msgr2=0x7f98b419a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98b9b20640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b4109240 0x7f98b419a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98b9b20640 1 -- 172.21.15.16:0/1876303628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b4103900 msgr2=0x7f98b419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98b9b20640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b4103900 0x7f98b419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98b9b20640 1 -- 172.21.15.16:0/1876303628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b41a01f0 con 0x7f98b4109b90 2024-08-28T19:20:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.720+0000 7f98b8b1e640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b4109240 0x7f98b419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.721+0000 7f98b9b20640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b4109b90 0x7f98b419fac0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f98ac005cc0 tx=0x7f98ac003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.721+0000 7f98aa7fc640 1 -- 172.21.15.16:0/1876303628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac004040 con 0x7f98b4109b90 2024-08-28T19:20:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.721+0000 7f98aa7fc640 1 -- 172.21.15.16:0/1876303628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ac004690 con 0x7f98b4109b90 2024-08-28T19:20:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.721+0000 7f98bb583640 1 -- 172.21.15.16:0/1876303628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b41a0480 con 0x7f98b4109b90 2024-08-28T19:20:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.721+0000 7f98aa7fc640 1 -- 172.21.15.16:0/1876303628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac030e70 con 0x7f98b4109b90 2024-08-28T19:20:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.721+0000 7f98bb583640 1 -- 172.21.15.16:0/1876303628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b41a6c40 con 0x7f98b4109b90 2024-08-28T19:20:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98c1af8740 1 Processor -- start 2024-08-28T19:20:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98aa7fc640 1 -- 172.21.15.16:0/1876303628 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7f98ac0305e0 con 0x7f98b4109b90 2024-08-28T19:20:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98c1af8740 1 -- start start 2024-08-28T19:20:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803cbcf60 0x55e8040627b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e804062cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e804063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98c1af8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e803fd93f0 con 0x55e803c940a0 2024-08-28T19:20:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98c1af8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e803fc8980 con 0x55e803cbcf60 2024-08-28T19:20:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98c1af8740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e803fcc930 con 0x55e803c9c280 2024-08-28T19:20:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98b931f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e804062cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98b931f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e804062cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33278/0 (socket says 172.21.15.16:33278) 2024-08-28T19:20:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98b931f640 1 -- 172.21.15.16:0/4113442122 learned_addr learned my addr 172.21.15.16:0/4113442122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98b8b1e640 1 --2- 172.21.15.16:0/4113442122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803cbcf60 0x55e8040627b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.724+0000 7f98b8b1e640 1 --2- 172.21.15.16:0/4113442122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e804063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f98b931f640 1 -- 172.21.15.16:0/4113442122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 msgr2=0x55e804063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f98b931f640 1 --2- 172.21.15.16:0/4113442122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e804063230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f98b931f640 1 -- 172.21.15.16:0/4113442122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803cbcf60 msgr2=0x55e8040627b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f98b931f640 1 --2- 172.21.15.16:0/4113442122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803cbcf60 0x55e8040627b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f98b931f640 1 -- 172.21.15.16:0/4113442122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e803d28540 con 0x55e803c940a0 2024-08-28T19:20:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f98aa7fc640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f98a00b4c00 0x7f98a00b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f98b8b1e640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f98a00b4c00 0x7f98a00b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f98b931f640 1 --2- 172.21.15.16:0/4113442122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e804062cf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f98b000ac20 tx=0x7f98b000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963c24c4b435d598 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f98aa7fc640 1 -- 172.21.15.16:0/1876303628 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f98ac1069e0 con 0x7f98b4109b90 2024-08-28T19:20:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f989f7fe640 1 -- 172.21.15.16:0/4113442122 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b001d070 con 0x55e803c940a0 2024-08-28T19:20:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f989f7fe640 1 -- 172.21.15.16:0/4113442122 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98b0018af0 con 0x55e803c940a0 2024-08-28T19:20:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.725+0000 7f989f7fe640 1 -- 172.21.15.16:0/4113442122 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b0010ab0 con 0x55e803c940a0 2024-08-28T19:20:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.726+0000 7f98c1af8740 1 -- 172.21.15.16:0/4113442122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 msgr2=0x55e804062cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.726+0000 7f98b8b1e640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f98a00b4c00 0x7f98a00b7110 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98a4006fd0 tx=0x7f98a4008040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.726+0000 7f98c1af8740 1 --2- 172.21.15.16:0/4113442122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e804062cf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f98b000ac20 tx=0x7f98b000f790 comp rx=0 tx=0).stop 2024-08-28T19:20:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.726+0000 7f98c1af8740 1 -- 172.21.15.16:0/4113442122 shutdown_connections 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.726+0000 7f98c1af8740 1 --2- 172.21.15.16:0/4113442122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e804063230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.726+0000 7f98c1af8740 1 --2- 172.21.15.16:0/4113442122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e804062cf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.726+0000 7f98c1af8740 1 --2- 172.21.15.16:0/4113442122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803cbcf60 0x55e8040627b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.726+0000 7f98c1af8740 1 -- 172.21.15.16:0/4113442122 >> 172.21.15.16:0/4113442122 conn(0x55e803cb1350 msgr2=0x55e8040606a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.726+0000 7f98c1af8740 1 -- 172.21.15.16:0/4113442122 shutdown_connections 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.726+0000 7f98c1af8740 1 -- 172.21.15.16:0/4113442122 wait complete. 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.726+0000 7f98c1af8740 1 -- start start 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98c1af8740 1 Processor -- start 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e80406de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e80406e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803cbcf60 0x55e80407a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98c1af8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e803e2b5f0 con 0x55e803c940a0 2024-08-28T19:20:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98c1af8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e803f4f600 con 0x55e803cbcf60 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98c1af8740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e804070f90 con 0x55e803c9c280 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98c1af8740 10 client.0 _ll_register_callbacks cb 0x55e803f75af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98b931f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e80406de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98b931f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e80406de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33290/0 (socket says 172.21.15.16:33290) 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98b931f640 1 -- 172.21.15.16:0/2938569286 learned_addr learned my addr 172.21.15.16:0/2938569286 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98b9b20640 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803cbcf60 0x55e80407a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98b8b1e640 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e80406e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98b931f640 1 -- 172.21.15.16:0/2938569286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 msgr2=0x55e80406e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98b931f640 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e80406e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98b931f640 1 -- 172.21.15.16:0/2938569286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803cbcf60 msgr2=0x55e80407a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.727+0000 7f98b931f640 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803cbcf60 0x55e80407a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f98b931f640 1 -- 172.21.15.16:0/2938569286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e803f887b0 con 0x55e803c940a0 2024-08-28T19:20:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f98b931f640 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e80406de90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f98b001b040 tx=0x7f98b00186f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e7ad469aa57aac server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f989cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f989cff9640 1 -- 172.21.15.16:0/2938569286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b001d050 con 0x55e803c940a0 2024-08-28T19:20:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f989cff9640 1 -- 172.21.15.16:0/2938569286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f98b0005240 con 0x55e803c940a0 2024-08-28T19:20:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f98c1af8740 1 -- 172.21.15.16:0/2938569286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98b0018af0 con 0x55e803c940a0 2024-08-28T19:20:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f989cff9640 1 -- 172.21.15.16:0/2938569286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f98b0032050 con 0x55e803c940a0 2024-08-28T19:20:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f98c1af8740 10 client.15453 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f98c1af8740 1 -- 172.21.15.16:0/2938569286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e80407cc90 con 0x55e803c940a0 2024-08-28T19:20:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f989cff9640 1 -- 172.21.15.16:0/2938569286 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98b0018af0 con 0x55e803c940a0 2024-08-28T19:20:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f989cff9640 1 -- 172.21.15.16:0/2938569286 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f98b00046c0 con 0x55e803c940a0 2024-08-28T19:20:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f98c1af8740 10 client.15453 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.728+0000 7f98c1af8740 10 client.15453 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-28T19:20:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.729+0000 7f98c1af8740 20 client.15453 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.729+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e803c7b8e0 0x55e8040802e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.729+0000 7f98c1af8740 4 client.15453 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:20:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.729+0000 7f98c1af8740 1 -- 172.21.15.16:0/2938569286 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e803fdb340 con 0x55e803c7b8e0 2024-08-28T19:20:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.729+0000 7f98b8b1e640 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e803c7b8e0 0x55e8040802e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.729+0000 7f989cff9640 10 client.15453 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.729+0000 7f98b8b1e640 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e803c7b8e0 0x55e8040802e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1f0a3163b038d466 server_cookie=5a8d2dbf6b399b56 in_seq=0 out_seq=0 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.730+0000 7f989cff9640 1 -- 172.21.15.16:0/2938569286 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e803fdb340 con 0x55e803c7b8e0 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.730+0000 7f989cff9640 10 client.15453 handle_command_reply: tid=0 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.730+0000 7f98c1af8740 1 client.15453 shutdown 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.730+0000 7f98c1af8740 1 -- 172.21.15.16:0/2938569286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 msgr2=0x55e80406de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.730+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e80406de90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f98b001b040 tx=0x7f98b00186f0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.731+0000 7f98c1af8740 1 -- 172.21.15.16:0/2938569286 shutdown_connections 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.731+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e803c7b8e0 0x55e8040802e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.731+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803cbcf60 0x55e80407a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.731+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e80406e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.731+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2938569286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803c940a0 0x55e80406de90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.731+0000 7f989cff9640 0 client.15453 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:20:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.731+0000 7f98c1af8740 1 -- 172.21.15.16:0/2938569286 >> 172.21.15.16:0/2938569286 conn(0x55e803cb1350 msgr2=0x55e804065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.731+0000 7f98c1af8740 1 -- 172.21.15.16:0/2938569286 shutdown_connections 2024-08-28T19:20:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.731+0000 7f98c1af8740 1 -- 172.21.15.16:0/2938569286 wait complete. 2024-08-28T19:20:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.731+0000 7f98c1af8740 20 client.15453 trim_cache size 0 max 16384 2024-08-28T19:20:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.749+0000 7f98c1af8740 1 Processor -- start 2024-08-28T19:20:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.749+0000 7f98c1af8740 1 -- start start 2024-08-28T19:20:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.749+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c940a0 0x55e80407a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.749+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803c9c280 0x55e80406dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.749+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e80406e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.749+0000 7f98c1af8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e80407e620 con 0x55e803cbcf60 2024-08-28T19:20:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.749+0000 7f98c1af8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e80407ecf0 con 0x55e803c9c280 2024-08-28T19:20:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.749+0000 7f98c1af8740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e80407f3c0 con 0x55e803c940a0 2024-08-28T19:20:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.749+0000 7f98b9b20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e80406e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f98b9b20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e80406e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33306/0 (socket says 172.21.15.16:33306) 2024-08-28T19:20:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f98b9b20640 1 -- 172.21.15.16:0/3975467876 learned_addr learned my addr 172.21.15.16:0/3975467876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f98b8b1e640 1 --2- 172.21.15.16:0/3975467876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803c9c280 0x55e80406dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f98b931f640 1 --2- 172.21.15.16:0/3975467876 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c940a0 0x55e80407a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f98b9b20640 1 -- 172.21.15.16:0/3975467876 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c940a0 msgr2=0x55e80407a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:45.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f98b9b20640 1 --2- 172.21.15.16:0/3975467876 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c940a0 0x55e80407a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f98b9b20640 1 -- 172.21.15.16:0/3975467876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803c9c280 msgr2=0x55e80406dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:45.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f98b9b20640 1 --2- 172.21.15.16:0/3975467876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803c9c280 0x55e80406dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f98b9b20640 1 -- 172.21.15.16:0/3975467876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e804060f90 con 0x55e803cbcf60 2024-08-28T19:20:45.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f98b9b20640 1 --2- 172.21.15.16:0/3975467876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e80406e180 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f98ac1043c0 tx=0x7f98ac003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceeba2cce46f5aff server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:45.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f989effd640 1 -- 172.21.15.16:0/3975467876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0cca90 con 0x55e803cbcf60 2024-08-28T19:20:45.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f989effd640 1 -- 172.21.15.16:0/3975467876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ac036040 con 0x55e803cbcf60 2024-08-28T19:20:45.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.750+0000 7f989effd640 1 -- 172.21.15.16:0/3975467876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0d5e10 con 0x55e803cbcf60 2024-08-28T19:20:45.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.751+0000 7f98c1af8740 1 -- 172.21.15.16:0/3975467876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 msgr2=0x55e80406e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:45.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.751+0000 7f98c1af8740 1 --2- 172.21.15.16:0/3975467876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e80406e180 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f98ac1043c0 tx=0x7f98ac003f90 comp rx=0 tx=0).stop 2024-08-28T19:20:45.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.751+0000 7f98c1af8740 1 -- 172.21.15.16:0/3975467876 shutdown_connections 2024-08-28T19:20:45.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.751+0000 7f98c1af8740 1 --2- 172.21.15.16:0/3975467876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e80406e180 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.751+0000 7f98c1af8740 1 --2- 172.21.15.16:0/3975467876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803c9c280 0x55e80406dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.751+0000 7f98c1af8740 1 --2- 172.21.15.16:0/3975467876 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c940a0 0x55e80407a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.751+0000 7f98c1af8740 1 -- 172.21.15.16:0/3975467876 >> 172.21.15.16:0/3975467876 conn(0x55e803cb1350 msgr2=0x55e80406c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:45.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.751+0000 7f98c1af8740 1 -- 172.21.15.16:0/3975467876 shutdown_connections 2024-08-28T19:20:45.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.751+0000 7f98c1af8740 1 -- 172.21.15.16:0/3975467876 wait complete. 2024-08-28T19:20:45.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.751+0000 7f98c1af8740 1 -- start start 2024-08-28T19:20:45.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98c1af8740 1 Processor -- start 2024-08-28T19:20:45.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803c940a0 0x55e804088e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e804082340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98c1af8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e804082990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98c1af8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e804080160 con 0x55e803cbcf60 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98c1af8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e80407fa90 con 0x55e803c940a0 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98c1af8740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e804080830 con 0x55e803c9c280 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98c1af8740 10 client.0 _ll_register_callbacks cb 0x7f98b0005dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98b9b20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e804082990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98b9b20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e804082990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33314/0 (socket says 172.21.15.16:33314) 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98b9b20640 1 -- 172.21.15.16:0/2788643063 learned_addr learned my addr 172.21.15.16:0/2788643063 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98b931f640 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803c940a0 0x55e804088e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.752+0000 7f98b8b1e640 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e804082340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f98b9b20640 1 -- 172.21.15.16:0/2788643063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 msgr2=0x55e804082340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f98b9b20640 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e804082340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f98b9b20640 1 -- 172.21.15.16:0/2788643063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803c940a0 msgr2=0x55e804088e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f98b9b20640 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803c940a0 0x55e804088e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f98b9b20640 1 -- 172.21.15.16:0/2788643063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e804083100 con 0x55e803cbcf60 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f98b931f640 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803c940a0 0x55e804088e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f98b9b20640 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e804082990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f98ac006010 tx=0x7f98ac0c94b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f8a862420d2851 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f989cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f989cff9640 1 -- 172.21.15.16:0/2788643063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0d62a0 con 0x55e803cbcf60 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f989cff9640 1 -- 172.21.15.16:0/2788643063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f98ac0d5e10 con 0x55e803cbcf60 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f98c1af8740 1 -- 172.21.15.16:0/2788643063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98ac036040 con 0x55e803cbcf60 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f989cff9640 1 -- 172.21.15.16:0/2788643063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f98ac0cd680 con 0x55e803cbcf60 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f98c1af8740 10 client.15459 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f98c1af8740 1 -- 172.21.15.16:0/2788643063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e8040eb7b0 con 0x55e803cbcf60 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.753+0000 7f989cff9640 1 -- 172.21.15.16:0/2788643063 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ac036040 con 0x55e803cbcf60 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.754+0000 7f989cff9640 1 -- 172.21.15.16:0/2788643063 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f98ac004b40 con 0x55e803cbcf60 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.754+0000 7f98c1af8740 10 client.15459 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.754+0000 7f98c1af8740 10 client.15459 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-28T19:20:45.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.754+0000 7f98c1af8740 20 client.15459 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:45.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.754+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e803c50c90 0x55e8040eeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:45.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.754+0000 7f98c1af8740 4 client.15459 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:20:45.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.754+0000 7f98c1af8740 1 -- 172.21.15.16:0/2788643063 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "status"}) -- 0x55e803fd6de0 con 0x55e803c50c90 2024-08-28T19:20:45.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.754+0000 7f98b931f640 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e803c50c90 0x55e8040eeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:45.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.754+0000 7f98b931f640 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e803c50c90 0x55e8040eeee0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=39ef6694c0b2a4cc server_cookie=a6281d708eabcba5 in_seq=0 out_seq=0 2024-08-28T19:20:45.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.754+0000 7f989cff9640 10 client.15459 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:20:45.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f989cff9640 1 -- 172.21.15.16:0/2788643063 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e803fd6de0 con 0x55e803c50c90 2024-08-28T19:20:45.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f989cff9640 10 client.15459 handle_command_reply: tid=0 2024-08-28T19:20:45.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 1 client.15459 shutdown 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 1 -- 172.21.15.16:0/2788643063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 msgr2=0x55e804082990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e804082990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f98ac006010 tx=0x7f98ac0c94b0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 1 -- 172.21.15.16:0/2788643063 shutdown_connections 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e803c50c90 0x55e8040eeee0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e803cbcf60 0x55e804082990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e803c9c280 0x55e804082340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 1 --2- 172.21.15.16:0/2788643063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e803c940a0 0x55e804088e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 1 -- 172.21.15.16:0/2788643063 >> 172.21.15.16:0/2788643063 conn(0x55e803cb1350 msgr2=0x55e8040637e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f989cff9640 0 client.15459 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 1 -- 172.21.15.16:0/2788643063 shutdown_connections 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 1 -- 172.21.15.16:0/2788643063 wait complete. 2024-08-28T19:20:45.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.755+0000 7f98c1af8740 20 client.15459 trim_cache size 0 max 16384 2024-08-28T19:20:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:20:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:20:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-08-28T19:20:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24625, 2024-08-28T19:20:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:20:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:20:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:20:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 95.124176316000003, 2024-08-28T19:20:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-08-28T19:20:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:20:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 82, 2024-08-28T19:20:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 156.692976423 2024-08-28T19:20:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:20:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 -- 172.21.15.16:0/1876303628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f98a00b4c00 msgr2=0x7f98a00b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f98a00b4c00 0x7f98a00b7110 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98a4006fd0 tx=0x7f98a4008040 comp rx=0 tx=0).stop 2024-08-28T19:20:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 -- 172.21.15.16:0/1876303628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b4109b90 msgr2=0x7f98b419fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b4109b90 0x7f98b419fac0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f98ac005cc0 tx=0x7f98ac003ba0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 -- 172.21.15.16:0/1876303628 shutdown_connections 2024-08-28T19:20:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b4109b90 0x7f98b419fac0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b4109240 0x7f98b419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f98a00b4c00 0x7f98a00b7110 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 --2- 172.21.15.16:0/1876303628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b4103900 0x7f98b419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:45.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 -- 172.21.15.16:0/1876303628 >> 172.21.15.16:0/1876303628 conn(0x7f98b407b5f0 msgr2=0x7f98b41072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:45.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 -- 172.21.15.16:0/1876303628 shutdown_connections 2024-08-28T19:20:45.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:45.756+0000 7f98bb583640 1 -- 172.21.15.16:0/1876303628 wait complete. 2024-08-28T19:20:46.613 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.h status 2024-08-28T19:20:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:47.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.364+0000 7f57f5255640 1 -- 172.21.15.16:0/4160321408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f0102820 msgr2=0x7f57f0102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:47.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.364+0000 7f57f5255640 1 --2- 172.21.15.16:0/4160321408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f0102820 0x7f57f0102c30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009900 tx=0x7f57e002a410 comp rx=0 tx=0).stop 2024-08-28T19:20:47.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.364+0000 7f57f5255640 1 -- 172.21.15.16:0/4160321408 shutdown_connections 2024-08-28T19:20:47.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.364+0000 7f57f5255640 1 --2- 172.21.15.16:0/4160321408 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f0108ab0 0x7f57f010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.364+0000 7f57f5255640 1 --2- 172.21.15.16:0/4160321408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0108160 0x7f57f0108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.364+0000 7f57f5255640 1 --2- 172.21.15.16:0/4160321408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f0102820 0x7f57f0102c30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.364+0000 7f57f5255640 1 -- 172.21.15.16:0/4160321408 >> 172.21.15.16:0/4160321408 conn(0x7f57f00794e0 msgr2=0x7f57f00798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.364+0000 7f57f5255640 1 -- 172.21.15.16:0/4160321408 shutdown_connections 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.365+0000 7f57f5255640 1 -- 172.21.15.16:0/4160321408 wait complete. 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.365+0000 7f57f5255640 1 Processor -- start 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.365+0000 7f57f5255640 1 -- start start 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57f5255640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0102820 0x7f57f019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57f5255640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f0108160 0x7f57f019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57f5255640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f0108ab0 0x7f57f019fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57f5255640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0112790 con 0x7f57f0108160 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57f5255640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0112610 con 0x7f57f0102820 2024-08-28T19:20:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57f5255640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0112910 con 0x7f57f0108ab0 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f0108160 0x7f57f019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57eed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0102820 0x7f57f019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f0108160 0x7f57f019a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33342/0 (socket says 172.21.15.16:33342) 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57ee59c640 1 -- 172.21.15.16:0/1845698687 learned_addr learned my addr 172.21.15.16:0/1845698687 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57ef59e640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f0108ab0 0x7f57f019fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57ee59c640 1 -- 172.21.15.16:0/1845698687 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f0108ab0 msgr2=0x7f57f019fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57ee59c640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f0108ab0 0x7f57f019fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57ee59c640 1 -- 172.21.15.16:0/1845698687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0102820 msgr2=0x7f57f019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57ee59c640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0102820 0x7f57f019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57ee59c640 1 -- 172.21.15.16:0/1845698687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f01a0200 con 0x7f57f0108160 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.366+0000 7f57ef59e640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f0108ab0 0x7f57f019fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.367+0000 7f57ee59c640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f0108160 0x7f57f019a830 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f57e400c970 tx=0x7f57e400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:47.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.367+0000 7f57d7fff640 1 -- 172.21.15.16:0/1845698687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e4007bb0 con 0x7f57f0108160 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.367+0000 7f57f5255640 1 -- 172.21.15.16:0/1845698687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f01a04f0 con 0x7f57f0108160 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.367+0000 7f57d7fff640 1 -- 172.21.15.16:0/1845698687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57e4007d50 con 0x7f57f0108160 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.367+0000 7f57d7fff640 1 -- 172.21.15.16:0/1845698687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e400f450 con 0x7f57f0108160 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.367+0000 7f57f5255640 1 -- 172.21.15.16:0/1845698687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f01a6cd0 con 0x7f57f0108160 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.369+0000 7f57fb7ca740 1 Processor -- start 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.369+0000 7f57fb7ca740 1 -- start start 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.369+0000 7f57d7fff640 1 -- 172.21.15.16:0/1845698687 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7f57e4004ec0 con 0x7f57f0108160 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.369+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be0557ef60 0x55be059247b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.369+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be05924cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be05925230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55be0589b3f0 con 0x55be055560a0 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55be0588a980 con 0x55be0555e280 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55be0588e930 con 0x55be0557ef60 2024-08-28T19:20:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be05924cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be05924cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33350/0 (socket says 172.21.15.16:33350) 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57eed9d640 1 -- 172.21.15.16:0/1749579492 learned_addr learned my addr 172.21.15.16:0/1749579492 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57ee59c640 1 --2- 172.21.15.16:0/1749579492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be05925230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57ef59e640 1 --2- 172.21.15.16:0/1749579492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be0557ef60 0x55be059247b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57eed9d640 1 -- 172.21.15.16:0/1749579492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be0557ef60 msgr2=0x55be059247b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57eed9d640 1 --2- 172.21.15.16:0/1749579492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be0557ef60 0x55be059247b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57eed9d640 1 -- 172.21.15.16:0/1749579492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 msgr2=0x55be05925230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57eed9d640 1 --2- 172.21.15.16:0/1749579492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be05925230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57eed9d640 1 -- 172.21.15.16:0/1749579492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be055ea540 con 0x55be055560a0 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57d7fff640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57d80b4b80 0x7f57d80b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57eed9d640 1 --2- 172.21.15.16:0/1749579492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be05924cf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009900 tx=0x7f57e0031e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41795ac0868606e7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57ef59e640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57d80b4b80 0x7f57d80b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57d7fff640 1 -- 172.21.15.16:0/1845698687 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f57e40e5380 con 0x7f57f0108160 2024-08-28T19:20:47.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.370+0000 7f57d57fa640 1 -- 172.21.15.16:0/1749579492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0040070 con 0x55be055560a0 2024-08-28T19:20:47.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.371+0000 7f57d57fa640 1 -- 172.21.15.16:0/1749579492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57e00316a0 con 0x55be055560a0 2024-08-28T19:20:47.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.371+0000 7f57d57fa640 1 -- 172.21.15.16:0/1749579492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0035040 con 0x55be055560a0 2024-08-28T19:20:47.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1749579492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 msgr2=0x55be05924cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:47.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57ef59e640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57d80b4b80 0x7f57d80b7090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f57e800c610 tx=0x7f57e800e040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:47.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/1749579492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be05924cf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009900 tx=0x7f57e0031e50 comp rx=0 tx=0).stop 2024-08-28T19:20:47.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1749579492 shutdown_connections 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/1749579492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be05925230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/1749579492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be05924cf0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/1749579492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be0557ef60 0x55be059247b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1749579492 >> 172.21.15.16:0/1749579492 conn(0x55be05573350 msgr2=0x55be059226a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1749579492 shutdown_connections 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1749579492 wait complete. 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57fb7ca740 1 -- start start 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.372+0000 7f57fb7ca740 1 Processor -- start 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be055560a0 0x55be0592feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be059303f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0557ef60 0x55be0593c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55be056ed5f0 con 0x55be0557ef60 2024-08-28T19:20:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55be05811600 con 0x55be0555e280 2024-08-28T19:20:47.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55be05932f30 con 0x55be055560a0 2024-08-28T19:20:47.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57fb7ca740 10 client.0 _ll_register_callbacks cb 0x55be05837af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:47.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57ef59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0557ef60 0x55be0593c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57ef59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0557ef60 0x55be0593c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33360/0 (socket says 172.21.15.16:33360) 2024-08-28T19:20:47.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57ef59e640 1 -- 172.21.15.16:0/2582197344 learned_addr learned my addr 172.21.15.16:0/2582197344 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:47.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57eed9d640 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be055560a0 0x55be0592feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57ee59c640 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be059303f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57ef59e640 1 -- 172.21.15.16:0/2582197344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be055560a0 msgr2=0x55be0592feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57ef59e640 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be055560a0 0x55be0592feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57ef59e640 1 -- 172.21.15.16:0/2582197344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 msgr2=0x55be059303f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57ef59e640 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be059303f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57ef59e640 1 -- 172.21.15.16:0/2582197344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be0584a7b0 con 0x55be0557ef60 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57ef59e640 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0557ef60 0x55be0593c250 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f57e8007e60 tx=0x7f57e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee2dd3ce4cc88be server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.373+0000 7f57d2ff5640 1 -- 172.21.15.16:0/2582197344 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8026020 con 0x55be0557ef60 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57fb7ca740 1 -- 172.21.15.16:0/2582197344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57e00316a0 con 0x55be0557ef60 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57d2ff5640 1 -- 172.21.15.16:0/2582197344 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f57e80223f0 con 0x55be0557ef60 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57d2ff5640 1 -- 172.21.15.16:0/2582197344 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f57e8031050 con 0x55be0557ef60 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57fb7ca740 10 client.15471 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57fb7ca740 1 -- 172.21.15.16:0/2582197344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55be0593ed00 con 0x55be0557ef60 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57d2ff5640 1 -- 172.21.15.16:0/2582197344 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57e00316a0 con 0x55be0557ef60 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57d2ff5640 1 -- 172.21.15.16:0/2582197344 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f57e8021030 con 0x55be0557ef60 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57fb7ca740 10 client.15471 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57fb7ca740 10 client.15471 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57fb7ca740 20 client.15471 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55be0553d8e0 0x55be059427e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57fb7ca740 4 client.15471 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.374+0000 7f57fb7ca740 1 -- 172.21.15.16:0/2582197344 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be0589d340 con 0x55be0553d8e0 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.375+0000 7f57eed9d640 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55be0553d8e0 0x55be059427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.375+0000 7f57eed9d640 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55be0553d8e0 0x55be059427e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f2fd96608144a4c7 server_cookie=fd264e47c7254e9e in_seq=0 out_seq=0 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.375+0000 7f57d2ff5640 10 client.15471 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:20:47.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57d2ff5640 1 -- 172.21.15.16:0/2582197344 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55be0589d340 con 0x55be0553d8e0 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57d2ff5640 10 client.15471 handle_command_reply: tid=0 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 1 client.15471 shutdown 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 1 -- 172.21.15.16:0/2582197344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0557ef60 msgr2=0x55be0593c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0557ef60 0x55be0593c250 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f57e8007e60 tx=0x7f57e8004a40 comp rx=0 tx=0).stop 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 1 -- 172.21.15.16:0/2582197344 shutdown_connections 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55be0553d8e0 0x55be059427e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0557ef60 0x55be0593c250 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be059303f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/2582197344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be055560a0 0x55be0592feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57d2ff5640 0 client.15471 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 1 -- 172.21.15.16:0/2582197344 >> 172.21.15.16:0/2582197344 conn(0x55be05573350 msgr2=0x55be05927940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 1 -- 172.21.15.16:0/2582197344 shutdown_connections 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 1 -- 172.21.15.16:0/2582197344 wait complete. 2024-08-28T19:20:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.376+0000 7f57fb7ca740 20 client.15471 trim_cache size 0 max 16384 2024-08-28T19:20:47.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.394+0000 7f57fb7ca740 1 Processor -- start 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.394+0000 7f57fb7ca740 1 -- start start 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.394+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be055560a0 0x55be0593c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0555e280 0x55be05940170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0557ef60 0x55be0592e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55be05940b20 con 0x55be0555e280 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55be059411f0 con 0x55be0557ef60 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55be059418c0 con 0x55be055560a0 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0555e280 0x55be05940170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0555e280 0x55be05940170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33372/0 (socket says 172.21.15.16:33372) 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57ee59c640 1 -- 172.21.15.16:0/3854822901 learned_addr learned my addr 172.21.15.16:0/3854822901 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57eed9d640 1 --2- 172.21.15.16:0/3854822901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be055560a0 0x55be0593c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57ef59e640 1 --2- 172.21.15.16:0/3854822901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0557ef60 0x55be0592e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57ee59c640 1 -- 172.21.15.16:0/3854822901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be055560a0 msgr2=0x55be0593c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:47.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57ee59c640 1 --2- 172.21.15.16:0/3854822901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be055560a0 0x55be0593c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57ee59c640 1 -- 172.21.15.16:0/3854822901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0557ef60 msgr2=0x55be0592e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:47.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57ee59c640 1 --2- 172.21.15.16:0/3854822901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0557ef60 0x55be0592e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57ee59c640 1 -- 172.21.15.16:0/3854822901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be0592aae0 con 0x55be0555e280 2024-08-28T19:20:47.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57ee59c640 1 --2- 172.21.15.16:0/3854822901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0555e280 0x55be05940170 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f57e40ab8b0 tx=0x7f57e40a7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d4fddb6b79718b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:47.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57d4ff9640 1 -- 172.21.15.16:0/3854822901 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e401d040 con 0x55be0555e280 2024-08-28T19:20:47.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57d4ff9640 1 -- 172.21.15.16:0/3854822901 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57e40046c0 con 0x55be0555e280 2024-08-28T19:20:47.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.395+0000 7f57d4ff9640 1 -- 172.21.15.16:0/3854822901 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e40ac020 con 0x55be0555e280 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.396+0000 7f57fb7ca740 1 -- 172.21.15.16:0/3854822901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0555e280 msgr2=0x55be05940170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.396+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/3854822901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0555e280 0x55be05940170 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f57e40ab8b0 tx=0x7f57e40a7040 comp rx=0 tx=0).stop 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.396+0000 7f57fb7ca740 1 -- 172.21.15.16:0/3854822901 shutdown_connections 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.396+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/3854822901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0557ef60 0x55be0592e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.396+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/3854822901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be0555e280 0x55be05940170 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.396+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/3854822901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be055560a0 0x55be0593c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.396+0000 7f57fb7ca740 1 -- 172.21.15.16:0/3854822901 >> 172.21.15.16:0/3854822901 conn(0x55be05573350 msgr2=0x55be0592fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.396+0000 7f57fb7ca740 1 -- 172.21.15.16:0/3854822901 shutdown_connections 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.396+0000 7f57fb7ca740 1 -- 172.21.15.16:0/3854822901 wait complete. 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.396+0000 7f57fb7ca740 1 -- start start 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57fb7ca740 1 Processor -- start 2024-08-28T19:20:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be0594b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be059447b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57fb7ca740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be0557ef60 0x55be05944d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55be05942660 con 0x55be055560a0 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55be05941f90 con 0x55be0555e280 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57fb7ca740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55be05942d30 con 0x55be0557ef60 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be0594b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57fb7ca740 10 client.0 _ll_register_callbacks cb 0x7f57e8007a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be0594b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33374/0 (socket says 172.21.15.16:33374) 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57eed9d640 1 -- 172.21.15.16:0/1821241598 learned_addr learned my addr 172.21.15.16:0/1821241598 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57ee59c640 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be059447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57ef59e640 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be0557ef60 0x55be05944d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57eed9d640 1 -- 172.21.15.16:0/1821241598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be0557ef60 msgr2=0x55be05944d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57eed9d640 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be0557ef60 0x55be05944d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57eed9d640 1 -- 172.21.15.16:0/1821241598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 msgr2=0x55be059447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:47.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57eed9d640 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be059447b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57eed9d640 1 -- 172.21.15.16:0/1821241598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be059454c0 con 0x55be055560a0 2024-08-28T19:20:47.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57ee59c640 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be059447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:47.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.397+0000 7f57ef59e640 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be0557ef60 0x55be05944d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:47.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57eed9d640 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be0594b2b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f57e0005400 tx=0x7f57e002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc8d7c9ee7c7532 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:47.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:47.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57d2ff5640 1 -- 172.21.15.16:0/1821241598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0040040 con 0x55be055560a0 2024-08-28T19:20:47.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57d2ff5640 1 -- 172.21.15.16:0/1821241598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f57e00316a0 con 0x55be055560a0 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1821241598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55be05898de0 con 0x55be055560a0 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57d2ff5640 1 -- 172.21.15.16:0/1821241598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f57e0060050 con 0x55be055560a0 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57fb7ca740 10 client.15477 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1821241598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55be059adcb0 con 0x55be055560a0 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57d2ff5640 1 -- 172.21.15.16:0/1821241598 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55be05898de0 con 0x55be055560a0 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57d2ff5640 1 -- 172.21.15.16:0/1821241598 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f57e003c560 con 0x55be055560a0 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57fb7ca740 10 client.15477 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57fb7ca740 10 client.15477 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57fb7ca740 20 client.15477 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55be05512c90 0x55be059b1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57fb7ca740 4 client.15477 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.398+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1821241598 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "status"}) -- 0x55be05896520 con 0x55be05512c90 2024-08-28T19:20:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57ee59c640 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55be05512c90 0x55be059b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57d2ff5640 10 client.15477 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57ee59c640 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55be05512c90 0x55be059b1000 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1aac218ecaee3d59 server_cookie=dac2ef02fcab10c0 in_seq=0 out_seq=0 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57d2ff5640 1 -- 172.21.15.16:0/1821241598 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55be05896520 con 0x55be05512c90 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57d2ff5640 10 client.15477 handle_command_reply: tid=0 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57fb7ca740 1 client.15477 shutdown 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1821241598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 msgr2=0x55be0594b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be0594b2b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f57e0005400 tx=0x7f57e002a410 comp rx=0 tx=0).stop 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1821241598 shutdown_connections 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55be05512c90 0x55be059b1000 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55be0557ef60 0x55be05944d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be0555e280 0x55be059447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57fb7ca740 1 --2- 172.21.15.16:0/1821241598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be055560a0 0x55be0594b2b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57d2ff5640 0 client.15477 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:20:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.399+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1821241598 >> 172.21.15.16:0/1821241598 conn(0x55be05573350 msgr2=0x55be05940170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:47.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.400+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1821241598 shutdown_connections 2024-08-28T19:20:47.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.400+0000 7f57fb7ca740 1 -- 172.21.15.16:0/1821241598 wait complete. 2024-08-28T19:20:47.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.400+0000 7f57fb7ca740 20 client.15477 trim_cache size 0 max 16384 2024-08-28T19:20:47.407 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:20:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:20:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-08-28T19:20:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24635, 2024-08-28T19:20:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:20:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:20:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:20:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 97.774720056000007, 2024-08-28T19:20:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-08-28T19:20:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:20:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 82, 2024-08-28T19:20:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 155.45441597499999 2024-08-28T19:20:47.408 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:20:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.400+0000 7f57f5255640 1 -- 172.21.15.16:0/1845698687 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57d80b4b80 msgr2=0x7f57d80b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.400+0000 7f57f5255640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57d80b4b80 0x7f57d80b7090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f57e800c610 tx=0x7f57e800e040 comp rx=0 tx=0).stop 2024-08-28T19:20:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.400+0000 7f57f5255640 1 -- 172.21.15.16:0/1845698687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f0108160 msgr2=0x7f57f019a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.400+0000 7f57f5255640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f0108160 0x7f57f019a830 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f57e400c970 tx=0x7f57e400ce30 comp rx=0 tx=0).stop 2024-08-28T19:20:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.401+0000 7f57f5255640 1 -- 172.21.15.16:0/1845698687 shutdown_connections 2024-08-28T19:20:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.401+0000 7f57f5255640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f0108ab0 0x7f57f019fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.401+0000 7f57f5255640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f0108160 0x7f57f019a830 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.401+0000 7f57f5255640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57d80b4b80 0x7f57d80b7090 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.401+0000 7f57f5255640 1 --2- 172.21.15.16:0/1845698687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0102820 0x7f57f019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.401+0000 7f57f5255640 1 -- 172.21.15.16:0/1845698687 >> 172.21.15.16:0/1845698687 conn(0x7f57f00794e0 msgr2=0x7f57f0101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.401+0000 7f57f5255640 1 -- 172.21.15.16:0/1845698687 shutdown_connections 2024-08-28T19:20:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:47.401+0000 7f57f5255640 1 -- 172.21.15.16:0/1845698687 wait complete. 2024-08-28T19:20:47.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:47 smithi016 ceph-mon[18556]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-08-28T19:20:47.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:47 smithi028 ceph-mon[21769]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-08-28T19:20:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:47 smithi115 ceph-mon[22184]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-08-28T19:20:48.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.i status 2024-08-28T19:20:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:48.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.830+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/2128988118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b04109bd0 msgr2=0x7f3b04110400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.830+0000 7f3b0b0d6640 1 --2- 172.21.15.16:0/2128988118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b04109bd0 0x7f3b04110400 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009ca0 tx=0x7f3afc02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.830+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/2128988118 shutdown_connections 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.830+0000 7f3b0b0d6640 1 --2- 172.21.15.16:0/2128988118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b04109bd0 0x7f3b04110400 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.830+0000 7f3b0b0d6640 1 --2- 172.21.15.16:0/2128988118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b04109280 0x7f3b04109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.830+0000 7f3b0b0d6640 1 --2- 172.21.15.16:0/2128988118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04103940 0x7f3b04103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.830+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/2128988118 >> 172.21.15.16:0/2128988118 conn(0x7f3b040ff3d0 msgr2=0x7f3b04101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.830+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/2128988118 shutdown_connections 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.830+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/2128988118 wait complete. 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3b0b0d6640 1 Processor -- start 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3b0b0d6640 1 -- start start 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3b0b0d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04103940 0x7f3b0419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3b0b0d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b04109280 0x7f3b0419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3b0b0d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b04109bd0 0x7f3b0419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3b0b0d6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3b04113890 con 0x7f3b04109280 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3b0b0d6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b04113710 con 0x7f3b04103940 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3b0b0d6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3b04113a10 con 0x7f3b04109bd0 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3afbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b04109280 0x7f3b0419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3afbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b04109280 0x7f3b0419a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33398/0 (socket says 172.21.15.16:33398) 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3afbfff640 1 -- 172.21.15.16:0/584787520 learned_addr learned my addr 172.21.15.16:0/584787520 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3b09673640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b04109bd0 0x7f3b0419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.831+0000 7f3b08e72640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04103940 0x7f3b0419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3afbfff640 1 -- 172.21.15.16:0/584787520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b04109bd0 msgr2=0x7f3b0419fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3afbfff640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b04109bd0 0x7f3b0419fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3afbfff640 1 -- 172.21.15.16:0/584787520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04103940 msgr2=0x7f3b0419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3afbfff640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04103940 0x7f3b0419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3afbfff640 1 -- 172.21.15.16:0/584787520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b041a0170 con 0x7f3b04109280 2024-08-28T19:20:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3b09673640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b04109bd0 0x7f3b0419fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3afbfff640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b04109280 0x7f3b0419a7a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3af400d740 tx=0x7f3af400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3af9ffb640 1 -- 172.21.15.16:0/584787520 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4017070 con 0x7f3b04109280 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3af9ffb640 1 -- 172.21.15.16:0/584787520 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3af400de50 con 0x7f3b04109280 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/584787520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b041a0460 con 0x7f3b04109280 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3af9ffb640 1 -- 172.21.15.16:0/584787520 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af400f620 con 0x7f3b04109280 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.832+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/584787520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b041a6bc0 con 0x7f3b04109280 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3af9ffb640 1 -- 172.21.15.16:0/584787520 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7f3af401a070 con 0x7f3b04109280 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3b1164b740 1 Processor -- start 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3b1164b740 1 -- start start 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c064b6f60 0x557c0685c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c0648e0a0 0x557c0685ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c06496280 0x557c0685d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3b1164b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557c067d33f0 con 0x557c06496280 2024-08-28T19:20:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3b1164b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557c067c2980 con 0x557c0648e0a0 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3b1164b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557c067c6930 con 0x557c064b6f60 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3afbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c06496280 0x557c0685d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3b08e72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c0648e0a0 0x557c0685ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3afbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c06496280 0x557c0685d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33404/0 (socket says 172.21.15.16:33404) 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3b09673640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c064b6f60 0x557c0685c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.835+0000 7f3afbfff640 1 -- 172.21.15.16:0/2191559976 learned_addr learned my addr 172.21.15.16:0/2191559976 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3af9ffb640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3aec0b4b80 0x7f3aec0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3afbfff640 1 -- 172.21.15.16:0/2191559976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c064b6f60 msgr2=0x557c0685c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3afbfff640 1 --2- 172.21.15.16:0/2191559976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c064b6f60 0x557c0685c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3afbfff640 1 -- 172.21.15.16:0/2191559976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c0648e0a0 msgr2=0x557c0685ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3afbfff640 1 --2- 172.21.15.16:0/2191559976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c0648e0a0 0x557c0685ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3b09673640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3aec0b4b80 0x7f3aec0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3af9ffb640 1 -- 172.21.15.16:0/584787520 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3af40e5130 con 0x7f3b04109280 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3afbfff640 1 -- 172.21.15.16:0/2191559976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c06522540 con 0x557c06496280 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3b08e72640 1 --2- 172.21.15.16:0/2191559976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c0648e0a0 0x557c0685ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3afbfff640 1 --2- 172.21.15.16:0/2191559976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c06496280 0x557c0685d230 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3af40ab660 tx=0x7f3af4005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b051b374c67580df server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3aeaffd640 1 -- 172.21.15.16:0/2191559976 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4021070 con 0x557c06496280 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3b09673640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3aec0b4b80 0x7f3aec0b7090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3afc005cc0 tx=0x7f3afc0095c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3aeaffd640 1 -- 172.21.15.16:0/2191559976 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3af40ab1e0 con 0x557c06496280 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.836+0000 7f3aeaffd640 1 -- 172.21.15.16:0/2191559976 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af40ab6d0 con 0x557c06496280 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.837+0000 7f3b1164b740 1 -- 172.21.15.16:0/2191559976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c06496280 msgr2=0x557c0685d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.837+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2191559976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c06496280 0x557c0685d230 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3af40ab660 tx=0x7f3af4005cc0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.837+0000 7f3b1164b740 1 -- 172.21.15.16:0/2191559976 shutdown_connections 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.837+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2191559976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c06496280 0x557c0685d230 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.837+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2191559976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c0648e0a0 0x557c0685ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.837+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2191559976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c064b6f60 0x557c0685c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.837+0000 7f3b1164b740 1 -- 172.21.15.16:0/2191559976 >> 172.21.15.16:0/2191559976 conn(0x557c064ab350 msgr2=0x557c0685a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.837+0000 7f3b1164b740 1 -- 172.21.15.16:0/2191559976 shutdown_connections 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.837+0000 7f3b1164b740 1 -- 172.21.15.16:0/2191559976 wait complete. 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.838+0000 7f3b1164b740 1 -- start start 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.838+0000 7f3b1164b740 1 Processor -- start 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.838+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c0686c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.838+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c06496280 0x557c0686c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.838+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c064b6f60 0x557c0686cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.838+0000 7f3b1164b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557c066255f0 con 0x557c0648e0a0 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.838+0000 7f3b1164b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557c06749600 con 0x557c064b6f60 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.838+0000 7f3b1164b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557c0686f200 con 0x557c06496280 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.838+0000 7f3b1164b740 10 client.0 _ll_register_callbacks cb 0x557c0676faf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.838+0000 7f3b08e72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c0686c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.838+0000 7f3b08e72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c0686c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33412/0 (socket says 172.21.15.16:33412) 2024-08-28T19:20:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b08e72640 1 -- 172.21.15.16:0/2666288552 learned_addr learned my addr 172.21.15.16:0/2666288552 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b09673640 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c064b6f60 0x557c0686cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3afbfff640 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c06496280 0x557c0686c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b08e72640 1 -- 172.21.15.16:0/2666288552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c06496280 msgr2=0x557c0686c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b08e72640 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c06496280 0x557c0686c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b08e72640 1 -- 172.21.15.16:0/2666288552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c064b6f60 msgr2=0x557c0686cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b08e72640 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c064b6f60 0x557c0686cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b08e72640 1 -- 172.21.15.16:0/2666288552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c067827b0 con 0x557c0648e0a0 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b09673640 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c064b6f60 0x557c0686cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b08e72640 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c0686c210 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3b000117c0 tx=0x7f3b00011c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99da3fdd280339a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3ae3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2666288552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0000fe70 con 0x557c0648e0a0 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2666288552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3b00005020 con 0x557c0648e0a0 2024-08-28T19:20:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b1164b740 1 -- 172.21.15.16:0/2666288552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3af40ab1e0 con 0x557c0648e0a0 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2666288552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3b00024050 con 0x557c0648e0a0 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b1164b740 10 client.15489 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.839+0000 7f3b1164b740 1 -- 172.21.15.16:0/2666288552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557c0687af30 con 0x557c0648e0a0 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.840+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2666288552 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3af40ab1e0 con 0x557c0648e0a0 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.840+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2666288552 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f3b00018030 con 0x557c0648e0a0 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.840+0000 7f3b1164b740 10 client.15489 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.840+0000 7f3b1164b740 10 client.15489 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.840+0000 7f3b1164b740 20 client.15489 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.840+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x557c064758e0 0x557c0687eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.840+0000 7f3b1164b740 4 client.15489 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.840+0000 7f3b1164b740 1 -- 172.21.15.16:0/2666288552 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c067d5340 con 0x557c064758e0 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.840+0000 7f3afbfff640 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x557c064758e0 0x557c0687eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.841+0000 7f3ae3fff640 10 client.15489 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.841+0000 7f3afbfff640 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x557c064758e0 0x557c0687eab0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ca8fb70d983a32ab server_cookie=69bff9a9ce419c36 in_seq=0 out_seq=0 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.841+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2666288552 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557c067d5340 con 0x557c064758e0 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.841+0000 7f3ae3fff640 10 client.15489 handle_command_reply: tid=0 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.841+0000 7f3b1164b740 1 client.15489 shutdown 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.841+0000 7f3b1164b740 1 -- 172.21.15.16:0/2666288552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 msgr2=0x557c0686c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.841+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c0686c210 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3b000117c0 tx=0x7f3b00011c80 comp rx=0 tx=0).stop 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.842+0000 7f3b1164b740 1 -- 172.21.15.16:0/2666288552 shutdown_connections 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.842+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x557c064758e0 0x557c0687eab0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.842+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c064b6f60 0x557c0686cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.842+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c06496280 0x557c0686c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.842+0000 7f3ae3fff640 0 client.15489 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.842+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2666288552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c0686c210 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.842+0000 7f3b1164b740 1 -- 172.21.15.16:0/2666288552 >> 172.21.15.16:0/2666288552 conn(0x557c064ab350 msgr2=0x557c0685f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:48.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.842+0000 7f3b1164b740 1 -- 172.21.15.16:0/2666288552 shutdown_connections 2024-08-28T19:20:48.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.842+0000 7f3b1164b740 1 -- 172.21.15.16:0/2666288552 wait complete. 2024-08-28T19:20:48.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.842+0000 7f3b1164b740 20 client.15489 trim_cache size 0 max 16384 2024-08-28T19:20:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.860+0000 7f3b1164b740 1 Processor -- start 2024-08-28T19:20:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.860+0000 7f3b1164b740 1 -- start start 2024-08-28T19:20:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.860+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c0648e0a0 0x557c06859cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.860+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c06496280 0x557c0685a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.860+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c064b6f60 0x557c06869400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.860+0000 7f3b1164b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557c0687cdf0 con 0x557c064b6f60 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.860+0000 7f3b1164b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557c0687d4c0 con 0x557c0648e0a0 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.860+0000 7f3b1164b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557c0687db90 con 0x557c06496280 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3b09673640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c064b6f60 0x557c06869400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3b09673640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c064b6f60 0x557c06869400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33418/0 (socket says 172.21.15.16:33418) 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3b09673640 1 -- 172.21.15.16:0/1689544237 learned_addr learned my addr 172.21.15.16:0/1689544237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3b08e72640 1 --2- 172.21.15.16:0/1689544237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c0648e0a0 0x557c06859cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3afbfff640 1 --2- 172.21.15.16:0/1689544237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c06496280 0x557c0685a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3b09673640 1 -- 172.21.15.16:0/1689544237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c06496280 msgr2=0x557c0685a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3b09673640 1 --2- 172.21.15.16:0/1689544237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c06496280 0x557c0685a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3b09673640 1 -- 172.21.15.16:0/1689544237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c0648e0a0 msgr2=0x557c06859cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3b09673640 1 --2- 172.21.15.16:0/1689544237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c0648e0a0 0x557c06859cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3b09673640 1 -- 172.21.15.16:0/1689544237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c0686a710 con 0x557c064b6f60 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3b09673640 1 --2- 172.21.15.16:0/1689544237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c064b6f60 0x557c06869400 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3afc031f00 tx=0x7f3afc041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937f5610cbdbf6c1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3aea7fc640 1 -- 172.21.15.16:0/1689544237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc053070 con 0x557c064b6f60 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3aea7fc640 1 -- 172.21.15.16:0/1689544237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3afc036040 con 0x557c064b6f60 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.861+0000 7f3aea7fc640 1 -- 172.21.15.16:0/1689544237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc02f8a0 con 0x557c064b6f60 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.862+0000 7f3b1164b740 1 -- 172.21.15.16:0/1689544237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c064b6f60 msgr2=0x557c06869400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.862+0000 7f3b1164b740 1 --2- 172.21.15.16:0/1689544237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c064b6f60 0x557c06869400 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3afc031f00 tx=0x7f3afc041000 comp rx=0 tx=0).stop 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.862+0000 7f3b1164b740 1 -- 172.21.15.16:0/1689544237 shutdown_connections 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.862+0000 7f3b1164b740 1 --2- 172.21.15.16:0/1689544237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c064b6f60 0x557c06869400 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.862+0000 7f3b1164b740 1 --2- 172.21.15.16:0/1689544237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c06496280 0x557c0685a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.862+0000 7f3b1164b740 1 --2- 172.21.15.16:0/1689544237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c0648e0a0 0x557c06859cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.862+0000 7f3b1164b740 1 -- 172.21.15.16:0/1689544237 >> 172.21.15.16:0/1689544237 conn(0x557c064ab350 msgr2=0x557c0686aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.862+0000 7f3b1164b740 1 -- 172.21.15.16:0/1689544237 shutdown_connections 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.862+0000 7f3b1164b740 1 -- 172.21.15.16:0/1689544237 wait complete. 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b1164b740 1 -- start start 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b1164b740 1 Processor -- start 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c06881030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c06496280 0x557c06881570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b1164b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c064b6f60 0x557c068867c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b1164b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557c0687e930 con 0x557c0648e0a0 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b1164b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557c0687e260 con 0x557c06496280 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b1164b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557c0687f000 con 0x557c064b6f60 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b1164b740 10 client.0 _ll_register_callbacks cb 0x7f3b00011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b08e72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c06881030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b08e72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c06881030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33434/0 (socket says 172.21.15.16:33434) 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b08e72640 1 -- 172.21.15.16:0/2779553830 learned_addr learned my addr 172.21.15.16:0/2779553830 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3afbfff640 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c06496280 0x557c06881570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.863+0000 7f3b09673640 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c064b6f60 0x557c068867c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3b08e72640 1 -- 172.21.15.16:0/2779553830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c064b6f60 msgr2=0x557c068867c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3b08e72640 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c064b6f60 0x557c068867c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3b08e72640 1 -- 172.21.15.16:0/2779553830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c06496280 msgr2=0x557c06881570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3b08e72640 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c06496280 0x557c06881570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3b08e72640 1 -- 172.21.15.16:0/2779553830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c06881e10 con 0x557c0648e0a0 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3afbfff640 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c06496280 0x557c06881570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3b08e72640 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c06881030 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3b000194e0 tx=0x7f3b0000ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41390da10cb399 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3ae3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2779553830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b00004a70 con 0x557c0648e0a0 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2779553830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3b00004c10 con 0x557c0648e0a0 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3b1164b740 1 -- 172.21.15.16:0/2779553830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3afc036040 con 0x557c0648e0a0 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2779553830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3b00039050 con 0x557c0648e0a0 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3b1164b740 10 client.15495 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3b1164b740 1 -- 172.21.15.16:0/2779553830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557c068ea540 con 0x557c0648e0a0 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.864+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2779553830 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3afc036040 con 0x557c0648e0a0 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.865+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2779553830 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f3b00018040 con 0x557c0648e0a0 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.865+0000 7f3b1164b740 10 client.15495 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.865+0000 7f3b1164b740 10 client.15495 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.865+0000 7f3b1164b740 20 client.15495 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.865+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x557c0644ac90 0x557c068ede80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.865+0000 7f3b1164b740 4 client.15495 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:20:48.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.865+0000 7f3b1164b740 1 -- 172.21.15.16:0/2779553830 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "status"}) -- 0x557c067d0de0 con 0x557c0644ac90 2024-08-28T19:20:48.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.865+0000 7f3afbfff640 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x557c0644ac90 0x557c068ede80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:48.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.865+0000 7f3afbfff640 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x557c0644ac90 0x557c068ede80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc16cec3aa6930d8 server_cookie=6955ed4e6b47787c in_seq=0 out_seq=0 2024-08-28T19:20:48.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.865+0000 7f3ae3fff640 10 client.15495 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:20:48.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3ae3fff640 1 -- 172.21.15.16:0/2779553830 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557c067d0de0 con 0x557c0644ac90 2024-08-28T19:20:48.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3ae3fff640 10 client.15495 handle_command_reply: tid=0 2024-08-28T19:20:48.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 1 client.15495 shutdown 2024-08-28T19:20:48.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 1 -- 172.21.15.16:0/2779553830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 msgr2=0x557c06881030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:48.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c06881030 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3b000194e0 tx=0x7f3b0000ff20 comp rx=0 tx=0).stop 2024-08-28T19:20:48.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 1 -- 172.21.15.16:0/2779553830 shutdown_connections 2024-08-28T19:20:48.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x557c0644ac90 0x557c068ede80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c064b6f60 0x557c068867c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c06496280 0x557c06881570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 1 --2- 172.21.15.16:0/2779553830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c0648e0a0 0x557c06881030 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 1 -- 172.21.15.16:0/2779553830 >> 172.21.15.16:0/2779553830 conn(0x557c064ab350 msgr2=0x557c0685d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3ae3fff640 0 client.15495 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:20:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 1 -- 172.21.15.16:0/2779553830 shutdown_connections 2024-08-28T19:20:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 1 -- 172.21.15.16:0/2779553830 wait complete. 2024-08-28T19:20:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.866+0000 7f3b1164b740 20 client.15495 trim_cache size 0 max 16384 2024-08-28T19:20:48.874 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24649, 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 100.25371381799999, 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 82, 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 154.34864790699999 2024-08-28T19:20:48.875 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:20:48.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/584787520 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3aec0b4b80 msgr2=0x7f3aec0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:48.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3aec0b4b80 0x7f3aec0b7090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3afc005cc0 tx=0x7f3afc0095c0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/584787520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b04109280 msgr2=0x7f3b0419a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:48.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b04109280 0x7f3b0419a7a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3af400d740 tx=0x7f3af400dc00 comp rx=0 tx=0).stop 2024-08-28T19:20:48.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/584787520 shutdown_connections 2024-08-28T19:20:48.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b04109bd0 0x7f3b0419fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b04109280 0x7f3b0419a7a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3aec0b4b80 0x7f3aec0b7090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 --2- 172.21.15.16:0/584787520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04103940 0x7f3b0419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:48.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/584787520 >> 172.21.15.16:0/584787520 conn(0x7f3b040ff3d0 msgr2=0x7f3b041088d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:48.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/584787520 shutdown_connections 2024-08-28T19:20:48.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:48.867+0000 7f3b0b0d6640 1 -- 172.21.15.16:0/584787520 wait complete. 2024-08-28T19:20:49.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:49 smithi028 ceph-mon[21769]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-08-28T19:20:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:49 smithi115 ceph-mon[22184]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-08-28T19:20:49.647 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:49 smithi016 ceph-mon[18556]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-08-28T19:20:49.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.b status 2024-08-28T19:20:49.930 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.415+0000 7fa6c1409640 1 -- 172.21.15.16:0/137051369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc109bb0 msgr2=0x7fa6bc1103e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.415+0000 7fa6c1409640 1 --2- 172.21.15.16:0/137051369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc109bb0 0x7fa6bc1103e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009ca0 tx=0x7fa6b402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.415+0000 7fa6c1409640 1 -- 172.21.15.16:0/137051369 shutdown_connections 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.415+0000 7fa6c1409640 1 --2- 172.21.15.16:0/137051369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc109bb0 0x7fa6bc1103e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.415+0000 7fa6c1409640 1 --2- 172.21.15.16:0/137051369 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6bc109260 0x7fa6bc109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.415+0000 7fa6c1409640 1 --2- 172.21.15.16:0/137051369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6bc103920 0x7fa6bc103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.416+0000 7fa6c1409640 1 -- 172.21.15.16:0/137051369 >> 172.21.15.16:0/137051369 conn(0x7fa6bc0ff370 msgr2=0x7fa6bc1017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.416+0000 7fa6c1409640 1 -- 172.21.15.16:0/137051369 shutdown_connections 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.416+0000 7fa6c1409640 1 -- 172.21.15.16:0/137051369 wait complete. 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.416+0000 7fa6c1409640 1 Processor -- start 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6c1409640 1 -- start start 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6c1409640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6bc103920 0x7fa6bc19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6c1409640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc109260 0x7fa6bc19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6c1409640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6bc109bb0 0x7fa6bc19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6c1409640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc113870 con 0x7fa6bc109260 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6c1409640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc1136f0 con 0x7fa6bc109bb0 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6c1409640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc1139f0 con 0x7fa6bc103920 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6ba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc109260 0x7fa6bc19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6ba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc109260 0x7fa6bc19a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33460/0 (socket says 172.21.15.16:33460) 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6ba7fc640 1 -- 172.21.15.16:0/685701722 learned_addr learned my addr 172.21.15.16:0/685701722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6bb7fe640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6bc109bb0 0x7fa6bc19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6baffd640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6bc103920 0x7fa6bc19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.417+0000 7fa6ba7fc640 1 -- 172.21.15.16:0/685701722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6bc103920 msgr2=0x7fa6bc19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.418+0000 7fa6ba7fc640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6bc103920 0x7fa6bc19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.418+0000 7fa6ba7fc640 1 -- 172.21.15.16:0/685701722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6bc109bb0 msgr2=0x7fa6bc19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.418+0000 7fa6ba7fc640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6bc109bb0 0x7fa6bc19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.418+0000 7fa6ba7fc640 1 -- 172.21.15.16:0/685701722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6bc1a0190 con 0x7fa6bc109260 2024-08-28T19:20:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.418+0000 7fa6ba7fc640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc109260 0x7fa6bc19a7c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00d750 tx=0x7fa6ac00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.418+0000 7fa6b3fff640 1 -- 172.21.15.16:0/685701722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac017070 con 0x7fa6bc109260 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.418+0000 7fa6c1409640 1 -- 172.21.15.16:0/685701722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6bc1a0480 con 0x7fa6bc109260 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.418+0000 7fa6b3fff640 1 -- 172.21.15.16:0/685701722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6ac0054d0 con 0x7fa6bc109260 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.418+0000 7fa6b3fff640 1 -- 172.21.15.16:0/685701722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac004c30 con 0x7fa6bc109260 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.418+0000 7fa6c1409640 1 -- 172.21.15.16:0/685701722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6bc1a6be0 con 0x7fa6bc109260 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.420+0000 7fa6c797e740 1 Processor -- start 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.420+0000 7fa6c797e740 1 -- start start 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.420+0000 7fa6b3fff640 1 -- 172.21.15.16:0/685701722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fa6ac004dd0 con 0x7fa6bc109260 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x5606452df7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452e0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6c797e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5606452563f0 con 0x560644f110a0 2024-08-28T19:20:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6c797e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560645245980 con 0x560644f19280 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6c797e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560645249930 con 0x560644f39f60 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33474/0 (socket says 172.21.15.16:33474) 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6baffd640 1 -- 172.21.15.16:0/793335028 learned_addr learned my addr 172.21.15.16:0/793335028 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6ba7fc640 1 --2- 172.21.15.16:0/793335028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452e0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6bb7fe640 1 --2- 172.21.15.16:0/793335028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x5606452df7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6baffd640 1 -- 172.21.15.16:0/793335028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 msgr2=0x5606452df7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6baffd640 1 --2- 172.21.15.16:0/793335028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x5606452df7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6baffd640 1 -- 172.21.15.16:0/793335028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 msgr2=0x5606452e0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6baffd640 1 --2- 172.21.15.16:0/793335028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452e0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6b3fff640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6a80b4b80 0x7fa6a80b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.421+0000 7fa6baffd640 1 -- 172.21.15.16:0/793335028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560644fa5540 con 0x560644f110a0 2024-08-28T19:20:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.422+0000 7fa6bb7fe640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6a80b4b80 0x7fa6a80b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.422+0000 7fa6baffd640 1 --2- 172.21.15.16:0/793335028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dfcf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009ca0 tx=0x7fa6a40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c632ce244a2f5e32 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.422+0000 7fa6b3fff640 1 -- 172.21.15.16:0/685701722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa6ac0e5050 con 0x7fa6bc109260 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.422+0000 7fa6b17fa640 1 -- 172.21.15.16:0/793335028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4012450 con 0x560644f110a0 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.422+0000 7fa6b17fa640 1 -- 172.21.15.16:0/793335028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6a4012aa0 con 0x560644f110a0 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.422+0000 7fa6b17fa640 1 -- 172.21.15.16:0/793335028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a400fab0 con 0x560644f110a0 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6c797e740 1 -- 172.21.15.16:0/793335028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 msgr2=0x5606452dfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6c797e740 1 --2- 172.21.15.16:0/793335028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dfcf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009ca0 tx=0x7fa6a40117b0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6bb7fe640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6a80b4b80 0x7fa6a80b7090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4006010 tx=0x7fa6b4009440 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6c797e740 1 -- 172.21.15.16:0/793335028 shutdown_connections 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6c797e740 1 --2- 172.21.15.16:0/793335028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452e0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6c797e740 1 --2- 172.21.15.16:0/793335028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dfcf0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6c797e740 1 --2- 172.21.15.16:0/793335028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x5606452df7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6c797e740 1 -- 172.21.15.16:0/793335028 >> 172.21.15.16:0/793335028 conn(0x560644f2e350 msgr2=0x5606452dd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6c797e740 1 -- 172.21.15.16:0/793335028 shutdown_connections 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6c797e740 1 -- 172.21.15.16:0/793335028 wait complete. 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6c797e740 1 -- start start 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.423+0000 7fa6c797e740 1 Processor -- start 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f110a0 0x5606452ef210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452ef750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f39f60 0x5606452efcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6c797e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5606450a85f0 con 0x560644f39f60 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6c797e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5606451cc600 con 0x560644f19280 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6c797e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5606452f22f0 con 0x560644f110a0 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6c797e740 10 client.0 _ll_register_callbacks cb 0x5606451f2af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f39f60 0x5606452efcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f39f60 0x5606452efcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33480/0 (socket says 172.21.15.16:33480) 2024-08-28T19:20:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6bb7fe640 1 -- 172.21.15.16:0/309949412 learned_addr learned my addr 172.21.15.16:0/309949412 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6ba7fc640 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452ef750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6baffd640 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f110a0 0x5606452ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6bb7fe640 1 -- 172.21.15.16:0/309949412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f110a0 msgr2=0x5606452ef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6bb7fe640 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f110a0 0x5606452ef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6bb7fe640 1 -- 172.21.15.16:0/309949412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 msgr2=0x5606452ef750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6bb7fe640 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452ef750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6bb7fe640 1 -- 172.21.15.16:0/309949412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606452057b0 con 0x560644f39f60 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6ba7fc640 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452ef750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6bb7fe640 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f39f60 0x5606452efcb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40099e0 tx=0x7fa6b402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c6b97a8a5b9fc5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.424+0000 7fa6a27fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6a27fc640 1 -- 172.21.15.16:0/309949412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4003c40 con 0x560644f39f60 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6a27fc640 1 -- 172.21.15.16:0/309949412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa6b4003de0 con 0x560644f39f60 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6c797e740 1 -- 172.21.15.16:0/309949412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6a4012aa0 con 0x560644f39f60 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6a27fc640 1 -- 172.21.15.16:0/309949412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa6b4004040 con 0x560644f39f60 2024-08-28T19:20:50.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6c797e740 10 client.15507 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6c797e740 1 -- 172.21.15.16:0/309949412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5606452fdfb0 con 0x560644f39f60 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6a27fc640 1 -- 172.21.15.16:0/309949412 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6a4012aa0 con 0x560644f39f60 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6a27fc640 1 -- 172.21.15.16:0/309949412 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa6b4051070 con 0x560644f39f60 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6c797e740 10 client.15507 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6c797e740 10 client.15507 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6c797e740 20 client.15507 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6c797e740 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560644ef88e0 0x560645301b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6c797e740 4 client.15507 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.425+0000 7fa6c797e740 1 -- 172.21.15.16:0/309949412 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560645258340 con 0x560644ef88e0 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.426+0000 7fa6baffd640 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560644ef88e0 0x560645301b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.426+0000 7fa6baffd640 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560644ef88e0 0x560645301b10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6e1c68945df3d90 server_cookie=6609a10e6c34aa9a in_seq=0 out_seq=0 2024-08-28T19:20:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.426+0000 7fa6a27fc640 10 client.15507 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:20:50.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6a27fc640 1 -- 172.21.15.16:0/309949412 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560645258340 con 0x560644ef88e0 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6a27fc640 10 client.15507 handle_command_reply: tid=0 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 1 client.15507 shutdown 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 1 -- 172.21.15.16:0/309949412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f39f60 msgr2=0x5606452efcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f39f60 0x5606452efcb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40099e0 tx=0x7fa6b402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 1 -- 172.21.15.16:0/309949412 shutdown_connections 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f39f60 0x5606452efcb0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452ef750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560644ef88e0 0x560645301b10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 1 --2- 172.21.15.16:0/309949412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f110a0 0x5606452ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 1 -- 172.21.15.16:0/309949412 >> 172.21.15.16:0/309949412 conn(0x560644f2e350 msgr2=0x5606452e11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6a27fc640 0 client.15507 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:20:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 1 -- 172.21.15.16:0/309949412 shutdown_connections 2024-08-28T19:20:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 1 -- 172.21.15.16:0/309949412 wait complete. 2024-08-28T19:20:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.427+0000 7fa6c797e740 20 client.15507 trim_cache size 0 max 16384 2024-08-28T19:20:50.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.445+0000 7fa6c797e740 1 Processor -- start 2024-08-28T19:20:50.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.445+0000 7fa6c797e740 1 -- start start 2024-08-28T19:20:50.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452dd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x5606452ec400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6c797e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5606452ffe50 con 0x560644f110a0 2024-08-28T19:20:50.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6c797e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560645300520 con 0x560644f19280 2024-08-28T19:20:50.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6c797e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560645300bf0 con 0x560644f39f60 2024-08-28T19:20:50.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dcfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33496/0 (socket says 172.21.15.16:33496) 2024-08-28T19:20:50.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6baffd640 1 -- 172.21.15.16:0/4087159119 learned_addr learned my addr 172.21.15.16:0/4087159119 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6ba7fc640 1 --2- 172.21.15.16:0/4087159119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452dd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6bb7fe640 1 --2- 172.21.15.16:0/4087159119 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x5606452ec400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6baffd640 1 -- 172.21.15.16:0/4087159119 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 msgr2=0x5606452ec400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6baffd640 1 --2- 172.21.15.16:0/4087159119 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x5606452ec400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6baffd640 1 -- 172.21.15.16:0/4087159119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 msgr2=0x5606452dd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6baffd640 1 --2- 172.21.15.16:0/4087159119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452dd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6baffd640 1 -- 172.21.15.16:0/4087159119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606452e05f0 con 0x560644f110a0 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6ba7fc640 1 --2- 172.21.15.16:0/4087159119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452dd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.446+0000 7fa6baffd640 1 --2- 172.21.15.16:0/4087159119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dcfe0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x5606452de430 tx=0x7fa6a4011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d33573c367c9330 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.447+0000 7fa6b0ff9640 1 -- 172.21.15.16:0/4087159119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4004020 con 0x560644f110a0 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.447+0000 7fa6b0ff9640 1 -- 172.21.15.16:0/4087159119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6a400fbc0 con 0x560644f110a0 2024-08-28T19:20:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.447+0000 7fa6b0ff9640 1 -- 172.21.15.16:0/4087159119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a40052a0 con 0x560644f110a0 2024-08-28T19:20:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.447+0000 7fa6c797e740 1 -- 172.21.15.16:0/4087159119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 msgr2=0x5606452dcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.447+0000 7fa6c797e740 1 --2- 172.21.15.16:0/4087159119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dcfe0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x5606452de430 tx=0x7fa6a4011de0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.447+0000 7fa6c797e740 1 -- 172.21.15.16:0/4087159119 shutdown_connections 2024-08-28T19:20:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.447+0000 7fa6c797e740 1 --2- 172.21.15.16:0/4087159119 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x5606452ec400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.447+0000 7fa6c797e740 1 --2- 172.21.15.16:0/4087159119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f19280 0x5606452dd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.447+0000 7fa6c797e740 1 --2- 172.21.15.16:0/4087159119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f110a0 0x5606452dcfe0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.447+0000 7fa6c797e740 1 -- 172.21.15.16:0/4087159119 >> 172.21.15.16:0/4087159119 conn(0x560644f2e350 msgr2=0x5606452eda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6c797e740 1 -- 172.21.15.16:0/4087159119 shutdown_connections 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6c797e740 1 -- 172.21.15.16:0/4087159119 wait complete. 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6c797e740 1 -- start start 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6c797e740 1 Processor -- start 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f110a0 0x56064530a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f19280 0x56064530a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6c797e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x560645303590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6c797e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560645301990 con 0x560644f19280 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6c797e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5606453012c0 con 0x560644f110a0 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6c797e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560645302060 con 0x560644f39f60 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6c797e740 10 client.0 _ll_register_callbacks cb 0x7fa6b4031f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:20:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6ba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f19280 0x56064530a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6ba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f19280 0x56064530a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33504/0 (socket says 172.21.15.16:33504) 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6ba7fc640 1 -- 172.21.15.16:0/3142096095 learned_addr learned my addr 172.21.15.16:0/3142096095 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.448+0000 7fa6baffd640 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f110a0 0x56064530a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6bb7fe640 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x560645303590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6ba7fc640 1 -- 172.21.15.16:0/3142096095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 msgr2=0x560645303590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6ba7fc640 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x560645303590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6ba7fc640 1 -- 172.21.15.16:0/3142096095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f110a0 msgr2=0x56064530a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6ba7fc640 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f110a0 0x56064530a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6ba7fc640 1 -- 172.21.15.16:0/3142096095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560645303d20 con 0x560644f19280 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6ba7fc640 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f19280 0x56064530a680 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00d720 tx=0x7fa6ac0e8ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7884a18ab52ad9e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6a27fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6a27fc640 1 -- 172.21.15.16:0/3142096095 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac020040 con 0x560644f19280 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6a27fc640 1 -- 172.21.15.16:0/3142096095 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa6ac005680 con 0x560644f19280 2024-08-28T19:20:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6c797e740 1 -- 172.21.15.16:0/3142096095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6a400fbc0 con 0x560644f19280 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6a27fc640 1 -- 172.21.15.16:0/3142096095 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa6ac0ab100 con 0x560644f19280 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6c797e740 10 client.15513 fetch_fsmap learned FSMap version 28 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.449+0000 7fa6c797e740 1 -- 172.21.15.16:0/3142096095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5606453097d0 con 0x560644f19280 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6a27fc640 1 -- 172.21.15.16:0/3142096095 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6a400fbc0 con 0x560644f19280 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6a27fc640 1 -- 172.21.15.16:0/3142096095 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa6ac0d2190 con 0x560644f19280 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6c797e740 10 client.15513 fetch_fsmap finished waiting for FSMap version 28 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6c797e740 10 client.15513 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6c797e740 20 client.15513 populate_metadata read hostname 'smithi016' 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6c797e740 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560644ecdc90 0x560645370390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6c797e740 4 client.15513 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6c797e740 1 -- 172.21.15.16:0/3142096095 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "status"}) -- 0x560645253de0 con 0x560644ecdc90 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6baffd640 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560644ecdc90 0x560645370390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6baffd640 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560644ecdc90 0x560645370390 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e469a9d21880d35 server_cookie=81a4b1284044fe16 in_seq=0 out_seq=0 2024-08-28T19:20:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6a27fc640 10 client.15513 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6a27fc640 1 -- 172.21.15.16:0/3142096095 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x560645253de0 con 0x560644ecdc90 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.450+0000 7fa6a27fc640 10 client.15513 handle_command_reply: tid=0 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 1 client.15513 shutdown 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 1 -- 172.21.15.16:0/3142096095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f19280 msgr2=0x56064530a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f19280 0x56064530a680 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00d720 tx=0x7fa6ac0e8ab0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 1 -- 172.21.15.16:0/3142096095 shutdown_connections 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560644ecdc90 0x560645370390 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560644f39f60 0x560645303590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560644f19280 0x56064530a680 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6a27fc640 0 client.15513 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 1 --2- 172.21.15.16:0/3142096095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560644f110a0 0x56064530a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 1 -- 172.21.15.16:0/3142096095 >> 172.21.15.16:0/3142096095 conn(0x560644f2e350 msgr2=0x5606452e07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 1 -- 172.21.15.16:0/3142096095 shutdown_connections 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 1 -- 172.21.15.16:0/3142096095 wait complete. 2024-08-28T19:20:50.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c797e740 20 client.15513 trim_cache size 0 max 16384 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24665, 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 136.976644822, 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 82, 2024-08-28T19:20:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 147.191685732 2024-08-28T19:20:50.459 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:20:50.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c1409640 1 -- 172.21.15.16:0/685701722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6a80b4b80 msgr2=0x7fa6a80b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:50.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c1409640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6a80b4b80 0x7fa6a80b7090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4006010 tx=0x7fa6b4009440 comp rx=0 tx=0).stop 2024-08-28T19:20:50.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c1409640 1 -- 172.21.15.16:0/685701722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc109260 msgr2=0x7fa6bc19a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:50.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c1409640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc109260 0x7fa6bc19a7c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00d750 tx=0x7fa6ac00dc10 comp rx=0 tx=0).stop 2024-08-28T19:20:50.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c1409640 1 -- 172.21.15.16:0/685701722 shutdown_connections 2024-08-28T19:20:50.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.451+0000 7fa6c1409640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6a80b4b80 0x7fa6a80b7090 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.452+0000 7fa6c1409640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6bc109bb0 0x7fa6bc19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.452+0000 7fa6c1409640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc109260 0x7fa6bc19a7c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.452+0000 7fa6c1409640 1 --2- 172.21.15.16:0/685701722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6bc103920 0x7fa6bc19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.452+0000 7fa6c1409640 1 -- 172.21.15.16:0/685701722 >> 172.21.15.16:0/685701722 conn(0x7fa6bc0ff370 msgr2=0x7fa6bc101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.452+0000 7fa6c1409640 1 -- 172.21.15.16:0/685701722 shutdown_connections 2024-08-28T19:20:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:50.452+0000 7fa6c1409640 1 -- 172.21.15.16:0/685701722 wait complete. 2024-08-28T19:20:51.185 INFO:teuthology.run_tasks:Running task kclient... 2024-08-28T19:20:51.199 INFO:tasks.kclient:Mounting kernel clients... 2024-08-28T19:20:51.199 INFO:tasks.kclient:config is {'client.0': {}} 2024-08-28T19:20:51.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:20:51.200 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-08-28T19:20:51.216 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-08-28T19:20:51.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:20:51.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-28T19:20:51.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:20:51.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-08-28T19:20:51.389 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-08-28T19:20:51.389 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-08-28T19:20:51.389 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-28T19:20:51.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs ls 2024-08-28T19:20:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:51 smithi016 ceph-mon[18556]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-08-28T19:20:51.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:51 smithi028 ceph-mon[21769]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-08-28T19:20:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:51 smithi115 ceph-mon[22184]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-08-28T19:20:51.644 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:52.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.144+0000 7fcaca3f3640 1 -- 172.21.15.16:0/278672072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4108ab0 msgr2=0x7fcac410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.144+0000 7fcaca3f3640 1 --2- 172.21.15.16:0/278672072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4108ab0 0x7fcac410f2e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fcabc009ca0 tx=0x7fcabc02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.145+0000 7fcaca3f3640 1 -- 172.21.15.16:0/278672072 shutdown_connections 2024-08-28T19:20:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.145+0000 7fcaca3f3640 1 --2- 172.21.15.16:0/278672072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4108ab0 0x7fcac410f2e0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.145+0000 7fcaca3f3640 1 --2- 172.21.15.16:0/278672072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcac4108160 0x7fcac4108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.145+0000 7fcaca3f3640 1 --2- 172.21.15.16:0/278672072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcac4102820 0x7fcac4102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.145+0000 7fcaca3f3640 1 -- 172.21.15.16:0/278672072 >> 172.21.15.16:0/278672072 conn(0x7fcac40794e0 msgr2=0x7fcac40798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.145+0000 7fcaca3f3640 1 -- 172.21.15.16:0/278672072 shutdown_connections 2024-08-28T19:20:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.145+0000 7fcaca3f3640 1 -- 172.21.15.16:0/278672072 wait complete. 2024-08-28T19:20:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.145+0000 7fcaca3f3640 1 Processor -- start 2024-08-28T19:20:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.145+0000 7fcaca3f3640 1 -- start start 2024-08-28T19:20:52.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcaca3f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4102820 0x7fcac419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:52.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcaca3f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcac4108160 0x7fcac419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:52.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcaca3f3640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcac4108ab0 0x7fcac419faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:52.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcaca3f3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcac4113840 con 0x7fcac4102820 2024-08-28T19:20:52.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcaca3f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcac41136c0 con 0x7fcac4108160 2024-08-28T19:20:52.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcaca3f3640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcac41139c0 con 0x7fcac4108ab0 2024-08-28T19:20:52.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcac3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4102820 0x7fcac419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:52.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcac3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4102820 0x7fcac419a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33508/0 (socket says 172.21.15.16:33508) 2024-08-28T19:20:52.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcac3fff640 1 -- 172.21.15.16:0/2660961710 learned_addr learned my addr 172.21.15.16:0/2660961710 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:52.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcac8990640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcac4108ab0 0x7fcac419faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcac37fe640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcac4108160 0x7fcac419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcac3fff640 1 -- 172.21.15.16:0/2660961710 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcac4108ab0 msgr2=0x7fcac419faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcac3fff640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcac4108ab0 0x7fcac419faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcac3fff640 1 -- 172.21.15.16:0/2660961710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcac4108160 msgr2=0x7fcac419a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcac3fff640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcac4108160 0x7fcac419a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcac3fff640 1 -- 172.21.15.16:0/2660961710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac41a0220 con 0x7fcac4102820 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.146+0000 7fcac3fff640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4102820 0x7fcac419a2b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcab400e9f0 tx=0x7fcab400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.147+0000 7fcac17fa640 1 -- 172.21.15.16:0/2660961710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab400cd80 con 0x7fcac4102820 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.147+0000 7fcac17fa640 1 -- 172.21.15.16:0/2660961710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcab40108e0 con 0x7fcac4102820 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.147+0000 7fcaca3f3640 1 -- 172.21.15.16:0/2660961710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac41a0510 con 0x7fcac4102820 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.147+0000 7fcac17fa640 1 -- 172.21.15.16:0/2660961710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab4018980 con 0x7fcac4102820 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.147+0000 7fcaca3f3640 1 -- 172.21.15.16:0/2660961710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac41a6c70 con 0x7fcac4102820 2024-08-28T19:20:52.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.148+0000 7fcaca3f3640 1 -- 172.21.15.16:0/2660961710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcac4103aa0 con 0x7fcac4102820 2024-08-28T19:20:52.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.150+0000 7fcac17fa640 1 -- 172.21.15.16:0/2660961710 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fcab4010430 con 0x7fcac4102820 2024-08-28T19:20:52.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.150+0000 7fcac17fa640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcaac0b4c00 0x7fcaac0b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:52.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.150+0000 7fcac37fe640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcaac0b4c00 0x7fcaac0b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:52.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.150+0000 7fcac17fa640 1 -- 172.21.15.16:0/2660961710 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fcab40e5d90 con 0x7fcac4102820 2024-08-28T19:20:52.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.151+0000 7fcac37fe640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcaac0b4c00 0x7fcaac0b7110 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcac4105ea0 tx=0x7fcab8005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:52.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.153+0000 7fcac17fa640 1 -- 172.21.15.16:0/2660961710 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fcab40ac380 con 0x7fcac4102820 2024-08-28T19:20:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.378+0000 7fcaca3f3640 1 -- 172.21.15.16:0/2660961710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcac410a8d0 con 0x7fcac4102820 2024-08-28T19:20:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.378+0000 7fcac17fa640 1 -- 172.21.15.16:0/2660961710 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7fcab40b0b40 con 0x7fcac4102820 2024-08-28T19:20:52.381 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-08-28T19:20:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 -- 172.21.15.16:0/2660961710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcaac0b4c00 msgr2=0x7fcaac0b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcaac0b4c00 0x7fcaac0b7110 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcac4105ea0 tx=0x7fcab8005f90 comp rx=0 tx=0).stop 2024-08-28T19:20:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 -- 172.21.15.16:0/2660961710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4102820 msgr2=0x7fcac419a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4102820 0x7fcac419a2b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcab400e9f0 tx=0x7fcab400eeb0 comp rx=0 tx=0).stop 2024-08-28T19:20:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 -- 172.21.15.16:0/2660961710 shutdown_connections 2024-08-28T19:20:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcac4108ab0 0x7fcac419faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcaac0b4c00 0x7fcaac0b7110 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcac4108160 0x7fcac419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 --2- 172.21.15.16:0/2660961710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4102820 0x7fcac419a2b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 -- 172.21.15.16:0/2660961710 >> 172.21.15.16:0/2660961710 conn(0x7fcac40794e0 msgr2=0x7fcac4104220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 -- 172.21.15.16:0/2660961710 shutdown_connections 2024-08-28T19:20:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:52.381+0000 7fcaca3f3640 1 -- 172.21.15.16:0/2660961710 wait complete. 2024-08-28T19:20:53.056 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-08-28T19:20:53.056 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-28T19:20:53.056 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-08-28T19:20:53.056 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-28T19:20:53.056 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-28T19:20:53.056 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-08-28T19:20:53.056 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-08-28T19:20:53.056 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-28T19:20:53.056 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-28T19:20:53.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:20:53.057 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-08-28T19:20:53.113 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-08-28T19:20:53.114 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-28T19:20:53.115 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-08-28T19:20:53.115 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-08-28T19:20:53.115 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-28T19:20:53.115 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42143sec preferred_lft 42143sec 2024-08-28T19:20:53.115 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::1876:14a0:73d4:8f28/64 scope link noprefixroute 2024-08-28T19:20:53.115 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-08-28T19:20:53.115 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-28T19:20:53.115 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-08-28T19:20:53.115 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-08-28T19:20:53.115 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-08-28T19:20:53.116 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-28T19:20:53.116 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-08-28T19:20:53.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-08-28T19:20:53.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-08-28T19:20:53.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-08-28T19:20:53.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-08-28T19:20:53.116 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-08-28T19:20:53.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-28T19:20:53.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:20:53.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:20:53.255 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-28T19:20:53.279 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-08-28T19:20:53.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:20:53.281 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-08-28T19:20:53.338 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-08-28T19:20:53.338 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-28T19:20:53.338 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-28T19:20:53.338 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-28T19:20:53.338 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-28T19:20:53.338 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-28T19:20:53.339 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-08-28T19:20:53.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-28T19:20:53.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-28T19:20:53.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-28T19:20:53.339 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-08-28T19:20:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-28T19:20:53.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:53 smithi016 ceph-mon[18556]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 511 B/s wr, 12 op/s 2024-08-28T19:20:53.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:53 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2660961710' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-08-28T19:20:53.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:20:53.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:20:53.466 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-08-28T19:20:53.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:20:53.481 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-08-28T19:20:53.536 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-28T19:20:53.536 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-08-28T19:20:53.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-28T19:20:53.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-08-28T19:20:53.537 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-08-28T19:20:53.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:53 smithi028 ceph-mon[21769]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 511 B/s wr, 12 op/s 2024-08-28T19:20:53.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:53 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2660961710' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-08-28T19:20:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:53 smithi115 ceph-mon[22184]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 511 B/s wr, 12 op/s 2024-08-28T19:20:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:53 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2660961710' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-08-28T19:20:53.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-28T19:20:53.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:20:53.655 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-08-28T19:20:53.655 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-28T19:20:53.656 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-08-28T19:20:53.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-08-28T19:20:53.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-08-28T19:20:53.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-08-28T19:20:53.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-08-28T19:20:53.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-08-28T19:20:53.656 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-08-28T19:20:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-28T19:20:53.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:20:53.771 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-28T19:20:53.771 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-08-28T19:20:53.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-08-28T19:20:53.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-08-28T19:20:53.771 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-08-28T19:20:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-28T19:20:53.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:20:53.848 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-28T19:20:53.849 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-28T19:20:53.864 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-28T19:20:53.864 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-28T19:20:53.864 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-28T19:20:53.919 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-08-28T19:20:53.920 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-08-28T19:20:53.920 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/d7e0e328-6597-4453-9c11-805312a036d5 2024-08-28T19:20:53.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:20:53.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/d7e0e328-6597-4453-9c11-805312a036d5 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-08-28T19:20:54.014 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-08-28T19:20:54.014 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-08-28T19:20:54.163 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-08-28T19:20:54.163 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-08-28T19:20:54.163 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.28:3300,172.21.15.115:3300" 2024-08-28T19:20:54.163 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with new device syntax: 0@eabc438a-6570-11ef-bcd4-c7b262605968.cephfs=/volumes/qa/sv_1/d7e0e328-6597-4453-9c11-805312a036d5 2024-08-28T19:20:54.163 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.16:3300/172.21.15.28:3300/172.21.15.115:3300" will pass to kernel 2024-08-28T19:20:54.164 INFO:teuthology.orchestra.run.smithi016.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-28T19:20:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: You just mounted a file system that supports labels which does not 2024-08-28T19:20:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-28T19:20:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: applications will generate AVC messages and not be allowed access to 2024-08-28T19:20:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-28T19:20:54.166 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-28T19:20:54.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:20:54.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-28T19:20:54.192 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-28T19:20:54.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-28T19:20:54.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:20:54.289 INFO:teuthology.orchestra.run.smithi016.stdout:client25199 2024-08-28T19:20:54.289 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:20:54.289 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-28T19:20:54.307 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-28T19:20:54.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-28T19:20:54.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:20:54.402 INFO:teuthology.orchestra.run.smithi016.stdout:client25199 2024-08-28T19:20:54.402 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-28T19:20:54.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-28T19:20:54.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:20:54.458 INFO:teuthology.orchestra.run.smithi016.stdout:client25199 2024-08-28T19:20:54.458 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/eabc438a-6570-11ef-bcd4-c7b262605968.client25199/status') 2024-08-28T19:20:54.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-28T19:20:54.517 INFO:teuthology.orchestra.run.smithi016.stdout:instance: client.25199 (3)172.21.15.16:0/1401403411 2024-08-28T19:20:54.517 INFO:teuthology.orchestra.run.smithi016.stdout:blocklisted: false 2024-08-28T19:20:54.517 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records in 2024-08-28T19:20:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records out 2024-08-28T19:20:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:71 bytes copied, 2.3171e-05 s, 3.1 MB/s 2024-08-28T19:20:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-28T19:20:54.525 INFO:teuthology.run_tasks:Running task exec... 2024-08-28T19:20:54.535 INFO:teuthology.task.exec:Executing custom commands... 2024-08-28T19:20:54.535 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-08-28T19:20:54.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-08-28T19:20:54.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.645+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2542805653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e901080b0 msgr2=0x7f6e901084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.645+0000 7f6e97ecc640 1 --2- 172.21.15.16:0/2542805653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e901080b0 0x7f6e901084c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009900 tx=0x7f6e8002a410 comp rx=0 tx=0).stop 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.645+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2542805653 shutdown_connections 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.645+0000 7f6e97ecc640 1 --2- 172.21.15.16:0/2542805653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e90108a00 0x7f6e9010f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.645+0000 7f6e97ecc640 1 --2- 172.21.15.16:0/2542805653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e901080b0 0x7f6e901084c0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.645+0000 7f6e97ecc640 1 --2- 172.21.15.16:0/2542805653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e90102780 0x7f6e90102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.645+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2542805653 >> 172.21.15.16:0/2542805653 conn(0x7f6e900fe180 msgr2=0x7f6e901005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.645+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2542805653 shutdown_connections 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.645+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2542805653 wait complete. 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e97ecc640 1 Processor -- start 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e97ecc640 1 -- start start 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e97ecc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e90102780 0x7f6e9019b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e97ecc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e901080b0 0x7f6e9019b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e97ecc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90108a00 0x7f6e9019bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e97ecc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6e90112690 con 0x7f6e90108a00 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e97ecc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e90112510 con 0x7f6e901080b0 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e97ecc640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6e90112810 con 0x7f6e90102780 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e96469640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90108a00 0x7f6e9019bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e96469640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90108a00 0x7f6e9019bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36940/0 (socket says 172.21.15.16:36940) 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e96469640 1 -- 172.21.15.16:0/2583865671 learned_addr learned my addr 172.21.15.16:0/2583865671 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e95c68640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e90102780 0x7f6e9019b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.646+0000 7f6e95467640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e901080b0 0x7f6e9019b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e96469640 1 -- 172.21.15.16:0/2583865671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e90102780 msgr2=0x7f6e9019b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e96469640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e90102780 0x7f6e9019b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e96469640 1 -- 172.21.15.16:0/2583865671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e901080b0 msgr2=0x7f6e9019b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e96469640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e901080b0 0x7f6e9019b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e96469640 1 -- 172.21.15.16:0/2583865671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e901a6300 con 0x7f6e90108a00 2024-08-28T19:20:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e95c68640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e90102780 0x7f6e9019b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e96469640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90108a00 0x7f6e9019bac0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800cad0 tx=0x7f6e8800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e86ffd640 1 -- 172.21.15.16:0/2583865671 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8800ae20 con 0x7f6e90108a00 2024-08-28T19:20:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2583865671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e901a65f0 con 0x7f6e90108a00 2024-08-28T19:20:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e86ffd640 1 -- 172.21.15.16:0/2583865671 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e88005020 con 0x7f6e90108a00 2024-08-28T19:20:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e86ffd640 1 -- 172.21.15.16:0/2583865671 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8800f6b0 con 0x7f6e90108a00 2024-08-28T19:20:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.647+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2583865671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e901a6b60 con 0x7f6e90108a00 2024-08-28T19:20:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.648+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2583865671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e90102b90 con 0x7f6e90108a00 2024-08-28T19:20:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.649+0000 7f6e86ffd640 1 -- 172.21.15.16:0/2583865671 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7f6e880051d0 con 0x7f6e90108a00 2024-08-28T19:20:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.650+0000 7f6e86ffd640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6e7c0b4b80 0x7f6e7c0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.650+0000 7f6e95c68640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6e7c0b4b80 0x7f6e7c0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.650+0000 7f6e86ffd640 1 -- 172.21.15.16:0/2583865671 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6e880e5b10 con 0x7f6e90108a00 2024-08-28T19:20:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.651+0000 7f6e95c68640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6e7c0b4b80 0x7f6e7c0b7090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c005d00 tx=0x7f6e8c005c90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.653+0000 7f6e86ffd640 1 -- 172.21.15.16:0/2583865671 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f6e88014030 con 0x7f6e90108a00 2024-08-28T19:20:54.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:54.878+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2583865671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f6e9002e430 con 0x7f6e90108a00 2024-08-28T19:20:55.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.257+0000 7f6e86ffd640 1 -- 172.21.15.16:0/2583865671 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v29) ==== 117+0+0 (secure 0 0 0) 0x7f6e880ac750 con 0x7f6e90108a00 2024-08-28T19:20:55.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2583865671 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6e7c0b4b80 msgr2=0x7f6e7c0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:55.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6e7c0b4b80 0x7f6e7c0b7090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c005d00 tx=0x7f6e8c005c90 comp rx=0 tx=0).stop 2024-08-28T19:20:55.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2583865671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90108a00 msgr2=0x7f6e9019bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:55.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90108a00 0x7f6e9019bac0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800cad0 tx=0x7f6e8800cf90 comp rx=0 tx=0).stop 2024-08-28T19:20:55.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2583865671 shutdown_connections 2024-08-28T19:20:55.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90108a00 0x7f6e9019bac0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6e7c0b4b80 0x7f6e7c0b7090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e901080b0 0x7f6e9019b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 --2- 172.21.15.16:0/2583865671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e90102780 0x7f6e9019b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2583865671 >> 172.21.15.16:0/2583865671 conn(0x7f6e900fe180 msgr2=0x7f6e90107770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:55.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2583865671 shutdown_connections 2024-08-28T19:20:55.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.259+0000 7f6e97ecc640 1 -- 172.21.15.16:0/2583865671 wait complete. 2024-08-28T19:20:55.274 INFO:teuthology.run_tasks:Running task check-counter... 2024-08-28T19:20:55.285 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-08-28T19:20:55.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-28T19:20:55.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.476+0000 7f5d16884640 1 -- 172.21.15.16:0/3007729696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d10102740 msgr2=0x7f5d10102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.476+0000 7f5d16884640 1 --2- 172.21.15.16:0/3007729696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d10102740 0x7f5d10102b50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009900 tx=0x7f5d0002a410 comp rx=0 tx=0).stop 2024-08-28T19:20:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.476+0000 7f5d16884640 1 -- 172.21.15.16:0/3007729696 shutdown_connections 2024-08-28T19:20:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.476+0000 7f5d16884640 1 --2- 172.21.15.16:0/3007729696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d101089c0 0x7f5d1010f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.476+0000 7f5d16884640 1 --2- 172.21.15.16:0/3007729696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d10108070 0x7f5d10108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.476+0000 7f5d16884640 1 --2- 172.21.15.16:0/3007729696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d10102740 0x7f5d10102b50 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.476+0000 7f5d16884640 1 -- 172.21.15.16:0/3007729696 >> 172.21.15.16:0/3007729696 conn(0x7f5d100fe180 msgr2=0x7f5d101005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.476+0000 7f5d16884640 1 -- 172.21.15.16:0/3007729696 shutdown_connections 2024-08-28T19:20:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.476+0000 7f5d16884640 1 -- 172.21.15.16:0/3007729696 wait complete. 2024-08-28T19:20:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.476+0000 7f5d16884640 1 Processor -- start 2024-08-28T19:20:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.476+0000 7f5d16884640 1 -- start start 2024-08-28T19:20:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d16884640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d10102740 0x7f5d1010fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d16884640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d10108070 0x7f5d10110210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d16884640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d101089c0 0x7f5d10110750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d16884640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d101137a0 con 0x7f5d10108070 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d16884640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d10113620 con 0x7f5d101089c0 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d16884640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5d10113920 con 0x7f5d10102740 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d10108070 0x7f5d10110210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d10108070 0x7f5d10110210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36958/0 (socket says 172.21.15.16:36958) 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0f7fe640 1 -- 172.21.15.16:0/3672177276 learned_addr learned my addr 172.21.15.16:0/3672177276 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0ffff640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d10102740 0x7f5d1010fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d14e21640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d101089c0 0x7f5d10110750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0f7fe640 1 -- 172.21.15.16:0/3672177276 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d10102740 msgr2=0x7f5d1010fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0f7fe640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d10102740 0x7f5d1010fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0f7fe640 1 -- 172.21.15.16:0/3672177276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d101089c0 msgr2=0x7f5d10110750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0f7fe640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d101089c0 0x7f5d10110750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0f7fe640 1 -- 172.21.15.16:0/3672177276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d101a8490 con 0x7f5d10108070 2024-08-28T19:20:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0ffff640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d10102740 0x7f5d1010fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d14e21640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d101089c0 0x7f5d10110750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0f7fe640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d10108070 0x7f5d10110210 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400d750 tx=0x7f5d0400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.477+0000 7f5d0d7fa640 1 -- 172.21.15.16:0/3672177276 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d04017070 con 0x7f5d10108070 2024-08-28T19:20:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.478+0000 7f5d16884640 1 -- 172.21.15.16:0/3672177276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d101a8780 con 0x7f5d10108070 2024-08-28T19:20:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.478+0000 7f5d0d7fa640 1 -- 172.21.15.16:0/3672177276 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d040054d0 con 0x7f5d10108070 2024-08-28T19:20:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.478+0000 7f5d0d7fa640 1 -- 172.21.15.16:0/3672177276 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d04004c30 con 0x7f5d10108070 2024-08-28T19:20:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.478+0000 7f5d16884640 1 -- 172.21.15.16:0/3672177276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d101a8c80 con 0x7f5d10108070 2024-08-28T19:20:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.479+0000 7f5d16884640 1 -- 172.21.15.16:0/3672177276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d10102b50 con 0x7f5d10108070 2024-08-28T19:20:55.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.480+0000 7f5d0d7fa640 1 -- 172.21.15.16:0/3672177276 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7f5d04004dd0 con 0x7f5d10108070 2024-08-28T19:20:55.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.481+0000 7f5d0d7fa640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cf80b4bb0 0x7f5cf80b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:55.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.481+0000 7f5d0d7fa640 1 -- 172.21.15.16:0/3672177276 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f5d040e54c0 con 0x7f5d10108070 2024-08-28T19:20:55.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.481+0000 7f5d0ffff640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cf80b4bb0 0x7f5cf80b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:55.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.481+0000 7f5d0ffff640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cf80b4bb0 0x7f5cf80b70c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5d00000c00 tx=0x7f5d00009680 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:55.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.484+0000 7f5d0d7fa640 1 -- 172.21.15.16:0/3672177276 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f5d040ac160 con 0x7f5d10108070 2024-08-28T19:20:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:55 smithi028 ceph-mon[21769]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:55 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2583865671' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-08-28T19:20:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:55 smithi115 ceph-mon[22184]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:55 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2583865671' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-08-28T19:20:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.676+0000 7f5d16884640 1 -- 172.21.15.16:0/3672177276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d1002e430 con 0x7f5d10108070 2024-08-28T19:20:55.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.677+0000 7f5d0d7fa640 1 -- 172.21.15.16:0/3672177276 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+25180 (secure 0 0 0) 0x7f5d040aba60 con 0x7f5d10108070 2024-08-28T19:20:55.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:55.682 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:20:09.003260+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:20:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.679+0000 7f5d16884640 1 -- 172.21.15.16:0/3672177276 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cf80b4bb0 msgr2=0x7f5cf80b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.679+0000 7f5d16884640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cf80b4bb0 0x7f5cf80b70c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5d00000c00 tx=0x7f5d00009680 comp rx=0 tx=0).stop 2024-08-28T19:20:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.679+0000 7f5d16884640 1 -- 172.21.15.16:0/3672177276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d10108070 msgr2=0x7f5d10110210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.679+0000 7f5d16884640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d10108070 0x7f5d10110210 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400d750 tx=0x7f5d0400dc10 comp rx=0 tx=0).stop 2024-08-28T19:20:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.679+0000 7f5d16884640 1 -- 172.21.15.16:0/3672177276 shutdown_connections 2024-08-28T19:20:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.679+0000 7f5d16884640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cf80b4bb0 0x7f5cf80b70c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.679+0000 7f5d16884640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d101089c0 0x7f5d10110750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.679+0000 7f5d16884640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d10108070 0x7f5d10110210 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.679+0000 7f5d16884640 1 --2- 172.21.15.16:0/3672177276 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d10102740 0x7f5d1010fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.679+0000 7f5d16884640 1 -- 172.21.15.16:0/3672177276 >> 172.21.15.16:0/3672177276 conn(0x7f5d100fe180 msgr2=0x7f5d101005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.680+0000 7f5d16884640 1 -- 172.21.15.16:0/3672177276 shutdown_connections 2024-08-28T19:20:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.680+0000 7f5d16884640 1 -- 172.21.15.16:0/3672177276 wait complete. 2024-08-28T19:20:55.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:55 smithi016 ceph-mon[18556]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:20:55.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:55 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2583865671' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-08-28T19:20:55.695 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-28T19:13:55.828283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-08-28T19:18:29.423422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-08-28T19:18:31.212294+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-08-28T19:18:37.500166+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-08-28T19:20:55.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-28T19:20:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.809+0000 7f6aebb67640 1 -- 172.21.15.16:0/1711994188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae4108a10 msgr2=0x7f6ae410f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.809+0000 7f6aebb67640 1 --2- 172.21.15.16:0/1711994188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae4108a10 0x7f6ae410f240 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009ca0 tx=0x7f6adc02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.809+0000 7f6aebb67640 1 -- 172.21.15.16:0/1711994188 shutdown_connections 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.809+0000 7f6aebb67640 1 --2- 172.21.15.16:0/1711994188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae4108a10 0x7f6ae410f240 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.809+0000 7f6aebb67640 1 --2- 172.21.15.16:0/1711994188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae41080c0 0x7f6ae41084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.809+0000 7f6aebb67640 1 --2- 172.21.15.16:0/1711994188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ae4102790 0x7f6ae4102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.809+0000 7f6aebb67640 1 -- 172.21.15.16:0/1711994188 >> 172.21.15.16:0/1711994188 conn(0x7f6ae40fe190 msgr2=0x7f6ae41005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.809+0000 7f6aebb67640 1 -- 172.21.15.16:0/1711994188 shutdown_connections 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.809+0000 7f6aebb67640 1 -- 172.21.15.16:0/1711994188 wait complete. 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aebb67640 1 Processor -- start 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aebb67640 1 -- start start 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aebb67640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ae4102790 0x7f6ae419af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aebb67640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae41080c0 0x7f6ae419b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aebb67640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae4108a10 0x7f6ae419ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aebb67640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae41126e0 con 0x7f6ae4108a10 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aebb67640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae4112560 con 0x7f6ae41080c0 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aebb67640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae4112860 con 0x7f6ae4102790 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aea104640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae4108a10 0x7f6ae419ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aea104640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae4108a10 0x7f6ae419ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36964/0 (socket says 172.21.15.16:36964) 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aea104640 1 -- 172.21.15.16:0/3239594286 learned_addr learned my addr 172.21.15.16:0/3239594286 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6ae9903640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ae4102790 0x7f6ae419af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6ae9102640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae41080c0 0x7f6ae419b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aea104640 1 -- 172.21.15.16:0/3239594286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ae4102790 msgr2=0x7f6ae419af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aea104640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ae4102790 0x7f6ae419af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aea104640 1 -- 172.21.15.16:0/3239594286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae41080c0 msgr2=0x7f6ae419b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aea104640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae41080c0 0x7f6ae419b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.810+0000 7f6aea104640 1 -- 172.21.15.16:0/3239594286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae41a62a0 con 0x7f6ae4108a10 2024-08-28T19:20:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.811+0000 7f6aea104640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae4108a10 0x7f6ae419ba10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6adc005cc0 tx=0x7f6adc031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.811+0000 7f6adaffd640 1 -- 172.21.15.16:0/3239594286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc003b70 con 0x7f6ae4108a10 2024-08-28T19:20:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.811+0000 7f6aebb67640 1 -- 172.21.15.16:0/3239594286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae41a6530 con 0x7f6ae4108a10 2024-08-28T19:20:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.811+0000 7f6adaffd640 1 -- 172.21.15.16:0/3239594286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6adc0318f0 con 0x7f6ae4108a10 2024-08-28T19:20:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.811+0000 7f6adaffd640 1 -- 172.21.15.16:0/3239594286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc02f950 con 0x7f6ae4108a10 2024-08-28T19:20:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.811+0000 7f6aebb67640 1 -- 172.21.15.16:0/3239594286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae41a6a80 con 0x7f6ae4108a10 2024-08-28T19:20:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.812+0000 7f6aebb67640 1 -- 172.21.15.16:0/3239594286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae4102ba0 con 0x7f6ae4108a10 2024-08-28T19:20:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.814+0000 7f6adaffd640 1 -- 172.21.15.16:0/3239594286 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7f6adc02faf0 con 0x7f6ae4108a10 2024-08-28T19:20:55.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.815+0000 7f6adaffd640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ad00b4b80 0x7f6ad00b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:55.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.815+0000 7f6ae9903640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ad00b4b80 0x7f6ad00b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:55.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.815+0000 7f6adaffd640 1 -- 172.21.15.16:0/3239594286 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6adc106ae0 con 0x7f6ae4108a10 2024-08-28T19:20:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.816+0000 7f6ae9903640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ad00b4b80 0x7f6ad00b7090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6ae406bde0 tx=0x7f6ae0005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:55.818+0000 7f6adaffd640 1 -- 172.21.15.16:0/3239594286 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f6adc0cd720 con 0x7f6ae4108a10 2024-08-28T19:20:56.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.005+0000 7f6aebb67640 1 -- 172.21.15.16:0/3239594286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6ae402e430 con 0x7f6ae4108a10 2024-08-28T19:20:56.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.006+0000 7f6adaffd640 1 -- 172.21.15.16:0/3239594286 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7f6adc0cd020 con 0x7f6ae4108a10 2024-08-28T19:20:56.008 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-08-28T19:20:56.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 -- 172.21.15.16:0/3239594286 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ad00b4b80 msgr2=0x7f6ad00b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ad00b4b80 0x7f6ad00b7090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6ae406bde0 tx=0x7f6ae0005f90 comp rx=0 tx=0).stop 2024-08-28T19:20:56.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 -- 172.21.15.16:0/3239594286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae4108a10 msgr2=0x7f6ae419ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae4108a10 0x7f6ae419ba10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6adc005cc0 tx=0x7f6adc031700 comp rx=0 tx=0).stop 2024-08-28T19:20:56.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 -- 172.21.15.16:0/3239594286 shutdown_connections 2024-08-28T19:20:56.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae4108a10 0x7f6ae419ba10 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ad00b4b80 0x7f6ad00b7090 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae41080c0 0x7f6ae419b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 --2- 172.21.15.16:0/3239594286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ae4102790 0x7f6ae419af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 -- 172.21.15.16:0/3239594286 >> 172.21.15.16:0/3239594286 conn(0x7f6ae40fe190 msgr2=0x7f6ae41076b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:56.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 -- 172.21.15.16:0/3239594286 shutdown_connections 2024-08-28T19:20:56.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.008+0000 7f6aebb67640 1 -- 172.21.15.16:0/3239594286 wait complete. 2024-08-28T19:20:56.022 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-08-28T19:20:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.134+0000 7f835958c640 1 -- 172.21.15.16:0/615774942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8354104680 msgr2=0x7f8354104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.134+0000 7f835958c640 1 --2- 172.21.15.16:0/615774942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8354104680 0x7f8354104a90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8344009ca0 tx=0x7f834402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.134+0000 7f835958c640 1 -- 172.21.15.16:0/615774942 shutdown_connections 2024-08-28T19:20:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.134+0000 7f835958c640 1 --2- 172.21.15.16:0/615774942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f835410dae0 0x7f835410ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.134+0000 7f835958c640 1 --2- 172.21.15.16:0/615774942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8354104fd0 0x7f835410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.134+0000 7f835958c640 1 --2- 172.21.15.16:0/615774942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8354104680 0x7f8354104a90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.134+0000 7f835958c640 1 -- 172.21.15.16:0/615774942 >> 172.21.15.16:0/615774942 conn(0x7f83540fe150 msgr2=0x7f83541005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.134+0000 7f835958c640 1 -- 172.21.15.16:0/615774942 shutdown_connections 2024-08-28T19:20:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.134+0000 7f835958c640 1 -- 172.21.15.16:0/615774942 wait complete. 2024-08-28T19:20:56.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f835958c640 1 Processor -- start 2024-08-28T19:20:56.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f835958c640 1 -- start start 2024-08-28T19:20:56.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f835958c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8354104680 0x7f835419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f835958c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8354104fd0 0x7f835419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f835958c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f835410dae0 0x7f835419fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f835958c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8354113750 con 0x7f835410dae0 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f835958c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83541135d0 con 0x7f8354104fd0 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f835958c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f83541138d0 con 0x7f8354104680 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f83537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f835410dae0 0x7f835419fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f83537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f835410dae0 0x7f835419fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36978/0 (socket says 172.21.15.16:36978) 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f83537fe640 1 -- 172.21.15.16:0/3809298501 learned_addr learned my addr 172.21.15.16:0/3809298501 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f83527fc640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8354104fd0 0x7f835419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f8352ffd640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8354104680 0x7f835419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f83537fe640 1 -- 172.21.15.16:0/3809298501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8354104680 msgr2=0x7f835419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f83537fe640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8354104680 0x7f835419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f83537fe640 1 -- 172.21.15.16:0/3809298501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8354104fd0 msgr2=0x7f835419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f83537fe640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8354104fd0 0x7f835419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.135+0000 7f83537fe640 1 -- 172.21.15.16:0/3809298501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83541a0120 con 0x7f835410dae0 2024-08-28T19:20:56.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.136+0000 7f8352ffd640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8354104680 0x7f835419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:20:56.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.136+0000 7f83537fe640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f835410dae0 0x7f835419fa20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f834c00d950 tx=0x7f834c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:56.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.136+0000 7f833bfff640 1 -- 172.21.15.16:0/3809298501 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c017030 con 0x7f835410dae0 2024-08-28T19:20:56.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.136+0000 7f835958c640 1 -- 172.21.15.16:0/3809298501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83541a0410 con 0x7f835410dae0 2024-08-28T19:20:56.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.136+0000 7f833bfff640 1 -- 172.21.15.16:0/3809298501 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f834c004020 con 0x7f835410dae0 2024-08-28T19:20:56.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.136+0000 7f833bfff640 1 -- 172.21.15.16:0/3809298501 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c003b50 con 0x7f835410dae0 2024-08-28T19:20:56.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.136+0000 7f835958c640 1 -- 172.21.15.16:0/3809298501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83541a6ba0 con 0x7f835410dae0 2024-08-28T19:20:56.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.137+0000 7f835958c640 1 -- 172.21.15.16:0/3809298501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8354104a90 con 0x7f835410dae0 2024-08-28T19:20:56.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.139+0000 7f833bfff640 1 -- 172.21.15.16:0/3809298501 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7f834c01a070 con 0x7f835410dae0 2024-08-28T19:20:56.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.140+0000 7f833bfff640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f833c0b4b80 0x7f833c0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.140+0000 7f8352ffd640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f833c0b4b80 0x7f833c0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.140+0000 7f833bfff640 1 -- 172.21.15.16:0/3809298501 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f834c0e5170 con 0x7f835410dae0 2024-08-28T19:20:56.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.141+0000 7f8352ffd640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f833c0b4b80 0x7f833c0b7090 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8344006010 tx=0x7f8344000f40 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:56.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.143+0000 7f833bfff640 1 -- 172.21.15.16:0/3809298501 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f834c021070 con 0x7f835410dae0 2024-08-28T19:20:56.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.329+0000 7f835958c640 1 -- 172.21.15.16:0/3809298501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f83541a6e40 con 0x7f835410dae0 2024-08-28T19:20:56.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.329+0000 7f833bfff640 1 -- 172.21.15.16:0/3809298501 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) ==== 104+0+11 (secure 0 0 0) 0x7f834c0abe10 con 0x7f835410dae0 2024-08-28T19:20:56.332 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-08-28T19:20:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 -- 172.21.15.16:0/3809298501 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f833c0b4b80 msgr2=0x7f833c0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f833c0b4b80 0x7f833c0b7090 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8344006010 tx=0x7f8344000f40 comp rx=0 tx=0).stop 2024-08-28T19:20:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 -- 172.21.15.16:0/3809298501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f835410dae0 msgr2=0x7f835419fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f835410dae0 0x7f835419fa20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f834c00d950 tx=0x7f834c00de10 comp rx=0 tx=0).stop 2024-08-28T19:20:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 -- 172.21.15.16:0/3809298501 shutdown_connections 2024-08-28T19:20:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f835410dae0 0x7f835419fa20 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f833c0b4b80 0x7f833c0b7090 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8354104fd0 0x7f835419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 --2- 172.21.15.16:0/3809298501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8354104680 0x7f835419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 -- 172.21.15.16:0/3809298501 >> 172.21.15.16:0/3809298501 conn(0x7f83540fe150 msgr2=0x7f83541075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 -- 172.21.15.16:0/3809298501 shutdown_connections 2024-08-28T19:20:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.331+0000 7f835958c640 1 -- 172.21.15.16:0/3809298501 wait complete. 2024-08-28T19:20:56.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:56 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2583865671' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-08-28T19:20:56.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:56 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:20:56.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:56 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3672177276' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:20:56.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:56 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3239594286' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-28T19:20:56.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-08-28T19:20:56.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.457+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/4144273901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff810db60 msgr2=0x7f8ff8110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.457+0000 7f8ffd4f0640 1 --2- 172.21.15.16:0/4144273901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff810db60 0x7f8ff8110000 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009ca0 tx=0x7f8ff002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.457+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/4144273901 shutdown_connections 2024-08-28T19:20:56.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.457+0000 7f8ffd4f0640 1 --2- 172.21.15.16:0/4144273901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff810db60 0x7f8ff8110000 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.457+0000 7f8ffd4f0640 1 --2- 172.21.15.16:0/4144273901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff8104fe0 0x7f8ff810d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.457+0000 7f8ffd4f0640 1 --2- 172.21.15.16:0/4144273901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff81045b0 0x7f8ff81049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.457+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/4144273901 >> 172.21.15.16:0/4144273901 conn(0x7f8ff80793d0 msgr2=0x7f8ff80797e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:56.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.457+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/4144273901 shutdown_connections 2024-08-28T19:20:56.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.457+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/4144273901 wait complete. 2024-08-28T19:20:56.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ffd4f0640 1 Processor -- start 2024-08-28T19:20:56.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ffd4f0640 1 -- start start 2024-08-28T19:20:56.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ffd4f0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff81045b0 0x7f8ff819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ffd4f0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff8104fe0 0x7f8ff819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ffd4f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff810db60 0x7f8ff819f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ffd4f0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8113780 con 0x7f8ff810db60 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ffd4f0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8113600 con 0x7f8ff81045b0 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ffd4f0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8113900 con 0x7f8ff8104fe0 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ff77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff810db60 0x7f8ff819f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ff77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff810db60 0x7f8ff819f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36986/0 (socket says 172.21.15.16:36986) 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ff77fe640 1 -- 172.21.15.16:0/1971204652 learned_addr learned my addr 172.21.15.16:0/1971204652 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ff6ffd640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff81045b0 0x7f8ff819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ff67fc640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff8104fe0 0x7f8ff819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ff77fe640 1 -- 172.21.15.16:0/1971204652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff8104fe0 msgr2=0x7f8ff819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ff77fe640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff8104fe0 0x7f8ff819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ff77fe640 1 -- 172.21.15.16:0/1971204652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff81045b0 msgr2=0x7f8ff819a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.458+0000 7f8ff77fe640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff81045b0 0x7f8ff819a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.459+0000 7f8ff77fe640 1 -- 172.21.15.16:0/1971204652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff81a00b0 con 0x7f8ff810db60 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.459+0000 7f8ff77fe640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff810db60 0x7f8ff819f9b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0005cc0 tx=0x7f8ff002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:56.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.459+0000 7f8fdffff640 1 -- 172.21.15.16:0/1971204652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0003bf0 con 0x7f8ff810db60 2024-08-28T19:20:56.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.459+0000 7f8fdffff640 1 -- 172.21.15.16:0/1971204652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ff0003d90 con 0x7f8ff810db60 2024-08-28T19:20:56.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.459+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/1971204652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff81a0340 con 0x7f8ff810db60 2024-08-28T19:20:56.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.459+0000 7f8fdffff640 1 -- 172.21.15.16:0/1971204652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff002fbc0 con 0x7f8ff810db60 2024-08-28T19:20:56.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.459+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/1971204652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a6b30 con 0x7f8ff810db60 2024-08-28T19:20:56.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.460+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/1971204652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff81049c0 con 0x7f8ff810db60 2024-08-28T19:20:56.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.462+0000 7f8fdffff640 1 -- 172.21.15.16:0/1971204652 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7f8ff002fd60 con 0x7f8ff810db60 2024-08-28T19:20:56.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.463+0000 7f8fdffff640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe00b4b80 0x7f8fe00b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.463+0000 7f8ff6ffd640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe00b4b80 0x7f8fe00b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.463+0000 7f8fdffff640 1 -- 172.21.15.16:0/1971204652 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f8ff0106ce0 con 0x7f8ff810db60 2024-08-28T19:20:56.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.463+0000 7f8ff6ffd640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe00b4b80 0x7f8fe00b7090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800bbe0 tx=0x7f8fe800b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:56.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.466+0000 7f8fdffff640 1 -- 172.21.15.16:0/1971204652 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8ff00cd980 con 0x7f8ff810db60 2024-08-28T19:20:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:56 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2583865671' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-08-28T19:20:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:56 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:20:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:56 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3672177276' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:20:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:56 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3239594286' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-28T19:20:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:56 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2583865671' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-08-28T19:20:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:56 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:20:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:56 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3672177276' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:20:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:56 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3239594286' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-28T19:20:56.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.658+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/1971204652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8ff8108cd0 con 0x7f8ff810db60 2024-08-28T19:20:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.658+0000 7f8fdffff640 1 -- 172.21.15.16:0/1971204652 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) ==== 100+0+11 (secure 0 0 0) 0x7f8ff00098b0 con 0x7f8ff810db60 2024-08-28T19:20:56.661 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-08-28T19:20:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/1971204652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe00b4b80 msgr2=0x7f8fe00b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe00b4b80 0x7f8fe00b7090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800bbe0 tx=0x7f8fe800b390 comp rx=0 tx=0).stop 2024-08-28T19:20:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/1971204652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff810db60 msgr2=0x7f8ff819f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff810db60 0x7f8ff819f9b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0005cc0 tx=0x7f8ff002f380 comp rx=0 tx=0).stop 2024-08-28T19:20:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/1971204652 shutdown_connections 2024-08-28T19:20:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff810db60 0x7f8ff819f9b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff8104fe0 0x7f8ff819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe00b4b80 0x7f8fe00b7090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 --2- 172.21.15.16:0/1971204652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff81045b0 0x7f8ff819a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/1971204652 >> 172.21.15.16:0/1971204652 conn(0x7f8ff80793d0 msgr2=0x7f8ff80ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/1971204652 shutdown_connections 2024-08-28T19:20:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.661+0000 7f8ffd4f0640 1 -- 172.21.15.16:0/1971204652 wait complete. 2024-08-28T19:20:56.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-08-28T19:20:56.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.786+0000 7fc5d44bb640 1 -- 172.21.15.16:0/3752015576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5cc101d80 msgr2=0x7fc5cc102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.786+0000 7fc5d44bb640 1 --2- 172.21.15.16:0/3752015576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5cc101d80 0x7fc5cc102190 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009900 tx=0x7fc5c802a410 comp rx=0 tx=0).stop 2024-08-28T19:20:56.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.786+0000 7fc5d44bb640 1 -- 172.21.15.16:0/3752015576 shutdown_connections 2024-08-28T19:20:56.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.786+0000 7fc5d44bb640 1 --2- 172.21.15.16:0/3752015576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5cc1068b0 0x7fc5cc110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.786+0000 7fc5d44bb640 1 --2- 172.21.15.16:0/3752015576 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5cc105f60 0x7fc5cc106370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.786+0000 7fc5d44bb640 1 --2- 172.21.15.16:0/3752015576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5cc101d80 0x7fc5cc102190 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.786+0000 7fc5d44bb640 1 -- 172.21.15.16:0/3752015576 >> 172.21.15.16:0/3752015576 conn(0x7fc5cc079bb0 msgr2=0x7fc5cc079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:56.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.786+0000 7fc5d44bb640 1 -- 172.21.15.16:0/3752015576 shutdown_connections 2024-08-28T19:20:56.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.786+0000 7fc5d44bb640 1 -- 172.21.15.16:0/3752015576 wait complete. 2024-08-28T19:20:56.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.787+0000 7fc5d44bb640 1 Processor -- start 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.787+0000 7fc5d44bb640 1 -- start start 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.787+0000 7fc5d44bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5cc101d80 0x7fc5cc19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.787+0000 7fc5d44bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5cc105f60 0x7fc5cc19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.787+0000 7fc5d44bb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5cc1068b0 0x7fc5cc19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.787+0000 7fc5d44bb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc1137b0 con 0x7fc5cc105f60 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.787+0000 7fc5d44bb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc113630 con 0x7fc5cc101d80 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.787+0000 7fc5d44bb640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc113930 con 0x7fc5cc1068b0 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.787+0000 7fc5d1a56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5cc105f60 0x7fc5cc19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d1a56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5cc105f60 0x7fc5cc19a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37004/0 (socket says 172.21.15.16:37004) 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d1a56640 1 -- 172.21.15.16:0/909627237 learned_addr learned my addr 172.21.15.16:0/909627237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d2257640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5cc101d80 0x7fc5cc19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d2a58640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5cc1068b0 0x7fc5cc19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d1a56640 1 -- 172.21.15.16:0/909627237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5cc1068b0 msgr2=0x7fc5cc19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d1a56640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5cc1068b0 0x7fc5cc19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d1a56640 1 -- 172.21.15.16:0/909627237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5cc101d80 msgr2=0x7fc5cc19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d1a56640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5cc101d80 0x7fc5cc19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d1a56640 1 -- 172.21.15.16:0/909627237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5cc1a0160 con 0x7fc5cc105f60 2024-08-28T19:20:56.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d1a56640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5cc105f60 0x7fc5cc19a7d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc00c970 tx=0x7fc5bc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:56.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5c37fe640 1 -- 172.21.15.16:0/909627237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc004220 con 0x7fc5cc105f60 2024-08-28T19:20:56.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5c37fe640 1 -- 172.21.15.16:0/909627237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5bc0043c0 con 0x7fc5cc105f60 2024-08-28T19:20:56.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d44bb640 1 -- 172.21.15.16:0/909627237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5cc1a0450 con 0x7fc5cc105f60 2024-08-28T19:20:56.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5c37fe640 1 -- 172.21.15.16:0/909627237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc004a00 con 0x7fc5cc105f60 2024-08-28T19:20:56.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.788+0000 7fc5d44bb640 1 -- 172.21.15.16:0/909627237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5cc1a6be0 con 0x7fc5cc105f60 2024-08-28T19:20:56.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.789+0000 7fc5d44bb640 1 -- 172.21.15.16:0/909627237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5cc102190 con 0x7fc5cc105f60 2024-08-28T19:20:56.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.792+0000 7fc5c37fe640 1 -- 172.21.15.16:0/909627237 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fc5bc0079b0 con 0x7fc5cc105f60 2024-08-28T19:20:56.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.792+0000 7fc5c37fe640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5b80b4c00 0x7fc5b80b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:56.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.793+0000 7fc5d2257640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5b80b4c00 0x7fc5b80b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:56.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.793+0000 7fc5c37fe640 1 -- 172.21.15.16:0/909627237 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc5bc0e5780 con 0x7fc5cc105f60 2024-08-28T19:20:56.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.793+0000 7fc5d2257640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5b80b4c00 0x7fc5b80b7110 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8000c00 tx=0x7fc5c8031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:56.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.796+0000 7fc5c37fe640 1 -- 172.21.15.16:0/909627237 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fc5bc014030 con 0x7fc5cc105f60 2024-08-28T19:20:56.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.981+0000 7fc5d44bb640 1 -- 172.21.15.16:0/909627237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc5cc02e430 con 0x7fc5cc105f60 2024-08-28T19:20:56.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.981+0000 7fc5c37fe640 1 -- 172.21.15.16:0/909627237 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v84) ==== 103+0+11 (secure 0 0 0) 0x7fc5bc0ac440 con 0x7fc5cc105f60 2024-08-28T19:20:56.983 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 -- 172.21.15.16:0/909627237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5b80b4c00 msgr2=0x7fc5b80b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5b80b4c00 0x7fc5b80b7110 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8000c00 tx=0x7fc5c8031400 comp rx=0 tx=0).stop 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 -- 172.21.15.16:0/909627237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5cc105f60 msgr2=0x7fc5cc19a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5cc105f60 0x7fc5cc19a7d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc00c970 tx=0x7fc5bc00ce30 comp rx=0 tx=0).stop 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 -- 172.21.15.16:0/909627237 shutdown_connections 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5cc1068b0 0x7fc5cc19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5cc105f60 0x7fc5cc19a7d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5b80b4c00 0x7fc5b80b7110 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 --2- 172.21.15.16:0/909627237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5cc101d80 0x7fc5cc19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 -- 172.21.15.16:0/909627237 >> 172.21.15.16:0/909627237 conn(0x7fc5cc079bb0 msgr2=0x7fc5cc107230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 -- 172.21.15.16:0/909627237 shutdown_connections 2024-08-28T19:20:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:56.983+0000 7fc5d44bb640 1 -- 172.21.15.16:0/909627237 wait complete. 2024-08-28T19:20:56.997 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-08-28T19:20:56.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:20:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:20:57.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:57 smithi028 ceph-mon[21769]: pgmap v336: 129 pgs: 129 active+clean; 647 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-08-28T19:20:57.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:57 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3809298501' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-08-28T19:20:57.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:57 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1971204652' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-08-28T19:20:57.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:57 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/909627237' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-08-28T19:20:57.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:57 smithi115 ceph-mon[22184]: pgmap v336: 129 pgs: 129 active+clean; 647 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-08-28T19:20:57.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:57 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3809298501' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-08-28T19:20:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:57 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1971204652' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-08-28T19:20:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:57 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/909627237' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-08-28T19:20:57.634 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:57 smithi016 ceph-mon[18556]: pgmap v336: 129 pgs: 129 active+clean; 647 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-08-28T19:20:57.634 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:57 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3809298501' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-08-28T19:20:57.634 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:57 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1971204652' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-08-28T19:20:57.634 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:57 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/909627237' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-08-28T19:20:57.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.724+0000 7fe43fee7640 1 -- 172.21.15.16:0/1381002363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe438108160 msgr2=0x7fe438108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:57.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.724+0000 7fe43fee7640 1 --2- 172.21.15.16:0/1381002363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe438108160 0x7fe438108570 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe428009900 tx=0x7fe42802a410 comp rx=0 tx=0).stop 2024-08-28T19:20:57.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.724+0000 7fe43fee7640 1 -- 172.21.15.16:0/1381002363 shutdown_connections 2024-08-28T19:20:57.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.724+0000 7fe43fee7640 1 --2- 172.21.15.16:0/1381002363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe438108ab0 0x7fe43810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:57.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.724+0000 7fe43fee7640 1 --2- 172.21.15.16:0/1381002363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe438108160 0x7fe438108570 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:57.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.724+0000 7fe43fee7640 1 --2- 172.21.15.16:0/1381002363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe438102820 0x7fe438102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.724+0000 7fe43fee7640 1 -- 172.21.15.16:0/1381002363 >> 172.21.15.16:0/1381002363 conn(0x7fe4380794e0 msgr2=0x7fe4380798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.724+0000 7fe43fee7640 1 -- 172.21.15.16:0/1381002363 shutdown_connections 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.724+0000 7fe43fee7640 1 -- 172.21.15.16:0/1381002363 wait complete. 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.725+0000 7fe43fee7640 1 Processor -- start 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.725+0000 7fe43fee7640 1 -- start start 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.725+0000 7fe43fee7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe438102820 0x7fe43819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.725+0000 7fe43fee7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe438108160 0x7fe43819a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.725+0000 7fe43fee7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe438108ab0 0x7fe43819fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.725+0000 7fe43fee7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4381126e0 con 0x7fe438108ab0 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.725+0000 7fe43fee7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe438112560 con 0x7fe438102820 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.725+0000 7fe43fee7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe438112860 con 0x7fe438108160 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.725+0000 7fe43e484640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe438108ab0 0x7fe43819fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.725+0000 7fe43e484640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe438108ab0 0x7fe43819fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37036/0 (socket says 172.21.15.16:37036) 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.725+0000 7fe43e484640 1 -- 172.21.15.16:0/1960835766 learned_addr learned my addr 172.21.15.16:0/1960835766 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe43dc83640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe438102820 0x7fe43819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe43d482640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe438108160 0x7fe43819a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe43e484640 1 -- 172.21.15.16:0/1960835766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe438108160 msgr2=0x7fe43819a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe43e484640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe438108160 0x7fe43819a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe43e484640 1 -- 172.21.15.16:0/1960835766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe438102820 msgr2=0x7fe43819a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe43e484640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe438102820 0x7fe43819a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe43e484640 1 -- 172.21.15.16:0/1960835766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4381a0250 con 0x7fe438108ab0 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe43dc83640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe438102820 0x7fe43819a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe43e484640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe438108ab0 0x7fe43819fb20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe43000cad0 tx=0x7fe43000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe42effd640 1 -- 172.21.15.16:0/1960835766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43000ae20 con 0x7fe438108ab0 2024-08-28T19:20:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe42effd640 1 -- 172.21.15.16:0/1960835766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe430005020 con 0x7fe438108ab0 2024-08-28T19:20:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe43fee7640 1 -- 172.21.15.16:0/1960835766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4381a0540 con 0x7fe438108ab0 2024-08-28T19:20:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe42effd640 1 -- 172.21.15.16:0/1960835766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43000f6b0 con 0x7fe438108ab0 2024-08-28T19:20:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.726+0000 7fe43fee7640 1 -- 172.21.15.16:0/1960835766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4381a6ca0 con 0x7fe438108ab0 2024-08-28T19:20:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.727+0000 7fe43fee7640 1 -- 172.21.15.16:0/1960835766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe438103aa0 con 0x7fe438108ab0 2024-08-28T19:20:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.729+0000 7fe42effd640 1 -- 172.21.15.16:0/1960835766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fe4300051d0 con 0x7fe438108ab0 2024-08-28T19:20:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.729+0000 7fe42effd640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe4240b4c00 0x7fe4240b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.730+0000 7fe43dc83640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe4240b4c00 0x7fe4240b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.730+0000 7fe42effd640 1 -- 172.21.15.16:0/1960835766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe4300e5a00 con 0x7fe438108ab0 2024-08-28T19:20:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.730+0000 7fe43dc83640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe4240b4c00 0x7fe4240b7110 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe434009c20 tx=0x7fe434005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.733+0000 7fe42effd640 1 -- 172.21.15.16:0/1960835766 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fe430014030 con 0x7fe438108ab0 2024-08-28T19:20:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.959+0000 7fe43fee7640 1 -- 172.21.15.16:0/1960835766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe438102c30 con 0x7fe438108ab0 2024-08-28T19:20:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.960+0000 7fe42effd640 1 -- 172.21.15.16:0/1960835766 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11977 (secure 0 0 0) 0x7fe4300ac720 con 0x7fe438108ab0 2024-08-28T19:20:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:57.963 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"btime":"2024-08-28T19:20:55:257311+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:20:54.880238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:20:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 -- 172.21.15.16:0/1960835766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe4240b4c00 msgr2=0x7fe4240b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe4240b4c00 0x7fe4240b7110 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe434009c20 tx=0x7fe434005f90 comp rx=0 tx=0).stop 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 -- 172.21.15.16:0/1960835766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe438108ab0 msgr2=0x7fe43819fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe438108ab0 0x7fe43819fb20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe43000cad0 tx=0x7fe43000cf90 comp rx=0 tx=0).stop 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 -- 172.21.15.16:0/1960835766 shutdown_connections 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe438108ab0 0x7fe43819fb20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe438108160 0x7fe43819a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe4240b4c00 0x7fe4240b7110 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 --2- 172.21.15.16:0/1960835766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe438102820 0x7fe43819a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 -- 172.21.15.16:0/1960835766 >> 172.21.15.16:0/1960835766 conn(0x7fe4380794e0 msgr2=0x7fe43806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 -- 172.21.15.16:0/1960835766 shutdown_connections 2024-08-28T19:20:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:57.962+0000 7fe43fee7640 1 -- 172.21.15.16:0/1960835766 wait complete. 2024-08-28T19:20:58.556 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:58 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1960835766' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:58.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:58 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1960835766' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:58.586 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-08-28T19:20:58.586 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-08-28T19:20:58.586 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-28T19:20:58.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:58 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1960835766' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:20:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.699+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1706862108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2c0103880 msgr2=0x7fd2c0103c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.699+0000 7fd2c7f56640 1 --2- 172.21.15.16:0/1706862108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2c0103880 0x7fd2c0103c90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009ca0 tx=0x7fd2bc02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:20:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.699+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1706862108 shutdown_connections 2024-08-28T19:20:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.699+0000 7fd2c7f56640 1 --2- 172.21.15.16:0/1706862108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2c0109b00 0x7fd2c0110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.699+0000 7fd2c7f56640 1 --2- 172.21.15.16:0/1706862108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2c01091b0 0x7fd2c01095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.699+0000 7fd2c7f56640 1 --2- 172.21.15.16:0/1706862108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2c0103880 0x7fd2c0103c90 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.699+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1706862108 >> 172.21.15.16:0/1706862108 conn(0x7fd2c00ff2a0 msgr2=0x7fd2c01016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.699+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1706862108 shutdown_connections 2024-08-28T19:20:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.699+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1706862108 wait complete. 2024-08-28T19:20:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c7f56640 1 Processor -- start 2024-08-28T19:20:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c7f56640 1 -- start start 2024-08-28T19:20:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c7f56640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2c0103880 0x7fd2c019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c7f56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2c01091b0 0x7fd2c019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c7f56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2c0109b00 0x7fd2c019fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c7f56640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c01137d0 con 0x7fd2c0109b00 2024-08-28T19:20:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c7f56640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c0113650 con 0x7fd2c01091b0 2024-08-28T19:20:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c7f56640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c0113950 con 0x7fd2c0103880 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c64f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2c0109b00 0x7fd2c019fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c64f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2c0109b00 0x7fd2c019fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37052/0 (socket says 172.21.15.16:37052) 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c64f3640 1 -- 172.21.15.16:0/1698697446 learned_addr learned my addr 172.21.15.16:0/1698697446 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.700+0000 7fd2c54f1640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2c01091b0 0x7fd2c019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2c64f3640 1 -- 172.21.15.16:0/1698697446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2c0103880 msgr2=0x7fd2c019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2c5cf2640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2c0103880 0x7fd2c019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2c64f3640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2c0103880 0x7fd2c019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2c64f3640 1 -- 172.21.15.16:0/1698697446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2c01091b0 msgr2=0x7fd2c019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2c64f3640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2c01091b0 0x7fd2c019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2c64f3640 1 -- 172.21.15.16:0/1698697446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c01a0100 con 0x7fd2c0109b00 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2c64f3640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2c0109b00 0x7fd2c019fa00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800d750 tx=0x7fd2b800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2b6ffd640 1 -- 172.21.15.16:0/1698697446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8017070 con 0x7fd2c0109b00 2024-08-28T19:20:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2b6ffd640 1 -- 172.21.15.16:0/1698697446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2b80054d0 con 0x7fd2c0109b00 2024-08-28T19:20:58.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1698697446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c01a03f0 con 0x7fd2c0109b00 2024-08-28T19:20:58.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2b6ffd640 1 -- 172.21.15.16:0/1698697446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8004c60 con 0x7fd2c0109b00 2024-08-28T19:20:58.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.701+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1698697446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c01a6b80 con 0x7fd2c0109b00 2024-08-28T19:20:58.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.702+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1698697446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c0103c90 con 0x7fd2c0109b00 2024-08-28T19:20:58.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.704+0000 7fd2b6ffd640 1 -- 172.21.15.16:0/1698697446 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fd2b8009bd0 con 0x7fd2c0109b00 2024-08-28T19:20:58.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.704+0000 7fd2b6ffd640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd2ac0b4c00 0x7fd2ac0b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:58.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.705+0000 7fd2c5cf2640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd2ac0b4c00 0x7fd2ac0b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:58.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.705+0000 7fd2b6ffd640 1 -- 172.21.15.16:0/1698697446 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fd2b80e5350 con 0x7fd2c0109b00 2024-08-28T19:20:58.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.705+0000 7fd2c5cf2640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd2ac0b4c00 0x7fd2ac0b7110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009ca0 tx=0x7fd2bc005fd0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:58.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.708+0000 7fd2b6ffd640 1 -- 172.21.15.16:0/1698697446 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fd2b80ac070 con 0x7fd2c0109b00 2024-08-28T19:20:58.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.892+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1698697446 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2c00635a0 con 0x7fd2ac0b4c00 2024-08-28T19:20:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.898+0000 7fd2b6ffd640 1 -- 172.21.15.16:0/1698697446 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340519 (secure 0 0 0) 0x7fd2c00635a0 con 0x7fd2ac0b4c00 2024-08-28T19:20:58.904 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-08-28T19:20:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.901+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1698697446 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd2ac0b4c00 msgr2=0x7fd2ac0b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.901+0000 7fd2c7f56640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd2ac0b4c00 0x7fd2ac0b7110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009ca0 tx=0x7fd2bc005fd0 comp rx=0 tx=0).stop 2024-08-28T19:20:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.901+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1698697446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2c0109b00 msgr2=0x7fd2c019fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.901+0000 7fd2c7f56640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2c0109b00 0x7fd2c019fa00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800d750 tx=0x7fd2b800dc10 comp rx=0 tx=0).stop 2024-08-28T19:20:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.901+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1698697446 shutdown_connections 2024-08-28T19:20:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.901+0000 7fd2c7f56640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2c0109b00 0x7fd2c019fa00 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.901+0000 7fd2c7f56640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd2ac0b4c00 0x7fd2ac0b7110 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.901+0000 7fd2c7f56640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2c01091b0 0x7fd2c019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.901+0000 7fd2c7f56640 1 --2- 172.21.15.16:0/1698697446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2c0103880 0x7fd2c019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.902+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1698697446 >> 172.21.15.16:0/1698697446 conn(0x7fd2c00ff2a0 msgr2=0x7fd2c01016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.902+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1698697446 shutdown_connections 2024-08-28T19:20:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:58.902+0000 7fd2c7f56640 1 -- 172.21.15.16:0/1698697446 wait complete. 2024-08-28T19:20:58.916 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-08-28T19:20:57.495480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662120,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1155,"num_read_kb":687,"num_write":387,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":819,"ondisk_log_size":819,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":416992,"kb_used_data":13664,"kb_used_omap":18,"kb_used_meta":403117,"kb_avail":1124524832,"statfs":{"total":1151940427776,"available":1151513427968,"internally_reserved":0,"allocated":13991936,"data_stored":7952798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":412792122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":308,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":56,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003941"},"pg_stats":[{"pgid":"2.1f","version":"84'14","reported_seq":172,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.806536+0000","last_change":"2024-08-28T19:19:08.633326+0000","last_active":"2024-08-28T19:20:54.806536+0000","last_peered":"2024-08-28T19:20:54.806536+0000","last_clean":"2024-08-28T19:20:54.806536+0000","last_became_active":"2024-08-28T19:19:08.623876+0000","last_became_peered":"2024-08-28T19:19:08.623876+0000","last_unstale":"2024-08-28T19:20:54.806536+0000","last_undegraded":"2024-08-28T19:20:54.806536+0000","last_fullsized":"2024-08-28T19:20:54.806536+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:19:50.632337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866980+0000","last_change":"2024-08-28T19:18:32.492604+0000","last_active":"2024-08-28T19:20:37.866980+0000","last_peered":"2024-08-28T19:20:37.866980+0000","last_clean":"2024-08-28T19:20:37.866980+0000","last_became_active":"2024-08-28T19:18:32.489427+0000","last_became_peered":"2024-08-28T19:18:32.489427+0000","last_unstale":"2024-08-28T19:20:37.866980+0000","last_undegraded":"2024-08-28T19:20:37.866980+0000","last_fullsized":"2024-08-28T19:20:37.866980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:36:14.841315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010445+0000","last_change":"2024-08-28T19:18:39.528815+0000","last_active":"2024-08-28T19:20:09.010445+0000","last_peered":"2024-08-28T19:20:09.010445+0000","last_clean":"2024-08-28T19:20:09.010445+0000","last_became_active":"2024-08-28T19:18:39.527781+0000","last_became_peered":"2024-08-28T19:18:39.527781+0000","last_unstale":"2024-08-28T19:20:09.010445+0000","last_undegraded":"2024-08-28T19:20:09.010445+0000","last_fullsized":"2024-08-28T19:20:09.010445+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:39.831426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525559+0000","last_change":"2024-08-28T19:18:31.082641+0000","last_active":"2024-08-28T19:20:09.525559+0000","last_peered":"2024-08-28T19:20:09.525559+0000","last_clean":"2024-08-28T19:20:09.525559+0000","last_became_active":"2024-08-28T19:18:31.082260+0000","last_became_peered":"2024-08-28T19:18:31.082260+0000","last_unstale":"2024-08-28T19:20:09.525559+0000","last_undegraded":"2024-08-28T19:20:09.525559+0000","last_fullsized":"2024-08-28T19:20:09.525559+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:09:55.836463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886294+0000","last_change":"2024-08-28T19:18:32.493498+0000","last_active":"2024-08-28T19:20:37.886294+0000","last_peered":"2024-08-28T19:20:37.886294+0000","last_clean":"2024-08-28T19:20:37.886294+0000","last_became_active":"2024-08-28T19:18:32.490969+0000","last_became_peered":"2024-08-28T19:18:32.490969+0000","last_unstale":"2024-08-28T19:20:37.886294+0000","last_undegraded":"2024-08-28T19:20:37.886294+0000","last_fullsized":"2024-08-28T19:20:37.886294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:34:29.537622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614531+0000","last_change":"2024-08-28T19:18:39.520922+0000","last_active":"2024-08-28T19:20:09.614531+0000","last_peered":"2024-08-28T19:20:09.614531+0000","last_clean":"2024-08-28T19:20:09.614531+0000","last_became_active":"2024-08-28T19:18:39.520362+0000","last_became_peered":"2024-08-28T19:18:39.520362+0000","last_unstale":"2024-08-28T19:20:09.614531+0000","last_undegraded":"2024-08-28T19:20:09.614531+0000","last_fullsized":"2024-08-28T19:20:09.614531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:40.658820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":52,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110351+0000","last_change":"2024-08-28T19:18:30.471870+0000","last_active":"2024-08-28T19:20:09.110351+0000","last_peered":"2024-08-28T19:20:09.110351+0000","last_clean":"2024-08-28T19:20:09.110351+0000","last_became_active":"2024-08-28T19:18:30.471438+0000","last_became_peered":"2024-08-28T19:18:30.471438+0000","last_unstale":"2024-08-28T19:20:09.110351+0000","last_undegraded":"2024-08-28T19:20:09.110351+0000","last_fullsized":"2024-08-28T19:20:09.110351+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:44:34.137395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"84'9","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.881074+0000","last_change":"2024-08-28T19:18:32.486485+0000","last_active":"2024-08-28T19:20:37.881074+0000","last_peered":"2024-08-28T19:20:37.881074+0000","last_clean":"2024-08-28T19:20:37.881074+0000","last_became_active":"2024-08-28T19:18:32.485888+0000","last_became_peered":"2024-08-28T19:18:32.485888+0000","last_unstale":"2024-08-28T19:20:37.881074+0000","last_undegraded":"2024-08-28T19:20:37.881074+0000","last_fullsized":"2024-08-28T19:20:37.881074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:45:48.757318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615105+0000","last_change":"2024-08-28T19:19:08.633317+0000","last_active":"2024-08-28T19:20:09.615105+0000","last_peered":"2024-08-28T19:20:09.615105+0000","last_clean":"2024-08-28T19:20:09.615105+0000","last_became_active":"2024-08-28T19:19:08.632689+0000","last_became_peered":"2024-08-28T19:19:08.632689+0000","last_unstale":"2024-08-28T19:20:09.615105+0000","last_undegraded":"2024-08-28T19:20:09.615105+0000","last_fullsized":"2024-08-28T19:20:09.615105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:35:08.740645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110282+0000","last_change":"2024-08-28T19:18:30.473982+0000","last_active":"2024-08-28T19:20:09.110282+0000","last_peered":"2024-08-28T19:20:09.110282+0000","last_clean":"2024-08-28T19:20:09.110282+0000","last_became_active":"2024-08-28T19:18:30.473552+0000","last_became_peered":"2024-08-28T19:18:30.473552+0000","last_unstale":"2024-08-28T19:20:09.110282+0000","last_undegraded":"2024-08-28T19:20:09.110282+0000","last_fullsized":"2024-08-28T19:20:09.110282+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:22:29.435517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"84'15","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884248+0000","last_change":"2024-08-28T19:18:32.487881+0000","last_active":"2024-08-28T19:20:37.884248+0000","last_peered":"2024-08-28T19:20:37.884248+0000","last_clean":"2024-08-28T19:20:37.884248+0000","last_became_active":"2024-08-28T19:18:32.487039+0000","last_became_peered":"2024-08-28T19:18:32.487039+0000","last_unstale":"2024-08-28T19:20:37.884248+0000","last_undegraded":"2024-08-28T19:20:37.884248+0000","last_fullsized":"2024-08-28T19:20:37.884248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:23:35.757811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614594+0000","last_change":"2024-08-28T19:18:39.518654+0000","last_active":"2024-08-28T19:20:09.614594+0000","last_peered":"2024-08-28T19:20:09.614594+0000","last_clean":"2024-08-28T19:20:09.614594+0000","last_became_active":"2024-08-28T19:18:39.518087+0000","last_became_peered":"2024-08-28T19:18:39.518087+0000","last_unstale":"2024-08-28T19:20:09.614594+0000","last_undegraded":"2024-08-28T19:20:09.614594+0000","last_fullsized":"2024-08-28T19:20:09.614594+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:08.284865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.887381+0000","last_change":"2024-08-28T19:18:32.491928+0000","last_active":"2024-08-28T19:20:37.887381+0000","last_peered":"2024-08-28T19:20:37.887381+0000","last_clean":"2024-08-28T19:20:37.887381+0000","last_became_active":"2024-08-28T19:18:32.491538+0000","last_became_peered":"2024-08-28T19:18:32.491538+0000","last_unstale":"2024-08-28T19:20:37.887381+0000","last_undegraded":"2024-08-28T19:20:37.887381+0000","last_fullsized":"2024-08-28T19:20:37.887381+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:39:06.077830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":24,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663446+0000","last_change":"2024-08-28T19:19:08.634752+0000","last_active":"2024-08-28T19:20:09.663446+0000","last_peered":"2024-08-28T19:20:09.663446+0000","last_clean":"2024-08-28T19:20:09.663446+0000","last_became_active":"2024-08-28T19:19:08.620784+0000","last_became_peered":"2024-08-28T19:19:08.620784+0000","last_unstale":"2024-08-28T19:20:09.663446+0000","last_undegraded":"2024-08-28T19:20:09.663446+0000","last_fullsized":"2024-08-28T19:20:09.663446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:51.640445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614895+0000","last_change":"2024-08-28T19:18:39.681413+0000","last_active":"2024-08-28T19:20:09.614895+0000","last_peered":"2024-08-28T19:20:09.614895+0000","last_clean":"2024-08-28T19:20:09.614895+0000","last_became_active":"2024-08-28T19:18:39.680793+0000","last_became_peered":"2024-08-28T19:18:39.680793+0000","last_unstale":"2024-08-28T19:20:09.614895+0000","last_undegraded":"2024-08-28T19:20:09.614895+0000","last_fullsized":"2024-08-28T19:20:09.614895+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:45:43.381019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870888+0000","last_change":"2024-08-28T19:18:32.486739+0000","last_active":"2024-08-28T19:20:37.870888+0000","last_peered":"2024-08-28T19:20:37.870888+0000","last_clean":"2024-08-28T19:20:37.870888+0000","last_became_active":"2024-08-28T19:18:32.486347+0000","last_became_peered":"2024-08-28T19:18:32.486347+0000","last_unstale":"2024-08-28T19:20:37.870888+0000","last_undegraded":"2024-08-28T19:20:37.870888+0000","last_fullsized":"2024-08-28T19:20:37.870888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:27:36.448569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010653+0000","last_change":"2024-08-28T19:18:30.467690+0000","last_active":"2024-08-28T19:20:09.010653+0000","last_peered":"2024-08-28T19:20:09.010653+0000","last_clean":"2024-08-28T19:20:09.010653+0000","last_became_active":"2024-08-28T19:18:30.467249+0000","last_became_peered":"2024-08-28T19:18:30.467249+0000","last_unstale":"2024-08-28T19:20:09.010653+0000","last_undegraded":"2024-08-28T19:20:09.010653+0000","last_fullsized":"2024-08-28T19:20:09.010653+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:37:28.876666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615167+0000","last_change":"2024-08-28T19:18:39.523889+0000","last_active":"2024-08-28T19:20:09.615167+0000","last_peered":"2024-08-28T19:20:09.615167+0000","last_clean":"2024-08-28T19:20:09.615167+0000","last_became_active":"2024-08-28T19:18:39.523384+0000","last_became_peered":"2024-08-28T19:18:39.523384+0000","last_unstale":"2024-08-28T19:20:09.615167+0000","last_undegraded":"2024-08-28T19:20:09.615167+0000","last_fullsized":"2024-08-28T19:20:09.615167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:57:32.277680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873180+0000","last_change":"2024-08-28T19:18:32.482257+0000","last_active":"2024-08-28T19:20:37.873180+0000","last_peered":"2024-08-28T19:20:37.873180+0000","last_clean":"2024-08-28T19:20:37.873180+0000","last_became_active":"2024-08-28T19:18:32.481884+0000","last_became_peered":"2024-08-28T19:18:32.481884+0000","last_unstale":"2024-08-28T19:20:37.873180+0000","last_undegraded":"2024-08-28T19:20:37.873180+0000","last_fullsized":"2024-08-28T19:20:37.873180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:56:28.754369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":47,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525435+0000","last_change":"2024-08-28T19:18:30.467307+0000","last_active":"2024-08-28T19:20:09.525435+0000","last_peered":"2024-08-28T19:20:09.525435+0000","last_clean":"2024-08-28T19:20:09.525435+0000","last_became_active":"2024-08-28T19:18:30.466942+0000","last_became_peered":"2024-08-28T19:18:30.466942+0000","last_unstale":"2024-08-28T19:20:09.525435+0000","last_undegraded":"2024-08-28T19:20:09.525435+0000","last_fullsized":"2024-08-28T19:20:09.525435+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:05:23.682110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007238+0000","last_change":"2024-08-28T19:18:39.531421+0000","last_active":"2024-08-28T19:20:09.007238+0000","last_peered":"2024-08-28T19:20:09.007238+0000","last_clean":"2024-08-28T19:20:09.007238+0000","last_became_active":"2024-08-28T19:18:39.531008+0000","last_became_peered":"2024-08-28T19:18:39.531008+0000","last_unstale":"2024-08-28T19:20:09.007238+0000","last_undegraded":"2024-08-28T19:20:09.007238+0000","last_fullsized":"2024-08-28T19:20:09.007238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:46:49.580601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868264+0000","last_change":"2024-08-28T19:18:32.485204+0000","last_active":"2024-08-28T19:20:37.868264+0000","last_peered":"2024-08-28T19:20:37.868264+0000","last_clean":"2024-08-28T19:20:37.868264+0000","last_became_active":"2024-08-28T19:18:32.484830+0000","last_became_peered":"2024-08-28T19:18:32.484830+0000","last_unstale":"2024-08-28T19:20:37.868264+0000","last_undegraded":"2024-08-28T19:20:37.868264+0000","last_fullsized":"2024-08-28T19:20:37.868264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:55:41.358718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007905+0000","last_change":"2024-08-28T19:18:31.082361+0000","last_active":"2024-08-28T19:20:09.007905+0000","last_peered":"2024-08-28T19:20:09.007905+0000","last_clean":"2024-08-28T19:20:09.007905+0000","last_became_active":"2024-08-28T19:18:31.081930+0000","last_became_peered":"2024-08-28T19:18:31.081930+0000","last_unstale":"2024-08-28T19:20:09.007905+0000","last_undegraded":"2024-08-28T19:20:09.007905+0000","last_fullsized":"2024-08-28T19:20:09.007905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:41:05.543088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615411+0000","last_change":"2024-08-28T19:18:39.682341+0000","last_active":"2024-08-28T19:20:09.615411+0000","last_peered":"2024-08-28T19:20:09.615411+0000","last_clean":"2024-08-28T19:20:09.615411+0000","last_became_active":"2024-08-28T19:18:39.681751+0000","last_became_peered":"2024-08-28T19:18:39.681751+0000","last_unstale":"2024-08-28T19:20:09.615411+0000","last_undegraded":"2024-08-28T19:20:09.615411+0000","last_fullsized":"2024-08-28T19:20:09.615411+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:51:24.371270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880906+0000","last_change":"2024-08-28T19:18:32.491069+0000","last_active":"2024-08-28T19:20:37.880906+0000","last_peered":"2024-08-28T19:20:37.880906+0000","last_clean":"2024-08-28T19:20:37.880906+0000","last_became_active":"2024-08-28T19:18:32.490688+0000","last_became_peered":"2024-08-28T19:18:32.490688+0000","last_unstale":"2024-08-28T19:20:37.880906+0000","last_undegraded":"2024-08-28T19:20:37.880906+0000","last_fullsized":"2024-08-28T19:20:37.880906+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:24:31.560519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615273+0000","last_change":"2024-08-28T19:18:30.739812+0000","last_active":"2024-08-28T19:20:09.615273+0000","last_peered":"2024-08-28T19:20:09.615273+0000","last_clean":"2024-08-28T19:20:09.615273+0000","last_became_active":"2024-08-28T19:18:30.739414+0000","last_became_peered":"2024-08-28T19:18:30.739414+0000","last_unstale":"2024-08-28T19:20:09.615273+0000","last_undegraded":"2024-08-28T19:20:09.615273+0000","last_fullsized":"2024-08-28T19:20:09.615273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:56:56.594993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010962+0000","last_change":"2024-08-28T19:18:39.528281+0000","last_active":"2024-08-28T19:20:09.010962+0000","last_peered":"2024-08-28T19:20:09.010962+0000","last_clean":"2024-08-28T19:20:09.010962+0000","last_became_active":"2024-08-28T19:18:39.527142+0000","last_became_peered":"2024-08-28T19:18:39.527142+0000","last_unstale":"2024-08-28T19:20:09.010962+0000","last_undegraded":"2024-08-28T19:20:09.010962+0000","last_fullsized":"2024-08-28T19:20:09.010962+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:44:19.123662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872571+0000","last_change":"2024-08-28T19:18:32.490430+0000","last_active":"2024-08-28T19:20:37.872571+0000","last_peered":"2024-08-28T19:20:37.872571+0000","last_clean":"2024-08-28T19:20:37.872571+0000","last_became_active":"2024-08-28T19:18:32.490020+0000","last_became_peered":"2024-08-28T19:18:32.490020+0000","last_unstale":"2024-08-28T19:20:37.872571+0000","last_undegraded":"2024-08-28T19:20:37.872571+0000","last_fullsized":"2024-08-28T19:20:37.872571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:52:31.153012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'17","reported_seq":57,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:41.222712+0000","last_change":"2024-08-28T19:18:30.467060+0000","last_active":"2024-08-28T19:20:41.222712+0000","last_peered":"2024-08-28T19:20:41.222712+0000","last_clean":"2024-08-28T19:20:41.222712+0000","last_became_active":"2024-08-28T19:18:30.466689+0000","last_became_peered":"2024-08-28T19:18:30.466689+0000","last_unstale":"2024-08-28T19:20:41.222712+0000","last_undegraded":"2024-08-28T19:20:41.222712+0000","last_fullsized":"2024-08-28T19:20:41.222712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:18:36.508131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.524970+0000","last_change":"2024-08-28T19:18:39.682874+0000","last_active":"2024-08-28T19:20:09.524970+0000","last_peered":"2024-08-28T19:20:09.524970+0000","last_clean":"2024-08-28T19:20:09.524970+0000","last_became_active":"2024-08-28T19:18:39.681500+0000","last_became_peered":"2024-08-28T19:18:39.681500+0000","last_unstale":"2024-08-28T19:20:09.524970+0000","last_undegraded":"2024-08-28T19:20:09.524970+0000","last_fullsized":"2024-08-28T19:20:09.524970+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:45:41.815506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.856367+0000","last_change":"2024-08-28T19:18:32.482047+0000","last_active":"2024-08-28T19:20:37.856367+0000","last_peered":"2024-08-28T19:20:37.856367+0000","last_clean":"2024-08-28T19:20:37.856367+0000","last_became_active":"2024-08-28T19:18:32.481640+0000","last_became_peered":"2024-08-28T19:18:32.481640+0000","last_unstale":"2024-08-28T19:20:37.856367+0000","last_undegraded":"2024-08-28T19:20:37.856367+0000","last_fullsized":"2024-08-28T19:20:37.856367+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:34.717764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110454+0000","last_change":"2024-08-28T19:18:31.083524+0000","last_active":"2024-08-28T19:20:09.110454+0000","last_peered":"2024-08-28T19:20:09.110454+0000","last_clean":"2024-08-28T19:20:09.110454+0000","last_became_active":"2024-08-28T19:18:31.083106+0000","last_became_peered":"2024-08-28T19:18:31.083106+0000","last_unstale":"2024-08-28T19:20:09.110454+0000","last_undegraded":"2024-08-28T19:20:09.110454+0000","last_fullsized":"2024-08-28T19:20:09.110454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:16.180754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663607+0000","last_change":"2024-08-28T19:18:39.531884+0000","last_active":"2024-08-28T19:20:09.663607+0000","last_peered":"2024-08-28T19:20:09.663607+0000","last_clean":"2024-08-28T19:20:09.663607+0000","last_became_active":"2024-08-28T19:18:39.531255+0000","last_became_peered":"2024-08-28T19:18:39.531255+0000","last_unstale":"2024-08-28T19:20:09.663607+0000","last_undegraded":"2024-08-28T19:20:09.663607+0000","last_fullsized":"2024-08-28T19:20:09.663607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:36:55.496594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.862241+0000","last_change":"2024-08-28T19:18:32.485180+0000","last_active":"2024-08-28T19:20:37.862241+0000","last_peered":"2024-08-28T19:20:37.862241+0000","last_clean":"2024-08-28T19:20:37.862241+0000","last_became_active":"2024-08-28T19:18:32.484804+0000","last_became_peered":"2024-08-28T19:18:32.484804+0000","last_unstale":"2024-08-28T19:20:37.862241+0000","last_undegraded":"2024-08-28T19:20:37.862241+0000","last_fullsized":"2024-08-28T19:20:37.862241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:12:40.958251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007361+0000","last_change":"2024-08-28T19:18:31.082226+0000","last_active":"2024-08-28T19:20:09.007361+0000","last_peered":"2024-08-28T19:20:09.007361+0000","last_clean":"2024-08-28T19:20:09.007361+0000","last_became_active":"2024-08-28T19:18:31.081833+0000","last_became_peered":"2024-08-28T19:18:31.081833+0000","last_unstale":"2024-08-28T19:20:09.007361+0000","last_undegraded":"2024-08-28T19:20:09.007361+0000","last_fullsized":"2024-08-28T19:20:09.007361+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:33:56.298093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109762+0000","last_change":"2024-08-28T19:18:39.681288+0000","last_active":"2024-08-28T19:20:09.109762+0000","last_peered":"2024-08-28T19:20:09.109762+0000","last_clean":"2024-08-28T19:20:09.109762+0000","last_became_active":"2024-08-28T19:18:39.680719+0000","last_became_peered":"2024-08-28T19:18:39.680719+0000","last_unstale":"2024-08-28T19:20:09.109762+0000","last_undegraded":"2024-08-28T19:20:09.109762+0000","last_fullsized":"2024-08-28T19:20:09.109762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:53:35.554181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880497+0000","last_change":"2024-08-28T19:18:32.487105+0000","last_active":"2024-08-28T19:20:37.880497+0000","last_peered":"2024-08-28T19:20:37.880497+0000","last_clean":"2024-08-28T19:20:37.880497+0000","last_became_active":"2024-08-28T19:18:32.486643+0000","last_became_peered":"2024-08-28T19:18:32.486643+0000","last_unstale":"2024-08-28T19:20:37.880497+0000","last_undegraded":"2024-08-28T19:20:37.880497+0000","last_fullsized":"2024-08-28T19:20:37.880497+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:18:43.274493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"84'7","reported_seq":153,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:53.805823+0000","last_change":"2024-08-28T19:18:30.471447+0000","last_active":"2024-08-28T19:20:53.805823+0000","last_peered":"2024-08-28T19:20:53.805823+0000","last_clean":"2024-08-28T19:20:53.805823+0000","last_became_active":"2024-08-28T19:18:30.470798+0000","last_became_peered":"2024-08-28T19:18:30.470798+0000","last_unstale":"2024-08-28T19:20:53.805823+0000","last_undegraded":"2024-08-28T19:20:53.805823+0000","last_fullsized":"2024-08-28T19:20:53.805823+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:50:13.948439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162958+0000","last_change":"2024-08-28T19:18:39.524190+0000","last_active":"2024-08-28T19:20:09.162958+0000","last_peered":"2024-08-28T19:20:09.162958+0000","last_clean":"2024-08-28T19:20:09.162958+0000","last_became_active":"2024-08-28T19:18:39.523771+0000","last_became_peered":"2024-08-28T19:18:39.523771+0000","last_unstale":"2024-08-28T19:20:09.162958+0000","last_undegraded":"2024-08-28T19:20:09.162958+0000","last_fullsized":"2024-08-28T19:20:09.162958+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:43:43.984625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.874995+0000","last_change":"2024-08-28T19:18:32.487212+0000","last_active":"2024-08-28T19:20:37.874995+0000","last_peered":"2024-08-28T19:20:37.874995+0000","last_clean":"2024-08-28T19:20:37.874995+0000","last_became_active":"2024-08-28T19:18:32.485432+0000","last_became_peered":"2024-08-28T19:18:32.485432+0000","last_unstale":"2024-08-28T19:20:37.874995+0000","last_undegraded":"2024-08-28T19:20:37.874995+0000","last_fullsized":"2024-08-28T19:20:37.874995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:52:11.672193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":149,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:53.805226+0000","last_change":"2024-08-28T19:18:31.082210+0000","last_active":"2024-08-28T19:20:53.805226+0000","last_peered":"2024-08-28T19:20:53.805226+0000","last_clean":"2024-08-28T19:20:53.805226+0000","last_became_active":"2024-08-28T19:18:31.081814+0000","last_became_peered":"2024-08-28T19:18:31.081814+0000","last_unstale":"2024-08-28T19:20:53.805226+0000","last_undegraded":"2024-08-28T19:20:53.805226+0000","last_fullsized":"2024-08-28T19:20:53.805226+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:33:32.281431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163227+0000","last_change":"2024-08-28T19:18:39.681858+0000","last_active":"2024-08-28T19:20:09.163227+0000","last_peered":"2024-08-28T19:20:09.163227+0000","last_clean":"2024-08-28T19:20:09.163227+0000","last_became_active":"2024-08-28T19:18:39.681382+0000","last_became_peered":"2024-08-28T19:18:39.681382+0000","last_unstale":"2024-08-28T19:20:09.163227+0000","last_undegraded":"2024-08-28T19:20:09.163227+0000","last_fullsized":"2024-08-28T19:20:09.163227+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:23:22.021022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.857639+0000","last_change":"2024-08-28T19:18:32.486802+0000","last_active":"2024-08-28T19:20:37.857639+0000","last_peered":"2024-08-28T19:20:37.857639+0000","last_clean":"2024-08-28T19:20:37.857639+0000","last_became_active":"2024-08-28T19:18:32.486479+0000","last_became_peered":"2024-08-28T19:18:32.486479+0000","last_unstale":"2024-08-28T19:20:37.857639+0000","last_undegraded":"2024-08-28T19:20:37.857639+0000","last_fullsized":"2024-08-28T19:20:37.857639+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:17:17.654757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529084+0000","last_change":"2024-08-28T19:18:30.470447+0000","last_active":"2024-08-28T19:20:09.529084+0000","last_peered":"2024-08-28T19:20:09.529084+0000","last_clean":"2024-08-28T19:20:09.529084+0000","last_became_active":"2024-08-28T19:18:30.469862+0000","last_became_peered":"2024-08-28T19:18:30.469862+0000","last_unstale":"2024-08-28T19:20:09.529084+0000","last_undegraded":"2024-08-28T19:20:09.529084+0000","last_fullsized":"2024-08-28T19:20:09.529084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:27:46.660492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009270+0000","last_change":"2024-08-28T19:19:08.624635+0000","last_active":"2024-08-28T19:20:09.009270+0000","last_peered":"2024-08-28T19:20:09.009270+0000","last_clean":"2024-08-28T19:20:09.009270+0000","last_became_active":"2024-08-28T19:19:08.622845+0000","last_became_peered":"2024-08-28T19:19:08.622845+0000","last_unstale":"2024-08-28T19:20:09.009270+0000","last_undegraded":"2024-08-28T19:20:09.009270+0000","last_fullsized":"2024-08-28T19:20:09.009270+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:56:40.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865852+0000","last_change":"2024-08-28T19:18:32.487947+0000","last_active":"2024-08-28T19:20:37.865852+0000","last_peered":"2024-08-28T19:20:37.865852+0000","last_clean":"2024-08-28T19:20:37.865852+0000","last_became_active":"2024-08-28T19:18:32.487592+0000","last_became_peered":"2024-08-28T19:18:32.487592+0000","last_unstale":"2024-08-28T19:20:37.865852+0000","last_undegraded":"2024-08-28T19:20:37.865852+0000","last_fullsized":"2024-08-28T19:20:37.865852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:30:29.550644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615576+0000","last_change":"2024-08-28T19:18:30.465044+0000","last_active":"2024-08-28T19:20:09.615576+0000","last_peered":"2024-08-28T19:20:09.615576+0000","last_clean":"2024-08-28T19:20:09.615576+0000","last_became_active":"2024-08-28T19:18:30.464580+0000","last_became_peered":"2024-08-28T19:18:30.464580+0000","last_unstale":"2024-08-28T19:20:09.615576+0000","last_undegraded":"2024-08-28T19:20:09.615576+0000","last_fullsized":"2024-08-28T19:20:09.615576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:21:17.223311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":114,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.528964+0000","last_change":"2024-08-28T19:17:17.063316+0000","last_active":"2024-08-28T19:20:09.528964+0000","last_peered":"2024-08-28T19:20:09.528964+0000","last_clean":"2024-08-28T19:20:09.528964+0000","last_became_active":"2024-08-28T19:17:17.057091+0000","last_became_peered":"2024-08-28T19:17:17.057091+0000","last_unstale":"2024-08-28T19:20:09.528964+0000","last_undegraded":"2024-08-28T19:20:09.528964+0000","last_fullsized":"2024-08-28T19:20:09.528964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_clean_scrub_stamp":"2024-08-28T19:13:56.815467+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:35:51.499703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"82'1","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529202+0000","last_change":"2024-08-28T19:19:08.621402+0000","last_active":"2024-08-28T19:20:09.529202+0000","last_peered":"2024-08-28T19:20:09.529202+0000","last_clean":"2024-08-28T19:20:09.529202+0000","last_became_active":"2024-08-28T19:19:08.620779+0000","last_became_peered":"2024-08-28T19:19:08.620779+0000","last_unstale":"2024-08-28T19:20:09.529202+0000","last_undegraded":"2024-08-28T19:20:09.529202+0000","last_fullsized":"2024-08-28T19:20:09.529202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:39.391414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886997+0000","last_change":"2024-08-28T19:18:32.490066+0000","last_active":"2024-08-28T19:20:37.886997+0000","last_peered":"2024-08-28T19:20:37.886997+0000","last_clean":"2024-08-28T19:20:37.886997+0000","last_became_active":"2024-08-28T19:18:32.489560+0000","last_became_peered":"2024-08-28T19:18:32.489560+0000","last_unstale":"2024-08-28T19:20:37.886997+0000","last_undegraded":"2024-08-28T19:20:37.886997+0000","last_fullsized":"2024-08-28T19:20:37.886997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:22:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'5","reported_seq":151,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.851167+0000","last_change":"2024-08-28T19:18:31.083078+0000","last_active":"2024-08-28T19:20:54.851167+0000","last_peered":"2024-08-28T19:20:54.851167+0000","last_clean":"2024-08-28T19:20:54.851167+0000","last_became_active":"2024-08-28T19:18:31.082575+0000","last_became_peered":"2024-08-28T19:18:31.082575+0000","last_unstale":"2024-08-28T19:20:54.851167+0000","last_undegraded":"2024-08-28T19:20:54.851167+0000","last_fullsized":"2024-08-28T19:20:54.851167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007862+0000","last_change":"2024-08-28T19:18:39.519814+0000","last_active":"2024-08-28T19:20:09.007862+0000","last_peered":"2024-08-28T19:20:09.007862+0000","last_clean":"2024-08-28T19:20:09.007862+0000","last_became_active":"2024-08-28T19:18:39.519258+0000","last_became_peered":"2024-08-28T19:18:39.519258+0000","last_unstale":"2024-08-28T19:20:09.007862+0000","last_undegraded":"2024-08-28T19:20:09.007862+0000","last_fullsized":"2024-08-28T19:20:09.007862+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:03:04.051376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871177+0000","last_change":"2024-08-28T19:18:32.495426+0000","last_active":"2024-08-28T19:20:37.871177+0000","last_peered":"2024-08-28T19:20:37.871177+0000","last_clean":"2024-08-28T19:20:37.871177+0000","last_became_active":"2024-08-28T19:18:32.495075+0000","last_became_peered":"2024-08-28T19:18:32.495075+0000","last_unstale":"2024-08-28T19:20:37.871177+0000","last_undegraded":"2024-08-28T19:20:37.871177+0000","last_fullsized":"2024-08-28T19:20:37.871177+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:40:22.456666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009115+0000","last_change":"2024-08-28T19:18:30.472708+0000","last_active":"2024-08-28T19:20:09.009115+0000","last_peered":"2024-08-28T19:20:09.009115+0000","last_clean":"2024-08-28T19:20:09.009115+0000","last_became_active":"2024-08-28T19:18:30.472320+0000","last_became_peered":"2024-08-28T19:18:30.472320+0000","last_unstale":"2024-08-28T19:20:09.009115+0000","last_undegraded":"2024-08-28T19:20:09.009115+0000","last_fullsized":"2024-08-28T19:20:09.009115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:57:12.055626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010437+0000","last_change":"2024-08-28T19:18:39.519398+0000","last_active":"2024-08-28T19:20:09.010437+0000","last_peered":"2024-08-28T19:20:09.010437+0000","last_clean":"2024-08-28T19:20:09.010437+0000","last_became_active":"2024-08-28T19:18:39.518997+0000","last_became_peered":"2024-08-28T19:18:39.518997+0000","last_unstale":"2024-08-28T19:20:09.010437+0000","last_undegraded":"2024-08-28T19:20:09.010437+0000","last_fullsized":"2024-08-28T19:20:09.010437+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:33.431608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.879604+0000","last_change":"2024-08-28T19:18:32.489439+0000","last_active":"2024-08-28T19:20:37.879604+0000","last_peered":"2024-08-28T19:20:37.879604+0000","last_clean":"2024-08-28T19:20:37.879604+0000","last_became_active":"2024-08-28T19:18:32.489054+0000","last_became_peered":"2024-08-28T19:18:32.489054+0000","last_unstale":"2024-08-28T19:20:37.879604+0000","last_undegraded":"2024-08-28T19:20:37.879604+0000","last_fullsized":"2024-08-28T19:20:37.879604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:31:32.647384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163358+0000","last_change":"2024-08-28T19:18:30.464993+0000","last_active":"2024-08-28T19:20:09.163358+0000","last_peered":"2024-08-28T19:20:09.163358+0000","last_clean":"2024-08-28T19:20:09.163358+0000","last_became_active":"2024-08-28T19:18:30.463951+0000","last_became_peered":"2024-08-28T19:18:30.463951+0000","last_unstale":"2024-08-28T19:20:09.163358+0000","last_undegraded":"2024-08-28T19:20:09.163358+0000","last_fullsized":"2024-08-28T19:20:09.163358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:06.780058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008234+0000","last_change":"2024-08-28T19:18:39.533061+0000","last_active":"2024-08-28T19:20:09.008234+0000","last_peered":"2024-08-28T19:20:09.008234+0000","last_clean":"2024-08-28T19:20:09.008234+0000","last_became_active":"2024-08-28T19:18:39.532607+0000","last_became_peered":"2024-08-28T19:18:39.532607+0000","last_unstale":"2024-08-28T19:20:09.008234+0000","last_undegraded":"2024-08-28T19:20:09.008234+0000","last_fullsized":"2024-08-28T19:20:09.008234+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:00:57.997665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875549+0000","last_change":"2024-08-28T19:18:32.487937+0000","last_active":"2024-08-28T19:20:37.875549+0000","last_peered":"2024-08-28T19:20:37.875549+0000","last_clean":"2024-08-28T19:20:37.875549+0000","last_became_active":"2024-08-28T19:18:32.487417+0000","last_became_peered":"2024-08-28T19:18:32.487417+0000","last_unstale":"2024-08-28T19:20:37.875549+0000","last_undegraded":"2024-08-28T19:20:37.875549+0000","last_fullsized":"2024-08-28T19:20:37.875549+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:42.497364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110517+0000","last_change":"2024-08-28T19:18:30.470742+0000","last_active":"2024-08-28T19:20:09.110517+0000","last_peered":"2024-08-28T19:20:09.110517+0000","last_clean":"2024-08-28T19:20:09.110517+0000","last_became_active":"2024-08-28T19:18:30.470135+0000","last_became_peered":"2024-08-28T19:18:30.470135+0000","last_unstale":"2024-08-28T19:20:09.110517+0000","last_undegraded":"2024-08-28T19:20:09.110517+0000","last_fullsized":"2024-08-28T19:20:09.110517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:49:39.581119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008429+0000","last_change":"2024-08-28T19:18:39.524052+0000","last_active":"2024-08-28T19:20:09.008429+0000","last_peered":"2024-08-28T19:20:09.008429+0000","last_clean":"2024-08-28T19:20:09.008429+0000","last_became_active":"2024-08-28T19:18:39.523601+0000","last_became_peered":"2024-08-28T19:18:39.523601+0000","last_unstale":"2024-08-28T19:20:09.008429+0000","last_undegraded":"2024-08-28T19:20:09.008429+0000","last_fullsized":"2024-08-28T19:20:09.008429+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:54:53.302424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883838+0000","last_change":"2024-08-28T19:18:32.491151+0000","last_active":"2024-08-28T19:20:37.883838+0000","last_peered":"2024-08-28T19:20:37.883838+0000","last_clean":"2024-08-28T19:20:37.883838+0000","last_became_active":"2024-08-28T19:18:32.490798+0000","last_became_peered":"2024-08-28T19:18:32.490798+0000","last_unstale":"2024-08-28T19:20:37.883838+0000","last_undegraded":"2024-08-28T19:20:37.883838+0000","last_fullsized":"2024-08-28T19:20:37.883838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:41:00.771019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109743+0000","last_change":"2024-08-28T19:18:31.084029+0000","last_active":"2024-08-28T19:20:09.109743+0000","last_peered":"2024-08-28T19:20:09.109743+0000","last_clean":"2024-08-28T19:20:09.109743+0000","last_became_active":"2024-08-28T19:18:31.083507+0000","last_became_peered":"2024-08-28T19:18:31.083507+0000","last_unstale":"2024-08-28T19:20:09.109743+0000","last_undegraded":"2024-08-28T19:20:09.109743+0000","last_fullsized":"2024-08-28T19:20:09.109743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:15:36.451702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163253+0000","last_change":"2024-08-28T19:18:39.527249+0000","last_active":"2024-08-28T19:20:09.163253+0000","last_peered":"2024-08-28T19:20:09.163253+0000","last_clean":"2024-08-28T19:20:09.163253+0000","last_became_active":"2024-08-28T19:18:39.526834+0000","last_became_peered":"2024-08-28T19:18:39.526834+0000","last_unstale":"2024-08-28T19:20:09.163253+0000","last_undegraded":"2024-08-28T19:20:09.163253+0000","last_fullsized":"2024-08-28T19:20:09.163253+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:48:02.294738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.876604+0000","last_change":"2024-08-28T19:18:32.497337+0000","last_active":"2024-08-28T19:20:37.876604+0000","last_peered":"2024-08-28T19:20:37.876604+0000","last_clean":"2024-08-28T19:20:37.876604+0000","last_became_active":"2024-08-28T19:18:32.496929+0000","last_became_peered":"2024-08-28T19:18:32.496929+0000","last_unstale":"2024-08-28T19:20:37.876604+0000","last_undegraded":"2024-08-28T19:20:37.876604+0000","last_fullsized":"2024-08-28T19:20:37.876604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:14.748765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":149,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:53.806036+0000","last_change":"2024-08-28T19:18:30.471353+0000","last_active":"2024-08-28T19:20:53.806036+0000","last_peered":"2024-08-28T19:20:53.806036+0000","last_clean":"2024-08-28T19:20:53.806036+0000","last_became_active":"2024-08-28T19:18:30.470283+0000","last_became_peered":"2024-08-28T19:18:30.470283+0000","last_unstale":"2024-08-28T19:20:53.806036+0000","last_undegraded":"2024-08-28T19:20:53.806036+0000","last_fullsized":"2024-08-28T19:20:53.806036+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:09:46.444864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007249+0000","last_change":"2024-08-28T19:18:39.519226+0000","last_active":"2024-08-28T19:20:09.007249+0000","last_peered":"2024-08-28T19:20:09.007249+0000","last_clean":"2024-08-28T19:20:09.007249+0000","last_became_active":"2024-08-28T19:18:39.518738+0000","last_became_peered":"2024-08-28T19:18:39.518738+0000","last_unstale":"2024-08-28T19:20:09.007249+0000","last_undegraded":"2024-08-28T19:20:09.007249+0000","last_fullsized":"2024-08-28T19:20:09.007249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:39:35.114702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.877692+0000","last_change":"2024-08-28T19:18:32.493060+0000","last_active":"2024-08-28T19:20:37.877692+0000","last_peered":"2024-08-28T19:20:37.877692+0000","last_clean":"2024-08-28T19:20:37.877692+0000","last_became_active":"2024-08-28T19:18:32.492590+0000","last_became_peered":"2024-08-28T19:18:32.492590+0000","last_unstale":"2024-08-28T19:20:37.877692+0000","last_undegraded":"2024-08-28T19:20:37.877692+0000","last_fullsized":"2024-08-28T19:20:37.877692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:02:02.165301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008761+0000","last_change":"2024-08-28T19:18:30.470674+0000","last_active":"2024-08-28T19:20:09.008761+0000","last_peered":"2024-08-28T19:20:09.008761+0000","last_clean":"2024-08-28T19:20:09.008761+0000","last_became_active":"2024-08-28T19:18:30.470275+0000","last_became_peered":"2024-08-28T19:18:30.470275+0000","last_unstale":"2024-08-28T19:20:09.008761+0000","last_undegraded":"2024-08-28T19:20:09.008761+0000","last_fullsized":"2024-08-28T19:20:09.008761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:01:48.547315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663504+0000","last_change":"2024-08-28T19:19:08.624194+0000","last_active":"2024-08-28T19:20:09.663504+0000","last_peered":"2024-08-28T19:20:09.663504+0000","last_clean":"2024-08-28T19:20:09.663504+0000","last_became_active":"2024-08-28T19:19:08.623681+0000","last_became_peered":"2024-08-28T19:19:08.623681+0000","last_unstale":"2024-08-28T19:20:09.663504+0000","last_undegraded":"2024-08-28T19:20:09.663504+0000","last_fullsized":"2024-08-28T19:20:09.663504+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:21:11.099931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872115+0000","last_change":"2024-08-28T19:18:32.488210+0000","last_active":"2024-08-28T19:20:37.872115+0000","last_peered":"2024-08-28T19:20:37.872115+0000","last_clean":"2024-08-28T19:20:37.872115+0000","last_became_active":"2024-08-28T19:18:32.487853+0000","last_became_peered":"2024-08-28T19:18:32.487853+0000","last_unstale":"2024-08-28T19:20:37.872115+0000","last_undegraded":"2024-08-28T19:20:37.872115+0000","last_fullsized":"2024-08-28T19:20:37.872115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:20:51.808958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011302+0000","last_change":"2024-08-28T19:18:30.465306+0000","last_active":"2024-08-28T19:20:09.011302+0000","last_peered":"2024-08-28T19:20:09.011302+0000","last_clean":"2024-08-28T19:20:09.011302+0000","last_became_active":"2024-08-28T19:18:30.464850+0000","last_became_peered":"2024-08-28T19:18:30.464850+0000","last_unstale":"2024-08-28T19:20:09.011302+0000","last_undegraded":"2024-08-28T19:20:09.011302+0000","last_fullsized":"2024-08-28T19:20:09.011302+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:24:26.628999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008657+0000","last_change":"2024-08-28T19:18:39.524005+0000","last_active":"2024-08-28T19:20:09.008657+0000","last_peered":"2024-08-28T19:20:09.008657+0000","last_clean":"2024-08-28T19:20:09.008657+0000","last_became_active":"2024-08-28T19:18:39.523594+0000","last_became_peered":"2024-08-28T19:18:39.523594+0000","last_unstale":"2024-08-28T19:20:09.008657+0000","last_undegraded":"2024-08-28T19:20:09.008657+0000","last_fullsized":"2024-08-28T19:20:09.008657+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:35:32.484399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"84'6","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872604+0000","last_change":"2024-08-28T19:18:32.489088+0000","last_active":"2024-08-28T19:20:37.872604+0000","last_peered":"2024-08-28T19:20:37.872604+0000","last_clean":"2024-08-28T19:20:37.872604+0000","last_became_active":"2024-08-28T19:18:32.488717+0000","last_became_peered":"2024-08-28T19:18:32.488717+0000","last_unstale":"2024-08-28T19:20:37.872604+0000","last_undegraded":"2024-08-28T19:20:37.872604+0000","last_fullsized":"2024-08-28T19:20:37.872604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:25.468919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"84'40","reported_seq":297,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.807945+0000","last_change":"2024-08-28T19:18:30.468674+0000","last_active":"2024-08-28T19:20:54.807945+0000","last_peered":"2024-08-28T19:20:54.807945+0000","last_clean":"2024-08-28T19:20:54.807945+0000","last_became_active":"2024-08-28T19:18:30.468300+0000","last_became_peered":"2024-08-28T19:18:30.468300+0000","last_unstale":"2024-08-28T19:20:54.807945+0000","last_undegraded":"2024-08-28T19:20:54.807945+0000","last_fullsized":"2024-08-28T19:20:54.807945+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:12:06.237498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64032,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":217,"num_read_kb":171,"num_write":42,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007081+0000","last_change":"2024-08-28T19:18:39.526131+0000","last_active":"2024-08-28T19:20:09.007081+0000","last_peered":"2024-08-28T19:20:09.007081+0000","last_clean":"2024-08-28T19:20:09.007081+0000","last_became_active":"2024-08-28T19:18:39.525708+0000","last_became_peered":"2024-08-28T19:18:39.525708+0000","last_unstale":"2024-08-28T19:20:09.007081+0000","last_undegraded":"2024-08-28T19:20:09.007081+0000","last_fullsized":"2024-08-28T19:20:09.007081+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:21:52.156487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.885078+0000","last_change":"2024-08-28T19:18:32.487634+0000","last_active":"2024-08-28T19:20:37.885078+0000","last_peered":"2024-08-28T19:20:37.885078+0000","last_clean":"2024-08-28T19:20:37.885078+0000","last_became_active":"2024-08-28T19:18:32.486702+0000","last_became_peered":"2024-08-28T19:18:32.486702+0000","last_unstale":"2024-08-28T19:20:37.885078+0000","last_undegraded":"2024-08-28T19:20:37.885078+0000","last_fullsized":"2024-08-28T19:20:37.885078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:08:12.606905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008985+0000","last_change":"2024-08-28T19:18:31.083494+0000","last_active":"2024-08-28T19:20:09.008985+0000","last_peered":"2024-08-28T19:20:09.008985+0000","last_clean":"2024-08-28T19:20:09.008985+0000","last_became_active":"2024-08-28T19:18:31.083086+0000","last_became_peered":"2024-08-28T19:18:31.083086+0000","last_unstale":"2024-08-28T19:20:09.008985+0000","last_undegraded":"2024-08-28T19:20:09.008985+0000","last_fullsized":"2024-08-28T19:20:09.008985+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:14:54.388364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615373+0000","last_change":"2024-08-28T19:18:39.532155+0000","last_active":"2024-08-28T19:20:09.615373+0000","last_peered":"2024-08-28T19:20:09.615373+0000","last_clean":"2024-08-28T19:20:09.615373+0000","last_became_active":"2024-08-28T19:18:39.531565+0000","last_became_peered":"2024-08-28T19:18:39.531565+0000","last_unstale":"2024-08-28T19:20:09.615373+0000","last_undegraded":"2024-08-28T19:20:09.615373+0000","last_fullsized":"2024-08-28T19:20:09.615373+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:29:48.824622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"84'9","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880917+0000","last_change":"2024-08-28T19:18:32.496296+0000","last_active":"2024-08-28T19:20:37.880917+0000","last_peered":"2024-08-28T19:20:37.880917+0000","last_clean":"2024-08-28T19:20:37.880917+0000","last_became_active":"2024-08-28T19:18:32.495680+0000","last_became_peered":"2024-08-28T19:18:32.495680+0000","last_unstale":"2024-08-28T19:20:37.880917+0000","last_undegraded":"2024-08-28T19:20:37.880917+0000","last_fullsized":"2024-08-28T19:20:37.880917+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:27:34.544529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529311+0000","last_change":"2024-08-28T19:19:08.631758+0000","last_active":"2024-08-28T19:20:09.529311+0000","last_peered":"2024-08-28T19:20:09.529311+0000","last_clean":"2024-08-28T19:20:09.529311+0000","last_became_active":"2024-08-28T19:19:08.631202+0000","last_became_peered":"2024-08-28T19:19:08.631202+0000","last_unstale":"2024-08-28T19:20:09.529311+0000","last_undegraded":"2024-08-28T19:20:09.529311+0000","last_fullsized":"2024-08-28T19:20:09.529311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:27:43.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.006719+0000","last_change":"2024-08-28T19:18:39.531157+0000","last_active":"2024-08-28T19:20:09.006719+0000","last_peered":"2024-08-28T19:20:09.006719+0000","last_clean":"2024-08-28T19:20:09.006719+0000","last_became_active":"2024-08-28T19:18:39.530675+0000","last_became_peered":"2024-08-28T19:18:39.530675+0000","last_unstale":"2024-08-28T19:20:09.006719+0000","last_undegraded":"2024-08-28T19:20:09.006719+0000","last_fullsized":"2024-08-28T19:20:09.006719+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:38:40.414854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"84'10","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883543+0000","last_change":"2024-08-28T19:18:32.482182+0000","last_active":"2024-08-28T19:20:37.883543+0000","last_peered":"2024-08-28T19:20:37.883543+0000","last_clean":"2024-08-28T19:20:37.883543+0000","last_became_active":"2024-08-28T19:18:32.481689+0000","last_became_peered":"2024-08-28T19:18:32.481689+0000","last_unstale":"2024-08-28T19:20:37.883543+0000","last_undegraded":"2024-08-28T19:20:37.883543+0000","last_fullsized":"2024-08-28T19:20:37.883543+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:13:29.278469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'5","reported_seq":251,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.850315+0000","last_change":"2024-08-28T19:18:30.475244+0000","last_active":"2024-08-28T19:20:54.850315+0000","last_peered":"2024-08-28T19:20:54.850315+0000","last_clean":"2024-08-28T19:20:54.850315+0000","last_became_active":"2024-08-28T19:18:30.474861+0000","last_became_peered":"2024-08-28T19:18:30.474861+0000","last_unstale":"2024-08-28T19:20:54.850315+0000","last_undegraded":"2024-08-28T19:20:54.850315+0000","last_fullsized":"2024-08-28T19:20:54.850315+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:27:09.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":206,"num_read_kb":206,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525040+0000","last_change":"2024-08-28T19:18:39.520022+0000","last_active":"2024-08-28T19:20:09.525040+0000","last_peered":"2024-08-28T19:20:09.525040+0000","last_clean":"2024-08-28T19:20:09.525040+0000","last_became_active":"2024-08-28T19:18:39.519584+0000","last_became_peered":"2024-08-28T19:18:39.519584+0000","last_unstale":"2024-08-28T19:20:09.525040+0000","last_undegraded":"2024-08-28T19:20:09.525040+0000","last_fullsized":"2024-08-28T19:20:09.525040+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:27:15.851303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010363+0000","last_change":"2024-08-28T19:18:39.522607+0000","last_active":"2024-08-28T19:20:09.010363+0000","last_peered":"2024-08-28T19:20:09.010363+0000","last_clean":"2024-08-28T19:20:09.010363+0000","last_became_active":"2024-08-28T19:18:39.522204+0000","last_became_peered":"2024-08-28T19:18:39.522204+0000","last_unstale":"2024-08-28T19:20:09.010363+0000","last_undegraded":"2024-08-28T19:20:09.010363+0000","last_fullsized":"2024-08-28T19:20:09.010363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:02:08.865294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"84'11","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865503+0000","last_change":"2024-08-28T19:18:32.485577+0000","last_active":"2024-08-28T19:20:37.865503+0000","last_peered":"2024-08-28T19:20:37.865503+0000","last_clean":"2024-08-28T19:20:37.865503+0000","last_became_active":"2024-08-28T19:18:32.485243+0000","last_became_peered":"2024-08-28T19:18:32.485243+0000","last_unstale":"2024-08-28T19:20:37.865503+0000","last_undegraded":"2024-08-28T19:20:37.865503+0000","last_fullsized":"2024-08-28T19:20:37.865503+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:05:53.019365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008741+0000","last_change":"2024-08-28T19:18:30.470670+0000","last_active":"2024-08-28T19:20:09.008741+0000","last_peered":"2024-08-28T19:20:09.008741+0000","last_clean":"2024-08-28T19:20:09.008741+0000","last_became_active":"2024-08-28T19:18:30.470264+0000","last_became_peered":"2024-08-28T19:18:30.470264+0000","last_unstale":"2024-08-28T19:20:09.008741+0000","last_undegraded":"2024-08-28T19:20:09.008741+0000","last_fullsized":"2024-08-28T19:20:09.008741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:35.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663847+0000","last_change":"2024-08-28T19:18:39.530948+0000","last_active":"2024-08-28T19:20:09.663847+0000","last_peered":"2024-08-28T19:20:09.663847+0000","last_clean":"2024-08-28T19:20:09.663847+0000","last_became_active":"2024-08-28T19:18:39.530473+0000","last_became_peered":"2024-08-28T19:18:39.530473+0000","last_unstale":"2024-08-28T19:20:09.663847+0000","last_undegraded":"2024-08-28T19:20:09.663847+0000","last_fullsized":"2024-08-28T19:20:09.663847+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:21:57.504582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"84'8","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.869758+0000","last_change":"2024-08-28T19:18:32.486027+0000","last_active":"2024-08-28T19:20:37.869758+0000","last_peered":"2024-08-28T19:20:37.869758+0000","last_clean":"2024-08-28T19:20:37.869758+0000","last_became_active":"2024-08-28T19:18:32.483273+0000","last_became_peered":"2024-08-28T19:18:32.483273+0000","last_unstale":"2024-08-28T19:20:37.869758+0000","last_undegraded":"2024-08-28T19:20:37.869758+0000","last_fullsized":"2024-08-28T19:20:37.869758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:47:37.680695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162978+0000","last_change":"2024-08-28T19:18:30.464811+0000","last_active":"2024-08-28T19:20:09.162978+0000","last_peered":"2024-08-28T19:20:09.162978+0000","last_clean":"2024-08-28T19:20:09.162978+0000","last_became_active":"2024-08-28T19:18:30.463986+0000","last_became_peered":"2024-08-28T19:18:30.463986+0000","last_unstale":"2024-08-28T19:20:09.162978+0000","last_undegraded":"2024-08-28T19:20:09.162978+0000","last_fullsized":"2024-08-28T19:20:09.162978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:43:46.582993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":53,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663544+0000","last_change":"2024-08-28T19:19:08.631382+0000","last_active":"2024-08-28T19:20:09.663544+0000","last_peered":"2024-08-28T19:20:09.663544+0000","last_clean":"2024-08-28T19:20:09.663544+0000","last_became_active":"2024-08-28T19:19:08.630869+0000","last_became_peered":"2024-08-28T19:19:08.630869+0000","last_unstale":"2024-08-28T19:20:09.663544+0000","last_undegraded":"2024-08-28T19:20:09.663544+0000","last_fullsized":"2024-08-28T19:20:09.663544+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:32:54.942408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"84'8","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875341+0000","last_change":"2024-08-28T19:18:32.484120+0000","last_active":"2024-08-28T19:20:37.875341+0000","last_peered":"2024-08-28T19:20:37.875341+0000","last_clean":"2024-08-28T19:20:37.875341+0000","last_became_active":"2024-08-28T19:18:32.483013+0000","last_became_peered":"2024-08-28T19:18:32.483013+0000","last_unstale":"2024-08-28T19:20:37.875341+0000","last_undegraded":"2024-08-28T19:20:37.875341+0000","last_fullsized":"2024-08-28T19:20:37.875341+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:59:41.195873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010737+0000","last_change":"2024-08-28T19:18:39.531046+0000","last_active":"2024-08-28T19:20:09.010737+0000","last_peered":"2024-08-28T19:20:09.010737+0000","last_clean":"2024-08-28T19:20:09.010737+0000","last_became_active":"2024-08-28T19:18:39.530407+0000","last_became_peered":"2024-08-28T19:18:39.530407+0000","last_unstale":"2024-08-28T19:20:09.010737+0000","last_undegraded":"2024-08-28T19:20:09.010737+0000","last_fullsized":"2024-08-28T19:20:09.010737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:47:15.173021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'7","reported_seq":151,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:51.803528+0000","last_change":"2024-08-28T19:18:30.471268+0000","last_active":"2024-08-28T19:20:51.803528+0000","last_peered":"2024-08-28T19:20:51.803528+0000","last_clean":"2024-08-28T19:20:51.803528+0000","last_became_active":"2024-08-28T19:18:30.470845+0000","last_became_peered":"2024-08-28T19:18:30.470845+0000","last_unstale":"2024-08-28T19:20:51.803528+0000","last_undegraded":"2024-08-28T19:20:51.803528+0000","last_fullsized":"2024-08-28T19:20:51.803528+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:18:18.086205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"84'7","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.859389+0000","last_change":"2024-08-28T19:18:32.479728+0000","last_active":"2024-08-28T19:20:37.859389+0000","last_peered":"2024-08-28T19:20:37.859389+0000","last_clean":"2024-08-28T19:20:37.859389+0000","last_became_active":"2024-08-28T19:18:32.478623+0000","last_became_peered":"2024-08-28T19:18:32.478623+0000","last_unstale":"2024-08-28T19:20:37.859389+0000","last_undegraded":"2024-08-28T19:20:37.859389+0000","last_fullsized":"2024-08-28T19:20:37.859389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:14:49.914808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011224+0000","last_change":"2024-08-28T19:18:39.531308+0000","last_active":"2024-08-28T19:20:09.011224+0000","last_peered":"2024-08-28T19:20:09.011224+0000","last_clean":"2024-08-28T19:20:09.011224+0000","last_became_active":"2024-08-28T19:18:39.530403+0000","last_became_peered":"2024-08-28T19:18:39.530403+0000","last_unstale":"2024-08-28T19:20:09.011224+0000","last_undegraded":"2024-08-28T19:20:09.011224+0000","last_fullsized":"2024-08-28T19:20:09.011224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:33.139668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873003+0000","last_change":"2024-08-28T19:18:32.494621+0000","last_active":"2024-08-28T19:20:37.873003+0000","last_peered":"2024-08-28T19:20:37.873003+0000","last_clean":"2024-08-28T19:20:37.873003+0000","last_became_active":"2024-08-28T19:18:32.494217+0000","last_became_peered":"2024-08-28T19:18:32.494217+0000","last_unstale":"2024-08-28T19:20:37.873003+0000","last_undegraded":"2024-08-28T19:20:37.873003+0000","last_fullsized":"2024-08-28T19:20:37.873003+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:54:15.419470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"84'12","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872043+0000","last_change":"2024-08-28T19:20:09.616552+0000","last_active":"2024-08-28T19:20:37.872043+0000","last_peered":"2024-08-28T19:20:37.872043+0000","last_clean":"2024-08-28T19:20:37.872043+0000","last_became_active":"2024-08-28T19:20:09.616159+0000","last_became_peered":"2024-08-28T19:20:09.616159+0000","last_unstale":"2024-08-28T19:20:37.872043+0000","last_undegraded":"2024-08-28T19:20:37.872043+0000","last_fullsized":"2024-08-28T19:20:37.872043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:04:10.354703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"84'8","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866959+0000","last_change":"2024-08-28T19:20:09.012824+0000","last_active":"2024-08-28T19:20:37.866959+0000","last_peered":"2024-08-28T19:20:37.866959+0000","last_clean":"2024-08-28T19:20:37.866959+0000","last_became_active":"2024-08-28T19:20:09.012490+0000","last_became_peered":"2024-08-28T19:20:09.012490+0000","last_unstale":"2024-08-28T19:20:37.866959+0000","last_undegraded":"2024-08-28T19:20:37.866959+0000","last_fullsized":"2024-08-28T19:20:37.866959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:49:02.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871666+0000","last_change":"2024-08-28T19:20:09.164523+0000","last_active":"2024-08-28T19:20:37.871666+0000","last_peered":"2024-08-28T19:20:37.871666+0000","last_clean":"2024-08-28T19:20:37.871666+0000","last_became_active":"2024-08-28T19:20:09.163825+0000","last_became_peered":"2024-08-28T19:20:09.163825+0000","last_unstale":"2024-08-28T19:20:37.871666+0000","last_undegraded":"2024-08-28T19:20:37.871666+0000","last_fullsized":"2024-08-28T19:20:37.871666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:42:48.141103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868174+0000","last_change":"2024-08-28T19:18:32.485446+0000","last_active":"2024-08-28T19:20:37.868174+0000","last_peered":"2024-08-28T19:20:37.868174+0000","last_clean":"2024-08-28T19:20:37.868174+0000","last_became_active":"2024-08-28T19:18:32.484197+0000","last_became_peered":"2024-08-28T19:18:32.484197+0000","last_unstale":"2024-08-28T19:20:37.868174+0000","last_undegraded":"2024-08-28T19:20:37.868174+0000","last_fullsized":"2024-08-28T19:20:37.868174+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:47:50.527915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868339+0000","last_change":"2024-08-28T19:18:32.489452+0000","last_active":"2024-08-28T19:20:37.868339+0000","last_peered":"2024-08-28T19:20:37.868339+0000","last_clean":"2024-08-28T19:20:37.868339+0000","last_became_active":"2024-08-28T19:18:32.487482+0000","last_became_peered":"2024-08-28T19:18:32.487482+0000","last_unstale":"2024-08-28T19:20:37.868339+0000","last_undegraded":"2024-08-28T19:20:37.868339+0000","last_fullsized":"2024-08-28T19:20:37.868339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:52:08.175080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884068+0000","last_change":"2024-08-28T19:18:32.483850+0000","last_active":"2024-08-28T19:20:37.884068+0000","last_peered":"2024-08-28T19:20:37.884068+0000","last_clean":"2024-08-28T19:20:37.884068+0000","last_became_active":"2024-08-28T19:18:32.483476+0000","last_became_peered":"2024-08-28T19:18:32.483476+0000","last_unstale":"2024-08-28T19:20:37.884068+0000","last_undegraded":"2024-08-28T19:20:37.884068+0000","last_fullsized":"2024-08-28T19:20:37.884068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:32:37.465218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868836+0000","last_change":"2024-08-28T19:18:32.492252+0000","last_active":"2024-08-28T19:20:37.868836+0000","last_peered":"2024-08-28T19:20:37.868836+0000","last_clean":"2024-08-28T19:20:37.868836+0000","last_became_active":"2024-08-28T19:18:32.491659+0000","last_became_peered":"2024-08-28T19:18:32.491659+0000","last_unstale":"2024-08-28T19:20:37.868836+0000","last_undegraded":"2024-08-28T19:20:37.868836+0000","last_fullsized":"2024-08-28T19:20:37.868836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:19:28.901559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865090+0000","last_change":"2024-08-28T19:18:32.491940+0000","last_active":"2024-08-28T19:20:37.865090+0000","last_peered":"2024-08-28T19:20:37.865090+0000","last_clean":"2024-08-28T19:20:37.865090+0000","last_became_active":"2024-08-28T19:18:32.487474+0000","last_became_peered":"2024-08-28T19:18:32.487474+0000","last_unstale":"2024-08-28T19:20:37.865090+0000","last_undegraded":"2024-08-28T19:20:37.865090+0000","last_fullsized":"2024-08-28T19:20:37.865090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:04:32.508620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871438+0000","last_change":"2024-08-28T19:18:32.485200+0000","last_active":"2024-08-28T19:20:37.871438+0000","last_peered":"2024-08-28T19:20:37.871438+0000","last_clean":"2024-08-28T19:20:37.871438+0000","last_became_active":"2024-08-28T19:18:32.484858+0000","last_became_peered":"2024-08-28T19:18:32.484858+0000","last_unstale":"2024-08-28T19:20:37.871438+0000","last_undegraded":"2024-08-28T19:20:37.871438+0000","last_fullsized":"2024-08-28T19:20:37.871438+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:10:57.925659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886493+0000","last_change":"2024-08-28T19:18:32.489903+0000","last_active":"2024-08-28T19:20:37.886493+0000","last_peered":"2024-08-28T19:20:37.886493+0000","last_clean":"2024-08-28T19:20:37.886493+0000","last_became_active":"2024-08-28T19:18:32.488950+0000","last_became_peered":"2024-08-28T19:18:32.488950+0000","last_unstale":"2024-08-28T19:20:37.886493+0000","last_undegraded":"2024-08-28T19:20:37.886493+0000","last_fullsized":"2024-08-28T19:20:37.886493+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:06:42.284177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872318+0000","last_change":"2024-08-28T19:18:32.488236+0000","last_active":"2024-08-28T19:20:37.872318+0000","last_peered":"2024-08-28T19:20:37.872318+0000","last_clean":"2024-08-28T19:20:37.872318+0000","last_became_active":"2024-08-28T19:18:32.487776+0000","last_became_peered":"2024-08-28T19:18:32.487776+0000","last_unstale":"2024-08-28T19:20:37.872318+0000","last_undegraded":"2024-08-28T19:20:37.872318+0000","last_fullsized":"2024-08-28T19:20:37.872318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:25:58.754740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875857+0000","last_change":"2024-08-28T19:18:32.485860+0000","last_active":"2024-08-28T19:20:37.875857+0000","last_peered":"2024-08-28T19:20:37.875857+0000","last_clean":"2024-08-28T19:20:37.875857+0000","last_became_active":"2024-08-28T19:18:32.485415+0000","last_became_peered":"2024-08-28T19:18:32.485415+0000","last_unstale":"2024-08-28T19:20:37.875857+0000","last_undegraded":"2024-08-28T19:20:37.875857+0000","last_fullsized":"2024-08-28T19:20:37.875857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:45:32.868651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873054+0000","last_change":"2024-08-28T19:18:32.485515+0000","last_active":"2024-08-28T19:20:37.873054+0000","last_peered":"2024-08-28T19:20:37.873054+0000","last_clean":"2024-08-28T19:20:37.873054+0000","last_became_active":"2024-08-28T19:18:32.485075+0000","last_became_peered":"2024-08-28T19:18:32.485075+0000","last_unstale":"2024-08-28T19:20:37.873054+0000","last_undegraded":"2024-08-28T19:20:37.873054+0000","last_fullsized":"2024-08-28T19:20:37.873054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:01:28.035479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864600+0000","last_change":"2024-08-28T19:18:32.484685+0000","last_active":"2024-08-28T19:20:37.864600+0000","last_peered":"2024-08-28T19:20:37.864600+0000","last_clean":"2024-08-28T19:20:37.864600+0000","last_became_active":"2024-08-28T19:18:32.484095+0000","last_became_peered":"2024-08-28T19:18:32.484095+0000","last_unstale":"2024-08-28T19:20:37.864600+0000","last_undegraded":"2024-08-28T19:20:37.864600+0000","last_fullsized":"2024-08-28T19:20:37.864600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:53:27.229186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884050+0000","last_change":"2024-08-28T19:18:32.488030+0000","last_active":"2024-08-28T19:20:37.884050+0000","last_peered":"2024-08-28T19:20:37.884050+0000","last_clean":"2024-08-28T19:20:37.884050+0000","last_became_active":"2024-08-28T19:18:32.487042+0000","last_became_peered":"2024-08-28T19:18:32.487042+0000","last_unstale":"2024-08-28T19:20:37.884050+0000","last_undegraded":"2024-08-28T19:20:37.884050+0000","last_fullsized":"2024-08-28T19:20:37.884050+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:56.966475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886125+0000","last_change":"2024-08-28T19:18:32.489442+0000","last_active":"2024-08-28T19:20:37.886125+0000","last_peered":"2024-08-28T19:20:37.886125+0000","last_clean":"2024-08-28T19:20:37.886125+0000","last_became_active":"2024-08-28T19:18:32.488405+0000","last_became_peered":"2024-08-28T19:18:32.488405+0000","last_unstale":"2024-08-28T19:20:37.886125+0000","last_undegraded":"2024-08-28T19:20:37.886125+0000","last_fullsized":"2024-08-28T19:20:37.886125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:21:03.026893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870138+0000","last_change":"2024-08-28T19:20:09.164305+0000","last_active":"2024-08-28T19:20:37.870138+0000","last_peered":"2024-08-28T19:20:37.870138+0000","last_clean":"2024-08-28T19:20:37.870138+0000","last_became_active":"2024-08-28T19:20:09.163834+0000","last_became_peered":"2024-08-28T19:20:09.163834+0000","last_unstale":"2024-08-28T19:20:37.870138+0000","last_undegraded":"2024-08-28T19:20:37.870138+0000","last_fullsized":"2024-08-28T19:20:37.870138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:20:10.137952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870501+0000","last_change":"2024-08-28T19:18:32.492077+0000","last_active":"2024-08-28T19:20:37.870501+0000","last_peered":"2024-08-28T19:20:37.870501+0000","last_clean":"2024-08-28T19:20:37.870501+0000","last_became_active":"2024-08-28T19:18:32.491722+0000","last_became_peered":"2024-08-28T19:18:32.491722+0000","last_unstale":"2024-08-28T19:20:37.870501+0000","last_undegraded":"2024-08-28T19:20:37.870501+0000","last_fullsized":"2024-08-28T19:20:37.870501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:16:21.697045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"84'10","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865421+0000","last_change":"2024-08-28T19:18:32.479845+0000","last_active":"2024-08-28T19:20:37.865421+0000","last_peered":"2024-08-28T19:20:37.865421+0000","last_clean":"2024-08-28T19:20:37.865421+0000","last_became_active":"2024-08-28T19:18:32.477690+0000","last_became_peered":"2024-08-28T19:18:32.477690+0000","last_unstale":"2024-08-28T19:20:37.865421+0000","last_undegraded":"2024-08-28T19:20:37.865421+0000","last_fullsized":"2024-08-28T19:20:37.865421+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:11.281415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866009+0000","last_change":"2024-08-28T19:19:08.624185+0000","last_active":"2024-08-28T19:20:37.866009+0000","last_peered":"2024-08-28T19:20:37.866009+0000","last_clean":"2024-08-28T19:20:37.866009+0000","last_became_active":"2024-08-28T19:19:08.623745+0000","last_became_peered":"2024-08-28T19:19:08.623745+0000","last_unstale":"2024-08-28T19:20:37.866009+0000","last_undegraded":"2024-08-28T19:20:37.866009+0000","last_fullsized":"2024-08-28T19:20:37.866009+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:56.657173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"84'6","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886865+0000","last_change":"2024-08-28T19:18:32.499038+0000","last_active":"2024-08-28T19:20:37.886865+0000","last_peered":"2024-08-28T19:20:37.886865+0000","last_clean":"2024-08-28T19:20:37.886865+0000","last_became_active":"2024-08-28T19:18:32.498276+0000","last_became_peered":"2024-08-28T19:18:32.498276+0000","last_unstale":"2024-08-28T19:20:37.886865+0000","last_undegraded":"2024-08-28T19:20:37.886865+0000","last_fullsized":"2024-08-28T19:20:37.886865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:17:45.468448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"84'16","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.882260+0000","last_change":"2024-08-28T19:18:32.492183+0000","last_active":"2024-08-28T19:20:37.882260+0000","last_peered":"2024-08-28T19:20:37.882260+0000","last_clean":"2024-08-28T19:20:37.882260+0000","last_became_active":"2024-08-28T19:18:32.491815+0000","last_became_peered":"2024-08-28T19:18:32.491815+0000","last_unstale":"2024-08-28T19:20:37.882260+0000","last_undegraded":"2024-08-28T19:20:37.882260+0000","last_fullsized":"2024-08-28T19:20:37.882260+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:13:11.183955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"84'11","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886721+0000","last_change":"2024-08-28T19:18:32.497399+0000","last_active":"2024-08-28T19:20:37.886721+0000","last_peered":"2024-08-28T19:20:37.886721+0000","last_clean":"2024-08-28T19:20:37.886721+0000","last_became_active":"2024-08-28T19:18:32.496901+0000","last_became_peered":"2024-08-28T19:18:32.496901+0000","last_unstale":"2024-08-28T19:20:37.886721+0000","last_undegraded":"2024-08-28T19:20:37.886721+0000","last_fullsized":"2024-08-28T19:20:37.886721+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:05:57.634590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"84'9","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864415+0000","last_change":"2024-08-28T19:18:32.495871+0000","last_active":"2024-08-28T19:20:37.864415+0000","last_peered":"2024-08-28T19:20:37.864415+0000","last_clean":"2024-08-28T19:20:37.864415+0000","last_became_active":"2024-08-28T19:18:32.495515+0000","last_became_peered":"2024-08-28T19:18:32.495515+0000","last_unstale":"2024-08-28T19:20:37.864415+0000","last_undegraded":"2024-08-28T19:20:37.864415+0000","last_fullsized":"2024-08-28T19:20:37.864415+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:53:56.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"84'5","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875276+0000","last_change":"2024-08-28T19:18:32.495711+0000","last_active":"2024-08-28T19:20:37.875276+0000","last_peered":"2024-08-28T19:20:37.875276+0000","last_clean":"2024-08-28T19:20:37.875276+0000","last_became_active":"2024-08-28T19:18:32.495271+0000","last_became_peered":"2024-08-28T19:18:32.495271+0000","last_unstale":"2024-08-28T19:20:37.875276+0000","last_undegraded":"2024-08-28T19:20:37.875276+0000","last_fullsized":"2024-08-28T19:20:37.875276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:05.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"84'6","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880581+0000","last_change":"2024-08-28T19:18:32.489731+0000","last_active":"2024-08-28T19:20:37.880581+0000","last_peered":"2024-08-28T19:20:37.880581+0000","last_clean":"2024-08-28T19:20:37.880581+0000","last_became_active":"2024-08-28T19:18:32.489366+0000","last_became_peered":"2024-08-28T19:18:32.489366+0000","last_unstale":"2024-08-28T19:20:37.880581+0000","last_undegraded":"2024-08-28T19:20:37.880581+0000","last_fullsized":"2024-08-28T19:20:37.880581+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:52:24.180155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"84'4","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.861147+0000","last_change":"2024-08-28T19:18:32.490270+0000","last_active":"2024-08-28T19:20:37.861147+0000","last_peered":"2024-08-28T19:20:37.861147+0000","last_clean":"2024-08-28T19:20:37.861147+0000","last_became_active":"2024-08-28T19:18:32.489667+0000","last_became_peered":"2024-08-28T19:18:32.489667+0000","last_unstale":"2024-08-28T19:20:37.861147+0000","last_undegraded":"2024-08-28T19:20:37.861147+0000","last_fullsized":"2024-08-28T19:20:37.861147+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:18.571978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873527+0000","last_change":"2024-08-28T19:18:32.482553+0000","last_active":"2024-08-28T19:20:37.873527+0000","last_peered":"2024-08-28T19:20:37.873527+0000","last_clean":"2024-08-28T19:20:37.873527+0000","last_became_active":"2024-08-28T19:18:32.481161+0000","last_became_peered":"2024-08-28T19:18:32.481161+0000","last_unstale":"2024-08-28T19:20:37.873527+0000","last_undegraded":"2024-08-28T19:20:37.873527+0000","last_fullsized":"2024-08-28T19:20:37.873527+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:13.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883684+0000","last_change":"2024-08-28T19:18:32.492473+0000","last_active":"2024-08-28T19:20:37.883684+0000","last_peered":"2024-08-28T19:20:37.883684+0000","last_clean":"2024-08-28T19:20:37.883684+0000","last_became_active":"2024-08-28T19:18:32.491897+0000","last_became_peered":"2024-08-28T19:18:32.491897+0000","last_unstale":"2024-08-28T19:20:37.883684+0000","last_undegraded":"2024-08-28T19:20:37.883684+0000","last_fullsized":"2024-08-28T19:20:37.883684+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:24.279391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870908+0000","last_change":"2024-08-28T19:18:32.485621+0000","last_active":"2024-08-28T19:20:37.870908+0000","last_peered":"2024-08-28T19:20:37.870908+0000","last_clean":"2024-08-28T19:20:37.870908+0000","last_became_active":"2024-08-28T19:18:32.485267+0000","last_became_peered":"2024-08-28T19:18:32.485267+0000","last_unstale":"2024-08-28T19:20:37.870908+0000","last_undegraded":"2024-08-28T19:20:37.870908+0000","last_fullsized":"2024-08-28T19:20:37.870908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:30:08.932135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884032+0000","last_change":"2024-08-28T19:18:32.488457+0000","last_active":"2024-08-28T19:20:37.884032+0000","last_peered":"2024-08-28T19:20:37.884032+0000","last_clean":"2024-08-28T19:20:37.884032+0000","last_became_active":"2024-08-28T19:18:32.487555+0000","last_became_peered":"2024-08-28T19:18:32.487555+0000","last_unstale":"2024-08-28T19:20:37.884032+0000","last_undegraded":"2024-08-28T19:20:37.884032+0000","last_fullsized":"2024-08-28T19:20:37.884032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:03:26.390481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71518,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1074,"num_read_kb":618,"num_write":262,"num_write_kb":152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":225,"ondisk_log_size":225,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":68,"seq":292057776171,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":500491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":62,"seq":266287972399,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1683456,"data_stored":1154763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":57,"seq":244813135925,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":503845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":51,"seq":219043332153,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36324,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708828,"statfs":{"total":95995035648,"available":95957839872,"internally_reserved":0,"allocated":1003520,"data_stored":500127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":45,"seq":193273528382,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1593344,"data_stored":1096625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":40,"seq":171798691908,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":495785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888135,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1605632,"data_stored":1088251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051662,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":499327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215186,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40476,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704676,"statfs":{"total":95995035648,"available":95953588224,"internally_reserved":0,"allocated":995328,"data_stored":494961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":561067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1052672,"data_stored":560847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705761,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":496709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-28T19:20:58.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-28T19:20:59.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.065+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/4077214834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2a8102760 msgr2=0x7fc2a8102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:59.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.065+0000 7fc2ad2f0640 1 --2- 172.21.15.16:0/4077214834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2a8102760 0x7fc2a8102b70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc298009900 tx=0x7fc29802a410 comp rx=0 tx=0).stop 2024-08-28T19:20:59.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.065+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/4077214834 shutdown_connections 2024-08-28T19:20:59.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.065+0000 7fc2ad2f0640 1 --2- 172.21.15.16:0/4077214834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a81089e0 0x7fc2a810f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.065+0000 7fc2ad2f0640 1 --2- 172.21.15.16:0/4077214834 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a8108090 0x7fc2a81084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.065+0000 7fc2ad2f0640 1 --2- 172.21.15.16:0/4077214834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2a8102760 0x7fc2a8102b70 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.065+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/4077214834 >> 172.21.15.16:0/4077214834 conn(0x7fc2a807a410 msgr2=0x7fc2a807a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.065+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/4077214834 shutdown_connections 2024-08-28T19:20:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.065+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/4077214834 wait complete. 2024-08-28T19:20:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.066+0000 7fc2ad2f0640 1 Processor -- start 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.066+0000 7fc2ad2f0640 1 -- start start 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.066+0000 7fc2ad2f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2a8102760 0x7fc2a80ff360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.066+0000 7fc2ad2f0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8108090 0x7fc2a80ff8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.066+0000 7fc2ad2f0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a81089e0 0x7fc2a80ffde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.066+0000 7fc2ad2f0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a81126b0 con 0x7fc2a8102760 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.066+0000 7fc2ad2f0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a8112530 con 0x7fc2a8108090 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.066+0000 7fc2ad2f0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a8112830 con 0x7fc2a81089e0 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2a8102760 0x7fc2a80ff360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2a8102760 0x7fc2a80ff360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37072/0 (socket says 172.21.15.16:37072) 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a6ffd640 1 -- 172.21.15.16:0/29424091 learned_addr learned my addr 172.21.15.16:0/29424091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc29e7fc640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8108090 0x7fc2a80ff8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a77fe640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a81089e0 0x7fc2a80ffde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a6ffd640 1 -- 172.21.15.16:0/29424091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a81089e0 msgr2=0x7fc2a80ffde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a6ffd640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a81089e0 0x7fc2a80ffde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a6ffd640 1 -- 172.21.15.16:0/29424091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8108090 msgr2=0x7fc2a80ff8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a6ffd640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8108090 0x7fc2a80ff8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a6ffd640 1 -- 172.21.15.16:0/29424091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a81a8470 con 0x7fc2a8102760 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a77fe640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a81089e0 0x7fc2a80ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a6ffd640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2a8102760 0x7fc2a80ff360 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc298005940 tx=0x7fc298004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a4ff9640 1 -- 172.21.15.16:0/29424091 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29803f030 con 0x7fc2a8102760 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a4ff9640 1 -- 172.21.15.16:0/29424091 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc298003aa0 con 0x7fc2a8102760 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/29424091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a81a8700 con 0x7fc2a8102760 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2a4ff9640 1 -- 172.21.15.16:0/29424091 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298035070 con 0x7fc2a8102760 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.067+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/29424091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a81a8c40 con 0x7fc2a8102760 2024-08-28T19:20:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.068+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/29424091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2a8102b70 con 0x7fc2a8102760 2024-08-28T19:20:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.070+0000 7fc2a4ff9640 1 -- 172.21.15.16:0/29424091 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fc298042030 con 0x7fc2a8102760 2024-08-28T19:20:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.071+0000 7fc2a4ff9640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2940b4c30 0x7fc2940b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.071+0000 7fc29e7fc640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2940b4c30 0x7fc2940b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.071+0000 7fc2a4ff9640 1 -- 172.21.15.16:0/29424091 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc298105e80 con 0x7fc2a8102760 2024-08-28T19:20:59.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.071+0000 7fc29e7fc640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2940b4c30 0x7fc2940b7140 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc2a806a080 tx=0x7fc290005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:59.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.074+0000 7fc2a4ff9640 1 -- 172.21.15.16:0/29424091 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fc2980ccac0 con 0x7fc2a8102760 2024-08-28T19:20:59.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.256+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/29424091 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc2a80635a0 con 0x7fc2940b4c30 2024-08-28T19:20:59.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.262+0000 7fc2a4ff9640 1 -- 172.21.15.16:0/29424091 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340519 (secure 0 0 0) 0x7fc2a80635a0 con 0x7fc2940b4c30 2024-08-28T19:20:59.267 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:59.269 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-08-28T19:20:59.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.265+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/29424091 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2940b4c30 msgr2=0x7fc2940b7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:59.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.265+0000 7fc2ad2f0640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2940b4c30 0x7fc2940b7140 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc2a806a080 tx=0x7fc290005f90 comp rx=0 tx=0).stop 2024-08-28T19:20:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.265+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/29424091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2a8102760 msgr2=0x7fc2a80ff360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.265+0000 7fc2ad2f0640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2a8102760 0x7fc2a80ff360 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc298005940 tx=0x7fc298004270 comp rx=0 tx=0).stop 2024-08-28T19:20:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.265+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/29424091 shutdown_connections 2024-08-28T19:20:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.265+0000 7fc2ad2f0640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a81089e0 0x7fc2a80ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.265+0000 7fc2ad2f0640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2940b4c30 0x7fc2940b7140 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.265+0000 7fc2ad2f0640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8108090 0x7fc2a80ff8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.265+0000 7fc2ad2f0640 1 --2- 172.21.15.16:0/29424091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2a8102760 0x7fc2a80ff360 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.265+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/29424091 >> 172.21.15.16:0/29424091 conn(0x7fc2a807a410 msgr2=0x7fc2a806ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.266+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/29424091 shutdown_connections 2024-08-28T19:20:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.266+0000 7fc2ad2f0640 1 -- 172.21.15.16:0/29424091 wait complete. 2024-08-28T19:20:59.280 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-08-28T19:20:57.495480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662120,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1155,"num_read_kb":687,"num_write":387,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":819,"ondisk_log_size":819,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":416992,"kb_used_data":13664,"kb_used_omap":18,"kb_used_meta":403117,"kb_avail":1124524832,"statfs":{"total":1151940427776,"available":1151513427968,"internally_reserved":0,"allocated":13991936,"data_stored":7952798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":412792122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":308,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":56,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003941"},"pg_stats":[{"pgid":"2.1f","version":"84'14","reported_seq":172,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.806536+0000","last_change":"2024-08-28T19:19:08.633326+0000","last_active":"2024-08-28T19:20:54.806536+0000","last_peered":"2024-08-28T19:20:54.806536+0000","last_clean":"2024-08-28T19:20:54.806536+0000","last_became_active":"2024-08-28T19:19:08.623876+0000","last_became_peered":"2024-08-28T19:19:08.623876+0000","last_unstale":"2024-08-28T19:20:54.806536+0000","last_undegraded":"2024-08-28T19:20:54.806536+0000","last_fullsized":"2024-08-28T19:20:54.806536+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:19:50.632337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866980+0000","last_change":"2024-08-28T19:18:32.492604+0000","last_active":"2024-08-28T19:20:37.866980+0000","last_peered":"2024-08-28T19:20:37.866980+0000","last_clean":"2024-08-28T19:20:37.866980+0000","last_became_active":"2024-08-28T19:18:32.489427+0000","last_became_peered":"2024-08-28T19:18:32.489427+0000","last_unstale":"2024-08-28T19:20:37.866980+0000","last_undegraded":"2024-08-28T19:20:37.866980+0000","last_fullsized":"2024-08-28T19:20:37.866980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:36:14.841315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010445+0000","last_change":"2024-08-28T19:18:39.528815+0000","last_active":"2024-08-28T19:20:09.010445+0000","last_peered":"2024-08-28T19:20:09.010445+0000","last_clean":"2024-08-28T19:20:09.010445+0000","last_became_active":"2024-08-28T19:18:39.527781+0000","last_became_peered":"2024-08-28T19:18:39.527781+0000","last_unstale":"2024-08-28T19:20:09.010445+0000","last_undegraded":"2024-08-28T19:20:09.010445+0000","last_fullsized":"2024-08-28T19:20:09.010445+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:39.831426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525559+0000","last_change":"2024-08-28T19:18:31.082641+0000","last_active":"2024-08-28T19:20:09.525559+0000","last_peered":"2024-08-28T19:20:09.525559+0000","last_clean":"2024-08-28T19:20:09.525559+0000","last_became_active":"2024-08-28T19:18:31.082260+0000","last_became_peered":"2024-08-28T19:18:31.082260+0000","last_unstale":"2024-08-28T19:20:09.525559+0000","last_undegraded":"2024-08-28T19:20:09.525559+0000","last_fullsized":"2024-08-28T19:20:09.525559+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:09:55.836463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886294+0000","last_change":"2024-08-28T19:18:32.493498+0000","last_active":"2024-08-28T19:20:37.886294+0000","last_peered":"2024-08-28T19:20:37.886294+0000","last_clean":"2024-08-28T19:20:37.886294+0000","last_became_active":"2024-08-28T19:18:32.490969+0000","last_became_peered":"2024-08-28T19:18:32.490969+0000","last_unstale":"2024-08-28T19:20:37.886294+0000","last_undegraded":"2024-08-28T19:20:37.886294+0000","last_fullsized":"2024-08-28T19:20:37.886294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:34:29.537622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614531+0000","last_change":"2024-08-28T19:18:39.520922+0000","last_active":"2024-08-28T19:20:09.614531+0000","last_peered":"2024-08-28T19:20:09.614531+0000","last_clean":"2024-08-28T19:20:09.614531+0000","last_became_active":"2024-08-28T19:18:39.520362+0000","last_became_peered":"2024-08-28T19:18:39.520362+0000","last_unstale":"2024-08-28T19:20:09.614531+0000","last_undegraded":"2024-08-28T19:20:09.614531+0000","last_fullsized":"2024-08-28T19:20:09.614531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:40.658820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":52,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110351+0000","last_change":"2024-08-28T19:18:30.471870+0000","last_active":"2024-08-28T19:20:09.110351+0000","last_peered":"2024-08-28T19:20:09.110351+0000","last_clean":"2024-08-28T19:20:09.110351+0000","last_became_active":"2024-08-28T19:18:30.471438+0000","last_became_peered":"2024-08-28T19:18:30.471438+0000","last_unstale":"2024-08-28T19:20:09.110351+0000","last_undegraded":"2024-08-28T19:20:09.110351+0000","last_fullsized":"2024-08-28T19:20:09.110351+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:44:34.137395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"84'9","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.881074+0000","last_change":"2024-08-28T19:18:32.486485+0000","last_active":"2024-08-28T19:20:37.881074+0000","last_peered":"2024-08-28T19:20:37.881074+0000","last_clean":"2024-08-28T19:20:37.881074+0000","last_became_active":"2024-08-28T19:18:32.485888+0000","last_became_peered":"2024-08-28T19:18:32.485888+0000","last_unstale":"2024-08-28T19:20:37.881074+0000","last_undegraded":"2024-08-28T19:20:37.881074+0000","last_fullsized":"2024-08-28T19:20:37.881074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:45:48.757318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615105+0000","last_change":"2024-08-28T19:19:08.633317+0000","last_active":"2024-08-28T19:20:09.615105+0000","last_peered":"2024-08-28T19:20:09.615105+0000","last_clean":"2024-08-28T19:20:09.615105+0000","last_became_active":"2024-08-28T19:19:08.632689+0000","last_became_peered":"2024-08-28T19:19:08.632689+0000","last_unstale":"2024-08-28T19:20:09.615105+0000","last_undegraded":"2024-08-28T19:20:09.615105+0000","last_fullsized":"2024-08-28T19:20:09.615105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:35:08.740645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110282+0000","last_change":"2024-08-28T19:18:30.473982+0000","last_active":"2024-08-28T19:20:09.110282+0000","last_peered":"2024-08-28T19:20:09.110282+0000","last_clean":"2024-08-28T19:20:09.110282+0000","last_became_active":"2024-08-28T19:18:30.473552+0000","last_became_peered":"2024-08-28T19:18:30.473552+0000","last_unstale":"2024-08-28T19:20:09.110282+0000","last_undegraded":"2024-08-28T19:20:09.110282+0000","last_fullsized":"2024-08-28T19:20:09.110282+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:22:29.435517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"84'15","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884248+0000","last_change":"2024-08-28T19:18:32.487881+0000","last_active":"2024-08-28T19:20:37.884248+0000","last_peered":"2024-08-28T19:20:37.884248+0000","last_clean":"2024-08-28T19:20:37.884248+0000","last_became_active":"2024-08-28T19:18:32.487039+0000","last_became_peered":"2024-08-28T19:18:32.487039+0000","last_unstale":"2024-08-28T19:20:37.884248+0000","last_undegraded":"2024-08-28T19:20:37.884248+0000","last_fullsized":"2024-08-28T19:20:37.884248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:23:35.757811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614594+0000","last_change":"2024-08-28T19:18:39.518654+0000","last_active":"2024-08-28T19:20:09.614594+0000","last_peered":"2024-08-28T19:20:09.614594+0000","last_clean":"2024-08-28T19:20:09.614594+0000","last_became_active":"2024-08-28T19:18:39.518087+0000","last_became_peered":"2024-08-28T19:18:39.518087+0000","last_unstale":"2024-08-28T19:20:09.614594+0000","last_undegraded":"2024-08-28T19:20:09.614594+0000","last_fullsized":"2024-08-28T19:20:09.614594+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:08.284865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.887381+0000","last_change":"2024-08-28T19:18:32.491928+0000","last_active":"2024-08-28T19:20:37.887381+0000","last_peered":"2024-08-28T19:20:37.887381+0000","last_clean":"2024-08-28T19:20:37.887381+0000","last_became_active":"2024-08-28T19:18:32.491538+0000","last_became_peered":"2024-08-28T19:18:32.491538+0000","last_unstale":"2024-08-28T19:20:37.887381+0000","last_undegraded":"2024-08-28T19:20:37.887381+0000","last_fullsized":"2024-08-28T19:20:37.887381+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:39:06.077830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":24,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663446+0000","last_change":"2024-08-28T19:19:08.634752+0000","last_active":"2024-08-28T19:20:09.663446+0000","last_peered":"2024-08-28T19:20:09.663446+0000","last_clean":"2024-08-28T19:20:09.663446+0000","last_became_active":"2024-08-28T19:19:08.620784+0000","last_became_peered":"2024-08-28T19:19:08.620784+0000","last_unstale":"2024-08-28T19:20:09.663446+0000","last_undegraded":"2024-08-28T19:20:09.663446+0000","last_fullsized":"2024-08-28T19:20:09.663446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:51.640445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614895+0000","last_change":"2024-08-28T19:18:39.681413+0000","last_active":"2024-08-28T19:20:09.614895+0000","last_peered":"2024-08-28T19:20:09.614895+0000","last_clean":"2024-08-28T19:20:09.614895+0000","last_became_active":"2024-08-28T19:18:39.680793+0000","last_became_peered":"2024-08-28T19:18:39.680793+0000","last_unstale":"2024-08-28T19:20:09.614895+0000","last_undegraded":"2024-08-28T19:20:09.614895+0000","last_fullsized":"2024-08-28T19:20:09.614895+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:45:43.381019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870888+0000","last_change":"2024-08-28T19:18:32.486739+0000","last_active":"2024-08-28T19:20:37.870888+0000","last_peered":"2024-08-28T19:20:37.870888+0000","last_clean":"2024-08-28T19:20:37.870888+0000","last_became_active":"2024-08-28T19:18:32.486347+0000","last_became_peered":"2024-08-28T19:18:32.486347+0000","last_unstale":"2024-08-28T19:20:37.870888+0000","last_undegraded":"2024-08-28T19:20:37.870888+0000","last_fullsized":"2024-08-28T19:20:37.870888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:27:36.448569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010653+0000","last_change":"2024-08-28T19:18:30.467690+0000","last_active":"2024-08-28T19:20:09.010653+0000","last_peered":"2024-08-28T19:20:09.010653+0000","last_clean":"2024-08-28T19:20:09.010653+0000","last_became_active":"2024-08-28T19:18:30.467249+0000","last_became_peered":"2024-08-28T19:18:30.467249+0000","last_unstale":"2024-08-28T19:20:09.010653+0000","last_undegraded":"2024-08-28T19:20:09.010653+0000","last_fullsized":"2024-08-28T19:20:09.010653+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:37:28.876666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615167+0000","last_change":"2024-08-28T19:18:39.523889+0000","last_active":"2024-08-28T19:20:09.615167+0000","last_peered":"2024-08-28T19:20:09.615167+0000","last_clean":"2024-08-28T19:20:09.615167+0000","last_became_active":"2024-08-28T19:18:39.523384+0000","last_became_peered":"2024-08-28T19:18:39.523384+0000","last_unstale":"2024-08-28T19:20:09.615167+0000","last_undegraded":"2024-08-28T19:20:09.615167+0000","last_fullsized":"2024-08-28T19:20:09.615167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:57:32.277680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873180+0000","last_change":"2024-08-28T19:18:32.482257+0000","last_active":"2024-08-28T19:20:37.873180+0000","last_peered":"2024-08-28T19:20:37.873180+0000","last_clean":"2024-08-28T19:20:37.873180+0000","last_became_active":"2024-08-28T19:18:32.481884+0000","last_became_peered":"2024-08-28T19:18:32.481884+0000","last_unstale":"2024-08-28T19:20:37.873180+0000","last_undegraded":"2024-08-28T19:20:37.873180+0000","last_fullsized":"2024-08-28T19:20:37.873180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:56:28.754369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":47,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525435+0000","last_change":"2024-08-28T19:18:30.467307+0000","last_active":"2024-08-28T19:20:09.525435+0000","last_peered":"2024-08-28T19:20:09.525435+0000","last_clean":"2024-08-28T19:20:09.525435+0000","last_became_active":"2024-08-28T19:18:30.466942+0000","last_became_peered":"2024-08-28T19:18:30.466942+0000","last_unstale":"2024-08-28T19:20:09.525435+0000","last_undegraded":"2024-08-28T19:20:09.525435+0000","last_fullsized":"2024-08-28T19:20:09.525435+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:05:23.682110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007238+0000","last_change":"2024-08-28T19:18:39.531421+0000","last_active":"2024-08-28T19:20:09.007238+0000","last_peered":"2024-08-28T19:20:09.007238+0000","last_clean":"2024-08-28T19:20:09.007238+0000","last_became_active":"2024-08-28T19:18:39.531008+0000","last_became_peered":"2024-08-28T19:18:39.531008+0000","last_unstale":"2024-08-28T19:20:09.007238+0000","last_undegraded":"2024-08-28T19:20:09.007238+0000","last_fullsized":"2024-08-28T19:20:09.007238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:46:49.580601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868264+0000","last_change":"2024-08-28T19:18:32.485204+0000","last_active":"2024-08-28T19:20:37.868264+0000","last_peered":"2024-08-28T19:20:37.868264+0000","last_clean":"2024-08-28T19:20:37.868264+0000","last_became_active":"2024-08-28T19:18:32.484830+0000","last_became_peered":"2024-08-28T19:18:32.484830+0000","last_unstale":"2024-08-28T19:20:37.868264+0000","last_undegraded":"2024-08-28T19:20:37.868264+0000","last_fullsized":"2024-08-28T19:20:37.868264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:55:41.358718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007905+0000","last_change":"2024-08-28T19:18:31.082361+0000","last_active":"2024-08-28T19:20:09.007905+0000","last_peered":"2024-08-28T19:20:09.007905+0000","last_clean":"2024-08-28T19:20:09.007905+0000","last_became_active":"2024-08-28T19:18:31.081930+0000","last_became_peered":"2024-08-28T19:18:31.081930+0000","last_unstale":"2024-08-28T19:20:09.007905+0000","last_undegraded":"2024-08-28T19:20:09.007905+0000","last_fullsized":"2024-08-28T19:20:09.007905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:41:05.543088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615411+0000","last_change":"2024-08-28T19:18:39.682341+0000","last_active":"2024-08-28T19:20:09.615411+0000","last_peered":"2024-08-28T19:20:09.615411+0000","last_clean":"2024-08-28T19:20:09.615411+0000","last_became_active":"2024-08-28T19:18:39.681751+0000","last_became_peered":"2024-08-28T19:18:39.681751+0000","last_unstale":"2024-08-28T19:20:09.615411+0000","last_undegraded":"2024-08-28T19:20:09.615411+0000","last_fullsized":"2024-08-28T19:20:09.615411+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:51:24.371270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880906+0000","last_change":"2024-08-28T19:18:32.491069+0000","last_active":"2024-08-28T19:20:37.880906+0000","last_peered":"2024-08-28T19:20:37.880906+0000","last_clean":"2024-08-28T19:20:37.880906+0000","last_became_active":"2024-08-28T19:18:32.490688+0000","last_became_peered":"2024-08-28T19:18:32.490688+0000","last_unstale":"2024-08-28T19:20:37.880906+0000","last_undegraded":"2024-08-28T19:20:37.880906+0000","last_fullsized":"2024-08-28T19:20:37.880906+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:24:31.560519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615273+0000","last_change":"2024-08-28T19:18:30.739812+0000","last_active":"2024-08-28T19:20:09.615273+0000","last_peered":"2024-08-28T19:20:09.615273+0000","last_clean":"2024-08-28T19:20:09.615273+0000","last_became_active":"2024-08-28T19:18:30.739414+0000","last_became_peered":"2024-08-28T19:18:30.739414+0000","last_unstale":"2024-08-28T19:20:09.615273+0000","last_undegraded":"2024-08-28T19:20:09.615273+0000","last_fullsized":"2024-08-28T19:20:09.615273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:56:56.594993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010962+0000","last_change":"2024-08-28T19:18:39.528281+0000","last_active":"2024-08-28T19:20:09.010962+0000","last_peered":"2024-08-28T19:20:09.010962+0000","last_clean":"2024-08-28T19:20:09.010962+0000","last_became_active":"2024-08-28T19:18:39.527142+0000","last_became_peered":"2024-08-28T19:18:39.527142+0000","last_unstale":"2024-08-28T19:20:09.010962+0000","last_undegraded":"2024-08-28T19:20:09.010962+0000","last_fullsized":"2024-08-28T19:20:09.010962+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:44:19.123662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872571+0000","last_change":"2024-08-28T19:18:32.490430+0000","last_active":"2024-08-28T19:20:37.872571+0000","last_peered":"2024-08-28T19:20:37.872571+0000","last_clean":"2024-08-28T19:20:37.872571+0000","last_became_active":"2024-08-28T19:18:32.490020+0000","last_became_peered":"2024-08-28T19:18:32.490020+0000","last_unstale":"2024-08-28T19:20:37.872571+0000","last_undegraded":"2024-08-28T19:20:37.872571+0000","last_fullsized":"2024-08-28T19:20:37.872571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:52:31.153012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'17","reported_seq":57,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:41.222712+0000","last_change":"2024-08-28T19:18:30.467060+0000","last_active":"2024-08-28T19:20:41.222712+0000","last_peered":"2024-08-28T19:20:41.222712+0000","last_clean":"2024-08-28T19:20:41.222712+0000","last_became_active":"2024-08-28T19:18:30.466689+0000","last_became_peered":"2024-08-28T19:18:30.466689+0000","last_unstale":"2024-08-28T19:20:41.222712+0000","last_undegraded":"2024-08-28T19:20:41.222712+0000","last_fullsized":"2024-08-28T19:20:41.222712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:18:36.508131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.524970+0000","last_change":"2024-08-28T19:18:39.682874+0000","last_active":"2024-08-28T19:20:09.524970+0000","last_peered":"2024-08-28T19:20:09.524970+0000","last_clean":"2024-08-28T19:20:09.524970+0000","last_became_active":"2024-08-28T19:18:39.681500+0000","last_became_peered":"2024-08-28T19:18:39.681500+0000","last_unstale":"2024-08-28T19:20:09.524970+0000","last_undegraded":"2024-08-28T19:20:09.524970+0000","last_fullsized":"2024-08-28T19:20:09.524970+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:45:41.815506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.856367+0000","last_change":"2024-08-28T19:18:32.482047+0000","last_active":"2024-08-28T19:20:37.856367+0000","last_peered":"2024-08-28T19:20:37.856367+0000","last_clean":"2024-08-28T19:20:37.856367+0000","last_became_active":"2024-08-28T19:18:32.481640+0000","last_became_peered":"2024-08-28T19:18:32.481640+0000","last_unstale":"2024-08-28T19:20:37.856367+0000","last_undegraded":"2024-08-28T19:20:37.856367+0000","last_fullsized":"2024-08-28T19:20:37.856367+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:34.717764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110454+0000","last_change":"2024-08-28T19:18:31.083524+0000","last_active":"2024-08-28T19:20:09.110454+0000","last_peered":"2024-08-28T19:20:09.110454+0000","last_clean":"2024-08-28T19:20:09.110454+0000","last_became_active":"2024-08-28T19:18:31.083106+0000","last_became_peered":"2024-08-28T19:18:31.083106+0000","last_unstale":"2024-08-28T19:20:09.110454+0000","last_undegraded":"2024-08-28T19:20:09.110454+0000","last_fullsized":"2024-08-28T19:20:09.110454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:16.180754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663607+0000","last_change":"2024-08-28T19:18:39.531884+0000","last_active":"2024-08-28T19:20:09.663607+0000","last_peered":"2024-08-28T19:20:09.663607+0000","last_clean":"2024-08-28T19:20:09.663607+0000","last_became_active":"2024-08-28T19:18:39.531255+0000","last_became_peered":"2024-08-28T19:18:39.531255+0000","last_unstale":"2024-08-28T19:20:09.663607+0000","last_undegraded":"2024-08-28T19:20:09.663607+0000","last_fullsized":"2024-08-28T19:20:09.663607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:36:55.496594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.862241+0000","last_change":"2024-08-28T19:18:32.485180+0000","last_active":"2024-08-28T19:20:37.862241+0000","last_peered":"2024-08-28T19:20:37.862241+0000","last_clean":"2024-08-28T19:20:37.862241+0000","last_became_active":"2024-08-28T19:18:32.484804+0000","last_became_peered":"2024-08-28T19:18:32.484804+0000","last_unstale":"2024-08-28T19:20:37.862241+0000","last_undegraded":"2024-08-28T19:20:37.862241+0000","last_fullsized":"2024-08-28T19:20:37.862241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:12:40.958251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007361+0000","last_change":"2024-08-28T19:18:31.082226+0000","last_active":"2024-08-28T19:20:09.007361+0000","last_peered":"2024-08-28T19:20:09.007361+0000","last_clean":"2024-08-28T19:20:09.007361+0000","last_became_active":"2024-08-28T19:18:31.081833+0000","last_became_peered":"2024-08-28T19:18:31.081833+0000","last_unstale":"2024-08-28T19:20:09.007361+0000","last_undegraded":"2024-08-28T19:20:09.007361+0000","last_fullsized":"2024-08-28T19:20:09.007361+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:33:56.298093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109762+0000","last_change":"2024-08-28T19:18:39.681288+0000","last_active":"2024-08-28T19:20:09.109762+0000","last_peered":"2024-08-28T19:20:09.109762+0000","last_clean":"2024-08-28T19:20:09.109762+0000","last_became_active":"2024-08-28T19:18:39.680719+0000","last_became_peered":"2024-08-28T19:18:39.680719+0000","last_unstale":"2024-08-28T19:20:09.109762+0000","last_undegraded":"2024-08-28T19:20:09.109762+0000","last_fullsized":"2024-08-28T19:20:09.109762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:53:35.554181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880497+0000","last_change":"2024-08-28T19:18:32.487105+0000","last_active":"2024-08-28T19:20:37.880497+0000","last_peered":"2024-08-28T19:20:37.880497+0000","last_clean":"2024-08-28T19:20:37.880497+0000","last_became_active":"2024-08-28T19:18:32.486643+0000","last_became_peered":"2024-08-28T19:18:32.486643+0000","last_unstale":"2024-08-28T19:20:37.880497+0000","last_undegraded":"2024-08-28T19:20:37.880497+0000","last_fullsized":"2024-08-28T19:20:37.880497+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:18:43.274493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"84'7","reported_seq":153,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:53.805823+0000","last_change":"2024-08-28T19:18:30.471447+0000","last_active":"2024-08-28T19:20:53.805823+0000","last_peered":"2024-08-28T19:20:53.805823+0000","last_clean":"2024-08-28T19:20:53.805823+0000","last_became_active":"2024-08-28T19:18:30.470798+0000","last_became_peered":"2024-08-28T19:18:30.470798+0000","last_unstale":"2024-08-28T19:20:53.805823+0000","last_undegraded":"2024-08-28T19:20:53.805823+0000","last_fullsized":"2024-08-28T19:20:53.805823+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:50:13.948439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162958+0000","last_change":"2024-08-28T19:18:39.524190+0000","last_active":"2024-08-28T19:20:09.162958+0000","last_peered":"2024-08-28T19:20:09.162958+0000","last_clean":"2024-08-28T19:20:09.162958+0000","last_became_active":"2024-08-28T19:18:39.523771+0000","last_became_peered":"2024-08-28T19:18:39.523771+0000","last_unstale":"2024-08-28T19:20:09.162958+0000","last_undegraded":"2024-08-28T19:20:09.162958+0000","last_fullsized":"2024-08-28T19:20:09.162958+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:43:43.984625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.874995+0000","last_change":"2024-08-28T19:18:32.487212+0000","last_active":"2024-08-28T19:20:37.874995+0000","last_peered":"2024-08-28T19:20:37.874995+0000","last_clean":"2024-08-28T19:20:37.874995+0000","last_became_active":"2024-08-28T19:18:32.485432+0000","last_became_peered":"2024-08-28T19:18:32.485432+0000","last_unstale":"2024-08-28T19:20:37.874995+0000","last_undegraded":"2024-08-28T19:20:37.874995+0000","last_fullsized":"2024-08-28T19:20:37.874995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:52:11.672193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":149,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:53.805226+0000","last_change":"2024-08-28T19:18:31.082210+0000","last_active":"2024-08-28T19:20:53.805226+0000","last_peered":"2024-08-28T19:20:53.805226+0000","last_clean":"2024-08-28T19:20:53.805226+0000","last_became_active":"2024-08-28T19:18:31.081814+0000","last_became_peered":"2024-08-28T19:18:31.081814+0000","last_unstale":"2024-08-28T19:20:53.805226+0000","last_undegraded":"2024-08-28T19:20:53.805226+0000","last_fullsized":"2024-08-28T19:20:53.805226+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:33:32.281431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163227+0000","last_change":"2024-08-28T19:18:39.681858+0000","last_active":"2024-08-28T19:20:09.163227+0000","last_peered":"2024-08-28T19:20:09.163227+0000","last_clean":"2024-08-28T19:20:09.163227+0000","last_became_active":"2024-08-28T19:18:39.681382+0000","last_became_peered":"2024-08-28T19:18:39.681382+0000","last_unstale":"2024-08-28T19:20:09.163227+0000","last_undegraded":"2024-08-28T19:20:09.163227+0000","last_fullsized":"2024-08-28T19:20:09.163227+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:23:22.021022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.857639+0000","last_change":"2024-08-28T19:18:32.486802+0000","last_active":"2024-08-28T19:20:37.857639+0000","last_peered":"2024-08-28T19:20:37.857639+0000","last_clean":"2024-08-28T19:20:37.857639+0000","last_became_active":"2024-08-28T19:18:32.486479+0000","last_became_peered":"2024-08-28T19:18:32.486479+0000","last_unstale":"2024-08-28T19:20:37.857639+0000","last_undegraded":"2024-08-28T19:20:37.857639+0000","last_fullsized":"2024-08-28T19:20:37.857639+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:17:17.654757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529084+0000","last_change":"2024-08-28T19:18:30.470447+0000","last_active":"2024-08-28T19:20:09.529084+0000","last_peered":"2024-08-28T19:20:09.529084+0000","last_clean":"2024-08-28T19:20:09.529084+0000","last_became_active":"2024-08-28T19:18:30.469862+0000","last_became_peered":"2024-08-28T19:18:30.469862+0000","last_unstale":"2024-08-28T19:20:09.529084+0000","last_undegraded":"2024-08-28T19:20:09.529084+0000","last_fullsized":"2024-08-28T19:20:09.529084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:27:46.660492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009270+0000","last_change":"2024-08-28T19:19:08.624635+0000","last_active":"2024-08-28T19:20:09.009270+0000","last_peered":"2024-08-28T19:20:09.009270+0000","last_clean":"2024-08-28T19:20:09.009270+0000","last_became_active":"2024-08-28T19:19:08.622845+0000","last_became_peered":"2024-08-28T19:19:08.622845+0000","last_unstale":"2024-08-28T19:20:09.009270+0000","last_undegraded":"2024-08-28T19:20:09.009270+0000","last_fullsized":"2024-08-28T19:20:09.009270+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:56:40.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865852+0000","last_change":"2024-08-28T19:18:32.487947+0000","last_active":"2024-08-28T19:20:37.865852+0000","last_peered":"2024-08-28T19:20:37.865852+0000","last_clean":"2024-08-28T19:20:37.865852+0000","last_became_active":"2024-08-28T19:18:32.487592+0000","last_became_peered":"2024-08-28T19:18:32.487592+0000","last_unstale":"2024-08-28T19:20:37.865852+0000","last_undegraded":"2024-08-28T19:20:37.865852+0000","last_fullsized":"2024-08-28T19:20:37.865852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:30:29.550644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615576+0000","last_change":"2024-08-28T19:18:30.465044+0000","last_active":"2024-08-28T19:20:09.615576+0000","last_peered":"2024-08-28T19:20:09.615576+0000","last_clean":"2024-08-28T19:20:09.615576+0000","last_became_active":"2024-08-28T19:18:30.464580+0000","last_became_peered":"2024-08-28T19:18:30.464580+0000","last_unstale":"2024-08-28T19:20:09.615576+0000","last_undegraded":"2024-08-28T19:20:09.615576+0000","last_fullsized":"2024-08-28T19:20:09.615576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:21:17.223311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":114,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.528964+0000","last_change":"2024-08-28T19:17:17.063316+0000","last_active":"2024-08-28T19:20:09.528964+0000","last_peered":"2024-08-28T19:20:09.528964+0000","last_clean":"2024-08-28T19:20:09.528964+0000","last_became_active":"2024-08-28T19:17:17.057091+0000","last_became_peered":"2024-08-28T19:17:17.057091+0000","last_unstale":"2024-08-28T19:20:09.528964+0000","last_undegraded":"2024-08-28T19:20:09.528964+0000","last_fullsized":"2024-08-28T19:20:09.528964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_clean_scrub_stamp":"2024-08-28T19:13:56.815467+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:35:51.499703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"82'1","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529202+0000","last_change":"2024-08-28T19:19:08.621402+0000","last_active":"2024-08-28T19:20:09.529202+0000","last_peered":"2024-08-28T19:20:09.529202+0000","last_clean":"2024-08-28T19:20:09.529202+0000","last_became_active":"2024-08-28T19:19:08.620779+0000","last_became_peered":"2024-08-28T19:19:08.620779+0000","last_unstale":"2024-08-28T19:20:09.529202+0000","last_undegraded":"2024-08-28T19:20:09.529202+0000","last_fullsized":"2024-08-28T19:20:09.529202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:39.391414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886997+0000","last_change":"2024-08-28T19:18:32.490066+0000","last_active":"2024-08-28T19:20:37.886997+0000","last_peered":"2024-08-28T19:20:37.886997+0000","last_clean":"2024-08-28T19:20:37.886997+0000","last_became_active":"2024-08-28T19:18:32.489560+0000","last_became_peered":"2024-08-28T19:18:32.489560+0000","last_unstale":"2024-08-28T19:20:37.886997+0000","last_undegraded":"2024-08-28T19:20:37.886997+0000","last_fullsized":"2024-08-28T19:20:37.886997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:22:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'5","reported_seq":151,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.851167+0000","last_change":"2024-08-28T19:18:31.083078+0000","last_active":"2024-08-28T19:20:54.851167+0000","last_peered":"2024-08-28T19:20:54.851167+0000","last_clean":"2024-08-28T19:20:54.851167+0000","last_became_active":"2024-08-28T19:18:31.082575+0000","last_became_peered":"2024-08-28T19:18:31.082575+0000","last_unstale":"2024-08-28T19:20:54.851167+0000","last_undegraded":"2024-08-28T19:20:54.851167+0000","last_fullsized":"2024-08-28T19:20:54.851167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007862+0000","last_change":"2024-08-28T19:18:39.519814+0000","last_active":"2024-08-28T19:20:09.007862+0000","last_peered":"2024-08-28T19:20:09.007862+0000","last_clean":"2024-08-28T19:20:09.007862+0000","last_became_active":"2024-08-28T19:18:39.519258+0000","last_became_peered":"2024-08-28T19:18:39.519258+0000","last_unstale":"2024-08-28T19:20:09.007862+0000","last_undegraded":"2024-08-28T19:20:09.007862+0000","last_fullsized":"2024-08-28T19:20:09.007862+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:03:04.051376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871177+0000","last_change":"2024-08-28T19:18:32.495426+0000","last_active":"2024-08-28T19:20:37.871177+0000","last_peered":"2024-08-28T19:20:37.871177+0000","last_clean":"2024-08-28T19:20:37.871177+0000","last_became_active":"2024-08-28T19:18:32.495075+0000","last_became_peered":"2024-08-28T19:18:32.495075+0000","last_unstale":"2024-08-28T19:20:37.871177+0000","last_undegraded":"2024-08-28T19:20:37.871177+0000","last_fullsized":"2024-08-28T19:20:37.871177+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:40:22.456666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009115+0000","last_change":"2024-08-28T19:18:30.472708+0000","last_active":"2024-08-28T19:20:09.009115+0000","last_peered":"2024-08-28T19:20:09.009115+0000","last_clean":"2024-08-28T19:20:09.009115+0000","last_became_active":"2024-08-28T19:18:30.472320+0000","last_became_peered":"2024-08-28T19:18:30.472320+0000","last_unstale":"2024-08-28T19:20:09.009115+0000","last_undegraded":"2024-08-28T19:20:09.009115+0000","last_fullsized":"2024-08-28T19:20:09.009115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:57:12.055626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010437+0000","last_change":"2024-08-28T19:18:39.519398+0000","last_active":"2024-08-28T19:20:09.010437+0000","last_peered":"2024-08-28T19:20:09.010437+0000","last_clean":"2024-08-28T19:20:09.010437+0000","last_became_active":"2024-08-28T19:18:39.518997+0000","last_became_peered":"2024-08-28T19:18:39.518997+0000","last_unstale":"2024-08-28T19:20:09.010437+0000","last_undegraded":"2024-08-28T19:20:09.010437+0000","last_fullsized":"2024-08-28T19:20:09.010437+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:33.431608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.879604+0000","last_change":"2024-08-28T19:18:32.489439+0000","last_active":"2024-08-28T19:20:37.879604+0000","last_peered":"2024-08-28T19:20:37.879604+0000","last_clean":"2024-08-28T19:20:37.879604+0000","last_became_active":"2024-08-28T19:18:32.489054+0000","last_became_peered":"2024-08-28T19:18:32.489054+0000","last_unstale":"2024-08-28T19:20:37.879604+0000","last_undegraded":"2024-08-28T19:20:37.879604+0000","last_fullsized":"2024-08-28T19:20:37.879604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:31:32.647384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163358+0000","last_change":"2024-08-28T19:18:30.464993+0000","last_active":"2024-08-28T19:20:09.163358+0000","last_peered":"2024-08-28T19:20:09.163358+0000","last_clean":"2024-08-28T19:20:09.163358+0000","last_became_active":"2024-08-28T19:18:30.463951+0000","last_became_peered":"2024-08-28T19:18:30.463951+0000","last_unstale":"2024-08-28T19:20:09.163358+0000","last_undegraded":"2024-08-28T19:20:09.163358+0000","last_fullsized":"2024-08-28T19:20:09.163358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:06.780058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008234+0000","last_change":"2024-08-28T19:18:39.533061+0000","last_active":"2024-08-28T19:20:09.008234+0000","last_peered":"2024-08-28T19:20:09.008234+0000","last_clean":"2024-08-28T19:20:09.008234+0000","last_became_active":"2024-08-28T19:18:39.532607+0000","last_became_peered":"2024-08-28T19:18:39.532607+0000","last_unstale":"2024-08-28T19:20:09.008234+0000","last_undegraded":"2024-08-28T19:20:09.008234+0000","last_fullsized":"2024-08-28T19:20:09.008234+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:00:57.997665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875549+0000","last_change":"2024-08-28T19:18:32.487937+0000","last_active":"2024-08-28T19:20:37.875549+0000","last_peered":"2024-08-28T19:20:37.875549+0000","last_clean":"2024-08-28T19:20:37.875549+0000","last_became_active":"2024-08-28T19:18:32.487417+0000","last_became_peered":"2024-08-28T19:18:32.487417+0000","last_unstale":"2024-08-28T19:20:37.875549+0000","last_undegraded":"2024-08-28T19:20:37.875549+0000","last_fullsized":"2024-08-28T19:20:37.875549+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:42.497364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110517+0000","last_change":"2024-08-28T19:18:30.470742+0000","last_active":"2024-08-28T19:20:09.110517+0000","last_peered":"2024-08-28T19:20:09.110517+0000","last_clean":"2024-08-28T19:20:09.110517+0000","last_became_active":"2024-08-28T19:18:30.470135+0000","last_became_peered":"2024-08-28T19:18:30.470135+0000","last_unstale":"2024-08-28T19:20:09.110517+0000","last_undegraded":"2024-08-28T19:20:09.110517+0000","last_fullsized":"2024-08-28T19:20:09.110517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:49:39.581119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008429+0000","last_change":"2024-08-28T19:18:39.524052+0000","last_active":"2024-08-28T19:20:09.008429+0000","last_peered":"2024-08-28T19:20:09.008429+0000","last_clean":"2024-08-28T19:20:09.008429+0000","last_became_active":"2024-08-28T19:18:39.523601+0000","last_became_peered":"2024-08-28T19:18:39.523601+0000","last_unstale":"2024-08-28T19:20:09.008429+0000","last_undegraded":"2024-08-28T19:20:09.008429+0000","last_fullsized":"2024-08-28T19:20:09.008429+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:54:53.302424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883838+0000","last_change":"2024-08-28T19:18:32.491151+0000","last_active":"2024-08-28T19:20:37.883838+0000","last_peered":"2024-08-28T19:20:37.883838+0000","last_clean":"2024-08-28T19:20:37.883838+0000","last_became_active":"2024-08-28T19:18:32.490798+0000","last_became_peered":"2024-08-28T19:18:32.490798+0000","last_unstale":"2024-08-28T19:20:37.883838+0000","last_undegraded":"2024-08-28T19:20:37.883838+0000","last_fullsized":"2024-08-28T19:20:37.883838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:41:00.771019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109743+0000","last_change":"2024-08-28T19:18:31.084029+0000","last_active":"2024-08-28T19:20:09.109743+0000","last_peered":"2024-08-28T19:20:09.109743+0000","last_clean":"2024-08-28T19:20:09.109743+0000","last_became_active":"2024-08-28T19:18:31.083507+0000","last_became_peered":"2024-08-28T19:18:31.083507+0000","last_unstale":"2024-08-28T19:20:09.109743+0000","last_undegraded":"2024-08-28T19:20:09.109743+0000","last_fullsized":"2024-08-28T19:20:09.109743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:15:36.451702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163253+0000","last_change":"2024-08-28T19:18:39.527249+0000","last_active":"2024-08-28T19:20:09.163253+0000","last_peered":"2024-08-28T19:20:09.163253+0000","last_clean":"2024-08-28T19:20:09.163253+0000","last_became_active":"2024-08-28T19:18:39.526834+0000","last_became_peered":"2024-08-28T19:18:39.526834+0000","last_unstale":"2024-08-28T19:20:09.163253+0000","last_undegraded":"2024-08-28T19:20:09.163253+0000","last_fullsized":"2024-08-28T19:20:09.163253+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:48:02.294738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.876604+0000","last_change":"2024-08-28T19:18:32.497337+0000","last_active":"2024-08-28T19:20:37.876604+0000","last_peered":"2024-08-28T19:20:37.876604+0000","last_clean":"2024-08-28T19:20:37.876604+0000","last_became_active":"2024-08-28T19:18:32.496929+0000","last_became_peered":"2024-08-28T19:18:32.496929+0000","last_unstale":"2024-08-28T19:20:37.876604+0000","last_undegraded":"2024-08-28T19:20:37.876604+0000","last_fullsized":"2024-08-28T19:20:37.876604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:14.748765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":149,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:53.806036+0000","last_change":"2024-08-28T19:18:30.471353+0000","last_active":"2024-08-28T19:20:53.806036+0000","last_peered":"2024-08-28T19:20:53.806036+0000","last_clean":"2024-08-28T19:20:53.806036+0000","last_became_active":"2024-08-28T19:18:30.470283+0000","last_became_peered":"2024-08-28T19:18:30.470283+0000","last_unstale":"2024-08-28T19:20:53.806036+0000","last_undegraded":"2024-08-28T19:20:53.806036+0000","last_fullsized":"2024-08-28T19:20:53.806036+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:09:46.444864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007249+0000","last_change":"2024-08-28T19:18:39.519226+0000","last_active":"2024-08-28T19:20:09.007249+0000","last_peered":"2024-08-28T19:20:09.007249+0000","last_clean":"2024-08-28T19:20:09.007249+0000","last_became_active":"2024-08-28T19:18:39.518738+0000","last_became_peered":"2024-08-28T19:18:39.518738+0000","last_unstale":"2024-08-28T19:20:09.007249+0000","last_undegraded":"2024-08-28T19:20:09.007249+0000","last_fullsized":"2024-08-28T19:20:09.007249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:39:35.114702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.877692+0000","last_change":"2024-08-28T19:18:32.493060+0000","last_active":"2024-08-28T19:20:37.877692+0000","last_peered":"2024-08-28T19:20:37.877692+0000","last_clean":"2024-08-28T19:20:37.877692+0000","last_became_active":"2024-08-28T19:18:32.492590+0000","last_became_peered":"2024-08-28T19:18:32.492590+0000","last_unstale":"2024-08-28T19:20:37.877692+0000","last_undegraded":"2024-08-28T19:20:37.877692+0000","last_fullsized":"2024-08-28T19:20:37.877692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:02:02.165301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008761+0000","last_change":"2024-08-28T19:18:30.470674+0000","last_active":"2024-08-28T19:20:09.008761+0000","last_peered":"2024-08-28T19:20:09.008761+0000","last_clean":"2024-08-28T19:20:09.008761+0000","last_became_active":"2024-08-28T19:18:30.470275+0000","last_became_peered":"2024-08-28T19:18:30.470275+0000","last_unstale":"2024-08-28T19:20:09.008761+0000","last_undegraded":"2024-08-28T19:20:09.008761+0000","last_fullsized":"2024-08-28T19:20:09.008761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:01:48.547315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663504+0000","last_change":"2024-08-28T19:19:08.624194+0000","last_active":"2024-08-28T19:20:09.663504+0000","last_peered":"2024-08-28T19:20:09.663504+0000","last_clean":"2024-08-28T19:20:09.663504+0000","last_became_active":"2024-08-28T19:19:08.623681+0000","last_became_peered":"2024-08-28T19:19:08.623681+0000","last_unstale":"2024-08-28T19:20:09.663504+0000","last_undegraded":"2024-08-28T19:20:09.663504+0000","last_fullsized":"2024-08-28T19:20:09.663504+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:21:11.099931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872115+0000","last_change":"2024-08-28T19:18:32.488210+0000","last_active":"2024-08-28T19:20:37.872115+0000","last_peered":"2024-08-28T19:20:37.872115+0000","last_clean":"2024-08-28T19:20:37.872115+0000","last_became_active":"2024-08-28T19:18:32.487853+0000","last_became_peered":"2024-08-28T19:18:32.487853+0000","last_unstale":"2024-08-28T19:20:37.872115+0000","last_undegraded":"2024-08-28T19:20:37.872115+0000","last_fullsized":"2024-08-28T19:20:37.872115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:20:51.808958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011302+0000","last_change":"2024-08-28T19:18:30.465306+0000","last_active":"2024-08-28T19:20:09.011302+0000","last_peered":"2024-08-28T19:20:09.011302+0000","last_clean":"2024-08-28T19:20:09.011302+0000","last_became_active":"2024-08-28T19:18:30.464850+0000","last_became_peered":"2024-08-28T19:18:30.464850+0000","last_unstale":"2024-08-28T19:20:09.011302+0000","last_undegraded":"2024-08-28T19:20:09.011302+0000","last_fullsized":"2024-08-28T19:20:09.011302+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:24:26.628999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008657+0000","last_change":"2024-08-28T19:18:39.524005+0000","last_active":"2024-08-28T19:20:09.008657+0000","last_peered":"2024-08-28T19:20:09.008657+0000","last_clean":"2024-08-28T19:20:09.008657+0000","last_became_active":"2024-08-28T19:18:39.523594+0000","last_became_peered":"2024-08-28T19:18:39.523594+0000","last_unstale":"2024-08-28T19:20:09.008657+0000","last_undegraded":"2024-08-28T19:20:09.008657+0000","last_fullsized":"2024-08-28T19:20:09.008657+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:35:32.484399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"84'6","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872604+0000","last_change":"2024-08-28T19:18:32.489088+0000","last_active":"2024-08-28T19:20:37.872604+0000","last_peered":"2024-08-28T19:20:37.872604+0000","last_clean":"2024-08-28T19:20:37.872604+0000","last_became_active":"2024-08-28T19:18:32.488717+0000","last_became_peered":"2024-08-28T19:18:32.488717+0000","last_unstale":"2024-08-28T19:20:37.872604+0000","last_undegraded":"2024-08-28T19:20:37.872604+0000","last_fullsized":"2024-08-28T19:20:37.872604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:25.468919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"84'40","reported_seq":297,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.807945+0000","last_change":"2024-08-28T19:18:30.468674+0000","last_active":"2024-08-28T19:20:54.807945+0000","last_peered":"2024-08-28T19:20:54.807945+0000","last_clean":"2024-08-28T19:20:54.807945+0000","last_became_active":"2024-08-28T19:18:30.468300+0000","last_became_peered":"2024-08-28T19:18:30.468300+0000","last_unstale":"2024-08-28T19:20:54.807945+0000","last_undegraded":"2024-08-28T19:20:54.807945+0000","last_fullsized":"2024-08-28T19:20:54.807945+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:12:06.237498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64032,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":217,"num_read_kb":171,"num_write":42,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007081+0000","last_change":"2024-08-28T19:18:39.526131+0000","last_active":"2024-08-28T19:20:09.007081+0000","last_peered":"2024-08-28T19:20:09.007081+0000","last_clean":"2024-08-28T19:20:09.007081+0000","last_became_active":"2024-08-28T19:18:39.525708+0000","last_became_peered":"2024-08-28T19:18:39.525708+0000","last_unstale":"2024-08-28T19:20:09.007081+0000","last_undegraded":"2024-08-28T19:20:09.007081+0000","last_fullsized":"2024-08-28T19:20:09.007081+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:21:52.156487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.885078+0000","last_change":"2024-08-28T19:18:32.487634+0000","last_active":"2024-08-28T19:20:37.885078+0000","last_peered":"2024-08-28T19:20:37.885078+0000","last_clean":"2024-08-28T19:20:37.885078+0000","last_became_active":"2024-08-28T19:18:32.486702+0000","last_became_peered":"2024-08-28T19:18:32.486702+0000","last_unstale":"2024-08-28T19:20:37.885078+0000","last_undegraded":"2024-08-28T19:20:37.885078+0000","last_fullsized":"2024-08-28T19:20:37.885078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:08:12.606905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008985+0000","last_change":"2024-08-28T19:18:31.083494+0000","last_active":"2024-08-28T19:20:09.008985+0000","last_peered":"2024-08-28T19:20:09.008985+0000","last_clean":"2024-08-28T19:20:09.008985+0000","last_became_active":"2024-08-28T19:18:31.083086+0000","last_became_peered":"2024-08-28T19:18:31.083086+0000","last_unstale":"2024-08-28T19:20:09.008985+0000","last_undegraded":"2024-08-28T19:20:09.008985+0000","last_fullsized":"2024-08-28T19:20:09.008985+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:14:54.388364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615373+0000","last_change":"2024-08-28T19:18:39.532155+0000","last_active":"2024-08-28T19:20:09.615373+0000","last_peered":"2024-08-28T19:20:09.615373+0000","last_clean":"2024-08-28T19:20:09.615373+0000","last_became_active":"2024-08-28T19:18:39.531565+0000","last_became_peered":"2024-08-28T19:18:39.531565+0000","last_unstale":"2024-08-28T19:20:09.615373+0000","last_undegraded":"2024-08-28T19:20:09.615373+0000","last_fullsized":"2024-08-28T19:20:09.615373+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:29:48.824622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"84'9","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880917+0000","last_change":"2024-08-28T19:18:32.496296+0000","last_active":"2024-08-28T19:20:37.880917+0000","last_peered":"2024-08-28T19:20:37.880917+0000","last_clean":"2024-08-28T19:20:37.880917+0000","last_became_active":"2024-08-28T19:18:32.495680+0000","last_became_peered":"2024-08-28T19:18:32.495680+0000","last_unstale":"2024-08-28T19:20:37.880917+0000","last_undegraded":"2024-08-28T19:20:37.880917+0000","last_fullsized":"2024-08-28T19:20:37.880917+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:27:34.544529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529311+0000","last_change":"2024-08-28T19:19:08.631758+0000","last_active":"2024-08-28T19:20:09.529311+0000","last_peered":"2024-08-28T19:20:09.529311+0000","last_clean":"2024-08-28T19:20:09.529311+0000","last_became_active":"2024-08-28T19:19:08.631202+0000","last_became_peered":"2024-08-28T19:19:08.631202+0000","last_unstale":"2024-08-28T19:20:09.529311+0000","last_undegraded":"2024-08-28T19:20:09.529311+0000","last_fullsized":"2024-08-28T19:20:09.529311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:27:43.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.006719+0000","last_change":"2024-08-28T19:18:39.531157+0000","last_active":"2024-08-28T19:20:09.006719+0000","last_peered":"2024-08-28T19:20:09.006719+0000","last_clean":"2024-08-28T19:20:09.006719+0000","last_became_active":"2024-08-28T19:18:39.530675+0000","last_became_peered":"2024-08-28T19:18:39.530675+0000","last_unstale":"2024-08-28T19:20:09.006719+0000","last_undegraded":"2024-08-28T19:20:09.006719+0000","last_fullsized":"2024-08-28T19:20:09.006719+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:38:40.414854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"84'10","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883543+0000","last_change":"2024-08-28T19:18:32.482182+0000","last_active":"2024-08-28T19:20:37.883543+0000","last_peered":"2024-08-28T19:20:37.883543+0000","last_clean":"2024-08-28T19:20:37.883543+0000","last_became_active":"2024-08-28T19:18:32.481689+0000","last_became_peered":"2024-08-28T19:18:32.481689+0000","last_unstale":"2024-08-28T19:20:37.883543+0000","last_undegraded":"2024-08-28T19:20:37.883543+0000","last_fullsized":"2024-08-28T19:20:37.883543+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:13:29.278469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'5","reported_seq":251,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.850315+0000","last_change":"2024-08-28T19:18:30.475244+0000","last_active":"2024-08-28T19:20:54.850315+0000","last_peered":"2024-08-28T19:20:54.850315+0000","last_clean":"2024-08-28T19:20:54.850315+0000","last_became_active":"2024-08-28T19:18:30.474861+0000","last_became_peered":"2024-08-28T19:18:30.474861+0000","last_unstale":"2024-08-28T19:20:54.850315+0000","last_undegraded":"2024-08-28T19:20:54.850315+0000","last_fullsized":"2024-08-28T19:20:54.850315+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:27:09.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":206,"num_read_kb":206,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525040+0000","last_change":"2024-08-28T19:18:39.520022+0000","last_active":"2024-08-28T19:20:09.525040+0000","last_peered":"2024-08-28T19:20:09.525040+0000","last_clean":"2024-08-28T19:20:09.525040+0000","last_became_active":"2024-08-28T19:18:39.519584+0000","last_became_peered":"2024-08-28T19:18:39.519584+0000","last_unstale":"2024-08-28T19:20:09.525040+0000","last_undegraded":"2024-08-28T19:20:09.525040+0000","last_fullsized":"2024-08-28T19:20:09.525040+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:27:15.851303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010363+0000","last_change":"2024-08-28T19:18:39.522607+0000","last_active":"2024-08-28T19:20:09.010363+0000","last_peered":"2024-08-28T19:20:09.010363+0000","last_clean":"2024-08-28T19:20:09.010363+0000","last_became_active":"2024-08-28T19:18:39.522204+0000","last_became_peered":"2024-08-28T19:18:39.522204+0000","last_unstale":"2024-08-28T19:20:09.010363+0000","last_undegraded":"2024-08-28T19:20:09.010363+0000","last_fullsized":"2024-08-28T19:20:09.010363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:02:08.865294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"84'11","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865503+0000","last_change":"2024-08-28T19:18:32.485577+0000","last_active":"2024-08-28T19:20:37.865503+0000","last_peered":"2024-08-28T19:20:37.865503+0000","last_clean":"2024-08-28T19:20:37.865503+0000","last_became_active":"2024-08-28T19:18:32.485243+0000","last_became_peered":"2024-08-28T19:18:32.485243+0000","last_unstale":"2024-08-28T19:20:37.865503+0000","last_undegraded":"2024-08-28T19:20:37.865503+0000","last_fullsized":"2024-08-28T19:20:37.865503+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:05:53.019365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008741+0000","last_change":"2024-08-28T19:18:30.470670+0000","last_active":"2024-08-28T19:20:09.008741+0000","last_peered":"2024-08-28T19:20:09.008741+0000","last_clean":"2024-08-28T19:20:09.008741+0000","last_became_active":"2024-08-28T19:18:30.470264+0000","last_became_peered":"2024-08-28T19:18:30.470264+0000","last_unstale":"2024-08-28T19:20:09.008741+0000","last_undegraded":"2024-08-28T19:20:09.008741+0000","last_fullsized":"2024-08-28T19:20:09.008741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:35.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663847+0000","last_change":"2024-08-28T19:18:39.530948+0000","last_active":"2024-08-28T19:20:09.663847+0000","last_peered":"2024-08-28T19:20:09.663847+0000","last_clean":"2024-08-28T19:20:09.663847+0000","last_became_active":"2024-08-28T19:18:39.530473+0000","last_became_peered":"2024-08-28T19:18:39.530473+0000","last_unstale":"2024-08-28T19:20:09.663847+0000","last_undegraded":"2024-08-28T19:20:09.663847+0000","last_fullsized":"2024-08-28T19:20:09.663847+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:21:57.504582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"84'8","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.869758+0000","last_change":"2024-08-28T19:18:32.486027+0000","last_active":"2024-08-28T19:20:37.869758+0000","last_peered":"2024-08-28T19:20:37.869758+0000","last_clean":"2024-08-28T19:20:37.869758+0000","last_became_active":"2024-08-28T19:18:32.483273+0000","last_became_peered":"2024-08-28T19:18:32.483273+0000","last_unstale":"2024-08-28T19:20:37.869758+0000","last_undegraded":"2024-08-28T19:20:37.869758+0000","last_fullsized":"2024-08-28T19:20:37.869758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:47:37.680695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162978+0000","last_change":"2024-08-28T19:18:30.464811+0000","last_active":"2024-08-28T19:20:09.162978+0000","last_peered":"2024-08-28T19:20:09.162978+0000","last_clean":"2024-08-28T19:20:09.162978+0000","last_became_active":"2024-08-28T19:18:30.463986+0000","last_became_peered":"2024-08-28T19:18:30.463986+0000","last_unstale":"2024-08-28T19:20:09.162978+0000","last_undegraded":"2024-08-28T19:20:09.162978+0000","last_fullsized":"2024-08-28T19:20:09.162978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:43:46.582993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":53,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663544+0000","last_change":"2024-08-28T19:19:08.631382+0000","last_active":"2024-08-28T19:20:09.663544+0000","last_peered":"2024-08-28T19:20:09.663544+0000","last_clean":"2024-08-28T19:20:09.663544+0000","last_became_active":"2024-08-28T19:19:08.630869+0000","last_became_peered":"2024-08-28T19:19:08.630869+0000","last_unstale":"2024-08-28T19:20:09.663544+0000","last_undegraded":"2024-08-28T19:20:09.663544+0000","last_fullsized":"2024-08-28T19:20:09.663544+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:32:54.942408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"84'8","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875341+0000","last_change":"2024-08-28T19:18:32.484120+0000","last_active":"2024-08-28T19:20:37.875341+0000","last_peered":"2024-08-28T19:20:37.875341+0000","last_clean":"2024-08-28T19:20:37.875341+0000","last_became_active":"2024-08-28T19:18:32.483013+0000","last_became_peered":"2024-08-28T19:18:32.483013+0000","last_unstale":"2024-08-28T19:20:37.875341+0000","last_undegraded":"2024-08-28T19:20:37.875341+0000","last_fullsized":"2024-08-28T19:20:37.875341+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:59:41.195873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010737+0000","last_change":"2024-08-28T19:18:39.531046+0000","last_active":"2024-08-28T19:20:09.010737+0000","last_peered":"2024-08-28T19:20:09.010737+0000","last_clean":"2024-08-28T19:20:09.010737+0000","last_became_active":"2024-08-28T19:18:39.530407+0000","last_became_peered":"2024-08-28T19:18:39.530407+0000","last_unstale":"2024-08-28T19:20:09.010737+0000","last_undegraded":"2024-08-28T19:20:09.010737+0000","last_fullsized":"2024-08-28T19:20:09.010737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:47:15.173021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'7","reported_seq":151,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:51.803528+0000","last_change":"2024-08-28T19:18:30.471268+0000","last_active":"2024-08-28T19:20:51.803528+0000","last_peered":"2024-08-28T19:20:51.803528+0000","last_clean":"2024-08-28T19:20:51.803528+0000","last_became_active":"2024-08-28T19:18:30.470845+0000","last_became_peered":"2024-08-28T19:18:30.470845+0000","last_unstale":"2024-08-28T19:20:51.803528+0000","last_undegraded":"2024-08-28T19:20:51.803528+0000","last_fullsized":"2024-08-28T19:20:51.803528+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:18:18.086205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"84'7","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.859389+0000","last_change":"2024-08-28T19:18:32.479728+0000","last_active":"2024-08-28T19:20:37.859389+0000","last_peered":"2024-08-28T19:20:37.859389+0000","last_clean":"2024-08-28T19:20:37.859389+0000","last_became_active":"2024-08-28T19:18:32.478623+0000","last_became_peered":"2024-08-28T19:18:32.478623+0000","last_unstale":"2024-08-28T19:20:37.859389+0000","last_undegraded":"2024-08-28T19:20:37.859389+0000","last_fullsized":"2024-08-28T19:20:37.859389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:14:49.914808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011224+0000","last_change":"2024-08-28T19:18:39.531308+0000","last_active":"2024-08-28T19:20:09.011224+0000","last_peered":"2024-08-28T19:20:09.011224+0000","last_clean":"2024-08-28T19:20:09.011224+0000","last_became_active":"2024-08-28T19:18:39.530403+0000","last_became_peered":"2024-08-28T19:18:39.530403+0000","last_unstale":"2024-08-28T19:20:09.011224+0000","last_undegraded":"2024-08-28T19:20:09.011224+0000","last_fullsized":"2024-08-28T19:20:09.011224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:33.139668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873003+0000","last_change":"2024-08-28T19:18:32.494621+0000","last_active":"2024-08-28T19:20:37.873003+0000","last_peered":"2024-08-28T19:20:37.873003+0000","last_clean":"2024-08-28T19:20:37.873003+0000","last_became_active":"2024-08-28T19:18:32.494217+0000","last_became_peered":"2024-08-28T19:18:32.494217+0000","last_unstale":"2024-08-28T19:20:37.873003+0000","last_undegraded":"2024-08-28T19:20:37.873003+0000","last_fullsized":"2024-08-28T19:20:37.873003+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:54:15.419470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"84'12","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872043+0000","last_change":"2024-08-28T19:20:09.616552+0000","last_active":"2024-08-28T19:20:37.872043+0000","last_peered":"2024-08-28T19:20:37.872043+0000","last_clean":"2024-08-28T19:20:37.872043+0000","last_became_active":"2024-08-28T19:20:09.616159+0000","last_became_peered":"2024-08-28T19:20:09.616159+0000","last_unstale":"2024-08-28T19:20:37.872043+0000","last_undegraded":"2024-08-28T19:20:37.872043+0000","last_fullsized":"2024-08-28T19:20:37.872043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:04:10.354703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"84'8","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866959+0000","last_change":"2024-08-28T19:20:09.012824+0000","last_active":"2024-08-28T19:20:37.866959+0000","last_peered":"2024-08-28T19:20:37.866959+0000","last_clean":"2024-08-28T19:20:37.866959+0000","last_became_active":"2024-08-28T19:20:09.012490+0000","last_became_peered":"2024-08-28T19:20:09.012490+0000","last_unstale":"2024-08-28T19:20:37.866959+0000","last_undegraded":"2024-08-28T19:20:37.866959+0000","last_fullsized":"2024-08-28T19:20:37.866959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:49:02.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871666+0000","last_change":"2024-08-28T19:20:09.164523+0000","last_active":"2024-08-28T19:20:37.871666+0000","last_peered":"2024-08-28T19:20:37.871666+0000","last_clean":"2024-08-28T19:20:37.871666+0000","last_became_active":"2024-08-28T19:20:09.163825+0000","last_became_peered":"2024-08-28T19:20:09.163825+0000","last_unstale":"2024-08-28T19:20:37.871666+0000","last_undegraded":"2024-08-28T19:20:37.871666+0000","last_fullsized":"2024-08-28T19:20:37.871666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:42:48.141103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868174+0000","last_change":"2024-08-28T19:18:32.485446+0000","last_active":"2024-08-28T19:20:37.868174+0000","last_peered":"2024-08-28T19:20:37.868174+0000","last_clean":"2024-08-28T19:20:37.868174+0000","last_became_active":"2024-08-28T19:18:32.484197+0000","last_became_peered":"2024-08-28T19:18:32.484197+0000","last_unstale":"2024-08-28T19:20:37.868174+0000","last_undegraded":"2024-08-28T19:20:37.868174+0000","last_fullsized":"2024-08-28T19:20:37.868174+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:47:50.527915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868339+0000","last_change":"2024-08-28T19:18:32.489452+0000","last_active":"2024-08-28T19:20:37.868339+0000","last_peered":"2024-08-28T19:20:37.868339+0000","last_clean":"2024-08-28T19:20:37.868339+0000","last_became_active":"2024-08-28T19:18:32.487482+0000","last_became_peered":"2024-08-28T19:18:32.487482+0000","last_unstale":"2024-08-28T19:20:37.868339+0000","last_undegraded":"2024-08-28T19:20:37.868339+0000","last_fullsized":"2024-08-28T19:20:37.868339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:52:08.175080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884068+0000","last_change":"2024-08-28T19:18:32.483850+0000","last_active":"2024-08-28T19:20:37.884068+0000","last_peered":"2024-08-28T19:20:37.884068+0000","last_clean":"2024-08-28T19:20:37.884068+0000","last_became_active":"2024-08-28T19:18:32.483476+0000","last_became_peered":"2024-08-28T19:18:32.483476+0000","last_unstale":"2024-08-28T19:20:37.884068+0000","last_undegraded":"2024-08-28T19:20:37.884068+0000","last_fullsized":"2024-08-28T19:20:37.884068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:32:37.465218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868836+0000","last_change":"2024-08-28T19:18:32.492252+0000","last_active":"2024-08-28T19:20:37.868836+0000","last_peered":"2024-08-28T19:20:37.868836+0000","last_clean":"2024-08-28T19:20:37.868836+0000","last_became_active":"2024-08-28T19:18:32.491659+0000","last_became_peered":"2024-08-28T19:18:32.491659+0000","last_unstale":"2024-08-28T19:20:37.868836+0000","last_undegraded":"2024-08-28T19:20:37.868836+0000","last_fullsized":"2024-08-28T19:20:37.868836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:19:28.901559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865090+0000","last_change":"2024-08-28T19:18:32.491940+0000","last_active":"2024-08-28T19:20:37.865090+0000","last_peered":"2024-08-28T19:20:37.865090+0000","last_clean":"2024-08-28T19:20:37.865090+0000","last_became_active":"2024-08-28T19:18:32.487474+0000","last_became_peered":"2024-08-28T19:18:32.487474+0000","last_unstale":"2024-08-28T19:20:37.865090+0000","last_undegraded":"2024-08-28T19:20:37.865090+0000","last_fullsized":"2024-08-28T19:20:37.865090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:04:32.508620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871438+0000","last_change":"2024-08-28T19:18:32.485200+0000","last_active":"2024-08-28T19:20:37.871438+0000","last_peered":"2024-08-28T19:20:37.871438+0000","last_clean":"2024-08-28T19:20:37.871438+0000","last_became_active":"2024-08-28T19:18:32.484858+0000","last_became_peered":"2024-08-28T19:18:32.484858+0000","last_unstale":"2024-08-28T19:20:37.871438+0000","last_undegraded":"2024-08-28T19:20:37.871438+0000","last_fullsized":"2024-08-28T19:20:37.871438+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:10:57.925659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886493+0000","last_change":"2024-08-28T19:18:32.489903+0000","last_active":"2024-08-28T19:20:37.886493+0000","last_peered":"2024-08-28T19:20:37.886493+0000","last_clean":"2024-08-28T19:20:37.886493+0000","last_became_active":"2024-08-28T19:18:32.488950+0000","last_became_peered":"2024-08-28T19:18:32.488950+0000","last_unstale":"2024-08-28T19:20:37.886493+0000","last_undegraded":"2024-08-28T19:20:37.886493+0000","last_fullsized":"2024-08-28T19:20:37.886493+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:06:42.284177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872318+0000","last_change":"2024-08-28T19:18:32.488236+0000","last_active":"2024-08-28T19:20:37.872318+0000","last_peered":"2024-08-28T19:20:37.872318+0000","last_clean":"2024-08-28T19:20:37.872318+0000","last_became_active":"2024-08-28T19:18:32.487776+0000","last_became_peered":"2024-08-28T19:18:32.487776+0000","last_unstale":"2024-08-28T19:20:37.872318+0000","last_undegraded":"2024-08-28T19:20:37.872318+0000","last_fullsized":"2024-08-28T19:20:37.872318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:25:58.754740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875857+0000","last_change":"2024-08-28T19:18:32.485860+0000","last_active":"2024-08-28T19:20:37.875857+0000","last_peered":"2024-08-28T19:20:37.875857+0000","last_clean":"2024-08-28T19:20:37.875857+0000","last_became_active":"2024-08-28T19:18:32.485415+0000","last_became_peered":"2024-08-28T19:18:32.485415+0000","last_unstale":"2024-08-28T19:20:37.875857+0000","last_undegraded":"2024-08-28T19:20:37.875857+0000","last_fullsized":"2024-08-28T19:20:37.875857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:45:32.868651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873054+0000","last_change":"2024-08-28T19:18:32.485515+0000","last_active":"2024-08-28T19:20:37.873054+0000","last_peered":"2024-08-28T19:20:37.873054+0000","last_clean":"2024-08-28T19:20:37.873054+0000","last_became_active":"2024-08-28T19:18:32.485075+0000","last_became_peered":"2024-08-28T19:18:32.485075+0000","last_unstale":"2024-08-28T19:20:37.873054+0000","last_undegraded":"2024-08-28T19:20:37.873054+0000","last_fullsized":"2024-08-28T19:20:37.873054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:01:28.035479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864600+0000","last_change":"2024-08-28T19:18:32.484685+0000","last_active":"2024-08-28T19:20:37.864600+0000","last_peered":"2024-08-28T19:20:37.864600+0000","last_clean":"2024-08-28T19:20:37.864600+0000","last_became_active":"2024-08-28T19:18:32.484095+0000","last_became_peered":"2024-08-28T19:18:32.484095+0000","last_unstale":"2024-08-28T19:20:37.864600+0000","last_undegraded":"2024-08-28T19:20:37.864600+0000","last_fullsized":"2024-08-28T19:20:37.864600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:53:27.229186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884050+0000","last_change":"2024-08-28T19:18:32.488030+0000","last_active":"2024-08-28T19:20:37.884050+0000","last_peered":"2024-08-28T19:20:37.884050+0000","last_clean":"2024-08-28T19:20:37.884050+0000","last_became_active":"2024-08-28T19:18:32.487042+0000","last_became_peered":"2024-08-28T19:18:32.487042+0000","last_unstale":"2024-08-28T19:20:37.884050+0000","last_undegraded":"2024-08-28T19:20:37.884050+0000","last_fullsized":"2024-08-28T19:20:37.884050+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:56.966475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886125+0000","last_change":"2024-08-28T19:18:32.489442+0000","last_active":"2024-08-28T19:20:37.886125+0000","last_peered":"2024-08-28T19:20:37.886125+0000","last_clean":"2024-08-28T19:20:37.886125+0000","last_became_active":"2024-08-28T19:18:32.488405+0000","last_became_peered":"2024-08-28T19:18:32.488405+0000","last_unstale":"2024-08-28T19:20:37.886125+0000","last_undegraded":"2024-08-28T19:20:37.886125+0000","last_fullsized":"2024-08-28T19:20:37.886125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:21:03.026893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870138+0000","last_change":"2024-08-28T19:20:09.164305+0000","last_active":"2024-08-28T19:20:37.870138+0000","last_peered":"2024-08-28T19:20:37.870138+0000","last_clean":"2024-08-28T19:20:37.870138+0000","last_became_active":"2024-08-28T19:20:09.163834+0000","last_became_peered":"2024-08-28T19:20:09.163834+0000","last_unstale":"2024-08-28T19:20:37.870138+0000","last_undegraded":"2024-08-28T19:20:37.870138+0000","last_fullsized":"2024-08-28T19:20:37.870138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:20:10.137952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870501+0000","last_change":"2024-08-28T19:18:32.492077+0000","last_active":"2024-08-28T19:20:37.870501+0000","last_peered":"2024-08-28T19:20:37.870501+0000","last_clean":"2024-08-28T19:20:37.870501+0000","last_became_active":"2024-08-28T19:18:32.491722+0000","last_became_peered":"2024-08-28T19:18:32.491722+0000","last_unstale":"2024-08-28T19:20:37.870501+0000","last_undegraded":"2024-08-28T19:20:37.870501+0000","last_fullsized":"2024-08-28T19:20:37.870501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:16:21.697045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"84'10","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865421+0000","last_change":"2024-08-28T19:18:32.479845+0000","last_active":"2024-08-28T19:20:37.865421+0000","last_peered":"2024-08-28T19:20:37.865421+0000","last_clean":"2024-08-28T19:20:37.865421+0000","last_became_active":"2024-08-28T19:18:32.477690+0000","last_became_peered":"2024-08-28T19:18:32.477690+0000","last_unstale":"2024-08-28T19:20:37.865421+0000","last_undegraded":"2024-08-28T19:20:37.865421+0000","last_fullsized":"2024-08-28T19:20:37.865421+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:11.281415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866009+0000","last_change":"2024-08-28T19:19:08.624185+0000","last_active":"2024-08-28T19:20:37.866009+0000","last_peered":"2024-08-28T19:20:37.866009+0000","last_clean":"2024-08-28T19:20:37.866009+0000","last_became_active":"2024-08-28T19:19:08.623745+0000","last_became_peered":"2024-08-28T19:19:08.623745+0000","last_unstale":"2024-08-28T19:20:37.866009+0000","last_undegraded":"2024-08-28T19:20:37.866009+0000","last_fullsized":"2024-08-28T19:20:37.866009+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:56.657173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"84'6","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886865+0000","last_change":"2024-08-28T19:18:32.499038+0000","last_active":"2024-08-28T19:20:37.886865+0000","last_peered":"2024-08-28T19:20:37.886865+0000","last_clean":"2024-08-28T19:20:37.886865+0000","last_became_active":"2024-08-28T19:18:32.498276+0000","last_became_peered":"2024-08-28T19:18:32.498276+0000","last_unstale":"2024-08-28T19:20:37.886865+0000","last_undegraded":"2024-08-28T19:20:37.886865+0000","last_fullsized":"2024-08-28T19:20:37.886865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:17:45.468448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"84'16","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.882260+0000","last_change":"2024-08-28T19:18:32.492183+0000","last_active":"2024-08-28T19:20:37.882260+0000","last_peered":"2024-08-28T19:20:37.882260+0000","last_clean":"2024-08-28T19:20:37.882260+0000","last_became_active":"2024-08-28T19:18:32.491815+0000","last_became_peered":"2024-08-28T19:18:32.491815+0000","last_unstale":"2024-08-28T19:20:37.882260+0000","last_undegraded":"2024-08-28T19:20:37.882260+0000","last_fullsized":"2024-08-28T19:20:37.882260+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:13:11.183955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"84'11","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886721+0000","last_change":"2024-08-28T19:18:32.497399+0000","last_active":"2024-08-28T19:20:37.886721+0000","last_peered":"2024-08-28T19:20:37.886721+0000","last_clean":"2024-08-28T19:20:37.886721+0000","last_became_active":"2024-08-28T19:18:32.496901+0000","last_became_peered":"2024-08-28T19:18:32.496901+0000","last_unstale":"2024-08-28T19:20:37.886721+0000","last_undegraded":"2024-08-28T19:20:37.886721+0000","last_fullsized":"2024-08-28T19:20:37.886721+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:05:57.634590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"84'9","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864415+0000","last_change":"2024-08-28T19:18:32.495871+0000","last_active":"2024-08-28T19:20:37.864415+0000","last_peered":"2024-08-28T19:20:37.864415+0000","last_clean":"2024-08-28T19:20:37.864415+0000","last_became_active":"2024-08-28T19:18:32.495515+0000","last_became_peered":"2024-08-28T19:18:32.495515+0000","last_unstale":"2024-08-28T19:20:37.864415+0000","last_undegraded":"2024-08-28T19:20:37.864415+0000","last_fullsized":"2024-08-28T19:20:37.864415+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:53:56.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"84'5","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875276+0000","last_change":"2024-08-28T19:18:32.495711+0000","last_active":"2024-08-28T19:20:37.875276+0000","last_peered":"2024-08-28T19:20:37.875276+0000","last_clean":"2024-08-28T19:20:37.875276+0000","last_became_active":"2024-08-28T19:18:32.495271+0000","last_became_peered":"2024-08-28T19:18:32.495271+0000","last_unstale":"2024-08-28T19:20:37.875276+0000","last_undegraded":"2024-08-28T19:20:37.875276+0000","last_fullsized":"2024-08-28T19:20:37.875276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:05.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"84'6","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880581+0000","last_change":"2024-08-28T19:18:32.489731+0000","last_active":"2024-08-28T19:20:37.880581+0000","last_peered":"2024-08-28T19:20:37.880581+0000","last_clean":"2024-08-28T19:20:37.880581+0000","last_became_active":"2024-08-28T19:18:32.489366+0000","last_became_peered":"2024-08-28T19:18:32.489366+0000","last_unstale":"2024-08-28T19:20:37.880581+0000","last_undegraded":"2024-08-28T19:20:37.880581+0000","last_fullsized":"2024-08-28T19:20:37.880581+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:52:24.180155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"84'4","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.861147+0000","last_change":"2024-08-28T19:18:32.490270+0000","last_active":"2024-08-28T19:20:37.861147+0000","last_peered":"2024-08-28T19:20:37.861147+0000","last_clean":"2024-08-28T19:20:37.861147+0000","last_became_active":"2024-08-28T19:18:32.489667+0000","last_became_peered":"2024-08-28T19:18:32.489667+0000","last_unstale":"2024-08-28T19:20:37.861147+0000","last_undegraded":"2024-08-28T19:20:37.861147+0000","last_fullsized":"2024-08-28T19:20:37.861147+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:18.571978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873527+0000","last_change":"2024-08-28T19:18:32.482553+0000","last_active":"2024-08-28T19:20:37.873527+0000","last_peered":"2024-08-28T19:20:37.873527+0000","last_clean":"2024-08-28T19:20:37.873527+0000","last_became_active":"2024-08-28T19:18:32.481161+0000","last_became_peered":"2024-08-28T19:18:32.481161+0000","last_unstale":"2024-08-28T19:20:37.873527+0000","last_undegraded":"2024-08-28T19:20:37.873527+0000","last_fullsized":"2024-08-28T19:20:37.873527+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:13.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883684+0000","last_change":"2024-08-28T19:18:32.492473+0000","last_active":"2024-08-28T19:20:37.883684+0000","last_peered":"2024-08-28T19:20:37.883684+0000","last_clean":"2024-08-28T19:20:37.883684+0000","last_became_active":"2024-08-28T19:18:32.491897+0000","last_became_peered":"2024-08-28T19:18:32.491897+0000","last_unstale":"2024-08-28T19:20:37.883684+0000","last_undegraded":"2024-08-28T19:20:37.883684+0000","last_fullsized":"2024-08-28T19:20:37.883684+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:24.279391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870908+0000","last_change":"2024-08-28T19:18:32.485621+0000","last_active":"2024-08-28T19:20:37.870908+0000","last_peered":"2024-08-28T19:20:37.870908+0000","last_clean":"2024-08-28T19:20:37.870908+0000","last_became_active":"2024-08-28T19:18:32.485267+0000","last_became_peered":"2024-08-28T19:18:32.485267+0000","last_unstale":"2024-08-28T19:20:37.870908+0000","last_undegraded":"2024-08-28T19:20:37.870908+0000","last_fullsized":"2024-08-28T19:20:37.870908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:30:08.932135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884032+0000","last_change":"2024-08-28T19:18:32.488457+0000","last_active":"2024-08-28T19:20:37.884032+0000","last_peered":"2024-08-28T19:20:37.884032+0000","last_clean":"2024-08-28T19:20:37.884032+0000","last_became_active":"2024-08-28T19:18:32.487555+0000","last_became_peered":"2024-08-28T19:18:32.487555+0000","last_unstale":"2024-08-28T19:20:37.884032+0000","last_undegraded":"2024-08-28T19:20:37.884032+0000","last_fullsized":"2024-08-28T19:20:37.884032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:03:26.390481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71518,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1074,"num_read_kb":618,"num_write":262,"num_write_kb":152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":225,"ondisk_log_size":225,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":68,"seq":292057776171,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":500491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":62,"seq":266287972399,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1683456,"data_stored":1154763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":57,"seq":244813135925,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":503845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":51,"seq":219043332153,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36324,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708828,"statfs":{"total":95995035648,"available":95957839872,"internally_reserved":0,"allocated":1003520,"data_stored":500127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":45,"seq":193273528382,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1593344,"data_stored":1096625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":40,"seq":171798691908,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":495785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888135,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1605632,"data_stored":1088251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051662,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":499327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215186,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40476,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704676,"statfs":{"total":95995035648,"available":95953588224,"internally_reserved":0,"allocated":995328,"data_stored":494961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":561067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1052672,"data_stored":560847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705761,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":496709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-28T19:20:59.284 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-08-28T19:20:59.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-28T19:20:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.428+0000 7f46168c3640 1 -- 172.21.15.16:0/2010783869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4610104e60 msgr2=0x7f4610105270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.428+0000 7f46168c3640 1 --2- 172.21.15.16:0/2010783869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4610104e60 0x7f4610105270 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4604009900 tx=0x7f460402a410 comp rx=0 tx=0).stop 2024-08-28T19:20:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.428+0000 7f46168c3640 1 -- 172.21.15.16:0/2010783869 shutdown_connections 2024-08-28T19:20:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.428+0000 7f46168c3640 1 --2- 172.21.15.16:0/2010783869 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46101057b0 0x7f461010f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.428+0000 7f46168c3640 1 --2- 172.21.15.16:0/2010783869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4610104e60 0x7f4610105270 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.428+0000 7f46168c3640 1 --2- 172.21.15.16:0/2010783869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4610102ea0 0x7f46101032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.428+0000 7f46168c3640 1 -- 172.21.15.16:0/2010783869 >> 172.21.15.16:0/2010783869 conn(0x7f4610079a80 msgr2=0x7f4610079e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.428+0000 7f46168c3640 1 -- 172.21.15.16:0/2010783869 shutdown_connections 2024-08-28T19:20:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.428+0000 7f46168c3640 1 -- 172.21.15.16:0/2010783869 wait complete. 2024-08-28T19:20:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.429+0000 7f46168c3640 1 Processor -- start 2024-08-28T19:20:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.429+0000 7f46168c3640 1 -- start start 2024-08-28T19:20:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.429+0000 7f46168c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4610102ea0 0x7f461019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.429+0000 7f46168c3640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4610104e60 0x7f461019a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.429+0000 7f46168c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46101057b0 0x7f461019fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.429+0000 7f46168c3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46101137a0 con 0x7f4610102ea0 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.429+0000 7f46168c3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4610113620 con 0x7f46101057b0 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.429+0000 7f46168c3640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4610113920 con 0x7f4610104e60 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.429+0000 7f460ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4610102ea0 0x7f461019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.429+0000 7f460ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4610102ea0 0x7f461019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37086/0 (socket says 172.21.15.16:37086) 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.429+0000 7f460ffff640 1 -- 172.21.15.16:0/1431702764 learned_addr learned my addr 172.21.15.16:0/1431702764 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f4614e60640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46101057b0 0x7f461019fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f460f7fe640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4610104e60 0x7f461019a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f460ffff640 1 -- 172.21.15.16:0/1431702764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4610104e60 msgr2=0x7f461019a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f460ffff640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4610104e60 0x7f461019a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f460ffff640 1 -- 172.21.15.16:0/1431702764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46101057b0 msgr2=0x7f461019fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f460ffff640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46101057b0 0x7f461019fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f460ffff640 1 -- 172.21.15.16:0/1431702764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46101a0170 con 0x7f4610102ea0 2024-08-28T19:20:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f460ffff640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4610102ea0 0x7f461019a240 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f460000e9c0 tx=0x7f460000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f460d7fa640 1 -- 172.21.15.16:0/1431702764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460000cd80 con 0x7f4610102ea0 2024-08-28T19:20:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f460d7fa640 1 -- 172.21.15.16:0/1431702764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46000035f0 con 0x7f4610102ea0 2024-08-28T19:20:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f460d7fa640 1 -- 172.21.15.16:0/1431702764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4600010690 con 0x7f4610102ea0 2024-08-28T19:20:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f46168c3640 1 -- 172.21.15.16:0/1431702764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46101a0460 con 0x7f4610102ea0 2024-08-28T19:20:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.430+0000 7f46168c3640 1 -- 172.21.15.16:0/1431702764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46101a6bf0 con 0x7f4610102ea0 2024-08-28T19:20:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.431+0000 7f46168c3640 1 -- 172.21.15.16:0/1431702764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46101032b0 con 0x7f4610102ea0 2024-08-28T19:20:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.433+0000 7f460d7fa640 1 -- 172.21.15.16:0/1431702764 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7f46000037a0 con 0x7f4610102ea0 2024-08-28T19:20:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.433+0000 7f460d7fa640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45f80b4c00 0x7f45f80b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:20:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.434+0000 7f460f7fe640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45f80b4c00 0x7f45f80b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:20:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.434+0000 7f460d7fa640 1 -- 172.21.15.16:0/1431702764 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f46000e5a20 con 0x7f4610102ea0 2024-08-28T19:20:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.434+0000 7f460f7fe640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45f80b4c00 0x7f45f80b7110 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4604009230 tx=0x7f4604031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:20:59.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.436+0000 7f460d7fa640 1 -- 172.21.15.16:0/1431702764 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f46000ac6c0 con 0x7f4610102ea0 2024-08-28T19:20:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:20:59 smithi028 ceph-mon[21769]: pgmap v337: 129 pgs: 129 active+clean; 647 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-08-28T19:20:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:20:59 smithi115 ceph-mon[22184]: pgmap v337: 129 pgs: 129 active+clean; 647 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-08-28T19:20:59.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.625+0000 7f46168c3640 1 -- 172.21.15.16:0/1431702764 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f46100635a0 con 0x7f45f80b4c00 2024-08-28T19:20:59.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.631+0000 7f460d7fa640 1 -- 172.21.15.16:0/1431702764 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340519 (secure 0 0 0) 0x7f46100635a0 con 0x7f45f80b4c00 2024-08-28T19:20:59.636 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:20:59.638 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 -- 172.21.15.16:0/1431702764 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45f80b4c00 msgr2=0x7f45f80b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45f80b4c00 0x7f45f80b7110 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4604009230 tx=0x7f4604031400 comp rx=0 tx=0).stop 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 -- 172.21.15.16:0/1431702764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4610102ea0 msgr2=0x7f461019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4610102ea0 0x7f461019a240 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f460000e9c0 tx=0x7f460000ee80 comp rx=0 tx=0).stop 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 -- 172.21.15.16:0/1431702764 shutdown_connections 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45f80b4c00 0x7f45f80b7110 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46101057b0 0x7f461019fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4610104e60 0x7f461019a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 --2- 172.21.15.16:0/1431702764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4610102ea0 0x7f461019a240 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 -- 172.21.15.16:0/1431702764 >> 172.21.15.16:0/1431702764 conn(0x7f4610079a80 msgr2=0x7f461006a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 -- 172.21.15.16:0/1431702764 shutdown_connections 2024-08-28T19:20:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:20:59.634+0000 7f46168c3640 1 -- 172.21.15.16:0/1431702764 wait complete. 2024-08-28T19:20:59.647 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:20:59 smithi016 ceph-mon[18556]: pgmap v337: 129 pgs: 129 active+clean; 647 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-08-28T19:20:59.648 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-08-28T19:20:59.496125+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662120,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1165,"num_read_kb":687,"num_write":387,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":819,"ondisk_log_size":819,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":416992,"kb_used_data":13664,"kb_used_omap":18,"kb_used_meta":403117,"kb_avail":1124524832,"statfs":{"total":1151940427776,"available":1151513427968,"internally_reserved":0,"allocated":13991936,"data_stored":7952798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":412792122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":308,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":51,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003891"},"pg_stats":[{"pgid":"2.1f","version":"84'14","reported_seq":172,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.806536+0000","last_change":"2024-08-28T19:19:08.633326+0000","last_active":"2024-08-28T19:20:54.806536+0000","last_peered":"2024-08-28T19:20:54.806536+0000","last_clean":"2024-08-28T19:20:54.806536+0000","last_became_active":"2024-08-28T19:19:08.623876+0000","last_became_peered":"2024-08-28T19:19:08.623876+0000","last_unstale":"2024-08-28T19:20:54.806536+0000","last_undegraded":"2024-08-28T19:20:54.806536+0000","last_fullsized":"2024-08-28T19:20:54.806536+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:19:50.632337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866980+0000","last_change":"2024-08-28T19:18:32.492604+0000","last_active":"2024-08-28T19:20:37.866980+0000","last_peered":"2024-08-28T19:20:37.866980+0000","last_clean":"2024-08-28T19:20:37.866980+0000","last_became_active":"2024-08-28T19:18:32.489427+0000","last_became_peered":"2024-08-28T19:18:32.489427+0000","last_unstale":"2024-08-28T19:20:37.866980+0000","last_undegraded":"2024-08-28T19:20:37.866980+0000","last_fullsized":"2024-08-28T19:20:37.866980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:36:14.841315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010445+0000","last_change":"2024-08-28T19:18:39.528815+0000","last_active":"2024-08-28T19:20:09.010445+0000","last_peered":"2024-08-28T19:20:09.010445+0000","last_clean":"2024-08-28T19:20:09.010445+0000","last_became_active":"2024-08-28T19:18:39.527781+0000","last_became_peered":"2024-08-28T19:18:39.527781+0000","last_unstale":"2024-08-28T19:20:09.010445+0000","last_undegraded":"2024-08-28T19:20:09.010445+0000","last_fullsized":"2024-08-28T19:20:09.010445+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:39.831426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525559+0000","last_change":"2024-08-28T19:18:31.082641+0000","last_active":"2024-08-28T19:20:09.525559+0000","last_peered":"2024-08-28T19:20:09.525559+0000","last_clean":"2024-08-28T19:20:09.525559+0000","last_became_active":"2024-08-28T19:18:31.082260+0000","last_became_peered":"2024-08-28T19:18:31.082260+0000","last_unstale":"2024-08-28T19:20:09.525559+0000","last_undegraded":"2024-08-28T19:20:09.525559+0000","last_fullsized":"2024-08-28T19:20:09.525559+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:09:55.836463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886294+0000","last_change":"2024-08-28T19:18:32.493498+0000","last_active":"2024-08-28T19:20:37.886294+0000","last_peered":"2024-08-28T19:20:37.886294+0000","last_clean":"2024-08-28T19:20:37.886294+0000","last_became_active":"2024-08-28T19:18:32.490969+0000","last_became_peered":"2024-08-28T19:18:32.490969+0000","last_unstale":"2024-08-28T19:20:37.886294+0000","last_undegraded":"2024-08-28T19:20:37.886294+0000","last_fullsized":"2024-08-28T19:20:37.886294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:34:29.537622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614531+0000","last_change":"2024-08-28T19:18:39.520922+0000","last_active":"2024-08-28T19:20:09.614531+0000","last_peered":"2024-08-28T19:20:09.614531+0000","last_clean":"2024-08-28T19:20:09.614531+0000","last_became_active":"2024-08-28T19:18:39.520362+0000","last_became_peered":"2024-08-28T19:18:39.520362+0000","last_unstale":"2024-08-28T19:20:09.614531+0000","last_undegraded":"2024-08-28T19:20:09.614531+0000","last_fullsized":"2024-08-28T19:20:09.614531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:40.658820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":52,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110351+0000","last_change":"2024-08-28T19:18:30.471870+0000","last_active":"2024-08-28T19:20:09.110351+0000","last_peered":"2024-08-28T19:20:09.110351+0000","last_clean":"2024-08-28T19:20:09.110351+0000","last_became_active":"2024-08-28T19:18:30.471438+0000","last_became_peered":"2024-08-28T19:18:30.471438+0000","last_unstale":"2024-08-28T19:20:09.110351+0000","last_undegraded":"2024-08-28T19:20:09.110351+0000","last_fullsized":"2024-08-28T19:20:09.110351+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:44:34.137395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"84'9","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.881074+0000","last_change":"2024-08-28T19:18:32.486485+0000","last_active":"2024-08-28T19:20:37.881074+0000","last_peered":"2024-08-28T19:20:37.881074+0000","last_clean":"2024-08-28T19:20:37.881074+0000","last_became_active":"2024-08-28T19:18:32.485888+0000","last_became_peered":"2024-08-28T19:18:32.485888+0000","last_unstale":"2024-08-28T19:20:37.881074+0000","last_undegraded":"2024-08-28T19:20:37.881074+0000","last_fullsized":"2024-08-28T19:20:37.881074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:45:48.757318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615105+0000","last_change":"2024-08-28T19:19:08.633317+0000","last_active":"2024-08-28T19:20:09.615105+0000","last_peered":"2024-08-28T19:20:09.615105+0000","last_clean":"2024-08-28T19:20:09.615105+0000","last_became_active":"2024-08-28T19:19:08.632689+0000","last_became_peered":"2024-08-28T19:19:08.632689+0000","last_unstale":"2024-08-28T19:20:09.615105+0000","last_undegraded":"2024-08-28T19:20:09.615105+0000","last_fullsized":"2024-08-28T19:20:09.615105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:35:08.740645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110282+0000","last_change":"2024-08-28T19:18:30.473982+0000","last_active":"2024-08-28T19:20:09.110282+0000","last_peered":"2024-08-28T19:20:09.110282+0000","last_clean":"2024-08-28T19:20:09.110282+0000","last_became_active":"2024-08-28T19:18:30.473552+0000","last_became_peered":"2024-08-28T19:18:30.473552+0000","last_unstale":"2024-08-28T19:20:09.110282+0000","last_undegraded":"2024-08-28T19:20:09.110282+0000","last_fullsized":"2024-08-28T19:20:09.110282+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:22:29.435517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"84'15","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884248+0000","last_change":"2024-08-28T19:18:32.487881+0000","last_active":"2024-08-28T19:20:37.884248+0000","last_peered":"2024-08-28T19:20:37.884248+0000","last_clean":"2024-08-28T19:20:37.884248+0000","last_became_active":"2024-08-28T19:18:32.487039+0000","last_became_peered":"2024-08-28T19:18:32.487039+0000","last_unstale":"2024-08-28T19:20:37.884248+0000","last_undegraded":"2024-08-28T19:20:37.884248+0000","last_fullsized":"2024-08-28T19:20:37.884248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:23:35.757811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614594+0000","last_change":"2024-08-28T19:18:39.518654+0000","last_active":"2024-08-28T19:20:09.614594+0000","last_peered":"2024-08-28T19:20:09.614594+0000","last_clean":"2024-08-28T19:20:09.614594+0000","last_became_active":"2024-08-28T19:18:39.518087+0000","last_became_peered":"2024-08-28T19:18:39.518087+0000","last_unstale":"2024-08-28T19:20:09.614594+0000","last_undegraded":"2024-08-28T19:20:09.614594+0000","last_fullsized":"2024-08-28T19:20:09.614594+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:08.284865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.887381+0000","last_change":"2024-08-28T19:18:32.491928+0000","last_active":"2024-08-28T19:20:37.887381+0000","last_peered":"2024-08-28T19:20:37.887381+0000","last_clean":"2024-08-28T19:20:37.887381+0000","last_became_active":"2024-08-28T19:18:32.491538+0000","last_became_peered":"2024-08-28T19:18:32.491538+0000","last_unstale":"2024-08-28T19:20:37.887381+0000","last_undegraded":"2024-08-28T19:20:37.887381+0000","last_fullsized":"2024-08-28T19:20:37.887381+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:39:06.077830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":24,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663446+0000","last_change":"2024-08-28T19:19:08.634752+0000","last_active":"2024-08-28T19:20:09.663446+0000","last_peered":"2024-08-28T19:20:09.663446+0000","last_clean":"2024-08-28T19:20:09.663446+0000","last_became_active":"2024-08-28T19:19:08.620784+0000","last_became_peered":"2024-08-28T19:19:08.620784+0000","last_unstale":"2024-08-28T19:20:09.663446+0000","last_undegraded":"2024-08-28T19:20:09.663446+0000","last_fullsized":"2024-08-28T19:20:09.663446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:51.640445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614895+0000","last_change":"2024-08-28T19:18:39.681413+0000","last_active":"2024-08-28T19:20:09.614895+0000","last_peered":"2024-08-28T19:20:09.614895+0000","last_clean":"2024-08-28T19:20:09.614895+0000","last_became_active":"2024-08-28T19:18:39.680793+0000","last_became_peered":"2024-08-28T19:18:39.680793+0000","last_unstale":"2024-08-28T19:20:09.614895+0000","last_undegraded":"2024-08-28T19:20:09.614895+0000","last_fullsized":"2024-08-28T19:20:09.614895+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:45:43.381019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870888+0000","last_change":"2024-08-28T19:18:32.486739+0000","last_active":"2024-08-28T19:20:37.870888+0000","last_peered":"2024-08-28T19:20:37.870888+0000","last_clean":"2024-08-28T19:20:37.870888+0000","last_became_active":"2024-08-28T19:18:32.486347+0000","last_became_peered":"2024-08-28T19:18:32.486347+0000","last_unstale":"2024-08-28T19:20:37.870888+0000","last_undegraded":"2024-08-28T19:20:37.870888+0000","last_fullsized":"2024-08-28T19:20:37.870888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:27:36.448569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010653+0000","last_change":"2024-08-28T19:18:30.467690+0000","last_active":"2024-08-28T19:20:09.010653+0000","last_peered":"2024-08-28T19:20:09.010653+0000","last_clean":"2024-08-28T19:20:09.010653+0000","last_became_active":"2024-08-28T19:18:30.467249+0000","last_became_peered":"2024-08-28T19:18:30.467249+0000","last_unstale":"2024-08-28T19:20:09.010653+0000","last_undegraded":"2024-08-28T19:20:09.010653+0000","last_fullsized":"2024-08-28T19:20:09.010653+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:37:28.876666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615167+0000","last_change":"2024-08-28T19:18:39.523889+0000","last_active":"2024-08-28T19:20:09.615167+0000","last_peered":"2024-08-28T19:20:09.615167+0000","last_clean":"2024-08-28T19:20:09.615167+0000","last_became_active":"2024-08-28T19:18:39.523384+0000","last_became_peered":"2024-08-28T19:18:39.523384+0000","last_unstale":"2024-08-28T19:20:09.615167+0000","last_undegraded":"2024-08-28T19:20:09.615167+0000","last_fullsized":"2024-08-28T19:20:09.615167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:57:32.277680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873180+0000","last_change":"2024-08-28T19:18:32.482257+0000","last_active":"2024-08-28T19:20:37.873180+0000","last_peered":"2024-08-28T19:20:37.873180+0000","last_clean":"2024-08-28T19:20:37.873180+0000","last_became_active":"2024-08-28T19:18:32.481884+0000","last_became_peered":"2024-08-28T19:18:32.481884+0000","last_unstale":"2024-08-28T19:20:37.873180+0000","last_undegraded":"2024-08-28T19:20:37.873180+0000","last_fullsized":"2024-08-28T19:20:37.873180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:56:28.754369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":47,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525435+0000","last_change":"2024-08-28T19:18:30.467307+0000","last_active":"2024-08-28T19:20:09.525435+0000","last_peered":"2024-08-28T19:20:09.525435+0000","last_clean":"2024-08-28T19:20:09.525435+0000","last_became_active":"2024-08-28T19:18:30.466942+0000","last_became_peered":"2024-08-28T19:18:30.466942+0000","last_unstale":"2024-08-28T19:20:09.525435+0000","last_undegraded":"2024-08-28T19:20:09.525435+0000","last_fullsized":"2024-08-28T19:20:09.525435+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:05:23.682110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007238+0000","last_change":"2024-08-28T19:18:39.531421+0000","last_active":"2024-08-28T19:20:09.007238+0000","last_peered":"2024-08-28T19:20:09.007238+0000","last_clean":"2024-08-28T19:20:09.007238+0000","last_became_active":"2024-08-28T19:18:39.531008+0000","last_became_peered":"2024-08-28T19:18:39.531008+0000","last_unstale":"2024-08-28T19:20:09.007238+0000","last_undegraded":"2024-08-28T19:20:09.007238+0000","last_fullsized":"2024-08-28T19:20:09.007238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:46:49.580601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868264+0000","last_change":"2024-08-28T19:18:32.485204+0000","last_active":"2024-08-28T19:20:37.868264+0000","last_peered":"2024-08-28T19:20:37.868264+0000","last_clean":"2024-08-28T19:20:37.868264+0000","last_became_active":"2024-08-28T19:18:32.484830+0000","last_became_peered":"2024-08-28T19:18:32.484830+0000","last_unstale":"2024-08-28T19:20:37.868264+0000","last_undegraded":"2024-08-28T19:20:37.868264+0000","last_fullsized":"2024-08-28T19:20:37.868264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:55:41.358718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007905+0000","last_change":"2024-08-28T19:18:31.082361+0000","last_active":"2024-08-28T19:20:09.007905+0000","last_peered":"2024-08-28T19:20:09.007905+0000","last_clean":"2024-08-28T19:20:09.007905+0000","last_became_active":"2024-08-28T19:18:31.081930+0000","last_became_peered":"2024-08-28T19:18:31.081930+0000","last_unstale":"2024-08-28T19:20:09.007905+0000","last_undegraded":"2024-08-28T19:20:09.007905+0000","last_fullsized":"2024-08-28T19:20:09.007905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:41:05.543088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615411+0000","last_change":"2024-08-28T19:18:39.682341+0000","last_active":"2024-08-28T19:20:09.615411+0000","last_peered":"2024-08-28T19:20:09.615411+0000","last_clean":"2024-08-28T19:20:09.615411+0000","last_became_active":"2024-08-28T19:18:39.681751+0000","last_became_peered":"2024-08-28T19:18:39.681751+0000","last_unstale":"2024-08-28T19:20:09.615411+0000","last_undegraded":"2024-08-28T19:20:09.615411+0000","last_fullsized":"2024-08-28T19:20:09.615411+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:51:24.371270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880906+0000","last_change":"2024-08-28T19:18:32.491069+0000","last_active":"2024-08-28T19:20:37.880906+0000","last_peered":"2024-08-28T19:20:37.880906+0000","last_clean":"2024-08-28T19:20:37.880906+0000","last_became_active":"2024-08-28T19:18:32.490688+0000","last_became_peered":"2024-08-28T19:18:32.490688+0000","last_unstale":"2024-08-28T19:20:37.880906+0000","last_undegraded":"2024-08-28T19:20:37.880906+0000","last_fullsized":"2024-08-28T19:20:37.880906+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:24:31.560519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615273+0000","last_change":"2024-08-28T19:18:30.739812+0000","last_active":"2024-08-28T19:20:09.615273+0000","last_peered":"2024-08-28T19:20:09.615273+0000","last_clean":"2024-08-28T19:20:09.615273+0000","last_became_active":"2024-08-28T19:18:30.739414+0000","last_became_peered":"2024-08-28T19:18:30.739414+0000","last_unstale":"2024-08-28T19:20:09.615273+0000","last_undegraded":"2024-08-28T19:20:09.615273+0000","last_fullsized":"2024-08-28T19:20:09.615273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:56:56.594993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010962+0000","last_change":"2024-08-28T19:18:39.528281+0000","last_active":"2024-08-28T19:20:09.010962+0000","last_peered":"2024-08-28T19:20:09.010962+0000","last_clean":"2024-08-28T19:20:09.010962+0000","last_became_active":"2024-08-28T19:18:39.527142+0000","last_became_peered":"2024-08-28T19:18:39.527142+0000","last_unstale":"2024-08-28T19:20:09.010962+0000","last_undegraded":"2024-08-28T19:20:09.010962+0000","last_fullsized":"2024-08-28T19:20:09.010962+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:44:19.123662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872571+0000","last_change":"2024-08-28T19:18:32.490430+0000","last_active":"2024-08-28T19:20:37.872571+0000","last_peered":"2024-08-28T19:20:37.872571+0000","last_clean":"2024-08-28T19:20:37.872571+0000","last_became_active":"2024-08-28T19:18:32.490020+0000","last_became_peered":"2024-08-28T19:18:32.490020+0000","last_unstale":"2024-08-28T19:20:37.872571+0000","last_undegraded":"2024-08-28T19:20:37.872571+0000","last_fullsized":"2024-08-28T19:20:37.872571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:52:31.153012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'17","reported_seq":57,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:41.222712+0000","last_change":"2024-08-28T19:18:30.467060+0000","last_active":"2024-08-28T19:20:41.222712+0000","last_peered":"2024-08-28T19:20:41.222712+0000","last_clean":"2024-08-28T19:20:41.222712+0000","last_became_active":"2024-08-28T19:18:30.466689+0000","last_became_peered":"2024-08-28T19:18:30.466689+0000","last_unstale":"2024-08-28T19:20:41.222712+0000","last_undegraded":"2024-08-28T19:20:41.222712+0000","last_fullsized":"2024-08-28T19:20:41.222712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:18:36.508131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.524970+0000","last_change":"2024-08-28T19:18:39.682874+0000","last_active":"2024-08-28T19:20:09.524970+0000","last_peered":"2024-08-28T19:20:09.524970+0000","last_clean":"2024-08-28T19:20:09.524970+0000","last_became_active":"2024-08-28T19:18:39.681500+0000","last_became_peered":"2024-08-28T19:18:39.681500+0000","last_unstale":"2024-08-28T19:20:09.524970+0000","last_undegraded":"2024-08-28T19:20:09.524970+0000","last_fullsized":"2024-08-28T19:20:09.524970+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:45:41.815506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.856367+0000","last_change":"2024-08-28T19:18:32.482047+0000","last_active":"2024-08-28T19:20:37.856367+0000","last_peered":"2024-08-28T19:20:37.856367+0000","last_clean":"2024-08-28T19:20:37.856367+0000","last_became_active":"2024-08-28T19:18:32.481640+0000","last_became_peered":"2024-08-28T19:18:32.481640+0000","last_unstale":"2024-08-28T19:20:37.856367+0000","last_undegraded":"2024-08-28T19:20:37.856367+0000","last_fullsized":"2024-08-28T19:20:37.856367+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:34.717764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110454+0000","last_change":"2024-08-28T19:18:31.083524+0000","last_active":"2024-08-28T19:20:09.110454+0000","last_peered":"2024-08-28T19:20:09.110454+0000","last_clean":"2024-08-28T19:20:09.110454+0000","last_became_active":"2024-08-28T19:18:31.083106+0000","last_became_peered":"2024-08-28T19:18:31.083106+0000","last_unstale":"2024-08-28T19:20:09.110454+0000","last_undegraded":"2024-08-28T19:20:09.110454+0000","last_fullsized":"2024-08-28T19:20:09.110454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:16.180754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663607+0000","last_change":"2024-08-28T19:18:39.531884+0000","last_active":"2024-08-28T19:20:09.663607+0000","last_peered":"2024-08-28T19:20:09.663607+0000","last_clean":"2024-08-28T19:20:09.663607+0000","last_became_active":"2024-08-28T19:18:39.531255+0000","last_became_peered":"2024-08-28T19:18:39.531255+0000","last_unstale":"2024-08-28T19:20:09.663607+0000","last_undegraded":"2024-08-28T19:20:09.663607+0000","last_fullsized":"2024-08-28T19:20:09.663607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:36:55.496594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.862241+0000","last_change":"2024-08-28T19:18:32.485180+0000","last_active":"2024-08-28T19:20:37.862241+0000","last_peered":"2024-08-28T19:20:37.862241+0000","last_clean":"2024-08-28T19:20:37.862241+0000","last_became_active":"2024-08-28T19:18:32.484804+0000","last_became_peered":"2024-08-28T19:18:32.484804+0000","last_unstale":"2024-08-28T19:20:37.862241+0000","last_undegraded":"2024-08-28T19:20:37.862241+0000","last_fullsized":"2024-08-28T19:20:37.862241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:12:40.958251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007361+0000","last_change":"2024-08-28T19:18:31.082226+0000","last_active":"2024-08-28T19:20:09.007361+0000","last_peered":"2024-08-28T19:20:09.007361+0000","last_clean":"2024-08-28T19:20:09.007361+0000","last_became_active":"2024-08-28T19:18:31.081833+0000","last_became_peered":"2024-08-28T19:18:31.081833+0000","last_unstale":"2024-08-28T19:20:09.007361+0000","last_undegraded":"2024-08-28T19:20:09.007361+0000","last_fullsized":"2024-08-28T19:20:09.007361+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:33:56.298093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109762+0000","last_change":"2024-08-28T19:18:39.681288+0000","last_active":"2024-08-28T19:20:09.109762+0000","last_peered":"2024-08-28T19:20:09.109762+0000","last_clean":"2024-08-28T19:20:09.109762+0000","last_became_active":"2024-08-28T19:18:39.680719+0000","last_became_peered":"2024-08-28T19:18:39.680719+0000","last_unstale":"2024-08-28T19:20:09.109762+0000","last_undegraded":"2024-08-28T19:20:09.109762+0000","last_fullsized":"2024-08-28T19:20:09.109762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:53:35.554181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880497+0000","last_change":"2024-08-28T19:18:32.487105+0000","last_active":"2024-08-28T19:20:37.880497+0000","last_peered":"2024-08-28T19:20:37.880497+0000","last_clean":"2024-08-28T19:20:37.880497+0000","last_became_active":"2024-08-28T19:18:32.486643+0000","last_became_peered":"2024-08-28T19:18:32.486643+0000","last_unstale":"2024-08-28T19:20:37.880497+0000","last_undegraded":"2024-08-28T19:20:37.880497+0000","last_fullsized":"2024-08-28T19:20:37.880497+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:18:43.274493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"84'7","reported_seq":153,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:53.805823+0000","last_change":"2024-08-28T19:18:30.471447+0000","last_active":"2024-08-28T19:20:53.805823+0000","last_peered":"2024-08-28T19:20:53.805823+0000","last_clean":"2024-08-28T19:20:53.805823+0000","last_became_active":"2024-08-28T19:18:30.470798+0000","last_became_peered":"2024-08-28T19:18:30.470798+0000","last_unstale":"2024-08-28T19:20:53.805823+0000","last_undegraded":"2024-08-28T19:20:53.805823+0000","last_fullsized":"2024-08-28T19:20:53.805823+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:50:13.948439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162958+0000","last_change":"2024-08-28T19:18:39.524190+0000","last_active":"2024-08-28T19:20:09.162958+0000","last_peered":"2024-08-28T19:20:09.162958+0000","last_clean":"2024-08-28T19:20:09.162958+0000","last_became_active":"2024-08-28T19:18:39.523771+0000","last_became_peered":"2024-08-28T19:18:39.523771+0000","last_unstale":"2024-08-28T19:20:09.162958+0000","last_undegraded":"2024-08-28T19:20:09.162958+0000","last_fullsized":"2024-08-28T19:20:09.162958+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:43:43.984625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.874995+0000","last_change":"2024-08-28T19:18:32.487212+0000","last_active":"2024-08-28T19:20:37.874995+0000","last_peered":"2024-08-28T19:20:37.874995+0000","last_clean":"2024-08-28T19:20:37.874995+0000","last_became_active":"2024-08-28T19:18:32.485432+0000","last_became_peered":"2024-08-28T19:18:32.485432+0000","last_unstale":"2024-08-28T19:20:37.874995+0000","last_undegraded":"2024-08-28T19:20:37.874995+0000","last_fullsized":"2024-08-28T19:20:37.874995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:52:11.672193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":149,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:53.805226+0000","last_change":"2024-08-28T19:18:31.082210+0000","last_active":"2024-08-28T19:20:53.805226+0000","last_peered":"2024-08-28T19:20:53.805226+0000","last_clean":"2024-08-28T19:20:53.805226+0000","last_became_active":"2024-08-28T19:18:31.081814+0000","last_became_peered":"2024-08-28T19:18:31.081814+0000","last_unstale":"2024-08-28T19:20:53.805226+0000","last_undegraded":"2024-08-28T19:20:53.805226+0000","last_fullsized":"2024-08-28T19:20:53.805226+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:33:32.281431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163227+0000","last_change":"2024-08-28T19:18:39.681858+0000","last_active":"2024-08-28T19:20:09.163227+0000","last_peered":"2024-08-28T19:20:09.163227+0000","last_clean":"2024-08-28T19:20:09.163227+0000","last_became_active":"2024-08-28T19:18:39.681382+0000","last_became_peered":"2024-08-28T19:18:39.681382+0000","last_unstale":"2024-08-28T19:20:09.163227+0000","last_undegraded":"2024-08-28T19:20:09.163227+0000","last_fullsized":"2024-08-28T19:20:09.163227+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:23:22.021022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.857639+0000","last_change":"2024-08-28T19:18:32.486802+0000","last_active":"2024-08-28T19:20:37.857639+0000","last_peered":"2024-08-28T19:20:37.857639+0000","last_clean":"2024-08-28T19:20:37.857639+0000","last_became_active":"2024-08-28T19:18:32.486479+0000","last_became_peered":"2024-08-28T19:18:32.486479+0000","last_unstale":"2024-08-28T19:20:37.857639+0000","last_undegraded":"2024-08-28T19:20:37.857639+0000","last_fullsized":"2024-08-28T19:20:37.857639+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:17:17.654757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529084+0000","last_change":"2024-08-28T19:18:30.470447+0000","last_active":"2024-08-28T19:20:09.529084+0000","last_peered":"2024-08-28T19:20:09.529084+0000","last_clean":"2024-08-28T19:20:09.529084+0000","last_became_active":"2024-08-28T19:18:30.469862+0000","last_became_peered":"2024-08-28T19:18:30.469862+0000","last_unstale":"2024-08-28T19:20:09.529084+0000","last_undegraded":"2024-08-28T19:20:09.529084+0000","last_fullsized":"2024-08-28T19:20:09.529084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:27:46.660492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009270+0000","last_change":"2024-08-28T19:19:08.624635+0000","last_active":"2024-08-28T19:20:09.009270+0000","last_peered":"2024-08-28T19:20:09.009270+0000","last_clean":"2024-08-28T19:20:09.009270+0000","last_became_active":"2024-08-28T19:19:08.622845+0000","last_became_peered":"2024-08-28T19:19:08.622845+0000","last_unstale":"2024-08-28T19:20:09.009270+0000","last_undegraded":"2024-08-28T19:20:09.009270+0000","last_fullsized":"2024-08-28T19:20:09.009270+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:56:40.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865852+0000","last_change":"2024-08-28T19:18:32.487947+0000","last_active":"2024-08-28T19:20:37.865852+0000","last_peered":"2024-08-28T19:20:37.865852+0000","last_clean":"2024-08-28T19:20:37.865852+0000","last_became_active":"2024-08-28T19:18:32.487592+0000","last_became_peered":"2024-08-28T19:18:32.487592+0000","last_unstale":"2024-08-28T19:20:37.865852+0000","last_undegraded":"2024-08-28T19:20:37.865852+0000","last_fullsized":"2024-08-28T19:20:37.865852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:30:29.550644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615576+0000","last_change":"2024-08-28T19:18:30.465044+0000","last_active":"2024-08-28T19:20:09.615576+0000","last_peered":"2024-08-28T19:20:09.615576+0000","last_clean":"2024-08-28T19:20:09.615576+0000","last_became_active":"2024-08-28T19:18:30.464580+0000","last_became_peered":"2024-08-28T19:18:30.464580+0000","last_unstale":"2024-08-28T19:20:09.615576+0000","last_undegraded":"2024-08-28T19:20:09.615576+0000","last_fullsized":"2024-08-28T19:20:09.615576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:21:17.223311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":114,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.528964+0000","last_change":"2024-08-28T19:17:17.063316+0000","last_active":"2024-08-28T19:20:09.528964+0000","last_peered":"2024-08-28T19:20:09.528964+0000","last_clean":"2024-08-28T19:20:09.528964+0000","last_became_active":"2024-08-28T19:17:17.057091+0000","last_became_peered":"2024-08-28T19:17:17.057091+0000","last_unstale":"2024-08-28T19:20:09.528964+0000","last_undegraded":"2024-08-28T19:20:09.528964+0000","last_fullsized":"2024-08-28T19:20:09.528964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_clean_scrub_stamp":"2024-08-28T19:13:56.815467+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:35:51.499703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"82'1","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529202+0000","last_change":"2024-08-28T19:19:08.621402+0000","last_active":"2024-08-28T19:20:09.529202+0000","last_peered":"2024-08-28T19:20:09.529202+0000","last_clean":"2024-08-28T19:20:09.529202+0000","last_became_active":"2024-08-28T19:19:08.620779+0000","last_became_peered":"2024-08-28T19:19:08.620779+0000","last_unstale":"2024-08-28T19:20:09.529202+0000","last_undegraded":"2024-08-28T19:20:09.529202+0000","last_fullsized":"2024-08-28T19:20:09.529202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:39.391414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886997+0000","last_change":"2024-08-28T19:18:32.490066+0000","last_active":"2024-08-28T19:20:37.886997+0000","last_peered":"2024-08-28T19:20:37.886997+0000","last_clean":"2024-08-28T19:20:37.886997+0000","last_became_active":"2024-08-28T19:18:32.489560+0000","last_became_peered":"2024-08-28T19:18:32.489560+0000","last_unstale":"2024-08-28T19:20:37.886997+0000","last_undegraded":"2024-08-28T19:20:37.886997+0000","last_fullsized":"2024-08-28T19:20:37.886997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:22:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'5","reported_seq":151,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.851167+0000","last_change":"2024-08-28T19:18:31.083078+0000","last_active":"2024-08-28T19:20:54.851167+0000","last_peered":"2024-08-28T19:20:54.851167+0000","last_clean":"2024-08-28T19:20:54.851167+0000","last_became_active":"2024-08-28T19:18:31.082575+0000","last_became_peered":"2024-08-28T19:18:31.082575+0000","last_unstale":"2024-08-28T19:20:54.851167+0000","last_undegraded":"2024-08-28T19:20:54.851167+0000","last_fullsized":"2024-08-28T19:20:54.851167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007862+0000","last_change":"2024-08-28T19:18:39.519814+0000","last_active":"2024-08-28T19:20:09.007862+0000","last_peered":"2024-08-28T19:20:09.007862+0000","last_clean":"2024-08-28T19:20:09.007862+0000","last_became_active":"2024-08-28T19:18:39.519258+0000","last_became_peered":"2024-08-28T19:18:39.519258+0000","last_unstale":"2024-08-28T19:20:09.007862+0000","last_undegraded":"2024-08-28T19:20:09.007862+0000","last_fullsized":"2024-08-28T19:20:09.007862+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:03:04.051376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871177+0000","last_change":"2024-08-28T19:18:32.495426+0000","last_active":"2024-08-28T19:20:37.871177+0000","last_peered":"2024-08-28T19:20:37.871177+0000","last_clean":"2024-08-28T19:20:37.871177+0000","last_became_active":"2024-08-28T19:18:32.495075+0000","last_became_peered":"2024-08-28T19:18:32.495075+0000","last_unstale":"2024-08-28T19:20:37.871177+0000","last_undegraded":"2024-08-28T19:20:37.871177+0000","last_fullsized":"2024-08-28T19:20:37.871177+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:40:22.456666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009115+0000","last_change":"2024-08-28T19:18:30.472708+0000","last_active":"2024-08-28T19:20:09.009115+0000","last_peered":"2024-08-28T19:20:09.009115+0000","last_clean":"2024-08-28T19:20:09.009115+0000","last_became_active":"2024-08-28T19:18:30.472320+0000","last_became_peered":"2024-08-28T19:18:30.472320+0000","last_unstale":"2024-08-28T19:20:09.009115+0000","last_undegraded":"2024-08-28T19:20:09.009115+0000","last_fullsized":"2024-08-28T19:20:09.009115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:57:12.055626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010493+0000","last_change":"2024-08-28T19:18:39.519398+0000","last_active":"2024-08-28T19:20:09.010493+0000","last_peered":"2024-08-28T19:20:09.010493+0000","last_clean":"2024-08-28T19:20:09.010493+0000","last_became_active":"2024-08-28T19:18:39.518997+0000","last_became_peered":"2024-08-28T19:18:39.518997+0000","last_unstale":"2024-08-28T19:20:09.010493+0000","last_undegraded":"2024-08-28T19:20:09.010493+0000","last_fullsized":"2024-08-28T19:20:09.010493+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:33.431608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.879604+0000","last_change":"2024-08-28T19:18:32.489439+0000","last_active":"2024-08-28T19:20:37.879604+0000","last_peered":"2024-08-28T19:20:37.879604+0000","last_clean":"2024-08-28T19:20:37.879604+0000","last_became_active":"2024-08-28T19:18:32.489054+0000","last_became_peered":"2024-08-28T19:18:32.489054+0000","last_unstale":"2024-08-28T19:20:37.879604+0000","last_undegraded":"2024-08-28T19:20:37.879604+0000","last_fullsized":"2024-08-28T19:20:37.879604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:31:32.647384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163358+0000","last_change":"2024-08-28T19:18:30.464993+0000","last_active":"2024-08-28T19:20:09.163358+0000","last_peered":"2024-08-28T19:20:09.163358+0000","last_clean":"2024-08-28T19:20:09.163358+0000","last_became_active":"2024-08-28T19:18:30.463951+0000","last_became_peered":"2024-08-28T19:18:30.463951+0000","last_unstale":"2024-08-28T19:20:09.163358+0000","last_undegraded":"2024-08-28T19:20:09.163358+0000","last_fullsized":"2024-08-28T19:20:09.163358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:06.780058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008234+0000","last_change":"2024-08-28T19:18:39.533061+0000","last_active":"2024-08-28T19:20:09.008234+0000","last_peered":"2024-08-28T19:20:09.008234+0000","last_clean":"2024-08-28T19:20:09.008234+0000","last_became_active":"2024-08-28T19:18:39.532607+0000","last_became_peered":"2024-08-28T19:18:39.532607+0000","last_unstale":"2024-08-28T19:20:09.008234+0000","last_undegraded":"2024-08-28T19:20:09.008234+0000","last_fullsized":"2024-08-28T19:20:09.008234+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:00:57.997665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875549+0000","last_change":"2024-08-28T19:18:32.487937+0000","last_active":"2024-08-28T19:20:37.875549+0000","last_peered":"2024-08-28T19:20:37.875549+0000","last_clean":"2024-08-28T19:20:37.875549+0000","last_became_active":"2024-08-28T19:18:32.487417+0000","last_became_peered":"2024-08-28T19:18:32.487417+0000","last_unstale":"2024-08-28T19:20:37.875549+0000","last_undegraded":"2024-08-28T19:20:37.875549+0000","last_fullsized":"2024-08-28T19:20:37.875549+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:42.497364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110517+0000","last_change":"2024-08-28T19:18:30.470742+0000","last_active":"2024-08-28T19:20:09.110517+0000","last_peered":"2024-08-28T19:20:09.110517+0000","last_clean":"2024-08-28T19:20:09.110517+0000","last_became_active":"2024-08-28T19:18:30.470135+0000","last_became_peered":"2024-08-28T19:18:30.470135+0000","last_unstale":"2024-08-28T19:20:09.110517+0000","last_undegraded":"2024-08-28T19:20:09.110517+0000","last_fullsized":"2024-08-28T19:20:09.110517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:49:39.581119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008429+0000","last_change":"2024-08-28T19:18:39.524052+0000","last_active":"2024-08-28T19:20:09.008429+0000","last_peered":"2024-08-28T19:20:09.008429+0000","last_clean":"2024-08-28T19:20:09.008429+0000","last_became_active":"2024-08-28T19:18:39.523601+0000","last_became_peered":"2024-08-28T19:18:39.523601+0000","last_unstale":"2024-08-28T19:20:09.008429+0000","last_undegraded":"2024-08-28T19:20:09.008429+0000","last_fullsized":"2024-08-28T19:20:09.008429+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:54:53.302424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883838+0000","last_change":"2024-08-28T19:18:32.491151+0000","last_active":"2024-08-28T19:20:37.883838+0000","last_peered":"2024-08-28T19:20:37.883838+0000","last_clean":"2024-08-28T19:20:37.883838+0000","last_became_active":"2024-08-28T19:18:32.490798+0000","last_became_peered":"2024-08-28T19:18:32.490798+0000","last_unstale":"2024-08-28T19:20:37.883838+0000","last_undegraded":"2024-08-28T19:20:37.883838+0000","last_fullsized":"2024-08-28T19:20:37.883838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:41:00.771019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109743+0000","last_change":"2024-08-28T19:18:31.084029+0000","last_active":"2024-08-28T19:20:09.109743+0000","last_peered":"2024-08-28T19:20:09.109743+0000","last_clean":"2024-08-28T19:20:09.109743+0000","last_became_active":"2024-08-28T19:18:31.083507+0000","last_became_peered":"2024-08-28T19:18:31.083507+0000","last_unstale":"2024-08-28T19:20:09.109743+0000","last_undegraded":"2024-08-28T19:20:09.109743+0000","last_fullsized":"2024-08-28T19:20:09.109743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:15:36.451702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163253+0000","last_change":"2024-08-28T19:18:39.527249+0000","last_active":"2024-08-28T19:20:09.163253+0000","last_peered":"2024-08-28T19:20:09.163253+0000","last_clean":"2024-08-28T19:20:09.163253+0000","last_became_active":"2024-08-28T19:18:39.526834+0000","last_became_peered":"2024-08-28T19:18:39.526834+0000","last_unstale":"2024-08-28T19:20:09.163253+0000","last_undegraded":"2024-08-28T19:20:09.163253+0000","last_fullsized":"2024-08-28T19:20:09.163253+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:48:02.294738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.876604+0000","last_change":"2024-08-28T19:18:32.497337+0000","last_active":"2024-08-28T19:20:37.876604+0000","last_peered":"2024-08-28T19:20:37.876604+0000","last_clean":"2024-08-28T19:20:37.876604+0000","last_became_active":"2024-08-28T19:18:32.496929+0000","last_became_peered":"2024-08-28T19:18:32.496929+0000","last_unstale":"2024-08-28T19:20:37.876604+0000","last_undegraded":"2024-08-28T19:20:37.876604+0000","last_fullsized":"2024-08-28T19:20:37.876604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:14.748765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":154,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:57.812014+0000","last_change":"2024-08-28T19:18:30.471353+0000","last_active":"2024-08-28T19:20:57.812014+0000","last_peered":"2024-08-28T19:20:57.812014+0000","last_clean":"2024-08-28T19:20:57.812014+0000","last_became_active":"2024-08-28T19:18:30.470283+0000","last_became_peered":"2024-08-28T19:18:30.470283+0000","last_unstale":"2024-08-28T19:20:57.812014+0000","last_undegraded":"2024-08-28T19:20:57.812014+0000","last_fullsized":"2024-08-28T19:20:57.812014+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:09:46.444864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":110,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007249+0000","last_change":"2024-08-28T19:18:39.519226+0000","last_active":"2024-08-28T19:20:09.007249+0000","last_peered":"2024-08-28T19:20:09.007249+0000","last_clean":"2024-08-28T19:20:09.007249+0000","last_became_active":"2024-08-28T19:18:39.518738+0000","last_became_peered":"2024-08-28T19:18:39.518738+0000","last_unstale":"2024-08-28T19:20:09.007249+0000","last_undegraded":"2024-08-28T19:20:09.007249+0000","last_fullsized":"2024-08-28T19:20:09.007249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:39:35.114702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.877692+0000","last_change":"2024-08-28T19:18:32.493060+0000","last_active":"2024-08-28T19:20:37.877692+0000","last_peered":"2024-08-28T19:20:37.877692+0000","last_clean":"2024-08-28T19:20:37.877692+0000","last_became_active":"2024-08-28T19:18:32.492590+0000","last_became_peered":"2024-08-28T19:18:32.492590+0000","last_unstale":"2024-08-28T19:20:37.877692+0000","last_undegraded":"2024-08-28T19:20:37.877692+0000","last_fullsized":"2024-08-28T19:20:37.877692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:02:02.165301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008761+0000","last_change":"2024-08-28T19:18:30.470674+0000","last_active":"2024-08-28T19:20:09.008761+0000","last_peered":"2024-08-28T19:20:09.008761+0000","last_clean":"2024-08-28T19:20:09.008761+0000","last_became_active":"2024-08-28T19:18:30.470275+0000","last_became_peered":"2024-08-28T19:18:30.470275+0000","last_unstale":"2024-08-28T19:20:09.008761+0000","last_undegraded":"2024-08-28T19:20:09.008761+0000","last_fullsized":"2024-08-28T19:20:09.008761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:01:48.547315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663504+0000","last_change":"2024-08-28T19:19:08.624194+0000","last_active":"2024-08-28T19:20:09.663504+0000","last_peered":"2024-08-28T19:20:09.663504+0000","last_clean":"2024-08-28T19:20:09.663504+0000","last_became_active":"2024-08-28T19:19:08.623681+0000","last_became_peered":"2024-08-28T19:19:08.623681+0000","last_unstale":"2024-08-28T19:20:09.663504+0000","last_undegraded":"2024-08-28T19:20:09.663504+0000","last_fullsized":"2024-08-28T19:20:09.663504+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:21:11.099931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872115+0000","last_change":"2024-08-28T19:18:32.488210+0000","last_active":"2024-08-28T19:20:37.872115+0000","last_peered":"2024-08-28T19:20:37.872115+0000","last_clean":"2024-08-28T19:20:37.872115+0000","last_became_active":"2024-08-28T19:18:32.487853+0000","last_became_peered":"2024-08-28T19:18:32.487853+0000","last_unstale":"2024-08-28T19:20:37.872115+0000","last_undegraded":"2024-08-28T19:20:37.872115+0000","last_fullsized":"2024-08-28T19:20:37.872115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:20:51.808958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011302+0000","last_change":"2024-08-28T19:18:30.465306+0000","last_active":"2024-08-28T19:20:09.011302+0000","last_peered":"2024-08-28T19:20:09.011302+0000","last_clean":"2024-08-28T19:20:09.011302+0000","last_became_active":"2024-08-28T19:18:30.464850+0000","last_became_peered":"2024-08-28T19:18:30.464850+0000","last_unstale":"2024-08-28T19:20:09.011302+0000","last_undegraded":"2024-08-28T19:20:09.011302+0000","last_fullsized":"2024-08-28T19:20:09.011302+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:24:26.628999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008657+0000","last_change":"2024-08-28T19:18:39.524005+0000","last_active":"2024-08-28T19:20:09.008657+0000","last_peered":"2024-08-28T19:20:09.008657+0000","last_clean":"2024-08-28T19:20:09.008657+0000","last_became_active":"2024-08-28T19:18:39.523594+0000","last_became_peered":"2024-08-28T19:18:39.523594+0000","last_unstale":"2024-08-28T19:20:09.008657+0000","last_undegraded":"2024-08-28T19:20:09.008657+0000","last_fullsized":"2024-08-28T19:20:09.008657+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:35:32.484399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"84'6","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872604+0000","last_change":"2024-08-28T19:18:32.489088+0000","last_active":"2024-08-28T19:20:37.872604+0000","last_peered":"2024-08-28T19:20:37.872604+0000","last_clean":"2024-08-28T19:20:37.872604+0000","last_became_active":"2024-08-28T19:18:32.488717+0000","last_became_peered":"2024-08-28T19:18:32.488717+0000","last_unstale":"2024-08-28T19:20:37.872604+0000","last_undegraded":"2024-08-28T19:20:37.872604+0000","last_fullsized":"2024-08-28T19:20:37.872604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:25.468919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"84'40","reported_seq":297,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.807945+0000","last_change":"2024-08-28T19:18:30.468674+0000","last_active":"2024-08-28T19:20:54.807945+0000","last_peered":"2024-08-28T19:20:54.807945+0000","last_clean":"2024-08-28T19:20:54.807945+0000","last_became_active":"2024-08-28T19:18:30.468300+0000","last_became_peered":"2024-08-28T19:18:30.468300+0000","last_unstale":"2024-08-28T19:20:54.807945+0000","last_undegraded":"2024-08-28T19:20:54.807945+0000","last_fullsized":"2024-08-28T19:20:54.807945+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:12:06.237498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64032,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":217,"num_read_kb":171,"num_write":42,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007081+0000","last_change":"2024-08-28T19:18:39.526131+0000","last_active":"2024-08-28T19:20:09.007081+0000","last_peered":"2024-08-28T19:20:09.007081+0000","last_clean":"2024-08-28T19:20:09.007081+0000","last_became_active":"2024-08-28T19:18:39.525708+0000","last_became_peered":"2024-08-28T19:18:39.525708+0000","last_unstale":"2024-08-28T19:20:09.007081+0000","last_undegraded":"2024-08-28T19:20:09.007081+0000","last_fullsized":"2024-08-28T19:20:09.007081+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:21:52.156487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.885078+0000","last_change":"2024-08-28T19:18:32.487634+0000","last_active":"2024-08-28T19:20:37.885078+0000","last_peered":"2024-08-28T19:20:37.885078+0000","last_clean":"2024-08-28T19:20:37.885078+0000","last_became_active":"2024-08-28T19:18:32.486702+0000","last_became_peered":"2024-08-28T19:18:32.486702+0000","last_unstale":"2024-08-28T19:20:37.885078+0000","last_undegraded":"2024-08-28T19:20:37.885078+0000","last_fullsized":"2024-08-28T19:20:37.885078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:08:12.606905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009190+0000","last_change":"2024-08-28T19:18:31.083494+0000","last_active":"2024-08-28T19:20:09.009190+0000","last_peered":"2024-08-28T19:20:09.009190+0000","last_clean":"2024-08-28T19:20:09.009190+0000","last_became_active":"2024-08-28T19:18:31.083086+0000","last_became_peered":"2024-08-28T19:18:31.083086+0000","last_unstale":"2024-08-28T19:20:09.009190+0000","last_undegraded":"2024-08-28T19:20:09.009190+0000","last_fullsized":"2024-08-28T19:20:09.009190+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:14:54.388364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615438+0000","last_change":"2024-08-28T19:18:39.532155+0000","last_active":"2024-08-28T19:20:09.615438+0000","last_peered":"2024-08-28T19:20:09.615438+0000","last_clean":"2024-08-28T19:20:09.615438+0000","last_became_active":"2024-08-28T19:18:39.531565+0000","last_became_peered":"2024-08-28T19:18:39.531565+0000","last_unstale":"2024-08-28T19:20:09.615438+0000","last_undegraded":"2024-08-28T19:20:09.615438+0000","last_fullsized":"2024-08-28T19:20:09.615438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:29:48.824622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.881019+0000","last_change":"2024-08-28T19:18:32.496296+0000","last_active":"2024-08-28T19:20:37.881019+0000","last_peered":"2024-08-28T19:20:37.881019+0000","last_clean":"2024-08-28T19:20:37.881019+0000","last_became_active":"2024-08-28T19:18:32.495680+0000","last_became_peered":"2024-08-28T19:18:32.495680+0000","last_unstale":"2024-08-28T19:20:37.881019+0000","last_undegraded":"2024-08-28T19:20:37.881019+0000","last_fullsized":"2024-08-28T19:20:37.881019+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:27:34.544529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529311+0000","last_change":"2024-08-28T19:19:08.631758+0000","last_active":"2024-08-28T19:20:09.529311+0000","last_peered":"2024-08-28T19:20:09.529311+0000","last_clean":"2024-08-28T19:20:09.529311+0000","last_became_active":"2024-08-28T19:19:08.631202+0000","last_became_peered":"2024-08-28T19:19:08.631202+0000","last_unstale":"2024-08-28T19:20:09.529311+0000","last_undegraded":"2024-08-28T19:20:09.529311+0000","last_fullsized":"2024-08-28T19:20:09.529311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:27:43.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.006719+0000","last_change":"2024-08-28T19:18:39.531157+0000","last_active":"2024-08-28T19:20:09.006719+0000","last_peered":"2024-08-28T19:20:09.006719+0000","last_clean":"2024-08-28T19:20:09.006719+0000","last_became_active":"2024-08-28T19:18:39.530675+0000","last_became_peered":"2024-08-28T19:18:39.530675+0000","last_unstale":"2024-08-28T19:20:09.006719+0000","last_undegraded":"2024-08-28T19:20:09.006719+0000","last_fullsized":"2024-08-28T19:20:09.006719+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:38:40.414854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"84'10","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883543+0000","last_change":"2024-08-28T19:18:32.482182+0000","last_active":"2024-08-28T19:20:37.883543+0000","last_peered":"2024-08-28T19:20:37.883543+0000","last_clean":"2024-08-28T19:20:37.883543+0000","last_became_active":"2024-08-28T19:18:32.481689+0000","last_became_peered":"2024-08-28T19:18:32.481689+0000","last_unstale":"2024-08-28T19:20:37.883543+0000","last_undegraded":"2024-08-28T19:20:37.883543+0000","last_fullsized":"2024-08-28T19:20:37.883543+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:13:29.278469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'5","reported_seq":251,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:54.850315+0000","last_change":"2024-08-28T19:18:30.475244+0000","last_active":"2024-08-28T19:20:54.850315+0000","last_peered":"2024-08-28T19:20:54.850315+0000","last_clean":"2024-08-28T19:20:54.850315+0000","last_became_active":"2024-08-28T19:18:30.474861+0000","last_became_peered":"2024-08-28T19:18:30.474861+0000","last_unstale":"2024-08-28T19:20:54.850315+0000","last_undegraded":"2024-08-28T19:20:54.850315+0000","last_fullsized":"2024-08-28T19:20:54.850315+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:27:09.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":206,"num_read_kb":206,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525040+0000","last_change":"2024-08-28T19:18:39.520022+0000","last_active":"2024-08-28T19:20:09.525040+0000","last_peered":"2024-08-28T19:20:09.525040+0000","last_clean":"2024-08-28T19:20:09.525040+0000","last_became_active":"2024-08-28T19:18:39.519584+0000","last_became_peered":"2024-08-28T19:18:39.519584+0000","last_unstale":"2024-08-28T19:20:09.525040+0000","last_undegraded":"2024-08-28T19:20:09.525040+0000","last_fullsized":"2024-08-28T19:20:09.525040+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:27:15.851303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010363+0000","last_change":"2024-08-28T19:18:39.522607+0000","last_active":"2024-08-28T19:20:09.010363+0000","last_peered":"2024-08-28T19:20:09.010363+0000","last_clean":"2024-08-28T19:20:09.010363+0000","last_became_active":"2024-08-28T19:18:39.522204+0000","last_became_peered":"2024-08-28T19:18:39.522204+0000","last_unstale":"2024-08-28T19:20:09.010363+0000","last_undegraded":"2024-08-28T19:20:09.010363+0000","last_fullsized":"2024-08-28T19:20:09.010363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:02:08.865294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"84'11","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865503+0000","last_change":"2024-08-28T19:18:32.485577+0000","last_active":"2024-08-28T19:20:37.865503+0000","last_peered":"2024-08-28T19:20:37.865503+0000","last_clean":"2024-08-28T19:20:37.865503+0000","last_became_active":"2024-08-28T19:18:32.485243+0000","last_became_peered":"2024-08-28T19:18:32.485243+0000","last_unstale":"2024-08-28T19:20:37.865503+0000","last_undegraded":"2024-08-28T19:20:37.865503+0000","last_fullsized":"2024-08-28T19:20:37.865503+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:05:53.019365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008799+0000","last_change":"2024-08-28T19:18:30.470670+0000","last_active":"2024-08-28T19:20:09.008799+0000","last_peered":"2024-08-28T19:20:09.008799+0000","last_clean":"2024-08-28T19:20:09.008799+0000","last_became_active":"2024-08-28T19:18:30.470264+0000","last_became_peered":"2024-08-28T19:18:30.470264+0000","last_unstale":"2024-08-28T19:20:09.008799+0000","last_undegraded":"2024-08-28T19:20:09.008799+0000","last_fullsized":"2024-08-28T19:20:09.008799+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:35.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663847+0000","last_change":"2024-08-28T19:18:39.530948+0000","last_active":"2024-08-28T19:20:09.663847+0000","last_peered":"2024-08-28T19:20:09.663847+0000","last_clean":"2024-08-28T19:20:09.663847+0000","last_became_active":"2024-08-28T19:18:39.530473+0000","last_became_peered":"2024-08-28T19:18:39.530473+0000","last_unstale":"2024-08-28T19:20:09.663847+0000","last_undegraded":"2024-08-28T19:20:09.663847+0000","last_fullsized":"2024-08-28T19:20:09.663847+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:21:57.504582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"84'8","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.869758+0000","last_change":"2024-08-28T19:18:32.486027+0000","last_active":"2024-08-28T19:20:37.869758+0000","last_peered":"2024-08-28T19:20:37.869758+0000","last_clean":"2024-08-28T19:20:37.869758+0000","last_became_active":"2024-08-28T19:18:32.483273+0000","last_became_peered":"2024-08-28T19:18:32.483273+0000","last_unstale":"2024-08-28T19:20:37.869758+0000","last_undegraded":"2024-08-28T19:20:37.869758+0000","last_fullsized":"2024-08-28T19:20:37.869758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:47:37.680695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162978+0000","last_change":"2024-08-28T19:18:30.464811+0000","last_active":"2024-08-28T19:20:09.162978+0000","last_peered":"2024-08-28T19:20:09.162978+0000","last_clean":"2024-08-28T19:20:09.162978+0000","last_became_active":"2024-08-28T19:18:30.463986+0000","last_became_peered":"2024-08-28T19:18:30.463986+0000","last_unstale":"2024-08-28T19:20:09.162978+0000","last_undegraded":"2024-08-28T19:20:09.162978+0000","last_fullsized":"2024-08-28T19:20:09.162978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:43:46.582993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":53,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663544+0000","last_change":"2024-08-28T19:19:08.631382+0000","last_active":"2024-08-28T19:20:09.663544+0000","last_peered":"2024-08-28T19:20:09.663544+0000","last_clean":"2024-08-28T19:20:09.663544+0000","last_became_active":"2024-08-28T19:19:08.630869+0000","last_became_peered":"2024-08-28T19:19:08.630869+0000","last_unstale":"2024-08-28T19:20:09.663544+0000","last_undegraded":"2024-08-28T19:20:09.663544+0000","last_fullsized":"2024-08-28T19:20:09.663544+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:32:54.942408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"84'8","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875341+0000","last_change":"2024-08-28T19:18:32.484120+0000","last_active":"2024-08-28T19:20:37.875341+0000","last_peered":"2024-08-28T19:20:37.875341+0000","last_clean":"2024-08-28T19:20:37.875341+0000","last_became_active":"2024-08-28T19:18:32.483013+0000","last_became_peered":"2024-08-28T19:18:32.483013+0000","last_unstale":"2024-08-28T19:20:37.875341+0000","last_undegraded":"2024-08-28T19:20:37.875341+0000","last_fullsized":"2024-08-28T19:20:37.875341+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:59:41.195873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010737+0000","last_change":"2024-08-28T19:18:39.531046+0000","last_active":"2024-08-28T19:20:09.010737+0000","last_peered":"2024-08-28T19:20:09.010737+0000","last_clean":"2024-08-28T19:20:09.010737+0000","last_became_active":"2024-08-28T19:18:39.530407+0000","last_became_peered":"2024-08-28T19:18:39.530407+0000","last_unstale":"2024-08-28T19:20:09.010737+0000","last_undegraded":"2024-08-28T19:20:09.010737+0000","last_fullsized":"2024-08-28T19:20:09.010737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:47:15.173021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'7","reported_seq":156,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:56.811024+0000","last_change":"2024-08-28T19:18:30.471268+0000","last_active":"2024-08-28T19:20:56.811024+0000","last_peered":"2024-08-28T19:20:56.811024+0000","last_clean":"2024-08-28T19:20:56.811024+0000","last_became_active":"2024-08-28T19:18:30.470845+0000","last_became_peered":"2024-08-28T19:18:30.470845+0000","last_unstale":"2024-08-28T19:20:56.811024+0000","last_undegraded":"2024-08-28T19:20:56.811024+0000","last_fullsized":"2024-08-28T19:20:56.811024+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:18:18.086205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"84'7","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.859389+0000","last_change":"2024-08-28T19:18:32.479728+0000","last_active":"2024-08-28T19:20:37.859389+0000","last_peered":"2024-08-28T19:20:37.859389+0000","last_clean":"2024-08-28T19:20:37.859389+0000","last_became_active":"2024-08-28T19:18:32.478623+0000","last_became_peered":"2024-08-28T19:18:32.478623+0000","last_unstale":"2024-08-28T19:20:37.859389+0000","last_undegraded":"2024-08-28T19:20:37.859389+0000","last_fullsized":"2024-08-28T19:20:37.859389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:14:49.914808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011224+0000","last_change":"2024-08-28T19:18:39.531308+0000","last_active":"2024-08-28T19:20:09.011224+0000","last_peered":"2024-08-28T19:20:09.011224+0000","last_clean":"2024-08-28T19:20:09.011224+0000","last_became_active":"2024-08-28T19:18:39.530403+0000","last_became_peered":"2024-08-28T19:18:39.530403+0000","last_unstale":"2024-08-28T19:20:09.011224+0000","last_undegraded":"2024-08-28T19:20:09.011224+0000","last_fullsized":"2024-08-28T19:20:09.011224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:33.139668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873003+0000","last_change":"2024-08-28T19:18:32.494621+0000","last_active":"2024-08-28T19:20:37.873003+0000","last_peered":"2024-08-28T19:20:37.873003+0000","last_clean":"2024-08-28T19:20:37.873003+0000","last_became_active":"2024-08-28T19:18:32.494217+0000","last_became_peered":"2024-08-28T19:18:32.494217+0000","last_unstale":"2024-08-28T19:20:37.873003+0000","last_undegraded":"2024-08-28T19:20:37.873003+0000","last_fullsized":"2024-08-28T19:20:37.873003+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:54:15.419470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"84'12","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872043+0000","last_change":"2024-08-28T19:20:09.616552+0000","last_active":"2024-08-28T19:20:37.872043+0000","last_peered":"2024-08-28T19:20:37.872043+0000","last_clean":"2024-08-28T19:20:37.872043+0000","last_became_active":"2024-08-28T19:20:09.616159+0000","last_became_peered":"2024-08-28T19:20:09.616159+0000","last_unstale":"2024-08-28T19:20:37.872043+0000","last_undegraded":"2024-08-28T19:20:37.872043+0000","last_fullsized":"2024-08-28T19:20:37.872043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:04:10.354703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"84'8","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866959+0000","last_change":"2024-08-28T19:20:09.012824+0000","last_active":"2024-08-28T19:20:37.866959+0000","last_peered":"2024-08-28T19:20:37.866959+0000","last_clean":"2024-08-28T19:20:37.866959+0000","last_became_active":"2024-08-28T19:20:09.012490+0000","last_became_peered":"2024-08-28T19:20:09.012490+0000","last_unstale":"2024-08-28T19:20:37.866959+0000","last_undegraded":"2024-08-28T19:20:37.866959+0000","last_fullsized":"2024-08-28T19:20:37.866959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:49:02.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871666+0000","last_change":"2024-08-28T19:20:09.164523+0000","last_active":"2024-08-28T19:20:37.871666+0000","last_peered":"2024-08-28T19:20:37.871666+0000","last_clean":"2024-08-28T19:20:37.871666+0000","last_became_active":"2024-08-28T19:20:09.163825+0000","last_became_peered":"2024-08-28T19:20:09.163825+0000","last_unstale":"2024-08-28T19:20:37.871666+0000","last_undegraded":"2024-08-28T19:20:37.871666+0000","last_fullsized":"2024-08-28T19:20:37.871666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:42:48.141103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868174+0000","last_change":"2024-08-28T19:18:32.485446+0000","last_active":"2024-08-28T19:20:37.868174+0000","last_peered":"2024-08-28T19:20:37.868174+0000","last_clean":"2024-08-28T19:20:37.868174+0000","last_became_active":"2024-08-28T19:18:32.484197+0000","last_became_peered":"2024-08-28T19:18:32.484197+0000","last_unstale":"2024-08-28T19:20:37.868174+0000","last_undegraded":"2024-08-28T19:20:37.868174+0000","last_fullsized":"2024-08-28T19:20:37.868174+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:47:50.527915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868339+0000","last_change":"2024-08-28T19:18:32.489452+0000","last_active":"2024-08-28T19:20:37.868339+0000","last_peered":"2024-08-28T19:20:37.868339+0000","last_clean":"2024-08-28T19:20:37.868339+0000","last_became_active":"2024-08-28T19:18:32.487482+0000","last_became_peered":"2024-08-28T19:18:32.487482+0000","last_unstale":"2024-08-28T19:20:37.868339+0000","last_undegraded":"2024-08-28T19:20:37.868339+0000","last_fullsized":"2024-08-28T19:20:37.868339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:52:08.175080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884068+0000","last_change":"2024-08-28T19:18:32.483850+0000","last_active":"2024-08-28T19:20:37.884068+0000","last_peered":"2024-08-28T19:20:37.884068+0000","last_clean":"2024-08-28T19:20:37.884068+0000","last_became_active":"2024-08-28T19:18:32.483476+0000","last_became_peered":"2024-08-28T19:18:32.483476+0000","last_unstale":"2024-08-28T19:20:37.884068+0000","last_undegraded":"2024-08-28T19:20:37.884068+0000","last_fullsized":"2024-08-28T19:20:37.884068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:32:37.465218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868836+0000","last_change":"2024-08-28T19:18:32.492252+0000","last_active":"2024-08-28T19:20:37.868836+0000","last_peered":"2024-08-28T19:20:37.868836+0000","last_clean":"2024-08-28T19:20:37.868836+0000","last_became_active":"2024-08-28T19:18:32.491659+0000","last_became_peered":"2024-08-28T19:18:32.491659+0000","last_unstale":"2024-08-28T19:20:37.868836+0000","last_undegraded":"2024-08-28T19:20:37.868836+0000","last_fullsized":"2024-08-28T19:20:37.868836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:19:28.901559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865090+0000","last_change":"2024-08-28T19:18:32.491940+0000","last_active":"2024-08-28T19:20:37.865090+0000","last_peered":"2024-08-28T19:20:37.865090+0000","last_clean":"2024-08-28T19:20:37.865090+0000","last_became_active":"2024-08-28T19:18:32.487474+0000","last_became_peered":"2024-08-28T19:18:32.487474+0000","last_unstale":"2024-08-28T19:20:37.865090+0000","last_undegraded":"2024-08-28T19:20:37.865090+0000","last_fullsized":"2024-08-28T19:20:37.865090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:04:32.508620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871438+0000","last_change":"2024-08-28T19:18:32.485200+0000","last_active":"2024-08-28T19:20:37.871438+0000","last_peered":"2024-08-28T19:20:37.871438+0000","last_clean":"2024-08-28T19:20:37.871438+0000","last_became_active":"2024-08-28T19:18:32.484858+0000","last_became_peered":"2024-08-28T19:18:32.484858+0000","last_unstale":"2024-08-28T19:20:37.871438+0000","last_undegraded":"2024-08-28T19:20:37.871438+0000","last_fullsized":"2024-08-28T19:20:37.871438+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:10:57.925659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886493+0000","last_change":"2024-08-28T19:18:32.489903+0000","last_active":"2024-08-28T19:20:37.886493+0000","last_peered":"2024-08-28T19:20:37.886493+0000","last_clean":"2024-08-28T19:20:37.886493+0000","last_became_active":"2024-08-28T19:18:32.488950+0000","last_became_peered":"2024-08-28T19:18:32.488950+0000","last_unstale":"2024-08-28T19:20:37.886493+0000","last_undegraded":"2024-08-28T19:20:37.886493+0000","last_fullsized":"2024-08-28T19:20:37.886493+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:06:42.284177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872318+0000","last_change":"2024-08-28T19:18:32.488236+0000","last_active":"2024-08-28T19:20:37.872318+0000","last_peered":"2024-08-28T19:20:37.872318+0000","last_clean":"2024-08-28T19:20:37.872318+0000","last_became_active":"2024-08-28T19:18:32.487776+0000","last_became_peered":"2024-08-28T19:18:32.487776+0000","last_unstale":"2024-08-28T19:20:37.872318+0000","last_undegraded":"2024-08-28T19:20:37.872318+0000","last_fullsized":"2024-08-28T19:20:37.872318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:25:58.754740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875857+0000","last_change":"2024-08-28T19:18:32.485860+0000","last_active":"2024-08-28T19:20:37.875857+0000","last_peered":"2024-08-28T19:20:37.875857+0000","last_clean":"2024-08-28T19:20:37.875857+0000","last_became_active":"2024-08-28T19:18:32.485415+0000","last_became_peered":"2024-08-28T19:18:32.485415+0000","last_unstale":"2024-08-28T19:20:37.875857+0000","last_undegraded":"2024-08-28T19:20:37.875857+0000","last_fullsized":"2024-08-28T19:20:37.875857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:45:32.868651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873054+0000","last_change":"2024-08-28T19:18:32.485515+0000","last_active":"2024-08-28T19:20:37.873054+0000","last_peered":"2024-08-28T19:20:37.873054+0000","last_clean":"2024-08-28T19:20:37.873054+0000","last_became_active":"2024-08-28T19:18:32.485075+0000","last_became_peered":"2024-08-28T19:18:32.485075+0000","last_unstale":"2024-08-28T19:20:37.873054+0000","last_undegraded":"2024-08-28T19:20:37.873054+0000","last_fullsized":"2024-08-28T19:20:37.873054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:01:28.035479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864600+0000","last_change":"2024-08-28T19:18:32.484685+0000","last_active":"2024-08-28T19:20:37.864600+0000","last_peered":"2024-08-28T19:20:37.864600+0000","last_clean":"2024-08-28T19:20:37.864600+0000","last_became_active":"2024-08-28T19:18:32.484095+0000","last_became_peered":"2024-08-28T19:18:32.484095+0000","last_unstale":"2024-08-28T19:20:37.864600+0000","last_undegraded":"2024-08-28T19:20:37.864600+0000","last_fullsized":"2024-08-28T19:20:37.864600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:53:27.229186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884050+0000","last_change":"2024-08-28T19:18:32.488030+0000","last_active":"2024-08-28T19:20:37.884050+0000","last_peered":"2024-08-28T19:20:37.884050+0000","last_clean":"2024-08-28T19:20:37.884050+0000","last_became_active":"2024-08-28T19:18:32.487042+0000","last_became_peered":"2024-08-28T19:18:32.487042+0000","last_unstale":"2024-08-28T19:20:37.884050+0000","last_undegraded":"2024-08-28T19:20:37.884050+0000","last_fullsized":"2024-08-28T19:20:37.884050+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:56.966475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886125+0000","last_change":"2024-08-28T19:18:32.489442+0000","last_active":"2024-08-28T19:20:37.886125+0000","last_peered":"2024-08-28T19:20:37.886125+0000","last_clean":"2024-08-28T19:20:37.886125+0000","last_became_active":"2024-08-28T19:18:32.488405+0000","last_became_peered":"2024-08-28T19:18:32.488405+0000","last_unstale":"2024-08-28T19:20:37.886125+0000","last_undegraded":"2024-08-28T19:20:37.886125+0000","last_fullsized":"2024-08-28T19:20:37.886125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:21:03.026893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870138+0000","last_change":"2024-08-28T19:20:09.164305+0000","last_active":"2024-08-28T19:20:37.870138+0000","last_peered":"2024-08-28T19:20:37.870138+0000","last_clean":"2024-08-28T19:20:37.870138+0000","last_became_active":"2024-08-28T19:20:09.163834+0000","last_became_peered":"2024-08-28T19:20:09.163834+0000","last_unstale":"2024-08-28T19:20:37.870138+0000","last_undegraded":"2024-08-28T19:20:37.870138+0000","last_fullsized":"2024-08-28T19:20:37.870138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:20:10.137952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870501+0000","last_change":"2024-08-28T19:18:32.492077+0000","last_active":"2024-08-28T19:20:37.870501+0000","last_peered":"2024-08-28T19:20:37.870501+0000","last_clean":"2024-08-28T19:20:37.870501+0000","last_became_active":"2024-08-28T19:18:32.491722+0000","last_became_peered":"2024-08-28T19:18:32.491722+0000","last_unstale":"2024-08-28T19:20:37.870501+0000","last_undegraded":"2024-08-28T19:20:37.870501+0000","last_fullsized":"2024-08-28T19:20:37.870501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:16:21.697045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"84'10","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865421+0000","last_change":"2024-08-28T19:18:32.479845+0000","last_active":"2024-08-28T19:20:37.865421+0000","last_peered":"2024-08-28T19:20:37.865421+0000","last_clean":"2024-08-28T19:20:37.865421+0000","last_became_active":"2024-08-28T19:18:32.477690+0000","last_became_peered":"2024-08-28T19:18:32.477690+0000","last_unstale":"2024-08-28T19:20:37.865421+0000","last_undegraded":"2024-08-28T19:20:37.865421+0000","last_fullsized":"2024-08-28T19:20:37.865421+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:11.281415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866009+0000","last_change":"2024-08-28T19:19:08.624185+0000","last_active":"2024-08-28T19:20:37.866009+0000","last_peered":"2024-08-28T19:20:37.866009+0000","last_clean":"2024-08-28T19:20:37.866009+0000","last_became_active":"2024-08-28T19:19:08.623745+0000","last_became_peered":"2024-08-28T19:19:08.623745+0000","last_unstale":"2024-08-28T19:20:37.866009+0000","last_undegraded":"2024-08-28T19:20:37.866009+0000","last_fullsized":"2024-08-28T19:20:37.866009+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:56.657173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"84'6","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886865+0000","last_change":"2024-08-28T19:18:32.499038+0000","last_active":"2024-08-28T19:20:37.886865+0000","last_peered":"2024-08-28T19:20:37.886865+0000","last_clean":"2024-08-28T19:20:37.886865+0000","last_became_active":"2024-08-28T19:18:32.498276+0000","last_became_peered":"2024-08-28T19:18:32.498276+0000","last_unstale":"2024-08-28T19:20:37.886865+0000","last_undegraded":"2024-08-28T19:20:37.886865+0000","last_fullsized":"2024-08-28T19:20:37.886865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:17:45.468448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"84'16","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.882260+0000","last_change":"2024-08-28T19:18:32.492183+0000","last_active":"2024-08-28T19:20:37.882260+0000","last_peered":"2024-08-28T19:20:37.882260+0000","last_clean":"2024-08-28T19:20:37.882260+0000","last_became_active":"2024-08-28T19:18:32.491815+0000","last_became_peered":"2024-08-28T19:18:32.491815+0000","last_unstale":"2024-08-28T19:20:37.882260+0000","last_undegraded":"2024-08-28T19:20:37.882260+0000","last_fullsized":"2024-08-28T19:20:37.882260+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:13:11.183955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"84'11","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886721+0000","last_change":"2024-08-28T19:18:32.497399+0000","last_active":"2024-08-28T19:20:37.886721+0000","last_peered":"2024-08-28T19:20:37.886721+0000","last_clean":"2024-08-28T19:20:37.886721+0000","last_became_active":"2024-08-28T19:18:32.496901+0000","last_became_peered":"2024-08-28T19:18:32.496901+0000","last_unstale":"2024-08-28T19:20:37.886721+0000","last_undegraded":"2024-08-28T19:20:37.886721+0000","last_fullsized":"2024-08-28T19:20:37.886721+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:05:57.634590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"84'9","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864415+0000","last_change":"2024-08-28T19:18:32.495871+0000","last_active":"2024-08-28T19:20:37.864415+0000","last_peered":"2024-08-28T19:20:37.864415+0000","last_clean":"2024-08-28T19:20:37.864415+0000","last_became_active":"2024-08-28T19:18:32.495515+0000","last_became_peered":"2024-08-28T19:18:32.495515+0000","last_unstale":"2024-08-28T19:20:37.864415+0000","last_undegraded":"2024-08-28T19:20:37.864415+0000","last_fullsized":"2024-08-28T19:20:37.864415+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:53:56.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"84'5","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875276+0000","last_change":"2024-08-28T19:18:32.495711+0000","last_active":"2024-08-28T19:20:37.875276+0000","last_peered":"2024-08-28T19:20:37.875276+0000","last_clean":"2024-08-28T19:20:37.875276+0000","last_became_active":"2024-08-28T19:18:32.495271+0000","last_became_peered":"2024-08-28T19:18:32.495271+0000","last_unstale":"2024-08-28T19:20:37.875276+0000","last_undegraded":"2024-08-28T19:20:37.875276+0000","last_fullsized":"2024-08-28T19:20:37.875276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:05.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"84'6","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880581+0000","last_change":"2024-08-28T19:18:32.489731+0000","last_active":"2024-08-28T19:20:37.880581+0000","last_peered":"2024-08-28T19:20:37.880581+0000","last_clean":"2024-08-28T19:20:37.880581+0000","last_became_active":"2024-08-28T19:18:32.489366+0000","last_became_peered":"2024-08-28T19:18:32.489366+0000","last_unstale":"2024-08-28T19:20:37.880581+0000","last_undegraded":"2024-08-28T19:20:37.880581+0000","last_fullsized":"2024-08-28T19:20:37.880581+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:52:24.180155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"84'4","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.861147+0000","last_change":"2024-08-28T19:18:32.490270+0000","last_active":"2024-08-28T19:20:37.861147+0000","last_peered":"2024-08-28T19:20:37.861147+0000","last_clean":"2024-08-28T19:20:37.861147+0000","last_became_active":"2024-08-28T19:18:32.489667+0000","last_became_peered":"2024-08-28T19:18:32.489667+0000","last_unstale":"2024-08-28T19:20:37.861147+0000","last_undegraded":"2024-08-28T19:20:37.861147+0000","last_fullsized":"2024-08-28T19:20:37.861147+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:18.571978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873527+0000","last_change":"2024-08-28T19:18:32.482553+0000","last_active":"2024-08-28T19:20:37.873527+0000","last_peered":"2024-08-28T19:20:37.873527+0000","last_clean":"2024-08-28T19:20:37.873527+0000","last_became_active":"2024-08-28T19:18:32.481161+0000","last_became_peered":"2024-08-28T19:18:32.481161+0000","last_unstale":"2024-08-28T19:20:37.873527+0000","last_undegraded":"2024-08-28T19:20:37.873527+0000","last_fullsized":"2024-08-28T19:20:37.873527+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:13.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883684+0000","last_change":"2024-08-28T19:18:32.492473+0000","last_active":"2024-08-28T19:20:37.883684+0000","last_peered":"2024-08-28T19:20:37.883684+0000","last_clean":"2024-08-28T19:20:37.883684+0000","last_became_active":"2024-08-28T19:18:32.491897+0000","last_became_peered":"2024-08-28T19:18:32.491897+0000","last_unstale":"2024-08-28T19:20:37.883684+0000","last_undegraded":"2024-08-28T19:20:37.883684+0000","last_fullsized":"2024-08-28T19:20:37.883684+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:24.279391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870908+0000","last_change":"2024-08-28T19:18:32.485621+0000","last_active":"2024-08-28T19:20:37.870908+0000","last_peered":"2024-08-28T19:20:37.870908+0000","last_clean":"2024-08-28T19:20:37.870908+0000","last_became_active":"2024-08-28T19:18:32.485267+0000","last_became_peered":"2024-08-28T19:18:32.485267+0000","last_unstale":"2024-08-28T19:20:37.870908+0000","last_undegraded":"2024-08-28T19:20:37.870908+0000","last_fullsized":"2024-08-28T19:20:37.870908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:30:08.932135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884032+0000","last_change":"2024-08-28T19:18:32.488457+0000","last_active":"2024-08-28T19:20:37.884032+0000","last_peered":"2024-08-28T19:20:37.884032+0000","last_clean":"2024-08-28T19:20:37.884032+0000","last_became_active":"2024-08-28T19:18:32.487555+0000","last_became_peered":"2024-08-28T19:18:32.487555+0000","last_unstale":"2024-08-28T19:20:37.884032+0000","last_undegraded":"2024-08-28T19:20:37.884032+0000","last_fullsized":"2024-08-28T19:20:37.884032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:03:26.390481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71518,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1084,"num_read_kb":618,"num_write":262,"num_write_kb":152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":225,"ondisk_log_size":225,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":68,"seq":292057776171,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":500491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":62,"seq":266287972399,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1683456,"data_stored":1154763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":57,"seq":244813135925,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":503845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":51,"seq":219043332153,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36324,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708828,"statfs":{"total":95995035648,"available":95957839872,"internally_reserved":0,"allocated":1003520,"data_stored":500127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":45,"seq":193273528382,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1593344,"data_stored":1096625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":40,"seq":171798691908,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":495785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888136,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1605632,"data_stored":1088251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051663,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":499327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215186,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40476,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704676,"statfs":{"total":95995035648,"available":95953588224,"internally_reserved":0,"allocated":995328,"data_stored":494961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":561067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1052672,"data_stored":560847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705762,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":496709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-28T19:20:59.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:20:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:00.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:00 smithi016 ceph-mon[18556]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:00.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:00 smithi016 ceph-mon[18556]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.441+0000 7fddcf874640 1 -- 172.21.15.16:0/59722150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddc8103950 msgr2=0x7fddc8103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.441+0000 7fddcf874640 1 --2- 172.21.15.16:0/59722150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddc8103950 0x7fddc8103d60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009900 tx=0x7fddc402a410 comp rx=0 tx=0).stop 2024-08-28T19:21:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.441+0000 7fddcf874640 1 -- 172.21.15.16:0/59722150 shutdown_connections 2024-08-28T19:21:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.441+0000 7fddcf874640 1 --2- 172.21.15.16:0/59722150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddc8109be0 0x7fddc8110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.441+0000 7fddcf874640 1 --2- 172.21.15.16:0/59722150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc8109290 0x7fddc81096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.441+0000 7fddcf874640 1 --2- 172.21.15.16:0/59722150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddc8103950 0x7fddc8103d60 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.441+0000 7fddcf874640 1 -- 172.21.15.16:0/59722150 >> 172.21.15.16:0/59722150 conn(0x7fddc80ff3a0 msgr2=0x7fddc81017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.442+0000 7fddcf874640 1 -- 172.21.15.16:0/59722150 shutdown_connections 2024-08-28T19:21:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.442+0000 7fddcf874640 1 -- 172.21.15.16:0/59722150 wait complete. 2024-08-28T19:21:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.442+0000 7fddcf874640 1 Processor -- start 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.442+0000 7fddcf874640 1 -- start start 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcf874640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddc8103950 0x7fddc80754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcf874640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddc8109290 0x7fddc80759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcf874640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc8109be0 0x7fddc8075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcf874640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fddc81138a0 con 0x7fddc8109290 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcf874640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fddc8113720 con 0x7fddc8109be0 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcf874640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fddc8113a20 con 0x7fddc8103950 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcce0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddc8109290 0x7fddc80759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcce0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddc8109290 0x7fddc80759f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37114/0 (socket says 172.21.15.16:37114) 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcce0f640 1 -- 172.21.15.16:0/3992812423 learned_addr learned my addr 172.21.15.16:0/3992812423 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcd610640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddc8103950 0x7fddc80754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcde11640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc8109be0 0x7fddc8075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcce0f640 1 -- 172.21.15.16:0/3992812423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddc8103950 msgr2=0x7fddc80754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcce0f640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddc8103950 0x7fddc80754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcce0f640 1 -- 172.21.15.16:0/3992812423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc8109be0 msgr2=0x7fddc8075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcce0f640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc8109be0 0x7fddc8075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcce0f640 1 -- 172.21.15.16:0/3992812423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc81a84a0 con 0x7fddc8109290 2024-08-28T19:21:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddcce0f640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddc8109290 0x7fddc80759f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fddb800e9f0 tx=0x7fddb800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.443+0000 7fddbe7fc640 1 -- 172.21.15.16:0/3992812423 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb800ccc0 con 0x7fddc8109290 2024-08-28T19:21:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.444+0000 7fddbe7fc640 1 -- 172.21.15.16:0/3992812423 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddb80108e0 con 0x7fddc8109290 2024-08-28T19:21:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.444+0000 7fddcf874640 1 -- 172.21.15.16:0/3992812423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc81a8790 con 0x7fddc8109290 2024-08-28T19:21:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.444+0000 7fddbe7fc640 1 -- 172.21.15.16:0/3992812423 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8018860 con 0x7fddc8109290 2024-08-28T19:21:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.444+0000 7fddcf874640 1 -- 172.21.15.16:0/3992812423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc81a8d20 con 0x7fddc8109290 2024-08-28T19:21:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.445+0000 7fddcf874640 1 -- 172.21.15.16:0/3992812423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc8103d60 con 0x7fddc8109290 2024-08-28T19:21:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.446+0000 7fddbe7fc640 1 -- 172.21.15.16:0/3992812423 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fddb8010430 con 0x7fddc8109290 2024-08-28T19:21:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.447+0000 7fddbe7fc640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddb40b4c00 0x7fddb40b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.447+0000 7fddcd610640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddb40b4c00 0x7fddb40b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.447+0000 7fddbe7fc640 1 -- 172.21.15.16:0/3992812423 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fddb80e5e30 con 0x7fddc8109290 2024-08-28T19:21:00.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.448+0000 7fddcd610640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddb40b4c00 0x7fddb40b7110 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fddc4000c00 tx=0x7fddc4031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.450+0000 7fddbe7fc640 1 -- 172.21.15.16:0/3992812423 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fddb80aca70 con 0x7fddc8109290 2024-08-28T19:21:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:00 smithi028 ceph-mon[21769]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:00 smithi028 ceph-mon[21769]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:00.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:00 smithi115 ceph-mon[22184]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:00.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:00 smithi115 ceph-mon[22184]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:00.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.677+0000 7fddcf874640 1 -- 172.21.15.16:0/3992812423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddc80767b0 con 0x7fddc8109290 2024-08-28T19:21:00.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.678+0000 7fddbe7fc640 1 -- 172.21.15.16:0/3992812423 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11977 (secure 0 0 0) 0x7fddc80767b0 con 0x7fddc8109290 2024-08-28T19:21:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:21:00.681 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"btime":"2024-08-28T19:20:55:257311+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:20:54.880238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:21:00.681 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2024-08-28T19:21:00.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 -- 172.21.15.16:0/3992812423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddb40b4c00 msgr2=0x7fddb40b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:00.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddb40b4c00 0x7fddb40b7110 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fddc4000c00 tx=0x7fddc4031400 comp rx=0 tx=0).stop 2024-08-28T19:21:00.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 -- 172.21.15.16:0/3992812423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddc8109290 msgr2=0x7fddc80759f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:00.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddc8109290 0x7fddc80759f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fddb800e9f0 tx=0x7fddb800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:21:00.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 -- 172.21.15.16:0/3992812423 shutdown_connections 2024-08-28T19:21:00.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddb40b4c00 0x7fddb40b7110 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:00.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc8109be0 0x7fddc8075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:00.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddc8109290 0x7fddc80759f0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:00.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 --2- 172.21.15.16:0/3992812423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddc8103950 0x7fddc80754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:00.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 -- 172.21.15.16:0/3992812423 >> 172.21.15.16:0/3992812423 conn(0x7fddc80ff3a0 msgr2=0x7fddc81017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:00.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 -- 172.21.15.16:0/3992812423 shutdown_connections 2024-08-28T19:21:00.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:00.682+0000 7fddcf874640 1 -- 172.21.15.16:0/3992812423 wait complete. 2024-08-28T19:21:01.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:21:01.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:01 smithi016 ceph-mon[18556]: pgmap v338: 129 pgs: 129 active+clean; 647 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-08-28T19:21:01.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:01 smithi016 ceph-mon[18556]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:01.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:01 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3992812423' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:01 smithi028 ceph-mon[21769]: pgmap v338: 129 pgs: 129 active+clean; 647 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-08-28T19:21:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:01 smithi028 ceph-mon[21769]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:01 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3992812423' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:01.586 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:01.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:01 smithi115 ceph-mon[22184]: pgmap v338: 129 pgs: 129 active+clean; 647 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-08-28T19:21:01.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:01 smithi115 ceph-mon[22184]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:01.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:01 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3992812423' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.126+0000 7fc373e90640 1 -- 172.21.15.16:0/1909340137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc36c103900 msgr2=0x7fc36c103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.126+0000 7fc373e90640 1 --2- 172.21.15.16:0/1909340137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc36c103900 0x7fc36c103d10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc368009900 tx=0x7fc36802a410 comp rx=0 tx=0).stop 2024-08-28T19:21:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.126+0000 7fc373e90640 1 -- 172.21.15.16:0/1909340137 shutdown_connections 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.126+0000 7fc373e90640 1 --2- 172.21.15.16:0/1909340137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c109b90 0x7fc36c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.126+0000 7fc373e90640 1 --2- 172.21.15.16:0/1909340137 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc36c109240 0x7fc36c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.126+0000 7fc373e90640 1 --2- 172.21.15.16:0/1909340137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc36c103900 0x7fc36c103d10 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.126+0000 7fc373e90640 1 -- 172.21.15.16:0/1909340137 >> 172.21.15.16:0/1909340137 conn(0x7fc36c07b5f0 msgr2=0x7fc36c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.126+0000 7fc373e90640 1 -- 172.21.15.16:0/1909340137 shutdown_connections 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.126+0000 7fc373e90640 1 -- 172.21.15.16:0/1909340137 wait complete. 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc373e90640 1 Processor -- start 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc373e90640 1 -- start start 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc373e90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc36c103900 0x7fc36c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc373e90640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc36c109240 0x7fc36c19c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc373e90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c109b90 0x7fc36c1a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc373e90640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c113850 con 0x7fc36c103900 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc373e90640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1136d0 con 0x7fc36c109b90 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc373e90640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1139d0 con 0x7fc36c109240 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc371c2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc36c103900 0x7fc36c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc371c2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc36c103900 0x7fc36c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37142/0 (socket says 172.21.15.16:37142) 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc371c2c640 1 -- 172.21.15.16:0/3459427339 learned_addr learned my addr 172.21.15.16:0/3459427339 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc37142b640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc36c109240 0x7fc36c19c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc37242d640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c109b90 0x7fc36c1a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc371c2c640 1 -- 172.21.15.16:0/3459427339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc36c109240 msgr2=0x7fc36c19c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc371c2c640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc36c109240 0x7fc36c19c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc371c2c640 1 -- 172.21.15.16:0/3459427339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c109b90 msgr2=0x7fc36c1a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc371c2c640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c109b90 0x7fc36c1a1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.127+0000 7fc371c2c640 1 -- 172.21.15.16:0/3459427339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36c1a2390 con 0x7fc36c103900 2024-08-28T19:21:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.128+0000 7fc371c2c640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc36c103900 0x7fc36c19c420 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc368005940 tx=0x7fc3680042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.128+0000 7fc362ffd640 1 -- 172.21.15.16:0/3459427339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36803f030 con 0x7fc36c103900 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.128+0000 7fc373e90640 1 -- 172.21.15.16:0/3459427339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc36c1a2620 con 0x7fc36c103900 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.128+0000 7fc362ffd640 1 -- 172.21.15.16:0/3459427339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc368003cb0 con 0x7fc36c103900 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.128+0000 7fc362ffd640 1 -- 172.21.15.16:0/3459427339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc368035070 con 0x7fc36c103900 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.128+0000 7fc373e90640 1 -- 172.21.15.16:0/3459427339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc36c1a8de0 con 0x7fc36c103900 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.129+0000 7fc373e90640 1 -- 172.21.15.16:0/3459427339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc36c103d10 con 0x7fc36c103900 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.131+0000 7fc362ffd640 1 -- 172.21.15.16:0/3459427339 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fc368042030 con 0x7fc36c103900 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.132+0000 7fc362ffd640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3580b4b80 0x7fc3580b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.132+0000 7fc37142b640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3580b4b80 0x7fc3580b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.132+0000 7fc362ffd640 1 -- 172.21.15.16:0/3459427339 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc3681061d0 con 0x7fc36c103900 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.133+0000 7fc37142b640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3580b4b80 0x7fc3580b7090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc35c005d00 tx=0x7fc35c005c50 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.136+0000 7fc362ffd640 1 -- 172.21.15.16:0/3459427339 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fc3680cce70 con 0x7fc36c103900 2024-08-28T19:21:02.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.322+0000 7fc373e90640 1 -- 172.21.15.16:0/3459427339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc36c1a9010 con 0x7fc36c103900 2024-08-28T19:21:02.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.324+0000 7fc362ffd640 1 -- 172.21.15.16:0/3459427339 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+25180 (secure 0 0 0) 0x7fc3680cc770 con 0x7fc36c103900 2024-08-28T19:21:02.326 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:21:02.327 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:20:09.003260+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:21:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.326+0000 7fc373e90640 1 -- 172.21.15.16:0/3459427339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3580b4b80 msgr2=0x7fc3580b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.326+0000 7fc373e90640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3580b4b80 0x7fc3580b7090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc35c005d00 tx=0x7fc35c005c50 comp rx=0 tx=0).stop 2024-08-28T19:21:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.326+0000 7fc373e90640 1 -- 172.21.15.16:0/3459427339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc36c103900 msgr2=0x7fc36c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.326+0000 7fc373e90640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc36c103900 0x7fc36c19c420 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc368005940 tx=0x7fc3680042f0 comp rx=0 tx=0).stop 2024-08-28T19:21:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.327+0000 7fc373e90640 1 -- 172.21.15.16:0/3459427339 shutdown_connections 2024-08-28T19:21:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.327+0000 7fc373e90640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3580b4b80 0x7fc3580b7090 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.327+0000 7fc373e90640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c109b90 0x7fc36c1a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.327+0000 7fc373e90640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc36c109240 0x7fc36c19c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:02.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.327+0000 7fc373e90640 1 --2- 172.21.15.16:0/3459427339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc36c103900 0x7fc36c19c420 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:02.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.327+0000 7fc373e90640 1 -- 172.21.15.16:0/3459427339 >> 172.21.15.16:0/3459427339 conn(0x7fc36c07b5f0 msgr2=0x7fc36c1070b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:02.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.327+0000 7fc373e90640 1 -- 172.21.15.16:0/3459427339 shutdown_connections 2024-08-28T19:21:02.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:02.327+0000 7fc373e90640 1 -- 172.21.15.16:0/3459427339 wait complete. 2024-08-28T19:21:02.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:02 smithi028 ceph-mon[21769]: pgmap v339: 129 pgs: 129 active+clean; 652 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 767 B/s wr, 12 op/s 2024-08-28T19:21:02.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:02 smithi115 ceph-mon[22184]: pgmap v339: 129 pgs: 129 active+clean; 652 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 767 B/s wr, 12 op/s 2024-08-28T19:21:02.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:02 smithi016 ceph-mon[18556]: pgmap v339: 129 pgs: 129 active+clean; 652 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 767 B/s wr, 12 op/s 2024-08-28T19:21:03.008 DEBUG:tasks.fwd_scrub:Yielding 2024-08-28T19:21:03.008 INFO:teuthology.run_tasks:Running task exec... 2024-08-28T19:21:03.017 INFO:teuthology.task.exec:Executing custom commands... 2024-08-28T19:21:03.017 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-08-28T19:21:03.017 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-08-28T19:21:03.017 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-08-28T19:21:03.017 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:21:03.018 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:21:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.165+0000 7fc2c3258640 1 -- 172.21.15.16:0/2990244778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2bc108b30 msgr2=0x7fc2bc10f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.165+0000 7fc2c3258640 1 --2- 172.21.15.16:0/2990244778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2bc108b30 0x7fc2bc10f360 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009ca0 tx=0x7fc2b402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.165+0000 7fc2c3258640 1 -- 172.21.15.16:0/2990244778 shutdown_connections 2024-08-28T19:21:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.165+0000 7fc2c3258640 1 --2- 172.21.15.16:0/2990244778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2bc108b30 0x7fc2bc10f360 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.165+0000 7fc2c3258640 1 --2- 172.21.15.16:0/2990244778 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2bc1081e0 0x7fc2bc1085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.165+0000 7fc2c3258640 1 --2- 172.21.15.16:0/2990244778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2bc102f10 0x7fc2bc103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.165+0000 7fc2c3258640 1 -- 172.21.15.16:0/2990244778 >> 172.21.15.16:0/2990244778 conn(0x7fc2bc079c30 msgr2=0x7fc2bc07a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.165+0000 7fc2c3258640 1 -- 172.21.15.16:0/2990244778 shutdown_connections 2024-08-28T19:21:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.165+0000 7fc2c3258640 1 -- 172.21.15.16:0/2990244778 wait complete. 2024-08-28T19:21:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.166+0000 7fc2c3258640 1 Processor -- start 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.166+0000 7fc2c3258640 1 -- start start 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.166+0000 7fc2c3258640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2bc102f10 0x7fc2bc19b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.166+0000 7fc2c3258640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2bc1081e0 0x7fc2bc19b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.166+0000 7fc2c3258640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2bc108b30 0x7fc2bc19bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.166+0000 7fc2c3258640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc112800 con 0x7fc2bc1081e0 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.166+0000 7fc2c3258640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc112680 con 0x7fc2bc108b30 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.166+0000 7fc2c3258640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc112980 con 0x7fc2bc102f10 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.166+0000 7fc2bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2bc1081e0 0x7fc2bc19b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2bc1081e0 0x7fc2bc19b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37168/0 (socket says 172.21.15.16:37168) 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2bbfff640 1 -- 172.21.15.16:0/3411547874 learned_addr learned my addr 172.21.15.16:0/3411547874 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2c17f5640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2bc108b30 0x7fc2bc19bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2c0ff4640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2bc102f10 0x7fc2bc19b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2bbfff640 1 -- 172.21.15.16:0/3411547874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2bc102f10 msgr2=0x7fc2bc19b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2bbfff640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2bc102f10 0x7fc2bc19b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2bbfff640 1 -- 172.21.15.16:0/3411547874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2bc108b30 msgr2=0x7fc2bc19bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2bbfff640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2bc108b30 0x7fc2bc19bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2bbfff640 1 -- 172.21.15.16:0/3411547874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2bc1a63f0 con 0x7fc2bc1081e0 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2c17f5640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2bc108b30 0x7fc2bc19bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2bbfff640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2bc1081e0 0x7fc2bc19b670 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800c970 tx=0x7fc2a800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2b9ffb640 1 -- 172.21.15.16:0/3411547874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8004220 con 0x7fc2bc1081e0 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2b9ffb640 1 -- 172.21.15.16:0/3411547874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2a80043c0 con 0x7fc2bc1081e0 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2b9ffb640 1 -- 172.21.15.16:0/3411547874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8004a00 con 0x7fc2bc1081e0 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.167+0000 7fc2c3258640 1 -- 172.21.15.16:0/3411547874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2bc1a66e0 con 0x7fc2bc1081e0 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.168+0000 7fc2c3258640 1 -- 172.21.15.16:0/3411547874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2bc1a6be0 con 0x7fc2bc1081e0 2024-08-28T19:21:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.169+0000 7fc2c3258640 1 -- 172.21.15.16:0/3411547874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2bc06b450 con 0x7fc2bc1081e0 2024-08-28T19:21:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.170+0000 7fc2b9ffb640 1 -- 172.21.15.16:0/3411547874 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fc2a80079b0 con 0x7fc2bc1081e0 2024-08-28T19:21:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.171+0000 7fc2b9ffb640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ac0b4b80 0x7fc2ac0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.171+0000 7fc2c0ff4640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ac0b4b80 0x7fc2ac0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.172+0000 7fc2b9ffb640 1 -- 172.21.15.16:0/3411547874 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc2a80e5850 con 0x7fc2bc1081e0 2024-08-28T19:21:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.172+0000 7fc2c0ff4640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ac0b4b80 0x7fc2ac0b7090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0005fd0 tx=0x7fc2b0005850 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.174+0000 7fc2b9ffb640 1 -- 172.21.15.16:0/3411547874 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fc2a8014030 con 0x7fc2bc1081e0 2024-08-28T19:21:03.319 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:03.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.411+0000 7fc2c3258640 1 -- 172.21.15.16:0/3411547874 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc2bc0635a0 con 0x7fc2ac0b4b80 2024-08-28T19:21:03.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:03 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3459427339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:21:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.567+0000 7fc2b9ffb640 1 -- 172.21.15.16:0/3411547874 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc2bc0635a0 con 0x7fc2ac0b4b80 2024-08-28T19:21:03.568 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path / 2024-08-28T19:21:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 -- 172.21.15.16:0/3411547874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ac0b4b80 msgr2=0x7fc2ac0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ac0b4b80 0x7fc2ac0b7090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0005fd0 tx=0x7fc2b0005850 comp rx=0 tx=0).stop 2024-08-28T19:21:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 -- 172.21.15.16:0/3411547874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2bc1081e0 msgr2=0x7fc2bc19b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2bc1081e0 0x7fc2bc19b670 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800c970 tx=0x7fc2a800ce30 comp rx=0 tx=0).stop 2024-08-28T19:21:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 -- 172.21.15.16:0/3411547874 shutdown_connections 2024-08-28T19:21:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ac0b4b80 0x7fc2ac0b7090 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2bc108b30 0x7fc2bc19bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2bc1081e0 0x7fc2bc19b670 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 --2- 172.21.15.16:0/3411547874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2bc102f10 0x7fc2bc19b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 -- 172.21.15.16:0/3411547874 >> 172.21.15.16:0/3411547874 conn(0x7fc2bc079c30 msgr2=0x7fc2bc106230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 -- 172.21.15.16:0/3411547874 shutdown_connections 2024-08-28T19:21:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.569+0000 7fc2c3258640 1 -- 172.21.15.16:0/3411547874 wait complete. 2024-08-28T19:21:03.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:03 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3459427339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:21:03.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-08-28T19:21:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:03 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3459427339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:21:03.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.698+0000 7f25658da640 1 -- 172.21.15.16:0/3481781601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25601089d0 msgr2=0x7f256010f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.698+0000 7f25658da640 1 --2- 172.21.15.16:0/3481781601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25601089d0 0x7f256010f200 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2558009ca0 tx=0x7f255802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.698+0000 7f25658da640 1 -- 172.21.15.16:0/3481781601 shutdown_connections 2024-08-28T19:21:03.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.698+0000 7f25658da640 1 --2- 172.21.15.16:0/3481781601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25601089d0 0x7f256010f200 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.698+0000 7f25658da640 1 --2- 172.21.15.16:0/3481781601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2560108080 0x7f2560108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.698+0000 7f25658da640 1 --2- 172.21.15.16:0/3481781601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2560102750 0x7f2560102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.698+0000 7f25658da640 1 -- 172.21.15.16:0/3481781601 >> 172.21.15.16:0/3481781601 conn(0x7f256007a4e0 msgr2=0x7f256007a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:03.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.698+0000 7f25658da640 1 -- 172.21.15.16:0/3481781601 shutdown_connections 2024-08-28T19:21:03.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.698+0000 7f25658da640 1 -- 172.21.15.16:0/3481781601 wait complete. 2024-08-28T19:21:03.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.699+0000 7f25658da640 1 Processor -- start 2024-08-28T19:21:03.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.699+0000 7f25658da640 1 -- start start 2024-08-28T19:21:03.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.699+0000 7f25658da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2560102750 0x7f25601990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.699+0000 7f25658da640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2560108080 0x7f2560199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f25658da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25601089d0 0x7f256019e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f25658da640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25601126a0 con 0x7f2560102750 2024-08-28T19:21:03.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f25658da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2560112520 con 0x7f25601089d0 2024-08-28T19:21:03.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f25658da640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2560112820 con 0x7f2560108080 2024-08-28T19:21:03.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f255effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2560102750 0x7f25601990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f255effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2560102750 0x7f25601990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37218/0 (socket says 172.21.15.16:37218) 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f255effd640 1 -- 172.21.15.16:0/3963784707 learned_addr learned my addr 172.21.15.16:0/3963784707 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f255e7fc640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2560108080 0x7f2560199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f255f7fe640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25601089d0 0x7f256019e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f255effd640 1 -- 172.21.15.16:0/3963784707 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2560108080 msgr2=0x7f2560199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f255effd640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2560108080 0x7f2560199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f255effd640 1 -- 172.21.15.16:0/3963784707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25601089d0 msgr2=0x7f256019e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f255effd640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25601089d0 0x7f256019e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f255effd640 1 -- 172.21.15.16:0/3963784707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256019f010 con 0x7f2560102750 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f255effd640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2560102750 0x7f25601990e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f255000e9f0 tx=0x7f255000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f25648d8640 1 -- 172.21.15.16:0/3963784707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255000cd80 con 0x7f2560102750 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f25648d8640 1 -- 172.21.15.16:0/3963784707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25500108e0 con 0x7f2560102750 2024-08-28T19:21:03.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f25658da640 1 -- 172.21.15.16:0/3963784707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256019f300 con 0x7f2560102750 2024-08-28T19:21:03.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f25648d8640 1 -- 172.21.15.16:0/3963784707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2550018980 con 0x7f2560102750 2024-08-28T19:21:03.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.700+0000 7f25658da640 1 -- 172.21.15.16:0/3963784707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25601a5a90 con 0x7f2560102750 2024-08-28T19:21:03.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.701+0000 7f25658da640 1 -- 172.21.15.16:0/3963784707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2560102b60 con 0x7f2560102750 2024-08-28T19:21:03.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.703+0000 7f25648d8640 1 -- 172.21.15.16:0/3963784707 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7f2550010430 con 0x7f2560102750 2024-08-28T19:21:03.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.704+0000 7f25648d8640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25480b4b80 0x7f25480b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.704+0000 7f255e7fc640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25480b4b80 0x7f25480b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.704+0000 7f25648d8640 1 -- 172.21.15.16:0/3963784707 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f25500e5e70 con 0x7f2560102750 2024-08-28T19:21:03.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.704+0000 7f255e7fc640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25480b4b80 0x7f25480b7090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f256007a930 tx=0x7f2554005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:03.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.707+0000 7f25648d8640 1 -- 172.21.15.16:0/3963784707 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f25500ac460 con 0x7f2560102750 2024-08-28T19:21:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.833+0000 7fa2dca0d640 1 -- 172.21.15.16:0/503477860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d8102880 msgr2=0x7fa2d8102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.833+0000 7fa2dca0d640 1 --2- 172.21.15.16:0/503477860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d8102880 0x7fa2d8102c90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009900 tx=0x7fa2c802a410 comp rx=0 tx=0).stop 2024-08-28T19:21:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.833+0000 7fa2dca0d640 1 -- 172.21.15.16:0/503477860 shutdown_connections 2024-08-28T19:21:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.833+0000 7fa2dca0d640 1 --2- 172.21.15.16:0/503477860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d8108ac0 0x7fa2d810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.833+0000 7fa2dca0d640 1 --2- 172.21.15.16:0/503477860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d8108170 0x7fa2d8108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.833+0000 7fa2dca0d640 1 --2- 172.21.15.16:0/503477860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d8102880 0x7fa2d8102c90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.833+0000 7fa2dca0d640 1 -- 172.21.15.16:0/503477860 >> 172.21.15.16:0/503477860 conn(0x7fa2d8079530 msgr2=0x7fa2d8079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.833+0000 7fa2dca0d640 1 -- 172.21.15.16:0/503477860 shutdown_connections 2024-08-28T19:21:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.834+0000 7fa2dca0d640 1 -- 172.21.15.16:0/503477860 wait complete. 2024-08-28T19:21:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.834+0000 7fa2dca0d640 1 Processor -- start 2024-08-28T19:21:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2dca0d640 1 -- start start 2024-08-28T19:21:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2dca0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d8102880 0x7fa2d819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2dca0d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d8108170 0x7fa2d819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2dca0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d8108ac0 0x7fa2d819fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2dca0d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81138a0 con 0x7fa2d8108ac0 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2dca0d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8113720 con 0x7fa2d8102880 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2dca0d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8113a20 con 0x7fa2d8108170 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2d6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d8108ac0 0x7fa2d819fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d8102880 0x7fa2d819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2d6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d8108ac0 0x7fa2d819fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37236/0 (socket says 172.21.15.16:37236) 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d8102880 0x7fa2d819a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:40568/0 (socket says 172.21.15.16:40568) 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2d6d9d640 1 -- 172.21.15.16:0/2388335081 learned_addr learned my addr 172.21.15.16:0/2388335081 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2cffff640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d8108170 0x7fa2d819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2d6d9d640 1 -- 172.21.15.16:0/2388335081 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d8108170 msgr2=0x7fa2d819a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2d6d9d640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d8108170 0x7fa2d819a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2d6d9d640 1 -- 172.21.15.16:0/2388335081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d8102880 msgr2=0x7fa2d819a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2d6d9d640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d8102880 0x7fa2d819a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2d6d9d640 1 -- 172.21.15.16:0/2388335081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d81a0180 con 0x7fa2d8108ac0 2024-08-28T19:21:03.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.835+0000 7fa2d659c640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d8102880 0x7fa2d819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:03.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.836+0000 7fa2d6d9d640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d8108ac0 0x7fa2d819fa50 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000eb10 tx=0x7fa2d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:03.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.836+0000 7fa2cf7fe640 1 -- 172.21.15.16:0/2388335081 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0004b20 con 0x7fa2d8108ac0 2024-08-28T19:21:03.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.836+0000 7fa2dca0d640 1 -- 172.21.15.16:0/2388335081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d81a0470 con 0x7fa2d8108ac0 2024-08-28T19:21:03.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.836+0000 7fa2cf7fe640 1 -- 172.21.15.16:0/2388335081 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2d0004cc0 con 0x7fa2d8108ac0 2024-08-28T19:21:03.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.836+0000 7fa2cf7fe640 1 -- 172.21.15.16:0/2388335081 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0010670 con 0x7fa2d8108ac0 2024-08-28T19:21:03.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.836+0000 7fa2dca0d640 1 -- 172.21.15.16:0/2388335081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d81a6bd0 con 0x7fa2d8108ac0 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.839+0000 7fa2e2f82740 1 Processor -- start 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.840+0000 7fa2cf7fe640 1 -- 172.21.15.16:0/2388335081 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fa2d0005020 con 0x7fa2d8108ac0 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.840+0000 7fa2e2f82740 1 -- start start 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.840+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abf8350 0x55e12af9e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abcf490 0x55e12af9ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abd7670 0x55e12af9f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e12af145f0 con 0x55e12abf8350 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e12af03b80 con 0x55e12abcf490 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e12af07b30 con 0x55e12abd7670 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abf8350 0x55e12af9e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abf8350 0x55e12af9e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37244/0 (socket says 172.21.15.16:37244) 2024-08-28T19:21:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cffff640 1 -- 172.21.15.16:0/1506519647 learned_addr learned my addr 172.21.15.16:0/1506519647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cffff640 1 --2- 172.21.15.16:0/1506519647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abd7670 0x55e12af9f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cf7fe640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c40b4b80 0x7fa2c40b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cffff640 1 -- 172.21.15.16:0/1506519647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abd7670 msgr2=0x55e12af9f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cffff640 1 --2- 172.21.15.16:0/1506519647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abd7670 0x55e12af9f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cffff640 1 -- 172.21.15.16:0/1506519647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abcf490 msgr2=0x55e12af9ecc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cffff640 1 --2- 172.21.15.16:0/1506519647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abcf490 0x55e12af9ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cffff640 1 -- 172.21.15.16:0/1506519647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e12ac63930 con 0x55e12abf8350 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cf7fe640 1 -- 172.21.15.16:0/2388335081 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa2d00e5b50 con 0x7fa2d8108ac0 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2d6d9d640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c40b4b80 0x7fa2c40b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.841+0000 7fa2cffff640 1 --2- 172.21.15.16:0/1506519647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abf8350 0x55e12af9e780 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00125c0 tx=0x7fa2c000ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad565fb14e0a162 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.842+0000 7fa2ccff9640 1 -- 172.21.15.16:0/1506519647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0004020 con 0x55e12abf8350 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.842+0000 7fa2ccff9640 1 -- 172.21.15.16:0/1506519647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c00054d0 con 0x55e12abf8350 2024-08-28T19:21:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.842+0000 7fa2ccff9640 1 -- 172.21.15.16:0/1506519647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0004a50 con 0x55e12abf8350 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1506519647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abf8350 msgr2=0x55e12af9e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2d6d9d640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c40b4b80 0x7fa2c40b7090 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa2d00a9360 tx=0x7fa2d000cee0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/1506519647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abf8350 0x55e12af9e780 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00125c0 tx=0x7fa2c000ba40 comp rx=0 tx=0).stop 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1506519647 shutdown_connections 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/1506519647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abd7670 0x55e12af9f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/1506519647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abcf490 0x55e12af9ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/1506519647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abf8350 0x55e12af9e780 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1506519647 >> 172.21.15.16:0/1506519647 conn(0x55e12abec740 msgr2=0x55e12af9c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1506519647 shutdown_connections 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1506519647 wait complete. 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2e2f82740 1 -- start start 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.843+0000 7fa2e2f82740 1 Processor -- start 2024-08-28T19:21:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afa9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12afaa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 0x55e12afb6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e12ad66890 con 0x55e12abcf490 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e12ae8a790 con 0x55e12abd7670 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e12afacf60 con 0x55e12abf8350 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afa9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2e2f82740 10 client.0 _ll_register_callbacks cb 0x55e12ac0fba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afa9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37246/0 (socket says 172.21.15.16:37246) 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2cffff640 1 -- 172.21.15.16:0/2142687794 learned_addr learned my addr 172.21.15.16:0/2142687794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2d659c640 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12afaa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2cffff640 1 -- 172.21.15.16:0/2142687794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 msgr2=0x55e12afb6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2cffff640 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 0x55e12afb6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2cffff640 1 -- 172.21.15.16:0/2142687794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 msgr2=0x55e12afaa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2cffff640 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12afaa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2cffff640 1 -- 172.21.15.16:0/2142687794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e12aec1f30 con 0x55e12abcf490 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2cffff640 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afa9ef0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0006010 tx=0x7fa2c0012300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e6f3f1d999cea5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.844+0000 7fa2bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2bdffb640 1 -- 172.21.15.16:0/2142687794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0010870 con 0x55e12abcf490 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2bdffb640 1 -- 172.21.15.16:0/2142687794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa2c0010a10 con 0x55e12abcf490 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2e2f82740 1 -- 172.21.15.16:0/2142687794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2c00054d0 con 0x55e12abcf490 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2bdffb640 1 -- 172.21.15.16:0/2142687794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa2c0034050 con 0x55e12abcf490 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2e2f82740 10 client.15621 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2e2f82740 1 -- 172.21.15.16:0/2142687794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e12afb8cd0 con 0x55e12abcf490 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2bdffb640 1 -- 172.21.15.16:0/2142687794 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c00054d0 con 0x55e12abcf490 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2bdffb640 1 -- 172.21.15.16:0/2142687794 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa2c002ba40 con 0x55e12abcf490 2024-08-28T19:21:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2e2f82740 10 client.15621 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:03.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2e2f82740 10 client.15621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:03.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2e2f82740 20 client.15621 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:03.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e12abb6cd0 0x55e12afbc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.845+0000 7fa2e2f82740 4 client.15621 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:03.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.846+0000 7fa2e2f82740 1 -- 172.21.15.16:0/2142687794 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e12af16540 con 0x55e12abb6cd0 2024-08-28T19:21:03.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.846+0000 7fa2d659c640 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e12abb6cd0 0x55e12afbc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.846+0000 7fa2d659c640 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e12abb6cd0 0x55e12afbc030 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49d0678b7b3ba323 server_cookie=2d9342a787b185c in_seq=0 out_seq=0 2024-08-28T19:21:03.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.846+0000 7fa2bdffb640 10 client.15621 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:03.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2bdffb640 1 -- 172.21.15.16:0/2142687794 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e12af16540 con 0x55e12abb6cd0 2024-08-28T19:21:03.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2bdffb640 10 client.15621 handle_command_reply: tid=0 2024-08-28T19:21:03.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 1 client.15621 shutdown 2024-08-28T19:21:03.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 1 -- 172.21.15.16:0/2142687794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 msgr2=0x55e12afa9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afa9ef0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0006010 tx=0x7fa2c0012300 comp rx=0 tx=0).stop 2024-08-28T19:21:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 1 -- 172.21.15.16:0/2142687794 shutdown_connections 2024-08-28T19:21:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e12abb6cd0 0x55e12afbc030 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 0x55e12afb6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12afaa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/2142687794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afa9ef0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2bdffb640 0 client.15621 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 1 -- 172.21.15.16:0/2142687794 >> 172.21.15.16:0/2142687794 conn(0x55e12abec740 msgr2=0x55e12afa1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 1 -- 172.21.15.16:0/2142687794 shutdown_connections 2024-08-28T19:21:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 1 -- 172.21.15.16:0/2142687794 wait complete. 2024-08-28T19:21:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.847+0000 7fa2e2f82740 20 client.15621 trim_cache size 0 max 16384 2024-08-28T19:21:03.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.865+0000 7fa2e2f82740 1 Processor -- start 2024-08-28T19:21:03.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.865+0000 7fa2e2f82740 1 -- start start 2024-08-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12af9bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12af9c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 0x55e12afa70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b0c00 con 0x55e12abcf490 2024-08-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b2740 con 0x55e12abd7670 2024-08-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b12d0 con 0x55e12abf8350 2024-08-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12af9bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12af9bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37260/0 (socket says 172.21.15.16:37260) 2024-08-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2cffff640 1 -- 172.21.15.16:0/3087060021 learned_addr learned my addr 172.21.15.16:0/3087060021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2d659c640 1 --2- 172.21.15.16:0/3087060021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12af9c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3087060021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 0x55e12afa70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2cffff640 1 -- 172.21.15.16:0/3087060021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 msgr2=0x55e12afa70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:03.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3087060021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 0x55e12afa70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2cffff640 1 -- 172.21.15.16:0/3087060021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 msgr2=0x55e12af9c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:03.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3087060021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12af9c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2cffff640 1 -- 172.21.15.16:0/3087060021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e12afac2c0 con 0x55e12abcf490 2024-08-28T19:21:03.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2d659c640 1 --2- 172.21.15.16:0/3087060021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12af9c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:03.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2cffff640 1 --2- 172.21.15.16:0/3087060021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12af9bca0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00058f0 tx=0x7fa2c0005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a48e2fc453aea3b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:03.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2bffff640 1 -- 172.21.15.16:0/3087060021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c000b810 con 0x55e12abcf490 2024-08-28T19:21:03.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2bffff640 1 -- 172.21.15.16:0/3087060021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c0026070 con 0x55e12abcf490 2024-08-28T19:21:03.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.866+0000 7fa2bffff640 1 -- 172.21.15.16:0/3087060021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c00108d0 con 0x55e12abcf490 2024-08-28T19:21:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.867+0000 7fa2e2f82740 1 -- 172.21.15.16:0/3087060021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 msgr2=0x55e12af9bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.867+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/3087060021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12af9bca0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00058f0 tx=0x7fa2c0005bf0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.867+0000 7fa2e2f82740 1 -- 172.21.15.16:0/3087060021 shutdown_connections 2024-08-28T19:21:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.867+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/3087060021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 0x55e12afa70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.867+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/3087060021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12af9c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.867+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/3087060021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12af9bca0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.867+0000 7fa2e2f82740 1 -- 172.21.15.16:0/3087060021 >> 172.21.15.16:0/3087060021 conn(0x55e12abec740 msgr2=0x55e12af9fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.867+0000 7fa2e2f82740 1 -- 172.21.15.16:0/3087060021 shutdown_connections 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.867+0000 7fa2e2f82740 1 -- 172.21.15.16:0/3087060021 wait complete. 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2e2f82740 1 -- start start 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2e2f82740 1 Processor -- start 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afae9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12afaef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2e2f82740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 0x55e12b025890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b2070 con 0x55e12abcf490 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b19a0 con 0x55e12abd7670 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2e2f82740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81aebe0 con 0x55e12abf8350 2024-08-28T19:21:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2e2f82740 10 client.0 _ll_register_callbacks cb 0x7fa2c00124a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afae9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afae9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37264/0 (socket says 172.21.15.16:37264) 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.868+0000 7fa2cffff640 1 -- 172.21.15.16:0/1055386209 learned_addr learned my addr 172.21.15.16:0/1055386209 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2cffff640 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 0x55e12b025890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2d659c640 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12afaef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2cffff640 1 -- 172.21.15.16:0/1055386209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 msgr2=0x55e12b025890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2cffff640 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 0x55e12b025890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2cffff640 1 -- 172.21.15.16:0/1055386209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 msgr2=0x55e12afaef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2cffff640 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12afaef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2cffff640 1 -- 172.21.15.16:0/1055386209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e12b025ea0 con 0x55e12abcf490 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2cffff640 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afae9a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa2c002be40 tx=0x7fa2c002bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19fd0dab0124d0bd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2bdffb640 1 -- 172.21.15.16:0/1055386209 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0042040 con 0x55e12abcf490 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2bdffb640 1 -- 172.21.15.16:0/1055386209 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa2c002b880 con 0x55e12abcf490 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1055386209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2c0026070 con 0x55e12abcf490 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2bdffb640 1 -- 172.21.15.16:0/1055386209 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa2c004c050 con 0x55e12abcf490 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2e2f82740 10 client.15627 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.869+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1055386209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e12b027160 con 0x55e12abcf490 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.870+0000 7fa2bdffb640 1 -- 172.21.15.16:0/1055386209 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c0026070 con 0x55e12abcf490 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.870+0000 7fa2bdffb640 1 -- 172.21.15.16:0/1055386209 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa2c000e520 con 0x55e12abcf490 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.870+0000 7fa2e2f82740 10 client.15627 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.870+0000 7fa2e2f82740 10 client.15627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.870+0000 7fa2e2f82740 20 client.15627 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.870+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e12ab8c080 0x55e12b0297b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.870+0000 7fa2e2f82740 4 client.15627 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:21:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.870+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1055386209 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e12af11fe0 con 0x55e12ab8c080 2024-08-28T19:21:03.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.870+0000 7fa2d659c640 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e12ab8c080 0x55e12b0297b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:03.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.871+0000 7fa2d659c640 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e12ab8c080 0x55e12b0297b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a79f0218d20eafe server_cookie=c60e19edf75ba471 in_seq=0 out_seq=0 2024-08-28T19:21:03.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.871+0000 7fa2bdffb640 10 client.15627 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:03.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2bdffb640 1 -- 172.21.15.16:0/1055386209 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e12af11fe0 con 0x55e12ab8c080 2024-08-28T19:21:03.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2bdffb640 10 client.15627 handle_command_reply: tid=0 2024-08-28T19:21:03.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2e2f82740 1 client.15627 shutdown 2024-08-28T19:21:03.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1055386209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 msgr2=0x55e12afae9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afae9a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa2c002be40 tx=0x7fa2c002bcd0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1055386209 shutdown_connections 2024-08-28T19:21:03.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e12ab8c080 0x55e12b0297b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e12abf8350 0x55e12b025890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12abd7670 0x55e12afaef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2e2f82740 1 --2- 172.21.15.16:0/1055386209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12abcf490 0x55e12afae9a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1055386209 >> 172.21.15.16:0/1055386209 conn(0x55e12abec740 msgr2=0x55e12afa3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:03.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2bdffb640 0 client.15627 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:03.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1055386209 shutdown_connections 2024-08-28T19:21:03.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.873+0000 7fa2e2f82740 1 -- 172.21.15.16:0/1055386209 wait complete. 2024-08-28T19:21:03.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2e2f82740 20 client.15627 trim_cache size 0 max 16384 2024-08-28T19:21:03.878 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:21:03.878 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:21:03.879 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "b842a7f8-9a3b-435e-88a7-49e794ad2bb8", 2024-08-28T19:21:03.879 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:21:03.879 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:21:03.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 -- 172.21.15.16:0/2388335081 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c40b4b80 msgr2=0x7fa2c40b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c40b4b80 0x7fa2c40b7090 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa2d00a9360 tx=0x7fa2d000cee0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 -- 172.21.15.16:0/2388335081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d8108ac0 msgr2=0x7fa2d819fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d8108ac0 0x7fa2d819fa50 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000eb10 tx=0x7fa2d000c2d0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 -- 172.21.15.16:0/2388335081 shutdown_connections 2024-08-28T19:21:03.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d8108ac0 0x7fa2d819fa50 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2d8108170 0x7fa2d819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa2c40b4b80 0x7fa2c40b7090 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 --2- 172.21.15.16:0/2388335081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d8102880 0x7fa2d819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 -- 172.21.15.16:0/2388335081 >> 172.21.15.16:0/2388335081 conn(0x7fa2d8079530 msgr2=0x7fa2d8106200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:03.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 -- 172.21.15.16:0/2388335081 shutdown_connections 2024-08-28T19:21:03.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.874+0000 7fa2dca0d640 1 -- 172.21.15.16:0/2388335081 wait complete. 2024-08-28T19:21:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.928+0000 7f25658da640 1 -- 172.21.15.16:0/3963784707 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f2560063640 con 0x7f25480b4b80 2024-08-28T19:21:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.988+0000 7f25648d8640 1 -- 172.21.15.16:0/3963784707 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f2560063640 con 0x7f25480b4b80 2024-08-28T19:21:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:Retention added to path / 2024-08-28T19:21:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.990+0000 7f25658da640 1 -- 172.21.15.16:0/3963784707 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25480b4b80 msgr2=0x7f25480b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.990+0000 7f25658da640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25480b4b80 0x7f25480b7090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f256007a930 tx=0x7f2554005f90 comp rx=0 tx=0).stop 2024-08-28T19:21:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.990+0000 7f25658da640 1 -- 172.21.15.16:0/3963784707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2560102750 msgr2=0x7f25601990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.990+0000 7f25658da640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2560102750 0x7f25601990e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f255000e9f0 tx=0x7f255000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.991+0000 7f25658da640 1 -- 172.21.15.16:0/3963784707 shutdown_connections 2024-08-28T19:21:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.991+0000 7f25658da640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25480b4b80 0x7f25480b7090 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.991+0000 7f25658da640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25601089d0 0x7f256019e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.991+0000 7f25658da640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2560108080 0x7f2560199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.991+0000 7f25658da640 1 --2- 172.21.15.16:0/3963784707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2560102750 0x7f25601990e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.991+0000 7f25658da640 1 -- 172.21.15.16:0/3963784707 >> 172.21.15.16:0/3963784707 conn(0x7f256007a4e0 msgr2=0x7f2560106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.991+0000 7f25658da640 1 -- 172.21.15.16:0/3963784707 shutdown_connections 2024-08-28T19:21:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:03.991+0000 7f25658da640 1 -- 172.21.15.16:0/3963784707 wait complete. 2024-08-28T19:21:04.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-08-28T19:21:04.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.116+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2590346730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb8c108080 msgr2=0x7fdb8c108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:04.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.116+0000 7fdb90ea7640 1 --2- 172.21.15.16:0/2590346730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb8c108080 0x7fdb8c108490 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdb80009900 tx=0x7fdb8002a410 comp rx=0 tx=0).stop 2024-08-28T19:21:04.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.116+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2590346730 shutdown_connections 2024-08-28T19:21:04.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.116+0000 7fdb90ea7640 1 --2- 172.21.15.16:0/2590346730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c1089d0 0x7fdb8c10f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.116+0000 7fdb90ea7640 1 --2- 172.21.15.16:0/2590346730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb8c108080 0x7fdb8c108490 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.116+0000 7fdb90ea7640 1 --2- 172.21.15.16:0/2590346730 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb8c102750 0x7fdb8c102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.116+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2590346730 >> 172.21.15.16:0/2590346730 conn(0x7fdb8c07a4e0 msgr2=0x7fdb8c07a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:04.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.116+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2590346730 shutdown_connections 2024-08-28T19:21:04.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.116+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2590346730 wait complete. 2024-08-28T19:21:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb90ea7640 1 Processor -- start 2024-08-28T19:21:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb90ea7640 1 -- start start 2024-08-28T19:21:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb90ea7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c102750 0x7fdb8c19afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb90ea7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb8c108080 0x7fdb8c19b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb90ea7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb8c1089d0 0x7fdb8c19ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb90ea7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c112690 con 0x7fdb8c108080 2024-08-28T19:21:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb90ea7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c112510 con 0x7fdb8c102750 2024-08-28T19:21:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb90ea7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c112810 con 0x7fdb8c1089d0 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb89d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb8c108080 0x7fdb8c19b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb89d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb8c108080 0x7fdb8c19b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37280/0 (socket says 172.21.15.16:37280) 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb89d9b640 1 -- 172.21.15.16:0/2543524549 learned_addr learned my addr 172.21.15.16:0/2543524549 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb8a59c640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c102750 0x7fdb8c19afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.117+0000 7fdb8ad9d640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb8c1089d0 0x7fdb8c19ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.118+0000 7fdb89d9b640 1 -- 172.21.15.16:0/2543524549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb8c1089d0 msgr2=0x7fdb8c19ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.118+0000 7fdb89d9b640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb8c1089d0 0x7fdb8c19ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.118+0000 7fdb89d9b640 1 -- 172.21.15.16:0/2543524549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c102750 msgr2=0x7fdb8c19afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.118+0000 7fdb89d9b640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c102750 0x7fdb8c19afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.118+0000 7fdb89d9b640 1 -- 172.21.15.16:0/2543524549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb8c1a6250 con 0x7fdb8c108080 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.118+0000 7fdb89d9b640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb8c108080 0x7fdb8c19b500 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdb80005ab0 tx=0x7fdb80003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.118+0000 7fdb7b7fe640 1 -- 172.21.15.16:0/2543524549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8003f070 con 0x7fdb8c108080 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.118+0000 7fdb7b7fe640 1 -- 172.21.15.16:0/2543524549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb80031d60 con 0x7fdb8c108080 2024-08-28T19:21:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.118+0000 7fdb7b7fe640 1 -- 172.21.15.16:0/2543524549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb80035030 con 0x7fdb8c108080 2024-08-28T19:21:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.118+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2543524549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb8c1a64e0 con 0x7fdb8c108080 2024-08-28T19:21:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.118+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2543524549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb8c1a6a20 con 0x7fdb8c108080 2024-08-28T19:21:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.119+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2543524549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb8c102b60 con 0x7fdb8c108080 2024-08-28T19:21:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.121+0000 7fdb7b7fe640 1 -- 172.21.15.16:0/2543524549 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 152594+0+0 (secure 0 0 0) 0x7fdb80042070 con 0x7fdb8c108080 2024-08-28T19:21:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.121+0000 7fdb7b7fe640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb740b4c00 0x7fdb740b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.121+0000 7fdb8a59c640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb740b4c00 0x7fdb740b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.122+0000 7fdb7b7fe640 1 -- 172.21.15.16:0/2543524549 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdb80106410 con 0x7fdb8c108080 2024-08-28T19:21:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.122+0000 7fdb8a59c640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb740b4c00 0x7fdb740b7110 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c06b4e0 tx=0x7fdb7c00b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.125+0000 7fdb7b7fe640 1 -- 172.21.15.16:0/2543524549 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fdb800cd0b0 con 0x7fdb8c108080 2024-08-28T19:21:04.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.311+0000 7fdb7b7fe640 1 -- 172.21.15.16:0/2543524549 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7fdb800c8c40 con 0x7fdb8c108080 2024-08-28T19:21:04.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.337+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2543524549 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fdb8c0635a0 con 0x7fdb740b4c00 2024-08-28T19:21:04.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.344+0000 7fdb7b7fe640 1 -- 172.21.15.16:0/2543524549 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fdb8c0635a0 con 0x7fdb740b4c00 2024-08-28T19:21:04.345 INFO:teuthology.orchestra.run.smithi016.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-08-28T00:00:00", "created": "2024-08-28T19:21:03", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-08-28T19:21:04.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2543524549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb740b4c00 msgr2=0x7fdb740b7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:04.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb740b4c00 0x7fdb740b7110 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c06b4e0 tx=0x7fdb7c00b390 comp rx=0 tx=0).stop 2024-08-28T19:21:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2543524549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb8c108080 msgr2=0x7fdb8c19b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb8c108080 0x7fdb8c19b500 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdb80005ab0 tx=0x7fdb80003cd0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2543524549 shutdown_connections 2024-08-28T19:21:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb8c1089d0 0x7fdb8c19ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb8c108080 0x7fdb8c19b500 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb740b4c00 0x7fdb740b7110 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 --2- 172.21.15.16:0/2543524549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c102750 0x7fdb8c19afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2543524549 >> 172.21.15.16:0/2543524549 conn(0x7fdb8c07a4e0 msgr2=0x7fdb8c106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2543524549 shutdown_connections 2024-08-28T19:21:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.346+0000 7fdb90ea7640 1 -- 172.21.15.16:0/2543524549 wait complete. 2024-08-28T19:21:04.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-08-28T19:21:04.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.471+0000 7f6382244640 1 -- 172.21.15.16:0/2053944257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c108a00 msgr2=0x7f637c10f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:04.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.471+0000 7f6382244640 1 --2- 172.21.15.16:0/2053944257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c108a00 0x7f637c10f230 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6374009ca0 tx=0x7f637402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.471+0000 7f6382244640 1 -- 172.21.15.16:0/2053944257 shutdown_connections 2024-08-28T19:21:04.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.471+0000 7f6382244640 1 --2- 172.21.15.16:0/2053944257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c108a00 0x7f637c10f230 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.471+0000 7f6382244640 1 --2- 172.21.15.16:0/2053944257 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f637c1080b0 0x7f637c1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.471+0000 7f6382244640 1 --2- 172.21.15.16:0/2053944257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637c102780 0x7f637c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.471+0000 7f6382244640 1 -- 172.21.15.16:0/2053944257 >> 172.21.15.16:0/2053944257 conn(0x7f637c0fe160 msgr2=0x7f637c1005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:04.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.471+0000 7f6382244640 1 -- 172.21.15.16:0/2053944257 shutdown_connections 2024-08-28T19:21:04.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.471+0000 7f6382244640 1 -- 172.21.15.16:0/2053944257 wait complete. 2024-08-28T19:21:04.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.471+0000 7f6382244640 1 Processor -- start 2024-08-28T19:21:04.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f6382244640 1 -- start start 2024-08-28T19:21:04.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f6382244640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f637c102780 0x7f637c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:04.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f6382244640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637c1080b0 0x7f637c19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:04.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f6382244640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c108a00 0x7f637c19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f6382244640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f637c1126d0 con 0x7f637c108a00 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f6382244640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f637c112550 con 0x7f637c1080b0 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f6382244640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f637c112850 con 0x7f637c102780 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f637bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c108a00 0x7f637c19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f637bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c108a00 0x7f637c19fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45212/0 (socket says 172.21.15.16:45212) 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f637bfff640 1 -- 172.21.15.16:0/3323212048 learned_addr learned my addr 172.21.15.16:0/3323212048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f637affd640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637c1080b0 0x7f637c19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f637b7fe640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f637c102780 0x7f637c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f637bfff640 1 -- 172.21.15.16:0/3323212048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f637c102780 msgr2=0x7f637c19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f637bfff640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f637c102780 0x7f637c19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f637bfff640 1 -- 172.21.15.16:0/3323212048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637c1080b0 msgr2=0x7f637c19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f637bfff640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637c1080b0 0x7f637c19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.472+0000 7f637bfff640 1 -- 172.21.15.16:0/3323212048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637c1a0150 con 0x7f637c108a00 2024-08-28T19:21:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.473+0000 7f637bfff640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c108a00 0x7f637c19fa50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6374005cc0 tx=0x7f6374031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.473+0000 7f6378ff9640 1 -- 172.21.15.16:0/3323212048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6374003b70 con 0x7f637c108a00 2024-08-28T19:21:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.473+0000 7f6378ff9640 1 -- 172.21.15.16:0/3323212048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63740318f0 con 0x7f637c108a00 2024-08-28T19:21:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.473+0000 7f6382244640 1 -- 172.21.15.16:0/3323212048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637c1a03e0 con 0x7f637c108a00 2024-08-28T19:21:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.473+0000 7f6378ff9640 1 -- 172.21.15.16:0/3323212048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637402f950 con 0x7f637c108a00 2024-08-28T19:21:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.473+0000 7f6382244640 1 -- 172.21.15.16:0/3323212048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f637c1a6bd0 con 0x7f637c108a00 2024-08-28T19:21:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.474+0000 7f6382244640 1 -- 172.21.15.16:0/3323212048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f637c102b90 con 0x7f637c108a00 2024-08-28T19:21:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.476+0000 7f6378ff9640 1 -- 172.21.15.16:0/3323212048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f637402fb10 con 0x7f637c108a00 2024-08-28T19:21:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.476+0000 7f6378ff9640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63640b4c10 0x7f63640b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.477+0000 7f637b7fe640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63640b4c10 0x7f63640b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.477+0000 7f6378ff9640 1 -- 172.21.15.16:0/3323212048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6374106ae0 con 0x7f637c108a00 2024-08-28T19:21:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.477+0000 7f637b7fe640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63640b4c10 0x7f63640b7120 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f636c009bd0 tx=0x7f636c005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:04.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.479+0000 7f6378ff9640 1 -- 172.21.15.16:0/3323212048 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f63740cd620 con 0x7f637c108a00 2024-08-28T19:21:04.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:04 smithi028 ceph-mon[21769]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:04.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:04 smithi028 ceph-mon[21769]: pgmap v340: 129 pgs: 129 active+clean; 652 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-08-28T19:21:04.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:04 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:21:04.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:04 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:04.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:04 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:21:04.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:04 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:04.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:04 smithi028 ceph-mon[21769]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:04.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:04 smithi016 ceph-mon[18556]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:04.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:04 smithi016 ceph-mon[18556]: pgmap v340: 129 pgs: 129 active+clean; 652 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-08-28T19:21:04.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:04 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:21:04.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:04 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:04.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:04 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:21:04.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:04 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:04.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:04 smithi016 ceph-mon[18556]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:04.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:04 smithi115 ceph-mon[22184]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:04.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:04 smithi115 ceph-mon[22184]: pgmap v340: 129 pgs: 129 active+clean; 652 KiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-08-28T19:21:04.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:04 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:21:04.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:04 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:04.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:04 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:21:04.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:04 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:04.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:04 smithi115 ceph-mon[22184]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:04.639 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:21:04.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.692+0000 7f6382244640 1 -- 172.21.15.16:0/3323212048 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f637c0635a0 con 0x7f63640b4c10 2024-08-28T19:21:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.698+0000 7f6378ff9640 1 -- 172.21.15.16:0/3323212048 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f637c0635a0 con 0x7f63640b4c10 2024-08-28T19:21:04.700 INFO:teuthology.orchestra.run.smithi016.stdout:/ 1m 6m3h 2024-08-28T19:21:04.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.700+0000 7f6382244640 1 -- 172.21.15.16:0/3323212048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63640b4c10 msgr2=0x7f63640b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:04.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.700+0000 7f6382244640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63640b4c10 0x7f63640b7120 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f636c009bd0 tx=0x7f636c005f90 comp rx=0 tx=0).stop 2024-08-28T19:21:04.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.700+0000 7f6382244640 1 -- 172.21.15.16:0/3323212048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c108a00 msgr2=0x7f637c19fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:04.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.700+0000 7f6382244640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c108a00 0x7f637c19fa50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6374005cc0 tx=0x7f6374031700 comp rx=0 tx=0).stop 2024-08-28T19:21:04.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.700+0000 7f6382244640 1 -- 172.21.15.16:0/3323212048 shutdown_connections 2024-08-28T19:21:04.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.701+0000 7f6382244640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c108a00 0x7f637c19fa50 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.701+0000 7f6382244640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63640b4c10 0x7f63640b7120 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.701+0000 7f6382244640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637c1080b0 0x7f637c19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.701+0000 7f6382244640 1 --2- 172.21.15.16:0/3323212048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f637c102780 0x7f637c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:04.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.701+0000 7f6382244640 1 -- 172.21.15.16:0/3323212048 >> 172.21.15.16:0/3323212048 conn(0x7f637c0fe160 msgr2=0x7f637c101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:04.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.701+0000 7f6382244640 1 -- 172.21.15.16:0/3323212048 shutdown_connections 2024-08-28T19:21:04.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:04.701+0000 7f6382244640 1 -- 172.21.15.16:0/3323212048 wait complete. 2024-08-28T19:21:04.714 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-08-28T19:21:04.739 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-08-28T19:21:04.750 INFO:teuthology.run_tasks:Running task quiescer... 2024-08-28T19:21:04.761 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-28T19:21:04.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-28T19:21:04.895 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:05.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.409+0000 7f50815a0640 1 -- 172.21.15.16:0/479618804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f507c102610 msgr2=0x7f507c102a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:05.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.409+0000 7f50815a0640 1 --2- 172.21.15.16:0/479618804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f507c102610 0x7f507c102a20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f506c009900 tx=0x7f506c02a410 comp rx=0 tx=0).stop 2024-08-28T19:21:05.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.409+0000 7f50815a0640 1 -- 172.21.15.16:0/479618804 shutdown_connections 2024-08-28T19:21:05.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.409+0000 7f50815a0640 1 --2- 172.21.15.16:0/479618804 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f507c1088a0 0x7f507c10f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.409+0000 7f50815a0640 1 --2- 172.21.15.16:0/479618804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c107f50 0x7f507c108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.410+0000 7f50815a0640 1 --2- 172.21.15.16:0/479618804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f507c102610 0x7f507c102a20 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.410+0000 7f50815a0640 1 -- 172.21.15.16:0/479618804 >> 172.21.15.16:0/479618804 conn(0x7f507c0fe060 msgr2=0x7f507c1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:05.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.410+0000 7f50815a0640 1 -- 172.21.15.16:0/479618804 shutdown_connections 2024-08-28T19:21:05.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.410+0000 7f50815a0640 1 -- 172.21.15.16:0/479618804 wait complete. 2024-08-28T19:21:05.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.410+0000 7f50815a0640 1 Processor -- start 2024-08-28T19:21:05.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.410+0000 7f50815a0640 1 -- start start 2024-08-28T19:21:05.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f50815a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f507c102610 0x7f507c117e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f50815a0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f507c107f50 0x7f507c1120d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f50815a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c1088a0 0x7f507c1126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f50815a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f507c1105e0 con 0x7f507c102610 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f50815a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f507c110460 con 0x7f507c1088a0 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f50815a0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f507c110760 con 0x7f507c107f50 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f507bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f507c102610 0x7f507c117e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f507bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f507c102610 0x7f507c117e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45222/0 (socket says 172.21.15.16:45222) 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f507bfff640 1 -- 172.21.15.16:0/2419717139 learned_addr learned my addr 172.21.15.16:0/2419717139 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f507b7fe640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f507c107f50 0x7f507c1120d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f5080d9f640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c1088a0 0x7f507c1126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f507bfff640 1 -- 172.21.15.16:0/2419717139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f507c107f50 msgr2=0x7f507c1120d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f507bfff640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f507c107f50 0x7f507c1120d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f507bfff640 1 -- 172.21.15.16:0/2419717139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c1088a0 msgr2=0x7f507c1126a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f507bfff640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c1088a0 0x7f507c1126a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.411+0000 7f507bfff640 1 -- 172.21.15.16:0/2419717139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507c112ec0 con 0x7f507c102610 2024-08-28T19:21:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.412+0000 7f507bfff640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f507c102610 0x7f507c117e10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f506c005ab0 tx=0x7f506c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.412+0000 7f50797fa640 1 -- 172.21.15.16:0/2419717139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506c03f070 con 0x7f507c102610 2024-08-28T19:21:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.412+0000 7f50797fa640 1 -- 172.21.15.16:0/2419717139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f506c031b50 con 0x7f507c102610 2024-08-28T19:21:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.412+0000 7f50815a0640 1 -- 172.21.15.16:0/2419717139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f507c1b5200 con 0x7f507c102610 2024-08-28T19:21:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.412+0000 7f50797fa640 1 -- 172.21.15.16:0/2419717139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506c035030 con 0x7f507c102610 2024-08-28T19:21:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.412+0000 7f50815a0640 1 -- 172.21.15.16:0/2419717139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f507c1b5620 con 0x7f507c102610 2024-08-28T19:21:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f5088d77740 1 Processor -- start 2024-08-28T19:21:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f50797fa640 1 -- 172.21.15.16:0/2419717139 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f506c042070 con 0x7f507c102610 2024-08-28T19:21:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f5088d77740 1 -- start start 2024-08-28T19:21:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f5088d77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da976070 0x5639dad1b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f5088d77740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad1b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f5088d77740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 0x5639dad1bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f5088d77740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5639dac92630 con 0x5639da94d1b0 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f5088d77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5639dac81bc0 con 0x5639da976070 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f5088d77740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5639dac85b70 con 0x5639da955390 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f507bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad1b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f5080d9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da976070 0x5639dad1b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f507bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad1b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45236/0 (socket says 172.21.15.16:45236) 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.415+0000 7f507bfff640 1 -- 172.21.15.16:0/530158528 learned_addr learned my addr 172.21.15.16:0/530158528 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.416+0000 7f507b7fe640 1 --2- 172.21.15.16:0/530158528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 0x5639dad1bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.416+0000 7f50797fa640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50640b4cc0 0x7f50640b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.416+0000 7f507bfff640 1 -- 172.21.15.16:0/530158528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 msgr2=0x5639dad1bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.416+0000 7f507bfff640 1 --2- 172.21.15.16:0/530158528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 0x5639dad1bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.416+0000 7f507bfff640 1 -- 172.21.15.16:0/530158528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da976070 msgr2=0x5639dad1b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.416+0000 7f507bfff640 1 --2- 172.21.15.16:0/530158528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da976070 0x5639dad1b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.416+0000 7f507bfff640 1 -- 172.21.15.16:0/530158528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639da9e1650 con 0x5639da94d1b0 2024-08-28T19:21:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.416+0000 7f5080d9f640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50640b4cc0 0x7f50640b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.416+0000 7f507bfff640 1 --2- 172.21.15.16:0/530158528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad1b9a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f506c1032d0 tx=0x7f506c090d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1dca0832ff82f9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.416+0000 7f50797fa640 1 -- 172.21.15.16:0/2419717139 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f506c105e70 con 0x7f507c102610 2024-08-28T19:21:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.416+0000 7f50627fc640 1 -- 172.21.15.16:0/530158528 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506c0cbad0 con 0x5639da94d1b0 2024-08-28T19:21:05.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.417+0000 7f50627fc640 1 -- 172.21.15.16:0/530158528 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f506c0cbe70 con 0x5639da94d1b0 2024-08-28T19:21:05.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.417+0000 7f5080d9f640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50640b4cc0 0x7f50640b71d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5074013520 tx=0x7f5074004ab0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.417+0000 7f50627fc640 1 -- 172.21.15.16:0/530158528 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506c03b460 con 0x5639da94d1b0 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.417+0000 7f5088d77740 1 -- 172.21.15.16:0/530158528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 msgr2=0x5639dad1b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.417+0000 7f5088d77740 1 --2- 172.21.15.16:0/530158528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad1b9a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f506c1032d0 tx=0x7f506c090d10 comp rx=0 tx=0).stop 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.418+0000 7f5088d77740 1 -- 172.21.15.16:0/530158528 shutdown_connections 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.418+0000 7f5088d77740 1 --2- 172.21.15.16:0/530158528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 0x5639dad1bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.418+0000 7f5088d77740 1 --2- 172.21.15.16:0/530158528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad1b9a0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.418+0000 7f5088d77740 1 --2- 172.21.15.16:0/530158528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da976070 0x5639dad1b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.418+0000 7f5088d77740 1 -- 172.21.15.16:0/530158528 >> 172.21.15.16:0/530158528 conn(0x5639da96a460 msgr2=0x5639dad19350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.418+0000 7f5088d77740 1 -- 172.21.15.16:0/530158528 shutdown_connections 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.418+0000 7f5088d77740 1 -- 172.21.15.16:0/530158528 wait complete. 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.418+0000 7f5088d77740 1 -- start start 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.418+0000 7f5088d77740 1 Processor -- start 2024-08-28T19:21:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f5088d77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da94d1b0 0x5639dad26b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f5088d77740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 0x5639dad270b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f5088d77740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da976070 0x5639dad32b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f5088d77740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5639daae4700 con 0x5639da976070 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f5088d77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5639dac08700 con 0x5639da94d1b0 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f5088d77740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5639dac92630 con 0x5639da955390 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f5088d77740 10 client.0 _ll_register_callbacks cb 0x5639dac37dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f5080d9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da976070 0x5639dad32b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f507b7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 0x5639dad270b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f507b7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 0x5639dad270b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:48728/0 (socket says 172.21.15.16:48728) 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f507bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da94d1b0 0x5639dad26b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f507b7fe640 1 -- 172.21.15.16:0/3909053597 learned_addr learned my addr 172.21.15.16:0/3909053597 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.419+0000 7f5080d9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da976070 0x5639dad32b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45238/0 (socket says 172.21.15.16:45238) 2024-08-28T19:21:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f507b7fe640 1 -- 172.21.15.16:0/3909053597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da94d1b0 msgr2=0x5639dad26b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f507b7fe640 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da94d1b0 0x5639dad26b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f507b7fe640 1 -- 172.21.15.16:0/3909053597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da976070 msgr2=0x5639dad32b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f507b7fe640 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da976070 0x5639dad32b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f507b7fe640 1 -- 172.21.15.16:0/3909053597 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639dac418b0 con 0x5639da955390 2024-08-28T19:21:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f507bfff640 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da94d1b0 0x5639dad26b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f507b7fe640 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 0x5639dad270b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5070012700 tx=0x7f5070012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=991a79d64e026418 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f505b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:21:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f505b7fe640 1 -- 172.21.15.16:0/3909053597 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070004050 con 0x5639da955390 2024-08-28T19:21:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f5088d77740 1 -- 172.21.15.16:0/3909053597 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f506c0cbe70 con 0x5639da955390 2024-08-28T19:21:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f505b7fe640 1 -- 172.21.15.16:0/3909053597 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f50700041f0 con 0x5639da955390 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.420+0000 7f505b7fe640 1 -- 172.21.15.16:0/3909053597 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50700043f0 con 0x5639da955390 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.421+0000 7f5088d77740 10 client.25246 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.421+0000 7f5088d77740 1 -- 172.21.15.16:0/3909053597 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5639dad35670 con 0x5639da955390 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.421+0000 7f505b7fe640 1 -- 172.21.15.16:0/3909053597 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f506c0cbe70 con 0x5639da955390 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.421+0000 7f505b7fe640 1 -- 172.21.15.16:0/3909053597 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5070003b20 con 0x5639da955390 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.421+0000 7f5088d77740 10 client.25246 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.421+0000 7f5088d77740 10 client.25246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.421+0000 7f5088d77740 20 client.25246 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.421+0000 7f5088d77740 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5639da9349f0 0x5639dad38ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.421+0000 7f5088d77740 4 client.25246 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.421+0000 7f5088d77740 1 -- 172.21.15.16:0/3909053597 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639dac94580 con 0x5639da9349f0 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.421+0000 7f507bfff640 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5639da9349f0 0x5639dad38ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.422+0000 7f505b7fe640 10 client.25246 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:05.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.422+0000 7f507bfff640 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5639da9349f0 0x5639dad38ad0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=835374cc2ae67938 server_cookie=9dbf0a095b5e3099 in_seq=0 out_seq=0 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f505b7fe640 1 -- 172.21.15.16:0/3909053597 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5639dac94580 con 0x5639da9349f0 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f505b7fe640 10 client.25246 handle_command_reply: tid=0 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 1 client.25246 shutdown 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 1 -- 172.21.15.16:0/3909053597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 msgr2=0x5639dad270b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 0x5639dad270b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5070012700 tx=0x7f5070012bc0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 1 -- 172.21.15.16:0/3909053597 shutdown_connections 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da976070 0x5639dad32b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5639da9349f0 0x5639dad38ad0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da955390 0x5639dad270b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 1 --2- 172.21.15.16:0/3909053597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da94d1b0 0x5639dad26b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f505b7fe640 0 client.25246 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 1 -- 172.21.15.16:0/3909053597 >> 172.21.15.16:0/3909053597 conn(0x5639da96a460 msgr2=0x5639dad1c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:05.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 1 -- 172.21.15.16:0/3909053597 shutdown_connections 2024-08-28T19:21:05.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 1 -- 172.21.15.16:0/3909053597 wait complete. 2024-08-28T19:21:05.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.423+0000 7f5088d77740 20 client.25246 trim_cache size 0 max 16384 2024-08-28T19:21:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.442+0000 7f5088d77740 1 Processor -- start 2024-08-28T19:21:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.442+0000 7f5088d77740 1 -- start start 2024-08-28T19:21:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f5088d77740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da94d1b0 0x5639dad1cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f5088d77740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da955390 0x5639dad32a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f5088d77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da976070 0x5639dad23d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f5088d77740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f507c1bf850 con 0x5639da955390 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f5088d77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f507c1c1390 con 0x5639da976070 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f5088d77740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f507c1bff20 con 0x5639da94d1b0 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f507b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da955390 0x5639dad32a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f507b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da955390 0x5639dad32a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45244/0 (socket says 172.21.15.16:45244) 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f507b7fe640 1 -- 172.21.15.16:0/4100305317 learned_addr learned my addr 172.21.15.16:0/4100305317 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f507bfff640 1 --2- 172.21.15.16:0/4100305317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da94d1b0 0x5639dad1cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f5080d9f640 1 --2- 172.21.15.16:0/4100305317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da976070 0x5639dad23d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f507b7fe640 1 -- 172.21.15.16:0/4100305317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da94d1b0 msgr2=0x5639dad1cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f507b7fe640 1 --2- 172.21.15.16:0/4100305317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da94d1b0 0x5639dad1cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f507b7fe640 1 -- 172.21.15.16:0/4100305317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da976070 msgr2=0x5639dad23d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f507b7fe640 1 --2- 172.21.15.16:0/4100305317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da976070 0x5639dad23d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f507b7fe640 1 -- 172.21.15.16:0/4100305317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639dad1e2c0 con 0x5639da955390 2024-08-28T19:21:05.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.443+0000 7f5080d9f640 1 --2- 172.21.15.16:0/4100305317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da976070 0x5639dad23d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:05.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.444+0000 7f507b7fe640 1 --2- 172.21.15.16:0/4100305317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da955390 0x5639dad32a70 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5070009560 tx=0x7f5070010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7f4f778e79eb88 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:05.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.444+0000 7f5061ffb640 1 -- 172.21.15.16:0/4100305317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070004050 con 0x5639da955390 2024-08-28T19:21:05.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.444+0000 7f5061ffb640 1 -- 172.21.15.16:0/4100305317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5070010e20 con 0x5639da955390 2024-08-28T19:21:05.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.444+0000 7f5061ffb640 1 -- 172.21.15.16:0/4100305317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070005070 con 0x5639da955390 2024-08-28T19:21:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.445+0000 7f5088d77740 1 -- 172.21.15.16:0/4100305317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da955390 msgr2=0x5639dad32a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.445+0000 7f5088d77740 1 --2- 172.21.15.16:0/4100305317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da955390 0x5639dad32a70 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5070009560 tx=0x7f5070010830 comp rx=0 tx=0).stop 2024-08-28T19:21:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.445+0000 7f5088d77740 1 -- 172.21.15.16:0/4100305317 shutdown_connections 2024-08-28T19:21:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.445+0000 7f5088d77740 1 --2- 172.21.15.16:0/4100305317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da976070 0x5639dad23d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.445+0000 7f5088d77740 1 --2- 172.21.15.16:0/4100305317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da955390 0x5639dad32a70 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.445+0000 7f5088d77740 1 --2- 172.21.15.16:0/4100305317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da94d1b0 0x5639dad1cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.445+0000 7f5088d77740 1 -- 172.21.15.16:0/4100305317 >> 172.21.15.16:0/4100305317 conn(0x5639da96a460 msgr2=0x5639dad1c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.445+0000 7f5088d77740 1 -- 172.21.15.16:0/4100305317 shutdown_connections 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.445+0000 7f5088d77740 1 -- 172.21.15.16:0/4100305317 wait complete. 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.445+0000 7f5088d77740 1 -- start start 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f5088d77740 1 Processor -- start 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f5088d77740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad2ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f5088d77740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da976070 0x5639dad2f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f5088d77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da909da0 0x5639dad29a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f5088d77740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f507c1c0cc0 con 0x5639da94d1b0 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f507bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad2ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f5088d77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f507c1c05f0 con 0x5639da909da0 2024-08-28T19:21:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f5088d77740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f507c1bd830 con 0x5639da976070 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f507bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad2ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45248/0 (socket says 172.21.15.16:45248) 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f5088d77740 10 client.0 _ll_register_callbacks cb 0x7f50700097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f507bfff640 1 -- 172.21.15.16:0/4043662082 learned_addr learned my addr 172.21.15.16:0/4043662082 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f5080d9f640 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da909da0 0x5639dad29a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.446+0000 7f507b7fe640 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da976070 0x5639dad2f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f507bfff640 1 -- 172.21.15.16:0/4043662082 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da976070 msgr2=0x5639dad2f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f507bfff640 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da976070 0x5639dad2f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f507bfff640 1 -- 172.21.15.16:0/4043662082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da909da0 msgr2=0x5639dad29a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f507bfff640 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da909da0 0x5639dad29a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f507bfff640 1 -- 172.21.15.16:0/4043662082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639dad2a180 con 0x5639da94d1b0 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f507b7fe640 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da976070 0x5639dad2f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f507bfff640 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad2ece0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f506c1032d0 tx=0x7f506c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9ab0cf1933107d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f505b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f505b7fe640 1 -- 172.21.15.16:0/4043662082 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506c049040 con 0x5639da94d1b0 2024-08-28T19:21:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f505b7fe640 1 -- 172.21.15.16:0/4043662082 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f506c0c85f0 con 0x5639da94d1b0 2024-08-28T19:21:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f5088d77740 1 -- 172.21.15.16:0/4043662082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5070010e20 con 0x5639da94d1b0 2024-08-28T19:21:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f505b7fe640 1 -- 172.21.15.16:0/4043662082 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f506c0b8db0 con 0x5639da94d1b0 2024-08-28T19:21:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f5088d77740 10 client.15657 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.447+0000 7f5088d77740 1 -- 172.21.15.16:0/4043662082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5639dada3e90 con 0x5639da94d1b0 2024-08-28T19:21:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.448+0000 7f505b7fe640 1 -- 172.21.15.16:0/4043662082 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5070010e20 con 0x5639da94d1b0 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.448+0000 7f505b7fe640 1 -- 172.21.15.16:0/4043662082 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f506c0acf90 con 0x5639da94d1b0 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.448+0000 7f5088d77740 10 client.15657 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.448+0000 7f5088d77740 10 client.15657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.448+0000 7f5088d77740 20 client.15657 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.448+0000 7f5088d77740 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5639dada4450 0x5639dada6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.448+0000 7f5088d77740 4 client.15657 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.448+0000 7f5088d77740 1 -- 172.21.15.16:0/4043662082 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5639dac90020 con 0x5639dada4450 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.448+0000 7f507b7fe640 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5639dada4450 0x5639dada6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f505b7fe640 10 client.15657 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f507b7fe640 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5639dada4450 0x5639dada6870 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd84f32170dbcf7c server_cookie=582ce4b1760250da in_seq=0 out_seq=0 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f505b7fe640 1 -- 172.21.15.16:0/4043662082 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+570 (crc 0 0 0) 0x5639dac90020 con 0x5639dada4450 2024-08-28T19:21:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f505b7fe640 10 client.15657 handle_command_reply: tid=0 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 1 client.15657 shutdown 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 1 -- 172.21.15.16:0/4043662082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 msgr2=0x5639dad2ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad2ece0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f506c1032d0 tx=0x7f506c009260 comp rx=0 tx=0).stop 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 1 -- 172.21.15.16:0/4043662082 shutdown_connections 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639da909da0 0x5639dad29a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5639dada4450 0x5639dada6870 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f505b7fe640 0 client.15657 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639da976070 0x5639dad2f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 1 --2- 172.21.15.16:0/4043662082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639da94d1b0 0x5639dad2ece0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 1 -- 172.21.15.16:0/4043662082 >> 172.21.15.16:0/4043662082 conn(0x5639da96a460 msgr2=0x5639dad28020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 1 -- 172.21.15.16:0/4043662082 shutdown_connections 2024-08-28T19:21:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 1 -- 172.21.15.16:0/4043662082 wait complete. 2024-08-28T19:21:05.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.449+0000 7f5088d77740 20 client.15657 trim_cache size 0 max 16384 2024-08-28T19:21:05.456 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:21:05.456 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:21:05.456 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:21:05.456 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:21:05.456 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:21:05.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "b842a7f8-9a3b-435e-88a7-49e794ad2bb8", 2024-08-28T19:21:05.456 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:21:05.456 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:05.457 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:21:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 -- 172.21.15.16:0/2419717139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50640b4cc0 msgr2=0x7f50640b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50640b4cc0 0x7f50640b71d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5074013520 tx=0x7f5074004ab0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 -- 172.21.15.16:0/2419717139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f507c102610 msgr2=0x7f507c117e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f507c102610 0x7f507c117e10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f506c005ab0 tx=0x7f506c003c20 comp rx=0 tx=0).stop 2024-08-28T19:21:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 -- 172.21.15.16:0/2419717139 shutdown_connections 2024-08-28T19:21:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50640b4cc0 0x7f50640b71d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c1088a0 0x7f507c1126a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f507c107f50 0x7f507c1120d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 --2- 172.21.15.16:0/2419717139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f507c102610 0x7f507c117e10 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:05.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 -- 172.21.15.16:0/2419717139 >> 172.21.15.16:0/2419717139 conn(0x7f507c0fe060 msgr2=0x7f507c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:05.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 -- 172.21.15.16:0/2419717139 shutdown_connections 2024-08-28T19:21:05.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:05.450+0000 7f50815a0640 1 -- 172.21.15.16:0/2419717139 wait complete. 2024-08-28T19:21:05.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:05 smithi028 ceph-mon[21769]: mgrmap e23: x(active, since 8m), standbys: z, y 2024-08-28T19:21:05.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:05 smithi028 ceph-mon[21769]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:05.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:05 smithi028 ceph-mon[21769]: from='client.15639 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:05 smithi115 ceph-mon[22184]: mgrmap e23: x(active, since 8m), standbys: z, y 2024-08-28T19:21:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:05 smithi115 ceph-mon[22184]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:05 smithi115 ceph-mon[22184]: from='client.15639 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:05.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:05 smithi016 ceph-mon[18556]: mgrmap e23: x(active, since 8m), standbys: z, y 2024-08-28T19:21:05.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:05 smithi016 ceph-mon[18556]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:05.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:05 smithi016 ceph-mon[18556]: from='client.15639 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:21:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.145+0000 7f1942ca4640 1 -- 172.21.15.16:0/3123773492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c108170 msgr2=0x7f193c108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.145+0000 7f1942ca4640 1 --2- 172.21.15.16:0/3123773492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c108170 0x7f193c108580 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1928009900 tx=0x7f192802a410 comp rx=0 tx=0).stop 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.145+0000 7f1942ca4640 1 -- 172.21.15.16:0/3123773492 shutdown_connections 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.145+0000 7f1942ca4640 1 --2- 172.21.15.16:0/3123773492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f193c108ac0 0x7f193c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.145+0000 7f1942ca4640 1 --2- 172.21.15.16:0/3123773492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c108170 0x7f193c108580 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.145+0000 7f1942ca4640 1 --2- 172.21.15.16:0/3123773492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f193c102880 0x7f193c102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.145+0000 7f1942ca4640 1 -- 172.21.15.16:0/3123773492 >> 172.21.15.16:0/3123773492 conn(0x7f193c079500 msgr2=0x7f193c079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.145+0000 7f1942ca4640 1 -- 172.21.15.16:0/3123773492 shutdown_connections 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.145+0000 7f1942ca4640 1 -- 172.21.15.16:0/3123773492 wait complete. 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.146+0000 7f1942ca4640 1 Processor -- start 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.146+0000 7f1942ca4640 1 -- start start 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.146+0000 7f1942ca4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c102880 0x7f193c19c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.146+0000 7f1942ca4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f193c108170 0x7f193c19c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.146+0000 7f1942ca4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f193c108ac0 0x7f193c1a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.146+0000 7f1942ca4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f193c112710 con 0x7f193c102880 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.146+0000 7f1942ca4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f193c112590 con 0x7f193c108170 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.146+0000 7f1942ca4640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f193c112890 con 0x7f193c108ac0 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1940a40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c102880 0x7f193c19c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1940a40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c102880 0x7f193c19c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45266/0 (socket says 172.21.15.16:45266) 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1940a40640 1 -- 172.21.15.16:0/758774942 learned_addr learned my addr 172.21.15.16:0/758774942 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f193bfff640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f193c108170 0x7f193c19c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1941241640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f193c108ac0 0x7f193c1a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1940a40640 1 -- 172.21.15.16:0/758774942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f193c108ac0 msgr2=0x7f193c1a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1940a40640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f193c108ac0 0x7f193c1a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1940a40640 1 -- 172.21.15.16:0/758774942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f193c108170 msgr2=0x7f193c19c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1940a40640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f193c108170 0x7f193c19c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1940a40640 1 -- 172.21.15.16:0/758774942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193c1a23c0 con 0x7f193c102880 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1940a40640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c102880 0x7f193c19c450 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f193000d770 tx=0x7f193000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1939ffb640 1 -- 172.21.15.16:0/758774942 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930017070 con 0x7f193c102880 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1939ffb640 1 -- 172.21.15.16:0/758774942 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19300054d0 con 0x7f193c102880 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1942ca4640 1 -- 172.21.15.16:0/758774942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193c1a26b0 con 0x7f193c102880 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1939ffb640 1 -- 172.21.15.16:0/758774942 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930004c30 con 0x7f193c102880 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.147+0000 7f1942ca4640 1 -- 172.21.15.16:0/758774942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f193c1a8e10 con 0x7f193c102880 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.148+0000 7f1942ca4640 1 -- 172.21.15.16:0/758774942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f193c103ab0 con 0x7f193c102880 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.150+0000 7f1939ffb640 1 -- 172.21.15.16:0/758774942 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f1930004e10 con 0x7f193c102880 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.150+0000 7f1939ffb640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f192c0b4c10 0x7f192c0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.151+0000 7f193bfff640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f192c0b4c10 0x7f192c0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.151+0000 7f1939ffb640 1 -- 172.21.15.16:0/758774942 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f19300e52d0 con 0x7f193c102880 2024-08-28T19:21:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.151+0000 7f193bfff640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f192c0b4c10 0x7f192c0b7120 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1928000c00 tx=0x7f1928031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:06.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.155+0000 7f1939ffb640 1 -- 172.21.15.16:0/758774942 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f19300abe10 con 0x7f193c102880 2024-08-28T19:21:06.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.340+0000 7f1942ca4640 1 -- 172.21.15.16:0/758774942 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f193c06c130 con 0x7f192c0b4c10 2024-08-28T19:21:06.342 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:06 smithi016 ceph-mon[18556]: pgmap v341: 129 pgs: 129 active+clean; 1.0 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 62 KiB/s wr, 18 op/s 2024-08-28T19:21:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.346+0000 7f1939ffb640 1 -- 172.21.15.16:0/758774942 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340581 (secure 0 0 0) 0x7f193c06c130 con 0x7f192c0b4c10 2024-08-28T19:21:06.351 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:21:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-08-28T19:21:06.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 -- 172.21.15.16:0/758774942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f192c0b4c10 msgr2=0x7f192c0b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:06.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f192c0b4c10 0x7f192c0b7120 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1928000c00 tx=0x7f1928031400 comp rx=0 tx=0).stop 2024-08-28T19:21:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 -- 172.21.15.16:0/758774942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c102880 msgr2=0x7f193c19c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c102880 0x7f193c19c450 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f193000d770 tx=0x7f193000dc30 comp rx=0 tx=0).stop 2024-08-28T19:21:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 -- 172.21.15.16:0/758774942 shutdown_connections 2024-08-28T19:21:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f193c108ac0 0x7f193c1a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f192c0b4c10 0x7f192c0b7120 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f193c108170 0x7f193c19c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 --2- 172.21.15.16:0/758774942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c102880 0x7f193c19c450 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 -- 172.21.15.16:0/758774942 >> 172.21.15.16:0/758774942 conn(0x7f193c079500 msgr2=0x7f193c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 -- 172.21.15.16:0/758774942 shutdown_connections 2024-08-28T19:21:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:06.351+0000 7f1942ca4640 1 -- 172.21.15.16:0/758774942 wait complete. 2024-08-28T19:21:06.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:06 smithi028 ceph-mon[21769]: pgmap v341: 129 pgs: 129 active+clean; 1.0 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 62 KiB/s wr, 18 op/s 2024-08-28T19:21:06.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:06 smithi115 ceph-mon[22184]: pgmap v341: 129 pgs: 129 active+clean; 1.0 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 62 KiB/s wr, 18 op/s 2024-08-28T19:21:06.639 INFO:tasks.cephfs.filesystem:scrub status for tag:b842a7f8-9a3b-435e-88a7-49e794ad2bb8 - {'path': '/', 'tag': 'b842a7f8-9a3b-435e-88a7-49e794ad2bb8', 'options': 'recursive,force'} 2024-08-28T19:21:07.383 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-08-28T19:21:05.497707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1061269,"num_objects":103,"num_object_clones":0,"num_object_copies":311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":1265,"num_read_kb":757,"num_write":450,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":871,"ondisk_log_size":871,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":418608,"kb_used_data":15220,"kb_used_omap":18,"kb_used_meta":403117,"kb_avail":1124523216,"statfs":{"total":1151940427776,"available":1151511773184,"internally_reserved":0,"allocated":15585280,"data_stored":9542901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":412792122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":399457,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":161,"num_read_kb":96,"num_write":67,"num_write_kb":747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003228"},"pg_stats":[{"pgid":"2.1f","version":"84'14","reported_seq":177,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:59.813914+0000","last_change":"2024-08-28T19:19:08.633326+0000","last_active":"2024-08-28T19:20:59.813914+0000","last_peered":"2024-08-28T19:20:59.813914+0000","last_clean":"2024-08-28T19:20:59.813914+0000","last_became_active":"2024-08-28T19:19:08.623876+0000","last_became_peered":"2024-08-28T19:19:08.623876+0000","last_unstale":"2024-08-28T19:20:59.813914+0000","last_undegraded":"2024-08-28T19:20:59.813914+0000","last_fullsized":"2024-08-28T19:20:59.813914+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:19:50.632337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866980+0000","last_change":"2024-08-28T19:18:32.492604+0000","last_active":"2024-08-28T19:20:37.866980+0000","last_peered":"2024-08-28T19:20:37.866980+0000","last_clean":"2024-08-28T19:20:37.866980+0000","last_became_active":"2024-08-28T19:18:32.489427+0000","last_became_peered":"2024-08-28T19:18:32.489427+0000","last_unstale":"2024-08-28T19:20:37.866980+0000","last_undegraded":"2024-08-28T19:20:37.866980+0000","last_fullsized":"2024-08-28T19:20:37.866980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:36:14.841315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010445+0000","last_change":"2024-08-28T19:18:39.528815+0000","last_active":"2024-08-28T19:20:09.010445+0000","last_peered":"2024-08-28T19:20:09.010445+0000","last_clean":"2024-08-28T19:20:09.010445+0000","last_became_active":"2024-08-28T19:18:39.527781+0000","last_became_peered":"2024-08-28T19:18:39.527781+0000","last_unstale":"2024-08-28T19:20:09.010445+0000","last_undegraded":"2024-08-28T19:20:09.010445+0000","last_fullsized":"2024-08-28T19:20:09.010445+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:39.831426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525559+0000","last_change":"2024-08-28T19:18:31.082641+0000","last_active":"2024-08-28T19:20:09.525559+0000","last_peered":"2024-08-28T19:20:09.525559+0000","last_clean":"2024-08-28T19:20:09.525559+0000","last_became_active":"2024-08-28T19:18:31.082260+0000","last_became_peered":"2024-08-28T19:18:31.082260+0000","last_unstale":"2024-08-28T19:20:09.525559+0000","last_undegraded":"2024-08-28T19:20:09.525559+0000","last_fullsized":"2024-08-28T19:20:09.525559+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:09:55.836463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886294+0000","last_change":"2024-08-28T19:18:32.493498+0000","last_active":"2024-08-28T19:20:37.886294+0000","last_peered":"2024-08-28T19:20:37.886294+0000","last_clean":"2024-08-28T19:20:37.886294+0000","last_became_active":"2024-08-28T19:18:32.490969+0000","last_became_peered":"2024-08-28T19:18:32.490969+0000","last_unstale":"2024-08-28T19:20:37.886294+0000","last_undegraded":"2024-08-28T19:20:37.886294+0000","last_fullsized":"2024-08-28T19:20:37.886294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:34:29.537622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614607+0000","last_change":"2024-08-28T19:18:39.520922+0000","last_active":"2024-08-28T19:20:09.614607+0000","last_peered":"2024-08-28T19:20:09.614607+0000","last_clean":"2024-08-28T19:20:09.614607+0000","last_became_active":"2024-08-28T19:18:39.520362+0000","last_became_peered":"2024-08-28T19:18:39.520362+0000","last_unstale":"2024-08-28T19:20:09.614607+0000","last_undegraded":"2024-08-28T19:20:09.614607+0000","last_fullsized":"2024-08-28T19:20:09.614607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:40.658820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":53,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110426+0000","last_change":"2024-08-28T19:18:30.471870+0000","last_active":"2024-08-28T19:20:09.110426+0000","last_peered":"2024-08-28T19:20:09.110426+0000","last_clean":"2024-08-28T19:20:09.110426+0000","last_became_active":"2024-08-28T19:18:30.471438+0000","last_became_peered":"2024-08-28T19:18:30.471438+0000","last_unstale":"2024-08-28T19:20:09.110426+0000","last_undegraded":"2024-08-28T19:20:09.110426+0000","last_fullsized":"2024-08-28T19:20:09.110426+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:44:34.137395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"84'9","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.881074+0000","last_change":"2024-08-28T19:18:32.486485+0000","last_active":"2024-08-28T19:20:37.881074+0000","last_peered":"2024-08-28T19:20:37.881074+0000","last_clean":"2024-08-28T19:20:37.881074+0000","last_became_active":"2024-08-28T19:18:32.485888+0000","last_became_peered":"2024-08-28T19:18:32.485888+0000","last_unstale":"2024-08-28T19:20:37.881074+0000","last_undegraded":"2024-08-28T19:20:37.881074+0000","last_fullsized":"2024-08-28T19:20:37.881074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:45:48.757318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615105+0000","last_change":"2024-08-28T19:19:08.633317+0000","last_active":"2024-08-28T19:20:09.615105+0000","last_peered":"2024-08-28T19:20:09.615105+0000","last_clean":"2024-08-28T19:20:09.615105+0000","last_became_active":"2024-08-28T19:19:08.632689+0000","last_became_peered":"2024-08-28T19:19:08.632689+0000","last_unstale":"2024-08-28T19:20:09.615105+0000","last_undegraded":"2024-08-28T19:20:09.615105+0000","last_fullsized":"2024-08-28T19:20:09.615105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:35:08.740645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110357+0000","last_change":"2024-08-28T19:18:30.473982+0000","last_active":"2024-08-28T19:20:09.110357+0000","last_peered":"2024-08-28T19:20:09.110357+0000","last_clean":"2024-08-28T19:20:09.110357+0000","last_became_active":"2024-08-28T19:18:30.473552+0000","last_became_peered":"2024-08-28T19:18:30.473552+0000","last_unstale":"2024-08-28T19:20:09.110357+0000","last_undegraded":"2024-08-28T19:20:09.110357+0000","last_fullsized":"2024-08-28T19:20:09.110357+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:22:29.435517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884392+0000","last_change":"2024-08-28T19:18:32.487881+0000","last_active":"2024-08-28T19:20:37.884392+0000","last_peered":"2024-08-28T19:20:37.884392+0000","last_clean":"2024-08-28T19:20:37.884392+0000","last_became_active":"2024-08-28T19:18:32.487039+0000","last_became_peered":"2024-08-28T19:18:32.487039+0000","last_unstale":"2024-08-28T19:20:37.884392+0000","last_undegraded":"2024-08-28T19:20:37.884392+0000","last_fullsized":"2024-08-28T19:20:37.884392+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:23:35.757811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614594+0000","last_change":"2024-08-28T19:18:39.518654+0000","last_active":"2024-08-28T19:20:09.614594+0000","last_peered":"2024-08-28T19:20:09.614594+0000","last_clean":"2024-08-28T19:20:09.614594+0000","last_became_active":"2024-08-28T19:18:39.518087+0000","last_became_peered":"2024-08-28T19:18:39.518087+0000","last_unstale":"2024-08-28T19:20:09.614594+0000","last_undegraded":"2024-08-28T19:20:09.614594+0000","last_fullsized":"2024-08-28T19:20:09.614594+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:08.284865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.887381+0000","last_change":"2024-08-28T19:18:32.491928+0000","last_active":"2024-08-28T19:20:37.887381+0000","last_peered":"2024-08-28T19:20:37.887381+0000","last_clean":"2024-08-28T19:20:37.887381+0000","last_became_active":"2024-08-28T19:18:32.491538+0000","last_became_peered":"2024-08-28T19:18:32.491538+0000","last_unstale":"2024-08-28T19:20:37.887381+0000","last_undegraded":"2024-08-28T19:20:37.887381+0000","last_fullsized":"2024-08-28T19:20:37.887381+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:39:06.077830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":24,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663446+0000","last_change":"2024-08-28T19:19:08.634752+0000","last_active":"2024-08-28T19:20:09.663446+0000","last_peered":"2024-08-28T19:20:09.663446+0000","last_clean":"2024-08-28T19:20:09.663446+0000","last_became_active":"2024-08-28T19:19:08.620784+0000","last_became_peered":"2024-08-28T19:19:08.620784+0000","last_unstale":"2024-08-28T19:20:09.663446+0000","last_undegraded":"2024-08-28T19:20:09.663446+0000","last_fullsized":"2024-08-28T19:20:09.663446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:51.640445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614895+0000","last_change":"2024-08-28T19:18:39.681413+0000","last_active":"2024-08-28T19:20:09.614895+0000","last_peered":"2024-08-28T19:20:09.614895+0000","last_clean":"2024-08-28T19:20:09.614895+0000","last_became_active":"2024-08-28T19:18:39.680793+0000","last_became_peered":"2024-08-28T19:18:39.680793+0000","last_unstale":"2024-08-28T19:20:09.614895+0000","last_undegraded":"2024-08-28T19:20:09.614895+0000","last_fullsized":"2024-08-28T19:20:09.614895+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:45:43.381019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870888+0000","last_change":"2024-08-28T19:18:32.486739+0000","last_active":"2024-08-28T19:20:37.870888+0000","last_peered":"2024-08-28T19:20:37.870888+0000","last_clean":"2024-08-28T19:20:37.870888+0000","last_became_active":"2024-08-28T19:18:32.486347+0000","last_became_peered":"2024-08-28T19:18:32.486347+0000","last_unstale":"2024-08-28T19:20:37.870888+0000","last_undegraded":"2024-08-28T19:20:37.870888+0000","last_fullsized":"2024-08-28T19:20:37.870888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:27:36.448569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010653+0000","last_change":"2024-08-28T19:18:30.467690+0000","last_active":"2024-08-28T19:20:09.010653+0000","last_peered":"2024-08-28T19:20:09.010653+0000","last_clean":"2024-08-28T19:20:09.010653+0000","last_became_active":"2024-08-28T19:18:30.467249+0000","last_became_peered":"2024-08-28T19:18:30.467249+0000","last_unstale":"2024-08-28T19:20:09.010653+0000","last_undegraded":"2024-08-28T19:20:09.010653+0000","last_fullsized":"2024-08-28T19:20:09.010653+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:37:28.876666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615167+0000","last_change":"2024-08-28T19:18:39.523889+0000","last_active":"2024-08-28T19:20:09.615167+0000","last_peered":"2024-08-28T19:20:09.615167+0000","last_clean":"2024-08-28T19:20:09.615167+0000","last_became_active":"2024-08-28T19:18:39.523384+0000","last_became_peered":"2024-08-28T19:18:39.523384+0000","last_unstale":"2024-08-28T19:20:09.615167+0000","last_undegraded":"2024-08-28T19:20:09.615167+0000","last_fullsized":"2024-08-28T19:20:09.615167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:57:32.277680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873180+0000","last_change":"2024-08-28T19:18:32.482257+0000","last_active":"2024-08-28T19:20:37.873180+0000","last_peered":"2024-08-28T19:20:37.873180+0000","last_clean":"2024-08-28T19:20:37.873180+0000","last_became_active":"2024-08-28T19:18:32.481884+0000","last_became_peered":"2024-08-28T19:18:32.481884+0000","last_unstale":"2024-08-28T19:20:37.873180+0000","last_undegraded":"2024-08-28T19:20:37.873180+0000","last_fullsized":"2024-08-28T19:20:37.873180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:56:28.754369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":47,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525435+0000","last_change":"2024-08-28T19:18:30.467307+0000","last_active":"2024-08-28T19:20:09.525435+0000","last_peered":"2024-08-28T19:20:09.525435+0000","last_clean":"2024-08-28T19:20:09.525435+0000","last_became_active":"2024-08-28T19:18:30.466942+0000","last_became_peered":"2024-08-28T19:18:30.466942+0000","last_unstale":"2024-08-28T19:20:09.525435+0000","last_undegraded":"2024-08-28T19:20:09.525435+0000","last_fullsized":"2024-08-28T19:20:09.525435+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:05:23.682110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007238+0000","last_change":"2024-08-28T19:18:39.531421+0000","last_active":"2024-08-28T19:20:09.007238+0000","last_peered":"2024-08-28T19:20:09.007238+0000","last_clean":"2024-08-28T19:20:09.007238+0000","last_became_active":"2024-08-28T19:18:39.531008+0000","last_became_peered":"2024-08-28T19:18:39.531008+0000","last_unstale":"2024-08-28T19:20:09.007238+0000","last_undegraded":"2024-08-28T19:20:09.007238+0000","last_fullsized":"2024-08-28T19:20:09.007238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:46:49.580601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868264+0000","last_change":"2024-08-28T19:18:32.485204+0000","last_active":"2024-08-28T19:20:37.868264+0000","last_peered":"2024-08-28T19:20:37.868264+0000","last_clean":"2024-08-28T19:20:37.868264+0000","last_became_active":"2024-08-28T19:18:32.484830+0000","last_became_peered":"2024-08-28T19:18:32.484830+0000","last_unstale":"2024-08-28T19:20:37.868264+0000","last_undegraded":"2024-08-28T19:20:37.868264+0000","last_fullsized":"2024-08-28T19:20:37.868264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:55:41.358718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007905+0000","last_change":"2024-08-28T19:18:31.082361+0000","last_active":"2024-08-28T19:20:09.007905+0000","last_peered":"2024-08-28T19:20:09.007905+0000","last_clean":"2024-08-28T19:20:09.007905+0000","last_became_active":"2024-08-28T19:18:31.081930+0000","last_became_peered":"2024-08-28T19:18:31.081930+0000","last_unstale":"2024-08-28T19:20:09.007905+0000","last_undegraded":"2024-08-28T19:20:09.007905+0000","last_fullsized":"2024-08-28T19:20:09.007905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:41:05.543088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615411+0000","last_change":"2024-08-28T19:18:39.682341+0000","last_active":"2024-08-28T19:20:09.615411+0000","last_peered":"2024-08-28T19:20:09.615411+0000","last_clean":"2024-08-28T19:20:09.615411+0000","last_became_active":"2024-08-28T19:18:39.681751+0000","last_became_peered":"2024-08-28T19:18:39.681751+0000","last_unstale":"2024-08-28T19:20:09.615411+0000","last_undegraded":"2024-08-28T19:20:09.615411+0000","last_fullsized":"2024-08-28T19:20:09.615411+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:51:24.371270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880906+0000","last_change":"2024-08-28T19:18:32.491069+0000","last_active":"2024-08-28T19:20:37.880906+0000","last_peered":"2024-08-28T19:20:37.880906+0000","last_clean":"2024-08-28T19:20:37.880906+0000","last_became_active":"2024-08-28T19:18:32.490688+0000","last_became_peered":"2024-08-28T19:18:32.490688+0000","last_unstale":"2024-08-28T19:20:37.880906+0000","last_undegraded":"2024-08-28T19:20:37.880906+0000","last_fullsized":"2024-08-28T19:20:37.880906+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:24:31.560519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615273+0000","last_change":"2024-08-28T19:18:30.739812+0000","last_active":"2024-08-28T19:20:09.615273+0000","last_peered":"2024-08-28T19:20:09.615273+0000","last_clean":"2024-08-28T19:20:09.615273+0000","last_became_active":"2024-08-28T19:18:30.739414+0000","last_became_peered":"2024-08-28T19:18:30.739414+0000","last_unstale":"2024-08-28T19:20:09.615273+0000","last_undegraded":"2024-08-28T19:20:09.615273+0000","last_fullsized":"2024-08-28T19:20:09.615273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:56:56.594993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010962+0000","last_change":"2024-08-28T19:18:39.528281+0000","last_active":"2024-08-28T19:20:09.010962+0000","last_peered":"2024-08-28T19:20:09.010962+0000","last_clean":"2024-08-28T19:20:09.010962+0000","last_became_active":"2024-08-28T19:18:39.527142+0000","last_became_peered":"2024-08-28T19:18:39.527142+0000","last_unstale":"2024-08-28T19:20:09.010962+0000","last_undegraded":"2024-08-28T19:20:09.010962+0000","last_fullsized":"2024-08-28T19:20:09.010962+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:44:19.123662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872571+0000","last_change":"2024-08-28T19:18:32.490430+0000","last_active":"2024-08-28T19:20:37.872571+0000","last_peered":"2024-08-28T19:20:37.872571+0000","last_clean":"2024-08-28T19:20:37.872571+0000","last_became_active":"2024-08-28T19:18:32.490020+0000","last_became_peered":"2024-08-28T19:18:32.490020+0000","last_unstale":"2024-08-28T19:20:37.872571+0000","last_undegraded":"2024-08-28T19:20:37.872571+0000","last_fullsized":"2024-08-28T19:20:37.872571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:52:31.153012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'17","reported_seq":57,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:41.222712+0000","last_change":"2024-08-28T19:18:30.467060+0000","last_active":"2024-08-28T19:20:41.222712+0000","last_peered":"2024-08-28T19:20:41.222712+0000","last_clean":"2024-08-28T19:20:41.222712+0000","last_became_active":"2024-08-28T19:18:30.466689+0000","last_became_peered":"2024-08-28T19:18:30.466689+0000","last_unstale":"2024-08-28T19:20:41.222712+0000","last_undegraded":"2024-08-28T19:20:41.222712+0000","last_fullsized":"2024-08-28T19:20:41.222712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:18:36.508131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.524970+0000","last_change":"2024-08-28T19:18:39.682874+0000","last_active":"2024-08-28T19:20:09.524970+0000","last_peered":"2024-08-28T19:20:09.524970+0000","last_clean":"2024-08-28T19:20:09.524970+0000","last_became_active":"2024-08-28T19:18:39.681500+0000","last_became_peered":"2024-08-28T19:18:39.681500+0000","last_unstale":"2024-08-28T19:20:09.524970+0000","last_undegraded":"2024-08-28T19:20:09.524970+0000","last_fullsized":"2024-08-28T19:20:09.524970+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:45:41.815506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.856367+0000","last_change":"2024-08-28T19:18:32.482047+0000","last_active":"2024-08-28T19:20:37.856367+0000","last_peered":"2024-08-28T19:20:37.856367+0000","last_clean":"2024-08-28T19:20:37.856367+0000","last_became_active":"2024-08-28T19:18:32.481640+0000","last_became_peered":"2024-08-28T19:18:32.481640+0000","last_unstale":"2024-08-28T19:20:37.856367+0000","last_undegraded":"2024-08-28T19:20:37.856367+0000","last_fullsized":"2024-08-28T19:20:37.856367+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:34.717764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110454+0000","last_change":"2024-08-28T19:18:31.083524+0000","last_active":"2024-08-28T19:20:09.110454+0000","last_peered":"2024-08-28T19:20:09.110454+0000","last_clean":"2024-08-28T19:20:09.110454+0000","last_became_active":"2024-08-28T19:18:31.083106+0000","last_became_peered":"2024-08-28T19:18:31.083106+0000","last_unstale":"2024-08-28T19:20:09.110454+0000","last_undegraded":"2024-08-28T19:20:09.110454+0000","last_fullsized":"2024-08-28T19:20:09.110454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:16.180754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663607+0000","last_change":"2024-08-28T19:18:39.531884+0000","last_active":"2024-08-28T19:20:09.663607+0000","last_peered":"2024-08-28T19:20:09.663607+0000","last_clean":"2024-08-28T19:20:09.663607+0000","last_became_active":"2024-08-28T19:18:39.531255+0000","last_became_peered":"2024-08-28T19:18:39.531255+0000","last_unstale":"2024-08-28T19:20:09.663607+0000","last_undegraded":"2024-08-28T19:20:09.663607+0000","last_fullsized":"2024-08-28T19:20:09.663607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:36:55.496594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.862241+0000","last_change":"2024-08-28T19:18:32.485180+0000","last_active":"2024-08-28T19:20:37.862241+0000","last_peered":"2024-08-28T19:20:37.862241+0000","last_clean":"2024-08-28T19:20:37.862241+0000","last_became_active":"2024-08-28T19:18:32.484804+0000","last_became_peered":"2024-08-28T19:18:32.484804+0000","last_unstale":"2024-08-28T19:20:37.862241+0000","last_undegraded":"2024-08-28T19:20:37.862241+0000","last_fullsized":"2024-08-28T19:20:37.862241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:12:40.958251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007361+0000","last_change":"2024-08-28T19:18:31.082226+0000","last_active":"2024-08-28T19:20:09.007361+0000","last_peered":"2024-08-28T19:20:09.007361+0000","last_clean":"2024-08-28T19:20:09.007361+0000","last_became_active":"2024-08-28T19:18:31.081833+0000","last_became_peered":"2024-08-28T19:18:31.081833+0000","last_unstale":"2024-08-28T19:20:09.007361+0000","last_undegraded":"2024-08-28T19:20:09.007361+0000","last_fullsized":"2024-08-28T19:20:09.007361+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:33:56.298093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109762+0000","last_change":"2024-08-28T19:18:39.681288+0000","last_active":"2024-08-28T19:20:09.109762+0000","last_peered":"2024-08-28T19:20:09.109762+0000","last_clean":"2024-08-28T19:20:09.109762+0000","last_became_active":"2024-08-28T19:18:39.680719+0000","last_became_peered":"2024-08-28T19:18:39.680719+0000","last_unstale":"2024-08-28T19:20:09.109762+0000","last_undegraded":"2024-08-28T19:20:09.109762+0000","last_fullsized":"2024-08-28T19:20:09.109762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:53:35.554181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880497+0000","last_change":"2024-08-28T19:18:32.487105+0000","last_active":"2024-08-28T19:20:37.880497+0000","last_peered":"2024-08-28T19:20:37.880497+0000","last_clean":"2024-08-28T19:20:37.880497+0000","last_became_active":"2024-08-28T19:18:32.486643+0000","last_became_peered":"2024-08-28T19:18:32.486643+0000","last_unstale":"2024-08-28T19:20:37.880497+0000","last_undegraded":"2024-08-28T19:20:37.880497+0000","last_fullsized":"2024-08-28T19:20:37.880497+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:18:43.274493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"84'7","reported_seq":163,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:03.821152+0000","last_change":"2024-08-28T19:18:30.471447+0000","last_active":"2024-08-28T19:21:03.821152+0000","last_peered":"2024-08-28T19:21:03.821152+0000","last_clean":"2024-08-28T19:21:03.821152+0000","last_became_active":"2024-08-28T19:18:30.470798+0000","last_became_peered":"2024-08-28T19:18:30.470798+0000","last_unstale":"2024-08-28T19:21:03.821152+0000","last_undegraded":"2024-08-28T19:21:03.821152+0000","last_fullsized":"2024-08-28T19:21:03.821152+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:50:13.948439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162958+0000","last_change":"2024-08-28T19:18:39.524190+0000","last_active":"2024-08-28T19:20:09.162958+0000","last_peered":"2024-08-28T19:20:09.162958+0000","last_clean":"2024-08-28T19:20:09.162958+0000","last_became_active":"2024-08-28T19:18:39.523771+0000","last_became_peered":"2024-08-28T19:18:39.523771+0000","last_unstale":"2024-08-28T19:20:09.162958+0000","last_undegraded":"2024-08-28T19:20:09.162958+0000","last_fullsized":"2024-08-28T19:20:09.162958+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:43:43.984625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.874995+0000","last_change":"2024-08-28T19:18:32.487212+0000","last_active":"2024-08-28T19:20:37.874995+0000","last_peered":"2024-08-28T19:20:37.874995+0000","last_clean":"2024-08-28T19:20:37.874995+0000","last_became_active":"2024-08-28T19:18:32.485432+0000","last_became_peered":"2024-08-28T19:18:32.485432+0000","last_unstale":"2024-08-28T19:20:37.874995+0000","last_undegraded":"2024-08-28T19:20:37.874995+0000","last_fullsized":"2024-08-28T19:20:37.874995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:52:11.672193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":159,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:03.820333+0000","last_change":"2024-08-28T19:18:31.082210+0000","last_active":"2024-08-28T19:21:03.820333+0000","last_peered":"2024-08-28T19:21:03.820333+0000","last_clean":"2024-08-28T19:21:03.820333+0000","last_became_active":"2024-08-28T19:18:31.081814+0000","last_became_peered":"2024-08-28T19:18:31.081814+0000","last_unstale":"2024-08-28T19:21:03.820333+0000","last_undegraded":"2024-08-28T19:21:03.820333+0000","last_fullsized":"2024-08-28T19:21:03.820333+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:33:32.281431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163227+0000","last_change":"2024-08-28T19:18:39.681858+0000","last_active":"2024-08-28T19:20:09.163227+0000","last_peered":"2024-08-28T19:20:09.163227+0000","last_clean":"2024-08-28T19:20:09.163227+0000","last_became_active":"2024-08-28T19:18:39.681382+0000","last_became_peered":"2024-08-28T19:18:39.681382+0000","last_unstale":"2024-08-28T19:20:09.163227+0000","last_undegraded":"2024-08-28T19:20:09.163227+0000","last_fullsized":"2024-08-28T19:20:09.163227+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:23:22.021022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.857639+0000","last_change":"2024-08-28T19:18:32.486802+0000","last_active":"2024-08-28T19:20:37.857639+0000","last_peered":"2024-08-28T19:20:37.857639+0000","last_clean":"2024-08-28T19:20:37.857639+0000","last_became_active":"2024-08-28T19:18:32.486479+0000","last_became_peered":"2024-08-28T19:18:32.486479+0000","last_unstale":"2024-08-28T19:20:37.857639+0000","last_undegraded":"2024-08-28T19:20:37.857639+0000","last_fullsized":"2024-08-28T19:20:37.857639+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:17:17.654757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529084+0000","last_change":"2024-08-28T19:18:30.470447+0000","last_active":"2024-08-28T19:20:09.529084+0000","last_peered":"2024-08-28T19:20:09.529084+0000","last_clean":"2024-08-28T19:20:09.529084+0000","last_became_active":"2024-08-28T19:18:30.469862+0000","last_became_peered":"2024-08-28T19:18:30.469862+0000","last_unstale":"2024-08-28T19:20:09.529084+0000","last_undegraded":"2024-08-28T19:20:09.529084+0000","last_fullsized":"2024-08-28T19:20:09.529084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:27:46.660492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009270+0000","last_change":"2024-08-28T19:19:08.624635+0000","last_active":"2024-08-28T19:20:09.009270+0000","last_peered":"2024-08-28T19:20:09.009270+0000","last_clean":"2024-08-28T19:20:09.009270+0000","last_became_active":"2024-08-28T19:19:08.622845+0000","last_became_peered":"2024-08-28T19:19:08.622845+0000","last_unstale":"2024-08-28T19:20:09.009270+0000","last_undegraded":"2024-08-28T19:20:09.009270+0000","last_fullsized":"2024-08-28T19:20:09.009270+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:56:40.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865852+0000","last_change":"2024-08-28T19:18:32.487947+0000","last_active":"2024-08-28T19:20:37.865852+0000","last_peered":"2024-08-28T19:20:37.865852+0000","last_clean":"2024-08-28T19:20:37.865852+0000","last_became_active":"2024-08-28T19:18:32.487592+0000","last_became_peered":"2024-08-28T19:18:32.487592+0000","last_unstale":"2024-08-28T19:20:37.865852+0000","last_undegraded":"2024-08-28T19:20:37.865852+0000","last_fullsized":"2024-08-28T19:20:37.865852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:30:29.550644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615576+0000","last_change":"2024-08-28T19:18:30.465044+0000","last_active":"2024-08-28T19:20:09.615576+0000","last_peered":"2024-08-28T19:20:09.615576+0000","last_clean":"2024-08-28T19:20:09.615576+0000","last_became_active":"2024-08-28T19:18:30.464580+0000","last_became_peered":"2024-08-28T19:18:30.464580+0000","last_unstale":"2024-08-28T19:20:09.615576+0000","last_undegraded":"2024-08-28T19:20:09.615576+0000","last_fullsized":"2024-08-28T19:20:09.615576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:21:17.223311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":114,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.528964+0000","last_change":"2024-08-28T19:17:17.063316+0000","last_active":"2024-08-28T19:20:09.528964+0000","last_peered":"2024-08-28T19:20:09.528964+0000","last_clean":"2024-08-28T19:20:09.528964+0000","last_became_active":"2024-08-28T19:17:17.057091+0000","last_became_peered":"2024-08-28T19:17:17.057091+0000","last_unstale":"2024-08-28T19:20:09.528964+0000","last_undegraded":"2024-08-28T19:20:09.528964+0000","last_fullsized":"2024-08-28T19:20:09.528964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_clean_scrub_stamp":"2024-08-28T19:13:56.815467+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:35:51.499703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"82'1","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529202+0000","last_change":"2024-08-28T19:19:08.621402+0000","last_active":"2024-08-28T19:20:09.529202+0000","last_peered":"2024-08-28T19:20:09.529202+0000","last_clean":"2024-08-28T19:20:09.529202+0000","last_became_active":"2024-08-28T19:19:08.620779+0000","last_became_peered":"2024-08-28T19:19:08.620779+0000","last_unstale":"2024-08-28T19:20:09.529202+0000","last_undegraded":"2024-08-28T19:20:09.529202+0000","last_fullsized":"2024-08-28T19:20:09.529202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:39.391414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886997+0000","last_change":"2024-08-28T19:18:32.490066+0000","last_active":"2024-08-28T19:20:37.886997+0000","last_peered":"2024-08-28T19:20:37.886997+0000","last_clean":"2024-08-28T19:20:37.886997+0000","last_became_active":"2024-08-28T19:18:32.489560+0000","last_became_peered":"2024-08-28T19:18:32.489560+0000","last_unstale":"2024-08-28T19:20:37.886997+0000","last_undegraded":"2024-08-28T19:20:37.886997+0000","last_fullsized":"2024-08-28T19:20:37.886997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:22:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'5","reported_seq":155,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:58.857569+0000","last_change":"2024-08-28T19:18:31.083078+0000","last_active":"2024-08-28T19:20:58.857569+0000","last_peered":"2024-08-28T19:20:58.857569+0000","last_clean":"2024-08-28T19:20:58.857569+0000","last_became_active":"2024-08-28T19:18:31.082575+0000","last_became_peered":"2024-08-28T19:18:31.082575+0000","last_unstale":"2024-08-28T19:20:58.857569+0000","last_undegraded":"2024-08-28T19:20:58.857569+0000","last_fullsized":"2024-08-28T19:20:58.857569+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":110,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007862+0000","last_change":"2024-08-28T19:18:39.519814+0000","last_active":"2024-08-28T19:20:09.007862+0000","last_peered":"2024-08-28T19:20:09.007862+0000","last_clean":"2024-08-28T19:20:09.007862+0000","last_became_active":"2024-08-28T19:18:39.519258+0000","last_became_peered":"2024-08-28T19:18:39.519258+0000","last_unstale":"2024-08-28T19:20:09.007862+0000","last_undegraded":"2024-08-28T19:20:09.007862+0000","last_fullsized":"2024-08-28T19:20:09.007862+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:03:04.051376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871177+0000","last_change":"2024-08-28T19:18:32.495426+0000","last_active":"2024-08-28T19:20:37.871177+0000","last_peered":"2024-08-28T19:20:37.871177+0000","last_clean":"2024-08-28T19:20:37.871177+0000","last_became_active":"2024-08-28T19:18:32.495075+0000","last_became_peered":"2024-08-28T19:18:32.495075+0000","last_unstale":"2024-08-28T19:20:37.871177+0000","last_undegraded":"2024-08-28T19:20:37.871177+0000","last_fullsized":"2024-08-28T19:20:37.871177+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:40:22.456666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'54","reported_seq":111,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:03.560174+0000","last_change":"2024-08-28T19:18:30.472708+0000","last_active":"2024-08-28T19:21:03.560174+0000","last_peered":"2024-08-28T19:21:03.560174+0000","last_clean":"2024-08-28T19:21:03.560174+0000","last_became_active":"2024-08-28T19:18:30.472320+0000","last_became_peered":"2024-08-28T19:18:30.472320+0000","last_unstale":"2024-08-28T19:21:03.560174+0000","last_undegraded":"2024-08-28T19:21:03.560174+0000","last_fullsized":"2024-08-28T19:21:03.560174+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:57:12.055626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":19,"num_write":63,"num_write_kb":739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010493+0000","last_change":"2024-08-28T19:18:39.519398+0000","last_active":"2024-08-28T19:20:09.010493+0000","last_peered":"2024-08-28T19:20:09.010493+0000","last_clean":"2024-08-28T19:20:09.010493+0000","last_became_active":"2024-08-28T19:18:39.518997+0000","last_became_peered":"2024-08-28T19:18:39.518997+0000","last_unstale":"2024-08-28T19:20:09.010493+0000","last_undegraded":"2024-08-28T19:20:09.010493+0000","last_fullsized":"2024-08-28T19:20:09.010493+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:33.431608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.879604+0000","last_change":"2024-08-28T19:18:32.489439+0000","last_active":"2024-08-28T19:20:37.879604+0000","last_peered":"2024-08-28T19:20:37.879604+0000","last_clean":"2024-08-28T19:20:37.879604+0000","last_became_active":"2024-08-28T19:18:32.489054+0000","last_became_peered":"2024-08-28T19:18:32.489054+0000","last_unstale":"2024-08-28T19:20:37.879604+0000","last_undegraded":"2024-08-28T19:20:37.879604+0000","last_fullsized":"2024-08-28T19:20:37.879604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:31:32.647384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163358+0000","last_change":"2024-08-28T19:18:30.464993+0000","last_active":"2024-08-28T19:20:09.163358+0000","last_peered":"2024-08-28T19:20:09.163358+0000","last_clean":"2024-08-28T19:20:09.163358+0000","last_became_active":"2024-08-28T19:18:30.463951+0000","last_became_peered":"2024-08-28T19:18:30.463951+0000","last_unstale":"2024-08-28T19:20:09.163358+0000","last_undegraded":"2024-08-28T19:20:09.163358+0000","last_fullsized":"2024-08-28T19:20:09.163358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:06.780058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008234+0000","last_change":"2024-08-28T19:18:39.533061+0000","last_active":"2024-08-28T19:20:09.008234+0000","last_peered":"2024-08-28T19:20:09.008234+0000","last_clean":"2024-08-28T19:20:09.008234+0000","last_became_active":"2024-08-28T19:18:39.532607+0000","last_became_peered":"2024-08-28T19:18:39.532607+0000","last_unstale":"2024-08-28T19:20:09.008234+0000","last_undegraded":"2024-08-28T19:20:09.008234+0000","last_fullsized":"2024-08-28T19:20:09.008234+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:00:57.997665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875549+0000","last_change":"2024-08-28T19:18:32.487937+0000","last_active":"2024-08-28T19:20:37.875549+0000","last_peered":"2024-08-28T19:20:37.875549+0000","last_clean":"2024-08-28T19:20:37.875549+0000","last_became_active":"2024-08-28T19:18:32.487417+0000","last_became_peered":"2024-08-28T19:18:32.487417+0000","last_unstale":"2024-08-28T19:20:37.875549+0000","last_undegraded":"2024-08-28T19:20:37.875549+0000","last_fullsized":"2024-08-28T19:20:37.875549+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:42.497364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110517+0000","last_change":"2024-08-28T19:18:30.470742+0000","last_active":"2024-08-28T19:20:09.110517+0000","last_peered":"2024-08-28T19:20:09.110517+0000","last_clean":"2024-08-28T19:20:09.110517+0000","last_became_active":"2024-08-28T19:18:30.470135+0000","last_became_peered":"2024-08-28T19:18:30.470135+0000","last_unstale":"2024-08-28T19:20:09.110517+0000","last_undegraded":"2024-08-28T19:20:09.110517+0000","last_fullsized":"2024-08-28T19:20:09.110517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:49:39.581119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008429+0000","last_change":"2024-08-28T19:18:39.524052+0000","last_active":"2024-08-28T19:20:09.008429+0000","last_peered":"2024-08-28T19:20:09.008429+0000","last_clean":"2024-08-28T19:20:09.008429+0000","last_became_active":"2024-08-28T19:18:39.523601+0000","last_became_peered":"2024-08-28T19:18:39.523601+0000","last_unstale":"2024-08-28T19:20:09.008429+0000","last_undegraded":"2024-08-28T19:20:09.008429+0000","last_fullsized":"2024-08-28T19:20:09.008429+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:54:53.302424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883838+0000","last_change":"2024-08-28T19:18:32.491151+0000","last_active":"2024-08-28T19:20:37.883838+0000","last_peered":"2024-08-28T19:20:37.883838+0000","last_clean":"2024-08-28T19:20:37.883838+0000","last_became_active":"2024-08-28T19:18:32.490798+0000","last_became_peered":"2024-08-28T19:18:32.490798+0000","last_unstale":"2024-08-28T19:20:37.883838+0000","last_undegraded":"2024-08-28T19:20:37.883838+0000","last_fullsized":"2024-08-28T19:20:37.883838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:41:00.771019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109743+0000","last_change":"2024-08-28T19:18:31.084029+0000","last_active":"2024-08-28T19:20:09.109743+0000","last_peered":"2024-08-28T19:20:09.109743+0000","last_clean":"2024-08-28T19:20:09.109743+0000","last_became_active":"2024-08-28T19:18:31.083507+0000","last_became_peered":"2024-08-28T19:18:31.083507+0000","last_unstale":"2024-08-28T19:20:09.109743+0000","last_undegraded":"2024-08-28T19:20:09.109743+0000","last_fullsized":"2024-08-28T19:20:09.109743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:15:36.451702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163318+0000","last_change":"2024-08-28T19:18:39.527249+0000","last_active":"2024-08-28T19:20:09.163318+0000","last_peered":"2024-08-28T19:20:09.163318+0000","last_clean":"2024-08-28T19:20:09.163318+0000","last_became_active":"2024-08-28T19:18:39.526834+0000","last_became_peered":"2024-08-28T19:18:39.526834+0000","last_unstale":"2024-08-28T19:20:09.163318+0000","last_undegraded":"2024-08-28T19:20:09.163318+0000","last_fullsized":"2024-08-28T19:20:09.163318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:48:02.294738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.876604+0000","last_change":"2024-08-28T19:18:32.497337+0000","last_active":"2024-08-28T19:20:37.876604+0000","last_peered":"2024-08-28T19:20:37.876604+0000","last_clean":"2024-08-28T19:20:37.876604+0000","last_became_active":"2024-08-28T19:18:32.496929+0000","last_became_peered":"2024-08-28T19:18:32.496929+0000","last_unstale":"2024-08-28T19:20:37.876604+0000","last_undegraded":"2024-08-28T19:20:37.876604+0000","last_fullsized":"2024-08-28T19:20:37.876604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:14.748765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":159,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:02.819590+0000","last_change":"2024-08-28T19:18:30.471353+0000","last_active":"2024-08-28T19:21:02.819590+0000","last_peered":"2024-08-28T19:21:02.819590+0000","last_clean":"2024-08-28T19:21:02.819590+0000","last_became_active":"2024-08-28T19:18:30.470283+0000","last_became_peered":"2024-08-28T19:18:30.470283+0000","last_unstale":"2024-08-28T19:21:02.819590+0000","last_undegraded":"2024-08-28T19:21:02.819590+0000","last_fullsized":"2024-08-28T19:21:02.819590+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:09:46.444864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007249+0000","last_change":"2024-08-28T19:18:39.519226+0000","last_active":"2024-08-28T19:20:09.007249+0000","last_peered":"2024-08-28T19:20:09.007249+0000","last_clean":"2024-08-28T19:20:09.007249+0000","last_became_active":"2024-08-28T19:18:39.518738+0000","last_became_peered":"2024-08-28T19:18:39.518738+0000","last_unstale":"2024-08-28T19:20:09.007249+0000","last_undegraded":"2024-08-28T19:20:09.007249+0000","last_fullsized":"2024-08-28T19:20:09.007249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:39:35.114702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.877692+0000","last_change":"2024-08-28T19:18:32.493060+0000","last_active":"2024-08-28T19:20:37.877692+0000","last_peered":"2024-08-28T19:20:37.877692+0000","last_clean":"2024-08-28T19:20:37.877692+0000","last_became_active":"2024-08-28T19:18:32.492590+0000","last_became_peered":"2024-08-28T19:18:32.492590+0000","last_unstale":"2024-08-28T19:20:37.877692+0000","last_undegraded":"2024-08-28T19:20:37.877692+0000","last_fullsized":"2024-08-28T19:20:37.877692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:02:02.165301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008761+0000","last_change":"2024-08-28T19:18:30.470674+0000","last_active":"2024-08-28T19:20:09.008761+0000","last_peered":"2024-08-28T19:20:09.008761+0000","last_clean":"2024-08-28T19:20:09.008761+0000","last_became_active":"2024-08-28T19:18:30.470275+0000","last_became_peered":"2024-08-28T19:18:30.470275+0000","last_unstale":"2024-08-28T19:20:09.008761+0000","last_undegraded":"2024-08-28T19:20:09.008761+0000","last_fullsized":"2024-08-28T19:20:09.008761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:01:48.547315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663581+0000","last_change":"2024-08-28T19:19:08.624194+0000","last_active":"2024-08-28T19:20:09.663581+0000","last_peered":"2024-08-28T19:20:09.663581+0000","last_clean":"2024-08-28T19:20:09.663581+0000","last_became_active":"2024-08-28T19:19:08.623681+0000","last_became_peered":"2024-08-28T19:19:08.623681+0000","last_unstale":"2024-08-28T19:20:09.663581+0000","last_undegraded":"2024-08-28T19:20:09.663581+0000","last_fullsized":"2024-08-28T19:20:09.663581+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:21:11.099931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872115+0000","last_change":"2024-08-28T19:18:32.488210+0000","last_active":"2024-08-28T19:20:37.872115+0000","last_peered":"2024-08-28T19:20:37.872115+0000","last_clean":"2024-08-28T19:20:37.872115+0000","last_became_active":"2024-08-28T19:18:32.487853+0000","last_became_peered":"2024-08-28T19:18:32.487853+0000","last_unstale":"2024-08-28T19:20:37.872115+0000","last_undegraded":"2024-08-28T19:20:37.872115+0000","last_fullsized":"2024-08-28T19:20:37.872115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:20:51.808958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011302+0000","last_change":"2024-08-28T19:18:30.465306+0000","last_active":"2024-08-28T19:20:09.011302+0000","last_peered":"2024-08-28T19:20:09.011302+0000","last_clean":"2024-08-28T19:20:09.011302+0000","last_became_active":"2024-08-28T19:18:30.464850+0000","last_became_peered":"2024-08-28T19:18:30.464850+0000","last_unstale":"2024-08-28T19:20:09.011302+0000","last_undegraded":"2024-08-28T19:20:09.011302+0000","last_fullsized":"2024-08-28T19:20:09.011302+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:24:26.628999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008722+0000","last_change":"2024-08-28T19:18:39.524005+0000","last_active":"2024-08-28T19:20:09.008722+0000","last_peered":"2024-08-28T19:20:09.008722+0000","last_clean":"2024-08-28T19:20:09.008722+0000","last_became_active":"2024-08-28T19:18:39.523594+0000","last_became_peered":"2024-08-28T19:18:39.523594+0000","last_unstale":"2024-08-28T19:20:09.008722+0000","last_undegraded":"2024-08-28T19:20:09.008722+0000","last_fullsized":"2024-08-28T19:20:09.008722+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:35:32.484399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872816+0000","last_change":"2024-08-28T19:18:32.489088+0000","last_active":"2024-08-28T19:20:37.872816+0000","last_peered":"2024-08-28T19:20:37.872816+0000","last_clean":"2024-08-28T19:20:37.872816+0000","last_became_active":"2024-08-28T19:18:32.488717+0000","last_became_peered":"2024-08-28T19:18:32.488717+0000","last_unstale":"2024-08-28T19:20:37.872816+0000","last_undegraded":"2024-08-28T19:20:37.872816+0000","last_fullsized":"2024-08-28T19:20:37.872816+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:25.468919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"84'41","reported_seq":320,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:04.822762+0000","last_change":"2024-08-28T19:18:30.468674+0000","last_active":"2024-08-28T19:21:04.822762+0000","last_peered":"2024-08-28T19:21:04.822762+0000","last_clean":"2024-08-28T19:21:04.822762+0000","last_became_active":"2024-08-28T19:18:30.468300+0000","last_became_peered":"2024-08-28T19:18:30.468300+0000","last_unstale":"2024-08-28T19:21:04.822762+0000","last_undegraded":"2024-08-28T19:21:04.822762+0000","last_fullsized":"2024-08-28T19:21:04.822762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:12:06.237498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69405,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":239,"num_read_kb":188,"num_write":43,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007140+0000","last_change":"2024-08-28T19:18:39.526131+0000","last_active":"2024-08-28T19:20:09.007140+0000","last_peered":"2024-08-28T19:20:09.007140+0000","last_clean":"2024-08-28T19:20:09.007140+0000","last_became_active":"2024-08-28T19:18:39.525708+0000","last_became_peered":"2024-08-28T19:18:39.525708+0000","last_unstale":"2024-08-28T19:20:09.007140+0000","last_undegraded":"2024-08-28T19:20:09.007140+0000","last_fullsized":"2024-08-28T19:20:09.007140+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:21:52.156487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.885078+0000","last_change":"2024-08-28T19:18:32.487634+0000","last_active":"2024-08-28T19:20:37.885078+0000","last_peered":"2024-08-28T19:20:37.885078+0000","last_clean":"2024-08-28T19:20:37.885078+0000","last_became_active":"2024-08-28T19:18:32.486702+0000","last_became_peered":"2024-08-28T19:18:32.486702+0000","last_unstale":"2024-08-28T19:20:37.885078+0000","last_undegraded":"2024-08-28T19:20:37.885078+0000","last_fullsized":"2024-08-28T19:20:37.885078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:08:12.606905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009190+0000","last_change":"2024-08-28T19:18:31.083494+0000","last_active":"2024-08-28T19:20:09.009190+0000","last_peered":"2024-08-28T19:20:09.009190+0000","last_clean":"2024-08-28T19:20:09.009190+0000","last_became_active":"2024-08-28T19:18:31.083086+0000","last_became_peered":"2024-08-28T19:18:31.083086+0000","last_unstale":"2024-08-28T19:20:09.009190+0000","last_undegraded":"2024-08-28T19:20:09.009190+0000","last_fullsized":"2024-08-28T19:20:09.009190+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:14:54.388364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615438+0000","last_change":"2024-08-28T19:18:39.532155+0000","last_active":"2024-08-28T19:20:09.615438+0000","last_peered":"2024-08-28T19:20:09.615438+0000","last_clean":"2024-08-28T19:20:09.615438+0000","last_became_active":"2024-08-28T19:18:39.531565+0000","last_became_peered":"2024-08-28T19:18:39.531565+0000","last_unstale":"2024-08-28T19:20:09.615438+0000","last_undegraded":"2024-08-28T19:20:09.615438+0000","last_fullsized":"2024-08-28T19:20:09.615438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:29:48.824622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.881019+0000","last_change":"2024-08-28T19:18:32.496296+0000","last_active":"2024-08-28T19:20:37.881019+0000","last_peered":"2024-08-28T19:20:37.881019+0000","last_clean":"2024-08-28T19:20:37.881019+0000","last_became_active":"2024-08-28T19:18:32.495680+0000","last_became_peered":"2024-08-28T19:18:32.495680+0000","last_unstale":"2024-08-28T19:20:37.881019+0000","last_undegraded":"2024-08-28T19:20:37.881019+0000","last_fullsized":"2024-08-28T19:20:37.881019+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:27:34.544529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529311+0000","last_change":"2024-08-28T19:19:08.631758+0000","last_active":"2024-08-28T19:20:09.529311+0000","last_peered":"2024-08-28T19:20:09.529311+0000","last_clean":"2024-08-28T19:20:09.529311+0000","last_became_active":"2024-08-28T19:19:08.631202+0000","last_became_peered":"2024-08-28T19:19:08.631202+0000","last_unstale":"2024-08-28T19:20:09.529311+0000","last_undegraded":"2024-08-28T19:20:09.529311+0000","last_fullsized":"2024-08-28T19:20:09.529311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:27:43.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.006784+0000","last_change":"2024-08-28T19:18:39.531157+0000","last_active":"2024-08-28T19:20:09.006784+0000","last_peered":"2024-08-28T19:20:09.006784+0000","last_clean":"2024-08-28T19:20:09.006784+0000","last_became_active":"2024-08-28T19:18:39.530675+0000","last_became_peered":"2024-08-28T19:18:39.530675+0000","last_unstale":"2024-08-28T19:20:09.006784+0000","last_undegraded":"2024-08-28T19:20:09.006784+0000","last_fullsized":"2024-08-28T19:20:09.006784+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:38:40.414854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883692+0000","last_change":"2024-08-28T19:18:32.482182+0000","last_active":"2024-08-28T19:20:37.883692+0000","last_peered":"2024-08-28T19:20:37.883692+0000","last_clean":"2024-08-28T19:20:37.883692+0000","last_became_active":"2024-08-28T19:18:32.481689+0000","last_became_peered":"2024-08-28T19:18:32.481689+0000","last_unstale":"2024-08-28T19:20:37.883692+0000","last_undegraded":"2024-08-28T19:20:37.883692+0000","last_fullsized":"2024-08-28T19:20:37.883692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:13:29.278469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'5","reported_seq":271,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:04.822781+0000","last_change":"2024-08-28T19:18:30.475244+0000","last_active":"2024-08-28T19:21:04.822781+0000","last_peered":"2024-08-28T19:21:04.822781+0000","last_clean":"2024-08-28T19:21:04.822781+0000","last_became_active":"2024-08-28T19:18:30.474861+0000","last_became_peered":"2024-08-28T19:18:30.474861+0000","last_unstale":"2024-08-28T19:21:04.822781+0000","last_undegraded":"2024-08-28T19:21:04.822781+0000","last_fullsized":"2024-08-28T19:21:04.822781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:27:09.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525108+0000","last_change":"2024-08-28T19:18:39.520022+0000","last_active":"2024-08-28T19:20:09.525108+0000","last_peered":"2024-08-28T19:20:09.525108+0000","last_clean":"2024-08-28T19:20:09.525108+0000","last_became_active":"2024-08-28T19:18:39.519584+0000","last_became_peered":"2024-08-28T19:18:39.519584+0000","last_unstale":"2024-08-28T19:20:09.525108+0000","last_undegraded":"2024-08-28T19:20:09.525108+0000","last_fullsized":"2024-08-28T19:20:09.525108+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:27:15.851303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010363+0000","last_change":"2024-08-28T19:18:39.522607+0000","last_active":"2024-08-28T19:20:09.010363+0000","last_peered":"2024-08-28T19:20:09.010363+0000","last_clean":"2024-08-28T19:20:09.010363+0000","last_became_active":"2024-08-28T19:18:39.522204+0000","last_became_peered":"2024-08-28T19:18:39.522204+0000","last_unstale":"2024-08-28T19:20:09.010363+0000","last_undegraded":"2024-08-28T19:20:09.010363+0000","last_fullsized":"2024-08-28T19:20:09.010363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:02:08.865294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865617+0000","last_change":"2024-08-28T19:18:32.485577+0000","last_active":"2024-08-28T19:20:37.865617+0000","last_peered":"2024-08-28T19:20:37.865617+0000","last_clean":"2024-08-28T19:20:37.865617+0000","last_became_active":"2024-08-28T19:18:32.485243+0000","last_became_peered":"2024-08-28T19:18:32.485243+0000","last_unstale":"2024-08-28T19:20:37.865617+0000","last_undegraded":"2024-08-28T19:20:37.865617+0000","last_fullsized":"2024-08-28T19:20:37.865617+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:05:53.019365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008799+0000","last_change":"2024-08-28T19:18:30.470670+0000","last_active":"2024-08-28T19:20:09.008799+0000","last_peered":"2024-08-28T19:20:09.008799+0000","last_clean":"2024-08-28T19:20:09.008799+0000","last_became_active":"2024-08-28T19:18:30.470264+0000","last_became_peered":"2024-08-28T19:18:30.470264+0000","last_unstale":"2024-08-28T19:20:09.008799+0000","last_undegraded":"2024-08-28T19:20:09.008799+0000","last_fullsized":"2024-08-28T19:20:09.008799+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:35.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663847+0000","last_change":"2024-08-28T19:18:39.530948+0000","last_active":"2024-08-28T19:20:09.663847+0000","last_peered":"2024-08-28T19:20:09.663847+0000","last_clean":"2024-08-28T19:20:09.663847+0000","last_became_active":"2024-08-28T19:18:39.530473+0000","last_became_peered":"2024-08-28T19:18:39.530473+0000","last_unstale":"2024-08-28T19:20:09.663847+0000","last_undegraded":"2024-08-28T19:20:09.663847+0000","last_fullsized":"2024-08-28T19:20:09.663847+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:21:57.504582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.869892+0000","last_change":"2024-08-28T19:18:32.486027+0000","last_active":"2024-08-28T19:20:37.869892+0000","last_peered":"2024-08-28T19:20:37.869892+0000","last_clean":"2024-08-28T19:20:37.869892+0000","last_became_active":"2024-08-28T19:18:32.483273+0000","last_became_peered":"2024-08-28T19:18:32.483273+0000","last_unstale":"2024-08-28T19:20:37.869892+0000","last_undegraded":"2024-08-28T19:20:37.869892+0000","last_fullsized":"2024-08-28T19:20:37.869892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:47:37.680695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162978+0000","last_change":"2024-08-28T19:18:30.464811+0000","last_active":"2024-08-28T19:20:09.162978+0000","last_peered":"2024-08-28T19:20:09.162978+0000","last_clean":"2024-08-28T19:20:09.162978+0000","last_became_active":"2024-08-28T19:18:30.463986+0000","last_became_peered":"2024-08-28T19:18:30.463986+0000","last_unstale":"2024-08-28T19:20:09.162978+0000","last_undegraded":"2024-08-28T19:20:09.162978+0000","last_fullsized":"2024-08-28T19:20:09.162978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:43:46.582993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663619+0000","last_change":"2024-08-28T19:19:08.631382+0000","last_active":"2024-08-28T19:20:09.663619+0000","last_peered":"2024-08-28T19:20:09.663619+0000","last_clean":"2024-08-28T19:20:09.663619+0000","last_became_active":"2024-08-28T19:19:08.630869+0000","last_became_peered":"2024-08-28T19:19:08.630869+0000","last_unstale":"2024-08-28T19:20:09.663619+0000","last_undegraded":"2024-08-28T19:20:09.663619+0000","last_fullsized":"2024-08-28T19:20:09.663619+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:32:54.942408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875456+0000","last_change":"2024-08-28T19:18:32.484120+0000","last_active":"2024-08-28T19:20:37.875456+0000","last_peered":"2024-08-28T19:20:37.875456+0000","last_clean":"2024-08-28T19:20:37.875456+0000","last_became_active":"2024-08-28T19:18:32.483013+0000","last_became_peered":"2024-08-28T19:18:32.483013+0000","last_unstale":"2024-08-28T19:20:37.875456+0000","last_undegraded":"2024-08-28T19:20:37.875456+0000","last_fullsized":"2024-08-28T19:20:37.875456+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:59:41.195873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010737+0000","last_change":"2024-08-28T19:18:39.531046+0000","last_active":"2024-08-28T19:20:09.010737+0000","last_peered":"2024-08-28T19:20:09.010737+0000","last_clean":"2024-08-28T19:20:09.010737+0000","last_became_active":"2024-08-28T19:18:39.530407+0000","last_became_peered":"2024-08-28T19:18:39.530407+0000","last_unstale":"2024-08-28T19:20:09.010737+0000","last_undegraded":"2024-08-28T19:20:09.010737+0000","last_fullsized":"2024-08-28T19:20:09.010737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:47:15.173021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'7","reported_seq":162,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:02.820158+0000","last_change":"2024-08-28T19:18:30.471268+0000","last_active":"2024-08-28T19:21:02.820158+0000","last_peered":"2024-08-28T19:21:02.820158+0000","last_clean":"2024-08-28T19:21:02.820158+0000","last_became_active":"2024-08-28T19:18:30.470845+0000","last_became_peered":"2024-08-28T19:18:30.470845+0000","last_unstale":"2024-08-28T19:21:02.820158+0000","last_undegraded":"2024-08-28T19:21:02.820158+0000","last_fullsized":"2024-08-28T19:21:02.820158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:18:18.086205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.860520+0000","last_change":"2024-08-28T19:18:32.479728+0000","last_active":"2024-08-28T19:20:37.860520+0000","last_peered":"2024-08-28T19:20:37.860520+0000","last_clean":"2024-08-28T19:20:37.860520+0000","last_became_active":"2024-08-28T19:18:32.478623+0000","last_became_peered":"2024-08-28T19:18:32.478623+0000","last_unstale":"2024-08-28T19:20:37.860520+0000","last_undegraded":"2024-08-28T19:20:37.860520+0000","last_fullsized":"2024-08-28T19:20:37.860520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:14:49.914808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011224+0000","last_change":"2024-08-28T19:18:39.531308+0000","last_active":"2024-08-28T19:20:09.011224+0000","last_peered":"2024-08-28T19:20:09.011224+0000","last_clean":"2024-08-28T19:20:09.011224+0000","last_became_active":"2024-08-28T19:18:39.530403+0000","last_became_peered":"2024-08-28T19:18:39.530403+0000","last_unstale":"2024-08-28T19:20:09.011224+0000","last_undegraded":"2024-08-28T19:20:09.011224+0000","last_fullsized":"2024-08-28T19:20:09.011224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:33.139668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873003+0000","last_change":"2024-08-28T19:18:32.494621+0000","last_active":"2024-08-28T19:20:37.873003+0000","last_peered":"2024-08-28T19:20:37.873003+0000","last_clean":"2024-08-28T19:20:37.873003+0000","last_became_active":"2024-08-28T19:18:32.494217+0000","last_became_peered":"2024-08-28T19:18:32.494217+0000","last_unstale":"2024-08-28T19:20:37.873003+0000","last_undegraded":"2024-08-28T19:20:37.873003+0000","last_fullsized":"2024-08-28T19:20:37.873003+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:54:15.419470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"84'12","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872043+0000","last_change":"2024-08-28T19:20:09.616552+0000","last_active":"2024-08-28T19:20:37.872043+0000","last_peered":"2024-08-28T19:20:37.872043+0000","last_clean":"2024-08-28T19:20:37.872043+0000","last_became_active":"2024-08-28T19:20:09.616159+0000","last_became_peered":"2024-08-28T19:20:09.616159+0000","last_unstale":"2024-08-28T19:20:37.872043+0000","last_undegraded":"2024-08-28T19:20:37.872043+0000","last_fullsized":"2024-08-28T19:20:37.872043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:04:10.354703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"84'8","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866959+0000","last_change":"2024-08-28T19:20:09.012824+0000","last_active":"2024-08-28T19:20:37.866959+0000","last_peered":"2024-08-28T19:20:37.866959+0000","last_clean":"2024-08-28T19:20:37.866959+0000","last_became_active":"2024-08-28T19:20:09.012490+0000","last_became_peered":"2024-08-28T19:20:09.012490+0000","last_unstale":"2024-08-28T19:20:37.866959+0000","last_undegraded":"2024-08-28T19:20:37.866959+0000","last_fullsized":"2024-08-28T19:20:37.866959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:49:02.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871666+0000","last_change":"2024-08-28T19:20:09.164523+0000","last_active":"2024-08-28T19:20:37.871666+0000","last_peered":"2024-08-28T19:20:37.871666+0000","last_clean":"2024-08-28T19:20:37.871666+0000","last_became_active":"2024-08-28T19:20:09.163825+0000","last_became_peered":"2024-08-28T19:20:09.163825+0000","last_unstale":"2024-08-28T19:20:37.871666+0000","last_undegraded":"2024-08-28T19:20:37.871666+0000","last_fullsized":"2024-08-28T19:20:37.871666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:42:48.141103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868174+0000","last_change":"2024-08-28T19:18:32.485446+0000","last_active":"2024-08-28T19:20:37.868174+0000","last_peered":"2024-08-28T19:20:37.868174+0000","last_clean":"2024-08-28T19:20:37.868174+0000","last_became_active":"2024-08-28T19:18:32.484197+0000","last_became_peered":"2024-08-28T19:18:32.484197+0000","last_unstale":"2024-08-28T19:20:37.868174+0000","last_undegraded":"2024-08-28T19:20:37.868174+0000","last_fullsized":"2024-08-28T19:20:37.868174+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:47:50.527915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868339+0000","last_change":"2024-08-28T19:18:32.489452+0000","last_active":"2024-08-28T19:20:37.868339+0000","last_peered":"2024-08-28T19:20:37.868339+0000","last_clean":"2024-08-28T19:20:37.868339+0000","last_became_active":"2024-08-28T19:18:32.487482+0000","last_became_peered":"2024-08-28T19:18:32.487482+0000","last_unstale":"2024-08-28T19:20:37.868339+0000","last_undegraded":"2024-08-28T19:20:37.868339+0000","last_fullsized":"2024-08-28T19:20:37.868339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:52:08.175080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884068+0000","last_change":"2024-08-28T19:18:32.483850+0000","last_active":"2024-08-28T19:20:37.884068+0000","last_peered":"2024-08-28T19:20:37.884068+0000","last_clean":"2024-08-28T19:20:37.884068+0000","last_became_active":"2024-08-28T19:18:32.483476+0000","last_became_peered":"2024-08-28T19:18:32.483476+0000","last_unstale":"2024-08-28T19:20:37.884068+0000","last_undegraded":"2024-08-28T19:20:37.884068+0000","last_fullsized":"2024-08-28T19:20:37.884068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:32:37.465218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868836+0000","last_change":"2024-08-28T19:18:32.492252+0000","last_active":"2024-08-28T19:20:37.868836+0000","last_peered":"2024-08-28T19:20:37.868836+0000","last_clean":"2024-08-28T19:20:37.868836+0000","last_became_active":"2024-08-28T19:18:32.491659+0000","last_became_peered":"2024-08-28T19:18:32.491659+0000","last_unstale":"2024-08-28T19:20:37.868836+0000","last_undegraded":"2024-08-28T19:20:37.868836+0000","last_fullsized":"2024-08-28T19:20:37.868836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:19:28.901559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865090+0000","last_change":"2024-08-28T19:18:32.491940+0000","last_active":"2024-08-28T19:20:37.865090+0000","last_peered":"2024-08-28T19:20:37.865090+0000","last_clean":"2024-08-28T19:20:37.865090+0000","last_became_active":"2024-08-28T19:18:32.487474+0000","last_became_peered":"2024-08-28T19:18:32.487474+0000","last_unstale":"2024-08-28T19:20:37.865090+0000","last_undegraded":"2024-08-28T19:20:37.865090+0000","last_fullsized":"2024-08-28T19:20:37.865090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:04:32.508620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871438+0000","last_change":"2024-08-28T19:18:32.485200+0000","last_active":"2024-08-28T19:20:37.871438+0000","last_peered":"2024-08-28T19:20:37.871438+0000","last_clean":"2024-08-28T19:20:37.871438+0000","last_became_active":"2024-08-28T19:18:32.484858+0000","last_became_peered":"2024-08-28T19:18:32.484858+0000","last_unstale":"2024-08-28T19:20:37.871438+0000","last_undegraded":"2024-08-28T19:20:37.871438+0000","last_fullsized":"2024-08-28T19:20:37.871438+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:10:57.925659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886493+0000","last_change":"2024-08-28T19:18:32.489903+0000","last_active":"2024-08-28T19:20:37.886493+0000","last_peered":"2024-08-28T19:20:37.886493+0000","last_clean":"2024-08-28T19:20:37.886493+0000","last_became_active":"2024-08-28T19:18:32.488950+0000","last_became_peered":"2024-08-28T19:18:32.488950+0000","last_unstale":"2024-08-28T19:20:37.886493+0000","last_undegraded":"2024-08-28T19:20:37.886493+0000","last_fullsized":"2024-08-28T19:20:37.886493+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:06:42.284177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872318+0000","last_change":"2024-08-28T19:18:32.488236+0000","last_active":"2024-08-28T19:20:37.872318+0000","last_peered":"2024-08-28T19:20:37.872318+0000","last_clean":"2024-08-28T19:20:37.872318+0000","last_became_active":"2024-08-28T19:18:32.487776+0000","last_became_peered":"2024-08-28T19:18:32.487776+0000","last_unstale":"2024-08-28T19:20:37.872318+0000","last_undegraded":"2024-08-28T19:20:37.872318+0000","last_fullsized":"2024-08-28T19:20:37.872318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:25:58.754740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875857+0000","last_change":"2024-08-28T19:18:32.485860+0000","last_active":"2024-08-28T19:20:37.875857+0000","last_peered":"2024-08-28T19:20:37.875857+0000","last_clean":"2024-08-28T19:20:37.875857+0000","last_became_active":"2024-08-28T19:18:32.485415+0000","last_became_peered":"2024-08-28T19:18:32.485415+0000","last_unstale":"2024-08-28T19:20:37.875857+0000","last_undegraded":"2024-08-28T19:20:37.875857+0000","last_fullsized":"2024-08-28T19:20:37.875857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:45:32.868651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873054+0000","last_change":"2024-08-28T19:18:32.485515+0000","last_active":"2024-08-28T19:20:37.873054+0000","last_peered":"2024-08-28T19:20:37.873054+0000","last_clean":"2024-08-28T19:20:37.873054+0000","last_became_active":"2024-08-28T19:18:32.485075+0000","last_became_peered":"2024-08-28T19:18:32.485075+0000","last_unstale":"2024-08-28T19:20:37.873054+0000","last_undegraded":"2024-08-28T19:20:37.873054+0000","last_fullsized":"2024-08-28T19:20:37.873054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:01:28.035479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864600+0000","last_change":"2024-08-28T19:18:32.484685+0000","last_active":"2024-08-28T19:20:37.864600+0000","last_peered":"2024-08-28T19:20:37.864600+0000","last_clean":"2024-08-28T19:20:37.864600+0000","last_became_active":"2024-08-28T19:18:32.484095+0000","last_became_peered":"2024-08-28T19:18:32.484095+0000","last_unstale":"2024-08-28T19:20:37.864600+0000","last_undegraded":"2024-08-28T19:20:37.864600+0000","last_fullsized":"2024-08-28T19:20:37.864600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:53:27.229186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884050+0000","last_change":"2024-08-28T19:18:32.488030+0000","last_active":"2024-08-28T19:20:37.884050+0000","last_peered":"2024-08-28T19:20:37.884050+0000","last_clean":"2024-08-28T19:20:37.884050+0000","last_became_active":"2024-08-28T19:18:32.487042+0000","last_became_peered":"2024-08-28T19:18:32.487042+0000","last_unstale":"2024-08-28T19:20:37.884050+0000","last_undegraded":"2024-08-28T19:20:37.884050+0000","last_fullsized":"2024-08-28T19:20:37.884050+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:56.966475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886125+0000","last_change":"2024-08-28T19:18:32.489442+0000","last_active":"2024-08-28T19:20:37.886125+0000","last_peered":"2024-08-28T19:20:37.886125+0000","last_clean":"2024-08-28T19:20:37.886125+0000","last_became_active":"2024-08-28T19:18:32.488405+0000","last_became_peered":"2024-08-28T19:18:32.488405+0000","last_unstale":"2024-08-28T19:20:37.886125+0000","last_undegraded":"2024-08-28T19:20:37.886125+0000","last_fullsized":"2024-08-28T19:20:37.886125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:21:03.026893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870138+0000","last_change":"2024-08-28T19:20:09.164305+0000","last_active":"2024-08-28T19:20:37.870138+0000","last_peered":"2024-08-28T19:20:37.870138+0000","last_clean":"2024-08-28T19:20:37.870138+0000","last_became_active":"2024-08-28T19:20:09.163834+0000","last_became_peered":"2024-08-28T19:20:09.163834+0000","last_unstale":"2024-08-28T19:20:37.870138+0000","last_undegraded":"2024-08-28T19:20:37.870138+0000","last_fullsized":"2024-08-28T19:20:37.870138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:20:10.137952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870501+0000","last_change":"2024-08-28T19:18:32.492077+0000","last_active":"2024-08-28T19:20:37.870501+0000","last_peered":"2024-08-28T19:20:37.870501+0000","last_clean":"2024-08-28T19:20:37.870501+0000","last_became_active":"2024-08-28T19:18:32.491722+0000","last_became_peered":"2024-08-28T19:18:32.491722+0000","last_unstale":"2024-08-28T19:20:37.870501+0000","last_undegraded":"2024-08-28T19:20:37.870501+0000","last_fullsized":"2024-08-28T19:20:37.870501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:16:21.697045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865541+0000","last_change":"2024-08-28T19:18:32.479845+0000","last_active":"2024-08-28T19:20:37.865541+0000","last_peered":"2024-08-28T19:20:37.865541+0000","last_clean":"2024-08-28T19:20:37.865541+0000","last_became_active":"2024-08-28T19:18:32.477690+0000","last_became_peered":"2024-08-28T19:18:32.477690+0000","last_unstale":"2024-08-28T19:20:37.865541+0000","last_undegraded":"2024-08-28T19:20:37.865541+0000","last_fullsized":"2024-08-28T19:20:37.865541+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:11.281415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866009+0000","last_change":"2024-08-28T19:19:08.624185+0000","last_active":"2024-08-28T19:20:37.866009+0000","last_peered":"2024-08-28T19:20:37.866009+0000","last_clean":"2024-08-28T19:20:37.866009+0000","last_became_active":"2024-08-28T19:19:08.623745+0000","last_became_peered":"2024-08-28T19:19:08.623745+0000","last_unstale":"2024-08-28T19:20:37.866009+0000","last_undegraded":"2024-08-28T19:20:37.866009+0000","last_fullsized":"2024-08-28T19:20:37.866009+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:56.657173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.887026+0000","last_change":"2024-08-28T19:18:32.499038+0000","last_active":"2024-08-28T19:20:37.887026+0000","last_peered":"2024-08-28T19:20:37.887026+0000","last_clean":"2024-08-28T19:20:37.887026+0000","last_became_active":"2024-08-28T19:18:32.498276+0000","last_became_peered":"2024-08-28T19:18:32.498276+0000","last_unstale":"2024-08-28T19:20:37.887026+0000","last_undegraded":"2024-08-28T19:20:37.887026+0000","last_fullsized":"2024-08-28T19:20:37.887026+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:17:45.468448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"84'16","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.882492+0000","last_change":"2024-08-28T19:18:32.492183+0000","last_active":"2024-08-28T19:20:37.882492+0000","last_peered":"2024-08-28T19:20:37.882492+0000","last_clean":"2024-08-28T19:20:37.882492+0000","last_became_active":"2024-08-28T19:18:32.491815+0000","last_became_peered":"2024-08-28T19:18:32.491815+0000","last_unstale":"2024-08-28T19:20:37.882492+0000","last_undegraded":"2024-08-28T19:20:37.882492+0000","last_fullsized":"2024-08-28T19:20:37.882492+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:13:11.183955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886985+0000","last_change":"2024-08-28T19:18:32.497399+0000","last_active":"2024-08-28T19:20:37.886985+0000","last_peered":"2024-08-28T19:20:37.886985+0000","last_clean":"2024-08-28T19:20:37.886985+0000","last_became_active":"2024-08-28T19:18:32.496901+0000","last_became_peered":"2024-08-28T19:18:32.496901+0000","last_unstale":"2024-08-28T19:20:37.886985+0000","last_undegraded":"2024-08-28T19:20:37.886985+0000","last_fullsized":"2024-08-28T19:20:37.886985+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:05:57.634590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864593+0000","last_change":"2024-08-28T19:18:32.495871+0000","last_active":"2024-08-28T19:20:37.864593+0000","last_peered":"2024-08-28T19:20:37.864593+0000","last_clean":"2024-08-28T19:20:37.864593+0000","last_became_active":"2024-08-28T19:18:32.495515+0000","last_became_peered":"2024-08-28T19:18:32.495515+0000","last_unstale":"2024-08-28T19:20:37.864593+0000","last_undegraded":"2024-08-28T19:20:37.864593+0000","last_fullsized":"2024-08-28T19:20:37.864593+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:53:56.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875472+0000","last_change":"2024-08-28T19:18:32.495711+0000","last_active":"2024-08-28T19:20:37.875472+0000","last_peered":"2024-08-28T19:20:37.875472+0000","last_clean":"2024-08-28T19:20:37.875472+0000","last_became_active":"2024-08-28T19:18:32.495271+0000","last_became_peered":"2024-08-28T19:18:32.495271+0000","last_unstale":"2024-08-28T19:20:37.875472+0000","last_undegraded":"2024-08-28T19:20:37.875472+0000","last_fullsized":"2024-08-28T19:20:37.875472+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:05.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880842+0000","last_change":"2024-08-28T19:18:32.489731+0000","last_active":"2024-08-28T19:20:37.880842+0000","last_peered":"2024-08-28T19:20:37.880842+0000","last_clean":"2024-08-28T19:20:37.880842+0000","last_became_active":"2024-08-28T19:18:32.489366+0000","last_became_peered":"2024-08-28T19:18:32.489366+0000","last_unstale":"2024-08-28T19:20:37.880842+0000","last_undegraded":"2024-08-28T19:20:37.880842+0000","last_fullsized":"2024-08-28T19:20:37.880842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:52:24.180155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.861421+0000","last_change":"2024-08-28T19:18:32.490270+0000","last_active":"2024-08-28T19:20:37.861421+0000","last_peered":"2024-08-28T19:20:37.861421+0000","last_clean":"2024-08-28T19:20:37.861421+0000","last_became_active":"2024-08-28T19:18:32.489667+0000","last_became_peered":"2024-08-28T19:18:32.489667+0000","last_unstale":"2024-08-28T19:20:37.861421+0000","last_undegraded":"2024-08-28T19:20:37.861421+0000","last_fullsized":"2024-08-28T19:20:37.861421+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:18.571978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873527+0000","last_change":"2024-08-28T19:18:32.482553+0000","last_active":"2024-08-28T19:20:37.873527+0000","last_peered":"2024-08-28T19:20:37.873527+0000","last_clean":"2024-08-28T19:20:37.873527+0000","last_became_active":"2024-08-28T19:18:32.481161+0000","last_became_peered":"2024-08-28T19:18:32.481161+0000","last_unstale":"2024-08-28T19:20:37.873527+0000","last_undegraded":"2024-08-28T19:20:37.873527+0000","last_fullsized":"2024-08-28T19:20:37.873527+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:13.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883684+0000","last_change":"2024-08-28T19:18:32.492473+0000","last_active":"2024-08-28T19:20:37.883684+0000","last_peered":"2024-08-28T19:20:37.883684+0000","last_clean":"2024-08-28T19:20:37.883684+0000","last_became_active":"2024-08-28T19:18:32.491897+0000","last_became_peered":"2024-08-28T19:18:32.491897+0000","last_unstale":"2024-08-28T19:20:37.883684+0000","last_undegraded":"2024-08-28T19:20:37.883684+0000","last_fullsized":"2024-08-28T19:20:37.883684+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:24.279391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870908+0000","last_change":"2024-08-28T19:18:32.485621+0000","last_active":"2024-08-28T19:20:37.870908+0000","last_peered":"2024-08-28T19:20:37.870908+0000","last_clean":"2024-08-28T19:20:37.870908+0000","last_became_active":"2024-08-28T19:18:32.485267+0000","last_became_peered":"2024-08-28T19:18:32.485267+0000","last_unstale":"2024-08-28T19:20:37.870908+0000","last_undegraded":"2024-08-28T19:20:37.870908+0000","last_fullsized":"2024-08-28T19:20:37.870908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:30:08.932135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884032+0000","last_change":"2024-08-28T19:18:32.488457+0000","last_active":"2024-08-28T19:20:37.884032+0000","last_peered":"2024-08-28T19:20:37.884032+0000","last_clean":"2024-08-28T19:20:37.884032+0000","last_became_active":"2024-08-28T19:18:32.487555+0000","last_became_peered":"2024-08-28T19:18:32.487555+0000","last_unstale":"2024-08-28T19:20:37.884032+0000","last_undegraded":"2024-08-28T19:20:37.884032+0000","last_fullsized":"2024-08-28T19:20:37.884032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:03:26.390481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":470667,"num_objects":99,"num_object_clones":0,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":1184,"num_read_kb":688,"num_write":325,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2568192,"data_stored":2198433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":68,"seq":292057776172,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":500491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":62,"seq":266287972401,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29120,"kb_used_data":2036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716032,"statfs":{"total":95995035648,"available":95965216768,"internally_reserved":0,"allocated":2084864,"data_stored":1553912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":57,"seq":244813135926,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":503845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":51,"seq":219043332155,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36708,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708444,"statfs":{"total":95995035648,"available":95957446656,"internally_reserved":0,"allocated":1396736,"data_stored":893343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":45,"seq":193273528384,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1593344,"data_stored":1096625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":40,"seq":171798691910,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":495785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888137,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1605632,"data_stored":1088251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051664,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":499327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215188,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41248,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703904,"statfs":{"total":95995035648,"available":95952797696,"internally_reserved":0,"allocated":1785856,"data_stored":1281953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1077248,"data_stored":566440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":566220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":496709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":819200,"data_stored":787422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":397128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":464917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-28T19:21:07.434 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-28T19:21:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:07 smithi028 ceph-mon[21769]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:07.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:07 smithi115 ceph-mon[22184]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:07.692 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:07.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:07 smithi016 ceph-mon[18556]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:08.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.180+0000 7fccfd124640 1 -- 172.21.15.16:0/1910995494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccf8109bb0 msgr2=0x7fccf81103e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.180+0000 7fccfd124640 1 --2- 172.21.15.16:0/1910995494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccf8109bb0 0x7fccf81103e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009ca0 tx=0x7fcce802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.180+0000 7fccfd124640 1 -- 172.21.15.16:0/1910995494 shutdown_connections 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.180+0000 7fccfd124640 1 --2- 172.21.15.16:0/1910995494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccf8109bb0 0x7fccf81103e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.180+0000 7fccfd124640 1 --2- 172.21.15.16:0/1910995494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf8109260 0x7fccf8109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.180+0000 7fccfd124640 1 --2- 172.21.15.16:0/1910995494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fccf8103920 0x7fccf8103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.180+0000 7fccfd124640 1 -- 172.21.15.16:0/1910995494 >> 172.21.15.16:0/1910995494 conn(0x7fccf80ff350 msgr2=0x7fccf81017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.180+0000 7fccfd124640 1 -- 172.21.15.16:0/1910995494 shutdown_connections 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.180+0000 7fccfd124640 1 -- 172.21.15.16:0/1910995494 wait complete. 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccfd124640 1 Processor -- start 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccfd124640 1 -- start start 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccfd124640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf8103920 0x7fccf819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccfd124640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccf8109260 0x7fccf819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccfd124640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fccf8109bb0 0x7fccf819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccfd124640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fccf8113870 con 0x7fccf8109260 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccfd124640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81136f0 con 0x7fccf8103920 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccfd124640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81139f0 con 0x7fccf8109bb0 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccf659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccf8109260 0x7fccf819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccf659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccf8109260 0x7fccf819a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45284/0 (socket says 172.21.15.16:45284) 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccf659c640 1 -- 172.21.15.16:0/3008782363 learned_addr learned my addr 172.21.15.16:0/3008782363 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.181+0000 7fccf6d9d640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf8103920 0x7fccf819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fccf759e640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fccf8109bb0 0x7fccf819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fccf6d9d640 1 -- 172.21.15.16:0/3008782363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fccf8109bb0 msgr2=0x7fccf819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fccf6d9d640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fccf8109bb0 0x7fccf819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fccf6d9d640 1 -- 172.21.15.16:0/3008782363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccf8109260 msgr2=0x7fccf819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fccf6d9d640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccf8109260 0x7fccf819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fccf6d9d640 1 -- 172.21.15.16:0/3008782363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf81a01d0 con 0x7fccf8103920 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fccf759e640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fccf8109bb0 0x7fccf819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fccf659c640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccf8109260 0x7fccf819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fccf6d9d640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf8103920 0x7fccf819a260 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fccf000d720 tx=0x7fccf000dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fcceffff640 1 -- 172.21.15.16:0/3008782363 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf0017070 con 0x7fccf8103920 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fccfd124640 1 -- 172.21.15.16:0/3008782363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf81a04c0 con 0x7fccf8103920 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fcceffff640 1 -- 172.21.15.16:0/3008782363 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccf000de30 con 0x7fccf8103920 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fcceffff640 1 -- 172.21.15.16:0/3008782363 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf00047a0 con 0x7fccf8103920 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.182+0000 7fccfd124640 1 -- 172.21.15.16:0/3008782363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf81a6c20 con 0x7fccf8103920 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.183+0000 7fccfd124640 1 -- 172.21.15.16:0/3008782363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf8104ba0 con 0x7fccf8103920 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.185+0000 7fcceffff640 1 -- 172.21.15.16:0/3008782363 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7fccf00099d0 con 0x7fccf8103920 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.186+0000 7fcceffff640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcce40b4c10 0x7fcce40b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.186+0000 7fccf659c640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcce40b4c10 0x7fcce40b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.186+0000 7fcceffff640 1 -- 172.21.15.16:0/3008782363 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fccf00e5880 con 0x7fccf8103920 2024-08-28T19:21:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.186+0000 7fccf659c640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcce40b4c10 0x7fcce40b7120 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fccf8102c10 tx=0x7fcce000b410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.189+0000 7fcceffff640 1 -- 172.21.15.16:0/3008782363 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fccf0020020 con 0x7fccf8103920 2024-08-28T19:21:08.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.372+0000 7fccfd124640 1 -- 172.21.15.16:0/3008782363 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fccf80635a0 con 0x7fcce40b4c10 2024-08-28T19:21:08.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:08 smithi016 ceph-mon[18556]: pgmap v342: 129 pgs: 129 active+clean; 1.4 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 163 KiB/s wr, 54 op/s 2024-08-28T19:21:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.378+0000 7fcceffff640 1 -- 172.21.15.16:0/3008782363 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340585 (secure 0 0 0) 0x7fccf80635a0 con 0x7fcce40b4c10 2024-08-28T19:21:08.382 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:21:08.384 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-08-28T19:21:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.381+0000 7fccfd124640 1 -- 172.21.15.16:0/3008782363 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcce40b4c10 msgr2=0x7fcce40b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.381+0000 7fccfd124640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcce40b4c10 0x7fcce40b7120 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fccf8102c10 tx=0x7fcce000b410 comp rx=0 tx=0).stop 2024-08-28T19:21:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.381+0000 7fccfd124640 1 -- 172.21.15.16:0/3008782363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf8103920 msgr2=0x7fccf819a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.381+0000 7fccfd124640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf8103920 0x7fccf819a260 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fccf000d720 tx=0x7fccf000dbe0 comp rx=0 tx=0).stop 2024-08-28T19:21:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.381+0000 7fccfd124640 1 -- 172.21.15.16:0/3008782363 shutdown_connections 2024-08-28T19:21:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.381+0000 7fccfd124640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fccf8109bb0 0x7fccf819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.381+0000 7fccfd124640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccf8109260 0x7fccf819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.381+0000 7fccfd124640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcce40b4c10 0x7fcce40b7120 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.381+0000 7fccfd124640 1 --2- 172.21.15.16:0/3008782363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf8103920 0x7fccf819a260 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.381+0000 7fccfd124640 1 -- 172.21.15.16:0/3008782363 >> 172.21.15.16:0/3008782363 conn(0x7fccf80ff350 msgr2=0x7fccf8108830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.382+0000 7fccfd124640 1 -- 172.21.15.16:0/3008782363 shutdown_connections 2024-08-28T19:21:08.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:08.382+0000 7fccfd124640 1 -- 172.21.15.16:0/3008782363 wait complete. 2024-08-28T19:21:08.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:08 smithi115 ceph-mon[22184]: pgmap v342: 129 pgs: 129 active+clean; 1.4 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 163 KiB/s wr, 54 op/s 2024-08-28T19:21:08.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:08 smithi028 ceph-mon[21769]: pgmap v342: 129 pgs: 129 active+clean; 1.4 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 163 KiB/s wr, 54 op/s 2024-08-28T19:21:09.076 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-08-28T19:21:07.498354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1454485,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1578,"num_read_kb":1031,"num_write":604,"num_write_kb":4264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":955,"ondisk_log_size":955,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":423152,"kb_used_data":15604,"kb_used_omap":18,"kb_used_meta":407277,"kb_avail":1124518672,"statfs":{"total":1151940427776,"available":1151507120128,"internally_reserved":0,"allocated":15978496,"data_stored":9936117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":417051962},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":792365,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":433,"num_read_kb":349,"num_write":219,"num_write_kb":1961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003431"},"pg_stats":[{"pgid":"2.1f","version":"84'14","reported_seq":183,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:04.821906+0000","last_change":"2024-08-28T19:19:08.633326+0000","last_active":"2024-08-28T19:21:04.821906+0000","last_peered":"2024-08-28T19:21:04.821906+0000","last_clean":"2024-08-28T19:21:04.821906+0000","last_became_active":"2024-08-28T19:19:08.623876+0000","last_became_peered":"2024-08-28T19:19:08.623876+0000","last_unstale":"2024-08-28T19:21:04.821906+0000","last_undegraded":"2024-08-28T19:21:04.821906+0000","last_fullsized":"2024-08-28T19:21:04.821906+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:19:50.632337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866980+0000","last_change":"2024-08-28T19:18:32.492604+0000","last_active":"2024-08-28T19:20:37.866980+0000","last_peered":"2024-08-28T19:20:37.866980+0000","last_clean":"2024-08-28T19:20:37.866980+0000","last_became_active":"2024-08-28T19:18:32.489427+0000","last_became_peered":"2024-08-28T19:18:32.489427+0000","last_unstale":"2024-08-28T19:20:37.866980+0000","last_undegraded":"2024-08-28T19:20:37.866980+0000","last_fullsized":"2024-08-28T19:20:37.866980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:36:14.841315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010515+0000","last_change":"2024-08-28T19:18:39.528815+0000","last_active":"2024-08-28T19:20:09.010515+0000","last_peered":"2024-08-28T19:20:09.010515+0000","last_clean":"2024-08-28T19:20:09.010515+0000","last_became_active":"2024-08-28T19:18:39.527781+0000","last_became_peered":"2024-08-28T19:18:39.527781+0000","last_unstale":"2024-08-28T19:20:09.010515+0000","last_undegraded":"2024-08-28T19:20:09.010515+0000","last_fullsized":"2024-08-28T19:20:09.010515+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:39.831426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525559+0000","last_change":"2024-08-28T19:18:31.082641+0000","last_active":"2024-08-28T19:20:09.525559+0000","last_peered":"2024-08-28T19:20:09.525559+0000","last_clean":"2024-08-28T19:20:09.525559+0000","last_became_active":"2024-08-28T19:18:31.082260+0000","last_became_peered":"2024-08-28T19:18:31.082260+0000","last_unstale":"2024-08-28T19:20:09.525559+0000","last_undegraded":"2024-08-28T19:20:09.525559+0000","last_fullsized":"2024-08-28T19:20:09.525559+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:09:55.836463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886294+0000","last_change":"2024-08-28T19:18:32.493498+0000","last_active":"2024-08-28T19:20:37.886294+0000","last_peered":"2024-08-28T19:20:37.886294+0000","last_clean":"2024-08-28T19:20:37.886294+0000","last_became_active":"2024-08-28T19:18:32.490969+0000","last_became_peered":"2024-08-28T19:18:32.490969+0000","last_unstale":"2024-08-28T19:20:37.886294+0000","last_undegraded":"2024-08-28T19:20:37.886294+0000","last_fullsized":"2024-08-28T19:20:37.886294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:34:29.537622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614607+0000","last_change":"2024-08-28T19:18:39.520922+0000","last_active":"2024-08-28T19:20:09.614607+0000","last_peered":"2024-08-28T19:20:09.614607+0000","last_clean":"2024-08-28T19:20:09.614607+0000","last_became_active":"2024-08-28T19:18:39.520362+0000","last_became_peered":"2024-08-28T19:18:39.520362+0000","last_unstale":"2024-08-28T19:20:09.614607+0000","last_undegraded":"2024-08-28T19:20:09.614607+0000","last_fullsized":"2024-08-28T19:20:09.614607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:40.658820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":53,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110426+0000","last_change":"2024-08-28T19:18:30.471870+0000","last_active":"2024-08-28T19:20:09.110426+0000","last_peered":"2024-08-28T19:20:09.110426+0000","last_clean":"2024-08-28T19:20:09.110426+0000","last_became_active":"2024-08-28T19:18:30.471438+0000","last_became_peered":"2024-08-28T19:18:30.471438+0000","last_unstale":"2024-08-28T19:20:09.110426+0000","last_undegraded":"2024-08-28T19:20:09.110426+0000","last_fullsized":"2024-08-28T19:20:09.110426+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:44:34.137395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.881211+0000","last_change":"2024-08-28T19:18:32.486485+0000","last_active":"2024-08-28T19:20:37.881211+0000","last_peered":"2024-08-28T19:20:37.881211+0000","last_clean":"2024-08-28T19:20:37.881211+0000","last_became_active":"2024-08-28T19:18:32.485888+0000","last_became_peered":"2024-08-28T19:18:32.485888+0000","last_unstale":"2024-08-28T19:20:37.881211+0000","last_undegraded":"2024-08-28T19:20:37.881211+0000","last_fullsized":"2024-08-28T19:20:37.881211+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:45:48.757318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615105+0000","last_change":"2024-08-28T19:19:08.633317+0000","last_active":"2024-08-28T19:20:09.615105+0000","last_peered":"2024-08-28T19:20:09.615105+0000","last_clean":"2024-08-28T19:20:09.615105+0000","last_became_active":"2024-08-28T19:19:08.632689+0000","last_became_peered":"2024-08-28T19:19:08.632689+0000","last_unstale":"2024-08-28T19:20:09.615105+0000","last_undegraded":"2024-08-28T19:20:09.615105+0000","last_fullsized":"2024-08-28T19:20:09.615105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:35:08.740645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110357+0000","last_change":"2024-08-28T19:18:30.473982+0000","last_active":"2024-08-28T19:20:09.110357+0000","last_peered":"2024-08-28T19:20:09.110357+0000","last_clean":"2024-08-28T19:20:09.110357+0000","last_became_active":"2024-08-28T19:18:30.473552+0000","last_became_peered":"2024-08-28T19:18:30.473552+0000","last_unstale":"2024-08-28T19:20:09.110357+0000","last_undegraded":"2024-08-28T19:20:09.110357+0000","last_fullsized":"2024-08-28T19:20:09.110357+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:22:29.435517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884392+0000","last_change":"2024-08-28T19:18:32.487881+0000","last_active":"2024-08-28T19:20:37.884392+0000","last_peered":"2024-08-28T19:20:37.884392+0000","last_clean":"2024-08-28T19:20:37.884392+0000","last_became_active":"2024-08-28T19:18:32.487039+0000","last_became_peered":"2024-08-28T19:18:32.487039+0000","last_unstale":"2024-08-28T19:20:37.884392+0000","last_undegraded":"2024-08-28T19:20:37.884392+0000","last_fullsized":"2024-08-28T19:20:37.884392+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:23:35.757811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614594+0000","last_change":"2024-08-28T19:18:39.518654+0000","last_active":"2024-08-28T19:20:09.614594+0000","last_peered":"2024-08-28T19:20:09.614594+0000","last_clean":"2024-08-28T19:20:09.614594+0000","last_became_active":"2024-08-28T19:18:39.518087+0000","last_became_peered":"2024-08-28T19:18:39.518087+0000","last_unstale":"2024-08-28T19:20:09.614594+0000","last_undegraded":"2024-08-28T19:20:09.614594+0000","last_fullsized":"2024-08-28T19:20:09.614594+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:08.284865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.887381+0000","last_change":"2024-08-28T19:18:32.491928+0000","last_active":"2024-08-28T19:20:37.887381+0000","last_peered":"2024-08-28T19:20:37.887381+0000","last_clean":"2024-08-28T19:20:37.887381+0000","last_became_active":"2024-08-28T19:18:32.491538+0000","last_became_peered":"2024-08-28T19:18:32.491538+0000","last_unstale":"2024-08-28T19:20:37.887381+0000","last_undegraded":"2024-08-28T19:20:37.887381+0000","last_fullsized":"2024-08-28T19:20:37.887381+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:39:06.077830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":24,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663446+0000","last_change":"2024-08-28T19:19:08.634752+0000","last_active":"2024-08-28T19:20:09.663446+0000","last_peered":"2024-08-28T19:20:09.663446+0000","last_clean":"2024-08-28T19:20:09.663446+0000","last_became_active":"2024-08-28T19:19:08.620784+0000","last_became_peered":"2024-08-28T19:19:08.620784+0000","last_unstale":"2024-08-28T19:20:09.663446+0000","last_undegraded":"2024-08-28T19:20:09.663446+0000","last_fullsized":"2024-08-28T19:20:09.663446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:51.640445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614895+0000","last_change":"2024-08-28T19:18:39.681413+0000","last_active":"2024-08-28T19:20:09.614895+0000","last_peered":"2024-08-28T19:20:09.614895+0000","last_clean":"2024-08-28T19:20:09.614895+0000","last_became_active":"2024-08-28T19:18:39.680793+0000","last_became_peered":"2024-08-28T19:18:39.680793+0000","last_unstale":"2024-08-28T19:20:09.614895+0000","last_undegraded":"2024-08-28T19:20:09.614895+0000","last_fullsized":"2024-08-28T19:20:09.614895+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:45:43.381019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870888+0000","last_change":"2024-08-28T19:18:32.486739+0000","last_active":"2024-08-28T19:20:37.870888+0000","last_peered":"2024-08-28T19:20:37.870888+0000","last_clean":"2024-08-28T19:20:37.870888+0000","last_became_active":"2024-08-28T19:18:32.486347+0000","last_became_peered":"2024-08-28T19:18:32.486347+0000","last_unstale":"2024-08-28T19:20:37.870888+0000","last_undegraded":"2024-08-28T19:20:37.870888+0000","last_fullsized":"2024-08-28T19:20:37.870888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:27:36.448569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010653+0000","last_change":"2024-08-28T19:18:30.467690+0000","last_active":"2024-08-28T19:20:09.010653+0000","last_peered":"2024-08-28T19:20:09.010653+0000","last_clean":"2024-08-28T19:20:09.010653+0000","last_became_active":"2024-08-28T19:18:30.467249+0000","last_became_peered":"2024-08-28T19:18:30.467249+0000","last_unstale":"2024-08-28T19:20:09.010653+0000","last_undegraded":"2024-08-28T19:20:09.010653+0000","last_fullsized":"2024-08-28T19:20:09.010653+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:37:28.876666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615167+0000","last_change":"2024-08-28T19:18:39.523889+0000","last_active":"2024-08-28T19:20:09.615167+0000","last_peered":"2024-08-28T19:20:09.615167+0000","last_clean":"2024-08-28T19:20:09.615167+0000","last_became_active":"2024-08-28T19:18:39.523384+0000","last_became_peered":"2024-08-28T19:18:39.523384+0000","last_unstale":"2024-08-28T19:20:09.615167+0000","last_undegraded":"2024-08-28T19:20:09.615167+0000","last_fullsized":"2024-08-28T19:20:09.615167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:57:32.277680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873180+0000","last_change":"2024-08-28T19:18:32.482257+0000","last_active":"2024-08-28T19:20:37.873180+0000","last_peered":"2024-08-28T19:20:37.873180+0000","last_clean":"2024-08-28T19:20:37.873180+0000","last_became_active":"2024-08-28T19:18:32.481884+0000","last_became_peered":"2024-08-28T19:18:32.481884+0000","last_unstale":"2024-08-28T19:20:37.873180+0000","last_undegraded":"2024-08-28T19:20:37.873180+0000","last_fullsized":"2024-08-28T19:20:37.873180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:56:28.754369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":47,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525435+0000","last_change":"2024-08-28T19:18:30.467307+0000","last_active":"2024-08-28T19:20:09.525435+0000","last_peered":"2024-08-28T19:20:09.525435+0000","last_clean":"2024-08-28T19:20:09.525435+0000","last_became_active":"2024-08-28T19:18:30.466942+0000","last_became_peered":"2024-08-28T19:18:30.466942+0000","last_unstale":"2024-08-28T19:20:09.525435+0000","last_undegraded":"2024-08-28T19:20:09.525435+0000","last_fullsized":"2024-08-28T19:20:09.525435+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:05:23.682110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007238+0000","last_change":"2024-08-28T19:18:39.531421+0000","last_active":"2024-08-28T19:20:09.007238+0000","last_peered":"2024-08-28T19:20:09.007238+0000","last_clean":"2024-08-28T19:20:09.007238+0000","last_became_active":"2024-08-28T19:18:39.531008+0000","last_became_peered":"2024-08-28T19:18:39.531008+0000","last_unstale":"2024-08-28T19:20:09.007238+0000","last_undegraded":"2024-08-28T19:20:09.007238+0000","last_fullsized":"2024-08-28T19:20:09.007238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:46:49.580601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868264+0000","last_change":"2024-08-28T19:18:32.485204+0000","last_active":"2024-08-28T19:20:37.868264+0000","last_peered":"2024-08-28T19:20:37.868264+0000","last_clean":"2024-08-28T19:20:37.868264+0000","last_became_active":"2024-08-28T19:18:32.484830+0000","last_became_peered":"2024-08-28T19:18:32.484830+0000","last_unstale":"2024-08-28T19:20:37.868264+0000","last_undegraded":"2024-08-28T19:20:37.868264+0000","last_fullsized":"2024-08-28T19:20:37.868264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:55:41.358718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007905+0000","last_change":"2024-08-28T19:18:31.082361+0000","last_active":"2024-08-28T19:20:09.007905+0000","last_peered":"2024-08-28T19:20:09.007905+0000","last_clean":"2024-08-28T19:20:09.007905+0000","last_became_active":"2024-08-28T19:18:31.081930+0000","last_became_peered":"2024-08-28T19:18:31.081930+0000","last_unstale":"2024-08-28T19:20:09.007905+0000","last_undegraded":"2024-08-28T19:20:09.007905+0000","last_fullsized":"2024-08-28T19:20:09.007905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:41:05.543088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615411+0000","last_change":"2024-08-28T19:18:39.682341+0000","last_active":"2024-08-28T19:20:09.615411+0000","last_peered":"2024-08-28T19:20:09.615411+0000","last_clean":"2024-08-28T19:20:09.615411+0000","last_became_active":"2024-08-28T19:18:39.681751+0000","last_became_peered":"2024-08-28T19:18:39.681751+0000","last_unstale":"2024-08-28T19:20:09.615411+0000","last_undegraded":"2024-08-28T19:20:09.615411+0000","last_fullsized":"2024-08-28T19:20:09.615411+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:51:24.371270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880906+0000","last_change":"2024-08-28T19:18:32.491069+0000","last_active":"2024-08-28T19:20:37.880906+0000","last_peered":"2024-08-28T19:20:37.880906+0000","last_clean":"2024-08-28T19:20:37.880906+0000","last_became_active":"2024-08-28T19:18:32.490688+0000","last_became_peered":"2024-08-28T19:18:32.490688+0000","last_unstale":"2024-08-28T19:20:37.880906+0000","last_undegraded":"2024-08-28T19:20:37.880906+0000","last_fullsized":"2024-08-28T19:20:37.880906+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:24:31.560519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615273+0000","last_change":"2024-08-28T19:18:30.739812+0000","last_active":"2024-08-28T19:20:09.615273+0000","last_peered":"2024-08-28T19:20:09.615273+0000","last_clean":"2024-08-28T19:20:09.615273+0000","last_became_active":"2024-08-28T19:18:30.739414+0000","last_became_peered":"2024-08-28T19:18:30.739414+0000","last_unstale":"2024-08-28T19:20:09.615273+0000","last_undegraded":"2024-08-28T19:20:09.615273+0000","last_fullsized":"2024-08-28T19:20:09.615273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:56:56.594993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010962+0000","last_change":"2024-08-28T19:18:39.528281+0000","last_active":"2024-08-28T19:20:09.010962+0000","last_peered":"2024-08-28T19:20:09.010962+0000","last_clean":"2024-08-28T19:20:09.010962+0000","last_became_active":"2024-08-28T19:18:39.527142+0000","last_became_peered":"2024-08-28T19:18:39.527142+0000","last_unstale":"2024-08-28T19:20:09.010962+0000","last_undegraded":"2024-08-28T19:20:09.010962+0000","last_fullsized":"2024-08-28T19:20:09.010962+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:44:19.123662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872571+0000","last_change":"2024-08-28T19:18:32.490430+0000","last_active":"2024-08-28T19:20:37.872571+0000","last_peered":"2024-08-28T19:20:37.872571+0000","last_clean":"2024-08-28T19:20:37.872571+0000","last_became_active":"2024-08-28T19:18:32.490020+0000","last_became_peered":"2024-08-28T19:18:32.490020+0000","last_unstale":"2024-08-28T19:20:37.872571+0000","last_undegraded":"2024-08-28T19:20:37.872571+0000","last_fullsized":"2024-08-28T19:20:37.872571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:52:31.153012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'101","reported_seq":230,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:04.697489+0000","last_change":"2024-08-28T19:18:30.467060+0000","last_active":"2024-08-28T19:21:04.697489+0000","last_peered":"2024-08-28T19:21:04.697489+0000","last_clean":"2024-08-28T19:21:04.697489+0000","last_became_active":"2024-08-28T19:18:30.466689+0000","last_became_peered":"2024-08-28T19:18:30.466689+0000","last_unstale":"2024-08-28T19:21:04.697489+0000","last_undegraded":"2024-08-28T19:21:04.697489+0000","last_fullsized":"2024-08-28T19:21:04.697489+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:18:36.508131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":302,"num_read_kb":268,"num_write":185,"num_write_kb":1222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.524970+0000","last_change":"2024-08-28T19:18:39.682874+0000","last_active":"2024-08-28T19:20:09.524970+0000","last_peered":"2024-08-28T19:20:09.524970+0000","last_clean":"2024-08-28T19:20:09.524970+0000","last_became_active":"2024-08-28T19:18:39.681500+0000","last_became_peered":"2024-08-28T19:18:39.681500+0000","last_unstale":"2024-08-28T19:20:09.524970+0000","last_undegraded":"2024-08-28T19:20:09.524970+0000","last_fullsized":"2024-08-28T19:20:09.524970+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:45:41.815506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.856367+0000","last_change":"2024-08-28T19:18:32.482047+0000","last_active":"2024-08-28T19:20:37.856367+0000","last_peered":"2024-08-28T19:20:37.856367+0000","last_clean":"2024-08-28T19:20:37.856367+0000","last_became_active":"2024-08-28T19:18:32.481640+0000","last_became_peered":"2024-08-28T19:18:32.481640+0000","last_unstale":"2024-08-28T19:20:37.856367+0000","last_undegraded":"2024-08-28T19:20:37.856367+0000","last_fullsized":"2024-08-28T19:20:37.856367+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:34.717764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110454+0000","last_change":"2024-08-28T19:18:31.083524+0000","last_active":"2024-08-28T19:20:09.110454+0000","last_peered":"2024-08-28T19:20:09.110454+0000","last_clean":"2024-08-28T19:20:09.110454+0000","last_became_active":"2024-08-28T19:18:31.083106+0000","last_became_peered":"2024-08-28T19:18:31.083106+0000","last_unstale":"2024-08-28T19:20:09.110454+0000","last_undegraded":"2024-08-28T19:20:09.110454+0000","last_fullsized":"2024-08-28T19:20:09.110454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:16.180754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663607+0000","last_change":"2024-08-28T19:18:39.531884+0000","last_active":"2024-08-28T19:20:09.663607+0000","last_peered":"2024-08-28T19:20:09.663607+0000","last_clean":"2024-08-28T19:20:09.663607+0000","last_became_active":"2024-08-28T19:18:39.531255+0000","last_became_peered":"2024-08-28T19:18:39.531255+0000","last_unstale":"2024-08-28T19:20:09.663607+0000","last_undegraded":"2024-08-28T19:20:09.663607+0000","last_fullsized":"2024-08-28T19:20:09.663607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:36:55.496594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.862241+0000","last_change":"2024-08-28T19:18:32.485180+0000","last_active":"2024-08-28T19:20:37.862241+0000","last_peered":"2024-08-28T19:20:37.862241+0000","last_clean":"2024-08-28T19:20:37.862241+0000","last_became_active":"2024-08-28T19:18:32.484804+0000","last_became_peered":"2024-08-28T19:18:32.484804+0000","last_unstale":"2024-08-28T19:20:37.862241+0000","last_undegraded":"2024-08-28T19:20:37.862241+0000","last_fullsized":"2024-08-28T19:20:37.862241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:12:40.958251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007361+0000","last_change":"2024-08-28T19:18:31.082226+0000","last_active":"2024-08-28T19:20:09.007361+0000","last_peered":"2024-08-28T19:20:09.007361+0000","last_clean":"2024-08-28T19:20:09.007361+0000","last_became_active":"2024-08-28T19:18:31.081833+0000","last_became_peered":"2024-08-28T19:18:31.081833+0000","last_unstale":"2024-08-28T19:20:09.007361+0000","last_undegraded":"2024-08-28T19:20:09.007361+0000","last_fullsized":"2024-08-28T19:20:09.007361+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:33:56.298093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109762+0000","last_change":"2024-08-28T19:18:39.681288+0000","last_active":"2024-08-28T19:20:09.109762+0000","last_peered":"2024-08-28T19:20:09.109762+0000","last_clean":"2024-08-28T19:20:09.109762+0000","last_became_active":"2024-08-28T19:18:39.680719+0000","last_became_peered":"2024-08-28T19:18:39.680719+0000","last_unstale":"2024-08-28T19:20:09.109762+0000","last_undegraded":"2024-08-28T19:20:09.109762+0000","last_fullsized":"2024-08-28T19:20:09.109762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:53:35.554181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880497+0000","last_change":"2024-08-28T19:18:32.487105+0000","last_active":"2024-08-28T19:20:37.880497+0000","last_peered":"2024-08-28T19:20:37.880497+0000","last_clean":"2024-08-28T19:20:37.880497+0000","last_became_active":"2024-08-28T19:18:32.486643+0000","last_became_peered":"2024-08-28T19:18:32.486643+0000","last_unstale":"2024-08-28T19:20:37.880497+0000","last_undegraded":"2024-08-28T19:20:37.880497+0000","last_fullsized":"2024-08-28T19:20:37.880497+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:18:43.274493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"84'7","reported_seq":163,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:03.821152+0000","last_change":"2024-08-28T19:18:30.471447+0000","last_active":"2024-08-28T19:21:03.821152+0000","last_peered":"2024-08-28T19:21:03.821152+0000","last_clean":"2024-08-28T19:21:03.821152+0000","last_became_active":"2024-08-28T19:18:30.470798+0000","last_became_peered":"2024-08-28T19:18:30.470798+0000","last_unstale":"2024-08-28T19:21:03.821152+0000","last_undegraded":"2024-08-28T19:21:03.821152+0000","last_fullsized":"2024-08-28T19:21:03.821152+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:50:13.948439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162958+0000","last_change":"2024-08-28T19:18:39.524190+0000","last_active":"2024-08-28T19:20:09.162958+0000","last_peered":"2024-08-28T19:20:09.162958+0000","last_clean":"2024-08-28T19:20:09.162958+0000","last_became_active":"2024-08-28T19:18:39.523771+0000","last_became_peered":"2024-08-28T19:18:39.523771+0000","last_unstale":"2024-08-28T19:20:09.162958+0000","last_undegraded":"2024-08-28T19:20:09.162958+0000","last_fullsized":"2024-08-28T19:20:09.162958+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:43:43.984625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.874995+0000","last_change":"2024-08-28T19:18:32.487212+0000","last_active":"2024-08-28T19:20:37.874995+0000","last_peered":"2024-08-28T19:20:37.874995+0000","last_clean":"2024-08-28T19:20:37.874995+0000","last_became_active":"2024-08-28T19:18:32.485432+0000","last_became_peered":"2024-08-28T19:18:32.485432+0000","last_unstale":"2024-08-28T19:20:37.874995+0000","last_undegraded":"2024-08-28T19:20:37.874995+0000","last_fullsized":"2024-08-28T19:20:37.874995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:52:11.672193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":159,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:03.820333+0000","last_change":"2024-08-28T19:18:31.082210+0000","last_active":"2024-08-28T19:21:03.820333+0000","last_peered":"2024-08-28T19:21:03.820333+0000","last_clean":"2024-08-28T19:21:03.820333+0000","last_became_active":"2024-08-28T19:18:31.081814+0000","last_became_peered":"2024-08-28T19:18:31.081814+0000","last_unstale":"2024-08-28T19:21:03.820333+0000","last_undegraded":"2024-08-28T19:21:03.820333+0000","last_fullsized":"2024-08-28T19:21:03.820333+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:33:32.281431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163227+0000","last_change":"2024-08-28T19:18:39.681858+0000","last_active":"2024-08-28T19:20:09.163227+0000","last_peered":"2024-08-28T19:20:09.163227+0000","last_clean":"2024-08-28T19:20:09.163227+0000","last_became_active":"2024-08-28T19:18:39.681382+0000","last_became_peered":"2024-08-28T19:18:39.681382+0000","last_unstale":"2024-08-28T19:20:09.163227+0000","last_undegraded":"2024-08-28T19:20:09.163227+0000","last_fullsized":"2024-08-28T19:20:09.163227+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:23:22.021022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.857639+0000","last_change":"2024-08-28T19:18:32.486802+0000","last_active":"2024-08-28T19:20:37.857639+0000","last_peered":"2024-08-28T19:20:37.857639+0000","last_clean":"2024-08-28T19:20:37.857639+0000","last_became_active":"2024-08-28T19:18:32.486479+0000","last_became_peered":"2024-08-28T19:18:32.486479+0000","last_unstale":"2024-08-28T19:20:37.857639+0000","last_undegraded":"2024-08-28T19:20:37.857639+0000","last_fullsized":"2024-08-28T19:20:37.857639+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:17:17.654757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529084+0000","last_change":"2024-08-28T19:18:30.470447+0000","last_active":"2024-08-28T19:20:09.529084+0000","last_peered":"2024-08-28T19:20:09.529084+0000","last_clean":"2024-08-28T19:20:09.529084+0000","last_became_active":"2024-08-28T19:18:30.469862+0000","last_became_peered":"2024-08-28T19:18:30.469862+0000","last_unstale":"2024-08-28T19:20:09.529084+0000","last_undegraded":"2024-08-28T19:20:09.529084+0000","last_fullsized":"2024-08-28T19:20:09.529084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:27:46.660492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009270+0000","last_change":"2024-08-28T19:19:08.624635+0000","last_active":"2024-08-28T19:20:09.009270+0000","last_peered":"2024-08-28T19:20:09.009270+0000","last_clean":"2024-08-28T19:20:09.009270+0000","last_became_active":"2024-08-28T19:19:08.622845+0000","last_became_peered":"2024-08-28T19:19:08.622845+0000","last_unstale":"2024-08-28T19:20:09.009270+0000","last_undegraded":"2024-08-28T19:20:09.009270+0000","last_fullsized":"2024-08-28T19:20:09.009270+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:56:40.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865852+0000","last_change":"2024-08-28T19:18:32.487947+0000","last_active":"2024-08-28T19:20:37.865852+0000","last_peered":"2024-08-28T19:20:37.865852+0000","last_clean":"2024-08-28T19:20:37.865852+0000","last_became_active":"2024-08-28T19:18:32.487592+0000","last_became_peered":"2024-08-28T19:18:32.487592+0000","last_unstale":"2024-08-28T19:20:37.865852+0000","last_undegraded":"2024-08-28T19:20:37.865852+0000","last_fullsized":"2024-08-28T19:20:37.865852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:30:29.550644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615576+0000","last_change":"2024-08-28T19:18:30.465044+0000","last_active":"2024-08-28T19:20:09.615576+0000","last_peered":"2024-08-28T19:20:09.615576+0000","last_clean":"2024-08-28T19:20:09.615576+0000","last_became_active":"2024-08-28T19:18:30.464580+0000","last_became_peered":"2024-08-28T19:18:30.464580+0000","last_unstale":"2024-08-28T19:20:09.615576+0000","last_undegraded":"2024-08-28T19:20:09.615576+0000","last_fullsized":"2024-08-28T19:20:09.615576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:21:17.223311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":114,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.528964+0000","last_change":"2024-08-28T19:17:17.063316+0000","last_active":"2024-08-28T19:20:09.528964+0000","last_peered":"2024-08-28T19:20:09.528964+0000","last_clean":"2024-08-28T19:20:09.528964+0000","last_became_active":"2024-08-28T19:17:17.057091+0000","last_became_peered":"2024-08-28T19:17:17.057091+0000","last_unstale":"2024-08-28T19:20:09.528964+0000","last_undegraded":"2024-08-28T19:20:09.528964+0000","last_fullsized":"2024-08-28T19:20:09.528964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_clean_scrub_stamp":"2024-08-28T19:13:56.815467+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:35:51.499703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"82'1","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529202+0000","last_change":"2024-08-28T19:19:08.621402+0000","last_active":"2024-08-28T19:20:09.529202+0000","last_peered":"2024-08-28T19:20:09.529202+0000","last_clean":"2024-08-28T19:20:09.529202+0000","last_became_active":"2024-08-28T19:19:08.620779+0000","last_became_peered":"2024-08-28T19:19:08.620779+0000","last_unstale":"2024-08-28T19:20:09.529202+0000","last_undegraded":"2024-08-28T19:20:09.529202+0000","last_fullsized":"2024-08-28T19:20:09.529202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:39.391414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886997+0000","last_change":"2024-08-28T19:18:32.490066+0000","last_active":"2024-08-28T19:20:37.886997+0000","last_peered":"2024-08-28T19:20:37.886997+0000","last_clean":"2024-08-28T19:20:37.886997+0000","last_became_active":"2024-08-28T19:18:32.489560+0000","last_became_peered":"2024-08-28T19:18:32.489560+0000","last_unstale":"2024-08-28T19:20:37.886997+0000","last_undegraded":"2024-08-28T19:20:37.886997+0000","last_fullsized":"2024-08-28T19:20:37.886997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:22:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'5","reported_seq":160,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:03.865971+0000","last_change":"2024-08-28T19:18:31.083078+0000","last_active":"2024-08-28T19:21:03.865971+0000","last_peered":"2024-08-28T19:21:03.865971+0000","last_clean":"2024-08-28T19:21:03.865971+0000","last_became_active":"2024-08-28T19:18:31.082575+0000","last_became_peered":"2024-08-28T19:18:31.082575+0000","last_unstale":"2024-08-28T19:21:03.865971+0000","last_undegraded":"2024-08-28T19:21:03.865971+0000","last_fullsized":"2024-08-28T19:21:03.865971+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007862+0000","last_change":"2024-08-28T19:18:39.519814+0000","last_active":"2024-08-28T19:20:09.007862+0000","last_peered":"2024-08-28T19:20:09.007862+0000","last_clean":"2024-08-28T19:20:09.007862+0000","last_became_active":"2024-08-28T19:18:39.519258+0000","last_became_peered":"2024-08-28T19:18:39.519258+0000","last_unstale":"2024-08-28T19:20:09.007862+0000","last_undegraded":"2024-08-28T19:20:09.007862+0000","last_fullsized":"2024-08-28T19:20:09.007862+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:03:04.051376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871177+0000","last_change":"2024-08-28T19:18:32.495426+0000","last_active":"2024-08-28T19:20:37.871177+0000","last_peered":"2024-08-28T19:20:37.871177+0000","last_clean":"2024-08-28T19:20:37.871177+0000","last_became_active":"2024-08-28T19:18:32.495075+0000","last_became_peered":"2024-08-28T19:18:32.495075+0000","last_unstale":"2024-08-28T19:20:37.871177+0000","last_undegraded":"2024-08-28T19:20:37.871177+0000","last_fullsized":"2024-08-28T19:20:37.871177+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:40:22.456666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'54","reported_seq":111,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:03.560174+0000","last_change":"2024-08-28T19:18:30.472708+0000","last_active":"2024-08-28T19:21:03.560174+0000","last_peered":"2024-08-28T19:21:03.560174+0000","last_clean":"2024-08-28T19:21:03.560174+0000","last_became_active":"2024-08-28T19:18:30.472320+0000","last_became_peered":"2024-08-28T19:18:30.472320+0000","last_unstale":"2024-08-28T19:21:03.560174+0000","last_undegraded":"2024-08-28T19:21:03.560174+0000","last_fullsized":"2024-08-28T19:21:03.560174+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:57:12.055626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":19,"num_write":63,"num_write_kb":739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010493+0000","last_change":"2024-08-28T19:18:39.519398+0000","last_active":"2024-08-28T19:20:09.010493+0000","last_peered":"2024-08-28T19:20:09.010493+0000","last_clean":"2024-08-28T19:20:09.010493+0000","last_became_active":"2024-08-28T19:18:39.518997+0000","last_became_peered":"2024-08-28T19:18:39.518997+0000","last_unstale":"2024-08-28T19:20:09.010493+0000","last_undegraded":"2024-08-28T19:20:09.010493+0000","last_fullsized":"2024-08-28T19:20:09.010493+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:33.431608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.879604+0000","last_change":"2024-08-28T19:18:32.489439+0000","last_active":"2024-08-28T19:20:37.879604+0000","last_peered":"2024-08-28T19:20:37.879604+0000","last_clean":"2024-08-28T19:20:37.879604+0000","last_became_active":"2024-08-28T19:18:32.489054+0000","last_became_peered":"2024-08-28T19:18:32.489054+0000","last_unstale":"2024-08-28T19:20:37.879604+0000","last_undegraded":"2024-08-28T19:20:37.879604+0000","last_fullsized":"2024-08-28T19:20:37.879604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:31:32.647384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163358+0000","last_change":"2024-08-28T19:18:30.464993+0000","last_active":"2024-08-28T19:20:09.163358+0000","last_peered":"2024-08-28T19:20:09.163358+0000","last_clean":"2024-08-28T19:20:09.163358+0000","last_became_active":"2024-08-28T19:18:30.463951+0000","last_became_peered":"2024-08-28T19:18:30.463951+0000","last_unstale":"2024-08-28T19:20:09.163358+0000","last_undegraded":"2024-08-28T19:20:09.163358+0000","last_fullsized":"2024-08-28T19:20:09.163358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:06.780058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008234+0000","last_change":"2024-08-28T19:18:39.533061+0000","last_active":"2024-08-28T19:20:09.008234+0000","last_peered":"2024-08-28T19:20:09.008234+0000","last_clean":"2024-08-28T19:20:09.008234+0000","last_became_active":"2024-08-28T19:18:39.532607+0000","last_became_peered":"2024-08-28T19:18:39.532607+0000","last_unstale":"2024-08-28T19:20:09.008234+0000","last_undegraded":"2024-08-28T19:20:09.008234+0000","last_fullsized":"2024-08-28T19:20:09.008234+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:00:57.997665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875549+0000","last_change":"2024-08-28T19:18:32.487937+0000","last_active":"2024-08-28T19:20:37.875549+0000","last_peered":"2024-08-28T19:20:37.875549+0000","last_clean":"2024-08-28T19:20:37.875549+0000","last_became_active":"2024-08-28T19:18:32.487417+0000","last_became_peered":"2024-08-28T19:18:32.487417+0000","last_unstale":"2024-08-28T19:20:37.875549+0000","last_undegraded":"2024-08-28T19:20:37.875549+0000","last_fullsized":"2024-08-28T19:20:37.875549+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:42.497364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110517+0000","last_change":"2024-08-28T19:18:30.470742+0000","last_active":"2024-08-28T19:20:09.110517+0000","last_peered":"2024-08-28T19:20:09.110517+0000","last_clean":"2024-08-28T19:20:09.110517+0000","last_became_active":"2024-08-28T19:18:30.470135+0000","last_became_peered":"2024-08-28T19:18:30.470135+0000","last_unstale":"2024-08-28T19:20:09.110517+0000","last_undegraded":"2024-08-28T19:20:09.110517+0000","last_fullsized":"2024-08-28T19:20:09.110517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:49:39.581119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008429+0000","last_change":"2024-08-28T19:18:39.524052+0000","last_active":"2024-08-28T19:20:09.008429+0000","last_peered":"2024-08-28T19:20:09.008429+0000","last_clean":"2024-08-28T19:20:09.008429+0000","last_became_active":"2024-08-28T19:18:39.523601+0000","last_became_peered":"2024-08-28T19:18:39.523601+0000","last_unstale":"2024-08-28T19:20:09.008429+0000","last_undegraded":"2024-08-28T19:20:09.008429+0000","last_fullsized":"2024-08-28T19:20:09.008429+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:54:53.302424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883838+0000","last_change":"2024-08-28T19:18:32.491151+0000","last_active":"2024-08-28T19:20:37.883838+0000","last_peered":"2024-08-28T19:20:37.883838+0000","last_clean":"2024-08-28T19:20:37.883838+0000","last_became_active":"2024-08-28T19:18:32.490798+0000","last_became_peered":"2024-08-28T19:18:32.490798+0000","last_unstale":"2024-08-28T19:20:37.883838+0000","last_undegraded":"2024-08-28T19:20:37.883838+0000","last_fullsized":"2024-08-28T19:20:37.883838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:41:00.771019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109743+0000","last_change":"2024-08-28T19:18:31.084029+0000","last_active":"2024-08-28T19:20:09.109743+0000","last_peered":"2024-08-28T19:20:09.109743+0000","last_clean":"2024-08-28T19:20:09.109743+0000","last_became_active":"2024-08-28T19:18:31.083507+0000","last_became_peered":"2024-08-28T19:18:31.083507+0000","last_unstale":"2024-08-28T19:20:09.109743+0000","last_undegraded":"2024-08-28T19:20:09.109743+0000","last_fullsized":"2024-08-28T19:20:09.109743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:15:36.451702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163318+0000","last_change":"2024-08-28T19:18:39.527249+0000","last_active":"2024-08-28T19:20:09.163318+0000","last_peered":"2024-08-28T19:20:09.163318+0000","last_clean":"2024-08-28T19:20:09.163318+0000","last_became_active":"2024-08-28T19:18:39.526834+0000","last_became_peered":"2024-08-28T19:18:39.526834+0000","last_unstale":"2024-08-28T19:20:09.163318+0000","last_undegraded":"2024-08-28T19:20:09.163318+0000","last_fullsized":"2024-08-28T19:20:09.163318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:48:02.294738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.876604+0000","last_change":"2024-08-28T19:18:32.497337+0000","last_active":"2024-08-28T19:20:37.876604+0000","last_peered":"2024-08-28T19:20:37.876604+0000","last_clean":"2024-08-28T19:20:37.876604+0000","last_became_active":"2024-08-28T19:18:32.496929+0000","last_became_peered":"2024-08-28T19:18:32.496929+0000","last_unstale":"2024-08-28T19:20:37.876604+0000","last_undegraded":"2024-08-28T19:20:37.876604+0000","last_fullsized":"2024-08-28T19:20:37.876604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:14.748765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":159,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:02.819590+0000","last_change":"2024-08-28T19:18:30.471353+0000","last_active":"2024-08-28T19:21:02.819590+0000","last_peered":"2024-08-28T19:21:02.819590+0000","last_clean":"2024-08-28T19:21:02.819590+0000","last_became_active":"2024-08-28T19:18:30.470283+0000","last_became_peered":"2024-08-28T19:18:30.470283+0000","last_unstale":"2024-08-28T19:21:02.819590+0000","last_undegraded":"2024-08-28T19:21:02.819590+0000","last_fullsized":"2024-08-28T19:21:02.819590+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:09:46.444864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007301+0000","last_change":"2024-08-28T19:18:39.519226+0000","last_active":"2024-08-28T19:20:09.007301+0000","last_peered":"2024-08-28T19:20:09.007301+0000","last_clean":"2024-08-28T19:20:09.007301+0000","last_became_active":"2024-08-28T19:18:39.518738+0000","last_became_peered":"2024-08-28T19:18:39.518738+0000","last_unstale":"2024-08-28T19:20:09.007301+0000","last_undegraded":"2024-08-28T19:20:09.007301+0000","last_fullsized":"2024-08-28T19:20:09.007301+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:39:35.114702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.877692+0000","last_change":"2024-08-28T19:18:32.493060+0000","last_active":"2024-08-28T19:20:37.877692+0000","last_peered":"2024-08-28T19:20:37.877692+0000","last_clean":"2024-08-28T19:20:37.877692+0000","last_became_active":"2024-08-28T19:18:32.492590+0000","last_became_peered":"2024-08-28T19:18:32.492590+0000","last_unstale":"2024-08-28T19:20:37.877692+0000","last_undegraded":"2024-08-28T19:20:37.877692+0000","last_fullsized":"2024-08-28T19:20:37.877692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:02:02.165301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008761+0000","last_change":"2024-08-28T19:18:30.470674+0000","last_active":"2024-08-28T19:20:09.008761+0000","last_peered":"2024-08-28T19:20:09.008761+0000","last_clean":"2024-08-28T19:20:09.008761+0000","last_became_active":"2024-08-28T19:18:30.470275+0000","last_became_peered":"2024-08-28T19:18:30.470275+0000","last_unstale":"2024-08-28T19:20:09.008761+0000","last_undegraded":"2024-08-28T19:20:09.008761+0000","last_fullsized":"2024-08-28T19:20:09.008761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:01:48.547315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663581+0000","last_change":"2024-08-28T19:19:08.624194+0000","last_active":"2024-08-28T19:20:09.663581+0000","last_peered":"2024-08-28T19:20:09.663581+0000","last_clean":"2024-08-28T19:20:09.663581+0000","last_became_active":"2024-08-28T19:19:08.623681+0000","last_became_peered":"2024-08-28T19:19:08.623681+0000","last_unstale":"2024-08-28T19:20:09.663581+0000","last_undegraded":"2024-08-28T19:20:09.663581+0000","last_fullsized":"2024-08-28T19:20:09.663581+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:21:11.099931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872115+0000","last_change":"2024-08-28T19:18:32.488210+0000","last_active":"2024-08-28T19:20:37.872115+0000","last_peered":"2024-08-28T19:20:37.872115+0000","last_clean":"2024-08-28T19:20:37.872115+0000","last_became_active":"2024-08-28T19:18:32.487853+0000","last_became_peered":"2024-08-28T19:18:32.487853+0000","last_unstale":"2024-08-28T19:20:37.872115+0000","last_undegraded":"2024-08-28T19:20:37.872115+0000","last_fullsized":"2024-08-28T19:20:37.872115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:20:51.808958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011302+0000","last_change":"2024-08-28T19:18:30.465306+0000","last_active":"2024-08-28T19:20:09.011302+0000","last_peered":"2024-08-28T19:20:09.011302+0000","last_clean":"2024-08-28T19:20:09.011302+0000","last_became_active":"2024-08-28T19:18:30.464850+0000","last_became_peered":"2024-08-28T19:18:30.464850+0000","last_unstale":"2024-08-28T19:20:09.011302+0000","last_undegraded":"2024-08-28T19:20:09.011302+0000","last_fullsized":"2024-08-28T19:20:09.011302+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:24:26.628999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008722+0000","last_change":"2024-08-28T19:18:39.524005+0000","last_active":"2024-08-28T19:20:09.008722+0000","last_peered":"2024-08-28T19:20:09.008722+0000","last_clean":"2024-08-28T19:20:09.008722+0000","last_became_active":"2024-08-28T19:18:39.523594+0000","last_became_peered":"2024-08-28T19:18:39.523594+0000","last_unstale":"2024-08-28T19:20:09.008722+0000","last_undegraded":"2024-08-28T19:20:09.008722+0000","last_fullsized":"2024-08-28T19:20:09.008722+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:35:32.484399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872816+0000","last_change":"2024-08-28T19:18:32.489088+0000","last_active":"2024-08-28T19:20:37.872816+0000","last_peered":"2024-08-28T19:20:37.872816+0000","last_clean":"2024-08-28T19:20:37.872816+0000","last_became_active":"2024-08-28T19:18:32.488717+0000","last_became_peered":"2024-08-28T19:18:32.488717+0000","last_unstale":"2024-08-28T19:20:37.872816+0000","last_undegraded":"2024-08-28T19:20:37.872816+0000","last_fullsized":"2024-08-28T19:20:37.872816+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:25.468919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"84'41","reported_seq":320,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:04.822762+0000","last_change":"2024-08-28T19:18:30.468674+0000","last_active":"2024-08-28T19:21:04.822762+0000","last_peered":"2024-08-28T19:21:04.822762+0000","last_clean":"2024-08-28T19:21:04.822762+0000","last_became_active":"2024-08-28T19:18:30.468300+0000","last_became_peered":"2024-08-28T19:18:30.468300+0000","last_unstale":"2024-08-28T19:21:04.822762+0000","last_undegraded":"2024-08-28T19:21:04.822762+0000","last_fullsized":"2024-08-28T19:21:04.822762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:12:06.237498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69405,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":239,"num_read_kb":188,"num_write":43,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007140+0000","last_change":"2024-08-28T19:18:39.526131+0000","last_active":"2024-08-28T19:20:09.007140+0000","last_peered":"2024-08-28T19:20:09.007140+0000","last_clean":"2024-08-28T19:20:09.007140+0000","last_became_active":"2024-08-28T19:18:39.525708+0000","last_became_peered":"2024-08-28T19:18:39.525708+0000","last_unstale":"2024-08-28T19:20:09.007140+0000","last_undegraded":"2024-08-28T19:20:09.007140+0000","last_fullsized":"2024-08-28T19:20:09.007140+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:21:52.156487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.885078+0000","last_change":"2024-08-28T19:18:32.487634+0000","last_active":"2024-08-28T19:20:37.885078+0000","last_peered":"2024-08-28T19:20:37.885078+0000","last_clean":"2024-08-28T19:20:37.885078+0000","last_became_active":"2024-08-28T19:18:32.486702+0000","last_became_peered":"2024-08-28T19:18:32.486702+0000","last_unstale":"2024-08-28T19:20:37.885078+0000","last_undegraded":"2024-08-28T19:20:37.885078+0000","last_fullsized":"2024-08-28T19:20:37.885078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:08:12.606905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009190+0000","last_change":"2024-08-28T19:18:31.083494+0000","last_active":"2024-08-28T19:20:09.009190+0000","last_peered":"2024-08-28T19:20:09.009190+0000","last_clean":"2024-08-28T19:20:09.009190+0000","last_became_active":"2024-08-28T19:18:31.083086+0000","last_became_peered":"2024-08-28T19:18:31.083086+0000","last_unstale":"2024-08-28T19:20:09.009190+0000","last_undegraded":"2024-08-28T19:20:09.009190+0000","last_fullsized":"2024-08-28T19:20:09.009190+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:14:54.388364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615438+0000","last_change":"2024-08-28T19:18:39.532155+0000","last_active":"2024-08-28T19:20:09.615438+0000","last_peered":"2024-08-28T19:20:09.615438+0000","last_clean":"2024-08-28T19:20:09.615438+0000","last_became_active":"2024-08-28T19:18:39.531565+0000","last_became_peered":"2024-08-28T19:18:39.531565+0000","last_unstale":"2024-08-28T19:20:09.615438+0000","last_undegraded":"2024-08-28T19:20:09.615438+0000","last_fullsized":"2024-08-28T19:20:09.615438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:29:48.824622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.881019+0000","last_change":"2024-08-28T19:18:32.496296+0000","last_active":"2024-08-28T19:20:37.881019+0000","last_peered":"2024-08-28T19:20:37.881019+0000","last_clean":"2024-08-28T19:20:37.881019+0000","last_became_active":"2024-08-28T19:18:32.495680+0000","last_became_peered":"2024-08-28T19:18:32.495680+0000","last_unstale":"2024-08-28T19:20:37.881019+0000","last_undegraded":"2024-08-28T19:20:37.881019+0000","last_fullsized":"2024-08-28T19:20:37.881019+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:27:34.544529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529311+0000","last_change":"2024-08-28T19:19:08.631758+0000","last_active":"2024-08-28T19:20:09.529311+0000","last_peered":"2024-08-28T19:20:09.529311+0000","last_clean":"2024-08-28T19:20:09.529311+0000","last_became_active":"2024-08-28T19:19:08.631202+0000","last_became_peered":"2024-08-28T19:19:08.631202+0000","last_unstale":"2024-08-28T19:20:09.529311+0000","last_undegraded":"2024-08-28T19:20:09.529311+0000","last_fullsized":"2024-08-28T19:20:09.529311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:27:43.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.006784+0000","last_change":"2024-08-28T19:18:39.531157+0000","last_active":"2024-08-28T19:20:09.006784+0000","last_peered":"2024-08-28T19:20:09.006784+0000","last_clean":"2024-08-28T19:20:09.006784+0000","last_became_active":"2024-08-28T19:18:39.530675+0000","last_became_peered":"2024-08-28T19:18:39.530675+0000","last_unstale":"2024-08-28T19:20:09.006784+0000","last_undegraded":"2024-08-28T19:20:09.006784+0000","last_fullsized":"2024-08-28T19:20:09.006784+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:38:40.414854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883692+0000","last_change":"2024-08-28T19:18:32.482182+0000","last_active":"2024-08-28T19:20:37.883692+0000","last_peered":"2024-08-28T19:20:37.883692+0000","last_clean":"2024-08-28T19:20:37.883692+0000","last_became_active":"2024-08-28T19:18:32.481689+0000","last_became_peered":"2024-08-28T19:18:32.481689+0000","last_unstale":"2024-08-28T19:20:37.883692+0000","last_undegraded":"2024-08-28T19:20:37.883692+0000","last_fullsized":"2024-08-28T19:20:37.883692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:13:29.278469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'5","reported_seq":271,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:04.822781+0000","last_change":"2024-08-28T19:18:30.475244+0000","last_active":"2024-08-28T19:21:04.822781+0000","last_peered":"2024-08-28T19:21:04.822781+0000","last_clean":"2024-08-28T19:21:04.822781+0000","last_became_active":"2024-08-28T19:18:30.474861+0000","last_became_peered":"2024-08-28T19:18:30.474861+0000","last_unstale":"2024-08-28T19:21:04.822781+0000","last_undegraded":"2024-08-28T19:21:04.822781+0000","last_fullsized":"2024-08-28T19:21:04.822781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:27:09.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525108+0000","last_change":"2024-08-28T19:18:39.520022+0000","last_active":"2024-08-28T19:20:09.525108+0000","last_peered":"2024-08-28T19:20:09.525108+0000","last_clean":"2024-08-28T19:20:09.525108+0000","last_became_active":"2024-08-28T19:18:39.519584+0000","last_became_peered":"2024-08-28T19:18:39.519584+0000","last_unstale":"2024-08-28T19:20:09.525108+0000","last_undegraded":"2024-08-28T19:20:09.525108+0000","last_fullsized":"2024-08-28T19:20:09.525108+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:27:15.851303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010363+0000","last_change":"2024-08-28T19:18:39.522607+0000","last_active":"2024-08-28T19:20:09.010363+0000","last_peered":"2024-08-28T19:20:09.010363+0000","last_clean":"2024-08-28T19:20:09.010363+0000","last_became_active":"2024-08-28T19:18:39.522204+0000","last_became_peered":"2024-08-28T19:18:39.522204+0000","last_unstale":"2024-08-28T19:20:09.010363+0000","last_undegraded":"2024-08-28T19:20:09.010363+0000","last_fullsized":"2024-08-28T19:20:09.010363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:02:08.865294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865617+0000","last_change":"2024-08-28T19:18:32.485577+0000","last_active":"2024-08-28T19:20:37.865617+0000","last_peered":"2024-08-28T19:20:37.865617+0000","last_clean":"2024-08-28T19:20:37.865617+0000","last_became_active":"2024-08-28T19:18:32.485243+0000","last_became_peered":"2024-08-28T19:18:32.485243+0000","last_unstale":"2024-08-28T19:20:37.865617+0000","last_undegraded":"2024-08-28T19:20:37.865617+0000","last_fullsized":"2024-08-28T19:20:37.865617+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:05:53.019365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008799+0000","last_change":"2024-08-28T19:18:30.470670+0000","last_active":"2024-08-28T19:20:09.008799+0000","last_peered":"2024-08-28T19:20:09.008799+0000","last_clean":"2024-08-28T19:20:09.008799+0000","last_became_active":"2024-08-28T19:18:30.470264+0000","last_became_peered":"2024-08-28T19:18:30.470264+0000","last_unstale":"2024-08-28T19:20:09.008799+0000","last_undegraded":"2024-08-28T19:20:09.008799+0000","last_fullsized":"2024-08-28T19:20:09.008799+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:35.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663847+0000","last_change":"2024-08-28T19:18:39.530948+0000","last_active":"2024-08-28T19:20:09.663847+0000","last_peered":"2024-08-28T19:20:09.663847+0000","last_clean":"2024-08-28T19:20:09.663847+0000","last_became_active":"2024-08-28T19:18:39.530473+0000","last_became_peered":"2024-08-28T19:18:39.530473+0000","last_unstale":"2024-08-28T19:20:09.663847+0000","last_undegraded":"2024-08-28T19:20:09.663847+0000","last_fullsized":"2024-08-28T19:20:09.663847+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:21:57.504582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.869892+0000","last_change":"2024-08-28T19:18:32.486027+0000","last_active":"2024-08-28T19:20:37.869892+0000","last_peered":"2024-08-28T19:20:37.869892+0000","last_clean":"2024-08-28T19:20:37.869892+0000","last_became_active":"2024-08-28T19:18:32.483273+0000","last_became_peered":"2024-08-28T19:18:32.483273+0000","last_unstale":"2024-08-28T19:20:37.869892+0000","last_undegraded":"2024-08-28T19:20:37.869892+0000","last_fullsized":"2024-08-28T19:20:37.869892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:47:37.680695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162978+0000","last_change":"2024-08-28T19:18:30.464811+0000","last_active":"2024-08-28T19:20:09.162978+0000","last_peered":"2024-08-28T19:20:09.162978+0000","last_clean":"2024-08-28T19:20:09.162978+0000","last_became_active":"2024-08-28T19:18:30.463986+0000","last_became_peered":"2024-08-28T19:18:30.463986+0000","last_unstale":"2024-08-28T19:20:09.162978+0000","last_undegraded":"2024-08-28T19:20:09.162978+0000","last_fullsized":"2024-08-28T19:20:09.162978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:43:46.582993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663619+0000","last_change":"2024-08-28T19:19:08.631382+0000","last_active":"2024-08-28T19:20:09.663619+0000","last_peered":"2024-08-28T19:20:09.663619+0000","last_clean":"2024-08-28T19:20:09.663619+0000","last_became_active":"2024-08-28T19:19:08.630869+0000","last_became_peered":"2024-08-28T19:19:08.630869+0000","last_unstale":"2024-08-28T19:20:09.663619+0000","last_undegraded":"2024-08-28T19:20:09.663619+0000","last_fullsized":"2024-08-28T19:20:09.663619+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:32:54.942408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875456+0000","last_change":"2024-08-28T19:18:32.484120+0000","last_active":"2024-08-28T19:20:37.875456+0000","last_peered":"2024-08-28T19:20:37.875456+0000","last_clean":"2024-08-28T19:20:37.875456+0000","last_became_active":"2024-08-28T19:18:32.483013+0000","last_became_peered":"2024-08-28T19:18:32.483013+0000","last_unstale":"2024-08-28T19:20:37.875456+0000","last_undegraded":"2024-08-28T19:20:37.875456+0000","last_fullsized":"2024-08-28T19:20:37.875456+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:59:41.195873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010737+0000","last_change":"2024-08-28T19:18:39.531046+0000","last_active":"2024-08-28T19:20:09.010737+0000","last_peered":"2024-08-28T19:20:09.010737+0000","last_clean":"2024-08-28T19:20:09.010737+0000","last_became_active":"2024-08-28T19:18:39.530407+0000","last_became_peered":"2024-08-28T19:18:39.530407+0000","last_unstale":"2024-08-28T19:20:09.010737+0000","last_undegraded":"2024-08-28T19:20:09.010737+0000","last_fullsized":"2024-08-28T19:20:09.010737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:47:15.173021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'7","reported_seq":162,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:02.820158+0000","last_change":"2024-08-28T19:18:30.471268+0000","last_active":"2024-08-28T19:21:02.820158+0000","last_peered":"2024-08-28T19:21:02.820158+0000","last_clean":"2024-08-28T19:21:02.820158+0000","last_became_active":"2024-08-28T19:18:30.470845+0000","last_became_peered":"2024-08-28T19:18:30.470845+0000","last_unstale":"2024-08-28T19:21:02.820158+0000","last_undegraded":"2024-08-28T19:21:02.820158+0000","last_fullsized":"2024-08-28T19:21:02.820158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:18:18.086205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.860520+0000","last_change":"2024-08-28T19:18:32.479728+0000","last_active":"2024-08-28T19:20:37.860520+0000","last_peered":"2024-08-28T19:20:37.860520+0000","last_clean":"2024-08-28T19:20:37.860520+0000","last_became_active":"2024-08-28T19:18:32.478623+0000","last_became_peered":"2024-08-28T19:18:32.478623+0000","last_unstale":"2024-08-28T19:20:37.860520+0000","last_undegraded":"2024-08-28T19:20:37.860520+0000","last_fullsized":"2024-08-28T19:20:37.860520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:14:49.914808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011224+0000","last_change":"2024-08-28T19:18:39.531308+0000","last_active":"2024-08-28T19:20:09.011224+0000","last_peered":"2024-08-28T19:20:09.011224+0000","last_clean":"2024-08-28T19:20:09.011224+0000","last_became_active":"2024-08-28T19:18:39.530403+0000","last_became_peered":"2024-08-28T19:18:39.530403+0000","last_unstale":"2024-08-28T19:20:09.011224+0000","last_undegraded":"2024-08-28T19:20:09.011224+0000","last_fullsized":"2024-08-28T19:20:09.011224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:33.139668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873003+0000","last_change":"2024-08-28T19:18:32.494621+0000","last_active":"2024-08-28T19:20:37.873003+0000","last_peered":"2024-08-28T19:20:37.873003+0000","last_clean":"2024-08-28T19:20:37.873003+0000","last_became_active":"2024-08-28T19:18:32.494217+0000","last_became_peered":"2024-08-28T19:18:32.494217+0000","last_unstale":"2024-08-28T19:20:37.873003+0000","last_undegraded":"2024-08-28T19:20:37.873003+0000","last_fullsized":"2024-08-28T19:20:37.873003+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:54:15.419470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"84'12","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872043+0000","last_change":"2024-08-28T19:20:09.616552+0000","last_active":"2024-08-28T19:20:37.872043+0000","last_peered":"2024-08-28T19:20:37.872043+0000","last_clean":"2024-08-28T19:20:37.872043+0000","last_became_active":"2024-08-28T19:20:09.616159+0000","last_became_peered":"2024-08-28T19:20:09.616159+0000","last_unstale":"2024-08-28T19:20:37.872043+0000","last_undegraded":"2024-08-28T19:20:37.872043+0000","last_fullsized":"2024-08-28T19:20:37.872043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:04:10.354703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"84'8","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866959+0000","last_change":"2024-08-28T19:20:09.012824+0000","last_active":"2024-08-28T19:20:37.866959+0000","last_peered":"2024-08-28T19:20:37.866959+0000","last_clean":"2024-08-28T19:20:37.866959+0000","last_became_active":"2024-08-28T19:20:09.012490+0000","last_became_peered":"2024-08-28T19:20:09.012490+0000","last_unstale":"2024-08-28T19:20:37.866959+0000","last_undegraded":"2024-08-28T19:20:37.866959+0000","last_fullsized":"2024-08-28T19:20:37.866959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:49:02.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871666+0000","last_change":"2024-08-28T19:20:09.164523+0000","last_active":"2024-08-28T19:20:37.871666+0000","last_peered":"2024-08-28T19:20:37.871666+0000","last_clean":"2024-08-28T19:20:37.871666+0000","last_became_active":"2024-08-28T19:20:09.163825+0000","last_became_peered":"2024-08-28T19:20:09.163825+0000","last_unstale":"2024-08-28T19:20:37.871666+0000","last_undegraded":"2024-08-28T19:20:37.871666+0000","last_fullsized":"2024-08-28T19:20:37.871666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:42:48.141103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868174+0000","last_change":"2024-08-28T19:18:32.485446+0000","last_active":"2024-08-28T19:20:37.868174+0000","last_peered":"2024-08-28T19:20:37.868174+0000","last_clean":"2024-08-28T19:20:37.868174+0000","last_became_active":"2024-08-28T19:18:32.484197+0000","last_became_peered":"2024-08-28T19:18:32.484197+0000","last_unstale":"2024-08-28T19:20:37.868174+0000","last_undegraded":"2024-08-28T19:20:37.868174+0000","last_fullsized":"2024-08-28T19:20:37.868174+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:47:50.527915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868339+0000","last_change":"2024-08-28T19:18:32.489452+0000","last_active":"2024-08-28T19:20:37.868339+0000","last_peered":"2024-08-28T19:20:37.868339+0000","last_clean":"2024-08-28T19:20:37.868339+0000","last_became_active":"2024-08-28T19:18:32.487482+0000","last_became_peered":"2024-08-28T19:18:32.487482+0000","last_unstale":"2024-08-28T19:20:37.868339+0000","last_undegraded":"2024-08-28T19:20:37.868339+0000","last_fullsized":"2024-08-28T19:20:37.868339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:52:08.175080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884068+0000","last_change":"2024-08-28T19:18:32.483850+0000","last_active":"2024-08-28T19:20:37.884068+0000","last_peered":"2024-08-28T19:20:37.884068+0000","last_clean":"2024-08-28T19:20:37.884068+0000","last_became_active":"2024-08-28T19:18:32.483476+0000","last_became_peered":"2024-08-28T19:18:32.483476+0000","last_unstale":"2024-08-28T19:20:37.884068+0000","last_undegraded":"2024-08-28T19:20:37.884068+0000","last_fullsized":"2024-08-28T19:20:37.884068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:32:37.465218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868836+0000","last_change":"2024-08-28T19:18:32.492252+0000","last_active":"2024-08-28T19:20:37.868836+0000","last_peered":"2024-08-28T19:20:37.868836+0000","last_clean":"2024-08-28T19:20:37.868836+0000","last_became_active":"2024-08-28T19:18:32.491659+0000","last_became_peered":"2024-08-28T19:18:32.491659+0000","last_unstale":"2024-08-28T19:20:37.868836+0000","last_undegraded":"2024-08-28T19:20:37.868836+0000","last_fullsized":"2024-08-28T19:20:37.868836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:19:28.901559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865090+0000","last_change":"2024-08-28T19:18:32.491940+0000","last_active":"2024-08-28T19:20:37.865090+0000","last_peered":"2024-08-28T19:20:37.865090+0000","last_clean":"2024-08-28T19:20:37.865090+0000","last_became_active":"2024-08-28T19:18:32.487474+0000","last_became_peered":"2024-08-28T19:18:32.487474+0000","last_unstale":"2024-08-28T19:20:37.865090+0000","last_undegraded":"2024-08-28T19:20:37.865090+0000","last_fullsized":"2024-08-28T19:20:37.865090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:04:32.508620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871438+0000","last_change":"2024-08-28T19:18:32.485200+0000","last_active":"2024-08-28T19:20:37.871438+0000","last_peered":"2024-08-28T19:20:37.871438+0000","last_clean":"2024-08-28T19:20:37.871438+0000","last_became_active":"2024-08-28T19:18:32.484858+0000","last_became_peered":"2024-08-28T19:18:32.484858+0000","last_unstale":"2024-08-28T19:20:37.871438+0000","last_undegraded":"2024-08-28T19:20:37.871438+0000","last_fullsized":"2024-08-28T19:20:37.871438+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:10:57.925659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886493+0000","last_change":"2024-08-28T19:18:32.489903+0000","last_active":"2024-08-28T19:20:37.886493+0000","last_peered":"2024-08-28T19:20:37.886493+0000","last_clean":"2024-08-28T19:20:37.886493+0000","last_became_active":"2024-08-28T19:18:32.488950+0000","last_became_peered":"2024-08-28T19:18:32.488950+0000","last_unstale":"2024-08-28T19:20:37.886493+0000","last_undegraded":"2024-08-28T19:20:37.886493+0000","last_fullsized":"2024-08-28T19:20:37.886493+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:06:42.284177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872318+0000","last_change":"2024-08-28T19:18:32.488236+0000","last_active":"2024-08-28T19:20:37.872318+0000","last_peered":"2024-08-28T19:20:37.872318+0000","last_clean":"2024-08-28T19:20:37.872318+0000","last_became_active":"2024-08-28T19:18:32.487776+0000","last_became_peered":"2024-08-28T19:18:32.487776+0000","last_unstale":"2024-08-28T19:20:37.872318+0000","last_undegraded":"2024-08-28T19:20:37.872318+0000","last_fullsized":"2024-08-28T19:20:37.872318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:25:58.754740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875857+0000","last_change":"2024-08-28T19:18:32.485860+0000","last_active":"2024-08-28T19:20:37.875857+0000","last_peered":"2024-08-28T19:20:37.875857+0000","last_clean":"2024-08-28T19:20:37.875857+0000","last_became_active":"2024-08-28T19:18:32.485415+0000","last_became_peered":"2024-08-28T19:18:32.485415+0000","last_unstale":"2024-08-28T19:20:37.875857+0000","last_undegraded":"2024-08-28T19:20:37.875857+0000","last_fullsized":"2024-08-28T19:20:37.875857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:45:32.868651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873054+0000","last_change":"2024-08-28T19:18:32.485515+0000","last_active":"2024-08-28T19:20:37.873054+0000","last_peered":"2024-08-28T19:20:37.873054+0000","last_clean":"2024-08-28T19:20:37.873054+0000","last_became_active":"2024-08-28T19:18:32.485075+0000","last_became_peered":"2024-08-28T19:18:32.485075+0000","last_unstale":"2024-08-28T19:20:37.873054+0000","last_undegraded":"2024-08-28T19:20:37.873054+0000","last_fullsized":"2024-08-28T19:20:37.873054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:01:28.035479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864600+0000","last_change":"2024-08-28T19:18:32.484685+0000","last_active":"2024-08-28T19:20:37.864600+0000","last_peered":"2024-08-28T19:20:37.864600+0000","last_clean":"2024-08-28T19:20:37.864600+0000","last_became_active":"2024-08-28T19:18:32.484095+0000","last_became_peered":"2024-08-28T19:18:32.484095+0000","last_unstale":"2024-08-28T19:20:37.864600+0000","last_undegraded":"2024-08-28T19:20:37.864600+0000","last_fullsized":"2024-08-28T19:20:37.864600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:53:27.229186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884050+0000","last_change":"2024-08-28T19:18:32.488030+0000","last_active":"2024-08-28T19:20:37.884050+0000","last_peered":"2024-08-28T19:20:37.884050+0000","last_clean":"2024-08-28T19:20:37.884050+0000","last_became_active":"2024-08-28T19:18:32.487042+0000","last_became_peered":"2024-08-28T19:18:32.487042+0000","last_unstale":"2024-08-28T19:20:37.884050+0000","last_undegraded":"2024-08-28T19:20:37.884050+0000","last_fullsized":"2024-08-28T19:20:37.884050+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:56.966475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886125+0000","last_change":"2024-08-28T19:18:32.489442+0000","last_active":"2024-08-28T19:20:37.886125+0000","last_peered":"2024-08-28T19:20:37.886125+0000","last_clean":"2024-08-28T19:20:37.886125+0000","last_became_active":"2024-08-28T19:18:32.488405+0000","last_became_peered":"2024-08-28T19:18:32.488405+0000","last_unstale":"2024-08-28T19:20:37.886125+0000","last_undegraded":"2024-08-28T19:20:37.886125+0000","last_fullsized":"2024-08-28T19:20:37.886125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:21:03.026893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870138+0000","last_change":"2024-08-28T19:20:09.164305+0000","last_active":"2024-08-28T19:20:37.870138+0000","last_peered":"2024-08-28T19:20:37.870138+0000","last_clean":"2024-08-28T19:20:37.870138+0000","last_became_active":"2024-08-28T19:20:09.163834+0000","last_became_peered":"2024-08-28T19:20:09.163834+0000","last_unstale":"2024-08-28T19:20:37.870138+0000","last_undegraded":"2024-08-28T19:20:37.870138+0000","last_fullsized":"2024-08-28T19:20:37.870138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:20:10.137952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870501+0000","last_change":"2024-08-28T19:18:32.492077+0000","last_active":"2024-08-28T19:20:37.870501+0000","last_peered":"2024-08-28T19:20:37.870501+0000","last_clean":"2024-08-28T19:20:37.870501+0000","last_became_active":"2024-08-28T19:18:32.491722+0000","last_became_peered":"2024-08-28T19:18:32.491722+0000","last_unstale":"2024-08-28T19:20:37.870501+0000","last_undegraded":"2024-08-28T19:20:37.870501+0000","last_fullsized":"2024-08-28T19:20:37.870501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:16:21.697045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865541+0000","last_change":"2024-08-28T19:18:32.479845+0000","last_active":"2024-08-28T19:20:37.865541+0000","last_peered":"2024-08-28T19:20:37.865541+0000","last_clean":"2024-08-28T19:20:37.865541+0000","last_became_active":"2024-08-28T19:18:32.477690+0000","last_became_peered":"2024-08-28T19:18:32.477690+0000","last_unstale":"2024-08-28T19:20:37.865541+0000","last_undegraded":"2024-08-28T19:20:37.865541+0000","last_fullsized":"2024-08-28T19:20:37.865541+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:11.281415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866009+0000","last_change":"2024-08-28T19:19:08.624185+0000","last_active":"2024-08-28T19:20:37.866009+0000","last_peered":"2024-08-28T19:20:37.866009+0000","last_clean":"2024-08-28T19:20:37.866009+0000","last_became_active":"2024-08-28T19:19:08.623745+0000","last_became_peered":"2024-08-28T19:19:08.623745+0000","last_unstale":"2024-08-28T19:20:37.866009+0000","last_undegraded":"2024-08-28T19:20:37.866009+0000","last_fullsized":"2024-08-28T19:20:37.866009+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:56.657173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.887026+0000","last_change":"2024-08-28T19:18:32.499038+0000","last_active":"2024-08-28T19:20:37.887026+0000","last_peered":"2024-08-28T19:20:37.887026+0000","last_clean":"2024-08-28T19:20:37.887026+0000","last_became_active":"2024-08-28T19:18:32.498276+0000","last_became_peered":"2024-08-28T19:18:32.498276+0000","last_unstale":"2024-08-28T19:20:37.887026+0000","last_undegraded":"2024-08-28T19:20:37.887026+0000","last_fullsized":"2024-08-28T19:20:37.887026+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:17:45.468448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"84'16","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.882492+0000","last_change":"2024-08-28T19:18:32.492183+0000","last_active":"2024-08-28T19:20:37.882492+0000","last_peered":"2024-08-28T19:20:37.882492+0000","last_clean":"2024-08-28T19:20:37.882492+0000","last_became_active":"2024-08-28T19:18:32.491815+0000","last_became_peered":"2024-08-28T19:18:32.491815+0000","last_unstale":"2024-08-28T19:20:37.882492+0000","last_undegraded":"2024-08-28T19:20:37.882492+0000","last_fullsized":"2024-08-28T19:20:37.882492+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:13:11.183955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886985+0000","last_change":"2024-08-28T19:18:32.497399+0000","last_active":"2024-08-28T19:20:37.886985+0000","last_peered":"2024-08-28T19:20:37.886985+0000","last_clean":"2024-08-28T19:20:37.886985+0000","last_became_active":"2024-08-28T19:18:32.496901+0000","last_became_peered":"2024-08-28T19:18:32.496901+0000","last_unstale":"2024-08-28T19:20:37.886985+0000","last_undegraded":"2024-08-28T19:20:37.886985+0000","last_fullsized":"2024-08-28T19:20:37.886985+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:05:57.634590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864593+0000","last_change":"2024-08-28T19:18:32.495871+0000","last_active":"2024-08-28T19:20:37.864593+0000","last_peered":"2024-08-28T19:20:37.864593+0000","last_clean":"2024-08-28T19:20:37.864593+0000","last_became_active":"2024-08-28T19:18:32.495515+0000","last_became_peered":"2024-08-28T19:18:32.495515+0000","last_unstale":"2024-08-28T19:20:37.864593+0000","last_undegraded":"2024-08-28T19:20:37.864593+0000","last_fullsized":"2024-08-28T19:20:37.864593+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:53:56.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875472+0000","last_change":"2024-08-28T19:18:32.495711+0000","last_active":"2024-08-28T19:20:37.875472+0000","last_peered":"2024-08-28T19:20:37.875472+0000","last_clean":"2024-08-28T19:20:37.875472+0000","last_became_active":"2024-08-28T19:18:32.495271+0000","last_became_peered":"2024-08-28T19:18:32.495271+0000","last_unstale":"2024-08-28T19:20:37.875472+0000","last_undegraded":"2024-08-28T19:20:37.875472+0000","last_fullsized":"2024-08-28T19:20:37.875472+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:05.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880842+0000","last_change":"2024-08-28T19:18:32.489731+0000","last_active":"2024-08-28T19:20:37.880842+0000","last_peered":"2024-08-28T19:20:37.880842+0000","last_clean":"2024-08-28T19:20:37.880842+0000","last_became_active":"2024-08-28T19:18:32.489366+0000","last_became_peered":"2024-08-28T19:18:32.489366+0000","last_unstale":"2024-08-28T19:20:37.880842+0000","last_undegraded":"2024-08-28T19:20:37.880842+0000","last_fullsized":"2024-08-28T19:20:37.880842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:52:24.180155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.861421+0000","last_change":"2024-08-28T19:18:32.490270+0000","last_active":"2024-08-28T19:20:37.861421+0000","last_peered":"2024-08-28T19:20:37.861421+0000","last_clean":"2024-08-28T19:20:37.861421+0000","last_became_active":"2024-08-28T19:18:32.489667+0000","last_became_peered":"2024-08-28T19:18:32.489667+0000","last_unstale":"2024-08-28T19:20:37.861421+0000","last_undegraded":"2024-08-28T19:20:37.861421+0000","last_fullsized":"2024-08-28T19:20:37.861421+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:18.571978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873527+0000","last_change":"2024-08-28T19:18:32.482553+0000","last_active":"2024-08-28T19:20:37.873527+0000","last_peered":"2024-08-28T19:20:37.873527+0000","last_clean":"2024-08-28T19:20:37.873527+0000","last_became_active":"2024-08-28T19:18:32.481161+0000","last_became_peered":"2024-08-28T19:18:32.481161+0000","last_unstale":"2024-08-28T19:20:37.873527+0000","last_undegraded":"2024-08-28T19:20:37.873527+0000","last_fullsized":"2024-08-28T19:20:37.873527+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:13.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883684+0000","last_change":"2024-08-28T19:18:32.492473+0000","last_active":"2024-08-28T19:20:37.883684+0000","last_peered":"2024-08-28T19:20:37.883684+0000","last_clean":"2024-08-28T19:20:37.883684+0000","last_became_active":"2024-08-28T19:18:32.491897+0000","last_became_peered":"2024-08-28T19:18:32.491897+0000","last_unstale":"2024-08-28T19:20:37.883684+0000","last_undegraded":"2024-08-28T19:20:37.883684+0000","last_fullsized":"2024-08-28T19:20:37.883684+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:24.279391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870908+0000","last_change":"2024-08-28T19:18:32.485621+0000","last_active":"2024-08-28T19:20:37.870908+0000","last_peered":"2024-08-28T19:20:37.870908+0000","last_clean":"2024-08-28T19:20:37.870908+0000","last_became_active":"2024-08-28T19:18:32.485267+0000","last_became_peered":"2024-08-28T19:18:32.485267+0000","last_unstale":"2024-08-28T19:20:37.870908+0000","last_undegraded":"2024-08-28T19:20:37.870908+0000","last_fullsized":"2024-08-28T19:20:37.870908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:30:08.932135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884032+0000","last_change":"2024-08-28T19:18:32.488457+0000","last_active":"2024-08-28T19:20:37.884032+0000","last_peered":"2024-08-28T19:20:37.884032+0000","last_clean":"2024-08-28T19:20:37.884032+0000","last_became_active":"2024-08-28T19:18:32.487555+0000","last_became_peered":"2024-08-28T19:18:32.487555+0000","last_unstale":"2024-08-28T19:20:37.884032+0000","last_undegraded":"2024-08-28T19:20:37.884032+0000","last_fullsized":"2024-08-28T19:20:37.884032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:03:26.390481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":863883,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1497,"num_read_kb":962,"num_write":479,"num_write_kb":2112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2961408,"data_stored":2591649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":361,"ondisk_log_size":361,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":68,"seq":292057776173,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":500491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":62,"seq":266287972401,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29120,"kb_used_data":2036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716032,"statfs":{"total":95995035648,"available":95965216768,"internally_reserved":0,"allocated":2084864,"data_stored":1553912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":57,"seq":244813135927,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40876,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704276,"statfs":{"total":95995035648,"available":95953178624,"internally_reserved":0,"allocated":1404928,"data_stored":897061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":51,"seq":219043332155,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36708,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708444,"statfs":{"total":95995035648,"available":95957446656,"internally_reserved":0,"allocated":1396736,"data_stored":893343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":45,"seq":193273528384,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1593344,"data_stored":1096625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":40,"seq":171798691910,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":495785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888137,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1605632,"data_stored":1088251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051664,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":499327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215188,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41248,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703904,"statfs":{"total":95995035648,"available":95952797696,"internally_reserved":0,"allocated":1785856,"data_stored":1281953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1077248,"data_stored":566440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":566220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":496709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":819200,"data_stored":787422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":397128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":464917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-28T19:21:09.081 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-28T19:21:09.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-28T19:21:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:09.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:09 smithi016 ceph-mon[18556]: from='client.25337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:09 smithi115 ceph-mon[22184]: from='client.25337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:09.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:09 smithi028 ceph-mon[21769]: from='client.25337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:09.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.859+0000 7f828d2cc640 1 -- 172.21.15.16:0/1154176620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8288103900 msgr2=0x7f8288103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:09.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.859+0000 7f828d2cc640 1 --2- 172.21.15.16:0/1154176620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8288103900 0x7f8288103d10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8278009900 tx=0x7f827802a410 comp rx=0 tx=0).stop 2024-08-28T19:21:09.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.859+0000 7f828d2cc640 1 -- 172.21.15.16:0/1154176620 shutdown_connections 2024-08-28T19:21:09.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.859+0000 7f828d2cc640 1 --2- 172.21.15.16:0/1154176620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8288109b90 0x7f82881103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:09.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.859+0000 7f828d2cc640 1 --2- 172.21.15.16:0/1154176620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8288109240 0x7f8288109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:09.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.859+0000 7f828d2cc640 1 --2- 172.21.15.16:0/1154176620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8288103900 0x7f8288103d10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:09.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.859+0000 7f828d2cc640 1 -- 172.21.15.16:0/1154176620 >> 172.21.15.16:0/1154176620 conn(0x7f828807b5f0 msgr2=0x7f828807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:09.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.859+0000 7f828d2cc640 1 -- 172.21.15.16:0/1154176620 shutdown_connections 2024-08-28T19:21:09.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.859+0000 7f828d2cc640 1 -- 172.21.15.16:0/1154176620 wait complete. 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.860+0000 7f828d2cc640 1 Processor -- start 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.860+0000 7f828d2cc640 1 -- start start 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.860+0000 7f828d2cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8288103900 0x7f82880754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.860+0000 7f828d2cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8288109240 0x7f82880759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.860+0000 7f828d2cc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8288109b90 0x7f8288075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.860+0000 7f828d2cc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8288113850 con 0x7f8288109240 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.860+0000 7f828d2cc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82881136d0 con 0x7f8288103900 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.860+0000 7f828d2cc640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f82881139d0 con 0x7f8288109b90 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.860+0000 7f82867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8288109240 0x7f82880759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.860+0000 7f82867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8288109240 0x7f82880759f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45308/0 (socket says 172.21.15.16:45308) 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.860+0000 7f82867fc640 1 -- 172.21.15.16:0/2419165227 learned_addr learned my addr 172.21.15.16:0/2419165227 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:09.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f8286ffd640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8288103900 0x7f82880754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f82877fe640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8288109b90 0x7f8288075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f82867fc640 1 -- 172.21.15.16:0/2419165227 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8288109b90 msgr2=0x7f8288075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f82867fc640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8288109b90 0x7f8288075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f82867fc640 1 -- 172.21.15.16:0/2419165227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8288103900 msgr2=0x7f82880754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f82867fc640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8288103900 0x7f82880754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f82867fc640 1 -- 172.21.15.16:0/2419165227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82881a8520 con 0x7f8288109240 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f82867fc640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8288109240 0x7f82880759f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f827c00d950 tx=0x7f827c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f826ffff640 1 -- 172.21.15.16:0/2419165227 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c017070 con 0x7f8288109240 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f828d2cc640 1 -- 172.21.15.16:0/2419165227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82881a8810 con 0x7f8288109240 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f826ffff640 1 -- 172.21.15.16:0/2419165227 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f827c009c30 con 0x7f8288109240 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f826ffff640 1 -- 172.21.15.16:0/2419165227 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c0047a0 con 0x7f8288109240 2024-08-28T19:21:09.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.861+0000 7f828d2cc640 1 -- 172.21.15.16:0/2419165227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82881a8da0 con 0x7f8288109240 2024-08-28T19:21:09.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.862+0000 7f828d2cc640 1 -- 172.21.15.16:0/2419165227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8288103d10 con 0x7f8288109240 2024-08-28T19:21:09.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.864+0000 7f826ffff640 1 -- 172.21.15.16:0/2419165227 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f827c004020 con 0x7f8288109240 2024-08-28T19:21:09.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.864+0000 7f826ffff640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f82700b4c10 0x7f82700b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:09.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.865+0000 7f8286ffd640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f82700b4c10 0x7f82700b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:09.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.865+0000 7f826ffff640 1 -- 172.21.15.16:0/2419165227 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f827c0e58c0 con 0x7f8288109240 2024-08-28T19:21:09.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.865+0000 7f8286ffd640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f82700b4c10 0x7f82700b7120 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8278000c00 tx=0x7f8278031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:09.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:09.868+0000 7f826ffff640 1 -- 172.21.15.16:0/2419165227 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f827c020020 con 0x7f8288109240 2024-08-28T19:21:10.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.049+0000 7f828d2cc640 1 -- 172.21.15.16:0/2419165227 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f82880635a0 con 0x7f82700b4c10 2024-08-28T19:21:10.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.055+0000 7f826ffff640 1 -- 172.21.15.16:0/2419165227 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340586 (secure 0 0 0) 0x7f82880635a0 con 0x7f82700b4c10 2024-08-28T19:21:10.059 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:21:10.061 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-08-28T19:21:10.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.058+0000 7f828d2cc640 1 -- 172.21.15.16:0/2419165227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f82700b4c10 msgr2=0x7f82700b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:10.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.058+0000 7f828d2cc640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f82700b4c10 0x7f82700b7120 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8278000c00 tx=0x7f8278031400 comp rx=0 tx=0).stop 2024-08-28T19:21:10.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.058+0000 7f828d2cc640 1 -- 172.21.15.16:0/2419165227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8288109240 msgr2=0x7f82880759f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:10.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.058+0000 7f828d2cc640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8288109240 0x7f82880759f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f827c00d950 tx=0x7f827c00de10 comp rx=0 tx=0).stop 2024-08-28T19:21:10.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.058+0000 7f828d2cc640 1 -- 172.21.15.16:0/2419165227 shutdown_connections 2024-08-28T19:21:10.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.058+0000 7f828d2cc640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8288109b90 0x7f8288075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:10.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.058+0000 7f828d2cc640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8288109240 0x7f82880759f0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:10.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.058+0000 7f828d2cc640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f82700b4c10 0x7f82700b7120 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:10.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.058+0000 7f828d2cc640 1 --2- 172.21.15.16:0/2419165227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8288103900 0x7f82880754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:10.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.058+0000 7f828d2cc640 1 -- 172.21.15.16:0/2419165227 >> 172.21.15.16:0/2419165227 conn(0x7f828807b5f0 msgr2=0x7f8288107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:10.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.059+0000 7f828d2cc640 1 -- 172.21.15.16:0/2419165227 shutdown_connections 2024-08-28T19:21:10.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:10.059+0000 7f828d2cc640 1 -- 172.21.15.16:0/2419165227 wait complete. 2024-08-28T19:21:10.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:10 smithi115 ceph-mon[22184]: pgmap v343: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 174 KiB/s wr, 55 op/s 2024-08-28T19:21:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:10 smithi115 ceph-mon[22184]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:10.731 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:10 smithi016 ceph-mon[18556]: pgmap v343: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 174 KiB/s wr, 55 op/s 2024-08-28T19:21:10.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:10 smithi016 ceph-mon[18556]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:10.760 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-08-28T19:21:09.498850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1454485,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1601,"num_read_kb":1044,"num_write":612,"num_write_kb":4398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":963,"ondisk_log_size":963,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":423600,"kb_used_data":15992,"kb_used_omap":18,"kb_used_meta":407277,"kb_avail":1124518224,"statfs":{"total":1151940427776,"available":1151506661376,"internally_reserved":0,"allocated":16375808,"data_stored":10329893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":417051962},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":792365,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":446,"num_read_kb":357,"num_write":225,"num_write_kb":2094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003369"},"pg_stats":[{"pgid":"2.1f","version":"84'14","reported_seq":183,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:04.821906+0000","last_change":"2024-08-28T19:19:08.633326+0000","last_active":"2024-08-28T19:21:04.821906+0000","last_peered":"2024-08-28T19:21:04.821906+0000","last_clean":"2024-08-28T19:21:04.821906+0000","last_became_active":"2024-08-28T19:19:08.623876+0000","last_became_peered":"2024-08-28T19:19:08.623876+0000","last_unstale":"2024-08-28T19:21:04.821906+0000","last_undegraded":"2024-08-28T19:21:04.821906+0000","last_fullsized":"2024-08-28T19:21:04.821906+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:19:50.632337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866980+0000","last_change":"2024-08-28T19:18:32.492604+0000","last_active":"2024-08-28T19:20:37.866980+0000","last_peered":"2024-08-28T19:20:37.866980+0000","last_clean":"2024-08-28T19:20:37.866980+0000","last_became_active":"2024-08-28T19:18:32.489427+0000","last_became_peered":"2024-08-28T19:18:32.489427+0000","last_unstale":"2024-08-28T19:20:37.866980+0000","last_undegraded":"2024-08-28T19:20:37.866980+0000","last_fullsized":"2024-08-28T19:20:37.866980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:36:14.841315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010515+0000","last_change":"2024-08-28T19:18:39.528815+0000","last_active":"2024-08-28T19:20:09.010515+0000","last_peered":"2024-08-28T19:20:09.010515+0000","last_clean":"2024-08-28T19:20:09.010515+0000","last_became_active":"2024-08-28T19:18:39.527781+0000","last_became_peered":"2024-08-28T19:18:39.527781+0000","last_unstale":"2024-08-28T19:20:09.010515+0000","last_undegraded":"2024-08-28T19:20:09.010515+0000","last_fullsized":"2024-08-28T19:20:09.010515+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:39.831426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525559+0000","last_change":"2024-08-28T19:18:31.082641+0000","last_active":"2024-08-28T19:20:09.525559+0000","last_peered":"2024-08-28T19:20:09.525559+0000","last_clean":"2024-08-28T19:20:09.525559+0000","last_became_active":"2024-08-28T19:18:31.082260+0000","last_became_peered":"2024-08-28T19:18:31.082260+0000","last_unstale":"2024-08-28T19:20:09.525559+0000","last_undegraded":"2024-08-28T19:20:09.525559+0000","last_fullsized":"2024-08-28T19:20:09.525559+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:09:55.836463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886294+0000","last_change":"2024-08-28T19:18:32.493498+0000","last_active":"2024-08-28T19:20:37.886294+0000","last_peered":"2024-08-28T19:20:37.886294+0000","last_clean":"2024-08-28T19:20:37.886294+0000","last_became_active":"2024-08-28T19:18:32.490969+0000","last_became_peered":"2024-08-28T19:18:32.490969+0000","last_unstale":"2024-08-28T19:20:37.886294+0000","last_undegraded":"2024-08-28T19:20:37.886294+0000","last_fullsized":"2024-08-28T19:20:37.886294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:34:29.537622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614607+0000","last_change":"2024-08-28T19:18:39.520922+0000","last_active":"2024-08-28T19:20:09.614607+0000","last_peered":"2024-08-28T19:20:09.614607+0000","last_clean":"2024-08-28T19:20:09.614607+0000","last_became_active":"2024-08-28T19:18:39.520362+0000","last_became_peered":"2024-08-28T19:18:39.520362+0000","last_unstale":"2024-08-28T19:20:09.614607+0000","last_undegraded":"2024-08-28T19:20:09.614607+0000","last_fullsized":"2024-08-28T19:20:09.614607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:35:40.658820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":53,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110426+0000","last_change":"2024-08-28T19:18:30.471870+0000","last_active":"2024-08-28T19:20:09.110426+0000","last_peered":"2024-08-28T19:20:09.110426+0000","last_clean":"2024-08-28T19:20:09.110426+0000","last_became_active":"2024-08-28T19:18:30.471438+0000","last_became_peered":"2024-08-28T19:18:30.471438+0000","last_unstale":"2024-08-28T19:20:09.110426+0000","last_undegraded":"2024-08-28T19:20:09.110426+0000","last_fullsized":"2024-08-28T19:20:09.110426+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:44:34.137395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.881211+0000","last_change":"2024-08-28T19:18:32.486485+0000","last_active":"2024-08-28T19:20:37.881211+0000","last_peered":"2024-08-28T19:20:37.881211+0000","last_clean":"2024-08-28T19:20:37.881211+0000","last_became_active":"2024-08-28T19:18:32.485888+0000","last_became_peered":"2024-08-28T19:18:32.485888+0000","last_unstale":"2024-08-28T19:20:37.881211+0000","last_undegraded":"2024-08-28T19:20:37.881211+0000","last_fullsized":"2024-08-28T19:20:37.881211+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:45:48.757318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615286+0000","last_change":"2024-08-28T19:19:08.633317+0000","last_active":"2024-08-28T19:20:09.615286+0000","last_peered":"2024-08-28T19:20:09.615286+0000","last_clean":"2024-08-28T19:20:09.615286+0000","last_became_active":"2024-08-28T19:19:08.632689+0000","last_became_peered":"2024-08-28T19:19:08.632689+0000","last_unstale":"2024-08-28T19:20:09.615286+0000","last_undegraded":"2024-08-28T19:20:09.615286+0000","last_fullsized":"2024-08-28T19:20:09.615286+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:35:08.740645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110357+0000","last_change":"2024-08-28T19:18:30.473982+0000","last_active":"2024-08-28T19:20:09.110357+0000","last_peered":"2024-08-28T19:20:09.110357+0000","last_clean":"2024-08-28T19:20:09.110357+0000","last_became_active":"2024-08-28T19:18:30.473552+0000","last_became_peered":"2024-08-28T19:18:30.473552+0000","last_unstale":"2024-08-28T19:20:09.110357+0000","last_undegraded":"2024-08-28T19:20:09.110357+0000","last_fullsized":"2024-08-28T19:20:09.110357+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:22:29.435517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884392+0000","last_change":"2024-08-28T19:18:32.487881+0000","last_active":"2024-08-28T19:20:37.884392+0000","last_peered":"2024-08-28T19:20:37.884392+0000","last_clean":"2024-08-28T19:20:37.884392+0000","last_became_active":"2024-08-28T19:18:32.487039+0000","last_became_peered":"2024-08-28T19:18:32.487039+0000","last_unstale":"2024-08-28T19:20:37.884392+0000","last_undegraded":"2024-08-28T19:20:37.884392+0000","last_fullsized":"2024-08-28T19:20:37.884392+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:23:35.757811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614817+0000","last_change":"2024-08-28T19:18:39.518654+0000","last_active":"2024-08-28T19:20:09.614817+0000","last_peered":"2024-08-28T19:20:09.614817+0000","last_clean":"2024-08-28T19:20:09.614817+0000","last_became_active":"2024-08-28T19:18:39.518087+0000","last_became_peered":"2024-08-28T19:18:39.518087+0000","last_unstale":"2024-08-28T19:20:09.614817+0000","last_undegraded":"2024-08-28T19:20:09.614817+0000","last_fullsized":"2024-08-28T19:20:09.614817+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:08.284865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.887381+0000","last_change":"2024-08-28T19:18:32.491928+0000","last_active":"2024-08-28T19:20:37.887381+0000","last_peered":"2024-08-28T19:20:37.887381+0000","last_clean":"2024-08-28T19:20:37.887381+0000","last_became_active":"2024-08-28T19:18:32.491538+0000","last_became_peered":"2024-08-28T19:18:32.491538+0000","last_unstale":"2024-08-28T19:20:37.887381+0000","last_undegraded":"2024-08-28T19:20:37.887381+0000","last_fullsized":"2024-08-28T19:20:37.887381+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:39:06.077830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":24,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663446+0000","last_change":"2024-08-28T19:19:08.634752+0000","last_active":"2024-08-28T19:20:09.663446+0000","last_peered":"2024-08-28T19:20:09.663446+0000","last_clean":"2024-08-28T19:20:09.663446+0000","last_became_active":"2024-08-28T19:19:08.620784+0000","last_became_peered":"2024-08-28T19:19:08.620784+0000","last_unstale":"2024-08-28T19:20:09.663446+0000","last_undegraded":"2024-08-28T19:20:09.663446+0000","last_fullsized":"2024-08-28T19:20:09.663446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:51.640445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.614895+0000","last_change":"2024-08-28T19:18:39.681413+0000","last_active":"2024-08-28T19:20:09.614895+0000","last_peered":"2024-08-28T19:20:09.614895+0000","last_clean":"2024-08-28T19:20:09.614895+0000","last_became_active":"2024-08-28T19:18:39.680793+0000","last_became_peered":"2024-08-28T19:18:39.680793+0000","last_unstale":"2024-08-28T19:20:09.614895+0000","last_undegraded":"2024-08-28T19:20:09.614895+0000","last_fullsized":"2024-08-28T19:20:09.614895+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:45:43.381019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870888+0000","last_change":"2024-08-28T19:18:32.486739+0000","last_active":"2024-08-28T19:20:37.870888+0000","last_peered":"2024-08-28T19:20:37.870888+0000","last_clean":"2024-08-28T19:20:37.870888+0000","last_became_active":"2024-08-28T19:18:32.486347+0000","last_became_peered":"2024-08-28T19:18:32.486347+0000","last_unstale":"2024-08-28T19:20:37.870888+0000","last_undegraded":"2024-08-28T19:20:37.870888+0000","last_fullsized":"2024-08-28T19:20:37.870888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:27:36.448569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010653+0000","last_change":"2024-08-28T19:18:30.467690+0000","last_active":"2024-08-28T19:20:09.010653+0000","last_peered":"2024-08-28T19:20:09.010653+0000","last_clean":"2024-08-28T19:20:09.010653+0000","last_became_active":"2024-08-28T19:18:30.467249+0000","last_became_peered":"2024-08-28T19:18:30.467249+0000","last_unstale":"2024-08-28T19:20:09.010653+0000","last_undegraded":"2024-08-28T19:20:09.010653+0000","last_fullsized":"2024-08-28T19:20:09.010653+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:37:28.876666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615167+0000","last_change":"2024-08-28T19:18:39.523889+0000","last_active":"2024-08-28T19:20:09.615167+0000","last_peered":"2024-08-28T19:20:09.615167+0000","last_clean":"2024-08-28T19:20:09.615167+0000","last_became_active":"2024-08-28T19:18:39.523384+0000","last_became_peered":"2024-08-28T19:18:39.523384+0000","last_unstale":"2024-08-28T19:20:09.615167+0000","last_undegraded":"2024-08-28T19:20:09.615167+0000","last_fullsized":"2024-08-28T19:20:09.615167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:57:32.277680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873180+0000","last_change":"2024-08-28T19:18:32.482257+0000","last_active":"2024-08-28T19:20:37.873180+0000","last_peered":"2024-08-28T19:20:37.873180+0000","last_clean":"2024-08-28T19:20:37.873180+0000","last_became_active":"2024-08-28T19:18:32.481884+0000","last_became_peered":"2024-08-28T19:18:32.481884+0000","last_unstale":"2024-08-28T19:20:37.873180+0000","last_undegraded":"2024-08-28T19:20:37.873180+0000","last_fullsized":"2024-08-28T19:20:37.873180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T03:56:28.754369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":47,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525435+0000","last_change":"2024-08-28T19:18:30.467307+0000","last_active":"2024-08-28T19:20:09.525435+0000","last_peered":"2024-08-28T19:20:09.525435+0000","last_clean":"2024-08-28T19:20:09.525435+0000","last_became_active":"2024-08-28T19:18:30.466942+0000","last_became_peered":"2024-08-28T19:18:30.466942+0000","last_unstale":"2024-08-28T19:20:09.525435+0000","last_undegraded":"2024-08-28T19:20:09.525435+0000","last_fullsized":"2024-08-28T19:20:09.525435+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:05:23.682110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007238+0000","last_change":"2024-08-28T19:18:39.531421+0000","last_active":"2024-08-28T19:20:09.007238+0000","last_peered":"2024-08-28T19:20:09.007238+0000","last_clean":"2024-08-28T19:20:09.007238+0000","last_became_active":"2024-08-28T19:18:39.531008+0000","last_became_peered":"2024-08-28T19:18:39.531008+0000","last_unstale":"2024-08-28T19:20:09.007238+0000","last_undegraded":"2024-08-28T19:20:09.007238+0000","last_fullsized":"2024-08-28T19:20:09.007238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:46:49.580601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868264+0000","last_change":"2024-08-28T19:18:32.485204+0000","last_active":"2024-08-28T19:20:37.868264+0000","last_peered":"2024-08-28T19:20:37.868264+0000","last_clean":"2024-08-28T19:20:37.868264+0000","last_became_active":"2024-08-28T19:18:32.484830+0000","last_became_peered":"2024-08-28T19:18:32.484830+0000","last_unstale":"2024-08-28T19:20:37.868264+0000","last_undegraded":"2024-08-28T19:20:37.868264+0000","last_fullsized":"2024-08-28T19:20:37.868264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:55:41.358718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007905+0000","last_change":"2024-08-28T19:18:31.082361+0000","last_active":"2024-08-28T19:20:09.007905+0000","last_peered":"2024-08-28T19:20:09.007905+0000","last_clean":"2024-08-28T19:20:09.007905+0000","last_became_active":"2024-08-28T19:18:31.081930+0000","last_became_peered":"2024-08-28T19:18:31.081930+0000","last_unstale":"2024-08-28T19:20:09.007905+0000","last_undegraded":"2024-08-28T19:20:09.007905+0000","last_fullsized":"2024-08-28T19:20:09.007905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:41:05.543088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615411+0000","last_change":"2024-08-28T19:18:39.682341+0000","last_active":"2024-08-28T19:20:09.615411+0000","last_peered":"2024-08-28T19:20:09.615411+0000","last_clean":"2024-08-28T19:20:09.615411+0000","last_became_active":"2024-08-28T19:18:39.681751+0000","last_became_peered":"2024-08-28T19:18:39.681751+0000","last_unstale":"2024-08-28T19:20:09.615411+0000","last_undegraded":"2024-08-28T19:20:09.615411+0000","last_fullsized":"2024-08-28T19:20:09.615411+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:51:24.371270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880906+0000","last_change":"2024-08-28T19:18:32.491069+0000","last_active":"2024-08-28T19:20:37.880906+0000","last_peered":"2024-08-28T19:20:37.880906+0000","last_clean":"2024-08-28T19:20:37.880906+0000","last_became_active":"2024-08-28T19:18:32.490688+0000","last_became_peered":"2024-08-28T19:18:32.490688+0000","last_unstale":"2024-08-28T19:20:37.880906+0000","last_undegraded":"2024-08-28T19:20:37.880906+0000","last_fullsized":"2024-08-28T19:20:37.880906+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:24:31.560519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615273+0000","last_change":"2024-08-28T19:18:30.739812+0000","last_active":"2024-08-28T19:20:09.615273+0000","last_peered":"2024-08-28T19:20:09.615273+0000","last_clean":"2024-08-28T19:20:09.615273+0000","last_became_active":"2024-08-28T19:18:30.739414+0000","last_became_peered":"2024-08-28T19:18:30.739414+0000","last_unstale":"2024-08-28T19:20:09.615273+0000","last_undegraded":"2024-08-28T19:20:09.615273+0000","last_fullsized":"2024-08-28T19:20:09.615273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:56:56.594993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010962+0000","last_change":"2024-08-28T19:18:39.528281+0000","last_active":"2024-08-28T19:20:09.010962+0000","last_peered":"2024-08-28T19:20:09.010962+0000","last_clean":"2024-08-28T19:20:09.010962+0000","last_became_active":"2024-08-28T19:18:39.527142+0000","last_became_peered":"2024-08-28T19:18:39.527142+0000","last_unstale":"2024-08-28T19:20:09.010962+0000","last_undegraded":"2024-08-28T19:20:09.010962+0000","last_fullsized":"2024-08-28T19:20:09.010962+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:44:19.123662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872571+0000","last_change":"2024-08-28T19:18:32.490430+0000","last_active":"2024-08-28T19:20:37.872571+0000","last_peered":"2024-08-28T19:20:37.872571+0000","last_clean":"2024-08-28T19:20:37.872571+0000","last_became_active":"2024-08-28T19:18:32.490020+0000","last_became_peered":"2024-08-28T19:18:32.490020+0000","last_unstale":"2024-08-28T19:20:37.872571+0000","last_undegraded":"2024-08-28T19:20:37.872571+0000","last_fullsized":"2024-08-28T19:20:37.872571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:52:31.153012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'101","reported_seq":230,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:04.697489+0000","last_change":"2024-08-28T19:18:30.467060+0000","last_active":"2024-08-28T19:21:04.697489+0000","last_peered":"2024-08-28T19:21:04.697489+0000","last_clean":"2024-08-28T19:21:04.697489+0000","last_became_active":"2024-08-28T19:18:30.466689+0000","last_became_peered":"2024-08-28T19:18:30.466689+0000","last_unstale":"2024-08-28T19:21:04.697489+0000","last_undegraded":"2024-08-28T19:21:04.697489+0000","last_fullsized":"2024-08-28T19:21:04.697489+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:18:36.508131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":302,"num_read_kb":268,"num_write":185,"num_write_kb":1222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.524970+0000","last_change":"2024-08-28T19:18:39.682874+0000","last_active":"2024-08-28T19:20:09.524970+0000","last_peered":"2024-08-28T19:20:09.524970+0000","last_clean":"2024-08-28T19:20:09.524970+0000","last_became_active":"2024-08-28T19:18:39.681500+0000","last_became_peered":"2024-08-28T19:18:39.681500+0000","last_unstale":"2024-08-28T19:20:09.524970+0000","last_undegraded":"2024-08-28T19:20:09.524970+0000","last_fullsized":"2024-08-28T19:20:09.524970+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:45:41.815506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.856367+0000","last_change":"2024-08-28T19:18:32.482047+0000","last_active":"2024-08-28T19:20:37.856367+0000","last_peered":"2024-08-28T19:20:37.856367+0000","last_clean":"2024-08-28T19:20:37.856367+0000","last_became_active":"2024-08-28T19:18:32.481640+0000","last_became_peered":"2024-08-28T19:18:32.481640+0000","last_unstale":"2024-08-28T19:20:37.856367+0000","last_undegraded":"2024-08-28T19:20:37.856367+0000","last_fullsized":"2024-08-28T19:20:37.856367+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:34.717764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110454+0000","last_change":"2024-08-28T19:18:31.083524+0000","last_active":"2024-08-28T19:20:09.110454+0000","last_peered":"2024-08-28T19:20:09.110454+0000","last_clean":"2024-08-28T19:20:09.110454+0000","last_became_active":"2024-08-28T19:18:31.083106+0000","last_became_peered":"2024-08-28T19:18:31.083106+0000","last_unstale":"2024-08-28T19:20:09.110454+0000","last_undegraded":"2024-08-28T19:20:09.110454+0000","last_fullsized":"2024-08-28T19:20:09.110454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:16.180754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663607+0000","last_change":"2024-08-28T19:18:39.531884+0000","last_active":"2024-08-28T19:20:09.663607+0000","last_peered":"2024-08-28T19:20:09.663607+0000","last_clean":"2024-08-28T19:20:09.663607+0000","last_became_active":"2024-08-28T19:18:39.531255+0000","last_became_peered":"2024-08-28T19:18:39.531255+0000","last_unstale":"2024-08-28T19:20:09.663607+0000","last_undegraded":"2024-08-28T19:20:09.663607+0000","last_fullsized":"2024-08-28T19:20:09.663607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:36:55.496594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.862241+0000","last_change":"2024-08-28T19:18:32.485180+0000","last_active":"2024-08-28T19:20:37.862241+0000","last_peered":"2024-08-28T19:20:37.862241+0000","last_clean":"2024-08-28T19:20:37.862241+0000","last_became_active":"2024-08-28T19:18:32.484804+0000","last_became_peered":"2024-08-28T19:18:32.484804+0000","last_unstale":"2024-08-28T19:20:37.862241+0000","last_undegraded":"2024-08-28T19:20:37.862241+0000","last_fullsized":"2024-08-28T19:20:37.862241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:12:40.958251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007361+0000","last_change":"2024-08-28T19:18:31.082226+0000","last_active":"2024-08-28T19:20:09.007361+0000","last_peered":"2024-08-28T19:20:09.007361+0000","last_clean":"2024-08-28T19:20:09.007361+0000","last_became_active":"2024-08-28T19:18:31.081833+0000","last_became_peered":"2024-08-28T19:18:31.081833+0000","last_unstale":"2024-08-28T19:20:09.007361+0000","last_undegraded":"2024-08-28T19:20:09.007361+0000","last_fullsized":"2024-08-28T19:20:09.007361+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:33:56.298093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109762+0000","last_change":"2024-08-28T19:18:39.681288+0000","last_active":"2024-08-28T19:20:09.109762+0000","last_peered":"2024-08-28T19:20:09.109762+0000","last_clean":"2024-08-28T19:20:09.109762+0000","last_became_active":"2024-08-28T19:18:39.680719+0000","last_became_peered":"2024-08-28T19:18:39.680719+0000","last_unstale":"2024-08-28T19:20:09.109762+0000","last_undegraded":"2024-08-28T19:20:09.109762+0000","last_fullsized":"2024-08-28T19:20:09.109762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:53:35.554181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880497+0000","last_change":"2024-08-28T19:18:32.487105+0000","last_active":"2024-08-28T19:20:37.880497+0000","last_peered":"2024-08-28T19:20:37.880497+0000","last_clean":"2024-08-28T19:20:37.880497+0000","last_became_active":"2024-08-28T19:18:32.486643+0000","last_became_peered":"2024-08-28T19:18:32.486643+0000","last_unstale":"2024-08-28T19:20:37.880497+0000","last_undegraded":"2024-08-28T19:20:37.880497+0000","last_fullsized":"2024-08-28T19:20:37.880497+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:18:43.274493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"84'7","reported_seq":163,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:03.821152+0000","last_change":"2024-08-28T19:18:30.471447+0000","last_active":"2024-08-28T19:21:03.821152+0000","last_peered":"2024-08-28T19:21:03.821152+0000","last_clean":"2024-08-28T19:21:03.821152+0000","last_became_active":"2024-08-28T19:18:30.470798+0000","last_became_peered":"2024-08-28T19:18:30.470798+0000","last_unstale":"2024-08-28T19:21:03.821152+0000","last_undegraded":"2024-08-28T19:21:03.821152+0000","last_fullsized":"2024-08-28T19:21:03.821152+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:50:13.948439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162958+0000","last_change":"2024-08-28T19:18:39.524190+0000","last_active":"2024-08-28T19:20:09.162958+0000","last_peered":"2024-08-28T19:20:09.162958+0000","last_clean":"2024-08-28T19:20:09.162958+0000","last_became_active":"2024-08-28T19:18:39.523771+0000","last_became_peered":"2024-08-28T19:18:39.523771+0000","last_unstale":"2024-08-28T19:20:09.162958+0000","last_undegraded":"2024-08-28T19:20:09.162958+0000","last_fullsized":"2024-08-28T19:20:09.162958+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:43:43.984625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.874995+0000","last_change":"2024-08-28T19:18:32.487212+0000","last_active":"2024-08-28T19:20:37.874995+0000","last_peered":"2024-08-28T19:20:37.874995+0000","last_clean":"2024-08-28T19:20:37.874995+0000","last_became_active":"2024-08-28T19:18:32.485432+0000","last_became_peered":"2024-08-28T19:18:32.485432+0000","last_unstale":"2024-08-28T19:20:37.874995+0000","last_undegraded":"2024-08-28T19:20:37.874995+0000","last_fullsized":"2024-08-28T19:20:37.874995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:52:11.672193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":159,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:03.820333+0000","last_change":"2024-08-28T19:18:31.082210+0000","last_active":"2024-08-28T19:21:03.820333+0000","last_peered":"2024-08-28T19:21:03.820333+0000","last_clean":"2024-08-28T19:21:03.820333+0000","last_became_active":"2024-08-28T19:18:31.081814+0000","last_became_peered":"2024-08-28T19:18:31.081814+0000","last_unstale":"2024-08-28T19:21:03.820333+0000","last_undegraded":"2024-08-28T19:21:03.820333+0000","last_fullsized":"2024-08-28T19:21:03.820333+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:33:32.281431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163227+0000","last_change":"2024-08-28T19:18:39.681858+0000","last_active":"2024-08-28T19:20:09.163227+0000","last_peered":"2024-08-28T19:20:09.163227+0000","last_clean":"2024-08-28T19:20:09.163227+0000","last_became_active":"2024-08-28T19:18:39.681382+0000","last_became_peered":"2024-08-28T19:18:39.681382+0000","last_unstale":"2024-08-28T19:20:09.163227+0000","last_undegraded":"2024-08-28T19:20:09.163227+0000","last_fullsized":"2024-08-28T19:20:09.163227+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:23:22.021022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.857639+0000","last_change":"2024-08-28T19:18:32.486802+0000","last_active":"2024-08-28T19:20:37.857639+0000","last_peered":"2024-08-28T19:20:37.857639+0000","last_clean":"2024-08-28T19:20:37.857639+0000","last_became_active":"2024-08-28T19:18:32.486479+0000","last_became_peered":"2024-08-28T19:18:32.486479+0000","last_unstale":"2024-08-28T19:20:37.857639+0000","last_undegraded":"2024-08-28T19:20:37.857639+0000","last_fullsized":"2024-08-28T19:20:37.857639+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:17:17.654757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529084+0000","last_change":"2024-08-28T19:18:30.470447+0000","last_active":"2024-08-28T19:20:09.529084+0000","last_peered":"2024-08-28T19:20:09.529084+0000","last_clean":"2024-08-28T19:20:09.529084+0000","last_became_active":"2024-08-28T19:18:30.469862+0000","last_became_peered":"2024-08-28T19:18:30.469862+0000","last_unstale":"2024-08-28T19:20:09.529084+0000","last_undegraded":"2024-08-28T19:20:09.529084+0000","last_fullsized":"2024-08-28T19:20:09.529084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:27:46.660492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009270+0000","last_change":"2024-08-28T19:19:08.624635+0000","last_active":"2024-08-28T19:20:09.009270+0000","last_peered":"2024-08-28T19:20:09.009270+0000","last_clean":"2024-08-28T19:20:09.009270+0000","last_became_active":"2024-08-28T19:19:08.622845+0000","last_became_peered":"2024-08-28T19:19:08.622845+0000","last_unstale":"2024-08-28T19:20:09.009270+0000","last_undegraded":"2024-08-28T19:20:09.009270+0000","last_fullsized":"2024-08-28T19:20:09.009270+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:56:40.137680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865852+0000","last_change":"2024-08-28T19:18:32.487947+0000","last_active":"2024-08-28T19:20:37.865852+0000","last_peered":"2024-08-28T19:20:37.865852+0000","last_clean":"2024-08-28T19:20:37.865852+0000","last_became_active":"2024-08-28T19:18:32.487592+0000","last_became_peered":"2024-08-28T19:18:32.487592+0000","last_unstale":"2024-08-28T19:20:37.865852+0000","last_undegraded":"2024-08-28T19:20:37.865852+0000","last_fullsized":"2024-08-28T19:20:37.865852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:30:29.550644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615576+0000","last_change":"2024-08-28T19:18:30.465044+0000","last_active":"2024-08-28T19:20:09.615576+0000","last_peered":"2024-08-28T19:20:09.615576+0000","last_clean":"2024-08-28T19:20:09.615576+0000","last_became_active":"2024-08-28T19:18:30.464580+0000","last_became_peered":"2024-08-28T19:18:30.464580+0000","last_unstale":"2024-08-28T19:20:09.615576+0000","last_undegraded":"2024-08-28T19:20:09.615576+0000","last_fullsized":"2024-08-28T19:20:09.615576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:21:17.223311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":114,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.528964+0000","last_change":"2024-08-28T19:17:17.063316+0000","last_active":"2024-08-28T19:20:09.528964+0000","last_peered":"2024-08-28T19:20:09.528964+0000","last_clean":"2024-08-28T19:20:09.528964+0000","last_became_active":"2024-08-28T19:17:17.057091+0000","last_became_peered":"2024-08-28T19:17:17.057091+0000","last_unstale":"2024-08-28T19:20:09.528964+0000","last_undegraded":"2024-08-28T19:20:09.528964+0000","last_fullsized":"2024-08-28T19:20:09.528964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:13:56.815467+0000","last_clean_scrub_stamp":"2024-08-28T19:13:56.815467+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:35:51.499703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"82'1","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529202+0000","last_change":"2024-08-28T19:19:08.621402+0000","last_active":"2024-08-28T19:20:09.529202+0000","last_peered":"2024-08-28T19:20:09.529202+0000","last_clean":"2024-08-28T19:20:09.529202+0000","last_became_active":"2024-08-28T19:19:08.620779+0000","last_became_peered":"2024-08-28T19:19:08.620779+0000","last_unstale":"2024-08-28T19:20:09.529202+0000","last_undegraded":"2024-08-28T19:20:09.529202+0000","last_fullsized":"2024-08-28T19:20:09.529202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:39.391414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886997+0000","last_change":"2024-08-28T19:18:32.490066+0000","last_active":"2024-08-28T19:20:37.886997+0000","last_peered":"2024-08-28T19:20:37.886997+0000","last_clean":"2024-08-28T19:20:37.886997+0000","last_became_active":"2024-08-28T19:18:32.489560+0000","last_became_peered":"2024-08-28T19:18:32.489560+0000","last_unstale":"2024-08-28T19:20:37.886997+0000","last_undegraded":"2024-08-28T19:20:37.886997+0000","last_fullsized":"2024-08-28T19:20:37.886997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:22:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'5","reported_seq":160,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:03.865971+0000","last_change":"2024-08-28T19:18:31.083078+0000","last_active":"2024-08-28T19:21:03.865971+0000","last_peered":"2024-08-28T19:21:03.865971+0000","last_clean":"2024-08-28T19:21:03.865971+0000","last_became_active":"2024-08-28T19:18:31.082575+0000","last_became_peered":"2024-08-28T19:18:31.082575+0000","last_unstale":"2024-08-28T19:21:03.865971+0000","last_undegraded":"2024-08-28T19:21:03.865971+0000","last_fullsized":"2024-08-28T19:21:03.865971+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007862+0000","last_change":"2024-08-28T19:18:39.519814+0000","last_active":"2024-08-28T19:20:09.007862+0000","last_peered":"2024-08-28T19:20:09.007862+0000","last_clean":"2024-08-28T19:20:09.007862+0000","last_became_active":"2024-08-28T19:18:39.519258+0000","last_became_peered":"2024-08-28T19:18:39.519258+0000","last_unstale":"2024-08-28T19:20:09.007862+0000","last_undegraded":"2024-08-28T19:20:09.007862+0000","last_fullsized":"2024-08-28T19:20:09.007862+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:03:04.051376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871177+0000","last_change":"2024-08-28T19:18:32.495426+0000","last_active":"2024-08-28T19:20:37.871177+0000","last_peered":"2024-08-28T19:20:37.871177+0000","last_clean":"2024-08-28T19:20:37.871177+0000","last_became_active":"2024-08-28T19:18:32.495075+0000","last_became_peered":"2024-08-28T19:18:32.495075+0000","last_unstale":"2024-08-28T19:20:37.871177+0000","last_undegraded":"2024-08-28T19:20:37.871177+0000","last_fullsized":"2024-08-28T19:20:37.871177+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:40:22.456666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'62","reported_seq":132,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:04.342332+0000","last_change":"2024-08-28T19:18:30.472708+0000","last_active":"2024-08-28T19:21:04.342332+0000","last_peered":"2024-08-28T19:21:04.342332+0000","last_clean":"2024-08-28T19:21:04.342332+0000","last_became_active":"2024-08-28T19:18:30.472320+0000","last_became_peered":"2024-08-28T19:18:30.472320+0000","last_unstale":"2024-08-28T19:21:04.342332+0000","last_undegraded":"2024-08-28T19:21:04.342332+0000","last_fullsized":"2024-08-28T19:21:04.342332+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:57:12.055626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":32,"num_write":71,"num_write_kb":873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010493+0000","last_change":"2024-08-28T19:18:39.519398+0000","last_active":"2024-08-28T19:20:09.010493+0000","last_peered":"2024-08-28T19:20:09.010493+0000","last_clean":"2024-08-28T19:20:09.010493+0000","last_became_active":"2024-08-28T19:18:39.518997+0000","last_became_peered":"2024-08-28T19:18:39.518997+0000","last_unstale":"2024-08-28T19:20:09.010493+0000","last_undegraded":"2024-08-28T19:20:09.010493+0000","last_fullsized":"2024-08-28T19:20:09.010493+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:01:33.431608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.879604+0000","last_change":"2024-08-28T19:18:32.489439+0000","last_active":"2024-08-28T19:20:37.879604+0000","last_peered":"2024-08-28T19:20:37.879604+0000","last_clean":"2024-08-28T19:20:37.879604+0000","last_became_active":"2024-08-28T19:18:32.489054+0000","last_became_peered":"2024-08-28T19:18:32.489054+0000","last_unstale":"2024-08-28T19:20:37.879604+0000","last_undegraded":"2024-08-28T19:20:37.879604+0000","last_fullsized":"2024-08-28T19:20:37.879604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:31:32.647384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163358+0000","last_change":"2024-08-28T19:18:30.464993+0000","last_active":"2024-08-28T19:20:09.163358+0000","last_peered":"2024-08-28T19:20:09.163358+0000","last_clean":"2024-08-28T19:20:09.163358+0000","last_became_active":"2024-08-28T19:18:30.463951+0000","last_became_peered":"2024-08-28T19:18:30.463951+0000","last_unstale":"2024-08-28T19:20:09.163358+0000","last_undegraded":"2024-08-28T19:20:09.163358+0000","last_fullsized":"2024-08-28T19:20:09.163358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:04:06.780058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008234+0000","last_change":"2024-08-28T19:18:39.533061+0000","last_active":"2024-08-28T19:20:09.008234+0000","last_peered":"2024-08-28T19:20:09.008234+0000","last_clean":"2024-08-28T19:20:09.008234+0000","last_became_active":"2024-08-28T19:18:39.532607+0000","last_became_peered":"2024-08-28T19:18:39.532607+0000","last_unstale":"2024-08-28T19:20:09.008234+0000","last_undegraded":"2024-08-28T19:20:09.008234+0000","last_fullsized":"2024-08-28T19:20:09.008234+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:00:57.997665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875549+0000","last_change":"2024-08-28T19:18:32.487937+0000","last_active":"2024-08-28T19:20:37.875549+0000","last_peered":"2024-08-28T19:20:37.875549+0000","last_clean":"2024-08-28T19:20:37.875549+0000","last_became_active":"2024-08-28T19:18:32.487417+0000","last_became_peered":"2024-08-28T19:18:32.487417+0000","last_unstale":"2024-08-28T19:20:37.875549+0000","last_undegraded":"2024-08-28T19:20:37.875549+0000","last_fullsized":"2024-08-28T19:20:37.875549+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:30:42.497364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.110517+0000","last_change":"2024-08-28T19:18:30.470742+0000","last_active":"2024-08-28T19:20:09.110517+0000","last_peered":"2024-08-28T19:20:09.110517+0000","last_clean":"2024-08-28T19:20:09.110517+0000","last_became_active":"2024-08-28T19:18:30.470135+0000","last_became_peered":"2024-08-28T19:18:30.470135+0000","last_unstale":"2024-08-28T19:20:09.110517+0000","last_undegraded":"2024-08-28T19:20:09.110517+0000","last_fullsized":"2024-08-28T19:20:09.110517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:49:39.581119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008429+0000","last_change":"2024-08-28T19:18:39.524052+0000","last_active":"2024-08-28T19:20:09.008429+0000","last_peered":"2024-08-28T19:20:09.008429+0000","last_clean":"2024-08-28T19:20:09.008429+0000","last_became_active":"2024-08-28T19:18:39.523601+0000","last_became_peered":"2024-08-28T19:18:39.523601+0000","last_unstale":"2024-08-28T19:20:09.008429+0000","last_undegraded":"2024-08-28T19:20:09.008429+0000","last_fullsized":"2024-08-28T19:20:09.008429+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:54:53.302424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883838+0000","last_change":"2024-08-28T19:18:32.491151+0000","last_active":"2024-08-28T19:20:37.883838+0000","last_peered":"2024-08-28T19:20:37.883838+0000","last_clean":"2024-08-28T19:20:37.883838+0000","last_became_active":"2024-08-28T19:18:32.490798+0000","last_became_peered":"2024-08-28T19:18:32.490798+0000","last_unstale":"2024-08-28T19:20:37.883838+0000","last_undegraded":"2024-08-28T19:20:37.883838+0000","last_fullsized":"2024-08-28T19:20:37.883838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:41:00.771019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.109743+0000","last_change":"2024-08-28T19:18:31.084029+0000","last_active":"2024-08-28T19:20:09.109743+0000","last_peered":"2024-08-28T19:20:09.109743+0000","last_clean":"2024-08-28T19:20:09.109743+0000","last_became_active":"2024-08-28T19:18:31.083507+0000","last_became_peered":"2024-08-28T19:18:31.083507+0000","last_unstale":"2024-08-28T19:20:09.109743+0000","last_undegraded":"2024-08-28T19:20:09.109743+0000","last_fullsized":"2024-08-28T19:20:09.109743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:15:36.451702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.163318+0000","last_change":"2024-08-28T19:18:39.527249+0000","last_active":"2024-08-28T19:20:09.163318+0000","last_peered":"2024-08-28T19:20:09.163318+0000","last_clean":"2024-08-28T19:20:09.163318+0000","last_became_active":"2024-08-28T19:18:39.526834+0000","last_became_peered":"2024-08-28T19:18:39.526834+0000","last_unstale":"2024-08-28T19:20:09.163318+0000","last_undegraded":"2024-08-28T19:20:09.163318+0000","last_fullsized":"2024-08-28T19:20:09.163318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:48:02.294738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.876604+0000","last_change":"2024-08-28T19:18:32.497337+0000","last_active":"2024-08-28T19:20:37.876604+0000","last_peered":"2024-08-28T19:20:37.876604+0000","last_clean":"2024-08-28T19:20:37.876604+0000","last_became_active":"2024-08-28T19:18:32.496929+0000","last_became_peered":"2024-08-28T19:18:32.496929+0000","last_unstale":"2024-08-28T19:20:37.876604+0000","last_undegraded":"2024-08-28T19:20:37.876604+0000","last_fullsized":"2024-08-28T19:20:37.876604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:23:14.748765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":164,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:07.827169+0000","last_change":"2024-08-28T19:18:30.471353+0000","last_active":"2024-08-28T19:21:07.827169+0000","last_peered":"2024-08-28T19:21:07.827169+0000","last_clean":"2024-08-28T19:21:07.827169+0000","last_became_active":"2024-08-28T19:18:30.470283+0000","last_became_peered":"2024-08-28T19:18:30.470283+0000","last_unstale":"2024-08-28T19:21:07.827169+0000","last_undegraded":"2024-08-28T19:21:07.827169+0000","last_fullsized":"2024-08-28T19:21:07.827169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:09:46.444864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007301+0000","last_change":"2024-08-28T19:18:39.519226+0000","last_active":"2024-08-28T19:20:09.007301+0000","last_peered":"2024-08-28T19:20:09.007301+0000","last_clean":"2024-08-28T19:20:09.007301+0000","last_became_active":"2024-08-28T19:18:39.518738+0000","last_became_peered":"2024-08-28T19:18:39.518738+0000","last_unstale":"2024-08-28T19:20:09.007301+0000","last_undegraded":"2024-08-28T19:20:09.007301+0000","last_fullsized":"2024-08-28T19:20:09.007301+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:39:35.114702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.877692+0000","last_change":"2024-08-28T19:18:32.493060+0000","last_active":"2024-08-28T19:20:37.877692+0000","last_peered":"2024-08-28T19:20:37.877692+0000","last_clean":"2024-08-28T19:20:37.877692+0000","last_became_active":"2024-08-28T19:18:32.492590+0000","last_became_peered":"2024-08-28T19:18:32.492590+0000","last_unstale":"2024-08-28T19:20:37.877692+0000","last_undegraded":"2024-08-28T19:20:37.877692+0000","last_fullsized":"2024-08-28T19:20:37.877692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:02:02.165301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008761+0000","last_change":"2024-08-28T19:18:30.470674+0000","last_active":"2024-08-28T19:20:09.008761+0000","last_peered":"2024-08-28T19:20:09.008761+0000","last_clean":"2024-08-28T19:20:09.008761+0000","last_became_active":"2024-08-28T19:18:30.470275+0000","last_became_peered":"2024-08-28T19:18:30.470275+0000","last_unstale":"2024-08-28T19:20:09.008761+0000","last_undegraded":"2024-08-28T19:20:09.008761+0000","last_fullsized":"2024-08-28T19:20:09.008761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:01:48.547315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663581+0000","last_change":"2024-08-28T19:19:08.624194+0000","last_active":"2024-08-28T19:20:09.663581+0000","last_peered":"2024-08-28T19:20:09.663581+0000","last_clean":"2024-08-28T19:20:09.663581+0000","last_became_active":"2024-08-28T19:19:08.623681+0000","last_became_peered":"2024-08-28T19:19:08.623681+0000","last_unstale":"2024-08-28T19:20:09.663581+0000","last_undegraded":"2024-08-28T19:20:09.663581+0000","last_fullsized":"2024-08-28T19:20:09.663581+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:21:11.099931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872115+0000","last_change":"2024-08-28T19:18:32.488210+0000","last_active":"2024-08-28T19:20:37.872115+0000","last_peered":"2024-08-28T19:20:37.872115+0000","last_clean":"2024-08-28T19:20:37.872115+0000","last_became_active":"2024-08-28T19:18:32.487853+0000","last_became_peered":"2024-08-28T19:18:32.487853+0000","last_unstale":"2024-08-28T19:20:37.872115+0000","last_undegraded":"2024-08-28T19:20:37.872115+0000","last_fullsized":"2024-08-28T19:20:37.872115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:20:51.808958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011302+0000","last_change":"2024-08-28T19:18:30.465306+0000","last_active":"2024-08-28T19:20:09.011302+0000","last_peered":"2024-08-28T19:20:09.011302+0000","last_clean":"2024-08-28T19:20:09.011302+0000","last_became_active":"2024-08-28T19:18:30.464850+0000","last_became_peered":"2024-08-28T19:18:30.464850+0000","last_unstale":"2024-08-28T19:20:09.011302+0000","last_undegraded":"2024-08-28T19:20:09.011302+0000","last_fullsized":"2024-08-28T19:20:09.011302+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:24:26.628999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008722+0000","last_change":"2024-08-28T19:18:39.524005+0000","last_active":"2024-08-28T19:20:09.008722+0000","last_peered":"2024-08-28T19:20:09.008722+0000","last_clean":"2024-08-28T19:20:09.008722+0000","last_became_active":"2024-08-28T19:18:39.523594+0000","last_became_peered":"2024-08-28T19:18:39.523594+0000","last_unstale":"2024-08-28T19:20:09.008722+0000","last_undegraded":"2024-08-28T19:20:09.008722+0000","last_fullsized":"2024-08-28T19:20:09.008722+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:35:32.484399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872816+0000","last_change":"2024-08-28T19:18:32.489088+0000","last_active":"2024-08-28T19:20:37.872816+0000","last_peered":"2024-08-28T19:20:37.872816+0000","last_clean":"2024-08-28T19:20:37.872816+0000","last_became_active":"2024-08-28T19:18:32.488717+0000","last_became_peered":"2024-08-28T19:18:32.488717+0000","last_unstale":"2024-08-28T19:20:37.872816+0000","last_undegraded":"2024-08-28T19:20:37.872816+0000","last_fullsized":"2024-08-28T19:20:37.872816+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:25.468919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"84'41","reported_seq":320,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:04.822762+0000","last_change":"2024-08-28T19:18:30.468674+0000","last_active":"2024-08-28T19:21:04.822762+0000","last_peered":"2024-08-28T19:21:04.822762+0000","last_clean":"2024-08-28T19:21:04.822762+0000","last_became_active":"2024-08-28T19:18:30.468300+0000","last_became_peered":"2024-08-28T19:18:30.468300+0000","last_unstale":"2024-08-28T19:21:04.822762+0000","last_undegraded":"2024-08-28T19:21:04.822762+0000","last_fullsized":"2024-08-28T19:21:04.822762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:12:06.237498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69405,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":239,"num_read_kb":188,"num_write":43,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.007140+0000","last_change":"2024-08-28T19:18:39.526131+0000","last_active":"2024-08-28T19:20:09.007140+0000","last_peered":"2024-08-28T19:20:09.007140+0000","last_clean":"2024-08-28T19:20:09.007140+0000","last_became_active":"2024-08-28T19:18:39.525708+0000","last_became_peered":"2024-08-28T19:18:39.525708+0000","last_unstale":"2024-08-28T19:20:09.007140+0000","last_undegraded":"2024-08-28T19:20:09.007140+0000","last_fullsized":"2024-08-28T19:20:09.007140+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:21:52.156487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.885078+0000","last_change":"2024-08-28T19:18:32.487634+0000","last_active":"2024-08-28T19:20:37.885078+0000","last_peered":"2024-08-28T19:20:37.885078+0000","last_clean":"2024-08-28T19:20:37.885078+0000","last_became_active":"2024-08-28T19:18:32.486702+0000","last_became_peered":"2024-08-28T19:18:32.486702+0000","last_unstale":"2024-08-28T19:20:37.885078+0000","last_undegraded":"2024-08-28T19:20:37.885078+0000","last_fullsized":"2024-08-28T19:20:37.885078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:08:12.606905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.009190+0000","last_change":"2024-08-28T19:18:31.083494+0000","last_active":"2024-08-28T19:20:09.009190+0000","last_peered":"2024-08-28T19:20:09.009190+0000","last_clean":"2024-08-28T19:20:09.009190+0000","last_became_active":"2024-08-28T19:18:31.083086+0000","last_became_peered":"2024-08-28T19:18:31.083086+0000","last_unstale":"2024-08-28T19:20:09.009190+0000","last_undegraded":"2024-08-28T19:20:09.009190+0000","last_fullsized":"2024-08-28T19:20:09.009190+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:14:54.388364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.615438+0000","last_change":"2024-08-28T19:18:39.532155+0000","last_active":"2024-08-28T19:20:09.615438+0000","last_peered":"2024-08-28T19:20:09.615438+0000","last_clean":"2024-08-28T19:20:09.615438+0000","last_became_active":"2024-08-28T19:18:39.531565+0000","last_became_peered":"2024-08-28T19:18:39.531565+0000","last_unstale":"2024-08-28T19:20:09.615438+0000","last_undegraded":"2024-08-28T19:20:09.615438+0000","last_fullsized":"2024-08-28T19:20:09.615438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:29:48.824622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.881019+0000","last_change":"2024-08-28T19:18:32.496296+0000","last_active":"2024-08-28T19:20:37.881019+0000","last_peered":"2024-08-28T19:20:37.881019+0000","last_clean":"2024-08-28T19:20:37.881019+0000","last_became_active":"2024-08-28T19:18:32.495680+0000","last_became_peered":"2024-08-28T19:18:32.495680+0000","last_unstale":"2024-08-28T19:20:37.881019+0000","last_undegraded":"2024-08-28T19:20:37.881019+0000","last_fullsized":"2024-08-28T19:20:37.881019+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:27:34.544529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.529311+0000","last_change":"2024-08-28T19:19:08.631758+0000","last_active":"2024-08-28T19:20:09.529311+0000","last_peered":"2024-08-28T19:20:09.529311+0000","last_clean":"2024-08-28T19:20:09.529311+0000","last_became_active":"2024-08-28T19:19:08.631202+0000","last_became_peered":"2024-08-28T19:19:08.631202+0000","last_unstale":"2024-08-28T19:20:09.529311+0000","last_undegraded":"2024-08-28T19:20:09.529311+0000","last_fullsized":"2024-08-28T19:20:09.529311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:27:43.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.006784+0000","last_change":"2024-08-28T19:18:39.531157+0000","last_active":"2024-08-28T19:20:09.006784+0000","last_peered":"2024-08-28T19:20:09.006784+0000","last_clean":"2024-08-28T19:20:09.006784+0000","last_became_active":"2024-08-28T19:18:39.530675+0000","last_became_peered":"2024-08-28T19:18:39.530675+0000","last_unstale":"2024-08-28T19:20:09.006784+0000","last_undegraded":"2024-08-28T19:20:09.006784+0000","last_fullsized":"2024-08-28T19:20:09.006784+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:38:40.414854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883692+0000","last_change":"2024-08-28T19:18:32.482182+0000","last_active":"2024-08-28T19:20:37.883692+0000","last_peered":"2024-08-28T19:20:37.883692+0000","last_clean":"2024-08-28T19:20:37.883692+0000","last_became_active":"2024-08-28T19:18:32.481689+0000","last_became_peered":"2024-08-28T19:18:32.481689+0000","last_unstale":"2024-08-28T19:20:37.883692+0000","last_undegraded":"2024-08-28T19:20:37.883692+0000","last_fullsized":"2024-08-28T19:20:37.883692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:13:29.278469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'5","reported_seq":271,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:04.822781+0000","last_change":"2024-08-28T19:18:30.475244+0000","last_active":"2024-08-28T19:21:04.822781+0000","last_peered":"2024-08-28T19:21:04.822781+0000","last_clean":"2024-08-28T19:21:04.822781+0000","last_became_active":"2024-08-28T19:18:30.474861+0000","last_became_peered":"2024-08-28T19:18:30.474861+0000","last_unstale":"2024-08-28T19:21:04.822781+0000","last_undegraded":"2024-08-28T19:21:04.822781+0000","last_fullsized":"2024-08-28T19:21:04.822781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:27:09.076440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.525108+0000","last_change":"2024-08-28T19:18:39.520022+0000","last_active":"2024-08-28T19:20:09.525108+0000","last_peered":"2024-08-28T19:20:09.525108+0000","last_clean":"2024-08-28T19:20:09.525108+0000","last_became_active":"2024-08-28T19:18:39.519584+0000","last_became_peered":"2024-08-28T19:18:39.519584+0000","last_unstale":"2024-08-28T19:20:09.525108+0000","last_undegraded":"2024-08-28T19:20:09.525108+0000","last_fullsized":"2024-08-28T19:20:09.525108+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:27:15.851303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010363+0000","last_change":"2024-08-28T19:18:39.522607+0000","last_active":"2024-08-28T19:20:09.010363+0000","last_peered":"2024-08-28T19:20:09.010363+0000","last_clean":"2024-08-28T19:20:09.010363+0000","last_became_active":"2024-08-28T19:18:39.522204+0000","last_became_peered":"2024-08-28T19:18:39.522204+0000","last_unstale":"2024-08-28T19:20:09.010363+0000","last_undegraded":"2024-08-28T19:20:09.010363+0000","last_fullsized":"2024-08-28T19:20:09.010363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:02:08.865294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865617+0000","last_change":"2024-08-28T19:18:32.485577+0000","last_active":"2024-08-28T19:20:37.865617+0000","last_peered":"2024-08-28T19:20:37.865617+0000","last_clean":"2024-08-28T19:20:37.865617+0000","last_became_active":"2024-08-28T19:18:32.485243+0000","last_became_peered":"2024-08-28T19:18:32.485243+0000","last_unstale":"2024-08-28T19:20:37.865617+0000","last_undegraded":"2024-08-28T19:20:37.865617+0000","last_fullsized":"2024-08-28T19:20:37.865617+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:05:53.019365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.008799+0000","last_change":"2024-08-28T19:18:30.470670+0000","last_active":"2024-08-28T19:20:09.008799+0000","last_peered":"2024-08-28T19:20:09.008799+0000","last_clean":"2024-08-28T19:20:09.008799+0000","last_became_active":"2024-08-28T19:18:30.470264+0000","last_became_peered":"2024-08-28T19:18:30.470264+0000","last_unstale":"2024-08-28T19:20:09.008799+0000","last_undegraded":"2024-08-28T19:20:09.008799+0000","last_fullsized":"2024-08-28T19:20:09.008799+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:35.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663847+0000","last_change":"2024-08-28T19:18:39.530948+0000","last_active":"2024-08-28T19:20:09.663847+0000","last_peered":"2024-08-28T19:20:09.663847+0000","last_clean":"2024-08-28T19:20:09.663847+0000","last_became_active":"2024-08-28T19:18:39.530473+0000","last_became_peered":"2024-08-28T19:18:39.530473+0000","last_unstale":"2024-08-28T19:20:09.663847+0000","last_undegraded":"2024-08-28T19:20:09.663847+0000","last_fullsized":"2024-08-28T19:20:09.663847+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:21:57.504582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.869892+0000","last_change":"2024-08-28T19:18:32.486027+0000","last_active":"2024-08-28T19:20:37.869892+0000","last_peered":"2024-08-28T19:20:37.869892+0000","last_clean":"2024-08-28T19:20:37.869892+0000","last_became_active":"2024-08-28T19:18:32.483273+0000","last_became_peered":"2024-08-28T19:18:32.483273+0000","last_unstale":"2024-08-28T19:20:37.869892+0000","last_undegraded":"2024-08-28T19:20:37.869892+0000","last_fullsized":"2024-08-28T19:20:37.869892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:47:37.680695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.162978+0000","last_change":"2024-08-28T19:18:30.464811+0000","last_active":"2024-08-28T19:20:09.162978+0000","last_peered":"2024-08-28T19:20:09.162978+0000","last_clean":"2024-08-28T19:20:09.162978+0000","last_became_active":"2024-08-28T19:18:30.463986+0000","last_became_peered":"2024-08-28T19:18:30.463986+0000","last_unstale":"2024-08-28T19:20:09.162978+0000","last_undegraded":"2024-08-28T19:20:09.162978+0000","last_fullsized":"2024-08-28T19:20:09.162978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:43:46.582993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.663619+0000","last_change":"2024-08-28T19:19:08.631382+0000","last_active":"2024-08-28T19:20:09.663619+0000","last_peered":"2024-08-28T19:20:09.663619+0000","last_clean":"2024-08-28T19:20:09.663619+0000","last_became_active":"2024-08-28T19:19:08.630869+0000","last_became_peered":"2024-08-28T19:19:08.630869+0000","last_unstale":"2024-08-28T19:20:09.663619+0000","last_undegraded":"2024-08-28T19:20:09.663619+0000","last_fullsized":"2024-08-28T19:20:09.663619+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:32:54.942408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875456+0000","last_change":"2024-08-28T19:18:32.484120+0000","last_active":"2024-08-28T19:20:37.875456+0000","last_peered":"2024-08-28T19:20:37.875456+0000","last_clean":"2024-08-28T19:20:37.875456+0000","last_became_active":"2024-08-28T19:18:32.483013+0000","last_became_peered":"2024-08-28T19:18:32.483013+0000","last_unstale":"2024-08-28T19:20:37.875456+0000","last_undegraded":"2024-08-28T19:20:37.875456+0000","last_fullsized":"2024-08-28T19:20:37.875456+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:59:41.195873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.010737+0000","last_change":"2024-08-28T19:18:39.531046+0000","last_active":"2024-08-28T19:20:09.010737+0000","last_peered":"2024-08-28T19:20:09.010737+0000","last_clean":"2024-08-28T19:20:09.010737+0000","last_became_active":"2024-08-28T19:18:39.530407+0000","last_became_peered":"2024-08-28T19:18:39.530407+0000","last_unstale":"2024-08-28T19:20:09.010737+0000","last_undegraded":"2024-08-28T19:20:09.010737+0000","last_fullsized":"2024-08-28T19:20:09.010737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:47:15.173021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'7","reported_seq":167,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:21:06.826273+0000","last_change":"2024-08-28T19:18:30.471268+0000","last_active":"2024-08-28T19:21:06.826273+0000","last_peered":"2024-08-28T19:21:06.826273+0000","last_clean":"2024-08-28T19:21:06.826273+0000","last_became_active":"2024-08-28T19:18:30.470845+0000","last_became_peered":"2024-08-28T19:18:30.470845+0000","last_unstale":"2024-08-28T19:21:06.826273+0000","last_undegraded":"2024-08-28T19:21:06.826273+0000","last_fullsized":"2024-08-28T19:21:06.826273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:29.441530+0000","last_clean_scrub_stamp":"2024-08-28T19:18:29.441530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:18:18.086205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.860520+0000","last_change":"2024-08-28T19:18:32.479728+0000","last_active":"2024-08-28T19:20:37.860520+0000","last_peered":"2024-08-28T19:20:37.860520+0000","last_clean":"2024-08-28T19:20:37.860520+0000","last_became_active":"2024-08-28T19:18:32.478623+0000","last_became_peered":"2024-08-28T19:18:32.478623+0000","last_unstale":"2024-08-28T19:20:37.860520+0000","last_undegraded":"2024-08-28T19:20:37.860520+0000","last_fullsized":"2024-08-28T19:20:37.860520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T00:14:49.914808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:09.011224+0000","last_change":"2024-08-28T19:18:39.531308+0000","last_active":"2024-08-28T19:20:09.011224+0000","last_peered":"2024-08-28T19:20:09.011224+0000","last_clean":"2024-08-28T19:20:09.011224+0000","last_became_active":"2024-08-28T19:18:39.530403+0000","last_became_peered":"2024-08-28T19:18:39.530403+0000","last_unstale":"2024-08-28T19:20:09.011224+0000","last_undegraded":"2024-08-28T19:20:09.011224+0000","last_fullsized":"2024-08-28T19:20:09.011224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:38.493204+0000","last_clean_scrub_stamp":"2024-08-28T19:18:38.493204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:33.139668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873003+0000","last_change":"2024-08-28T19:18:32.494621+0000","last_active":"2024-08-28T19:20:37.873003+0000","last_peered":"2024-08-28T19:20:37.873003+0000","last_clean":"2024-08-28T19:20:37.873003+0000","last_became_active":"2024-08-28T19:18:32.494217+0000","last_became_peered":"2024-08-28T19:18:32.494217+0000","last_unstale":"2024-08-28T19:20:37.873003+0000","last_undegraded":"2024-08-28T19:20:37.873003+0000","last_fullsized":"2024-08-28T19:20:37.873003+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:54:15.419470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"84'12","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872043+0000","last_change":"2024-08-28T19:20:09.616552+0000","last_active":"2024-08-28T19:20:37.872043+0000","last_peered":"2024-08-28T19:20:37.872043+0000","last_clean":"2024-08-28T19:20:37.872043+0000","last_became_active":"2024-08-28T19:20:09.616159+0000","last_became_peered":"2024-08-28T19:20:09.616159+0000","last_unstale":"2024-08-28T19:20:37.872043+0000","last_undegraded":"2024-08-28T19:20:37.872043+0000","last_fullsized":"2024-08-28T19:20:37.872043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:04:10.354703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"84'8","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866959+0000","last_change":"2024-08-28T19:20:09.012824+0000","last_active":"2024-08-28T19:20:37.866959+0000","last_peered":"2024-08-28T19:20:37.866959+0000","last_clean":"2024-08-28T19:20:37.866959+0000","last_became_active":"2024-08-28T19:20:09.012490+0000","last_became_peered":"2024-08-28T19:20:09.012490+0000","last_unstale":"2024-08-28T19:20:37.866959+0000","last_undegraded":"2024-08-28T19:20:37.866959+0000","last_fullsized":"2024-08-28T19:20:37.866959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:49:02.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871666+0000","last_change":"2024-08-28T19:20:09.164523+0000","last_active":"2024-08-28T19:20:37.871666+0000","last_peered":"2024-08-28T19:20:37.871666+0000","last_clean":"2024-08-28T19:20:37.871666+0000","last_became_active":"2024-08-28T19:20:09.163825+0000","last_became_peered":"2024-08-28T19:20:09.163825+0000","last_unstale":"2024-08-28T19:20:37.871666+0000","last_undegraded":"2024-08-28T19:20:37.871666+0000","last_fullsized":"2024-08-28T19:20:37.871666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:42:48.141103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868174+0000","last_change":"2024-08-28T19:18:32.485446+0000","last_active":"2024-08-28T19:20:37.868174+0000","last_peered":"2024-08-28T19:20:37.868174+0000","last_clean":"2024-08-28T19:20:37.868174+0000","last_became_active":"2024-08-28T19:18:32.484197+0000","last_became_peered":"2024-08-28T19:18:32.484197+0000","last_unstale":"2024-08-28T19:20:37.868174+0000","last_undegraded":"2024-08-28T19:20:37.868174+0000","last_fullsized":"2024-08-28T19:20:37.868174+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:47:50.527915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868339+0000","last_change":"2024-08-28T19:18:32.489452+0000","last_active":"2024-08-28T19:20:37.868339+0000","last_peered":"2024-08-28T19:20:37.868339+0000","last_clean":"2024-08-28T19:20:37.868339+0000","last_became_active":"2024-08-28T19:18:32.487482+0000","last_became_peered":"2024-08-28T19:18:32.487482+0000","last_unstale":"2024-08-28T19:20:37.868339+0000","last_undegraded":"2024-08-28T19:20:37.868339+0000","last_fullsized":"2024-08-28T19:20:37.868339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:52:08.175080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"84'15","reported_seq":50,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884068+0000","last_change":"2024-08-28T19:18:32.483850+0000","last_active":"2024-08-28T19:20:37.884068+0000","last_peered":"2024-08-28T19:20:37.884068+0000","last_clean":"2024-08-28T19:20:37.884068+0000","last_became_active":"2024-08-28T19:18:32.483476+0000","last_became_peered":"2024-08-28T19:18:32.483476+0000","last_unstale":"2024-08-28T19:20:37.884068+0000","last_undegraded":"2024-08-28T19:20:37.884068+0000","last_fullsized":"2024-08-28T19:20:37.884068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:32:37.465218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.868836+0000","last_change":"2024-08-28T19:18:32.492252+0000","last_active":"2024-08-28T19:20:37.868836+0000","last_peered":"2024-08-28T19:20:37.868836+0000","last_clean":"2024-08-28T19:20:37.868836+0000","last_became_active":"2024-08-28T19:18:32.491659+0000","last_became_peered":"2024-08-28T19:18:32.491659+0000","last_unstale":"2024-08-28T19:20:37.868836+0000","last_undegraded":"2024-08-28T19:20:37.868836+0000","last_fullsized":"2024-08-28T19:20:37.868836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:19:28.901559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865090+0000","last_change":"2024-08-28T19:18:32.491940+0000","last_active":"2024-08-28T19:20:37.865090+0000","last_peered":"2024-08-28T19:20:37.865090+0000","last_clean":"2024-08-28T19:20:37.865090+0000","last_became_active":"2024-08-28T19:18:32.487474+0000","last_became_peered":"2024-08-28T19:18:32.487474+0000","last_unstale":"2024-08-28T19:20:37.865090+0000","last_undegraded":"2024-08-28T19:20:37.865090+0000","last_fullsized":"2024-08-28T19:20:37.865090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T07:04:32.508620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.871438+0000","last_change":"2024-08-28T19:18:32.485200+0000","last_active":"2024-08-28T19:20:37.871438+0000","last_peered":"2024-08-28T19:20:37.871438+0000","last_clean":"2024-08-28T19:20:37.871438+0000","last_became_active":"2024-08-28T19:18:32.484858+0000","last_became_peered":"2024-08-28T19:18:32.484858+0000","last_unstale":"2024-08-28T19:20:37.871438+0000","last_undegraded":"2024-08-28T19:20:37.871438+0000","last_fullsized":"2024-08-28T19:20:37.871438+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:10:57.925659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886493+0000","last_change":"2024-08-28T19:18:32.489903+0000","last_active":"2024-08-28T19:20:37.886493+0000","last_peered":"2024-08-28T19:20:37.886493+0000","last_clean":"2024-08-28T19:20:37.886493+0000","last_became_active":"2024-08-28T19:18:32.488950+0000","last_became_peered":"2024-08-28T19:18:32.488950+0000","last_unstale":"2024-08-28T19:20:37.886493+0000","last_undegraded":"2024-08-28T19:20:37.886493+0000","last_fullsized":"2024-08-28T19:20:37.886493+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:06:42.284177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.872318+0000","last_change":"2024-08-28T19:18:32.488236+0000","last_active":"2024-08-28T19:20:37.872318+0000","last_peered":"2024-08-28T19:20:37.872318+0000","last_clean":"2024-08-28T19:20:37.872318+0000","last_became_active":"2024-08-28T19:18:32.487776+0000","last_became_peered":"2024-08-28T19:18:32.487776+0000","last_unstale":"2024-08-28T19:20:37.872318+0000","last_undegraded":"2024-08-28T19:20:37.872318+0000","last_fullsized":"2024-08-28T19:20:37.872318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:25:58.754740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875857+0000","last_change":"2024-08-28T19:18:32.485860+0000","last_active":"2024-08-28T19:20:37.875857+0000","last_peered":"2024-08-28T19:20:37.875857+0000","last_clean":"2024-08-28T19:20:37.875857+0000","last_became_active":"2024-08-28T19:18:32.485415+0000","last_became_peered":"2024-08-28T19:18:32.485415+0000","last_unstale":"2024-08-28T19:20:37.875857+0000","last_undegraded":"2024-08-28T19:20:37.875857+0000","last_fullsized":"2024-08-28T19:20:37.875857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:45:32.868651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873054+0000","last_change":"2024-08-28T19:18:32.485515+0000","last_active":"2024-08-28T19:20:37.873054+0000","last_peered":"2024-08-28T19:20:37.873054+0000","last_clean":"2024-08-28T19:20:37.873054+0000","last_became_active":"2024-08-28T19:18:32.485075+0000","last_became_peered":"2024-08-28T19:18:32.485075+0000","last_unstale":"2024-08-28T19:20:37.873054+0000","last_undegraded":"2024-08-28T19:20:37.873054+0000","last_fullsized":"2024-08-28T19:20:37.873054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:01:28.035479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864600+0000","last_change":"2024-08-28T19:18:32.484685+0000","last_active":"2024-08-28T19:20:37.864600+0000","last_peered":"2024-08-28T19:20:37.864600+0000","last_clean":"2024-08-28T19:20:37.864600+0000","last_became_active":"2024-08-28T19:18:32.484095+0000","last_became_peered":"2024-08-28T19:18:32.484095+0000","last_unstale":"2024-08-28T19:20:37.864600+0000","last_undegraded":"2024-08-28T19:20:37.864600+0000","last_fullsized":"2024-08-28T19:20:37.864600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:53:27.229186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884050+0000","last_change":"2024-08-28T19:18:32.488030+0000","last_active":"2024-08-28T19:20:37.884050+0000","last_peered":"2024-08-28T19:20:37.884050+0000","last_clean":"2024-08-28T19:20:37.884050+0000","last_became_active":"2024-08-28T19:18:32.487042+0000","last_became_peered":"2024-08-28T19:18:32.487042+0000","last_unstale":"2024-08-28T19:20:37.884050+0000","last_undegraded":"2024-08-28T19:20:37.884050+0000","last_fullsized":"2024-08-28T19:20:37.884050+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:13:56.966475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886125+0000","last_change":"2024-08-28T19:18:32.489442+0000","last_active":"2024-08-28T19:20:37.886125+0000","last_peered":"2024-08-28T19:20:37.886125+0000","last_clean":"2024-08-28T19:20:37.886125+0000","last_became_active":"2024-08-28T19:18:32.488405+0000","last_became_peered":"2024-08-28T19:18:32.488405+0000","last_unstale":"2024-08-28T19:20:37.886125+0000","last_undegraded":"2024-08-28T19:20:37.886125+0000","last_fullsized":"2024-08-28T19:20:37.886125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T20:21:03.026893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"84'7","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870138+0000","last_change":"2024-08-28T19:20:09.164305+0000","last_active":"2024-08-28T19:20:37.870138+0000","last_peered":"2024-08-28T19:20:37.870138+0000","last_clean":"2024-08-28T19:20:37.870138+0000","last_became_active":"2024-08-28T19:20:09.163834+0000","last_became_peered":"2024-08-28T19:20:09.163834+0000","last_unstale":"2024-08-28T19:20:37.870138+0000","last_undegraded":"2024-08-28T19:20:37.870138+0000","last_fullsized":"2024-08-28T19:20:37.870138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:20:10.137952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870501+0000","last_change":"2024-08-28T19:18:32.492077+0000","last_active":"2024-08-28T19:20:37.870501+0000","last_peered":"2024-08-28T19:20:37.870501+0000","last_clean":"2024-08-28T19:20:37.870501+0000","last_became_active":"2024-08-28T19:18:32.491722+0000","last_became_peered":"2024-08-28T19:18:32.491722+0000","last_unstale":"2024-08-28T19:20:37.870501+0000","last_undegraded":"2024-08-28T19:20:37.870501+0000","last_fullsized":"2024-08-28T19:20:37.870501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:16:21.697045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"84'10","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.865541+0000","last_change":"2024-08-28T19:18:32.479845+0000","last_active":"2024-08-28T19:20:37.865541+0000","last_peered":"2024-08-28T19:20:37.865541+0000","last_clean":"2024-08-28T19:20:37.865541+0000","last_became_active":"2024-08-28T19:18:32.477690+0000","last_became_peered":"2024-08-28T19:18:32.477690+0000","last_unstale":"2024-08-28T19:20:37.865541+0000","last_undegraded":"2024-08-28T19:20:37.865541+0000","last_fullsized":"2024-08-28T19:20:37.865541+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:11.281415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.866009+0000","last_change":"2024-08-28T19:19:08.624185+0000","last_active":"2024-08-28T19:20:37.866009+0000","last_peered":"2024-08-28T19:20:37.866009+0000","last_clean":"2024-08-28T19:20:37.866009+0000","last_became_active":"2024-08-28T19:19:08.623745+0000","last_became_peered":"2024-08-28T19:19:08.623745+0000","last_unstale":"2024-08-28T19:20:37.866009+0000","last_undegraded":"2024-08-28T19:20:37.866009+0000","last_fullsized":"2024-08-28T19:20:37.866009+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:56:56.657173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.887026+0000","last_change":"2024-08-28T19:18:32.499038+0000","last_active":"2024-08-28T19:20:37.887026+0000","last_peered":"2024-08-28T19:20:37.887026+0000","last_clean":"2024-08-28T19:20:37.887026+0000","last_became_active":"2024-08-28T19:18:32.498276+0000","last_became_peered":"2024-08-28T19:18:32.498276+0000","last_unstale":"2024-08-28T19:20:37.887026+0000","last_undegraded":"2024-08-28T19:20:37.887026+0000","last_fullsized":"2024-08-28T19:20:37.887026+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T21:17:45.468448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"84'16","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.882492+0000","last_change":"2024-08-28T19:18:32.492183+0000","last_active":"2024-08-28T19:20:37.882492+0000","last_peered":"2024-08-28T19:20:37.882492+0000","last_clean":"2024-08-28T19:20:37.882492+0000","last_became_active":"2024-08-28T19:18:32.491815+0000","last_became_peered":"2024-08-28T19:18:32.491815+0000","last_unstale":"2024-08-28T19:20:37.882492+0000","last_undegraded":"2024-08-28T19:20:37.882492+0000","last_fullsized":"2024-08-28T19:20:37.882492+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T04:13:11.183955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"84'11","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.886985+0000","last_change":"2024-08-28T19:18:32.497399+0000","last_active":"2024-08-28T19:20:37.886985+0000","last_peered":"2024-08-28T19:20:37.886985+0000","last_clean":"2024-08-28T19:20:37.886985+0000","last_became_active":"2024-08-28T19:18:32.496901+0000","last_became_peered":"2024-08-28T19:18:32.496901+0000","last_unstale":"2024-08-28T19:20:37.886985+0000","last_undegraded":"2024-08-28T19:20:37.886985+0000","last_fullsized":"2024-08-28T19:20:37.886985+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T02:05:57.634590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"84'9","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.864593+0000","last_change":"2024-08-28T19:18:32.495871+0000","last_active":"2024-08-28T19:20:37.864593+0000","last_peered":"2024-08-28T19:20:37.864593+0000","last_clean":"2024-08-28T19:20:37.864593+0000","last_became_active":"2024-08-28T19:18:32.495515+0000","last_became_peered":"2024-08-28T19:18:32.495515+0000","last_unstale":"2024-08-28T19:20:37.864593+0000","last_undegraded":"2024-08-28T19:20:37.864593+0000","last_fullsized":"2024-08-28T19:20:37.864593+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T23:53:56.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.875472+0000","last_change":"2024-08-28T19:18:32.495711+0000","last_active":"2024-08-28T19:20:37.875472+0000","last_peered":"2024-08-28T19:20:37.875472+0000","last_clean":"2024-08-28T19:20:37.875472+0000","last_became_active":"2024-08-28T19:18:32.495271+0000","last_became_peered":"2024-08-28T19:18:32.495271+0000","last_unstale":"2024-08-28T19:20:37.875472+0000","last_undegraded":"2024-08-28T19:20:37.875472+0000","last_fullsized":"2024-08-28T19:20:37.875472+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:49:05.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"84'6","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.880842+0000","last_change":"2024-08-28T19:18:32.489731+0000","last_active":"2024-08-28T19:20:37.880842+0000","last_peered":"2024-08-28T19:20:37.880842+0000","last_clean":"2024-08-28T19:20:37.880842+0000","last_became_active":"2024-08-28T19:18:32.489366+0000","last_became_peered":"2024-08-28T19:18:32.489366+0000","last_unstale":"2024-08-28T19:20:37.880842+0000","last_undegraded":"2024-08-28T19:20:37.880842+0000","last_fullsized":"2024-08-28T19:20:37.880842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T22:52:24.180155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.861421+0000","last_change":"2024-08-28T19:18:32.490270+0000","last_active":"2024-08-28T19:20:37.861421+0000","last_peered":"2024-08-28T19:20:37.861421+0000","last_clean":"2024-08-28T19:20:37.861421+0000","last_became_active":"2024-08-28T19:18:32.489667+0000","last_became_peered":"2024-08-28T19:18:32.489667+0000","last_unstale":"2024-08-28T19:20:37.861421+0000","last_undegraded":"2024-08-28T19:20:37.861421+0000","last_fullsized":"2024-08-28T19:20:37.861421+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-29T19:45:18.571978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.873527+0000","last_change":"2024-08-28T19:18:32.482553+0000","last_active":"2024-08-28T19:20:37.873527+0000","last_peered":"2024-08-28T19:20:37.873527+0000","last_clean":"2024-08-28T19:20:37.873527+0000","last_became_active":"2024-08-28T19:18:32.481161+0000","last_became_peered":"2024-08-28T19:18:32.481161+0000","last_unstale":"2024-08-28T19:20:37.873527+0000","last_undegraded":"2024-08-28T19:20:37.873527+0000","last_fullsized":"2024-08-28T19:20:37.873527+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:13:13.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.883684+0000","last_change":"2024-08-28T19:18:32.492473+0000","last_active":"2024-08-28T19:20:37.883684+0000","last_peered":"2024-08-28T19:20:37.883684+0000","last_clean":"2024-08-28T19:20:37.883684+0000","last_became_active":"2024-08-28T19:18:32.491897+0000","last_became_peered":"2024-08-28T19:18:32.491897+0000","last_unstale":"2024-08-28T19:20:37.883684+0000","last_undegraded":"2024-08-28T19:20:37.883684+0000","last_fullsized":"2024-08-28T19:20:37.883684+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T06:48:24.279391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.870908+0000","last_change":"2024-08-28T19:18:32.485621+0000","last_active":"2024-08-28T19:20:37.870908+0000","last_peered":"2024-08-28T19:20:37.870908+0000","last_clean":"2024-08-28T19:20:37.870908+0000","last_became_active":"2024-08-28T19:18:32.485267+0000","last_became_peered":"2024-08-28T19:18:32.485267+0000","last_unstale":"2024-08-28T19:20:37.870908+0000","last_undegraded":"2024-08-28T19:20:37.870908+0000","last_fullsized":"2024-08-28T19:20:37.870908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T05:30:08.932135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"84'8","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-08-28T19:20:37.884032+0000","last_change":"2024-08-28T19:18:32.488457+0000","last_active":"2024-08-28T19:20:37.884032+0000","last_peered":"2024-08-28T19:20:37.884032+0000","last_clean":"2024-08-28T19:20:37.884032+0000","last_became_active":"2024-08-28T19:18:32.487555+0000","last_became_peered":"2024-08-28T19:18:32.487555+0000","last_unstale":"2024-08-28T19:20:37.884032+0000","last_undegraded":"2024-08-28T19:20:37.884032+0000","last_fullsized":"2024-08-28T19:20:37.884032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-28T19:18:31.453879+0000","last_clean_scrub_stamp":"2024-08-28T19:18:31.453879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T01:03:26.390481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":863883,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1520,"num_read_kb":975,"num_write":487,"num_write_kb":2246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2961408,"data_stored":2591649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":369,"ondisk_log_size":369,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":68,"seq":292057776173,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":500491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":62,"seq":266287972401,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29120,"kb_used_data":2036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716032,"statfs":{"total":95995035648,"available":95965216768,"internally_reserved":0,"allocated":2084864,"data_stored":1553912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":57,"seq":244813135927,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40876,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704276,"statfs":{"total":95995035648,"available":95953178624,"internally_reserved":0,"allocated":1404928,"data_stored":897061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":51,"seq":219043332155,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36708,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708444,"statfs":{"total":95995035648,"available":95957446656,"internally_reserved":0,"allocated":1396736,"data_stored":893343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":45,"seq":193273528384,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1593344,"data_stored":1096625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":40,"seq":171798691910,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":495785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888138,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29040,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716112,"statfs":{"total":95995035648,"available":95965298688,"internally_reserved":0,"allocated":2002944,"data_stored":1482027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051665,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":499327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215188,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41248,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703904,"statfs":{"total":95995035648,"available":95952797696,"internally_reserved":0,"allocated":1785856,"data_stored":1281953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1077248,"data_stored":566440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":566220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":496709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":819200,"data_stored":787422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":397128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":464917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-28T19:21:10.765 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:21:10.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:10 smithi028 ceph-mon[21769]: pgmap v343: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 174 KiB/s wr, 55 op/s 2024-08-28T19:21:10.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:10 smithi028 ceph-mon[21769]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-28T19:21:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:11.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.484+0000 7f005af13640 1 -- 172.21.15.16:0/111454047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0054109240 msgr2=0x7f0054109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:11.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.484+0000 7f005af13640 1 --2- 172.21.15.16:0/111454047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0054109240 0x7f0054109650 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0040009900 tx=0x7f004002a410 comp rx=0 tx=0).stop 2024-08-28T19:21:11.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.484+0000 7f005af13640 1 -- 172.21.15.16:0/111454047 shutdown_connections 2024-08-28T19:21:11.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.484+0000 7f005af13640 1 --2- 172.21.15.16:0/111454047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0054109b90 0x7f00541103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:11.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.484+0000 7f005af13640 1 --2- 172.21.15.16:0/111454047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0054109240 0x7f0054109650 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.484+0000 7f005af13640 1 --2- 172.21.15.16:0/111454047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0054103900 0x7f0054103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.484+0000 7f005af13640 1 -- 172.21.15.16:0/111454047 >> 172.21.15.16:0/111454047 conn(0x7f005407b5f0 msgr2=0x7f005407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.484+0000 7f005af13640 1 -- 172.21.15.16:0/111454047 shutdown_connections 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.484+0000 7f005af13640 1 -- 172.21.15.16:0/111454047 wait complete. 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.484+0000 7f005af13640 1 Processor -- start 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.484+0000 7f005af13640 1 -- start start 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f005af13640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0054103900 0x7f005419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f005af13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0054109240 0x7f005419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f005af13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0054109b90 0x7f005419fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f005af13640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0054113850 con 0x7f0054109240 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f005af13640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00541136d0 con 0x7f0054109b90 2024-08-28T19:21:11.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f005af13640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00541139d0 con 0x7f0054103900 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f0053fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0054109240 0x7f005419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f0053fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0054109240 0x7f005419a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45320/0 (socket says 172.21.15.16:45320) 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f0053fff640 1 -- 172.21.15.16:0/1250189252 learned_addr learned my addr 172.21.15.16:0/1250189252 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f0058caf640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0054103900 0x7f005419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f00594b0640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0054109b90 0x7f005419fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f0053fff640 1 -- 172.21.15.16:0/1250189252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0054103900 msgr2=0x7f005419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f0053fff640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0054103900 0x7f005419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f0053fff640 1 -- 172.21.15.16:0/1250189252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0054109b90 msgr2=0x7f005419fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f0053fff640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0054109b90 0x7f005419fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f0053fff640 1 -- 172.21.15.16:0/1250189252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00541a00d0 con 0x7f0054109240 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f0058caf640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0054103900 0x7f005419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.485+0000 7f0053fff640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0054109240 0x7f005419a7c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f004002a930 tx=0x7f0040004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:11.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.486+0000 7f0051ffb640 1 -- 172.21.15.16:0/1250189252 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004003f030 con 0x7f0054109240 2024-08-28T19:21:11.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.486+0000 7f0051ffb640 1 -- 172.21.15.16:0/1250189252 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0040003aa0 con 0x7f0054109240 2024-08-28T19:21:11.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.486+0000 7f005af13640 1 -- 172.21.15.16:0/1250189252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00541a0360 con 0x7f0054109240 2024-08-28T19:21:11.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.486+0000 7f0051ffb640 1 -- 172.21.15.16:0/1250189252 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040035070 con 0x7f0054109240 2024-08-28T19:21:11.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.486+0000 7f005af13640 1 -- 172.21.15.16:0/1250189252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00541a0840 con 0x7f0054109240 2024-08-28T19:21:11.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.487+0000 7f005af13640 1 -- 172.21.15.16:0/1250189252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0054103d10 con 0x7f0054109240 2024-08-28T19:21:11.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.488+0000 7f0051ffb640 1 -- 172.21.15.16:0/1250189252 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f0040042030 con 0x7f0054109240 2024-08-28T19:21:11.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.489+0000 7f0051ffb640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00440b4c40 0x7f00440b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:11.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.489+0000 7f0058caf640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00440b4c40 0x7f00440b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:11.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.489+0000 7f0051ffb640 1 -- 172.21.15.16:0/1250189252 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f00401063d0 con 0x7f0054109240 2024-08-28T19:21:11.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.490+0000 7f0058caf640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00440b4c40 0x7f00440b7150 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f004800abe0 tx=0x7f004800a5c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:11.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.492+0000 7f0051ffb640 1 -- 172.21.15.16:0/1250189252 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f00400cceb0 con 0x7f0054109240 2024-08-28T19:21:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.722+0000 7f005af13640 1 -- 172.21.15.16:0/1250189252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f005419b350 con 0x7f0054109240 2024-08-28T19:21:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.723+0000 7f0051ffb640 1 -- 172.21.15.16:0/1250189252 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11977 (secure 0 0 0) 0x7f00400cc7b0 con 0x7f0054109240 2024-08-28T19:21:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:21:11.724 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"btime":"2024-08-28T19:20:55:257311+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:20:54.880238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:21:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2024-08-28T19:21:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 -- 172.21.15.16:0/1250189252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00440b4c40 msgr2=0x7f00440b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00440b4c40 0x7f00440b7150 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f004800abe0 tx=0x7f004800a5c0 comp rx=0 tx=0).stop 2024-08-28T19:21:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 -- 172.21.15.16:0/1250189252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0054109240 msgr2=0x7f005419a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0054109240 0x7f005419a7c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f004002a930 tx=0x7f0040004270 comp rx=0 tx=0).stop 2024-08-28T19:21:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 -- 172.21.15.16:0/1250189252 shutdown_connections 2024-08-28T19:21:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00440b4c40 0x7f00440b7150 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0054109b90 0x7f005419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0054109240 0x7f005419a7c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 --2- 172.21.15.16:0/1250189252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0054103900 0x7f005419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 -- 172.21.15.16:0/1250189252 >> 172.21.15.16:0/1250189252 conn(0x7f005407b5f0 msgr2=0x7f0054107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 -- 172.21.15.16:0/1250189252 shutdown_connections 2024-08-28T19:21:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:11.725+0000 7f005af13640 1 -- 172.21.15.16:0/1250189252 wait complete. 2024-08-28T19:21:12.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:21:12.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:12 smithi016 ceph-mon[18556]: pgmap v344: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 174 KiB/s wr, 58 op/s 2024-08-28T19:21:12.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:12 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1250189252' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:12.771 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:12.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:12 smithi028 ceph-mon[21769]: pgmap v344: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 174 KiB/s wr, 58 op/s 2024-08-28T19:21:12.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:12 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1250189252' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:12 smithi115 ceph-mon[22184]: pgmap v344: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 174 KiB/s wr, 58 op/s 2024-08-28T19:21:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:12 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1250189252' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.235+0000 7fadca81f640 1 -- 172.21.15.16:0/1728649036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadc4102880 msgr2=0x7fadc4102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.235+0000 7fadca81f640 1 --2- 172.21.15.16:0/1728649036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadc4102880 0x7fadc4102c90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fadb4009900 tx=0x7fadb402a410 comp rx=0 tx=0).stop 2024-08-28T19:21:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.235+0000 7fadca81f640 1 -- 172.21.15.16:0/1728649036 shutdown_connections 2024-08-28T19:21:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.235+0000 7fadca81f640 1 --2- 172.21.15.16:0/1728649036 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadc4108b10 0x7fadc410f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.235+0000 7fadca81f640 1 --2- 172.21.15.16:0/1728649036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc41081c0 0x7fadc41085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.235+0000 7fadca81f640 1 --2- 172.21.15.16:0/1728649036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadc4102880 0x7fadc4102c90 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.235+0000 7fadca81f640 1 -- 172.21.15.16:0/1728649036 >> 172.21.15.16:0/1728649036 conn(0x7fadc4079510 msgr2=0x7fadc4079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.235+0000 7fadca81f640 1 -- 172.21.15.16:0/1728649036 shutdown_connections 2024-08-28T19:21:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.235+0000 7fadca81f640 1 -- 172.21.15.16:0/1728649036 wait complete. 2024-08-28T19:21:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadca81f640 1 Processor -- start 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadca81f640 1 -- start start 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadca81f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadc4102880 0x7fadc4199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadca81f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadc41081c0 0x7fadc41997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadca81f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc4108b10 0x7fadc419ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadca81f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fadc4112810 con 0x7fadc41081c0 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadca81f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadc4112690 con 0x7fadc4108b10 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadca81f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fadc4112990 con 0x7fadc4102880 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadc37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadc41081c0 0x7fadc41997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadc37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadc41081c0 0x7fadc41997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45340/0 (socket says 172.21.15.16:45340) 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadc37fe640 1 -- 172.21.15.16:0/1067847086 learned_addr learned my addr 172.21.15.16:0/1067847086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.236+0000 7fadc8dbc640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc4108b10 0x7fadc419ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc3fff640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadc4102880 0x7fadc4199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc37fe640 1 -- 172.21.15.16:0/1067847086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadc4102880 msgr2=0x7fadc4199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc37fe640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadc4102880 0x7fadc4199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc37fe640 1 -- 172.21.15.16:0/1067847086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc4108b10 msgr2=0x7fadc419ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc37fe640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc4108b10 0x7fadc419ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc37fe640 1 -- 172.21.15.16:0/1067847086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc419f0d0 con 0x7fadc41081c0 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc3fff640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadc4102880 0x7fadc4199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc8dbc640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc4108b10 0x7fadc419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc37fe640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadc41081c0 0x7fadc41997c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fadb8006fd0 tx=0x7fadb800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc17fa640 1 -- 172.21.15.16:0/1067847086 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8011830 con 0x7fadc41081c0 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadca81f640 1 -- 172.21.15.16:0/1067847086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc419f3c0 con 0x7fadc41081c0 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc17fa640 1 -- 172.21.15.16:0/1067847086 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadb8016070 con 0x7fadc41081c0 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadc17fa640 1 -- 172.21.15.16:0/1067847086 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8010e00 con 0x7fadc41081c0 2024-08-28T19:21:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.237+0000 7fadca81f640 1 -- 172.21.15.16:0/1067847086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc41a5be0 con 0x7fadc41081c0 2024-08-28T19:21:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.238+0000 7fadca81f640 1 -- 172.21.15.16:0/1067847086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc4103b00 con 0x7fadc41081c0 2024-08-28T19:21:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.240+0000 7fadc17fa640 1 -- 172.21.15.16:0/1067847086 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7fadb80119d0 con 0x7fadc41081c0 2024-08-28T19:21:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.240+0000 7fadc17fa640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadac0b4c40 0x7fadac0b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.240+0000 7fadc3fff640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadac0b4c40 0x7fadac0b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.241+0000 7fadc17fa640 1 -- 172.21.15.16:0/1067847086 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fadb80e5420 con 0x7fadc41081c0 2024-08-28T19:21:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.241+0000 7fadc3fff640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadac0b4c40 0x7fadac0b7150 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fadb4000c00 tx=0x7fadb4031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.243+0000 7fadc17fa640 1 -- 172.21.15.16:0/1067847086 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fadb80abf60 con 0x7fadc41081c0 2024-08-28T19:21:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.468+0000 7fadca81f640 1 -- 172.21.15.16:0/1067847086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadc4102c90 con 0x7fadc41081c0 2024-08-28T19:21:13.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.469+0000 7fadc17fa640 1 -- 172.21.15.16:0/1067847086 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11977 (secure 0 0 0) 0x7fadc4102c90 con 0x7fadc41081c0 2024-08-28T19:21:13.470 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:21:13.470 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"btime":"2024-08-28T19:20:55:257311+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:20:54.880238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:21:13.471 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2024-08-28T19:21:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 -- 172.21.15.16:0/1067847086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadac0b4c40 msgr2=0x7fadac0b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadac0b4c40 0x7fadac0b7150 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fadb4000c00 tx=0x7fadb4031400 comp rx=0 tx=0).stop 2024-08-28T19:21:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 -- 172.21.15.16:0/1067847086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadc41081c0 msgr2=0x7fadc41997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadc41081c0 0x7fadc41997c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fadb8006fd0 tx=0x7fadb800b8e0 comp rx=0 tx=0).stop 2024-08-28T19:21:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 -- 172.21.15.16:0/1067847086 shutdown_connections 2024-08-28T19:21:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadac0b4c40 0x7fadac0b7150 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc4108b10 0x7fadc419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadc41081c0 0x7fadc41997c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 --2- 172.21.15.16:0/1067847086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadc4102880 0x7fadc4199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 -- 172.21.15.16:0/1067847086 >> 172.21.15.16:0/1067847086 conn(0x7fadc4079510 msgr2=0x7fadc41062c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 -- 172.21.15.16:0/1067847086 shutdown_connections 2024-08-28T19:21:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:13.471+0000 7fadca81f640 1 -- 172.21.15.16:0/1067847086 wait complete. 2024-08-28T19:21:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:13 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1067847086' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:13 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1067847086' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:14.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:13 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1067847086' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:14.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:21:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:14.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:14 smithi016 ceph-mon[18556]: pgmap v345: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 174 KiB/s wr, 54 op/s 2024-08-28T19:21:14.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:14 smithi115 ceph-mon[22184]: pgmap v345: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 174 KiB/s wr, 54 op/s 2024-08-28T19:21:15.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:14 smithi028 ceph-mon[21769]: pgmap v345: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 174 KiB/s wr, 54 op/s 2024-08-28T19:21:15.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.120+0000 7f8e467d6640 1 -- 172.21.15.16:0/3413466707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e4010dc30 msgr2=0x7f8e401100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:15.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.120+0000 7f8e467d6640 1 --2- 172.21.15.16:0/3413466707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e4010dc30 0x7f8e401100d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009ca0 tx=0x7f8e3802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.120+0000 7f8e467d6640 1 -- 172.21.15.16:0/3413466707 shutdown_connections 2024-08-28T19:21:15.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.120+0000 7f8e467d6640 1 --2- 172.21.15.16:0/3413466707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e4010dc30 0x7f8e401100d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.120+0000 7f8e467d6640 1 --2- 172.21.15.16:0/3413466707 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e401050b0 0x7f8e4010d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.120+0000 7f8e467d6640 1 --2- 172.21.15.16:0/3413466707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e40104680 0x7f8e40104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.120+0000 7f8e467d6640 1 -- 172.21.15.16:0/3413466707 >> 172.21.15.16:0/3413466707 conn(0x7f8e40079530 msgr2=0x7f8e40079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:15.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.120+0000 7f8e467d6640 1 -- 172.21.15.16:0/3413466707 shutdown_connections 2024-08-28T19:21:15.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.120+0000 7f8e467d6640 1 -- 172.21.15.16:0/3413466707 wait complete. 2024-08-28T19:21:15.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e467d6640 1 Processor -- start 2024-08-28T19:21:15.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e467d6640 1 -- start start 2024-08-28T19:21:15.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e467d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e40104680 0x7f8e40102590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:15.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e467d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e401050b0 0x7f8e40102ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:15.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e467d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e4010dc30 0x7f8e40103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:15.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e467d6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e40113860 con 0x7f8e40104680 2024-08-28T19:21:15.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e467d6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e401136e0 con 0x7f8e401050b0 2024-08-28T19:21:15.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e467d6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8e401139e0 con 0x7f8e4010dc30 2024-08-28T19:21:15.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e40104680 0x7f8e40102590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:15.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e40104680 0x7f8e40102590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54604/0 (socket says 172.21.15.16:54604) 2024-08-28T19:21:15.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e3ffff640 1 -- 172.21.15.16:0/1836246543 learned_addr learned my addr 172.21.15.16:0/1836246543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:15.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.121+0000 7f8e37fff640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e401050b0 0x7f8e40102ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:15.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e3ffff640 1 -- 172.21.15.16:0/1836246543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e4010dc30 msgr2=0x7f8e40103010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:15.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e3ffff640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e4010dc30 0x7f8e40103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e3ffff640 1 -- 172.21.15.16:0/1836246543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e401050b0 msgr2=0x7f8e40102ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:15.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e3ffff640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e401050b0 0x7f8e40102ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e3ffff640 1 -- 172.21.15.16:0/1836246543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e401a83e0 con 0x7f8e40104680 2024-08-28T19:21:15.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e37fff640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e401050b0 0x7f8e40102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:15.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e3ffff640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e40104680 0x7f8e40102590 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000c920 tx=0x7f8e3000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:15.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e3dffb640 1 -- 172.21.15.16:0/1836246543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30007c70 con 0x7f8e40104680 2024-08-28T19:21:15.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e3dffb640 1 -- 172.21.15.16:0/1836246543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e30007e10 con 0x7f8e40104680 2024-08-28T19:21:15.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e467d6640 1 -- 172.21.15.16:0/1836246543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e401a86d0 con 0x7f8e40104680 2024-08-28T19:21:15.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e3dffb640 1 -- 172.21.15.16:0/1836246543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30018820 con 0x7f8e40104680 2024-08-28T19:21:15.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.122+0000 7f8e467d6640 1 -- 172.21.15.16:0/1836246543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e401a8c10 con 0x7f8e40104680 2024-08-28T19:21:15.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.123+0000 7f8e467d6640 1 -- 172.21.15.16:0/1836246543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e40104a90 con 0x7f8e40104680 2024-08-28T19:21:15.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.125+0000 7f8e3dffb640 1 -- 172.21.15.16:0/1836246543 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f8e3000f450 con 0x7f8e40104680 2024-08-28T19:21:15.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.125+0000 7f8e3dffb640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e2c0b4c10 0x7f8e2c0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:15.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.125+0000 7f8e37fff640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e2c0b4c10 0x7f8e2c0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:15.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.125+0000 7f8e3dffb640 1 -- 172.21.15.16:0/1836246543 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f8e300e5c60 con 0x7f8e40104680 2024-08-28T19:21:15.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.126+0000 7f8e37fff640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e2c0b4c10 0x7f8e2c0b7120 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8e28005fd0 tx=0x7f8e28005ea0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:15.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.128+0000 7f8e3dffb640 1 -- 172.21.15.16:0/1836246543 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8e30014030 con 0x7f8e40104680 2024-08-28T19:21:15.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.317+0000 7f8e467d6640 1 -- 172.21.15.16:0/1836246543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e40103890 con 0x7f8e40104680 2024-08-28T19:21:15.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.319+0000 7f8e3dffb640 1 -- 172.21.15.16:0/1836246543 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+25180 (secure 0 0 0) 0x7f8e300ac830 con 0x7f8e40104680 2024-08-28T19:21:15.321 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:21:15.321 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:20:09.003260+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:21:15.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.322+0000 7f8e467d6640 1 -- 172.21.15.16:0/1836246543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e2c0b4c10 msgr2=0x7f8e2c0b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:15.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.322+0000 7f8e467d6640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e2c0b4c10 0x7f8e2c0b7120 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8e28005fd0 tx=0x7f8e28005ea0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.322+0000 7f8e467d6640 1 -- 172.21.15.16:0/1836246543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e40104680 msgr2=0x7f8e40102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:15.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.322+0000 7f8e467d6640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e40104680 0x7f8e40102590 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000c920 tx=0x7f8e3000cde0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.322+0000 7f8e467d6640 1 -- 172.21.15.16:0/1836246543 shutdown_connections 2024-08-28T19:21:15.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.322+0000 7f8e467d6640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e4010dc30 0x7f8e40103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.323+0000 7f8e467d6640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e2c0b4c10 0x7f8e2c0b7120 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.323+0000 7f8e467d6640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e401050b0 0x7f8e40102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.323+0000 7f8e467d6640 1 --2- 172.21.15.16:0/1836246543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e40104680 0x7f8e40102590 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:15.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.323+0000 7f8e467d6640 1 -- 172.21.15.16:0/1836246543 >> 172.21.15.16:0/1836246543 conn(0x7f8e40079530 msgr2=0x7f8e40107760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:15.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.323+0000 7f8e467d6640 1 -- 172.21.15.16:0/1836246543 shutdown_connections 2024-08-28T19:21:15.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:15.323+0000 7f8e467d6640 1 -- 172.21.15.16:0/1836246543 wait complete. 2024-08-28T19:21:15.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:15 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1836246543' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:21:15.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:15 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1836246543' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:21:16.018 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-08-28T19:21:16.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:15 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1836246543' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:21:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:16.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:16 smithi016 ceph-mon[18556]: pgmap v346: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 174 KiB/s wr, 58 op/s 2024-08-28T19:21:16.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:16 smithi115 ceph-mon[22184]: pgmap v346: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 174 KiB/s wr, 58 op/s 2024-08-28T19:21:17.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:16 smithi028 ceph-mon[21769]: pgmap v346: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 174 KiB/s wr, 58 op/s 2024-08-28T19:21:17.399 INFO:teuthology.orchestra.run.smithi016.stdout:{"run_dir":"/var/run/ceph"} 2024-08-28T19:21:17.399 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-08-28T19:21:17.399 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 7797 2024-08-28T19:21:17.400 DEBUG:tasks.quiescer:Yielding 2024-08-28T19:21:17.400 INFO:teuthology.run_tasks:Running task workunit... 2024-08-28T19:21:17.421 INFO:tasks.workunit:Pulling workunits from ref d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:21:17.421 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-28T19:21:17.421 INFO:tasks.workunit:timeout=3h 2024-08-28T19:21:17.421 INFO:tasks.workunit:cleanup=True 2024-08-28T19:21:17.422 DEBUG:teuthology.orchestra.run.smithi016:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-28T19:21:17.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:21:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-08-28T19:21:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-08-28T19:21:17.439 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 79h/121d Inode: 1099511628283 Links: 2 2024-08-28T19:21:17.439 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-28T19:21:17.439 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-08-28T19:21:17.439 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-08-28 19:19:19.641567787 +0000 2024-08-28T19:21:17.439 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-08-28 19:19:19.641567787 +0000 2024-08-28T19:21:17.439 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-08-28 19:20:54.187949970 +0000 2024-08-28T19:21:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-08-28 19:19:19.641567787 +0000 2024-08-28T19:21:17.439 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-08-28T19:21:17.439 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-08-28T19:21:17.512 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-28T19:21:17.568 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-28T19:21:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.181+0000 7f5910ed0640 1 -- 172.21.15.16:0/1019322808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f590c102810 msgr2=0x7f590c102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.181+0000 7f5910ed0640 1 --2- 172.21.15.16:0/1019322808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f590c102810 0x7f590c102c20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009900 tx=0x7f58fc02a410 comp rx=0 tx=0).stop 2024-08-28T19:21:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.181+0000 7f5910ed0640 1 -- 172.21.15.16:0/1019322808 shutdown_connections 2024-08-28T19:21:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.181+0000 7f5910ed0640 1 --2- 172.21.15.16:0/1019322808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c108aa0 0x7f590c10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.181+0000 7f5910ed0640 1 --2- 172.21.15.16:0/1019322808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f590c108150 0x7f590c108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.181+0000 7f5910ed0640 1 --2- 172.21.15.16:0/1019322808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f590c102810 0x7f590c102c20 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.181+0000 7f5910ed0640 1 -- 172.21.15.16:0/1019322808 >> 172.21.15.16:0/1019322808 conn(0x7f590c0794b0 msgr2=0x7f590c0798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.181+0000 7f5910ed0640 1 -- 172.21.15.16:0/1019322808 shutdown_connections 2024-08-28T19:21:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.181+0000 7f5910ed0640 1 -- 172.21.15.16:0/1019322808 wait complete. 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f5910ed0640 1 Processor -- start 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f5910ed0640 1 -- start start 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f5910ed0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c102810 0x7f590c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f5910ed0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f590c108150 0x7f590c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f5910ed0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f590c108aa0 0x7f590c19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f5910ed0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f590c112770 con 0x7f590c108150 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f5910ed0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f590c1125f0 con 0x7f590c102810 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f5910ed0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f590c1128f0 con 0x7f590c108aa0 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f5909d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f590c108150 0x7f590c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f5909d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f590c108150 0x7f590c19a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54618/0 (socket says 172.21.15.16:54618) 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f5909d9b640 1 -- 172.21.15.16:0/4159067669 learned_addr learned my addr 172.21.15.16:0/4159067669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.182+0000 7f590a59c640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c102810 0x7f590c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.183+0000 7f5909d9b640 1 -- 172.21.15.16:0/4159067669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f590c108aa0 msgr2=0x7f590c19fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.183+0000 7f5909d9b640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f590c108aa0 0x7f590c19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.183+0000 7f5909d9b640 1 -- 172.21.15.16:0/4159067669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c102810 msgr2=0x7f590c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.183+0000 7f5909d9b640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c102810 0x7f590c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.183+0000 7f5909d9b640 1 -- 172.21.15.16:0/4159067669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590c1a0200 con 0x7f590c108150 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.183+0000 7f5909d9b640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f590c108150 0x7f590c19a830 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f590000d750 tx=0x7f590000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.183+0000 7f58fb7fe640 1 -- 172.21.15.16:0/4159067669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900017070 con 0x7f590c108150 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.183+0000 7f58fb7fe640 1 -- 172.21.15.16:0/4159067669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59000054d0 con 0x7f590c108150 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.183+0000 7f58fb7fe640 1 -- 172.21.15.16:0/4159067669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900004c30 con 0x7f590c108150 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.183+0000 7f5910ed0640 1 -- 172.21.15.16:0/4159067669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f590c1a04f0 con 0x7f590c108150 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.183+0000 7f5910ed0640 1 -- 172.21.15.16:0/4159067669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f590c1a6c50 con 0x7f590c108150 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.184+0000 7f5910ed0640 1 -- 172.21.15.16:0/4159067669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f590c103a90 con 0x7f590c108150 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.185+0000 7f58fb7fe640 1 -- 172.21.15.16:0/4159067669 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f5900004dd0 con 0x7f590c108150 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.186+0000 7f58fb7fe640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58f40b4c10 0x7f58f40b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.186+0000 7f590a59c640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58f40b4c10 0x7f58f40b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.186+0000 7f58fb7fe640 1 -- 172.21.15.16:0/4159067669 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f59000e5680 con 0x7f590c108150 2024-08-28T19:21:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.187+0000 7f590a59c640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58f40b4c10 0x7f58f40b7120 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f58fc031440 tx=0x7f58fc005d50 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.189+0000 7f58fb7fe640 1 -- 172.21.15.16:0/4159067669 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f59000ac1c0 con 0x7f590c108150 2024-08-28T19:21:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.415+0000 7f5910ed0640 1 -- 172.21.15.16:0/4159067669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f590c102c20 con 0x7f590c108150 2024-08-28T19:21:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.416+0000 7f58fb7fe640 1 -- 172.21.15.16:0/4159067669 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11977 (secure 0 0 0) 0x7f590c102c20 con 0x7f590c108150 2024-08-28T19:21:18.417 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:21:18.418 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"btime":"2024-08-28T19:20:55:257311+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:20:54.880238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:21:18.418 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2024-08-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 -- 172.21.15.16:0/4159067669 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58f40b4c10 msgr2=0x7f58f40b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58f40b4c10 0x7f58f40b7120 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f58fc031440 tx=0x7f58fc005d50 comp rx=0 tx=0).stop 2024-08-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 -- 172.21.15.16:0/4159067669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f590c108150 msgr2=0x7f590c19a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f590c108150 0x7f590c19a830 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f590000d750 tx=0x7f590000dc10 comp rx=0 tx=0).stop 2024-08-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 -- 172.21.15.16:0/4159067669 shutdown_connections 2024-08-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f590c108aa0 0x7f590c19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f590c108150 0x7f590c19a830 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58f40b4c10 0x7f58f40b7120 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 --2- 172.21.15.16:0/4159067669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c102810 0x7f590c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 -- 172.21.15.16:0/4159067669 >> 172.21.15.16:0/4159067669 conn(0x7f590c0794b0 msgr2=0x7f590c1018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 -- 172.21.15.16:0/4159067669 shutdown_connections 2024-08-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:18.418+0000 7f5910ed0640 1 -- 172.21.15.16:0/4159067669 wait complete. 2024-08-28T19:21:18.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:18 smithi016 ceph-mon[18556]: pgmap v347: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 112 KiB/s wr, 48 op/s 2024-08-28T19:21:18.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:18 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:21:18.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:18 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/4159067669' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:18.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:18 smithi115 ceph-mon[22184]: pgmap v347: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 112 KiB/s wr, 48 op/s 2024-08-28T19:21:18.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:18 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:21:18.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:18 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/4159067669' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:19.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:18 smithi028 ceph-mon[21769]: pgmap v347: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 112 KiB/s wr, 48 op/s 2024-08-28T19:21:19.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:18 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:21:19.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:18 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/4159067669' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:19.162 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-08-28T19:18:33.178088+0000', 'modified': '2024-08-28T19:20:54.880238+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24665, 'mds_1': 24649, 'mds_2': 24635, 'mds_3': 24625, 'mds_4': 24619}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14817': {'gid': 14817, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6839/3069459721', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 3069459721}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 3069459721}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14823': {'gid': 14823, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6841/2698492790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6840', 'nonce': 2698492790}, {'type': 'v1', 'addr': '172.21.15.16:6841', 'nonce': 2698492790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24583': {'gid': 24583, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.115:6833/2581346583', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6832', 'nonce': 2581346583}, {'type': 'v1', 'addr': '172.21.15.115:6833', 'nonce': 2581346583}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24590': {'gid': 24590, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6833/798033587', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 798033587}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 798033587}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24605': {'gid': 24605, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6835/2983279759', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2983279759}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2983279759}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24619': {'gid': 24619, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.115:6835/1297102593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6834', 'nonce': 1297102593}, {'type': 'v1', 'addr': '172.21.15.115:6835', 'nonce': 1297102593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24625': {'gid': 24625, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.115:6837/4186796507', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6836', 'nonce': 4186796507}, {'type': 'v1', 'addr': '172.21.15.115:6837', 'nonce': 4186796507}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24635': {'gid': 24635, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.28:6837/2888152030', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6836', 'nonce': 2888152030}, {'type': 'v1', 'addr': '172.21.15.28:6837', 'nonce': 2888152030}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24649': {'gid': 24649, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.115:6839/3027291414', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6838', 'nonce': 3027291414}, {'type': 'v1', 'addr': '172.21.15.115:6839', 'nonce': 3027291414}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24665': {'gid': 24665, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6839/2518320595', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 2518320595}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 2518320595}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24665, 'qdb_cluster': [24619, 24625, 24635, 24649, 24665]} 2024-08-28T19:21:19.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-08-28T19:21:19.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.f status 2024-08-28T19:21:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.934+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/2777059792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a5410dc30 msgr2=0x7f0a54110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.934+0000 7f0a5c5b9640 1 --2- 172.21.15.16:0/2777059792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a5410dc30 0x7f0a54110120 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c009ca0 tx=0x7f0a4c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.934+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/2777059792 shutdown_connections 2024-08-28T19:21:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.934+0000 7f0a5c5b9640 1 --2- 172.21.15.16:0/2777059792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a5410dc30 0x7f0a54110120 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.934+0000 7f0a5c5b9640 1 --2- 172.21.15.16:0/2777059792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a54105120 0x7f0a5410d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.934+0000 7f0a5c5b9640 1 --2- 172.21.15.16:0/2777059792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a541047d0 0x7f0a54104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.934+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/2777059792 >> 172.21.15.16:0/2777059792 conn(0x7f0a54079510 msgr2=0x7f0a54079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.935+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/2777059792 shutdown_connections 2024-08-28T19:21:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.935+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/2777059792 wait complete. 2024-08-28T19:21:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.935+0000 7f0a5c5b9640 1 Processor -- start 2024-08-28T19:21:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.935+0000 7f0a5c5b9640 1 -- start start 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.935+0000 7f0a5c5b9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a541047d0 0x7f0a5419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.935+0000 7f0a5c5b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a54105120 0x7f0a5419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.935+0000 7f0a5c5b9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a5410dc30 0x7f0a5419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.935+0000 7f0a5c5b9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a54113890 con 0x7f0a54105120 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.935+0000 7f0a5c5b9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a54113710 con 0x7f0a5410dc30 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.935+0000 7f0a5c5b9640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0a54113a10 con 0x7f0a541047d0 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a59b54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a54105120 0x7f0a5419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a59b54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a54105120 0x7f0a5419a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54644/0 (socket says 172.21.15.16:54644) 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a59b54640 1 -- 172.21.15.16:0/3387383625 learned_addr learned my addr 172.21.15.16:0/3387383625 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a5a355640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a541047d0 0x7f0a5419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a5ab56640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a5410dc30 0x7f0a5419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a59b54640 1 -- 172.21.15.16:0/3387383625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a541047d0 msgr2=0x7f0a5419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a59b54640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a541047d0 0x7f0a5419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a59b54640 1 -- 172.21.15.16:0/3387383625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a5410dc30 msgr2=0x7f0a5419faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a59b54640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a5410dc30 0x7f0a5419faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a59b54640 1 -- 172.21.15.16:0/3387383625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a541a0110 con 0x7f0a54105120 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a5a355640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a541047d0 0x7f0a5419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a59b54640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a54105120 0x7f0a5419a800 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0a4400d950 tx=0x7f0a4400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a4b7fe640 1 -- 172.21.15.16:0/3387383625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a44017070 con 0x7f0a54105120 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a4b7fe640 1 -- 172.21.15.16:0/3387383625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a44009c30 con 0x7f0a54105120 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/3387383625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a541a0400 con 0x7f0a54105120 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a4b7fe640 1 -- 172.21.15.16:0/3387383625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a440047a0 con 0x7f0a54105120 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.936+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/3387383625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a541a6c20 con 0x7f0a54105120 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.939+0000 7f0a62b2e740 1 Processor -- start 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.939+0000 7f0a62b2e740 1 -- start start 2024-08-28T19:21:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.939+0000 7f0a4b7fe640 1 -- 172.21.15.16:0/3387383625 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f0a44004020 con 0x7f0a54105120 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.939+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a603f60 0x56519a9a97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.939+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5db0a0 0x56519a9a9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.939+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9aa230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.939+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56519a9203f0 con 0x56519a5e3280 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.939+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56519a90f980 con 0x56519a5db0a0 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.939+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56519a913930 con 0x56519a603f60 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a59b54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a5a355640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5db0a0 0x56519a9a9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a59b54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9aa230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54646/0 (socket says 172.21.15.16:54646) 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a59b54640 1 -- 172.21.15.16:0/4254893563 learned_addr learned my addr 172.21.15.16:0/4254893563 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a5ab56640 1 --2- 172.21.15.16:0/4254893563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a603f60 0x56519a9a97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a4b7fe640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a400b4c40 0x7f0a400b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a59b54640 1 -- 172.21.15.16:0/4254893563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a603f60 msgr2=0x56519a9a97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a59b54640 1 --2- 172.21.15.16:0/4254893563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a603f60 0x56519a9a97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a59b54640 1 -- 172.21.15.16:0/4254893563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5db0a0 msgr2=0x56519a9a9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a59b54640 1 --2- 172.21.15.16:0/4254893563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5db0a0 0x56519a9a9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a59b54640 1 -- 172.21.15.16:0/4254893563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56519a66f540 con 0x56519a5e3280 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a5ab56640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a400b4c40 0x7f0a400b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a4b7fe640 1 -- 172.21.15.16:0/3387383625 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0a440e5440 con 0x7f0a54105120 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a59b54640 1 --2- 172.21.15.16:0/4254893563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9aa230 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0a440e28a0 tx=0x7f0a440bd3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bce4fd47c7f2be0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.940+0000 7f0a48ff9640 1 -- 172.21.15.16:0/4254893563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a440ab1d0 con 0x56519a5e3280 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.941+0000 7f0a48ff9640 1 -- 172.21.15.16:0/4254893563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a440d7970 con 0x56519a5e3280 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.941+0000 7f0a48ff9640 1 -- 172.21.15.16:0/4254893563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a440b1280 con 0x56519a5e3280 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.941+0000 7f0a62b2e740 1 -- 172.21.15.16:0/4254893563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 msgr2=0x56519a9aa230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.941+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/4254893563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9aa230 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0a440e28a0 tx=0x7f0a440bd3f0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.941+0000 7f0a5ab56640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a400b4c40 0x7f0a400b7150 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c006010 tx=0x7f0a4c0095c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.941+0000 7f0a62b2e740 1 -- 172.21.15.16:0/4254893563 shutdown_connections 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.941+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/4254893563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9aa230 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.941+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/4254893563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5db0a0 0x56519a9a9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.941+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/4254893563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a603f60 0x56519a9a97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 1 -- 172.21.15.16:0/4254893563 >> 172.21.15.16:0/4254893563 conn(0x56519a5f8350 msgr2=0x56519a9a76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 1 -- 172.21.15.16:0/4254893563 shutdown_connections 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 1 -- 172.21.15.16:0/4254893563 wait complete. 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 1 -- start start 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 1 Processor -- start 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a5db0a0 0x56519a9b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5e3280 0x56519a9b5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a603f60 0x56519a9b5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56519a7725f0 con 0x56519a603f60 2024-08-28T19:21:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56519a896600 con 0x56519a5e3280 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56519a9b7f30 con 0x56519a5db0a0 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a62b2e740 10 client.0 _ll_register_callbacks cb 0x56519a8bcaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.942+0000 7f0a5ab56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a603f60 0x56519a9b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a5ab56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a603f60 0x56519a9b5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54656/0 (socket says 172.21.15.16:54656) 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a5ab56640 1 -- 172.21.15.16:0/2448747795 learned_addr learned my addr 172.21.15.16:0/2448747795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a5a355640 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a5db0a0 0x56519a9b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a59b54640 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5e3280 0x56519a9b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a5ab56640 1 -- 172.21.15.16:0/2448747795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a5db0a0 msgr2=0x56519a9b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a5ab56640 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a5db0a0 0x56519a9b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a5ab56640 1 -- 172.21.15.16:0/2448747795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5e3280 msgr2=0x56519a9b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a5ab56640 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5e3280 0x56519a9b5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a5ab56640 1 -- 172.21.15.16:0/2448747795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56519a8cf7b0 con 0x56519a603f60 2024-08-28T19:21:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a5ab56640 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a603f60 0x56519a9b5870 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c031610 tx=0x7f0a4c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6af20440c257bf2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2448747795 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c003c00 con 0x56519a603f60 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2448747795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56519a922340 con 0x56519a603f60 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2448747795 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0a4c003da0 con 0x56519a603f60 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2448747795 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0a4c05c050 con 0x56519a603f60 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a62b2e740 10 client.15711 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.943+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2448747795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56519a9c3c00 con 0x56519a603f60 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.944+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2448747795 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56519a922340 con 0x56519a603f60 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.944+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2448747795 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0a4c02fa80 con 0x56519a603f60 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.944+0000 7f0a62b2e740 10 client.15711 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.944+0000 7f0a62b2e740 10 client.15711 resolve_mds: resolved name 'f' to daemon mds.f 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.944+0000 7f0a62b2e740 20 client.15711 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.944+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56519a5c28e0 0x56519a9c7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.944+0000 7f0a62b2e740 4 client.15711 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.944+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2448747795 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56519a91dde0 con 0x56519a5c28e0 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.944+0000 7f0a5a355640 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56519a5c28e0 0x56519a9c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.945+0000 7f0a5a355640 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56519a5c28e0 0x56519a9c7640 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bea60a129329733e server_cookie=2b2654f0968c4e94 in_seq=0 out_seq=0 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.945+0000 7f0a3dffb640 10 client.15711 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2448747795 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56519a91dde0 con 0x56519a5c28e0 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a3dffb640 10 client.15711 handle_command_reply: tid=0 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 1 client.15711 shutdown 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2448747795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a603f60 msgr2=0x56519a9b5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a603f60 0x56519a9b5870 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c031610 tx=0x7f0a4c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2448747795 shutdown_connections 2024-08-28T19:21:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56519a5c28e0 0x56519a9c7640 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a603f60 0x56519a9b5870 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5e3280 0x56519a9b5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a3dffb640 0 client.15711 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:21:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2448747795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a5db0a0 0x56519a9b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2448747795 >> 172.21.15.16:0/2448747795 conn(0x56519a5f8350 msgr2=0x56519a9ac940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2448747795 shutdown_connections 2024-08-28T19:21:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2448747795 wait complete. 2024-08-28T19:21:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.946+0000 7f0a62b2e740 20 client.15711 trim_cache size 0 max 16384 2024-08-28T19:21:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.964+0000 7f0a62b2e740 1 Processor -- start 2024-08-28T19:21:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.964+0000 7f0a62b2e740 1 -- start start 2024-08-28T19:21:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a5db0a0 0x56519a9a2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9a6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a603f60 0x56519a9a7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56519a9c5980 con 0x56519a5e3280 2024-08-28T19:21:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56519a9c6050 con 0x56519a603f60 2024-08-28T19:21:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56519a9c6720 con 0x56519a5db0a0 2024-08-28T19:21:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a59b54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9a6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a59b54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9a6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54668/0 (socket says 172.21.15.16:54668) 2024-08-28T19:21:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a59b54640 1 -- 172.21.15.16:0/2371125888 learned_addr learned my addr 172.21.15.16:0/2371125888 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a5a355640 1 --2- 172.21.15.16:0/2371125888 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a5db0a0 0x56519a9a2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a5ab56640 1 --2- 172.21.15.16:0/2371125888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a603f60 0x56519a9a7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a59b54640 1 -- 172.21.15.16:0/2371125888 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a5db0a0 msgr2=0x56519a9a2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a59b54640 1 --2- 172.21.15.16:0/2371125888 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a5db0a0 0x56519a9a2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a59b54640 1 -- 172.21.15.16:0/2371125888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a603f60 msgr2=0x56519a9a7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a59b54640 1 --2- 172.21.15.16:0/2371125888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a603f60 0x56519a9a7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a59b54640 1 -- 172.21.15.16:0/2371125888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56519a9b7510 con 0x56519a5e3280 2024-08-28T19:21:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.965+0000 7f0a59b54640 1 --2- 172.21.15.16:0/2371125888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9a6cf0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0a440a8af0 tx=0x7f0a440e2900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d8eed27cee1cce server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.966+0000 7f0a3ffff640 1 -- 172.21.15.16:0/2371125888 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a440aaff0 con 0x56519a5e3280 2024-08-28T19:21:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.966+0000 7f0a3ffff640 1 -- 172.21.15.16:0/2371125888 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a440ab390 con 0x56519a5e3280 2024-08-28T19:21:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.966+0000 7f0a3ffff640 1 -- 172.21.15.16:0/2371125888 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a440ab880 con 0x56519a5e3280 2024-08-28T19:21:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.966+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2371125888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 msgr2=0x56519a9a6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.966+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2371125888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9a6cf0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0a440a8af0 tx=0x7f0a440e2900 comp rx=0 tx=0).stop 2024-08-28T19:21:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.966+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2371125888 shutdown_connections 2024-08-28T19:21:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.966+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2371125888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a603f60 0x56519a9a7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.966+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2371125888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9a6cf0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.966+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2371125888 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a5db0a0 0x56519a9a2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.966+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2371125888 >> 172.21.15.16:0/2371125888 conn(0x56519a5f8350 msgr2=0x56519a9b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.966+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2371125888 shutdown_connections 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2371125888 wait complete. 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a62b2e740 1 -- start start 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a62b2e740 1 Processor -- start 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5db0a0 0x56519a9cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9c9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a62b2e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a603f60 0x56519a9c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56519a9c74c0 con 0x56519a5e3280 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56519a9c6df0 con 0x56519a5db0a0 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a62b2e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56519a9c7b90 con 0x56519a603f60 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a62b2e740 10 client.0 _ll_register_callbacks cb 0x7f0a4c031450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a59b54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a59b54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9c9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54684/0 (socket says 172.21.15.16:54684) 2024-08-28T19:21:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.967+0000 7f0a59b54640 1 -- 172.21.15.16:0/2744456219 learned_addr learned my addr 172.21.15.16:0/2744456219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a5a355640 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5db0a0 0x56519a9cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a5ab56640 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a603f60 0x56519a9c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a59b54640 1 -- 172.21.15.16:0/2744456219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a603f60 msgr2=0x56519a9c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a59b54640 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a603f60 0x56519a9c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a59b54640 1 -- 172.21.15.16:0/2744456219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5db0a0 msgr2=0x56519a9cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a59b54640 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5db0a0 0x56519a9cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a59b54640 1 -- 172.21.15.16:0/2744456219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56519a9c9f40 con 0x56519a5e3280 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a5ab56640 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a603f60 0x56519a9c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a59b54640 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9c9410 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0a440e2990 tx=0x7f0a440e2de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb75359f8a55e243 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2744456219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a440aaff0 con 0x56519a5e3280 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2744456219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0a440b14c0 con 0x56519a5e3280 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2744456219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a440ab390 con 0x56519a5e3280 2024-08-28T19:21:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2744456219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0a440b00f0 con 0x56519a5e3280 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a62b2e740 10 client.15717 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2744456219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56519a9cf920 con 0x56519a5e3280 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.968+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2744456219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a440ab390 con 0x56519a5e3280 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.969+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2744456219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0a440a7b10 con 0x56519a5e3280 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.969+0000 7f0a62b2e740 10 client.15717 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.969+0000 7f0a62b2e740 10 client.15717 resolve_mds: resolved name 'f' to daemon mds.f 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.969+0000 7f0a62b2e740 20 client.15717 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.969+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56519a597c90 0x56519aa359c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.969+0000 7f0a62b2e740 4 client.15717 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.969+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2744456219 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "status"}) -- 0x56519a91b520 con 0x56519a597c90 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.969+0000 7f0a5a355640 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56519a597c90 0x56519aa359c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.969+0000 7f0a3dffb640 10 client.15717 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.969+0000 7f0a5a355640 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56519a597c90 0x56519aa359c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c097ffcc73d8ab02 server_cookie=fdcd6d757c689af0 in_seq=0 out_seq=0 2024-08-28T19:21:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a3dffb640 1 -- 172.21.15.16:0/2744456219 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56519a91b520 con 0x56519a597c90 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a3dffb640 10 client.15717 handle_command_reply: tid=0 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 1 client.15717 shutdown 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2744456219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 msgr2=0x56519a9c9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9c9410 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0a440e2990 tx=0x7f0a440e2de0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2744456219 shutdown_connections 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56519a597c90 0x56519aa359c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519a603f60 0x56519a9c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519a5e3280 0x56519a9c9410 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a3dffb640 0 client.15717 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 1 --2- 172.21.15.16:0/2744456219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519a5db0a0 0x56519a9cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2744456219 >> 172.21.15.16:0/2744456219 conn(0x56519a5f8350 msgr2=0x56519a9b4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2744456219 shutdown_connections 2024-08-28T19:21:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 1 -- 172.21.15.16:0/2744456219 wait complete. 2024-08-28T19:21:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.970+0000 7f0a62b2e740 20 client.15717 trim_cache size 0 max 16384 2024-08-28T19:21:19.977 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:21:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:21:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-08-28T19:21:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24619, 2024-08-28T19:21:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:21:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:21:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:21:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 128.329118671, 2024-08-28T19:21:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 29, 2024-08-28T19:21:19.978 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:21:19.978 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-08-28T19:21:19.978 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 193.39701486800001 2024-08-28T19:21:19.978 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:21:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/3387383625 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a400b4c40 msgr2=0x7f0a400b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a400b4c40 0x7f0a400b7150 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c006010 tx=0x7f0a4c0095c0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/3387383625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a54105120 msgr2=0x7f0a5419a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a54105120 0x7f0a5419a800 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0a4400d950 tx=0x7f0a4400de10 comp rx=0 tx=0).stop 2024-08-28T19:21:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/3387383625 shutdown_connections 2024-08-28T19:21:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a400b4c40 0x7f0a400b7150 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a5410dc30 0x7f0a5419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a54105120 0x7f0a5419a800 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 --2- 172.21.15.16:0/3387383625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a541047d0 0x7f0a5419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/3387383625 >> 172.21.15.16:0/3387383625 conn(0x7f0a54079510 msgr2=0x7f0a54100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/3387383625 shutdown_connections 2024-08-28T19:21:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:19.971+0000 7f0a5c5b9640 1 -- 172.21.15.16:0/3387383625 wait complete. 2024-08-28T19:21:20.635 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.c status 2024-08-28T19:21:20.861 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:20 smithi028 ceph-mon[21769]: pgmap v348: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-08-28T19:21:20.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:20 smithi115 ceph-mon[22184]: pgmap v348: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-08-28T19:21:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:21.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:20 smithi016 ceph-mon[18556]: pgmap v348: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-08-28T19:21:21.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.370+0000 7f5c041fd640 1 -- 172.21.15.16:0/3422511088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bfc075550 msgr2=0x7f5bfc075960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:21.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.370+0000 7f5c041fd640 1 --2- 172.21.15.16:0/3422511088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bfc075550 0x7f5bfc075960 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009900 tx=0x7f5bf802a410 comp rx=0 tx=0).stop 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.370+0000 7f5c041fd640 1 -- 172.21.15.16:0/3422511088 shutdown_connections 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.370+0000 7f5c041fd640 1 --2- 172.21.15.16:0/3422511088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bfc10fec0 0x7f5bfc1122e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.370+0000 7f5c041fd640 1 --2- 172.21.15.16:0/3422511088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bfc075ea0 0x7f5bfc10f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.370+0000 7f5c041fd640 1 --2- 172.21.15.16:0/3422511088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bfc075550 0x7f5bfc075960 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.370+0000 7f5c041fd640 1 -- 172.21.15.16:0/3422511088 >> 172.21.15.16:0/3422511088 conn(0x7f5bfc07b6b0 msgr2=0x7f5bfc07bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.370+0000 7f5c041fd640 1 -- 172.21.15.16:0/3422511088 shutdown_connections 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.370+0000 7f5c041fd640 1 -- 172.21.15.16:0/3422511088 wait complete. 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c041fd640 1 Processor -- start 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c041fd640 1 -- start start 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c041fd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bfc075550 0x7f5bfc19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c041fd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bfc075ea0 0x7f5bfc19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c041fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bfc10fec0 0x7f5bfc1a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c041fd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc115a30 con 0x7f5bfc075ea0 2024-08-28T19:21:21.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c041fd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc1158b0 con 0x7f5bfc10fec0 2024-08-28T19:21:21.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c041fd640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc115bb0 con 0x7f5bfc075550 2024-08-28T19:21:21.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c01798640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bfc075ea0 0x7f5bfc19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c01798640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bfc075ea0 0x7f5bfc19ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54710/0 (socket says 172.21.15.16:54710) 2024-08-28T19:21:21.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c01798640 1 -- 172.21.15.16:0/804810361 learned_addr learned my addr 172.21.15.16:0/804810361 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:21.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c01f99640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bfc075550 0x7f5bfc19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.371+0000 7f5c0279a640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bfc10fec0 0x7f5bfc1a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5c01798640 1 -- 172.21.15.16:0/804810361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bfc075550 msgr2=0x7f5bfc19c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5c01798640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bfc075550 0x7f5bfc19c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5c01798640 1 -- 172.21.15.16:0/804810361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bfc10fec0 msgr2=0x7f5bfc1a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5c01798640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bfc10fec0 0x7f5bfc1a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5c01798640 1 -- 172.21.15.16:0/804810361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bfc1a23f0 con 0x7f5bfc075ea0 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5c0279a640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bfc10fec0 0x7f5bfc1a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5c01798640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bfc075ea0 0x7f5bfc19ca20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5bec005fd0 tx=0x7f5bec00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5bf2ffd640 1 -- 172.21.15.16:0/804810361 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec017070 con 0x7f5bfc075ea0 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5bf2ffd640 1 -- 172.21.15.16:0/804810361 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bec009c30 con 0x7f5bfc075ea0 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5c041fd640 1 -- 172.21.15.16:0/804810361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bfc1a26e0 con 0x7f5bfc075ea0 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5bf2ffd640 1 -- 172.21.15.16:0/804810361 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec003d30 con 0x7f5bfc075ea0 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.372+0000 7f5c041fd640 1 -- 172.21.15.16:0/804810361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bfc1a8e40 con 0x7f5bfc075ea0 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.375+0000 7f5c0a772740 1 Processor -- start 2024-08-28T19:21:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.375+0000 7f5c0a772740 1 -- start start 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe314f60 0x559cbe6ba7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe2ec0a0 0x559cbe6bacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5bf2ffd640 1 -- 172.21.15.16:0/804810361 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f5bec003ed0 con 0x7f5bfc075ea0 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6bb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c0a772740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559cbe6313f0 con 0x559cbe2f4280 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c0a772740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559cbe620980 con 0x559cbe314f60 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c0a772740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559cbe624930 con 0x559cbe2ec0a0 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c01798640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6bb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c01798640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6bb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54724/0 (socket says 172.21.15.16:54724) 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c01798640 1 -- 172.21.15.16:0/559774565 learned_addr learned my addr 172.21.15.16:0/559774565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c0279a640 1 --2- 172.21.15.16:0/559774565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe314f60 0x559cbe6ba7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c01f99640 1 --2- 172.21.15.16:0/559774565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe2ec0a0 0x559cbe6bacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c01798640 1 -- 172.21.15.16:0/559774565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe2ec0a0 msgr2=0x559cbe6bacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c01798640 1 --2- 172.21.15.16:0/559774565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe2ec0a0 0x559cbe6bacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c01798640 1 -- 172.21.15.16:0/559774565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe314f60 msgr2=0x559cbe6ba7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c01798640 1 --2- 172.21.15.16:0/559774565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe314f60 0x559cbe6ba7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c01798640 1 -- 172.21.15.16:0/559774565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cbe380540 con 0x559cbe2f4280 2024-08-28T19:21:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5c01798640 1 --2- 172.21.15.16:0/559774565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6bb230 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0e2910 tx=0x7f5bec0ebb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2283caf16b2e21d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.376+0000 7f5bf2ffd640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5be80b4c10 0x7f5be80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.377+0000 7f5be7fff640 1 -- 172.21.15.16:0/559774565 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec0f8070 con 0x559cbe2f4280 2024-08-28T19:21:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.377+0000 7f5c0279a640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5be80b4c10 0x7f5be80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.377+0000 7f5bf2ffd640 1 -- 172.21.15.16:0/804810361 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f5bec0e5480 con 0x7f5bfc075ea0 2024-08-28T19:21:21.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.377+0000 7f5be7fff640 1 -- 172.21.15.16:0/559774565 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bec012680 con 0x559cbe2f4280 2024-08-28T19:21:21.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.377+0000 7f5be7fff640 1 -- 172.21.15.16:0/559774565 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec01e120 con 0x559cbe2f4280 2024-08-28T19:21:21.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0a772740 1 -- 172.21.15.16:0/559774565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 msgr2=0x559cbe6bb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:21.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0279a640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5be80b4c10 0x7f5be80b7120 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4005fd0 tx=0x7f5bf40085b0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:21.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0a772740 1 --2- 172.21.15.16:0/559774565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6bb230 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0e2910 tx=0x7f5bec0ebb00 comp rx=0 tx=0).stop 2024-08-28T19:21:21.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0a772740 1 -- 172.21.15.16:0/559774565 shutdown_connections 2024-08-28T19:21:21.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0a772740 1 --2- 172.21.15.16:0/559774565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6bb230 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0a772740 1 --2- 172.21.15.16:0/559774565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe2ec0a0 0x559cbe6bacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0a772740 1 --2- 172.21.15.16:0/559774565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe314f60 0x559cbe6ba7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0a772740 1 -- 172.21.15.16:0/559774565 >> 172.21.15.16:0/559774565 conn(0x559cbe309350 msgr2=0x559cbe6b86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0a772740 1 -- 172.21.15.16:0/559774565 shutdown_connections 2024-08-28T19:21:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0a772740 1 -- 172.21.15.16:0/559774565 wait complete. 2024-08-28T19:21:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0a772740 1 -- start start 2024-08-28T19:21:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.378+0000 7f5c0a772740 1 Processor -- start 2024-08-28T19:21:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe2ec0a0 0x559cbe6c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6c6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe314f60 0x559cbe6d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c0a772740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559cbe4835f0 con 0x559cbe2f4280 2024-08-28T19:21:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c0a772740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559cbe5a7600 con 0x559cbe2ec0a0 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c0a772740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559cbe6c8f90 con 0x559cbe314f60 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c0a772740 10 client.0 _ll_register_callbacks cb 0x559cbe5cdaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c01798640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c01798640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6c6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54726/0 (socket says 172.21.15.16:54726) 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c01798640 1 -- 172.21.15.16:0/4267815273 learned_addr learned my addr 172.21.15.16:0/4267815273 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c0279a640 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe314f60 0x559cbe6d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c01f99640 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe2ec0a0 0x559cbe6c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c01798640 1 -- 172.21.15.16:0/4267815273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe314f60 msgr2=0x559cbe6d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c01798640 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe314f60 0x559cbe6d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c01798640 1 -- 172.21.15.16:0/4267815273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe2ec0a0 msgr2=0x559cbe6c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c01798640 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe2ec0a0 0x559cbe6c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c01798640 1 -- 172.21.15.16:0/4267815273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cbe5e07b0 con 0x559cbe2f4280 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c01f99640 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe2ec0a0 0x559cbe6c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c0279a640 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe314f60 0x559cbe6d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5c01798640 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6c6340 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0e2a20 tx=0x7f5bec0e8da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e96b43495d9599 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5be57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.379+0000 7f5be57fa640 1 -- 172.21.15.16:0/4267815273 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec0a7780 con 0x559cbe2f4280 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5be57fa640 1 -- 172.21.15.16:0/4267815273 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f5bec0a7eb0 con 0x559cbe2f4280 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5c0a772740 1 -- 172.21.15.16:0/4267815273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bec012680 con 0x559cbe2f4280 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5be57fa640 1 -- 172.21.15.16:0/4267815273 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5bec0ab210 con 0x559cbe2f4280 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5c0a772740 10 client.15729 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5c0a772740 1 -- 172.21.15.16:0/4267815273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559cbe6d4c70 con 0x559cbe2f4280 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5be57fa640 1 -- 172.21.15.16:0/4267815273 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bec012680 con 0x559cbe2f4280 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5be57fa640 1 -- 172.21.15.16:0/4267815273 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5bec0e5150 con 0x559cbe2f4280 2024-08-28T19:21:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5c0a772740 10 client.15729 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5c0a772740 10 client.15729 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5c0a772740 20 client.15729 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5c0a772740 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x559cbe2d38e0 0x559cbe6d8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5c0a772740 4 client.15729 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.380+0000 7f5c0a772740 1 -- 172.21.15.16:0/4267815273 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559cbe633340 con 0x559cbe2d38e0 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.381+0000 7f5c01f99640 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x559cbe2d38e0 0x559cbe6d8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.381+0000 7f5c01f99640 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x559cbe2d38e0 0x559cbe6d8140 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=18ecb847d6ba7808 server_cookie=31d9169dff541a1a in_seq=0 out_seq=0 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.381+0000 7f5be57fa640 10 client.15729 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5be57fa640 1 -- 172.21.15.16:0/4267815273 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559cbe633340 con 0x559cbe2d38e0 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5be57fa640 10 client.15729 handle_command_reply: tid=0 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 1 client.15729 shutdown 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 1 -- 172.21.15.16:0/4267815273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 msgr2=0x559cbe6c6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6c6340 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0e2a20 tx=0x7f5bec0e8da0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 1 -- 172.21.15.16:0/4267815273 shutdown_connections 2024-08-28T19:21:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x559cbe2d38e0 0x559cbe6d8140 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe314f60 0x559cbe6d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6c6340 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5be57fa640 0 client.15729 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:21:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 1 --2- 172.21.15.16:0/4267815273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe2ec0a0 0x559cbe6c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 1 -- 172.21.15.16:0/4267815273 >> 172.21.15.16:0/4267815273 conn(0x559cbe309350 msgr2=0x559cbe6bd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 1 -- 172.21.15.16:0/4267815273 shutdown_connections 2024-08-28T19:21:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 1 -- 172.21.15.16:0/4267815273 wait complete. 2024-08-28T19:21:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.382+0000 7f5c0a772740 20 client.15729 trim_cache size 0 max 16384 2024-08-28T19:21:21.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.401+0000 7f5c0a772740 1 Processor -- start 2024-08-28T19:21:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.401+0000 7f5c0a772740 1 -- start start 2024-08-28T19:21:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.401+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2ec0a0 0x559cbe6d8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.401+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe2f4280 0x559cbe6d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.401+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe314f60 0x559cbe6bbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.401+0000 7f5c0a772740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559cbe6d6480 con 0x559cbe2ec0a0 2024-08-28T19:21:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.401+0000 7f5c0a772740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559cbe6d6b50 con 0x559cbe314f60 2024-08-28T19:21:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.401+0000 7f5c0a772740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559cbe6d7220 con 0x559cbe2f4280 2024-08-28T19:21:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.401+0000 7f5c01f99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2ec0a0 0x559cbe6d8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.401+0000 7f5c01f99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2ec0a0 0x559cbe6d8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54736/0 (socket says 172.21.15.16:54736) 2024-08-28T19:21:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.401+0000 7f5c01f99640 1 -- 172.21.15.16:0/193206667 learned_addr learned my addr 172.21.15.16:0/193206667 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5c0279a640 1 --2- 172.21.15.16:0/193206667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe314f60 0x559cbe6bbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5c01798640 1 --2- 172.21.15.16:0/193206667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe2f4280 0x559cbe6d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5c01f99640 1 -- 172.21.15.16:0/193206667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe2f4280 msgr2=0x559cbe6d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5c01f99640 1 --2- 172.21.15.16:0/193206667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe2f4280 0x559cbe6d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5c01f99640 1 -- 172.21.15.16:0/193206667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe314f60 msgr2=0x559cbe6bbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5c01f99640 1 --2- 172.21.15.16:0/193206667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe314f60 0x559cbe6bbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5c01f99640 1 -- 172.21.15.16:0/193206667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cbe6bb5e0 con 0x559cbe2ec0a0 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5c0279a640 1 --2- 172.21.15.16:0/193206667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe314f60 0x559cbe6bbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5c01f99640 1 --2- 172.21.15.16:0/193206667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2ec0a0 0x559cbe6d8140 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009900 tx=0x7f5bf8004310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921f89aee094bb04 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5be77fe640 1 -- 172.21.15.16:0/193206667 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8035040 con 0x559cbe2ec0a0 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5be77fe640 1 -- 172.21.15.16:0/193206667 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf8049070 con 0x559cbe2ec0a0 2024-08-28T19:21:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.402+0000 7f5be77fe640 1 -- 172.21.15.16:0/193206667 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8004c10 con 0x559cbe2ec0a0 2024-08-28T19:21:21.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.403+0000 7f5c0a772740 1 -- 172.21.15.16:0/193206667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2ec0a0 msgr2=0x559cbe6d8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:21.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.403+0000 7f5c0a772740 1 --2- 172.21.15.16:0/193206667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2ec0a0 0x559cbe6d8140 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009900 tx=0x7f5bf8004310 comp rx=0 tx=0).stop 2024-08-28T19:21:21.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.403+0000 7f5c0a772740 1 -- 172.21.15.16:0/193206667 shutdown_connections 2024-08-28T19:21:21.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.403+0000 7f5c0a772740 1 --2- 172.21.15.16:0/193206667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe314f60 0x559cbe6bbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.403+0000 7f5c0a772740 1 --2- 172.21.15.16:0/193206667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe2f4280 0x559cbe6d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.403+0000 7f5c0a772740 1 --2- 172.21.15.16:0/193206667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2ec0a0 0x559cbe6d8140 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.403+0000 7f5c0a772740 1 -- 172.21.15.16:0/193206667 >> 172.21.15.16:0/193206667 conn(0x559cbe309350 msgr2=0x559cbe6bb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.403+0000 7f5c0a772740 1 -- 172.21.15.16:0/193206667 shutdown_connections 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.403+0000 7f5c0a772740 1 -- 172.21.15.16:0/193206667 wait complete. 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.403+0000 7f5c0a772740 1 -- start start 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.403+0000 7f5c0a772740 1 Processor -- start 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe2ec0a0 0x559cbe6e0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6d9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c0a772740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe314f60 0x559cbe6da440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c0a772740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559cbe6d7fc0 con 0x559cbe2f4280 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c0a772740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559cbe6d78f0 con 0x559cbe2ec0a0 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c0a772740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559cbe6d8690 con 0x559cbe314f60 2024-08-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c0a772740 10 client.0 _ll_register_callbacks cb 0x7f5bf8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:21.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c01798640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6d9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c01798640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6d9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54738/0 (socket says 172.21.15.16:54738) 2024-08-28T19:21:21.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c01798640 1 -- 172.21.15.16:0/1650258726 learned_addr learned my addr 172.21.15.16:0/1650258726 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:21.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c01f99640 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe2ec0a0 0x559cbe6e0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c01798640 1 -- 172.21.15.16:0/1650258726 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe314f60 msgr2=0x559cbe6da440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:21.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c0279a640 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe314f60 0x559cbe6da440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c01798640 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe314f60 0x559cbe6da440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c01798640 1 -- 172.21.15.16:0/1650258726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe2ec0a0 msgr2=0x559cbe6e0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:21.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c01798640 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe2ec0a0 0x559cbe6e0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.404+0000 7f5c01798640 1 -- 172.21.15.16:0/1650258726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cbe6daa50 con 0x559cbe2f4280 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5c01798640 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6d9f00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0ab850 tx=0x7f5bec0b4cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f934ab27fe6f60 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5be57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5be57fa640 1 -- 172.21.15.16:0/1650258726 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec0ab690 con 0x559cbe2f4280 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5be57fa640 1 -- 172.21.15.16:0/1650258726 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f5bec0b4750 con 0x559cbe2f4280 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5c0a772740 1 -- 172.21.15.16:0/1650258726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bf8049070 con 0x559cbe2f4280 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5be57fa640 1 -- 172.21.15.16:0/1650258726 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5bec0a7780 con 0x559cbe2f4280 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5c0a772740 10 client.15735 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5c0a772740 1 -- 172.21.15.16:0/1650258726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559cbe6e0430 con 0x559cbe2f4280 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5be57fa640 1 -- 172.21.15.16:0/1650258726 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf8049070 con 0x559cbe2f4280 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5be57fa640 1 -- 172.21.15.16:0/1650258726 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5bec0abb20 con 0x559cbe2f4280 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5c0a772740 10 client.15735 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5c0a772740 10 client.15735 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5c0a772740 20 client.15735 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5c0a772740 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x559cbe2a8c90 0x559cbe746a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5c0a772740 4 client.15735 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.405+0000 7f5c0a772740 1 -- 172.21.15.16:0/1650258726 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "status"}) -- 0x559cbe62ede0 con 0x559cbe2a8c90 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.406+0000 7f5c01f99640 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x559cbe2a8c90 0x559cbe746a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.406+0000 7f5be57fa640 10 client.15735 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.406+0000 7f5c01f99640 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x559cbe2a8c90 0x559cbe746a40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=64134dccce65d182 server_cookie=272a9f5221d9af2d in_seq=0 out_seq=0 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.406+0000 7f5be57fa640 1 -- 172.21.15.16:0/1650258726 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559cbe62ede0 con 0x559cbe2a8c90 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.406+0000 7f5be57fa640 10 client.15735 handle_command_reply: tid=0 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.406+0000 7f5c0a772740 1 client.15735 shutdown 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c0a772740 1 -- 172.21.15.16:0/1650258726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 msgr2=0x559cbe6d9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c0a772740 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6d9f00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0ab850 tx=0x7f5bec0b4cb0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c0a772740 1 -- 172.21.15.16:0/1650258726 shutdown_connections 2024-08-28T19:21:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c0a772740 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x559cbe2a8c90 0x559cbe746a40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c0a772740 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559cbe314f60 0x559cbe6da440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c0a772740 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559cbe2f4280 0x559cbe6d9f00 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5be57fa640 0 client.15735 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:21:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c0a772740 1 --2- 172.21.15.16:0/1650258726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cbe2ec0a0 0x559cbe6e0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c0a772740 1 -- 172.21.15.16:0/1650258726 >> 172.21.15.16:0/1650258726 conn(0x559cbe309350 msgr2=0x559cbe6c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c0a772740 1 -- 172.21.15.16:0/1650258726 shutdown_connections 2024-08-28T19:21:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c0a772740 1 -- 172.21.15.16:0/1650258726 wait complete. 2024-08-28T19:21:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c0a772740 20 client.15735 trim_cache size 0 max 16384 2024-08-28T19:21:21.414 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:21:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-08-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24625, 2024-08-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 130.775938115, 2024-08-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 29, 2024-08-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:21:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-08-28T19:21:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 192.34473840499999 2024-08-28T19:21:21.416 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:21:21.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c041fd640 1 -- 172.21.15.16:0/804810361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5be80b4c10 msgr2=0x7f5be80b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c041fd640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5be80b4c10 0x7f5be80b7120 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4005fd0 tx=0x7f5bf40085b0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c041fd640 1 -- 172.21.15.16:0/804810361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bfc075ea0 msgr2=0x7f5bfc19ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.407+0000 7f5c041fd640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bfc075ea0 0x7f5bfc19ca20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5bec005fd0 tx=0x7f5bec00dde0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.408+0000 7f5c041fd640 1 -- 172.21.15.16:0/804810361 shutdown_connections 2024-08-28T19:21:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.408+0000 7f5c041fd640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5be80b4c10 0x7f5be80b7120 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.408+0000 7f5c041fd640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bfc10fec0 0x7f5bfc1a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.408+0000 7f5c041fd640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bfc075ea0 0x7f5bfc19ca20 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.408+0000 7f5c041fd640 1 --2- 172.21.15.16:0/804810361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bfc075550 0x7f5bfc19c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.408+0000 7f5c041fd640 1 -- 172.21.15.16:0/804810361 >> 172.21.15.16:0/804810361 conn(0x7f5bfc07b6b0 msgr2=0x7f5bfc109a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.408+0000 7f5c041fd640 1 -- 172.21.15.16:0/804810361 shutdown_connections 2024-08-28T19:21:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:21.408+0000 7f5c041fd640 1 -- 172.21.15.16:0/804810361 wait complete. 2024-08-28T19:21:22.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.h status 2024-08-28T19:21:22.453 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:22.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:22 smithi016 ceph-mon[18556]: pgmap v349: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 853 B/s wr, 12 op/s 2024-08-28T19:21:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:22 smithi115 ceph-mon[22184]: pgmap v349: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 853 B/s wr, 12 op/s 2024-08-28T19:21:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.948+0000 7f471c0c2640 1 -- 172.21.15.16:0/913184606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4714103900 msgr2=0x7f4714103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.948+0000 7f471c0c2640 1 --2- 172.21.15.16:0/913184606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4714103900 0x7f4714103d10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4710009900 tx=0x7f471002a410 comp rx=0 tx=0).stop 2024-08-28T19:21:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.948+0000 7f471c0c2640 1 -- 172.21.15.16:0/913184606 shutdown_connections 2024-08-28T19:21:22.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.948+0000 7f471c0c2640 1 --2- 172.21.15.16:0/913184606 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4714109b90 0x7f47141103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.948+0000 7f471c0c2640 1 --2- 172.21.15.16:0/913184606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109240 0x7f4714109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.948+0000 7f471c0c2640 1 --2- 172.21.15.16:0/913184606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4714103900 0x7f4714103d10 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.948+0000 7f471c0c2640 1 -- 172.21.15.16:0/913184606 >> 172.21.15.16:0/913184606 conn(0x7f471407b5f0 msgr2=0x7f471407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:22.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.948+0000 7f471c0c2640 1 -- 172.21.15.16:0/913184606 shutdown_connections 2024-08-28T19:21:22.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.948+0000 7f471c0c2640 1 -- 172.21.15.16:0/913184606 wait complete. 2024-08-28T19:21:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.948+0000 7f471c0c2640 1 Processor -- start 2024-08-28T19:21:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.948+0000 7f471c0c2640 1 -- start start 2024-08-28T19:21:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f471c0c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4714103900 0x7f47140754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f471c0c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109240 0x7f47140759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f471c0c2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4714109b90 0x7f4714075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f471c0c2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4714113850 con 0x7f4714103900 2024-08-28T19:21:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f471c0c2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47141136d0 con 0x7f4714109240 2024-08-28T19:21:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f471c0c2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f47141139d0 con 0x7f4714109b90 2024-08-28T19:21:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f4719e5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4714103900 0x7f47140754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f4719e5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4714103900 0x7f47140754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54752/0 (socket says 172.21.15.16:54752) 2024-08-28T19:21:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f4719e5e640 1 -- 172.21.15.16:0/1850486584 learned_addr learned my addr 172.21.15.16:0/1850486584 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f471a65f640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4714109b90 0x7f4714075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f471965d640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109240 0x7f47140759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f4719e5e640 1 -- 172.21.15.16:0/1850486584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4714109b90 msgr2=0x7f4714075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f4719e5e640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4714109b90 0x7f4714075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f4719e5e640 1 -- 172.21.15.16:0/1850486584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109240 msgr2=0x7f47140759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f4719e5e640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109240 0x7f47140759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f4719e5e640 1 -- 172.21.15.16:0/1850486584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47141a8500 con 0x7f4714103900 2024-08-28T19:21:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.949+0000 7f4719e5e640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4714103900 0x7f47140754b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4710005940 tx=0x7f47100040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:22.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.950+0000 7f470affd640 1 -- 172.21.15.16:0/1850486584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471003f030 con 0x7f4714103900 2024-08-28T19:21:22.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.950+0000 7f470affd640 1 -- 172.21.15.16:0/1850486584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47100044c0 con 0x7f4714103900 2024-08-28T19:21:22.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.950+0000 7f471c0c2640 1 -- 172.21.15.16:0/1850486584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47141a8790 con 0x7f4714103900 2024-08-28T19:21:22.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.950+0000 7f471c0c2640 1 -- 172.21.15.16:0/1850486584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47141a8cc0 con 0x7f4714103900 2024-08-28T19:21:22.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.950+0000 7f470affd640 1 -- 172.21.15.16:0/1850486584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4710035070 con 0x7f4714103900 2024-08-28T19:21:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.952+0000 7f4722637740 1 Processor -- start 2024-08-28T19:21:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.952+0000 7f4722637740 1 -- start start 2024-08-28T19:21:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.952+0000 7f470affd640 1 -- 172.21.15.16:0/1850486584 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f4710042030 con 0x7f4714103900 2024-08-28T19:21:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4722637740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811915ef60 0x5581195047b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4722637740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x558119504cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4722637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x558119505230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4722637740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55811947b3f0 con 0x5581191360a0 2024-08-28T19:21:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4722637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55811946a980 con 0x55811913e280 2024-08-28T19:21:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4722637740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55811946e930 con 0x55811915ef60 2024-08-28T19:21:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4719e5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x558119504cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f471a65f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811915ef60 0x5581195047b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4719e5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x558119504cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54764/0 (socket says 172.21.15.16:54764) 2024-08-28T19:21:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f471a65f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811915ef60 0x5581195047b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:37288/0 (socket says 172.21.15.16:37288) 2024-08-28T19:21:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4719e5e640 1 -- 172.21.15.16:0/1415009098 learned_addr learned my addr 172.21.15.16:0/1415009098 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f471965d640 1 --2- 172.21.15.16:0/1415009098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x558119505230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4719e5e640 1 -- 172.21.15.16:0/1415009098 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811915ef60 msgr2=0x5581195047b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4719e5e640 1 --2- 172.21.15.16:0/1415009098 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811915ef60 0x5581195047b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4719e5e640 1 -- 172.21.15.16:0/1415009098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 msgr2=0x558119505230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4719e5e640 1 --2- 172.21.15.16:0/1415009098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x558119505230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f4719e5e640 1 -- 172.21.15.16:0/1415009098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581191ca540 con 0x5581191360a0 2024-08-28T19:21:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f471a65f640 1 --2- 172.21.15.16:0/1415009098 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811915ef60 0x5581195047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f471965d640 1 --2- 172.21.15.16:0/1415009098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x558119505230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.953+0000 7f470affd640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47000b4c90 0x7f47000b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.954+0000 7f471a65f640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47000b4c90 0x7f47000b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.954+0000 7f4719e5e640 1 --2- 172.21.15.16:0/1415009098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x558119504cf0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4710103090 tx=0x7f4710109f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a9c587ab28c1d3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.954+0000 7f470affd640 1 -- 172.21.15.16:0/1850486584 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f4710105c30 con 0x7f4714103900 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.954+0000 7f46fffff640 1 -- 172.21.15.16:0/1415009098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4710049070 con 0x5581191360a0 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.954+0000 7f46fffff640 1 -- 172.21.15.16:0/1415009098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47100cbb80 con 0x5581191360a0 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.954+0000 7f46fffff640 1 -- 172.21.15.16:0/1415009098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47100b1e60 con 0x5581191360a0 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.955+0000 7f4722637740 1 -- 172.21.15.16:0/1415009098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 msgr2=0x558119504cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.955+0000 7f471a65f640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47000b4c90 0x7f47000b71a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f470c005fd0 tx=0x7f470c009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.955+0000 7f4722637740 1 --2- 172.21.15.16:0/1415009098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x558119504cf0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4710103090 tx=0x7f4710109f00 comp rx=0 tx=0).stop 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.955+0000 7f4722637740 1 -- 172.21.15.16:0/1415009098 shutdown_connections 2024-08-28T19:21:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.955+0000 7f4722637740 1 --2- 172.21.15.16:0/1415009098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x558119505230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.955+0000 7f4722637740 1 --2- 172.21.15.16:0/1415009098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x558119504cf0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.955+0000 7f4722637740 1 --2- 172.21.15.16:0/1415009098 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811915ef60 0x5581195047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.955+0000 7f4722637740 1 -- 172.21.15.16:0/1415009098 >> 172.21.15.16:0/1415009098 conn(0x558119153350 msgr2=0x5581195026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.955+0000 7f4722637740 1 -- 172.21.15.16:0/1415009098 shutdown_connections 2024-08-28T19:21:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.955+0000 7f4722637740 1 -- 172.21.15.16:0/1415009098 wait complete. 2024-08-28T19:21:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.955+0000 7f4722637740 1 -- start start 2024-08-28T19:21:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4722637740 1 Processor -- start 2024-08-28T19:21:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4722637740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x55811950fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4722637740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811913e280 0x558119510370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4722637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811915ef60 0x5581195108d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4722637740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5581192cd5f0 con 0x5581191360a0 2024-08-28T19:21:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4722637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5581193f1600 con 0x55811915ef60 2024-08-28T19:21:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4722637740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558119512d50 con 0x55811913e280 2024-08-28T19:21:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4722637740 10 client.0 _ll_register_callbacks cb 0x558119417af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4719e5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x55811950fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4719e5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x55811950fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54776/0 (socket says 172.21.15.16:54776) 2024-08-28T19:21:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4719e5e640 1 -- 172.21.15.16:0/1342535349 learned_addr learned my addr 172.21.15.16:0/1342535349 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f471965d640 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811913e280 0x558119510370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f4719e5e640 1 -- 172.21.15.16:0/1342535349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811913e280 msgr2=0x558119510370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.956+0000 7f471a65f640 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811915ef60 0x5581195108d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f4719e5e640 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811913e280 0x558119510370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f4719e5e640 1 -- 172.21.15.16:0/1342535349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811915ef60 msgr2=0x5581195108d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f4719e5e640 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811915ef60 0x5581195108d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f4719e5e640 1 -- 172.21.15.16:0/1342535349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55811942a7b0 con 0x5581191360a0 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f4719e5e640 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x55811950fe30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f47101036d0 tx=0x7f471003b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a337f7eed21343d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f46fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f46fd7fa640 1 -- 172.21.15.16:0/1342535349 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4710049040 con 0x5581191360a0 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f46fd7fa640 1 -- 172.21.15.16:0/1342535349 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f47100c8300 con 0x5581191360a0 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f4722637740 1 -- 172.21.15.16:0/1342535349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47100cbb80 con 0x5581191360a0 2024-08-28T19:21:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f46fd7fa640 1 -- 172.21.15.16:0/1342535349 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f47100b1e60 con 0x5581191360a0 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f4722637740 10 client.15747 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f4722637740 1 -- 172.21.15.16:0/1342535349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55811951e940 con 0x5581191360a0 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f46fd7fa640 1 -- 172.21.15.16:0/1342535349 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47100cbb80 con 0x5581191360a0 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.957+0000 7f46fd7fa640 1 -- 172.21.15.16:0/1342535349 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f47100b71a0 con 0x5581191360a0 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.958+0000 7f4722637740 10 client.15747 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.958+0000 7f4722637740 10 client.15747 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.958+0000 7f4722637740 20 client.15747 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.958+0000 7f4722637740 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55811911d8e0 0x558119521f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.958+0000 7f4722637740 4 client.15747 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.958+0000 7f4722637740 1 -- 172.21.15.16:0/1342535349 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55811947d340 con 0x55811911d8e0 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.958+0000 7f471965d640 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55811911d8e0 0x558119521f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.958+0000 7f471965d640 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55811911d8e0 0x558119521f70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8921d856757f24b6 server_cookie=82b3c6fec6ce12ad in_seq=0 out_seq=0 2024-08-28T19:21:22.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.958+0000 7f46fd7fa640 10 client.15747 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.959+0000 7f46fd7fa640 1 -- 172.21.15.16:0/1342535349 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55811947d340 con 0x55811911d8e0 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.959+0000 7f46fd7fa640 10 client.15747 handle_command_reply: tid=0 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.959+0000 7f4722637740 1 client.15747 shutdown 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.959+0000 7f4722637740 1 -- 172.21.15.16:0/1342535349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 msgr2=0x55811950fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.959+0000 7f4722637740 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x55811950fe30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f47101036d0 tx=0x7f471003b460 comp rx=0 tx=0).stop 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.960+0000 7f4722637740 1 -- 172.21.15.16:0/1342535349 shutdown_connections 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.960+0000 7f4722637740 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55811911d8e0 0x558119521f70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.960+0000 7f4722637740 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811915ef60 0x5581195108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.960+0000 7f4722637740 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55811913e280 0x558119510370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.960+0000 7f4722637740 1 --2- 172.21.15.16:0/1342535349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5581191360a0 0x55811950fe30 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.960+0000 7f46fd7fa640 0 client.15747 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.960+0000 7f4722637740 1 -- 172.21.15.16:0/1342535349 >> 172.21.15.16:0/1342535349 conn(0x558119153350 msgr2=0x558119507940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.960+0000 7f4722637740 1 -- 172.21.15.16:0/1342535349 shutdown_connections 2024-08-28T19:21:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.960+0000 7f4722637740 1 -- 172.21.15.16:0/1342535349 wait complete. 2024-08-28T19:21:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.960+0000 7f4722637740 20 client.15747 trim_cache size 0 max 16384 2024-08-28T19:21:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f4722637740 1 Processor -- start 2024-08-28T19:21:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f4722637740 1 -- start start 2024-08-28T19:21:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f4722637740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5581191360a0 0x55811950e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f4722637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x55811950ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f4722637740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119501cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f4722637740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5581195202b0 con 0x55811915ef60 2024-08-28T19:21:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f4722637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558119520980 con 0x55811913e280 2024-08-28T19:21:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f4722637740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558119521050 con 0x5581191360a0 2024-08-28T19:21:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f471a65f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119501cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f471a65f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119501cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54782/0 (socket says 172.21.15.16:54782) 2024-08-28T19:21:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f471a65f640 1 -- 172.21.15.16:0/1396141669 learned_addr learned my addr 172.21.15.16:0/1396141669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.978+0000 7f4719e5e640 1 --2- 172.21.15.16:0/1396141669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5581191360a0 0x55811950e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.979+0000 7f471965d640 1 --2- 172.21.15.16:0/1396141669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x55811950ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.979+0000 7f471a65f640 1 -- 172.21.15.16:0/1396141669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5581191360a0 msgr2=0x55811950e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.979+0000 7f471a65f640 1 --2- 172.21.15.16:0/1396141669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5581191360a0 0x55811950e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.979+0000 7f471a65f640 1 -- 172.21.15.16:0/1396141669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 msgr2=0x55811950ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.979+0000 7f471a65f640 1 --2- 172.21.15.16:0/1396141669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x55811950ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.979+0000 7f471a65f640 1 -- 172.21.15.16:0/1396141669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55811950e330 con 0x55811915ef60 2024-08-28T19:21:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.979+0000 7f471965d640 1 --2- 172.21.15.16:0/1396141669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x55811950ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.979+0000 7f471a65f640 1 --2- 172.21.15.16:0/1396141669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119501cf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f470c010930 tx=0x7f470c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31ddf280fd7c513 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.979+0000 7f46ff7fe640 1 -- 172.21.15.16:0/1396141669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c02c070 con 0x55811915ef60 2024-08-28T19:21:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.979+0000 7f46ff7fe640 1 -- 172.21.15.16:0/1396141669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f470c01edd0 con 0x55811915ef60 2024-08-28T19:21:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.979+0000 7f46ff7fe640 1 -- 172.21.15.16:0/1396141669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c027730 con 0x55811915ef60 2024-08-28T19:21:22.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.980+0000 7f4722637740 1 -- 172.21.15.16:0/1396141669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 msgr2=0x558119501cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.980+0000 7f4722637740 1 --2- 172.21.15.16:0/1396141669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119501cf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f470c010930 tx=0x7f470c016000 comp rx=0 tx=0).stop 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.980+0000 7f4722637740 1 -- 172.21.15.16:0/1396141669 shutdown_connections 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.980+0000 7f4722637740 1 --2- 172.21.15.16:0/1396141669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119501cf0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.980+0000 7f4722637740 1 --2- 172.21.15.16:0/1396141669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x55811950ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.980+0000 7f4722637740 1 --2- 172.21.15.16:0/1396141669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5581191360a0 0x55811950e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.980+0000 7f4722637740 1 -- 172.21.15.16:0/1396141669 >> 172.21.15.16:0/1396141669 conn(0x558119153350 msgr2=0x55811951fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.980+0000 7f4722637740 1 -- 172.21.15.16:0/1396141669 shutdown_connections 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.980+0000 7f4722637740 1 -- 172.21.15.16:0/1396141669 wait complete. 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.980+0000 7f4722637740 1 -- start start 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.980+0000 7f4722637740 1 Processor -- start 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f4722637740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5581191360a0 0x55811952aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f4722637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x558119524050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f4722637740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119524740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f4722637740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558119521df0 con 0x55811915ef60 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f4722637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558119521720 con 0x55811913e280 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f4722637740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5581195224c0 con 0x5581191360a0 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f4722637740 10 client.0 _ll_register_callbacks cb 0x7f4710109cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f471a65f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119524740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f471a65f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119524740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54792/0 (socket says 172.21.15.16:54792) 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f471a65f640 1 -- 172.21.15.16:0/216948476 learned_addr learned my addr 172.21.15.16:0/216948476 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f4719e5e640 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5581191360a0 0x55811952aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f471965d640 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x558119524050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f471a65f640 1 -- 172.21.15.16:0/216948476 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5581191360a0 msgr2=0x55811952aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f471a65f640 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5581191360a0 0x55811952aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f471a65f640 1 -- 172.21.15.16:0/216948476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 msgr2=0x558119524050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f471a65f640 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x558119524050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.981+0000 7f471a65f640 1 -- 172.21.15.16:0/216948476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581195297e0 con 0x55811915ef60 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.982+0000 7f471a65f640 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119524740 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f470c009a00 tx=0x7f470c0106a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89afa84b2fe75f74 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.982+0000 7f46fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.982+0000 7f46fd7fa640 1 -- 172.21.15.16:0/216948476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c035060 con 0x55811915ef60 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.982+0000 7f46fd7fa640 1 -- 172.21.15.16:0/216948476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f470c024040 con 0x55811915ef60 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.982+0000 7f4722637740 1 -- 172.21.15.16:0/216948476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f470c01edd0 con 0x55811915ef60 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.982+0000 7f46fd7fa640 1 -- 172.21.15.16:0/216948476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f470c02c030 con 0x55811915ef60 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.982+0000 7f4722637740 10 client.15753 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.982+0000 7f4722637740 1 -- 172.21.15.16:0/216948476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55811958d570 con 0x55811915ef60 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.982+0000 7f46fd7fa640 1 -- 172.21.15.16:0/216948476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f470c01edd0 con 0x55811915ef60 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.982+0000 7f46fd7fa640 1 -- 172.21.15.16:0/216948476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f470c027a80 con 0x55811915ef60 2024-08-28T19:21:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.983+0000 7f4722637740 10 client.15753 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.983+0000 7f4722637740 10 client.15753 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.983+0000 7f4722637740 20 client.15753 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.983+0000 7f4722637740 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5581190f2c90 0x558119590a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.983+0000 7f4722637740 4 client.15753 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.983+0000 7f4722637740 1 -- 172.21.15.16:0/216948476 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "status"}) -- 0x558119478de0 con 0x5581190f2c90 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.983+0000 7f4719e5e640 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5581190f2c90 0x558119590a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.983+0000 7f4719e5e640 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5581190f2c90 0x558119590a70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3cb5a7f78176f59d server_cookie=79496b7621387869 in_seq=0 out_seq=0 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.983+0000 7f46fd7fa640 10 client.15753 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f46fd7fa640 1 -- 172.21.15.16:0/216948476 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558119478de0 con 0x5581190f2c90 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f46fd7fa640 10 client.15753 handle_command_reply: tid=0 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 1 client.15753 shutdown 2024-08-28T19:21:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 1 -- 172.21.15.16:0/216948476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 msgr2=0x558119524740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119524740 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f470c009a00 tx=0x7f470c0106a0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 1 -- 172.21.15.16:0/216948476 shutdown_connections 2024-08-28T19:21:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5581190f2c90 0x558119590a70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55811915ef60 0x558119524740 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55811913e280 0x558119524050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 1 --2- 172.21.15.16:0/216948476 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5581191360a0 0x55811952aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f46fd7fa640 0 client.15753 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:21:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 1 -- 172.21.15.16:0/216948476 >> 172.21.15.16:0/216948476 conn(0x558119153350 msgr2=0x55811950e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 1 -- 172.21.15.16:0/216948476 shutdown_connections 2024-08-28T19:21:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 1 -- 172.21.15.16:0/216948476 wait complete. 2024-08-28T19:21:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f4722637740 20 client.15753 trim_cache size 0 max 16384 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24635, 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 133.35900811499999, 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 29, 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-08-28T19:21:22.992 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 191.038705687 2024-08-28T19:21:22.993 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:21:22.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f471c0c2640 1 -- 172.21.15.16:0/1850486584 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47000b4c90 msgr2=0x7f47000b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:22.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f471c0c2640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47000b4c90 0x7f47000b71a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f470c005fd0 tx=0x7f470c009500 comp rx=0 tx=0).stop 2024-08-28T19:21:22.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f471c0c2640 1 -- 172.21.15.16:0/1850486584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4714103900 msgr2=0x7f47140754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:22.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.984+0000 7f471c0c2640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4714103900 0x7f47140754b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4710005940 tx=0x7f47100040c0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.985+0000 7f471c0c2640 1 -- 172.21.15.16:0/1850486584 shutdown_connections 2024-08-28T19:21:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.985+0000 7f471c0c2640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4714109b90 0x7f4714075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.985+0000 7f471c0c2640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47000b4c90 0x7f47000b71a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.985+0000 7f471c0c2640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109240 0x7f47140759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.985+0000 7f471c0c2640 1 --2- 172.21.15.16:0/1850486584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4714103900 0x7f47140754b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.985+0000 7f471c0c2640 1 -- 172.21.15.16:0/1850486584 >> 172.21.15.16:0/1850486584 conn(0x7f471407b5f0 msgr2=0x7f47141072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.985+0000 7f471c0c2640 1 -- 172.21.15.16:0/1850486584 shutdown_connections 2024-08-28T19:21:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:22.985+0000 7f471c0c2640 1 -- 172.21.15.16:0/1850486584 wait complete. 2024-08-28T19:21:23.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:22 smithi028 ceph-mon[21769]: pgmap v349: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 853 B/s wr, 12 op/s 2024-08-28T19:21:23.842 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.i status 2024-08-28T19:21:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:24.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.533+0000 7fd50b466640 1 -- 172.21.15.16:0/1409882049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd504109260 msgr2=0x7fd504109670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:24.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.533+0000 7fd50b466640 1 --2- 172.21.15.16:0/1409882049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd504109260 0x7fd504109670 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4009900 tx=0x7fd4f402a410 comp rx=0 tx=0).stop 2024-08-28T19:21:24.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.533+0000 7fd50b466640 1 -- 172.21.15.16:0/1409882049 shutdown_connections 2024-08-28T19:21:24.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.533+0000 7fd50b466640 1 --2- 172.21.15.16:0/1409882049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504109bb0 0x7fd5041103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.533+0000 7fd50b466640 1 --2- 172.21.15.16:0/1409882049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd504109260 0x7fd504109670 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.533+0000 7fd50b466640 1 --2- 172.21.15.16:0/1409882049 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd504103920 0x7fd504103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.533+0000 7fd50b466640 1 -- 172.21.15.16:0/1409882049 >> 172.21.15.16:0/1409882049 conn(0x7fd5040ff3b0 msgr2=0x7fd504101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:24.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.533+0000 7fd50b466640 1 -- 172.21.15.16:0/1409882049 shutdown_connections 2024-08-28T19:21:24.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.533+0000 7fd50b466640 1 -- 172.21.15.16:0/1409882049 wait complete. 2024-08-28T19:21:24.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.534+0000 7fd50b466640 1 Processor -- start 2024-08-28T19:21:24.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.534+0000 7fd50b466640 1 -- start start 2024-08-28T19:21:24.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.534+0000 7fd50b466640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504103920 0x7fd50419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd50b466640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd504109260 0x7fd50419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd50b466640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd504109bb0 0x7fd50419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd50b466640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd504113870 con 0x7fd504109bb0 2024-08-28T19:21:24.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd50b466640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5041136f0 con 0x7fd504103920 2024-08-28T19:21:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd50b466640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd5041139f0 con 0x7fd504109260 2024-08-28T19:21:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd509a03640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd504109bb0 0x7fd50419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd509a03640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd504109bb0 0x7fd50419faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42484/0 (socket says 172.21.15.16:42484) 2024-08-28T19:21:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd509a03640 1 -- 172.21.15.16:0/2343314250 learned_addr learned my addr 172.21.15.16:0/2343314250 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd509202640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504103920 0x7fd50419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd508a01640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd504109260 0x7fd50419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd509a03640 1 -- 172.21.15.16:0/2343314250 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd504109260 msgr2=0x7fd50419a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd509a03640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd504109260 0x7fd50419a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd509a03640 1 -- 172.21.15.16:0/2343314250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504103920 msgr2=0x7fd50419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:24.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd509a03640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504103920 0x7fd50419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd509a03640 1 -- 172.21.15.16:0/2343314250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5041a01d0 con 0x7fd504109bb0 2024-08-28T19:21:24.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd509a03640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd504109bb0 0x7fd50419faa0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc00d950 tx=0x7fd4fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:24.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd4fa7fc640 1 -- 172.21.15.16:0/2343314250 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc017030 con 0x7fd504109bb0 2024-08-28T19:21:24.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd50b466640 1 -- 172.21.15.16:0/2343314250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5041a04c0 con 0x7fd504109bb0 2024-08-28T19:21:24.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd4fa7fc640 1 -- 172.21.15.16:0/2343314250 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4fc004020 con 0x7fd504109bb0 2024-08-28T19:21:24.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.535+0000 7fd4fa7fc640 1 -- 172.21.15.16:0/2343314250 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc003b50 con 0x7fd504109bb0 2024-08-28T19:21:24.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.536+0000 7fd50b466640 1 -- 172.21.15.16:0/2343314250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5041a6c20 con 0x7fd504109bb0 2024-08-28T19:21:24.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.538+0000 7fd5119db740 1 Processor -- start 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.538+0000 7fd4fa7fc640 1 -- 172.21.15.16:0/2343314250 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7fd4fc01a070 con 0x7fd504109bb0 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd5119db740 1 -- start start 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd5119db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdebf60 0x55efe01917b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd5119db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdc30a0 0x55efe0191cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd5119db740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe0192230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd5119db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55efe01083f0 con 0x55efdfdebf60 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd5119db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55efe00f7980 con 0x55efdfdc30a0 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd5119db740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55efe00fb930 con 0x55efdfdcb280 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd508a01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdebf60 0x55efe01917b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd508a01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdebf60 0x55efe01917b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42488/0 (socket says 172.21.15.16:42488) 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd508a01640 1 -- 172.21.15.16:0/2268711020 learned_addr learned my addr 172.21.15.16:0/2268711020 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd509202640 1 --2- 172.21.15.16:0/2268711020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdc30a0 0x55efe0191cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd508a01640 1 -- 172.21.15.16:0/2268711020 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 msgr2=0x55efe0192230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:24.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd508a01640 1 --2- 172.21.15.16:0/2268711020 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe0192230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd508a01640 1 -- 172.21.15.16:0/2268711020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdc30a0 msgr2=0x55efe0191cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:24.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd508a01640 1 --2- 172.21.15.16:0/2268711020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdc30a0 0x55efe0191cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd508a01640 1 -- 172.21.15.16:0/2268711020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efdfe57540 con 0x55efdfdebf60 2024-08-28T19:21:24.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.539+0000 7fd4fa7fc640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4f00b4c10 0x7fd4f00b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.540+0000 7fd508a01640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4f00b4c10 0x7fd4f00b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.540+0000 7fd4fa7fc640 1 -- 172.21.15.16:0/2343314250 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fd4fc0e4ee0 con 0x7fd504109bb0 2024-08-28T19:21:24.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.540+0000 7fd508a01640 1 --2- 172.21.15.16:0/2268711020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdebf60 0x55efe01917b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4009900 tx=0x7fd4f4032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dfbffa29ea9b51f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:24.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.540+0000 7fd4ef7fe640 1 -- 172.21.15.16:0/2268711020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4035040 con 0x55efdfdebf60 2024-08-28T19:21:24.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.540+0000 7fd4ef7fe640 1 -- 172.21.15.16:0/2268711020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4f4005040 con 0x55efdfdebf60 2024-08-28T19:21:24.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.540+0000 7fd4ef7fe640 1 -- 172.21.15.16:0/2268711020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4004950 con 0x55efdfdebf60 2024-08-28T19:21:24.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.541+0000 7fd5119db740 1 -- 172.21.15.16:0/2268711020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdebf60 msgr2=0x55efe01917b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:24.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.541+0000 7fd5119db740 1 --2- 172.21.15.16:0/2268711020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdebf60 0x55efe01917b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4009900 tx=0x7fd4f4032c60 comp rx=0 tx=0).stop 2024-08-28T19:21:24.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.541+0000 7fd508a01640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4f00b4c10 0x7fd4f00b7120 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd4f40096c0 tx=0x7fd4f404c000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:24.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.541+0000 7fd5119db740 1 -- 172.21.15.16:0/2268711020 shutdown_connections 2024-08-28T19:21:24.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.541+0000 7fd5119db740 1 --2- 172.21.15.16:0/2268711020 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe0192230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.541+0000 7fd5119db740 1 --2- 172.21.15.16:0/2268711020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdc30a0 0x55efe0191cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.541+0000 7fd5119db740 1 --2- 172.21.15.16:0/2268711020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdebf60 0x55efe01917b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.541+0000 7fd5119db740 1 -- 172.21.15.16:0/2268711020 >> 172.21.15.16:0/2268711020 conn(0x55efdfde0350 msgr2=0x55efe018f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.541+0000 7fd5119db740 1 -- 172.21.15.16:0/2268711020 shutdown_connections 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.541+0000 7fd5119db740 1 -- 172.21.15.16:0/2268711020 wait complete. 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.541+0000 7fd5119db740 1 -- start start 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd5119db740 1 Processor -- start 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd5119db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd5119db740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe019d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd5119db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe01a9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd5119db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55efdff5a5f0 con 0x55efdfdc30a0 2024-08-28T19:21:24.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd5119db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55efe007e600 con 0x55efdfdebf60 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd5119db740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55efe019ff30 con 0x55efdfdcb280 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd509202640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd5119db740 10 client.0 _ll_register_callbacks cb 0x55efe00a4af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd509202640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42494/0 (socket says 172.21.15.16:42494) 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd509202640 1 -- 172.21.15.16:0/3761328321 learned_addr learned my addr 172.21.15.16:0/3761328321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd508a01640 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.542+0000 7fd509a03640 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe01a9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd509202640 1 -- 172.21.15.16:0/3761328321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 msgr2=0x55efe019d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd509202640 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe019d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd509202640 1 -- 172.21.15.16:0/3761328321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 msgr2=0x55efe01a9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd509202640 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe01a9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd509202640 1 -- 172.21.15.16:0/3761328321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efe00b77b0 con 0x55efdfdc30a0 2024-08-28T19:21:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd509a03640 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe01a9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd509202640 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019ce30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd500011d10 tx=0x7fd50000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fba0a9d1f37754a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd4ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd4ecff9640 1 -- 172.21.15.16:0/3761328321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50001f060 con 0x55efdfdc30a0 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd4ecff9640 1 -- 172.21.15.16:0/3761328321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fd500017050 con 0x55efdfdc30a0 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd5119db740 1 -- 172.21.15.16:0/3761328321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4f4005040 con 0x55efdfdc30a0 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd4ecff9640 1 -- 172.21.15.16:0/3761328321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd500026050 con 0x55efdfdc30a0 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd5119db740 10 client.15765 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd5119db740 1 -- 172.21.15.16:0/3761328321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55efe01b0080 con 0x55efdfdc30a0 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.543+0000 7fd4ecff9640 1 -- 172.21.15.16:0/3761328321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4f4005040 con 0x55efdfdc30a0 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.544+0000 7fd4ecff9640 1 -- 172.21.15.16:0/3761328321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd500019030 con 0x55efdfdc30a0 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.544+0000 7fd5119db740 10 client.15765 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.544+0000 7fd5119db740 10 client.15765 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.544+0000 7fd5119db740 20 client.15765 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:24.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.544+0000 7fd5119db740 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55efdfdaa8e0 0x55efe01b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.544+0000 7fd5119db740 4 client.15765 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:24.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.544+0000 7fd5119db740 1 -- 172.21.15.16:0/3761328321 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efe010a340 con 0x55efdfdaa8e0 2024-08-28T19:21:24.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.544+0000 7fd508a01640 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55efdfdaa8e0 0x55efe01b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.544+0000 7fd4ecff9640 10 client.15765 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:21:24.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.544+0000 7fd508a01640 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55efdfdaa8e0 0x55efe01b3b50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=630463455e12aec1 server_cookie=7a2ed140469e43b8 in_seq=0 out_seq=0 2024-08-28T19:21:24.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.545+0000 7fd4ecff9640 1 -- 172.21.15.16:0/3761328321 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55efe010a340 con 0x55efdfdaa8e0 2024-08-28T19:21:24.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.545+0000 7fd4ecff9640 10 client.15765 handle_command_reply: tid=0 2024-08-28T19:21:24.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.545+0000 7fd5119db740 1 client.15765 shutdown 2024-08-28T19:21:24.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.545+0000 7fd5119db740 1 -- 172.21.15.16:0/3761328321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 msgr2=0x55efe019ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:24.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.545+0000 7fd5119db740 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019ce30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd500011d10 tx=0x7fd50000f790 comp rx=0 tx=0).stop 2024-08-28T19:21:24.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.545+0000 7fd5119db740 1 -- 172.21.15.16:0/3761328321 shutdown_connections 2024-08-28T19:21:24.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.545+0000 7fd5119db740 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55efdfdaa8e0 0x55efe01b3b50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.545+0000 7fd5119db740 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe01a9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.545+0000 7fd5119db740 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.545+0000 7fd5119db740 1 --2- 172.21.15.16:0/3761328321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019ce30 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.545+0000 7fd5119db740 1 -- 172.21.15.16:0/3761328321 >> 172.21.15.16:0/3761328321 conn(0x55efdfde0350 msgr2=0x55efe0194990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:24.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.546+0000 7fd4ecff9640 0 client.15765 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:21:24.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.546+0000 7fd5119db740 1 -- 172.21.15.16:0/3761328321 shutdown_connections 2024-08-28T19:21:24.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.546+0000 7fd5119db740 1 -- 172.21.15.16:0/3761328321 wait complete. 2024-08-28T19:21:24.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.546+0000 7fd5119db740 20 client.15765 trim_cache size 0 max 16384 2024-08-28T19:21:24.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.564+0000 7fd5119db740 1 Processor -- start 2024-08-28T19:21:24.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.564+0000 7fd5119db740 1 -- start start 2024-08-28T19:21:24.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.564+0000 7fd5119db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.564+0000 7fd5119db740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe01a91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.564+0000 7fd5119db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe018ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.564+0000 7fd5119db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55efe01b1e90 con 0x55efdfdc30a0 2024-08-28T19:21:24.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.564+0000 7fd5119db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55efe01b2560 con 0x55efdfdebf60 2024-08-28T19:21:24.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.564+0000 7fd5119db740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55efe01b2c30 con 0x55efdfdcb280 2024-08-28T19:21:24.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.564+0000 7fd509202640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.564+0000 7fd509202640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42510/0 (socket says 172.21.15.16:42510) 2024-08-28T19:21:24.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd509202640 1 -- 172.21.15.16:0/3824288724 learned_addr learned my addr 172.21.15.16:0/3824288724 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:24.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd509202640 1 -- 172.21.15.16:0/3824288724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 msgr2=0x55efe01a91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:24.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd508a01640 1 --2- 172.21.15.16:0/3824288724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe01a91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd509a03640 1 --2- 172.21.15.16:0/3824288724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe018ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd509202640 1 --2- 172.21.15.16:0/3824288724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe01a91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd509202640 1 -- 172.21.15.16:0/3824288724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 msgr2=0x55efe018ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:24.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd509202640 1 --2- 172.21.15.16:0/3824288724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe018ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd509202640 1 -- 172.21.15.16:0/3824288724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efe0197ae0 con 0x55efdfdc30a0 2024-08-28T19:21:24.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd508a01640 1 --2- 172.21.15.16:0/3824288724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe01a91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:24.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd509202640 1 --2- 172.21.15.16:0/3824288724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019d2f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd500011a30 tx=0x7fd50000fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d21c16559b8f9d39 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:24.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd4eeffd640 1 -- 172.21.15.16:0/3824288724 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5000049c0 con 0x55efdfdc30a0 2024-08-28T19:21:24.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd4eeffd640 1 -- 172.21.15.16:0/3824288724 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd500019c60 con 0x55efdfdc30a0 2024-08-28T19:21:24.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.565+0000 7fd4eeffd640 1 -- 172.21.15.16:0/3824288724 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50002a530 con 0x55efdfdc30a0 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.566+0000 7fd5119db740 1 -- 172.21.15.16:0/3824288724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 msgr2=0x55efe019d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.566+0000 7fd5119db740 1 --2- 172.21.15.16:0/3824288724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019d2f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd500011a30 tx=0x7fd50000fc90 comp rx=0 tx=0).stop 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.566+0000 7fd5119db740 1 -- 172.21.15.16:0/3824288724 shutdown_connections 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.566+0000 7fd5119db740 1 --2- 172.21.15.16:0/3824288724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe018ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.566+0000 7fd5119db740 1 --2- 172.21.15.16:0/3824288724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe01a91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.566+0000 7fd5119db740 1 --2- 172.21.15.16:0/3824288724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe019d2f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.566+0000 7fd5119db740 1 -- 172.21.15.16:0/3824288724 >> 172.21.15.16:0/3824288724 conn(0x55efdfde0350 msgr2=0x55efe019cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.566+0000 7fd5119db740 1 -- 172.21.15.16:0/3824288724 shutdown_connections 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.566+0000 7fd5119db740 1 -- 172.21.15.16:0/3824288724 wait complete. 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd5119db740 1 -- start start 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd5119db740 1 Processor -- start 2024-08-28T19:21:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd5119db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe01bc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd5119db740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe01bc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd5119db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe01b5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd5119db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55efe01b39d0 con 0x55efdfdc30a0 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd5119db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55efe01b3300 con 0x55efdfdebf60 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd5119db740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55efe01b40a0 con 0x55efdfdcb280 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd5119db740 10 client.0 _ll_register_callbacks cb 0x7fd500011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd509202640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe01bc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd509202640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe01bc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42524/0 (socket says 172.21.15.16:42524) 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd509202640 1 -- 172.21.15.16:0/2524614514 learned_addr learned my addr 172.21.15.16:0/2524614514 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd509a03640 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe01b5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.567+0000 7fd508a01640 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe01bc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd509202640 1 -- 172.21.15.16:0/2524614514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 msgr2=0x55efe01bc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd509202640 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe01bc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd509202640 1 -- 172.21.15.16:0/2524614514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 msgr2=0x55efe01b5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd509202640 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe01b5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd509202640 1 -- 172.21.15.16:0/2524614514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efe01b5f00 con 0x55efdfdc30a0 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd509202640 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe01bc1b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd50001a040 tx=0x7fd500012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1aed083d144bd3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd4ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd4ecff9640 1 -- 172.21.15.16:0/2524614514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500039060 con 0x55efdfdc30a0 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd4ecff9640 1 -- 172.21.15.16:0/2524614514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fd500019a00 con 0x55efdfdc30a0 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd5119db740 1 -- 172.21.15.16:0/2524614514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd500019c60 con 0x55efdfdc30a0 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd4ecff9640 1 -- 172.21.15.16:0/2524614514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd500040050 con 0x55efdfdc30a0 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd5119db740 10 client.15771 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd5119db740 1 -- 172.21.15.16:0/2524614514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55efe01bb8c0 con 0x55efdfdc30a0 2024-08-28T19:21:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.568+0000 7fd4ecff9640 1 -- 172.21.15.16:0/2524614514 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd500019c60 con 0x55efdfdc30a0 2024-08-28T19:21:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.569+0000 7fd4ecff9640 1 -- 172.21.15.16:0/2524614514 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd500019040 con 0x55efdfdc30a0 2024-08-28T19:21:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.569+0000 7fd5119db740 10 client.15771 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.569+0000 7fd5119db740 10 client.15771 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-28T19:21:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.569+0000 7fd5119db740 20 client.15771 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.569+0000 7fd5119db740 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55efdfd7fc90 0x55efe0221f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.569+0000 7fd5119db740 4 client.15771 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:21:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.569+0000 7fd5119db740 1 -- 172.21.15.16:0/2524614514 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "status"}) -- 0x55efe0105de0 con 0x55efdfd7fc90 2024-08-28T19:21:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.569+0000 7fd508a01640 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55efdfd7fc90 0x55efe0221f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd508a01640 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55efdfd7fc90 0x55efe0221f90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ea2d12f2c1c274f server_cookie=597fd6a849cda89c in_seq=0 out_seq=0 2024-08-28T19:21:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd4ecff9640 10 client.15771 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:21:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd4ecff9640 1 -- 172.21.15.16:0/2524614514 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55efe0105de0 con 0x55efdfd7fc90 2024-08-28T19:21:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd4ecff9640 10 client.15771 handle_command_reply: tid=0 2024-08-28T19:21:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 1 client.15771 shutdown 2024-08-28T19:21:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 1 -- 172.21.15.16:0/2524614514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 msgr2=0x55efe01bc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe01bc1b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd50001a040 tx=0x7fd500012f90 comp rx=0 tx=0).stop 2024-08-28T19:21:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 1 -- 172.21.15.16:0/2524614514 shutdown_connections 2024-08-28T19:21:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55efdfd7fc90 0x55efe0221f90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efdfdebf60 0x55efe01b5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55efdfdcb280 0x55efe01bc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 1 --2- 172.21.15.16:0/2524614514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdfdc30a0 0x55efe01bc1b0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 1 -- 172.21.15.16:0/2524614514 >> 172.21.15.16:0/2524614514 conn(0x55efdfde0350 msgr2=0x55efe01927e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd4ecff9640 0 client.15771 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:21:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 1 -- 172.21.15.16:0/2524614514 shutdown_connections 2024-08-28T19:21:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 1 -- 172.21.15.16:0/2524614514 wait complete. 2024-08-28T19:21:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.570+0000 7fd5119db740 20 client.15771 trim_cache size 0 max 16384 2024-08-28T19:21:24.579 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:21:24.579 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:21:24.579 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-08-28T19:21:24.580 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24649, 2024-08-28T19:21:24.580 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:21:24.580 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:21:24.580 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:21:24.580 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 135.95797342399999, 2024-08-28T19:21:24.580 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 29, 2024-08-28T19:21:24.580 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:21:24.580 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-08-28T19:21:24.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 190.05290719600001 2024-08-28T19:21:24.580 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:21:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 -- 172.21.15.16:0/2343314250 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4f00b4c10 msgr2=0x7fd4f00b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4f00b4c10 0x7fd4f00b7120 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd4f40096c0 tx=0x7fd4f404c000 comp rx=0 tx=0).stop 2024-08-28T19:21:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 -- 172.21.15.16:0/2343314250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd504109bb0 msgr2=0x7fd50419faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd504109bb0 0x7fd50419faa0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc00d950 tx=0x7fd4fc00de10 comp rx=0 tx=0).stop 2024-08-28T19:21:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 -- 172.21.15.16:0/2343314250 shutdown_connections 2024-08-28T19:21:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd504109bb0 0x7fd50419faa0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd504109260 0x7fd50419a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4f00b4c10 0x7fd4f00b7120 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 --2- 172.21.15.16:0/2343314250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504103920 0x7fd50419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 -- 172.21.15.16:0/2343314250 >> 172.21.15.16:0/2343314250 conn(0x7fd5040ff3b0 msgr2=0x7fd5041017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 -- 172.21.15.16:0/2343314250 shutdown_connections 2024-08-28T19:21:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:24.571+0000 7fd50b466640 1 -- 172.21.15.16:0/2343314250 wait complete. 2024-08-28T19:21:24.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:24 smithi115 ceph-mon[22184]: pgmap v350: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 8 op/s 2024-08-28T19:21:25.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:24 smithi016 ceph-mon[18556]: pgmap v350: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 8 op/s 2024-08-28T19:21:25.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:24 smithi028 ceph-mon[21769]: pgmap v350: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 8 op/s 2024-08-28T19:21:25.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.b status 2024-08-28T19:21:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.195+0000 7ff03475e640 1 -- 172.21.15.16:0/571353985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02c103900 msgr2=0x7ff02c103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.195+0000 7ff03475e640 1 --2- 172.21.15.16:0/571353985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02c103900 0x7ff02c103d10 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff01c009ca0 tx=0x7ff01c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.195+0000 7ff03475e640 1 -- 172.21.15.16:0/571353985 shutdown_connections 2024-08-28T19:21:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.195+0000 7ff03475e640 1 --2- 172.21.15.16:0/571353985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff02c109b90 0x7ff02c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.195+0000 7ff03475e640 1 --2- 172.21.15.16:0/571353985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff02c109240 0x7ff02c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.195+0000 7ff03475e640 1 --2- 172.21.15.16:0/571353985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02c103900 0x7ff02c103d10 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.195+0000 7ff03475e640 1 -- 172.21.15.16:0/571353985 >> 172.21.15.16:0/571353985 conn(0x7ff02c07b5f0 msgr2=0x7ff02c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.195+0000 7ff03475e640 1 -- 172.21.15.16:0/571353985 shutdown_connections 2024-08-28T19:21:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.195+0000 7ff03475e640 1 -- 172.21.15.16:0/571353985 wait complete. 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff03475e640 1 Processor -- start 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff03475e640 1 -- start start 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff03475e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02c103900 0x7ff02c0754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff03475e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff02c109240 0x7ff02c0759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff03475e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff02c109b90 0x7ff02c075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff03475e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff02c113850 con 0x7ff02c103900 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff03475e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff02c1136d0 con 0x7ff02c109b90 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff03475e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff02c1139d0 con 0x7ff02c109240 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff0324fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02c103900 0x7ff02c0754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff0324fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02c103900 0x7ff02c0754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42540/0 (socket says 172.21.15.16:42540) 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff0324fa640 1 -- 172.21.15.16:0/312060675 learned_addr learned my addr 172.21.15.16:0/312060675 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff031cf9640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff02c109240 0x7ff02c0759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.196+0000 7ff032cfb640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff02c109b90 0x7ff02c075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.197+0000 7ff0324fa640 1 -- 172.21.15.16:0/312060675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff02c109240 msgr2=0x7ff02c0759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.197+0000 7ff0324fa640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff02c109240 0x7ff02c0759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.197+0000 7ff0324fa640 1 -- 172.21.15.16:0/312060675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff02c109b90 msgr2=0x7ff02c075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.197+0000 7ff0324fa640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff02c109b90 0x7ff02c075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.197+0000 7ff0324fa640 1 -- 172.21.15.16:0/312060675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff02c1a84e0 con 0x7ff02c103900 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.197+0000 7ff0324fa640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02c103900 0x7ff02c0754b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff01c006010 tx=0x7ff01c0319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.197+0000 7ff02b7fe640 1 -- 172.21.15.16:0/312060675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c003bf0 con 0x7ff02c103900 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.197+0000 7ff03475e640 1 -- 172.21.15.16:0/312060675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff02c1a8770 con 0x7ff02c103900 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.197+0000 7ff02b7fe640 1 -- 172.21.15.16:0/312060675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff01c003d90 con 0x7ff02c103900 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.197+0000 7ff02b7fe640 1 -- 172.21.15.16:0/312060675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c02fc20 con 0x7ff02c103900 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.197+0000 7ff03475e640 1 -- 172.21.15.16:0/312060675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff02c1a8d00 con 0x7ff02c103900 2024-08-28T19:21:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff03acd3740 1 Processor -- start 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff03acd3740 1 -- start start 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff02b7fe640 1 -- 172.21.15.16:0/312060675 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7ff01c0046a0 con 0x7ff02c103900 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0afb6f60 0x556d0b35c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b35ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b35d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff03acd3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556d0b2d33f0 con 0x556d0af96280 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff03acd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d0b2c2980 con 0x556d0afb6f60 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff03acd3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556d0b2c6930 con 0x556d0af8e0a0 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff031cf9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b35d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff031cf9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b35d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42542/0 (socket says 172.21.15.16:42542) 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff031cf9640 1 -- 172.21.15.16:0/2512511421 learned_addr learned my addr 172.21.15.16:0/2512511421 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff032cfb640 1 --2- 172.21.15.16:0/2512511421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0afb6f60 0x556d0b35c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.200+0000 7ff0324fa640 1 --2- 172.21.15.16:0/2512511421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b35ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.201+0000 7ff031cf9640 1 -- 172.21.15.16:0/2512511421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 msgr2=0x556d0b35ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.201+0000 7ff031cf9640 1 --2- 172.21.15.16:0/2512511421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b35ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.201+0000 7ff031cf9640 1 -- 172.21.15.16:0/2512511421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0afb6f60 msgr2=0x556d0b35c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.201+0000 7ff031cf9640 1 --2- 172.21.15.16:0/2512511421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0afb6f60 0x556d0b35c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.201+0000 7ff031cf9640 1 -- 172.21.15.16:0/2512511421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d0b022540 con 0x556d0af96280 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.201+0000 7ff031cf9640 1 --2- 172.21.15.16:0/2512511421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b35d230 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff02000ab70 tx=0x7ff02000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b936233c3e33fb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.201+0000 7ff028ff9640 1 -- 172.21.15.16:0/2512511421 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020004220 con 0x556d0af96280 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.201+0000 7ff028ff9640 1 -- 172.21.15.16:0/2512511421 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0200043c0 con 0x556d0af96280 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.201+0000 7ff028ff9640 1 -- 172.21.15.16:0/2512511421 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020018770 con 0x556d0af96280 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.201+0000 7ff02b7fe640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0140b4c10 0x7ff0140b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.201+0000 7ff032cfb640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0140b4c10 0x7ff0140b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.202+0000 7ff02b7fe640 1 -- 172.21.15.16:0/312060675 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff01c106b30 con 0x7ff02c103900 2024-08-28T19:21:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.202+0000 7ff03acd3740 1 -- 172.21.15.16:0/2512511421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 msgr2=0x556d0b35d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.202+0000 7ff032cfb640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0140b4c10 0x7ff0140b7120 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff02400bca0 tx=0x7ff02400b450 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.202+0000 7ff03acd3740 1 --2- 172.21.15.16:0/2512511421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b35d230 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff02000ab70 tx=0x7ff02000f790 comp rx=0 tx=0).stop 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.202+0000 7ff03acd3740 1 -- 172.21.15.16:0/2512511421 shutdown_connections 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.202+0000 7ff03acd3740 1 --2- 172.21.15.16:0/2512511421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b35d230 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.202+0000 7ff03acd3740 1 --2- 172.21.15.16:0/2512511421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b35ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.202+0000 7ff03acd3740 1 --2- 172.21.15.16:0/2512511421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0afb6f60 0x556d0b35c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.202+0000 7ff03acd3740 1 -- 172.21.15.16:0/2512511421 >> 172.21.15.16:0/2512511421 conn(0x556d0afab350 msgr2=0x556d0b35a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.202+0000 7ff03acd3740 1 -- 172.21.15.16:0/2512511421 shutdown_connections 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.202+0000 7ff03acd3740 1 -- 172.21.15.16:0/2512511421 wait complete. 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff03acd3740 1 -- start start 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff03acd3740 1 Processor -- start 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b367e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0af96280 0x556d0b368350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0afb6f60 0x556d0b3742b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff03acd3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556d0b1255f0 con 0x556d0afb6f60 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff03acd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d0b249600 con 0x556d0af96280 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff03acd3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556d0b36af90 con 0x556d0af8e0a0 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff03acd3740 10 client.0 _ll_register_callbacks cb 0x556d0b26faf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff032cfb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0afb6f60 0x556d0b3742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff032cfb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0afb6f60 0x556d0b3742b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42552/0 (socket says 172.21.15.16:42552) 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff032cfb640 1 -- 172.21.15.16:0/3017260218 learned_addr learned my addr 172.21.15.16:0/3017260218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff0324fa640 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b367e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.203+0000 7ff031cf9640 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0af96280 0x556d0b368350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff032cfb640 1 -- 172.21.15.16:0/3017260218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 msgr2=0x556d0b367e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff032cfb640 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b367e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff032cfb640 1 -- 172.21.15.16:0/3017260218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0af96280 msgr2=0x556d0b368350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff032cfb640 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0af96280 0x556d0b368350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff032cfb640 1 -- 172.21.15.16:0/3017260218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d0b2827b0 con 0x556d0afb6f60 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff032cfb640 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0afb6f60 0x556d0b3742b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff024013fd0 tx=0x7ff0240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be41033a4385dd4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff011ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff011ffb640 1 -- 172.21.15.16:0/3017260218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024029680 con 0x556d0afb6f60 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff011ffb640 1 -- 172.21.15.16:0/3017260218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff024029820 con 0x556d0afb6f60 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff03acd3740 1 -- 172.21.15.16:0/3017260218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0200043c0 con 0x556d0afb6f60 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff011ffb640 1 -- 172.21.15.16:0/3017260218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff024029a20 con 0x556d0afb6f60 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff03acd3740 10 client.15783 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff03acd3740 1 -- 172.21.15.16:0/3017260218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556d0b376cd0 con 0x556d0afb6f60 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.204+0000 7ff011ffb640 1 -- 172.21.15.16:0/3017260218 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0200043c0 con 0x556d0afb6f60 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.205+0000 7ff011ffb640 1 -- 172.21.15.16:0/3017260218 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff02401b390 con 0x556d0afb6f60 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.205+0000 7ff03acd3740 10 client.15783 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.205+0000 7ff03acd3740 10 client.15783 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.205+0000 7ff03acd3740 20 client.15783 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.205+0000 7ff03acd3740 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d0af758e0 0x556d0b37a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.205+0000 7ff03acd3740 4 client.15783 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.205+0000 7ff03acd3740 1 -- 172.21.15.16:0/3017260218 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d0b2d5340 con 0x556d0af758e0 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.205+0000 7ff0324fa640 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d0af758e0 0x556d0b37a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.205+0000 7ff011ffb640 10 client.15783 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.205+0000 7ff0324fa640 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d0af758e0 0x556d0b37a7a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa1a2103aa75a4f server_cookie=b6a728e9d6445a30 in_seq=0 out_seq=0 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff011ffb640 1 -- 172.21.15.16:0/3017260218 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556d0b2d5340 con 0x556d0af758e0 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff011ffb640 10 client.15783 handle_command_reply: tid=0 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff03acd3740 1 client.15783 shutdown 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff03acd3740 1 -- 172.21.15.16:0/3017260218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0afb6f60 msgr2=0x556d0b3742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff03acd3740 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0afb6f60 0x556d0b3742b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff024013fd0 tx=0x7ff0240046c0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff03acd3740 1 -- 172.21.15.16:0/3017260218 shutdown_connections 2024-08-28T19:21:26.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff03acd3740 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0afb6f60 0x556d0b3742b0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff03acd3740 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0af96280 0x556d0b368350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff03acd3740 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d0af758e0 0x556d0b37a7a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff03acd3740 1 --2- 172.21.15.16:0/3017260218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b367e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff03acd3740 1 -- 172.21.15.16:0/3017260218 >> 172.21.15.16:0/3017260218 conn(0x556d0afab350 msgr2=0x556d0b35f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:26.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.206+0000 7ff011ffb640 0 client.15783 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:26.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.207+0000 7ff03acd3740 1 -- 172.21.15.16:0/3017260218 shutdown_connections 2024-08-28T19:21:26.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.207+0000 7ff03acd3740 1 -- 172.21.15.16:0/3017260218 wait complete. 2024-08-28T19:21:26.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.207+0000 7ff03acd3740 20 client.15783 trim_cache size 0 max 16384 2024-08-28T19:21:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.225+0000 7ff03acd3740 1 Processor -- start 2024-08-28T19:21:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.225+0000 7ff03acd3740 1 -- start start 2024-08-28T19:21:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.225+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0af8e0a0 0x556d0b374220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.225+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b359cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.225+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0afb6f60 0x556d0b35a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.225+0000 7ff03acd3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556d0b378ae0 con 0x556d0af96280 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.225+0000 7ff03acd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d0b3791b0 con 0x556d0af8e0a0 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.225+0000 7ff03acd3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556d0b379880 con 0x556d0afb6f60 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.225+0000 7ff031cf9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b359cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.225+0000 7ff031cf9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b359cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42568/0 (socket says 172.21.15.16:42568) 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.225+0000 7ff031cf9640 1 -- 172.21.15.16:0/3934114852 learned_addr learned my addr 172.21.15.16:0/3934114852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff0324fa640 1 --2- 172.21.15.16:0/3934114852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0af8e0a0 0x556d0b374220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff032cfb640 1 --2- 172.21.15.16:0/3934114852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0afb6f60 0x556d0b35a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff031cf9640 1 -- 172.21.15.16:0/3934114852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0afb6f60 msgr2=0x556d0b35a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff031cf9640 1 --2- 172.21.15.16:0/3934114852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0afb6f60 0x556d0b35a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff031cf9640 1 -- 172.21.15.16:0/3934114852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0af8e0a0 msgr2=0x556d0b374220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff031cf9640 1 --2- 172.21.15.16:0/3934114852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0af8e0a0 0x556d0b374220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff031cf9640 1 -- 172.21.15.16:0/3934114852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d0b368eb0 con 0x556d0af96280 2024-08-28T19:21:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff0324fa640 1 --2- 172.21.15.16:0/3934114852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0af8e0a0 0x556d0b374220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff031cf9640 1 --2- 172.21.15.16:0/3934114852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b359cf0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff02000ab40 tx=0x7ff020005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf38946dda9f03 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff013fff640 1 -- 172.21.15.16:0/3934114852 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02000ae70 con 0x556d0af96280 2024-08-28T19:21:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff013fff640 1 -- 172.21.15.16:0/3934114852 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff020004320 con 0x556d0af96280 2024-08-28T19:21:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.226+0000 7ff013fff640 1 -- 172.21.15.16:0/3934114852 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020003850 con 0x556d0af96280 2024-08-28T19:21:26.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.227+0000 7ff03acd3740 1 -- 172.21.15.16:0/3934114852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 msgr2=0x556d0b359cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:26.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.227+0000 7ff03acd3740 1 --2- 172.21.15.16:0/3934114852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b359cf0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff02000ab40 tx=0x7ff020005da0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.227+0000 7ff03acd3740 1 -- 172.21.15.16:0/3934114852 shutdown_connections 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.227+0000 7ff03acd3740 1 --2- 172.21.15.16:0/3934114852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0afb6f60 0x556d0b35a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.227+0000 7ff03acd3740 1 --2- 172.21.15.16:0/3934114852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b359cf0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.227+0000 7ff03acd3740 1 --2- 172.21.15.16:0/3934114852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0af8e0a0 0x556d0b374220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.227+0000 7ff03acd3740 1 -- 172.21.15.16:0/3934114852 >> 172.21.15.16:0/3934114852 conn(0x556d0afab350 msgr2=0x556d0b366690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.227+0000 7ff03acd3740 1 -- 172.21.15.16:0/3934114852 shutdown_connections 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.227+0000 7ff03acd3740 1 -- 172.21.15.16:0/3934114852 wait complete. 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.227+0000 7ff03acd3740 1 -- start start 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.227+0000 7ff03acd3740 1 Processor -- start 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b380a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b381020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff03acd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0afb6f60 0x556d0b37c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff03acd3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556d0b37a620 con 0x556d0af96280 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff03acd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d0b379f50 con 0x556d0afb6f60 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff03acd3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556d0b37acf0 con 0x556d0af8e0a0 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff03acd3740 10 client.0 _ll_register_callbacks cb 0x7ff02400be40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff031cf9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b381020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff031cf9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b381020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42578/0 (socket says 172.21.15.16:42578) 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff031cf9640 1 -- 172.21.15.16:0/543061855 learned_addr learned my addr 172.21.15.16:0/543061855 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff032cfb640 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0afb6f60 0x556d0b37c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff0324fa640 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b380a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff031cf9640 1 -- 172.21.15.16:0/543061855 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 msgr2=0x556d0b380a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff031cf9640 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b380a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff031cf9640 1 -- 172.21.15.16:0/543061855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0afb6f60 msgr2=0x556d0b37c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff031cf9640 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0afb6f60 0x556d0b37c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff031cf9640 1 -- 172.21.15.16:0/543061855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d0b37c7d0 con 0x556d0af96280 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff031cf9640 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b381020 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff0200058c0 tx=0x7ff020024b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2435f1b4143690 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.228+0000 7ff011ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff011ffb640 1 -- 172.21.15.16:0/543061855 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020018640 con 0x556d0af96280 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff011ffb640 1 -- 172.21.15.16:0/543061855 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff0200187e0 con 0x556d0af96280 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff03acd3740 1 -- 172.21.15.16:0/543061855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff020004320 con 0x556d0af96280 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff011ffb640 1 -- 172.21.15.16:0/543061855 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff02003f050 con 0x556d0af96280 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff03acd3740 10 client.15789 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff03acd3740 1 -- 172.21.15.16:0/543061855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556d0b382140 con 0x556d0af96280 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff011ffb640 1 -- 172.21.15.16:0/543061855 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff020004320 con 0x556d0af96280 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff011ffb640 1 -- 172.21.15.16:0/543061855 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff020010690 con 0x556d0af96280 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff03acd3740 10 client.15789 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff03acd3740 10 client.15789 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff03acd3740 20 client.15789 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff03acd3740 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d0af4ac90 0x556d0b3e86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff03acd3740 4 client.15789 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.229+0000 7ff03acd3740 1 -- 172.21.15.16:0/543061855 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "status"}) -- 0x556d0b2d0de0 con 0x556d0af4ac90 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff0324fa640 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d0af4ac90 0x556d0b3e86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff0324fa640 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d0af4ac90 0x556d0b3e86f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae809db541e3f581 server_cookie=2f5712133d6ba659 in_seq=0 out_seq=0 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff011ffb640 10 client.15789 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff011ffb640 1 -- 172.21.15.16:0/543061855 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556d0b2d0de0 con 0x556d0af4ac90 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff011ffb640 10 client.15789 handle_command_reply: tid=0 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff03acd3740 1 client.15789 shutdown 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff03acd3740 1 -- 172.21.15.16:0/543061855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 msgr2=0x556d0b381020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff03acd3740 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b381020 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff0200058c0 tx=0x7ff020024b10 comp rx=0 tx=0).stop 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff03acd3740 1 -- 172.21.15.16:0/543061855 shutdown_connections 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff03acd3740 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d0afb6f60 0x556d0b37c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff03acd3740 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d0af96280 0x556d0b381020 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff03acd3740 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d0af4ac90 0x556d0b3e86f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff011ffb640 0 client.15789 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.230+0000 7ff03acd3740 1 --2- 172.21.15.16:0/543061855 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d0af8e0a0 0x556d0b380a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03acd3740 1 -- 172.21.15.16:0/543061855 >> 172.21.15.16:0/543061855 conn(0x556d0afab350 msgr2=0x556d0b3692c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03acd3740 1 -- 172.21.15.16:0/543061855 shutdown_connections 2024-08-28T19:21:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03acd3740 1 -- 172.21.15.16:0/543061855 wait complete. 2024-08-28T19:21:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03acd3740 20 client.15789 trim_cache size 0 max 16384 2024-08-28T19:21:26.237 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:21:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "eabc438a-6570-11ef-bcd4-c7b262605968", 2024-08-28T19:21:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-08-28T19:21:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24665, 2024-08-28T19:21:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-08-28T19:21:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-08-28T19:21:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-08-28T19:21:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 172.75644882700001, 2024-08-28T19:21:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 29, 2024-08-28T19:21:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-08-28T19:21:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-08-28T19:21:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 182.971485925 2024-08-28T19:21:26.238 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:21:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 -- 172.21.15.16:0/312060675 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0140b4c10 msgr2=0x7ff0140b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0140b4c10 0x7ff0140b7120 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff02400bca0 tx=0x7ff02400b450 comp rx=0 tx=0).stop 2024-08-28T19:21:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 -- 172.21.15.16:0/312060675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02c103900 msgr2=0x7ff02c0754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02c103900 0x7ff02c0754b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff01c006010 tx=0x7ff01c0319f0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 -- 172.21.15.16:0/312060675 shutdown_connections 2024-08-28T19:21:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0140b4c10 0x7ff0140b7120 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff02c109b90 0x7ff02c075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff02c109240 0x7ff02c0759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 --2- 172.21.15.16:0/312060675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02c103900 0x7ff02c0754b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 -- 172.21.15.16:0/312060675 >> 172.21.15.16:0/312060675 conn(0x7ff02c07b5f0 msgr2=0x7ff02c1072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 -- 172.21.15.16:0/312060675 shutdown_connections 2024-08-28T19:21:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:26.231+0000 7ff03475e640 1 -- 172.21.15.16:0/312060675 wait complete. 2024-08-28T19:21:26.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:26 smithi115 ceph-mon[22184]: pgmap v351: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 853 B/s wr, 12 op/s 2024-08-28T19:21:26.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:26 smithi016 ceph-mon[18556]: pgmap v351: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 853 B/s wr, 12 op/s 2024-08-28T19:21:26.964 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-08-28T19:21:27.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:26 smithi028 ceph-mon[21769]: pgmap v351: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 853 B/s wr, 12 op/s 2024-08-28T19:21:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:28 smithi115 ceph-mon[22184]: pgmap v352: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 9 op/s 2024-08-28T19:21:29.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:28 smithi016 ceph-mon[18556]: pgmap v352: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 9 op/s 2024-08-28T19:21:29.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:28 smithi028 ceph-mon[21769]: pgmap v352: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 9 op/s 2024-08-28T19:21:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:30 smithi115 ceph-mon[22184]: pgmap v353: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 9 op/s 2024-08-28T19:21:31.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:30 smithi016 ceph-mon[18556]: pgmap v353: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 9 op/s 2024-08-28T19:21:31.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:30 smithi028 ceph-mon[21769]: pgmap v353: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 9 op/s 2024-08-28T19:21:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:32 smithi115 ceph-mon[22184]: pgmap v354: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 853 B/s wr, 12 op/s 2024-08-28T19:21:33.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:32 smithi016 ceph-mon[18556]: pgmap v354: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 853 B/s wr, 12 op/s 2024-08-28T19:21:33.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:32 smithi028 ceph-mon[21769]: pgmap v354: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 853 B/s wr, 12 op/s 2024-08-28T19:21:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:34 smithi115 ceph-mon[22184]: pgmap v355: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:21:35.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:34 smithi016 ceph-mon[18556]: pgmap v355: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:21:35.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:34 smithi028 ceph-mon[21769]: pgmap v355: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:21:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:35 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:21:36.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:35 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:21:36.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:35 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:21:36.640 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:21:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:37.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:36 smithi016 ceph-mon[18556]: pgmap v356: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:21:37.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:21:37.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:21:37.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:36 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:36 smithi028 ceph-mon[21769]: pgmap v356: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:21:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:21:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:21:37.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:36 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:36 smithi115 ceph-mon[22184]: pgmap v356: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:21:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:21:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:21:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:36 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:21:37.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.427+0000 7f55bf420640 1 -- 172.21.15.16:0/3123743094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b810dbf0 msgr2=0x7f55b81100e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:37.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.427+0000 7f55bf420640 1 --2- 172.21.15.16:0/3123743094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b810dbf0 0x7f55b81100e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009ca0 tx=0x7f55b002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.427+0000 7f55bf420640 1 -- 172.21.15.16:0/3123743094 shutdown_connections 2024-08-28T19:21:37.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.427+0000 7f55bf420640 1 --2- 172.21.15.16:0/3123743094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b810dbf0 0x7f55b81100e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.427+0000 7f55bf420640 1 --2- 172.21.15.16:0/3123743094 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b81050e0 0x7f55b810d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.427+0000 7f55bf420640 1 --2- 172.21.15.16:0/3123743094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b8104790 0x7f55b8104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.427+0000 7f55bf420640 1 -- 172.21.15.16:0/3123743094 >> 172.21.15.16:0/3123743094 conn(0x7f55b8079580 msgr2=0x7f55b8079990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.427+0000 7f55bf420640 1 -- 172.21.15.16:0/3123743094 shutdown_connections 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.427+0000 7f55bf420640 1 -- 172.21.15.16:0/3123743094 wait complete. 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.428+0000 7f55bf420640 1 Processor -- start 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.428+0000 7f55bf420640 1 -- start start 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.428+0000 7f55bf420640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b8104790 0x7f55b819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.428+0000 7f55bf420640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b81050e0 0x7f55b819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.428+0000 7f55bf420640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b810dbf0 0x7f55b819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.428+0000 7f55bf420640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55b8113850 con 0x7f55b810dbf0 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.428+0000 7f55bf420640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55b81136d0 con 0x7f55b81050e0 2024-08-28T19:21:37.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.428+0000 7f55bf420640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f55b81139d0 con 0x7f55b8104790 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bd9bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b810dbf0 0x7f55b819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bd9bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b810dbf0 0x7f55b819fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48024/0 (socket says 172.21.15.16:48024) 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bd9bd640 1 -- 172.21.15.16:0/878507233 learned_addr learned my addr 172.21.15.16:0/878507233 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b81050e0 0x7f55b819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bd1bc640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b8104790 0x7f55b819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bd9bd640 1 -- 172.21.15.16:0/878507233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b8104790 msgr2=0x7f55b819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bd9bd640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b8104790 0x7f55b819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bd9bd640 1 -- 172.21.15.16:0/878507233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b81050e0 msgr2=0x7f55b819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bd9bd640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b81050e0 0x7f55b819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bd9bd640 1 -- 172.21.15.16:0/878507233 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b81a0210 con 0x7f55b810dbf0 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bd9bd640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b810dbf0 0x7f55b819fae0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f55b0005cc0 tx=0x7f55b002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55ae7fc640 1 -- 172.21.15.16:0/878507233 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b0003bf0 con 0x7f55b810dbf0 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55ae7fc640 1 -- 172.21.15.16:0/878507233 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55b0003d90 con 0x7f55b810dbf0 2024-08-28T19:21:37.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bf420640 1 -- 172.21.15.16:0/878507233 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b81a04a0 con 0x7f55b810dbf0 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55ae7fc640 1 -- 172.21.15.16:0/878507233 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b002fbc0 con 0x7f55b810dbf0 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.429+0000 7f55bf420640 1 -- 172.21.15.16:0/878507233 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b81a6c60 con 0x7f55b810dbf0 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.432+0000 7f55c5995740 1 Processor -- start 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55ae7fc640 1 -- 172.21.15.16:0/878507233 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f55b002fd60 con 0x7f55b810dbf0 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55c5995740 1 -- start start 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55c5995740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d76c070 0x55713db11300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55c5995740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d7431b0 0x55713db11840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55c5995740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db11d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55c5995740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55713da88630 con 0x55713d74b390 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55c5995740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55713da77bc0 con 0x55713d76c070 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55c5995740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55713da7bb70 con 0x55713d7431b0 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55bc9bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db11d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55bc9bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db11d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48034/0 (socket says 172.21.15.16:48034) 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55bc9bb640 1 -- 172.21.15.16:0/2369072883 learned_addr learned my addr 172.21.15.16:0/2369072883 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/2369072883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d76c070 0x55713db11300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.433+0000 7f55bd1bc640 1 --2- 172.21.15.16:0/2369072883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d7431b0 0x55713db11840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55ae7fc640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f55a40b4c10 0x7f55a40b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55bc9bb640 1 -- 172.21.15.16:0/2369072883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d7431b0 msgr2=0x55713db11840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/2369072883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d7431b0 0x55713db11840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55bc9bb640 1 -- 172.21.15.16:0/2369072883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d76c070 msgr2=0x55713db11300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/2369072883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d76c070 0x55713db11300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55bc9bb640 1 -- 172.21.15.16:0/2369072883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55713d7d7650 con 0x55713d74b390 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55bd9bd640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f55a40b4c10 0x7f55a40b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55ae7fc640 1 -- 172.21.15.16:0/878507233 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f55b0106d10 con 0x7f55b810dbf0 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/2369072883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db11d80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f55a80126c0 tx=0x7f55a800b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cea9e834af136f4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55a37fe640 1 -- 172.21.15.16:0/2369072883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a8012c40 con 0x55713d74b390 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55a37fe640 1 -- 172.21.15.16:0/2369072883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55a8012de0 con 0x55713d74b390 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.434+0000 7f55a37fe640 1 -- 172.21.15.16:0/2369072883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a8005650 con 0x55713d74b390 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.435+0000 7f55c5995740 1 -- 172.21.15.16:0/2369072883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 msgr2=0x55713db11d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.435+0000 7f55c5995740 1 --2- 172.21.15.16:0/2369072883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db11d80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f55a80126c0 tx=0x7f55a800b6e0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.435+0000 7f55bd9bd640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f55a40b4c10 0x7f55a40b7120 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f55b00cce50 tx=0x7f55b00c19d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.435+0000 7f55c5995740 1 -- 172.21.15.16:0/2369072883 shutdown_connections 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.435+0000 7f55c5995740 1 --2- 172.21.15.16:0/2369072883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db11d80 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.435+0000 7f55c5995740 1 --2- 172.21.15.16:0/2369072883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d7431b0 0x55713db11840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.435+0000 7f55c5995740 1 --2- 172.21.15.16:0/2369072883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d76c070 0x55713db11300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.435+0000 7f55c5995740 1 -- 172.21.15.16:0/2369072883 >> 172.21.15.16:0/2369072883 conn(0x55713d760460 msgr2=0x55713db0f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.435+0000 7f55c5995740 1 -- 172.21.15.16:0/2369072883 shutdown_connections 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.435+0000 7f55c5995740 1 -- 172.21.15.16:0/2369072883 wait complete. 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.436+0000 7f55c5995740 1 -- start start 2024-08-28T19:21:37.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.436+0000 7f55c5995740 1 Processor -- start 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.436+0000 7f55c5995740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d7431b0 0x55713db1ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.436+0000 7f55c5995740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d74b390 0x55713db1cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.436+0000 7f55c5995740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d76c070 0x55713db28e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.436+0000 7f55c5995740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55713d8da700 con 0x55713d76c070 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.436+0000 7f55c5995740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55713d9fe700 con 0x55713d74b390 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.436+0000 7f55c5995740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55713db1fae0 con 0x55713d7431b0 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.436+0000 7f55c5995740 10 client.0 _ll_register_callbacks cb 0x55713da2ddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bc9bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d76c070 0x55713db28e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bc9bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d76c070 0x55713db28e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48042/0 (socket says 172.21.15.16:48042) 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bc9bb640 1 -- 172.21.15.16:0/1564767026 learned_addr learned my addr 172.21.15.16:0/1564767026 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d7431b0 0x55713db1ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bd1bc640 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d74b390 0x55713db1cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bc9bb640 1 -- 172.21.15.16:0/1564767026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d7431b0 msgr2=0x55713db1ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d7431b0 0x55713db1ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bc9bb640 1 -- 172.21.15.16:0/1564767026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d74b390 msgr2=0x55713db1cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d74b390 0x55713db1cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bc9bb640 1 -- 172.21.15.16:0/1564767026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55713da378b0 con 0x55713d76c070 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bd1bc640 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d74b390 0x55713db1cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d76c070 0x55713db28e00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x55713db1f270 tx=0x7f55a801b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce70a9fd53cc7cd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55a0ff9640 1 -- 172.21.15.16:0/1564767026 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a8010a40 con 0x55713d76c070 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55a0ff9640 1 -- 172.21.15.16:0/1564767026 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f55a8010be0 con 0x55713d76c070 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.437+0000 7f55c5995740 1 -- 172.21.15.16:0/1564767026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55a8012de0 con 0x55713d76c070 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.438+0000 7f55a0ff9640 1 -- 172.21.15.16:0/1564767026 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f55a8034050 con 0x55713d76c070 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.438+0000 7f55c5995740 10 client.15801 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:37.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.438+0000 7f55c5995740 1 -- 172.21.15.16:0/1564767026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55713db2b880 con 0x55713d76c070 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.438+0000 7f55a0ff9640 1 -- 172.21.15.16:0/1564767026 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55a8012de0 con 0x55713d76c070 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.438+0000 7f55a0ff9640 1 -- 172.21.15.16:0/1564767026 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f55a802e4f0 con 0x55713d76c070 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.438+0000 7f55c5995740 10 client.15801 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.438+0000 7f55c5995740 10 client.15801 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.438+0000 7f55c5995740 20 client.15801 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.438+0000 7f55c5995740 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55713d72a9f0 0x55713db2ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.438+0000 7f55c5995740 4 client.15801 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.438+0000 7f55c5995740 1 -- 172.21.15.16:0/1564767026 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55713da8a580 con 0x55713d72a9f0 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.439+0000 7f55bd1bc640 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55713d72a9f0 0x55713db2ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.439+0000 7f55a0ff9640 10 client.15801 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.439+0000 7f55bd1bc640 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55713d72a9f0 0x55713db2ebe0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74f109dda5baf2b9 server_cookie=f104fee539e4a68e in_seq=0 out_seq=0 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55a0ff9640 1 -- 172.21.15.16:0/1564767026 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55713da8a580 con 0x55713d72a9f0 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55a0ff9640 10 client.15801 handle_command_reply: tid=0 2024-08-28T19:21:37.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55c5995740 1 client.15801 shutdown 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55c5995740 1 -- 172.21.15.16:0/1564767026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d76c070 msgr2=0x55713db28e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55c5995740 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d76c070 0x55713db28e00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x55713db1f270 tx=0x7f55a801b040 comp rx=0 tx=0).stop 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55c5995740 1 -- 172.21.15.16:0/1564767026 shutdown_connections 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55c5995740 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d76c070 0x55713db28e00 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55c5995740 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d74b390 0x55713db1cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55c5995740 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55713d72a9f0 0x55713db2ebe0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55c5995740 1 --2- 172.21.15.16:0/1564767026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d7431b0 0x55713db1ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55a0ff9640 0 client.15801 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.440+0000 7f55c5995740 1 -- 172.21.15.16:0/1564767026 >> 172.21.15.16:0/1564767026 conn(0x55713d760460 msgr2=0x55713db14490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.441+0000 7f55c5995740 1 -- 172.21.15.16:0/1564767026 shutdown_connections 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.441+0000 7f55c5995740 1 -- 172.21.15.16:0/1564767026 wait complete. 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.441+0000 7f55c5995740 20 client.15801 trim_cache size 0 max 16384 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.459+0000 7f55c5995740 1 Processor -- start 2024-08-28T19:21:37.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.459+0000 7f55c5995740 1 -- start start 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55c5995740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 0x55713db14b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55c5995740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db1c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55c5995740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d76c070 0x55713db19c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55c5995740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55b81b0c80 con 0x55713d74b390 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55c5995740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55b81b27c0 con 0x55713d7431b0 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55c5995740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f55b81b1350 con 0x55713d76c070 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bd1bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db1c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bd1bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db1c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48050/0 (socket says 172.21.15.16:48050) 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bd1bc640 1 -- 172.21.15.16:0/1136524361 learned_addr learned my addr 172.21.15.16:0/1136524361 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/1136524361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d76c070 0x55713db19c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/1136524361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 0x55713db14b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bd1bc640 1 -- 172.21.15.16:0/1136524361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d76c070 msgr2=0x55713db19c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bd1bc640 1 --2- 172.21.15.16:0/1136524361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d76c070 0x55713db19c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bd1bc640 1 -- 172.21.15.16:0/1136524361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 msgr2=0x55713db14b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bd1bc640 1 --2- 172.21.15.16:0/1136524361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 0x55713db14b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bd1bc640 1 -- 172.21.15.16:0/1136524361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55713db1b980 con 0x55713d74b390 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/1136524361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 0x55713db14b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.460+0000 7f55bd1bc640 1 --2- 172.21.15.16:0/1136524361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db1c990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x55713d8da7c0 tx=0x7f55b40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739374535d28196f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.461+0000 7f55a2ffd640 1 -- 172.21.15.16:0/1136524361 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b401d620 con 0x55713d74b390 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.461+0000 7f55a2ffd640 1 -- 172.21.15.16:0/1136524361 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55b401dc70 con 0x55713d74b390 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.461+0000 7f55a2ffd640 1 -- 172.21.15.16:0/1136524361 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4024e40 con 0x55713d74b390 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.461+0000 7f55c5995740 1 -- 172.21.15.16:0/1136524361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 msgr2=0x55713db1c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.461+0000 7f55c5995740 1 --2- 172.21.15.16:0/1136524361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db1c990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x55713d8da7c0 tx=0x7f55b40118d0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.462+0000 7f55c5995740 1 -- 172.21.15.16:0/1136524361 shutdown_connections 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.462+0000 7f55c5995740 1 --2- 172.21.15.16:0/1136524361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d76c070 0x55713db19c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.462+0000 7f55c5995740 1 --2- 172.21.15.16:0/1136524361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db1c990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.462+0000 7f55c5995740 1 --2- 172.21.15.16:0/1136524361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 0x55713db14b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.462+0000 7f55c5995740 1 -- 172.21.15.16:0/1136524361 >> 172.21.15.16:0/1136524361 conn(0x55713d760460 msgr2=0x55713db14400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.462+0000 7f55c5995740 1 -- 172.21.15.16:0/1136524361 shutdown_connections 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.462+0000 7f55c5995740 1 -- 172.21.15.16:0/1136524361 wait complete. 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.462+0000 7f55c5995740 1 -- start start 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.462+0000 7f55c5995740 1 Processor -- start 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55c5995740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 0x55713db2fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55c5995740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db302c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55c5995740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d76c070 0x55713db30800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55c5995740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55b81b20f0 con 0x55713d74b390 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55c5995740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55b81b1a20 con 0x55713d7431b0 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55c5995740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f55b81aec60 con 0x55713d76c070 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55c5995740 10 client.0 _ll_register_callbacks cb 0x7f55a800bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55bc9bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 0x55713db2fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55bc9bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 0x55713db2fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:55336/0 (socket says 172.21.15.16:55336) 2024-08-28T19:21:37.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55bc9bb640 1 -- 172.21.15.16:0/853484629 learned_addr learned my addr 172.21.15.16:0/853484629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d76c070 0x55713db30800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55bc9bb640 1 -- 172.21.15.16:0/853484629 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d76c070 msgr2=0x55713db30800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d76c070 0x55713db30800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55bc9bb640 1 -- 172.21.15.16:0/853484629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 msgr2=0x55713db302c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db302c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.463+0000 7f55bc9bb640 1 -- 172.21.15.16:0/853484629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55713db984e0 con 0x55713d7431b0 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.464+0000 7f55bc9bb640 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 0x55713db2fd80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f55a8032040 tx=0x7f55a8017040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75bbeb74b1d0afd5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.464+0000 7f55a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.464+0000 7f55a0ff9640 1 -- 172.21.15.16:0/853484629 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a800bac0 con 0x55713d7431b0 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.464+0000 7f55c5995740 1 -- 172.21.15.16:0/853484629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55b401dc70 con 0x55713d7431b0 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.464+0000 7f55a0ff9640 1 -- 172.21.15.16:0/853484629 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f55a8010c50 con 0x55713d7431b0 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.464+0000 7f55a0ff9640 1 -- 172.21.15.16:0/853484629 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f55a8045050 con 0x55713d7431b0 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.464+0000 7f55c5995740 10 client.25445 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:37.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.464+0000 7f55c5995740 1 -- 172.21.15.16:0/853484629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55713db997a0 con 0x55713d7431b0 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.464+0000 7f55a0ff9640 1 -- 172.21.15.16:0/853484629 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55b401dc70 con 0x55713d7431b0 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.464+0000 7f55a0ff9640 1 -- 172.21.15.16:0/853484629 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f55a802e340 con 0x55713d7431b0 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55c5995740 10 client.25445 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55c5995740 10 client.25445 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55c5995740 20 client.25445 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55c5995740 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55713d6ffda0 0x55713db9bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55c5995740 4 client.25445 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55c5995740 1 -- 172.21.15.16:0/853484629 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55713da86020 con 0x55713d6ffda0 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55bd1bc640 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55713d6ffda0 0x55713db9bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55bd1bc640 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55713d6ffda0 0x55713db9bf60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8af5d504f7137c87 server_cookie=c840bc8d033605a8 in_seq=0 out_seq=0 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55a0ff9640 10 client.25445 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55a0ff9640 1 -- 172.21.15.16:0/853484629 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+374 (crc 0 0 0) 0x55713da86020 con 0x55713d6ffda0 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55a0ff9640 10 client.25445 handle_command_reply: tid=0 2024-08-28T19:21:37.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.465+0000 7f55c5995740 1 client.25445 shutdown 2024-08-28T19:21:37.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55c5995740 1 -- 172.21.15.16:0/853484629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 msgr2=0x55713db2fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:37.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55c5995740 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 0x55713db2fd80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f55a8032040 tx=0x7f55a8017040 comp rx=0 tx=0).stop 2024-08-28T19:21:37.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55c5995740 1 -- 172.21.15.16:0/853484629 shutdown_connections 2024-08-28T19:21:37.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55c5995740 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55713d6ffda0 0x55713db9bf60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55c5995740 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55713d76c070 0x55713db30800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55c5995740 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55713d74b390 0x55713db302c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55a0ff9640 0 client.25445 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:37.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55c5995740 1 --2- 172.21.15.16:0/853484629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55713d7431b0 0x55713db2fd80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55c5995740 1 -- 172.21.15.16:0/853484629 >> 172.21.15.16:0/853484629 conn(0x55713d760460 msgr2=0x55713db12330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:37.542 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:37.543 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:21:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55c5995740 1 -- 172.21.15.16:0/853484629 shutdown_connections 2024-08-28T19:21:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55c5995740 1 -- 172.21.15.16:0/853484629 wait complete. 2024-08-28T19:21:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55c5995740 20 client.25445 trim_cache size 0 max 16384 2024-08-28T19:21:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55bf420640 1 -- 172.21.15.16:0/878507233 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f55a40b4c10 msgr2=0x7f55a40b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55bf420640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f55a40b4c10 0x7f55a40b7120 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f55b00cce50 tx=0x7f55b00c19d0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55bf420640 1 -- 172.21.15.16:0/878507233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b810dbf0 msgr2=0x7f55b819fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.466+0000 7f55bf420640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b810dbf0 0x7f55b819fae0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f55b0005cc0 tx=0x7f55b002f380 comp rx=0 tx=0).stop 2024-08-28T19:21:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.467+0000 7f55bf420640 1 -- 172.21.15.16:0/878507233 shutdown_connections 2024-08-28T19:21:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.467+0000 7f55bf420640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b810dbf0 0x7f55b819fae0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.467+0000 7f55bf420640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f55a40b4c10 0x7f55a40b7120 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.467+0000 7f55bf420640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b81050e0 0x7f55b819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.467+0000 7f55bf420640 1 --2- 172.21.15.16:0/878507233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b8104790 0x7f55b819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.467+0000 7f55bf420640 1 -- 172.21.15.16:0/878507233 >> 172.21.15.16:0/878507233 conn(0x7f55b8079580 msgr2=0x7f55b80ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.467+0000 7f55bf420640 1 -- 172.21.15.16:0/878507233 shutdown_connections 2024-08-28T19:21:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:37.467+0000 7f55bf420640 1 -- 172.21.15.16:0/878507233 wait complete. 2024-08-28T19:21:38.400 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:21:38.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:21:38.670 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:38.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:38 smithi016 ceph-mon[18556]: pgmap v357: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:21:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:38 smithi028 ceph-mon[21769]: pgmap v357: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:21:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:38 smithi115 ceph-mon[22184]: pgmap v357: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:21:39.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.194+0000 7f6dac359640 1 -- 172.21.15.16:0/1121579412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da4109bd0 msgr2=0x7f6da4110400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:39.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.194+0000 7f6dac359640 1 --2- 172.21.15.16:0/1121579412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da4109bd0 0x7f6da4110400 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009ca0 tx=0x7f6d9c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:39.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.194+0000 7f6dac359640 1 -- 172.21.15.16:0/1121579412 shutdown_connections 2024-08-28T19:21:39.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.194+0000 7f6dac359640 1 --2- 172.21.15.16:0/1121579412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da4109bd0 0x7f6da4110400 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.194+0000 7f6dac359640 1 --2- 172.21.15.16:0/1121579412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da4109280 0x7f6da4109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.194+0000 7f6dac359640 1 --2- 172.21.15.16:0/1121579412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da4103940 0x7f6da4103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.194+0000 7f6dac359640 1 -- 172.21.15.16:0/1121579412 >> 172.21.15.16:0/1121579412 conn(0x7f6da40ff3d0 msgr2=0x7f6da4101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.194+0000 7f6dac359640 1 -- 172.21.15.16:0/1121579412 shutdown_connections 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.194+0000 7f6dac359640 1 -- 172.21.15.16:0/1121579412 wait complete. 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.195+0000 7f6dac359640 1 Processor -- start 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.195+0000 7f6dac359640 1 -- start start 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6dac359640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da4103940 0x7f6da419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6dac359640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da4109280 0x7f6da419a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6dac359640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da4109bd0 0x7f6da419fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6dac359640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6da41138f0 con 0x7f6da4109280 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6dac359640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4113770 con 0x7f6da4103940 2024-08-28T19:21:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6dac359640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4113a70 con 0x7f6da4109bd0 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6da98f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da4109280 0x7f6da419a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6da98f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da4109280 0x7f6da419a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48072/0 (socket says 172.21.15.16:48072) 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6da98f4640 1 -- 172.21.15.16:0/940477548 learned_addr learned my addr 172.21.15.16:0/940477548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6daa0f5640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da4103940 0x7f6da419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6daa8f6640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da4109bd0 0x7f6da419fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6da98f4640 1 -- 172.21.15.16:0/940477548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da4109bd0 msgr2=0x7f6da419fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6da98f4640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da4109bd0 0x7f6da419fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6da98f4640 1 -- 172.21.15.16:0/940477548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da4103940 msgr2=0x7f6da419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6da98f4640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da4103940 0x7f6da419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6da98f4640 1 -- 172.21.15.16:0/940477548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da41a0250 con 0x7f6da4109280 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6daa8f6640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da4109bd0 0x7f6da419fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.196+0000 7f6da98f4640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da4109280 0x7f6da419a880 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400c970 tx=0x7f6d9400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.197+0000 7f6d9b7fe640 1 -- 172.21.15.16:0/940477548 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94007bb0 con 0x7f6da4109280 2024-08-28T19:21:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.197+0000 7f6d9b7fe640 1 -- 172.21.15.16:0/940477548 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d94007d50 con 0x7f6da4109280 2024-08-28T19:21:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.197+0000 7f6dac359640 1 -- 172.21.15.16:0/940477548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da41a0540 con 0x7f6da4109280 2024-08-28T19:21:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.197+0000 7f6d9b7fe640 1 -- 172.21.15.16:0/940477548 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94004e10 con 0x7f6da4109280 2024-08-28T19:21:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.197+0000 7f6dac359640 1 -- 172.21.15.16:0/940477548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da41a6ca0 con 0x7f6da4109280 2024-08-28T19:21:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.198+0000 7f6dac359640 1 -- 172.21.15.16:0/940477548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da4103d50 con 0x7f6da4109280 2024-08-28T19:21:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.200+0000 7f6d9b7fe640 1 -- 172.21.15.16:0/940477548 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f6d94005000 con 0x7f6da4109280 2024-08-28T19:21:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.201+0000 7f6d9b7fe640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d900b4c10 0x7f6d900b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.201+0000 7f6daa0f5640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d900b4c10 0x7f6d900b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.201+0000 7f6d9b7fe640 1 -- 172.21.15.16:0/940477548 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6d940e5620 con 0x7f6da4109280 2024-08-28T19:21:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.201+0000 7f6daa0f5640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d900b4c10 0x7f6d900b7120 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009890 tx=0x7f6da0008040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.205+0000 7f6d9b7fe640 1 -- 172.21.15.16:0/940477548 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f6d94014030 con 0x7f6da4109280 2024-08-28T19:21:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.441+0000 7f6dac359640 1 -- 172.21.15.16:0/940477548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6da402e430 con 0x7f6da4109280 2024-08-28T19:21:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.442+0000 7f6d9b7fe640 1 -- 172.21.15.16:0/940477548 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11977 (secure 0 0 0) 0x7f6d940ac100 con 0x7f6da4109280 2024-08-28T19:21:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:21:39.444 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"btime":"2024-08-28T19:20:55:257311+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:20:54.880238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:21:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2024-08-28T19:21:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.444+0000 7f6dac359640 1 -- 172.21.15.16:0/940477548 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d900b4c10 msgr2=0x7f6d900b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.444+0000 7f6dac359640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d900b4c10 0x7f6d900b7120 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009890 tx=0x7f6da0008040 comp rx=0 tx=0).stop 2024-08-28T19:21:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.444+0000 7f6dac359640 1 -- 172.21.15.16:0/940477548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da4109280 msgr2=0x7f6da419a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.444+0000 7f6dac359640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da4109280 0x7f6da419a880 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400c970 tx=0x7f6d9400ce30 comp rx=0 tx=0).stop 2024-08-28T19:21:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.444+0000 7f6dac359640 1 -- 172.21.15.16:0/940477548 shutdown_connections 2024-08-28T19:21:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.444+0000 7f6dac359640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6da4109bd0 0x7f6da419fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.444+0000 7f6dac359640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da4109280 0x7f6da419a880 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.444+0000 7f6dac359640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6d900b4c10 0x7f6d900b7120 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.444+0000 7f6dac359640 1 --2- 172.21.15.16:0/940477548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da4103940 0x7f6da419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.444+0000 7f6dac359640 1 -- 172.21.15.16:0/940477548 >> 172.21.15.16:0/940477548 conn(0x7f6da40ff3d0 msgr2=0x7f6da4108890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.445+0000 7f6dac359640 1 -- 172.21.15.16:0/940477548 shutdown_connections 2024-08-28T19:21:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:39.445+0000 7f6dac359640 1 -- 172.21.15.16:0/940477548 wait complete. 2024-08-28T19:21:39.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:39 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/940477548' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:40.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:39 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/940477548' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:40.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:21:40.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:39 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/940477548' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:40.376 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:40.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:40 smithi016 ceph-mon[18556]: pgmap v358: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-08-28T19:21:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.899+0000 7f15d0c78640 1 -- 172.21.15.16:0/171064596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc108aa0 msgr2=0x7f15cc10f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.899+0000 7f15d0c78640 1 --2- 172.21.15.16:0/171064596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc108aa0 0x7f15cc10f2d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009ca0 tx=0x7f15c402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.899+0000 7f15d0c78640 1 -- 172.21.15.16:0/171064596 shutdown_connections 2024-08-28T19:21:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.899+0000 7f15d0c78640 1 --2- 172.21.15.16:0/171064596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc108aa0 0x7f15cc10f2d0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.899+0000 7f15d0c78640 1 --2- 172.21.15.16:0/171064596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc108150 0x7f15cc108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.899+0000 7f15d0c78640 1 --2- 172.21.15.16:0/171064596 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15cc102810 0x7f15cc102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.899+0000 7f15d0c78640 1 -- 172.21.15.16:0/171064596 >> 172.21.15.16:0/171064596 conn(0x7f15cc0794d0 msgr2=0x7f15cc0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:40.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.899+0000 7f15d0c78640 1 -- 172.21.15.16:0/171064596 shutdown_connections 2024-08-28T19:21:40.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.899+0000 7f15d0c78640 1 -- 172.21.15.16:0/171064596 wait complete. 2024-08-28T19:21:40.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.900+0000 7f15d0c78640 1 Processor -- start 2024-08-28T19:21:40.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15d0c78640 1 -- start start 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15d0c78640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc102810 0x7f15cc19afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15d0c78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc108150 0x7f15cc19b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15d0c78640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15cc108aa0 0x7f15cc19ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15d0c78640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc112700 con 0x7f15cc108150 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15d0c78640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc112580 con 0x7f15cc102810 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15d0c78640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc112880 con 0x7f15cc108aa0 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15c9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc108150 0x7f15cc19b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15c9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc108150 0x7f15cc19b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48090/0 (socket says 172.21.15.16:48090) 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15c9d9b640 1 -- 172.21.15.16:0/1429220278 learned_addr learned my addr 172.21.15.16:0/1429220278 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15ca59c640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc102810 0x7f15cc19afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.901+0000 7f15cad9d640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15cc108aa0 0x7f15cc19ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.902+0000 7f15c9d9b640 1 -- 172.21.15.16:0/1429220278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15cc108aa0 msgr2=0x7f15cc19ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.902+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15cc108aa0 0x7f15cc19ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.902+0000 7f15c9d9b640 1 -- 172.21.15.16:0/1429220278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc102810 msgr2=0x7f15cc19afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.902+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc102810 0x7f15cc19afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.902+0000 7f15c9d9b640 1 -- 172.21.15.16:0/1429220278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15cc1a61d0 con 0x7f15cc108150 2024-08-28T19:21:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.902+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc108150 0x7f15cc19b510 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f15c000e9c0 tx=0x7f15c000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.902+0000 7f15bb7fe640 1 -- 172.21.15.16:0/1429220278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c000ccc0 con 0x7f15cc108150 2024-08-28T19:21:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.902+0000 7f15d0c78640 1 -- 172.21.15.16:0/1429220278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15cc1a64c0 con 0x7f15cc108150 2024-08-28T19:21:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.902+0000 7f15bb7fe640 1 -- 172.21.15.16:0/1429220278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15c00054d0 con 0x7f15cc108150 2024-08-28T19:21:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.902+0000 7f15bb7fe640 1 -- 172.21.15.16:0/1429220278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c0010430 con 0x7f15cc108150 2024-08-28T19:21:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.902+0000 7f15d0c78640 1 -- 172.21.15.16:0/1429220278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15cc1a6a50 con 0x7f15cc108150 2024-08-28T19:21:40.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.905+0000 7f15d71ed740 1 Processor -- start 2024-08-28T19:21:40.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.905+0000 7f15d71ed740 1 -- start start 2024-08-28T19:21:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376280c070 0x563762bb1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627e31b0 0x563762bb1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637627eb390 0x563762bb1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15d71ed740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563762b28680 con 0x5637627e31b0 2024-08-28T19:21:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15d71ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563762b17c10 con 0x5637627eb390 2024-08-28T19:21:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15d71ed740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563762b1bbc0 con 0x56376280c070 2024-08-28T19:21:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15ca59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627e31b0 0x563762bb1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15ca59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627e31b0 0x563762bb1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48104/0 (socket says 172.21.15.16:48104) 2024-08-28T19:21:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15ca59c640 1 -- 172.21.15.16:0/749540008 learned_addr learned my addr 172.21.15.16:0/749540008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15bb7fe640 1 -- 172.21.15.16:0/1429220278 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f15c0004f20 con 0x7f15cc108150 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15cad9d640 1 --2- 172.21.15.16:0/749540008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376280c070 0x563762bb1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15ca59c640 1 -- 172.21.15.16:0/749540008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376280c070 msgr2=0x563762bb1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15ca59c640 1 --2- 172.21.15.16:0/749540008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376280c070 0x563762bb1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15ca59c640 1 -- 172.21.15.16:0/749540008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637627eb390 msgr2=0x563762bb1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15ca59c640 1 --2- 172.21.15.16:0/749540008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637627eb390 0x563762bb1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.906+0000 7f15ca59c640 1 -- 172.21.15.16:0/749540008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563762877650 con 0x5637627e31b0 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.907+0000 7f15ca59c640 1 --2- 172.21.15.16:0/749540008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627e31b0 0x563762bb1850 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00ac80 tx=0x7f15bc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84140af26f6752f9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.907+0000 7f15b8ff9640 1 -- 172.21.15.16:0/749540008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc01d070 con 0x5637627e31b0 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.907+0000 7f15b8ff9640 1 -- 172.21.15.16:0/749540008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15bc018af0 con 0x5637627e31b0 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.907+0000 7f15b8ff9640 1 -- 172.21.15.16:0/749540008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc010b40 con 0x5637627e31b0 2024-08-28T19:21:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.907+0000 7f15bb7fe640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15b40b4cc0 0x7f15b40b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.907+0000 7f15cad9d640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15b40b4cc0 0x7f15b40b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.907+0000 7f15bb7fe640 1 -- 172.21.15.16:0/1429220278 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f15c00e5b30 con 0x7f15cc108150 2024-08-28T19:21:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.908+0000 7f15d71ed740 1 -- 172.21.15.16:0/749540008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627e31b0 msgr2=0x563762bb1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.908+0000 7f15d71ed740 1 --2- 172.21.15.16:0/749540008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627e31b0 0x563762bb1850 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00ac80 tx=0x7f15bc00f790 comp rx=0 tx=0).stop 2024-08-28T19:21:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.908+0000 7f15cad9d640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15b40b4cc0 0x7f15b40b71d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f15c402f500 tx=0x7f15c402f330 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.908+0000 7f15d71ed740 1 -- 172.21.15.16:0/749540008 shutdown_connections 2024-08-28T19:21:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.908+0000 7f15d71ed740 1 --2- 172.21.15.16:0/749540008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637627eb390 0x563762bb1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.908+0000 7f15d71ed740 1 --2- 172.21.15.16:0/749540008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627e31b0 0x563762bb1850 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.908+0000 7f15d71ed740 1 --2- 172.21.15.16:0/749540008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376280c070 0x563762bb1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.908+0000 7f15d71ed740 1 -- 172.21.15.16:0/749540008 >> 172.21.15.16:0/749540008 conn(0x563762800460 msgr2=0x563762baf200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.908+0000 7f15d71ed740 1 -- 172.21.15.16:0/749540008 shutdown_connections 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.908+0000 7f15d71ed740 1 -- 172.21.15.16:0/749540008 wait complete. 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15d71ed740 1 -- start start 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15d71ed740 1 Processor -- start 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762bbcb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bbd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376280c070 0x563762bcd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15d71ed740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56376297a700 con 0x5637627eb390 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15d71ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563762a9e750 con 0x56376280c070 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15d71ed740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563762bbfaf0 con 0x5637627e31b0 2024-08-28T19:21:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15d71ed740 10 client.0 _ll_register_callbacks cb 0x563762acde10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15c9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bbd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15c9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bbd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48118/0 (socket says 172.21.15.16:48118) 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3622492587 learned_addr learned my addr 172.21.15.16:0/3622492587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762bbcb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.909+0000 7f15cad9d640 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376280c070 0x563762bcd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3622492587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 msgr2=0x563762bbcb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762bbcb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3622492587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376280c070 msgr2=0x563762bcd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376280c070 0x563762bcd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3622492587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563762ad7900 con 0x5637627eb390 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762bbcb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bbd060 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f15c00abf00 tx=0x7f15c00e34d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2466fe4cb9e75040 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3622492587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c00ac670 con 0x5637627eb390 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3622492587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f15c00b03f0 con 0x5637627eb390 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15d71ed740 1 -- 172.21.15.16:0/3622492587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15bc018af0 con 0x5637627eb390 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3622492587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f15c00c6050 con 0x5637627eb390 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15d71ed740 10 client.15822 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15d71ed740 1 -- 172.21.15.16:0/3622492587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563762bcfd60 con 0x5637627eb390 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.910+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3622492587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15bc018af0 con 0x5637627eb390 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.911+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3622492587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f15c00b4e00 con 0x5637627eb390 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.911+0000 7f15d71ed740 10 client.15822 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.911+0000 7f15d71ed740 10 client.15822 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.911+0000 7f15d71ed740 20 client.15822 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.911+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5637627ca9f0 0x563762bd3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.911+0000 7f15d71ed740 4 client.15822 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.911+0000 7f15d71ed740 1 -- 172.21.15.16:0/3622492587 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563762b2a5d0 con 0x5637627ca9f0 2024-08-28T19:21:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.911+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5637627ca9f0 0x563762bd3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.912+0000 7f15b1ffb640 10 client.15822 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:21:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.912+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5637627ca9f0 0x563762bd3230 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b902d6e9eecea89b server_cookie=4b63b5cb50dac19b in_seq=0 out_seq=0 2024-08-28T19:21:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3622492587 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563762b2a5d0 con 0x5637627ca9f0 2024-08-28T19:21:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15b1ffb640 10 client.15822 handle_command_reply: tid=0 2024-08-28T19:21:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 1 client.15822 shutdown 2024-08-28T19:21:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 1 -- 172.21.15.16:0/3622492587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 msgr2=0x563762bbd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bbd060 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f15c00abf00 tx=0x7f15c00e34d0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 1 -- 172.21.15.16:0/3622492587 shutdown_connections 2024-08-28T19:21:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5637627ca9f0 0x563762bd3230 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376280c070 0x563762bcd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bbd060 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15b1ffb640 0 client.15822 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:21:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3622492587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762bbcb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 1 -- 172.21.15.16:0/3622492587 >> 172.21.15.16:0/3622492587 conn(0x563762800460 msgr2=0x563762bb44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 1 -- 172.21.15.16:0/3622492587 shutdown_connections 2024-08-28T19:21:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 1 -- 172.21.15.16:0/3622492587 wait complete. 2024-08-28T19:21:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.913+0000 7f15d71ed740 20 client.15822 trim_cache size 0 max 16384 2024-08-28T19:21:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.940+0000 7f15d71ed740 1 Processor -- start 2024-08-28T19:21:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.940+0000 7f15d71ed740 1 -- start start 2024-08-28T19:21:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.940+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762baf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.940+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637627eb390 0x563762bcd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.940+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376280c070 0x563762bb4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.940+0000 7f15d71ed740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc1ae8e0 con 0x56376280c070 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.940+0000 7f15d71ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc1b0420 con 0x5637627eb390 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.940+0000 7f15d71ed740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc1aefb0 con 0x5637627e31b0 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.940+0000 7f15cad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376280c070 0x563762bb4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.940+0000 7f15cad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376280c070 0x563762bb4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48130/0 (socket says 172.21.15.16:48130) 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.940+0000 7f15cad9d640 1 -- 172.21.15.16:0/177035273 learned_addr learned my addr 172.21.15.16:0/177035273 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.941+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/177035273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637627eb390 0x563762bcd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.941+0000 7f15ca59c640 1 --2- 172.21.15.16:0/177035273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762baf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.941+0000 7f15cad9d640 1 -- 172.21.15.16:0/177035273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 msgr2=0x563762baf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.941+0000 7f15cad9d640 1 --2- 172.21.15.16:0/177035273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762baf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.941+0000 7f15cad9d640 1 -- 172.21.15.16:0/177035273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637627eb390 msgr2=0x563762bcd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.941+0000 7f15cad9d640 1 --2- 172.21.15.16:0/177035273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637627eb390 0x563762bcd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.941+0000 7f15cad9d640 1 -- 172.21.15.16:0/177035273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563762bbd9a0 con 0x56376280c070 2024-08-28T19:21:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.941+0000 7f15cad9d640 1 --2- 172.21.15.16:0/177035273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376280c070 0x563762bb4410 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f15c4035000 tx=0x7f15c4041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3084cecdfc5f648 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.941+0000 7f15b3fff640 1 -- 172.21.15.16:0/177035273 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c4003970 con 0x56376280c070 2024-08-28T19:21:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.941+0000 7f15b3fff640 1 -- 172.21.15.16:0/177035273 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15c4036050 con 0x56376280c070 2024-08-28T19:21:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.941+0000 7f15b3fff640 1 -- 172.21.15.16:0/177035273 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c4004800 con 0x56376280c070 2024-08-28T19:21:40.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.942+0000 7f15d71ed740 1 -- 172.21.15.16:0/177035273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376280c070 msgr2=0x563762bb4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:40.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.942+0000 7f15d71ed740 1 --2- 172.21.15.16:0/177035273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376280c070 0x563762bb4410 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f15c4035000 tx=0x7f15c4041000 comp rx=0 tx=0).stop 2024-08-28T19:21:40.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.942+0000 7f15d71ed740 1 -- 172.21.15.16:0/177035273 shutdown_connections 2024-08-28T19:21:40.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.942+0000 7f15d71ed740 1 --2- 172.21.15.16:0/177035273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376280c070 0x563762bb4410 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.942+0000 7f15d71ed740 1 --2- 172.21.15.16:0/177035273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637627eb390 0x563762bcd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.942+0000 7f15d71ed740 1 --2- 172.21.15.16:0/177035273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762baf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.942+0000 7f15d71ed740 1 -- 172.21.15.16:0/177035273 >> 172.21.15.16:0/177035273 conn(0x563762800460 msgr2=0x563762baeb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.942+0000 7f15d71ed740 1 -- 172.21.15.16:0/177035273 shutdown_connections 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.942+0000 7f15d71ed740 1 -- 172.21.15.16:0/177035273 wait complete. 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.943+0000 7f15d71ed740 1 -- start start 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.943+0000 7f15d71ed740 1 Processor -- start 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.943+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762bc5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.943+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bc59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.943+0000 7f15d71ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376280c070 0x563762c3cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.943+0000 7f15d71ed740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc1afd50 con 0x5637627eb390 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.943+0000 7f15d71ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc1af680 con 0x56376280c070 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.943+0000 7f15d71ed740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc1ac8c0 con 0x5637627e31b0 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.943+0000 7f15c9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bc59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15c9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bc59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48142/0 (socket says 172.21.15.16:48142) 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3590982387 learned_addr learned my addr 172.21.15.16:0/3590982387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15cad9d640 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376280c070 0x563762c3cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15d71ed740 10 client.0 _ll_register_callbacks cb 0x7f15c0009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762bc5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3590982387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 msgr2=0x563762bc5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762bc5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3590982387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376280c070 msgr2=0x563762c3cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376280c070 0x563762c3cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3590982387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563762bc61e0 con 0x5637627eb390 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15cad9d640 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376280c070 0x563762c3cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bc59d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f15c00b5170 tx=0x7f15c0004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4efa496ee819f6f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3590982387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c000ce60 con 0x5637627eb390 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3590982387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f15c00a8560 con 0x5637627eb390 2024-08-28T19:21:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15d71ed740 1 -- 172.21.15.16:0/3590982387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15c4036050 con 0x5637627eb390 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3590982387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f15c00e5df0 con 0x5637627eb390 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15d71ed740 10 client.15828 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.944+0000 7f15d71ed740 1 -- 172.21.15.16:0/3590982387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563762c3e070 con 0x5637627eb390 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.945+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3590982387 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15c4036050 con 0x5637627eb390 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.945+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3590982387 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f15c00cd2b0 con 0x5637627eb390 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.945+0000 7f15d71ed740 10 client.15828 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.945+0000 7f15d71ed740 10 client.15828 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.945+0000 7f15d71ed740 20 client.15828 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.945+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56376279fda0 0x563762c407a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.945+0000 7f15d71ed740 4 client.15828 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.945+0000 7f15d71ed740 1 -- 172.21.15.16:0/3590982387 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563762b26070 con 0x56376279fda0 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.945+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56376279fda0 0x563762c407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56376279fda0 0x563762c407a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fcf7876f49df0a59 server_cookie=b42ccd7582194a78 in_seq=0 out_seq=0 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15b1ffb640 10 client.15828 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:21:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15b1ffb640 1 -- 172.21.15.16:0/3590982387 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563762b26070 con 0x56376279fda0 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15b1ffb640 10 client.15828 handle_command_reply: tid=0 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15d71ed740 1 client.15828 shutdown 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15d71ed740 1 -- 172.21.15.16:0/3590982387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 msgr2=0x563762bc59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bc59d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f15c00b5170 tx=0x7f15c0004840 comp rx=0 tx=0).stop 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15d71ed740 1 -- 172.21.15.16:0/3590982387 shutdown_connections 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56376279fda0 0x563762c407a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376280c070 0x563762c3cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637627eb390 0x563762bc59d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15d71ed740 1 --2- 172.21.15.16:0/3590982387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5637627e31b0 0x563762bc5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.946+0000 7f15d71ed740 1 -- 172.21.15.16:0/3590982387 >> 172.21.15.16:0/3590982387 conn(0x563762800460 msgr2=0x563762bb4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15b1ffb640 0 client.15828 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d71ed740 1 -- 172.21.15.16:0/3590982387 shutdown_connections 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d71ed740 1 -- 172.21.15.16:0/3590982387 wait complete. 2024-08-28T19:21:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d71ed740 20 client.15828 trim_cache size 0 max 16384 2024-08-28T19:21:40.953 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:21:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d0c78640 1 -- 172.21.15.16:0/1429220278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15b40b4cc0 msgr2=0x7f15b40b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d0c78640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15b40b4cc0 0x7f15b40b71d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f15c402f500 tx=0x7f15c402f330 comp rx=0 tx=0).stop 2024-08-28T19:21:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d0c78640 1 -- 172.21.15.16:0/1429220278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc108150 msgr2=0x7f15cc19b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d0c78640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc108150 0x7f15cc19b510 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f15c000e9c0 tx=0x7f15c000edc0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d0c78640 1 -- 172.21.15.16:0/1429220278 shutdown_connections 2024-08-28T19:21:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d0c78640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15cc108aa0 0x7f15cc19ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d0c78640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc108150 0x7f15cc19b510 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d0c78640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15b40b4cc0 0x7f15b40b71d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d0c78640 1 --2- 172.21.15.16:0/1429220278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc102810 0x7f15cc19afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.947+0000 7f15d0c78640 1 -- 172.21.15.16:0/1429220278 >> 172.21.15.16:0/1429220278 conn(0x7f15cc0794d0 msgr2=0x7f15cc104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.948+0000 7f15d0c78640 1 -- 172.21.15.16:0/1429220278 shutdown_connections 2024-08-28T19:21:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:40.948+0000 7f15d0c78640 1 -- 172.21.15.16:0/1429220278 wait complete. 2024-08-28T19:21:41.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:40 smithi028 ceph-mon[21769]: pgmap v358: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-08-28T19:21:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:40 smithi115 ceph-mon[22184]: pgmap v358: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-08-28T19:21:41.757 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:21:41.966 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 34.4 2024-08-28T19:21:41.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:21:42.022 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:42.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.562+0000 7fba7f3d5640 1 -- 172.21.15.16:0/1663254611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba7810dbb0 msgr2=0x7fba781100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:42.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.562+0000 7fba7f3d5640 1 --2- 172.21.15.16:0/1663254611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba7810dbb0 0x7fba781100a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fba70009ca0 tx=0x7fba7002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.562+0000 7fba7f3d5640 1 -- 172.21.15.16:0/1663254611 shutdown_connections 2024-08-28T19:21:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.562+0000 7fba7f3d5640 1 --2- 172.21.15.16:0/1663254611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba7810dbb0 0x7fba781100a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.562+0000 7fba7f3d5640 1 --2- 172.21.15.16:0/1663254611 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba781050a0 0x7fba7810d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.562+0000 7fba7f3d5640 1 --2- 172.21.15.16:0/1663254611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba78104750 0x7fba78104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.562+0000 7fba7f3d5640 1 -- 172.21.15.16:0/1663254611 >> 172.21.15.16:0/1663254611 conn(0x7fba780794d0 msgr2=0x7fba780798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.562+0000 7fba7f3d5640 1 -- 172.21.15.16:0/1663254611 shutdown_connections 2024-08-28T19:21:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.562+0000 7fba7f3d5640 1 -- 172.21.15.16:0/1663254611 wait complete. 2024-08-28T19:21:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.563+0000 7fba7f3d5640 1 Processor -- start 2024-08-28T19:21:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7f3d5640 1 -- start start 2024-08-28T19:21:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7f3d5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba78104750 0x7fba7819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7f3d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba781050a0 0x7fba7819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7f3d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7810dbb0 0x7fba7819fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7f3d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba78113810 con 0x7fba781050a0 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7f3d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba78113690 con 0x7fba7810dbb0 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7f3d5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fba78113990 con 0x7fba78104750 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7c970640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba781050a0 0x7fba7819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7c970640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba781050a0 0x7fba7819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48150/0 (socket says 172.21.15.16:48150) 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7c970640 1 -- 172.21.15.16:0/322096122 learned_addr learned my addr 172.21.15.16:0/322096122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7d171640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba78104750 0x7fba7819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7d972640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7810dbb0 0x7fba7819fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7c970640 1 -- 172.21.15.16:0/322096122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba78104750 msgr2=0x7fba7819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7c970640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba78104750 0x7fba7819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7c970640 1 -- 172.21.15.16:0/322096122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7810dbb0 msgr2=0x7fba7819fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7c970640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7810dbb0 0x7fba7819fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7c970640 1 -- 172.21.15.16:0/322096122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba781a0070 con 0x7fba781050a0 2024-08-28T19:21:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.564+0000 7fba7d972640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7810dbb0 0x7fba7819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.565+0000 7fba7c970640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba781050a0 0x7fba7819a760 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fba6800c8c0 tx=0x7fba6800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.565+0000 7fba6e7fc640 1 -- 172.21.15.16:0/322096122 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68007bb0 con 0x7fba781050a0 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.565+0000 7fba6e7fc640 1 -- 172.21.15.16:0/322096122 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba68007d50 con 0x7fba781050a0 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.565+0000 7fba7f3d5640 1 -- 172.21.15.16:0/322096122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba781a0360 con 0x7fba781050a0 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.565+0000 7fba6e7fc640 1 -- 172.21.15.16:0/322096122 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba680187c0 con 0x7fba781050a0 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.565+0000 7fba7f3d5640 1 -- 172.21.15.16:0/322096122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba781a6b80 con 0x7fba781050a0 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.568+0000 7fba8594a740 1 Processor -- start 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.568+0000 7fba8594a740 1 -- start start 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.568+0000 7fba8594a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b793d070 0x55a8b7ce2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.568+0000 7fba8594a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b79141b0 0x55a8b7ce2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.568+0000 7fba8594a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b791c390 0x55a8b7ce2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.568+0000 7fba8594a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a8b7c59680 con 0x55a8b791c390 2024-08-28T19:21:42.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.568+0000 7fba8594a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a8b7c48c10 con 0x55a8b793d070 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.568+0000 7fba8594a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a8b7c4cbc0 con 0x55a8b79141b0 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.568+0000 7fba6e7fc640 1 -- 172.21.15.16:0/322096122 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7fba6800fa20 con 0x7fba781050a0 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba7c970640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b791c390 0x55a8b7ce2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba7d972640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b793d070 0x55a8b7ce2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba7d972640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b793d070 0x55a8b7ce2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:55426/0 (socket says 172.21.15.16:55426) 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba7d972640 1 -- 172.21.15.16:0/3756651321 learned_addr learned my addr 172.21.15.16:0/3756651321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba7d171640 1 --2- 172.21.15.16:0/3756651321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b79141b0 0x55a8b7ce2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba7d171640 1 -- 172.21.15.16:0/3756651321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b793d070 msgr2=0x55a8b7ce2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba7d171640 1 --2- 172.21.15.16:0/3756651321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b793d070 0x55a8b7ce2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba7d171640 1 -- 172.21.15.16:0/3756651321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b791c390 msgr2=0x55a8b7ce2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba7d171640 1 --2- 172.21.15.16:0/3756651321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b791c390 0x55a8b7ce2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba7d171640 1 -- 172.21.15.16:0/3756651321 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8b79a8650 con 0x55a8b79141b0 2024-08-28T19:21:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba7d171640 1 --2- 172.21.15.16:0/3756651321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b79141b0 0x55a8b7ce2850 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fba7400aba0 tx=0x7fba7400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59a269f00529d1ea server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.569+0000 7fba6e7fc640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba640b4c40 0x7fba640b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.570+0000 7fba637fe640 1 -- 172.21.15.16:0/3756651321 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74005220 con 0x55a8b79141b0 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.570+0000 7fba7d972640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba640b4c40 0x7fba640b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.570+0000 7fba6e7fc640 1 -- 172.21.15.16:0/322096122 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fba680e5890 con 0x7fba781050a0 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.570+0000 7fba637fe640 1 -- 172.21.15.16:0/3756651321 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba74004c50 con 0x55a8b79141b0 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.570+0000 7fba637fe640 1 -- 172.21.15.16:0/3756651321 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74018820 con 0x55a8b79141b0 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba8594a740 1 -- 172.21.15.16:0/3756651321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b79141b0 msgr2=0x55a8b7ce2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba8594a740 1 --2- 172.21.15.16:0/3756651321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b79141b0 0x55a8b7ce2850 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fba7400aba0 tx=0x7fba7400f790 comp rx=0 tx=0).stop 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba7d972640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba640b4c40 0x7fba640b7150 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fba7002f500 tx=0x7fba70009360 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba8594a740 1 -- 172.21.15.16:0/3756651321 shutdown_connections 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba8594a740 1 --2- 172.21.15.16:0/3756651321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b791c390 0x55a8b7ce2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba8594a740 1 --2- 172.21.15.16:0/3756651321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b79141b0 0x55a8b7ce2850 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba8594a740 1 --2- 172.21.15.16:0/3756651321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b793d070 0x55a8b7ce2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba8594a740 1 -- 172.21.15.16:0/3756651321 >> 172.21.15.16:0/3756651321 conn(0x55a8b7931460 msgr2=0x55a8b7ce0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba8594a740 1 -- 172.21.15.16:0/3756651321 shutdown_connections 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba8594a740 1 -- 172.21.15.16:0/3756651321 wait complete. 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba8594a740 1 -- start start 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.571+0000 7fba8594a740 1 Processor -- start 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba8594a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7ceda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba8594a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cedfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba8594a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b793d070 0x55a8b7cf9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba8594a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a8b7aab700 con 0x55a8b79141b0 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba8594a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a8b7bcf750 con 0x55a8b793d070 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba8594a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a8b7cf0af0 con 0x55a8b791c390 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba8594a740 10 client.0 _ll_register_callbacks cb 0x55a8b7bfee10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba7d171640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7ceda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba7d171640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7ceda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48164/0 (socket says 172.21.15.16:48164) 2024-08-28T19:21:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba7d171640 1 -- 172.21.15.16:0/1575736777 learned_addr learned my addr 172.21.15.16:0/1575736777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba7c970640 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cedfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba7d972640 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b793d070 0x55a8b7cf9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba7d171640 1 -- 172.21.15.16:0/1575736777 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 msgr2=0x55a8b7cedfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba7d171640 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cedfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba7d171640 1 -- 172.21.15.16:0/1575736777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b793d070 msgr2=0x55a8b7cf9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba7d171640 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b793d070 0x55a8b7cf9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba7d171640 1 -- 172.21.15.16:0/1575736777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8b7c08900 con 0x55a8b79141b0 2024-08-28T19:21:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba7d171640 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7ceda80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fba740042f0 tx=0x7fba74004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c672d0a01e18edd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.572+0000 7fba60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba60ff9640 1 -- 172.21.15.16:0/1575736777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74027020 con 0x55a8b79141b0 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba60ff9640 1 -- 172.21.15.16:0/1575736777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fba74005010 con 0x55a8b79141b0 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba8594a740 1 -- 172.21.15.16:0/1575736777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba74004c50 con 0x55a8b79141b0 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba60ff9640 1 -- 172.21.15.16:0/1575736777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fba74032050 con 0x55a8b79141b0 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba8594a740 10 client.15840 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba8594a740 1 -- 172.21.15.16:0/1575736777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8b7cfc8c0 con 0x55a8b79141b0 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba60ff9640 1 -- 172.21.15.16:0/1575736777 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba74004c50 con 0x55a8b79141b0 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba60ff9640 1 -- 172.21.15.16:0/1575736777 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fba74003aa0 con 0x55a8b79141b0 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba8594a740 10 client.15840 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba8594a740 10 client.15840 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba8594a740 20 client.15840 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba8594a740 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a8b78fb9f0 0x55a8b7cffd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba8594a740 4 client.15840 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.573+0000 7fba8594a740 1 -- 172.21.15.16:0/1575736777 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8b7c5b5d0 con 0x55a8b78fb9f0 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.574+0000 7fba7c970640 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a8b78fb9f0 0x55a8b7cffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.574+0000 7fba7c970640 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a8b78fb9f0 0x55a8b7cffd30 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aa887f54249f8665 server_cookie=f819fca583cf88f7 in_seq=0 out_seq=0 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.574+0000 7fba60ff9640 10 client.15840 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba60ff9640 1 -- 172.21.15.16:0/1575736777 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a8b7c5b5d0 con 0x55a8b78fb9f0 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba60ff9640 10 client.15840 handle_command_reply: tid=0 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 1 client.15840 shutdown 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 1 -- 172.21.15.16:0/1575736777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 msgr2=0x55a8b7ceda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7ceda80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fba740042f0 tx=0x7fba74004320 comp rx=0 tx=0).stop 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 1 -- 172.21.15.16:0/1575736777 shutdown_connections 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a8b78fb9f0 0x55a8b7cffd30 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b793d070 0x55a8b7cf9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cedfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba60ff9640 0 client.15840 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 1 --2- 172.21.15.16:0/1575736777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7ceda80 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 1 -- 172.21.15.16:0/1575736777 >> 172.21.15.16:0/1575736777 conn(0x55a8b7931460 msgr2=0x55a8b7ce54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 1 -- 172.21.15.16:0/1575736777 shutdown_connections 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 1 -- 172.21.15.16:0/1575736777 wait complete. 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.575+0000 7fba8594a740 20 client.15840 trim_cache size 0 max 16384 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba8594a740 1 Processor -- start 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba8594a740 1 -- start start 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba8594a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b78fb9f0 0x55a8b7ce3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba8594a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7cf9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba8594a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba8594a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba781b0b90 con 0x55a8b79141b0 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba8594a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba781b26d0 con 0x55a8b78fb9f0 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba8594a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fba781b1260 con 0x55a8b791c390 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba7c970640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7cf9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba7c970640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7cf9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48166/0 (socket says 172.21.15.16:48166) 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba7c970640 1 -- 172.21.15.16:0/1745543981 learned_addr learned my addr 172.21.15.16:0/1745543981 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.590+0000 7fba7d171640 1 --2- 172.21.15.16:0/1745543981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b78fb9f0 0x55a8b7ce3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.591+0000 7fba7d972640 1 --2- 172.21.15.16:0/1745543981 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.591+0000 7fba7c970640 1 -- 172.21.15.16:0/1745543981 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 msgr2=0x55a8b7cef6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.591+0000 7fba7c970640 1 --2- 172.21.15.16:0/1745543981 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cef6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.591+0000 7fba7c970640 1 -- 172.21.15.16:0/1745543981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b78fb9f0 msgr2=0x55a8b7ce3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.591+0000 7fba7c970640 1 --2- 172.21.15.16:0/1745543981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b78fb9f0 0x55a8b7ce3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.591+0000 7fba7c970640 1 -- 172.21.15.16:0/1745543981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8b7ceeb20 con 0x55a8b79141b0 2024-08-28T19:21:42.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.591+0000 7fba7d171640 1 --2- 172.21.15.16:0/1745543981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b78fb9f0 0x55a8b7ce3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.591+0000 7fba7c970640 1 --2- 172.21.15.16:0/1745543981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7cf9ce0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fba6800cfd0 tx=0x7fba680e7d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387b6a7afb2a75ca server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.591+0000 7fba62ffd640 1 -- 172.21.15.16:0/1745543981 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba680ac3d0 con 0x55a8b79141b0 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.591+0000 7fba62ffd640 1 -- 172.21.15.16:0/1745543981 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba680abcd0 con 0x55a8b79141b0 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.591+0000 7fba62ffd640 1 -- 172.21.15.16:0/1745543981 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba680a7f60 con 0x55a8b79141b0 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.592+0000 7fba8594a740 1 -- 172.21.15.16:0/1745543981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 msgr2=0x55a8b7cf9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.592+0000 7fba8594a740 1 --2- 172.21.15.16:0/1745543981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7cf9ce0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fba6800cfd0 tx=0x7fba680e7d80 comp rx=0 tx=0).stop 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.592+0000 7fba8594a740 1 -- 172.21.15.16:0/1745543981 shutdown_connections 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.592+0000 7fba8594a740 1 --2- 172.21.15.16:0/1745543981 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cef6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.592+0000 7fba8594a740 1 --2- 172.21.15.16:0/1745543981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b79141b0 0x55a8b7cf9ce0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.592+0000 7fba8594a740 1 --2- 172.21.15.16:0/1745543981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b78fb9f0 0x55a8b7ce3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.592+0000 7fba8594a740 1 -- 172.21.15.16:0/1745543981 >> 172.21.15.16:0/1745543981 conn(0x55a8b7931460 msgr2=0x55a8b7ceefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.592+0000 7fba8594a740 1 -- 172.21.15.16:0/1745543981 shutdown_connections 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.592+0000 7fba8594a740 1 -- 172.21.15.16:0/1745543981 wait complete. 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.593+0000 7fba8594a740 1 -- start start 2024-08-28T19:21:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.593+0000 7fba8594a740 1 Processor -- start 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba8594a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b78fb9f0 0x55a8b7cf3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba8594a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b79141b0 0x55a8b7cf42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba8594a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cf47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba8594a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba781b2000 con 0x55a8b78fb9f0 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba8594a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba781b1930 con 0x55a8b79141b0 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba8594a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fba781aeb70 con 0x55a8b791c390 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba8594a740 10 client.0 _ll_register_callbacks cb 0x7fba680eb400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7d171640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b78fb9f0 0x55a8b7cf3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7d171640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b78fb9f0 0x55a8b7cf3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48180/0 (socket says 172.21.15.16:48180) 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7d171640 1 -- 172.21.15.16:0/4167209821 learned_addr learned my addr 172.21.15.16:0/4167209821 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7d972640 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cf47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7c970640 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b79141b0 0x55a8b7cf42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7d171640 1 -- 172.21.15.16:0/4167209821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 msgr2=0x55a8b7cf47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7d171640 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cf47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7d171640 1 -- 172.21.15.16:0/4167209821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b79141b0 msgr2=0x55a8b7cf42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7d171640 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b79141b0 0x55a8b7cf42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7d171640 1 -- 172.21.15.16:0/4167209821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8b7d69510 con 0x55a8b78fb9f0 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7d972640 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cf47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba7d171640 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b78fb9f0 0x55a8b7cf3d60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fba7400f7d0 tx=0x7fba7400aa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a6df6f3494c301 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.594+0000 7fba60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba60ff9640 1 -- 172.21.15.16:0/4167209821 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7403a020 con 0x55a8b78fb9f0 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba60ff9640 1 -- 172.21.15.16:0/4167209821 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fba74004a80 con 0x55a8b78fb9f0 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba8594a740 1 -- 172.21.15.16:0/4167209821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba680abcd0 con 0x55a8b78fb9f0 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba60ff9640 1 -- 172.21.15.16:0/4167209821 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fba74045050 con 0x55a8b78fb9f0 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba8594a740 10 client.15846 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba8594a740 1 -- 172.21.15.16:0/4167209821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8b7d6a790 con 0x55a8b78fb9f0 2024-08-28T19:21:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba60ff9640 1 -- 172.21.15.16:0/4167209821 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba680abcd0 con 0x55a8b78fb9f0 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba60ff9640 1 -- 172.21.15.16:0/4167209821 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fba740188f0 con 0x55a8b78fb9f0 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba8594a740 10 client.15846 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba8594a740 10 client.15846 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba8594a740 20 client.15846 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba8594a740 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a8b793d070 0x55a8b7d6cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba8594a740 4 client.15846 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.595+0000 7fba8594a740 1 -- 172.21.15.16:0/4167209821 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a8b7c57070 con 0x55a8b793d070 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba7c970640 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a8b793d070 0x55a8b7d6cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba60ff9640 10 client.15846 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba7c970640 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a8b793d070 0x55a8b7d6cf60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=545ece7be71280fc server_cookie=1db9b7b4a0b11977 in_seq=0 out_seq=0 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba60ff9640 1 -- 172.21.15.16:0/4167209821 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a8b7c57070 con 0x55a8b793d070 2024-08-28T19:21:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba60ff9640 10 client.15846 handle_command_reply: tid=0 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba8594a740 1 client.15846 shutdown 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba8594a740 1 -- 172.21.15.16:0/4167209821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b78fb9f0 msgr2=0x55a8b7cf3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba8594a740 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b78fb9f0 0x55a8b7cf3d60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fba7400f7d0 tx=0x7fba7400aa20 comp rx=0 tx=0).stop 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba8594a740 1 -- 172.21.15.16:0/4167209821 shutdown_connections 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba8594a740 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a8b793d070 0x55a8b7d6cf60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba8594a740 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a8b791c390 0x55a8b7cf47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba8594a740 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b79141b0 0x55a8b7cf42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba60ff9640 0 client.15846 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba8594a740 1 --2- 172.21.15.16:0/4167209821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b78fb9f0 0x55a8b7cf3d60 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.596+0000 7fba8594a740 1 -- 172.21.15.16:0/4167209821 >> 172.21.15.16:0/4167209821 conn(0x55a8b7931460 msgr2=0x55a8b7ce3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.597+0000 7fba8594a740 1 -- 172.21.15.16:0/4167209821 shutdown_connections 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.597+0000 7fba8594a740 1 -- 172.21.15.16:0/4167209821 wait complete. 2024-08-28T19:21:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.597+0000 7fba8594a740 20 client.15846 trim_cache size 0 max 16384 2024-08-28T19:21:42.632 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:21:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.597+0000 7fba7f3d5640 1 -- 172.21.15.16:0/322096122 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba640b4c40 msgr2=0x7fba640b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.597+0000 7fba7f3d5640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba640b4c40 0x7fba640b7150 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fba7002f500 tx=0x7fba70009360 comp rx=0 tx=0).stop 2024-08-28T19:21:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.597+0000 7fba7f3d5640 1 -- 172.21.15.16:0/322096122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba781050a0 msgr2=0x7fba7819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.597+0000 7fba7f3d5640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba781050a0 0x7fba7819a760 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fba6800c8c0 tx=0x7fba6800cd80 comp rx=0 tx=0).stop 2024-08-28T19:21:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.598+0000 7fba7f3d5640 1 -- 172.21.15.16:0/322096122 shutdown_connections 2024-08-28T19:21:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.598+0000 7fba7f3d5640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba640b4c40 0x7fba640b7150 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.598+0000 7fba7f3d5640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7810dbb0 0x7fba7819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.598+0000 7fba7f3d5640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba781050a0 0x7fba7819a760 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.598+0000 7fba7f3d5640 1 --2- 172.21.15.16:0/322096122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba78104750 0x7fba7819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:42.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.598+0000 7fba7f3d5640 1 -- 172.21.15.16:0/322096122 >> 172.21.15.16:0/322096122 conn(0x7fba780794d0 msgr2=0x7fba780ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:42.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.598+0000 7fba7f3d5640 1 -- 172.21.15.16:0/322096122 shutdown_connections 2024-08-28T19:21:42.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:42.598+0000 7fba7f3d5640 1 -- 172.21.15.16:0/322096122 wait complete. 2024-08-28T19:21:43.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:42 smithi016 ceph-mon[18556]: pgmap v359: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:21:43.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:42 smithi028 ceph-mon[21769]: pgmap v359: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:21:43.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:42 smithi115 ceph-mon[22184]: pgmap v359: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-08-28T19:21:45.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:44 smithi016 ceph-mon[18556]: pgmap v360: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:21:45.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:44 smithi028 ceph-mon[21769]: pgmap v360: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:21:45.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:44 smithi115 ceph-mon[22184]: pgmap v360: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-08-28T19:21:46.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:21:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:46.967 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:46 smithi016 ceph-mon[18556]: pgmap v361: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:21:47.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:46 smithi028 ceph-mon[21769]: pgmap v361: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:21:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:46 smithi115 ceph-mon[22184]: pgmap v361: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-08-28T19:21:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.203+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/919068883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b0108140 msgr2=0x7fb3b0108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.203+0000 7fb3b5ccf640 1 --2- 172.21.15.16:0/919068883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b0108140 0x7fb3b0108550 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4009900 tx=0x7fb3a402a410 comp rx=0 tx=0).stop 2024-08-28T19:21:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.203+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/919068883 shutdown_connections 2024-08-28T19:21:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.203+0000 7fb3b5ccf640 1 --2- 172.21.15.16:0/919068883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb3b0108a90 0x7fb3b010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.203+0000 7fb3b5ccf640 1 --2- 172.21.15.16:0/919068883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b0108140 0x7fb3b0108550 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.203+0000 7fb3b5ccf640 1 --2- 172.21.15.16:0/919068883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b0102800 0x7fb3b0102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.203+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/919068883 >> 172.21.15.16:0/919068883 conn(0x7fb3b00794d0 msgr2=0x7fb3b00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:47.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.203+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/919068883 shutdown_connections 2024-08-28T19:21:47.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.203+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/919068883 wait complete. 2024-08-28T19:21:47.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.204+0000 7fb3b5ccf640 1 Processor -- start 2024-08-28T19:21:47.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.204+0000 7fb3b5ccf640 1 -- start start 2024-08-28T19:21:47.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.204+0000 7fb3b5ccf640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb3b0102800 0x7fb3b019c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.204+0000 7fb3b5ccf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b0108140 0x7fb3b019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.204+0000 7fb3aeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b0108140 0x7fb3b019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.204+0000 7fb3aeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b0108140 0x7fb3b019c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40946/0 (socket says 172.21.15.16:40946) 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.204+0000 7fb3b5ccf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b0108a90 0x7fb3b01a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3af7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb3b0102800 0x7fb3b019c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3aeffd640 1 -- 172.21.15.16:0/2034598285 learned_addr learned my addr 172.21.15.16:0/2034598285 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b01126c0 con 0x7fb3b0108140 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b0112540 con 0x7fb3b0108a90 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b0112840 con 0x7fb3b0102800 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3affff640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b0108a90 0x7fb3b01a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3aeffd640 1 -- 172.21.15.16:0/2034598285 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb3b0102800 msgr2=0x7fb3b019c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3aeffd640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb3b0102800 0x7fb3b019c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3aeffd640 1 -- 172.21.15.16:0/2034598285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b0108a90 msgr2=0x7fb3b01a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3aeffd640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b0108a90 0x7fb3b01a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3aeffd640 1 -- 172.21.15.16:0/2034598285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b01a2360 con 0x7fb3b0108140 2024-08-28T19:21:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3af7fe640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb3b0102800 0x7fb3b019c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.205+0000 7fb3aeffd640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b0108140 0x7fb3b019c990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4032fd0 tx=0x7fb3a4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.206+0000 7fb3acff9640 1 -- 172.21.15.16:0/2034598285 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a403f030 con 0x7fb3b0108140 2024-08-28T19:21:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.206+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3b01a25f0 con 0x7fb3b0108140 2024-08-28T19:21:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.206+0000 7fb3acff9640 1 -- 172.21.15.16:0/2034598285 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3a4004490 con 0x7fb3b0108140 2024-08-28T19:21:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.206+0000 7fb3acff9640 1 -- 172.21.15.16:0/2034598285 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4035070 con 0x7fb3b0108140 2024-08-28T19:21:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.206+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3b01a8db0 con 0x7fb3b0108140 2024-08-28T19:21:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.207+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3b0103a80 con 0x7fb3b0108140 2024-08-28T19:21:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.209+0000 7fb3acff9640 1 -- 172.21.15.16:0/2034598285 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7fb3a4042030 con 0x7fb3b0108140 2024-08-28T19:21:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.209+0000 7fb3acff9640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3980b4c90 0x7fb3980b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.210+0000 7fb3af7fe640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3980b4c90 0x7fb3980b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.210+0000 7fb3acff9640 1 -- 172.21.15.16:0/2034598285 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb3a41060b0 con 0x7fb3b0108140 2024-08-28T19:21:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.210+0000 7fb3af7fe640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3980b4c90 0x7fb3980b71a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000bfd0 tx=0x7fb3a000b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.214+0000 7fb3acff9640 1 -- 172.21.15.16:0/2034598285 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fb3a40cc540 con 0x7fb3b0108140 2024-08-28T19:21:47.259 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.518+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3b0102c10 con 0x7fb3b0108140 2024-08-28T19:21:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:21:47.521 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"btime":"2024-08-28T19:20:55:257311+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:20:54.880238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:21:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.519+0000 7fb3acff9640 1 -- 172.21.15.16:0/2034598285 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11977 (secure 0 0 0) 0x7fb3a40d0d00 con 0x7fb3b0108140 2024-08-28T19:21:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2024-08-28T19:21:47.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3980b4c90 msgr2=0x7fb3980b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:47.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3980b4c90 0x7fb3980b71a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000bfd0 tx=0x7fb3a000b390 comp rx=0 tx=0).stop 2024-08-28T19:21:47.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b0108140 msgr2=0x7fb3b019c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:47.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b0108140 0x7fb3b019c990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4032fd0 tx=0x7fb3a4004270 comp rx=0 tx=0).stop 2024-08-28T19:21:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 shutdown_connections 2024-08-28T19:21:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb3980b4c90 0x7fb3980b71a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b0108a90 0x7fb3b01a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b0108140 0x7fb3b019c990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 --2- 172.21.15.16:0/2034598285 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb3b0102800 0x7fb3b019c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 >> 172.21.15.16:0/2034598285 conn(0x7fb3b00794d0 msgr2=0x7fb3b01040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 shutdown_connections 2024-08-28T19:21:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:47.523+0000 7fb3b5ccf640 1 -- 172.21.15.16:0/2034598285 wait complete. 2024-08-28T19:21:47.888 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:47 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2034598285' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.003+0000 7f489354a640 1 -- 172.21.15.16:0/4033531432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bb0 msgr2=0x7f488c1103e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.003+0000 7f489354a640 1 --2- 172.21.15.16:0/4033531432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bb0 0x7f488c1103e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4884009ca0 tx=0x7f488402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.003+0000 7f489354a640 1 -- 172.21.15.16:0/4033531432 shutdown_connections 2024-08-28T19:21:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.003+0000 7f489354a640 1 --2- 172.21.15.16:0/4033531432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bb0 0x7f488c1103e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.003+0000 7f489354a640 1 --2- 172.21.15.16:0/4033531432 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c109260 0x7f488c109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.003+0000 7f489354a640 1 --2- 172.21.15.16:0/4033531432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c103920 0x7f488c103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.003+0000 7f489354a640 1 -- 172.21.15.16:0/4033531432 >> 172.21.15.16:0/4033531432 conn(0x7f488c0ff3b0 msgr2=0x7f488c101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.003+0000 7f489354a640 1 -- 172.21.15.16:0/4033531432 shutdown_connections 2024-08-28T19:21:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.003+0000 7f489354a640 1 -- 172.21.15.16:0/4033531432 wait complete. 2024-08-28T19:21:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.004+0000 7f489354a640 1 Processor -- start 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.004+0000 7f489354a640 1 -- start start 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.004+0000 7f489354a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c103920 0x7f488c0754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.004+0000 7f489354a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c109260 0x7f488c0759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.004+0000 7f489354a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bb0 0x7f488c075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.004+0000 7f489354a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f488c113870 con 0x7f488c109bb0 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.004+0000 7f489354a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1136f0 con 0x7f488c103920 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.004+0000 7f489354a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1139f0 con 0x7f488c109260 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f4891ae7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bb0 0x7f488c075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f4891ae7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bb0 0x7f488c075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40970/0 (socket says 172.21.15.16:40970) 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f4891ae7640 1 -- 172.21.15.16:0/1815026277 learned_addr learned my addr 172.21.15.16:0/1815026277 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f4890ae5640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c109260 0x7f488c0759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f48912e6640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c103920 0x7f488c0754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f4891ae7640 1 -- 172.21.15.16:0/1815026277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c109260 msgr2=0x7f488c0759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:48.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f4891ae7640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c109260 0x7f488c0759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f4891ae7640 1 -- 172.21.15.16:0/1815026277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c103920 msgr2=0x7f488c0754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:48.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f4891ae7640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c103920 0x7f488c0754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f4891ae7640 1 -- 172.21.15.16:0/1815026277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c1a8540 con 0x7f488c109bb0 2024-08-28T19:21:48.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f48912e6640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c103920 0x7f488c0754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:48.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f4891ae7640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bb0 0x7f488c075f30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f48840098d0 tx=0x7f48840317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:48.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.005+0000 7f487e7fc640 1 -- 172.21.15.16:0/1815026277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4884003b70 con 0x7f488c109bb0 2024-08-28T19:21:48.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.006+0000 7f487e7fc640 1 -- 172.21.15.16:0/1815026277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4884003d10 con 0x7f488c109bb0 2024-08-28T19:21:48.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.006+0000 7f489354a640 1 -- 172.21.15.16:0/1815026277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488c1a87d0 con 0x7f488c109bb0 2024-08-28T19:21:48.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.006+0000 7f487e7fc640 1 -- 172.21.15.16:0/1815026277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488402fa20 con 0x7f488c109bb0 2024-08-28T19:21:48.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.006+0000 7f489354a640 1 -- 172.21.15.16:0/1815026277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488c1a8d10 con 0x7f488c109bb0 2024-08-28T19:21:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.008+0000 7f4899abf740 1 Processor -- start 2024-08-28T19:21:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f487e7fc640 1 -- 172.21.15.16:0/1815026277 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f488402fc00 con 0x7f488c109bb0 2024-08-28T19:21:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f4899abf740 1 -- start start 2024-08-28T19:21:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f4899abf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf35205070 0x55cf355aa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f4899abf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355aa850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f4899abf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf351e4390 0x55cf355aad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f4899abf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cf35521680 con 0x55cf351dc1b0 2024-08-28T19:21:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f4899abf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf35510c10 con 0x55cf35205070 2024-08-28T19:21:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f4899abf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cf35514bc0 con 0x55cf351e4390 2024-08-28T19:21:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f48912e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355aa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f48912e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355aa850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40972/0 (socket says 172.21.15.16:40972) 2024-08-28T19:21:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f48912e6640 1 -- 172.21.15.16:0/2621139094 learned_addr learned my addr 172.21.15.16:0/2621139094 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.009+0000 7f4890ae5640 1 --2- 172.21.15.16:0/2621139094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf35205070 0x55cf355aa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f487e7fc640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4c10 0x7f48780b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f4890ae5640 1 --2- 172.21.15.16:0/2621139094 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf351e4390 0x55cf355aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f48912e6640 1 -- 172.21.15.16:0/2621139094 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf351e4390 msgr2=0x55cf355aad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f48912e6640 1 --2- 172.21.15.16:0/2621139094 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf351e4390 0x55cf355aad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f48912e6640 1 -- 172.21.15.16:0/2621139094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf35205070 msgr2=0x55cf355aa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f48912e6640 1 --2- 172.21.15.16:0/2621139094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf35205070 0x55cf355aa310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f48912e6640 1 -- 172.21.15.16:0/2621139094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf35270650 con 0x55cf351dc1b0 2024-08-28T19:21:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f4891ae7640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4c10 0x7f48780b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f487e7fc640 1 -- 172.21.15.16:0/1815026277 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f4884106c10 con 0x7f488c109bb0 2024-08-28T19:21:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f48912e6640 1 --2- 172.21.15.16:0/2621139094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355aa850 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f48880114f0 tx=0x7f48880119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54174832b8ae5499 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f48777fe640 1 -- 172.21.15.16:0/2621139094 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488801e070 con 0x55cf351dc1b0 2024-08-28T19:21:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f48777fe640 1 -- 172.21.15.16:0/2621139094 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48880035f0 con 0x55cf351dc1b0 2024-08-28T19:21:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.010+0000 7f48777fe640 1 -- 172.21.15.16:0/2621139094 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488800f780 con 0x55cf351dc1b0 2024-08-28T19:21:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.011+0000 7f4899abf740 1 -- 172.21.15.16:0/2621139094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 msgr2=0x55cf355aa850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.011+0000 7f4899abf740 1 --2- 172.21.15.16:0/2621139094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355aa850 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f48880114f0 tx=0x7f48880119b0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.011+0000 7f4891ae7640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4c10 0x7f48780b7120 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f48840c90d0 tx=0x7f48840c18d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.011+0000 7f4899abf740 1 -- 172.21.15.16:0/2621139094 shutdown_connections 2024-08-28T19:21:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.011+0000 7f4899abf740 1 --2- 172.21.15.16:0/2621139094 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf351e4390 0x55cf355aad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.011+0000 7f4899abf740 1 --2- 172.21.15.16:0/2621139094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355aa850 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.011+0000 7f4899abf740 1 --2- 172.21.15.16:0/2621139094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf35205070 0x55cf355aa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.011+0000 7f4899abf740 1 -- 172.21.15.16:0/2621139094 >> 172.21.15.16:0/2621139094 conn(0x55cf351f9460 msgr2=0x55cf355a8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.011+0000 7f4899abf740 1 -- 172.21.15.16:0/2621139094 shutdown_connections 2024-08-28T19:21:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.011+0000 7f4899abf740 1 -- 172.21.15.16:0/2621139094 wait complete. 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.012+0000 7f4899abf740 1 -- start start 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.012+0000 7f4899abf740 1 Processor -- start 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.012+0000 7f4899abf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351dc1b0 0x55cf355b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.012+0000 7f4899abf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355b6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.012+0000 7f4899abf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf35205070 0x55cf355c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.012+0000 7f4899abf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cf35373700 con 0x55cf351e4390 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f4899abf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf35497750 con 0x55cf351dc1b0 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f4899abf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cf355b8af0 con 0x55cf35205070 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f4899abf740 10 client.0 _ll_register_callbacks cb 0x55cf354c6e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f4890ae5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351dc1b0 0x55cf355b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f48912e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f4890ae5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351dc1b0 0x55cf355b5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49134/0 (socket says 172.21.15.16:49134) 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f48912e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355b6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40980/0 (socket says 172.21.15.16:40980) 2024-08-28T19:21:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f4890ae5640 1 -- 172.21.15.16:0/704823334 learned_addr learned my addr 172.21.15.16:0/704823334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f4890ae5640 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf35205070 0x55cf355c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f48912e6640 1 -- 172.21.15.16:0/704823334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf35205070 msgr2=0x55cf355c1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f48912e6640 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf35205070 0x55cf355c1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f48912e6640 1 -- 172.21.15.16:0/704823334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351dc1b0 msgr2=0x55cf355b5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f48912e6640 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351dc1b0 0x55cf355b5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f48912e6640 1 -- 172.21.15.16:0/704823334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf354d0900 con 0x55cf351e4390 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f4890ae5640 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351dc1b0 0x55cf355b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f48912e6640 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355b6010 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4888011eb0 tx=0x7f4888009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55523ff3c0b91ac server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f4874ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f4874ff9640 1 -- 172.21.15.16:0/704823334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488801e040 con 0x55cf351e4390 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.013+0000 7f4874ff9640 1 -- 172.21.15.16:0/704823334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f4888016050 con 0x55cf351e4390 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4899abf740 1 -- 172.21.15.16:0/704823334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48880035f0 con 0x55cf351e4390 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4874ff9640 1 -- 172.21.15.16:0/704823334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4888031050 con 0x55cf351e4390 2024-08-28T19:21:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4899abf740 10 client.15864 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:48.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4899abf740 1 -- 172.21.15.16:0/704823334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cf355c48a0 con 0x55cf351e4390 2024-08-28T19:21:48.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4874ff9640 1 -- 172.21.15.16:0/704823334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48880035f0 con 0x55cf351e4390 2024-08-28T19:21:48.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4874ff9640 1 -- 172.21.15.16:0/704823334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4888021030 con 0x55cf351e4390 2024-08-28T19:21:48.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4899abf740 10 client.15864 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:48.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4899abf740 10 client.15864 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:21:48.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4899abf740 20 client.15864 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:48.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4899abf740 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55cf351c39f0 0x55cf355c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4899abf740 4 client.15864 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:48.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.014+0000 7f4899abf740 1 -- 172.21.15.16:0/704823334 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf355235d0 con 0x55cf351c39f0 2024-08-28T19:21:48.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.015+0000 7f4890ae5640 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55cf351c39f0 0x55cf355c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.015+0000 7f4874ff9640 10 client.15864 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.015+0000 7f4890ae5640 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55cf351c39f0 0x55cf355c7cb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cd7b0943db51dceb server_cookie=329580eb20303c1d in_seq=0 out_seq=0 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4874ff9640 1 -- 172.21.15.16:0/704823334 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cf355235d0 con 0x55cf351c39f0 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4874ff9640 10 client.15864 handle_command_reply: tid=0 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4899abf740 1 client.15864 shutdown 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4899abf740 1 -- 172.21.15.16:0/704823334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 msgr2=0x55cf355b6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4899abf740 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355b6010 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4888011eb0 tx=0x7f4888009d40 comp rx=0 tx=0).stop 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4899abf740 1 -- 172.21.15.16:0/704823334 shutdown_connections 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4899abf740 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55cf351c39f0 0x55cf355c7cb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4899abf740 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf35205070 0x55cf355c1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4899abf740 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355b6010 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4899abf740 1 --2- 172.21.15.16:0/704823334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351dc1b0 0x55cf355b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4899abf740 1 -- 172.21.15.16:0/704823334 >> 172.21.15.16:0/704823334 conn(0x55cf351f9460 msgr2=0x55cf355ad4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:48.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4874ff9640 0 client.15864 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:21:48.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4899abf740 1 -- 172.21.15.16:0/704823334 shutdown_connections 2024-08-28T19:21:48.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.016+0000 7f4899abf740 1 -- 172.21.15.16:0/704823334 wait complete. 2024-08-28T19:21:48.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.017+0000 7f4899abf740 20 client.15864 trim_cache size 0 max 16384 2024-08-28T19:21:48.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.030+0000 7f4899abf740 1 Processor -- start 2024-08-28T19:21:48.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.030+0000 7f4899abf740 1 -- start start 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.030+0000 7f4899abf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf351dc1b0 0x55cf355b60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.030+0000 7f4899abf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.030+0000 7f4899abf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf35205070 0x55cf355ad950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.030+0000 7f4899abf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1b2e80 con 0x55cf351e4390 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.030+0000 7f4899abf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1b49c0 con 0x55cf35205070 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.030+0000 7f4899abf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1b3550 con 0x55cf351dc1b0 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f48912e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f48912e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355ad410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40982/0 (socket says 172.21.15.16:40982) 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f48912e6640 1 -- 172.21.15.16:0/328469101 learned_addr learned my addr 172.21.15.16:0/328469101 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f4890ae5640 1 --2- 172.21.15.16:0/328469101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf35205070 0x55cf355ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f4890ae5640 1 --2- 172.21.15.16:0/328469101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf351dc1b0 0x55cf355b60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f48912e6640 1 -- 172.21.15.16:0/328469101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf351dc1b0 msgr2=0x55cf355b60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f48912e6640 1 --2- 172.21.15.16:0/328469101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf351dc1b0 0x55cf355b60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f48912e6640 1 -- 172.21.15.16:0/328469101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf35205070 msgr2=0x55cf355ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:48.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f48912e6640 1 --2- 172.21.15.16:0/328469101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf35205070 0x55cf355ad950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f48912e6640 1 -- 172.21.15.16:0/328469101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf355b8210 con 0x55cf351e4390 2024-08-28T19:21:48.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f4890ae5640 1 --2- 172.21.15.16:0/328469101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf351dc1b0 0x55cf355b60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:48.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f48912e6640 1 --2- 172.21.15.16:0/328469101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355ad410 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f4888000c00 tx=0x7f48880119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74975421a3cd3605 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:48.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f4876ffd640 1 -- 172.21.15.16:0/328469101 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488801e030 con 0x55cf351e4390 2024-08-28T19:21:48.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f4876ffd640 1 -- 172.21.15.16:0/328469101 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48880035f0 con 0x55cf351e4390 2024-08-28T19:21:48.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.031+0000 7f4876ffd640 1 -- 172.21.15.16:0/328469101 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888003f70 con 0x55cf351e4390 2024-08-28T19:21:48.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.032+0000 7f4899abf740 1 -- 172.21.15.16:0/328469101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 msgr2=0x55cf355ad410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:48.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.032+0000 7f4899abf740 1 --2- 172.21.15.16:0/328469101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355ad410 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f4888000c00 tx=0x7f48880119b0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.032+0000 7f4899abf740 1 -- 172.21.15.16:0/328469101 shutdown_connections 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.032+0000 7f4899abf740 1 --2- 172.21.15.16:0/328469101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf35205070 0x55cf355ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.032+0000 7f4899abf740 1 --2- 172.21.15.16:0/328469101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351e4390 0x55cf355ad410 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.032+0000 7f4899abf740 1 --2- 172.21.15.16:0/328469101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf351dc1b0 0x55cf355b60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.032+0000 7f4899abf740 1 -- 172.21.15.16:0/328469101 >> 172.21.15.16:0/328469101 conn(0x55cf351f9460 msgr2=0x55cf355b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.033+0000 7f4899abf740 1 -- 172.21.15.16:0/328469101 shutdown_connections 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.033+0000 7f4899abf740 1 -- 172.21.15.16:0/328469101 wait complete. 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.033+0000 7f4899abf740 1 -- start start 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.033+0000 7f4899abf740 1 Processor -- start 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4899abf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355be3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4899abf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351e4390 0x55cf355b98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4899abf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf35205070 0x55cf355b9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4899abf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1b42f0 con 0x55cf351dc1b0 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4899abf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1b3c20 con 0x55cf351e4390 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4899abf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1b0e60 con 0x55cf35205070 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4899abf740 10 client.0 _ll_register_callbacks cb 0x7f488801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4890ae5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355be3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4890ae5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355be3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40986/0 (socket says 172.21.15.16:40986) 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4890ae5640 1 -- 172.21.15.16:0/673758823 learned_addr learned my addr 172.21.15.16:0/673758823 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f48912e6640 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351e4390 0x55cf355b98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4890ae5640 1 -- 172.21.15.16:0/673758823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf35205070 msgr2=0x55cf355b9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4890ae5640 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf35205070 0x55cf355b9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4890ae5640 1 -- 172.21.15.16:0/673758823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351e4390 msgr2=0x55cf355b98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4890ae5640 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351e4390 0x55cf355b98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f4890ae5640 1 -- 172.21.15.16:0/673758823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf355ba400 con 0x55cf351dc1b0 2024-08-28T19:21:48.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.034+0000 7f48912e6640 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351e4390 0x55cf355b98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4890ae5640 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355be3b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4880014010 tx=0x7f4880022800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2f955c5f31dd94 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4874ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4874ff9640 1 -- 172.21.15.16:0/673758823 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4880004020 con 0x55cf351dc1b0 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4874ff9640 1 -- 172.21.15.16:0/673758823 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f488001b040 con 0x55cf351dc1b0 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4899abf740 1 -- 172.21.15.16:0/673758823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48880035f0 con 0x55cf351dc1b0 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4874ff9640 1 -- 172.21.15.16:0/673758823 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4880038050 con 0x55cf351dc1b0 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4899abf740 10 client.15870 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4899abf740 1 -- 172.21.15.16:0/673758823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cf35632310 con 0x55cf351dc1b0 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4874ff9640 1 -- 172.21.15.16:0/673758823 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48880035f0 con 0x55cf351dc1b0 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4874ff9640 1 -- 172.21.15.16:0/673758823 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f488002c440 con 0x55cf351dc1b0 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4899abf740 10 client.15870 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4899abf740 10 client.15870 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:21:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4899abf740 20 client.15870 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.035+0000 7f4899abf740 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55cf35198da0 0x55cf35634b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.036+0000 7f4899abf740 4 client.15870 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.036+0000 7f4899abf740 1 -- 172.21.15.16:0/673758823 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf3551f070 con 0x55cf35198da0 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.036+0000 7f48912e6640 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55cf35198da0 0x55cf35634b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.036+0000 7f4874ff9640 10 client.15870 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.036+0000 7f48912e6640 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55cf35198da0 0x55cf35634b00 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=363fbce64c174913 server_cookie=22458ec9002cc78e in_seq=0 out_seq=0 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.036+0000 7f4874ff9640 1 -- 172.21.15.16:0/673758823 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf3551f070 con 0x55cf35198da0 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.036+0000 7f4874ff9640 10 client.15870 handle_command_reply: tid=0 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.036+0000 7f4899abf740 1 client.15870 shutdown 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4899abf740 1 -- 172.21.15.16:0/673758823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 msgr2=0x55cf355be3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4899abf740 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355be3b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4880014010 tx=0x7f4880022800 comp rx=0 tx=0).stop 2024-08-28T19:21:48.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4899abf740 1 -- 172.21.15.16:0/673758823 shutdown_connections 2024-08-28T19:21:48.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4899abf740 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55cf35198da0 0x55cf35634b00 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4899abf740 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf35205070 0x55cf355b9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4874ff9640 0 client.15870 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:21:48.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4899abf740 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf351e4390 0x55cf355b98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4899abf740 1 --2- 172.21.15.16:0/673758823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf351dc1b0 0x55cf355be3b0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4899abf740 1 -- 172.21.15.16:0/673758823 >> 172.21.15.16:0/673758823 conn(0x55cf351f9460 msgr2=0x55cf355ad410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:48.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4899abf740 1 -- 172.21.15.16:0/673758823 shutdown_connections 2024-08-28T19:21:48.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4899abf740 1 -- 172.21.15.16:0/673758823 wait complete. 2024-08-28T19:21:48.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.037+0000 7f4899abf740 20 client.15870 trim_cache size 0 max 16384 2024-08-28T19:21:48.044 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 -- 172.21.15.16:0/1815026277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4c10 msgr2=0x7f48780b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4c10 0x7f48780b7120 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f48840c90d0 tx=0x7f48840c18d0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 -- 172.21.15.16:0/1815026277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bb0 msgr2=0x7f488c075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bb0 0x7f488c075f30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f48840098d0 tx=0x7f48840317b0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 -- 172.21.15.16:0/1815026277 shutdown_connections 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bb0 0x7f488c075f30 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c109260 0x7f488c0759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4c10 0x7f48780b7120 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 --2- 172.21.15.16:0/1815026277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c103920 0x7f488c0754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 -- 172.21.15.16:0/1815026277 >> 172.21.15.16:0/1815026277 conn(0x7f488c0ff3b0 msgr2=0x7f488c106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 -- 172.21.15.16:0/1815026277 shutdown_connections 2024-08-28T19:21:48.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:48.038+0000 7f489354a640 1 -- 172.21.15.16:0/1815026277 wait complete. 2024-08-28T19:21:48.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:47 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2034598285' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:47 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2034598285' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:21:48.590 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 154.4 --await-for 45.0' 2024-08-28T19:21:48.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 154.4 --await-for 45.0 2024-08-28T19:21:48.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:48 smithi016 ceph-mon[18556]: pgmap v362: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:21:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:48 smithi028 ceph-mon[21769]: pgmap v362: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:21:49.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:48 smithi115 ceph-mon[22184]: pgmap v362: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-08-28T19:21:49.249 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:21:49.383 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:50.034 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:50.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.033+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1377022657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c0101de0 msgr2=0x7fd8c01021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.033+0000 7fd8c5db5640 1 --2- 172.21.15.16:0/1377022657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c0101de0 0x7fd8c01021f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009900 tx=0x7fd8b0032c20 comp rx=0 tx=0).stop 2024-08-28T19:21:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.033+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1377022657 shutdown_connections 2024-08-28T19:21:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.033+0000 7fd8c5db5640 1 --2- 172.21.15.16:0/1377022657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0106980 0x7fd8c0110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.033+0000 7fd8c5db5640 1 --2- 172.21.15.16:0/1377022657 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c0106030 0x7fd8c0106440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.033+0000 7fd8c5db5640 1 --2- 172.21.15.16:0/1377022657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c0101de0 0x7fd8c01021f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.033+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1377022657 >> 172.21.15.16:0/1377022657 conn(0x7fd8c0079cf0 msgr2=0x7fd8c007a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.034+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1377022657 shutdown_connections 2024-08-28T19:21:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.034+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1377022657 wait complete. 2024-08-28T19:21:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.035+0000 7fd8c5db5640 1 Processor -- start 2024-08-28T19:21:50.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.035+0000 7fd8c5db5640 1 -- start start 2024-08-28T19:21:50.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.035+0000 7fd8c5db5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c0101de0 0x7fd8c01a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.035+0000 7fd8c5db5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0106030 0x7fd8c01a9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.035+0000 7fd8c5db5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c0106980 0x7fd8c01ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.035+0000 7fd8c5db5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0122690 con 0x7fd8c0101de0 2024-08-28T19:21:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.035+0000 7fd8c5db5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0122510 con 0x7fd8c0106030 2024-08-28T19:21:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.035+0000 7fd8c5db5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0122810 con 0x7fd8c0106980 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8bf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c0101de0 0x7fd8c01a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0106030 0x7fd8c01a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8bf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c0101de0 0x7fd8c01a90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41010/0 (socket says 172.21.15.16:41010) 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0106030 0x7fd8c01a9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49160/0 (socket says 172.21.15.16:49160) 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8bf7fe640 1 -- 172.21.15.16:0/1314040591 learned_addr learned my addr 172.21.15.16:0/1314040591 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8bffff640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c0106980 0x7fd8c01ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8beffd640 1 -- 172.21.15.16:0/1314040591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c0106980 msgr2=0x7fd8c01ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8beffd640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c0106980 0x7fd8c01ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8beffd640 1 -- 172.21.15.16:0/1314040591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c0101de0 msgr2=0x7fd8c01a90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8beffd640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c0101de0 0x7fd8c01a90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8beffd640 1 -- 172.21.15.16:0/1314040591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c01af030 con 0x7fd8c0106030 2024-08-28T19:21:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.036+0000 7fd8beffd640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0106030 0x7fd8c01a9660 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0075550 tx=0x7fd8b400dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.037+0000 7fd8bcff9640 1 -- 172.21.15.16:0/1314040591 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4017070 con 0x7fd8c0106030 2024-08-28T19:21:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.037+0000 7fd8bcff9640 1 -- 172.21.15.16:0/1314040591 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b40099a0 con 0x7fd8c0106030 2024-08-28T19:21:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.037+0000 7fd8bcff9640 1 -- 172.21.15.16:0/1314040591 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4005630 con 0x7fd8c0106030 2024-08-28T19:21:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.037+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1314040591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c01af320 con 0x7fd8c0106030 2024-08-28T19:21:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.037+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1314040591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c01b5a80 con 0x7fd8c0106030 2024-08-28T19:21:50.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.040+0000 7fd8bcff9640 1 -- 172.21.15.16:0/1314040591 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7fd8b4009b50 con 0x7fd8c0106030 2024-08-28T19:21:50.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.040+0000 7fd8cc32a740 1 Processor -- start 2024-08-28T19:21:50.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.040+0000 7fd8cc32a740 1 -- start start 2024-08-28T19:21:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.041+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8e00d840 0x556d8e3b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.041+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.041+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8dfecb60 0x556d8e3b5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.041+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556d8e329330 con 0x556d8dfecb60 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.041+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d8e3188c0 con 0x556d8e00d840 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.041+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556d8e31c870 con 0x556d8dfe4980 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.041+0000 7fd8beffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8dfecb60 0x556d8e3b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.041+0000 7fd8beffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8dfecb60 0x556d8e3b5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41022/0 (socket says 172.21.15.16:41022) 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.041+0000 7fd8beffd640 1 -- 172.21.15.16:0/3574954976 learned_addr learned my addr 172.21.15.16:0/3574954976 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.041+0000 7fd8bcff9640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8a80b4c10 0x7fd8a80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.041+0000 7fd8bffff640 1 --2- 172.21.15.16:0/3574954976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8e00d840 0x556d8e3b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.042+0000 7fd8beffd640 1 -- 172.21.15.16:0/3574954976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 msgr2=0x556d8e3b51d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.042+0000 7fd8bf7fe640 1 --2- 172.21.15.16:0/3574954976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.042+0000 7fd8beffd640 1 --2- 172.21.15.16:0/3574954976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3b51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.042+0000 7fd8beffd640 1 -- 172.21.15.16:0/3574954976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8e00d840 msgr2=0x556d8e3b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.042+0000 7fd8beffd640 1 --2- 172.21.15.16:0/3574954976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8e00d840 0x556d8e3b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.042+0000 7fd8bcff9640 1 -- 172.21.15.16:0/1314040591 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fd8b40e5380 con 0x7fd8c0106030 2024-08-28T19:21:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.042+0000 7fd8beffd640 1 -- 172.21.15.16:0/3574954976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d8e2d0df0 con 0x556d8dfecb60 2024-08-28T19:21:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.042+0000 7fd8bffff640 1 --2- 172.21.15.16:0/3574954976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8e00d840 0x556d8e3b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.042+0000 7fd8bffff640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8a80b4c10 0x7fd8a80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.042+0000 7fd8beffd640 1 --2- 172.21.15.16:0/3574954976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8dfecb60 0x556d8e3b5710 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40a8a30 tx=0x7fd8b4009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b536183f100e0647 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.043+0000 7fd8a5ffb640 1 -- 172.21.15.16:0/3574954976 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40aaf30 con 0x556d8dfecb60 2024-08-28T19:21:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.043+0000 7fd8a5ffb640 1 -- 172.21.15.16:0/3574954976 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b40ab2d0 con 0x556d8dfecb60 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.043+0000 7fd8a5ffb640 1 -- 172.21.15.16:0/3574954976 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40b4ae0 con 0x556d8dfecb60 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.044+0000 7fd8bffff640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8a80b4c10 0x7fd8a80b7120 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009fd0 tx=0x7fd8b8004a40 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.044+0000 7fd8cc32a740 1 -- 172.21.15.16:0/3574954976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8dfecb60 msgr2=0x556d8e3b5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.044+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/3574954976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8dfecb60 0x556d8e3b5710 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40a8a30 tx=0x7fd8b4009f30 comp rx=0 tx=0).stop 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.044+0000 7fd8cc32a740 1 -- 172.21.15.16:0/3574954976 shutdown_connections 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.044+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/3574954976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8dfecb60 0x556d8e3b5710 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.044+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/3574954976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.044+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/3574954976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8e00d840 0x556d8e3b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.044+0000 7fd8cc32a740 1 -- 172.21.15.16:0/3574954976 >> 172.21.15.16:0/3574954976 conn(0x556d8e001990 msgr2=0x556d8e3b2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.044+0000 7fd8cc32a740 1 -- 172.21.15.16:0/3574954976 shutdown_connections 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.044+0000 7fd8cc32a740 1 -- 172.21.15.16:0/3574954976 wait complete. 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.045+0000 7fd8cc32a740 1 -- start start 2024-08-28T19:21:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.045+0000 7fd8cc32a740 1 Processor -- start 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3c0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3cc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556d8e17b7a0 con 0x556d8e00d840 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d8e29f240 con 0x556d8dfecb60 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556d8e3c3410 con 0x556d8dfe4980 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3cc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41028/0 (socket says 172.21.15.16:41028) 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8bffff640 1 -- 172.21.15.16:0/443243442 learned_addr learned my addr 172.21.15.16:0/443243442 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8bf7fe640 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3c0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8cc32a740 10 client.0 _ll_register_callbacks cb 0x556d8e025090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8beffd640 1 -- 172.21.15.16:0/443243442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 msgr2=0x556d8e3c0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8beffd640 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3c0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8beffd640 1 -- 172.21.15.16:0/443243442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 msgr2=0x556d8e3cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8beffd640 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8beffd640 1 -- 172.21.15.16:0/443243442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d8e2d52b0 con 0x556d8dfecb60 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8bffff640 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.046+0000 7fd8cc32a740 1 -- 172.21.15.16:0/443243442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b40ab2d0 con 0x556d8dfecb60 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.047+0000 7fd8beffd640 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3c08d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40a7840 tx=0x7fd8b40b44c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=302b8bea352fe459 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.047+0000 7fd89effd640 10 client.25511 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.047+0000 7fd89effd640 1 -- 172.21.15.16:0/443243442 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40aac20 con 0x556d8dfecb60 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.047+0000 7fd89effd640 1 -- 172.21.15.16:0/443243442 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fd8b40acaf0 con 0x556d8dfecb60 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.047+0000 7fd89effd640 1 -- 172.21.15.16:0/443243442 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd8b40acd50 con 0x556d8dfecb60 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.047+0000 7fd8cc32a740 10 client.25511 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.047+0000 7fd8cc32a740 1 -- 172.21.15.16:0/443243442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=29}) -- 0x556d8e087b90 con 0x556d8dfecb60 2024-08-28T19:21:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.048+0000 7fd89effd640 1 -- 172.21.15.16:0/443243442 <== mon.2 v2:172.21.15.28:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd8b40a7130 con 0x556d8dfecb60 2024-08-28T19:21:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.048+0000 7fd8cc32a740 10 client.25511 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.048+0000 7fd8cc32a740 10 client.25511 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:21:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.048+0000 7fd8cc32a740 20 client.25511 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.048+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d8dfcc1c0 0x556d8e3d23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.048+0000 7fd8cc32a740 4 client.25511 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.048+0000 7fd8cc32a740 1 -- 172.21.15.16:0/443243442 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d8e32b280 con 0x556d8dfcc1c0 2024-08-28T19:21:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.048+0000 7fd8bf7fe640 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d8dfcc1c0 0x556d8e3d23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.048+0000 7fd8bf7fe640 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d8dfcc1c0 0x556d8e3d23e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc1aa8fd84326682 server_cookie=213fd827c8115752 in_seq=0 out_seq=0 2024-08-28T19:21:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.048+0000 7fd89effd640 10 client.25511 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.049+0000 7fd89effd640 1 -- 172.21.15.16:0/443243442 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556d8e32b280 con 0x556d8dfcc1c0 2024-08-28T19:21:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.049+0000 7fd89effd640 10 client.25511 handle_command_reply: tid=0 2024-08-28T19:21:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.049+0000 7fd8cc32a740 1 client.25511 shutdown 2024-08-28T19:21:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd8cc32a740 1 -- 172.21.15.16:0/443243442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 msgr2=0x556d8e3c08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3c08d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40a7840 tx=0x7fd8b40b44c0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd8cc32a740 1 -- 172.21.15.16:0/443243442 shutdown_connections 2024-08-28T19:21:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3c08d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d8dfcc1c0 0x556d8e3d23e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/443243442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3c0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd89effd640 0 client.25511 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd8cc32a740 1 -- 172.21.15.16:0/443243442 >> 172.21.15.16:0/443243442 conn(0x556d8e001990 msgr2=0x556d8e3b5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd8cc32a740 1 -- 172.21.15.16:0/443243442 shutdown_connections 2024-08-28T19:21:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd8cc32a740 1 -- 172.21.15.16:0/443243442 wait complete. 2024-08-28T19:21:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.050+0000 7fd8cc32a740 20 client.25511 trim_cache size 0 max 16384 2024-08-28T19:21:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.070+0000 7fd8cc32a740 1 Processor -- start 2024-08-28T19:21:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.070+0000 7fd8cc32a740 1 -- start start 2024-08-28T19:21:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.070+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.070+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.070+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3bd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.070+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01bfaf0 con 0x556d8e00d840 2024-08-28T19:21:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.070+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01c1630 con 0x556d8dfecb60 2024-08-28T19:21:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.070+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01c01c0 con 0x556d8dfe4980 2024-08-28T19:21:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.070+0000 7fd8bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3bd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.070+0000 7fd8bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3bd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41036/0 (socket says 172.21.15.16:41036) 2024-08-28T19:21:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.070+0000 7fd8bffff640 1 -- 172.21.15.16:0/3355730580 learned_addr learned my addr 172.21.15.16:0/3355730580 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.071+0000 7fd8bf7fe640 1 --2- 172.21.15.16:0/3355730580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.071+0000 7fd8beffd640 1 --2- 172.21.15.16:0/3355730580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.071+0000 7fd8bffff640 1 -- 172.21.15.16:0/3355730580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 msgr2=0x556d8e3b1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.071+0000 7fd8bffff640 1 --2- 172.21.15.16:0/3355730580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3b1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.071+0000 7fd8bffff640 1 -- 172.21.15.16:0/3355730580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 msgr2=0x556d8e3cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.071+0000 7fd8bffff640 1 --2- 172.21.15.16:0/3355730580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.071+0000 7fd8bffff640 1 -- 172.21.15.16:0/3355730580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d8e3bf5e0 con 0x556d8e00d840 2024-08-28T19:21:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.071+0000 7fd8beffd640 1 --2- 172.21.15.16:0/3355730580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.071+0000 7fd8bffff640 1 --2- 172.21.15.16:0/3355730580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3bd580 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd8b801a710 tx=0x7fd8b8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e847e269eda7710a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.072+0000 7fd8a57fa640 1 -- 172.21.15.16:0/3355730580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b80105d0 con 0x556d8e00d840 2024-08-28T19:21:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.072+0000 7fd8a57fa640 1 -- 172.21.15.16:0/3355730580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b802b070 con 0x556d8e00d840 2024-08-28T19:21:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.072+0000 7fd8a57fa640 1 -- 172.21.15.16:0/3355730580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8026aa0 con 0x556d8e00d840 2024-08-28T19:21:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.073+0000 7fd8cc32a740 1 -- 172.21.15.16:0/3355730580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 msgr2=0x556d8e3bd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.073+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/3355730580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3bd580 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd8b801a710 tx=0x7fd8b8018000 comp rx=0 tx=0).stop 2024-08-28T19:21:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.073+0000 7fd8cc32a740 1 -- 172.21.15.16:0/3355730580 shutdown_connections 2024-08-28T19:21:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.073+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/3355730580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3bd580 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.073+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/3355730580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.073+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/3355730580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.073+0000 7fd8cc32a740 1 -- 172.21.15.16:0/3355730580 >> 172.21.15.16:0/3355730580 conn(0x556d8e001990 msgr2=0x556d8e3b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:50.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.073+0000 7fd8cc32a740 1 -- 172.21.15.16:0/3355730580 shutdown_connections 2024-08-28T19:21:50.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.073+0000 7fd8cc32a740 1 -- 172.21.15.16:0/3355730580 wait complete. 2024-08-28T19:21:50.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.074+0000 7fd8cc32a740 1 -- start start 2024-08-28T19:21:50.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.074+0000 7fd8cc32a740 1 Processor -- start 2024-08-28T19:21:50.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.074+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3d16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.074+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8cc32a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01c0f60 con 0x556d8e00d840 2024-08-28T19:21:50.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01c0890 con 0x556d8dfecb60 2024-08-28T19:21:50.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8cc32a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01bdad0 con 0x556d8dfe4980 2024-08-28T19:21:50.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8cc32a740 10 client.0 _ll_register_callbacks cb 0x7fd8b40a7000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3d1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49190/0 (socket says 172.21.15.16:49190) 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8beffd640 1 -- 172.21.15.16:0/167942761 learned_addr learned my addr 172.21.15.16:0/167942761 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8bffff640 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8bf7fe640 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3d16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8bffff640 1 -- 172.21.15.16:0/167942761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 msgr2=0x556d8e3d16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8bffff640 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3d16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8bffff640 1 -- 172.21.15.16:0/167942761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 msgr2=0x556d8e3d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8bffff640 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3d1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8bffff640 1 -- 172.21.15.16:0/167942761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d8e2e8d90 con 0x556d8e00d840 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd8bffff640 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3d2120 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd8b802a000 tx=0x7fd8b8026ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22750407d6e3a74f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.075+0000 7fd89effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.076+0000 7fd89effd640 1 -- 172.21.15.16:0/167942761 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8033020 con 0x556d8e00d840 2024-08-28T19:21:50.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.076+0000 7fd89effd640 1 -- 172.21.15.16:0/167942761 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fd8b80105d0 con 0x556d8e00d840 2024-08-28T19:21:50.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.076+0000 7fd8cc32a740 1 -- 172.21.15.16:0/167942761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b802b070 con 0x556d8e00d840 2024-08-28T19:21:50.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.076+0000 7fd89effd640 1 -- 172.21.15.16:0/167942761 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd8b80263f0 con 0x556d8e00d840 2024-08-28T19:21:50.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.076+0000 7fd8cc32a740 10 client.15888 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:50.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.076+0000 7fd8cc32a740 1 -- 172.21.15.16:0/167942761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556d8e43cea0 con 0x556d8e00d840 2024-08-28T19:21:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.076+0000 7fd89effd640 1 -- 172.21.15.16:0/167942761 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b802b070 con 0x556d8e00d840 2024-08-28T19:21:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.076+0000 7fd89effd640 1 -- 172.21.15.16:0/167942761 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd8b8005170 con 0x556d8e00d840 2024-08-28T19:21:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.077+0000 7fd8cc32a740 10 client.15888 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.077+0000 7fd8cc32a740 10 client.15888 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:21:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.077+0000 7fd8cc32a740 20 client.15888 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.077+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d8dfa1570 0x556d8e43f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.077+0000 7fd8cc32a740 4 client.15888 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 154.4, "await_for": 45.0}] 2024-08-28T19:21:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.077+0000 7fd8cc32a740 1 -- 172.21.15.16:0/167942761 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 154.4, "await_for": 45.0}) -- 0x556d8e326d20 con 0x556d8dfa1570 2024-08-28T19:21:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.077+0000 7fd8bf7fe640 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d8dfa1570 0x556d8e43f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.077+0000 7fd89effd640 10 client.15888 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.077+0000 7fd8bf7fe640 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d8dfa1570 0x556d8e43f630 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cba5a8d9234d3b1b server_cookie=226c889efd498192 in_seq=0 out_seq=0 2024-08-28T19:21:50.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd89effd640 1 -- 172.21.15.16:0/167942761 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x556d8e326d20 con 0x556d8dfa1570 2024-08-28T19:21:50.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd89effd640 10 client.15888 handle_command_reply: tid=0 2024-08-28T19:21:50.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 1 client.15888 shutdown 2024-08-28T19:21:50.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 1 -- 172.21.15.16:0/167942761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 msgr2=0x556d8e3d2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:50.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3d2120 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd8b802a000 tx=0x7fd8b8026ea0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 1 -- 172.21.15.16:0/167942761 shutdown_connections 2024-08-28T19:21:50.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d8e00d840 0x556d8e3d2120 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d8dfecb60 0x556d8e3d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d8dfa1570 0x556d8e43f630 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd89effd640 0 client.15888 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:50.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 1 --2- 172.21.15.16:0/167942761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d8dfe4980 0x556d8e3d16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 1 -- 172.21.15.16:0/167942761 >> 172.21.15.16:0/167942761 conn(0x556d8e001990 msgr2=0x556d8e3b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:50.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 1 -- 172.21.15.16:0/167942761 shutdown_connections 2024-08-28T19:21:50.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 1 -- 172.21.15.16:0/167942761 wait complete. 2024-08-28T19:21:50.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.087+0000 7fd8cc32a740 20 client.15888 trim_cache size 0 max 16384 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 29, 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 2, 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "3e074952": { 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 2, 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:21:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 154.4, 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:50.091 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:21:50.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1314040591 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8a80b4c10 msgr2=0x7fd8a80b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:50.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8a80b4c10 0x7fd8a80b7120 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009fd0 tx=0x7fd8b8004a40 comp rx=0 tx=0).stop 2024-08-28T19:21:50.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1314040591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0106030 msgr2=0x7fd8c01a9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:50.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0106030 0x7fd8c01a9660 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0075550 tx=0x7fd8b400dbb0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1314040591 shutdown_connections 2024-08-28T19:21:50.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8c0106980 0x7fd8c01ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd8a80b4c10 0x7fd8a80b7120 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0106030 0x7fd8c01a9660 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 --2- 172.21.15.16:0/1314040591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c0101de0 0x7fd8c01a90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:50.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1314040591 >> 172.21.15.16:0/1314040591 conn(0x7fd8c0079cf0 msgr2=0x7fd8c01072c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:50.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1314040591 shutdown_connections 2024-08-28T19:21:50.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:50.088+0000 7fd8c5db5640 1 -- 172.21.15.16:0/1314040591 wait complete. 2024-08-28T19:21:50.877 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '3e074952', quiesce took 8.9 seconds. Will release after: 25.5 2024-08-28T19:21:51.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:50 smithi016 ceph-mon[18556]: pgmap v363: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-08-28T19:21:51.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:50 smithi028 ceph-mon[21769]: pgmap v363: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-08-28T19:21:51.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:50 smithi115 ceph-mon[22184]: pgmap v363: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-08-28T19:21:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.250+0000 7fe6cc06c640 1 -- 172.21.15.16:0/1382946605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c4105060 msgr2=0x7fe6c410d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.250+0000 7fe6cc06c640 1 --2- 172.21.15.16:0/1382946605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c4105060 0x7fe6c410d6a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4009900 tx=0x7fe6b402a410 comp rx=0 tx=0).stop 2024-08-28T19:21:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.250+0000 7fe6cc06c640 1 -- 172.21.15.16:0/1382946605 shutdown_connections 2024-08-28T19:21:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.250+0000 7fe6cc06c640 1 --2- 172.21.15.16:0/1382946605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c410dbe0 0x7fe6c41100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.250+0000 7fe6cc06c640 1 --2- 172.21.15.16:0/1382946605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c4105060 0x7fe6c410d6a0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.250+0000 7fe6cc06c640 1 --2- 172.21.15.16:0/1382946605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6c4104630 0x7fe6c4104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.250+0000 7fe6cc06c640 1 -- 172.21.15.16:0/1382946605 >> 172.21.15.16:0/1382946605 conn(0x7fe6c4079410 msgr2=0x7fe6c4079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:51.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.250+0000 7fe6cc06c640 1 -- 172.21.15.16:0/1382946605 shutdown_connections 2024-08-28T19:21:51.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.250+0000 7fe6cc06c640 1 -- 172.21.15.16:0/1382946605 wait complete. 2024-08-28T19:21:51.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.251+0000 7fe6cc06c640 1 Processor -- start 2024-08-28T19:21:51.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.251+0000 7fe6cc06c640 1 -- start start 2024-08-28T19:21:51.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.251+0000 7fe6cc06c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c4104630 0x7fe6c4102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.251+0000 7fe6cc06c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6c4105060 0x7fe6c4102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.251+0000 7fe6cc06c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c410dbe0 0x7fe6c4102fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6ca609640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c410dbe0 0x7fe6c4102fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6cc06c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c4113860 con 0x7fe6c410dbe0 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6cc06c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c41136e0 con 0x7fe6c4104630 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6cc06c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c41139e0 con 0x7fe6c4105060 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6ca609640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c410dbe0 0x7fe6c4102fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41062/0 (socket says 172.21.15.16:41062) 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6c9e08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c4104630 0x7fe6c4102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6ca609640 1 -- 172.21.15.16:0/2510094713 learned_addr learned my addr 172.21.15.16:0/2510094713 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6c9607640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6c4105060 0x7fe6c4102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6c9e08640 1 -- 172.21.15.16:0/2510094713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6c4105060 msgr2=0x7fe6c4102a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6c4105060 0x7fe6c4102a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6c9e08640 1 -- 172.21.15.16:0/2510094713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c410dbe0 msgr2=0x7fe6c4102fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c410dbe0 0x7fe6c4102fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6c9e08640 1 -- 172.21.15.16:0/2510094713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c41a84a0 con 0x7fe6c4104630 2024-08-28T19:21:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.252+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c4104630 0x7fe6c4102540 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000c970 tx=0x7fe6c000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:51.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.253+0000 7fe6baffd640 1 -- 172.21.15.16:0/2510094713 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0007bb0 con 0x7fe6c4104630 2024-08-28T19:21:51.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.253+0000 7fe6cc06c640 1 -- 172.21.15.16:0/2510094713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6c41a8790 con 0x7fe6c4104630 2024-08-28T19:21:51.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.253+0000 7fe6baffd640 1 -- 172.21.15.16:0/2510094713 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c0007d50 con 0x7fe6c4104630 2024-08-28T19:21:51.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.253+0000 7fe6cc06c640 1 -- 172.21.15.16:0/2510094713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6c41a8cf0 con 0x7fe6c4104630 2024-08-28T19:21:51.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.253+0000 7fe6baffd640 1 -- 172.21.15.16:0/2510094713 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0018780 con 0x7fe6c4104630 2024-08-28T19:21:51.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.256+0000 7fe6d25e1740 1 Processor -- start 2024-08-28T19:21:51.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6d25e1740 1 -- start start 2024-08-28T19:21:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6baffd640 1 -- 172.21.15.16:0/2510094713 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7fe6c000f450 con 0x7fe6c4104630 2024-08-28T19:21:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6d25e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf47a070 0x5625bf81f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6d25e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf4511b0 0x5625bf81f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6d25e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf459390 0x5625bf81fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6d25e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5625bf796680 con 0x5625bf459390 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6d25e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625bf785c10 con 0x5625bf4511b0 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6d25e1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5625bf789bc0 con 0x5625bf47a070 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6c9607640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf459390 0x5625bf81fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6c9607640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf459390 0x5625bf81fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41068/0 (socket says 172.21.15.16:41068) 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6c9607640 1 -- 172.21.15.16:0/4284600884 learned_addr learned my addr 172.21.15.16:0/4284600884 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/4284600884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf4511b0 0x5625bf81f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.257+0000 7fe6ca609640 1 --2- 172.21.15.16:0/4284600884 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf47a070 0x5625bf81f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.258+0000 7fe6c9607640 1 -- 172.21.15.16:0/4284600884 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf47a070 msgr2=0x5625bf81f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.258+0000 7fe6c9607640 1 --2- 172.21.15.16:0/4284600884 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf47a070 0x5625bf81f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.258+0000 7fe6c9607640 1 -- 172.21.15.16:0/4284600884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf4511b0 msgr2=0x5625bf81f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.258+0000 7fe6c9607640 1 --2- 172.21.15.16:0/4284600884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf4511b0 0x5625bf81f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.258+0000 7fe6c9607640 1 -- 172.21.15.16:0/4284600884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625bf4e5650 con 0x5625bf459390 2024-08-28T19:21:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.258+0000 7fe6ca609640 1 --2- 172.21.15.16:0/4284600884 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf47a070 0x5625bf81f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:51.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.258+0000 7fe6c9607640 1 --2- 172.21.15.16:0/4284600884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf459390 0x5625bf81fd90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4009900 tx=0x7fe6b4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a941a19f835786 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:51.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.258+0000 7fe6affff640 1 -- 172.21.15.16:0/4284600884 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4040070 con 0x5625bf459390 2024-08-28T19:21:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.258+0000 7fe6baffd640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b00b4c90 0x7fe6b00b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.258+0000 7fe6affff640 1 -- 172.21.15.16:0/4284600884 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6b4005db0 con 0x5625bf459390 2024-08-28T19:21:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.258+0000 7fe6affff640 1 -- 172.21.15.16:0/4284600884 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4035040 con 0x5625bf459390 2024-08-28T19:21:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.259+0000 7fe6ca609640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b00b4c90 0x7fe6b00b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.259+0000 7fe6baffd640 1 -- 172.21.15.16:0/2510094713 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe6c00e5850 con 0x7fe6c4104630 2024-08-28T19:21:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.259+0000 7fe6d25e1740 1 -- 172.21.15.16:0/4284600884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf459390 msgr2=0x5625bf81fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.259+0000 7fe6ca609640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b00b4c90 0x7fe6b00b71a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009270 tx=0x7fe6bc009580 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.259+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/4284600884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf459390 0x5625bf81fd90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4009900 tx=0x7fe6b4000bc0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.259+0000 7fe6d25e1740 1 -- 172.21.15.16:0/4284600884 shutdown_connections 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.259+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/4284600884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf459390 0x5625bf81fd90 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.259+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/4284600884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf4511b0 0x5625bf81f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.259+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/4284600884 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf47a070 0x5625bf81f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.259+0000 7fe6d25e1740 1 -- 172.21.15.16:0/4284600884 >> 172.21.15.16:0/4284600884 conn(0x5625bf46e460 msgr2=0x5625bf81d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.260+0000 7fe6d25e1740 1 -- 172.21.15.16:0/4284600884 shutdown_connections 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.260+0000 7fe6d25e1740 1 -- 172.21.15.16:0/4284600884 wait complete. 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.260+0000 7fe6d25e1740 1 -- start start 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.260+0000 7fe6d25e1740 1 Processor -- start 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6d25e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf82a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6d25e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 0x5625bf82af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6d25e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf47a070 0x5625bf836de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6d25e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5625bf5e8700 con 0x5625bf4511b0 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6d25e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625bf70c750 con 0x5625bf47a070 2024-08-28T19:21:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6d25e1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5625bf82da90 con 0x5625bf459390 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6d25e1740 10 client.0 _ll_register_callbacks cb 0x5625bf73be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6c9607640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 0x5625bf82af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6c9607640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 0x5625bf82af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:50744/0 (socket says 172.21.15.16:50744) 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6c9607640 1 -- 172.21.15.16:0/1120940313 learned_addr learned my addr 172.21.15.16:0/1120940313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6c9607640 1 -- 172.21.15.16:0/1120940313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf47a070 msgr2=0x5625bf836de0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6c9607640 1 --2- 172.21.15.16:0/1120940313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf47a070 0x5625bf836de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6c9607640 1 -- 172.21.15.16:0/1120940313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 msgr2=0x5625bf82a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6c9607640 1 --2- 172.21.15.16:0/1120940313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf82a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.261+0000 7fe6c9607640 1 -- 172.21.15.16:0/1120940313 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625bf745900 con 0x5625bf459390 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.262+0000 7fe6c9607640 1 --2- 172.21.15.16:0/1120940313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 0x5625bf82af30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4005660 tx=0x7fe6b4005690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69ae8df35f1a5789 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.262+0000 7fe6ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.262+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/1120940313 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b404b020 con 0x5625bf459390 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.262+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/1120940313 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe6b4004750 con 0x5625bf459390 2024-08-28T19:21:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.262+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1120940313 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6b4005db0 con 0x5625bf459390 2024-08-28T19:21:51.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.262+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/1120940313 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe6b4056050 con 0x5625bf459390 2024-08-28T19:21:51.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.262+0000 7fe6d25e1740 10 client.25438 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:51.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.262+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1120940313 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625bf839830 con 0x5625bf459390 2024-08-28T19:21:51.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.263+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/1120940313 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6b4004e00 con 0x5625bf459390 2024-08-28T19:21:51.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.263+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/1120940313 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe6b403cb20 con 0x5625bf459390 2024-08-28T19:21:51.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.263+0000 7fe6d25e1740 10 client.25438 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.263+0000 7fe6d25e1740 10 client.25438 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.263+0000 7fe6d25e1740 20 client.25438 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.263+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/1120940313 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5625bf4389f0 0x5625bf83cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.263+0000 7fe6d25e1740 4 client.25438 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.263+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1120940313 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe6b4005db0 con 0x5625bf4389f0 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.264+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/1120940313 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5625bf4389f0 0x5625bf83cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.264+0000 7fe6ad7fa640 10 client.25438 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.264+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/1120940313 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5625bf4389f0 0x5625bf83cb90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ff7bf55c48671a70 server_cookie=a919532256df2878 in_seq=0 out_seq=0 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/1120940313 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fe6b4005db0 con 0x5625bf4389f0 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6ad7fa640 10 client.25438 handle_command_reply: tid=0 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6d25e1740 1 client.25438 shutdown 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1120940313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 msgr2=0x5625bf82af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/1120940313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 0x5625bf82af30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4005660 tx=0x7fe6b4005690 comp rx=0 tx=0).stop 2024-08-28T19:21:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1120940313 shutdown_connections 2024-08-28T19:21:51.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/1120940313 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5625bf4389f0 0x5625bf83cb90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6ad7fa640 0 client.25438 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:21:51.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/1120940313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf47a070 0x5625bf836de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/1120940313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 0x5625bf82af30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/1120940313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf82a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.265+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1120940313 >> 172.21.15.16:0/1120940313 conn(0x5625bf46e460 msgr2=0x5625bf8224a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:51.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.266+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1120940313 shutdown_connections 2024-08-28T19:21:51.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.266+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1120940313 wait complete. 2024-08-28T19:21:51.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.266+0000 7fe6d25e1740 20 client.25438 trim_cache size 0 max 16384 2024-08-28T19:21:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.282+0000 7fe6d25e1740 1 Processor -- start 2024-08-28T19:21:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.282+0000 7fe6d25e1740 1 -- start start 2024-08-28T19:21:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.282+0000 7fe6d25e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf820e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.282+0000 7fe6d25e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf459390 0x5625bf82c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.282+0000 7fe6d25e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf47a070 0x5625bf836ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.282+0000 7fe6d25e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c41b2cf0 con 0x5625bf4511b0 2024-08-28T19:21:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6d25e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c41b4830 con 0x5625bf459390 2024-08-28T19:21:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6d25e1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c41b33c0 con 0x5625bf47a070 2024-08-28T19:21:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6c9e08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf820e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6c9e08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf820e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41084/0 (socket says 172.21.15.16:41084) 2024-08-28T19:21:51.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6c9e08640 1 -- 172.21.15.16:0/1165862874 learned_addr learned my addr 172.21.15.16:0/1165862874 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:51.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6c9607640 1 --2- 172.21.15.16:0/1165862874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf459390 0x5625bf82c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6c9e08640 1 -- 172.21.15.16:0/1165862874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf47a070 msgr2=0x5625bf836ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:21:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/1165862874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf47a070 0x5625bf836ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6c9e08640 1 -- 172.21.15.16:0/1165862874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf459390 msgr2=0x5625bf82c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/1165862874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf459390 0x5625bf82c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6c9e08640 1 -- 172.21.15.16:0/1165862874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625bf82bcb0 con 0x5625bf4511b0 2024-08-28T19:21:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.283+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/1165862874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf820e10 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000ffd0 tx=0x7fe6c00e7ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b2f3ff2c990ffa server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.284+0000 7fe6af7fe640 1 -- 172.21.15.16:0/1165862874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c00a73c0 con 0x5625bf4511b0 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.284+0000 7fe6af7fe640 1 -- 172.21.15.16:0/1165862874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c00930b0 con 0x5625bf4511b0 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.284+0000 7fe6af7fe640 1 -- 172.21.15.16:0/1165862874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c00b4f50 con 0x5625bf4511b0 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.285+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1165862874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 msgr2=0x5625bf820e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.285+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/1165862874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf820e10 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000ffd0 tx=0x7fe6c00e7ca0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.285+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1165862874 shutdown_connections 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.285+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/1165862874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf47a070 0x5625bf836ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.285+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/1165862874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf459390 0x5625bf82c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.285+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/1165862874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf820e10 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.285+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1165862874 >> 172.21.15.16:0/1165862874 conn(0x5625bf46e460 msgr2=0x5625bf824420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.285+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1165862874 shutdown_connections 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.285+0000 7fe6d25e1740 1 -- 172.21.15.16:0/1165862874 wait complete. 2024-08-28T19:21:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.285+0000 7fe6d25e1740 1 -- start start 2024-08-28T19:21:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.285+0000 7fe6d25e1740 1 Processor -- start 2024-08-28T19:21:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.286+0000 7fe6d25e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf82f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.286+0000 7fe6d25e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 0x5625bf82fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.286+0000 7fe6c9e08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf82f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.286+0000 7fe6c9e08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf82f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41094/0 (socket says 172.21.15.16:41094) 2024-08-28T19:21:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.286+0000 7fe6c9e08640 1 -- 172.21.15.16:0/2169743385 learned_addr learned my addr 172.21.15.16:0/2169743385 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.286+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf47a070 0x5625bf8a6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.286+0000 7fe6d25e1740 1 -- 172.21.15.16:0/2169743385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c41b4160 con 0x5625bf4511b0 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.287+0000 7fe6c9607640 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 0x5625bf82fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.287+0000 7fe6d25e1740 1 -- 172.21.15.16:0/2169743385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c41b3a90 con 0x5625bf47a070 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.287+0000 7fe6ca609640 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf47a070 0x5625bf8a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.287+0000 7fe6d25e1740 1 -- 172.21.15.16:0/2169743385 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c41b0cd0 con 0x5625bf459390 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.287+0000 7fe6d25e1740 10 client.0 _ll_register_callbacks cb 0x7fe6c00eb400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.287+0000 7fe6c9e08640 1 -- 172.21.15.16:0/2169743385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 msgr2=0x5625bf82fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.287+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 0x5625bf82fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.287+0000 7fe6c9e08640 1 -- 172.21.15.16:0/2169743385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf47a070 msgr2=0x5625bf8a6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.287+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf47a070 0x5625bf8a6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.287+0000 7fe6c9e08640 1 -- 172.21.15.16:0/2169743385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625bf8a6920 con 0x5625bf4511b0 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.288+0000 7fe6c9e08640 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf82f530 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00a7cb0 tx=0x7fe6c00eb380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef8ab667399d035 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.288+0000 7fe6ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.288+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/2169743385 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c00a73c0 con 0x5625bf4511b0 2024-08-28T19:21:51.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.288+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/2169743385 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe6c000f740 con 0x5625bf4511b0 2024-08-28T19:21:51.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.288+0000 7fe6d25e1740 1 -- 172.21.15.16:0/2169743385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5625bf7985d0 con 0x5625bf4511b0 2024-08-28T19:21:51.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.288+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/2169743385 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe6c00abc30 con 0x5625bf4511b0 2024-08-28T19:21:51.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.288+0000 7fe6d25e1740 10 client.15903 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.288+0000 7fe6d25e1740 1 -- 172.21.15.16:0/2169743385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625bf8a7b40 con 0x5625bf4511b0 2024-08-28T19:21:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.289+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/2169743385 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5625bf7985d0 con 0x5625bf4511b0 2024-08-28T19:21:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.289+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/2169743385 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe6c00e2f60 con 0x5625bf4511b0 2024-08-28T19:21:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.289+0000 7fe6d25e1740 10 client.15903 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.289+0000 7fe6d25e1740 10 client.15903 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:21:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.289+0000 7fe6d25e1740 20 client.15903 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.289+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5625bf40dda0 0x5625bf8aa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.289+0000 7fe6d25e1740 4 client.15903 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.289+0000 7fe6d25e1740 1 -- 172.21.15.16:0/2169743385 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625bf794070 con 0x5625bf40dda0 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.289+0000 7fe6c9607640 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5625bf40dda0 0x5625bf8aa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6ad7fa640 10 client.15903 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6c9607640 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5625bf40dda0 0x5625bf8aa2f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e10b6123e002f7bc server_cookie=8a44a81c99e1871c in_seq=0 out_seq=0 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6ad7fa640 1 -- 172.21.15.16:0/2169743385 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625bf794070 con 0x5625bf40dda0 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6ad7fa640 10 client.15903 handle_command_reply: tid=0 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6d25e1740 1 client.15903 shutdown 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6d25e1740 1 -- 172.21.15.16:0/2169743385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 msgr2=0x5625bf82f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf82f530 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00a7cb0 tx=0x7fe6c00eb380 comp rx=0 tx=0).stop 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6d25e1740 1 -- 172.21.15.16:0/2169743385 shutdown_connections 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5625bf40dda0 0x5625bf8aa2f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bf47a070 0x5625bf8a6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6ad7fa640 0 client.15903 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:21:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625bf459390 0x5625bf82fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6d25e1740 1 --2- 172.21.15.16:0/2169743385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625bf4511b0 0x5625bf82f530 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6d25e1740 1 -- 172.21.15.16:0/2169743385 >> 172.21.15.16:0/2169743385 conn(0x5625bf46e460 msgr2=0x5625bf82c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6d25e1740 1 -- 172.21.15.16:0/2169743385 shutdown_connections 2024-08-28T19:21:51.295 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.290+0000 7fe6d25e1740 1 -- 172.21.15.16:0/2169743385 wait complete. 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.291+0000 7fe6d25e1740 20 client.15903 trim_cache size 0 max 16384 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.291+0000 7fe6cc06c640 1 -- 172.21.15.16:0/2510094713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b00b4c90 msgr2=0x7fe6b00b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.291+0000 7fe6cc06c640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b00b4c90 0x7fe6b00b71a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009270 tx=0x7fe6bc009580 comp rx=0 tx=0).stop 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.291+0000 7fe6cc06c640 1 -- 172.21.15.16:0/2510094713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c4104630 msgr2=0x7fe6c4102540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.291+0000 7fe6cc06c640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c4104630 0x7fe6c4102540 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000c970 tx=0x7fe6c000ce30 comp rx=0 tx=0).stop 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.291+0000 7fe6cc06c640 1 -- 172.21.15.16:0/2510094713 shutdown_connections 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.291+0000 7fe6cc06c640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c410dbe0 0x7fe6c4102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.291+0000 7fe6cc06c640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6c4105060 0x7fe6c4102a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.291+0000 7fe6cc06c640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b00b4c90 0x7fe6b00b71a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.291+0000 7fe6cc06c640 1 --2- 172.21.15.16:0/2510094713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c4104630 0x7fe6c4102540 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.291+0000 7fe6cc06c640 1 -- 172.21.15.16:0/2510094713 >> 172.21.15.16:0/2510094713 conn(0x7fe6c4079410 msgr2=0x7fe6c4100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.292+0000 7fe6cc06c640 1 -- 172.21.15.16:0/2510094713 shutdown_connections 2024-08-28T19:21:51.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:51.292+0000 7fe6cc06c640 1 -- 172.21.15.16:0/2510094713 wait complete. 2024-08-28T19:21:51.975 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:21:52.247 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:52.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.722+0000 7f6ef3b45640 1 -- 172.21.15.16:0/1535077196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec109bc0 msgr2=0x7f6eec1103f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.722+0000 7f6ef3b45640 1 --2- 172.21.15.16:0/1535077196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec109bc0 0x7f6eec1103f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009ca0 tx=0x7f6ee402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.722+0000 7f6ef3b45640 1 -- 172.21.15.16:0/1535077196 shutdown_connections 2024-08-28T19:21:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.722+0000 7f6ef3b45640 1 --2- 172.21.15.16:0/1535077196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec109bc0 0x7f6eec1103f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.722+0000 7f6ef3b45640 1 --2- 172.21.15.16:0/1535077196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eec109270 0x7f6eec109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.722+0000 7f6ef3b45640 1 --2- 172.21.15.16:0/1535077196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6eec103930 0x7f6eec103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.722+0000 7f6ef3b45640 1 -- 172.21.15.16:0/1535077196 >> 172.21.15.16:0/1535077196 conn(0x7f6eec0ff380 msgr2=0x7f6eec1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.722+0000 7f6ef3b45640 1 -- 172.21.15.16:0/1535077196 shutdown_connections 2024-08-28T19:21:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.723+0000 7f6ef3b45640 1 -- 172.21.15.16:0/1535077196 wait complete. 2024-08-28T19:21:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.724+0000 7f6ef3b45640 1 Processor -- start 2024-08-28T19:21:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.724+0000 7f6ef3b45640 1 -- start start 2024-08-28T19:21:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.724+0000 7f6ef3b45640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec103930 0x7f6eec19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.724+0000 7f6ef3b45640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6eec109270 0x7f6eec19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef3b45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eec109bc0 0x7f6eec19fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef3b45640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec113880 con 0x7f6eec103930 2024-08-28T19:21:52.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef3b45640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec113700 con 0x7f6eec109bc0 2024-08-28T19:21:52.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef3b45640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec113a00 con 0x7f6eec109270 2024-08-28T19:21:52.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef10e0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6eec109270 0x7f6eec19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef20e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eec109bc0 0x7f6eec19fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef10e0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6eec109270 0x7f6eec19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:50772/0 (socket says 172.21.15.16:50772) 2024-08-28T19:21:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef10e0640 1 -- 172.21.15.16:0/3124500638 learned_addr learned my addr 172.21.15.16:0/3124500638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec103930 0x7f6eec19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef20e2640 1 -- 172.21.15.16:0/3124500638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6eec109270 msgr2=0x7f6eec19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6eec109270 0x7f6eec19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef20e2640 1 -- 172.21.15.16:0/3124500638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec103930 msgr2=0x7f6eec19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec103930 0x7f6eec19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef20e2640 1 -- 172.21.15.16:0/3124500638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eec1a0160 con 0x7f6eec109bc0 2024-08-28T19:21:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.725+0000 7f6ef10e0640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6eec109270 0x7f6eec19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.726+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec103930 0x7f6eec19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.726+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eec109bc0 0x7f6eec19fa30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6ee402ade0 tx=0x7f6ee4031660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.726+0000 7f6ee2ffd640 1 -- 172.21.15.16:0/3124500638 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4003b70 con 0x7f6eec109bc0 2024-08-28T19:21:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.726+0000 7f6ee2ffd640 1 -- 172.21.15.16:0/3124500638 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ee4031910 con 0x7f6eec109bc0 2024-08-28T19:21:52.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.726+0000 7f6ee2ffd640 1 -- 172.21.15.16:0/3124500638 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee402f950 con 0x7f6eec109bc0 2024-08-28T19:21:52.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.726+0000 7f6ef3b45640 1 -- 172.21.15.16:0/3124500638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eec1a03f0 con 0x7f6eec109bc0 2024-08-28T19:21:52.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.726+0000 7f6ef3b45640 1 -- 172.21.15.16:0/3124500638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eec1a0820 con 0x7f6eec109bc0 2024-08-28T19:21:52.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.729+0000 7f6ee2ffd640 1 -- 172.21.15.16:0/3124500638 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f6ee402faf0 con 0x7f6eec109bc0 2024-08-28T19:21:52.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.729+0000 7f6ee2ffd640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ed80b4c10 0x7f6ed80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.730+0000 7f6ee2ffd640 1 -- 172.21.15.16:0/3124500638 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6ee41068d0 con 0x7f6eec109bc0 2024-08-28T19:21:52.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.730+0000 7f6efa0ba740 1 Processor -- start 2024-08-28T19:21:52.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.730+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ed80b4c10 0x7f6ed80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.730+0000 7f6efa0ba740 1 -- start start 2024-08-28T19:21:52.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.731+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c8061f8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.731+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e2a1b0 0x55c8061f8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.731+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e32390 0x55c8061f8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.731+0000 7f6ef18e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c8061f8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.731+0000 7f6ef18e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c8061f8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41126/0 (socket says 172.21.15.16:41126) 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.731+0000 7f6ef18e1640 1 -- 172.21.15.16:0/1160348721 learned_addr learned my addr 172.21.15.16:0/1160348721 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.731+0000 7f6ef10e0640 1 --2- 172.21.15.16:0/1160348721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e2a1b0 0x55c8061f8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.731+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1160348721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c80616f680 con 0x55c805e53070 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.731+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1160348721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c80615ec10 con 0x55c805e2a1b0 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.731+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1160348721 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c806162bc0 con 0x55c805e32390 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.731+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/1160348721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e32390 0x55c8061f8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.732+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ed80b4c10 0x7f6ed80b7120 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009b50 tx=0x7f6ee8005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.732+0000 7f6ef20e2640 1 -- 172.21.15.16:0/1160348721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e2a1b0 msgr2=0x55c8061f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.732+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/1160348721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e2a1b0 0x55c8061f8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.732+0000 7f6ef20e2640 1 -- 172.21.15.16:0/1160348721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 msgr2=0x55c8061f8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.732+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/1160348721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c8061f8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.732+0000 7f6ef20e2640 1 -- 172.21.15.16:0/1160348721 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c805ebe650 con 0x55c805e32390 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.732+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/1160348721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c8061f8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.732+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/1160348721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e32390 0x55c8061f8d90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4103df0 tx=0x7f6ee4005c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b877d2fba39cbb41 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.733+0000 7f6ed7fff640 1 -- 172.21.15.16:0/1160348721 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4003d10 con 0x55c805e32390 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.733+0000 7f6ed7fff640 1 -- 172.21.15.16:0/1160348721 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ee4036040 con 0x55c805e32390 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.733+0000 7f6ed7fff640 1 -- 172.21.15.16:0/1160348721 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee40b40e0 con 0x55c805e32390 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.734+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1160348721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e32390 msgr2=0x55c8061f8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.734+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1160348721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e32390 0x55c8061f8d90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4103df0 tx=0x7f6ee4005c30 comp rx=0 tx=0).stop 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.734+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1160348721 shutdown_connections 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.734+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1160348721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e32390 0x55c8061f8d90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.734+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1160348721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e2a1b0 0x55c8061f8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.734+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1160348721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c8061f8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.734+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1160348721 >> 172.21.15.16:0/1160348721 conn(0x55c805e47460 msgr2=0x55c8061f6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.734+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1160348721 shutdown_connections 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.734+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1160348721 wait complete. 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.735+0000 7f6efa0ba740 1 -- start start 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.735+0000 7f6efa0ba740 1 Processor -- start 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.735+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c806203af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.735+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 0x55c806204030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.735+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 0x55c80620f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.735+0000 7f6efa0ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c805fc1700 con 0x55c805e32390 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.735+0000 7f6efa0ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c8060e5750 con 0x55c805e53070 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.735+0000 7f6efa0ba740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c806162bc0 con 0x55c805e2a1b0 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.735+0000 7f6efa0ba740 10 client.0 _ll_register_callbacks cb 0x55c806114e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.736+0000 7f6ef18e1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c806203af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.736+0000 7f6ef18e1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c806203af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:50804/0 (socket says 172.21.15.16:50804) 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.736+0000 7f6ef18e1640 1 -- 172.21.15.16:0/1312810888 learned_addr learned my addr 172.21.15.16:0/1312810888 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.736+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 0x55c80620f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.736+0000 7f6ef20e2640 1 -- 172.21.15.16:0/1312810888 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 msgr2=0x55c806203af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.736+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c806203af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.736+0000 7f6ef20e2640 1 -- 172.21.15.16:0/1312810888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 msgr2=0x55c806204030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.736+0000 7f6ef10e0640 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 0x55c806204030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.736+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 0x55c806204030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.736+0000 7f6ef20e2640 1 -- 172.21.15.16:0/1312810888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c80611e900 con 0x55c805e53070 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.736+0000 7f6ef10e0640 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 0x55c806204030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.737+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 0x55c80620f910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4103df0 tx=0x7f6ee40c91d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81aeef0e00e7949d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.737+0000 7f6ed57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.737+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1312810888 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee40d1580 con 0x55c805e53070 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.737+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1312810888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ee4036040 con 0x55c805e53070 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.737+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1312810888 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6ee40d1160 con 0x55c805e53070 2024-08-28T19:21:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.737+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1312810888 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6ee40d6030 con 0x55c805e53070 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.737+0000 7f6efa0ba740 10 client.25544 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.737+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1312810888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c806212400 con 0x55c805e53070 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.737+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1312810888 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ee4036040 con 0x55c805e53070 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.739+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1312810888 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6ee40ccf70 con 0x55c805e53070 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.739+0000 7f6efa0ba740 10 client.25544 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.739+0000 7f6efa0ba740 10 client.25544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.739+0000 7f6efa0ba740 20 client.25544 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.739+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c805e119f0 0x55c806215890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.739+0000 7f6efa0ba740 4 client.25544 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.739+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1312810888 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8061715d0 con 0x55c805e119f0 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.739+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c805e119f0 0x55c806215890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.740+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c805e119f0 0x55c806215890 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bc620e08a1a1930 server_cookie=89bec8b6f1de02bc in_seq=0 out_seq=0 2024-08-28T19:21:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.740+0000 7f6ed57fa640 10 client.25544 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.740+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1312810888 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c8061715d0 con 0x55c805e119f0 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.740+0000 7f6ed57fa640 10 client.25544 handle_command_reply: tid=0 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6efa0ba740 1 client.25544 shutdown 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1312810888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 msgr2=0x55c80620f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 0x55c80620f910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4103df0 tx=0x7f6ee40c91d0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1312810888 shutdown_connections 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 0x55c80620f910 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 0x55c806204030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c805e119f0 0x55c806215890 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1312810888 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c806203af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1312810888 >> 172.21.15.16:0/1312810888 conn(0x55c805e47460 msgr2=0x55c8061f9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6ed57fa640 0 client.25544 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1312810888 shutdown_connections 2024-08-28T19:21:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.741+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1312810888 wait complete. 2024-08-28T19:21:52.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.742+0000 7f6efa0ba740 20 client.25544 trim_cache size 0 max 16384 2024-08-28T19:21:52.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.768+0000 7f6efa0ba740 1 Processor -- start 2024-08-28T19:21:52.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.769+0000 7f6efa0ba740 1 -- start start 2024-08-28T19:21:52.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.769+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c8062059d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.769+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e32390 0x55c80620f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.769+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c80620fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.769+0000 7f6efa0ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec1b2da0 con 0x55c805e53070 2024-08-28T19:21:52.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.769+0000 7f6efa0ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec1b48e0 con 0x55c805e32390 2024-08-28T19:21:52.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.769+0000 7f6efa0ba740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec1b3470 con 0x55c805e2a1b0 2024-08-28T19:21:52.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.769+0000 7f6ef20e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c80620fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.769+0000 7f6ef20e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c80620fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41142/0 (socket says 172.21.15.16:41142) 2024-08-28T19:21:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.769+0000 7f6ef20e2640 1 -- 172.21.15.16:0/1592513142 learned_addr learned my addr 172.21.15.16:0/1592513142 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.769+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/1592513142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c8062059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.770+0000 7f6ef10e0640 1 --2- 172.21.15.16:0/1592513142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e32390 0x55c80620f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.770+0000 7f6ef18e1640 1 -- 172.21.15.16:0/1592513142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e32390 msgr2=0x55c80620f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.770+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/1592513142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e32390 0x55c80620f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.770+0000 7f6ef18e1640 1 -- 172.21.15.16:0/1592513142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 msgr2=0x55c80620fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:52.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.770+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/1592513142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c80620fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.770+0000 7f6ef18e1640 1 -- 172.21.15.16:0/1592513142 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8061f6fc0 con 0x55c805e2a1b0 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.770+0000 7f6ef10e0640 1 --2- 172.21.15.16:0/1592513142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e32390 0x55c80620f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.770+0000 7f6ef20e2640 1 --2- 172.21.15.16:0/1592513142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c80620fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.771+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/1592513142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c8062059d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8010800 tx=0x7f6ee8013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbb61c0232fcd215 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.771+0000 7f6ed77fe640 1 -- 172.21.15.16:0/1592513142 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee80221d0 con 0x55c805e2a1b0 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.771+0000 7f6ed77fe640 1 -- 172.21.15.16:0/1592513142 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ee8003610 con 0x55c805e2a1b0 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.771+0000 7f6ed77fe640 1 -- 172.21.15.16:0/1592513142 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8022750 con 0x55c805e2a1b0 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.772+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1592513142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 msgr2=0x55c8062059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.772+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1592513142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c8062059d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8010800 tx=0x7f6ee8013000 comp rx=0 tx=0).stop 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.772+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1592513142 shutdown_connections 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.772+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1592513142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e53070 0x55c80620fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.772+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1592513142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e32390 0x55c80620f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.772+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1592513142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c8062059d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.772+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1592513142 >> 172.21.15.16:0/1592513142 conn(0x55c805e47460 msgr2=0x55c8061fd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.772+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1592513142 shutdown_connections 2024-08-28T19:21:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.773+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1592513142 wait complete. 2024-08-28T19:21:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.773+0000 7f6efa0ba740 1 -- start start 2024-08-28T19:21:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.773+0000 7f6efa0ba740 1 Processor -- start 2024-08-28T19:21:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.774+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c806209b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.774+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 0x55c80620a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.774+0000 7f6efa0ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 0x55c80627f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.775+0000 7f6efa0ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec1b4210 con 0x55c805e32390 2024-08-28T19:21:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.775+0000 7f6efa0ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec1b3b40 con 0x55c805e53070 2024-08-28T19:21:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.775+0000 7f6efa0ba740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec1b0d80 con 0x55c805e2a1b0 2024-08-28T19:21:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.775+0000 7f6ef10e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 0x55c80620a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.775+0000 7f6ef18e1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c806209b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.775+0000 7f6ef20e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 0x55c80627f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.775+0000 7f6ef20e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 0x55c80627f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49310/0 (socket says 172.21.15.16:49310) 2024-08-28T19:21:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.775+0000 7f6ef20e2640 1 -- 172.21.15.16:0/1604540156 learned_addr learned my addr 172.21.15.16:0/1604540156 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.775+0000 7f6efa0ba740 10 client.0 _ll_register_callbacks cb 0x7f6ee8016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.777+0000 7f6ef10e0640 1 -- 172.21.15.16:0/1604540156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 msgr2=0x55c806209b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.777+0000 7f6ef10e0640 1 --2- 172.21.15.16:0/1604540156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c806209b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.777+0000 7f6ef10e0640 1 -- 172.21.15.16:0/1604540156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 msgr2=0x55c80627f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.777+0000 7f6ef10e0640 1 --2- 172.21.15.16:0/1604540156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 0x55c80627f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.777+0000 7f6ef10e0640 1 -- 172.21.15.16:0/1604540156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c80627f770 con 0x55c805e32390 2024-08-28T19:21:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.778+0000 7f6ef10e0640 1 --2- 172.21.15.16:0/1604540156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 0x55c80620a0f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x55c8061f6f90 tx=0x7f6edc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de760ef537cb6825 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.778+0000 7f6ed57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.778+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1604540156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6edc021060 con 0x55c805e32390 2024-08-28T19:21:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.778+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1604540156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6edc004050 con 0x55c805e32390 2024-08-28T19:21:52.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.778+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1604540156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ee8003610 con 0x55c805e32390 2024-08-28T19:21:52.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.779+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1604540156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6edc004250 con 0x55c805e32390 2024-08-28T19:21:52.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.779+0000 7f6efa0ba740 10 client.15915 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:52.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.779+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1604540156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c806280980 con 0x55c805e32390 2024-08-28T19:21:52.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.779+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1604540156 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ee8003610 con 0x55c805e32390 2024-08-28T19:21:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.779+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1604540156 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6edc025540 con 0x55c805e32390 2024-08-28T19:21:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.779+0000 7f6efa0ba740 10 client.15915 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.779+0000 7f6efa0ba740 10 client.15915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.780+0000 7f6efa0ba740 20 client.15915 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.780+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1604540156 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c805de6da0 0x55c806283100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.780+0000 7f6efa0ba740 4 client.15915 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:21:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.780+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1604540156 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c80616d070 con 0x55c805de6da0 2024-08-28T19:21:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.780+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/1604540156 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c805de6da0 0x55c806283100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:52.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.780+0000 7f6ed57fa640 10 client.15915 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:52.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.781+0000 7f6ef18e1640 1 --2- 172.21.15.16:0/1604540156 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c805de6da0 0x55c806283100 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=646ce30dbfab39a8 server_cookie=c6fbb361d1901b47 in_seq=0 out_seq=0 2024-08-28T19:21:52.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.781+0000 7f6ed57fa640 1 -- 172.21.15.16:0/1604540156 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c80616d070 con 0x55c805de6da0 2024-08-28T19:21:52.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.781+0000 7f6ed57fa640 10 client.15915 handle_command_reply: tid=0 2024-08-28T19:21:52.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.781+0000 7f6efa0ba740 1 client.15915 shutdown 2024-08-28T19:21:52.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1604540156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 msgr2=0x55c80620a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:52.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1604540156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 0x55c80620a0f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x55c8061f6f90 tx=0x7f6edc00f790 comp rx=0 tx=0).stop 2024-08-28T19:21:52.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1604540156 shutdown_connections 2024-08-28T19:21:52.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1604540156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c805e53070 0x55c80627f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1604540156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c805e32390 0x55c80620a0f0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1604540156 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c805de6da0 0x55c806283100 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6ed57fa640 0 client.15915 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:52.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6efa0ba740 1 --2- 172.21.15.16:0/1604540156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c805e2a1b0 0x55c806209b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1604540156 >> 172.21.15.16:0/1604540156 conn(0x55c805e47460 msgr2=0x55c8062056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:52.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1604540156 shutdown_connections 2024-08-28T19:21:52.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6efa0ba740 1 -- 172.21.15.16:0/1604540156 wait complete. 2024-08-28T19:21:52.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.782+0000 7f6efa0ba740 20 client.15915 trim_cache size 0 max 16384 2024-08-28T19:21:52.786 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:21:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 -- 172.21.15.16:0/3124500638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ed80b4c10 msgr2=0x7f6ed80b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ed80b4c10 0x7f6ed80b7120 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009b50 tx=0x7f6ee8005f90 comp rx=0 tx=0).stop 2024-08-28T19:21:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 -- 172.21.15.16:0/3124500638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eec109bc0 msgr2=0x7f6eec19fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eec109bc0 0x7f6eec19fa30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6ee402ade0 tx=0x7f6ee4031660 comp rx=0 tx=0).stop 2024-08-28T19:21:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 -- 172.21.15.16:0/3124500638 shutdown_connections 2024-08-28T19:21:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6ed80b4c10 0x7f6ed80b7120 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eec109bc0 0x7f6eec19fa30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6eec109270 0x7f6eec19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 --2- 172.21.15.16:0/3124500638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec103930 0x7f6eec19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 -- 172.21.15.16:0/3124500638 >> 172.21.15.16:0/3124500638 conn(0x7f6eec0ff380 msgr2=0x7f6eec1017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 -- 172.21.15.16:0/3124500638 shutdown_connections 2024-08-28T19:21:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:52.783+0000 7f6ef3b45640 1 -- 172.21.15.16:0/3124500638 wait complete. 2024-08-28T19:21:53.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:52 smithi016 ceph-mon[18556]: pgmap v364: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-08-28T19:21:53.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:52 smithi028 ceph-mon[21769]: pgmap v364: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-08-28T19:21:53.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:52 smithi115 ceph-mon[22184]: pgmap v364: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-08-28T19:21:54.561 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:21:54.561 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:21:54.921 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:55.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:54 smithi016 ceph-mon[18556]: pgmap v365: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-08-28T19:21:55.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:54 smithi028 ceph-mon[21769]: pgmap v365: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-08-28T19:21:55.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:54 smithi115 ceph-mon[22184]: pgmap v365: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-08-28T19:21:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.447+0000 7f6ca6b91640 1 -- 172.21.15.16:0/3560644872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0108170 msgr2=0x7f6ca0108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.447+0000 7f6ca6b91640 1 --2- 172.21.15.16:0/3560644872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0108170 0x7f6ca0108580 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c009900 tx=0x7f6c8c02a410 comp rx=0 tx=0).stop 2024-08-28T19:21:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.447+0000 7f6ca6b91640 1 -- 172.21.15.16:0/3560644872 shutdown_connections 2024-08-28T19:21:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.448+0000 7f6ca6b91640 1 --2- 172.21.15.16:0/3560644872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ca0108ac0 0x7f6ca010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.448+0000 7f6ca6b91640 1 --2- 172.21.15.16:0/3560644872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0108170 0x7f6ca0108580 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.448+0000 7f6ca6b91640 1 --2- 172.21.15.16:0/3560644872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca0102880 0x7f6ca0102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.448+0000 7f6ca6b91640 1 -- 172.21.15.16:0/3560644872 >> 172.21.15.16:0/3560644872 conn(0x7f6ca0079530 msgr2=0x7f6ca0079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.448+0000 7f6ca6b91640 1 -- 172.21.15.16:0/3560644872 shutdown_connections 2024-08-28T19:21:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.448+0000 7f6ca6b91640 1 -- 172.21.15.16:0/3560644872 wait complete. 2024-08-28T19:21:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.449+0000 7f6ca6b91640 1 Processor -- start 2024-08-28T19:21:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.449+0000 7f6ca6b91640 1 -- start start 2024-08-28T19:21:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6ca6b91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca0102880 0x7f6ca0199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6ca6b91640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ca0108170 0x7f6ca01996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6ca6b91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0108ac0 0x7f6ca019e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6ca492d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca0102880 0x7f6ca0199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ca0108170 0x7f6ca01996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ca0108170 0x7f6ca01996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:52098/0 (socket says 172.21.15.16:52098) 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6ca512e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0108ac0 0x7f6ca019e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6ca492d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca0102880 0x7f6ca0199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49736/0 (socket says 172.21.15.16:49736) 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6ca512e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0108ac0 0x7f6ca019e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51752/0 (socket says 172.21.15.16:51752) 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6c9ffff640 1 -- 172.21.15.16:0/2996970232 learned_addr learned my addr 172.21.15.16:0/2996970232 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6ca6b91640 1 -- 172.21.15.16:0/2996970232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca0112730 con 0x7f6ca0108ac0 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6ca6b91640 1 -- 172.21.15.16:0/2996970232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01125b0 con 0x7f6ca0102880 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.450+0000 7f6ca6b91640 1 -- 172.21.15.16:0/2996970232 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01128b0 con 0x7f6ca0108170 2024-08-28T19:21:55.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.451+0000 7f6ca492d640 1 -- 172.21.15.16:0/2996970232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ca0108170 msgr2=0x7f6ca01996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.451+0000 7f6ca492d640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ca0108170 0x7f6ca01996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.451+0000 7f6ca492d640 1 -- 172.21.15.16:0/2996970232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0108ac0 msgr2=0x7f6ca019e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.451+0000 7f6ca492d640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0108ac0 0x7f6ca019e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.451+0000 7f6ca492d640 1 -- 172.21.15.16:0/2996970232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca019f0b0 con 0x7f6ca0102880 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.451+0000 7f6ca512e640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0108ac0 0x7f6ca019e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.451+0000 7f6ca492d640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca0102880 0x7f6ca0199140 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400e9f0 tx=0x7f6c9400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.451+0000 7f6c9dffb640 1 -- 172.21.15.16:0/2996970232 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9400ccc0 con 0x7f6ca0102880 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.451+0000 7f6ca6b91640 1 -- 172.21.15.16:0/2996970232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca019f3a0 con 0x7f6ca0102880 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.452+0000 7f6ca6b91640 1 -- 172.21.15.16:0/2996970232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca01a5b00 con 0x7f6ca0102880 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.452+0000 7f6c9dffb640 1 -- 172.21.15.16:0/2996970232 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c940108e0 con 0x7f6ca0102880 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.452+0000 7f6c9dffb640 1 -- 172.21.15.16:0/2996970232 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c94018880 con 0x7f6ca0102880 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.455+0000 7f6c9dffb640 1 -- 172.21.15.16:0/2996970232 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f6c94010430 con 0x7f6ca0102880 2024-08-28T19:21:55.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.455+0000 7f6cad106740 1 Processor -- start 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.455+0000 7f6cad106740 1 -- start start 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.455+0000 7f6cad106740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c51350 0x55d411ff7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.455+0000 7f6cad106740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c28490 0x55d411ff7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.455+0000 7f6cad106740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d411ff8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6cad106740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d411f6d5f0 con 0x55d411c28490 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6cad106740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d411f5cb80 con 0x55d411c30670 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6cad106740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d411f60b30 con 0x55d411c51350 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6ca492d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c28490 0x55d411ff7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d411ff8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d411ff8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49742/0 (socket says 172.21.15.16:49742) 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6c9ffff640 1 -- 172.21.15.16:0/2565651472 learned_addr learned my addr 172.21.15.16:0/2565651472 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:55.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6ca492d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c28490 0x55d411ff7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51762/0 (socket says 172.21.15.16:51762) 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6ca512e640 1 --2- 172.21.15.16:0/2565651472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c51350 0x55d411ff7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6c9ffff640 1 -- 172.21.15.16:0/2565651472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c51350 msgr2=0x55d411ff7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6c9ffff640 1 --2- 172.21.15.16:0/2565651472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c51350 0x55d411ff7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6c9ffff640 1 -- 172.21.15.16:0/2565651472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c28490 msgr2=0x55d411ff7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6c9ffff640 1 --2- 172.21.15.16:0/2565651472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c28490 0x55d411ff7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6c9ffff640 1 -- 172.21.15.16:0/2565651472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d411cbc930 con 0x55d411c30670 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.456+0000 7f6ca492d640 1 --2- 172.21.15.16:0/2565651472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c28490 0x55d411ff7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.457+0000 7f6c9ffff640 1 --2- 172.21.15.16:0/2565651472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d411ff8200 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c009230 tx=0x7f6c8c031bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb0eff643f090d9e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.457+0000 7f6c9dffb640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6c900b4c90 0x7f6c900b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.457+0000 7f6c8affd640 1 -- 172.21.15.16:0/2565651472 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c040070 con 0x55d411c30670 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.457+0000 7f6ca512e640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6c900b4c90 0x7f6c900b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.457+0000 7f6c8affd640 1 -- 172.21.15.16:0/2565651472 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c8c0316a0 con 0x55d411c30670 2024-08-28T19:21:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.457+0000 7f6c8affd640 1 -- 172.21.15.16:0/2565651472 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c035040 con 0x55d411c30670 2024-08-28T19:21:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.458+0000 7f6c9dffb640 1 -- 172.21.15.16:0/2996970232 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6c940e58a0 con 0x7f6ca0102880 2024-08-28T19:21:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.458+0000 7f6cad106740 1 -- 172.21.15.16:0/2565651472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 msgr2=0x55d411ff8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.458+0000 7f6ca512e640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6c900b4c90 0x7f6c900b71a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6c98005fd0 tx=0x7f6c98009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.458+0000 7f6cad106740 1 --2- 172.21.15.16:0/2565651472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d411ff8200 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c009230 tx=0x7f6c8c031bc0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.458+0000 7f6cad106740 1 -- 172.21.15.16:0/2565651472 shutdown_connections 2024-08-28T19:21:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.458+0000 7f6cad106740 1 --2- 172.21.15.16:0/2565651472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d411ff8200 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.458+0000 7f6cad106740 1 --2- 172.21.15.16:0/2565651472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c28490 0x55d411ff7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.458+0000 7f6cad106740 1 --2- 172.21.15.16:0/2565651472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c51350 0x55d411ff7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.458+0000 7f6cad106740 1 -- 172.21.15.16:0/2565651472 >> 172.21.15.16:0/2565651472 conn(0x55d411c45740 msgr2=0x55d411ff5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.458+0000 7f6cad106740 1 -- 172.21.15.16:0/2565651472 shutdown_connections 2024-08-28T19:21:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.459+0000 7f6cad106740 1 -- 172.21.15.16:0/2565651472 wait complete. 2024-08-28T19:21:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.459+0000 7f6cad106740 1 -- start start 2024-08-28T19:21:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.459+0000 7f6cad106740 1 Processor -- start 2024-08-28T19:21:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.459+0000 7f6cad106740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c28490 0x55d412002f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6cad106740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d412003490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6cad106740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d41200f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6cad106740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d411dbf890 con 0x55d411c51350 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6cad106740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d411ee3790 con 0x55d411c30670 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6cad106740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d412005f00 con 0x55d411c28490 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6cad106740 10 client.0 _ll_register_callbacks cb 0x55d411c68ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6ca492d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c28490 0x55d412002f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d412003490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6ca492d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c28490 0x55d412002f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:52130/0 (socket says 172.21.15.16:52130) 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6ca492d640 1 -- 172.21.15.16:0/4024181007 learned_addr learned my addr 172.21.15.16:0/4024181007 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6ca512e640 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d41200f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6c9ffff640 1 -- 172.21.15.16:0/4024181007 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c28490 msgr2=0x55d412002f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6c9ffff640 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c28490 0x55d412002f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6c9ffff640 1 -- 172.21.15.16:0/4024181007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 msgr2=0x55d41200f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6c9ffff640 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d41200f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6c9ffff640 1 -- 172.21.15.16:0/4024181007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d411f1af30 con 0x55d411c30670 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.460+0000 7f6ca492d640 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c28490 0x55d412002f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.461+0000 7f6ca512e640 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d41200f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.461+0000 7f6c9ffff640 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d412003490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c042000 tx=0x7f6c8c02a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2536133b6da7924 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.461+0000 7f6c7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.461+0000 7f6c7ffff640 1 -- 172.21.15.16:0/4024181007 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c035030 con 0x55d411c30670 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.461+0000 7f6cad106740 1 -- 172.21.15.16:0/4024181007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c8c0316a0 con 0x55d411c30670 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.461+0000 7f6c7ffff640 1 -- 172.21.15.16:0/4024181007 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6c8c004240 con 0x55d411c30670 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.461+0000 7f6c7ffff640 1 -- 172.21.15.16:0/4024181007 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6c8c040040 con 0x55d411c30670 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.461+0000 7f6cad106740 10 client.25562 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.461+0000 7f6cad106740 1 -- 172.21.15.16:0/4024181007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d412011d60 con 0x55d411c30670 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.462+0000 7f6c7ffff640 1 -- 172.21.15.16:0/4024181007 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c8c005db0 con 0x55d411c30670 2024-08-28T19:21:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.462+0000 7f6c7ffff640 1 -- 172.21.15.16:0/4024181007 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6c8c0037b0 con 0x55d411c30670 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.462+0000 7f6cad106740 10 client.25562 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.462+0000 7f6cad106740 10 client.25562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.462+0000 7f6cad106740 20 client.25562 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.462+0000 7f6cad106740 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d411c0fcd0 0x55d4120150c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.462+0000 7f6cad106740 4 client.25562 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.462+0000 7f6cad106740 1 -- 172.21.15.16:0/4024181007 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6c8c0316a0 con 0x55d411c0fcd0 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.462+0000 7f6ca492d640 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d411c0fcd0 0x55d4120150c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.463+0000 7f6c7ffff640 10 client.25562 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.463+0000 7f6ca492d640 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d411c0fcd0 0x55d4120150c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=381476a2aea070f1 server_cookie=4c83befc08d17ded in_seq=0 out_seq=0 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6c7ffff640 1 -- 172.21.15.16:0/4024181007 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f6c8c0316a0 con 0x55d411c0fcd0 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6c7ffff640 10 client.25562 handle_command_reply: tid=0 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 1 client.25562 shutdown 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 1 -- 172.21.15.16:0/4024181007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 msgr2=0x55d412003490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d412003490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c042000 tx=0x7f6c8c02a410 comp rx=0 tx=0).stop 2024-08-28T19:21:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 1 -- 172.21.15.16:0/4024181007 shutdown_connections 2024-08-28T19:21:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d41200f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d412003490 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d411c0fcd0 0x55d4120150c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 1 --2- 172.21.15.16:0/4024181007 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c28490 0x55d412002f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 1 -- 172.21.15.16:0/4024181007 >> 172.21.15.16:0/4024181007 conn(0x55d411c45740 msgr2=0x55d411ffa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6c7ffff640 0 client.25562 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 1 -- 172.21.15.16:0/4024181007 shutdown_connections 2024-08-28T19:21:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 1 -- 172.21.15.16:0/4024181007 wait complete. 2024-08-28T19:21:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.464+0000 7f6cad106740 20 client.25562 trim_cache size 0 max 16384 2024-08-28T19:21:55.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6cad106740 1 Processor -- start 2024-08-28T19:21:55.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6cad106740 1 -- start start 2024-08-28T19:21:55.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6cad106740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c28490 0x55d411ffaf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6cad106740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d412000140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6cad106740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d412000680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6cad106740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8400feb0 con 0x55d411c51350 2024-08-28T19:21:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6cad106740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c840119f0 con 0x55d411c30670 2024-08-28T19:21:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6cad106740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6c84010580 con 0x55d411c28490 2024-08-28T19:21:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6ca512e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d412000680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6ca512e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d412000680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51772/0 (socket says 172.21.15.16:51772) 2024-08-28T19:21:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6ca512e640 1 -- 172.21.15.16:0/2262824396 learned_addr learned my addr 172.21.15.16:0/2262824396 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.497+0000 7f6c9ffff640 1 --2- 172.21.15.16:0/2262824396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d412000140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.498+0000 7f6ca512e640 1 -- 172.21.15.16:0/2262824396 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c28490 msgr2=0x55d411ffaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.498+0000 7f6ca512e640 1 --2- 172.21.15.16:0/2262824396 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c28490 0x55d411ffaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.498+0000 7f6ca512e640 1 -- 172.21.15.16:0/2262824396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 msgr2=0x55d412000140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.498+0000 7f6ca512e640 1 --2- 172.21.15.16:0/2262824396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d412000140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.498+0000 7f6ca512e640 1 -- 172.21.15.16:0/2262824396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d412005690 con 0x55d411c51350 2024-08-28T19:21:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.498+0000 7f6ca512e640 1 --2- 172.21.15.16:0/2262824396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d412000680 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6c98012a20 tx=0x7f6c98015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1267f808886152 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.498+0000 7f6c8a7fc640 1 -- 172.21.15.16:0/2262824396 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9802b070 con 0x55d411c51350 2024-08-28T19:21:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.498+0000 7f6c8a7fc640 1 -- 172.21.15.16:0/2262824396 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c9801ec50 con 0x55d411c51350 2024-08-28T19:21:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.498+0000 7f6c8a7fc640 1 -- 172.21.15.16:0/2262824396 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98026650 con 0x55d411c51350 2024-08-28T19:21:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.499+0000 7f6cad106740 1 -- 172.21.15.16:0/2262824396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 msgr2=0x55d412000680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.499+0000 7f6cad106740 1 --2- 172.21.15.16:0/2262824396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d412000680 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6c98012a20 tx=0x7f6c98015000 comp rx=0 tx=0).stop 2024-08-28T19:21:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.499+0000 7f6cad106740 1 -- 172.21.15.16:0/2262824396 shutdown_connections 2024-08-28T19:21:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.499+0000 7f6cad106740 1 --2- 172.21.15.16:0/2262824396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d412000680 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.499+0000 7f6cad106740 1 --2- 172.21.15.16:0/2262824396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c30670 0x55d412000140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.499+0000 7f6cad106740 1 --2- 172.21.15.16:0/2262824396 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c28490 0x55d411ffaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.499+0000 7f6cad106740 1 -- 172.21.15.16:0/2262824396 >> 172.21.15.16:0/2262824396 conn(0x55d411c45740 msgr2=0x55d411ffa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.500+0000 7f6cad106740 1 -- 172.21.15.16:0/2262824396 shutdown_connections 2024-08-28T19:21:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.500+0000 7f6cad106740 1 -- 172.21.15.16:0/2262824396 wait complete. 2024-08-28T19:21:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.500+0000 7f6cad106740 1 -- start start 2024-08-28T19:21:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.500+0000 7f6cad106740 1 Processor -- start 2024-08-28T19:21:55.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.501+0000 7f6cad106740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c28490 0x55d4120739d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.501+0000 7f6cad106740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c30670 0x55d412073f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.501+0000 7f6cad106740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d41207e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.501+0000 7f6ca492d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c28490 0x55d4120739d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.501+0000 7f6ca492d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c28490 0x55d4120739d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49764/0 (socket says 172.21.15.16:49764) 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.501+0000 7f6ca492d640 1 -- 172.21.15.16:0/1119105706 learned_addr learned my addr 172.21.15.16:0/1119105706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.501+0000 7f6cad106740 1 -- 172.21.15.16:0/1119105706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c84011320 con 0x55d411c51350 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.501+0000 7f6cad106740 1 -- 172.21.15.16:0/1119105706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c84010c50 con 0x55d411c28490 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.501+0000 7f6cad106740 1 -- 172.21.15.16:0/1119105706 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8400de90 con 0x55d411c30670 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.501+0000 7f6cad106740 10 client.0 _ll_register_callbacks cb 0x7f6c940ed610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.502+0000 7f6ca512e640 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d41207e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.502+0000 7f6ca492d640 1 -- 172.21.15.16:0/1119105706 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c30670 msgr2=0x55d412073f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.502+0000 7f6ca492d640 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c30670 0x55d412073f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.502+0000 7f6ca492d640 1 -- 172.21.15.16:0/1119105706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 msgr2=0x55d41207e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:55.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.502+0000 7f6ca492d640 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d41207e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.502+0000 7f6ca492d640 1 -- 172.21.15.16:0/1119105706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d41207ee10 con 0x55d411c28490 2024-08-28T19:21:55.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.502+0000 7f6ca512e640 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d41207e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:55.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.502+0000 7f6ca492d640 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c28490 0x55d4120739d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6c940a7db0 tx=0x7f6c94018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b3f9367a8d09abb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:55.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.502+0000 7f6c7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6c7ffff640 1 -- 172.21.15.16:0/1119105706 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c94010720 con 0x55d411c28490 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6c7ffff640 1 -- 172.21.15.16:0/1119105706 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6c940aaf90 con 0x55d411c28490 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6cad106740 1 -- 172.21.15.16:0/1119105706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c9801ec50 con 0x55d411c28490 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6c7ffff640 1 -- 172.21.15.16:0/1119105706 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6c940ab840 con 0x55d411c28490 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6cad106740 10 client.25568 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6cad106740 1 -- 172.21.15.16:0/1119105706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d4120800d0 con 0x55d411c28490 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6c7ffff640 1 -- 172.21.15.16:0/1119105706 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c9801ec50 con 0x55d411c28490 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6c7ffff640 1 -- 172.21.15.16:0/1119105706 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6c940a76a0 con 0x55d411c28490 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6cad106740 10 client.25568 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6cad106740 10 client.25568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6cad106740 20 client.25568 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:55.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6cad106740 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d411be5080 0x55d412082840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:55.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6cad106740 4 client.25568 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:21:55.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.503+0000 7f6cad106740 1 -- 172.21.15.16:0/1119105706 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d411f6f540 con 0x55d411be5080 2024-08-28T19:21:55.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.504+0000 7f6c9ffff640 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d411be5080 0x55d412082840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:55.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.504+0000 7f6c7ffff640 10 client.25568 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:55.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.504+0000 7f6c9ffff640 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d411be5080 0x55d412082840 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20496db8e85fc53f server_cookie=15a989cc048590d7 in_seq=0 out_seq=0 2024-08-28T19:21:55.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.506+0000 7f6c7ffff640 1 -- 172.21.15.16:0/1119105706 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d411f6f540 con 0x55d411be5080 2024-08-28T19:21:55.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.506+0000 7f6c7ffff640 10 client.25568 handle_command_reply: tid=0 2024-08-28T19:21:55.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.506+0000 7f6cad106740 1 client.25568 shutdown 2024-08-28T19:21:55.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.506+0000 7f6cad106740 1 -- 172.21.15.16:0/1119105706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c28490 msgr2=0x55d4120739d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.506+0000 7f6cad106740 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c28490 0x55d4120739d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6c940a7db0 tx=0x7f6c94018420 comp rx=0 tx=0).stop 2024-08-28T19:21:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.507+0000 7f6cad106740 1 -- 172.21.15.16:0/1119105706 shutdown_connections 2024-08-28T19:21:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.507+0000 7f6cad106740 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d411c51350 0x55d41207e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.507+0000 7f6cad106740 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d411be5080 0x55d412082840 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.507+0000 7f6cad106740 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d411c30670 0x55d412073f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.507+0000 7f6cad106740 1 --2- 172.21.15.16:0/1119105706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d411c28490 0x55d4120739d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.507+0000 7f6cad106740 1 -- 172.21.15.16:0/1119105706 >> 172.21.15.16:0/1119105706 conn(0x55d411c45740 msgr2=0x55d411ff87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.507+0000 7f6c7ffff640 0 client.25568 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.507+0000 7f6cad106740 1 -- 172.21.15.16:0/1119105706 shutdown_connections 2024-08-28T19:21:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.507+0000 7f6cad106740 1 -- 172.21.15.16:0/1119105706 wait complete. 2024-08-28T19:21:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.507+0000 7f6cad106740 20 client.25568 trim_cache size 0 max 16384 2024-08-28T19:21:55.511 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:21:55.511 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:21:55.511 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f", 2024-08-28T19:21:55.511 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:21:55.511 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:21:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 -- 172.21.15.16:0/2996970232 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6c900b4c90 msgr2=0x7f6c900b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6c900b4c90 0x7f6c900b71a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6c98005fd0 tx=0x7f6c98009600 comp rx=0 tx=0).stop 2024-08-28T19:21:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 -- 172.21.15.16:0/2996970232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca0102880 msgr2=0x7f6ca0199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca0102880 0x7f6ca0199140 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400e9f0 tx=0x7f6c9400eeb0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 -- 172.21.15.16:0/2996970232 shutdown_connections 2024-08-28T19:21:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0108ac0 0x7f6ca019e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ca0108170 0x7f6ca01996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6c900b4c90 0x7f6c900b71a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 --2- 172.21.15.16:0/2996970232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca0102880 0x7f6ca0199140 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 -- 172.21.15.16:0/2996970232 >> 172.21.15.16:0/2996970232 conn(0x7f6ca0079530 msgr2=0x7f6ca0106200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 -- 172.21.15.16:0/2996970232 shutdown_connections 2024-08-28T19:21:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:55.508+0000 7f6ca6b91640 1 -- 172.21.15.16:0/2996970232 wait complete. 2024-08-28T19:21:56.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:21:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:21:56.929 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:56 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:21:56.929 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:56 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:56.929 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:56 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:21:56.929 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:56 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:56.929 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:56 smithi016 ceph-mon[18556]: pgmap v366: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-08-28T19:21:57.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:56 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:21:57.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:56 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:57.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:56 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:21:57.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:56 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:57.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:56 smithi028 ceph-mon[21769]: pgmap v366: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-08-28T19:21:57.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.075+0000 7f057f635640 1 -- 172.21.15.16:0/1560102570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578108150 msgr2=0x7f0578108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:57.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.075+0000 7f057f635640 1 --2- 172.21.15.16:0/1560102570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578108150 0x7f0578108560 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0568009900 tx=0x7f056802a410 comp rx=0 tx=0).stop 2024-08-28T19:21:57.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.075+0000 7f057f635640 1 -- 172.21.15.16:0/1560102570 shutdown_connections 2024-08-28T19:21:57.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.075+0000 7f057f635640 1 --2- 172.21.15.16:0/1560102570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0578108aa0 0x7f057810f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.075+0000 7f057f635640 1 --2- 172.21.15.16:0/1560102570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578108150 0x7f0578108560 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.075+0000 7f057f635640 1 --2- 172.21.15.16:0/1560102570 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0578102810 0x7f0578102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.075+0000 7f057f635640 1 -- 172.21.15.16:0/1560102570 >> 172.21.15.16:0/1560102570 conn(0x7f05780794d0 msgr2=0x7f05780798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:57.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.076+0000 7f057f635640 1 -- 172.21.15.16:0/1560102570 shutdown_connections 2024-08-28T19:21:57.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.076+0000 7f057f635640 1 -- 172.21.15.16:0/1560102570 wait complete. 2024-08-28T19:21:57.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.077+0000 7f057f635640 1 Processor -- start 2024-08-28T19:21:57.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.077+0000 7f057f635640 1 -- start start 2024-08-28T19:21:57.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.078+0000 7f057f635640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0578102810 0x7f05781991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.078+0000 7f057f635640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0578108150 0x7f0578199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.078+0000 7f057f635640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578108aa0 0x7f057819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.078+0000 7f057f635640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0578112720 con 0x7f0578108aa0 2024-08-28T19:21:57.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.078+0000 7f057f635640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05781125a0 con 0x7f0578108150 2024-08-28T19:21:57.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.078+0000 7f057f635640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f05781128a0 con 0x7f0578102810 2024-08-28T19:21:57.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.078+0000 7f057cbd0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0578108150 0x7f0578199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.078+0000 7f057d3d1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0578102810 0x7f05781991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.078+0000 7f057dbd2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578108aa0 0x7f057819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.078+0000 7f057d3d1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0578102810 0x7f05781991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:52152/0 (socket says 172.21.15.16:52152) 2024-08-28T19:21:57.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.078+0000 7f057d3d1640 1 -- 172.21.15.16:0/3711001106 learned_addr learned my addr 172.21.15.16:0/3711001106 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:57.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.079+0000 7f057d3d1640 1 -- 172.21.15.16:0/3711001106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0578108150 msgr2=0x7f0578199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:57.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.079+0000 7f057d3d1640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0578108150 0x7f0578199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.079+0000 7f057d3d1640 1 -- 172.21.15.16:0/3711001106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578108aa0 msgr2=0x7f057819e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:57.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.079+0000 7f057d3d1640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578108aa0 0x7f057819e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.079+0000 7f057d3d1640 1 -- 172.21.15.16:0/3711001106 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f057819f120 con 0x7f0578102810 2024-08-28T19:21:57.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.079+0000 7f057cbd0640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0578108150 0x7f0578199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:57.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.079+0000 7f057dbd2640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578108aa0 0x7f057819e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:21:57.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.079+0000 7f057d3d1640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0578102810 0x7f05781991b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f057400e9f0 tx=0x7f057400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:57.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.080+0000 7f056e7fc640 1 -- 172.21.15.16:0/3711001106 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057400cd80 con 0x7f0578102810 2024-08-28T19:21:57.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.080+0000 7f056e7fc640 1 -- 172.21.15.16:0/3711001106 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05740108e0 con 0x7f0578102810 2024-08-28T19:21:57.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.080+0000 7f056e7fc640 1 -- 172.21.15.16:0/3711001106 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0574018980 con 0x7f0578102810 2024-08-28T19:21:57.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.080+0000 7f057f635640 1 -- 172.21.15.16:0/3711001106 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f057819f410 con 0x7f0578102810 2024-08-28T19:21:57.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.080+0000 7f057f635640 1 -- 172.21.15.16:0/3711001106 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f057806b920 con 0x7f0578102810 2024-08-28T19:21:57.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.084+0000 7f056e7fc640 1 -- 172.21.15.16:0/3711001106 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 23) ==== 152651+0+0 (secure 0 0 0) 0x7f0574010430 con 0x7f0578102810 2024-08-28T19:21:57.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.084+0000 7f0585baa740 1 Processor -- start 2024-08-28T19:21:57.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.084+0000 7f0585baa740 1 -- start start 2024-08-28T19:21:57.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.085+0000 7f0585baa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 0x56220670b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.085+0000 7f0585baa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x56220670b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.085+0000 7f056e7fc640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05640b6e20 0x7f05640b9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.085+0000 7f0585baa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x56220670bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.085+0000 7f0585baa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562206682630 con 0x562206345390 2024-08-28T19:21:57.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.085+0000 7f0585baa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562206671bc0 con 0x562206366070 2024-08-28T19:21:57.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.085+0000 7f057dbd2640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05640b6e20 0x7f05640b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.085+0000 7f056e7fc640 1 -- 172.21.15.16:0/3711001106 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f05740e5b70 con 0x7f0578102810 2024-08-28T19:21:57.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.086+0000 7f057dbd2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 0x56220670b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.086+0000 7f057dbd2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 0x56220670b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49788/0 (socket says 172.21.15.16:49788) 2024-08-28T19:21:57.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.085+0000 7f057d3d1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x56220670b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.086+0000 7f057dbd2640 1 -- 172.21.15.16:0/3241359758 learned_addr learned my addr 172.21.15.16:0/3241359758 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.085+0000 7f0585baa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562206675b70 con 0x56220633d1b0 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.086+0000 7f057cbd0640 1 --2- 172.21.15.16:0/3241359758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x56220670bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.086+0000 7f057dbd2640 1 -- 172.21.15.16:0/3241359758 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 msgr2=0x56220670b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.086+0000 7f057dbd2640 1 --2- 172.21.15.16:0/3241359758 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x56220670b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.086+0000 7f057dbd2640 1 -- 172.21.15.16:0/3241359758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 msgr2=0x56220670bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.087+0000 7f057dbd2640 1 --2- 172.21.15.16:0/3241359758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x56220670bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.087+0000 7f057dbd2640 1 -- 172.21.15.16:0/3241359758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622063d1650 con 0x562206366070 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.087+0000 7f057cbd0640 1 --2- 172.21.15.16:0/3241359758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x56220670bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.087+0000 7f057dbd2640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05640b6e20 0x7f05640b9330 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0570014c00 tx=0x7f0570009e10 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.087+0000 7f057dbd2640 1 --2- 172.21.15.16:0/3241359758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 0x56220670b2c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f05700146d0 tx=0x7f057001b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f88453158ccb6ce server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.087+0000 7f05637fe640 1 -- 172.21.15.16:0/3241359758 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0570024070 con 0x562206366070 2024-08-28T19:21:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.088+0000 7f05637fe640 1 -- 172.21.15.16:0/3241359758 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0570003e40 con 0x562206366070 2024-08-28T19:21:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.088+0000 7f05637fe640 1 -- 172.21.15.16:0/3241359758 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0570020a80 con 0x562206366070 2024-08-28T19:21:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.089+0000 7f0585baa740 1 -- 172.21.15.16:0/3241359758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 msgr2=0x56220670b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.089+0000 7f0585baa740 1 --2- 172.21.15.16:0/3241359758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 0x56220670b2c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f05700146d0 tx=0x7f057001b000 comp rx=0 tx=0).stop 2024-08-28T19:21:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.089+0000 7f0585baa740 1 -- 172.21.15.16:0/3241359758 shutdown_connections 2024-08-28T19:21:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.089+0000 7f0585baa740 1 --2- 172.21.15.16:0/3241359758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x56220670bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.089+0000 7f0585baa740 1 --2- 172.21.15.16:0/3241359758 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x56220670b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.089+0000 7f0585baa740 1 --2- 172.21.15.16:0/3241359758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 0x56220670b2c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.089+0000 7f0585baa740 1 -- 172.21.15.16:0/3241359758 >> 172.21.15.16:0/3241359758 conn(0x56220635a460 msgr2=0x5622067091b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.089+0000 7f0585baa740 1 -- 172.21.15.16:0/3241359758 shutdown_connections 2024-08-28T19:21:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.089+0000 7f0585baa740 1 -- 172.21.15.16:0/3241359758 wait complete. 2024-08-28T19:21:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.090+0000 7f0585baa740 1 -- start start 2024-08-28T19:21:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.090+0000 7f0585baa740 1 Processor -- start 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.090+0000 7f0585baa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x562206716ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.091+0000 7f0585baa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x562206717050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.091+0000 7f0585baa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 0x562206722d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.091+0000 7f0585baa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5622064d4700 con 0x562206345390 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.091+0000 7f0585baa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5622065f8700 con 0x562206366070 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.091+0000 7f0585baa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562206719a40 con 0x56220633d1b0 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.091+0000 7f057d3d1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x562206716ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.091+0000 7f057cbd0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x562206717050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.091+0000 7f057dbd2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 0x562206722d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.091+0000 7f057dbd2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 0x562206722d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49790/0 (socket says 172.21.15.16:49790) 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.091+0000 7f057dbd2640 1 -- 172.21.15.16:0/1632029588 learned_addr learned my addr 172.21.15.16:0/1632029588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.091+0000 7f0585baa740 10 client.0 _ll_register_callbacks cb 0x562206627dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.092+0000 7f057cbd0640 1 -- 172.21.15.16:0/1632029588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 msgr2=0x562206716ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.092+0000 7f057cbd0640 1 --2- 172.21.15.16:0/1632029588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x562206716ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.092+0000 7f057cbd0640 1 -- 172.21.15.16:0/1632029588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 msgr2=0x562206722d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.092+0000 7f057cbd0640 1 --2- 172.21.15.16:0/1632029588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 0x562206722d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.092+0000 7f057cbd0640 1 -- 172.21.15.16:0/1632029588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622066318b0 con 0x562206345390 2024-08-28T19:21:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.092+0000 7f057cbd0640 1 --2- 172.21.15.16:0/1632029588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x562206717050 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f056802a930 tx=0x7f05680053b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938ee587c6c55437 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.092+0000 7f0560ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:21:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.092+0000 7f0560ff9640 1 -- 172.21.15.16:0/1632029588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0568035040 con 0x562206345390 2024-08-28T19:21:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.093+0000 7f0560ff9640 1 -- 172.21.15.16:0/1632029588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0568004770 con 0x562206345390 2024-08-28T19:21:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.093+0000 7f0585baa740 1 -- 172.21.15.16:0/1632029588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562206684580 con 0x562206345390 2024-08-28T19:21:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.093+0000 7f0560ff9640 1 -- 172.21.15.16:0/1632029588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f056804f050 con 0x562206345390 2024-08-28T19:21:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.093+0000 7f0585baa740 10 client.15939 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.093+0000 7f0585baa740 1 -- 172.21.15.16:0/1632029588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562206725880 con 0x562206345390 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.093+0000 7f0560ff9640 1 -- 172.21.15.16:0/1632029588 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0568004e20 con 0x562206345390 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.095+0000 7f0560ff9640 1 -- 172.21.15.16:0/1632029588 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0568044020 con 0x562206345390 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.095+0000 7f0585baa740 10 client.15939 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.095+0000 7f0585baa740 10 client.15939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.095+0000 7f0585baa740 20 client.15939 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.095+0000 7f0585baa740 1 --2- 172.21.15.16:0/1632029588 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5622063249f0 0x562206728d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.095+0000 7f0585baa740 4 client.15939 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.095+0000 7f0585baa740 1 -- 172.21.15.16:0/1632029588 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562206684580 con 0x5622063249f0 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.095+0000 7f057d3d1640 1 --2- 172.21.15.16:0/1632029588 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5622063249f0 0x562206728d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.096+0000 7f057d3d1640 1 --2- 172.21.15.16:0/1632029588 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5622063249f0 0x562206728d50 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75fd2236a3af8886 server_cookie=6d8f154fe348a3f in_seq=0 out_seq=0 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.096+0000 7f0560ff9640 10 client.15939 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.096+0000 7f0560ff9640 1 -- 172.21.15.16:0/1632029588 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562206684580 con 0x5622063249f0 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.096+0000 7f0560ff9640 10 client.15939 handle_command_reply: tid=0 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.096+0000 7f0585baa740 1 client.15939 shutdown 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0585baa740 1 -- 172.21.15.16:0/1632029588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 msgr2=0x562206717050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0585baa740 1 --2- 172.21.15.16:0/1632029588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x562206717050 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f056802a930 tx=0x7f05680053b0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0585baa740 1 -- 172.21.15.16:0/1632029588 shutdown_connections 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0585baa740 1 --2- 172.21.15.16:0/1632029588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206366070 0x562206722d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0585baa740 1 --2- 172.21.15.16:0/1632029588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x562206717050 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0585baa740 1 --2- 172.21.15.16:0/1632029588 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5622063249f0 0x562206728d50 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0585baa740 1 --2- 172.21.15.16:0/1632029588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x562206716ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0585baa740 1 -- 172.21.15.16:0/1632029588 >> 172.21.15.16:0/1632029588 conn(0x56220635a460 msgr2=0x56220670e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0560ff9640 0 client.15939 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0585baa740 1 -- 172.21.15.16:0/1632029588 shutdown_connections 2024-08-28T19:21:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0585baa740 1 -- 172.21.15.16:0/1632029588 wait complete. 2024-08-28T19:21:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.097+0000 7f0585baa740 20 client.15939 trim_cache size 0 max 16384 2024-08-28T19:21:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:56 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:21:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:56 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:56 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:21:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:56 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:21:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:56 smithi115 ceph-mon[22184]: pgmap v366: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-08-28T19:21:57.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.133+0000 7f0585baa740 1 Processor -- start 2024-08-28T19:21:57.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.133+0000 7f0585baa740 1 -- start start 2024-08-28T19:21:57.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.133+0000 7f0585baa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56220633d1b0 0x56220670ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.133+0000 7f0585baa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x562206713cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.133+0000 7f0585baa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562206366070 0x562206714200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.133+0000 7f0585baa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05781ae930 con 0x562206345390 2024-08-28T19:21:57.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.133+0000 7f0585baa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05781b0470 con 0x56220633d1b0 2024-08-28T19:21:57.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.133+0000 7f0585baa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f05781af000 con 0x562206366070 2024-08-28T19:21:57.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057dbd2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562206366070 0x562206714200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057d3d1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56220633d1b0 0x56220670ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057dbd2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562206366070 0x562206714200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:52170/0 (socket says 172.21.15.16:52170) 2024-08-28T19:21:57.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057dbd2640 1 -- 172.21.15.16:0/1921011588 learned_addr learned my addr 172.21.15.16:0/1921011588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:57.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057cbd0640 1 --2- 172.21.15.16:0/1921011588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x562206713cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057d3d1640 1 -- 172.21.15.16:0/1921011588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562206366070 msgr2=0x562206714200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:57.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057d3d1640 1 --2- 172.21.15.16:0/1921011588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562206366070 0x562206714200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057d3d1640 1 -- 172.21.15.16:0/1921011588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 msgr2=0x562206713cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:57.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057d3d1640 1 --2- 172.21.15.16:0/1921011588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x562206713cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057d3d1640 1 -- 172.21.15.16:0/1921011588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56220670dac0 con 0x56220633d1b0 2024-08-28T19:21:57.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057dbd2640 1 --2- 172.21.15.16:0/1921011588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562206366070 0x562206714200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057cbd0640 1 --2- 172.21.15.16:0/1921011588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x562206713cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f057d3d1640 1 --2- 172.21.15.16:0/1921011588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56220633d1b0 0x56220670ead0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f05740e3060 tx=0x7f05740a7e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6012f78afa291853 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f0562ffd640 1 -- 172.21.15.16:0/1921011588 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05740b0820 con 0x56220633d1b0 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f0562ffd640 1 -- 172.21.15.16:0/1921011588 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05740b52d0 con 0x56220633d1b0 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.134+0000 7f0562ffd640 1 -- 172.21.15.16:0/1921011588 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05740b3ba0 con 0x56220633d1b0 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.136+0000 7f0585baa740 1 -- 172.21.15.16:0/1921011588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56220633d1b0 msgr2=0x56220670ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.136+0000 7f0585baa740 1 --2- 172.21.15.16:0/1921011588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56220633d1b0 0x56220670ead0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f05740e3060 tx=0x7f05740a7e70 comp rx=0 tx=0).stop 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.136+0000 7f0585baa740 1 -- 172.21.15.16:0/1921011588 shutdown_connections 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.136+0000 7f0585baa740 1 --2- 172.21.15.16:0/1921011588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562206366070 0x562206714200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.136+0000 7f0585baa740 1 --2- 172.21.15.16:0/1921011588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206345390 0x562206713cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.136+0000 7f0585baa740 1 --2- 172.21.15.16:0/1921011588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56220633d1b0 0x56220670ead0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.136+0000 7f0585baa740 1 -- 172.21.15.16:0/1921011588 >> 172.21.15.16:0/1921011588 conn(0x56220635a460 msgr2=0x56220670e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:57.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.136+0000 7f0585baa740 1 -- 172.21.15.16:0/1921011588 shutdown_connections 2024-08-28T19:21:57.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.136+0000 7f0585baa740 1 -- 172.21.15.16:0/1921011588 wait complete. 2024-08-28T19:21:57.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.136+0000 7f0585baa740 1 -- start start 2024-08-28T19:21:57.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.136+0000 7f0585baa740 1 Processor -- start 2024-08-28T19:21:57.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f0585baa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x56220671b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f0585baa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206345390 0x56220671b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f0585baa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206366070 0x562206792620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f057cbd0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206345390 0x56220671b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f057cbd0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206345390 0x56220671b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49808/0 (socket says 172.21.15.16:49808) 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f057dbd2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206366070 0x562206792620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f057d3d1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x56220671b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f057dbd2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206366070 0x562206792620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51808/0 (socket says 172.21.15.16:51808) 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f0585baa740 1 -- 172.21.15.16:0/3343744988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05781afda0 con 0x562206366070 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f057cbd0640 1 -- 172.21.15.16:0/3343744988 learned_addr learned my addr 172.21.15.16:0/3343744988 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f0585baa740 1 -- 172.21.15.16:0/3343744988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05781af6d0 con 0x562206345390 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.137+0000 7f0585baa740 1 -- 172.21.15.16:0/3343744988 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f05781ac910 con 0x56220633d1b0 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.138+0000 7f0585baa740 10 client.0 _ll_register_callbacks cb 0x7f0568009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.138+0000 7f057d3d1640 1 -- 172.21.15.16:0/3343744988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206345390 msgr2=0x56220671b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.138+0000 7f057d3d1640 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206345390 0x56220671b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.138+0000 7f057d3d1640 1 -- 172.21.15.16:0/3343744988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206366070 msgr2=0x562206792620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:21:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.138+0000 7f057d3d1640 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206366070 0x562206792620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.138+0000 7f057d3d1640 1 -- 172.21.15.16:0/3343744988 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562206792c30 con 0x56220633d1b0 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.138+0000 7f057dbd2640 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206366070 0x562206792620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.138+0000 7f057d3d1640 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x56220671b470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f05740e3060 tx=0x7f05740ac130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83f6c364eb533b74 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.138+0000 7f0560ff9640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.138+0000 7f0560ff9640 1 -- 172.21.15.16:0/3343744988 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0574097200 con 0x56220633d1b0 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.139+0000 7f0560ff9640 1 -- 172.21.15.16:0/3343744988 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f05740b0390 con 0x56220633d1b0 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.139+0000 7f0585baa740 1 -- 172.21.15.16:0/3343744988 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05740b52d0 con 0x56220633d1b0 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.139+0000 7f0560ff9640 1 -- 172.21.15.16:0/3343744988 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f05740b05d0 con 0x56220633d1b0 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.139+0000 7f0585baa740 10 client.25489 fetch_fsmap learned FSMap version 29 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.139+0000 7f0585baa740 1 -- 172.21.15.16:0/3343744988 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562206793ec0 con 0x56220633d1b0 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.139+0000 7f0560ff9640 1 -- 172.21.15.16:0/3343744988 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05740b52d0 con 0x56220633d1b0 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.140+0000 7f0560ff9640 1 -- 172.21.15.16:0/3343744988 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f05740b0070 con 0x56220633d1b0 2024-08-28T19:21:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.140+0000 7f0585baa740 10 client.25489 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:21:57.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.140+0000 7f0585baa740 10 client.25489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:21:57.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.140+0000 7f0585baa740 20 client.25489 populate_metadata read hostname 'smithi016' 2024-08-28T19:21:57.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.141+0000 7f0585baa740 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5622062f9da0 0x5622067965e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:21:57.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.141+0000 7f0585baa740 4 client.25489 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:21:57.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.141+0000 7f0585baa740 1 -- 172.21.15.16:0/3343744988 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562206680020 con 0x5622062f9da0 2024-08-28T19:21:57.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.141+0000 7f057cbd0640 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5622062f9da0 0x5622067965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:21:57.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.141+0000 7f0560ff9640 10 client.25489 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:57.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.141+0000 7f057cbd0640 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5622062f9da0 0x5622067965e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cf334eeb2646c6 server_cookie=e9550bda77a9fe23 in_seq=0 out_seq=0 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0560ff9640 1 -- 172.21.15.16:0/3343744988 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+854 (crc 0 0 0) 0x562206680020 con 0x5622062f9da0 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0560ff9640 10 client.25489 handle_command_reply: tid=0 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 1 client.25489 shutdown 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 1 -- 172.21.15.16:0/3343744988 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 msgr2=0x56220671b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x56220671b470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f05740e3060 tx=0x7f05740ac130 comp rx=0 tx=0).stop 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 1 -- 172.21.15.16:0/3343744988 shutdown_connections 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562206366070 0x562206792620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562206345390 0x56220671b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5622062f9da0 0x5622067965e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 1 --2- 172.21.15.16:0/3343744988 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56220633d1b0 0x56220671b470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0560ff9640 0 client.25489 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:21:57.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 1 -- 172.21.15.16:0/3343744988 >> 172.21.15.16:0/3343744988 conn(0x56220635a460 msgr2=0x56220670c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:57.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 1 -- 172.21.15.16:0/3343744988 shutdown_connections 2024-08-28T19:21:57.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 1 -- 172.21.15.16:0/3343744988 wait complete. 2024-08-28T19:21:57.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.142+0000 7f0585baa740 20 client.25489 trim_cache size 0 max 16384 2024-08-28T19:21:57.148 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:21:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:21:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:21:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:21:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:21:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f", 2024-08-28T19:21:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:21:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:21:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:21:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:21:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:21:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:21:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:21:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:21:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:21:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:21:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:21:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:21:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:21:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:21:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:21:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:21:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:21:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:21:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:21:57.150 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.143+0000 7f057f635640 1 -- 172.21.15.16:0/3711001106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05640b6e20 msgr2=0x7f05640b9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.143+0000 7f057f635640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05640b6e20 0x7f05640b9330 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0570014c00 tx=0x7f0570009e10 comp rx=0 tx=0).stop 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.143+0000 7f057f635640 1 -- 172.21.15.16:0/3711001106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0578102810 msgr2=0x7f05781991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.143+0000 7f057f635640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0578102810 0x7f05781991b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f057400e9f0 tx=0x7f057400eeb0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.144+0000 7f057f635640 1 -- 172.21.15.16:0/3711001106 shutdown_connections 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.144+0000 7f057f635640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578108aa0 0x7f057819e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.144+0000 7f057f635640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05640b6e20 0x7f05640b9330 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.144+0000 7f057f635640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0578108150 0x7f0578199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.144+0000 7f057f635640 1 --2- 172.21.15.16:0/3711001106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0578102810 0x7f05781991b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.144+0000 7f057f635640 1 -- 172.21.15.16:0/3711001106 >> 172.21.15.16:0/3711001106 conn(0x7f05780794d0 msgr2=0x7f05781041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.144+0000 7f057f635640 1 -- 172.21.15.16:0/3711001106 shutdown_connections 2024-08-28T19:21:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:21:57.144+0000 7f057f635640 1 -- 172.21.15.16:0/3711001106 wait complete. 2024-08-28T19:21:58.025 INFO:tasks.cephfs.filesystem:scrub status for tag:fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f - {'path': '/', 'tag': 'fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f', 'options': 'recursive,force'} 2024-08-28T19:21:59.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:21:58 smithi016 ceph-mon[18556]: pgmap v367: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-08-28T19:21:59.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:21:58 smithi028 ceph-mon[21769]: pgmap v367: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-08-28T19:21:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:21:58 smithi115 ceph-mon[22184]: pgmap v367: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-08-28T19:22:01.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:00 smithi016 ceph-mon[18556]: pgmap v368: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 9 op/s 2024-08-28T19:22:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:00 smithi028 ceph-mon[21769]: pgmap v368: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 9 op/s 2024-08-28T19:22:01.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:00 smithi115 ceph-mon[22184]: pgmap v368: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 9 op/s 2024-08-28T19:22:01.500 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:22:01 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:22:00.999+0000 7f9902779640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-28T19:22:01.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:22:01 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:22:00.999+0000 7f9902779640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-28T19:22:01.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:22:01 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:22:00.999+0000 7f9902779640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-28T19:22:01.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:22:01 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:22:00.999+0000 7f9902779640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-28T19:22:01.501 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:22:01 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x[18784]: 2024-08-28T19:22:00.999+0000 7f9902779640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-28T19:22:03.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:02 smithi016 ceph-mon[18556]: pgmap v369: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 25 op/s 2024-08-28T19:22:03.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:02 smithi016 ceph-mon[18556]: mgrmap e24: x(active, since 9m), standbys: z, y 2024-08-28T19:22:03.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:02 smithi028 ceph-mon[21769]: pgmap v369: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 25 op/s 2024-08-28T19:22:03.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:02 smithi028 ceph-mon[21769]: mgrmap e24: x(active, since 9m), standbys: z, y 2024-08-28T19:22:03.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:02 smithi115 ceph-mon[22184]: pgmap v369: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 25 op/s 2024-08-28T19:22:03.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:02 smithi115 ceph-mon[22184]: mgrmap e24: x(active, since 9m), standbys: z, y 2024-08-28T19:22:05.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:04 smithi016 ceph-mon[18556]: pgmap v370: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-08-28T19:22:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:04 smithi028 ceph-mon[21769]: pgmap v370: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-08-28T19:22:05.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:04 smithi115 ceph-mon[22184]: pgmap v370: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-08-28T19:22:07.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:06 smithi016 ceph-mon[18556]: pgmap v371: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-08-28T19:22:07.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:06 smithi028 ceph-mon[21769]: pgmap v371: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-08-28T19:22:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:06 smithi115 ceph-mon[22184]: pgmap v371: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-08-28T19:22:09.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:08 smithi016 ceph-mon[18556]: pgmap v372: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-08-28T19:22:09.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:08 smithi028 ceph-mon[21769]: pgmap v372: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-08-28T19:22:09.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:08 smithi115 ceph-mon[22184]: pgmap v372: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-08-28T19:22:11.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:10 smithi016 ceph-mon[18556]: pgmap v373: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-08-28T19:22:11.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:10 smithi028 ceph-mon[21769]: pgmap v373: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-08-28T19:22:11.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:10 smithi115 ceph-mon[22184]: pgmap v373: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-08-28T19:22:13.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:12 smithi016 ceph-mon[18556]: pgmap v374: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-08-28T19:22:13.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:12 smithi028 ceph-mon[21769]: pgmap v374: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-08-28T19:22:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:12 smithi115 ceph-mon[22184]: pgmap v374: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-08-28T19:22:15.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:14 smithi016 ceph-mon[18556]: pgmap v375: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-08-28T19:22:15.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:14 smithi028 ceph-mon[21769]: pgmap v375: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-08-28T19:22:15.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:14 smithi115 ceph-mon[22184]: pgmap v375: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-08-28T19:22:16.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:22:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:17.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:16 smithi016 ceph-mon[18556]: pgmap v376: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-08-28T19:22:17.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:16 smithi028 ceph-mon[21769]: pgmap v376: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-08-28T19:22:17.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:16 smithi115 ceph-mon[22184]: pgmap v376: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-08-28T19:22:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.494+0000 7f4831e45640 1 -- 172.21.15.16:0/859184724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482c10dc00 msgr2=0x7f482c1100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.494+0000 7f4831e45640 1 --2- 172.21.15.16:0/859184724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482c10dc00 0x7f482c1100d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4824009ca0 tx=0x7f482402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:22:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.494+0000 7f4831e45640 1 -- 172.21.15.16:0/859184724 shutdown_connections 2024-08-28T19:22:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.494+0000 7f4831e45640 1 --2- 172.21.15.16:0/859184724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482c10dc00 0x7f482c1100d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.494+0000 7f4831e45640 1 --2- 172.21.15.16:0/859184724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c105120 0x7f482c10d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.494+0000 7f4831e45640 1 --2- 172.21.15.16:0/859184724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f482c1047d0 0x7f482c104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.494+0000 7f4831e45640 1 -- 172.21.15.16:0/859184724 >> 172.21.15.16:0/859184724 conn(0x7f482c079530 msgr2=0x7f482c079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.495+0000 7f4831e45640 1 -- 172.21.15.16:0/859184724 shutdown_connections 2024-08-28T19:22:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.495+0000 7f4831e45640 1 -- 172.21.15.16:0/859184724 wait complete. 2024-08-28T19:22:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.495+0000 7f4831e45640 1 Processor -- start 2024-08-28T19:22:17.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.495+0000 7f4831e45640 1 -- start start 2024-08-28T19:22:17.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f4831e45640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482c1047d0 0x7f482c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:17.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f4831e45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c105120 0x7f482c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:17.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f4831e45640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f482c10dc00 0x7f482c19fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:17.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f4831e45640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f482c113860 con 0x7f482c1047d0 2024-08-28T19:22:17.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f4831e45640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f482c1136e0 con 0x7f482c105120 2024-08-28T19:22:17.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f4831e45640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f482c1139e0 con 0x7f482c10dc00 2024-08-28T19:22:17.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f482b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482c1047d0 0x7f482c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:17.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f482b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482c1047d0 0x7f482c19a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56258/0 (socket says 172.21.15.16:56258) 2024-08-28T19:22:17.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f482b7fe640 1 -- 172.21.15.16:0/4265652389 learned_addr learned my addr 172.21.15.16:0/4265652389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:17.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f4822ffd640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c105120 0x7f482c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f482bfff640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f482c10dc00 0x7f482c19fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f482b7fe640 1 -- 172.21.15.16:0/4265652389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f482c10dc00 msgr2=0x7f482c19fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f482b7fe640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f482c10dc00 0x7f482c19fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f482b7fe640 1 -- 172.21.15.16:0/4265652389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c105120 msgr2=0x7f482c19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.496+0000 7f482b7fe640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c105120 0x7f482c19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.497+0000 7f482b7fe640 1 -- 172.21.15.16:0/4265652389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482c1a0140 con 0x7f482c1047d0 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.497+0000 7f482b7fe640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482c1047d0 0x7f482c19a1d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f481c00d770 tx=0x7f481c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.497+0000 7f48297fa640 1 -- 172.21.15.16:0/4265652389 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c017070 con 0x7f482c1047d0 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.497+0000 7f4831e45640 1 -- 172.21.15.16:0/4265652389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f482c1a0430 con 0x7f482c1047d0 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.497+0000 7f48297fa640 1 -- 172.21.15.16:0/4265652389 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f481c0054d0 con 0x7f482c1047d0 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.497+0000 7f48297fa640 1 -- 172.21.15.16:0/4265652389 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c004c30 con 0x7f482c1047d0 2024-08-28T19:22:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.497+0000 7f4831e45640 1 -- 172.21.15.16:0/4265652389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f482c1a6b90 con 0x7f482c1047d0 2024-08-28T19:22:17.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.498+0000 7f4831e45640 1 -- 172.21.15.16:0/4265652389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f482c104be0 con 0x7f482c1047d0 2024-08-28T19:22:17.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.501+0000 7f48297fa640 1 -- 172.21.15.16:0/4265652389 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f481c004dd0 con 0x7f482c1047d0 2024-08-28T19:22:17.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.502+0000 7f48297fa640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48180b4ca0 0x7f48180b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:17.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.502+0000 7f4822ffd640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48180b4ca0 0x7f48180b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:17.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.502+0000 7f48297fa640 1 -- 172.21.15.16:0/4265652389 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f481c0e57c0 con 0x7f482c1047d0 2024-08-28T19:22:17.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.502+0000 7f4822ffd640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48180b4ca0 0x7f48180b71b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f482c10f830 tx=0x7f481400b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:17.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.504+0000 7f48297fa640 1 -- 172.21.15.16:0/4265652389 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f481c0ac1e0 con 0x7f482c1047d0 2024-08-28T19:22:17.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.728+0000 7f4831e45640 1 -- 172.21.15.16:0/4265652389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f482c1024c0 con 0x7f482c1047d0 2024-08-28T19:22:17.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.729+0000 7f48297fa640 1 -- 172.21.15.16:0/4265652389 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11977 (secure 0 0 0) 0x7f481c0abae0 con 0x7f482c1047d0 2024-08-28T19:22:17.731 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:22:17.731 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"btime":"2024-08-28T19:20:55:257311+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:20:54.880238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:22:17.731 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2024-08-28T19:22:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.732+0000 7f4831e45640 1 -- 172.21.15.16:0/4265652389 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48180b4ca0 msgr2=0x7f48180b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.732+0000 7f4831e45640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48180b4ca0 0x7f48180b71b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f482c10f830 tx=0x7f481400b390 comp rx=0 tx=0).stop 2024-08-28T19:22:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.732+0000 7f4831e45640 1 -- 172.21.15.16:0/4265652389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482c1047d0 msgr2=0x7f482c19a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.732+0000 7f4831e45640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482c1047d0 0x7f482c19a1d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f481c00d770 tx=0x7f481c00dc30 comp rx=0 tx=0).stop 2024-08-28T19:22:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.733+0000 7f4831e45640 1 -- 172.21.15.16:0/4265652389 shutdown_connections 2024-08-28T19:22:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.733+0000 7f4831e45640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f482c10dc00 0x7f482c19fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.733+0000 7f4831e45640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48180b4ca0 0x7f48180b71b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.733+0000 7f4831e45640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c105120 0x7f482c19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.733+0000 7f4831e45640 1 --2- 172.21.15.16:0/4265652389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482c1047d0 0x7f482c19a1d0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.733+0000 7f4831e45640 1 -- 172.21.15.16:0/4265652389 >> 172.21.15.16:0/4265652389 conn(0x7f482c079530 msgr2=0x7f482c0ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.733+0000 7f4831e45640 1 -- 172.21.15.16:0/4265652389 shutdown_connections 2024-08-28T19:22:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:17.733+0000 7f4831e45640 1 -- 172.21.15.16:0/4265652389 wait complete. 2024-08-28T19:22:18.433 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id 3e074952 --release --await' 2024-08-28T19:22:18.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id 3e074952 --release --await 2024-08-28T19:22:18.693 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:18.926 INFO:tasks.workunit.client.0.smithi016.stderr:Updating files: 90% (11456/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-08-28T19:22:18.983 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:18 smithi016 ceph-mon[18556]: pgmap v377: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-08-28T19:22:18.983 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:18 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/4265652389' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:22:19.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:18 smithi028 ceph-mon[21769]: pgmap v377: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-08-28T19:22:19.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:18 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/4265652389' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:22:19.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:18 smithi115 ceph-mon[22184]: pgmap v377: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-08-28T19:22:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:18 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/4265652389' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:22:19.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.226+0000 7f2808916640 1 -- 172.21.15.16:0/2923524713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2804108140 msgr2=0x7f2804108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:19.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.226+0000 7f2808916640 1 --2- 172.21.15.16:0/2923524713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2804108140 0x7f2804108550 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009900 tx=0x7f27f802a410 comp rx=0 tx=0).stop 2024-08-28T19:22:19.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.227+0000 7f2808916640 1 -- 172.21.15.16:0/2923524713 shutdown_connections 2024-08-28T19:22:19.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.227+0000 7f2808916640 1 --2- 172.21.15.16:0/2923524713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2804108a90 0x7f280410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.227+0000 7f2808916640 1 --2- 172.21.15.16:0/2923524713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2804108140 0x7f2804108550 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.227+0000 7f2808916640 1 --2- 172.21.15.16:0/2923524713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2804102800 0x7f2804102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.227+0000 7f2808916640 1 -- 172.21.15.16:0/2923524713 >> 172.21.15.16:0/2923524713 conn(0x7f28040794d0 msgr2=0x7f28040798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:19.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.227+0000 7f2808916640 1 -- 172.21.15.16:0/2923524713 shutdown_connections 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.227+0000 7f2808916640 1 -- 172.21.15.16:0/2923524713 wait complete. 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.227+0000 7f2808916640 1 Processor -- start 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.227+0000 7f2808916640 1 -- start start 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f2808916640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2804102800 0x7f280419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f2808916640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2804108140 0x7f280419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f2808916640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2804108a90 0x7f280419fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f2808916640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28041126e0 con 0x7f2804102800 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f2808916640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2804112560 con 0x7f2804108a90 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f2808916640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2804112860 con 0x7f2804108140 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f280259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2804102800 0x7f280419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f280259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2804102800 0x7f280419a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56280/0 (socket says 172.21.15.16:56280) 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f280259c640 1 -- 172.21.15.16:0/3054301119 learned_addr learned my addr 172.21.15.16:0/3054301119 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:19.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f2802d9d640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2804108a90 0x7f280419fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f2801d9b640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2804108140 0x7f280419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f280259c640 1 -- 172.21.15.16:0/3054301119 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2804108140 msgr2=0x7f280419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f280259c640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2804108140 0x7f280419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f280259c640 1 -- 172.21.15.16:0/3054301119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2804108a90 msgr2=0x7f280419fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f280259c640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2804108a90 0x7f280419fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f280259c640 1 -- 172.21.15.16:0/3054301119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28041a0190 con 0x7f2804102800 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f2802d9d640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2804108a90 0x7f280419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.228+0000 7f280259c640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2804102800 0x7f280419a220 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f27f400d950 tx=0x7f27f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.229+0000 7f27f37fe640 1 -- 172.21.15.16:0/3054301119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4017070 con 0x7f2804102800 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.229+0000 7f27f37fe640 1 -- 172.21.15.16:0/3054301119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27f4009c30 con 0x7f2804102800 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.229+0000 7f2808916640 1 -- 172.21.15.16:0/3054301119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28041a0480 con 0x7f2804102800 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.229+0000 7f27f37fe640 1 -- 172.21.15.16:0/3054301119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f400f620 con 0x7f2804102800 2024-08-28T19:22:19.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.229+0000 7f2808916640 1 -- 172.21.15.16:0/3054301119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28041a6be0 con 0x7f2804102800 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.232+0000 7f280ee8b740 1 Processor -- start 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.232+0000 7f280ee8b740 1 -- start start 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.232+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.232+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645f5660 0x55a2649c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.232+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a2645fd840 0x55a2649c4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.232+0000 7f280ee8b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a26493aa70 con 0x55a26461e520 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.232+0000 7f280ee8b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a26492a000 con 0x55a2645f5660 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.232+0000 7f280ee8b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a26492dfb0 con 0x55a2645fd840 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.233+0000 7f2802d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.233+0000 7f2802d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649c3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56292/0 (socket says 172.21.15.16:56292) 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.233+0000 7f2802d9d640 1 -- 172.21.15.16:0/4053351124 learned_addr learned my addr 172.21.15.16:0/4053351124 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.233+0000 7f27f37fe640 1 -- 172.21.15.16:0/3054301119 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f27f400f7c0 con 0x7f2804102800 2024-08-28T19:22:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.233+0000 7f2802d9d640 1 -- 172.21.15.16:0/4053351124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a2645fd840 msgr2=0x55a2649c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.233+0000 7f2802d9d640 1 --2- 172.21.15.16:0/4053351124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a2645fd840 0x55a2649c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.233+0000 7f2802d9d640 1 -- 172.21.15.16:0/4053351124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645f5660 msgr2=0x55a2649c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.233+0000 7f2802d9d640 1 --2- 172.21.15.16:0/4053351124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645f5660 0x55a2649c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.233+0000 7f2802d9d640 1 -- 172.21.15.16:0/4053351124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a264689b00 con 0x55a26461e520 2024-08-28T19:22:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.233+0000 7f2802d9d640 1 --2- 172.21.15.16:0/4053351124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649c3c80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00af40 tx=0x7f27fc00e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87255fe130e03035 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.233+0000 7f27f0ff9640 1 -- 172.21.15.16:0/4053351124 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc003b70 con 0x55a26461e520 2024-08-28T19:22:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.234+0000 7f27f0ff9640 1 -- 172.21.15.16:0/4053351124 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27fc003d10 con 0x55a26461e520 2024-08-28T19:22:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.234+0000 7f27f0ff9640 1 -- 172.21.15.16:0/4053351124 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc010ab0 con 0x55a26461e520 2024-08-28T19:22:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.234+0000 7f27f37fe640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27ec0b4d20 0x7f27ec0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.234+0000 7f280259c640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27ec0b4d20 0x7f27ec0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.234+0000 7f27f37fe640 1 -- 172.21.15.16:0/3054301119 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f27f40e5640 con 0x7f2804102800 2024-08-28T19:22:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.235+0000 7f280ee8b740 1 -- 172.21.15.16:0/4053351124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 msgr2=0x55a2649c3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.235+0000 7f280ee8b740 1 --2- 172.21.15.16:0/4053351124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649c3c80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00af40 tx=0x7f27fc00e7e0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.235+0000 7f280ee8b740 1 -- 172.21.15.16:0/4053351124 shutdown_connections 2024-08-28T19:22:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.235+0000 7f280ee8b740 1 --2- 172.21.15.16:0/4053351124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a2645fd840 0x55a2649c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.235+0000 7f280ee8b740 1 --2- 172.21.15.16:0/4053351124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645f5660 0x55a2649c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.235+0000 7f280ee8b740 1 --2- 172.21.15.16:0/4053351124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649c3c80 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.235+0000 7f280ee8b740 1 -- 172.21.15.16:0/4053351124 >> 172.21.15.16:0/4053351124 conn(0x55a264612910 msgr2=0x55a2649c1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.235+0000 7f280259c640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27ec0b4d20 0x7f27ec0b7230 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55a2649c2900 tx=0x7f27f4012000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.235+0000 7f280ee8b740 1 -- 172.21.15.16:0/4053351124 shutdown_connections 2024-08-28T19:22:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.235+0000 7f280ee8b740 1 -- 172.21.15.16:0/4053351124 wait complete. 2024-08-28T19:22:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.235+0000 7f280ee8b740 1 -- start start 2024-08-28T19:22:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f280ee8b740 1 Processor -- start 2024-08-28T19:22:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645fd840 0x55a2649c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a26461e520 0x55a2649c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f280ee8b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a2649d0370 con 0x55a2645f5660 2024-08-28T19:22:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f280ee8b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a2648eb520 con 0x55a2645fd840 2024-08-28T19:22:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f280ee8b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a2649d04f0 con 0x55a26461e520 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f280ee8b740 10 client.0 _ll_register_callbacks cb 0x55a264923c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f2802d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f2802d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649cf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56298/0 (socket says 172.21.15.16:56298) 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f2802d9d640 1 -- 172.21.15.16:0/3764118179 learned_addr learned my addr 172.21.15.16:0/3764118179 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.236+0000 7f2801d9b640 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645fd840 0x55a2649c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f2802d9d640 1 -- 172.21.15.16:0/3764118179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a26461e520 msgr2=0x55a2649c9f20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f2802d9d640 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a26461e520 0x55a2649c9f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f2802d9d640 1 -- 172.21.15.16:0/3764118179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645fd840 msgr2=0x55a2649c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f2802d9d640 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645fd840 0x55a2649c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f2802d9d640 1 -- 172.21.15.16:0/3764118179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a264700960 con 0x55a2645f5660 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f2802d9d640 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649cf540 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00ac80 tx=0x7f27fc005ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff1554a3a6a36a4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f27e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f27e9ffb640 1 -- 172.21.15.16:0/3764118179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc010a60 con 0x55a2645f5660 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f280ee8b740 1 -- 172.21.15.16:0/3764118179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a2648c99a0 con 0x55a2645f5660 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f27e9ffb640 1 -- 172.21.15.16:0/3764118179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f27fc010c00 con 0x55a2645f5660 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f27e9ffb640 1 -- 172.21.15.16:0/3764118179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f27fc010e00 con 0x55a2645f5660 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f280ee8b740 10 client.15969 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.237+0000 7f280ee8b740 1 -- 172.21.15.16:0/3764118179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a2649e6e40 con 0x55a2645f5660 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.238+0000 7f27e9ffb640 1 -- 172.21.15.16:0/3764118179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27fc004000 con 0x55a2645f5660 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.238+0000 7f27e9ffb640 1 -- 172.21.15.16:0/3764118179 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f27fc0050c0 con 0x55a2645f5660 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.238+0000 7f280ee8b740 10 client.15969 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.238+0000 7f280ee8b740 10 client.15969 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.238+0000 7f280ee8b740 20 client.15969 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.238+0000 7f280ee8b740 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a2645dcea0 0x55a2649ea200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.238+0000 7f280ee8b740 4 client.15969 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.238+0000 7f280ee8b740 1 -- 172.21.15.16:0/3764118179 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2648c99a0 con 0x55a2645dcea0 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.238+0000 7f2801d9b640 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a2645dcea0 0x55a2649ea200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.239+0000 7f27e9ffb640 10 client.15969 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.239+0000 7f2801d9b640 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a2645dcea0 0x55a2649ea200 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7509c6c9558e530b server_cookie=3fd448eb94ae9cfe in_seq=0 out_seq=0 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.239+0000 7f27e9ffb640 1 -- 172.21.15.16:0/3764118179 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a2648c99a0 con 0x55a2645dcea0 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.239+0000 7f27e9ffb640 10 client.15969 handle_command_reply: tid=0 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 1 client.15969 shutdown 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 1 -- 172.21.15.16:0/3764118179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 msgr2=0x55a2649cf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649cf540 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00ac80 tx=0x7f27fc005ab0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 1 -- 172.21.15.16:0/3764118179 shutdown_connections 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a2645dcea0 0x55a2649ea200 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a26461e520 0x55a2649c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645fd840 0x55a2649c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 1 --2- 172.21.15.16:0/3764118179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649cf540 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 1 -- 172.21.15.16:0/3764118179 >> 172.21.15.16:0/3764118179 conn(0x55a264612910 msgr2=0x55a2649c6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f27e9ffb640 0 client.15969 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 1 -- 172.21.15.16:0/3764118179 shutdown_connections 2024-08-28T19:22:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 1 -- 172.21.15.16:0/3764118179 wait complete. 2024-08-28T19:22:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.240+0000 7f280ee8b740 20 client.15969 trim_cache size 0 max 16384 2024-08-28T19:22:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f280ee8b740 1 Processor -- start 2024-08-28T19:22:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f280ee8b740 1 -- start start 2024-08-28T19:22:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645f5660 0x55a2649c0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a2645fd840 0x55a2649d3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649d51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f280ee8b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28041ae9f0 con 0x55a26461e520 2024-08-28T19:22:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f280ee8b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28041b0530 con 0x55a2645f5660 2024-08-28T19:22:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f280ee8b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f28041af0c0 con 0x55a2645fd840 2024-08-28T19:22:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f2802d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649d51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f2802d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649d51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56310/0 (socket says 172.21.15.16:56310) 2024-08-28T19:22:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f2802d9d640 1 -- 172.21.15.16:0/764961365 learned_addr learned my addr 172.21.15.16:0/764961365 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f2802d9d640 1 --2- 172.21.15.16:0/764961365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645f5660 0x55a2649c0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f2801d9b640 1 --2- 172.21.15.16:0/764961365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a2645fd840 0x55a2649d3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f2802d9d640 1 -- 172.21.15.16:0/764961365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a2645fd840 msgr2=0x55a2649d3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f2802d9d640 1 --2- 172.21.15.16:0/764961365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a2645fd840 0x55a2649d3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f2802d9d640 1 -- 172.21.15.16:0/764961365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645f5660 msgr2=0x55a2649c0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f2802d9d640 1 --2- 172.21.15.16:0/764961365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645f5660 0x55a2649c0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.258+0000 7f2802d9d640 1 -- 172.21.15.16:0/764961365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2645dd0d0 con 0x55a26461e520 2024-08-28T19:22:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.259+0000 7f2802d9d640 1 --2- 172.21.15.16:0/764961365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649d51a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f27fc004800 tx=0x7f27fc01e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2086c4525d94e57 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.259+0000 7f27ebfff640 1 -- 172.21.15.16:0/764961365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc03a070 con 0x55a26461e520 2024-08-28T19:22:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.259+0000 7f27ebfff640 1 -- 172.21.15.16:0/764961365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27fc004300 con 0x55a26461e520 2024-08-28T19:22:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.259+0000 7f27ebfff640 1 -- 172.21.15.16:0/764961365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc01a790 con 0x55a26461e520 2024-08-28T19:22:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.260+0000 7f280ee8b740 1 -- 172.21.15.16:0/764961365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 msgr2=0x55a2649d51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.260+0000 7f280ee8b740 1 --2- 172.21.15.16:0/764961365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649d51a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f27fc004800 tx=0x7f27fc01e040 comp rx=0 tx=0).stop 2024-08-28T19:22:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.260+0000 7f280ee8b740 1 -- 172.21.15.16:0/764961365 shutdown_connections 2024-08-28T19:22:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.260+0000 7f280ee8b740 1 --2- 172.21.15.16:0/764961365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a26461e520 0x55a2649d51a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.260+0000 7f280ee8b740 1 --2- 172.21.15.16:0/764961365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a2645fd840 0x55a2649d3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.260+0000 7f280ee8b740 1 --2- 172.21.15.16:0/764961365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645f5660 0x55a2649c0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.260+0000 7f280ee8b740 1 -- 172.21.15.16:0/764961365 >> 172.21.15.16:0/764961365 conn(0x55a264612910 msgr2=0x55a2649c1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.260+0000 7f280ee8b740 1 -- 172.21.15.16:0/764961365 shutdown_connections 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.260+0000 7f280ee8b740 1 -- 172.21.15.16:0/764961365 wait complete. 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.260+0000 7f280ee8b740 1 -- start start 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.260+0000 7f280ee8b740 1 Processor -- start 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649d5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645fd840 0x55a2649e0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f280ee8b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a26461e520 0x55a2649e1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f280ee8b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28041afe60 con 0x55a2645f5660 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f280ee8b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28041af790 con 0x55a2645fd840 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f280ee8b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f28041ac9d0 con 0x55a26461e520 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f280ee8b740 10 client.0 _ll_register_callbacks cb 0x7f27fc014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f2802d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649d5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f2802d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649d5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56322/0 (socket says 172.21.15.16:56322) 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f2802d9d640 1 -- 172.21.15.16:0/191347028 learned_addr learned my addr 172.21.15.16:0/191347028 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f2802d9d640 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a26461e520 0x55a2649e1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f2801d9b640 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645fd840 0x55a2649e0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f2802d9d640 1 -- 172.21.15.16:0/191347028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a26461e520 msgr2=0x55a2649e1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f2802d9d640 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a26461e520 0x55a2649e1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f2802d9d640 1 -- 172.21.15.16:0/191347028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645fd840 msgr2=0x55a2649e0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f2802d9d640 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645fd840 0x55a2649e0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f2802d9d640 1 -- 172.21.15.16:0/191347028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2649e1ad0 con 0x55a2645f5660 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f2802d9d640 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649d5a90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f27fc005560 tx=0x7f27fc010790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfb480fcc837364 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.261+0000 7f27e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f27e9ffb640 1 -- 172.21.15.16:0/191347028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc03a070 con 0x55a2645f5660 2024-08-28T19:22:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f27e9ffb640 1 -- 172.21.15.16:0/191347028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f27fc003f80 con 0x55a2645f5660 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f280ee8b740 1 -- 172.21.15.16:0/191347028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27fc004300 con 0x55a2645f5660 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f27e9ffb640 1 -- 172.21.15.16:0/191347028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f27fc01a530 con 0x55a2645f5660 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f280ee8b740 10 client.15975 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f280ee8b740 1 -- 172.21.15.16:0/191347028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a264a54ae0 con 0x55a2645f5660 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f27e9ffb640 1 -- 172.21.15.16:0/191347028 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27fc004300 con 0x55a2645f5660 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f27e9ffb640 1 -- 172.21.15.16:0/191347028 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f27fc004a70 con 0x55a2645f5660 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f280ee8b740 10 client.15975 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f280ee8b740 10 client.15975 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f280ee8b740 20 client.15975 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f280ee8b740 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a2645b2250 0x55a264a57080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f280ee8b740 4 client.15975 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3e074952", "release": true, "await": true}] 2024-08-28T19:22:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.262+0000 7f280ee8b740 1 -- 172.21.15.16:0/191347028 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3e074952", "release": true, "await": true}) -- 0x55a26493c9c0 con 0x55a2645b2250 2024-08-28T19:22:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.263+0000 7f2801d9b640 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a2645b2250 0x55a264a57080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.263+0000 7f27e9ffb640 10 client.15975 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.263+0000 7f2801d9b640 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a2645b2250 0x55a264a57080 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb1dda05dd34e4dc server_cookie=354d04dcd1c138db in_seq=0 out_seq=0 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f27e9ffb640 1 -- 172.21.15.16:0/191347028 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55a26493c9c0 con 0x55a2645b2250 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f27e9ffb640 10 client.15975 handle_command_reply: tid=0 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 1 client.15975 shutdown 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 1 -- 172.21.15.16:0/191347028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 msgr2=0x55a2649d5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649d5a90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f27fc005560 tx=0x7f27fc010790 comp rx=0 tx=0).stop 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 1 -- 172.21.15.16:0/191347028 shutdown_connections 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a2645b2250 0x55a264a57080 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a26461e520 0x55a2649e1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2645fd840 0x55a2649e0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 1 --2- 172.21.15.16:0/191347028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2645f5660 0x55a2649d5a90 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f27e9ffb640 0 client.15975 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 1 -- 172.21.15.16:0/191347028 >> 172.21.15.16:0/191347028 conn(0x55a264612910 msgr2=0x55a2649cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 1 -- 172.21.15.16:0/191347028 shutdown_connections 2024-08-28T19:22:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 1 -- 172.21.15.16:0/191347028 wait complete. 2024-08-28T19:22:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.266+0000 7f280ee8b740 20 client.15975 trim_cache size 0 max 16384 2024-08-28T19:22:19.270 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:22:19.270 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 29, 2024-08-28T19:22:19.270 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:22:19.270 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 4, 2024-08-28T19:22:19.270 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "3e074952": { 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 4, 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 154.4, 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:22:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:22:19.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:22:19.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:22:19.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:22:19.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:22:19.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:22:19.272 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 -- 172.21.15.16:0/3054301119 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27ec0b4d20 msgr2=0x7f27ec0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27ec0b4d20 0x7f27ec0b7230 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55a2649c2900 tx=0x7f27f4012000 comp rx=0 tx=0).stop 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 -- 172.21.15.16:0/3054301119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2804102800 msgr2=0x7f280419a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2804102800 0x7f280419a220 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f27f400d950 tx=0x7f27f400de10 comp rx=0 tx=0).stop 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 -- 172.21.15.16:0/3054301119 shutdown_connections 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27ec0b4d20 0x7f27ec0b7230 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2804108a90 0x7f280419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2804108140 0x7f280419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 --2- 172.21.15.16:0/3054301119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2804102800 0x7f280419a220 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 -- 172.21.15.16:0/3054301119 >> 172.21.15.16:0/3054301119 conn(0x7f28040794d0 msgr2=0x7f2804100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 -- 172.21.15.16:0/3054301119 shutdown_connections 2024-08-28T19:22:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:19.267+0000 7f2808916640 1 -- 172.21.15.16:0/3054301119 wait complete. 2024-08-28T19:22:19.834 INFO:tasks.workunit.client.0.smithi016.stderr:Note: switching to 'd19e6118f7015f0047264006cb0c873528eb3ae6'. 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by switching back to a branch. 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr: git switch -c 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr:Or undo this operation with: 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-08-28T19:22:19.835 INFO:tasks.workunit.client.0.smithi016.stderr: git switch - 2024-08-28T19:22:19.836 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-08-28T19:22:19.836 INFO:tasks.workunit.client.0.smithi016.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-28T19:22:19.836 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-08-28T19:22:19.836 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at d19e6118f70 Merge PR #44359 into wip-mchangir-testing-20240828.085847-main-debug 2024-08-28T19:22:19.839 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-28T19:22:19.908 INFO:tasks.workunit.client.0.smithi016.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-28T19:22:19.910 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-28T19:22:19.910 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-28T19:22:20.009 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '3e074952', total seconds elapsed: 38.0 2024-08-28T19:22:20.596 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-28T19:22:20.662 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-28T19:22:20.702 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-28T19:22:20.703 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-28T19:22:20.703 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-28T19:22:20.742 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-28T19:22:20.746 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:22:20.746 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-28T19:22:20.762 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-08-28T19:22:20.763 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-08-28T19:22:20.763 DEBUG:teuthology.orchestra.run.smithi016:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d19e6118f7015f0047264006cb0c873528eb3ae6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-08-28T19:22:20.824 INFO:tasks.workunit.client.0.smithi016.stderr:+ dbench 1 2024-08-28T19:22:20.832 INFO:tasks.workunit.client.0.smithi016.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-08-28T19:22:20.833 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-08-28T19:22:20.833 INFO:tasks.workunit.client.0.smithi016.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-08-28T19:22:20.833 INFO:tasks.workunit.client.0.smithi016.stdout:0 of 1 processes prepared for launch 0 sec 2024-08-28T19:22:20.933 INFO:tasks.workunit.client.0.smithi016.stdout:1 of 1 processes prepared for launch 0 sec 2024-08-28T19:22:20.933 INFO:tasks.workunit.client.0.smithi016.stdout:releasing clients 2024-08-28T19:22:21.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:20 smithi016 ceph-mon[18556]: pgmap v378: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-08-28T19:22:21.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:20 smithi028 ceph-mon[21769]: pgmap v378: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-08-28T19:22:21.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:20 smithi115 ceph-mon[22184]: pgmap v378: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-08-28T19:22:21.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 3715 38.78 MB/sec warmup 1 sec latency 6.900 ms 2024-08-28T19:22:22.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 6907 27.25 MB/sec warmup 2 sec latency 13.400 ms 2024-08-28T19:22:23.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:22 smithi016 ceph-mon[18556]: pgmap v379: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-08-28T19:22:23.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:22 smithi028 ceph-mon[21769]: pgmap v379: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-08-28T19:22:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:22 smithi115 ceph-mon[22184]: pgmap v379: 129 pgs: 129 active+clean; 1.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-08-28T19:22:23.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 10021 23.91 MB/sec warmup 3 sec latency 13.841 ms 2024-08-28T19:22:24.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:23 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:22:24.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:23 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:22:24.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:23 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:22:24.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 12793 21.01 MB/sec warmup 4 sec latency 16.447 ms 2024-08-28T19:22:25.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:24 smithi016 ceph-mon[18556]: pgmap v380: 129 pgs: 129 active+clean; 1.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-08-28T19:22:25.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:24 smithi028 ceph-mon[21769]: pgmap v380: 129 pgs: 129 active+clean; 1.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-08-28T19:22:25.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:24 smithi115 ceph-mon[22184]: pgmap v380: 129 pgs: 129 active+clean; 1.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-08-28T19:22:25.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 15427 19.91 MB/sec warmup 5 sec latency 42.656 ms 2024-08-28T19:22:26.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 18646 19.55 MB/sec warmup 6 sec latency 12.958 ms 2024-08-28T19:22:27.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:27 smithi115 ceph-mon[22184]: pgmap v381: 129 pgs: 129 active+clean; 6.7 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 526 KiB/s wr, 84 op/s 2024-08-28T19:22:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:27 smithi016 ceph-mon[18556]: pgmap v381: 129 pgs: 129 active+clean; 6.7 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 526 KiB/s wr, 84 op/s 2024-08-28T19:22:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:27 smithi028 ceph-mon[21769]: pgmap v381: 129 pgs: 129 active+clean; 6.7 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 526 KiB/s wr, 84 op/s 2024-08-28T19:22:27.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21991 19.41 MB/sec warmup 7 sec latency 14.587 ms 2024-08-28T19:22:28.025 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:22:28.285 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:28.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.790+0000 7f6b5d10b640 1 -- 172.21.15.16:0/3167372272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b58108140 msgr2=0x7f6b58108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:28.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.790+0000 7f6b5d10b640 1 --2- 172.21.15.16:0/3167372272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b58108140 0x7f6b58108550 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c009900 tx=0x7f6b4c02a410 comp rx=0 tx=0).stop 2024-08-28T19:22:28.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.790+0000 7f6b5d10b640 1 -- 172.21.15.16:0/3167372272 shutdown_connections 2024-08-28T19:22:28.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.790+0000 7f6b5d10b640 1 --2- 172.21.15.16:0/3167372272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58108a90 0x7f6b5810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.790+0000 7f6b5d10b640 1 --2- 172.21.15.16:0/3167372272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b58108140 0x7f6b58108550 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.790+0000 7f6b5d10b640 1 --2- 172.21.15.16:0/3167372272 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b58102800 0x7f6b58102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.790+0000 7f6b5d10b640 1 -- 172.21.15.16:0/3167372272 >> 172.21.15.16:0/3167372272 conn(0x7f6b580794d0 msgr2=0x7f6b580798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:28.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.790+0000 7f6b5d10b640 1 -- 172.21.15.16:0/3167372272 shutdown_connections 2024-08-28T19:22:28.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.790+0000 7f6b5d10b640 1 -- 172.21.15.16:0/3167372272 wait complete. 2024-08-28T19:22:28.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.791+0000 7f6b5d10b640 1 Processor -- start 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.791+0000 7f6b5d10b640 1 -- start start 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b5d10b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b58102800 0x7f6b5819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b5d10b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58108140 0x7f6b5819a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b5d10b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b58108a90 0x7f6b5819fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b5d10b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b581126e0 con 0x7f6b58102800 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b5d10b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b58112560 con 0x7f6b58108140 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b5d10b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6b58112860 con 0x7f6b58108a90 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b58102800 0x7f6b5819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b58102800 0x7f6b5819a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50926/0 (socket says 172.21.15.16:50926) 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b56d9d640 1 -- 172.21.15.16:0/110234305 learned_addr learned my addr 172.21.15.16:0/110234305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b5659c640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58108140 0x7f6b5819a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b5759e640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b58108a90 0x7f6b5819fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b56d9d640 1 -- 172.21.15.16:0/110234305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b58108a90 msgr2=0x7f6b5819fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b58108a90 0x7f6b5819fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b56d9d640 1 -- 172.21.15.16:0/110234305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58108140 msgr2=0x7f6b5819a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58108140 0x7f6b5819a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b56d9d640 1 -- 172.21.15.16:0/110234305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b581a0250 con 0x7f6b58102800 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b5659c640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58108140 0x7f6b5819a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b5759e640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b58108a90 0x7f6b5819fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b58102800 0x7f6b5819a2e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800e9f0 tx=0x7f6b4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.792+0000 7f6b3ffff640 1 -- 172.21.15.16:0/110234305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4800cd80 con 0x7f6b58102800 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.793+0000 7f6b5d10b640 1 -- 172.21.15.16:0/110234305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b581a0540 con 0x7f6b58102800 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.793+0000 7f6b3ffff640 1 -- 172.21.15.16:0/110234305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b480054d0 con 0x7f6b58102800 2024-08-28T19:22:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.793+0000 7f6b3ffff640 1 -- 172.21.15.16:0/110234305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b48010690 con 0x7f6b58102800 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.793+0000 7f6b5d10b640 1 -- 172.21.15.16:0/110234305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b581a6ca0 con 0x7f6b58102800 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.795+0000 7f6b3ffff640 1 -- 172.21.15.16:0/110234305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f6b48010830 con 0x7f6b58102800 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.795+0000 7f6b63680740 1 Processor -- start 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.795+0000 7f6b63680740 1 -- start start 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b63680740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 0x560b346b92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b63680740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346b9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b63680740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 0x560b346b9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b63680740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560b34630630 con 0x560b342eb1b0 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b63680740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560b3461fbc0 con 0x560b342f3390 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b63680740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560b34623b70 con 0x560b34314070 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346b9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346b9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50932/0 (socket says 172.21.15.16:50932) 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b5659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 0x560b346b9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b56d9d640 1 -- 172.21.15.16:0/3624720759 learned_addr learned my addr 172.21.15.16:0/3624720759 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b5759e640 1 --2- 172.21.15.16:0/3624720759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 0x560b346b92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b3ffff640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b400b4ca0 0x7f6b400b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b5759e640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b400b4ca0 0x7f6b400b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b3ffff640 1 -- 172.21.15.16:0/110234305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6b480e5d00 con 0x7f6b58102800 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b56d9d640 1 -- 172.21.15.16:0/3624720759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 msgr2=0x560b346b92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/3624720759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 0x560b346b92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b56d9d640 1 -- 172.21.15.16:0/3624720759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 msgr2=0x560b346b9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/3624720759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 0x560b346b9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b56d9d640 1 -- 172.21.15.16:0/3624720759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b3437f650 con 0x560b342eb1b0 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b5659c640 1 --2- 172.21.15.16:0/3624720759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 0x560b346b9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.796+0000 7f6b5759e640 1 --2- 172.21.15.16:0/3624720759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 0x560b346b92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.797+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/3624720759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346b9800 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6b480e2ff0 tx=0x7f6b48004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a83f4bfb09edad server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.797+0000 7f6b3d7fa640 1 -- 172.21.15.16:0/3624720759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b480ab970 con 0x560b342eb1b0 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.797+0000 7f6b3d7fa640 1 -- 172.21.15.16:0/3624720759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b480cae50 con 0x560b342eb1b0 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.797+0000 7f6b3d7fa640 1 -- 172.21.15.16:0/3624720759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b480ab790 con 0x560b342eb1b0 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.797+0000 7f6b5759e640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b400b4ca0 0x7f6b400b71b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6b500094d0 tx=0x7f6b50004ab0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.798+0000 7f6b63680740 1 -- 172.21.15.16:0/3624720759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 msgr2=0x560b346b9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.798+0000 7f6b63680740 1 --2- 172.21.15.16:0/3624720759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346b9800 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6b480e2ff0 tx=0x7f6b48004a90 comp rx=0 tx=0).stop 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.798+0000 7f6b63680740 1 -- 172.21.15.16:0/3624720759 shutdown_connections 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.798+0000 7f6b63680740 1 --2- 172.21.15.16:0/3624720759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 0x560b346b9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.798+0000 7f6b63680740 1 --2- 172.21.15.16:0/3624720759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346b9800 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.798+0000 7f6b63680740 1 --2- 172.21.15.16:0/3624720759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 0x560b346b92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.798+0000 7f6b63680740 1 -- 172.21.15.16:0/3624720759 >> 172.21.15.16:0/3624720759 conn(0x560b34308460 msgr2=0x560b346b71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.798+0000 7f6b63680740 1 -- 172.21.15.16:0/3624720759 shutdown_connections 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.798+0000 7f6b63680740 1 -- 172.21.15.16:0/3624720759 wait complete. 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.798+0000 7f6b63680740 1 -- start start 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.798+0000 7f6b63680740 1 Processor -- start 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b63680740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b342eb1b0 0x560b346c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b63680740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b63680740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b34314070 0x560b346d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b63680740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560b34482700 con 0x560b342f3390 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b63680740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560b345a6700 con 0x560b34314070 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b63680740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560b346c79e0 con 0x560b342eb1b0 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b63680740 10 client.0 _ll_register_callbacks cb 0x560b345d5dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b5659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b5659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346c4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50934/0 (socket says 172.21.15.16:50934) 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b5659c640 1 -- 172.21.15.16:0/3612031274 learned_addr learned my addr 172.21.15.16:0/3612031274 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b5759e640 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b34314070 0x560b346d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b342eb1b0 0x560b346c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b5659c640 1 -- 172.21.15.16:0/3612031274 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b342eb1b0 msgr2=0x560b346c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b5659c640 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b342eb1b0 0x560b346c4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b5659c640 1 -- 172.21.15.16:0/3612031274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b34314070 msgr2=0x560b346d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b5659c640 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b34314070 0x560b346d0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b5659c640 1 -- 172.21.15.16:0/3612031274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b345df8b0 con 0x560b342f3390 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b5759e640 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b34314070 0x560b346d0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b5659c640 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346c4f40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c009900 tx=0x7f6b4c031740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50f87de4906e8bb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b3aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.799+0000 7f6b3aff5640 1 -- 172.21.15.16:0/3612031274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c035040 con 0x560b342f3390 2024-08-28T19:22:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b3aff5640 1 -- 172.21.15.16:0/3612031274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6b4c004040 con 0x560b342f3390 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b63680740 1 -- 172.21.15.16:0/3612031274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b480cae50 con 0x560b342f3390 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b3aff5640 1 -- 172.21.15.16:0/3612031274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6b4c004240 con 0x560b342f3390 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b63680740 10 client.15987 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b63680740 1 -- 172.21.15.16:0/3612031274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560b346d3770 con 0x560b342f3390 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b3aff5640 1 -- 172.21.15.16:0/3612031274 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b480cae50 con 0x560b342f3390 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b3aff5640 1 -- 172.21.15.16:0/3612031274 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6b4c004af0 con 0x560b342f3390 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b63680740 10 client.15987 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b63680740 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b63680740 20 client.15987 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b63680740 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b342d29f0 0x560b346d6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b63680740 4 client.15987 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b63680740 1 -- 172.21.15.16:0/3612031274 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b34632580 con 0x560b342d29f0 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.800+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b342d29f0 0x560b346d6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.801+0000 7f6b3aff5640 10 client.15987 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.801+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b342d29f0 0x560b346d6c70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7df703a7d047d004 server_cookie=508687e91e70dd8c in_seq=0 out_seq=0 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b3aff5640 1 -- 172.21.15.16:0/3612031274 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560b34632580 con 0x560b342d29f0 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b3aff5640 10 client.15987 handle_command_reply: tid=0 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 1 client.15987 shutdown 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 1 -- 172.21.15.16:0/3612031274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 msgr2=0x560b346c4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346c4f40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c009900 tx=0x7f6b4c031740 comp rx=0 tx=0).stop 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 1 -- 172.21.15.16:0/3612031274 shutdown_connections 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b34314070 0x560b346d0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346c4f40 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b342d29f0 0x560b346d6c70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 1 --2- 172.21.15.16:0/3612031274 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b342eb1b0 0x560b346c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b3aff5640 0 client.15987 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:28.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 1 -- 172.21.15.16:0/3612031274 >> 172.21.15.16:0/3612031274 conn(0x560b34308460 msgr2=0x560b346bc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:28.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 1 -- 172.21.15.16:0/3612031274 shutdown_connections 2024-08-28T19:22:28.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 1 -- 172.21.15.16:0/3612031274 wait complete. 2024-08-28T19:22:28.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.802+0000 7f6b63680740 20 client.15987 trim_cache size 0 max 16384 2024-08-28T19:22:28.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b63680740 1 Processor -- start 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b63680740 1 -- start start 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b63680740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b342eb1b0 0x560b346d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b63680740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346bc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b63680740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b34314070 0x560b346bc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b63680740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b581b0be0 con 0x560b342f3390 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b63680740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b581b2720 con 0x560b34314070 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b63680740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6b581b12b0 con 0x560b342eb1b0 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b5659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346bc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b56d9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b342eb1b0 0x560b346d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b5659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346bc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50948/0 (socket says 172.21.15.16:50948) 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b5659c640 1 -- 172.21.15.16:0/3273761025 learned_addr learned my addr 172.21.15.16:0/3273761025 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.820+0000 7f6b5759e640 1 --2- 172.21.15.16:0/3273761025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b34314070 0x560b346bc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.821+0000 7f6b5659c640 1 -- 172.21.15.16:0/3273761025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b342eb1b0 msgr2=0x560b346d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:28.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.821+0000 7f6b5659c640 1 --2- 172.21.15.16:0/3273761025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b342eb1b0 0x560b346d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.821+0000 7f6b5659c640 1 -- 172.21.15.16:0/3273761025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b34314070 msgr2=0x560b346bc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:28.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.821+0000 7f6b5659c640 1 --2- 172.21.15.16:0/3273761025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b34314070 0x560b346bc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.821+0000 7f6b5659c640 1 -- 172.21.15.16:0/3273761025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b346bc120 con 0x560b342f3390 2024-08-28T19:22:28.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.821+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/3273761025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b342eb1b0 0x560b346d0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:28.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.821+0000 7f6b5659c640 1 --2- 172.21.15.16:0/3273761025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346bc3c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c000c00 tx=0x7f6b4c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4acd21df17a23ac9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:28.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.821+0000 7f6b3cff9640 1 -- 172.21.15.16:0/3273761025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c035030 con 0x560b342f3390 2024-08-28T19:22:28.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.821+0000 7f6b3cff9640 1 -- 172.21.15.16:0/3273761025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b4c004c20 con 0x560b342f3390 2024-08-28T19:22:28.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.821+0000 7f6b3cff9640 1 -- 172.21.15.16:0/3273761025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c000e60 con 0x560b342f3390 2024-08-28T19:22:28.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.822+0000 7f6b63680740 1 -- 172.21.15.16:0/3273761025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 msgr2=0x560b346bc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:28.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.822+0000 7f6b63680740 1 --2- 172.21.15.16:0/3273761025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346bc3c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c000c00 tx=0x7f6b4c005850 comp rx=0 tx=0).stop 2024-08-28T19:22:28.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.822+0000 7f6b63680740 1 -- 172.21.15.16:0/3273761025 shutdown_connections 2024-08-28T19:22:28.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.822+0000 7f6b63680740 1 --2- 172.21.15.16:0/3273761025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b34314070 0x560b346bc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.822+0000 7f6b63680740 1 --2- 172.21.15.16:0/3273761025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342f3390 0x560b346bc3c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.822+0000 7f6b63680740 1 --2- 172.21.15.16:0/3273761025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b342eb1b0 0x560b346d0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.822+0000 7f6b63680740 1 -- 172.21.15.16:0/3273761025 >> 172.21.15.16:0/3273761025 conn(0x560b34308460 msgr2=0x560b346c3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.822+0000 7f6b63680740 1 -- 172.21.15.16:0/3273761025 shutdown_connections 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.822+0000 7f6b63680740 1 -- 172.21.15.16:0/3273761025 wait complete. 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.822+0000 7f6b63680740 1 -- start start 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b63680740 1 Processor -- start 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b63680740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b63680740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 0x560b346c9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b63680740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 0x560b346c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b63680740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b581b2050 con 0x560b342eb1b0 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b63680740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b581b1980 con 0x560b342f3390 2024-08-28T19:22:28.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b63680740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6b581aebc0 con 0x560b34314070 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b63680740 10 client.0 _ll_register_callbacks cb 0x7f6b4c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50960/0 (socket says 172.21.15.16:50960) 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b56d9d640 1 -- 172.21.15.16:0/4284191393 learned_addr learned my addr 172.21.15.16:0/4284191393 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b5659c640 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 0x560b346c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b5759e640 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 0x560b346c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b56d9d640 1 -- 172.21.15.16:0/4284191393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 msgr2=0x560b346c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 0x560b346c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b56d9d640 1 -- 172.21.15.16:0/4284191393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 msgr2=0x560b346c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 0x560b346c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b56d9d640 1 -- 172.21.15.16:0/4284191393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b34740740 con 0x560b342eb1b0 2024-08-28T19:22:28.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b5759e640 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 0x560b346c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.823+0000 7f6b5659c640 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 0x560b346c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b56d9d640 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346c8d20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6b480abfb0 tx=0x7f6b480abc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68bb0b06fcb778b2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b3aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b3aff5640 1 -- 172.21.15.16:0/4284191393 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b48097070 con 0x560b342eb1b0 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b3aff5640 1 -- 172.21.15.16:0/4284191393 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f6b480e9280 con 0x560b342eb1b0 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b63680740 1 -- 172.21.15.16:0/4284191393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b4c004c20 con 0x560b342eb1b0 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b3aff5640 1 -- 172.21.15.16:0/4284191393 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6b480ab970 con 0x560b342eb1b0 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b63680740 10 client.15993 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b63680740 1 -- 172.21.15.16:0/4284191393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560b34741a10 con 0x560b342eb1b0 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b3aff5640 1 -- 172.21.15.16:0/4284191393 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b4c004c20 con 0x560b342eb1b0 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b3aff5640 1 -- 172.21.15.16:0/4284191393 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6b48097210 con 0x560b342eb1b0 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b63680740 10 client.15993 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b63680740 10 client.15993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.824+0000 7f6b63680740 20 client.15993 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b63680740 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b342a7da0 0x560b34743d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b63680740 4 client.15993 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b63680740 1 -- 172.21.15.16:0/4284191393 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560b3462e020 con 0x560b342a7da0 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b5659c640 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b342a7da0 0x560b34743d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b5659c640 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b342a7da0 0x560b34743d60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16c2f2a85686b735 server_cookie=ab308d31d44c233e in_seq=0 out_seq=0 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b3aff5640 10 client.15993 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b3aff5640 1 -- 172.21.15.16:0/4284191393 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+658 (crc 0 0 0) 0x560b3462e020 con 0x560b342a7da0 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b3aff5640 10 client.15993 handle_command_reply: tid=0 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b63680740 1 client.15993 shutdown 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b63680740 1 -- 172.21.15.16:0/4284191393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 msgr2=0x560b346c8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b63680740 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346c8d20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6b480abfb0 tx=0x7f6b480abc40 comp rx=0 tx=0).stop 2024-08-28T19:22:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b63680740 1 -- 172.21.15.16:0/4284191393 shutdown_connections 2024-08-28T19:22:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b63680740 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b342a7da0 0x560b34743d60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b63680740 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b34314070 0x560b346c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b63680740 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b342f3390 0x560b346c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b63680740 1 --2- 172.21.15.16:0/4284191393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b342eb1b0 0x560b346c8d20 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.825+0000 7f6b3aff5640 0 client.15993 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b63680740 1 -- 172.21.15.16:0/4284191393 >> 172.21.15.16:0/4284191393 conn(0x560b34308460 msgr2=0x560b346bc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b63680740 1 -- 172.21.15.16:0/4284191393 shutdown_connections 2024-08-28T19:22:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b63680740 1 -- 172.21.15.16:0/4284191393 wait complete. 2024-08-28T19:22:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b63680740 20 client.15993 trim_cache size 0 max 16384 2024-08-28T19:22:28.833 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:22:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:22:28.835 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:22:28.835 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:22:28.835 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:22:28.835 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:22:28.835 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:22:28.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:22:28.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:22:28.835 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:22:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 -- 172.21.15.16:0/110234305 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b400b4ca0 msgr2=0x7f6b400b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b400b4ca0 0x7f6b400b71b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6b500094d0 tx=0x7f6b50004ab0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 -- 172.21.15.16:0/110234305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b58102800 msgr2=0x7f6b5819a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b58102800 0x7f6b5819a2e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800e9f0 tx=0x7f6b4800edf0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 -- 172.21.15.16:0/110234305 shutdown_connections 2024-08-28T19:22:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b58108a90 0x7f6b5819fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6b400b4ca0 0x7f6b400b71b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58108140 0x7f6b5819a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 --2- 172.21.15.16:0/110234305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b58102800 0x7f6b5819a2e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:28.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 -- 172.21.15.16:0/110234305 >> 172.21.15.16:0/110234305 conn(0x7f6b580794d0 msgr2=0x7f6b5806b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:28.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 -- 172.21.15.16:0/110234305 shutdown_connections 2024-08-28T19:22:28.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:28.826+0000 7f6b5d10b640 1 -- 172.21.15.16:0/110234305 wait complete. 2024-08-28T19:22:28.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 25193 19.32 MB/sec warmup 8 sec latency 14.890 ms 2024-08-28T19:22:29.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:29 smithi115 ceph-mon[22184]: pgmap v382: 129 pgs: 129 active+clean; 8.0 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 674 KiB/s wr, 89 op/s 2024-08-28T19:22:29.427 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:29 smithi016 ceph-mon[18556]: pgmap v382: 129 pgs: 129 active+clean; 8.0 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 674 KiB/s wr, 89 op/s 2024-08-28T19:22:29.457 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:22:29.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:22:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:29 smithi028 ceph-mon[21769]: pgmap v382: 129 pgs: 129 active+clean; 8.0 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 674 KiB/s wr, 89 op/s 2024-08-28T19:22:29.720 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:29.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 28434 19.15 MB/sec warmup 9 sec latency 25.589 ms 2024-08-28T19:22:30.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.209+0000 7f0e89e44640 1 -- 172.21.15.16:0/2316752879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e84108170 msgr2=0x7f0e84108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:30.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.209+0000 7f0e89e44640 1 --2- 172.21.15.16:0/2316752879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e84108170 0x7f0e84108580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009860 tx=0x7f0e6c02a340 comp rx=0 tx=0).stop 2024-08-28T19:22:30.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.209+0000 7f0e89e44640 1 -- 172.21.15.16:0/2316752879 shutdown_connections 2024-08-28T19:22:30.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.209+0000 7f0e89e44640 1 --2- 172.21.15.16:0/2316752879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e84108ac0 0x7f0e8410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:30.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.209+0000 7f0e89e44640 1 --2- 172.21.15.16:0/2316752879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e84108170 0x7f0e84108580 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:30.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.209+0000 7f0e89e44640 1 --2- 172.21.15.16:0/2316752879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e84102880 0x7f0e84102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:30.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.209+0000 7f0e89e44640 1 -- 172.21.15.16:0/2316752879 >> 172.21.15.16:0/2316752879 conn(0x7f0e84079500 msgr2=0x7f0e84079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:30.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.209+0000 7f0e89e44640 1 -- 172.21.15.16:0/2316752879 shutdown_connections 2024-08-28T19:22:30.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.209+0000 7f0e89e44640 1 -- 172.21.15.16:0/2316752879 wait complete. 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e89e44640 1 Processor -- start 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e89e44640 1 -- start start 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e89e44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e84102880 0x7f0e84198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e89e44640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e84108170 0x7f0e84199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e89e44640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e84108ac0 0x7f0e84199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e89e44640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e841126f0 con 0x7f0e84102880 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e89e44640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e84112570 con 0x7f0e84108170 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e89e44640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0e84112870 con 0x7f0e84108ac0 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e837fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e84102880 0x7f0e84198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e837fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e84102880 0x7f0e84198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50988/0 (socket says 172.21.15.16:50988) 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e837fe640 1 -- 172.21.15.16:0/2826625061 learned_addr learned my addr 172.21.15.16:0/2826625061 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.210+0000 7f0e7bfff640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e84108170 0x7f0e84199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e83fff640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e84108ac0 0x7f0e84199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e837fe640 1 -- 172.21.15.16:0/2826625061 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e84108ac0 msgr2=0x7f0e84199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e837fe640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e84108ac0 0x7f0e84199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e837fe640 1 -- 172.21.15.16:0/2826625061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e84108170 msgr2=0x7f0e84199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e837fe640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e84108170 0x7f0e84199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e837fe640 1 -- 172.21.15.16:0/2826625061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e841a8480 con 0x7f0e84102880 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e7bfff640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e84108170 0x7f0e84199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e837fe640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e84102880 0x7f0e84198dd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400e9c0 tx=0x7f0e7400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e817fa640 1 -- 172.21.15.16:0/2826625061 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7400ccc0 con 0x7f0e84102880 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e89e44640 1 -- 172.21.15.16:0/2826625061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e841a8770 con 0x7f0e84102880 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e817fa640 1 -- 172.21.15.16:0/2826625061 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e740054d0 con 0x7f0e84102880 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e817fa640 1 -- 172.21.15.16:0/2826625061 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e74010430 con 0x7f0e84102880 2024-08-28T19:22:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.211+0000 7f0e89e44640 1 -- 172.21.15.16:0/2826625061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e841a8cd0 con 0x7f0e84102880 2024-08-28T19:22:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.212+0000 7f0e89e44640 1 -- 172.21.15.16:0/2826625061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e84103ab0 con 0x7f0e84102880 2024-08-28T19:22:30.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.215+0000 7f0e817fa640 1 -- 172.21.15.16:0/2826625061 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0e74004f80 con 0x7f0e84102880 2024-08-28T19:22:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.216+0000 7f0e817fa640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e700b4ca0 0x7f0e700b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.216+0000 7f0e7bfff640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e700b4ca0 0x7f0e700b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.216+0000 7f0e817fa640 1 -- 172.21.15.16:0/2826625061 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0e740e6300 con 0x7f0e84102880 2024-08-28T19:22:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.216+0000 7f0e7bfff640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e700b4ca0 0x7f0e700b71b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009830 tx=0x7f0e6c0058d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:30.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.219+0000 7f0e817fa640 1 -- 172.21.15.16:0/2826625061 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f0e740acd80 con 0x7f0e84102880 2024-08-28T19:22:30.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.455+0000 7f0e89e44640 1 -- 172.21.15.16:0/2826625061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e84102c90 con 0x7f0e84102880 2024-08-28T19:22:30.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.456+0000 7f0e817fa640 1 -- 172.21.15.16:0/2826625061 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11977 (secure 0 0 0) 0x7f0e84102c90 con 0x7f0e84102880 2024-08-28T19:22:30.457 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:22:30.457 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"btime":"2024-08-28T19:20:55:257311+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:20:54.880238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:22:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2024-08-28T19:22:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 -- 172.21.15.16:0/2826625061 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e700b4ca0 msgr2=0x7f0e700b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e700b4ca0 0x7f0e700b71b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009830 tx=0x7f0e6c0058d0 comp rx=0 tx=0).stop 2024-08-28T19:22:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 -- 172.21.15.16:0/2826625061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e84102880 msgr2=0x7f0e84198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e84102880 0x7f0e84198dd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400e9c0 tx=0x7f0e7400ee80 comp rx=0 tx=0).stop 2024-08-28T19:22:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 -- 172.21.15.16:0/2826625061 shutdown_connections 2024-08-28T19:22:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e84108ac0 0x7f0e84199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e700b4ca0 0x7f0e700b71b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e84108170 0x7f0e84199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 --2- 172.21.15.16:0/2826625061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e84102880 0x7f0e84198dd0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 -- 172.21.15.16:0/2826625061 >> 172.21.15.16:0/2826625061 conn(0x7f0e84079500 msgr2=0x7f0e841041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 -- 172.21.15.16:0/2826625061 shutdown_connections 2024-08-28T19:22:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:30.459+0000 7f0e89e44640 1 -- 172.21.15.16:0/2826625061 wait complete. 2024-08-28T19:22:30.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 31183 18.50 MB/sec warmup 10 sec latency 23.227 ms 2024-08-28T19:22:31.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:22:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:31 smithi115 ceph-mon[22184]: pgmap v383: 129 pgs: 129 active+clean; 12 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-08-28T19:22:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:31 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2826625061' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:22:31.476 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:31 smithi016 ceph-mon[18556]: pgmap v383: 129 pgs: 129 active+clean; 12 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-08-28T19:22:31.476 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:31 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2826625061' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:22:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:31 smithi028 ceph-mon[21769]: pgmap v383: 129 pgs: 129 active+clean; 12 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-08-28T19:22:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:31 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2826625061' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:22:31.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.844+0000 7f008bcfb640 1 -- 172.21.15.16:0/1415386706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0084108a90 msgr2=0x7f008410f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:31.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.844+0000 7f008bcfb640 1 --2- 172.21.15.16:0/1415386706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0084108a90 0x7f008410f2c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f007c009ca0 tx=0x7f007c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.844+0000 7f008bcfb640 1 -- 172.21.15.16:0/1415386706 shutdown_connections 2024-08-28T19:22:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.844+0000 7f008bcfb640 1 --2- 172.21.15.16:0/1415386706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0084108a90 0x7f008410f2c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.844+0000 7f008bcfb640 1 --2- 172.21.15.16:0/1415386706 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0084108140 0x7f0084108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.844+0000 7f008bcfb640 1 --2- 172.21.15.16:0/1415386706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0084102800 0x7f0084102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.844+0000 7f008bcfb640 1 -- 172.21.15.16:0/1415386706 >> 172.21.15.16:0/1415386706 conn(0x7f00840794d0 msgr2=0x7f00840798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.844+0000 7f008bcfb640 1 -- 172.21.15.16:0/1415386706 shutdown_connections 2024-08-28T19:22:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.845+0000 7f008bcfb640 1 -- 172.21.15.16:0/1415386706 wait complete. 2024-08-28T19:22:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.845+0000 7f008bcfb640 1 Processor -- start 2024-08-28T19:22:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.845+0000 7f008bcfb640 1 -- start start 2024-08-28T19:22:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.845+0000 7f008bcfb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0084102800 0x7f008419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f008bcfb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0084108140 0x7f008419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f008bcfb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0084108a90 0x7f008419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f008bcfb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0084112710 con 0x7f0084102800 2024-08-28T19:22:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f008bcfb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0084112590 con 0x7f0084108140 2024-08-28T19:22:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f008bcfb640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0084112890 con 0x7f0084108a90 2024-08-28T19:22:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f0089a97640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0084102800 0x7f008419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f0089a97640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0084102800 0x7f008419a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50998/0 (socket says 172.21.15.16:50998) 2024-08-28T19:22:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f0089a97640 1 -- 172.21.15.16:0/418389049 learned_addr learned my addr 172.21.15.16:0/418389049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f0089296640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0084108140 0x7f008419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f008a298640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0084108a90 0x7f008419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f0089a97640 1 -- 172.21.15.16:0/418389049 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0084108a90 msgr2=0x7f008419fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f0089a97640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0084108a90 0x7f008419fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f0089a97640 1 -- 172.21.15.16:0/418389049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0084108140 msgr2=0x7f008419a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f0089a97640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0084108140 0x7f008419a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f0089a97640 1 -- 172.21.15.16:0/418389049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00841a01c0 con 0x7f0084102800 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f0089296640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0084108140 0x7f008419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f0089a97640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0084102800 0x7f008419a250 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f008000ea60 tx=0x7f008000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f007affd640 1 -- 172.21.15.16:0/418389049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008000cde0 con 0x7f0084102800 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.846+0000 7f008bcfb640 1 -- 172.21.15.16:0/418389049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00841a04b0 con 0x7f0084102800 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.847+0000 7f007affd640 1 -- 172.21.15.16:0/418389049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00800035f0 con 0x7f0084102800 2024-08-28T19:22:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.847+0000 7f007affd640 1 -- 172.21.15.16:0/418389049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080010690 con 0x7f0084102800 2024-08-28T19:22:31.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.847+0000 7f008bcfb640 1 -- 172.21.15.16:0/418389049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00841a6c10 con 0x7f0084102800 2024-08-28T19:22:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.849+0000 7f007affd640 1 -- 172.21.15.16:0/418389049 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f00800037a0 con 0x7f0084102800 2024-08-28T19:22:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.849+0000 7f0092270740 1 Processor -- start 2024-08-28T19:22:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f0092270740 1 -- start start 2024-08-28T19:22:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f0092270740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 0x556abc6ea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f0092270740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 0x556abc6eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f0092270740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 0x556abc6eb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f0092270740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556abc65d680 con 0x556abc320390 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f0092270740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556abc64cc10 con 0x556abc341070 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f0092270740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556abc650bc0 con 0x556abc3181b0 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f0089296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 0x556abc6eb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f008a298640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 0x556abc6ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f008a298640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 0x556abc6ea760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41564/0 (socket says 172.21.15.16:41564) 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f0089296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 0x556abc6eb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51014/0 (socket says 172.21.15.16:51014) 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f008a298640 1 -- 172.21.15.16:0/3121157218 learned_addr learned my addr 172.21.15.16:0/3121157218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f0089a97640 1 --2- 172.21.15.16:0/3121157218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 0x556abc6eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f007affd640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00700b4ca0 0x7f00700b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f008a298640 1 -- 172.21.15.16:0/3121157218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 msgr2=0x556abc6eaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f008a298640 1 --2- 172.21.15.16:0/3121157218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 0x556abc6eaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f008a298640 1 -- 172.21.15.16:0/3121157218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 msgr2=0x556abc6eb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:31.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f008a298640 1 --2- 172.21.15.16:0/3121157218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 0x556abc6eb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f008a298640 1 -- 172.21.15.16:0/3121157218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556abc3ac650 con 0x556abc341070 2024-08-28T19:22:31.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.850+0000 7f0089a97640 1 --2- 172.21.15.16:0/3121157218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 0x556abc6eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:31.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.851+0000 7f008a298640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00700b4ca0 0x7f00700b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.851+0000 7f007affd640 1 -- 172.21.15.16:0/418389049 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f00800e5bb0 con 0x7f0084102800 2024-08-28T19:22:31.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.851+0000 7f008a298640 1 --2- 172.21.15.16:0/3121157218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 0x556abc6ea760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f007c009820 tx=0x7f007c009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9b2001f2386a137 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:31.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.851+0000 7f006ffff640 1 -- 172.21.15.16:0/3121157218 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c042040 con 0x556abc341070 2024-08-28T19:22:31.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.851+0000 7f008a298640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00700b4ca0 0x7f00700b71b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f007c04d000 tx=0x7f007c04e000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:31.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.851+0000 7f006ffff640 1 -- 172.21.15.16:0/3121157218 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f007c036070 con 0x556abc341070 2024-08-28T19:22:31.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.851+0000 7f006ffff640 1 -- 172.21.15.16:0/3121157218 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c02f660 con 0x556abc341070 2024-08-28T19:22:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.852+0000 7f0092270740 1 -- 172.21.15.16:0/3121157218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 msgr2=0x556abc6ea760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.852+0000 7f0092270740 1 --2- 172.21.15.16:0/3121157218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 0x556abc6ea760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f007c009820 tx=0x7f007c009850 comp rx=0 tx=0).stop 2024-08-28T19:22:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.852+0000 7f0092270740 1 -- 172.21.15.16:0/3121157218 shutdown_connections 2024-08-28T19:22:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.852+0000 7f0092270740 1 --2- 172.21.15.16:0/3121157218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 0x556abc6eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.852+0000 7f0092270740 1 --2- 172.21.15.16:0/3121157218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 0x556abc6eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.852+0000 7f0092270740 1 --2- 172.21.15.16:0/3121157218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 0x556abc6ea760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.852+0000 7f0092270740 1 -- 172.21.15.16:0/3121157218 >> 172.21.15.16:0/3121157218 conn(0x556abc335460 msgr2=0x556abc6e8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.852+0000 7f0092270740 1 -- 172.21.15.16:0/3121157218 shutdown_connections 2024-08-28T19:22:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.852+0000 7f0092270740 1 -- 172.21.15.16:0/3121157218 wait complete. 2024-08-28T19:22:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.852+0000 7f0092270740 1 -- start start 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0092270740 1 Processor -- start 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0092270740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 0x556abc6f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0092270740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 0x556abc6f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0092270740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 0x556abc702230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0092270740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556abc4af700 con 0x556abc341070 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0092270740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556abc5d3750 con 0x556abc3181b0 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0092270740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556abc6f8f40 con 0x556abc320390 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0092270740 10 client.0 _ll_register_callbacks cb 0x556abc602e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0089a97640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 0x556abc6f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0089a97640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 0x556abc6f5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41572/0 (socket says 172.21.15.16:41572) 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0089a97640 1 -- 172.21.15.16:0/2134950354 learned_addr learned my addr 172.21.15.16:0/2134950354 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0089296640 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 0x556abc6f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f008a298640 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 0x556abc702230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.853+0000 7f0089a97640 1 -- 172.21.15.16:0/2134950354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 msgr2=0x556abc6f63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f0089a97640 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 0x556abc6f63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f0089a97640 1 -- 172.21.15.16:0/2134950354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 msgr2=0x556abc702230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f0089a97640 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 0x556abc702230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f0089a97640 1 -- 172.21.15.16:0/2134950354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556abc60c900 con 0x556abc3181b0 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f0089296640 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 0x556abc6f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f0089a97640 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 0x556abc6f5e90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0080009920 tx=0x7f00800095a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2e2c1fb3ed289e2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f006d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f006d7fa640 1 -- 172.21.15.16:0/2134950354 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00800a7df0 con 0x556abc3181b0 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f0092270740 1 -- 172.21.15.16:0/2134950354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f007c036070 con 0x556abc3181b0 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f006d7fa640 1 -- 172.21.15.16:0/2134950354 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f00800932e0 con 0x556abc3181b0 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f006d7fa640 1 -- 172.21.15.16:0/2134950354 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f00800aba40 con 0x556abc3181b0 2024-08-28T19:22:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f0092270740 10 client.25649 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f0092270740 1 -- 172.21.15.16:0/2134950354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556abc704d80 con 0x556abc3181b0 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.854+0000 7f006d7fa640 1 -- 172.21.15.16:0/2134950354 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f007c036070 con 0x556abc3181b0 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.855+0000 7f006d7fa640 1 -- 172.21.15.16:0/2134950354 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f00800b0390 con 0x556abc3181b0 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.855+0000 7f0092270740 10 client.25649 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.855+0000 7f0092270740 10 client.25649 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.855+0000 7f0092270740 20 client.25649 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.855+0000 7f0092270740 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x556abc2ff9f0 0x556abc708260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.855+0000 7f0092270740 4 client.25649 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.855+0000 7f0092270740 1 -- 172.21.15.16:0/2134950354 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556abc65f5d0 con 0x556abc2ff9f0 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.855+0000 7f0089296640 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x556abc2ff9f0 0x556abc708260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.855+0000 7f006d7fa640 10 client.25649 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.855+0000 7f0089296640 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x556abc2ff9f0 0x556abc708260 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7742c5e124915a43 server_cookie=3fe17ad05906f698 in_seq=0 out_seq=0 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f006d7fa640 1 -- 172.21.15.16:0/2134950354 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556abc65f5d0 con 0x556abc2ff9f0 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f006d7fa640 10 client.25649 handle_command_reply: tid=0 2024-08-28T19:22:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f0092270740 1 client.25649 shutdown 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f0092270740 1 -- 172.21.15.16:0/2134950354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 msgr2=0x556abc6f5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f0092270740 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 0x556abc6f5e90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0080009920 tx=0x7f00800095a0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f0092270740 1 -- 172.21.15.16:0/2134950354 shutdown_connections 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f0092270740 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x556abc2ff9f0 0x556abc708260 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f0092270740 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 0x556abc702230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f0092270740 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 0x556abc6f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f0092270740 1 --2- 172.21.15.16:0/2134950354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 0x556abc6f5e90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f006d7fa640 0 client.25649 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f0092270740 1 -- 172.21.15.16:0/2134950354 >> 172.21.15.16:0/2134950354 conn(0x556abc335460 msgr2=0x556abc6ed940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f0092270740 1 -- 172.21.15.16:0/2134950354 shutdown_connections 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.856+0000 7f0092270740 1 -- 172.21.15.16:0/2134950354 wait complete. 2024-08-28T19:22:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.857+0000 7f0092270740 20 client.25649 trim_cache size 0 max 16384 2024-08-28T19:22:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.870+0000 7f0092270740 1 Processor -- start 2024-08-28T19:22:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.870+0000 7f0092270740 1 -- start start 2024-08-28T19:22:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0092270740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 0x556abc702130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0092270740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 0x556abc702670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0092270740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 0x556abc6f3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0092270740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00841ae820 con 0x556abc341070 2024-08-28T19:22:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0092270740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00841b0360 con 0x556abc3181b0 2024-08-28T19:22:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0092270740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00841aeef0 con 0x556abc320390 2024-08-28T19:22:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f008a298640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 0x556abc6f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f008a298640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 0x556abc6f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51044/0 (socket says 172.21.15.16:51044) 2024-08-28T19:22:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f008a298640 1 -- 172.21.15.16:0/3373564350 learned_addr learned my addr 172.21.15.16:0/3373564350 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0089296640 1 --2- 172.21.15.16:0/3373564350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 0x556abc702670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0089a97640 1 --2- 172.21.15.16:0/3373564350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 0x556abc702130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0089296640 1 -- 172.21.15.16:0/3373564350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 msgr2=0x556abc702130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0089296640 1 --2- 172.21.15.16:0/3373564350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 0x556abc702130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0089296640 1 -- 172.21.15.16:0/3373564350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 msgr2=0x556abc6f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0089296640 1 --2- 172.21.15.16:0/3373564350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 0x556abc6f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0089296640 1 -- 172.21.15.16:0/3373564350 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556abc6ecca0 con 0x556abc320390 2024-08-28T19:22:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f008a298640 1 --2- 172.21.15.16:0/3373564350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 0x556abc6f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.871+0000 7f0089296640 1 --2- 172.21.15.16:0/3373564350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 0x556abc702670 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x556abc7089a0 tx=0x7f007400b390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b1f9fee85fdb11a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.872+0000 7f006f7fe640 1 -- 172.21.15.16:0/3373564350 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0074005220 con 0x556abc320390 2024-08-28T19:22:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.872+0000 7f006f7fe640 1 -- 172.21.15.16:0/3373564350 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0074004c50 con 0x556abc320390 2024-08-28T19:22:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.872+0000 7f006f7fe640 1 -- 172.21.15.16:0/3373564350 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0074003b20 con 0x556abc320390 2024-08-28T19:22:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.872+0000 7f0092270740 1 -- 172.21.15.16:0/3373564350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 msgr2=0x556abc702670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.872+0000 7f0092270740 1 --2- 172.21.15.16:0/3373564350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 0x556abc702670 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x556abc7089a0 tx=0x7f007400b390 comp rx=0 tx=0).stop 2024-08-28T19:22:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.872+0000 7f0092270740 1 -- 172.21.15.16:0/3373564350 shutdown_connections 2024-08-28T19:22:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.872+0000 7f0092270740 1 --2- 172.21.15.16:0/3373564350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc341070 0x556abc6f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.872+0000 7f0092270740 1 --2- 172.21.15.16:0/3373564350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc320390 0x556abc702670 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.872+0000 7f0092270740 1 --2- 172.21.15.16:0/3373564350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc3181b0 0x556abc702130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.872+0000 7f0092270740 1 -- 172.21.15.16:0/3373564350 >> 172.21.15.16:0/3373564350 conn(0x556abc335460 msgr2=0x556abc6f5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.873+0000 7f0092270740 1 -- 172.21.15.16:0/3373564350 shutdown_connections 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.873+0000 7f0092270740 1 -- 172.21.15.16:0/3373564350 wait complete. 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.873+0000 7f0092270740 1 -- start start 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.873+0000 7f0092270740 1 Processor -- start 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.873+0000 7f0092270740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 0x556abc6fa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.873+0000 7f0092270740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 0x556abc6fa990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.873+0000 7f0092270740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 0x556abc771c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.873+0000 7f0092270740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00841afc90 con 0x556abc320390 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.873+0000 7f0092270740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00841af5c0 con 0x556abc341070 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.873+0000 7f0092270740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00841ac800 con 0x556abc3181b0 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0092270740 10 client.0 _ll_register_callbacks cb 0x7f0080003d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0089a97640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 0x556abc6fa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0089296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 0x556abc6fa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0089a97640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 0x556abc6fa450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34106/0 (socket says 172.21.15.16:34106) 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0089296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 0x556abc6fa990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51060/0 (socket says 172.21.15.16:51060) 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0089a97640 1 -- 172.21.15.16:0/2252610136 learned_addr learned my addr 172.21.15.16:0/2252610136 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f008a298640 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 0x556abc771c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0089a97640 1 -- 172.21.15.16:0/2252610136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 msgr2=0x556abc771c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0089a97640 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 0x556abc771c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0089a97640 1 -- 172.21.15.16:0/2252610136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 msgr2=0x556abc6fa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0089a97640 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 0x556abc6fa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0089a97640 1 -- 172.21.15.16:0/2252610136 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556abc6fb210 con 0x556abc3181b0 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0089a97640 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 0x556abc6fa450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f00800e3590 tx=0x7f00800972f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6701653cfd372c89 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f006d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:22:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f006d7fa640 1 -- 172.21.15.16:0/2252610136 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00800a7df0 con 0x556abc3181b0 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f006d7fa640 1 -- 172.21.15.16:0/2252610136 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f00800ab390 con 0x556abc3181b0 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.874+0000 7f0092270740 1 -- 172.21.15.16:0/2252610136 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0074004c50 con 0x556abc3181b0 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f006d7fa640 1 -- 172.21.15.16:0/2252610136 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f00800ac630 con 0x556abc3181b0 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f0092270740 10 client.25549 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f0092270740 1 -- 172.21.15.16:0/2252610136 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556abc7731e0 con 0x556abc3181b0 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f006d7fa640 1 -- 172.21.15.16:0/2252610136 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0074004c50 con 0x556abc3181b0 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f006d7fa640 1 -- 172.21.15.16:0/2252610136 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f00800c6ce0 con 0x556abc3181b0 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f0092270740 10 client.25549 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f0092270740 10 client.25549 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f0092270740 20 client.25549 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f0092270740 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x556abc2d4da0 0x556abc775910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f0092270740 4 client.25549 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f0092270740 1 -- 172.21.15.16:0/2252610136 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556abc65b070 con 0x556abc2d4da0 2024-08-28T19:22:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.875+0000 7f0089296640 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x556abc2d4da0 0x556abc775910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f006d7fa640 10 client.25549 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0089296640 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x556abc2d4da0 0x556abc775910 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=788c42a91ea9c565 server_cookie=30ad2452aba36c93 in_seq=0 out_seq=0 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f006d7fa640 1 -- 172.21.15.16:0/2252610136 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556abc65b070 con 0x556abc2d4da0 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f006d7fa640 10 client.25549 handle_command_reply: tid=0 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 1 client.25549 shutdown 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 1 -- 172.21.15.16:0/2252610136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 msgr2=0x556abc6fa450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 0x556abc6fa450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f00800e3590 tx=0x7f00800972f0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 1 -- 172.21.15.16:0/2252610136 shutdown_connections 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x556abc2d4da0 0x556abc775910 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556abc341070 0x556abc771c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556abc320390 0x556abc6fa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f006d7fa640 0 client.25549 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:22:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 1 --2- 172.21.15.16:0/2252610136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556abc3181b0 0x556abc6fa450 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 1 -- 172.21.15.16:0/2252610136 >> 172.21.15.16:0/2252610136 conn(0x556abc335460 msgr2=0x556abc6eb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:31.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 1 -- 172.21.15.16:0/2252610136 shutdown_connections 2024-08-28T19:22:31.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 1 -- 172.21.15.16:0/2252610136 wait complete. 2024-08-28T19:22:31.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.876+0000 7f0092270740 20 client.25549 trim_cache size 0 max 16384 2024-08-28T19:22:31.883 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:22:31.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 -- 172.21.15.16:0/418389049 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00700b4ca0 msgr2=0x7f00700b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:31.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00700b4ca0 0x7f00700b71b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f007c04d000 tx=0x7f007c04e000 comp rx=0 tx=0).stop 2024-08-28T19:22:31.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 -- 172.21.15.16:0/418389049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0084102800 msgr2=0x7f008419a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:31.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0084102800 0x7f008419a250 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f008000ea60 tx=0x7f008000ef20 comp rx=0 tx=0).stop 2024-08-28T19:22:31.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 -- 172.21.15.16:0/418389049 shutdown_connections 2024-08-28T19:22:31.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0084108a90 0x7f008419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00700b4ca0 0x7f00700b71b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0084108140 0x7f008419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 --2- 172.21.15.16:0/418389049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0084102800 0x7f008419a250 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:31.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 -- 172.21.15.16:0/418389049 >> 172.21.15.16:0/418389049 conn(0x7f00840794d0 msgr2=0x7f0084101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 -- 172.21.15.16:0/418389049 shutdown_connections 2024-08-28T19:22:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:31.877+0000 7f008bcfb640 1 -- 172.21.15.16:0/418389049 wait complete. 2024-08-28T19:22:31.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 33740 18.04 MB/sec warmup 11 sec latency 36.394 ms 2024-08-28T19:22:32.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:22:32.861 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:32.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 36581 17.85 MB/sec warmup 12 sec latency 23.141 ms 2024-08-28T19:22:33.245 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:33 smithi016 ceph-mon[18556]: pgmap v384: 129 pgs: 129 active+clean; 13 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.2 MiB/s wr, 155 op/s 2024-08-28T19:22:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.339+0000 7f478efe6640 1 -- 172.21.15.16:0/1268261263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478810dbd0 msgr2=0x7f47881100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.339+0000 7f478efe6640 1 --2- 172.21.15.16:0/1268261263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478810dbd0 0x7f47881100a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4780009ca0 tx=0x7f478002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.339+0000 7f478efe6640 1 -- 172.21.15.16:0/1268261263 shutdown_connections 2024-08-28T19:22:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.339+0000 7f478efe6640 1 --2- 172.21.15.16:0/1268261263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478810dbd0 0x7f47881100a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.339+0000 7f478efe6640 1 --2- 172.21.15.16:0/1268261263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4788105050 0x7f478810d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.339+0000 7f478efe6640 1 --2- 172.21.15.16:0/1268261263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4788104620 0x7f4788104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.339+0000 7f478efe6640 1 -- 172.21.15.16:0/1268261263 >> 172.21.15.16:0/1268261263 conn(0x7f47880794d0 msgr2=0x7f47880798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.339+0000 7f478efe6640 1 -- 172.21.15.16:0/1268261263 shutdown_connections 2024-08-28T19:22:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.339+0000 7f478efe6640 1 -- 172.21.15.16:0/1268261263 wait complete. 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478efe6640 1 Processor -- start 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478efe6640 1 -- start start 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478efe6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4788104620 0x7f478819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478efe6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4788105050 0x7f478819a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478efe6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f478810dbd0 0x7f478819fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478efe6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4788113810 con 0x7f4788104620 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478efe6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4788113690 con 0x7f4788105050 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478efe6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4788113990 con 0x7f478810dbd0 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478cd82640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4788104620 0x7f478819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478cd82640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4788104620 0x7f478819a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51092/0 (socket says 172.21.15.16:51092) 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478cd82640 1 -- 172.21.15.16:0/1552550879 learned_addr learned my addr 172.21.15.16:0/1552550879 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f4787fff640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4788105050 0x7f478819a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478d583640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f478810dbd0 0x7f478819fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478cd82640 1 -- 172.21.15.16:0/1552550879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f478810dbd0 msgr2=0x7f478819fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478cd82640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f478810dbd0 0x7f478819fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478cd82640 1 -- 172.21.15.16:0/1552550879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4788105050 msgr2=0x7f478819a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478cd82640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4788105050 0x7f478819a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.340+0000 7f478cd82640 1 -- 172.21.15.16:0/1552550879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47881a01f0 con 0x7f4788104620 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.341+0000 7f478d583640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f478810dbd0 0x7f478819fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.341+0000 7f4787fff640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4788105050 0x7f478819a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.341+0000 7f478cd82640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4788104620 0x7f478819a280 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f477c00e9c0 tx=0x7f477c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.341+0000 7f4785ffb640 1 -- 172.21.15.16:0/1552550879 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c00cd80 con 0x7f4788104620 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.341+0000 7f4785ffb640 1 -- 172.21.15.16:0/1552550879 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f477c0035f0 con 0x7f4788104620 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.341+0000 7f478efe6640 1 -- 172.21.15.16:0/1552550879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47881a04e0 con 0x7f4788104620 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.341+0000 7f4785ffb640 1 -- 172.21.15.16:0/1552550879 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c010690 con 0x7f4788104620 2024-08-28T19:22:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.341+0000 7f478efe6640 1 -- 172.21.15.16:0/1552550879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47881a6c40 con 0x7f4788104620 2024-08-28T19:22:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.344+0000 7f479555b740 1 Processor -- start 2024-08-28T19:22:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f4785ffb640 1 -- 172.21.15.16:0/1552550879 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f477c0037a0 con 0x7f4788104620 2024-08-28T19:22:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f479555b740 1 -- start start 2024-08-28T19:22:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f479555b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 0x5588db0c3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f479555b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacf51b0 0x5588db0c3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f479555b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacfd390 0x5588db0c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f479555b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5588db03a680 con 0x5588dad1e070 2024-08-28T19:22:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f479555b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5588db029c10 con 0x5588dacfd390 2024-08-28T19:22:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f479555b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5588db02dbc0 con 0x5588dacf51b0 2024-08-28T19:22:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f478d583640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 0x5588db0c3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f478d583640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 0x5588db0c3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51102/0 (socket says 172.21.15.16:51102) 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f478d583640 1 -- 172.21.15.16:0/3723485126 learned_addr learned my addr 172.21.15.16:0/3723485126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f4787fff640 1 --2- 172.21.15.16:0/3723485126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacfd390 0x5588db0c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f478cd82640 1 --2- 172.21.15.16:0/3723485126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacf51b0 0x5588db0c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f4787fff640 1 -- 172.21.15.16:0/3723485126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacf51b0 msgr2=0x5588db0c3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f4787fff640 1 --2- 172.21.15.16:0/3723485126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacf51b0 0x5588db0c3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f4787fff640 1 -- 172.21.15.16:0/3723485126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 msgr2=0x5588db0c3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f4787fff640 1 --2- 172.21.15.16:0/3723485126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 0x5588db0c3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.345+0000 7f4787fff640 1 -- 172.21.15.16:0/3723485126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588dad89650 con 0x5588dacfd390 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.346+0000 7f478d583640 1 --2- 172.21.15.16:0/3723485126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 0x5588db0c3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.346+0000 7f4785ffb640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47780b4ca0 0x7f47780b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.346+0000 7f4787fff640 1 --2- 172.21.15.16:0/3723485126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacfd390 0x5588db0c3d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f478806aad0 tx=0x7f47740118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c60f0742f123279 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.346+0000 7f478d583640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47780b4ca0 0x7f47780b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.346+0000 7f4772ffd640 1 -- 172.21.15.16:0/3723485126 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774004310 con 0x5588dacfd390 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.346+0000 7f4785ffb640 1 -- 172.21.15.16:0/1552550879 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f477c0e5d00 con 0x7f4788104620 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.346+0000 7f4772ffd640 1 -- 172.21.15.16:0/3723485126 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47740044b0 con 0x5588dacfd390 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.346+0000 7f4772ffd640 1 -- 172.21.15.16:0/3723485126 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47740035f0 con 0x5588dacfd390 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.347+0000 7f479555b740 1 -- 172.21.15.16:0/3723485126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacfd390 msgr2=0x5588db0c3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.347+0000 7f478d583640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47780b4ca0 0x7f47780b71b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f478002f500 tx=0x7f4780005d80 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.347+0000 7f479555b740 1 --2- 172.21.15.16:0/3723485126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacfd390 0x5588db0c3d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f478806aad0 tx=0x7f47740118d0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.347+0000 7f479555b740 1 -- 172.21.15.16:0/3723485126 shutdown_connections 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.347+0000 7f479555b740 1 --2- 172.21.15.16:0/3723485126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacfd390 0x5588db0c3d90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.347+0000 7f479555b740 1 --2- 172.21.15.16:0/3723485126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacf51b0 0x5588db0c3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.347+0000 7f479555b740 1 --2- 172.21.15.16:0/3723485126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 0x5588db0c3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.347+0000 7f479555b740 1 -- 172.21.15.16:0/3723485126 >> 172.21.15.16:0/3723485126 conn(0x5588dad12460 msgr2=0x5588db0c1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.347+0000 7f479555b740 1 -- 172.21.15.16:0/3723485126 shutdown_connections 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.347+0000 7f479555b740 1 -- 172.21.15.16:0/3723485126 wait complete. 2024-08-28T19:22:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.348+0000 7f479555b740 1 -- start start 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.348+0000 7f479555b740 1 Processor -- start 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.348+0000 7f479555b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacf51b0 0x5588db0d2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.348+0000 7f479555b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacfd390 0x5588db0d32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.348+0000 7f479555b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 0x5588db0df180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.348+0000 7f479555b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5588dae8c700 con 0x5588dad1e070 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.348+0000 7f479555b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5588dafb0750 con 0x5588dacf51b0 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.348+0000 7f479555b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5588db0d5e20 con 0x5588dacfd390 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.348+0000 7f479555b740 10 client.0 _ll_register_callbacks cb 0x5588dafdfe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f478d583640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 0x5588db0df180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f478cd82640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacf51b0 0x5588db0d2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f4787fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacfd390 0x5588db0d32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f478cd82640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacf51b0 0x5588db0d2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41624/0 (socket says 172.21.15.16:41624) 2024-08-28T19:22:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f478cd82640 1 -- 172.21.15.16:0/2641316278 learned_addr learned my addr 172.21.15.16:0/2641316278 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f478d583640 1 -- 172.21.15.16:0/2641316278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacfd390 msgr2=0x5588db0d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f478d583640 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacfd390 0x5588db0d32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f478d583640 1 -- 172.21.15.16:0/2641316278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacf51b0 msgr2=0x5588db0d2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f478d583640 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacf51b0 0x5588db0d2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f478d583640 1 -- 172.21.15.16:0/2641316278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588dafe9900 con 0x5588dad1e070 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f478cd82640 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacf51b0 0x5588db0d2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f4787fff640 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacfd390 0x5588db0d32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f478d583640 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 0x5588db0df180 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4780009a60 tx=0x7f478002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47dc92ff24fc4b05 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f4767fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f4767fff640 1 -- 172.21.15.16:0/2641316278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47800318e0 con 0x5588dad1e070 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f4767fff640 1 -- 172.21.15.16:0/2641316278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f4780004250 con 0x5588dad1e070 2024-08-28T19:22:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.349+0000 7f479555b740 1 -- 172.21.15.16:0/2641316278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47740044b0 con 0x5588dad1e070 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f4767fff640 1 -- 172.21.15.16:0/2641316278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4780004450 con 0x5588dad1e070 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f479555b740 10 client.16029 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f479555b740 1 -- 172.21.15.16:0/2641316278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588db0e1b70 con 0x5588dad1e070 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f4767fff640 1 -- 172.21.15.16:0/2641316278 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47740044b0 con 0x5588dad1e070 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f4767fff640 1 -- 172.21.15.16:0/2641316278 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4780050070 con 0x5588dad1e070 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f479555b740 10 client.16029 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f479555b740 10 client.16029 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f479555b740 20 client.16029 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f479555b740 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5588dacdc9f0 0x5588db0e5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f479555b740 4 client.16029 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f479555b740 1 -- 172.21.15.16:0/2641316278 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588db03c5d0 con 0x5588dacdc9f0 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.350+0000 7f478cd82640 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5588dacdc9f0 0x5588db0e5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.351+0000 7f478cd82640 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5588dacdc9f0 0x5588db0e5010 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ccf3839bdaf0fc server_cookie=30ae0ab0a1c81dc0 in_seq=0 out_seq=0 2024-08-28T19:22:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.351+0000 7f4767fff640 10 client.16029 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:22:33.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.351+0000 7f4767fff640 1 -- 172.21.15.16:0/2641316278 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5588db03c5d0 con 0x5588dacdc9f0 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.351+0000 7f4767fff640 10 client.16029 handle_command_reply: tid=0 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 1 client.16029 shutdown 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 1 -- 172.21.15.16:0/2641316278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 msgr2=0x5588db0df180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 0x5588db0df180 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4780009a60 tx=0x7f478002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 1 -- 172.21.15.16:0/2641316278 shutdown_connections 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5588dacdc9f0 0x5588db0e5010 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f4767fff640 0 client.16029 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dad1e070 0x5588db0df180 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacfd390 0x5588db0d32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 1 --2- 172.21.15.16:0/2641316278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacf51b0 0x5588db0d2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 1 -- 172.21.15.16:0/2641316278 >> 172.21.15.16:0/2641316278 conn(0x5588dad12460 msgr2=0x5588db0c63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 1 -- 172.21.15.16:0/2641316278 shutdown_connections 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 1 -- 172.21.15.16:0/2641316278 wait complete. 2024-08-28T19:22:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.352+0000 7f479555b740 20 client.16029 trim_cache size 0 max 16384 2024-08-28T19:22:33.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.365+0000 7f479555b740 1 Processor -- start 2024-08-28T19:22:33.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.365+0000 7f479555b740 1 -- start start 2024-08-28T19:22:33.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f479555b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacdc9f0 0x5588db0df080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f479555b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f479555b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacfd390 0x5588db0c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f479555b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47881aea50 con 0x5588dacf51b0 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f479555b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47881b0590 con 0x5588dacdc9f0 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f479555b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f47881af120 con 0x5588dacfd390 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f4787fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f4787fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51112/0 (socket says 172.21.15.16:51112) 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f4787fff640 1 -- 172.21.15.16:0/2126781883 learned_addr learned my addr 172.21.15.16:0/2126781883 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f478cd82640 1 --2- 172.21.15.16:0/2126781883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacdc9f0 0x5588db0df080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f478d583640 1 --2- 172.21.15.16:0/2126781883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacfd390 0x5588db0c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f4787fff640 1 -- 172.21.15.16:0/2126781883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacfd390 msgr2=0x5588db0c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f4787fff640 1 --2- 172.21.15.16:0/2126781883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacfd390 0x5588db0c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f4787fff640 1 -- 172.21.15.16:0/2126781883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacdc9f0 msgr2=0x5588db0df080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:33.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f4787fff640 1 --2- 172.21.15.16:0/2126781883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacdc9f0 0x5588db0df080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f4787fff640 1 -- 172.21.15.16:0/2126781883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588db0c6110 con 0x5588dacf51b0 2024-08-28T19:22:33.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.366+0000 7f478cd82640 1 --2- 172.21.15.16:0/2126781883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacdc9f0 0x5588db0df080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:33.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.367+0000 7f4787fff640 1 --2- 172.21.15.16:0/2126781883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0c63b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4774004c40 tx=0x7f4774009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a168d6a8ce28f8f6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.367+0000 7f47727fc640 1 -- 172.21.15.16:0/2126781883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774028070 con 0x5588dacf51b0 2024-08-28T19:22:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.367+0000 7f47727fc640 1 -- 172.21.15.16:0/2126781883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f477401f040 con 0x5588dacf51b0 2024-08-28T19:22:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.367+0000 7f47727fc640 1 -- 172.21.15.16:0/2126781883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774005530 con 0x5588dacf51b0 2024-08-28T19:22:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.368+0000 7f479555b740 1 -- 172.21.15.16:0/2126781883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 msgr2=0x5588db0c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.368+0000 7f479555b740 1 --2- 172.21.15.16:0/2126781883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0c63b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4774004c40 tx=0x7f4774009b30 comp rx=0 tx=0).stop 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.368+0000 7f479555b740 1 -- 172.21.15.16:0/2126781883 shutdown_connections 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.368+0000 7f479555b740 1 --2- 172.21.15.16:0/2126781883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacfd390 0x5588db0c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.368+0000 7f479555b740 1 --2- 172.21.15.16:0/2126781883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0c63b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.368+0000 7f479555b740 1 --2- 172.21.15.16:0/2126781883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacdc9f0 0x5588db0df080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.368+0000 7f479555b740 1 -- 172.21.15.16:0/2126781883 >> 172.21.15.16:0/2126781883 conn(0x5588dad12460 msgr2=0x5588db0d1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.368+0000 7f479555b740 1 -- 172.21.15.16:0/2126781883 shutdown_connections 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.368+0000 7f479555b740 1 -- 172.21.15.16:0/2126781883 wait complete. 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.368+0000 7f479555b740 1 -- start start 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.368+0000 7f479555b740 1 Processor -- start 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f479555b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacdc9f0 0x5588db0d71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f479555b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0d7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f479555b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacfd390 0x5588db0d7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f479555b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47881afec0 con 0x5588dacf51b0 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f479555b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47881af7f0 con 0x5588dacfd390 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f479555b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f47881aca30 con 0x5588dacdc9f0 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f4787fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0d7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f479555b740 10 client.0 _ll_register_callbacks cb 0x7f4774005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f4787fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0d7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51128/0 (socket says 172.21.15.16:51128) 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f4787fff640 1 -- 172.21.15.16:0/3026110540 learned_addr learned my addr 172.21.15.16:0/3026110540 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f478cd82640 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacdc9f0 0x5588db0d71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f478d583640 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacfd390 0x5588db0d7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f4787fff640 1 -- 172.21.15.16:0/3026110540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacdc9f0 msgr2=0x5588db0d71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f4787fff640 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacdc9f0 0x5588db0d71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f4787fff640 1 -- 172.21.15.16:0/3026110540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacfd390 msgr2=0x5588db0d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f4787fff640 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacfd390 0x5588db0d7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.369+0000 7f4787fff640 1 -- 172.21.15.16:0/3026110540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588db14ead0 con 0x5588dacf51b0 2024-08-28T19:22:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.370+0000 7f4787fff640 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0d7700 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4774011dd0 tx=0x7f4774005070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea192f7f8e99bb9e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.370+0000 7f4767fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.370+0000 7f4767fff640 1 -- 172.21.15.16:0/3026110540 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774031070 con 0x5588dacf51b0 2024-08-28T19:22:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.370+0000 7f4767fff640 1 -- 172.21.15.16:0/3026110540 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f4774004420 con 0x5588dacf51b0 2024-08-28T19:22:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.370+0000 7f479555b740 1 -- 172.21.15.16:0/3026110540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f477401f040 con 0x5588dacf51b0 2024-08-28T19:22:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.370+0000 7f4767fff640 1 -- 172.21.15.16:0/3026110540 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f477403d050 con 0x5588dacf51b0 2024-08-28T19:22:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.370+0000 7f479555b740 10 client.16035 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.370+0000 7f479555b740 1 -- 172.21.15.16:0/3026110540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588db14fd50 con 0x5588dacf51b0 2024-08-28T19:22:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.370+0000 7f4767fff640 1 -- 172.21.15.16:0/3026110540 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f477401f040 con 0x5588dacf51b0 2024-08-28T19:22:33.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:33 smithi115 ceph-mon[22184]: pgmap v384: 129 pgs: 129 active+clean; 13 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.2 MiB/s wr, 155 op/s 2024-08-28T19:22:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.370+0000 7f4767fff640 1 -- 172.21.15.16:0/3026110540 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4774018070 con 0x5588dacf51b0 2024-08-28T19:22:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.370+0000 7f479555b740 10 client.16035 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.371+0000 7f479555b740 10 client.16035 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:22:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.371+0000 7f479555b740 20 client.16035 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.371+0000 7f479555b740 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5588dad1e070 0x5588db152240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.371+0000 7f479555b740 4 client.16035 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:22:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.371+0000 7f479555b740 1 -- 172.21.15.16:0/3026110540 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5588db038070 con 0x5588dad1e070 2024-08-28T19:22:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.371+0000 7f478cd82640 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5588dad1e070 0x5588db152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.371+0000 7f478cd82640 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5588dad1e070 0x5588db152240 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d5303083c8c4fd53 server_cookie=4c613507d23dc142 in_seq=0 out_seq=0 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.371+0000 7f4767fff640 10 client.16035 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.371+0000 7f4767fff640 1 -- 172.21.15.16:0/3026110540 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5588db038070 con 0x5588dad1e070 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.371+0000 7f4767fff640 10 client.16035 handle_command_reply: tid=0 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.371+0000 7f479555b740 1 client.16035 shutdown 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f479555b740 1 -- 172.21.15.16:0/3026110540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 msgr2=0x5588db0d7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f479555b740 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0d7700 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4774011dd0 tx=0x7f4774005070 comp rx=0 tx=0).stop 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f479555b740 1 -- 172.21.15.16:0/3026110540 shutdown_connections 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f479555b740 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5588dad1e070 0x5588db152240 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f479555b740 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588dacfd390 0x5588db0d7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f479555b740 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5588dacf51b0 0x5588db0d7700 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f479555b740 1 --2- 172.21.15.16:0/3026110540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5588dacdc9f0 0x5588db0d71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f4767fff640 0 client.16035 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:22:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f479555b740 1 -- 172.21.15.16:0/3026110540 >> 172.21.15.16:0/3026110540 conn(0x5588dad12460 msgr2=0x5588db0d13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f479555b740 1 -- 172.21.15.16:0/3026110540 shutdown_connections 2024-08-28T19:22:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f479555b740 1 -- 172.21.15.16:0/3026110540 wait complete. 2024-08-28T19:22:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f479555b740 20 client.16035 trim_cache size 0 max 16384 2024-08-28T19:22:33.378 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f478efe6640 1 -- 172.21.15.16:0/1552550879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47780b4ca0 msgr2=0x7f47780b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f478efe6640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47780b4ca0 0x7f47780b71b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f478002f500 tx=0x7f4780005d80 comp rx=0 tx=0).stop 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f478efe6640 1 -- 172.21.15.16:0/1552550879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4788104620 msgr2=0x7f478819a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.372+0000 7f478efe6640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4788104620 0x7f478819a280 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f477c00e9c0 tx=0x7f477c00ee80 comp rx=0 tx=0).stop 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.373+0000 7f478efe6640 1 -- 172.21.15.16:0/1552550879 shutdown_connections 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.373+0000 7f478efe6640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f478810dbd0 0x7f478819fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.373+0000 7f478efe6640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47780b4ca0 0x7f47780b71b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.373+0000 7f478efe6640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4788105050 0x7f478819a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.373+0000 7f478efe6640 1 --2- 172.21.15.16:0/1552550879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4788104620 0x7f478819a280 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.373+0000 7f478efe6640 1 -- 172.21.15.16:0/1552550879 >> 172.21.15.16:0/1552550879 conn(0x7f47880794d0 msgr2=0x7f4788106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.373+0000 7f478efe6640 1 -- 172.21.15.16:0/1552550879 shutdown_connections 2024-08-28T19:22:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:33.373+0000 7f478efe6640 1 -- 172.21.15.16:0/1552550879 wait complete. 2024-08-28T19:22:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:33 smithi028 ceph-mon[21769]: pgmap v384: 129 pgs: 129 active+clean; 13 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.2 MiB/s wr, 155 op/s 2024-08-28T19:22:33.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 39613 17.80 MB/sec warmup 13 sec latency 15.094 ms 2024-08-28T19:22:34.044 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:22:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.796+0000 7f47b9566640 1 -- 172.21.15.16:0/2240177442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b4108140 msgr2=0x7f47b4108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.796+0000 7f47b9566640 1 --2- 172.21.15.16:0/2240177442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b4108140 0x7f47b4108550 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009900 tx=0x7f47a802a410 comp rx=0 tx=0).stop 2024-08-28T19:22:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.796+0000 7f47b9566640 1 -- 172.21.15.16:0/2240177442 shutdown_connections 2024-08-28T19:22:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.796+0000 7f47b9566640 1 --2- 172.21.15.16:0/2240177442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f47b4108a90 0x7f47b410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.796+0000 7f47b9566640 1 --2- 172.21.15.16:0/2240177442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b4108140 0x7f47b4108550 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.796+0000 7f47b9566640 1 --2- 172.21.15.16:0/2240177442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b4102800 0x7f47b4102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.796+0000 7f47b9566640 1 -- 172.21.15.16:0/2240177442 >> 172.21.15.16:0/2240177442 conn(0x7f47b40794d0 msgr2=0x7f47b40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.796+0000 7f47b9566640 1 -- 172.21.15.16:0/2240177442 shutdown_connections 2024-08-28T19:22:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.796+0000 7f47b9566640 1 -- 172.21.15.16:0/2240177442 wait complete. 2024-08-28T19:22:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.797+0000 7f47b9566640 1 Processor -- start 2024-08-28T19:22:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.797+0000 7f47b9566640 1 -- start start 2024-08-28T19:22:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b9566640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b4102800 0x7f47b4104e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b9566640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f47b4108140 0x7f47b41053c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b9566640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b4108a90 0x7f47b4105900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b9566640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41126e0 con 0x7f47b4108a90 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b9566640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47b4112560 con 0x7f47b4102800 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b9566640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f47b4112860 con 0x7f47b4108140 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b4108a90 0x7f47b4105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b4102800 0x7f47b4104e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b4108a90 0x7f47b4105900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45666/0 (socket says 172.21.15.16:45666) 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b37fe640 1 -- 172.21.15.16:0/1078843240 learned_addr learned my addr 172.21.15.16:0/1078843240 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b27fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f47b4108140 0x7f47b41053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b4102800 0x7f47b4104e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41384/0 (socket says 172.21.15.16:41384) 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b2ffd640 1 -- 172.21.15.16:0/1078843240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f47b4108140 msgr2=0x7f47b41053c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f47b4108140 0x7f47b41053c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b2ffd640 1 -- 172.21.15.16:0/1078843240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b4108a90 msgr2=0x7f47b4105900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b4108a90 0x7f47b4105900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b2ffd640 1 -- 172.21.15.16:0/1078843240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b41a84b0 con 0x7f47b4102800 2024-08-28T19:22:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b37fe640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b4108a90 0x7f47b4105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b27fc640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f47b4108140 0x7f47b41053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.798+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b4102800 0x7f47b4104e80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f47a400e9f0 tx=0x7f47a400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.799+0000 7f479bfff640 1 -- 172.21.15.16:0/1078843240 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a400ccc0 con 0x7f47b4102800 2024-08-28T19:22:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.799+0000 7f479bfff640 1 -- 172.21.15.16:0/1078843240 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47a40108e0 con 0x7f47b4102800 2024-08-28T19:22:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.799+0000 7f47b9566640 1 -- 172.21.15.16:0/1078843240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b41a87a0 con 0x7f47b4102800 2024-08-28T19:22:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.799+0000 7f479bfff640 1 -- 172.21.15.16:0/1078843240 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a4018880 con 0x7f47b4102800 2024-08-28T19:22:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.799+0000 7f47b9566640 1 -- 172.21.15.16:0/1078843240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b41a8cb0 con 0x7f47b4102800 2024-08-28T19:22:34.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.801+0000 7f47bfadb740 1 Processor -- start 2024-08-28T19:22:34.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.801+0000 7f47bfadb740 1 -- start start 2024-08-28T19:22:34.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f479bfff640 1 -- 172.21.15.16:0/1078843240 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f47a4010430 con 0x7f47b4102800 2024-08-28T19:22:34.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61ad070 0x558ba6552310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61841b0 0x558ba6552850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba618c390 0x558ba6552d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47bfadb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558ba64c9680 con 0x558ba61ad070 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47bfadb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ba64b8c10 con 0x558ba61841b0 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47bfadb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558ba64bcbc0 con 0x558ba618c390 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61ad070 0x558ba6552310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61841b0 0x558ba6552850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47b27fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba618c390 0x558ba6552d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47b27fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba618c390 0x558ba6552d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:37510/0 (socket says 172.21.15.16:37510) 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47b27fc640 1 -- 172.21.15.16:0/3686591745 learned_addr learned my addr 172.21.15.16:0/3686591745 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47b2ffd640 1 -- 172.21.15.16:0/3686591745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba618c390 msgr2=0x558ba6552d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/3686591745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba618c390 0x558ba6552d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47b2ffd640 1 -- 172.21.15.16:0/3686591745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61ad070 msgr2=0x558ba6552310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/3686591745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61ad070 0x558ba6552310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47b2ffd640 1 -- 172.21.15.16:0/3686591745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ba6218650 con 0x558ba61841b0 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.802+0000 7f47b27fc640 1 --2- 172.21.15.16:0/3686591745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba618c390 0x558ba6552d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.803+0000 7f479bfff640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f479c0b4ca0 0x7f479c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.803+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/3686591745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61841b0 0x558ba6552850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f47a40e2fd0 tx=0x7f47a40ee730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18c91a0f9277d20e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.803+0000 7f47b37fe640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f479c0b4ca0 0x7f479c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.803+0000 7f479bfff640 1 -- 172.21.15.16:0/1078843240 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f47a40e5bf0 con 0x7f47b4102800 2024-08-28T19:22:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.803+0000 7f47997fa640 1 -- 172.21.15.16:0/3686591745 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a40a7ee0 con 0x558ba61841b0 2024-08-28T19:22:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.803+0000 7f47997fa640 1 -- 172.21.15.16:0/3686591745 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47a4010720 con 0x558ba61841b0 2024-08-28T19:22:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.803+0000 7f47997fa640 1 -- 172.21.15.16:0/3686591745 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a40ab790 con 0x558ba61841b0 2024-08-28T19:22:34.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47bfadb740 1 -- 172.21.15.16:0/3686591745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61841b0 msgr2=0x558ba6552850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47bfadb740 1 --2- 172.21.15.16:0/3686591745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61841b0 0x558ba6552850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f47a40e2fd0 tx=0x7f47a40ee730 comp rx=0 tx=0).stop 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47b37fe640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f479c0b4ca0 0x7f479c0b71b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0049d0 tx=0x7f47ac009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47bfadb740 1 -- 172.21.15.16:0/3686591745 shutdown_connections 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47bfadb740 1 --2- 172.21.15.16:0/3686591745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba618c390 0x558ba6552d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47bfadb740 1 --2- 172.21.15.16:0/3686591745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61841b0 0x558ba6552850 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47bfadb740 1 --2- 172.21.15.16:0/3686591745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61ad070 0x558ba6552310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47bfadb740 1 -- 172.21.15.16:0/3686591745 >> 172.21.15.16:0/3686591745 conn(0x558ba61a1460 msgr2=0x558ba6550200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47bfadb740 1 -- 172.21.15.16:0/3686591745 shutdown_connections 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47bfadb740 1 -- 172.21.15.16:0/3686591745 wait complete. 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47bfadb740 1 -- start start 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.804+0000 7f47bfadb740 1 Processor -- start 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61841b0 0x558ba655db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba618c390 0x558ba655e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61ad070 0x558ba6569de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47bfadb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558ba631b700 con 0x558ba61841b0 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47bfadb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ba643f750 con 0x558ba618c390 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47bfadb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558ba6560af0 con 0x558ba61ad070 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47bfadb740 10 client.0 _ll_register_callbacks cb 0x558ba646ee10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61841b0 0x558ba655db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61841b0 0x558ba655db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45692/0 (socket says 172.21.15.16:45692) 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b2ffd640 1 -- 172.21.15.16:0/1655847623 learned_addr learned my addr 172.21.15.16:0/1655847623 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b27fc640 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba618c390 0x558ba655e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b37fe640 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61ad070 0x558ba6569de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b2ffd640 1 -- 172.21.15.16:0/1655847623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61ad070 msgr2=0x558ba6569de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61ad070 0x558ba6569de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b2ffd640 1 -- 172.21.15.16:0/1655847623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba618c390 msgr2=0x558ba655e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba618c390 0x558ba655e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b2ffd640 1 -- 172.21.15.16:0/1655847623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ba6478900 con 0x558ba61841b0 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b27fc640 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba618c390 0x558ba655e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61841b0 0x558ba655db60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f47a40e2fa0 tx=0x7f47a400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1b4df836cd848d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.805+0000 7f4796ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.806+0000 7f4796ff5640 1 -- 172.21.15.16:0/1655847623 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a40e9050 con 0x558ba61841b0 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.806+0000 7f4796ff5640 1 -- 172.21.15.16:0/1655847623 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f47a40e9e00 con 0x558ba61841b0 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.806+0000 7f47bfadb740 1 -- 172.21.15.16:0/1655847623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558ba64cb5d0 con 0x558ba61841b0 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.806+0000 7f4796ff5640 1 -- 172.21.15.16:0/1655847623 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f47a40ab970 con 0x558ba61841b0 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.806+0000 7f47bfadb740 10 client.16047 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.806+0000 7f47bfadb740 1 -- 172.21.15.16:0/1655847623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558ba656c950 con 0x558ba61841b0 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.806+0000 7f4796ff5640 1 -- 172.21.15.16:0/1655847623 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558ba64cb5d0 con 0x558ba61841b0 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.806+0000 7f4796ff5640 1 -- 172.21.15.16:0/1655847623 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f47a40af030 con 0x558ba61841b0 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.807+0000 7f47bfadb740 10 client.16047 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.807+0000 7f47bfadb740 10 client.16047 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.807+0000 7f47bfadb740 20 client.16047 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.807+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558ba616b9f0 0x558ba656fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.807+0000 7f47bfadb740 4 client.16047 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.807+0000 7f47bfadb740 1 -- 172.21.15.16:0/1655847623 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ba64c7070 con 0x558ba616b9f0 2024-08-28T19:22:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.807+0000 7f47b27fc640 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558ba616b9f0 0x558ba656fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.807+0000 7f4796ff5640 10 client.16047 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:22:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.807+0000 7f47b27fc640 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558ba616b9f0 0x558ba656fcb0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9c94b218da20b262 server_cookie=fa2c9edb263410b4 in_seq=0 out_seq=0 2024-08-28T19:22:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f4796ff5640 1 -- 172.21.15.16:0/1655847623 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558ba64c7070 con 0x558ba616b9f0 2024-08-28T19:22:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f4796ff5640 10 client.16047 handle_command_reply: tid=0 2024-08-28T19:22:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 1 client.16047 shutdown 2024-08-28T19:22:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 1 -- 172.21.15.16:0/1655847623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61841b0 msgr2=0x558ba655db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61841b0 0x558ba655db60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f47a40e2fa0 tx=0x7f47a400c7f0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 1 -- 172.21.15.16:0/1655847623 shutdown_connections 2024-08-28T19:22:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558ba616b9f0 0x558ba656fcb0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61ad070 0x558ba6569de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba618c390 0x558ba655e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1655847623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba61841b0 0x558ba655db60 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 1 -- 172.21.15.16:0/1655847623 >> 172.21.15.16:0/1655847623 conn(0x558ba61a1460 msgr2=0x558ba65554f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f4796ff5640 0 client.16047 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:22:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 1 -- 172.21.15.16:0/1655847623 shutdown_connections 2024-08-28T19:22:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 1 -- 172.21.15.16:0/1655847623 wait complete. 2024-08-28T19:22:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.808+0000 7f47bfadb740 20 client.16047 trim_cache size 0 max 16384 2024-08-28T19:22:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47bfadb740 1 Processor -- start 2024-08-28T19:22:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47bfadb740 1 -- start start 2024-08-28T19:22:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61841b0 0x558ba6555410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 0x558ba6555950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 0x558ba655ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47bfadb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b2d30 con 0x558ba618c390 2024-08-28T19:22:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47bfadb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b4870 con 0x558ba61ad070 2024-08-28T19:22:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47bfadb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b3400 con 0x558ba61841b0 2024-08-28T19:22:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47b27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 0x558ba6555950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47b2ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61841b0 0x558ba6555410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47b27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 0x558ba6555950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45706/0 (socket says 172.21.15.16:45706) 2024-08-28T19:22:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47b27fc640 1 -- 172.21.15.16:0/797009286 learned_addr learned my addr 172.21.15.16:0/797009286 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.822+0000 7f47b37fe640 1 --2- 172.21.15.16:0/797009286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 0x558ba655ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.823+0000 7f47b27fc640 1 -- 172.21.15.16:0/797009286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61841b0 msgr2=0x558ba6555410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.823+0000 7f47b27fc640 1 --2- 172.21.15.16:0/797009286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61841b0 0x558ba6555410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.823+0000 7f47b27fc640 1 -- 172.21.15.16:0/797009286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 msgr2=0x558ba655ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.823+0000 7f47b27fc640 1 --2- 172.21.15.16:0/797009286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 0x558ba655ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.823+0000 7f47b27fc640 1 -- 172.21.15.16:0/797009286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ba655c060 con 0x558ba618c390 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.823+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/797009286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61841b0 0x558ba6555410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.823+0000 7f47b27fc640 1 --2- 172.21.15.16:0/797009286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 0x558ba6555950 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f47b406a840 tx=0x7f47a8031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20cb54d3872f6493 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.823+0000 7f4798ff9640 1 -- 172.21.15.16:0/797009286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a804a020 con 0x558ba618c390 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.823+0000 7f4798ff9640 1 -- 172.21.15.16:0/797009286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47a80054f0 con 0x558ba618c390 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.823+0000 7f4798ff9640 1 -- 172.21.15.16:0/797009286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8035040 con 0x558ba618c390 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.824+0000 7f47bfadb740 1 -- 172.21.15.16:0/797009286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 msgr2=0x558ba6555950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.824+0000 7f47bfadb740 1 --2- 172.21.15.16:0/797009286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 0x558ba6555950 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f47b406a840 tx=0x7f47a8031400 comp rx=0 tx=0).stop 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.824+0000 7f47bfadb740 1 -- 172.21.15.16:0/797009286 shutdown_connections 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.824+0000 7f47bfadb740 1 --2- 172.21.15.16:0/797009286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 0x558ba655ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.824+0000 7f47bfadb740 1 --2- 172.21.15.16:0/797009286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 0x558ba6555950 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.824+0000 7f47bfadb740 1 --2- 172.21.15.16:0/797009286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61841b0 0x558ba6555410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.824+0000 7f47bfadb740 1 -- 172.21.15.16:0/797009286 >> 172.21.15.16:0/797009286 conn(0x558ba61a1460 msgr2=0x558ba6558d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:34.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.824+0000 7f47bfadb740 1 -- 172.21.15.16:0/797009286 shutdown_connections 2024-08-28T19:22:34.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.824+0000 7f47bfadb740 1 -- 172.21.15.16:0/797009286 wait complete. 2024-08-28T19:22:34.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.824+0000 7f47bfadb740 1 -- start start 2024-08-28T19:22:34.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47bfadb740 1 Processor -- start 2024-08-28T19:22:34.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61841b0 0x558ba6570e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 0x558ba65713a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47bfadb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 0x558ba65718e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47bfadb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b41a0 con 0x558ba618c390 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47bfadb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b3ad0 con 0x558ba61ad070 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47bfadb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b0d10 con 0x558ba61841b0 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47bfadb740 10 client.0 _ll_register_callbacks cb 0x7f47ac0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47b27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 0x558ba65713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 0x558ba65718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 0x558ba65718e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41396/0 (socket says 172.21.15.16:41396) 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47b37fe640 1 -- 172.21.15.16:0/1732657362 learned_addr learned my addr 172.21.15.16:0/1732657362 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61841b0 0x558ba6570e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47b37fe640 1 -- 172.21.15.16:0/1732657362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61841b0 msgr2=0x558ba6570e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47b37fe640 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61841b0 0x558ba6570e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47b37fe640 1 -- 172.21.15.16:0/1732657362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 msgr2=0x558ba65713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47b37fe640 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 0x558ba65713a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.825+0000 7f47b37fe640 1 -- 172.21.15.16:0/1732657362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ba65d95b0 con 0x558ba61ad070 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.826+0000 7f47b37fe640 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 0x558ba65718e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00ffb0 tx=0x7f47ac004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=864461324d8303b8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.826+0000 7f4796ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.826+0000 7f4796ff5640 1 -- 172.21.15.16:0/1732657362 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac02f060 con 0x558ba61ad070 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.826+0000 7f4796ff5640 1 -- 172.21.15.16:0/1732657362 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f47ac011c10 con 0x558ba61ad070 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.826+0000 7f47bfadb740 1 -- 172.21.15.16:0/1732657362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47a80054f0 con 0x558ba61ad070 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.826+0000 7f4796ff5640 1 -- 172.21.15.16:0/1732657362 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f47ac036050 con 0x558ba61ad070 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.826+0000 7f47bfadb740 10 client.25691 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.826+0000 7f47bfadb740 1 -- 172.21.15.16:0/1732657362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558ba65da890 con 0x558ba61ad070 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.826+0000 7f4796ff5640 1 -- 172.21.15.16:0/1732657362 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47a80054f0 con 0x558ba61ad070 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.827+0000 7f4796ff5640 1 -- 172.21.15.16:0/1732657362 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f47ac01e410 con 0x558ba61ad070 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.827+0000 7f47bfadb740 10 client.25691 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:34.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.827+0000 7f47bfadb740 10 client.25691 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.827+0000 7f47bfadb740 20 client.25691 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.827+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558ba6140da0 0x558ba65dd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.827+0000 7f47bfadb740 4 client.25691 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.827+0000 7f47bfadb740 1 -- 172.21.15.16:0/1732657362 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558ba64c47b0 con 0x558ba6140da0 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.827+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558ba6140da0 0x558ba65dd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.827+0000 7f47b2ffd640 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558ba6140da0 0x558ba65dd0b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c0016a45b9cd9e07 server_cookie=ba714d7cdf6a2991 in_seq=0 out_seq=0 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.827+0000 7f4796ff5640 10 client.25691 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f4796ff5640 1 -- 172.21.15.16:0/1732657362 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558ba64c47b0 con 0x558ba6140da0 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f4796ff5640 10 client.25691 handle_command_reply: tid=0 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 1 client.25691 shutdown 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 1 -- 172.21.15.16:0/1732657362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 msgr2=0x558ba65718e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 0x558ba65718e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00ffb0 tx=0x7f47ac004a40 comp rx=0 tx=0).stop 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 1 -- 172.21.15.16:0/1732657362 shutdown_connections 2024-08-28T19:22:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558ba6140da0 0x558ba65dd0b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ba61ad070 0x558ba65718e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ba618c390 0x558ba65713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 1 --2- 172.21.15.16:0/1732657362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558ba61841b0 0x558ba6570e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 1 -- 172.21.15.16:0/1732657362 >> 172.21.15.16:0/1732657362 conn(0x558ba61a1460 msgr2=0x558ba6555410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f4796ff5640 0 client.25691 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:22:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 1 -- 172.21.15.16:0/1732657362 shutdown_connections 2024-08-28T19:22:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 1 -- 172.21.15.16:0/1732657362 wait complete. 2024-08-28T19:22:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.828+0000 7f47bfadb740 20 client.25691 trim_cache size 0 max 16384 2024-08-28T19:22:34.833 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:22:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 -- 172.21.15.16:0/1078843240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f479c0b4ca0 msgr2=0x7f479c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f479c0b4ca0 0x7f479c0b71b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0049d0 tx=0x7f47ac009500 comp rx=0 tx=0).stop 2024-08-28T19:22:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 -- 172.21.15.16:0/1078843240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b4102800 msgr2=0x7f47b4104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b4102800 0x7f47b4104e80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f47a400e9f0 tx=0x7f47a400eeb0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 -- 172.21.15.16:0/1078843240 shutdown_connections 2024-08-28T19:22:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b4108a90 0x7f47b4105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f47b4108140 0x7f47b41053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f479c0b4ca0 0x7f479c0b71b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 --2- 172.21.15.16:0/1078843240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b4102800 0x7f47b4104e80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 -- 172.21.15.16:0/1078843240 >> 172.21.15.16:0/1078843240 conn(0x7f47b40794d0 msgr2=0x7f47b406af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 -- 172.21.15.16:0/1078843240 shutdown_connections 2024-08-28T19:22:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:34.829+0000 7f47b9566640 1 -- 172.21.15.16:0/1078843240 wait complete. 2024-08-28T19:22:34.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 42854 17.87 MB/sec warmup 14 sec latency 39.591 ms 2024-08-28T19:22:35.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:35 smithi115 ceph-mon[22184]: pgmap v385: 129 pgs: 129 active+clean; 16 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-08-28T19:22:35.478 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:35 smithi016 ceph-mon[18556]: pgmap v385: 129 pgs: 129 active+clean; 16 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-08-28T19:22:35.507 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:22:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:35 smithi028 ceph-mon[21769]: pgmap v385: 129 pgs: 129 active+clean; 16 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-08-28T19:22:35.779 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:35.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 45962 17.66 MB/sec warmup 15 sec latency 24.143 ms 2024-08-28T19:22:36.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.280+0000 7f966033a640 1 -- 172.21.15.16:0/2958468456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96581050b0 msgr2=0x7f965810d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:36.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.280+0000 7f966033a640 1 --2- 172.21.15.16:0/2958468456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96581050b0 0x7f965810d600 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9648009900 tx=0x7f964802a410 comp rx=0 tx=0).stop 2024-08-28T19:22:36.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.280+0000 7f966033a640 1 -- 172.21.15.16:0/2958468456 shutdown_connections 2024-08-28T19:22:36.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.280+0000 7f966033a640 1 --2- 172.21.15.16:0/2958468456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965810dbc0 0x7f96581100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.280+0000 7f966033a640 1 --2- 172.21.15.16:0/2958468456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96581050b0 0x7f965810d600 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.280+0000 7f966033a640 1 --2- 172.21.15.16:0/2958468456 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9658104760 0x7f9658104b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.280+0000 7f966033a640 1 -- 172.21.15.16:0/2958468456 >> 172.21.15.16:0/2958468456 conn(0x7f96580794d0 msgr2=0x7f96580798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:36.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.280+0000 7f966033a640 1 -- 172.21.15.16:0/2958468456 shutdown_connections 2024-08-28T19:22:36.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.280+0000 7f966033a640 1 -- 172.21.15.16:0/2958468456 wait complete. 2024-08-28T19:22:36.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f966033a640 1 Processor -- start 2024-08-28T19:22:36.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f966033a640 1 -- start start 2024-08-28T19:22:36.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f966033a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9658104760 0x7f965819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f966033a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96581050b0 0x7f965819a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f966033a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965810dbc0 0x7f965819fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f966033a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9658113800 con 0x7f965810dbc0 2024-08-28T19:22:36.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f966033a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9658113680 con 0x7f96581050b0 2024-08-28T19:22:36.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f966033a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9658113980 con 0x7f9658104760 2024-08-28T19:22:36.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f965d8d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96581050b0 0x7f965819a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f965e8d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965810dbc0 0x7f965819fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f965e0d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9658104760 0x7f965819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f965e8d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965810dbc0 0x7f965819fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45744/0 (socket says 172.21.15.16:45744) 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f965d8d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96581050b0 0x7f965819a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41412/0 (socket says 172.21.15.16:41412) 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f965e8d7640 1 -- 172.21.15.16:0/786173470 learned_addr learned my addr 172.21.15.16:0/786173470 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.281+0000 7f965e0d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9658104760 0x7f965819a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:37574/0 (socket says 172.21.15.16:37574) 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f965e8d7640 1 -- 172.21.15.16:0/786173470 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9658104760 msgr2=0x7f965819a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f965e8d7640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9658104760 0x7f965819a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f965e8d7640 1 -- 172.21.15.16:0/786173470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96581050b0 msgr2=0x7f965819a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f965e8d7640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96581050b0 0x7f965819a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f965e8d7640 1 -- 172.21.15.16:0/786173470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96581a0240 con 0x7f965810dbc0 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f965d8d5640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96581050b0 0x7f965819a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f965e8d7640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965810dbc0 0x7f965819fb10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f965000eb10 tx=0x7f965000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:36.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f964f7fe640 1 -- 172.21.15.16:0/786173470 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650004b20 con 0x7f965810dbc0 2024-08-28T19:22:36.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f966033a640 1 -- 172.21.15.16:0/786173470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96581a0530 con 0x7f965810dbc0 2024-08-28T19:22:36.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f966033a640 1 -- 172.21.15.16:0/786173470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96581a6c90 con 0x7f965810dbc0 2024-08-28T19:22:36.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f964f7fe640 1 -- 172.21.15.16:0/786173470 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9650004cc0 con 0x7f965810dbc0 2024-08-28T19:22:36.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.282+0000 7f964f7fe640 1 -- 172.21.15.16:0/786173470 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650010690 con 0x7f965810dbc0 2024-08-28T19:22:36.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.285+0000 7f96668af740 1 Processor -- start 2024-08-28T19:22:36.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.286+0000 7f964f7fe640 1 -- 172.21.15.16:0/786173470 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f9650010830 con 0x7f965810dbc0 2024-08-28T19:22:36.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.286+0000 7f96668af740 1 -- start start 2024-08-28T19:22:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.286+0000 7f96668af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009d1070 0x556e00d76310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.286+0000 7f96668af740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d76850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.286+0000 7f96668af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 0x556e00d76d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.286+0000 7f96668af740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556e00ced680 con 0x556e009b0390 2024-08-28T19:22:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.286+0000 7f96668af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e00cdcc10 con 0x556e009d1070 2024-08-28T19:22:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.286+0000 7f96668af740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556e00ce0bc0 con 0x556e009a81b0 2024-08-28T19:22:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965d8d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009d1070 0x556e00d76310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965e0d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d76850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965e0d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d76850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:37580/0 (socket says 172.21.15.16:37580) 2024-08-28T19:22:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965d8d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009d1070 0x556e00d76310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41424/0 (socket says 172.21.15.16:41424) 2024-08-28T19:22:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965e0d6640 1 -- 172.21.15.16:0/2330448429 learned_addr learned my addr 172.21.15.16:0/2330448429 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965d8d5640 1 --2- 172.21.15.16:0/2330448429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 0x556e00d76d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965e0d6640 1 -- 172.21.15.16:0/2330448429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009d1070 msgr2=0x556e00d76310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:36.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965e0d6640 1 --2- 172.21.15.16:0/2330448429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009d1070 0x556e00d76310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965e0d6640 1 -- 172.21.15.16:0/2330448429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 msgr2=0x556e00d76d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:36.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965e0d6640 1 --2- 172.21.15.16:0/2330448429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 0x556e00d76d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965e0d6640 1 -- 172.21.15.16:0/2330448429 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e00a3c650 con 0x556e009a81b0 2024-08-28T19:22:36.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f964f7fe640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f96440b4ca0 0x7f96440b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965e0d6640 1 --2- 172.21.15.16:0/2330448429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d76850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f965400bce0 tx=0x7f96540107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4e6317475dca61b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f965d8d5640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f96440b4ca0 0x7f96440b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.287+0000 7f964f7fe640 1 -- 172.21.15.16:0/786173470 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f96500e5d40 con 0x7f965810dbc0 2024-08-28T19:22:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.288+0000 7f964cd9a640 1 -- 172.21.15.16:0/2330448429 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654005020 con 0x556e009a81b0 2024-08-28T19:22:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.288+0000 7f964cd9a640 1 -- 172.21.15.16:0/2330448429 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96540047a0 con 0x556e009a81b0 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.288+0000 7f964cd9a640 1 -- 172.21.15.16:0/2330448429 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96540116a0 con 0x556e009a81b0 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f96668af740 1 -- 172.21.15.16:0/2330448429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 msgr2=0x556e00d76850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f965d8d5640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f96440b4ca0 0x7f96440b71b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9648009230 tx=0x7f9648031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f96668af740 1 --2- 172.21.15.16:0/2330448429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d76850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f965400bce0 tx=0x7f96540107a0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f96668af740 1 -- 172.21.15.16:0/2330448429 shutdown_connections 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f96668af740 1 --2- 172.21.15.16:0/2330448429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 0x556e00d76d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f96668af740 1 --2- 172.21.15.16:0/2330448429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d76850 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f96668af740 1 --2- 172.21.15.16:0/2330448429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009d1070 0x556e00d76310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f96668af740 1 -- 172.21.15.16:0/2330448429 >> 172.21.15.16:0/2330448429 conn(0x556e009c5460 msgr2=0x556e00d74200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f96668af740 1 -- 172.21.15.16:0/2330448429 shutdown_connections 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f96668af740 1 -- 172.21.15.16:0/2330448429 wait complete. 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f96668af740 1 -- start start 2024-08-28T19:22:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.289+0000 7f96668af740 1 Processor -- start 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f96668af740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d85df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f96668af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009b0390 0x556e00d86330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f96668af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 0x556e00d921e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f96668af740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556e00b3f700 con 0x556e009d1070 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f96668af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e00c63750 con 0x556e009b0390 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f96668af740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556e00d88e20 con 0x556e009a81b0 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f965e0d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d85df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f96668af740 10 client.0 _ll_register_callbacks cb 0x556e00c92e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f965e0d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d85df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:37582/0 (socket says 172.21.15.16:37582) 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f965d8d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009b0390 0x556e00d86330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f965e0d6640 1 -- 172.21.15.16:0/3215643734 learned_addr learned my addr 172.21.15.16:0/3215643734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f965e8d7640 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 0x556e00d921e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f965e0d6640 1 -- 172.21.15.16:0/3215643734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009b0390 msgr2=0x556e00d86330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f965e0d6640 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009b0390 0x556e00d86330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f965e0d6640 1 -- 172.21.15.16:0/3215643734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 msgr2=0x556e00d921e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f965e0d6640 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 0x556e00d921e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f965e0d6640 1 -- 172.21.15.16:0/3215643734 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e00c9c900 con 0x556e009a81b0 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.290+0000 7f965e8d7640 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 0x556e00d921e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.291+0000 7f965e0d6640 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d85df0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9654005d00 tx=0x7f9654004b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cda89ff7bc9d477c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.291+0000 7f9641ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.291+0000 7f9641ffb640 1 -- 172.21.15.16:0/3215643734 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965402a060 con 0x556e009a81b0 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.291+0000 7f9641ffb640 1 -- 172.21.15.16:0/3215643734 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f9654005000 con 0x556e009a81b0 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.291+0000 7f96668af740 1 -- 172.21.15.16:0/3215643734 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96540047a0 con 0x556e009a81b0 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.291+0000 7f9641ffb640 1 -- 172.21.15.16:0/3215643734 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9654031050 con 0x556e009a81b0 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.291+0000 7f96668af740 10 client.25594 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.291+0000 7f96668af740 1 -- 172.21.15.16:0/3215643734 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556e00d94c10 con 0x556e009a81b0 2024-08-28T19:22:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.291+0000 7f9641ffb640 1 -- 172.21.15.16:0/3215643734 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96540047a0 con 0x556e009a81b0 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.291+0000 7f9641ffb640 1 -- 172.21.15.16:0/3215643734 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9654024070 con 0x556e009a81b0 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.292+0000 7f96668af740 10 client.25594 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.292+0000 7f96668af740 10 client.25594 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.292+0000 7f96668af740 20 client.25594 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.292+0000 7f96668af740 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e0098f9f0 0x556e00d98020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.292+0000 7f96668af740 4 client.25594 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.292+0000 7f96668af740 1 -- 172.21.15.16:0/3215643734 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e00cef5d0 con 0x556e0098f9f0 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.292+0000 7f965d8d5640 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e0098f9f0 0x556e00d98020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.293+0000 7f965d8d5640 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e0098f9f0 0x556e00d98020 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=795631e52e74a18c server_cookie=bd505536d4d926d in_seq=0 out_seq=0 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.293+0000 7f9641ffb640 10 client.25594 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.293+0000 7f9641ffb640 1 -- 172.21.15.16:0/3215643734 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556e00cef5d0 con 0x556e0098f9f0 2024-08-28T19:22:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.293+0000 7f9641ffb640 10 client.25594 handle_command_reply: tid=0 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.293+0000 7f96668af740 1 client.25594 shutdown 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f96668af740 1 -- 172.21.15.16:0/3215643734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 msgr2=0x556e00d85df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f96668af740 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d85df0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9654005d00 tx=0x7f9654004b80 comp rx=0 tx=0).stop 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f96668af740 1 -- 172.21.15.16:0/3215643734 shutdown_connections 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f96668af740 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e0098f9f0 0x556e00d98020 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f96668af740 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 0x556e00d921e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f96668af740 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009b0390 0x556e00d86330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f9641ffb640 0 client.25594 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f96668af740 1 --2- 172.21.15.16:0/3215643734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009a81b0 0x556e00d85df0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f96668af740 1 -- 172.21.15.16:0/3215643734 >> 172.21.15.16:0/3215643734 conn(0x556e009c5460 msgr2=0x556e00d794a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f96668af740 1 -- 172.21.15.16:0/3215643734 shutdown_connections 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f96668af740 1 -- 172.21.15.16:0/3215643734 wait complete. 2024-08-28T19:22:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.294+0000 7f96668af740 20 client.25594 trim_cache size 0 max 16384 2024-08-28T19:22:36.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.309+0000 7f96668af740 1 Processor -- start 2024-08-28T19:22:36.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.309+0000 7f96668af740 1 -- start start 2024-08-28T19:22:36.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.309+0000 7f96668af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 0x556e00d737d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.309+0000 7f96668af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 0x556e00d73d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.309+0000 7f96668af740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009d1070 0x556e00d82fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.309+0000 7f96668af740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f963c00fec0 con 0x556e009b0390 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.309+0000 7f96668af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f963c011a00 con 0x556e009a81b0 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.309+0000 7f96668af740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f963c010590 con 0x556e009d1070 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965d8d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 0x556e00d73d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965d8d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 0x556e00d73d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45778/0 (socket says 172.21.15.16:45778) 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965d8d5640 1 -- 172.21.15.16:0/3427025823 learned_addr learned my addr 172.21.15.16:0/3427025823 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965e0d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 0x556e00d737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965e8d7640 1 --2- 172.21.15.16:0/3427025823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009d1070 0x556e00d82fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965e0d6640 1 -- 172.21.15.16:0/3427025823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009d1070 msgr2=0x556e00d82fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965e0d6640 1 --2- 172.21.15.16:0/3427025823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009d1070 0x556e00d82fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965e0d6640 1 -- 172.21.15.16:0/3427025823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 msgr2=0x556e00d73d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965e0d6640 1 --2- 172.21.15.16:0/3427025823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 0x556e00d73d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965e0d6640 1 -- 172.21.15.16:0/3427025823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e00d842f0 con 0x556e009a81b0 2024-08-28T19:22:36.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965d8d5640 1 --2- 172.21.15.16:0/3427025823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 0x556e00d73d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:36.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f965e0d6640 1 --2- 172.21.15.16:0/3427025823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 0x556e00d737d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9654010f90 tx=0x7f9654005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eece8ad0b8d2090 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:36.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.310+0000 7f9643fff640 1 -- 172.21.15.16:0/3427025823 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654039020 con 0x556e009a81b0 2024-08-28T19:22:36.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.311+0000 7f9643fff640 1 -- 172.21.15.16:0/3427025823 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9654004780 con 0x556e009a81b0 2024-08-28T19:22:36.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.311+0000 7f9643fff640 1 -- 172.21.15.16:0/3427025823 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965402a040 con 0x556e009a81b0 2024-08-28T19:22:36.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.311+0000 7f96668af740 1 -- 172.21.15.16:0/3427025823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 msgr2=0x556e00d737d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:36.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.311+0000 7f96668af740 1 --2- 172.21.15.16:0/3427025823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 0x556e00d737d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9654010f90 tx=0x7f9654005850 comp rx=0 tx=0).stop 2024-08-28T19:22:36.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.311+0000 7f96668af740 1 -- 172.21.15.16:0/3427025823 shutdown_connections 2024-08-28T19:22:36.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.311+0000 7f96668af740 1 --2- 172.21.15.16:0/3427025823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009d1070 0x556e00d82fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.311+0000 7f96668af740 1 --2- 172.21.15.16:0/3427025823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009b0390 0x556e00d73d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.311+0000 7f96668af740 1 --2- 172.21.15.16:0/3427025823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 0x556e00d737d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.311+0000 7f96668af740 1 -- 172.21.15.16:0/3427025823 >> 172.21.15.16:0/3427025823 conn(0x556e009c5460 msgr2=0x556e00d84670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:36.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.312+0000 7f96668af740 1 -- 172.21.15.16:0/3427025823 shutdown_connections 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.312+0000 7f96668af740 1 -- 172.21.15.16:0/3427025823 wait complete. 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.312+0000 7f96668af740 1 -- start start 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.312+0000 7f96668af740 1 Processor -- start 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.312+0000 7f96668af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 0x556e00d99350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.312+0000 7f96668af740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009b0390 0x556e00d99890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.312+0000 7f96668af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 0x556e00e017a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.312+0000 7f96668af740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f963c011330 con 0x556e009d1070 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.312+0000 7f96668af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f963c010c60 con 0x556e009a81b0 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.312+0000 7f96668af740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f963c00dea0 con 0x556e009b0390 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f96668af740 10 client.0 _ll_register_callbacks cb 0x7f96480314e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965e0d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 0x556e00d99350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965d8d5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009b0390 0x556e00d99890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965e0d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 0x556e00d99350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41438/0 (socket says 172.21.15.16:41438) 2024-08-28T19:22:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965e0d6640 1 -- 172.21.15.16:0/196172735 learned_addr learned my addr 172.21.15.16:0/196172735 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965e8d7640 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 0x556e00e017a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965d8d5640 1 -- 172.21.15.16:0/196172735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 msgr2=0x556e00d99350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965d8d5640 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 0x556e00d99350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965d8d5640 1 -- 172.21.15.16:0/196172735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 msgr2=0x556e00e017a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965d8d5640 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 0x556e00e017a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965d8d5640 1 -- 172.21.15.16:0/196172735 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e00d9a130 con 0x556e009b0390 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965e8d7640 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 0x556e00e017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f965d8d5640 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009b0390 0x556e00d99890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9648005f70 tx=0x7f9648000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=765d17acda0820aa server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f9641ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.313+0000 7f9641ffb640 1 -- 172.21.15.16:0/196172735 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648035040 con 0x556e009b0390 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f9641ffb640 1 -- 172.21.15.16:0/196172735 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f9648004af0 con 0x556e009b0390 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f96668af740 1 -- 172.21.15.16:0/196172735 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9654004780 con 0x556e009b0390 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f9641ffb640 1 -- 172.21.15.16:0/196172735 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9648004cf0 con 0x556e009b0390 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f96668af740 10 client.25600 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f96668af740 1 -- 172.21.15.16:0/196172735 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556e00e02d90 con 0x556e009b0390 2024-08-28T19:22:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f9641ffb640 1 -- 172.21.15.16:0/196172735 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9654004780 con 0x556e009b0390 2024-08-28T19:22:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f9641ffb640 1 -- 172.21.15.16:0/196172735 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f96480039c0 con 0x556e009b0390 2024-08-28T19:22:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f96668af740 10 client.25600 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f96668af740 10 client.25600 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:22:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f96668af740 20 client.25600 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f96668af740 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e00964da0 0x556e00e054b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f96668af740 4 client.25600 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:22:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.314+0000 7f96668af740 1 -- 172.21.15.16:0/196172735 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e00ceb070 con 0x556e00964da0 2024-08-28T19:22:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.315+0000 7f965e0d6640 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e00964da0 0x556e00e054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.315+0000 7f965e0d6640 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e00964da0 0x556e00e054b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c75c6af2eb74a03a server_cookie=bf22b2b5817d5a36 in_seq=0 out_seq=0 2024-08-28T19:22:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.315+0000 7f9641ffb640 10 client.25600 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:22:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.315+0000 7f9641ffb640 1 -- 172.21.15.16:0/196172735 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e00ceb070 con 0x556e00964da0 2024-08-28T19:22:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.315+0000 7f9641ffb640 10 client.25600 handle_command_reply: tid=0 2024-08-28T19:22:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.315+0000 7f96668af740 1 client.25600 shutdown 2024-08-28T19:22:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f96668af740 1 -- 172.21.15.16:0/196172735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009b0390 msgr2=0x556e00d99890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f96668af740 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009b0390 0x556e00d99890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9648005f70 tx=0x7f9648000bc0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f96668af740 1 -- 172.21.15.16:0/196172735 shutdown_connections 2024-08-28T19:22:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f96668af740 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e00964da0 0x556e00e054b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f96668af740 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e009d1070 0x556e00e017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f96668af740 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e009b0390 0x556e00d99890 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f96668af740 1 --2- 172.21.15.16:0/196172735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e009a81b0 0x556e00d99350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f96668af740 1 -- 172.21.15.16:0/196172735 >> 172.21.15.16:0/196172735 conn(0x556e009c5460 msgr2=0x556e00d794a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f9641ffb640 0 client.25600 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:22:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f96668af740 1 -- 172.21.15.16:0/196172735 shutdown_connections 2024-08-28T19:22:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f96668af740 1 -- 172.21.15.16:0/196172735 wait complete. 2024-08-28T19:22:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f96668af740 20 client.25600 trim_cache size 0 max 16384 2024-08-28T19:22:36.321 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:22:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f966033a640 1 -- 172.21.15.16:0/786173470 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f96440b4ca0 msgr2=0x7f96440b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f966033a640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f96440b4ca0 0x7f96440b71b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9648009230 tx=0x7f9648031400 comp rx=0 tx=0).stop 2024-08-28T19:22:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f966033a640 1 -- 172.21.15.16:0/786173470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965810dbc0 msgr2=0x7f965819fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f966033a640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965810dbc0 0x7f965819fb10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f965000eb10 tx=0x7f965000c2d0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f966033a640 1 -- 172.21.15.16:0/786173470 shutdown_connections 2024-08-28T19:22:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f966033a640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965810dbc0 0x7f965819fb10 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f966033a640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f96440b4ca0 0x7f96440b71b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f966033a640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96581050b0 0x7f965819a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f966033a640 1 --2- 172.21.15.16:0/786173470 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9658104760 0x7f965819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.316+0000 7f966033a640 1 -- 172.21.15.16:0/786173470 >> 172.21.15.16:0/786173470 conn(0x7f96580794d0 msgr2=0x7f96580ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.317+0000 7f966033a640 1 -- 172.21.15.16:0/786173470 shutdown_connections 2024-08-28T19:22:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:36.317+0000 7f966033a640 1 -- 172.21.15.16:0/786173470 wait complete. 2024-08-28T19:22:36.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 48407 17.21 MB/sec warmup 16 sec latency 33.886 ms 2024-08-28T19:22:37.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:37 smithi115 ceph-mon[22184]: pgmap v386: 129 pgs: 129 active+clean; 18 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-08-28T19:22:37.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:37 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:22:37.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:37 smithi016 ceph-mon[18556]: pgmap v386: 129 pgs: 129 active+clean; 18 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-08-28T19:22:37.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:37 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:22:37.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:22:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:37 smithi028 ceph-mon[21769]: pgmap v386: 129 pgs: 129 active+clean; 18 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-08-28T19:22:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:37 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:22:37.696 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:37.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 51126 17.28 MB/sec warmup 17 sec latency 14.550 ms 2024-08-28T19:22:38.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:38 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:22:38.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:38 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:22:38.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:38 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:22:38.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.443+0000 7f27b50c3640 1 -- 172.21.15.16:0/3467112102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27b0103910 msgr2=0x7f27b0103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:38.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.443+0000 7f27b50c3640 1 --2- 172.21.15.16:0/3467112102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27b0103910 0x7f27b0103d20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009900 tx=0x7f27a002a410 comp rx=0 tx=0).stop 2024-08-28T19:22:38.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.443+0000 7f27b50c3640 1 -- 172.21.15.16:0/3467112102 shutdown_connections 2024-08-28T19:22:38.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.443+0000 7f27b50c3640 1 --2- 172.21.15.16:0/3467112102 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b0109ba0 0x7f27b01103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.443+0000 7f27b50c3640 1 --2- 172.21.15.16:0/3467112102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0109250 0x7f27b0109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.443+0000 7f27b50c3640 1 --2- 172.21.15.16:0/3467112102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27b0103910 0x7f27b0103d20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.443+0000 7f27b50c3640 1 -- 172.21.15.16:0/3467112102 >> 172.21.15.16:0/3467112102 conn(0x7f27b00ff380 msgr2=0x7f27b01017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.443+0000 7f27b50c3640 1 -- 172.21.15.16:0/3467112102 shutdown_connections 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.443+0000 7f27b50c3640 1 -- 172.21.15.16:0/3467112102 wait complete. 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.444+0000 7f27b50c3640 1 Processor -- start 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27b50c3640 1 -- start start 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27b50c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27b0103910 0x7f27b01a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27b50c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0109250 0x7f27b01a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27b50c3640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b0109ba0 0x7f27b01ac5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27b50c3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27b0113860 con 0x7f27b0109250 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27b50c3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27b01136e0 con 0x7f27b0103910 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27b50c3640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f27b01139e0 con 0x7f27b0109ba0 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27ae59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0109250 0x7f27b01a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27aed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27b0103910 0x7f27b01a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27ae59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0109250 0x7f27b01a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45822/0 (socket says 172.21.15.16:45822) 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27ae59c640 1 -- 172.21.15.16:0/4070743417 learned_addr learned my addr 172.21.15.16:0/4070743417 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27af59e640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b0109ba0 0x7f27b01ac5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27aed9d640 1 -- 172.21.15.16:0/4070743417 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b0109ba0 msgr2=0x7f27b01ac5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.445+0000 7f27aed9d640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b0109ba0 0x7f27b01ac5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.446+0000 7f27aed9d640 1 -- 172.21.15.16:0/4070743417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0109250 msgr2=0x7f27b01a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.446+0000 7f27aed9d640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0109250 0x7f27b01a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.446+0000 7f27aed9d640 1 -- 172.21.15.16:0/4070743417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b01a2350 con 0x7f27b0103910 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.446+0000 7f27ae59c640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0109250 0x7f27b01a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.446+0000 7f27aed9d640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27b0103910 0x7f27b01a1520 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f27a002a930 tx=0x7f27a0031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.446+0000 7f2797fff640 1 -- 172.21.15.16:0/4070743417 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a003f030 con 0x7f27b0103910 2024-08-28T19:22:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.446+0000 7f27b50c3640 1 -- 172.21.15.16:0/4070743417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b01acba0 con 0x7f27b0103910 2024-08-28T19:22:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.446+0000 7f2797fff640 1 -- 172.21.15.16:0/4070743417 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27a00043d0 con 0x7f27b0103910 2024-08-28T19:22:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.446+0000 7f2797fff640 1 -- 172.21.15.16:0/4070743417 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a0035070 con 0x7f27b0103910 2024-08-28T19:22:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.446+0000 7f27b50c3640 1 -- 172.21.15.16:0/4070743417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b01ad0d0 con 0x7f27b0103910 2024-08-28T19:22:38.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.448+0000 7f27bb638740 1 Processor -- start 2024-08-28T19:22:38.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27bb638740 1 -- start start 2024-08-28T19:22:38.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27bb638740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 0x55abe09a3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27bb638740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05d51b0 0x55abe09a3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27bb638740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05dd390 0x55abe09a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27bb638740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55abe091a680 con 0x55abe05dd390 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27bb638740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55abe0909c10 con 0x55abe05fe070 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27bb638740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55abe090dbc0 con 0x55abe05d51b0 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27af59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 0x55abe09a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27af59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 0x55abe09a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41462/0 (socket says 172.21.15.16:41462) 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27af59e640 1 -- 172.21.15.16:0/3720315121 learned_addr learned my addr 172.21.15.16:0/3720315121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f2797fff640 1 -- 172.21.15.16:0/4070743417 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f27a0042030 con 0x7f27b0103910 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27ae59c640 1 --2- 172.21.15.16:0/3720315121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05dd390 0x55abe09a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27aed9d640 1 --2- 172.21.15.16:0/3720315121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05d51b0 0x55abe09a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27af59e640 1 -- 172.21.15.16:0/3720315121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05d51b0 msgr2=0x55abe09a3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27af59e640 1 --2- 172.21.15.16:0/3720315121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05d51b0 0x55abe09a3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27af59e640 1 -- 172.21.15.16:0/3720315121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05dd390 msgr2=0x55abe09a3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27af59e640 1 --2- 172.21.15.16:0/3720315121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05dd390 0x55abe09a3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27af59e640 1 -- 172.21.15.16:0/3720315121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abe0669650 con 0x55abe05fe070 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.449+0000 7f27aed9d640 1 --2- 172.21.15.16:0/3720315121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05d51b0 0x55abe09a3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.450+0000 7f27af59e640 1 --2- 172.21.15.16:0/3720315121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 0x55abe09a3310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f27a8011440 tx=0x7f27a8011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9339ddefab4e184a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.450+0000 7f27957fa640 1 -- 172.21.15.16:0/3720315121 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a8003c30 con 0x55abe05fe070 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.450+0000 7f27957fa640 1 -- 172.21.15.16:0/3720315121 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27a8003dd0 con 0x55abe05fe070 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.450+0000 7f27957fa640 1 -- 172.21.15.16:0/3720315121 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a800fae0 con 0x55abe05fe070 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.450+0000 7f2797fff640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27980b4d20 0x7f27980b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.450+0000 7f27aed9d640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27980b4d20 0x7f27980b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.451+0000 7f2797fff640 1 -- 172.21.15.16:0/4070743417 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f27a0105b70 con 0x7f27b0103910 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.451+0000 7f27bb638740 1 -- 172.21.15.16:0/3720315121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 msgr2=0x55abe09a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.451+0000 7f27aed9d640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27980b4d20 0x7f27980b7230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55abe09a1f90 tx=0x7f27a002a410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.451+0000 7f27bb638740 1 --2- 172.21.15.16:0/3720315121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 0x55abe09a3310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f27a8011440 tx=0x7f27a8011900 comp rx=0 tx=0).stop 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.451+0000 7f27bb638740 1 -- 172.21.15.16:0/3720315121 shutdown_connections 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.451+0000 7f27bb638740 1 --2- 172.21.15.16:0/3720315121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05dd390 0x55abe09a3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.451+0000 7f27bb638740 1 --2- 172.21.15.16:0/3720315121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05d51b0 0x55abe09a3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.451+0000 7f27bb638740 1 --2- 172.21.15.16:0/3720315121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 0x55abe09a3310 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.451+0000 7f27bb638740 1 -- 172.21.15.16:0/3720315121 >> 172.21.15.16:0/3720315121 conn(0x55abe05f2460 msgr2=0x55abe09a1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.451+0000 7f27bb638740 1 -- 172.21.15.16:0/3720315121 shutdown_connections 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.451+0000 7f27bb638740 1 -- 172.21.15.16:0/3720315121 wait complete. 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27bb638740 1 -- start start 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27bb638740 1 Processor -- start 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27bb638740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27bb638740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 0x55abe09af000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27bb638740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05fe070 0x55abe09bae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27bb638740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55abe076c700 con 0x55abe05d51b0 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27bb638740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55abe0890750 con 0x55abe05dd390 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27bb638740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55abe09b1af0 con 0x55abe05fe070 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27bb638740 10 client.0 _ll_register_callbacks cb 0x55abe08bfe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27af59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27ae59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 0x55abe09af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27af59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09aeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45852/0 (socket says 172.21.15.16:45852) 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27af59e640 1 -- 172.21.15.16:0/3538704605 learned_addr learned my addr 172.21.15.16:0/3538704605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.452+0000 7f27af59e640 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05fe070 0x55abe09bae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f27ae59c640 1 -- 172.21.15.16:0/3538704605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05fe070 msgr2=0x55abe09bae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f27ae59c640 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05fe070 0x55abe09bae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f27ae59c640 1 -- 172.21.15.16:0/3538704605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 msgr2=0x55abe09aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f27ae59c640 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09aeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f27ae59c640 1 -- 172.21.15.16:0/3538704605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abe08c9900 con 0x55abe05dd390 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f27ae59c640 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 0x55abe09af000 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f27a40058c0 tx=0x7f27a400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81d9a17066874b1c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f2792ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f2792ff5640 1 -- 172.21.15.16:0/3538704605 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a40120d0 con 0x55abe05dd390 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f27bb638740 1 -- 172.21.15.16:0/3538704605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27a8003dd0 con 0x55abe05dd390 2024-08-28T19:22:38.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f2792ff5640 1 -- 172.21.15.16:0/3538704605 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f27a4004020 con 0x55abe05dd390 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f2792ff5640 1 -- 172.21.15.16:0/3538704605 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f27a4025050 con 0x55abe05dd390 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f27bb638740 10 client.25721 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f27bb638740 1 -- 172.21.15.16:0/3538704605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55abe09bd8f0 con 0x55abe05dd390 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.453+0000 7f2792ff5640 1 -- 172.21.15.16:0/3538704605 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27a8003dd0 con 0x55abe05dd390 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.454+0000 7f2792ff5640 1 -- 172.21.15.16:0/3538704605 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f27a401c470 con 0x55abe05dd390 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.454+0000 7f27bb638740 10 client.25721 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.454+0000 7f27bb638740 10 client.25721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.454+0000 7f27bb638740 20 client.25721 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.454+0000 7f27bb638740 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55abe05bc9f0 0x55abe09c0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.454+0000 7f27bb638740 4 client.25721 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.454+0000 7f27bb638740 1 -- 172.21.15.16:0/3538704605 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abe091c5d0 con 0x55abe05bc9f0 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.454+0000 7f27af59e640 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55abe05bc9f0 0x55abe09c0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.454+0000 7f2792ff5640 10 client.25721 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.454+0000 7f27af59e640 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55abe05bc9f0 0x55abe09c0e50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=947c34b07adff05f server_cookie=edfc0b3dc49f7e86 in_seq=0 out_seq=0 2024-08-28T19:22:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f2792ff5640 1 -- 172.21.15.16:0/3538704605 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55abe091c5d0 con 0x55abe05bc9f0 2024-08-28T19:22:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f2792ff5640 10 client.25721 handle_command_reply: tid=0 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 1 client.25721 shutdown 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 1 -- 172.21.15.16:0/3538704605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 msgr2=0x55abe09af000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 0x55abe09af000 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f27a40058c0 tx=0x7f27a400f790 comp rx=0 tx=0).stop 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 1 -- 172.21.15.16:0/3538704605 shutdown_connections 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55abe05bc9f0 0x55abe09c0e50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05fe070 0x55abe09bae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 0x55abe09af000 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f2792ff5640 0 client.25721 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 1 --2- 172.21.15.16:0/3538704605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 1 -- 172.21.15.16:0/3538704605 >> 172.21.15.16:0/3538704605 conn(0x55abe05f2460 msgr2=0x55abe09a64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 1 -- 172.21.15.16:0/3538704605 shutdown_connections 2024-08-28T19:22:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 1 -- 172.21.15.16:0/3538704605 wait complete. 2024-08-28T19:22:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.456+0000 7f27bb638740 20 client.25721 trim_cache size 0 max 16384 2024-08-28T19:22:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.470+0000 7f27bb638740 1 Processor -- start 2024-08-28T19:22:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.470+0000 7f27bb638740 1 -- start start 2024-08-28T19:22:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27bb638740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09a6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27bb638740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 0x55abe09bad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27bb638740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05fe070 0x55abe09bb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27bb638740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27b01b7210 con 0x55abe05d51b0 2024-08-28T19:22:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27bb638740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27b01b8d50 con 0x55abe05dd390 2024-08-28T19:22:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27bb638740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f27b01b78e0 con 0x55abe05fe070 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27af59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09a6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27af59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09a6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45868/0 (socket says 172.21.15.16:45868) 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27af59e640 1 -- 172.21.15.16:0/4085403484 learned_addr learned my addr 172.21.15.16:0/4085403484 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27ae59c640 1 --2- 172.21.15.16:0/4085403484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 0x55abe09bad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27af59e640 1 --2- 172.21.15.16:0/4085403484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05fe070 0x55abe09bb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27ae59c640 1 -- 172.21.15.16:0/4085403484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05fe070 msgr2=0x55abe09bb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27ae59c640 1 --2- 172.21.15.16:0/4085403484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05fe070 0x55abe09bb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27ae59c640 1 -- 172.21.15.16:0/4085403484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 msgr2=0x55abe09a6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27ae59c640 1 --2- 172.21.15.16:0/4085403484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09a6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27ae59c640 1 -- 172.21.15.16:0/4085403484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abe09a7060 con 0x55abe05dd390 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27af59e640 1 --2- 172.21.15.16:0/4085403484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09a6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.471+0000 7f27ae59c640 1 --2- 172.21.15.16:0/4085403484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 0x55abe09bad10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f27a401c870 tx=0x7f27a40059b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ca60cae207f101c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.472+0000 7f2794ff9640 1 -- 172.21.15.16:0/4085403484 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a4022040 con 0x55abe05dd390 2024-08-28T19:22:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.472+0000 7f2794ff9640 1 -- 172.21.15.16:0/4085403484 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27a40120d0 con 0x55abe05dd390 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.472+0000 7f2794ff9640 1 -- 172.21.15.16:0/4085403484 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a4005040 con 0x55abe05dd390 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.473+0000 7f27bb638740 1 -- 172.21.15.16:0/4085403484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 msgr2=0x55abe09bad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.473+0000 7f27bb638740 1 --2- 172.21.15.16:0/4085403484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 0x55abe09bad10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f27a401c870 tx=0x7f27a40059b0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.473+0000 7f27bb638740 1 -- 172.21.15.16:0/4085403484 shutdown_connections 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.473+0000 7f27bb638740 1 --2- 172.21.15.16:0/4085403484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05fe070 0x55abe09bb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.473+0000 7f27bb638740 1 --2- 172.21.15.16:0/4085403484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05dd390 0x55abe09bad10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.473+0000 7f27bb638740 1 --2- 172.21.15.16:0/4085403484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09a6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.473+0000 7f27bb638740 1 -- 172.21.15.16:0/4085403484 >> 172.21.15.16:0/4085403484 conn(0x55abe05f2460 msgr2=0x55abe09a6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.473+0000 7f27bb638740 1 -- 172.21.15.16:0/4085403484 shutdown_connections 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.473+0000 7f27bb638740 1 -- 172.21.15.16:0/4085403484 wait complete. 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.473+0000 7f27bb638740 1 -- start start 2024-08-28T19:22:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27bb638740 1 Processor -- start 2024-08-28T19:22:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27bb638740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27bb638740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05dd390 0x55abe09b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27bb638740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 0x55abe0a247e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27bb638740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27b01b8680 con 0x55abe05d51b0 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27bb638740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27b01b7fb0 con 0x55abe05fe070 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27bb638740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f27b01b51f0 con 0x55abe05dd390 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27bb638740 10 client.0 _ll_register_callbacks cb 0x7f27a4009ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27af59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27af59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09b35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45884/0 (socket says 172.21.15.16:45884) 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27af59e640 1 -- 172.21.15.16:0/3345168160 learned_addr learned my addr 172.21.15.16:0/3345168160 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27ae59c640 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05dd390 0x55abe09b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27af59e640 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 0x55abe0a247e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27ae59c640 1 -- 172.21.15.16:0/3345168160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 msgr2=0x55abe0a247e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27ae59c640 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 0x55abe0a247e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27ae59c640 1 -- 172.21.15.16:0/3345168160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 msgr2=0x55abe09b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27ae59c640 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27ae59c640 1 -- 172.21.15.16:0/3345168160 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abe0a24e20 con 0x55abe05dd390 2024-08-28T19:22:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.474+0000 7f27af59e640 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.475+0000 7f27ae59c640 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05dd390 0x55abe09b3b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55abe099f6f0 tx=0x7f27a401c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dd130a90a5fe245 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.475+0000 7f2792ff5640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.475+0000 7f2792ff5640 1 -- 172.21.15.16:0/3345168160 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a4037060 con 0x55abe05dd390 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.475+0000 7f27bb638740 1 -- 172.21.15.16:0/3345168160 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27a40120d0 con 0x55abe05dd390 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.475+0000 7f2792ff5640 1 -- 172.21.15.16:0/3345168160 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f27a4004020 con 0x55abe05dd390 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.475+0000 7f2792ff5640 1 -- 172.21.15.16:0/3345168160 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f27a403e050 con 0x55abe05dd390 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.475+0000 7f27bb638740 10 client.25612 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.475+0000 7f27bb638740 1 -- 172.21.15.16:0/3345168160 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55abe0a2c150 con 0x55abe05dd390 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.475+0000 7f2792ff5640 1 -- 172.21.15.16:0/3345168160 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27a40120d0 con 0x55abe05dd390 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.476+0000 7f2792ff5640 1 -- 172.21.15.16:0/3345168160 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f27a4004680 con 0x55abe05dd390 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.476+0000 7f27bb638740 10 client.25612 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.476+0000 7f27bb638740 10 client.25612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.476+0000 7f27bb638740 20 client.25612 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.476+0000 7f27bb638740 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55abe0591da0 0x55abe0a2e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.476+0000 7f27bb638740 4 client.25612 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.476+0000 7f27bb638740 1 -- 172.21.15.16:0/3345168160 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55abe0918070 con 0x55abe0591da0 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.476+0000 7f27af59e640 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55abe0591da0 0x55abe0a2e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.476+0000 7f27af59e640 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55abe0591da0 0x55abe0a2e7c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5e7d93c193136b6 server_cookie=b2ee17ef8f153e75 in_seq=0 out_seq=0 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.476+0000 7f2792ff5640 10 client.25612 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f2792ff5640 1 -- 172.21.15.16:0/3345168160 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55abe0918070 con 0x55abe0591da0 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f2792ff5640 10 client.25612 handle_command_reply: tid=0 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 1 client.25612 shutdown 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 1 -- 172.21.15.16:0/3345168160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05dd390 msgr2=0x55abe09b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05dd390 0x55abe09b3b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55abe099f6f0 tx=0x7f27a401c6a0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 1 -- 172.21.15.16:0/3345168160 shutdown_connections 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abe05fe070 0x55abe0a247e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55abe0591da0 0x55abe0a2e7c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55abe05dd390 0x55abe09b3b80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 1 --2- 172.21.15.16:0/3345168160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abe05d51b0 0x55abe09b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f2792ff5640 0 client.25612 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 1 -- 172.21.15.16:0/3345168160 >> 172.21.15.16:0/3345168160 conn(0x55abe05f2460 msgr2=0x55abe09a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 1 -- 172.21.15.16:0/3345168160 shutdown_connections 2024-08-28T19:22:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 1 -- 172.21.15.16:0/3345168160 wait complete. 2024-08-28T19:22:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.477+0000 7f27bb638740 20 client.25612 trim_cache size 0 max 16384 2024-08-28T19:22:38.482 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:22:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 -- 172.21.15.16:0/4070743417 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27980b4d20 msgr2=0x7f27980b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27980b4d20 0x7f27980b7230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55abe09a1f90 tx=0x7f27a002a410 comp rx=0 tx=0).stop 2024-08-28T19:22:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 -- 172.21.15.16:0/4070743417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27b0103910 msgr2=0x7f27b01a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27b0103910 0x7f27b01a1520 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f27a002a930 tx=0x7f27a0031a20 comp rx=0 tx=0).stop 2024-08-28T19:22:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 -- 172.21.15.16:0/4070743417 shutdown_connections 2024-08-28T19:22:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b0109ba0 0x7f27b01ac5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0109250 0x7f27b01a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27980b4d20 0x7f27980b7230 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 --2- 172.21.15.16:0/4070743417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27b0103910 0x7f27b01a1520 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 -- 172.21.15.16:0/4070743417 >> 172.21.15.16:0/4070743417 conn(0x7f27b00ff380 msgr2=0x7f27b00ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 -- 172.21.15.16:0/4070743417 shutdown_connections 2024-08-28T19:22:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:38.478+0000 7f27b50c3640 1 -- 172.21.15.16:0/4070743417 wait complete. 2024-08-28T19:22:38.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:38 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:22:38.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:38 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:22:38.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:38 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:22:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:38 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:22:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:38 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:22:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:38 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:22:38.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 53953 17.11 MB/sec warmup 18 sec latency 15.017 ms 2024-08-28T19:22:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:39 smithi016 ceph-mon[18556]: pgmap v387: 129 pgs: 129 active+clean; 22 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-08-28T19:22:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:39 smithi028 ceph-mon[21769]: pgmap v387: 129 pgs: 129 active+clean; 22 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-08-28T19:22:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:39 smithi115 ceph-mon[22184]: pgmap v387: 129 pgs: 129 active+clean; 22 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-08-28T19:22:39.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 57171 17.21 MB/sec warmup 19 sec latency 19.132 ms 2024-08-28T19:22:40.602 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:22:40.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:22:40.862 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:40.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 60552 17.29 MB/sec warmup 20 sec latency 13.358 ms 2024-08-28T19:22:41.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:41 smithi016 ceph-mon[18556]: pgmap v388: 129 pgs: 129 active+clean; 23 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-08-28T19:22:41.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.451+0000 7fcf88141640 1 -- 172.21.15.16:0/2574596971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80108170 msgr2=0x7fcf80108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.451+0000 7fcf88141640 1 --2- 172.21.15.16:0/2574596971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80108170 0x7fcf80108580 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fcf74009900 tx=0x7fcf7402a3f0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.451+0000 7fcf88141640 1 -- 172.21.15.16:0/2574596971 shutdown_connections 2024-08-28T19:22:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.451+0000 7fcf88141640 1 --2- 172.21.15.16:0/2574596971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf80108ac0 0x7fcf8010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.451+0000 7fcf88141640 1 --2- 172.21.15.16:0/2574596971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80108170 0x7fcf80108580 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.451+0000 7fcf88141640 1 --2- 172.21.15.16:0/2574596971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80102880 0x7fcf80102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.451+0000 7fcf88141640 1 -- 172.21.15.16:0/2574596971 >> 172.21.15.16:0/2574596971 conn(0x7fcf80079530 msgr2=0x7fcf80079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.451+0000 7fcf88141640 1 -- 172.21.15.16:0/2574596971 shutdown_connections 2024-08-28T19:22:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.451+0000 7fcf88141640 1 -- 172.21.15.16:0/2574596971 wait complete. 2024-08-28T19:22:41.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf88141640 1 Processor -- start 2024-08-28T19:22:41.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf88141640 1 -- start start 2024-08-28T19:22:41.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf88141640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf80102880 0x7fcf8019a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf88141640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80108170 0x7fcf8019a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf88141640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80108ac0 0x7fcf8019fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf88141640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf80112730 con 0x7fcf80108ac0 2024-08-28T19:22:41.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf88141640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801125b0 con 0x7fcf80108170 2024-08-28T19:22:41.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf88141640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801128b0 con 0x7fcf80102880 2024-08-28T19:22:41.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf856dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80108170 0x7fcf8019a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf856dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80108170 0x7fcf8019a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41496/0 (socket says 172.21.15.16:41496) 2024-08-28T19:22:41.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf856dc640 1 -- 172.21.15.16:0/3136897624 learned_addr learned my addr 172.21.15.16:0/3136897624 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:41.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.452+0000 7fcf866de640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80108ac0 0x7fcf8019fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf85edd640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf80102880 0x7fcf8019a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf856dc640 1 -- 172.21.15.16:0/3136897624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf80102880 msgr2=0x7fcf8019a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf856dc640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf80102880 0x7fcf8019a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf856dc640 1 -- 172.21.15.16:0/3136897624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80108ac0 msgr2=0x7fcf8019fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf856dc640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80108ac0 0x7fcf8019fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf856dc640 1 -- 172.21.15.16:0/3136897624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf801a0240 con 0x7fcf80108170 2024-08-28T19:22:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf856dc640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80108170 0x7fcf8019a870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcf74006010 tx=0x7fcf74004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf7effd640 1 -- 172.21.15.16:0/3136897624 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7403f030 con 0x7fcf80108170 2024-08-28T19:22:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf88141640 1 -- 172.21.15.16:0/3136897624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf801a04d0 con 0x7fcf80108170 2024-08-28T19:22:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf7effd640 1 -- 172.21.15.16:0/3136897624 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf74003aa0 con 0x7fcf80108170 2024-08-28T19:22:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf7effd640 1 -- 172.21.15.16:0/3136897624 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf74035070 con 0x7fcf80108170 2024-08-28T19:22:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.453+0000 7fcf88141640 1 -- 172.21.15.16:0/3136897624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf800ff2f0 con 0x7fcf80108170 2024-08-28T19:22:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.456+0000 7fcf7effd640 1 -- 172.21.15.16:0/3136897624 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fcf74042030 con 0x7fcf80108170 2024-08-28T19:22:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.456+0000 7fcf8e6b6740 1 Processor -- start 2024-08-28T19:22:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.456+0000 7fcf8e6b6740 1 -- start start 2024-08-28T19:22:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c8e350 0x557d72034780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 0x557d72034cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c6d670 0x557d72035200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557d71faa5f0 con 0x557d71c65490 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d71f99b80 con 0x557d71c6d670 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557d71f9db30 con 0x557d71c8e350 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf85edd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 0x557d72034cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf866de640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c8e350 0x557d72034780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf85edd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 0x557d72034cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45912/0 (socket says 172.21.15.16:45912) 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf866de640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c8e350 0x557d72034780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:37694/0 (socket says 172.21.15.16:37694) 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf85edd640 1 -- 172.21.15.16:0/2226828506 learned_addr learned my addr 172.21.15.16:0/2226828506 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf856dc640 1 --2- 172.21.15.16:0/2226828506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c6d670 0x557d72035200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf866de640 1 -- 172.21.15.16:0/2226828506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c6d670 msgr2=0x557d72035200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf866de640 1 --2- 172.21.15.16:0/2226828506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c6d670 0x557d72035200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf866de640 1 -- 172.21.15.16:0/2226828506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 msgr2=0x557d72034cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf866de640 1 --2- 172.21.15.16:0/2226828506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 0x557d72034cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf866de640 1 -- 172.21.15.16:0/2226828506 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d71cf9930 con 0x557d71c8e350 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf7effd640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf680b4ca0 0x7fcf680b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf85edd640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf680b4ca0 0x7fcf680b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.457+0000 7fcf7effd640 1 -- 172.21.15.16:0/3136897624 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fcf74105d20 con 0x7fcf80108170 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.458+0000 7fcf866de640 1 --2- 172.21.15.16:0/2226828506 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c8e350 0x557d72034780 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800ff10 tx=0x7fcf78004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a23656973207003a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.458+0000 7fcf67fff640 1 -- 172.21.15.16:0/2226828506 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78003dd0 con 0x557d71c8e350 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.458+0000 7fcf85edd640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf680b4ca0 0x7fcf680b71b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x557d72033400 tx=0x7fcf70009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.458+0000 7fcf67fff640 1 -- 172.21.15.16:0/2226828506 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf7801e070 con 0x557d71c8e350 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.458+0000 7fcf67fff640 1 -- 172.21.15.16:0/2226828506 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78018650 con 0x557d71c8e350 2024-08-28T19:22:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.459+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/2226828506 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c8e350 msgr2=0x557d72034780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.459+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/2226828506 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c8e350 0x557d72034780 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800ff10 tx=0x7fcf78004410 comp rx=0 tx=0).stop 2024-08-28T19:22:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.459+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/2226828506 shutdown_connections 2024-08-28T19:22:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.459+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/2226828506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c6d670 0x557d72035200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.459+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/2226828506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 0x557d72034cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.459+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/2226828506 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c8e350 0x557d72034780 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.459+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/2226828506 >> 172.21.15.16:0/2226828506 conn(0x557d71c82740 msgr2=0x557d72032670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.459+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/2226828506 shutdown_connections 2024-08-28T19:22:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.459+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/2226828506 wait complete. 2024-08-28T19:22:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.459+0000 7fcf8e6b6740 1 -- start start 2024-08-28T19:22:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf8e6b6740 1 Processor -- start 2024-08-28T19:22:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 0x557d7203ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c6d670 0x557d72040490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d7204c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557d71dfc890 con 0x557d71c65490 2024-08-28T19:22:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d71f20790 con 0x557d71c8e350 2024-08-28T19:22:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557d72042f60 con 0x557d71c6d670 2024-08-28T19:22:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf8e6b6740 10 client.0 _ll_register_callbacks cb 0x557d71ca5ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf85edd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 0x557d7203ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf856dc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c6d670 0x557d72040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf85edd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 0x557d7203ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45918/0 (socket says 172.21.15.16:45918) 2024-08-28T19:22:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf85edd640 1 -- 172.21.15.16:0/1576223761 learned_addr learned my addr 172.21.15.16:0/1576223761 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.460+0000 7fcf866de640 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d7204c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf856dc640 1 -- 172.21.15.16:0/1576223761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 msgr2=0x557d7204c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf856dc640 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d7204c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf856dc640 1 -- 172.21.15.16:0/1576223761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 msgr2=0x557d7203ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf856dc640 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 0x557d7203ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf856dc640 1 -- 172.21.15.16:0/1576223761 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d71f57f30 con 0x557d71c6d670 2024-08-28T19:22:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf866de640 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d7204c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf856dc640 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c6d670 0x557d72040490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcf74103100 tx=0x7fcf741093c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ae4d9cc219b5a53 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf657fa640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:22:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf657fa640 1 -- 172.21.15.16:0/1576223761 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf74049070 con 0x557d71c6d670 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf657fa640 1 -- 172.21.15.16:0/1576223761 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fcf740c8740 con 0x557d71c6d670 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/1576223761 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf7801e070 con 0x557d71c6d670 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf657fa640 1 -- 172.21.15.16:0/1576223761 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf740c8010 con 0x557d71c6d670 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf8e6b6740 10 client.25624 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/1576223761 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557d7204ed60 con 0x557d71c6d670 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.461+0000 7fcf657fa640 1 -- 172.21.15.16:0/1576223761 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf7801e070 con 0x557d71c6d670 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.462+0000 7fcf657fa640 1 -- 172.21.15.16:0/1576223761 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcf740c83e0 con 0x557d71c6d670 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.462+0000 7fcf8e6b6740 10 client.25624 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.462+0000 7fcf8e6b6740 10 client.25624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.462+0000 7fcf8e6b6740 20 client.25624 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.462+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557d71c4ccd0 0x557d72052250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.462+0000 7fcf8e6b6740 4 client.25624 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.462+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/1576223761 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d71fac540 con 0x557d71c4ccd0 2024-08-28T19:22:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.462+0000 7fcf85edd640 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557d71c4ccd0 0x557d72052250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.462+0000 7fcf657fa640 10 client.25624 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.462+0000 7fcf85edd640 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557d71c4ccd0 0x557d72052250 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5254c23d2be6b7ae server_cookie=f2f6b684faba9ccd in_seq=0 out_seq=0 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.463+0000 7fcf657fa640 1 -- 172.21.15.16:0/1576223761 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557d71fac540 con 0x557d71c4ccd0 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.463+0000 7fcf657fa640 10 client.25624 handle_command_reply: tid=0 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.463+0000 7fcf8e6b6740 1 client.25624 shutdown 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/1576223761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c6d670 msgr2=0x557d72040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c6d670 0x557d72040490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcf74103100 tx=0x7fcf741093c0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/1576223761 shutdown_connections 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d7204c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557d71c4ccd0 0x557d72052250 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c6d670 0x557d72040490 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/1576223761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c65490 0x557d7203ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf657fa640 0 client.25624 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/1576223761 >> 172.21.15.16:0/1576223761 conn(0x557d71c82740 msgr2=0x557d72037910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/1576223761 shutdown_connections 2024-08-28T19:22:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/1576223761 wait complete. 2024-08-28T19:22:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.464+0000 7fcf8e6b6740 20 client.25624 trim_cache size 0 max 16384 2024-08-28T19:22:41.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.482+0000 7fcf8e6b6740 1 Processor -- start 2024-08-28T19:22:41.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.482+0000 7fcf8e6b6740 1 -- start start 2024-08-28T19:22:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.482+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c65490 0x557d72037f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.482+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d7204c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.482+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d7203d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.482+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801ae9d0 con 0x557d71c6d670 2024-08-28T19:22:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.482+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801b0510 con 0x557d71c8e350 2024-08-28T19:22:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.482+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801af0a0 con 0x557d71c65490 2024-08-28T19:22:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf856dc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d7204c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf856dc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d7204c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45922/0 (socket says 172.21.15.16:45922) 2024-08-28T19:22:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf856dc640 1 -- 172.21.15.16:0/436792889 learned_addr learned my addr 172.21.15.16:0/436792889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf866de640 1 --2- 172.21.15.16:0/436792889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d7203d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf85edd640 1 --2- 172.21.15.16:0/436792889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c65490 0x557d72037f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf856dc640 1 -- 172.21.15.16:0/436792889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c65490 msgr2=0x557d72037f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf856dc640 1 --2- 172.21.15.16:0/436792889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c65490 0x557d72037f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf856dc640 1 -- 172.21.15.16:0/436792889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 msgr2=0x557d7203d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf856dc640 1 --2- 172.21.15.16:0/436792889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d7203d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf856dc640 1 -- 172.21.15.16:0/436792889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d72041210 con 0x557d71c6d670 2024-08-28T19:22:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf856dc640 1 --2- 172.21.15.16:0/436792889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d7204c180 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcf74009320 tx=0x7fcf7403b520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6ef27a3c28e3ef server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf677fe640 1 -- 172.21.15.16:0/436792889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf74049070 con 0x557d71c6d670 2024-08-28T19:22:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf677fe640 1 -- 172.21.15.16:0/436792889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf74109560 con 0x557d71c6d670 2024-08-28T19:22:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.483+0000 7fcf677fe640 1 -- 172.21.15.16:0/436792889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf740cb8c0 con 0x557d71c6d670 2024-08-28T19:22:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.484+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/436792889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 msgr2=0x557d7204c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.484+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/436792889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d7204c180 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcf74009320 tx=0x7fcf7403b520 comp rx=0 tx=0).stop 2024-08-28T19:22:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.484+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/436792889 shutdown_connections 2024-08-28T19:22:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.484+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/436792889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d7203d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.484+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/436792889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d7204c180 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.484+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/436792889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c65490 0x557d72037f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.484+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/436792889 >> 172.21.15.16:0/436792889 conn(0x557d71c82740 msgr2=0x557d72037880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.484+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/436792889 shutdown_connections 2024-08-28T19:22:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.484+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/436792889 wait complete. 2024-08-28T19:22:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.485+0000 7fcf8e6b6740 1 -- start start 2024-08-28T19:22:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.485+0000 7fcf8e6b6740 1 Processor -- start 2024-08-28T19:22:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.485+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c65490 0x557d720489e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.485+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d72044ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.485+0000 7fcf8e6b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d72045400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.485+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801afe40 con 0x557d71c6d670 2024-08-28T19:22:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.485+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801af770 con 0x557d71c8e350 2024-08-28T19:22:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.485+0000 7fcf8e6b6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801ac9b0 con 0x557d71c65490 2024-08-28T19:22:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.485+0000 7fcf8e6b6740 10 client.0 _ll_register_callbacks cb 0x7fcf74005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf866de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d72045400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf856dc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d72044ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf866de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d72045400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41510/0 (socket says 172.21.15.16:41510) 2024-08-28T19:22:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf856dc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d72044ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45930/0 (socket says 172.21.15.16:45930) 2024-08-28T19:22:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf866de640 1 -- 172.21.15.16:0/245100342 learned_addr learned my addr 172.21.15.16:0/245100342 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf85edd640 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c65490 0x557d720489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf866de640 1 -- 172.21.15.16:0/245100342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c65490 msgr2=0x557d720489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf866de640 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c65490 0x557d720489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf866de640 1 -- 172.21.15.16:0/245100342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 msgr2=0x557d72044ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf866de640 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d72044ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf866de640 1 -- 172.21.15.16:0/245100342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d72045b90 con 0x557d71c8e350 2024-08-28T19:22:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf856dc640 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d72044ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf866de640 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d72045400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800fa90 tx=0x7fcf78005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6370d0cbf9766d37 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.486+0000 7fcf657fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf657fa640 1 -- 172.21.15.16:0/245100342 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78031060 con 0x557d71c8e350 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf657fa640 1 -- 172.21.15.16:0/245100342 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fcf78004660 con 0x557d71c8e350 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/245100342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf74109560 con 0x557d71c8e350 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf657fa640 1 -- 172.21.15.16:0/245100342 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf78038050 con 0x557d71c8e350 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf8e6b6740 10 client.25745 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/245100342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557d720bdba0 con 0x557d71c8e350 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf657fa640 1 -- 172.21.15.16:0/245100342 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf74109560 con 0x557d71c8e350 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf657fa640 1 -- 172.21.15.16:0/245100342 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcf780197a0 con 0x557d71c8e350 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf8e6b6740 10 client.25745 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf8e6b6740 10 client.25745 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf8e6b6740 20 client.25745 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557d71c22080 0x557d720bffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf8e6b6740 4 client.25745 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:22:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/245100342 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557d71fa7fe0 con 0x557d71c22080 2024-08-28T19:22:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.487+0000 7fcf85edd640 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557d71c22080 0x557d720bffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.488+0000 7fcf657fa640 10 client.25745 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.488+0000 7fcf85edd640 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557d71c22080 0x557d720bffd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95a5b8778f4ede16 server_cookie=79b9a3912e5e81aa in_seq=0 out_seq=0 2024-08-28T19:22:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.490+0000 7fcf657fa640 1 -- 172.21.15.16:0/245100342 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557d71fa7fe0 con 0x557d71c22080 2024-08-28T19:22:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.490+0000 7fcf657fa640 10 client.25745 handle_command_reply: tid=0 2024-08-28T19:22:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.490+0000 7fcf8e6b6740 1 client.25745 shutdown 2024-08-28T19:22:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/245100342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 msgr2=0x557d72045400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d72045400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800fa90 tx=0x7fcf78005250 comp rx=0 tx=0).stop 2024-08-28T19:22:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/245100342 shutdown_connections 2024-08-28T19:22:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d71c8e350 0x557d72045400 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d71c6d670 0x557d72044ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557d71c22080 0x557d720bffd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf8e6b6740 1 --2- 172.21.15.16:0/245100342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d71c65490 0x557d720489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf657fa640 0 client.25745 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/245100342 >> 172.21.15.16:0/245100342 conn(0x557d71c82740 msgr2=0x557d72035760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/245100342 shutdown_connections 2024-08-28T19:22:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf8e6b6740 1 -- 172.21.15.16:0/245100342 wait complete. 2024-08-28T19:22:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.491+0000 7fcf8e6b6740 20 client.25745 trim_cache size 0 max 16384 2024-08-28T19:22:41.496 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:22:41.496 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:22:41.496 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "fa70608a-b5e8-4395-998a-2f7edb54b221", 2024-08-28T19:22:41.496 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:22:41.496 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:22:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 -- 172.21.15.16:0/3136897624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf680b4ca0 msgr2=0x7fcf680b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf680b4ca0 0x7fcf680b71b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x557d72033400 tx=0x7fcf70009500 comp rx=0 tx=0).stop 2024-08-28T19:22:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 -- 172.21.15.16:0/3136897624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80108170 msgr2=0x7fcf8019a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80108170 0x7fcf8019a870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcf74006010 tx=0x7fcf74004060 comp rx=0 tx=0).stop 2024-08-28T19:22:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 -- 172.21.15.16:0/3136897624 shutdown_connections 2024-08-28T19:22:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80108ac0 0x7fcf8019fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf680b4ca0 0x7fcf680b71b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80108170 0x7fcf8019a870 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 --2- 172.21.15.16:0/3136897624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf80102880 0x7fcf8019a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 -- 172.21.15.16:0/3136897624 >> 172.21.15.16:0/3136897624 conn(0x7fcf80079530 msgr2=0x7fcf80101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 -- 172.21.15.16:0/3136897624 shutdown_connections 2024-08-28T19:22:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:41.492+0000 7fcf88141640 1 -- 172.21.15.16:0/3136897624 wait complete. 2024-08-28T19:22:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:41 smithi028 ceph-mon[21769]: pgmap v388: 129 pgs: 129 active+clean; 23 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-08-28T19:22:41.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:41 smithi115 ceph-mon[22184]: pgmap v388: 129 pgs: 129 active+clean; 23 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-08-28T19:22:41.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 62895 16.86 MB/sec warmup 21 sec latency 45.661 ms 2024-08-28T19:22:42.487 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:22:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:42.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 65510 16.84 MB/sec warmup 22 sec latency 14.266 ms 2024-08-28T19:22:43.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:43 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:22:43.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:43 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:43.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:43 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:22:43.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:43 smithi016 ceph-mon[18556]: pgmap v389: 129 pgs: 129 active+clean; 28 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.7 MiB/s wr, 217 op/s 2024-08-28T19:22:43.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:43 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.308+0000 7f4b22dde640 1 -- 172.21.15.16:0/2080288736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c102830 msgr2=0x7f4b1c102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.308+0000 7f4b22dde640 1 --2- 172.21.15.16:0/2080288736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c102830 0x7f4b1c102c40 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009ca0 tx=0x7f4b0c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.308+0000 7f4b22dde640 1 -- 172.21.15.16:0/2080288736 shutdown_connections 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.308+0000 7f4b22dde640 1 --2- 172.21.15.16:0/2080288736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b1c108ac0 0x7f4b1c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.308+0000 7f4b22dde640 1 --2- 172.21.15.16:0/2080288736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c108170 0x7f4b1c108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.308+0000 7f4b22dde640 1 --2- 172.21.15.16:0/2080288736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c102830 0x7f4b1c102c40 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.308+0000 7f4b22dde640 1 -- 172.21.15.16:0/2080288736 >> 172.21.15.16:0/2080288736 conn(0x7f4b1c0794d0 msgr2=0x7f4b1c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.308+0000 7f4b22dde640 1 -- 172.21.15.16:0/2080288736 shutdown_connections 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.308+0000 7f4b22dde640 1 -- 172.21.15.16:0/2080288736 wait complete. 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b22dde640 1 Processor -- start 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b22dde640 1 -- start start 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b22dde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c102830 0x7f4b1c19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b22dde640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c108170 0x7f4b1c19b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b22dde640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b1c108ac0 0x7f4b1c19bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b22dde640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c112770 con 0x7f4b1c108170 2024-08-28T19:22:43.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b22dde640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c1125f0 con 0x7f4b1c102830 2024-08-28T19:22:43.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b22dde640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c1128f0 con 0x7f4b1c108ac0 2024-08-28T19:22:43.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b1bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c108170 0x7f4b1c19b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b20b7a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c102830 0x7f4b1c19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b1bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c108170 0x7f4b1c19b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45946/0 (socket says 172.21.15.16:45946) 2024-08-28T19:22:43.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b1bfff640 1 -- 172.21.15.16:0/97718910 learned_addr learned my addr 172.21.15.16:0/97718910 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:43.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b2137b640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b1c108ac0 0x7f4b1c19bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b1bfff640 1 -- 172.21.15.16:0/97718910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b1c108ac0 msgr2=0x7f4b1c19bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:43.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b1c108ac0 0x7f4b1c19bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b1bfff640 1 -- 172.21.15.16:0/97718910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c102830 msgr2=0x7f4b1c19b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c102830 0x7f4b1c19b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.309+0000 7f4b1bfff640 1 -- 172.21.15.16:0/97718910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1c1a6330 con 0x7f4b1c108170 2024-08-28T19:22:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.310+0000 7f4b2137b640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b1c108ac0 0x7f4b1c19bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.310+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c108170 0x7f4b1c19b5e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400e9f0 tx=0x7f4b0400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.310+0000 7f4b19ffb640 1 -- 172.21.15.16:0/97718910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0400ccc0 con 0x7f4b1c108170 2024-08-28T19:22:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.310+0000 7f4b19ffb640 1 -- 172.21.15.16:0/97718910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b040108e0 con 0x7f4b1c108170 2024-08-28T19:22:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.310+0000 7f4b22dde640 1 -- 172.21.15.16:0/97718910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b1c198ef0 con 0x7f4b1c108170 2024-08-28T19:22:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.310+0000 7f4b19ffb640 1 -- 172.21.15.16:0/97718910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b04018860 con 0x7f4b1c108170 2024-08-28T19:22:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.310+0000 7f4b22dde640 1 -- 172.21.15.16:0/97718910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b1c199430 con 0x7f4b1c108170 2024-08-28T19:22:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.313+0000 7f4b29353740 1 Processor -- start 2024-08-28T19:22:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.313+0000 7f4b19ffb640 1 -- 172.21.15.16:0/97718910 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4b04010430 con 0x7f4b1c108170 2024-08-28T19:22:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.313+0000 7f4b29353740 1 -- start start 2024-08-28T19:22:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.313+0000 7f4b29353740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 0x5645af97f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.313+0000 7f4b29353740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 0x5645af97f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.313+0000 7f4b29353740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 0x5645af97fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.313+0000 7f4b29353740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5645af8f6630 con 0x5645af5da070 2024-08-28T19:22:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.313+0000 7f4b29353740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5645af8e5bc0 con 0x5645af5b9390 2024-08-28T19:22:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.313+0000 7f4b29353740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5645af8e9b70 con 0x5645af5b11b0 2024-08-28T19:22:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.313+0000 7f4b2137b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 0x5645af97f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.313+0000 7f4b2137b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 0x5645af97f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45960/0 (socket says 172.21.15.16:45960) 2024-08-28T19:22:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b2137b640 1 -- 172.21.15.16:0/3579819079 learned_addr learned my addr 172.21.15.16:0/3579819079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/3579819079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 0x5645af97fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/3579819079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 0x5645af97f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b2137b640 1 -- 172.21.15.16:0/3579819079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 msgr2=0x5645af97f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b2137b640 1 --2- 172.21.15.16:0/3579819079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 0x5645af97f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b2137b640 1 -- 172.21.15.16:0/3579819079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 msgr2=0x5645af97fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b2137b640 1 --2- 172.21.15.16:0/3579819079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 0x5645af97fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b2137b640 1 -- 172.21.15.16:0/3579819079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645af645650 con 0x5645af5da070 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/3579819079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 0x5645af97f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b19ffb640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b080b4cd0 0x7f4b080b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/3579819079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 0x5645af97fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b080b4cd0 0x7f4b080b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b19ffb640 1 -- 172.21.15.16:0/97718910 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f4b040e5be0 con 0x7f4b1c108170 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b2137b640 1 --2- 172.21.15.16:0/3579819079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 0x5645af97f2c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400bcf0 tx=0x7f4b1400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29e16e8676f11a6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.314+0000 7f4b12ffd640 1 -- 172.21.15.16:0/3579819079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b14015070 con 0x5645af5da070 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.315+0000 7f4b12ffd640 1 -- 172.21.15.16:0/3579819079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b140118f0 con 0x5645af5da070 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.315+0000 7f4b12ffd640 1 -- 172.21.15.16:0/3579819079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b14010840 con 0x5645af5da070 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.315+0000 7f4b29353740 1 -- 172.21.15.16:0/3579819079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 msgr2=0x5645af97f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.315+0000 7f4b29353740 1 --2- 172.21.15.16:0/3579819079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 0x5645af97f2c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400bcf0 tx=0x7f4b1400f730 comp rx=0 tx=0).stop 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.315+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b080b4cd0 0x7f4b080b71e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c005e30 tx=0x7f4b0c009360 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.315+0000 7f4b29353740 1 -- 172.21.15.16:0/3579819079 shutdown_connections 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.315+0000 7f4b29353740 1 --2- 172.21.15.16:0/3579819079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 0x5645af97fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.315+0000 7f4b29353740 1 --2- 172.21.15.16:0/3579819079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 0x5645af97f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.315+0000 7f4b29353740 1 --2- 172.21.15.16:0/3579819079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 0x5645af97f2c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.316+0000 7f4b29353740 1 -- 172.21.15.16:0/3579819079 >> 172.21.15.16:0/3579819079 conn(0x5645af5ce460 msgr2=0x5645af97d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.316+0000 7f4b29353740 1 -- 172.21.15.16:0/3579819079 shutdown_connections 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.316+0000 7f4b29353740 1 -- 172.21.15.16:0/3579819079 wait complete. 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.316+0000 7f4b29353740 1 -- start start 2024-08-28T19:22:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.316+0000 7f4b29353740 1 Processor -- start 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b29353740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b11b0 0x5645af98aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b29353740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b9390 0x5645af98b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b29353740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5da070 0x5645af996d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b29353740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5645af748700 con 0x5645af5b9390 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b29353740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5645af86c700 con 0x5645af5b11b0 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b29353740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5645af98daa0 con 0x5645af5da070 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b29353740 10 client.0 _ll_register_callbacks cb 0x5645af89bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b1bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b9390 0x5645af98b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b1bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b9390 0x5645af98b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45972/0 (socket says 172.21.15.16:45972) 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b1bfff640 1 -- 172.21.15.16:0/630858441 learned_addr learned my addr 172.21.15.16:0/630858441 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b11b0 0x5645af98aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b2137b640 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5da070 0x5645af996d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b1bfff640 1 -- 172.21.15.16:0/630858441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5da070 msgr2=0x5645af996d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5da070 0x5645af996d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b1bfff640 1 -- 172.21.15.16:0/630858441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b11b0 msgr2=0x5645af98aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b11b0 0x5645af98aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b1bfff640 1 -- 172.21.15.16:0/630858441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645af8a58b0 con 0x5645af5b9390 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b11b0 0x5645af98aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b9390 0x5645af98b000 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400c6a0 tx=0x7f4b0400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f4b57d279dc10e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.317+0000 7f4afbfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4afbfff640 1 -- 172.21.15.16:0/630858441 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b040aba70 con 0x5645af5b9390 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4afbfff640 1 -- 172.21.15.16:0/630858441 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f4b040b3b50 con 0x5645af5b9390 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4b29353740 1 -- 172.21.15.16:0/630858441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b140118f0 con 0x5645af5b9390 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4afbfff640 1 -- 172.21.15.16:0/630858441 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4b040ac660 con 0x5645af5b9390 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4b29353740 10 client.16119 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4b29353740 1 -- 172.21.15.16:0/630858441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5645af999940 con 0x5645af5b9390 2024-08-28T19:22:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4afbfff640 1 -- 172.21.15.16:0/630858441 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b140118f0 con 0x5645af5b9390 2024-08-28T19:22:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4afbfff640 1 -- 172.21.15.16:0/630858441 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4b040b0420 con 0x5645af5b9390 2024-08-28T19:22:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4b29353740 10 client.16119 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4b29353740 10 client.16119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4b29353740 20 client.16119 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4b29353740 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5645af5989f0 0x5645af99ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4b29353740 4 client.16119 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4b29353740 1 -- 172.21.15.16:0/630858441 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645af8f8580 con 0x5645af5989f0 2024-08-28T19:22:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.318+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5645af5989f0 0x5645af99ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.319+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5645af5989f0 0x5645af99ce10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7202db246b30aaa server_cookie=6461c39f8b5ed9d1 in_seq=0 out_seq=0 2024-08-28T19:22:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.319+0000 7f4afbfff640 10 client.16119 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.320+0000 7f4afbfff640 1 -- 172.21.15.16:0/630858441 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5645af8f8580 con 0x5645af5989f0 2024-08-28T19:22:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.320+0000 7f4afbfff640 10 client.16119 handle_command_reply: tid=0 2024-08-28T19:22:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.320+0000 7f4b29353740 1 client.16119 shutdown 2024-08-28T19:22:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.320+0000 7f4b29353740 1 -- 172.21.15.16:0/630858441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b9390 msgr2=0x5645af98b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.320+0000 7f4b29353740 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b9390 0x5645af98b000 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400c6a0 tx=0x7f4b0400ce60 comp rx=0 tx=0).stop 2024-08-28T19:22:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.320+0000 7f4b29353740 1 -- 172.21.15.16:0/630858441 shutdown_connections 2024-08-28T19:22:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.320+0000 7f4b29353740 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5645af5989f0 0x5645af99ce10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.321+0000 7f4b29353740 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5da070 0x5645af996d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.321+0000 7f4b29353740 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b9390 0x5645af98b000 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.321+0000 7f4b29353740 1 --2- 172.21.15.16:0/630858441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b11b0 0x5645af98aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.321+0000 7f4b29353740 1 -- 172.21.15.16:0/630858441 >> 172.21.15.16:0/630858441 conn(0x5645af5ce460 msgr2=0x5645af9824a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.321+0000 7f4afbfff640 0 client.16119 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.321+0000 7f4b29353740 1 -- 172.21.15.16:0/630858441 shutdown_connections 2024-08-28T19:22:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.321+0000 7f4b29353740 1 -- 172.21.15.16:0/630858441 wait complete. 2024-08-28T19:22:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.321+0000 7f4b29353740 20 client.16119 trim_cache size 0 max 16384 2024-08-28T19:22:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.339+0000 7f4b29353740 1 Processor -- start 2024-08-28T19:22:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.339+0000 7f4b29353740 1 -- start start 2024-08-28T19:22:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b29353740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b11b0 0x5645af982ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b29353740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b9390 0x5645af996c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b29353740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5da070 0x5645af9971d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b29353740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c1aeac0 con 0x5645af5b11b0 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b29353740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c1b0600 con 0x5645af5da070 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b29353740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c1af190 con 0x5645af5b9390 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b20b7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b11b0 0x5645af982ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b20b7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b11b0 0x5645af982ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45980/0 (socket says 172.21.15.16:45980) 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b20b7a640 1 -- 172.21.15.16:0/80808587 learned_addr learned my addr 172.21.15.16:0/80808587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b2137b640 1 --2- 172.21.15.16:0/80808587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5da070 0x5645af9971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/80808587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b9390 0x5645af996c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b20b7a640 1 -- 172.21.15.16:0/80808587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b9390 msgr2=0x5645af996c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/80808587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b9390 0x5645af996c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b20b7a640 1 -- 172.21.15.16:0/80808587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5da070 msgr2=0x5645af9971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/80808587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5da070 0x5645af9971d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b20b7a640 1 -- 172.21.15.16:0/80808587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645af97d6f0 con 0x5645af5b11b0 2024-08-28T19:22:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.340+0000 7f4b2137b640 1 --2- 172.21.15.16:0/80808587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5da070 0x5645af9971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.341+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/80808587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b11b0 0x5645af982ad0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009ca0 tx=0x7f4b0c047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efa64e7f6053aac server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.341+0000 7f4b127fc640 1 -- 172.21.15.16:0/80808587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c003b70 con 0x5645af5b11b0 2024-08-28T19:22:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.341+0000 7f4b127fc640 1 -- 172.21.15.16:0/80808587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b0c036040 con 0x5645af5b11b0 2024-08-28T19:22:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.341+0000 7f4b127fc640 1 -- 172.21.15.16:0/80808587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c0043c0 con 0x5645af5b11b0 2024-08-28T19:22:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.342+0000 7f4b29353740 1 -- 172.21.15.16:0/80808587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b11b0 msgr2=0x5645af982ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.342+0000 7f4b29353740 1 --2- 172.21.15.16:0/80808587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b11b0 0x5645af982ad0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009ca0 tx=0x7f4b0c047000 comp rx=0 tx=0).stop 2024-08-28T19:22:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.342+0000 7f4b29353740 1 -- 172.21.15.16:0/80808587 shutdown_connections 2024-08-28T19:22:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.342+0000 7f4b29353740 1 --2- 172.21.15.16:0/80808587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5da070 0x5645af9971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.342+0000 7f4b29353740 1 --2- 172.21.15.16:0/80808587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b9390 0x5645af996c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.342+0000 7f4b29353740 1 --2- 172.21.15.16:0/80808587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5b11b0 0x5645af982ad0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.342+0000 7f4b29353740 1 -- 172.21.15.16:0/80808587 >> 172.21.15.16:0/80808587 conn(0x5645af5ce460 msgr2=0x5645af9823c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.342+0000 7f4b29353740 1 -- 172.21.15.16:0/80808587 shutdown_connections 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.342+0000 7f4b29353740 1 -- 172.21.15.16:0/80808587 wait complete. 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.342+0000 7f4b29353740 1 -- start start 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b29353740 1 Processor -- start 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b29353740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 0x5645af9931e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b29353740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 0x5645af98f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b29353740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 0x5645af98fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b29353740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c1aff30 con 0x5645af5da070 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b29353740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c1af860 con 0x5645af5b9390 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b29353740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c1acaa0 con 0x5645af5b11b0 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b29353740 10 client.0 _ll_register_callbacks cb 0x7f4b040a7e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b20b7a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 0x5645af9931e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b20b7a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 0x5645af9931e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:37774/0 (socket says 172.21.15.16:37774) 2024-08-28T19:22:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b20b7a640 1 -- 172.21.15.16:0/1322447230 learned_addr learned my addr 172.21.15.16:0/1322447230 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 0x5645af98f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.343+0000 7f4b20b7a640 1 -- 172.21.15.16:0/1322447230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 msgr2=0x5645af98f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 0x5645af98f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4b20b7a640 1 -- 172.21.15.16:0/1322447230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 msgr2=0x5645af98fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4b2137b640 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 0x5645af98fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 0x5645af98fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4b20b7a640 1 -- 172.21.15.16:0/1322447230 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645af990120 con 0x5645af5b11b0 2024-08-28T19:22:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 0x5645af98f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4b2137b640 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 0x5645af98fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:22:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4b20b7a640 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 0x5645af9931e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c02a900 tx=0x7f4b0c031d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87281d8218b463c2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4afbfff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:22:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4afbfff640 1 -- 172.21.15.16:0/1322447230 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c004b40 con 0x5645af5b11b0 2024-08-28T19:22:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4afbfff640 1 -- 172.21.15.16:0/1322447230 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f4b0c004ce0 con 0x5645af5b11b0 2024-08-28T19:22:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.344+0000 7f4b29353740 1 -- 172.21.15.16:0/1322447230 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b0c036040 con 0x5645af5b11b0 2024-08-28T19:22:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4afbfff640 1 -- 172.21.15.16:0/1322447230 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4b0c06d050 con 0x5645af5b11b0 2024-08-28T19:22:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4b29353740 10 client.25648 fetch_fsmap learned FSMap version 29 2024-08-28T19:22:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4b29353740 1 -- 172.21.15.16:0/1322447230 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5645afa08430 con 0x5645af5b11b0 2024-08-28T19:22:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4afbfff640 1 -- 172.21.15.16:0/1322447230 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b0c036040 con 0x5645af5b11b0 2024-08-28T19:22:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4afbfff640 1 -- 172.21.15.16:0/1322447230 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4b0c004460 con 0x5645af5b11b0 2024-08-28T19:22:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4b29353740 10 client.25648 fetch_fsmap finished waiting for FSMap version 29 2024-08-28T19:22:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4b29353740 10 client.25648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4b29353740 20 client.25648 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4b29353740 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5645af56dda0 0x5645afa0ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4b29353740 4 client.25648 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:22:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4b29353740 1 -- 172.21.15.16:0/1322447230 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5645af8f4020 con 0x5645af56dda0 2024-08-28T19:22:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.345+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5645af56dda0 0x5645afa0ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.346+0000 7f4b1bfff640 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5645af56dda0 0x5645afa0ac10 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ddcdfce4860d47b server_cookie=aaf70a59d5b7c893 in_seq=0 out_seq=0 2024-08-28T19:22:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.346+0000 7f4afbfff640 10 client.25648 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.346+0000 7f4afbfff640 1 -- 172.21.15.16:0/1322447230 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+942 (crc 0 0 0) 0x5645af8f4020 con 0x5645af56dda0 2024-08-28T19:22:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.346+0000 7f4afbfff640 10 client.25648 handle_command_reply: tid=0 2024-08-28T19:22:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.346+0000 7f4b29353740 1 client.25648 shutdown 2024-08-28T19:22:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4b29353740 1 -- 172.21.15.16:0/1322447230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 msgr2=0x5645af9931e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4b29353740 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 0x5645af9931e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c02a900 tx=0x7f4b0c031d90 comp rx=0 tx=0).stop 2024-08-28T19:22:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4b29353740 1 -- 172.21.15.16:0/1322447230 shutdown_connections 2024-08-28T19:22:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4b29353740 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645af5da070 0x5645af98fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4b29353740 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645af5b9390 0x5645af98f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4b29353740 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5645af56dda0 0x5645afa0ac10 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4b29353740 1 --2- 172.21.15.16:0/1322447230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5645af5b11b0 0x5645af9931e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4afbfff640 0 client.25648 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4b29353740 1 -- 172.21.15.16:0/1322447230 >> 172.21.15.16:0/1322447230 conn(0x5645af5ce460 msgr2=0x5645af980250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4b29353740 1 -- 172.21.15.16:0/1322447230 shutdown_connections 2024-08-28T19:22:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4b29353740 1 -- 172.21.15.16:0/1322447230 wait complete. 2024-08-28T19:22:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.347+0000 7f4b29353740 20 client.25648 trim_cache size 0 max 16384 2024-08-28T19:22:43.354 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:22:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:22:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:22:43.356 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:22:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 -- 172.21.15.16:0/97718910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b080b4cd0 msgr2=0x7f4b080b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b080b4cd0 0x7f4b080b71e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c005e30 tx=0x7f4b0c009360 comp rx=0 tx=0).stop 2024-08-28T19:22:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 -- 172.21.15.16:0/97718910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c108170 msgr2=0x7f4b1c19b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c108170 0x7f4b1c19b5e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400e9f0 tx=0x7f4b0400eeb0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 -- 172.21.15.16:0/97718910 shutdown_connections 2024-08-28T19:22:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b1c108ac0 0x7f4b1c19bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c108170 0x7f4b1c19b5e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b080b4cd0 0x7f4b080b71e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 --2- 172.21.15.16:0/97718910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c102830 0x7f4b1c19b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 -- 172.21.15.16:0/97718910 >> 172.21.15.16:0/97718910 conn(0x7f4b1c0794d0 msgr2=0x7f4b1c106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 -- 172.21.15.16:0/97718910 shutdown_connections 2024-08-28T19:22:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:43.348+0000 7f4b22dde640 1 -- 172.21.15.16:0/97718910 wait complete. 2024-08-28T19:22:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:43 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:22:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:43 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:43 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:22:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:43 smithi028 ceph-mon[21769]: pgmap v389: 129 pgs: 129 active+clean; 28 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.7 MiB/s wr, 217 op/s 2024-08-28T19:22:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:43 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:43.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:43 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:22:43.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:43 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:43.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:43 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:22:43.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:43 smithi115 ceph-mon[22184]: pgmap v389: 129 pgs: 129 active+clean; 28 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.7 MiB/s wr, 217 op/s 2024-08-28T19:22:43.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:43 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:43.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 68295 16.74 MB/sec warmup 23 sec latency 20.765 ms 2024-08-28T19:22:44.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:44 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:22:44.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:44 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:22:44.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:44 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:22:44.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71035 16.75 MB/sec warmup 24 sec latency 194.931 ms 2024-08-28T19:22:45.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:45 smithi016 ceph-mon[18556]: pgmap v390: 129 pgs: 129 active+clean; 29 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-08-28T19:22:45.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:45 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:22:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:45 smithi028 ceph-mon[21769]: pgmap v390: 129 pgs: 129 active+clean; 29 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-08-28T19:22:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:45 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:22:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:45 smithi115 ceph-mon[22184]: pgmap v390: 129 pgs: 129 active+clean; 29 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-08-28T19:22:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:45 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:22:45.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71035 16.08 MB/sec warmup 25 sec latency 1195.033 ms 2024-08-28T19:22:46.864 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:22:46.864 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:22:46.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71035 15.46 MB/sec warmup 26 sec latency 2195.080 ms 2024-08-28T19:22:47.120 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:47.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:47 smithi016 ceph-mon[18556]: pgmap v391: 129 pgs: 129 active+clean; 38 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 248 op/s 2024-08-28T19:22:47.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:47 smithi028 ceph-mon[21769]: pgmap v391: 129 pgs: 129 active+clean; 38 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 248 op/s 2024-08-28T19:22:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:47 smithi115 ceph-mon[22184]: pgmap v391: 129 pgs: 129 active+clean; 38 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 248 op/s 2024-08-28T19:22:47.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.609+0000 7f1b6341e640 1 -- 172.21.15.16:0/1284411947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b5c1028b0 msgr2=0x7f1b5c102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:47.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.609+0000 7f1b6341e640 1 --2- 172.21.15.16:0/1284411947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b5c1028b0 0x7f1b5c102cc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1b58009900 tx=0x7f1b5802a410 comp rx=0 tx=0).stop 2024-08-28T19:22:47.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.610+0000 7f1b6341e640 1 -- 172.21.15.16:0/1284411947 shutdown_connections 2024-08-28T19:22:47.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.610+0000 7f1b6341e640 1 --2- 172.21.15.16:0/1284411947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b5c108b40 0x7f1b5c10f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:47.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.610+0000 7f1b6341e640 1 --2- 172.21.15.16:0/1284411947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b5c1081f0 0x7f1b5c108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:47.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.610+0000 7f1b6341e640 1 --2- 172.21.15.16:0/1284411947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b5c1028b0 0x7f1b5c102cc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:47.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.610+0000 7f1b6341e640 1 -- 172.21.15.16:0/1284411947 >> 172.21.15.16:0/1284411947 conn(0x7f1b5c079470 msgr2=0x7f1b5c079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.610+0000 7f1b6341e640 1 -- 172.21.15.16:0/1284411947 shutdown_connections 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.610+0000 7f1b6341e640 1 -- 172.21.15.16:0/1284411947 wait complete. 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.610+0000 7f1b6341e640 1 Processor -- start 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b6341e640 1 -- start start 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b6341e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b5c1028b0 0x7f1b5c199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b6341e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b5c1081f0 0x7f1b5c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b6341e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b5c108b40 0x7f1b5c19ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b6341e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c112800 con 0x7f1b5c1081f0 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b6341e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c112680 con 0x7f1b5c1028b0 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b6341e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c112980 con 0x7f1b5c108b40 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b609b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b5c1081f0 0x7f1b5c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b609b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b5c1081f0 0x7f1b5c1997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41704/0 (socket says 172.21.15.16:41704) 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b609b9640 1 -- 172.21.15.16:0/1621023071 learned_addr learned my addr 172.21.15.16:0/1621023071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b611ba640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b5c1028b0 0x7f1b5c199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b609b9640 1 -- 172.21.15.16:0/1621023071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b5c108b40 msgr2=0x7f1b5c19ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b609b9640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b5c108b40 0x7f1b5c19ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b609b9640 1 -- 172.21.15.16:0/1621023071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b5c1028b0 msgr2=0x7f1b5c199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b609b9640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b5c1028b0 0x7f1b5c199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.611+0000 7f1b609b9640 1 -- 172.21.15.16:0/1621023071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5c19f1c0 con 0x7f1b5c1081f0 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.612+0000 7f1b609b9640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b5c1081f0 0x7f1b5c1997f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c00e9c0 tx=0x7f1b4c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.612+0000 7f1b527fc640 1 -- 172.21.15.16:0/1621023071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c00ccc0 con 0x7f1b5c1081f0 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.612+0000 7f1b527fc640 1 -- 172.21.15.16:0/1621023071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b4c0054d0 con 0x7f1b5c1081f0 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.612+0000 7f1b6341e640 1 -- 172.21.15.16:0/1621023071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5c19f4b0 con 0x7f1b5c1081f0 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.612+0000 7f1b527fc640 1 -- 172.21.15.16:0/1621023071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c010430 con 0x7f1b5c1081f0 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.612+0000 7f1b6341e640 1 -- 172.21.15.16:0/1621023071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b5c1a5c10 con 0x7f1b5c1081f0 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.613+0000 7f1b6341e640 1 -- 172.21.15.16:0/1621023071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b5c102cc0 con 0x7f1b5c1081f0 2024-08-28T19:22:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.614+0000 7f1b527fc640 1 -- 172.21.15.16:0/1621023071 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1b4c005000 con 0x7f1b5c1081f0 2024-08-28T19:22:47.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.615+0000 7f1b527fc640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1b480b4ca0 0x7f1b480b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:47.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.615+0000 7f1b611ba640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1b480b4ca0 0x7f1b480b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:47.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.615+0000 7f1b527fc640 1 -- 172.21.15.16:0/1621023071 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1b4c0e5ea0 con 0x7f1b5c1081f0 2024-08-28T19:22:47.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.616+0000 7f1b611ba640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1b480b4ca0 0x7f1b480b71b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1b58031440 tx=0x7f1b58009560 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:47.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.618+0000 7f1b527fc640 1 -- 172.21.15.16:0/1621023071 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f1b4c056130 con 0x7f1b5c1081f0 2024-08-28T19:22:47.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.932+0000 7f1b6341e640 1 -- 172.21.15.16:0/1621023071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b5c19a380 con 0x7f1b5c1081f0 2024-08-28T19:22:47.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71035 14.89 MB/sec warmup 27 sec latency 3195.111 ms 2024-08-28T19:22:47.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.933+0000 7f1b527fc640 1 -- 172.21.15.16:0/1621023071 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11978 (secure 0 0 0) 0x7f1b5c19a380 con 0x7f1b5c1081f0 2024-08-28T19:22:47.935 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:22:47.935 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"btime":"2024-08-28T19:22:44:127609+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:22:43.270302+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:22:47.935 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2024-08-28T19:22:47.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.937+0000 7f1b6341e640 1 -- 172.21.15.16:0/1621023071 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1b480b4ca0 msgr2=0x7f1b480b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:47.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.937+0000 7f1b6341e640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1b480b4ca0 0x7f1b480b71b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1b58031440 tx=0x7f1b58009560 comp rx=0 tx=0).stop 2024-08-28T19:22:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.937+0000 7f1b6341e640 1 -- 172.21.15.16:0/1621023071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b5c1081f0 msgr2=0x7f1b5c1997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.937+0000 7f1b6341e640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b5c1081f0 0x7f1b5c1997f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c00e9c0 tx=0x7f1b4c00ee80 comp rx=0 tx=0).stop 2024-08-28T19:22:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.937+0000 7f1b6341e640 1 -- 172.21.15.16:0/1621023071 shutdown_connections 2024-08-28T19:22:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.937+0000 7f1b6341e640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b5c108b40 0x7f1b5c19ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.937+0000 7f1b6341e640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b5c1081f0 0x7f1b5c1997f0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.937+0000 7f1b6341e640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1b480b4ca0 0x7f1b480b71b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.937+0000 7f1b6341e640 1 --2- 172.21.15.16:0/1621023071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b5c1028b0 0x7f1b5c199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.937+0000 7f1b6341e640 1 -- 172.21.15.16:0/1621023071 >> 172.21.15.16:0/1621023071 conn(0x7f1b5c079470 msgr2=0x7f1b5c106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.938+0000 7f1b6341e640 1 -- 172.21.15.16:0/1621023071 shutdown_connections 2024-08-28T19:22:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:47.938+0000 7f1b6341e640 1 -- 172.21.15.16:0/1621023071 wait complete. 2024-08-28T19:22:48.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:48 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1621023071' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:22:48.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:48 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1621023071' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:22:48.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:48 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1621023071' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:22:48.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:22:48.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71035 14.35 MB/sec warmup 28 sec latency 4195.142 ms 2024-08-28T19:22:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:49.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:49 smithi016 ceph-mon[18556]: pgmap v392: 129 pgs: 129 active+clean; 44 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 2.5 MiB/s wr, 229 op/s 2024-08-28T19:22:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.423+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/568891998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c8109b90 msgr2=0x7fb1c81103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.423+0000 7fb1cd0a6640 1 --2- 172.21.15.16:0/568891998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c8109b90 0x7fb1c81103c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0009ca0 tx=0x7fb1c002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.423+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/568891998 shutdown_connections 2024-08-28T19:22:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.423+0000 7fb1cd0a6640 1 --2- 172.21.15.16:0/568891998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c8109b90 0x7fb1c81103c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.423+0000 7fb1cd0a6640 1 --2- 172.21.15.16:0/568891998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c8109240 0x7fb1c8109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.423+0000 7fb1cd0a6640 1 --2- 172.21.15.16:0/568891998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1c8103900 0x7fb1c8103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.423+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/568891998 >> 172.21.15.16:0/568891998 conn(0x7fb1c807b5f0 msgr2=0x7fb1c807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.424+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/568891998 shutdown_connections 2024-08-28T19:22:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.424+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/568891998 wait complete. 2024-08-28T19:22:49.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.424+0000 7fb1cd0a6640 1 Processor -- start 2024-08-28T19:22:49.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.424+0000 7fb1cd0a6640 1 -- start start 2024-08-28T19:22:49.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1cd0a6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1c8103900 0x7fb1c80754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1cd0a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c8109240 0x7fb1c80759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1cd0a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c8109b90 0x7fb1c8075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1cd0a6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c8113850 con 0x7fb1c8109240 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1cd0a6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81136d0 con 0x7fb1c8109b90 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1cd0a6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81139d0 con 0x7fb1c8103900 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c8109240 0x7fb1c80759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c8109240 0x7fb1c80759f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41718/0 (socket says 172.21.15.16:41718) 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1bffff640 1 -- 172.21.15.16:0/3935643739 learned_addr learned my addr 172.21.15.16:0/3935643739 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1c8103900 0x7fb1c80754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1c759e640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c8109b90 0x7fb1c8075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1bffff640 1 -- 172.21.15.16:0/3935643739 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1c8103900 msgr2=0x7fb1c80754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1bffff640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1c8103900 0x7fb1c80754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1bffff640 1 -- 172.21.15.16:0/3935643739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c8109b90 msgr2=0x7fb1c8075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1bffff640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c8109b90 0x7fb1c8075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1bffff640 1 -- 172.21.15.16:0/3935643739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c81a8540 con 0x7fb1c8109240 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.425+0000 7fb1bffff640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c8109240 0x7fb1c80759f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb1b000e9f0 tx=0x7fb1b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:49.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.426+0000 7fb1c4d99640 1 -- 172.21.15.16:0/3935643739 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b000ccc0 con 0x7fb1c8109240 2024-08-28T19:22:49.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.426+0000 7fb1c4d99640 1 -- 172.21.15.16:0/3935643739 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b00108e0 con 0x7fb1c8109240 2024-08-28T19:22:49.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.426+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/3935643739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c81a8830 con 0x7fb1c8109240 2024-08-28T19:22:49.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.426+0000 7fb1c4d99640 1 -- 172.21.15.16:0/3935643739 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0018860 con 0x7fb1c8109240 2024-08-28T19:22:49.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.426+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/3935643739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c81a8d70 con 0x7fb1c8109240 2024-08-28T19:22:49.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.428+0000 7fb1d361b740 1 Processor -- start 2024-08-28T19:22:49.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.428+0000 7fb1d361b740 1 -- start start 2024-08-28T19:22:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a72d070 0x55ae1aad2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aad2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a70c390 0x55ae1aad2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1d361b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ae1aa49680 con 0x55ae1a7041b0 2024-08-28T19:22:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1d361b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae1aa38c10 con 0x55ae1a72d070 2024-08-28T19:22:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1d361b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ae1aa3cbc0 con 0x55ae1a70c390 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1c6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aad2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1c6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aad2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41728/0 (socket says 172.21.15.16:41728) 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/4201735998 learned_addr learned my addr 172.21.15.16:0/4201735998 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1c4d99640 1 -- 172.21.15.16:0/3935643739 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fb1b0010430 con 0x7fb1c8109240 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1c759e640 1 --2- 172.21.15.16:0/4201735998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a72d070 0x55ae1aad2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/4201735998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a70c390 msgr2=0x55ae1aad2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/4201735998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a70c390 0x55ae1aad2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/4201735998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a72d070 msgr2=0x55ae1aad2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/4201735998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a72d070 0x55ae1aad2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/4201735998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae1a798650 con 0x55ae1a7041b0 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.429+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/4201735998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aad2850 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8004650 tx=0x7fb1b800bba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e01c154d2a367a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.430+0000 7fb1bd7fa640 1 -- 172.21.15.16:0/4201735998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8003bd0 con 0x55ae1a7041b0 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.430+0000 7fb1bd7fa640 1 -- 172.21.15.16:0/4201735998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b8015070 con 0x55ae1a7041b0 2024-08-28T19:22:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.430+0000 7fb1bd7fa640 1 -- 172.21.15.16:0/4201735998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8004d20 con 0x55ae1a7041b0 2024-08-28T19:22:49.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.430+0000 7fb1c4d99640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1b40b4ca0 0x7fb1b40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.430+0000 7fb1c759e640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1b40b4ca0 0x7fb1b40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.430+0000 7fb1c4d99640 1 -- 172.21.15.16:0/3935643739 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb1b00e5b20 con 0x7fb1c8109240 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1d361b740 1 -- 172.21.15.16:0/4201735998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 msgr2=0x55ae1aad2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1d361b740 1 --2- 172.21.15.16:0/4201735998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aad2850 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8004650 tx=0x7fb1b800bba0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1c759e640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1b40b4ca0 0x7fb1b40b71b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb1c002f500 tx=0x7fb1c002f310 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1d361b740 1 -- 172.21.15.16:0/4201735998 shutdown_connections 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1d361b740 1 --2- 172.21.15.16:0/4201735998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a70c390 0x55ae1aad2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1d361b740 1 --2- 172.21.15.16:0/4201735998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aad2850 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1d361b740 1 --2- 172.21.15.16:0/4201735998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a72d070 0x55ae1aad2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1d361b740 1 -- 172.21.15.16:0/4201735998 >> 172.21.15.16:0/4201735998 conn(0x55ae1a721460 msgr2=0x55ae1aad0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1d361b740 1 -- 172.21.15.16:0/4201735998 shutdown_connections 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1d361b740 1 -- 172.21.15.16:0/4201735998 wait complete. 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1d361b740 1 -- start start 2024-08-28T19:22:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.431+0000 7fb1d361b740 1 Processor -- start 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaddae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aade020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a72d070 0x55ae1aae9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1d361b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ae1a89b700 con 0x55ae1a7041b0 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1d361b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae1a9bf750 con 0x55ae1a70c390 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1d361b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ae1aae0af0 con 0x55ae1a72d070 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1d361b740 10 client.0 _ll_register_callbacks cb 0x55ae1a9eee10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1c6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaddae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1c6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaddae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41738/0 (socket says 172.21.15.16:41738) 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/1562478790 learned_addr learned my addr 172.21.15.16:0/1562478790 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1c759e640 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a72d070 0x55ae1aae9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1bffff640 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aade020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/1562478790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a72d070 msgr2=0x55ae1aae9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a72d070 0x55ae1aae9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/1562478790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 msgr2=0x55ae1aade020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aade020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/1562478790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae1a9f8900 con 0x55ae1a7041b0 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaddae0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8005880 tx=0x7fb1b8005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a839bbe62bfda31 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.432+0000 7fb1ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/1562478790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8028060 con 0x55ae1a7041b0 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/1562478790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb1b8011640 con 0x55ae1a7041b0 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1d361b740 1 -- 172.21.15.16:0/1562478790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b8015070 con 0x55ae1a7041b0 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/1562478790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb1b8003ea0 con 0x55ae1a7041b0 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1d361b740 10 client.16140 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1d361b740 1 -- 172.21.15.16:0/1562478790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ae1aaec8f0 con 0x55ae1a7041b0 2024-08-28T19:22:49.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/1562478790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b8015070 con 0x55ae1a7041b0 2024-08-28T19:22:49.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/1562478790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fb1b8022070 con 0x55ae1a7041b0 2024-08-28T19:22:49.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1d361b740 10 client.16140 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:49.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1d361b740 10 client.16140 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:22:49.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1d361b740 20 client.16140 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:49.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1d361b740 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55ae1a6eb9f0 0x55ae1aaefd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1d361b740 4 client.16140 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:49.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.433+0000 7fb1d361b740 1 -- 172.21.15.16:0/1562478790 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae1aa4b5d0 con 0x55ae1a6eb9f0 2024-08-28T19:22:49.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.434+0000 7fb1bffff640 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55ae1a6eb9f0 0x55ae1aaefd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.434+0000 7fb1bffff640 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55ae1a6eb9f0 0x55ae1aaefd00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8140f0ab17ef628e server_cookie=c3a419450660f61d in_seq=0 out_seq=0 2024-08-28T19:22:49.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.434+0000 7fb1ae7fc640 10 client.16140 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:22:49.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/1562478790 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ae1aa4b5d0 con 0x55ae1a6eb9f0 2024-08-28T19:22:49.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1ae7fc640 10 client.16140 handle_command_reply: tid=0 2024-08-28T19:22:49.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 1 client.16140 shutdown 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 1 -- 172.21.15.16:0/1562478790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 msgr2=0x55ae1aaddae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaddae0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8005880 tx=0x7fb1b8005750 comp rx=0 tx=0).stop 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 1 -- 172.21.15.16:0/1562478790 shutdown_connections 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55ae1a6eb9f0 0x55ae1aaefd00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a72d070 0x55ae1aae9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aade020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1ae7fc640 0 client.16140 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 1 --2- 172.21.15.16:0/1562478790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaddae0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 1 -- 172.21.15.16:0/1562478790 >> 172.21.15.16:0/1562478790 conn(0x55ae1a721460 msgr2=0x55ae1aad54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 1 -- 172.21.15.16:0/1562478790 shutdown_connections 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 1 -- 172.21.15.16:0/1562478790 wait complete. 2024-08-28T19:22:49.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.435+0000 7fb1d361b740 20 client.16140 trim_cache size 0 max 16384 2024-08-28T19:22:49.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1d361b740 1 Processor -- start 2024-08-28T19:22:49.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1d361b740 1 -- start start 2024-08-28T19:22:49.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a7041b0 0x55ae1aadacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aadb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a72d070 0x55ae1aaf2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1d361b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81b2e70 con 0x55ae1a72d070 2024-08-28T19:22:49.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1d361b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81b4a70 con 0x55ae1a70c390 2024-08-28T19:22:49.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1d361b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81b3540 con 0x55ae1a7041b0 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1c759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a72d070 0x55ae1aaf2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1c759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a72d070 0x55ae1aaf2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41744/0 (socket says 172.21.15.16:41744) 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1c759e640 1 -- 172.21.15.16:0/444047723 learned_addr learned my addr 172.21.15.16:0/444047723 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1bffff640 1 --2- 172.21.15.16:0/444047723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aadb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/444047723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a7041b0 0x55ae1aadacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1c759e640 1 -- 172.21.15.16:0/444047723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a7041b0 msgr2=0x55ae1aadacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1c759e640 1 --2- 172.21.15.16:0/444047723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a7041b0 0x55ae1aadacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.449+0000 7fb1c759e640 1 -- 172.21.15.16:0/444047723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 msgr2=0x55ae1aadb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.450+0000 7fb1c759e640 1 --2- 172.21.15.16:0/444047723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aadb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.450+0000 7fb1c759e640 1 -- 172.21.15.16:0/444047723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae1aad4600 con 0x55ae1a72d070 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.450+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/444047723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a7041b0 0x55ae1aadacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.450+0000 7fb1c759e640 1 --2- 172.21.15.16:0/444047723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a72d070 0x55ae1aaf2d30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0035000 tx=0x7fb1c0041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29eaa4c0b022cf70 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.450+0000 7fb1bcff9640 1 -- 172.21.15.16:0/444047723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0003970 con 0x55ae1a72d070 2024-08-28T19:22:49.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.450+0000 7fb1bcff9640 1 -- 172.21.15.16:0/444047723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1c0036050 con 0x55ae1a72d070 2024-08-28T19:22:49.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.450+0000 7fb1bcff9640 1 -- 172.21.15.16:0/444047723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c002f660 con 0x55ae1a72d070 2024-08-28T19:22:49.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.451+0000 7fb1d361b740 1 -- 172.21.15.16:0/444047723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a72d070 msgr2=0x55ae1aaf2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:49.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.451+0000 7fb1d361b740 1 --2- 172.21.15.16:0/444047723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a72d070 0x55ae1aaf2d30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0035000 tx=0x7fb1c0041000 comp rx=0 tx=0).stop 2024-08-28T19:22:49.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.451+0000 7fb1d361b740 1 -- 172.21.15.16:0/444047723 shutdown_connections 2024-08-28T19:22:49.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.451+0000 7fb1d361b740 1 --2- 172.21.15.16:0/444047723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a72d070 0x55ae1aaf2d30 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.451+0000 7fb1d361b740 1 --2- 172.21.15.16:0/444047723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aadb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.451+0000 7fb1d361b740 1 --2- 172.21.15.16:0/444047723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a7041b0 0x55ae1aadacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.451+0000 7fb1d361b740 1 -- 172.21.15.16:0/444047723 >> 172.21.15.16:0/444047723 conn(0x55ae1a721460 msgr2=0x55ae1aadc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.451+0000 7fb1d361b740 1 -- 172.21.15.16:0/444047723 shutdown_connections 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.451+0000 7fb1d361b740 1 -- 172.21.15.16:0/444047723 wait complete. 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.451+0000 7fb1d361b740 1 -- start start 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.451+0000 7fb1d361b740 1 Processor -- start 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaf3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aaf1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1d361b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a72d070 0x55ae1aaf1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1d361b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81b42e0 con 0x55ae1a7041b0 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1d361b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81b3c10 con 0x55ae1a70c390 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1d361b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c81b0e50 con 0x55ae1a72d070 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1d361b740 10 client.0 _ll_register_callbacks cb 0x7fb1c0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1c6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaf3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1c6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaf3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41750/0 (socket says 172.21.15.16:41750) 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/491583408 learned_addr learned my addr 172.21.15.16:0/491583408 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:49.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1c759e640 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a72d070 0x55ae1aaf1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1bffff640 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aaf1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/491583408 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a72d070 msgr2=0x55ae1aaf1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a72d070 0x55ae1aaf1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/491583408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 msgr2=0x55ae1aaf1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aaf1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1c6d9d640 1 -- 172.21.15.16:0/491583408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae1aaf24f0 con 0x55ae1a7041b0 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1c6d9d640 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaf3430 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800bbe0 tx=0x7fb1b8004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4655188c602a0b35 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/491583408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8038060 con 0x55ae1a7041b0 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/491583408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb1b8019620 con 0x55ae1a7041b0 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.452+0000 7fb1d361b740 1 -- 172.21.15.16:0/491583408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1c0036050 con 0x55ae1a7041b0 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/491583408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb1b803f050 con 0x55ae1a7041b0 2024-08-28T19:22:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1d361b740 10 client.16146 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1d361b740 1 -- 172.21.15.16:0/491583408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ae1ab5b2e0 con 0x55ae1a7041b0 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/491583408 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1c0036050 con 0x55ae1a7041b0 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/491583408 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fb1b8022040 con 0x55ae1a7041b0 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1d361b740 10 client.16146 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1d361b740 10 client.16146 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1d361b740 20 client.16146 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1d361b740 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55ae1a6c0da0 0x55ae1ab5da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1d361b740 4 client.16146 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1d361b740 1 -- 172.21.15.16:0/491583408 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ae1aa47070 con 0x55ae1a6c0da0 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.453+0000 7fb1bffff640 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55ae1a6c0da0 0x55ae1ab5da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1ae7fc640 10 client.16146 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1bffff640 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55ae1a6c0da0 0x55ae1ab5da70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d48730f89d26bbd2 server_cookie=ecde850a566a99b in_seq=0 out_seq=0 2024-08-28T19:22:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1ae7fc640 1 -- 172.21.15.16:0/491583408 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ae1aa47070 con 0x55ae1a6c0da0 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1ae7fc640 10 client.16146 handle_command_reply: tid=0 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 1 client.16146 shutdown 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 1 -- 172.21.15.16:0/491583408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 msgr2=0x55ae1aaf3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaf3430 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800bbe0 tx=0x7fb1b8004550 comp rx=0 tx=0).stop 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 1 -- 172.21.15.16:0/491583408 shutdown_connections 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55ae1a6c0da0 0x55ae1ab5da70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1ae7fc640 0 client.16146 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae1a72d070 0x55ae1aaf1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae1a70c390 0x55ae1aaf1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 1 --2- 172.21.15.16:0/491583408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae1a7041b0 0x55ae1aaf3430 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 1 -- 172.21.15.16:0/491583408 >> 172.21.15.16:0/491583408 conn(0x55ae1a721460 msgr2=0x55ae1aad3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 1 -- 172.21.15.16:0/491583408 shutdown_connections 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 1 -- 172.21.15.16:0/491583408 wait complete. 2024-08-28T19:22:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.454+0000 7fb1d361b740 20 client.16146 trim_cache size 0 max 16384 2024-08-28T19:22:49.461 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:22:49.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/3935643739 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1b40b4ca0 msgr2=0x7fb1b40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:49.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1b40b4ca0 0x7fb1b40b71b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb1c002f500 tx=0x7fb1c002f310 comp rx=0 tx=0).stop 2024-08-28T19:22:49.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/3935643739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c8109240 msgr2=0x7fb1c80759f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:49.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c8109240 0x7fb1c80759f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb1b000e9f0 tx=0x7fb1b000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/3935643739 shutdown_connections 2024-08-28T19:22:49.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1b40b4ca0 0x7fb1b40b71b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c8109b90 0x7fb1c8075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c8109240 0x7fb1c80759f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 --2- 172.21.15.16:0/3935643739 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1c8103900 0x7fb1c80754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:49.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/3935643739 >> 172.21.15.16:0/3935643739 conn(0x7fb1c807b5f0 msgr2=0x7fb1c81051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:49.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/3935643739 shutdown_connections 2024-08-28T19:22:49.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:49.455+0000 7fb1cd0a6640 1 -- 172.21.15.16:0/3935643739 wait complete. 2024-08-28T19:22:49.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:49 smithi028 ceph-mon[21769]: pgmap v392: 129 pgs: 129 active+clean; 44 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 2.5 MiB/s wr, 229 op/s 2024-08-28T19:22:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:49 smithi115 ceph-mon[22184]: pgmap v392: 129 pgs: 129 active+clean; 44 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 2.5 MiB/s wr, 229 op/s 2024-08-28T19:22:49.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71102 13.87 MB/sec warmup 29 sec latency 4797.391 ms 2024-08-28T19:22:50.311 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:50 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:22:50.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:22:50.460 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:50 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:22:50.606 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:50 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:22:50.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71102 13.40 MB/sec warmup 30 sec latency 1339.265 ms 2024-08-28T19:22:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.065+0000 7fe87571b640 1 -- 172.21.15.16:0/3550307488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87010ff00 msgr2=0x7fe870112320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.065+0000 7fe87571b640 1 --2- 172.21.15.16:0/3550307488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87010ff00 0x7fe870112320 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe868009900 tx=0x7fe86802a410 comp rx=0 tx=0).stop 2024-08-28T19:22:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.065+0000 7fe87571b640 1 -- 172.21.15.16:0/3550307488 shutdown_connections 2024-08-28T19:22:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.065+0000 7fe87571b640 1 --2- 172.21.15.16:0/3550307488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87010ff00 0x7fe870112320 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.065+0000 7fe87571b640 1 --2- 172.21.15.16:0/3550307488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe870075e20 0x7fe87010f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.065+0000 7fe87571b640 1 --2- 172.21.15.16:0/3550307488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8700754d0 0x7fe8700758e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.065+0000 7fe87571b640 1 -- 172.21.15.16:0/3550307488 >> 172.21.15.16:0/3550307488 conn(0x7fe8700ff3b0 msgr2=0x7fe870101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.065+0000 7fe87571b640 1 -- 172.21.15.16:0/3550307488 shutdown_connections 2024-08-28T19:22:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.065+0000 7fe87571b640 1 -- 172.21.15.16:0/3550307488 wait complete. 2024-08-28T19:22:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe87571b640 1 Processor -- start 2024-08-28T19:22:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe87571b640 1 -- start start 2024-08-28T19:22:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe87571b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8700754d0 0x7fe87019c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe87571b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe870075e20 0x7fe87019c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe87571b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe87010ff00 0x7fe8701a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe87571b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe870115a40 con 0x7fe8700754d0 2024-08-28T19:22:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe87571b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701158c0 con 0x7fe870075e20 2024-08-28T19:22:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe87571b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe870115bc0 con 0x7fe87010ff00 2024-08-28T19:22:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe86effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8700754d0 0x7fe87019c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe86effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8700754d0 0x7fe87019c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41764/0 (socket says 172.21.15.16:41764) 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe86effd640 1 -- 172.21.15.16:0/632251825 learned_addr learned my addr 172.21.15.16:0/632251825 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe87010ff00 0x7fe8701a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe86e7fc640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe870075e20 0x7fe87019c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe86effd640 1 -- 172.21.15.16:0/632251825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe87010ff00 msgr2=0x7fe8701a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.066+0000 7fe86effd640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe87010ff00 0x7fe8701a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.067+0000 7fe86effd640 1 -- 172.21.15.16:0/632251825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe870075e20 msgr2=0x7fe87019c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.067+0000 7fe86effd640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe870075e20 0x7fe87019c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.067+0000 7fe86effd640 1 -- 172.21.15.16:0/632251825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8701a22c0 con 0x7fe8700754d0 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.067+0000 7fe86e7fc640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe870075e20 0x7fe87019c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.067+0000 7fe86effd640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8700754d0 0x7fe87019c410 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe86000cce0 tx=0x7fe86000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.067+0000 7fe857fff640 1 -- 172.21.15.16:0/632251825 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86000f650 con 0x7fe8700754d0 2024-08-28T19:22:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.067+0000 7fe87571b640 1 -- 172.21.15.16:0/632251825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8701a25b0 con 0x7fe8700754d0 2024-08-28T19:22:51.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.067+0000 7fe87571b640 1 -- 172.21.15.16:0/632251825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8701a8dd0 con 0x7fe8700754d0 2024-08-28T19:22:51.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.067+0000 7fe857fff640 1 -- 172.21.15.16:0/632251825 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe86000fca0 con 0x7fe8700754d0 2024-08-28T19:22:51.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.067+0000 7fe857fff640 1 -- 172.21.15.16:0/632251825 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860018b40 con 0x7fe8700754d0 2024-08-28T19:22:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.069+0000 7fe87bc90740 1 Processor -- start 2024-08-28T19:22:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe87bc90740 1 -- start start 2024-08-28T19:22:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe857fff640 1 -- 172.21.15.16:0/632251825 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe86000f7f0 con 0x7fe8700754d0 2024-08-28T19:22:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bdc070 0x55b054f81310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bb31b0 0x55b054f81850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f81d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe87bc90740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b054ef8680 con 0x55b054bdc070 2024-08-28T19:22:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe87bc90740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b054ee7c10 con 0x55b054bbb390 2024-08-28T19:22:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe87bc90740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b054eebbc0 con 0x55b054bb31b0 2024-08-28T19:22:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe86f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bdc070 0x55b054f81310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe86f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bdc070 0x55b054f81310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41772/0 (socket says 172.21.15.16:41772) 2024-08-28T19:22:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe86f7fe640 1 -- 172.21.15.16:0/3823627880 learned_addr learned my addr 172.21.15.16:0/3823627880 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe86effd640 1 --2- 172.21.15.16:0/3823627880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bb31b0 0x55b054f81850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe86e7fc640 1 --2- 172.21.15.16:0/3823627880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f81d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe86f7fe640 1 -- 172.21.15.16:0/3823627880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bb31b0 msgr2=0x55b054f81850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/3823627880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bb31b0 0x55b054f81850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe86f7fe640 1 -- 172.21.15.16:0/3823627880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 msgr2=0x55b054f81d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/3823627880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f81d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.070+0000 7fe86f7fe640 1 -- 172.21.15.16:0/3823627880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b054c47650 con 0x55b054bdc070 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.071+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/3823627880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bdc070 0x55b054f81310 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe868009900 tx=0x7fe868032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7389f096ff564d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.071+0000 7fe857fff640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8580b4d50 0x7fe8580b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.071+0000 7fe86effd640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8580b4d50 0x7fe8580b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.071+0000 7fe8557fa640 1 -- 172.21.15.16:0/3823627880 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86803f070 con 0x55b054bdc070 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.071+0000 7fe8557fa640 1 -- 172.21.15.16:0/3823627880 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8680316a0 con 0x55b054bdc070 2024-08-28T19:22:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.071+0000 7fe857fff640 1 -- 172.21.15.16:0/632251825 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe8600e5df0 con 0x7fe8700754d0 2024-08-28T19:22:51.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.071+0000 7fe8557fa640 1 -- 172.21.15.16:0/3823627880 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868035070 con 0x55b054bdc070 2024-08-28T19:22:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.072+0000 7fe87bc90740 1 -- 172.21.15.16:0/3823627880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bdc070 msgr2=0x55b054f81310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.072+0000 7fe86effd640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8580b4d50 0x7fe8580b7260 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe8600e3280 tx=0x7fe860093040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.072+0000 7fe87bc90740 1 --2- 172.21.15.16:0/3823627880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bdc070 0x55b054f81310 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe868009900 tx=0x7fe868032dc0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.072+0000 7fe87bc90740 1 -- 172.21.15.16:0/3823627880 shutdown_connections 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.072+0000 7fe87bc90740 1 --2- 172.21.15.16:0/3823627880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f81d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.072+0000 7fe87bc90740 1 --2- 172.21.15.16:0/3823627880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bb31b0 0x55b054f81850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.072+0000 7fe87bc90740 1 --2- 172.21.15.16:0/3823627880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bdc070 0x55b054f81310 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.072+0000 7fe87bc90740 1 -- 172.21.15.16:0/3823627880 >> 172.21.15.16:0/3823627880 conn(0x55b054bd0460 msgr2=0x55b054f7f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.072+0000 7fe87bc90740 1 -- 172.21.15.16:0/3823627880 shutdown_connections 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.072+0000 7fe87bc90740 1 -- 172.21.15.16:0/3823627880 wait complete. 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.072+0000 7fe87bc90740 1 -- start start 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe87bc90740 1 Processor -- start 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bbb390 0x55b054f8cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bdc070 0x55b054f98dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe87bc90740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b054d4a700 con 0x55b054bb31b0 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe87bc90740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b054e6e750 con 0x55b054bdc070 2024-08-28T19:22:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe87bc90740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b054f8fae0 con 0x55b054bbb390 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe87bc90740 10 client.0 _ll_register_callbacks cb 0x55b054e9de10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe86f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe86f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41786/0 (socket says 172.21.15.16:41786) 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe86f7fe640 1 -- 172.21.15.16:0/1953465591 learned_addr learned my addr 172.21.15.16:0/1953465591 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe86e7fc640 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bbb390 0x55b054f8cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe86f7fe640 1 -- 172.21.15.16:0/1953465591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bbb390 msgr2=0x55b054f8cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bbb390 0x55b054f8cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe86f7fe640 1 -- 172.21.15.16:0/1953465591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bdc070 msgr2=0x55b054f98dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bdc070 0x55b054f98dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.073+0000 7fe86f7fe640 1 -- 172.21.15.16:0/1953465591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b054ea7900 con 0x55b054bb31b0 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8ca80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe86802a930 tx=0x7fe868032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5d45548b7461a6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe852ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe852ff5640 1 -- 172.21.15.16:0/1953465591 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86804a070 con 0x55b054bb31b0 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe852ff5640 1 -- 172.21.15.16:0/1953465591 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe868004460 con 0x55b054bb31b0 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe87bc90740 1 -- 172.21.15.16:0/1953465591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8680316a0 con 0x55b054bb31b0 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe852ff5640 1 -- 172.21.15.16:0/1953465591 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe868056050 con 0x55b054bb31b0 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe87bc90740 10 client.16158 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe87bc90740 1 -- 172.21.15.16:0/1953465591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b054f9b8b0 con 0x55b054bb31b0 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe852ff5640 1 -- 172.21.15.16:0/1953465591 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8680316a0 con 0x55b054bb31b0 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe852ff5640 1 -- 172.21.15.16:0/1953465591 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fe868004990 con 0x55b054bb31b0 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe87bc90740 10 client.16158 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe87bc90740 10 client.16158 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.074+0000 7fe87bc90740 20 client.16158 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.075+0000 7fe87bc90740 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b054b9a9f0 0x55b054f9ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.075+0000 7fe87bc90740 4 client.16158 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.075+0000 7fe87bc90740 1 -- 172.21.15.16:0/1953465591 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b054efa5d0 con 0x55b054b9a9f0 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.075+0000 7fe86e7fc640 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b054b9a9f0 0x55b054f9ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.075+0000 7fe852ff5640 10 client.16158 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.075+0000 7fe86e7fc640 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b054b9a9f0 0x55b054f9ec10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f588262d24e7ddff server_cookie=f1e52f58b37dc357 in_seq=0 out_seq=0 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.076+0000 7fe852ff5640 1 -- 172.21.15.16:0/1953465591 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b054efa5d0 con 0x55b054b9a9f0 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.076+0000 7fe852ff5640 10 client.16158 handle_command_reply: tid=0 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.076+0000 7fe87bc90740 1 client.16158 shutdown 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe87bc90740 1 -- 172.21.15.16:0/1953465591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 msgr2=0x55b054f8ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe87bc90740 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8ca80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe86802a930 tx=0x7fe868032b20 comp rx=0 tx=0).stop 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe87bc90740 1 -- 172.21.15.16:0/1953465591 shutdown_connections 2024-08-28T19:22:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe87bc90740 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b054b9a9f0 0x55b054f9ec10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe87bc90740 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bdc070 0x55b054f98dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe87bc90740 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bbb390 0x55b054f8cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe87bc90740 1 --2- 172.21.15.16:0/1953465591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8ca80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe87bc90740 1 -- 172.21.15.16:0/1953465591 >> 172.21.15.16:0/1953465591 conn(0x55b054bd0460 msgr2=0x55b054f82340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:51.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe852ff5640 0 client.16158 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:22:51.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe87bc90740 1 -- 172.21.15.16:0/1953465591 shutdown_connections 2024-08-28T19:22:51.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe87bc90740 1 -- 172.21.15.16:0/1953465591 wait complete. 2024-08-28T19:22:51.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.077+0000 7fe87bc90740 20 client.16158 trim_cache size 0 max 16384 2024-08-28T19:22:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.090+0000 7fe87bc90740 1 Processor -- start 2024-08-28T19:22:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.090+0000 7fe87bc90740 1 -- start start 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.090+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.090+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f821f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.090+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bdc070 0x55b054f82730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.090+0000 7fe87bc90740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701b2d00 con 0x55b054bb31b0 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.090+0000 7fe87bc90740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701b4900 con 0x55b054bbb390 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe87bc90740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701b33d0 con 0x55b054bdc070 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe86f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe86f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41802/0 (socket says 172.21.15.16:41802) 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe86f7fe640 1 -- 172.21.15.16:0/2185386817 learned_addr learned my addr 172.21.15.16:0/2185386817 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe86e7fc640 1 --2- 172.21.15.16:0/2185386817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f821f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/2185386817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bdc070 0x55b054f82730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe86f7fe640 1 -- 172.21.15.16:0/2185386817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bdc070 msgr2=0x55b054f82730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/2185386817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bdc070 0x55b054f82730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe86f7fe640 1 -- 172.21.15.16:0/2185386817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 msgr2=0x55b054f821f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/2185386817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f821f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe86f7fe640 1 -- 172.21.15.16:0/2185386817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b054f83600 con 0x55b054bb31b0 2024-08-28T19:22:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/2185386817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8d060 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe8680096c0 tx=0x7fe868005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf653b2420c0ba0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe854ff9640 1 -- 172.21.15.16:0/2185386817 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86804a020 con 0x55b054bb31b0 2024-08-28T19:22:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe854ff9640 1 -- 172.21.15.16:0/2185386817 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe868032dc0 con 0x55b054bb31b0 2024-08-28T19:22:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.091+0000 7fe854ff9640 1 -- 172.21.15.16:0/2185386817 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86803f030 con 0x55b054bb31b0 2024-08-28T19:22:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.092+0000 7fe87bc90740 1 -- 172.21.15.16:0/2185386817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 msgr2=0x55b054f8d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.092+0000 7fe87bc90740 1 --2- 172.21.15.16:0/2185386817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8d060 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe8680096c0 tx=0x7fe868005370 comp rx=0 tx=0).stop 2024-08-28T19:22:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.092+0000 7fe87bc90740 1 -- 172.21.15.16:0/2185386817 shutdown_connections 2024-08-28T19:22:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.092+0000 7fe87bc90740 1 --2- 172.21.15.16:0/2185386817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bdc070 0x55b054f82730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.092+0000 7fe87bc90740 1 --2- 172.21.15.16:0/2185386817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f821f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.092+0000 7fe87bc90740 1 --2- 172.21.15.16:0/2185386817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f8d060 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.092+0000 7fe87bc90740 1 -- 172.21.15.16:0/2185386817 >> 172.21.15.16:0/2185386817 conn(0x55b054bd0460 msgr2=0x55b054f8b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.092+0000 7fe87bc90740 1 -- 172.21.15.16:0/2185386817 shutdown_connections 2024-08-28T19:22:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.092+0000 7fe87bc90740 1 -- 172.21.15.16:0/2185386817 wait complete. 2024-08-28T19:22:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe87bc90740 1 -- start start 2024-08-28T19:22:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe87bc90740 1 Processor -- start 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f9f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f9fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe87bc90740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bdc070 0x55b055001350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe87bc90740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701b4170 con 0x55b054bb31b0 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe87bc90740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701b3aa0 con 0x55b054bbb390 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe87bc90740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701b0ce0 con 0x55b054bdc070 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe87bc90740 10 client.0 _ll_register_callbacks cb 0x7fe8600e9810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe86f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f9f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe86f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f9f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41804/0 (socket says 172.21.15.16:41804) 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe86f7fe640 1 -- 172.21.15.16:0/3582871545 learned_addr learned my addr 172.21.15.16:0/3582871545 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe86e7fc640 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f9fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.093+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bdc070 0x55b055001350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe86f7fe640 1 -- 172.21.15.16:0/3582871545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bdc070 msgr2=0x55b055001350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bdc070 0x55b055001350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe86f7fe640 1 -- 172.21.15.16:0/3582871545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 msgr2=0x55b054f9fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f9fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe86f7fe640 1 -- 172.21.15.16:0/3582871545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0550019c0 con 0x55b054bb31b0 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe86e7fc640 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f9fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe86f7fe640 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f9f5c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe868006010 tx=0x7fe8680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b32f820b1eaf52 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe852ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe852ff5640 1 -- 172.21.15.16:0/3582871545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86803f070 con 0x55b054bb31b0 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe852ff5640 1 -- 172.21.15.16:0/3582871545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe8680049a0 con 0x55b054bb31b0 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe87bc90740 1 -- 172.21.15.16:0/3582871545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe868032dc0 con 0x55b054bb31b0 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe852ff5640 1 -- 172.21.15.16:0/3582871545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe86804a070 con 0x55b054bb31b0 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe87bc90740 10 client.16164 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe87bc90740 1 -- 172.21.15.16:0/3582871545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b055008d90 con 0x55b054bb31b0 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.094+0000 7fe852ff5640 1 -- 172.21.15.16:0/3582871545 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe868032dc0 con 0x55b054bb31b0 2024-08-28T19:22:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.095+0000 7fe852ff5640 1 -- 172.21.15.16:0/3582871545 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fe868003ca0 con 0x55b054bb31b0 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.095+0000 7fe87bc90740 10 client.16164 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.095+0000 7fe87bc90740 10 client.16164 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.095+0000 7fe87bc90740 20 client.16164 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.095+0000 7fe87bc90740 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b054b6fda0 0x55b05500b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.095+0000 7fe87bc90740 4 client.16164 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.095+0000 7fe87bc90740 1 -- 172.21.15.16:0/3582871545 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b054ef6070 con 0x55b054b6fda0 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.095+0000 7fe86e7fc640 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b054b6fda0 0x55b05500b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.095+0000 7fe86e7fc640 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b054b6fda0 0x55b05500b250 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=708520b4a6398d56 server_cookie=b51bf56928ac7250 in_seq=0 out_seq=0 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.095+0000 7fe852ff5640 10 client.16164 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe852ff5640 1 -- 172.21.15.16:0/3582871545 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b054ef6070 con 0x55b054b6fda0 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe852ff5640 10 client.16164 handle_command_reply: tid=0 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 1 client.16164 shutdown 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 1 -- 172.21.15.16:0/3582871545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 msgr2=0x55b054f9f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f9f5c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe868006010 tx=0x7fe8680046c0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 1 -- 172.21.15.16:0/3582871545 shutdown_connections 2024-08-28T19:22:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b054b6fda0 0x55b05500b250 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b054bdc070 0x55b055001350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b054bbb390 0x55b054f9fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 1 --2- 172.21.15.16:0/3582871545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b054bb31b0 0x55b054f9f5c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe852ff5640 0 client.16164 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:22:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 1 -- 172.21.15.16:0/3582871545 >> 172.21.15.16:0/3582871545 conn(0x55b054bd0460 msgr2=0x55b054f8df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 1 -- 172.21.15.16:0/3582871545 shutdown_connections 2024-08-28T19:22:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 1 -- 172.21.15.16:0/3582871545 wait complete. 2024-08-28T19:22:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.096+0000 7fe87bc90740 20 client.16164 trim_cache size 0 max 16384 2024-08-28T19:22:51.102 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:22:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 -- 172.21.15.16:0/632251825 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8580b4d50 msgr2=0x7fe8580b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8580b4d50 0x7fe8580b7260 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe8600e3280 tx=0x7fe860093040 comp rx=0 tx=0).stop 2024-08-28T19:22:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 -- 172.21.15.16:0/632251825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8700754d0 msgr2=0x7fe87019c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8700754d0 0x7fe87019c410 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe86000cce0 tx=0x7fe86000a620 comp rx=0 tx=0).stop 2024-08-28T19:22:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 -- 172.21.15.16:0/632251825 shutdown_connections 2024-08-28T19:22:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe87010ff00 0x7fe8701a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8580b4d50 0x7fe8580b7260 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe870075e20 0x7fe87019c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 --2- 172.21.15.16:0/632251825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8700754d0 0x7fe87019c410 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 -- 172.21.15.16:0/632251825 >> 172.21.15.16:0/632251825 conn(0x7fe8700ff3b0 msgr2=0x7fe870100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 -- 172.21.15.16:0/632251825 shutdown_connections 2024-08-28T19:22:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:51.097+0000 7fe87571b640 1 -- 172.21.15.16:0/632251825 wait complete. 2024-08-28T19:22:51.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:51 smithi016 ceph-mon[18556]: pgmap v393: 129 pgs: 129 active+clean; 51 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 4.7 MiB/s wr, 237 op/s 2024-08-28T19:22:51.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:51 smithi028 ceph-mon[21769]: pgmap v393: 129 pgs: 129 active+clean; 51 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 4.7 MiB/s wr, 237 op/s 2024-08-28T19:22:51.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:51 smithi115 ceph-mon[22184]: pgmap v393: 129 pgs: 129 active+clean; 51 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 4.7 MiB/s wr, 237 op/s 2024-08-28T19:22:51.923 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:22:51.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71102 12.97 MB/sec warmup 31 sec latency 2339.308 ms 2024-08-28T19:22:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:52.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.677+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1335599036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbcc10f870 msgr2=0x7fdbcc10fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.677+0000 7fdbd0e2c640 1 --2- 172.21.15.16:0/1335599036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbcc10f870 0x7fdbcc10fc80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009900 tx=0x7fdbb402a410 comp rx=0 tx=0).stop 2024-08-28T19:22:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.677+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1335599036 shutdown_connections 2024-08-28T19:22:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.677+0000 7fdbd0e2c640 1 --2- 172.21.15.16:0/1335599036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbcc1101c0 0x7fdbcc1169f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.677+0000 7fdbd0e2c640 1 --2- 172.21.15.16:0/1335599036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbcc10f870 0x7fdbcc10fc80 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.677+0000 7fdbd0e2c640 1 --2- 172.21.15.16:0/1335599036 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbcc109f30 0x7fdbcc10a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.677+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1335599036 >> 172.21.15.16:0/1335599036 conn(0x7fdbcc077f10 msgr2=0x7fdbcc107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.677+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1335599036 shutdown_connections 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.677+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1335599036 wait complete. 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.678+0000 7fdbd0e2c640 1 Processor -- start 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.678+0000 7fdbd0e2c640 1 -- start start 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.678+0000 7fdbd0e2c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbcc109f30 0x7fdbcc1a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.678+0000 7fdbd0e2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbcc10f870 0x7fdbcc1a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.678+0000 7fdbd0e2c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbcc1101c0 0x7fdbcc1a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.678+0000 7fdbd0e2c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc119e80 con 0x7fdbcc10f870 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.678+0000 7fdbd0e2c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc119d00 con 0x7fdbcc1101c0 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.678+0000 7fdbd0e2c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc11a000 con 0x7fdbcc109f30 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.678+0000 7fdbc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbcc10f870 0x7fdbcc1a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.679+0000 7fdbca59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbcc109f30 0x7fdbcc1a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.679+0000 7fdbca59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbcc109f30 0x7fdbcc1a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:44020/0 (socket says 172.21.15.16:44020) 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.679+0000 7fdbca59c640 1 -- 172.21.15.16:0/1081199872 learned_addr learned my addr 172.21.15.16:0/1081199872 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.679+0000 7fdbca59c640 1 -- 172.21.15.16:0/1081199872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbcc1101c0 msgr2=0x7fdbcc1a6110 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.679+0000 7fdbca59c640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbcc1101c0 0x7fdbcc1a6110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.679+0000 7fdbca59c640 1 -- 172.21.15.16:0/1081199872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbcc10f870 msgr2=0x7fdbcc1a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.679+0000 7fdbca59c640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbcc10f870 0x7fdbcc1a0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.679+0000 7fdbca59c640 1 -- 172.21.15.16:0/1081199872 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbcc1a6840 con 0x7fdbcc109f30 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.680+0000 7fdbca59c640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbcc109f30 0x7fdbcc1a08d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00db80 tx=0x7fdbbc00b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.680+0000 7fdbc37fe640 1 -- 172.21.15.16:0/1081199872 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc017070 con 0x7fdbcc109f30 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.680+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1081199872 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbcc1a6b30 con 0x7fdbcc109f30 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.680+0000 7fdbc37fe640 1 -- 172.21.15.16:0/1081199872 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbbc003b70 con 0x7fdbcc109f30 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.680+0000 7fdbc37fe640 1 -- 172.21.15.16:0/1081199872 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc0106c0 con 0x7fdbcc109f30 2024-08-28T19:22:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.680+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1081199872 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbcc1ad290 con 0x7fdbcc109f30 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.683+0000 7fdbd73a1740 1 Processor -- start 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.683+0000 7fdbd73a1740 1 -- start start 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb345070 0x560edb6ea310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb31c1b0 0x560edb6ea850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb324390 0x560edb6ead90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560edb661680 con 0x560edb345070 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560edb650c10 con 0x560edb324390 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560edb654bc0 con 0x560edb31c1b0 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbcad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb345070 0x560edb6ea310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbcad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb345070 0x560edb6ea310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41834/0 (socket says 172.21.15.16:41834) 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbcad9d640 1 -- 172.21.15.16:0/2742966085 learned_addr learned my addr 172.21.15.16:0/2742966085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbc3fff640 1 --2- 172.21.15.16:0/2742966085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb324390 0x560edb6ead90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbcad9d640 1 -- 172.21.15.16:0/2742966085 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb31c1b0 msgr2=0x560edb6ea850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:22:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/2742966085 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb31c1b0 0x560edb6ea850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbcad9d640 1 -- 172.21.15.16:0/2742966085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb324390 msgr2=0x560edb6ead90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/2742966085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb324390 0x560edb6ead90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.684+0000 7fdbcad9d640 1 -- 172.21.15.16:0/2742966085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560edb3b0650 con 0x560edb345070 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.685+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/2742966085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb345070 0x560edb6ea310 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400b980 tx=0x7fdbc400be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67267c988318934 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.685+0000 7fdbc37fe640 1 -- 172.21.15.16:0/1081199872 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fdbbc004020 con 0x7fdbcc109f30 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.685+0000 7fdbc0ff9640 1 -- 172.21.15.16:0/2742966085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4016070 con 0x560edb345070 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.685+0000 7fdbc0ff9640 1 -- 172.21.15.16:0/2742966085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbc40054d0 con 0x560edb345070 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.685+0000 7fdbc0ff9640 1 -- 172.21.15.16:0/2742966085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4004d60 con 0x560edb345070 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.686+0000 7fdbc37fe640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbb80b4d20 0x7fdbb80b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.686+0000 7fdbca59c640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbb80b4d20 0x7fdbb80b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.686+0000 7fdbd73a1740 1 -- 172.21.15.16:0/2742966085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb345070 msgr2=0x560edb6ea310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.686+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/2742966085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb345070 0x560edb6ea310 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400b980 tx=0x7fdbc400be40 comp rx=0 tx=0).stop 2024-08-28T19:22:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.687+0000 7fdbd73a1740 1 -- 172.21.15.16:0/2742966085 shutdown_connections 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.687+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/2742966085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb324390 0x560edb6ead90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.687+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/2742966085 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb31c1b0 0x560edb6ea850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.687+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/2742966085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb345070 0x560edb6ea310 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.687+0000 7fdbd73a1740 1 -- 172.21.15.16:0/2742966085 >> 172.21.15.16:0/2742966085 conn(0x560edb339460 msgr2=0x560edb6e8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.687+0000 7fdbc37fe640 1 -- 172.21.15.16:0/1081199872 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdbbc0e5940 con 0x7fdbcc109f30 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.687+0000 7fdbd73a1740 1 -- 172.21.15.16:0/2742966085 shutdown_connections 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.687+0000 7fdbd73a1740 1 -- 172.21.15.16:0/2742966085 wait complete. 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.687+0000 7fdbd73a1740 1 -- start start 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.688+0000 7fdbca59c640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbb80b4d20 0x7fdbb80b7230 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x560edb6e8f90 tx=0x7fdbbc079100 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.688+0000 7fdbd73a1740 1 Processor -- start 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.689+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb31c1b0 0x560edb6f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.689+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb6f6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.689+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb345070 0x560edb701de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.689+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560edb4b3700 con 0x560edb324390 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.689+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560edb5d7750 con 0x560edb345070 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.689+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560edb6f8af0 con 0x560edb31c1b0 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.689+0000 7fdbd73a1740 10 client.0 _ll_register_callbacks cb 0x560edb606e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.689+0000 7fdbc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb6f6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.689+0000 7fdbc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb6f6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41840/0 (socket says 172.21.15.16:41840) 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.689+0000 7fdbc3fff640 1 -- 172.21.15.16:0/1917363520 learned_addr learned my addr 172.21.15.16:0/1917363520 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.689+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb345070 0x560edb701de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.690+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb31c1b0 0x560edb6f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.690+0000 7fdbc3fff640 1 -- 172.21.15.16:0/1917363520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb31c1b0 msgr2=0x560edb6f5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.690+0000 7fdbc3fff640 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb31c1b0 0x560edb6f5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.690+0000 7fdbc3fff640 1 -- 172.21.15.16:0/1917363520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb345070 msgr2=0x560edb701de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.690+0000 7fdbc3fff640 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb345070 0x560edb701de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.690+0000 7fdbc3fff640 1 -- 172.21.15.16:0/1917363520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560edb610900 con 0x560edb324390 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.690+0000 7fdbb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.690+0000 7fdbc3fff640 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb6f6110 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009900 tx=0x7fdbb4032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f4fa6088f07e17 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.691+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1917363520 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4035040 con 0x560edb324390 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.691+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1917363520 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdbb4031860 con 0x560edb324390 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.691+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1917363520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbc40054d0 con 0x560edb324390 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.691+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1917363520 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdbb404f050 con 0x560edb324390 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.691+0000 7fdbd73a1740 10 client.16176 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.691+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1917363520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560edb704990 con 0x560edb324390 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.691+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1917363520 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbc40054d0 con 0x560edb324390 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.692+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1917363520 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fdbb4004750 con 0x560edb324390 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.692+0000 7fdbd73a1740 10 client.16176 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.692+0000 7fdbd73a1740 10 client.16176 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.692+0000 7fdbd73a1740 20 client.16176 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.692+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x560edb3039f0 0x560edb707f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.692+0000 7fdbd73a1740 4 client.16176 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.692+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1917363520 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560edb6635d0 con 0x560edb3039f0 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.692+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x560edb3039f0 0x560edb707f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.693+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x560edb3039f0 0x560edb707f00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b5fc3090a7db1fc server_cookie=be34e2360b0e178d in_seq=0 out_seq=0 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.693+0000 7fdbb1ffb640 10 client.16176 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.694+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1917363520 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560edb6635d0 con 0x560edb3039f0 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.694+0000 7fdbb1ffb640 10 client.16176 handle_command_reply: tid=0 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.695+0000 7fdbd73a1740 1 client.16176 shutdown 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.695+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1917363520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 msgr2=0x560edb6f6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.695+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb6f6110 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009900 tx=0x7fdbb4032f60 comp rx=0 tx=0).stop 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.695+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1917363520 shutdown_connections 2024-08-28T19:22:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.695+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x560edb3039f0 0x560edb707f00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.695+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb345070 0x560edb701de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.695+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb6f6110 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.695+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1917363520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb31c1b0 0x560edb6f5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.695+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1917363520 >> 172.21.15.16:0/1917363520 conn(0x560edb339460 msgr2=0x560edb6ed4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:52.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.695+0000 7fdbb1ffb640 0 client.16176 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:22:52.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.695+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1917363520 shutdown_connections 2024-08-28T19:22:52.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.696+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1917363520 wait complete. 2024-08-28T19:22:52.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.696+0000 7fdbd73a1740 20 client.16176 trim_cache size 0 max 16384 2024-08-28T19:22:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.721+0000 7fdbd73a1740 1 Processor -- start 2024-08-28T19:22:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.721+0000 7fdbd73a1740 1 -- start start 2024-08-28T19:22:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.721+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 0x560edb6f7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb701ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 0x560edb702220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1b72c0 con 0x560edb324390 2024-08-28T19:22:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1b8ec0 con 0x560edb31c1b0 2024-08-28T19:22:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1b7990 con 0x560edb345070 2024-08-28T19:22:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbcad9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 0x560edb702220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbcad9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 0x560edb702220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:44036/0 (socket says 172.21.15.16:44036) 2024-08-28T19:22:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbcad9d640 1 -- 172.21.15.16:0/1682322363 learned_addr learned my addr 172.21.15.16:0/1682322363 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbc3fff640 1 --2- 172.21.15.16:0/1682322363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb701ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1682322363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 0x560edb6f7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbcad9d640 1 -- 172.21.15.16:0/1682322363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 msgr2=0x560edb6f7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1682322363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 0x560edb6f7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbcad9d640 1 -- 172.21.15.16:0/1682322363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 msgr2=0x560edb701ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.722+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1682322363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb701ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.723+0000 7fdbcad9d640 1 -- 172.21.15.16:0/1682322363 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560edb6ec600 con 0x560edb345070 2024-08-28T19:22:52.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.723+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1682322363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 0x560edb702220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4004a10 tx=0x7fdbc400be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38b9feff1340f032 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:52.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.723+0000 7fdbb3fff640 1 -- 172.21.15.16:0/1682322363 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4016040 con 0x560edb345070 2024-08-28T19:22:52.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.724+0000 7fdbb3fff640 1 -- 172.21.15.16:0/1682322363 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbc402a040 con 0x560edb345070 2024-08-28T19:22:52.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.724+0000 7fdbb3fff640 1 -- 172.21.15.16:0/1682322363 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc401a400 con 0x560edb345070 2024-08-28T19:22:52.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.724+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1682322363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 msgr2=0x560edb702220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:52.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.725+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1682322363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 0x560edb702220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4004a10 tx=0x7fdbc400be40 comp rx=0 tx=0).stop 2024-08-28T19:22:52.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.725+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1682322363 shutdown_connections 2024-08-28T19:22:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.725+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1682322363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 0x560edb702220 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.725+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1682322363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb701ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.725+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1682322363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 0x560edb6f7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.725+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1682322363 >> 172.21.15.16:0/1682322363 conn(0x560edb339460 msgr2=0x560edb6f7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.725+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1682322363 shutdown_connections 2024-08-28T19:22:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.725+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1682322363 wait complete. 2024-08-28T19:22:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.725+0000 7fdbd73a1740 1 -- start start 2024-08-28T19:22:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.725+0000 7fdbd73a1740 1 Processor -- start 2024-08-28T19:22:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 0x560edb6f9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb6fa1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbd73a1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 0x560edb6fa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1b8730 con 0x560edb324390 2024-08-28T19:22:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1b8060 con 0x560edb31c1b0 2024-08-28T19:22:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbd73a1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1b52a0 con 0x560edb345070 2024-08-28T19:22:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbcad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 0x560edb6f9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbcad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 0x560edb6f9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:50494/0 (socket says 172.21.15.16:50494) 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbcad9d640 1 -- 172.21.15.16:0/1710352222 learned_addr learned my addr 172.21.15.16:0/1710352222 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 0x560edb6fa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbc3fff640 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb6fa1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.726+0000 7fdbd73a1740 10 client.0 _ll_register_callbacks cb 0x7fdbb4005db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.727+0000 7fdbcad9d640 1 -- 172.21.15.16:0/1710352222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 msgr2=0x560edb6fa6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.727+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 0x560edb6fa6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.727+0000 7fdbcad9d640 1 -- 172.21.15.16:0/1710352222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 msgr2=0x560edb6fa1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.727+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb6fa1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.727+0000 7fdbcad9d640 1 -- 172.21.15.16:0/1710352222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560edb771900 con 0x560edb31c1b0 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.727+0000 7fdbcad9d640 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 0x560edb6f9c60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40049e0 tx=0x7fdbc4010a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d989f352796c5fa server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.727+0000 7fdbb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.727+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1710352222 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc401e030 con 0x560edb31c1b0 2024-08-28T19:22:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1710352222 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdbc4010bc0 con 0x560edb31c1b0 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1710352222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbc402a040 con 0x560edb31c1b0 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1710352222 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdbc4016070 con 0x560edb31c1b0 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbd73a1740 10 client.25814 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1710352222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560edb772a40 con 0x560edb31c1b0 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1710352222 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbc4010dc0 con 0x560edb31c1b0 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1710352222 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fdbc40307b0 con 0x560edb31c1b0 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbd73a1740 10 client.25814 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbd73a1740 10 client.25814 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbd73a1740 20 client.25814 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x560edb2d8da0 0x560edb774fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbd73a1740 4 client.25814 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:22:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.728+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1710352222 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fdbc402a040 con 0x560edb2d8da0 2024-08-28T19:22:52.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.729+0000 7fdbc3fff640 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x560edb2d8da0 0x560edb774fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:52.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.729+0000 7fdbb1ffb640 10 client.25814 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:22:52.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.729+0000 7fdbc3fff640 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x560edb2d8da0 0x560edb774fb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7175368a46cc6029 server_cookie=fa5d2bb10c4e071c in_seq=0 out_seq=0 2024-08-28T19:22:52.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.730+0000 7fdbb1ffb640 1 -- 172.21.15.16:0/1710352222 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fdbc402a040 con 0x560edb2d8da0 2024-08-28T19:22:52.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.730+0000 7fdbb1ffb640 10 client.25814 handle_command_reply: tid=0 2024-08-28T19:22:52.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.730+0000 7fdbd73a1740 1 client.25814 shutdown 2024-08-28T19:22:52.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.730+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1710352222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 msgr2=0x560edb6f9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.730+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 0x560edb6f9c60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40049e0 tx=0x7fdbc4010a20 comp rx=0 tx=0).stop 2024-08-28T19:22:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.731+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1710352222 shutdown_connections 2024-08-28T19:22:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.731+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x560edb2d8da0 0x560edb774fb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.731+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560edb345070 0x560edb6fa6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.731+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560edb324390 0x560edb6fa1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.731+0000 7fdbd73a1740 1 --2- 172.21.15.16:0/1710352222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560edb31c1b0 0x560edb6f9c60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.731+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1710352222 >> 172.21.15.16:0/1710352222 conn(0x560edb339460 msgr2=0x560edb6ebd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.731+0000 7fdbb1ffb640 0 client.25814 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:22:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.731+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1710352222 shutdown_connections 2024-08-28T19:22:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.731+0000 7fdbd73a1740 1 -- 172.21.15.16:0/1710352222 wait complete. 2024-08-28T19:22:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.732+0000 7fdbd73a1740 20 client.25814 trim_cache size 0 max 16384 2024-08-28T19:22:52.736 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:22:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.733+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1081199872 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbb80b4d20 msgr2=0x7fdbb80b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.733+0000 7fdbd0e2c640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbb80b4d20 0x7fdbb80b7230 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x560edb6e8f90 tx=0x7fdbbc079100 comp rx=0 tx=0).stop 2024-08-28T19:22:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.733+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1081199872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbcc109f30 msgr2=0x7fdbcc1a08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.733+0000 7fdbd0e2c640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbcc109f30 0x7fdbcc1a08d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00db80 tx=0x7fdbbc00b630 comp rx=0 tx=0).stop 2024-08-28T19:22:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.734+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1081199872 shutdown_connections 2024-08-28T19:22:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.734+0000 7fdbd0e2c640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbb80b4d20 0x7fdbb80b7230 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.734+0000 7fdbd0e2c640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbcc1101c0 0x7fdbcc1a6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.734+0000 7fdbd0e2c640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbcc10f870 0x7fdbcc1a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.734+0000 7fdbd0e2c640 1 --2- 172.21.15.16:0/1081199872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbcc109f30 0x7fdbcc1a08d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.734+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1081199872 >> 172.21.15.16:0/1081199872 conn(0x7fdbcc077f10 msgr2=0x7fdbcc108410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:52.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.734+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1081199872 shutdown_connections 2024-08-28T19:22:52.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:52.734+0000 7fdbd0e2c640 1 -- 172.21.15.16:0/1081199872 wait complete. 2024-08-28T19:22:52.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71102 12.57 MB/sec warmup 32 sec latency 3339.383 ms 2024-08-28T19:22:53.422 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:53 smithi115 ceph-mon[22184]: pgmap v394: 129 pgs: 129 active+clean; 54 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 6.8 MiB/s wr, 304 op/s 2024-08-28T19:22:53.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:53 smithi016 ceph-mon[18556]: pgmap v394: 129 pgs: 129 active+clean; 54 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 6.8 MiB/s wr, 304 op/s 2024-08-28T19:22:53.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:53 smithi028 ceph-mon[21769]: pgmap v394: 129 pgs: 129 active+clean; 54 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 6.8 MiB/s wr, 304 op/s 2024-08-28T19:22:53.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:22:53.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71102 12.18 MB/sec warmup 33 sec latency 4339.421 ms 2024-08-28T19:22:53.942 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:54.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.436+0000 7f3f9b617640 1 -- 172.21.15.16:0/1927968737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f94102880 msgr2=0x7f3f94102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:54.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.436+0000 7f3f9b617640 1 --2- 172.21.15.16:0/1927968737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f94102880 0x7f3f94102c90 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009ca0 tx=0x7f3f8402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.436+0000 7f3f9b617640 1 -- 172.21.15.16:0/1927968737 shutdown_connections 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.436+0000 7f3f9b617640 1 --2- 172.21.15.16:0/1927968737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f94108ac0 0x7f3f9410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.436+0000 7f3f9b617640 1 --2- 172.21.15.16:0/1927968737 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f94108170 0x7f3f94108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.436+0000 7f3f9b617640 1 --2- 172.21.15.16:0/1927968737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f94102880 0x7f3f94102c90 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.436+0000 7f3f9b617640 1 -- 172.21.15.16:0/1927968737 >> 172.21.15.16:0/1927968737 conn(0x7f3f94079530 msgr2=0x7f3f94079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.436+0000 7f3f9b617640 1 -- 172.21.15.16:0/1927968737 shutdown_connections 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.436+0000 7f3f9b617640 1 -- 172.21.15.16:0/1927968737 wait complete. 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.437+0000 7f3f9b617640 1 Processor -- start 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.437+0000 7f3f9b617640 1 -- start start 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f9b617640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f94102880 0x7f3f9419b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f9b617640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f94108170 0x7f3f9419b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f9b617640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f94108ac0 0x7f3f9419bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f9b617640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f94112770 con 0x7f3f94108170 2024-08-28T19:22:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f9b617640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941125f0 con 0x7f3f94108ac0 2024-08-28T19:22:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f9b617640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941128f0 con 0x7f3f94102880 2024-08-28T19:22:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f98bb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f94108170 0x7f3f9419b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f98bb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f94108170 0x7f3f9419b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58688/0 (socket says 172.21.15.16:58688) 2024-08-28T19:22:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f98bb2640 1 -- 172.21.15.16:0/791898444 learned_addr learned my addr 172.21.15.16:0/791898444 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f99bb4640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f94108ac0 0x7f3f9419bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f993b3640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f94102880 0x7f3f9419b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f98bb2640 1 -- 172.21.15.16:0/791898444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f94102880 msgr2=0x7f3f9419b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f98bb2640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f94102880 0x7f3f9419b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f98bb2640 1 -- 172.21.15.16:0/791898444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f94108ac0 msgr2=0x7f3f9419bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f98bb2640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f94108ac0 0x7f3f9419bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f98bb2640 1 -- 172.21.15.16:0/791898444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f941a6350 con 0x7f3f94108170 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f99bb4640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f94108ac0 0x7f3f9419bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f98bb2640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f94108170 0x7f3f9419b600 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3f8800d740 tx=0x7f3f8800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f927fc640 1 -- 172.21.15.16:0/791898444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88017070 con 0x7f3f94108170 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f927fc640 1 -- 172.21.15.16:0/791898444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f8800de50 con 0x7f3f94108170 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f9b617640 1 -- 172.21.15.16:0/791898444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f941a6640 con 0x7f3f94108170 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.438+0000 7f3f927fc640 1 -- 172.21.15.16:0/791898444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8800f620 con 0x7f3f94108170 2024-08-28T19:22:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.439+0000 7f3f9b617640 1 -- 172.21.15.16:0/791898444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f941a6b80 con 0x7f3f94108170 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.441+0000 7f3fa1b8c740 1 Processor -- start 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.441+0000 7f3fa1b8c740 1 -- start start 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.441+0000 7f3f927fc640 1 -- 172.21.15.16:0/791898444 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3f8801a070 con 0x7f3f94108170 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.441+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 0x55c11b99b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.441+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.441+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5d5390 0x55c11b99bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.441+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c11b912680 con 0x55c11b5cd1b0 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.441+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c11b901c10 con 0x55c11b5f6070 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.441+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c11b905bc0 con 0x55c11b5d5390 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f993b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f993b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58702/0 (socket says 172.21.15.16:58702) 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f993b3640 1 -- 172.21.15.16:0/1071848473 learned_addr learned my addr 172.21.15.16:0/1071848473 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f99bb4640 1 --2- 172.21.15.16:0/1071848473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 0x55c11b99b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f98bb2640 1 --2- 172.21.15.16:0/1071848473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5d5390 0x55c11b99bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f993b3640 1 -- 172.21.15.16:0/1071848473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5d5390 msgr2=0x55c11b99bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f993b3640 1 --2- 172.21.15.16:0/1071848473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5d5390 0x55c11b99bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f993b3640 1 -- 172.21.15.16:0/1071848473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 msgr2=0x55c11b99b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f993b3640 1 --2- 172.21.15.16:0/1071848473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 0x55c11b99b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f993b3640 1 -- 172.21.15.16:0/1071848473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c11b661650 con 0x55c11b5cd1b0 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f927fc640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f7c0b4cd0 0x7f3f7c0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f993b3640 1 --2- 172.21.15.16:0/1071848473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99b850 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3f84005cc0 tx=0x7f3f84009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2256df32618ebc2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f99bb4640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f7c0b4cd0 0x7f3f7c0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f927fc640 1 -- 172.21.15.16:0/791898444 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3f880e5380 con 0x7f3f94108170 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.442+0000 7f3f7b7fe640 1 -- 172.21.15.16:0/1071848473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84031db0 con 0x55c11b5cd1b0 2024-08-28T19:22:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.443+0000 7f3f7b7fe640 1 -- 172.21.15.16:0/1071848473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f84036070 con 0x55c11b5cd1b0 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.443+0000 7f3f7b7fe640 1 -- 172.21.15.16:0/1071848473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84004240 con 0x55c11b5cd1b0 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.443+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1071848473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 msgr2=0x55c11b99b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.443+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1071848473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99b850 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3f84005cc0 tx=0x7f3f84009800 comp rx=0 tx=0).stop 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.443+0000 7f3f99bb4640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f7c0b4cd0 0x7f3f7c0b71e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00bd60 tx=0x7f3f8c00b550 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.443+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1071848473 shutdown_connections 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.443+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1071848473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5d5390 0x55c11b99bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.443+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1071848473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99b850 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.444+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1071848473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 0x55c11b99b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.444+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1071848473 >> 172.21.15.16:0/1071848473 conn(0x55c11b5ea460 msgr2=0x55c11b999200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.444+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1071848473 shutdown_connections 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.444+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1071848473 wait complete. 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.444+0000 7f3fa1b8c740 1 -- start start 2024-08-28T19:22:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.444+0000 7f3fa1b8c740 1 Processor -- start 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5cd1b0 0x55c11b9a6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9a6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5f6070 0x55c11b9b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c11b764700 con 0x55c11b5d5390 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c11b888750 con 0x55c11b5cd1b0 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c11b9a9a90 con 0x55c11b5f6070 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3fa1b8c740 10 client.0 _ll_register_callbacks cb 0x55c11b8b7e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f98bb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9a6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f98bb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9a6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58704/0 (socket says 172.21.15.16:58704) 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f98bb2640 1 -- 172.21.15.16:0/1113692335 learned_addr learned my addr 172.21.15.16:0/1113692335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f99bb4640 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5f6070 0x55c11b9b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f993b3640 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5cd1b0 0x55c11b9a6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f98bb2640 1 -- 172.21.15.16:0/1113692335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5f6070 msgr2=0x55c11b9b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f98bb2640 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5f6070 0x55c11b9b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f98bb2640 1 -- 172.21.15.16:0/1113692335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5cd1b0 msgr2=0x55c11b9a6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f98bb2640 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5cd1b0 0x55c11b9a6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f98bb2640 1 -- 172.21.15.16:0/1113692335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c11b8c1900 con 0x55c11b5d5390 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f98bb2640 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9a6f70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009610 tx=0x7f3f880e8ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cdac0aa27d40492 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f78ff9640 1 -- 172.21.15.16:0/1113692335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88021070 con 0x55c11b5d5390 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3f78ff9640 1 -- 172.21.15.16:0/1113692335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3f880e5150 con 0x55c11b5d5390 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.445+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1113692335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f84036070 con 0x55c11b5d5390 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3f78ff9640 1 -- 172.21.15.16:0/1113692335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3f88092e60 con 0x55c11b5d5390 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3fa1b8c740 10 client.16194 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1113692335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c11b9b5820 con 0x55c11b5d5390 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3f78ff9640 1 -- 172.21.15.16:0/1113692335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f84036070 con 0x55c11b5d5390 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3f78ff9640 1 -- 172.21.15.16:0/1113692335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f3f880b4040 con 0x55c11b5d5390 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3fa1b8c740 10 client.16194 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3fa1b8c740 10 client.16194 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3fa1b8c740 20 client.16194 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55c11b5b49f0 0x55c11b9b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3fa1b8c740 4 client.16194 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1113692335 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c11b9145d0 con 0x55c11b5b49f0 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.446+0000 7f3f993b3640 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55c11b5b49f0 0x55c11b9b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.447+0000 7f3f78ff9640 10 client.16194 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.447+0000 7f3f993b3640 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55c11b5b49f0 0x55c11b9b8c10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3bbaed59eb80fb2 server_cookie=b784ecfa91b37958 in_seq=0 out_seq=0 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3f78ff9640 1 -- 172.21.15.16:0/1113692335 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c11b9145d0 con 0x55c11b5b49f0 2024-08-28T19:22:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3f78ff9640 10 client.16194 handle_command_reply: tid=0 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 1 client.16194 shutdown 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1113692335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 msgr2=0x55c11b9a6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9a6f70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009610 tx=0x7f3f880e8ac0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1113692335 shutdown_connections 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55c11b5b49f0 0x55c11b9b8c10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5f6070 0x55c11b9b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9a6f70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3f78ff9640 0 client.16194 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1113692335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5cd1b0 0x55c11b9a6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1113692335 >> 172.21.15.16:0/1113692335 conn(0x55c11b5ea460 msgr2=0x55c11b99e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1113692335 shutdown_connections 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1113692335 wait complete. 2024-08-28T19:22:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.448+0000 7f3fa1b8c740 20 client.16194 trim_cache size 0 max 16384 2024-08-28T19:22:54.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.461+0000 7f3fa1b8c740 1 Processor -- start 2024-08-28T19:22:54.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.461+0000 7f3fa1b8c740 1 -- start start 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5d5390 0x55c11b9a8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 0x55c11b9b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941b0c00 con 0x55c11b5cd1b0 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941b2800 con 0x55c11b5f6070 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941b12d0 con 0x55c11b5d5390 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3f993b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3f993b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58714/0 (socket says 172.21.15.16:58714) 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3f993b3640 1 -- 172.21.15.16:0/1842988859 learned_addr learned my addr 172.21.15.16:0/1842988859 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3f98bb2640 1 --2- 172.21.15.16:0/1842988859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5d5390 0x55c11b9a8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3f993b3640 1 -- 172.21.15.16:0/1842988859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5d5390 msgr2=0x55c11b9a8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3f993b3640 1 --2- 172.21.15.16:0/1842988859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5d5390 0x55c11b9a8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3f993b3640 1 -- 172.21.15.16:0/1842988859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 msgr2=0x55c11b9b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3f993b3640 1 --2- 172.21.15.16:0/1842988859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 0x55c11b9b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3f993b3640 1 -- 172.21.15.16:0/1842988859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c11b99d600 con 0x55c11b5cd1b0 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3f993b3640 1 --2- 172.21.15.16:0/1842988859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99cd00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3f84000f80 tx=0x7f3f84009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf581496e2020a1c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.462+0000 7f3f7affd640 1 -- 172.21.15.16:0/1842988859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84049040 con 0x55c11b5cd1b0 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.463+0000 7f3f7affd640 1 -- 172.21.15.16:0/1842988859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f84036040 con 0x55c11b5cd1b0 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.463+0000 7f3f7affd640 1 -- 172.21.15.16:0/1842988859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f840305c0 con 0x55c11b5cd1b0 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.463+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1842988859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 msgr2=0x55c11b99cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.463+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1842988859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99cd00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3f84000f80 tx=0x7f3f84009b40 comp rx=0 tx=0).stop 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.463+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1842988859 shutdown_connections 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.463+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1842988859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 0x55c11b9b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.463+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1842988859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5d5390 0x55c11b9a8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.463+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/1842988859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5cd1b0 0x55c11b99cd00 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.463+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1842988859 >> 172.21.15.16:0/1842988859 conn(0x55c11b5ea460 msgr2=0x55c11b9a7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.463+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1842988859 shutdown_connections 2024-08-28T19:22:54.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.463+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/1842988859 wait complete. 2024-08-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3fa1b8c740 1 -- start start 2024-08-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3fa1b8c740 1 Processor -- start 2024-08-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5cd1b0 0x55c11b9acda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3fa1b8c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 0x55c11b9ad820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941b2070 con 0x55c11b5d5390 2024-08-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941b19a0 con 0x55c11b5f6070 2024-08-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3fa1b8c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941aebe0 con 0x55c11b5cd1b0 2024-08-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3fa1b8c740 10 client.0 _ll_register_callbacks cb 0x7f3f88009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3f98bb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3f98bb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9ad2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58730/0 (socket says 172.21.15.16:58730) 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3f98bb2640 1 -- 172.21.15.16:0/2732612454 learned_addr learned my addr 172.21.15.16:0/2732612454 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.464+0000 7f3f99bb4640 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 0x55c11b9ad820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3f993b3640 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5cd1b0 0x55c11b9acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3f98bb2640 1 -- 172.21.15.16:0/2732612454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5cd1b0 msgr2=0x55c11b9acda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3f98bb2640 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5cd1b0 0x55c11b9acda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3f98bb2640 1 -- 172.21.15.16:0/2732612454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 msgr2=0x55c11b9ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3f98bb2640 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 0x55c11b9ad820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3f98bb2640 1 -- 172.21.15.16:0/2732612454 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c11ba22690 con 0x55c11b5d5390 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3f98bb2640 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9ad2e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f3f880ae040 tx=0x7f3f88009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d4f61e1d83486d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3f78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3f78ff9640 1 -- 172.21.15.16:0/2732612454 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88021070 con 0x55c11b5d5390 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3f78ff9640 1 -- 172.21.15.16:0/2732612454 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3f880a7280 con 0x55c11b5d5390 2024-08-28T19:22:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/2732612454 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f84036040 con 0x55c11b5d5390 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3f78ff9640 1 -- 172.21.15.16:0/2732612454 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3f880ab050 con 0x55c11b5d5390 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3fa1b8c740 10 client.16200 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.465+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/2732612454 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c11ba238f0 con 0x55c11b5d5390 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.466+0000 7f3f78ff9640 1 -- 172.21.15.16:0/2732612454 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f84036040 con 0x55c11b5d5390 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.466+0000 7f3f78ff9640 1 -- 172.21.15.16:0/2732612454 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f3f880b4040 con 0x55c11b5d5390 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.466+0000 7f3fa1b8c740 10 client.16200 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.466+0000 7f3fa1b8c740 10 client.16200 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.466+0000 7f3fa1b8c740 20 client.16200 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.466+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55c11b589da0 0x55c11ba26020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.466+0000 7f3fa1b8c740 4 client.16200 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.466+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/2732612454 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c11b910070 con 0x55c11b589da0 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.466+0000 7f3f993b3640 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55c11b589da0 0x55c11ba26020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3f993b3640 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55c11b589da0 0x55c11ba26020 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4ae33d25edfb7266 server_cookie=8f66ac9d7e04dcaa in_seq=0 out_seq=0 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3f78ff9640 10 client.16200 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3f78ff9640 1 -- 172.21.15.16:0/2732612454 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c11b910070 con 0x55c11b589da0 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3f78ff9640 10 client.16200 handle_command_reply: tid=0 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 1 client.16200 shutdown 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/2732612454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 msgr2=0x55c11b9ad2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9ad2e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f3f880ae040 tx=0x7f3f88009f30 comp rx=0 tx=0).stop 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/2732612454 shutdown_connections 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55c11b589da0 0x55c11ba26020 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c11b5f6070 0x55c11b9ad820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3f78ff9640 0 client.16200 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c11b5d5390 0x55c11b9ad2e0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 1 --2- 172.21.15.16:0/2732612454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c11b5cd1b0 0x55c11b9acda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/2732612454 >> 172.21.15.16:0/2732612454 conn(0x55c11b5ea460 msgr2=0x55c11b9a8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:54.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/2732612454 shutdown_connections 2024-08-28T19:22:54.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 1 -- 172.21.15.16:0/2732612454 wait complete. 2024-08-28T19:22:54.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.467+0000 7f3fa1b8c740 20 client.16200 trim_cache size 0 max 16384 2024-08-28T19:22:54.475 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 -- 172.21.15.16:0/791898444 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f7c0b4cd0 msgr2=0x7f3f7c0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f7c0b4cd0 0x7f3f7c0b71e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00bd60 tx=0x7f3f8c00b550 comp rx=0 tx=0).stop 2024-08-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 -- 172.21.15.16:0/791898444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f94108170 msgr2=0x7f3f9419b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f94108170 0x7f3f9419b600 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3f8800d740 tx=0x7f3f8800dc00 comp rx=0 tx=0).stop 2024-08-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 -- 172.21.15.16:0/791898444 shutdown_connections 2024-08-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f7c0b4cd0 0x7f3f7c0b71e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f94108ac0 0x7f3f9419bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f94108170 0x7f3f9419b600 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 --2- 172.21.15.16:0/791898444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f94102880 0x7f3f9419b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 -- 172.21.15.16:0/791898444 >> 172.21.15.16:0/791898444 conn(0x7f3f94079530 msgr2=0x7f3f941062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 -- 172.21.15.16:0/791898444 shutdown_connections 2024-08-28T19:22:54.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:54.468+0000 7f3f9b617640 1 -- 172.21.15.16:0/791898444 wait complete. 2024-08-28T19:22:54.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71202 11.86 MB/sec warmup 34 sec latency 4944.281 ms 2024-08-28T19:22:55.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:55 smithi016 ceph-mon[18556]: pgmap v395: 129 pgs: 129 active+clean; 47 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 6.3 MiB/s wr, 246 op/s 2024-08-28T19:22:55.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:22:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:55 smithi028 ceph-mon[21769]: pgmap v395: 129 pgs: 129 active+clean; 47 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 6.3 MiB/s wr, 246 op/s 2024-08-28T19:22:55.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:55 smithi115 ceph-mon[22184]: pgmap v395: 129 pgs: 129 active+clean; 47 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 6.3 MiB/s wr, 246 op/s 2024-08-28T19:22:55.648 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:55.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71202 11.52 MB/sec warmup 35 sec latency 1304.383 ms 2024-08-28T19:22:56.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.137+0000 7fce5c729640 1 -- 172.21.15.16:0/1980108096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce54109240 msgr2=0x7fce54109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.137+0000 7fce5c729640 1 --2- 172.21.15.16:0/1980108096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce54109240 0x7fce54109650 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fce44009900 tx=0x7fce4402a410 comp rx=0 tx=0).stop 2024-08-28T19:22:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.137+0000 7fce5c729640 1 -- 172.21.15.16:0/1980108096 shutdown_connections 2024-08-28T19:22:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.137+0000 7fce5c729640 1 --2- 172.21.15.16:0/1980108096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce54109b90 0x7fce541103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.137+0000 7fce5c729640 1 --2- 172.21.15.16:0/1980108096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce54109240 0x7fce54109650 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.137+0000 7fce5c729640 1 --2- 172.21.15.16:0/1980108096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce54103900 0x7fce54103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.137+0000 7fce5c729640 1 -- 172.21.15.16:0/1980108096 >> 172.21.15.16:0/1980108096 conn(0x7fce5407b5f0 msgr2=0x7fce5407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:56.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.137+0000 7fce5c729640 1 -- 172.21.15.16:0/1980108096 shutdown_connections 2024-08-28T19:22:56.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.137+0000 7fce5c729640 1 -- 172.21.15.16:0/1980108096 wait complete. 2024-08-28T19:22:56.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce5c729640 1 Processor -- start 2024-08-28T19:22:56.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce5c729640 1 -- start start 2024-08-28T19:22:56.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce5c729640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce54103900 0x7fce5419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce5c729640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce54109240 0x7fce5419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce5c729640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce54109b90 0x7fce5419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce5c729640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce54113850 con 0x7fce54109240 2024-08-28T19:22:56.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce5c729640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce541136d0 con 0x7fce54109b90 2024-08-28T19:22:56.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce5c729640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fce541139d0 con 0x7fce54103900 2024-08-28T19:22:56.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce59cc4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce54109240 0x7fce5419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce59cc4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce54109240 0x7fce5419a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58744/0 (socket says 172.21.15.16:58744) 2024-08-28T19:22:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce59cc4640 1 -- 172.21.15.16:0/1271863991 learned_addr learned my addr 172.21.15.16:0/1271863991 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.138+0000 7fce5acc6640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce54109b90 0x7fce5419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce54103900 0x7fce5419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce59cc4640 1 -- 172.21.15.16:0/1271863991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce54103900 msgr2=0x7fce5419a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce59cc4640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce54103900 0x7fce5419a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce59cc4640 1 -- 172.21.15.16:0/1271863991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce54109b90 msgr2=0x7fce5419fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce59cc4640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce54109b90 0x7fce5419fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce59cc4640 1 -- 172.21.15.16:0/1271863991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce541a01d0 con 0x7fce54109240 2024-08-28T19:22:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce54103900 0x7fce5419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:22:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce59cc4640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce54109240 0x7fce5419a8c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fce4402a930 tx=0x7fce44004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce4b7fe640 1 -- 172.21.15.16:0/1271863991 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4403f030 con 0x7fce54109240 2024-08-28T19:22:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce5c729640 1 -- 172.21.15.16:0/1271863991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce541a0460 con 0x7fce54109240 2024-08-28T19:22:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce4b7fe640 1 -- 172.21.15.16:0/1271863991 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce44003aa0 con 0x7fce54109240 2024-08-28T19:22:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce4b7fe640 1 -- 172.21.15.16:0/1271863991 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44035070 con 0x7fce54109240 2024-08-28T19:22:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.139+0000 7fce5c729640 1 -- 172.21.15.16:0/1271863991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce541a0990 con 0x7fce54109240 2024-08-28T19:22:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce62c9e740 1 Processor -- start 2024-08-28T19:22:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce62c9e740 1 -- start start 2024-08-28T19:22:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce4b7fe640 1 -- 172.21.15.16:0/1271863991 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fce44042030 con 0x7fce54109240 2024-08-28T19:22:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aed6070 0x55ae0b27b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aeb5390 0x55ae0b27bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce62c9e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b1f2680 con 0x55ae0aead1b0 2024-08-28T19:22:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce62c9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b1e1c10 con 0x55ae0aed6070 2024-08-28T19:22:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce62c9e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b1e5bc0 con 0x55ae0aeb5390 2024-08-28T19:22:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce5a4c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce5acc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aed6070 0x55ae0b27b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce5a4c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58758/0 (socket says 172.21.15.16:58758) 2024-08-28T19:22:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce5a4c5640 1 -- 172.21.15.16:0/111228906 learned_addr learned my addr 172.21.15.16:0/111228906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.142+0000 7fce59cc4640 1 --2- 172.21.15.16:0/111228906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aeb5390 0x55ae0b27bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce5a4c5640 1 -- 172.21.15.16:0/111228906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aeb5390 msgr2=0x55ae0b27bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/111228906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aeb5390 0x55ae0b27bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce5a4c5640 1 -- 172.21.15.16:0/111228906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aed6070 msgr2=0x55ae0b27b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/111228906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aed6070 0x55ae0b27b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce5a4c5640 1 -- 172.21.15.16:0/111228906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae0af41650 con 0x55ae0aead1b0 2024-08-28T19:22:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce4b7fe640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fce400b4cd0 0x7fce400b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/111228906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27b850 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fce50005fd0 tx=0x7fce5000e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1984a450c06d4e3e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce5acc6640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fce400b4cd0 0x7fce400b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce4b7fe640 1 -- 172.21.15.16:0/1271863991 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fce44105d80 con 0x7fce54109240 2024-08-28T19:22:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce48ff9640 1 -- 172.21.15.16:0/111228906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50016070 con 0x55ae0aead1b0 2024-08-28T19:22:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce48ff9640 1 -- 172.21.15.16:0/111228906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce500054d0 con 0x55ae0aead1b0 2024-08-28T19:22:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.143+0000 7fce48ff9640 1 -- 172.21.15.16:0/111228906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50011470 con 0x55ae0aead1b0 2024-08-28T19:22:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.144+0000 7fce62c9e740 1 -- 172.21.15.16:0/111228906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 msgr2=0x55ae0b27b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.144+0000 7fce5acc6640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fce400b4cd0 0x7fce400b71e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fce4c007f00 tx=0x7fce4c007520 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.144+0000 7fce62c9e740 1 --2- 172.21.15.16:0/111228906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27b850 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fce50005fd0 tx=0x7fce5000e6c0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.144+0000 7fce62c9e740 1 -- 172.21.15.16:0/111228906 shutdown_connections 2024-08-28T19:22:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.144+0000 7fce62c9e740 1 --2- 172.21.15.16:0/111228906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aeb5390 0x55ae0b27bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.144+0000 7fce62c9e740 1 --2- 172.21.15.16:0/111228906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27b850 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.144+0000 7fce62c9e740 1 --2- 172.21.15.16:0/111228906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aed6070 0x55ae0b27b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.144+0000 7fce62c9e740 1 -- 172.21.15.16:0/111228906 >> 172.21.15.16:0/111228906 conn(0x55ae0aeca460 msgr2=0x55ae0b279200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.144+0000 7fce62c9e740 1 -- 172.21.15.16:0/111228906 shutdown_connections 2024-08-28T19:22:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.144+0000 7fce62c9e740 1 -- 172.21.15.16:0/111228906 wait complete. 2024-08-28T19:22:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce62c9e740 1 -- start start 2024-08-28T19:22:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce62c9e740 1 Processor -- start 2024-08-28T19:22:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b286a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aeb5390 0x55ae0b286f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aed6070 0x55ae0b292de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce62c9e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b044700 con 0x55ae0aead1b0 2024-08-28T19:22:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce62c9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b168750 con 0x55ae0aeb5390 2024-08-28T19:22:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce62c9e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b289af0 con 0x55ae0aed6070 2024-08-28T19:22:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce62c9e740 10 client.0 _ll_register_callbacks cb 0x55ae0b197e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce5a4c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b286a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce5a4c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b286a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58764/0 (socket says 172.21.15.16:58764) 2024-08-28T19:22:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce5a4c5640 1 -- 172.21.15.16:0/3960647957 learned_addr learned my addr 172.21.15.16:0/3960647957 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.145+0000 7fce59cc4640 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aeb5390 0x55ae0b286f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce5a4c5640 1 -- 172.21.15.16:0/3960647957 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aed6070 msgr2=0x55ae0b292de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aed6070 0x55ae0b292de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce5a4c5640 1 -- 172.21.15.16:0/3960647957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aeb5390 msgr2=0x55ae0b286f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aeb5390 0x55ae0b286f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce5a4c5640 1 -- 172.21.15.16:0/3960647957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae0b1a1900 con 0x55ae0aead1b0 2024-08-28T19:22:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b286a50 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fce50005940 tx=0x7fce5000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0d286c4180b882 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce3dffb640 1 -- 172.21.15.16:0/3960647957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50016040 con 0x55ae0aead1b0 2024-08-28T19:22:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce3dffb640 1 -- 172.21.15.16:0/3960647957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fce50004ec0 con 0x55ae0aead1b0 2024-08-28T19:22:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce62c9e740 1 -- 172.21.15.16:0/3960647957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce500054d0 con 0x55ae0aead1b0 2024-08-28T19:22:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce3dffb640 1 -- 172.21.15.16:0/3960647957 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fce500098a0 con 0x55ae0aead1b0 2024-08-28T19:22:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce62c9e740 10 client.16212 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce62c9e740 1 -- 172.21.15.16:0/3960647957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ae0b295830 con 0x55ae0aead1b0 2024-08-28T19:22:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.146+0000 7fce3dffb640 1 -- 172.21.15.16:0/3960647957 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce500054d0 con 0x55ae0aead1b0 2024-08-28T19:22:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.147+0000 7fce3dffb640 1 -- 172.21.15.16:0/3960647957 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fce5000f040 con 0x55ae0aead1b0 2024-08-28T19:22:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.147+0000 7fce62c9e740 10 client.16212 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.147+0000 7fce62c9e740 10 client.16212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:56.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.147+0000 7fce62c9e740 20 client.16212 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:56.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.147+0000 7fce62c9e740 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae0ae949f0 0x55ae0b298b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.147+0000 7fce62c9e740 4 client.16212 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:56.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.147+0000 7fce62c9e740 1 -- 172.21.15.16:0/3960647957 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae0b1f45d0 con 0x55ae0ae949f0 2024-08-28T19:22:56.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.147+0000 7fce59cc4640 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae0ae949f0 0x55ae0b298b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.148+0000 7fce59cc4640 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae0ae949f0 0x55ae0b298b90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89ccdb32a9648cfd server_cookie=10bf8d835cc98558 in_seq=0 out_seq=0 2024-08-28T19:22:56.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.148+0000 7fce3dffb640 10 client.16212 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:56.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.148+0000 7fce3dffb640 1 -- 172.21.15.16:0/3960647957 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ae0b1f45d0 con 0x55ae0ae949f0 2024-08-28T19:22:56.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.148+0000 7fce3dffb640 10 client.16212 handle_command_reply: tid=0 2024-08-28T19:22:56.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.148+0000 7fce62c9e740 1 client.16212 shutdown 2024-08-28T19:22:56.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce62c9e740 1 -- 172.21.15.16:0/3960647957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 msgr2=0x55ae0b286a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:56.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce62c9e740 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b286a50 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fce50005940 tx=0x7fce5000ef60 comp rx=0 tx=0).stop 2024-08-28T19:22:56.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce62c9e740 1 -- 172.21.15.16:0/3960647957 shutdown_connections 2024-08-28T19:22:56.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce62c9e740 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae0ae949f0 0x55ae0b298b90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce62c9e740 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aed6070 0x55ae0b292de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce62c9e740 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aeb5390 0x55ae0b286f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce62c9e740 1 --2- 172.21.15.16:0/3960647957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b286a50 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce62c9e740 1 -- 172.21.15.16:0/3960647957 >> 172.21.15.16:0/3960647957 conn(0x55ae0aeca460 msgr2=0x55ae0b27e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:56.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce3dffb640 0 client.16212 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:56.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce62c9e740 1 -- 172.21.15.16:0/3960647957 shutdown_connections 2024-08-28T19:22:56.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce62c9e740 1 -- 172.21.15.16:0/3960647957 wait complete. 2024-08-28T19:22:56.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.149+0000 7fce62c9e740 20 client.16212 trim_cache size 0 max 16384 2024-08-28T19:22:56.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.162+0000 7fce62c9e740 1 Processor -- start 2024-08-28T19:22:56.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.162+0000 7fce62c9e740 1 -- start start 2024-08-28T19:22:56.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.162+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aeb5390 0x55ae0b292ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aed6070 0x55ae0b278b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce62c9e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce541b0ce0 con 0x55ae0aead1b0 2024-08-28T19:22:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce5a4c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce62c9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce541b28e0 con 0x55ae0aeb5390 2024-08-28T19:22:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce62c9e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fce541b13b0 con 0x55ae0aed6070 2024-08-28T19:22:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce5a4c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58776/0 (socket says 172.21.15.16:58776) 2024-08-28T19:22:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce5a4c5640 1 -- 172.21.15.16:0/905236160 learned_addr learned my addr 172.21.15.16:0/905236160 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce59cc4640 1 --2- 172.21.15.16:0/905236160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aeb5390 0x55ae0b292ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce5acc6640 1 --2- 172.21.15.16:0/905236160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aed6070 0x55ae0b278b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce5a4c5640 1 -- 172.21.15.16:0/905236160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aed6070 msgr2=0x55ae0b278b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:56.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/905236160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aed6070 0x55ae0b278b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce5a4c5640 1 -- 172.21.15.16:0/905236160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aeb5390 msgr2=0x55ae0b292ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:56.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/905236160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aeb5390 0x55ae0b292ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce5a4c5640 1 -- 172.21.15.16:0/905236160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae0b287d10 con 0x55ae0aead1b0 2024-08-28T19:22:56.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce5acc6640 1 --2- 172.21.15.16:0/905236160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aed6070 0x55ae0b278b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:56.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.163+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/905236160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27eb20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fce50006010 tx=0x7fce5000e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f748d2483eabb23f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:56.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.164+0000 7fce3ffff640 1 -- 172.21.15.16:0/905236160 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50005c50 con 0x55ae0aead1b0 2024-08-28T19:22:56.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.164+0000 7fce3ffff640 1 -- 172.21.15.16:0/905236160 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce500054d0 con 0x55ae0aead1b0 2024-08-28T19:22:56.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.164+0000 7fce3ffff640 1 -- 172.21.15.16:0/905236160 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5001ac10 con 0x55ae0aead1b0 2024-08-28T19:22:56.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.165+0000 7fce62c9e740 1 -- 172.21.15.16:0/905236160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 msgr2=0x55ae0b27eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:56.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.165+0000 7fce62c9e740 1 --2- 172.21.15.16:0/905236160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27eb20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fce50006010 tx=0x7fce5000e6c0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.165+0000 7fce62c9e740 1 -- 172.21.15.16:0/905236160 shutdown_connections 2024-08-28T19:22:56.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.165+0000 7fce62c9e740 1 --2- 172.21.15.16:0/905236160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aed6070 0x55ae0b278b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.165+0000 7fce62c9e740 1 --2- 172.21.15.16:0/905236160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aeb5390 0x55ae0b292ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.165+0000 7fce62c9e740 1 --2- 172.21.15.16:0/905236160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b27eb20 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.165+0000 7fce62c9e740 1 -- 172.21.15.16:0/905236160 >> 172.21.15.16:0/905236160 conn(0x55ae0aeca460 msgr2=0x55ae0b27e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:56.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.165+0000 7fce62c9e740 1 -- 172.21.15.16:0/905236160 shutdown_connections 2024-08-28T19:22:56.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.165+0000 7fce62c9e740 1 -- 172.21.15.16:0/905236160 wait complete. 2024-08-28T19:22:56.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce62c9e740 1 -- start start 2024-08-28T19:22:56.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce62c9e740 1 Processor -- start 2024-08-28T19:22:56.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b28aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aeb5390 0x55ae0b28b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce62c9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aed6070 0x55ae0b28b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce62c9e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce541b2150 con 0x55ae0aead1b0 2024-08-28T19:22:56.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce62c9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce541b1a80 con 0x55ae0aed6070 2024-08-28T19:22:56.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce62c9e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fce541aecc0 con 0x55ae0aeb5390 2024-08-28T19:22:56.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce62c9e740 10 client.0 _ll_register_callbacks cb 0x7fce4c007860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:56.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce5a4c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b28aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce5a4c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b28aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58792/0 (socket says 172.21.15.16:58792) 2024-08-28T19:22:56.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce5a4c5640 1 -- 172.21.15.16:0/1524840263 learned_addr learned my addr 172.21.15.16:0/1524840263 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:56.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce5acc6640 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aed6070 0x55ae0b28b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.166+0000 7fce59cc4640 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aeb5390 0x55ae0b28b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce5a4c5640 1 -- 172.21.15.16:0/1524840263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aeb5390 msgr2=0x55ae0b28b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:56.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aeb5390 0x55ae0b28b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce5a4c5640 1 -- 172.21.15.16:0/1524840263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aed6070 msgr2=0x55ae0b28b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:56.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aed6070 0x55ae0b28b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce5a4c5640 1 -- 172.21.15.16:0/1524840263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae0b3025f0 con 0x55ae0aead1b0 2024-08-28T19:22:56.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce5a4c5640 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b28aee0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fce50014040 tx=0x7fce50005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff3e5008ab224e8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:56.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:56.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce3dffb640 1 -- 172.21.15.16:0/1524840263 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50016030 con 0x55ae0aead1b0 2024-08-28T19:22:56.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce3dffb640 1 -- 172.21.15.16:0/1524840263 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fce50004ec0 con 0x55ae0aead1b0 2024-08-28T19:22:56.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce62c9e740 1 -- 172.21.15.16:0/1524840263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce500054d0 con 0x55ae0aead1b0 2024-08-28T19:22:56.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce3dffb640 1 -- 172.21.15.16:0/1524840263 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fce50043050 con 0x55ae0aead1b0 2024-08-28T19:22:56.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce62c9e740 10 client.16218 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:56.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce62c9e740 1 -- 172.21.15.16:0/1524840263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ae0b303830 con 0x55ae0aead1b0 2024-08-28T19:22:56.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.167+0000 7fce3dffb640 1 -- 172.21.15.16:0/1524840263 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce500054d0 con 0x55ae0aead1b0 2024-08-28T19:22:56.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.168+0000 7fce3dffb640 1 -- 172.21.15.16:0/1524840263 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fce500116e0 con 0x55ae0aead1b0 2024-08-28T19:22:56.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.168+0000 7fce62c9e740 10 client.16218 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:56.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.168+0000 7fce62c9e740 10 client.16218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:56.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.168+0000 7fce62c9e740 20 client.16218 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:56.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.168+0000 7fce62c9e740 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae0ae69da0 0x55ae0b305fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:56.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.168+0000 7fce62c9e740 4 client.16218 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:22:56.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.168+0000 7fce62c9e740 1 -- 172.21.15.16:0/1524840263 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ae0b1f0070 con 0x55ae0ae69da0 2024-08-28T19:22:56.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.168+0000 7fce59cc4640 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae0ae69da0 0x55ae0b305fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:56.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce3dffb640 10 client.16218 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:56.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce59cc4640 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae0ae69da0 0x55ae0b305fa0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b9ba3453f8b37a server_cookie=44c9ca7424ef2eb0 in_seq=0 out_seq=0 2024-08-28T19:22:56.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce3dffb640 1 -- 172.21.15.16:0/1524840263 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ae0b1f0070 con 0x55ae0ae69da0 2024-08-28T19:22:56.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce3dffb640 10 client.16218 handle_command_reply: tid=0 2024-08-28T19:22:56.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 1 client.16218 shutdown 2024-08-28T19:22:56.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 1 -- 172.21.15.16:0/1524840263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 msgr2=0x55ae0b28aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:56.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b28aee0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fce50014040 tx=0x7fce50005680 comp rx=0 tx=0).stop 2024-08-28T19:22:56.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 1 -- 172.21.15.16:0/1524840263 shutdown_connections 2024-08-28T19:22:56.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae0aed6070 0x55ae0b28b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae0ae69da0 0x55ae0b305fa0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae0aeb5390 0x55ae0b28b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce3dffb640 0 client.16218 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:56.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 1 --2- 172.21.15.16:0/1524840263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae0aead1b0 0x55ae0b28aee0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 1 -- 172.21.15.16:0/1524840263 >> 172.21.15.16:0/1524840263 conn(0x55ae0aeca460 msgr2=0x55ae0b278b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:56.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 1 -- 172.21.15.16:0/1524840263 shutdown_connections 2024-08-28T19:22:56.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 1 -- 172.21.15.16:0/1524840263 wait complete. 2024-08-28T19:22:56.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.169+0000 7fce62c9e740 20 client.16218 trim_cache size 0 max 16384 2024-08-28T19:22:56.197 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:22:56.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 -- 172.21.15.16:0/1271863991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fce400b4cd0 msgr2=0x7fce400b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:56.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fce400b4cd0 0x7fce400b71e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fce4c007f00 tx=0x7fce4c007520 comp rx=0 tx=0).stop 2024-08-28T19:22:56.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 -- 172.21.15.16:0/1271863991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce54109240 msgr2=0x7fce5419a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:56.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce54109240 0x7fce5419a8c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fce4402a930 tx=0x7fce44004270 comp rx=0 tx=0).stop 2024-08-28T19:22:56.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 -- 172.21.15.16:0/1271863991 shutdown_connections 2024-08-28T19:22:56.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fce400b4cd0 0x7fce400b71e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce54109b90 0x7fce5419fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce54109240 0x7fce5419a8c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 --2- 172.21.15.16:0/1271863991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce54103900 0x7fce5419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:56.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 -- 172.21.15.16:0/1271863991 >> 172.21.15.16:0/1271863991 conn(0x7fce5407b5f0 msgr2=0x7fce541053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:56.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 -- 172.21.15.16:0/1271863991 shutdown_connections 2024-08-28T19:22:56.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:56.170+0000 7fce5c729640 1 -- 172.21.15.16:0/1271863991 wait complete. 2024-08-28T19:22:56.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71202 11.20 MB/sec warmup 36 sec latency 2304.455 ms 2024-08-28T19:22:57.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:57 smithi016 ceph-mon[18556]: pgmap v396: 129 pgs: 129 active+clean; 48 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 6.6 MiB/s wr, 264 op/s 2024-08-28T19:22:57.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:57 smithi028 ceph-mon[21769]: pgmap v396: 129 pgs: 129 active+clean; 48 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 6.6 MiB/s wr, 264 op/s 2024-08-28T19:22:57.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:57 smithi115 ceph-mon[22184]: pgmap v396: 129 pgs: 129 active+clean; 48 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 6.6 MiB/s wr, 264 op/s 2024-08-28T19:22:57.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71202 10.89 MB/sec warmup 37 sec latency 3304.947 ms 2024-08-28T19:22:57.972 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:22:57.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:22:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:58.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.735+0000 7f627d50c640 1 -- 172.21.15.16:0/4003440763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6278108140 msgr2=0x7f6278108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.735+0000 7f627d50c640 1 --2- 172.21.15.16:0/4003440763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6278108140 0x7f6278108550 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f626c009900 tx=0x7f626c02a410 comp rx=0 tx=0).stop 2024-08-28T19:22:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.735+0000 7f627d50c640 1 -- 172.21.15.16:0/4003440763 shutdown_connections 2024-08-28T19:22:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.735+0000 7f627d50c640 1 --2- 172.21.15.16:0/4003440763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6278108a90 0x7f627810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.735+0000 7f627d50c640 1 --2- 172.21.15.16:0/4003440763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6278108140 0x7f6278108550 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.735+0000 7f627d50c640 1 --2- 172.21.15.16:0/4003440763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6278102800 0x7f6278102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.735+0000 7f627d50c640 1 -- 172.21.15.16:0/4003440763 >> 172.21.15.16:0/4003440763 conn(0x7f62780794d0 msgr2=0x7f62780798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.735+0000 7f627d50c640 1 -- 172.21.15.16:0/4003440763 shutdown_connections 2024-08-28T19:22:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.735+0000 7f627d50c640 1 -- 172.21.15.16:0/4003440763 wait complete. 2024-08-28T19:22:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f627d50c640 1 Processor -- start 2024-08-28T19:22:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f627d50c640 1 -- start start 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f627d50c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6278102800 0x7f627819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f627d50c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6278108140 0x7f627819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f627d50c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6278108a90 0x7f627819fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f627d50c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6278113820 con 0x7f6278102800 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f627d50c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62781136a0 con 0x7f6278108140 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f627d50c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f62781139a0 con 0x7f6278108a90 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f6276ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6278102800 0x7f627819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f6276ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6278102800 0x7f627819a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58816/0 (socket says 172.21.15.16:58816) 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f6276ffd640 1 -- 172.21.15.16:0/2262524449 learned_addr learned my addr 172.21.15.16:0/2262524449 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f62777fe640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6278108a90 0x7f627819fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.736+0000 7f62767fc640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6278108140 0x7f627819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f6276ffd640 1 -- 172.21.15.16:0/2262524449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6278108a90 msgr2=0x7f627819fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f6276ffd640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6278108a90 0x7f627819fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f6276ffd640 1 -- 172.21.15.16:0/2262524449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6278108140 msgr2=0x7f627819a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f6276ffd640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6278108140 0x7f627819a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f6276ffd640 1 -- 172.21.15.16:0/2262524449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62781a01c0 con 0x7f6278102800 2024-08-28T19:22:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f62767fc640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6278108140 0x7f627819a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f6276ffd640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6278102800 0x7f627819a250 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f626800c890 tx=0x7f626800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f625ffff640 1 -- 172.21.15.16:0/2262524449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268004220 con 0x7f6278102800 2024-08-28T19:22:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f627d50c640 1 -- 172.21.15.16:0/2262524449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62781a04b0 con 0x7f6278102800 2024-08-28T19:22:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f625ffff640 1 -- 172.21.15.16:0/2262524449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62680043c0 con 0x7f6278102800 2024-08-28T19:22:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f625ffff640 1 -- 172.21.15.16:0/2262524449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268004a00 con 0x7f6278102800 2024-08-28T19:22:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.737+0000 7f627d50c640 1 -- 172.21.15.16:0/2262524449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62781a6c10 con 0x7f6278102800 2024-08-28T19:22:58.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f6283a81740 1 Processor -- start 2024-08-28T19:22:58.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f6283a81740 1 -- start start 2024-08-28T19:22:58.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f625ffff640 1 -- 172.21.15.16:0/2262524449 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f62680079b0 con 0x7f6278102800 2024-08-28T19:22:58.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f6283a81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d0817350 0x55a0d0bbd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f6283a81740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bbdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f6283a81740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d07f6670 0x55a0d0bbe200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f6283a81740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a0d0b335f0 con 0x55a0d07ee490 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f6283a81740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0d0b22b80 con 0x55a0d0817350 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f6283a81740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a0d0b26b30 con 0x55a0d07f6670 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f6276ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bbdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f6276ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bbdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58828/0 (socket says 172.21.15.16:58828) 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.740+0000 7f6276ffd640 1 -- 172.21.15.16:0/1603445279 learned_addr learned my addr 172.21.15.16:0/1603445279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f62777fe640 1 --2- 172.21.15.16:0/1603445279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d0817350 0x55a0d0bbd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f62767fc640 1 --2- 172.21.15.16:0/1603445279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d07f6670 0x55a0d0bbe200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f6276ffd640 1 -- 172.21.15.16:0/1603445279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d07f6670 msgr2=0x55a0d0bbe200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f6276ffd640 1 --2- 172.21.15.16:0/1603445279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d07f6670 0x55a0d0bbe200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f6276ffd640 1 -- 172.21.15.16:0/1603445279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d0817350 msgr2=0x55a0d0bbd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f6276ffd640 1 --2- 172.21.15.16:0/1603445279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d0817350 0x55a0d0bbd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f6276ffd640 1 -- 172.21.15.16:0/1603445279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0d0882930 con 0x55a0d07ee490 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f62777fe640 1 --2- 172.21.15.16:0/1603445279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d0817350 0x55a0d0bbd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f6276ffd640 1 --2- 172.21.15.16:0/1603445279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bbdcc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f62680e2a70 tx=0x7f62680ed690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3919fb1ecd0431d5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f625ffff640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62600b4ca0 0x7f62600b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f62777fe640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62600b4ca0 0x7f62600b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.741+0000 7f625d7fa640 1 -- 172.21.15.16:0/1603445279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62680f4070 con 0x55a0d07ee490 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.742+0000 7f625d7fa640 1 -- 172.21.15.16:0/1603445279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6268005290 con 0x55a0d07ee490 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.742+0000 7f625d7fa640 1 -- 172.21.15.16:0/1603445279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62680ab870 con 0x55a0d07ee490 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.742+0000 7f625ffff640 1 -- 172.21.15.16:0/2262524449 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f62680e5720 con 0x7f6278102800 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.742+0000 7f6283a81740 1 -- 172.21.15.16:0/1603445279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 msgr2=0x55a0d0bbdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.742+0000 7f6283a81740 1 --2- 172.21.15.16:0/1603445279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bbdcc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f62680e2a70 tx=0x7f62680ed690 comp rx=0 tx=0).stop 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.742+0000 7f6283a81740 1 -- 172.21.15.16:0/1603445279 shutdown_connections 2024-08-28T19:22:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.742+0000 7f6283a81740 1 --2- 172.21.15.16:0/1603445279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d07f6670 0x55a0d0bbe200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.742+0000 7f6283a81740 1 --2- 172.21.15.16:0/1603445279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bbdcc0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.742+0000 7f6283a81740 1 --2- 172.21.15.16:0/1603445279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d0817350 0x55a0d0bbd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.742+0000 7f6283a81740 1 -- 172.21.15.16:0/1603445279 >> 172.21.15.16:0/1603445279 conn(0x55a0d080b740 msgr2=0x55a0d0bbb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.742+0000 7f6283a81740 1 -- 172.21.15.16:0/1603445279 shutdown_connections 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.743+0000 7f6283a81740 1 -- 172.21.15.16:0/1603445279 wait complete. 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.743+0000 7f62777fe640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62600b4ca0 0x7f62600b71b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6270009fd0 tx=0x7f62700097e0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.743+0000 7f6283a81740 1 -- start start 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.743+0000 7f6283a81740 1 Processor -- start 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.743+0000 7f6283a81740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d07ee490 0x55a0d0bc8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.743+0000 7f6283a81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 0x55a0d0bc94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.743+0000 7f6283a81740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d0817350 0x55a0d0bd5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f6283a81740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a0d0985890 con 0x55a0d0817350 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f6283a81740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0d0aa9790 con 0x55a0d07f6670 2024-08-28T19:22:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f6283a81740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a0d0bcbf60 con 0x55a0d07ee490 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f6283a81740 10 client.0 _ll_register_callbacks cb 0x55a0d082eba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f62777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d0817350 0x55a0d0bd5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f62777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d0817350 0x55a0d0bd5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58836/0 (socket says 172.21.15.16:58836) 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f62777fe640 1 -- 172.21.15.16:0/321455335 learned_addr learned my addr 172.21.15.16:0/321455335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f62767fc640 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 0x55a0d0bc94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f6276ffd640 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d07ee490 0x55a0d0bc8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f62777fe640 1 -- 172.21.15.16:0/321455335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d07ee490 msgr2=0x55a0d0bc8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f62777fe640 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d07ee490 0x55a0d0bc8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f62777fe640 1 -- 172.21.15.16:0/321455335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 msgr2=0x55a0d0bc94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f62777fe640 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 0x55a0d0bc94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f62777fe640 1 -- 172.21.15.16:0/321455335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0d0ae0f30 con 0x55a0d0817350 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f62767fc640 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 0x55a0d0bc94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f62777fe640 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d0817350 0x55a0d0bd5250 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6270010980 tx=0x7f6270004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14267da005698692 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f625aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.744+0000 7f625aff5640 1 -- 172.21.15.16:0/321455335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627001e6d0 con 0x55a0d0817350 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f625aff5640 1 -- 172.21.15.16:0/321455335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f627001e870 con 0x55a0d0817350 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f6283a81740 1 -- 172.21.15.16:0/321455335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a0d0b35540 con 0x55a0d0817350 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f625aff5640 1 -- 172.21.15.16:0/321455335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f627001ea70 con 0x55a0d0817350 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f6283a81740 10 client.16230 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f6283a81740 1 -- 172.21.15.16:0/321455335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a0d0bd7da0 con 0x55a0d0817350 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f625aff5640 1 -- 172.21.15.16:0/321455335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a0d0b35540 con 0x55a0d0817350 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f625aff5640 1 -- 172.21.15.16:0/321455335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f62700043a0 con 0x55a0d0817350 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f6283a81740 10 client.16230 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f6283a81740 10 client.16230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f6283a81740 20 client.16230 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f6283a81740 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0d07d5cd0 0x55a0d0bdb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f6283a81740 4 client.16230 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:22:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f6283a81740 1 -- 172.21.15.16:0/321455335 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0d0b30fe0 con 0x55a0d07d5cd0 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.745+0000 7f6276ffd640 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0d07d5cd0 0x55a0d0bdb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.746+0000 7f6276ffd640 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0d07d5cd0 0x55a0d0bdb3b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b662ebed575f007 server_cookie=9167f33db98728f8 in_seq=0 out_seq=0 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.746+0000 7f625aff5640 10 client.16230 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f625aff5640 1 -- 172.21.15.16:0/321455335 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a0d0b30fe0 con 0x55a0d07d5cd0 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f625aff5640 10 client.16230 handle_command_reply: tid=0 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 1 client.16230 shutdown 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 1 -- 172.21.15.16:0/321455335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d0817350 msgr2=0x55a0d0bd5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d0817350 0x55a0d0bd5250 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6270010980 tx=0x7f6270004a40 comp rx=0 tx=0).stop 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 1 -- 172.21.15.16:0/321455335 shutdown_connections 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d0817350 0x55a0d0bd5250 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 0x55a0d0bc94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0d07d5cd0 0x55a0d0bdb3b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 1 --2- 172.21.15.16:0/321455335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d07ee490 0x55a0d0bc8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 1 -- 172.21.15.16:0/321455335 >> 172.21.15.16:0/321455335 conn(0x55a0d080b740 msgr2=0x55a0d0bc0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f625aff5640 0 client.16230 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 1 -- 172.21.15.16:0/321455335 shutdown_connections 2024-08-28T19:22:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 1 -- 172.21.15.16:0/321455335 wait complete. 2024-08-28T19:22:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.747+0000 7f6283a81740 20 client.16230 trim_cache size 0 max 16384 2024-08-28T19:22:58.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6283a81740 1 Processor -- start 2024-08-28T19:22:58.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6283a81740 1 -- start start 2024-08-28T19:22:58.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6283a81740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bc0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6283a81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 0x55a0d0bd5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6283a81740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 0x55a0d0bd5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6283a81740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62781b0cf0 con 0x55a0d07ee490 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6283a81740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62781b28f0 con 0x55a0d07f6670 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6283a81740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f62781b13c0 con 0x55a0d0817350 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6276ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bc0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6276ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bc0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58848/0 (socket says 172.21.15.16:58848) 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6276ffd640 1 -- 172.21.15.16:0/2520156771 learned_addr learned my addr 172.21.15.16:0/2520156771 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f62767fc640 1 --2- 172.21.15.16:0/2520156771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 0x55a0d0bd5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6276ffd640 1 -- 172.21.15.16:0/2520156771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 msgr2=0x55a0d0bd5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f62777fe640 1 --2- 172.21.15.16:0/2520156771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 0x55a0d0bd5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6276ffd640 1 --2- 172.21.15.16:0/2520156771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 0x55a0d0bd5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6276ffd640 1 -- 172.21.15.16:0/2520156771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 msgr2=0x55a0d0bd5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6276ffd640 1 --2- 172.21.15.16:0/2520156771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 0x55a0d0bd5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f6276ffd640 1 -- 172.21.15.16:0/2520156771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0d0bc99b0 con 0x55a0d07ee490 2024-08-28T19:22:58.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f62777fe640 1 --2- 172.21.15.16:0/2520156771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 0x55a0d0bd5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:22:58.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.765+0000 7f62767fc640 1 --2- 172.21.15.16:0/2520156771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 0x55a0d0bd5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:58.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.766+0000 7f6276ffd640 1 --2- 172.21.15.16:0/2520156771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bc0f90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f62680e2bb0 tx=0x7f62680eb550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da4b32db94e707c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:58.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.766+0000 7f625cff9640 1 -- 172.21.15.16:0/2520156771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268052970 con 0x55a0d07ee490 2024-08-28T19:22:58.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.766+0000 7f625cff9640 1 -- 172.21.15.16:0/2520156771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62680ede90 con 0x55a0d07ee490 2024-08-28T19:22:58.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.766+0000 7f625cff9640 1 -- 172.21.15.16:0/2520156771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62680536f0 con 0x55a0d07ee490 2024-08-28T19:22:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.767+0000 7f6283a81740 1 -- 172.21.15.16:0/2520156771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 msgr2=0x55a0d0bc0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.767+0000 7f6283a81740 1 --2- 172.21.15.16:0/2520156771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bc0f90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f62680e2bb0 tx=0x7f62680eb550 comp rx=0 tx=0).stop 2024-08-28T19:22:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.767+0000 7f6283a81740 1 -- 172.21.15.16:0/2520156771 shutdown_connections 2024-08-28T19:22:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.767+0000 7f6283a81740 1 --2- 172.21.15.16:0/2520156771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 0x55a0d0bd5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.767+0000 7f6283a81740 1 --2- 172.21.15.16:0/2520156771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07f6670 0x55a0d0bd5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.767+0000 7f6283a81740 1 --2- 172.21.15.16:0/2520156771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07ee490 0x55a0d0bc0f90 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.767+0000 7f6283a81740 1 -- 172.21.15.16:0/2520156771 >> 172.21.15.16:0/2520156771 conn(0x55a0d080b740 msgr2=0x55a0d0bc0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.767+0000 7f6283a81740 1 -- 172.21.15.16:0/2520156771 shutdown_connections 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.767+0000 7f6283a81740 1 -- 172.21.15.16:0/2520156771 wait complete. 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.767+0000 7f6283a81740 1 -- start start 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.767+0000 7f6283a81740 1 Processor -- start 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f6283a81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07ee490 0x55a0d0bd1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f6283a81740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07f6670 0x55a0d0bd1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f6283a81740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 0x55a0d0bccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f6283a81740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62781b2160 con 0x55a0d07f6670 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f6283a81740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62781b1a90 con 0x55a0d07ee490 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f6283a81740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f62781aecd0 con 0x55a0d0817350 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f6283a81740 10 client.0 _ll_register_callbacks cb 0x7f6270009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f62767fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07f6670 0x55a0d0bd1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f62767fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07f6670 0x55a0d0bd1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58860/0 (socket says 172.21.15.16:58860) 2024-08-28T19:22:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f62767fc640 1 -- 172.21.15.16:0/4017519213 learned_addr learned my addr 172.21.15.16:0/4017519213 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f6276ffd640 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07ee490 0x55a0d0bd1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f62777fe640 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 0x55a0d0bccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f62767fc640 1 -- 172.21.15.16:0/4017519213 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 msgr2=0x55a0d0bccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f62767fc640 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 0x55a0d0bccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f62767fc640 1 -- 172.21.15.16:0/4017519213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07ee490 msgr2=0x55a0d0bd1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f62767fc640 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07ee490 0x55a0d0bd1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f62767fc640 1 -- 172.21.15.16:0/4017519213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0d0bcd400 con 0x55a0d07f6670 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f62777fe640 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 0x55a0d0bccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f62767fc640 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07f6670 0x55a0d0bd1bd0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f626c005ab0 tx=0x7f626c0042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d49a0b824a2f09 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f625aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.768+0000 7f625aff5640 1 -- 172.21.15.16:0/4017519213 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c035040 con 0x55a0d07f6670 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f625aff5640 1 -- 172.21.15.16:0/4017519213 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f626c031b10 con 0x55a0d07f6670 2024-08-28T19:22:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f6283a81740 1 -- 172.21.15.16:0/4017519213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62680ede90 con 0x55a0d07f6670 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f625aff5640 1 -- 172.21.15.16:0/4017519213 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f626c050050 con 0x55a0d07f6670 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f6283a81740 10 client.16236 fetch_fsmap learned FSMap version 31 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f6283a81740 1 -- 172.21.15.16:0/4017519213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a0d0c45920 con 0x55a0d07f6670 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f625aff5640 1 -- 172.21.15.16:0/4017519213 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62680ede90 con 0x55a0d07f6670 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f625aff5640 1 -- 172.21.15.16:0/4017519213 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f626c004a30 con 0x55a0d07f6670 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f6283a81740 10 client.16236 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f6283a81740 10 client.16236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f6283a81740 20 client.16236 populate_metadata read hostname 'smithi016' 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f6283a81740 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0d07ab080 0x55a0d0c48220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f6283a81740 4 client.16236 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.769+0000 7f6283a81740 1 -- 172.21.15.16:0/4017519213 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a0d0b2e720 con 0x55a0d07ab080 2024-08-28T19:22:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.770+0000 7f6276ffd640 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0d07ab080 0x55a0d0c48220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:22:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.770+0000 7f625aff5640 10 client.16236 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.770+0000 7f6276ffd640 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0d07ab080 0x55a0d0c48220 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=176d554c3841274c server_cookie=c9e27b7c40ab856a in_seq=0 out_seq=0 2024-08-28T19:22:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f625aff5640 1 -- 172.21.15.16:0/4017519213 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a0d0b2e720 con 0x55a0d07ab080 2024-08-28T19:22:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f625aff5640 10 client.16236 handle_command_reply: tid=0 2024-08-28T19:22:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 1 client.16236 shutdown 2024-08-28T19:22:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 1 -- 172.21.15.16:0/4017519213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07f6670 msgr2=0x55a0d0bd1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07f6670 0x55a0d0bd1bd0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f626c005ab0 tx=0x7f626c0042e0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 1 -- 172.21.15.16:0/4017519213 shutdown_connections 2024-08-28T19:22:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0d07ab080 0x55a0d0c48220 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0d0817350 0x55a0d0bccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0d07f6670 0x55a0d0bd1bd0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f625aff5640 0 client.16236 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:22:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 1 --2- 172.21.15.16:0/4017519213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0d07ee490 0x55a0d0bd1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 1 -- 172.21.15.16:0/4017519213 >> 172.21.15.16:0/4017519213 conn(0x55a0d080b740 msgr2=0x55a0d0bca230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 1 -- 172.21.15.16:0/4017519213 shutdown_connections 2024-08-28T19:22:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 1 -- 172.21.15.16:0/4017519213 wait complete. 2024-08-28T19:22:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.772+0000 7f6283a81740 20 client.16236 trim_cache size 0 max 16384 2024-08-28T19:22:58.779 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:22:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:22:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4", 2024-08-28T19:22:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:22:58.780 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:22:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 -- 172.21.15.16:0/2262524449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62600b4ca0 msgr2=0x7f62600b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62600b4ca0 0x7f62600b71b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6270009fd0 tx=0x7f62700097e0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 -- 172.21.15.16:0/2262524449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6278102800 msgr2=0x7f627819a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:22:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6278102800 0x7f627819a250 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f626800c890 tx=0x7f626800cd50 comp rx=0 tx=0).stop 2024-08-28T19:22:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 -- 172.21.15.16:0/2262524449 shutdown_connections 2024-08-28T19:22:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6278108a90 0x7f627819fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f62600b4ca0 0x7f62600b71b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6278108140 0x7f627819a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 --2- 172.21.15.16:0/2262524449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6278102800 0x7f627819a250 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:22:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 -- 172.21.15.16:0/2262524449 >> 172.21.15.16:0/2262524449 conn(0x7f62780794d0 msgr2=0x7f6278104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:22:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 -- 172.21.15.16:0/2262524449 shutdown_connections 2024-08-28T19:22:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:22:58.773+0000 7f627d50c640 1 -- 172.21.15.16:0/2262524449 wait complete. 2024-08-28T19:22:58.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71264 10.61 MB/sec warmup 38 sec latency 4227.739 ms 2024-08-28T19:22:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:59 smithi016 ceph-mon[18556]: pgmap v397: 129 pgs: 129 active+clean; 48 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 6.3 MiB/s wr, 203 op/s 2024-08-28T19:22:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:59 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:22:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:59 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:22:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:59 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:59 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:59 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:22:59.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:22:59 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:59 smithi028 ceph-mon[21769]: pgmap v397: 129 pgs: 129 active+clean; 48 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 6.3 MiB/s wr, 203 op/s 2024-08-28T19:22:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:59 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:22:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:59 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:22:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:59 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:59 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:59 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:22:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:22:59 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:59.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:22:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:59 smithi115 ceph-mon[22184]: pgmap v397: 129 pgs: 129 active+clean; 48 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 6.3 MiB/s wr, 203 op/s 2024-08-28T19:22:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:59 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:22:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:59 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:22:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:59 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:59 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:59 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:22:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:22:59 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:22:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:22:59.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 73405 10.53 MB/sec warmup 39 sec latency 14.468 ms 2024-08-28T19:23:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:00 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:23:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:00 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:00 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:23:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:00 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:23:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:00 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:23:00.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:00 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:00 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:23:00.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:00 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:00 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:00 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.345+0000 7fa4e2208640 1 -- 172.21.15.16:0/1974184679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4dc1050a0 msgr2=0x7fa4dc10d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:00.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.345+0000 7fa4e2208640 1 --2- 172.21.15.16:0/1974184679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4dc1050a0 0x7fa4dc10d5f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0009900 tx=0x7fa4d002a410 comp rx=0 tx=0).stop 2024-08-28T19:23:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.345+0000 7fa4e2208640 1 -- 172.21.15.16:0/1974184679 shutdown_connections 2024-08-28T19:23:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.345+0000 7fa4e2208640 1 --2- 172.21.15.16:0/1974184679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4dc10dbb0 0x7fa4dc1100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.345+0000 7fa4e2208640 1 --2- 172.21.15.16:0/1974184679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4dc1050a0 0x7fa4dc10d5f0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.345+0000 7fa4e2208640 1 --2- 172.21.15.16:0/1974184679 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4dc104750 0x7fa4dc104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.345+0000 7fa4e2208640 1 -- 172.21.15.16:0/1974184679 >> 172.21.15.16:0/1974184679 conn(0x7fa4dc0794d0 msgr2=0x7fa4dc0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.345+0000 7fa4e2208640 1 -- 172.21.15.16:0/1974184679 shutdown_connections 2024-08-28T19:23:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.345+0000 7fa4e2208640 1 -- 172.21.15.16:0/1974184679 wait complete. 2024-08-28T19:23:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.346+0000 7fa4e2208640 1 Processor -- start 2024-08-28T19:23:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4e2208640 1 -- start start 2024-08-28T19:23:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4e2208640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4dc104750 0x7fa4dc19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4e2208640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4dc1050a0 0x7fa4dc19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4e2208640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4dc10dbb0 0x7fa4dc19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4e2208640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4dc1137d0 con 0x7fa4dc10dbb0 2024-08-28T19:23:00.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4e2208640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4dc113650 con 0x7fa4dc104750 2024-08-28T19:23:00.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4e2208640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa4dc113950 con 0x7fa4dc1050a0 2024-08-28T19:23:00.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4dc10dbb0 0x7fa4dc19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4dc104750 0x7fa4dc19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4dc10dbb0 0x7fa4dc19fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58888/0 (socket says 172.21.15.16:58888) 2024-08-28T19:23:00.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4dc104750 0x7fa4dc19a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60206/0 (socket says 172.21.15.16:60206) 2024-08-28T19:23:00.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4dbfff640 1 -- 172.21.15.16:0/4063554525 learned_addr learned my addr 172.21.15.16:0/4063554525 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:00.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4daffd640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4dc1050a0 0x7fa4dc19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4dbfff640 1 -- 172.21.15.16:0/4063554525 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4dc1050a0 msgr2=0x7fa4dc19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:00.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4dbfff640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4dc1050a0 0x7fa4dc19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4dbfff640 1 -- 172.21.15.16:0/4063554525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4dc104750 msgr2=0x7fa4dc19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:00.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4dbfff640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4dc104750 0x7fa4dc19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4dbfff640 1 -- 172.21.15.16:0/4063554525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4dc1a0190 con 0x7fa4dc10dbb0 2024-08-28T19:23:00.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.347+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4dc104750 0x7fa4dc19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:00.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.348+0000 7fa4dbfff640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4dc10dbb0 0x7fa4dc19fa60 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa4d400eb10 tx=0x7fa4d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:00.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.348+0000 7fa4d8ff9640 1 -- 172.21.15.16:0/4063554525 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4004b20 con 0x7fa4dc10dbb0 2024-08-28T19:23:00.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.348+0000 7fa4e2208640 1 -- 172.21.15.16:0/4063554525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4dc1a0480 con 0x7fa4dc10dbb0 2024-08-28T19:23:00.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.348+0000 7fa4e2208640 1 -- 172.21.15.16:0/4063554525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4dc1a6be0 con 0x7fa4dc10dbb0 2024-08-28T19:23:00.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.348+0000 7fa4d8ff9640 1 -- 172.21.15.16:0/4063554525 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4d4004cc0 con 0x7fa4dc10dbb0 2024-08-28T19:23:00.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.348+0000 7fa4d8ff9640 1 -- 172.21.15.16:0/4063554525 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4010670 con 0x7fa4dc10dbb0 2024-08-28T19:23:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.351+0000 7fa4e877d740 1 Processor -- start 2024-08-28T19:23:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.351+0000 7fa4d8ff9640 1 -- 172.21.15.16:0/4063554525 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa4d4005020 con 0x7fa4dc10dbb0 2024-08-28T19:23:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.351+0000 7fa4e877d740 1 -- start start 2024-08-28T19:23:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574dd070 0x55b6578822c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657882800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b657882d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4e877d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b6577f9630 con 0x55b6574dd070 2024-08-28T19:23:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4e877d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6577e8bc0 con 0x55b6574b41b0 2024-08-28T19:23:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4e877d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b6577ecb70 con 0x55b6574bc390 2024-08-28T19:23:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574dd070 0x55b6578822c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657882800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574dd070 0x55b6578822c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58898/0 (socket says 172.21.15.16:58898) 2024-08-28T19:23:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4daffd640 1 -- 172.21.15.16:0/1385115774 learned_addr learned my addr 172.21.15.16:0/1385115774 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657882800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60210/0 (socket says 172.21.15.16:60210) 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4daffd640 1 --2- 172.21.15.16:0/1385115774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b657882d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4db7fe640 1 -- 172.21.15.16:0/1385115774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 msgr2=0x55b657882d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/1385115774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b657882d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4db7fe640 1 -- 172.21.15.16:0/1385115774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574dd070 msgr2=0x55b6578822c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/1385115774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574dd070 0x55b6578822c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4db7fe640 1 -- 172.21.15.16:0/1385115774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b657548650 con 0x55b6574b41b0 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.352+0000 7fa4d8ff9640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4c40b4d20 0x7fa4c40b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.353+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/1385115774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657882800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009fa0 tx=0x7fa4cc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da63c66b2f63ce14 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.353+0000 7fa4daffd640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4c40b4d20 0x7fa4c40b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.353+0000 7fa4c1ffb640 1 -- 172.21.15.16:0/1385115774 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc01e070 con 0x55b6574b41b0 2024-08-28T19:23:00.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.353+0000 7fa4c1ffb640 1 -- 172.21.15.16:0/1385115774 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4cc00f9f0 con 0x55b6574b41b0 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.353+0000 7fa4c1ffb640 1 -- 172.21.15.16:0/1385115774 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc019a90 con 0x55b6574b41b0 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.353+0000 7fa4d8ff9640 1 -- 172.21.15.16:0/4063554525 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa4d40e5c00 con 0x7fa4dc10dbb0 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.354+0000 7fa4e877d740 1 -- 172.21.15.16:0/1385115774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 msgr2=0x55b657882800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.354+0000 7fa4e877d740 1 --2- 172.21.15.16:0/1385115774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657882800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009fa0 tx=0x7fa4cc0118d0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.354+0000 7fa4daffd640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4c40b4d20 0x7fa4c40b7230 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00096c0 tx=0x7fa4d0041040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.354+0000 7fa4e877d740 1 -- 172.21.15.16:0/1385115774 shutdown_connections 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.354+0000 7fa4e877d740 1 --2- 172.21.15.16:0/1385115774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b657882d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.354+0000 7fa4e877d740 1 --2- 172.21.15.16:0/1385115774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657882800 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.354+0000 7fa4e877d740 1 --2- 172.21.15.16:0/1385115774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574dd070 0x55b6578822c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.354+0000 7fa4e877d740 1 -- 172.21.15.16:0/1385115774 >> 172.21.15.16:0/1385115774 conn(0x55b6574d1460 msgr2=0x55b6578801b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.354+0000 7fa4e877d740 1 -- 172.21.15.16:0/1385115774 shutdown_connections 2024-08-28T19:23:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.354+0000 7fa4e877d740 1 -- 172.21.15.16:0/1385115774 wait complete. 2024-08-28T19:23:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.354+0000 7fa4e877d740 1 -- start start 2024-08-28T19:23:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4e877d740 1 Processor -- start 2024-08-28T19:23:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574b41b0 0x55b65788da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b65788df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574dd070 0x55b65789e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4e877d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b65764b700 con 0x55b6574b41b0 2024-08-28T19:23:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4e877d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b65776f700 con 0x55b6574dd070 2024-08-28T19:23:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4e877d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b657894eb0 con 0x55b6574bc390 2024-08-28T19:23:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4e877d740 10 client.0 _ll_register_callbacks cb 0x55b65779edc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4daffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b65788df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574dd070 0x55b65789e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574b41b0 0x55b65788da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4daffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b65788df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:51100/0 (socket says 172.21.15.16:51100) 2024-08-28T19:23:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.355+0000 7fa4daffd640 1 -- 172.21.15.16:0/3057752286 learned_addr learned my addr 172.21.15.16:0/3057752286 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4daffd640 1 -- 172.21.15.16:0/3057752286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574dd070 msgr2=0x55b65789e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4daffd640 1 --2- 172.21.15.16:0/3057752286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574dd070 0x55b65789e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4daffd640 1 -- 172.21.15.16:0/3057752286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574b41b0 msgr2=0x55b65788da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4daffd640 1 --2- 172.21.15.16:0/3057752286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574b41b0 0x55b65788da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4daffd640 1 -- 172.21.15.16:0/3057752286 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6577a88b0 con 0x55b6574bc390 2024-08-28T19:23:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4daffd640 1 --2- 172.21.15.16:0/3057752286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b65788df40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa4d002a930 tx=0x7fa4d0031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d70bf1a1463e824 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4baffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:23:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4baffd640 1 -- 172.21.15.16:0/3057752286 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0035040 con 0x55b6574bc390 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4e877d740 1 -- 172.21.15.16:0/3057752286 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4cc00f9f0 con 0x55b6574bc390 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4baffd640 1 -- 172.21.15.16:0/3057752286 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa4d0004460 con 0x55b6574bc390 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4baffd640 1 -- 172.21.15.16:0/3057752286 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa4d005c050 con 0x55b6574bc390 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4e877d740 10 client.25744 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.356+0000 7fa4e877d740 1 -- 172.21.15.16:0/3057752286 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b6578a0c80 con 0x55b6574bc390 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.357+0000 7fa4baffd640 1 -- 172.21.15.16:0/3057752286 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4cc00f9f0 con 0x55b6574bc390 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.357+0000 7fa4baffd640 1 -- 172.21.15.16:0/3057752286 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fa4d0004c60 con 0x55b6574bc390 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.357+0000 7fa4e877d740 10 client.25744 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.357+0000 7fa4e877d740 10 client.25744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.357+0000 7fa4e877d740 20 client.25744 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.357+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3057752286 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b65749b9f0 0x55b6578a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.357+0000 7fa4e877d740 4 client.25744 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:23:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.357+0000 7fa4e877d740 1 -- 172.21.15.16:0/3057752286 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6577fb580 con 0x55b65749b9f0 2024-08-28T19:23:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.357+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/3057752286 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b65749b9f0 0x55b6578a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.358+0000 7fa4baffd640 10 client.25744 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.358+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/3057752286 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b65749b9f0 0x55b6578a42c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2dd28fa9fe970b1 server_cookie=db7eb65949f4b90 in_seq=0 out_seq=0 2024-08-28T19:23:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4baffd640 1 -- 172.21.15.16:0/3057752286 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b6577fb580 con 0x55b65749b9f0 2024-08-28T19:23:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4baffd640 10 client.25744 handle_command_reply: tid=0 2024-08-28T19:23:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 1 client.25744 shutdown 2024-08-28T19:23:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 1 -- 172.21.15.16:0/3057752286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 msgr2=0x55b65788df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3057752286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b65788df40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa4d002a930 tx=0x7fa4d0031400 comp rx=0 tx=0).stop 2024-08-28T19:23:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 1 -- 172.21.15.16:0/3057752286 shutdown_connections 2024-08-28T19:23:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3057752286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574dd070 0x55b65789e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3057752286 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b65749b9f0 0x55b6578a42c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3057752286 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b65788df40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3057752286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574b41b0 0x55b65788da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4baffd640 0 client.25744 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 1 -- 172.21.15.16:0/3057752286 >> 172.21.15.16:0/3057752286 conn(0x55b6574d1460 msgr2=0x55b657885450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 1 -- 172.21.15.16:0/3057752286 shutdown_connections 2024-08-28T19:23:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 1 -- 172.21.15.16:0/3057752286 wait complete. 2024-08-28T19:23:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.359+0000 7fa4e877d740 20 client.25744 trim_cache size 0 max 16384 2024-08-28T19:23:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.378+0000 7fa4e877d740 1 Processor -- start 2024-08-28T19:23:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.378+0000 7fa4e877d740 1 -- start start 2024-08-28T19:23:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.378+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b657883cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.378+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574b41b0 0x55b65788f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.378+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574bc390 0x55b65789e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4e877d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c800fec0 con 0x55b6574d1460 2024-08-28T19:23:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4e877d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c8011ac0 con 0x55b6574bc390 2024-08-28T19:23:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b657883cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4e877d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c8010590 con 0x55b6574b41b0 2024-08-28T19:23:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b657883cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58920/0 (socket says 172.21.15.16:58920) 2024-08-28T19:23:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4db7fe640 1 -- 172.21.15.16:0/3671678022 learned_addr learned my addr 172.21.15.16:0/3671678022 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4dbfff640 1 --2- 172.21.15.16:0/3671678022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574bc390 0x55b65789e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4daffd640 1 --2- 172.21.15.16:0/3671678022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574b41b0 0x55b65788f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4db7fe640 1 -- 172.21.15.16:0/3671678022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574b41b0 msgr2=0x55b65788f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/3671678022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574b41b0 0x55b65788f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4db7fe640 1 -- 172.21.15.16:0/3671678022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574bc390 msgr2=0x55b65789e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/3671678022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574bc390 0x55b65789e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4db7fe640 1 -- 172.21.15.16:0/3671678022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b657894d90 con 0x55b6574d1460 2024-08-28T19:23:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/3671678022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b657883cb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x55b657880f40 tx=0x7fa4cc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6f21674a3957f1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.379+0000 7fa4c17fa640 1 -- 172.21.15.16:0/3671678022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc02a070 con 0x55b6574d1460 2024-08-28T19:23:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.380+0000 7fa4c17fa640 1 -- 172.21.15.16:0/3671678022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4cc00f9f0 con 0x55b6574d1460 2024-08-28T19:23:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.380+0000 7fa4c17fa640 1 -- 172.21.15.16:0/3671678022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc01e040 con 0x55b6574d1460 2024-08-28T19:23:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.380+0000 7fa4e877d740 1 -- 172.21.15.16:0/3671678022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 msgr2=0x55b657883cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.380+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3671678022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b657883cb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x55b657880f40 tx=0x7fa4cc005d50 comp rx=0 tx=0).stop 2024-08-28T19:23:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.380+0000 7fa4e877d740 1 -- 172.21.15.16:0/3671678022 shutdown_connections 2024-08-28T19:23:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.380+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3671678022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574bc390 0x55b65789e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.380+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3671678022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574b41b0 0x55b65788f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.380+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3671678022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b657883cb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.381+0000 7fa4e877d740 1 -- 172.21.15.16:0/3671678022 >> 172.21.15.16:0/3671678022 conn(0x55b65788e0c0 msgr2=0x55b65788ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.381+0000 7fa4e877d740 1 -- 172.21.15.16:0/3671678022 shutdown_connections 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.381+0000 7fa4e877d740 1 -- 172.21.15.16:0/3671678022 wait complete. 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.381+0000 7fa4e877d740 1 -- start start 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.381+0000 7fa4e877d740 1 Processor -- start 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657898860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b657898da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4e877d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b65790dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4e877d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c8011330 con 0x55b6574d1460 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4e877d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c8010c60 con 0x55b6574b41b0 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4e877d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c800dea0 con 0x55b6574bc390 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4e877d740 10 client.0 _ll_register_callbacks cb 0x7fa4d0032e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657898860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b65790dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4daffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b657898da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b65790dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58926/0 (socket says 172.21.15.16:58926) 2024-08-28T19:23:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657898860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60238/0 (socket says 172.21.15.16:60238) 2024-08-28T19:23:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4dbfff640 1 -- 172.21.15.16:0/3741922477 learned_addr learned my addr 172.21.15.16:0/3741922477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4dbfff640 1 -- 172.21.15.16:0/3741922477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 msgr2=0x55b657898da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4dbfff640 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b657898da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4dbfff640 1 -- 172.21.15.16:0/3741922477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 msgr2=0x55b657898860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4dbfff640 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657898860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4dbfff640 1 -- 172.21.15.16:0/3741922477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b65790e210 con 0x55b6574d1460 2024-08-28T19:23:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657898860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4daffd640 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b657898da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4dbfff640 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b65790dcd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40049d0 tx=0x7fa4d40b06e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61c319d918cb804 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4baffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.382+0000 7fa4baffd640 1 -- 172.21.15.16:0/3741922477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4097070 con 0x55b6574d1460 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4baffd640 1 -- 172.21.15.16:0/3741922477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa4d40b5020 con 0x55b6574d1460 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4e877d740 1 -- 172.21.15.16:0/3741922477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4cc00f9f0 con 0x55b6574d1460 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4baffd640 1 -- 172.21.15.16:0/3741922477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa4d40d2030 con 0x55b6574d1460 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4e877d740 10 client.16254 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4e877d740 1 -- 172.21.15.16:0/3741922477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b65790f3e0 con 0x55b6574d1460 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4baffd640 1 -- 172.21.15.16:0/3741922477 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4cc00f9f0 con 0x55b6574d1460 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4baffd640 1 -- 172.21.15.16:0/3741922477 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fa4d40cd320 con 0x55b6574d1460 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4e877d740 10 client.16254 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4e877d740 10 client.16254 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:23:00.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4e877d740 20 client.16254 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:00.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6574dd070 0x55b657911c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:00.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4e877d740 4 client.16254 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:23:00.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.383+0000 7fa4e877d740 1 -- 172.21.15.16:0/3741922477 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b6577f7020 con 0x55b6574dd070 2024-08-28T19:23:00.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6574dd070 0x55b657911c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:00.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4baffd640 10 client.16254 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:00.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4db7fe640 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6574dd070 0x55b657911c30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61f09813ed5f9cd7 server_cookie=de807ce6111182da in_seq=0 out_seq=0 2024-08-28T19:23:00.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4baffd640 1 -- 172.21.15.16:0/3741922477 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1422 (crc 0 0 0) 0x55b6577f7020 con 0x55b6574dd070 2024-08-28T19:23:00.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4baffd640 10 client.16254 handle_command_reply: tid=0 2024-08-28T19:23:00.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4e877d740 1 client.16254 shutdown 2024-08-28T19:23:00.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4e877d740 1 -- 172.21.15.16:0/3741922477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 msgr2=0x55b65790dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b65790dcd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40049d0 tx=0x7fa4d40b06e0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4e877d740 1 -- 172.21.15.16:0/3741922477 shutdown_connections 2024-08-28T19:23:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6574d1460 0x55b65790dcd0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6574dd070 0x55b657911c30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4baffd640 0 client.16254 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6574bc390 0x55b657898da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4e877d740 1 --2- 172.21.15.16:0/3741922477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6574b41b0 0x55b657898860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.384+0000 7fa4e877d740 1 -- 172.21.15.16:0/3741922477 >> 172.21.15.16:0/3741922477 conn(0x55b65788e0c0 msgr2=0x55b65788f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e877d740 1 -- 172.21.15.16:0/3741922477 shutdown_connections 2024-08-28T19:23:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e877d740 1 -- 172.21.15.16:0/3741922477 wait complete. 2024-08-28T19:23:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e877d740 20 client.16254 trim_cache size 0 max 16384 2024-08-28T19:23:00.394 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:23:00.394 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:23:00.394 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:23:00.394 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:23:00.394 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:00.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4", 2024-08-28T19:23:00.394 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:23:00.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:00.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:00.394 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:00.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:00.396 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:23:00.397 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:23:00.397 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:00.397 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:00.397 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:00.397 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:00.397 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:00.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:00.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:00.397 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:23:00.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e2208640 1 -- 172.21.15.16:0/4063554525 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4c40b4d20 msgr2=0x7fa4c40b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:00.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e2208640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4c40b4d20 0x7fa4c40b7230 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00096c0 tx=0x7fa4d0041040 comp rx=0 tx=0).stop 2024-08-28T19:23:00.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e2208640 1 -- 172.21.15.16:0/4063554525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4dc10dbb0 msgr2=0x7fa4dc19fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:00.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e2208640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4dc10dbb0 0x7fa4dc19fa60 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa4d400eb10 tx=0x7fa4d400c2d0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e2208640 1 -- 172.21.15.16:0/4063554525 shutdown_connections 2024-08-28T19:23:00.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e2208640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4dc10dbb0 0x7fa4dc19fa60 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e2208640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4dc1050a0 0x7fa4dc19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e2208640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4c40b4d20 0x7fa4c40b7230 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e2208640 1 --2- 172.21.15.16:0/4063554525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4dc104750 0x7fa4dc19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:00.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.385+0000 7fa4e2208640 1 -- 172.21.15.16:0/4063554525 >> 172.21.15.16:0/4063554525 conn(0x7fa4dc0794d0 msgr2=0x7fa4dc0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:00.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.386+0000 7fa4e2208640 1 -- 172.21.15.16:0/4063554525 shutdown_connections 2024-08-28T19:23:00.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:00.386+0000 7fa4e2208640 1 -- 172.21.15.16:0/4063554525 wait complete. 2024-08-28T19:23:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:00 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:23:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:00 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:00 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:23:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:00 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:23:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:00 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:23:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:00 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:00 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:23:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:00 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:00 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:00 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:00 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:23:00.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:00 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:00 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:23:00.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:00 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:23:00.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:00 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:23:00.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:00 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:00 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:23:00.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:00 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:00 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:00 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:00.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 74538 10.52 MB/sec warmup 40 sec latency 53.327 ms 2024-08-28T19:23:01.127 INFO:tasks.cephfs.filesystem:scrub status for tag:f6d83c9f-bbbd-4891-ba71-cffd5e9780a4 - {'path': '/', 'tag': 'f6d83c9f-bbbd-4891-ba71-cffd5e9780a4', 'options': 'recursive,force'} 2024-08-28T19:23:01.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:01 smithi016 ceph-mon[18556]: pgmap v398: 129 pgs: 129 active+clean; 48 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 5.7 MiB/s wr, 197 op/s 2024-08-28T19:23:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:01 smithi028 ceph-mon[21769]: pgmap v398: 129 pgs: 129 active+clean; 48 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 5.7 MiB/s wr, 197 op/s 2024-08-28T19:23:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:01 smithi115 ceph-mon[22184]: pgmap v398: 129 pgs: 129 active+clean; 48 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 5.7 MiB/s wr, 197 op/s 2024-08-28T19:23:01.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 75781 10.36 MB/sec warmup 41 sec latency 24.014 ms 2024-08-28T19:23:02.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 78378 10.51 MB/sec warmup 42 sec latency 10.095 ms 2024-08-28T19:23:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:03 smithi028 ceph-mon[21769]: pgmap v399: 129 pgs: 129 active+clean; 47 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 3.2 MiB/s wr, 158 op/s 2024-08-28T19:23:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:03 smithi016 ceph-mon[18556]: pgmap v399: 129 pgs: 129 active+clean; 47 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 3.2 MiB/s wr, 158 op/s 2024-08-28T19:23:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:03 smithi115 ceph-mon[22184]: pgmap v399: 129 pgs: 129 active+clean; 47 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 3.2 MiB/s wr, 158 op/s 2024-08-28T19:23:03.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 80476 10.44 MB/sec warmup 43 sec latency 11.032 ms 2024-08-28T19:23:04.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 82084 10.47 MB/sec warmup 44 sec latency 14.474 ms 2024-08-28T19:23:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:05 smithi016 ceph-mon[18556]: pgmap v400: 129 pgs: 129 active+clean; 48 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.0 MiB/s wr, 86 op/s 2024-08-28T19:23:05.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:05 smithi028 ceph-mon[21769]: pgmap v400: 129 pgs: 129 active+clean; 48 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.0 MiB/s wr, 86 op/s 2024-08-28T19:23:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:05 smithi115 ceph-mon[22184]: pgmap v400: 129 pgs: 129 active+clean; 48 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.0 MiB/s wr, 86 op/s 2024-08-28T19:23:05.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 84563 10.52 MB/sec warmup 45 sec latency 11.216 ms 2024-08-28T19:23:06.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 85984 10.44 MB/sec warmup 46 sec latency 16.323 ms 2024-08-28T19:23:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:07 smithi016 ceph-mon[18556]: pgmap v401: 129 pgs: 129 active+clean; 52 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 189 op/s 2024-08-28T19:23:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:07 smithi028 ceph-mon[21769]: pgmap v401: 129 pgs: 129 active+clean; 52 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 189 op/s 2024-08-28T19:23:07.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:07 smithi115 ceph-mon[22184]: pgmap v401: 129 pgs: 129 active+clean; 52 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 189 op/s 2024-08-28T19:23:07.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 88796 10.59 MB/sec warmup 47 sec latency 9.090 ms 2024-08-28T19:23:08.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 90136 10.50 MB/sec warmup 48 sec latency 11.931 ms 2024-08-28T19:23:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:09 smithi016 ceph-mon[18556]: pgmap v402: 129 pgs: 129 active+clean; 53 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 182 op/s 2024-08-28T19:23:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:09 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:23:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:09 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:23:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:09 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:23:09.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:09 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:23:09.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:09 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:23:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:09 smithi028 ceph-mon[21769]: pgmap v402: 129 pgs: 129 active+clean; 53 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 182 op/s 2024-08-28T19:23:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:09 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:23:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:09 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:23:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:09 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:23:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:09 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:23:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:09 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:23:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:09 smithi115 ceph-mon[22184]: pgmap v402: 129 pgs: 129 active+clean; 53 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 182 op/s 2024-08-28T19:23:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:09 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:23:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:09 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:23:09.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:09 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:23:09.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:09 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:23:09.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:09 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:23:09.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 92563 10.57 MB/sec warmup 49 sec latency 10.410 ms 2024-08-28T19:23:10.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 94567 10.50 MB/sec warmup 50 sec latency 15.485 ms 2024-08-28T19:23:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:11 smithi016 ceph-mon[18556]: pgmap v403: 129 pgs: 129 active+clean; 55 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 213 op/s 2024-08-28T19:23:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:11 smithi028 ceph-mon[21769]: pgmap v403: 129 pgs: 129 active+clean; 55 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 213 op/s 2024-08-28T19:23:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:11 smithi115 ceph-mon[22184]: pgmap v403: 129 pgs: 129 active+clean; 55 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 213 op/s 2024-08-28T19:23:11.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 96226 10.54 MB/sec warmup 51 sec latency 19.322 ms 2024-08-28T19:23:12.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 98423 10.49 MB/sec warmup 52 sec latency 15.198 ms 2024-08-28T19:23:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:13 smithi016 ceph-mon[18556]: pgmap v404: 129 pgs: 129 active+clean; 58 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 270 op/s 2024-08-28T19:23:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:13 smithi028 ceph-mon[21769]: pgmap v404: 129 pgs: 129 active+clean; 58 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 270 op/s 2024-08-28T19:23:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:13 smithi115 ceph-mon[22184]: pgmap v404: 129 pgs: 129 active+clean; 58 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 270 op/s 2024-08-28T19:23:13.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100038 10.50 MB/sec warmup 53 sec latency 11.525 ms 2024-08-28T19:23:14.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 102709 10.57 MB/sec warmup 54 sec latency 8.954 ms 2024-08-28T19:23:15.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:15 smithi016 ceph-mon[18556]: pgmap v405: 129 pgs: 129 active+clean; 58 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 240 op/s 2024-08-28T19:23:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:15 smithi028 ceph-mon[21769]: pgmap v405: 129 pgs: 129 active+clean; 58 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 240 op/s 2024-08-28T19:23:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:15 smithi115 ceph-mon[22184]: pgmap v405: 129 pgs: 129 active+clean; 58 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 240 op/s 2024-08-28T19:23:15.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 103968 10.50 MB/sec warmup 55 sec latency 12.924 ms 2024-08-28T19:23:16.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 106570 10.60 MB/sec warmup 56 sec latency 45.084 ms 2024-08-28T19:23:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:17 smithi016 ceph-mon[18556]: pgmap v406: 129 pgs: 129 active+clean; 64 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 334 op/s 2024-08-28T19:23:17.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:17 smithi028 ceph-mon[21769]: pgmap v406: 129 pgs: 129 active+clean; 64 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 334 op/s 2024-08-28T19:23:17.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:17 smithi115 ceph-mon[22184]: pgmap v406: 129 pgs: 129 active+clean; 64 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 334 op/s 2024-08-28T19:23:17.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 107980 10.52 MB/sec warmup 57 sec latency 16.502 ms 2024-08-28T19:23:18.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 110343 10.58 MB/sec warmup 58 sec latency 15.259 ms 2024-08-28T19:23:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:19 smithi016 ceph-mon[18556]: pgmap v407: 129 pgs: 129 active+clean; 65 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 235 op/s 2024-08-28T19:23:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:19 smithi028 ceph-mon[21769]: pgmap v407: 129 pgs: 129 active+clean; 65 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 235 op/s 2024-08-28T19:23:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:19 smithi115 ceph-mon[22184]: pgmap v407: 129 pgs: 129 active+clean; 65 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 235 op/s 2024-08-28T19:23:19.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 112441 10.52 MB/sec warmup 59 sec latency 12.176 ms 2024-08-28T19:23:20.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 114020 10.55 MB/sec warmup 60 sec latency 23.584 ms 2024-08-28T19:23:21.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:21 smithi016 ceph-mon[18556]: pgmap v408: 129 pgs: 129 active+clean; 67 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 259 op/s 2024-08-28T19:23:21.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:21 smithi028 ceph-mon[21769]: pgmap v408: 129 pgs: 129 active+clean; 67 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 259 op/s 2024-08-28T19:23:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:21 smithi115 ceph-mon[22184]: pgmap v408: 129 pgs: 129 active+clean; 67 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 259 op/s 2024-08-28T19:23:21.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 116563 10.58 MB/sec warmup 61 sec latency 11.759 ms 2024-08-28T19:23:22.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 117973 10.53 MB/sec warmup 62 sec latency 16.741 ms 2024-08-28T19:23:23.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:23 smithi016 ceph-mon[18556]: pgmap v409: 129 pgs: 129 active+clean; 70 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 281 op/s 2024-08-28T19:23:23.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:23 smithi028 ceph-mon[21769]: pgmap v409: 129 pgs: 129 active+clean; 70 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 281 op/s 2024-08-28T19:23:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:23 smithi115 ceph-mon[22184]: pgmap v409: 129 pgs: 129 active+clean; 70 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 281 op/s 2024-08-28T19:23:23.895 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.6 2024-08-28T19:23:23.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:23:23.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 120465 10.58 MB/sec warmup 63 sec latency 8.548 ms 2024-08-28T19:23:24.154 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:24.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.678+0000 7fc191b96640 1 -- 172.21.15.16:0/2360426738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc18c103900 msgr2=0x7fc18c103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:24.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.678+0000 7fc191b96640 1 --2- 172.21.15.16:0/2360426738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc18c103900 0x7fc18c103d10 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009900 tx=0x7fc17c02a410 comp rx=0 tx=0).stop 2024-08-28T19:23:24.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.678+0000 7fc191b96640 1 -- 172.21.15.16:0/2360426738 shutdown_connections 2024-08-28T19:23:24.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.678+0000 7fc191b96640 1 --2- 172.21.15.16:0/2360426738 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc18c109b90 0x7fc18c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:24.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.678+0000 7fc191b96640 1 --2- 172.21.15.16:0/2360426738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc18c109240 0x7fc18c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:24.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.678+0000 7fc191b96640 1 --2- 172.21.15.16:0/2360426738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc18c103900 0x7fc18c103d10 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:24.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.678+0000 7fc191b96640 1 -- 172.21.15.16:0/2360426738 >> 172.21.15.16:0/2360426738 conn(0x7fc18c07b5f0 msgr2=0x7fc18c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:24.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.678+0000 7fc191b96640 1 -- 172.21.15.16:0/2360426738 shutdown_connections 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.678+0000 7fc191b96640 1 -- 172.21.15.16:0/2360426738 wait complete. 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.678+0000 7fc191b96640 1 Processor -- start 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc191b96640 1 -- start start 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc191b96640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc18c103900 0x7fc18c0754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc191b96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc18c109240 0x7fc18c0759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc191b96640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc18c109b90 0x7fc18c075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc191b96640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c1138a0 con 0x7fc18c103900 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc191b96640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c113720 con 0x7fc18c109240 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc191b96640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c113a20 con 0x7fc18c109b90 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc18b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc18c103900 0x7fc18c0754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc18b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc18c103900 0x7fc18c0754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47838/0 (socket says 172.21.15.16:47838) 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc18b7fe640 1 -- 172.21.15.16:0/1229655022 learned_addr learned my addr 172.21.15.16:0/1229655022 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc18affd640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc18c109240 0x7fc18c0759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc18bfff640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc18c109b90 0x7fc18c075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc18b7fe640 1 -- 172.21.15.16:0/1229655022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc18c109b90 msgr2=0x7fc18c075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc18b7fe640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc18c109b90 0x7fc18c075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc18b7fe640 1 -- 172.21.15.16:0/1229655022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc18c109240 msgr2=0x7fc18c0759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.679+0000 7fc18b7fe640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc18c109240 0x7fc18c0759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.680+0000 7fc18b7fe640 1 -- 172.21.15.16:0/1229655022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18c1a84d0 con 0x7fc18c103900 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.680+0000 7fc18b7fe640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc18c103900 0x7fc18c0754b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc17c005ab0 tx=0x7fc17c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.680+0000 7fc188ff9640 1 -- 172.21.15.16:0/1229655022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c03f070 con 0x7fc18c103900 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.680+0000 7fc188ff9640 1 -- 172.21.15.16:0/1229655022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc17c031b60 con 0x7fc18c103900 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.680+0000 7fc191b96640 1 -- 172.21.15.16:0/1229655022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc18c1a8760 con 0x7fc18c103900 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.680+0000 7fc188ff9640 1 -- 172.21.15.16:0/1229655022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c035030 con 0x7fc18c103900 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.680+0000 7fc191b96640 1 -- 172.21.15.16:0/1229655022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc18c1a8c40 con 0x7fc18c103900 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.681+0000 7fc191b96640 1 -- 172.21.15.16:0/1229655022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc18c103d10 con 0x7fc18c103900 2024-08-28T19:23:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.682+0000 7fc188ff9640 1 -- 172.21.15.16:0/1229655022 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc17c042070 con 0x7fc18c103900 2024-08-28T19:23:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.683+0000 7fc188ff9640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc1740b4ca0 0x7fc1740b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.683+0000 7fc18affd640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc1740b4ca0 0x7fc1740b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.683+0000 7fc188ff9640 1 -- 172.21.15.16:0/1229655022 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc17c105fd0 con 0x7fc18c103900 2024-08-28T19:23:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.684+0000 7fc18affd640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc1740b4ca0 0x7fc1740b71b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc18c108f80 tx=0x7fc180005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.688+0000 7fc188ff9640 1 -- 172.21.15.16:0/1229655022 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fc17c0cc400 con 0x7fc18c103900 2024-08-28T19:23:24.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 121737 10.52 MB/sec warmup 64 sec latency 45.062 ms 2024-08-28T19:23:24.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.948+0000 7fc191b96640 1 -- 172.21.15.16:0/1229655022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc18c02e430 con 0x7fc18c103900 2024-08-28T19:23:24.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.949+0000 7fc188ff9640 1 -- 172.21.15.16:0/1229655022 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11984 (secure 0 0 0) 0x7fc17c0d0bc0 con 0x7fc18c103900 2024-08-28T19:23:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:23:24.950 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":31,"btime":"2024-08-28T19:22:49:142320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:22:48.274394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:23:24.950 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 31 2024-08-28T19:23:24.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 -- 172.21.15.16:0/1229655022 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc1740b4ca0 msgr2=0x7fc1740b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:24.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc1740b4ca0 0x7fc1740b71b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc18c108f80 tx=0x7fc180005f90 comp rx=0 tx=0).stop 2024-08-28T19:23:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 -- 172.21.15.16:0/1229655022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc18c103900 msgr2=0x7fc18c0754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc18c103900 0x7fc18c0754b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc17c005ab0 tx=0x7fc17c003c10 comp rx=0 tx=0).stop 2024-08-28T19:23:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 -- 172.21.15.16:0/1229655022 shutdown_connections 2024-08-28T19:23:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc18c109b90 0x7fc18c075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc1740b4ca0 0x7fc1740b71b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc18c109240 0x7fc18c0759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 --2- 172.21.15.16:0/1229655022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc18c103900 0x7fc18c0754b0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 -- 172.21.15.16:0/1229655022 >> 172.21.15.16:0/1229655022 conn(0x7fc18c07b5f0 msgr2=0x7fc18c107290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 -- 172.21.15.16:0/1229655022 shutdown_connections 2024-08-28T19:23:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:24.951+0000 7fc191b96640 1 -- 172.21.15.16:0/1229655022 wait complete. 2024-08-28T19:23:25.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:25 smithi016 ceph-mon[18556]: pgmap v410: 129 pgs: 129 active+clean; 70 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-08-28T19:23:25.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:25 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1229655022' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:23:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:25 smithi028 ceph-mon[21769]: pgmap v410: 129 pgs: 129 active+clean; 70 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-08-28T19:23:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:25 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1229655022' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:23:25.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:25 smithi115 ceph-mon[22184]: pgmap v410: 129 pgs: 129 active+clean; 70 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-08-28T19:23:25.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:25 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1229655022' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:23:25.738 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 146.6 --await-for 45.0' 2024-08-28T19:23:25.738 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 146.6 --await-for 45.0 2024-08-28T19:23:25.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124418 10.61 MB/sec warmup 65 sec latency 10.584 ms 2024-08-28T19:23:26.030 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.534+0000 7fdd97618640 1 -- 172.21.15.16:0/3740264159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd90108160 msgr2=0x7fdd90108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.534+0000 7fdd97618640 1 --2- 172.21.15.16:0/3740264159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd90108160 0x7fdd90108570 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fdd80009900 tx=0x7fdd8002a410 comp rx=0 tx=0).stop 2024-08-28T19:23:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.534+0000 7fdd97618640 1 -- 172.21.15.16:0/3740264159 shutdown_connections 2024-08-28T19:23:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.534+0000 7fdd97618640 1 --2- 172.21.15.16:0/3740264159 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90108ab0 0x7fdd9010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.534+0000 7fdd97618640 1 --2- 172.21.15.16:0/3740264159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd90108160 0x7fdd90108570 secure :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fdd80009900 tx=0x7fdd8002a410 comp rx=0 tx=0).stop 2024-08-28T19:23:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.535+0000 7fdd97618640 1 --2- 172.21.15.16:0/3740264159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd90102820 0x7fdd90102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.535+0000 7fdd97618640 1 -- 172.21.15.16:0/3740264159 >> 172.21.15.16:0/3740264159 conn(0x7fdd90079500 msgr2=0x7fdd90079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.535+0000 7fdd97618640 1 -- 172.21.15.16:0/3740264159 shutdown_connections 2024-08-28T19:23:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.535+0000 7fdd97618640 1 -- 172.21.15.16:0/3740264159 wait complete. 2024-08-28T19:23:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.536+0000 7fdd97618640 1 Processor -- start 2024-08-28T19:23:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.536+0000 7fdd97618640 1 -- start start 2024-08-28T19:23:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.537+0000 7fdd97618640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd90102820 0x7fdd9019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.537+0000 7fdd97618640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90108ab0 0x7fdd9019a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.537+0000 7fdd97618640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9019fb70 0x7fdd9019ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.537+0000 7fdd97618640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd90113820 con 0x7fdd90102820 2024-08-28T19:23:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.537+0000 7fdd97618640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd901136a0 con 0x7fdd9019fb70 2024-08-28T19:23:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.537+0000 7fdd97618640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdd901139a0 con 0x7fdd90108ab0 2024-08-28T19:23:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.537+0000 7fdd953b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd90102820 0x7fdd9019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.537+0000 7fdd953b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd90102820 0x7fdd9019a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47858/0 (socket says 172.21.15.16:47858) 2024-08-28T19:23:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.537+0000 7fdd953b4640 1 -- 172.21.15.16:0/4103599468 learned_addr learned my addr 172.21.15.16:0/4103599468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.537+0000 7fdd94bb3640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90108ab0 0x7fdd9019a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.537+0000 7fdd95bb5640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9019fb70 0x7fdd9019ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.538+0000 7fdd953b4640 1 -- 172.21.15.16:0/4103599468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90108ab0 msgr2=0x7fdd9019a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.538+0000 7fdd953b4640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90108ab0 0x7fdd9019a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.538+0000 7fdd953b4640 1 -- 172.21.15.16:0/4103599468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9019fb70 msgr2=0x7fdd9019ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.538+0000 7fdd953b4640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9019fb70 0x7fdd9019ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.538+0000 7fdd953b4640 1 -- 172.21.15.16:0/4103599468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd901a0750 con 0x7fdd90102820 2024-08-28T19:23:26.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.538+0000 7fdd953b4640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd90102820 0x7fdd9019a3f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00e9f0 tx=0x7fdd8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:26.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.538+0000 7fdd867fc640 1 -- 172.21.15.16:0/4103599468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c00cd80 con 0x7fdd90102820 2024-08-28T19:23:26.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.538+0000 7fdd97618640 1 -- 172.21.15.16:0/4103599468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9010f850 con 0x7fdd90102820 2024-08-28T19:23:26.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.538+0000 7fdd97618640 1 -- 172.21.15.16:0/4103599468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd9010fdb0 con 0x7fdd90102820 2024-08-28T19:23:26.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.539+0000 7fdd867fc640 1 -- 172.21.15.16:0/4103599468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd8c0108e0 con 0x7fdd90102820 2024-08-28T19:23:26.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.539+0000 7fdd867fc640 1 -- 172.21.15.16:0/4103599468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c018980 con 0x7fdd90102820 2024-08-28T19:23:26.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.542+0000 7fdd9db8d740 1 Processor -- start 2024-08-28T19:23:26.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.542+0000 7fdd9db8d740 1 -- start start 2024-08-28T19:23:26.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x5612775704a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 0x5612775709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 0x561277570f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5612774dc330 con 0x5612771c0840 2024-08-28T19:23:26.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5612774cb8c0 con 0x561277197980 2024-08-28T19:23:26.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5612774cf870 con 0x56127719fb60 2024-08-28T19:23:26.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd867fc640 1 -- 172.21.15.16:0/4103599468 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fdd8c018b20 con 0x7fdd90102820 2024-08-28T19:23:26.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd94bb3640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 0x561277570f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd94bb3640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 0x561277570f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56024/0 (socket says 172.21.15.16:56024) 2024-08-28T19:23:26.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd94bb3640 1 -- 172.21.15.16:0/2241600040 learned_addr learned my addr 172.21.15.16:0/2241600040 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:26.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd95bb5640 1 --2- 172.21.15.16:0/2241600040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x5612775704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.543+0000 7fdd953b4640 1 --2- 172.21.15.16:0/2241600040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 0x5612775709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.544+0000 7fdd94bb3640 1 -- 172.21.15.16:0/2241600040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 msgr2=0x5612775709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.544+0000 7fdd94bb3640 1 --2- 172.21.15.16:0/2241600040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 0x5612775709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.544+0000 7fdd94bb3640 1 -- 172.21.15.16:0/2241600040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 msgr2=0x5612775704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.544+0000 7fdd94bb3640 1 --2- 172.21.15.16:0/2241600040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x5612775704a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.544+0000 7fdd94bb3640 1 -- 172.21.15.16:0/2241600040 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561277483df0 con 0x56127719fb60 2024-08-28T19:23:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.544+0000 7fdd95bb5640 1 --2- 172.21.15.16:0/2241600040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x5612775704a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.544+0000 7fdd953b4640 1 --2- 172.21.15.16:0/2241600040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 0x5612775709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:23:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.544+0000 7fdd94bb3640 1 --2- 172.21.15.16:0/2241600040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 0x561277570f20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdd80005940 tx=0x7fdd80032b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb80e7eb89f45047 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.544+0000 7fdd7b7fe640 1 -- 172.21.15.16:0/2241600040 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd80040070 con 0x56127719fb60 2024-08-28T19:23:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.544+0000 7fdd7b7fe640 1 -- 172.21.15.16:0/2241600040 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd80004030 con 0x56127719fb60 2024-08-28T19:23:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.545+0000 7fdd7b7fe640 1 -- 172.21.15.16:0/2241600040 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd80035040 con 0x56127719fb60 2024-08-28T19:23:26.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.545+0000 7fdd867fc640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdd7c0b4d20 0x7fdd7c0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.545+0000 7fdd95bb5640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdd7c0b4d20 0x7fdd7c0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.545+0000 7fdd867fc640 1 -- 172.21.15.16:0/4103599468 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdd8c0e5b60 con 0x7fdd90102820 2024-08-28T19:23:26.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2241600040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 msgr2=0x561277570f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:26.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd95bb5640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdd7c0b4d20 0x7fdd7c0b7230 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009fd0 tx=0x7fdd88009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:26.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/2241600040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 0x561277570f20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdd80005940 tx=0x7fdd80032b20 comp rx=0 tx=0).stop 2024-08-28T19:23:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2241600040 shutdown_connections 2024-08-28T19:23:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/2241600040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 0x561277570f20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/2241600040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 0x5612775709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/2241600040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x5612775704a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2241600040 >> 172.21.15.16:0/2241600040 conn(0x5612771b4990 msgr2=0x56127756e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2241600040 shutdown_connections 2024-08-28T19:23:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2241600040 wait complete. 2024-08-28T19:23:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd9db8d740 1 -- start start 2024-08-28T19:23:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.546+0000 7fdd9db8d740 1 Processor -- start 2024-08-28T19:23:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561277197980 0x56127757bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56127719fb60 0x56127757c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612771c0840 0x561277587f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56127732e7a0 con 0x56127719fb60 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561277452240 con 0x5612771c0840 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56127757ec70 con 0x561277197980 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd9db8d740 10 client.0 _ll_register_callbacks cb 0x5612771d8090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd94bb3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56127719fb60 0x56127757c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd94bb3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56127719fb60 0x56127757c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47862/0 (socket says 172.21.15.16:47862) 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd94bb3640 1 -- 172.21.15.16:0/2652285542 learned_addr learned my addr 172.21.15.16:0/2652285542 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd95bb5640 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612771c0840 0x561277587f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd953b4640 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561277197980 0x56127757bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd95bb5640 1 -- 172.21.15.16:0/2652285542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561277197980 msgr2=0x56127757bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:26.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd95bb5640 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561277197980 0x56127757bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd95bb5640 1 -- 172.21.15.16:0/2652285542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56127719fb60 msgr2=0x56127757c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.547+0000 7fdd95bb5640 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56127719fb60 0x56127757c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.548+0000 7fdd95bb5640 1 -- 172.21.15.16:0/2652285542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612774882b0 con 0x5612771c0840 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.548+0000 7fdd95bb5640 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612771c0840 0x561277587f60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009d20 tx=0x7fdd88004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12c37fa776d5f1e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.548+0000 7fdd78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.548+0000 7fdd78ff9640 1 -- 172.21.15.16:0/2652285542 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88003c40 con 0x5612771c0840 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.548+0000 7fdd78ff9640 1 -- 172.21.15.16:0/2652285542 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdd88017030 con 0x5612771c0840 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.548+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2652285542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd80004030 con 0x5612771c0840 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.548+0000 7fdd78ff9640 1 -- 172.21.15.16:0/2652285542 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdd88003de0 con 0x5612771c0840 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.548+0000 7fdd9db8d740 10 client.25898 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.548+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2652285542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56127723ab90 con 0x5612771c0840 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.549+0000 7fdd78ff9640 1 -- 172.21.15.16:0/2652285542 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd80004030 con 0x5612771c0840 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.549+0000 7fdd78ff9640 1 -- 172.21.15.16:0/2652285542 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fdd8801f8d0 con 0x5612771c0840 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.549+0000 7fdd9db8d740 10 client.25898 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.549+0000 7fdd9db8d740 10 client.25898 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:23:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.549+0000 7fdd9db8d740 20 client.25898 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.549+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56127717f1c0 0x56127758dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.549+0000 7fdd9db8d740 4 client.25898 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:23:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.549+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2652285542 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612774de280 con 0x56127717f1c0 2024-08-28T19:23:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.549+0000 7fdd953b4640 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56127717f1c0 0x56127758dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.550+0000 7fdd78ff9640 10 client.25898 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.550+0000 7fdd953b4640 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56127717f1c0 0x56127758dd80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3280658dcd15bc26 server_cookie=7350437667bae248 in_seq=0 out_seq=0 2024-08-28T19:23:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.550+0000 7fdd78ff9640 1 -- 172.21.15.16:0/2652285542 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5612774de280 con 0x56127717f1c0 2024-08-28T19:23:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.550+0000 7fdd78ff9640 10 client.25898 handle_command_reply: tid=0 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.550+0000 7fdd9db8d740 1 client.25898 shutdown 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2652285542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612771c0840 msgr2=0x561277587f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612771c0840 0x561277587f60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009d20 tx=0x7fdd88004ab0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2652285542 shutdown_connections 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612771c0840 0x561277587f60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56127719fb60 0x56127757c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56127717f1c0 0x56127758dd80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/2652285542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561277197980 0x56127757bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2652285542 >> 172.21.15.16:0/2652285542 conn(0x5612771b4990 msgr2=0x561277564340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd78ff9640 0 client.25898 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2652285542 shutdown_connections 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd9db8d740 1 -- 172.21.15.16:0/2652285542 wait complete. 2024-08-28T19:23:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.551+0000 7fdd9db8d740 20 client.25898 trim_cache size 0 max 16384 2024-08-28T19:23:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.569+0000 7fdd9db8d740 1 Processor -- start 2024-08-28T19:23:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.569+0000 7fdd9db8d740 1 -- start start 2024-08-28T19:23:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.569+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561277197980 0x56127757bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.569+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127719fb60 0x56127757d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.569+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x561277587e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.569+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd901af050 con 0x5612771c0840 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.569+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd901b0c50 con 0x56127719fb60 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.569+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdd901af720 con 0x561277197980 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd95bb5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x561277587e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd94bb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127719fb60 0x56127757d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd95bb5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x561277587e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47876/0 (socket says 172.21.15.16:47876) 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd95bb5640 1 -- 172.21.15.16:0/4257400080 learned_addr learned my addr 172.21.15.16:0/4257400080 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd953b4640 1 --2- 172.21.15.16:0/4257400080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561277197980 0x56127757bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd94bb3640 1 -- 172.21.15.16:0/4257400080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561277197980 msgr2=0x56127757bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd94bb3640 1 --2- 172.21.15.16:0/4257400080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561277197980 0x56127757bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd94bb3640 1 -- 172.21.15.16:0/4257400080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 msgr2=0x561277587e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd94bb3640 1 --2- 172.21.15.16:0/4257400080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x561277587e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd94bb3640 1 -- 172.21.15.16:0/4257400080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56127749bd90 con 0x56127719fb60 2024-08-28T19:23:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd953b4640 1 --2- 172.21.15.16:0/4257400080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561277197980 0x56127757bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd94bb3640 1 --2- 172.21.15.16:0/4257400080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127719fb60 0x56127757d850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdd80005940 tx=0x7fdd80005180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4847bac9195229d7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.570+0000 7fdd7affd640 1 -- 172.21.15.16:0/4257400080 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd80040040 con 0x56127719fb60 2024-08-28T19:23:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.571+0000 7fdd7affd640 1 -- 172.21.15.16:0/4257400080 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd80032d10 con 0x56127719fb60 2024-08-28T19:23:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.571+0000 7fdd7affd640 1 -- 172.21.15.16:0/4257400080 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd80035040 con 0x56127719fb60 2024-08-28T19:23:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.571+0000 7fdd9db8d740 1 -- 172.21.15.16:0/4257400080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127719fb60 msgr2=0x56127757d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.571+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/4257400080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127719fb60 0x56127757d850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdd80005940 tx=0x7fdd80005180 comp rx=0 tx=0).stop 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.571+0000 7fdd9db8d740 1 -- 172.21.15.16:0/4257400080 shutdown_connections 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.571+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/4257400080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x561277587e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.571+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/4257400080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127719fb60 0x56127757d850 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.571+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/4257400080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561277197980 0x56127757bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.572+0000 7fdd9db8d740 1 -- 172.21.15.16:0/4257400080 >> 172.21.15.16:0/4257400080 conn(0x5612771b4990 msgr2=0x56127757d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.572+0000 7fdd9db8d740 1 -- 172.21.15.16:0/4257400080 shutdown_connections 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.572+0000 7fdd9db8d740 1 -- 172.21.15.16:0/4257400080 wait complete. 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.572+0000 7fdd9db8d740 1 -- start start 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.572+0000 7fdd9db8d740 1 Processor -- start 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 0x5612775840d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 0x561277584610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd9db8d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x56127757ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd901b04c0 con 0x5612771c0840 2024-08-28T19:23:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd901afdf0 con 0x561277197980 2024-08-28T19:23:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd9db8d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdd901ad030 con 0x56127719fb60 2024-08-28T19:23:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd95bb5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x56127757ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd9db8d740 10 client.0 _ll_register_callbacks cb 0x7fdd88009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd953b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 0x5612775840d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd95bb5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x56127757ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47882/0 (socket says 172.21.15.16:47882) 2024-08-28T19:23:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd95bb5640 1 -- 172.21.15.16:0/3115091615 learned_addr learned my addr 172.21.15.16:0/3115091615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd94bb3640 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 0x561277584610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd953b4640 1 -- 172.21.15.16:0/3115091615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 msgr2=0x561277584610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd953b4640 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 0x561277584610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd953b4640 1 -- 172.21.15.16:0/3115091615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 msgr2=0x56127757ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd953b4640 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x56127757ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.573+0000 7fdd953b4640 1 -- 172.21.15.16:0/3115091615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56127757f470 con 0x561277197980 2024-08-28T19:23:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.574+0000 7fdd953b4640 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 0x5612775840d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0e2f40 tx=0x7fdd8c018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e53146077717d1e6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.574+0000 7fdd78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:23:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.574+0000 7fdd78ff9640 1 -- 172.21.15.16:0/3115091615 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c0b5200 con 0x561277197980 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.574+0000 7fdd78ff9640 1 -- 172.21.15.16:0/3115091615 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdd8c0a8220 con 0x561277197980 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.574+0000 7fdd9db8d740 1 -- 172.21.15.16:0/3115091615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd80032d10 con 0x561277197980 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.574+0000 7fdd78ff9640 1 -- 172.21.15.16:0/3115091615 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdd8c0b03a0 con 0x561277197980 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.574+0000 7fdd9db8d740 10 client.25904 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.574+0000 7fdd9db8d740 1 -- 172.21.15.16:0/3115091615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5612775f8fc0 con 0x561277197980 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.575+0000 7fdd78ff9640 1 -- 172.21.15.16:0/3115091615 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd80032d10 con 0x561277197980 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.575+0000 7fdd78ff9640 1 -- 172.21.15.16:0/3115091615 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fdd8c0ab280 con 0x561277197980 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.575+0000 7fdd9db8d740 10 client.25904 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.575+0000 7fdd9db8d740 10 client.25904 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.575+0000 7fdd9db8d740 20 client.25904 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.575+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561277154570 0x5612775fb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.575+0000 7fdd9db8d740 4 client.25904 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.6, "await_for": 45.0}] 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.575+0000 7fdd9db8d740 1 -- 172.21.15.16:0/3115091615 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.6, "await_for": 45.0}) -- 0x5612774d9d20 con 0x561277154570 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.575+0000 7fdd94bb3640 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561277154570 0x5612775fb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.576+0000 7fdd78ff9640 10 client.25904 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:26.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.576+0000 7fdd94bb3640 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561277154570 0x5612775fb8a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4083e4d0246d33d server_cookie=1eff28a3078dcb36 in_seq=0 out_seq=0 2024-08-28T19:23:26.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd78ff9640 1 -- 172.21.15.16:0/3115091615 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5612774d9d20 con 0x561277154570 2024-08-28T19:23:26.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd78ff9640 10 client.25904 handle_command_reply: tid=0 2024-08-28T19:23:26.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd9db8d740 1 client.25904 shutdown 2024-08-28T19:23:26.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd9db8d740 1 -- 172.21.15.16:0/3115091615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 msgr2=0x5612775840d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:26.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 0x5612775840d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0e2f40 tx=0x7fdd8c018420 comp rx=0 tx=0).stop 2024-08-28T19:23:26.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd9db8d740 1 -- 172.21.15.16:0/3115091615 shutdown_connections 2024-08-28T19:23:26.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612771c0840 0x56127757ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561277154570 0x5612775fb8a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56127719fb60 0x561277584610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd9db8d740 1 --2- 172.21.15.16:0/3115091615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561277197980 0x5612775840d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd78ff9640 0 client.25904 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd9db8d740 1 -- 172.21.15.16:0/3115091615 >> 172.21.15.16:0/3115091615 conn(0x5612771b4990 msgr2=0x56127757ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd9db8d740 1 -- 172.21.15.16:0/3115091615 shutdown_connections 2024-08-28T19:23:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.647+0000 7fdd9db8d740 1 -- 172.21.15.16:0/3115091615 wait complete. 2024-08-28T19:23:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd9db8d740 20 client.25904 trim_cache size 0 max 16384 2024-08-28T19:23:26.650 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 31, 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 6, 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "99a527e6": { 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 6, 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:23:26.651 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 146.6, 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:26.652 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:23:26.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 -- 172.21.15.16:0/4103599468 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdd7c0b4d20 msgr2=0x7fdd7c0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:26.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdd7c0b4d20 0x7fdd7c0b7230 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009fd0 tx=0x7fdd88009500 comp rx=0 tx=0).stop 2024-08-28T19:23:26.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 -- 172.21.15.16:0/4103599468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd90102820 msgr2=0x7fdd9019a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:26.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd90102820 0x7fdd9019a3f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00e9f0 tx=0x7fdd8c00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 -- 172.21.15.16:0/4103599468 shutdown_connections 2024-08-28T19:23:26.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdd7c0b4d20 0x7fdd7c0b7230 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9019fb70 0x7fdd9019ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90108ab0 0x7fdd9019a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 --2- 172.21.15.16:0/4103599468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd90102820 0x7fdd9019a3f0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:26.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 -- 172.21.15.16:0/4103599468 >> 172.21.15.16:0/4103599468 conn(0x7fdd90079500 msgr2=0x7fdd90104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:26.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 -- 172.21.15.16:0/4103599468 shutdown_connections 2024-08-28T19:23:26.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:26.648+0000 7fdd97618640 1 -- 172.21.15.16:0/4103599468 wait complete. 2024-08-28T19:23:26.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 10.47 MB/sec warmup 66 sec latency 354.108 ms 2024-08-28T19:23:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:27 smithi016 ceph-mon[18556]: pgmap v411: 129 pgs: 129 active+clean; 75 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 310 op/s 2024-08-28T19:23:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:27 smithi028 ceph-mon[21769]: pgmap v411: 129 pgs: 129 active+clean; 75 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 310 op/s 2024-08-28T19:23:27.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:27 smithi115 ceph-mon[22184]: pgmap v411: 129 pgs: 129 active+clean; 75 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 310 op/s 2024-08-28T19:23:27.680 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '99a527e6', quiesce took 3.8 seconds. Will release after: 22.8 2024-08-28T19:23:27.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 10.31 MB/sec warmup 67 sec latency 1354.188 ms 2024-08-28T19:23:28.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 10.16 MB/sec warmup 68 sec latency 2354.289 ms 2024-08-28T19:23:29.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:29 smithi016 ceph-mon[18556]: pgmap v412: 129 pgs: 129 active+clean; 75 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 228 op/s 2024-08-28T19:23:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:29 smithi028 ceph-mon[21769]: pgmap v412: 129 pgs: 129 active+clean; 75 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 228 op/s 2024-08-28T19:23:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:29 smithi115 ceph-mon[22184]: pgmap v412: 129 pgs: 129 active+clean; 75 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 228 op/s 2024-08-28T19:23:29.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 10.01 MB/sec warmup 69 sec latency 3354.398 ms 2024-08-28T19:23:30.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 9.87 MB/sec warmup 70 sec latency 4354.461 ms 2024-08-28T19:23:31.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:23:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:31.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:31 smithi016 ceph-mon[18556]: pgmap v413: 129 pgs: 129 active+clean; 77 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 239 op/s 2024-08-28T19:23:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:31 smithi028 ceph-mon[21769]: pgmap v413: 129 pgs: 129 active+clean; 77 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 239 op/s 2024-08-28T19:23:31.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:31 smithi115 ceph-mon[22184]: pgmap v413: 129 pgs: 129 active+clean; 77 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 239 op/s 2024-08-28T19:23:31.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.917+0000 7f437d07a640 1 -- 172.21.15.16:0/1014386782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378109b90 msgr2=0x7f43781103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:31.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.917+0000 7f437d07a640 1 --2- 172.21.15.16:0/1014386782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378109b90 0x7f43781103c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4370009ca0 tx=0x7f437002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.917+0000 7f437d07a640 1 -- 172.21.15.16:0/1014386782 shutdown_connections 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.917+0000 7f437d07a640 1 --2- 172.21.15.16:0/1014386782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378109b90 0x7f43781103c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.917+0000 7f437d07a640 1 --2- 172.21.15.16:0/1014386782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4378109240 0x7f4378109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.917+0000 7f437d07a640 1 --2- 172.21.15.16:0/1014386782 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4378103900 0x7f4378103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.917+0000 7f437d07a640 1 -- 172.21.15.16:0/1014386782 >> 172.21.15.16:0/1014386782 conn(0x7f437807b5f0 msgr2=0x7f437807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.917+0000 7f437d07a640 1 -- 172.21.15.16:0/1014386782 shutdown_connections 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.917+0000 7f437d07a640 1 -- 172.21.15.16:0/1014386782 wait complete. 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.918+0000 7f437d07a640 1 Processor -- start 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.918+0000 7f437d07a640 1 -- start start 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.918+0000 7f437d07a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4378103900 0x7f437819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.918+0000 7f437d07a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4378109240 0x7f437819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.918+0000 7f437d07a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378109b90 0x7f437819fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.918+0000 7f437d07a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4378113850 con 0x7f4378109b90 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.918+0000 7f437d07a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43781136d0 con 0x7f4378109240 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.918+0000 7f437d07a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f43781139d0 con 0x7f4378103900 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.918+0000 7f437759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378109b90 0x7f437819fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f437759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378109b90 0x7f437819fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47892/0 (socket says 172.21.15.16:47892) 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f437759e640 1 -- 172.21.15.16:0/4244881469 learned_addr learned my addr 172.21.15.16:0/4244881469 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f437659c640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4378109240 0x7f437819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f4376d9d640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4378103900 0x7f437819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f437759e640 1 -- 172.21.15.16:0/4244881469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4378103900 msgr2=0x7f437819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f437759e640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4378103900 0x7f437819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f437759e640 1 -- 172.21.15.16:0/4244881469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4378109240 msgr2=0x7f437819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f437759e640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4378109240 0x7f437819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f437759e640 1 -- 172.21.15.16:0/4244881469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43781a00d0 con 0x7f4378109b90 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f437759e640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378109b90 0x7f437819fa60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4370005cc0 tx=0x7f437002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f435ffff640 1 -- 172.21.15.16:0/4244881469 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4370003bf0 con 0x7f4378109b90 2024-08-28T19:23:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f435ffff640 1 -- 172.21.15.16:0/4244881469 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4370003d90 con 0x7f4378109b90 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f437d07a640 1 -- 172.21.15.16:0/4244881469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43781a0360 con 0x7f4378109b90 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f435ffff640 1 -- 172.21.15.16:0/4244881469 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437002fb20 con 0x7f4378109b90 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.919+0000 7f437d07a640 1 -- 172.21.15.16:0/4244881469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43781a08a0 con 0x7f4378109b90 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.922+0000 7f435ffff640 1 -- 172.21.15.16:0/4244881469 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4370004580 con 0x7f4378109b90 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.922+0000 7f43835ef740 1 Processor -- start 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.922+0000 7f43835ef740 1 -- start start 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.922+0000 7f43835ef740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3b2070 0x55da7f7572c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.922+0000 7f43835ef740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3891b0 0x55da7f757800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.922+0000 7f43835ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f757d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.922+0000 7f43835ef740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55da7f6ce630 con 0x55da7f3b2070 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f43835ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55da7f6bdbc0 con 0x55da7f391390 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f43835ef740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55da7f6c1b70 con 0x55da7f3891b0 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3b2070 0x55da7f7572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f435ffff640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43600b4cd0 0x7f43600b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3b2070 0x55da7f7572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47898/0 (socket says 172.21.15.16:47898) 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437659c640 1 -- 172.21.15.16:0/3971178111 learned_addr learned my addr 172.21.15.16:0/3971178111 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f4376d9d640 1 --2- 172.21.15.16:0/3971178111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3891b0 0x55da7f757800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437759e640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43600b4cd0 0x7f43600b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437659c640 1 --2- 172.21.15.16:0/3971178111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f757d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437659c640 1 -- 172.21.15.16:0/3971178111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3891b0 msgr2=0x55da7f757800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f435ffff640 1 -- 172.21.15.16:0/4244881469 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f4370106c90 con 0x7f4378109b90 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437659c640 1 --2- 172.21.15.16:0/3971178111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3891b0 0x55da7f757800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437659c640 1 -- 172.21.15.16:0/3971178111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 msgr2=0x55da7f757d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437659c640 1 --2- 172.21.15.16:0/3971178111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f757d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437659c640 1 -- 172.21.15.16:0/3971178111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da7f41d650 con 0x55da7f3b2070 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437659c640 1 --2- 172.21.15.16:0/3971178111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3b2070 0x55da7f7572c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f436c00af90 tx=0x7f436c00a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbd35b4ab2de7f3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f437759e640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43600b4cd0 0x7f43600b71e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f43700ccf40 tx=0x7f437004e000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f435d7fa640 1 -- 172.21.15.16:0/3971178111 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c019030 con 0x55da7f3b2070 2024-08-28T19:23:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f435d7fa640 1 -- 172.21.15.16:0/3971178111 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f436c0110d0 con 0x55da7f3b2070 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.923+0000 7f435d7fa640 1 -- 172.21.15.16:0/3971178111 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c004c70 con 0x55da7f3b2070 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.924+0000 7f43835ef740 1 -- 172.21.15.16:0/3971178111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3b2070 msgr2=0x55da7f7572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.924+0000 7f43835ef740 1 --2- 172.21.15.16:0/3971178111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3b2070 0x55da7f7572c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f436c00af90 tx=0x7f436c00a780 comp rx=0 tx=0).stop 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.924+0000 7f43835ef740 1 -- 172.21.15.16:0/3971178111 shutdown_connections 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.924+0000 7f43835ef740 1 --2- 172.21.15.16:0/3971178111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f757d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.924+0000 7f43835ef740 1 --2- 172.21.15.16:0/3971178111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3891b0 0x55da7f757800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.924+0000 7f43835ef740 1 --2- 172.21.15.16:0/3971178111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3b2070 0x55da7f7572c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.924+0000 7f43835ef740 1 -- 172.21.15.16:0/3971178111 >> 172.21.15.16:0/3971178111 conn(0x55da7f3a6460 msgr2=0x55da7f7551b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.924+0000 7f43835ef740 1 -- 172.21.15.16:0/3971178111 shutdown_connections 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.924+0000 7f43835ef740 1 -- 172.21.15.16:0/3971178111 wait complete. 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f43835ef740 1 -- start start 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f43835ef740 1 Processor -- start 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f43835ef740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f762a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f43835ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f762f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f43835ef740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3b2070 0x55da7f76edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f43835ef740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55da7f520700 con 0x55da7f3891b0 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f43835ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55da7f644700 con 0x55da7f391390 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f43835ef740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55da7f765aa0 con 0x55da7f3b2070 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f43835ef740 10 client.0 _ll_register_callbacks cb 0x55da7f673dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f437659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f762a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f437659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f762a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47910/0 (socket says 172.21.15.16:47910) 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.925+0000 7f437659c640 1 -- 172.21.15.16:0/1169965812 learned_addr learned my addr 172.21.15.16:0/1169965812 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f4376d9d640 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f762f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f437659c640 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3b2070 0x55da7f76edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f437659c640 1 -- 172.21.15.16:0/1169965812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3b2070 msgr2=0x55da7f76edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f437659c640 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3b2070 0x55da7f76edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f437659c640 1 -- 172.21.15.16:0/1169965812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 msgr2=0x55da7f762f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f437659c640 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f762f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f437659c640 1 -- 172.21.15.16:0/1169965812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da7f67d8b0 con 0x55da7f3891b0 2024-08-28T19:23:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f4376d9d640 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f762f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f437659c640 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f762a20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f436c0058c0 tx=0x7f436c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9357a450706d1373 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f435aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f435aff5640 1 -- 172.21.15.16:0/1169965812 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c019040 con 0x55da7f3891b0 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f435aff5640 1 -- 172.21.15.16:0/1169965812 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f436c005200 con 0x55da7f3891b0 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f43835ef740 1 -- 172.21.15.16:0/1169965812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f436c0110d0 con 0x55da7f3891b0 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f435aff5640 1 -- 172.21.15.16:0/1169965812 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f436c031050 con 0x55da7f3891b0 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f43835ef740 10 client.16290 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.926+0000 7f43835ef740 1 -- 172.21.15.16:0/1169965812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55da7f771840 con 0x55da7f3891b0 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.927+0000 7f435aff5640 1 -- 172.21.15.16:0/1169965812 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f436c0110d0 con 0x55da7f3891b0 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.927+0000 7f435aff5640 1 -- 172.21.15.16:0/1169965812 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f436c01e460 con 0x55da7f3891b0 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.927+0000 7f43835ef740 10 client.16290 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.927+0000 7f43835ef740 10 client.16290 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.927+0000 7f43835ef740 20 client.16290 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.927+0000 7f43835ef740 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55da7f3709f0 0x55da7f774bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.927+0000 7f43835ef740 4 client.16290 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:23:31.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.927+0000 7f43835ef740 1 -- 172.21.15.16:0/1169965812 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da7f6d0580 con 0x55da7f3709f0 2024-08-28T19:23:31.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.927+0000 7f4376d9d640 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55da7f3709f0 0x55da7f774bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.928+0000 7f435aff5640 10 client.16290 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:31.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.928+0000 7f4376d9d640 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55da7f3709f0 0x55da7f774bf0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=979808541e37cb52 server_cookie=d47e90cfb0b3b695 in_seq=0 out_seq=0 2024-08-28T19:23:31.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.928+0000 7f435aff5640 1 -- 172.21.15.16:0/1169965812 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55da7f6d0580 con 0x55da7f3709f0 2024-08-28T19:23:31.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.928+0000 7f435aff5640 10 client.16290 handle_command_reply: tid=0 2024-08-28T19:23:31.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.928+0000 7f43835ef740 1 client.16290 shutdown 2024-08-28T19:23:31.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f43835ef740 1 -- 172.21.15.16:0/1169965812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 msgr2=0x55da7f762a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:31.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f43835ef740 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f762a20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f436c0058c0 tx=0x7f436c005850 comp rx=0 tx=0).stop 2024-08-28T19:23:31.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f43835ef740 1 -- 172.21.15.16:0/1169965812 shutdown_connections 2024-08-28T19:23:31.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f43835ef740 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55da7f3709f0 0x55da7f774bf0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f43835ef740 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3b2070 0x55da7f76edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f43835ef740 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f762f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f435aff5640 0 client.16290 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:31.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f43835ef740 1 --2- 172.21.15.16:0/1169965812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f762a20 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f43835ef740 1 -- 172.21.15.16:0/1169965812 >> 172.21.15.16:0/1169965812 conn(0x55da7f3a6460 msgr2=0x55da7f75a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:31.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f43835ef740 1 -- 172.21.15.16:0/1169965812 shutdown_connections 2024-08-28T19:23:31.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f43835ef740 1 -- 172.21.15.16:0/1169965812 wait complete. 2024-08-28T19:23:31.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.929+0000 7f43835ef740 20 client.16290 trim_cache size 0 max 16384 2024-08-28T19:23:31.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 9.73 MB/sec warmup 71 sec latency 5354.506 ms 2024-08-28T19:23:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f43835ef740 1 Processor -- start 2024-08-28T19:23:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f43835ef740 1 -- start start 2024-08-28T19:23:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f43835ef740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3709f0 0x55da7f75aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f43835ef740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f75fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f43835ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f760150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f43835ef740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f43781b0ae0 con 0x55da7f3891b0 2024-08-28T19:23:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f43835ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43781b26e0 con 0x55da7f391390 2024-08-28T19:23:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f43835ef740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f43781b11b0 con 0x55da7f3709f0 2024-08-28T19:23:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f4376d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f75fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f437659c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3709f0 0x55da7f75aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f4376d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f75fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47924/0 (socket says 172.21.15.16:47924) 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f4376d9d640 1 -- 172.21.15.16:0/1288385514 learned_addr learned my addr 172.21.15.16:0/1288385514 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f437659c640 1 --2- 172.21.15.16:0/1288385514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f760150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f4376d9d640 1 -- 172.21.15.16:0/1288385514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3709f0 msgr2=0x55da7f75aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f4376d9d640 1 --2- 172.21.15.16:0/1288385514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3709f0 0x55da7f75aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f4376d9d640 1 -- 172.21.15.16:0/1288385514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 msgr2=0x55da7f760150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f4376d9d640 1 --2- 172.21.15.16:0/1288385514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f760150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.947+0000 7f4376d9d640 1 -- 172.21.15.16:0/1288385514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da7f761980 con 0x55da7f3891b0 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.948+0000 7f437659c640 1 --2- 172.21.15.16:0/1288385514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3709f0 0x55da7f75aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.948+0000 7f4376d9d640 1 --2- 172.21.15.16:0/1288385514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f75fc10 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4368005f30 tx=0x7f4368005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1f676948fff301 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.948+0000 7f435cff9640 1 -- 172.21.15.16:0/1288385514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4368013190 con 0x55da7f3891b0 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.948+0000 7f435cff9640 1 -- 172.21.15.16:0/1288385514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f436801d8e0 con 0x55da7f3891b0 2024-08-28T19:23:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.948+0000 7f435cff9640 1 -- 172.21.15.16:0/1288385514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43680118a0 con 0x55da7f3891b0 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 -- 172.21.15.16:0/1288385514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 msgr2=0x55da7f75fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 --2- 172.21.15.16:0/1288385514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f75fc10 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4368005f30 tx=0x7f4368005f60 comp rx=0 tx=0).stop 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 -- 172.21.15.16:0/1288385514 shutdown_connections 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 --2- 172.21.15.16:0/1288385514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f391390 0x55da7f760150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 --2- 172.21.15.16:0/1288385514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f3891b0 0x55da7f75fc10 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 --2- 172.21.15.16:0/1288385514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3709f0 0x55da7f75aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 -- 172.21.15.16:0/1288385514 >> 172.21.15.16:0/1288385514 conn(0x55da7f3a6460 msgr2=0x55da7f75a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 -- 172.21.15.16:0/1288385514 shutdown_connections 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 -- 172.21.15.16:0/1288385514 wait complete. 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 -- start start 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 Processor -- start 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3709f0 0x55da7f766da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f3891b0 0x55da7f7672e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f391390 0x55da7f767820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f43781b1f50 con 0x55da7f391390 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43781b1880 con 0x55da7f3891b0 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.949+0000 7f43835ef740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f43781aeac0 con 0x55da7f3709f0 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f43835ef740 10 client.0 _ll_register_callbacks cb 0x7f436800b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f437659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f391390 0x55da7f767820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f4376d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f3891b0 0x55da7f7672e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f437659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f391390 0x55da7f767820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47934/0 (socket says 172.21.15.16:47934) 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f437659c640 1 -- 172.21.15.16:0/3906346308 learned_addr learned my addr 172.21.15.16:0/3906346308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f437659c640 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3709f0 0x55da7f766da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f437659c640 1 -- 172.21.15.16:0/3906346308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3709f0 msgr2=0x55da7f766da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f437659c640 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3709f0 0x55da7f766da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f437659c640 1 -- 172.21.15.16:0/3906346308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f3891b0 msgr2=0x55da7f7672e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f437659c640 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f3891b0 0x55da7f7672e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f437659c640 1 -- 172.21.15.16:0/3906346308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da7f7de520 con 0x55da7f391390 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f4376d9d640 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f3891b0 0x55da7f7672e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f437659c640 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f391390 0x55da7f767820 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f436c016020 tx=0x7f436c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d572f3d5892e40 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f435aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f435aff5640 1 -- 172.21.15.16:0/3906346308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c004860 con 0x55da7f391390 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f43835ef740 1 -- 172.21.15.16:0/3906346308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f436801d8e0 con 0x55da7f391390 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.950+0000 7f435aff5640 1 -- 172.21.15.16:0/3906346308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f436c01eaa0 con 0x55da7f391390 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f435aff5640 1 -- 172.21.15.16:0/3906346308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f436c046050 con 0x55da7f391390 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f43835ef740 10 client.16296 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f43835ef740 1 -- 172.21.15.16:0/3906346308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55da7f7df7b0 con 0x55da7f391390 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f435aff5640 1 -- 172.21.15.16:0/3906346308 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f436801d8e0 con 0x55da7f391390 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f435aff5640 1 -- 172.21.15.16:0/3906346308 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f436c004ea0 con 0x55da7f391390 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f43835ef740 10 client.16296 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f43835ef740 10 client.16296 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:23:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f43835ef740 20 client.16296 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f43835ef740 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55da7f3b2070 0x55da7f7e1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f43835ef740 4 client.16296 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f43835ef740 1 -- 172.21.15.16:0/3906346308 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55da7f6cc020 con 0x55da7f3b2070 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.951+0000 7f4376d9d640 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55da7f3b2070 0x55da7f7e1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f435aff5640 10 client.16296 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f4376d9d640 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55da7f3b2070 0x55da7f7e1e80 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b81e4b04b937eec server_cookie=5fc895315454dda8 in_seq=0 out_seq=0 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f435aff5640 1 -- 172.21.15.16:0/3906346308 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1226 (crc 0 0 0) 0x55da7f6cc020 con 0x55da7f3b2070 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f435aff5640 10 client.16296 handle_command_reply: tid=0 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 1 client.16296 shutdown 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 1 -- 172.21.15.16:0/3906346308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f391390 msgr2=0x55da7f767820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f391390 0x55da7f767820 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f436c016020 tx=0x7f436c005850 comp rx=0 tx=0).stop 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 1 -- 172.21.15.16:0/3906346308 shutdown_connections 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da7f391390 0x55da7f767820 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da7f3891b0 0x55da7f7672e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55da7f3b2070 0x55da7f7e1e80 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f435aff5640 0 client.16296 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 1 --2- 172.21.15.16:0/3906346308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55da7f3709f0 0x55da7f766da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 1 -- 172.21.15.16:0/3906346308 >> 172.21.15.16:0/3906346308 conn(0x55da7f3a6460 msgr2=0x55da7f763e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 1 -- 172.21.15.16:0/3906346308 shutdown_connections 2024-08-28T19:23:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 1 -- 172.21.15.16:0/3906346308 wait complete. 2024-08-28T19:23:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.952+0000 7f43835ef740 20 client.16296 trim_cache size 0 max 16384 2024-08-28T19:23:31.958 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:23:31.958 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:23:31.958 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:23:31.958 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:31.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:31.961 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:31.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:31.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:31.961 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:23:31.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 -- 172.21.15.16:0/4244881469 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43600b4cd0 msgr2=0x7f43600b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:31.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43600b4cd0 0x7f43600b71e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f43700ccf40 tx=0x7f437004e000 comp rx=0 tx=0).stop 2024-08-28T19:23:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 -- 172.21.15.16:0/4244881469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378109b90 msgr2=0x7f437819fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378109b90 0x7f437819fa60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4370005cc0 tx=0x7f437002f380 comp rx=0 tx=0).stop 2024-08-28T19:23:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 -- 172.21.15.16:0/4244881469 shutdown_connections 2024-08-28T19:23:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378109b90 0x7f437819fa60 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43600b4cd0 0x7f43600b71e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4378109240 0x7f437819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 --2- 172.21.15.16:0/4244881469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4378103900 0x7f437819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 -- 172.21.15.16:0/4244881469 >> 172.21.15.16:0/4244881469 conn(0x7f437807b5f0 msgr2=0x7f4378105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 -- 172.21.15.16:0/4244881469 shutdown_connections 2024-08-28T19:23:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:31.953+0000 7f437d07a640 1 -- 172.21.15.16:0/4244881469 wait complete. 2024-08-28T19:23:32.854 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:23:32.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:23:32.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 9.59 MB/sec warmup 72 sec latency 6354.551 ms 2024-08-28T19:23:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:33.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:33 smithi016 ceph-mon[18556]: pgmap v414: 129 pgs: 129 active+clean; 78 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 225 op/s 2024-08-28T19:23:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.526+0000 7ff51106a640 1 -- 172.21.15.16:0/3216232480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff50c108a90 msgr2=0x7ff50c10f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.526+0000 7ff51106a640 1 --2- 172.21.15.16:0/3216232480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff50c108a90 0x7ff50c10f2c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff504009ca0 tx=0x7ff50402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:23:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.526+0000 7ff51106a640 1 -- 172.21.15.16:0/3216232480 shutdown_connections 2024-08-28T19:23:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.526+0000 7ff51106a640 1 --2- 172.21.15.16:0/3216232480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff50c108a90 0x7ff50c10f2c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.526+0000 7ff51106a640 1 --2- 172.21.15.16:0/3216232480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c108140 0x7ff50c108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.526+0000 7ff51106a640 1 --2- 172.21.15.16:0/3216232480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff50c102800 0x7ff50c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.526+0000 7ff51106a640 1 -- 172.21.15.16:0/3216232480 >> 172.21.15.16:0/3216232480 conn(0x7ff50c0794d0 msgr2=0x7ff50c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.526+0000 7ff51106a640 1 -- 172.21.15.16:0/3216232480 shutdown_connections 2024-08-28T19:23:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.526+0000 7ff51106a640 1 -- 172.21.15.16:0/3216232480 wait complete. 2024-08-28T19:23:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.527+0000 7ff51106a640 1 Processor -- start 2024-08-28T19:23:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.527+0000 7ff51106a640 1 -- start start 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.527+0000 7ff51106a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff50c102800 0x7ff50c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.527+0000 7ff51106a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff50c108140 0x7ff50c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.527+0000 7ff51106a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c108a90 0x7ff50c19fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.527+0000 7ff51106a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c112710 con 0x7ff50c102800 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.527+0000 7ff51106a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c112590 con 0x7ff50c108a90 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.527+0000 7ff51106a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c112890 con 0x7ff50c108140 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.527+0000 7ff50ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff50c102800 0x7ff50c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.527+0000 7ff50ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff50c102800 0x7ff50c19a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47946/0 (socket says 172.21.15.16:47946) 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.527+0000 7ff50ad9d640 1 -- 172.21.15.16:0/80991903 learned_addr learned my addr 172.21.15.16:0/80991903 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff50a59c640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff50c108140 0x7ff50c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff50b59e640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c108a90 0x7ff50c19fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff50ad9d640 1 -- 172.21.15.16:0/80991903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff50c108140 msgr2=0x7ff50c19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff50ad9d640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff50c108140 0x7ff50c19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff50ad9d640 1 -- 172.21.15.16:0/80991903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c108a90 msgr2=0x7ff50c19fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff50ad9d640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c108a90 0x7ff50c19fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff50ad9d640 1 -- 172.21.15.16:0/80991903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50c1a0080 con 0x7ff50c102800 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff50ad9d640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff50c102800 0x7ff50c19a1d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00d770 tx=0x7ff4fc00db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff4f3fff640 1 -- 172.21.15.16:0/80991903 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc017070 con 0x7ff50c102800 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff51106a640 1 -- 172.21.15.16:0/80991903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff50c1a0370 con 0x7ff50c102800 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff4f3fff640 1 -- 172.21.15.16:0/80991903 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4fc009820 con 0x7ff50c102800 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff4f3fff640 1 -- 172.21.15.16:0/80991903 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc004bc0 con 0x7ff50c102800 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.528+0000 7ff51106a640 1 -- 172.21.15.16:0/80991903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff50c1a6b90 con 0x7ff50c102800 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.529+0000 7ff51106a640 1 -- 172.21.15.16:0/80991903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff50c103a80 con 0x7ff50c102800 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.531+0000 7ff4f3fff640 1 -- 172.21.15.16:0/80991903 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff4fc0099d0 con 0x7ff50c102800 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.531+0000 7ff4f3fff640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4f40b4cd0 0x7ff4f40b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.532+0000 7ff50a59c640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4f40b4cd0 0x7ff4f40b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:33.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.532+0000 7ff4f3fff640 1 -- 172.21.15.16:0/80991903 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff4fc0e5780 con 0x7ff50c102800 2024-08-28T19:23:33.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.532+0000 7ff50a59c640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4f40b4cd0 0x7ff4f40b71e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff50c102350 tx=0x7ff50000a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:33.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.535+0000 7ff4f3fff640 1 -- 172.21.15.16:0/80991903 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ff4fc0ac200 con 0x7ff50c102800 2024-08-28T19:23:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:33 smithi028 ceph-mon[21769]: pgmap v414: 129 pgs: 129 active+clean; 78 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 225 op/s 2024-08-28T19:23:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:33 smithi115 ceph-mon[22184]: pgmap v414: 129 pgs: 129 active+clean; 78 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 225 op/s 2024-08-28T19:23:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.770+0000 7ff51106a640 1 -- 172.21.15.16:0/80991903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff50c10b8f0 con 0x7ff50c102800 2024-08-28T19:23:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.771+0000 7ff4f3fff640 1 -- 172.21.15.16:0/80991903 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11984 (secure 0 0 0) 0x7ff4fc0abb00 con 0x7ff50c102800 2024-08-28T19:23:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:23:33.773 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":31,"btime":"2024-08-28T19:22:49:142320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:22:48.274394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:23:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 31 2024-08-28T19:23:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 -- 172.21.15.16:0/80991903 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4f40b4cd0 msgr2=0x7ff4f40b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4f40b4cd0 0x7ff4f40b71e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff50c102350 tx=0x7ff50000a2d0 comp rx=0 tx=0).stop 2024-08-28T19:23:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 -- 172.21.15.16:0/80991903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff50c102800 msgr2=0x7ff50c19a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff50c102800 0x7ff50c19a1d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00d770 tx=0x7ff4fc00db70 comp rx=0 tx=0).stop 2024-08-28T19:23:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 -- 172.21.15.16:0/80991903 shutdown_connections 2024-08-28T19:23:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4f40b4cd0 0x7ff4f40b71e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c108a90 0x7ff50c19fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff50c108140 0x7ff50c19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 --2- 172.21.15.16:0/80991903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff50c102800 0x7ff50c19a1d0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 -- 172.21.15.16:0/80991903 >> 172.21.15.16:0/80991903 conn(0x7ff50c0794d0 msgr2=0x7ff50c06aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 -- 172.21.15.16:0/80991903 shutdown_connections 2024-08-28T19:23:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:33.773+0000 7ff51106a640 1 -- 172.21.15.16:0/80991903 wait complete. 2024-08-28T19:23:33.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 9.46 MB/sec warmup 73 sec latency 7354.595 ms 2024-08-28T19:23:34.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:34 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/80991903' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:23:34.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:23:34.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:34 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/80991903' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:23:34.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:34 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/80991903' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:23:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:34.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 9.33 MB/sec warmup 74 sec latency 8354.630 ms 2024-08-28T19:23:35.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.264+0000 7fbdd9848640 1 -- 172.21.15.16:0/3692183752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd4104770 msgr2=0x7fbdd4104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:35.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.264+0000 7fbdd9848640 1 --2- 172.21.15.16:0/3692183752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd4104770 0x7fbdd4104b80 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009900 tx=0x7fbdc402a410 comp rx=0 tx=0).stop 2024-08-28T19:23:35.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.264+0000 7fbdd9848640 1 -- 172.21.15.16:0/3692183752 shutdown_connections 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.264+0000 7fbdd9848640 1 --2- 172.21.15.16:0/3692183752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd410dbd0 0x7fbdd41100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.264+0000 7fbdd9848640 1 --2- 172.21.15.16:0/3692183752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd41050c0 0x7fbdd410d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.264+0000 7fbdd9848640 1 --2- 172.21.15.16:0/3692183752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd4104770 0x7fbdd4104b80 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.264+0000 7fbdd9848640 1 -- 172.21.15.16:0/3692183752 >> 172.21.15.16:0/3692183752 conn(0x7fbdd40794e0 msgr2=0x7fbdd40798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.264+0000 7fbdd9848640 1 -- 172.21.15.16:0/3692183752 shutdown_connections 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.264+0000 7fbdd9848640 1 -- 172.21.15.16:0/3692183752 wait complete. 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.264+0000 7fbdd9848640 1 Processor -- start 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd9848640 1 -- start start 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd9848640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd4104770 0x7fbdd419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd9848640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd41050c0 0x7fbdd419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd9848640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd410dbd0 0x7fbdd419fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd9848640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd4113830 con 0x7fbdd4104770 2024-08-28T19:23:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd9848640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd41136b0 con 0x7fbdd410dbd0 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd9848640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd41139b0 con 0x7fbdd41050c0 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd4104770 0x7fbdd419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd4104770 0x7fbdd419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45204/0 (socket says 172.21.15.16:45204) 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/241369432 learned_addr learned my addr 172.21.15.16:0/241369432 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd37fe640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd410dbd0 0x7fbdd419fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd41050c0 0x7fbdd419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/241369432 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd41050c0 msgr2=0x7fbdd419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd41050c0 0x7fbdd419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/241369432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd410dbd0 msgr2=0x7fbdd419fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd410dbd0 0x7fbdd419fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/241369432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd41a0210 con 0x7fbdd4104770 2024-08-28T19:23:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd41050c0 0x7fbdd419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.265+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd4104770 0x7fbdd419a2a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4032fd0 tx=0x7fbdc4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.266+0000 7fbdd8846640 1 -- 172.21.15.16:0/241369432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc403f030 con 0x7fbdd4104770 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.266+0000 7fbdd8846640 1 -- 172.21.15.16:0/241369432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdc4004490 con 0x7fbdd4104770 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.266+0000 7fbdd9848640 1 -- 172.21.15.16:0/241369432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd41a04a0 con 0x7fbdd4104770 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.266+0000 7fbdd8846640 1 -- 172.21.15.16:0/241369432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4035070 con 0x7fbdd4104770 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.266+0000 7fbdd9848640 1 -- 172.21.15.16:0/241369432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd41a6c60 con 0x7fbdd4104770 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbddfdbd740 1 Processor -- start 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbddfdbd740 1 -- start start 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 0x56460e592350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 0x56460e592890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e592dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56460e509680 con 0x56460e1cc390 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56460e4f8c10 con 0x56460e1c41b0 2024-08-28T19:23:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56460e4fcbc0 con 0x56460e1ed070 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbdd8846640 1 -- 172.21.15.16:0/241369432 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fbdc4042030 con 0x7fbdd4104770 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbdd27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e592dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbdd27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e592dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45216/0 (socket says 172.21.15.16:45216) 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.269+0000 7fbdd27fc640 1 -- 172.21.15.16:0/2915281246 learned_addr learned my addr 172.21.15.16:0/2915281246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd37fe640 1 --2- 172.21.15.16:0/2915281246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 0x56460e592350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/2915281246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 0x56460e592890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd27fc640 1 -- 172.21.15.16:0/2915281246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 msgr2=0x56460e592350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/2915281246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 0x56460e592350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd27fc640 1 -- 172.21.15.16:0/2915281246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 msgr2=0x56460e592890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/2915281246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 0x56460e592890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd27fc640 1 -- 172.21.15.16:0/2915281246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56460e258650 con 0x56460e1cc390 2024-08-28T19:23:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd37fe640 1 --2- 172.21.15.16:0/2915281246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 0x56460e592350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/2915281246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 0x56460e592890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/2915281246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e592dd0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800bcd0 tx=0x7fbdc800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb50ffb2f8464641 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd8846640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdbc0b4d20 0x7fbdbc0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdb9ffb640 1 -- 172.21.15.16:0/2915281246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8015070 con 0x56460e1cc390 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd37fe640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdbc0b4d20 0x7fbdbc0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdb9ffb640 1 -- 172.21.15.16:0/2915281246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdc80118f0 con 0x56460e1cc390 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdd8846640 1 -- 172.21.15.16:0/241369432 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fbdc4105d40 con 0x7fbdd4104770 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.270+0000 7fbdb9ffb640 1 -- 172.21.15.16:0/2915281246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8010930 con 0x56460e1cc390 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.271+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2915281246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 msgr2=0x56460e592dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.271+0000 7fbdd37fe640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdbc0b4d20 0x7fbdbc0b7230 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc005fd0 tx=0x7fbdcc009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.271+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/2915281246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e592dd0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800bcd0 tx=0x7fbdc800f730 comp rx=0 tx=0).stop 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.271+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2915281246 shutdown_connections 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.271+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/2915281246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e592dd0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.271+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/2915281246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 0x56460e592890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.271+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/2915281246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 0x56460e592350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.271+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2915281246 >> 172.21.15.16:0/2915281246 conn(0x56460e1e1460 msgr2=0x56460e590240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.271+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2915281246 shutdown_connections 2024-08-28T19:23:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.271+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2915281246 wait complete. 2024-08-28T19:23:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.272+0000 7fbddfdbd740 1 -- start start 2024-08-28T19:23:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.272+0000 7fbddfdbd740 1 Processor -- start 2024-08-28T19:23:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.272+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 0x56460e59da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.272+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e59df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.272+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 0x56460e5a9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.272+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56460e35b700 con 0x56460e1cc390 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.272+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56460e47f750 con 0x56460e1c41b0 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.272+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56460e5a0b30 con 0x56460e1ed070 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbddfdbd740 10 client.0 _ll_register_callbacks cb 0x56460e4aee10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e59df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e59df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45228/0 (socket says 172.21.15.16:45228) 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd27fc640 1 -- 172.21.15.16:0/3251984059 learned_addr learned my addr 172.21.15.16:0/3251984059 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 0x56460e59da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd37fe640 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 0x56460e5a9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd27fc640 1 -- 172.21.15.16:0/3251984059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 msgr2=0x56460e5a9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 0x56460e5a9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd27fc640 1 -- 172.21.15.16:0/3251984059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 msgr2=0x56460e59da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 0x56460e59da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd27fc640 1 -- 172.21.15.16:0/3251984059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56460e4b8900 con 0x56460e1cc390 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd37fe640 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 0x56460e5a9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 0x56460e59da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e59df50 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fbdc80104b0 tx=0x7fbdc8011670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e43712ea286089 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/3251984059 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8028060 con 0x56460e1cc390 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/3251984059 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fbdc8011cb0 con 0x56460e1cc390 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.273+0000 7fbddfdbd740 1 -- 172.21.15.16:0/3251984059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdc80118f0 con 0x56460e1cc390 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/3251984059 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbdc802f050 con 0x56460e1cc390 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbddfdbd740 10 client.16314 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbddfdbd740 1 -- 172.21.15.16:0/3251984059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56460e5ac750 con 0x56460e1cc390 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/3251984059 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdc80118f0 con 0x56460e1cc390 2024-08-28T19:23:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/3251984059 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fbdc8004990 con 0x56460e1cc390 2024-08-28T19:23:35.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbddfdbd740 10 client.16314 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:35.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbddfdbd740 10 client.16314 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:23:35.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbddfdbd740 20 client.16314 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:35.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56460e1ab9f0 0x56460e5afab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbddfdbd740 4 client.16314 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:23:35.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbddfdbd740 1 -- 172.21.15.16:0/3251984059 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56460e50b5d0 con 0x56460e1ab9f0 2024-08-28T19:23:35.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.274+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56460e1ab9f0 0x56460e5afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.275+0000 7fbdb2ffd640 10 client.16314 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.275+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56460e1ab9f0 0x56460e5afab0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=37f4d55675f1ab33 server_cookie=a1b70540e69a3936 in_seq=0 out_seq=0 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/3251984059 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56460e50b5d0 con 0x56460e1ab9f0 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbdb2ffd640 10 client.16314 handle_command_reply: tid=0 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 1 client.16314 shutdown 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 1 -- 172.21.15.16:0/3251984059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 msgr2=0x56460e59df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e59df50 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fbdc80104b0 tx=0x7fbdc8011670 comp rx=0 tx=0).stop 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 1 -- 172.21.15.16:0/3251984059 shutdown_connections 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56460e1ab9f0 0x56460e5afab0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1ed070 0x56460e5a9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1cc390 0x56460e59df50 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbdb2ffd640 0 client.16314 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:23:35.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/3251984059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1c41b0 0x56460e59da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 1 -- 172.21.15.16:0/3251984059 >> 172.21.15.16:0/3251984059 conn(0x56460e1e1460 msgr2=0x56460e5954e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:35.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 1 -- 172.21.15.16:0/3251984059 shutdown_connections 2024-08-28T19:23:35.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 1 -- 172.21.15.16:0/3251984059 wait complete. 2024-08-28T19:23:35.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.276+0000 7fbddfdbd740 20 client.16314 trim_cache size 0 max 16384 2024-08-28T19:23:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.290+0000 7fbddfdbd740 1 Processor -- start 2024-08-28T19:23:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.290+0000 7fbddfdbd740 1 -- start start 2024-08-28T19:23:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.290+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5a9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.290+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1cc390 0x56460e59ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.290+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1ed070 0x56460e59b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.290+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd41b0c30 con 0x56460e1c41b0 2024-08-28T19:23:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.290+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd41b2830 con 0x56460e1ed070 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.290+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd41b1300 con 0x56460e1cc390 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.290+0000 7fbdd2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5a9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.290+0000 7fbdd2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5a9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45236/0 (socket says 172.21.15.16:45236) 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.290+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/1254391999 learned_addr learned my addr 172.21.15.16:0/1254391999 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdd37fe640 1 --2- 172.21.15.16:0/1254391999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1ed070 0x56460e59b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/1254391999 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1cc390 0x56460e59ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/1254391999 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1cc390 msgr2=0x56460e59ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/1254391999 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1cc390 0x56460e59ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/1254391999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1ed070 msgr2=0x56460e59b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/1254391999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1ed070 0x56460e59b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/1254391999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56460e59ecd0 con 0x56460e1c41b0 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdd37fe640 1 --2- 172.21.15.16:0/1254391999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1ed070 0x56460e59b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/1254391999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5a9dc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40cc050 tx=0x7fbdc4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d4db3631b707ef server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdb97fa640 1 -- 172.21.15.16:0/1254391999 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4049040 con 0x56460e1c41b0 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdb97fa640 1 -- 172.21.15.16:0/1254391999 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdc40cc0c0 con 0x56460e1c41b0 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.291+0000 7fbdb97fa640 1 -- 172.21.15.16:0/1254391999 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc403bc20 con 0x56460e1c41b0 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.292+0000 7fbddfdbd740 1 -- 172.21.15.16:0/1254391999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 msgr2=0x56460e5a9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.292+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/1254391999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5a9dc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40cc050 tx=0x7fbdc4009260 comp rx=0 tx=0).stop 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.292+0000 7fbddfdbd740 1 -- 172.21.15.16:0/1254391999 shutdown_connections 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.292+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/1254391999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1ed070 0x56460e59b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.292+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/1254391999 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1cc390 0x56460e59ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.292+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/1254391999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5a9dc0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.292+0000 7fbddfdbd740 1 -- 172.21.15.16:0/1254391999 >> 172.21.15.16:0/1254391999 conn(0x56460e1e1460 msgr2=0x56460e59c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.292+0000 7fbddfdbd740 1 -- 172.21.15.16:0/1254391999 shutdown_connections 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.292+0000 7fbddfdbd740 1 -- 172.21.15.16:0/1254391999 wait complete. 2024-08-28T19:23:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbddfdbd740 1 -- start start 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbddfdbd740 1 Processor -- start 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1cc390 0x56460e5b1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbddfdbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1ed070 0x56460e5b1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd41b20a0 con 0x56460e1c41b0 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd41b19d0 con 0x56460e1ed070 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbddfdbd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd41aec10 con 0x56460e1cc390 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbddfdbd740 10 client.0 _ll_register_callbacks cb 0x7fbdc800bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbdd2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbdd2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45238/0 (socket says 172.21.15.16:45238) 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/2162974694 learned_addr learned my addr 172.21.15.16:0/2162974694 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.293+0000 7fbdd37fe640 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1ed070 0x56460e5b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1cc390 0x56460e5b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/2162974694 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1cc390 msgr2=0x56460e5b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1cc390 0x56460e5b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/2162974694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1ed070 msgr2=0x56460e5b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1ed070 0x56460e5b1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdd2ffd640 1 -- 172.21.15.16:0/2162974694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56460e619330 con 0x56460e1c41b0 2024-08-28T19:23:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdd2ffd640 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5b0bc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4103720 tx=0x7fbdc403b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5151c4e4d9657335 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/2162974694 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40b7070 con 0x56460e1c41b0 2024-08-28T19:23:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/2162974694 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fbdc40c8350 con 0x56460e1c41b0 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2162974694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdc40cc0c0 con 0x56460e1c41b0 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/2162974694 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbdc4049040 con 0x56460e1c41b0 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbddfdbd740 10 client.16320 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2162974694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56460e61a590 con 0x56460e1c41b0 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.294+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/2162974694 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdc40cc0c0 con 0x56460e1c41b0 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.295+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/2162974694 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7fbdc40b7210 con 0x56460e1c41b0 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.295+0000 7fbddfdbd740 10 client.16320 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.295+0000 7fbddfdbd740 10 client.16320 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.295+0000 7fbddfdbd740 20 client.16320 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.295+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56460e180da0 0x56460e61cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.295+0000 7fbddfdbd740 4 client.16320 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.295+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2162974694 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56460e507070 con 0x56460e180da0 2024-08-28T19:23:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.295+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56460e180da0 0x56460e61cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.295+0000 7fbdb2ffd640 10 client.16320 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.295+0000 7fbdd27fc640 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56460e180da0 0x56460e61cad0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d43d3eebb0ea5e6a server_cookie=bdf92ef86d138d63 in_seq=0 out_seq=0 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbdb2ffd640 1 -- 172.21.15.16:0/2162974694 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56460e507070 con 0x56460e180da0 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbdb2ffd640 10 client.16320 handle_command_reply: tid=0 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 1 client.16320 shutdown 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2162974694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 msgr2=0x56460e5b0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5b0bc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4103720 tx=0x7fbdc403b690 comp rx=0 tx=0).stop 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2162974694 shutdown_connections 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56460e180da0 0x56460e61cad0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56460e1ed070 0x56460e5b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56460e1cc390 0x56460e5b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 1 --2- 172.21.15.16:0/2162974694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56460e1c41b0 0x56460e5b0bc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2162974694 >> 172.21.15.16:0/2162974694 conn(0x56460e1e1460 msgr2=0x56460e59eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:35.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbdb2ffd640 0 client.16320 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:23:35.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2162974694 shutdown_connections 2024-08-28T19:23:35.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 1 -- 172.21.15.16:0/2162974694 wait complete. 2024-08-28T19:23:35.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.296+0000 7fbddfdbd740 20 client.16320 trim_cache size 0 max 16384 2024-08-28T19:23:35.306 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:23:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 -- 172.21.15.16:0/241369432 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdbc0b4d20 msgr2=0x7fbdbc0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdbc0b4d20 0x7fbdbc0b7230 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc005fd0 tx=0x7fbdcc009600 comp rx=0 tx=0).stop 2024-08-28T19:23:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 -- 172.21.15.16:0/241369432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd4104770 msgr2=0x7fbdd419a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd4104770 0x7fbdd419a2a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4032fd0 tx=0x7fbdc4004270 comp rx=0 tx=0).stop 2024-08-28T19:23:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 -- 172.21.15.16:0/241369432 shutdown_connections 2024-08-28T19:23:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdbc0b4d20 0x7fbdbc0b7230 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd410dbd0 0x7fbdd419fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd41050c0 0x7fbdd419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 --2- 172.21.15.16:0/241369432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd4104770 0x7fbdd419a2a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 -- 172.21.15.16:0/241369432 >> 172.21.15.16:0/241369432 conn(0x7fbdd40794e0 msgr2=0x7fbdd40ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 -- 172.21.15.16:0/241369432 shutdown_connections 2024-08-28T19:23:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:35.297+0000 7fbdd9848640 1 -- 172.21.15.16:0/241369432 wait complete. 2024-08-28T19:23:35.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:35 smithi016 ceph-mon[18556]: pgmap v415: 129 pgs: 129 active+clean; 78 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 946 KiB/s wr, 160 op/s 2024-08-28T19:23:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:35 smithi028 ceph-mon[21769]: pgmap v415: 129 pgs: 129 active+clean; 78 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 946 KiB/s wr, 160 op/s 2024-08-28T19:23:35.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:35 smithi115 ceph-mon[22184]: pgmap v415: 129 pgs: 129 active+clean; 78 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 946 KiB/s wr, 160 op/s 2024-08-28T19:23:35.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 9.21 MB/sec warmup 75 sec latency 9354.732 ms 2024-08-28T19:23:36.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:23:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.828+0000 7f7a53552640 1 -- 172.21.15.16:0/2465902851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c108140 msgr2=0x7f7a4c108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.828+0000 7f7a53552640 1 --2- 172.21.15.16:0/2465902851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c108140 0x7f7a4c108550 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009900 tx=0x7f7a3c02a410 comp rx=0 tx=0).stop 2024-08-28T19:23:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.828+0000 7f7a53552640 1 -- 172.21.15.16:0/2465902851 shutdown_connections 2024-08-28T19:23:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.828+0000 7f7a53552640 1 --2- 172.21.15.16:0/2465902851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4c108a90 0x7f7a4c10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.828+0000 7f7a53552640 1 --2- 172.21.15.16:0/2465902851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c108140 0x7f7a4c108550 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.828+0000 7f7a53552640 1 --2- 172.21.15.16:0/2465902851 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a4c102800 0x7f7a4c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.828+0000 7f7a53552640 1 -- 172.21.15.16:0/2465902851 >> 172.21.15.16:0/2465902851 conn(0x7f7a4c0794d0 msgr2=0x7f7a4c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.828+0000 7f7a53552640 1 -- 172.21.15.16:0/2465902851 shutdown_connections 2024-08-28T19:23:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.828+0000 7f7a53552640 1 -- 172.21.15.16:0/2465902851 wait complete. 2024-08-28T19:23:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a53552640 1 Processor -- start 2024-08-28T19:23:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a53552640 1 -- start start 2024-08-28T19:23:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a53552640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c102800 0x7f7a4c199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a53552640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4c108140 0x7f7a4c1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a53552640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a4c108a90 0x7f7a4c19e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a53552640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c1126e0 con 0x7f7a4c102800 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a53552640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c112560 con 0x7f7a4c108140 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a53552640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c112860 con 0x7f7a4c108a90 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a512ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c102800 0x7f7a4c199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a512ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c102800 0x7f7a4c199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45260/0 (socket says 172.21.15.16:45260) 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a512ee640 1 -- 172.21.15.16:0/1975357407 learned_addr learned my addr 172.21.15.16:0/1975357407 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a50aed640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4c108140 0x7f7a4c1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a51aef640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a4c108a90 0x7f7a4c19e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a512ee640 1 -- 172.21.15.16:0/1975357407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a4c108a90 msgr2=0x7f7a4c19e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a512ee640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a4c108a90 0x7f7a4c19e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a512ee640 1 -- 172.21.15.16:0/1975357407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4c108140 msgr2=0x7f7a4c1996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a512ee640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4c108140 0x7f7a4c1996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a512ee640 1 -- 172.21.15.16:0/1975357407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4c19f080 con 0x7f7a4c102800 2024-08-28T19:23:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a51aef640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a4c108a90 0x7f7a4c19e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:23:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.829+0000 7f7a50aed640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4c108140 0x7f7a4c1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:23:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.830+0000 7f7a512ee640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c102800 0x7f7a4c199110 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7a4800e9f0 tx=0x7f7a4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.830+0000 7f7a427fc640 1 -- 172.21.15.16:0/1975357407 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4800cd80 con 0x7f7a4c102800 2024-08-28T19:23:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.830+0000 7f7a427fc640 1 -- 172.21.15.16:0/1975357407 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a480054d0 con 0x7f7a4c102800 2024-08-28T19:23:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.830+0000 7f7a427fc640 1 -- 172.21.15.16:0/1975357407 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a48010690 con 0x7f7a4c102800 2024-08-28T19:23:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.830+0000 7f7a53552640 1 -- 172.21.15.16:0/1975357407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a4c19f370 con 0x7f7a4c102800 2024-08-28T19:23:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.830+0000 7f7a53552640 1 -- 172.21.15.16:0/1975357407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a4c06ac00 con 0x7f7a4c102800 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a59ac7740 1 Processor -- start 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a59ac7740 1 -- start start 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d6440dc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d0e1b0 0x55d6440dc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440dcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d644053680 con 0x55d643d37070 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d644042c10 con 0x55d643d16390 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d644046bc0 con 0x55d643d0e1b0 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a427fc640 1 -- 172.21.15.16:0/1975357407 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7a48004e90 con 0x7f7a4c102800 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a51aef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d6440dc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a51aef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d6440dc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45264/0 (socket says 172.21.15.16:45264) 2024-08-28T19:23:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a51aef640 1 -- 172.21.15.16:0/3701181530 learned_addr learned my addr 172.21.15.16:0/3701181530 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a50aed640 1 --2- 172.21.15.16:0/3701181530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440dcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.833+0000 7f7a51aef640 1 -- 172.21.15.16:0/3701181530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d0e1b0 msgr2=0x55d6440dc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a512ee640 1 --2- 172.21.15.16:0/3701181530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d0e1b0 0x55d6440dc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a51aef640 1 --2- 172.21.15.16:0/3701181530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d0e1b0 0x55d6440dc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a51aef640 1 -- 172.21.15.16:0/3701181530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 msgr2=0x55d6440dcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a51aef640 1 --2- 172.21.15.16:0/3701181530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440dcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a51aef640 1 -- 172.21.15.16:0/3701181530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d643da2650 con 0x55d643d37070 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a512ee640 1 --2- 172.21.15.16:0/3701181530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d0e1b0 0x55d6440dc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a51aef640 1 --2- 172.21.15.16:0/3701181530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d6440dc310 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009ea0 tx=0x7f7a44009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41140e9e1384e21 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a427fc640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a380b4ca0 0x7f7a380b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a377fe640 1 -- 172.21.15.16:0/3701181530 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a44004d20 con 0x55d643d37070 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a377fe640 1 -- 172.21.15.16:0/3701181530 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a440054d0 con 0x55d643d37070 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a377fe640 1 -- 172.21.15.16:0/3701181530 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a440117c0 con 0x55d643d37070 2024-08-28T19:23:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a512ee640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a380b4ca0 0x7f7a380b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.834+0000 7f7a427fc640 1 -- 172.21.15.16:0/1975357407 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7a480e5e90 con 0x7f7a4c102800 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.835+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3701181530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 msgr2=0x55d6440dc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.835+0000 7f7a512ee640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a380b4ca0 0x7f7a380b71b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55d6440daf90 tx=0x7f7a48004a90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.835+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/3701181530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d6440dc310 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009ea0 tx=0x7f7a44009f90 comp rx=0 tx=0).stop 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.835+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3701181530 shutdown_connections 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.835+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/3701181530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440dcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.835+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/3701181530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d0e1b0 0x55d6440dc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.835+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/3701181530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d6440dc310 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.835+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3701181530 >> 172.21.15.16:0/3701181530 conn(0x55d643d2b460 msgr2=0x55d6440da200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.835+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3701181530 shutdown_connections 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.835+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3701181530 wait complete. 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.835+0000 7f7a59ac7740 1 -- start start 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a59ac7740 1 Processor -- start 2024-08-28T19:23:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d16390 0x55d6440e8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d37070 0x55d6440f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d643ea5700 con 0x55d643d0e1b0 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d643fc9750 con 0x55d643d37070 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d6440eaaf0 con 0x55d643d16390 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a59ac7740 10 client.0 _ll_register_callbacks cb 0x55d643ff8e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a51aef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a51aef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45266/0 (socket says 172.21.15.16:45266) 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a51aef640 1 -- 172.21.15.16:0/2027920580 learned_addr learned my addr 172.21.15.16:0/2027920580 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a50aed640 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d16390 0x55d6440e8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.836+0000 7f7a51aef640 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d37070 0x55d6440f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a51aef640 1 -- 172.21.15.16:0/2027920580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d16390 msgr2=0x55d6440e8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a51aef640 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d16390 0x55d6440e8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a51aef640 1 -- 172.21.15.16:0/2027920580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d37070 msgr2=0x55d6440f3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a51aef640 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d37070 0x55d6440f3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a51aef640 1 -- 172.21.15.16:0/2027920580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d644002900 con 0x55d643d0e1b0 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a50aed640 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d16390 0x55d6440e8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a51aef640 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e7b10 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7a440054a0 tx=0x7f7a4400ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327b72fc43872992 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a34ff9640 1 -- 172.21.15.16:0/2027920580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4401e070 con 0x55d643d0e1b0 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a59ac7740 1 -- 172.21.15.16:0/2027920580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a440054d0 con 0x55d643d0e1b0 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a34ff9640 1 -- 172.21.15.16:0/2027920580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7a44011740 con 0x55d643d0e1b0 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a34ff9640 1 -- 172.21.15.16:0/2027920580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7a44011940 con 0x55d643d0e1b0 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a59ac7740 10 client.16332 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a59ac7740 1 -- 172.21.15.16:0/2027920580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d6440f6890 con 0x55d643d0e1b0 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.837+0000 7f7a34ff9640 1 -- 172.21.15.16:0/2027920580 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a440054d0 con 0x55d643d0e1b0 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.838+0000 7f7a34ff9640 1 -- 172.21.15.16:0/2027920580 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f7a44019a10 con 0x55d643d0e1b0 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.838+0000 7f7a59ac7740 10 client.16332 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.838+0000 7f7a59ac7740 10 client.16332 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.838+0000 7f7a59ac7740 20 client.16332 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.838+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55d643cf59f0 0x55d6440f9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.838+0000 7f7a59ac7740 4 client.16332 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.838+0000 7f7a59ac7740 1 -- 172.21.15.16:0/2027920580 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6440555d0 con 0x55d643cf59f0 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.838+0000 7f7a50aed640 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55d643cf59f0 0x55d6440f9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.838+0000 7f7a50aed640 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55d643cf59f0 0x55d6440f9bf0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a8a64678d622c283 server_cookie=3dd82f7180c58e0b in_seq=0 out_seq=0 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.838+0000 7f7a34ff9640 10 client.16332 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.839+0000 7f7a34ff9640 1 -- 172.21.15.16:0/2027920580 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d6440555d0 con 0x55d643cf59f0 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.839+0000 7f7a34ff9640 10 client.16332 handle_command_reply: tid=0 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.839+0000 7f7a59ac7740 1 client.16332 shutdown 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.839+0000 7f7a59ac7740 1 -- 172.21.15.16:0/2027920580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 msgr2=0x55d6440e7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.839+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e7b10 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7a440054a0 tx=0x7f7a4400ff30 comp rx=0 tx=0).stop 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.839+0000 7f7a59ac7740 1 -- 172.21.15.16:0/2027920580 shutdown_connections 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.839+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55d643cf59f0 0x55d6440f9bf0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.839+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d37070 0x55d6440f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.839+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d16390 0x55d6440e8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.839+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/2027920580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e7b10 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.839+0000 7f7a59ac7740 1 -- 172.21.15.16:0/2027920580 >> 172.21.15.16:0/2027920580 conn(0x55d643d2b460 msgr2=0x55d6440df4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.840+0000 7f7a34ff9640 0 client.16332 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:23:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.840+0000 7f7a59ac7740 1 -- 172.21.15.16:0/2027920580 shutdown_connections 2024-08-28T19:23:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.840+0000 7f7a59ac7740 1 -- 172.21.15.16:0/2027920580 wait complete. 2024-08-28T19:23:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.840+0000 7f7a59ac7740 20 client.16332 trim_cache size 0 max 16384 2024-08-28T19:23:36.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.853+0000 7f7a59ac7740 1 Processor -- start 2024-08-28T19:23:36.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.853+0000 7f7a59ac7740 1 -- start start 2024-08-28T19:23:36.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.853+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.853+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.853+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d37070 0x55d6440f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.853+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c1b0b90 con 0x55d643d0e1b0 2024-08-28T19:23:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.853+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c1b2790 con 0x55d643d16390 2024-08-28T19:23:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.853+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c1b1260 con 0x55d643d37070 2024-08-28T19:23:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a51aef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a51aef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45270/0 (socket says 172.21.15.16:45270) 2024-08-28T19:23:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a51aef640 1 -- 172.21.15.16:0/4182442172 learned_addr learned my addr 172.21.15.16:0/4182442172 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a50aed640 1 --2- 172.21.15.16:0/4182442172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a51aef640 1 -- 172.21.15.16:0/4182442172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d37070 msgr2=0x55d6440f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a51aef640 1 --2- 172.21.15.16:0/4182442172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d37070 0x55d6440f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a51aef640 1 -- 172.21.15.16:0/4182442172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 msgr2=0x55d6440f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a51aef640 1 --2- 172.21.15.16:0/4182442172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a51aef640 1 -- 172.21.15.16:0/4182442172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6440e6010 con 0x55d643d0e1b0 2024-08-28T19:23:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a51aef640 1 --2- 172.21.15.16:0/4182442172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e80f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7a4401c040 tx=0x7f7a44009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158fc7134e6edde8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a36ffd640 1 -- 172.21.15.16:0/4182442172 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4403a020 con 0x55d643d0e1b0 2024-08-28T19:23:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a36ffd640 1 -- 172.21.15.16:0/4182442172 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a44025040 con 0x55d643d0e1b0 2024-08-28T19:23:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.854+0000 7f7a36ffd640 1 -- 172.21.15.16:0/4182442172 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4401e040 con 0x55d643d0e1b0 2024-08-28T19:23:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.855+0000 7f7a59ac7740 1 -- 172.21.15.16:0/4182442172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 msgr2=0x55d6440e80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.855+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/4182442172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e80f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7a4401c040 tx=0x7f7a44009f90 comp rx=0 tx=0).stop 2024-08-28T19:23:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.855+0000 7f7a59ac7740 1 -- 172.21.15.16:0/4182442172 shutdown_connections 2024-08-28T19:23:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.855+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/4182442172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d37070 0x55d6440f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.855+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/4182442172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.855+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/4182442172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d0e1b0 0x55d6440e80f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.855+0000 7f7a59ac7740 1 -- 172.21.15.16:0/4182442172 >> 172.21.15.16:0/4182442172 conn(0x55d643d2b460 msgr2=0x55d6440e6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:36.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.855+0000 7f7a59ac7740 1 -- 172.21.15.16:0/4182442172 shutdown_connections 2024-08-28T19:23:36.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.855+0000 7f7a59ac7740 1 -- 172.21.15.16:0/4182442172 wait complete. 2024-08-28T19:23:36.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.855+0000 7f7a59ac7740 1 -- start start 2024-08-28T19:23:36.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a59ac7740 1 Processor -- start 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d0e1b0 0x55d6440fa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440face0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a59ac7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d64415b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c1b2000 con 0x55d643d37070 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c1b1930 con 0x55d643d16390 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a59ac7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c1aeb70 con 0x55d643d0e1b0 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a59ac7740 10 client.0 _ll_register_callbacks cb 0x7f7a44009cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a51aef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d64415b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a51aef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d64415b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45282/0 (socket says 172.21.15.16:45282) 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a51aef640 1 -- 172.21.15.16:0/3481251669 learned_addr learned my addr 172.21.15.16:0/3481251669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a50aed640 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440face0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a51aef640 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d0e1b0 0x55d6440fa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a51aef640 1 -- 172.21.15.16:0/3481251669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d0e1b0 msgr2=0x55d6440fa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a51aef640 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d0e1b0 0x55d6440fa740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a51aef640 1 -- 172.21.15.16:0/3481251669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 msgr2=0x55d6440face0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a51aef640 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440face0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.856+0000 7f7a51aef640 1 -- 172.21.15.16:0/3481251669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d64415ba00 con 0x55d643d37070 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a51aef640 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d64415b390 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7a440056e0 tx=0x7f7a44005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70b2abd001cd0c3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a34ff9640 1 -- 172.21.15.16:0/3481251669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4403a070 con 0x55d643d37070 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a34ff9640 1 -- 172.21.15.16:0/3481251669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7a44010520 con 0x55d643d37070 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3481251669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a44025040 con 0x55d643d37070 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a34ff9640 1 -- 172.21.15.16:0/3481251669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7a4400fbc0 con 0x55d643d37070 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a59ac7740 10 client.16338 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3481251669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d644163f00 con 0x55d643d37070 2024-08-28T19:23:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a34ff9640 1 -- 172.21.15.16:0/3481251669 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a44025040 con 0x55d643d37070 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a34ff9640 1 -- 172.21.15.16:0/3481251669 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f7a44010d10 con 0x55d643d37070 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a59ac7740 10 client.16338 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a59ac7740 10 client.16338 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.857+0000 7f7a59ac7740 20 client.16338 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.858+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55d643ccada0 0x55d644166510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.858+0000 7f7a59ac7740 4 client.16338 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.858+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3481251669 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d644051070 con 0x55d643ccada0 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.858+0000 7f7a50aed640 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55d643ccada0 0x55d644166510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.858+0000 7f7a34ff9640 10 client.16338 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.858+0000 7f7a50aed640 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55d643ccada0 0x55d644166510 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=80504e03af24a6ac server_cookie=75b588386b300ad1 in_seq=0 out_seq=0 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a34ff9640 1 -- 172.21.15.16:0/3481251669 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d644051070 con 0x55d643ccada0 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a34ff9640 10 client.16338 handle_command_reply: tid=0 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 1 client.16338 shutdown 2024-08-28T19:23:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3481251669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 msgr2=0x55d64415b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d64415b390 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7a440056e0 tx=0x7f7a44005fd0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3481251669 shutdown_connections 2024-08-28T19:23:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55d643ccada0 0x55d644166510 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d643d37070 0x55d64415b390 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a34ff9640 0 client.16338 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:23:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d643d16390 0x55d6440face0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 1 --2- 172.21.15.16:0/3481251669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d643d0e1b0 0x55d6440fa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3481251669 >> 172.21.15.16:0/3481251669 conn(0x55d643d2b460 msgr2=0x55d6440df3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3481251669 shutdown_connections 2024-08-28T19:23:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 1 -- 172.21.15.16:0/3481251669 wait complete. 2024-08-28T19:23:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a59ac7740 20 client.16338 trim_cache size 0 max 16384 2024-08-28T19:23:36.866 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:23:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a53552640 1 -- 172.21.15.16:0/1975357407 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a380b4ca0 msgr2=0x7f7a380b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a53552640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a380b4ca0 0x7f7a380b71b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55d6440daf90 tx=0x7f7a48004a90 comp rx=0 tx=0).stop 2024-08-28T19:23:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a53552640 1 -- 172.21.15.16:0/1975357407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c102800 msgr2=0x7f7a4c199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.859+0000 7f7a53552640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c102800 0x7f7a4c199110 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7a4800e9f0 tx=0x7f7a4800edf0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.860+0000 7f7a53552640 1 -- 172.21.15.16:0/1975357407 shutdown_connections 2024-08-28T19:23:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.860+0000 7f7a53552640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a4c108a90 0x7f7a4c19e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.860+0000 7f7a53552640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7a380b4ca0 0x7f7a380b71b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.860+0000 7f7a53552640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4c108140 0x7f7a4c1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.860+0000 7f7a53552640 1 --2- 172.21.15.16:0/1975357407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c102800 0x7f7a4c199110 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.860+0000 7f7a53552640 1 -- 172.21.15.16:0/1975357407 >> 172.21.15.16:0/1975357407 conn(0x7f7a4c0794d0 msgr2=0x7f7a4c1041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.860+0000 7f7a53552640 1 -- 172.21.15.16:0/1975357407 shutdown_connections 2024-08-28T19:23:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:36.860+0000 7f7a53552640 1 -- 172.21.15.16:0/1975357407 wait complete. 2024-08-28T19:23:36.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 9.09 MB/sec warmup 76 sec latency 10354.800 ms 2024-08-28T19:23:37.492 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:37 smithi016 ceph-mon[18556]: pgmap v416: 129 pgs: 129 active+clean; 81 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-08-28T19:23:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:37 smithi028 ceph-mon[21769]: pgmap v416: 129 pgs: 129 active+clean; 81 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-08-28T19:23:37.586 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:23:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:37 smithi115 ceph-mon[22184]: pgmap v416: 129 pgs: 129 active+clean; 81 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-08-28T19:23:37.890 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:37.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 8.97 MB/sec warmup 77 sec latency 11354.844 ms 2024-08-28T19:23:38.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:38 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:23:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.371+0000 7f28b39e5640 1 -- 172.21.15.16:0/1672613473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28ac102800 msgr2=0x7f28ac102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.371+0000 7f28b39e5640 1 --2- 172.21.15.16:0/1672613473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28ac102800 0x7f28ac102c10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f28a8009900 tx=0x7f28a802a410 comp rx=0 tx=0).stop 2024-08-28T19:23:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.371+0000 7f28b39e5640 1 -- 172.21.15.16:0/1672613473 shutdown_connections 2024-08-28T19:23:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.371+0000 7f28b39e5640 1 --2- 172.21.15.16:0/1672613473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac108a90 0x7f28ac10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.371+0000 7f28b39e5640 1 --2- 172.21.15.16:0/1672613473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28ac108140 0x7f28ac108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.371+0000 7f28b39e5640 1 --2- 172.21.15.16:0/1672613473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28ac102800 0x7f28ac102c10 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.371+0000 7f28b39e5640 1 -- 172.21.15.16:0/1672613473 >> 172.21.15.16:0/1672613473 conn(0x7f28ac0794d0 msgr2=0x7f28ac0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.371+0000 7f28b39e5640 1 -- 172.21.15.16:0/1672613473 shutdown_connections 2024-08-28T19:23:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.371+0000 7f28b39e5640 1 -- 172.21.15.16:0/1672613473 wait complete. 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b39e5640 1 Processor -- start 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b39e5640 1 -- start start 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b39e5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac102800 0x7f28ac103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b39e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28ac108140 0x7f28ac104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b39e5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28ac108a90 0x7f28ac104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b39e5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac1126f0 con 0x7f28ac108140 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b39e5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac112570 con 0x7f28ac102800 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b39e5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac112870 con 0x7f28ac108a90 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b0f80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28ac108140 0x7f28ac104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b1781640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac102800 0x7f28ac103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b0f80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28ac108140 0x7f28ac104530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45300/0 (socket says 172.21.15.16:45300) 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b0f80640 1 -- 172.21.15.16:0/3426189912 learned_addr learned my addr 172.21.15.16:0/3426189912 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.372+0000 7f28b1f82640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28ac108a90 0x7f28ac104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28b0f80640 1 -- 172.21.15.16:0/3426189912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28ac108a90 msgr2=0x7f28ac104a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28b0f80640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28ac108a90 0x7f28ac104a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28b0f80640 1 -- 172.21.15.16:0/3426189912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac102800 msgr2=0x7f28ac103ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28b0f80640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac102800 0x7f28ac103ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28b0f80640 1 -- 172.21.15.16:0/3426189912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ac1a84d0 con 0x7f28ac108140 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28b1781640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac102800 0x7f28ac103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28b0f80640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28ac108140 0x7f28ac104530 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f289c00d770 tx=0x7f289c00db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28a27fc640 1 -- 172.21.15.16:0/3426189912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c017070 con 0x7f28ac108140 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28b39e5640 1 -- 172.21.15.16:0/3426189912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ac1a87c0 con 0x7f28ac108140 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28a27fc640 1 -- 172.21.15.16:0/3426189912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f289c010460 con 0x7f28ac108140 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28a27fc640 1 -- 172.21.15.16:0/3426189912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c00f620 con 0x7f28ac108140 2024-08-28T19:23:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.373+0000 7f28b39e5640 1 -- 172.21.15.16:0/3426189912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ac1a8d00 con 0x7f28ac108140 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.376+0000 7f28a27fc640 1 -- 172.21.15.16:0/3426189912 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f289c01a070 con 0x7f28ac108140 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.376+0000 7f28b9f5a740 1 Processor -- start 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b9f5a740 1 -- start start 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28a27fc640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28980b4ca0 0x7f28980b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b1f82640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28980b4ca0 0x7f28980b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28a27fc640 1 -- 172.21.15.16:0/3426189912 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f289c0e5280 con 0x7f28ac108140 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38932070 0x561d38cd7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cd7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 0x561d38cd7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561d38c4e680 con 0x561d389091b0 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d38c3dc10 con 0x561d38932070 2024-08-28T19:23:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561d38c41bc0 con 0x561d38911390 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b0f80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cd7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b0f80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cd7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45302/0 (socket says 172.21.15.16:45302) 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b0f80640 1 -- 172.21.15.16:0/382235656 learned_addr learned my addr 172.21.15.16:0/382235656 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b1781640 1 --2- 172.21.15.16:0/382235656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38932070 0x561d38cd7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b1f82640 1 --2- 172.21.15.16:0/382235656 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 0x561d38cd7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b0f80640 1 -- 172.21.15.16:0/382235656 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 msgr2=0x561d38cd7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.377+0000 7f28b0f80640 1 --2- 172.21.15.16:0/382235656 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 0x561d38cd7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.378+0000 7f28b0f80640 1 -- 172.21.15.16:0/382235656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38932070 msgr2=0x561d38cd7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.378+0000 7f28b0f80640 1 --2- 172.21.15.16:0/382235656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38932070 0x561d38cd7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.378+0000 7f28b0f80640 1 -- 172.21.15.16:0/382235656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d3899d650 con 0x561d389091b0 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.378+0000 7f28b1f82640 1 --2- 172.21.15.16:0/382235656 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 0x561d38cd7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.378+0000 7f28b1f82640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28980b4ca0 0x7f28980b71b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f28a4005fd0 tx=0x7f28a4011810 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.378+0000 7f28b0f80640 1 --2- 172.21.15.16:0/382235656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cd7890 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f289c0ab590 tx=0x7f289c0ab600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9759b7aca9d79d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.378+0000 7f28977fe640 1 -- 172.21.15.16:0/382235656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c021070 con 0x561d389091b0 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.378+0000 7f28977fe640 1 -- 172.21.15.16:0/382235656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f289c0b4920 con 0x561d389091b0 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.378+0000 7f28977fe640 1 -- 172.21.15.16:0/382235656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c010930 con 0x561d389091b0 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.379+0000 7f28b9f5a740 1 -- 172.21.15.16:0/382235656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 msgr2=0x561d38cd7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.379+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/382235656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cd7890 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f289c0ab590 tx=0x7f289c0ab600 comp rx=0 tx=0).stop 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.379+0000 7f28b9f5a740 1 -- 172.21.15.16:0/382235656 shutdown_connections 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.379+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/382235656 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 0x561d38cd7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.379+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/382235656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cd7890 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.379+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/382235656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38932070 0x561d38cd7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.379+0000 7f28b9f5a740 1 -- 172.21.15.16:0/382235656 >> 172.21.15.16:0/382235656 conn(0x561d38926460 msgr2=0x561d38cd5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.379+0000 7f28b9f5a740 1 -- 172.21.15.16:0/382235656 shutdown_connections 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.379+0000 7f28b9f5a740 1 -- 172.21.15.16:0/382235656 wait complete. 2024-08-28T19:23:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.379+0000 7f28b9f5a740 1 -- start start 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.379+0000 7f28b9f5a740 1 Processor -- start 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d389091b0 0x561d38ce2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 0x561d38ce3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38ceee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561d38aa0700 con 0x561d38932070 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d38bc4750 con 0x561d389091b0 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561d38ce5b30 con 0x561d38911390 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b9f5a740 10 client.0 _ll_register_callbacks cb 0x561d38bf3e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b1f82640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38ceee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b0f80640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 0x561d38ce3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b1f82640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38ceee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45312/0 (socket says 172.21.15.16:45312) 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b1f82640 1 -- 172.21.15.16:0/1487439753 learned_addr learned my addr 172.21.15.16:0/1487439753 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b1781640 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d389091b0 0x561d38ce2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b0f80640 1 -- 172.21.15.16:0/1487439753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d389091b0 msgr2=0x561d38ce2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b0f80640 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d389091b0 0x561d38ce2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b0f80640 1 -- 172.21.15.16:0/1487439753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 msgr2=0x561d38ceee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b0f80640 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38ceee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b0f80640 1 -- 172.21.15.16:0/1487439753 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d38bfd900 con 0x561d38911390 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.380+0000 7f28b1f82640 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38ceee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f28b0f80640 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 0x561d38ce3090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x561d38ce5700 tx=0x7f289c010820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4343d841a9475307 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f2894ff9640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f2894ff9640 1 -- 172.21.15.16:0/1487439753 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c021050 con 0x561d38911390 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f28b9f5a740 1 -- 172.21.15.16:0/1487439753 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f289c0b4920 con 0x561d38911390 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f2894ff9640 1 -- 172.21.15.16:0/1487439753 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f289c010930 con 0x561d38911390 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f2894ff9640 1 -- 172.21.15.16:0/1487439753 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f289c05b170 con 0x561d38911390 2024-08-28T19:23:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f28b9f5a740 10 client.25831 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f28b9f5a740 1 -- 172.21.15.16:0/1487439753 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561d38cf1970 con 0x561d38911390 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f2894ff9640 1 -- 172.21.15.16:0/1487439753 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f289c0b4920 con 0x561d38911390 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f2894ff9640 1 -- 172.21.15.16:0/1487439753 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f289c0ce0b0 con 0x561d38911390 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f28b9f5a740 10 client.25831 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f28b9f5a740 10 client.25831 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f28b9f5a740 20 client.25831 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561d388f09f0 0x561d38cf4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.381+0000 7f28b9f5a740 4 client.25831 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.382+0000 7f28b9f5a740 1 -- 172.21.15.16:0/1487439753 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d38c505d0 con 0x561d388f09f0 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.382+0000 7f28b1781640 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561d388f09f0 0x561d38cf4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.382+0000 7f2894ff9640 10 client.25831 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.382+0000 7f28b1781640 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561d388f09f0 0x561d38cf4cd0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=db81592c2408fe03 server_cookie=206ff451aa62f8a1 in_seq=0 out_seq=0 2024-08-28T19:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f2894ff9640 1 -- 172.21.15.16:0/1487439753 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561d38c505d0 con 0x561d388f09f0 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f2894ff9640 10 client.25831 handle_command_reply: tid=0 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 1 client.25831 shutdown 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 1 -- 172.21.15.16:0/1487439753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 msgr2=0x561d38ce3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 0x561d38ce3090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x561d38ce5700 tx=0x7f289c010820 comp rx=0 tx=0).stop 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 1 -- 172.21.15.16:0/1487439753 shutdown_connections 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561d388f09f0 0x561d38cf4cd0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38ceee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38911390 0x561d38ce3090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/1487439753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d389091b0 0x561d38ce2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f2894ff9640 0 client.25831 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 1 -- 172.21.15.16:0/1487439753 >> 172.21.15.16:0/1487439753 conn(0x561d38926460 msgr2=0x561d38cda530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 1 -- 172.21.15.16:0/1487439753 shutdown_connections 2024-08-28T19:23:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 1 -- 172.21.15.16:0/1487439753 wait complete. 2024-08-28T19:23:38.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.383+0000 7f28b9f5a740 20 client.25831 trim_cache size 0 max 16384 2024-08-28T19:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.396+0000 7f28b9f5a740 1 Processor -- start 2024-08-28T19:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b9f5a740 1 -- start start 2024-08-28T19:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d389091b0 0x561d38cdab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38911390 0x561d38ceed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38cdfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac1b2d70 con 0x561d38932070 2024-08-28T19:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac1b4970 con 0x561d38911390 2024-08-28T19:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac1b3440 con 0x561d389091b0 2024-08-28T19:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b1f82640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38cdfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b1f82640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38cdfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45320/0 (socket says 172.21.15.16:45320) 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b1f82640 1 -- 172.21.15.16:0/2743697505 learned_addr learned my addr 172.21.15.16:0/2743697505 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b1781640 1 --2- 172.21.15.16:0/2743697505 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d389091b0 0x561d38cdab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b0f80640 1 --2- 172.21.15.16:0/2743697505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38911390 0x561d38ceed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b1f82640 1 -- 172.21.15.16:0/2743697505 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d389091b0 msgr2=0x561d38cdab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b1f82640 1 --2- 172.21.15.16:0/2743697505 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d389091b0 0x561d38cdab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b1f82640 1 -- 172.21.15.16:0/2743697505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38911390 msgr2=0x561d38ceed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b1f82640 1 --2- 172.21.15.16:0/2743697505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38911390 0x561d38ceed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b1f82640 1 -- 172.21.15.16:0/2743697505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d38ce1030 con 0x561d38932070 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b1781640 1 --2- 172.21.15.16:0/2743697505 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d389091b0 0x561d38cdab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.397+0000 7f28b1f82640 1 --2- 172.21.15.16:0/2743697505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38cdfd20 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f28a400fe10 tx=0x7f28a401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96e3eb223f6948f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.398+0000 7f2896ffd640 1 -- 172.21.15.16:0/2743697505 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a402d070 con 0x561d38932070 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.398+0000 7f2896ffd640 1 -- 172.21.15.16:0/2743697505 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28a4019050 con 0x561d38932070 2024-08-28T19:23:38.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.398+0000 7f2896ffd640 1 -- 172.21.15.16:0/2743697505 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a4017040 con 0x561d38932070 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.398+0000 7f28b9f5a740 1 -- 172.21.15.16:0/2743697505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 msgr2=0x561d38cdfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.398+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/2743697505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38cdfd20 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f28a400fe10 tx=0x7f28a401a000 comp rx=0 tx=0).stop 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 -- 172.21.15.16:0/2743697505 shutdown_connections 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/2743697505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d38932070 0x561d38cdfd20 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/2743697505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38911390 0x561d38ceed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/2743697505 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d389091b0 0x561d38cdab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 -- 172.21.15.16:0/2743697505 >> 172.21.15.16:0/2743697505 conn(0x561d38926460 msgr2=0x561d38cda450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 -- 172.21.15.16:0/2743697505 shutdown_connections 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 -- 172.21.15.16:0/2743697505 wait complete. 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 -- start start 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 Processor -- start 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cf8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38911390 0x561d38cf5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.399+0000 7f28b9f5a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38932070 0x561d38cf5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac1b41e0 con 0x561d389091b0 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac1b3b10 con 0x561d38911390 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b9f5a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac1b0d50 con 0x561d38932070 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b9f5a740 10 client.0 _ll_register_callbacks cb 0x7f28a8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b1781640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cf8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b1781640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cf8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45332/0 (socket says 172.21.15.16:45332) 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b1781640 1 -- 172.21.15.16:0/3683322223 learned_addr learned my addr 172.21.15.16:0/3683322223 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b0f80640 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38911390 0x561d38cf5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b1f82640 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38932070 0x561d38cf5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b1781640 1 -- 172.21.15.16:0/3683322223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38932070 msgr2=0x561d38cf5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b1781640 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38932070 0x561d38cf5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b1781640 1 -- 172.21.15.16:0/3683322223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38911390 msgr2=0x561d38cf5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b1781640 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38911390 0x561d38cf5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b1781640 1 -- 172.21.15.16:0/3683322223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d38cf6750 con 0x561d389091b0 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b0f80640 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38911390 0x561d38cf5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f28b1781640 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cf8520 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f28a802a930 tx=0x7f28a8004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf647bb3be5baf8b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f2894ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f2894ff9640 1 -- 172.21.15.16:0/3683322223 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a8035040 con 0x561d389091b0 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.400+0000 7f2894ff9640 1 -- 172.21.15.16:0/3683322223 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f28a8005240 con 0x561d389091b0 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f28b9f5a740 1 -- 172.21.15.16:0/3683322223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28a4019050 con 0x561d389091b0 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f2894ff9640 1 -- 172.21.15.16:0/3683322223 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f28a8055050 con 0x561d389091b0 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f28b9f5a740 10 client.16356 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f28b9f5a740 1 -- 172.21.15.16:0/3683322223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561d38d5f3d0 con 0x561d389091b0 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f2894ff9640 1 -- 172.21.15.16:0/3683322223 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28a4019050 con 0x561d389091b0 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f2894ff9640 1 -- 172.21.15.16:0/3683322223 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f28a8031800 con 0x561d389091b0 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f28b9f5a740 10 client.16356 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:38.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f28b9f5a740 10 client.16356 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f28b9f5a740 20 client.16356 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561d388c5da0 0x561d38d61b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f28b9f5a740 4 client.16356 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f28b9f5a740 1 -- 172.21.15.16:0/3683322223 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561d38ce3420 con 0x561d388c5da0 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.401+0000 7f28b0f80640 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561d388c5da0 0x561d38d61b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f2894ff9640 10 client.16356 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b0f80640 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561d388c5da0 0x561d38d61b60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9b99c0c4a818b18 server_cookie=66898b0605e7f328 in_seq=0 out_seq=0 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f2894ff9640 1 -- 172.21.15.16:0/3683322223 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561d38ce3420 con 0x561d388c5da0 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f2894ff9640 10 client.16356 handle_command_reply: tid=0 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 1 client.16356 shutdown 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 1 -- 172.21.15.16:0/3683322223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 msgr2=0x561d38cf8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cf8520 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f28a802a930 tx=0x7f28a8004320 comp rx=0 tx=0).stop 2024-08-28T19:23:38.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 1 -- 172.21.15.16:0/3683322223 shutdown_connections 2024-08-28T19:23:38.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561d388c5da0 0x561d38d61b60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d38932070 0x561d38cf5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d38911390 0x561d38cf5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 1 --2- 172.21.15.16:0/3683322223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d389091b0 0x561d38cf8520 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 1 -- 172.21.15.16:0/3683322223 >> 172.21.15.16:0/3683322223 conn(0x561d38926460 msgr2=0x561d38cd82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:38.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f2894ff9640 0 client.16356 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:23:38.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 1 -- 172.21.15.16:0/3683322223 shutdown_connections 2024-08-28T19:23:38.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 1 -- 172.21.15.16:0/3683322223 wait complete. 2024-08-28T19:23:38.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.402+0000 7f28b9f5a740 20 client.16356 trim_cache size 0 max 16384 2024-08-28T19:23:38.414 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:23:38.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 -- 172.21.15.16:0/3426189912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28980b4ca0 msgr2=0x7f28980b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:38.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28980b4ca0 0x7f28980b71b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f28a4005fd0 tx=0x7f28a4011810 comp rx=0 tx=0).stop 2024-08-28T19:23:38.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 -- 172.21.15.16:0/3426189912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28ac108140 msgr2=0x7f28ac104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:38.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28ac108140 0x7f28ac104530 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f289c00d770 tx=0x7f289c00db70 comp rx=0 tx=0).stop 2024-08-28T19:23:38.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 -- 172.21.15.16:0/3426189912 shutdown_connections 2024-08-28T19:23:38.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28ac108a90 0x7f28ac104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28ac108140 0x7f28ac104530 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28980b4ca0 0x7f28980b71b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 --2- 172.21.15.16:0/3426189912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac102800 0x7f28ac103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:38.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 -- 172.21.15.16:0/3426189912 >> 172.21.15.16:0/3426189912 conn(0x7f28ac0794d0 msgr2=0x7f28ac101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:38.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 -- 172.21.15.16:0/3426189912 shutdown_connections 2024-08-28T19:23:38.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:38.403+0000 7f28b39e5640 1 -- 172.21.15.16:0/3426189912 wait complete. 2024-08-28T19:23:38.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:38 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:23:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:38 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:23:38.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 8.86 MB/sec warmup 78 sec latency 12354.925 ms 2024-08-28T19:23:39.255 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:23:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:39 smithi016 ceph-mon[18556]: pgmap v417: 129 pgs: 129 active+clean; 82 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 852 KiB/s wr, 82 op/s 2024-08-28T19:23:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:39 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:23:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:39 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:23:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:39 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:23:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:39 smithi028 ceph-mon[21769]: pgmap v417: 129 pgs: 129 active+clean; 82 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 852 KiB/s wr, 82 op/s 2024-08-28T19:23:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:39 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:23:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:39 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:23:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:39 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:23:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:39 smithi115 ceph-mon[22184]: pgmap v417: 129 pgs: 129 active+clean; 82 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 852 KiB/s wr, 82 op/s 2024-08-28T19:23:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:39 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:23:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:39 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:23:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:39 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:23:39.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 8.74 MB/sec warmup 79 sec latency 13354.990 ms 2024-08-28T19:23:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.037+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3309057397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f98109b90 msgr2=0x7f0f981103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.037+0000 7f0f9de1d640 1 --2- 172.21.15.16:0/3309057397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f98109b90 0x7f0f981103c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009ca0 tx=0x7f0f9002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.037+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3309057397 shutdown_connections 2024-08-28T19:23:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.037+0000 7f0f9de1d640 1 --2- 172.21.15.16:0/3309057397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f98109b90 0x7f0f981103c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.038+0000 7f0f9de1d640 1 --2- 172.21.15.16:0/3309057397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f98109240 0x7f0f98109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.038+0000 7f0f9de1d640 1 --2- 172.21.15.16:0/3309057397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f98103900 0x7f0f98103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.038+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3309057397 >> 172.21.15.16:0/3309057397 conn(0x7f0f9807b5f0 msgr2=0x7f0f9807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.038+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3309057397 shutdown_connections 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.038+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3309057397 wait complete. 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.038+0000 7f0f9de1d640 1 Processor -- start 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.038+0000 7f0f9de1d640 1 -- start start 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f9de1d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f98103900 0x7f0f9819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f9de1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f98109240 0x7f0f9819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f9de1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f98109b90 0x7f0f9819fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f9de1d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f98113850 con 0x7f0f98109240 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f9de1d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f981136d0 con 0x7f0f98109b90 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f9de1d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0f981139d0 con 0x7f0f98103900 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f96ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f98109240 0x7f0f9819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f96ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f98109240 0x7f0f9819a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45336/0 (socket says 172.21.15.16:45336) 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f96ffd640 1 -- 172.21.15.16:0/3102656145 learned_addr learned my addr 172.21.15.16:0/3102656145 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f977fe640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f98103900 0x7f0f9819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f97fff640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f98109b90 0x7f0f9819fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f96ffd640 1 -- 172.21.15.16:0/3102656145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f98103900 msgr2=0x7f0f9819a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f98103900 0x7f0f9819a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f96ffd640 1 -- 172.21.15.16:0/3102656145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f98109b90 msgr2=0x7f0f9819fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f98109b90 0x7f0f9819fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f96ffd640 1 -- 172.21.15.16:0/3102656145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f981a0090 con 0x7f0f98109240 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f97fff640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f98109b90 0x7f0f9819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.039+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f98109240 0x7f0f9819a780 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c00e9f0 tx=0x7f0f8c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.040+0000 7f0f94ff9640 1 -- 172.21.15.16:0/3102656145 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c00ccc0 con 0x7f0f98109240 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.040+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3102656145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f981a0380 con 0x7f0f98109240 2024-08-28T19:23:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.040+0000 7f0f94ff9640 1 -- 172.21.15.16:0/3102656145 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f8c0108e0 con 0x7f0f98109240 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.040+0000 7f0f94ff9640 1 -- 172.21.15.16:0/3102656145 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c018880 con 0x7f0f98109240 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.040+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3102656145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f981a6ba0 con 0x7f0f98109240 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.042+0000 7f0fa4392740 1 Processor -- start 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0fa4392740 1 -- start start 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b5843310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b54751b0 0x5603b5843850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b5843d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0fa4392740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5603b57ba680 con 0x5603b547d390 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0fa4392740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5603b57a9c10 con 0x5603b549e070 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0fa4392740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5603b57adbc0 con 0x5603b54751b0 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0f94ff9640 1 -- 172.21.15.16:0/3102656145 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0f8c010430 con 0x7f0f98109240 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0f96ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b5843d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0f977fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b54751b0 0x5603b5843850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0f96ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b5843d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45350/0 (socket says 172.21.15.16:45350) 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0f96ffd640 1 -- 172.21.15.16:0/2185816830 learned_addr learned my addr 172.21.15.16:0/2185816830 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0f97fff640 1 --2- 172.21.15.16:0/2185816830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b5843310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.043+0000 7f0f96ffd640 1 -- 172.21.15.16:0/2185816830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b54751b0 msgr2=0x5603b5843850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.044+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/2185816830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b54751b0 0x5603b5843850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.044+0000 7f0f96ffd640 1 -- 172.21.15.16:0/2185816830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 msgr2=0x5603b5843310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.044+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/2185816830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b5843310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.044+0000 7f0f96ffd640 1 -- 172.21.15.16:0/2185816830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603b5509650 con 0x5603b547d390 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.044+0000 7f0f94ff9640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f800b4cd0 0x7f0f800b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.044+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/2185816830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b5843d90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0e3020 tx=0x7f0f8c0ee810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0704ca677b32465 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.044+0000 7f0f97fff640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f800b4cd0 0x7f0f800b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.044+0000 7f0f94ff9640 1 -- 172.21.15.16:0/3102656145 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0f8c0e5cd0 con 0x7f0f98109240 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.044+0000 7f0f7dffb640 1 -- 172.21.15.16:0/2185816830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0a7f10 con 0x5603b547d390 2024-08-28T19:23:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.044+0000 7f0f7dffb640 1 -- 172.21.15.16:0/2185816830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f8c0e7030 con 0x5603b547d390 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.044+0000 7f0f7dffb640 1 -- 172.21.15.16:0/2185816830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0ab7c0 con 0x5603b547d390 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.045+0000 7f0fa4392740 1 -- 172.21.15.16:0/2185816830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 msgr2=0x5603b5843d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.045+0000 7f0f97fff640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f800b4cd0 0x7f0f800b71e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0f90005cc0 tx=0x7f0f90005c30 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.045+0000 7f0fa4392740 1 --2- 172.21.15.16:0/2185816830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b5843d90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0e3020 tx=0x7f0f8c0ee810 comp rx=0 tx=0).stop 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.045+0000 7f0fa4392740 1 -- 172.21.15.16:0/2185816830 shutdown_connections 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.045+0000 7f0fa4392740 1 --2- 172.21.15.16:0/2185816830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b5843d90 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.045+0000 7f0fa4392740 1 --2- 172.21.15.16:0/2185816830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b54751b0 0x5603b5843850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.045+0000 7f0fa4392740 1 --2- 172.21.15.16:0/2185816830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b5843310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.045+0000 7f0fa4392740 1 -- 172.21.15.16:0/2185816830 >> 172.21.15.16:0/2185816830 conn(0x5603b5492460 msgr2=0x5603b5841200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.045+0000 7f0fa4392740 1 -- 172.21.15.16:0/2185816830 shutdown_connections 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.045+0000 7f0fa4392740 1 -- 172.21.15.16:0/2185816830 wait complete. 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0fa4392740 1 -- start start 2024-08-28T19:23:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0fa4392740 1 Processor -- start 2024-08-28T19:23:40.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b54751b0 0x5603b584ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b584ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b585ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0fa4392740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5603b560c700 con 0x5603b547d390 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0fa4392740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5603b5730750 con 0x5603b549e070 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0fa4392740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5603b5851b40 con 0x5603b54751b0 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0fa4392740 10 client.0 _ll_register_callbacks cb 0x5603b575fe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0f96ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b584ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0f96ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b584ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45356/0 (socket says 172.21.15.16:45356) 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0f96ffd640 1 -- 172.21.15.16:0/3027572745 learned_addr learned my addr 172.21.15.16:0/3027572745 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0f977fe640 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b54751b0 0x5603b584ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.046+0000 7f0f97fff640 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b585ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0f96ffd640 1 -- 172.21.15.16:0/3027572745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b54751b0 msgr2=0x5603b584ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b54751b0 0x5603b584ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0f96ffd640 1 -- 172.21.15.16:0/3027572745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 msgr2=0x5603b585ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b585ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0f96ffd640 1 -- 172.21.15.16:0/3027572745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603b5769900 con 0x5603b547d390 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b584ef70 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0a92c0 tx=0x7f0f8c0096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f750ac17c5788f2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0f76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0f76ffd640 1 -- 172.21.15.16:0/3027572745 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0ab9a0 con 0x5603b547d390 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0f76ffd640 1 -- 172.21.15.16:0/3027572745 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0f8c0a8640 con 0x5603b547d390 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0fa4392740 1 -- 172.21.15.16:0/3027572745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f8c0e7030 con 0x5603b547d390 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0f76ffd640 1 -- 172.21.15.16:0/3027572745 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0f8c0abe20 con 0x5603b547d390 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0fa4392740 10 client.16368 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.047+0000 7f0fa4392740 1 -- 172.21.15.16:0/3027572745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603b585d8b0 con 0x5603b547d390 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.048+0000 7f0f76ffd640 1 -- 172.21.15.16:0/3027572745 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f8c0e7030 con 0x5603b547d390 2024-08-28T19:23:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.048+0000 7f0f76ffd640 1 -- 172.21.15.16:0/3027572745 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f0f8c0971a0 con 0x5603b547d390 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.048+0000 7f0fa4392740 10 client.16368 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.048+0000 7f0fa4392740 10 client.16368 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.048+0000 7f0fa4392740 20 client.16368 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.048+0000 7f0fa4392740 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5603b545c9f0 0x5603b5860c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.048+0000 7f0fa4392740 4 client.16368 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.048+0000 7f0fa4392740 1 -- 172.21.15.16:0/3027572745 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603b57bc5d0 con 0x5603b545c9f0 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.048+0000 7f0f977fe640 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5603b545c9f0 0x5603b5860c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.049+0000 7f0f977fe640 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5603b545c9f0 0x5603b5860c10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f84e396d4e0372fd server_cookie=92c1ff5831c03984 in_seq=0 out_seq=0 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.049+0000 7f0f76ffd640 10 client.16368 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.049+0000 7f0f76ffd640 1 -- 172.21.15.16:0/3027572745 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5603b57bc5d0 con 0x5603b545c9f0 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.049+0000 7f0f76ffd640 10 client.16368 handle_command_reply: tid=0 2024-08-28T19:23:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.049+0000 7f0fa4392740 1 client.16368 shutdown 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0fa4392740 1 -- 172.21.15.16:0/3027572745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 msgr2=0x5603b584ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0fa4392740 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b584ef70 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0a92c0 tx=0x7f0f8c0096a0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0fa4392740 1 -- 172.21.15.16:0/3027572745 shutdown_connections 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0fa4392740 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5603b545c9f0 0x5603b5860c10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0fa4392740 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b585ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0fa4392740 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b547d390 0x5603b584ef70 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0fa4392740 1 --2- 172.21.15.16:0/3027572745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b54751b0 0x5603b584ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0f76ffd640 0 client.16368 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0fa4392740 1 -- 172.21.15.16:0/3027572745 >> 172.21.15.16:0/3027572745 conn(0x5603b5492460 msgr2=0x5603b58464f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0fa4392740 1 -- 172.21.15.16:0/3027572745 shutdown_connections 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0fa4392740 1 -- 172.21.15.16:0/3027572745 wait complete. 2024-08-28T19:23:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.050+0000 7f0fa4392740 20 client.16368 trim_cache size 0 max 16384 2024-08-28T19:23:40.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.063+0000 7f0fa4392740 1 Processor -- start 2024-08-28T19:23:40.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.063+0000 7f0fa4392740 1 -- start start 2024-08-28T19:23:40.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.063+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5841230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.063+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b547d390 0x5603b585ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.063+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b549e070 0x5603b584bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.063+0000 7f0fa4392740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f981b0c70 con 0x5603b54751b0 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.063+0000 7f0fa4392740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f981b2870 con 0x5603b547d390 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.063+0000 7f0fa4392740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0f981b1340 con 0x5603b549e070 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5841230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5841230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45368/0 (socket says 172.21.15.16:45368) 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f977fe640 1 -- 172.21.15.16:0/831854232 learned_addr learned my addr 172.21.15.16:0/831854232 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/831854232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b547d390 0x5603b585ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f977fe640 1 -- 172.21.15.16:0/831854232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b549e070 msgr2=0x5603b584bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f977fe640 1 --2- 172.21.15.16:0/831854232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b549e070 0x5603b584bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f977fe640 1 -- 172.21.15.16:0/831854232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b547d390 msgr2=0x5603b585ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f977fe640 1 --2- 172.21.15.16:0/831854232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b547d390 0x5603b585ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f977fe640 1 -- 172.21.15.16:0/831854232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603b584cf30 con 0x5603b54751b0 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f977fe640 1 --2- 172.21.15.16:0/831854232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5841230 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800ac40 tx=0x7f0f8800a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d193879913289d5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f7d7fa640 1 -- 172.21.15.16:0/831854232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f88028020 con 0x5603b54751b0 2024-08-28T19:23:40.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f7d7fa640 1 -- 172.21.15.16:0/831854232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f880047a0 con 0x5603b54751b0 2024-08-28T19:23:40.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.064+0000 7f0f7d7fa640 1 -- 172.21.15.16:0/831854232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f88024700 con 0x5603b54751b0 2024-08-28T19:23:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.065+0000 7f0fa4392740 1 -- 172.21.15.16:0/831854232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 msgr2=0x5603b5841230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.065+0000 7f0fa4392740 1 --2- 172.21.15.16:0/831854232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5841230 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800ac40 tx=0x7f0f8800a2d0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.065+0000 7f0fa4392740 1 -- 172.21.15.16:0/831854232 shutdown_connections 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.065+0000 7f0fa4392740 1 --2- 172.21.15.16:0/831854232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b549e070 0x5603b584bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.065+0000 7f0fa4392740 1 --2- 172.21.15.16:0/831854232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b547d390 0x5603b585ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.065+0000 7f0fa4392740 1 --2- 172.21.15.16:0/831854232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5841230 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.065+0000 7f0fa4392740 1 -- 172.21.15.16:0/831854232 >> 172.21.15.16:0/831854232 conn(0x5603b5492460 msgr2=0x5603b5840b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.065+0000 7f0fa4392740 1 -- 172.21.15.16:0/831854232 shutdown_connections 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.065+0000 7f0fa4392740 1 -- 172.21.15.16:0/831854232 wait complete. 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.065+0000 7f0fa4392740 1 -- start start 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0fa4392740 1 Processor -- start 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5861e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b547d390 0x5603b58623d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0fa4392740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b58ca390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0fa4392740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f981b20e0 con 0x5603b54751b0 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0fa4392740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f981b1a10 con 0x5603b549e070 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0fa4392740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0f981aec50 con 0x5603b547d390 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0fa4392740 10 client.0 _ll_register_callbacks cb 0x7f0f8c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0f977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5861e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0f977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5861e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45370/0 (socket says 172.21.15.16:45370) 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0f977fe640 1 -- 172.21.15.16:0/2505668323 learned_addr learned my addr 172.21.15.16:0/2505668323 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0f97fff640 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b58ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b547d390 0x5603b58623d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0f977fe640 1 -- 172.21.15.16:0/2505668323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b547d390 msgr2=0x5603b58623d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0f977fe640 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b547d390 0x5603b58623d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0f977fe640 1 -- 172.21.15.16:0/2505668323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 msgr2=0x5603b58ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0f977fe640 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b58ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0f977fe640 1 -- 172.21.15.16:0/2505668323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603b5862bc0 con 0x5603b54751b0 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.066+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b547d390 0x5603b58623d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.067+0000 7f0f977fe640 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5861e90 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0f88004d60 tx=0x7f0f88004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21d5d9a2366341e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.067+0000 7f0f76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.067+0000 7f0f76ffd640 1 -- 172.21.15.16:0/2505668323 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f88031070 con 0x5603b54751b0 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.067+0000 7f0fa4392740 1 -- 172.21.15.16:0/2505668323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f880047a0 con 0x5603b54751b0 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.067+0000 7f0f76ffd640 1 -- 172.21.15.16:0/2505668323 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0f88005430 con 0x5603b54751b0 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.067+0000 7f0f76ffd640 1 -- 172.21.15.16:0/2505668323 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0f8803d050 con 0x5603b54751b0 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.067+0000 7f0fa4392740 10 client.16374 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.067+0000 7f0fa4392740 1 -- 172.21.15.16:0/2505668323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603b58cb880 con 0x5603b54751b0 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.067+0000 7f0f76ffd640 1 -- 172.21.15.16:0/2505668323 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f880047a0 con 0x5603b54751b0 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.067+0000 7f0f76ffd640 1 -- 172.21.15.16:0/2505668323 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f0f8801e520 con 0x5603b54751b0 2024-08-28T19:23:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0fa4392740 10 client.16374 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0fa4392740 10 client.16374 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0fa4392740 20 client.16374 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0fa4392740 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5603b5431da0 0x5603b58cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0fa4392740 4 client.16374 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0fa4392740 1 -- 172.21.15.16:0/2505668323 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5603b57b8070 con 0x5603b5431da0 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5603b5431da0 0x5603b58cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0f96ffd640 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5603b5431da0 0x5603b58cdfb0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=76169ab833b6e435 server_cookie=874f85994e5c3f90 in_seq=0 out_seq=0 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0f76ffd640 10 client.16374 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0f76ffd640 1 -- 172.21.15.16:0/2505668323 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5603b57b8070 con 0x5603b5431da0 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0f76ffd640 10 client.16374 handle_command_reply: tid=0 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.068+0000 7f0fa4392740 1 client.16374 shutdown 2024-08-28T19:23:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0fa4392740 1 -- 172.21.15.16:0/2505668323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 msgr2=0x5603b5861e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0fa4392740 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5861e90 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0f88004d60 tx=0x7f0f88004d90 comp rx=0 tx=0).stop 2024-08-28T19:23:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0fa4392740 1 -- 172.21.15.16:0/2505668323 shutdown_connections 2024-08-28T19:23:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0fa4392740 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5603b5431da0 0x5603b58cdfb0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0fa4392740 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603b549e070 0x5603b58ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0fa4392740 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603b547d390 0x5603b58623d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0fa4392740 1 --2- 172.21.15.16:0/2505668323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603b54751b0 0x5603b5861e90 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0fa4392740 1 -- 172.21.15.16:0/2505668323 >> 172.21.15.16:0/2505668323 conn(0x5603b5492460 msgr2=0x5603b5846280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0f76ffd640 0 client.16374 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:23:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0fa4392740 1 -- 172.21.15.16:0/2505668323 shutdown_connections 2024-08-28T19:23:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0fa4392740 1 -- 172.21.15.16:0/2505668323 wait complete. 2024-08-28T19:23:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0fa4392740 20 client.16374 trim_cache size 0 max 16384 2024-08-28T19:23:40.076 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:23:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3102656145 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f800b4cd0 msgr2=0x7f0f800b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0f9de1d640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f800b4cd0 0x7f0f800b71e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0f90005cc0 tx=0x7f0f90005c30 comp rx=0 tx=0).stop 2024-08-28T19:23:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3102656145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f98109240 msgr2=0x7f0f9819a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.069+0000 7f0f9de1d640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f98109240 0x7f0f9819a780 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c00e9f0 tx=0x7f0f8c00edf0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.070+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3102656145 shutdown_connections 2024-08-28T19:23:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.070+0000 7f0f9de1d640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f800b4cd0 0x7f0f800b71e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.070+0000 7f0f9de1d640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f98109b90 0x7f0f9819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.070+0000 7f0f9de1d640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f98109240 0x7f0f9819a780 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.070+0000 7f0f9de1d640 1 --2- 172.21.15.16:0/3102656145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f98103900 0x7f0f9819a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.070+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3102656145 >> 172.21.15.16:0/3102656145 conn(0x7f0f9807b5f0 msgr2=0x7f0f98107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.070+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3102656145 shutdown_connections 2024-08-28T19:23:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:40.070+0000 7f0f9de1d640 1 -- 172.21.15.16:0/3102656145 wait complete. 2024-08-28T19:23:40.790 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:23:40.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 8.63 MB/sec warmup 80 sec latency 14355.047 ms 2024-08-28T19:23:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:41.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:41 smithi016 ceph-mon[18556]: pgmap v418: 129 pgs: 129 active+clean; 84 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.1 MiB/s wr, 78 op/s 2024-08-28T19:23:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.507+0000 7f0091b67640 1 -- 172.21.15.16:0/3627155133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c102880 msgr2=0x7f008c102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.507+0000 7f0091b67640 1 --2- 172.21.15.16:0/3627155133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c102880 0x7f008c102c90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f007c009900 tx=0x7f007c02a410 comp rx=0 tx=0).stop 2024-08-28T19:23:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.507+0000 7f0091b67640 1 -- 172.21.15.16:0/3627155133 shutdown_connections 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.507+0000 7f0091b67640 1 --2- 172.21.15.16:0/3627155133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f008c108b10 0x7f008c10f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.507+0000 7f0091b67640 1 --2- 172.21.15.16:0/3627155133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f008c1081c0 0x7f008c1085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.507+0000 7f0091b67640 1 --2- 172.21.15.16:0/3627155133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c102880 0x7f008c102c90 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.507+0000 7f0091b67640 1 -- 172.21.15.16:0/3627155133 >> 172.21.15.16:0/3627155133 conn(0x7f008c079510 msgr2=0x7f008c079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.507+0000 7f0091b67640 1 -- 172.21.15.16:0/3627155133 shutdown_connections 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.507+0000 7f0091b67640 1 -- 172.21.15.16:0/3627155133 wait complete. 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.508+0000 7f0091b67640 1 Processor -- start 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.508+0000 7f0091b67640 1 -- start start 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f0091b67640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f008c102880 0x7f008c19b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f0091b67640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f008c1081c0 0x7f008c19b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f0091b67640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c108b10 0x7f008c19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f0091b67640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1127e0 con 0x7f008c108b10 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f0091b67640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f008c112660 con 0x7f008c1081c0 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f0091b67640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f008c112960 con 0x7f008c102880 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f008bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c108b10 0x7f008c19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f008bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c108b10 0x7f008c19bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45374/0 (socket says 172.21.15.16:45374) 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f008bfff640 1 -- 172.21.15.16:0/2932991028 learned_addr learned my addr 172.21.15.16:0/2932991028 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f008affd640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f008c1081c0 0x7f008c19b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f008b7fe640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f008c102880 0x7f008c19b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f008bfff640 1 -- 172.21.15.16:0/2932991028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f008c102880 msgr2=0x7f008c19b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f008bfff640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f008c102880 0x7f008c19b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f008bfff640 1 -- 172.21.15.16:0/2932991028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f008c1081c0 msgr2=0x7f008c19b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f008bfff640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f008c1081c0 0x7f008c19b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f008bfff640 1 -- 172.21.15.16:0/2932991028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008c1a6370 con 0x7f008c108b10 2024-08-28T19:23:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f008bfff640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c108b10 0x7f008c19bb30 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f008400eaa0 tx=0x7f008400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f0088ff9640 1 -- 172.21.15.16:0/2932991028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008400cdb0 con 0x7f008c108b10 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f0088ff9640 1 -- 172.21.15.16:0/2932991028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0084005020 con 0x7f008c108b10 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f0088ff9640 1 -- 172.21.15.16:0/2932991028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00840116d0 con 0x7f008c108b10 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.509+0000 7f0091b67640 1 -- 172.21.15.16:0/2932991028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f008c1a6660 con 0x7f008c108b10 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.510+0000 7f0091b67640 1 -- 172.21.15.16:0/2932991028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f008c1a6b50 con 0x7f008c108b10 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.512+0000 7f00980dc740 1 Processor -- start 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.512+0000 7f0088ff9640 1 -- 172.21.15.16:0/2932991028 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f00840051d0 con 0x7f008c108b10 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.512+0000 7f00980dc740 1 -- start start 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.512+0000 7f00980dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f375686070 0x55f375a2b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f00980dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f37565d1b0 0x55f375a2b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f00980dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a2bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f00980dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f3759a2680 con 0x55f375665390 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f00980dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f375991c10 con 0x55f375686070 2024-08-28T19:23:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f00980dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f375995bc0 con 0x55f37565d1b0 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a2bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a2bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45386/0 (socket says 172.21.15.16:45386) 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008affd640 1 -- 172.21.15.16:0/1262432583 learned_addr learned my addr 172.21.15.16:0/1262432583 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008affd640 1 --2- 172.21.15.16:0/1262432583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f375686070 0x55f375a2b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f0088ff9640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00740b4ca0 0x7f00740b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008b7fe640 1 --2- 172.21.15.16:0/1262432583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f37565d1b0 0x55f375a2b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008affd640 1 -- 172.21.15.16:0/1262432583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f37565d1b0 msgr2=0x55f375a2b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008affd640 1 --2- 172.21.15.16:0/1262432583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f37565d1b0 0x55f375a2b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008affd640 1 -- 172.21.15.16:0/1262432583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f375686070 msgr2=0x55f375a2b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008affd640 1 --2- 172.21.15.16:0/1262432583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f375686070 0x55f375a2b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008affd640 1 -- 172.21.15.16:0/1262432583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3756f1650 con 0x55f375665390 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008bfff640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00740b4ca0 0x7f00740b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f008affd640 1 --2- 172.21.15.16:0/1262432583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a2bd90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f00800114f0 tx=0x7f008000a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9144beec2f7378 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f0088ff9640 1 -- 172.21.15.16:0/2932991028 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f00840e5a20 con 0x7f008c108b10 2024-08-28T19:23:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.513+0000 7f0071ffb640 1 -- 172.21.15.16:0/1262432583 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080004220 con 0x55f375665390 2024-08-28T19:23:41.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.514+0000 7f0071ffb640 1 -- 172.21.15.16:0/1262432583 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00800043c0 con 0x55f375665390 2024-08-28T19:23:41.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.514+0000 7f0071ffb640 1 -- 172.21.15.16:0/1262432583 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080004e20 con 0x55f375665390 2024-08-28T19:23:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.514+0000 7f00980dc740 1 -- 172.21.15.16:0/1262432583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 msgr2=0x55f375a2bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.514+0000 7f008bfff640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00740b4ca0 0x7f00740b71b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f00840e2eb0 tx=0x7f0084093040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.514+0000 7f00980dc740 1 --2- 172.21.15.16:0/1262432583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a2bd90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f00800114f0 tx=0x7f008000a6f0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.515+0000 7f00980dc740 1 -- 172.21.15.16:0/1262432583 shutdown_connections 2024-08-28T19:23:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.515+0000 7f00980dc740 1 --2- 172.21.15.16:0/1262432583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a2bd90 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.515+0000 7f00980dc740 1 --2- 172.21.15.16:0/1262432583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f37565d1b0 0x55f375a2b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.515+0000 7f00980dc740 1 --2- 172.21.15.16:0/1262432583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f375686070 0x55f375a2b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.515+0000 7f00980dc740 1 -- 172.21.15.16:0/1262432583 >> 172.21.15.16:0/1262432583 conn(0x55f37567a460 msgr2=0x55f375a29200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.515+0000 7f00980dc740 1 -- 172.21.15.16:0/1262432583 shutdown_connections 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.515+0000 7f00980dc740 1 -- 172.21.15.16:0/1262432583 wait complete. 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.515+0000 7f00980dc740 1 -- start start 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.515+0000 7f00980dc740 1 Processor -- start 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f00980dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f37565d1b0 0x55f375a36a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f00980dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f375665390 0x55f375a36f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f00980dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375686070 0x55f375a42de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f00980dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f3757f4700 con 0x55f375686070 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f00980dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f375918750 con 0x55f37565d1b0 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f00980dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f375a39af0 con 0x55f375665390 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f00980dc740 10 client.0 _ll_register_callbacks cb 0x55f375947e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f008affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375686070 0x55f375a42de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f008affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375686070 0x55f375a42de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45400/0 (socket says 172.21.15.16:45400) 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f008affd640 1 -- 172.21.15.16:0/2825153228 learned_addr learned my addr 172.21.15.16:0/2825153228 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f008b7fe640 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f375665390 0x55f375a36f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f008affd640 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f37565d1b0 0x55f375a36a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f008affd640 1 -- 172.21.15.16:0/2825153228 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f375665390 msgr2=0x55f375a36f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f008affd640 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f375665390 0x55f375a36f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f008affd640 1 -- 172.21.15.16:0/2825153228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f37565d1b0 msgr2=0x55f375a36a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f008affd640 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f37565d1b0 0x55f375a36a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f008affd640 1 -- 172.21.15.16:0/2825153228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f375951900 con 0x55f375686070 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f008affd640 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375686070 0x55f375a42de0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x55f375a39250 tx=0x7f00800049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce190d6be51f891d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.516+0000 7f006affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f006affd640 1 -- 172.21.15.16:0/2825153228 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00800052c0 con 0x55f375686070 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f006affd640 1 -- 172.21.15.16:0/2825153228 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0080005460 con 0x55f375686070 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f00980dc740 1 -- 172.21.15.16:0/2825153228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00800043c0 con 0x55f375686070 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f006affd640 1 -- 172.21.15.16:0/2825153228 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f008000f690 con 0x55f375686070 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f00980dc740 10 client.16386 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f00980dc740 1 -- 172.21.15.16:0/2825153228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f375a45830 con 0x55f375686070 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f006affd640 1 -- 172.21.15.16:0/2825153228 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00800043c0 con 0x55f375686070 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f006affd640 1 -- 172.21.15.16:0/2825153228 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f008001b030 con 0x55f375686070 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f00980dc740 10 client.16386 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f00980dc740 10 client.16386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f00980dc740 20 client.16386 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f00980dc740 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f3756449f0 0x55f375a48b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f00980dc740 4 client.16386 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.517+0000 7f00980dc740 1 -- 172.21.15.16:0/2825153228 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3759a45d0 con 0x55f3756449f0 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.518+0000 7f008b7fe640 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f3756449f0 0x55f375a48b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.518+0000 7f008b7fe640 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f3756449f0 0x55f375a48b90 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=849c33ecc3bb5b6e server_cookie=b57574a68ea7b2f3 in_seq=0 out_seq=0 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.518+0000 7f006affd640 10 client.16386 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f006affd640 1 -- 172.21.15.16:0/2825153228 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f3759a45d0 con 0x55f3756449f0 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f006affd640 10 client.16386 handle_command_reply: tid=0 2024-08-28T19:23:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 1 client.16386 shutdown 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 1 -- 172.21.15.16:0/2825153228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375686070 msgr2=0x55f375a42de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375686070 0x55f375a42de0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x55f375a39250 tx=0x7f00800049c0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 1 -- 172.21.15.16:0/2825153228 shutdown_connections 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375686070 0x55f375a42de0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f3756449f0 0x55f375a48b90 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f375665390 0x55f375a36f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 1 --2- 172.21.15.16:0/2825153228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f37565d1b0 0x55f375a36a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 1 -- 172.21.15.16:0/2825153228 >> 172.21.15.16:0/2825153228 conn(0x55f37567a460 msgr2=0x55f375a2e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f006affd640 0 client.16386 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 1 -- 172.21.15.16:0/2825153228 shutdown_connections 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 1 -- 172.21.15.16:0/2825153228 wait complete. 2024-08-28T19:23:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.519+0000 7f00980dc740 20 client.16386 trim_cache size 0 max 16384 2024-08-28T19:23:41.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.532+0000 7f00980dc740 1 Processor -- start 2024-08-28T19:23:41.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.532+0000 7f00980dc740 1 -- start start 2024-08-28T19:23:41.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f00980dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f37565d1b0 0x55f375a287d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f00980dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a28d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f00980dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f375686070 0x55f375a33bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f00980dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1b0bc0 con 0x55f375665390 2024-08-28T19:23:41.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f00980dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1b27c0 con 0x55f37565d1b0 2024-08-28T19:23:41.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f00980dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1b1290 con 0x55f375686070 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f008b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a28d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f008b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a28d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45404/0 (socket says 172.21.15.16:45404) 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f008b7fe640 1 -- 172.21.15.16:0/3777849670 learned_addr learned my addr 172.21.15.16:0/3777849670 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f008affd640 1 --2- 172.21.15.16:0/3777849670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f37565d1b0 0x55f375a287d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f008b7fe640 1 -- 172.21.15.16:0/3777849670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f375686070 msgr2=0x55f375a33bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f008b7fe640 1 --2- 172.21.15.16:0/3777849670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f375686070 0x55f375a33bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f008b7fe640 1 -- 172.21.15.16:0/3777849670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f37565d1b0 msgr2=0x55f375a287d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f008b7fe640 1 --2- 172.21.15.16:0/3777849670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f37565d1b0 0x55f375a287d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f008b7fe640 1 -- 172.21.15.16:0/3777849670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f375a245a0 con 0x55f375665390 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f008affd640 1 --2- 172.21.15.16:0/3777849670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f37565d1b0 0x55f375a287d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.533+0000 7f008b7fe640 1 --2- 172.21.15.16:0/3777849670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a28d10 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f007c02a930 tx=0x7f007c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08534ced08a4528 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00717fa640 1 -- 172.21.15.16:0/3777849670 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c035040 con 0x55f375665390 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00717fa640 1 -- 172.21.15.16:0/3777849670 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f007c004480 con 0x55f375665390 2024-08-28T19:23:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00717fa640 1 -- 172.21.15.16:0/3777849670 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c0035f0 con 0x55f375665390 2024-08-28T19:23:41.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00980dc740 1 -- 172.21.15.16:0/3777849670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 msgr2=0x55f375a28d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:41.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00980dc740 1 --2- 172.21.15.16:0/3777849670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a28d10 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f007c02a930 tx=0x7f007c031400 comp rx=0 tx=0).stop 2024-08-28T19:23:41.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00980dc740 1 -- 172.21.15.16:0/3777849670 shutdown_connections 2024-08-28T19:23:41.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00980dc740 1 --2- 172.21.15.16:0/3777849670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f375686070 0x55f375a33bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00980dc740 1 --2- 172.21.15.16:0/3777849670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a28d10 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00980dc740 1 --2- 172.21.15.16:0/3777849670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f37565d1b0 0x55f375a287d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00980dc740 1 -- 172.21.15.16:0/3777849670 >> 172.21.15.16:0/3777849670 conn(0x55f37567a460 msgr2=0x55f375a35280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:41.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00980dc740 1 -- 172.21.15.16:0/3777849670 shutdown_connections 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.534+0000 7f00980dc740 1 -- 172.21.15.16:0/3777849670 wait complete. 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.535+0000 7f00980dc740 1 -- start start 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.535+0000 7f00980dc740 1 Processor -- start 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.535+0000 7f00980dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f37565d1b0 0x55f375a3aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.535+0000 7f00980dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a3b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.535+0000 7f00980dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f375686070 0x55f375a3b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.535+0000 7f00980dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1b2030 con 0x55f375665390 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.535+0000 7f00980dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1b1960 con 0x55f375686070 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.535+0000 7f00980dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1aeba0 con 0x55f37565d1b0 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f00980dc740 10 client.0 _ll_register_callbacks cb 0x7f008000aec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f008b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a3b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f008b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a3b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45406/0 (socket says 172.21.15.16:45406) 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f008b7fe640 1 -- 172.21.15.16:0/113530127 learned_addr learned my addr 172.21.15.16:0/113530127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f008affd640 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f37565d1b0 0x55f375a3aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f008affd640 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f375686070 0x55f375a3b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f008b7fe640 1 -- 172.21.15.16:0/113530127 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f37565d1b0 msgr2=0x55f375a3aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f008b7fe640 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f37565d1b0 0x55f375a3aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f008b7fe640 1 -- 172.21.15.16:0/113530127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f375686070 msgr2=0x55f375a3b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f008b7fe640 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f375686070 0x55f375a3b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f008b7fe640 1 -- 172.21.15.16:0/113530127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f375ab2640 con 0x55f375665390 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f008b7fe640 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a3b3f0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f007c004440 tx=0x7f007c005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e029d311b31cafa1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f006affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f006affd640 1 -- 172.21.15.16:0/113530127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c052070 con 0x55f375665390 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f006affd640 1 -- 172.21.15.16:0/113530127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f007c004750 con 0x55f375665390 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.536+0000 7f00980dc740 1 -- 172.21.15.16:0/113530127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f007c004480 con 0x55f375665390 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f006affd640 1 -- 172.21.15.16:0/113530127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f007c035070 con 0x55f375665390 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f00980dc740 10 client.16392 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f00980dc740 1 -- 172.21.15.16:0/113530127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f375ab3810 con 0x55f375665390 2024-08-28T19:23:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f006affd640 1 -- 172.21.15.16:0/113530127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f007c004480 con 0x55f375665390 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f006affd640 1 -- 172.21.15.16:0/113530127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f007c003c00 con 0x55f375665390 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f00980dc740 10 client.16392 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f00980dc740 10 client.16392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f00980dc740 20 client.16392 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f00980dc740 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f375619da0 0x55f375ab6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f00980dc740 4 client.16392 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f00980dc740 1 -- 172.21.15.16:0/113530127 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3759a0070 con 0x55f375619da0 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.537+0000 7f008affd640 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f375619da0 0x55f375ab6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f006affd640 10 client.16392 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f008affd640 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f375619da0 0x55f375ab6000 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e49a83fd265bfff server_cookie=50aee56851f42311 in_seq=0 out_seq=0 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f006affd640 1 -- 172.21.15.16:0/113530127 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3759a0070 con 0x55f375619da0 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f006affd640 10 client.16392 handle_command_reply: tid=0 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f00980dc740 1 client.16392 shutdown 2024-08-28T19:23:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f00980dc740 1 -- 172.21.15.16:0/113530127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 msgr2=0x55f375a3b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f00980dc740 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a3b3f0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f007c004440 tx=0x7f007c005040 comp rx=0 tx=0).stop 2024-08-28T19:23:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f00980dc740 1 -- 172.21.15.16:0/113530127 shutdown_connections 2024-08-28T19:23:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f00980dc740 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f375686070 0x55f375a3b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f00980dc740 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f375665390 0x55f375a3b3f0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f00980dc740 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f375619da0 0x55f375ab6000 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f00980dc740 1 --2- 172.21.15.16:0/113530127 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f37565d1b0 0x55f375a3aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f006affd640 0 client.16392 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f00980dc740 1 -- 172.21.15.16:0/113530127 >> 172.21.15.16:0/113530127 conn(0x55f37567a460 msgr2=0x55f375a2c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.538+0000 7f00980dc740 1 -- 172.21.15.16:0/113530127 shutdown_connections 2024-08-28T19:23:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f00980dc740 1 -- 172.21.15.16:0/113530127 wait complete. 2024-08-28T19:23:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f00980dc740 20 client.16392 trim_cache size 0 max 16384 2024-08-28T19:23:41.545 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:23:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f0091b67640 1 -- 172.21.15.16:0/2932991028 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00740b4ca0 msgr2=0x7f00740b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f0091b67640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00740b4ca0 0x7f00740b71b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f00840e2eb0 tx=0x7f0084093040 comp rx=0 tx=0).stop 2024-08-28T19:23:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f0091b67640 1 -- 172.21.15.16:0/2932991028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c108b10 msgr2=0x7f008c19bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:41.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f0091b67640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c108b10 0x7f008c19bb30 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f008400eaa0 tx=0x7f008400ef60 comp rx=0 tx=0).stop 2024-08-28T19:23:41.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f0091b67640 1 -- 172.21.15.16:0/2932991028 shutdown_connections 2024-08-28T19:23:41.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f0091b67640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c108b10 0x7f008c19bb30 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f0091b67640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00740b4ca0 0x7f00740b71b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f0091b67640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f008c1081c0 0x7f008c19b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f0091b67640 1 --2- 172.21.15.16:0/2932991028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f008c102880 0x7f008c19b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:41.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f0091b67640 1 -- 172.21.15.16:0/2932991028 >> 172.21.15.16:0/2932991028 conn(0x7f008c079510 msgr2=0x7f008c106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:41.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.539+0000 7f0091b67640 1 -- 172.21.15.16:0/2932991028 shutdown_connections 2024-08-28T19:23:41.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:41.540+0000 7f0091b67640 1 -- 172.21.15.16:0/2932991028 wait complete. 2024-08-28T19:23:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:41 smithi028 ceph-mon[21769]: pgmap v418: 129 pgs: 129 active+clean; 84 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.1 MiB/s wr, 78 op/s 2024-08-28T19:23:41.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:41 smithi115 ceph-mon[22184]: pgmap v418: 129 pgs: 129 active+clean; 84 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.1 MiB/s wr, 78 op/s 2024-08-28T19:23:41.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 8.53 MB/sec warmup 81 sec latency 15355.118 ms 2024-08-28T19:23:42.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 8.42 MB/sec warmup 82 sec latency 16355.182 ms 2024-08-28T19:23:43.334 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:23:43.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:23:43.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:43 smithi016 ceph-mon[18556]: pgmap v419: 129 pgs: 129 active+clean; 84 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-08-28T19:23:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:43 smithi028 ceph-mon[21769]: pgmap v419: 129 pgs: 129 active+clean; 84 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-08-28T19:23:43.604 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:43.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:43 smithi115 ceph-mon[22184]: pgmap v419: 129 pgs: 129 active+clean; 84 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-08-28T19:23:43.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 8.32 MB/sec warmup 83 sec latency 17355.281 ms 2024-08-28T19:23:44.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.069+0000 7f9cbac1f640 1 -- 172.21.15.16:0/2807672394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cb4109240 msgr2=0x7f9cb4109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.069+0000 7f9cbac1f640 1 --2- 172.21.15.16:0/2807672394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cb4109240 0x7f9cb4109650 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0009900 tx=0x7f9ca002a410 comp rx=0 tx=0).stop 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.069+0000 7f9cbac1f640 1 -- 172.21.15.16:0/2807672394 shutdown_connections 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.069+0000 7f9cbac1f640 1 --2- 172.21.15.16:0/2807672394 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9cb4109b90 0x7f9cb41103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.069+0000 7f9cbac1f640 1 --2- 172.21.15.16:0/2807672394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cb4109240 0x7f9cb4109650 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.069+0000 7f9cbac1f640 1 --2- 172.21.15.16:0/2807672394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb4103900 0x7f9cb4103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.069+0000 7f9cbac1f640 1 -- 172.21.15.16:0/2807672394 >> 172.21.15.16:0/2807672394 conn(0x7f9cb407b5f0 msgr2=0x7f9cb407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.069+0000 7f9cbac1f640 1 -- 172.21.15.16:0/2807672394 shutdown_connections 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.069+0000 7f9cbac1f640 1 -- 172.21.15.16:0/2807672394 wait complete. 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cbac1f640 1 Processor -- start 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cbac1f640 1 -- start start 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cbac1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb4103900 0x7f9cb419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cbac1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cb4109240 0x7f9cb419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cbac1f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9cb4109b90 0x7f9cb419fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cbac1f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb4113850 con 0x7f9cb4109240 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cbac1f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41136d0 con 0x7f9cb4103900 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cbac1f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41139d0 con 0x7f9cb4109b90 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cb4109240 0x7f9cb419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cb4109240 0x7f9cb419a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45424/0 (socket says 172.21.15.16:45424) 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cb3fff640 1 -- 172.21.15.16:0/1330697510 learned_addr learned my addr 172.21.15.16:0/1330697510 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb4103900 0x7f9cb419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.070+0000 7f9cb91bc640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9cb4109b90 0x7f9cb419fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.071+0000 7f9cb3fff640 1 -- 172.21.15.16:0/1330697510 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9cb4109b90 msgr2=0x7f9cb419fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.071+0000 7f9cb3fff640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9cb4109b90 0x7f9cb419fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.071+0000 7f9cb3fff640 1 -- 172.21.15.16:0/1330697510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb4103900 msgr2=0x7f9cb419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.071+0000 7f9cb3fff640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb4103900 0x7f9cb419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.071+0000 7f9cb3fff640 1 -- 172.21.15.16:0/1330697510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb41a00d0 con 0x7f9cb4109240 2024-08-28T19:23:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.071+0000 7f9cb3fff640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cb4109240 0x7f9cb419a7c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9ca002a930 tx=0x7f9ca0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.071+0000 7f9cb1ffb640 1 -- 172.21.15.16:0/1330697510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca003f030 con 0x7f9cb4109240 2024-08-28T19:23:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.071+0000 7f9cb1ffb640 1 -- 172.21.15.16:0/1330697510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ca0003aa0 con 0x7f9cb4109240 2024-08-28T19:23:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.071+0000 7f9cbac1f640 1 -- 172.21.15.16:0/1330697510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cb41a0360 con 0x7f9cb4109240 2024-08-28T19:23:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.071+0000 7f9cb1ffb640 1 -- 172.21.15.16:0/1330697510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0035070 con 0x7f9cb4109240 2024-08-28T19:23:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.071+0000 7f9cbac1f640 1 -- 172.21.15.16:0/1330697510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cb41a0840 con 0x7f9cb4109240 2024-08-28T19:23:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.074+0000 7f9cc1194740 1 Processor -- start 2024-08-28T19:23:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.074+0000 7f9cb1ffb640 1 -- 172.21.15.16:0/1330697510 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f9ca0042030 con 0x7f9cb4109240 2024-08-28T19:23:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.074+0000 7f9cc1194740 1 -- start start 2024-08-28T19:23:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.074+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602433350 0x5556027d9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.074+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55560240a490 0x5556027d9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.074+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602412670 0x5556027da200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.074+0000 7f9cc1194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55560274f5f0 con 0x555602412670 2024-08-28T19:23:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.074+0000 7f9cc1194740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55560273eb80 con 0x555602433350 2024-08-28T19:23:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.074+0000 7f9cc1194740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555602742b30 con 0x55560240a490 2024-08-28T19:23:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602412670 0x5556027da200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb91bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602433350 0x5556027d9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb91bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602433350 0x5556027d9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51574/0 (socket says 172.21.15.16:51574) 2024-08-28T19:23:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb91bc640 1 -- 172.21.15.16:0/3275954203 learned_addr learned my addr 172.21.15.16:0/3275954203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/3275954203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55560240a490 0x5556027d9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602412670 0x5556027da200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45430/0 (socket says 172.21.15.16:45430) 2024-08-28T19:23:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb91bc640 1 -- 172.21.15.16:0/3275954203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55560240a490 msgr2=0x5556027d9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb91bc640 1 --2- 172.21.15.16:0/3275954203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55560240a490 0x5556027d9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb91bc640 1 -- 172.21.15.16:0/3275954203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602412670 msgr2=0x5556027da200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb91bc640 1 --2- 172.21.15.16:0/3275954203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602412670 0x5556027da200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb1ffb640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ca40b4cd0 0x7f9ca40b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb91bc640 1 -- 172.21.15.16:0/3275954203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55560249e930 con 0x555602433350 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/3275954203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55560240a490 0x5556027d9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ca40b4cd0 0x7f9ca40b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb1ffb640 1 -- 172.21.15.16:0/1330697510 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f9ca0105f00 con 0x7f9cb4109240 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.075+0000 7f9cb91bc640 1 --2- 172.21.15.16:0/3275954203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602433350 0x5556027d9780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9cac010370 tx=0x7f9cac010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2869f9cee977cf97 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.076+0000 7f9c9effd640 1 -- 172.21.15.16:0/3275954203 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac016070 con 0x555602433350 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.076+0000 7f9c9effd640 1 -- 172.21.15.16:0/3275954203 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cac004d20 con 0x555602433350 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.076+0000 7f9c9effd640 1 -- 172.21.15.16:0/3275954203 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac005350 con 0x555602433350 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.076+0000 7f9cc1194740 1 -- 172.21.15.16:0/3275954203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602433350 msgr2=0x5556027d9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.076+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ca40b4cd0 0x7f9ca40b71e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x5556027d8400 tx=0x7f9ca800a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.076+0000 7f9cc1194740 1 --2- 172.21.15.16:0/3275954203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602433350 0x5556027d9780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9cac010370 tx=0x7f9cac010830 comp rx=0 tx=0).stop 2024-08-28T19:23:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.077+0000 7f9cc1194740 1 -- 172.21.15.16:0/3275954203 shutdown_connections 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.077+0000 7f9cc1194740 1 --2- 172.21.15.16:0/3275954203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602412670 0x5556027da200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.077+0000 7f9cc1194740 1 --2- 172.21.15.16:0/3275954203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55560240a490 0x5556027d9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.077+0000 7f9cc1194740 1 --2- 172.21.15.16:0/3275954203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602433350 0x5556027d9780 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.077+0000 7f9cc1194740 1 -- 172.21.15.16:0/3275954203 >> 172.21.15.16:0/3275954203 conn(0x555602427740 msgr2=0x5556027d7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.077+0000 7f9cc1194740 1 -- 172.21.15.16:0/3275954203 shutdown_connections 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.077+0000 7f9cc1194740 1 -- 172.21.15.16:0/3275954203 wait complete. 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.077+0000 7f9cc1194740 1 -- start start 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.077+0000 7f9cc1194740 1 Processor -- start 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602412670 0x5556027e5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602433350 0x5556027f1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cc1194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5556025a1890 con 0x55560240a490 2024-08-28T19:23:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cc1194740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5556026c5790 con 0x555602412670 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cc1194740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5556027e7f60 con 0x555602433350 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cc1194740 10 client.0 _ll_register_callbacks cb 0x55560244aba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cb89bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cb89bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45444/0 (socket says 172.21.15.16:45444) 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cb89bb640 1 -- 172.21.15.16:0/1594140997 learned_addr learned my addr 172.21.15.16:0/1594140997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cb3fff640 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602412670 0x5556027e5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cb91bc640 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602433350 0x5556027f1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cb89bb640 1 -- 172.21.15.16:0/1594140997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602433350 msgr2=0x5556027f1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602433350 0x5556027f1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cb89bb640 1 -- 172.21.15.16:0/1594140997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602412670 msgr2=0x5556027e5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602412670 0x5556027e5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cb89bb640 1 -- 172.21.15.16:0/1594140997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556026fcf30 con 0x55560240a490 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e4f00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8011010 tx=0x7f9ca8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a2cede5eebbcfc3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9c93fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:44.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9c93fff640 1 -- 172.21.15.16:0/1594140997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca802c060 con 0x55560240a490 2024-08-28T19:23:44.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9c93fff640 1 -- 172.21.15.16:0/1594140997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f9ca8016030 con 0x55560240a490 2024-08-28T19:23:44.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.078+0000 7f9cc1194740 1 -- 172.21.15.16:0/1594140997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cac004d20 con 0x55560240a490 2024-08-28T19:23:44.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.079+0000 7f9c93fff640 1 -- 172.21.15.16:0/1594140997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9ca8033050 con 0x55560240a490 2024-08-28T19:23:44.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.079+0000 7f9cc1194740 10 client.16404 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:44.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.079+0000 7f9cc1194740 1 -- 172.21.15.16:0/1594140997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5556027f3dd0 con 0x55560240a490 2024-08-28T19:23:44.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.079+0000 7f9c93fff640 1 -- 172.21.15.16:0/1594140997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cac004d20 con 0x55560240a490 2024-08-28T19:23:44.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.079+0000 7f9c93fff640 1 -- 172.21.15.16:0/1594140997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f9ca801a4c0 con 0x55560240a490 2024-08-28T19:23:44.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.079+0000 7f9cc1194740 10 client.16404 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:44.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.079+0000 7f9cc1194740 10 client.16404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:23:44.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.079+0000 7f9cc1194740 20 client.16404 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.079+0000 7f9cc1194740 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5556023f1cd0 0x5556027f7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.079+0000 7f9cc1194740 4 client.16404 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.079+0000 7f9cc1194740 1 -- 172.21.15.16:0/1594140997 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555602751540 con 0x5556023f1cd0 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.080+0000 7f9cb3fff640 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5556023f1cd0 0x5556027f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.080+0000 7f9cb3fff640 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5556023f1cd0 0x5556027f7330 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a81a8062277593a server_cookie=527c7854098f1f7d in_seq=0 out_seq=0 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.080+0000 7f9c93fff640 10 client.16404 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9c93fff640 1 -- 172.21.15.16:0/1594140997 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555602751540 con 0x5556023f1cd0 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9c93fff640 10 client.16404 handle_command_reply: tid=0 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 1 client.16404 shutdown 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 1 -- 172.21.15.16:0/1594140997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 msgr2=0x5556027e4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e4f00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8011010 tx=0x7f9ca8004730 comp rx=0 tx=0).stop 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 1 -- 172.21.15.16:0/1594140997 shutdown_connections 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5556023f1cd0 0x5556027f7330 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9c93fff640 0 client.16404 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602433350 0x5556027f1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602412670 0x5556027e5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 1 --2- 172.21.15.16:0/1594140997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e4f00 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 1 -- 172.21.15.16:0/1594140997 >> 172.21.15.16:0/1594140997 conn(0x555602427740 msgr2=0x5556027dc910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:44.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 1 -- 172.21.15.16:0/1594140997 shutdown_connections 2024-08-28T19:23:44.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 1 -- 172.21.15.16:0/1594140997 wait complete. 2024-08-28T19:23:44.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.081+0000 7f9cc1194740 20 client.16404 trim_cache size 0 max 16384 2024-08-28T19:23:44.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.099+0000 7f9cc1194740 1 Processor -- start 2024-08-28T19:23:44.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.099+0000 7f9cc1194740 1 -- start start 2024-08-28T19:23:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.099+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.099+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602412670 0x5556027f1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.099+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602433350 0x5556027f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.099+0000 7f9cc1194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41ae8b0 con 0x55560240a490 2024-08-28T19:23:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.099+0000 7f9cc1194740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41b04b0 con 0x555602412670 2024-08-28T19:23:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.099+0000 7f9cc1194740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41aef80 con 0x555602433350 2024-08-28T19:23:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb89bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb89bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45452/0 (socket says 172.21.15.16:45452) 2024-08-28T19:23:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb89bb640 1 -- 172.21.15.16:0/705269427 learned_addr learned my addr 172.21.15.16:0/705269427 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb91bc640 1 --2- 172.21.15.16:0/705269427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602433350 0x5556027f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb3fff640 1 --2- 172.21.15.16:0/705269427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602412670 0x5556027f1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb89bb640 1 -- 172.21.15.16:0/705269427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602433350 msgr2=0x5556027f16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/705269427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602433350 0x5556027f16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb89bb640 1 -- 172.21.15.16:0/705269427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602412670 msgr2=0x5556027f1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/705269427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602412670 0x5556027f1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb89bb640 1 -- 172.21.15.16:0/705269427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556027dba70 con 0x55560240a490 2024-08-28T19:23:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb3fff640 1 --2- 172.21.15.16:0/705269427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602412670 0x5556027f1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:23:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/705269427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e6b30 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8017040 tx=0x7f9ca8010f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16cefb9882bfb4e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9c9e7fc640 1 -- 172.21.15.16:0/705269427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca801cc80 con 0x55560240a490 2024-08-28T19:23:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9c9e7fc640 1 -- 172.21.15.16:0/705269427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ca803c070 con 0x55560240a490 2024-08-28T19:23:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.100+0000 7f9c9e7fc640 1 -- 172.21.15.16:0/705269427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca801a5e0 con 0x55560240a490 2024-08-28T19:23:44.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.101+0000 7f9cc1194740 1 -- 172.21.15.16:0/705269427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 msgr2=0x5556027e6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:44.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.101+0000 7f9cc1194740 1 --2- 172.21.15.16:0/705269427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e6b30 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8017040 tx=0x7f9ca8010f60 comp rx=0 tx=0).stop 2024-08-28T19:23:44.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.101+0000 7f9cc1194740 1 -- 172.21.15.16:0/705269427 shutdown_connections 2024-08-28T19:23:44.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.101+0000 7f9cc1194740 1 --2- 172.21.15.16:0/705269427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602433350 0x5556027f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.101+0000 7f9cc1194740 1 --2- 172.21.15.16:0/705269427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555602412670 0x5556027f1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.101+0000 7f9cc1194740 1 --2- 172.21.15.16:0/705269427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55560240a490 0x5556027e6b30 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.101+0000 7f9cc1194740 1 -- 172.21.15.16:0/705269427 >> 172.21.15.16:0/705269427 conn(0x555602427740 msgr2=0x5556027e6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.101+0000 7f9cc1194740 1 -- 172.21.15.16:0/705269427 shutdown_connections 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.101+0000 7f9cc1194740 1 -- 172.21.15.16:0/705269427 wait complete. 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cc1194740 1 -- start start 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cc1194740 1 Processor -- start 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55560240a490 0x5556027edac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602412670 0x5556027e9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cc1194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602433350 0x5556027ea5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cc1194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41afd20 con 0x555602433350 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cc1194740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41af650 con 0x55560240a490 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cc1194740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41ac890 con 0x555602412670 2024-08-28T19:23:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cc1194740 10 client.0 _ll_register_callbacks cb 0x7f9ca8005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cb91bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602433350 0x5556027ea5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cb91bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602433350 0x5556027ea5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45458/0 (socket says 172.21.15.16:45458) 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cb91bc640 1 -- 172.21.15.16:0/667830754 learned_addr learned my addr 172.21.15.16:0/667830754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.102+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55560240a490 0x5556027edac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9cb3fff640 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602412670 0x5556027e9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9cb91bc640 1 -- 172.21.15.16:0/667830754 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602412670 msgr2=0x5556027e9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9cb91bc640 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602412670 0x5556027e9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9cb91bc640 1 -- 172.21.15.16:0/667830754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55560240a490 msgr2=0x5556027edac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9cb91bc640 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55560240a490 0x5556027edac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9cb91bc640 1 -- 172.21.15.16:0/667830754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556027ead40 con 0x555602433350 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55560240a490 0x5556027edac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9cb91bc640 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602433350 0x5556027ea5b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9cac010370 tx=0x7f9cac004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377a9094416a84a2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9c93fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9c93fff640 1 -- 172.21.15.16:0/667830754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac016070 con 0x555602433350 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9c93fff640 1 -- 172.21.15.16:0/667830754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f9cac012710 con 0x555602433350 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9cc1194740 1 -- 172.21.15.16:0/667830754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ca803c070 con 0x555602433350 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9c93fff640 1 -- 172.21.15.16:0/667830754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9cac035050 con 0x555602433350 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9cc1194740 10 client.16410 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.103+0000 7f9cc1194740 1 -- 172.21.15.16:0/667830754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555602862cf0 con 0x555602433350 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.104+0000 7f9c93fff640 1 -- 172.21.15.16:0/667830754 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ca803c070 con 0x555602433350 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.104+0000 7f9c93fff640 1 -- 172.21.15.16:0/667830754 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f9cac00f040 con 0x555602433350 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.104+0000 7f9cc1194740 10 client.16410 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.104+0000 7f9cc1194740 10 client.16410 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.104+0000 7f9cc1194740 20 client.16410 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.104+0000 7f9cc1194740 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5556023c7080 0x555602865160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.104+0000 7f9cc1194740 4 client.16410 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:23:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.104+0000 7f9cc1194740 1 -- 172.21.15.16:0/667830754 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55560274cfe0 con 0x5556023c7080 2024-08-28T19:23:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.104+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5556023c7080 0x555602865160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.105+0000 7f9cb89bb640 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5556023c7080 0x555602865160 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=608cffd88faf695d server_cookie=9029807968bfaf0a in_seq=0 out_seq=0 2024-08-28T19:23:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.105+0000 7f9c93fff640 10 client.16410 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.106+0000 7f9c93fff640 1 -- 172.21.15.16:0/667830754 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55560274cfe0 con 0x5556023c7080 2024-08-28T19:23:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.106+0000 7f9c93fff640 10 client.16410 handle_command_reply: tid=0 2024-08-28T19:23:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.106+0000 7f9cc1194740 1 client.16410 shutdown 2024-08-28T19:23:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.106+0000 7f9cc1194740 1 -- 172.21.15.16:0/667830754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602433350 msgr2=0x5556027ea5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.106+0000 7f9cc1194740 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602433350 0x5556027ea5b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9cac010370 tx=0x7f9cac004bb0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cc1194740 1 -- 172.21.15.16:0/667830754 shutdown_connections 2024-08-28T19:23:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cc1194740 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555602433350 0x5556027ea5b0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cc1194740 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5556023c7080 0x555602865160 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cc1194740 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555602412670 0x5556027e9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cc1194740 1 --2- 172.21.15.16:0/667830754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55560240a490 0x5556027edac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9c93fff640 0 client.16410 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cc1194740 1 -- 172.21.15.16:0/667830754 >> 172.21.15.16:0/667830754 conn(0x555602427740 msgr2=0x5556027e63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cc1194740 1 -- 172.21.15.16:0/667830754 shutdown_connections 2024-08-28T19:23:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cc1194740 1 -- 172.21.15.16:0/667830754 wait complete. 2024-08-28T19:23:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cc1194740 20 client.16410 trim_cache size 0 max 16384 2024-08-28T19:23:44.112 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:23:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:23:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2", 2024-08-28T19:23:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:23:44.113 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:23:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cbac1f640 1 -- 172.21.15.16:0/1330697510 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ca40b4cd0 msgr2=0x7f9ca40b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cbac1f640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ca40b4cd0 0x7f9ca40b71e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x5556027d8400 tx=0x7f9ca800a2d0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cbac1f640 1 -- 172.21.15.16:0/1330697510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cb4109240 msgr2=0x7f9cb419a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cbac1f640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cb4109240 0x7f9cb419a7c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9ca002a930 tx=0x7f9ca0004270 comp rx=0 tx=0).stop 2024-08-28T19:23:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cbac1f640 1 -- 172.21.15.16:0/1330697510 shutdown_connections 2024-08-28T19:23:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cbac1f640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9cb4109b90 0x7f9cb419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cbac1f640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cb4109240 0x7f9cb419a7c0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cbac1f640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ca40b4cd0 0x7f9ca40b71e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cbac1f640 1 --2- 172.21.15.16:0/1330697510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb4103900 0x7f9cb419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.107+0000 7f9cbac1f640 1 -- 172.21.15.16:0/1330697510 >> 172.21.15.16:0/1330697510 conn(0x7f9cb407b5f0 msgr2=0x7f9cb4107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.108+0000 7f9cbac1f640 1 -- 172.21.15.16:0/1330697510 shutdown_connections 2024-08-28T19:23:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:44.108+0000 7f9cbac1f640 1 -- 172.21.15.16:0/1330697510 wait complete. 2024-08-28T19:23:44.901 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:23:44.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 8.22 MB/sec warmup 84 sec latency 18355.343 ms 2024-08-28T19:23:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:45.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: pgmap v420: 129 pgs: 129 active+clean; 85 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1.2 MiB/s wr, 51 op/s 2024-08-28T19:23:45.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:45 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: pgmap v420: 129 pgs: 129 active+clean; 85 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1.2 MiB/s wr, 51 op/s 2024-08-28T19:23:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:23:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:45 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: pgmap v420: 129 pgs: 129 active+clean; 85 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1.2 MiB/s wr, 51 op/s 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:45 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.638+0000 7f0049cc7640 1 -- 172.21.15.16:0/678713702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0044102820 msgr2=0x7f0044102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.638+0000 7f0049cc7640 1 --2- 172.21.15.16:0/678713702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0044102820 0x7f0044102c30 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0034009900 tx=0x7f003402a410 comp rx=0 tx=0).stop 2024-08-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.638+0000 7f0049cc7640 1 -- 172.21.15.16:0/678713702 shutdown_connections 2024-08-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.638+0000 7f0049cc7640 1 --2- 172.21.15.16:0/678713702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0044108ab0 0x7f004410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.638+0000 7f0049cc7640 1 --2- 172.21.15.16:0/678713702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0044108160 0x7f0044108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.638+0000 7f0049cc7640 1 --2- 172.21.15.16:0/678713702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0044102820 0x7f0044102c30 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.638+0000 7f0049cc7640 1 -- 172.21.15.16:0/678713702 >> 172.21.15.16:0/678713702 conn(0x7f00440794e0 msgr2=0x7f00440798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.638+0000 7f0049cc7640 1 -- 172.21.15.16:0/678713702 shutdown_connections 2024-08-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.638+0000 7f0049cc7640 1 -- 172.21.15.16:0/678713702 wait complete. 2024-08-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.639+0000 7f0049cc7640 1 Processor -- start 2024-08-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.639+0000 7f0049cc7640 1 -- start start 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.639+0000 7f0049cc7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0044102820 0x7f004419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.639+0000 7f0049cc7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0044108160 0x7f004419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.639+0000 7f0049cc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0044108ab0 0x7f004419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.639+0000 7f0049cc7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0044113840 con 0x7f0044108ab0 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.639+0000 7f0049cc7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00441136c0 con 0x7f0044108160 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.639+0000 7f0049cc7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00441139c0 con 0x7f0044102820 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0043fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0044108ab0 0x7f004419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0043fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0044108ab0 0x7f004419fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55842/0 (socket says 172.21.15.16:55842) 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0043fff640 1 -- 172.21.15.16:0/4078971799 learned_addr learned my addr 172.21.15.16:0/4078971799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0042ffd640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0044108160 0x7f004419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f00437fe640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0044102820 0x7f004419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0043fff640 1 -- 172.21.15.16:0/4078971799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0044102820 msgr2=0x7f004419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0043fff640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0044102820 0x7f004419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0043fff640 1 -- 172.21.15.16:0/4078971799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0044108160 msgr2=0x7f004419a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0043fff640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0044108160 0x7f004419a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0043fff640 1 -- 172.21.15.16:0/4078971799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00441a01e0 con 0x7f0044108ab0 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0042ffd640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0044108160 0x7f004419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0043fff640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0044108ab0 0x7f004419fab0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f003c00eb10 tx=0x7f003c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0040ff9640 1 -- 172.21.15.16:0/4078971799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c004b20 con 0x7f0044108ab0 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0040ff9640 1 -- 172.21.15.16:0/4078971799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f003c004cc0 con 0x7f0044108ab0 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0049cc7640 1 -- 172.21.15.16:0/4078971799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00441a04d0 con 0x7f0044108ab0 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.640+0000 7f0040ff9640 1 -- 172.21.15.16:0/4078971799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c010670 con 0x7f0044108ab0 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.641+0000 7f0049cc7640 1 -- 172.21.15.16:0/4078971799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00441a6c30 con 0x7f0044108ab0 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.643+0000 7f005023c740 1 Processor -- start 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.643+0000 7f0040ff9640 1 -- 172.21.15.16:0/4078971799 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f003c010810 con 0x7f0044108ab0 2024-08-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.643+0000 7f005023c740 1 -- start start 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.643+0000 7f005023c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 0x555f064612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.643+0000 7f005023c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06461800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.643+0000 7f005023c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0609b390 0x555f06461d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.643+0000 7f005023c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555f063d8630 con 0x555f060931b0 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.643+0000 7f005023c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555f063c7bc0 con 0x555f0609b390 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.643+0000 7f005023c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555f063cbb70 con 0x555f060bc070 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f00437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06461800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f00437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06461800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55854/0 (socket says 172.21.15.16:55854) 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f00437fe640 1 -- 172.21.15.16:0/1860912993 learned_addr learned my addr 172.21.15.16:0/1860912993 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f0042ffd640 1 --2- 172.21.15.16:0/1860912993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0609b390 0x555f06461d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f0042ffd640 1 --2- 172.21.15.16:0/1860912993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 0x555f064612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f00437fe640 1 -- 172.21.15.16:0/1860912993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 msgr2=0x555f064612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f00437fe640 1 --2- 172.21.15.16:0/1860912993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 0x555f064612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f00437fe640 1 -- 172.21.15.16:0/1860912993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0609b390 msgr2=0x555f06461d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f00437fe640 1 --2- 172.21.15.16:0/1860912993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0609b390 0x555f06461d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f00437fe640 1 -- 172.21.15.16:0/1860912993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f06127650 con 0x555f060931b0 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f0042ffd640 1 --2- 172.21.15.16:0/1860912993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 0x555f064612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f0040ff9640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f002c0b4ca0 0x7f002c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f00437fe640 1 --2- 172.21.15.16:0/1860912993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06461800 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0034009900 tx=0x7f0034032c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf6e7d52e0518ce server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f0042ffd640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f002c0b4ca0 0x7f002c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f0029ffb640 1 -- 172.21.15.16:0/1860912993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0034040070 con 0x555f060931b0 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f0040ff9640 1 -- 172.21.15.16:0/4078971799 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f003c0e5df0 con 0x7f0044108ab0 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f0029ffb640 1 -- 172.21.15.16:0/1860912993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0034004030 con 0x555f060931b0 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.644+0000 7f0029ffb640 1 -- 172.21.15.16:0/1860912993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0034035040 con 0x555f060931b0 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.645+0000 7f005023c740 1 -- 172.21.15.16:0/1860912993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 msgr2=0x555f06461800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.645+0000 7f005023c740 1 --2- 172.21.15.16:0/1860912993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06461800 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0034009900 tx=0x7f0034032c40 comp rx=0 tx=0).stop 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.645+0000 7f0042ffd640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f002c0b4ca0 0x7f002c0b71b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0044100a80 tx=0x7f0038014040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.645+0000 7f005023c740 1 -- 172.21.15.16:0/1860912993 shutdown_connections 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.645+0000 7f005023c740 1 --2- 172.21.15.16:0/1860912993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0609b390 0x555f06461d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.645+0000 7f005023c740 1 --2- 172.21.15.16:0/1860912993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06461800 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.645+0000 7f005023c740 1 --2- 172.21.15.16:0/1860912993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 0x555f064612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.645+0000 7f005023c740 1 -- 172.21.15.16:0/1860912993 >> 172.21.15.16:0/1860912993 conn(0x555f060b0460 msgr2=0x555f0645f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.645+0000 7f005023c740 1 -- 172.21.15.16:0/1860912993 shutdown_connections 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.645+0000 7f005023c740 1 -- 172.21.15.16:0/1860912993 wait complete. 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.646+0000 7f005023c740 1 -- start start 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.646+0000 7f005023c740 1 Processor -- start 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.646+0000 7f005023c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06470dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.646+0000 7f005023c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0609b390 0x555f06471300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.646+0000 7f005023c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060bc070 0x555f0647d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.646+0000 7f005023c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555f0622a700 con 0x555f060931b0 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.646+0000 7f005023c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555f0634e700 con 0x555f060bc070 2024-08-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f005023c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555f06473da0 con 0x555f0609b390 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f005023c740 10 client.0 _ll_register_callbacks cb 0x555f0637ddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f00437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06470dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f00437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06470dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55864/0 (socket says 172.21.15.16:55864) 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f00437fe640 1 -- 172.21.15.16:0/732356502 learned_addr learned my addr 172.21.15.16:0/732356502 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f0043fff640 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060bc070 0x555f0647d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f0042ffd640 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0609b390 0x555f06471300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f00437fe640 1 -- 172.21.15.16:0/732356502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0609b390 msgr2=0x555f06471300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f00437fe640 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0609b390 0x555f06471300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f00437fe640 1 -- 172.21.15.16:0/732356502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060bc070 msgr2=0x555f0647d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f00437fe640 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060bc070 0x555f0647d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f00437fe640 1 -- 172.21.15.16:0/732356502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f063878b0 con 0x555f060931b0 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f0043fff640 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060bc070 0x555f0647d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f00437fe640 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06470dc0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f003402a930 tx=0x7f00340051c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a3220bb194b4c3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f0022ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f0022ffd640 1 -- 172.21.15.16:0/732356502 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003404d060 con 0x555f060931b0 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f0022ffd640 1 -- 172.21.15.16:0/732356502 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f00340314a0 con 0x555f060931b0 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f005023c740 1 -- 172.21.15.16:0/732356502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0034004030 con 0x555f060931b0 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f0022ffd640 1 -- 172.21.15.16:0/732356502 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0034054050 con 0x555f060931b0 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.647+0000 7f005023c740 10 client.16422 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.648+0000 7f005023c740 1 -- 172.21.15.16:0/732356502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555f0647fa90 con 0x555f060931b0 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.648+0000 7f0022ffd640 1 -- 172.21.15.16:0/732356502 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0034004030 con 0x555f060931b0 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.648+0000 7f0022ffd640 1 -- 172.21.15.16:0/732356502 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f00340035f0 con 0x555f060931b0 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.648+0000 7f005023c740 10 client.16422 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.648+0000 7f005023c740 10 client.16422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.648+0000 7f005023c740 20 client.16422 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.648+0000 7f005023c740 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555f0607a9f0 0x555f06482df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.648+0000 7f005023c740 4 client.16422 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.648+0000 7f005023c740 1 -- 172.21.15.16:0/732356502 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f063da580 con 0x555f0607a9f0 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.648+0000 7f0042ffd640 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555f0607a9f0 0x555f06482df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.649+0000 7f0042ffd640 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555f0607a9f0 0x555f06482df0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=941bfd6922c65ed1 server_cookie=fbcae44d9b7dbcf6 in_seq=0 out_seq=0 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.649+0000 7f0022ffd640 10 client.16422 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f0022ffd640 1 -- 172.21.15.16:0/732356502 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555f063da580 con 0x555f0607a9f0 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f0022ffd640 10 client.16422 handle_command_reply: tid=0 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 1 client.16422 shutdown 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 1 -- 172.21.15.16:0/732356502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 msgr2=0x555f06470dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06470dc0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f003402a930 tx=0x7f00340051c0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 1 -- 172.21.15.16:0/732356502 shutdown_connections 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060bc070 0x555f0647d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555f0607a9f0 0x555f06482df0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0609b390 0x555f06471300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 1 --2- 172.21.15.16:0/732356502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f060931b0 0x555f06470dc0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 1 -- 172.21.15.16:0/732356502 >> 172.21.15.16:0/732356502 conn(0x555f060b0460 msgr2=0x555f06462cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f0022ffd640 0 client.16422 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 1 -- 172.21.15.16:0/732356502 shutdown_connections 2024-08-28T19:23:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 1 -- 172.21.15.16:0/732356502 wait complete. 2024-08-28T19:23:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.650+0000 7f005023c740 20 client.16422 trim_cache size 0 max 16384 2024-08-28T19:23:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f005023c740 1 Processor -- start 2024-08-28T19:23:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f005023c740 1 -- start start 2024-08-28T19:23:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f005023c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060931b0 0x555f0646f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f005023c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f0646fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f005023c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060bc070 0x555f0646dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f005023c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00441ae8e0 con 0x555f0609b390 2024-08-28T19:23:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f005023c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00441b04e0 con 0x555f060bc070 2024-08-28T19:23:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f005023c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00441aefb0 con 0x555f060931b0 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f0042ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f0646fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f0042ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f0646fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55872/0 (socket says 172.21.15.16:55872) 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f0042ffd640 1 -- 172.21.15.16:0/786194285 learned_addr learned my addr 172.21.15.16:0/786194285 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f00437fe640 1 --2- 172.21.15.16:0/786194285 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060931b0 0x555f0646f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f0043fff640 1 --2- 172.21.15.16:0/786194285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060bc070 0x555f0646dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f0042ffd640 1 -- 172.21.15.16:0/786194285 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060931b0 msgr2=0x555f0646f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f0042ffd640 1 --2- 172.21.15.16:0/786194285 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060931b0 0x555f0646f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f0042ffd640 1 -- 172.21.15.16:0/786194285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060bc070 msgr2=0x555f0646dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f0042ffd640 1 --2- 172.21.15.16:0/786194285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060bc070 0x555f0646dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.668+0000 7f0042ffd640 1 -- 172.21.15.16:0/786194285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f0645a550 con 0x555f0609b390 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.669+0000 7f00437fe640 1 --2- 172.21.15.16:0/786194285 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060931b0 0x555f0646f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.669+0000 7f0042ffd640 1 --2- 172.21.15.16:0/786194285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f0646fb80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f0038011890 tx=0x7f0038016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3948960e33c589c0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.669+0000 7f00297fa640 1 -- 172.21.15.16:0/786194285 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003801c970 con 0x555f0609b390 2024-08-28T19:23:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.669+0000 7f00297fa640 1 -- 172.21.15.16:0/786194285 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0038029470 con 0x555f0609b390 2024-08-28T19:23:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.669+0000 7f00297fa640 1 -- 172.21.15.16:0/786194285 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003802f5f0 con 0x555f0609b390 2024-08-28T19:23:45.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.670+0000 7f005023c740 1 -- 172.21.15.16:0/786194285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 msgr2=0x555f0646fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:45.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.670+0000 7f005023c740 1 --2- 172.21.15.16:0/786194285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f0646fb80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f0038011890 tx=0x7f0038016000 comp rx=0 tx=0).stop 2024-08-28T19:23:45.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.670+0000 7f005023c740 1 -- 172.21.15.16:0/786194285 shutdown_connections 2024-08-28T19:23:45.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.670+0000 7f005023c740 1 --2- 172.21.15.16:0/786194285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060bc070 0x555f0646dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.670+0000 7f005023c740 1 --2- 172.21.15.16:0/786194285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f0646fb80 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.670+0000 7f005023c740 1 --2- 172.21.15.16:0/786194285 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060931b0 0x555f0646f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.670+0000 7f005023c740 1 -- 172.21.15.16:0/786194285 >> 172.21.15.16:0/786194285 conn(0x555f060b0460 msgr2=0x555f0646bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.670+0000 7f005023c740 1 -- 172.21.15.16:0/786194285 shutdown_connections 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.670+0000 7f005023c740 1 -- 172.21.15.16:0/786194285 wait complete. 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.670+0000 7f005023c740 1 -- start start 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.670+0000 7f005023c740 1 Processor -- start 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f005023c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060931b0 0x555f06483f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f005023c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f064844c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f005023c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 0x555f06484a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f005023c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00441afd50 con 0x555f0609b390 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f005023c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00441af680 con 0x555f060931b0 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f005023c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00441ac8c0 con 0x555f060bc070 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f005023c740 10 client.0 _ll_register_callbacks cb 0x7f0034032b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0042ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f064844c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0042ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f064844c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55886/0 (socket says 172.21.15.16:55886) 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0042ffd640 1 -- 172.21.15.16:0/2678539540 learned_addr learned my addr 172.21.15.16:0/2678539540 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f00437fe640 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060931b0 0x555f06483f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0043fff640 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 0x555f06484a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0042ffd640 1 -- 172.21.15.16:0/2678539540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 msgr2=0x555f06484a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0042ffd640 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 0x555f06484a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0042ffd640 1 -- 172.21.15.16:0/2678539540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060931b0 msgr2=0x555f06483f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0042ffd640 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060931b0 0x555f06483f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0042ffd640 1 -- 172.21.15.16:0/2678539540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f064ec6d0 con 0x555f0609b390 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0043fff640 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 0x555f06484a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0042ffd640 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f064844c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f003800bd60 tx=0x7f0038011fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3666138ff5baecf server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.671+0000 7f0022ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f0022ffd640 1 -- 172.21.15.16:0/2678539540 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0038004030 con 0x555f0609b390 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f0022ffd640 1 -- 172.21.15.16:0/2678539540 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f00380041d0 con 0x555f0609b390 2024-08-28T19:23:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f005023c740 1 -- 172.21.15.16:0/2678539540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0038029470 con 0x555f0609b390 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f0022ffd640 1 -- 172.21.15.16:0/2678539540 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0038041050 con 0x555f0609b390 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f005023c740 10 client.16428 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f005023c740 1 -- 172.21.15.16:0/2678539540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555f064ed960 con 0x555f0609b390 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f0022ffd640 1 -- 172.21.15.16:0/2678539540 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0038029470 con 0x555f0609b390 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f0022ffd640 1 -- 172.21.15.16:0/2678539540 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f0038005450 con 0x555f0609b390 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f005023c740 10 client.16428 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f005023c740 10 client.16428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f005023c740 20 client.16428 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f005023c740 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555f0604fda0 0x555f064f0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f005023c740 4 client.16428 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.672+0000 7f005023c740 1 -- 172.21.15.16:0/2678539540 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555f063d6020 con 0x555f0604fda0 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f00437fe640 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555f0604fda0 0x555f064f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f00437fe640 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555f0604fda0 0x555f064f0120 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce462d32096d0a87 server_cookie=8e60b8d7268dd0bc in_seq=0 out_seq=0 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f0022ffd640 10 client.16428 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f0022ffd640 1 -- 172.21.15.16:0/2678539540 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1706 (crc 0 0 0) 0x555f063d6020 con 0x555f0604fda0 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f0022ffd640 10 client.16428 handle_command_reply: tid=0 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f005023c740 1 client.16428 shutdown 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f005023c740 1 -- 172.21.15.16:0/2678539540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 msgr2=0x555f064844c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f005023c740 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f064844c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f003800bd60 tx=0x7f0038011fd0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f005023c740 1 -- 172.21.15.16:0/2678539540 shutdown_connections 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f005023c740 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555f0604fda0 0x555f064f0120 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f005023c740 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f060bc070 0x555f06484a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f005023c740 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0609b390 0x555f064844c0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f0022ffd640 0 client.16428 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f005023c740 1 --2- 172.21.15.16:0/2678539540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f060931b0 0x555f06483f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.673+0000 7f005023c740 1 -- 172.21.15.16:0/2678539540 >> 172.21.15.16:0/2678539540 conn(0x555f060b0460 msgr2=0x555f0646f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f005023c740 1 -- 172.21.15.16:0/2678539540 shutdown_connections 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f005023c740 1 -- 172.21.15.16:0/2678539540 wait complete. 2024-08-28T19:23:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f005023c740 20 client.16428 trim_cache size 0 max 16384 2024-08-28T19:23:45.680 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:23:45.680 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:23:45.680 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:23:45.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:23:45.680 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:45.680 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2", 2024-08-28T19:23:45.680 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:45.681 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:23:45.682 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:45.683 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:23:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 -- 172.21.15.16:0/4078971799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f002c0b4ca0 msgr2=0x7f002c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f002c0b4ca0 0x7f002c0b71b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0044100a80 tx=0x7f0038014040 comp rx=0 tx=0).stop 2024-08-28T19:23:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 -- 172.21.15.16:0/4078971799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0044108ab0 msgr2=0x7f004419fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0044108ab0 0x7f004419fab0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f003c00eb10 tx=0x7f003c00c2d0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 -- 172.21.15.16:0/4078971799 shutdown_connections 2024-08-28T19:23:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0044108ab0 0x7f004419fab0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f002c0b4ca0 0x7f002c0b71b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0044108160 0x7f004419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 --2- 172.21.15.16:0/4078971799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0044102820 0x7f004419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 -- 172.21.15.16:0/4078971799 >> 172.21.15.16:0/4078971799 conn(0x7f00440794e0 msgr2=0x7f00441012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 -- 172.21.15.16:0/4078971799 shutdown_connections 2024-08-28T19:23:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:45.674+0000 7f0049cc7640 1 -- 172.21.15.16:0/4078971799 wait complete. 2024-08-28T19:23:45.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 8.13 MB/sec warmup 85 sec latency 19355.393 ms 2024-08-28T19:23:46.497 INFO:tasks.cephfs.filesystem:scrub status for tag:bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2 - {'path': '/', 'tag': 'bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2', 'options': 'recursive,force'} 2024-08-28T19:23:46.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 8.03 MB/sec warmup 86 sec latency 20355.503 ms 2024-08-28T19:23:47.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:47 smithi016 ceph-mon[18556]: pgmap v421: 129 pgs: 129 active+clean; 84 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-08-28T19:23:47.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:47 smithi028 ceph-mon[21769]: pgmap v421: 129 pgs: 129 active+clean; 84 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-08-28T19:23:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:47 smithi115 ceph-mon[22184]: pgmap v421: 129 pgs: 129 active+clean; 84 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-08-28T19:23:47.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 7.94 MB/sec warmup 87 sec latency 21355.605 ms 2024-08-28T19:23:48.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 7.85 MB/sec warmup 88 sec latency 22355.645 ms 2024-08-28T19:23:49.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:49 smithi016 ceph-mon[18556]: pgmap v422: 129 pgs: 129 active+clean; 82 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.4 MiB/s wr, 70 op/s 2024-08-28T19:23:49.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:49 smithi028 ceph-mon[21769]: pgmap v422: 129 pgs: 129 active+clean; 82 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.4 MiB/s wr, 70 op/s 2024-08-28T19:23:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:49 smithi115 ceph-mon[22184]: pgmap v422: 129 pgs: 129 active+clean; 82 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.4 MiB/s wr, 70 op/s 2024-08-28T19:23:49.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 7.76 MB/sec warmup 89 sec latency 23355.723 ms 2024-08-28T19:23:50.481 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:23:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:50.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 7.68 MB/sec warmup 90 sec latency 24355.809 ms 2024-08-28T19:23:51.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.244+0000 7f09b21e6640 1 -- 172.21.15.16:0/2080159202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ac108ac0 msgr2=0x7f09ac10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:51.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.244+0000 7f09b21e6640 1 --2- 172.21.15.16:0/2080159202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ac108ac0 0x7f09ac10f2f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009ca0 tx=0x7f09a402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:23:51.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.244+0000 7f09b21e6640 1 -- 172.21.15.16:0/2080159202 shutdown_connections 2024-08-28T19:23:51.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.244+0000 7f09b21e6640 1 --2- 172.21.15.16:0/2080159202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ac108ac0 0x7f09ac10f2f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:51.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.244+0000 7f09b21e6640 1 --2- 172.21.15.16:0/2080159202 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09ac108170 0x7f09ac108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:51.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.244+0000 7f09b21e6640 1 --2- 172.21.15.16:0/2080159202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09ac102830 0x7f09ac102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:51.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.244+0000 7f09b21e6640 1 -- 172.21.15.16:0/2080159202 >> 172.21.15.16:0/2080159202 conn(0x7f09ac0794c0 msgr2=0x7f09ac0798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:51.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.244+0000 7f09b21e6640 1 -- 172.21.15.16:0/2080159202 shutdown_connections 2024-08-28T19:23:51.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.244+0000 7f09b21e6640 1 -- 172.21.15.16:0/2080159202 wait complete. 2024-08-28T19:23:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.245+0000 7f09b21e6640 1 Processor -- start 2024-08-28T19:23:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.245+0000 7f09b21e6640 1 -- start start 2024-08-28T19:23:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.245+0000 7f09b21e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ac102830 0x7f09ac19c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.245+0000 7f09b21e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09ac108170 0x7f09ac19c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.245+0000 7f09b21e6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09ac108ac0 0x7f09ac1a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.245+0000 7f09b21e6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac112770 con 0x7f09ac102830 2024-08-28T19:23:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.245+0000 7f09b21e6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac1125f0 con 0x7f09ac108170 2024-08-28T19:23:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.245+0000 7f09b21e6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac1128f0 con 0x7f09ac108ac0 2024-08-28T19:23:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.246+0000 7f09ab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ac102830 0x7f09ac19c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.246+0000 7f09ab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ac102830 0x7f09ac19c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55910/0 (socket says 172.21.15.16:55910) 2024-08-28T19:23:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.246+0000 7f09ab7fe640 1 -- 172.21.15.16:0/1000080335 learned_addr learned my addr 172.21.15.16:0/1000080335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.246+0000 7f09aaffd640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09ac108170 0x7f09ac19c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.246+0000 7f09abfff640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09ac108ac0 0x7f09ac1a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.246+0000 7f09ab7fe640 1 -- 172.21.15.16:0/1000080335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09ac108ac0 msgr2=0x7f09ac1a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.246+0000 7f09ab7fe640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09ac108ac0 0x7f09ac1a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.246+0000 7f09ab7fe640 1 -- 172.21.15.16:0/1000080335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09ac108170 msgr2=0x7f09ac19c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.246+0000 7f09ab7fe640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09ac108170 0x7f09ac19c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.246+0000 7f09ab7fe640 1 -- 172.21.15.16:0/1000080335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac1a2250 con 0x7f09ac102830 2024-08-28T19:23:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.246+0000 7f09ab7fe640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ac102830 0x7f09ac19c3a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f099c00c920 tx=0x7f099c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.247+0000 7f09a8ff9640 1 -- 172.21.15.16:0/1000080335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c007c70 con 0x7f09ac102830 2024-08-28T19:23:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.247+0000 7f09b21e6640 1 -- 172.21.15.16:0/1000080335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09ac1a2540 con 0x7f09ac102830 2024-08-28T19:23:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.247+0000 7f09a8ff9640 1 -- 172.21.15.16:0/1000080335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f099c007e10 con 0x7f09ac102830 2024-08-28T19:23:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.247+0000 7f09b21e6640 1 -- 172.21.15.16:0/1000080335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09ac1a8d60 con 0x7f09ac102830 2024-08-28T19:23:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.248+0000 7f09a8ff9640 1 -- 172.21.15.16:0/1000080335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c018820 con 0x7f09ac102830 2024-08-28T19:23:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.249+0000 7f09b21e6640 1 -- 172.21.15.16:0/1000080335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f098c0052c0 con 0x7f09ac102830 2024-08-28T19:23:51.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.250+0000 7f09a8ff9640 1 -- 172.21.15.16:0/1000080335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f099c0189c0 con 0x7f09ac102830 2024-08-28T19:23:51.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.251+0000 7f09a8ff9640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09980b4cd0 0x7f09980b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:51.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.251+0000 7f09aaffd640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09980b4cd0 0x7f09980b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:51.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.251+0000 7f09a8ff9640 1 -- 172.21.15.16:0/1000080335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f099c0e57e0 con 0x7f09ac102830 2024-08-28T19:23:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.251+0000 7f09aaffd640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09980b4cd0 0x7f09980b71e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f09ac079ac0 tx=0x7f099400b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:51.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.255+0000 7f09a8ff9640 1 -- 172.21.15.16:0/1000080335 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f099c014030 con 0x7f09ac102830 2024-08-28T19:23:51.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.477+0000 7f09b21e6640 1 -- 172.21.15.16:0/1000080335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f098c005880 con 0x7f09ac102830 2024-08-28T19:23:51.479 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:51 smithi016 ceph-mon[18556]: pgmap v423: 129 pgs: 129 active+clean; 80 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-08-28T19:23:51.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.478+0000 7f09a8ff9640 1 -- 172.21.15.16:0/1000080335 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11984 (secure 0 0 0) 0x7f099c0ac200 con 0x7f09ac102830 2024-08-28T19:23:51.480 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:23:51.480 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":31,"btime":"2024-08-28T19:22:49:142320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:22:48.274394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:23:51.480 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 31 2024-08-28T19:23:51.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 -- 172.21.15.16:0/1000080335 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09980b4cd0 msgr2=0x7f09980b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:51.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09980b4cd0 0x7f09980b71e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f09ac079ac0 tx=0x7f099400b390 comp rx=0 tx=0).stop 2024-08-28T19:23:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 -- 172.21.15.16:0/1000080335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ac102830 msgr2=0x7f09ac19c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ac102830 0x7f09ac19c3a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f099c00c920 tx=0x7f099c00cde0 comp rx=0 tx=0).stop 2024-08-28T19:23:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 -- 172.21.15.16:0/1000080335 shutdown_connections 2024-08-28T19:23:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09ac108ac0 0x7f09ac1a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09980b4cd0 0x7f09980b71e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09ac108170 0x7f09ac19c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 --2- 172.21.15.16:0/1000080335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ac102830 0x7f09ac19c3a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 -- 172.21.15.16:0/1000080335 >> 172.21.15.16:0/1000080335 conn(0x7f09ac0794c0 msgr2=0x7f09ac06b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 -- 172.21.15.16:0/1000080335 shutdown_connections 2024-08-28T19:23:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:51.480+0000 7f09b21e6640 1 -- 172.21.15.16:0/1000080335 wait complete. 2024-08-28T19:23:51.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:51 smithi028 ceph-mon[21769]: pgmap v423: 129 pgs: 129 active+clean; 80 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-08-28T19:23:51.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:51 smithi115 ceph-mon[22184]: pgmap v423: 129 pgs: 129 active+clean; 80 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-08-28T19:23:51.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 7.59 MB/sec warmup 91 sec latency 25355.869 ms 2024-08-28T19:23:52.327 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id 99a527e6 --release --await' 2024-08-28T19:23:52.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id 99a527e6 --release --await 2024-08-28T19:23:52.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:52 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1000080335' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:23:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:23:52.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:52 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1000080335' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:23:52.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:52 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1000080335' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:23:52.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124984 7.51 MB/sec warmup 92 sec latency 26355.963 ms 2024-08-28T19:23:53.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.087+0000 7f24d4537640 1 -- 172.21.15.16:0/1507784257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24cc102820 msgr2=0x7f24cc102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:53.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.087+0000 7f24d4537640 1 --2- 172.21.15.16:0/1507784257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24cc102820 0x7f24cc102c30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009900 tx=0x7f24c802a410 comp rx=0 tx=0).stop 2024-08-28T19:23:53.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.087+0000 7f24d4537640 1 -- 172.21.15.16:0/1507784257 shutdown_connections 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.087+0000 7f24d4537640 1 --2- 172.21.15.16:0/1507784257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24cc108ab0 0x7f24cc10f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.087+0000 7f24d4537640 1 --2- 172.21.15.16:0/1507784257 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24cc108160 0x7f24cc108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.087+0000 7f24d4537640 1 --2- 172.21.15.16:0/1507784257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24cc102820 0x7f24cc102c30 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.087+0000 7f24d4537640 1 -- 172.21.15.16:0/1507784257 >> 172.21.15.16:0/1507784257 conn(0x7f24cc0794e0 msgr2=0x7f24cc0798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.087+0000 7f24d4537640 1 -- 172.21.15.16:0/1507784257 shutdown_connections 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.087+0000 7f24d4537640 1 -- 172.21.15.16:0/1507784257 wait complete. 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.088+0000 7f24d4537640 1 Processor -- start 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.088+0000 7f24d4537640 1 -- start start 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.088+0000 7f24d4537640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24cc102820 0x7f24cc19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.088+0000 7f24d4537640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24cc108160 0x7f24cc19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.088+0000 7f24d4537640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24cc108ab0 0x7f24cc19fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.088+0000 7f24d4537640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc113840 con 0x7f24cc108ab0 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.088+0000 7f24d4537640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1136c0 con 0x7f24cc108160 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.088+0000 7f24d4537640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1139c0 con 0x7f24cc102820 2024-08-28T19:23:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.088+0000 7f24d2ad4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24cc108ab0 0x7f24cc19fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d2ad4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24cc108ab0 0x7f24cc19fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55930/0 (socket says 172.21.15.16:55930) 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d2ad4640 1 -- 172.21.15.16:0/2146249372 learned_addr learned my addr 172.21.15.16:0/2146249372 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24cc108160 0x7f24cc19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d22d3640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24cc102820 0x7f24cc19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d2ad4640 1 -- 172.21.15.16:0/2146249372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24cc102820 msgr2=0x7f24cc19a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d2ad4640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24cc102820 0x7f24cc19a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d2ad4640 1 -- 172.21.15.16:0/2146249372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24cc108160 msgr2=0x7f24cc19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d2ad4640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24cc108160 0x7f24cc19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d2ad4640 1 -- 172.21.15.16:0/2146249372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24cc1a01e0 con 0x7f24cc108ab0 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d2ad4640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24cc108ab0 0x7f24cc19fab0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f24c400eb10 tx=0x7f24c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24c37fe640 1 -- 172.21.15.16:0/2146249372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4004b20 con 0x7f24cc108ab0 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24c37fe640 1 -- 172.21.15.16:0/2146249372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24c4004cc0 con 0x7f24cc108ab0 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d4537640 1 -- 172.21.15.16:0/2146249372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24cc1a04d0 con 0x7f24cc108ab0 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24c37fe640 1 -- 172.21.15.16:0/2146249372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4010670 con 0x7f24cc108ab0 2024-08-28T19:23:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.089+0000 7f24d4537640 1 -- 172.21.15.16:0/2146249372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24cc1a6c30 con 0x7f24cc108ab0 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.092+0000 7f24daaac740 1 Processor -- start 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24c37fe640 1 -- 172.21.15.16:0/2146249372 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f24c4010830 con 0x7f24cc108ab0 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24daaac740 1 -- start start 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24daaac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24daaac740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24daaac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24daaac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cbc332ba70 con 0x55cbc300f520 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24daaac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbc331b000 con 0x55cbc2fee840 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24daaac740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cbc331efb0 con 0x55cbc2fe6660 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24d1ad2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24d1ad2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33b4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55934/0 (socket says 172.21.15.16:55934) 2024-08-28T19:23:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24d1ad2640 1 -- 172.21.15.16:0/1181507383 learned_addr learned my addr 172.21.15.16:0/1181507383 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.093+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/1181507383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24c37fe640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24b80b4ca0 0x7f24b80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24d22d3640 1 --2- 172.21.15.16:0/1181507383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24d2ad4640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24b80b4ca0 0x7f24b80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24d1ad2640 1 -- 172.21.15.16:0/1181507383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 msgr2=0x55cbc33b51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/1181507383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33b51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24d1ad2640 1 -- 172.21.15.16:0/1181507383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 msgr2=0x55cbc33b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24c37fe640 1 -- 172.21.15.16:0/2146249372 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f24c40e5e90 con 0x7f24cc108ab0 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/1181507383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24d1ad2640 1 -- 172.21.15.16:0/1181507383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbc307ab00 con 0x55cbc300f520 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/1181507383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33b4c80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00bde0 tx=0x7f24bc004120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36bc5ef2370a450b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24d2ad4640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24b80b4ca0 0x7f24b80b71b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f24c40a8290 tx=0x7f24c40762b0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24c0ff9640 1 -- 172.21.15.16:0/1181507383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc003e80 con 0x55cbc300f520 2024-08-28T19:23:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24c0ff9640 1 -- 172.21.15.16:0/1181507383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24bc0054d0 con 0x55cbc300f520 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.094+0000 7f24c0ff9640 1 -- 172.21.15.16:0/1181507383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc004e50 con 0x55cbc300f520 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.095+0000 7f24daaac740 1 -- 172.21.15.16:0/1181507383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 msgr2=0x55cbc33b4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.095+0000 7f24daaac740 1 --2- 172.21.15.16:0/1181507383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33b4c80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00bde0 tx=0x7f24bc004120 comp rx=0 tx=0).stop 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.095+0000 7f24daaac740 1 -- 172.21.15.16:0/1181507383 shutdown_connections 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.095+0000 7f24daaac740 1 --2- 172.21.15.16:0/1181507383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.095+0000 7f24daaac740 1 --2- 172.21.15.16:0/1181507383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33b51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.095+0000 7f24daaac740 1 --2- 172.21.15.16:0/1181507383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33b4c80 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.095+0000 7f24daaac740 1 -- 172.21.15.16:0/1181507383 >> 172.21.15.16:0/1181507383 conn(0x55cbc3003910 msgr2=0x55cbc33b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.095+0000 7f24daaac740 1 -- 172.21.15.16:0/1181507383 shutdown_connections 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.095+0000 7f24daaac740 1 -- 172.21.15.16:0/1181507383 wait complete. 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24daaac740 1 -- start start 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24daaac740 1 Processor -- start 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24daaac740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33c0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24daaac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fee840 0x55cbc33c0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24daaac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc300f520 0x55cbc33cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24daaac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cbc33c3400 con 0x55cbc2fee840 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24daaac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbc32dc520 con 0x55cbc300f520 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24daaac740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cbc33c3580 con 0x55cbc2fe6660 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24daaac740 10 client.0 _ll_register_callbacks cb 0x55cbc3314c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24d22d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fee840 0x55cbc33c0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24d22d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fee840 0x55cbc33c0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55950/0 (socket says 172.21.15.16:55950) 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.096+0000 7f24d22d3640 1 -- 172.21.15.16:0/2162682740 learned_addr learned my addr 172.21.15.16:0/2162682740 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc300f520 0x55cbc33cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33c0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24d22d3640 1 -- 172.21.15.16:0/2162682740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 msgr2=0x55cbc33c0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24d22d3640 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33c0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24d22d3640 1 -- 172.21.15.16:0/2162682740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc300f520 msgr2=0x55cbc33cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24d22d3640 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc300f520 0x55cbc33cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24d22d3640 1 -- 172.21.15.16:0/2162682740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbc30f1960 con 0x55cbc2fee840 2024-08-28T19:23:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24d22d3640 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fee840 0x55cbc33c0c90 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f24c802a930 tx=0x7f24c80316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff3437d17dbd1bd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2162682740 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8035040 con 0x55cbc2fee840 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2162682740 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f24c8004240 con 0x55cbc2fee840 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.097+0000 7f24daaac740 1 -- 172.21.15.16:0/2162682740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cbc32ba9a0 con 0x55cbc2fee840 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.098+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2162682740 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f24c804e050 con 0x55cbc2fee840 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.098+0000 7f24daaac740 10 client.16446 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.098+0000 7f24daaac740 1 -- 172.21.15.16:0/2162682740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cbc33cf4a0 con 0x55cbc2fee840 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.098+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2162682740 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55cbc32ba9a0 con 0x55cbc2fee840 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.098+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2162682740 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f24c80047e0 con 0x55cbc2fee840 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.098+0000 7f24daaac740 10 client.16446 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.098+0000 7f24daaac740 10 client.16446 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.098+0000 7f24daaac740 20 client.16446 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.098+0000 7f24daaac740 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cbc2fcdea0 0x55cbc33d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.098+0000 7f24daaac740 4 client.16446 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:23:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.098+0000 7f24daaac740 1 -- 172.21.15.16:0/2162682740 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbc332d9c0 con 0x55cbc2fcdea0 2024-08-28T19:23:53.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.099+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cbc2fcdea0 0x55cbc33d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.100+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cbc2fcdea0 0x55cbc33d2a90 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c0c9bd4cf64d20f server_cookie=68ef3fcc42cf88d9 in_seq=0 out_seq=0 2024-08-28T19:23:53.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.100+0000 7f24b5ffb640 10 client.16446 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:53.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.101+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2162682740 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cbc332d9c0 con 0x55cbc2fcdea0 2024-08-28T19:23:53.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.101+0000 7f24b5ffb640 10 client.16446 handle_command_reply: tid=0 2024-08-28T19:23:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.107+0000 7f24daaac740 1 client.16446 shutdown 2024-08-28T19:23:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.110+0000 7f24daaac740 1 -- 172.21.15.16:0/2162682740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fee840 msgr2=0x55cbc33c0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.110+0000 7f24daaac740 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fee840 0x55cbc33c0c90 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f24c802a930 tx=0x7f24c80316d0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.111+0000 7f24daaac740 1 -- 172.21.15.16:0/2162682740 shutdown_connections 2024-08-28T19:23:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.111+0000 7f24daaac740 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc300f520 0x55cbc33cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.111+0000 7f24daaac740 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fee840 0x55cbc33c0c90 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.111+0000 7f24daaac740 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cbc2fcdea0 0x55cbc33d2a90 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.111+0000 7f24daaac740 1 --2- 172.21.15.16:0/2162682740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33c0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.111+0000 7f24daaac740 1 -- 172.21.15.16:0/2162682740 >> 172.21.15.16:0/2162682740 conn(0x55cbc3003910 msgr2=0x55cbc33b7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.111+0000 7f24b5ffb640 0 client.16446 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.111+0000 7f24daaac740 1 -- 172.21.15.16:0/2162682740 shutdown_connections 2024-08-28T19:23:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.111+0000 7f24daaac740 1 -- 172.21.15.16:0/2162682740 wait complete. 2024-08-28T19:23:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.113+0000 7f24daaac740 20 client.16446 trim_cache size 0 max 16384 2024-08-28T19:23:53.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.156+0000 7f24daaac740 1 Processor -- start 2024-08-28T19:23:53.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.156+0000 7f24daaac740 1 -- start start 2024-08-28T19:23:53.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.156+0000 7f24daaac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fe6660 0x55cbc33b6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24daaac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33cc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24daaac740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc300f520 0x55cbc33cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24daaac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1b0be0 con 0x55cbc2fe6660 2024-08-28T19:23:53.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24daaac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1b27e0 con 0x55cbc2fee840 2024-08-28T19:23:53.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24daaac740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1b12b0 con 0x55cbc300f520 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24d22d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24d22d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:52860/0 (socket says 172.21.15.16:52860) 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24d22d3640 1 -- 172.21.15.16:0/2687959610 learned_addr learned my addr 172.21.15.16:0/2687959610 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/2687959610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fe6660 0x55cbc33b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24d22d3640 1 -- 172.21.15.16:0/2687959610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc300f520 msgr2=0x55cbc33cccb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24d22d3640 1 --2- 172.21.15.16:0/2687959610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc300f520 0x55cbc33cccb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24d22d3640 1 -- 172.21.15.16:0/2687959610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fe6660 msgr2=0x55cbc33b6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24d22d3640 1 --2- 172.21.15.16:0/2687959610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fe6660 0x55cbc33b6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24d22d3640 1 -- 172.21.15.16:0/2687959610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbc33c1810 con 0x55cbc2fee840 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.157+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/2687959610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fe6660 0x55cbc33b6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.158+0000 7f24d22d3640 1 --2- 172.21.15.16:0/2687959610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33cc770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24c804c040 tx=0x7f24c8031740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6802600747d509f4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.158+0000 7f24b7fff640 1 -- 172.21.15.16:0/2687959610 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8035070 con 0x55cbc2fee840 2024-08-28T19:23:53.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.158+0000 7f24b7fff640 1 -- 172.21.15.16:0/2687959610 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24c80035f0 con 0x55cbc2fee840 2024-08-28T19:23:53.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.158+0000 7f24b7fff640 1 -- 172.21.15.16:0/2687959610 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8004fe0 con 0x55cbc2fee840 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.159+0000 7f24daaac740 1 -- 172.21.15.16:0/2687959610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 msgr2=0x55cbc33cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.159+0000 7f24daaac740 1 --2- 172.21.15.16:0/2687959610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33cc770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24c804c040 tx=0x7f24c8031740 comp rx=0 tx=0).stop 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.159+0000 7f24daaac740 1 -- 172.21.15.16:0/2687959610 shutdown_connections 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.159+0000 7f24daaac740 1 --2- 172.21.15.16:0/2687959610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc300f520 0x55cbc33cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.159+0000 7f24daaac740 1 --2- 172.21.15.16:0/2687959610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33cc770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.159+0000 7f24daaac740 1 --2- 172.21.15.16:0/2687959610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc2fe6660 0x55cbc33b6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.159+0000 7f24daaac740 1 -- 172.21.15.16:0/2687959610 >> 172.21.15.16:0/2687959610 conn(0x55cbc3003910 msgr2=0x55cbc33b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.159+0000 7f24daaac740 1 -- 172.21.15.16:0/2687959610 shutdown_connections 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.159+0000 7f24daaac740 1 -- 172.21.15.16:0/2687959610 wait complete. 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24daaac740 1 -- start start 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24daaac740 1 Processor -- start 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24daaac740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24daaac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33c5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24daaac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24daaac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1b2050 con 0x55cbc300f520 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24daaac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1b1980 con 0x55cbc2fee840 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24daaac740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1aebc0 con 0x55cbc2fe6660 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24daaac740 10 client.0 _ll_register_callbacks cb 0x7f24c8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24d1ad2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24d22d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24d1ad2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33c5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55972/0 (socket says 172.21.15.16:55972) 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24d1ad2640 1 -- 172.21.15.16:0/2083536113 learned_addr learned my addr 172.21.15.16:0/2083536113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24d22d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33c5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:52874/0 (socket says 172.21.15.16:52874) 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.160+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24d1ad2640 1 -- 172.21.15.16:0/2083536113 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 msgr2=0x55cbc33c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24d1ad2640 1 -- 172.21.15.16:0/2083536113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 msgr2=0x55cbc33c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24d1ad2640 1 -- 172.21.15.16:0/2083536113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbc33c5cd0 con 0x55cbc300f520 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24d22d3640 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24d1ad2640 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33c5790 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f24bc004aa0 tx=0x7f24bc017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eebc00523685d4b8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2083536113 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc012740 con 0x55cbc300f520 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2083536113 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f24bc02ff90 con 0x55cbc300f520 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24daaac740 1 -- 172.21.15.16:0/2083536113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24c80035f0 con 0x55cbc300f520 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2083536113 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f24bc03d050 con 0x55cbc300f520 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24daaac740 10 client.16449 fetch_fsmap learned FSMap version 31 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24daaac740 1 -- 172.21.15.16:0/2083536113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cbc343e0d0 con 0x55cbc300f520 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.161+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2083536113 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24c80035f0 con 0x55cbc300f520 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.162+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2083536113 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8386+0+0 (secure 0 0 0) 0x7f24bc0106c0 con 0x55cbc300f520 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.162+0000 7f24daaac740 10 client.16449 fetch_fsmap finished waiting for FSMap version 31 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.162+0000 7f24daaac740 10 client.16449 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:23:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.162+0000 7f24daaac740 20 client.16449 populate_metadata read hostname 'smithi016' 2024-08-28T19:23:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.162+0000 7f24daaac740 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cbc2fa3250 0x55cbc3440780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:23:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.162+0000 7f24daaac740 4 client.16449 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "99a527e6", "release": true, "await": true}] 2024-08-28T19:23:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.162+0000 7f24daaac740 1 -- 172.21.15.16:0/2083536113 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "99a527e6", "release": true, "await": true}) -- 0x55cbc3329460 con 0x55cbc2fa3250 2024-08-28T19:23:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.162+0000 7f24d22d3640 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cbc2fa3250 0x55cbc3440780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:23:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.162+0000 7f24d22d3640 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cbc2fa3250 0x55cbc3440780 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8a2354c402c0622 server_cookie=d9357a5d2719790 in_seq=0 out_seq=0 2024-08-28T19:23:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.162+0000 7f24b5ffb640 10 client.16449 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.185+0000 7f24b5ffb640 1 -- 172.21.15.16:0/2083536113 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55cbc3329460 con 0x55cbc2fa3250 2024-08-28T19:23:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.185+0000 7f24b5ffb640 10 client.16449 handle_command_reply: tid=0 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 1 client.16449 shutdown 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 1 -- 172.21.15.16:0/2083536113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 msgr2=0x55cbc33c5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33c5790 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f24bc004aa0 tx=0x7f24bc017040 comp rx=0 tx=0).stop 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 1 -- 172.21.15.16:0/2083536113 shutdown_connections 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbc300f520 0x55cbc33c5790 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc2fee840 0x55cbc33c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cbc2fa3250 0x55cbc3440780 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 1 --2- 172.21.15.16:0/2083536113 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cbc2fe6660 0x55cbc33c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24b5ffb640 0 client.16449 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 1 -- 172.21.15.16:0/2083536113 >> 172.21.15.16:0/2083536113 conn(0x55cbc3003910 msgr2=0x55cbc33c19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 1 -- 172.21.15.16:0/2083536113 shutdown_connections 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 1 -- 172.21.15.16:0/2083536113 wait complete. 2024-08-28T19:23:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.186+0000 7f24daaac740 20 client.16449 trim_cache size 0 max 16384 2024-08-28T19:23:53.190 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:23:53.190 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 31, 2024-08-28T19:23:53.190 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:23:53.190 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 8, 2024-08-28T19:23:53.190 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:23:53.190 INFO:teuthology.orchestra.run.smithi016.stdout: "99a527e6": { 2024-08-28T19:23:53.190 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 8, 2024-08-28T19:23:53.190 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 146.6, 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:23:53.191 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:23:53.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:53.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:53.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:53.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:53.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:23:53.192 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:23:53.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 -- 172.21.15.16:0/2146249372 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24b80b4ca0 msgr2=0x7f24b80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24b80b4ca0 0x7f24b80b71b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f24c40a8290 tx=0x7f24c40762b0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 -- 172.21.15.16:0/2146249372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24cc108ab0 msgr2=0x7f24cc19fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:23:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24cc108ab0 0x7f24cc19fab0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f24c400eb10 tx=0x7f24c400c2d0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 -- 172.21.15.16:0/2146249372 shutdown_connections 2024-08-28T19:23:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24cc108ab0 0x7f24cc19fab0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f24b80b4ca0 0x7f24b80b71b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24cc108160 0x7f24cc19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 --2- 172.21.15.16:0/2146249372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f24cc102820 0x7f24cc19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:23:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 -- 172.21.15.16:0/2146249372 >> 172.21.15.16:0/2146249372 conn(0x7f24cc0794e0 msgr2=0x7f24cc1012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:23:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 -- 172.21.15.16:0/2146249372 shutdown_connections 2024-08-28T19:23:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:23:53.187+0000 7f24d4537640 1 -- 172.21.15.16:0/2146249372 wait complete. 2024-08-28T19:23:53.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:53 smithi115 ceph-mon[22184]: pgmap v424: 129 pgs: 129 active+clean; 82 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-08-28T19:23:53.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:53 smithi028 ceph-mon[21769]: pgmap v424: 129 pgs: 129 active+clean; 82 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-08-28T19:23:53.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:53 smithi016 ceph-mon[18556]: pgmap v424: 129 pgs: 129 active+clean; 82 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-08-28T19:23:53.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 126757 7.51 MB/sec warmup 93 sec latency 26605.388 ms 2024-08-28T19:23:54.095 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '99a527e6', total seconds elapsed: 30.2 2024-08-28T19:23:54.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 128351 7.55 MB/sec warmup 94 sec latency 16.976 ms 2024-08-28T19:23:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:55 smithi028 ceph-mon[21769]: pgmap v425: 129 pgs: 129 active+clean; 82 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.4 MiB/s wr, 67 op/s 2024-08-28T19:23:55.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:55 smithi115 ceph-mon[22184]: pgmap v425: 129 pgs: 129 active+clean; 82 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.4 MiB/s wr, 67 op/s 2024-08-28T19:23:55.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:55 smithi016 ceph-mon[18556]: pgmap v425: 129 pgs: 129 active+clean; 82 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.4 MiB/s wr, 67 op/s 2024-08-28T19:23:55.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 130895 7.61 MB/sec warmup 95 sec latency 11.392 ms 2024-08-28T19:23:56.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 132380 7.61 MB/sec warmup 96 sec latency 12.105 ms 2024-08-28T19:23:57.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:57 smithi028 ceph-mon[21769]: pgmap v426: 129 pgs: 129 active+clean; 79 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.6 MiB/s wr, 115 op/s 2024-08-28T19:23:57.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:57 smithi115 ceph-mon[22184]: pgmap v426: 129 pgs: 129 active+clean; 79 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.6 MiB/s wr, 115 op/s 2024-08-28T19:23:57.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:57 smithi016 ceph-mon[18556]: pgmap v426: 129 pgs: 129 active+clean; 79 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.6 MiB/s wr, 115 op/s 2024-08-28T19:23:57.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 133679 7.58 MB/sec warmup 97 sec latency 636.570 ms 2024-08-28T19:23:58.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:58 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:23:58.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:58 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:23:58.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:58 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:23:58.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 133679 7.51 MB/sec warmup 98 sec latency 1636.667 ms 2024-08-28T19:23:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:59 smithi028 ceph-mon[21769]: pgmap v427: 129 pgs: 129 active+clean; 76 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 1.8 MiB/s wr, 122 op/s 2024-08-28T19:23:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:59 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:23:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:59 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:23:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:59 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:23:59.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:59 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:23:59.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:23:59 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:23:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:59 smithi115 ceph-mon[22184]: pgmap v427: 129 pgs: 129 active+clean; 76 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 1.8 MiB/s wr, 122 op/s 2024-08-28T19:23:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:59 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:23:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:59 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:23:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:59 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:23:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:59 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:23:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:23:59 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:23:59.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:59 smithi016 ceph-mon[18556]: pgmap v427: 129 pgs: 129 active+clean; 76 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 1.8 MiB/s wr, 122 op/s 2024-08-28T19:23:59.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:59 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:23:59.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:59 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:23:59.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:59 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:23:59.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:59 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:23:59.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:23:59 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:23:59.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 133679 7.43 MB/sec warmup 99 sec latency 2636.772 ms 2024-08-28T19:24:00.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 133707 7.36 MB/sec warmup 100 sec latency 2880.026 ms 2024-08-28T19:24:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:01 smithi028 ceph-mon[21769]: pgmap v428: 129 pgs: 129 active+clean; 78 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 148 op/s 2024-08-28T19:24:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:01 smithi115 ceph-mon[22184]: pgmap v428: 129 pgs: 129 active+clean; 78 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 148 op/s 2024-08-28T19:24:01.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:01 smithi016 ceph-mon[18556]: pgmap v428: 129 pgs: 129 active+clean; 78 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 148 op/s 2024-08-28T19:24:01.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 133719 7.28 MB/sec warmup 101 sec latency 1393.525 ms 2024-08-28T19:24:02.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 133719 7.21 MB/sec warmup 102 sec latency 1325.860 ms 2024-08-28T19:24:03.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:03 smithi028 ceph-mon[21769]: pgmap v429: 129 pgs: 129 active+clean; 81 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 185 op/s 2024-08-28T19:24:03.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:03 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:24:03.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:03 smithi115 ceph-mon[22184]: pgmap v429: 129 pgs: 129 active+clean; 81 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 185 op/s 2024-08-28T19:24:03.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:03 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:24:03.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:03 smithi016 ceph-mon[18556]: pgmap v429: 129 pgs: 129 active+clean; 81 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 185 op/s 2024-08-28T19:24:03.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:03 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:24:03.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 133719 7.14 MB/sec warmup 103 sec latency 2325.974 ms 2024-08-28T19:24:04.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 133719 7.07 MB/sec warmup 104 sec latency 3326.079 ms 2024-08-28T19:24:05.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:05 smithi028 ceph-mon[21769]: pgmap v430: 129 pgs: 129 active+clean; 82 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 170 op/s 2024-08-28T19:24:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:05 smithi115 ceph-mon[22184]: pgmap v430: 129 pgs: 129 active+clean; 82 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 170 op/s 2024-08-28T19:24:05.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:05 smithi016 ceph-mon[18556]: pgmap v430: 129 pgs: 129 active+clean; 82 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 170 op/s 2024-08-28T19:24:05.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 133719 7.01 MB/sec warmup 105 sec latency 4326.164 ms 2024-08-28T19:24:06.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134205 6.99 MB/sec warmup 106 sec latency 4981.982 ms 2024-08-28T19:24:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:07 smithi028 ceph-mon[21769]: pgmap v431: 129 pgs: 129 active+clean; 80 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-08-28T19:24:07.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:07 smithi115 ceph-mon[22184]: pgmap v431: 129 pgs: 129 active+clean; 80 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-08-28T19:24:07.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:07 smithi016 ceph-mon[18556]: pgmap v431: 129 pgs: 129 active+clean; 80 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-08-28T19:24:07.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134205 6.93 MB/sec warmup 107 sec latency 1117.667 ms 2024-08-28T19:24:08.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134205 6.86 MB/sec warmup 108 sec latency 2117.715 ms 2024-08-28T19:24:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:09 smithi028 ceph-mon[21769]: pgmap v432: 129 pgs: 129 active+clean; 77 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-08-28T19:24:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:09 smithi115 ceph-mon[22184]: pgmap v432: 129 pgs: 129 active+clean; 77 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-08-28T19:24:09.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:09 smithi016 ceph-mon[18556]: pgmap v432: 129 pgs: 129 active+clean; 77 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-08-28T19:24:09.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134205 6.80 MB/sec warmup 109 sec latency 3117.823 ms 2024-08-28T19:24:10.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134205 6.74 MB/sec warmup 110 sec latency 4117.937 ms 2024-08-28T19:24:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:11 smithi028 ceph-mon[21769]: pgmap v433: 129 pgs: 129 active+clean; 77 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 915 KiB/s wr, 103 op/s 2024-08-28T19:24:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:11 smithi115 ceph-mon[22184]: pgmap v433: 129 pgs: 129 active+clean; 77 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 915 KiB/s wr, 103 op/s 2024-08-28T19:24:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:11 smithi016 ceph-mon[18556]: pgmap v433: 129 pgs: 129 active+clean; 77 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 915 KiB/s wr, 103 op/s 2024-08-28T19:24:11.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134286 6.68 MB/sec warmup 111 sec latency 4777.952 ms 2024-08-28T19:24:12.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134286 6.62 MB/sec warmup 112 sec latency 1282.861 ms 2024-08-28T19:24:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:13 smithi028 ceph-mon[21769]: pgmap v434: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 572 KiB/s wr, 76 op/s 2024-08-28T19:24:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:13 smithi115 ceph-mon[22184]: pgmap v434: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 572 KiB/s wr, 76 op/s 2024-08-28T19:24:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:13 smithi016 ceph-mon[18556]: pgmap v434: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 572 KiB/s wr, 76 op/s 2024-08-28T19:24:13.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134286 6.56 MB/sec warmup 113 sec latency 2282.963 ms 2024-08-28T19:24:14.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134286 6.50 MB/sec warmup 114 sec latency 3283.068 ms 2024-08-28T19:24:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:15 smithi028 ceph-mon[21769]: pgmap v435: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 205 KiB/s wr, 28 op/s 2024-08-28T19:24:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:15 smithi115 ceph-mon[22184]: pgmap v435: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 205 KiB/s wr, 28 op/s 2024-08-28T19:24:15.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:15 smithi016 ceph-mon[18556]: pgmap v435: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 205 KiB/s wr, 28 op/s 2024-08-28T19:24:15.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134286 6.44 MB/sec warmup 115 sec latency 4283.179 ms 2024-08-28T19:24:16.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:24:16.763 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:24:16.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134298 6.39 MB/sec warmup 116 sec latency 4943.274 ms 2024-08-28T19:24:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.245+0000 7f5440267640 1 -- 172.21.15.16:0/3891074247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5438109280 msgr2=0x7f5438109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.245+0000 7f5440267640 1 --2- 172.21.15.16:0/3891074247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5438109280 0x7f5438109690 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f542c009900 tx=0x7f542c02a410 comp rx=0 tx=0).stop 2024-08-28T19:24:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.246+0000 7f5440267640 1 -- 172.21.15.16:0/3891074247 shutdown_connections 2024-08-28T19:24:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.246+0000 7f5440267640 1 --2- 172.21.15.16:0/3891074247 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5438109bd0 0x7f5438110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.246+0000 7f5440267640 1 --2- 172.21.15.16:0/3891074247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5438109280 0x7f5438109690 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.246+0000 7f5440267640 1 --2- 172.21.15.16:0/3891074247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5438103940 0x7f5438103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.246+0000 7f5440267640 1 -- 172.21.15.16:0/3891074247 >> 172.21.15.16:0/3891074247 conn(0x7f54380ff3d0 msgr2=0x7f5438101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.246+0000 7f5440267640 1 -- 172.21.15.16:0/3891074247 shutdown_connections 2024-08-28T19:24:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.246+0000 7f5440267640 1 -- 172.21.15.16:0/3891074247 wait complete. 2024-08-28T19:24:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.246+0000 7f5440267640 1 Processor -- start 2024-08-28T19:24:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.246+0000 7f5440267640 1 -- start start 2024-08-28T19:24:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f5440267640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5438103940 0x7f543819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f5440267640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5438109280 0x7f543819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f5440267640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5438109bd0 0x7f543819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f5440267640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54381138e0 con 0x7f5438109bd0 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f5440267640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5438113760 con 0x7f5438109280 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f5440267640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5438113a60 con 0x7f5438103940 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f543e804640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5438109bd0 0x7f543819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f543e804640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5438109bd0 0x7f543819fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58964/0 (socket says 172.21.15.16:58964) 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f543e804640 1 -- 172.21.15.16:0/1162915809 learned_addr learned my addr 172.21.15.16:0/1162915809 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f543e003640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5438103940 0x7f543819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f543d802640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5438109280 0x7f543819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f543e804640 1 -- 172.21.15.16:0/1162915809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5438103940 msgr2=0x7f543819a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f543e804640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5438103940 0x7f543819a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f543e804640 1 -- 172.21.15.16:0/1162915809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5438109280 msgr2=0x7f543819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f543e804640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5438109280 0x7f543819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f543e804640 1 -- 172.21.15.16:0/1162915809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54381a01e0 con 0x7f5438109bd0 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.247+0000 7f543e804640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5438109bd0 0x7f543819fab0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f543000d750 tx=0x7f543000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.248+0000 7f54377fe640 1 -- 172.21.15.16:0/1162915809 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5430017070 con 0x7f5438109bd0 2024-08-28T19:24:17.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.248+0000 7f54377fe640 1 -- 172.21.15.16:0/1162915809 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54300054d0 con 0x7f5438109bd0 2024-08-28T19:24:17.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.248+0000 7f5440267640 1 -- 172.21.15.16:0/1162915809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54381a04d0 con 0x7f5438109bd0 2024-08-28T19:24:17.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.248+0000 7f54377fe640 1 -- 172.21.15.16:0/1162915809 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5430004c60 con 0x7f5438109bd0 2024-08-28T19:24:17.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.248+0000 7f5440267640 1 -- 172.21.15.16:0/1162915809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54381a6c30 con 0x7f5438109bd0 2024-08-28T19:24:17.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.250+0000 7f54467dc740 1 Processor -- start 2024-08-28T19:24:17.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.250+0000 7f54377fe640 1 -- 172.21.15.16:0/1162915809 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5430009bd0 con 0x7f5438109bd0 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f54467dc740 1 -- start start 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f54467dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1d3070 0x56459b5782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f54467dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1aa1b0 0x56459b578800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f54467dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1b2390 0x56459b578d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f54467dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56459b4ef630 con 0x56459b1b2390 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f54467dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56459b4debc0 con 0x56459b1d3070 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f54467dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56459b4e2b70 con 0x56459b1aa1b0 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f543d802640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1b2390 0x56459b578d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f543d802640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1b2390 0x56459b578d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58974/0 (socket says 172.21.15.16:58974) 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f543d802640 1 -- 172.21.15.16:0/1932624475 learned_addr learned my addr 172.21.15.16:0/1932624475 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f543e003640 1 --2- 172.21.15.16:0/1932624475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1aa1b0 0x56459b578800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f543d802640 1 --2- 172.21.15.16:0/1932624475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1d3070 0x56459b5782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.251+0000 7f54377fe640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54200b4ca0 0x7f54200b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f543d802640 1 -- 172.21.15.16:0/1932624475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1aa1b0 msgr2=0x56459b578800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f543d802640 1 --2- 172.21.15.16:0/1932624475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1aa1b0 0x56459b578800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f543d802640 1 -- 172.21.15.16:0/1932624475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1d3070 msgr2=0x56459b5782c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f543d802640 1 --2- 172.21.15.16:0/1932624475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1d3070 0x56459b5782c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f543e804640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54200b4ca0 0x7f54200b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f543d802640 1 -- 172.21.15.16:0/1932624475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56459b23e650 con 0x56459b1b2390 2024-08-28T19:24:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f54377fe640 1 -- 172.21.15.16:0/1162915809 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f54300e5330 con 0x7f5438109bd0 2024-08-28T19:24:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f543d802640 1 --2- 172.21.15.16:0/1932624475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1b2390 0x56459b578d40 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f542c009900 tx=0x7f542c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa051db1c3c0d73 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f5434d9a640 1 -- 172.21.15.16:0/1932624475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c035040 con 0x56459b1b2390 2024-08-28T19:24:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f5434d9a640 1 -- 172.21.15.16:0/1932624475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f542c0038c0 con 0x56459b1b2390 2024-08-28T19:24:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f543e804640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54200b4ca0 0x7f54200b71b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f543000d720 tx=0x7f54300ce040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.252+0000 7f5434d9a640 1 -- 172.21.15.16:0/1932624475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c004920 con 0x56459b1b2390 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.253+0000 7f54467dc740 1 -- 172.21.15.16:0/1932624475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1b2390 msgr2=0x56459b578d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.253+0000 7f54467dc740 1 --2- 172.21.15.16:0/1932624475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1b2390 0x56459b578d40 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f542c009900 tx=0x7f542c000bc0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.253+0000 7f54467dc740 1 -- 172.21.15.16:0/1932624475 shutdown_connections 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.253+0000 7f54467dc740 1 --2- 172.21.15.16:0/1932624475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1b2390 0x56459b578d40 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.253+0000 7f54467dc740 1 --2- 172.21.15.16:0/1932624475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1aa1b0 0x56459b578800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.253+0000 7f54467dc740 1 --2- 172.21.15.16:0/1932624475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1d3070 0x56459b5782c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.253+0000 7f54467dc740 1 -- 172.21.15.16:0/1932624475 >> 172.21.15.16:0/1932624475 conn(0x56459b1c7460 msgr2=0x56459b5761b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.253+0000 7f54467dc740 1 -- 172.21.15.16:0/1932624475 shutdown_connections 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.253+0000 7f54467dc740 1 -- 172.21.15.16:0/1932624475 wait complete. 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f54467dc740 1 -- start start 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f54467dc740 1 Processor -- start 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f54467dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1aa1b0 0x56459b5839e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f54467dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1b2390 0x56459b583f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f54467dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1d3070 0x56459b58fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f54467dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56459b341700 con 0x56459b1d3070 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f54467dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56459b465700 con 0x56459b1aa1b0 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f54467dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56459b586a40 con 0x56459b1b2390 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f54467dc740 10 client.0 _ll_register_callbacks cb 0x56459b494dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f543d802640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1d3070 0x56459b58fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f543d802640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1d3070 0x56459b58fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58976/0 (socket says 172.21.15.16:58976) 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f543d802640 1 -- 172.21.15.16:0/3346485446 learned_addr learned my addr 172.21.15.16:0/3346485446 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.254+0000 7f543d802640 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1aa1b0 0x56459b5839e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f543e003640 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1b2390 0x56459b583f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f543d802640 1 -- 172.21.15.16:0/3346485446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1b2390 msgr2=0x56459b583f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f543d802640 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1b2390 0x56459b583f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f543d802640 1 -- 172.21.15.16:0/3346485446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1aa1b0 msgr2=0x56459b5839e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f543d802640 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1aa1b0 0x56459b5839e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f543d802640 1 -- 172.21.15.16:0/3346485446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56459b49e8b0 con 0x56459b1d3070 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f543e003640 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1b2390 0x56459b583f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:24:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f543d802640 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1d3070 0x56459b58fd60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x56459b586230 tx=0x7f542c004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c757ebe59d38401 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f541dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f541dffb640 1 -- 172.21.15.16:0/3346485446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c035040 con 0x56459b1d3070 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f541dffb640 1 -- 172.21.15.16:0/3346485446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f542c005030 con 0x56459b1d3070 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f54467dc740 1 -- 172.21.15.16:0/3346485446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f542c0038c0 con 0x56459b1d3070 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f541dffb640 1 -- 172.21.15.16:0/3346485446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f542c005230 con 0x56459b1d3070 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f54467dc740 10 client.16461 fetch_fsmap learned FSMap version 33 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.255+0000 7f54467dc740 1 -- 172.21.15.16:0/3346485446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56459b5927c0 con 0x56459b1d3070 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.256+0000 7f541dffb640 1 -- 172.21.15.16:0/3346485446 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f542c0038c0 con 0x56459b1d3070 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.256+0000 7f541dffb640 1 -- 172.21.15.16:0/3346485446 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f542c048070 con 0x56459b1d3070 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.256+0000 7f54467dc740 10 client.16461 fetch_fsmap finished waiting for FSMap version 33 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.256+0000 7f54467dc740 10 client.16461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.256+0000 7f54467dc740 20 client.16461 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.256+0000 7f54467dc740 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56459b1919f0 0x56459b595b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.256+0000 7f54467dc740 4 client.16461 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:24:17.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.256+0000 7f54467dc740 1 -- 172.21.15.16:0/3346485446 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56459b4f1580 con 0x56459b1919f0 2024-08-28T19:24:17.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.256+0000 7f543e003640 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56459b1919f0 0x56459b595b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.257+0000 7f541dffb640 10 client.16461 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:17.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.257+0000 7f543e003640 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56459b1919f0 0x56459b595b20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfabf4eb660c205f server_cookie=6310a6d8a832b1f1 in_seq=0 out_seq=0 2024-08-28T19:24:17.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f541dffb640 1 -- 172.21.15.16:0/3346485446 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56459b4f1580 con 0x56459b1919f0 2024-08-28T19:24:17.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f541dffb640 10 client.16461 handle_command_reply: tid=0 2024-08-28T19:24:17.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 1 client.16461 shutdown 2024-08-28T19:24:17.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 1 -- 172.21.15.16:0/3346485446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1d3070 msgr2=0x56459b58fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:17.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1d3070 0x56459b58fd60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x56459b586230 tx=0x7f542c004580 comp rx=0 tx=0).stop 2024-08-28T19:24:17.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 1 -- 172.21.15.16:0/3346485446 shutdown_connections 2024-08-28T19:24:17.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1d3070 0x56459b58fd60 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56459b1919f0 0x56459b595b20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1b2390 0x56459b583f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 1 --2- 172.21.15.16:0/3346485446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1aa1b0 0x56459b5839e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 1 -- 172.21.15.16:0/3346485446 >> 172.21.15.16:0/3346485446 conn(0x56459b1c7460 msgr2=0x56459b57b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:17.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f541dffb640 0 client.16461 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:17.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 1 -- 172.21.15.16:0/3346485446 shutdown_connections 2024-08-28T19:24:17.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 1 -- 172.21.15.16:0/3346485446 wait complete. 2024-08-28T19:24:17.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.258+0000 7f54467dc740 20 client.16461 trim_cache size 0 max 16384 2024-08-28T19:24:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f54467dc740 1 Processor -- start 2024-08-28T19:24:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f54467dc740 1 -- start start 2024-08-28T19:24:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f54467dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b579a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f54467dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1b2390 0x56459b575780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f54467dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1d3070 0x56459b575cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f54467dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54381b0b60 con 0x56459b1aa1b0 2024-08-28T19:24:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f54467dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54381b27c0 con 0x56459b1b2390 2024-08-28T19:24:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f54467dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f54381b1230 con 0x56459b1d3070 2024-08-28T19:24:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f543d802640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b579a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f543d802640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b579a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58988/0 (socket says 172.21.15.16:58988) 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f543d802640 1 -- 172.21.15.16:0/2865088774 learned_addr learned my addr 172.21.15.16:0/2865088774 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f543e003640 1 --2- 172.21.15.16:0/2865088774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1b2390 0x56459b575780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f543d802640 1 --2- 172.21.15.16:0/2865088774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1d3070 0x56459b575cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f543d802640 1 -- 172.21.15.16:0/2865088774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1d3070 msgr2=0x56459b575cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f543d802640 1 --2- 172.21.15.16:0/2865088774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1d3070 0x56459b575cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f543d802640 1 -- 172.21.15.16:0/2865088774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1b2390 msgr2=0x56459b575780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f543d802640 1 --2- 172.21.15.16:0/2865088774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1b2390 0x56459b575780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.276+0000 7f543d802640 1 -- 172.21.15.16:0/2865088774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56459b582940 con 0x56459b1aa1b0 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.277+0000 7f543d802640 1 --2- 172.21.15.16:0/2865088774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b579a00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f542c02a450 tx=0x7f542c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247c9aa7cf2913a4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.277+0000 7f541ffff640 1 -- 172.21.15.16:0/2865088774 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c035070 con 0x56459b1aa1b0 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.277+0000 7f541ffff640 1 -- 172.21.15.16:0/2865088774 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f542c04e070 con 0x56459b1aa1b0 2024-08-28T19:24:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.277+0000 7f541ffff640 1 -- 172.21.15.16:0/2865088774 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c0053d0 con 0x56459b1aa1b0 2024-08-28T19:24:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.277+0000 7f54467dc740 1 -- 172.21.15.16:0/2865088774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 msgr2=0x56459b579a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.278+0000 7f54467dc740 1 --2- 172.21.15.16:0/2865088774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b579a00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f542c02a450 tx=0x7f542c031400 comp rx=0 tx=0).stop 2024-08-28T19:24:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.278+0000 7f54467dc740 1 -- 172.21.15.16:0/2865088774 shutdown_connections 2024-08-28T19:24:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.278+0000 7f54467dc740 1 --2- 172.21.15.16:0/2865088774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1d3070 0x56459b575cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.278+0000 7f54467dc740 1 --2- 172.21.15.16:0/2865088774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1b2390 0x56459b575780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.278+0000 7f54467dc740 1 --2- 172.21.15.16:0/2865088774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b579a00 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.278+0000 7f54467dc740 1 -- 172.21.15.16:0/2865088774 >> 172.21.15.16:0/2865088774 conn(0x56459b1c7460 msgr2=0x56459b5792f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.278+0000 7f54467dc740 1 -- 172.21.15.16:0/2865088774 shutdown_connections 2024-08-28T19:24:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.278+0000 7f54467dc740 1 -- 172.21.15.16:0/2865088774 wait complete. 2024-08-28T19:24:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.278+0000 7f54467dc740 1 -- start start 2024-08-28T19:24:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.278+0000 7f54467dc740 1 Processor -- start 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.278+0000 7f54467dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b599490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f54467dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1b2390 0x56459b596960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f54467dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1d3070 0x56459b597000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f54467dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54381b2030 con 0x56459b1aa1b0 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f54467dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54381b1960 con 0x56459b1b2390 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f54467dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f54381aeb40 con 0x56459b1d3070 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f54467dc740 10 client.0 _ll_register_callbacks cb 0x7f542c0041a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543d802640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b599490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543d802640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b599490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58992/0 (socket says 172.21.15.16:58992) 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543d802640 1 -- 172.21.15.16:0/4093974613 learned_addr learned my addr 172.21.15.16:0/4093974613 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543e003640 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1b2390 0x56459b596960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543d802640 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1d3070 0x56459b597000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543d802640 1 -- 172.21.15.16:0/4093974613 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1d3070 msgr2=0x56459b597000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543d802640 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1d3070 0x56459b597000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543d802640 1 -- 172.21.15.16:0/4093974613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1b2390 msgr2=0x56459b596960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543d802640 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1b2390 0x56459b596960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543d802640 1 -- 172.21.15.16:0/4093974613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56459b597790 con 0x56459b1aa1b0 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543e003640 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1b2390 0x56459b596960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f543d802640 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b599490 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f542c04d040 tx=0x7f542c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32fc86abf5e9a951 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f541dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f541dffb640 1 -- 172.21.15.16:0/4093974613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c035070 con 0x56459b1aa1b0 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f541dffb640 1 -- 172.21.15.16:0/4093974613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f542c053040 con 0x56459b1aa1b0 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.279+0000 7f54467dc740 1 -- 172.21.15.16:0/4093974613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f542c04e070 con 0x56459b1aa1b0 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f541dffb640 1 -- 172.21.15.16:0/4093974613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f542c004480 con 0x56459b1aa1b0 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f54467dc740 10 client.16467 fetch_fsmap learned FSMap version 33 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f54467dc740 1 -- 172.21.15.16:0/4093974613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56459b600360 con 0x56459b1aa1b0 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f541dffb640 1 -- 172.21.15.16:0/4093974613 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f542c04e070 con 0x56459b1aa1b0 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f541dffb640 1 -- 172.21.15.16:0/4093974613 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f542c048070 con 0x56459b1aa1b0 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f54467dc740 10 client.16467 fetch_fsmap finished waiting for FSMap version 33 2024-08-28T19:24:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f54467dc740 10 client.16467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f54467dc740 20 client.16467 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f54467dc740 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56459b166da0 0x56459b6028f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f54467dc740 4 client.16467 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f54467dc740 1 -- 172.21.15.16:0/4093974613 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56459b4ed020 con 0x56459b166da0 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.280+0000 7f543e003640 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56459b166da0 0x56459b6028f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f541dffb640 10 client.16467 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f543e003640 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56459b166da0 0x56459b6028f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=995724d3eb0bcbd server_cookie=1aa0385baf27bb8f in_seq=0 out_seq=0 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f541dffb640 1 -- 172.21.15.16:0/4093974613 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1510 (crc 0 0 0) 0x56459b4ed020 con 0x56459b166da0 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f541dffb640 10 client.16467 handle_command_reply: tid=0 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 1 client.16467 shutdown 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 1 -- 172.21.15.16:0/4093974613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 msgr2=0x56459b599490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b599490 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f542c04d040 tx=0x7f542c005c90 comp rx=0 tx=0).stop 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 1 -- 172.21.15.16:0/4093974613 shutdown_connections 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56459b166da0 0x56459b6028f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56459b1d3070 0x56459b597000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459b1b2390 0x56459b596960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 1 --2- 172.21.15.16:0/4093974613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56459b1aa1b0 0x56459b599490 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 1 -- 172.21.15.16:0/4093974613 >> 172.21.15.16:0/4093974613 conn(0x56459b1c7460 msgr2=0x56459b584ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f541dffb640 0 client.16467 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 1 -- 172.21.15.16:0/4093974613 shutdown_connections 2024-08-28T19:24:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 1 -- 172.21.15.16:0/4093974613 wait complete. 2024-08-28T19:24:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.281+0000 7f54467dc740 20 client.16467 trim_cache size 0 max 16384 2024-08-28T19:24:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:24:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:24:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:24:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:24:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:24:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:24:17.291 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:24:17.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 -- 172.21.15.16:0/1162915809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54200b4ca0 msgr2=0x7f54200b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:17.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54200b4ca0 0x7f54200b71b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f543000d720 tx=0x7f54300ce040 comp rx=0 tx=0).stop 2024-08-28T19:24:17.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 -- 172.21.15.16:0/1162915809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5438109bd0 msgr2=0x7f543819fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:17.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5438109bd0 0x7f543819fab0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f543000d750 tx=0x7f543000dc10 comp rx=0 tx=0).stop 2024-08-28T19:24:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 -- 172.21.15.16:0/1162915809 shutdown_connections 2024-08-28T19:24:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5438109bd0 0x7f543819fab0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f54200b4ca0 0x7f54200b71b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5438109280 0x7f543819a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 --2- 172.21.15.16:0/1162915809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5438103940 0x7f543819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 -- 172.21.15.16:0/1162915809 >> 172.21.15.16:0/1162915809 conn(0x7f54380ff3d0 msgr2=0x7f5438108930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 -- 172.21.15.16:0/1162915809 shutdown_connections 2024-08-28T19:24:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:17.282+0000 7f5440267640 1 -- 172.21.15.16:0/1162915809 wait complete. 2024-08-28T19:24:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:17 smithi016 ceph-mon[18556]: pgmap v436: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 84 KiB/s wr, 31 op/s 2024-08-28T19:24:17.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:17 smithi028 ceph-mon[21769]: pgmap v436: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 84 KiB/s wr, 31 op/s 2024-08-28T19:24:17.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:17 smithi115 ceph-mon[22184]: pgmap v436: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 84 KiB/s wr, 31 op/s 2024-08-28T19:24:17.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134298 6.33 MB/sec warmup 117 sec latency 1326.994 ms 2024-08-28T19:24:18.131 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:24:18.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:24:18.386 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:24:18.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134298 6.28 MB/sec warmup 118 sec latency 2327.023 ms 2024-08-28T19:24:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.955+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/3243098703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0c4108ab0 msgr2=0x7ff0c410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.955+0000 7ff0cb4c6640 1 --2- 172.21.15.16:0/3243098703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0c4108ab0 0x7ff0c410f2e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009ca0 tx=0x7ff0bc02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.955+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/3243098703 shutdown_connections 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.955+0000 7ff0cb4c6640 1 --2- 172.21.15.16:0/3243098703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0c4108ab0 0x7ff0c410f2e0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.955+0000 7ff0cb4c6640 1 --2- 172.21.15.16:0/3243098703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0c4108160 0x7ff0c4108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.955+0000 7ff0cb4c6640 1 --2- 172.21.15.16:0/3243098703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c4102820 0x7ff0c4102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.955+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/3243098703 >> 172.21.15.16:0/3243098703 conn(0x7ff0c40794e0 msgr2=0x7ff0c40798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.955+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/3243098703 shutdown_connections 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.955+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/3243098703 wait complete. 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0cb4c6640 1 Processor -- start 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0cb4c6640 1 -- start start 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0cb4c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0c4102820 0x7ff0c419af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0cb4c6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0c4108160 0x7ff0c419b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0cb4c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c4108ab0 0x7ff0c419ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0cb4c6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c4112710 con 0x7ff0c4102820 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0cb4c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c4112590 con 0x7ff0c4108ab0 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0cb4c6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c4112890 con 0x7ff0c4108160 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0c9262640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0c4102820 0x7ff0c419af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0c9262640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0c4102820 0x7ff0c419af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59002/0 (socket says 172.21.15.16:59002) 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0c9262640 1 -- 172.21.15.16:0/115958457 learned_addr learned my addr 172.21.15.16:0/115958457 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0c8a61640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0c4108160 0x7ff0c419b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.956+0000 7ff0c9a63640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c4108ab0 0x7ff0c419ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0c9262640 1 -- 172.21.15.16:0/115958457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0c4108160 msgr2=0x7ff0c419b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0c9262640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0c4108160 0x7ff0c419b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0c9262640 1 -- 172.21.15.16:0/115958457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c4108ab0 msgr2=0x7ff0c419ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0c9262640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c4108ab0 0x7ff0c419ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0c9262640 1 -- 172.21.15.16:0/115958457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c41a6200 con 0x7ff0c4102820 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0c8a61640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0c4108160 0x7ff0c419b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0c9a63640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c4108ab0 0x7ff0c419ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0c9262640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0c4102820 0x7ff0c419af80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000c920 tx=0x7ff0c000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0ba7fc640 1 -- 172.21.15.16:0/115958457 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0007c70 con 0x7ff0c4102820 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/115958457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c41a64f0 con 0x7ff0c4102820 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0ba7fc640 1 -- 172.21.15.16:0/115958457 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0c0007e10 con 0x7ff0c4102820 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0ba7fc640 1 -- 172.21.15.16:0/115958457 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0018820 con 0x7ff0c4102820 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.957+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/115958457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c41a6a80 con 0x7ff0c4102820 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.958+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/115958457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c4103aa0 con 0x7ff0c4102820 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.960+0000 7ff0ba7fc640 1 -- 172.21.15.16:0/115958457 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff0c000f450 con 0x7ff0c4102820 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.960+0000 7ff0ba7fc640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0b00b4ca0 0x7ff0b00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.961+0000 7ff0c8a61640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0b00b4ca0 0x7ff0b00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.961+0000 7ff0ba7fc640 1 -- 172.21.15.16:0/115958457 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff0c00e5de0 con 0x7ff0c4102820 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.961+0000 7ff0c8a61640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0b00b4ca0 0x7ff0b00b71b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4005e10 tx=0x7ff0b4005da0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:18.964+0000 7ff0ba7fc640 1 -- 172.21.15.16:0/115958457 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ff0c0014030 con 0x7ff0c4102820 2024-08-28T19:24:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.185+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/115958457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0c4102c30 con 0x7ff0c4102820 2024-08-28T19:24:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.186+0000 7ff0ba7fc640 1 -- 172.21.15.16:0/115958457 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11987 (secure 0 0 0) 0x7ff0c00ac890 con 0x7ff0c4102820 2024-08-28T19:24:19.187 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:24:19.188 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":33,"btime":"2024-08-28T19:24:02:274795+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:24:01.956748+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:24:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 33 2024-08-28T19:24:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/115958457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0b00b4ca0 msgr2=0x7ff0b00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0b00b4ca0 0x7ff0b00b71b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4005e10 tx=0x7ff0b4005da0 comp rx=0 tx=0).stop 2024-08-28T19:24:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/115958457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0c4102820 msgr2=0x7ff0c419af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0c4102820 0x7ff0c419af80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000c920 tx=0x7ff0c000cde0 comp rx=0 tx=0).stop 2024-08-28T19:24:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/115958457 shutdown_connections 2024-08-28T19:24:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0b00b4ca0 0x7ff0b00b71b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c4108ab0 0x7ff0c419ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0c4108160 0x7ff0c419b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 --2- 172.21.15.16:0/115958457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0c4102820 0x7ff0c419af80 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/115958457 >> 172.21.15.16:0/115958457 conn(0x7ff0c40794e0 msgr2=0x7ff0c4105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/115958457 shutdown_connections 2024-08-28T19:24:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:19.188+0000 7ff0cb4c6640 1 -- 172.21.15.16:0/115958457 wait complete. 2024-08-28T19:24:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:19 smithi016 ceph-mon[18556]: pgmap v437: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-08-28T19:24:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:19 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/115958457' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:24:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:19 smithi028 ceph-mon[21769]: pgmap v437: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-08-28T19:24:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:19 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/115958457' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:24:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:19 smithi115 ceph-mon[22184]: pgmap v437: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-08-28T19:24:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:19 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/115958457' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:24:19.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134298 6.23 MB/sec warmup 119 sec latency 3327.112 ms 2024-08-28T19:24:20.089 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:24:20.349 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:24:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.832+0000 7f5090ba2640 1 -- 172.21.15.16:0/147786711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f508c10dbb0 msgr2=0x7f508c110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.832+0000 7f5090ba2640 1 --2- 172.21.15.16:0/147786711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f508c10dbb0 0x7f508c110090 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5084009ca0 tx=0x7f508402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.832+0000 7f5090ba2640 1 -- 172.21.15.16:0/147786711 shutdown_connections 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.832+0000 7f5090ba2640 1 --2- 172.21.15.16:0/147786711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f508c10dbb0 0x7f508c110090 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.832+0000 7f5090ba2640 1 --2- 172.21.15.16:0/147786711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508c105030 0x7f508c10d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.832+0000 7f5090ba2640 1 --2- 172.21.15.16:0/147786711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f508c104600 0x7f508c104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.832+0000 7f5090ba2640 1 -- 172.21.15.16:0/147786711 >> 172.21.15.16:0/147786711 conn(0x7f508c0794d0 msgr2=0x7f508c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.832+0000 7f5090ba2640 1 -- 172.21.15.16:0/147786711 shutdown_connections 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.832+0000 7f5090ba2640 1 -- 172.21.15.16:0/147786711 wait complete. 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.833+0000 7f5090ba2640 1 Processor -- start 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.833+0000 7f5090ba2640 1 -- start start 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.833+0000 7f5090ba2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508c104600 0x7f508c102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.833+0000 7f5090ba2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f508c105030 0x7f508c102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.833+0000 7f5090ba2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f508c10dbb0 0x7f508c102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.833+0000 7f5090ba2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f508c113800 con 0x7f508c10dbb0 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.833+0000 7f5090ba2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f508c113680 con 0x7f508c104600 2024-08-28T19:24:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.833+0000 7f5090ba2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f508c113980 con 0x7f508c105030 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.833+0000 7f508ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f508c10dbb0 0x7f508c102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.833+0000 7f508ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f508c10dbb0 0x7f508c102f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59028/0 (socket says 172.21.15.16:59028) 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.833+0000 7f508ad9d640 1 -- 172.21.15.16:0/3394852449 learned_addr learned my addr 172.21.15.16:0/3394852449 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f508a59c640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508c104600 0x7f508c102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f5089d9b640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f508c105030 0x7f508c102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f508ad9d640 1 -- 172.21.15.16:0/3394852449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f508c105030 msgr2=0x7f508c102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f508ad9d640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f508c105030 0x7f508c102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f508ad9d640 1 -- 172.21.15.16:0/3394852449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508c104600 msgr2=0x7f508c102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f508ad9d640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508c104600 0x7f508c102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f508ad9d640 1 -- 172.21.15.16:0/3394852449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508c1a8470 con 0x7f508c10dbb0 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f508ad9d640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f508c10dbb0 0x7f508c102f90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f50840098d0 tx=0x7f50840317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f507b7fe640 1 -- 172.21.15.16:0/3394852449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084003b70 con 0x7f508c10dbb0 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f507b7fe640 1 -- 172.21.15.16:0/3394852449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5084003d10 con 0x7f508c10dbb0 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f5090ba2640 1 -- 172.21.15.16:0/3394852449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f508c1a8700 con 0x7f508c10dbb0 2024-08-28T19:24:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f507b7fe640 1 -- 172.21.15.16:0/3394852449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508402fa20 con 0x7f508c10dbb0 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.834+0000 7f5090ba2640 1 -- 172.21.15.16:0/3394852449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f508c1a8c40 con 0x7f508c10dbb0 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f5097117740 1 Processor -- start 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f507b7fe640 1 -- 172.21.15.16:0/3394852449 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f508402fbe0 con 0x7f508c10dbb0 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f5097117740 1 -- start start 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f5097117740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed056070 0x55a5ed3fb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f5097117740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed02d1b0 0x55a5ed3fb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f5097117740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 0x55a5ed3fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f5097117740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5ed372680 con 0x55a5ed02d1b0 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f5097117740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5ed361c10 con 0x55a5ed035390 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f5097117740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a5ed365bc0 con 0x55a5ed056070 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f508a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed02d1b0 0x55a5ed3fb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f508a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed02d1b0 0x55a5ed3fb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59032/0 (socket says 172.21.15.16:59032) 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.837+0000 7f508a59c640 1 -- 172.21.15.16:0/2907492608 learned_addr learned my addr 172.21.15.16:0/2907492608 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2907492608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 0x55a5ed3fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2907492608 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed056070 0x55a5ed3fb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f507b7fe640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50740b4ca0 0x7f50740b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f508a59c640 1 -- 172.21.15.16:0/2907492608 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed056070 msgr2=0x55a5ed3fb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f508a59c640 1 --2- 172.21.15.16:0/2907492608 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed056070 0x55a5ed3fb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f508a59c640 1 -- 172.21.15.16:0/2907492608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 msgr2=0x55a5ed3fbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f508a59c640 1 --2- 172.21.15.16:0/2907492608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 0x55a5ed3fbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f508a59c640 1 -- 172.21.15.16:0/2907492608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5ed0c1650 con 0x55a5ed02d1b0 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2907492608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 0x55a5ed3fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f508ad9d640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50740b4ca0 0x7f50740b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f508a59c640 1 --2- 172.21.15.16:0/2907492608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed02d1b0 0x55a5ed3fb850 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f507c00cbb0 tx=0x7f507c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad1258306aab447 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f507b7fe640 1 -- 172.21.15.16:0/3394852449 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f5084106c50 con 0x7f508c10dbb0 2024-08-28T19:24:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f5078ff9640 1 -- 172.21.15.16:0/2907492608 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c005220 con 0x55a5ed02d1b0 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f5078ff9640 1 -- 172.21.15.16:0/2907492608 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f507c004c50 con 0x55a5ed02d1b0 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.838+0000 7f5078ff9640 1 -- 172.21.15.16:0/2907492608 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c010720 con 0x55a5ed02d1b0 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.839+0000 7f5097117740 1 -- 172.21.15.16:0/2907492608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed02d1b0 msgr2=0x55a5ed3fb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.839+0000 7f5097117740 1 --2- 172.21.15.16:0/2907492608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed02d1b0 0x55a5ed3fb850 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f507c00cbb0 tx=0x7f507c00e720 comp rx=0 tx=0).stop 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.839+0000 7f508ad9d640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50740b4ca0 0x7f50740b71b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5084104630 tx=0x7f508402a8c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.839+0000 7f5097117740 1 -- 172.21.15.16:0/2907492608 shutdown_connections 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.839+0000 7f5097117740 1 --2- 172.21.15.16:0/2907492608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 0x55a5ed3fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.839+0000 7f5097117740 1 --2- 172.21.15.16:0/2907492608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed02d1b0 0x55a5ed3fb850 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.839+0000 7f5097117740 1 --2- 172.21.15.16:0/2907492608 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed056070 0x55a5ed3fb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.839+0000 7f5097117740 1 -- 172.21.15.16:0/2907492608 >> 172.21.15.16:0/2907492608 conn(0x55a5ed04a460 msgr2=0x55a5ed3f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.839+0000 7f5097117740 1 -- 172.21.15.16:0/2907492608 shutdown_connections 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.839+0000 7f5097117740 1 -- 172.21.15.16:0/2907492608 wait complete. 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5097117740 1 -- start start 2024-08-28T19:24:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5097117740 1 Processor -- start 2024-08-28T19:24:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5097117740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed02d1b0 0x55a5ed406a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5097117740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed406f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5097117740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed056070 0x55a5ed412de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5097117740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5ed1c4700 con 0x55a5ed035390 2024-08-28T19:24:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5097117740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5ed2e8750 con 0x55a5ed02d1b0 2024-08-28T19:24:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5097117740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a5ed409af0 con 0x55a5ed056070 2024-08-28T19:24:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5097117740 10 client.0 _ll_register_callbacks cb 0x55a5ed317e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:20.890 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:24:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f508a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed406f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f508a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed406f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59038/0 (socket says 172.21.15.16:59038) 2024-08-28T19:24:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f508a59c640 1 -- 172.21.15.16:0/3048072131 learned_addr learned my addr 172.21.15.16:0/3048072131 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5089d9b640 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed02d1b0 0x55a5ed406a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5089d9b640 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed056070 0x55a5ed412de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f508a59c640 1 -- 172.21.15.16:0/3048072131 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed056070 msgr2=0x55a5ed412de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f508a59c640 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed056070 0x55a5ed412de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f508a59c640 1 -- 172.21.15.16:0/3048072131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed02d1b0 msgr2=0x55a5ed406a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f508a59c640 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed02d1b0 0x55a5ed406a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f508a59c640 1 -- 172.21.15.16:0/3048072131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5ed321900 con 0x55a5ed035390 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.840+0000 7f5089d9b640 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed02d1b0 0x55a5ed406a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.841+0000 7f508a59c640 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed406f90 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f507c000c00 tx=0x7f507c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5379be596d9a2e98 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.841+0000 7f5071ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.841+0000 7f5071ffb640 1 -- 172.21.15.16:0/3048072131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c029060 con 0x55a5ed035390 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.841+0000 7f5071ffb640 1 -- 172.21.15.16:0/3048072131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f507c005010 con 0x55a5ed035390 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.841+0000 7f5097117740 1 -- 172.21.15.16:0/3048072131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f507c004c50 con 0x55a5ed035390 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.841+0000 7f5071ffb640 1 -- 172.21.15.16:0/3048072131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f507c030050 con 0x55a5ed035390 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.841+0000 7f5097117740 10 client.16485 fetch_fsmap learned FSMap version 33 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.841+0000 7f5097117740 1 -- 172.21.15.16:0/3048072131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a5ed415880 con 0x55a5ed035390 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.841+0000 7f5071ffb640 1 -- 172.21.15.16:0/3048072131 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f507c004c50 con 0x55a5ed035390 2024-08-28T19:24:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.841+0000 7f5071ffb640 1 -- 172.21.15.16:0/3048072131 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f507c01a410 con 0x55a5ed035390 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.841+0000 7f5097117740 10 client.16485 fetch_fsmap finished waiting for FSMap version 33 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.842+0000 7f5097117740 10 client.16485 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.842+0000 7f5097117740 20 client.16485 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.842+0000 7f5097117740 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55a5ed0149f0 0x55a5ed418c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.842+0000 7f5097117740 4 client.16485 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.842+0000 7f5097117740 1 -- 172.21.15.16:0/3048072131 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5ed3745d0 con 0x55a5ed0149f0 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.842+0000 7f5089d9b640 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55a5ed0149f0 0x55a5ed418c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.842+0000 7f5089d9b640 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55a5ed0149f0 0x55a5ed418c90 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=11f1b668e036c969 server_cookie=8eab066ac1bab82b in_seq=0 out_seq=0 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.842+0000 7f5071ffb640 10 client.16485 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5071ffb640 1 -- 172.21.15.16:0/3048072131 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a5ed3745d0 con 0x55a5ed0149f0 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5071ffb640 10 client.16485 handle_command_reply: tid=0 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5097117740 1 client.16485 shutdown 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5097117740 1 -- 172.21.15.16:0/3048072131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 msgr2=0x55a5ed406f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5097117740 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed406f90 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f507c000c00 tx=0x7f507c00c920 comp rx=0 tx=0).stop 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5097117740 1 -- 172.21.15.16:0/3048072131 shutdown_connections 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5097117740 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55a5ed0149f0 0x55a5ed418c90 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5097117740 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed056070 0x55a5ed412de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5097117740 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed406f90 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5097117740 1 --2- 172.21.15.16:0/3048072131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed02d1b0 0x55a5ed406a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5071ffb640 0 client.16485 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.843+0000 7f5097117740 1 -- 172.21.15.16:0/3048072131 >> 172.21.15.16:0/3048072131 conn(0x55a5ed04a460 msgr2=0x55a5ed3fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.844+0000 7f5097117740 1 -- 172.21.15.16:0/3048072131 shutdown_connections 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.844+0000 7f5097117740 1 -- 172.21.15.16:0/3048072131 wait complete. 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.844+0000 7f5097117740 20 client.16485 trim_cache size 0 max 16384 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5097117740 1 Processor -- start 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5097117740 1 -- start start 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5097117740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed02d1b0 0x55a5ed4052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5097117740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 0x55a5ed405810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5097117740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed056070 0x55a5ed3fe410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5097117740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f508c1b2e00 con 0x55a5ed056070 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5097117740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f508c1b4a60 con 0x55a5ed035390 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5097117740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f508c1b34d0 con 0x55a5ed02d1b0 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5089d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed056070 0x55a5ed3fe410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5089d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed056070 0x55a5ed3fe410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59042/0 (socket says 172.21.15.16:59042) 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5089d9b640 1 -- 172.21.15.16:0/2901552269 learned_addr learned my addr 172.21.15.16:0/2901552269 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.857+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2901552269 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed02d1b0 0x55a5ed4052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.858+0000 7f508a59c640 1 --2- 172.21.15.16:0/2901552269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 0x55a5ed405810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.858+0000 7f5089d9b640 1 -- 172.21.15.16:0/2901552269 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed02d1b0 msgr2=0x55a5ed4052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.858+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2901552269 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed02d1b0 0x55a5ed4052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.858+0000 7f5089d9b640 1 -- 172.21.15.16:0/2901552269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 msgr2=0x55a5ed405810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.858+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2901552269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 0x55a5ed405810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.858+0000 7f5089d9b640 1 -- 172.21.15.16:0/2901552269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5ed407610 con 0x55a5ed056070 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.858+0000 7f508a59c640 1 --2- 172.21.15.16:0/2901552269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 0x55a5ed405810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:24:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.858+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2901552269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed056070 0x55a5ed3fe410 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5080000c00 tx=0x7f5080004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36cab49190ff620f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.858+0000 7f5073fff640 1 -- 172.21.15.16:0/2901552269 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080004110 con 0x55a5ed056070 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.858+0000 7f5073fff640 1 -- 172.21.15.16:0/2901552269 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5080010db0 con 0x55a5ed056070 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.858+0000 7f5073fff640 1 -- 172.21.15.16:0/2901552269 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50800039f0 con 0x55a5ed056070 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 -- 172.21.15.16:0/2901552269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed056070 msgr2=0x55a5ed3fe410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 --2- 172.21.15.16:0/2901552269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed056070 0x55a5ed3fe410 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5080000c00 tx=0x7f5080004730 comp rx=0 tx=0).stop 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 -- 172.21.15.16:0/2901552269 shutdown_connections 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 --2- 172.21.15.16:0/2901552269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed056070 0x55a5ed3fe410 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 --2- 172.21.15.16:0/2901552269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed035390 0x55a5ed405810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 --2- 172.21.15.16:0/2901552269 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed02d1b0 0x55a5ed4052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 -- 172.21.15.16:0/2901552269 >> 172.21.15.16:0/2901552269 conn(0x55a5ed04a460 msgr2=0x55a5ed401b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 -- 172.21.15.16:0/2901552269 shutdown_connections 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 -- 172.21.15.16:0/2901552269 wait complete. 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 -- start start 2024-08-28T19:24:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 Processor -- start 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.859+0000 7f5097117740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed02d1b0 0x55a5ed41c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5097117740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed419850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5097117740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed056070 0x55a5ed419d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5097117740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f508c1b42d0 con 0x55a5ed035390 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5097117740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f508c1b3c00 con 0x55a5ed056070 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5097117740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f508c1b0de0 con 0x55a5ed02d1b0 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5097117740 10 client.0 _ll_register_callbacks cb 0x7f507c014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f508a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed419850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f508a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed419850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59052/0 (socket says 172.21.15.16:59052) 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f508a59c640 1 -- 172.21.15.16:0/2182296206 learned_addr learned my addr 172.21.15.16:0/2182296206 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed056070 0x55a5ed419d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed02d1b0 0x55a5ed41c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f508a59c640 1 -- 172.21.15.16:0/2182296206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed02d1b0 msgr2=0x55a5ed41c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f508a59c640 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed02d1b0 0x55a5ed41c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f508a59c640 1 -- 172.21.15.16:0/2182296206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed056070 msgr2=0x55a5ed419d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f508a59c640 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed056070 0x55a5ed419d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f508a59c640 1 -- 172.21.15.16:0/2182296206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5ed41a400 con 0x55a5ed035390 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed056070 0x55a5ed419d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f508a59c640 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed419850 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f507c01a860 tx=0x7f507c00c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da2bc7f7781740f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5071ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5071ffb640 1 -- 172.21.15.16:0/2182296206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c01e070 con 0x55a5ed035390 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5071ffb640 1 -- 172.21.15.16:0/2182296206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f507c004a40 con 0x55a5ed035390 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.860+0000 7f5097117740 1 -- 172.21.15.16:0/2182296206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5080010db0 con 0x55a5ed035390 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5071ffb640 1 -- 172.21.15.16:0/2182296206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f507c029070 con 0x55a5ed035390 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5097117740 10 client.16491 fetch_fsmap learned FSMap version 33 2024-08-28T19:24:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5097117740 1 -- 172.21.15.16:0/2182296206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a5ed4831b0 con 0x55a5ed035390 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5071ffb640 1 -- 172.21.15.16:0/2182296206 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5080010db0 con 0x55a5ed035390 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5071ffb640 1 -- 172.21.15.16:0/2182296206 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f507c035440 con 0x55a5ed035390 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5097117740 10 client.16491 fetch_fsmap finished waiting for FSMap version 33 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5097117740 10 client.16491 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5097117740 20 client.16491 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5097117740 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55a5ecfe9da0 0x55a5ed4859f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5097117740 4 client.16491 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5097117740 1 -- 172.21.15.16:0/2182296206 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5ed370070 con 0x55a5ecfe9da0 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.861+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55a5ecfe9da0 0x55a5ed4859f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5089d9b640 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55a5ecfe9da0 0x55a5ed4859f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9145a563a1c619d1 server_cookie=9bb959375c3d9bd2 in_seq=0 out_seq=0 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5071ffb640 10 client.16491 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5071ffb640 1 -- 172.21.15.16:0/2182296206 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5ed370070 con 0x55a5ecfe9da0 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5071ffb640 10 client.16491 handle_command_reply: tid=0 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 1 client.16491 shutdown 2024-08-28T19:24:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 1 -- 172.21.15.16:0/2182296206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 msgr2=0x55a5ed419850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed419850 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f507c01a860 tx=0x7f507c00c450 comp rx=0 tx=0).stop 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 1 -- 172.21.15.16:0/2182296206 shutdown_connections 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55a5ecfe9da0 0x55a5ed4859f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ed056070 0x55a5ed419d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ed035390 0x55a5ed419850 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5071ffb640 0 client.16491 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 1 --2- 172.21.15.16:0/2182296206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ed02d1b0 0x55a5ed41c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 1 -- 172.21.15.16:0/2182296206 >> 172.21.15.16:0/2182296206 conn(0x55a5ed04a460 msgr2=0x55a5ed401b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 1 -- 172.21.15.16:0/2182296206 shutdown_connections 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 1 -- 172.21.15.16:0/2182296206 wait complete. 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.862+0000 7f5097117740 20 client.16491 trim_cache size 0 max 16384 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 -- 172.21.15.16:0/3394852449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50740b4ca0 msgr2=0x7f50740b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50740b4ca0 0x7f50740b71b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5084104630 tx=0x7f508402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 -- 172.21.15.16:0/3394852449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f508c10dbb0 msgr2=0x7f508c102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f508c10dbb0 0x7f508c102f90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f50840098d0 tx=0x7f50840317b0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 -- 172.21.15.16:0/3394852449 shutdown_connections 2024-08-28T19:24:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f508c10dbb0 0x7f508c102f90 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f508c105030 0x7f508c102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50740b4ca0 0x7f50740b71b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 --2- 172.21.15.16:0/3394852449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508c104600 0x7f508c102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 -- 172.21.15.16:0/3394852449 >> 172.21.15.16:0/3394852449 conn(0x7f508c0794d0 msgr2=0x7f508c1055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 -- 172.21.15.16:0/3394852449 shutdown_connections 2024-08-28T19:24:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:20.863+0000 7f5090ba2640 1 -- 172.21.15.16:0/3394852449 wait complete. 2024-08-28T19:24:21.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:21 smithi016 ceph-mon[18556]: pgmap v438: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 43 KiB/s wr, 18 op/s 2024-08-28T19:24:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:21 smithi115 ceph-mon[22184]: pgmap v438: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 43 KiB/s wr, 18 op/s 2024-08-28T19:24:21.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:24:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:21 smithi028 ceph-mon[21769]: pgmap v438: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 43 KiB/s wr, 18 op/s 2024-08-28T19:24:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:24:21.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134483 0.74 MB/sec execute 1 sec latency 4980.745 ms 2024-08-28T19:24:22.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.374+0000 7fe883a39640 1 -- 172.21.15.16:0/144046727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c1047d0 msgr2=0x7fe87c104be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.374+0000 7fe883a39640 1 --2- 172.21.15.16:0/144046727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c1047d0 0x7fe87c104be0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe878009900 tx=0x7fe87802a410 comp rx=0 tx=0).stop 2024-08-28T19:24:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.374+0000 7fe883a39640 1 -- 172.21.15.16:0/144046727 shutdown_connections 2024-08-28T19:24:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.374+0000 7fe883a39640 1 --2- 172.21.15.16:0/144046727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c10dc30 0x7fe87c110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.374+0000 7fe883a39640 1 --2- 172.21.15.16:0/144046727 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe87c105120 0x7fe87c10d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.374+0000 7fe883a39640 1 --2- 172.21.15.16:0/144046727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c1047d0 0x7fe87c104be0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.374+0000 7fe883a39640 1 -- 172.21.15.16:0/144046727 >> 172.21.15.16:0/144046727 conn(0x7fe87c079510 msgr2=0x7fe87c079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.374+0000 7fe883a39640 1 -- 172.21.15.16:0/144046727 shutdown_connections 2024-08-28T19:24:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.375+0000 7fe883a39640 1 -- 172.21.15.16:0/144046727 wait complete. 2024-08-28T19:24:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.375+0000 7fe883a39640 1 Processor -- start 2024-08-28T19:24:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.375+0000 7fe883a39640 1 -- start start 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.375+0000 7fe883a39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c1047d0 0x7fe87c19a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.375+0000 7fe883a39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c105120 0x7fe87c19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.375+0000 7fe883a39640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe87c10dc30 0x7fe87c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.375+0000 7fe883a39640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1138b0 con 0x7fe87c105120 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.375+0000 7fe883a39640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c113730 con 0x7fe87c1047d0 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.375+0000 7fe883a39640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c113a30 con 0x7fe87c10dc30 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe880fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c105120 0x7fe87c19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe880fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c105120 0x7fe87c19a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59058/0 (socket says 172.21.15.16:59058) 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe880fd4640 1 -- 172.21.15.16:0/1581665628 learned_addr learned my addr 172.21.15.16:0/1581665628 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe8817d5640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c1047d0 0x7fe87c19a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe880fd4640 1 -- 172.21.15.16:0/1581665628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe87c10dc30 msgr2=0x7fe87c19fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe881fd6640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe87c10dc30 0x7fe87c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe880fd4640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe87c10dc30 0x7fe87c19fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe880fd4640 1 -- 172.21.15.16:0/1581665628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c1047d0 msgr2=0x7fe87c19a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe880fd4640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c1047d0 0x7fe87c19a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe880fd4640 1 -- 172.21.15.16:0/1581665628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c1a01f0 con 0x7fe87c105120 2024-08-28T19:24:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe880fd4640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c105120 0x7fe87c19a8e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00d750 tx=0x7fe86c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe8727fc640 1 -- 172.21.15.16:0/1581665628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c017070 con 0x7fe87c105120 2024-08-28T19:24:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe8727fc640 1 -- 172.21.15.16:0/1581665628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe86c0054d0 con 0x7fe87c105120 2024-08-28T19:24:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe883a39640 1 -- 172.21.15.16:0/1581665628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87c1a04e0 con 0x7fe87c105120 2024-08-28T19:24:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe8727fc640 1 -- 172.21.15.16:0/1581665628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c004c30 con 0x7fe87c105120 2024-08-28T19:24:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.376+0000 7fe883a39640 1 -- 172.21.15.16:0/1581665628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87c1a6d00 con 0x7fe87c105120 2024-08-28T19:24:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.379+0000 7fe889fae740 1 Processor -- start 2024-08-28T19:24:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.379+0000 7fe889fae740 1 -- start start 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe889fae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b3f070 0x5654d1ee4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe889fae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b161b0 0x5654d1ee4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe889fae740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b1e390 0x5654d1ee4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe889fae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5654d1e5b680 con 0x5654d1b3f070 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe889fae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5654d1e4ac10 con 0x5654d1b161b0 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe889fae740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5654d1e4ebc0 con 0x5654d1b1e390 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe881fd6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b3f070 0x5654d1ee4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe881fd6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b3f070 0x5654d1ee4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59062/0 (socket says 172.21.15.16:59062) 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe881fd6640 1 -- 172.21.15.16:0/2707486336 learned_addr learned my addr 172.21.15.16:0/2707486336 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe8727fc640 1 -- 172.21.15.16:0/1581665628 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe86c004dd0 con 0x7fe87c105120 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe8817d5640 1 --2- 172.21.15.16:0/2707486336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b161b0 0x5654d1ee4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe881fd6640 1 -- 172.21.15.16:0/2707486336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b1e390 msgr2=0x5654d1ee4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe881fd6640 1 --2- 172.21.15.16:0/2707486336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b1e390 0x5654d1ee4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe881fd6640 1 -- 172.21.15.16:0/2707486336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b161b0 msgr2=0x5654d1ee4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe881fd6640 1 --2- 172.21.15.16:0/2707486336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b161b0 0x5654d1ee4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe881fd6640 1 -- 172.21.15.16:0/2707486336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654d1baa650 con 0x5654d1b3f070 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.380+0000 7fe881fd6640 1 --2- 172.21.15.16:0/2707486336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b3f070 0x5654d1ee4350 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe874005fd0 tx=0x7fe87400d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618ee471d235c99f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.381+0000 7fe8677fe640 1 -- 172.21.15.16:0/2707486336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874018680 con 0x5654d1b3f070 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.381+0000 7fe8677fe640 1 -- 172.21.15.16:0/2707486336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe874018cd0 con 0x5654d1b3f070 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.381+0000 7fe8677fe640 1 -- 172.21.15.16:0/2707486336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87400fdf0 con 0x5654d1b3f070 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.381+0000 7fe8727fc640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8680b4d50 0x7fe8680b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.381+0000 7fe8817d5640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8680b4d50 0x7fe8680b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.381+0000 7fe8727fc640 1 -- 172.21.15.16:0/1581665628 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe86c0e5250 con 0x7fe87c105120 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe889fae740 1 -- 172.21.15.16:0/2707486336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b3f070 msgr2=0x5654d1ee4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe889fae740 1 --2- 172.21.15.16:0/2707486336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b3f070 0x5654d1ee4350 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe874005fd0 tx=0x7fe87400d710 comp rx=0 tx=0).stop 2024-08-28T19:24:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe8817d5640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8680b4d50 0x7fe8680b7260 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe878031440 tx=0x7fe878005db0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe889fae740 1 -- 172.21.15.16:0/2707486336 shutdown_connections 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe889fae740 1 --2- 172.21.15.16:0/2707486336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b1e390 0x5654d1ee4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe889fae740 1 --2- 172.21.15.16:0/2707486336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b161b0 0x5654d1ee4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe889fae740 1 --2- 172.21.15.16:0/2707486336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b3f070 0x5654d1ee4350 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe889fae740 1 -- 172.21.15.16:0/2707486336 >> 172.21.15.16:0/2707486336 conn(0x5654d1b33460 msgr2=0x5654d1ee2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe889fae740 1 -- 172.21.15.16:0/2707486336 shutdown_connections 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe889fae740 1 -- 172.21.15.16:0/2707486336 wait complete. 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe889fae740 1 -- start start 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.382+0000 7fe889fae740 1 Processor -- start 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe889fae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b161b0 0x5654d1eefb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe889fae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe889fae740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b3f070 0x5654d1efbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe889fae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5654d1cad700 con 0x5654d1b1e390 2024-08-28T19:24:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe889fae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5654d1dd1750 con 0x5654d1b161b0 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe889fae740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5654d1ef2b30 con 0x5654d1b3f070 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe889fae740 10 client.0 _ll_register_callbacks cb 0x5654d1e00e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe880fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe880fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59064/0 (socket says 172.21.15.16:59064) 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe880fd4640 1 -- 172.21.15.16:0/4166701216 learned_addr learned my addr 172.21.15.16:0/4166701216 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe8817d5640 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b161b0 0x5654d1eefb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe881fd6640 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b3f070 0x5654d1efbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe880fd4640 1 -- 172.21.15.16:0/4166701216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b3f070 msgr2=0x5654d1efbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe880fd4640 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b3f070 0x5654d1efbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe880fd4640 1 -- 172.21.15.16:0/4166701216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b161b0 msgr2=0x5654d1eefb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe880fd4640 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b161b0 0x5654d1eefb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe880fd4640 1 -- 172.21.15.16:0/4166701216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654d1e0a900 con 0x5654d1b1e390 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe8817d5640 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b161b0 0x5654d1eefb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:24:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe880fd4640 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef00c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0e2690 tx=0x7fe86c0b42f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5e9868af593df0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.383+0000 7fe864ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe864ff9640 1 -- 172.21.15.16:0/4166701216 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c020040 con 0x5654d1b1e390 2024-08-28T19:24:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe864ff9640 1 -- 172.21.15.16:0/4166701216 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe86c0a7860 con 0x5654d1b1e390 2024-08-28T19:24:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe889fae740 1 -- 172.21.15.16:0/4166701216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe874018cd0 con 0x5654d1b1e390 2024-08-28T19:24:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe864ff9640 1 -- 172.21.15.16:0/4166701216 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe86c0ab5d0 con 0x5654d1b1e390 2024-08-28T19:24:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe889fae740 10 client.16503 fetch_fsmap learned FSMap version 33 2024-08-28T19:24:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe889fae740 1 -- 172.21.15.16:0/4166701216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5654d1efe960 con 0x5654d1b1e390 2024-08-28T19:24:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe864ff9640 1 -- 172.21.15.16:0/4166701216 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe874018cd0 con 0x5654d1b1e390 2024-08-28T19:24:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe864ff9640 1 -- 172.21.15.16:0/4166701216 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fe86c0b4460 con 0x5654d1b1e390 2024-08-28T19:24:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe889fae740 10 client.16503 fetch_fsmap finished waiting for FSMap version 33 2024-08-28T19:24:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe889fae740 10 client.16503 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:24:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe889fae740 20 client.16503 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe889fae740 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5654d1afd9f0 0x5654d1f01dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe889fae740 4 client.16503 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:24:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.384+0000 7fe889fae740 1 -- 172.21.15.16:0/4166701216 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654d1e5d5d0 con 0x5654d1afd9f0 2024-08-28T19:24:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.385+0000 7fe8817d5640 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5654d1afd9f0 0x5654d1f01dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.385+0000 7fe864ff9640 10 client.16503 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:24:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.385+0000 7fe8817d5640 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5654d1afd9f0 0x5654d1f01dd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=55f0ce8b114fbde server_cookie=59862ca989a507ec in_seq=0 out_seq=0 2024-08-28T19:24:22.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe864ff9640 1 -- 172.21.15.16:0/4166701216 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5654d1e5d5d0 con 0x5654d1afd9f0 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe864ff9640 10 client.16503 handle_command_reply: tid=0 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 1 client.16503 shutdown 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 1 -- 172.21.15.16:0/4166701216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 msgr2=0x5654d1ef00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef00c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0e2690 tx=0x7fe86c0b42f0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 1 -- 172.21.15.16:0/4166701216 shutdown_connections 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5654d1afd9f0 0x5654d1f01dd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b3f070 0x5654d1efbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef00c0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 1 --2- 172.21.15.16:0/4166701216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b161b0 0x5654d1eefb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 1 -- 172.21.15.16:0/4166701216 >> 172.21.15.16:0/4166701216 conn(0x5654d1b33460 msgr2=0x5654d1ee7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe864ff9640 0 client.16503 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:24:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 1 -- 172.21.15.16:0/4166701216 shutdown_connections 2024-08-28T19:24:22.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 1 -- 172.21.15.16:0/4166701216 wait complete. 2024-08-28T19:24:22.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.386+0000 7fe889fae740 20 client.16503 trim_cache size 0 max 16384 2024-08-28T19:24:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.399+0000 7fe889fae740 1 Processor -- start 2024-08-28T19:24:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.399+0000 7fe889fae740 1 -- start start 2024-08-28T19:24:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe889fae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b161b0 0x5654d1ee5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe889fae740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b1e390 0x5654d1efbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe889fae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 0x5654d1efc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe889fae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1b0bd0 con 0x5654d1b161b0 2024-08-28T19:24:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe889fae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1b2830 con 0x5654d1b3f070 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe889fae740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1b12a0 con 0x5654d1b1e390 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe8817d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b161b0 0x5654d1ee5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe8817d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b161b0 0x5654d1ee5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59078/0 (socket says 172.21.15.16:59078) 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe8817d5640 1 -- 172.21.15.16:0/1538525109 learned_addr learned my addr 172.21.15.16:0/1538525109 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe880fd4640 1 --2- 172.21.15.16:0/1538525109 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b1e390 0x5654d1efbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe881fd6640 1 --2- 172.21.15.16:0/1538525109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 0x5654d1efc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe8817d5640 1 -- 172.21.15.16:0/1538525109 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b1e390 msgr2=0x5654d1efbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe8817d5640 1 --2- 172.21.15.16:0/1538525109 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b1e390 0x5654d1efbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe8817d5640 1 -- 172.21.15.16:0/1538525109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 msgr2=0x5654d1efc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe8817d5640 1 --2- 172.21.15.16:0/1538525109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 0x5654d1efc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe8817d5640 1 -- 172.21.15.16:0/1538525109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654d1ef22c0 con 0x5654d1b161b0 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe880fd4640 1 --2- 172.21.15.16:0/1538525109 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b1e390 0x5654d1efbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe881fd6640 1 --2- 172.21.15.16:0/1538525109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 0x5654d1efc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:24:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.400+0000 7fe8817d5640 1 --2- 172.21.15.16:0/1538525109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b161b0 0x5654d1ee5a90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe87802a930 tx=0x7fe87803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de3947cb8c553db server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.401+0000 7fe866ffd640 1 -- 172.21.15.16:0/1538525109 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878058070 con 0x5654d1b161b0 2024-08-28T19:24:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.401+0000 7fe866ffd640 1 -- 172.21.15.16:0/1538525109 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8780054f0 con 0x5654d1b161b0 2024-08-28T19:24:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.401+0000 7fe866ffd640 1 -- 172.21.15.16:0/1538525109 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878035040 con 0x5654d1b161b0 2024-08-28T19:24:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.401+0000 7fe889fae740 1 -- 172.21.15.16:0/1538525109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b161b0 msgr2=0x5654d1ee5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.401+0000 7fe889fae740 1 --2- 172.21.15.16:0/1538525109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b161b0 0x5654d1ee5a90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe87802a930 tx=0x7fe87803f000 comp rx=0 tx=0).stop 2024-08-28T19:24:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.401+0000 7fe889fae740 1 -- 172.21.15.16:0/1538525109 shutdown_connections 2024-08-28T19:24:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.401+0000 7fe889fae740 1 --2- 172.21.15.16:0/1538525109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 0x5654d1efc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.401+0000 7fe889fae740 1 --2- 172.21.15.16:0/1538525109 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b1e390 0x5654d1efbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.401+0000 7fe889fae740 1 --2- 172.21.15.16:0/1538525109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b161b0 0x5654d1ee5a90 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.401+0000 7fe889fae740 1 -- 172.21.15.16:0/1538525109 >> 172.21.15.16:0/1538525109 conn(0x5654d1b33460 msgr2=0x5654d1ee5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.401+0000 7fe889fae740 1 -- 172.21.15.16:0/1538525109 shutdown_connections 2024-08-28T19:24:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.402+0000 7fe889fae740 1 -- 172.21.15.16:0/1538525109 wait complete. 2024-08-28T19:24:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.402+0000 7fe889fae740 1 -- start start 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.402+0000 7fe889fae740 1 Processor -- start 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.402+0000 7fe889fae740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b161b0 0x5654d1ef3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.402+0000 7fe889fae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.402+0000 7fe889fae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 0x5654d1ef49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.402+0000 7fe889fae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1b20a0 con 0x5654d1b1e390 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.402+0000 7fe889fae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1b19d0 con 0x5654d1b3f070 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.402+0000 7fe889fae740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1aebb0 con 0x5654d1b161b0 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.402+0000 7fe889fae740 10 client.0 _ll_register_callbacks cb 0x7fe86c009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe880fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe880fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59084/0 (socket says 172.21.15.16:59084) 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe880fd4640 1 -- 172.21.15.16:0/911129322 learned_addr learned my addr 172.21.15.16:0/911129322 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe8817d5640 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b161b0 0x5654d1ef3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe881fd6640 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 0x5654d1ef49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe880fd4640 1 -- 172.21.15.16:0/911129322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b161b0 msgr2=0x5654d1ef3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe880fd4640 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b161b0 0x5654d1ef3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe880fd4640 1 -- 172.21.15.16:0/911129322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 msgr2=0x5654d1ef49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe880fd4640 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 0x5654d1ef49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe880fd4640 1 -- 172.21.15.16:0/911129322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654d1f6b8b0 con 0x5654d1b1e390 2024-08-28T19:24:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe881fd6640 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 0x5654d1ef49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:24:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe880fd4640 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef44a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0ab2d0 tx=0x7fe86c0b4690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f95bc5960e007c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe864ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe864ff9640 1 -- 172.21.15.16:0/911129322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c020030 con 0x5654d1b1e390 2024-08-28T19:24:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe864ff9640 1 -- 172.21.15.16:0/911129322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe86c0ce0d0 con 0x5654d1b1e390 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe889fae740 1 -- 172.21.15.16:0/911129322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8780054f0 con 0x5654d1b1e390 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.403+0000 7fe864ff9640 1 -- 172.21.15.16:0/911129322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe86c0aaf20 con 0x5654d1b1e390 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.404+0000 7fe889fae740 10 client.16509 fetch_fsmap learned FSMap version 33 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.404+0000 7fe889fae740 1 -- 172.21.15.16:0/911129322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5654d1f6ca10 con 0x5654d1b1e390 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.404+0000 7fe864ff9640 1 -- 172.21.15.16:0/911129322 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8780054f0 con 0x5654d1b1e390 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.404+0000 7fe864ff9640 1 -- 172.21.15.16:0/911129322 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fe86c0a6f40 con 0x5654d1b1e390 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.404+0000 7fe889fae740 10 client.16509 fetch_fsmap finished waiting for FSMap version 33 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.404+0000 7fe889fae740 10 client.16509 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.404+0000 7fe889fae740 20 client.16509 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.404+0000 7fe889fae740 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5654d1ad2da0 0x5654d1f6f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.404+0000 7fe889fae740 4 client.16509 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.404+0000 7fe889fae740 1 -- 172.21.15.16:0/911129322 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5654d1e59070 con 0x5654d1ad2da0 2024-08-28T19:24:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.404+0000 7fe8817d5640 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5654d1ad2da0 0x5654d1f6f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe864ff9640 10 client.16509 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe8817d5640 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5654d1ad2da0 0x5654d1f6f190 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=653c1e9e870da567 server_cookie=afcaf97fae5d6dc in_seq=0 out_seq=0 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe864ff9640 1 -- 172.21.15.16:0/911129322 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5654d1e59070 con 0x5654d1ad2da0 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe864ff9640 10 client.16509 handle_command_reply: tid=0 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 1 client.16509 shutdown 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 1 -- 172.21.15.16:0/911129322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 msgr2=0x5654d1ef44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef44a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0ab2d0 tx=0x7fe86c0b4690 comp rx=0 tx=0).stop 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 1 -- 172.21.15.16:0/911129322 shutdown_connections 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5654d1ad2da0 0x5654d1f6f190 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d1b3f070 0x5654d1ef49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d1b1e390 0x5654d1ef44a0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 1 --2- 172.21.15.16:0/911129322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d1b161b0 0x5654d1ef3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 1 -- 172.21.15.16:0/911129322 >> 172.21.15.16:0/911129322 conn(0x5654d1b33460 msgr2=0x5654d1ee7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe864ff9640 0 client.16509 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:24:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 1 -- 172.21.15.16:0/911129322 shutdown_connections 2024-08-28T19:24:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 1 -- 172.21.15.16:0/911129322 wait complete. 2024-08-28T19:24:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.405+0000 7fe889fae740 20 client.16509 trim_cache size 0 max 16384 2024-08-28T19:24:22.413 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:24:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 -- 172.21.15.16:0/1581665628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8680b4d50 msgr2=0x7fe8680b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8680b4d50 0x7fe8680b7260 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe878031440 tx=0x7fe878005db0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 -- 172.21.15.16:0/1581665628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c105120 msgr2=0x7fe87c19a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c105120 0x7fe87c19a8e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00d750 tx=0x7fe86c00dc10 comp rx=0 tx=0).stop 2024-08-28T19:24:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 -- 172.21.15.16:0/1581665628 shutdown_connections 2024-08-28T19:24:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe87c10dc30 0x7fe87c19fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c105120 0x7fe87c19a8e0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8680b4d50 0x7fe8680b7260 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 --2- 172.21.15.16:0/1581665628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c1047d0 0x7fe87c19a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 -- 172.21.15.16:0/1581665628 >> 172.21.15.16:0/1581665628 conn(0x7fe87c079510 msgr2=0x7fe87c100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 -- 172.21.15.16:0/1581665628 shutdown_connections 2024-08-28T19:24:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:22.406+0000 7fe883a39640 1 -- 172.21.15.16:0/1581665628 wait complete. 2024-08-28T19:24:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:22 smithi115 ceph-mon[22184]: pgmap v439: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 40 KiB/s wr, 22 op/s 2024-08-28T19:24:22.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:22 smithi016 ceph-mon[18556]: pgmap v439: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 40 KiB/s wr, 22 op/s 2024-08-28T19:24:22.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:22 smithi028 ceph-mon[21769]: pgmap v439: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 40 KiB/s wr, 22 op/s 2024-08-28T19:24:22.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134483 0.37 MB/sec execute 2 sec latency 1298.788 ms 2024-08-28T19:24:23.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:24:23.490 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:24:23.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134483 0.25 MB/sec execute 3 sec latency 2298.844 ms 2024-08-28T19:24:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.955+0000 7f22cc53b640 1 -- 172.21.15.16:0/1382412396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22c4108140 msgr2=0x7f22c4108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.955+0000 7f22cc53b640 1 --2- 172.21.15.16:0/1382412396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22c4108140 0x7f22c4108550 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f22b4009900 tx=0x7f22b402a410 comp rx=0 tx=0).stop 2024-08-28T19:24:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.955+0000 7f22cc53b640 1 -- 172.21.15.16:0/1382412396 shutdown_connections 2024-08-28T19:24:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.955+0000 7f22cc53b640 1 --2- 172.21.15.16:0/1382412396 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c4108a90 0x7f22c410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.955+0000 7f22cc53b640 1 --2- 172.21.15.16:0/1382412396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22c4108140 0x7f22c4108550 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.955+0000 7f22cc53b640 1 --2- 172.21.15.16:0/1382412396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c4102800 0x7f22c4102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.955+0000 7f22cc53b640 1 -- 172.21.15.16:0/1382412396 >> 172.21.15.16:0/1382412396 conn(0x7f22c40794d0 msgr2=0x7f22c40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.955+0000 7f22cc53b640 1 -- 172.21.15.16:0/1382412396 shutdown_connections 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.955+0000 7f22cc53b640 1 -- 172.21.15.16:0/1382412396 wait complete. 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22cc53b640 1 Processor -- start 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22cc53b640 1 -- start start 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22cc53b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c4102800 0x7f22c419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22cc53b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22c4108140 0x7f22c419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22cc53b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c4108a90 0x7f22c419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22cc53b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22c4112740 con 0x7f22c4108140 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22cc53b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22c41125c0 con 0x7f22c4108a90 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22cc53b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f22c41128c0 con 0x7f22c4102800 2024-08-28T19:24:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22c9ad6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22c4108140 0x7f22c419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22c9ad6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22c4108140 0x7f22c419a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59094/0 (socket says 172.21.15.16:59094) 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22c9ad6640 1 -- 172.21.15.16:0/2904162259 learned_addr learned my addr 172.21.15.16:0/2904162259 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c4102800 0x7f22c419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.956+0000 7f22caad8640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c4108a90 0x7f22c419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.957+0000 7f22c9ad6640 1 -- 172.21.15.16:0/2904162259 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c4102800 msgr2=0x7f22c419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.957+0000 7f22c9ad6640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c4102800 0x7f22c419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.957+0000 7f22c9ad6640 1 -- 172.21.15.16:0/2904162259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c4108a90 msgr2=0x7f22c419faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.957+0000 7f22c9ad6640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c4108a90 0x7f22c419faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.957+0000 7f22c9ad6640 1 -- 172.21.15.16:0/2904162259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c41a0110 con 0x7f22c4108140 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.957+0000 7f22c9ad6640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22c4108140 0x7f22c419a800 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f22b4031a90 tx=0x7f22b4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.957+0000 7f22bb7fe640 1 -- 172.21.15.16:0/2904162259 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b403f030 con 0x7f22c4108140 2024-08-28T19:24:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.957+0000 7f22bb7fe640 1 -- 172.21.15.16:0/2904162259 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22b4004490 con 0x7f22c4108140 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.957+0000 7f22cc53b640 1 -- 172.21.15.16:0/2904162259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c41a03a0 con 0x7f22c4108140 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.957+0000 7f22bb7fe640 1 -- 172.21.15.16:0/2904162259 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b4035070 con 0x7f22c4108140 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.957+0000 7f22cc53b640 1 -- 172.21.15.16:0/2904162259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c41a6c20 con 0x7f22c4108140 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22d2ab0740 1 Processor -- start 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22d2ab0740 1 -- start start 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65623a070 0x55a6565df310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565df850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a656219390 0x55a6565dfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a656556680 con 0x55a6562111b0 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a656545c10 con 0x55a65623a070 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a656549bc0 con 0x55a656219390 2024-08-28T19:24:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22bb7fe640 1 -- 172.21.15.16:0/2904162259 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f22b4042030 con 0x7f22c4108140 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22ca2d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565df850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22ca2d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565df850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59110/0 (socket says 172.21.15.16:59110) 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.960+0000 7f22ca2d7640 1 -- 172.21.15.16:0/1290030779 learned_addr learned my addr 172.21.15.16:0/1290030779 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22caad8640 1 --2- 172.21.15.16:0/1290030779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65623a070 0x55a6565df310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22ca2d7640 1 -- 172.21.15.16:0/1290030779 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a656219390 msgr2=0x55a6565dfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/1290030779 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a656219390 0x55a6565dfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22ca2d7640 1 -- 172.21.15.16:0/1290030779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65623a070 msgr2=0x55a6565df310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/1290030779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65623a070 0x55a6565df310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22ca2d7640 1 -- 172.21.15.16:0/1290030779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6562a5650 con 0x55a6562111b0 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/1290030779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565df850 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f22c000ad00 tx=0x7f22c000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5ad909ad5453b6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22b8ff9640 1 -- 172.21.15.16:0/1290030779 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c001d070 con 0x55a6562111b0 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22b8ff9640 1 -- 172.21.15.16:0/1290030779 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22c0018af0 con 0x55a6562111b0 2024-08-28T19:24:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22b8ff9640 1 -- 172.21.15.16:0/1290030779 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0010b30 con 0x55a6562111b0 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22bb7fe640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22b00b4d50 0x7f22b00b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.961+0000 7f22caad8640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22b00b4d50 0x7f22b00b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.962+0000 7f22bb7fe640 1 -- 172.21.15.16:0/2904162259 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f22b4105d60 con 0x7f22c4108140 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.962+0000 7f22d2ab0740 1 -- 172.21.15.16:0/1290030779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 msgr2=0x55a6565df850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.962+0000 7f22caad8640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22b00b4d50 0x7f22b00b7260 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00a510 tx=0x7f22bc00d040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.962+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/1290030779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565df850 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f22c000ad00 tx=0x7f22c000f790 comp rx=0 tx=0).stop 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.962+0000 7f22d2ab0740 1 -- 172.21.15.16:0/1290030779 shutdown_connections 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.962+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/1290030779 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a656219390 0x55a6565dfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.962+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/1290030779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565df850 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.962+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/1290030779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65623a070 0x55a6565df310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.962+0000 7f22d2ab0740 1 -- 172.21.15.16:0/1290030779 >> 172.21.15.16:0/1290030779 conn(0x55a65622e460 msgr2=0x55a6565dd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.962+0000 7f22d2ab0740 1 -- 172.21.15.16:0/1290030779 shutdown_connections 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.962+0000 7f22d2ab0740 1 -- 172.21.15.16:0/1290030779 wait complete. 2024-08-28T19:24:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22d2ab0740 1 -- start start 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22d2ab0740 1 Processor -- start 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6562111b0 0x55a6565eaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565eb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65623a070 0x55a6565f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a6563a8700 con 0x55a656219390 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a6564cc750 con 0x55a65623a070 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a6565edaf0 con 0x55a6562111b0 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22c9ad6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22d2ab0740 10 client.0 _ll_register_callbacks cb 0x55a6564fbe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22c9ad6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565eb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59118/0 (socket says 172.21.15.16:59118) 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22c9ad6640 1 -- 172.21.15.16:0/2305806609 learned_addr learned my addr 172.21.15.16:0/2305806609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6562111b0 0x55a6565eaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.963+0000 7f22caad8640 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65623a070 0x55a6565f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22c9ad6640 1 -- 172.21.15.16:0/2305806609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6562111b0 msgr2=0x55a6565eaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22c9ad6640 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6562111b0 0x55a6565eaae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22c9ad6640 1 -- 172.21.15.16:0/2305806609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65623a070 msgr2=0x55a6565f6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22c9ad6640 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65623a070 0x55a6565f6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22c9ad6640 1 -- 172.21.15.16:0/2305806609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a656505900 con 0x55a656219390 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6562111b0 0x55a6565eaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22c9ad6640 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565eb020 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f22b40c8160 tx=0x7f22b40c80b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d02ebec41945267 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22adffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22adffb640 1 -- 172.21.15.16:0/2305806609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b4049070 con 0x55a656219390 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22d2ab0740 1 -- 172.21.15.16:0/2305806609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22c0018af0 con 0x55a656219390 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22adffb640 1 -- 172.21.15.16:0/2305806609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f22b40d0530 con 0x55a656219390 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22adffb640 1 -- 172.21.15.16:0/2305806609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f22b40d0730 con 0x55a656219390 2024-08-28T19:24:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22d2ab0740 10 client.16521 fetch_fsmap learned FSMap version 33 2024-08-28T19:24:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22d2ab0740 1 -- 172.21.15.16:0/2305806609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a6565f98f0 con 0x55a656219390 2024-08-28T19:24:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.964+0000 7f22adffb640 1 -- 172.21.15.16:0/2305806609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22c0018af0 con 0x55a656219390 2024-08-28T19:24:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.965+0000 7f22adffb640 1 -- 172.21.15.16:0/2305806609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f22b40d0150 con 0x55a656219390 2024-08-28T19:24:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.965+0000 7f22d2ab0740 10 client.16521 fetch_fsmap finished waiting for FSMap version 33 2024-08-28T19:24:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.965+0000 7f22d2ab0740 10 client.16521 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:24:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.965+0000 7f22d2ab0740 20 client.16521 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.965+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a6561f89f0 0x55a6565fcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.965+0000 7f22d2ab0740 4 client.16521 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:24:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.965+0000 7f22d2ab0740 1 -- 172.21.15.16:0/2305806609 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6565585d0 con 0x55a6561f89f0 2024-08-28T19:24:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.965+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a6561f89f0 0x55a6565fcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.966+0000 7f22adffb640 10 client.16521 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.966+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a6561f89f0 0x55a6565fcd40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a91910f134ecf34e server_cookie=f97d134e1f0c923e in_seq=0 out_seq=0 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.966+0000 7f22adffb640 1 -- 172.21.15.16:0/2305806609 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a6565585d0 con 0x55a6561f89f0 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.966+0000 7f22adffb640 10 client.16521 handle_command_reply: tid=0 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 1 client.16521 shutdown 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 1 -- 172.21.15.16:0/2305806609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 msgr2=0x55a6565eb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565eb020 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f22b40c8160 tx=0x7f22b40c80b0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 1 -- 172.21.15.16:0/2305806609 shutdown_connections 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a6561f89f0 0x55a6565fcd40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65623a070 0x55a6565f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565eb020 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/2305806609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6562111b0 0x55a6565eaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22adffb640 0 client.16521 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:24:23.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 1 -- 172.21.15.16:0/2305806609 >> 172.21.15.16:0/2305806609 conn(0x55a65622e460 msgr2=0x55a6565e24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:23.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 1 -- 172.21.15.16:0/2305806609 shutdown_connections 2024-08-28T19:24:23.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 1 -- 172.21.15.16:0/2305806609 wait complete. 2024-08-28T19:24:23.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.967+0000 7f22d2ab0740 20 client.16521 trim_cache size 0 max 16384 2024-08-28T19:24:23.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.980+0000 7f22d2ab0740 1 Processor -- start 2024-08-28T19:24:23.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.980+0000 7f22d2ab0740 1 -- start start 2024-08-28T19:24:23.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6561f89f0 0x55a6565dd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6562111b0 0x55a6565f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565e2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22c41b0b20 con 0x55a656219390 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22c41b2780 con 0x55a6562111b0 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f22c41b11f0 con 0x55a6561f89f0 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22caad8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565e2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22caad8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565e2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59134/0 (socket says 172.21.15.16:59134) 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22caad8640 1 -- 172.21.15.16:0/909770168 learned_addr learned my addr 172.21.15.16:0/909770168 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/909770168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6561f89f0 0x55a6565dd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22c9ad6640 1 --2- 172.21.15.16:0/909770168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6562111b0 0x55a6565f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22caad8640 1 -- 172.21.15.16:0/909770168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6561f89f0 msgr2=0x55a6565dd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22caad8640 1 --2- 172.21.15.16:0/909770168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6561f89f0 0x55a6565dd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22caad8640 1 -- 172.21.15.16:0/909770168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6562111b0 msgr2=0x55a6565f6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22caad8640 1 --2- 172.21.15.16:0/909770168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6562111b0 0x55a6565f6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22caad8640 1 -- 172.21.15.16:0/909770168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6565ebda0 con 0x55a656219390 2024-08-28T19:24:23.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22caad8640 1 --2- 172.21.15.16:0/909770168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565e2410 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f22bc018940 tx=0x7f22bc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc40bc110af6e4cc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.981+0000 7f22affff640 1 -- 172.21.15.16:0/909770168 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc007dc0 con 0x55a656219390 2024-08-28T19:24:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.982+0000 7f22affff640 1 -- 172.21.15.16:0/909770168 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22bc029070 con 0x55a656219390 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.982+0000 7f22affff640 1 -- 172.21.15.16:0/909770168 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc016630 con 0x55a656219390 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.982+0000 7f22d2ab0740 1 -- 172.21.15.16:0/909770168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 msgr2=0x55a6565e2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.982+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/909770168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565e2410 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f22bc018940 tx=0x7f22bc017000 comp rx=0 tx=0).stop 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.982+0000 7f22d2ab0740 1 -- 172.21.15.16:0/909770168 shutdown_connections 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.982+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/909770168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a656219390 0x55a6565e2410 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.982+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/909770168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6562111b0 0x55a6565f6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.982+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/909770168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6561f89f0 0x55a6565dd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.982+0000 7f22d2ab0740 1 -- 172.21.15.16:0/909770168 >> 172.21.15.16:0/909770168 conn(0x55a65622e460 msgr2=0x55a6565dcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.982+0000 7f22d2ab0740 1 -- 172.21.15.16:0/909770168 shutdown_connections 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.982+0000 7f22d2ab0740 1 -- 172.21.15.16:0/909770168 wait complete. 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22d2ab0740 1 -- start start 2024-08-28T19:24:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22d2ab0740 1 Processor -- start 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6561f89f0 0x55a6565ef600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565efbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22d2ab0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a656219390 0x55a6566606d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22c41b1ff0 con 0x55a6562111b0 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22c41b1920 con 0x55a6561f89f0 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22d2ab0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f22c41aeb00 con 0x55a656219390 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22d2ab0740 10 client.0 _ll_register_callbacks cb 0x7f22b4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22c9ad6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565efbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22c9ad6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565efbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59142/0 (socket says 172.21.15.16:59142) 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22c9ad6640 1 -- 172.21.15.16:0/867280312 learned_addr learned my addr 172.21.15.16:0/867280312 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.983+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6561f89f0 0x55a6565ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22caad8640 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a656219390 0x55a6566606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22c9ad6640 1 -- 172.21.15.16:0/867280312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a656219390 msgr2=0x55a6566606d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22c9ad6640 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a656219390 0x55a6566606d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22c9ad6640 1 -- 172.21.15.16:0/867280312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6561f89f0 msgr2=0x55a6565ef600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22c9ad6640 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6561f89f0 0x55a6565ef600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22c9ad6640 1 -- 172.21.15.16:0/867280312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a656660d40 con 0x55a6562111b0 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22c9ad6640 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565efbc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f22b4006010 tx=0x7f22b40c8030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3faf76df328d7fbb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22adffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22adffb640 1 -- 172.21.15.16:0/867280312 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b4049070 con 0x55a6562111b0 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22adffb640 1 -- 172.21.15.16:0/867280312 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f22b40d4f70 con 0x55a6562111b0 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22d2ab0740 1 -- 172.21.15.16:0/867280312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22bc029070 con 0x55a6562111b0 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22adffb640 1 -- 172.21.15.16:0/867280312 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f22b40cb850 con 0x55a6562111b0 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22d2ab0740 10 client.16527 fetch_fsmap learned FSMap version 33 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22d2ab0740 1 -- 172.21.15.16:0/867280312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a656668040 con 0x55a6562111b0 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.984+0000 7f22adffb640 1 -- 172.21.15.16:0/867280312 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22bc029070 con 0x55a6562111b0 2024-08-28T19:24:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.985+0000 7f22adffb640 1 -- 172.21.15.16:0/867280312 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f22b403bbb0 con 0x55a6562111b0 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.985+0000 7f22d2ab0740 10 client.16527 fetch_fsmap finished waiting for FSMap version 33 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.985+0000 7f22d2ab0740 10 client.16527 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.985+0000 7f22d2ab0740 20 client.16527 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.985+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a65623a070 0x55a65666a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.985+0000 7f22d2ab0740 4 client.16527 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.985+0000 7f22d2ab0740 1 -- 172.21.15.16:0/867280312 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a656554070 con 0x55a65623a070 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.985+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a65623a070 0x55a65666a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22adffb640 10 client.16527 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22ca2d7640 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a65623a070 0x55a65666a5e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d46bdb8098839f06 server_cookie=e25436760439da67 in_seq=0 out_seq=0 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22adffb640 1 -- 172.21.15.16:0/867280312 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a656554070 con 0x55a65623a070 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22adffb640 10 client.16527 handle_command_reply: tid=0 2024-08-28T19:24:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 1 client.16527 shutdown 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 1 -- 172.21.15.16:0/867280312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 msgr2=0x55a6565efbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565efbc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f22b4006010 tx=0x7f22b40c8030 comp rx=0 tx=0).stop 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 1 -- 172.21.15.16:0/867280312 shutdown_connections 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a65623a070 0x55a65666a5e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a656219390 0x55a6566606d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6562111b0 0x55a6565efbc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 1 --2- 172.21.15.16:0/867280312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6561f89f0 0x55a6565ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 1 -- 172.21.15.16:0/867280312 >> 172.21.15.16:0/867280312 conn(0x55a65622e460 msgr2=0x55a6565e2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22adffb640 0 client.16527 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 1 -- 172.21.15.16:0/867280312 shutdown_connections 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 1 -- 172.21.15.16:0/867280312 wait complete. 2024-08-28T19:24:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.986+0000 7f22d2ab0740 20 client.16527 trim_cache size 0 max 16384 2024-08-28T19:24:23.993 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:24:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 -- 172.21.15.16:0/2904162259 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22b00b4d50 msgr2=0x7f22b00b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22b00b4d50 0x7f22b00b7260 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00a510 tx=0x7f22bc00d040 comp rx=0 tx=0).stop 2024-08-28T19:24:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 -- 172.21.15.16:0/2904162259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22c4108140 msgr2=0x7f22c419a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22c4108140 0x7f22c419a800 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f22b4031a90 tx=0x7f22b4004270 comp rx=0 tx=0).stop 2024-08-28T19:24:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 -- 172.21.15.16:0/2904162259 shutdown_connections 2024-08-28T19:24:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22b00b4d50 0x7f22b00b7260 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c4108a90 0x7f22c419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22c4108140 0x7f22c419a800 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 --2- 172.21.15.16:0/2904162259 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c4102800 0x7f22c419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 -- 172.21.15.16:0/2904162259 >> 172.21.15.16:0/2904162259 conn(0x7f22c40794d0 msgr2=0x7f22c406afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 -- 172.21.15.16:0/2904162259 shutdown_connections 2024-08-28T19:24:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:23.987+0000 7f22cc53b640 1 -- 172.21.15.16:0/2904162259 wait complete. 2024-08-28T19:24:24.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:24 smithi016 ceph-mon[18556]: pgmap v440: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 13 op/s 2024-08-28T19:24:24.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:24:24.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:24 smithi115 ceph-mon[22184]: pgmap v440: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 13 op/s 2024-08-28T19:24:24.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134483 0.19 MB/sec execute 4 sec latency 3298.893 ms 2024-08-28T19:24:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:24:25.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:24 smithi028 ceph-mon[21769]: pgmap v440: 129 pgs: 129 active+clean; 78 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 13 op/s 2024-08-28T19:24:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.533+0000 7fdf0091d640 1 -- 172.21.15.16:0/441222874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc105910 msgr2=0x7fdefc10f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.533+0000 7fdf0091d640 1 --2- 172.21.15.16:0/441222874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc105910 0x7fdefc10f3a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009ca0 tx=0x7fdef402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.533+0000 7fdf0091d640 1 -- 172.21.15.16:0/441222874 shutdown_connections 2024-08-28T19:24:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.533+0000 7fdf0091d640 1 --2- 172.21.15.16:0/441222874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc105910 0x7fdefc10f3a0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.533+0000 7fdf0091d640 1 --2- 172.21.15.16:0/441222874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdefc104fc0 0x7fdefc1053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.533+0000 7fdf0091d640 1 --2- 172.21.15.16:0/441222874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdefc102fb0 0x7fdefc1033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.533+0000 7fdf0091d640 1 -- 172.21.15.16:0/441222874 >> 172.21.15.16:0/441222874 conn(0x7fdefc079c40 msgr2=0x7fdefc07a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.533+0000 7fdf0091d640 1 -- 172.21.15.16:0/441222874 shutdown_connections 2024-08-28T19:24:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.533+0000 7fdf0091d640 1 -- 172.21.15.16:0/441222874 wait complete. 2024-08-28T19:24:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.534+0000 7fdf0091d640 1 Processor -- start 2024-08-28T19:24:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.534+0000 7fdf0091d640 1 -- start start 2024-08-28T19:24:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.534+0000 7fdf0091d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdefc102fb0 0x7fdefc19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdf0091d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc104fc0 0x7fdefc19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdf0091d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdefc105910 0x7fdefc19fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdf0091d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1127f0 con 0x7fdefc104fc0 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdf0091d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc112670 con 0x7fdefc102fb0 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdf0091d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc112970 con 0x7fdefc105910 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdef9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc104fc0 0x7fdefc19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdefad9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdefc105910 0x7fdefc19fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdefa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdefc102fb0 0x7fdefc19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdefa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdefc102fb0 0x7fdefc19a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:58950/0 (socket says 172.21.15.16:58950) 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdefad9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdefc105910 0x7fdefc19fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:42786/0 (socket says 172.21.15.16:42786) 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdefa59c640 1 -- 172.21.15.16:0/2914946857 learned_addr learned my addr 172.21.15.16:0/2914946857 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdef9d9b640 1 -- 172.21.15.16:0/2914946857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdefc105910 msgr2=0x7fdefc19fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdef9d9b640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdefc105910 0x7fdefc19fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdef9d9b640 1 -- 172.21.15.16:0/2914946857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdefc102fb0 msgr2=0x7fdefc19a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdef9d9b640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdefc102fb0 0x7fdefc19a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdef9d9b640 1 -- 172.21.15.16:0/2914946857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdefc1a01c0 con 0x7fdefc104fc0 2024-08-28T19:24:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.535+0000 7fdef9d9b640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc104fc0 0x7fdefc19a8b0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fdefc075800 tx=0x7fdef0010050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.536+0000 7fdeeb7fe640 1 -- 172.21.15.16:0/2914946857 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef000e2e0 con 0x7fdefc104fc0 2024-08-28T19:24:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.536+0000 7fdeeb7fe640 1 -- 172.21.15.16:0/2914946857 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdef000e480 con 0x7fdefc104fc0 2024-08-28T19:24:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.536+0000 7fdf0091d640 1 -- 172.21.15.16:0/2914946857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdefc1a04b0 con 0x7fdefc104fc0 2024-08-28T19:24:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.536+0000 7fdeeb7fe640 1 -- 172.21.15.16:0/2914946857 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef000ce10 con 0x7fdefc104fc0 2024-08-28T19:24:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.536+0000 7fdf0091d640 1 -- 172.21.15.16:0/2914946857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdefc0feb50 con 0x7fdefc104fc0 2024-08-28T19:24:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.538+0000 7fdf06e92740 1 Processor -- start 2024-08-28T19:24:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.538+0000 7fdf06e92740 1 -- start start 2024-08-28T19:24:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.538+0000 7fdeeb7fe640 1 -- 172.21.15.16:0/2914946857 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fdef0010820 con 0x7fdefc104fc0 2024-08-28T19:24:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1dc8070 0x5647a216d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1d9f1b0 0x5647a216d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1da7390 0x5647a216dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdf06e92740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5647a20e4680 con 0x5647a1dc8070 2024-08-28T19:24:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdf06e92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5647a20d3c10 con 0x5647a1d9f1b0 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdf06e92740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5647a20d7bc0 con 0x5647a1da7390 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdefad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1dc8070 0x5647a216d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdefad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1dc8070 0x5647a216d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45970/0 (socket says 172.21.15.16:45970) 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdefad9d640 1 -- 172.21.15.16:0/3381584246 learned_addr learned my addr 172.21.15.16:0/3381584246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdefa59c640 1 --2- 172.21.15.16:0/3381584246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1d9f1b0 0x5647a216d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdef9d9b640 1 --2- 172.21.15.16:0/3381584246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1da7390 0x5647a216dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdefad9d640 1 -- 172.21.15.16:0/3381584246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1da7390 msgr2=0x5647a216dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdefad9d640 1 --2- 172.21.15.16:0/3381584246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1da7390 0x5647a216dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdefad9d640 1 -- 172.21.15.16:0/3381584246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1d9f1b0 msgr2=0x5647a216d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdefad9d640 1 --2- 172.21.15.16:0/3381584246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1d9f1b0 0x5647a216d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdefad9d640 1 -- 172.21.15.16:0/3381584246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647a1e33650 con 0x5647a1dc8070 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdefa59c640 1 --2- 172.21.15.16:0/3381584246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1d9f1b0 0x5647a216d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.539+0000 7fdeeb7fe640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdee40b4d50 0x7fdee40b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.540+0000 7fdefa59c640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdee40b4d50 0x7fdee40b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.540+0000 7fdefad9d640 1 --2- 172.21.15.16:0/3381584246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1dc8070 0x5647a216d310 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fdef402f4c0 tx=0x7fdef4031ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90945ce7b66fb143 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.540+0000 7fdeeb7fe640 1 -- 172.21.15.16:0/2914946857 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdef00e8dc0 con 0x7fdefc104fc0 2024-08-28T19:24:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.540+0000 7fdee8ff9640 1 -- 172.21.15.16:0/3381584246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef4031450 con 0x5647a1dc8070 2024-08-28T19:24:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.540+0000 7fdee8ff9640 1 -- 172.21.15.16:0/3381584246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdef4036030 con 0x5647a1dc8070 2024-08-28T19:24:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.540+0000 7fdee8ff9640 1 -- 172.21.15.16:0/3381584246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef4004460 con 0x5647a1dc8070 2024-08-28T19:24:25.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.541+0000 7fdf06e92740 1 -- 172.21.15.16:0/3381584246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1dc8070 msgr2=0x5647a216d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:25.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.541+0000 7fdefa59c640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdee40b4d50 0x7fdee40b7260 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x5647a216bf90 tx=0x7fdeec00a380 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.541+0000 7fdf06e92740 1 --2- 172.21.15.16:0/3381584246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1dc8070 0x5647a216d310 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fdef402f4c0 tx=0x7fdef4031ae0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.541+0000 7fdf06e92740 1 -- 172.21.15.16:0/3381584246 shutdown_connections 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.541+0000 7fdf06e92740 1 --2- 172.21.15.16:0/3381584246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1da7390 0x5647a216dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.541+0000 7fdf06e92740 1 --2- 172.21.15.16:0/3381584246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1d9f1b0 0x5647a216d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.541+0000 7fdf06e92740 1 --2- 172.21.15.16:0/3381584246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1dc8070 0x5647a216d310 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.541+0000 7fdf06e92740 1 -- 172.21.15.16:0/3381584246 >> 172.21.15.16:0/3381584246 conn(0x5647a1dbc460 msgr2=0x5647a216b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.541+0000 7fdf06e92740 1 -- 172.21.15.16:0/3381584246 shutdown_connections 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.541+0000 7fdf06e92740 1 -- 172.21.15.16:0/3381584246 wait complete. 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.541+0000 7fdf06e92740 1 -- start start 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdf06e92740 1 Processor -- start 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1d9f1b0 0x5647a2178aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a2178fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1dc8070 0x5647a2184c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdf06e92740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5647a1f36700 con 0x5647a1da7390 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdf06e92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5647a205a750 con 0x5647a1d9f1b0 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdf06e92740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5647a217b910 con 0x5647a1dc8070 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdf06e92740 10 client.0 _ll_register_callbacks cb 0x5647a2089e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdef9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a2178fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdef9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a2178fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45972/0 (socket says 172.21.15.16:45972) 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdef9d9b640 1 -- 172.21.15.16:0/3138896403 learned_addr learned my addr 172.21.15.16:0/3138896403 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdefa59c640 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1d9f1b0 0x5647a2178aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdefad9d640 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1dc8070 0x5647a2184c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdef9d9b640 1 -- 172.21.15.16:0/3138896403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1dc8070 msgr2=0x5647a2184c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdef9d9b640 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1dc8070 0x5647a2184c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdef9d9b640 1 -- 172.21.15.16:0/3138896403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1d9f1b0 msgr2=0x5647a2178aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdef9d9b640 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1d9f1b0 0x5647a2178aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdef9d9b640 1 -- 172.21.15.16:0/3138896403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647a2093900 con 0x5647a1da7390 2024-08-28T19:24:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdef9d9b640 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a2178fe0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fdef0010550 tx=0x7fdef00aefa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67ad58f1accb0b4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.542+0000 7fdee1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdee1ffb640 1 -- 172.21.15.16:0/3138896403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0010b10 con 0x5647a1da7390 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdee1ffb640 1 -- 172.21.15.16:0/3138896403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdef00b4de0 con 0x5647a1da7390 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdf06e92740 1 -- 172.21.15.16:0/3138896403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdef4036030 con 0x5647a1da7390 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdee1ffb640 1 -- 172.21.15.16:0/3138896403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdef00aece0 con 0x5647a1da7390 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdf06e92740 10 client.16539 fetch_fsmap learned FSMap version 33 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdf06e92740 1 -- 172.21.15.16:0/3138896403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5647a2187700 con 0x5647a1da7390 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdee1ffb640 1 -- 172.21.15.16:0/3138896403 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdef4036030 con 0x5647a1da7390 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdee1ffb640 1 -- 172.21.15.16:0/3138896403 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fdef0016070 con 0x5647a1da7390 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdf06e92740 10 client.16539 fetch_fsmap finished waiting for FSMap version 33 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdf06e92740 10 client.16539 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdf06e92740 20 client.16539 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdf06e92740 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5647a1d869f0 0x5647a218abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdf06e92740 4 client.16539 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:24:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.543+0000 7fdf06e92740 1 -- 172.21.15.16:0/3138896403 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647a20e65d0 con 0x5647a1d869f0 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.544+0000 7fdefa59c640 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5647a1d869f0 0x5647a218abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.544+0000 7fdee1ffb640 10 client.16539 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.544+0000 7fdefa59c640 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5647a1d869f0 0x5647a218abd0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=49efb90e8aff3c21 server_cookie=f94ab6404adf81fe in_seq=0 out_seq=0 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdee1ffb640 1 -- 172.21.15.16:0/3138896403 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5647a20e65d0 con 0x5647a1d869f0 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdee1ffb640 10 client.16539 handle_command_reply: tid=0 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 1 client.16539 shutdown 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 1 -- 172.21.15.16:0/3138896403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 msgr2=0x5647a2178fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a2178fe0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fdef0010550 tx=0x7fdef00aefa0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 1 -- 172.21.15.16:0/3138896403 shutdown_connections 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5647a1d869f0 0x5647a218abd0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1dc8070 0x5647a2184c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a2178fe0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 1 --2- 172.21.15.16:0/3138896403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1d9f1b0 0x5647a2178aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 1 -- 172.21.15.16:0/3138896403 >> 172.21.15.16:0/3138896403 conn(0x5647a1dbc460 msgr2=0x5647a21704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdee1ffb640 0 client.16539 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:24:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 1 -- 172.21.15.16:0/3138896403 shutdown_connections 2024-08-28T19:24:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 1 -- 172.21.15.16:0/3138896403 wait complete. 2024-08-28T19:24:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.545+0000 7fdf06e92740 20 client.16539 trim_cache size 0 max 16384 2024-08-28T19:24:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.558+0000 7fdf06e92740 1 Processor -- start 2024-08-28T19:24:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.558+0000 7fdf06e92740 1 -- start start 2024-08-28T19:24:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1d9f1b0 0x5647a2175c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1da7390 0x5647a21761d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1dc8070 0x5647a2190360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdf06e92740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1ae990 con 0x5647a1d9f1b0 2024-08-28T19:24:25.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdf06e92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1b05f0 con 0x5647a1dc8070 2024-08-28T19:24:25.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdf06e92740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1af060 con 0x5647a1da7390 2024-08-28T19:24:25.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdefa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1d9f1b0 0x5647a2175c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdefa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1d9f1b0 0x5647a2175c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45982/0 (socket says 172.21.15.16:45982) 2024-08-28T19:24:25.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdefa59c640 1 -- 172.21.15.16:0/1482974205 learned_addr learned my addr 172.21.15.16:0/1482974205 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:25.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdef9d9b640 1 --2- 172.21.15.16:0/1482974205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1da7390 0x5647a21761d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdefad9d640 1 --2- 172.21.15.16:0/1482974205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1dc8070 0x5647a2190360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdefa59c640 1 -- 172.21.15.16:0/1482974205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1da7390 msgr2=0x5647a21761d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdefa59c640 1 --2- 172.21.15.16:0/1482974205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1da7390 0x5647a21761d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdefa59c640 1 -- 172.21.15.16:0/1482974205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1dc8070 msgr2=0x5647a2190360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdefa59c640 1 --2- 172.21.15.16:0/1482974205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1dc8070 0x5647a2190360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdefa59c640 1 -- 172.21.15.16:0/1482974205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647a216f8f0 con 0x5647a1d9f1b0 2024-08-28T19:24:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.559+0000 7fdefa59c640 1 --2- 172.21.15.16:0/1482974205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1d9f1b0 0x5647a2175c90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00acd0 tx=0x7fdeec019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7407f74fa05f0b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.560+0000 7fdee3fff640 1 -- 172.21.15.16:0/1482974205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeec003af0 con 0x5647a1d9f1b0 2024-08-28T19:24:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.560+0000 7fdee3fff640 1 -- 172.21.15.16:0/1482974205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdeec0054d0 con 0x5647a1d9f1b0 2024-08-28T19:24:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.560+0000 7fdee3fff640 1 -- 172.21.15.16:0/1482974205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeec004c60 con 0x5647a1d9f1b0 2024-08-28T19:24:25.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.560+0000 7fdf06e92740 1 -- 172.21.15.16:0/1482974205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1d9f1b0 msgr2=0x5647a2175c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:25.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.560+0000 7fdf06e92740 1 --2- 172.21.15.16:0/1482974205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1d9f1b0 0x5647a2175c90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00acd0 tx=0x7fdeec019000 comp rx=0 tx=0).stop 2024-08-28T19:24:25.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.560+0000 7fdf06e92740 1 -- 172.21.15.16:0/1482974205 shutdown_connections 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.560+0000 7fdf06e92740 1 --2- 172.21.15.16:0/1482974205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1dc8070 0x5647a2190360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.560+0000 7fdf06e92740 1 --2- 172.21.15.16:0/1482974205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1da7390 0x5647a21761d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.560+0000 7fdf06e92740 1 --2- 172.21.15.16:0/1482974205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1d9f1b0 0x5647a2175c90 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.560+0000 7fdf06e92740 1 -- 172.21.15.16:0/1482974205 >> 172.21.15.16:0/1482974205 conn(0x5647a1dbc460 msgr2=0x5647a2177320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdf06e92740 1 -- 172.21.15.16:0/1482974205 shutdown_connections 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdf06e92740 1 -- 172.21.15.16:0/1482974205 wait complete. 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdf06e92740 1 -- start start 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdf06e92740 1 Processor -- start 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1d9f1b0 0x5647a2181190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a217d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdf06e92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1dc8070 0x5647a217dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdf06e92740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1afe60 con 0x5647a1da7390 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdf06e92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1af790 con 0x5647a1dc8070 2024-08-28T19:24:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdf06e92740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1ac970 con 0x5647a1d9f1b0 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdf06e92740 10 client.0 _ll_register_callbacks cb 0x7fdef00ec420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.561+0000 7fdef9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a217d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdefa59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1d9f1b0 0x5647a2181190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdef9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a217d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45990/0 (socket says 172.21.15.16:45990) 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdef9d9b640 1 -- 172.21.15.16:0/1916879518 learned_addr learned my addr 172.21.15.16:0/1916879518 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdefad9d640 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1dc8070 0x5647a217dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdef9d9b640 1 -- 172.21.15.16:0/1916879518 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1d9f1b0 msgr2=0x5647a2181190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdef9d9b640 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1d9f1b0 0x5647a2181190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdef9d9b640 1 -- 172.21.15.16:0/1916879518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1dc8070 msgr2=0x5647a217dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdef9d9b640 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1dc8070 0x5647a217dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdef9d9b640 1 -- 172.21.15.16:0/1916879518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647a217e1d0 con 0x5647a1da7390 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdefad9d640 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1dc8070 0x5647a217dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdef9d9b640 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a217d6a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fdef00ab250 tx=0x7fdef00ae220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=273cad37642e2417 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdee1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdee1ffb640 1 -- 172.21.15.16:0/1916879518 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0010b10 con 0x5647a1da7390 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdee1ffb640 1 -- 172.21.15.16:0/1916879518 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdef009a3b0 con 0x5647a1da7390 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdf06e92740 1 -- 172.21.15.16:0/1916879518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdeec0054d0 con 0x5647a1da7390 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdee1ffb640 1 -- 172.21.15.16:0/1916879518 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdef00aeb20 con 0x5647a1da7390 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdf06e92740 10 client.16545 fetch_fsmap learned FSMap version 33 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.562+0000 7fdf06e92740 1 -- 172.21.15.16:0/1916879518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5647a21f62f0 con 0x5647a1da7390 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.563+0000 7fdee1ffb640 1 -- 172.21.15.16:0/1916879518 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdeec0054d0 con 0x5647a1da7390 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.563+0000 7fdee1ffb640 1 -- 172.21.15.16:0/1916879518 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fdef0016070 con 0x5647a1da7390 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.563+0000 7fdf06e92740 10 client.16545 fetch_fsmap finished waiting for FSMap version 33 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.563+0000 7fdf06e92740 10 client.16545 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.563+0000 7fdf06e92740 20 client.16545 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.563+0000 7fdf06e92740 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5647a1d5bda0 0x5647a21f8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.563+0000 7fdf06e92740 4 client.16545 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.563+0000 7fdf06e92740 1 -- 172.21.15.16:0/1916879518 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5647a20e2070 con 0x5647a1d5bda0 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.563+0000 7fdefa59c640 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5647a1d5bda0 0x5647a21f8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdee1ffb640 10 client.16545 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdefa59c640 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5647a1d5bda0 0x5647a21f8a20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c75daef5919f9074 server_cookie=40731e5f89d2eed0 in_seq=0 out_seq=0 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdee1ffb640 1 -- 172.21.15.16:0/1916879518 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5647a20e2070 con 0x5647a1d5bda0 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdee1ffb640 10 client.16545 handle_command_reply: tid=0 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 1 client.16545 shutdown 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 1 -- 172.21.15.16:0/1916879518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 msgr2=0x5647a217d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a217d6a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fdef00ab250 tx=0x7fdef00ae220 comp rx=0 tx=0).stop 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 1 -- 172.21.15.16:0/1916879518 shutdown_connections 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5647a1d5bda0 0x5647a21f8a20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647a1dc8070 0x5647a217dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a1da7390 0x5647a217d6a0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdee1ffb640 0 client.16545 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:24:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 1 --2- 172.21.15.16:0/1916879518 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5647a1d9f1b0 0x5647a2181190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 1 -- 172.21.15.16:0/1916879518 >> 172.21.15.16:0/1916879518 conn(0x5647a1dbc460 msgr2=0x5647a21772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 1 -- 172.21.15.16:0/1916879518 shutdown_connections 2024-08-28T19:24:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 1 -- 172.21.15.16:0/1916879518 wait complete. 2024-08-28T19:24:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.564+0000 7fdf06e92740 20 client.16545 trim_cache size 0 max 16384 2024-08-28T19:24:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 -- 172.21.15.16:0/2914946857 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdee40b4d50 msgr2=0x7fdee40b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdee40b4d50 0x7fdee40b7260 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x5647a216bf90 tx=0x7fdeec00a380 comp rx=0 tx=0).stop 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 -- 172.21.15.16:0/2914946857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc104fc0 msgr2=0x7fdefc19a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc104fc0 0x7fdefc19a8b0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fdefc075800 tx=0x7fdef0010050 comp rx=0 tx=0).stop 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 -- 172.21.15.16:0/2914946857 shutdown_connections 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdefc105910 0x7fdefc19fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc104fc0 0x7fdefc19a8b0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdee40b4d50 0x7fdee40b7260 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 --2- 172.21.15.16:0/2914946857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdefc102fb0 0x7fdefc19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 -- 172.21.15.16:0/2914946857 >> 172.21.15.16:0/2914946857 conn(0x7fdefc079c40 msgr2=0x7fdefc1085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 -- 172.21.15.16:0/2914946857 shutdown_connections 2024-08-28T19:24:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:25.565+0000 7fdf0091d640 1 -- 172.21.15.16:0/2914946857 wait complete. 2024-08-28T19:24:25.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134483 0.15 MB/sec execute 5 sec latency 4298.972 ms 2024-08-28T19:24:26.237 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:24:26.492 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:24:26.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:26 smithi016 ceph-mon[18556]: pgmap v441: 129 pgs: 129 active+clean; 78 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 105 KiB/s wr, 19 op/s 2024-08-28T19:24:26.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 135047 1.04 MB/sec execute 6 sec latency 4961.647 ms 2024-08-28T19:24:26.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.963+0000 7f281eefd640 1 -- 172.21.15.16:0/3913311963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f281810db90 msgr2=0x7f28181100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:26.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.963+0000 7f281eefd640 1 --2- 172.21.15.16:0/3913311963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f281810db90 0x7f28181100a0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2810009ca0 tx=0x7f281002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.963+0000 7f281eefd640 1 -- 172.21.15.16:0/3913311963 shutdown_connections 2024-08-28T19:24:26.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.963+0000 7f281eefd640 1 --2- 172.21.15.16:0/3913311963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f281810db90 0x7f28181100a0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.963+0000 7f281eefd640 1 --2- 172.21.15.16:0/3913311963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2818105010 0x7f281810d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.963+0000 7f281eefd640 1 --2- 172.21.15.16:0/3913311963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28181045e0 0x7f28181049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.963+0000 7f281eefd640 1 -- 172.21.15.16:0/3913311963 >> 172.21.15.16:0/3913311963 conn(0x7f28180794d0 msgr2=0x7f28180798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:26.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.963+0000 7f281eefd640 1 -- 172.21.15.16:0/3913311963 shutdown_connections 2024-08-28T19:24:26.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.963+0000 7f281eefd640 1 -- 172.21.15.16:0/3913311963 wait complete. 2024-08-28T19:24:26.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.964+0000 7f281eefd640 1 Processor -- start 2024-08-28T19:24:26.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.964+0000 7f281eefd640 1 -- start start 2024-08-28T19:24:26.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.964+0000 7f281eefd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28181045e0 0x7f281819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.964+0000 7f281eefd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2818105010 0x7f281819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.964+0000 7f281eefd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f281810db90 0x7f281819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.964+0000 7f281eefd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2818113830 con 0x7f281810db90 2024-08-28T19:24:26.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.964+0000 7f281eefd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28181136b0 con 0x7f2818105010 2024-08-28T19:24:26.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.964+0000 7f281eefd640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f28181139b0 con 0x7f28181045e0 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f281cc99640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28181045e0 0x7f281819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f2817fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2818105010 0x7f281819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f281cc99640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28181045e0 0x7f281819a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:42818/0 (socket says 172.21.15.16:42818) 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f281cc99640 1 -- 172.21.15.16:0/731506819 learned_addr learned my addr 172.21.15.16:0/731506819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f281d49a640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f281810db90 0x7f281819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f2817fff640 1 -- 172.21.15.16:0/731506819 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28181045e0 msgr2=0x7f281819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f2817fff640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28181045e0 0x7f281819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f2817fff640 1 -- 172.21.15.16:0/731506819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f281810db90 msgr2=0x7f281819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f2817fff640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f281810db90 0x7f281819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f2817fff640 1 -- 172.21.15.16:0/731506819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28181a01d0 con 0x7f2818105010 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f281cc99640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28181045e0 0x7f281819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f281d49a640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f281810db90 0x7f281819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:26.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f2817fff640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2818105010 0x7f281819a800 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f281806aad0 tx=0x7f280c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:26.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f2815ffb640 1 -- 172.21.15.16:0/731506819 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c00cc40 con 0x7f2818105010 2024-08-28T19:24:26.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f281eefd640 1 -- 172.21.15.16:0/731506819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28181a04c0 con 0x7f2818105010 2024-08-28T19:24:26.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f2815ffb640 1 -- 172.21.15.16:0/731506819 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f280c00cde0 con 0x7f2818105010 2024-08-28T19:24:26.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.965+0000 7f2815ffb640 1 -- 172.21.15.16:0/731506819 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c004a00 con 0x7f2818105010 2024-08-28T19:24:26.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.966+0000 7f281eefd640 1 -- 172.21.15.16:0/731506819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28181a6c20 con 0x7f2818105010 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.968+0000 7f2815ffb640 1 -- 172.21.15.16:0/731506819 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f280c004ba0 con 0x7f2818105010 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.968+0000 7f2825472740 1 Processor -- start 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.968+0000 7f2825472740 1 -- start start 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.968+0000 7f2825472740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3dda0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.968+0000 7f2825472740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 0x559f3dda0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.968+0000 7f2825472740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 0x559f3dda0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.968+0000 7f2825472740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559f3dd17680 con 0x559f3d9da390 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.968+0000 7f2825472740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559f3dd06c10 con 0x559f3d9fb070 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.968+0000 7f2825472740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559f3dd0abc0 con 0x559f3d9d21b0 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281d49a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3dda0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f2817fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 0x559f3dda0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281d49a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3dda0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:59006/0 (socket says 172.21.15.16:59006) 2024-08-28T19:24:26.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281d49a640 1 -- 172.21.15.16:0/2088084548 learned_addr learned my addr 172.21.15.16:0/2088084548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281cc99640 1 --2- 172.21.15.16:0/2088084548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 0x559f3dda0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281d49a640 1 -- 172.21.15.16:0/2088084548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 msgr2=0x559f3dda0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281d49a640 1 --2- 172.21.15.16:0/2088084548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 0x559f3dda0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281d49a640 1 -- 172.21.15.16:0/2088084548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 msgr2=0x559f3dda0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281d49a640 1 --2- 172.21.15.16:0/2088084548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 0x559f3dda0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281d49a640 1 -- 172.21.15.16:0/2088084548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f3da66650 con 0x559f3d9fb070 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f2815ffb640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28000b4ca0 0x7f28000b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281cc99640 1 --2- 172.21.15.16:0/2088084548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 0x559f3dda0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281cc99640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28000b4ca0 0x7f28000b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f2815ffb640 1 -- 172.21.15.16:0/731506819 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f280c0e5800 con 0x7f2818105010 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f281d49a640 1 --2- 172.21.15.16:0/2088084548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3dda0310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2810033040 tx=0x7f281002f310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecbd7e57e6715269 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.969+0000 7f27feffd640 1 -- 172.21.15.16:0/2088084548 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2810031650 con 0x559f3d9fb070 2024-08-28T19:24:26.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.970+0000 7f27feffd640 1 -- 172.21.15.16:0/2088084548 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2810036040 con 0x559f3d9fb070 2024-08-28T19:24:26.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.970+0000 7f27feffd640 1 -- 172.21.15.16:0/2088084548 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28100044c0 con 0x559f3d9fb070 2024-08-28T19:24:26.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.970+0000 7f2825472740 1 -- 172.21.15.16:0/2088084548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 msgr2=0x559f3dda0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:26.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.970+0000 7f2825472740 1 --2- 172.21.15.16:0/2088084548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3dda0310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2810033040 tx=0x7f281002f310 comp rx=0 tx=0).stop 2024-08-28T19:24:26.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.970+0000 7f281cc99640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28000b4ca0 0x7f28000b71b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x559f3dd9ef90 tx=0x7f2808005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:26.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.970+0000 7f2825472740 1 -- 172.21.15.16:0/2088084548 shutdown_connections 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.970+0000 7f2825472740 1 --2- 172.21.15.16:0/2088084548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 0x559f3dda0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.970+0000 7f2825472740 1 --2- 172.21.15.16:0/2088084548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 0x559f3dda0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.970+0000 7f2825472740 1 --2- 172.21.15.16:0/2088084548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3dda0310 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.970+0000 7f2825472740 1 -- 172.21.15.16:0/2088084548 >> 172.21.15.16:0/2088084548 conn(0x559f3d9ef460 msgr2=0x559f3dd9e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.971+0000 7f2825472740 1 -- 172.21.15.16:0/2088084548 shutdown_connections 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.971+0000 7f2825472740 1 -- 172.21.15.16:0/2088084548 wait complete. 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.971+0000 7f2825472740 1 -- start start 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.971+0000 7f2825472740 1 Processor -- start 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.971+0000 7f2825472740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 0x559f3ddafeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.971+0000 7f2825472740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 0x559f3ddb03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.971+0000 7f2825472740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3ddbbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2825472740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559f3db69700 con 0x559f3d9da390 2024-08-28T19:24:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2825472740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559f3dc8d750 con 0x559f3d9fb070 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2825472740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559f3ddb2c10 con 0x559f3d9d21b0 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2817fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 0x559f3ddb03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2825472740 10 client.0 _ll_register_callbacks cb 0x559f3dcbce10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2817fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 0x559f3ddb03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46018/0 (socket says 172.21.15.16:46018) 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f281cc99640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 0x559f3ddafeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2817fff640 1 -- 172.21.15.16:0/559545002 learned_addr learned my addr 172.21.15.16:0/559545002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f281d49a640 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3ddbbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2817fff640 1 -- 172.21.15.16:0/559545002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 msgr2=0x559f3ddafeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2817fff640 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 0x559f3ddafeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2817fff640 1 -- 172.21.15.16:0/559545002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 msgr2=0x559f3ddbbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2817fff640 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3ddbbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2817fff640 1 -- 172.21.15.16:0/559545002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f3dcc6900 con 0x559f3d9da390 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f281d49a640 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3ddbbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f281cc99640 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 0x559f3ddafeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2817fff640 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 0x559f3ddb03f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f280c00c6a0 tx=0x7f280c0096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e568f9bf083ae0c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f27f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f27f7fff640 1 -- 172.21.15.16:0/559545002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c0ab4d0 con 0x559f3d9da390 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f27f7fff640 1 -- 172.21.15.16:0/559545002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f280c0e53e0 con 0x559f3d9da390 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.972+0000 7f2825472740 1 -- 172.21.15.16:0/559545002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2810036040 con 0x559f3d9da390 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f27f7fff640 1 -- 172.21.15.16:0/559545002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f280c0e3070 con 0x559f3d9da390 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f2825472740 10 client.16557 fetch_fsmap learned FSMap version 34 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f2825472740 1 -- 172.21.15.16:0/559545002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559f3ddbea90 con 0x559f3d9da390 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f27f7fff640 1 -- 172.21.15.16:0/559545002 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2810036040 con 0x559f3d9da390 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f27f7fff640 1 -- 172.21.15.16:0/559545002 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 34) ==== 8398+0+0 (secure 0 0 0) 0x7f280c0ce040 con 0x559f3d9da390 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f2825472740 10 client.16557 fetch_fsmap finished waiting for FSMap version 34 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f2825472740 10 client.16557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f2825472740 20 client.16557 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f2825472740 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559f3d9b99f0 0x559f3ddc2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f2825472740 4 client.16557 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:24:26.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f2825472740 1 -- 172.21.15.16:0/559545002 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f3dd195d0 con 0x559f3d9b99f0 2024-08-28T19:24:26.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.973+0000 7f281cc99640 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559f3d9b99f0 0x559f3ddc2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.974+0000 7f27f7fff640 10 client.16557 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:26.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.974+0000 7f281cc99640 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559f3d9b99f0 0x559f3ddc2010 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7be2ff2bf8330e0a server_cookie=5ebcbb2df6580d2 in_seq=0 out_seq=0 2024-08-28T19:24:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f27f7fff640 1 -- 172.21.15.16:0/559545002 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559f3dd195d0 con 0x559f3d9b99f0 2024-08-28T19:24:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f27f7fff640 10 client.16557 handle_command_reply: tid=0 2024-08-28T19:24:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 1 client.16557 shutdown 2024-08-28T19:24:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 1 -- 172.21.15.16:0/559545002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 msgr2=0x559f3ddb03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 0x559f3ddb03f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f280c00c6a0 tx=0x7f280c0096e0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 1 -- 172.21.15.16:0/559545002 shutdown_connections 2024-08-28T19:24:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3ddbbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9da390 0x559f3ddb03f0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559f3d9b99f0 0x559f3ddc2010 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f27f7fff640 0 client.16557 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 1 --2- 172.21.15.16:0/559545002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9d21b0 0x559f3ddafeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 1 -- 172.21.15.16:0/559545002 >> 172.21.15.16:0/559545002 conn(0x559f3d9ef460 msgr2=0x559f3dd9c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 1 -- 172.21.15.16:0/559545002 shutdown_connections 2024-08-28T19:24:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 1 -- 172.21.15.16:0/559545002 wait complete. 2024-08-28T19:24:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.975+0000 7f2825472740 20 client.16557 trim_cache size 0 max 16384 2024-08-28T19:24:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f2825472740 1 Processor -- start 2024-08-28T19:24:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f2825472740 1 -- start start 2024-08-28T19:24:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f2825472740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9d21b0 0x559f3ddbbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f2825472740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9da390 0x559f3ddbc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f2825472740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3dd9db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f2825472740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28181ae8a0 con 0x559f3d9d21b0 2024-08-28T19:24:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f2825472740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28181b0530 con 0x559f3d9fb070 2024-08-28T19:24:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f2825472740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f28181aefa0 con 0x559f3d9da390 2024-08-28T19:24:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f281cc99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9d21b0 0x559f3ddbbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f2817fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9da390 0x559f3ddbc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f281cc99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9d21b0 0x559f3ddbbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46024/0 (socket says 172.21.15.16:46024) 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f281cc99640 1 -- 172.21.15.16:0/16363550 learned_addr learned my addr 172.21.15.16:0/16363550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f281d49a640 1 --2- 172.21.15.16:0/16363550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3dd9db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f281cc99640 1 -- 172.21.15.16:0/16363550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9da390 msgr2=0x559f3ddbc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f281cc99640 1 --2- 172.21.15.16:0/16363550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9da390 0x559f3ddbc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f281cc99640 1 -- 172.21.15.16:0/16363550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 msgr2=0x559f3dd9db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f281cc99640 1 --2- 172.21.15.16:0/16363550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3dd9db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f281cc99640 1 -- 172.21.15.16:0/16363550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f3ddae3b0 con 0x559f3d9d21b0 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.989+0000 7f281d49a640 1 --2- 172.21.15.16:0/16363550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3dd9db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.990+0000 7f281cc99640 1 --2- 172.21.15.16:0/16363550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9d21b0 0x559f3ddbbe40 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2808009620 tx=0x7f280801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82dc3f86082e4082 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.990+0000 7f27fe7fc640 1 -- 172.21.15.16:0/16363550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280800fc60 con 0x559f3d9d21b0 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.990+0000 7f27fe7fc640 1 -- 172.21.15.16:0/16363550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f280800fe00 con 0x559f3d9d21b0 2024-08-28T19:24:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.990+0000 7f27fe7fc640 1 -- 172.21.15.16:0/16363550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28080326b0 con 0x559f3d9d21b0 2024-08-28T19:24:26.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.991+0000 7f2825472740 1 -- 172.21.15.16:0/16363550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9d21b0 msgr2=0x559f3ddbbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:26.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.991+0000 7f2825472740 1 --2- 172.21.15.16:0/16363550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9d21b0 0x559f3ddbbe40 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2808009620 tx=0x7f280801b000 comp rx=0 tx=0).stop 2024-08-28T19:24:26.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.991+0000 7f2825472740 1 -- 172.21.15.16:0/16363550 shutdown_connections 2024-08-28T19:24:26.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.991+0000 7f2825472740 1 --2- 172.21.15.16:0/16363550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9fb070 0x559f3dd9db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.991+0000 7f2825472740 1 --2- 172.21.15.16:0/16363550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9da390 0x559f3ddbc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.991+0000 7f2825472740 1 --2- 172.21.15.16:0/16363550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9d21b0 0x559f3ddbbe40 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.991+0000 7f2825472740 1 -- 172.21.15.16:0/16363550 >> 172.21.15.16:0/16363550 conn(0x559f3d9ef460 msgr2=0x559f3ddae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.991+0000 7f2825472740 1 -- 172.21.15.16:0/16363550 shutdown_connections 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.991+0000 7f2825472740 1 -- 172.21.15.16:0/16363550 wait complete. 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.991+0000 7f2825472740 1 -- start start 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.991+0000 7f2825472740 1 Processor -- start 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f2825472740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9d21b0 0x559f3ddb4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f2825472740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9da390 0x559f3ddb4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f2825472740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9fb070 0x559f3ddb4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f2825472740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28181afda0 con 0x559f3d9fb070 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f2825472740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28181af6d0 con 0x559f3d9d21b0 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f2825472740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f28181ac880 con 0x559f3d9da390 2024-08-28T19:24:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f2825472740 10 client.0 _ll_register_callbacks cb 0x7f280c0eb240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f281cc99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9d21b0 0x559f3ddb4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f281d49a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9fb070 0x559f3ddb4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f281cc99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9d21b0 0x559f3ddb4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:59032/0 (socket says 172.21.15.16:59032) 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f2817fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9da390 0x559f3ddb4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f281cc99640 1 -- 172.21.15.16:0/4110179160 learned_addr learned my addr 172.21.15.16:0/4110179160 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f281cc99640 1 -- 172.21.15.16:0/4110179160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9da390 msgr2=0x559f3ddb4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f281cc99640 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9da390 0x559f3ddb4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f281cc99640 1 -- 172.21.15.16:0/4110179160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9fb070 msgr2=0x559f3ddb4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f281cc99640 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9fb070 0x559f3ddb4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f281cc99640 1 -- 172.21.15.16:0/4110179160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f3de2bb30 con 0x559f3d9d21b0 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.992+0000 7f281d49a640 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9fb070 0x559f3ddb4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.993+0000 7f281cc99640 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9d21b0 0x559f3ddb4010 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2808021eb0 tx=0x7f2808021ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae0cea4b6ba31c0c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.993+0000 7f27f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:24:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.993+0000 7f27f7fff640 1 -- 172.21.15.16:0/4110179160 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280800fc60 con 0x559f3d9d21b0 2024-08-28T19:24:26.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.993+0000 7f2825472740 1 -- 172.21.15.16:0/4110179160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f280800fe00 con 0x559f3d9d21b0 2024-08-28T19:24:26.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.993+0000 7f27f7fff640 1 -- 172.21.15.16:0/4110179160 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f2808005250 con 0x559f3d9d21b0 2024-08-28T19:24:26.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.993+0000 7f27f7fff640 1 -- 172.21.15.16:0/4110179160 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2808045050 con 0x559f3d9d21b0 2024-08-28T19:24:26.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.993+0000 7f2825472740 10 client.26147 fetch_fsmap learned FSMap version 34 2024-08-28T19:24:26.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.993+0000 7f2825472740 1 -- 172.21.15.16:0/4110179160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559f3de2cd90 con 0x559f3d9d21b0 2024-08-28T19:24:26.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.993+0000 7f27f7fff640 1 -- 172.21.15.16:0/4110179160 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f280800fe00 con 0x559f3d9d21b0 2024-08-28T19:24:26.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.993+0000 7f27f7fff640 1 -- 172.21.15.16:0/4110179160 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8398+0+0 (secure 0 0 0) 0x7f280802b560 con 0x559f3d9d21b0 2024-08-28T19:24:26.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f2825472740 10 client.26147 fetch_fsmap finished waiting for FSMap version 34 2024-08-28T19:24:26.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f2825472740 10 client.26147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:24:26.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f2825472740 20 client.26147 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f2825472740 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559f3d98eda0 0x559f3de2f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f2825472740 4 client.26147 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f2825472740 1 -- 172.21.15.16:0/4110179160 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f3dd15070 con 0x559f3d98eda0 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f2817fff640 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559f3d98eda0 0x559f3de2f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f27f7fff640 10 client.26147 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f2817fff640 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559f3d98eda0 0x559f3de2f5b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1369619c896b2c4 server_cookie=9cc4f6d91b4313b9 in_seq=0 out_seq=0 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f27f7fff640 1 -- 172.21.15.16:0/4110179160 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f3dd15070 con 0x559f3d98eda0 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f27f7fff640 10 client.26147 handle_command_reply: tid=0 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.994+0000 7f2825472740 1 client.26147 shutdown 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f2825472740 1 -- 172.21.15.16:0/4110179160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9d21b0 msgr2=0x559f3ddb4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f2825472740 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9d21b0 0x559f3ddb4010 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2808021eb0 tx=0x7f2808021ee0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f2825472740 1 -- 172.21.15.16:0/4110179160 shutdown_connections 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f2825472740 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f3d9fb070 0x559f3ddb4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f2825472740 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559f3d98eda0 0x559f3de2f5b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f2825472740 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559f3d9da390 0x559f3ddb4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f2825472740 1 --2- 172.21.15.16:0/4110179160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f3d9d21b0 0x559f3ddb4010 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f27f7fff640 0 client.26147 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f2825472740 1 -- 172.21.15.16:0/4110179160 >> 172.21.15.16:0/4110179160 conn(0x559f3d9ef460 msgr2=0x559f3dd9db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f2825472740 1 -- 172.21.15.16:0/4110179160 shutdown_connections 2024-08-28T19:24:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f2825472740 1 -- 172.21.15.16:0/4110179160 wait complete. 2024-08-28T19:24:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f2825472740 20 client.26147 trim_cache size 0 max 16384 2024-08-28T19:24:27.000 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:24:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f281eefd640 1 -- 172.21.15.16:0/731506819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28000b4ca0 msgr2=0x7f28000b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f281eefd640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28000b4ca0 0x7f28000b71b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x559f3dd9ef90 tx=0x7f2808005f90 comp rx=0 tx=0).stop 2024-08-28T19:24:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f281eefd640 1 -- 172.21.15.16:0/731506819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2818105010 msgr2=0x7f281819a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.995+0000 7f281eefd640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2818105010 0x7f281819a800 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f281806aad0 tx=0x7f280c00ee50 comp rx=0 tx=0).stop 2024-08-28T19:24:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.996+0000 7f281eefd640 1 -- 172.21.15.16:0/731506819 shutdown_connections 2024-08-28T19:24:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.996+0000 7f281eefd640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f281810db90 0x7f281819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.996+0000 7f281eefd640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28000b4ca0 0x7f28000b71b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.996+0000 7f281eefd640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2818105010 0x7f281819a800 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.996+0000 7f281eefd640 1 --2- 172.21.15.16:0/731506819 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28181045e0 0x7f281819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.996+0000 7f281eefd640 1 -- 172.21.15.16:0/731506819 >> 172.21.15.16:0/731506819 conn(0x7f28180794d0 msgr2=0x7f2818107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:27.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.996+0000 7f281eefd640 1 -- 172.21.15.16:0/731506819 shutdown_connections 2024-08-28T19:24:27.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:26.996+0000 7f281eefd640 1 -- 172.21.15.16:0/731506819 wait complete. 2024-08-28T19:24:27.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:26 smithi028 ceph-mon[21769]: pgmap v441: 129 pgs: 129 active+clean; 78 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 105 KiB/s wr, 19 op/s 2024-08-28T19:24:27.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:26 smithi115 ceph-mon[22184]: pgmap v441: 129 pgs: 129 active+clean; 78 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 105 KiB/s wr, 19 op/s 2024-08-28T19:24:27.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 136512 1.69 MB/sec execute 7 sec latency 13.336 ms 2024-08-28T19:24:28.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:27 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:24:28.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:27 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:24:28.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:27 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:24:28.830 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:24:28.830 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:24:28.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 138761 3.21 MB/sec execute 8 sec latency 17.430 ms 2024-08-28T19:24:29.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:28 smithi016 ceph-mon[18556]: pgmap v442: 129 pgs: 129 active+clean; 78 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 110 KiB/s wr, 18 op/s 2024-08-28T19:24:29.058 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:28 smithi028 ceph-mon[21769]: pgmap v442: 129 pgs: 129 active+clean; 78 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 110 KiB/s wr, 18 op/s 2024-08-28T19:24:29.114 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:24:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:28 smithi115 ceph-mon[22184]: pgmap v442: 129 pgs: 129 active+clean; 78 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 110 KiB/s wr, 18 op/s 2024-08-28T19:24:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.815+0000 7f264cde7640 1 -- 172.21.15.16:0/1265421941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2648101b00 msgr2=0x7f2648101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.815+0000 7f264cde7640 1 --2- 172.21.15.16:0/1265421941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2648101b00 0x7f2648101f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2638009900 tx=0x7f263802a480 comp rx=0 tx=0).stop 2024-08-28T19:24:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.815+0000 7f264cde7640 1 -- 172.21.15.16:0/1265421941 shutdown_connections 2024-08-28T19:24:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.815+0000 7f264cde7640 1 --2- 172.21.15.16:0/1265421941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2648102660 0x7f26481122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.815+0000 7f264cde7640 1 --2- 172.21.15.16:0/1265421941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2648101b00 0x7f2648101f50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.815+0000 7f264cde7640 1 --2- 172.21.15.16:0/1265421941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2648104460 0x7f2648104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.815+0000 7f264cde7640 1 -- 172.21.15.16:0/1265421941 >> 172.21.15.16:0/1265421941 conn(0x7f2648077a60 msgr2=0x7f2648077ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.815+0000 7f264cde7640 1 -- 172.21.15.16:0/1265421941 shutdown_connections 2024-08-28T19:24:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.815+0000 7f264cde7640 1 -- 172.21.15.16:0/1265421941 wait complete. 2024-08-28T19:24:29.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.816+0000 7f264cde7640 1 Processor -- start 2024-08-28T19:24:29.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.817+0000 7f264cde7640 1 -- start start 2024-08-28T19:24:29.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.817+0000 7f264cde7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2648101b00 0x7f264819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.817+0000 7f264cde7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2648102660 0x7f264819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.817+0000 7f264cde7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2648104460 0x7f26481a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.817+0000 7f264cde7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2648115ab0 con 0x7f2648104460 2024-08-28T19:24:29.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.817+0000 7f264cde7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2648115930 con 0x7f2648101b00 2024-08-28T19:24:29.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.817+0000 7f264cde7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2648115c30 con 0x7f2648102660 2024-08-28T19:24:29.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.817+0000 7f264659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2648101b00 0x7f264819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.817+0000 7f264659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2648101b00 0x7f264819c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:59050/0 (socket says 172.21.15.16:59050) 2024-08-28T19:24:29.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.817+0000 7f264659c640 1 -- 172.21.15.16:0/4049974002 learned_addr learned my addr 172.21.15.16:0/4049974002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:29.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.817+0000 7f2646d9d640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2648104460 0x7f26481a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.818+0000 7f2645d9b640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2648102660 0x7f264819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.818+0000 7f2646d9d640 1 -- 172.21.15.16:0/4049974002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2648102660 msgr2=0x7f264819ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.818+0000 7f2646d9d640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2648102660 0x7f264819ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.818+0000 7f2646d9d640 1 -- 172.21.15.16:0/4049974002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2648101b00 msgr2=0x7f264819c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.818+0000 7f2646d9d640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2648101b00 0x7f264819c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.818+0000 7f2646d9d640 1 -- 172.21.15.16:0/4049974002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26481a23d0 con 0x7f2648104460 2024-08-28T19:24:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.819+0000 7f2646d9d640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2648104460 0x7f26481a1ca0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f264000e940 tx=0x7f264000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:29.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.819+0000 7f26377fe640 1 -- 172.21.15.16:0/4049974002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264000ccc0 con 0x7f2648104460 2024-08-28T19:24:29.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.819+0000 7f26377fe640 1 -- 172.21.15.16:0/4049974002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2640005020 con 0x7f2648104460 2024-08-28T19:24:29.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.819+0000 7f26377fe640 1 -- 172.21.15.16:0/4049974002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640004a00 con 0x7f2648104460 2024-08-28T19:24:29.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.819+0000 7f264cde7640 1 -- 172.21.15.16:0/4049974002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26481a26c0 con 0x7f2648104460 2024-08-28T19:24:29.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.819+0000 7f264cde7640 1 -- 172.21.15.16:0/4049974002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26481a8e20 con 0x7f2648104460 2024-08-28T19:24:29.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.823+0000 7f265335c740 1 Processor -- start 2024-08-28T19:24:29.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.824+0000 7f26377fe640 1 -- 172.21.15.16:0/4049974002 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2640004ba0 con 0x7f2648104460 2024-08-28T19:24:29.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.824+0000 7f265335c740 1 -- start start 2024-08-28T19:24:29.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.825+0000 7f265335c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 0x5600c34c5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.825+0000 7f265335c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30f6490 0x5600c34c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.825+0000 7f265335c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34c6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.825+0000 7f265335c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5600c343b5f0 con 0x5600c30f6490 2024-08-28T19:24:29.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.825+0000 7f265335c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5600c342ab80 con 0x5600c311f350 2024-08-28T19:24:29.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.825+0000 7f265335c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5600c342eb30 con 0x5600c30fe670 2024-08-28T19:24:29.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.826+0000 7f264659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30f6490 0x5600c34c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.826+0000 7f2645d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 0x5600c34c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.826+0000 7f2645d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 0x5600c34c5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:59052/0 (socket says 172.21.15.16:59052) 2024-08-28T19:24:29.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.826+0000 7f264659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30f6490 0x5600c34c5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46056/0 (socket says 172.21.15.16:46056) 2024-08-28T19:24:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.826+0000 7f2645d9b640 1 -- 172.21.15.16:0/1399632296 learned_addr learned my addr 172.21.15.16:0/1399632296 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.826+0000 7f264659c640 1 -- 172.21.15.16:0/1399632296 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 msgr2=0x5600c34c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.826+0000 7f264659c640 1 --2- 172.21.15.16:0/1399632296 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.826+0000 7f264659c640 1 -- 172.21.15.16:0/1399632296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 msgr2=0x5600c34c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.827+0000 7f26377fe640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26300b4e70 0x7f26300b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.828+0000 7f264659c640 1 --2- 172.21.15.16:0/1399632296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 0x5600c34c5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.828+0000 7f264659c640 1 -- 172.21.15.16:0/1399632296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600c318a930 con 0x5600c30f6490 2024-08-28T19:24:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.828+0000 7f2645d9b640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26300b4e70 0x7f26300b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.828+0000 7f26377fe640 1 -- 172.21.15.16:0/4049974002 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f26400e5740 con 0x7f2648104460 2024-08-28T19:24:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.829+0000 7f264659c640 1 --2- 172.21.15.16:0/1399632296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30f6490 0x5600c34c5cc0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f263c009fc0 tx=0x7f263c011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95efab3efbeb228d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.830+0000 7f2634ff9640 1 -- 172.21.15.16:0/1399632296 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263c003b70 con 0x5600c30f6490 2024-08-28T19:24:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.830+0000 7f2634ff9640 1 -- 172.21.15.16:0/1399632296 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f263c016070 con 0x5600c30f6490 2024-08-28T19:24:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.830+0000 7f2634ff9640 1 -- 172.21.15.16:0/1399632296 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263c012950 con 0x5600c30f6490 2024-08-28T19:24:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.831+0000 7f265335c740 1 -- 172.21.15.16:0/1399632296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30f6490 msgr2=0x5600c34c5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.831+0000 7f265335c740 1 --2- 172.21.15.16:0/1399632296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30f6490 0x5600c34c5cc0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f263c009fc0 tx=0x7f263c011870 comp rx=0 tx=0).stop 2024-08-28T19:24:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.831+0000 7f265335c740 1 -- 172.21.15.16:0/1399632296 shutdown_connections 2024-08-28T19:24:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.831+0000 7f265335c740 1 --2- 172.21.15.16:0/1399632296 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.831+0000 7f265335c740 1 --2- 172.21.15.16:0/1399632296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30f6490 0x5600c34c5cc0 secure :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f263c009fc0 tx=0x7f263c011870 comp rx=0 tx=0).stop 2024-08-28T19:24:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.832+0000 7f265335c740 1 --2- 172.21.15.16:0/1399632296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 0x5600c34c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.832+0000 7f265335c740 1 -- 172.21.15.16:0/1399632296 >> 172.21.15.16:0/1399632296 conn(0x5600c3113740 msgr2=0x5600c34c3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.832+0000 7f2645d9b640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26300b4e70 0x7f26300b7380 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f26380096c0 tx=0x7f2638009260 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:29.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.832+0000 7f265335c740 1 -- 172.21.15.16:0/1399632296 shutdown_connections 2024-08-28T19:24:29.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.832+0000 7f265335c740 1 -- 172.21.15.16:0/1399632296 wait complete. 2024-08-28T19:24:29.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.833+0000 7f265335c740 1 -- start start 2024-08-28T19:24:29.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.833+0000 7f265335c740 1 Processor -- start 2024-08-28T19:24:29.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f265335c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f265335c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 0x5600c34d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f265335c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30ddcd0 0x5600c34e5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f265335c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5600c328d890 con 0x5600c30ddcd0 2024-08-28T19:24:29.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f265335c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5600c33b1790 con 0x5600c311f350 2024-08-28T19:24:29.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f265335c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5600c34d3fb0 con 0x5600c30fe670 2024-08-28T19:24:29.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f265335c740 10 client.0 _ll_register_callbacks cb 0x5600c3136ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:29.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f2645d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 0x5600c34d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f2646d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30ddcd0 0x5600c34e5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f2646d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30ddcd0 0x5600c34e5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46058/0 (socket says 172.21.15.16:46058) 2024-08-28T19:24:29.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f2646d9d640 1 -- 172.21.15.16:0/3140573447 learned_addr learned my addr 172.21.15.16:0/3140573447 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:29.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.834+0000 7f264659c640 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.835+0000 7f2646d9d640 1 -- 172.21.15.16:0/3140573447 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 msgr2=0x5600c34d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:29.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.835+0000 7f2646d9d640 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.835+0000 7f2646d9d640 1 -- 172.21.15.16:0/3140573447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 msgr2=0x5600c34d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:29.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.835+0000 7f2646d9d640 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 0x5600c34d17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.835+0000 7f2646d9d640 1 -- 172.21.15.16:0/3140573447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600c33e8f30 con 0x5600c30ddcd0 2024-08-28T19:24:29.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.835+0000 7f2645d9b640 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 0x5600c34d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:29.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.835+0000 7f2646d9d640 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30ddcd0 0x5600c34e5ab0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f26400ab9f0 tx=0x7f26400ab290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8610d2bf5cf23a1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:29.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.835+0000 7f262dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:29.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.835+0000 7f262dffb640 1 -- 172.21.15.16:0/3140573447 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264001c040 con 0x5600c30ddcd0 2024-08-28T19:24:29.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.836+0000 7f262dffb640 1 -- 172.21.15.16:0/3140573447 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f26400a8050 con 0x5600c30ddcd0 2024-08-28T19:24:29.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.836+0000 7f265335c740 1 -- 172.21.15.16:0/3140573447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f263c016070 con 0x5600c30ddcd0 2024-08-28T19:24:29.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.836+0000 7f262dffb640 1 -- 172.21.15.16:0/3140573447 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f26400b00d0 con 0x5600c30ddcd0 2024-08-28T19:24:29.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.836+0000 7f265335c740 10 client.16575 fetch_fsmap learned FSMap version 34 2024-08-28T19:24:29.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.836+0000 7f265335c740 1 -- 172.21.15.16:0/3140573447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5600c34d6960 con 0x5600c30ddcd0 2024-08-28T19:24:29.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.837+0000 7f262dffb640 1 -- 172.21.15.16:0/3140573447 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f263c016070 con 0x5600c30ddcd0 2024-08-28T19:24:29.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.837+0000 7f262dffb640 1 -- 172.21.15.16:0/3140573447 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 34) ==== 8398+0+0 (secure 0 0 0) 0x7f26400b48f0 con 0x5600c30ddcd0 2024-08-28T19:24:29.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.837+0000 7f265335c740 10 client.16575 fetch_fsmap finished waiting for FSMap version 34 2024-08-28T19:24:29.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.837+0000 7f265335c740 10 client.16575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:24:29.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.837+0000 7f265335c740 20 client.16575 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:29.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.837+0000 7f265335c740 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600c30b3080 0x5600c34d9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.837+0000 7f265335c740 4 client.16575 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:24:29.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.837+0000 7f265335c740 1 -- 172.21.15.16:0/3140573447 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600c343d540 con 0x5600c30b3080 2024-08-28T19:24:29.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.838+0000 7f264659c640 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600c30b3080 0x5600c34d9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.839+0000 7f264659c640 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600c30b3080 0x5600c34d9fd0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d0d40ecce0fc39d server_cookie=704fde83d5f5999b in_seq=0 out_seq=0 2024-08-28T19:24:29.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.839+0000 7f262dffb640 10 client.16575 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:29.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.840+0000 7f262dffb640 1 -- 172.21.15.16:0/3140573447 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5600c343d540 con 0x5600c30b3080 2024-08-28T19:24:29.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.840+0000 7f262dffb640 10 client.16575 handle_command_reply: tid=0 2024-08-28T19:24:29.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.840+0000 7f265335c740 1 client.16575 shutdown 2024-08-28T19:24:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.841+0000 7f265335c740 1 -- 172.21.15.16:0/3140573447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30ddcd0 msgr2=0x5600c34e5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.841+0000 7f265335c740 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30ddcd0 0x5600c34e5ab0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f26400ab9f0 tx=0x7f26400ab290 comp rx=0 tx=0).stop 2024-08-28T19:24:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.841+0000 7f265335c740 1 -- 172.21.15.16:0/3140573447 shutdown_connections 2024-08-28T19:24:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.841+0000 7f265335c740 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c30ddcd0 0x5600c34e5ab0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.841+0000 7f265335c740 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c311f350 0x5600c34d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.841+0000 7f265335c740 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600c30b3080 0x5600c34d9fd0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.841+0000 7f265335c740 1 --2- 172.21.15.16:0/3140573447 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.841+0000 7f262dffb640 0 client.16575 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.841+0000 7f265335c740 1 -- 172.21.15.16:0/3140573447 >> 172.21.15.16:0/3140573447 conn(0x5600c3113740 msgr2=0x5600c34c8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.842+0000 7f265335c740 1 -- 172.21.15.16:0/3140573447 shutdown_connections 2024-08-28T19:24:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.842+0000 7f265335c740 1 -- 172.21.15.16:0/3140573447 wait complete. 2024-08-28T19:24:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.842+0000 7f265335c740 20 client.16575 trim_cache size 0 max 16384 2024-08-28T19:24:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.865+0000 7f265335c740 1 Processor -- start 2024-08-28T19:24:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.865+0000 7f265335c740 1 -- start start 2024-08-28T19:24:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.865+0000 7f265335c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30ddcd0 0x5600c34e5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.865+0000 7f265335c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30fe670 0x5600c34c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.865+0000 7f265335c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 0x5600c34c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.865+0000 7f265335c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26481ae9c0 con 0x5600c311f350 2024-08-28T19:24:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.865+0000 7f265335c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26481b0650 con 0x5600c30fe670 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.865+0000 7f265335c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f26481af0c0 con 0x5600c30ddcd0 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f2646d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 0x5600c34c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f2645d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30fe670 0x5600c34c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f2646d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 0x5600c34c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46060/0 (socket says 172.21.15.16:46060) 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f2645d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30fe670 0x5600c34c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:59070/0 (socket says 172.21.15.16:59070) 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f2646d9d640 1 -- 172.21.15.16:0/2117560087 learned_addr learned my addr 172.21.15.16:0/2117560087 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f264659c640 1 --2- 172.21.15.16:0/2117560087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30ddcd0 0x5600c34e5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f2645d9b640 1 -- 172.21.15.16:0/2117560087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30ddcd0 msgr2=0x5600c34e5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f2645d9b640 1 --2- 172.21.15.16:0/2117560087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30ddcd0 0x5600c34e5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f2645d9b640 1 -- 172.21.15.16:0/2117560087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 msgr2=0x5600c34c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f2645d9b640 1 --2- 172.21.15.16:0/2117560087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 0x5600c34c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f2645d9b640 1 -- 172.21.15.16:0/2117560087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600c34cf570 con 0x5600c30fe670 2024-08-28T19:24:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f264659c640 1 --2- 172.21.15.16:0/2117560087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30ddcd0 0x5600c34e5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:24:29.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.866+0000 7f2645d9b640 1 --2- 172.21.15.16:0/2117560087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30fe670 0x5600c34c8880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2638005ab0 tx=0x7f263803f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c348ea78c312c922 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:29.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.867+0000 7f262ffff640 1 -- 172.21.15.16:0/2117560087 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2638035040 con 0x5600c30fe670 2024-08-28T19:24:29.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.867+0000 7f262ffff640 1 -- 172.21.15.16:0/2117560087 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2638041070 con 0x5600c30fe670 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.867+0000 7f262ffff640 1 -- 172.21.15.16:0/2117560087 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263803d450 con 0x5600c30fe670 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.867+0000 7f265335c740 1 -- 172.21.15.16:0/2117560087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30fe670 msgr2=0x5600c34c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.867+0000 7f265335c740 1 --2- 172.21.15.16:0/2117560087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30fe670 0x5600c34c8880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2638005ab0 tx=0x7f263803f000 comp rx=0 tx=0).stop 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.868+0000 7f265335c740 1 -- 172.21.15.16:0/2117560087 shutdown_connections 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.868+0000 7f265335c740 1 --2- 172.21.15.16:0/2117560087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 0x5600c34c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.868+0000 7f265335c740 1 --2- 172.21.15.16:0/2117560087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30fe670 0x5600c34c8880 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.868+0000 7f265335c740 1 --2- 172.21.15.16:0/2117560087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30ddcd0 0x5600c34e5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.868+0000 7f265335c740 1 -- 172.21.15.16:0/2117560087 >> 172.21.15.16:0/2117560087 conn(0x5600c3113740 msgr2=0x5600c34cc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.868+0000 7f265335c740 1 -- 172.21.15.16:0/2117560087 shutdown_connections 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.868+0000 7f265335c740 1 -- 172.21.15.16:0/2117560087 wait complete. 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.868+0000 7f265335c740 1 -- start start 2024-08-28T19:24:29.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.868+0000 7f265335c740 1 Processor -- start 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f265335c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30ddcd0 0x5600c34dd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f265335c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34dddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f265335c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 0x5600c355e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f264659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30ddcd0 0x5600c34dd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f264659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30ddcd0 0x5600c34dd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:59084/0 (socket says 172.21.15.16:59084) 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f2645d9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34dddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f2646d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 0x5600c355e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f2645d9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34dddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:42928/0 (socket says 172.21.15.16:42928) 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f264659c640 1 -- 172.21.15.16:0/3158728898 learned_addr learned my addr 172.21.15.16:0/3158728898 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f265335c740 1 -- 172.21.15.16:0/3158728898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26481afec0 con 0x5600c311f350 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f265335c740 1 -- 172.21.15.16:0/3158728898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26481af7f0 con 0x5600c30ddcd0 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f265335c740 1 -- 172.21.15.16:0/3158728898 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f26481ac9e0 con 0x5600c30fe670 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.869+0000 7f265335c740 10 client.0 _ll_register_callbacks cb 0x7f26400046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:29.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f264659c640 1 -- 172.21.15.16:0/3158728898 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 msgr2=0x5600c34dddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f264659c640 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34dddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f264659c640 1 -- 172.21.15.16:0/3158728898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 msgr2=0x5600c355e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f264659c640 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 0x5600c355e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f264659c640 1 -- 172.21.15.16:0/3158728898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600c34de690 con 0x5600c30ddcd0 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f2645d9b640 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34dddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f2646d9d640 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 0x5600c355e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f264659c640 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30ddcd0 0x5600c34dd8b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x5600c34c4430 tx=0x7f263c004ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fb462fa38cb908e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f262dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f262dffb640 1 -- 172.21.15.16:0/3158728898 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263c012710 con 0x5600c30ddcd0 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f262dffb640 1 -- 172.21.15.16:0/3158728898 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f263c0128b0 con 0x5600c30ddcd0 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f265335c740 1 -- 172.21.15.16:0/3158728898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2638041070 con 0x5600c30ddcd0 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.870+0000 7f262dffb640 1 -- 172.21.15.16:0/3158728898 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f263c037050 con 0x5600c30ddcd0 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.871+0000 7f265335c740 10 client.26165 fetch_fsmap learned FSMap version 34 2024-08-28T19:24:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.871+0000 7f265335c740 1 -- 172.21.15.16:0/3158728898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5600c355f870 con 0x5600c30ddcd0 2024-08-28T19:24:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.871+0000 7f262dffb640 1 -- 172.21.15.16:0/3158728898 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2638041070 con 0x5600c30ddcd0 2024-08-28T19:24:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.871+0000 7f262dffb640 1 -- 172.21.15.16:0/3158728898 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8398+0+0 (secure 0 0 0) 0x7f263c0056e0 con 0x5600c30ddcd0 2024-08-28T19:24:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.871+0000 7f265335c740 10 client.26165 fetch_fsmap finished waiting for FSMap version 34 2024-08-28T19:24:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.871+0000 7f265335c740 10 client.26165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:24:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.871+0000 7f265335c740 20 client.26165 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.871+0000 7f265335c740 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600c355fe10 0x5600c3562230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.871+0000 7f265335c740 4 client.26165 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:24:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.871+0000 7f265335c740 1 -- 172.21.15.16:0/3158728898 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5600c3438fe0 con 0x5600c355fe10 2024-08-28T19:24:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.871+0000 7f2645d9b640 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600c355fe10 0x5600c3562230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:29.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.872+0000 7f262dffb640 10 client.26165 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:29.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.872+0000 7f2645d9b640 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600c355fe10 0x5600c3562230 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc5979855b0fd331 server_cookie=74388ebe89667e06 in_seq=0 out_seq=0 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.874+0000 7f262dffb640 1 -- 172.21.15.16:0/3158728898 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5600c3438fe0 con 0x5600c355fe10 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.874+0000 7f262dffb640 10 client.26165 handle_command_reply: tid=0 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.874+0000 7f265335c740 1 client.26165 shutdown 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f265335c740 1 -- 172.21.15.16:0/3158728898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30ddcd0 msgr2=0x5600c34dd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f265335c740 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30ddcd0 0x5600c34dd8b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x5600c34c4430 tx=0x7f263c004ba0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f265335c740 1 -- 172.21.15.16:0/3158728898 shutdown_connections 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f265335c740 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600c311f350 0x5600c355e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f265335c740 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600c355fe10 0x5600c3562230 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f265335c740 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600c30fe670 0x5600c34dddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f262dffb640 0 client.26165 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f265335c740 1 --2- 172.21.15.16:0/3158728898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600c30ddcd0 0x5600c34dd8b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f265335c740 1 -- 172.21.15.16:0/3158728898 >> 172.21.15.16:0/3158728898 conn(0x5600c3113740 msgr2=0x5600c34c8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f265335c740 1 -- 172.21.15.16:0/3158728898 shutdown_connections 2024-08-28T19:24:29.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f265335c740 1 -- 172.21.15.16:0/3158728898 wait complete. 2024-08-28T19:24:29.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.875+0000 7f265335c740 20 client.26165 trim_cache size 0 max 16384 2024-08-28T19:24:29.879 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:24:29.879 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:24:29.879 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d", 2024-08-28T19:24:29.879 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:24:29.879 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:24:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 -- 172.21.15.16:0/4049974002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26300b4e70 msgr2=0x7f26300b7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26300b4e70 0x7f26300b7380 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f26380096c0 tx=0x7f2638009260 comp rx=0 tx=0).stop 2024-08-28T19:24:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 -- 172.21.15.16:0/4049974002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2648104460 msgr2=0x7f26481a1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2648104460 0x7f26481a1ca0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f264000e940 tx=0x7f264000ee00 comp rx=0 tx=0).stop 2024-08-28T19:24:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 -- 172.21.15.16:0/4049974002 shutdown_connections 2024-08-28T19:24:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2648104460 0x7f26481a1ca0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2648102660 0x7f264819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26300b4e70 0x7f26300b7380 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 --2- 172.21.15.16:0/4049974002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2648101b00 0x7f264819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 -- 172.21.15.16:0/4049974002 >> 172.21.15.16:0/4049974002 conn(0x7f2648077a60 msgr2=0x7f2648100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 -- 172.21.15.16:0/4049974002 shutdown_connections 2024-08-28T19:24:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:29.876+0000 7f264cde7640 1 -- 172.21.15.16:0/4049974002 wait complete. 2024-08-28T19:24:29.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 140702 3.61 MB/sec execute 9 sec latency 50.924 ms 2024-08-28T19:24:30.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:24:30.900 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:24:30.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 142377 4.48 MB/sec execute 10 sec latency 71.500 ms 2024-08-28T19:24:31.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: pgmap v443: 129 pgs: 129 active+clean; 81 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 362 KiB/s wr, 41 op/s 2024-08-28T19:24:31.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:24:31.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:24:31.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:24:31.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: pgmap v443: 129 pgs: 129 active+clean; 81 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 362 KiB/s wr, 41 op/s 2024-08-28T19:24:31.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:24:31.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:24:31.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:24:31.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: pgmap v443: 129 pgs: 129 active+clean; 81 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 362 KiB/s wr, 41 op/s 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:24:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:24:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:24:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:24:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.348+0000 7f46c1d22640 1 -- 172.21.15.16:0/1927422961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc102810 msgr2=0x7f46bc102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.348+0000 7f46c1d22640 1 --2- 172.21.15.16:0/1927422961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc102810 0x7f46bc102c20 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009900 tx=0x7f46ac02a410 comp rx=0 tx=0).stop 2024-08-28T19:24:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.348+0000 7f46c1d22640 1 -- 172.21.15.16:0/1927422961 shutdown_connections 2024-08-28T19:24:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.348+0000 7f46c1d22640 1 --2- 172.21.15.16:0/1927422961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46bc108aa0 0x7f46bc10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.348+0000 7f46c1d22640 1 --2- 172.21.15.16:0/1927422961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46bc108150 0x7f46bc108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.348+0000 7f46c1d22640 1 --2- 172.21.15.16:0/1927422961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc102810 0x7f46bc102c20 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.348+0000 7f46c1d22640 1 -- 172.21.15.16:0/1927422961 >> 172.21.15.16:0/1927422961 conn(0x7f46bc0794d0 msgr2=0x7f46bc0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.348+0000 7f46c1d22640 1 -- 172.21.15.16:0/1927422961 shutdown_connections 2024-08-28T19:24:31.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.348+0000 7f46c1d22640 1 -- 172.21.15.16:0/1927422961 wait complete. 2024-08-28T19:24:31.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.349+0000 7f46c1d22640 1 Processor -- start 2024-08-28T19:24:31.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.349+0000 7f46c1d22640 1 -- start start 2024-08-28T19:24:31.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46c1d22640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc102810 0x7f46bc19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46c1d22640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46bc108150 0x7f46bc19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46c1d22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46bc108aa0 0x7f46bc19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46c1d22640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc112720 con 0x7f46bc102810 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46c1d22640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc1125a0 con 0x7f46bc108aa0 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46c1d22640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc1128a0 con 0x7f46bc108150 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc102810 0x7f46bc19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc102810 0x7f46bc19a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46078/0 (socket says 172.21.15.16:46078) 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46bb7fe640 1 -- 172.21.15.16:0/350625474 learned_addr learned my addr 172.21.15.16:0/350625474 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46baffd640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46bc108150 0x7f46bc19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46bbfff640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46bc108aa0 0x7f46bc19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46bb7fe640 1 -- 172.21.15.16:0/350625474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46bc108150 msgr2=0x7f46bc19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46bb7fe640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46bc108150 0x7f46bc19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46bb7fe640 1 -- 172.21.15.16:0/350625474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46bc108aa0 msgr2=0x7f46bc19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46bb7fe640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46bc108aa0 0x7f46bc19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46bb7fe640 1 -- 172.21.15.16:0/350625474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46bc1a0190 con 0x7f46bc102810 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46bbfff640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46bc108aa0 0x7f46bc19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.350+0000 7f46baffd640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46bc108150 0x7f46bc19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.351+0000 7f46bb7fe640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc102810 0x7f46bc19a220 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f46ac005940 tx=0x7f46ac0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.351+0000 7f46b8ff9640 1 -- 172.21.15.16:0/350625474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ac03f030 con 0x7f46bc102810 2024-08-28T19:24:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.351+0000 7f46c1d22640 1 -- 172.21.15.16:0/350625474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46bc1a0420 con 0x7f46bc102810 2024-08-28T19:24:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.351+0000 7f46c1d22640 1 -- 172.21.15.16:0/350625474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46bc1a6be0 con 0x7f46bc102810 2024-08-28T19:24:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.352+0000 7f46b8ff9640 1 -- 172.21.15.16:0/350625474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46ac0044c0 con 0x7f46bc102810 2024-08-28T19:24:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.352+0000 7f46b8ff9640 1 -- 172.21.15.16:0/350625474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ac035070 con 0x7f46bc102810 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46c8297740 1 Processor -- start 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46c8297740 1 -- start start 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46c8297740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035cc070 0x55f0039712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46c8297740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f003971800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46c8297740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035ab390 0x55f003971d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46c8297740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f0038e8630 con 0x55f0035ab390 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46c8297740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f0038d7bc0 con 0x55f0035cc070 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46c8297740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f0038dbb70 con 0x55f0035a31b0 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46b8ff9640 1 -- 172.21.15.16:0/350625474 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f46ac042030 con 0x7f46bc102810 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035cc070 0x55f0039712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035cc070 0x55f0039712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:59118/0 (socket says 172.21.15.16:59118) 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46bbfff640 1 -- 172.21.15.16:0/33839623 learned_addr learned my addr 172.21.15.16:0/33839623 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46bb7fe640 1 --2- 172.21.15.16:0/33839623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f003971800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.354+0000 7f46baffd640 1 --2- 172.21.15.16:0/33839623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035ab390 0x55f003971d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46bbfff640 1 -- 172.21.15.16:0/33839623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 msgr2=0x55f003971800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46bbfff640 1 --2- 172.21.15.16:0/33839623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f003971800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46bbfff640 1 -- 172.21.15.16:0/33839623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035ab390 msgr2=0x55f003971d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46bbfff640 1 --2- 172.21.15.16:0/33839623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035ab390 0x55f003971d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46bbfff640 1 -- 172.21.15.16:0/33839623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f003637650 con 0x55f0035cc070 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46bb7fe640 1 --2- 172.21.15.16:0/33839623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f003971800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46bbfff640 1 --2- 172.21.15.16:0/33839623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035cc070 0x55f0039712c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f46b40117e0 tx=0x7f46b4011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ade74e37a1e8110 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46a1ffb640 1 -- 172.21.15.16:0/33839623 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4011310 con 0x55f0035cc070 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46a1ffb640 1 -- 172.21.15.16:0/33839623 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46b4003e20 con 0x55f0035cc070 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46a1ffb640 1 -- 172.21.15.16:0/33839623 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b40044f0 con 0x55f0035cc070 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46b8ff9640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f46a40b4d20 0x7f46a40b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.355+0000 7f46bb7fe640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f46a40b4d20 0x7f46a40b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.356+0000 7f46b8ff9640 1 -- 172.21.15.16:0/350625474 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f46ac1059d0 con 0x7f46bc102810 2024-08-28T19:24:31.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.356+0000 7f46c8297740 1 -- 172.21.15.16:0/33839623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035cc070 msgr2=0x55f0039712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:31.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.356+0000 7f46bb7fe640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f46a40b4d20 0x7f46a40b7230 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f46ac102e60 tx=0x7f46ac03b460 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:31.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.356+0000 7f46c8297740 1 --2- 172.21.15.16:0/33839623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035cc070 0x55f0039712c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f46b40117e0 tx=0x7f46b4011ca0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.356+0000 7f46c8297740 1 -- 172.21.15.16:0/33839623 shutdown_connections 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.356+0000 7f46c8297740 1 --2- 172.21.15.16:0/33839623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035ab390 0x55f003971d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.356+0000 7f46c8297740 1 --2- 172.21.15.16:0/33839623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f003971800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.356+0000 7f46c8297740 1 --2- 172.21.15.16:0/33839623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035cc070 0x55f0039712c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.356+0000 7f46c8297740 1 -- 172.21.15.16:0/33839623 >> 172.21.15.16:0/33839623 conn(0x55f0035c0460 msgr2=0x55f00396f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.356+0000 7f46c8297740 1 -- 172.21.15.16:0/33839623 shutdown_connections 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.356+0000 7f46c8297740 1 -- 172.21.15.16:0/33839623 wait complete. 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46c8297740 1 -- start start 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46c8297740 1 Processor -- start 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46c8297740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f00397ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46c8297740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f00397cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46c8297740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f003988df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46c8297740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f00373a700 con 0x55f0035cc070 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46c8297740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f00385e700 con 0x55f0035ab390 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46c8297740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f00397faa0 con 0x55f0035a31b0 2024-08-28T19:24:31.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46c8297740 10 client.0 _ll_register_callbacks cb 0x55f00388ddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46bbfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f00397ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46bbfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f00397ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:42966/0 (socket says 172.21.15.16:42966) 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46baffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f00397cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.357+0000 7f46bbfff640 1 -- 172.21.15.16:0/3615976426 learned_addr learned my addr 172.21.15.16:0/3615976426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f46bbfff640 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f003988df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f46baffd640 1 -- 172.21.15.16:0/3615976426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 msgr2=0x55f00397ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f46baffd640 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f00397ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f46baffd640 1 -- 172.21.15.16:0/3615976426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 msgr2=0x55f003988df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f46baffd640 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f003988df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f46baffd640 1 -- 172.21.15.16:0/3615976426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0038978b0 con 0x55f0035ab390 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f46baffd640 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f00397cfb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f46bc06b460 tx=0x7f46b000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7b3bbe7532100c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f469affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f469affd640 1 -- 172.21.15.16:0/3615976426 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b000fe60 con 0x55f0035ab390 2024-08-28T19:24:31.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f469affd640 1 -- 172.21.15.16:0/3615976426 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f46b0012110 con 0x55f0035ab390 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f46c8297740 1 -- 172.21.15.16:0/3615976426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46b4003e20 con 0x55f0035ab390 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f469affd640 1 -- 172.21.15.16:0/3615976426 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f46b0025050 con 0x55f0035ab390 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f46c8297740 10 client.26177 fetch_fsmap learned FSMap version 34 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.358+0000 7f46c8297740 1 -- 172.21.15.16:0/3615976426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55f00398b940 con 0x55f0035ab390 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.359+0000 7f469affd640 1 -- 172.21.15.16:0/3615976426 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46b4003e20 con 0x55f0035ab390 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.359+0000 7f469affd640 1 -- 172.21.15.16:0/3615976426 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8398+0+0 (secure 0 0 0) 0x7f46b0003bf0 con 0x55f0035ab390 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.359+0000 7f46c8297740 10 client.26177 fetch_fsmap finished waiting for FSMap version 34 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.359+0000 7f46c8297740 10 client.26177 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.359+0000 7f46c8297740 20 client.26177 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.359+0000 7f46c8297740 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f00358a9f0 0x55f00398eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.359+0000 7f46c8297740 4 client.26177 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.359+0000 7f46c8297740 1 -- 172.21.15.16:0/3615976426 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0038ea580 con 0x55f00358a9f0 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.359+0000 7f46bbfff640 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f00358a9f0 0x55f00398eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.359+0000 7f469affd640 10 client.26177 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:31.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.360+0000 7f46bbfff640 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f00358a9f0 0x55f00398eeb0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd3fdb48e1af9402 server_cookie=c7a2de710ceb7cee in_seq=0 out_seq=0 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.360+0000 7f469affd640 1 -- 172.21.15.16:0/3615976426 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f0038ea580 con 0x55f00358a9f0 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.360+0000 7f469affd640 10 client.26177 handle_command_reply: tid=0 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.360+0000 7f46c8297740 1 client.26177 shutdown 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f46c8297740 1 -- 172.21.15.16:0/3615976426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 msgr2=0x55f00397cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f46c8297740 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f00397cfb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f46bc06b460 tx=0x7f46b000f790 comp rx=0 tx=0).stop 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f46c8297740 1 -- 172.21.15.16:0/3615976426 shutdown_connections 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f46c8297740 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f003988df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f46c8297740 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f00397cfb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f46c8297740 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f00358a9f0 0x55f00398eeb0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f46c8297740 1 --2- 172.21.15.16:0/3615976426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f00397ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f46c8297740 1 -- 172.21.15.16:0/3615976426 >> 172.21.15.16:0/3615976426 conn(0x55f0035c0460 msgr2=0x55f003974450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f469affd640 0 client.26177 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f46c8297740 1 -- 172.21.15.16:0/3615976426 shutdown_connections 2024-08-28T19:24:31.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f46c8297740 1 -- 172.21.15.16:0/3615976426 wait complete. 2024-08-28T19:24:31.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.361+0000 7f46c8297740 20 client.26177 trim_cache size 0 max 16384 2024-08-28T19:24:31.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46c8297740 1 Processor -- start 2024-08-28T19:24:31.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46c8297740 1 -- start start 2024-08-28T19:24:31.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46c8297740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f003972a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46c8297740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f003988cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46c8297740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f003979c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46c8297740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46a800fec0 con 0x55f0035cc070 2024-08-28T19:24:31.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46c8297740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46a8011b50 con 0x55f0035ab390 2024-08-28T19:24:31.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46c8297740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f46a80105c0 con 0x55f0035a31b0 2024-08-28T19:24:31.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f003979c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f003979c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46092/0 (socket says 172.21.15.16:46092) 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46bbfff640 1 -- 172.21.15.16:0/275173132 learned_addr learned my addr 172.21.15.16:0/275173132 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46baffd640 1 --2- 172.21.15.16:0/275173132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f003988cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46bbfff640 1 --2- 172.21.15.16:0/275173132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f003972a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46bbfff640 1 -- 172.21.15.16:0/275173132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 msgr2=0x55f003972a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46bbfff640 1 --2- 172.21.15.16:0/275173132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f003972a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46bbfff640 1 -- 172.21.15.16:0/275173132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 msgr2=0x55f003988cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.380+0000 7f46bbfff640 1 --2- 172.21.15.16:0/275173132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f003988cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.381+0000 7f46bbfff640 1 -- 172.21.15.16:0/275173132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0039735b0 con 0x55f0035cc070 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.381+0000 7f46baffd640 1 --2- 172.21.15.16:0/275173132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f003988cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.381+0000 7f46bbfff640 1 --2- 172.21.15.16:0/275173132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f003979c60 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x55f00397f2c0 tx=0x7f46b4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487ef9706e59fcfb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.381+0000 7f46a17fa640 1 -- 172.21.15.16:0/275173132 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b400f9a0 con 0x55f0035cc070 2024-08-28T19:24:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.381+0000 7f46a17fa640 1 -- 172.21.15.16:0/275173132 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46b4005220 con 0x55f0035cc070 2024-08-28T19:24:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.381+0000 7f46a17fa640 1 -- 172.21.15.16:0/275173132 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b402b6b0 con 0x55f0035cc070 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.382+0000 7f46c8297740 1 -- 172.21.15.16:0/275173132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 msgr2=0x55f003979c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.382+0000 7f46c8297740 1 --2- 172.21.15.16:0/275173132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f003979c60 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x55f00397f2c0 tx=0x7f46b4009500 comp rx=0 tx=0).stop 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.382+0000 7f46c8297740 1 -- 172.21.15.16:0/275173132 shutdown_connections 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.382+0000 7f46c8297740 1 --2- 172.21.15.16:0/275173132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f003979c60 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.382+0000 7f46c8297740 1 --2- 172.21.15.16:0/275173132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f003988cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.382+0000 7f46c8297740 1 --2- 172.21.15.16:0/275173132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f003972a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.382+0000 7f46c8297740 1 -- 172.21.15.16:0/275173132 >> 172.21.15.16:0/275173132 conn(0x55f0035c0460 msgr2=0x55f0039722f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.382+0000 7f46c8297740 1 -- 172.21.15.16:0/275173132 shutdown_connections 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.382+0000 7f46c8297740 1 -- 172.21.15.16:0/275173132 wait complete. 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.382+0000 7f46c8297740 1 -- start start 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46c8297740 1 Processor -- start 2024-08-28T19:24:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46c8297740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f0039f2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46c8297740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f0039f3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46c8297740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f0039814d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46c8297740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46a80113c0 con 0x55f0035cc070 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46c8297740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46a8010cf0 con 0x55f0035ab390 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46c8297740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f46a800dea0 con 0x55f0035a31b0 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46c8297740 10 client.0 _ll_register_callbacks cb 0x7f46b0009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f0039814d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46baffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f0039f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f0039814d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46094/0 (socket says 172.21.15.16:46094) 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46bbfff640 1 -- 172.21.15.16:0/3190735287 learned_addr learned my addr 172.21.15.16:0/3190735287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46bbfff640 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f0039f2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46baffd640 1 -- 172.21.15.16:0/3190735287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 msgr2=0x55f0039f2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46baffd640 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f0039f2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46baffd640 1 -- 172.21.15.16:0/3190735287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 msgr2=0x55f0039814d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46baffd640 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f0039814d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46baffd640 1 -- 172.21.15.16:0/3190735287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f003981c40 con 0x55f0035ab390 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.383+0000 7f46bbfff640 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f0039814d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.384+0000 7f46baffd640 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f0039f3190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009710 tx=0x7f46b00129e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=633a6f8495f6b7d4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.384+0000 7f469affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.384+0000 7f469affd640 1 -- 172.21.15.16:0/3190735287 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0000e60 con 0x55f0035ab390 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.384+0000 7f46c8297740 1 -- 172.21.15.16:0/3190735287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46b4005220 con 0x55f0035ab390 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.384+0000 7f469affd640 1 -- 172.21.15.16:0/3190735287 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f46b0022030 con 0x55f0035ab390 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.384+0000 7f469affd640 1 -- 172.21.15.16:0/3190735287 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f46b0039050 con 0x55f0035ab390 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.384+0000 7f46c8297740 10 client.26183 fetch_fsmap learned FSMap version 34 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.384+0000 7f46c8297740 1 -- 172.21.15.16:0/3190735287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55f0039fa390 con 0x55f0035ab390 2024-08-28T19:24:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.384+0000 7f469affd640 1 -- 172.21.15.16:0/3190735287 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46b4005220 con 0x55f0035ab390 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.385+0000 7f469affd640 1 -- 172.21.15.16:0/3190735287 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8398+0+0 (secure 0 0 0) 0x7f46b00035f0 con 0x55f0035ab390 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.385+0000 7f46c8297740 10 client.26183 fetch_fsmap finished waiting for FSMap version 34 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.385+0000 7f46c8297740 10 client.26183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.385+0000 7f46c8297740 20 client.26183 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.385+0000 7f46c8297740 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f00355fda0 0x55f0039fcc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.385+0000 7f46c8297740 4 client.26183 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.385+0000 7f46c8297740 1 -- 172.21.15.16:0/3190735287 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f0038e6020 con 0x55f00355fda0 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.385+0000 7f46bbfff640 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f00355fda0 0x55f0039fcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.385+0000 7f469affd640 10 client.26183 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46bbfff640 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f00355fda0 0x55f0039fcc40 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65a47fb68ee8fec2 server_cookie=3b8794e796681a9 in_seq=0 out_seq=0 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f469affd640 1 -- 172.21.15.16:0/3190735287 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1990 (crc 0 0 0) 0x55f0038e6020 con 0x55f00355fda0 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f469affd640 10 client.26183 handle_command_reply: tid=0 2024-08-28T19:24:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 1 client.26183 shutdown 2024-08-28T19:24:31.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 1 -- 172.21.15.16:0/3190735287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 msgr2=0x55f0039f3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:31.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f0039f3190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009710 tx=0x7f46b00129e0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 1 -- 172.21.15.16:0/3190735287 shutdown_connections 2024-08-28T19:24:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0035cc070 0x55f0039814d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0035ab390 0x55f0039f3190 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f00355fda0 0x55f0039fcc40 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 1 --2- 172.21.15.16:0/3190735287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0035a31b0 0x55f0039f2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f469affd640 0 client.26183 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 1 -- 172.21.15.16:0/3190735287 >> 172.21.15.16:0/3190735287 conn(0x55f0035c0460 msgr2=0x55f00397deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 1 -- 172.21.15.16:0/3190735287 shutdown_connections 2024-08-28T19:24:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 1 -- 172.21.15.16:0/3190735287 wait complete. 2024-08-28T19:24:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.386+0000 7f46c8297740 20 client.26183 trim_cache size 0 max 16384 2024-08-28T19:24:31.392 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:24:31.392 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d", 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:24:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:24:31.394 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:24:31.395 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:24:31.396 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:24:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 -- 172.21.15.16:0/350625474 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f46a40b4d20 msgr2=0x7f46a40b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f46a40b4d20 0x7f46a40b7230 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f46ac102e60 tx=0x7f46ac03b460 comp rx=0 tx=0).stop 2024-08-28T19:24:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 -- 172.21.15.16:0/350625474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc102810 msgr2=0x7f46bc19a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc102810 0x7f46bc19a220 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f46ac005940 tx=0x7f46ac0040c0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 -- 172.21.15.16:0/350625474 shutdown_connections 2024-08-28T19:24:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f46a40b4d20 0x7f46a40b7230 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46bc108aa0 0x7f46bc19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46bc108150 0x7f46bc19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 --2- 172.21.15.16:0/350625474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc102810 0x7f46bc19a220 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:31.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 -- 172.21.15.16:0/350625474 >> 172.21.15.16:0/350625474 conn(0x7f46bc0794d0 msgr2=0x7f46bc101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:31.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 -- 172.21.15.16:0/350625474 shutdown_connections 2024-08-28T19:24:31.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:31.387+0000 7f46c1d22640 1 -- 172.21.15.16:0/350625474 wait complete. 2024-08-28T19:24:31.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 144678 5.07 MB/sec execute 11 sec latency 39.046 ms 2024-08-28T19:24:32.360 INFO:tasks.cephfs.filesystem:scrub status for tag:2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d - {'path': '/', 'tag': '2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d', 'options': 'recursive,force'} 2024-08-28T19:24:32.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 146201 5.35 MB/sec execute 12 sec latency 14.048 ms 2024-08-28T19:24:33.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:32 smithi016 ceph-mon[18556]: pgmap v444: 129 pgs: 129 active+clean; 84 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 947 KiB/s wr, 110 op/s 2024-08-28T19:24:33.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:32 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:24:33.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:32 smithi028 ceph-mon[21769]: pgmap v444: 129 pgs: 129 active+clean; 84 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 947 KiB/s wr, 110 op/s 2024-08-28T19:24:33.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:32 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:24:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:32 smithi115 ceph-mon[22184]: pgmap v444: 129 pgs: 129 active+clean; 84 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 947 KiB/s wr, 110 op/s 2024-08-28T19:24:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:32 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:24:33.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 148830 6.03 MB/sec execute 13 sec latency 18.527 ms 2024-08-28T19:24:34.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 150259 6.08 MB/sec execute 14 sec latency 14.187 ms 2024-08-28T19:24:35.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:34 smithi016 ceph-mon[18556]: pgmap v445: 129 pgs: 129 active+clean; 84 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-08-28T19:24:35.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:34 smithi028 ceph-mon[21769]: pgmap v445: 129 pgs: 129 active+clean; 84 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-08-28T19:24:35.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:34 smithi115 ceph-mon[22184]: pgmap v445: 129 pgs: 129 active+clean; 84 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-08-28T19:24:35.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 152835 6.79 MB/sec execute 15 sec latency 10.200 ms 2024-08-28T19:24:36.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 154878 6.81 MB/sec execute 16 sec latency 12.770 ms 2024-08-28T19:24:37.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:36 smithi016 ceph-mon[18556]: pgmap v446: 129 pgs: 129 active+clean; 86 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 196 op/s 2024-08-28T19:24:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:36 smithi028 ceph-mon[21769]: pgmap v446: 129 pgs: 129 active+clean; 86 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 196 op/s 2024-08-28T19:24:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:36 smithi115 ceph-mon[22184]: pgmap v446: 129 pgs: 129 active+clean; 86 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 196 op/s 2024-08-28T19:24:37.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 156517 7.13 MB/sec execute 17 sec latency 15.601 ms 2024-08-28T19:24:38.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 158584 7.17 MB/sec execute 18 sec latency 12.713 ms 2024-08-28T19:24:39.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:38 smithi016 ceph-mon[18556]: pgmap v447: 129 pgs: 129 active+clean; 88 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 210 op/s 2024-08-28T19:24:39.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:38 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:24:39.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:38 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:24:39.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:38 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:24:39.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:38 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:24:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:38 smithi028 ceph-mon[21769]: pgmap v447: 129 pgs: 129 active+clean; 88 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 210 op/s 2024-08-28T19:24:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:38 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:24:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:38 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:24:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:38 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:24:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:38 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:24:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:38 smithi115 ceph-mon[22184]: pgmap v447: 129 pgs: 129 active+clean; 88 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 210 op/s 2024-08-28T19:24:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:38 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:24:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:38 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:24:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:38 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:24:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:38 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:24:39.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 160112 7.39 MB/sec execute 19 sec latency 19.959 ms 2024-08-28T19:24:40.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:39 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:24:40.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:39 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:24:40.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:39 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:24:40.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:39 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:24:40.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:39 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:24:40.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:39 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:24:40.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:39 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:24:40.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:39 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:24:40.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:39 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:24:40.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:39 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:24:40.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:39 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:24:40.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:39 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:24:40.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:39 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:24:40.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:39 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:24:40.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:39 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:24:40.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 162561 7.64 MB/sec execute 20 sec latency 11.307 ms 2024-08-28T19:24:41.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:40 smithi016 ceph-mon[18556]: pgmap v448: 129 pgs: 129 active+clean; 89 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 231 op/s 2024-08-28T19:24:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:40 smithi028 ceph-mon[21769]: pgmap v448: 129 pgs: 129 active+clean; 89 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 231 op/s 2024-08-28T19:24:41.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:40 smithi115 ceph-mon[22184]: pgmap v448: 129 pgs: 129 active+clean; 89 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 231 op/s 2024-08-28T19:24:41.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 163905 7.61 MB/sec execute 21 sec latency 11.943 ms 2024-08-28T19:24:42.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 166520 7.90 MB/sec execute 22 sec latency 10.568 ms 2024-08-28T19:24:43.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:42 smithi016 ceph-mon[18556]: pgmap v449: 129 pgs: 129 active+clean; 94 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 297 op/s 2024-08-28T19:24:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:42 smithi028 ceph-mon[21769]: pgmap v449: 129 pgs: 129 active+clean; 94 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 297 op/s 2024-08-28T19:24:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:42 smithi115 ceph-mon[22184]: pgmap v449: 129 pgs: 129 active+clean; 94 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 297 op/s 2024-08-28T19:24:43.496 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.0 2024-08-28T19:24:43.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:24:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:24:43.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 167800 7.85 MB/sec execute 23 sec latency 12.324 ms 2024-08-28T19:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.268+0000 7fe89023e640 1 -- 172.21.15.16:0/2504471917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe888101bc0 msgr2=0x7fe888101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.268+0000 7fe89023e640 1 --2- 172.21.15.16:0/2504471917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe888101bc0 0x7fe888101fd0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe878009930 tx=0x7fe87802a420 comp rx=0 tx=0).stop 2024-08-28T19:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.268+0000 7fe89023e640 1 -- 172.21.15.16:0/2504471917 shutdown_connections 2024-08-28T19:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.268+0000 7fe89023e640 1 --2- 172.21.15.16:0/2504471917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe888102510 0x7fe888110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.268+0000 7fe89023e640 1 --2- 172.21.15.16:0/2504471917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe888101bc0 0x7fe888101fd0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.268+0000 7fe89023e640 1 --2- 172.21.15.16:0/2504471917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8881084d0 0x7fe8881088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.268+0000 7fe89023e640 1 -- 172.21.15.16:0/2504471917 >> 172.21.15.16:0/2504471917 conn(0x7fe888077a60 msgr2=0x7fe888077ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.268+0000 7fe89023e640 1 -- 172.21.15.16:0/2504471917 shutdown_connections 2024-08-28T19:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.268+0000 7fe89023e640 1 -- 172.21.15.16:0/2504471917 wait complete. 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe89023e640 1 Processor -- start 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe89023e640 1 -- start start 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe89023e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe888101bc0 0x7fe88819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe89023e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe888102510 0x7fe88819a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe89023e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8881084d0 0x7fe88819fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe89023e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe888113850 con 0x7fe888101bc0 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe89023e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881136d0 con 0x7fe8881084d0 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe89023e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881139d0 con 0x7fe888102510 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe88dfda640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe888101bc0 0x7fe88819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe88dfda640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe888101bc0 0x7fe88819a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56796/0 (socket says 172.21.15.16:56796) 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe88dfda640 1 -- 172.21.15.16:0/514463183 learned_addr learned my addr 172.21.15.16:0/514463183 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe88e7db640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8881084d0 0x7fe88819fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.269+0000 7fe88d7d9640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe888102510 0x7fe88819a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe88e7db640 1 -- 172.21.15.16:0/514463183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe888102510 msgr2=0x7fe88819a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe88e7db640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe888102510 0x7fe88819a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe88e7db640 1 -- 172.21.15.16:0/514463183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe888101bc0 msgr2=0x7fe88819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe88e7db640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe888101bc0 0x7fe88819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe88e7db640 1 -- 172.21.15.16:0/514463183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8881a01f0 con 0x7fe8881084d0 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe88dfda640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe888101bc0 0x7fe88819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe88e7db640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8881084d0 0x7fe88819fac0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe88000ec50 tx=0x7fe88000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe87effd640 1 -- 172.21.15.16:0/514463183 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880003b70 con 0x7fe8881084d0 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe89023e640 1 -- 172.21.15.16:0/514463183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8881a04e0 con 0x7fe8881084d0 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe87effd640 1 -- 172.21.15.16:0/514463183 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe880003d10 con 0x7fe8881084d0 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe87effd640 1 -- 172.21.15.16:0/514463183 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880010830 con 0x7fe8881084d0 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.270+0000 7fe89023e640 1 -- 172.21.15.16:0/514463183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8881a6c40 con 0x7fe8881084d0 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.271+0000 7fe89023e640 1 -- 172.21.15.16:0/514463183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe888101fd0 con 0x7fe8881084d0 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.273+0000 7fe87effd640 1 -- 172.21.15.16:0/514463183 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe880004b40 con 0x7fe8881084d0 2024-08-28T19:24:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.274+0000 7fe87effd640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8740b4ca0 0x7fe8740b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.274+0000 7fe87effd640 1 -- 172.21.15.16:0/514463183 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe8800e5e90 con 0x7fe8881084d0 2024-08-28T19:24:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.274+0000 7fe88dfda640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8740b4ca0 0x7fe8740b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.274+0000 7fe88dfda640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8740b4ca0 0x7fe8740b71b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe884011500 tx=0x7fe884005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.278+0000 7fe87effd640 1 -- 172.21.15.16:0/514463183 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fe8800ac9a0 con 0x7fe8881084d0 2024-08-28T19:24:44.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.505+0000 7fe89023e640 1 -- 172.21.15.16:0/514463183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe88819b3b0 con 0x7fe8881084d0 2024-08-28T19:24:44.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.506+0000 7fe87effd640 1 -- 172.21.15.16:0/514463183 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11990 (secure 0 0 0) 0x7fe88819b3b0 con 0x7fe8881084d0 2024-08-28T19:24:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:24:44.508 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"btime":"2024-08-28T19:24:31:632183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:24:31.584516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:24:44.508 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2024-08-28T19:24:44.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 -- 172.21.15.16:0/514463183 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8740b4ca0 msgr2=0x7fe8740b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:44.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8740b4ca0 0x7fe8740b71b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe884011500 tx=0x7fe884005f90 comp rx=0 tx=0).stop 2024-08-28T19:24:44.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 -- 172.21.15.16:0/514463183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8881084d0 msgr2=0x7fe88819fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:44.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8881084d0 0x7fe88819fac0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe88000ec50 tx=0x7fe88000c6a0 comp rx=0 tx=0).stop 2024-08-28T19:24:44.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 -- 172.21.15.16:0/514463183 shutdown_connections 2024-08-28T19:24:44.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8740b4ca0 0x7fe8740b71b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:44.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8881084d0 0x7fe88819fac0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:44.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe888102510 0x7fe88819a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:44.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 --2- 172.21.15.16:0/514463183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe888101bc0 0x7fe88819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:44.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 -- 172.21.15.16:0/514463183 >> 172.21.15.16:0/514463183 conn(0x7fe888077a60 msgr2=0x7fe8881050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:44.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 -- 172.21.15.16:0/514463183 shutdown_connections 2024-08-28T19:24:44.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:44.508+0000 7fe89023e640 1 -- 172.21.15.16:0/514463183 wait complete. 2024-08-28T19:24:44.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 170433 8.18 MB/sec execute 24 sec latency 38.047 ms 2024-08-28T19:24:45.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:44 smithi016 ceph-mon[18556]: pgmap v450: 129 pgs: 129 active+clean; 94 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 228 op/s 2024-08-28T19:24:45.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:44 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/514463183' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:24:45.221 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 147.0 --await-for 45.0' 2024-08-28T19:24:45.221 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 147.0 --await-for 45.0 2024-08-28T19:24:45.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:44 smithi028 ceph-mon[21769]: pgmap v450: 129 pgs: 129 active+clean; 94 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 228 op/s 2024-08-28T19:24:45.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:44 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/514463183' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:24:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:44 smithi115 ceph-mon[22184]: pgmap v450: 129 pgs: 129 active+clean; 94 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 228 op/s 2024-08-28T19:24:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:44 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/514463183' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:24:45.483 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:24:45.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 171832 8.11 MB/sec execute 25 sec latency 15.661 ms 2024-08-28T19:24:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.026+0000 7f33b2fdb640 1 -- 172.21.15.16:0/1707882431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac109280 msgr2=0x7f33ac109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.026+0000 7f33b2fdb640 1 --2- 172.21.15.16:0/1707882431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac109280 0x7f33ac109690 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3398009900 tx=0x7f339802a410 comp rx=0 tx=0).stop 2024-08-28T19:24:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.026+0000 7f33b2fdb640 1 -- 172.21.15.16:0/1707882431 shutdown_connections 2024-08-28T19:24:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.026+0000 7f33b2fdb640 1 --2- 172.21.15.16:0/1707882431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33ac109bd0 0x7f33ac110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.026+0000 7f33b2fdb640 1 --2- 172.21.15.16:0/1707882431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac109280 0x7f33ac109690 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.026+0000 7f33b2fdb640 1 --2- 172.21.15.16:0/1707882431 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33ac103940 0x7f33ac103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.026+0000 7f33b2fdb640 1 -- 172.21.15.16:0/1707882431 >> 172.21.15.16:0/1707882431 conn(0x7f33ac0ff3b0 msgr2=0x7f33ac101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.026+0000 7f33b2fdb640 1 -- 172.21.15.16:0/1707882431 shutdown_connections 2024-08-28T19:24:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.026+0000 7f33b2fdb640 1 -- 172.21.15.16:0/1707882431 wait complete. 2024-08-28T19:24:46.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b2fdb640 1 Processor -- start 2024-08-28T19:24:46.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b2fdb640 1 -- start start 2024-08-28T19:24:46.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b2fdb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33ac103940 0x7f33ac0754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b2fdb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33ac109280 0x7f33ac0759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b2fdb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac109bd0 0x7f33ac075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b2fdb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac113890 con 0x7f33ac109bd0 2024-08-28T19:24:46.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b2fdb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac113710 con 0x7f33ac109280 2024-08-28T19:24:46.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b2fdb640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac113a10 con 0x7f33ac103940 2024-08-28T19:24:46.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b0d77640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33ac103940 0x7f33ac0754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b1578640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac109bd0 0x7f33ac075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b0d77640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33ac103940 0x7f33ac0754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:37152/0 (socket says 172.21.15.16:37152) 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.027+0000 7f33b0d77640 1 -- 172.21.15.16:0/2128023985 learned_addr learned my addr 172.21.15.16:0/2128023985 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.028+0000 7f33abfff640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33ac109280 0x7f33ac0759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.028+0000 7f33b0d77640 1 -- 172.21.15.16:0/2128023985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33ac109280 msgr2=0x7f33ac0759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.028+0000 7f33b0d77640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33ac109280 0x7f33ac0759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.028+0000 7f33b0d77640 1 -- 172.21.15.16:0/2128023985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac109bd0 msgr2=0x7f33ac075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.028+0000 7f33b0d77640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac109bd0 0x7f33ac075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.028+0000 7f33b0d77640 1 -- 172.21.15.16:0/2128023985 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33ac1a8540 con 0x7f33ac103940 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.028+0000 7f33b1578640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac109bd0 0x7f33ac075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.028+0000 7f33b0d77640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33ac103940 0x7f33ac0754b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f33a000e9f0 tx=0x7f33a000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.028+0000 7f33a9ffb640 1 -- 172.21.15.16:0/2128023985 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a000cd80 con 0x7f33ac103940 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.028+0000 7f33b2fdb640 1 -- 172.21.15.16:0/2128023985 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33ac1a8830 con 0x7f33ac103940 2024-08-28T19:24:46.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.029+0000 7f33a9ffb640 1 -- 172.21.15.16:0/2128023985 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33a00054d0 con 0x7f33ac103940 2024-08-28T19:24:46.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.029+0000 7f33a9ffb640 1 -- 172.21.15.16:0/2128023985 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a0010430 con 0x7f33ac103940 2024-08-28T19:24:46.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.029+0000 7f33b2fdb640 1 -- 172.21.15.16:0/2128023985 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33ac1a8d40 con 0x7f33ac103940 2024-08-28T19:24:46.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.031+0000 7f33b9550740 1 Processor -- start 2024-08-28T19:24:46.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b9550740 1 -- start start 2024-08-28T19:24:46.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33a9ffb640 1 -- 172.21.15.16:0/2128023985 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f33a00046c0 con 0x7f33ac103940 2024-08-28T19:24:46.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b9550740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 0x5577b8401c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b9550740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 0x5577b84021d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b9550740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8039b60 0x5577b8402710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b9550740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5577b8376330 con 0x5577b8031980 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b9550740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5577b83658c0 con 0x5577b805a840 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b9550740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5577b8369870 con 0x5577b8039b60 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b1578640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 0x5577b8401c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b1578640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 0x5577b8401c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48756/0 (socket says 172.21.15.16:48756) 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33abfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8039b60 0x5577b8402710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b1578640 1 -- 172.21.15.16:0/2325075550 learned_addr learned my addr 172.21.15.16:0/2325075550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b0d77640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 0x5577b84021d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b1578640 1 -- 172.21.15.16:0/2325075550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8039b60 msgr2=0x5577b8402710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b1578640 1 --2- 172.21.15.16:0/2325075550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8039b60 0x5577b8402710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b1578640 1 -- 172.21.15.16:0/2325075550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 msgr2=0x5577b84021d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:46.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b1578640 1 --2- 172.21.15.16:0/2325075550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 0x5577b84021d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.032+0000 7f33b1578640 1 -- 172.21.15.16:0/2325075550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577b831ddf0 con 0x5577b805a840 2024-08-28T19:24:46.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.033+0000 7f33b1578640 1 --2- 172.21.15.16:0/2325075550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 0x5577b8401c90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f33a40117e0 tx=0x7f33a4011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bc2224eddaec326 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:46.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.033+0000 7f33a9ffb640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f339c0b4ca0 0x7f339c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.033+0000 7f33b0d77640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f339c0b4ca0 0x7f339c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.033+0000 7f3396ffd640 1 -- 172.21.15.16:0/2325075550 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4011310 con 0x5577b805a840 2024-08-28T19:24:46.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.033+0000 7f3396ffd640 1 -- 172.21.15.16:0/2325075550 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33a4003e20 con 0x5577b805a840 2024-08-28T19:24:46.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.033+0000 7f3396ffd640 1 -- 172.21.15.16:0/2325075550 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a40044f0 con 0x5577b805a840 2024-08-28T19:24:46.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.033+0000 7f33a9ffb640 1 -- 172.21.15.16:0/2128023985 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f33a00e5c90 con 0x7f33ac103940 2024-08-28T19:24:46.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b9550740 1 -- 172.21.15.16:0/2325075550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 msgr2=0x5577b8401c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:46.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b0d77640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f339c0b4ca0 0x7f339c0b71b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f33a00e3120 tx=0x7f33a00e97e0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:46.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b9550740 1 --2- 172.21.15.16:0/2325075550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 0x5577b8401c90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f33a40117e0 tx=0x7f33a4011ca0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b9550740 1 -- 172.21.15.16:0/2325075550 shutdown_connections 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b9550740 1 --2- 172.21.15.16:0/2325075550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8039b60 0x5577b8402710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b9550740 1 --2- 172.21.15.16:0/2325075550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 0x5577b84021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b9550740 1 --2- 172.21.15.16:0/2325075550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 0x5577b8401c90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b9550740 1 -- 172.21.15.16:0/2325075550 >> 172.21.15.16:0/2325075550 conn(0x5577b804e990 msgr2=0x5577b83ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b9550740 1 -- 172.21.15.16:0/2325075550 shutdown_connections 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b9550740 1 -- 172.21.15.16:0/2325075550 wait complete. 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b9550740 1 -- start start 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.034+0000 7f33b9550740 1 Processor -- start 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33b9550740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8031980 0x5577b840d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33b9550740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8039b60 0x5577b840d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33b9550740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b805a840 0x5577b8419760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33b9550740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5577b81c87a0 con 0x5577b805a840 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33b9550740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5577b82ec240 con 0x5577b8031980 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33b9550740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5577b8410470 con 0x5577b8039b60 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33b9550740 10 client.0 _ll_register_callbacks cb 0x5577b8072090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33b1578640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8031980 0x5577b840d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33b1578640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8031980 0x5577b840d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48772/0 (socket says 172.21.15.16:48772) 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33b1578640 1 -- 172.21.15.16:0/191853204 learned_addr learned my addr 172.21.15.16:0/191853204 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33abfff640 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8039b60 0x5577b840d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.035+0000 7f33b1578640 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b805a840 0x5577b8419760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f33abfff640 1 -- 172.21.15.16:0/191853204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8031980 msgr2=0x5577b840d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f33abfff640 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8031980 0x5577b840d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f33abfff640 1 -- 172.21.15.16:0/191853204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b805a840 msgr2=0x5577b8419760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f33abfff640 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b805a840 0x5577b8419760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f33abfff640 1 -- 172.21.15.16:0/191853204 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577b83222b0 con 0x5577b8039b60 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f33b1578640 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8031980 0x5577b840d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f33abfff640 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8039b60 0x5577b840d950 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f339802a930 tx=0x7f3398032e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcc0427fb19e2bed server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f338bfff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f338bfff640 1 -- 172.21.15.16:0/191853204 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3398035040 con 0x5577b8039b60 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f338bfff640 1 -- 172.21.15.16:0/191853204 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3398031760 con 0x5577b8039b60 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f33b9550740 1 -- 172.21.15.16:0/191853204 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33a4003e20 con 0x5577b8039b60 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f338bfff640 1 -- 172.21.15.16:0/191853204 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f339804f050 con 0x5577b8039b60 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f33b9550740 10 client.26044 fetch_fsmap learned FSMap version 35 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f33b9550740 1 -- 172.21.15.16:0/191853204 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5577b80d4b90 con 0x5577b8039b60 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.036+0000 7f338bfff640 1 -- 172.21.15.16:0/191853204 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33a4003e20 con 0x5577b8039b60 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.037+0000 7f338bfff640 1 -- 172.21.15.16:0/191853204 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f33980054b0 con 0x5577b8039b60 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.037+0000 7f33b9550740 10 client.26044 fetch_fsmap finished waiting for FSMap version 35 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.037+0000 7f33b9550740 10 client.26044 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:24:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.037+0000 7f33b9550740 20 client.26044 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:46.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.037+0000 7f33b9550740 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577b80191c0 0x5577b841f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.037+0000 7f33b9550740 4 client.26044 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:24:46.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.037+0000 7f33b9550740 1 -- 172.21.15.16:0/191853204 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577b8378280 con 0x5577b80191c0 2024-08-28T19:24:46.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.037+0000 7f33b1578640 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577b80191c0 0x5577b841f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.037+0000 7f338bfff640 10 client.26044 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:46.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.037+0000 7f33b1578640 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577b80191c0 0x5577b841f620 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd39fc309ed36c94 server_cookie=b32899b09064837b in_seq=0 out_seq=0 2024-08-28T19:24:46.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f338bfff640 1 -- 172.21.15.16:0/191853204 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5577b8378280 con 0x5577b80191c0 2024-08-28T19:24:46.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f338bfff640 10 client.26044 handle_command_reply: tid=0 2024-08-28T19:24:46.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 1 client.26044 shutdown 2024-08-28T19:24:46.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 1 -- 172.21.15.16:0/191853204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8039b60 msgr2=0x5577b840d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:46.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8039b60 0x5577b840d950 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f339802a930 tx=0x7f3398032e90 comp rx=0 tx=0).stop 2024-08-28T19:24:46.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 1 -- 172.21.15.16:0/191853204 shutdown_connections 2024-08-28T19:24:46.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b805a840 0x5577b8419760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577b80191c0 0x5577b841f620 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8039b60 0x5577b840d950 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 1 --2- 172.21.15.16:0/191853204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8031980 0x5577b840d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f338bfff640 0 client.26044 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 1 -- 172.21.15.16:0/191853204 >> 172.21.15.16:0/191853204 conn(0x5577b804e990 msgr2=0x5577b83fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 1 -- 172.21.15.16:0/191853204 shutdown_connections 2024-08-28T19:24:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 1 -- 172.21.15.16:0/191853204 wait complete. 2024-08-28T19:24:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.040+0000 7f33b9550740 20 client.26044 trim_cache size 0 max 16384 2024-08-28T19:24:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b9550740 1 Processor -- start 2024-08-28T19:24:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b9550740 1 -- start start 2024-08-28T19:24:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b9550740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 0x5577b840f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b9550740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8039b60 0x5577b84033d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b9550740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b805a840 0x5577b8419660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b9550740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac1b2df0 con 0x5577b8031980 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b9550740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac1b4ab0 con 0x5577b8039b60 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b9550740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac1b3520 con 0x5577b805a840 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b1578640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 0x5577b840f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b1578640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 0x5577b840f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56866/0 (socket says 172.21.15.16:56866) 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b1578640 1 -- 172.21.15.16:0/3101170248 learned_addr learned my addr 172.21.15.16:0/3101170248 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33abfff640 1 --2- 172.21.15.16:0/3101170248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8039b60 0x5577b84033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b1578640 1 --2- 172.21.15.16:0/3101170248 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b805a840 0x5577b8419660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b1578640 1 -- 172.21.15.16:0/3101170248 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b805a840 msgr2=0x5577b8419660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b1578640 1 --2- 172.21.15.16:0/3101170248 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b805a840 0x5577b8419660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b1578640 1 -- 172.21.15.16:0/3101170248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8039b60 msgr2=0x5577b84033d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b1578640 1 --2- 172.21.15.16:0/3101170248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8039b60 0x5577b84033d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.058+0000 7f33b1578640 1 -- 172.21.15.16:0/3101170248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577b8335d90 con 0x5577b8031980 2024-08-28T19:24:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.059+0000 7f33b1578640 1 --2- 172.21.15.16:0/3101170248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 0x5577b840f2f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009fd0 tx=0x7f33a4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946e046834112381 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.059+0000 7f33967fc640 1 -- 172.21.15.16:0/3101170248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a400fa20 con 0x5577b8031980 2024-08-28T19:24:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.059+0000 7f33967fc640 1 -- 172.21.15.16:0/3101170248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33a400fbc0 con 0x5577b8031980 2024-08-28T19:24:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.059+0000 7f33967fc640 1 -- 172.21.15.16:0/3101170248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a40059d0 con 0x5577b8031980 2024-08-28T19:24:46.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.060+0000 7f33b9550740 1 -- 172.21.15.16:0/3101170248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 msgr2=0x5577b840f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.060+0000 7f33b9550740 1 --2- 172.21.15.16:0/3101170248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 0x5577b840f2f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009fd0 tx=0x7f33a4009500 comp rx=0 tx=0).stop 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.060+0000 7f33b9550740 1 -- 172.21.15.16:0/3101170248 shutdown_connections 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.060+0000 7f33b9550740 1 --2- 172.21.15.16:0/3101170248 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b805a840 0x5577b8419660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.060+0000 7f33b9550740 1 --2- 172.21.15.16:0/3101170248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b8039b60 0x5577b84033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.060+0000 7f33b9550740 1 --2- 172.21.15.16:0/3101170248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8031980 0x5577b840f2f0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.060+0000 7f33b9550740 1 -- 172.21.15.16:0/3101170248 >> 172.21.15.16:0/3101170248 conn(0x5577b804e990 msgr2=0x5577b8402cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.060+0000 7f33b9550740 1 -- 172.21.15.16:0/3101170248 shutdown_connections 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.060+0000 7f33b9550740 1 -- 172.21.15.16:0/3101170248 wait complete. 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.060+0000 7f33b9550740 1 -- start start 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.060+0000 7f33b9550740 1 Processor -- start 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33b9550740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8031980 0x5577b84158d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33b9550740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8039b60 0x5577b8415e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33b9550740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 0x5577b8410680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33b9550740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac1b4320 con 0x5577b8039b60 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33b9550740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac1b3c50 con 0x5577b805a840 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33b9550740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac1b0dd0 con 0x5577b8031980 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33b9550740 10 client.0 _ll_register_callbacks cb 0x7f3398009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8039b60 0x5577b8415e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8039b60 0x5577b8415e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56878/0 (socket says 172.21.15.16:56878) 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33abfff640 1 -- 172.21.15.16:0/2956759683 learned_addr learned my addr 172.21.15.16:0/2956759683 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33b1578640 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8031980 0x5577b84158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33b1578640 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 0x5577b8410680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33abfff640 1 -- 172.21.15.16:0/2956759683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8031980 msgr2=0x5577b84158d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33abfff640 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8031980 0x5577b84158d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33abfff640 1 -- 172.21.15.16:0/2956759683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 msgr2=0x5577b8410680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:24:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33abfff640 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 0x5577b8410680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33abfff640 1 -- 172.21.15.16:0/2956759683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577b8410cf0 con 0x5577b8039b60 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f33abfff640 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8039b60 0x5577b8415e10 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3398000c00 tx=0x7f3398004c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7abb66fe5fe1c734 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f338bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.061+0000 7f338bfff640 1 -- 172.21.15.16:0/2956759683 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3398035050 con 0x5577b8039b60 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f338bfff640 1 -- 172.21.15.16:0/2956759683 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3398004250 con 0x5577b8039b60 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f33b9550740 1 -- 172.21.15.16:0/2956759683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33a400fbc0 con 0x5577b8039b60 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f338bfff640 1 -- 172.21.15.16:0/2956759683 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3398004450 con 0x5577b8039b60 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f33b9550740 10 client.16623 fetch_fsmap learned FSMap version 35 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f33b9550740 1 -- 172.21.15.16:0/2956759683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5577b848a8d0 con 0x5577b8039b60 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f338bfff640 1 -- 172.21.15.16:0/2956759683 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3398005320 con 0x5577b8039b60 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f338bfff640 1 -- 172.21.15.16:0/2956759683 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f339805b030 con 0x5577b8039b60 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f33b9550740 10 client.16623 fetch_fsmap finished waiting for FSMap version 35 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f33b9550740 10 client.16623 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:24:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f33b9550740 20 client.16623 populate_metadata read hostname 'smithi016' 2024-08-28T19:24:46.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f33b9550740 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577b7fee570 0x5577b848d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:24:46.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f33b9550740 4 client.16623 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.0, "await_for": 45.0}] 2024-08-28T19:24:46.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f33b9550740 1 -- 172.21.15.16:0/2956759683 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.0, "await_for": 45.0}) -- 0x7f33a400fbc0 con 0x5577b7fee570 2024-08-28T19:24:46.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.062+0000 7f33b1578640 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577b7fee570 0x5577b848d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:24:46.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.063+0000 7f338bfff640 10 client.16623 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:46.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.063+0000 7f33b1578640 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577b7fee570 0x5577b848d240 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2af2e6e6b65f617b server_cookie=fd7dbd2e9e2b92be in_seq=0 out_seq=0 2024-08-28T19:24:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.144+0000 7f338bfff640 1 -- 172.21.15.16:0/2956759683 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f33a400fbc0 con 0x5577b7fee570 2024-08-28T19:24:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.144+0000 7f338bfff640 10 client.16623 handle_command_reply: tid=0 2024-08-28T19:24:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 1 client.16623 shutdown 2024-08-28T19:24:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 1 -- 172.21.15.16:0/2956759683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8039b60 msgr2=0x5577b8415e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8039b60 0x5577b8415e10 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3398000c00 tx=0x7f3398004c20 comp rx=0 tx=0).stop 2024-08-28T19:24:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 1 -- 172.21.15.16:0/2956759683 shutdown_connections 2024-08-28T19:24:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b805a840 0x5577b8410680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577b8039b60 0x5577b8415e10 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577b7fee570 0x5577b848d240 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 1 --2- 172.21.15.16:0/2956759683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577b8031980 0x5577b84158d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f338bfff640 0 client.16623 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:24:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 1 -- 172.21.15.16:0/2956759683 >> 172.21.15.16:0/2956759683 conn(0x5577b804e990 msgr2=0x5577b840f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 1 -- 172.21.15.16:0/2956759683 shutdown_connections 2024-08-28T19:24:46.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 1 -- 172.21.15.16:0/2956759683 wait complete. 2024-08-28T19:24:46.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.145+0000 7f33b9550740 20 client.16623 trim_cache size 0 max 16384 2024-08-28T19:24:46.148 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:24:46.148 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 35, 2024-08-28T19:24:46.148 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:24:46.148 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 10, 2024-08-28T19:24:46.148 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:24:46.148 INFO:teuthology.orchestra.run.smithi016.stdout: "e6f999f4": { 2024-08-28T19:24:46.148 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 10, 2024-08-28T19:24:46.148 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.0, 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:24:46.149 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:24:46.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:24:46.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:24:46.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:24:46.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:24:46.150 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:24:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 -- 172.21.15.16:0/2128023985 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f339c0b4ca0 msgr2=0x7f339c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f339c0b4ca0 0x7f339c0b71b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f33a00e3120 tx=0x7f33a00e97e0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 -- 172.21.15.16:0/2128023985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33ac103940 msgr2=0x7f33ac0754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:24:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33ac103940 0x7f33ac0754b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f33a000e9f0 tx=0x7f33a000edf0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 -- 172.21.15.16:0/2128023985 shutdown_connections 2024-08-28T19:24:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac109bd0 0x7f33ac075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f339c0b4ca0 0x7f339c0b71b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33ac109280 0x7f33ac0759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 --2- 172.21.15.16:0/2128023985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33ac103940 0x7f33ac0754b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:24:46.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 -- 172.21.15.16:0/2128023985 >> 172.21.15.16:0/2128023985 conn(0x7f33ac0ff3b0 msgr2=0x7f33ac1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:24:46.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 -- 172.21.15.16:0/2128023985 shutdown_connections 2024-08-28T19:24:46.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:24:46.146+0000 7f33b2fdb640 1 -- 172.21.15.16:0/2128023985 wait complete. 2024-08-28T19:24:46.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 7.83 MB/sec execute 26 sec latency 854.532 ms 2024-08-28T19:24:47.041 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'e6f999f4', quiesce took 3.5 seconds. Will release after: 23.5 2024-08-28T19:24:47.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:46 smithi016 ceph-mon[18556]: pgmap v451: 129 pgs: 129 active+clean; 99 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 315 op/s 2024-08-28T19:24:47.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:46 smithi028 ceph-mon[21769]: pgmap v451: 129 pgs: 129 active+clean; 99 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 315 op/s 2024-08-28T19:24:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:46 smithi115 ceph-mon[22184]: pgmap v451: 129 pgs: 129 active+clean; 99 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 315 op/s 2024-08-28T19:24:47.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 7.54 MB/sec execute 27 sec latency 1854.601 ms 2024-08-28T19:24:48.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 7.27 MB/sec execute 28 sec latency 2854.705 ms 2024-08-28T19:24:49.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:48 smithi016 ceph-mon[18556]: pgmap v452: 129 pgs: 129 active+clean; 99 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 242 op/s 2024-08-28T19:24:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:48 smithi028 ceph-mon[21769]: pgmap v452: 129 pgs: 129 active+clean; 99 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 242 op/s 2024-08-28T19:24:49.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:48 smithi115 ceph-mon[22184]: pgmap v452: 129 pgs: 129 active+clean; 99 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 242 op/s 2024-08-28T19:24:49.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 7.02 MB/sec execute 29 sec latency 3854.809 ms 2024-08-28T19:24:50.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 6.79 MB/sec execute 30 sec latency 4854.858 ms 2024-08-28T19:24:51.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:50 smithi016 ceph-mon[18556]: pgmap v453: 129 pgs: 129 active+clean; 100 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 230 op/s 2024-08-28T19:24:51.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:50 smithi028 ceph-mon[21769]: pgmap v453: 129 pgs: 129 active+clean; 100 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 230 op/s 2024-08-28T19:24:51.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:50 smithi115 ceph-mon[22184]: pgmap v453: 129 pgs: 129 active+clean; 100 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 230 op/s 2024-08-28T19:24:51.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 6.57 MB/sec execute 31 sec latency 5854.968 ms 2024-08-28T19:24:52.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 6.36 MB/sec execute 32 sec latency 6855.054 ms 2024-08-28T19:24:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:52 smithi016 ceph-mon[18556]: pgmap v454: 129 pgs: 129 active+clean; 101 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-08-28T19:24:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:52 smithi028 ceph-mon[21769]: pgmap v454: 129 pgs: 129 active+clean; 101 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-08-28T19:24:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:52 smithi115 ceph-mon[22184]: pgmap v454: 129 pgs: 129 active+clean; 101 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-08-28T19:24:53.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 6.17 MB/sec execute 33 sec latency 7855.132 ms 2024-08-28T19:24:54.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 5.99 MB/sec execute 34 sec latency 8855.207 ms 2024-08-28T19:24:55.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:54 smithi115 ceph-mon[22184]: pgmap v455: 129 pgs: 129 active+clean; 101 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 883 KiB/s wr, 138 op/s 2024-08-28T19:24:55.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:54 smithi016 ceph-mon[18556]: pgmap v455: 129 pgs: 129 active+clean; 101 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 883 KiB/s wr, 138 op/s 2024-08-28T19:24:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:54 smithi028 ceph-mon[21769]: pgmap v455: 129 pgs: 129 active+clean; 101 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 883 KiB/s wr, 138 op/s 2024-08-28T19:24:55.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 5.82 MB/sec execute 35 sec latency 9855.299 ms 2024-08-28T19:24:56.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 5.65 MB/sec execute 36 sec latency 10855.394 ms 2024-08-28T19:24:57.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:56 smithi016 ceph-mon[18556]: pgmap v456: 129 pgs: 129 active+clean; 105 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-08-28T19:24:57.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:56 smithi028 ceph-mon[21769]: pgmap v456: 129 pgs: 129 active+clean; 105 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-08-28T19:24:57.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:56 smithi115 ceph-mon[22184]: pgmap v456: 129 pgs: 129 active+clean; 105 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-08-28T19:24:57.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 5.50 MB/sec execute 37 sec latency 11855.441 ms 2024-08-28T19:24:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 5.36 MB/sec execute 38 sec latency 12855.520 ms 2024-08-28T19:24:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:58 smithi016 ceph-mon[18556]: pgmap v457: 129 pgs: 129 active+clean; 115 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-08-28T19:24:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:58 smithi028 ceph-mon[21769]: pgmap v457: 129 pgs: 129 active+clean; 115 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-08-28T19:24:59.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:58 smithi115 ceph-mon[22184]: pgmap v457: 129 pgs: 129 active+clean; 115 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-08-28T19:24:59.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 5.22 MB/sec execute 39 sec latency 13855.633 ms 2024-08-28T19:25:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:24:59 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:24:59 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:00.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:24:59 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:00.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 5.09 MB/sec execute 40 sec latency 14855.740 ms 2024-08-28T19:25:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:00 smithi016 ceph-mon[18556]: pgmap v458: 129 pgs: 129 active+clean; 116 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 2.6 MiB/s wr, 100 op/s 2024-08-28T19:25:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:00 smithi028 ceph-mon[21769]: pgmap v458: 129 pgs: 129 active+clean; 116 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 2.6 MiB/s wr, 100 op/s 2024-08-28T19:25:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:00 smithi115 ceph-mon[22184]: pgmap v458: 129 pgs: 129 active+clean; 116 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 2.6 MiB/s wr, 100 op/s 2024-08-28T19:25:01.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 4.96 MB/sec execute 41 sec latency 15855.834 ms 2024-08-28T19:25:02.362 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:25:02.632 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:02.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 4.85 MB/sec execute 42 sec latency 16855.878 ms 2024-08-28T19:25:03.042 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:02 smithi016 ceph-mon[18556]: pgmap v459: 129 pgs: 129 active+clean; 102 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 2.7 MiB/s wr, 131 op/s 2024-08-28T19:25:03.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.134+0000 7fda40444640 1 -- 172.21.15.16:0/634890355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda3810dc30 msgr2=0x7fda38110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:03.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.134+0000 7fda40444640 1 --2- 172.21.15.16:0/634890355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda3810dc30 0x7fda38110120 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fda30009ca0 tx=0x7fda3002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.134+0000 7fda40444640 1 -- 172.21.15.16:0/634890355 shutdown_connections 2024-08-28T19:25:03.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.134+0000 7fda40444640 1 --2- 172.21.15.16:0/634890355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda3810dc30 0x7fda38110120 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.134+0000 7fda40444640 1 --2- 172.21.15.16:0/634890355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda38105120 0x7fda3810d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.134+0000 7fda40444640 1 --2- 172.21.15.16:0/634890355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda381047d0 0x7fda38104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.134+0000 7fda40444640 1 -- 172.21.15.16:0/634890355 >> 172.21.15.16:0/634890355 conn(0x7fda38079510 msgr2=0x7fda38079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.134+0000 7fda40444640 1 -- 172.21.15.16:0/634890355 shutdown_connections 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.134+0000 7fda40444640 1 -- 172.21.15.16:0/634890355 wait complete. 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda40444640 1 Processor -- start 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda40444640 1 -- start start 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda40444640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda381047d0 0x7fda3819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda40444640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda38105120 0x7fda3819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda40444640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda3810dc30 0x7fda3819fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda40444640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda381138e0 con 0x7fda381047d0 2024-08-28T19:25:03.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda40444640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda38113760 con 0x7fda38105120 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda40444640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fda38113a60 con 0x7fda3810dc30 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3e1e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda381047d0 0x7fda3819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3e1e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda381047d0 0x7fda3819a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49484/0 (socket says 172.21.15.16:49484) 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3e1e0640 1 -- 172.21.15.16:0/961742243 learned_addr learned my addr 172.21.15.16:0/961742243 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda3810dc30 0x7fda3819fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3d9df640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda38105120 0x7fda3819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3e1e0640 1 -- 172.21.15.16:0/961742243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda3810dc30 msgr2=0x7fda3819fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda3810dc30 0x7fda3819fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3e1e0640 1 -- 172.21.15.16:0/961742243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda38105120 msgr2=0x7fda3819a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda38105120 0x7fda3819a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3e1e0640 1 -- 172.21.15.16:0/961742243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda381a00e0 con 0x7fda381047d0 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda3810dc30 0x7fda3819fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.135+0000 7fda3d9df640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda38105120 0x7fda3819a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.136+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda381047d0 0x7fda3819a230 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fda3400c920 tx=0x7fda3400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.136+0000 7fda2f7fe640 1 -- 172.21.15.16:0/961742243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34007c70 con 0x7fda381047d0 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.136+0000 7fda2f7fe640 1 -- 172.21.15.16:0/961742243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda34007e10 con 0x7fda381047d0 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.136+0000 7fda40444640 1 -- 172.21.15.16:0/961742243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda381a03d0 con 0x7fda381047d0 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.136+0000 7fda2f7fe640 1 -- 172.21.15.16:0/961742243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34018820 con 0x7fda381047d0 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.136+0000 7fda40444640 1 -- 172.21.15.16:0/961742243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda381a6bf0 con 0x7fda381047d0 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.138+0000 7fda2f7fe640 1 -- 172.21.15.16:0/961742243 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fda3400f450 con 0x7fda381047d0 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.138+0000 7fda469b9740 1 Processor -- start 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda469b9740 1 -- start start 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda469b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79cef5070 0x55a79d29a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda469b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d29a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda469b9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 0x55a79d29ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda469b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a79d211630 con 0x55a79cecc1b0 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda469b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a79d200bc0 con 0x55a79cef5070 2024-08-28T19:25:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda469b9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a79d204b70 con 0x55a79ced4390 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda3e1e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d29a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda3e1e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d29a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49494/0 (socket says 172.21.15.16:49494) 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda3e1e0640 1 -- 172.21.15.16:0/3852301848 learned_addr learned my addr 172.21.15.16:0/3852301848 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda3d9df640 1 --2- 172.21.15.16:0/3852301848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 0x55a79d29ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/3852301848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79cef5070 0x55a79d29a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda2f7fe640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda240b4cd0 0x7fda240b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda3e1e0640 1 -- 172.21.15.16:0/3852301848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 msgr2=0x55a79d29ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/3852301848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 0x55a79d29ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda3e1e0640 1 -- 172.21.15.16:0/3852301848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79cef5070 msgr2=0x55a79d29a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/3852301848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79cef5070 0x55a79d29a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda3e1e0640 1 -- 172.21.15.16:0/3852301848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a79cf60650 con 0x55a79cecc1b0 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.139+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/3852301848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79cef5070 0x55a79d29a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.140+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda240b4cd0 0x7fda240b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.140+0000 7fda2f7fe640 1 -- 172.21.15.16:0/961742243 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fda340e5a70 con 0x7fda381047d0 2024-08-28T19:25:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.140+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/3852301848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d29a800 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fda340e2dc0 tx=0x7fda34020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2511e19d5d9f37b4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.140+0000 7fda2cd9a640 1 -- 172.21.15.16:0/3852301848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda340abbc0 con 0x55a79cecc1b0 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.140+0000 7fda2cd9a640 1 -- 172.21.15.16:0/3852301848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda3400e3b0 con 0x55a79cecc1b0 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.140+0000 7fda2cd9a640 1 -- 172.21.15.16:0/3852301848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3401b120 con 0x55a79cecc1b0 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.141+0000 7fda469b9740 1 -- 172.21.15.16:0/3852301848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 msgr2=0x55a79d29a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.141+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda240b4cd0 0x7fda240b71e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fda30009a60 tx=0x7fda300095c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.141+0000 7fda469b9740 1 --2- 172.21.15.16:0/3852301848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d29a800 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fda340e2dc0 tx=0x7fda34020000 comp rx=0 tx=0).stop 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.141+0000 7fda469b9740 1 -- 172.21.15.16:0/3852301848 shutdown_connections 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.141+0000 7fda469b9740 1 --2- 172.21.15.16:0/3852301848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 0x55a79d29ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.141+0000 7fda469b9740 1 --2- 172.21.15.16:0/3852301848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d29a800 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.141+0000 7fda469b9740 1 --2- 172.21.15.16:0/3852301848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79cef5070 0x55a79d29a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.141+0000 7fda469b9740 1 -- 172.21.15.16:0/3852301848 >> 172.21.15.16:0/3852301848 conn(0x55a79cee9460 msgr2=0x55a79d2981b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.141+0000 7fda469b9740 1 -- 172.21.15.16:0/3852301848 shutdown_connections 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.141+0000 7fda469b9740 1 -- 172.21.15.16:0/3852301848 wait complete. 2024-08-28T19:25:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.141+0000 7fda469b9740 1 -- start start 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda469b9740 1 Processor -- start 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda469b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79cecc1b0 0x55a79d2a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda469b9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 0x55a79d2aa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda469b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d2b6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda469b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a79d063700 con 0x55a79cef5070 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda469b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a79d187700 con 0x55a79cecc1b0 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda469b9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a79d2acd40 con 0x55a79ced4390 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda469b9740 10 client.0 _ll_register_callbacks cb 0x55a79d1b6dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda3e9e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d2b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda3d9df640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 0x55a79d2aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda3d9df640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 0x55a79d2aa480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:52504/0 (socket says 172.21.15.16:52504) 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda3d9df640 1 -- 172.21.15.16:0/2159765843 learned_addr learned my addr 172.21.15.16:0/2159765843 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda3e9e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d2b6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49506/0 (socket says 172.21.15.16:49506) 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.142+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79cecc1b0 0x55a79d2a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda3d9df640 1 -- 172.21.15.16:0/2159765843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79cecc1b0 msgr2=0x55a79d2a9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda3d9df640 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79cecc1b0 0x55a79d2a9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda3d9df640 1 -- 172.21.15.16:0/2159765843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 msgr2=0x55a79d2b6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda3d9df640 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d2b6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda3d9df640 1 -- 172.21.15.16:0/2159765843 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a79d1c08b0 con 0x55a79ced4390 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d2b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda3d9df640 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 0x55a79d2aa480 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda28009d40 tx=0x7fda280039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c60e7b35293f17f7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda21ffb640 1 -- 172.21.15.16:0/2159765843 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda28013930 con 0x55a79ced4390 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda469b9740 1 -- 172.21.15.16:0/2159765843 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda3400e3b0 con 0x55a79ced4390 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda21ffb640 1 -- 172.21.15.16:0/2159765843 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fda28013ad0 con 0x55a79ced4390 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda21ffb640 1 -- 172.21.15.16:0/2159765843 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fda28013cd0 con 0x55a79ced4390 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda469b9740 10 client.26059 fetch_fsmap learned FSMap version 36 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.143+0000 7fda469b9740 1 -- 172.21.15.16:0/2159765843 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a79d2b8c90 con 0x55a79ced4390 2024-08-28T19:25:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.144+0000 7fda21ffb640 1 -- 172.21.15.16:0/2159765843 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda3400e3b0 con 0x55a79ced4390 2024-08-28T19:25:03.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.144+0000 7fda21ffb640 1 -- 172.21.15.16:0/2159765843 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7fda2801a070 con 0x55a79ced4390 2024-08-28T19:25:03.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.144+0000 7fda469b9740 10 client.26059 fetch_fsmap finished waiting for FSMap version 36 2024-08-28T19:25:03.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.144+0000 7fda469b9740 10 client.26059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:25:03.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.144+0000 7fda469b9740 20 client.26059 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:03.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.144+0000 7fda469b9740 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a79ceb39f0 0x55a79d2bc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.144+0000 7fda469b9740 4 client.26059 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:03.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.144+0000 7fda469b9740 1 -- 172.21.15.16:0/2159765843 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a79d213580 con 0x55a79ceb39f0 2024-08-28T19:25:03.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.144+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a79ceb39f0 0x55a79d2bc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.145+0000 7fda21ffb640 10 client.26059 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:03.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.145+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a79ceb39f0 0x55a79d2bc240 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49eab4faf87613f9 server_cookie=8c345e36d35bbc42 in_seq=0 out_seq=0 2024-08-28T19:25:03.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.145+0000 7fda21ffb640 1 -- 172.21.15.16:0/2159765843 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a79d213580 con 0x55a79ceb39f0 2024-08-28T19:25:03.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.145+0000 7fda21ffb640 10 client.26059 handle_command_reply: tid=0 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.145+0000 7fda469b9740 1 client.26059 shutdown 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda469b9740 1 -- 172.21.15.16:0/2159765843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 msgr2=0x55a79d2aa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda469b9740 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 0x55a79d2aa480 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda28009d40 tx=0x7fda280039c0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda469b9740 1 -- 172.21.15.16:0/2159765843 shutdown_connections 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda469b9740 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d2b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda469b9740 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a79ceb39f0 0x55a79d2bc240 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda469b9740 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79ced4390 0x55a79d2aa480 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda469b9740 1 --2- 172.21.15.16:0/2159765843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79cecc1b0 0x55a79d2a9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda21ffb640 0 client.26059 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda469b9740 1 -- 172.21.15.16:0/2159765843 >> 172.21.15.16:0/2159765843 conn(0x55a79cee9460 msgr2=0x55a79d29bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda469b9740 1 -- 172.21.15.16:0/2159765843 shutdown_connections 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda469b9740 1 -- 172.21.15.16:0/2159765843 wait complete. 2024-08-28T19:25:03.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.146+0000 7fda469b9740 20 client.26059 trim_cache size 0 max 16384 2024-08-28T19:25:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.163+0000 7fda469b9740 1 Processor -- start 2024-08-28T19:25:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.163+0000 7fda469b9740 1 -- start start 2024-08-28T19:25:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.163+0000 7fda469b9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79cecc1b0 0x55a79d2b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.163+0000 7fda469b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 0x55a79d2b64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.163+0000 7fda469b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d297b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.163+0000 7fda469b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda381ae950 con 0x55a79cef5070 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.163+0000 7fda469b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda381b05e0 con 0x55a79ced4390 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.163+0000 7fda469b9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fda381af080 con 0x55a79cecc1b0 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3e9e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d297b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3e9e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d297b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49518/0 (socket says 172.21.15.16:49518) 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3e9e1640 1 -- 172.21.15.16:0/2956237365 learned_addr learned my addr 172.21.15.16:0/2956237365 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/2956237365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79cecc1b0 0x55a79d2b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3d9df640 1 --2- 172.21.15.16:0/2956237365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 0x55a79d2b64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3e9e1640 1 -- 172.21.15.16:0/2956237365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79cecc1b0 msgr2=0x55a79d2b5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/2956237365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79cecc1b0 0x55a79d2b5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3e9e1640 1 -- 172.21.15.16:0/2956237365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 msgr2=0x55a79d2b64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/2956237365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 0x55a79d2b64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3e9e1640 1 -- 172.21.15.16:0/2956237365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a79d298aa0 con 0x55a79cef5070 2024-08-28T19:25:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3d9df640 1 --2- 172.21.15.16:0/2956237365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 0x55a79d2b64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/2956237365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d297b10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fda30035000 tx=0x7fda30040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2260e2d987eef8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda23fff640 1 -- 172.21.15.16:0/2956237365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda30031d60 con 0x55a79cef5070 2024-08-28T19:25:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda23fff640 1 -- 172.21.15.16:0/2956237365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda30036040 con 0x55a79cef5070 2024-08-28T19:25:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.164+0000 7fda23fff640 1 -- 172.21.15.16:0/2956237365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda300042b0 con 0x55a79cef5070 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.165+0000 7fda469b9740 1 -- 172.21.15.16:0/2956237365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 msgr2=0x55a79d297b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.165+0000 7fda469b9740 1 --2- 172.21.15.16:0/2956237365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d297b10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fda30035000 tx=0x7fda30040000 comp rx=0 tx=0).stop 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.165+0000 7fda469b9740 1 -- 172.21.15.16:0/2956237365 shutdown_connections 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.165+0000 7fda469b9740 1 --2- 172.21.15.16:0/2956237365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cef5070 0x55a79d297b10 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.165+0000 7fda469b9740 1 --2- 172.21.15.16:0/2956237365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 0x55a79d2b64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.165+0000 7fda469b9740 1 --2- 172.21.15.16:0/2956237365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79cecc1b0 0x55a79d2b5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.165+0000 7fda469b9740 1 -- 172.21.15.16:0/2956237365 >> 172.21.15.16:0/2956237365 conn(0x55a79cee9460 msgr2=0x55a79d2a8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.165+0000 7fda469b9740 1 -- 172.21.15.16:0/2956237365 shutdown_connections 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.165+0000 7fda469b9740 1 -- 172.21.15.16:0/2956237365 wait complete. 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda469b9740 1 -- start start 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda469b9740 1 Processor -- start 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda469b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d2ae640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda469b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 0x55a79d2aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda469b9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79cef5070 0x55a79d325c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda469b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda381afe80 con 0x55a79cecc1b0 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda469b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda381af7b0 con 0x55a79ced4390 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda469b9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fda381ac930 con 0x55a79cef5070 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3e1e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d2ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3e1e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d2ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49520/0 (socket says 172.21.15.16:49520) 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3e1e0640 1 -- 172.21.15.16:0/235724615 learned_addr learned my addr 172.21.15.16:0/235724615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda469b9740 10 client.0 _ll_register_callbacks cb 0x7fda28004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79cef5070 0x55a79d325c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3d9df640 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 0x55a79d2aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3e1e0640 1 -- 172.21.15.16:0/235724615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79cef5070 msgr2=0x55a79d325c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79cef5070 0x55a79d325c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3e1e0640 1 -- 172.21.15.16:0/235724615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 msgr2=0x55a79d2aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:03.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 0x55a79d2aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3e1e0640 1 -- 172.21.15.16:0/235724615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a79d326140 con 0x55a79cecc1b0 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3e9e1640 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79cef5070 0x55a79d325c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.166+0000 7fda3d9df640 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 0x55a79d2aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda3e1e0640 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d2ae640 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fda3400c8f0 tx=0x7fda3400fe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec04f1a8c11ad870 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda21ffb640 1 -- 172.21.15.16:0/235724615 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda340af060 con 0x55a79cecc1b0 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda21ffb640 1 -- 172.21.15.16:0/235724615 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fda3400fa90 con 0x55a79cecc1b0 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda469b9740 1 -- 172.21.15.16:0/235724615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda30036040 con 0x55a79cecc1b0 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda21ffb640 1 -- 172.21.15.16:0/235724615 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fda340ab560 con 0x55a79cecc1b0 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda469b9740 10 client.16641 fetch_fsmap learned FSMap version 36 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda469b9740 1 -- 172.21.15.16:0/235724615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a79d327390 con 0x55a79cecc1b0 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda21ffb640 1 -- 172.21.15.16:0/235724615 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda30036040 con 0x55a79cecc1b0 2024-08-28T19:25:03.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda21ffb640 1 -- 172.21.15.16:0/235724615 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7fda34093110 con 0x55a79cecc1b0 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda469b9740 10 client.16641 fetch_fsmap finished waiting for FSMap version 36 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda469b9740 10 client.16641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda469b9740 20 client.16641 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda469b9740 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a79ce88da0 0x55a79d3296e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda469b9740 4 client.16641 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.167+0000 7fda469b9740 1 -- 172.21.15.16:0/235724615 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a79d2a0440 con 0x55a79ce88da0 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.168+0000 7fda3d9df640 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a79ce88da0 0x55a79d3296e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.168+0000 7fda3d9df640 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a79ce88da0 0x55a79d3296e0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=150fa86819b43d51 server_cookie=bd7524703584e98c in_seq=0 out_seq=0 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.168+0000 7fda21ffb640 10 client.16641 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.168+0000 7fda21ffb640 1 -- 172.21.15.16:0/235724615 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1794 (crc 0 0 0) 0x55a79d2a0440 con 0x55a79ce88da0 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.168+0000 7fda21ffb640 10 client.16641 handle_command_reply: tid=0 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.168+0000 7fda469b9740 1 client.16641 shutdown 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.168+0000 7fda469b9740 1 -- 172.21.15.16:0/235724615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 msgr2=0x55a79d2ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.168+0000 7fda469b9740 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d2ae640 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fda3400c8f0 tx=0x7fda3400fe60 comp rx=0 tx=0).stop 2024-08-28T19:25:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda469b9740 1 -- 172.21.15.16:0/235724615 shutdown_connections 2024-08-28T19:25:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda469b9740 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a79ce88da0 0x55a79d3296e0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda469b9740 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a79cef5070 0x55a79d325c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda469b9740 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a79ced4390 0x55a79d2aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda21ffb640 0 client.16641 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda469b9740 1 --2- 172.21.15.16:0/235724615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a79cecc1b0 0x55a79d2ae640 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda469b9740 1 -- 172.21.15.16:0/235724615 >> 172.21.15.16:0/235724615 conn(0x55a79cee9460 msgr2=0x55a79d2a9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda469b9740 1 -- 172.21.15.16:0/235724615 shutdown_connections 2024-08-28T19:25:03.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda469b9740 1 -- 172.21.15.16:0/235724615 wait complete. 2024-08-28T19:25:03.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda469b9740 20 client.16641 trim_cache size 0 max 16384 2024-08-28T19:25:03.176 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:25:03.176 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:25:03.176 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:25:03.176 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:25:03.176 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:25:03.176 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:25:03.176 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:03.176 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:25:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:03.178 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:03.179 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:03.180 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:03.180 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:03.180 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:03.180 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:03.180 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:03.180 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:25:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda40444640 1 -- 172.21.15.16:0/961742243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda240b4cd0 msgr2=0x7fda240b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda40444640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda240b4cd0 0x7fda240b71e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fda30009a60 tx=0x7fda300095c0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda40444640 1 -- 172.21.15.16:0/961742243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda381047d0 msgr2=0x7fda3819a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda40444640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda381047d0 0x7fda3819a230 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fda3400c920 tx=0x7fda3400cde0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda40444640 1 -- 172.21.15.16:0/961742243 shutdown_connections 2024-08-28T19:25:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda40444640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda3810dc30 0x7fda3819fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda40444640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda240b4cd0 0x7fda240b71e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda40444640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda38105120 0x7fda3819a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda40444640 1 --2- 172.21.15.16:0/961742243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda381047d0 0x7fda3819a230 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.169+0000 7fda40444640 1 -- 172.21.15.16:0/961742243 >> 172.21.15.16:0/961742243 conn(0x7fda38079510 msgr2=0x7fda380ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.170+0000 7fda40444640 1 -- 172.21.15.16:0/961742243 shutdown_connections 2024-08-28T19:25:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:03.170+0000 7fda40444640 1 -- 172.21.15.16:0/961742243 wait complete. 2024-08-28T19:25:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:02 smithi028 ceph-mon[21769]: pgmap v459: 129 pgs: 129 active+clean; 102 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 2.7 MiB/s wr, 131 op/s 2024-08-28T19:25:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:02 smithi115 ceph-mon[22184]: pgmap v459: 129 pgs: 129 active+clean; 102 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 2.7 MiB/s wr, 131 op/s 2024-08-28T19:25:03.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 4.73 MB/sec execute 43 sec latency 17855.921 ms 2024-08-28T19:25:04.035 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:25:04.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:25:04.288 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.768+0000 7fca4e88e640 1 -- 172.21.15.16:0/661892722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca48109240 msgr2=0x7fca48109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.768+0000 7fca4e88e640 1 --2- 172.21.15.16:0/661892722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca48109240 0x7fca48109650 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009900 tx=0x7fca3c02a3f0 comp rx=0 tx=0).stop 2024-08-28T19:25:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.768+0000 7fca4e88e640 1 -- 172.21.15.16:0/661892722 shutdown_connections 2024-08-28T19:25:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.768+0000 7fca4e88e640 1 --2- 172.21.15.16:0/661892722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca48109b90 0x7fca481103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.768+0000 7fca4e88e640 1 --2- 172.21.15.16:0/661892722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca48109240 0x7fca48109650 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.768+0000 7fca4e88e640 1 --2- 172.21.15.16:0/661892722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca48103900 0x7fca48103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.768+0000 7fca4e88e640 1 -- 172.21.15.16:0/661892722 >> 172.21.15.16:0/661892722 conn(0x7fca4807b5f0 msgr2=0x7fca4807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.768+0000 7fca4e88e640 1 -- 172.21.15.16:0/661892722 shutdown_connections 2024-08-28T19:25:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.768+0000 7fca4e88e640 1 -- 172.21.15.16:0/661892722 wait complete. 2024-08-28T19:25:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.768+0000 7fca4e88e640 1 Processor -- start 2024-08-28T19:25:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.768+0000 7fca4e88e640 1 -- start start 2024-08-28T19:25:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca4e88e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca48103900 0x7fca4819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca4e88e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca48109240 0x7fca4819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca4e88e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca48109b90 0x7fca4819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca4e88e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fca48113850 con 0x7fca48109240 2024-08-28T19:25:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca4e88e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca481136d0 con 0x7fca48109b90 2024-08-28T19:25:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca4e88e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fca481139d0 con 0x7fca48103900 2024-08-28T19:25:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca48109240 0x7fca4819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca48109240 0x7fca4819a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41568/0 (socket says 172.21.15.16:41568) 2024-08-28T19:25:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca477fe640 1 -- 172.21.15.16:0/1716663783 learned_addr learned my addr 172.21.15.16:0/1716663783 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca47fff640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca48103900 0x7fca4819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca4ce2b640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca48109b90 0x7fca4819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca477fe640 1 -- 172.21.15.16:0/1716663783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca48103900 msgr2=0x7fca4819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca477fe640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca48103900 0x7fca4819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca477fe640 1 -- 172.21.15.16:0/1716663783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca48109b90 msgr2=0x7fca4819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca477fe640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca48109b90 0x7fca4819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.769+0000 7fca477fe640 1 -- 172.21.15.16:0/1716663783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca481a0110 con 0x7fca48109240 2024-08-28T19:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.770+0000 7fca477fe640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca48109240 0x7fca4819a800 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fca3c02a910 tx=0x7fca3c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.770+0000 7fca457fa640 1 -- 172.21.15.16:0/1716663783 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3c03f030 con 0x7fca48109240 2024-08-28T19:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.770+0000 7fca4e88e640 1 -- 172.21.15.16:0/1716663783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca481a03a0 con 0x7fca48109240 2024-08-28T19:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.770+0000 7fca457fa640 1 -- 172.21.15.16:0/1716663783 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca3c003aa0 con 0x7fca48109240 2024-08-28T19:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.770+0000 7fca457fa640 1 -- 172.21.15.16:0/1716663783 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3c035070 con 0x7fca48109240 2024-08-28T19:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.770+0000 7fca4e88e640 1 -- 172.21.15.16:0/1716663783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca481a08d0 con 0x7fca48109240 2024-08-28T19:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.771+0000 7fca4e88e640 1 -- 172.21.15.16:0/1716663783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca48103d10 con 0x7fca48109240 2024-08-28T19:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.772+0000 7fca457fa640 1 -- 172.21.15.16:0/1716663783 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fca3c042030 con 0x7fca48109240 2024-08-28T19:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.773+0000 7fca457fa640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca300b4cd0 0x7fca300b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.773+0000 7fca47fff640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca300b4cd0 0x7fca300b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.773+0000 7fca457fa640 1 -- 172.21.15.16:0/1716663783 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fca3c1063a0 con 0x7fca48109240 2024-08-28T19:25:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.774+0000 7fca47fff640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca300b4cd0 0x7fca300b71e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fca3800abe0 tx=0x7fca3800a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:04.776+0000 7fca457fa640 1 -- 172.21.15.16:0/1716663783 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fca481a08d0 con 0x7fca48109240 2024-08-28T19:25:04.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 4.63 MB/sec execute 44 sec latency 18855.949 ms 2024-08-28T19:25:05.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.001+0000 7fca4e88e640 1 -- 172.21.15.16:0/1716663783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca4819b390 con 0x7fca48109240 2024-08-28T19:25:05.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.001+0000 7fca457fa640 1 -- 172.21.15.16:0/1716663783 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11984 (secure 0 0 0) 0x7fca4819b390 con 0x7fca48109240 2024-08-28T19:25:05.003 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:25:05.003 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":37,"btime":"2024-08-28T19:25:03:931196+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:25:03.274930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:25:05.003 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 37 2024-08-28T19:25:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 -- 172.21.15.16:0/1716663783 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca300b4cd0 msgr2=0x7fca300b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca300b4cd0 0x7fca300b71e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fca3800abe0 tx=0x7fca3800a2d0 comp rx=0 tx=0).stop 2024-08-28T19:25:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 -- 172.21.15.16:0/1716663783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca48109240 msgr2=0x7fca4819a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca48109240 0x7fca4819a800 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fca3c02a910 tx=0x7fca3c004270 comp rx=0 tx=0).stop 2024-08-28T19:25:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 -- 172.21.15.16:0/1716663783 shutdown_connections 2024-08-28T19:25:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca300b4cd0 0x7fca300b71e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca48109b90 0x7fca4819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca48109240 0x7fca4819a800 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 --2- 172.21.15.16:0/1716663783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca48103900 0x7fca4819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 -- 172.21.15.16:0/1716663783 >> 172.21.15.16:0/1716663783 conn(0x7fca4807b5f0 msgr2=0x7fca48105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 -- 172.21.15.16:0/1716663783 shutdown_connections 2024-08-28T19:25:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:05.004+0000 7fca4e88e640 1 -- 172.21.15.16:0/1716663783 wait complete. 2024-08-28T19:25:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:04 smithi016 ceph-mon[18556]: pgmap v460: 129 pgs: 129 active+clean; 99 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 2.6 MiB/s wr, 114 op/s 2024-08-28T19:25:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:04 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:04 smithi028 ceph-mon[21769]: pgmap v460: 129 pgs: 129 active+clean; 99 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 2.6 MiB/s wr, 114 op/s 2024-08-28T19:25:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:04 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:04 smithi115 ceph-mon[22184]: pgmap v460: 129 pgs: 129 active+clean; 99 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 2.6 MiB/s wr, 114 op/s 2024-08-28T19:25:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:04 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:05.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:25:05.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 4.52 MB/sec execute 45 sec latency 19855.975 ms 2024-08-28T19:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:06.188 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:05 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1716663783' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:25:06.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:05 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1716663783' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:25:06.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:05 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1716663783' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:25:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.612+0000 7f919bcf6640 1 -- 172.21.15.16:0/3258586365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91941039d0 msgr2=0x7f9194103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.612+0000 7f919bcf6640 1 --2- 172.21.15.16:0/3258586365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91941039d0 0x7f9194103de0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f9190009900 tx=0x7f919002a410 comp rx=0 tx=0).stop 2024-08-28T19:25:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.612+0000 7f919bcf6640 1 -- 172.21.15.16:0/3258586365 shutdown_connections 2024-08-28T19:25:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.612+0000 7f919bcf6640 1 --2- 172.21.15.16:0/3258586365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9194109c60 0x7f9194110490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.612+0000 7f919bcf6640 1 --2- 172.21.15.16:0/3258586365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9194109310 0x7f9194109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.612+0000 7f919bcf6640 1 --2- 172.21.15.16:0/3258586365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91941039d0 0x7f9194103de0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.612+0000 7f919bcf6640 1 -- 172.21.15.16:0/3258586365 >> 172.21.15.16:0/3258586365 conn(0x7f919407b670 msgr2=0x7f919407ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.612+0000 7f919bcf6640 1 -- 172.21.15.16:0/3258586365 shutdown_connections 2024-08-28T19:25:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.612+0000 7f919bcf6640 1 -- 172.21.15.16:0/3258586365 wait complete. 2024-08-28T19:25:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919bcf6640 1 Processor -- start 2024-08-28T19:25:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919bcf6640 1 -- start start 2024-08-28T19:25:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919bcf6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91941039d0 0x7f919419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919bcf6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9194109310 0x7f919419c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919bcf6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9194109c60 0x7f91941a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919bcf6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9194113880 con 0x7f9194109c60 2024-08-28T19:25:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919bcf6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9194113700 con 0x7f91941039d0 2024-08-28T19:25:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919bcf6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9194113a00 con 0x7f9194109310 2024-08-28T19:25:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919a293640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9194109c60 0x7f91941a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919a293640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9194109c60 0x7f91941a1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41590/0 (socket says 172.21.15.16:41590) 2024-08-28T19:25:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919a293640 1 -- 172.21.15.16:0/1963892271 learned_addr learned my addr 172.21.15.16:0/1963892271 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f9199a92640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91941039d0 0x7f919419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f9199291640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9194109310 0x7f919419c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919a293640 1 -- 172.21.15.16:0/1963892271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9194109310 msgr2=0x7f919419c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.613+0000 7f919a293640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9194109310 0x7f919419c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.614+0000 7f919a293640 1 -- 172.21.15.16:0/1963892271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91941039d0 msgr2=0x7f919419c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.614+0000 7f919a293640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91941039d0 0x7f919419c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.614+0000 7f919a293640 1 -- 172.21.15.16:0/1963892271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91941a23a0 con 0x7f9194109c60 2024-08-28T19:25:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.614+0000 7f919a293640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9194109c60 0x7f91941a1c70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f918c00eaa0 tx=0x7f918c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.614+0000 7f918affd640 1 -- 172.21.15.16:0/1963892271 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c00cdb0 con 0x7f9194109c60 2024-08-28T19:25:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.614+0000 7f918affd640 1 -- 172.21.15.16:0/1963892271 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f918c005020 con 0x7f9194109c60 2024-08-28T19:25:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.614+0000 7f919bcf6640 1 -- 172.21.15.16:0/1963892271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91941a2690 con 0x7f9194109c60 2024-08-28T19:25:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.614+0000 7f918affd640 1 -- 172.21.15.16:0/1963892271 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c0116d0 con 0x7f9194109c60 2024-08-28T19:25:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.614+0000 7f919bcf6640 1 -- 172.21.15.16:0/1963892271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91941a8df0 con 0x7f9194109c60 2024-08-28T19:25:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.617+0000 7f91a226b740 1 Processor -- start 2024-08-28T19:25:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.617+0000 7f918affd640 1 -- 172.21.15.16:0/1963892271 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f918c0051d0 con 0x7f9194109c60 2024-08-28T19:25:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.617+0000 7f91a226b740 1 -- start start 2024-08-28T19:25:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.617+0000 7f91a226b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 0x55bf5c8a6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.617+0000 7f91a226b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8a6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.617+0000 7f91a226b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4e0390 0x55bf5c8a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.617+0000 7f91a226b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c81d680 con 0x55bf5c4d81b0 2024-08-28T19:25:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.617+0000 7f91a226b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c80cc10 con 0x55bf5c4e0390 2024-08-28T19:25:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.617+0000 7f91a226b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c810bc0 con 0x55bf5c501070 2024-08-28T19:25:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8a6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41594/0 (socket says 172.21.15.16:41594) 2024-08-28T19:25:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199a92640 1 -- 172.21.15.16:0/2022768781 learned_addr learned my addr 172.21.15.16:0/2022768781 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199291640 1 --2- 172.21.15.16:0/2022768781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4e0390 0x55bf5c8a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199291640 1 --2- 172.21.15.16:0/2022768781 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 0x55bf5c8a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f918affd640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91800b4ca0 0x7f91800b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f919a293640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91800b4ca0 0x7f91800b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199a92640 1 -- 172.21.15.16:0/2022768781 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 msgr2=0x55bf5c8a6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199a92640 1 --2- 172.21.15.16:0/2022768781 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 0x55bf5c8a6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199a92640 1 -- 172.21.15.16:0/2022768781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4e0390 msgr2=0x55bf5c8a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199a92640 1 --2- 172.21.15.16:0/2022768781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4e0390 0x55bf5c8a6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199a92640 1 -- 172.21.15.16:0/2022768781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf5c56c650 con 0x55bf5c4d81b0 2024-08-28T19:25:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199291640 1 --2- 172.21.15.16:0/2022768781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4e0390 0x55bf5c8a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f918affd640 1 -- 172.21.15.16:0/1963892271 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f918c0e5a00 con 0x7f9194109c60 2024-08-28T19:25:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.618+0000 7f9199a92640 1 --2- 172.21.15.16:0/2022768781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8a6850 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9190009900 tx=0x7f9190032cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f248e87dd80f7915 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.619+0000 7f917ffff640 1 -- 172.21.15.16:0/2022768781 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190040070 con 0x55bf5c4d81b0 2024-08-28T19:25:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.619+0000 7f919a293640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91800b4ca0 0x7f91800b71b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f918c0e33e0 tx=0x7f918c0792d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.619+0000 7f917ffff640 1 -- 172.21.15.16:0/2022768781 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9190032e40 con 0x55bf5c4d81b0 2024-08-28T19:25:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.619+0000 7f917ffff640 1 -- 172.21.15.16:0/2022768781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190035040 con 0x55bf5c4d81b0 2024-08-28T19:25:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.619+0000 7f91a226b740 1 -- 172.21.15.16:0/2022768781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4d81b0 msgr2=0x55bf5c8a6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.619+0000 7f91a226b740 1 --2- 172.21.15.16:0/2022768781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8a6850 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9190009900 tx=0x7f9190032cd0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.619+0000 7f91a226b740 1 -- 172.21.15.16:0/2022768781 shutdown_connections 2024-08-28T19:25:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.619+0000 7f91a226b740 1 --2- 172.21.15.16:0/2022768781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4e0390 0x55bf5c8a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.619+0000 7f91a226b740 1 --2- 172.21.15.16:0/2022768781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8a6850 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.620+0000 7f91a226b740 1 --2- 172.21.15.16:0/2022768781 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 0x55bf5c8a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.620+0000 7f91a226b740 1 -- 172.21.15.16:0/2022768781 >> 172.21.15.16:0/2022768781 conn(0x55bf5c4f5460 msgr2=0x55bf5c8a4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.620+0000 7f91a226b740 1 -- 172.21.15.16:0/2022768781 shutdown_connections 2024-08-28T19:25:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.620+0000 7f91a226b740 1 -- 172.21.15.16:0/2022768781 wait complete. 2024-08-28T19:25:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.620+0000 7f91a226b740 1 -- start start 2024-08-28T19:25:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.620+0000 7f91a226b740 1 Processor -- start 2024-08-28T19:25:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f91a226b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f91a226b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f91a226b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 0x55bf5c8bddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f91a226b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c66f700 con 0x55bf5c4e0390 2024-08-28T19:25:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f91a226b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c793750 con 0x55bf5c4d81b0 2024-08-28T19:25:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f91a226b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c8b4a90 con 0x55bf5c501070 2024-08-28T19:25:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f91a226b740 10 client.0 _ll_register_callbacks cb 0x55bf5c7c2e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41604/0 (socket says 172.21.15.16:41604) 2024-08-28T19:25:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199a92640 1 -- 172.21.15.16:0/2489963856 learned_addr learned my addr 172.21.15.16:0/2489963856 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199291640 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 0x55bf5c8bddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199291640 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199a92640 1 -- 172.21.15.16:0/2489963856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 msgr2=0x55bf5c8bddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199a92640 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 0x55bf5c8bddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199a92640 1 -- 172.21.15.16:0/2489963856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 msgr2=0x55bf5c8b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199a92640 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8b1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199a92640 1 -- 172.21.15.16:0/2489963856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf5c7cc900 con 0x55bf5c4e0390 2024-08-28T19:25:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199291640 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:25:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f9199a92640 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b1fe0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f919002a930 tx=0x7f919002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69178e1f6e8ae7a6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f917d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f917d7fa640 1 -- 172.21.15.16:0/2489963856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919004b030 con 0x55bf5c4e0390 2024-08-28T19:25:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.621+0000 7f917d7fa640 1 -- 172.21.15.16:0/2489963856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f91900316b0 con 0x55bf5c4e0390 2024-08-28T19:25:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f91a226b740 1 -- 172.21.15.16:0/2489963856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bf5c81f5d0 con 0x55bf5c4e0390 2024-08-28T19:25:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f917d7fa640 1 -- 172.21.15.16:0/2489963856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9190055050 con 0x55bf5c4e0390 2024-08-28T19:25:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f91a226b740 10 client.16659 fetch_fsmap learned FSMap version 37 2024-08-28T19:25:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f91a226b740 1 -- 172.21.15.16:0/2489963856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bf5c8c0850 con 0x55bf5c4e0390 2024-08-28T19:25:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f917d7fa640 1 -- 172.21.15.16:0/2489963856 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55bf5c81f5d0 con 0x55bf5c4e0390 2024-08-28T19:25:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f917d7fa640 1 -- 172.21.15.16:0/2489963856 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) ==== 8390+0+0 (secure 0 0 0) 0x7f9190003a90 con 0x55bf5c4e0390 2024-08-28T19:25:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f91a226b740 10 client.16659 fetch_fsmap finished waiting for FSMap version 37 2024-08-28T19:25:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f91a226b740 10 client.16659 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:25:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f91a226b740 20 client.16659 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f91a226b740 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55bf5c4bf9f0 0x55bf5c8c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f91a226b740 4 client.16659 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.622+0000 7f91a226b740 1 -- 172.21.15.16:0/2489963856 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf5c81b070 con 0x55bf5c4bf9f0 2024-08-28T19:25:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.623+0000 7f9199291640 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55bf5c4bf9f0 0x55bf5c8c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.623+0000 7f9199291640 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55bf5c4bf9f0 0x55bf5c8c3bb0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=240274532a3cc13a server_cookie=db5c76f5b99db69c in_seq=0 out_seq=0 2024-08-28T19:25:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.623+0000 7f917d7fa640 10 client.16659 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:25:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f917d7fa640 1 -- 172.21.15.16:0/2489963856 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55bf5c81b070 con 0x55bf5c4bf9f0 2024-08-28T19:25:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f917d7fa640 10 client.16659 handle_command_reply: tid=0 2024-08-28T19:25:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 1 client.16659 shutdown 2024-08-28T19:25:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 1 -- 172.21.15.16:0/2489963856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 msgr2=0x55bf5c8b1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b1fe0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f919002a930 tx=0x7f919002a410 comp rx=0 tx=0).stop 2024-08-28T19:25:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 1 -- 172.21.15.16:0/2489963856 shutdown_connections 2024-08-28T19:25:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55bf5c4bf9f0 0x55bf5c8c3bb0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 0x55bf5c8bddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b1fe0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 1 --2- 172.21.15.16:0/2489963856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f917d7fa640 0 client.16659 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:25:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 1 -- 172.21.15.16:0/2489963856 >> 172.21.15.16:0/2489963856 conn(0x55bf5c4f5460 msgr2=0x55bf5c8a94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 1 -- 172.21.15.16:0/2489963856 shutdown_connections 2024-08-28T19:25:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 1 -- 172.21.15.16:0/2489963856 wait complete. 2024-08-28T19:25:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.624+0000 7f91a226b740 20 client.16659 trim_cache size 0 max 16384 2024-08-28T19:25:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f91a226b740 1 Processor -- start 2024-08-28T19:25:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f91a226b740 1 -- start start 2024-08-28T19:25:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f91a226b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8b0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f91a226b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f91a226b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 0x55bf5c8a9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f91a226b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91941b2d20 con 0x55bf5c4e0390 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f91a226b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91941b4950 con 0x55bf5c4d81b0 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f91a226b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f91941b3450 con 0x55bf5c501070 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f9199a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f9199a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41606/0 (socket says 172.21.15.16:41606) 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f9199a92640 1 -- 172.21.15.16:0/2433604468 learned_addr learned my addr 172.21.15.16:0/2433604468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f9199291640 1 --2- 172.21.15.16:0/2433604468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f9199a92640 1 -- 172.21.15.16:0/2433604468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 msgr2=0x55bf5c8a9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f9199a92640 1 --2- 172.21.15.16:0/2433604468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 0x55bf5c8a9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f9199a92640 1 -- 172.21.15.16:0/2433604468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 msgr2=0x55bf5c8b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f9199a92640 1 --2- 172.21.15.16:0/2433604468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8b0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f9199a92640 1 -- 172.21.15.16:0/2433604468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf5c8b49d0 con 0x55bf5c4e0390 2024-08-28T19:25:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f9199291640 1 --2- 172.21.15.16:0/2433604468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8b0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:25:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.638+0000 7f9199a92640 1 --2- 172.21.15.16:0/2433604468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b0860 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9190009900 tx=0x7f91900058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3ec65bf3a9cd9d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.639+0000 7f917f7fe640 1 -- 172.21.15.16:0/2433604468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919004b040 con 0x55bf5c4e0390 2024-08-28T19:25:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.639+0000 7f917f7fe640 1 -- 172.21.15.16:0/2433604468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9190004480 con 0x55bf5c4e0390 2024-08-28T19:25:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.639+0000 7f917f7fe640 1 -- 172.21.15.16:0/2433604468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190035070 con 0x55bf5c4e0390 2024-08-28T19:25:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.639+0000 7f91a226b740 1 -- 172.21.15.16:0/2433604468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 msgr2=0x55bf5c8b0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.639+0000 7f91a226b740 1 --2- 172.21.15.16:0/2433604468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b0860 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9190009900 tx=0x7f91900058f0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.640+0000 7f91a226b740 1 -- 172.21.15.16:0/2433604468 shutdown_connections 2024-08-28T19:25:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.640+0000 7f91a226b740 1 --2- 172.21.15.16:0/2433604468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c501070 0x55bf5c8a9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.640+0000 7f91a226b740 1 --2- 172.21.15.16:0/2433604468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8b0860 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.640+0000 7f91a226b740 1 --2- 172.21.15.16:0/2433604468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8b0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.640+0000 7f91a226b740 1 -- 172.21.15.16:0/2433604468 >> 172.21.15.16:0/2433604468 conn(0x55bf5c4f5460 msgr2=0x55bf5c8acbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:06.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.640+0000 7f91a226b740 1 -- 172.21.15.16:0/2433604468 shutdown_connections 2024-08-28T19:25:06.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.640+0000 7f91a226b740 1 -- 172.21.15.16:0/2433604468 wait complete. 2024-08-28T19:25:06.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.640+0000 7f91a226b740 1 -- start start 2024-08-28T19:25:06.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.640+0000 7f91a226b740 1 Processor -- start 2024-08-28T19:25:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f91a226b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8c4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f91a226b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f91a226b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c501070 0x55bf5c925350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f91a226b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91941b4250 con 0x55bf5c4e0390 2024-08-28T19:25:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f91a226b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91941b3b80 con 0x55bf5c501070 2024-08-28T19:25:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f91a226b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f91941b0d00 con 0x55bf5c4d81b0 2024-08-28T19:25:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f91a226b740 10 client.0 _ll_register_callbacks cb 0x7f918400ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8c4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41618/0 (socket says 172.21.15.16:41618) 2024-08-28T19:25:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199a92640 1 -- 172.21.15.16:0/3398090537 learned_addr learned my addr 172.21.15.16:0/3398090537 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199291640 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199a92640 1 -- 172.21.15.16:0/3398090537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c4d81b0 msgr2=0x55bf5c8c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199291640 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c501070 0x55bf5c925350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199a92640 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8c4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199a92640 1 -- 172.21.15.16:0/3398090537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c501070 msgr2=0x55bf5c925350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199a92640 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c501070 0x55bf5c925350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199a92640 1 -- 172.21.15.16:0/3398090537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf5c9259c0 con 0x55bf5c4e0390 2024-08-28T19:25:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199291640 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c501070 0x55bf5c925350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:25:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f9199a92640 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8c4cf0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f91900095e0 tx=0x7f9190005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d304eece85ad992 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f917d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.641+0000 7f917d7fa640 1 -- 172.21.15.16:0/3398090537 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190035040 con 0x55bf5c4e0390 2024-08-28T19:25:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f917d7fa640 1 -- 172.21.15.16:0/3398090537 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f91900035f0 con 0x55bf5c4e0390 2024-08-28T19:25:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f91a226b740 1 -- 172.21.15.16:0/3398090537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9190004480 con 0x55bf5c4e0390 2024-08-28T19:25:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f917d7fa640 1 -- 172.21.15.16:0/3398090537 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f919006f050 con 0x55bf5c4e0390 2024-08-28T19:25:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f91a226b740 10 client.16665 fetch_fsmap learned FSMap version 37 2024-08-28T19:25:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f91a226b740 1 -- 172.21.15.16:0/3398090537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bf5c92de50 con 0x55bf5c4e0390 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f917d7fa640 1 -- 172.21.15.16:0/3398090537 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9190004480 con 0x55bf5c4e0390 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f917d7fa640 1 -- 172.21.15.16:0/3398090537 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) ==== 8390+0+0 (secure 0 0 0) 0x7f919006a030 con 0x55bf5c4e0390 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f91a226b740 10 client.16665 fetch_fsmap finished waiting for FSMap version 37 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f91a226b740 10 client.16665 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f91a226b740 20 client.16665 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f91a226b740 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55bf5c494da0 0x55bf5c930630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f91a226b740 4 client.16665 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f91a226b740 1 -- 172.21.15.16:0/3398090537 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bf5c8187b0 con 0x55bf5c494da0 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.642+0000 7f9199291640 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55bf5c494da0 0x55bf5c930630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f9199291640 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55bf5c494da0 0x55bf5c930630 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b1aa69000f001d6e server_cookie=1f3c79c479cb21fe in_seq=0 out_seq=0 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f917d7fa640 10 client.16665 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:25:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f917d7fa640 1 -- 172.21.15.16:0/3398090537 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bf5c8187b0 con 0x55bf5c494da0 2024-08-28T19:25:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f917d7fa640 10 client.16665 handle_command_reply: tid=0 2024-08-28T19:25:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f91a226b740 1 client.16665 shutdown 2024-08-28T19:25:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f91a226b740 1 -- 172.21.15.16:0/3398090537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 msgr2=0x55bf5c8c4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f91a226b740 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8c4cf0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f91900095e0 tx=0x7f9190005710 comp rx=0 tx=0).stop 2024-08-28T19:25:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f91a226b740 1 -- 172.21.15.16:0/3398090537 shutdown_connections 2024-08-28T19:25:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f91a226b740 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55bf5c494da0 0x55bf5c930630 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f91a226b740 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf5c501070 0x55bf5c925350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f91a226b740 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf5c4e0390 0x55bf5c8c4cf0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f91a226b740 1 --2- 172.21.15.16:0/3398090537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bf5c4d81b0 0x55bf5c8c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f917d7fa640 0 client.16665 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:25:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f91a226b740 1 -- 172.21.15.16:0/3398090537 >> 172.21.15.16:0/3398090537 conn(0x55bf5c4f5460 msgr2=0x55bf5c8a9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.643+0000 7f91a226b740 1 -- 172.21.15.16:0/3398090537 shutdown_connections 2024-08-28T19:25:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f91a226b740 1 -- 172.21.15.16:0/3398090537 wait complete. 2024-08-28T19:25:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f91a226b740 20 client.16665 trim_cache size 0 max 16384 2024-08-28T19:25:06.652 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:25:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 -- 172.21.15.16:0/1963892271 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91800b4ca0 msgr2=0x7f91800b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91800b4ca0 0x7f91800b71b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f918c0e33e0 tx=0x7f918c0792d0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 -- 172.21.15.16:0/1963892271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9194109c60 msgr2=0x7f91941a1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9194109c60 0x7f91941a1c70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f918c00eaa0 tx=0x7f918c00ef60 comp rx=0 tx=0).stop 2024-08-28T19:25:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 -- 172.21.15.16:0/1963892271 shutdown_connections 2024-08-28T19:25:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9194109c60 0x7f91941a1c70 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9194109310 0x7f919419c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91800b4ca0 0x7f91800b71b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 --2- 172.21.15.16:0/1963892271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91941039d0 0x7f919419c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 -- 172.21.15.16:0/1963892271 >> 172.21.15.16:0/1963892271 conn(0x7f919407b670 msgr2=0x7f9194107120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 -- 172.21.15.16:0/1963892271 shutdown_connections 2024-08-28T19:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:06.644+0000 7f919bcf6640 1 -- 172.21.15.16:0/1963892271 wait complete. 2024-08-28T19:25:06.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 4.43 MB/sec execute 46 sec latency 20856.001 ms 2024-08-28T19:25:07.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:06 smithi016 ceph-mon[18556]: pgmap v461: 129 pgs: 129 active+clean; 97 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 2.6 MiB/s wr, 127 op/s 2024-08-28T19:25:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:06 smithi028 ceph-mon[21769]: pgmap v461: 129 pgs: 129 active+clean; 97 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 2.6 MiB/s wr, 127 op/s 2024-08-28T19:25:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:06 smithi115 ceph-mon[22184]: pgmap v461: 129 pgs: 129 active+clean; 97 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 2.6 MiB/s wr, 127 op/s 2024-08-28T19:25:07.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:25:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:07.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 4.33 MB/sec execute 47 sec latency 21856.038 ms 2024-08-28T19:25:08.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.275+0000 7f7f5dd23640 1 -- 172.21.15.16:0/576278868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f58102880 msgr2=0x7f7f58102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:08.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.275+0000 7f7f5dd23640 1 --2- 172.21.15.16:0/576278868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f58102880 0x7f7f58102c90 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009900 tx=0x7f7f4802a410 comp rx=0 tx=0).stop 2024-08-28T19:25:08.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.275+0000 7f7f5dd23640 1 -- 172.21.15.16:0/576278868 shutdown_connections 2024-08-28T19:25:08.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.275+0000 7f7f5dd23640 1 --2- 172.21.15.16:0/576278868 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f58108b10 0x7f7f5810f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.275+0000 7f7f5dd23640 1 --2- 172.21.15.16:0/576278868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f581081c0 0x7f7f581085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.275+0000 7f7f5dd23640 1 --2- 172.21.15.16:0/576278868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f58102880 0x7f7f58102c90 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.275+0000 7f7f5dd23640 1 -- 172.21.15.16:0/576278868 >> 172.21.15.16:0/576278868 conn(0x7f7f58079510 msgr2=0x7f7f58079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.275+0000 7f7f5dd23640 1 -- 172.21.15.16:0/576278868 shutdown_connections 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.275+0000 7f7f5dd23640 1 -- 172.21.15.16:0/576278868 wait complete. 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.276+0000 7f7f5dd23640 1 Processor -- start 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.276+0000 7f7f5dd23640 1 -- start start 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.276+0000 7f7f5dd23640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f58102880 0x7f7f580ff410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.276+0000 7f7f5dd23640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f581081c0 0x7f7f580ff950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.276+0000 7f7f5dd23640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58108b10 0x7f7f580ffe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.276+0000 7f7f5dd23640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581127e0 con 0x7f7f581081c0 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.276+0000 7f7f5dd23640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f58112660 con 0x7f7f58108b10 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.276+0000 7f7f5dd23640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7f58112960 con 0x7f7f58102880 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.276+0000 7f7f4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f581081c0 0x7f7f580ff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.276+0000 7f7f4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f581081c0 0x7f7f580ff950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41646/0 (socket says 172.21.15.16:41646) 2024-08-28T19:25:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.276+0000 7f7f4effd640 1 -- 172.21.15.16:0/372106952 learned_addr learned my addr 172.21.15.16:0/372106952 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f577fe640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f58102880 0x7f7f580ff410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f57fff640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58108b10 0x7f7f580ffe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f4effd640 1 -- 172.21.15.16:0/372106952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f58102880 msgr2=0x7f7f580ff410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f4effd640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f58102880 0x7f7f580ff410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f4effd640 1 -- 172.21.15.16:0/372106952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58108b10 msgr2=0x7f7f580ffe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f4effd640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58108b10 0x7f7f580ffe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f4effd640 1 -- 172.21.15.16:0/372106952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f581a85c0 con 0x7f7f581081c0 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f4effd640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f581081c0 0x7f7f580ff950 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000d950 tx=0x7f7f4000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f557fa640 1 -- 172.21.15.16:0/372106952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f40017070 con 0x7f7f581081c0 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f557fa640 1 -- 172.21.15.16:0/372106952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f40009c30 con 0x7f7f581081c0 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f557fa640 1 -- 172.21.15.16:0/372106952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f400047a0 con 0x7f7f581081c0 2024-08-28T19:25:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f5dd23640 1 -- 172.21.15.16:0/372106952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f581a88b0 con 0x7f7f581081c0 2024-08-28T19:25:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.277+0000 7f7f5dd23640 1 -- 172.21.15.16:0/372106952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f581a8df0 con 0x7f7f581081c0 2024-08-28T19:25:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.280+0000 7f7f64298740 1 Processor -- start 2024-08-28T19:25:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.280+0000 7f7f64298740 1 -- start start 2024-08-28T19:25:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.280+0000 7f7f557fa640 1 -- 172.21.15.16:0/372106952 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7f40004020 con 0x7f7f581081c0 2024-08-28T19:25:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.280+0000 7f7f64298740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5f1070 0x5591df996350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.280+0000 7f7f64298740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5c81b0 0x5591df996890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.280+0000 7f7f64298740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5d0390 0x5591df996dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f64298740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591df90d680 con 0x5591df5d0390 2024-08-28T19:25:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f64298740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591df8fcc10 con 0x5591df5c81b0 2024-08-28T19:25:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f64298740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5591df900bc0 con 0x5591df5f1070 2024-08-28T19:25:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5d0390 0x5591df996dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f57fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5f1070 0x5591df996350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5d0390 0x5591df996dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41658/0 (socket says 172.21.15.16:41658) 2024-08-28T19:25:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5c81b0 0x5591df996890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f4effd640 1 -- 172.21.15.16:0/1323504748 learned_addr learned my addr 172.21.15.16:0/1323504748 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f4effd640 1 -- 172.21.15.16:0/1323504748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5f1070 msgr2=0x5591df996350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f4effd640 1 --2- 172.21.15.16:0/1323504748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5f1070 0x5591df996350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f4effd640 1 -- 172.21.15.16:0/1323504748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5c81b0 msgr2=0x5591df996890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f4effd640 1 --2- 172.21.15.16:0/1323504748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5c81b0 0x5591df996890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f4effd640 1 -- 172.21.15.16:0/1323504748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591df65c650 con 0x5591df5d0390 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f4effd640 1 --2- 172.21.15.16:0/1323504748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5d0390 0x5591df996dd0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7f400e2880 tx=0x7f7f400ebba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61658f0e1f913032 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f557fa640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f440b4ca0 0x7f7f440b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.281+0000 7f7f4dffb640 1 -- 172.21.15.16:0/1323504748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f400f8070 con 0x5591df5d0390 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.282+0000 7f7f4dffb640 1 -- 172.21.15.16:0/1323504748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f400ab3c0 con 0x5591df5d0390 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.282+0000 7f7f4dffb640 1 -- 172.21.15.16:0/1323504748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f400ca4f0 con 0x5591df5d0390 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.282+0000 7f7f57fff640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f440b4ca0 0x7f7f440b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.282+0000 7f7f557fa640 1 -- 172.21.15.16:0/372106952 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7f400e5530 con 0x7f7f581081c0 2024-08-28T19:25:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f64298740 1 -- 172.21.15.16:0/1323504748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5d0390 msgr2=0x5591df996dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f57fff640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f440b4ca0 0x7f7f440b71b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7f50005fd0 tx=0x7f7f50009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f64298740 1 --2- 172.21.15.16:0/1323504748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5d0390 0x5591df996dd0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7f400e2880 tx=0x7f7f400ebba0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f64298740 1 -- 172.21.15.16:0/1323504748 shutdown_connections 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f64298740 1 --2- 172.21.15.16:0/1323504748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5d0390 0x5591df996dd0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f64298740 1 --2- 172.21.15.16:0/1323504748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5c81b0 0x5591df996890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f64298740 1 --2- 172.21.15.16:0/1323504748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5f1070 0x5591df996350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f64298740 1 -- 172.21.15.16:0/1323504748 >> 172.21.15.16:0/1323504748 conn(0x5591df5e5460 msgr2=0x5591df994240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f64298740 1 -- 172.21.15.16:0/1323504748 shutdown_connections 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f64298740 1 -- 172.21.15.16:0/1323504748 wait complete. 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f64298740 1 -- start start 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.283+0000 7f7f64298740 1 Processor -- start 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f64298740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f64298740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5d0390 0x5591df9a1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f64298740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5f1070 0x5591df9ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f64298740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591df75f700 con 0x5591df5c81b0 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f64298740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591df883750 con 0x5591df5f1070 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f64298740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5591df9a4b30 con 0x5591df5d0390 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41670/0 (socket says 172.21.15.16:41670) 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f577fe640 1 -- 172.21.15.16:0/11949306 learned_addr learned my addr 172.21.15.16:0/11949306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f57fff640 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5f1070 0x5591df9ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f4effd640 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5d0390 0x5591df9a1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f64298740 10 client.0 _ll_register_callbacks cb 0x5591df8b2e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f577fe640 1 -- 172.21.15.16:0/11949306 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5d0390 msgr2=0x5591df9a1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f577fe640 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5d0390 0x5591df9a1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f577fe640 1 -- 172.21.15.16:0/11949306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5f1070 msgr2=0x5591df9ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f577fe640 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5f1070 0x5591df9ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f577fe640 1 -- 172.21.15.16:0/11949306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591df8bc900 con 0x5591df5c81b0 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f57fff640 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5f1070 0x5591df9ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f577fe640 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a1a20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009900 tx=0x7f7f480316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eeb257084232e73 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.284+0000 7f7f36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f36ffd640 1 -- 172.21.15.16:0/11949306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48035040 con 0x5591df5c81b0 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f36ffd640 1 -- 172.21.15.16:0/11949306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7f48004040 con 0x5591df5c81b0 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f64298740 1 -- 172.21.15.16:0/11949306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f400ab3c0 con 0x5591df5c81b0 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f36ffd640 1 -- 172.21.15.16:0/11949306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7f48032e30 con 0x5591df5c81b0 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f64298740 10 client.16677 fetch_fsmap learned FSMap version 37 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f64298740 1 -- 172.21.15.16:0/11949306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5591df9b0860 con 0x5591df5c81b0 2024-08-28T19:25:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f36ffd640 1 -- 172.21.15.16:0/11949306 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f400ab3c0 con 0x5591df5c81b0 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f36ffd640 1 -- 172.21.15.16:0/11949306 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) ==== 8390+0+0 (secure 0 0 0) 0x7f7f48004770 con 0x5591df5c81b0 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f64298740 10 client.16677 fetch_fsmap finished waiting for FSMap version 37 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f64298740 10 client.16677 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f64298740 20 client.16677 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f64298740 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5591df5af9f0 0x5591df9b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f64298740 4 client.16677 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.285+0000 7f7f64298740 1 -- 172.21.15.16:0/11949306 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591df90f5d0 con 0x5591df5af9f0 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.286+0000 7f7f4effd640 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5591df5af9f0 0x5591df9b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.286+0000 7f7f36ffd640 10 client.16677 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.286+0000 7f7f4effd640 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5591df5af9f0 0x5591df9b3de0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=10e2406b285db2e6 server_cookie=d47c16096b28a5b1 in_seq=0 out_seq=0 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f36ffd640 1 -- 172.21.15.16:0/11949306 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5591df90f5d0 con 0x5591df5af9f0 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f36ffd640 10 client.16677 handle_command_reply: tid=0 2024-08-28T19:25:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 1 client.16677 shutdown 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 1 -- 172.21.15.16:0/11949306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 msgr2=0x5591df9a1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a1a20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009900 tx=0x7f7f480316d0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 1 -- 172.21.15.16:0/11949306 shutdown_connections 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5591df5af9f0 0x5591df9b3de0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5f1070 0x5591df9ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5d0390 0x5591df9a1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 1 --2- 172.21.15.16:0/11949306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a1a20 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f36ffd640 0 client.16677 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 1 -- 172.21.15.16:0/11949306 >> 172.21.15.16:0/11949306 conn(0x5591df5e5460 msgr2=0x5591df9994e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 1 -- 172.21.15.16:0/11949306 shutdown_connections 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 1 -- 172.21.15.16:0/11949306 wait complete. 2024-08-28T19:25:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.287+0000 7f7f64298740 20 client.16677 trim_cache size 0 max 16384 2024-08-28T19:25:08.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f64298740 1 Processor -- start 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f64298740 1 -- start start 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f64298740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9add50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f64298740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5d0390 0x5591df99ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f64298740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5f1070 0x5591df99f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f64298740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581b2e60 con 0x5591df5c81b0 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f64298740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581b4a90 con 0x5591df5d0390 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f64298740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581b3590 con 0x5591df5f1070 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9add50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41678/0 (socket says 172.21.15.16:41678) 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f577fe640 1 -- 172.21.15.16:0/3027284391 learned_addr learned my addr 172.21.15.16:0/3027284391 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.301+0000 7f7f4effd640 1 --2- 172.21.15.16:0/3027284391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5d0390 0x5591df99ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.302+0000 7f7f57fff640 1 --2- 172.21.15.16:0/3027284391 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5f1070 0x5591df99f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.302+0000 7f7f577fe640 1 -- 172.21.15.16:0/3027284391 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5f1070 msgr2=0x5591df99f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.302+0000 7f7f577fe640 1 --2- 172.21.15.16:0/3027284391 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5f1070 0x5591df99f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.302+0000 7f7f577fe640 1 -- 172.21.15.16:0/3027284391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5d0390 msgr2=0x5591df99ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.302+0000 7f7f577fe640 1 --2- 172.21.15.16:0/3027284391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5d0390 0x5591df99ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.302+0000 7f7f577fe640 1 -- 172.21.15.16:0/3027284391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591df9a2ce0 con 0x5591df5c81b0 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.302+0000 7f7f57fff640 1 --2- 172.21.15.16:0/3027284391 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5f1070 0x5591df99f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.302+0000 7f7f577fe640 1 --2- 172.21.15.16:0/3027284391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9add50 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7f48004bc0 tx=0x7f7f4802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36156bfc98742743 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.302+0000 7f7f4d7fa640 1 -- 172.21.15.16:0/3027284391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48035030 con 0x5591df5c81b0 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.302+0000 7f7f4d7fa640 1 -- 172.21.15.16:0/3027284391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f480048c0 con 0x5591df5c81b0 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.302+0000 7f7f4d7fa640 1 -- 172.21.15.16:0/3027284391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48005230 con 0x5591df5c81b0 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.303+0000 7f7f64298740 1 -- 172.21.15.16:0/3027284391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 msgr2=0x5591df9add50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.303+0000 7f7f64298740 1 --2- 172.21.15.16:0/3027284391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9add50 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7f48004bc0 tx=0x7f7f4802a410 comp rx=0 tx=0).stop 2024-08-28T19:25:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.303+0000 7f7f64298740 1 -- 172.21.15.16:0/3027284391 shutdown_connections 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.303+0000 7f7f64298740 1 --2- 172.21.15.16:0/3027284391 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5f1070 0x5591df99f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.303+0000 7f7f64298740 1 --2- 172.21.15.16:0/3027284391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5d0390 0x5591df99ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.303+0000 7f7f64298740 1 --2- 172.21.15.16:0/3027284391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9add50 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.303+0000 7f7f64298740 1 -- 172.21.15.16:0/3027284391 >> 172.21.15.16:0/3027284391 conn(0x5591df5e5460 msgr2=0x5591df9a02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.303+0000 7f7f64298740 1 -- 172.21.15.16:0/3027284391 shutdown_connections 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.303+0000 7f7f64298740 1 -- 172.21.15.16:0/3027284391 wait complete. 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.303+0000 7f7f64298740 1 -- start start 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f64298740 1 Processor -- start 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f64298740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f64298740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5d0390 0x5591df9a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f64298740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5f1070 0x5591dfa17770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f64298740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581b4390 con 0x5591df5c81b0 2024-08-28T19:25:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f64298740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581b3cc0 con 0x5591df5f1070 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f64298740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581b0e40 con 0x5591df5d0390 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f64298740 10 client.0 _ll_register_callbacks cb 0x7f7f48005db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41690/0 (socket says 172.21.15.16:41690) 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f577fe640 1 -- 172.21.15.16:0/2895479470 learned_addr learned my addr 172.21.15.16:0/2895479470 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f57fff640 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5f1070 0x5591dfa17770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f4effd640 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5d0390 0x5591df9a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f577fe640 1 -- 172.21.15.16:0/2895479470 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5d0390 msgr2=0x5591df9a6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f577fe640 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5d0390 0x5591df9a6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f577fe640 1 -- 172.21.15.16:0/2895479470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5f1070 msgr2=0x5591dfa17770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f577fe640 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5f1070 0x5591dfa17770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f577fe640 1 -- 172.21.15.16:0/2895479470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591dfa17de0 con 0x5591df5c81b0 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.304+0000 7f7f4effd640 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5d0390 0x5591df9a6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f577fe640 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a6680 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7f48004d30 tx=0x7f7f48005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c7ad218f505e1de server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f36ffd640 1 -- 172.21.15.16:0/2895479470 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48061060 con 0x5591df5c81b0 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f36ffd640 1 -- 172.21.15.16:0/2895479470 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7f48005320 con 0x5591df5c81b0 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f64298740 1 -- 172.21.15.16:0/2895479470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f480048c0 con 0x5591df5c81b0 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f36ffd640 1 -- 172.21.15.16:0/2895479470 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7f48068050 con 0x5591df5c81b0 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f64298740 10 client.16683 fetch_fsmap learned FSMap version 37 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f64298740 1 -- 172.21.15.16:0/2895479470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5591dfa1f070 con 0x5591df5c81b0 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f36ffd640 1 -- 172.21.15.16:0/2895479470 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f480048c0 con 0x5591df5c81b0 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f36ffd640 1 -- 172.21.15.16:0/2895479470 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) ==== 8390+0+0 (secure 0 0 0) 0x7f7f48046470 con 0x5591df5c81b0 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f64298740 10 client.16683 fetch_fsmap finished waiting for FSMap version 37 2024-08-28T19:25:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f64298740 10 client.16683 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f64298740 20 client.16683 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.305+0000 7f7f64298740 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5591df584da0 0x5591dfa21720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.306+0000 7f7f64298740 4 client.16683 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.306+0000 7f7f64298740 1 -- 172.21.15.16:0/2895479470 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5591df90b070 con 0x5591df584da0 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.306+0000 7f7f4effd640 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5591df584da0 0x5591dfa21720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.306+0000 7f7f4effd640 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5591df584da0 0x5591dfa21720 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3d5f6b88119cb136 server_cookie=90ebe9289c837f67 in_seq=0 out_seq=0 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.306+0000 7f7f36ffd640 10 client.16683 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f36ffd640 1 -- 172.21.15.16:0/2895479470 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5591df90b070 con 0x5591df584da0 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f36ffd640 10 client.16683 handle_command_reply: tid=0 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 1 client.16683 shutdown 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 1 -- 172.21.15.16:0/2895479470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 msgr2=0x5591df9a6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a6680 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7f48004d30 tx=0x7f7f48005a20 comp rx=0 tx=0).stop 2024-08-28T19:25:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 1 -- 172.21.15.16:0/2895479470 shutdown_connections 2024-08-28T19:25:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5591df584da0 0x5591dfa21720 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591df5f1070 0x5591dfa17770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591df5d0390 0x5591df9a6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f36ffd640 0 client.16683 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:25:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 1 --2- 172.21.15.16:0/2895479470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591df5c81b0 0x5591df9a6680 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 1 -- 172.21.15.16:0/2895479470 >> 172.21.15.16:0/2895479470 conn(0x5591df5e5460 msgr2=0x5591df997380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 1 -- 172.21.15.16:0/2895479470 shutdown_connections 2024-08-28T19:25:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 1 -- 172.21.15.16:0/2895479470 wait complete. 2024-08-28T19:25:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.307+0000 7f7f64298740 20 client.16683 trim_cache size 0 max 16384 2024-08-28T19:25:08.317 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:25:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 -- 172.21.15.16:0/372106952 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f440b4ca0 msgr2=0x7f7f440b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f440b4ca0 0x7f7f440b71b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7f50005fd0 tx=0x7f7f50009600 comp rx=0 tx=0).stop 2024-08-28T19:25:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 -- 172.21.15.16:0/372106952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f581081c0 msgr2=0x7f7f580ff950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f581081c0 0x7f7f580ff950 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000d950 tx=0x7f7f4000de10 comp rx=0 tx=0).stop 2024-08-28T19:25:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 -- 172.21.15.16:0/372106952 shutdown_connections 2024-08-28T19:25:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7f440b4ca0 0x7f7f440b71b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58108b10 0x7f7f580ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f581081c0 0x7f7f580ff950 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 --2- 172.21.15.16:0/372106952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f58102880 0x7f7f580ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 -- 172.21.15.16:0/372106952 >> 172.21.15.16:0/372106952 conn(0x7f7f58079510 msgr2=0x7f7f581018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 -- 172.21.15.16:0/372106952 shutdown_connections 2024-08-28T19:25:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:08.308+0000 7f7f5dd23640 1 -- 172.21.15.16:0/372106952 wait complete. 2024-08-28T19:25:08.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 4.24 MB/sec execute 48 sec latency 22856.086 ms 2024-08-28T19:25:09.026 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:25:09.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:08 smithi016 ceph-mon[18556]: pgmap v462: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-08-28T19:25:09.279 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:08 smithi028 ceph-mon[21769]: pgmap v462: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-08-28T19:25:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:08 smithi115 ceph-mon[22184]: pgmap v462: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-08-28T19:25:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.776+0000 7fe79d427640 1 -- 172.21.15.16:0/1632476380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe798108ab0 msgr2=0x7fe79810f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.776+0000 7fe79d427640 1 --2- 172.21.15.16:0/1632476380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe798108ab0 0x7fe79810f2e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe790009ca0 tx=0x7fe79002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.776+0000 7fe79d427640 1 -- 172.21.15.16:0/1632476380 shutdown_connections 2024-08-28T19:25:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.776+0000 7fe79d427640 1 --2- 172.21.15.16:0/1632476380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe798108ab0 0x7fe79810f2e0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.776+0000 7fe79d427640 1 --2- 172.21.15.16:0/1632476380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe798108160 0x7fe798108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.776+0000 7fe79d427640 1 --2- 172.21.15.16:0/1632476380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe798102820 0x7fe798102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.776+0000 7fe79d427640 1 -- 172.21.15.16:0/1632476380 >> 172.21.15.16:0/1632476380 conn(0x7fe7980794e0 msgr2=0x7fe7980798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.776+0000 7fe79d427640 1 -- 172.21.15.16:0/1632476380 shutdown_connections 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.776+0000 7fe79d427640 1 -- 172.21.15.16:0/1632476380 wait complete. 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.777+0000 7fe79d427640 1 Processor -- start 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.777+0000 7fe79d427640 1 -- start start 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe79d427640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe798102820 0x7fe79810fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe79d427640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe798108160 0x7fe7981102b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe79d427640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe798108ab0 0x7fe7981107f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe79d427640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe798113840 con 0x7fe798108160 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe79d427640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7981136c0 con 0x7fe798108ab0 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe79d427640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe7981139c0 con 0x7fe798102820 2024-08-28T19:25:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe7967fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe798108160 0x7fe7981102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe7967fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe798108160 0x7fe7981102b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41712/0 (socket says 172.21.15.16:41712) 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe7967fc640 1 -- 172.21.15.16:0/2760018134 learned_addr learned my addr 172.21.15.16:0/2760018134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe7977fe640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe798108ab0 0x7fe7981107f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe796ffd640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe798102820 0x7fe79810fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe7967fc640 1 -- 172.21.15.16:0/2760018134 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe798102820 msgr2=0x7fe79810fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe7967fc640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe798102820 0x7fe79810fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe7967fc640 1 -- 172.21.15.16:0/2760018134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe798108ab0 msgr2=0x7fe7981107f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe7967fc640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe798108ab0 0x7fe7981107f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe7967fc640 1 -- 172.21.15.16:0/2760018134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7981a8550 con 0x7fe798108160 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe7967fc640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe798108160 0x7fe7981102b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe78000d770 tx=0x7fe78000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.778+0000 7fe78ffff640 1 -- 172.21.15.16:0/2760018134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780017070 con 0x7fe798108160 2024-08-28T19:25:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.779+0000 7fe79d427640 1 -- 172.21.15.16:0/2760018134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7981a8840 con 0x7fe798108160 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.779+0000 7fe78ffff640 1 -- 172.21.15.16:0/2760018134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe780009c30 con 0x7fe798108160 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.779+0000 7fe78ffff640 1 -- 172.21.15.16:0/2760018134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78000f620 con 0x7fe798108160 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.779+0000 7fe79d427640 1 -- 172.21.15.16:0/2760018134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7981a8d80 con 0x7fe798108160 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.781+0000 7fe78ffff640 1 -- 172.21.15.16:0/2760018134 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe78001a070 con 0x7fe798108160 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.781+0000 7fe7a399c740 1 Processor -- start 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.781+0000 7fe7a399c740 1 -- start start 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac11f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd511b0 0x558aac11f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd59390 0x558aac11fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7a399c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558aac096680 con 0x558aabd7a070 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7a399c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558aac085c10 con 0x558aabd511b0 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7a399c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558aac089bc0 con 0x558aabd59390 2024-08-28T19:25:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac11f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac11f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41720/0 (socket says 172.21.15.16:41720) 2024-08-28T19:25:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7977fe640 1 -- 172.21.15.16:0/2795844690 learned_addr learned my addr 172.21.15.16:0/2795844690 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe796ffd640 1 --2- 172.21.15.16:0/2795844690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd511b0 0x558aac11f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe78ffff640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe7840b4ca0 0x7fe7840b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7967fc640 1 --2- 172.21.15.16:0/2795844690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd59390 0x558aac11fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7977fe640 1 -- 172.21.15.16:0/2795844690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd59390 msgr2=0x558aac11fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7977fe640 1 --2- 172.21.15.16:0/2795844690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd59390 0x558aac11fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.782+0000 7fe7977fe640 1 -- 172.21.15.16:0/2795844690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd511b0 msgr2=0x558aac11f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.783+0000 7fe7977fe640 1 --2- 172.21.15.16:0/2795844690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd511b0 0x558aac11f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.783+0000 7fe7977fe640 1 -- 172.21.15.16:0/2795844690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aabde5650 con 0x558aabd7a070 2024-08-28T19:25:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.783+0000 7fe796ffd640 1 --2- 172.21.15.16:0/2795844690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd511b0 0x558aac11f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.783+0000 7fe78ffff640 1 -- 172.21.15.16:0/2760018134 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe7800e5370 con 0x7fe798108160 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.783+0000 7fe7977fe640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe7840b4ca0 0x7fe7840b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.783+0000 7fe7977fe640 1 --2- 172.21.15.16:0/2795844690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac11f310 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe790009820 tx=0x7fe790009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1180efe60bdeef server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.783+0000 7fe78d7fa640 1 -- 172.21.15.16:0/2795844690 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe790042040 con 0x558aabd7a070 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.783+0000 7fe78d7fa640 1 -- 172.21.15.16:0/2795844690 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe790036070 con 0x558aabd7a070 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.783+0000 7fe78d7fa640 1 -- 172.21.15.16:0/2795844690 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79002f660 con 0x558aabd7a070 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.784+0000 7fe7a399c740 1 -- 172.21.15.16:0/2795844690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 msgr2=0x558aac11f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.784+0000 7fe7a399c740 1 --2- 172.21.15.16:0/2795844690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac11f310 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe790009820 tx=0x7fe790009850 comp rx=0 tx=0).stop 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.784+0000 7fe7977fe640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe7840b4ca0 0x7fe7840b71b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe79004d000 tx=0x7fe79004e000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.784+0000 7fe7a399c740 1 -- 172.21.15.16:0/2795844690 shutdown_connections 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.784+0000 7fe7a399c740 1 --2- 172.21.15.16:0/2795844690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd59390 0x558aac11fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.784+0000 7fe7a399c740 1 --2- 172.21.15.16:0/2795844690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd511b0 0x558aac11f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.784+0000 7fe7a399c740 1 --2- 172.21.15.16:0/2795844690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac11f310 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.784+0000 7fe7a399c740 1 -- 172.21.15.16:0/2795844690 >> 172.21.15.16:0/2795844690 conn(0x558aabd6e460 msgr2=0x558aac11d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.784+0000 7fe7a399c740 1 -- 172.21.15.16:0/2795844690 shutdown_connections 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.784+0000 7fe7a399c740 1 -- 172.21.15.16:0/2795844690 wait complete. 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7a399c740 1 -- start start 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7a399c740 1 Processor -- start 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd511b0 0x558aac12aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd59390 0x558aac12b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd7a070 0x558aac136de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7a399c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558aabee8700 con 0x558aabd59390 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7a399c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558aac00c750 con 0x558aabd7a070 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7a399c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558aac12daf0 con 0x558aabd511b0 2024-08-28T19:25:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7a399c740 10 client.0 _ll_register_callbacks cb 0x558aac03be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7967fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd59390 0x558aac12b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7967fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd59390 0x558aac12b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41726/0 (socket says 172.21.15.16:41726) 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7967fc640 1 -- 172.21.15.16:0/3563208340 learned_addr learned my addr 172.21.15.16:0/3563208340 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7977fe640 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd7a070 0x558aac136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe796ffd640 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd511b0 0x558aac12aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7967fc640 1 -- 172.21.15.16:0/3563208340 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd511b0 msgr2=0x558aac12aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7967fc640 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd511b0 0x558aac12aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7967fc640 1 -- 172.21.15.16:0/3563208340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd7a070 msgr2=0x558aac136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.785+0000 7fe7967fc640 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd7a070 0x558aac136de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.786+0000 7fe7967fc640 1 -- 172.21.15.16:0/3563208340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aac045900 con 0x558aabd59390 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.786+0000 7fe7967fc640 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd59390 0x558aac12b030 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe780009610 tx=0x7fe7800748b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b8592ddcef18e3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.786+0000 7fe77e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.786+0000 7fe77e7fc640 1 -- 172.21.15.16:0/3563208340 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780021050 con 0x558aabd59390 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.786+0000 7fe77e7fc640 1 -- 172.21.15.16:0/3563208340 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe7800a7d30 con 0x558aabd59390 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.786+0000 7fe7a399c740 1 -- 172.21.15.16:0/3563208340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe790036070 con 0x558aabd59390 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.786+0000 7fe77e7fc640 1 -- 172.21.15.16:0/3563208340 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe7800a5e60 con 0x558aabd59390 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.786+0000 7fe7a399c740 10 client.16695 fetch_fsmap learned FSMap version 37 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.786+0000 7fe7a399c740 1 -- 172.21.15.16:0/3563208340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558aac1398d0 con 0x558aabd59390 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.786+0000 7fe77e7fc640 1 -- 172.21.15.16:0/3563208340 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe790036070 con 0x558aabd59390 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.786+0000 7fe77e7fc640 1 -- 172.21.15.16:0/3563208340 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) ==== 8390+0+0 (secure 0 0 0) 0x7fe78005b1b0 con 0x558aabd59390 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.787+0000 7fe7a399c740 10 client.16695 fetch_fsmap finished waiting for FSMap version 37 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.787+0000 7fe7a399c740 10 client.16695 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.787+0000 7fe7a399c740 20 client.16695 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.787+0000 7fe7a399c740 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558aabd389f0 0x558aac13cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.787+0000 7fe7a399c740 4 client.16695 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.787+0000 7fe7a399c740 1 -- 172.21.15.16:0/3563208340 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558aac0985d0 con 0x558aabd389f0 2024-08-28T19:25:09.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.787+0000 7fe796ffd640 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558aabd389f0 0x558aac13cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.787+0000 7fe77e7fc640 10 client.16695 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:25:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.787+0000 7fe796ffd640 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558aabd389f0 0x558aac13cda0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dc01e5f4113c0857 server_cookie=1b4997938bd6a8c7 in_seq=0 out_seq=0 2024-08-28T19:25:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe77e7fc640 1 -- 172.21.15.16:0/3563208340 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558aac0985d0 con 0x558aabd389f0 2024-08-28T19:25:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe77e7fc640 10 client.16695 handle_command_reply: tid=0 2024-08-28T19:25:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe7a399c740 1 client.16695 shutdown 2024-08-28T19:25:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe7a399c740 1 -- 172.21.15.16:0/3563208340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd59390 msgr2=0x558aac12b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe7a399c740 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd59390 0x558aac12b030 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe780009610 tx=0x7fe7800748b0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe7a399c740 1 -- 172.21.15.16:0/3563208340 shutdown_connections 2024-08-28T19:25:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe7a399c740 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558aabd389f0 0x558aac13cda0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe7a399c740 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd7a070 0x558aac136de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe7a399c740 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd59390 0x558aac12b030 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe7a399c740 1 --2- 172.21.15.16:0/3563208340 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd511b0 0x558aac12aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe7a399c740 1 -- 172.21.15.16:0/3563208340 >> 172.21.15.16:0/3563208340 conn(0x558aabd6e460 msgr2=0x558aac1224f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.788+0000 7fe77e7fc640 0 client.16695 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:25:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.789+0000 7fe7a399c740 1 -- 172.21.15.16:0/3563208340 shutdown_connections 2024-08-28T19:25:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.789+0000 7fe7a399c740 1 -- 172.21.15.16:0/3563208340 wait complete. 2024-08-28T19:25:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.789+0000 7fe7a399c740 20 client.16695 trim_cache size 0 max 16384 2024-08-28T19:25:09.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.802+0000 7fe7a399c740 1 Processor -- start 2024-08-28T19:25:09.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.802+0000 7fe7a399c740 1 -- start start 2024-08-28T19:25:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.802+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd511b0 0x558aac11d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.802+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd59390 0x558aac136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.802+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac137220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.802+0000 7fe7a399c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe7981b2e20 con 0x558aabd7a070 2024-08-28T19:25:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7a399c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7981b4a50 con 0x558aabd59390 2024-08-28T19:25:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7a399c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe7981b3550 con 0x558aabd511b0 2024-08-28T19:25:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac137220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41736/0 (socket says 172.21.15.16:41736) 2024-08-28T19:25:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7977fe640 1 -- 172.21.15.16:0/3279021858 learned_addr learned my addr 172.21.15.16:0/3279021858 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe796ffd640 1 --2- 172.21.15.16:0/3279021858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd511b0 0x558aac11d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7967fc640 1 --2- 172.21.15.16:0/3279021858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd59390 0x558aac136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7977fe640 1 -- 172.21.15.16:0/3279021858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd511b0 msgr2=0x558aac11d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7977fe640 1 --2- 172.21.15.16:0/3279021858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd511b0 0x558aac11d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7977fe640 1 -- 172.21.15.16:0/3279021858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd59390 msgr2=0x558aac136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7977fe640 1 --2- 172.21.15.16:0/3279021858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd59390 0x558aac136ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7977fe640 1 -- 172.21.15.16:0/3279021858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aac1173d0 con 0x558aabd7a070 2024-08-28T19:25:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe7977fe640 1 --2- 172.21.15.16:0/3279021858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac137220 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe790009a60 tx=0x7fe790003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68140b5f579e9976 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe78cff9640 1 -- 172.21.15.16:0/3279021858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe790042030 con 0x558aabd7a070 2024-08-28T19:25:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe78cff9640 1 -- 172.21.15.16:0/3279021858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe790036050 con 0x558aabd7a070 2024-08-28T19:25:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.803+0000 7fe78cff9640 1 -- 172.21.15.16:0/3279021858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe790003b70 con 0x558aabd7a070 2024-08-28T19:25:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.804+0000 7fe7a399c740 1 -- 172.21.15.16:0/3279021858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 msgr2=0x558aac137220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.804+0000 7fe7a399c740 1 --2- 172.21.15.16:0/3279021858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac137220 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe790009a60 tx=0x7fe790003f20 comp rx=0 tx=0).stop 2024-08-28T19:25:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.804+0000 7fe7a399c740 1 -- 172.21.15.16:0/3279021858 shutdown_connections 2024-08-28T19:25:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.804+0000 7fe7a399c740 1 --2- 172.21.15.16:0/3279021858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd7a070 0x558aac137220 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.804+0000 7fe7a399c740 1 --2- 172.21.15.16:0/3279021858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd59390 0x558aac136ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.804+0000 7fe7a399c740 1 --2- 172.21.15.16:0/3279021858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd511b0 0x558aac11d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.804+0000 7fe7a399c740 1 -- 172.21.15.16:0/3279021858 >> 172.21.15.16:0/3279021858 conn(0x558aabd6e460 msgr2=0x558aac11cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.804+0000 7fe7a399c740 1 -- 172.21.15.16:0/3279021858 shutdown_connections 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.804+0000 7fe7a399c740 1 -- 172.21.15.16:0/3279021858 wait complete. 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.805+0000 7fe7a399c740 1 -- start start 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.805+0000 7fe7a399c740 1 Processor -- start 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.805+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd511b0 0x558aac12f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.805+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd59390 0x558aac12fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.805+0000 7fe7a399c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd7a070 0x558aac1a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.805+0000 7fe7a399c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe7981b4350 con 0x558aabd511b0 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.805+0000 7fe7a399c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7981b3c80 con 0x558aabd7a070 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.805+0000 7fe7a399c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe7981b0e00 con 0x558aabd59390 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.805+0000 7fe7a399c740 10 client.0 _ll_register_callbacks cb 0x7fe78000d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.805+0000 7fe796ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd511b0 0x558aac12f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.805+0000 7fe796ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd511b0 0x558aac12f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41742/0 (socket says 172.21.15.16:41742) 2024-08-28T19:25:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe796ffd640 1 -- 172.21.15.16:0/1804613451 learned_addr learned my addr 172.21.15.16:0/1804613451 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe7977fe640 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd7a070 0x558aac1a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe7967fc640 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd59390 0x558aac12fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe796ffd640 1 -- 172.21.15.16:0/1804613451 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd59390 msgr2=0x558aac12fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe796ffd640 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd59390 0x558aac12fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe796ffd640 1 -- 172.21.15.16:0/1804613451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd7a070 msgr2=0x558aac1a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe796ffd640 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd7a070 0x558aac1a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe796ffd640 1 -- 172.21.15.16:0/1804613451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aac1a0cb0 con 0x558aabd511b0 2024-08-28T19:25:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe796ffd640 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd511b0 0x558aac12f770 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe788000c00 tx=0x7fe7880039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4aa5b2185e3d667 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe77e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe77e7fc640 1 -- 172.21.15.16:0/1804613451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78802a060 con 0x558aabd511b0 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe77e7fc640 1 -- 172.21.15.16:0/1804613451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fe788010840 con 0x558aabd511b0 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe7a399c740 1 -- 172.21.15.16:0/1804613451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe790036050 con 0x558aabd511b0 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe77e7fc640 1 -- 172.21.15.16:0/1804613451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe788031050 con 0x558aabd511b0 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe7a399c740 10 client.16701 fetch_fsmap learned FSMap version 37 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.806+0000 7fe7a399c740 1 -- 172.21.15.16:0/1804613451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558aac1a8110 con 0x558aabd511b0 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.807+0000 7fe77e7fc640 1 -- 172.21.15.16:0/1804613451 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe790036050 con 0x558aabd511b0 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.807+0000 7fe77e7fc640 1 -- 172.21.15.16:0/1804613451 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) ==== 8390+0+0 (secure 0 0 0) 0x7fe788025bc0 con 0x558aabd511b0 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.807+0000 7fe7a399c740 10 client.16701 fetch_fsmap finished waiting for FSMap version 37 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.807+0000 7fe7a399c740 10 client.16701 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.807+0000 7fe7a399c740 20 client.16701 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.807+0000 7fe7a399c740 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558aabd0dda0 0x558aac1aa910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.807+0000 7fe7a399c740 4 client.16701 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.807+0000 7fe7a399c740 1 -- 172.21.15.16:0/1804613451 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558aac094070 con 0x558aabd0dda0 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.807+0000 7fe7967fc640 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558aabd0dda0 0x558aac1aa910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7967fc640 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558aabd0dda0 0x558aac1aa910 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5986a113c5592e36 server_cookie=42a34630d01e6678 in_seq=0 out_seq=0 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe77e7fc640 10 client.16701 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe77e7fc640 1 -- 172.21.15.16:0/1804613451 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558aac094070 con 0x558aabd0dda0 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe77e7fc640 10 client.16701 handle_command_reply: tid=0 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 1 client.16701 shutdown 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 1 -- 172.21.15.16:0/1804613451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd511b0 msgr2=0x558aac12f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd511b0 0x558aac12f770 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe788000c00 tx=0x7fe7880039c0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 1 -- 172.21.15.16:0/1804613451 shutdown_connections 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x558aabd0dda0 0x558aac1aa910 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aabd7a070 0x558aac1a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558aabd59390 0x558aac12fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 1 --2- 172.21.15.16:0/1804613451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aabd511b0 0x558aac12f770 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 1 -- 172.21.15.16:0/1804613451 >> 172.21.15.16:0/1804613451 conn(0x558aabd6e460 msgr2=0x558aac122410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:09.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe77e7fc640 0 client.16701 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:25:09.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 1 -- 172.21.15.16:0/1804613451 shutdown_connections 2024-08-28T19:25:09.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 1 -- 172.21.15.16:0/1804613451 wait complete. 2024-08-28T19:25:09.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.808+0000 7fe7a399c740 20 client.16701 trim_cache size 0 max 16384 2024-08-28T19:25:09.815 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:25:09.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 -- 172.21.15.16:0/2760018134 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe7840b4ca0 msgr2=0x7fe7840b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:09.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe7840b4ca0 0x7fe7840b71b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe79004d000 tx=0x7fe79004e000 comp rx=0 tx=0).stop 2024-08-28T19:25:09.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 -- 172.21.15.16:0/2760018134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe798108160 msgr2=0x7fe7981102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:09.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe798108160 0x7fe7981102b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe78000d770 tx=0x7fe78000dc30 comp rx=0 tx=0).stop 2024-08-28T19:25:09.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 -- 172.21.15.16:0/2760018134 shutdown_connections 2024-08-28T19:25:09.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe7840b4ca0 0x7fe7840b71b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe798108ab0 0x7fe7981107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe798108160 0x7fe7981102b0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 --2- 172.21.15.16:0/2760018134 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe798102820 0x7fe79810fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:09.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 -- 172.21.15.16:0/2760018134 >> 172.21.15.16:0/2760018134 conn(0x7fe7980794e0 msgr2=0x7fe7981050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:09.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 -- 172.21.15.16:0/2760018134 shutdown_connections 2024-08-28T19:25:09.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:09.809+0000 7fe79d427640 1 -- 172.21.15.16:0/2760018134 wait complete. 2024-08-28T19:25:09.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 4.15 MB/sec execute 49 sec latency 23856.186 ms 2024-08-28T19:25:10.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:25:10.547 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:25:10.804 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:10.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 4.07 MB/sec execute 50 sec latency 24856.273 ms 2024-08-28T19:25:11.189 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:10 smithi016 ceph-mon[18556]: pgmap v463: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 96 op/s 2024-08-28T19:25:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:10 smithi028 ceph-mon[21769]: pgmap v463: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 96 op/s 2024-08-28T19:25:11.359 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:10 smithi115 ceph-mon[22184]: pgmap v463: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 96 op/s 2024-08-28T19:25:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.369+0000 7f7bec34f640 1 -- 172.21.15.16:0/207159601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be4102800 msgr2=0x7f7be4102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.369+0000 7f7bec34f640 1 --2- 172.21.15.16:0/207159601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be4102800 0x7f7be4102c10 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009900 tx=0x7f7be002a410 comp rx=0 tx=0).stop 2024-08-28T19:25:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.369+0000 7f7bec34f640 1 -- 172.21.15.16:0/207159601 shutdown_connections 2024-08-28T19:25:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.369+0000 7f7bec34f640 1 --2- 172.21.15.16:0/207159601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7be4108a90 0x7f7be410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.369+0000 7f7bec34f640 1 --2- 172.21.15.16:0/207159601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be4108140 0x7f7be4108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.369+0000 7f7bec34f640 1 --2- 172.21.15.16:0/207159601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be4102800 0x7f7be4102c10 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.369+0000 7f7bec34f640 1 -- 172.21.15.16:0/207159601 >> 172.21.15.16:0/207159601 conn(0x7f7be40794d0 msgr2=0x7f7be40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.369+0000 7f7bec34f640 1 -- 172.21.15.16:0/207159601 shutdown_connections 2024-08-28T19:25:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.369+0000 7f7bec34f640 1 -- 172.21.15.16:0/207159601 wait complete. 2024-08-28T19:25:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.370+0000 7f7bec34f640 1 Processor -- start 2024-08-28T19:25:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.370+0000 7f7bec34f640 1 -- start start 2024-08-28T19:25:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.370+0000 7f7bec34f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be4102800 0x7f7be4103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.370+0000 7f7bec34f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7be4108140 0x7f7be4104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7bec34f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be4108a90 0x7f7be4104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7bec34f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7be41126f0 con 0x7f7be4102800 2024-08-28T19:25:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7bec34f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7be4112570 con 0x7f7be4108140 2024-08-28T19:25:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7bec34f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7be4112870 con 0x7f7be4108a90 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7bea0eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be4102800 0x7f7be4103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7bea0eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be4102800 0x7f7be4103ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41766/0 (socket says 172.21.15.16:41766) 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7bea0eb640 1 -- 172.21.15.16:0/3228411623 learned_addr learned my addr 172.21.15.16:0/3228411623 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7be98ea640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7be4108140 0x7f7be4104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7bea8ec640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be4108a90 0x7f7be4104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7be98ea640 1 -- 172.21.15.16:0/3228411623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be4108a90 msgr2=0x7f7be4104a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7be98ea640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be4108a90 0x7f7be4104a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7be98ea640 1 -- 172.21.15.16:0/3228411623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be4102800 msgr2=0x7f7be4103ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7be98ea640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be4102800 0x7f7be4103ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7be98ea640 1 -- 172.21.15.16:0/3228411623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be41a84d0 con 0x7f7be4108140 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.371+0000 7f7bea0eb640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be4102800 0x7f7be4103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.372+0000 7f7be98ea640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7be4108140 0x7f7be4104530 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800d6a0 tx=0x7f7bd800db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:11.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.372+0000 7f7bd77fe640 1 -- 172.21.15.16:0/3228411623 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8017070 con 0x7f7be4108140 2024-08-28T19:25:11.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.372+0000 7f7bec34f640 1 -- 172.21.15.16:0/3228411623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be41a87c0 con 0x7f7be4108140 2024-08-28T19:25:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.372+0000 7f7bec34f640 1 -- 172.21.15.16:0/3228411623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be41a8d50 con 0x7f7be4108140 2024-08-28T19:25:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.372+0000 7f7bd77fe640 1 -- 172.21.15.16:0/3228411623 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bd8009820 con 0x7f7be4108140 2024-08-28T19:25:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.372+0000 7f7bd77fe640 1 -- 172.21.15.16:0/3228411623 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8004c40 con 0x7f7be4108140 2024-08-28T19:25:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.373+0000 7f7bec34f640 1 -- 172.21.15.16:0/3228411623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bc80052c0 con 0x7f7be4108140 2024-08-28T19:25:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.375+0000 7f7bd77fe640 1 -- 172.21.15.16:0/3228411623 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7bd8005220 con 0x7f7be4108140 2024-08-28T19:25:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.376+0000 7f7bd77fe640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bd00b4ca0 0x7f7bd00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.376+0000 7f7bea0eb640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bd00b4ca0 0x7f7bd00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.376+0000 7f7bd77fe640 1 -- 172.21.15.16:0/3228411623 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7bd80e5330 con 0x7f7be4108140 2024-08-28T19:25:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.377+0000 7f7bea0eb640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bd00b4ca0 0x7f7bd00b71b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009230 tx=0x7f7be00058d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:11.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.379+0000 7f7bd77fe640 1 -- 172.21.15.16:0/3228411623 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f7bd80ab700 con 0x7f7be4108140 2024-08-28T19:25:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.613+0000 7f7bec34f640 1 -- 172.21.15.16:0/3228411623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bc80055b0 con 0x7f7be4108140 2024-08-28T19:25:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.614+0000 7f7bd77fe640 1 -- 172.21.15.16:0/3228411623 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11984 (secure 0 0 0) 0x7f7bd80a5e20 con 0x7f7be4108140 2024-08-28T19:25:11.615 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:25:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":37,"btime":"2024-08-28T19:25:03:931196+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:25:03.274930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:25:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 37 2024-08-28T19:25:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 -- 172.21.15.16:0/3228411623 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bd00b4ca0 msgr2=0x7f7bd00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bd00b4ca0 0x7f7bd00b71b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009230 tx=0x7f7be00058d0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 -- 172.21.15.16:0/3228411623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7be4108140 msgr2=0x7f7be4104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7be4108140 0x7f7be4104530 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800d6a0 tx=0x7f7bd800db60 comp rx=0 tx=0).stop 2024-08-28T19:25:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 -- 172.21.15.16:0/3228411623 shutdown_connections 2024-08-28T19:25:11.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be4108a90 0x7f7be4104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bd00b4ca0 0x7f7bd00b71b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7be4108140 0x7f7be4104530 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 --2- 172.21.15.16:0/3228411623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be4102800 0x7f7be4103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 -- 172.21.15.16:0/3228411623 >> 172.21.15.16:0/3228411623 conn(0x7f7be40794d0 msgr2=0x7f7be4101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:11.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 -- 172.21.15.16:0/3228411623 shutdown_connections 2024-08-28T19:25:11.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.617+0000 7f7bec34f640 1 -- 172.21.15.16:0/3228411623 wait complete. 2024-08-28T19:25:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.841+0000 7f90eceff640 1 -- 172.21.15.16:0/3797297170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8109b90 msgr2=0x7f90e81103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.841+0000 7f90eceff640 1 --2- 172.21.15.16:0/3797297170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8109b90 0x7f90e81103c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009ca0 tx=0x7f90e002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.841+0000 7f90eceff640 1 -- 172.21.15.16:0/3797297170 shutdown_connections 2024-08-28T19:25:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.841+0000 7f90eceff640 1 --2- 172.21.15.16:0/3797297170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8109b90 0x7f90e81103c0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.841+0000 7f90eceff640 1 --2- 172.21.15.16:0/3797297170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e8109240 0x7f90e8109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.841+0000 7f90eceff640 1 --2- 172.21.15.16:0/3797297170 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f90e8103900 0x7f90e8103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.841+0000 7f90eceff640 1 -- 172.21.15.16:0/3797297170 >> 172.21.15.16:0/3797297170 conn(0x7f90e807b5f0 msgr2=0x7f90e807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.841+0000 7f90eceff640 1 -- 172.21.15.16:0/3797297170 shutdown_connections 2024-08-28T19:25:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.842+0000 7f90eceff640 1 -- 172.21.15.16:0/3797297170 wait complete. 2024-08-28T19:25:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.842+0000 7f90eceff640 1 Processor -- start 2024-08-28T19:25:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.842+0000 7f90eceff640 1 -- start start 2024-08-28T19:25:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90eceff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f90e8103900 0x7f90e80754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90eceff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8109240 0x7f90e80759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90eceff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e8109b90 0x7f90e8075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90eceff640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90e8113850 con 0x7f90e8109240 2024-08-28T19:25:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90eceff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81136d0 con 0x7f90e8109b90 2024-08-28T19:25:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90eceff640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81139d0 con 0x7f90e8103900 2024-08-28T19:25:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8109240 0x7f90e80759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8109240 0x7f90e80759f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41770/0 (socket says 172.21.15.16:41770) 2024-08-28T19:25:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90dffff640 1 -- 172.21.15.16:0/2880466845 learned_addr learned my addr 172.21.15.16:0/2880466845 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90e659c640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f90e8103900 0x7f90e80754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90e6d9d640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e8109b90 0x7f90e8075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90dffff640 1 -- 172.21.15.16:0/2880466845 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f90e8103900 msgr2=0x7f90e80754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90dffff640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f90e8103900 0x7f90e80754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90dffff640 1 -- 172.21.15.16:0/2880466845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e8109b90 msgr2=0x7f90e8075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90dffff640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e8109b90 0x7f90e8075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90dffff640 1 -- 172.21.15.16:0/2880466845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e81a84a0 con 0x7f90e8109240 2024-08-28T19:25:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90dffff640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8109240 0x7f90e80759f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f90d000d740 tx=0x7f90d000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90df7fe640 1 -- 172.21.15.16:0/2880466845 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0017070 con 0x7f90e8109240 2024-08-28T19:25:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90eceff640 1 -- 172.21.15.16:0/2880466845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e81a8790 con 0x7f90e8109240 2024-08-28T19:25:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.843+0000 7f90df7fe640 1 -- 172.21.15.16:0/2880466845 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90d000de50 con 0x7f90e8109240 2024-08-28T19:25:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.844+0000 7f90df7fe640 1 -- 172.21.15.16:0/2880466845 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d000f620 con 0x7f90e8109240 2024-08-28T19:25:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.844+0000 7f90eceff640 1 -- 172.21.15.16:0/2880466845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e81a8cd0 con 0x7f90e8109240 2024-08-28T19:25:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.846+0000 7f90df7fe640 1 -- 172.21.15.16:0/2880466845 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f90d001a070 con 0x7f90e8109240 2024-08-28T19:25:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.846+0000 7f90f3474740 1 Processor -- start 2024-08-28T19:25:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.846+0000 7f90f3474740 1 -- start start 2024-08-28T19:25:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90f3474740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cf17070 0x560d9d2bc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90f3474740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9ceee1b0 0x560d9d2bc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90f3474740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2bcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90f3474740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560d9d233680 con 0x560d9cef6390 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90f3474740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d9d222c10 con 0x560d9cf17070 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90f3474740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560d9d226bc0 con 0x560d9ceee1b0 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2bcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2bcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41784/0 (socket says 172.21.15.16:41784) 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90dffff640 1 -- 172.21.15.16:0/604555003 learned_addr learned my addr 172.21.15.16:0/604555003 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90df7fe640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f90d40b4ca0 0x7f90d40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90e659c640 1 --2- 172.21.15.16:0/604555003 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9ceee1b0 0x560d9d2bc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90e6d9d640 1 --2- 172.21.15.16:0/604555003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cf17070 0x560d9d2bc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90e6d9d640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f90d40b4ca0 0x7f90d40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90dffff640 1 -- 172.21.15.16:0/604555003 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9ceee1b0 msgr2=0x560d9d2bc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90dffff640 1 --2- 172.21.15.16:0/604555003 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9ceee1b0 0x560d9d2bc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90dffff640 1 -- 172.21.15.16:0/604555003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cf17070 msgr2=0x560d9d2bc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90dffff640 1 --2- 172.21.15.16:0/604555003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cf17070 0x560d9d2bc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90dffff640 1 -- 172.21.15.16:0/604555003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d9cf82650 con 0x560d9cef6390 2024-08-28T19:25:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.847+0000 7f90e659c640 1 --2- 172.21.15.16:0/604555003 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9ceee1b0 0x560d9d2bc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.848+0000 7f90df7fe640 1 -- 172.21.15.16:0/2880466845 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f90d00e5280 con 0x7f90e8109240 2024-08-28T19:25:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.848+0000 7f90dffff640 1 --2- 172.21.15.16:0/604555003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2bcd90 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f90d00a7730 tx=0x7f90d0005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e730f2ec6078bb7f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.848+0000 7f90e6d9d640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f90d40b4ca0 0x7f90d40b71b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009a60 tx=0x7f90e00095c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.848+0000 7f90dcff9640 1 -- 172.21.15.16:0/604555003 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0021020 con 0x560d9cef6390 2024-08-28T19:25:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.848+0000 7f90dcff9640 1 -- 172.21.15.16:0/604555003 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90d00ab1c0 con 0x560d9cef6390 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.848+0000 7f90dcff9640 1 -- 172.21.15.16:0/604555003 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d00e28e0 con 0x560d9cef6390 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.849+0000 7f90f3474740 1 -- 172.21.15.16:0/604555003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 msgr2=0x560d9d2bcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.849+0000 7f90f3474740 1 --2- 172.21.15.16:0/604555003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2bcd90 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f90d00a7730 tx=0x7f90d0005cc0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.849+0000 7f90f3474740 1 -- 172.21.15.16:0/604555003 shutdown_connections 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.849+0000 7f90f3474740 1 --2- 172.21.15.16:0/604555003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2bcd90 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.849+0000 7f90f3474740 1 --2- 172.21.15.16:0/604555003 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9ceee1b0 0x560d9d2bc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.849+0000 7f90f3474740 1 --2- 172.21.15.16:0/604555003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cf17070 0x560d9d2bc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.849+0000 7f90f3474740 1 -- 172.21.15.16:0/604555003 >> 172.21.15.16:0/604555003 conn(0x560d9cf0b460 msgr2=0x560d9d2ba200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.849+0000 7f90f3474740 1 -- 172.21.15.16:0/604555003 shutdown_connections 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.849+0000 7f90f3474740 1 -- 172.21.15.16:0/604555003 wait complete. 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.849+0000 7f90f3474740 1 -- start start 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.849+0000 7f90f3474740 1 Processor -- start 2024-08-28T19:25:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90f3474740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90f3474740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cef6390 0x560d9d2c8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90f3474740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cf17070 0x560d9d2d3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90f3474740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560d9d085700 con 0x560d9ceee1b0 2024-08-28T19:25:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90f3474740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d9d1a9750 con 0x560d9cef6390 2024-08-28T19:25:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90f3474740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560d9d2caa30 con 0x560d9cf17070 2024-08-28T19:25:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90f3474740 10 client.0 _ll_register_callbacks cb 0x560d9d1d8e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90e659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90e659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2c7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41794/0 (socket says 172.21.15.16:41794) 2024-08-28T19:25:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90e659c640 1 -- 172.21.15.16:0/628788259 learned_addr learned my addr 172.21.15.16:0/628788259 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90e6d9d640 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cf17070 0x560d9d2d3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90dffff640 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cef6390 0x560d9d2c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90e659c640 1 -- 172.21.15.16:0/628788259 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cf17070 msgr2=0x560d9d2d3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90e659c640 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cf17070 0x560d9d2d3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90e659c640 1 -- 172.21.15.16:0/628788259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cef6390 msgr2=0x560d9d2c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90e659c640 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cef6390 0x560d9d2c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.850+0000 7f90e659c640 1 -- 172.21.15.16:0/628788259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d9d1e2900 con 0x560d9ceee1b0 2024-08-28T19:25:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90e659c640 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2c7ac0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f90d8010820 tx=0x7f90d8010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af00b87f74a1540e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90cdffb640 1 -- 172.21.15.16:0/628788259 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d800fc50 con 0x560d9ceee1b0 2024-08-28T19:25:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90cdffb640 1 -- 172.21.15.16:0/628788259 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f90d800fdf0 con 0x560d9ceee1b0 2024-08-28T19:25:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90f3474740 1 -- 172.21.15.16:0/628788259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90d00ab1c0 con 0x560d9ceee1b0 2024-08-28T19:25:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90cdffb640 1 -- 172.21.15.16:0/628788259 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f90d8024050 con 0x560d9ceee1b0 2024-08-28T19:25:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90f3474740 10 client.16719 fetch_fsmap learned FSMap version 37 2024-08-28T19:25:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90f3474740 1 -- 172.21.15.16:0/628788259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560d9d2d68e0 con 0x560d9ceee1b0 2024-08-28T19:25:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90cdffb640 1 -- 172.21.15.16:0/628788259 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90d00ab1c0 con 0x560d9ceee1b0 2024-08-28T19:25:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90cdffb640 1 -- 172.21.15.16:0/628788259 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) ==== 8390+0+0 (secure 0 0 0) 0x7f90d800e4d0 con 0x560d9ceee1b0 2024-08-28T19:25:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90f3474740 10 client.16719 fetch_fsmap finished waiting for FSMap version 37 2024-08-28T19:25:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90f3474740 10 client.16719 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:25:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90f3474740 20 client.16719 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90f3474740 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x560d9ced59f0 0x560d9d2d9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90f3474740 4 client.16719 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.851+0000 7f90f3474740 1 -- 172.21.15.16:0/628788259 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d9d2355d0 con 0x560d9ced59f0 2024-08-28T19:25:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.852+0000 7f90dffff640 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x560d9ced59f0 0x560d9d2d9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.852+0000 7f90cdffb640 10 client.16719 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:25:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.852+0000 7f90dffff640 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x560d9ced59f0 0x560d9d2d9e40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=50b2718aa3b1d68 server_cookie=3263950b060e6d2a in_seq=0 out_seq=0 2024-08-28T19:25:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.853+0000 7f90cdffb640 1 -- 172.21.15.16:0/628788259 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560d9d2355d0 con 0x560d9ced59f0 2024-08-28T19:25:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.853+0000 7f90cdffb640 10 client.16719 handle_command_reply: tid=0 2024-08-28T19:25:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.853+0000 7f90f3474740 1 client.16719 shutdown 2024-08-28T19:25:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90f3474740 1 -- 172.21.15.16:0/628788259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 msgr2=0x560d9d2c7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90f3474740 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2c7ac0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f90d8010820 tx=0x7f90d8010ce0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90f3474740 1 -- 172.21.15.16:0/628788259 shutdown_connections 2024-08-28T19:25:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90f3474740 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x560d9ced59f0 0x560d9d2d9e40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90f3474740 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cf17070 0x560d9d2d3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90f3474740 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cef6390 0x560d9d2c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90cdffb640 0 client.16719 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:25:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90f3474740 1 --2- 172.21.15.16:0/628788259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2c7ac0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90f3474740 1 -- 172.21.15.16:0/628788259 >> 172.21.15.16:0/628788259 conn(0x560d9cf0b460 msgr2=0x560d9d2bd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90f3474740 1 -- 172.21.15.16:0/628788259 shutdown_connections 2024-08-28T19:25:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90f3474740 1 -- 172.21.15.16:0/628788259 wait complete. 2024-08-28T19:25:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.854+0000 7f90f3474740 20 client.16719 trim_cache size 0 max 16384 2024-08-28T19:25:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90f3474740 1 Processor -- start 2024-08-28T19:25:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90f3474740 1 -- start start 2024-08-28T19:25:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90f3474740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9ceee1b0 0x560d9d2c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90f3474740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2d3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90f3474740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cf17070 0x560d9d2d4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90f3474740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81b2d60 con 0x560d9cef6390 2024-08-28T19:25:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90f3474740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81b4990 con 0x560d9ceee1b0 2024-08-28T19:25:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90f3474740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81b3490 con 0x560d9cf17070 2024-08-28T19:25:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2d3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2d3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41808/0 (socket says 172.21.15.16:41808) 2024-08-28T19:25:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90dffff640 1 -- 172.21.15.16:0/2649352947 learned_addr learned my addr 172.21.15.16:0/2649352947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90e659c640 1 --2- 172.21.15.16:0/2649352947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9ceee1b0 0x560d9d2c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.869+0000 7f90e6d9d640 1 --2- 172.21.15.16:0/2649352947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cf17070 0x560d9d2d4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.870+0000 7f90e659c640 1 -- 172.21.15.16:0/2649352947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cf17070 msgr2=0x560d9d2d4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.870+0000 7f90e659c640 1 --2- 172.21.15.16:0/2649352947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cf17070 0x560d9d2d4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.870+0000 7f90e659c640 1 -- 172.21.15.16:0/2649352947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 msgr2=0x560d9d2d3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.870+0000 7f90e659c640 1 --2- 172.21.15.16:0/2649352947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2d3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.870+0000 7f90e659c640 1 -- 172.21.15.16:0/2649352947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d9d2c5fc0 con 0x560d9ceee1b0 2024-08-28T19:25:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.870+0000 7f90dffff640 1 --2- 172.21.15.16:0/2649352947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2d3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.870+0000 7f90e659c640 1 --2- 172.21.15.16:0/2649352947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9ceee1b0 0x560d9d2c7f80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f90d80058c0 tx=0x7f90d800f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aa2d2e504fc822b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.870+0000 7f90cffff640 1 -- 172.21.15.16:0/2649352947 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d802d070 con 0x560d9ceee1b0 2024-08-28T19:25:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.870+0000 7f90cffff640 1 -- 172.21.15.16:0/2649352947 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90d800eb30 con 0x560d9ceee1b0 2024-08-28T19:25:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.870+0000 7f90cffff640 1 -- 172.21.15.16:0/2649352947 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d80053b0 con 0x560d9ceee1b0 2024-08-28T19:25:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.871+0000 7f90f3474740 1 -- 172.21.15.16:0/2649352947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9ceee1b0 msgr2=0x560d9d2c7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.871+0000 7f90f3474740 1 --2- 172.21.15.16:0/2649352947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9ceee1b0 0x560d9d2c7f80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f90d80058c0 tx=0x7f90d800f7f0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.871+0000 7f90f3474740 1 -- 172.21.15.16:0/2649352947 shutdown_connections 2024-08-28T19:25:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.871+0000 7f90f3474740 1 --2- 172.21.15.16:0/2649352947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cf17070 0x560d9d2d4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.871+0000 7f90f3474740 1 --2- 172.21.15.16:0/2649352947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9cef6390 0x560d9d2d3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.871+0000 7f90f3474740 1 --2- 172.21.15.16:0/2649352947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9ceee1b0 0x560d9d2c7f80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.871+0000 7f90f3474740 1 -- 172.21.15.16:0/2649352947 >> 172.21.15.16:0/2649352947 conn(0x560d9cf0b460 msgr2=0x560d9d2c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.871+0000 7f90f3474740 1 -- 172.21.15.16:0/2649352947 shutdown_connections 2024-08-28T19:25:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.871+0000 7f90f3474740 1 -- 172.21.15.16:0/2649352947 wait complete. 2024-08-28T19:25:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90f3474740 1 -- start start 2024-08-28T19:25:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90f3474740 1 Processor -- start 2024-08-28T19:25:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90f3474740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90f3474740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cef6390 0x560d9d2ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90f3474740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cf17070 0x560d9d33d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90f3474740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81b4290 con 0x560d9ceee1b0 2024-08-28T19:25:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90f3474740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81b3bc0 con 0x560d9cf17070 2024-08-28T19:25:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90f3474740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81b0d40 con 0x560d9cef6390 2024-08-28T19:25:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90e659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90f3474740 10 client.0 _ll_register_callbacks cb 0x7f90d800a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90e659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41820/0 (socket says 172.21.15.16:41820) 2024-08-28T19:25:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90e659c640 1 -- 172.21.15.16:0/1272735950 learned_addr learned my addr 172.21.15.16:0/1272735950 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90dffff640 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cef6390 0x560d9d2ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.872+0000 7f90e6d9d640 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cf17070 0x560d9d33d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90e659c640 1 -- 172.21.15.16:0/1272735950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cef6390 msgr2=0x560d9d2ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90e659c640 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cef6390 0x560d9d2ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90e659c640 1 -- 172.21.15.16:0/1272735950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cf17070 msgr2=0x560d9d33d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90e659c640 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cf17070 0x560d9d33d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90e659c640 1 -- 172.21.15.16:0/1272735950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d9d33def0 con 0x560d9ceee1b0 2024-08-28T19:25:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90e6d9d640 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cf17070 0x560d9d33d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90dffff640 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cef6390 0x560d9d2ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90e659c640 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2cc5c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f90d8013040 tx=0x7f90d80056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48023d94762a1ae3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90cdffb640 1 -- 172.21.15.16:0/1272735950 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d802d040 con 0x560d9ceee1b0 2024-08-28T19:25:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90cdffb640 1 -- 172.21.15.16:0/1272735950 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f90d800fc50 con 0x560d9ceee1b0 2024-08-28T19:25:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90f3474740 1 -- 172.21.15.16:0/1272735950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90d800eb30 con 0x560d9ceee1b0 2024-08-28T19:25:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90cdffb640 1 -- 172.21.15.16:0/1272735950 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f90d800fe50 con 0x560d9ceee1b0 2024-08-28T19:25:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90f3474740 10 client.16725 fetch_fsmap learned FSMap version 37 2024-08-28T19:25:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.873+0000 7f90f3474740 1 -- 172.21.15.16:0/1272735950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560d9d345210 con 0x560d9ceee1b0 2024-08-28T19:25:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.874+0000 7f90cdffb640 1 -- 172.21.15.16:0/1272735950 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90d800eb30 con 0x560d9ceee1b0 2024-08-28T19:25:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.874+0000 7f90cdffb640 1 -- 172.21.15.16:0/1272735950 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) ==== 8390+0+0 (secure 0 0 0) 0x7f90d8030070 con 0x560d9ceee1b0 2024-08-28T19:25:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.874+0000 7f90f3474740 10 client.16725 fetch_fsmap finished waiting for FSMap version 37 2024-08-28T19:25:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.874+0000 7f90f3474740 10 client.16725 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:25:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.874+0000 7f90f3474740 20 client.16725 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.874+0000 7f90f3474740 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x560d9ceaada0 0x560d9d347880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.874+0000 7f90f3474740 4 client.16725 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:25:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.874+0000 7f90f3474740 1 -- 172.21.15.16:0/1272735950 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d9d231070 con 0x560d9ceaada0 2024-08-28T19:25:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.874+0000 7f90dffff640 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x560d9ceaada0 0x560d9d347880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90dffff640 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x560d9ceaada0 0x560d9d347880 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d7b28b3336db0521 server_cookie=9034655aff053cc4 in_seq=0 out_seq=0 2024-08-28T19:25:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90cdffb640 10 client.16725 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:25:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90cdffb640 1 -- 172.21.15.16:0/1272735950 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d9d231070 con 0x560d9ceaada0 2024-08-28T19:25:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90cdffb640 10 client.16725 handle_command_reply: tid=0 2024-08-28T19:25:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90f3474740 1 client.16725 shutdown 2024-08-28T19:25:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90f3474740 1 -- 172.21.15.16:0/1272735950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 msgr2=0x560d9d2cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90f3474740 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2cc5c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f90d8013040 tx=0x7f90d80056f0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90f3474740 1 -- 172.21.15.16:0/1272735950 shutdown_connections 2024-08-28T19:25:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90f3474740 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x560d9ceaada0 0x560d9d347880 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90cdffb640 0 client.16725 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:25:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90f3474740 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d9cf17070 0x560d9d33d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90f3474740 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560d9cef6390 0x560d9d2ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90f3474740 1 --2- 172.21.15.16:0/1272735950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d9ceee1b0 0x560d9d2cc5c0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.875+0000 7f90f3474740 1 -- 172.21.15.16:0/1272735950 >> 172.21.15.16:0/1272735950 conn(0x560d9cf0b460 msgr2=0x560d9d2c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90f3474740 1 -- 172.21.15.16:0/1272735950 shutdown_connections 2024-08-28T19:25:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90f3474740 1 -- 172.21.15.16:0/1272735950 wait complete. 2024-08-28T19:25:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90f3474740 20 client.16725 trim_cache size 0 max 16384 2024-08-28T19:25:11.883 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:25:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 -- 172.21.15.16:0/2880466845 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f90d40b4ca0 msgr2=0x7f90d40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f90d40b4ca0 0x7f90d40b71b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009a60 tx=0x7f90e00095c0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 -- 172.21.15.16:0/2880466845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8109240 msgr2=0x7f90e80759f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8109240 0x7f90e80759f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f90d000d740 tx=0x7f90d000dc00 comp rx=0 tx=0).stop 2024-08-28T19:25:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 -- 172.21.15.16:0/2880466845 shutdown_connections 2024-08-28T19:25:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f90d40b4ca0 0x7f90d40b71b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e8109b90 0x7f90e8075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8109240 0x7f90e80759f0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 --2- 172.21.15.16:0/2880466845 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f90e8103900 0x7f90e80754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 -- 172.21.15.16:0/2880466845 >> 172.21.15.16:0/2880466845 conn(0x7f90e807b5f0 msgr2=0x7f90e8107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 -- 172.21.15.16:0/2880466845 shutdown_connections 2024-08-28T19:25:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:11.876+0000 7f90eceff640 1 -- 172.21.15.16:0/2880466845 wait complete. 2024-08-28T19:25:11.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 3.99 MB/sec execute 51 sec latency 25856.366 ms 2024-08-28T19:25:12.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:11 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3228411623' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:25:12.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:11 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3228411623' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:25:12.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:11 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3228411623' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:25:12.927 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id e6f999f4 --release --await' 2024-08-28T19:25:12.927 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id e6f999f4 --release --await 2024-08-28T19:25:12.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 3.91 MB/sec execute 52 sec latency 26856.425 ms 2024-08-28T19:25:13.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:12 smithi016 ceph-mon[18556]: pgmap v464: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 203 KiB/s wr, 67 op/s 2024-08-28T19:25:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:12 smithi028 ceph-mon[21769]: pgmap v464: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 203 KiB/s wr, 67 op/s 2024-08-28T19:25:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:12 smithi115 ceph-mon[22184]: pgmap v464: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 203 KiB/s wr, 67 op/s 2024-08-28T19:25:13.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:25:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:13.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172537 3.84 MB/sec execute 53 sec latency 27856.470 ms 2024-08-28T19:25:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.059+0000 7f3c72624640 1 -- 172.21.15.16:0/4094343844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c6c109260 msgr2=0x7f3c6c109670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.059+0000 7f3c72624640 1 --2- 172.21.15.16:0/4094343844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c6c109260 0x7f3c6c109670 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009900 tx=0x7f3c5402a410 comp rx=0 tx=0).stop 2024-08-28T19:25:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.059+0000 7f3c72624640 1 -- 172.21.15.16:0/4094343844 shutdown_connections 2024-08-28T19:25:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.059+0000 7f3c72624640 1 --2- 172.21.15.16:0/4094343844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c6c109bb0 0x7f3c6c1103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.059+0000 7f3c72624640 1 --2- 172.21.15.16:0/4094343844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c6c109260 0x7f3c6c109670 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.059+0000 7f3c72624640 1 --2- 172.21.15.16:0/4094343844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c6c103920 0x7f3c6c103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.059+0000 7f3c72624640 1 -- 172.21.15.16:0/4094343844 >> 172.21.15.16:0/4094343844 conn(0x7f3c6c0ff350 msgr2=0x7f3c6c1017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.059+0000 7f3c72624640 1 -- 172.21.15.16:0/4094343844 shutdown_connections 2024-08-28T19:25:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.059+0000 7f3c72624640 1 -- 172.21.15.16:0/4094343844 wait complete. 2024-08-28T19:25:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.060+0000 7f3c72624640 1 Processor -- start 2024-08-28T19:25:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.060+0000 7f3c72624640 1 -- start start 2024-08-28T19:25:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.060+0000 7f3c72624640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c6c103920 0x7f3c6c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.060+0000 7f3c72624640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c6c109260 0x7f3c6c19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.060+0000 7f3c72624640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c6c109bb0 0x7f3c6c19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.060+0000 7f3c72624640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c113870 con 0x7f3c6c109bb0 2024-08-28T19:25:14.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.060+0000 7f3c72624640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1136f0 con 0x7f3c6c103920 2024-08-28T19:25:14.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.060+0000 7f3c72624640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1139f0 con 0x7f3c6c109260 2024-08-28T19:25:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c70bc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c6c109bb0 0x7f3c6c19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c70bc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c6c109bb0 0x7f3c6c19faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41836/0 (socket says 172.21.15.16:41836) 2024-08-28T19:25:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c70bc1640 1 -- 172.21.15.16:0/859958923 learned_addr learned my addr 172.21.15.16:0/859958923 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c6bfff640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c6c103920 0x7f3c6c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c6c109260 0x7f3c6c19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c70bc1640 1 -- 172.21.15.16:0/859958923 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c6c109260 msgr2=0x7f3c6c19a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c70bc1640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c6c109260 0x7f3c6c19a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c70bc1640 1 -- 172.21.15.16:0/859958923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c6c103920 msgr2=0x7f3c6c19a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c70bc1640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c6c103920 0x7f3c6c19a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c70bc1640 1 -- 172.21.15.16:0/859958923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6c1a01d0 con 0x7f3c6c109bb0 2024-08-28T19:25:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c70bc1640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c6c109bb0 0x7f3c6c19faa0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3c6400eb10 tx=0x7f3c6400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c697fa640 1 -- 172.21.15.16:0/859958923 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c64004b20 con 0x7f3c6c109bb0 2024-08-28T19:25:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c72624640 1 -- 172.21.15.16:0/859958923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c6c1a04c0 con 0x7f3c6c109bb0 2024-08-28T19:25:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c72624640 1 -- 172.21.15.16:0/859958923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c6c1a6c20 con 0x7f3c6c109bb0 2024-08-28T19:25:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c697fa640 1 -- 172.21.15.16:0/859958923 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c64004cc0 con 0x7f3c6c109bb0 2024-08-28T19:25:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.061+0000 7f3c697fa640 1 -- 172.21.15.16:0/859958923 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c64010670 con 0x7f3c6c109bb0 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.064+0000 7f3c78b99740 1 Processor -- start 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.065+0000 7f3c697fa640 1 -- 172.21.15.16:0/859958923 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3c64010830 con 0x7f3c6c109bb0 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.065+0000 7f3c78b99740 1 -- start start 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.066+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadd03520 0x560dae0a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.066+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcda660 0x560dae0a91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.066+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadce2840 0x560dae0a9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.066+0000 7f3c78b99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560dae01fa70 con 0x560dadd03520 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.066+0000 7f3c78b99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560dae00f000 con 0x560dadcda660 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.066+0000 7f3c78b99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560dae012fb0 con 0x560dadce2840 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.066+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadd03520 0x560dae0a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.066+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadd03520 0x560dae0a8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41840/0 (socket says 172.21.15.16:41840) 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.066+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/2559758267 learned_addr learned my addr 172.21.15.16:0/2559758267 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.067+0000 7f3c6bfff640 1 --2- 172.21.15.16:0/2559758267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcda660 0x560dae0a91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.067+0000 7f3c697fa640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c580b4ca0 0x7f3c580b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.067+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/2559758267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadce2840 msgr2=0x560dae0a9700 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:25:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.067+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/2559758267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadce2840 0x560dae0a9700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.067+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/2559758267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcda660 msgr2=0x560dae0a91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.067+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/2559758267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcda660 0x560dae0a91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.067+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/2559758267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dadd6eb00 con 0x560dadd03520 2024-08-28T19:25:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.067+0000 7f3c697fa640 1 -- 172.21.15.16:0/859958923 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3c640e5f60 con 0x7f3c6c109bb0 2024-08-28T19:25:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.067+0000 7f3c70bc1640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c580b4ca0 0x7f3c580b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.067+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/2559758267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadd03520 0x560dae0a8c80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3c54000c00 tx=0x7f3c54032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7faa2baffb8d7c12 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.068+0000 7f3c627fc640 1 -- 172.21.15.16:0/2559758267 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5403f070 con 0x560dadd03520 2024-08-28T19:25:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.069+0000 7f3c627fc640 1 -- 172.21.15.16:0/2559758267 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c54004030 con 0x560dadd03520 2024-08-28T19:25:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.069+0000 7f3c627fc640 1 -- 172.21.15.16:0/2559758267 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54035070 con 0x560dadd03520 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.070+0000 7f3c78b99740 1 -- 172.21.15.16:0/2559758267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadd03520 msgr2=0x560dae0a8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.070+0000 7f3c78b99740 1 --2- 172.21.15.16:0/2559758267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadd03520 0x560dae0a8c80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3c54000c00 tx=0x7f3c54032b20 comp rx=0 tx=0).stop 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.070+0000 7f3c70bc1640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c580b4ca0 0x7f3c580b71b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3c640e3340 tx=0x7f3c6400cf70 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.070+0000 7f3c78b99740 1 -- 172.21.15.16:0/2559758267 shutdown_connections 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.070+0000 7f3c78b99740 1 --2- 172.21.15.16:0/2559758267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadce2840 0x560dae0a9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.070+0000 7f3c78b99740 1 --2- 172.21.15.16:0/2559758267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcda660 0x560dae0a91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.070+0000 7f3c78b99740 1 --2- 172.21.15.16:0/2559758267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadd03520 0x560dae0a8c80 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.070+0000 7f3c78b99740 1 -- 172.21.15.16:0/2559758267 >> 172.21.15.16:0/2559758267 conn(0x560dadcf7910 msgr2=0x560dae0a6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.070+0000 7f3c78b99740 1 -- 172.21.15.16:0/2559758267 shutdown_connections 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.070+0000 7f3c78b99740 1 -- 172.21.15.16:0/2559758267 wait complete. 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.071+0000 7f3c78b99740 1 -- start start 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.071+0000 7f3c78b99740 1 Processor -- start 2024-08-28T19:25:14.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0b9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadd03520 0x560dae0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c78b99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560dae0bb7b0 con 0x560dadce2840 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c78b99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560dadfd0520 con 0x560dadd03520 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c78b99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560dae0bb930 con 0x560dadcda660 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c6bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0b9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c6bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0b9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41842/0 (socket says 172.21.15.16:41842) 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadd03520 0x560dae0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c6bfff640 1 -- 172.21.15.16:0/2341646135 learned_addr learned my addr 172.21.15.16:0/2341646135 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c78b99740 10 client.0 _ll_register_callbacks cb 0x560dae008c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c6bfff640 1 -- 172.21.15.16:0/2341646135 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 msgr2=0x560dae0b8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.072+0000 7f3c6bfff640 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0b8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.073+0000 7f3c6bfff640 1 -- 172.21.15.16:0/2341646135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadd03520 msgr2=0x560dae0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:14.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.073+0000 7f3c6bfff640 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadd03520 0x560dae0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.073+0000 7f3c6bfff640 1 -- 172.21.15.16:0/2341646135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dadde5960 con 0x560dadce2840 2024-08-28T19:25:14.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.073+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadd03520 0x560dae0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:14.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.073+0000 7f3c6bfff640 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0b9180 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c012a70 tx=0x7f3c5c012f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3426c544ea280dbd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:14.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.073+0000 7f3c4b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:14.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.073+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/2341646135 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c0120d0 con 0x560dadce2840 2024-08-28T19:25:14.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.073+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/2341646135 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3c5c004020 con 0x560dadce2840 2024-08-28T19:25:14.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.073+0000 7f3c78b99740 1 -- 172.21.15.16:0/2341646135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560dadfae9a0 con 0x560dadce2840 2024-08-28T19:25:14.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.074+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/2341646135 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3c5c025050 con 0x560dadce2840 2024-08-28T19:25:14.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.074+0000 7f3c78b99740 10 client.16737 fetch_fsmap learned FSMap version 38 2024-08-28T19:25:14.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.074+0000 7f3c78b99740 1 -- 172.21.15.16:0/2341646135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560dae0c7980 con 0x560dadce2840 2024-08-28T19:25:14.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.074+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/2341646135 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560dadfae9a0 con 0x560dadce2840 2024-08-28T19:25:14.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.074+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/2341646135 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 38) ==== 8386+0+0 (secure 0 0 0) 0x7f3c5c003780 con 0x560dadce2840 2024-08-28T19:25:14.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.075+0000 7f3c78b99740 10 client.16737 fetch_fsmap finished waiting for FSMap version 38 2024-08-28T19:25:14.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.075+0000 7f3c78b99740 10 client.16737 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:25:14.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.075+0000 7f3c78b99740 20 client.16737 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:14.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.075+0000 7f3c78b99740 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560dadcc1ea0 0x560dae0cae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.075+0000 7f3c78b99740 4 client.16737 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:14.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.075+0000 7f3c78b99740 1 -- 172.21.15.16:0/2341646135 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560dae0219c0 con 0x560dadcc1ea0 2024-08-28T19:25:14.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.075+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560dadcc1ea0 0x560dae0cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.075+0000 7f3c4b7fe640 10 client.16737 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:14.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.075+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560dadcc1ea0 0x560dae0cae70 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18e91dd4fda6f9c6 server_cookie=43c81c7afe6c78ae in_seq=0 out_seq=0 2024-08-28T19:25:14.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.076+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/2341646135 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560dae0219c0 con 0x560dadcc1ea0 2024-08-28T19:25:14.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.076+0000 7f3c4b7fe640 10 client.16737 handle_command_reply: tid=0 2024-08-28T19:25:14.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.076+0000 7f3c78b99740 1 client.16737 shutdown 2024-08-28T19:25:14.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.076+0000 7f3c78b99740 1 -- 172.21.15.16:0/2341646135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 msgr2=0x560dae0b9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:14.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.076+0000 7f3c78b99740 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0b9180 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c012a70 tx=0x7f3c5c012f30 comp rx=0 tx=0).stop 2024-08-28T19:25:14.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.076+0000 7f3c78b99740 1 -- 172.21.15.16:0/2341646135 shutdown_connections 2024-08-28T19:25:14.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.076+0000 7f3c78b99740 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadd03520 0x560dae0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.076+0000 7f3c78b99740 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0b9180 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.076+0000 7f3c78b99740 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560dadcc1ea0 0x560dae0cae70 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.076+0000 7f3c78b99740 1 --2- 172.21.15.16:0/2341646135 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.076+0000 7f3c78b99740 1 -- 172.21.15.16:0/2341646135 >> 172.21.15.16:0/2341646135 conn(0x560dadcf7910 msgr2=0x560dae0aa670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:14.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.077+0000 7f3c4b7fe640 0 client.16737 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:14.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.077+0000 7f3c78b99740 1 -- 172.21.15.16:0/2341646135 shutdown_connections 2024-08-28T19:25:14.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.077+0000 7f3c78b99740 1 -- 172.21.15.16:0/2341646135 wait complete. 2024-08-28T19:25:14.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.077+0000 7f3c78b99740 20 client.16737 trim_cache size 0 max 16384 2024-08-28T19:25:14.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c78b99740 1 Processor -- start 2024-08-28T19:25:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c78b99740 1 -- start start 2024-08-28T19:25:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcf7910 0x560dae0b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0b7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c78b99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1ae900 con 0x560dadce2840 2024-08-28T19:25:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c78b99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1b0500 con 0x560dadcf7910 2024-08-28T19:25:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c78b99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1af030 con 0x560dadcda660 2024-08-28T19:25:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0c4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41850/0 (socket says 172.21.15.16:41850) 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/1604977848 learned_addr learned my addr 172.21.15.16:0/1604977848 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.097+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/1604977848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcf7910 0x560dae0b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.098+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/1604977848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 msgr2=0x560dae0b7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.098+0000 7f3c6bfff640 1 --2- 172.21.15.16:0/1604977848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0b7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.098+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/1604977848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0b7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.098+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/1604977848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcf7910 msgr2=0x560dae0b72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.098+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/1604977848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcf7910 0x560dae0b72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.098+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/1604977848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dae0b9320 con 0x560dadce2840 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.098+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/1604977848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0c4b90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3c54005f70 tx=0x7f3c54032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad117527f6021447 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.098+0000 7f3c6bfff640 1 --2- 172.21.15.16:0/1604977848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0b7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.098+0000 7f3c61ffb640 1 -- 172.21.15.16:0/1604977848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54035040 con 0x560dadce2840 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.098+0000 7f3c61ffb640 1 -- 172.21.15.16:0/1604977848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c540046c0 con 0x560dadce2840 2024-08-28T19:25:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.098+0000 7f3c61ffb640 1 -- 172.21.15.16:0/1604977848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5403f040 con 0x560dadce2840 2024-08-28T19:25:14.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.099+0000 7f3c78b99740 1 -- 172.21.15.16:0/1604977848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 msgr2=0x560dae0c4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.099+0000 7f3c78b99740 1 --2- 172.21.15.16:0/1604977848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0c4b90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3c54005f70 tx=0x7f3c54032b20 comp rx=0 tx=0).stop 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.099+0000 7f3c78b99740 1 -- 172.21.15.16:0/1604977848 shutdown_connections 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.099+0000 7f3c78b99740 1 --2- 172.21.15.16:0/1604977848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0c4b90 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.099+0000 7f3c78b99740 1 --2- 172.21.15.16:0/1604977848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0b7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.099+0000 7f3c78b99740 1 --2- 172.21.15.16:0/1604977848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcf7910 0x560dae0b72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.099+0000 7f3c78b99740 1 -- 172.21.15.16:0/1604977848 >> 172.21.15.16:0/1604977848 conn(0x560dae0cb1c0 msgr2=0x560dae0a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.099+0000 7f3c78b99740 1 -- 172.21.15.16:0/1604977848 shutdown_connections 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.099+0000 7f3c78b99740 1 -- 172.21.15.16:0/1604977848 wait complete. 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.100+0000 7f3c78b99740 1 -- start start 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.100+0000 7f3c78b99740 1 Processor -- start 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0bd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0bd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c78b99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcf7910 0x560dae134830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c78b99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1afe30 con 0x560dadce2840 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c78b99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1af760 con 0x560dadcf7910 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c78b99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1ac8e0 con 0x560dadcda660 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c78b99740 10 client.0 _ll_register_callbacks cb 0x7f3c5c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0bd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c6bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0bd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0bd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:60050/0 (socket says 172.21.15.16:60050) 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/1404285113 learned_addr learned my addr 172.21.15.16:0/1404285113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcf7910 0x560dae134830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/1404285113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcf7910 msgr2=0x560dae134830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcf7910 0x560dae134830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/1404285113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 msgr2=0x560dae0bd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0bd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c6b7fe640 1 -- 172.21.15.16:0/1404285113 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dae134d70 con 0x560dadcda660 2024-08-28T19:25:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.101+0000 7f3c6bfff640 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0bd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.102+0000 7f3c6b7fe640 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0bd290 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3c54032df0 tx=0x7f3c54045040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=485818a0fe12d258 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.102+0000 7f3c4b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:25:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.102+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/1404285113 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54035030 con 0x560dadcda660 2024-08-28T19:25:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.102+0000 7f3c78b99740 1 -- 172.21.15.16:0/1404285113 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c540046c0 con 0x560dadcda660 2024-08-28T19:25:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.102+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/1404285113 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3c54004b30 con 0x560dadcda660 2024-08-28T19:25:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.102+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/1404285113 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3c5403f040 con 0x560dadcda660 2024-08-28T19:25:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.102+0000 7f3c78b99740 10 client.26146 fetch_fsmap learned FSMap version 38 2024-08-28T19:25:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.102+0000 7f3c78b99740 1 -- 172.21.15.16:0/1404285113 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560dae135f30 con 0x560dadcda660 2024-08-28T19:25:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.103+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/1404285113 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c540046c0 con 0x560dadcda660 2024-08-28T19:25:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.103+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/1404285113 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 38) ==== 8386+0+0 (secure 0 0 0) 0x7f3c54052040 con 0x560dadcda660 2024-08-28T19:25:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.103+0000 7f3c78b99740 10 client.26146 fetch_fsmap finished waiting for FSMap version 38 2024-08-28T19:25:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.103+0000 7f3c78b99740 10 client.26146 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:25:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.103+0000 7f3c78b99740 20 client.26146 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.103+0000 7f3c78b99740 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560dadd03520 0x560dae138220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:14.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.103+0000 7f3c78b99740 4 client.26146 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e6f999f4", "release": true, "await": true}] 2024-08-28T19:25:14.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.103+0000 7f3c78b99740 1 -- 172.21.15.16:0/1404285113 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e6f999f4", "release": true, "await": true}) -- 0x560dae01d460 con 0x560dadd03520 2024-08-28T19:25:14.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.104+0000 7f3c6bfff640 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560dadd03520 0x560dae138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:14.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.104+0000 7f3c4b7fe640 10 client.26146 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:14.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.104+0000 7f3c6bfff640 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560dadd03520 0x560dae138220 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d55700c5308eb4ea server_cookie=58567af0c9ea3de0 in_seq=0 out_seq=0 2024-08-28T19:25:14.110 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.117+0000 7f3c4b7fe640 1 -- 172.21.15.16:0/1404285113 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x560dae01d460 con 0x560dadd03520 2024-08-28T19:25:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.117+0000 7f3c4b7fe640 10 client.26146 handle_command_reply: tid=0 2024-08-28T19:25:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.117+0000 7f3c78b99740 1 client.26146 shutdown 2024-08-28T19:25:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c78b99740 1 -- 172.21.15.16:0/1404285113 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 msgr2=0x560dae0bd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c78b99740 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0bd290 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3c54032df0 tx=0x7f3c54045040 comp rx=0 tx=0).stop 2024-08-28T19:25:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c78b99740 1 -- 172.21.15.16:0/1404285113 shutdown_connections 2024-08-28T19:25:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c78b99740 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dadcf7910 0x560dae134830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c78b99740 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560dadce2840 0x560dae0bd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c78b99740 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560dadd03520 0x560dae138220 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c78b99740 1 --2- 172.21.15.16:0/1404285113 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560dadcda660 0x560dae0bd290 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c78b99740 1 -- 172.21.15.16:0/1404285113 >> 172.21.15.16:0/1404285113 conn(0x560dae0cb1c0 msgr2=0x560dae0a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c4b7fe640 0 client.26146 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c78b99740 1 -- 172.21.15.16:0/1404285113 shutdown_connections 2024-08-28T19:25:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c78b99740 1 -- 172.21.15.16:0/1404285113 wait complete. 2024-08-28T19:25:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.118+0000 7f3c78b99740 20 client.26146 trim_cache size 0 max 16384 2024-08-28T19:25:14.121 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:25:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 38, 2024-08-28T19:25:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:25:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 12, 2024-08-28T19:25:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:25:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "e6f999f4": { 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 12, 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.0, 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:25:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:14.123 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:25:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 -- 172.21.15.16:0/859958923 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c580b4ca0 msgr2=0x7f3c580b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c580b4ca0 0x7f3c580b71b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3c640e3340 tx=0x7f3c6400cf70 comp rx=0 tx=0).stop 2024-08-28T19:25:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 -- 172.21.15.16:0/859958923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c6c109bb0 msgr2=0x7f3c6c19faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c6c109bb0 0x7f3c6c19faa0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3c6400eb10 tx=0x7f3c6400c2d0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 -- 172.21.15.16:0/859958923 shutdown_connections 2024-08-28T19:25:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c6c109bb0 0x7f3c6c19faa0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3c6c109260 0x7f3c6c19a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3c580b4ca0 0x7f3c580b71b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 --2- 172.21.15.16:0/859958923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c6c103920 0x7f3c6c19a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 -- 172.21.15.16:0/859958923 >> 172.21.15.16:0/859958923 conn(0x7f3c6c0ff350 msgr2=0x7f3c6c1088f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 -- 172.21.15.16:0/859958923 shutdown_connections 2024-08-28T19:25:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:14.119+0000 7f3c72624640 1 -- 172.21.15.16:0/859958923 wait complete. 2024-08-28T19:25:14.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 174088 4.01 MB/sec execute 54 sec latency 28018.751 ms 2024-08-28T19:25:15.038 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'e6f999f4', total seconds elapsed: 31.5 2024-08-28T19:25:15.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:14 smithi016 ceph-mon[18556]: pgmap v465: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-08-28T19:25:15.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:14 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:15.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:14 smithi028 ceph-mon[21769]: pgmap v465: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-08-28T19:25:15.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:14 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.347+0000 7fb21c2c5640 1 -- 172.21.15.16:0/699105955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214108b40 msgr2=0x7fb21410f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.347+0000 7fb21c2c5640 1 --2- 172.21.15.16:0/699105955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214108b40 0x7fb21410f370 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009ca0 tx=0x7fb20c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.347+0000 7fb21c2c5640 1 -- 172.21.15.16:0/699105955 shutdown_connections 2024-08-28T19:25:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.347+0000 7fb21c2c5640 1 --2- 172.21.15.16:0/699105955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214108b40 0x7fb21410f370 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.347+0000 7fb21c2c5640 1 --2- 172.21.15.16:0/699105955 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2141081f0 0x7fb214108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.347+0000 7fb21c2c5640 1 --2- 172.21.15.16:0/699105955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2141028b0 0x7fb214102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.347+0000 7fb21c2c5640 1 -- 172.21.15.16:0/699105955 >> 172.21.15.16:0/699105955 conn(0x7fb214079470 msgr2=0x7fb214079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.347+0000 7fb21c2c5640 1 -- 172.21.15.16:0/699105955 shutdown_connections 2024-08-28T19:25:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.347+0000 7fb21c2c5640 1 -- 172.21.15.16:0/699105955 wait complete. 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.348+0000 7fb21c2c5640 1 Processor -- start 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.348+0000 7fb21c2c5640 1 -- start start 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.348+0000 7fb21c2c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2141028b0 0x7fb21419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.348+0000 7fb21c2c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2141081f0 0x7fb21419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.348+0000 7fb21c2c5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb214108b40 0x7fb21419faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.348+0000 7fb21c2c5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb214112810 con 0x7fb2141028b0 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.348+0000 7fb21c2c5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb214112690 con 0x7fb2141081f0 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.348+0000 7fb21c2c5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb214112990 con 0x7fb214108b40 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21a061640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2141028b0 0x7fb21419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21a061640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2141028b0 0x7fb21419a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43256/0 (socket says 172.21.15.16:43256) 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb219860640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2141081f0 0x7fb21419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21a061640 1 -- 172.21.15.16:0/3908824352 learned_addr learned my addr 172.21.15.16:0/3908824352 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21a862640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb214108b40 0x7fb21419faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21a061640 1 -- 172.21.15.16:0/3908824352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb214108b40 msgr2=0x7fb21419faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21a061640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb214108b40 0x7fb21419faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21a061640 1 -- 172.21.15.16:0/3908824352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2141081f0 msgr2=0x7fb21419a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21a061640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2141081f0 0x7fb21419a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21a061640 1 -- 172.21.15.16:0/3908824352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2141a0220 con 0x7fb2141028b0 2024-08-28T19:25:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb219860640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2141081f0 0x7fb21419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21a061640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2141028b0 0x7fb21419a2b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb21000ea60 tx=0x7fb21000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb20b7fe640 1 -- 172.21.15.16:0/3908824352 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21000cde0 con 0x7fb2141028b0 2024-08-28T19:25:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb20b7fe640 1 -- 172.21.15.16:0/3908824352 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2100035f0 con 0x7fb2141028b0 2024-08-28T19:25:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21c2c5640 1 -- 172.21.15.16:0/3908824352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2141a0510 con 0x7fb2141028b0 2024-08-28T19:25:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb20b7fe640 1 -- 172.21.15.16:0/3908824352 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210010690 con 0x7fb2141028b0 2024-08-28T19:25:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.349+0000 7fb21c2c5640 1 -- 172.21.15.16:0/3908824352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2141a6c70 con 0x7fb2141028b0 2024-08-28T19:25:15.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.352+0000 7fb20b7fe640 1 -- 172.21.15.16:0/3908824352 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fb2100037a0 con 0x7fb2141028b0 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb22283a740 1 Processor -- start 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb22283a740 1 -- start start 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb22283a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 0x562b2f195350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb22283a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edc71b0 0x562b2f195890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb22283a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 0x562b2f195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb22283a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562b2f10c680 con 0x562b2edc71b0 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb22283a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b2f0fbc10 con 0x562b2edcf390 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb22283a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562b2f0ffbc0 con 0x562b2edf0070 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb21a061640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edc71b0 0x562b2f195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb21a862640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 0x562b2f195350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb21a061640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edc71b0 0x562b2f195890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43270/0 (socket says 172.21.15.16:43270) 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb21a061640 1 -- 172.21.15.16:0/2754588157 learned_addr learned my addr 172.21.15.16:0/2754588157 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.353+0000 7fb219860640 1 --2- 172.21.15.16:0/2754588157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 0x562b2f195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb21a862640 1 -- 172.21.15.16:0/2754588157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 msgr2=0x562b2f195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb21a862640 1 --2- 172.21.15.16:0/2754588157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 0x562b2f195dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb21a862640 1 -- 172.21.15.16:0/2754588157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edc71b0 msgr2=0x562b2f195890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb21a862640 1 --2- 172.21.15.16:0/2754588157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edc71b0 0x562b2f195890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb21a862640 1 -- 172.21.15.16:0/2754588157 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b2ee5b650 con 0x562b2edf0070 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb219860640 1 --2- 172.21.15.16:0/2754588157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 0x562b2f195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb20b7fe640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 0x7fb2000b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb21a862640 1 --2- 172.21.15.16:0/2754588157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 0x562b2f195350 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb20c005cc0 tx=0x7fb20c02f420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=217f233bbdc8475b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb21a061640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 0x7fb2000b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb20b7fe640 1 -- 172.21.15.16:0/3908824352 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb2100e5c60 con 0x7fb2141028b0 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb208d9a640 1 -- 172.21.15.16:0/2754588157 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c031cf0 con 0x562b2edf0070 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb208d9a640 1 -- 172.21.15.16:0/2754588157 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb20c036040 con 0x562b2edf0070 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.354+0000 7fb208d9a640 1 -- 172.21.15.16:0/2754588157 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c004040 con 0x562b2edf0070 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.355+0000 7fb22283a740 1 -- 172.21.15.16:0/2754588157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 msgr2=0x562b2f195350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.355+0000 7fb21a061640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 0x7fb2000b71b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb2100e30a0 tx=0x7fb2100e97b0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:15.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.355+0000 7fb22283a740 1 --2- 172.21.15.16:0/2754588157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 0x562b2f195350 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb20c005cc0 tx=0x7fb20c02f420 comp rx=0 tx=0).stop 2024-08-28T19:25:15.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.355+0000 7fb22283a740 1 -- 172.21.15.16:0/2754588157 shutdown_connections 2024-08-28T19:25:15.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.355+0000 7fb22283a740 1 --2- 172.21.15.16:0/2754588157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 0x562b2f195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.355+0000 7fb22283a740 1 --2- 172.21.15.16:0/2754588157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edc71b0 0x562b2f195890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.355+0000 7fb22283a740 1 --2- 172.21.15.16:0/2754588157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 0x562b2f195350 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.355+0000 7fb22283a740 1 -- 172.21.15.16:0/2754588157 >> 172.21.15.16:0/2754588157 conn(0x562b2ede4460 msgr2=0x562b2f193240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:15.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb22283a740 1 -- 172.21.15.16:0/2754588157 shutdown_connections 2024-08-28T19:25:15.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb22283a740 1 -- 172.21.15.16:0/2754588157 wait complete. 2024-08-28T19:25:15.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb22283a740 1 -- start start 2024-08-28T19:25:15.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb22283a740 1 Processor -- start 2024-08-28T19:25:15.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb22283a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edc71b0 0x562b2f1a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb22283a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 0x562b2f1a5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb22283a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edf0070 0x562b2f1b1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb22283a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562b2ef5e700 con 0x562b2edf0070 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb22283a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b2f082750 con 0x562b2edcf390 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb22283a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562b2f1a7e30 con 0x562b2edc71b0 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb22283a740 10 client.0 _ll_register_callbacks cb 0x562b2f0b1e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.356+0000 7fb21a862640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edf0070 0x562b2f1b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb21a862640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edf0070 0x562b2f1b1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43280/0 (socket says 172.21.15.16:43280) 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb219860640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 0x562b2f1a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb21a862640 1 -- 172.21.15.16:0/3763956989 learned_addr learned my addr 172.21.15.16:0/3763956989 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb21a862640 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edc71b0 0x562b2f1a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb219860640 1 -- 172.21.15.16:0/3763956989 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edc71b0 msgr2=0x562b2f1a4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb219860640 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edc71b0 0x562b2f1a4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb219860640 1 -- 172.21.15.16:0/3763956989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edf0070 msgr2=0x562b2f1b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb219860640 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edf0070 0x562b2f1b1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb219860640 1 -- 172.21.15.16:0/3763956989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b2f0bb900 con 0x562b2edcf390 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb21a862640 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edf0070 0x562b2f1b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb219860640 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 0x562b2f1a5410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb21406b420 tx=0x7fb20400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb52eb23944f3857 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb1fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb1fdffb640 1 -- 172.21.15.16:0/3763956989 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb204012110 con 0x562b2edcf390 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb1fdffb640 1 -- 172.21.15.16:0/3763956989 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb204004050 con 0x562b2edcf390 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.357+0000 7fb22283a740 1 -- 172.21.15.16:0/3763956989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb20c036040 con 0x562b2edcf390 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb1fdffb640 1 -- 172.21.15.16:0/3763956989 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb2040122b0 con 0x562b2edcf390 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb22283a740 10 client.26327 fetch_fsmap learned FSMap version 38 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb22283a740 1 -- 172.21.15.16:0/3763956989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562b2f1b3c10 con 0x562b2edcf390 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb1fdffb640 1 -- 172.21.15.16:0/3763956989 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2040054d0 con 0x562b2edcf390 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb1fdffb640 1 -- 172.21.15.16:0/3763956989 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8386+0+0 (secure 0 0 0) 0x7fb204004a70 con 0x562b2edcf390 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb22283a740 10 client.26327 fetch_fsmap finished waiting for FSMap version 38 2024-08-28T19:25:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb22283a740 10 client.26327 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb22283a740 20 client.26327 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb22283a740 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562b2edae9f0 0x562b2f1b7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb22283a740 4 client.26327 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb22283a740 1 -- 172.21.15.16:0/3763956989 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb20c036040 con 0x562b2edae9f0 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.358+0000 7fb21a862640 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562b2edae9f0 0x562b2f1b7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.359+0000 7fb1fdffb640 10 client.26327 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.359+0000 7fb21a862640 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562b2edae9f0 0x562b2f1b7130 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7ece3b7e303c50a server_cookie=25e1b1f5714f80a in_seq=0 out_seq=0 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.359+0000 7fb1fdffb640 1 -- 172.21.15.16:0/3763956989 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fb20c036040 con 0x562b2edae9f0 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.359+0000 7fb1fdffb640 10 client.26327 handle_command_reply: tid=0 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.359+0000 7fb22283a740 1 client.26327 shutdown 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.360+0000 7fb22283a740 1 -- 172.21.15.16:0/3763956989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 msgr2=0x562b2f1a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.360+0000 7fb22283a740 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 0x562b2f1a5410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb21406b420 tx=0x7fb20400f790 comp rx=0 tx=0).stop 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.360+0000 7fb22283a740 1 -- 172.21.15.16:0/3763956989 shutdown_connections 2024-08-28T19:25:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.360+0000 7fb22283a740 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edf0070 0x562b2f1b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.360+0000 7fb22283a740 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edcf390 0x562b2f1a5410 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.360+0000 7fb22283a740 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562b2edae9f0 0x562b2f1b7130 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.360+0000 7fb22283a740 1 --2- 172.21.15.16:0/3763956989 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edc71b0 0x562b2f1a4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.360+0000 7fb1fdffb640 0 client.26327 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.360+0000 7fb22283a740 1 -- 172.21.15.16:0/3763956989 >> 172.21.15.16:0/3763956989 conn(0x562b2ede4460 msgr2=0x562b2f196d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.360+0000 7fb22283a740 1 -- 172.21.15.16:0/3763956989 shutdown_connections 2024-08-28T19:25:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.360+0000 7fb22283a740 1 -- 172.21.15.16:0/3763956989 wait complete. 2024-08-28T19:25:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.361+0000 7fb22283a740 20 client.26327 trim_cache size 0 max 16384 2024-08-28T19:25:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:14 smithi115 ceph-mon[22184]: pgmap v465: 129 pgs: 129 active+clean; 98 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-08-28T19:25:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:14 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.376+0000 7fb22283a740 1 Processor -- start 2024-08-28T19:25:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.376+0000 7fb22283a740 1 -- start start 2024-08-28T19:25:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.376+0000 7fb22283a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edc71b0 0x562b2f18e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb22283a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 0x562b2f1a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb22283a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 0x562b2f1a2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb22283a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141aea50 con 0x562b2edcf390 2024-08-28T19:25:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb22283a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141b0650 con 0x562b2edc71b0 2024-08-28T19:25:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb22283a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141af180 con 0x562b2edf0070 2024-08-28T19:25:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb219860640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 0x562b2f1a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb219860640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 0x562b2f1a20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43286/0 (socket says 172.21.15.16:43286) 2024-08-28T19:25:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb219860640 1 -- 172.21.15.16:0/2811088931 learned_addr learned my addr 172.21.15.16:0/2811088931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb21a862640 1 --2- 172.21.15.16:0/2811088931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edc71b0 0x562b2f18e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb21a862640 1 --2- 172.21.15.16:0/2811088931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 0x562b2f1a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb219860640 1 -- 172.21.15.16:0/2811088931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 msgr2=0x562b2f1a2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb219860640 1 --2- 172.21.15.16:0/2811088931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 0x562b2f1a2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb219860640 1 -- 172.21.15.16:0/2811088931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edc71b0 msgr2=0x562b2f18e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb219860640 1 --2- 172.21.15.16:0/2811088931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edc71b0 0x562b2f18e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb219860640 1 -- 172.21.15.16:0/2811088931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b2f196190 con 0x562b2edcf390 2024-08-28T19:25:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb21a862640 1 --2- 172.21.15.16:0/2811088931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 0x562b2f1a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.377+0000 7fb219860640 1 --2- 172.21.15.16:0/2811088931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 0x562b2f1a20c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb204004e60 tx=0x7fb204012990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe46b49daee0e6a2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:15.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.378+0000 7fb1fffff640 1 -- 172.21.15.16:0/2811088931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb204003840 con 0x562b2edcf390 2024-08-28T19:25:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.378+0000 7fb1fffff640 1 -- 172.21.15.16:0/2811088931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb20402e070 con 0x562b2edcf390 2024-08-28T19:25:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.378+0000 7fb1fffff640 1 -- 172.21.15.16:0/2811088931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb204029450 con 0x562b2edcf390 2024-08-28T19:25:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.379+0000 7fb22283a740 1 -- 172.21.15.16:0/2811088931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 msgr2=0x562b2f1a20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.379+0000 7fb22283a740 1 --2- 172.21.15.16:0/2811088931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 0x562b2f1a20c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb204004e60 tx=0x7fb204012990 comp rx=0 tx=0).stop 2024-08-28T19:25:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.379+0000 7fb22283a740 1 -- 172.21.15.16:0/2811088931 shutdown_connections 2024-08-28T19:25:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.379+0000 7fb22283a740 1 --2- 172.21.15.16:0/2811088931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edf0070 0x562b2f1a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.379+0000 7fb22283a740 1 --2- 172.21.15.16:0/2811088931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 0x562b2f1a20c0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.379+0000 7fb22283a740 1 --2- 172.21.15.16:0/2811088931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edc71b0 0x562b2f18e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.379+0000 7fb22283a740 1 -- 172.21.15.16:0/2811088931 >> 172.21.15.16:0/2811088931 conn(0x562b2ede4460 msgr2=0x562b2f1a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.379+0000 7fb22283a740 1 -- 172.21.15.16:0/2811088931 shutdown_connections 2024-08-28T19:25:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.379+0000 7fb22283a740 1 -- 172.21.15.16:0/2811088931 wait complete. 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.379+0000 7fb22283a740 1 -- start start 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.379+0000 7fb22283a740 1 Processor -- start 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb22283a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edc71b0 0x562b2f1ad830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb22283a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 0x562b2f1abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb22283a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edf0070 0x562b2f1ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb22283a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141aff80 con 0x562b2edcf390 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb22283a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141af8b0 con 0x562b2edf0070 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb22283a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141aca30 con 0x562b2edc71b0 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb21a862640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edf0070 0x562b2f1ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb21a862640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edf0070 0x562b2f1ac250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:33638/0 (socket says 172.21.15.16:33638) 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb21a862640 1 -- 172.21.15.16:0/2443641565 learned_addr learned my addr 172.21.15.16:0/2443641565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb21a862640 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edc71b0 0x562b2f1ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb219860640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 0x562b2f1abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb22283a740 10 client.0 _ll_register_callbacks cb 0x7fb204009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb21a862640 1 -- 172.21.15.16:0/2443641565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edc71b0 msgr2=0x562b2f1ad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb21a862640 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edc71b0 0x562b2f1ad830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb21a862640 1 -- 172.21.15.16:0/2443641565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 msgr2=0x562b2f1abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb21a862640 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 0x562b2f1abd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.380+0000 7fb21a862640 1 -- 172.21.15.16:0/2443641565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b2f1ac940 con 0x562b2edf0070 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.381+0000 7fb21a862640 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edf0070 0x562b2f1ac250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009ca0 tx=0x7fb20c02f420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bf428a5456237bb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.381+0000 7fb1fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.381+0000 7fb1fdffb640 1 -- 172.21.15.16:0/2443641565 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c04a040 con 0x562b2edf0070 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.381+0000 7fb22283a740 1 -- 172.21.15.16:0/2443641565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb20402e070 con 0x562b2edf0070 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.381+0000 7fb1fdffb640 1 -- 172.21.15.16:0/2443641565 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb20c031450 con 0x562b2edf0070 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.381+0000 7fb1fdffb640 1 -- 172.21.15.16:0/2443641565 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb20c031650 con 0x562b2edf0070 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.381+0000 7fb22283a740 10 client.26333 fetch_fsmap learned FSMap version 38 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.381+0000 7fb22283a740 1 -- 172.21.15.16:0/2443641565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562b2f19b9c0 con 0x562b2edf0070 2024-08-28T19:25:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.381+0000 7fb1fdffb640 1 -- 172.21.15.16:0/2443641565 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb20402e070 con 0x562b2edf0070 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.382+0000 7fb1fdffb640 1 -- 172.21.15.16:0/2443641565 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8386+0+0 (secure 0 0 0) 0x7fb20c050040 con 0x562b2edf0070 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.382+0000 7fb22283a740 10 client.26333 fetch_fsmap finished waiting for FSMap version 38 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.382+0000 7fb22283a740 10 client.26333 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.382+0000 7fb22283a740 20 client.26333 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.382+0000 7fb22283a740 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562b2ed83da0 0x562b2f19e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.382+0000 7fb22283a740 4 client.26333 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.382+0000 7fb22283a740 1 -- 172.21.15.16:0/2443641565 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562b2f10e5d0 con 0x562b2ed83da0 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.382+0000 7fb219860640 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562b2ed83da0 0x562b2f19e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb1fdffb640 10 client.26333 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb219860640 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562b2ed83da0 0x562b2f19e120 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d00723e5c98ca08 server_cookie=f14ca1c0f6f04e70 in_seq=0 out_seq=0 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb1fdffb640 1 -- 172.21.15.16:0/2443641565 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562b2f10e5d0 con 0x562b2ed83da0 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb1fdffb640 10 client.26333 handle_command_reply: tid=0 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 1 client.26333 shutdown 2024-08-28T19:25:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 1 -- 172.21.15.16:0/2443641565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edf0070 msgr2=0x562b2f1ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edf0070 0x562b2f1ac250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009ca0 tx=0x7fb20c02f420 comp rx=0 tx=0).stop 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 1 -- 172.21.15.16:0/2443641565 shutdown_connections 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2edf0070 0x562b2f1ac250 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b2edcf390 0x562b2f1abd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562b2ed83da0 0x562b2f19e120 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 1 --2- 172.21.15.16:0/2443641565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b2edc71b0 0x562b2f1ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb1fdffb640 0 client.26333 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 1 -- 172.21.15.16:0/2443641565 >> 172.21.15.16:0/2443641565 conn(0x562b2ede4460 msgr2=0x562b2f196380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 1 -- 172.21.15.16:0/2443641565 shutdown_connections 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 1 -- 172.21.15.16:0/2443641565 wait complete. 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.383+0000 7fb22283a740 20 client.26333 trim_cache size 0 max 16384 2024-08-28T19:25:15.389 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:25:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 -- 172.21.15.16:0/3908824352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 msgr2=0x7fb2000b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 0x7fb2000b71b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb2100e30a0 tx=0x7fb2100e97b0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 -- 172.21.15.16:0/3908824352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2141028b0 msgr2=0x7fb21419a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2141028b0 0x7fb21419a2b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb21000ea60 tx=0x7fb21000ef20 comp rx=0 tx=0).stop 2024-08-28T19:25:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 -- 172.21.15.16:0/3908824352 shutdown_connections 2024-08-28T19:25:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb214108b40 0x7fb21419faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 0x7fb2000b71b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2141081f0 0x7fb21419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 --2- 172.21.15.16:0/3908824352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2141028b0 0x7fb21419a2b0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:15.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 -- 172.21.15.16:0/3908824352 >> 172.21.15.16:0/3908824352 conn(0x7fb214079470 msgr2=0x7fb214101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:15.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 -- 172.21.15.16:0/3908824352 shutdown_connections 2024-08-28T19:25:15.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:15.384+0000 7fb21c2c5640 1 -- 172.21.15.16:0/3908824352 wait complete. 2024-08-28T19:25:15.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 175882 4.05 MB/sec execute 55 sec latency 15.233 ms 2024-08-28T19:25:16.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 177762 4.21 MB/sec execute 56 sec latency 14.979 ms 2024-08-28T19:25:17.213 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:25:17.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:25:17.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:16 smithi016 ceph-mon[18556]: pgmap v466: 129 pgs: 129 active+clean; 98 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 82 KiB/s wr, 30 op/s 2024-08-28T19:25:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:16 smithi028 ceph-mon[21769]: pgmap v466: 129 pgs: 129 active+clean; 98 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 82 KiB/s wr, 30 op/s 2024-08-28T19:25:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:16 smithi115 ceph-mon[22184]: pgmap v466: 129 pgs: 129 active+clean; 98 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 82 KiB/s wr, 30 op/s 2024-08-28T19:25:17.484 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:17.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 179814 4.27 MB/sec execute 57 sec latency 12.106 ms 2024-08-28T19:25:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.195+0000 7f855ac78640 1 -- 172.21.15.16:0/1916314490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855410dbd0 msgr2=0x7f85541100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.195+0000 7f855ac78640 1 --2- 172.21.15.16:0/1916314490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855410dbd0 0x7f85541100a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f854c009900 tx=0x7f854c02a410 comp rx=0 tx=0).stop 2024-08-28T19:25:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.195+0000 7f855ac78640 1 -- 172.21.15.16:0/1916314490 shutdown_connections 2024-08-28T19:25:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.195+0000 7f855ac78640 1 --2- 172.21.15.16:0/1916314490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855410dbd0 0x7f85541100a0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.195+0000 7f855ac78640 1 --2- 172.21.15.16:0/1916314490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8554105050 0x7f855410d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.195+0000 7f855ac78640 1 --2- 172.21.15.16:0/1916314490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8554104620 0x7f8554104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.195+0000 7f855ac78640 1 -- 172.21.15.16:0/1916314490 >> 172.21.15.16:0/1916314490 conn(0x7f85540794d0 msgr2=0x7f85540798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.195+0000 7f855ac78640 1 -- 172.21.15.16:0/1916314490 shutdown_connections 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.195+0000 7f855ac78640 1 -- 172.21.15.16:0/1916314490 wait complete. 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.195+0000 7f855ac78640 1 Processor -- start 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f855ac78640 1 -- start start 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f855ac78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8554104620 0x7f855419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f855ac78640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8554105050 0x7f855419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f855ac78640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f855410dbd0 0x7f855419fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f855ac78640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8554113810 con 0x7f8554104620 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f855ac78640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8554113690 con 0x7f8554105050 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f855ac78640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8554113990 con 0x7f855410dbd0 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f8558a14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8554104620 0x7f855419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f8558a14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8554104620 0x7f855419a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43316/0 (socket says 172.21.15.16:43316) 2024-08-28T19:25:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f8558a14640 1 -- 172.21.15.16:0/2431488656 learned_addr learned my addr 172.21.15.16:0/2431488656 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f8559215640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f855410dbd0 0x7f855419fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f8553fff640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8554105050 0x7f855419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f8558a14640 1 -- 172.21.15.16:0/2431488656 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f855410dbd0 msgr2=0x7f855419fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f8558a14640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f855410dbd0 0x7f855419fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f8558a14640 1 -- 172.21.15.16:0/2431488656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8554105050 msgr2=0x7f855419a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f8558a14640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8554105050 0x7f855419a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.196+0000 7f8558a14640 1 -- 172.21.15.16:0/2431488656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85541a01f0 con 0x7f8554104620 2024-08-28T19:25:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.197+0000 7f8558a14640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8554104620 0x7f855419a280 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f854400ebc0 tx=0x7f854400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:18.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.197+0000 7f8551ffb640 1 -- 172.21.15.16:0/2431488656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8544003b70 con 0x7f8554104620 2024-08-28T19:25:18.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.197+0000 7f855ac78640 1 -- 172.21.15.16:0/2431488656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85541a04e0 con 0x7f8554104620 2024-08-28T19:25:18.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.197+0000 7f8551ffb640 1 -- 172.21.15.16:0/2431488656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8544003d10 con 0x7f8554104620 2024-08-28T19:25:18.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.197+0000 7f8551ffb640 1 -- 172.21.15.16:0/2431488656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8544019800 con 0x7f8554104620 2024-08-28T19:25:18.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.198+0000 7f855ac78640 1 -- 172.21.15.16:0/2431488656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8554102410 con 0x7f8554104620 2024-08-28T19:25:18.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.200+0000 7f85611ed740 1 Processor -- start 2024-08-28T19:25:18.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f85611ed740 1 -- start start 2024-08-28T19:25:18.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f8551ffb640 1 -- 172.21.15.16:0/2431488656 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f85440199a0 con 0x7f8554104620 2024-08-28T19:25:18.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f85611ed740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a63350 0x556e37e09780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f85611ed740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e09cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f85611ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 0x556e37e0a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f85611ed740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556e37d7f5f0 con 0x556e37a63350 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f85611ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e37d6eb80 con 0x556e37a42670 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f85611ed740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556e37d72b30 con 0x556e37a3a490 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f8559215640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a63350 0x556e37e09780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f8559215640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a63350 0x556e37e09780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43326/0 (socket says 172.21.15.16:43326) 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f8559215640 1 -- 172.21.15.16:0/748208015 learned_addr learned my addr 172.21.15.16:0/748208015 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f8558a14640 1 --2- 172.21.15.16:0/748208015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e09cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.201+0000 7f8553fff640 1 --2- 172.21.15.16:0/748208015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 0x556e37e0a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f8559215640 1 -- 172.21.15.16:0/748208015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 msgr2=0x556e37e09cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f8559215640 1 --2- 172.21.15.16:0/748208015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e09cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f8559215640 1 -- 172.21.15.16:0/748208015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 msgr2=0x556e37e0a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:18.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f8559215640 1 --2- 172.21.15.16:0/748208015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 0x556e37e0a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f8559215640 1 -- 172.21.15.16:0/748208015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e37ace930 con 0x556e37a63350 2024-08-28T19:25:18.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f8558a14640 1 --2- 172.21.15.16:0/748208015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e09cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:18.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f8551ffb640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85400b9130 0x7f85400bb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f8559215640 1 --2- 172.21.15.16:0/748208015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a63350 0x556e37e09780 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f854c009900 tx=0x7f854c032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef3f5bce9478048 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:18.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f8558a14640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85400b9130 0x7f85400bb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f854affd640 1 -- 172.21.15.16:0/748208015 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c040070 con 0x556e37a63350 2024-08-28T19:25:18.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f854affd640 1 -- 172.21.15.16:0/748208015 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f854c004030 con 0x556e37a63350 2024-08-28T19:25:18.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f854affd640 1 -- 172.21.15.16:0/748208015 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c035040 con 0x556e37a63350 2024-08-28T19:25:18.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.202+0000 7f8551ffb640 1 -- 172.21.15.16:0/2431488656 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f85440e60d0 con 0x7f8554104620 2024-08-28T19:25:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.203+0000 7f85611ed740 1 -- 172.21.15.16:0/748208015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a63350 msgr2=0x556e37e09780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.203+0000 7f85611ed740 1 --2- 172.21.15.16:0/748208015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a63350 0x556e37e09780 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f854c009900 tx=0x7f854c032b20 comp rx=0 tx=0).stop 2024-08-28T19:25:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.203+0000 7f8558a14640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85400b9130 0x7f85400bb640 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f85440e3560 tx=0x7f85440b5040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.203+0000 7f85611ed740 1 -- 172.21.15.16:0/748208015 shutdown_connections 2024-08-28T19:25:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.203+0000 7f85611ed740 1 --2- 172.21.15.16:0/748208015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 0x556e37e0a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.203+0000 7f85611ed740 1 --2- 172.21.15.16:0/748208015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e09cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.203+0000 7f85611ed740 1 --2- 172.21.15.16:0/748208015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a63350 0x556e37e09780 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.203+0000 7f85611ed740 1 -- 172.21.15.16:0/748208015 >> 172.21.15.16:0/748208015 conn(0x556e37a57740 msgr2=0x556e37e07670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.203+0000 7f85611ed740 1 -- 172.21.15.16:0/748208015 shutdown_connections 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.203+0000 7f85611ed740 1 -- 172.21.15.16:0/748208015 wait complete. 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.204+0000 7f85611ed740 1 -- start start 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.204+0000 7f85611ed740 1 Processor -- start 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f85611ed740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a3a490 0x556e37e14f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f85611ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 0x556e37e154c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f85611ed740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a63350 0x556e37e211f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f85611ed740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556e37bd1890 con 0x556e37a3a490 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f85611ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e37cf5790 con 0x556e37a42670 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f85611ed740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556e37e17f00 con 0x556e37a63350 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f85611ed740 10 client.0 _ll_register_callbacks cb 0x556e37a7aba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8559215640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a3a490 0x556e37e14f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8553fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 0x556e37e154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8553fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 0x556e37e154c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:33682/0 (socket says 172.21.15.16:33682) 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8553fff640 1 -- 172.21.15.16:0/3145966146 learned_addr learned my addr 172.21.15.16:0/3145966146 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8559215640 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a63350 0x556e37e211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8553fff640 1 -- 172.21.15.16:0/3145966146 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a63350 msgr2=0x556e37e211f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8553fff640 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a63350 0x556e37e211f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8553fff640 1 -- 172.21.15.16:0/3145966146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a3a490 msgr2=0x556e37e14f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8553fff640 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a3a490 0x556e37e14f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8553fff640 1 -- 172.21.15.16:0/3145966146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e37d2cf30 con 0x556e37a42670 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8559215640 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a63350 0x556e37e211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8553fff640 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 0x556e37e154c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f853c012700 tx=0x7f853c012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff2beba1bc49dab5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8533fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.205+0000 7f8533fff640 1 -- 172.21.15.16:0/3145966146 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c00f8c0 con 0x556e37a42670 2024-08-28T19:25:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f8533fff640 1 -- 172.21.15.16:0/3145966146 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f853c00fa60 con 0x556e37a42670 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f85611ed740 1 -- 172.21.15.16:0/3145966146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f854c004030 con 0x556e37a42670 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f8533fff640 1 -- 172.21.15.16:0/3145966146 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f853c025050 con 0x556e37a42670 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f85611ed740 10 client.26345 fetch_fsmap learned FSMap version 38 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f85611ed740 1 -- 172.21.15.16:0/3145966146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556e37e23da0 con 0x556e37a42670 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f8533fff640 1 -- 172.21.15.16:0/3145966146 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f854c004030 con 0x556e37a42670 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f8533fff640 1 -- 172.21.15.16:0/3145966146 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8386+0+0 (secure 0 0 0) 0x7f853c010430 con 0x556e37a42670 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f85611ed740 10 client.26345 fetch_fsmap finished waiting for FSMap version 38 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f85611ed740 10 client.26345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f85611ed740 20 client.26345 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f85611ed740 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556e37a21cd0 0x556e37e27300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f85611ed740 4 client.26345 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f85611ed740 1 -- 172.21.15.16:0/3145966146 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e37d81540 con 0x556e37a21cd0 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.206+0000 7f8559215640 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556e37a21cd0 0x556e37e27300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.207+0000 7f8559215640 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556e37a21cd0 0x556e37e27300 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2b314fb4b848e49 server_cookie=11dfb9ffa6d9d657 in_seq=0 out_seq=0 2024-08-28T19:25:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.207+0000 7f8533fff640 10 client.26345 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f8533fff640 1 -- 172.21.15.16:0/3145966146 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556e37d81540 con 0x556e37a21cd0 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f8533fff640 10 client.26345 handle_command_reply: tid=0 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 1 client.26345 shutdown 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 1 -- 172.21.15.16:0/3145966146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 msgr2=0x556e37e154c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 0x556e37e154c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f853c012700 tx=0x7f853c012bc0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 1 -- 172.21.15.16:0/3145966146 shutdown_connections 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556e37a21cd0 0x556e37e27300 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a63350 0x556e37e211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a42670 0x556e37e154c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f8533fff640 0 client.26345 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 1 --2- 172.21.15.16:0/3145966146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a3a490 0x556e37e14f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 1 -- 172.21.15.16:0/3145966146 >> 172.21.15.16:0/3145966146 conn(0x556e37a57740 msgr2=0x556e37e0c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 1 -- 172.21.15.16:0/3145966146 shutdown_connections 2024-08-28T19:25:18.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 1 -- 172.21.15.16:0/3145966146 wait complete. 2024-08-28T19:25:18.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.208+0000 7f85611ed740 20 client.26345 trim_cache size 0 max 16384 2024-08-28T19:25:18.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.226+0000 7f85611ed740 1 Processor -- start 2024-08-28T19:25:18.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.226+0000 7f85611ed740 1 -- start start 2024-08-28T19:25:18.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f85611ed740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e07770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f85611ed740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e210f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f85611ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a63350 0x556e37e21630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f85611ed740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85541ae9d0 con 0x556e37a42670 2024-08-28T19:25:18.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f85611ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85541b05d0 con 0x556e37a63350 2024-08-28T19:25:18.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f85611ed740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f85541af100 con 0x556e37a3a490 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f8553fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e210f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f8553fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e210f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43350/0 (socket says 172.21.15.16:43350) 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f8553fff640 1 -- 172.21.15.16:0/2305881376 learned_addr learned my addr 172.21.15.16:0/2305881376 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f8559215640 1 --2- 172.21.15.16:0/2305881376 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e07770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f8559215640 1 --2- 172.21.15.16:0/2305881376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a63350 0x556e37e21630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f8553fff640 1 -- 172.21.15.16:0/2305881376 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 msgr2=0x556e37e07770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f8553fff640 1 --2- 172.21.15.16:0/2305881376 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e07770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f8553fff640 1 -- 172.21.15.16:0/2305881376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a63350 msgr2=0x556e37e21630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f8553fff640 1 --2- 172.21.15.16:0/2305881376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a63350 0x556e37e21630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f8553fff640 1 -- 172.21.15.16:0/2305881376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e37e17a90 con 0x556e37a42670 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.227+0000 7f8553fff640 1 --2- 172.21.15.16:0/2305881376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e210f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f853c001eb0 tx=0x7f853c009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c9039936f4b548 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:18.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.228+0000 7f854a7fc640 1 -- 172.21.15.16:0/2305881376 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c010430 con 0x556e37a42670 2024-08-28T19:25:18.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.228+0000 7f854a7fc640 1 -- 172.21.15.16:0/2305881376 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f853c0105d0 con 0x556e37a42670 2024-08-28T19:25:18.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.228+0000 7f854a7fc640 1 -- 172.21.15.16:0/2305881376 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c00fea0 con 0x556e37a42670 2024-08-28T19:25:18.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.228+0000 7f85611ed740 1 -- 172.21.15.16:0/2305881376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 msgr2=0x556e37e210f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:18.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.228+0000 7f85611ed740 1 --2- 172.21.15.16:0/2305881376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e210f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f853c001eb0 tx=0x7f853c009e40 comp rx=0 tx=0).stop 2024-08-28T19:25:18.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.229+0000 7f85611ed740 1 -- 172.21.15.16:0/2305881376 shutdown_connections 2024-08-28T19:25:18.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.229+0000 7f85611ed740 1 --2- 172.21.15.16:0/2305881376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a63350 0x556e37e21630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.229+0000 7f85611ed740 1 --2- 172.21.15.16:0/2305881376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e210f0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.229+0000 7f85611ed740 1 --2- 172.21.15.16:0/2305881376 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e07770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.229+0000 7f85611ed740 1 -- 172.21.15.16:0/2305881376 >> 172.21.15.16:0/2305881376 conn(0x556e37a57740 msgr2=0x556e37e07060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.229+0000 7f85611ed740 1 -- 172.21.15.16:0/2305881376 shutdown_connections 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.229+0000 7f85611ed740 1 -- 172.21.15.16:0/2305881376 wait complete. 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.229+0000 7f85611ed740 1 -- start start 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.229+0000 7f85611ed740 1 Processor -- start 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f85611ed740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e19a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f85611ed740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e1a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f85611ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a63350 0x556e37e8acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f85611ed740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85541aff00 con 0x556e37a42670 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f85611ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85541af830 con 0x556e37a63350 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f85611ed740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f85541ac9b0 con 0x556e37a3a490 2024-08-28T19:25:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f85611ed740 10 client.0 _ll_register_callbacks cb 0x7f853c009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8553fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e1a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8559215640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e19a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8553fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e1a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43362/0 (socket says 172.21.15.16:43362) 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8553fff640 1 -- 172.21.15.16:0/2171458127 learned_addr learned my addr 172.21.15.16:0/2171458127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8559215640 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a63350 0x556e37e8acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8553fff640 1 -- 172.21.15.16:0/2171458127 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 msgr2=0x556e37e19a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8553fff640 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e19a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8553fff640 1 -- 172.21.15.16:0/2171458127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a63350 msgr2=0x556e37e8acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8553fff640 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a63350 0x556e37e8acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8553fff640 1 -- 172.21.15.16:0/2171458127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e37e8b340 con 0x556e37a42670 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8553fff640 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e1a010 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f853c0126a0 tx=0x7f853c005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8dab71bbaefc16f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.230+0000 7f8533fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f8533fff640 1 -- 172.21.15.16:0/2171458127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c00f9a0 con 0x556e37a42670 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f8533fff640 1 -- 172.21.15.16:0/2171458127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f853c018040 con 0x556e37a42670 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f85611ed740 1 -- 172.21.15.16:0/2171458127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f853c0105d0 con 0x556e37a42670 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f8533fff640 1 -- 172.21.15.16:0/2171458127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f853c010a70 con 0x556e37a42670 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f85611ed740 10 client.16779 fetch_fsmap learned FSMap version 38 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f85611ed740 1 -- 172.21.15.16:0/2171458127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556e37e926c0 con 0x556e37a42670 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f8533fff640 1 -- 172.21.15.16:0/2171458127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f853c0105d0 con 0x556e37a42670 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f8533fff640 1 -- 172.21.15.16:0/2171458127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 38) ==== 8386+0+0 (secure 0 0 0) 0x7f853c003860 con 0x556e37a42670 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f85611ed740 10 client.16779 fetch_fsmap finished waiting for FSMap version 38 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f85611ed740 10 client.16779 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f85611ed740 20 client.16779 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f85611ed740 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556e379f7080 0x556e37e94d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f85611ed740 4 client.16779 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:25:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f85611ed740 1 -- 172.21.15.16:0/2171458127 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556e37d7cfe0 con 0x556e379f7080 2024-08-28T19:25:18.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.231+0000 7f8559215640 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556e379f7080 0x556e37e94d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:18.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.232+0000 7f8533fff640 10 client.16779 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:18.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.232+0000 7f8559215640 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556e379f7080 0x556e37e94d90 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b76d24c2df5f32b5 server_cookie=3b1e1b7d7ec5eab7 in_seq=0 out_seq=0 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f8533fff640 1 -- 172.21.15.16:0/2171458127 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556e37d7cfe0 con 0x556e379f7080 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f8533fff640 10 client.16779 handle_command_reply: tid=0 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f85611ed740 1 client.16779 shutdown 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f85611ed740 1 -- 172.21.15.16:0/2171458127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 msgr2=0x556e37e1a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f85611ed740 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e1a010 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f853c0126a0 tx=0x7f853c005f10 comp rx=0 tx=0).stop 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f85611ed740 1 -- 172.21.15.16:0/2171458127 shutdown_connections 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f85611ed740 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e37a63350 0x556e37e8acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f85611ed740 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e37a42670 0x556e37e1a010 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f85611ed740 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556e379f7080 0x556e37e94d90 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f85611ed740 1 --2- 172.21.15.16:0/2171458127 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e37a3a490 0x556e37e19a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f8533fff640 0 client.16779 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.234+0000 7f85611ed740 1 -- 172.21.15.16:0/2171458127 >> 172.21.15.16:0/2171458127 conn(0x556e37a57740 msgr2=0x556e37e0c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f85611ed740 1 -- 172.21.15.16:0/2171458127 shutdown_connections 2024-08-28T19:25:18.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f85611ed740 1 -- 172.21.15.16:0/2171458127 wait complete. 2024-08-28T19:25:18.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f85611ed740 20 client.16779 trim_cache size 0 max 16384 2024-08-28T19:25:18.239 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:25:18.239 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:25:18.239 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "7af73992-6dbf-427c-84e6-e23aa2c95123", 2024-08-28T19:25:18.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:25:18.239 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:25:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f855ac78640 1 -- 172.21.15.16:0/2431488656 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85400b9130 msgr2=0x7f85400bb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f855ac78640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85400b9130 0x7f85400bb640 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f85440e3560 tx=0x7f85440b5040 comp rx=0 tx=0).stop 2024-08-28T19:25:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f855ac78640 1 -- 172.21.15.16:0/2431488656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8554104620 msgr2=0x7f855419a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f855ac78640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8554104620 0x7f855419a280 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f854400ebc0 tx=0x7f854400c6a0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f855ac78640 1 -- 172.21.15.16:0/2431488656 shutdown_connections 2024-08-28T19:25:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f855ac78640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f855410dbd0 0x7f855419fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f855ac78640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85400b9130 0x7f85400bb640 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f855ac78640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8554105050 0x7f855419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f855ac78640 1 --2- 172.21.15.16:0/2431488656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8554104620 0x7f855419a280 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.235+0000 7f855ac78640 1 -- 172.21.15.16:0/2431488656 >> 172.21.15.16:0/2431488656 conn(0x7f85540794d0 msgr2=0x7f8554106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.236+0000 7f855ac78640 1 -- 172.21.15.16:0/2431488656 shutdown_connections 2024-08-28T19:25:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:18.236+0000 7f855ac78640 1 -- 172.21.15.16:0/2431488656 wait complete. 2024-08-28T19:25:18.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 181361 4.40 MB/sec execute 58 sec latency 56.111 ms 2024-08-28T19:25:19.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: pgmap v467: 129 pgs: 129 active+clean; 99 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 162 KiB/s wr, 37 op/s 2024-08-28T19:25:19.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:25:19.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:25:19.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:25:19.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:25:19.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:25:19.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:25:19.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:18 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:25:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: pgmap v467: 129 pgs: 129 active+clean; 99 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 162 KiB/s wr, 37 op/s 2024-08-28T19:25:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:25:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:25:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:25:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:25:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:25:19.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:25:19.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:18 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: pgmap v467: 129 pgs: 129 active+clean; 99 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 162 KiB/s wr, 37 op/s 2024-08-28T19:25:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:25:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:25:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:25:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:25:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:25:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:25:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:18 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:19.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.922+0000 7f2f813b5640 1 -- 172.21.15.16:0/4227062271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c102810 msgr2=0x7f2f7c102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:19.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.922+0000 7f2f813b5640 1 --2- 172.21.15.16:0/4227062271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c102810 0x7f2f7c102c20 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009ca0 tx=0x7f2f6c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.922+0000 7f2f813b5640 1 -- 172.21.15.16:0/4227062271 shutdown_connections 2024-08-28T19:25:19.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.922+0000 7f2f813b5640 1 --2- 172.21.15.16:0/4227062271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2f7c108aa0 0x7f2f7c10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.922+0000 7f2f813b5640 1 --2- 172.21.15.16:0/4227062271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f7c108150 0x7f2f7c108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.922+0000 7f2f813b5640 1 --2- 172.21.15.16:0/4227062271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c102810 0x7f2f7c102c20 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.922+0000 7f2f813b5640 1 -- 172.21.15.16:0/4227062271 >> 172.21.15.16:0/4227062271 conn(0x7f2f7c0794d0 msgr2=0x7f2f7c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:19.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.922+0000 7f2f813b5640 1 -- 172.21.15.16:0/4227062271 shutdown_connections 2024-08-28T19:25:19.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.922+0000 7f2f813b5640 1 -- 172.21.15.16:0/4227062271 wait complete. 2024-08-28T19:25:19.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.923+0000 7f2f813b5640 1 Processor -- start 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.923+0000 7f2f813b5640 1 -- start start 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f813b5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2f7c102810 0x7f2f7c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f813b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c108150 0x7f2f7c19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f813b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f7c108aa0 0x7f2f7c19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f813b5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c112700 con 0x7f2f7c108150 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f813b5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c112580 con 0x7f2f7c108aa0 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f813b5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c112880 con 0x7f2f7c102810 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c108150 0x7f2f7c19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c108150 0x7f2f7c19a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43382/0 (socket says 172.21.15.16:43382) 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7a7fc640 1 -- 172.21.15.16:0/4188898565 learned_addr learned my addr 172.21.15.16:0/4188898565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7b7fe640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f7c108aa0 0x7f2f7c19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7affd640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2f7c102810 0x7f2f7c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7a7fc640 1 -- 172.21.15.16:0/4188898565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2f7c102810 msgr2=0x7f2f7c19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2f7c102810 0x7f2f7c19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7a7fc640 1 -- 172.21.15.16:0/4188898565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f7c108aa0 msgr2=0x7f2f7c19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f7c108aa0 0x7f2f7c19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7a7fc640 1 -- 172.21.15.16:0/4188898565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7c1a0190 con 0x7f2f7c108150 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7affd640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2f7c102810 0x7f2f7c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7b7fe640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f7c108aa0 0x7f2f7c19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c108150 0x7f2f7c19a7c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000e9c0 tx=0x7f2f7000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.924+0000 7f2f63fff640 1 -- 172.21.15.16:0/4188898565 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7000cd80 con 0x7f2f7c108150 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.925+0000 7f2f813b5640 1 -- 172.21.15.16:0/4188898565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7c1a0480 con 0x7f2f7c108150 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.925+0000 7f2f63fff640 1 -- 172.21.15.16:0/4188898565 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f700054d0 con 0x7f2f7c108150 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.925+0000 7f2f63fff640 1 -- 172.21.15.16:0/4188898565 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70010430 con 0x7f2f7c108150 2024-08-28T19:25:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.925+0000 7f2f813b5640 1 -- 172.21.15.16:0/4188898565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f7c1a6be0 con 0x7f2f7c108150 2024-08-28T19:25:19.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.927+0000 7f2f8792a740 1 Processor -- start 2024-08-28T19:25:19.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.927+0000 7f2f8792a740 1 -- start start 2024-08-28T19:25:19.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.927+0000 7f2f63fff640 1 -- 172.21.15.16:0/4188898565 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2f70004fd0 con 0x7f2f7c108150 2024-08-28T19:25:19.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc09b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc09b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 0x55fbbc09bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f8792a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fbbc012630 con 0x55fbbbcf6070 2024-08-28T19:25:19.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f8792a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fbbc001bc0 con 0x55fbbbccd1b0 2024-08-28T19:25:19.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f8792a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55fbbc005b70 con 0x55fbbbcd5390 2024-08-28T19:25:19.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc09b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc09b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7a7fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 0x55fbbc09bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc09b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:33736/0 (socket says 172.21.15.16:33736) 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc09b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43396/0 (socket says 172.21.15.16:43396) 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7affd640 1 -- 172.21.15.16:0/3704622734 learned_addr learned my addr 172.21.15.16:0/3704622734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7affd640 1 -- 172.21.15.16:0/3704622734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 msgr2=0x55fbbc09bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7affd640 1 --2- 172.21.15.16:0/3704622734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 0x55fbbc09bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7affd640 1 -- 172.21.15.16:0/3704622734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 msgr2=0x55fbbc09b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7affd640 1 --2- 172.21.15.16:0/3704622734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc09b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7affd640 1 -- 172.21.15.16:0/3704622734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbbbd61650 con 0x55fbbbccd1b0 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.928+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/3704622734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 0x55fbbc09bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.929+0000 7f2f63fff640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2f640b4ca0 0x7f2f640b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.929+0000 7f2f7affd640 1 --2- 172.21.15.16:0/3704622734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc09b800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c035000 tx=0x7f2f6c0094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c0e3d87659b84b1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.929+0000 7f2f7b7fe640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2f640b4ca0 0x7f2f640b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.929+0000 7f2f63fff640 1 -- 172.21.15.16:0/4188898565 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f2f700e5cc0 con 0x7f2f7c108150 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.929+0000 7f2f617fa640 1 -- 172.21.15.16:0/3704622734 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c031650 con 0x55fbbbccd1b0 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.929+0000 7f2f617fa640 1 -- 172.21.15.16:0/3704622734 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f6c036040 con 0x55fbbbccd1b0 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.929+0000 7f2f617fa640 1 -- 172.21.15.16:0/3704622734 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c0044c0 con 0x55fbbbccd1b0 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.930+0000 7f2f8792a740 1 -- 172.21.15.16:0/3704622734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 msgr2=0x55fbbc09b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.930+0000 7f2f8792a740 1 --2- 172.21.15.16:0/3704622734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc09b800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c035000 tx=0x7f2f6c0094a0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.930+0000 7f2f7b7fe640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2f640b4ca0 0x7f2f640b71b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2f74012640 tx=0x7f2f74005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.930+0000 7f2f8792a740 1 -- 172.21.15.16:0/3704622734 shutdown_connections 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.930+0000 7f2f8792a740 1 --2- 172.21.15.16:0/3704622734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 0x55fbbc09bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.930+0000 7f2f8792a740 1 --2- 172.21.15.16:0/3704622734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc09b800 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.930+0000 7f2f8792a740 1 --2- 172.21.15.16:0/3704622734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc09b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.930+0000 7f2f8792a740 1 -- 172.21.15.16:0/3704622734 >> 172.21.15.16:0/3704622734 conn(0x55fbbbcea460 msgr2=0x55fbbc0991b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:19.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.930+0000 7f2f8792a740 1 -- 172.21.15.16:0/3704622734 shutdown_connections 2024-08-28T19:25:19.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.930+0000 7f2f8792a740 1 -- 172.21.15.16:0/3704622734 wait complete. 2024-08-28T19:25:19.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.930+0000 7f2f8792a740 1 -- start start 2024-08-28T19:25:19.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f8792a740 1 Processor -- start 2024-08-28T19:25:19.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcd5390 0x55fbbc0a6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcf6070 0x55fbbc0b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f8792a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fbbbe64700 con 0x55fbbbcd5390 2024-08-28T19:25:19.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f8792a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fbbbf88700 con 0x55fbbbccd1b0 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f8792a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55fbbc0a98c0 con 0x55fbbbcf6070 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f8792a740 10 client.0 _ll_register_callbacks cb 0x55fbbbfb7dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f7a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcd5390 0x55fbbc0a6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f7a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcd5390 0x55fbbc0a6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43410/0 (socket says 172.21.15.16:43410) 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f7affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f7a7fc640 1 -- 172.21.15.16:0/1101001096 learned_addr learned my addr 172.21.15.16:0/1101001096 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f7b7fe640 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcf6070 0x55fbbc0b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f7a7fc640 1 -- 172.21.15.16:0/1101001096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcf6070 msgr2=0x55fbbc0b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcf6070 0x55fbbc0b2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f7a7fc640 1 -- 172.21.15.16:0/1101001096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 msgr2=0x55fbbc0a6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0a6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.931+0000 7f2f7a7fc640 1 -- 172.21.15.16:0/1101001096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbbbfc18b0 con 0x55fbbbcd5390 2024-08-28T19:25:19.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.932+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcd5390 0x55fbbc0a6fa0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2f700abfd0 tx=0x7f2f700048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fbe9356329392c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.932+0000 7f2f5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.932+0000 7f2f5eff5640 1 -- 172.21.15.16:0/1101001096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f700e7030 con 0x55fbbbcd5390 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.932+0000 7f2f8792a740 1 -- 172.21.15.16:0/1101001096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f6c036040 con 0x55fbbbcd5390 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.932+0000 7f2f5eff5640 1 -- 172.21.15.16:0/1101001096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f2f700ac4b0 con 0x55fbbbcd5390 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.932+0000 7f2f5eff5640 1 -- 172.21.15.16:0/1101001096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f2f700abe10 con 0x55fbbbcd5390 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.932+0000 7f2f8792a740 10 client.16791 fetch_fsmap learned FSMap version 38 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.932+0000 7f2f8792a740 1 -- 172.21.15.16:0/1101001096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fbbc0b56e0 con 0x55fbbbcd5390 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.932+0000 7f2f5eff5640 1 -- 172.21.15.16:0/1101001096 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f6c036040 con 0x55fbbbcd5390 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.932+0000 7f2f5eff5640 1 -- 172.21.15.16:0/1101001096 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 38) ==== 8386+0+0 (secure 0 0 0) 0x7f2f700971a0 con 0x55fbbbcd5390 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.933+0000 7f2f8792a740 10 client.16791 fetch_fsmap finished waiting for FSMap version 38 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.933+0000 7f2f8792a740 10 client.16791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.933+0000 7f2f8792a740 20 client.16791 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:19.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.933+0000 7f2f8792a740 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fbbbcb49f0 0x55fbbc0b8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.933+0000 7f2f8792a740 4 client.16791 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.933+0000 7f2f8792a740 1 -- 172.21.15.16:0/1101001096 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbbc014580 con 0x55fbbbcb49f0 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.933+0000 7f2f7affd640 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fbbbcb49f0 0x55fbbc0b8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.933+0000 7f2f5eff5640 10 client.16791 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.933+0000 7f2f7affd640 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fbbbcb49f0 0x55fbbc0b8ff0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11c8adeed503e7fb server_cookie=9b1735b4b5338ff3 in_seq=0 out_seq=0 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f5eff5640 1 -- 172.21.15.16:0/1101001096 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55fbbc014580 con 0x55fbbbcb49f0 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f5eff5640 10 client.16791 handle_command_reply: tid=0 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 1 client.16791 shutdown 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 1 -- 172.21.15.16:0/1101001096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcd5390 msgr2=0x55fbbc0a6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcd5390 0x55fbbc0a6fa0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2f700abfd0 tx=0x7f2f700048b0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 1 -- 172.21.15.16:0/1101001096 shutdown_connections 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fbbbcb49f0 0x55fbbc0b8ff0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcf6070 0x55fbbc0b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcd5390 0x55fbbc0a6fa0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 1 --2- 172.21.15.16:0/1101001096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 1 -- 172.21.15.16:0/1101001096 >> 172.21.15.16:0/1101001096 conn(0x55fbbbcea460 msgr2=0x55fbbc09e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f5eff5640 0 client.16791 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 1 -- 172.21.15.16:0/1101001096 shutdown_connections 2024-08-28T19:25:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 1 -- 172.21.15.16:0/1101001096 wait complete. 2024-08-28T19:25:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.934+0000 7f2f8792a740 20 client.16791 trim_cache size 0 max 16384 2024-08-28T19:25:19.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 183682 4.51 MB/sec execute 59 sec latency 10.506 ms 2024-08-28T19:25:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.954+0000 7f2f8792a740 1 Processor -- start 2024-08-28T19:25:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.954+0000 7f2f8792a740 1 -- start start 2024-08-28T19:25:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.954+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.954+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 0x55fbbc0a5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.954+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc09e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.954+0000 7f2f8792a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c1b0b40 con 0x55fbbbcf6070 2024-08-28T19:25:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.954+0000 7f2f8792a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c1b2740 con 0x55fbbbccd1b0 2024-08-28T19:25:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.954+0000 7f2f8792a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c1b1270 con 0x55fbbbcd5390 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc09e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc09e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43416/0 (socket says 172.21.15.16:43416) 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7b7fe640 1 -- 172.21.15.16:0/1317156397 learned_addr learned my addr 172.21.15.16:0/1317156397 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/1317156397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 0x55fbbc0a5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7affd640 1 -- 172.21.15.16:0/1317156397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 msgr2=0x55fbbc0a5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7affd640 1 --2- 172.21.15.16:0/1317156397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 0x55fbbc0a5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7affd640 1 -- 172.21.15.16:0/1317156397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 msgr2=0x55fbbc09e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7affd640 1 --2- 172.21.15.16:0/1317156397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc09e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7affd640 1 -- 172.21.15.16:0/1317156397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbbc0a75c0 con 0x55fbbbccd1b0 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7b7fe640 1 --2- 172.21.15.16:0/1317156397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc09e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/1317156397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 0x55fbbc0a5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f7affd640 1 --2- 172.21.15.16:0/1317156397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0a52e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009ca0 tx=0x7f2f6c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d267afef6103d20b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f60ff9640 1 -- 172.21.15.16:0/1317156397 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c05d070 con 0x55fbbbccd1b0 2024-08-28T19:25:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f60ff9640 1 -- 172.21.15.16:0/1317156397 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f6c036040 con 0x55fbbbccd1b0 2024-08-28T19:25:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.955+0000 7f2f60ff9640 1 -- 172.21.15.16:0/1317156397 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c004150 con 0x55fbbbccd1b0 2024-08-28T19:25:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.956+0000 7f2f8792a740 1 -- 172.21.15.16:0/1317156397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 msgr2=0x55fbbc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.956+0000 7f2f8792a740 1 --2- 172.21.15.16:0/1317156397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0a52e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009ca0 tx=0x7f2f6c005fd0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.956+0000 7f2f8792a740 1 -- 172.21.15.16:0/1317156397 shutdown_connections 2024-08-28T19:25:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.956+0000 7f2f8792a740 1 --2- 172.21.15.16:0/1317156397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc09e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.956+0000 7f2f8792a740 1 --2- 172.21.15.16:0/1317156397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbcd5390 0x55fbbc0a5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.956+0000 7f2f8792a740 1 --2- 172.21.15.16:0/1317156397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0a52e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.956+0000 7f2f8792a740 1 -- 172.21.15.16:0/1317156397 >> 172.21.15.16:0/1317156397 conn(0x55fbbbcea460 msgr2=0x55fbbc0a1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.957+0000 7f2f8792a740 1 -- 172.21.15.16:0/1317156397 shutdown_connections 2024-08-28T19:25:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.957+0000 7f2f8792a740 1 -- 172.21.15.16:0/1317156397 wait complete. 2024-08-28T19:25:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.957+0000 7f2f8792a740 1 -- start start 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.957+0000 7f2f8792a740 1 Processor -- start 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0af150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbcd5390 0x55fbbc0aa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f8792a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc0aab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f8792a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c1b2070 con 0x55fbbbcf6070 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f8792a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c1b19a0 con 0x55fbbbcd5390 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f8792a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c1aeb20 con 0x55fbbbccd1b0 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f8792a740 10 client.0 _ll_register_callbacks cb 0x7f2f700e9500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7affd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc0aab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7affd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0af150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:46868/0 (socket says 172.21.15.16:46868) 2024-08-28T19:25:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7affd640 1 -- 172.21.15.16:0/3155980906 learned_addr learned my addr 172.21.15.16:0/3155980906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7b7fe640 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc0aab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43428/0 (socket says 172.21.15.16:43428) 2024-08-28T19:25:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbcd5390 0x55fbbc0aa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7affd640 1 -- 172.21.15.16:0/3155980906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbcd5390 msgr2=0x55fbbc0aa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7affd640 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbcd5390 0x55fbbc0aa600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7affd640 1 -- 172.21.15.16:0/3155980906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 msgr2=0x55fbbc0aab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7affd640 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc0aab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7affd640 1 -- 172.21.15.16:0/3155980906 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbbc0ab100 con 0x55fbbbccd1b0 2024-08-28T19:25:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:25:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.958+0000 7f2f7affd640 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0af150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c05b040 tx=0x7f2f6c000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38a49efe1cf2acd2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f5eff5640 1 -- 172.21.15.16:0/3155980906 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c066060 con 0x55fbbbccd1b0 2024-08-28T19:25:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f5eff5640 1 -- 172.21.15.16:0/3155980906 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f2f6c0315f0 con 0x55fbbbccd1b0 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f8792a740 1 -- 172.21.15.16:0/3155980906 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f6c036040 con 0x55fbbbccd1b0 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f5eff5640 1 -- 172.21.15.16:0/3155980906 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f2f6c06d050 con 0x55fbbbccd1b0 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f8792a740 10 client.26200 fetch_fsmap learned FSMap version 38 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f8792a740 1 -- 172.21.15.16:0/3155980906 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fbbc123730 con 0x55fbbbccd1b0 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f5eff5640 1 -- 172.21.15.16:0/3155980906 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f6c036040 con 0x55fbbbccd1b0 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f5eff5640 1 -- 172.21.15.16:0/3155980906 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 38) ==== 8386+0+0 (secure 0 0 0) 0x7f2f6c044070 con 0x55fbbbccd1b0 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f8792a740 10 client.26200 fetch_fsmap finished waiting for FSMap version 38 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f8792a740 10 client.26200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f8792a740 20 client.26200 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f8792a740 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fbbbc89da0 0x55fbbc125bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f8792a740 4 client.26200 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:25:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f8792a740 1 -- 172.21.15.16:0/3155980906 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fbbc010020 con 0x55fbbbc89da0 2024-08-28T19:25:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.959+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fbbbc89da0 0x55fbbc125bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f5eff5640 10 client.26200 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f7a7fc640 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fbbbc89da0 0x55fbbc125bd0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f16cdd147e9d2f34 server_cookie=1ad7f44bc46fdf28 in_seq=0 out_seq=0 2024-08-28T19:25:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f5eff5640 1 -- 172.21.15.16:0/3155980906 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2274 (crc 0 0 0) 0x55fbbc010020 con 0x55fbbbc89da0 2024-08-28T19:25:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f5eff5640 10 client.26200 handle_command_reply: tid=0 2024-08-28T19:25:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f8792a740 1 client.26200 shutdown 2024-08-28T19:25:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f8792a740 1 -- 172.21.15.16:0/3155980906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbccd1b0 msgr2=0x55fbbc0af150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f8792a740 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0af150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c05b040 tx=0x7f2f6c000f40 comp rx=0 tx=0).stop 2024-08-28T19:25:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f8792a740 1 -- 172.21.15.16:0/3155980906 shutdown_connections 2024-08-28T19:25:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f8792a740 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbbbcf6070 0x55fbbc0aab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f8792a740 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fbbbcd5390 0x55fbbc0aa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f8792a740 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fbbbc89da0 0x55fbbc125bd0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f8792a740 1 --2- 172.21.15.16:0/3155980906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fbbbccd1b0 0x55fbbc0af150 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f5eff5640 0 client.26200 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f8792a740 1 -- 172.21.15.16:0/3155980906 >> 172.21.15.16:0/3155980906 conn(0x55fbbbcea460 msgr2=0x55fbbc0a1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.960+0000 7f2f8792a740 1 -- 172.21.15.16:0/3155980906 shutdown_connections 2024-08-28T19:25:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f8792a740 1 -- 172.21.15.16:0/3155980906 wait complete. 2024-08-28T19:25:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f8792a740 20 client.26200 trim_cache size 0 max 16384 2024-08-28T19:25:19.968 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:25:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "7af73992-6dbf-427c-84e6-e23aa2c95123", 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:25:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:19.973 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:25:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f813b5640 1 -- 172.21.15.16:0/4188898565 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2f640b4ca0 msgr2=0x7f2f640b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f813b5640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2f640b4ca0 0x7f2f640b71b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2f74012640 tx=0x7f2f74005f90 comp rx=0 tx=0).stop 2024-08-28T19:25:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f813b5640 1 -- 172.21.15.16:0/4188898565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c108150 msgr2=0x7f2f7c19a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f813b5640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c108150 0x7f2f7c19a7c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000e9c0 tx=0x7f2f7000ee80 comp rx=0 tx=0).stop 2024-08-28T19:25:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f813b5640 1 -- 172.21.15.16:0/4188898565 shutdown_connections 2024-08-28T19:25:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f813b5640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2f640b4ca0 0x7f2f640b71b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f813b5640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f7c108aa0 0x7f2f7c19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f813b5640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c108150 0x7f2f7c19a7c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f813b5640 1 --2- 172.21.15.16:0/4188898565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2f7c102810 0x7f2f7c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.961+0000 7f2f813b5640 1 -- 172.21.15.16:0/4188898565 >> 172.21.15.16:0/4188898565 conn(0x7f2f7c0794d0 msgr2=0x7f2f7c101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.962+0000 7f2f813b5640 1 -- 172.21.15.16:0/4188898565 shutdown_connections 2024-08-28T19:25:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:19.962+0000 7f2f813b5640 1 -- 172.21.15.16:0/4188898565 wait complete. 2024-08-28T19:25:20.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:25:20.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:25:20.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:20.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:20.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:25:20.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:20.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:25:20.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:25:20.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:20.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:20.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:25:20.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:20.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:25:20.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:25:20.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:20.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:20.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:25:20.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:25:20.812 INFO:tasks.cephfs.filesystem:scrub status for tag:7af73992-6dbf-427c-84e6-e23aa2c95123 - {'path': '/', 'tag': '7af73992-6dbf-427c-84e6-e23aa2c95123', 'options': 'recursive,force'} 2024-08-28T19:25:20.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 185178 4.57 MB/sec execute 60 sec latency 15.952 ms 2024-08-28T19:25:21.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:20 smithi016 ceph-mon[18556]: pgmap v468: 129 pgs: 129 active+clean; 101 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 321 KiB/s wr, 55 op/s 2024-08-28T19:25:21.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:20 smithi028 ceph-mon[21769]: pgmap v468: 129 pgs: 129 active+clean; 101 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 321 KiB/s wr, 55 op/s 2024-08-28T19:25:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:20 smithi115 ceph-mon[22184]: pgmap v468: 129 pgs: 129 active+clean; 101 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 321 KiB/s wr, 55 op/s 2024-08-28T19:25:21.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 187832 4.73 MB/sec execute 61 sec latency 10.797 ms 2024-08-28T19:25:22.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 189237 4.77 MB/sec execute 62 sec latency 14.826 ms 2024-08-28T19:25:23.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:22 smithi016 ceph-mon[18556]: pgmap v469: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 841 KiB/s wr, 143 op/s 2024-08-28T19:25:23.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:22 smithi028 ceph-mon[21769]: pgmap v469: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 841 KiB/s wr, 143 op/s 2024-08-28T19:25:23.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:22 smithi115 ceph-mon[22184]: pgmap v469: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 841 KiB/s wr, 143 op/s 2024-08-28T19:25:23.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 191783 4.95 MB/sec execute 63 sec latency 24.773 ms 2024-08-28T19:25:24.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 193796 4.97 MB/sec execute 64 sec latency 11.738 ms 2024-08-28T19:25:25.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:24 smithi016 ceph-mon[18556]: pgmap v470: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 851 KiB/s wr, 139 op/s 2024-08-28T19:25:25.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:24 smithi028 ceph-mon[21769]: pgmap v470: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 851 KiB/s wr, 139 op/s 2024-08-28T19:25:25.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:24 smithi115 ceph-mon[22184]: pgmap v470: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 851 KiB/s wr, 139 op/s 2024-08-28T19:25:25.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 195537 5.10 MB/sec execute 65 sec latency 19.854 ms 2024-08-28T19:25:26.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 197861 5.19 MB/sec execute 66 sec latency 11.148 ms 2024-08-28T19:25:27.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:26 smithi016 ceph-mon[18556]: pgmap v471: 129 pgs: 129 active+clean; 111 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-08-28T19:25:27.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:26 smithi028 ceph-mon[21769]: pgmap v471: 129 pgs: 129 active+clean; 111 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-08-28T19:25:27.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:26 smithi115 ceph-mon[22184]: pgmap v471: 129 pgs: 129 active+clean; 111 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-08-28T19:25:27.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 199360 5.24 MB/sec execute 67 sec latency 12.822 ms 2024-08-28T19:25:28.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 202001 5.37 MB/sec execute 68 sec latency 14.966 ms 2024-08-28T19:25:29.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:28 smithi016 ceph-mon[18556]: pgmap v472: 129 pgs: 129 active+clean; 111 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-08-28T19:25:29.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:28 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:25:29.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:28 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:25:29.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:28 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:25:29.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:28 smithi028 ceph-mon[21769]: pgmap v472: 129 pgs: 129 active+clean; 111 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-08-28T19:25:29.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:28 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:25:29.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:28 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:25:29.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:28 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:25:29.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:28 smithi115 ceph-mon[22184]: pgmap v472: 129 pgs: 129 active+clean; 111 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-08-28T19:25:29.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:28 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:25:29.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:28 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:25:29.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:28 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:25:29.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 203417 5.39 MB/sec execute 69 sec latency 12.674 ms 2024-08-28T19:25:30.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:29 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:25:30.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:29 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:25:30.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:29 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:25:30.377 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:29 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:25:30.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:29 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:25:30.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:29 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:25:30.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 206033 5.55 MB/sec execute 70 sec latency 11.699 ms 2024-08-28T19:25:31.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:30 smithi016 ceph-mon[18556]: pgmap v473: 129 pgs: 129 active+clean; 116 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 259 op/s 2024-08-28T19:25:31.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:30 smithi028 ceph-mon[21769]: pgmap v473: 129 pgs: 129 active+clean; 116 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 259 op/s 2024-08-28T19:25:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:30 smithi115 ceph-mon[22184]: pgmap v473: 129 pgs: 129 active+clean; 116 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 259 op/s 2024-08-28T19:25:31.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 208075 5.58 MB/sec execute 71 sec latency 15.646 ms 2024-08-28T19:25:32.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 209746 5.67 MB/sec execute 72 sec latency 20.110 ms 2024-08-28T19:25:33.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:32 smithi016 ceph-mon[18556]: pgmap v474: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 305 op/s 2024-08-28T19:25:33.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:32 smithi028 ceph-mon[21769]: pgmap v474: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 305 op/s 2024-08-28T19:25:33.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:32 smithi115 ceph-mon[22184]: pgmap v474: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 305 op/s 2024-08-28T19:25:33.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 212171 5.75 MB/sec execute 73 sec latency 14.086 ms 2024-08-28T19:25:34.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213599 5.78 MB/sec execute 74 sec latency 15.082 ms 2024-08-28T19:25:35.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:34 smithi016 ceph-mon[18556]: pgmap v475: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 219 op/s 2024-08-28T19:25:35.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:34 smithi028 ceph-mon[21769]: pgmap v475: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 219 op/s 2024-08-28T19:25:35.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:34 smithi115 ceph-mon[22184]: pgmap v475: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 219 op/s 2024-08-28T19:25:35.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 216250 5.89 MB/sec execute 75 sec latency 11.099 ms 2024-08-28T19:25:36.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:36 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:36.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:36 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:36.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:36 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:36.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 217643 5.90 MB/sec execute 76 sec latency 11.394 ms 2024-08-28T19:25:37.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:37 smithi028 ceph-mon[21769]: pgmap v476: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-08-28T19:25:37.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:37 smithi115 ceph-mon[22184]: pgmap v476: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-08-28T19:25:37.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:37 smithi016 ceph-mon[18556]: pgmap v476: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-08-28T19:25:37.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220223 6.04 MB/sec execute 77 sec latency 10.128 ms 2024-08-28T19:25:38.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 222326 6.06 MB/sec execute 78 sec latency 11.861 ms 2024-08-28T19:25:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:39 smithi028 ceph-mon[21769]: pgmap v477: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 241 op/s 2024-08-28T19:25:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:39 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:25:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:39 smithi115 ceph-mon[22184]: pgmap v477: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 241 op/s 2024-08-28T19:25:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:39 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:25:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:39 smithi016 ceph-mon[18556]: pgmap v477: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 241 op/s 2024-08-28T19:25:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:39 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:25:39.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 223922 6.13 MB/sec execute 79 sec latency 27.323 ms 2024-08-28T19:25:40.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:25:40.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:25:40.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:25:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:25:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:25:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:25:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:25:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:25:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:25:40.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 226095 6.16 MB/sec execute 80 sec latency 12.100 ms 2024-08-28T19:25:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:41 smithi028 ceph-mon[21769]: pgmap v478: 129 pgs: 129 active+clean; 131 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 293 op/s 2024-08-28T19:25:41.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:41 smithi115 ceph-mon[22184]: pgmap v478: 129 pgs: 129 active+clean; 131 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 293 op/s 2024-08-28T19:25:41.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:41 smithi016 ceph-mon[18556]: pgmap v478: 129 pgs: 129 active+clean; 131 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 293 op/s 2024-08-28T19:25:41.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 227675 6.22 MB/sec execute 81 sec latency 9.908 ms 2024-08-28T19:25:42.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 229971 6.30 MB/sec execute 82 sec latency 11.638 ms 2024-08-28T19:25:43.302 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:43 smithi016 ceph-mon[18556]: pgmap v479: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 275 op/s 2024-08-28T19:25:43.308 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:43 smithi028 ceph-mon[21769]: pgmap v479: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 275 op/s 2024-08-28T19:25:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:43 smithi115 ceph-mon[22184]: pgmap v479: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 275 op/s 2024-08-28T19:25:43.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 231326 6.30 MB/sec execute 83 sec latency 19.824 ms 2024-08-28T19:25:44.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 233534 6.37 MB/sec execute 84 sec latency 15.335 ms 2024-08-28T19:25:45.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:45 smithi028 ceph-mon[21769]: pgmap v480: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 212 op/s 2024-08-28T19:25:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:45 smithi115 ceph-mon[22184]: pgmap v480: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 212 op/s 2024-08-28T19:25:45.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:45 smithi016 ceph-mon[18556]: pgmap v480: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 212 op/s 2024-08-28T19:25:45.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234928 6.38 MB/sec execute 85 sec latency 16.644 ms 2024-08-28T19:25:46.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 237517 6.47 MB/sec execute 86 sec latency 11.621 ms 2024-08-28T19:25:47.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:47 smithi028 ceph-mon[21769]: pgmap v481: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 316 op/s 2024-08-28T19:25:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:47 smithi115 ceph-mon[22184]: pgmap v481: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 316 op/s 2024-08-28T19:25:47.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:47 smithi016 ceph-mon[18556]: pgmap v481: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 316 op/s 2024-08-28T19:25:47.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 238765 6.47 MB/sec execute 87 sec latency 12.761 ms 2024-08-28T19:25:48.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 241378 6.58 MB/sec execute 88 sec latency 9.681 ms 2024-08-28T19:25:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:49 smithi028 ceph-mon[21769]: pgmap v482: 129 pgs: 129 active+clean; 140 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 276 op/s 2024-08-28T19:25:49.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:49 smithi115 ceph-mon[22184]: pgmap v482: 129 pgs: 129 active+clean; 140 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 276 op/s 2024-08-28T19:25:49.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:49 smithi016 ceph-mon[18556]: pgmap v482: 129 pgs: 129 active+clean; 140 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 276 op/s 2024-08-28T19:25:49.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 242477 6.54 MB/sec execute 89 sec latency 61.469 ms 2024-08-28T19:25:50.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:25:50.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 245000 6.66 MB/sec execute 90 sec latency 18.719 ms 2024-08-28T19:25:51.120 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:51.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:51 smithi016 ceph-mon[18556]: pgmap v483: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 256 op/s 2024-08-28T19:25:51.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:51 smithi028 ceph-mon[21769]: pgmap v483: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 256 op/s 2024-08-28T19:25:51.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:51 smithi115 ceph-mon[22184]: pgmap v483: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 256 op/s 2024-08-28T19:25:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.772+0000 7f497416a640 1 -- 172.21.15.16:0/971867394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f496c102840 msgr2=0x7f496c102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.772+0000 7f497416a640 1 --2- 172.21.15.16:0/971867394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f496c102840 0x7f496c102c50 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f4968009900 tx=0x7f496802a410 comp rx=0 tx=0).stop 2024-08-28T19:25:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.772+0000 7f497416a640 1 -- 172.21.15.16:0/971867394 shutdown_connections 2024-08-28T19:25:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.772+0000 7f497416a640 1 --2- 172.21.15.16:0/971867394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c108ad0 0x7f496c10f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.772+0000 7f497416a640 1 --2- 172.21.15.16:0/971867394 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f496c108180 0x7f496c108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.772+0000 7f497416a640 1 --2- 172.21.15.16:0/971867394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f496c102840 0x7f496c102c50 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.772+0000 7f497416a640 1 -- 172.21.15.16:0/971867394 >> 172.21.15.16:0/971867394 conn(0x7f496c079580 msgr2=0x7f496c079990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.772+0000 7f497416a640 1 -- 172.21.15.16:0/971867394 shutdown_connections 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.772+0000 7f497416a640 1 -- 172.21.15.16:0/971867394 wait complete. 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.773+0000 7f497416a640 1 Processor -- start 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.773+0000 7f497416a640 1 -- start start 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.773+0000 7f497416a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f496c102840 0x7f496c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f497416a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f496c108180 0x7f496c19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f497416a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c108ad0 0x7f496c19fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f497416a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f496c113850 con 0x7f496c102840 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f497416a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1136d0 con 0x7f496c108ad0 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f497416a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1139d0 con 0x7f496c108180 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f4971f06640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f496c102840 0x7f496c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f4971f06640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f496c102840 0x7f496c19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34886/0 (socket says 172.21.15.16:34886) 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f4971f06640 1 -- 172.21.15.16:0/1805143339 learned_addr learned my addr 172.21.15.16:0/1805143339 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f4971705640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f496c108180 0x7f496c19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f4972707640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c108ad0 0x7f496c19fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f4971f06640 1 -- 172.21.15.16:0/1805143339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f496c108180 msgr2=0x7f496c19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f4971f06640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f496c108180 0x7f496c19a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f4971f06640 1 -- 172.21.15.16:0/1805143339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c108ad0 msgr2=0x7f496c19fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f4971f06640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c108ad0 0x7f496c19fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f4971f06640 1 -- 172.21.15.16:0/1805143339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c1a01b0 con 0x7f496c102840 2024-08-28T19:25:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.774+0000 7f4971f06640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f496c102840 0x7f496c19a240 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f4968005940 tx=0x7f49680040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.775+0000 7f4962ffd640 1 -- 172.21.15.16:0/1805143339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496803f030 con 0x7f496c102840 2024-08-28T19:25:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.775+0000 7f497416a640 1 -- 172.21.15.16:0/1805143339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496c1a0440 con 0x7f496c102840 2024-08-28T19:25:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.775+0000 7f4962ffd640 1 -- 172.21.15.16:0/1805143339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49680044c0 con 0x7f496c102840 2024-08-28T19:25:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.775+0000 7f4962ffd640 1 -- 172.21.15.16:0/1805143339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4968035070 con 0x7f496c102840 2024-08-28T19:25:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.775+0000 7f497416a640 1 -- 172.21.15.16:0/1805143339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f496c1a6c00 con 0x7f496c102840 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f497a6df740 1 Processor -- start 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f497a6df740 1 -- start start 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f497a6df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 0x55db2ce522c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f4962ffd640 1 -- 172.21.15.16:0/1805143339 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4968042030 con 0x7f496c102840 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f497a6df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca841b0 0x55db2ce52800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f497a6df740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 0x55db2ce52d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f497a6df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55db2cdc9630 con 0x55db2caad070 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f497a6df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db2cdb8bc0 con 0x55db2ca841b0 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f497a6df740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55db2cdbcb70 con 0x55db2ca8c390 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f4972707640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 0x55db2ce522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f4971f06640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca841b0 0x55db2ce52800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f4972707640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 0x55db2ce522c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34900/0 (socket says 172.21.15.16:34900) 2024-08-28T19:25:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f4972707640 1 -- 172.21.15.16:0/2602264475 learned_addr learned my addr 172.21.15.16:0/2602264475 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f4971f06640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca841b0 0x55db2ce52800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:38998/0 (socket says 172.21.15.16:38998) 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.778+0000 7f4971705640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 0x55db2ce52d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.779+0000 7f4972707640 1 -- 172.21.15.16:0/2602264475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 msgr2=0x55db2ce52d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.779+0000 7f4972707640 1 --2- 172.21.15.16:0/2602264475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 0x55db2ce52d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.779+0000 7f4972707640 1 -- 172.21.15.16:0/2602264475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca841b0 msgr2=0x55db2ce52800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.779+0000 7f4972707640 1 --2- 172.21.15.16:0/2602264475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca841b0 0x55db2ce52800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.779+0000 7f4972707640 1 -- 172.21.15.16:0/2602264475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db2cb18650 con 0x55db2caad070 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.779+0000 7f4971f06640 1 --2- 172.21.15.16:0/2602264475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca841b0 0x55db2ce52800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.779+0000 7f4971705640 1 --2- 172.21.15.16:0/2602264475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 0x55db2ce52d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.779+0000 7f4972707640 1 --2- 172.21.15.16:0/2602264475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 0x55db2ce522c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4964009fd0 tx=0x7f49640118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4132debbe1428f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.779+0000 7f4957fff640 1 -- 172.21.15.16:0/2602264475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964004d20 con 0x55db2caad070 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.779+0000 7f4957fff640 1 -- 172.21.15.16:0/2602264475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4964004050 con 0x55db2caad070 2024-08-28T19:25:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.779+0000 7f4957fff640 1 -- 172.21.15.16:0/2602264475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964005220 con 0x55db2caad070 2024-08-28T19:25:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.780+0000 7f4962ffd640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f49580b4d20 0x7f49580b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.780+0000 7f4971f06640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f49580b4d20 0x7f49580b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.780+0000 7f4962ffd640 1 -- 172.21.15.16:0/1805143339 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f4968105b80 con 0x7f496c102840 2024-08-28T19:25:51.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f497a6df740 1 -- 172.21.15.16:0/2602264475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 msgr2=0x55db2ce522c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f4971f06640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f49580b4d20 0x7f49580b7230 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4968103010 tx=0x7f4968110990 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f497a6df740 1 --2- 172.21.15.16:0/2602264475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 0x55db2ce522c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4964009fd0 tx=0x7f49640118d0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f497a6df740 1 -- 172.21.15.16:0/2602264475 shutdown_connections 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f497a6df740 1 --2- 172.21.15.16:0/2602264475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 0x55db2ce52d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f497a6df740 1 --2- 172.21.15.16:0/2602264475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca841b0 0x55db2ce52800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f497a6df740 1 --2- 172.21.15.16:0/2602264475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 0x55db2ce522c0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f497a6df740 1 -- 172.21.15.16:0/2602264475 >> 172.21.15.16:0/2602264475 conn(0x55db2caa1460 msgr2=0x55db2ce501b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f497a6df740 1 -- 172.21.15.16:0/2602264475 shutdown_connections 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f497a6df740 1 -- 172.21.15.16:0/2602264475 wait complete. 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f497a6df740 1 -- start start 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.781+0000 7f497a6df740 1 Processor -- start 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f497a6df740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca841b0 0x55db2ce5d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f497a6df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca8c390 0x55db2ce5def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f497a6df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 0x55db2ce69dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f497a6df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55db2cc1b700 con 0x55db2caad070 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f497a6df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db2cd3f700 con 0x55db2ca8c390 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f497a6df740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55db2ce60aa0 con 0x55db2ca841b0 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f497a6df740 10 client.0 _ll_register_callbacks cb 0x55db2cd6edc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f4972707640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca841b0 0x55db2ce5d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f4971705640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca8c390 0x55db2ce5def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f4972707640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca841b0 0x55db2ce5d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:38434/0 (socket says 172.21.15.16:38434) 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f4972707640 1 -- 172.21.15.16:0/1101682246 learned_addr learned my addr 172.21.15.16:0/1101682246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f4972707640 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 0x55db2ce69dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f4971705640 1 -- 172.21.15.16:0/1101682246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca841b0 msgr2=0x55db2ce5d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f4971705640 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca841b0 0x55db2ce5d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f4971705640 1 -- 172.21.15.16:0/1101682246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 msgr2=0x55db2ce69dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f4971705640 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 0x55db2ce69dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f4971705640 1 -- 172.21.15.16:0/1101682246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db2cd788b0 con 0x55db2ca8c390 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f4971705640 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca8c390 0x55db2ce5def0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f496c06a840 tx=0x7f495c012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4b967341d6a3799 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.782+0000 7f49557fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f49557fa640 1 -- 172.21.15.16:0/1101682246 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c004030 con 0x55db2ca8c390 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f497a6df740 1 -- 172.21.15.16:0/1101682246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4964004050 con 0x55db2ca8c390 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f49557fa640 1 -- 172.21.15.16:0/1101682246 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f495c0041d0 con 0x55db2ca8c390 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f49557fa640 1 -- 172.21.15.16:0/1101682246 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f495c025050 con 0x55db2ca8c390 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f497a6df740 10 client.26381 fetch_fsmap learned FSMap version 39 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f497a6df740 1 -- 172.21.15.16:0/1101682246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55db2ce6c810 con 0x55db2ca8c390 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f49557fa640 1 -- 172.21.15.16:0/1101682246 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4964004050 con 0x55db2ca8c390 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f49557fa640 1 -- 172.21.15.16:0/1101682246 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) ==== 8390+0+0 (secure 0 0 0) 0x7f495c003930 con 0x55db2ca8c390 2024-08-28T19:25:51.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f497a6df740 10 client.26381 fetch_fsmap finished waiting for FSMap version 39 2024-08-28T19:25:51.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f497a6df740 10 client.26381 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:25:51.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f497a6df740 20 client.26381 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:51.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f497a6df740 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db2ca6b9f0 0x55db2ce6fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f497a6df740 4 client.26381 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:51.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.783+0000 7f497a6df740 1 -- 172.21.15.16:0/1101682246 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db2cdcb580 con 0x55db2ca6b9f0 2024-08-28T19:25:51.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.784+0000 7f4972707640 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db2ca6b9f0 0x55db2ce6fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.784+0000 7f49557fa640 10 client.26381 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:51.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.784+0000 7f4972707640 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db2ca6b9f0 0x55db2ce6fdb0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91f98802e2bd3b25 server_cookie=a43e654467ddc751 in_seq=0 out_seq=0 2024-08-28T19:25:51.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f49557fa640 1 -- 172.21.15.16:0/1101682246 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55db2cdcb580 con 0x55db2ca6b9f0 2024-08-28T19:25:51.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f49557fa640 10 client.26381 handle_command_reply: tid=0 2024-08-28T19:25:51.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 1 client.26381 shutdown 2024-08-28T19:25:51.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 1 -- 172.21.15.16:0/1101682246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca8c390 msgr2=0x55db2ce5def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:51.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca8c390 0x55db2ce5def0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f496c06a840 tx=0x7f495c012b60 comp rx=0 tx=0).stop 2024-08-28T19:25:51.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 1 -- 172.21.15.16:0/1101682246 shutdown_connections 2024-08-28T19:25:51.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2caad070 0x55db2ce69dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2ca8c390 0x55db2ce5def0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db2ca6b9f0 0x55db2ce6fdb0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 1 --2- 172.21.15.16:0/1101682246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca841b0 0x55db2ce5d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f49557fa640 0 client.26381 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:51.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 1 -- 172.21.15.16:0/1101682246 >> 172.21.15.16:0/1101682246 conn(0x55db2caa1460 msgr2=0x55db2ce554a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:51.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 1 -- 172.21.15.16:0/1101682246 shutdown_connections 2024-08-28T19:25:51.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 1 -- 172.21.15.16:0/1101682246 wait complete. 2024-08-28T19:25:51.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.785+0000 7f497a6df740 20 client.26381 trim_cache size 0 max 16384 2024-08-28T19:25:51.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f497a6df740 1 Processor -- start 2024-08-28T19:25:51.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f497a6df740 1 -- start start 2024-08-28T19:25:51.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f497a6df740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca841b0 0x55db2ce55ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f497a6df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca8c390 0x55db2ce69cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f497a6df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2caad070 0x55db2ce4fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f497a6df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1ae970 con 0x55db2ca8c390 2024-08-28T19:25:51.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f497a6df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1b05a0 con 0x55db2caad070 2024-08-28T19:25:51.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f497a6df740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1af0a0 con 0x55db2ca841b0 2024-08-28T19:25:51.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f4971705640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca8c390 0x55db2ce69cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f4971705640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca8c390 0x55db2ce69cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34906/0 (socket says 172.21.15.16:34906) 2024-08-28T19:25:51.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f4971705640 1 -- 172.21.15.16:0/4009144412 learned_addr learned my addr 172.21.15.16:0/4009144412 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:51.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f4972707640 1 --2- 172.21.15.16:0/4009144412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca841b0 0x55db2ce55ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f4972707640 1 --2- 172.21.15.16:0/4009144412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2caad070 0x55db2ce4fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f4971705640 1 -- 172.21.15.16:0/4009144412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca841b0 msgr2=0x55db2ce55ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f4971705640 1 --2- 172.21.15.16:0/4009144412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca841b0 0x55db2ce55ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f4971705640 1 -- 172.21.15.16:0/4009144412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2caad070 msgr2=0x55db2ce4fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f4971705640 1 --2- 172.21.15.16:0/4009144412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2caad070 0x55db2ce4fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.804+0000 7f4971705640 1 -- 172.21.15.16:0/4009144412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db2ce55120 con 0x55db2ca8c390 2024-08-28T19:25:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.805+0000 7f4971705640 1 --2- 172.21.15.16:0/4009144412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca8c390 0x55db2ce69cc0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f495c009de0 tx=0x7f495c010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a08fe94a0f225e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.805+0000 7f49577fe640 1 -- 172.21.15.16:0/4009144412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c022040 con 0x55db2ca8c390 2024-08-28T19:25:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.805+0000 7f49577fe640 1 -- 172.21.15.16:0/4009144412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f495c003aa0 con 0x55db2ca8c390 2024-08-28T19:25:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.805+0000 7f49577fe640 1 -- 172.21.15.16:0/4009144412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c005370 con 0x55db2ca8c390 2024-08-28T19:25:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.806+0000 7f497a6df740 1 -- 172.21.15.16:0/4009144412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca8c390 msgr2=0x55db2ce69cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.806+0000 7f497a6df740 1 --2- 172.21.15.16:0/4009144412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca8c390 0x55db2ce69cc0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f495c009de0 tx=0x7f495c010430 comp rx=0 tx=0).stop 2024-08-28T19:25:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.806+0000 7f497a6df740 1 -- 172.21.15.16:0/4009144412 shutdown_connections 2024-08-28T19:25:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.806+0000 7f497a6df740 1 --2- 172.21.15.16:0/4009144412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2caad070 0x55db2ce4fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.806+0000 7f497a6df740 1 --2- 172.21.15.16:0/4009144412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca8c390 0x55db2ce69cc0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.806+0000 7f497a6df740 1 --2- 172.21.15.16:0/4009144412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca841b0 0x55db2ce55ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.806+0000 7f497a6df740 1 -- 172.21.15.16:0/4009144412 >> 172.21.15.16:0/4009144412 conn(0x55db2caa1460 msgr2=0x55db2ce553c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.806+0000 7f497a6df740 1 -- 172.21.15.16:0/4009144412 shutdown_connections 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.806+0000 7f497a6df740 1 -- 172.21.15.16:0/4009144412 wait complete. 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.806+0000 7f497a6df740 1 -- start start 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.806+0000 7f497a6df740 1 Processor -- start 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f497a6df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca841b0 0x55db2ce62680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f497a6df740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 0x55db2ce62c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f497a6df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2caad070 0x55db2ced3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f497a6df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1afed0 con 0x55db2ca841b0 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f497a6df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1af7d0 con 0x55db2caad070 2024-08-28T19:25:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f497a6df740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1ac950 con 0x55db2ca8c390 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f497a6df740 10 client.0 _ll_register_callbacks cb 0x7f495c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f4972707640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca841b0 0x55db2ce62680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f4972707640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca841b0 0x55db2ce62680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34920/0 (socket says 172.21.15.16:34920) 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f4972707640 1 -- 172.21.15.16:0/2385498846 learned_addr learned my addr 172.21.15.16:0/2385498846 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f4971705640 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 0x55db2ce62c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f4972707640 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2caad070 0x55db2ced3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f4971705640 1 -- 172.21.15.16:0/2385498846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2caad070 msgr2=0x55db2ced3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f4971705640 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2caad070 0x55db2ced3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f4971705640 1 -- 172.21.15.16:0/2385498846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca841b0 msgr2=0x55db2ce62680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f4971705640 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca841b0 0x55db2ce62680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f4971705640 1 -- 172.21.15.16:0/2385498846 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db2ce53100 con 0x55db2ca8c390 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.807+0000 7f4972707640 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca841b0 0x55db2ce62680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.808+0000 7f4971705640 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 0x55db2ce62c40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x55db2cd581d0 tx=0x7f495c003880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98d985887e37db49 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.808+0000 7f49557fa640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.808+0000 7f49557fa640 1 -- 172.21.15.16:0/2385498846 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c00fb30 con 0x55db2ca8c390 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.808+0000 7f497a6df740 1 -- 172.21.15.16:0/2385498846 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f495c003aa0 con 0x55db2ca8c390 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.808+0000 7f49557fa640 1 -- 172.21.15.16:0/2385498846 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f495c00fcd0 con 0x55db2ca8c390 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.808+0000 7f49557fa640 1 -- 172.21.15.16:0/2385498846 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f495c03b050 con 0x55db2ca8c390 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.808+0000 7f497a6df740 10 client.26218 fetch_fsmap learned FSMap version 39 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.808+0000 7f497a6df740 1 -- 172.21.15.16:0/2385498846 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55db2cedaf70 con 0x55db2ca8c390 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.808+0000 7f49557fa640 1 -- 172.21.15.16:0/2385498846 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f495c003aa0 con 0x55db2ca8c390 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.808+0000 7f49557fa640 1 -- 172.21.15.16:0/2385498846 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 39) ==== 8390+0+0 (secure 0 0 0) 0x7f495c010970 con 0x55db2ca8c390 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f497a6df740 10 client.26218 fetch_fsmap finished waiting for FSMap version 39 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f497a6df740 10 client.26218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f497a6df740 20 client.26218 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f497a6df740 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db2ca40da0 0x55db2cedd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f497a6df740 4 client.26218 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f497a6df740 1 -- 172.21.15.16:0/2385498846 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55db2cdc7020 con 0x55db2ca40da0 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f4972707640 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db2ca40da0 0x55db2cedd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f49557fa640 10 client.26218 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f4972707640 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db2ca40da0 0x55db2cedd5f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dde63696f2a2679 server_cookie=782b4fd8e665322 in_seq=0 out_seq=0 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f49557fa640 1 -- 172.21.15.16:0/2385498846 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2078 (crc 0 0 0) 0x55db2cdc7020 con 0x55db2ca40da0 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f49557fa640 10 client.26218 handle_command_reply: tid=0 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.809+0000 7f497a6df740 1 client.26218 shutdown 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f497a6df740 1 -- 172.21.15.16:0/2385498846 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 msgr2=0x55db2ce62c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f497a6df740 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 0x55db2ce62c40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x55db2cd581d0 tx=0x7f495c003880 comp rx=0 tx=0).stop 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f497a6df740 1 -- 172.21.15.16:0/2385498846 shutdown_connections 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f497a6df740 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2caad070 0x55db2ced3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f497a6df740 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db2ca40da0 0x55db2cedd5f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f497a6df740 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db2ca8c390 0x55db2ce62c40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f497a6df740 1 --2- 172.21.15.16:0/2385498846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db2ca841b0 0x55db2ce62680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f49557fa640 0 client.26218 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:25:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f497a6df740 1 -- 172.21.15.16:0/2385498846 >> 172.21.15.16:0/2385498846 conn(0x55db2caa1460 msgr2=0x55db2ce4f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f497a6df740 1 -- 172.21.15.16:0/2385498846 shutdown_connections 2024-08-28T19:25:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f497a6df740 1 -- 172.21.15.16:0/2385498846 wait complete. 2024-08-28T19:25:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.810+0000 7f497a6df740 20 client.26218 trim_cache size 0 max 16384 2024-08-28T19:25:51.817 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:25:51.817 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:25:51.817 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:25:51.817 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:25:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:25:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:25:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:25:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:25:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:51.819 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:51.820 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:51.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:25:51.822 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:25:51.822 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:25:51.822 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:25:51.822 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:25:51.822 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:25:51.822 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:25:51.822 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:25:51.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:51.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:25:51.822 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:25:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 -- 172.21.15.16:0/1805143339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f49580b4d20 msgr2=0x7f49580b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f49580b4d20 0x7f49580b7230 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4968103010 tx=0x7f4968110990 comp rx=0 tx=0).stop 2024-08-28T19:25:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 -- 172.21.15.16:0/1805143339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f496c102840 msgr2=0x7f496c19a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f496c102840 0x7f496c19a240 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f4968005940 tx=0x7f49680040c0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 -- 172.21.15.16:0/1805143339 shutdown_connections 2024-08-28T19:25:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f49580b4d20 0x7f49580b7230 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c108ad0 0x7f496c19fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f496c108180 0x7f496c19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 --2- 172.21.15.16:0/1805143339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f496c102840 0x7f496c19a240 secure :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f4968005940 tx=0x7f49680040c0 comp rx=0 tx=0).stop 2024-08-28T19:25:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 -- 172.21.15.16:0/1805143339 >> 172.21.15.16:0/1805143339 conn(0x7f496c079580 msgr2=0x7f496c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 -- 172.21.15.16:0/1805143339 shutdown_connections 2024-08-28T19:25:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:51.811+0000 7f497416a640 1 -- 172.21.15.16:0/1805143339 wait complete. 2024-08-28T19:25:51.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 246309 6.64 MB/sec execute 91 sec latency 27.594 ms 2024-08-28T19:25:52.753 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:25:52.753 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:25:52.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 248651 6.72 MB/sec execute 92 sec latency 12.007 ms 2024-08-28T19:25:53.023 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:53 smithi028 ceph-mon[21769]: pgmap v484: 129 pgs: 129 active+clean; 147 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 268 op/s 2024-08-28T19:25:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:53 smithi115 ceph-mon[22184]: pgmap v484: 129 pgs: 129 active+clean; 147 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 268 op/s 2024-08-28T19:25:53.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:53 smithi016 ceph-mon[18556]: pgmap v484: 129 pgs: 129 active+clean; 147 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 268 op/s 2024-08-28T19:25:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.808+0000 7fabe6227640 1 -- 172.21.15.16:0/4240275488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabe01045e0 msgr2=0x7fabe01049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.808+0000 7fabe6227640 1 --2- 172.21.15.16:0/4240275488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabe01045e0 0x7fabe01049f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fabc8009900 tx=0x7fabc802a410 comp rx=0 tx=0).stop 2024-08-28T19:25:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.808+0000 7fabe6227640 1 -- 172.21.15.16:0/4240275488 shutdown_connections 2024-08-28T19:25:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.808+0000 7fabe6227640 1 --2- 172.21.15.16:0/4240275488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabe010db90 0x7fabe01100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.808+0000 7fabe6227640 1 --2- 172.21.15.16:0/4240275488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe0105010 0x7fabe010d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.808+0000 7fabe6227640 1 --2- 172.21.15.16:0/4240275488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabe01045e0 0x7fabe01049f0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.808+0000 7fabe6227640 1 -- 172.21.15.16:0/4240275488 >> 172.21.15.16:0/4240275488 conn(0x7fabe00794d0 msgr2=0x7fabe00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.809+0000 7fabe6227640 1 -- 172.21.15.16:0/4240275488 shutdown_connections 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.809+0000 7fabe6227640 1 -- 172.21.15.16:0/4240275488 wait complete. 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.809+0000 7fabe6227640 1 Processor -- start 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.809+0000 7fabe6227640 1 -- start start 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabe6227640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabe01045e0 0x7fabe019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabe6227640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabe0105010 0x7fabe019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabe6227640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe010db90 0x7fabe019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabe6227640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabe0113830 con 0x7fabe01045e0 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabe6227640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabe01136b0 con 0x7fabe0105010 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabe6227640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fabe01139b0 con 0x7fabe010db90 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabe01045e0 0x7fabe019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabe01045e0 0x7fabe019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34936/0 (socket says 172.21.15.16:34936) 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdf7fe640 1 -- 172.21.15.16:0/4234541230 learned_addr learned my addr 172.21.15.16:0/4234541230 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdffff640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe010db90 0x7fabe019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdeffd640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabe0105010 0x7fabe019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdf7fe640 1 -- 172.21.15.16:0/4234541230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe010db90 msgr2=0x7fabe019faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdf7fe640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe010db90 0x7fabe019faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdf7fe640 1 -- 172.21.15.16:0/4234541230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabe0105010 msgr2=0x7fabe019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdf7fe640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabe0105010 0x7fabe019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdf7fe640 1 -- 172.21.15.16:0/4234541230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe01a01d0 con 0x7fabe01045e0 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdffff640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe010db90 0x7fabe019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.810+0000 7fabdeffd640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabe0105010 0x7fabe019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.811+0000 7fabdf7fe640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabe01045e0 0x7fabe019a260 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fabc8032fd0 tx=0x7fabc8031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.811+0000 7fabdcff9640 1 -- 172.21.15.16:0/4234541230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc803f030 con 0x7fabe01045e0 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.811+0000 7fabe6227640 1 -- 172.21.15.16:0/4234541230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe01a0460 con 0x7fabe01045e0 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.811+0000 7fabdcff9640 1 -- 172.21.15.16:0/4234541230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabc8004490 con 0x7fabe01045e0 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.811+0000 7fabdcff9640 1 -- 172.21.15.16:0/4234541230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8035070 con 0x7fabe01045e0 2024-08-28T19:25:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.811+0000 7fabe6227640 1 -- 172.21.15.16:0/4234541230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe01a6c20 con 0x7fabe01045e0 2024-08-28T19:25:53.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.812+0000 7fabe6227640 1 -- 172.21.15.16:0/4234541230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabe01049f0 con 0x7fabe01045e0 2024-08-28T19:25:53.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.814+0000 7fabdcff9640 1 -- 172.21.15.16:0/4234541230 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fabc8042030 con 0x7fabe01045e0 2024-08-28T19:25:53.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.815+0000 7fabdcff9640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabcc0b4ca0 0x7fabcc0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:53.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.815+0000 7fabdeffd640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabcc0b4ca0 0x7fabcc0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:53.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.815+0000 7fabdcff9640 1 -- 172.21.15.16:0/4234541230 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fabc8105fe0 con 0x7fabe01045e0 2024-08-28T19:25:53.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.816+0000 7fabdeffd640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabcc0b4ca0 0x7fabcc0b71b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fabe010f930 tx=0x7fabd0005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:53.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:53.818+0000 7fabdcff9640 1 -- 172.21.15.16:0/4234541230 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fabc80cc3d0 con 0x7fabe01045e0 2024-08-28T19:25:53.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 250624 6.72 MB/sec execute 93 sec latency 20.860 ms 2024-08-28T19:25:54.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.048+0000 7fabe6227640 1 -- 172.21.15.16:0/4234541230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabe019b390 con 0x7fabe01045e0 2024-08-28T19:25:54.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.048+0000 7fabdcff9640 1 -- 172.21.15.16:0/4234541230 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11984 (secure 0 0 0) 0x7fabc80d0b90 con 0x7fabe01045e0 2024-08-28T19:25:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:25:54.051 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-08-28T19:25:34:993166+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:25:34.531024+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:25:54.051 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-08-28T19:25:54.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 -- 172.21.15.16:0/4234541230 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabcc0b4ca0 msgr2=0x7fabcc0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:54.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabcc0b4ca0 0x7fabcc0b71b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fabe010f930 tx=0x7fabd0005f90 comp rx=0 tx=0).stop 2024-08-28T19:25:54.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 -- 172.21.15.16:0/4234541230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabe01045e0 msgr2=0x7fabe019a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabe01045e0 0x7fabe019a260 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fabc8032fd0 tx=0x7fabc8031a20 comp rx=0 tx=0).stop 2024-08-28T19:25:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 -- 172.21.15.16:0/4234541230 shutdown_connections 2024-08-28T19:25:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe010db90 0x7fabe019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabcc0b4ca0 0x7fabcc0b71b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabe0105010 0x7fabe019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 --2- 172.21.15.16:0/4234541230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabe01045e0 0x7fabe019a260 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 -- 172.21.15.16:0/4234541230 >> 172.21.15.16:0/4234541230 conn(0x7fabe00794d0 msgr2=0x7fabe0107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 -- 172.21.15.16:0/4234541230 shutdown_connections 2024-08-28T19:25:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:54.051+0000 7fabe6227640 1 -- 172.21.15.16:0/4234541230 wait complete. 2024-08-28T19:25:54.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 252278 6.78 MB/sec execute 94 sec latency 30.943 ms 2024-08-28T19:25:55.022 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:25:55.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:55 smithi016 ceph-mon[18556]: pgmap v485: 129 pgs: 129 active+clean; 147 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 229 op/s 2024-08-28T19:25:55.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:55 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/4234541230' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:25:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:55 smithi028 ceph-mon[21769]: pgmap v485: 129 pgs: 129 active+clean; 147 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 229 op/s 2024-08-28T19:25:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:55 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/4234541230' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:25:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:55 smithi115 ceph-mon[22184]: pgmap v485: 129 pgs: 129 active+clean; 147 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 229 op/s 2024-08-28T19:25:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:55 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/4234541230' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:25:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.812+0000 7fc07d59a640 1 -- 172.21.15.16:0/385690866 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc078102860 msgr2=0x7fc078102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.812+0000 7fc07d59a640 1 --2- 172.21.15.16:0/385690866 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc078102860 0x7fc078102c70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc068009900 tx=0x7fc06802a3f0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.813+0000 7fc07d59a640 1 -- 172.21.15.16:0/385690866 shutdown_connections 2024-08-28T19:25:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.813+0000 7fc07d59a640 1 --2- 172.21.15.16:0/385690866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078108aa0 0x7fc07806b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.813+0000 7fc07d59a640 1 --2- 172.21.15.16:0/385690866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc078108150 0x7fc078108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.813+0000 7fc07d59a640 1 --2- 172.21.15.16:0/385690866 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc078102860 0x7fc078102c70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.813+0000 7fc07d59a640 1 -- 172.21.15.16:0/385690866 >> 172.21.15.16:0/385690866 conn(0x7fc078079530 msgr2=0x7fc078079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.813+0000 7fc07d59a640 1 -- 172.21.15.16:0/385690866 shutdown_connections 2024-08-28T19:25:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.813+0000 7fc07d59a640 1 -- 172.21.15.16:0/385690866 wait complete. 2024-08-28T19:25:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.814+0000 7fc07d59a640 1 Processor -- start 2024-08-28T19:25:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.814+0000 7fc07d59a640 1 -- start start 2024-08-28T19:25:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.814+0000 7fc07d59a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc078102860 0x7fc07819c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.814+0000 7fc07d59a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc078108150 0x7fc07819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.814+0000 7fc07d59a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078108aa0 0x7fc0781a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.814+0000 7fc07d59a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc078115ac0 con 0x7fc078108150 2024-08-28T19:25:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.814+0000 7fc07d59a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc078115940 con 0x7fc078108aa0 2024-08-28T19:25:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.814+0000 7fc07d59a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc078115c40 con 0x7fc078102860 2024-08-28T19:25:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.815+0000 7fc0767fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc078108150 0x7fc07819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.815+0000 7fc076ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc078102860 0x7fc07819c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.815+0000 7fc0767fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc078108150 0x7fc07819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45590/0 (socket says 172.21.15.16:45590) 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.815+0000 7fc0767fc640 1 -- 172.21.15.16:0/1107152736 learned_addr learned my addr 172.21.15.16:0/1107152736 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.815+0000 7fc0777fe640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078108aa0 0x7fc0781a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.815+0000 7fc076ffd640 1 -- 172.21.15.16:0/1107152736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078108aa0 msgr2=0x7fc0781a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.815+0000 7fc076ffd640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078108aa0 0x7fc0781a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.815+0000 7fc076ffd640 1 -- 172.21.15.16:0/1107152736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc078108150 msgr2=0x7fc07819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.815+0000 7fc076ffd640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc078108150 0x7fc07819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.815+0000 7fc076ffd640 1 -- 172.21.15.16:0/1107152736 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0781a2420 con 0x7fc078102860 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.815+0000 7fc076ffd640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc078102860 0x7fc07819c4b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc06802a9e0 tx=0x7fc068003b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.816+0000 7fc06ffff640 1 -- 172.21.15.16:0/1107152736 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06803f070 con 0x7fc078102860 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.816+0000 7fc06ffff640 1 -- 172.21.15.16:0/1107152736 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc068031b50 con 0x7fc078102860 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.816+0000 7fc06ffff640 1 -- 172.21.15.16:0/1107152736 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc068035030 con 0x7fc078102860 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.816+0000 7fc07d59a640 1 -- 172.21.15.16:0/1107152736 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0781a26b0 con 0x7fc078102860 2024-08-28T19:25:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.816+0000 7fc07d59a640 1 -- 172.21.15.16:0/1107152736 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0781a2b10 con 0x7fc078102860 2024-08-28T19:25:55.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.820+0000 7fc083b0f740 1 Processor -- start 2024-08-28T19:25:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.821+0000 7fc06ffff640 1 -- 172.21.15.16:0/1107152736 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc068042070 con 0x7fc078102860 2024-08-28T19:25:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.821+0000 7fc083b0f740 1 -- start start 2024-08-28T19:25:55.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.822+0000 7fc083b0f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb829f070 0x563bb8644350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.822+0000 7fc083b0f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb82761b0 0x563bb8644890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.822+0000 7fc083b0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb8644dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.822+0000 7fc083b0f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563bb85bb680 con 0x563bb82761b0 2024-08-28T19:25:55.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.822+0000 7fc083b0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563bb85aac10 con 0x563bb827e390 2024-08-28T19:25:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.822+0000 7fc083b0f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563bb85aebc0 con 0x563bb829f070 2024-08-28T19:25:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.822+0000 7fc0767fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb8644dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.822+0000 7fc0767fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb8644dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48078/0 (socket says 172.21.15.16:48078) 2024-08-28T19:25:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.822+0000 7fc0767fc640 1 -- 172.21.15.16:0/695818322 learned_addr learned my addr 172.21.15.16:0/695818322 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.822+0000 7fc06ffff640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc05c0b4cd0 0x7fc05c0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.822+0000 7fc0777fe640 1 --2- 172.21.15.16:0/695818322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb829f070 0x563bb8644350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.823+0000 7fc076ffd640 1 --2- 172.21.15.16:0/695818322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb82761b0 0x563bb8644890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.823+0000 7fc0777fe640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc05c0b4cd0 0x7fc05c0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.823+0000 7fc06ffff640 1 -- 172.21.15.16:0/1107152736 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc068105ba0 con 0x7fc078102860 2024-08-28T19:25:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.823+0000 7fc0767fc640 1 -- 172.21.15.16:0/695818322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb829f070 msgr2=0x563bb8644350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.823+0000 7fc0767fc640 1 --2- 172.21.15.16:0/695818322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb829f070 0x563bb8644350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.823+0000 7fc0767fc640 1 -- 172.21.15.16:0/695818322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb82761b0 msgr2=0x563bb8644890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.823+0000 7fc0767fc640 1 --2- 172.21.15.16:0/695818322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb82761b0 0x563bb8644890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.823+0000 7fc0767fc640 1 -- 172.21.15.16:0/695818322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bb830a650 con 0x563bb827e390 2024-08-28T19:25:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.823+0000 7fc0777fe640 1 --2- 172.21.15.16:0/695818322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb829f070 0x563bb8644350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.823+0000 7fc076ffd640 1 --2- 172.21.15.16:0/695818322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb82761b0 0x563bb8644890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:25:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.823+0000 7fc0767fc640 1 --2- 172.21.15.16:0/695818322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb8644dd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc064009d00 tx=0x7fc064011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22d9799fdc1a6da6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.824+0000 7fc06d7fa640 1 -- 172.21.15.16:0/695818322 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06401e070 con 0x563bb827e390 2024-08-28T19:25:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.824+0000 7fc06d7fa640 1 -- 172.21.15.16:0/695818322 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc064019ae0 con 0x563bb827e390 2024-08-28T19:25:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.824+0000 7fc06d7fa640 1 -- 172.21.15.16:0/695818322 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06400ea30 con 0x563bb827e390 2024-08-28T19:25:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.825+0000 7fc083b0f740 1 -- 172.21.15.16:0/695818322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 msgr2=0x563bb8644dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.825+0000 7fc083b0f740 1 --2- 172.21.15.16:0/695818322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb8644dd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc064009d00 tx=0x7fc064011810 comp rx=0 tx=0).stop 2024-08-28T19:25:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.825+0000 7fc0777fe640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc05c0b4cd0 0x7fc05c0b71e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc070005fd0 tx=0x7fc070004ab0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.826+0000 7fc083b0f740 1 -- 172.21.15.16:0/695818322 shutdown_connections 2024-08-28T19:25:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.826+0000 7fc083b0f740 1 --2- 172.21.15.16:0/695818322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb8644dd0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.826+0000 7fc083b0f740 1 --2- 172.21.15.16:0/695818322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb82761b0 0x563bb8644890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.826+0000 7fc083b0f740 1 --2- 172.21.15.16:0/695818322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb829f070 0x563bb8644350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.826+0000 7fc083b0f740 1 -- 172.21.15.16:0/695818322 >> 172.21.15.16:0/695818322 conn(0x563bb8293460 msgr2=0x563bb8642240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.827+0000 7fc083b0f740 1 -- 172.21.15.16:0/695818322 shutdown_connections 2024-08-28T19:25:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.827+0000 7fc083b0f740 1 -- 172.21.15.16:0/695818322 wait complete. 2024-08-28T19:25:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.827+0000 7fc083b0f740 1 -- start start 2024-08-28T19:25:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.827+0000 7fc083b0f740 1 Processor -- start 2024-08-28T19:25:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc083b0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb82761b0 0x563bb864fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc083b0f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb827e390 0x563bb8650090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc076ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb82761b0 0x563bb864fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc076ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb82761b0 0x563bb864fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48086/0 (socket says 172.21.15.16:48086) 2024-08-28T19:25:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc076ffd640 1 -- 172.21.15.16:0/3400689570 learned_addr learned my addr 172.21.15.16:0/3400689570 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc083b0f740 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb829f070 0x563bb865be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc083b0f740 1 -- 172.21.15.16:0/3400689570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563bb840d700 con 0x563bb827e390 2024-08-28T19:25:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc083b0f740 1 -- 172.21.15.16:0/3400689570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563bb8531750 con 0x563bb82761b0 2024-08-28T19:25:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc083b0f740 1 -- 172.21.15.16:0/3400689570 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563bb8652b30 con 0x563bb829f070 2024-08-28T19:25:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc083b0f740 10 client.0 _ll_register_callbacks cb 0x563bb8560e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc0777fe640 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb829f070 0x563bb865be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.828+0000 7fc0767fc640 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb827e390 0x563bb8650090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.829+0000 7fc0767fc640 1 -- 172.21.15.16:0/3400689570 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb829f070 msgr2=0x563bb865be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.829+0000 7fc0767fc640 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb829f070 0x563bb865be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.829+0000 7fc0767fc640 1 -- 172.21.15.16:0/3400689570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb82761b0 msgr2=0x563bb864fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.829+0000 7fc0767fc640 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb82761b0 0x563bb864fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.829+0000 7fc0767fc640 1 -- 172.21.15.16:0/3400689570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bb856a900 con 0x563bb827e390 2024-08-28T19:25:55.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.830+0000 7fc0767fc640 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb827e390 0x563bb8650090 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc064011d10 tx=0x7fc064000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ade22bcefff2c08 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:55.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.830+0000 7fc05a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:55.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.831+0000 7fc05a7fc640 1 -- 172.21.15.16:0/3400689570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc064029020 con 0x563bb827e390 2024-08-28T19:25:55.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.832+0000 7fc083b0f740 1 -- 172.21.15.16:0/3400689570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc064019ae0 con 0x563bb827e390 2024-08-28T19:25:55.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.832+0000 7fc05a7fc640 1 -- 172.21.15.16:0/3400689570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc064005240 con 0x563bb827e390 2024-08-28T19:25:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.832+0000 7fc05a7fc640 1 -- 172.21.15.16:0/3400689570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc064034050 con 0x563bb827e390 2024-08-28T19:25:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.832+0000 7fc083b0f740 10 client.16833 fetch_fsmap learned FSMap version 39 2024-08-28T19:25:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.832+0000 7fc083b0f740 1 -- 172.21.15.16:0/3400689570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563bb865e950 con 0x563bb827e390 2024-08-28T19:25:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.833+0000 7fc05a7fc640 1 -- 172.21.15.16:0/3400689570 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc064009920 con 0x563bb827e390 2024-08-28T19:25:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.833+0000 7fc05a7fc640 1 -- 172.21.15.16:0/3400689570 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8390+0+0 (secure 0 0 0) 0x7fc06400e6d0 con 0x563bb827e390 2024-08-28T19:25:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.833+0000 7fc083b0f740 10 client.16833 fetch_fsmap finished waiting for FSMap version 39 2024-08-28T19:25:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.833+0000 7fc083b0f740 10 client.16833 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:25:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.833+0000 7fc083b0f740 20 client.16833 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.833+0000 7fc083b0f740 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563bb825d9f0 0x563bb86621f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.833+0000 7fc083b0f740 4 client.16833 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.833+0000 7fc083b0f740 1 -- 172.21.15.16:0/3400689570 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc064019ae0 con 0x563bb825d9f0 2024-08-28T19:25:55.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.834+0000 7fc076ffd640 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563bb825d9f0 0x563bb86621f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.834+0000 7fc05a7fc640 10 client.16833 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:25:55.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.834+0000 7fc076ffd640 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563bb825d9f0 0x563bb86621f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9796885459490cf6 server_cookie=66e3352281dfb541 in_seq=0 out_seq=0 2024-08-28T19:25:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.836+0000 7fc05a7fc640 1 -- 172.21.15.16:0/3400689570 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fc064019ae0 con 0x563bb825d9f0 2024-08-28T19:25:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.836+0000 7fc05a7fc640 10 client.16833 handle_command_reply: tid=0 2024-08-28T19:25:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.836+0000 7fc083b0f740 1 client.16833 shutdown 2024-08-28T19:25:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc083b0f740 1 -- 172.21.15.16:0/3400689570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb827e390 msgr2=0x563bb8650090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc083b0f740 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb827e390 0x563bb8650090 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc064011d10 tx=0x7fc064000bc0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc083b0f740 1 -- 172.21.15.16:0/3400689570 shutdown_connections 2024-08-28T19:25:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc083b0f740 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563bb825d9f0 0x563bb86621f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc083b0f740 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb829f070 0x563bb865be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc083b0f740 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb827e390 0x563bb8650090 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc083b0f740 1 --2- 172.21.15.16:0/3400689570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb82761b0 0x563bb864fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc083b0f740 1 -- 172.21.15.16:0/3400689570 >> 172.21.15.16:0/3400689570 conn(0x563bb8293460 msgr2=0x563bb8647530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc05a7fc640 0 client.16833 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:25:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc083b0f740 1 -- 172.21.15.16:0/3400689570 shutdown_connections 2024-08-28T19:25:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc083b0f740 1 -- 172.21.15.16:0/3400689570 wait complete. 2024-08-28T19:25:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.837+0000 7fc083b0f740 20 client.16833 trim_cache size 0 max 16384 2024-08-28T19:25:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.858+0000 7fc083b0f740 1 Processor -- start 2024-08-28T19:25:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.858+0000 7fc083b0f740 1 -- start start 2024-08-28T19:25:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.859+0000 7fc083b0f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb82761b0 0x563bb864fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.859+0000 7fc083b0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb865bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.859+0000 7fc083b0f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 0x563bb865c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.859+0000 7fc083b0f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781aeb20 con 0x563bb829f070 2024-08-28T19:25:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.859+0000 7fc083b0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781b0750 con 0x563bb827e390 2024-08-28T19:25:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.859+0000 7fc083b0f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781af250 con 0x563bb82761b0 2024-08-28T19:25:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.859+0000 7fc0767fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb865bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.859+0000 7fc0767fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb865bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48102/0 (socket says 172.21.15.16:48102) 2024-08-28T19:25:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.859+0000 7fc0767fc640 1 -- 172.21.15.16:0/1388701424 learned_addr learned my addr 172.21.15.16:0/1388701424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.860+0000 7fc076ffd640 1 --2- 172.21.15.16:0/1388701424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb82761b0 0x563bb864fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.860+0000 7fc0777fe640 1 --2- 172.21.15.16:0/1388701424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 0x563bb865c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.860+0000 7fc0767fc640 1 -- 172.21.15.16:0/1388701424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb82761b0 msgr2=0x563bb864fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.860+0000 7fc0767fc640 1 --2- 172.21.15.16:0/1388701424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb82761b0 0x563bb864fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.860+0000 7fc0767fc640 1 -- 172.21.15.16:0/1388701424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 msgr2=0x563bb865c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.860+0000 7fc0767fc640 1 --2- 172.21.15.16:0/1388701424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 0x563bb865c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.860+0000 7fc0767fc640 1 -- 172.21.15.16:0/1388701424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bb8646be0 con 0x563bb827e390 2024-08-28T19:25:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.860+0000 7fc0777fe640 1 --2- 172.21.15.16:0/1388701424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 0x563bb865c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:25:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.860+0000 7fc0767fc640 1 --2- 172.21.15.16:0/1388701424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb865bd20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc0640059f0 tx=0x7fc064019640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=278ef61b1743a0ae server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.861+0000 7fc06cff9640 1 -- 172.21.15.16:0/1388701424 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06401e040 con 0x563bb827e390 2024-08-28T19:25:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.861+0000 7fc06cff9640 1 -- 172.21.15.16:0/1388701424 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc064009920 con 0x563bb827e390 2024-08-28T19:25:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.861+0000 7fc06cff9640 1 -- 172.21.15.16:0/1388701424 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc064019dd0 con 0x563bb827e390 2024-08-28T19:25:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.862+0000 7fc083b0f740 1 -- 172.21.15.16:0/1388701424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 msgr2=0x563bb865bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.862+0000 7fc083b0f740 1 --2- 172.21.15.16:0/1388701424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb865bd20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc0640059f0 tx=0x7fc064019640 comp rx=0 tx=0).stop 2024-08-28T19:25:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.863+0000 7fc083b0f740 1 -- 172.21.15.16:0/1388701424 shutdown_connections 2024-08-28T19:25:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.863+0000 7fc083b0f740 1 --2- 172.21.15.16:0/1388701424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 0x563bb865c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.863+0000 7fc083b0f740 1 --2- 172.21.15.16:0/1388701424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb865bd20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.863+0000 7fc083b0f740 1 --2- 172.21.15.16:0/1388701424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb82761b0 0x563bb864fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.863+0000 7fc083b0f740 1 -- 172.21.15.16:0/1388701424 >> 172.21.15.16:0/1388701424 conn(0x563bb8293460 msgr2=0x563bb864f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.863+0000 7fc083b0f740 1 -- 172.21.15.16:0/1388701424 shutdown_connections 2024-08-28T19:25:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.863+0000 7fc083b0f740 1 -- 172.21.15.16:0/1388701424 wait complete. 2024-08-28T19:25:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.864+0000 7fc083b0f740 1 -- start start 2024-08-28T19:25:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.864+0000 7fc083b0f740 1 Processor -- start 2024-08-28T19:25:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.864+0000 7fc083b0f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb82761b0 0x563bb8653550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.865+0000 7fc083b0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb8653a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.865+0000 7fc076ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb82761b0 0x563bb8653550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.865+0000 7fc076ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb82761b0 0x563bb8653550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55630/0 (socket says 172.21.15.16:55630) 2024-08-28T19:25:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.865+0000 7fc076ffd640 1 -- 172.21.15.16:0/292476196 learned_addr learned my addr 172.21.15.16:0/292476196 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.865+0000 7fc083b0f740 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 0x563bb86caba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.865+0000 7fc083b0f740 1 -- 172.21.15.16:0/292476196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781b0080 con 0x563bb829f070 2024-08-28T19:25:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.865+0000 7fc083b0f740 1 -- 172.21.15.16:0/292476196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781af980 con 0x563bb827e390 2024-08-28T19:25:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.865+0000 7fc0767fc640 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb8653a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.865+0000 7fc083b0f740 1 -- 172.21.15.16:0/292476196 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781acb40 con 0x563bb82761b0 2024-08-28T19:25:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.865+0000 7fc0777fe640 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 0x563bb86caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.866+0000 7fc083b0f740 10 client.0 _ll_register_callbacks cb 0x7fc06401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:55.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.866+0000 7fc0777fe640 1 -- 172.21.15.16:0/292476196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb82761b0 msgr2=0x563bb8653550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:55.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.866+0000 7fc0777fe640 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb82761b0 0x563bb8653550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.866+0000 7fc0777fe640 1 -- 172.21.15.16:0/292476196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 msgr2=0x563bb8653a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:55.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.866+0000 7fc0777fe640 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb8653a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.866+0000 7fc0777fe640 1 -- 172.21.15.16:0/292476196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bb86cb0e0 con 0x563bb829f070 2024-08-28T19:25:55.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.867+0000 7fc0777fe640 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 0x563bb86caba0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x563bb85aed10 tx=0x7fc070016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7cbb32a3dba3db server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:55.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.867+0000 7fc05a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:55.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.867+0000 7fc05a7fc640 1 -- 172.21.15.16:0/292476196 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc070021700 con 0x563bb829f070 2024-08-28T19:25:55.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.867+0000 7fc083b0f740 1 -- 172.21.15.16:0/292476196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc064009920 con 0x563bb829f070 2024-08-28T19:25:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.867+0000 7fc05a7fc640 1 -- 172.21.15.16:0/292476196 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc0700218a0 con 0x563bb829f070 2024-08-28T19:25:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.868+0000 7fc05a7fc640 1 -- 172.21.15.16:0/292476196 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc07003c050 con 0x563bb829f070 2024-08-28T19:25:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.868+0000 7fc083b0f740 10 client.16836 fetch_fsmap learned FSMap version 39 2024-08-28T19:25:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.868+0000 7fc083b0f740 1 -- 172.21.15.16:0/292476196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563bb86cc2c0 con 0x563bb829f070 2024-08-28T19:25:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.868+0000 7fc05a7fc640 1 -- 172.21.15.16:0/292476196 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc064009920 con 0x563bb829f070 2024-08-28T19:25:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.868+0000 7fc05a7fc640 1 -- 172.21.15.16:0/292476196 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8390+0+0 (secure 0 0 0) 0x7fc07002ece0 con 0x563bb829f070 2024-08-28T19:25:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.869+0000 7fc083b0f740 10 client.16836 fetch_fsmap finished waiting for FSMap version 39 2024-08-28T19:25:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.869+0000 7fc083b0f740 10 client.16836 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:25:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.869+0000 7fc083b0f740 20 client.16836 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.869+0000 7fc083b0f740 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563bb8232da0 0x563bb86ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.869+0000 7fc083b0f740 4 client.16836 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:25:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.869+0000 7fc083b0f740 1 -- 172.21.15.16:0/292476196 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563bb85bd5d0 con 0x563bb8232da0 2024-08-28T19:25:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.869+0000 7fc076ffd640 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563bb8232da0 0x563bb86ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.870+0000 7fc05a7fc640 10 client.16836 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:25:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.870+0000 7fc076ffd640 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563bb8232da0 0x563bb86ceb10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=90f680b68b20f87e server_cookie=897df3fab2440e7c in_seq=0 out_seq=0 2024-08-28T19:25:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.871+0000 7fc05a7fc640 1 -- 172.21.15.16:0/292476196 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563bb85bd5d0 con 0x563bb8232da0 2024-08-28T19:25:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.871+0000 7fc05a7fc640 10 client.16836 handle_command_reply: tid=0 2024-08-28T19:25:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.871+0000 7fc083b0f740 1 client.16836 shutdown 2024-08-28T19:25:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.872+0000 7fc083b0f740 1 -- 172.21.15.16:0/292476196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 msgr2=0x563bb86caba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.872+0000 7fc083b0f740 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 0x563bb86caba0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x563bb85aed10 tx=0x7fc070016040 comp rx=0 tx=0).stop 2024-08-28T19:25:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.872+0000 7fc083b0f740 1 -- 172.21.15.16:0/292476196 shutdown_connections 2024-08-28T19:25:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.872+0000 7fc083b0f740 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563bb8232da0 0x563bb86ceb10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.872+0000 7fc083b0f740 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb829f070 0x563bb86caba0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.872+0000 7fc083b0f740 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bb827e390 0x563bb8653a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.872+0000 7fc083b0f740 1 --2- 172.21.15.16:0/292476196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563bb82761b0 0x563bb8653550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.872+0000 7fc083b0f740 1 -- 172.21.15.16:0/292476196 >> 172.21.15.16:0/292476196 conn(0x563bb8293460 msgr2=0x563bb8641b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.872+0000 7fc05a7fc640 0 client.16836 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:25:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.872+0000 7fc083b0f740 1 -- 172.21.15.16:0/292476196 shutdown_connections 2024-08-28T19:25:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.872+0000 7fc083b0f740 1 -- 172.21.15.16:0/292476196 wait complete. 2024-08-28T19:25:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.873+0000 7fc083b0f740 20 client.16836 trim_cache size 0 max 16384 2024-08-28T19:25:55.876 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:25:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.874+0000 7fc07d59a640 1 -- 172.21.15.16:0/1107152736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc05c0b4cd0 msgr2=0x7fc05c0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.874+0000 7fc07d59a640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc05c0b4cd0 0x7fc05c0b71e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc070005fd0 tx=0x7fc070004ab0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.874+0000 7fc07d59a640 1 -- 172.21.15.16:0/1107152736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc078102860 msgr2=0x7fc07819c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.874+0000 7fc07d59a640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc078102860 0x7fc07819c4b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc06802a9e0 tx=0x7fc068003b20 comp rx=0 tx=0).stop 2024-08-28T19:25:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.874+0000 7fc07d59a640 1 -- 172.21.15.16:0/1107152736 shutdown_connections 2024-08-28T19:25:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.874+0000 7fc07d59a640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc05c0b4cd0 0x7fc05c0b71e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.874+0000 7fc07d59a640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078108aa0 0x7fc0781a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.874+0000 7fc07d59a640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc078108150 0x7fc07819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.874+0000 7fc07d59a640 1 --2- 172.21.15.16:0/1107152736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc078102860 0x7fc07819c4b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.875+0000 7fc07d59a640 1 -- 172.21.15.16:0/1107152736 >> 172.21.15.16:0/1107152736 conn(0x7fc078079530 msgr2=0x7fc078106ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.875+0000 7fc07d59a640 1 -- 172.21.15.16:0/1107152736 shutdown_connections 2024-08-28T19:25:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:55.877+0000 7fc07d59a640 1 -- 172.21.15.16:0/1107152736 wait complete. 2024-08-28T19:25:55.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 254218 6.78 MB/sec execute 95 sec latency 82.445 ms 2024-08-28T19:25:56.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:25:56.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 255839 6.84 MB/sec execute 96 sec latency 63.164 ms 2024-08-28T19:25:57.114 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:57.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:57 smithi016 ceph-mon[18556]: pgmap v486: 129 pgs: 129 active+clean; 152 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 340 op/s 2024-08-28T19:25:57.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:57 smithi028 ceph-mon[21769]: pgmap v486: 129 pgs: 129 active+clean; 152 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 340 op/s 2024-08-28T19:25:57.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:57 smithi115 ceph-mon[22184]: pgmap v486: 129 pgs: 129 active+clean; 152 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 340 op/s 2024-08-28T19:25:57.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 257889 6.85 MB/sec execute 97 sec latency 63.281 ms 2024-08-28T19:25:58.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.598+0000 7f0b9859e640 1 -- 172.21.15.16:0/3719477222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b90108170 msgr2=0x7f0b90108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:58.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.598+0000 7f0b9859e640 1 --2- 172.21.15.16:0/3719477222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b90108170 0x7f0b90108580 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f0b80009900 tx=0x7f0b8002a410 comp rx=0 tx=0).stop 2024-08-28T19:25:58.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.598+0000 7f0b9859e640 1 -- 172.21.15.16:0/3719477222 shutdown_connections 2024-08-28T19:25:58.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.598+0000 7f0b9859e640 1 --2- 172.21.15.16:0/3719477222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b90108ac0 0x7f0b9010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.598+0000 7f0b9859e640 1 --2- 172.21.15.16:0/3719477222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b90108170 0x7f0b90108580 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.598+0000 7f0b9859e640 1 --2- 172.21.15.16:0/3719477222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b90102830 0x7f0b90102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.598+0000 7f0b9859e640 1 -- 172.21.15.16:0/3719477222 >> 172.21.15.16:0/3719477222 conn(0x7f0b900794d0 msgr2=0x7f0b900798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:58.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.598+0000 7f0b9859e640 1 -- 172.21.15.16:0/3719477222 shutdown_connections 2024-08-28T19:25:58.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.598+0000 7f0b9859e640 1 -- 172.21.15.16:0/3719477222 wait complete. 2024-08-28T19:25:58.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.598+0000 7f0b9859e640 1 Processor -- start 2024-08-28T19:25:58.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.598+0000 7f0b9859e640 1 -- start start 2024-08-28T19:25:58.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b9859e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b90102830 0x7f0b9019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b9859e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b90108170 0x7f0b9019a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b9859e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b90108ac0 0x7f0b9019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b9859e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0b90112730 con 0x7f0b90102830 2024-08-28T19:25:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b9859e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0b901125b0 con 0x7f0b90108170 2024-08-28T19:25:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b9859e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0b901128b0 con 0x7f0b90108ac0 2024-08-28T19:25:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b9633a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b90102830 0x7f0b9019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b9633a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b90102830 0x7f0b9019a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45674/0 (socket says 172.21.15.16:45674) 2024-08-28T19:25:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b9633a640 1 -- 172.21.15.16:0/3514812208 learned_addr learned my addr 172.21.15.16:0/3514812208 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:58.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b95b39640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b90108170 0x7f0b9019a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b96b3b640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b90108ac0 0x7f0b9019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b95b39640 1 -- 172.21.15.16:0/3514812208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b90108ac0 msgr2=0x7f0b9019fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:58.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b95b39640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b90108ac0 0x7f0b9019fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b95b39640 1 -- 172.21.15.16:0/3514812208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b90102830 msgr2=0x7f0b9019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:58.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b95b39640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b90102830 0x7f0b9019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b95b39640 1 -- 172.21.15.16:0/3514812208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b901a01e0 con 0x7f0b90108170 2024-08-28T19:25:58.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.599+0000 7f0b9633a640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b90102830 0x7f0b9019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:58.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.600+0000 7f0b95b39640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b90108170 0x7f0b9019a810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0b8002a930 tx=0x7f0b80004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:58.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.600+0000 7f0b877fe640 1 -- 172.21.15.16:0/3514812208 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8003f070 con 0x7f0b90108170 2024-08-28T19:25:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.600+0000 7f0b9859e640 1 -- 172.21.15.16:0/3514812208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b901a0470 con 0x7f0b90108170 2024-08-28T19:25:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.600+0000 7f0b877fe640 1 -- 172.21.15.16:0/3514812208 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b80003ca0 con 0x7f0b90108170 2024-08-28T19:25:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.600+0000 7f0b877fe640 1 -- 172.21.15.16:0/3514812208 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80035070 con 0x7f0b90108170 2024-08-28T19:25:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.600+0000 7f0b9859e640 1 -- 172.21.15.16:0/3514812208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b901a6c30 con 0x7f0b90108170 2024-08-28T19:25:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.602+0000 7f0b9eb13740 1 Processor -- start 2024-08-28T19:25:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9eb13740 1 -- start start 2024-08-28T19:25:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 0x55aed85a9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81db1b0 0x55aed85a9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81e3390 0x55aed85a9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aed8520680 con 0x55aed81db1b0 2024-08-28T19:25:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aed850fc10 con 0x55aed8204070 2024-08-28T19:25:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55aed8513bc0 con 0x55aed81e3390 2024-08-28T19:25:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9633a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81db1b0 0x55aed85a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9633a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81db1b0 0x55aed85a9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45678/0 (socket says 172.21.15.16:45678) 2024-08-28T19:25:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9633a640 1 -- 172.21.15.16:0/2819246334 learned_addr learned my addr 172.21.15.16:0/2819246334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b96b3b640 1 --2- 172.21.15.16:0/2819246334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 0x55aed85a9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9633a640 1 -- 172.21.15.16:0/2819246334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81e3390 msgr2=0x55aed85a9d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:25:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9633a640 1 --2- 172.21.15.16:0/2819246334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81e3390 0x55aed85a9d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9633a640 1 -- 172.21.15.16:0/2819246334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 msgr2=0x55aed85a9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.603+0000 7f0b9633a640 1 --2- 172.21.15.16:0/2819246334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 0x55aed85a9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.604+0000 7f0b9633a640 1 -- 172.21.15.16:0/2819246334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aed826f650 con 0x55aed81db1b0 2024-08-28T19:25:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.604+0000 7f0b96b3b640 1 --2- 172.21.15.16:0/2819246334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 0x55aed85a9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.604+0000 7f0b877fe640 1 -- 172.21.15.16:0/3514812208 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0b8003b460 con 0x7f0b90108170 2024-08-28T19:25:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.604+0000 7f0b9633a640 1 --2- 172.21.15.16:0/2819246334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81db1b0 0x55aed85a9850 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c011650 tx=0x7f0b8c011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3793162258dde1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.604+0000 7f0b84ff9640 1 -- 172.21.15.16:0/2819246334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c01e070 con 0x55aed81db1b0 2024-08-28T19:25:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.604+0000 7f0b84ff9640 1 -- 172.21.15.16:0/2819246334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b8c00fba0 con 0x55aed81db1b0 2024-08-28T19:25:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.604+0000 7f0b84ff9640 1 -- 172.21.15.16:0/2819246334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c00e5d0 con 0x55aed81db1b0 2024-08-28T19:25:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.604+0000 7f0b877fe640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0b7c0b4ca0 0x7f0b7c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b96b3b640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0b7c0b4ca0 0x7f0b7c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b877fe640 1 -- 172.21.15.16:0/3514812208 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0b80105ab0 con 0x7f0b90108170 2024-08-28T19:25:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b9eb13740 1 -- 172.21.15.16:0/2819246334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81db1b0 msgr2=0x55aed85a9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/2819246334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81db1b0 0x55aed85a9850 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c011650 tx=0x7f0b8c011b10 comp rx=0 tx=0).stop 2024-08-28T19:25:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b9eb13740 1 -- 172.21.15.16:0/2819246334 shutdown_connections 2024-08-28T19:25:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/2819246334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81e3390 0x55aed85a9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/2819246334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81db1b0 0x55aed85a9850 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/2819246334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 0x55aed85a9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b9eb13740 1 -- 172.21.15.16:0/2819246334 >> 172.21.15.16:0/2819246334 conn(0x55aed81f8460 msgr2=0x55aed85a7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b96b3b640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0b7c0b4ca0 0x7f0b7c0b71b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0b88005fd0 tx=0x7f0b88009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b9eb13740 1 -- 172.21.15.16:0/2819246334 shutdown_connections 2024-08-28T19:25:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b9eb13740 1 -- 172.21.15.16:0/2819246334 wait complete. 2024-08-28T19:25:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.605+0000 7f0b9eb13740 1 -- start start 2024-08-28T19:25:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b9eb13740 1 Processor -- start 2024-08-28T19:25:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81e3390 0x55aed85b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 0x55aed85b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81c29f0 0x55aed85ba840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aed8372700 con 0x55aed81c29f0 2024-08-28T19:25:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aed8496750 con 0x55aed8204070 2024-08-28T19:25:58.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55aed85af880 con 0x55aed81e3390 2024-08-28T19:25:58.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b9633a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81e3390 0x55aed85b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b9633a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81e3390 0x55aed85b52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55652/0 (socket says 172.21.15.16:55652) 2024-08-28T19:25:58.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b9633a640 1 -- 172.21.15.16:0/81659390 learned_addr learned my addr 172.21.15.16:0/81659390 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:58.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b9eb13740 10 client.0 _ll_register_callbacks cb 0x55aed84c5e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:58.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b95b39640 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 0x55aed85b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.606+0000 7f0b96b3b640 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81c29f0 0x55aed85ba840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b9633a640 1 -- 172.21.15.16:0/81659390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 msgr2=0x55aed85b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b9633a640 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 0x55aed85b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b9633a640 1 -- 172.21.15.16:0/81659390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81c29f0 msgr2=0x55aed85ba840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b9633a640 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81c29f0 0x55aed85ba840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b9633a640 1 -- 172.21.15.16:0/81659390 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aed84cf900 con 0x55aed81e3390 2024-08-28T19:25:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b95b39640 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 0x55aed85b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b9633a640 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81e3390 0x55aed85b52b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c019c80 tx=0x7f0b8c01af10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0ca7d7098bc8cbc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:25:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b79ffb640 1 -- 172.21.15.16:0/81659390 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c01e050 con 0x55aed81e3390 2024-08-28T19:25:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b9eb13740 1 -- 172.21.15.16:0/81659390 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b8c00fba0 con 0x55aed81e3390 2024-08-28T19:25:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b79ffb640 1 -- 172.21.15.16:0/81659390 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0b8c009e40 con 0x55aed81e3390 2024-08-28T19:25:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b79ffb640 1 -- 172.21.15.16:0/81659390 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f0b8c032050 con 0x55aed81e3390 2024-08-28T19:25:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b9eb13740 10 client.26251 fetch_fsmap learned FSMap version 40 2024-08-28T19:25:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.607+0000 7f0b9eb13740 1 -- 172.21.15.16:0/81659390 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55aed85c3920 con 0x55aed81e3390 2024-08-28T19:25:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.608+0000 7f0b79ffb640 1 -- 172.21.15.16:0/81659390 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b8c00fba0 con 0x55aed81e3390 2024-08-28T19:25:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.608+0000 7f0b79ffb640 1 -- 172.21.15.16:0/81659390 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f0b8c02d030 con 0x55aed81e3390 2024-08-28T19:25:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.608+0000 7f0b9eb13740 10 client.26251 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:25:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.608+0000 7f0b9eb13740 10 client.26251 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:25:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.608+0000 7f0b9eb13740 20 client.26251 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.608+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55aed8197da0 0x55aed85c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.608+0000 7f0b9eb13740 4 client.26251 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.609+0000 7f0b9eb13740 1 -- 172.21.15.16:0/81659390 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aed85225d0 con 0x55aed8197da0 2024-08-28T19:25:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.609+0000 7f0b95b39640 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55aed8197da0 0x55aed85c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.609+0000 7f0b79ffb640 10 client.26251 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:25:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.609+0000 7f0b95b39640 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55aed8197da0 0x55aed85c6c80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fa8a08df0e748096 server_cookie=b4971a22860ffbe7 in_seq=0 out_seq=0 2024-08-28T19:25:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b79ffb640 1 -- 172.21.15.16:0/81659390 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55aed85225d0 con 0x55aed8197da0 2024-08-28T19:25:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b79ffb640 10 client.26251 handle_command_reply: tid=0 2024-08-28T19:25:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b9eb13740 1 client.26251 shutdown 2024-08-28T19:25:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b9eb13740 1 -- 172.21.15.16:0/81659390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81e3390 msgr2=0x55aed85b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81e3390 0x55aed85b52b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c019c80 tx=0x7f0b8c01af10 comp rx=0 tx=0).stop 2024-08-28T19:25:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b9eb13740 1 -- 172.21.15.16:0/81659390 shutdown_connections 2024-08-28T19:25:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55aed8197da0 0x55aed85c6c80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81c29f0 0x55aed85ba840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed8204070 0x55aed85b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/81659390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81e3390 0x55aed85b52b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b9eb13740 1 -- 172.21.15.16:0/81659390 >> 172.21.15.16:0/81659390 conn(0x55aed81f8460 msgr2=0x55aed85ac460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b79ffb640 0 client.26251 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:25:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b9eb13740 1 -- 172.21.15.16:0/81659390 shutdown_connections 2024-08-28T19:25:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.610+0000 7f0b9eb13740 1 -- 172.21.15.16:0/81659390 wait complete. 2024-08-28T19:25:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.611+0000 7f0b9eb13740 20 client.26251 trim_cache size 0 max 16384 2024-08-28T19:25:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b9eb13740 1 Processor -- start 2024-08-28T19:25:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b9eb13740 1 -- start start 2024-08-28T19:25:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81c29f0 0x55aed85b5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed81e3390 0x55aed85acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed8204070 0x55aed85b0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0b901ae960 con 0x55aed8204070 2024-08-28T19:25:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0b901b0560 con 0x55aed81e3390 2024-08-28T19:25:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0b901af090 con 0x55aed81c29f0 2024-08-28T19:25:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b96b3b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed8204070 0x55aed85b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b96b3b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed8204070 0x55aed85b0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45696/0 (socket says 172.21.15.16:45696) 2024-08-28T19:25:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b96b3b640 1 -- 172.21.15.16:0/4108508312 learned_addr learned my addr 172.21.15.16:0/4108508312 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b95b39640 1 --2- 172.21.15.16:0/4108508312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed81e3390 0x55aed85acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b9633a640 1 --2- 172.21.15.16:0/4108508312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81c29f0 0x55aed85b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b96b3b640 1 -- 172.21.15.16:0/4108508312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81c29f0 msgr2=0x55aed85b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b96b3b640 1 --2- 172.21.15.16:0/4108508312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81c29f0 0x55aed85b5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b96b3b640 1 -- 172.21.15.16:0/4108508312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed81e3390 msgr2=0x55aed85acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b96b3b640 1 --2- 172.21.15.16:0/4108508312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed81e3390 0x55aed85acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b96b3b640 1 -- 172.21.15.16:0/4108508312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aed85a13f0 con 0x55aed8204070 2024-08-28T19:25:58.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b95b39640 1 --2- 172.21.15.16:0/4108508312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed81e3390 0x55aed85acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:58.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.625+0000 7f0b9633a640 1 --2- 172.21.15.16:0/4108508312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81c29f0 0x55aed85b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:58.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.626+0000 7f0b96b3b640 1 --2- 172.21.15.16:0/4108508312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed8204070 0x55aed85b0860 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0b88025740 tx=0x7f0b88020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4aad7fc460c0cd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:58.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.626+0000 7f0b7bfff640 1 -- 172.21.15.16:0/4108508312 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88034070 con 0x55aed8204070 2024-08-28T19:25:58.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.626+0000 7f0b7bfff640 1 -- 172.21.15.16:0/4108508312 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b88024aa0 con 0x55aed8204070 2024-08-28T19:25:58.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.626+0000 7f0b7bfff640 1 -- 172.21.15.16:0/4108508312 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8802fbd0 con 0x55aed8204070 2024-08-28T19:25:58.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 -- 172.21.15.16:0/4108508312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed8204070 msgr2=0x55aed85b0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:58.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/4108508312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed8204070 0x55aed85b0860 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0b88025740 tx=0x7f0b88020000 comp rx=0 tx=0).stop 2024-08-28T19:25:58.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 -- 172.21.15.16:0/4108508312 shutdown_connections 2024-08-28T19:25:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/4108508312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed8204070 0x55aed85b0860 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/4108508312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed81e3390 0x55aed85acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/4108508312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed81c29f0 0x55aed85b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 -- 172.21.15.16:0/4108508312 >> 172.21.15.16:0/4108508312 conn(0x55aed81f8460 msgr2=0x55aed85aa340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 -- 172.21.15.16:0/4108508312 shutdown_connections 2024-08-28T19:25:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 -- 172.21.15.16:0/4108508312 wait complete. 2024-08-28T19:25:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 -- start start 2024-08-28T19:25:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 Processor -- start 2024-08-28T19:25:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed81c29f0 0x55aed85c0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81e3390 0x55aed85c11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed8204070 0x55aed8626770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.627+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0b901afe90 con 0x55aed81e3390 2024-08-28T19:25:58.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0b901af790 con 0x55aed81c29f0 2024-08-28T19:25:58.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b9eb13740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0b901ac940 con 0x55aed8204070 2024-08-28T19:25:58.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b9eb13740 10 client.0 _ll_register_callbacks cb 0x7f0b8c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:58.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b95b39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81e3390 0x55aed85c11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b95b39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81e3390 0x55aed85c11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45700/0 (socket says 172.21.15.16:45700) 2024-08-28T19:25:58.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b95b39640 1 -- 172.21.15.16:0/3211888790 learned_addr learned my addr 172.21.15.16:0/3211888790 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:58.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b96b3b640 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed8204070 0x55aed8626770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b9633a640 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed81c29f0 0x55aed85c0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b95b39640 1 -- 172.21.15.16:0/3211888790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed8204070 msgr2=0x55aed8626770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:58.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b95b39640 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed8204070 0x55aed8626770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b95b39640 1 -- 172.21.15.16:0/3211888790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed81c29f0 msgr2=0x55aed85c0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:58.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b95b39640 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed81c29f0 0x55aed85c0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b95b39640 1 -- 172.21.15.16:0/3211888790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aed8626de0 con 0x55aed81e3390 2024-08-28T19:25:58.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b95b39640 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81e3390 0x55aed85c11b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0b80102e90 tx=0x7f0b8003b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5775057829a19ea4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:58.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:25:58.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b79ffb640 1 -- 172.21.15.16:0/3211888790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b800cf040 con 0x55aed81e3390 2024-08-28T19:25:58.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b79ffb640 1 -- 172.21.15.16:0/3211888790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0b800c8110 con 0x55aed81e3390 2024-08-28T19:25:58.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b9eb13740 1 -- 172.21.15.16:0/3211888790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b88024aa0 con 0x55aed81e3390 2024-08-28T19:25:58.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b79ffb640 1 -- 172.21.15.16:0/3211888790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f0b80048040 con 0x55aed81e3390 2024-08-28T19:25:58.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.628+0000 7f0b9eb13740 10 client.16854 fetch_fsmap learned FSMap version 40 2024-08-28T19:25:58.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b9eb13740 1 -- 172.21.15.16:0/3211888790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55aed8631470 con 0x55aed81e3390 2024-08-28T19:25:58.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b79ffb640 1 -- 172.21.15.16:0/3211888790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b88024aa0 con 0x55aed81e3390 2024-08-28T19:25:58.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b79ffb640 1 -- 172.21.15.16:0/3211888790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f0b800b7020 con 0x55aed81e3390 2024-08-28T19:25:58.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b9eb13740 10 client.16854 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:25:58.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b9eb13740 10 client.16854 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:25:58.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b9eb13740 20 client.16854 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:58.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55aed8631bf0 0x55aed8634010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:58.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b9eb13740 4 client.16854 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:25:58.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b9eb13740 1 -- 172.21.15.16:0/3211888790 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aed851e070 con 0x55aed8631bf0 2024-08-28T19:25:58.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b9633a640 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55aed8631bf0 0x55aed8634010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:58.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b79ffb640 10 client.16854 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:25:58.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.629+0000 7f0b9633a640 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55aed8631bf0 0x55aed8634010 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ffeed629b2a53ecd server_cookie=83a10020ef9cfd74 in_seq=0 out_seq=0 2024-08-28T19:25:58.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b79ffb640 1 -- 172.21.15.16:0/3211888790 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aed851e070 con 0x55aed8631bf0 2024-08-28T19:25:58.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b79ffb640 10 client.16854 handle_command_reply: tid=0 2024-08-28T19:25:58.205 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:25:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 1 client.16854 shutdown 2024-08-28T19:25:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 1 -- 172.21.15.16:0/3211888790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81e3390 msgr2=0x55aed85c11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81e3390 0x55aed85c11b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0b80102e90 tx=0x7f0b8003b750 comp rx=0 tx=0).stop 2024-08-28T19:25:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 1 -- 172.21.15.16:0/3211888790 shutdown_connections 2024-08-28T19:25:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55aed8631bf0 0x55aed8634010 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aed8204070 0x55aed8626770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aed81e3390 0x55aed85c11b0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b79ffb640 0 client.16854 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:25:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 1 --2- 172.21.15.16:0/3211888790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aed81c29f0 0x55aed85c0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 1 -- 172.21.15.16:0/3211888790 >> 172.21.15.16:0/3211888790 conn(0x55aed81f8460 msgr2=0x55aed85bb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:58.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 1 -- 172.21.15.16:0/3211888790 shutdown_connections 2024-08-28T19:25:58.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 1 -- 172.21.15.16:0/3211888790 wait complete. 2024-08-28T19:25:58.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.630+0000 7f0b9eb13740 20 client.16854 trim_cache size 0 max 16384 2024-08-28T19:25:58.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 -- 172.21.15.16:0/3514812208 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0b7c0b4ca0 msgr2=0x7f0b7c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:58.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0b7c0b4ca0 0x7f0b7c0b71b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0b88005fd0 tx=0x7f0b88009600 comp rx=0 tx=0).stop 2024-08-28T19:25:58.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 -- 172.21.15.16:0/3514812208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b90108170 msgr2=0x7f0b9019a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:58.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b90108170 0x7f0b9019a810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0b8002a930 tx=0x7f0b80004060 comp rx=0 tx=0).stop 2024-08-28T19:25:58.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 -- 172.21.15.16:0/3514812208 shutdown_connections 2024-08-28T19:25:58.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b90108ac0 0x7f0b9019fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0b7c0b4ca0 0x7f0b7c0b71b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b90108170 0x7f0b9019a810 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 --2- 172.21.15.16:0/3514812208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b90102830 0x7f0b9019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:58.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 -- 172.21.15.16:0/3514812208 >> 172.21.15.16:0/3514812208 conn(0x7f0b900794d0 msgr2=0x7f0b90101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:58.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 -- 172.21.15.16:0/3514812208 shutdown_connections 2024-08-28T19:25:58.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:57.631+0000 7f0b9859e640 1 -- 172.21.15.16:0/3514812208 wait complete. 2024-08-28T19:25:58.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:58 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:58.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:58 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:58.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:58 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:25:58.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 259471 6.90 MB/sec execute 98 sec latency 41.341 ms 2024-08-28T19:25:59.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:25:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:25:59 smithi028 ceph-mon[21769]: pgmap v487: 129 pgs: 129 active+clean; 152 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 249 op/s 2024-08-28T19:25:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:25:59 smithi115 ceph-mon[22184]: pgmap v487: 129 pgs: 129 active+clean; 152 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 249 op/s 2024-08-28T19:25:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:25:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:25:59 smithi016 ceph-mon[18556]: pgmap v487: 129 pgs: 129 active+clean; 152 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 249 op/s 2024-08-28T19:25:59.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.885+0000 7fc846c70640 1 -- 172.21.15.16:0/1321619405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc840102800 msgr2=0x7fc840102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:59.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.885+0000 7fc846c70640 1 --2- 172.21.15.16:0/1321619405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc840102800 0x7fc840102c10 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fc834009900 tx=0x7fc83402a410 comp rx=0 tx=0).stop 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.885+0000 7fc846c70640 1 -- 172.21.15.16:0/1321619405 shutdown_connections 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.885+0000 7fc846c70640 1 --2- 172.21.15.16:0/1321619405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc840108a90 0x7fc84010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.885+0000 7fc846c70640 1 --2- 172.21.15.16:0/1321619405 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc840108140 0x7fc840108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.885+0000 7fc846c70640 1 --2- 172.21.15.16:0/1321619405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc840102800 0x7fc840102c10 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.885+0000 7fc846c70640 1 -- 172.21.15.16:0/1321619405 >> 172.21.15.16:0/1321619405 conn(0x7fc8400794d0 msgr2=0x7fc8400798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.885+0000 7fc846c70640 1 -- 172.21.15.16:0/1321619405 shutdown_connections 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.885+0000 7fc846c70640 1 -- 172.21.15.16:0/1321619405 wait complete. 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc846c70640 1 Processor -- start 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc846c70640 1 -- start start 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc846c70640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc840102800 0x7fc8401991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc846c70640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc840108140 0x7fc840199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc846c70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc840108a90 0x7fc84019ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc846c70640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401126f0 con 0x7fc840108140 2024-08-28T19:25:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc846c70640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc840112570 con 0x7fc840108a90 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc846c70640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc840112870 con 0x7fc840102800 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc83ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc840108140 0x7fc840199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc83ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc840108140 0x7fc840199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45706/0 (socket says 172.21.15.16:45706) 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc83ffff640 1 -- 172.21.15.16:0/2147812808 learned_addr learned my addr 172.21.15.16:0/2147812808 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.886+0000 7fc844a0c640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc840102800 0x7fc8401991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc84520d640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc840108a90 0x7fc84019ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc83ffff640 1 -- 172.21.15.16:0/2147812808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc840102800 msgr2=0x7fc8401991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc83ffff640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc840102800 0x7fc8401991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc83ffff640 1 -- 172.21.15.16:0/2147812808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc840108a90 msgr2=0x7fc84019ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc83ffff640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc840108a90 0x7fc84019ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc83ffff640 1 -- 172.21.15.16:0/2147812808 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84019f130 con 0x7fc840108140 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc84520d640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc840108a90 0x7fc84019ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc83ffff640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc840108140 0x7fc840199760 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00e9c0 tx=0x7fc82c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc83dffb640 1 -- 172.21.15.16:0/2147812808 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c00cd80 con 0x7fc840108140 2024-08-28T19:25:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc83dffb640 1 -- 172.21.15.16:0/2147812808 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc82c0054d0 con 0x7fc840108140 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc846c70640 1 -- 172.21.15.16:0/2147812808 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84019f420 con 0x7fc840108140 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.887+0000 7fc83dffb640 1 -- 172.21.15.16:0/2147812808 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c010430 con 0x7fc840108140 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.888+0000 7fc846c70640 1 -- 172.21.15.16:0/2147812808 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8401a5b80 con 0x7fc840108140 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.890+0000 7fc83dffb640 1 -- 172.21.15.16:0/2147812808 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc82c004fd0 con 0x7fc840108140 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.890+0000 7fc84d1e5740 1 Processor -- start 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.890+0000 7fc84d1e5740 1 -- start start 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3ca15070 0x564d3cdbcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdba310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9f4390 0x564d3cdba9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564d3cd31680 con 0x564d3c9f4390 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d3cd20c10 con 0x564d3ca15070 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564d3cd24bc0 con 0x564d3c9ec1b0 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc844a0c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdba310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc844a0c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdba310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55684/0 (socket says 172.21.15.16:55684) 2024-08-28T19:25:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc844a0c640 1 -- 172.21.15.16:0/3945999548 learned_addr learned my addr 172.21.15.16:0/3945999548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc83ffff640 1 --2- 172.21.15.16:0/3945999548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9f4390 0x564d3cdba9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc84520d640 1 --2- 172.21.15.16:0/3945999548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3ca15070 0x564d3cdbcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc844a0c640 1 -- 172.21.15.16:0/3945999548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3ca15070 msgr2=0x564d3cdbcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc844a0c640 1 --2- 172.21.15.16:0/3945999548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3ca15070 0x564d3cdbcdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc844a0c640 1 -- 172.21.15.16:0/3945999548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9f4390 msgr2=0x564d3cdba9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc844a0c640 1 --2- 172.21.15.16:0/3945999548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9f4390 0x564d3cdba9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.891+0000 7fc844a0c640 1 -- 172.21.15.16:0/3945999548 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d3ca80650 con 0x564d3c9ec1b0 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.892+0000 7fc83dffb640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8300b4ca0 0x7fc8300b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.892+0000 7fc844a0c640 1 --2- 172.21.15.16:0/3945999548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdba310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc834009900 tx=0x7fc834032b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd87c8d0c506d59c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.892+0000 7fc84520d640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8300b4ca0 0x7fc8300b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.892+0000 7fc83dffb640 1 -- 172.21.15.16:0/2147812808 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc82c0e5cc0 con 0x7fc840108140 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.892+0000 7fc82affd640 1 -- 172.21.15.16:0/3945999548 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834040070 con 0x564d3c9ec1b0 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.892+0000 7fc82affd640 1 -- 172.21.15.16:0/3945999548 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc834004030 con 0x564d3c9ec1b0 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.892+0000 7fc82affd640 1 -- 172.21.15.16:0/3945999548 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834035040 con 0x564d3c9ec1b0 2024-08-28T19:25:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.893+0000 7fc84d1e5740 1 -- 172.21.15.16:0/3945999548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 msgr2=0x564d3cdba310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.893+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/3945999548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdba310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc834009900 tx=0x7fc834032b90 comp rx=0 tx=0).stop 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.893+0000 7fc84520d640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8300b4ca0 0x7fc8300b71b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc838009fd0 tx=0x7fc838009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.893+0000 7fc84d1e5740 1 -- 172.21.15.16:0/3945999548 shutdown_connections 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.893+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/3945999548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9f4390 0x564d3cdba9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.893+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/3945999548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdba310 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.893+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/3945999548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3ca15070 0x564d3cdbcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.893+0000 7fc84d1e5740 1 -- 172.21.15.16:0/3945999548 >> 172.21.15.16:0/3945999548 conn(0x564d3ca09460 msgr2=0x564d3cdb8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.893+0000 7fc84d1e5740 1 -- 172.21.15.16:0/3945999548 shutdown_connections 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.893+0000 7fc84d1e5740 1 -- 172.21.15.16:0/3945999548 wait complete. 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc84d1e5740 1 -- start start 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc84d1e5740 1 Processor -- start 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9ec1b0 0x564d3cdca050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cdca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3ca15070 0x564d3cdd61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564d3cb83700 con 0x564d3c9ec1b0 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d3cca7750 con 0x564d3c9f4390 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564d3cdccde0 con 0x564d3ca15070 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc84d1e5740 10 client.0 _ll_register_callbacks cb 0x564d3ccd6e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc844a0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9ec1b0 0x564d3cdca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc844a0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9ec1b0 0x564d3cdca050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45720/0 (socket says 172.21.15.16:45720) 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc844a0c640 1 -- 172.21.15.16:0/56092002 learned_addr learned my addr 172.21.15.16:0/56092002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.894+0000 7fc84520d640 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3ca15070 0x564d3cdd61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc83ffff640 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cdca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc84520d640 1 -- 172.21.15.16:0/56092002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 msgr2=0x564d3cdca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc84520d640 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cdca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc84520d640 1 -- 172.21.15.16:0/56092002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9ec1b0 msgr2=0x564d3cdca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc84520d640 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9ec1b0 0x564d3cdca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc84520d640 1 -- 172.21.15.16:0/56092002 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d3cce0900 con 0x564d3ca15070 2024-08-28T19:25:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc844a0c640 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9ec1b0 0x564d3cdca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc84520d640 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3ca15070 0x564d3cdd61d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc838003f20 tx=0x7fc838004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27509b5f9ccf73bb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc81ffff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc81ffff640 1 -- 172.21.15.16:0/56092002 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83800fe60 con 0x564d3ca15070 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc81ffff640 1 -- 172.21.15.16:0/56092002 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc8380047d0 con 0x564d3ca15070 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc84d1e5740 1 -- 172.21.15.16:0/56092002 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc834004030 con 0x564d3ca15070 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc81ffff640 1 -- 172.21.15.16:0/56092002 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc838012020 con 0x564d3ca15070 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc84d1e5740 10 client.26269 fetch_fsmap learned FSMap version 40 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.895+0000 7fc84d1e5740 1 -- 172.21.15.16:0/56092002 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x564d3cdd8c80 con 0x564d3ca15070 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.896+0000 7fc81ffff640 1 -- 172.21.15.16:0/56092002 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc834004030 con 0x564d3ca15070 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.896+0000 7fc81ffff640 1 -- 172.21.15.16:0/56092002 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fc838027070 con 0x564d3ca15070 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.896+0000 7fc84d1e5740 10 client.26269 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.896+0000 7fc84d1e5740 10 client.26269 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.896+0000 7fc84d1e5740 20 client.26269 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.896+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564d3c9d39f0 0x564d3cddc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.896+0000 7fc84d1e5740 4 client.26269 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.896+0000 7fc84d1e5740 1 -- 172.21.15.16:0/56092002 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d3cd335d0 con 0x564d3c9d39f0 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.896+0000 7fc844a0c640 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564d3c9d39f0 0x564d3cddc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.896+0000 7fc81ffff640 10 client.26269 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.896+0000 7fc844a0c640 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564d3c9d39f0 0x564d3cddc2f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=15a5d8379f953e66 server_cookie=400402dd0ea13325 in_seq=0 out_seq=0 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc81ffff640 1 -- 172.21.15.16:0/56092002 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564d3cd335d0 con 0x564d3c9d39f0 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc81ffff640 10 client.26269 handle_command_reply: tid=0 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc84d1e5740 1 client.26269 shutdown 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc84d1e5740 1 -- 172.21.15.16:0/56092002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3ca15070 msgr2=0x564d3cdd61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3ca15070 0x564d3cdd61d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc838003f20 tx=0x7fc838004ab0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc84d1e5740 1 -- 172.21.15.16:0/56092002 shutdown_connections 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564d3c9d39f0 0x564d3cddc2f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3ca15070 0x564d3cdd61d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cdca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/56092002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c9ec1b0 0x564d3cdca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc81ffff640 0 client.26269 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:25:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.897+0000 7fc84d1e5740 1 -- 172.21.15.16:0/56092002 >> 172.21.15.16:0/56092002 conn(0x564d3ca09460 msgr2=0x564d3cdbbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.898+0000 7fc84d1e5740 1 -- 172.21.15.16:0/56092002 shutdown_connections 2024-08-28T19:25:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.898+0000 7fc84d1e5740 1 -- 172.21.15.16:0/56092002 wait complete. 2024-08-28T19:25:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.898+0000 7fc84d1e5740 20 client.26269 trim_cache size 0 max 16384 2024-08-28T19:25:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.912+0000 7fc84d1e5740 1 Processor -- start 2024-08-28T19:25:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc84d1e5740 1 -- start start 2024-08-28T19:25:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3ca09460 0x564d3cdba180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdba6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cde1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401aea60 con 0x564d3ca09460 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401b0660 con 0x564d3c9f4390 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401af190 con 0x564d3c9ec1b0 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc84520d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cde1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc844a0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3ca09460 0x564d3cdba180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc83ffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdba6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc84520d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cde1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48206/0 (socket says 172.21.15.16:48206) 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc84520d640 1 -- 172.21.15.16:0/3720652448 learned_addr learned my addr 172.21.15.16:0/3720652448 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc83ffff640 1 -- 172.21.15.16:0/3720652448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 msgr2=0x564d3cde1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc83ffff640 1 --2- 172.21.15.16:0/3720652448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cde1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc83ffff640 1 -- 172.21.15.16:0/3720652448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3ca09460 msgr2=0x564d3cdba180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc83ffff640 1 --2- 172.21.15.16:0/3720652448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3ca09460 0x564d3cdba180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc83ffff640 1 -- 172.21.15.16:0/3720652448 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d3cdb8af0 con 0x564d3c9ec1b0 2024-08-28T19:25:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc844a0c640 1 --2- 172.21.15.16:0/3720652448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3ca09460 0x564d3cdba180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.913+0000 7fc84520d640 1 --2- 172.21.15.16:0/3720652448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cde1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:25:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.914+0000 7fc83ffff640 1 --2- 172.21.15.16:0/3720652448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdba6c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0e30a0 tx=0x7fc82c0e9810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80403f5572fd1caa server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.914+0000 7fc82a7fc640 1 -- 172.21.15.16:0/3720652448 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c0e7070 con 0x564d3c9ec1b0 2024-08-28T19:25:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.914+0000 7fc82a7fc640 1 -- 172.21.15.16:0/3720652448 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc82c0046c0 con 0x564d3c9ec1b0 2024-08-28T19:25:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.914+0000 7fc82a7fc640 1 -- 172.21.15.16:0/3720652448 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c0ac040 con 0x564d3c9ec1b0 2024-08-28T19:25:59.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.915+0000 7fc84d1e5740 1 -- 172.21.15.16:0/3720652448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 msgr2=0x564d3cdba6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:59.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.915+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/3720652448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdba6c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0e30a0 tx=0x7fc82c0e9810 comp rx=0 tx=0).stop 2024-08-28T19:25:59.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.915+0000 7fc84d1e5740 1 -- 172.21.15.16:0/3720652448 shutdown_connections 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.915+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/3720652448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cde1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.915+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/3720652448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdba6c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.915+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/3720652448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3ca09460 0x564d3cdba180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.915+0000 7fc84d1e5740 1 -- 172.21.15.16:0/3720652448 >> 172.21.15.16:0/3720652448 conn(0x564d3cdc2420 msgr2=0x564d3cdc88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.915+0000 7fc84d1e5740 1 -- 172.21.15.16:0/3720652448 shutdown_connections 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.915+0000 7fc84d1e5740 1 -- 172.21.15.16:0/3720652448 wait complete. 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.915+0000 7fc84d1e5740 1 -- start start 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc84d1e5740 1 Processor -- start 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdd0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cdd0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc84d1e5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3ca09460 0x564d3cdd0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401aff90 con 0x564d3ca09460 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401af890 con 0x564d3c9f4390 2024-08-28T19:25:59.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc84d1e5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401aca40 con 0x564d3c9ec1b0 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc84d1e5740 10 client.0 _ll_register_callbacks cb 0x7fc8380125b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc844a0c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdd0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc84520d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3ca09460 0x564d3cdd0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc844a0c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdd0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55710/0 (socket says 172.21.15.16:55710) 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc844a0c640 1 -- 172.21.15.16:0/1459247880 learned_addr learned my addr 172.21.15.16:0/1459247880 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc83ffff640 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cdd0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc844a0c640 1 -- 172.21.15.16:0/1459247880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 msgr2=0x564d3cdd0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc844a0c640 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cdd0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc844a0c640 1 -- 172.21.15.16:0/1459247880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3ca09460 msgr2=0x564d3cdd0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc844a0c640 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3ca09460 0x564d3cdd0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.916+0000 7fc844a0c640 1 -- 172.21.15.16:0/1459247880 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d3ce45d30 con 0x564d3c9ec1b0 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.917+0000 7fc844a0c640 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdd0150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x564d3cdcc850 tx=0x7fc8340050e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3b434ff093b560 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.917+0000 7fc81ffff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.917+0000 7fc81ffff640 1 -- 172.21.15.16:0/1459247880 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834056060 con 0x564d3c9ec1b0 2024-08-28T19:25:59.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.917+0000 7fc81ffff640 1 -- 172.21.15.16:0/1459247880 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc834031d10 con 0x564d3c9ec1b0 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.917+0000 7fc84d1e5740 1 -- 172.21.15.16:0/1459247880 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564d3cd2f070 con 0x564d3c9ec1b0 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.917+0000 7fc81ffff640 1 -- 172.21.15.16:0/1459247880 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc834040040 con 0x564d3c9ec1b0 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.917+0000 7fc84d1e5740 10 client.26275 fetch_fsmap learned FSMap version 40 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.917+0000 7fc84d1e5740 1 -- 172.21.15.16:0/1459247880 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x564d3ce46fe0 con 0x564d3c9ec1b0 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.917+0000 7fc81ffff640 1 -- 172.21.15.16:0/1459247880 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564d3cd2f070 con 0x564d3c9ec1b0 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc81ffff640 1 -- 172.21.15.16:0/1459247880 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fc8340046c0 con 0x564d3c9ec1b0 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc84d1e5740 10 client.26275 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc84d1e5740 10 client.26275 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc84d1e5740 20 client.26275 populate_metadata read hostname 'smithi016' 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564d3ca15070 0x564d3ce493e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc84d1e5740 4 client.26275 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc84d1e5740 1 -- 172.21.15.16:0/1459247880 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564d3cd2c7b0 con 0x564d3ca15070 2024-08-28T19:25:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc83ffff640 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564d3ca15070 0x564d3ce493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc81ffff640 10 client.26275 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc83ffff640 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564d3ca15070 0x564d3ce493e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fe341dcba653d7db server_cookie=131c215beb340dd9 in_seq=0 out_seq=0 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc81ffff640 1 -- 172.21.15.16:0/1459247880 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564d3cd2c7b0 con 0x564d3ca15070 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc81ffff640 10 client.26275 handle_command_reply: tid=0 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.918+0000 7fc84d1e5740 1 client.26275 shutdown 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc84d1e5740 1 -- 172.21.15.16:0/1459247880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 msgr2=0x564d3cdd0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdd0150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x564d3cdcc850 tx=0x7fc8340050e0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc84d1e5740 1 -- 172.21.15.16:0/1459247880 shutdown_connections 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564d3ca15070 0x564d3ce493e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3ca09460 0x564d3cdd0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d3c9f4390 0x564d3cdd0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc84d1e5740 1 --2- 172.21.15.16:0/1459247880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564d3c9ec1b0 0x564d3cdd0150 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc81ffff640 0 client.26275 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:25:59.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc84d1e5740 1 -- 172.21.15.16:0/1459247880 >> 172.21.15.16:0/1459247880 conn(0x564d3cdc2420 msgr2=0x564d3cdcb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:59.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc84d1e5740 1 -- 172.21.15.16:0/1459247880 shutdown_connections 2024-08-28T19:25:59.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc84d1e5740 1 -- 172.21.15.16:0/1459247880 wait complete. 2024-08-28T19:25:59.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc84d1e5740 20 client.26275 trim_cache size 0 max 16384 2024-08-28T19:25:59.926 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc846c70640 1 -- 172.21.15.16:0/2147812808 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8300b4ca0 msgr2=0x7fc8300b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc846c70640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8300b4ca0 0x7fc8300b71b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc838009fd0 tx=0x7fc838009500 comp rx=0 tx=0).stop 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc846c70640 1 -- 172.21.15.16:0/2147812808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc840108140 msgr2=0x7fc840199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.919+0000 7fc846c70640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc840108140 0x7fc840199760 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00e9c0 tx=0x7fc82c00ee80 comp rx=0 tx=0).stop 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.920+0000 7fc846c70640 1 -- 172.21.15.16:0/2147812808 shutdown_connections 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.920+0000 7fc846c70640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc8300b4ca0 0x7fc8300b71b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.920+0000 7fc846c70640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc840108a90 0x7fc84019ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.920+0000 7fc846c70640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc840108140 0x7fc840199760 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.920+0000 7fc846c70640 1 --2- 172.21.15.16:0/2147812808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc840102800 0x7fc8401991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.920+0000 7fc846c70640 1 -- 172.21.15.16:0/2147812808 >> 172.21.15.16:0/2147812808 conn(0x7fc8400794d0 msgr2=0x7fc840101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.920+0000 7fc846c70640 1 -- 172.21.15.16:0/2147812808 shutdown_connections 2024-08-28T19:25:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:25:59.920+0000 7fc846c70640 1 -- 172.21.15.16:0/2147812808 wait complete. 2024-08-28T19:25:59.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 261539 6.91 MB/sec execute 99 sec latency 59.468 ms 2024-08-28T19:26:00.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:26:00.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 262338 6.90 MB/sec execute 100 sec latency 604.694 ms 2024-08-28T19:26:01.067 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:01 smithi028 ceph-mon[21769]: pgmap v488: 129 pgs: 129 active+clean; 153 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 270 op/s 2024-08-28T19:26:01.329 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:01 smithi016 ceph-mon[18556]: pgmap v488: 129 pgs: 129 active+clean; 153 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 270 op/s 2024-08-28T19:26:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:01 smithi115 ceph-mon[22184]: pgmap v488: 129 pgs: 129 active+clean; 153 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 270 op/s 2024-08-28T19:26:01.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.577+0000 7f716bb7d640 1 -- 172.21.15.16:0/3050496986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7164104790 msgr2=0x7f7164104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:01.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.577+0000 7f716bb7d640 1 --2- 172.21.15.16:0/3050496986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7164104790 0x7f7164104ba0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7160009900 tx=0x7f716002a410 comp rx=0 tx=0).stop 2024-08-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.577+0000 7f716bb7d640 1 -- 172.21.15.16:0/3050496986 shutdown_connections 2024-08-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.577+0000 7f716bb7d640 1 --2- 172.21.15.16:0/3050496986 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f716410dbf0 0x7f71641100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.577+0000 7f716bb7d640 1 --2- 172.21.15.16:0/3050496986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71641050e0 0x7f716410d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.577+0000 7f716bb7d640 1 --2- 172.21.15.16:0/3050496986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7164104790 0x7f7164104ba0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.577+0000 7f716bb7d640 1 -- 172.21.15.16:0/3050496986 >> 172.21.15.16:0/3050496986 conn(0x7f7164079580 msgr2=0x7f7164079990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.577+0000 7f716bb7d640 1 -- 172.21.15.16:0/3050496986 shutdown_connections 2024-08-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.577+0000 7f716bb7d640 1 -- 172.21.15.16:0/3050496986 wait complete. 2024-08-28T19:26:01.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f716bb7d640 1 Processor -- start 2024-08-28T19:26:01.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f716bb7d640 1 -- start start 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f716bb7d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7164104790 0x7f71641024d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f716bb7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71641050e0 0x7f7164102a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f716bb7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716410dbf0 0x7f7164102f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f716bb7d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7164113870 con 0x7f71641050e0 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f716bb7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71641136f0 con 0x7f716410dbf0 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f716bb7d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f71641139f0 con 0x7f7164104790 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f7169118640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71641050e0 0x7f7164102a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f716a11a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716410dbf0 0x7f7164102f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f716a11a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716410dbf0 0x7f7164102f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48236/0 (socket says 172.21.15.16:48236) 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f7169118640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71641050e0 0x7f7164102a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45740/0 (socket says 172.21.15.16:45740) 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.578+0000 7f716a11a640 1 -- 172.21.15.16:0/3018190652 learned_addr learned my addr 172.21.15.16:0/3018190652 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.579+0000 7f7169919640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7164104790 0x7f71641024d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.579+0000 7f716a11a640 1 -- 172.21.15.16:0/3018190652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7164104790 msgr2=0x7f71641024d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.579+0000 7f716a11a640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7164104790 0x7f71641024d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.579+0000 7f716a11a640 1 -- 172.21.15.16:0/3018190652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71641050e0 msgr2=0x7f7164102a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.579+0000 7f716a11a640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71641050e0 0x7f7164102a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.579+0000 7f716a11a640 1 -- 172.21.15.16:0/3018190652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71641a8560 con 0x7f716410dbf0 2024-08-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.579+0000 7f716a11a640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716410dbf0 0x7f7164102f50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f715c00cad0 tx=0x7f715c00cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.579+0000 7f7156ffd640 1 -- 172.21.15.16:0/3018190652 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c00ae20 con 0x7f716410dbf0 2024-08-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.579+0000 7f7156ffd640 1 -- 172.21.15.16:0/3018190652 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f715c005020 con 0x7f716410dbf0 2024-08-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.579+0000 7f7156ffd640 1 -- 172.21.15.16:0/3018190652 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c00f6b0 con 0x7f716410dbf0 2024-08-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.579+0000 7f716bb7d640 1 -- 172.21.15.16:0/3018190652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71641a8850 con 0x7f716410dbf0 2024-08-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.580+0000 7f716bb7d640 1 -- 172.21.15.16:0/3018190652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71641a8d40 con 0x7f716410dbf0 2024-08-28T19:26:01.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.582+0000 7f71720f2740 1 Processor -- start 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.582+0000 7f7156ffd640 1 -- 172.21.15.16:0/3018190652 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f715c00f850 con 0x7f716410dbf0 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.582+0000 7f71720f2740 1 -- start start 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f71720f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f19111c070 0x55f1914c1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f71720f2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910f31b0 0x55f1914c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f71720f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910fb390 0x55f1914c1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f71720f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f191438680 con 0x55f19111c070 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f71720f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f191427c10 con 0x55f1910fb390 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f71720f2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f19142bbc0 con 0x55f1910f31b0 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f7169118640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f19111c070 0x55f1914c1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f7169118640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f19111c070 0x55f1914c1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45746/0 (socket says 172.21.15.16:45746) 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f7169118640 1 -- 172.21.15.16:0/3061422439 learned_addr learned my addr 172.21.15.16:0/3061422439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f7169118640 1 --2- 172.21.15.16:0/3061422439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910fb390 0x55f1914c1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f7169919640 1 --2- 172.21.15.16:0/3061422439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910f31b0 0x55f1914c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f7156ffd640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f71500b4d20 0x7f71500b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f716a11a640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f71500b4d20 0x7f71500b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.583+0000 7f7169118640 1 -- 172.21.15.16:0/3061422439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910f31b0 msgr2=0x55f1914c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.584+0000 7f7169118640 1 --2- 172.21.15.16:0/3061422439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910f31b0 0x55f1914c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.584+0000 7f7169118640 1 -- 172.21.15.16:0/3061422439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910fb390 msgr2=0x55f1914c1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.584+0000 7f7169118640 1 --2- 172.21.15.16:0/3061422439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910fb390 0x55f1914c1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.584+0000 7f7169118640 1 -- 172.21.15.16:0/3061422439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f191187650 con 0x55f19111c070 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.584+0000 7f7156ffd640 1 -- 172.21.15.16:0/3018190652 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f715c0e57d0 con 0x7f716410dbf0 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.584+0000 7f7169118640 1 --2- 172.21.15.16:0/3061422439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f19111c070 0x55f1914c1310 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f71580058a0 tx=0x7f7158005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b88fe06af977155 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.584+0000 7f714ffff640 1 -- 172.21.15.16:0/3061422439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71580120d0 con 0x55f19111c070 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.584+0000 7f716a11a640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f71500b4d20 0x7f71500b7230 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f715c0abb90 tx=0x7f715c0a7ba0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.584+0000 7f714ffff640 1 -- 172.21.15.16:0/3061422439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7158012ca0 con 0x55f19111c070 2024-08-28T19:26:01.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.584+0000 7f714ffff640 1 -- 172.21.15.16:0/3061422439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71580047a0 con 0x55f19111c070 2024-08-28T19:26:01.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.585+0000 7f71720f2740 1 -- 172.21.15.16:0/3061422439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f19111c070 msgr2=0x55f1914c1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:01.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.585+0000 7f71720f2740 1 --2- 172.21.15.16:0/3061422439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f19111c070 0x55f1914c1310 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f71580058a0 tx=0x7f7158005c90 comp rx=0 tx=0).stop 2024-08-28T19:26:01.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.585+0000 7f71720f2740 1 -- 172.21.15.16:0/3061422439 shutdown_connections 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.585+0000 7f71720f2740 1 --2- 172.21.15.16:0/3061422439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910fb390 0x55f1914c1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.585+0000 7f71720f2740 1 --2- 172.21.15.16:0/3061422439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910f31b0 0x55f1914c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.585+0000 7f71720f2740 1 --2- 172.21.15.16:0/3061422439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f19111c070 0x55f1914c1310 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.585+0000 7f71720f2740 1 -- 172.21.15.16:0/3061422439 >> 172.21.15.16:0/3061422439 conn(0x55f191110460 msgr2=0x55f1914bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.585+0000 7f71720f2740 1 -- 172.21.15.16:0/3061422439 shutdown_connections 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.585+0000 7f71720f2740 1 -- 172.21.15.16:0/3061422439 wait complete. 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.586+0000 7f71720f2740 1 -- start start 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.586+0000 7f71720f2740 1 Processor -- start 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.586+0000 7f71720f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 0x55f1914cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.586+0000 7f71720f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910fb390 0x55f1914ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.586+0000 7f71720f2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f19111c070 0x55f1914d8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.586+0000 7f71720f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f19128a700 con 0x55f1910f31b0 2024-08-28T19:26:01.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.586+0000 7f71720f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f1913ae750 con 0x55f1910fb390 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.586+0000 7f71720f2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f1914cfa90 con 0x55f19111c070 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f71720f2740 10 client.0 _ll_register_callbacks cb 0x55f1913dde10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f7169118640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 0x55f1914cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f7169118640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 0x55f1914cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45758/0 (socket says 172.21.15.16:45758) 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f7169118640 1 -- 172.21.15.16:0/276775199 learned_addr learned my addr 172.21.15.16:0/276775199 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f7169919640 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910fb390 0x55f1914ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f7169118640 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f19111c070 0x55f1914d8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f7169919640 1 -- 172.21.15.16:0/276775199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f19111c070 msgr2=0x55f1914d8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f7169919640 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f19111c070 0x55f1914d8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f7169919640 1 -- 172.21.15.16:0/276775199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 msgr2=0x55f1914cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f7169919640 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 0x55f1914cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f7169919640 1 -- 172.21.15.16:0/276775199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1913e7900 con 0x55f1910fb390 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f7169919640 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910fb390 0x55f1914ccf90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7160006010 tx=0x7f71600316d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adeff30290b945d4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.587+0000 7f714d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:26:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f714d7fa640 1 -- 172.21.15.16:0/276775199 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7160035040 con 0x55f1910fb390 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f714d7fa640 1 -- 172.21.15.16:0/276775199 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7160031d10 con 0x55f1910fb390 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f71720f2740 1 -- 172.21.15.16:0/276775199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7158012ca0 con 0x55f1910fb390 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f714d7fa640 1 -- 172.21.15.16:0/276775199 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f716004e050 con 0x55f1910fb390 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f71720f2740 10 client.26459 fetch_fsmap learned FSMap version 40 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f71720f2740 1 -- 172.21.15.16:0/276775199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f1914db820 con 0x55f1910fb390 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f714d7fa640 1 -- 172.21.15.16:0/276775199 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7158012ca0 con 0x55f1910fb390 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f714d7fa640 1 -- 172.21.15.16:0/276775199 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f7160004750 con 0x55f1910fb390 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f71720f2740 10 client.26459 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f71720f2740 10 client.26459 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f71720f2740 20 client.26459 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f71720f2740 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f1910da9f0 0x55f1914ded30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f71720f2740 4 client.26459 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.588+0000 7f71720f2740 1 -- 172.21.15.16:0/276775199 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f19143a5d0 con 0x55f1910da9f0 2024-08-28T19:26:01.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.589+0000 7f7169118640 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f1910da9f0 0x55f1914ded30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.589+0000 7f714d7fa640 10 client.26459 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:26:01.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.589+0000 7f7169118640 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f1910da9f0 0x55f1914ded30 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=85026920ae8058df server_cookie=440924c3d875f765 in_seq=0 out_seq=0 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f714d7fa640 1 -- 172.21.15.16:0/276775199 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f19143a5d0 con 0x55f1910da9f0 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f714d7fa640 10 client.26459 handle_command_reply: tid=0 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f71720f2740 1 client.26459 shutdown 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f71720f2740 1 -- 172.21.15.16:0/276775199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910fb390 msgr2=0x55f1914ccf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f71720f2740 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910fb390 0x55f1914ccf90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7160006010 tx=0x7f71600316d0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f71720f2740 1 -- 172.21.15.16:0/276775199 shutdown_connections 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f71720f2740 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f1910da9f0 0x55f1914ded30 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f71720f2740 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f19111c070 0x55f1914d8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f71720f2740 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910fb390 0x55f1914ccf90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f714d7fa640 0 client.26459 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f71720f2740 1 --2- 172.21.15.16:0/276775199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 0x55f1914cca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f71720f2740 1 -- 172.21.15.16:0/276775199 >> 172.21.15.16:0/276775199 conn(0x55f191110460 msgr2=0x55f1914c44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:01.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f71720f2740 1 -- 172.21.15.16:0/276775199 shutdown_connections 2024-08-28T19:26:01.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.590+0000 7f71720f2740 1 -- 172.21.15.16:0/276775199 wait complete. 2024-08-28T19:26:01.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.591+0000 7f71720f2740 20 client.26459 trim_cache size 0 max 16384 2024-08-28T19:26:01.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f71720f2740 1 Processor -- start 2024-08-28T19:26:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f71720f2740 1 -- start start 2024-08-28T19:26:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f71720f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910da9f0 0x55f1914bf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f71720f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910f31b0 0x55f1914d8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f71720f2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910fb390 0x55f1914c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f71720f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f71641b2df0 con 0x55f1910da9f0 2024-08-28T19:26:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f71720f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71641b49f0 con 0x55f1910f31b0 2024-08-28T19:26:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f71720f2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f71641b3520 con 0x55f1910fb390 2024-08-28T19:26:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f7169118640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910da9f0 0x55f1914bf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f7169118640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910da9f0 0x55f1914bf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45768/0 (socket says 172.21.15.16:45768) 2024-08-28T19:26:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f7169118640 1 -- 172.21.15.16:0/492760093 learned_addr learned my addr 172.21.15.16:0/492760093 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.607+0000 7f7169919640 1 --2- 172.21.15.16:0/492760093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910f31b0 0x55f1914d8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.608+0000 7f7169118640 1 --2- 172.21.15.16:0/492760093 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910fb390 0x55f1914c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.608+0000 7f7169919640 1 -- 172.21.15.16:0/492760093 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910fb390 msgr2=0x55f1914c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.608+0000 7f7169919640 1 --2- 172.21.15.16:0/492760093 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910fb390 0x55f1914c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.608+0000 7f7169919640 1 -- 172.21.15.16:0/492760093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910da9f0 msgr2=0x55f1914bf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.608+0000 7f7169919640 1 --2- 172.21.15.16:0/492760093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910da9f0 0x55f1914bf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.608+0000 7f7169919640 1 -- 172.21.15.16:0/492760093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1914c3600 con 0x55f1910f31b0 2024-08-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.608+0000 7f7169118640 1 --2- 172.21.15.16:0/492760093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910da9f0 0x55f1914bf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.608+0000 7f7169919640 1 --2- 172.21.15.16:0/492760093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910f31b0 0x55f1914d8c80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7160009900 tx=0x7f7160032de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d834f18996eb618 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.608+0000 7f714f7fe640 1 -- 172.21.15.16:0/492760093 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7160035030 con 0x55f1910f31b0 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.608+0000 7f714f7fe640 1 -- 172.21.15.16:0/492760093 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7160041070 con 0x55f1910f31b0 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.608+0000 7f714f7fe640 1 -- 172.21.15.16:0/492760093 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716004b040 con 0x55f1910f31b0 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.609+0000 7f71720f2740 1 -- 172.21.15.16:0/492760093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910f31b0 msgr2=0x55f1914d8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.609+0000 7f71720f2740 1 --2- 172.21.15.16:0/492760093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910f31b0 0x55f1914d8c80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7160009900 tx=0x7f7160032de0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.609+0000 7f71720f2740 1 -- 172.21.15.16:0/492760093 shutdown_connections 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.609+0000 7f71720f2740 1 --2- 172.21.15.16:0/492760093 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910fb390 0x55f1914c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.609+0000 7f71720f2740 1 --2- 172.21.15.16:0/492760093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910f31b0 0x55f1914d8c80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.609+0000 7f71720f2740 1 --2- 172.21.15.16:0/492760093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910da9f0 0x55f1914bf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.609+0000 7f71720f2740 1 -- 172.21.15.16:0/492760093 >> 172.21.15.16:0/492760093 conn(0x55f191110460 msgr2=0x55f1914beb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.609+0000 7f71720f2740 1 -- 172.21.15.16:0/492760093 shutdown_connections 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.609+0000 7f71720f2740 1 -- 172.21.15.16:0/492760093 wait complete. 2024-08-28T19:26:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.610+0000 7f71720f2740 1 -- start start 2024-08-28T19:26:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.610+0000 7f71720f2740 1 Processor -- start 2024-08-28T19:26:01.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.610+0000 7f71720f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910da9f0 0x55f1914d1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.610+0000 7f71720f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 0x55f1914d1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.610+0000 7f71720f2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910fb390 0x55f1915486f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.610+0000 7f71720f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f71641b4320 con 0x55f1910f31b0 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.610+0000 7f71720f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71641b3c20 con 0x55f1910da9f0 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.610+0000 7f71720f2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f71641b0dd0 con 0x55f1910fb390 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.610+0000 7f71720f2740 10 client.0 _ll_register_callbacks cb 0x7f7160005e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169118640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910da9f0 0x55f1914d1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169919640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 0x55f1914d1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169118640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910da9f0 0x55f1914d1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48268/0 (socket says 172.21.15.16:48268) 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169118640 1 -- 172.21.15.16:0/601437841 learned_addr learned my addr 172.21.15.16:0/601437841 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169118640 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910fb390 0x55f1915486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169118640 1 -- 172.21.15.16:0/601437841 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910fb390 msgr2=0x55f1915486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169118640 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910fb390 0x55f1915486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169118640 1 -- 172.21.15.16:0/601437841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 msgr2=0x55f1914d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169118640 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 0x55f1914d1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169118640 1 -- 172.21.15.16:0/601437841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f191548d60 con 0x55f1910da9f0 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169919640 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 0x55f1914d1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f7169118640 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910da9f0 0x55f1914d1560 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7158005d00 tx=0x7f715802b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=459d573ca3207afe server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f714d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f714d7fa640 1 -- 172.21.15.16:0/601437841 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715800e9f0 con 0x55f1910da9f0 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.611+0000 7f714d7fa640 1 -- 172.21.15.16:0/601437841 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f715800eb90 con 0x55f1910da9f0 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.612+0000 7f71720f2740 1 -- 172.21.15.16:0/601437841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7160041070 con 0x55f1910da9f0 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.612+0000 7f714d7fa640 1 -- 172.21.15.16:0/601437841 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f715803c050 con 0x55f1910da9f0 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.612+0000 7f71720f2740 10 client.26465 fetch_fsmap learned FSMap version 40 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.612+0000 7f71720f2740 1 -- 172.21.15.16:0/601437841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f191549fc0 con 0x55f1910da9f0 2024-08-28T19:26:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.612+0000 7f714d7fa640 1 -- 172.21.15.16:0/601437841 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7160041070 con 0x55f1910da9f0 2024-08-28T19:26:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.612+0000 7f714d7fa640 1 -- 172.21.15.16:0/601437841 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f7158010460 con 0x55f1910da9f0 2024-08-28T19:26:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.613+0000 7f71720f2740 10 client.26465 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:26:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.613+0000 7f71720f2740 10 client.26465 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:26:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.613+0000 7f71720f2740 20 client.26465 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.613+0000 7f71720f2740 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f19111c070 0x55f19154c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.613+0000 7f71720f2740 4 client.26465 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:26:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.613+0000 7f71720f2740 1 -- 172.21.15.16:0/601437841 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f191436070 con 0x55f19111c070 2024-08-28T19:26:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.613+0000 7f7169919640 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f19111c070 0x55f19154c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.614+0000 7f7169919640 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f19111c070 0x55f19154c5b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=397c881457a291b1 server_cookie=b4de89f128ecf961 in_seq=0 out_seq=0 2024-08-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.614+0000 7f714d7fa640 10 client.26465 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.614+0000 7f714d7fa640 1 -- 172.21.15.16:0/601437841 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f191436070 con 0x55f19111c070 2024-08-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.614+0000 7f714d7fa640 10 client.26465 handle_command_reply: tid=0 2024-08-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.614+0000 7f71720f2740 1 client.26465 shutdown 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.614+0000 7f71720f2740 1 -- 172.21.15.16:0/601437841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910da9f0 msgr2=0x55f1914d1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.614+0000 7f71720f2740 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910da9f0 0x55f1914d1560 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7158005d00 tx=0x7f715802b040 comp rx=0 tx=0).stop 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.614+0000 7f71720f2740 1 -- 172.21.15.16:0/601437841 shutdown_connections 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.614+0000 7f71720f2740 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f19111c070 0x55f19154c5b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.615+0000 7f714d7fa640 0 client.26465 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.615+0000 7f71720f2740 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f1910fb390 0x55f1915486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.615+0000 7f71720f2740 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1910f31b0 0x55f1914d1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.615+0000 7f71720f2740 1 --2- 172.21.15.16:0/601437841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1910da9f0 0x55f1914d1560 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.615+0000 7f71720f2740 1 -- 172.21.15.16:0/601437841 >> 172.21.15.16:0/601437841 conn(0x55f191110460 msgr2=0x55f1914c4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.615+0000 7f71720f2740 1 -- 172.21.15.16:0/601437841 shutdown_connections 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.615+0000 7f71720f2740 1 -- 172.21.15.16:0/601437841 wait complete. 2024-08-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.615+0000 7f71720f2740 20 client.26465 trim_cache size 0 max 16384 2024-08-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 -- 172.21.15.16:0/3018190652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f71500b4d20 msgr2=0x7f71500b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f71500b4d20 0x7f71500b7230 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f715c0abb90 tx=0x7f715c0a7ba0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 -- 172.21.15.16:0/3018190652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716410dbf0 msgr2=0x7f7164102f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716410dbf0 0x7f7164102f50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f715c00cad0 tx=0x7f715c00cf90 comp rx=0 tx=0).stop 2024-08-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 -- 172.21.15.16:0/3018190652 shutdown_connections 2024-08-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f71500b4d20 0x7f71500b7230 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716410dbf0 0x7f7164102f50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71641050e0 0x7f7164102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 --2- 172.21.15.16:0/3018190652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7164104790 0x7f71641024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:01.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 -- 172.21.15.16:0/3018190652 >> 172.21.15.16:0/3018190652 conn(0x7f7164079580 msgr2=0x7f7164107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:01.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 -- 172.21.15.16:0/3018190652 shutdown_connections 2024-08-28T19:26:01.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:01.616+0000 7f716bb7d640 1 -- 172.21.15.16:0/3018190652 wait complete. 2024-08-28T19:26:01.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 262691 6.87 MB/sec execute 101 sec latency 1262.327 ms 2024-08-28T19:26:02.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:02.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 263132 6.81 MB/sec execute 102 sec latency 57.708 ms 2024-08-28T19:26:03.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:03 smithi016 ceph-mon[18556]: pgmap v489: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 347 op/s 2024-08-28T19:26:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:03 smithi028 ceph-mon[21769]: pgmap v489: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 347 op/s 2024-08-28T19:26:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.363+0000 7fde43924640 1 -- 172.21.15.16:0/1184342199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c10dbb0 msgr2=0x7fde3c1100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.363+0000 7fde43924640 1 --2- 172.21.15.16:0/1184342199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c10dbb0 0x7fde3c1100a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fde34009ca0 tx=0x7fde3402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.363+0000 7fde43924640 1 -- 172.21.15.16:0/1184342199 shutdown_connections 2024-08-28T19:26:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.363+0000 7fde43924640 1 --2- 172.21.15.16:0/1184342199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c10dbb0 0x7fde3c1100a0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.363+0000 7fde43924640 1 --2- 172.21.15.16:0/1184342199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde3c1050a0 0x7fde3c10d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.363+0000 7fde43924640 1 --2- 172.21.15.16:0/1184342199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c104750 0x7fde3c104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.364+0000 7fde43924640 1 -- 172.21.15.16:0/1184342199 >> 172.21.15.16:0/1184342199 conn(0x7fde3c0794d0 msgr2=0x7fde3c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.364+0000 7fde43924640 1 -- 172.21.15.16:0/1184342199 shutdown_connections 2024-08-28T19:26:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.364+0000 7fde43924640 1 -- 172.21.15.16:0/1184342199 wait complete. 2024-08-28T19:26:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.364+0000 7fde43924640 1 Processor -- start 2024-08-28T19:26:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.364+0000 7fde43924640 1 -- start start 2024-08-28T19:26:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde43924640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde3c104750 0x7fde3c19a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde43924640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c1050a0 0x7fde3c19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde43924640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c10dbb0 0x7fde3c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde43924640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c113830 con 0x7fde3c10dbb0 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde43924640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1136b0 con 0x7fde3c1050a0 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde43924640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1139b0 con 0x7fde3c104750 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde41ec1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c10dbb0 0x7fde3c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde40ebf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c1050a0 0x7fde3c19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde41ec1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c10dbb0 0x7fde3c19fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45782/0 (socket says 172.21.15.16:45782) 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde41ec1640 1 -- 172.21.15.16:0/2810016588 learned_addr learned my addr 172.21.15.16:0/2810016588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde416c0640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde3c104750 0x7fde3c19a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde41ec1640 1 -- 172.21.15.16:0/2810016588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde3c104750 msgr2=0x7fde3c19a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde41ec1640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde3c104750 0x7fde3c19a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde41ec1640 1 -- 172.21.15.16:0/2810016588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c1050a0 msgr2=0x7fde3c19a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde41ec1640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c1050a0 0x7fde3c19a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde41ec1640 1 -- 172.21.15.16:0/2810016588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c1a02b0 con 0x7fde3c10dbb0 2024-08-28T19:26:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde416c0640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde3c104750 0x7fde3c19a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.365+0000 7fde41ec1640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c10dbb0 0x7fde3c19fb80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fde34005cc0 tx=0x7fde34031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.366+0000 7fde327fc640 1 -- 172.21.15.16:0/2810016588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34003b70 con 0x7fde3c10dbb0 2024-08-28T19:26:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.366+0000 7fde327fc640 1 -- 172.21.15.16:0/2810016588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde340318f0 con 0x7fde3c10dbb0 2024-08-28T19:26:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.366+0000 7fde43924640 1 -- 172.21.15.16:0/2810016588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3c1a0540 con 0x7fde3c10dbb0 2024-08-28T19:26:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.366+0000 7fde327fc640 1 -- 172.21.15.16:0/2810016588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3402f950 con 0x7fde3c10dbb0 2024-08-28T19:26:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.366+0000 7fde43924640 1 -- 172.21.15.16:0/2810016588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3c1a6d00 con 0x7fde3c10dbb0 2024-08-28T19:26:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:03 smithi115 ceph-mon[22184]: pgmap v489: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 347 op/s 2024-08-28T19:26:03.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.368+0000 7fde49e99740 1 Processor -- start 2024-08-28T19:26:03.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde327fc640 1 -- 172.21.15.16:0/2810016588 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fde3402fb50 con 0x7fde3c10dbb0 2024-08-28T19:26:03.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde49e99740 1 -- start start 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde49e99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412bac4070 0x56412be69310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde49e99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412ba9b1b0 0x56412be69850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde49e99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 0x56412be69d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde49e99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56412bde0680 con 0x56412baa3390 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde49e99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56412bdcfc10 con 0x56412bac4070 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde49e99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56412bdd3bc0 con 0x56412ba9b1b0 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde416c0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412ba9b1b0 0x56412be69850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde416c0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412ba9b1b0 0x56412be69850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55786/0 (socket says 172.21.15.16:55786) 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde416c0640 1 -- 172.21.15.16:0/2318439548 learned_addr learned my addr 172.21.15.16:0/2318439548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde40ebf640 1 --2- 172.21.15.16:0/2318439548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412bac4070 0x56412be69310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.369+0000 7fde40ebf640 1 --2- 172.21.15.16:0/2318439548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 0x56412be69d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde327fc640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde280b4d20 0x7fde280b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde416c0640 1 -- 172.21.15.16:0/2318439548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412bac4070 msgr2=0x56412be69310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde416c0640 1 --2- 172.21.15.16:0/2318439548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412bac4070 0x56412be69310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde416c0640 1 -- 172.21.15.16:0/2318439548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 msgr2=0x56412be69d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde416c0640 1 --2- 172.21.15.16:0/2318439548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 0x56412be69d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde416c0640 1 -- 172.21.15.16:0/2318439548 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56412bb2f650 con 0x56412ba9b1b0 2024-08-28T19:26:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde41ec1640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde280b4d20 0x7fde280b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde416c0640 1 --2- 172.21.15.16:0/2318439548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412ba9b1b0 0x56412be69850 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde380114c0 tx=0x7fde38011980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b360d45fe6ad48f0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde327fc640 1 -- 172.21.15.16:0/2810016588 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fde34106960 con 0x7fde3c10dbb0 2024-08-28T19:26:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde277fe640 1 -- 172.21.15.16:0/2318439548 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38004020 con 0x56412ba9b1b0 2024-08-28T19:26:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde277fe640 1 -- 172.21.15.16:0/2318439548 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde380054d0 con 0x56412ba9b1b0 2024-08-28T19:26:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.370+0000 7fde277fe640 1 -- 172.21.15.16:0/2318439548 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38004c80 con 0x56412ba9b1b0 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.371+0000 7fde49e99740 1 -- 172.21.15.16:0/2318439548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412ba9b1b0 msgr2=0x56412be69850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.371+0000 7fde49e99740 1 --2- 172.21.15.16:0/2318439548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412ba9b1b0 0x56412be69850 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde380114c0 tx=0x7fde38011980 comp rx=0 tx=0).stop 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.371+0000 7fde41ec1640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde280b4d20 0x7fde280b7230 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fde340cc9e0 tx=0x7fde34039000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.371+0000 7fde49e99740 1 -- 172.21.15.16:0/2318439548 shutdown_connections 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.371+0000 7fde49e99740 1 --2- 172.21.15.16:0/2318439548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 0x56412be69d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.371+0000 7fde49e99740 1 --2- 172.21.15.16:0/2318439548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412ba9b1b0 0x56412be69850 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.371+0000 7fde49e99740 1 --2- 172.21.15.16:0/2318439548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412bac4070 0x56412be69310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.371+0000 7fde49e99740 1 -- 172.21.15.16:0/2318439548 >> 172.21.15.16:0/2318439548 conn(0x56412bab8460 msgr2=0x56412be67200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.371+0000 7fde49e99740 1 -- 172.21.15.16:0/2318439548 shutdown_connections 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.371+0000 7fde49e99740 1 -- 172.21.15.16:0/2318439548 wait complete. 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde49e99740 1 -- start start 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde49e99740 1 Processor -- start 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde49e99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 0x56412be749f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde49e99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 0x56412be74f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde49e99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412bac4070 0x56412be80e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde49e99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56412bc32700 con 0x56412baa3390 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde49e99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56412bd56750 con 0x56412ba9b1b0 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde49e99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56412be77af0 con 0x56412bac4070 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde49e99740 10 client.0 _ll_register_callbacks cb 0x56412bd85e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde416c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 0x56412be74f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde40ebf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 0x56412be749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde416c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 0x56412be74f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45800/0 (socket says 172.21.15.16:45800) 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.372+0000 7fde416c0640 1 -- 172.21.15.16:0/2799883798 learned_addr learned my addr 172.21.15.16:0/2799883798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde40ebf640 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412bac4070 0x56412be80e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde40ebf640 1 -- 172.21.15.16:0/2799883798 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412bac4070 msgr2=0x56412be80e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde40ebf640 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412bac4070 0x56412be80e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde40ebf640 1 -- 172.21.15.16:0/2799883798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 msgr2=0x56412be74f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde40ebf640 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 0x56412be74f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde40ebf640 1 -- 172.21.15.16:0/2799883798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56412bd8f900 con 0x56412ba9b1b0 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde416c0640 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 0x56412be74f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde40ebf640 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 0x56412be749f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009710 tx=0x7fde2c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9014042263e938ce server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde24ff9640 1 -- 172.21.15.16:0/2799883798 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c00fb60 con 0x56412ba9b1b0 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde49e99740 1 -- 172.21.15.16:0/2799883798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde380054d0 con 0x56412ba9b1b0 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde24ff9640 1 -- 172.21.15.16:0/2799883798 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fde2c00fd00 con 0x56412ba9b1b0 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde24ff9640 1 -- 172.21.15.16:0/2799883798 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fde2c02a050 con 0x56412ba9b1b0 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde49e99740 10 client.26477 fetch_fsmap learned FSMap version 40 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.373+0000 7fde49e99740 1 -- 172.21.15.16:0/2799883798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56412be83910 con 0x56412ba9b1b0 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.374+0000 7fde24ff9640 1 -- 172.21.15.16:0/2799883798 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde380054d0 con 0x56412ba9b1b0 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.374+0000 7fde24ff9640 1 -- 172.21.15.16:0/2799883798 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fde2c01d070 con 0x56412ba9b1b0 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.374+0000 7fde49e99740 10 client.26477 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.374+0000 7fde49e99740 10 client.26477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.374+0000 7fde49e99740 20 client.26477 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.374+0000 7fde49e99740 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56412ba829f0 0x56412be86d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.374+0000 7fde49e99740 4 client.26477 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:03.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.374+0000 7fde49e99740 1 -- 172.21.15.16:0/2799883798 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56412bde25d0 con 0x56412ba829f0 2024-08-28T19:26:03.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.374+0000 7fde416c0640 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56412ba829f0 0x56412be86d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.375+0000 7fde24ff9640 10 client.26477 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:03.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.375+0000 7fde416c0640 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56412ba829f0 0x56412be86d30 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa21dfb4034681bc server_cookie=96a4e20aeb575e3f in_seq=0 out_seq=0 2024-08-28T19:26:03.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.375+0000 7fde24ff9640 1 -- 172.21.15.16:0/2799883798 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56412bde25d0 con 0x56412ba829f0 2024-08-28T19:26:03.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.375+0000 7fde24ff9640 10 client.26477 handle_command_reply: tid=0 2024-08-28T19:26:03.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.375+0000 7fde49e99740 1 client.26477 shutdown 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde49e99740 1 -- 172.21.15.16:0/2799883798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 msgr2=0x56412be749f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde49e99740 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 0x56412be749f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009710 tx=0x7fde2c004730 comp rx=0 tx=0).stop 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde49e99740 1 -- 172.21.15.16:0/2799883798 shutdown_connections 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde49e99740 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56412ba829f0 0x56412be86d30 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde49e99740 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412bac4070 0x56412be80e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde49e99740 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412baa3390 0x56412be74f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde24ff9640 0 client.26477 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde49e99740 1 --2- 172.21.15.16:0/2799883798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 0x56412be749f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde49e99740 1 -- 172.21.15.16:0/2799883798 >> 172.21.15.16:0/2799883798 conn(0x56412bab8460 msgr2=0x56412be6c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde49e99740 1 -- 172.21.15.16:0/2799883798 shutdown_connections 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde49e99740 1 -- 172.21.15.16:0/2799883798 wait complete. 2024-08-28T19:26:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.376+0000 7fde49e99740 20 client.26477 trim_cache size 0 max 16384 2024-08-28T19:26:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.390+0000 7fde49e99740 1 Processor -- start 2024-08-28T19:26:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.390+0000 7fde49e99740 1 -- start start 2024-08-28T19:26:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.390+0000 7fde49e99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412ba9b1b0 0x56412be76620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.390+0000 7fde49e99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412baa3390 0x56412be667d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.390+0000 7fde49e99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412bac4070 0x56412be66d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.390+0000 7fde49e99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1b0ca0 con 0x56412ba9b1b0 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.390+0000 7fde49e99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1b28a0 con 0x56412baa3390 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.390+0000 7fde49e99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1b13d0 con 0x56412bac4070 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.390+0000 7fde40ebf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412ba9b1b0 0x56412be76620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.390+0000 7fde416c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412baa3390 0x56412be667d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde416c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412baa3390 0x56412be667d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48318/0 (socket says 172.21.15.16:48318) 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde40ebf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412ba9b1b0 0x56412be76620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45810/0 (socket says 172.21.15.16:45810) 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde416c0640 1 -- 172.21.15.16:0/2738283122 learned_addr learned my addr 172.21.15.16:0/2738283122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde40ebf640 1 --2- 172.21.15.16:0/2738283122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412bac4070 0x56412be66d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde416c0640 1 -- 172.21.15.16:0/2738283122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412bac4070 msgr2=0x56412be66d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde416c0640 1 --2- 172.21.15.16:0/2738283122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412bac4070 0x56412be66d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde416c0640 1 -- 172.21.15.16:0/2738283122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412ba9b1b0 msgr2=0x56412be76620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde416c0640 1 --2- 172.21.15.16:0/2738283122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412ba9b1b0 0x56412be76620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde416c0640 1 -- 172.21.15.16:0/2738283122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56412be6b600 con 0x56412baa3390 2024-08-28T19:26:03.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde416c0640 1 --2- 172.21.15.16:0/2738283122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412baa3390 0x56412be667d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fde38004810 tx=0x7fde38005870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c689fd0c0063a59c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:03.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde26ffd640 1 -- 172.21.15.16:0/2738283122 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3801d040 con 0x56412baa3390 2024-08-28T19:26:03.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde26ffd640 1 -- 172.21.15.16:0/2738283122 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde3800fd80 con 0x56412baa3390 2024-08-28T19:26:03.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.391+0000 7fde26ffd640 1 -- 172.21.15.16:0/2738283122 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38026dc0 con 0x56412baa3390 2024-08-28T19:26:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.392+0000 7fde49e99740 1 -- 172.21.15.16:0/2738283122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412baa3390 msgr2=0x56412be667d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.392+0000 7fde49e99740 1 --2- 172.21.15.16:0/2738283122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412baa3390 0x56412be667d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fde38004810 tx=0x7fde38005870 comp rx=0 tx=0).stop 2024-08-28T19:26:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.392+0000 7fde49e99740 1 -- 172.21.15.16:0/2738283122 shutdown_connections 2024-08-28T19:26:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.392+0000 7fde49e99740 1 --2- 172.21.15.16:0/2738283122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412bac4070 0x56412be66d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.392+0000 7fde49e99740 1 --2- 172.21.15.16:0/2738283122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412baa3390 0x56412be667d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.392+0000 7fde49e99740 1 --2- 172.21.15.16:0/2738283122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412ba9b1b0 0x56412be76620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.392+0000 7fde49e99740 1 -- 172.21.15.16:0/2738283122 >> 172.21.15.16:0/2738283122 conn(0x56412bab8460 msgr2=0x56412be75f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.392+0000 7fde49e99740 1 -- 172.21.15.16:0/2738283122 shutdown_connections 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.392+0000 7fde49e99740 1 -- 172.21.15.16:0/2738283122 wait complete. 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.392+0000 7fde49e99740 1 -- start start 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde49e99740 1 Processor -- start 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde49e99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 0x56412be791b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde49e99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412baa3390 0x56412be79790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde49e99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412bac4070 0x56412bef06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde49e99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1b21d0 con 0x56412bac4070 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde49e99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1b1ad0 con 0x56412ba9b1b0 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde49e99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1aec80 con 0x56412baa3390 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde49e99740 10 client.0 _ll_register_callbacks cb 0x7fde2c009ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde416c0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412baa3390 0x56412be79790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde40ebf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 0x56412be791b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde416c0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412baa3390 0x56412be79790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55814/0 (socket says 172.21.15.16:55814) 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde416c0640 1 -- 172.21.15.16:0/2847459490 learned_addr learned my addr 172.21.15.16:0/2847459490 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde40ebf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 0x56412be791b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48326/0 (socket says 172.21.15.16:48326) 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.393+0000 7fde40ebf640 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412bac4070 0x56412bef06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde416c0640 1 -- 172.21.15.16:0/2847459490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 msgr2=0x56412be791b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde416c0640 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 0x56412be791b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde416c0640 1 -- 172.21.15.16:0/2847459490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412bac4070 msgr2=0x56412bef06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde416c0640 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412bac4070 0x56412bef06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde416c0640 1 -- 172.21.15.16:0/2847459490 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56412bef0c10 con 0x56412baa3390 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde416c0640 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412baa3390 0x56412be79790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fde38009380 tx=0x7fde38004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=532d0072923e1483 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:26:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde24ff9640 1 -- 172.21.15.16:0/2847459490 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3801d030 con 0x56412baa3390 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde49e99740 1 -- 172.21.15.16:0/2847459490 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde3800fd80 con 0x56412baa3390 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde24ff9640 1 -- 172.21.15.16:0/2847459490 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fde38026720 con 0x56412baa3390 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde24ff9640 1 -- 172.21.15.16:0/2847459490 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fde38042050 con 0x56412baa3390 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde49e99740 10 client.26308 fetch_fsmap learned FSMap version 40 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde49e99740 1 -- 172.21.15.16:0/2847459490 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56412bef1dc0 con 0x56412baa3390 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.394+0000 7fde24ff9640 1 -- 172.21.15.16:0/2847459490 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde3800fd80 con 0x56412baa3390 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde24ff9640 1 -- 172.21.15.16:0/2847459490 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fde3802e070 con 0x56412baa3390 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde49e99740 10 client.26308 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde49e99740 10 client.26308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde49e99740 20 client.26308 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde49e99740 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56412ba57da0 0x56412bef4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde49e99740 4 client.26308 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde49e99740 1 -- 172.21.15.16:0/2847459490 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56412bdde070 con 0x56412ba57da0 2024-08-28T19:26:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde40ebf640 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56412ba57da0 0x56412bef4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde24ff9640 10 client.26308 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde40ebf640 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56412ba57da0 0x56412bef4300 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20dc720e4e16c70d server_cookie=b3d6b0f8d8a158a3 in_seq=0 out_seq=0 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde24ff9640 1 -- 172.21.15.16:0/2847459490 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56412bdde070 con 0x56412ba57da0 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde24ff9640 10 client.26308 handle_command_reply: tid=0 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.395+0000 7fde49e99740 1 client.26308 shutdown 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde49e99740 1 -- 172.21.15.16:0/2847459490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412baa3390 msgr2=0x56412be79790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde49e99740 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412baa3390 0x56412be79790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fde38009380 tx=0x7fde38004530 comp rx=0 tx=0).stop 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde49e99740 1 -- 172.21.15.16:0/2847459490 shutdown_connections 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde49e99740 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56412bac4070 0x56412bef06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde49e99740 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56412ba57da0 0x56412bef4300 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde49e99740 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56412baa3390 0x56412be79790 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde49e99740 1 --2- 172.21.15.16:0/2847459490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412ba9b1b0 0x56412be791b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde49e99740 1 -- 172.21.15.16:0/2847459490 >> 172.21.15.16:0/2847459490 conn(0x56412bab8460 msgr2=0x56412be75cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde24ff9640 0 client.26308 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde49e99740 1 -- 172.21.15.16:0/2847459490 shutdown_connections 2024-08-28T19:26:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde49e99740 1 -- 172.21.15.16:0/2847459490 wait complete. 2024-08-28T19:26:03.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.396+0000 7fde49e99740 20 client.26308 trim_cache size 0 max 16384 2024-08-28T19:26:03.403 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 -- 172.21.15.16:0/2810016588 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde280b4d20 msgr2=0x7fde280b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde280b4d20 0x7fde280b7230 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fde340cc9e0 tx=0x7fde34039000 comp rx=0 tx=0).stop 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 -- 172.21.15.16:0/2810016588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c10dbb0 msgr2=0x7fde3c19fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c10dbb0 0x7fde3c19fb80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fde34005cc0 tx=0x7fde34031700 comp rx=0 tx=0).stop 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 -- 172.21.15.16:0/2810016588 shutdown_connections 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c10dbb0 0x7fde3c19fb80 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde280b4d20 0x7fde280b7230 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c1050a0 0x7fde3c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 --2- 172.21.15.16:0/2810016588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde3c104750 0x7fde3c19a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 -- 172.21.15.16:0/2810016588 >> 172.21.15.16:0/2810016588 conn(0x7fde3c0794d0 msgr2=0x7fde3c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 -- 172.21.15.16:0/2810016588 shutdown_connections 2024-08-28T19:26:03.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:03.397+0000 7fde43924640 1 -- 172.21.15.16:0/2810016588 wait complete. 2024-08-28T19:26:03.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 265524 6.87 MB/sec execute 103 sec latency 39.926 ms 2024-08-28T19:26:04.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266994 6.87 MB/sec execute 104 sec latency 15.740 ms 2024-08-28T19:26:05.181 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-08-28T19:26:05.182 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-08-28T19:26:05.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:26:05.232 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:26:05.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:26:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:05 smithi028 ceph-mon[21769]: pgmap v490: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 274 op/s 2024-08-28T19:26:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:05 smithi115 ceph-mon[22184]: pgmap v490: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 274 op/s 2024-08-28T19:26:05.477 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:05 smithi016 ceph-mon[18556]: pgmap v490: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 274 op/s 2024-08-28T19:26:05.752 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:05.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 269255 6.93 MB/sec execute 105 sec latency 39.201 ms 2024-08-28T19:26:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.988+0000 7f01b1103640 1 -- 172.21.15.16:0/4030440759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac108ac0 msgr2=0x7f01ac10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.988+0000 7f01b1103640 1 --2- 172.21.15.16:0/4030440759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac108ac0 0x7f01ac10f2f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f01a4009ca0 tx=0x7f01a402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:26:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.988+0000 7f01b1103640 1 -- 172.21.15.16:0/4030440759 shutdown_connections 2024-08-28T19:26:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.988+0000 7f01b1103640 1 --2- 172.21.15.16:0/4030440759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac108ac0 0x7f01ac10f2f0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.988+0000 7f01b1103640 1 --2- 172.21.15.16:0/4030440759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac108170 0x7f01ac108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.988+0000 7f01b1103640 1 --2- 172.21.15.16:0/4030440759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ac102880 0x7f01ac102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.988+0000 7f01b1103640 1 -- 172.21.15.16:0/4030440759 >> 172.21.15.16:0/4030440759 conn(0x7f01ac079550 msgr2=0x7f01ac079960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.988+0000 7f01b1103640 1 -- 172.21.15.16:0/4030440759 shutdown_connections 2024-08-28T19:26:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.988+0000 7f01b1103640 1 -- 172.21.15.16:0/4030440759 wait complete. 2024-08-28T19:26:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.989+0000 7f01b1103640 1 Processor -- start 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.989+0000 7f01b1103640 1 -- start start 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.989+0000 7f01b1103640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac102880 0x7f01ac0ff410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.989+0000 7f01b1103640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ac108170 0x7f01ac0ff950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.989+0000 7f01b1103640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac108ac0 0x7f01ac0ffe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.989+0000 7f01b1103640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac112790 con 0x7f01ac102880 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.989+0000 7f01b1103640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac112610 con 0x7f01ac108ac0 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.989+0000 7f01b1103640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac112910 con 0x7f01ac108170 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.989+0000 7f01aad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac102880 0x7f01ac0ff410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.989+0000 7f01aa59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ac108170 0x7f01ac0ff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01aad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac102880 0x7f01ac0ff410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38406/0 (socket says 172.21.15.16:38406) 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01aa59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ac108170 0x7f01ac0ff950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56328/0 (socket says 172.21.15.16:56328) 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01ab59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac108ac0 0x7f01ac0ffe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01aad9d640 1 -- 172.21.15.16:0/2153823048 learned_addr learned my addr 172.21.15.16:0/2153823048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01aa59c640 1 -- 172.21.15.16:0/2153823048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac108ac0 msgr2=0x7f01ac0ffe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01aa59c640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac108ac0 0x7f01ac0ffe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01aa59c640 1 -- 172.21.15.16:0/2153823048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac102880 msgr2=0x7f01ac0ff410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01aa59c640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac102880 0x7f01ac0ff410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01aa59c640 1 -- 172.21.15.16:0/2153823048 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ac1a84f0 con 0x7f01ac108170 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01aad9d640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac102880 0x7f01ac0ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01aa59c640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ac108170 0x7f01ac0ff950 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f01ac06aad0 tx=0x7f01a000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f0193fff640 1 -- 172.21.15.16:0/2153823048 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a000cd00 con 0x7f01ac108170 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01b1103640 1 -- 172.21.15.16:0/2153823048 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01ac1a87e0 con 0x7f01ac108170 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f0193fff640 1 -- 172.21.15.16:0/2153823048 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01a00035f0 con 0x7f01ac108170 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f0193fff640 1 -- 172.21.15.16:0/2153823048 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a0010690 con 0x7f01ac108170 2024-08-28T19:26:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.990+0000 7f01b1103640 1 -- 172.21.15.16:0/2153823048 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01ac1a8d70 con 0x7f01ac108170 2024-08-28T19:26:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.992+0000 7f01b1103640 1 -- 172.21.15.16:0/2153823048 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01ac102c90 con 0x7f01ac108170 2024-08-28T19:26:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.993+0000 7f0193fff640 1 -- 172.21.15.16:0/2153823048 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f01a00037a0 con 0x7f01ac108170 2024-08-28T19:26:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.994+0000 7f0193fff640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01940b4ca0 0x7f01940b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.994+0000 7f01aad9d640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01940b4ca0 0x7f01940b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.994+0000 7f0193fff640 1 -- 172.21.15.16:0/2153823048 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f01a00e5b80 con 0x7f01ac108170 2024-08-28T19:26:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.995+0000 7f01aad9d640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01940b4ca0 0x7f01940b71b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f01ac100e60 tx=0x7f019c005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:05.998+0000 7f0193fff640 1 -- 172.21.15.16:0/2153823048 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f01a00ac5a0 con 0x7f01ac108170 2024-08-28T19:26:06.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.227+0000 7f01b1103640 1 -- 172.21.15.16:0/2153823048 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01ac100710 con 0x7f01ac108170 2024-08-28T19:26:06.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.228+0000 7f0193fff640 1 -- 172.21.15.16:0/2153823048 <== mon.1 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11982 (secure 0 0 0) 0x7f01ac100710 con 0x7f01ac108170 2024-08-28T19:26:06.229 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:26:06.235 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":40,"btime":"2024-08-28T19:25:57:024583+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:25:56.959375+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:26:06.235 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 40 2024-08-28T19:26:06.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 -- 172.21.15.16:0/2153823048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01940b4ca0 msgr2=0x7f01940b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:06.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01940b4ca0 0x7f01940b71b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f01ac100e60 tx=0x7f019c005f90 comp rx=0 tx=0).stop 2024-08-28T19:26:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 -- 172.21.15.16:0/2153823048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ac108170 msgr2=0x7f01ac0ff950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ac108170 0x7f01ac0ff950 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f01ac06aad0 tx=0x7f01a000ee50 comp rx=0 tx=0).stop 2024-08-28T19:26:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 -- 172.21.15.16:0/2153823048 shutdown_connections 2024-08-28T19:26:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f01940b4ca0 0x7f01940b71b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac108ac0 0x7f01ac0ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01ac108170 0x7f01ac0ff950 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 --2- 172.21.15.16:0/2153823048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac102880 0x7f01ac0ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 -- 172.21.15.16:0/2153823048 >> 172.21.15.16:0/2153823048 conn(0x7f01ac079550 msgr2=0x7f01ac101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 -- 172.21.15.16:0/2153823048 shutdown_connections 2024-08-28T19:26:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.230+0000 7f01b1103640 1 -- 172.21.15.16:0/2153823048 wait complete. 2024-08-28T19:26:06.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270029 6.92 MB/sec execute 106 sec latency 396.533 ms 2024-08-28T19:26:07.163 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-08-28T19:26:07.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-08-28T19:26:07.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.449+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/48338522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb64102800 msgr2=0x7fdb64102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.449+0000 7fdb6a9ed640 1 --2- 172.21.15.16:0/48338522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb64102800 0x7fdb64102c10 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009900 tx=0x7fdb5402a410 comp rx=0 tx=0).stop 2024-08-28T19:26:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.449+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/48338522 shutdown_connections 2024-08-28T19:26:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.449+0000 7fdb6a9ed640 1 --2- 172.21.15.16:0/48338522 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb64108a90 0x7fdb6410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.449+0000 7fdb6a9ed640 1 --2- 172.21.15.16:0/48338522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64108140 0x7fdb64108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.449+0000 7fdb6a9ed640 1 --2- 172.21.15.16:0/48338522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb64102800 0x7fdb64102c10 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.449+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/48338522 >> 172.21.15.16:0/48338522 conn(0x7fdb640794d0 msgr2=0x7fdb640798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.449+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/48338522 shutdown_connections 2024-08-28T19:26:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.449+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/48338522 wait complete. 2024-08-28T19:26:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.450+0000 7fdb6a9ed640 1 Processor -- start 2024-08-28T19:26:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.450+0000 7fdb6a9ed640 1 -- start start 2024-08-28T19:26:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb6a9ed640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb64102800 0x7fdb6419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb6a9ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb64108140 0x7fdb6419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb6a9ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64108a90 0x7fdb6419f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb6a9ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb64113820 con 0x7fdb64108140 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb6a9ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641136a0 con 0x7fdb64108a90 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb6a9ed640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641139a0 con 0x7fdb64102800 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb64108140 0x7fdb6419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb64108140 0x7fdb6419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38424/0 (socket says 172.21.15.16:38424) 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb64102800 0x7fdb6419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb637fe640 1 -- 172.21.15.16:0/3580655218 learned_addr learned my addr 172.21.15.16:0/3580655218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb68f8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64108a90 0x7fdb6419f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb637fe640 1 -- 172.21.15.16:0/3580655218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb64102800 msgr2=0x7fdb6419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:07.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb637fe640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb64102800 0x7fdb6419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb637fe640 1 -- 172.21.15.16:0/3580655218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64108a90 msgr2=0x7fdb6419f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb637fe640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64108a90 0x7fdb6419f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb637fe640 1 -- 172.21.15.16:0/3580655218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb641a0120 con 0x7fdb64108140 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb68f8a640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64108a90 0x7fdb6419f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.451+0000 7fdb637fe640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb64108140 0x7fdb6419a750 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fdb5800e9c0 tx=0x7fdb5800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.452+0000 7fdb617fa640 1 -- 172.21.15.16:0/3580655218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5800cd80 con 0x7fdb64108140 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.452+0000 7fdb617fa640 1 -- 172.21.15.16:0/3580655218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb580054d0 con 0x7fdb64108140 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.452+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/3580655218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb641a0410 con 0x7fdb64108140 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.452+0000 7fdb617fa640 1 -- 172.21.15.16:0/3580655218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb58010430 con 0x7fdb64108140 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.452+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/3580655218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb641a6b70 con 0x7fdb64108140 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.454+0000 7fdb70f62740 1 Processor -- start 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.454+0000 7fdb70f62740 1 -- start start 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb617fa640 1 -- 172.21.15.16:0/3580655218 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fdb58004fd0 con 0x7fdb64108140 2024-08-28T19:26:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebb9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 0x55f30ebb9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7f2670 0x55f30ebba200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb70f62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f30eb2f5f0 con 0x55f30e813350 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb70f62740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f30eb1eb80 con 0x55f30e7f2670 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb70f62740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f30eb22b30 con 0x55f30e7ea490 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 0x55f30ebb9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 0x55f30ebb9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56362/0 (socket says 172.21.15.16:56362) 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb63fff640 1 -- 172.21.15.16:0/1306379268 learned_addr learned my addr 172.21.15.16:0/1306379268 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb637fe640 1 --2- 172.21.15.16:0/1306379268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7f2670 0x55f30ebba200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb68f8a640 1 --2- 172.21.15.16:0/1306379268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebb9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb63fff640 1 -- 172.21.15.16:0/1306379268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7f2670 msgr2=0x55f30ebba200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb63fff640 1 --2- 172.21.15.16:0/1306379268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7f2670 0x55f30ebba200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb63fff640 1 -- 172.21.15.16:0/1306379268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 msgr2=0x55f30ebb9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb63fff640 1 --2- 172.21.15.16:0/1306379268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebb9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.455+0000 7fdb63fff640 1 -- 172.21.15.16:0/1306379268 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f30e87e930 con 0x55f30e7ea490 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.456+0000 7fdb617fa640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb4c0b4d20 0x7fdb4c0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.456+0000 7fdb63fff640 1 --2- 172.21.15.16:0/1306379268 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 0x55f30ebb9cc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdb54033000 tx=0x7fdb54032b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ac3f85b44bd5c87 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.456+0000 7fdb68f8a640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb4c0b4d20 0x7fdb4c0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.456+0000 7fdb617fa640 1 -- 172.21.15.16:0/3580655218 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdb580e5c40 con 0x7fdb64108140 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.456+0000 7fdb4a7fc640 1 -- 172.21.15.16:0/1306379268 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54035040 con 0x55f30e7ea490 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.456+0000 7fdb4a7fc640 1 -- 172.21.15.16:0/1306379268 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb54004030 con 0x55f30e7ea490 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.456+0000 7fdb4a7fc640 1 -- 172.21.15.16:0/1306379268 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54005580 con 0x55f30e7ea490 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.457+0000 7fdb70f62740 1 -- 172.21.15.16:0/1306379268 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 msgr2=0x55f30ebb9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.457+0000 7fdb68f8a640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb4c0b4d20 0x7fdb4c0b7230 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c009fd0 tx=0x7fdb5c009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.457+0000 7fdb70f62740 1 --2- 172.21.15.16:0/1306379268 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 0x55f30ebb9cc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdb54033000 tx=0x7fdb54032b90 comp rx=0 tx=0).stop 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.457+0000 7fdb70f62740 1 -- 172.21.15.16:0/1306379268 shutdown_connections 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.457+0000 7fdb70f62740 1 --2- 172.21.15.16:0/1306379268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7f2670 0x55f30ebba200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.457+0000 7fdb70f62740 1 --2- 172.21.15.16:0/1306379268 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 0x55f30ebb9cc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.457+0000 7fdb70f62740 1 --2- 172.21.15.16:0/1306379268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebb9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.457+0000 7fdb70f62740 1 -- 172.21.15.16:0/1306379268 >> 172.21.15.16:0/1306379268 conn(0x55f30e807740 msgr2=0x55f30ebb7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.457+0000 7fdb70f62740 1 -- 172.21.15.16:0/1306379268 shutdown_connections 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.457+0000 7fdb70f62740 1 -- 172.21.15.16:0/1306379268 wait complete. 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb70f62740 1 -- start start 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb70f62740 1 Processor -- start 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 0x55f30ebc9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7f2670 0x55f30ebc9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebd5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb70f62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f30e981890 con 0x55f30e813350 2024-08-28T19:26:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb70f62740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f30eaa5790 con 0x55f30e7ea490 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb70f62740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f30ebcc230 con 0x55f30e7f2670 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb70f62740 10 client.0 _ll_register_callbacks cb 0x55f30e82aba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb68f8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebd5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 0x55f30ebc9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb637fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7f2670 0x55f30ebc9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb68f8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebd5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38440/0 (socket says 172.21.15.16:38440) 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.458+0000 7fdb68f8a640 1 -- 172.21.15.16:0/322119203 learned_addr learned my addr 172.21.15.16:0/322119203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb63fff640 1 -- 172.21.15.16:0/322119203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7f2670 msgr2=0x55f30ebc9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb63fff640 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7f2670 0x55f30ebc9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb63fff640 1 -- 172.21.15.16:0/322119203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 msgr2=0x55f30ebd5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb63fff640 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebd5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb63fff640 1 -- 172.21.15.16:0/322119203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f30eadcf30 con 0x55f30e7ea490 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb68f8a640 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebd5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb63fff640 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 0x55f30ebc9300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb5402a930 tx=0x7fdb54005160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7a1d13302d6ef01 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb437fe640 1 -- 172.21.15.16:0/322119203 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54035070 con 0x55f30e7ea490 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb70f62740 1 -- 172.21.15.16:0/322119203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb54004030 con 0x55f30e7ea490 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb437fe640 1 -- 172.21.15.16:0/322119203 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdb54005590 con 0x55f30e7ea490 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb437fe640 1 -- 172.21.15.16:0/322119203 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdb540041e0 con 0x55f30e7ea490 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb70f62740 10 client.26501 fetch_fsmap learned FSMap version 40 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.459+0000 7fdb70f62740 1 -- 172.21.15.16:0/322119203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f30ebd7fe0 con 0x55f30e7ea490 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.460+0000 7fdb437fe640 1 -- 172.21.15.16:0/322119203 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb54004030 con 0x55f30e7ea490 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.460+0000 7fdb437fe640 1 -- 172.21.15.16:0/322119203 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fdb54004b10 con 0x55f30e7ea490 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.460+0000 7fdb70f62740 10 client.26501 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.460+0000 7fdb70f62740 10 client.26501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.460+0000 7fdb70f62740 20 client.26501 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.460+0000 7fdb70f62740 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f30e7d1cd0 0x55f30ebdb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.460+0000 7fdb70f62740 4 client.26501 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.460+0000 7fdb70f62740 1 -- 172.21.15.16:0/322119203 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f30eb31540 con 0x55f30e7d1cd0 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.460+0000 7fdb637fe640 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f30e7d1cd0 0x55f30ebdb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.460+0000 7fdb437fe640 10 client.26501 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.460+0000 7fdb637fe640 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f30e7d1cd0 0x55f30ebdb340 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c75fa3d6927f7c80 server_cookie=1ca416fd3f32bccf in_seq=0 out_seq=0 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb437fe640 1 -- 172.21.15.16:0/322119203 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f30eb31540 con 0x55f30e7d1cd0 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb437fe640 10 client.26501 handle_command_reply: tid=0 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb70f62740 1 client.26501 shutdown 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb70f62740 1 -- 172.21.15.16:0/322119203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 msgr2=0x55f30ebc9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb70f62740 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 0x55f30ebc9300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb5402a930 tx=0x7fdb54005160 comp rx=0 tx=0).stop 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb70f62740 1 -- 172.21.15.16:0/322119203 shutdown_connections 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb70f62740 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebd5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb70f62740 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f30e7d1cd0 0x55f30ebdb340 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb70f62740 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7f2670 0x55f30ebc9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb70f62740 1 --2- 172.21.15.16:0/322119203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 0x55f30ebc9300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb437fe640 0 client.26501 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb70f62740 1 -- 172.21.15.16:0/322119203 >> 172.21.15.16:0/322119203 conn(0x55f30e807740 msgr2=0x55f30ebbc910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb70f62740 1 -- 172.21.15.16:0/322119203 shutdown_connections 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.461+0000 7fdb70f62740 1 -- 172.21.15.16:0/322119203 wait complete. 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.462+0000 7fdb70f62740 20 client.26501 trim_cache size 0 max 16384 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.480+0000 7fdb70f62740 1 Processor -- start 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.480+0000 7fdb70f62740 1 -- start start 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.480+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 0x55f30ebbae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.480+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7f2670 0x55f30ebb6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.480+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebb71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.480+0000 7fdb70f62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641ae970 con 0x55f30e813350 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.480+0000 7fdb70f62740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641b0570 con 0x55f30e7f2670 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.480+0000 7fdb70f62740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641af0a0 con 0x55f30e7ea490 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb68f8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebb71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 0x55f30ebbae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb68f8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebb71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38452/0 (socket says 172.21.15.16:38452) 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb68f8a640 1 -- 172.21.15.16:0/3726284442 learned_addr learned my addr 172.21.15.16:0/3726284442 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb637fe640 1 --2- 172.21.15.16:0/3726284442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7f2670 0x55f30ebb6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb68f8a640 1 -- 172.21.15.16:0/3726284442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 msgr2=0x55f30ebbae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb68f8a640 1 --2- 172.21.15.16:0/3726284442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 0x55f30ebbae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb68f8a640 1 -- 172.21.15.16:0/3726284442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7f2670 msgr2=0x55f30ebb6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb68f8a640 1 --2- 172.21.15.16:0/3726284442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7f2670 0x55f30ebb6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb68f8a640 1 -- 172.21.15.16:0/3726284442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f30ebb1920 con 0x55f30e813350 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb68f8a640 1 --2- 172.21.15.16:0/3726284442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebb71e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c003e40 tx=0x7fdb5c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a4030da6f97035 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.481+0000 7fdb49ffb640 1 -- 172.21.15.16:0/3726284442 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c00fda0 con 0x55f30e813350 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.482+0000 7fdb49ffb640 1 -- 172.21.15.16:0/3726284442 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb5c004b20 con 0x55f30e813350 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.482+0000 7fdb49ffb640 1 -- 172.21.15.16:0/3726284442 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c0056c0 con 0x55f30e813350 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.482+0000 7fdb70f62740 1 -- 172.21.15.16:0/3726284442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 msgr2=0x55f30ebb71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.482+0000 7fdb70f62740 1 --2- 172.21.15.16:0/3726284442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebb71e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c003e40 tx=0x7fdb5c012000 comp rx=0 tx=0).stop 2024-08-28T19:26:07.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.482+0000 7fdb70f62740 1 -- 172.21.15.16:0/3726284442 shutdown_connections 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.482+0000 7fdb70f62740 1 --2- 172.21.15.16:0/3726284442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e813350 0x55f30ebb71e0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.482+0000 7fdb70f62740 1 --2- 172.21.15.16:0/3726284442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7f2670 0x55f30ebb6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.482+0000 7fdb70f62740 1 --2- 172.21.15.16:0/3726284442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e7ea490 0x55f30ebbae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.482+0000 7fdb70f62740 1 -- 172.21.15.16:0/3726284442 >> 172.21.15.16:0/3726284442 conn(0x55f30e807740 msgr2=0x55f30ebbd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.482+0000 7fdb70f62740 1 -- 172.21.15.16:0/3726284442 shutdown_connections 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.482+0000 7fdb70f62740 1 -- 172.21.15.16:0/3726284442 wait complete. 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.483+0000 7fdb70f62740 1 -- start start 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.483+0000 7fdb70f62740 1 Processor -- start 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.483+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 0x55f30ebdca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.483+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e7f2670 0x55f30ebdcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.483+0000 7fdb70f62740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e813350 0x55f30ec44ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.483+0000 7fdb70f62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641afea0 con 0x55f30e7f2670 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.483+0000 7fdb70f62740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641af7a0 con 0x55f30e7ea490 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.483+0000 7fdb70f62740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641ac950 con 0x55f30e813350 2024-08-28T19:26:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.483+0000 7fdb70f62740 10 client.0 _ll_register_callbacks cb 0x7fdb5c009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 0x55f30ebdca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 0x55f30ebdca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34120/0 (socket says 172.21.15.16:34120) 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e7f2670 0x55f30ebdcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb63fff640 1 -- 172.21.15.16:0/2407186658 learned_addr learned my addr 172.21.15.16:0/2407186658 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb68f8a640 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e813350 0x55f30ec44ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb63fff640 1 -- 172.21.15.16:0/2407186658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e813350 msgr2=0x55f30ec44ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb63fff640 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e813350 0x55f30ec44ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb63fff640 1 -- 172.21.15.16:0/2407186658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e7f2670 msgr2=0x55f30ebdcfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb63fff640 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e7f2670 0x55f30ebdcfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb63fff640 1 -- 172.21.15.16:0/2407186658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f30ec45000 con 0x55f30e7ea490 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb63fff640 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 0x55f30ebdca90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdb54004eb0 tx=0x7fdb54032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24870922a8b5f43d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb437fe640 1 -- 172.21.15.16:0/2407186658 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54035070 con 0x55f30e7ea490 2024-08-28T19:26:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb70f62740 1 -- 172.21.15.16:0/2407186658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb5c004b20 con 0x55f30e7ea490 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb437fe640 1 -- 172.21.15.16:0/2407186658 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdb54005590 con 0x55f30e7ea490 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb437fe640 1 -- 172.21.15.16:0/2407186658 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdb54067050 con 0x55f30e7ea490 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb70f62740 10 client.26507 fetch_fsmap learned FSMap version 40 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.484+0000 7fdb70f62740 1 -- 172.21.15.16:0/2407186658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f30ec462a0 con 0x55f30e7ea490 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.485+0000 7fdb437fe640 1 -- 172.21.15.16:0/2407186658 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb5c004b20 con 0x55f30e7ea490 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.485+0000 7fdb437fe640 1 -- 172.21.15.16:0/2407186658 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fdb54003d60 con 0x55f30e7ea490 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.485+0000 7fdb70f62740 10 client.26507 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.485+0000 7fdb70f62740 10 client.26507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.485+0000 7fdb70f62740 20 client.26507 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.485+0000 7fdb70f62740 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f30e7a7080 0x55f30ec48900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.485+0000 7fdb70f62740 4 client.26507 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.485+0000 7fdb70f62740 1 -- 172.21.15.16:0/2407186658 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f30eb2cfe0 con 0x55f30e7a7080 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.485+0000 7fdb637fe640 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f30e7a7080 0x55f30ec48900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.486+0000 7fdb437fe640 10 client.26507 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.486+0000 7fdb637fe640 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f30e7a7080 0x55f30ec48900 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae6ba3937fd4c4f server_cookie=d9b94ffcc38305f4 in_seq=0 out_seq=0 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb437fe640 1 -- 172.21.15.16:0/2407186658 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f30eb2cfe0 con 0x55f30e7a7080 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb437fe640 10 client.26507 handle_command_reply: tid=0 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb70f62740 1 client.26507 shutdown 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb70f62740 1 -- 172.21.15.16:0/2407186658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 msgr2=0x55f30ebdca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb70f62740 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 0x55f30ebdca90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdb54004eb0 tx=0x7fdb54032f40 comp rx=0 tx=0).stop 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb70f62740 1 -- 172.21.15.16:0/2407186658 shutdown_connections 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb70f62740 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f30e7a7080 0x55f30ec48900 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb70f62740 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f30e813350 0x55f30ec44ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb70f62740 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e7f2670 0x55f30ebdcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb70f62740 1 --2- 172.21.15.16:0/2407186658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f30e7ea490 0x55f30ebdca90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb70f62740 1 -- 172.21.15.16:0/2407186658 >> 172.21.15.16:0/2407186658 conn(0x55f30e807740 msgr2=0x55f30ebbd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.487+0000 7fdb437fe640 0 client.26507 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb70f62740 1 -- 172.21.15.16:0/2407186658 shutdown_connections 2024-08-28T19:26:07.320 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:26:07.320 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:26:07.320 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "b678eb41-8e9a-4031-8818-43d5f536898b", 2024-08-28T19:26:07.320 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:26:07.320 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb70f62740 1 -- 172.21.15.16:0/2407186658 wait complete. 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb70f62740 20 client.26507 trim_cache size 0 max 16384 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/3580655218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb4c0b4d20 msgr2=0x7fdb4c0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb4c0b4d20 0x7fdb4c0b7230 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c009fd0 tx=0x7fdb5c009500 comp rx=0 tx=0).stop 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/3580655218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb64108140 msgr2=0x7fdb6419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb64108140 0x7fdb6419a750 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fdb5800e9c0 tx=0x7fdb5800ee80 comp rx=0 tx=0).stop 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/3580655218 shutdown_connections 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdb4c0b4d20 0x7fdb4c0b7230 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64108a90 0x7fdb6419f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb64108140 0x7fdb6419a750 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 --2- 172.21.15.16:0/3580655218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb64102800 0x7fdb6419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/3580655218 >> 172.21.15.16:0/3580655218 conn(0x7fdb640794d0 msgr2=0x7fdb64101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/3580655218 shutdown_connections 2024-08-28T19:26:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:06.488+0000 7fdb6a9ed640 1 -- 172.21.15.16:0/3580655218 wait complete. 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: pgmap v491: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 359 op/s 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2153823048' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.323 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:07 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.323 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: pgmap v491: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 359 op/s 2024-08-28T19:26:07.323 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2153823048' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:07.323 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:07.323 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:07.323 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:07.323 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.323 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.323 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.323 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:07.324 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:07.324 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:07.324 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.324 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.324 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:07 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: pgmap v491: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 359 op/s 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2153823048' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:07 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:07.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:26:07.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270029 6.85 MB/sec execute 107 sec latency 1396.585 ms 2024-08-28T19:26:07.955 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:08.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:08 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:26:08.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:08 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:08.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:08 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:08.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:08 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:08.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:08 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:08 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:26:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:08 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:08 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:08 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:08 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:08.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:08 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:26:08.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:08 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:08.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:08 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:08.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:08 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:08.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:08 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.420+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3551050563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f4108270 msgr2=0x7f42f4108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.420+0000 7f42fb1ed640 1 --2- 172.21.15.16:0/3551050563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f4108270 0x7f42f4108680 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009900 tx=0x7f42e002a410 comp rx=0 tx=0).stop 2024-08-28T19:26:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.420+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3551050563 shutdown_connections 2024-08-28T19:26:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.420+0000 7f42fb1ed640 1 --2- 172.21.15.16:0/3551050563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42f4108bc0 0x7f42f410f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.420+0000 7f42fb1ed640 1 --2- 172.21.15.16:0/3551050563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f4108270 0x7f42f4108680 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.420+0000 7f42fb1ed640 1 --2- 172.21.15.16:0/3551050563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f4102820 0x7f42f4102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.420+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3551050563 >> 172.21.15.16:0/3551050563 conn(0x7f42f40794c0 msgr2=0x7f42f40798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:08.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.420+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3551050563 shutdown_connections 2024-08-28T19:26:08.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.420+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3551050563 wait complete. 2024-08-28T19:26:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42fb1ed640 1 Processor -- start 2024-08-28T19:26:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42fb1ed640 1 -- start start 2024-08-28T19:26:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42fb1ed640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42f4102820 0x7f42f4199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42fb1ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f4108270 0x7f42f4199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42fb1ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f4108bc0 0x7f42f419eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42fb1ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42f4112880 con 0x7f42f4108270 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42fb1ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42f4112700 con 0x7f42f4108bc0 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42fb1ed640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f42f4112a00 con 0x7f42f4102820 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f4108270 0x7f42f4199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f4108270 0x7f42f4199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38472/0 (socket says 172.21.15.16:38472) 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42f3fff640 1 -- 172.21.15.16:0/3118982294 learned_addr learned my addr 172.21.15.16:0/3118982294 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42f8f89640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42f4102820 0x7f42f4199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42f978a640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f4108bc0 0x7f42f419eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42f3fff640 1 -- 172.21.15.16:0/3118982294 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42f4102820 msgr2=0x7f42f4199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42f3fff640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42f4102820 0x7f42f4199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42f3fff640 1 -- 172.21.15.16:0/3118982294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f4108bc0 msgr2=0x7f42f419eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42f3fff640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f4108bc0 0x7f42f419eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42f3fff640 1 -- 172.21.15.16:0/3118982294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f419f1e0 con 0x7f42f4108270 2024-08-28T19:26:08.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.421+0000 7f42f978a640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f4108bc0 0x7f42f419eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:26:08.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.422+0000 7f42f3fff640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f4108270 0x7f42f4199810 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f42e0006010 tx=0x7f42e0004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.422+0000 7f42f1ffb640 1 -- 172.21.15.16:0/3118982294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e003f030 con 0x7f42f4108270 2024-08-28T19:26:08.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.422+0000 7f42f1ffb640 1 -- 172.21.15.16:0/3118982294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42e0003aa0 con 0x7f42f4108270 2024-08-28T19:26:08.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.422+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3118982294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f419f470 con 0x7f42f4108270 2024-08-28T19:26:08.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.422+0000 7f42f1ffb640 1 -- 172.21.15.16:0/3118982294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e0035070 con 0x7f42f4108270 2024-08-28T19:26:08.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.422+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3118982294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f41a5c30 con 0x7f42f4108270 2024-08-28T19:26:08.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f4301762740 1 Processor -- start 2024-08-28T19:26:08.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f42f1ffb640 1 -- 172.21.15.16:0/3118982294 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f42e0042030 con 0x7f42f4108270 2024-08-28T19:26:08.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f4301762740 1 -- start start 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f4301762740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1a5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f4301762740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 0x55e74d1a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f4301762740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdddb60 0x55e74d1a6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f4301762740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e74d11a330 con 0x55e74cdfe840 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f4301762740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e74d1098c0 con 0x55e74cdd5980 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f4301762740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e74d10d870 con 0x55e74cdddb60 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f42f978a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1a5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f42f978a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1a5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38486/0 (socket says 172.21.15.16:38486) 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f42f978a640 1 -- 172.21.15.16:0/1915845991 learned_addr learned my addr 172.21.15.16:0/1915845991 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.425+0000 7f42f8f89640 1 --2- 172.21.15.16:0/1915845991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 0x55e74d1a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42f978a640 1 -- 172.21.15.16:0/1915845991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdddb60 msgr2=0x55e74d1a6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42f978a640 1 --2- 172.21.15.16:0/1915845991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdddb60 0x55e74d1a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42f978a640 1 -- 172.21.15.16:0/1915845991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 msgr2=0x55e74d1a61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42f978a640 1 --2- 172.21.15.16:0/1915845991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 0x55e74d1a61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42f978a640 1 -- 172.21.15.16:0/1915845991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e74d0c1df0 con 0x55e74cdfe840 2024-08-28T19:26:08.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42f1ffb640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f42e40b4ca0 0x7f42e40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42f978a640 1 --2- 172.21.15.16:0/1915845991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1a5c90 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f42ec008dd0 tx=0x7f42ec00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6054379e70cf98b9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42f8f89640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f42e40b4ca0 0x7f42e40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42f1ffb640 1 -- 172.21.15.16:0/3118982294 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f42e0105e70 con 0x7f42f4108270 2024-08-28T19:26:08.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42deffd640 1 -- 172.21.15.16:0/1915845991 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec00f830 con 0x55e74cdfe840 2024-08-28T19:26:08.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42deffd640 1 -- 172.21.15.16:0/1915845991 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42ec00de20 con 0x55e74cdfe840 2024-08-28T19:26:08.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.426+0000 7f42deffd640 1 -- 172.21.15.16:0/1915845991 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec018d80 con 0x55e74cdfe840 2024-08-28T19:26:08.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.427+0000 7f4301762740 1 -- 172.21.15.16:0/1915845991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 msgr2=0x55e74d1a5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.427+0000 7f42f8f89640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f42e40b4ca0 0x7f42e40b71b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x55e74d1a4910 tx=0x7f42e8005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.427+0000 7f4301762740 1 --2- 172.21.15.16:0/1915845991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1a5c90 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f42ec008dd0 tx=0x7f42ec00d710 comp rx=0 tx=0).stop 2024-08-28T19:26:08.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.427+0000 7f4301762740 1 -- 172.21.15.16:0/1915845991 shutdown_connections 2024-08-28T19:26:08.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.427+0000 7f4301762740 1 --2- 172.21.15.16:0/1915845991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdddb60 0x55e74d1a6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.427+0000 7f4301762740 1 --2- 172.21.15.16:0/1915845991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 0x55e74d1a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.427+0000 7f4301762740 1 --2- 172.21.15.16:0/1915845991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1a5c90 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.427+0000 7f4301762740 1 -- 172.21.15.16:0/1915845991 >> 172.21.15.16:0/1915845991 conn(0x55e74cdf2990 msgr2=0x55e74d1a3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.428+0000 7f4301762740 1 -- 172.21.15.16:0/1915845991 shutdown_connections 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.428+0000 7f4301762740 1 -- 172.21.15.16:0/1915845991 wait complete. 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.428+0000 7f4301762740 1 -- start start 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.428+0000 7f4301762740 1 Processor -- start 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f4301762740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdd5980 0x55e74d1b3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f4301762740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdddb60 0x55e74d1b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f4301762740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1bf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f4301762740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e74cf6c7a0 con 0x55e74cdfe840 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f4301762740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e74d090240 con 0x55e74cdddb60 2024-08-28T19:26:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f4301762740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e74d1b65a0 con 0x55e74cdd5980 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f4301762740 10 client.0 _ll_register_callbacks cb 0x55e74ce16090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f42f3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdddb60 0x55e74d1b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f42f978a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1bf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f42f3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdddb60 0x55e74d1b3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34150/0 (socket says 172.21.15.16:34150) 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f42f978a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1bf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38494/0 (socket says 172.21.15.16:38494) 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f42f3fff640 1 -- 172.21.15.16:0/1561528176 learned_addr learned my addr 172.21.15.16:0/1561528176 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f42f8f89640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdd5980 0x55e74d1b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f42f3fff640 1 -- 172.21.15.16:0/1561528176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdd5980 msgr2=0x55e74d1b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f42f3fff640 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdd5980 0x55e74d1b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f42f3fff640 1 -- 172.21.15.16:0/1561528176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 msgr2=0x55e74d1bf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f42f3fff640 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1bf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.429+0000 7f42f3fff640 1 -- 172.21.15.16:0/1561528176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e74d0c62b0 con 0x55e74cdddb60 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.430+0000 7f42f978a640 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1bf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.430+0000 7f42f3fff640 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdddb60 0x55e74d1b3c10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f42e01032b0 tx=0x7f42e00c8170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bae8c129222f703 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.430+0000 7f42d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:26:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.430+0000 7f42d3fff640 1 -- 172.21.15.16:0/1561528176 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e0049070 con 0x55e74cdddb60 2024-08-28T19:26:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.430+0000 7f4301762740 1 -- 172.21.15.16:0/1561528176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42ec00de20 con 0x55e74cdddb60 2024-08-28T19:26:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.430+0000 7f42d3fff640 1 -- 172.21.15.16:0/1561528176 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f42e00c8480 con 0x55e74cdddb60 2024-08-28T19:26:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.430+0000 7f42d3fff640 1 -- 172.21.15.16:0/1561528176 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f42e00cb960 con 0x55e74cdddb60 2024-08-28T19:26:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.430+0000 7f4301762740 10 client.26516 fetch_fsmap learned FSMap version 40 2024-08-28T19:26:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.430+0000 7f4301762740 1 -- 172.21.15.16:0/1561528176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e74ce78b90 con 0x55e74cdddb60 2024-08-28T19:26:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.430+0000 7f42d3fff640 1 -- 172.21.15.16:0/1561528176 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42ec00de20 con 0x55e74cdddb60 2024-08-28T19:26:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.431+0000 7f42d3fff640 1 -- 172.21.15.16:0/1561528176 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f42e00cc450 con 0x55e74cdddb60 2024-08-28T19:26:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.431+0000 7f4301762740 10 client.26516 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.431+0000 7f4301762740 10 client.26516 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.431+0000 7f4301762740 20 client.26516 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.431+0000 7f4301762740 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e74cdbd1c0 0x55e74d1c5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.431+0000 7f4301762740 4 client.26516 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.431+0000 7f4301762740 1 -- 172.21.15.16:0/1561528176 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e74d11c280 con 0x55e74cdbd1c0 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.431+0000 7f42f8f89640 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e74cdbd1c0 0x55e74d1c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.431+0000 7f42f8f89640 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e74cdbd1c0 0x55e74d1c5760 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=851d287266225b09 server_cookie=65c792a40bbc000d in_seq=0 out_seq=0 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.431+0000 7f42d3fff640 10 client.26516 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.432+0000 7f42d3fff640 1 -- 172.21.15.16:0/1561528176 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e74d11c280 con 0x55e74cdbd1c0 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.432+0000 7f42d3fff640 10 client.26516 handle_command_reply: tid=0 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.432+0000 7f4301762740 1 client.26516 shutdown 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f4301762740 1 -- 172.21.15.16:0/1561528176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdddb60 msgr2=0x55e74d1b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f4301762740 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdddb60 0x55e74d1b3c10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f42e01032b0 tx=0x7f42e00c8170 comp rx=0 tx=0).stop 2024-08-28T19:26:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f4301762740 1 -- 172.21.15.16:0/1561528176 shutdown_connections 2024-08-28T19:26:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f4301762740 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1bf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f4301762740 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdddb60 0x55e74d1b3c10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f4301762740 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e74cdbd1c0 0x55e74d1c5760 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f4301762740 1 --2- 172.21.15.16:0/1561528176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdd5980 0x55e74d1b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f42d3fff640 0 client.26516 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f4301762740 1 -- 172.21.15.16:0/1561528176 >> 172.21.15.16:0/1561528176 conn(0x55e74cdf2990 msgr2=0x55e74d1a2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f4301762740 1 -- 172.21.15.16:0/1561528176 shutdown_connections 2024-08-28T19:26:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f4301762740 1 -- 172.21.15.16:0/1561528176 wait complete. 2024-08-28T19:26:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.433+0000 7f4301762740 20 client.26516 trim_cache size 0 max 16384 2024-08-28T19:26:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.451+0000 7f4301762740 1 Processor -- start 2024-08-28T19:26:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.451+0000 7f4301762740 1 -- start start 2024-08-28T19:26:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.451+0000 7f4301762740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdd5980 0x55e74d1b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.451+0000 7f4301762740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdddb60 0x55e74d1b3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.451+0000 7f4301762740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdfe840 0x55e74d1bf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.451+0000 7f4301762740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42f41ae910 con 0x55e74cdddb60 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.451+0000 7f4301762740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42f41b0510 con 0x55e74cdfe840 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.451+0000 7f4301762740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f42f41af040 con 0x55e74cdd5980 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.451+0000 7f42f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdddb60 0x55e74d1b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.451+0000 7f42f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdddb60 0x55e74d1b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38508/0 (socket says 172.21.15.16:38508) 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.451+0000 7f42f3fff640 1 -- 172.21.15.16:0/545171577 learned_addr learned my addr 172.21.15.16:0/545171577 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42f8f89640 1 --2- 172.21.15.16:0/545171577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdd5980 0x55e74d1b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42f978a640 1 --2- 172.21.15.16:0/545171577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdfe840 0x55e74d1bf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42f3fff640 1 -- 172.21.15.16:0/545171577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdd5980 msgr2=0x55e74d1b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42f3fff640 1 --2- 172.21.15.16:0/545171577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdd5980 0x55e74d1b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42f3fff640 1 -- 172.21.15.16:0/545171577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdfe840 msgr2=0x55e74d1bf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42f3fff640 1 --2- 172.21.15.16:0/545171577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdfe840 0x55e74d1bf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42f3fff640 1 -- 172.21.15.16:0/545171577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e74d1b3f80 con 0x55e74cdddb60 2024-08-28T19:26:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42f8f89640 1 --2- 172.21.15.16:0/545171577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdd5980 0x55e74d1b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42f978a640 1 --2- 172.21.15.16:0/545171577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdfe840 0x55e74d1bf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:26:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42f3fff640 1 --2- 172.21.15.16:0/545171577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdddb60 0x55e74d1b3440 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f42e0103850 tx=0x7f42e0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492fd8732398bbe9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42de7fc640 1 -- 172.21.15.16:0/545171577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e00cc8f0 con 0x55e74cdddb60 2024-08-28T19:26:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42de7fc640 1 -- 172.21.15.16:0/545171577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42e00d0a60 con 0x55e74cdddb60 2024-08-28T19:26:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.452+0000 7f42de7fc640 1 -- 172.21.15.16:0/545171577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e00d5080 con 0x55e74cdddb60 2024-08-28T19:26:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.453+0000 7f4301762740 1 -- 172.21.15.16:0/545171577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdddb60 msgr2=0x55e74d1b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.453+0000 7f4301762740 1 --2- 172.21.15.16:0/545171577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdddb60 0x55e74d1b3440 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f42e0103850 tx=0x7f42e0000bc0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.453+0000 7f4301762740 1 -- 172.21.15.16:0/545171577 shutdown_connections 2024-08-28T19:26:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.453+0000 7f4301762740 1 --2- 172.21.15.16:0/545171577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdfe840 0x55e74d1bf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.453+0000 7f4301762740 1 --2- 172.21.15.16:0/545171577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdddb60 0x55e74d1b3440 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.453+0000 7f4301762740 1 --2- 172.21.15.16:0/545171577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdd5980 0x55e74d1b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.453+0000 7f4301762740 1 -- 172.21.15.16:0/545171577 >> 172.21.15.16:0/545171577 conn(0x55e74cdf2990 msgr2=0x55e74d1ab7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.453+0000 7f4301762740 1 -- 172.21.15.16:0/545171577 shutdown_connections 2024-08-28T19:26:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.453+0000 7f4301762740 1 -- 172.21.15.16:0/545171577 wait complete. 2024-08-28T19:26:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.454+0000 7f4301762740 1 -- start start 2024-08-28T19:26:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.454+0000 7f4301762740 1 Processor -- start 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.454+0000 7f4301762740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 0x55e74d1bba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.454+0000 7f4301762740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdddb60 0x55e74d1bc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.454+0000 7f4301762740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1b6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.454+0000 7f4301762740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42f41afe40 con 0x55e74cdfe840 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.454+0000 7f4301762740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42f41af740 con 0x55e74cdd5980 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.454+0000 7f4301762740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f42f41ac8f0 con 0x55e74cdddb60 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.454+0000 7f4301762740 10 client.0 _ll_register_callbacks cb 0x7f42e0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f978a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f8f89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 0x55e74d1bba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f8f89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 0x55e74d1bba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34156/0 (socket says 172.21.15.16:34156) 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f3fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdddb60 0x55e74d1bc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f8f89640 1 -- 172.21.15.16:0/2818857387 learned_addr learned my addr 172.21.15.16:0/2818857387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f3fff640 1 -- 172.21.15.16:0/2818857387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 msgr2=0x55e74d1bba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f3fff640 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 0x55e74d1bba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f3fff640 1 -- 172.21.15.16:0/2818857387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 msgr2=0x55e74d1b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f3fff640 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f3fff640 1 -- 172.21.15.16:0/2818857387 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e74d0d9d90 con 0x55e74cdddb60 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f8f89640 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 0x55e74d1bba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f978a640 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42f3fff640 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdddb60 0x55e74d1bc010 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f42e003d040 tx=0x7f42e003b5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a792016a26f6875 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.455+0000 7f42d3fff640 1 -- 172.21.15.16:0/2818857387 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e0049070 con 0x55e74cdddb60 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f4301762740 1 -- 172.21.15.16:0/2818857387 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42e00d0a60 con 0x55e74cdddb60 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f42d3fff640 1 -- 172.21.15.16:0/2818857387 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f42e00d5290 con 0x55e74cdddb60 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f42d3fff640 1 -- 172.21.15.16:0/2818857387 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f42e00cbd00 con 0x55e74cdddb60 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f4301762740 10 client.26344 fetch_fsmap learned FSMap version 40 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f4301762740 1 -- 172.21.15.16:0/2818857387 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e74d0d9d90 con 0x55e74cdddb60 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f42d3fff640 1 -- 172.21.15.16:0/2818857387 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42e00d0a60 con 0x55e74cdddb60 2024-08-28T19:26:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f42d3fff640 1 -- 172.21.15.16:0/2818857387 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f42e00c7cf0 con 0x55e74cdddb60 2024-08-28T19:26:08.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f4301762740 10 client.26344 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:26:08.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f4301762740 10 client.26344 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:26:08.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f4301762740 20 client.26344 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:08.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f4301762740 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e74cd92570 0x55e74d234ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f4301762740 4 client.26344 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-08-28T19:26:08.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.456+0000 7f4301762740 1 -- 172.21.15.16:0/2818857387 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x55e74d117d20 con 0x55e74cd92570 2024-08-28T19:26:08.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.457+0000 7f42f8f89640 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e74cd92570 0x55e74d234ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.457+0000 7f42f8f89640 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e74cd92570 0x55e74d234ee0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd222c81a8ab395e server_cookie=6ea067e10f87e3c2 in_seq=0 out_seq=0 2024-08-28T19:26:08.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.457+0000 7f42d3fff640 10 client.26344 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:08.493 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:08.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270029 6.79 MB/sec execute 108 sec latency 2396.632 ms 2024-08-28T19:26:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.957+0000 7f13fc14a640 1 -- 172.21.15.16:0/3781620440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f4109be0 msgr2=0x7f13f4110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.957+0000 7f13fc14a640 1 --2- 172.21.15.16:0/3781620440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f4109be0 0x7f13f4110410 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f13ec009ca0 tx=0x7f13ec02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.957+0000 7f13fc14a640 1 -- 172.21.15.16:0/3781620440 shutdown_connections 2024-08-28T19:26:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.957+0000 7f13fc14a640 1 --2- 172.21.15.16:0/3781620440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f4109be0 0x7f13f4110410 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.957+0000 7f13fc14a640 1 --2- 172.21.15.16:0/3781620440 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13f4109290 0x7f13f41096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.957+0000 7f13fc14a640 1 --2- 172.21.15.16:0/3781620440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f4103950 0x7f13f4103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.957+0000 7f13fc14a640 1 -- 172.21.15.16:0/3781620440 >> 172.21.15.16:0/3781620440 conn(0x7f13f40ff3a0 msgr2=0x7f13f41017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.957+0000 7f13fc14a640 1 -- 172.21.15.16:0/3781620440 shutdown_connections 2024-08-28T19:26:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.957+0000 7f13fc14a640 1 -- 172.21.15.16:0/3781620440 wait complete. 2024-08-28T19:26:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.958+0000 7f13fc14a640 1 Processor -- start 2024-08-28T19:26:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.958+0000 7f13fc14a640 1 -- start start 2024-08-28T19:26:08.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.958+0000 7f13fc14a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f4103950 0x7f13f419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.958+0000 7f13fc14a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13f4109290 0x7f13f419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.958+0000 7f13fc14a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f4109be0 0x7f13f419fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.958+0000 7f13fc14a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13f41138a0 con 0x7f13f4109be0 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13fc14a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13f4113720 con 0x7f13f4103950 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13fc14a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f13f4113a20 con 0x7f13f4109290 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13fa6e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f4109be0 0x7f13f419fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13f9ee6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f4103950 0x7f13f419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13fa6e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f4109be0 0x7f13f419fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38536/0 (socket says 172.21.15.16:38536) 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13f9ee6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f4103950 0x7f13f419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34164/0 (socket says 172.21.15.16:34164) 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13fa6e7640 1 -- 172.21.15.16:0/1398964239 learned_addr learned my addr 172.21.15.16:0/1398964239 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13f4109290 0x7f13f419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13fa6e7640 1 -- 172.21.15.16:0/1398964239 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13f4109290 msgr2=0x7f13f419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13fa6e7640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13f4109290 0x7f13f419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13fa6e7640 1 -- 172.21.15.16:0/1398964239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f4103950 msgr2=0x7f13f419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13fa6e7640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f4103950 0x7f13f419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13fa6e7640 1 -- 172.21.15.16:0/1398964239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f41a0140 con 0x7f13f4109be0 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f4103950 0x7f13f419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13fa6e7640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f4109be0 0x7f13f419fa10 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0098d0 tx=0x7f13ec0317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.959+0000 7f13eaffd640 1 -- 172.21.15.16:0/1398964239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec003b70 con 0x7f13f4109be0 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.960+0000 7f13eaffd640 1 -- 172.21.15.16:0/1398964239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13ec003d10 con 0x7f13f4109be0 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.960+0000 7f13fc14a640 1 -- 172.21.15.16:0/1398964239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13f41a03d0 con 0x7f13f4109be0 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.960+0000 7f13eaffd640 1 -- 172.21.15.16:0/1398964239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec02fa20 con 0x7f13f4109be0 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.960+0000 7f13fc14a640 1 -- 172.21.15.16:0/1398964239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f41a6b90 con 0x7f13f4109be0 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.962+0000 7f14026bf740 1 Processor -- start 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.962+0000 7f13eaffd640 1 -- 172.21.15.16:0/1398964239 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f13ec02fc40 con 0x7f13f4109be0 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f14026bf740 1 -- start start 2024-08-28T19:26:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f14026bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd66012c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f14026bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 0x558cd6601800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f14026bf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd623b390 0x558cd6601d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f14026bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558cd6578630 con 0x558cd62331b0 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f14026bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558cd6567bc0 con 0x558cd625c070 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f14026bf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558cd656bb70 con 0x558cd623b390 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13f9ee6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 0x558cd6601800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13f9ee6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 0x558cd6601800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38552/0 (socket says 172.21.15.16:38552) 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1170619147 learned_addr learned my addr 172.21.15.16:0/1170619147 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1170619147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd66012c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13eaffd640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13e00b4d20 0x7f13e00b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1170619147 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd623b390 0x558cd6601d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1170619147 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd623b390 msgr2=0x558cd6601d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/1170619147 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd623b390 0x558cd6601d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1170619147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 msgr2=0x558cd66012c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/1170619147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd66012c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1170619147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cd62c7650 con 0x558cd62331b0 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13fa6e7640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13e00b4d20 0x7f13e00b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1170619147 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd623b390 0x558cd6601d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.963+0000 7f13eaffd640 1 -- 172.21.15.16:0/1398964239 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f13ec106b00 con 0x7f13f4109be0 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.964+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/1170619147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 0x558cd6601800 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f13f00115f0 tx=0x7f13f0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b54fa43a2bf1fc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.964+0000 7f13dffff640 1 -- 172.21.15.16:0/1170619147 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f001e070 con 0x558cd62331b0 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.964+0000 7f13dffff640 1 -- 172.21.15.16:0/1170619147 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13f000faf0 con 0x558cd62331b0 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.964+0000 7f13dffff640 1 -- 172.21.15.16:0/1170619147 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f0019b10 con 0x558cd62331b0 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.964+0000 7f13fa6e7640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13e00b4d20 0x7f13e00b7230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f13ec103f90 tx=0x7f13ec02a8c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.965+0000 7f14026bf740 1 -- 172.21.15.16:0/1170619147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 msgr2=0x558cd6601800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.965+0000 7f14026bf740 1 --2- 172.21.15.16:0/1170619147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 0x558cd6601800 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f13f00115f0 tx=0x7f13f0011ab0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.965+0000 7f14026bf740 1 -- 172.21.15.16:0/1170619147 shutdown_connections 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.965+0000 7f14026bf740 1 --2- 172.21.15.16:0/1170619147 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd623b390 0x558cd6601d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.965+0000 7f14026bf740 1 --2- 172.21.15.16:0/1170619147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 0x558cd6601800 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.965+0000 7f14026bf740 1 --2- 172.21.15.16:0/1170619147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd66012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.965+0000 7f14026bf740 1 -- 172.21.15.16:0/1170619147 >> 172.21.15.16:0/1170619147 conn(0x558cd6250460 msgr2=0x558cd65ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.965+0000 7f14026bf740 1 -- 172.21.15.16:0/1170619147 shutdown_connections 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.965+0000 7f14026bf740 1 -- 172.21.15.16:0/1170619147 wait complete. 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.965+0000 7f14026bf740 1 -- start start 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.966+0000 7f14026bf740 1 Processor -- start 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.966+0000 7f14026bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 0x558cd660cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.966+0000 7f14026bf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd623b390 0x558cd660d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.966+0000 7f14026bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd6618df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.966+0000 7f14026bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558cd63ca700 con 0x558cd62331b0 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.966+0000 7f14026bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558cd64ee700 con 0x558cd625c070 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.966+0000 7f14026bf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558cd660faa0 con 0x558cd623b390 2024-08-28T19:26:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.966+0000 7f14026bf740 10 client.0 _ll_register_callbacks cb 0x558cd651ddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.966+0000 7f13f96e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 0x558cd660cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.966+0000 7f13f96e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 0x558cd660cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38558/0 (socket says 172.21.15.16:38558) 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.966+0000 7f13f96e5640 1 -- 172.21.15.16:0/2631525012 learned_addr learned my addr 172.21.15.16:0/2631525012 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f13f96e5640 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd6618df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd623b390 0x558cd660d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f13f96e5640 1 -- 172.21.15.16:0/2631525012 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd623b390 msgr2=0x558cd660d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f13f96e5640 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd623b390 0x558cd660d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f13f96e5640 1 -- 172.21.15.16:0/2631525012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 msgr2=0x558cd660cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f13f96e5640 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 0x558cd660cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f13f96e5640 1 -- 172.21.15.16:0/2631525012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cd65278b0 con 0x558cd625c070 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f13f96e5640 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd6618df0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f13e400bc90 tx=0x7f13e4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e6927e139ce5cf1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f13dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f13dd7fa640 1 -- 172.21.15.16:0/2631525012 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e401b040 con 0x558cd625c070 2024-08-28T19:26:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f14026bf740 1 -- 172.21.15.16:0/2631525012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13f000faf0 con 0x558cd625c070 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.967+0000 7f13dd7fa640 1 -- 172.21.15.16:0/2631525012 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f13e4004020 con 0x558cd625c070 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f13dd7fa640 1 -- 172.21.15.16:0/2631525012 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f13e4029050 con 0x558cd625c070 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f14026bf740 10 client.26531 fetch_fsmap learned FSMap version 40 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f14026bf740 1 -- 172.21.15.16:0/2631525012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x558cd661b9b0 con 0x558cd625c070 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f13dd7fa640 1 -- 172.21.15.16:0/2631525012 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13f000faf0 con 0x558cd625c070 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f13dd7fa640 1 -- 172.21.15.16:0/2631525012 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f13e4012ce0 con 0x558cd625c070 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f14026bf740 10 client.26531 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f14026bf740 10 client.26531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f14026bf740 20 client.26531 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f14026bf740 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558cd621a9f0 0x558cd661ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f14026bf740 4 client.26531 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f14026bf740 1 -- 172.21.15.16:0/2631525012 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cd657a580 con 0x558cd621a9f0 2024-08-28T19:26:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.968+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558cd621a9f0 0x558cd661ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.969+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558cd621a9f0 0x558cd661ee10 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6a1c398d1dd41e0 server_cookie=2d43a495b2c6c571 in_seq=0 out_seq=0 2024-08-28T19:26:08.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.969+0000 7f13dd7fa640 10 client.26531 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:08.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f13dd7fa640 1 -- 172.21.15.16:0/2631525012 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558cd657a580 con 0x558cd621a9f0 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f13dd7fa640 10 client.26531 handle_command_reply: tid=0 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 1 client.26531 shutdown 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 1 -- 172.21.15.16:0/2631525012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 msgr2=0x558cd6618df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd6618df0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f13e400bc90 tx=0x7f13e4004730 comp rx=0 tx=0).stop 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 1 -- 172.21.15.16:0/2631525012 shutdown_connections 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd6618df0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558cd621a9f0 0x558cd661ee10 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd623b390 0x558cd660d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 1 --2- 172.21.15.16:0/2631525012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd62331b0 0x558cd660cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f13dd7fa640 0 client.26531 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 1 -- 172.21.15.16:0/2631525012 >> 172.21.15.16:0/2631525012 conn(0x558cd6250460 msgr2=0x558cd6604450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 1 -- 172.21.15.16:0/2631525012 shutdown_connections 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 1 -- 172.21.15.16:0/2631525012 wait complete. 2024-08-28T19:26:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.970+0000 7f14026bf740 20 client.26531 trim_cache size 0 max 16384 2024-08-28T19:26:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f14026bf740 1 Processor -- start 2024-08-28T19:26:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f14026bf740 1 -- start start 2024-08-28T19:26:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f14026bf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd62331b0 0x558cd6604ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f14026bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd6609cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f14026bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd660a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f14026bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13f41b0c00 con 0x558cd623b390 2024-08-28T19:26:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f14026bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13f41b2800 con 0x558cd625c070 2024-08-28T19:26:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f14026bf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f13f41b1330 con 0x558cd62331b0 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f13f9ee6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd6609cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f13f9ee6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd6609cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38570/0 (socket says 172.21.15.16:38570) 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1050731148 learned_addr learned my addr 172.21.15.16:0/1050731148 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1050731148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd660a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.988+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1050731148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd62331b0 0x558cd6604ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.989+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1050731148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd62331b0 msgr2=0x558cd6604ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.989+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/1050731148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd62331b0 0x558cd6604ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.989+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1050731148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 msgr2=0x558cd660a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.989+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/1050731148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd660a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.989+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1050731148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cd660dd80 con 0x558cd623b390 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.989+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1050731148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd660a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.989+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/1050731148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd6609cb0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f13f00196e0 tx=0x7f13f0019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866b62584e5fcadc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.989+0000 7f13df7fe640 1 -- 172.21.15.16:0/1050731148 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f0032070 con 0x558cd623b390 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.989+0000 7f13df7fe640 1 -- 172.21.15.16:0/1050731148 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13f000faf0 con 0x558cd623b390 2024-08-28T19:26:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.989+0000 7f13df7fe640 1 -- 172.21.15.16:0/1050731148 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f001e040 con 0x558cd623b390 2024-08-28T19:26:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.990+0000 7f14026bf740 1 -- 172.21.15.16:0/1050731148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 msgr2=0x558cd6609cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.990+0000 7f14026bf740 1 --2- 172.21.15.16:0/1050731148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd6609cb0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f13f00196e0 tx=0x7f13f0019670 comp rx=0 tx=0).stop 2024-08-28T19:26:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.990+0000 7f14026bf740 1 -- 172.21.15.16:0/1050731148 shutdown_connections 2024-08-28T19:26:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.990+0000 7f14026bf740 1 --2- 172.21.15.16:0/1050731148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd625c070 0x558cd660a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.990+0000 7f14026bf740 1 --2- 172.21.15.16:0/1050731148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd6609cb0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.990+0000 7f14026bf740 1 --2- 172.21.15.16:0/1050731148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd62331b0 0x558cd6604ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.990+0000 7f14026bf740 1 -- 172.21.15.16:0/1050731148 >> 172.21.15.16:0/1050731148 conn(0x558cd6250460 msgr2=0x558cd66043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.990+0000 7f14026bf740 1 -- 172.21.15.16:0/1050731148 shutdown_connections 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.990+0000 7f14026bf740 1 -- 172.21.15.16:0/1050731148 wait complete. 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.990+0000 7f14026bf740 1 -- start start 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f14026bf740 1 Processor -- start 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f14026bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd62331b0 0x558cd6615210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f14026bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd66116d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f14026bf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd625c070 0x558cd6611c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f14026bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13f41b2130 con 0x558cd623b390 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f14026bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13f41b1a30 con 0x558cd62331b0 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f14026bf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f13f41aebe0 con 0x558cd625c070 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f14026bf740 10 client.0 _ll_register_callbacks cb 0x7f13e4012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f13f9ee6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd66116d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f13f9ee6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd66116d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38572/0 (socket says 172.21.15.16:38572) 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1722341918 learned_addr learned my addr 172.21.15.16:0/1722341918 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd62331b0 0x558cd6615210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.991+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd625c070 0x558cd6611c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1722341918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd625c070 msgr2=0x558cd6611c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd625c070 0x558cd6611c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1722341918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd62331b0 msgr2=0x558cd6615210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd62331b0 0x558cd6615210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f13f9ee6640 1 -- 172.21.15.16:0/1722341918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cd6612150 con 0x558cd623b390 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd625c070 0x558cd6611c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f13f9ee6640 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd66116d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f13f0011af0 tx=0x7f13f0019530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33a72c297a26ef0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f13dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f13dd7fa640 1 -- 172.21.15.16:0/1722341918 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f0032030 con 0x558cd623b390 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f13dd7fa640 1 -- 172.21.15.16:0/1722341918 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f13f0016040 con 0x558cd623b390 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f14026bf740 1 -- 172.21.15.16:0/1722341918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13f000faf0 con 0x558cd623b390 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f13dd7fa640 1 -- 172.21.15.16:0/1722341918 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f13f001e050 con 0x558cd623b390 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f14026bf740 10 client.16968 fetch_fsmap learned FSMap version 40 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.992+0000 7f14026bf740 1 -- 172.21.15.16:0/1722341918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x558cd668a410 con 0x558cd623b390 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.993+0000 7f13dd7fa640 1 -- 172.21.15.16:0/1722341918 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13f000faf0 con 0x558cd623b390 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.993+0000 7f13dd7fa640 1 -- 172.21.15.16:0/1722341918 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f13f0019910 con 0x558cd623b390 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.993+0000 7f14026bf740 10 client.16968 fetch_fsmap finished waiting for FSMap version 40 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.993+0000 7f14026bf740 10 client.16968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.993+0000 7f14026bf740 20 client.16968 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.993+0000 7f14026bf740 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558cd61efda0 0x558cd668c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.993+0000 7f14026bf740 4 client.16968 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.993+0000 7f14026bf740 1 -- 172.21.15.16:0/1722341918 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558cd6576020 con 0x558cd61efda0 2024-08-28T19:26:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.993+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558cd61efda0 0x558cd668c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f13f96e5640 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558cd61efda0 0x558cd668c900 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=238e1d7d802c668c server_cookie=a9ed2e217d2d35ad in_seq=0 out_seq=0 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f13dd7fa640 10 client.16968 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f13dd7fa640 1 -- 172.21.15.16:0/1722341918 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2564 (crc 0 0 0) 0x558cd6576020 con 0x558cd61efda0 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f13dd7fa640 10 client.16968 handle_command_reply: tid=0 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f14026bf740 1 client.16968 shutdown 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f14026bf740 1 -- 172.21.15.16:0/1722341918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 msgr2=0x558cd66116d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f14026bf740 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd66116d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f13f0011af0 tx=0x7f13f0019530 comp rx=0 tx=0).stop 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f14026bf740 1 -- 172.21.15.16:0/1722341918 shutdown_connections 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f14026bf740 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558cd61efda0 0x558cd668c900 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f14026bf740 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558cd625c070 0x558cd6611c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f14026bf740 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cd623b390 0x558cd66116d0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f13dd7fa640 0 client.16968 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f14026bf740 1 --2- 172.21.15.16:0/1722341918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd62331b0 0x558cd6615210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f14026bf740 1 -- 172.21.15.16:0/1722341918 >> 172.21.15.16:0/1722341918 conn(0x558cd6250460 msgr2=0x558cd66022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f14026bf740 1 -- 172.21.15.16:0/1722341918 shutdown_connections 2024-08-28T19:26:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.994+0000 7f14026bf740 1 -- 172.21.15.16:0/1722341918 wait complete. 2024-08-28T19:26:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f14026bf740 20 client.16968 trim_cache size 0 max 16384 2024-08-28T19:26:09.000 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:26:09.000 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "scrub active (0 inodes in the stack)", 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "b678eb41-8e9a-4031-8818-43d5f536898b", 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:09.001 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:09.002 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:09.003 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:09.004 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:09.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:09.006 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:26:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 -- 172.21.15.16:0/1398964239 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13e00b4d20 msgr2=0x7f13e00b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13e00b4d20 0x7f13e00b7230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f13ec103f90 tx=0x7f13ec02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 -- 172.21.15.16:0/1398964239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f4109be0 msgr2=0x7f13f419fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f4109be0 0x7f13f419fa10 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0098d0 tx=0x7f13ec0317b0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 -- 172.21.15.16:0/1398964239 shutdown_connections 2024-08-28T19:26:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f4109be0 0x7f13f419fa10 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13f4109290 0x7f13f419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13e00b4d20 0x7f13e00b7230 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 --2- 172.21.15.16:0/1398964239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f4103950 0x7f13f419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 -- 172.21.15.16:0/1398964239 >> 172.21.15.16:0/1398964239 conn(0x7f13f40ff3a0 msgr2=0x7f13f41017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 -- 172.21.15.16:0/1398964239 shutdown_connections 2024-08-28T19:26:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:08.995+0000 7f13fc14a640 1 -- 172.21.15.16:0/1398964239 wait complete. 2024-08-28T19:26:09.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:09 smithi016 ceph-mon[18556]: pgmap v492: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 283 op/s 2024-08-28T19:26:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:09 smithi028 ceph-mon[21769]: pgmap v492: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 283 op/s 2024-08-28T19:26:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:09 smithi115 ceph-mon[22184]: pgmap v492: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 283 op/s 2024-08-28T19:26:09.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.547+0000 7f42d3fff640 1 -- 172.21.15.16:0/2818857387 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e74d117d20 con 0x55e74cd92570 2024-08-28T19:26:09.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.547+0000 7f42d3fff640 10 client.26344 handle_command_reply: tid=0 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.547+0000 7f4301762740 1 client.26344 shutdown 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f4301762740 1 -- 172.21.15.16:0/2818857387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdddb60 msgr2=0x55e74d1bc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f4301762740 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdddb60 0x55e74d1bc010 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f42e003d040 tx=0x7f42e003b5d0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f4301762740 1 -- 172.21.15.16:0/2818857387 shutdown_connections 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f4301762740 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74cdfe840 0x55e74d1b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f4301762740 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e74cd92570 0x55e74d234ee0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f4301762740 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e74cdddb60 0x55e74d1bc010 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f4301762740 1 --2- 172.21.15.16:0/2818857387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e74cdd5980 0x55e74d1bba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f42d3fff640 0 client.26344 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f4301762740 1 -- 172.21.15.16:0/2818857387 >> 172.21.15.16:0/2818857387 conn(0x55e74cdf2990 msgr2=0x55e74d1bf940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f4301762740 1 -- 172.21.15.16:0/2818857387 shutdown_connections 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f4301762740 1 -- 172.21.15.16:0/2818857387 wait complete. 2024-08-28T19:26:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.548+0000 7f4301762740 20 client.26344 trim_cache size 0 max 16384 2024-08-28T19:26:09.551 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:26:09.551 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 40, 2024-08-28T19:26:09.551 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:26:09.551 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 14, 2024-08-28T19:26:09.551 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:26:09.551 INFO:teuthology.orchestra.run.smithi016.stdout: "bbce1fd4": { 2024-08-28T19:26:09.551 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 14, 2024-08-28T19:26:09.551 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:26:09.551 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:26:09.551 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:26:09.551 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:09.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:09.553 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:09.553 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:26:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3118982294 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f42e40b4ca0 msgr2=0x7f42e40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f42e40b4ca0 0x7f42e40b71b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x55e74d1a4910 tx=0x7f42e8005f90 comp rx=0 tx=0).stop 2024-08-28T19:26:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3118982294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f4108270 msgr2=0x7f42f4199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f4108270 0x7f42f4199810 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f42e0006010 tx=0x7f42e0004060 comp rx=0 tx=0).stop 2024-08-28T19:26:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3118982294 shutdown_connections 2024-08-28T19:26:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f42e40b4ca0 0x7f42e40b71b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f4108bc0 0x7f42f419eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f4108270 0x7f42f4199810 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 --2- 172.21.15.16:0/3118982294 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42f4102820 0x7f42f4199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3118982294 >> 172.21.15.16:0/3118982294 conn(0x7f42f40794c0 msgr2=0x7f42f4106360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3118982294 shutdown_connections 2024-08-28T19:26:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:09.549+0000 7f42fb1ed640 1 -- 172.21.15.16:0/3118982294 wait complete. 2024-08-28T19:26:09.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.73 MB/sec execute 109 sec latency 2996.441 ms 2024-08-28T19:26:09.963 INFO:tasks.cephfs.filesystem:scrub status for tag:b678eb41-8e9a-4031-8818-43d5f536898b - {'path': '/', 'tag': 'b678eb41-8e9a-4031-8818-43d5f536898b', 'options': 'recursive,force'} 2024-08-28T19:26:10.947 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'bbce1fd4', quiesce took 5.8 seconds. Will release after: 24.2 2024-08-28T19:26:10.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.66 MB/sec execute 110 sec latency 1399.370 ms 2024-08-28T19:26:11.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:11 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:11.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:11 smithi016 ceph-mon[18556]: pgmap v493: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 297 op/s 2024-08-28T19:26:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:11 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:11 smithi028 ceph-mon[21769]: pgmap v493: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 297 op/s 2024-08-28T19:26:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:11 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:11 smithi115 ceph-mon[22184]: pgmap v493: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 297 op/s 2024-08-28T19:26:11.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.60 MB/sec execute 111 sec latency 2399.424 ms 2024-08-28T19:26:12.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.55 MB/sec execute 112 sec latency 3399.503 ms 2024-08-28T19:26:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:13 smithi115 ceph-mon[22184]: pgmap v494: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 295 op/s 2024-08-28T19:26:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:13 smithi016 ceph-mon[18556]: pgmap v494: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 295 op/s 2024-08-28T19:26:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:13 smithi028 ceph-mon[21769]: pgmap v494: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 295 op/s 2024-08-28T19:26:13.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.49 MB/sec execute 113 sec latency 4399.563 ms 2024-08-28T19:26:14.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:26:14.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:26:14.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:26:14.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:26:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:26:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:26:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:26:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:26:14.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:26:14.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:26:14.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:26:14.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:26:14.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.43 MB/sec execute 114 sec latency 5399.617 ms 2024-08-28T19:26:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:15 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:26:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:15 smithi115 ceph-mon[22184]: pgmap v495: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 198 op/s 2024-08-28T19:26:15.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:15 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:26:15.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:15 smithi016 ceph-mon[18556]: pgmap v495: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 198 op/s 2024-08-28T19:26:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:15 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:26:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:15 smithi028 ceph-mon[21769]: pgmap v495: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 198 op/s 2024-08-28T19:26:15.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.37 MB/sec execute 115 sec latency 6399.699 ms 2024-08-28T19:26:16.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.32 MB/sec execute 116 sec latency 7399.831 ms 2024-08-28T19:26:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:17 smithi115 ceph-mon[22184]: pgmap v496: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 214 op/s 2024-08-28T19:26:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:17 smithi016 ceph-mon[18556]: pgmap v496: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 214 op/s 2024-08-28T19:26:17.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:17 smithi028 ceph-mon[21769]: pgmap v496: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 214 op/s 2024-08-28T19:26:17.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.27 MB/sec execute 117 sec latency 8399.919 ms 2024-08-28T19:26:18.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:18 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:26:18.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:18 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:26:18.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:18 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:26:18.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.21 MB/sec execute 118 sec latency 9400.030 ms 2024-08-28T19:26:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:19 smithi115 ceph-mon[22184]: pgmap v497: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 1.9 MiB/s wr, 135 op/s 2024-08-28T19:26:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:19 smithi016 ceph-mon[18556]: pgmap v497: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 1.9 MiB/s wr, 135 op/s 2024-08-28T19:26:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:19 smithi028 ceph-mon[21769]: pgmap v497: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 1.9 MiB/s wr, 135 op/s 2024-08-28T19:26:19.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.16 MB/sec execute 119 sec latency 10400.155 ms 2024-08-28T19:26:20.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.11 MB/sec execute 120 sec latency 11400.249 ms 2024-08-28T19:26:21.361 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:21 smithi028 ceph-mon[21769]: pgmap v498: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 2.0 MiB/s wr, 113 op/s 2024-08-28T19:26:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:21 smithi115 ceph-mon[22184]: pgmap v498: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 2.0 MiB/s wr, 113 op/s 2024-08-28T19:26:21.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:21 smithi016 ceph-mon[18556]: pgmap v498: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 2.0 MiB/s wr, 113 op/s 2024-08-28T19:26:21.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.06 MB/sec execute 121 sec latency 12400.362 ms 2024-08-28T19:26:22.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 6.01 MB/sec execute 122 sec latency 13400.404 ms 2024-08-28T19:26:23.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:23 smithi115 ceph-mon[22184]: pgmap v499: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.1 MiB/s wr, 105 op/s 2024-08-28T19:26:23.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:23 smithi016 ceph-mon[18556]: pgmap v499: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.1 MiB/s wr, 105 op/s 2024-08-28T19:26:23.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:23 smithi028 ceph-mon[21769]: pgmap v499: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.1 MiB/s wr, 105 op/s 2024-08-28T19:26:23.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.96 MB/sec execute 123 sec latency 14400.510 ms 2024-08-28T19:26:24.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.91 MB/sec execute 124 sec latency 15400.607 ms 2024-08-28T19:26:25.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:25 smithi115 ceph-mon[22184]: pgmap v500: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.9 MiB/s wr, 77 op/s 2024-08-28T19:26:25.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:25 smithi016 ceph-mon[18556]: pgmap v500: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.9 MiB/s wr, 77 op/s 2024-08-28T19:26:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:25 smithi028 ceph-mon[21769]: pgmap v500: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.9 MiB/s wr, 77 op/s 2024-08-28T19:26:25.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.86 MB/sec execute 125 sec latency 16400.738 ms 2024-08-28T19:26:26.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.82 MB/sec execute 126 sec latency 17400.838 ms 2024-08-28T19:26:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:27 smithi016 ceph-mon[18556]: pgmap v501: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 2.2 MiB/s wr, 97 op/s 2024-08-28T19:26:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:27 smithi028 ceph-mon[21769]: pgmap v501: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 2.2 MiB/s wr, 97 op/s 2024-08-28T19:26:27.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:27 smithi115 ceph-mon[22184]: pgmap v501: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 2.2 MiB/s wr, 97 op/s 2024-08-28T19:26:27.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.77 MB/sec execute 127 sec latency 18400.898 ms 2024-08-28T19:26:28.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.73 MB/sec execute 128 sec latency 19400.954 ms 2024-08-28T19:26:29.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:29 smithi016 ceph-mon[18556]: pgmap v502: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-08-28T19:26:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:29 smithi028 ceph-mon[21769]: pgmap v502: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-08-28T19:26:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:29 smithi115 ceph-mon[22184]: pgmap v502: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-08-28T19:26:29.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.68 MB/sec execute 129 sec latency 20401.043 ms 2024-08-28T19:26:30.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.64 MB/sec execute 130 sec latency 21401.117 ms 2024-08-28T19:26:31.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:31 smithi016 ceph-mon[18556]: pgmap v503: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.9 MiB/s wr, 85 op/s 2024-08-28T19:26:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:31 smithi028 ceph-mon[21769]: pgmap v503: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.9 MiB/s wr, 85 op/s 2024-08-28T19:26:31.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:31 smithi115 ceph-mon[22184]: pgmap v503: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.9 MiB/s wr, 85 op/s 2024-08-28T19:26:31.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.60 MB/sec execute 131 sec latency 22401.175 ms 2024-08-28T19:26:32.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.55 MB/sec execute 132 sec latency 23401.281 ms 2024-08-28T19:26:33.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:33 smithi016 ceph-mon[18556]: pgmap v504: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-08-28T19:26:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:33 smithi028 ceph-mon[21769]: pgmap v504: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-08-28T19:26:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:33 smithi115 ceph-mon[22184]: pgmap v504: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-08-28T19:26:33.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.51 MB/sec execute 133 sec latency 24401.388 ms 2024-08-28T19:26:34.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.47 MB/sec execute 134 sec latency 25401.497 ms 2024-08-28T19:26:35.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:26:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:35.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:35 smithi016 ceph-mon[18556]: pgmap v505: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 1.2 MiB/s wr, 65 op/s 2024-08-28T19:26:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:35 smithi028 ceph-mon[21769]: pgmap v505: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 1.2 MiB/s wr, 65 op/s 2024-08-28T19:26:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:35 smithi115 ceph-mon[22184]: pgmap v505: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 1.2 MiB/s wr, 65 op/s 2024-08-28T19:26:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.874+0000 7f6141c4a640 1 -- 172.21.15.16:0/3915042024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f613c10bdc0 msgr2=0x7f613c1125f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.874+0000 7f6141c4a640 1 --2- 172.21.15.16:0/3915042024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f613c10bdc0 0x7f613c1125f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6134009ca0 tx=0x7f613402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:26:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.874+0000 7f6141c4a640 1 -- 172.21.15.16:0/3915042024 shutdown_connections 2024-08-28T19:26:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.874+0000 7f6141c4a640 1 --2- 172.21.15.16:0/3915042024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f613c10bdc0 0x7f613c1125f0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.874+0000 7f6141c4a640 1 --2- 172.21.15.16:0/3915042024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c10b470 0x7f613c10b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.874+0000 7f6141c4a640 1 --2- 172.21.15.16:0/3915042024 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f613c105b30 0x7f613c105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.874+0000 7f6141c4a640 1 -- 172.21.15.16:0/3915042024 >> 172.21.15.16:0/3915042024 conn(0x7f613c101600 msgr2=0x7f613c103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.874+0000 7f6141c4a640 1 -- 172.21.15.16:0/3915042024 shutdown_connections 2024-08-28T19:26:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.874+0000 7f6141c4a640 1 -- 172.21.15.16:0/3915042024 wait complete. 2024-08-28T19:26:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.875+0000 7f6141c4a640 1 Processor -- start 2024-08-28T19:26:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.875+0000 7f6141c4a640 1 -- start start 2024-08-28T19:26:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6141c4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c105b30 0x7f613c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6141c4a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f613c10b470 0x7f613c19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6141c4a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f613c10bdc0 0x7f613c1a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6141c4a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f613c115a80 con 0x7f613c10b470 2024-08-28T19:26:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6141c4a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f613c115900 con 0x7f613c105b30 2024-08-28T19:26:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6141c4a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f613c115c00 con 0x7f613c10bdc0 2024-08-28T19:26:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6132ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f613c10b470 0x7f613c19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6132ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f613c10b470 0x7f613c19cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34332/0 (socket says 172.21.15.16:34332) 2024-08-28T19:26:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6132ffd640 1 -- 172.21.15.16:0/1947278977 learned_addr learned my addr 172.21.15.16:0/1947278977 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f613b7fe640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c105b30 0x7f613c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f613bfff640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f613c10bdc0 0x7f613c1a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6132ffd640 1 -- 172.21.15.16:0/1947278977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f613c10bdc0 msgr2=0x7f613c1a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6132ffd640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f613c10bdc0 0x7f613c1a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:35.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6132ffd640 1 -- 172.21.15.16:0/1947278977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c105b30 msgr2=0x7f613c19c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:35.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6132ffd640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c105b30 0x7f613c19c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:35.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.876+0000 7f6132ffd640 1 -- 172.21.15.16:0/1947278977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c1a2480 con 0x7f613c10b470 2024-08-28T19:26:35.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.877+0000 7f6132ffd640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f613c10b470 0x7f613c19cab0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f612400d740 tx=0x7f612400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:35.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.877+0000 7f61397fa640 1 -- 172.21.15.16:0/1947278977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6124017070 con 0x7f613c10b470 2024-08-28T19:26:35.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.877+0000 7f6141c4a640 1 -- 172.21.15.16:0/1947278977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f613c1a2770 con 0x7f613c10b470 2024-08-28T19:26:35.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.877+0000 7f61397fa640 1 -- 172.21.15.16:0/1947278977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612400de50 con 0x7f613c10b470 2024-08-28T19:26:35.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.877+0000 7f61397fa640 1 -- 172.21.15.16:0/1947278977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612400f620 con 0x7f613c10b470 2024-08-28T19:26:35.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.877+0000 7f6141c4a640 1 -- 172.21.15.16:0/1947278977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f613c1a8ed0 con 0x7f613c10b470 2024-08-28T19:26:35.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.878+0000 7f6141c4a640 1 -- 172.21.15.16:0/1947278977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f613c105f40 con 0x7f613c10b470 2024-08-28T19:26:35.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.881+0000 7f61397fa640 1 -- 172.21.15.16:0/1947278977 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f612401a070 con 0x7f613c10b470 2024-08-28T19:26:35.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.881+0000 7f61397fa640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61280b4ca0 0x7f61280b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:35.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.881+0000 7f613b7fe640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61280b4ca0 0x7f61280b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.881+0000 7f61397fa640 1 -- 172.21.15.16:0/1947278977 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f61240e5130 con 0x7f613c10b470 2024-08-28T19:26:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.882+0000 7f613b7fe640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61280b4ca0 0x7f61280b71b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f612c005fd0 tx=0x7f612c00b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:35.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:35.885+0000 7f61397fa640 1 -- 172.21.15.16:0/1947278977 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f61240ab560 con 0x7f613c10b470 2024-08-28T19:26:35.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.43 MB/sec execute 135 sec latency 26401.541 ms 2024-08-28T19:26:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.136+0000 7f6141c4a640 1 -- 172.21.15.16:0/1947278977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f613c10dc10 con 0x7f613c10b470 2024-08-28T19:26:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.137+0000 7f61397fa640 1 -- 172.21.15.16:0/1947278977 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11979 (secure 0 0 0) 0x7f61240b40f0 con 0x7f613c10b470 2024-08-28T19:26:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:26:36.140 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":41,"btime":"2024-08-28T19:26:17:068335+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":41,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:26:16.960563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:26:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 41 2024-08-28T19:26:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 -- 172.21.15.16:0/1947278977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61280b4ca0 msgr2=0x7f61280b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61280b4ca0 0x7f61280b71b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f612c005fd0 tx=0x7f612c00b390 comp rx=0 tx=0).stop 2024-08-28T19:26:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 -- 172.21.15.16:0/1947278977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f613c10b470 msgr2=0x7f613c19cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f613c10b470 0x7f613c19cab0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f612400d740 tx=0x7f612400dc00 comp rx=0 tx=0).stop 2024-08-28T19:26:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 -- 172.21.15.16:0/1947278977 shutdown_connections 2024-08-28T19:26:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f613c10bdc0 0x7f613c1a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f613c10b470 0x7f613c19cab0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61280b4ca0 0x7f61280b71b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 --2- 172.21.15.16:0/1947278977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c105b30 0x7f613c19c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 -- 172.21.15.16:0/1947278977 >> 172.21.15.16:0/1947278977 conn(0x7f613c101600 msgr2=0x7f613c108a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 -- 172.21.15.16:0/1947278977 shutdown_connections 2024-08-28T19:26:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:36.140+0000 7f6141c4a640 1 -- 172.21.15.16:0/1947278977 wait complete. 2024-08-28T19:26:36.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:36 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1947278977' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:36.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:36 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1947278977' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:36.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:36 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1947278977' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:36.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.39 MB/sec execute 136 sec latency 27401.592 ms 2024-08-28T19:26:37.176 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id bbce1fd4 --release --await' 2024-08-28T19:26:37.177 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id bbce1fd4 --release --await 2024-08-28T19:26:37.431 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:37.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:37 smithi016 ceph-mon[18556]: pgmap v506: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-08-28T19:26:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:37 smithi028 ceph-mon[21769]: pgmap v506: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-08-28T19:26:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:37 smithi115 ceph-mon[22184]: pgmap v506: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-08-28T19:26:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.921+0000 7fc6abb4a640 1 -- 172.21.15.16:0/241169826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a410bde0 msgr2=0x7fc6a4112610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.921+0000 7fc6abb4a640 1 --2- 172.21.15.16:0/241169826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a410bde0 0x7fc6a4112610 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009ca0 tx=0x7fc69c02a8b0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.921+0000 7fc6abb4a640 1 -- 172.21.15.16:0/241169826 shutdown_connections 2024-08-28T19:26:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.921+0000 7fc6abb4a640 1 --2- 172.21.15.16:0/241169826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a410bde0 0x7fc6a4112610 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.921+0000 7fc6abb4a640 1 --2- 172.21.15.16:0/241169826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6a410b490 0x7fc6a410b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.921+0000 7fc6abb4a640 1 --2- 172.21.15.16:0/241169826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6a4105b50 0x7fc6a4105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.921+0000 7fc6abb4a640 1 -- 172.21.15.16:0/241169826 >> 172.21.15.16:0/241169826 conn(0x7fc6a4101580 msgr2=0x7fc6a41039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.921+0000 7fc6abb4a640 1 -- 172.21.15.16:0/241169826 shutdown_connections 2024-08-28T19:26:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.921+0000 7fc6abb4a640 1 -- 172.21.15.16:0/241169826 wait complete. 2024-08-28T19:26:37.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.922+0000 7fc6abb4a640 1 Processor -- start 2024-08-28T19:26:37.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.922+0000 7fc6abb4a640 1 -- start start 2024-08-28T19:26:37.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.923+0000 7fc6abb4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6a4105b50 0x7fc6a410fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.923+0000 7fc6abb4a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6a410b490 0x7fc6a410ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.923+0000 7fc6abb4a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a410bde0 0x7fc6a41104d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.923+0000 7fc6abb4a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a4115aa0 con 0x7fc6a410bde0 2024-08-28T19:26:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.923+0000 7fc6abb4a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a4115920 con 0x7fc6a4105b50 2024-08-28T19:26:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.923+0000 7fc6abb4a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a4115c20 con 0x7fc6a410b490 2024-08-28T19:26:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.923+0000 7fc6a98e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6a4105b50 0x7fc6a410fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.923+0000 7fc6a98e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6a4105b50 0x7fc6a410fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45214/0 (socket says 172.21.15.16:45214) 2024-08-28T19:26:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.923+0000 7fc6a98e6640 1 -- 172.21.15.16:0/3365621108 learned_addr learned my addr 172.21.15.16:0/3365621108 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.924+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a410bde0 0x7fc6a41104d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.924+0000 7fc6a98e6640 1 -- 172.21.15.16:0/3365621108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6a410b490 msgr2=0x7fc6a410ff90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-28T19:26:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.924+0000 7fc6a98e6640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6a410b490 0x7fc6a410ff90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.924+0000 7fc6a98e6640 1 -- 172.21.15.16:0/3365621108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a410bde0 msgr2=0x7fc6a41104d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.924+0000 7fc6a98e6640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a410bde0 0x7fc6a41104d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.924+0000 7fc6a98e6640 1 -- 172.21.15.16:0/3365621108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a41ac8f0 con 0x7fc6a4105b50 2024-08-28T19:26:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.924+0000 7fc6a98e6640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6a4105b50 0x7fc6a410fa50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc69800e9f0 tx=0x7fc69800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.925+0000 7fc696ffd640 1 -- 172.21.15.16:0/3365621108 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69800cd80 con 0x7fc6a4105b50 2024-08-28T19:26:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.925+0000 7fc696ffd640 1 -- 172.21.15.16:0/3365621108 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6980108e0 con 0x7fc6a4105b50 2024-08-28T19:26:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.925+0000 7fc6abb4a640 1 -- 172.21.15.16:0/3365621108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6a41acbe0 con 0x7fc6a4105b50 2024-08-28T19:26:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.925+0000 7fc696ffd640 1 -- 172.21.15.16:0/3365621108 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc698018980 con 0x7fc6a4105b50 2024-08-28T19:26:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.925+0000 7fc6abb4a640 1 -- 172.21.15.16:0/3365621108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6a41ad170 con 0x7fc6a4105b50 2024-08-28T19:26:37.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.928+0000 7fc6b20bf740 1 Processor -- start 2024-08-28T19:26:37.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.929+0000 7fc6b20bf740 1 -- start start 2024-08-28T19:26:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 0x562d80275c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fea7660 0x562d802761c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7feaf840 0x562d80276700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562d801eca70 con 0x562d7feaf840 2024-08-28T19:26:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d801dc000 con 0x562d7fed0520 2024-08-28T19:26:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562d801dffb0 con 0x562d7fea7660 2024-08-28T19:26:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6aa0e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 0x562d80275c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6aa0e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 0x562d80275c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45228/0 (socket says 172.21.15.16:45228) 2024-08-28T19:26:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/2427241222 learned_addr learned my addr 172.21.15.16:0/2427241222 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/2427241222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fea7660 msgr2=0x562d802761c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:26:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/2427241222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fea7660 0x562d802761c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/2427241222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7feaf840 msgr2=0x562d80276700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/2427241222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7feaf840 0x562d80276700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.932+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/2427241222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d7ff3bb00 con 0x562d7fed0520 2024-08-28T19:26:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.933+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/2427241222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 0x562d80275c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc69c005cc0 tx=0x7fc69c009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbafc799b3f64e3a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.933+0000 7fc696ffd640 1 -- 172.21.15.16:0/3365621108 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc698010430 con 0x7fc6a4105b50 2024-08-28T19:26:37.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.934+0000 7fc68ffff640 1 -- 172.21.15.16:0/2427241222 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c0305a0 con 0x562d7fed0520 2024-08-28T19:26:37.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.934+0000 7fc68ffff640 1 -- 172.21.15.16:0/2427241222 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc69c036040 con 0x562d7fed0520 2024-08-28T19:26:37.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.934+0000 7fc696ffd640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6900b4d20 0x7fc6900b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.934+0000 7fc6a98e6640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6900b4d20 0x7fc6900b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.934+0000 7fc68ffff640 1 -- 172.21.15.16:0/2427241222 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c0042c0 con 0x562d7fed0520 2024-08-28T19:26:37.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.935+0000 7fc696ffd640 1 -- 172.21.15.16:0/3365621108 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc6980e5d40 con 0x7fc6a4105b50 2024-08-28T19:26:37.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.935+0000 7fc6b20bf740 1 -- 172.21.15.16:0/2427241222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 msgr2=0x562d80275c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:37.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.935+0000 7fc6a98e6640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6900b4d20 0x7fc6900b7230 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x562d80274900 tx=0x7fc6980abe30 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.935+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/2427241222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 0x562d80275c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc69c005cc0 tx=0x7fc69c009800 comp rx=0 tx=0).stop 2024-08-28T19:26:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.935+0000 7fc6b20bf740 1 -- 172.21.15.16:0/2427241222 shutdown_connections 2024-08-28T19:26:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.935+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/2427241222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7feaf840 0x562d80276700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.935+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/2427241222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fea7660 0x562d802761c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.935+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/2427241222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 0x562d80275c80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.935+0000 7fc6b20bf740 1 -- 172.21.15.16:0/2427241222 >> 172.21.15.16:0/2427241222 conn(0x562d7fec4910 msgr2=0x562d80273b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.935+0000 7fc6b20bf740 1 -- 172.21.15.16:0/2427241222 shutdown_connections 2024-08-28T19:26:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.936+0000 7fc6b20bf740 1 -- 172.21.15.16:0/2427241222 wait complete. 2024-08-28T19:26:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.936+0000 7fc6b20bf740 1 -- start start 2024-08-28T19:26:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.936+0000 7fc6b20bf740 1 Processor -- start 2024-08-28T19:26:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.936+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 0x562d80281870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.936+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 0x562d80281df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.936+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fed0520 0x562d8028d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.936+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562d80284450 con 0x562d7fea7660 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.936+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d8019d520 con 0x562d7feaf840 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.936+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562d802845d0 con 0x562d7fed0520 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6b20bf740 10 client.0 _ll_register_callbacks cb 0x562d801d5c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6a90e5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 0x562d80281df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6aa0e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 0x562d80281870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6aa0e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 0x562d80281870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34380/0 (socket says 172.21.15.16:34380) 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6a90e5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 0x562d80281df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45242/0 (socket says 172.21.15.16:45242) 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/829721496 learned_addr learned my addr 172.21.15.16:0/829721496 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fed0520 0x562d8028d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6a90e5640 1 -- 172.21.15.16:0/829721496 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fed0520 msgr2=0x562d8028d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6a90e5640 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fed0520 0x562d8028d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6a90e5640 1 -- 172.21.15.16:0/829721496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 msgr2=0x562d80281870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6a90e5640 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 0x562d80281870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6a90e5640 1 -- 172.21.15.16:0/829721496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d7ffb2960 con 0x562d7feaf840 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc6a90e5640 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 0x562d80281df0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4104dd0 tx=0x7fc6a0009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1093e70df9fd46b4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.937+0000 7fc68d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.938+0000 7fc68d7fa640 1 -- 172.21.15.16:0/829721496 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0004220 con 0x562d7feaf840 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.938+0000 7fc68d7fa640 1 -- 172.21.15.16:0/829721496 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc6a00043c0 con 0x562d7feaf840 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.938+0000 7fc6b20bf740 1 -- 172.21.15.16:0/829721496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562d8017b9a0 con 0x562d7feaf840 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.938+0000 7fc68d7fa640 1 -- 172.21.15.16:0/829721496 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fc6a0005050 con 0x562d7feaf840 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.938+0000 7fc6b20bf740 10 client.26555 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.938+0000 7fc6b20bf740 1 -- 172.21.15.16:0/829721496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562d802906c0 con 0x562d7feaf840 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.938+0000 7fc68d7fa640 1 -- 172.21.15.16:0/829721496 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562d8017b9a0 con 0x562d7feaf840 2024-08-28T19:26:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.939+0000 7fc68d7fa640 1 -- 172.21.15.16:0/829721496 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7fc6a00035f0 con 0x562d7feaf840 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.939+0000 7fc6b20bf740 10 client.26555 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.939+0000 7fc6b20bf740 10 client.26555 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.939+0000 7fc6b20bf740 20 client.26555 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.939+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d7fe8eea0 0x562d80293b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.939+0000 7fc6b20bf740 4 client.26555 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.939+0000 7fc6b20bf740 1 -- 172.21.15.16:0/829721496 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d801ee9c0 con 0x562d7fe8eea0 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.939+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d7fe8eea0 0x562d80293b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.939+0000 7fc68d7fa640 10 client.26555 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.939+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d7fe8eea0 0x562d80293b90 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=331fef799ec71d5c server_cookie=b2fc7674ed9ae19b in_seq=0 out_seq=0 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc68d7fa640 1 -- 172.21.15.16:0/829721496 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562d801ee9c0 con 0x562d7fe8eea0 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc68d7fa640 10 client.26555 handle_command_reply: tid=0 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc6b20bf740 1 client.26555 shutdown 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc6b20bf740 1 -- 172.21.15.16:0/829721496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 msgr2=0x562d80281df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 0x562d80281df0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4104dd0 tx=0x7fc6a0009f90 comp rx=0 tx=0).stop 2024-08-28T19:26:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc6b20bf740 1 -- 172.21.15.16:0/829721496 shutdown_connections 2024-08-28T19:26:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d7fe8eea0 0x562d80293b90 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fed0520 0x562d8028d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 0x562d80281df0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc68d7fa640 0 client.26555 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/829721496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 0x562d80281870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.940+0000 7fc6b20bf740 1 -- 172.21.15.16:0/829721496 >> 172.21.15.16:0/829721496 conn(0x562d7fec4910 msgr2=0x562d80278e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.941+0000 7fc6b20bf740 1 -- 172.21.15.16:0/829721496 shutdown_connections 2024-08-28T19:26:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.941+0000 7fc6b20bf740 1 -- 172.21.15.16:0/829721496 wait complete. 2024-08-28T19:26:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.941+0000 7fc6b20bf740 20 client.26555 trim_cache size 0 max 16384 2024-08-28T19:26:37.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270033 5.35 MB/sec execute 137 sec latency 28401.636 ms 2024-08-28T19:26:37.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.958+0000 7fc6b20bf740 1 Processor -- start 2024-08-28T19:26:37.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.958+0000 7fc6b20bf740 1 -- start start 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.958+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fea7660 0x562d8027ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.958+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 0x562d80276490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.958+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fed0520 0x562d80273400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.958+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a41b7300 con 0x562d7fed0520 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.958+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a41b8e70 con 0x562d7feaf840 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.958+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a41b79d0 con 0x562d7fea7660 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.958+0000 7fc6aa0e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fed0520 0x562d80273400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.958+0000 7fc6aa0e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fed0520 0x562d80273400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34386/0 (socket says 172.21.15.16:34386) 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.958+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/3782087542 learned_addr learned my addr 172.21.15.16:0/3782087542 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.959+0000 7fc6a90e5640 1 --2- 172.21.15.16:0/3782087542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 0x562d80276490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.959+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/3782087542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fea7660 0x562d8027ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.959+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/3782087542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fea7660 msgr2=0x562d8027ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:37.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.959+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/3782087542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fea7660 0x562d8027ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.959+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/3782087542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 msgr2=0x562d80276490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:37.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.959+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/3782087542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 0x562d80276490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.959+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/3782087542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d80272e10 con 0x562d7fed0520 2024-08-28T19:26:37.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.959+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/3782087542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fed0520 0x562d80273400 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fc69c003de0 tx=0x7fc69c003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db645a0ac51919b6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:37.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.959+0000 7fc68f7fe640 1 -- 172.21.15.16:0/3782087542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c030c80 con 0x562d7fed0520 2024-08-28T19:26:37.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.959+0000 7fc68f7fe640 1 -- 172.21.15.16:0/3782087542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc69c04b040 con 0x562d7fed0520 2024-08-28T19:26:37.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.959+0000 7fc68f7fe640 1 -- 172.21.15.16:0/3782087542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c0316a0 con 0x562d7fed0520 2024-08-28T19:26:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.960+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3782087542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fed0520 msgr2=0x562d80273400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.960+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/3782087542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fed0520 0x562d80273400 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fc69c003de0 tx=0x7fc69c003d70 comp rx=0 tx=0).stop 2024-08-28T19:26:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.960+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3782087542 shutdown_connections 2024-08-28T19:26:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.960+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/3782087542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fed0520 0x562d80273400 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.960+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/3782087542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7feaf840 0x562d80276490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.960+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/3782087542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7fea7660 0x562d8027ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.960+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3782087542 >> 172.21.15.16:0/3782087542 conn(0x562d7fec4910 msgr2=0x562d80278d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.960+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3782087542 shutdown_connections 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.960+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3782087542 wait complete. 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.960+0000 7fc6b20bf740 1 -- start start 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.960+0000 7fc6b20bf740 1 Processor -- start 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 0x562d8027f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7feaf840 0x562d80289850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6b20bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 0x562d80289e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a41b87a0 con 0x562d7fea7660 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a41b80a0 con 0x562d7fed0520 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6b20bf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a41b52e0 con 0x562d7feaf840 2024-08-28T19:26:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6b20bf740 10 client.0 _ll_register_callbacks cb 0x7fc6a0009390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6aa0e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 0x562d8027f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6aa0e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 0x562d8027f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34392/0 (socket says 172.21.15.16:34392) 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/3121354624 learned_addr learned my addr 172.21.15.16:0/3121354624 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6a90e5640 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7feaf840 0x562d80289850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 0x562d80289e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/3121354624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7feaf840 msgr2=0x562d80289850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7feaf840 0x562d80289850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/3121354624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 msgr2=0x562d80289e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 0x562d80289e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.961+0000 7fc6aa0e7640 1 -- 172.21.15.16:0/3121354624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d8027d910 con 0x562d7fea7660 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc6aa0e7640 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 0x562d8027f1b0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc69c030800 tx=0x7fc69c0047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6781afe2c87f7163 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc68d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc68d7fa640 1 -- 172.21.15.16:0/3121354624 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c030c80 con 0x562d7fea7660 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc68d7fa640 1 -- 172.21.15.16:0/3121354624 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc69c003e10 con 0x562d7fea7660 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3121354624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc69c04b040 con 0x562d7fea7660 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc68d7fa640 1 -- 172.21.15.16:0/3121354624 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fc69c068050 con 0x562d7fea7660 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc6b20bf740 10 client.16989 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3121354624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562d802ff600 con 0x562d7fea7660 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc68d7fa640 1 -- 172.21.15.16:0/3121354624 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc69c04b040 con 0x562d7fea7660 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc68d7fa640 1 -- 172.21.15.16:0/3121354624 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7fc69c031a80 con 0x562d7fea7660 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc6b20bf740 10 client.16989 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.962+0000 7fc6b20bf740 10 client.16989 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.963+0000 7fc6b20bf740 20 client.16989 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.963+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d7fe64250 0x562d803018e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.963+0000 7fc6b20bf740 4 client.16989 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "bbce1fd4", "release": true, "await": true}] 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.963+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3121354624 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "bbce1fd4", "release": true, "await": true}) -- 0x562d802799a0 con 0x562d7fe64250 2024-08-28T19:26:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.963+0000 7fc6a90e5640 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d7fe64250 0x562d803018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.963+0000 7fc68d7fa640 10 client.16989 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.963+0000 7fc6a90e5640 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d7fe64250 0x562d803018e0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b27806f5024eeb5 server_cookie=af56e80ba7dfb6de in_seq=0 out_seq=0 2024-08-28T19:26:37.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.973+0000 7fc68d7fa640 1 -- 172.21.15.16:0/3121354624 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x562d802799a0 con 0x562d7fe64250 2024-08-28T19:26:37.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.973+0000 7fc68d7fa640 10 client.16989 handle_command_reply: tid=0 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.973+0000 7fc6b20bf740 1 client.16989 shutdown 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3121354624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 msgr2=0x562d8027f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 0x562d8027f1b0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc69c030800 tx=0x7fc69c0047a0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3121354624 shutdown_connections 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d7fed0520 0x562d80289e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d7fe64250 0x562d803018e0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7feaf840 0x562d80289850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6b20bf740 1 --2- 172.21.15.16:0/3121354624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d7fea7660 0x562d8027f1b0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc68d7fa640 0 client.16989 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3121354624 >> 172.21.15.16:0/3121354624 conn(0x562d7fec4910 msgr2=0x562d80273400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3121354624 shutdown_connections 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6b20bf740 1 -- 172.21.15.16:0/3121354624 wait complete. 2024-08-28T19:26:37.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6b20bf740 20 client.16989 trim_cache size 0 max 16384 2024-08-28T19:26:37.977 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:26:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 42, 2024-08-28T19:26:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:26:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 16, 2024-08-28T19:26:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:26:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "bbce1fd4": { 2024-08-28T19:26:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 16, 2024-08-28T19:26:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:26:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:26:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:26:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:37.978 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:26:37.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6abb4a640 1 -- 172.21.15.16:0/3365621108 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6900b4d20 msgr2=0x7fc6900b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:37.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6abb4a640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6900b4d20 0x7fc6900b7230 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x562d80274900 tx=0x7fc6980abe30 comp rx=0 tx=0).stop 2024-08-28T19:26:37.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6abb4a640 1 -- 172.21.15.16:0/3365621108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6a4105b50 msgr2=0x7fc6a410fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:37.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.974+0000 7fc6abb4a640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6a4105b50 0x7fc6a410fa50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc69800e9f0 tx=0x7fc69800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.975+0000 7fc6abb4a640 1 -- 172.21.15.16:0/3365621108 shutdown_connections 2024-08-28T19:26:37.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.975+0000 7fc6abb4a640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a410bde0 0x7fc6a41104d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.975+0000 7fc6abb4a640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6a410b490 0x7fc6a410ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.975+0000 7fc6abb4a640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6900b4d20 0x7fc6900b7230 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.975+0000 7fc6abb4a640 1 --2- 172.21.15.16:0/3365621108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6a4105b50 0x7fc6a410fa50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:37.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.975+0000 7fc6abb4a640 1 -- 172.21.15.16:0/3365621108 >> 172.21.15.16:0/3365621108 conn(0x7fc6a4101580 msgr2=0x7fc6a410aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:37.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.975+0000 7fc6abb4a640 1 -- 172.21.15.16:0/3365621108 shutdown_connections 2024-08-28T19:26:37.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:37.975+0000 7fc6abb4a640 1 -- 172.21.15.16:0/3365621108 wait complete. 2024-08-28T19:26:38.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:38 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:26:38.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:38 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:26:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:38 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:26:38.763 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'bbce1fd4', total seconds elapsed: 33.6 2024-08-28T19:26:38.763 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-08-28T19:26:38.764 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:26:38.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 272040 5.36 MB/sec execute 138 sec latency 28428.388 ms 2024-08-28T19:26:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:39 smithi016 ceph-mon[18556]: pgmap v507: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.2 MiB/s wr, 64 op/s 2024-08-28T19:26:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:39 smithi028 ceph-mon[21769]: pgmap v507: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.2 MiB/s wr, 64 op/s 2024-08-28T19:26:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.606+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2746941895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf410dbf0 msgr2=0x7fdbf41100e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.606+0000 7fdbf94c3640 1 --2- 172.21.15.16:0/2746941895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf410dbf0 0x7fdbf41100e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009ca0 tx=0x7fdbec02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:26:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.606+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2746941895 shutdown_connections 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.606+0000 7fdbf94c3640 1 --2- 172.21.15.16:0/2746941895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf410dbf0 0x7fdbf41100e0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.606+0000 7fdbf94c3640 1 --2- 172.21.15.16:0/2746941895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf41050e0 0x7fdbf410d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.606+0000 7fdbf94c3640 1 --2- 172.21.15.16:0/2746941895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbf4104790 0x7fdbf4104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.606+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2746941895 >> 172.21.15.16:0/2746941895 conn(0x7fdbf4079580 msgr2=0x7fdbf4079990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.606+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2746941895 shutdown_connections 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.606+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2746941895 wait complete. 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.607+0000 7fdbf94c3640 1 Processor -- start 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.607+0000 7fdbf94c3640 1 -- start start 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.607+0000 7fdbf94c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf4104790 0x7fdbf419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.607+0000 7fdbf94c3640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbf41050e0 0x7fdbf419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.607+0000 7fdbf94c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf410dbf0 0x7fdbf419fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.607+0000 7fdbf94c3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf41138a0 con 0x7fdbf410dbf0 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.607+0000 7fdbf94c3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf4113720 con 0x7fdbf4104790 2024-08-28T19:26:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.607+0000 7fdbf94c3640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf4113a20 con 0x7fdbf41050e0 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf410dbf0 0x7fdbf419fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf4104790 0x7fdbf419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf410dbf0 0x7fdbf419fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34404/0 (socket says 172.21.15.16:34404) 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf27fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbf41050e0 0x7fdbf419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf37fe640 1 -- 172.21.15.16:0/2766883002 learned_addr learned my addr 172.21.15.16:0/2766883002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf27fc640 1 -- 172.21.15.16:0/2766883002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf4104790 msgr2=0x7fdbf419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf27fc640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf4104790 0x7fdbf419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf27fc640 1 -- 172.21.15.16:0/2766883002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf410dbf0 msgr2=0x7fdbf419fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf27fc640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf410dbf0 0x7fdbf419fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf27fc640 1 -- 172.21.15.16:0/2766883002 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf41a0200 con 0x7fdbf41050e0 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf37fe640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf410dbf0 0x7fdbf419fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbf27fc640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbf41050e0 0x7fdbf419a830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdbf406aad0 tx=0x7fdbe400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.608+0000 7fdbebfff640 1 -- 172.21.15.16:0/2766883002 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe400cc40 con 0x7fdbf41050e0 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.609+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2766883002 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbf41a04f0 con 0x7fdbf41050e0 2024-08-28T19:26:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.609+0000 7fdbebfff640 1 -- 172.21.15.16:0/2766883002 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbe400cde0 con 0x7fdbf41050e0 2024-08-28T19:26:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.609+0000 7fdbebfff640 1 -- 172.21.15.16:0/2766883002 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4018760 con 0x7fdbf41050e0 2024-08-28T19:26:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.609+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2766883002 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbf41a6c50 con 0x7fdbf41050e0 2024-08-28T19:26:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.610+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2766883002 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbf4104ba0 con 0x7fdbf41050e0 2024-08-28T19:26:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.611+0000 7fdbebfff640 1 -- 172.21.15.16:0/2766883002 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fdbe4010a90 con 0x7fdbf41050e0 2024-08-28T19:26:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.612+0000 7fdbebfff640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbe00b4ca0 0x7fdbe00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.612+0000 7fdbebfff640 1 -- 172.21.15.16:0/2766883002 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fdbe40e5fe0 con 0x7fdbf41050e0 2024-08-28T19:26:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.613+0000 7fdbf2ffd640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbe00b4ca0 0x7fdbe00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:39.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.613+0000 7fdbf2ffd640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbe00b4ca0 0x7fdbe00b71b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdbf406c3c0 tx=0x7fdbdc005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:39.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.615+0000 7fdbebfff640 1 -- 172.21.15.16:0/2766883002 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fdbe40ac3b0 con 0x7fdbf41050e0 2024-08-28T19:26:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:39 smithi115 ceph-mon[22184]: pgmap v507: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.2 MiB/s wr, 64 op/s 2024-08-28T19:26:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.867+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2766883002 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbf419b3c0 con 0x7fdbf41050e0 2024-08-28T19:26:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:26:39.872 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":42,"btime":"2024-08-28T19:26:37:182944+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:26:36.587538+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:26:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.868+0000 7fdbebfff640 1 -- 172.21.15.16:0/2766883002 <== mon.1 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11978 (secure 0 0 0) 0x7fdbf419b3c0 con 0x7fdbf41050e0 2024-08-28T19:26:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 42 2024-08-28T19:26:39.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.873+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2766883002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbe00b4ca0 msgr2=0x7fdbe00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:39.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.873+0000 7fdbf94c3640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbe00b4ca0 0x7fdbe00b71b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdbf406c3c0 tx=0x7fdbdc005f90 comp rx=0 tx=0).stop 2024-08-28T19:26:39.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.873+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2766883002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbf41050e0 msgr2=0x7fdbf419a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:39.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.873+0000 7fdbf94c3640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbf41050e0 0x7fdbf419a830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdbf406aad0 tx=0x7fdbe400ee50 comp rx=0 tx=0).stop 2024-08-28T19:26:39.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.874+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2766883002 shutdown_connections 2024-08-28T19:26:39.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.875+0000 7fdbf94c3640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf410dbf0 0x7fdbf419fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:39.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.875+0000 7fdbf94c3640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdbf41050e0 0x7fdbf419a830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:39.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.875+0000 7fdbf94c3640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdbe00b4ca0 0x7fdbe00b71b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:39.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.875+0000 7fdbf94c3640 1 --2- 172.21.15.16:0/2766883002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf4104790 0x7fdbf419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:39.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.875+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2766883002 >> 172.21.15.16:0/2766883002 conn(0x7fdbf4079580 msgr2=0x7fdbf40ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:39.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.875+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2766883002 shutdown_connections 2024-08-28T19:26:39.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:39.875+0000 7fdbf94c3640 1 -- 172.21.15.16:0/2766883002 wait complete. 2024-08-28T19:26:39.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 273523 5.41 MB/sec execute 139 sec latency 26.730 ms 2024-08-28T19:26:39.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:26:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:26:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:40 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2766883002' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:26:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:26:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:26:40.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:26:40.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:40 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2766883002' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:40.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:26:40.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:26:40.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:26:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:26:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:40 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2766883002' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:26:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:26:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:26:40.647 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-08-28T19:26:40.648 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-08-28T19:26:40.767 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:40.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 275506 5.42 MB/sec execute 140 sec latency 12.372 ms 2024-08-28T19:26:41.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.246+0000 7f7fca9fd640 1 -- 172.21.15.16:0/1053645320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fc4102840 msgr2=0x7f7fc4102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:41.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.246+0000 7f7fca9fd640 1 --2- 172.21.15.16:0/1053645320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fc4102840 0x7f7fc4102c50 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc009900 tx=0x7f7fbc02a410 comp rx=0 tx=0).stop 2024-08-28T19:26:41.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.246+0000 7f7fca9fd640 1 -- 172.21.15.16:0/1053645320 shutdown_connections 2024-08-28T19:26:41.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.246+0000 7f7fca9fd640 1 --2- 172.21.15.16:0/1053645320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fc4108ad0 0x7f7fc410f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.246+0000 7f7fca9fd640 1 --2- 172.21.15.16:0/1053645320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc4108180 0x7f7fc4108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.246+0000 7f7fca9fd640 1 --2- 172.21.15.16:0/1053645320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fc4102840 0x7f7fc4102c50 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.246+0000 7f7fca9fd640 1 -- 172.21.15.16:0/1053645320 >> 172.21.15.16:0/1053645320 conn(0x7f7fc40794f0 msgr2=0x7f7fc4079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:41.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.246+0000 7f7fca9fd640 1 -- 172.21.15.16:0/1053645320 shutdown_connections 2024-08-28T19:26:41.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.246+0000 7f7fca9fd640 1 -- 172.21.15.16:0/1053645320 wait complete. 2024-08-28T19:26:41.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.247+0000 7f7fca9fd640 1 Processor -- start 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.247+0000 7f7fca9fd640 1 -- start start 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.247+0000 7f7fca9fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc4102840 0x7f7fc41991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.247+0000 7f7fca9fd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fc4108180 0x7f7fc4199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.247+0000 7f7fca9fd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fc4108ad0 0x7f7fc419ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.247+0000 7f7fca9fd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc41127a0 con 0x7f7fc4108180 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.247+0000 7f7fca9fd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc4112620 con 0x7f7fc4102840 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.247+0000 7f7fca9fd640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc4112920 con 0x7f7fc4108ad0 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.247+0000 7f7fc37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fc4108180 0x7f7fc4199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.247+0000 7f7fc37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fc4108180 0x7f7fc4199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34416/0 (socket says 172.21.15.16:34416) 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc37fe640 1 -- 172.21.15.16:0/2547410278 learned_addr learned my addr 172.21.15.16:0/2547410278 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc4102840 0x7f7fc41991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc8f9a640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fc4108ad0 0x7f7fc419ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc37fe640 1 -- 172.21.15.16:0/2547410278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fc4108ad0 msgr2=0x7f7fc419ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fc4108ad0 0x7f7fc419ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc37fe640 1 -- 172.21.15.16:0/2547410278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc4102840 msgr2=0x7f7fc41991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc4102840 0x7f7fc41991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc37fe640 1 -- 172.21.15.16:0/2547410278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc419f160 con 0x7f7fc4108180 2024-08-28T19:26:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc4102840 0x7f7fc41991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fc4108180 0x7f7fc4199790 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00e9f0 tx=0x7f7fac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc17fa640 1 -- 172.21.15.16:0/2547410278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac00ccc0 con 0x7f7fc4108180 2024-08-28T19:26:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fca9fd640 1 -- 172.21.15.16:0/2547410278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc419f450 con 0x7f7fc4108180 2024-08-28T19:26:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fca9fd640 1 -- 172.21.15.16:0/2547410278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc406b8e0 con 0x7f7fc4108180 2024-08-28T19:26:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.248+0000 7f7fc17fa640 1 -- 172.21.15.16:0/2547410278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fac0108e0 con 0x7f7fc4108180 2024-08-28T19:26:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.249+0000 7f7fc17fa640 1 -- 172.21.15.16:0/2547410278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac018860 con 0x7f7fc4108180 2024-08-28T19:26:41.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.251+0000 7f7fd0f72740 1 Processor -- start 2024-08-28T19:26:41.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.251+0000 7f7fd0f72740 1 -- start start 2024-08-28T19:26:41.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 0x55fb38d3d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d3d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38977390 0x55fb38d3dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fb38cb4630 con 0x55fb38998070 2024-08-28T19:26:41.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb38ca3bc0 con 0x55fb38977390 2024-08-28T19:26:41.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55fb38ca7b70 con 0x55fb3896f1b0 2024-08-28T19:26:41.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fc8f9a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 0x55fb38d3d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fc17fa640 1 -- 172.21.15.16:0/2547410278 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7fac010430 con 0x7f7fc4108180 2024-08-28T19:26:41.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fc8f9a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 0x55fb38d3d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34418/0 (socket says 172.21.15.16:34418) 2024-08-28T19:26:41.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fc8f9a640 1 -- 172.21.15.16:0/1561531074 learned_addr learned my addr 172.21.15.16:0/1561531074 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:41.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/1561531074 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d3d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/1561531074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38977390 0x55fb38d3dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fc8f9a640 1 -- 172.21.15.16:0/1561531074 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 msgr2=0x55fb38d3d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:41.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fc8f9a640 1 --2- 172.21.15.16:0/1561531074 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d3d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.252+0000 7f7fc8f9a640 1 -- 172.21.15.16:0/1561531074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38977390 msgr2=0x55fb38d3dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.253+0000 7f7fc8f9a640 1 --2- 172.21.15.16:0/1561531074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38977390 0x55fb38d3dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.253+0000 7f7fc8f9a640 1 -- 172.21.15.16:0/1561531074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb38a03650 con 0x55fb38998070 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.253+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/1561531074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38977390 0x55fb38d3dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.253+0000 7f7fc8f9a640 1 --2- 172.21.15.16:0/1561531074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 0x55fb38d3d2c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009f50 tx=0x7f7fb4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45809318e2715c78 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.253+0000 7f7fba7fc640 1 -- 172.21.15.16:0/1561531074 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb401b020 con 0x55fb38998070 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.253+0000 7f7fba7fc640 1 -- 172.21.15.16:0/1561531074 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb4003e20 con 0x55fb38998070 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.253+0000 7f7fba7fc640 1 -- 172.21.15.16:0/1561531074 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4005020 con 0x55fb38998070 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.253+0000 7f7fc17fa640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7fb00b4d20 0x7f7fb00b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.253+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7fb00b4d20 0x7f7fb00b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.254+0000 7f7fc17fa640 1 -- 172.21.15.16:0/2547410278 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7fac0e5990 con 0x7f7fc4108180 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.254+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1561531074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 msgr2=0x55fb38d3d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:41.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.254+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7fb00b4d20 0x7f7fb00b7230 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc031440 tx=0x7f7fbc032ed0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:41.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.254+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/1561531074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 0x55fb38d3d2c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009f50 tx=0x7f7fb4009500 comp rx=0 tx=0).stop 2024-08-28T19:26:41.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.254+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1561531074 shutdown_connections 2024-08-28T19:26:41.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.254+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/1561531074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38977390 0x55fb38d3dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.254+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/1561531074 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d3d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.254+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/1561531074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 0x55fb38d3d2c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.254+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1561531074 >> 172.21.15.16:0/1561531074 conn(0x55fb3898c460 msgr2=0x55fb38d3b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:41.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.254+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1561531074 shutdown_connections 2024-08-28T19:26:41.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.254+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1561531074 wait complete. 2024-08-28T19:26:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.255+0000 7f7fd0f72740 1 -- start start 2024-08-28T19:26:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.255+0000 7f7fd0f72740 1 Processor -- start 2024-08-28T19:26:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.255+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb3896f1b0 0x55fb38d4cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.255+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb38977390 0x55fb38d4d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.255+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 0x55fb38d59190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.255+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fb38b06700 con 0x55fb38998070 2024-08-28T19:26:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.255+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb38c2a700 con 0x55fb3896f1b0 2024-08-28T19:26:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.255+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55fb38d4fe00 con 0x55fb38977390 2024-08-28T19:26:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.255+0000 7f7fd0f72740 10 client.0 _ll_register_callbacks cb 0x55fb38c59dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc37fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb38977390 0x55fb38d4d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb3896f1b0 0x55fb38d4cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc8f9a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 0x55fb38d59190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb3896f1b0 0x55fb38d4cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45302/0 (socket says 172.21.15.16:45302) 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc3fff640 1 -- 172.21.15.16:0/1666695627 learned_addr learned my addr 172.21.15.16:0/1666695627 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc37fe640 1 -- 172.21.15.16:0/1666695627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb3896f1b0 msgr2=0x55fb38d4cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb3896f1b0 0x55fb38d4cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc37fe640 1 -- 172.21.15.16:0/1666695627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 msgr2=0x55fb38d59190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 0x55fb38d59190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc37fe640 1 -- 172.21.15.16:0/1666695627 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb38c638b0 con 0x55fb38977390 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc8f9a640 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 0x55fb38d59190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb3896f1b0 0x55fb38d4cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb38977390 0x55fb38d4d2c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0e2d10 tx=0x7f7fac0abbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa39e401d46b68bb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fa37fe640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fa37fe640 1 -- 172.21.15.16:0/1666695627 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac010720 con 0x55fb38977390 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fa37fe640 1 -- 172.21.15.16:0/1666695627 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7fac091b50 con 0x55fb38977390 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.256+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1666695627 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fb4003e20 con 0x55fb38977390 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fa37fe640 1 -- 172.21.15.16:0/1666695627 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7fac0ab870 con 0x55fb38977390 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fd0f72740 10 client.26395 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1666695627 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55fb38d5bb70 con 0x55fb38977390 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fa37fe640 1 -- 172.21.15.16:0/1666695627 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb4003e20 con 0x55fb38977390 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fa37fe640 1 -- 172.21.15.16:0/1666695627 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f7fac0a76d0 con 0x55fb38977390 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fd0f72740 10 client.26395 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fd0f72740 10 client.26395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fd0f72740 20 client.26395 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb389569f0 0x55fb38d5f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fd0f72740 4 client.26395 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1666695627 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb38cb6580 con 0x55fb389569f0 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.257+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb389569f0 0x55fb38d5f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.258+0000 7f7fa37fe640 10 client.26395 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.258+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb389569f0 0x55fb38d5f060 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b446822430fd1229 server_cookie=7c2668328a6dbdb0 in_seq=0 out_seq=0 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fa37fe640 1 -- 172.21.15.16:0/1666695627 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55fb38cb6580 con 0x55fb389569f0 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fa37fe640 10 client.26395 handle_command_reply: tid=0 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 1 client.26395 shutdown 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1666695627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb38977390 msgr2=0x55fb38d4d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb38977390 0x55fb38d4d2c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0e2d10 tx=0x7f7fac0abbf0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1666695627 shutdown_connections 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38998070 0x55fb38d59190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb389569f0 0x55fb38d5f060 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb38977390 0x55fb38d4d2c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/1666695627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb3896f1b0 0x55fb38d4cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1666695627 >> 172.21.15.16:0/1666695627 conn(0x55fb3898c460 msgr2=0x55fb38d3ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fa37fe640 0 client.26395 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1666695627 shutdown_connections 2024-08-28T19:26:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 1 -- 172.21.15.16:0/1666695627 wait complete. 2024-08-28T19:26:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.259+0000 7f7fd0f72740 20 client.26395 trim_cache size 0 max 16384 2024-08-28T19:26:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.288+0000 7f7fd0f72740 1 Processor -- start 2024-08-28T19:26:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.288+0000 7f7fd0f72740 1 -- start start 2024-08-28T19:26:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d59090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d4b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38998070 0x55fb38d4bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa400fec0 con 0x55fb38977390 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa4011a30 con 0x55fb38998070 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa4010590 con 0x55fb3896f1b0 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fc37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d4b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fc37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d4b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34440/0 (socket says 172.21.15.16:34440) 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fc37fe640 1 -- 172.21.15.16:0/159692419 learned_addr learned my addr 172.21.15.16:0/159692419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/159692419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d59090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fc8f9a640 1 --2- 172.21.15.16:0/159692419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38998070 0x55fb38d4bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fc37fe640 1 -- 172.21.15.16:0/159692419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 msgr2=0x55fb38d59090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/159692419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d59090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fc37fe640 1 -- 172.21.15.16:0/159692419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38998070 msgr2=0x55fb38d4bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/159692419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38998070 0x55fb38d4bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fc37fe640 1 -- 172.21.15.16:0/159692419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb38d352f0 con 0x55fb38977390 2024-08-28T19:26:41.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.289+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/159692419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d59090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:41.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.290+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/159692419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d4b600 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0aba10 tx=0x7f7fac0abc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20cd9ad7b1abadbd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:41.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.290+0000 7f7fb9ffb640 1 -- 172.21.15.16:0/159692419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac0ab4d0 con 0x55fb38977390 2024-08-28T19:26:41.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.290+0000 7f7fb9ffb640 1 -- 172.21.15.16:0/159692419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fac010720 con 0x55fb38977390 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.290+0000 7f7fb9ffb640 1 -- 172.21.15.16:0/159692419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac0aaa30 con 0x55fb38977390 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.291+0000 7f7fd0f72740 1 -- 172.21.15.16:0/159692419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 msgr2=0x55fb38d4b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.291+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/159692419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d4b600 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0aba10 tx=0x7f7fac0abc60 comp rx=0 tx=0).stop 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.291+0000 7f7fd0f72740 1 -- 172.21.15.16:0/159692419 shutdown_connections 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.291+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/159692419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38998070 0x55fb38d4bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.291+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/159692419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d4b600 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.291+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/159692419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d59090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.291+0000 7f7fd0f72740 1 -- 172.21.15.16:0/159692419 >> 172.21.15.16:0/159692419 conn(0x55fb3898c460 msgr2=0x55fb38d47ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.291+0000 7f7fd0f72740 1 -- 172.21.15.16:0/159692419 shutdown_connections 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.291+0000 7f7fd0f72740 1 -- 172.21.15.16:0/159692419 wait complete. 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.291+0000 7f7fd0f72740 1 -- start start 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fd0f72740 1 Processor -- start 2024-08-28T19:26:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d50c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fd0f72740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38998070 0x55fb38d511b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa4011360 con 0x55fb38977390 2024-08-28T19:26:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa4010c60 con 0x55fb38998070 2024-08-28T19:26:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fd0f72740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa400dea0 con 0x55fb3896f1b0 2024-08-28T19:26:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fd0f72740 10 client.0 _ll_register_callbacks cb 0x7f7fac0eb240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fc37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d50c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fc37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d50c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34444/0 (socket says 172.21.15.16:34444) 2024-08-28T19:26:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fc37fe640 1 -- 172.21.15.16:0/2527518063 learned_addr learned my addr 172.21.15.16:0/2527518063 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.292+0000 7f7fc8f9a640 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38998070 0x55fb38d511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fc37fe640 1 -- 172.21.15.16:0/2527518063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 msgr2=0x55fb38d557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fc37fe640 1 -- 172.21.15.16:0/2527518063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38998070 msgr2=0x55fb38d511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38998070 0x55fb38d511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fc37fe640 1 -- 172.21.15.16:0/2527518063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb38d51820 con 0x55fb38977390 2024-08-28T19:26:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fc37fe640 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d50c70 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0a8fd0 tx=0x7f7fac0abbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6095bc568c2251 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fa37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fa37fe640 1 -- 172.21.15.16:0/2527518063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac0ab4d0 con 0x55fb38977390 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fa37fe640 1 -- 172.21.15.16:0/2527518063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f7fac0ab1d0 con 0x55fb38977390 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fd0f72740 1 -- 172.21.15.16:0/2527518063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fb38cb2020 con 0x55fb38977390 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fa37fe640 1 -- 172.21.15.16:0/2527518063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7fac0abd60 con 0x55fb38977390 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fd0f72740 10 client.17013 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.293+0000 7f7fd0f72740 1 -- 172.21.15.16:0/2527518063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55fb38dc9840 con 0x55fb38977390 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.294+0000 7f7fa37fe640 1 -- 172.21.15.16:0/2527518063 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55fb38cb2020 con 0x55fb38977390 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.294+0000 7f7fa37fe640 1 -- 172.21.15.16:0/2527518063 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f7fac0b16b0 con 0x55fb38977390 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.294+0000 7f7fd0f72740 10 client.17013 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.294+0000 7f7fd0f72740 10 client.17013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.294+0000 7f7fd0f72740 20 client.17013 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.294+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb3892bda0 0x55fb38dcbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.294+0000 7f7fd0f72740 4 client.17013 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.294+0000 7f7fd0f72740 1 -- 172.21.15.16:0/2527518063 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fb38caf760 con 0x55fb3892bda0 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.294+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb3892bda0 0x55fb38dcbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fa37fe640 10 client.17013 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fc3fff640 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb3892bda0 0x55fb38dcbba0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dbd54693aed25a0 server_cookie=4643e5a20e239580 in_seq=0 out_seq=0 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fa37fe640 1 -- 172.21.15.16:0/2527518063 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2362 (crc 0 0 0) 0x55fb38caf760 con 0x55fb3892bda0 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fa37fe640 10 client.17013 handle_command_reply: tid=0 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fd0f72740 1 client.17013 shutdown 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fd0f72740 1 -- 172.21.15.16:0/2527518063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 msgr2=0x55fb38d50c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d50c70 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0a8fd0 tx=0x7f7fac0abbf0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fd0f72740 1 -- 172.21.15.16:0/2527518063 shutdown_connections 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb38998070 0x55fb38d511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb38977390 0x55fb38d50c70 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb3892bda0 0x55fb38dcbba0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fd0f72740 1 --2- 172.21.15.16:0/2527518063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb3896f1b0 0x55fb38d557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fd0f72740 1 -- 172.21.15.16:0/2527518063 >> 172.21.15.16:0/2527518063 conn(0x55fb3898c460 msgr2=0x55fb38d4b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.295+0000 7f7fa37fe640 0 client.17013 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.296+0000 7f7fd0f72740 1 -- 172.21.15.16:0/2527518063 shutdown_connections 2024-08-28T19:26:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.296+0000 7f7fd0f72740 1 -- 172.21.15.16:0/2527518063 wait complete. 2024-08-28T19:26:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.296+0000 7f7fd0f72740 20 client.17013 trim_cache size 0 max 16384 2024-08-28T19:26:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:41.301 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:26:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:41.306 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:26:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.296+0000 7f7fca9fd640 1 -- 172.21.15.16:0/2547410278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7fb00b4d20 msgr2=0x7f7fb00b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.296+0000 7f7fca9fd640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7fb00b4d20 0x7f7fb00b7230 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc031440 tx=0x7f7fbc032ed0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.296+0000 7f7fca9fd640 1 -- 172.21.15.16:0/2547410278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fc4108180 msgr2=0x7f7fc4199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.296+0000 7f7fca9fd640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fc4108180 0x7f7fc4199790 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00e9f0 tx=0x7f7fac00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.297+0000 7f7fca9fd640 1 -- 172.21.15.16:0/2547410278 shutdown_connections 2024-08-28T19:26:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.297+0000 7f7fca9fd640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fc4108ad0 0x7f7fc419ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.297+0000 7f7fca9fd640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fc4108180 0x7f7fc4199790 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.297+0000 7f7fca9fd640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7fb00b4d20 0x7f7fb00b7230 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.297+0000 7f7fca9fd640 1 --2- 172.21.15.16:0/2547410278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc4102840 0x7f7fc41991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.297+0000 7f7fca9fd640 1 -- 172.21.15.16:0/2547410278 >> 172.21.15.16:0/2547410278 conn(0x7f7fc40794f0 msgr2=0x7f7fc41062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.297+0000 7f7fca9fd640 1 -- 172.21.15.16:0/2547410278 shutdown_connections 2024-08-28T19:26:41.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:41.297+0000 7f7fca9fd640 1 -- 172.21.15.16:0/2547410278 wait complete. 2024-08-28T19:26:41.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:41 smithi016 ceph-mon[18556]: pgmap v508: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 1.1 MiB/s wr, 68 op/s 2024-08-28T19:26:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:41 smithi028 ceph-mon[21769]: pgmap v508: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 1.1 MiB/s wr, 68 op/s 2024-08-28T19:26:41.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:41 smithi115 ceph-mon[22184]: pgmap v508: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 1.1 MiB/s wr, 68 op/s 2024-08-28T19:26:41.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277085 5.47 MB/sec execute 141 sec latency 21.925 ms 2024-08-28T19:26:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.033+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3477913159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ed0108ab0 msgr2=0x7f1ed010f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.033+0000 7f1ed4bc2640 1 --2- 172.21.15.16:0/3477913159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ed0108ab0 0x7f1ed010f2e0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009ca0 tx=0x7f1ec802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.034+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3477913159 shutdown_connections 2024-08-28T19:26:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.034+0000 7f1ed4bc2640 1 --2- 172.21.15.16:0/3477913159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ed0108ab0 0x7f1ed010f2e0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.034+0000 7f1ed4bc2640 1 --2- 172.21.15.16:0/3477913159 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1ed0108160 0x7f1ed0108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.034+0000 7f1ed4bc2640 1 --2- 172.21.15.16:0/3477913159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed0102820 0x7f1ed0102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.034+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3477913159 >> 172.21.15.16:0/3477913159 conn(0x7f1ed0079500 msgr2=0x7f1ed0079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.034+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3477913159 shutdown_connections 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.034+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3477913159 wait complete. 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.034+0000 7f1ed4bc2640 1 Processor -- start 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.034+0000 7f1ed4bc2640 1 -- start start 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ed4bc2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1ed0102820 0x7f1ed019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ed4bc2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ed0108160 0x7f1ed019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ed4bc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed0108ab0 0x7f1ed019fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ed4bc2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed0113830 con 0x7f1ed0108160 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ed4bc2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed01136b0 con 0x7f1ed0108ab0 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ed4bc2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed01139b0 con 0x7f1ed0102820 2024-08-28T19:26:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ec7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ed0108160 0x7f1ed019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ece59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1ed0102820 0x7f1ed019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ec7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ed0108160 0x7f1ed019a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34462/0 (socket says 172.21.15.16:34462) 2024-08-28T19:26:42.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ec7fff640 1 -- 172.21.15.16:0/3829615270 learned_addr learned my addr 172.21.15.16:0/3829615270 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:42.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1eced9d640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed0108ab0 0x7f1ed019fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ece59c640 1 -- 172.21.15.16:0/3829615270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed0108ab0 msgr2=0x7f1ed019fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ece59c640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed0108ab0 0x7f1ed019fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ece59c640 1 -- 172.21.15.16:0/3829615270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ed0108160 msgr2=0x7f1ed019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ece59c640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ed0108160 0x7f1ed019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.035+0000 7f1ece59c640 1 -- 172.21.15.16:0/3829615270 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed01a0130 con 0x7f1ed0102820 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.036+0000 7f1ece59c640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1ed0102820 0x7f1ed019a1c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000e9f0 tx=0x7f1ec000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.036+0000 7f1ec77fe640 1 -- 172.21.15.16:0/3829615270 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec000ccc0 con 0x7f1ed0102820 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.036+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3829615270 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed01a0420 con 0x7f1ed0102820 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.036+0000 7f1ec77fe640 1 -- 172.21.15.16:0/3829615270 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ec00108e0 con 0x7f1ed0102820 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.036+0000 7f1ec77fe640 1 -- 172.21.15.16:0/3829615270 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0018880 con 0x7f1ed0102820 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.036+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3829615270 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed01a6b80 con 0x7f1ed0102820 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.038+0000 7f1edb137740 1 Processor -- start 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.038+0000 7f1edb137740 1 -- start start 2024-08-28T19:26:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1ec77fe640 1 -- 172.21.15.16:0/3829615270 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1ec0010430 con 0x7f1ed0102820 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1edb137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58823840 0x55ea58bcac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1edb137740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58bcb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1edb137740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58802b60 0x55ea58bcb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1edb137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea58b3f330 con 0x55ea58823840 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1edb137740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ea58b2e8c0 con 0x55ea58802b60 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1edb137740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ea58b32870 con 0x55ea587fa980 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1eced9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58823840 0x55ea58bcac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1eced9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58823840 0x55ea58bcac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34468/0 (socket says 172.21.15.16:34468) 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1eced9d640 1 -- 172.21.15.16:0/2712326086 learned_addr learned my addr 172.21.15.16:0/2712326086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1ec7fff640 1 --2- 172.21.15.16:0/2712326086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58802b60 0x55ea58bcb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1ece59c640 1 --2- 172.21.15.16:0/2712326086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58bcb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1eced9d640 1 -- 172.21.15.16:0/2712326086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 msgr2=0x55ea58bcb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1eced9d640 1 --2- 172.21.15.16:0/2712326086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58bcb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1eced9d640 1 -- 172.21.15.16:0/2712326086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58802b60 msgr2=0x55ea58bcb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1eced9d640 1 --2- 172.21.15.16:0/2712326086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58802b60 0x55ea58bcb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1eced9d640 1 -- 172.21.15.16:0/2712326086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea58ae6df0 con 0x55ea58823840 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.039+0000 7f1eced9d640 1 --2- 172.21.15.16:0/2712326086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58823840 0x55ea58bcac90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1ec802f440 tx=0x7f1ec802f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3270d0e9cf84d8ae server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.040+0000 7f1ec77fe640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ebc0b4ca0 0x7f1ebc0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.040+0000 7f1ec4ff9640 1 -- 172.21.15.16:0/2712326086 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8031cf0 con 0x55ea58823840 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.040+0000 7f1ec4ff9640 1 -- 172.21.15.16:0/2712326086 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ec8036070 con 0x55ea58823840 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.040+0000 7f1ec4ff9640 1 -- 172.21.15.16:0/2712326086 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80042a0 con 0x55ea58823840 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.040+0000 7f1ece59c640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ebc0b4ca0 0x7f1ebc0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.040+0000 7f1ec77fe640 1 -- 172.21.15.16:0/3829615270 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1ec00e5bc0 con 0x7f1ed0102820 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 -- 172.21.15.16:0/2712326086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58823840 msgr2=0x55ea58bcac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1ece59c640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ebc0b4ca0 0x7f1ebc0b71b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00e3050 tx=0x7f1ec00e9710 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 --2- 172.21.15.16:0/2712326086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58823840 0x55ea58bcac90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1ec802f440 tx=0x7f1ec802f470 comp rx=0 tx=0).stop 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 -- 172.21.15.16:0/2712326086 shutdown_connections 2024-08-28T19:26:42.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 --2- 172.21.15.16:0/2712326086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58802b60 0x55ea58bcb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 --2- 172.21.15.16:0/2712326086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58bcb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 --2- 172.21.15.16:0/2712326086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58823840 0x55ea58bcac90 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 -- 172.21.15.16:0/2712326086 >> 172.21.15.16:0/2712326086 conn(0x55ea58817990 msgr2=0x55ea58bc8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 -- 172.21.15.16:0/2712326086 shutdown_connections 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 -- 172.21.15.16:0/2712326086 wait complete. 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 -- start start 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 Processor -- start 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea587fa980 0x55ea58bda7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bdad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.041+0000 7f1edb137740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea58823840 0x55ea58be6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1edb137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea589917a0 con 0x55ea58802b60 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1edb137740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ea58ab5240 con 0x55ea587fa980 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1edb137740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ea58bdd5c0 con 0x55ea58823840 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1edb137740 10 client.0 _ll_register_callbacks cb 0x55ea5883b090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1ec7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bdad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1eced9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea587fa980 0x55ea58bda7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1ec7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bdad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34482/0 (socket says 172.21.15.16:34482) 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1ec7fff640 1 -- 172.21.15.16:0/1810774189 learned_addr learned my addr 172.21.15.16:0/1810774189 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1eced9d640 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea58823840 0x55ea58be6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1eced9d640 1 -- 172.21.15.16:0/1810774189 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea58823840 msgr2=0x55ea58be6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1eced9d640 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea58823840 0x55ea58be6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1eced9d640 1 -- 172.21.15.16:0/1810774189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 msgr2=0x55ea58bdad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1eced9d640 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bdad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1eced9d640 1 -- 172.21.15.16:0/1810774189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea58aeb2b0 con 0x55ea587fa980 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1ec7fff640 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bdad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1eced9d640 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea587fa980 0x55ea58bda7e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009bf0 tx=0x7f1ec8003d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f191ab005ea07992 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1eb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:26:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/1810774189 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8031d00 con 0x55ea587fa980 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1edb137740 1 -- 172.21.15.16:0/1810774189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ec8036070 con 0x55ea587fa980 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.042+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/1810774189 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1ec8004b40 con 0x55ea587fa980 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/1810774189 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f1ec805b050 con 0x55ea587fa980 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1edb137740 10 client.26597 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1edb137740 1 -- 172.21.15.16:0/1810774189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55ea5889db90 con 0x55ea587fa980 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/1810774189 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ec8036070 con 0x55ea587fa980 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/1810774189 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f1ec8053020 con 0x55ea587fa980 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1edb137740 10 client.26597 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1edb137740 10 client.26597 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1edb137740 20 client.26597 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1edb137740 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea587e21c0 0x55ea58bec550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1edb137740 4 client.26597 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1edb137740 1 -- 172.21.15.16:0/1810774189 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea58b41280 con 0x55ea587e21c0 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.043+0000 7f1ec7fff640 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea587e21c0 0x55ea58bec550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.044+0000 7f1eb5ffb640 10 client.26597 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.044+0000 7f1ec7fff640 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea587e21c0 0x55ea58bec550 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17f5b000a615bd85 server_cookie=b2b9833897ca67f in_seq=0 out_seq=0 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.044+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/1810774189 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ea58b41280 con 0x55ea587e21c0 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.044+0000 7f1eb5ffb640 10 client.26597 handle_command_reply: tid=0 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.044+0000 7f1edb137740 1 client.26597 shutdown 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.044+0000 7f1edb137740 1 -- 172.21.15.16:0/1810774189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea587fa980 msgr2=0x55ea58bda7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.044+0000 7f1edb137740 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea587fa980 0x55ea58bda7e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009bf0 tx=0x7f1ec8003d10 comp rx=0 tx=0).stop 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.044+0000 7f1edb137740 1 -- 172.21.15.16:0/1810774189 shutdown_connections 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.045+0000 7f1edb137740 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea587e21c0 0x55ea58bec550 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.045+0000 7f1edb137740 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea58823840 0x55ea58be6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.045+0000 7f1edb137740 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bdad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.045+0000 7f1eb5ffb640 0 client.26597 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.045+0000 7f1edb137740 1 --2- 172.21.15.16:0/1810774189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea587fa980 0x55ea58bda7e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.045+0000 7f1edb137740 1 -- 172.21.15.16:0/1810774189 >> 172.21.15.16:0/1810774189 conn(0x55ea58817990 msgr2=0x55ea58bc7340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.045+0000 7f1edb137740 1 -- 172.21.15.16:0/1810774189 shutdown_connections 2024-08-28T19:26:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.045+0000 7f1edb137740 1 -- 172.21.15.16:0/1810774189 wait complete. 2024-08-28T19:26:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.045+0000 7f1edb137740 20 client.26597 trim_cache size 0 max 16384 2024-08-28T19:26:42.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1edb137740 1 Processor -- start 2024-08-28T19:26:42.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1edb137740 1 -- start start 2024-08-28T19:26:42.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1edb137740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58bc7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1edb137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bd79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1edb137740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58823840 0x55ea58bd7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1edb137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed01ae8a0 con 0x55ea58802b60 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1edb137740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed01b0410 con 0x55ea58823840 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1edb137740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed01aef70 con 0x55ea587fa980 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1ec7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bd79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1ec7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bd79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34488/0 (socket says 172.21.15.16:34488) 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1ec7fff640 1 -- 172.21.15.16:0/1926605967 learned_addr learned my addr 172.21.15.16:0/1926605967 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1eced9d640 1 --2- 172.21.15.16:0/1926605967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58bc7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1eced9d640 1 --2- 172.21.15.16:0/1926605967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58823840 0x55ea58bd7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1ec7fff640 1 -- 172.21.15.16:0/1926605967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 msgr2=0x55ea58bc7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1ec7fff640 1 --2- 172.21.15.16:0/1926605967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58bc7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1ec7fff640 1 -- 172.21.15.16:0/1926605967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58823840 msgr2=0x55ea58bd7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1ec7fff640 1 --2- 172.21.15.16:0/1926605967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58823840 0x55ea58bd7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.063+0000 7f1ec7fff640 1 -- 172.21.15.16:0/1926605967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea58afed90 con 0x55ea58802b60 2024-08-28T19:26:42.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.064+0000 7f1ec7fff640 1 --2- 172.21.15.16:0/1926605967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bd79d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x55ea58b2ea10 tx=0x7f1eb8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158113abc4610b6e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.064+0000 7f1eb7fff640 1 -- 172.21.15.16:0/1926605967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8005020 con 0x55ea58802b60 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.064+0000 7f1eb7fff640 1 -- 172.21.15.16:0/1926605967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1eb800f870 con 0x55ea58802b60 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.064+0000 7f1eb7fff640 1 -- 172.21.15.16:0/1926605967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8004a00 con 0x55ea58802b60 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.065+0000 7f1edb137740 1 -- 172.21.15.16:0/1926605967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 msgr2=0x55ea58bd79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.065+0000 7f1edb137740 1 --2- 172.21.15.16:0/1926605967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bd79d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x55ea58b2ea10 tx=0x7f1eb8005f90 comp rx=0 tx=0).stop 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.065+0000 7f1edb137740 1 -- 172.21.15.16:0/1926605967 shutdown_connections 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.065+0000 7f1edb137740 1 --2- 172.21.15.16:0/1926605967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58823840 0x55ea58bd7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.065+0000 7f1edb137740 1 --2- 172.21.15.16:0/1926605967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58bd79d0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.065+0000 7f1edb137740 1 --2- 172.21.15.16:0/1926605967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58bc7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.065+0000 7f1edb137740 1 -- 172.21.15.16:0/1926605967 >> 172.21.15.16:0/1926605967 conn(0x55ea58817990 msgr2=0x55ea58bdac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.065+0000 7f1edb137740 1 -- 172.21.15.16:0/1926605967 shutdown_connections 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.065+0000 7f1edb137740 1 -- 172.21.15.16:0/1926605967 wait complete. 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.065+0000 7f1edb137740 1 -- start start 2024-08-28T19:26:42.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.065+0000 7f1edb137740 1 Processor -- start 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1edb137740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58be2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1edb137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58be3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1edb137740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58823840 0x55ea58bdd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1edb137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed01afd40 con 0x55ea58802b60 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1edb137740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed01af640 con 0x55ea58823840 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1edb137740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed01ac880 con 0x55ea587fa980 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1edb137740 10 client.0 _ll_register_callbacks cb 0x7f1eb8009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1ec7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58be3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1eced9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58be2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1ec7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58be3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34500/0 (socket says 172.21.15.16:34500) 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1ec7fff640 1 -- 172.21.15.16:0/2194919731 learned_addr learned my addr 172.21.15.16:0/2194919731 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1eced9d640 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58be2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:48696/0 (socket says 172.21.15.16:48696) 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1eced9d640 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58823840 0x55ea58bdd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1ec7fff640 1 -- 172.21.15.16:0/2194919731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 msgr2=0x55ea58be2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1ec7fff640 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58be2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1ec7fff640 1 -- 172.21.15.16:0/2194919731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58823840 msgr2=0x55ea58bdd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1ec7fff640 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58823840 0x55ea58bdd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1ec7fff640 1 -- 172.21.15.16:0/2194919731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea58bddf50 con 0x55ea58802b60 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1ec7fff640 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58be3070 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8005b10 tx=0x7f1eb8005b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faaa497218d2a5db server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.066+0000 7f1eb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/2194919731 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8035060 con 0x55ea58802b60 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/2194919731 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1eb800fc20 con 0x55ea58802b60 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1edb137740 1 -- 172.21.15.16:0/2194919731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1eb800f870 con 0x55ea58802b60 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/2194919731 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f1eb803c050 con 0x55ea58802b60 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1edb137740 10 client.17031 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1edb137740 1 -- 172.21.15.16:0/2194919731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55ea58c57680 con 0x55ea58802b60 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/2194919731 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1eb800f870 con 0x55ea58802b60 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/2194919731 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f1eb8010430 con 0x55ea58802b60 2024-08-28T19:26:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1edb137740 10 client.17031 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1edb137740 10 client.17031 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:26:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1edb137740 20 client.17031 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1edb137740 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea587b7570 0x55ea58c59d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1edb137740 4 client.17031 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-08-28T19:26:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1edb137740 1 -- 172.21.15.16:0/2194919731 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x55ea58b3cd20 con 0x55ea587b7570 2024-08-28T19:26:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.067+0000 7f1eced9d640 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea587b7570 0x55ea58c59d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.068+0000 7f1eb5ffb640 10 client.17031 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.068+0000 7f1eced9d640 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea587b7570 0x55ea58c59d80 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd6fb547ac11b271 server_cookie=97d3ad9f350b35e3 in_seq=0 out_seq=0 2024-08-28T19:26:42.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.153+0000 7f1eb5ffb640 1 -- 172.21.15.16:0/2194919731 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ea58b3cd20 con 0x55ea587b7570 2024-08-28T19:26:42.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.153+0000 7f1eb5ffb640 10 client.17031 handle_command_reply: tid=0 2024-08-28T19:26:42.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.153+0000 7f1edb137740 1 client.17031 shutdown 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.153+0000 7f1edb137740 1 -- 172.21.15.16:0/2194919731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 msgr2=0x55ea58be3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.153+0000 7f1edb137740 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58be3070 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8005b10 tx=0x7f1eb8005b40 comp rx=0 tx=0).stop 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.154+0000 7f1edb137740 1 -- 172.21.15.16:0/2194919731 shutdown_connections 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.154+0000 7f1edb137740 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea58823840 0x55ea58bdd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.154+0000 7f1edb137740 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea58802b60 0x55ea58be3070 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.154+0000 7f1edb137740 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea587b7570 0x55ea58c59d80 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.154+0000 7f1edb137740 1 --2- 172.21.15.16:0/2194919731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea587fa980 0x55ea58be2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.154+0000 7f1edb137740 1 -- 172.21.15.16:0/2194919731 >> 172.21.15.16:0/2194919731 conn(0x55ea58817990 msgr2=0x55ea58bdac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.154+0000 7f1eb5ffb640 0 client.17031 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.154+0000 7f1edb137740 1 -- 172.21.15.16:0/2194919731 shutdown_connections 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.154+0000 7f1edb137740 1 -- 172.21.15.16:0/2194919731 wait complete. 2024-08-28T19:26:42.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.154+0000 7f1edb137740 20 client.17031 trim_cache size 0 max 16384 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 42, 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 18, 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: "8e9c1d21": { 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 18, 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:42.157 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:42.158 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3829615270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ebc0b4ca0 msgr2=0x7f1ebc0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ebc0b4ca0 0x7f1ebc0b71b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00e3050 tx=0x7f1ec00e9710 comp rx=0 tx=0).stop 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3829615270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1ed0102820 msgr2=0x7f1ed019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1ed0102820 0x7f1ed019a1c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000e9f0 tx=0x7f1ec000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3829615270 shutdown_connections 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ebc0b4ca0 0x7f1ebc0b71b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed0108ab0 0x7f1ed019fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ed0108160 0x7f1ed019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 --2- 172.21.15.16:0/3829615270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1ed0102820 0x7f1ed019a1c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3829615270 >> 172.21.15.16:0/3829615270 conn(0x7f1ed0079500 msgr2=0x7f1ed0104130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3829615270 shutdown_connections 2024-08-28T19:26:42.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:42.155+0000 7f1ed4bc2640 1 -- 172.21.15.16:0/3829615270 wait complete. 2024-08-28T19:26:42.797 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:26:42.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:26:42.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.43 MB/sec execute 142 sec latency 846.169 ms 2024-08-28T19:26:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:43.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:43 smithi016 ceph-mon[18556]: pgmap v509: 129 pgs: 129 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-08-28T19:26:43.553 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '8e9c1d21', quiesce took 4.8 seconds. Will release after: 25.2 2024-08-28T19:26:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:43 smithi028 ceph-mon[21769]: pgmap v509: 129 pgs: 129 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-08-28T19:26:43.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:43 smithi115 ceph-mon[22184]: pgmap v509: 129 pgs: 129 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-08-28T19:26:43.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.39 MB/sec execute 143 sec latency 1846.220 ms 2024-08-28T19:26:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.985+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1705236471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1f0103900 msgr2=0x7fd1f0103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.985+0000 7fd1f7c5c640 1 --2- 172.21.15.16:0/1705236471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1f0103900 0x7fd1f0103d10 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec009900 tx=0x7fd1ec02a410 comp rx=0 tx=0).stop 2024-08-28T19:26:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.985+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1705236471 shutdown_connections 2024-08-28T19:26:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.985+0000 7fd1f7c5c640 1 --2- 172.21.15.16:0/1705236471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1f0109b90 0x7fd1f01103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.985+0000 7fd1f7c5c640 1 --2- 172.21.15.16:0/1705236471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1f0109240 0x7fd1f0109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.985+0000 7fd1f7c5c640 1 --2- 172.21.15.16:0/1705236471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1f0103900 0x7fd1f0103d10 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.985+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1705236471 >> 172.21.15.16:0/1705236471 conn(0x7fd1f007b5f0 msgr2=0x7fd1f007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.985+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1705236471 shutdown_connections 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.985+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1705236471 wait complete. 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.986+0000 7fd1f7c5c640 1 Processor -- start 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.986+0000 7fd1f7c5c640 1 -- start start 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.986+0000 7fd1f7c5c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1f0103900 0x7fd1f019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.986+0000 7fd1f7c5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1f0109240 0x7fd1f019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.986+0000 7fd1f7c5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1f0109b90 0x7fd1f019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.986+0000 7fd1f7c5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f0113850 con 0x7fd1f0109b90 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.986+0000 7fd1f7c5c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f01136d0 con 0x7fd1f0109240 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.986+0000 7fd1f7c5c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f01139d0 con 0x7fd1f0103900 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f61f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1f0109b90 0x7fd1f019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f61f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1f0109b90 0x7fd1f019fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34520/0 (socket says 172.21.15.16:34520) 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f61f9640 1 -- 172.21.15.16:0/1240098355 learned_addr learned my addr 172.21.15.16:0/1240098355 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f59f8640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1f0103900 0x7fd1f019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f51f7640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1f0109240 0x7fd1f019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f61f9640 1 -- 172.21.15.16:0/1240098355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1f0103900 msgr2=0x7fd1f019a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f61f9640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1f0103900 0x7fd1f019a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f61f9640 1 -- 172.21.15.16:0/1240098355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1f0109240 msgr2=0x7fd1f019a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f61f9640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1f0109240 0x7fd1f019a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f61f9640 1 -- 172.21.15.16:0/1240098355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f01a0190 con 0x7fd1f0109b90 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f61f9640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1f0109b90 0x7fd1f019fa60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd1e800eaa0 tx=0x7fd1e800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1e6ffd640 1 -- 172.21.15.16:0/1240098355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e800cdb0 con 0x7fd1f0109b90 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1e6ffd640 1 -- 172.21.15.16:0/1240098355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1e8005020 con 0x7fd1f0109b90 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1240098355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f01a0480 con 0x7fd1f0109b90 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1e6ffd640 1 -- 172.21.15.16:0/1240098355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e80116d0 con 0x7fd1f0109b90 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.987+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1240098355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f01a6be0 con 0x7fd1f0109b90 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.988+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1240098355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f0103d10 con 0x7fd1f0109b90 2024-08-28T19:26:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.990+0000 7fd1e6ffd640 1 -- 172.21.15.16:0/1240098355 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fd1e80051d0 con 0x7fd1f0109b90 2024-08-28T19:26:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.990+0000 7fd1e6ffd640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1dc0b4d20 0x7fd1dc0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.991+0000 7fd1f59f8640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1dc0b4d20 0x7fd1dc0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.991+0000 7fd1e6ffd640 1 -- 172.21.15.16:0/1240098355 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fd1e80e5d00 con 0x7fd1f0109b90 2024-08-28T19:26:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.991+0000 7fd1f59f8640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1dc0b4d20 0x7fd1dc0b7230 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec031440 tx=0x7fd1ec009560 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:43.994+0000 7fd1e6ffd640 1 -- 172.21.15.16:0/1240098355 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fd1e8016030 con 0x7fd1f0109b90 2024-08-28T19:26:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.217+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1240098355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1f019b350 con 0x7fd1f0109b90 2024-08-28T19:26:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.218+0000 7fd1e6ffd640 1 -- 172.21.15.16:0/1240098355 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11978 (secure 0 0 0) 0x7fd1f019b350 con 0x7fd1f0109b90 2024-08-28T19:26:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:26:44.220 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":42,"btime":"2024-08-28T19:26:37:182944+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:26:36.587538+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:26:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 42 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.220+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1240098355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1dc0b4d20 msgr2=0x7fd1dc0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.220+0000 7fd1f7c5c640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1dc0b4d20 0x7fd1dc0b7230 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec031440 tx=0x7fd1ec009560 comp rx=0 tx=0).stop 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.220+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1240098355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1f0109b90 msgr2=0x7fd1f019fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.220+0000 7fd1f7c5c640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1f0109b90 0x7fd1f019fa60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd1e800eaa0 tx=0x7fd1e800ef60 comp rx=0 tx=0).stop 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.220+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1240098355 shutdown_connections 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.220+0000 7fd1f7c5c640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1f0109b90 0x7fd1f019fa60 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.220+0000 7fd1f7c5c640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1dc0b4d20 0x7fd1dc0b7230 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.220+0000 7fd1f7c5c640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1f0109240 0x7fd1f019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.220+0000 7fd1f7c5c640 1 --2- 172.21.15.16:0/1240098355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1f0103900 0x7fd1f019a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.220+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1240098355 >> 172.21.15.16:0/1240098355 conn(0x7fd1f007b5f0 msgr2=0x7fd1f0107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.221+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1240098355 shutdown_connections 2024-08-28T19:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:44.221+0000 7fd1f7c5c640 1 -- 172.21.15.16:0/1240098355 wait complete. 2024-08-28T19:26:44.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.36 MB/sec execute 144 sec latency 2846.268 ms 2024-08-28T19:26:44.972 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:26:45.223 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:45.470 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:45 smithi016 ceph-mon[18556]: pgmap v510: 129 pgs: 129 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 94 op/s 2024-08-28T19:26:45.470 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:45 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1240098355' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:45 smithi028 ceph-mon[21769]: pgmap v510: 129 pgs: 129 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 94 op/s 2024-08-28T19:26:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:45 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1240098355' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:45 smithi115 ceph-mon[22184]: pgmap v510: 129 pgs: 129 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 94 op/s 2024-08-28T19:26:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:45 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1240098355' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:26:45.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.680+0000 7f1c8a51c640 1 -- 172.21.15.16:0/2061104460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84108160 msgr2=0x7f1c84108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:45.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.680+0000 7f1c8a51c640 1 --2- 172.21.15.16:0/2061104460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84108160 0x7f1c84108570 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009900 tx=0x7f1c7802a410 comp rx=0 tx=0).stop 2024-08-28T19:26:45.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.680+0000 7f1c8a51c640 1 -- 172.21.15.16:0/2061104460 shutdown_connections 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.680+0000 7f1c8a51c640 1 --2- 172.21.15.16:0/2061104460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c84108ab0 0x7f1c8410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.680+0000 7f1c8a51c640 1 --2- 172.21.15.16:0/2061104460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84108160 0x7f1c84108570 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.680+0000 7f1c8a51c640 1 --2- 172.21.15.16:0/2061104460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c84102820 0x7f1c84102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.680+0000 7f1c8a51c640 1 -- 172.21.15.16:0/2061104460 >> 172.21.15.16:0/2061104460 conn(0x7f1c840794e0 msgr2=0x7f1c840798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.680+0000 7f1c8a51c640 1 -- 172.21.15.16:0/2061104460 shutdown_connections 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.680+0000 7f1c8a51c640 1 -- 172.21.15.16:0/2061104460 wait complete. 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c8a51c640 1 Processor -- start 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c8a51c640 1 -- start start 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c8a51c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84102820 0x7f1c84100e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c8a51c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c84108160 0x7f1c840ff430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c8a51c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c84108ab0 0x7f1c840ff970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c8a51c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c84112700 con 0x7f1c84102820 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c8a51c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c84112580 con 0x7f1c84108ab0 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c8a51c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1c84112880 con 0x7f1c84108160 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c83fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84102820 0x7f1c84100e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c83fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84102820 0x7f1c84100e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60354/0 (socket says 172.21.15.16:60354) 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c83fff640 1 -- 172.21.15.16:0/615646460 learned_addr learned my addr 172.21.15.16:0/615646460 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c84108ab0 0x7f1c840ff970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.681+0000 7f1c837fe640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c84108160 0x7f1c840ff430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.682+0000 7f1c83fff640 1 -- 172.21.15.16:0/615646460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c84108160 msgr2=0x7f1c840ff430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.682+0000 7f1c83fff640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c84108160 0x7f1c840ff430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.682+0000 7f1c83fff640 1 -- 172.21.15.16:0/615646460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c84108ab0 msgr2=0x7f1c840ff970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.682+0000 7f1c83fff640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c84108ab0 0x7f1c840ff970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.682+0000 7f1c83fff640 1 -- 172.21.15.16:0/615646460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c84100100 con 0x7f1c84102820 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.682+0000 7f1c83fff640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84102820 0x7f1c84100e40 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400d770 tx=0x7f1c7400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.682+0000 7f1c817fa640 1 -- 172.21.15.16:0/615646460 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c74017070 con 0x7f1c84102820 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.682+0000 7f1c817fa640 1 -- 172.21.15.16:0/615646460 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c740054d0 con 0x7f1c84102820 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.682+0000 7f1c817fa640 1 -- 172.21.15.16:0/615646460 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c74004c30 con 0x7f1c84102820 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.682+0000 7f1c8a51c640 1 -- 172.21.15.16:0/615646460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c841acc40 con 0x7f1c84102820 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.682+0000 7f1c8a51c640 1 -- 172.21.15.16:0/615646460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c841ad100 con 0x7f1c84102820 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c90a91740 1 Processor -- start 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c90a91740 1 -- start start 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c817fa640 1 -- 172.21.15.16:0/615646460 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1c74004e10 con 0x7f1c84102820 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22e8070 0x5582c268d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c268dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c90a91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5582c2604680 con 0x5582c22bf1b0 2024-08-28T19:26:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c90a91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5582c25f3c10 con 0x5582c22c7390 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c90a91740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5582c25f7bc0 con 0x5582c22e8070 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c83fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c83fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60364/0 (socket says 172.21.15.16:60364) 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c83fff640 1 -- 172.21.15.16:0/2719737814 learned_addr learned my addr 172.21.15.16:0/2719737814 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c837fe640 1 --2- 172.21.15.16:0/2719737814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c268dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/2719737814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22e8070 0x5582c268d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c83fff640 1 -- 172.21.15.16:0/2719737814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22e8070 msgr2=0x5582c268d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c83fff640 1 --2- 172.21.15.16:0/2719737814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22e8070 0x5582c268d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.685+0000 7f1c83fff640 1 -- 172.21.15.16:0/2719737814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 msgr2=0x5582c268dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.686+0000 7f1c83fff640 1 --2- 172.21.15.16:0/2719737814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c268dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.686+0000 7f1c83fff640 1 -- 172.21.15.16:0/2719737814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582c2353650 con 0x5582c22bf1b0 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.686+0000 7f1c837fe640 1 --2- 172.21.15.16:0/2719737814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c268dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.686+0000 7f1c817fa640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c6c0b4ca0 0x7f1c6c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.686+0000 7f1c83fff640 1 --2- 172.21.15.16:0/2719737814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268d850 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f1c740e2670 tx=0x7f1c740ed9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6b11f735285d17 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.686+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c6c0b4ca0 0x7f1c6c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.686+0000 7f1c817fa640 1 -- 172.21.15.16:0/615646460 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1c740e5320 con 0x7f1c84102820 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.686+0000 7f1c6a7fc640 1 -- 172.21.15.16:0/2719737814 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c740a7610 con 0x5582c22bf1b0 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.686+0000 7f1c6a7fc640 1 -- 172.21.15.16:0/2719737814 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c740ab260 con 0x5582c22bf1b0 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.686+0000 7f1c6a7fc640 1 -- 172.21.15.16:0/2719737814 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c740aaec0 con 0x5582c22bf1b0 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.687+0000 7f1c90a91740 1 -- 172.21.15.16:0/2719737814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 msgr2=0x5582c268d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.687+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c6c0b4ca0 0x7f1c6c0b71b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1c841014f0 tx=0x7f1c7c007520 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.687+0000 7f1c90a91740 1 --2- 172.21.15.16:0/2719737814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268d850 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f1c740e2670 tx=0x7f1c740ed9b0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.687+0000 7f1c90a91740 1 -- 172.21.15.16:0/2719737814 shutdown_connections 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.687+0000 7f1c90a91740 1 --2- 172.21.15.16:0/2719737814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c268dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.687+0000 7f1c90a91740 1 --2- 172.21.15.16:0/2719737814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268d850 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.687+0000 7f1c90a91740 1 --2- 172.21.15.16:0/2719737814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22e8070 0x5582c268d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.687+0000 7f1c90a91740 1 -- 172.21.15.16:0/2719737814 >> 172.21.15.16:0/2719737814 conn(0x5582c22dc460 msgr2=0x5582c268b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.687+0000 7f1c90a91740 1 -- 172.21.15.16:0/2719737814 shutdown_connections 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.687+0000 7f1c90a91740 1 -- 172.21.15.16:0/2719737814 wait complete. 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c90a91740 1 -- start start 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c90a91740 1 Processor -- start 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22bf1b0 0x5582c2698b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c2699050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c26a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c90a91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5582c2456700 con 0x5582c22e8070 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c90a91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5582c257a750 con 0x5582c22c7390 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c90a91740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5582c269baf0 con 0x5582c22bf1b0 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c90a91740 10 client.0 _ll_register_callbacks cb 0x5582c25a9e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c88ab9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c26a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c88ab9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c26a4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60372/0 (socket says 172.21.15.16:60372) 2024-08-28T19:26:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c88ab9640 1 -- 172.21.15.16:0/2456292231 learned_addr learned my addr 172.21.15.16:0/2456292231 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.688+0000 7f1c837fe640 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c2699050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c83fff640 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22bf1b0 0x5582c2698b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c88ab9640 1 -- 172.21.15.16:0/2456292231 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22bf1b0 msgr2=0x5582c2698b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22bf1b0 0x5582c2698b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c88ab9640 1 -- 172.21.15.16:0/2456292231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 msgr2=0x5582c2699050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c2699050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c88ab9640 1 -- 172.21.15.16:0/2456292231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582c25b3900 con 0x5582c22e8070 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c83fff640 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22bf1b0 0x5582c2698b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c26a4e40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0072f0 tx=0x7f1c7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fef9f0a99c55ff server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c637fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c637fe640 1 -- 172.21.15.16:0/2456292231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00eb30 con 0x5582c22e8070 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c637fe640 1 -- 172.21.15.16:0/2456292231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1c7c00ecd0 con 0x5582c22e8070 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c90a91740 1 -- 172.21.15.16:0/2456292231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c740ab260 con 0x5582c22e8070 2024-08-28T19:26:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c637fe640 1 -- 172.21.15.16:0/2456292231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f1c7c02b620 con 0x5582c22e8070 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c90a91740 10 client.17049 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c90a91740 1 -- 172.21.15.16:0/2456292231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5582c26a79b0 con 0x5582c22e8070 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.689+0000 7f1c637fe640 1 -- 172.21.15.16:0/2456292231 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c740ab260 con 0x5582c22e8070 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.690+0000 7f1c637fe640 1 -- 172.21.15.16:0/2456292231 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f1c7c026b80 con 0x5582c22e8070 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.690+0000 7f1c90a91740 10 client.17049 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.690+0000 7f1c90a91740 10 client.17049 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.690+0000 7f1c90a91740 20 client.17049 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.690+0000 7f1c90a91740 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5582c22a69f0 0x5582c26aae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.690+0000 7f1c90a91740 4 client.17049 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.690+0000 7f1c90a91740 1 -- 172.21.15.16:0/2456292231 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582c26065d0 con 0x5582c22a69f0 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.690+0000 7f1c83fff640 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5582c22a69f0 0x5582c26aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.691+0000 7f1c83fff640 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5582c22a69f0 0x5582c26aae40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=887dd18ad69ac915 server_cookie=be283d048ede02b1 in_seq=0 out_seq=0 2024-08-28T19:26:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.691+0000 7f1c637fe640 10 client.17049 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.691+0000 7f1c637fe640 1 -- 172.21.15.16:0/2456292231 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5582c26065d0 con 0x5582c22a69f0 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.691+0000 7f1c637fe640 10 client.17049 handle_command_reply: tid=0 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.691+0000 7f1c90a91740 1 client.17049 shutdown 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c90a91740 1 -- 172.21.15.16:0/2456292231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 msgr2=0x5582c26a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c90a91740 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c26a4e40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0072f0 tx=0x7f1c7c004a40 comp rx=0 tx=0).stop 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c90a91740 1 -- 172.21.15.16:0/2456292231 shutdown_connections 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c90a91740 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5582c22a69f0 0x5582c26aae40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c90a91740 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c26a4e40 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c637fe640 0 client.17049 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c90a91740 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c2699050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c90a91740 1 --2- 172.21.15.16:0/2456292231 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22bf1b0 0x5582c2698b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c90a91740 1 -- 172.21.15.16:0/2456292231 >> 172.21.15.16:0/2456292231 conn(0x5582c22dc460 msgr2=0x5582c26904a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:45.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c90a91740 1 -- 172.21.15.16:0/2456292231 shutdown_connections 2024-08-28T19:26:45.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c90a91740 1 -- 172.21.15.16:0/2456292231 wait complete. 2024-08-28T19:26:45.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.692+0000 7f1c90a91740 20 client.17049 trim_cache size 0 max 16384 2024-08-28T19:26:45.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.705+0000 7f1c90a91740 1 Processor -- start 2024-08-28T19:26:45.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.705+0000 7f1c90a91740 1 -- start start 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c26a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22e8070 0x5582c26a5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c90a91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841b7210 con 0x5582c22bf1b0 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c90a91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841b8d80 con 0x5582c22c7390 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c90a91740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841b78e0 con 0x5582c22e8070 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c83fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c83fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60384/0 (socket says 172.21.15.16:60384) 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c83fff640 1 -- 172.21.15.16:0/592345032 learned_addr learned my addr 172.21.15.16:0/592345032 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/592345032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22e8070 0x5582c26a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c837fe640 1 --2- 172.21.15.16:0/592345032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c26a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c83fff640 1 -- 172.21.15.16:0/592345032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22e8070 msgr2=0x5582c26a5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c83fff640 1 --2- 172.21.15.16:0/592345032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22e8070 0x5582c26a5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c83fff640 1 -- 172.21.15.16:0/592345032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 msgr2=0x5582c26a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c83fff640 1 --2- 172.21.15.16:0/592345032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c26a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c83fff640 1 -- 172.21.15.16:0/592345032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582c2697010 con 0x5582c22bf1b0 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c837fe640 1 --2- 172.21.15.16:0/592345032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c26a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/592345032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22e8070 0x5582c26a5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.706+0000 7f1c83fff640 1 --2- 172.21.15.16:0/592345032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268b300 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f1c740e2d00 tx=0x7f1c740e2710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f577f3f558ede5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.707+0000 7f1c69ffb640 1 -- 172.21.15.16:0/592345032 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c74020040 con 0x5582c22bf1b0 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.707+0000 7f1c69ffb640 1 -- 172.21.15.16:0/592345032 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c740e2e50 con 0x5582c22bf1b0 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.707+0000 7f1c69ffb640 1 -- 172.21.15.16:0/592345032 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c740ab0a0 con 0x5582c22bf1b0 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.707+0000 7f1c90a91740 1 -- 172.21.15.16:0/592345032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 msgr2=0x5582c268b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.707+0000 7f1c90a91740 1 --2- 172.21.15.16:0/592345032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268b300 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f1c740e2d00 tx=0x7f1c740e2710 comp rx=0 tx=0).stop 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.707+0000 7f1c90a91740 1 -- 172.21.15.16:0/592345032 shutdown_connections 2024-08-28T19:26:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.707+0000 7f1c90a91740 1 --2- 172.21.15.16:0/592345032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22e8070 0x5582c26a5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.707+0000 7f1c90a91740 1 --2- 172.21.15.16:0/592345032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22c7390 0x5582c26a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.707+0000 7f1c90a91740 1 --2- 172.21.15.16:0/592345032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22bf1b0 0x5582c268b300 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.707+0000 7f1c90a91740 1 -- 172.21.15.16:0/592345032 >> 172.21.15.16:0/592345032 conn(0x5582c22dc460 msgr2=0x5582c268abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.708+0000 7f1c90a91740 1 -- 172.21.15.16:0/592345032 shutdown_connections 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.708+0000 7f1c90a91740 1 -- 172.21.15.16:0/592345032 wait complete. 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.708+0000 7f1c90a91740 1 -- start start 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.708+0000 7f1c90a91740 1 Processor -- start 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.708+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22bf1b0 0x5582c26a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22c7390 0x5582c26a1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c90a91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c269dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c90a91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841b86b0 con 0x5582c22e8070 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c90a91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841b7fb0 con 0x5582c22bf1b0 2024-08-28T19:26:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c90a91740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841b51f0 con 0x5582c22c7390 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c90a91740 10 client.0 _ll_register_callbacks cb 0x7f1c7c007860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c88ab9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c269dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c88ab9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c269dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60396/0 (socket says 172.21.15.16:60396) 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c88ab9640 1 -- 172.21.15.16:0/4293640042 learned_addr learned my addr 172.21.15.16:0/4293640042 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c83fff640 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22bf1b0 0x5582c26a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c837fe640 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22c7390 0x5582c26a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c88ab9640 1 -- 172.21.15.16:0/4293640042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22c7390 msgr2=0x5582c26a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22c7390 0x5582c26a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c88ab9640 1 -- 172.21.15.16:0/4293640042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22bf1b0 msgr2=0x5582c26a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22bf1b0 0x5582c26a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c88ab9640 1 -- 172.21.15.16:0/4293640042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582c269e2b0 con 0x5582c22e8070 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c88ab9640 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c269dd70 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c008010 tx=0x7f1c7c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53cbfc891b327727 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c637fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:45.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c637fe640 1 -- 172.21.15.16:0/4293640042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c01cbd0 con 0x5582c22e8070 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c637fe640 1 -- 172.21.15.16:0/4293640042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1c7c01cd70 con 0x5582c22e8070 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.709+0000 7f1c90a91740 1 -- 172.21.15.16:0/4293640042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c740e2e50 con 0x5582c22e8070 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c637fe640 1 -- 172.21.15.16:0/4293640042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f1c7c02b460 con 0x5582c22e8070 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c90a91740 10 client.17055 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c90a91740 1 -- 172.21.15.16:0/4293640042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5582c2715e50 con 0x5582c22e8070 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c637fe640 1 -- 172.21.15.16:0/4293640042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c740e2e50 con 0x5582c22e8070 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c637fe640 1 -- 172.21.15.16:0/4293640042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f1c7c00e780 con 0x5582c22e8070 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c90a91740 10 client.17055 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c90a91740 10 client.17055 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c90a91740 20 client.17055 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c90a91740 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5582c227bda0 0x5582c2718570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c90a91740 4 client.17055 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:26:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c90a91740 1 -- 172.21.15.16:0/4293640042 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5582c2602070 con 0x5582c227bda0 2024-08-28T19:26:45.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.710+0000 7f1c83fff640 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5582c227bda0 0x5582c2718570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:45.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c83fff640 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5582c227bda0 0x5582c2718570 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b2913666ecd33321 server_cookie=4a3bd4b61325f15a in_seq=0 out_seq=0 2024-08-28T19:26:45.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c637fe640 10 client.17055 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:26:45.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c637fe640 1 -- 172.21.15.16:0/4293640042 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5582c2602070 con 0x5582c227bda0 2024-08-28T19:26:45.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c637fe640 10 client.17055 handle_command_reply: tid=0 2024-08-28T19:26:45.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c90a91740 1 client.17055 shutdown 2024-08-28T19:26:45.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c90a91740 1 -- 172.21.15.16:0/4293640042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 msgr2=0x5582c269dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:45.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c90a91740 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c269dd70 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c008010 tx=0x7f1c7c01c8d0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c90a91740 1 -- 172.21.15.16:0/4293640042 shutdown_connections 2024-08-28T19:26:45.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c90a91740 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5582c227bda0 0x5582c2718570 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c90a91740 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582c22e8070 0x5582c269dd70 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c90a91740 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582c22c7390 0x5582c26a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c637fe640 0 client.17055 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:26:45.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c90a91740 1 --2- 172.21.15.16:0/4293640042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582c22bf1b0 0x5582c26a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c90a91740 1 -- 172.21.15.16:0/4293640042 >> 172.21.15.16:0/4293640042 conn(0x5582c22dc460 msgr2=0x5582c2690410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:45.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c90a91740 1 -- 172.21.15.16:0/4293640042 shutdown_connections 2024-08-28T19:26:45.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.711+0000 7f1c90a91740 1 -- 172.21.15.16:0/4293640042 wait complete. 2024-08-28T19:26:45.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c90a91740 20 client.17055 trim_cache size 0 max 16384 2024-08-28T19:26:45.719 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:26:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 -- 172.21.15.16:0/615646460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c6c0b4ca0 msgr2=0x7f1c6c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c6c0b4ca0 0x7f1c6c0b71b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1c841014f0 tx=0x7f1c7c007520 comp rx=0 tx=0).stop 2024-08-28T19:26:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 -- 172.21.15.16:0/615646460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84102820 msgr2=0x7f1c84100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84102820 0x7f1c84100e40 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400d770 tx=0x7f1c7400dc30 comp rx=0 tx=0).stop 2024-08-28T19:26:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 -- 172.21.15.16:0/615646460 shutdown_connections 2024-08-28T19:26:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c6c0b4ca0 0x7f1c6c0b71b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c84108ab0 0x7f1c840ff970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c84108160 0x7f1c840ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 --2- 172.21.15.16:0/615646460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84102820 0x7f1c84100e40 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:45.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 -- 172.21.15.16:0/615646460 >> 172.21.15.16:0/615646460 conn(0x7f1c840794e0 msgr2=0x7f1c8406b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:45.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 -- 172.21.15.16:0/615646460 shutdown_connections 2024-08-28T19:26:45.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:45.712+0000 7f1c8a51c640 1 -- 172.21.15.16:0/615646460 wait complete. 2024-08-28T19:26:45.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.32 MB/sec execute 145 sec latency 3846.306 ms 2024-08-28T19:26:46.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:26:46.814 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:46.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.28 MB/sec execute 146 sec latency 4846.356 ms 2024-08-28T19:26:47.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.293+0000 7ff080258640 1 -- 172.21.15.16:0/2708002155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0781024b0 msgr2=0x7ff0781079c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.293+0000 7ff080258640 1 --2- 172.21.15.16:0/2708002155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0781024b0 0x7ff0781079c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ff068009930 tx=0x7ff06802a420 comp rx=0 tx=0).stop 2024-08-28T19:26:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.293+0000 7ff080258640 1 -- 172.21.15.16:0/2708002155 shutdown_connections 2024-08-28T19:26:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.293+0000 7ff080258640 1 --2- 172.21.15.16:0/2708002155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff07810c5f0 0x7ff07810ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.293+0000 7ff080258640 1 --2- 172.21.15.16:0/2708002155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0781024b0 0x7ff0781079c0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.293+0000 7ff080258640 1 --2- 172.21.15.16:0/2708002155 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff078069e80 0x7ff078107480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.293+0000 7ff080258640 1 -- 172.21.15.16:0/2708002155 >> 172.21.15.16:0/2708002155 conn(0x7ff0780775a0 msgr2=0x7ff0780779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.293+0000 7ff080258640 1 -- 172.21.15.16:0/2708002155 shutdown_connections 2024-08-28T19:26:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.293+0000 7ff080258640 1 -- 172.21.15.16:0/2708002155 wait complete. 2024-08-28T19:26:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.293+0000 7ff080258640 1 Processor -- start 2024-08-28T19:26:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.293+0000 7ff080258640 1 -- start start 2024-08-28T19:26:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff080258640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff078069e80 0x7ff07819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff080258640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0781024b0 0x7ff07819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff080258640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff07810c5f0 0x7ff07819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff080258640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff078111ee0 con 0x7ff07810c5f0 2024-08-28T19:26:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff080258640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff078111d60 con 0x7ff078069e80 2024-08-28T19:26:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff080258640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff078112060 con 0x7ff0781024b0 2024-08-28T19:26:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07e7f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff07810c5f0 0x7ff07819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07e7f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff07810c5f0 0x7ff07819fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60416/0 (socket says 172.21.15.16:60416) 2024-08-28T19:26:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07e7f5640 1 -- 172.21.15.16:0/596068644 learned_addr learned my addr 172.21.15.16:0/596068644 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07dff4640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff078069e80 0x7ff07819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0781024b0 0x7ff07819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07e7f5640 1 -- 172.21.15.16:0/596068644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0781024b0 msgr2=0x7ff07819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07e7f5640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0781024b0 0x7ff07819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07e7f5640 1 -- 172.21.15.16:0/596068644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff078069e80 msgr2=0x7ff07819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07e7f5640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff078069e80 0x7ff07819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07e7f5640 1 -- 172.21.15.16:0/596068644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0781a0210 con 0x7ff07810c5f0 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07dff4640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff078069e80 0x7ff07819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0781024b0 0x7ff07819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.294+0000 7ff07e7f5640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff07810c5f0 0x7ff07819fae0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff07000eaa0 tx=0x7ff07000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.295+0000 7ff06effd640 1 -- 172.21.15.16:0/596068644 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07000cdb0 con 0x7ff07810c5f0 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.295+0000 7ff080258640 1 -- 172.21.15.16:0/596068644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0781a0500 con 0x7ff07810c5f0 2024-08-28T19:26:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.295+0000 7ff06effd640 1 -- 172.21.15.16:0/596068644 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff070005020 con 0x7ff07810c5f0 2024-08-28T19:26:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.295+0000 7ff06effd640 1 -- 172.21.15.16:0/596068644 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0700116d0 con 0x7ff07810c5f0 2024-08-28T19:26:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.295+0000 7ff080258640 1 -- 172.21.15.16:0/596068644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0781a6c60 con 0x7ff07810c5f0 2024-08-28T19:26:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.297+0000 7ff0867cd740 1 Processor -- start 2024-08-28T19:26:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.297+0000 7ff06effd640 1 -- 172.21.15.16:0/596068644 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff0700051d0 con 0x7ff07810c5f0 2024-08-28T19:26:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.297+0000 7ff0867cd740 1 -- start start 2024-08-28T19:26:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d700350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d3321b0 0x56376d700890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d33a390 0x56376d700dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff0867cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56376d677680 con 0x56376d3321b0 2024-08-28T19:26:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff0867cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56376d666c10 con 0x56376d35b070 2024-08-28T19:26:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff0867cd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56376d66abc0 con 0x56376d33a390 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07dff4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d3321b0 0x56376d700890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07dff4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d3321b0 0x56376d700890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60430/0 (socket says 172.21.15.16:60430) 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07dff4640 1 -- 172.21.15.16:0/2944774662 learned_addr learned my addr 172.21.15.16:0/2944774662 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/2944774662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d700350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/2944774662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d33a390 0x56376d700dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff06effd640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0640b4ca0 0x7ff0640b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07dff4640 1 -- 172.21.15.16:0/2944774662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d33a390 msgr2=0x56376d700dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07dff4640 1 --2- 172.21.15.16:0/2944774662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d33a390 0x56376d700dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07dff4640 1 -- 172.21.15.16:0/2944774662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 msgr2=0x56376d700350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07dff4640 1 --2- 172.21.15.16:0/2944774662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d700350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07dff4640 1 -- 172.21.15.16:0/2944774662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56376d3c6650 con 0x56376d3321b0 2024-08-28T19:26:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/2944774662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d700350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff07e7f5640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0640b4ca0 0x7ff0640b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.298+0000 7ff06effd640 1 -- 172.21.15.16:0/596068644 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff0700e5b40 con 0x7ff07810c5f0 2024-08-28T19:26:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.299+0000 7ff07dff4640 1 --2- 172.21.15.16:0/2944774662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d3321b0 0x56376d700890 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff07400aa10 tx=0x7ff07400aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b010dd64862d2e57 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.299+0000 7ff063fff640 1 -- 172.21.15.16:0/2944774662 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074015070 con 0x56376d3321b0 2024-08-28T19:26:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.299+0000 7ff063fff640 1 -- 172.21.15.16:0/2944774662 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff07400cc60 con 0x56376d3321b0 2024-08-28T19:26:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.299+0000 7ff063fff640 1 -- 172.21.15.16:0/2944774662 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0740105b0 con 0x56376d3321b0 2024-08-28T19:26:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.300+0000 7ff0867cd740 1 -- 172.21.15.16:0/2944774662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d3321b0 msgr2=0x56376d700890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.300+0000 7ff07e7f5640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0640b4ca0 0x7ff0640b71b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff0700abbc0 tx=0x7ff070079360 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.300+0000 7ff0867cd740 1 --2- 172.21.15.16:0/2944774662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d3321b0 0x56376d700890 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff07400aa10 tx=0x7ff07400aed0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.300+0000 7ff0867cd740 1 -- 172.21.15.16:0/2944774662 shutdown_connections 2024-08-28T19:26:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.300+0000 7ff0867cd740 1 --2- 172.21.15.16:0/2944774662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d33a390 0x56376d700dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.300+0000 7ff0867cd740 1 --2- 172.21.15.16:0/2944774662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d3321b0 0x56376d700890 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.300+0000 7ff0867cd740 1 --2- 172.21.15.16:0/2944774662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d700350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.300+0000 7ff0867cd740 1 -- 172.21.15.16:0/2944774662 >> 172.21.15.16:0/2944774662 conn(0x56376d34f460 msgr2=0x56376d6fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.300+0000 7ff0867cd740 1 -- 172.21.15.16:0/2944774662 shutdown_connections 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.300+0000 7ff0867cd740 1 -- 172.21.15.16:0/2944774662 wait complete. 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.300+0000 7ff0867cd740 1 -- start start 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff0867cd740 1 Processor -- start 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d3321b0 0x56376d70ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d70bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d717e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff0867cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56376d4c9700 con 0x56376d33a390 2024-08-28T19:26:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff0867cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56376d5ed750 con 0x56376d35b070 2024-08-28T19:26:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff0867cd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56376d70eb30 con 0x56376d3321b0 2024-08-28T19:26:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff0867cd740 10 client.0 _ll_register_callbacks cb 0x56376d61ce10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff07dff4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d70bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff07dff4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d70bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60440/0 (socket says 172.21.15.16:60440) 2024-08-28T19:26:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff07dff4640 1 -- 172.21.15.16:0/3637419741 learned_addr learned my addr 172.21.15.16:0/3637419741 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d3321b0 0x56376d70ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.301+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d717e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff07dff4640 1 -- 172.21.15.16:0/3637419741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d3321b0 msgr2=0x56376d70ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff07dff4640 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d3321b0 0x56376d70ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff07dff4640 1 -- 172.21.15.16:0/3637419741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 msgr2=0x56376d717e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff07dff4640 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d717e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff07dff4640 1 -- 172.21.15.16:0/3637419741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56376d626900 con 0x56376d33a390 2024-08-28T19:26:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d717e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff07dff4640 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d70bfb0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff07400a9e0 tx=0x7ff0740119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78b83c692c6080d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff0617fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff0617fa640 1 -- 172.21.15.16:0/3637419741 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074020070 con 0x56376d33a390 2024-08-28T19:26:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff0617fa640 1 -- 172.21.15.16:0/3637419741 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff07401a660 con 0x56376d33a390 2024-08-28T19:26:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff0867cd740 1 -- 172.21.15.16:0/3637419741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff07400cc60 con 0x56376d33a390 2024-08-28T19:26:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff0617fa640 1 -- 172.21.15.16:0/3637419741 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ff074015040 con 0x56376d33a390 2024-08-28T19:26:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff0867cd740 10 client.17067 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff0867cd740 1 -- 172.21.15.16:0/3637419741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56376d71a8a0 con 0x56376d33a390 2024-08-28T19:26:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.302+0000 7ff0617fa640 1 -- 172.21.15.16:0/3637419741 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff07400cc60 con 0x56376d33a390 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.303+0000 7ff0617fa640 1 -- 172.21.15.16:0/3637419741 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7ff074024730 con 0x56376d33a390 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.303+0000 7ff0867cd740 10 client.17067 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.303+0000 7ff0867cd740 10 client.17067 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.303+0000 7ff0867cd740 20 client.17067 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.303+0000 7ff0867cd740 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56376d3199f0 0x56376d71dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.303+0000 7ff0867cd740 4 client.17067 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.303+0000 7ff0867cd740 1 -- 172.21.15.16:0/3637419741 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56376d6795d0 con 0x56376d3199f0 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.303+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56376d3199f0 0x56376d71dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.304+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56376d3199f0 0x56376d71dc00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=15302f42c81d49c1 server_cookie=7cc2f089d171ee1c in_seq=0 out_seq=0 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.304+0000 7ff0617fa640 10 client.17067 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.304+0000 7ff0617fa640 1 -- 172.21.15.16:0/3637419741 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56376d6795d0 con 0x56376d3199f0 2024-08-28T19:26:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.304+0000 7ff0617fa640 10 client.17067 handle_command_reply: tid=0 2024-08-28T19:26:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.304+0000 7ff0867cd740 1 client.17067 shutdown 2024-08-28T19:26:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0867cd740 1 -- 172.21.15.16:0/3637419741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 msgr2=0x56376d70bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0867cd740 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d70bfb0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff07400a9e0 tx=0x7ff0740119a0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0867cd740 1 -- 172.21.15.16:0/3637419741 shutdown_connections 2024-08-28T19:26:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0867cd740 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56376d3199f0 0x56376d71dc00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0867cd740 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d717e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0867cd740 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d70bfb0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0867cd740 1 --2- 172.21.15.16:0/3637419741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d3321b0 0x56376d70ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0617fa640 0 client.17067 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:26:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0867cd740 1 -- 172.21.15.16:0/3637419741 >> 172.21.15.16:0/3637419741 conn(0x56376d34f460 msgr2=0x56376d7034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0867cd740 1 -- 172.21.15.16:0/3637419741 shutdown_connections 2024-08-28T19:26:47.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0867cd740 1 -- 172.21.15.16:0/3637419741 wait complete. 2024-08-28T19:26:47.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.305+0000 7ff0867cd740 20 client.17067 trim_cache size 0 max 16384 2024-08-28T19:26:47.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.318+0000 7ff0867cd740 1 Processor -- start 2024-08-28T19:26:47.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.318+0000 7ff0867cd740 1 -- start start 2024-08-28T19:26:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d3321b0 0x56376d70bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d717df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d35b070 0x56376d6fdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff0867cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff0781b0d20 con 0x56376d33a390 2024-08-28T19:26:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff0867cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0781b2890 con 0x56376d3321b0 2024-08-28T19:26:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff0867cd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff0781b13f0 con 0x56376d35b070 2024-08-28T19:26:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff07dff4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d717df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff07dff4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d717df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60454/0 (socket says 172.21.15.16:60454) 2024-08-28T19:26:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff07dff4640 1 -- 172.21.15.16:0/3963543298 learned_addr learned my addr 172.21.15.16:0/3963543298 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/3963543298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d3321b0 0x56376d70bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff07dff4640 1 -- 172.21.15.16:0/3963543298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d35b070 msgr2=0x56376d6fdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff07dff4640 1 --2- 172.21.15.16:0/3963543298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d35b070 0x56376d6fdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff07dff4640 1 -- 172.21.15.16:0/3963543298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d3321b0 msgr2=0x56376d70bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff07dff4640 1 --2- 172.21.15.16:0/3963543298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d3321b0 0x56376d70bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff07dff4640 1 -- 172.21.15.16:0/3963543298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56376d70c980 con 0x56376d33a390 2024-08-28T19:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/3963543298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d3321b0 0x56376d70bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff07dff4640 1 --2- 172.21.15.16:0/3963543298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d717df0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff074011f10 tx=0x7ff07400c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8f371ef49d8b2e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff0637fe640 1 -- 172.21.15.16:0/3963543298 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074020040 con 0x56376d33a390 2024-08-28T19:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff0637fe640 1 -- 172.21.15.16:0/3963543298 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff074032070 con 0x56376d33a390 2024-08-28T19:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.319+0000 7ff0637fe640 1 -- 172.21.15.16:0/3963543298 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07401a8c0 con 0x56376d33a390 2024-08-28T19:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.320+0000 7ff0867cd740 1 -- 172.21.15.16:0/3963543298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 msgr2=0x56376d717df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.320+0000 7ff0867cd740 1 --2- 172.21.15.16:0/3963543298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d717df0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff074011f10 tx=0x7ff07400c5d0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.320+0000 7ff0867cd740 1 -- 172.21.15.16:0/3963543298 shutdown_connections 2024-08-28T19:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.320+0000 7ff0867cd740 1 --2- 172.21.15.16:0/3963543298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d35b070 0x56376d6fdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.320+0000 7ff0867cd740 1 --2- 172.21.15.16:0/3963543298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d717df0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.320+0000 7ff0867cd740 1 --2- 172.21.15.16:0/3963543298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d3321b0 0x56376d70bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.320+0000 7ff0867cd740 1 -- 172.21.15.16:0/3963543298 >> 172.21.15.16:0/3963543298 conn(0x56376d34f460 msgr2=0x56376d70b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.320+0000 7ff0867cd740 1 -- 172.21.15.16:0/3963543298 shutdown_connections 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.320+0000 7ff0867cd740 1 -- 172.21.15.16:0/3963543298 wait complete. 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.320+0000 7ff0867cd740 1 -- start start 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff0867cd740 1 Processor -- start 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d3321b0 0x56376d71ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d71f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff0867cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d71f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff0867cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff0781b21c0 con 0x56376d33a390 2024-08-28T19:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff0867cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0781b1ac0 con 0x56376d35b070 2024-08-28T19:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff0867cd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff0781aed00 con 0x56376d3321b0 2024-08-28T19:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff0867cd740 10 client.0 _ll_register_callbacks cb 0x7ff07400a860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff07dff4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d71f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff07dff4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d71f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60458/0 (socket says 172.21.15.16:60458) 2024-08-28T19:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff07dff4640 1 -- 172.21.15.16:0/827156519 learned_addr learned my addr 172.21.15.16:0/827156519 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d71f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d3321b0 0x56376d71ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff07dff4640 1 -- 172.21.15.16:0/827156519 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d3321b0 msgr2=0x56376d71ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff07dff4640 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d3321b0 0x56376d71ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff07dff4640 1 -- 172.21.15.16:0/827156519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 msgr2=0x56376d71f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff07dff4640 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d71f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.321+0000 7ff07dff4640 1 -- 172.21.15.16:0/827156519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56376d787580 con 0x56376d33a390 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.322+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d71f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.322+0000 7ff07dff4640 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d71f340 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff074004700 tx=0x7ff07400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472ab888eec8c233 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.322+0000 7ff0617fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.322+0000 7ff0617fa640 1 -- 172.21.15.16:0/827156519 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074020040 con 0x56376d33a390 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.322+0000 7ff0617fa640 1 -- 172.21.15.16:0/827156519 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff074024600 con 0x56376d33a390 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.322+0000 7ff0867cd740 1 -- 172.21.15.16:0/827156519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff074032070 con 0x56376d33a390 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.322+0000 7ff0617fa640 1 -- 172.21.15.16:0/827156519 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ff074046050 con 0x56376d33a390 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.322+0000 7ff0867cd740 10 client.17073 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.322+0000 7ff0867cd740 1 -- 172.21.15.16:0/827156519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56376d7887e0 con 0x56376d33a390 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.322+0000 7ff0617fa640 1 -- 172.21.15.16:0/827156519 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff074032070 con 0x56376d33a390 2024-08-28T19:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.322+0000 7ff0617fa640 1 -- 172.21.15.16:0/827156519 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7ff074037460 con 0x56376d33a390 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff0867cd740 10 client.17073 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff0867cd740 10 client.17073 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff0867cd740 20 client.17073 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff0867cd740 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56376d2eeda0 0x56376d78ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff0867cd740 4 client.17073 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff0867cd740 1 -- 172.21.15.16:0/827156519 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56376d675070 con 0x56376d2eeda0 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56376d2eeda0 0x56376d78ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff07d7f3640 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56376d2eeda0 0x56376d78ae00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bedc7524fbae9ca9 server_cookie=3d9108ae835e9064 in_seq=0 out_seq=0 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff0617fa640 10 client.17073 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff0617fa640 1 -- 172.21.15.16:0/827156519 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56376d675070 con 0x56376d2eeda0 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff0617fa640 10 client.17073 handle_command_reply: tid=0 2024-08-28T19:26:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.323+0000 7ff0867cd740 1 client.17073 shutdown 2024-08-28T19:26:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0867cd740 1 -- 172.21.15.16:0/827156519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 msgr2=0x56376d71f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0867cd740 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d71f340 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff074004700 tx=0x7ff07400a660 comp rx=0 tx=0).stop 2024-08-28T19:26:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0867cd740 1 -- 172.21.15.16:0/827156519 shutdown_connections 2024-08-28T19:26:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0867cd740 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x56376d2eeda0 0x56376d78ae00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0867cd740 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56376d35b070 0x56376d71f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0867cd740 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56376d33a390 0x56376d71f340 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0617fa640 0 client.17073 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:26:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0867cd740 1 --2- 172.21.15.16:0/827156519 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56376d3321b0 0x56376d71ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0867cd740 1 -- 172.21.15.16:0/827156519 >> 172.21.15.16:0/827156519 conn(0x56376d34f460 msgr2=0x56376d703450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0867cd740 1 -- 172.21.15.16:0/827156519 shutdown_connections 2024-08-28T19:26:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0867cd740 1 -- 172.21.15.16:0/827156519 wait complete. 2024-08-28T19:26:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff0867cd740 20 client.17073 trim_cache size 0 max 16384 2024-08-28T19:26:47.332 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:26:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff080258640 1 -- 172.21.15.16:0/596068644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0640b4ca0 msgr2=0x7ff0640b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff080258640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0640b4ca0 0x7ff0640b71b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff0700abbc0 tx=0x7ff070079360 comp rx=0 tx=0).stop 2024-08-28T19:26:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff080258640 1 -- 172.21.15.16:0/596068644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff07810c5f0 msgr2=0x7ff07819fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff080258640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff07810c5f0 0x7ff07819fae0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff07000eaa0 tx=0x7ff07000ef60 comp rx=0 tx=0).stop 2024-08-28T19:26:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff080258640 1 -- 172.21.15.16:0/596068644 shutdown_connections 2024-08-28T19:26:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff080258640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff07810c5f0 0x7ff07819fae0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff080258640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0781024b0 0x7ff07819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff080258640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff0640b4ca0 0x7ff0640b71b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff080258640 1 --2- 172.21.15.16:0/596068644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff078069e80 0x7ff07819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.324+0000 7ff080258640 1 -- 172.21.15.16:0/596068644 >> 172.21.15.16:0/596068644 conn(0x7ff0780775a0 msgr2=0x7ff078100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.325+0000 7ff080258640 1 -- 172.21.15.16:0/596068644 shutdown_connections 2024-08-28T19:26:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:47.325+0000 7ff080258640 1 -- 172.21.15.16:0/596068644 wait complete. 2024-08-28T19:26:47.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:47 smithi016 ceph-mon[18556]: pgmap v511: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-08-28T19:26:47.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:47 smithi028 ceph-mon[21769]: pgmap v511: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-08-28T19:26:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:47 smithi115 ceph-mon[22184]: pgmap v511: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-08-28T19:26:47.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.25 MB/sec execute 147 sec latency 5846.425 ms 2024-08-28T19:26:48.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:26:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:48.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.809+0000 7f07537a2640 1 -- 172.21.15.16:0/4068549965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c102f20 msgr2=0x7f074c103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.809+0000 7f07537a2640 1 --2- 172.21.15.16:0/4068549965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c102f20 0x7f074c103330 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0748009900 tx=0x7f074802a410 comp rx=0 tx=0).stop 2024-08-28T19:26:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.809+0000 7f07537a2640 1 -- 172.21.15.16:0/4068549965 shutdown_connections 2024-08-28T19:26:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.809+0000 7f07537a2640 1 --2- 172.21.15.16:0/4068549965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c108aa0 0x7f074c10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.809+0000 7f07537a2640 1 --2- 172.21.15.16:0/4068549965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f074c108150 0x7f074c108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.809+0000 7f07537a2640 1 --2- 172.21.15.16:0/4068549965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c102f20 0x7f074c103330 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.809+0000 7f07537a2640 1 -- 172.21.15.16:0/4068549965 >> 172.21.15.16:0/4068549965 conn(0x7f074c077a00 msgr2=0x7f074c077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.809+0000 7f07537a2640 1 -- 172.21.15.16:0/4068549965 shutdown_connections 2024-08-28T19:26:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.809+0000 7f07537a2640 1 -- 172.21.15.16:0/4068549965 wait complete. 2024-08-28T19:26:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f07537a2640 1 Processor -- start 2024-08-28T19:26:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f07537a2640 1 -- start start 2024-08-28T19:26:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f07537a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c102f20 0x7f074c19b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f07537a2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f074c108150 0x7f074c19b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f07537a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c108aa0 0x7f074c19bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f07537a2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f074c112770 con 0x7f074c102f20 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f07537a2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f074c1125f0 con 0x7f074c108aa0 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f07537a2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f074c1128f0 con 0x7f074c108150 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f075153e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c102f20 0x7f074c19b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f075153e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c102f20 0x7f074c19b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60478/0 (socket says 172.21.15.16:60478) 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f075153e640 1 -- 172.21.15.16:0/2369001524 learned_addr learned my addr 172.21.15.16:0/2369001524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f0750d3d640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f074c108150 0x7f074c19b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.810+0000 7f0751d3f640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c108aa0 0x7f074c19bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.811+0000 7f075153e640 1 -- 172.21.15.16:0/2369001524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f074c108150 msgr2=0x7f074c19b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.811+0000 7f075153e640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f074c108150 0x7f074c19b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.811+0000 7f075153e640 1 -- 172.21.15.16:0/2369001524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c108aa0 msgr2=0x7f074c19bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.811+0000 7f075153e640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c108aa0 0x7f074c19bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.811+0000 7f075153e640 1 -- 172.21.15.16:0/2369001524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f074c1a63b0 con 0x7f074c102f20 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.811+0000 7f075153e640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c102f20 0x7f074c19b120 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0748005940 tx=0x7f07480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.811+0000 7f07427fc640 1 -- 172.21.15.16:0/2369001524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074803f030 con 0x7f074c102f20 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.811+0000 7f07427fc640 1 -- 172.21.15.16:0/2369001524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0748003cb0 con 0x7f074c102f20 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.811+0000 7f07537a2640 1 -- 172.21.15.16:0/2369001524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f074c1a6640 con 0x7f074c102f20 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.811+0000 7f07427fc640 1 -- 172.21.15.16:0/2369001524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748035070 con 0x7f074c102f20 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.811+0000 7f07537a2640 1 -- 172.21.15.16:0/2369001524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f074c1a6b70 con 0x7f074c102f20 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.814+0000 7f0759d17740 1 Processor -- start 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.814+0000 7f0759d17740 1 -- start start 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.814+0000 7f07427fc640 1 -- 172.21.15.16:0/2369001524 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0748042030 con 0x7f074c102f20 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.814+0000 7f0759d17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x564688561310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.814+0000 7f0759d17740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881931b0 0x564688561850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.814+0000 7f0759d17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56468819b390 0x564688561d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.814+0000 7f0759d17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5646884d8680 con 0x5646881bc070 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.814+0000 7f0759d17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5646884c7c10 con 0x56468819b390 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.814+0000 7f0759d17740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5646884cbbc0 con 0x5646881931b0 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f0751d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x564688561310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f0751d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x564688561310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60494/0 (socket says 172.21.15.16:60494) 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f0751d3f640 1 -- 172.21.15.16:0/1504027050 learned_addr learned my addr 172.21.15.16:0/1504027050 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f0750d3d640 1 --2- 172.21.15.16:0/1504027050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56468819b390 0x564688561d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f075153e640 1 --2- 172.21.15.16:0/1504027050 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881931b0 0x564688561850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f0751d3f640 1 -- 172.21.15.16:0/1504027050 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881931b0 msgr2=0x564688561850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f0751d3f640 1 --2- 172.21.15.16:0/1504027050 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881931b0 0x564688561850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f0751d3f640 1 -- 172.21.15.16:0/1504027050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56468819b390 msgr2=0x564688561d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f0751d3f640 1 --2- 172.21.15.16:0/1504027050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56468819b390 0x564688561d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f0751d3f640 1 -- 172.21.15.16:0/1504027050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564688227650 con 0x5646881bc070 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f075153e640 1 --2- 172.21.15.16:0/1504027050 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881931b0 0x564688561850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f07427fc640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07380b4d50 0x7f07380b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f0751d3f640 1 --2- 172.21.15.16:0/1504027050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x564688561310 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f0744009f90 tx=0x7f0744010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86bf2789447ede5c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f075153e640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07380b4d50 0x7f07380b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f07377fe640 1 -- 172.21.15.16:0/1504027050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0744011700 con 0x5646881bc070 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f07377fe640 1 -- 172.21.15.16:0/1504027050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0744011d50 con 0x5646881bc070 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.815+0000 7f07377fe640 1 -- 172.21.15.16:0/1504027050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0744019d10 con 0x5646881bc070 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.816+0000 7f07427fc640 1 -- 172.21.15.16:0/2369001524 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0748105cb0 con 0x7f074c102f20 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.816+0000 7f0759d17740 1 -- 172.21.15.16:0/1504027050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 msgr2=0x564688561310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.816+0000 7f075153e640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07380b4d50 0x7f07380b7260 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0748103140 tx=0x7f0748040e30 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.816+0000 7f0759d17740 1 --2- 172.21.15.16:0/1504027050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x564688561310 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f0744009f90 tx=0x7f0744010860 comp rx=0 tx=0).stop 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.816+0000 7f0759d17740 1 -- 172.21.15.16:0/1504027050 shutdown_connections 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.816+0000 7f0759d17740 1 --2- 172.21.15.16:0/1504027050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56468819b390 0x564688561d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.816+0000 7f0759d17740 1 --2- 172.21.15.16:0/1504027050 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881931b0 0x564688561850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.816+0000 7f0759d17740 1 --2- 172.21.15.16:0/1504027050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x564688561310 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.816+0000 7f0759d17740 1 -- 172.21.15.16:0/1504027050 >> 172.21.15.16:0/1504027050 conn(0x5646881b0460 msgr2=0x56468855f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.816+0000 7f0759d17740 1 -- 172.21.15.16:0/1504027050 shutdown_connections 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.816+0000 7f0759d17740 1 -- 172.21.15.16:0/1504027050 wait complete. 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.817+0000 7f0759d17740 1 -- start start 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.817+0000 7f0759d17740 1 Processor -- start 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.817+0000 7f0759d17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881931b0 0x564688570db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.817+0000 7f0759d17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56468819b390 0x5646885712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.817+0000 7f0759d17740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881bc070 0x56468857d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.817+0000 7f0759d17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56468832a700 con 0x5646881931b0 2024-08-28T19:26:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.817+0000 7f0759d17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56468844e750 con 0x56468819b390 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.817+0000 7f0759d17740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564688573e50 con 0x5646881bc070 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0759d17740 10 client.0 _ll_register_callbacks cb 0x56468847de10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0751d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881931b0 0x564688570db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0751d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881931b0 0x564688570db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60496/0 (socket says 172.21.15.16:60496) 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0751d3f640 1 -- 172.21.15.16:0/2419829330 learned_addr learned my addr 172.21.15.16:0/2419829330 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0750d3d640 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56468819b390 0x5646885712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0751d3f640 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881bc070 0x56468857d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0751d3f640 1 -- 172.21.15.16:0/2419829330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881bc070 msgr2=0x56468857d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0751d3f640 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881bc070 0x56468857d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0751d3f640 1 -- 172.21.15.16:0/2419829330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56468819b390 msgr2=0x5646885712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0751d3f640 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56468819b390 0x5646885712f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0751d3f640 1 -- 172.21.15.16:0/2419829330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564688487900 con 0x5646881931b0 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0751d3f640 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881931b0 0x564688570db0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0744009990 tx=0x7f07440098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593538f9058b4514 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0734ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0734ff9640 1 -- 172.21.15.16:0/2419829330 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0744019a10 con 0x5646881931b0 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0734ff9640 1 -- 172.21.15.16:0/2419829330 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0744019bb0 con 0x5646881931b0 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.818+0000 7f0759d17740 1 -- 172.21.15.16:0/2419829330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0744011d50 con 0x5646881931b0 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0734ff9640 1 -- 172.21.15.16:0/2419829330 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f0744019db0 con 0x5646881931b0 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0759d17740 10 client.17085 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0759d17740 1 -- 172.21.15.16:0/2419829330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56468857fb70 con 0x5646881931b0 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0734ff9640 1 -- 172.21.15.16:0/2419829330 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0744011d50 con 0x5646881931b0 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0734ff9640 1 -- 172.21.15.16:0/2419829330 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f0744025030 con 0x5646881931b0 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0759d17740 10 client.17085 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0759d17740 10 client.17085 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:26:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0759d17740 20 client.17085 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0759d17740 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56468817a9f0 0x564688582f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0759d17740 4 client.17085 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0759d17740 1 -- 172.21.15.16:0/2419829330 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646884da5d0 con 0x56468817a9f0 2024-08-28T19:26:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.819+0000 7f0750d3d640 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56468817a9f0 0x564688582f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.820+0000 7f0750d3d640 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56468817a9f0 0x564688582f20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=feb8ebc9c2205dcf server_cookie=30d272724d6661fa in_seq=0 out_seq=0 2024-08-28T19:26:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.820+0000 7f0734ff9640 10 client.17085 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:26:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0734ff9640 1 -- 172.21.15.16:0/2419829330 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5646884da5d0 con 0x56468817a9f0 2024-08-28T19:26:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0734ff9640 10 client.17085 handle_command_reply: tid=0 2024-08-28T19:26:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 1 client.17085 shutdown 2024-08-28T19:26:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 1 -- 172.21.15.16:0/2419829330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881931b0 msgr2=0x564688570db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881931b0 0x564688570db0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0744009990 tx=0x7f07440098e0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 1 -- 172.21.15.16:0/2419829330 shutdown_connections 2024-08-28T19:26:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56468817a9f0 0x564688582f20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881bc070 0x56468857d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56468819b390 0x5646885712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 1 --2- 172.21.15.16:0/2419829330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881931b0 0x564688570db0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0734ff9640 0 client.17085 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:26:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 1 -- 172.21.15.16:0/2419829330 >> 172.21.15.16:0/2419829330 conn(0x5646881b0460 msgr2=0x56468855d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 1 -- 172.21.15.16:0/2419829330 shutdown_connections 2024-08-28T19:26:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 1 -- 172.21.15.16:0/2419829330 wait complete. 2024-08-28T19:26:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.821+0000 7f0759d17740 20 client.17085 trim_cache size 0 max 16384 2024-08-28T19:26:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.834+0000 7f0759d17740 1 Processor -- start 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.834+0000 7f0759d17740 1 -- start start 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0759d17740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881931b0 0x564688562d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0759d17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468819b390 0x56468857d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0759d17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646881bc070 0x5646885729e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0759d17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f074c1aeac0 con 0x56468819b390 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0759d17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f074c1b0630 con 0x5646881bc070 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0759d17740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f074c1af190 con 0x5646881931b0 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0750d3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468819b390 0x56468857d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0750d3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468819b390 0x56468857d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60512/0 (socket says 172.21.15.16:60512) 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0750d3d640 1 -- 172.21.15.16:0/4201416373 learned_addr learned my addr 172.21.15.16:0/4201416373 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0751d3f640 1 --2- 172.21.15.16:0/4201416373 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881931b0 0x564688562d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0751d3f640 1 --2- 172.21.15.16:0/4201416373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646881bc070 0x5646885729e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0750d3d640 1 -- 172.21.15.16:0/4201416373 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881931b0 msgr2=0x564688562d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0750d3d640 1 --2- 172.21.15.16:0/4201416373 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881931b0 0x564688562d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0750d3d640 1 -- 172.21.15.16:0/4201416373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646881bc070 msgr2=0x5646885729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0750d3d640 1 --2- 172.21.15.16:0/4201416373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646881bc070 0x5646885729e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0750d3d640 1 -- 172.21.15.16:0/4201416373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56468856f2b0 con 0x56468819b390 2024-08-28T19:26:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0751d3f640 1 --2- 172.21.15.16:0/4201416373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646881bc070 0x5646885729e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:26:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0750d3d640 1 --2- 172.21.15.16:0/4201416373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468819b390 0x56468857d120 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f074c0755c0 tx=0x7f073c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ea04ba5e12824c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0736ffd640 1 -- 172.21.15.16:0/4201416373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c011440 con 0x56468819b390 2024-08-28T19:26:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0736ffd640 1 -- 172.21.15.16:0/4201416373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f073c011a90 con 0x56468819b390 2024-08-28T19:26:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.835+0000 7f0736ffd640 1 -- 172.21.15.16:0/4201416373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c024a90 con 0x56468819b390 2024-08-28T19:26:48.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.836+0000 7f0759d17740 1 -- 172.21.15.16:0/4201416373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468819b390 msgr2=0x56468857d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.836+0000 7f0759d17740 1 --2- 172.21.15.16:0/4201416373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468819b390 0x56468857d120 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f074c0755c0 tx=0x7f073c00b390 comp rx=0 tx=0).stop 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.836+0000 7f0759d17740 1 -- 172.21.15.16:0/4201416373 shutdown_connections 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.836+0000 7f0759d17740 1 --2- 172.21.15.16:0/4201416373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646881bc070 0x5646885729e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.836+0000 7f0759d17740 1 --2- 172.21.15.16:0/4201416373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468819b390 0x56468857d120 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.836+0000 7f0759d17740 1 --2- 172.21.15.16:0/4201416373 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646881931b0 0x564688562d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.836+0000 7f0759d17740 1 -- 172.21.15.16:0/4201416373 >> 172.21.15.16:0/4201416373 conn(0x5646881b0460 msgr2=0x5646885722d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.836+0000 7f0759d17740 1 -- 172.21.15.16:0/4201416373 shutdown_connections 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.836+0000 7f0759d17740 1 -- 172.21.15.16:0/4201416373 wait complete. 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0759d17740 1 -- start start 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0759d17740 1 Processor -- start 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0759d17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646881931b0 0x5646885846e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0759d17740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56468819b390 0x564688584c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0759d17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x5646885ec6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0759d17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f074c1aff60 con 0x5646881bc070 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0759d17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f074c1af860 con 0x5646881931b0 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0759d17740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f074c1acaa0 con 0x56468819b390 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0759d17740 10 client.0 _ll_register_callbacks cb 0x7f073c004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0751d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x5646885ec6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0751d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x5646885ec6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60516/0 (socket says 172.21.15.16:60516) 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0751d3f640 1 -- 172.21.15.16:0/4095070165 learned_addr learned my addr 172.21.15.16:0/4095070165 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.837+0000 7f0751d3f640 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646881931b0 0x5646885846e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0750d3d640 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56468819b390 0x564688584c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0751d3f640 1 -- 172.21.15.16:0/4095070165 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56468819b390 msgr2=0x564688584c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0751d3f640 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56468819b390 0x564688584c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0751d3f640 1 -- 172.21.15.16:0/4095070165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646881931b0 msgr2=0x5646885846e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0751d3f640 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646881931b0 0x5646885846e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0751d3f640 1 -- 172.21.15.16:0/4095070165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646885ecc30 con 0x5646881bc070 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0751d3f640 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x5646885ec6a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x5646884d8790 tx=0x7f07440194d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33e3cee492ebf8e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0734ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0734ff9640 1 -- 172.21.15.16:0/4095070165 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0744009b60 con 0x5646881bc070 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0734ff9640 1 -- 172.21.15.16:0/4095070165 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f0744019a20 con 0x5646881bc070 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0759d17740 1 -- 172.21.15.16:0/4095070165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f073c011a90 con 0x5646881bc070 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0734ff9640 1 -- 172.21.15.16:0/4095070165 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f07440054f0 con 0x5646881bc070 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0759d17740 10 client.17091 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.838+0000 7f0759d17740 1 -- 172.21.15.16:0/4095070165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5646885edd90 con 0x5646881bc070 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.839+0000 7f0734ff9640 1 -- 172.21.15.16:0/4095070165 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f073c011a90 con 0x5646881bc070 2024-08-28T19:26:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.839+0000 7f0734ff9640 1 -- 172.21.15.16:0/4095070165 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f0744025040 con 0x5646881bc070 2024-08-28T19:26:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.839+0000 7f0759d17740 10 client.17091 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.839+0000 7f0759d17740 10 client.17091 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:26:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.839+0000 7f0759d17740 20 client.17091 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.839+0000 7f0759d17740 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56468814fda0 0x5646885f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.839+0000 7f0759d17740 4 client.17091 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:26:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.839+0000 7f0759d17740 1 -- 172.21.15.16:0/4095070165 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5646884d6070 con 0x56468814fda0 2024-08-28T19:26:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.839+0000 7f0750d3d640 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56468814fda0 0x5646885f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0750d3d640 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56468814fda0 0x5646885f04f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=55af7f5992e6fadc server_cookie=4cbd0288e6f02bea in_seq=0 out_seq=0 2024-08-28T19:26:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0734ff9640 10 client.17091 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:26:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0734ff9640 1 -- 172.21.15.16:0/4095070165 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5646884d6070 con 0x56468814fda0 2024-08-28T19:26:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0734ff9640 10 client.17091 handle_command_reply: tid=0 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 1 client.17091 shutdown 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 1 -- 172.21.15.16:0/4095070165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 msgr2=0x5646885ec6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x5646885ec6a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x5646884d8790 tx=0x7f07440194d0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 1 -- 172.21.15.16:0/4095070165 shutdown_connections 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56468814fda0 0x5646885f04f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646881bc070 0x5646885ec6a0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56468819b390 0x564688584c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 1 --2- 172.21.15.16:0/4095070165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646881931b0 0x5646885846e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 1 -- 172.21.15.16:0/4095070165 >> 172.21.15.16:0/4095070165 conn(0x5646881b0460 msgr2=0x564688566c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0734ff9640 0 client.17091 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 1 -- 172.21.15.16:0/4095070165 shutdown_connections 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 1 -- 172.21.15.16:0/4095070165 wait complete. 2024-08-28T19:26:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.840+0000 7f0759d17740 20 client.17091 trim_cache size 0 max 16384 2024-08-28T19:26:48.847 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:26:48.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 -- 172.21.15.16:0/2369001524 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07380b4d50 msgr2=0x7f07380b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07380b4d50 0x7f07380b7260 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0748103140 tx=0x7f0748040e30 comp rx=0 tx=0).stop 2024-08-28T19:26:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 -- 172.21.15.16:0/2369001524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c102f20 msgr2=0x7f074c19b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c102f20 0x7f074c19b120 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0748005940 tx=0x7f07480042f0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 -- 172.21.15.16:0/2369001524 shutdown_connections 2024-08-28T19:26:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07380b4d50 0x7f07380b7260 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c108aa0 0x7f074c19bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f074c108150 0x7f074c19b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 --2- 172.21.15.16:0/2369001524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c102f20 0x7f074c19b120 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 -- 172.21.15.16:0/2369001524 >> 172.21.15.16:0/2369001524 conn(0x7f074c077a00 msgr2=0x7f074c106300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 -- 172.21.15.16:0/2369001524 shutdown_connections 2024-08-28T19:26:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:48.841+0000 7f07537a2640 1 -- 172.21.15.16:0/2369001524 wait complete. 2024-08-28T19:26:48.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.21 MB/sec execute 148 sec latency 6846.522 ms 2024-08-28T19:26:49.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:49 smithi016 ceph-mon[18556]: pgmap v512: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-08-28T19:26:49.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:49 smithi028 ceph-mon[21769]: pgmap v512: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-08-28T19:26:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:49 smithi115 ceph-mon[22184]: pgmap v512: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-08-28T19:26:49.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:26:49.876 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:49.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.18 MB/sec execute 149 sec latency 7846.597 ms 2024-08-28T19:26:50.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.437+0000 7f0705d87640 1 -- 172.21.15.16:0/3181955858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0700102800 msgr2=0x7f0700102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:50.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.437+0000 7f0705d87640 1 --2- 172.21.15.16:0/3181955858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0700102800 0x7f0700102c10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009900 tx=0x7f06f002a410 comp rx=0 tx=0).stop 2024-08-28T19:26:50.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.437+0000 7f0705d87640 1 -- 172.21.15.16:0/3181955858 shutdown_connections 2024-08-28T19:26:50.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.437+0000 7f0705d87640 1 --2- 172.21.15.16:0/3181955858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700108a90 0x7f070010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.437+0000 7f0705d87640 1 --2- 172.21.15.16:0/3181955858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0700108140 0x7f0700108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.437+0000 7f0705d87640 1 --2- 172.21.15.16:0/3181955858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0700102800 0x7f0700102c10 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.437+0000 7f0705d87640 1 -- 172.21.15.16:0/3181955858 >> 172.21.15.16:0/3181955858 conn(0x7f07000794d0 msgr2=0x7f07000798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.437+0000 7f0705d87640 1 -- 172.21.15.16:0/3181955858 shutdown_connections 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.437+0000 7f0705d87640 1 -- 172.21.15.16:0/3181955858 wait complete. 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.438+0000 7f0705d87640 1 Processor -- start 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.438+0000 7f0705d87640 1 -- start start 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.438+0000 7f0705d87640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0700102800 0x7f070019c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.438+0000 7f0705d87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0700108140 0x7f070019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.438+0000 7f0705d87640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700108a90 0x7f07001a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.438+0000 7f0705d87640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0700112710 con 0x7f0700108140 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.438+0000 7f0705d87640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0700112590 con 0x7f0700108a90 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.438+0000 7f0705d87640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0700112890 con 0x7f0700102800 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.438+0000 7f06feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0700108140 0x7f070019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0700108140 0x7f070019ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60526/0 (socket says 172.21.15.16:60526) 2024-08-28T19:26:50.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06feffd640 1 -- 172.21.15.16:0/4209891647 learned_addr learned my addr 172.21.15.16:0/4209891647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0700102800 0x7f070019c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06fffff640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700108a90 0x7f07001a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06feffd640 1 -- 172.21.15.16:0/4209891647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0700102800 msgr2=0x7f070019c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06feffd640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0700102800 0x7f070019c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06feffd640 1 -- 172.21.15.16:0/4209891647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700108a90 msgr2=0x7f07001a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06feffd640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700108a90 0x7f07001a1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06feffd640 1 -- 172.21.15.16:0/4209891647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07001a2410 con 0x7f0700108140 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06fffff640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700108a90 0x7f07001a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06feffd640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0700108140 0x7f070019ca40 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f06f400d740 tx=0x7f06f400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06fcff9640 1 -- 172.21.15.16:0/4209891647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f4017070 con 0x7f0700108140 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06fcff9640 1 -- 172.21.15.16:0/4209891647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06f400de50 con 0x7f0700108140 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f0705d87640 1 -- 172.21.15.16:0/4209891647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07001a2700 con 0x7f0700108140 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f06fcff9640 1 -- 172.21.15.16:0/4209891647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f4004ce0 con 0x7f0700108140 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.439+0000 7f0705d87640 1 -- 172.21.15.16:0/4209891647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07001a8e60 con 0x7f0700108140 2024-08-28T19:26:50.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f070c2fc740 1 Processor -- start 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f070c2fc740 1 -- start start 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704b09070 0x55d704eae310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae01b0 0x55d704eae850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae8390 0x55d704eaed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f070c2fc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d704e25680 con 0x55d704ae01b0 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f070c2fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d704e14c10 con 0x55d704ae8390 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f070c2fc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d704e18bc0 con 0x55d704b09070 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f06ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae01b0 0x55d704eae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f06ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae01b0 0x55d704eae850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60530/0 (socket says 172.21.15.16:60530) 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f06ff7fe640 1 -- 172.21.15.16:0/2305939658 learned_addr learned my addr 172.21.15.16:0/2305939658 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f06fffff640 1 --2- 172.21.15.16:0/2305939658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704b09070 0x55d704eae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f06ff7fe640 1 -- 172.21.15.16:0/2305939658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704b09070 msgr2=0x55d704eae310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/2305939658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704b09070 0x55d704eae310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f06ff7fe640 1 -- 172.21.15.16:0/2305939658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae8390 msgr2=0x55d704eaed90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/2305939658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae8390 0x55d704eaed90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f06ff7fe640 1 -- 172.21.15.16:0/2305939658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d704b74650 con 0x55d704ae01b0 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.442+0000 7f06fffff640 1 --2- 172.21.15.16:0/2305939658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704b09070 0x55d704eae310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.443+0000 7f06fcff9640 1 -- 172.21.15.16:0/4209891647 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f06f4004e80 con 0x7f0700108140 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.443+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/2305939658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae01b0 0x55d704eae850 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009900 tx=0x7f06f0032cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7802b40292e71391 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.443+0000 7f06e5ffb640 1 -- 172.21.15.16:0/2305939658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f0040070 con 0x55d704ae01b0 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.443+0000 7f06e5ffb640 1 -- 172.21.15.16:0/2305939658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06f0032e40 con 0x55d704ae01b0 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.443+0000 7f06e5ffb640 1 -- 172.21.15.16:0/2305939658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f0035040 con 0x55d704ae01b0 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.443+0000 7f06fcff9640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06e80b4ca0 0x7f06e80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.443+0000 7f06fffff640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06e80b4ca0 0x7f06e80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f06fcff9640 1 -- 172.21.15.16:0/4209891647 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f06f40e5270 con 0x7f0700108140 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f070c2fc740 1 -- 172.21.15.16:0/2305939658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae01b0 msgr2=0x55d704eae850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:50.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f070c2fc740 1 --2- 172.21.15.16:0/2305939658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae01b0 0x55d704eae850 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009900 tx=0x7f06f0032cd0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f070c2fc740 1 -- 172.21.15.16:0/2305939658 shutdown_connections 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f070c2fc740 1 --2- 172.21.15.16:0/2305939658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae8390 0x55d704eaed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f070c2fc740 1 --2- 172.21.15.16:0/2305939658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae01b0 0x55d704eae850 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f070c2fc740 1 --2- 172.21.15.16:0/2305939658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704b09070 0x55d704eae310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f070c2fc740 1 -- 172.21.15.16:0/2305939658 >> 172.21.15.16:0/2305939658 conn(0x55d704afd460 msgr2=0x55d704eac200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f070c2fc740 1 -- 172.21.15.16:0/2305939658 shutdown_connections 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f06fffff640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06e80b4ca0 0x7f06e80b71b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f06f8005fd0 tx=0x7f06f8009990 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f070c2fc740 1 -- 172.21.15.16:0/2305939658 wait complete. 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.444+0000 7f070c2fc740 1 -- start start 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f070c2fc740 1 Processor -- start 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704ae01b0 0x55d704eb9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae8390 0x55d704eba000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704b09070 0x55d704ec5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f070c2fc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d704c77700 con 0x55d704b09070 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f070c2fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d704d9b750 con 0x55d704ae8390 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f070c2fc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d704ebca90 con 0x55d704ae01b0 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f070c2fc740 10 client.0 _ll_register_callbacks cb 0x55d704dcae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f06fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704b09070 0x55d704ec5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f06fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704b09070 0x55d704ec5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60540/0 (socket says 172.21.15.16:60540) 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f06fffff640 1 -- 172.21.15.16:0/178755560 learned_addr learned my addr 172.21.15.16:0/178755560 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704ae01b0 0x55d704eb9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f06feffd640 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae8390 0x55d704eba000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f06fffff640 1 -- 172.21.15.16:0/178755560 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704ae01b0 msgr2=0x55d704eb9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f06fffff640 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704ae01b0 0x55d704eb9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f06fffff640 1 -- 172.21.15.16:0/178755560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae8390 msgr2=0x55d704eba000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f06fffff640 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae8390 0x55d704eba000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.445+0000 7f06fffff640 1 -- 172.21.15.16:0/178755560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d704dd4900 con 0x55d704b09070 2024-08-28T19:26:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f06fffff640 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704b09070 0x55d704ec5de0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f06f801d8c0 tx=0x7f06f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f285f954ac2b5e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f06deffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f06deffd640 1 -- 172.21.15.16:0/178755560 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f8004d20 con 0x55d704b09070 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f06deffd640 1 -- 172.21.15.16:0/178755560 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f06f801e6e0 con 0x55d704b09070 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f070c2fc740 1 -- 172.21.15.16:0/178755560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d704e275d0 con 0x55d704b09070 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f06deffd640 1 -- 172.21.15.16:0/178755560 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f06f8030050 con 0x55d704b09070 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f070c2fc740 10 client.17103 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f070c2fc740 1 -- 172.21.15.16:0/178755560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55d704ec8900 con 0x55d704b09070 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f06deffd640 1 -- 172.21.15.16:0/178755560 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d704e275d0 con 0x55d704b09070 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f06deffd640 1 -- 172.21.15.16:0/178755560 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f06f800f040 con 0x55d704b09070 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f070c2fc740 10 client.17103 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f070c2fc740 10 client.17103 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.446+0000 7f070c2fc740 20 client.17103 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.447+0000 7f070c2fc740 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55d704ac79f0 0x55d704ecbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.447+0000 7f070c2fc740 4 client.17103 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.447+0000 7f070c2fc740 1 -- 172.21.15.16:0/178755560 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d704e23070 con 0x55d704ac79f0 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.447+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55d704ac79f0 0x55d704ecbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.447+0000 7f06deffd640 10 client.17103 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.447+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55d704ac79f0 0x55d704ecbf20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f2f45916c43c50ff server_cookie=dc20fe8603cb9705 in_seq=0 out_seq=0 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f06deffd640 1 -- 172.21.15.16:0/178755560 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d704e23070 con 0x55d704ac79f0 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f06deffd640 10 client.17103 handle_command_reply: tid=0 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 1 client.17103 shutdown 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 1 -- 172.21.15.16:0/178755560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704b09070 msgr2=0x55d704ec5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704b09070 0x55d704ec5de0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f06f801d8c0 tx=0x7f06f8004a40 comp rx=0 tx=0).stop 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 1 -- 172.21.15.16:0/178755560 shutdown_connections 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55d704ac79f0 0x55d704ecbf20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704b09070 0x55d704ec5de0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae8390 0x55d704eba000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 1 --2- 172.21.15.16:0/178755560 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704ae01b0 0x55d704eb9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f06deffd640 0 client.17103 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 1 -- 172.21.15.16:0/178755560 >> 172.21.15.16:0/178755560 conn(0x55d704afd460 msgr2=0x55d704eb14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 1 -- 172.21.15.16:0/178755560 shutdown_connections 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 1 -- 172.21.15.16:0/178755560 wait complete. 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.448+0000 7f070c2fc740 20 client.17103 trim_cache size 0 max 16384 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f070c2fc740 1 Processor -- start 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f070c2fc740 1 -- start start 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704ae01b0 0x55d704eac270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ec5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704b09070 0x55d704ec6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f070c2fc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07001b2de0 con 0x55d704ae8390 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f070c2fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07001b4950 con 0x55d704b09070 2024-08-28T19:26:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f070c2fc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f07001b34b0 con 0x55d704ae01b0 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f06feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ec5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f06feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ec5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60546/0 (socket says 172.21.15.16:60546) 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f06feffd640 1 -- 172.21.15.16:0/2140489198 learned_addr learned my addr 172.21.15.16:0/2140489198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f06fffff640 1 --2- 172.21.15.16:0/2140489198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704b09070 0x55d704ec6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/2140489198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704ae01b0 0x55d704eac270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f06feffd640 1 -- 172.21.15.16:0/2140489198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704ae01b0 msgr2=0x55d704eac270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f06feffd640 1 --2- 172.21.15.16:0/2140489198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704ae01b0 0x55d704eac270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f06feffd640 1 -- 172.21.15.16:0/2140489198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704b09070 msgr2=0x55d704ec6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f06feffd640 1 --2- 172.21.15.16:0/2140489198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704b09070 0x55d704ec6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f06feffd640 1 -- 172.21.15.16:0/2140489198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d704eb7fc0 con 0x55d704ae8390 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.462+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/2140489198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704ae01b0 0x55d704eac270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.463+0000 7f06feffd640 1 --2- 172.21.15.16:0/2140489198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ec5ce0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009610 tx=0x7f06f40aae20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca7b4d73a070970 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.463+0000 7f06e57fa640 1 -- 172.21.15.16:0/2140489198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f4020040 con 0x55d704ae8390 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.463+0000 7f06e57fa640 1 -- 172.21.15.16:0/2140489198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06f40abcf0 con 0x55d704ae8390 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.463+0000 7f06e57fa640 1 -- 172.21.15.16:0/2140489198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f40a7880 con 0x55d704ae8390 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.463+0000 7f070c2fc740 1 -- 172.21.15.16:0/2140489198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 msgr2=0x55d704ec5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.463+0000 7f070c2fc740 1 --2- 172.21.15.16:0/2140489198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ec5ce0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009610 tx=0x7f06f40aae20 comp rx=0 tx=0).stop 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.463+0000 7f070c2fc740 1 -- 172.21.15.16:0/2140489198 shutdown_connections 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.463+0000 7f070c2fc740 1 --2- 172.21.15.16:0/2140489198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704b09070 0x55d704ec6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 --2- 172.21.15.16:0/2140489198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ec5ce0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 --2- 172.21.15.16:0/2140489198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704ae01b0 0x55d704eac270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 -- 172.21.15.16:0/2140489198 >> 172.21.15.16:0/2140489198 conn(0x55d704afd460 msgr2=0x55d704eabb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 -- 172.21.15.16:0/2140489198 shutdown_connections 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 -- 172.21.15.16:0/2140489198 wait complete. 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 -- start start 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 Processor -- start 2024-08-28T19:26:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae01b0 0x55d704ebe440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ebe980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704b09070 0x55d704f358e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07001b4280 con 0x55d704ae8390 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07001b3b80 con 0x55d704ae01b0 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.464+0000 7f070c2fc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f07001b0dc0 con 0x55d704b09070 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f070c2fc740 10 client.0 _ll_register_callbacks cb 0x7f06f8009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ebe980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae01b0 0x55d704ebe440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ebe980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60562/0 (socket says 172.21.15.16:60562) 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06feffd640 1 -- 172.21.15.16:0/1140966150 learned_addr learned my addr 172.21.15.16:0/1140966150 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06fffff640 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704b09070 0x55d704f358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06ff7fe640 1 -- 172.21.15.16:0/1140966150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704b09070 msgr2=0x55d704f358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704b09070 0x55d704f358e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06ff7fe640 1 -- 172.21.15.16:0/1140966150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 msgr2=0x55d704ebe980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ebe980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06ff7fe640 1 -- 172.21.15.16:0/1140966150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d704f35e20 con 0x55d704ae01b0 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06feffd640 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ebe980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06ff7fe640 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae01b0 0x55d704ebe440 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f06f0031440 tx=0x7f06f0005260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dbbbba7a94b96b7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06deffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06deffd640 1 -- 172.21.15.16:0/1140966150 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f0051070 con 0x55d704ae01b0 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f070c2fc740 1 -- 172.21.15.16:0/1140966150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06f40abcf0 con 0x55d704ae01b0 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.465+0000 7f06deffd640 1 -- 172.21.15.16:0/1140966150 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f06f00318b0 con 0x55d704ae01b0 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.466+0000 7f06deffd640 1 -- 172.21.15.16:0/1140966150 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f06f0035040 con 0x55d704ae01b0 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.466+0000 7f070c2fc740 10 client.26666 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.466+0000 7f070c2fc740 1 -- 172.21.15.16:0/1140966150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55d704f37080 con 0x55d704ae01b0 2024-08-28T19:26:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.466+0000 7f06deffd640 1 -- 172.21.15.16:0/1140966150 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06f40abcf0 con 0x55d704ae01b0 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.466+0000 7f06deffd640 1 -- 172.21.15.16:0/1140966150 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f06f0003b00 con 0x55d704ae01b0 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.466+0000 7f070c2fc740 10 client.26666 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.466+0000 7f070c2fc740 10 client.26666 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.466+0000 7f070c2fc740 20 client.26666 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.466+0000 7f070c2fc740 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55d704a9cda0 0x55d704f393d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.466+0000 7f070c2fc740 4 client.26666 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.466+0000 7f070c2fc740 1 -- 172.21.15.16:0/1140966150 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d704eba430 con 0x55d704a9cda0 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f06feffd640 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55d704a9cda0 0x55d704f393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f06deffd640 10 client.26666 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f06feffd640 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55d704a9cda0 0x55d704f393d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d7f1f8a9be344d23 server_cookie=3eaec7280e700ece in_seq=0 out_seq=0 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f06deffd640 1 -- 172.21.15.16:0/1140966150 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d704eba430 con 0x55d704a9cda0 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f06deffd640 10 client.26666 handle_command_reply: tid=0 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f070c2fc740 1 client.26666 shutdown 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f070c2fc740 1 -- 172.21.15.16:0/1140966150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae01b0 msgr2=0x55d704ebe440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f070c2fc740 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae01b0 0x55d704ebe440 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f06f0031440 tx=0x7f06f0005260 comp rx=0 tx=0).stop 2024-08-28T19:26:50.704 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:26:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f070c2fc740 1 -- 172.21.15.16:0/1140966150 shutdown_connections 2024-08-28T19:26:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f070c2fc740 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55d704a9cda0 0x55d704f393d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f070c2fc740 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d704b09070 0x55d704f358e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f070c2fc740 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d704ae8390 0x55d704ebe980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f06deffd640 0 client.26666 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:26:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f070c2fc740 1 --2- 172.21.15.16:0/1140966150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d704ae01b0 0x55d704ebe440 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f070c2fc740 1 -- 172.21.15.16:0/1140966150 >> 172.21.15.16:0/1140966150 conn(0x55d704afd460 msgr2=0x55d704eb1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f070c2fc740 1 -- 172.21.15.16:0/1140966150 shutdown_connections 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.467+0000 7f070c2fc740 1 -- 172.21.15.16:0/1140966150 wait complete. 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f070c2fc740 20 client.26666 trim_cache size 0 max 16384 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 -- 172.21.15.16:0/4209891647 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06e80b4ca0 msgr2=0x7f06e80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06e80b4ca0 0x7f06e80b71b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f06f8005fd0 tx=0x7f06f8009990 comp rx=0 tx=0).stop 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 -- 172.21.15.16:0/4209891647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0700108140 msgr2=0x7f070019ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0700108140 0x7f070019ca40 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f06f400d740 tx=0x7f06f400dc00 comp rx=0 tx=0).stop 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 -- 172.21.15.16:0/4209891647 shutdown_connections 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06e80b4ca0 0x7f06e80b71b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700108a90 0x7f07001a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0700108140 0x7f070019ca40 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 --2- 172.21.15.16:0/4209891647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0700102800 0x7f070019c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 -- 172.21.15.16:0/4209891647 >> 172.21.15.16:0/4209891647 conn(0x7f07000794d0 msgr2=0x7f070006acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 -- 172.21.15.16:0/4209891647 shutdown_connections 2024-08-28T19:26:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:50.468+0000 7f0705d87640 1 -- 172.21.15.16:0/4209891647 wait complete. 2024-08-28T19:26:50.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.14 MB/sec execute 150 sec latency 8846.646 ms 2024-08-28T19:26:51.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:51 smithi016 ceph-mon[18556]: pgmap v513: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 124 op/s 2024-08-28T19:26:51.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:51 smithi028 ceph-mon[21769]: pgmap v513: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 124 op/s 2024-08-28T19:26:51.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:26:51.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:51 smithi115 ceph-mon[22184]: pgmap v513: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 124 op/s 2024-08-28T19:26:51.829 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:51.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.11 MB/sec execute 151 sec latency 9846.719 ms 2024-08-28T19:26:52.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.324+0000 7fa16013a640 1 -- 172.21.15.16:0/2672455132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa158108ab0 msgr2=0x7fa15810f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:52.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.324+0000 7fa16013a640 1 --2- 172.21.15.16:0/2672455132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa158108ab0 0x7fa15810f2e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa150009ca0 tx=0x7fa15002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.324+0000 7fa16013a640 1 -- 172.21.15.16:0/2672455132 shutdown_connections 2024-08-28T19:26:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.324+0000 7fa16013a640 1 --2- 172.21.15.16:0/2672455132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa158108ab0 0x7fa15810f2e0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.324+0000 7fa16013a640 1 --2- 172.21.15.16:0/2672455132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa158108160 0x7fa158108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.324+0000 7fa16013a640 1 --2- 172.21.15.16:0/2672455132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa158102820 0x7fa158102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.324+0000 7fa16013a640 1 -- 172.21.15.16:0/2672455132 >> 172.21.15.16:0/2672455132 conn(0x7fa1580794e0 msgr2=0x7fa1580798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.324+0000 7fa16013a640 1 -- 172.21.15.16:0/2672455132 shutdown_connections 2024-08-28T19:26:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.325+0000 7fa16013a640 1 -- 172.21.15.16:0/2672455132 wait complete. 2024-08-28T19:26:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.325+0000 7fa16013a640 1 Processor -- start 2024-08-28T19:26:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.325+0000 7fa16013a640 1 -- start start 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.325+0000 7fa16013a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa158102820 0x7fa15819b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.325+0000 7fa16013a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa158108160 0x7fa15819b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa16013a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa158108ab0 0x7fa15819bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa16013a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa158112800 con 0x7fa158108ab0 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa16013a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa158112680 con 0x7fa158102820 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa16013a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa158112980 con 0x7fa158108160 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15e6d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa158108ab0 0x7fa15819bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15e6d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa158108ab0 0x7fa15819bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60570/0 (socket says 172.21.15.16:60570) 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15e6d7640 1 -- 172.21.15.16:0/1669438327 learned_addr learned my addr 172.21.15.16:0/1669438327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15ded6640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa158102820 0x7fa15819b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa158108160 0x7fa15819b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15e6d7640 1 -- 172.21.15.16:0/1669438327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa158108160 msgr2=0x7fa15819b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15e6d7640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa158108160 0x7fa15819b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15e6d7640 1 -- 172.21.15.16:0/1669438327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa158102820 msgr2=0x7fa15819b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15e6d7640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa158102820 0x7fa15819b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15e6d7640 1 -- 172.21.15.16:0/1669438327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1581a6400 con 0x7fa158108ab0 2024-08-28T19:26:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15ded6640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa158102820 0x7fa15819b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa15e6d7640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa158108ab0 0x7fa15819bb70 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa150005cc0 tx=0x7fa15002f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.326+0000 7fa14effd640 1 -- 172.21.15.16:0/1669438327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa150003cf0 con 0x7fa158108ab0 2024-08-28T19:26:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.327+0000 7fa16013a640 1 -- 172.21.15.16:0/1669438327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1581a6690 con 0x7fa158108ab0 2024-08-28T19:26:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.327+0000 7fa14effd640 1 -- 172.21.15.16:0/1669438327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1500044f0 con 0x7fa158108ab0 2024-08-28T19:26:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.327+0000 7fa14effd640 1 -- 172.21.15.16:0/1669438327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15002fd10 con 0x7fa158108ab0 2024-08-28T19:26:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.327+0000 7fa16013a640 1 -- 172.21.15.16:0/1669438327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1581a6ca0 con 0x7fa158108ab0 2024-08-28T19:26:52.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.329+0000 7fa1666af740 1 Processor -- start 2024-08-28T19:26:52.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.329+0000 7fa14effd640 1 -- 172.21.15.16:0/1669438327 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa150004040 con 0x7fa158108ab0 2024-08-28T19:26:52.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.329+0000 7fa1666af740 1 -- start start 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa1666af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd08e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa1666af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc01b0 0x5589bd08e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa1666af740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc8390 0x5589bd08ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa1666af740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589bd005680 con 0x5589bcce9070 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa1666af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5589bcff4c10 con 0x5589bccc01b0 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa1666af740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5589bcff8bc0 con 0x5589bccc8390 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15d6d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd08e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15d6d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd08e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60572/0 (socket says 172.21.15.16:60572) 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15d6d5640 1 -- 172.21.15.16:0/1173687882 learned_addr learned my addr 172.21.15.16:0/1173687882 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa14effd640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1440b4ca0 0x7fa1440b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15ded6640 1 --2- 172.21.15.16:0/1173687882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc01b0 0x5589bd08e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/1173687882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc8390 0x5589bd08ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15d6d5640 1 -- 172.21.15.16:0/1173687882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc8390 msgr2=0x5589bd08ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:52.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/1173687882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc8390 0x5589bd08ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15d6d5640 1 -- 172.21.15.16:0/1173687882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc01b0 msgr2=0x5589bd08e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:52.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/1173687882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc01b0 0x5589bd08e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15d6d5640 1 -- 172.21.15.16:0/1173687882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589bcd54650 con 0x5589bcce9070 2024-08-28T19:26:52.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15ded6640 1 --2- 172.21.15.16:0/1173687882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc01b0 0x5589bd08e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:52.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.330+0000 7fa15e6d7640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1440b4ca0 0x7fa1440b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.331+0000 7fa14effd640 1 -- 172.21.15.16:0/1669438327 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa150107060 con 0x7fa158108ab0 2024-08-28T19:26:52.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.331+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/1173687882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd08e310 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa14800bbc0 tx=0x7fa14800b400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356a71bce8f951b9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:52.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.331+0000 7fa143fff640 1 -- 172.21.15.16:0/1173687882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14801a070 con 0x5589bcce9070 2024-08-28T19:26:52.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.331+0000 7fa143fff640 1 -- 172.21.15.16:0/1173687882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa148010040 con 0x5589bcce9070 2024-08-28T19:26:52.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.331+0000 7fa143fff640 1 -- 172.21.15.16:0/1173687882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa148013570 con 0x5589bcce9070 2024-08-28T19:26:52.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa1666af740 1 -- 172.21.15.16:0/1173687882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 msgr2=0x5589bd08e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:52.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa1666af740 1 --2- 172.21.15.16:0/1173687882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd08e310 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa14800bbc0 tx=0x7fa14800b400 comp rx=0 tx=0).stop 2024-08-28T19:26:52.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa15e6d7640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1440b4ca0 0x7fa1440b71b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa150031a90 tx=0x7fa15004a000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:52.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa1666af740 1 -- 172.21.15.16:0/1173687882 shutdown_connections 2024-08-28T19:26:52.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa1666af740 1 --2- 172.21.15.16:0/1173687882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc8390 0x5589bd08ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa1666af740 1 --2- 172.21.15.16:0/1173687882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc01b0 0x5589bd08e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa1666af740 1 --2- 172.21.15.16:0/1173687882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd08e310 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa1666af740 1 -- 172.21.15.16:0/1173687882 >> 172.21.15.16:0/1173687882 conn(0x5589bccdd460 msgr2=0x5589bd08c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa1666af740 1 -- 172.21.15.16:0/1173687882 shutdown_connections 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa1666af740 1 -- 172.21.15.16:0/1173687882 wait complete. 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa1666af740 1 -- start start 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.332+0000 7fa1666af740 1 Processor -- start 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa1666af740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc01b0 0x5589bd099b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa1666af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc8390 0x5589bd09a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa1666af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bcce9070 0x5589bd0a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa1666af740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589bce57700 con 0x5589bccc8390 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa1666af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5589bcf7b750 con 0x5589bcce9070 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa1666af740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5589bd09caf0 con 0x5589bccc01b0 2024-08-28T19:26:52.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa1666af740 10 client.0 _ll_register_callbacks cb 0x5589bcfaae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa15ded6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc8390 0x5589bd09a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa15ded6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc8390 0x5589bd09a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60576/0 (socket says 172.21.15.16:60576) 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa15ded6640 1 -- 172.21.15.16:0/884216911 learned_addr learned my addr 172.21.15.16:0/884216911 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bcce9070 0x5589bd0a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc01b0 0x5589bd099b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa15ded6640 1 -- 172.21.15.16:0/884216911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc01b0 msgr2=0x5589bd099b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa15ded6640 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc01b0 0x5589bd099b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa15ded6640 1 -- 172.21.15.16:0/884216911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bcce9070 msgr2=0x5589bd0a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa15ded6640 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bcce9070 0x5589bd0a5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa15ded6640 1 -- 172.21.15.16:0/884216911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589bcfb4900 con 0x5589bccc8390 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.333+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bcce9070 0x5589bd0a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa15ded6640 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc8390 0x5589bd09a070 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa15400bf70 tx=0x7fa15400f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=482ee1145abb9c6b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:52.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa1417fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa1417fa640 1 -- 172.21.15.16:0/884216911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa154018020 con 0x5589bccc8390 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa1417fa640 1 -- 172.21.15.16:0/884216911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa154014670 con 0x5589bccc8390 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa1666af740 1 -- 172.21.15.16:0/884216911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa148010040 con 0x5589bccc8390 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa1417fa640 1 -- 172.21.15.16:0/884216911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fa154023050 con 0x5589bccc8390 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa1666af740 10 client.17121 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa1666af740 1 -- 172.21.15.16:0/884216911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5589bd0a8980 con 0x5589bccc8390 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa1417fa640 1 -- 172.21.15.16:0/884216911 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa148010040 con 0x5589bccc8390 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa1417fa640 1 -- 172.21.15.16:0/884216911 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7fa1540139d0 con 0x5589bccc8390 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa1666af740 10 client.17121 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.334+0000 7fa1666af740 10 client.17121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.335+0000 7fa1666af740 20 client.17121 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.335+0000 7fa1666af740 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5589bcca79f0 0x5589bd0abf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.335+0000 7fa1666af740 4 client.17121 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.335+0000 7fa1666af740 1 -- 172.21.15.16:0/884216911 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589bd0075d0 con 0x5589bcca79f0 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.335+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5589bcca79f0 0x5589bd0abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.335+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5589bcca79f0 0x5589bd0abf40 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82d48da895f07df3 server_cookie=5d55530b242a4a29 in_seq=0 out_seq=0 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.335+0000 7fa1417fa640 10 client.17121 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1417fa640 1 -- 172.21.15.16:0/884216911 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5589bd0075d0 con 0x5589bcca79f0 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1417fa640 10 client.17121 handle_command_reply: tid=0 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1666af740 1 client.17121 shutdown 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1666af740 1 -- 172.21.15.16:0/884216911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc8390 msgr2=0x5589bd09a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1666af740 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc8390 0x5589bd09a070 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa15400bf70 tx=0x7fa15400f670 comp rx=0 tx=0).stop 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1666af740 1 -- 172.21.15.16:0/884216911 shutdown_connections 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1666af740 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bcce9070 0x5589bd0a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1666af740 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc8390 0x5589bd09a070 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1666af740 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5589bcca79f0 0x5589bd0abf40 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1666af740 1 --2- 172.21.15.16:0/884216911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc01b0 0x5589bd099b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1666af740 1 -- 172.21.15.16:0/884216911 >> 172.21.15.16:0/884216911 conn(0x5589bccdd460 msgr2=0x5589bd0914f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:52.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1417fa640 0 client.17121 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:52.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1666af740 1 -- 172.21.15.16:0/884216911 shutdown_connections 2024-08-28T19:26:52.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.336+0000 7fa1666af740 1 -- 172.21.15.16:0/884216911 wait complete. 2024-08-28T19:26:52.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.337+0000 7fa1666af740 20 client.17121 trim_cache size 0 max 16384 2024-08-28T19:26:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa1666af740 1 Processor -- start 2024-08-28T19:26:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa1666af740 1 -- start start 2024-08-28T19:26:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa1666af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc01b0 0x5589bd08fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa1666af740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc8390 0x5589bd0a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa1666af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd0a6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa1666af740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581b0d00 con 0x5589bcce9070 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa1666af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581b2870 con 0x5589bccc01b0 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa1666af740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581b13d0 con 0x5589bccc8390 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa15d6d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd0a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa15d6d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd0a6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60578/0 (socket says 172.21.15.16:60578) 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa15d6d5640 1 -- 172.21.15.16:0/747276020 learned_addr learned my addr 172.21.15.16:0/747276020 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/747276020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc01b0 0x5589bd08fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa15ded6640 1 --2- 172.21.15.16:0/747276020 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc8390 0x5589bd0a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa15d6d5640 1 -- 172.21.15.16:0/747276020 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc8390 msgr2=0x5589bd0a5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/747276020 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc8390 0x5589bd0a5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa15d6d5640 1 -- 172.21.15.16:0/747276020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc01b0 msgr2=0x5589bd08fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.350+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/747276020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc01b0 0x5589bd08fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.351+0000 7fa15d6d5640 1 -- 172.21.15.16:0/747276020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589bd090600 con 0x5589bcce9070 2024-08-28T19:26:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.351+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/747276020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd0a6220 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa148015d80 tx=0x7fa14800b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad46b4d744fd0a13 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:52.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.351+0000 7fa1437fe640 1 -- 172.21.15.16:0/747276020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14801a040 con 0x5589bcce9070 2024-08-28T19:26:52.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.351+0000 7fa1437fe640 1 -- 172.21.15.16:0/747276020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1480157a0 con 0x5589bcce9070 2024-08-28T19:26:52.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.351+0000 7fa1437fe640 1 -- 172.21.15.16:0/747276020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa148025dc0 con 0x5589bcce9070 2024-08-28T19:26:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.352+0000 7fa1666af740 1 -- 172.21.15.16:0/747276020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 msgr2=0x5589bd0a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.352+0000 7fa1666af740 1 --2- 172.21.15.16:0/747276020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd0a6220 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa148015d80 tx=0x7fa14800b970 comp rx=0 tx=0).stop 2024-08-28T19:26:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.352+0000 7fa1666af740 1 -- 172.21.15.16:0/747276020 shutdown_connections 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.352+0000 7fa1666af740 1 --2- 172.21.15.16:0/747276020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bcce9070 0x5589bd0a6220 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.352+0000 7fa1666af740 1 --2- 172.21.15.16:0/747276020 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bccc8390 0x5589bd0a5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.352+0000 7fa1666af740 1 --2- 172.21.15.16:0/747276020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc01b0 0x5589bd08fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.352+0000 7fa1666af740 1 -- 172.21.15.16:0/747276020 >> 172.21.15.16:0/747276020 conn(0x5589bccdd460 msgr2=0x5589bd08f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.352+0000 7fa1666af740 1 -- 172.21.15.16:0/747276020 shutdown_connections 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.352+0000 7fa1666af740 1 -- 172.21.15.16:0/747276020 wait complete. 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.352+0000 7fa1666af740 1 -- start start 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.352+0000 7fa1666af740 1 Processor -- start 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa1666af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc01b0 0x5589bd09e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa1666af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc8390 0x5589bd09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa1666af740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bcce9070 0x5589bd10f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa1666af740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581b21a0 con 0x5589bccc01b0 2024-08-28T19:26:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa1666af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581b1aa0 con 0x5589bccc8390 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa1666af740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581aece0 con 0x5589bcce9070 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa1666af740 10 client.0 _ll_register_callbacks cb 0x7fa15400b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15d6d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc01b0 0x5589bd09e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15d6d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc01b0 0x5589bd09e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60588/0 (socket says 172.21.15.16:60588) 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15d6d5640 1 -- 172.21.15.16:0/2801755073 learned_addr learned my addr 172.21.15.16:0/2801755073 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15ded6640 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc8390 0x5589bd09ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bcce9070 0x5589bd10f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15d6d5640 1 -- 172.21.15.16:0/2801755073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bcce9070 msgr2=0x5589bd10f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bcce9070 0x5589bd10f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15d6d5640 1 -- 172.21.15.16:0/2801755073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc8390 msgr2=0x5589bd09ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc8390 0x5589bd09ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15d6d5640 1 -- 172.21.15.16:0/2801755073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589bd110020 con 0x5589bccc01b0 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15ded6640 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc8390 0x5589bd09ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa15d6d5640 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc01b0 0x5589bd09e6e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fa148004700 tx=0x7fa14800bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec52881c8b85b248 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa1417fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa1417fa640 1 -- 172.21.15.16:0/2801755073 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14801a070 con 0x5589bccc01b0 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa1417fa640 1 -- 172.21.15.16:0/2801755073 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fa148025600 con 0x5589bccc01b0 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.353+0000 7fa1666af740 1 -- 172.21.15.16:0/2801755073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1480157a0 con 0x5589bccc01b0 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.354+0000 7fa1417fa640 1 -- 172.21.15.16:0/2801755073 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fa148040050 con 0x5589bccc01b0 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.354+0000 7fa1666af740 10 client.17127 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.354+0000 7fa1666af740 1 -- 172.21.15.16:0/2801755073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5589bd117340 con 0x5589bccc01b0 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.354+0000 7fa1417fa640 1 -- 172.21.15.16:0/2801755073 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1480157a0 con 0x5589bccc01b0 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.354+0000 7fa1417fa640 1 -- 172.21.15.16:0/2801755073 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7fa148020040 con 0x5589bccc01b0 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.354+0000 7fa1666af740 10 client.17127 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.354+0000 7fa1666af740 10 client.17127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.354+0000 7fa1666af740 20 client.17127 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.354+0000 7fa1666af740 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5589bcc7cda0 0x5589bd119780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.354+0000 7fa1666af740 4 client.17127 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.354+0000 7fa1666af740 1 -- 172.21.15.16:0/2801755073 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5589bd003070 con 0x5589bcc7cda0 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.355+0000 7fa15ded6640 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5589bcc7cda0 0x5589bd119780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.355+0000 7fa1417fa640 10 client.17127 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.355+0000 7fa15ded6640 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5589bcc7cda0 0x5589bd119780 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c3c50f166dd55b3 server_cookie=264ce4c06b494bbb in_seq=0 out_seq=0 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.355+0000 7fa1417fa640 1 -- 172.21.15.16:0/2801755073 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5589bd003070 con 0x5589bcc7cda0 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.355+0000 7fa1417fa640 10 client.17127 handle_command_reply: tid=0 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.355+0000 7fa1666af740 1 client.17127 shutdown 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.355+0000 7fa1666af740 1 -- 172.21.15.16:0/2801755073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc01b0 msgr2=0x5589bd09e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.355+0000 7fa1666af740 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc01b0 0x5589bd09e6e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fa148004700 tx=0x7fa14800bad0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa1666af740 1 -- 172.21.15.16:0/2801755073 shutdown_connections 2024-08-28T19:26:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa1666af740 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5589bcc7cda0 0x5589bd119780 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa1666af740 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5589bcce9070 0x5589bd10f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa1666af740 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589bccc8390 0x5589bd09ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa1666af740 1 --2- 172.21.15.16:0/2801755073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589bccc01b0 0x5589bd09e6e0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa1417fa640 0 client.17127 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa1666af740 1 -- 172.21.15.16:0/2801755073 >> 172.21.15.16:0/2801755073 conn(0x5589bccdd460 msgr2=0x5589bd09b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa1666af740 1 -- 172.21.15.16:0/2801755073 shutdown_connections 2024-08-28T19:26:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa1666af740 1 -- 172.21.15.16:0/2801755073 wait complete. 2024-08-28T19:26:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa1666af740 20 client.17127 trim_cache size 0 max 16384 2024-08-28T19:26:52.362 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:26:52.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa16013a640 1 -- 172.21.15.16:0/1669438327 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1440b4ca0 msgr2=0x7fa1440b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:52.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa16013a640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1440b4ca0 0x7fa1440b71b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa150031a90 tx=0x7fa15004a000 comp rx=0 tx=0).stop 2024-08-28T19:26:52.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa16013a640 1 -- 172.21.15.16:0/1669438327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa158108ab0 msgr2=0x7fa15819bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:52.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa16013a640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa158108ab0 0x7fa15819bb70 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa150005cc0 tx=0x7fa15002f470 comp rx=0 tx=0).stop 2024-08-28T19:26:52.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa16013a640 1 -- 172.21.15.16:0/1669438327 shutdown_connections 2024-08-28T19:26:52.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa16013a640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa158108ab0 0x7fa15819bb70 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa16013a640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa158108160 0x7fa15819b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa16013a640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1440b4ca0 0x7fa1440b71b0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa16013a640 1 --2- 172.21.15.16:0/1669438327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa158102820 0x7fa15819b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:52.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.356+0000 7fa16013a640 1 -- 172.21.15.16:0/1669438327 >> 172.21.15.16:0/1669438327 conn(0x7fa1580794e0 msgr2=0x7fa1581050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:52.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.357+0000 7fa16013a640 1 -- 172.21.15.16:0/1669438327 shutdown_connections 2024-08-28T19:26:52.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:52.357+0000 7fa16013a640 1 -- 172.21.15.16:0/1669438327 wait complete. 2024-08-28T19:26:52.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.07 MB/sec execute 152 sec latency 10846.821 ms 2024-08-28T19:26:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:53 smithi016 ceph-mon[18556]: pgmap v514: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 854 KiB/s wr, 115 op/s 2024-08-28T19:26:53.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:53 smithi028 ceph-mon[21769]: pgmap v514: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 854 KiB/s wr, 115 op/s 2024-08-28T19:26:53.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:53 smithi115 ceph-mon[22184]: pgmap v514: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 854 KiB/s wr, 115 op/s 2024-08-28T19:26:53.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.04 MB/sec execute 153 sec latency 11846.909 ms 2024-08-28T19:26:54.136 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:26:54.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:26:54.409 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:54.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 5.01 MB/sec execute 154 sec latency 12846.968 ms 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.861+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3582698850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e8108ab0 msgr2=0x7ff9e810f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.861+0000 7ff9efbf9640 1 --2- 172.21.15.16:0/3582698850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e8108ab0 0x7ff9e810f2e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009ca0 tx=0x7ff9e002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.862+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3582698850 shutdown_connections 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.862+0000 7ff9efbf9640 1 --2- 172.21.15.16:0/3582698850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e8108ab0 0x7ff9e810f2e0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.862+0000 7ff9efbf9640 1 --2- 172.21.15.16:0/3582698850 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9e8108160 0x7ff9e8108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.862+0000 7ff9efbf9640 1 --2- 172.21.15.16:0/3582698850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e8102820 0x7ff9e8102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.862+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3582698850 >> 172.21.15.16:0/3582698850 conn(0x7ff9e80794e0 msgr2=0x7ff9e80798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.862+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3582698850 shutdown_connections 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.862+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3582698850 wait complete. 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9efbf9640 1 Processor -- start 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9efbf9640 1 -- start start 2024-08-28T19:26:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9efbf9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e8102820 0x7ff9e819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9efbf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e8108160 0x7ff9e819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9efbf9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9e8108ab0 0x7ff9e819fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9efbf9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e8112780 con 0x7ff9e8102820 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9efbf9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e8112600 con 0x7ff9e8108160 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9efbf9640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e8112900 con 0x7ff9e8108ab0 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9ed995640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e8102820 0x7ff9e819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9ed995640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e8102820 0x7ff9e819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49050/0 (socket says 172.21.15.16:49050) 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9ed995640 1 -- 172.21.15.16:0/3897169912 learned_addr learned my addr 172.21.15.16:0/3897169912 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.863+0000 7ff9ed194640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e8108160 0x7ff9e819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.864+0000 7ff9ed995640 1 -- 172.21.15.16:0/3897169912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9e8108ab0 msgr2=0x7ff9e819fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.864+0000 7ff9ed995640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9e8108ab0 0x7ff9e819fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.864+0000 7ff9ed995640 1 -- 172.21.15.16:0/3897169912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e8108160 msgr2=0x7ff9e819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.864+0000 7ff9ed995640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e8108160 0x7ff9e819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.864+0000 7ff9ed995640 1 -- 172.21.15.16:0/3897169912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e81a0190 con 0x7ff9e8102820 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.864+0000 7ff9ed995640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e8102820 0x7ff9e819a220 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400d950 tx=0x7ff9e400dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.864+0000 7ff9deffd640 1 -- 172.21.15.16:0/3897169912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4017070 con 0x7ff9e8102820 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.864+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3897169912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e81a0480 con 0x7ff9e8102820 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.864+0000 7ff9deffd640 1 -- 172.21.15.16:0/3897169912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9e400de80 con 0x7ff9e8102820 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.864+0000 7ff9deffd640 1 -- 172.21.15.16:0/3897169912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e400f620 con 0x7ff9e8102820 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.864+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3897169912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e81a6be0 con 0x7ff9e8102820 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.867+0000 7ff9deffd640 1 -- 172.21.15.16:0/3897169912 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff9e4009c30 con 0x7ff9e8102820 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.867+0000 7ff9f616e740 1 Processor -- start 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.867+0000 7ff9f616e740 1 -- start start 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x56277849f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d0490 0x56277849fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784a0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9f616e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5627784155f0 con 0x5627780d8670 2024-08-28T19:26:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9f616e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562778404b80 con 0x5627780f9350 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9f616e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562778408b30 con 0x5627780d0490 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9deffd640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9d40b4ca0 0x7ff9d40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ed194640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ed194640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784a0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49058/0 (socket says 172.21.15.16:49058) 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ed194640 1 -- 172.21.15.16:0/22905773 learned_addr learned my addr 172.21.15.16:0/22905773 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ed995640 1 --2- 172.21.15.16:0/22905773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d0490 0x56277849fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ee196640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9d40b4ca0 0x7ff9d40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ee196640 1 --2- 172.21.15.16:0/22905773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x56277849f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9deffd640 1 -- 172.21.15.16:0/3897169912 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff9e40e5530 con 0x7ff9e8102820 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ed194640 1 -- 172.21.15.16:0/22905773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d0490 msgr2=0x56277849fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ed194640 1 --2- 172.21.15.16:0/22905773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d0490 0x56277849fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ed194640 1 -- 172.21.15.16:0/22905773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 msgr2=0x56277849f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:54.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ed194640 1 --2- 172.21.15.16:0/22905773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x56277849f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ed194640 1 -- 172.21.15.16:0/22905773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562778164930 con 0x5627780d8670 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ed995640 1 --2- 172.21.15.16:0/22905773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d0490 0x56277849fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ee196640 1 --2- 172.21.15.16:0/22905773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x56277849f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ed194640 1 --2- 172.21.15.16:0/22905773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784a0200 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009a00 tx=0x7ff9d8009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a32c521edb6f76 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.868+0000 7ff9ee196640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9d40b4ca0 0x7ff9d40b71b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00094e0 tx=0x7ff9e0009360 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.869+0000 7ff9d3fff640 1 -- 172.21.15.16:0/22905773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8017070 con 0x5627780d8670 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.869+0000 7ff9d3fff640 1 -- 172.21.15.16:0/22905773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d8013470 con 0x5627780d8670 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.869+0000 7ff9d3fff640 1 -- 172.21.15.16:0/22905773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8012630 con 0x5627780d8670 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.870+0000 7ff9f616e740 1 -- 172.21.15.16:0/22905773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 msgr2=0x5627784a0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.870+0000 7ff9f616e740 1 --2- 172.21.15.16:0/22905773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784a0200 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009a00 tx=0x7ff9d8009ec0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.870+0000 7ff9f616e740 1 -- 172.21.15.16:0/22905773 shutdown_connections 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.870+0000 7ff9f616e740 1 --2- 172.21.15.16:0/22905773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784a0200 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.870+0000 7ff9f616e740 1 --2- 172.21.15.16:0/22905773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d0490 0x56277849fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.870+0000 7ff9f616e740 1 --2- 172.21.15.16:0/22905773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x56277849f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.870+0000 7ff9f616e740 1 -- 172.21.15.16:0/22905773 >> 172.21.15.16:0/22905773 conn(0x5627780ed740 msgr2=0x56277849d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.870+0000 7ff9f616e740 1 -- 172.21.15.16:0/22905773 shutdown_connections 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.870+0000 7ff9f616e740 1 -- 172.21.15.16:0/22905773 wait complete. 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.870+0000 7ff9f616e740 1 -- start start 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.870+0000 7ff9f616e740 1 Processor -- start 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d0490 0x5627784aaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d8670 0x5627784ab460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x5627784b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9f616e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562778267890 con 0x5627780d0490 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9f616e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56277838b790 con 0x5627780f9350 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9f616e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5627784adf60 con 0x5627780d8670 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9f616e740 10 client.0 _ll_register_callbacks cb 0x562778110ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ed995640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d0490 0x5627784aaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ed995640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d0490 0x5627784aaf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49068/0 (socket says 172.21.15.16:49068) 2024-08-28T19:26:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ed995640 1 -- 172.21.15.16:0/2589418414 learned_addr learned my addr 172.21.15.16:0/2589418414 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ee196640 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x5627784b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ed194640 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d8670 0x5627784ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ed995640 1 -- 172.21.15.16:0/2589418414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d8670 msgr2=0x5627784ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ed995640 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d8670 0x5627784ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ed995640 1 -- 172.21.15.16:0/2589418414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 msgr2=0x5627784b7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ed995640 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x5627784b7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ed995640 1 -- 172.21.15.16:0/2589418414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627783c2f30 con 0x5627780d0490 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ee196640 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x5627784b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ed194640 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d8670 0x5627784ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9ed995640 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d0490 0x5627784aaf20 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff9e40e2f10 tx=0x7ff9e4074a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9575a0e66558dafb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.871+0000 7ff9d17fa640 1 -- 172.21.15.16:0/2589418414 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4020070 con 0x5627780d0490 2024-08-28T19:26:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9d17fa640 1 -- 172.21.15.16:0/2589418414 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff9e40b4950 con 0x5627780d0490 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9f616e740 1 -- 172.21.15.16:0/2589418414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9d8013470 con 0x5627780d0490 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9d17fa640 1 -- 172.21.15.16:0/2589418414 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ff9e40ab3c0 con 0x5627780d0490 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9f616e740 10 client.17139 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9f616e740 1 -- 172.21.15.16:0/2589418414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5627784b9cd0 con 0x5627780d0490 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9d17fa640 1 -- 172.21.15.16:0/2589418414 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d8013470 con 0x5627780d0490 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9d17fa640 1 -- 172.21.15.16:0/2589418414 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7ff9e40acbe0 con 0x5627780d0490 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9f616e740 10 client.17139 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9f616e740 10 client.17139 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9f616e740 20 client.17139 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9f616e740 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5627780b7cd0 0x5627784bd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9f616e740 4 client.17139 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9f616e740 1 -- 172.21.15.16:0/2589418414 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562778417540 con 0x5627780b7cd0 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.872+0000 7ff9ed194640 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5627780b7cd0 0x5627784bd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.873+0000 7ff9d17fa640 10 client.17139 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.873+0000 7ff9ed194640 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5627780b7cd0 0x5627784bd220 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8e9c58773ab30b7 server_cookie=54230b3b40e20e9a in_seq=0 out_seq=0 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9d17fa640 1 -- 172.21.15.16:0/2589418414 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562778417540 con 0x5627780b7cd0 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9d17fa640 10 client.17139 handle_command_reply: tid=0 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 1 client.17139 shutdown 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 1 -- 172.21.15.16:0/2589418414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d0490 msgr2=0x5627784aaf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d0490 0x5627784aaf20 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff9e40e2f10 tx=0x7ff9e4074a20 comp rx=0 tx=0).stop 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 1 -- 172.21.15.16:0/2589418414 shutdown_connections 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x5627784b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5627780b7cd0 0x5627784bd220 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d8670 0x5627784ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 1 --2- 172.21.15.16:0/2589418414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d0490 0x5627784aaf20 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9d17fa640 0 client.17139 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 1 -- 172.21.15.16:0/2589418414 >> 172.21.15.16:0/2589418414 conn(0x5627780ed740 msgr2=0x5627784a2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 1 -- 172.21.15.16:0/2589418414 shutdown_connections 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 1 -- 172.21.15.16:0/2589418414 wait complete. 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.874+0000 7ff9f616e740 20 client.17139 trim_cache size 0 max 16384 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9f616e740 1 Processor -- start 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9f616e740 1 -- start start 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d0490 0x5627784a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x56277849cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x56277849d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9f616e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e81b0c30 con 0x5627780d8670 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9f616e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e81b27a0 con 0x5627780f9350 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9f616e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e81b1300 con 0x5627780d0490 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9ed194640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x56277849cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9ed194640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x56277849cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49080/0 (socket says 172.21.15.16:49080) 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9ed194640 1 -- 172.21.15.16:0/922875573 learned_addr learned my addr 172.21.15.16:0/922875573 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.892+0000 7ff9ed995640 1 --2- 172.21.15.16:0/922875573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d0490 0x5627784a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.893+0000 7ff9ee196640 1 --2- 172.21.15.16:0/922875573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x56277849d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.893+0000 7ff9ed194640 1 -- 172.21.15.16:0/922875573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d0490 msgr2=0x5627784a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.893+0000 7ff9ed194640 1 --2- 172.21.15.16:0/922875573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d0490 0x5627784a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.893+0000 7ff9ed194640 1 -- 172.21.15.16:0/922875573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 msgr2=0x56277849d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.893+0000 7ff9ed194640 1 --2- 172.21.15.16:0/922875573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x56277849d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.893+0000 7ff9ed194640 1 -- 172.21.15.16:0/922875573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627784a1a70 con 0x5627780d8670 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.893+0000 7ff9ee196640 1 --2- 172.21.15.16:0/922875573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x56277849d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.893+0000 7ff9ed194640 1 --2- 172.21.15.16:0/922875573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x56277849cca0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8013e00 tx=0x7ff9d8009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892a046a2669dce1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.893+0000 7ff9d37fe640 1 -- 172.21.15.16:0/922875573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8017040 con 0x5627780d8670 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.893+0000 7ff9d37fe640 1 -- 172.21.15.16:0/922875573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d80139b0 con 0x5627780d8670 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.893+0000 7ff9d37fe640 1 -- 172.21.15.16:0/922875573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d801cde0 con 0x5627780d8670 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.894+0000 7ff9f616e740 1 -- 172.21.15.16:0/922875573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 msgr2=0x56277849cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.894+0000 7ff9f616e740 1 --2- 172.21.15.16:0/922875573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x56277849cca0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8013e00 tx=0x7ff9d8009110 comp rx=0 tx=0).stop 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.894+0000 7ff9f616e740 1 -- 172.21.15.16:0/922875573 shutdown_connections 2024-08-28T19:26:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.894+0000 7ff9f616e740 1 --2- 172.21.15.16:0/922875573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780f9350 0x56277849d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.894+0000 7ff9f616e740 1 --2- 172.21.15.16:0/922875573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x56277849cca0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.894+0000 7ff9f616e740 1 --2- 172.21.15.16:0/922875573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780d0490 0x5627784a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.894+0000 7ff9f616e740 1 -- 172.21.15.16:0/922875573 >> 172.21.15.16:0/922875573 conn(0x5627780ed740 msgr2=0x5627784a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.894+0000 7ff9f616e740 1 -- 172.21.15.16:0/922875573 shutdown_connections 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.894+0000 7ff9f616e740 1 -- 172.21.15.16:0/922875573 wait complete. 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.894+0000 7ff9f616e740 1 -- start start 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.894+0000 7ff9f616e740 1 Processor -- start 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780d0490 0x5627784af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9f616e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780f9350 0x562778526be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9f616e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e81b20d0 con 0x5627780d8670 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9f616e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e81b19d0 con 0x5627780d0490 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9f616e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e81aec10 con 0x5627780f9350 2024-08-28T19:26:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9f616e740 10 client.0 _ll_register_callbacks cb 0x7ff9e40e2dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9ed194640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9ed194640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784af9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49086/0 (socket says 172.21.15.16:49086) 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9ed194640 1 -- 172.21.15.16:0/4138170048 learned_addr learned my addr 172.21.15.16:0/4138170048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9ed995640 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780d0490 0x5627784af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9ee196640 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780f9350 0x562778526be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9ed194640 1 -- 172.21.15.16:0/4138170048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780f9350 msgr2=0x562778526be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9ed194640 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780f9350 0x562778526be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9ed194640 1 -- 172.21.15.16:0/4138170048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780d0490 msgr2=0x5627784af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9ed194640 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780d0490 0x5627784af4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.895+0000 7ff9ed194640 1 -- 172.21.15.16:0/4138170048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627784b0280 con 0x5627780d8670 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9ed194640 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784af9e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8000c00 tx=0x7ff9d8006a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3323de76236d4f4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9d17fa640 1 -- 172.21.15.16:0/4138170048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8032070 con 0x5627780d8670 2024-08-28T19:26:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9d17fa640 1 -- 172.21.15.16:0/4138170048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7ff9d8013b60 con 0x5627780d8670 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9f616e740 1 -- 172.21.15.16:0/4138170048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9d80139b0 con 0x5627780d8670 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9d17fa640 1 -- 172.21.15.16:0/4138170048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ff9d803e050 con 0x5627780d8670 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9f616e740 10 client.17145 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9f616e740 1 -- 172.21.15.16:0/4138170048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562778528120 con 0x5627780d8670 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9d17fa640 1 -- 172.21.15.16:0/4138170048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d80139b0 con 0x5627780d8670 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9d17fa640 1 -- 172.21.15.16:0/4138170048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7ff9d802c630 con 0x5627780d8670 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9f616e740 10 client.17145 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9f616e740 10 client.17145 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9f616e740 20 client.17145 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9f616e740 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56277808d080 0x56277852a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9f616e740 4 client.17145 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.896+0000 7ff9f616e740 1 -- 172.21.15.16:0/4138170048 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562778412fe0 con 0x56277808d080 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.897+0000 7ff9ed995640 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56277808d080 0x56277852a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.897+0000 7ff9ed995640 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56277808d080 0x56277852a4c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5f90fd58d1b9c10 server_cookie=f5cb5d409d84c467 in_seq=0 out_seq=0 2024-08-28T19:26:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.897+0000 7ff9d17fa640 10 client.17145 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.898+0000 7ff9d17fa640 1 -- 172.21.15.16:0/4138170048 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562778412fe0 con 0x56277808d080 2024-08-28T19:26:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.898+0000 7ff9d17fa640 10 client.17145 handle_command_reply: tid=0 2024-08-28T19:26:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.898+0000 7ff9f616e740 1 client.17145 shutdown 2024-08-28T19:26:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9f616e740 1 -- 172.21.15.16:0/4138170048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 msgr2=0x5627784af9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9f616e740 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784af9e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8000c00 tx=0x7ff9d8006a30 comp rx=0 tx=0).stop 2024-08-28T19:26:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9f616e740 1 -- 172.21.15.16:0/4138170048 shutdown_connections 2024-08-28T19:26:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9f616e740 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56277808d080 0x56277852a4c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.968 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:26:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:26:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "c89de864-c285-403a-8de0-63c8123972fa", 2024-08-28T19:26:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:26:54.969 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:26:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9d17fa640 0 client.17145 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9f616e740 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627780f9350 0x562778526be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9f616e740 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627780d8670 0x5627784af9e0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9f616e740 1 --2- 172.21.15.16:0/4138170048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627780d0490 0x5627784af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9f616e740 1 -- 172.21.15.16:0/4138170048 >> 172.21.15.16:0/4138170048 conn(0x5627780ed740 msgr2=0x5627784a10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9f616e740 1 -- 172.21.15.16:0/4138170048 shutdown_connections 2024-08-28T19:26:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9f616e740 1 -- 172.21.15.16:0/4138170048 wait complete. 2024-08-28T19:26:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9f616e740 20 client.17145 trim_cache size 0 max 16384 2024-08-28T19:26:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3897169912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9d40b4ca0 msgr2=0x7ff9d40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.899+0000 7ff9efbf9640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9d40b4ca0 0x7ff9d40b71b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00094e0 tx=0x7ff9e0009360 comp rx=0 tx=0).stop 2024-08-28T19:26:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.900+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3897169912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e8102820 msgr2=0x7ff9e819a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.900+0000 7ff9efbf9640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e8102820 0x7ff9e819a220 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400d950 tx=0x7ff9e400dd50 comp rx=0 tx=0).stop 2024-08-28T19:26:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.900+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3897169912 shutdown_connections 2024-08-28T19:26:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.900+0000 7ff9efbf9640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9e8108ab0 0x7ff9e819fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.900+0000 7ff9efbf9640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9d40b4ca0 0x7ff9d40b71b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.900+0000 7ff9efbf9640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e8108160 0x7ff9e819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.900+0000 7ff9efbf9640 1 --2- 172.21.15.16:0/3897169912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e8102820 0x7ff9e819a220 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.900+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3897169912 >> 172.21.15.16:0/3897169912 conn(0x7ff9e80794e0 msgr2=0x7ff9e806bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.900+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3897169912 shutdown_connections 2024-08-28T19:26:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:54.900+0000 7ff9efbf9640 1 -- 172.21.15.16:0/3897169912 wait complete. 2024-08-28T19:26:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:55 smithi016 ceph-mon[18556]: pgmap v515: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 366 KiB/s wr, 64 op/s 2024-08-28T19:26:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:55 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:55 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:55 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:55 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:55.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:55 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:55.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:55 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:55 smithi028 ceph-mon[21769]: pgmap v515: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 366 KiB/s wr, 64 op/s 2024-08-28T19:26:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:55 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:55 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:55 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:55 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:55 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:55 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:55 smithi115 ceph-mon[22184]: pgmap v515: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 366 KiB/s wr, 64 op/s 2024-08-28T19:26:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:55 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:55 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:55 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:55 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:55 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:55 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:55.897 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:26:55.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.98 MB/sec execute 155 sec latency 13846.997 ms 2024-08-28T19:26:56.151 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:26:56.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:56 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:26:56.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:56 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:56 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:56.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:56 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:56.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:56 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:56.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:56 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:56 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:26:56.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:56 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:56 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:56 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:56 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:26:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:56 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:56 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:56 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:56 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:56 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:56 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:26:56.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:56 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:56 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:56 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:56 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:26:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:56 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:56 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:56 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:56 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:56 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:56 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:26:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:56 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:56 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:56 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:26:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.646+0000 7fb4acaee640 1 -- 172.21.15.16:0/3109746917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8108160 msgr2=0x7fb4a8108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.646+0000 7fb4acaee640 1 --2- 172.21.15.16:0/3109746917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8108160 0x7fb4a8108570 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb49c009900 tx=0x7fb49c02a410 comp rx=0 tx=0).stop 2024-08-28T19:26:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.646+0000 7fb4acaee640 1 -- 172.21.15.16:0/3109746917 shutdown_connections 2024-08-28T19:26:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.646+0000 7fb4acaee640 1 --2- 172.21.15.16:0/3109746917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a8108ab0 0x7fb4a810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.646+0000 7fb4acaee640 1 --2- 172.21.15.16:0/3109746917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8108160 0x7fb4a8108570 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.646+0000 7fb4acaee640 1 --2- 172.21.15.16:0/3109746917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4a8102820 0x7fb4a8102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.646+0000 7fb4acaee640 1 -- 172.21.15.16:0/3109746917 >> 172.21.15.16:0/3109746917 conn(0x7fb4a80794e0 msgr2=0x7fb4a80798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:56.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.646+0000 7fb4acaee640 1 -- 172.21.15.16:0/3109746917 shutdown_connections 2024-08-28T19:26:56.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.646+0000 7fb4acaee640 1 -- 172.21.15.16:0/3109746917 wait complete. 2024-08-28T19:26:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.647+0000 7fb4acaee640 1 Processor -- start 2024-08-28T19:26:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.647+0000 7fb4acaee640 1 -- start start 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.647+0000 7fb4acaee640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4a8102820 0x7fb4a819b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.647+0000 7fb4acaee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a8108160 0x7fb4a819b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.647+0000 7fb4acaee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8108ab0 0x7fb4a819bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.647+0000 7fb4acaee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81126e0 con 0x7fb4a8108ab0 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.647+0000 7fb4acaee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a8112560 con 0x7fb4a8108160 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.647+0000 7fb4acaee640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a8112860 con 0x7fb4a8102820 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.647+0000 7fb4a6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8108ab0 0x7fb4a819bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4a6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8108ab0 0x7fb4a819bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49106/0 (socket says 172.21.15.16:49106) 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4a6d9d640 1 -- 172.21.15.16:0/161299860 learned_addr learned my addr 172.21.15.16:0/161299860 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a8108160 0x7fb4a819b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4a659c640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4a8102820 0x7fb4a819b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4a6d9d640 1 -- 172.21.15.16:0/161299860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4a8102820 msgr2=0x7fb4a819b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4a6d9d640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4a8102820 0x7fb4a819b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4a6d9d640 1 -- 172.21.15.16:0/161299860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a8108160 msgr2=0x7fb4a819b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4a6d9d640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a8108160 0x7fb4a819b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4a6d9d640 1 -- 172.21.15.16:0/161299860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a81a63f0 con 0x7fb4a8108ab0 2024-08-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4a6d9d640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8108ab0 0x7fb4a819bbb0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000eb10 tx=0x7fb4a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4977fe640 1 -- 172.21.15.16:0/161299860 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0004b20 con 0x7fb4a8108ab0 2024-08-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4977fe640 1 -- 172.21.15.16:0/161299860 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4a0004cc0 con 0x7fb4a8108ab0 2024-08-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4977fe640 1 -- 172.21.15.16:0/161299860 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0010670 con 0x7fb4a8108ab0 2024-08-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4acaee640 1 -- 172.21.15.16:0/161299860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a81a66e0 con 0x7fb4a8108ab0 2024-08-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.648+0000 7fb4acaee640 1 -- 172.21.15.16:0/161299860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a81a6ba0 con 0x7fb4a8108ab0 2024-08-28T19:26:56.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.651+0000 7fb4977fe640 1 -- 172.21.15.16:0/161299860 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fb4a0005020 con 0x7fb4a8108ab0 2024-08-28T19:26:56.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.651+0000 7fb4b3063740 1 Processor -- start 2024-08-28T19:26:56.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.651+0000 7fb4b3063740 1 -- start start 2024-08-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745d51b0 0x5563749a3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745dd390 0x5563749a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4b3063740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55637491a630 con 0x5563745fe070 2024-08-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4b3063740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556374909bc0 con 0x5563745d51b0 2024-08-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4b3063740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55637490db70 con 0x5563745dd390 2024-08-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749a32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49122/0 (socket says 172.21.15.16:49122) 2024-08-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/266176054 learned_addr learned my addr 172.21.15.16:0/266176054 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a659c640 1 --2- 172.21.15.16:0/266176054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745d51b0 0x5563749a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/266176054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745dd390 0x5563749a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4977fe640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4900b4ca0 0x7fb4900b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a6d9d640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4900b4ca0 0x7fb4900b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/266176054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745dd390 msgr2=0x5563749a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4977fe640 1 -- 172.21.15.16:0/161299860 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb4a00e5ec0 con 0x7fb4a8108ab0 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/266176054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745dd390 0x5563749a3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/266176054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745d51b0 msgr2=0x5563749a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/266176054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745d51b0 0x5563749a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/266176054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556374669650 con 0x5563745fe070 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a659c640 1 --2- 172.21.15.16:0/266176054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745d51b0 0x5563749a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.652+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/266176054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749a32c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0096c0 tx=0x7fb49c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b64cd548068bc2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.653+0000 7fb4a6d9d640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4900b4ca0 0x7fb4900b71b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00e3350 tx=0x7fb4a000efd0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.653+0000 7fb494ff9640 1 -- 172.21.15.16:0/266176054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c035040 con 0x5563745fe070 2024-08-28T19:26:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.653+0000 7fb494ff9640 1 -- 172.21.15.16:0/266176054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb49c0035f0 con 0x5563745fe070 2024-08-28T19:26:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.653+0000 7fb494ff9640 1 -- 172.21.15.16:0/266176054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c0044e0 con 0x5563745fe070 2024-08-28T19:26:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.654+0000 7fb4b3063740 1 -- 172.21.15.16:0/266176054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 msgr2=0x5563749a32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.654+0000 7fb4b3063740 1 --2- 172.21.15.16:0/266176054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749a32c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0096c0 tx=0x7fb49c031400 comp rx=0 tx=0).stop 2024-08-28T19:26:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.654+0000 7fb4b3063740 1 -- 172.21.15.16:0/266176054 shutdown_connections 2024-08-28T19:26:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.654+0000 7fb4b3063740 1 --2- 172.21.15.16:0/266176054 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745dd390 0x5563749a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.654+0000 7fb4b3063740 1 --2- 172.21.15.16:0/266176054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745d51b0 0x5563749a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.654+0000 7fb4b3063740 1 --2- 172.21.15.16:0/266176054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749a32c0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.654+0000 7fb4b3063740 1 -- 172.21.15.16:0/266176054 >> 172.21.15.16:0/266176054 conn(0x5563745f2460 msgr2=0x5563749a11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.654+0000 7fb4b3063740 1 -- 172.21.15.16:0/266176054 shutdown_connections 2024-08-28T19:26:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.654+0000 7fb4b3063740 1 -- 172.21.15.16:0/266176054 wait complete. 2024-08-28T19:26:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.654+0000 7fb4b3063740 1 -- start start 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.654+0000 7fb4b3063740 1 Processor -- start 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745d51b0 0x5563749aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745dd390 0x5563749af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745fe070 0x5563749bad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4b3063740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55637476c700 con 0x5563745d51b0 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4b3063740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556374890700 con 0x5563745dd390 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4b3063740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5563749b1a40 con 0x5563745fe070 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4b3063740 10 client.0 _ll_register_callbacks cb 0x5563748bfdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4a5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745d51b0 0x5563749aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4a5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745d51b0 0x5563749aeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49132/0 (socket says 172.21.15.16:49132) 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/2765244357 learned_addr learned my addr 172.21.15.16:0/2765244357 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4a659c640 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745dd390 0x5563749af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745fe070 0x5563749bad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/2765244357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745fe070 msgr2=0x5563749bad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745fe070 0x5563749bad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/2765244357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745dd390 msgr2=0x5563749af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745dd390 0x5563749af0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/2765244357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563748c98b0 con 0x5563745d51b0 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745d51b0 0x5563749aeb30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb49c006010 tx=0x7fb49c032d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c03915b2c935df server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.655+0000 7fb48dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb48dffb640 1 -- 172.21.15.16:0/2765244357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c052060 con 0x5563745d51b0 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb48dffb640 1 -- 172.21.15.16:0/2765244357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb49c003bb0 con 0x5563745d51b0 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb4b3063740 1 -- 172.21.15.16:0/2765244357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb49c0035f0 con 0x5563745d51b0 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb48dffb640 1 -- 172.21.15.16:0/2765244357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fb49c035070 con 0x5563745d51b0 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb4b3063740 10 client.17157 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb4b3063740 1 -- 172.21.15.16:0/2765244357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5563749bd8a0 con 0x5563745d51b0 2024-08-28T19:26:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb48dffb640 1 -- 172.21.15.16:0/2765244357 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb49c0035f0 con 0x5563745d51b0 2024-08-28T19:26:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb48dffb640 1 -- 172.21.15.16:0/2765244357 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7fb49c005350 con 0x5563745d51b0 2024-08-28T19:26:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb4b3063740 10 client.17157 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb4b3063740 10 client.17157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb4b3063740 20 client.17157 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5563745bc9f0 0x5563749c0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb4b3063740 4 client.17157 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:26:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.656+0000 7fb4b3063740 1 -- 172.21.15.16:0/2765244357 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55637491c580 con 0x5563745bc9f0 2024-08-28T19:26:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.657+0000 7fb4a659c640 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5563745bc9f0 0x5563749c0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.657+0000 7fb4a659c640 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5563745bc9f0 0x5563749c0c00 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29832e78a0b3c636 server_cookie=e2d052ee2aab7d96 in_seq=0 out_seq=0 2024-08-28T19:26:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.657+0000 7fb48dffb640 10 client.17157 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb48dffb640 1 -- 172.21.15.16:0/2765244357 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55637491c580 con 0x5563745bc9f0 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb48dffb640 10 client.17157 handle_command_reply: tid=0 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 1 client.17157 shutdown 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 1 -- 172.21.15.16:0/2765244357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745d51b0 msgr2=0x5563749aeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745d51b0 0x5563749aeb30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb49c006010 tx=0x7fb49c032d80 comp rx=0 tx=0).stop 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 1 -- 172.21.15.16:0/2765244357 shutdown_connections 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5563745bc9f0 0x5563749c0c00 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745fe070 0x5563749bad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745dd390 0x5563749af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2765244357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745d51b0 0x5563749aeb30 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 1 -- 172.21.15.16:0/2765244357 >> 172.21.15.16:0/2765244357 conn(0x5563745f2460 msgr2=0x5563749a64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb48dffb640 0 client.17157 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 1 -- 172.21.15.16:0/2765244357 shutdown_connections 2024-08-28T19:26:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 1 -- 172.21.15.16:0/2765244357 wait complete. 2024-08-28T19:26:56.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.658+0000 7fb4b3063740 20 client.17157 trim_cache size 0 max 16384 2024-08-28T19:26:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.676+0000 7fb4b3063740 1 Processor -- start 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.676+0000 7fb4b3063740 1 -- start start 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745d51b0 0x5563749b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745dd390 0x5563749baca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749bb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4b3063740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81b0ba0 con 0x5563745fe070 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4b3063740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81b2710 con 0x5563745d51b0 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4b3063740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81b1270 con 0x5563745dd390 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4a5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749bb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4a5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749bb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49136/0 (socket says 172.21.15.16:49136) 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/4262545997 learned_addr learned my addr 172.21.15.16:0/4262545997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/4262545997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745d51b0 0x5563749b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4a659c640 1 --2- 172.21.15.16:0/4262545997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745dd390 0x5563749baca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/4262545997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745dd390 msgr2=0x5563749baca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/4262545997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745dd390 0x5563749baca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/4262545997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745d51b0 msgr2=0x5563749b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/4262545997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745d51b0 0x5563749b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4a5d9b640 1 -- 172.21.15.16:0/4262545997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563749a6120 con 0x5563745fe070 2024-08-28T19:26:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/4262545997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749bb1e0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb49c004720 tx=0x7fb49c032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593b61ea2e46af6a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb48ffff640 1 -- 172.21.15.16:0/4262545997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c052020 con 0x5563745fe070 2024-08-28T19:26:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb48ffff640 1 -- 172.21.15.16:0/4262545997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb49c031c30 con 0x5563745fe070 2024-08-28T19:26:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.677+0000 7fb48ffff640 1 -- 172.21.15.16:0/4262545997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c005500 con 0x5563745fe070 2024-08-28T19:26:56.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.678+0000 7fb4b3063740 1 -- 172.21.15.16:0/4262545997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 msgr2=0x5563749bb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:56.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.678+0000 7fb4b3063740 1 --2- 172.21.15.16:0/4262545997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749bb1e0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb49c004720 tx=0x7fb49c032c90 comp rx=0 tx=0).stop 2024-08-28T19:26:56.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.678+0000 7fb4b3063740 1 -- 172.21.15.16:0/4262545997 shutdown_connections 2024-08-28T19:26:56.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.678+0000 7fb4b3063740 1 --2- 172.21.15.16:0/4262545997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745fe070 0x5563749bb1e0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.678+0000 7fb4b3063740 1 --2- 172.21.15.16:0/4262545997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745dd390 0x5563749baca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.678+0000 7fb4b3063740 1 --2- 172.21.15.16:0/4262545997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745d51b0 0x5563749b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.678+0000 7fb4b3063740 1 -- 172.21.15.16:0/4262545997 >> 172.21.15.16:0/4262545997 conn(0x5563745f2460 msgr2=0x5563749a4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:56.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.678+0000 7fb4b3063740 1 -- 172.21.15.16:0/4262545997 shutdown_connections 2024-08-28T19:26:56.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.678+0000 7fb4b3063740 1 -- 172.21.15.16:0/4262545997 wait complete. 2024-08-28T19:26:56.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4b3063740 1 -- start start 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4b3063740 1 Processor -- start 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745d51b0 0x556374a1fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745dd390 0x556374a20070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4b3063740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745fe070 0x556374a23b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4b3063740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81b2040 con 0x5563745dd390 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4b3063740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81b1940 con 0x5563745fe070 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4b3063740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81aeb80 con 0x5563745d51b0 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4b3063740 10 client.0 _ll_register_callbacks cb 0x7fb49800c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4a659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745dd390 0x556374a20070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4a659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745dd390 0x556374a20070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49140/0 (socket says 172.21.15.16:49140) 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4a659c640 1 -- 172.21.15.16:0/2799354335 learned_addr learned my addr 172.21.15.16:0/2799354335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745d51b0 0x556374a1fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745fe070 0x556374a23b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4a659c640 1 -- 172.21.15.16:0/2799354335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745d51b0 msgr2=0x556374a1fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4a659c640 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745d51b0 0x556374a1fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4a659c640 1 -- 172.21.15.16:0/2799354335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745fe070 msgr2=0x556374a23b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4a659c640 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745fe070 0x556374a23b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4a659c640 1 -- 172.21.15.16:0/2799354335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556374a24170 con 0x5563745dd390 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.679+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745d51b0 0x556374a1fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb4a659c640 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745dd390 0x556374a20070 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb4980058c0 tx=0x7fb498012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d64f9e2eb1026e4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb48dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb48dffb640 1 -- 172.21.15.16:0/2799354335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb498004020 con 0x5563745dd390 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb48dffb640 1 -- 172.21.15.16:0/2799354335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fb4980041c0 con 0x5563745dd390 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb4b3063740 1 -- 172.21.15.16:0/2799354335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb49c031c30 con 0x5563745dd390 2024-08-28T19:26:56.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb48dffb640 1 -- 172.21.15.16:0/2799354335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fb4980043c0 con 0x5563745dd390 2024-08-28T19:26:56.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb4b3063740 10 client.17163 fetch_fsmap learned FSMap version 42 2024-08-28T19:26:56.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb4b3063740 1 -- 172.21.15.16:0/2799354335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x556374a2b600 con 0x5563745dd390 2024-08-28T19:26:56.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb48dffb640 1 -- 172.21.15.16:0/2799354335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb49c031c30 con 0x5563745dd390 2024-08-28T19:26:56.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb48dffb640 1 -- 172.21.15.16:0/2799354335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7fb498003960 con 0x5563745dd390 2024-08-28T19:26:56.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb4b3063740 10 client.17163 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:26:56.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb4b3063740 10 client.17163 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:26:56.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb4b3063740 20 client.17163 populate_metadata read hostname 'smithi016' 2024-08-28T19:26:56.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556374591da0 0x556374a2de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb4b3063740 4 client.17163 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.680+0000 7fb4b3063740 1 -- 172.21.15.16:0/2799354335 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556374918020 con 0x556374591da0 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556374591da0 0x556374a2de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb4a5d9b640 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556374591da0 0x556374a2de70 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc9cfc49da7d9a0d server_cookie=81c420ba5f72f46a in_seq=0 out_seq=0 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb48dffb640 10 client.17163 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb48dffb640 1 -- 172.21.15.16:0/2799354335 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2842 (crc 0 0 0) 0x556374918020 con 0x556374591da0 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb48dffb640 10 client.17163 handle_command_reply: tid=0 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb4b3063740 1 client.17163 shutdown 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb4b3063740 1 -- 172.21.15.16:0/2799354335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745dd390 msgr2=0x556374a20070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745dd390 0x556374a20070 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb4980058c0 tx=0x7fb498012b60 comp rx=0 tx=0).stop 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb4b3063740 1 -- 172.21.15.16:0/2799354335 shutdown_connections 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563745fe070 0x556374a23b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5563745dd390 0x556374a20070 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556374591da0 0x556374a2de70 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb4b3063740 1 --2- 172.21.15.16:0/2799354335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5563745d51b0 0x556374a1fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb48dffb640 0 client.17163 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:26:56.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.681+0000 7fb4b3063740 1 -- 172.21.15.16:0/2799354335 >> 172.21.15.16:0/2799354335 conn(0x5563745f2460 msgr2=0x5563749a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:56.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4b3063740 1 -- 172.21.15.16:0/2799354335 shutdown_connections 2024-08-28T19:26:56.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4b3063740 1 -- 172.21.15.16:0/2799354335 wait complete. 2024-08-28T19:26:56.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4b3063740 20 client.17163 trim_cache size 0 max 16384 2024-08-28T19:26:56.691 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:26:56.691 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:26:56.691 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "c89de864-c285-403a-8de0-63c8123972fa", 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:26:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:26:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:26:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:26:56.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:56.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:26:56.698 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:26:56.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 -- 172.21.15.16:0/161299860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4900b4ca0 msgr2=0x7fb4900b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:56.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4900b4ca0 0x7fb4900b71b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00e3350 tx=0x7fb4a000efd0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 -- 172.21.15.16:0/161299860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8108ab0 msgr2=0x7fb4a819bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:26:56.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8108ab0 0x7fb4a819bbb0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000eb10 tx=0x7fb4a000c2d0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 -- 172.21.15.16:0/161299860 shutdown_connections 2024-08-28T19:26:56.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8108ab0 0x7fb4a819bbb0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4900b4ca0 0x7fb4900b71b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a8108160 0x7fb4a819b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 --2- 172.21.15.16:0/161299860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4a8102820 0x7fb4a819b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:26:56.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 -- 172.21.15.16:0/161299860 >> 172.21.15.16:0/161299860 conn(0x7fb4a80794e0 msgr2=0x7fb4a81050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:26:56.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 -- 172.21.15.16:0/161299860 shutdown_connections 2024-08-28T19:26:56.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:26:56.682+0000 7fb4acaee640 1 -- 172.21.15.16:0/161299860 wait complete. 2024-08-28T19:26:56.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.94 MB/sec execute 156 sec latency 14847.042 ms 2024-08-28T19:26:57.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:57 smithi016 ceph-mon[18556]: pgmap v516: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 370 KiB/s wr, 67 op/s 2024-08-28T19:26:57.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:57 smithi028 ceph-mon[21769]: pgmap v516: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 370 KiB/s wr, 67 op/s 2024-08-28T19:26:57.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:57 smithi115 ceph-mon[22184]: pgmap v516: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 370 KiB/s wr, 67 op/s 2024-08-28T19:26:57.710 INFO:tasks.cephfs.filesystem:scrub status for tag:c89de864-c285-403a-8de0-63c8123972fa - {'path': '/', 'tag': 'c89de864-c285-403a-8de0-63c8123972fa', 'options': 'recursive,force'} 2024-08-28T19:26:57.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.91 MB/sec execute 157 sec latency 15847.071 ms 2024-08-28T19:26:58.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.88 MB/sec execute 158 sec latency 16847.167 ms 2024-08-28T19:26:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:26:59 smithi016 ceph-mon[18556]: pgmap v517: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 157 KiB/s wr, 19 op/s 2024-08-28T19:26:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:26:59 smithi028 ceph-mon[21769]: pgmap v517: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 157 KiB/s wr, 19 op/s 2024-08-28T19:26:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:26:59 smithi115 ceph-mon[22184]: pgmap v517: 129 pgs: 129 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 157 KiB/s wr, 19 op/s 2024-08-28T19:26:59.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.85 MB/sec execute 159 sec latency 17847.232 ms 2024-08-28T19:27:00.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.82 MB/sec execute 160 sec latency 18847.316 ms 2024-08-28T19:27:01.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:01 smithi016 ceph-mon[18556]: pgmap v518: 129 pgs: 129 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 297 KiB/s wr, 21 op/s 2024-08-28T19:27:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:01 smithi028 ceph-mon[21769]: pgmap v518: 129 pgs: 129 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 297 KiB/s wr, 21 op/s 2024-08-28T19:27:01.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:01 smithi115 ceph-mon[22184]: pgmap v518: 129 pgs: 129 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 297 KiB/s wr, 21 op/s 2024-08-28T19:27:01.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.79 MB/sec execute 161 sec latency 19847.439 ms 2024-08-28T19:27:02.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.76 MB/sec execute 162 sec latency 20847.526 ms 2024-08-28T19:27:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:03 smithi016 ceph-mon[18556]: pgmap v519: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 616 KiB/s wr, 48 op/s 2024-08-28T19:27:03.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:03 smithi028 ceph-mon[21769]: pgmap v519: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 616 KiB/s wr, 48 op/s 2024-08-28T19:27:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:03 smithi115 ceph-mon[22184]: pgmap v519: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 616 KiB/s wr, 48 op/s 2024-08-28T19:27:03.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.73 MB/sec execute 163 sec latency 21847.639 ms 2024-08-28T19:27:04.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.70 MB/sec execute 164 sec latency 22847.745 ms 2024-08-28T19:27:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:05 smithi016 ceph-mon[18556]: pgmap v520: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 610 KiB/s wr, 44 op/s 2024-08-28T19:27:05.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:05 smithi028 ceph-mon[21769]: pgmap v520: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 610 KiB/s wr, 44 op/s 2024-08-28T19:27:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:05 smithi115 ceph-mon[22184]: pgmap v520: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 610 KiB/s wr, 44 op/s 2024-08-28T19:27:05.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.67 MB/sec execute 165 sec latency 23847.841 ms 2024-08-28T19:27:06.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.65 MB/sec execute 166 sec latency 24847.954 ms 2024-08-28T19:27:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:07 smithi016 ceph-mon[18556]: pgmap v521: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 640 KiB/s wr, 52 op/s 2024-08-28T19:27:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:07 smithi028 ceph-mon[21769]: pgmap v521: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 640 KiB/s wr, 52 op/s 2024-08-28T19:27:07.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:07 smithi115 ceph-mon[22184]: pgmap v521: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 640 KiB/s wr, 52 op/s 2024-08-28T19:27:07.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.62 MB/sec execute 167 sec latency 25848.078 ms 2024-08-28T19:27:08.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:27:08.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.59 MB/sec execute 168 sec latency 26848.128 ms 2024-08-28T19:27:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:27:09.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:09 smithi016 ceph-mon[18556]: pgmap v522: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 625 KiB/s wr, 48 op/s 2024-08-28T19:27:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.500+0000 7f3977e10640 1 -- 172.21.15.16:0/3763930865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970101d10 msgr2=0x7f3970102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.500+0000 7f3977e10640 1 --2- 172.21.15.16:0/3763930865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970101d10 0x7f3970102120 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f396c009900 tx=0x7f396c02a410 comp rx=0 tx=0).stop 2024-08-28T19:27:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.500+0000 7f3977e10640 1 -- 172.21.15.16:0/3763930865 shutdown_connections 2024-08-28T19:27:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.500+0000 7f3977e10640 1 --2- 172.21.15.16:0/3763930865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3970109ba0 0x7f39701103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.500+0000 7f3977e10640 1 --2- 172.21.15.16:0/3763930865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3970109250 0x7f3970109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.500+0000 7f3977e10640 1 --2- 172.21.15.16:0/3763930865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970101d10 0x7f3970102120 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.500+0000 7f3977e10640 1 -- 172.21.15.16:0/3763930865 >> 172.21.15.16:0/3763930865 conn(0x7f3970077a00 msgr2=0x7f3970077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.500+0000 7f3977e10640 1 -- 172.21.15.16:0/3763930865 shutdown_connections 2024-08-28T19:27:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.500+0000 7f3977e10640 1 -- 172.21.15.16:0/3763930865 wait complete. 2024-08-28T19:27:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.501+0000 7f3977e10640 1 Processor -- start 2024-08-28T19:27:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f3977e10640 1 -- start start 2024-08-28T19:27:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f3977e10640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3970101d10 0x7f3970102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f3977e10640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3970109250 0x7f39701031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f3977e10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970109ba0 0x7f3970103730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f3977e10640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f39701138b0 con 0x7f3970109ba0 2024-08-28T19:27:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f3977e10640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3970113730 con 0x7f3970101d10 2024-08-28T19:27:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f3977e10640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3970113a30 con 0x7f3970109250 2024-08-28T19:27:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f39763ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970109ba0 0x7f3970103730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f39763ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970109ba0 0x7f3970103730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43368/0 (socket says 172.21.15.16:43368) 2024-08-28T19:27:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f39763ad640 1 -- 172.21.15.16:0/1982798583 learned_addr learned my addr 172.21.15.16:0/1982798583 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f3975bac640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3970101d10 0x7f3970102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f39753ab640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3970109250 0x7f39701031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f39763ad640 1 -- 172.21.15.16:0/1982798583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3970109250 msgr2=0x7f39701031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f39763ad640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3970109250 0x7f39701031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f39763ad640 1 -- 172.21.15.16:0/1982798583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3970101d10 msgr2=0x7f3970102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f39763ad640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3970101d10 0x7f3970102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f39763ad640 1 -- 172.21.15.16:0/1982798583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39701a84c0 con 0x7f3970109ba0 2024-08-28T19:27:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.502+0000 7f39753ab640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3970109250 0x7f39701031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:27:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.503+0000 7f39763ad640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970109ba0 0x7f3970103730 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f396800eb10 tx=0x7f396800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.503+0000 7f3966ffd640 1 -- 172.21.15.16:0/1982798583 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968004b20 con 0x7f3970109ba0 2024-08-28T19:27:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.503+0000 7f3966ffd640 1 -- 172.21.15.16:0/1982798583 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3968004cc0 con 0x7f3970109ba0 2024-08-28T19:27:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.503+0000 7f3966ffd640 1 -- 172.21.15.16:0/1982798583 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968010670 con 0x7f3970109ba0 2024-08-28T19:27:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.503+0000 7f3977e10640 1 -- 172.21.15.16:0/1982798583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39701a87b0 con 0x7f3970109ba0 2024-08-28T19:27:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.503+0000 7f3977e10640 1 -- 172.21.15.16:0/1982798583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39701a8cc0 con 0x7f3970109ba0 2024-08-28T19:27:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.504+0000 7f3977e10640 1 -- 172.21.15.16:0/1982798583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3970102120 con 0x7f3970109ba0 2024-08-28T19:27:09.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.507+0000 7f3966ffd640 1 -- 172.21.15.16:0/1982798583 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3968005000 con 0x7f3970109ba0 2024-08-28T19:27:09.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.507+0000 7f3966ffd640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f395c0b4ca0 0x7f395c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:09.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.508+0000 7f3975bac640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f395c0b4ca0 0x7f395c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:09.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.508+0000 7f3966ffd640 1 -- 172.21.15.16:0/1982798583 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f39680e60a0 con 0x7f3970109ba0 2024-08-28T19:27:09.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.508+0000 7f3975bac640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f395c0b4ca0 0x7f395c0b71b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f396c005ab0 tx=0x7f396c005a20 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:09.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.510+0000 7f3966ffd640 1 -- 172.21.15.16:0/1982798583 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f3968056130 con 0x7f3970109ba0 2024-08-28T19:27:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:09 smithi028 ceph-mon[21769]: pgmap v522: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 625 KiB/s wr, 48 op/s 2024-08-28T19:27:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:09 smithi115 ceph-mon[22184]: pgmap v522: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 625 KiB/s wr, 48 op/s 2024-08-28T19:27:09.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.733+0000 7f3977e10640 1 -- 172.21.15.16:0/1982798583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3970103fb0 con 0x7f3970109ba0 2024-08-28T19:27:09.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.734+0000 7f3966ffd640 1 -- 172.21.15.16:0/1982798583 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11978 (secure 0 0 0) 0x7f39680acb20 con 0x7f3970109ba0 2024-08-28T19:27:09.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:27:09.735 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":42,"btime":"2024-08-28T19:26:37:182944+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:26:36.587538+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:27:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 42 2024-08-28T19:27:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 -- 172.21.15.16:0/1982798583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f395c0b4ca0 msgr2=0x7f395c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f395c0b4ca0 0x7f395c0b71b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f396c005ab0 tx=0x7f396c005a20 comp rx=0 tx=0).stop 2024-08-28T19:27:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 -- 172.21.15.16:0/1982798583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970109ba0 msgr2=0x7f3970103730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970109ba0 0x7f3970103730 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f396800eb10 tx=0x7f396800c2d0 comp rx=0 tx=0).stop 2024-08-28T19:27:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 -- 172.21.15.16:0/1982798583 shutdown_connections 2024-08-28T19:27:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970109ba0 0x7f3970103730 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3970109250 0x7f39701031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f395c0b4ca0 0x7f395c0b71b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 --2- 172.21.15.16:0/1982798583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3970101d10 0x7f3970102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 -- 172.21.15.16:0/1982798583 >> 172.21.15.16:0/1982798583 conn(0x7f3970077a00 msgr2=0x7f3970107240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 -- 172.21.15.16:0/1982798583 shutdown_connections 2024-08-28T19:27:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:09.736+0000 7f3977e10640 1 -- 172.21.15.16:0/1982798583 wait complete. 2024-08-28T19:27:09.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.56 MB/sec execute 169 sec latency 27848.174 ms 2024-08-28T19:27:10.410 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id 8e9c1d21 --release --await' 2024-08-28T19:27:10.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id 8e9c1d21 --release --await 2024-08-28T19:27:10.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:10 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1982798583' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:27:10.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:10 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1982798583' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:27:10.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:10 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1982798583' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:27:10.672 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:27:10.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277160 4.54 MB/sec execute 170 sec latency 28848.215 ms 2024-08-28T19:27:11.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.192+0000 7f0064c45640 1 -- 172.21.15.16:0/2235234511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060107970 msgr2=0x7f0060107d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:11.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.192+0000 7f0064c45640 1 --2- 172.21.15.16:0/2235234511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060107970 0x7f0060107d80 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0054009900 tx=0x7f005402a410 comp rx=0 tx=0).stop 2024-08-28T19:27:11.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.192+0000 7f0064c45640 1 -- 172.21.15.16:0/2235234511 shutdown_connections 2024-08-28T19:27:11.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.192+0000 7f0064c45640 1 --2- 172.21.15.16:0/2235234511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00601082c0 0x7f006010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.192+0000 7f0064c45640 1 --2- 172.21.15.16:0/2235234511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060107970 0x7f0060107d80 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.192+0000 7f0064c45640 1 --2- 172.21.15.16:0/2235234511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0060102030 0x7f0060102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.192+0000 7f0064c45640 1 -- 172.21.15.16:0/2235234511 >> 172.21.15.16:0/2235234511 conn(0x7f0060079c40 msgr2=0x7f006007a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:11.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.192+0000 7f0064c45640 1 -- 172.21.15.16:0/2235234511 shutdown_connections 2024-08-28T19:27:11.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.192+0000 7f0064c45640 1 -- 172.21.15.16:0/2235234511 wait complete. 2024-08-28T19:27:11.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f0064c45640 1 Processor -- start 2024-08-28T19:27:11.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f0064c45640 1 -- start start 2024-08-28T19:27:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f0064c45640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060102030 0x7f006019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f0064c45640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0060107970 0x7f006019a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f0064c45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00601082c0 0x7f006019fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f0064c45640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00601127f0 con 0x7f0060102030 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f0064c45640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0060112670 con 0x7f00601082c0 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f0064c45640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0060112970 con 0x7f0060107970 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f005e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060102030 0x7f006019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f005e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060102030 0x7f006019a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43392/0 (socket says 172.21.15.16:43392) 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f005e59c640 1 -- 172.21.15.16:0/2908555942 learned_addr learned my addr 172.21.15.16:0/2908555942 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f005dd9b640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0060107970 0x7f006019a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.193+0000 7f005ed9d640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00601082c0 0x7f006019fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f005e59c640 1 -- 172.21.15.16:0/2908555942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0060107970 msgr2=0x7f006019a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f005e59c640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0060107970 0x7f006019a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f005e59c640 1 -- 172.21.15.16:0/2908555942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00601082c0 msgr2=0x7f006019fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f005e59c640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00601082c0 0x7f006019fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f005e59c640 1 -- 172.21.15.16:0/2908555942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00601a02a0 con 0x7f0060102030 2024-08-28T19:27:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f005ed9d640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00601082c0 0x7f006019fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:27:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f005e59c640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060102030 0x7f006019a330 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f005000d950 tx=0x7f005000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f004f7fe640 1 -- 172.21.15.16:0/2908555942 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050017030 con 0x7f0060102030 2024-08-28T19:27:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f0064c45640 1 -- 172.21.15.16:0/2908555942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00601a0590 con 0x7f0060102030 2024-08-28T19:27:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f004f7fe640 1 -- 172.21.15.16:0/2908555942 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0050004020 con 0x7f0060102030 2024-08-28T19:27:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f004f7fe640 1 -- 172.21.15.16:0/2908555942 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00500047a0 con 0x7f0060102030 2024-08-28T19:27:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.194+0000 7f0064c45640 1 -- 172.21.15.16:0/2908555942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00600feb50 con 0x7f0060102030 2024-08-28T19:27:11.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.197+0000 7f004f7fe640 1 -- 172.21.15.16:0/2908555942 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0050009a30 con 0x7f0060102030 2024-08-28T19:27:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.197+0000 7f006b1ba740 1 Processor -- start 2024-08-28T19:27:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f004f7fe640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00480b4ca0 0x7f00480b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f005ed9d640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00480b4ca0 0x7f00480b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f004f7fe640 1 -- 172.21.15.16:0/2908555942 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f00500e5380 con 0x7f0060102030 2024-08-28T19:27:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f006b1ba740 1 -- start start 2024-08-28T19:27:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f961520 0x55d89fd06c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f938660 0x55d89fd071c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd07700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f006b1ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d89fc7da70 con 0x55d89f940840 2024-08-28T19:27:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f006b1ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d89fc6d000 con 0x55d89f938660 2024-08-28T19:27:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f006b1ba740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d89fc70fb0 con 0x55d89f961520 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f005ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd07700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f005ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd07700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43406/0 (socket says 172.21.15.16:43406) 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f005ed9d640 1 -- 172.21.15.16:0/2379663142 learned_addr learned my addr 172.21.15.16:0/2379663142 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f005ed9d640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00480b4ca0 0x7f00480b71b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f00580092f0 tx=0x7f0058009280 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f005dd9b640 1 --2- 172.21.15.16:0/2379663142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f938660 0x55d89fd071c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.198+0000 7f005e59c640 1 --2- 172.21.15.16:0/2379663142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f961520 0x55d89fd06c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.199+0000 7f005ed9d640 1 -- 172.21.15.16:0/2379663142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f961520 msgr2=0x55d89fd06c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.199+0000 7f005ed9d640 1 --2- 172.21.15.16:0/2379663142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f961520 0x55d89fd06c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.199+0000 7f005ed9d640 1 -- 172.21.15.16:0/2379663142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f938660 msgr2=0x55d89fd071c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.199+0000 7f005ed9d640 1 --2- 172.21.15.16:0/2379663142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f938660 0x55d89fd071c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.199+0000 7f005ed9d640 1 -- 172.21.15.16:0/2379663142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d89f9ccb00 con 0x55d89f940840 2024-08-28T19:27:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.199+0000 7f005ed9d640 1 --2- 172.21.15.16:0/2379663142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd07700 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f00580125e0 tx=0x7f0058013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5963142809f5add9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.199+0000 7f004cff9640 1 -- 172.21.15.16:0/2379663142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0058023070 con 0x55d89f940840 2024-08-28T19:27:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.199+0000 7f004cff9640 1 -- 172.21.15.16:0/2379663142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0058011bd0 con 0x55d89f940840 2024-08-28T19:27:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.199+0000 7f004cff9640 1 -- 172.21.15.16:0/2379663142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005801fd30 con 0x55d89f940840 2024-08-28T19:27:11.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.200+0000 7f006b1ba740 1 -- 172.21.15.16:0/2379663142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 msgr2=0x55d89fd07700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.200+0000 7f006b1ba740 1 --2- 172.21.15.16:0/2379663142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd07700 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f00580125e0 tx=0x7f0058013000 comp rx=0 tx=0).stop 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.200+0000 7f006b1ba740 1 -- 172.21.15.16:0/2379663142 shutdown_connections 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.200+0000 7f006b1ba740 1 --2- 172.21.15.16:0/2379663142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd07700 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.200+0000 7f006b1ba740 1 --2- 172.21.15.16:0/2379663142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f938660 0x55d89fd071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.200+0000 7f006b1ba740 1 --2- 172.21.15.16:0/2379663142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f961520 0x55d89fd06c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.200+0000 7f006b1ba740 1 -- 172.21.15.16:0/2379663142 >> 172.21.15.16:0/2379663142 conn(0x55d89f955910 msgr2=0x55d89fd04b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.200+0000 7f006b1ba740 1 -- 172.21.15.16:0/2379663142 shutdown_connections 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.200+0000 7f006b1ba740 1 -- 172.21.15.16:0/2379663142 wait complete. 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.200+0000 7f006b1ba740 1 -- start start 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f006b1ba740 1 Processor -- start 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f938660 0x55d89fd12840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f940840 0x55d89fd12d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f961520 0x55d89fd1e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f006b1ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d89fd15460 con 0x55d89f961520 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f006b1ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d89fc2e520 con 0x55d89f940840 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f006b1ba740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d89fd155e0 con 0x55d89f938660 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f006b1ba740 10 client.0 _ll_register_callbacks cb 0x55d89fc66c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f005ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f961520 0x55d89fd1e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f005ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f961520 0x55d89fd1e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43418/0 (socket says 172.21.15.16:43418) 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f005ed9d640 1 -- 172.21.15.16:0/1176994131 learned_addr learned my addr 172.21.15.16:0/1176994131 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f005dd9b640 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f940840 0x55d89fd12d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f005e59c640 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f938660 0x55d89fd12840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f005ed9d640 1 -- 172.21.15.16:0/1176994131 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f938660 msgr2=0x55d89fd12840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f005ed9d640 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f938660 0x55d89fd12840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f005ed9d640 1 -- 172.21.15.16:0/1176994131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f940840 msgr2=0x55d89fd12d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f005ed9d640 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f940840 0x55d89fd12d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f005ed9d640 1 -- 172.21.15.16:0/1176994131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d89fa43960 con 0x55d89f961520 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.201+0000 7f005e59c640 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f938660 0x55d89fd12840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.202+0000 7f005ed9d640 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f961520 0x55d89fd1e8d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f005801f870 tx=0x7f0058010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d921fa9fa715c177 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.202+0000 7f0045ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.202+0000 7f0045ffb640 1 -- 172.21.15.16:0/1176994131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0058036060 con 0x55d89f961520 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.202+0000 7f0045ffb640 1 -- 172.21.15.16:0/1176994131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f005801fd40 con 0x55d89f961520 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.202+0000 7f006b1ba740 1 -- 172.21.15.16:0/1176994131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d89fc0c9a0 con 0x55d89f961520 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.202+0000 7f0045ffb640 1 -- 172.21.15.16:0/1176994131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f005803d050 con 0x55d89f961520 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.202+0000 7f006b1ba740 10 client.17181 fetch_fsmap learned FSMap version 42 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.202+0000 7f006b1ba740 1 -- 172.21.15.16:0/1176994131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55d89fd21600 con 0x55d89f961520 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.202+0000 7f0045ffb640 1 -- 172.21.15.16:0/1176994131 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d89fc0c9a0 con 0x55d89f961520 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.202+0000 7f0045ffb640 1 -- 172.21.15.16:0/1176994131 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f005800f040 con 0x55d89f961520 2024-08-28T19:27:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.203+0000 7f006b1ba740 10 client.17181 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.203+0000 7f006b1ba740 10 client.17181 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.203+0000 7f006b1ba740 20 client.17181 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.203+0000 7f006b1ba740 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d89f91fea0 0x55d89fd24a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.203+0000 7f006b1ba740 4 client.17181 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.203+0000 7f006b1ba740 1 -- 172.21.15.16:0/1176994131 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d89fc7f9c0 con 0x55d89f91fea0 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.203+0000 7f005e59c640 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d89f91fea0 0x55d89fd24a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.203+0000 7f005e59c640 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d89f91fea0 0x55d89fd24a10 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=938418233c8b879f server_cookie=9ec2e0cd9d60d3f1 in_seq=0 out_seq=0 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.203+0000 7f0045ffb640 10 client.17181 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f0045ffb640 1 -- 172.21.15.16:0/1176994131 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d89fc7f9c0 con 0x55d89f91fea0 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f0045ffb640 10 client.17181 handle_command_reply: tid=0 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 1 client.17181 shutdown 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 1 -- 172.21.15.16:0/1176994131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f961520 msgr2=0x55d89fd1e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f961520 0x55d89fd1e8d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f005801f870 tx=0x7f0058010ec0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 1 -- 172.21.15.16:0/1176994131 shutdown_connections 2024-08-28T19:27:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f961520 0x55d89fd1e8d0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f940840 0x55d89fd12d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d89f91fea0 0x55d89fd24a10 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f0045ffb640 0 client.17181 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 1 --2- 172.21.15.16:0/1176994131 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f938660 0x55d89fd12840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 1 -- 172.21.15.16:0/1176994131 >> 172.21.15.16:0/1176994131 conn(0x55d89f955910 msgr2=0x55d89fd09e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 1 -- 172.21.15.16:0/1176994131 shutdown_connections 2024-08-28T19:27:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 1 -- 172.21.15.16:0/1176994131 wait complete. 2024-08-28T19:27:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.204+0000 7f006b1ba740 20 client.17181 trim_cache size 0 max 16384 2024-08-28T19:27:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.221+0000 7f006b1ba740 1 Processor -- start 2024-08-28T19:27:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f006b1ba740 1 -- start start 2024-08-28T19:27:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f938660 0x55d89fd14520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd1e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f961520 0x55d89fd1ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f006b1ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00601b2ed0 con 0x55d89f940840 2024-08-28T19:27:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f006b1ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00601b4a40 con 0x55d89f961520 2024-08-28T19:27:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f006b1ba740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00601b35a0 con 0x55d89f938660 2024-08-28T19:27:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f005dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd1e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f005dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd1e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43432/0 (socket says 172.21.15.16:43432) 2024-08-28T19:27:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f005dd9b640 1 -- 172.21.15.16:0/519343827 learned_addr learned my addr 172.21.15.16:0/519343827 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f005dd9b640 1 -- 172.21.15.16:0/519343827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f938660 msgr2=0x55d89fd14520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f005e59c640 1 --2- 172.21.15.16:0/519343827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f938660 0x55d89fd14520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f005dd9b640 1 --2- 172.21.15.16:0/519343827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f938660 0x55d89fd14520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f005dd9b640 1 -- 172.21.15.16:0/519343827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f961520 msgr2=0x55d89fd1ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f005ed9d640 1 --2- 172.21.15.16:0/519343827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f961520 0x55d89fd1ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f005dd9b640 1 --2- 172.21.15.16:0/519343827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f961520 0x55d89fd1ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f005dd9b640 1 -- 172.21.15.16:0/519343827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d89fd1f2c0 con 0x55d89f940840 2024-08-28T19:27:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.222+0000 7f005dd9b640 1 --2- 172.21.15.16:0/519343827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd1e840 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f0054009900 tx=0x7f0054000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574b608b4e32aaf7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.223+0000 7f0047fff640 1 -- 172.21.15.16:0/519343827 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0054035040 con 0x55d89f940840 2024-08-28T19:27:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.223+0000 7f0047fff640 1 -- 172.21.15.16:0/519343827 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00540317e0 con 0x55d89f940840 2024-08-28T19:27:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.223+0000 7f0047fff640 1 -- 172.21.15.16:0/519343827 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0054004750 con 0x55d89f940840 2024-08-28T19:27:11.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.223+0000 7f006b1ba740 1 -- 172.21.15.16:0/519343827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 msgr2=0x55d89fd1e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:11.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.223+0000 7f006b1ba740 1 --2- 172.21.15.16:0/519343827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd1e840 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f0054009900 tx=0x7f0054000bc0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.223+0000 7f006b1ba740 1 -- 172.21.15.16:0/519343827 shutdown_connections 2024-08-28T19:27:11.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.223+0000 7f006b1ba740 1 --2- 172.21.15.16:0/519343827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f961520 0x55d89fd1ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.223+0000 7f006b1ba740 1 --2- 172.21.15.16:0/519343827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f940840 0x55d89fd1e840 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.223+0000 7f006b1ba740 1 --2- 172.21.15.16:0/519343827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f938660 0x55d89fd14520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.223+0000 7f006b1ba740 1 -- 172.21.15.16:0/519343827 >> 172.21.15.16:0/519343827 conn(0x55d89f955910 msgr2=0x55d89fd07780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f006b1ba740 1 -- 172.21.15.16:0/519343827 shutdown_connections 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f006b1ba740 1 -- 172.21.15.16:0/519343827 wait complete. 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f006b1ba740 1 -- start start 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f006b1ba740 1 Processor -- start 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f938660 0x55d89fd263e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f940840 0x55d89fd269c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f006b1ba740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f961520 0x55d89fd881d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f006b1ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00601b4370 con 0x55d89f938660 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f006b1ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00601b3c70 con 0x55d89f940840 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f006b1ba740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00601b0eb0 con 0x55d89f961520 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f006b1ba740 10 client.0 _ll_register_callbacks cb 0x7f0054009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f005e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f938660 0x55d89fd263e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f005e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f938660 0x55d89fd263e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43434/0 (socket says 172.21.15.16:43434) 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f005e59c640 1 -- 172.21.15.16:0/4192553784 learned_addr learned my addr 172.21.15.16:0/4192553784 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.224+0000 7f005dd9b640 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f940840 0x55d89fd269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f005ed9d640 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f961520 0x55d89fd881d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f005e59c640 1 -- 172.21.15.16:0/4192553784 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f961520 msgr2=0x55d89fd881d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f005e59c640 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f961520 0x55d89fd881d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f005e59c640 1 -- 172.21.15.16:0/4192553784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f940840 msgr2=0x55d89fd269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f005e59c640 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f940840 0x55d89fd269c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f005e59c640 1 -- 172.21.15.16:0/4192553784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d89fd88840 con 0x55d89f938660 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f005e59c640 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f938660 0x55d89fd263e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f00500ab640 tx=0x7f0050009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fae010092fb75a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f0045ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:27:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f0045ffb640 1 -- 172.21.15.16:0/4192553784 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050096020 con 0x55d89f938660 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f0045ffb640 1 -- 172.21.15.16:0/4192553784 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f00500e5080 con 0x55d89f938660 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f006b1ba740 1 -- 172.21.15.16:0/4192553784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00540317e0 con 0x55d89f938660 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f0045ffb640 1 -- 172.21.15.16:0/4192553784 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f0050092e60 con 0x55d89f938660 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f006b1ba740 10 client.17187 fetch_fsmap learned FSMap version 42 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f006b1ba740 1 -- 172.21.15.16:0/4192553784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55d89fd8fba0 con 0x55d89f938660 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.225+0000 7f0045ffb640 1 -- 172.21.15.16:0/4192553784 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00540317e0 con 0x55d89f938660 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.226+0000 7f0045ffb640 1 -- 172.21.15.16:0/4192553784 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 42) ==== 8374+0+0 (secure 0 0 0) 0x7f00500b4040 con 0x55d89f938660 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.226+0000 7f006b1ba740 10 client.17187 fetch_fsmap finished waiting for FSMap version 42 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.226+0000 7f006b1ba740 10 client.17187 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.226+0000 7f006b1ba740 20 client.17187 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.226+0000 7f006b1ba740 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d89f8f5250 0x55d89fd92310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.226+0000 7f006b1ba740 4 client.17187 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "8e9c1d21", "release": true, "await": true}] 2024-08-28T19:27:11.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.226+0000 7f006b1ba740 1 -- 172.21.15.16:0/4192553784 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "8e9c1d21", "release": true, "await": true}) -- 0x55d89fc7b460 con 0x55d89f8f5250 2024-08-28T19:27:11.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.226+0000 7f005dd9b640 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d89f8f5250 0x55d89fd92310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:11.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.226+0000 7f0045ffb640 10 client.17187 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:11.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.226+0000 7f005dd9b640 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d89f8f5250 0x55d89fd92310 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6051e689c0e01c9 server_cookie=b05a9e835951de99 in_seq=0 out_seq=0 2024-08-28T19:27:11.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.257+0000 7f0045ffb640 1 -- 172.21.15.16:0/4192553784 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d89fc7b460 con 0x55d89f8f5250 2024-08-28T19:27:11.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.257+0000 7f0045ffb640 10 client.17187 handle_command_reply: tid=0 2024-08-28T19:27:11.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.257+0000 7f006b1ba740 1 client.17187 shutdown 2024-08-28T19:27:11.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f006b1ba740 1 -- 172.21.15.16:0/4192553784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f938660 msgr2=0x55d89fd263e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f006b1ba740 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f938660 0x55d89fd263e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f00500ab640 tx=0x7f0050009f50 comp rx=0 tx=0).stop 2024-08-28T19:27:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f006b1ba740 1 -- 172.21.15.16:0/4192553784 shutdown_connections 2024-08-28T19:27:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f006b1ba740 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d89f8f5250 0x55d89fd92310 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f006b1ba740 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d89f961520 0x55d89fd881d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f006b1ba740 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89f940840 0x55d89fd269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f0045ffb640 0 client.17187 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f006b1ba740 1 --2- 172.21.15.16:0/4192553784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d89f938660 0x55d89fd263e0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f006b1ba740 1 -- 172.21.15.16:0/4192553784 >> 172.21.15.16:0/4192553784 conn(0x55d89f955910 msgr2=0x55d89fd141e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f006b1ba740 1 -- 172.21.15.16:0/4192553784 shutdown_connections 2024-08-28T19:27:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f006b1ba740 1 -- 172.21.15.16:0/4192553784 wait complete. 2024-08-28T19:27:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.258+0000 7f006b1ba740 20 client.17187 trim_cache size 0 max 16384 2024-08-28T19:27:11.261 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:27:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 42, 2024-08-28T19:27:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:27:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 20, 2024-08-28T19:27:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:27:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "8e9c1d21": { 2024-08-28T19:27:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 20, 2024-08-28T19:27:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:27:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:27:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:27:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:27:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:27:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-08-28T19:27:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:27:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:27:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:27:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:27:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:27:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:27:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:27:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:27:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:27:11.265 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:27:11.265 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:27:11.265 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:27:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 -- 172.21.15.16:0/2908555942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00480b4ca0 msgr2=0x7f00480b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00480b4ca0 0x7f00480b71b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f00580092f0 tx=0x7f0058009280 comp rx=0 tx=0).stop 2024-08-28T19:27:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 -- 172.21.15.16:0/2908555942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060102030 msgr2=0x7f006019a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060102030 0x7f006019a330 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f005000d950 tx=0x7f005000de10 comp rx=0 tx=0).stop 2024-08-28T19:27:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 -- 172.21.15.16:0/2908555942 shutdown_connections 2024-08-28T19:27:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00480b4ca0 0x7f00480b71b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00601082c0 0x7f006019fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0060107970 0x7f006019a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 --2- 172.21.15.16:0/2908555942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060102030 0x7f006019a330 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 -- 172.21.15.16:0/2908555942 >> 172.21.15.16:0/2908555942 conn(0x7f0060079c40 msgr2=0x7f0060104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 -- 172.21.15.16:0/2908555942 shutdown_connections 2024-08-28T19:27:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:11.259+0000 7f0064c45640 1 -- 172.21.15.16:0/2908555942 wait complete. 2024-08-28T19:27:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:11 smithi016 ceph-mon[18556]: pgmap v523: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 545 KiB/s wr, 47 op/s 2024-08-28T19:27:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:11 smithi028 ceph-mon[21769]: pgmap v523: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 545 KiB/s wr, 47 op/s 2024-08-28T19:27:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:11 smithi115 ceph-mon[22184]: pgmap v523: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 545 KiB/s wr, 47 op/s 2024-08-28T19:27:11.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 278501 4.54 MB/sec execute 171 sec latency 29162.029 ms 2024-08-28T19:27:11.960 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '8e9c1d21', total seconds elapsed: 33.2 2024-08-28T19:27:12.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 280592 4.60 MB/sec execute 172 sec latency 9.785 ms 2024-08-28T19:27:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:13 smithi028 ceph-mon[21769]: pgmap v524: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 354 KiB/s wr, 44 op/s 2024-08-28T19:27:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:13 smithi115 ceph-mon[22184]: pgmap v524: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 354 KiB/s wr, 44 op/s 2024-08-28T19:27:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:13 smithi016 ceph-mon[18556]: pgmap v524: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 354 KiB/s wr, 44 op/s 2024-08-28T19:27:13.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 282577 4.61 MB/sec execute 173 sec latency 56.608 ms 2024-08-28T19:27:14.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 284186 4.65 MB/sec execute 174 sec latency 12.927 ms 2024-08-28T19:27:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:15 smithi028 ceph-mon[21769]: pgmap v525: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 32 KiB/s wr, 15 op/s 2024-08-28T19:27:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:15 smithi115 ceph-mon[22184]: pgmap v525: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 32 KiB/s wr, 15 op/s 2024-08-28T19:27:15.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:15 smithi016 ceph-mon[18556]: pgmap v525: 129 pgs: 129 active+clean; 167 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 32 KiB/s wr, 15 op/s 2024-08-28T19:27:15.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 286240 4.67 MB/sec execute 175 sec latency 11.647 ms 2024-08-28T19:27:16.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 287796 4.71 MB/sec execute 176 sec latency 13.684 ms 2024-08-28T19:27:17.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:17 smithi028 ceph-mon[21769]: pgmap v526: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 475 KiB/s wr, 75 op/s 2024-08-28T19:27:17.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:17 smithi115 ceph-mon[22184]: pgmap v526: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 475 KiB/s wr, 75 op/s 2024-08-28T19:27:17.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:17 smithi016 ceph-mon[18556]: pgmap v526: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 475 KiB/s wr, 75 op/s 2024-08-28T19:27:17.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 287925 4.68 MB/sec execute 177 sec latency 895.879 ms 2024-08-28T19:27:18.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 287925 4.66 MB/sec execute 178 sec latency 1895.931 ms 2024-08-28T19:27:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:19 smithi028 ceph-mon[21769]: pgmap v527: 129 pgs: 129 active+clean; 173 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 585 KiB/s wr, 101 op/s 2024-08-28T19:27:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:27:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:27:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:27:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:27:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:27:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:19 smithi115 ceph-mon[22184]: pgmap v527: 129 pgs: 129 active+clean; 173 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 585 KiB/s wr, 101 op/s 2024-08-28T19:27:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:27:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:27:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:27:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:27:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:27:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:19 smithi016 ceph-mon[18556]: pgmap v527: 129 pgs: 129 active+clean; 173 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 585 KiB/s wr, 101 op/s 2024-08-28T19:27:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:27:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:27:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:27:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:27:19.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:27:19.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 287925 4.63 MB/sec execute 179 sec latency 2896.043 ms 2024-08-28T19:27:20.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 287925 4.60 MB/sec execute 180 sec latency 3896.138 ms 2024-08-28T19:27:21.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:21 smithi028 ceph-mon[21769]: pgmap v528: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 728 KiB/s wr, 123 op/s 2024-08-28T19:27:21.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:21 smithi115 ceph-mon[22184]: pgmap v528: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 728 KiB/s wr, 123 op/s 2024-08-28T19:27:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:21 smithi016 ceph-mon[18556]: pgmap v528: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 728 KiB/s wr, 123 op/s 2024-08-28T19:27:21.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 287925 4.58 MB/sec execute 181 sec latency 4896.238 ms 2024-08-28T19:27:22.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 287925 4.55 MB/sec execute 182 sec latency 5896.327 ms 2024-08-28T19:27:23.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:23 smithi028 ceph-mon[21769]: pgmap v529: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 906 KiB/s wr, 150 op/s 2024-08-28T19:27:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:23 smithi115 ceph-mon[22184]: pgmap v529: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 906 KiB/s wr, 150 op/s 2024-08-28T19:27:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:23 smithi016 ceph-mon[18556]: pgmap v529: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 906 KiB/s wr, 150 op/s 2024-08-28T19:27:23.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 287925 4.53 MB/sec execute 183 sec latency 6896.400 ms 2024-08-28T19:27:24.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 289642 4.54 MB/sec execute 184 sec latency 7055.372 ms 2024-08-28T19:27:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:25 smithi028 ceph-mon[21769]: pgmap v530: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 906 KiB/s wr, 146 op/s 2024-08-28T19:27:25.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:25 smithi115 ceph-mon[22184]: pgmap v530: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 906 KiB/s wr, 146 op/s 2024-08-28T19:27:25.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:25 smithi016 ceph-mon[18556]: pgmap v530: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 906 KiB/s wr, 146 op/s 2024-08-28T19:27:25.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 291282 4.58 MB/sec execute 185 sec latency 10.020 ms 2024-08-28T19:27:26.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:26 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:26.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:26 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:26.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:26 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:26.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 292651 4.59 MB/sec execute 186 sec latency 19.128 ms 2024-08-28T19:27:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:27 smithi028 ceph-mon[21769]: pgmap v531: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 932 KiB/s wr, 154 op/s 2024-08-28T19:27:27.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:27 smithi115 ceph-mon[22184]: pgmap v531: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 932 KiB/s wr, 154 op/s 2024-08-28T19:27:27.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:27:27.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:27 smithi016 ceph-mon[18556]: pgmap v531: 129 pgs: 129 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 932 KiB/s wr, 154 op/s 2024-08-28T19:27:27.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 294671 4.63 MB/sec execute 187 sec latency 14.604 ms 2024-08-28T19:27:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:27:28.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.444+0000 7faddb9ea640 1 -- 172.21.15.16:0/1046388073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd4107980 msgr2=0x7fadd4107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.444+0000 7faddb9ea640 1 --2- 172.21.15.16:0/1046388073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd4107980 0x7fadd4107d90 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009900 tx=0x7fadc402a410 comp rx=0 tx=0).stop 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.444+0000 7faddb9ea640 1 -- 172.21.15.16:0/1046388073 shutdown_connections 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.444+0000 7faddb9ea640 1 --2- 172.21.15.16:0/1046388073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd41082d0 0x7fadd410eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.444+0000 7faddb9ea640 1 --2- 172.21.15.16:0/1046388073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd4107980 0x7fadd4107d90 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.444+0000 7faddb9ea640 1 --2- 172.21.15.16:0/1046388073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4102040 0x7fadd4102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.444+0000 7faddb9ea640 1 -- 172.21.15.16:0/1046388073 >> 172.21.15.16:0/1046388073 conn(0x7fadd4079c60 msgr2=0x7fadd407a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.444+0000 7faddb9ea640 1 -- 172.21.15.16:0/1046388073 shutdown_connections 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.444+0000 7faddb9ea640 1 -- 172.21.15.16:0/1046388073 wait complete. 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7faddb9ea640 1 Processor -- start 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7faddb9ea640 1 -- start start 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7faddb9ea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd4102040 0x7fadd419b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7faddb9ea640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4107980 0x7fadd419b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7faddb9ea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd41082d0 0x7fadd419bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7faddb9ea640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fadd4112810 con 0x7fadd41082d0 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7faddb9ea640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadd4112690 con 0x7fadd4102040 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7faddb9ea640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fadd4112990 con 0x7fadd4107980 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7fadd9f87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd41082d0 0x7fadd419bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7fadd9786640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd4102040 0x7fadd419b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7fadd9f87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd41082d0 0x7fadd419bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52504/0 (socket says 172.21.15.16:52504) 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7fadd9f87640 1 -- 172.21.15.16:0/1820462237 learned_addr learned my addr 172.21.15.16:0/1820462237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.445+0000 7fadd8f85640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4107980 0x7fadd419b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7fadd9f87640 1 -- 172.21.15.16:0/1820462237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4107980 msgr2=0x7fadd419b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7fadd9f87640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4107980 0x7fadd419b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7fadd9f87640 1 -- 172.21.15.16:0/1820462237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd4102040 msgr2=0x7fadd419b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7fadd9f87640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd4102040 0x7fadd419b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7fadd9f87640 1 -- 172.21.15.16:0/1820462237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd4198fd0 con 0x7fadd41082d0 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7fadd9786640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd4102040 0x7fadd419b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7fadd9f87640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd41082d0 0x7fadd419bbe0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00eb10 tx=0x7fadcc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7fadca7fc640 1 -- 172.21.15.16:0/1820462237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc004b20 con 0x7fadd41082d0 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7faddb9ea640 1 -- 172.21.15.16:0/1820462237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd41992c0 con 0x7fadd41082d0 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7fadca7fc640 1 -- 172.21.15.16:0/1820462237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadcc004cc0 con 0x7fadd41082d0 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7faddb9ea640 1 -- 172.21.15.16:0/1820462237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd41997d0 con 0x7fadd41082d0 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.446+0000 7fadca7fc640 1 -- 172.21.15.16:0/1820462237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc010670 con 0x7fadd41082d0 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.449+0000 7fade1f5f740 1 Processor -- start 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.449+0000 7fadca7fc640 1 -- 172.21.15.16:0/1820462237 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fadcc0108b0 con 0x7fadd41082d0 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.449+0000 7fade1f5f740 1 -- start start 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 0x55b60f02c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 0x55b60f02c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f02cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fade1f5f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b60efa3630 con 0x55b60ec87070 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fade1f5f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b60ef92bc0 con 0x55b60ec66390 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fade1f5f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b60ef96b70 con 0x55b60ec5e1b0 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd8f85640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 0x55b60f02c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd8f85640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 0x55b60f02c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52512/0 (socket says 172.21.15.16:52512) 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd8f85640 1 -- 172.21.15.16:0/2845525298 learned_addr learned my addr 172.21.15.16:0/2845525298 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd9786640 1 --2- 172.21.15.16:0/2845525298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 0x55b60f02c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd8f85640 1 --2- 172.21.15.16:0/2845525298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f02cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd9786640 1 -- 172.21.15.16:0/2845525298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 msgr2=0x55b60f02cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd9786640 1 --2- 172.21.15.16:0/2845525298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f02cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd9786640 1 -- 172.21.15.16:0/2845525298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 msgr2=0x55b60f02c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd9786640 1 --2- 172.21.15.16:0/2845525298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 0x55b60f02c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd9786640 1 -- 172.21.15.16:0/2845525298 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b60ecf2650 con 0x55b60ec5e1b0 2024-08-28T19:27:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadca7fc640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadc00b4cd0 0x7fadc00b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd8f85640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadc00b4cd0 0x7fadc00b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadd9786640 1 --2- 172.21.15.16:0/2845525298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 0x55b60f02c800 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009fa0 tx=0x7fadd00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52a16d3e7d91eaec server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.450+0000 7fadca7fc640 1 -- 172.21.15.16:0/1820462237 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fadcc0e5e30 con 0x7fadd41082d0 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.451+0000 7fadbf7fe640 1 -- 172.21.15.16:0/2845525298 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0004030 con 0x55b60ec5e1b0 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.451+0000 7fadbf7fe640 1 -- 172.21.15.16:0/2845525298 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadd00054d0 con 0x55b60ec5e1b0 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.451+0000 7fadbf7fe640 1 -- 172.21.15.16:0/2845525298 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0004c80 con 0x55b60ec5e1b0 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fade1f5f740 1 -- 172.21.15.16:0/2845525298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 msgr2=0x55b60f02c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fade1f5f740 1 --2- 172.21.15.16:0/2845525298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 0x55b60f02c800 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009fa0 tx=0x7fadd00118d0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fadd8f85640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadc00b4cd0 0x7fadc00b71e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fadc40096c0 tx=0x7fadc4000bc0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fade1f5f740 1 -- 172.21.15.16:0/2845525298 shutdown_connections 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fade1f5f740 1 --2- 172.21.15.16:0/2845525298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f02cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fade1f5f740 1 --2- 172.21.15.16:0/2845525298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 0x55b60f02c800 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fade1f5f740 1 --2- 172.21.15.16:0/2845525298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 0x55b60f02c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fade1f5f740 1 -- 172.21.15.16:0/2845525298 >> 172.21.15.16:0/2845525298 conn(0x55b60ec7b460 msgr2=0x55b60f02a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fade1f5f740 1 -- 172.21.15.16:0/2845525298 shutdown_connections 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fade1f5f740 1 -- 172.21.15.16:0/2845525298 wait complete. 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fade1f5f740 1 -- start start 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.452+0000 7fade1f5f740 1 Processor -- start 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f0379e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f037f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec87070 0x55b60f043d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fade1f5f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b60edf5700 con 0x55b60ec5e1b0 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fade1f5f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b60ef19700 con 0x55b60ec66390 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fade1f5f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b60f03aaa0 con 0x55b60ec87070 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fade1f5f740 10 client.0 _ll_register_callbacks cb 0x55b60ef48dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd9786640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f0379e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd9786640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f0379e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52514/0 (socket says 172.21.15.16:52514) 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd9786640 1 -- 172.21.15.16:0/1166423463 learned_addr learned my addr 172.21.15.16:0/1166423463 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd8f85640 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd9f87640 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec87070 0x55b60f043d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd9786640 1 -- 172.21.15.16:0/1166423463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec87070 msgr2=0x55b60f043d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd9786640 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec87070 0x55b60f043d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd9786640 1 -- 172.21.15.16:0/1166423463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 msgr2=0x55b60f037f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd9786640 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f037f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd9786640 1 -- 172.21.15.16:0/1166423463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b60ef528b0 con 0x55b60ec5e1b0 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd9f87640 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec87070 0x55b60f043d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadd9786640 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f0379e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009700 tx=0x7fadd000fc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c1041404e329c9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.453+0000 7fadbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.454+0000 7fadbcff9640 1 -- 172.21.15.16:0/1166423463 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0026020 con 0x55b60ec5e1b0 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.454+0000 7fadbcff9640 1 -- 172.21.15.16:0/1166423463 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fadd000fe30 con 0x55b60ec5e1b0 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.454+0000 7fade1f5f740 1 -- 172.21.15.16:0/1166423463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadd00054d0 con 0x55b60ec5e1b0 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.454+0000 7fadbcff9640 1 -- 172.21.15.16:0/1166423463 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fadd0031050 con 0x55b60ec5e1b0 2024-08-28T19:27:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.454+0000 7fade1f5f740 10 client.17199 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.454+0000 7fade1f5f740 1 -- 172.21.15.16:0/1166423463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55b60f046810 con 0x55b60ec5e1b0 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.454+0000 7fadbcff9640 1 -- 172.21.15.16:0/1166423463 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadd00054d0 con 0x55b60ec5e1b0 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.454+0000 7fadbcff9640 1 -- 172.21.15.16:0/1166423463 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fadd0021030 con 0x55b60ec5e1b0 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.455+0000 7fade1f5f740 10 client.17199 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.455+0000 7fade1f5f740 10 client.17199 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.455+0000 7fade1f5f740 20 client.17199 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.455+0000 7fade1f5f740 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b60ec459f0 0x55b60f049c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.455+0000 7fade1f5f740 4 client.17199 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.455+0000 7fade1f5f740 1 -- 172.21.15.16:0/1166423463 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b60efa5580 con 0x55b60ec459f0 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.455+0000 7fadd8f85640 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b60ec459f0 0x55b60f049c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.455+0000 7fadbcff9640 10 client.17199 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.455+0000 7fadd8f85640 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b60ec459f0 0x55b60f049c20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18780c4500fb35b2 server_cookie=8770f170f234f423 in_seq=0 out_seq=0 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fadbcff9640 1 -- 172.21.15.16:0/1166423463 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b60efa5580 con 0x55b60ec459f0 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fadbcff9640 10 client.17199 handle_command_reply: tid=0 2024-08-28T19:27:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 1 client.17199 shutdown 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 1 -- 172.21.15.16:0/1166423463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 msgr2=0x55b60f0379e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f0379e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009700 tx=0x7fadd000fc40 comp rx=0 tx=0).stop 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 1 -- 172.21.15.16:0/1166423463 shutdown_connections 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b60ec459f0 0x55b60f049c20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec87070 0x55b60f043d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f037f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 1 --2- 172.21.15.16:0/1166423463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f0379e0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 1 -- 172.21.15.16:0/1166423463 >> 172.21.15.16:0/1166423463 conn(0x55b60ec7b460 msgr2=0x55b60f02f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fadbcff9640 0 client.17199 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 1 -- 172.21.15.16:0/1166423463 shutdown_connections 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 1 -- 172.21.15.16:0/1166423463 wait complete. 2024-08-28T19:27:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.456+0000 7fade1f5f740 20 client.17199 trim_cache size 0 max 16384 2024-08-28T19:27:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fade1f5f740 1 Processor -- start 2024-08-28T19:27:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fade1f5f740 1 -- start start 2024-08-28T19:27:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 0x55b60f02da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f034bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 0x55b60f035110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fade1f5f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fadd41b2e40 con 0x55b60ec87070 2024-08-28T19:27:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fade1f5f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadd41b4980 con 0x55b60ec66390 2024-08-28T19:27:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fade1f5f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fadd41b3510 con 0x55b60ec5e1b0 2024-08-28T19:27:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fadd9f87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 0x55b60f035110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fadd9786640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 0x55b60f02da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fadd9786640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 0x55b60f02da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:43052/0 (socket says 172.21.15.16:43052) 2024-08-28T19:27:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.478+0000 7fadd9786640 1 -- 172.21.15.16:0/2715069761 learned_addr learned my addr 172.21.15.16:0/2715069761 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.479+0000 7fadd9786640 1 -- 172.21.15.16:0/2715069761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 msgr2=0x55b60f034bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.479+0000 7fadd9786640 1 --2- 172.21.15.16:0/2715069761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f034bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.479+0000 7fadd9786640 1 -- 172.21.15.16:0/2715069761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 msgr2=0x55b60f035110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.479+0000 7fadd9786640 1 --2- 172.21.15.16:0/2715069761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 0x55b60f035110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.479+0000 7fadd9786640 1 -- 172.21.15.16:0/2715069761 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b60ec45c20 con 0x55b60ec5e1b0 2024-08-28T19:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.479+0000 7fadd9f87640 1 --2- 172.21.15.16:0/2715069761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 0x55b60f035110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.479+0000 7fadd9786640 1 --2- 172.21.15.16:0/2715069761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 0x55b60f02da00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009730 tx=0x7fadd0009ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c5dd412f6a5c7b6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.479+0000 7fadbeffd640 1 -- 172.21.15.16:0/2715069761 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0026040 con 0x55b60ec5e1b0 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.479+0000 7fadbeffd640 1 -- 172.21.15.16:0/2715069761 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadd00054a0 con 0x55b60ec5e1b0 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.479+0000 7fadbeffd640 1 -- 172.21.15.16:0/2715069761 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd001e040 con 0x55b60ec5e1b0 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.480+0000 7fade1f5f740 1 -- 172.21.15.16:0/2715069761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 msgr2=0x55b60f02da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.480+0000 7fade1f5f740 1 --2- 172.21.15.16:0/2715069761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 0x55b60f02da00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009730 tx=0x7fadd0009ea0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.480+0000 7fade1f5f740 1 -- 172.21.15.16:0/2715069761 shutdown_connections 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.480+0000 7fade1f5f740 1 --2- 172.21.15.16:0/2715069761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec87070 0x55b60f035110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.480+0000 7fade1f5f740 1 --2- 172.21.15.16:0/2715069761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f034bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.480+0000 7fade1f5f740 1 --2- 172.21.15.16:0/2715069761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec5e1b0 0x55b60f02da00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.480+0000 7fade1f5f740 1 -- 172.21.15.16:0/2715069761 >> 172.21.15.16:0/2715069761 conn(0x55b60ec7b460 msgr2=0x55b60f02d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.481+0000 7fade1f5f740 1 -- 172.21.15.16:0/2715069761 shutdown_connections 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.481+0000 7fade1f5f740 1 -- 172.21.15.16:0/2715069761 wait complete. 2024-08-28T19:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.481+0000 7fade1f5f740 1 -- start start 2024-08-28T19:27:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.481+0000 7fade1f5f740 1 Processor -- start 2024-08-28T19:27:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f03d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f03dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fade1f5f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec87070 0x55b60f0ab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fade1f5f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fadd41b42b0 con 0x55b60ec5e1b0 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fade1f5f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadd41b3be0 con 0x55b60ec66390 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fade1f5f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fadd41b0e20 con 0x55b60ec87070 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fade1f5f740 10 client.0 _ll_register_callbacks cb 0x7fadd0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fadd9786640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fadd9786640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f03d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52524/0 (socket says 172.21.15.16:52524) 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fadd9786640 1 -- 172.21.15.16:0/3200925783 learned_addr learned my addr 172.21.15.16:0/3200925783 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fadd8f85640 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f03dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fadd9f87640 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec87070 0x55b60f0ab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fadd9786640 1 -- 172.21.15.16:0/3200925783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec87070 msgr2=0x55b60f0ab4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fadd9786640 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec87070 0x55b60f0ab4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fadd9786640 1 -- 172.21.15.16:0/3200925783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 msgr2=0x55b60f03dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fadd9786640 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f03dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.482+0000 7fadd9786640 1 -- 172.21.15.16:0/3200925783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b60f0abb50 con 0x55b60ec5e1b0 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.483+0000 7fadd9786640 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f03d760 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fadd00092c0 tx=0x7fadd0005650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59679981db83552 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.483+0000 7fadbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.483+0000 7fadbcff9640 1 -- 172.21.15.16:0/3200925783 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0026070 con 0x55b60ec5e1b0 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.483+0000 7fadbcff9640 1 -- 172.21.15.16:0/3200925783 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fadd00050d0 con 0x55b60ec5e1b0 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.483+0000 7fade1f5f740 1 -- 172.21.15.16:0/3200925783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadd00054a0 con 0x55b60ec5e1b0 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.483+0000 7fadbcff9640 1 -- 172.21.15.16:0/3200925783 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fadd0044050 con 0x55b60ec5e1b0 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.483+0000 7fade1f5f740 10 client.17205 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:28.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.483+0000 7fade1f5f740 1 -- 172.21.15.16:0/3200925783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55b60f0b4020 con 0x55b60ec5e1b0 2024-08-28T19:27:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.483+0000 7fadbcff9640 1 -- 172.21.15.16:0/3200925783 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadd003b070 con 0x55b60ec5e1b0 2024-08-28T19:27:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.484+0000 7fadbcff9640 1 -- 172.21.15.16:0/3200925783 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fadd0021070 con 0x55b60ec5e1b0 2024-08-28T19:27:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.484+0000 7fade1f5f740 10 client.17205 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.484+0000 7fade1f5f740 10 client.17205 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:27:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.484+0000 7fade1f5f740 20 client.17205 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.484+0000 7fade1f5f740 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b60ec1ada0 0x55b60f0b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.484+0000 7fade1f5f740 4 client.17205 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:27:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.484+0000 7fade1f5f740 1 -- 172.21.15.16:0/3200925783 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fadd00054a0 con 0x55b60ec1ada0 2024-08-28T19:27:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.484+0000 7fadd8f85640 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b60ec1ada0 0x55b60f0b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fadbcff9640 10 client.17205 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fadd8f85640 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b60ec1ada0 0x55b60f0b66e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8139576a3250d2bc server_cookie=219581363656a15b in_seq=0 out_seq=0 2024-08-28T19:27:28.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fadbcff9640 1 -- 172.21.15.16:0/3200925783 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2646 (crc 0 0 0) 0x7fadd00054a0 con 0x55b60ec1ada0 2024-08-28T19:27:28.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fadbcff9640 10 client.17205 handle_command_reply: tid=0 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fade1f5f740 1 client.17205 shutdown 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fade1f5f740 1 -- 172.21.15.16:0/3200925783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 msgr2=0x55b60f03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fade1f5f740 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f03d760 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fadd00092c0 tx=0x7fadd0005650 comp rx=0 tx=0).stop 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fade1f5f740 1 -- 172.21.15.16:0/3200925783 shutdown_connections 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fade1f5f740 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b60ec1ada0 0x55b60f0b66e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fade1f5f740 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b60ec87070 0x55b60f0ab4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fade1f5f740 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b60ec66390 0x55b60f03dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fade1f5f740 1 --2- 172.21.15.16:0/3200925783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b60ec5e1b0 0x55b60f03d760 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fadbcff9640 0 client.17205 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.485+0000 7fade1f5f740 1 -- 172.21.15.16:0/3200925783 >> 172.21.15.16:0/3200925783 conn(0x55b60ec7b460 msgr2=0x55b60f038ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.486+0000 7fade1f5f740 1 -- 172.21.15.16:0/3200925783 shutdown_connections 2024-08-28T19:27:28.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.486+0000 7fade1f5f740 1 -- 172.21.15.16:0/3200925783 wait complete. 2024-08-28T19:27:28.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.486+0000 7fade1f5f740 20 client.17205 trim_cache size 0 max 16384 2024-08-28T19:27:28.491 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:27:28.491 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:27:28.491 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:27:28.491 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:27:28.491 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:27:28.491 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:27:28.491 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:28.491 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:28.491 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:27:28.492 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:27:28.493 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:28.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:28.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:28.496 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:27:28.496 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:27:28.496 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:28.496 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:28.496 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:28.496 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:28.496 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:28.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:27:28.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:27:28.496 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:27:28.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.486+0000 7faddb9ea640 1 -- 172.21.15.16:0/1820462237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadc00b4cd0 msgr2=0x7fadc00b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:28.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.486+0000 7faddb9ea640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadc00b4cd0 0x7fadc00b71e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fadc40096c0 tx=0x7fadc4000bc0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.486+0000 7faddb9ea640 1 -- 172.21.15.16:0/1820462237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd41082d0 msgr2=0x7fadd419bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:28.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.486+0000 7faddb9ea640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd41082d0 0x7fadd419bbe0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00eb10 tx=0x7fadcc00c2d0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.486+0000 7faddb9ea640 1 -- 172.21.15.16:0/1820462237 shutdown_connections 2024-08-28T19:27:28.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.486+0000 7faddb9ea640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd41082d0 0x7fadd419bbe0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.486+0000 7faddb9ea640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4107980 0x7fadd419b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.486+0000 7faddb9ea640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fadc00b4cd0 0x7fadc00b71e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.487+0000 7faddb9ea640 1 --2- 172.21.15.16:0/1820462237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd4102040 0x7fadd419b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.487+0000 7faddb9ea640 1 -- 172.21.15.16:0/1820462237 >> 172.21.15.16:0/1820462237 conn(0x7fadd4079c60 msgr2=0x7fadd4103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.487+0000 7faddb9ea640 1 -- 172.21.15.16:0/1820462237 shutdown_connections 2024-08-28T19:27:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:28.487+0000 7faddb9ea640 1 -- 172.21.15.16:0/1820462237 wait complete. 2024-08-28T19:27:28.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 295860 4.64 MB/sec execute 188 sec latency 41.635 ms 2024-08-28T19:27:29.330 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:27:29.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:27:29.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:29 smithi016 ceph-mon[18556]: pgmap v532: 129 pgs: 129 active+clean; 178 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 684 KiB/s wr, 124 op/s 2024-08-28T19:27:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:29 smithi028 ceph-mon[21769]: pgmap v532: 129 pgs: 129 active+clean; 178 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 684 KiB/s wr, 124 op/s 2024-08-28T19:27:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:29 smithi115 ceph-mon[22184]: pgmap v532: 129 pgs: 129 active+clean; 178 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 684 KiB/s wr, 124 op/s 2024-08-28T19:27:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:27:29.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 296834 4.62 MB/sec execute 189 sec latency 36.929 ms 2024-08-28T19:27:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.134+0000 7f5b22b49640 1 -- 172.21.15.16:0/3913336104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1c104750 msgr2=0x7f5b1c104b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.134+0000 7f5b22b49640 1 --2- 172.21.15.16:0/3913336104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1c104750 0x7f5b1c104b60 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009900 tx=0x7f5b1002a410 comp rx=0 tx=0).stop 2024-08-28T19:27:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.134+0000 7f5b22b49640 1 -- 172.21.15.16:0/3913336104 shutdown_connections 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.134+0000 7f5b22b49640 1 --2- 172.21.15.16:0/3913336104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b1c10dbb0 0x7f5b1c1100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.134+0000 7f5b22b49640 1 --2- 172.21.15.16:0/3913336104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b1c1050a0 0x7f5b1c10d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.134+0000 7f5b22b49640 1 --2- 172.21.15.16:0/3913336104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1c104750 0x7f5b1c104b60 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.134+0000 7f5b22b49640 1 -- 172.21.15.16:0/3913336104 >> 172.21.15.16:0/3913336104 conn(0x7f5b1c0794d0 msgr2=0x7f5b1c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.134+0000 7f5b22b49640 1 -- 172.21.15.16:0/3913336104 shutdown_connections 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.134+0000 7f5b22b49640 1 -- 172.21.15.16:0/3913336104 wait complete. 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.135+0000 7f5b22b49640 1 Processor -- start 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.135+0000 7f5b22b49640 1 -- start start 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.135+0000 7f5b22b49640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b1c104750 0x7f5b1c19a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.135+0000 7f5b22b49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1c1050a0 0x7f5b1c19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.135+0000 7f5b22b49640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b1c10dbb0 0x7f5b1c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.135+0000 7f5b22b49640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c113810 con 0x7f5b1c1050a0 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.135+0000 7f5b22b49640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c113690 con 0x7f5b1c104750 2024-08-28T19:27:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.135+0000 7f5b22b49640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c113990 con 0x7f5b1c10dbb0 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b208e5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b1c104750 0x7f5b1c19a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b1bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1c1050a0 0x7f5b1c19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b208e5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b1c104750 0x7f5b1c19a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41626/0 (socket says 172.21.15.16:41626) 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b208e5640 1 -- 172.21.15.16:0/1354398406 learned_addr learned my addr 172.21.15.16:0/1354398406 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b210e6640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b1c10dbb0 0x7f5b1c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b208e5640 1 -- 172.21.15.16:0/1354398406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b1c10dbb0 msgr2=0x7f5b1c19fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b208e5640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b1c10dbb0 0x7f5b1c19fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b208e5640 1 -- 172.21.15.16:0/1354398406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1c1050a0 msgr2=0x7f5b1c19a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b208e5640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1c1050a0 0x7f5b1c19a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b208e5640 1 -- 172.21.15.16:0/1354398406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1c1a02b0 con 0x7f5b1c104750 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b210e6640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b1c10dbb0 0x7f5b1c19fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b1bfff640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1c1050a0 0x7f5b1c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.136+0000 7f5b208e5640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b1c104750 0x7f5b1c19a340 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5b10032fd0 tx=0x7f5b10031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.137+0000 7f5b19ffb640 1 -- 172.21.15.16:0/1354398406 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1003f030 con 0x7f5b1c104750 2024-08-28T19:27:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.137+0000 7f5b19ffb640 1 -- 172.21.15.16:0/1354398406 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b10004490 con 0x7f5b1c104750 2024-08-28T19:27:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.137+0000 7f5b22b49640 1 -- 172.21.15.16:0/1354398406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b1c1a0540 con 0x7f5b1c104750 2024-08-28T19:27:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.137+0000 7f5b19ffb640 1 -- 172.21.15.16:0/1354398406 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b10035070 con 0x7f5b1c104750 2024-08-28T19:27:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.137+0000 7f5b22b49640 1 -- 172.21.15.16:0/1354398406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b1c1a6d00 con 0x7f5b1c104750 2024-08-28T19:27:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.138+0000 7f5b22b49640 1 -- 172.21.15.16:0/1354398406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b1c104b60 con 0x7f5b1c104750 2024-08-28T19:27:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.140+0000 7f5b19ffb640 1 -- 172.21.15.16:0/1354398406 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5b10042030 con 0x7f5b1c104750 2024-08-28T19:27:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.140+0000 7f5b19ffb640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b0c0b4ca0 0x7f5b0c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.141+0000 7f5b1bfff640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b0c0b4ca0 0x7f5b0c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.141+0000 7f5b19ffb640 1 -- 172.21.15.16:0/1354398406 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f5b10106030 con 0x7f5b1c104750 2024-08-28T19:27:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.141+0000 7f5b1bfff640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b0c0b4ca0 0x7f5b0c0b71b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c1021a0 tx=0x7f5b08005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:30.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.146+0000 7f5b19ffb640 1 -- 172.21.15.16:0/1354398406 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f5b100ccab0 con 0x7f5b1c104750 2024-08-28T19:27:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.382+0000 7f5b22b49640 1 -- 172.21.15.16:0/1354398406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b1c108cb0 con 0x7f5b1c104750 2024-08-28T19:27:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.382+0000 7f5b19ffb640 1 -- 172.21.15.16:0/1354398406 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11977 (secure 0 0 0) 0x7f5b100cc3b0 con 0x7f5b1c104750 2024-08-28T19:27:30.384 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:27:30.384 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"btime":"2024-08-28T19:27:25:273422+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":43,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:27:24.534899+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:27:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 43 2024-08-28T19:27:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 -- 172.21.15.16:0/1354398406 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b0c0b4ca0 msgr2=0x7f5b0c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b0c0b4ca0 0x7f5b0c0b71b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c1021a0 tx=0x7f5b08005f90 comp rx=0 tx=0).stop 2024-08-28T19:27:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 -- 172.21.15.16:0/1354398406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b1c104750 msgr2=0x7f5b1c19a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b1c104750 0x7f5b1c19a340 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5b10032fd0 tx=0x7f5b10031a20 comp rx=0 tx=0).stop 2024-08-28T19:27:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 -- 172.21.15.16:0/1354398406 shutdown_connections 2024-08-28T19:27:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b1c10dbb0 0x7f5b1c19fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1c1050a0 0x7f5b1c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b0c0b4ca0 0x7f5b0c0b71b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 --2- 172.21.15.16:0/1354398406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b1c104750 0x7f5b1c19a340 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 -- 172.21.15.16:0/1354398406 >> 172.21.15.16:0/1354398406 conn(0x7f5b1c0794d0 msgr2=0x7f5b1c100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 -- 172.21.15.16:0/1354398406 shutdown_connections 2024-08-28T19:27:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:30.385+0000 7f5b22b49640 1 -- 172.21.15.16:0/1354398406 wait complete. 2024-08-28T19:27:30.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 298393 4.66 MB/sec execute 190 sec latency 41.151 ms 2024-08-28T19:27:31.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:27:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:27:31.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:31 smithi016 ceph-mon[18556]: pgmap v533: 129 pgs: 129 active+clean; 181 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 779 KiB/s wr, 116 op/s 2024-08-28T19:27:31.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:31 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1354398406' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:27:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:31 smithi028 ceph-mon[21769]: pgmap v533: 129 pgs: 129 active+clean; 181 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 779 KiB/s wr, 116 op/s 2024-08-28T19:27:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:31 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1354398406' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:27:31.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:31 smithi115 ceph-mon[22184]: pgmap v533: 129 pgs: 129 active+clean; 181 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 779 KiB/s wr, 116 op/s 2024-08-28T19:27:31.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:31 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1354398406' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:27:31.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 300266 4.67 MB/sec execute 191 sec latency 26.818 ms 2024-08-28T19:27:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.965+0000 7fc51f52b640 1 -- 172.21.15.16:0/1757854543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc518105b50 msgr2=0x7fc518105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.965+0000 7fc51f52b640 1 --2- 172.21.15.16:0/1757854543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc518105b50 0x7fc518105f60 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fc514009900 tx=0x7fc51402a410 comp rx=0 tx=0).stop 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.965+0000 7fc51f52b640 1 -- 172.21.15.16:0/1757854543 shutdown_connections 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.965+0000 7fc51f52b640 1 --2- 172.21.15.16:0/1757854543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51810bde0 0x7fc518112610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.965+0000 7fc51f52b640 1 --2- 172.21.15.16:0/1757854543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc51810b490 0x7fc51810b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.965+0000 7fc51f52b640 1 --2- 172.21.15.16:0/1757854543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc518105b50 0x7fc518105f60 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.965+0000 7fc51f52b640 1 -- 172.21.15.16:0/1757854543 >> 172.21.15.16:0/1757854543 conn(0x7fc518101560 msgr2=0x7fc5181039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.965+0000 7fc51f52b640 1 -- 172.21.15.16:0/1757854543 shutdown_connections 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.965+0000 7fc51f52b640 1 -- 172.21.15.16:0/1757854543 wait complete. 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.965+0000 7fc51f52b640 1 Processor -- start 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51f52b640 1 -- start start 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51f52b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc518105b50 0x7fc51819c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51f52b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc51810b490 0x7fc51819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51f52b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc51810bde0 0x7fc5181a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51f52b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc518115aa0 con 0x7fc51810b490 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51f52b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc518115920 con 0x7fc51810bde0 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51f52b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc518115c20 con 0x7fc518105b50 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51cac6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc51810b490 0x7fc51819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51d2c7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc518105b50 0x7fc51819c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51cac6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc51810b490 0x7fc51819ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52558/0 (socket says 172.21.15.16:52558) 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51cac6640 1 -- 172.21.15.16:0/2198721217 learned_addr learned my addr 172.21.15.16:0/2198721217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51dac8640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc51810bde0 0x7fc5181a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51cac6640 1 -- 172.21.15.16:0/2198721217 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc518105b50 msgr2=0x7fc51819c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51cac6640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc518105b50 0x7fc51819c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51cac6640 1 -- 172.21.15.16:0/2198721217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc51810bde0 msgr2=0x7fc5181a1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51cac6640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc51810bde0 0x7fc5181a1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51cac6640 1 -- 172.21.15.16:0/2198721217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5181a2400 con 0x7fc51810b490 2024-08-28T19:27:31.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc518105b50 0x7fc51819c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:31.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.966+0000 7fc51dac8640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc51810bde0 0x7fc5181a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:31.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.967+0000 7fc51cac6640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc51810b490 0x7fc51819ca30 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fc50800e9f0 tx=0x7fc50800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:31.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.967+0000 7fc50e7fc640 1 -- 172.21.15.16:0/2198721217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50800ccc0 con 0x7fc51810b490 2024-08-28T19:27:31.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.967+0000 7fc50e7fc640 1 -- 172.21.15.16:0/2198721217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5080108e0 con 0x7fc51810b490 2024-08-28T19:27:31.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.967+0000 7fc51f52b640 1 -- 172.21.15.16:0/2198721217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5181a26f0 con 0x7fc51810b490 2024-08-28T19:27:31.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.967+0000 7fc50e7fc640 1 -- 172.21.15.16:0/2198721217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508018860 con 0x7fc51810b490 2024-08-28T19:27:31.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.967+0000 7fc51f52b640 1 -- 172.21.15.16:0/2198721217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5181a8e50 con 0x7fc51810b490 2024-08-28T19:27:31.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.970+0000 7fc525aa0740 1 Processor -- start 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.970+0000 7fc525aa0740 1 -- start start 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 0x558c12579310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121ab1b0 0x558c12579850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c12579d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc525aa0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558c124f0680 con 0x558c121b3390 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc525aa0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c124dfc10 con 0x558c121ab1b0 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc525aa0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558c124e3bc0 con 0x558c121d4070 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc51d2c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121ab1b0 0x558c12579850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc51dac8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 0x558c12579310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc51dac8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 0x558c12579310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:43112/0 (socket says 172.21.15.16:43112) 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc51d2c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121ab1b0 0x558c12579850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41652/0 (socket says 172.21.15.16:41652) 2024-08-28T19:27:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc51dac8640 1 -- 172.21.15.16:0/2115362063 learned_addr learned my addr 172.21.15.16:0/2115362063 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc50e7fc640 1 -- 172.21.15.16:0/2198721217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc508010430 con 0x7fc51810b490 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc51dac8640 1 -- 172.21.15.16:0/2115362063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121ab1b0 msgr2=0x558c12579850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc51dac8640 1 --2- 172.21.15.16:0/2115362063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121ab1b0 0x558c12579850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc51dac8640 1 -- 172.21.15.16:0/2115362063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 msgr2=0x558c12579d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc51dac8640 1 --2- 172.21.15.16:0/2115362063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c12579d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc51dac8640 1 -- 172.21.15.16:0/2115362063 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c1223f650 con 0x558c121d4070 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.971+0000 7fc51dac8640 1 --2- 172.21.15.16:0/2115362063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 0x558c12579310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc5100117e0 tx=0x7fc510011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=759c36beef2d5118 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.972+0000 7fc5037fe640 1 -- 172.21.15.16:0/2115362063 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc510011310 con 0x558c121d4070 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.972+0000 7fc5037fe640 1 -- 172.21.15.16:0/2115362063 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc510003e20 con 0x558c121d4070 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.972+0000 7fc5037fe640 1 -- 172.21.15.16:0/2115362063 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5100044f0 con 0x558c121d4070 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.972+0000 7fc50e7fc640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5040b4ca0 0x7fc5040b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.972+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5040b4ca0 0x7fc5040b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.972+0000 7fc50e7fc640 1 -- 172.21.15.16:0/2198721217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc5080e5b40 con 0x7fc51810b490 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc525aa0740 1 -- 172.21.15.16:0/2115362063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 msgr2=0x558c12579310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5040b4ca0 0x7fc5040b71b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc514031440 tx=0x7fc514032ed0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc525aa0740 1 --2- 172.21.15.16:0/2115362063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 0x558c12579310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc5100117e0 tx=0x7fc510011ca0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc525aa0740 1 -- 172.21.15.16:0/2115362063 shutdown_connections 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc525aa0740 1 --2- 172.21.15.16:0/2115362063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c12579d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc525aa0740 1 --2- 172.21.15.16:0/2115362063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121ab1b0 0x558c12579850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc525aa0740 1 --2- 172.21.15.16:0/2115362063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 0x558c12579310 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc525aa0740 1 -- 172.21.15.16:0/2115362063 >> 172.21.15.16:0/2115362063 conn(0x558c121c8460 msgr2=0x558c12577200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc525aa0740 1 -- 172.21.15.16:0/2115362063 shutdown_connections 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc525aa0740 1 -- 172.21.15.16:0/2115362063 wait complete. 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc525aa0740 1 -- start start 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.973+0000 7fc525aa0740 1 Processor -- start 2024-08-28T19:27:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121ab1b0 0x558c1258d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c1258d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 0x558c12599580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc525aa0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558c12342700 con 0x558c121b3390 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc525aa0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c12466750 con 0x558c121ab1b0 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc525aa0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558c12590220 con 0x558c121d4070 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc525aa0740 10 client.0 _ll_register_callbacks cb 0x558c12495e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc51cac6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c1258d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc51cac6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c1258d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52562/0 (socket says 172.21.15.16:52562) 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc51cac6640 1 -- 172.21.15.16:0/3518938099 learned_addr learned my addr 172.21.15.16:0/3518938099 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc51dac8640 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 0x558c12599580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121ab1b0 0x558c1258d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc51dac8640 1 -- 172.21.15.16:0/3518938099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121ab1b0 msgr2=0x558c1258d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc51dac8640 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121ab1b0 0x558c1258d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc51dac8640 1 -- 172.21.15.16:0/3518938099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 msgr2=0x558c1258d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc51dac8640 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c1258d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc51dac8640 1 -- 172.21.15.16:0/3518938099 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c1249f900 con 0x558c121d4070 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.974+0000 7fc51cac6640 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c1258d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc51dac8640 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 0x558c12599580 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc5100041d0 tx=0x7fc510011340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34f5067d3126d266 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc500ff9640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc500ff9640 1 -- 172.21.15.16:0/3518938099 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51000fdb0 con 0x558c121d4070 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc525aa0740 1 -- 172.21.15.16:0/3518938099 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc510003e20 con 0x558c121d4070 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc500ff9640 1 -- 172.21.15.16:0/3518938099 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc510004a40 con 0x558c121d4070 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc500ff9640 1 -- 172.21.15.16:0/3518938099 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fc510004c40 con 0x558c121d4070 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc525aa0740 10 client.26575 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc525aa0740 1 -- 172.21.15.16:0/3518938099 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x558c1259c040 con 0x558c121d4070 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc500ff9640 1 -- 172.21.15.16:0/3518938099 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc510003e20 con 0x558c121d4070 2024-08-28T19:27:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc500ff9640 1 -- 172.21.15.16:0/3518938099 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fc510018020 con 0x558c121d4070 2024-08-28T19:27:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc525aa0740 10 client.26575 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc525aa0740 10 client.26575 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:27:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc525aa0740 20 client.26575 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x558c121929f0 0x558c1259f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc525aa0740 4 client.26575 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:27:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.975+0000 7fc525aa0740 1 -- 172.21.15.16:0/3518938099 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c124f25d0 con 0x558c121929f0 2024-08-28T19:27:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.976+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x558c121929f0 0x558c1259f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.976+0000 7fc500ff9640 10 client.26575 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:27:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.976+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x558c121929f0 0x558c1259f3a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=97115fb78f02f80b server_cookie=27425549cad1ef50 in_seq=0 out_seq=0 2024-08-28T19:27:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc500ff9640 1 -- 172.21.15.16:0/3518938099 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558c124f25d0 con 0x558c121929f0 2024-08-28T19:27:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc500ff9640 10 client.26575 handle_command_reply: tid=0 2024-08-28T19:27:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc525aa0740 1 client.26575 shutdown 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc525aa0740 1 -- 172.21.15.16:0/3518938099 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 msgr2=0x558c12599580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 0x558c12599580 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc5100041d0 tx=0x7fc510011340 comp rx=0 tx=0).stop 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc525aa0740 1 -- 172.21.15.16:0/3518938099 shutdown_connections 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x558c121929f0 0x558c1259f3a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121d4070 0x558c12599580 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c1258d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3518938099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121ab1b0 0x558c1258d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc500ff9640 0 client.26575 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc525aa0740 1 -- 172.21.15.16:0/3518938099 >> 172.21.15.16:0/3518938099 conn(0x558c121c8460 msgr2=0x558c125757c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc525aa0740 1 -- 172.21.15.16:0/3518938099 shutdown_connections 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.977+0000 7fc525aa0740 1 -- 172.21.15.16:0/3518938099 wait complete. 2024-08-28T19:27:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.978+0000 7fc525aa0740 20 client.26575 trim_cache size 0 max 16384 2024-08-28T19:27:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.993+0000 7fc525aa0740 1 Processor -- start 2024-08-28T19:27:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.993+0000 7fc525aa0740 1 -- start start 2024-08-28T19:27:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.993+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121ab1b0 0x558c1258d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.993+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121b3390 0x558c1258ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.993+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 0x558c12599480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.993+0000 7fc525aa0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181aea30 con 0x558c121ab1b0 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.993+0000 7fc525aa0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181b0570 con 0x558c121d4070 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.993+0000 7fc525aa0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181af100 con 0x558c121b3390 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.993+0000 7fc51d2c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121ab1b0 0x558c1258d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.993+0000 7fc51d2c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121ab1b0 0x558c1258d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52568/0 (socket says 172.21.15.16:52568) 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.993+0000 7fc51d2c7640 1 -- 172.21.15.16:0/3844071919 learned_addr learned my addr 172.21.15.16:0/3844071919 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc51dac8640 1 --2- 172.21.15.16:0/3844071919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 0x558c12599480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc51cac6640 1 --2- 172.21.15.16:0/3844071919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121b3390 0x558c1258ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc51d2c7640 1 -- 172.21.15.16:0/3844071919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121b3390 msgr2=0x558c1258ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/3844071919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121b3390 0x558c1258ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc51d2c7640 1 -- 172.21.15.16:0/3844071919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 msgr2=0x558c12599480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/3844071919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 0x558c12599480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc51d2c7640 1 -- 172.21.15.16:0/3844071919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c1258b780 con 0x558c121ab1b0 2024-08-28T19:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc51dac8640 1 --2- 172.21.15.16:0/3844071919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 0x558c12599480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:31.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/3844071919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121ab1b0 0x558c1258d9f0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc51402a930 tx=0x7fc51403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48dec0830d20ee3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:31.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc502ffd640 1 -- 172.21.15.16:0/3844071919 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc514057070 con 0x558c121ab1b0 2024-08-28T19:27:31.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc502ffd640 1 -- 172.21.15.16:0/3844071919 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc514004770 con 0x558c121ab1b0 2024-08-28T19:27:31.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.994+0000 7fc502ffd640 1 -- 172.21.15.16:0/3844071919 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc514035040 con 0x558c121ab1b0 2024-08-28T19:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.995+0000 7fc525aa0740 1 -- 172.21.15.16:0/3844071919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121ab1b0 msgr2=0x558c1258d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.995+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3844071919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121ab1b0 0x558c1258d9f0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc51402a930 tx=0x7fc51403f000 comp rx=0 tx=0).stop 2024-08-28T19:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.995+0000 7fc525aa0740 1 -- 172.21.15.16:0/3844071919 shutdown_connections 2024-08-28T19:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.995+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3844071919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 0x558c12599480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.995+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3844071919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121b3390 0x558c1258ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.995+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3844071919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121ab1b0 0x558c1258d9f0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.995+0000 7fc525aa0740 1 -- 172.21.15.16:0/3844071919 >> 172.21.15.16:0/3844071919 conn(0x558c121c8460 msgr2=0x558c1258e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.995+0000 7fc525aa0740 1 -- 172.21.15.16:0/3844071919 shutdown_connections 2024-08-28T19:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.995+0000 7fc525aa0740 1 -- 172.21.15.16:0/3844071919 wait complete. 2024-08-28T19:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc525aa0740 1 -- start start 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc525aa0740 1 Processor -- start 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121ab1b0 0x558c12591810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c12591d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc525aa0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 0x558c12608c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc525aa0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181afea0 con 0x558c121b3390 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc525aa0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181af7d0 con 0x558c121d4070 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc525aa0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181aca10 con 0x558c121ab1b0 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc525aa0740 10 client.0 _ll_register_callbacks cb 0x7fc5100093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc51cac6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c12591d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc51d2c7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121ab1b0 0x558c12591810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc51cac6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c12591d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52578/0 (socket says 172.21.15.16:52578) 2024-08-28T19:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc51cac6640 1 -- 172.21.15.16:0/3227072018 learned_addr learned my addr 172.21.15.16:0/3227072018 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.996+0000 7fc51dac8640 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 0x558c12608c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc51cac6640 1 -- 172.21.15.16:0/3227072018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121ab1b0 msgr2=0x558c12591810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc51cac6640 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121ab1b0 0x558c12591810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc51cac6640 1 -- 172.21.15.16:0/3227072018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 msgr2=0x558c12608c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc51cac6640 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 0x558c12608c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc51cac6640 1 -- 172.21.15.16:0/3227072018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c12592640 con 0x558c121b3390 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121ab1b0 0x558c12591810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc51dac8640 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 0x558c12608c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc51cac6640 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c12591d50 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc5080abf00 tx=0x7fc50800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73780eddd08de84 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc500ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc500ff9640 1 -- 172.21.15.16:0/3227072018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5080abf70 con 0x558c121b3390 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc500ff9640 1 -- 172.21.15.16:0/3227072018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7fc5080930c0 con 0x558c121b3390 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc525aa0740 1 -- 172.21.15.16:0/3227072018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc514004770 con 0x558c121b3390 2024-08-28T19:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc500ff9640 1 -- 172.21.15.16:0/3227072018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fc5080b0730 con 0x558c121b3390 2024-08-28T19:27:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc525aa0740 10 client.17226 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.997+0000 7fc525aa0740 1 -- 172.21.15.16:0/3227072018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x558c1260a2d0 con 0x558c121b3390 2024-08-28T19:27:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.998+0000 7fc500ff9640 1 -- 172.21.15.16:0/3227072018 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc514004770 con 0x558c121b3390 2024-08-28T19:27:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.998+0000 7fc500ff9640 1 -- 172.21.15.16:0/3227072018 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fc5080b3ab0 con 0x558c121b3390 2024-08-28T19:27:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.998+0000 7fc525aa0740 10 client.17226 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.998+0000 7fc525aa0740 10 client.17226 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:27:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.998+0000 7fc525aa0740 20 client.17226 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.998+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x558c12167da0 0x558c1260ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.998+0000 7fc525aa0740 4 client.17226 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:27:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.998+0000 7fc525aa0740 1 -- 172.21.15.16:0/3227072018 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558c124ee070 con 0x558c12167da0 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.998+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x558c12167da0 0x558c1260ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.999+0000 7fc500ff9640 10 client.17226 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.999+0000 7fc51d2c7640 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x558c12167da0 0x558c1260ca70 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1e40327c1864ae2c server_cookie=4cd48e4f7a4526e3 in_seq=0 out_seq=0 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.999+0000 7fc500ff9640 1 -- 172.21.15.16:0/3227072018 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558c124ee070 con 0x558c12167da0 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.999+0000 7fc500ff9640 10 client.17226 handle_command_reply: tid=0 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.999+0000 7fc525aa0740 1 client.17226 shutdown 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.999+0000 7fc525aa0740 1 -- 172.21.15.16:0/3227072018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 msgr2=0x558c12591d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.999+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c12591d50 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc5080abf00 tx=0x7fc50800ce60 comp rx=0 tx=0).stop 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.999+0000 7fc525aa0740 1 -- 172.21.15.16:0/3227072018 shutdown_connections 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:31.999+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x558c12167da0 0x558c1260ca70 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c121d4070 0x558c12608c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c121b3390 0x558c12591d50 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc525aa0740 1 --2- 172.21.15.16:0/3227072018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558c121ab1b0 0x558c12591810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc500ff9640 0 client.17226 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:27:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc525aa0740 1 -- 172.21.15.16:0/3227072018 >> 172.21.15.16:0/3227072018 conn(0x558c121c8460 msgr2=0x558c1258d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc525aa0740 1 -- 172.21.15.16:0/3227072018 shutdown_connections 2024-08-28T19:27:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc525aa0740 1 -- 172.21.15.16:0/3227072018 wait complete. 2024-08-28T19:27:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc525aa0740 20 client.17226 trim_cache size 0 max 16384 2024-08-28T19:27:32.005 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc51f52b640 1 -- 172.21.15.16:0/2198721217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5040b4ca0 msgr2=0x7fc5040b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc51f52b640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5040b4ca0 0x7fc5040b71b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc514031440 tx=0x7fc514032ed0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc51f52b640 1 -- 172.21.15.16:0/2198721217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc51810b490 msgr2=0x7fc51819ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.000+0000 7fc51f52b640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc51810b490 0x7fc51819ca30 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fc50800e9f0 tx=0x7fc50800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.001+0000 7fc51f52b640 1 -- 172.21.15.16:0/2198721217 shutdown_connections 2024-08-28T19:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.001+0000 7fc51f52b640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc5040b4ca0 0x7fc5040b71b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.001+0000 7fc51f52b640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc51810bde0 0x7fc5181a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.001+0000 7fc51f52b640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc51810b490 0x7fc51819ca30 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.001+0000 7fc51f52b640 1 --2- 172.21.15.16:0/2198721217 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc518105b50 0x7fc51819c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.001+0000 7fc51f52b640 1 -- 172.21.15.16:0/2198721217 >> 172.21.15.16:0/2198721217 conn(0x7fc518101560 msgr2=0x7fc5181049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.001+0000 7fc51f52b640 1 -- 172.21.15.16:0/2198721217 shutdown_connections 2024-08-28T19:27:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:32.001+0000 7fc51f52b640 1 -- 172.21.15.16:0/2198721217 wait complete. 2024-08-28T19:27:32.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:27:32.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 301890 4.71 MB/sec execute 192 sec latency 10.305 ms 2024-08-28T19:27:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:27:33.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:33 smithi016 ceph-mon[18556]: pgmap v534: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-08-28T19:27:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:33 smithi028 ceph-mon[21769]: pgmap v534: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-08-28T19:27:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:33 smithi115 ceph-mon[22184]: pgmap v534: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-08-28T19:27:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.631+0000 7f9537238640 1 -- 172.21.15.16:0/1233365498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9530102880 msgr2=0x7f9530102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.631+0000 7f9537238640 1 --2- 172.21.15.16:0/1233365498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9530102880 0x7f9530102c90 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f9524009900 tx=0x7f952402a410 comp rx=0 tx=0).stop 2024-08-28T19:27:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.631+0000 7f9537238640 1 -- 172.21.15.16:0/1233365498 shutdown_connections 2024-08-28T19:27:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.631+0000 7f9537238640 1 --2- 172.21.15.16:0/1233365498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9530108ac0 0x7f953010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.631+0000 7f9537238640 1 --2- 172.21.15.16:0/1233365498 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9530108170 0x7f9530108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.631+0000 7f9537238640 1 --2- 172.21.15.16:0/1233365498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9530102880 0x7f9530102c90 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.631+0000 7f9537238640 1 -- 172.21.15.16:0/1233365498 >> 172.21.15.16:0/1233365498 conn(0x7f9530079530 msgr2=0x7f9530079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.631+0000 7f9537238640 1 -- 172.21.15.16:0/1233365498 shutdown_connections 2024-08-28T19:27:33.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.631+0000 7f9537238640 1 -- 172.21.15.16:0/1233365498 wait complete. 2024-08-28T19:27:33.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f9537238640 1 Processor -- start 2024-08-28T19:27:33.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f9537238640 1 -- start start 2024-08-28T19:27:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f9537238640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9530102880 0x7f953019c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f9537238640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9530108170 0x7f953019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f9537238640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9530108ac0 0x7f95301a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f9537238640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9530112790 con 0x7f9530108170 2024-08-28T19:27:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f9537238640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9530112610 con 0x7f9530108ac0 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f9537238640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9530112910 con 0x7f9530102880 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f952ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9530108170 0x7f953019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f9534fd4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9530102880 0x7f953019c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f9534fd4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9530102880 0x7f953019c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:43168/0 (socket says 172.21.15.16:43168) 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f9534fd4640 1 -- 172.21.15.16:0/2076392664 learned_addr learned my addr 172.21.15.16:0/2076392664 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.632+0000 7f95357d5640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9530108ac0 0x7f95301a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f95357d5640 1 -- 172.21.15.16:0/2076392664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9530102880 msgr2=0x7f953019c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f95357d5640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9530102880 0x7f953019c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f95357d5640 1 -- 172.21.15.16:0/2076392664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9530108170 msgr2=0x7f953019ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f95357d5640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9530108170 0x7f953019ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f95357d5640 1 -- 172.21.15.16:0/2076392664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95301a2400 con 0x7f9530108ac0 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f952ffff640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9530108170 0x7f953019ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f9534fd4640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9530102880 0x7f953019c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f95357d5640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9530108ac0 0x7f95301a1cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9528009fd0 tx=0x7f952800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f952dffb640 1 -- 172.21.15.16:0/2076392664 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f952800ce20 con 0x7f9530108ac0 2024-08-28T19:27:33.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f9537238640 1 -- 172.21.15.16:0/2076392664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95301a26f0 con 0x7f9530108ac0 2024-08-28T19:27:33.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f952dffb640 1 -- 172.21.15.16:0/2076392664 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9528005020 con 0x7f9530108ac0 2024-08-28T19:27:33.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f952dffb640 1 -- 172.21.15.16:0/2076392664 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528010690 con 0x7f9530108ac0 2024-08-28T19:27:33.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.633+0000 7f9537238640 1 -- 172.21.15.16:0/2076392664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95301a8e50 con 0x7f9530108ac0 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.636+0000 7f952dffb640 1 -- 172.21.15.16:0/2076392664 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f95280054d0 con 0x7f9530108ac0 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.636+0000 7f953d7ad740 1 Processor -- start 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.636+0000 7f953d7ad740 1 -- start start 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.636+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8a0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.636+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4d21b0 0x557a6f8a0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.636+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4da390 0x557a6f8a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f953d7ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557a6f817680 con 0x557a6f4d21b0 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f9534fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4d21b0 0x557a6f8a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f9534fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4d21b0 0x557a6f8a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52598/0 (socket says 172.21.15.16:52598) 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f953d7ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557a6f806c10 con 0x557a6f4da390 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f9534fd4640 1 -- 172.21.15.16:0/2183728583 learned_addr learned my addr 172.21.15.16:0/2183728583 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f952ffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f953d7ad740 1 -- 172.21.15.16:0/2183728583 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557a6f80abc0 con 0x557a6f4fb070 2024-08-28T19:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f952ffff640 1 --2- 172.21.15.16:0/2183728583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4da390 0x557a6f8a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f952dffb640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f95200b4ca0 0x7f95200b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f95357d5640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f95200b4ca0 0x7f95200b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f952ffff640 1 -- 172.21.15.16:0/2183728583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4da390 msgr2=0x557a6f8a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f952ffff640 1 --2- 172.21.15.16:0/2183728583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4da390 0x557a6f8a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f952ffff640 1 -- 172.21.15.16:0/2183728583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4d21b0 msgr2=0x557a6f8a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f952ffff640 1 --2- 172.21.15.16:0/2183728583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4d21b0 0x557a6f8a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f952ffff640 1 -- 172.21.15.16:0/2183728583 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a6f566650 con 0x557a6f4fb070 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f952dffb640 1 -- 172.21.15.16:0/2076392664 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f95280e5fc0 con 0x7f9530108ac0 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.637+0000 7f952ffff640 1 --2- 172.21.15.16:0/2183728583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8a0310 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f951c009fd0 tx=0x7f951c005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1fc2c8cbdb46fa4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.638+0000 7f951affd640 1 -- 172.21.15.16:0/2183728583 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c013130 con 0x557a6f4fb070 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.638+0000 7f95357d5640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f95200b4ca0 0x7f95200b71b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f95280a8360 tx=0x7f9528081040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.638+0000 7f951affd640 1 -- 172.21.15.16:0/2183728583 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f951c0054d0 con 0x557a6f4fb070 2024-08-28T19:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.638+0000 7f951affd640 1 -- 172.21.15.16:0/2183728583 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c0047a0 con 0x557a6f4fb070 2024-08-28T19:27:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.639+0000 7f953d7ad740 1 -- 172.21.15.16:0/2183728583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 msgr2=0x557a6f8a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.639+0000 7f953d7ad740 1 --2- 172.21.15.16:0/2183728583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8a0310 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f951c009fd0 tx=0x7f951c005850 comp rx=0 tx=0).stop 2024-08-28T19:27:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.639+0000 7f953d7ad740 1 -- 172.21.15.16:0/2183728583 shutdown_connections 2024-08-28T19:27:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.639+0000 7f953d7ad740 1 --2- 172.21.15.16:0/2183728583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4da390 0x557a6f8a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.639+0000 7f953d7ad740 1 --2- 172.21.15.16:0/2183728583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4d21b0 0x557a6f8a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.639+0000 7f953d7ad740 1 --2- 172.21.15.16:0/2183728583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8a0310 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.639+0000 7f953d7ad740 1 -- 172.21.15.16:0/2183728583 >> 172.21.15.16:0/2183728583 conn(0x557a6f4ef460 msgr2=0x557a6f89e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.639+0000 7f953d7ad740 1 -- 172.21.15.16:0/2183728583 shutdown_connections 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.639+0000 7f953d7ad740 1 -- 172.21.15.16:0/2183728583 wait complete. 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.639+0000 7f953d7ad740 1 -- start start 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.639+0000 7f953d7ad740 1 Processor -- start 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4d21b0 0x557a6f8aba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4fb070 0x557a6f8b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f953d7ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557a6f669700 con 0x557a6f4da390 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f953d7ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557a6f78d750 con 0x557a6f4fb070 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f953d7ad740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557a6f8aea90 con 0x557a6f4d21b0 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f953d7ad740 10 client.0 _ll_register_callbacks cb 0x557a6f7bce10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f952ffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4d21b0 0x557a6f8aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f952ffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4d21b0 0x557a6f8aba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:43188/0 (socket says 172.21.15.16:43188) 2024-08-28T19:27:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f9534fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f952ffff640 1 -- 172.21.15.16:0/4221402775 learned_addr learned my addr 172.21.15.16:0/4221402775 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f952ffff640 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4fb070 0x557a6f8b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f952ffff640 1 -- 172.21.15.16:0/4221402775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4fb070 msgr2=0x557a6f8b7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f952ffff640 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4fb070 0x557a6f8b7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f952ffff640 1 -- 172.21.15.16:0/4221402775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 msgr2=0x557a6f8abfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f952ffff640 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8abfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f952ffff640 1 -- 172.21.15.16:0/4221402775 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a6f7c6900 con 0x557a6f4d21b0 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.640+0000 7f9534fd4640 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8abfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.641+0000 7f952ffff640 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4d21b0 0x557a6f8aba70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f951c00fd70 tx=0x7f951c018040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea68d866e90f055b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.641+0000 7f950ffff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.641+0000 7f950ffff640 1 -- 172.21.15.16:0/4221402775 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c004460 con 0x557a6f4d21b0 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.641+0000 7f953d7ad740 1 -- 172.21.15.16:0/4221402775 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f951c0054d0 con 0x557a6f4d21b0 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.641+0000 7f950ffff640 1 -- 172.21.15.16:0/4221402775 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f951c0035f0 con 0x557a6f4d21b0 2024-08-28T19:27:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.641+0000 7f950ffff640 1 -- 172.21.15.16:0/4221402775 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f951c032050 con 0x557a6f4d21b0 2024-08-28T19:27:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.641+0000 7f953d7ad740 10 client.26593 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.641+0000 7f953d7ad740 1 -- 172.21.15.16:0/4221402775 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x557a6f8ba840 con 0x557a6f4d21b0 2024-08-28T19:27:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.641+0000 7f950ffff640 1 -- 172.21.15.16:0/4221402775 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f951c0054d0 con 0x557a6f4d21b0 2024-08-28T19:27:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.641+0000 7f950ffff640 1 -- 172.21.15.16:0/4221402775 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f951c0217b0 con 0x557a6f4d21b0 2024-08-28T19:27:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.642+0000 7f953d7ad740 10 client.26593 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.642+0000 7f953d7ad740 10 client.26593 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:27:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.642+0000 7f953d7ad740 20 client.26593 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.642+0000 7f953d7ad740 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557a6f4b99f0 0x557a6f8bdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.642+0000 7f953d7ad740 4 client.26593 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:27:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.642+0000 7f953d7ad740 1 -- 172.21.15.16:0/4221402775 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a6f8195d0 con 0x557a6f4b99f0 2024-08-28T19:27:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.642+0000 7f9534fd4640 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557a6f4b99f0 0x557a6f8bdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.642+0000 7f950ffff640 10 client.26593 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:27:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.642+0000 7f9534fd4640 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557a6f4b99f0 0x557a6f8bdc50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=29a5e938704e01da server_cookie=55bfe5c27c0c5047 in_seq=0 out_seq=0 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f950ffff640 1 -- 172.21.15.16:0/4221402775 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557a6f8195d0 con 0x557a6f4b99f0 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f950ffff640 10 client.26593 handle_command_reply: tid=0 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 1 client.26593 shutdown 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 1 -- 172.21.15.16:0/4221402775 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4d21b0 msgr2=0x557a6f8aba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4d21b0 0x557a6f8aba70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f951c00fd70 tx=0x7f951c018040 comp rx=0 tx=0).stop 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 1 -- 172.21.15.16:0/4221402775 shutdown_connections 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557a6f4b99f0 0x557a6f8bdc50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4fb070 0x557a6f8b7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8abfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 1 --2- 172.21.15.16:0/4221402775 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4d21b0 0x557a6f8aba70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f950ffff640 0 client.26593 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 1 -- 172.21.15.16:0/4221402775 >> 172.21.15.16:0/4221402775 conn(0x557a6f4ef460 msgr2=0x557a6f8a34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 1 -- 172.21.15.16:0/4221402775 shutdown_connections 2024-08-28T19:27:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 1 -- 172.21.15.16:0/4221402775 wait complete. 2024-08-28T19:27:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.643+0000 7f953d7ad740 20 client.26593 trim_cache size 0 max 16384 2024-08-28T19:27:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.658+0000 7f953d7ad740 1 Processor -- start 2024-08-28T19:27:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.658+0000 7f953d7ad740 1 -- start start 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.658+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4d21b0 0x557a6f8aa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.658+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8aa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.658+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8a3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.658+0000 7f953d7ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95301b2d80 con 0x557a6f4da390 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.658+0000 7f953d7ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95301b48c0 con 0x557a6f4d21b0 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.658+0000 7f953d7ad740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f95301b3450 con 0x557a6f4fb070 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.658+0000 7f9534fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.658+0000 7f9534fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8aa830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52608/0 (socket says 172.21.15.16:52608) 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.658+0000 7f9534fd4640 1 -- 172.21.15.16:0/1352152569 learned_addr learned my addr 172.21.15.16:0/1352152569 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f952ffff640 1 --2- 172.21.15.16:0/1352152569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4d21b0 0x557a6f8aa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f9534fd4640 1 -- 172.21.15.16:0/1352152569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 msgr2=0x557a6f8a3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f952ffff640 1 --2- 172.21.15.16:0/1352152569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8a3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f9534fd4640 1 --2- 172.21.15.16:0/1352152569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8a3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f9534fd4640 1 -- 172.21.15.16:0/1352152569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4d21b0 msgr2=0x557a6f8aa2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f9534fd4640 1 --2- 172.21.15.16:0/1352152569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4d21b0 0x557a6f8aa2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f9534fd4640 1 -- 172.21.15.16:0/1352152569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a6f8a3170 con 0x557a6f4da390 2024-08-28T19:27:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f952ffff640 1 --2- 172.21.15.16:0/1352152569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8a3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:27:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f9534fd4640 1 --2- 172.21.15.16:0/1352152569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8aa830 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f9524005e00 tx=0x7f95240329b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb0b1e6361cbf36 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f951a7fc640 1 -- 172.21.15.16:0/1352152569 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9524035040 con 0x557a6f4da390 2024-08-28T19:27:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f951a7fc640 1 -- 172.21.15.16:0/1352152569 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95240054f0 con 0x557a6f4da390 2024-08-28T19:27:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.659+0000 7f951a7fc640 1 -- 172.21.15.16:0/1352152569 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9524004c00 con 0x557a6f4da390 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.660+0000 7f953d7ad740 1 -- 172.21.15.16:0/1352152569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 msgr2=0x557a6f8aa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.660+0000 7f953d7ad740 1 --2- 172.21.15.16:0/1352152569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8aa830 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f9524005e00 tx=0x7f95240329b0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.660+0000 7f953d7ad740 1 -- 172.21.15.16:0/1352152569 shutdown_connections 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.660+0000 7f953d7ad740 1 --2- 172.21.15.16:0/1352152569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8a3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.660+0000 7f953d7ad740 1 --2- 172.21.15.16:0/1352152569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8aa830 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.660+0000 7f953d7ad740 1 --2- 172.21.15.16:0/1352152569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4d21b0 0x557a6f8aa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.660+0000 7f953d7ad740 1 -- 172.21.15.16:0/1352152569 >> 172.21.15.16:0/1352152569 conn(0x557a6f4ef460 msgr2=0x557a6f8a6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.660+0000 7f953d7ad740 1 -- 172.21.15.16:0/1352152569 shutdown_connections 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.660+0000 7f953d7ad740 1 -- 172.21.15.16:0/1352152569 wait complete. 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f953d7ad740 1 -- start start 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f953d7ad740 1 Processor -- start 2024-08-28T19:27:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4d21b0 0x557a6f8af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8afd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f953d7ad740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f953d7ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95301b41f0 con 0x557a6f4da390 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f953d7ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95301b3b20 con 0x557a6f4d21b0 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f953d7ad740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f95301b0d60 con 0x557a6f4fb070 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f953d7ad740 10 client.0 _ll_register_callbacks cb 0x7f951c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f9534fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f9534fd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8afd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52614/0 (socket says 172.21.15.16:52614) 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f952ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4d21b0 0x557a6f8af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f9534fd4640 1 -- 172.21.15.16:0/2672389549 learned_addr learned my addr 172.21.15.16:0/2672389549 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.661+0000 7f952ffff640 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f9534fd4640 1 -- 172.21.15.16:0/2672389549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 msgr2=0x557a6f8b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f9534fd4640 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f9534fd4640 1 -- 172.21.15.16:0/2672389549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4d21b0 msgr2=0x557a6f8af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f9534fd4640 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4d21b0 0x557a6f8af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f9534fd4640 1 -- 172.21.15.16:0/2672389549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a6f8b42f0 con 0x557a6f4da390 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f952ffff640 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4d21b0 0x557a6f8af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f9534fd4640 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8afd90 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f9524009230 tx=0x7f95240314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beaeb00f8299a3c3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f950ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f950ffff640 1 -- 172.21.15.16:0/2672389549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9524057020 con 0x557a6f4da390 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f950ffff640 1 -- 172.21.15.16:0/2672389549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f9524004a80 con 0x557a6f4da390 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f953d7ad740 1 -- 172.21.15.16:0/2672389549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95240054f0 con 0x557a6f4da390 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f950ffff640 1 -- 172.21.15.16:0/2672389549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f9524035070 con 0x557a6f4da390 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f953d7ad740 10 client.17244 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.662+0000 7f953d7ad740 1 -- 172.21.15.16:0/2672389549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x557a6f927f90 con 0x557a6f4da390 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.663+0000 7f950ffff640 1 -- 172.21.15.16:0/2672389549 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95240054f0 con 0x557a6f4da390 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.663+0000 7f950ffff640 1 -- 172.21.15.16:0/2672389549 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f95240316c0 con 0x557a6f4da390 2024-08-28T19:27:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.663+0000 7f953d7ad740 10 client.17244 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:33.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.663+0000 7f953d7ad740 10 client.17244 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:27:33.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.663+0000 7f953d7ad740 20 client.17244 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:33.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.663+0000 7f953d7ad740 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557a6f48eda0 0x557a6f92a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:33.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.663+0000 7f953d7ad740 4 client.17244 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:27:33.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.663+0000 7f953d7ad740 1 -- 172.21.15.16:0/2672389549 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557a6f815070 con 0x557a6f48eda0 2024-08-28T19:27:33.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.663+0000 7f952ffff640 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557a6f48eda0 0x557a6f92a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:33.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.663+0000 7f950ffff640 10 client.17244 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:27:33.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f952ffff640 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557a6f48eda0 0x557a6f92a6e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a994832b040fa691 server_cookie=4e4c4207a7d64394 in_seq=0 out_seq=0 2024-08-28T19:27:33.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f950ffff640 1 -- 172.21.15.16:0/2672389549 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557a6f815070 con 0x557a6f48eda0 2024-08-28T19:27:33.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f950ffff640 10 client.17244 handle_command_reply: tid=0 2024-08-28T19:27:33.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 1 client.17244 shutdown 2024-08-28T19:27:33.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 1 -- 172.21.15.16:0/2672389549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 msgr2=0x557a6f8afd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:33.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8afd90 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f9524009230 tx=0x7f95240314a0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 1 -- 172.21.15.16:0/2672389549 shutdown_connections 2024-08-28T19:27:33.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557a6f48eda0 0x557a6f92a6e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f950ffff640 0 client.17244 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:27:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a6f4fb070 0x557a6f8b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a6f4da390 0x557a6f8afd90 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 1 --2- 172.21.15.16:0/2672389549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a6f4d21b0 0x557a6f8af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 1 -- 172.21.15.16:0/2672389549 >> 172.21.15.16:0/2672389549 conn(0x557a6f4ef460 msgr2=0x557a6f8a3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 1 -- 172.21.15.16:0/2672389549 shutdown_connections 2024-08-28T19:27:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 1 -- 172.21.15.16:0/2672389549 wait complete. 2024-08-28T19:27:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.664+0000 7f953d7ad740 20 client.17244 trim_cache size 0 max 16384 2024-08-28T19:27:33.670 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:27:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 -- 172.21.15.16:0/2076392664 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f95200b4ca0 msgr2=0x7f95200b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f95200b4ca0 0x7f95200b71b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f95280a8360 tx=0x7f9528081040 comp rx=0 tx=0).stop 2024-08-28T19:27:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 -- 172.21.15.16:0/2076392664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9530108ac0 msgr2=0x7f95301a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9530108ac0 0x7f95301a1cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9528009fd0 tx=0x7f952800ef10 comp rx=0 tx=0).stop 2024-08-28T19:27:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 -- 172.21.15.16:0/2076392664 shutdown_connections 2024-08-28T19:27:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f95200b4ca0 0x7f95200b71b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9530108ac0 0x7f95301a1cd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9530108170 0x7f953019ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 --2- 172.21.15.16:0/2076392664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9530102880 0x7f953019c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 -- 172.21.15.16:0/2076392664 >> 172.21.15.16:0/2076392664 conn(0x7f9530079530 msgr2=0x7f9530106210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 -- 172.21.15.16:0/2076392664 shutdown_connections 2024-08-28T19:27:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:33.665+0000 7f9537238640 1 -- 172.21.15.16:0/2076392664 wait complete. 2024-08-28T19:27:33.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 303052 4.72 MB/sec execute 193 sec latency 46.862 ms 2024-08-28T19:27:34.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:27:34.747 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:27:34.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 305155 4.76 MB/sec execute 194 sec latency 15.021 ms 2024-08-28T19:27:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.229+0000 7f257c0dd640 1 -- 172.21.15.16:0/1118057985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2574109280 msgr2=0x7f2574109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.229+0000 7f257c0dd640 1 --2- 172.21.15.16:0/1118057985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2574109280 0x7f2574109690 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2564009900 tx=0x7f256402a410 comp rx=0 tx=0).stop 2024-08-28T19:27:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.229+0000 7f257c0dd640 1 -- 172.21.15.16:0/1118057985 shutdown_connections 2024-08-28T19:27:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.229+0000 7f257c0dd640 1 --2- 172.21.15.16:0/1118057985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2574109bd0 0x7f2574110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.229+0000 7f257c0dd640 1 --2- 172.21.15.16:0/1118057985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2574109280 0x7f2574109690 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.229+0000 7f257c0dd640 1 --2- 172.21.15.16:0/1118057985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574103940 0x7f2574103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.229+0000 7f257c0dd640 1 -- 172.21.15.16:0/1118057985 >> 172.21.15.16:0/1118057985 conn(0x7f25740ff3d0 msgr2=0x7f2574101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.229+0000 7f257c0dd640 1 -- 172.21.15.16:0/1118057985 shutdown_connections 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.229+0000 7f257c0dd640 1 -- 172.21.15.16:0/1118057985 wait complete. 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.230+0000 7f257c0dd640 1 Processor -- start 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.230+0000 7f257c0dd640 1 -- start start 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.230+0000 7f257c0dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2574103940 0x7f25740754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.230+0000 7f257c0dd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574109280 0x7f25740759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.230+0000 7f257c0dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2574109bd0 0x7f2574075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.230+0000 7f257c0dd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2574113890 con 0x7f2574109bd0 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.230+0000 7f257c0dd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2574113710 con 0x7f2574103940 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.230+0000 7f257c0dd640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2574113a10 con 0x7f2574109280 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f2579e79640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2574103940 0x7f25740754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f257a67a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2574109bd0 0x7f2574075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f2579e79640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2574103940 0x7f25740754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60262/0 (socket says 172.21.15.16:60262) 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f2579e79640 1 -- 172.21.15.16:0/880855106 learned_addr learned my addr 172.21.15.16:0/880855106 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f2579678640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574109280 0x7f25740759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f2579e79640 1 -- 172.21.15.16:0/880855106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574109280 msgr2=0x7f25740759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f2579e79640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574109280 0x7f25740759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f2579e79640 1 -- 172.21.15.16:0/880855106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2574109bd0 msgr2=0x7f2574075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f2579e79640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2574109bd0 0x7f2574075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f2579e79640 1 -- 172.21.15.16:0/880855106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25741a8520 con 0x7f2574103940 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f2579e79640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2574103940 0x7f25740754b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f257000e9f0 tx=0x7f257000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f256affd640 1 -- 172.21.15.16:0/880855106 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257000cd80 con 0x7f2574103940 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f257c0dd640 1 -- 172.21.15.16:0/880855106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25741a8810 con 0x7f2574103940 2024-08-28T19:27:35.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f256affd640 1 -- 172.21.15.16:0/880855106 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25700054d0 con 0x7f2574103940 2024-08-28T19:27:35.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f256affd640 1 -- 172.21.15.16:0/880855106 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2570010690 con 0x7f2574103940 2024-08-28T19:27:35.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.231+0000 7f257c0dd640 1 -- 172.21.15.16:0/880855106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25741a8cd0 con 0x7f2574103940 2024-08-28T19:27:35.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2582652740 1 Processor -- start 2024-08-28T19:27:35.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2582652740 1 -- start start 2024-08-28T19:27:35.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f256affd640 1 -- 172.21.15.16:0/880855106 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2570004e90 con 0x7f2574103940 2024-08-28T19:27:35.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2582652740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be67070 0x557d0c20c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2582652740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be3e1b0 0x557d0c20c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2582652740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be46390 0x557d0c20cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2582652740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557d0c183680 con 0x557d0be3e1b0 2024-08-28T19:27:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2582652740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d0c172c10 con 0x557d0be46390 2024-08-28T19:27:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2582652740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557d0c176bc0 con 0x557d0be67070 2024-08-28T19:27:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2579e79640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be3e1b0 0x557d0c20c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f257a67a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be67070 0x557d0c20c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2579e79640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be3e1b0 0x557d0c20c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39214/0 (socket says 172.21.15.16:39214) 2024-08-28T19:27:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2579e79640 1 -- 172.21.15.16:0/599586261 learned_addr learned my addr 172.21.15.16:0/599586261 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.234+0000 7f2579678640 1 --2- 172.21.15.16:0/599586261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be46390 0x557d0c20cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.235+0000 7f257a67a640 1 -- 172.21.15.16:0/599586261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be46390 msgr2=0x557d0c20cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.235+0000 7f257a67a640 1 --2- 172.21.15.16:0/599586261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be46390 0x557d0c20cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.235+0000 7f257a67a640 1 -- 172.21.15.16:0/599586261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be3e1b0 msgr2=0x557d0c20c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.235+0000 7f257a67a640 1 --2- 172.21.15.16:0/599586261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be3e1b0 0x557d0c20c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.235+0000 7f257a67a640 1 -- 172.21.15.16:0/599586261 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d0bed2650 con 0x557d0be67070 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.235+0000 7f257a67a640 1 --2- 172.21.15.16:0/599586261 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be67070 0x557d0c20c310 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f256c009f00 tx=0x7f256c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b10cddde4343c09e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.235+0000 7f255ffff640 1 -- 172.21.15.16:0/599586261 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c004d20 con 0x557d0be67070 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.235+0000 7f255ffff640 1 -- 172.21.15.16:0/599586261 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f256c004030 con 0x557d0be67070 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.235+0000 7f256affd640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25600b4ca0 0x7f25600b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.235+0000 7f255ffff640 1 -- 172.21.15.16:0/599586261 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c005280 con 0x557d0be67070 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.235+0000 7f2579e79640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25600b4ca0 0x7f25600b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.236+0000 7f256affd640 1 -- 172.21.15.16:0/880855106 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f25700e5d00 con 0x7f2574103940 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.236+0000 7f2582652740 1 -- 172.21.15.16:0/599586261 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be67070 msgr2=0x557d0c20c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.236+0000 7f2579e79640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25600b4ca0 0x7f25600b71b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f25700a8100 tx=0x7f25700e9840 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.236+0000 7f2582652740 1 --2- 172.21.15.16:0/599586261 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be67070 0x557d0c20c310 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f256c009f00 tx=0x7f256c011810 comp rx=0 tx=0).stop 2024-08-28T19:27:35.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.236+0000 7f2582652740 1 -- 172.21.15.16:0/599586261 shutdown_connections 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.236+0000 7f2582652740 1 --2- 172.21.15.16:0/599586261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be46390 0x557d0c20cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.236+0000 7f2582652740 1 --2- 172.21.15.16:0/599586261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be3e1b0 0x557d0c20c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.236+0000 7f2582652740 1 --2- 172.21.15.16:0/599586261 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be67070 0x557d0c20c310 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.236+0000 7f2582652740 1 -- 172.21.15.16:0/599586261 >> 172.21.15.16:0/599586261 conn(0x557d0be5b460 msgr2=0x557d0c20a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.236+0000 7f2582652740 1 -- 172.21.15.16:0/599586261 shutdown_connections 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.236+0000 7f2582652740 1 -- 172.21.15.16:0/599586261 wait complete. 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.237+0000 7f2582652740 1 -- start start 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.237+0000 7f2582652740 1 Processor -- start 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.237+0000 7f2582652740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 0x557d0c217b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.237+0000 7f2582652740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be46390 0x557d0c218040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.237+0000 7f2582652740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be67070 0x557d0c223e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.237+0000 7f2582652740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557d0bfd5700 con 0x557d0be67070 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.237+0000 7f2582652740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d0c0f9750 con 0x557d0be3e1b0 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.237+0000 7f2582652740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557d0c21aaf0 con 0x557d0be46390 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.237+0000 7f2582652740 10 client.0 _ll_register_callbacks cb 0x557d0c128e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f257a67a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 0x557d0c217b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f257a67a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 0x557d0c217b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60274/0 (socket says 172.21.15.16:60274) 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f257a67a640 1 -- 172.21.15.16:0/2370287438 learned_addr learned my addr 172.21.15.16:0/2370287438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f257a67a640 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be67070 0x557d0c223e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f2579678640 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be46390 0x557d0c218040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f257a67a640 1 -- 172.21.15.16:0/2370287438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be46390 msgr2=0x557d0c218040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f257a67a640 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be46390 0x557d0c218040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f257a67a640 1 -- 172.21.15.16:0/2370287438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be67070 msgr2=0x557d0c223e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f257a67a640 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be67070 0x557d0c223e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f257a67a640 1 -- 172.21.15.16:0/2370287438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d0c132900 con 0x557d0be3e1b0 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f257a67a640 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 0x557d0c217b00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f256c009370 tx=0x7f256c003f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e05f2ca20f5d851d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f255d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f255d7fa640 1 -- 172.21.15.16:0/2370287438 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c02d060 con 0x557d0be3e1b0 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f255d7fa640 1 -- 172.21.15.16:0/2370287438 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f256c004a40 con 0x557d0be3e1b0 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f2582652740 1 -- 172.21.15.16:0/2370287438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f256c004030 con 0x557d0be3e1b0 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f255d7fa640 1 -- 172.21.15.16:0/2370287438 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f256c034050 con 0x557d0be3e1b0 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f2582652740 10 client.26801 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.238+0000 7f2582652740 1 -- 172.21.15.16:0/2370287438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x557d0c226900 con 0x557d0be3e1b0 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.239+0000 7f255d7fa640 1 -- 172.21.15.16:0/2370287438 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f256c004030 con 0x557d0be3e1b0 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.239+0000 7f255d7fa640 1 -- 172.21.15.16:0/2370287438 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f256c024070 con 0x557d0be3e1b0 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.239+0000 7f2582652740 10 client.26801 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.239+0000 7f2582652740 10 client.26801 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.239+0000 7f2582652740 20 client.26801 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.239+0000 7f2582652740 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d0be259f0 0x557d0c229dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.239+0000 7f2582652740 4 client.26801 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.239+0000 7f2582652740 1 -- 172.21.15.16:0/2370287438 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d0c1855d0 con 0x557d0be259f0 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.239+0000 7f2579678640 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d0be259f0 0x557d0c229dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f2579678640 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d0be259f0 0x557d0c229dd0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e3d6e7f0d4e4490 server_cookie=f5456b8dec1e8ec5 in_seq=0 out_seq=0 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f255d7fa640 10 client.26801 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f255d7fa640 1 -- 172.21.15.16:0/2370287438 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557d0c1855d0 con 0x557d0be259f0 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f255d7fa640 10 client.26801 handle_command_reply: tid=0 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f2582652740 1 client.26801 shutdown 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f2582652740 1 -- 172.21.15.16:0/2370287438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 msgr2=0x557d0c217b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f2582652740 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 0x557d0c217b00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f256c009370 tx=0x7f256c003f60 comp rx=0 tx=0).stop 2024-08-28T19:27:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f2582652740 1 -- 172.21.15.16:0/2370287438 shutdown_connections 2024-08-28T19:27:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f2582652740 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d0be259f0 0x557d0c229dd0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f2582652740 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be67070 0x557d0c223e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f2582652740 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be46390 0x557d0c218040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f2582652740 1 --2- 172.21.15.16:0/2370287438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 0x557d0c217b00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f255d7fa640 0 client.26801 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:27:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.240+0000 7f2582652740 1 -- 172.21.15.16:0/2370287438 >> 172.21.15.16:0/2370287438 conn(0x557d0be5b460 msgr2=0x557d0c20f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.241+0000 7f2582652740 1 -- 172.21.15.16:0/2370287438 shutdown_connections 2024-08-28T19:27:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.241+0000 7f2582652740 1 -- 172.21.15.16:0/2370287438 wait complete. 2024-08-28T19:27:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.241+0000 7f2582652740 20 client.26801 trim_cache size 0 max 16384 2024-08-28T19:27:35.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.254+0000 7f2582652740 1 Processor -- start 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2582652740 1 -- start start 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2582652740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be3e1b0 0x557d0c2199e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2582652740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 0x557d0c20da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2582652740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be67070 0x557d0c223d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2582652740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25741b2ce0 con 0x557d0be46390 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2582652740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25741b4820 con 0x557d0be67070 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2582652740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f25741b33b0 con 0x557d0be3e1b0 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f257a67a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be3e1b0 0x557d0c2199e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2579678640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 0x557d0c20da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f257a67a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be3e1b0 0x557d0c2199e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:38456/0 (socket says 172.21.15.16:38456) 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f257a67a640 1 -- 172.21.15.16:0/2138018815 learned_addr learned my addr 172.21.15.16:0/2138018815 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f257a67a640 1 --2- 172.21.15.16:0/2138018815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be67070 0x557d0c223d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2579678640 1 -- 172.21.15.16:0/2138018815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be3e1b0 msgr2=0x557d0c2199e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:35.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2579678640 1 --2- 172.21.15.16:0/2138018815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be3e1b0 0x557d0c2199e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2579678640 1 -- 172.21.15.16:0/2138018815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be67070 msgr2=0x557d0c223d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:35.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2579678640 1 --2- 172.21.15.16:0/2138018815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be67070 0x557d0c223d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f2579678640 1 -- 172.21.15.16:0/2138018815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d0c218dc0 con 0x557d0be46390 2024-08-28T19:27:35.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.255+0000 7f257a67a640 1 --2- 172.21.15.16:0/2138018815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be3e1b0 0x557d0c2199e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:35.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.256+0000 7f2579678640 1 --2- 172.21.15.16:0/2138018815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 0x557d0c20da50 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f256402a930 tx=0x7f2564031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e58733f288b400c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:35.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.256+0000 7f255f7fe640 1 -- 172.21.15.16:0/2138018815 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564035040 con 0x557d0be46390 2024-08-28T19:27:35.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.256+0000 7f255f7fe640 1 -- 172.21.15.16:0/2138018815 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25640054f0 con 0x557d0be46390 2024-08-28T19:27:35.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.256+0000 7f255f7fe640 1 -- 172.21.15.16:0/2138018815 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564004c80 con 0x557d0be46390 2024-08-28T19:27:35.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.257+0000 7f2582652740 1 -- 172.21.15.16:0/2138018815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 msgr2=0x557d0c20da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:35.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.257+0000 7f2582652740 1 --2- 172.21.15.16:0/2138018815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 0x557d0c20da50 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f256402a930 tx=0x7f2564031400 comp rx=0 tx=0).stop 2024-08-28T19:27:35.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.257+0000 7f2582652740 1 -- 172.21.15.16:0/2138018815 shutdown_connections 2024-08-28T19:27:35.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.257+0000 7f2582652740 1 --2- 172.21.15.16:0/2138018815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be67070 0x557d0c223d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.257+0000 7f2582652740 1 --2- 172.21.15.16:0/2138018815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 0x557d0c20da50 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.257+0000 7f2582652740 1 --2- 172.21.15.16:0/2138018815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be3e1b0 0x557d0c2199e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.257+0000 7f2582652740 1 -- 172.21.15.16:0/2138018815 >> 172.21.15.16:0/2138018815 conn(0x557d0be5b460 msgr2=0x557d0c20d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:35.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.257+0000 7f2582652740 1 -- 172.21.15.16:0/2138018815 shutdown_connections 2024-08-28T19:27:35.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.257+0000 7f2582652740 1 -- 172.21.15.16:0/2138018815 wait complete. 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.257+0000 7f2582652740 1 -- start start 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.257+0000 7f2582652740 1 Processor -- start 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2582652740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 0x557d0c220540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2582652740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 0x557d0c21da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2582652740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be67070 0x557d0c21df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2582652740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25741b4150 con 0x557d0be46390 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2582652740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25741b3a80 con 0x557d0be3e1b0 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2582652740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f25741b0cc0 con 0x557d0be67070 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2582652740 10 client.0 _ll_register_callbacks cb 0x7f25700e9710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2579678640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 0x557d0c21da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f257a67a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 0x557d0c220540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2579678640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 0x557d0c21da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39230/0 (socket says 172.21.15.16:39230) 2024-08-28T19:27:35.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2579678640 1 -- 172.21.15.16:0/2696078958 learned_addr learned my addr 172.21.15.16:0/2696078958 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:35.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f257a67a640 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be67070 0x557d0c21df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2579678640 1 -- 172.21.15.16:0/2696078958 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be67070 msgr2=0x557d0c21df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:35.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2579678640 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be67070 0x557d0c21df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2579678640 1 -- 172.21.15.16:0/2696078958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 msgr2=0x557d0c220540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:35.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2579678640 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 0x557d0c220540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2579678640 1 -- 172.21.15.16:0/2696078958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d0c21e6c0 con 0x557d0be46390 2024-08-28T19:27:35.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f2579678640 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 0x557d0c21da10 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2564009230 tx=0x7f2564005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e45e8c466852eb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:35.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.258+0000 7f255d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:27:35.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f255d7fa640 1 -- 172.21.15.16:0/2696078958 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564054070 con 0x557d0be46390 2024-08-28T19:27:35.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f255d7fa640 1 -- 172.21.15.16:0/2696078958 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f25640035f0 con 0x557d0be46390 2024-08-28T19:27:35.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f2582652740 1 -- 172.21.15.16:0/2696078958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25640054f0 con 0x557d0be46390 2024-08-28T19:27:35.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f255d7fa640 1 -- 172.21.15.16:0/2696078958 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f2564060050 con 0x557d0be46390 2024-08-28T19:27:35.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f2582652740 10 client.17262 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:35.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f2582652740 1 -- 172.21.15.16:0/2696078958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x557d0c294530 con 0x557d0be46390 2024-08-28T19:27:35.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f255d7fa640 1 -- 172.21.15.16:0/2696078958 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25640054f0 con 0x557d0be46390 2024-08-28T19:27:35.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f255d7fa640 1 -- 172.21.15.16:0/2696078958 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f2564032500 con 0x557d0be46390 2024-08-28T19:27:35.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f2582652740 10 client.17262 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:35.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f2582652740 10 client.17262 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:27:35.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f2582652740 20 client.17262 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:35.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f2582652740 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d0bdfada0 0x557d0c296be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:35.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f2582652740 4 client.17262 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:27:35.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f2582652740 1 -- 172.21.15.16:0/2696078958 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557d0c181070 con 0x557d0bdfada0 2024-08-28T19:27:35.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.259+0000 7f257a67a640 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d0bdfada0 0x557d0c296be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:35.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f255d7fa640 10 client.17262 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:27:35.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f257a67a640 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d0bdfada0 0x557d0c296be0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d7bae076fced69f server_cookie=bbe3cf26e2dda431 in_seq=0 out_seq=0 2024-08-28T19:27:35.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f255d7fa640 1 -- 172.21.15.16:0/2696078958 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557d0c181070 con 0x557d0bdfada0 2024-08-28T19:27:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f255d7fa640 10 client.17262 handle_command_reply: tid=0 2024-08-28T19:27:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f2582652740 1 client.17262 shutdown 2024-08-28T19:27:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f2582652740 1 -- 172.21.15.16:0/2696078958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 msgr2=0x557d0c21da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f2582652740 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 0x557d0c21da10 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2564009230 tx=0x7f2564005720 comp rx=0 tx=0).stop 2024-08-28T19:27:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f2582652740 1 -- 172.21.15.16:0/2696078958 shutdown_connections 2024-08-28T19:27:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f2582652740 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d0bdfada0 0x557d0c296be0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f2582652740 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d0be67070 0x557d0c21df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f2582652740 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d0be46390 0x557d0c21da10 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f255d7fa640 0 client.17262 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:27:35.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f2582652740 1 --2- 172.21.15.16:0/2696078958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d0be3e1b0 0x557d0c220540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f2582652740 1 -- 172.21.15.16:0/2696078958 >> 172.21.15.16:0/2696078958 conn(0x557d0be5b460 msgr2=0x557d0c219720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:35.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.260+0000 7f2582652740 1 -- 172.21.15.16:0/2696078958 shutdown_connections 2024-08-28T19:27:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f2582652740 1 -- 172.21.15.16:0/2696078958 wait complete. 2024-08-28T19:27:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f2582652740 20 client.17262 trim_cache size 0 max 16384 2024-08-28T19:27:35.272 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:27:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 -- 172.21.15.16:0/880855106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25600b4ca0 msgr2=0x7f25600b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25600b4ca0 0x7f25600b71b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f25700a8100 tx=0x7f25700e9840 comp rx=0 tx=0).stop 2024-08-28T19:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 -- 172.21.15.16:0/880855106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2574103940 msgr2=0x7f25740754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2574103940 0x7f25740754b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f257000e9f0 tx=0x7f257000edf0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 -- 172.21.15.16:0/880855106 shutdown_connections 2024-08-28T19:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2574109bd0 0x7f2574075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574109280 0x7f25740759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f25600b4ca0 0x7f25600b71b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 --2- 172.21.15.16:0/880855106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2574103940 0x7f25740754b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 -- 172.21.15.16:0/880855106 >> 172.21.15.16:0/880855106 conn(0x7f25740ff3d0 msgr2=0x7f25741017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:35.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 -- 172.21.15.16:0/880855106 shutdown_connections 2024-08-28T19:27:35.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:35.261+0000 7f257c0dd640 1 -- 172.21.15.16:0/880855106 wait complete. 2024-08-28T19:27:35.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:35 smithi016 ceph-mon[18556]: pgmap v535: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 888 KiB/s wr, 129 op/s 2024-08-28T19:27:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:35 smithi028 ceph-mon[21769]: pgmap v535: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 888 KiB/s wr, 129 op/s 2024-08-28T19:27:35.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:35 smithi115 ceph-mon[22184]: pgmap v535: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 888 KiB/s wr, 129 op/s 2024-08-28T19:27:35.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 306134 4.75 MB/sec execute 195 sec latency 38.695 ms 2024-08-28T19:27:35.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:27:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:27:36.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.748+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3235742236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb4109270 msgr2=0x7f1bb4109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:36.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.748+0000 7f1bbc6b9640 1 --2- 172.21.15.16:0/3235742236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb4109270 0x7f1bb4109680 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009900 tx=0x7f1ba402a410 comp rx=0 tx=0).stop 2024-08-28T19:27:36.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.748+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3235742236 shutdown_connections 2024-08-28T19:27:36.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.748+0000 7f1bbc6b9640 1 --2- 172.21.15.16:0/3235742236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb4109bc0 0x7f1bb41103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.748+0000 7f1bbc6b9640 1 --2- 172.21.15.16:0/3235742236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb4109270 0x7f1bb4109680 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.748+0000 7f1bbc6b9640 1 --2- 172.21.15.16:0/3235742236 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1bb4103930 0x7f1bb4103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.748+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3235742236 >> 172.21.15.16:0/3235742236 conn(0x7f1bb40ff3a0 msgr2=0x7f1bb41017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.748+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3235742236 shutdown_connections 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.748+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3235742236 wait complete. 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bbc6b9640 1 Processor -- start 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bbc6b9640 1 -- start start 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bbc6b9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb4103930 0x7f1bb40754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bbc6b9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1bb4109270 0x7f1bb40759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bbc6b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb4109bc0 0x7f1bb4075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bbc6b9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41138d0 con 0x7f1bb4109bc0 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bbc6b9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4113750 con 0x7f1bb4103930 2024-08-28T19:27:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bbc6b9640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4113a50 con 0x7f1bb4109270 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bba455640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb4103930 0x7f1bb40754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bba455640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb4103930 0x7f1bb40754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60312/0 (socket says 172.21.15.16:60312) 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bba455640 1 -- 172.21.15.16:0/3089750974 learned_addr learned my addr 172.21.15.16:0/3089750974 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bb9c54640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1bb4109270 0x7f1bb40759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.749+0000 7f1bbac56640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb4109bc0 0x7f1bb4075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.750+0000 7f1bba455640 1 -- 172.21.15.16:0/3089750974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1bb4109270 msgr2=0x7f1bb40759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.750+0000 7f1bba455640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1bb4109270 0x7f1bb40759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.750+0000 7f1bba455640 1 -- 172.21.15.16:0/3089750974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb4109bc0 msgr2=0x7f1bb4075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.750+0000 7f1bba455640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb4109bc0 0x7f1bb4075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.750+0000 7f1bba455640 1 -- 172.21.15.16:0/3089750974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb41a8500 con 0x7f1bb4103930 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.750+0000 7f1bba455640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb4103930 0x7f1bb40754b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000e9f0 tx=0x7f1bb000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.750+0000 7f1bab7fe640 1 -- 172.21.15.16:0/3089750974 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000cd80 con 0x7f1bb4103930 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.750+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3089750974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb41a87f0 con 0x7f1bb4103930 2024-08-28T19:27:36.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.750+0000 7f1bab7fe640 1 -- 172.21.15.16:0/3089750974 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bb00054d0 con 0x7f1bb4103930 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.750+0000 7f1bab7fe640 1 -- 172.21.15.16:0/3089750974 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0010690 con 0x7f1bb4103930 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.750+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3089750974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb41a8cb0 con 0x7f1bb4103930 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bc2c2e740 1 Processor -- start 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bc2c2e740 1 -- start start 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec89070 0x555aff02e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 0x555aff02e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bab7fe640 1 -- 172.21.15.16:0/3089750974 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1bb0004e90 con 0x7f1bb4103930 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec68390 0x555aff02edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555afefa5680 con 0x555afec601b0 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555afef94c10 con 0x555afec89070 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555afef98bc0 con 0x555afec68390 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bbac56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec89070 0x555aff02e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bbac56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec89070 0x555aff02e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60318/0 (socket says 172.21.15.16:60318) 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bbac56640 1 -- 172.21.15.16:0/1419497853 learned_addr learned my addr 172.21.15.16:0/1419497853 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bba455640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 0x555aff02e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.753+0000 7f1bb9c54640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec68390 0x555aff02edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1bbac56640 1 -- 172.21.15.16:0/1419497853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec68390 msgr2=0x555aff02edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1bbac56640 1 --2- 172.21.15.16:0/1419497853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec68390 0x555aff02edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1bbac56640 1 -- 172.21.15.16:0/1419497853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 msgr2=0x555aff02e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1bbac56640 1 --2- 172.21.15.16:0/1419497853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 0x555aff02e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1bbac56640 1 -- 172.21.15.16:0/1419497853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555afecf4650 con 0x555afec89070 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1bb9c54640 1 --2- 172.21.15.16:0/1419497853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec68390 0x555aff02edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1bba455640 1 --2- 172.21.15.16:0/1419497853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 0x555aff02e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1bbac56640 1 --2- 172.21.15.16:0/1419497853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec89070 0x555aff02e350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009fd0 tx=0x7f1bac0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a027c596a95a99 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1bab7fe640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ba00b4ca0 0x7f1ba00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1ba8ff9640 1 -- 172.21.15.16:0/1419497853 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac004d20 con 0x555afec89070 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1ba8ff9640 1 -- 172.21.15.16:0/1419497853 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bac004050 con 0x555afec89070 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1ba8ff9640 1 -- 172.21.15.16:0/1419497853 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac005220 con 0x555afec89070 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1bba455640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ba00b4ca0 0x7f1ba00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.754+0000 7f1bab7fe640 1 -- 172.21.15.16:0/3089750974 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1bb00e5d00 con 0x7f1bb4103930 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.755+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/1419497853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec89070 msgr2=0x555aff02e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.755+0000 7f1bba455640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ba00b4ca0 0x7f1ba00b71b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00e3190 tx=0x7f1bb0004a90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.755+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/1419497853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec89070 0x555aff02e350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009fd0 tx=0x7f1bac0118d0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.755+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/1419497853 shutdown_connections 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.755+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/1419497853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec68390 0x555aff02edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.755+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/1419497853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 0x555aff02e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.755+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/1419497853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec89070 0x555aff02e350 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.755+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/1419497853 >> 172.21.15.16:0/1419497853 conn(0x555afec7d460 msgr2=0x555aff02c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.755+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/1419497853 shutdown_connections 2024-08-28T19:27:36.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.755+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/1419497853 wait complete. 2024-08-28T19:27:36.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bc2c2e740 1 -- start start 2024-08-28T19:27:36.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bc2c2e740 1 Processor -- start 2024-08-28T19:27:36.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 0x555aff039ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec68390 0x555aff03a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec89070 0x555aff045e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555afedf7700 con 0x555afec601b0 2024-08-28T19:27:36.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555afef1b750 con 0x555afec68390 2024-08-28T19:27:36.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555aff03cb30 con 0x555afec89070 2024-08-28T19:27:36.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bc2c2e740 10 client.0 _ll_register_callbacks cb 0x555afef4ae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:36.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bbac56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 0x555aff039ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bb9c54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec68390 0x555aff03a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bb9c54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec68390 0x555aff03a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60320/0 (socket says 172.21.15.16:60320) 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bbac56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 0x555aff039ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39258/0 (socket says 172.21.15.16:39258) 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bb9c54640 1 -- 172.21.15.16:0/4135935023 learned_addr learned my addr 172.21.15.16:0/4135935023 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.756+0000 7f1bbac56640 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec89070 0x555aff045e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1bb9c54640 1 -- 172.21.15.16:0/4135935023 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec89070 msgr2=0x555aff045e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1bb9c54640 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec89070 0x555aff045e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1bb9c54640 1 -- 172.21.15.16:0/4135935023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 msgr2=0x555aff039ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1bb9c54640 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 0x555aff039ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1bb9c54640 1 -- 172.21.15.16:0/4135935023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555afef54900 con 0x555afec68390 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1bb9c54640 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec68390 0x555aff03a010 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4000c00 tx=0x7f1ba40043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59b7d85782dab26b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1b9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4135935023 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4035040 con 0x555afec68390 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4135935023 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1ba4005450 con 0x555afec68390 2024-08-28T19:27:36.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4135935023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bac004050 con 0x555afec68390 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4135935023 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1ba4005650 con 0x555afec68390 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1bc2c2e740 10 client.26819 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4135935023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x555aff048910 con 0x555afec68390 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.757+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4135935023 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bac004050 con 0x555afec68390 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.758+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4135935023 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f1ba4003960 con 0x555afec68390 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.758+0000 7f1bc2c2e740 10 client.26819 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.758+0000 7f1bc2c2e740 10 client.26819 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.758+0000 7f1bc2c2e740 20 client.26819 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.758+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x555afec479f0 0x555aff04bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.758+0000 7f1bc2c2e740 4 client.26819 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.758+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4135935023 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555afefa75d0 con 0x555afec479f0 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.758+0000 7f1bbac56640 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x555afec479f0 0x555aff04bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.758+0000 7f1b9dffb640 10 client.26819 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.758+0000 7f1bbac56640 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x555afec479f0 0x555aff04bde0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=febe1576e7f1f0d6 server_cookie=4b76802473d50c0e in_seq=0 out_seq=0 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4135935023 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555afefa75d0 con 0x555afec479f0 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1b9dffb640 10 client.26819 handle_command_reply: tid=0 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1bc2c2e740 1 client.26819 shutdown 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4135935023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec68390 msgr2=0x555aff03a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec68390 0x555aff03a010 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4000c00 tx=0x7f1ba40043b0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4135935023 shutdown_connections 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x555afec479f0 0x555aff04bde0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec89070 0x555aff045e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec68390 0x555aff03a010 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1b9dffb640 0 client.26819 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4135935023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec601b0 0x555aff039ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4135935023 >> 172.21.15.16:0/4135935023 conn(0x555afec7d460 msgr2=0x555aff031530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:36.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.759+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4135935023 shutdown_connections 2024-08-28T19:27:36.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.760+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4135935023 wait complete. 2024-08-28T19:27:36.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.760+0000 7f1bc2c2e740 20 client.26819 trim_cache size 0 max 16384 2024-08-28T19:27:36.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bc2c2e740 1 Processor -- start 2024-08-28T19:27:36.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bc2c2e740 1 -- start start 2024-08-28T19:27:36.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec479f0 0x555aff039f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 0x555aff045d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec68390 0x555aff046260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41b2de0 con 0x555afec68390 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41b4920 con 0x555afec601b0 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41b34b0 con 0x555afec479f0 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bbac56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec68390 0x555aff046260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bbac56640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec479f0 0x555aff039f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bb9c54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 0x555aff045d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bbac56640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec479f0 0x555aff039f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:38518/0 (socket says 172.21.15.16:38518) 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bb9c54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 0x555aff045d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60322/0 (socket says 172.21.15.16:60322) 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.773+0000 7f1bbac56640 1 -- 172.21.15.16:0/3134267324 learned_addr learned my addr 172.21.15.16:0/3134267324 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.774+0000 7f1bbac56640 1 -- 172.21.15.16:0/3134267324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 msgr2=0x555aff045d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.774+0000 7f1bbac56640 1 --2- 172.21.15.16:0/3134267324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 0x555aff045d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.774+0000 7f1bbac56640 1 -- 172.21.15.16:0/3134267324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec68390 msgr2=0x555aff046260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:36.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.774+0000 7f1bbac56640 1 --2- 172.21.15.16:0/3134267324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec68390 0x555aff046260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.774+0000 7f1bbac56640 1 -- 172.21.15.16:0/3134267324 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555aff03ad90 con 0x555afec479f0 2024-08-28T19:27:36.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.774+0000 7f1bb9c54640 1 --2- 172.21.15.16:0/3134267324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 0x555aff045d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:36.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.774+0000 7f1bbac56640 1 --2- 172.21.15.16:0/3134267324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec479f0 0x555aff039f90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1bac011dd0 tx=0x7f1bac005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec836a7b1d7d7d4b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:36.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.774+0000 7f1b9ffff640 1 -- 172.21.15.16:0/3134267324 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac02c070 con 0x555afec479f0 2024-08-28T19:27:36.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.774+0000 7f1b9ffff640 1 -- 172.21.15.16:0/3134267324 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bac004990 con 0x555afec479f0 2024-08-28T19:27:36.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.774+0000 7f1b9ffff640 1 -- 172.21.15.16:0/3134267324 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac005040 con 0x555afec479f0 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.775+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/3134267324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec479f0 msgr2=0x555aff039f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.775+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/3134267324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec479f0 0x555aff039f90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1bac011dd0 tx=0x7f1bac005fd0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.775+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/3134267324 shutdown_connections 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.775+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/3134267324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec68390 0x555aff046260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.775+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/3134267324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 0x555aff045d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.775+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/3134267324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec479f0 0x555aff039f90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.775+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/3134267324 >> 172.21.15.16:0/3134267324 conn(0x555afec7d460 msgr2=0x555aff039880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.775+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/3134267324 shutdown_connections 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.775+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/3134267324 wait complete. 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.775+0000 7f1bc2c2e740 1 -- start start 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.775+0000 7f1bc2c2e740 1 Processor -- start 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec479f0 0x555aff03de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 0x555aff03e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bc2c2e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec68390 0x555aff03e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41b4250 con 0x555afec479f0 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41b3b80 con 0x555afec601b0 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bc2c2e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41b0dc0 con 0x555afec68390 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bc2c2e740 10 client.0 _ll_register_callbacks cb 0x7f1ba4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bbac56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec479f0 0x555aff03de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bbac56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec479f0 0x555aff03de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39274/0 (socket says 172.21.15.16:39274) 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bbac56640 1 -- 172.21.15.16:0/4095567749 learned_addr learned my addr 172.21.15.16:0/4095567749 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bb9c54640 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 0x555aff03e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bbac56640 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec68390 0x555aff03e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bb9c54640 1 -- 172.21.15.16:0/4095567749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec68390 msgr2=0x555aff03e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bb9c54640 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec68390 0x555aff03e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bb9c54640 1 -- 172.21.15.16:0/4095567749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec479f0 msgr2=0x555aff03de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bb9c54640 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec479f0 0x555aff03de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bb9c54640 1 -- 172.21.15.16:0/4095567749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555aff0b58c0 con 0x555afec601b0 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.776+0000 7f1bbac56640 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec479f0 0x555aff03de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1bb9c54640 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 0x555aff03e370 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4003ef0 tx=0x7f1ba402ab90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66bc876b2bcc978e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1b9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4095567749 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4059060 con 0x555afec601b0 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4095567749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bac004990 con 0x555afec601b0 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4095567749 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1ba40037f0 con 0x555afec601b0 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4095567749 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1ba4035070 con 0x555afec601b0 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1bc2c2e740 10 client.26825 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4095567749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x555aff0b6b40 con 0x555afec601b0 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4095567749 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bac004990 con 0x555afec601b0 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4095567749 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f1ba4004750 con 0x555afec601b0 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1bc2c2e740 10 client.26825 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1bc2c2e740 10 client.26825 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1bc2c2e740 20 client.26825 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x555afec89070 0x555aff0b9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1bc2c2e740 4 client.26825 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.777+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4095567749 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555afefa3070 con 0x555afec89070 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1bbac56640 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x555afec89070 0x555aff0b9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1bbac56640 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x555afec89070 0x555aff0b9270 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aa959b7bddd22203 server_cookie=b15b5ca984d6489c in_seq=0 out_seq=0 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1b9dffb640 10 client.26825 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1b9dffb640 1 -- 172.21.15.16:0/4095567749 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555afefa3070 con 0x555afec89070 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1b9dffb640 10 client.26825 handle_command_reply: tid=0 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1bc2c2e740 1 client.26825 shutdown 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4095567749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 msgr2=0x555aff03e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 0x555aff03e370 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4003ef0 tx=0x7f1ba402ab90 comp rx=0 tx=0).stop 2024-08-28T19:27:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4095567749 shutdown_connections 2024-08-28T19:27:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x555afec89070 0x555aff0b9270 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555afec68390 0x555aff03e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1b9dffb640 0 client.26825 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:27:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555afec601b0 0x555aff03e370 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1bc2c2e740 1 --2- 172.21.15.16:0/4095567749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555afec479f0 0x555aff03de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.778+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4095567749 >> 172.21.15.16:0/4095567749 conn(0x555afec7d460 msgr2=0x555aff02ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4095567749 shutdown_connections 2024-08-28T19:27:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bc2c2e740 1 -- 172.21.15.16:0/4095567749 wait complete. 2024-08-28T19:27:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bc2c2e740 20 client.26825 trim_cache size 0 max 16384 2024-08-28T19:27:36.785 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:27:36.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3089750974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ba00b4ca0 msgr2=0x7f1ba00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:36.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ba00b4ca0 0x7f1ba00b71b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00e3190 tx=0x7f1bb0004a90 comp rx=0 tx=0).stop 2024-08-28T19:27:36.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3089750974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb4103930 msgr2=0x7f1bb40754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:36.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb4103930 0x7f1bb40754b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000e9f0 tx=0x7f1bb000edf0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3089750974 shutdown_connections 2024-08-28T19:27:36.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb4109bc0 0x7f1bb4075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1bb4109270 0x7f1bb40759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1ba00b4ca0 0x7f1ba00b71b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 --2- 172.21.15.16:0/3089750974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb4103930 0x7f1bb40754b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:36.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3089750974 >> 172.21.15.16:0/3089750974 conn(0x7f1bb40ff3a0 msgr2=0x7f1bb41017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:36.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3089750974 shutdown_connections 2024-08-28T19:27:36.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:36.779+0000 7f1bbc6b9640 1 -- 172.21.15.16:0/3089750974 wait complete. 2024-08-28T19:27:36.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 307960 4.78 MB/sec execute 196 sec latency 32.856 ms 2024-08-28T19:27:37.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:37 smithi016 ceph-mon[18556]: pgmap v536: 129 pgs: 129 active+clean; 190 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 204 op/s 2024-08-28T19:27:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:37 smithi028 ceph-mon[21769]: pgmap v536: 129 pgs: 129 active+clean; 190 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 204 op/s 2024-08-28T19:27:37.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:27:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:37 smithi115 ceph-mon[22184]: pgmap v536: 129 pgs: 129 active+clean; 190 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 204 op/s 2024-08-28T19:27:37.861 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:27:37.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 309298 4.79 MB/sec execute 197 sec latency 17.438 ms 2024-08-28T19:27:38.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.354+0000 7f4001254640 1 -- 172.21.15.16:0/2450744798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc108a90 msgr2=0x7f3ffc10f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:38.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.354+0000 7f4001254640 1 --2- 172.21.15.16:0/2450744798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc108a90 0x7f3ffc10f2c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009ca0 tx=0x7f3ff402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.354+0000 7f4001254640 1 -- 172.21.15.16:0/2450744798 shutdown_connections 2024-08-28T19:27:38.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.354+0000 7f4001254640 1 --2- 172.21.15.16:0/2450744798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc108a90 0x7f3ffc10f2c0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.354+0000 7f4001254640 1 --2- 172.21.15.16:0/2450744798 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ffc108140 0x7f3ffc108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.354+0000 7f4001254640 1 --2- 172.21.15.16:0/2450744798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ffc102800 0x7f3ffc102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.354+0000 7f4001254640 1 -- 172.21.15.16:0/2450744798 >> 172.21.15.16:0/2450744798 conn(0x7f3ffc0794d0 msgr2=0x7f3ffc0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:38.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.354+0000 7f4001254640 1 -- 172.21.15.16:0/2450744798 shutdown_connections 2024-08-28T19:27:38.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.354+0000 7f4001254640 1 -- 172.21.15.16:0/2450744798 wait complete. 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.355+0000 7f4001254640 1 Processor -- start 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.355+0000 7f4001254640 1 -- start start 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f4001254640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc102800 0x7f3ffc19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f4001254640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ffc108140 0x7f3ffc19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f4001254640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ffc108a90 0x7f3ffc19f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f4001254640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc113820 con 0x7f3ffc102800 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f4001254640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc1136a0 con 0x7f3ffc108a90 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f4001254640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc1139a0 con 0x7f3ffc108140 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc102800 0x7f3ffc19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffa59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ffc108140 0x7f3ffc19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc102800 0x7f3ffc19a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39296/0 (socket says 172.21.15.16:39296) 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffad9d640 1 -- 172.21.15.16:0/1623259449 learned_addr learned my addr 172.21.15.16:0/1623259449 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ffc108a90 0x7f3ffc19f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffad9d640 1 -- 172.21.15.16:0/1623259449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ffc108140 msgr2=0x7f3ffc19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffad9d640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ffc108140 0x7f3ffc19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffad9d640 1 -- 172.21.15.16:0/1623259449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ffc108a90 msgr2=0x7f3ffc19f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffad9d640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ffc108a90 0x7f3ffc19f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffad9d640 1 -- 172.21.15.16:0/1623259449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ffc1a0120 con 0x7f3ffc102800 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ffc108a90 0x7f3ffc19f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.356+0000 7f3ffad9d640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc102800 0x7f3ffc19a1b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00ea60 tx=0x7f3fec00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.357+0000 7f3fe3fff640 1 -- 172.21.15.16:0/1623259449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec00cde0 con 0x7f3ffc102800 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.357+0000 7f4001254640 1 -- 172.21.15.16:0/1623259449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ffc1a0410 con 0x7f3ffc102800 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.357+0000 7f3fe3fff640 1 -- 172.21.15.16:0/1623259449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fec0035f0 con 0x7f3ffc102800 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.357+0000 7f3fe3fff640 1 -- 172.21.15.16:0/1623259449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec010690 con 0x7f3ffc102800 2024-08-28T19:27:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.357+0000 7f4001254640 1 -- 172.21.15.16:0/1623259449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ffc1a6b70 con 0x7f3ffc102800 2024-08-28T19:27:38.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.360+0000 7f40077c9740 1 Processor -- start 2024-08-28T19:27:38.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.360+0000 7f40077c9740 1 -- start start 2024-08-28T19:27:38.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.360+0000 7f3fe3fff640 1 -- 172.21.15.16:0/1623259449 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3fec0037a0 con 0x7f3ffc102800 2024-08-28T19:27:38.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.360+0000 7f40077c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f46310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.360+0000 7f40077c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 0x55e6f3f46850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.360+0000 7f40077c9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b80390 0x55e6f3f46d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.360+0000 7f40077c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e6f3ebd680 con 0x55e6f3b781b0 2024-08-28T19:27:38.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f40077c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e6f3eacc10 con 0x55e6f3ba1070 2024-08-28T19:27:38.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f40077c9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e6f3eb0bc0 con 0x55e6f3b80390 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffb59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f46310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffb59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f46310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60344/0 (socket says 172.21.15.16:60344) 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffb59e640 1 -- 172.21.15.16:0/2378530523 learned_addr learned my addr 172.21.15.16:0/2378530523 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffa59c640 1 --2- 172.21.15.16:0/2378530523 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b80390 0x55e6f3f46d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffad9d640 1 --2- 172.21.15.16:0/2378530523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 0x55e6f3f46850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffb59e640 1 -- 172.21.15.16:0/2378530523 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b80390 msgr2=0x55e6f3f46d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/2378530523 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b80390 0x55e6f3f46d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffb59e640 1 -- 172.21.15.16:0/2378530523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 msgr2=0x55e6f3f46850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/2378530523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 0x55e6f3f46850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffb59e640 1 -- 172.21.15.16:0/2378530523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6f3c0c650 con 0x55e6f3ba1070 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffad9d640 1 --2- 172.21.15.16:0/2378530523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 0x55e6f3f46850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/2378530523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f46310 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4033040 tx=0x7f3ff402f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94e353a1dd847718 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3fe3fff640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3fe40b4ca0 0x7f3fe40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.361+0000 7f3fe17fa640 1 -- 172.21.15.16:0/2378530523 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4031650 con 0x55e6f3ba1070 2024-08-28T19:27:38.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.362+0000 7f3ffad9d640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3fe40b4ca0 0x7f3fe40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.362+0000 7f3fe17fa640 1 -- 172.21.15.16:0/2378530523 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff4036040 con 0x55e6f3ba1070 2024-08-28T19:27:38.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.362+0000 7f3fe3fff640 1 -- 172.21.15.16:0/1623259449 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3fec0e5b90 con 0x7f3ffc102800 2024-08-28T19:27:38.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.362+0000 7f3fe17fa640 1 -- 172.21.15.16:0/2378530523 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4004540 con 0x55e6f3ba1070 2024-08-28T19:27:38.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.362+0000 7f40077c9740 1 -- 172.21.15.16:0/2378530523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 msgr2=0x55e6f3f46310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:38.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.362+0000 7f3ffad9d640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3fe40b4ca0 0x7f3fe40b71b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0e3020 tx=0x7f3fec0e7040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:38.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.362+0000 7f40077c9740 1 --2- 172.21.15.16:0/2378530523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f46310 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4033040 tx=0x7f3ff402f3c0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 -- 172.21.15.16:0/2378530523 shutdown_connections 2024-08-28T19:27:38.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 --2- 172.21.15.16:0/2378530523 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b80390 0x55e6f3f46d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 --2- 172.21.15.16:0/2378530523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 0x55e6f3f46850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 --2- 172.21.15.16:0/2378530523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f46310 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 -- 172.21.15.16:0/2378530523 >> 172.21.15.16:0/2378530523 conn(0x55e6f3b95460 msgr2=0x55e6f3f44200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 -- 172.21.15.16:0/2378530523 shutdown_connections 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 -- 172.21.15.16:0/2378530523 wait complete. 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 -- start start 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 Processor -- start 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3b781b0 0x55e6f3f565f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 0x55e6f3f56bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.363+0000 7f40077c9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3ba1070 0x55e6f3f5bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f40077c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e6f3d0f700 con 0x55e6f3b80390 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f40077c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e6f3e33750 con 0x55e6f3b781b0 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f40077c9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e6f3f54960 con 0x55e6f3ba1070 2024-08-28T19:27:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f40077c9740 10 client.0 _ll_register_callbacks cb 0x55e6f3e62e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 0x55e6f3f56bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffb59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3b781b0 0x55e6f3f565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 0x55e6f3f56bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39312/0 (socket says 172.21.15.16:39312) 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffa59c640 1 -- 172.21.15.16:0/3527518069 learned_addr learned my addr 172.21.15.16:0/3527518069 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffb59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3b781b0 0x55e6f3f565f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60346/0 (socket says 172.21.15.16:60346) 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3ba1070 0x55e6f3f5bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffb59e640 1 -- 172.21.15.16:0/3527518069 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3ba1070 msgr2=0x55e6f3f5bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3ba1070 0x55e6f3f5bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffb59e640 1 -- 172.21.15.16:0/3527518069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 msgr2=0x55e6f3f56bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 0x55e6f3f56bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffb59e640 1 -- 172.21.15.16:0/3527518069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6f3e6c900 con 0x55e6f3b781b0 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3b781b0 0x55e6f3f565f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3ff402f500 tx=0x7f3ff4004180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1816618dea643768 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3fdeff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.364+0000 7f3fdeff5640 1 -- 172.21.15.16:0/3527518069 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4059060 con 0x55e6f3b781b0 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f3fdeff5640 1 -- 172.21.15.16:0/3527518069 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3ff4031660 con 0x55e6f3b781b0 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f40077c9740 1 -- 172.21.15.16:0/3527518069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ff4036040 con 0x55e6f3b781b0 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f3fdeff5640 1 -- 172.21.15.16:0/3527518069 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3ff4060050 con 0x55e6f3b781b0 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f40077c9740 10 client.26837 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f40077c9740 1 -- 172.21.15.16:0/3527518069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e6f3f64a90 con 0x55e6f3b781b0 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f3fdeff5640 1 -- 172.21.15.16:0/3527518069 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff4036040 con 0x55e6f3b781b0 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f3fdeff5640 1 -- 172.21.15.16:0/3527518069 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f3ff404e030 con 0x55e6f3b781b0 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f40077c9740 10 client.26837 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f40077c9740 10 client.26837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f40077c9740 20 client.26837 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f40077c9740 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6f3b5f9f0 0x55e6f3f67df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f40077c9740 4 client.26837 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f40077c9740 1 -- 172.21.15.16:0/3527518069 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6f3ebf5d0 con 0x55e6f3b5f9f0 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.365+0000 7f3ffa59c640 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6f3b5f9f0 0x55e6f3f67df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.366+0000 7f3ffa59c640 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6f3b5f9f0 0x55e6f3f67df0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae8bc42bf9cbf8f3 server_cookie=96b566efe41ef782 in_seq=0 out_seq=0 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.366+0000 7f3fdeff5640 10 client.26837 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f3fdeff5640 1 -- 172.21.15.16:0/3527518069 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e6f3ebf5d0 con 0x55e6f3b5f9f0 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f3fdeff5640 10 client.26837 handle_command_reply: tid=0 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f40077c9740 1 client.26837 shutdown 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f40077c9740 1 -- 172.21.15.16:0/3527518069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3b781b0 msgr2=0x55e6f3f565f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f40077c9740 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3b781b0 0x55e6f3f565f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3ff402f500 tx=0x7f3ff4004180 comp rx=0 tx=0).stop 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f40077c9740 1 -- 172.21.15.16:0/3527518069 shutdown_connections 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f40077c9740 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6f3b5f9f0 0x55e6f3f67df0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f40077c9740 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3ba1070 0x55e6f3f5bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f40077c9740 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 0x55e6f3f56bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f3fdeff5640 0 client.26837 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f40077c9740 1 --2- 172.21.15.16:0/3527518069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3b781b0 0x55e6f3f565f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.367+0000 7f40077c9740 1 -- 172.21.15.16:0/3527518069 >> 172.21.15.16:0/3527518069 conn(0x55e6f3b95460 msgr2=0x55e6f3f494f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.368+0000 7f40077c9740 1 -- 172.21.15.16:0/3527518069 shutdown_connections 2024-08-28T19:27:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.368+0000 7f40077c9740 1 -- 172.21.15.16:0/3527518069 wait complete. 2024-08-28T19:27:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.368+0000 7f40077c9740 20 client.26837 trim_cache size 0 max 16384 2024-08-28T19:27:38.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f40077c9740 1 Processor -- start 2024-08-28T19:27:38.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f40077c9740 1 -- start start 2024-08-28T19:27:38.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f40077c9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b781b0 0x55e6f3f5b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f40077c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 0x55e6f3f47a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f40077c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f52fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f40077c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc1ae8a0 con 0x55e6f3b80390 2024-08-28T19:27:38.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f40077c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc1b03e0 con 0x55e6f3ba1070 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f40077c9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc1aef70 con 0x55e6f3b781b0 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f3ffa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 0x55e6f3f47a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f3ffa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 0x55e6f3f47a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39316/0 (socket says 172.21.15.16:39316) 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f3ffa59c640 1 -- 172.21.15.16:0/2117121021 learned_addr learned my addr 172.21.15.16:0/2117121021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/2117121021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f52fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.381+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/2117121021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b781b0 0x55e6f3f5b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.382+0000 7f3ffb59e640 1 -- 172.21.15.16:0/2117121021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b781b0 msgr2=0x55e6f3f5b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.382+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/2117121021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b781b0 0x55e6f3f5b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.382+0000 7f3ffb59e640 1 -- 172.21.15.16:0/2117121021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 msgr2=0x55e6f3f47a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.382+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/2117121021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 0x55e6f3f47a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.382+0000 7f3ffb59e640 1 -- 172.21.15.16:0/2117121021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6f3f576b0 con 0x55e6f3ba1070 2024-08-28T19:27:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.382+0000 7f3ffa59c640 1 --2- 172.21.15.16:0/2117121021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 0x55e6f3f47a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:38.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.382+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/2117121021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f52fe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3ff402fb60 tx=0x7f3ff402a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=276762d3cb75775a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:38.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.382+0000 7f3fe0ff9640 1 -- 172.21.15.16:0/2117121021 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4059070 con 0x55e6f3ba1070 2024-08-28T19:27:38.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.382+0000 7f3fe0ff9640 1 -- 172.21.15.16:0/2117121021 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff4036070 con 0x55e6f3ba1070 2024-08-28T19:27:38.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.382+0000 7f3fe0ff9640 1 -- 172.21.15.16:0/2117121021 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff402f660 con 0x55e6f3ba1070 2024-08-28T19:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.383+0000 7f40077c9740 1 -- 172.21.15.16:0/2117121021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 msgr2=0x55e6f3f52fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.383+0000 7f40077c9740 1 --2- 172.21.15.16:0/2117121021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f52fe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3ff402fb60 tx=0x7f3ff402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.383+0000 7f40077c9740 1 -- 172.21.15.16:0/2117121021 shutdown_connections 2024-08-28T19:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.383+0000 7f40077c9740 1 --2- 172.21.15.16:0/2117121021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f52fe0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.383+0000 7f40077c9740 1 --2- 172.21.15.16:0/2117121021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b80390 0x55e6f3f47a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.383+0000 7f40077c9740 1 --2- 172.21.15.16:0/2117121021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b781b0 0x55e6f3f5b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.383+0000 7f40077c9740 1 -- 172.21.15.16:0/2117121021 >> 172.21.15.16:0/2117121021 conn(0x55e6f3b95460 msgr2=0x55e6f3f47340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.383+0000 7f40077c9740 1 -- 172.21.15.16:0/2117121021 shutdown_connections 2024-08-28T19:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.383+0000 7f40077c9740 1 -- 172.21.15.16:0/2117121021 wait complete. 2024-08-28T19:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.383+0000 7f40077c9740 1 -- start start 2024-08-28T19:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.383+0000 7f40077c9740 1 Processor -- start 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f40077c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 0x55e6f3f62770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f40077c9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b80390 0x55e6f3f60c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f40077c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f611d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f40077c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc1afd10 con 0x55e6f3b781b0 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f40077c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc1af640 con 0x55e6f3ba1070 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f40077c9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc1ac880 con 0x55e6f3b80390 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f40077c9740 10 client.0 _ll_register_callbacks cb 0x7f3ff402f310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3ffb59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 0x55e6f3f62770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3ffb59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 0x55e6f3f62770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39318/0 (socket says 172.21.15.16:39318) 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3ffb59e640 1 -- 172.21.15.16:0/1379821571 learned_addr learned my addr 172.21.15.16:0/1379821571 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3ffa59c640 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b80390 0x55e6f3f60c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3ffb59e640 1 -- 172.21.15.16:0/1379821571 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b80390 msgr2=0x55e6f3f60c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b80390 0x55e6f3f60c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3ffb59e640 1 -- 172.21.15.16:0/1379821571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 msgr2=0x55e6f3f611d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f611d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3ffb59e640 1 -- 172.21.15.16:0/1379821571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6f3f61960 con 0x55e6f3b781b0 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3ffb59e640 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 0x55e6f3f62770 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4002230 tx=0x7f3ff4004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98adb452057e7ddb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3fdeff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.384+0000 7f3fdeff5640 1 -- 172.21.15.16:0/1379821571 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4059040 con 0x55e6f3b781b0 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f3fdeff5640 1 -- 172.21.15.16:0/1379821571 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f3ff402f660 con 0x55e6f3b781b0 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f40077c9740 1 -- 172.21.15.16:0/1379821571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ff4036070 con 0x55e6f3b781b0 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f3fdeff5640 1 -- 172.21.15.16:0/1379821571 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3ff4031870 con 0x55e6f3b781b0 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f40077c9740 10 client.17295 fetch_fsmap learned FSMap version 43 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f40077c9740 1 -- 172.21.15.16:0/1379821571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e6f3fd3500 con 0x55e6f3b781b0 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f3fdeff5640 1 -- 172.21.15.16:0/1379821571 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff4036070 con 0x55e6f3b781b0 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f3fdeff5640 1 -- 172.21.15.16:0/1379821571 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f3ff404e040 con 0x55e6f3b781b0 2024-08-28T19:27:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f40077c9740 10 client.17295 fetch_fsmap finished waiting for FSMap version 43 2024-08-28T19:27:38.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f40077c9740 10 client.17295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:27:38.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f40077c9740 20 client.17295 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:38.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f40077c9740 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6f3b34da0 0x55e6f3fd5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:38.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f40077c9740 4 client.17295 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:27:38.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f40077c9740 1 -- 172.21.15.16:0/1379821571 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e6f3ebb070 con 0x55e6f3b34da0 2024-08-28T19:27:38.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.385+0000 7f3ffa59c640 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6f3b34da0 0x55e6f3fd5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:38.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.386+0000 7f3ffa59c640 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6f3b34da0 0x55e6f3fd5850 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e894db2b67486b9 server_cookie=7ae0ae77fe3b9cf9 in_seq=0 out_seq=0 2024-08-28T19:27:38.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.386+0000 7f3fdeff5640 10 client.17295 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f3fdeff5640 1 -- 172.21.15.16:0/1379821571 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e6f3ebb070 con 0x55e6f3b34da0 2024-08-28T19:27:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f3fdeff5640 10 client.17295 handle_command_reply: tid=0 2024-08-28T19:27:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 1 client.17295 shutdown 2024-08-28T19:27:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 1 -- 172.21.15.16:0/1379821571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 msgr2=0x55e6f3f62770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 0x55e6f3f62770 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4002230 tx=0x7f3ff4004700 comp rx=0 tx=0).stop 2024-08-28T19:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 1 -- 172.21.15.16:0/1379821571 shutdown_connections 2024-08-28T19:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6f3ba1070 0x55e6f3f611d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6f3b34da0 0x55e6f3fd5850 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6f3b80390 0x55e6f3f60c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f3fdeff5640 0 client.17295 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 1 --2- 172.21.15.16:0/1379821571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6f3b781b0 0x55e6f3f62770 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 1 -- 172.21.15.16:0/1379821571 >> 172.21.15.16:0/1379821571 conn(0x55e6f3b95460 msgr2=0x55e6f3f47ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 1 -- 172.21.15.16:0/1379821571 shutdown_connections 2024-08-28T19:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 1 -- 172.21.15.16:0/1379821571 wait complete. 2024-08-28T19:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.388+0000 7f40077c9740 20 client.17295 trim_cache size 0 max 16384 2024-08-28T19:27:38.394 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:27:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 -- 172.21.15.16:0/1623259449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3fe40b4ca0 msgr2=0x7f3fe40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3fe40b4ca0 0x7f3fe40b71b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0e3020 tx=0x7f3fec0e7040 comp rx=0 tx=0).stop 2024-08-28T19:27:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 -- 172.21.15.16:0/1623259449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc102800 msgr2=0x7f3ffc19a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc102800 0x7f3ffc19a1b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00ea60 tx=0x7f3fec00ef20 comp rx=0 tx=0).stop 2024-08-28T19:27:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 -- 172.21.15.16:0/1623259449 shutdown_connections 2024-08-28T19:27:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3fe40b4ca0 0x7f3fe40b71b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ffc108a90 0x7f3ffc19f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ffc108140 0x7f3ffc19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 --2- 172.21.15.16:0/1623259449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc102800 0x7f3ffc19a1b0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 -- 172.21.15.16:0/1623259449 >> 172.21.15.16:0/1623259449 conn(0x7f3ffc0794d0 msgr2=0x7f3ffc104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 -- 172.21.15.16:0/1623259449 shutdown_connections 2024-08-28T19:27:38.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:38.389+0000 7f4001254640 1 -- 172.21.15.16:0/1623259449 wait complete. 2024-08-28T19:27:38.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 310704 4.79 MB/sec execute 198 sec latency 14.034 ms 2024-08-28T19:27:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:39 smithi028 ceph-mon[21769]: pgmap v537: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 229 op/s 2024-08-28T19:27:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:39 smithi115 ceph-mon[22184]: pgmap v537: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 229 op/s 2024-08-28T19:27:39.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:39 smithi016 ceph-mon[18556]: pgmap v537: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 229 op/s 2024-08-28T19:27:39.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 312422 4.84 MB/sec execute 199 sec latency 52.569 ms 2024-08-28T19:27:40.219 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:27:40.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:27:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:40 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:27:40.506 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:27:40.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:40 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:27:40.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:40 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:27:40.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 313661 4.84 MB/sec execute 200 sec latency 54.394 ms 2024-08-28T19:27:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.015+0000 7f197cf9d640 1 -- 172.21.15.16:0/4207954999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1978107f50 msgr2=0x7f1978108360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.015+0000 7f197cf9d640 1 --2- 172.21.15.16:0/4207954999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1978107f50 0x7f1978108360 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f1960009900 tx=0x7f196002a410 comp rx=0 tx=0).stop 2024-08-28T19:27:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.015+0000 7f197cf9d640 1 -- 172.21.15.16:0/4207954999 shutdown_connections 2024-08-28T19:27:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.015+0000 7f197cf9d640 1 --2- 172.21.15.16:0/4207954999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19781088a0 0x7f197810f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.015+0000 7f197cf9d640 1 --2- 172.21.15.16:0/4207954999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1978107f50 0x7f1978108360 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.015+0000 7f197cf9d640 1 --2- 172.21.15.16:0/4207954999 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1978102610 0x7f1978102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.015+0000 7f197cf9d640 1 -- 172.21.15.16:0/4207954999 >> 172.21.15.16:0/4207954999 conn(0x7f19780fe060 msgr2=0x7f19781004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.015+0000 7f197cf9d640 1 -- 172.21.15.16:0/4207954999 shutdown_connections 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.015+0000 7f197cf9d640 1 -- 172.21.15.16:0/4207954999 wait complete. 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.016+0000 7f197cf9d640 1 Processor -- start 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.016+0000 7f197cf9d640 1 -- start start 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.016+0000 7f197cf9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1978102610 0x7f197819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.016+0000 7f197cf9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978107f50 0x7f197819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.016+0000 7f197cf9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19781088a0 0x7f197819f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.016+0000 7f197cf9d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1978113630 con 0x7f19781088a0 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.016+0000 7f197cf9d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19781134b0 con 0x7f1978107f50 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.016+0000 7f197cf9d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f19781137b0 con 0x7f1978102610 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f1977fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19781088a0 0x7f197819f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19777fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1978102610 0x7f197819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19777fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1978102610 0x7f197819a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:38592/0 (socket says 172.21.15.16:38592) 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19777fe640 1 -- 172.21.15.16:0/3772598651 learned_addr learned my addr 172.21.15.16:0/3772598651 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f196effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978107f50 0x7f197819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19777fe640 1 -- 172.21.15.16:0/3772598651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978107f50 msgr2=0x7f197819a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19777fe640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978107f50 0x7f197819a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19777fe640 1 -- 172.21.15.16:0/3772598651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19781088a0 msgr2=0x7f197819f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19777fe640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19781088a0 0x7f197819f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19777fe640 1 -- 172.21.15.16:0/3772598651 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197819ffb0 con 0x7f1978102610 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f1977fff640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19781088a0 0x7f197819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19777fe640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1978102610 0x7f197819a040 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f196800e9f0 tx=0x7f196800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19757fa640 1 -- 172.21.15.16:0/3772598651 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196800cd80 con 0x7f1978102610 2024-08-28T19:27:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f197cf9d640 1 -- 172.21.15.16:0/3772598651 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19781a02a0 con 0x7f1978102610 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19757fa640 1 -- 172.21.15.16:0/3772598651 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19680054d0 con 0x7f1978102610 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f19757fa640 1 -- 172.21.15.16:0/3772598651 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1968010690 con 0x7f1978102610 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.017+0000 7f197cf9d640 1 -- 172.21.15.16:0/3772598651 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19781a6a00 con 0x7f1978102610 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f1984774740 1 Processor -- start 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f1984774740 1 -- start start 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f19757fa640 1 -- 172.21.15.16:0/3772598651 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1968010830 con 0x7f1978102610 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f1984774740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e212a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f1984774740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e212ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f1984774740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e212b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f1984774740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b2e20a05f0 con 0x55b2e1d5b490 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f1984774740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b2e208fb80 con 0x55b2e1d84350 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f1984774740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b2e2093b30 con 0x55b2e1d63670 2024-08-28T19:27:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f1977fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e212a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f19777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e212ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f1977fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e212a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60380/0 (socket says 172.21.15.16:60380) 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.020+0000 7f1977fff640 1 -- 172.21.15.16:0/253272852 learned_addr learned my addr 172.21.15.16:0/253272852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f196effd640 1 --2- 172.21.15.16:0/253272852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e212b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f1977fff640 1 -- 172.21.15.16:0/253272852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 msgr2=0x55b2e212b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f1977fff640 1 --2- 172.21.15.16:0/253272852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e212b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f1977fff640 1 -- 172.21.15.16:0/253272852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 msgr2=0x55b2e212ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f1977fff640 1 --2- 172.21.15.16:0/253272852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e212ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f1977fff640 1 -- 172.21.15.16:0/253272852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2e1def930 con 0x55b2e1d84350 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f196effd640 1 --2- 172.21.15.16:0/253272852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e212b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f19757fa640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19640b4ca0 0x7f19640b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f1977fff640 1 --2- 172.21.15.16:0/253272852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e212a950 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f19700117e0 tx=0x7f1970011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=102f6fe273f7913a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f19777fe640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19640b4ca0 0x7f19640b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f19757fa640 1 -- 172.21.15.16:0/3772598651 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f19680e5cd0 con 0x7f1978102610 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f196dffb640 1 -- 172.21.15.16:0/253272852 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970011310 con 0x55b2e1d84350 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.021+0000 7f196dffb640 1 -- 172.21.15.16:0/253272852 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1970003e20 con 0x55b2e1d84350 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.022+0000 7f196dffb640 1 -- 172.21.15.16:0/253272852 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970004460 con 0x55b2e1d84350 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.022+0000 7f19777fe640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19640b4ca0 0x7f19640b71b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f19680e3160 tx=0x7f19680e9820 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.022+0000 7f1984774740 1 -- 172.21.15.16:0/253272852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 msgr2=0x55b2e212a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.022+0000 7f1984774740 1 --2- 172.21.15.16:0/253272852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e212a950 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f19700117e0 tx=0x7f1970011ca0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.022+0000 7f1984774740 1 -- 172.21.15.16:0/253272852 shutdown_connections 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.022+0000 7f1984774740 1 --2- 172.21.15.16:0/253272852 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e212b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.022+0000 7f1984774740 1 --2- 172.21.15.16:0/253272852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e212ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.022+0000 7f1984774740 1 --2- 172.21.15.16:0/253272852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e212a950 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.022+0000 7f1984774740 1 -- 172.21.15.16:0/253272852 >> 172.21.15.16:0/253272852 conn(0x55b2e1d78740 msgr2=0x55b2e2128840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.022+0000 7f1984774740 1 -- 172.21.15.16:0/253272852 shutdown_connections 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.022+0000 7f1984774740 1 -- 172.21.15.16:0/253272852 wait complete. 2024-08-28T19:27:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.023+0000 7f1984774740 1 -- start start 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.023+0000 7f1984774740 1 Processor -- start 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1984774740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d5b490 0x55b2e21360d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1984774740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d63670 0x55b2e2136610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1984774740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d84350 0x55b2e2142420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1984774740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b2e1ef2890 con 0x55b2e1d63670 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1984774740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b2e2016790 con 0x55b2e1d5b490 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1984774740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b2e2139130 con 0x55b2e1d84350 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1984774740 10 client.0 _ll_register_callbacks cb 0x55b2e1d9bba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1977fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d5b490 0x55b2e21360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f196effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d63670 0x55b2e2136610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1977fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d5b490 0x55b2e21360d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60394/0 (socket says 172.21.15.16:60394) 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f196effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d63670 0x55b2e2136610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39360/0 (socket says 172.21.15.16:39360) 2024-08-28T19:27:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f196effd640 1 -- 172.21.15.16:0/869909194 learned_addr learned my addr 172.21.15.16:0/869909194 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1977fff640 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d84350 0x55b2e2142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f196effd640 1 -- 172.21.15.16:0/869909194 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d84350 msgr2=0x55b2e2142420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f196effd640 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d84350 0x55b2e2142420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f196effd640 1 -- 172.21.15.16:0/869909194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d5b490 msgr2=0x55b2e21360d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f196effd640 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d5b490 0x55b2e21360d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f196effd640 1 -- 172.21.15.16:0/869909194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2e204df30 con 0x55b2e1d63670 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f196effd640 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d63670 0x55b2e2136610 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1960009900 tx=0x7f1960032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef7139542723faa server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1956ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.024+0000 7f1956ffd640 1 -- 172.21.15.16:0/869909194 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1960035040 con 0x55b2e1d63670 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1956ffd640 1 -- 172.21.15.16:0/869909194 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1960031760 con 0x55b2e1d63670 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1984774740 1 -- 172.21.15.16:0/869909194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1970003e20 con 0x55b2e1d63670 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1956ffd640 1 -- 172.21.15.16:0/869909194 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1960004040 con 0x55b2e1d63670 2024-08-28T19:27:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1984774740 10 client.17307 fetch_fsmap learned FSMap version 44 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1984774740 1 -- 172.21.15.16:0/869909194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55b2e2144f10 con 0x55b2e1d63670 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1956ffd640 1 -- 172.21.15.16:0/869909194 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1970003e20 con 0x55b2e1d63670 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1956ffd640 1 -- 172.21.15.16:0/869909194 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 44) ==== 8374+0+0 (secure 0 0 0) 0x7f1960004750 con 0x55b2e1d63670 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1984774740 10 client.17307 fetch_fsmap finished waiting for FSMap version 44 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1984774740 10 client.17307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1984774740 20 client.17307 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1984774740 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b2e1d42cd0 0x55b2e2148400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1984774740 4 client.17307 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1984774740 1 -- 172.21.15.16:0/869909194 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2e20a2540 con 0x55b2e1d42cd0 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.025+0000 7f1977fff640 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b2e1d42cd0 0x55b2e2148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.026+0000 7f1956ffd640 10 client.17307 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.026+0000 7f1977fff640 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b2e1d42cd0 0x55b2e2148400 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be55859fc81c50ad server_cookie=e41889d37da2cdbe in_seq=0 out_seq=0 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.027+0000 7f1956ffd640 1 -- 172.21.15.16:0/869909194 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b2e20a2540 con 0x55b2e1d42cd0 2024-08-28T19:27:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.027+0000 7f1956ffd640 10 client.17307 handle_command_reply: tid=0 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.027+0000 7f1984774740 1 client.17307 shutdown 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1984774740 1 -- 172.21.15.16:0/869909194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d63670 msgr2=0x55b2e2136610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1984774740 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d63670 0x55b2e2136610 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1960009900 tx=0x7f1960032e90 comp rx=0 tx=0).stop 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1984774740 1 -- 172.21.15.16:0/869909194 shutdown_connections 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1984774740 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b2e1d42cd0 0x55b2e2148400 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1984774740 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d84350 0x55b2e2142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1984774740 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d63670 0x55b2e2136610 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1956ffd640 0 client.17307 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1984774740 1 --2- 172.21.15.16:0/869909194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d5b490 0x55b2e21360d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1984774740 1 -- 172.21.15.16:0/869909194 >> 172.21.15.16:0/869909194 conn(0x55b2e1d78740 msgr2=0x55b2e212db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1984774740 1 -- 172.21.15.16:0/869909194 shutdown_connections 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1984774740 1 -- 172.21.15.16:0/869909194 wait complete. 2024-08-28T19:27:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.028+0000 7f1984774740 20 client.17307 trim_cache size 0 max 16384 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.048+0000 7f1984774740 1 Processor -- start 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.048+0000 7f1984774740 1 -- start start 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1984774740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e2137d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1984774740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e2127f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1984774740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e2128490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1984774740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19781b09d0 con 0x55b2e1d5b490 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1984774740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19781b2540 con 0x55b2e1d84350 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1984774740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f19781b10a0 con 0x55b2e1d63670 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1977fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e2137d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1977fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e2137d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39372/0 (socket says 172.21.15.16:39372) 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1977fff640 1 -- 172.21.15.16:0/4059223744 learned_addr learned my addr 172.21.15.16:0/4059223744 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f196effd640 1 --2- 172.21.15.16:0/4059223744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e2127f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1977fff640 1 --2- 172.21.15.16:0/4059223744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e2128490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1977fff640 1 -- 172.21.15.16:0/4059223744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 msgr2=0x55b2e2127f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1977fff640 1 --2- 172.21.15.16:0/4059223744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e2127f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1977fff640 1 -- 172.21.15.16:0/4059223744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 msgr2=0x55b2e2128490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1977fff640 1 --2- 172.21.15.16:0/4059223744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e2128490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1977fff640 1 -- 172.21.15.16:0/4059223744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2e2137390 con 0x55b2e1d5b490 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f1977fff640 1 --2- 172.21.15.16:0/4059223744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e2137d00 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f197000f790 tx=0x7f1970009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b337101e7b2d0956 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.049+0000 7f196d7fa640 1 -- 172.21.15.16:0/4059223744 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970005030 con 0x55b2e1d5b490 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.050+0000 7f196d7fa640 1 -- 172.21.15.16:0/4059223744 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f197000fdf0 con 0x55b2e1d5b490 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.050+0000 7f196d7fa640 1 -- 172.21.15.16:0/4059223744 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970025040 con 0x55b2e1d5b490 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.050+0000 7f1984774740 1 -- 172.21.15.16:0/4059223744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 msgr2=0x55b2e2137d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.050+0000 7f1984774740 1 --2- 172.21.15.16:0/4059223744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e2137d00 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f197000f790 tx=0x7f1970009500 comp rx=0 tx=0).stop 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.050+0000 7f1984774740 1 -- 172.21.15.16:0/4059223744 shutdown_connections 2024-08-28T19:27:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.050+0000 7f1984774740 1 --2- 172.21.15.16:0/4059223744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e2128490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.050+0000 7f1984774740 1 --2- 172.21.15.16:0/4059223744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e2127f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.050+0000 7f1984774740 1 --2- 172.21.15.16:0/4059223744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e2137d00 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.050+0000 7f1984774740 1 -- 172.21.15.16:0/4059223744 >> 172.21.15.16:0/4059223744 conn(0x55b2e1d78740 msgr2=0x55b2e21375f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.051+0000 7f1984774740 1 -- 172.21.15.16:0/4059223744 shutdown_connections 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.051+0000 7f1984774740 1 -- 172.21.15.16:0/4059223744 wait complete. 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.051+0000 7f1984774740 1 -- start start 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.051+0000 7f1984774740 1 Processor -- start 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.051+0000 7f1984774740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e213a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.051+0000 7f1984774740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e213ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.051+0000 7f1984774740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e21b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.051+0000 7f1984774740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19781b1e70 con 0x55b2e1d5b490 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1984774740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19781b1770 con 0x55b2e1d84350 2024-08-28T19:27:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1984774740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f19781ae9b0 con 0x55b2e1d63670 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1984774740 10 client.0 _ll_register_callbacks cb 0x7f1960009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1977fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e213a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1977fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e213a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39378/0 (socket says 172.21.15.16:39378) 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1977fff640 1 -- 172.21.15.16:0/3463395575 learned_addr learned my addr 172.21.15.16:0/3463395575 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f196effd640 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e213ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1977fff640 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e21b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f196effd640 1 -- 172.21.15.16:0/3463395575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 msgr2=0x55b2e21b1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f196effd640 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e21b1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f196effd640 1 -- 172.21.15.16:0/3463395575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 msgr2=0x55b2e213a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f196effd640 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e213a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f196effd640 1 -- 172.21.15.16:0/3463395575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2e21b2300 con 0x55b2e1d63670 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1977fff640 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e213a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f196effd640 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e213ac70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1960001eb0 tx=0x7f1960004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a97cfbe3f32a0a0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1956ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1956ffd640 1 -- 172.21.15.16:0/3463395575 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1960058060 con 0x55b2e1d63670 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1956ffd640 1 -- 172.21.15.16:0/3463395575 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f1960003af0 con 0x55b2e1d63670 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.052+0000 7f1984774740 1 -- 172.21.15.16:0/3463395575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f197000fdf0 con 0x55b2e1d63670 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1956ffd640 1 -- 172.21.15.16:0/3463395575 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1960035070 con 0x55b2e1d63670 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1984774740 10 client.26668 fetch_fsmap learned FSMap version 44 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1984774740 1 -- 172.21.15.16:0/3463395575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55b2e21b3410 con 0x55b2e1d63670 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1956ffd640 1 -- 172.21.15.16:0/3463395575 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f197000fdf0 con 0x55b2e1d63670 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1956ffd640 1 -- 172.21.15.16:0/3463395575 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 44) ==== 8374+0+0 (secure 0 0 0) 0x7f1960005360 con 0x55b2e1d63670 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1984774740 10 client.26668 fetch_fsmap finished waiting for FSMap version 44 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1984774740 10 client.26668 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1984774740 20 client.26668 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1984774740 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b2e1d18080 0x55b2e21b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1984774740 4 client.26668 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1984774740 1 -- 172.21.15.16:0/3463395575 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b2e209dfe0 con 0x55b2e1d18080 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.053+0000 7f1977fff640 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b2e1d18080 0x55b2e21b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.054+0000 7f1956ffd640 10 client.26668 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.054+0000 7f1977fff640 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b2e1d18080 0x55b2e21b5c00 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c77c7c605da7016 server_cookie=9f0ec0d086f11e1d in_seq=0 out_seq=0 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1956ffd640 1 -- 172.21.15.16:0/3463395575 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b2e209dfe0 con 0x55b2e1d18080 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1956ffd640 10 client.26668 handle_command_reply: tid=0 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1984774740 1 client.26668 shutdown 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1984774740 1 -- 172.21.15.16:0/3463395575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 msgr2=0x55b2e213ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1984774740 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e213ac70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1960001eb0 tx=0x7f1960004410 comp rx=0 tx=0).stop 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1984774740 1 -- 172.21.15.16:0/3463395575 shutdown_connections 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1984774740 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e1d84350 0x55b2e21b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1984774740 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b2e1d18080 0x55b2e21b5c00 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1984774740 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b2e1d63670 0x55b2e213ac70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1984774740 1 --2- 172.21.15.16:0/3463395575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e1d5b490 0x55b2e213a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1984774740 1 -- 172.21.15.16:0/3463395575 >> 172.21.15.16:0/3463395575 conn(0x55b2e1d78740 msgr2=0x55b2e2137560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.056+0000 7f1956ffd640 0 client.26668 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f1984774740 1 -- 172.21.15.16:0/3463395575 shutdown_connections 2024-08-28T19:27:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f1984774740 1 -- 172.21.15.16:0/3463395575 wait complete. 2024-08-28T19:27:41.222 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:27:41.222 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:27:41.222 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "38d6d63f-9bab-468d-8eac-25cd6efa2607", 2024-08-28T19:27:41.222 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:27:41.222 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:27:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f1984774740 20 client.26668 trim_cache size 0 max 16384 2024-08-28T19:27:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f197cf9d640 1 -- 172.21.15.16:0/3772598651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19640b4ca0 msgr2=0x7f19640b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f197cf9d640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19640b4ca0 0x7f19640b71b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f19680e3160 tx=0x7f19680e9820 comp rx=0 tx=0).stop 2024-08-28T19:27:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f197cf9d640 1 -- 172.21.15.16:0/3772598651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1978102610 msgr2=0x7f197819a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f197cf9d640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1978102610 0x7f197819a040 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f196800e9f0 tx=0x7f196800edf0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f197cf9d640 1 -- 172.21.15.16:0/3772598651 shutdown_connections 2024-08-28T19:27:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f197cf9d640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19781088a0 0x7f197819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f197cf9d640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19640b4ca0 0x7f19640b71b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f197cf9d640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978107f50 0x7f197819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f197cf9d640 1 --2- 172.21.15.16:0/3772598651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1978102610 0x7f197819a040 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f197cf9d640 1 -- 172.21.15.16:0/3772598651 >> 172.21.15.16:0/3772598651 conn(0x7f19780fe060 msgr2=0x7f1978100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.057+0000 7f197cf9d640 1 -- 172.21.15.16:0/3772598651 shutdown_connections 2024-08-28T19:27:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:41.058+0000 7f197cf9d640 1 -- 172.21.15.16:0/3772598651 wait complete. 2024-08-28T19:27:41.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: pgmap v538: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 219 op/s 2024-08-28T19:27:41.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:41.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:27:41.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:27:41.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:27:41.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:41 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: pgmap v538: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 219 op/s 2024-08-28T19:27:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:27:41.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:27:41.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:27:41.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:27:41.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:41 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:41.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: pgmap v538: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 219 op/s 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-28T19:27:41.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:27:41.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:27:41.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:27:41.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:27:41.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:41 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:41.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 315772 4.88 MB/sec execute 201 sec latency 67.051 ms 2024-08-28T19:27:42.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:27:42.474 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:27:42.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-08-28T19:27:42.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi115 to 176.0M 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: Unable to set osd_memory_target on smithi016 to 185701376: error parsing value: Value '185701376' is below minimum 939524096 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: Unable to set osd_memory_target on smithi028 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: Unable to set osd_memory_target on smithi115 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:27:42.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:42 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi115 to 176.0M 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: Unable to set osd_memory_target on smithi016 to 185701376: error parsing value: Value '185701376' is below minimum 939524096 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: Unable to set osd_memory_target on smithi028 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: Unable to set osd_memory_target on smithi115 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:27:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:27:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:27:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:42 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-08-28T19:27:42.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-08-28T19:27:42.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi115 to 176.0M 2024-08-28T19:27:42.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: Unable to set osd_memory_target on smithi016 to 185701376: error parsing value: Value '185701376' is below minimum 939524096 2024-08-28T19:27:42.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: Unable to set osd_memory_target on smithi028 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-08-28T19:27:42.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: Unable to set osd_memory_target on smithi115 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-08-28T19:27:42.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:27:42.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:42 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:27:42.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 315859 4.85 MB/sec execute 202 sec latency 915.983 ms 2024-08-28T19:27:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.993+0000 7f5765a90640 1 -- 172.21.15.16:0/3979736843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57601081c0 msgr2=0x7f57601085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.993+0000 7f5765a90640 1 --2- 172.21.15.16:0/3979736843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57601081c0 0x7f57601085d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f5748009900 tx=0x7f574802a410 comp rx=0 tx=0).stop 2024-08-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.993+0000 7f5765a90640 1 -- 172.21.15.16:0/3979736843 shutdown_connections 2024-08-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.993+0000 7f5765a90640 1 --2- 172.21.15.16:0/3979736843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5760108b10 0x7f576010f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.993+0000 7f5765a90640 1 --2- 172.21.15.16:0/3979736843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57601081c0 0x7f57601085d0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.993+0000 7f5765a90640 1 --2- 172.21.15.16:0/3979736843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5760102880 0x7f5760102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.993+0000 7f5765a90640 1 -- 172.21.15.16:0/3979736843 >> 172.21.15.16:0/3979736843 conn(0x7f5760079510 msgr2=0x7f5760079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.993+0000 7f5765a90640 1 -- 172.21.15.16:0/3979736843 shutdown_connections 2024-08-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.993+0000 7f5765a90640 1 -- 172.21.15.16:0/3979736843 wait complete. 2024-08-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.993+0000 7f5765a90640 1 Processor -- start 2024-08-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5765a90640 1 -- start start 2024-08-28T19:27:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5765a90640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5760102880 0x7f576019b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5765a90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57601081c0 0x7f576019b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5765a90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5760108b10 0x7f576019bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5765a90640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5760112780 con 0x7f57601081c0 2024-08-28T19:27:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5765a90640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5760112600 con 0x7f5760108b10 2024-08-28T19:27:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5765a90640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5760112900 con 0x7f5760102880 2024-08-28T19:27:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5756ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57601081c0 0x7f576019b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5756ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57601081c0 0x7f576019b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39384/0 (socket says 172.21.15.16:39384) 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5756ffd640 1 -- 172.21.15.16:0/370182238 learned_addr learned my addr 172.21.15.16:0/370182238 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f575ffff640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5760108b10 0x7f576019bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f575f7fe640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5760102880 0x7f576019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5756ffd640 1 -- 172.21.15.16:0/370182238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5760102880 msgr2=0x7f576019b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5756ffd640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5760102880 0x7f576019b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5756ffd640 1 -- 172.21.15.16:0/370182238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5760108b10 msgr2=0x7f576019bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.994+0000 7f5756ffd640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5760108b10 0x7f576019bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.995+0000 7f5756ffd640 1 -- 172.21.15.16:0/370182238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57601a6390 con 0x7f57601081c0 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.995+0000 7f575ffff640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5760108b10 0x7f576019bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.995+0000 7f5756ffd640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57601081c0 0x7f576019b650 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f574802a930 tx=0x7f5748004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.995+0000 7f575d7fa640 1 -- 172.21.15.16:0/370182238 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574803f030 con 0x7f57601081c0 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.995+0000 7f5765a90640 1 -- 172.21.15.16:0/370182238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57601a6620 con 0x7f57601081c0 2024-08-28T19:27:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.995+0000 7f575d7fa640 1 -- 172.21.15.16:0/370182238 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5748003aa0 con 0x7f57601081c0 2024-08-28T19:27:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.995+0000 7f575d7fa640 1 -- 172.21.15.16:0/370182238 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5748035070 con 0x7f57601081c0 2024-08-28T19:27:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.995+0000 7f5765a90640 1 -- 172.21.15.16:0/370182238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57601a6b00 con 0x7f57601081c0 2024-08-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f576c005740 1 Processor -- start 2024-08-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f575d7fa640 1 -- 172.21.15.16:0/370182238 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5748042030 con 0x7f57601081c0 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f576c005740 1 -- start start 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f576c005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c521e070 0x55a6c55c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f576c005740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55c3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f576c005740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f576c005740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a6c553a630 con 0x55a6c51fd390 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f576c005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a6c5529bc0 con 0x55a6c521e070 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f576c005740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a6c552db70 con 0x55a6c51f51b0 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f5756ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f575f7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55c3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f575ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c521e070 0x55a6c55c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f5756ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55c3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39398/0 (socket says 172.21.15.16:39398) 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f5756ffd640 1 -- 172.21.15.16:0/735792456 learned_addr learned my addr 172.21.15.16:0/735792456 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f575ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c521e070 0x55a6c55c32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60446/0 (socket says 172.21.15.16:60446) 2024-08-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.998+0000 7f575f7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55c3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:38648/0 (socket says 172.21.15.16:38648) 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f575ffff640 1 -- 172.21.15.16:0/735792456 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 msgr2=0x55a6c55c3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f575ffff640 1 --2- 172.21.15.16:0/735792456 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55c3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f575ffff640 1 -- 172.21.15.16:0/735792456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 msgr2=0x55a6c55c3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f575ffff640 1 --2- 172.21.15.16:0/735792456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55c3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f575d7fa640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f574c0b4ca0 0x7f574c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f575ffff640 1 -- 172.21.15.16:0/735792456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6c5289650 con 0x55a6c521e070 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f5756ffd640 1 --2- 172.21.15.16:0/735792456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55c3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f575d7fa640 1 -- 172.21.15.16:0/370182238 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f57481060c0 con 0x7f57601081c0 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f575ffff640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f574c0b4ca0 0x7f574c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f575ffff640 1 --2- 172.21.15.16:0/735792456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c521e070 0x55a6c55c32c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5758005fd0 tx=0x7f5758010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3478d71000fc13bf server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f5755ffb640 1 -- 172.21.15.16:0/735792456 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758022070 con 0x55a6c521e070 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f5755ffb640 1 -- 172.21.15.16:0/735792456 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f575801cd90 con 0x55a6c521e070 2024-08-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:42.999+0000 7f5755ffb640 1 -- 172.21.15.16:0/735792456 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758016070 con 0x55a6c521e070 2024-08-28T19:27:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.000+0000 7f576c005740 1 -- 172.21.15.16:0/735792456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c521e070 msgr2=0x55a6c55c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.000+0000 7f576c005740 1 --2- 172.21.15.16:0/735792456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c521e070 0x55a6c55c32c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5758005fd0 tx=0x7f5758010740 comp rx=0 tx=0).stop 2024-08-28T19:27:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.000+0000 7f575ffff640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f574c0b4ca0 0x7f574c0b71b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f575801f000 tx=0x7f5758022000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.000+0000 7f576c005740 1 -- 172.21.15.16:0/735792456 shutdown_connections 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.000+0000 7f576c005740 1 --2- 172.21.15.16:0/735792456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55c3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.000+0000 7f576c005740 1 --2- 172.21.15.16:0/735792456 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55c3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.000+0000 7f576c005740 1 --2- 172.21.15.16:0/735792456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c521e070 0x55a6c55c32c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.000+0000 7f576c005740 1 -- 172.21.15.16:0/735792456 >> 172.21.15.16:0/735792456 conn(0x55a6c5212460 msgr2=0x55a6c55c11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.000+0000 7f576c005740 1 -- 172.21.15.16:0/735792456 shutdown_connections 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.000+0000 7f576c005740 1 -- 172.21.15.16:0/735792456 wait complete. 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.001+0000 7f576c005740 1 -- start start 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.001+0000 7f576c005740 1 Processor -- start 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f576c005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51f51b0 0x55a6c55ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f576c005740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f576c005740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c521e070 0x55a6c55dad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f576c005740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a6c538c700 con 0x55a6c51fd390 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f576c005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a6c54b0700 con 0x55a6c51f51b0 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f576c005740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a6c55d1aa0 con 0x55a6c521e070 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f576c005740 10 client.0 _ll_register_callbacks cb 0x55a6c54dfdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f5756ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f575f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51f51b0 0x55a6c55ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f575f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51f51b0 0x55a6c55ce9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60460/0 (socket says 172.21.15.16:60460) 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f5756ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39410/0 (socket says 172.21.15.16:39410) 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f575f7fe640 1 -- 172.21.15.16:0/4212687807 learned_addr learned my addr 172.21.15.16:0/4212687807 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f575ffff640 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c521e070 0x55a6c55dad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f575f7fe640 1 -- 172.21.15.16:0/4212687807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c521e070 msgr2=0x55a6c55dad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f575f7fe640 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c521e070 0x55a6c55dad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f575f7fe640 1 -- 172.21.15.16:0/4212687807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 msgr2=0x55a6c55cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f575f7fe640 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f575f7fe640 1 -- 172.21.15.16:0/4212687807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6c54e98b0 con 0x55a6c51f51b0 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f5756ffd640 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f575f7fe640 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51f51b0 0x55a6c55ce9e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f575000cde0 tx=0x7f57500039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3e7dffa48a705b4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f573effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.002+0000 7f573effd640 1 -- 172.21.15.16:0/4212687807 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5750011970 con 0x55a6c51f51b0 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f573effd640 1 -- 172.21.15.16:0/4212687807 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f5750011b10 con 0x55a6c51f51b0 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f576c005740 1 -- 172.21.15.16:0/4212687807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f575801cd90 con 0x55a6c51f51b0 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f573effd640 1 -- 172.21.15.16:0/4212687807 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f5750011d10 con 0x55a6c51f51b0 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f576c005740 10 client.26870 fetch_fsmap learned FSMap version 44 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f576c005740 1 -- 172.21.15.16:0/4212687807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a6c55dd870 con 0x55a6c51f51b0 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f573effd640 1 -- 172.21.15.16:0/4212687807 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f575801cd90 con 0x55a6c51f51b0 2024-08-28T19:27:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f573effd640 1 -- 172.21.15.16:0/4212687807 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 44) ==== 8374+0+0 (secure 0 0 0) 0x7f5750018070 con 0x55a6c51f51b0 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f576c005740 10 client.26870 fetch_fsmap finished waiting for FSMap version 44 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f576c005740 10 client.26870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f576c005740 20 client.26870 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f576c005740 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6c51dc9f0 0x55a6c55e0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f576c005740 4 client.26870 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f576c005740 1 -- 172.21.15.16:0/4212687807 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6c553c580 con 0x55a6c51dc9f0 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.003+0000 7f5756ffd640 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6c51dc9f0 0x55a6c55e0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.004+0000 7f573effd640 10 client.26870 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.004+0000 7f5756ffd640 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6c51dc9f0 0x55a6c55e0df0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d744e5307b357ed2 server_cookie=bd27089c26b361f4 in_seq=0 out_seq=0 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f573effd640 1 -- 172.21.15.16:0/4212687807 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a6c553c580 con 0x55a6c51dc9f0 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f573effd640 10 client.26870 handle_command_reply: tid=0 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 1 client.26870 shutdown 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 1 -- 172.21.15.16:0/4212687807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51f51b0 msgr2=0x55a6c55ce9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51f51b0 0x55a6c55ce9e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f575000cde0 tx=0x7f57500039c0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 1 -- 172.21.15.16:0/4212687807 shutdown_connections 2024-08-28T19:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6c51dc9f0 0x55a6c55e0df0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c521e070 0x55a6c55dad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f573effd640 0 client.26870 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 1 --2- 172.21.15.16:0/4212687807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51f51b0 0x55a6c55ce9e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 1 -- 172.21.15.16:0/4212687807 >> 172.21.15.16:0/4212687807 conn(0x55a6c5212460 msgr2=0x55a6c55c64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 1 -- 172.21.15.16:0/4212687807 shutdown_connections 2024-08-28T19:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 1 -- 172.21.15.16:0/4212687807 wait complete. 2024-08-28T19:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.005+0000 7f576c005740 20 client.26870 trim_cache size 0 max 16384 2024-08-28T19:27:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f576c005740 1 Processor -- start 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f576c005740 1 -- start start 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f576c005740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55c1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f576c005740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55dac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f576c005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c521e070 0x55a6c55db1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f576c005740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57601b0c00 con 0x55a6c51fd390 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f576c005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57601b2770 con 0x55a6c521e070 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f576c005740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f57601b12d0 con 0x55a6c51f51b0 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f5756ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f5756ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55dac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39414/0 (socket says 172.21.15.16:39414) 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f5756ffd640 1 -- 172.21.15.16:0/3201991995 learned_addr learned my addr 172.21.15.16:0/3201991995 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f575f7fe640 1 --2- 172.21.15.16:0/3201991995 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55c1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f5756ffd640 1 -- 172.21.15.16:0/3201991995 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 msgr2=0x55a6c55c1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f5756ffd640 1 --2- 172.21.15.16:0/3201991995 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55c1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f5756ffd640 1 -- 172.21.15.16:0/3201991995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c521e070 msgr2=0x55a6c55db1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f5756ffd640 1 --2- 172.21.15.16:0/3201991995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c521e070 0x55a6c55db1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f5756ffd640 1 -- 172.21.15.16:0/3201991995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6c55cd940 con 0x55a6c51fd390 2024-08-28T19:27:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.023+0000 7f5756ffd640 1 --2- 172.21.15.16:0/3201991995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55dac90 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f5748103550 tx=0x7f57481085d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41663a1a37496332 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.024+0000 7f57557fa640 1 -- 172.21.15.16:0/3201991995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5748049040 con 0x55a6c51fd390 2024-08-28T19:27:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.024+0000 7f57557fa640 1 -- 172.21.15.16:0/3201991995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57480d50a0 con 0x55a6c51fd390 2024-08-28T19:27:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.024+0000 7f57557fa640 1 -- 172.21.15.16:0/3201991995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57480d0cb0 con 0x55a6c51fd390 2024-08-28T19:27:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.024+0000 7f576c005740 1 -- 172.21.15.16:0/3201991995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 msgr2=0x55a6c55dac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.024+0000 7f576c005740 1 --2- 172.21.15.16:0/3201991995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55dac90 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f5748103550 tx=0x7f57481085d0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.024+0000 7f576c005740 1 -- 172.21.15.16:0/3201991995 shutdown_connections 2024-08-28T19:27:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.024+0000 7f576c005740 1 --2- 172.21.15.16:0/3201991995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c521e070 0x55a6c55db1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.024+0000 7f576c005740 1 --2- 172.21.15.16:0/3201991995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c51fd390 0x55a6c55dac90 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.024+0000 7f576c005740 1 --2- 172.21.15.16:0/3201991995 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55c1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.024+0000 7f576c005740 1 -- 172.21.15.16:0/3201991995 >> 172.21.15.16:0/3201991995 conn(0x55a6c5212460 msgr2=0x55a6c55c0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.025+0000 7f576c005740 1 -- 172.21.15.16:0/3201991995 shutdown_connections 2024-08-28T19:27:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.025+0000 7f576c005740 1 -- 172.21.15.16:0/3201991995 wait complete. 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.025+0000 7f576c005740 1 -- start start 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.025+0000 7f576c005740 1 Processor -- start 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.025+0000 7f576c005740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.025+0000 7f576c005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51fd390 0x55a6c55d3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.025+0000 7f576c005740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c521e070 0x55a6c564a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.025+0000 7f576c005740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57601b20a0 con 0x55a6c521e070 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.025+0000 7f576c005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57601b19a0 con 0x55a6c51fd390 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.025+0000 7f576c005740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f57601aebe0 con 0x55a6c51f51b0 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f576c005740 10 client.0 _ll_register_callbacks cb 0x7f5750004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f575ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c521e070 0x55a6c564a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f575ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c521e070 0x55a6c564a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39422/0 (socket says 172.21.15.16:39422) 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f575ffff640 1 -- 172.21.15.16:0/3203699865 learned_addr learned my addr 172.21.15.16:0/3203699865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f575f7fe640 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f5756ffd640 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51fd390 0x55a6c55d3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f575ffff640 1 -- 172.21.15.16:0/3203699865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 msgr2=0x55a6c55d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f575ffff640 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55d2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f575ffff640 1 -- 172.21.15.16:0/3203699865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51fd390 msgr2=0x55a6c55d3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f575ffff640 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51fd390 0x55a6c55d3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f575ffff640 1 -- 172.21.15.16:0/3203699865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6c55d3d80 con 0x55a6c521e070 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f575f7fe640 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55d2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f575ffff640 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c521e070 0x55a6c564a7b0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f5758011010 tx=0x7f5758011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576fbdaaf45cb4b7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f573effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f573effd640 1 -- 172.21.15.16:0/3203699865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758035060 con 0x55a6c521e070 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f573effd640 1 -- 172.21.15.16:0/3203699865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9402+0+0 (secure 0 0 0) 0x7f575801ca60 con 0x55a6c521e070 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.026+0000 7f576c005740 1 -- 172.21.15.16:0/3203699865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a6c55cf350 con 0x55a6c521e070 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.027+0000 7f573effd640 1 -- 172.21.15.16:0/3203699865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f575803c050 con 0x55a6c521e070 2024-08-28T19:27:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.027+0000 7f576c005740 10 client.17331 fetch_fsmap learned FSMap version 44 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.027+0000 7f576c005740 1 -- 172.21.15.16:0/3203699865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a6c564bd40 con 0x55a6c521e070 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.027+0000 7f573effd640 1 -- 172.21.15.16:0/3203699865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a6c55cf350 con 0x55a6c521e070 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.027+0000 7f573effd640 1 -- 172.21.15.16:0/3203699865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 44) ==== 8374+0+0 (secure 0 0 0) 0x7f575802f070 con 0x55a6c521e070 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f576c005740 10 client.17331 fetch_fsmap finished waiting for FSMap version 44 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f576c005740 10 client.17331 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f576c005740 20 client.17331 populate_metadata read hostname 'smithi016' 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f576c005740 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6c51b1da0 0x55a6c564e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f576c005740 4 client.17331 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f576c005740 1 -- 172.21.15.16:0/3203699865 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a6c5538020 con 0x55a6c51b1da0 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f575f7fe640 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6c51b1da0 0x55a6c564e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f575f7fe640 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6c51b1da0 0x55a6c564e160 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55098ca177baa902 server_cookie=398a59bfedebefd2 in_seq=0 out_seq=0 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f573effd640 10 client.17331 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f573effd640 1 -- 172.21.15.16:0/3203699865 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3126 (crc 0 0 0) 0x55a6c5538020 con 0x55a6c51b1da0 2024-08-28T19:27:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f573effd640 10 client.17331 handle_command_reply: tid=0 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.028+0000 7f576c005740 1 client.17331 shutdown 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f576c005740 1 -- 172.21.15.16:0/3203699865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c521e070 msgr2=0x55a6c564a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f576c005740 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c521e070 0x55a6c564a7b0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f5758011010 tx=0x7f5758011f80 comp rx=0 tx=0).stop 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f576c005740 1 -- 172.21.15.16:0/3203699865 shutdown_connections 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f576c005740 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6c521e070 0x55a6c564a7b0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f576c005740 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6c51fd390 0x55a6c55d3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f576c005740 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6c51b1da0 0x55a6c564e160 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f576c005740 1 --2- 172.21.15.16:0/3203699865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6c51f51b0 0x55a6c55d2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f576c005740 1 -- 172.21.15.16:0/3203699865 >> 172.21.15.16:0/3203699865 conn(0x55a6c5212460 msgr2=0x55a6c55c63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f573effd640 0 client.17331 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f576c005740 1 -- 172.21.15.16:0/3203699865 shutdown_connections 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f576c005740 1 -- 172.21.15.16:0/3203699865 wait complete. 2024-08-28T19:27:43.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f576c005740 20 client.17331 trim_cache size 0 max 16384 2024-08-28T19:27:43.035 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:27:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:27:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:27:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:27:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "38d6d63f-9bab-468d-8eac-25cd6efa2607", 2024-08-28T19:27:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:27:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:27:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:27:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:27:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:27:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:27:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:27:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:27:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:27:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:27:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:27:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:27:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:27:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:27:43.041 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:27:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f5765a90640 1 -- 172.21.15.16:0/370182238 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f574c0b4ca0 msgr2=0x7f574c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f5765a90640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f574c0b4ca0 0x7f574c0b71b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f575801f000 tx=0x7f5758022000 comp rx=0 tx=0).stop 2024-08-28T19:27:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f5765a90640 1 -- 172.21.15.16:0/370182238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57601081c0 msgr2=0x7f576019b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:27:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f5765a90640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57601081c0 0x7f576019b650 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f574802a930 tx=0x7f5748004270 comp rx=0 tx=0).stop 2024-08-28T19:27:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.029+0000 7f5765a90640 1 -- 172.21.15.16:0/370182238 shutdown_connections 2024-08-28T19:27:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.030+0000 7f5765a90640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f574c0b4ca0 0x7f574c0b71b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.030+0000 7f5765a90640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5760108b10 0x7f576019bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.030+0000 7f5765a90640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57601081c0 0x7f576019b650 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.030+0000 7f5765a90640 1 --2- 172.21.15.16:0/370182238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5760102880 0x7f576019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:27:43.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.030+0000 7f5765a90640 1 -- 172.21.15.16:0/370182238 >> 172.21.15.16:0/370182238 conn(0x7f5760079510 msgr2=0x7f57601062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:27:43.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.030+0000 7f5765a90640 1 -- 172.21.15.16:0/370182238 shutdown_connections 2024-08-28T19:27:43.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:27:43.030+0000 7f5765a90640 1 -- 172.21.15.16:0/370182238 wait complete. 2024-08-28T19:27:43.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:43 smithi115 ceph-mon[22184]: pgmap v539: 129 pgs: 129 active+clean; 195 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 234 op/s 2024-08-28T19:27:43.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:43 smithi016 ceph-mon[18556]: pgmap v539: 129 pgs: 129 active+clean; 195 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 234 op/s 2024-08-28T19:27:43.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:43 smithi028 ceph-mon[21769]: pgmap v539: 129 pgs: 129 active+clean; 195 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 234 op/s 2024-08-28T19:27:43.816 INFO:tasks.cephfs.filesystem:scrub status for tag:38d6d63f-9bab-468d-8eac-25cd6efa2607 - {'path': '/', 'tag': '38d6d63f-9bab-468d-8eac-25cd6efa2607', 'options': 'recursive,force'} 2024-08-28T19:27:43.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 316492 4.84 MB/sec execute 203 sec latency 1293.164 ms 2024-08-28T19:27:44.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:44 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:44.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:44 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:44 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:44.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318463 4.87 MB/sec execute 204 sec latency 15.804 ms 2024-08-28T19:27:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:45 smithi115 ceph-mon[22184]: pgmap v540: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.3 MiB/s wr, 216 op/s 2024-08-28T19:27:45.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:45 smithi016 ceph-mon[18556]: pgmap v540: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.3 MiB/s wr, 216 op/s 2024-08-28T19:27:45.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:45 smithi028 ceph-mon[21769]: pgmap v540: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.3 MiB/s wr, 216 op/s 2024-08-28T19:27:45.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 319866 4.88 MB/sec execute 205 sec latency 10.459 ms 2024-08-28T19:27:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:46 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:46.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:46 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:46.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:46 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:46.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 321745 4.90 MB/sec execute 206 sec latency 16.517 ms 2024-08-28T19:27:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:47 smithi115 ceph-mon[22184]: pgmap v541: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 283 op/s 2024-08-28T19:27:47.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:47 smithi016 ceph-mon[18556]: pgmap v541: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 283 op/s 2024-08-28T19:27:47.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:47 smithi028 ceph-mon[21769]: pgmap v541: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 283 op/s 2024-08-28T19:27:47.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 323233 4.93 MB/sec execute 207 sec latency 54.097 ms 2024-08-28T19:27:48.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:48 smithi016 ceph-mon[18556]: pgmap v542: 129 pgs: 129 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 227 op/s 2024-08-28T19:27:48.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:48 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:27:48.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:48 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:48.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:48 smithi028 ceph-mon[21769]: pgmap v542: 129 pgs: 129 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 227 op/s 2024-08-28T19:27:48.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:48 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:27:48.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:48 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:48 smithi115 ceph-mon[22184]: pgmap v542: 129 pgs: 129 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 227 op/s 2024-08-28T19:27:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:48 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:27:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:48 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:27:48.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 324487 4.93 MB/sec execute 208 sec latency 187.489 ms 2024-08-28T19:27:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:27:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:27:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:27:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:27:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:27:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:27:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:27:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:27:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:27:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:27:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:27:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:27:49.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 326416 4.96 MB/sec execute 209 sec latency 319.431 ms 2024-08-28T19:27:50.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:50 smithi016 ceph-mon[18556]: pgmap v543: 129 pgs: 129 active+clean; 204 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 219 op/s 2024-08-28T19:27:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:50 smithi028 ceph-mon[21769]: pgmap v543: 129 pgs: 129 active+clean; 204 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 219 op/s 2024-08-28T19:27:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:50 smithi115 ceph-mon[22184]: pgmap v543: 129 pgs: 129 active+clean; 204 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 219 op/s 2024-08-28T19:27:50.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 327441 4.96 MB/sec execute 210 sec latency 72.568 ms 2024-08-28T19:27:51.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 328029 4.95 MB/sec execute 211 sec latency 528.674 ms 2024-08-28T19:27:52.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 330114 4.99 MB/sec execute 212 sec latency 549.671 ms 2024-08-28T19:27:53.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:52 smithi016 ceph-mon[18556]: pgmap v544: 129 pgs: 129 active+clean; 209 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 288 op/s 2024-08-28T19:27:53.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:52 smithi028 ceph-mon[21769]: pgmap v544: 129 pgs: 129 active+clean; 209 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 288 op/s 2024-08-28T19:27:53.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:52 smithi115 ceph-mon[22184]: pgmap v544: 129 pgs: 129 active+clean; 209 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 288 op/s 2024-08-28T19:27:53.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 331355 4.99 MB/sec execute 213 sec latency 14.698 ms 2024-08-28T19:27:54.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 331577 4.97 MB/sec execute 214 sec latency 772.700 ms 2024-08-28T19:27:55.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:54 smithi016 ceph-mon[18556]: pgmap v545: 129 pgs: 129 active+clean; 209 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 253 op/s 2024-08-28T19:27:55.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:54 smithi028 ceph-mon[21769]: pgmap v545: 129 pgs: 129 active+clean; 209 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 253 op/s 2024-08-28T19:27:55.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:54 smithi115 ceph-mon[22184]: pgmap v545: 129 pgs: 129 active+clean; 209 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 253 op/s 2024-08-28T19:27:55.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 331577 4.95 MB/sec execute 215 sec latency 1772.778 ms 2024-08-28T19:27:56.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 332341 4.93 MB/sec execute 216 sec latency 2421.282 ms 2024-08-28T19:27:57.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:56 smithi016 ceph-mon[18556]: pgmap v546: 129 pgs: 129 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 283 op/s 2024-08-28T19:27:57.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:56 smithi028 ceph-mon[21769]: pgmap v546: 129 pgs: 129 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 283 op/s 2024-08-28T19:27:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:56 smithi115 ceph-mon[22184]: pgmap v546: 129 pgs: 129 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 283 op/s 2024-08-28T19:27:57.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 333871 4.96 MB/sec execute 217 sec latency 44.353 ms 2024-08-28T19:27:58.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 335120 4.97 MB/sec execute 218 sec latency 180.745 ms 2024-08-28T19:27:59.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:27:58 smithi016 ceph-mon[18556]: pgmap v547: 129 pgs: 129 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 218 op/s 2024-08-28T19:27:59.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:27:58 smithi028 ceph-mon[21769]: pgmap v547: 129 pgs: 129 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 218 op/s 2024-08-28T19:27:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:27:58 smithi115 ceph-mon[22184]: pgmap v547: 129 pgs: 129 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 218 op/s 2024-08-28T19:27:59.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 335120 4.94 MB/sec execute 219 sec latency 1180.831 ms 2024-08-28T19:28:00.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 335120 4.92 MB/sec execute 220 sec latency 2180.923 ms 2024-08-28T19:28:01.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:00 smithi016 ceph-mon[18556]: pgmap v548: 129 pgs: 129 active+clean; 213 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-08-28T19:28:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:00 smithi028 ceph-mon[21769]: pgmap v548: 129 pgs: 129 active+clean; 213 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-08-28T19:28:01.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:00 smithi115 ceph-mon[22184]: pgmap v548: 129 pgs: 129 active+clean; 213 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-08-28T19:28:01.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 335120 4.90 MB/sec execute 221 sec latency 3180.998 ms 2024-08-28T19:28:02.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 335882 4.88 MB/sec execute 222 sec latency 3198.415 ms 2024-08-28T19:28:03.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:02 smithi016 ceph-mon[18556]: pgmap v549: 129 pgs: 129 active+clean; 215 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 239 op/s 2024-08-28T19:28:03.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:02 smithi028 ceph-mon[21769]: pgmap v549: 129 pgs: 129 active+clean; 215 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 239 op/s 2024-08-28T19:28:03.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:02 smithi115 ceph-mon[22184]: pgmap v549: 129 pgs: 129 active+clean; 215 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 239 op/s 2024-08-28T19:28:03.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 337363 4.92 MB/sec execute 223 sec latency 43.318 ms 2024-08-28T19:28:04.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 338670 4.92 MB/sec execute 224 sec latency 85.986 ms 2024-08-28T19:28:05.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:04 smithi016 ceph-mon[18556]: pgmap v550: 129 pgs: 129 active+clean; 215 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-08-28T19:28:05.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:04 smithi016 ceph-mon[18556]: osdmap e85: 12 total, 12 up, 12 in 2024-08-28T19:28:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:04 smithi028 ceph-mon[21769]: pgmap v550: 129 pgs: 129 active+clean; 215 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-08-28T19:28:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:04 smithi028 ceph-mon[21769]: osdmap e85: 12 total, 12 up, 12 in 2024-08-28T19:28:05.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:04 smithi115 ceph-mon[22184]: pgmap v550: 129 pgs: 129 active+clean; 215 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-08-28T19:28:05.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:04 smithi115 ceph-mon[22184]: osdmap e85: 12 total, 12 up, 12 in 2024-08-28T19:28:05.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 338670 4.90 MB/sec execute 225 sec latency 1086.101 ms 2024-08-28T19:28:06.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 338670 4.87 MB/sec execute 226 sec latency 2086.188 ms 2024-08-28T19:28:07.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:06 smithi016 ceph-mon[18556]: pgmap v552: 129 pgs: 129 active+clean; 219 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-08-28T19:28:07.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:06 smithi028 ceph-mon[21769]: pgmap v552: 129 pgs: 129 active+clean; 219 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-08-28T19:28:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:06 smithi115 ceph-mon[22184]: pgmap v552: 129 pgs: 129 active+clean; 219 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-08-28T19:28:07.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 340665 4.91 MB/sec execute 227 sec latency 2105.789 ms 2024-08-28T19:28:08.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 341706 4.90 MB/sec execute 228 sec latency 43.976 ms 2024-08-28T19:28:09.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:08 smithi016 ceph-mon[18556]: pgmap v553: 129 pgs: 129 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 214 op/s 2024-08-28T19:28:09.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:08 smithi028 ceph-mon[21769]: pgmap v553: 129 pgs: 129 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 214 op/s 2024-08-28T19:28:09.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:08 smithi115 ceph-mon[22184]: pgmap v553: 129 pgs: 129 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 214 op/s 2024-08-28T19:28:09.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 342210 4.89 MB/sec execute 229 sec latency 728.048 ms 2024-08-28T19:28:10.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 342210 4.87 MB/sec execute 230 sec latency 1728.131 ms 2024-08-28T19:28:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:11 smithi016 ceph-mon[18556]: pgmap v554: 129 pgs: 129 active+clean; 217 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2024-08-28T19:28:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:11 smithi016 ceph-mon[18556]: osdmap e86: 12 total, 12 up, 12 in 2024-08-28T19:28:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:11 smithi028 ceph-mon[21769]: pgmap v554: 129 pgs: 129 active+clean; 217 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2024-08-28T19:28:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:11 smithi028 ceph-mon[21769]: osdmap e86: 12 total, 12 up, 12 in 2024-08-28T19:28:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:11 smithi115 ceph-mon[22184]: pgmap v554: 129 pgs: 129 active+clean; 217 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2024-08-28T19:28:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:11 smithi115 ceph-mon[22184]: osdmap e86: 12 total, 12 up, 12 in 2024-08-28T19:28:11.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 342210 4.85 MB/sec execute 231 sec latency 2728.230 ms 2024-08-28T19:28:12.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 344204 4.88 MB/sec execute 232 sec latency 2745.474 ms 2024-08-28T19:28:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:13 smithi016 ceph-mon[18556]: pgmap v556: 129 pgs: 129 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 262 op/s 2024-08-28T19:28:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:13 smithi028 ceph-mon[21769]: pgmap v556: 129 pgs: 129 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 262 op/s 2024-08-28T19:28:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:13 smithi115 ceph-mon[22184]: pgmap v556: 129 pgs: 129 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 262 op/s 2024-08-28T19:28:13.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:28:13.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345222 4.88 MB/sec execute 233 sec latency 45.510 ms 2024-08-28T19:28:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:28:14.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.763+0000 7f18cf2a8640 1 -- 172.21.15.16:0/2329059952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c8102880 msgr2=0x7f18c8102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:14.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.763+0000 7f18cf2a8640 1 --2- 172.21.15.16:0/2329059952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c8102880 0x7f18c8102c90 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f18c0009900 tx=0x7f18c002a410 comp rx=0 tx=0).stop 2024-08-28T19:28:14.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.763+0000 7f18cf2a8640 1 -- 172.21.15.16:0/2329059952 shutdown_connections 2024-08-28T19:28:14.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.763+0000 7f18cf2a8640 1 --2- 172.21.15.16:0/2329059952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8108ac0 0x7f18c810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.763+0000 7f18cf2a8640 1 --2- 172.21.15.16:0/2329059952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18c8108170 0x7f18c8108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.763+0000 7f18cf2a8640 1 --2- 172.21.15.16:0/2329059952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c8102880 0x7f18c8102c90 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.763+0000 7f18cf2a8640 1 -- 172.21.15.16:0/2329059952 >> 172.21.15.16:0/2329059952 conn(0x7f18c8079500 msgr2=0x7f18c8079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:14.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.763+0000 7f18cf2a8640 1 -- 172.21.15.16:0/2329059952 shutdown_connections 2024-08-28T19:28:14.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.763+0000 7f18cf2a8640 1 -- 172.21.15.16:0/2329059952 wait complete. 2024-08-28T19:28:14.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.764+0000 7f18cf2a8640 1 Processor -- start 2024-08-28T19:28:14.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.764+0000 7f18cf2a8640 1 -- start start 2024-08-28T19:28:14.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.764+0000 7f18cf2a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c8102880 0x7f18c819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.764+0000 7f18cf2a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8108170 0x7f18c819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cf2a8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18c8108ac0 0x7f18c819fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cf2a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18c8112720 con 0x7f18c8102880 2024-08-28T19:28:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cf2a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81125a0 con 0x7f18c8108170 2024-08-28T19:28:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cf2a8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81128a0 con 0x7f18c8108ac0 2024-08-28T19:28:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cd044640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c8102880 0x7f18c819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cd044640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c8102880 0x7f18c819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41546/0 (socket says 172.21.15.16:41546) 2024-08-28T19:28:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cd044640 1 -- 172.21.15.16:0/4002982784 learned_addr learned my addr 172.21.15.16:0/4002982784 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cc843640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8108170 0x7f18c819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cd845640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18c8108ac0 0x7f18c819fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cd044640 1 -- 172.21.15.16:0/4002982784 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18c8108ac0 msgr2=0x7f18c819fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cd044640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18c8108ac0 0x7f18c819fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cd044640 1 -- 172.21.15.16:0/4002982784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8108170 msgr2=0x7f18c819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cd044640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8108170 0x7f18c819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cd044640 1 -- 172.21.15.16:0/4002982784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c81a0170 con 0x7f18c8102880 2024-08-28T19:28:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cd845640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18c8108ac0 0x7f18c819fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:28:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cd044640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c8102880 0x7f18c819a200 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f18c0005940 tx=0x7f18c0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18c67fc640 1 -- 172.21.15.16:0/4002982784 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c003f030 con 0x7f18c8102880 2024-08-28T19:28:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18cf2a8640 1 -- 172.21.15.16:0/4002982784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c81a0400 con 0x7f18c8102880 2024-08-28T19:28:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.765+0000 7f18c67fc640 1 -- 172.21.15.16:0/4002982784 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18c0003aa0 con 0x7f18c8102880 2024-08-28T19:28:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.766+0000 7f18c67fc640 1 -- 172.21.15.16:0/4002982784 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c0035070 con 0x7f18c8102880 2024-08-28T19:28:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.766+0000 7f18cf2a8640 1 -- 172.21.15.16:0/4002982784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c8103f00 con 0x7f18c8102880 2024-08-28T19:28:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.768+0000 7f18d581d740 1 Processor -- start 2024-08-28T19:28:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.768+0000 7f18c67fc640 1 -- 172.21.15.16:0/4002982784 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f18c0042030 con 0x7f18c8102880 2024-08-28T19:28:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.768+0000 7f18d581d740 1 -- start start 2024-08-28T19:28:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18d581d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2e12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18d581d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2e1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18d581d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef1b390 0x55995f2e1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18d581d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55995f258630 con 0x55995ef3c070 2024-08-28T19:28:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18d581d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55995f247bc0 con 0x55995ef131b0 2024-08-28T19:28:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18d581d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55995f24bb70 con 0x55995ef1b390 2024-08-28T19:28:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18cd845640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2e12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18cd845640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2e12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41558/0 (socket says 172.21.15.16:41558) 2024-08-28T19:28:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18cd845640 1 -- 172.21.15.16:0/1594198797 learned_addr learned my addr 172.21.15.16:0/1594198797 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18cc843640 1 --2- 172.21.15.16:0/1594198797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef1b390 0x55995f2e1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18cd044640 1 --2- 172.21.15.16:0/1594198797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2e1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18cd845640 1 -- 172.21.15.16:0/1594198797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef1b390 msgr2=0x55995f2e1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18cd845640 1 --2- 172.21.15.16:0/1594198797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef1b390 0x55995f2e1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18cd845640 1 -- 172.21.15.16:0/1594198797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 msgr2=0x55995f2e1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18cd845640 1 --2- 172.21.15.16:0/1594198797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2e1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.769+0000 7f18cd845640 1 -- 172.21.15.16:0/1594198797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55995efa7650 con 0x55995ef3c070 2024-08-28T19:28:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.770+0000 7f18c67fc640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18b00b4ca0 0x7f18b00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.770+0000 7f18cd044640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18b00b4ca0 0x7f18b00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.770+0000 7f18c67fc640 1 -- 172.21.15.16:0/4002982784 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f18c0106020 con 0x7f18c8102880 2024-08-28T19:28:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.770+0000 7f18cd845640 1 --2- 172.21.15.16:0/1594198797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2e12c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f18b8010450 tx=0x7f18b8010910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43786ad21facec4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.770+0000 7f18af7fe640 1 -- 172.21.15.16:0/1594198797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b8017070 con 0x55995ef3c070 2024-08-28T19:28:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.770+0000 7f18af7fe640 1 -- 172.21.15.16:0/1594198797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18b8011c10 con 0x55995ef3c070 2024-08-28T19:28:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.771+0000 7f18af7fe640 1 -- 172.21.15.16:0/1594198797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b801ac50 con 0x55995ef3c070 2024-08-28T19:28:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.771+0000 7f18d581d740 1 -- 172.21.15.16:0/1594198797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 msgr2=0x55995f2e12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.771+0000 7f18cd044640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18b00b4ca0 0x7f18b00b71b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f18c0103450 tx=0x7f18c002a410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.771+0000 7f18d581d740 1 --2- 172.21.15.16:0/1594198797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2e12c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f18b8010450 tx=0x7f18b8010910 comp rx=0 tx=0).stop 2024-08-28T19:28:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.772+0000 7f18d581d740 1 -- 172.21.15.16:0/1594198797 shutdown_connections 2024-08-28T19:28:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.772+0000 7f18d581d740 1 --2- 172.21.15.16:0/1594198797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef1b390 0x55995f2e1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.772+0000 7f18d581d740 1 --2- 172.21.15.16:0/1594198797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2e1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.772+0000 7f18d581d740 1 --2- 172.21.15.16:0/1594198797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2e12c0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.772+0000 7f18d581d740 1 -- 172.21.15.16:0/1594198797 >> 172.21.15.16:0/1594198797 conn(0x55995ef30460 msgr2=0x55995f2df1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.772+0000 7f18d581d740 1 -- 172.21.15.16:0/1594198797 shutdown_connections 2024-08-28T19:28:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.772+0000 7f18d581d740 1 -- 172.21.15.16:0/1594198797 wait complete. 2024-08-28T19:28:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.772+0000 7f18d581d740 1 -- start start 2024-08-28T19:28:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.772+0000 7f18d581d740 1 Processor -- start 2024-08-28T19:28:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18d581d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2f5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18d581d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef1b390 0x55995f2f5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18d581d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18d581d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55995f0aa700 con 0x55995ef3c070 2024-08-28T19:28:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18d581d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55995f1ce700 con 0x55995ef131b0 2024-08-28T19:28:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18d581d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55995f2f8220 con 0x55995ef1b390 2024-08-28T19:28:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18d581d740 10 client.0 _ll_register_callbacks cb 0x55995f1fddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18cd845640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18cc843640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef1b390 0x55995f2f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18cd845640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2f5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41568/0 (socket says 172.21.15.16:41568) 2024-08-28T19:28:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18cd845640 1 -- 172.21.15.16:0/2087321903 learned_addr learned my addr 172.21.15.16:0/2087321903 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:14.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18cd845640 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18cd845640 1 -- 172.21.15.16:0/2087321903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef1b390 msgr2=0x55995f2f5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:14.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18cd845640 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef1b390 0x55995f2f5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18cd845640 1 -- 172.21.15.16:0/2087321903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 msgr2=0x55995f2f5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:14.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18cd845640 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2f5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18cd845640 1 -- 172.21.15.16:0/2087321903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55995f2078b0 con 0x55995ef3c070 2024-08-28T19:28:14.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.773+0000 7f18cc843640 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef1b390 0x55995f2f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:28:14.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.774+0000 7f18cd845640 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2f5bc0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f18b801a6b0 tx=0x7f18b801a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88306ebe26e9544 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:14.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.774+0000 7f18acff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:28:14.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.774+0000 7f18acff9640 1 -- 172.21.15.16:0/2087321903 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b802e060 con 0x55995ef3c070 2024-08-28T19:28:14.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.774+0000 7f18acff9640 1 -- 172.21.15.16:0/2087321903 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f18b8011770 con 0x55995ef3c070 2024-08-28T19:28:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.774+0000 7f18d581d740 1 -- 172.21.15.16:0/2087321903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18b8011c10 con 0x55995ef3c070 2024-08-28T19:28:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.774+0000 7f18acff9640 1 -- 172.21.15.16:0/2087321903 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f18b8017030 con 0x55995ef3c070 2024-08-28T19:28:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.774+0000 7f18d581d740 10 client.17343 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.774+0000 7f18d581d740 1 -- 172.21.15.16:0/2087321903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55995f303f50 con 0x55995ef3c070 2024-08-28T19:28:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.774+0000 7f18acff9640 1 -- 172.21.15.16:0/2087321903 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18b8011c10 con 0x55995ef3c070 2024-08-28T19:28:14.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.774+0000 7f18acff9640 1 -- 172.21.15.16:0/2087321903 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f18b8023030 con 0x55995ef3c070 2024-08-28T19:28:14.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.775+0000 7f18d581d740 10 client.17343 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:14.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.775+0000 7f18d581d740 10 client.17343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:28:14.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.775+0000 7f18d581d740 20 client.17343 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:14.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.775+0000 7f18d581d740 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55995eefa9f0 0x55995f3073c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.775+0000 7f18d581d740 4 client.17343 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:28:14.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.775+0000 7f18d581d740 1 -- 172.21.15.16:0/2087321903 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55995f25a580 con 0x55995eefa9f0 2024-08-28T19:28:14.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.775+0000 7f18cc843640 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55995eefa9f0 0x55995f3073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.775+0000 7f18acff9640 10 client.17343 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.775+0000 7f18cc843640 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55995eefa9f0 0x55995f3073c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fae4994f808bd2f8 server_cookie=af051564b16d5d44 in_seq=0 out_seq=0 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18acff9640 1 -- 172.21.15.16:0/2087321903 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55995f25a580 con 0x55995eefa9f0 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18acff9640 10 client.17343 handle_command_reply: tid=0 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18d581d740 1 client.17343 shutdown 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18d581d740 1 -- 172.21.15.16:0/2087321903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 msgr2=0x55995f2f5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18d581d740 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2f5bc0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f18b801a6b0 tx=0x7f18b801a710 comp rx=0 tx=0).stop 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18d581d740 1 -- 172.21.15.16:0/2087321903 shutdown_connections 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18d581d740 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef3c070 0x55995f2f5bc0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18d581d740 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55995eefa9f0 0x55995f3073c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18d581d740 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef1b390 0x55995f2f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18d581d740 1 --2- 172.21.15.16:0/2087321903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18acff9640 0 client.17343 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:14.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.776+0000 7f18d581d740 1 -- 172.21.15.16:0/2087321903 >> 172.21.15.16:0/2087321903 conn(0x55995ef30460 msgr2=0x55995f2e4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:14.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.777+0000 7f18d581d740 1 -- 172.21.15.16:0/2087321903 shutdown_connections 2024-08-28T19:28:14.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.777+0000 7f18d581d740 1 -- 172.21.15.16:0/2087321903 wait complete. 2024-08-28T19:28:14.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.777+0000 7f18d581d740 20 client.17343 trim_cache size 0 max 16384 2024-08-28T19:28:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18d581d740 1 Processor -- start 2024-08-28T19:28:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18d581d740 1 -- start start 2024-08-28T19:28:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18d581d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18d581d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f300ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18d581d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef3c070 0x55995f2e2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18d581d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81aeab0 con 0x55995ef1b390 2024-08-28T19:28:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18d581d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81b06e0 con 0x55995ef131b0 2024-08-28T19:28:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18d581d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81af180 con 0x55995ef3c070 2024-08-28T19:28:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18cc843640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f300ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18cc843640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f300ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41574/0 (socket says 172.21.15.16:41574) 2024-08-28T19:28:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18cc843640 1 -- 172.21.15.16:0/3810170567 learned_addr learned my addr 172.21.15.16:0/3810170567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18cd845640 1 --2- 172.21.15.16:0/3810170567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18cd845640 1 --2- 172.21.15.16:0/3810170567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef3c070 0x55995f2e2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18cc843640 1 -- 172.21.15.16:0/3810170567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef3c070 msgr2=0x55995f2e2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18cc843640 1 --2- 172.21.15.16:0/3810170567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef3c070 0x55995f2e2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18cc843640 1 -- 172.21.15.16:0/3810170567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 msgr2=0x55995f2f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18cc843640 1 --2- 172.21.15.16:0/3810170567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2f70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18cc843640 1 -- 172.21.15.16:0/3810170567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55995f2f80b0 con 0x55995ef1b390 2024-08-28T19:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.796+0000 7f18cd845640 1 --2- 172.21.15.16:0/3810170567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef3c070 0x55995f2e2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.797+0000 7f18cc843640 1 --2- 172.21.15.16:0/3810170567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f300ec0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x55995f0aa7c0 tx=0x7f18bc00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8179d7eec6bcdaa server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.797+0000 7f18aeffd640 1 -- 172.21.15.16:0/3810170567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc005020 con 0x55995ef1b390 2024-08-28T19:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.797+0000 7f18aeffd640 1 -- 172.21.15.16:0/3810170567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18bc0047a0 con 0x55995ef1b390 2024-08-28T19:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.797+0000 7f18aeffd640 1 -- 172.21.15.16:0/3810170567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc024690 con 0x55995ef1b390 2024-08-28T19:28:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.798+0000 7f18d581d740 1 -- 172.21.15.16:0/3810170567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 msgr2=0x55995f300ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.798+0000 7f18d581d740 1 --2- 172.21.15.16:0/3810170567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f300ec0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x55995f0aa7c0 tx=0x7f18bc00b390 comp rx=0 tx=0).stop 2024-08-28T19:28:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.798+0000 7f18d581d740 1 -- 172.21.15.16:0/3810170567 shutdown_connections 2024-08-28T19:28:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.798+0000 7f18d581d740 1 --2- 172.21.15.16:0/3810170567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef3c070 0x55995f2e2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.798+0000 7f18d581d740 1 --2- 172.21.15.16:0/3810170567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f300ec0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.798+0000 7f18d581d740 1 --2- 172.21.15.16:0/3810170567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef131b0 0x55995f2f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.798+0000 7f18d581d740 1 -- 172.21.15.16:0/3810170567 >> 172.21.15.16:0/3810170567 conn(0x55995ef30460 msgr2=0x55995f2e22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.798+0000 7f18d581d740 1 -- 172.21.15.16:0/3810170567 shutdown_connections 2024-08-28T19:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.798+0000 7f18d581d740 1 -- 172.21.15.16:0/3810170567 wait complete. 2024-08-28T19:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.798+0000 7f18d581d740 1 -- start start 2024-08-28T19:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.798+0000 7f18d581d740 1 Processor -- start 2024-08-28T19:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18d581d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef131b0 0x55995f30acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18d581d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f3081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18d581d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef3c070 0x55995f3087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18d581d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81aff80 con 0x55995ef1b390 2024-08-28T19:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18d581d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81af850 con 0x55995ef3c070 2024-08-28T19:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18d581d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81acab0 con 0x55995ef131b0 2024-08-28T19:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18d581d740 10 client.0 _ll_register_callbacks cb 0x7f18bc00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cc843640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f3081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cc843640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f3081a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41576/0 (socket says 172.21.15.16:41576) 2024-08-28T19:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cd845640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef131b0 0x55995f30acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cc843640 1 -- 172.21.15.16:0/1779842114 learned_addr learned my addr 172.21.15.16:0/1779842114 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cd845640 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef3c070 0x55995f3087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cc843640 1 -- 172.21.15.16:0/1779842114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef131b0 msgr2=0x55995f30acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cc843640 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef131b0 0x55995f30acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cc843640 1 -- 172.21.15.16:0/1779842114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef3c070 msgr2=0x55995f3087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cc843640 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef3c070 0x55995f3087d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cc843640 1 -- 172.21.15.16:0/1779842114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55995f308f60 con 0x55995ef1b390 2024-08-28T19:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cd845640 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef131b0 0x55995f30acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18cc843640 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f3081a0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f18bc00ba80 tx=0x7f18bc00bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135437ffaf0043d5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.799+0000 7f18acff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18acff9640 1 -- 172.21.15.16:0/1779842114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc032020 con 0x55995ef1b390 2024-08-28T19:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18acff9640 1 -- 172.21.15.16:0/1779842114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f18bc005000 con 0x55995ef1b390 2024-08-28T19:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18d581d740 1 -- 172.21.15.16:0/1779842114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18bc0047a0 con 0x55995ef1b390 2024-08-28T19:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18acff9640 1 -- 172.21.15.16:0/1779842114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f18bc03d050 con 0x55995ef1b390 2024-08-28T19:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18d581d740 10 client.17349 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18d581d740 1 -- 172.21.15.16:0/1779842114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55995f371a70 con 0x55995ef1b390 2024-08-28T19:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18acff9640 1 -- 172.21.15.16:0/1779842114 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18bc0047a0 con 0x55995ef1b390 2024-08-28T19:28:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18acff9640 1 -- 172.21.15.16:0/1779842114 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f18bc024af0 con 0x55995ef1b390 2024-08-28T19:28:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18d581d740 10 client.17349 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18d581d740 10 client.17349 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:28:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18d581d740 20 client.17349 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18d581d740 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55995eecfda0 0x55995f3741d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18d581d740 4 client.17349 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:28:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.800+0000 7f18d581d740 1 -- 172.21.15.16:0/1779842114 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55995f256020 con 0x55995eecfda0 2024-08-28T19:28:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18cd845640 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55995eecfda0 0x55995f3741d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18acff9640 10 client.17349 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18cd845640 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55995eecfda0 0x55995f3741d0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f99dd4d9eb8ee732 server_cookie=c7a3d139896e4464 in_seq=0 out_seq=0 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18acff9640 1 -- 172.21.15.16:0/1779842114 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2930 (crc 0 0 0) 0x55995f256020 con 0x55995eecfda0 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18acff9640 10 client.17349 handle_command_reply: tid=0 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18d581d740 1 client.17349 shutdown 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18d581d740 1 -- 172.21.15.16:0/1779842114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 msgr2=0x55995f3081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18d581d740 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f3081a0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f18bc00ba80 tx=0x7f18bc00bce0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18d581d740 1 -- 172.21.15.16:0/1779842114 shutdown_connections 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18d581d740 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55995ef3c070 0x55995f3087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18d581d740 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55995ef1b390 0x55995f3081a0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18d581d740 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55995eecfda0 0x55995f3741d0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18d581d740 1 --2- 172.21.15.16:0/1779842114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55995ef131b0 0x55995f30acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18acff9640 0 client.17349 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.801+0000 7f18d581d740 1 -- 172.21.15.16:0/1779842114 >> 172.21.15.16:0/1779842114 conn(0x55995ef30460 msgr2=0x55995f300c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18d581d740 1 -- 172.21.15.16:0/1779842114 shutdown_connections 2024-08-28T19:28:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18d581d740 1 -- 172.21.15.16:0/1779842114 wait complete. 2024-08-28T19:28:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18d581d740 20 client.17349 trim_cache size 0 max 16384 2024-08-28T19:28:14.810 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:28:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:28:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:28:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:28:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:28:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:28:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:28:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:28:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:28:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:28:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:28:14.817 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:28:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18cf2a8640 1 -- 172.21.15.16:0/4002982784 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18b00b4ca0 msgr2=0x7f18b00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18cf2a8640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18b00b4ca0 0x7f18b00b71b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f18c0103450 tx=0x7f18c002a410 comp rx=0 tx=0).stop 2024-08-28T19:28:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18cf2a8640 1 -- 172.21.15.16:0/4002982784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c8102880 msgr2=0x7f18c819a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18cf2a8640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c8102880 0x7f18c819a200 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f18c0005940 tx=0x7f18c0004270 comp rx=0 tx=0).stop 2024-08-28T19:28:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18cf2a8640 1 -- 172.21.15.16:0/4002982784 shutdown_connections 2024-08-28T19:28:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18cf2a8640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18c8108ac0 0x7f18c819fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18cf2a8640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18b00b4ca0 0x7f18b00b71b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18cf2a8640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8108170 0x7f18c819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18cf2a8640 1 --2- 172.21.15.16:0/4002982784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c8102880 0x7f18c819a200 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:14.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18cf2a8640 1 -- 172.21.15.16:0/4002982784 >> 172.21.15.16:0/4002982784 conn(0x7f18c8079500 msgr2=0x7f18c81018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:14.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.802+0000 7f18cf2a8640 1 -- 172.21.15.16:0/4002982784 shutdown_connections 2024-08-28T19:28:14.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:14.803+0000 7f18cf2a8640 1 -- 172.21.15.16:0/4002982784 wait complete. 2024-08-28T19:28:14.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345759 4.87 MB/sec execute 234 sec latency 542.852 ms 2024-08-28T19:28:15.421 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:15 smithi016 ceph-mon[18556]: pgmap v557: 129 pgs: 129 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 220 op/s 2024-08-28T19:28:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:15 smithi028 ceph-mon[21769]: pgmap v557: 129 pgs: 129 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 220 op/s 2024-08-28T19:28:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:15 smithi115 ceph-mon[22184]: pgmap v557: 129 pgs: 129 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 220 op/s 2024-08-28T19:28:15.649 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:28:15.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:28:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:28:15.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345759 4.85 MB/sec execute 235 sec latency 1542.880 ms 2024-08-28T19:28:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.381+0000 7fe9d107e640 1 -- 172.21.15.16:0/940131735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc107100 msgr2=0x7fe9cc107510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.381+0000 7fe9d107e640 1 --2- 172.21.15.16:0/940131735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc107100 0x7fe9cc107510 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009900 tx=0x7fe9c002a410 comp rx=0 tx=0).stop 2024-08-28T19:28:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.381+0000 7fe9d107e640 1 -- 172.21.15.16:0/940131735 shutdown_connections 2024-08-28T19:28:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.381+0000 7fe9d107e640 1 --2- 172.21.15.16:0/940131735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc107a50 0x7fe9cc110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.381+0000 7fe9d107e640 1 --2- 172.21.15.16:0/940131735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc107100 0x7fe9cc107510 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.381+0000 7fe9d107e640 1 --2- 172.21.15.16:0/940131735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9cc101f00 0x7fe9cc102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.381+0000 7fe9d107e640 1 -- 172.21.15.16:0/940131735 >> 172.21.15.16:0/940131735 conn(0x7fe9cc079c40 msgr2=0x7fe9cc07a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.381+0000 7fe9d107e640 1 -- 172.21.15.16:0/940131735 shutdown_connections 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.381+0000 7fe9d107e640 1 -- 172.21.15.16:0/940131735 wait complete. 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.381+0000 7fe9d107e640 1 Processor -- start 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9d107e640 1 -- start start 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9d107e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101f00 0x7fe9cc19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9d107e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc107100 0x7fe9cc19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9d107e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9cc107a50 0x7fe9cc19fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9d107e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc113900 con 0x7fe9cc107100 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9d107e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc113780 con 0x7fe9cc101f00 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9d107e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc113a80 con 0x7fe9cc107a50 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9ca59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc107100 0x7fe9cc19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9ca59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc107100 0x7fe9cc19a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41592/0 (socket says 172.21.15.16:41592) 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9ca59c640 1 -- 172.21.15.16:0/2188801132 learned_addr learned my addr 172.21.15.16:0/2188801132 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9cad9d640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101f00 0x7fe9cc19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.382+0000 7fe9cb59e640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9cc107a50 0x7fe9cc19fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.383+0000 7fe9ca59c640 1 -- 172.21.15.16:0/2188801132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9cc107a50 msgr2=0x7fe9cc19fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.383+0000 7fe9ca59c640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9cc107a50 0x7fe9cc19fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.383+0000 7fe9ca59c640 1 -- 172.21.15.16:0/2188801132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101f00 msgr2=0x7fe9cc19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.383+0000 7fe9ca59c640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101f00 0x7fe9cc19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.383+0000 7fe9ca59c640 1 -- 172.21.15.16:0/2188801132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc1a0180 con 0x7fe9cc107100 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.383+0000 7fe9ca59c640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc107100 0x7fe9cc19a870 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fe9c002a930 tx=0x7fe9c0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.383+0000 7fe9b3fff640 1 -- 172.21.15.16:0/2188801132 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c003f030 con 0x7fe9cc107100 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.383+0000 7fe9b3fff640 1 -- 172.21.15.16:0/2188801132 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9c0003aa0 con 0x7fe9cc107100 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.383+0000 7fe9b3fff640 1 -- 172.21.15.16:0/2188801132 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0035070 con 0x7fe9cc107100 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.383+0000 7fe9d107e640 1 -- 172.21.15.16:0/2188801132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9cc1a0410 con 0x7fe9cc107100 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.383+0000 7fe9d107e640 1 -- 172.21.15.16:0/2188801132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9cc1a08f0 con 0x7fe9cc107100 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.384+0000 7fe9d107e640 1 -- 172.21.15.16:0/2188801132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9cc102310 con 0x7fe9cc107100 2024-08-28T19:28:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.386+0000 7fe9b3fff640 1 -- 172.21.15.16:0/2188801132 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe9c0042030 con 0x7fe9cc107100 2024-08-28T19:28:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.386+0000 7fe9b3fff640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe9b40b4cd0 0x7fe9b40b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.386+0000 7fe9cad9d640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe9b40b4cd0 0x7fe9b40b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.387+0000 7fe9b3fff640 1 -- 172.21.15.16:0/2188801132 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7fe9c0106300 con 0x7fe9cc107100 2024-08-28T19:28:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.387+0000 7fe9cad9d640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe9b40b4cd0 0x7fe9b40b71e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00abe0 tx=0x7fe9bc00a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.390+0000 7fe9b3fff640 1 -- 172.21.15.16:0/2188801132 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fe9c00ccd20 con 0x7fe9cc107100 2024-08-28T19:28:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.617+0000 7fe9d107e640 1 -- 172.21.15.16:0/2188801132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9cc10ba30 con 0x7fe9cc107100 2024-08-28T19:28:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.618+0000 7fe9b3fff640 1 -- 172.21.15.16:0/2188801132 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+11985 (secure 0 0 0) 0x7fe9c00d5260 con 0x7fe9cc107100 2024-08-28T19:28:16.620 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:28:16.620 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":47,"btime":"2024-08-28T19:27:48:332467+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:27:48.279644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:28:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 47 2024-08-28T19:28:16.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.620+0000 7fe9d107e640 1 -- 172.21.15.16:0/2188801132 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe9b40b4cd0 msgr2=0x7fe9b40b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:16.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.620+0000 7fe9d107e640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe9b40b4cd0 0x7fe9b40b71e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00abe0 tx=0x7fe9bc00a2d0 comp rx=0 tx=0).stop 2024-08-28T19:28:16.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.620+0000 7fe9d107e640 1 -- 172.21.15.16:0/2188801132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc107100 msgr2=0x7fe9cc19a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:16.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.620+0000 7fe9d107e640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc107100 0x7fe9cc19a870 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fe9c002a930 tx=0x7fe9c0004270 comp rx=0 tx=0).stop 2024-08-28T19:28:16.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.620+0000 7fe9d107e640 1 -- 172.21.15.16:0/2188801132 shutdown_connections 2024-08-28T19:28:16.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.620+0000 7fe9d107e640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9cc107a50 0x7fe9cc19fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:16.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.620+0000 7fe9d107e640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc107100 0x7fe9cc19a870 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:16.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.620+0000 7fe9d107e640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe9b40b4cd0 0x7fe9b40b71e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:16.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.620+0000 7fe9d107e640 1 --2- 172.21.15.16:0/2188801132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101f00 0x7fe9cc19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:16.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.620+0000 7fe9d107e640 1 -- 172.21.15.16:0/2188801132 >> 172.21.15.16:0/2188801132 conn(0x7fe9cc079c40 msgr2=0x7fe9cc105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:16.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.621+0000 7fe9d107e640 1 -- 172.21.15.16:0/2188801132 shutdown_connections 2024-08-28T19:28:16.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:16.621+0000 7fe9d107e640 1 -- 172.21.15.16:0/2188801132 wait complete. 2024-08-28T19:28:16.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 346535 4.84 MB/sec execute 236 sec latency 2189.618 ms 2024-08-28T19:28:17.251 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:28:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:17 smithi016 ceph-mon[18556]: pgmap v558: 129 pgs: 129 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-08-28T19:28:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:17 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2188801132' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:28:17.523 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:28:17.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:17 smithi028 ceph-mon[21769]: pgmap v558: 129 pgs: 129 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-08-28T19:28:17.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:17 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2188801132' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:28:17.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:17 smithi115 ceph-mon[22184]: pgmap v558: 129 pgs: 129 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-08-28T19:28:17.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:17 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2188801132' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:28:17.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 348044 4.87 MB/sec execute 237 sec latency 30.355 ms 2024-08-28T19:28:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.987+0000 7f472df2a640 1 -- 172.21.15.16:0/3210741954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4728109be0 msgr2=0x7f4728110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.987+0000 7f472df2a640 1 --2- 172.21.15.16:0/3210741954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4728109be0 0x7f4728110410 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f4720009ca0 tx=0x7f472002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.987+0000 7f472df2a640 1 -- 172.21.15.16:0/3210741954 shutdown_connections 2024-08-28T19:28:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.987+0000 7f472df2a640 1 --2- 172.21.15.16:0/3210741954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4728109be0 0x7f4728110410 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.987+0000 7f472df2a640 1 --2- 172.21.15.16:0/3210741954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728109290 0x7f47281096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.987+0000 7f472df2a640 1 --2- 172.21.15.16:0/3210741954 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4728103950 0x7f4728103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.987+0000 7f472df2a640 1 -- 172.21.15.16:0/3210741954 >> 172.21.15.16:0/3210741954 conn(0x7f47280ff3a0 msgr2=0x7f47281017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.987+0000 7f472df2a640 1 -- 172.21.15.16:0/3210741954 shutdown_connections 2024-08-28T19:28:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.987+0000 7f472df2a640 1 -- 172.21.15.16:0/3210741954 wait complete. 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.988+0000 7f472df2a640 1 Processor -- start 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.988+0000 7f472df2a640 1 -- start start 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f472df2a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4728103950 0x7f472819c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f472df2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728109290 0x7f472819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f472df2a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4728109be0 0x7f47281a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f472df2a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47281138a0 con 0x7f4728109be0 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f472df2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4728113720 con 0x7f4728109290 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f472df2a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4728113a20 con 0x7f4728103950 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f4726ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728109290 0x7f472819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f4726ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728109290 0x7f472819c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37096/0 (socket says 172.21.15.16:37096) 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f4726ffd640 1 -- 172.21.15.16:0/2894900944 learned_addr learned my addr 172.21.15.16:0/2894900944 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f47277fe640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4728103950 0x7f472819c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f4726ffd640 1 -- 172.21.15.16:0/2894900944 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4728103950 msgr2=0x7f472819c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f4726ffd640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4728103950 0x7f472819c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f4726ffd640 1 -- 172.21.15.16:0/2894900944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4728109be0 msgr2=0x7f47281a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f4726ffd640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4728109be0 0x7f47281a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f4726ffd640 1 -- 172.21.15.16:0/2894900944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47281a2380 con 0x7f4728109290 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.989+0000 7f47277fe640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4728103950 0x7f472819c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.990+0000 7f4726ffd640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728109290 0x7f472819c9b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f471000e9f0 tx=0x7f471000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.990+0000 7f4724ff9640 1 -- 172.21.15.16:0/2894900944 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471000ccc0 con 0x7f4728109290 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.990+0000 7f4724ff9640 1 -- 172.21.15.16:0/2894900944 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47100108e0 con 0x7f4728109290 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.990+0000 7f4724ff9640 1 -- 172.21.15.16:0/2894900944 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4710018860 con 0x7f4728109290 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.990+0000 7f472df2a640 1 -- 172.21.15.16:0/2894900944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47281a2670 con 0x7f4728109290 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.990+0000 7f472df2a640 1 -- 172.21.15.16:0/2894900944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47281a8dd0 con 0x7f4728109290 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.993+0000 7f4724ff9640 1 -- 172.21.15.16:0/2894900944 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4710010430 con 0x7f4728109290 2024-08-28T19:28:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.993+0000 7f473449f740 1 Processor -- start 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.993+0000 7f473449f740 1 -- start start 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.993+0000 7f473449f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 0x55722d17e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.993+0000 7f473449f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb01b0 0x55722d17e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.993+0000 7f473449f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdb8390 0x55722d17ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.993+0000 7f473449f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55722d0f5680 con 0x55722cdd9070 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.993+0000 7f473449f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55722d0e4c10 con 0x55722cdb8390 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.993+0000 7f473449f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55722d0e8bc0 con 0x55722cdb01b0 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f4727fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 0x55722d17e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f47277fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb01b0 0x55722d17e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f4727fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 0x55722d17e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41624/0 (socket says 172.21.15.16:41624) 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f4727fff640 1 -- 172.21.15.16:0/2316025289 learned_addr learned my addr 172.21.15.16:0/2316025289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f4724ff9640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47140b4d20 0x7f47140b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f4726ffd640 1 --2- 172.21.15.16:0/2316025289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdb8390 0x55722d17ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f4727fff640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47140b4d20 0x7f47140b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f4727fff640 1 -- 172.21.15.16:0/2316025289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb01b0 msgr2=0x55722d17e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f4727fff640 1 --2- 172.21.15.16:0/2316025289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb01b0 0x55722d17e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f4727fff640 1 -- 172.21.15.16:0/2316025289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdb8390 msgr2=0x55722d17ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f4727fff640 1 --2- 172.21.15.16:0/2316025289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdb8390 0x55722d17ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.994+0000 7f4727fff640 1 -- 172.21.15.16:0/2316025289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55722ce44650 con 0x55722cdd9070 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.995+0000 7f4727fff640 1 --2- 172.21.15.16:0/2316025289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 0x55722d17e310 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f47200097b0 tx=0x7f4720044e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7576b05b308f8daf server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.995+0000 7f4724ff9640 1 -- 172.21.15.16:0/2894900944 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f47100e5cf0 con 0x7f4728109290 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.995+0000 7f4727fff640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47140b4d20 0x7f47140b7230 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4720054000 tx=0x7f472002fa30 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.996+0000 7f471dffb640 1 -- 172.21.15.16:0/2316025289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4720031450 con 0x55722cdd9070 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.996+0000 7f471dffb640 1 -- 172.21.15.16:0/2316025289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4720036050 con 0x55722cdd9070 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.996+0000 7f471dffb640 1 -- 172.21.15.16:0/2316025289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4720004460 con 0x55722cdd9070 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.997+0000 7f473449f740 1 -- 172.21.15.16:0/2316025289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 msgr2=0x55722d17e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.997+0000 7f473449f740 1 --2- 172.21.15.16:0/2316025289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 0x55722d17e310 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f47200097b0 tx=0x7f4720044e50 comp rx=0 tx=0).stop 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.997+0000 7f473449f740 1 -- 172.21.15.16:0/2316025289 shutdown_connections 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.997+0000 7f473449f740 1 --2- 172.21.15.16:0/2316025289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdb8390 0x55722d17ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.997+0000 7f473449f740 1 --2- 172.21.15.16:0/2316025289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb01b0 0x55722d17e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.997+0000 7f473449f740 1 --2- 172.21.15.16:0/2316025289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 0x55722d17e310 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.997+0000 7f473449f740 1 -- 172.21.15.16:0/2316025289 >> 172.21.15.16:0/2316025289 conn(0x55722cdcd460 msgr2=0x55722d17c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.998+0000 7f473449f740 1 -- 172.21.15.16:0/2316025289 shutdown_connections 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.998+0000 7f473449f740 1 -- 172.21.15.16:0/2316025289 wait complete. 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.998+0000 7f473449f740 1 -- start start 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.999+0000 7f473449f740 1 Processor -- start 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.999+0000 7f473449f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb01b0 0x55722d189ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.999+0000 7f473449f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdb8390 0x55722d18a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:17.999+0000 7f473449f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 0x55722d195ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f473449f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55722cf47700 con 0x55722cdd9070 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f473449f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55722d06b750 con 0x55722cdb8390 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f473449f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55722d18c970 con 0x55722cdb01b0 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f473449f740 10 client.0 _ll_register_callbacks cb 0x55722d09ae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f47277fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb01b0 0x55722d189ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f4727fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 0x55722d195ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f4726ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdb8390 0x55722d18a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f47277fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb01b0 0x55722d189ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56350/0 (socket says 172.21.15.16:56350) 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f47277fe640 1 -- 172.21.15.16:0/3092105827 learned_addr learned my addr 172.21.15.16:0/3092105827 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f4727fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 0x55722d195ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41626/0 (socket says 172.21.15.16:41626) 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f4726ffd640 1 -- 172.21.15.16:0/3092105827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb01b0 msgr2=0x55722d189ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f4726ffd640 1 --2- 172.21.15.16:0/3092105827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb01b0 0x55722d189ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f4726ffd640 1 -- 172.21.15.16:0/3092105827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 msgr2=0x55722d195ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f4726ffd640 1 --2- 172.21.15.16:0/3092105827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 0x55722d195ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.000+0000 7f4726ffd640 1 -- 172.21.15.16:0/3092105827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55722d0a4900 con 0x55722cdb8390 2024-08-28T19:28:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.001+0000 7f4726ffd640 1 --2- 172.21.15.16:0/3092105827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdb8390 0x55722d18a010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f47100e3010 tx=0x7f4710009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad6b8bdb29aa3d61 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.001+0000 7f4706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:28:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.001+0000 7f4706ffd640 1 -- 172.21.15.16:0/3092105827 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47100e34a0 con 0x55722cdb8390 2024-08-28T19:28:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.001+0000 7f473449f740 1 -- 172.21.15.16:0/3092105827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4720036050 con 0x55722cdb8390 2024-08-28T19:28:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.001+0000 7f4706ffd640 1 -- 172.21.15.16:0/3092105827 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f47100ab410 con 0x55722cdb8390 2024-08-28T19:28:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.001+0000 7f4706ffd640 1 -- 172.21.15.16:0/3092105827 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f47100abfb0 con 0x55722cdb8390 2024-08-28T19:28:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.001+0000 7f473449f740 10 client.26909 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.001+0000 7f473449f740 1 -- 172.21.15.16:0/3092105827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55722d1987f0 con 0x55722cdb8390 2024-08-28T19:28:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.002+0000 7f4706ffd640 1 -- 172.21.15.16:0/3092105827 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4720036050 con 0x55722cdb8390 2024-08-28T19:28:18.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.002+0000 7f4706ffd640 1 -- 172.21.15.16:0/3092105827 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f47100a7920 con 0x55722cdb8390 2024-08-28T19:28:18.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.002+0000 7f473449f740 10 client.26909 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.002+0000 7f473449f740 10 client.26909 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.002+0000 7f473449f740 20 client.26909 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.002+0000 7f473449f740 1 --2- 172.21.15.16:0/3092105827 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55722cd979f0 0x55722d19c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.002+0000 7f473449f740 4 client.26909 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.002+0000 7f473449f740 1 -- 172.21.15.16:0/3092105827 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55722d0f75d0 con 0x55722cd979f0 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.002+0000 7f47277fe640 1 --2- 172.21.15.16:0/3092105827 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55722cd979f0 0x55722d19c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.003+0000 7f4706ffd640 10 client.26909 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.003+0000 7f47277fe640 1 --2- 172.21.15.16:0/3092105827 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55722cd979f0 0x55722d19c040 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=50402fc00b017a4 server_cookie=761b8f72be6111a3 in_seq=0 out_seq=0 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.004+0000 7f4706ffd640 1 -- 172.21.15.16:0/3092105827 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55722d0f75d0 con 0x55722cd979f0 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.004+0000 7f4706ffd640 10 client.26909 handle_command_reply: tid=0 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.004+0000 7f473449f740 1 client.26909 shutdown 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.005+0000 7f473449f740 1 -- 172.21.15.16:0/3092105827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdb8390 msgr2=0x55722d18a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.005+0000 7f473449f740 1 --2- 172.21.15.16:0/3092105827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdb8390 0x55722d18a010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f47100e3010 tx=0x7f4710009530 comp rx=0 tx=0).stop 2024-08-28T19:28:18.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.005+0000 7f473449f740 1 -- 172.21.15.16:0/3092105827 shutdown_connections 2024-08-28T19:28:18.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.005+0000 7f473449f740 1 --2- 172.21.15.16:0/3092105827 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55722cd979f0 0x55722d19c040 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.005+0000 7f473449f740 1 --2- 172.21.15.16:0/3092105827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdd9070 0x55722d195ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.005+0000 7f473449f740 1 --2- 172.21.15.16:0/3092105827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdb8390 0x55722d18a010 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.005+0000 7f473449f740 1 --2- 172.21.15.16:0/3092105827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb01b0 0x55722d189ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.005+0000 7f473449f740 1 -- 172.21.15.16:0/3092105827 >> 172.21.15.16:0/3092105827 conn(0x55722cdcd460 msgr2=0x55722d1814f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:18.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.006+0000 7f4706ffd640 0 client.26909 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:28:18.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.006+0000 7f473449f740 1 -- 172.21.15.16:0/3092105827 shutdown_connections 2024-08-28T19:28:18.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.006+0000 7f473449f740 1 -- 172.21.15.16:0/3092105827 wait complete. 2024-08-28T19:28:18.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.006+0000 7f473449f740 20 client.26909 trim_cache size 0 max 16384 2024-08-28T19:28:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.020+0000 7f473449f740 1 Processor -- start 2024-08-28T19:28:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.020+0000 7f473449f740 1 -- start start 2024-08-28T19:28:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f473449f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d177230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f473449f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb8390 0x55722d189f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f473449f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 0x55722d181410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f473449f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47281aeb20 con 0x55722cdb01b0 2024-08-28T19:28:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f473449f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47281b0750 con 0x55722cdd9070 2024-08-28T19:28:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f473449f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f47281af1f0 con 0x55722cdb8390 2024-08-28T19:28:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f47277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d177230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f47277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d177230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41642/0 (socket says 172.21.15.16:41642) 2024-08-28T19:28:18.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f4727fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 0x55722d181410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f47277fe640 1 -- 172.21.15.16:0/2443498682 learned_addr learned my addr 172.21.15.16:0/2443498682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:18.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f4726ffd640 1 --2- 172.21.15.16:0/2443498682 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb8390 0x55722d189f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f47277fe640 1 -- 172.21.15.16:0/2443498682 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb8390 msgr2=0x55722d189f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f47277fe640 1 --2- 172.21.15.16:0/2443498682 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb8390 0x55722d189f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f47277fe640 1 -- 172.21.15.16:0/2443498682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 msgr2=0x55722d181410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f47277fe640 1 --2- 172.21.15.16:0/2443498682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 0x55722d181410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f47277fe640 1 -- 172.21.15.16:0/2443498682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55722d188a30 con 0x55722cdb01b0 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f4727fff640 1 --2- 172.21.15.16:0/2443498682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 0x55722d181410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.021+0000 7f47277fe640 1 --2- 172.21.15.16:0/2443498682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d177230 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f4718004700 tx=0x7f4718005410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b17fabc3a4f92f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.022+0000 7f471d7fa640 1 -- 172.21.15.16:0/2443498682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4718022040 con 0x55722cdb01b0 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.022+0000 7f471d7fa640 1 -- 172.21.15.16:0/2443498682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4718004e20 con 0x55722cdb01b0 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.022+0000 7f471d7fa640 1 -- 172.21.15.16:0/2443498682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471801f890 con 0x55722cdb01b0 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.023+0000 7f473449f740 1 -- 172.21.15.16:0/2443498682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 msgr2=0x55722d177230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.023+0000 7f473449f740 1 --2- 172.21.15.16:0/2443498682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d177230 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f4718004700 tx=0x7f4718005410 comp rx=0 tx=0).stop 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.023+0000 7f473449f740 1 -- 172.21.15.16:0/2443498682 shutdown_connections 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.023+0000 7f473449f740 1 --2- 172.21.15.16:0/2443498682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 0x55722d181410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.023+0000 7f473449f740 1 --2- 172.21.15.16:0/2443498682 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb8390 0x55722d189f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.023+0000 7f473449f740 1 --2- 172.21.15.16:0/2443498682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d177230 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.023+0000 7f473449f740 1 -- 172.21.15.16:0/2443498682 >> 172.21.15.16:0/2443498682 conn(0x55722cdcd460 msgr2=0x55722d189880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.023+0000 7f473449f740 1 -- 172.21.15.16:0/2443498682 shutdown_connections 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.023+0000 7f473449f740 1 -- 172.21.15.16:0/2443498682 wait complete. 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.023+0000 7f473449f740 1 -- start start 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.023+0000 7f473449f740 1 Processor -- start 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f473449f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d192500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f473449f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb8390 0x55722d18d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f473449f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 0x55722d18e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f473449f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47281afff0 con 0x55722cdb01b0 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f473449f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47281af8c0 con 0x55722cdd9070 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f473449f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f47281acb60 con 0x55722cdb8390 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f473449f740 10 client.0 _ll_register_callbacks cb 0x7f47100ab9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f47277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d192500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f4726ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb8390 0x55722d18d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f47277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d192500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41652/0 (socket says 172.21.15.16:41652) 2024-08-28T19:28:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f4726ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb8390 0x55722d18d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56364/0 (socket says 172.21.15.16:56364) 2024-08-28T19:28:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f4727fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 0x55722d18e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f47277fe640 1 -- 172.21.15.16:0/1302474140 learned_addr learned my addr 172.21.15.16:0/1302474140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f47277fe640 1 -- 172.21.15.16:0/1302474140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb8390 msgr2=0x55722d18d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f47277fe640 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb8390 0x55722d18d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f47277fe640 1 -- 172.21.15.16:0/1302474140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 msgr2=0x55722d18e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f47277fe640 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 0x55722d18e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f47277fe640 1 -- 172.21.15.16:0/1302474140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55722d18e770 con 0x55722cdb01b0 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f4727fff640 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 0x55722d18e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f47277fe640 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d192500 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f47180056d0 tx=0x7f4718005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0cb087a400a8992 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f4706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.024+0000 7f4706ffd640 1 -- 172.21.15.16:0/1302474140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4718022040 con 0x55722cdb01b0 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f4706ffd640 1 -- 172.21.15.16:0/1302474140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f471801f9e0 con 0x55722cdb01b0 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f473449f740 1 -- 172.21.15.16:0/1302474140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4718004e20 con 0x55722cdb01b0 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f4706ffd640 1 -- 172.21.15.16:0/1302474140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f471801fbe0 con 0x55722cdb01b0 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f473449f740 10 client.17370 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f473449f740 1 -- 172.21.15.16:0/1302474140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55722d206a50 con 0x55722cdb01b0 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f4706ffd640 1 -- 172.21.15.16:0/1302474140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4718004e20 con 0x55722cdb01b0 2024-08-28T19:28:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f4706ffd640 1 -- 172.21.15.16:0/1302474140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f47180203e0 con 0x55722cdb01b0 2024-08-28T19:28:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f473449f740 10 client.17370 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f473449f740 10 client.17370 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:28:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f473449f740 20 client.17370 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f473449f740 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55722cd6cda0 0x55722d209230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f473449f740 4 client.17370 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:28:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f473449f740 1 -- 172.21.15.16:0/1302474140 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55722d0f3070 con 0x55722cd6cda0 2024-08-28T19:28:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.025+0000 7f4726ffd640 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55722cd6cda0 0x55722d209230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:18.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f4726ffd640 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55722cd6cda0 0x55722d209230 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=464e488961c06150 server_cookie=17fc3315fd0b609d in_seq=0 out_seq=0 2024-08-28T19:28:18.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f4706ffd640 10 client.17370 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:28:18.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f4706ffd640 1 -- 172.21.15.16:0/1302474140 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55722d0f3070 con 0x55722cd6cda0 2024-08-28T19:28:18.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f4706ffd640 10 client.17370 handle_command_reply: tid=0 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f473449f740 1 client.17370 shutdown 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f473449f740 1 -- 172.21.15.16:0/1302474140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 msgr2=0x55722d192500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f473449f740 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d192500 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f47180056d0 tx=0x7f4718005700 comp rx=0 tx=0).stop 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f473449f740 1 -- 172.21.15.16:0/1302474140 shutdown_connections 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f473449f740 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55722cd6cda0 0x55722d209230 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f473449f740 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55722cdd9070 0x55722d18e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f4706ffd640 0 client.17370 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f473449f740 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55722cdb8390 0x55722d18d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f473449f740 1 --2- 172.21.15.16:0/1302474140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55722cdb01b0 0x55722d192500 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f473449f740 1 -- 172.21.15.16:0/1302474140 >> 172.21.15.16:0/1302474140 conn(0x55722cdcd460 msgr2=0x55722d181410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.026+0000 7f473449f740 1 -- 172.21.15.16:0/1302474140 shutdown_connections 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f473449f740 1 -- 172.21.15.16:0/1302474140 wait complete. 2024-08-28T19:28:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f473449f740 20 client.17370 trim_cache size 0 max 16384 2024-08-28T19:28:18.034 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:28:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f472df2a640 1 -- 172.21.15.16:0/2894900944 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47140b4d20 msgr2=0x7f47140b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f472df2a640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47140b4d20 0x7f47140b7230 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4720054000 tx=0x7f472002fa30 comp rx=0 tx=0).stop 2024-08-28T19:28:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f472df2a640 1 -- 172.21.15.16:0/2894900944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728109290 msgr2=0x7f472819c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f472df2a640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728109290 0x7f472819c9b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f471000e9f0 tx=0x7f471000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f472df2a640 1 -- 172.21.15.16:0/2894900944 shutdown_connections 2024-08-28T19:28:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f472df2a640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4728109be0 0x7f47281a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f472df2a640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f47140b4d20 0x7f47140b7230 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f472df2a640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728109290 0x7f472819c9b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f472df2a640 1 --2- 172.21.15.16:0/2894900944 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4728103950 0x7f472819c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.027+0000 7f472df2a640 1 -- 172.21.15.16:0/2894900944 >> 172.21.15.16:0/2894900944 conn(0x7f47280ff3a0 msgr2=0x7f47281088a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.028+0000 7f472df2a640 1 -- 172.21.15.16:0/2894900944 shutdown_connections 2024-08-28T19:28:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:18.028+0000 7f472df2a640 1 -- 172.21.15.16:0/2894900944 wait complete. 2024-08-28T19:28:18.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:28:18.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 349302 4.87 MB/sec execute 238 sec latency 118.897 ms 2024-08-28T19:28:19.100 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:28:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:19 smithi016 ceph-mon[18556]: pgmap v559: 129 pgs: 129 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-08-28T19:28:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:19 smithi028 ceph-mon[21769]: pgmap v559: 129 pgs: 129 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-08-28T19:28:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:19 smithi115 ceph-mon[22184]: pgmap v559: 129 pgs: 129 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-08-28T19:28:19.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.637+0000 7efcfbf8f640 1 -- 172.21.15.16:0/3983509537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf41081c0 msgr2=0x7efcf41085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:19.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.637+0000 7efcfbf8f640 1 --2- 172.21.15.16:0/3983509537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf41081c0 0x7efcf41085d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7efce4009900 tx=0x7efce402a410 comp rx=0 tx=0).stop 2024-08-28T19:28:19.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.638+0000 7efcfbf8f640 1 -- 172.21.15.16:0/3983509537 shutdown_connections 2024-08-28T19:28:19.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.638+0000 7efcfbf8f640 1 --2- 172.21.15.16:0/3983509537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf4108b10 0x7efcf410f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.638+0000 7efcfbf8f640 1 --2- 172.21.15.16:0/3983509537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf41081c0 0x7efcf41085d0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.638+0000 7efcfbf8f640 1 --2- 172.21.15.16:0/3983509537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcf4102880 0x7efcf4102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.638+0000 7efcfbf8f640 1 -- 172.21.15.16:0/3983509537 >> 172.21.15.16:0/3983509537 conn(0x7efcf4079510 msgr2=0x7efcf4079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:19.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.638+0000 7efcfbf8f640 1 -- 172.21.15.16:0/3983509537 shutdown_connections 2024-08-28T19:28:19.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.638+0000 7efcfbf8f640 1 -- 172.21.15.16:0/3983509537 wait complete. 2024-08-28T19:28:19.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.638+0000 7efcfbf8f640 1 Processor -- start 2024-08-28T19:28:19.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.638+0000 7efcfbf8f640 1 -- start start 2024-08-28T19:28:19.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcfbf8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf4102880 0x7efcf419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcfbf8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf41081c0 0x7efcf419a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcfbf8f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcf4108b10 0x7efcf419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcfbf8f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efcf41127a0 con 0x7efcf41081c0 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcfbf8f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcf4112620 con 0x7efcf4102880 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcfbf8f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efcf4112920 con 0x7efcf4108b10 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcf952a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf41081c0 0x7efcf419a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcf952a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf41081c0 0x7efcf419a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41670/0 (socket says 172.21.15.16:41670) 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcfa52c640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcf4108b10 0x7efcf419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcf952a640 1 -- 172.21.15.16:0/2365346605 learned_addr learned my addr 172.21.15.16:0/2365346605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf4102880 0x7efcf419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcf952a640 1 -- 172.21.15.16:0/2365346605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcf4108b10 msgr2=0x7efcf419fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcf952a640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcf4108b10 0x7efcf419fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcf952a640 1 -- 172.21.15.16:0/2365346605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf4102880 msgr2=0x7efcf419a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcf952a640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf4102880 0x7efcf419a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcf952a640 1 -- 172.21.15.16:0/2365346605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf41a01f0 con 0x7efcf41081c0 2024-08-28T19:28:19.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.639+0000 7efcf952a640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf41081c0 0x7efcf419a8e0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7efce402a930 tx=0x7efce4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:19.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.640+0000 7efceaffd640 1 -- 172.21.15.16:0/2365346605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce403f030 con 0x7efcf41081c0 2024-08-28T19:28:19.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.640+0000 7efcfbf8f640 1 -- 172.21.15.16:0/2365346605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf41a0480 con 0x7efcf41081c0 2024-08-28T19:28:19.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.640+0000 7efceaffd640 1 -- 172.21.15.16:0/2365346605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efce4003aa0 con 0x7efcf41081c0 2024-08-28T19:28:19.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.640+0000 7efceaffd640 1 -- 172.21.15.16:0/2365346605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce4035070 con 0x7efcf41081c0 2024-08-28T19:28:19.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.640+0000 7efcfbf8f640 1 -- 172.21.15.16:0/2365346605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf41a09b0 con 0x7efcf41081c0 2024-08-28T19:28:19.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.642+0000 7efd02504740 1 Processor -- start 2024-08-28T19:28:19.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efd02504740 1 -- start start 2024-08-28T19:28:19.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efceaffd640 1 -- 172.21.15.16:0/2365346605 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7efce4042030 con 0x7efcf41081c0 2024-08-28T19:28:19.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efd02504740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 0x562a64752350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efd02504740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643841b0 0x562a64752890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efd02504740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6438c390 0x562a64752dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efd02504740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562a646c9680 con 0x562a643ad070 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efd02504740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a646b8c10 con 0x562a6438c390 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efd02504740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562a646bcbc0 con 0x562a643841b0 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efcfa52c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 0x562a64752350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efcf9d2b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643841b0 0x562a64752890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efcfa52c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 0x562a64752350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41684/0 (socket says 172.21.15.16:41684) 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efcfa52c640 1 -- 172.21.15.16:0/2265330468 learned_addr learned my addr 172.21.15.16:0/2265330468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efcf952a640 1 --2- 172.21.15.16:0/2265330468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6438c390 0x562a64752dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efcf9d2b640 1 -- 172.21.15.16:0/2265330468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6438c390 msgr2=0x562a64752dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/2265330468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6438c390 0x562a64752dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efcf9d2b640 1 -- 172.21.15.16:0/2265330468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 msgr2=0x562a64752350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/2265330468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 0x562a64752350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.643+0000 7efcf9d2b640 1 -- 172.21.15.16:0/2265330468 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a64418650 con 0x562a643841b0 2024-08-28T19:28:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.644+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/2265330468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643841b0 0x562a64752890 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efcf0004630 tx=0x7efcf000bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8b6d2ea32ca6612 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:19.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.644+0000 7efceaffd640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efce00b4cd0 0x7efce00b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.644+0000 7efcdffff640 1 -- 172.21.15.16:0/2265330468 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0003b70 con 0x562a643841b0 2024-08-28T19:28:19.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.644+0000 7efcfa52c640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efce00b4cd0 0x7efce00b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.644+0000 7efcdffff640 1 -- 172.21.15.16:0/2265330468 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcf0015070 con 0x562a643841b0 2024-08-28T19:28:19.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.644+0000 7efcdffff640 1 -- 172.21.15.16:0/2265330468 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0004c10 con 0x562a643841b0 2024-08-28T19:28:19.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.644+0000 7efceaffd640 1 -- 172.21.15.16:0/2365346605 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7efce4105a30 con 0x7efcf41081c0 2024-08-28T19:28:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.645+0000 7efd02504740 1 -- 172.21.15.16:0/2265330468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643841b0 msgr2=0x562a64752890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.645+0000 7efcfa52c640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efce00b4cd0 0x7efce00b71e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efcec010540 tx=0x7efcec007520 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.645+0000 7efd02504740 1 --2- 172.21.15.16:0/2265330468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643841b0 0x562a64752890 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efcf0004630 tx=0x7efcf000bb20 comp rx=0 tx=0).stop 2024-08-28T19:28:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.645+0000 7efd02504740 1 -- 172.21.15.16:0/2265330468 shutdown_connections 2024-08-28T19:28:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.645+0000 7efd02504740 1 --2- 172.21.15.16:0/2265330468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6438c390 0x562a64752dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.645+0000 7efd02504740 1 --2- 172.21.15.16:0/2265330468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643841b0 0x562a64752890 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.645+0000 7efd02504740 1 --2- 172.21.15.16:0/2265330468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 0x562a64752350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.645+0000 7efd02504740 1 -- 172.21.15.16:0/2265330468 >> 172.21.15.16:0/2265330468 conn(0x562a643a1460 msgr2=0x562a64750240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.645+0000 7efd02504740 1 -- 172.21.15.16:0/2265330468 shutdown_connections 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.645+0000 7efd02504740 1 -- 172.21.15.16:0/2265330468 wait complete. 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.645+0000 7efd02504740 1 -- start start 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efd02504740 1 Processor -- start 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efd02504740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a643841b0 0x562a64761e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efd02504740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a6438c390 0x562a647623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efd02504740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643ad070 0x562a6476e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efd02504740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562a6451b700 con 0x562a6438c390 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efd02504740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a6463f750 con 0x562a643841b0 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efd02504740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562a64764e90 con 0x562a643ad070 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efd02504740 10 client.0 _ll_register_callbacks cb 0x562a6466ee10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efcf952a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a6438c390 0x562a647623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efcf952a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a6438c390 0x562a647623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41694/0 (socket says 172.21.15.16:41694) 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efcf952a640 1 -- 172.21.15.16:0/3872417870 learned_addr learned my addr 172.21.15.16:0/3872417870 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a643841b0 0x562a64761e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.646+0000 7efcfa52c640 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643ad070 0x562a6476e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcf952a640 1 -- 172.21.15.16:0/3872417870 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643ad070 msgr2=0x562a6476e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcf952a640 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643ad070 0x562a6476e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcf952a640 1 -- 172.21.15.16:0/3872417870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a643841b0 msgr2=0x562a64761e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcf952a640 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a643841b0 0x562a64761e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcf952a640 1 -- 172.21.15.16:0/3872417870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a64678900 con 0x562a6438c390 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcfa52c640 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643ad070 0x562a6476e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcf952a640 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a6438c390 0x562a647623a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7efce4031440 tx=0x7efce4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9558f92fb6d760 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcdd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcdd7fa640 1 -- 172.21.15.16:0/3872417870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce40d0060 con 0x562a6438c390 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcdd7fa640 1 -- 172.21.15.16:0/3872417870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7efce4103030 con 0x562a6438c390 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efd02504740 1 -- 172.21.15.16:0/3872417870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcf0015070 con 0x562a6438c390 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcdd7fa640 1 -- 172.21.15.16:0/3872417870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7efce4049050 con 0x562a6438c390 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efd02504740 10 client.17382 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efd02504740 1 -- 172.21.15.16:0/3872417870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562a64770c20 con 0x562a6438c390 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.647+0000 7efcdd7fa640 1 -- 172.21.15.16:0/3872417870 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcf0015070 con 0x562a6438c390 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.648+0000 7efcdd7fa640 1 -- 172.21.15.16:0/3872417870 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7efce404c060 con 0x562a6438c390 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.648+0000 7efd02504740 10 client.17382 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.648+0000 7efd02504740 10 client.17382 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.648+0000 7efd02504740 20 client.17382 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.648+0000 7efd02504740 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562a6436b9f0 0x562a64774060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.648+0000 7efd02504740 4 client.17382 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.648+0000 7efd02504740 1 -- 172.21.15.16:0/3872417870 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a646cb5d0 con 0x562a6436b9f0 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.648+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562a6436b9f0 0x562a64774060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.648+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562a6436b9f0 0x562a64774060 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d1855735cc2f3f04 server_cookie=5af66853e29e882f in_seq=0 out_seq=0 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.648+0000 7efcdd7fa640 10 client.17382 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efcdd7fa640 1 -- 172.21.15.16:0/3872417870 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562a646cb5d0 con 0x562a6436b9f0 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efcdd7fa640 10 client.17382 handle_command_reply: tid=0 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efd02504740 1 client.17382 shutdown 2024-08-28T19:28:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efd02504740 1 -- 172.21.15.16:0/3872417870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a6438c390 msgr2=0x562a647623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efd02504740 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a6438c390 0x562a647623a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7efce4031440 tx=0x7efce4000bc0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efd02504740 1 -- 172.21.15.16:0/3872417870 shutdown_connections 2024-08-28T19:28:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efd02504740 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562a6436b9f0 0x562a64774060 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efd02504740 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643ad070 0x562a6476e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efd02504740 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a6438c390 0x562a647623a0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efd02504740 1 --2- 172.21.15.16:0/3872417870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a643841b0 0x562a64761e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efcdd7fa640 0 client.17382 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:28:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efd02504740 1 -- 172.21.15.16:0/3872417870 >> 172.21.15.16:0/3872417870 conn(0x562a643a1460 msgr2=0x562a6474e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efd02504740 1 -- 172.21.15.16:0/3872417870 shutdown_connections 2024-08-28T19:28:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.649+0000 7efd02504740 1 -- 172.21.15.16:0/3872417870 wait complete. 2024-08-28T19:28:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.650+0000 7efd02504740 20 client.17382 trim_cache size 0 max 16384 2024-08-28T19:28:19.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efd02504740 1 Processor -- start 2024-08-28T19:28:19.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efd02504740 1 -- start start 2024-08-28T19:28:19.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efd02504740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a643841b0 0x562a64763a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efd02504740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a6438c390 0x562a6476e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efd02504740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 0x562a6475f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efd02504740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efcf41ae9e0 con 0x562a643ad070 2024-08-28T19:28:19.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efd02504740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcf41b0610 con 0x562a643841b0 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efd02504740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efcf41af0b0 con 0x562a6438c390 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efcfa52c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 0x562a6475f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efcfa52c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 0x562a6475f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41710/0 (socket says 172.21.15.16:41710) 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efcfa52c640 1 -- 172.21.15.16:0/575290319 learned_addr learned my addr 172.21.15.16:0/575290319 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/575290319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a643841b0 0x562a64763a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efcf952a640 1 --2- 172.21.15.16:0/575290319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a6438c390 0x562a6476e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efcfa52c640 1 -- 172.21.15.16:0/575290319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a6438c390 msgr2=0x562a6476e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efcfa52c640 1 --2- 172.21.15.16:0/575290319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a6438c390 0x562a6476e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.663+0000 7efcfa52c640 1 -- 172.21.15.16:0/575290319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a643841b0 msgr2=0x562a64763a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.664+0000 7efcfa52c640 1 --2- 172.21.15.16:0/575290319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a643841b0 0x562a64763a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.664+0000 7efcfa52c640 1 -- 172.21.15.16:0/575290319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a64753140 con 0x562a643ad070 2024-08-28T19:28:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.664+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/575290319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a643841b0 0x562a64763a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:28:19.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.664+0000 7efcfa52c640 1 --2- 172.21.15.16:0/575290319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 0x562a6475f050 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7efcec00f8d0 tx=0x7efcec015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b881340e72ff4223 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:19.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.664+0000 7efcdf7fe640 1 -- 172.21.15.16:0/575290319 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcec003970 con 0x562a643ad070 2024-08-28T19:28:19.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.664+0000 7efcdf7fe640 1 -- 172.21.15.16:0/575290319 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcec01e930 con 0x562a643ad070 2024-08-28T19:28:19.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.664+0000 7efcdf7fe640 1 -- 172.21.15.16:0/575290319 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcec004290 con 0x562a643ad070 2024-08-28T19:28:19.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.665+0000 7efd02504740 1 -- 172.21.15.16:0/575290319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 msgr2=0x562a6475f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:19.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.665+0000 7efd02504740 1 --2- 172.21.15.16:0/575290319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 0x562a6475f050 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7efcec00f8d0 tx=0x7efcec015000 comp rx=0 tx=0).stop 2024-08-28T19:28:19.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.665+0000 7efd02504740 1 -- 172.21.15.16:0/575290319 shutdown_connections 2024-08-28T19:28:19.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.665+0000 7efd02504740 1 --2- 172.21.15.16:0/575290319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643ad070 0x562a6475f050 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.665+0000 7efd02504740 1 --2- 172.21.15.16:0/575290319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a6438c390 0x562a6476e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.665+0000 7efd02504740 1 --2- 172.21.15.16:0/575290319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a643841b0 0x562a64763a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.665+0000 7efd02504740 1 -- 172.21.15.16:0/575290319 >> 172.21.15.16:0/575290319 conn(0x562a643a1460 msgr2=0x562a64763380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.665+0000 7efd02504740 1 -- 172.21.15.16:0/575290319 shutdown_connections 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.665+0000 7efd02504740 1 -- 172.21.15.16:0/575290319 wait complete. 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.665+0000 7efd02504740 1 -- start start 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.665+0000 7efd02504740 1 Processor -- start 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efd02504740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643841b0 0x562a6476a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efd02504740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6438c390 0x562a64766d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efd02504740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643ad070 0x562a64767250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efd02504740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efcf41afeb0 con 0x562a643841b0 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efd02504740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcf41af780 con 0x562a6438c390 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efd02504740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efcf41ac9e0 con 0x562a643ad070 2024-08-28T19:28:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efd02504740 10 client.0 _ll_register_callbacks cb 0x7efce4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcf9d2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643841b0 0x562a6476a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcf9d2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643841b0 0x562a6476a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41716/0 (socket says 172.21.15.16:41716) 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcf9d2b640 1 -- 172.21.15.16:0/665819830 learned_addr learned my addr 172.21.15.16:0/665819830 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcfa52c640 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643ad070 0x562a64767250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcf952a640 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6438c390 0x562a64766d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcf9d2b640 1 -- 172.21.15.16:0/665819830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643ad070 msgr2=0x562a64767250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643ad070 0x562a64767250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcf9d2b640 1 -- 172.21.15.16:0/665819830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6438c390 msgr2=0x562a64766d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6438c390 0x562a64766d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcf9d2b640 1 -- 172.21.15.16:0/665819830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a647678f0 con 0x562a643841b0 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcf9d2b640 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643841b0 0x562a6476a850 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x562a64750fd0 tx=0x7efcf0003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed62db1834b46292 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.666+0000 7efcdd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:28:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efcdd7fa640 1 -- 172.21.15.16:0/665819830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0011850 con 0x562a643841b0 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efd02504740 1 -- 172.21.15.16:0/665819830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcec01e930 con 0x562a643841b0 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efcdd7fa640 1 -- 172.21.15.16:0/665819830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7efcf00119f0 con 0x562a643841b0 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efcdd7fa640 1 -- 172.21.15.16:0/665819830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7efcf003a050 con 0x562a643841b0 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efd02504740 10 client.17388 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efd02504740 1 -- 172.21.15.16:0/665819830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562a647df8b0 con 0x562a643841b0 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efcdd7fa640 1 -- 172.21.15.16:0/665819830 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcec01e930 con 0x562a643841b0 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efcdd7fa640 1 -- 172.21.15.16:0/665819830 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7efcf0035030 con 0x562a643841b0 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efd02504740 10 client.17388 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efd02504740 10 client.17388 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efd02504740 20 client.17388 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efd02504740 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562a64340da0 0x562a647e1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efd02504740 4 client.17388 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efd02504740 1 -- 172.21.15.16:0/665819830 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a646c7070 con 0x562a64340da0 2024-08-28T19:28:19.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.667+0000 7efcf952a640 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562a64340da0 0x562a647e1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efcdd7fa640 10 client.17388 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efcf952a640 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562a64340da0 0x562a647e1df0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9eb9bb4e6c8bbe8c server_cookie=432d8e433af8b3df in_seq=0 out_seq=0 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efcdd7fa640 1 -- 172.21.15.16:0/665819830 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a646c7070 con 0x562a64340da0 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efcdd7fa640 10 client.17388 handle_command_reply: tid=0 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efd02504740 1 client.17388 shutdown 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efd02504740 1 -- 172.21.15.16:0/665819830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643841b0 msgr2=0x562a6476a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efd02504740 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643841b0 0x562a6476a850 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x562a64750fd0 tx=0x7efcf0003e20 comp rx=0 tx=0).stop 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efd02504740 1 -- 172.21.15.16:0/665819830 shutdown_connections 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efd02504740 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562a64340da0 0x562a647e1df0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efd02504740 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562a643ad070 0x562a64767250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efd02504740 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6438c390 0x562a64766d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efd02504740 1 --2- 172.21.15.16:0/665819830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a643841b0 0x562a6476a850 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efcdd7fa640 0 client.17388 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efd02504740 1 -- 172.21.15.16:0/665819830 >> 172.21.15.16:0/665819830 conn(0x562a643a1460 msgr2=0x562a64757eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efd02504740 1 -- 172.21.15.16:0/665819830 shutdown_connections 2024-08-28T19:28:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.668+0000 7efd02504740 1 -- 172.21.15.16:0/665819830 wait complete. 2024-08-28T19:28:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efd02504740 20 client.17388 trim_cache size 0 max 16384 2024-08-28T19:28:19.675 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:28:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 -- 172.21.15.16:0/2365346605 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efce00b4cd0 msgr2=0x7efce00b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efce00b4cd0 0x7efce00b71e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efcec010540 tx=0x7efcec007520 comp rx=0 tx=0).stop 2024-08-28T19:28:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 -- 172.21.15.16:0/2365346605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf41081c0 msgr2=0x7efcf419a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf41081c0 0x7efcf419a8e0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7efce402a930 tx=0x7efce4004270 comp rx=0 tx=0).stop 2024-08-28T19:28:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 -- 172.21.15.16:0/2365346605 shutdown_connections 2024-08-28T19:28:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcf4108b10 0x7efcf419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf41081c0 0x7efcf419a8e0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efce00b4cd0 0x7efce00b71e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 --2- 172.21.15.16:0/2365346605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf4102880 0x7efcf419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 -- 172.21.15.16:0/2365346605 >> 172.21.15.16:0/2365346605 conn(0x7efcf4079510 msgr2=0x7efcf4101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 -- 172.21.15.16:0/2365346605 shutdown_connections 2024-08-28T19:28:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:19.669+0000 7efcfbf8f640 1 -- 172.21.15.16:0/2365346605 wait complete. 2024-08-28T19:28:19.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 349302 4.85 MB/sec execute 239 sec latency 1118.983 ms 2024-08-28T19:28:20.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:28:20.629 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:28:20.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 349302 4.83 MB/sec execute 240 sec latency 2119.028 ms 2024-08-28T19:28:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.119+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2675374759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3b010b470 msgr2=0x7fc3b010b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.119+0000 7fc3b4b38640 1 --2- 172.21.15.16:0/2675374759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3b010b470 0x7fc3b010b880 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009900 tx=0x7fc3a402a410 comp rx=0 tx=0).stop 2024-08-28T19:28:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.119+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2675374759 shutdown_connections 2024-08-28T19:28:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.119+0000 7fc3b4b38640 1 --2- 172.21.15.16:0/2675374759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b010bdc0 0x7fc3b01125f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.119+0000 7fc3b4b38640 1 --2- 172.21.15.16:0/2675374759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3b010b470 0x7fc3b010b880 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.119+0000 7fc3b4b38640 1 --2- 172.21.15.16:0/2675374759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3b0105b30 0x7fc3b0105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.119+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2675374759 >> 172.21.15.16:0/2675374759 conn(0x7fc3b0101600 msgr2=0x7fc3b0103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.119+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2675374759 shutdown_connections 2024-08-28T19:28:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.119+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2675374759 wait complete. 2024-08-28T19:28:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3b4b38640 1 Processor -- start 2024-08-28T19:28:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3b4b38640 1 -- start start 2024-08-28T19:28:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3b4b38640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3b0105b30 0x7fc3b019c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3b4b38640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3b010b470 0x7fc3b019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3b4b38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b010bdc0 0x7fc3b01a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3b4b38640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b0115a80 con 0x7fc3b010b470 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3b4b38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b0115900 con 0x7fc3b010bdc0 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3b4b38640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b0115c00 con 0x7fc3b0105b30 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3b010b470 0x7fc3b019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3b010b470 0x7fc3b019ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41730/0 (socket says 172.21.15.16:41730) 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3add9b640 1 -- 172.21.15.16:0/2854314014 learned_addr learned my addr 172.21.15.16:0/2854314014 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3aed9d640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b010bdc0 0x7fc3b01a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3ae59c640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3b0105b30 0x7fc3b019c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3add9b640 1 -- 172.21.15.16:0/2854314014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3b0105b30 msgr2=0x7fc3b019c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3add9b640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3b0105b30 0x7fc3b019c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3add9b640 1 -- 172.21.15.16:0/2854314014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b010bdc0 msgr2=0x7fc3b01a1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3add9b640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b010bdc0 0x7fc3b01a1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.120+0000 7fc3add9b640 1 -- 172.21.15.16:0/2854314014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b01a2400 con 0x7fc3b010b470 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.121+0000 7fc3add9b640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3b010b470 0x7fc3b019ca30 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fc3a402a930 tx=0x7fc3a4031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.121+0000 7fc39f7fe640 1 -- 172.21.15.16:0/2854314014 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a403f030 con 0x7fc3b010b470 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.121+0000 7fc39f7fe640 1 -- 172.21.15.16:0/2854314014 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3a4004490 con 0x7fc3b010b470 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.121+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2854314014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3b01a2690 con 0x7fc3b010b470 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.121+0000 7fc39f7fe640 1 -- 172.21.15.16:0/2854314014 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4035070 con 0x7fc3b010b470 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.121+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2854314014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3b01a8e50 con 0x7fc3b010b470 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.123+0000 7fc3bb0ad740 1 Processor -- start 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.123+0000 7fc39f7fe640 1 -- 172.21.15.16:0/2854314014 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc3a4042030 con 0x7fc3b010b470 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3bb0ad740 1 -- start start 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aca070 0x563019e6f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aa11b0 0x563019e6f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e6fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563019de6680 con 0x563019aa9390 2024-08-28T19:28:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563019dd5c10 con 0x563019aa11b0 2024-08-28T19:28:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563019dd9bc0 con 0x563019aca070 2024-08-28T19:28:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e6fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e6fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41744/0 (socket says 172.21.15.16:41744) 2024-08-28T19:28:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3add9b640 1 -- 172.21.15.16:0/2502432393 learned_addr learned my addr 172.21.15.16:0/2502432393 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3ae59c640 1 --2- 172.21.15.16:0/2502432393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aa11b0 0x563019e6f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc3aed9d640 1 --2- 172.21.15.16:0/2502432393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aca070 0x563019e6f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.124+0000 7fc39f7fe640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3980b4d20 0x7fc3980b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc3aed9d640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3980b4d20 0x7fc3980b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc3add9b640 1 -- 172.21.15.16:0/2502432393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aca070 msgr2=0x563019e6f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc3add9b640 1 --2- 172.21.15.16:0/2502432393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aca070 0x563019e6f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc3add9b640 1 -- 172.21.15.16:0/2502432393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aa11b0 msgr2=0x563019e6f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc3add9b640 1 --2- 172.21.15.16:0/2502432393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aa11b0 0x563019e6f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc3add9b640 1 -- 172.21.15.16:0/2502432393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563019b35650 con 0x563019aa9390 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc3ae59c640 1 --2- 172.21.15.16:0/2502432393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aa11b0 0x563019e6f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc39f7fe640 1 -- 172.21.15.16:0/2854314014 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7fc3a4105d00 con 0x7fc3b010b470 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc3add9b640 1 --2- 172.21.15.16:0/2502432393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e6fd90 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4102ff0 tx=0x7fc3a4085230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f4c796e6e5014e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc3aed9d640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3980b4d20 0x7fc3980b7230 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8005c40 tx=0x7fc3a8004ab0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc39cff9640 1 -- 172.21.15.16:0/2502432393 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4049020 con 0x563019aa9390 2024-08-28T19:28:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc39cff9640 1 -- 172.21.15.16:0/2502432393 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3a40cbb30 con 0x563019aa9390 2024-08-28T19:28:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.125+0000 7fc39cff9640 1 -- 172.21.15.16:0/2502432393 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a40cb970 con 0x563019aa9390 2024-08-28T19:28:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.126+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/2502432393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 msgr2=0x563019e6fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.126+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/2502432393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e6fd90 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4102ff0 tx=0x7fc3a4085230 comp rx=0 tx=0).stop 2024-08-28T19:28:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.126+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/2502432393 shutdown_connections 2024-08-28T19:28:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.126+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/2502432393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e6fd90 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.126+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/2502432393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aa11b0 0x563019e6f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.126+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/2502432393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aca070 0x563019e6f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.126+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/2502432393 >> 172.21.15.16:0/2502432393 conn(0x563019abe460 msgr2=0x563019e6d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.126+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/2502432393 shutdown_connections 2024-08-28T19:28:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.126+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/2502432393 wait complete. 2024-08-28T19:28:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3bb0ad740 1 -- start start 2024-08-28T19:28:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3bb0ad740 1 Processor -- start 2024-08-28T19:28:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aa11b0 0x563019e7aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e7af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aca070 0x563019e86a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563019c38700 con 0x563019aa9390 2024-08-28T19:28:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563019d5c750 con 0x563019aca070 2024-08-28T19:28:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563019de6680 con 0x563019aa11b0 2024-08-28T19:28:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3bb0ad740 10 client.0 _ll_register_callbacks cb 0x563019d8be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e7af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e7af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41758/0 (socket says 172.21.15.16:41758) 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3add9b640 1 -- 172.21.15.16:0/536813319 learned_addr learned my addr 172.21.15.16:0/536813319 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3ae59c640 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aa11b0 0x563019e7aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.127+0000 7fc3aed9d640 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aca070 0x563019e86a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc3add9b640 1 -- 172.21.15.16:0/536813319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aa11b0 msgr2=0x563019e7aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc3add9b640 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aa11b0 0x563019e7aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc3add9b640 1 -- 172.21.15.16:0/536813319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aca070 msgr2=0x563019e86a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc3add9b640 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aca070 0x563019e86a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc3add9b640 1 -- 172.21.15.16:0/536813319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563019d95900 con 0x563019aa9390 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc3add9b640 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e7af90 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40092a0 tx=0x7fc3a40057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a7fb6a6437eb2a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc395ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc395ffb640 1 -- 172.21.15.16:0/536813319 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4049040 con 0x563019aa9390 2024-08-28T19:28:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc395ffb640 1 -- 172.21.15.16:0/536813319 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7fc3a402edc0 con 0x563019aa9390 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/536813319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3a40cbb30 con 0x563019aa9390 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc395ffb640 1 -- 172.21.15.16:0/536813319 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fc3a404a260 con 0x563019aa9390 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc3bb0ad740 10 client.17400 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/536813319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x563019e894f0 con 0x563019aa9390 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc395ffb640 1 -- 172.21.15.16:0/536813319 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3a40cbb30 con 0x563019aa9390 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.128+0000 7fc395ffb640 1 -- 172.21.15.16:0/536813319 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fc3a4103300 con 0x563019aa9390 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.129+0000 7fc3bb0ad740 10 client.17400 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.129+0000 7fc3bb0ad740 10 client.17400 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.129+0000 7fc3bb0ad740 20 client.17400 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.129+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563019a889f0 0x563019e8c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.129+0000 7fc3bb0ad740 4 client.17400 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.129+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/536813319 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563019de85d0 con 0x563019a889f0 2024-08-28T19:28:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.129+0000 7fc3ae59c640 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563019a889f0 0x563019e8c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.129+0000 7fc3ae59c640 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563019a889f0 0x563019e8c680 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4190e35d183b99bc server_cookie=7a1bf17598a4f39c in_seq=0 out_seq=0 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.129+0000 7fc395ffb640 10 client.17400 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc395ffb640 1 -- 172.21.15.16:0/536813319 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563019de85d0 con 0x563019a889f0 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc395ffb640 10 client.17400 handle_command_reply: tid=0 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 1 client.17400 shutdown 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/536813319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 msgr2=0x563019e7af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e7af90 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40092a0 tx=0x7fc3a40057e0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/536813319 shutdown_connections 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563019a889f0 0x563019e8c680 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aca070 0x563019e86a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc395ffb640 0 client.17400 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:28:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e7af90 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/536813319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aa11b0 0x563019e7aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/536813319 >> 172.21.15.16:0/536813319 conn(0x563019abe460 msgr2=0x563019e70d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:21.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/536813319 shutdown_connections 2024-08-28T19:28:21.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/536813319 wait complete. 2024-08-28T19:28:21.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.130+0000 7fc3bb0ad740 20 client.17400 trim_cache size 0 max 16384 2024-08-28T19:28:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3bb0ad740 1 Processor -- start 2024-08-28T19:28:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3bb0ad740 1 -- start start 2024-08-28T19:28:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aa11b0 0x563019e6d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e74590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aca070 0x563019e86950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b01b2ee0 con 0x563019aa9390 2024-08-28T19:28:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b01b4b10 con 0x563019aa11b0 2024-08-28T19:28:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b01b35b0 con 0x563019aca070 2024-08-28T19:28:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e74590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e74590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41762/0 (socket says 172.21.15.16:41762) 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3add9b640 1 -- 172.21.15.16:0/2851747095 learned_addr learned my addr 172.21.15.16:0/2851747095 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3aed9d640 1 --2- 172.21.15.16:0/2851747095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aca070 0x563019e86950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3add9b640 1 -- 172.21.15.16:0/2851747095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aca070 msgr2=0x563019e86950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3ae59c640 1 --2- 172.21.15.16:0/2851747095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aa11b0 0x563019e6d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3add9b640 1 --2- 172.21.15.16:0/2851747095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aca070 0x563019e86950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3add9b640 1 -- 172.21.15.16:0/2851747095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aa11b0 msgr2=0x563019e6d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3add9b640 1 --2- 172.21.15.16:0/2851747095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aa11b0 0x563019e6d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3add9b640 1 -- 172.21.15.16:0/2851747095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563019e8c9f0 con 0x563019aa9390 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.144+0000 7fc3aed9d640 1 --2- 172.21.15.16:0/2851747095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aca070 0x563019e86950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.145+0000 7fc3add9b640 1 --2- 172.21.15.16:0/2851747095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e74590 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fc3a41030e0 tx=0x7fc3a4005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d262b29f51f2a8e6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.145+0000 7fc397fff640 1 -- 172.21.15.16:0/2851747095 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4049070 con 0x563019aa9390 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.145+0000 7fc397fff640 1 -- 172.21.15.16:0/2851747095 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3a40cbb30 con 0x563019aa9390 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.145+0000 7fc397fff640 1 -- 172.21.15.16:0/2851747095 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a402e260 con 0x563019aa9390 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/2851747095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 msgr2=0x563019e74590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/2851747095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e74590 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fc3a41030e0 tx=0x7fc3a4005f30 comp rx=0 tx=0).stop 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/2851747095 shutdown_connections 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/2851747095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aca070 0x563019e86950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/2851747095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e74590 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/2851747095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aa11b0 0x563019e6d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/2851747095 >> 172.21.15.16:0/2851747095 conn(0x563019abe460 msgr2=0x563019e6cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/2851747095 shutdown_connections 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/2851747095 wait complete. 2024-08-28T19:28:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 -- start start 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 Processor -- start 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aa11b0 0x563019e829c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e82f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aca070 0x563019e7d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b01b43b0 con 0x563019aa9390 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b01b3c80 con 0x563019aca070 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b01b0ee0 con 0x563019aa11b0 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.146+0000 7fc3bb0ad740 10 client.0 _ll_register_callbacks cb 0x7fc3a0009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e82f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e82f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41776/0 (socket says 172.21.15.16:41776) 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3add9b640 1 -- 172.21.15.16:0/1591047617 learned_addr learned my addr 172.21.15.16:0/1591047617 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3aed9d640 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aca070 0x563019e7d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3ae59c640 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aa11b0 0x563019e829c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3add9b640 1 -- 172.21.15.16:0/1591047617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aa11b0 msgr2=0x563019e829c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3add9b640 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aa11b0 0x563019e829c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3add9b640 1 -- 172.21.15.16:0/1591047617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aca070 msgr2=0x563019e7d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3add9b640 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aca070 0x563019e7d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3add9b640 1 -- 172.21.15.16:0/1591047617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563019e83440 con 0x563019aa9390 2024-08-28T19:28:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3ae59c640 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aa11b0 0x563019e829c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:28:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3add9b640 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e82f00 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40d53b0 tx=0x7fc3a40c7ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952e4cb9d0a7c055 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc395ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:28:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc395ffb640 1 -- 172.21.15.16:0/1591047617 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4049070 con 0x563019aa9390 2024-08-28T19:28:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc395ffb640 1 -- 172.21.15.16:0/1591047617 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7fc3a403bab0 con 0x563019aa9390 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/1591047617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3a40cbb30 con 0x563019aa9390 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc395ffb640 1 -- 172.21.15.16:0/1591047617 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fc3a40cb790 con 0x563019aa9390 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3bb0ad740 10 client.17406 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.147+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/1591047617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x563019e83440 con 0x563019aa9390 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.148+0000 7fc395ffb640 1 -- 172.21.15.16:0/1591047617 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3a40cbb30 con 0x563019aa9390 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.148+0000 7fc395ffb640 1 -- 172.21.15.16:0/1591047617 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fc3a40b71c0 con 0x563019aa9390 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.148+0000 7fc3bb0ad740 10 client.17406 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.148+0000 7fc3bb0ad740 10 client.17406 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.148+0000 7fc3bb0ad740 20 client.17406 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.148+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563019a5dda0 0x563019ef9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.148+0000 7fc3bb0ad740 4 client.17406 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.148+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/1591047617 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563019de4070 con 0x563019a5dda0 2024-08-28T19:28:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.148+0000 7fc3ae59c640 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563019a5dda0 0x563019ef9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3ae59c640 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563019a5dda0 0x563019ef9570 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=47084fbb49350c83 server_cookie=76ec88b86a57b110 in_seq=0 out_seq=0 2024-08-28T19:28:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc395ffb640 10 client.17406 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:28:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc395ffb640 1 -- 172.21.15.16:0/1591047617 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563019de4070 con 0x563019a5dda0 2024-08-28T19:28:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc395ffb640 10 client.17406 handle_command_reply: tid=0 2024-08-28T19:28:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 1 client.17406 shutdown 2024-08-28T19:28:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/1591047617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 msgr2=0x563019e82f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e82f00 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40d53b0 tx=0x7fc3a40c7ff0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/1591047617 shutdown_connections 2024-08-28T19:28:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563019a5dda0 0x563019ef9570 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563019aca070 0x563019e7d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563019aa9390 0x563019e82f00 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 1 --2- 172.21.15.16:0/1591047617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563019aa11b0 0x563019e829c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/1591047617 >> 172.21.15.16:0/1591047617 conn(0x563019abe460 msgr2=0x563019e78ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc395ffb640 0 client.17406 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:28:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/1591047617 shutdown_connections 2024-08-28T19:28:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 1 -- 172.21.15.16:0/1591047617 wait complete. 2024-08-28T19:28:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.149+0000 7fc3bb0ad740 20 client.17406 trim_cache size 0 max 16384 2024-08-28T19:28:21.164 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:28:21.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2854314014 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3980b4d20 msgr2=0x7fc3980b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:21.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3980b4d20 0x7fc3980b7230 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8005c40 tx=0x7fc3a8004ab0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2854314014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3b010b470 msgr2=0x7fc3b019ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:21.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3b010b470 0x7fc3b019ca30 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fc3a402a930 tx=0x7fc3a4031a20 comp rx=0 tx=0).stop 2024-08-28T19:28:21.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2854314014 shutdown_connections 2024-08-28T19:28:21.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3980b4d20 0x7fc3980b7230 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b010bdc0 0x7fc3b01a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3b010b470 0x7fc3b019ca30 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 --2- 172.21.15.16:0/2854314014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3b0105b30 0x7fc3b019c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:21.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2854314014 >> 172.21.15.16:0/2854314014 conn(0x7fc3b0101600 msgr2=0x7fc3b0103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:21.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2854314014 shutdown_connections 2024-08-28T19:28:21.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:21.150+0000 7fc3b4b38640 1 -- 172.21.15.16:0/2854314014 wait complete. 2024-08-28T19:28:21.479 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:21 smithi016 ceph-mon[18556]: pgmap v560: 129 pgs: 129 active+clean; 224 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 186 op/s 2024-08-28T19:28:21.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:21 smithi028 ceph-mon[21769]: pgmap v560: 129 pgs: 129 active+clean; 224 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 186 op/s 2024-08-28T19:28:21.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:21 smithi115 ceph-mon[22184]: pgmap v560: 129 pgs: 129 active+clean; 224 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 186 op/s 2024-08-28T19:28:21.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:28:21.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 349302 4.81 MB/sec execute 241 sec latency 3119.074 ms 2024-08-28T19:28:22.094 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:28:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.574+0000 7ff96e0ad640 1 -- 172.21.15.16:0/2647902062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff968108140 msgr2=0x7ff968108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.574+0000 7ff96e0ad640 1 --2- 172.21.15.16:0/2647902062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff968108140 0x7ff968108550 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7ff950009900 tx=0x7ff95002a410 comp rx=0 tx=0).stop 2024-08-28T19:28:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.574+0000 7ff96e0ad640 1 -- 172.21.15.16:0/2647902062 shutdown_connections 2024-08-28T19:28:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.574+0000 7ff96e0ad640 1 --2- 172.21.15.16:0/2647902062 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff968108a90 0x7ff96810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.574+0000 7ff96e0ad640 1 --2- 172.21.15.16:0/2647902062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff968108140 0x7ff968108550 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.574+0000 7ff96e0ad640 1 --2- 172.21.15.16:0/2647902062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff968102800 0x7ff968102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.574+0000 7ff96e0ad640 1 -- 172.21.15.16:0/2647902062 >> 172.21.15.16:0/2647902062 conn(0x7ff9680794d0 msgr2=0x7ff9680798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.574+0000 7ff96e0ad640 1 -- 172.21.15.16:0/2647902062 shutdown_connections 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.574+0000 7ff96e0ad640 1 -- 172.21.15.16:0/2647902062 wait complete. 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.575+0000 7ff96e0ad640 1 Processor -- start 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.575+0000 7ff96e0ad640 1 -- start start 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.575+0000 7ff96e0ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff968102800 0x7ff96810fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.575+0000 7ff96e0ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff968108140 0x7ff968110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.575+0000 7ff96e0ad640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff968108a90 0x7ff9681107d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.575+0000 7ff96e0ad640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff968113850 con 0x7ff968102800 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.575+0000 7ff96e0ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9681136d0 con 0x7ff968108140 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.575+0000 7ff96e0ad640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff9681139d0 con 0x7ff968108a90 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.575+0000 7ff9677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff968102800 0x7ff96810fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.575+0000 7ff9677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff968102800 0x7ff96810fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41798/0 (socket says 172.21.15.16:41798) 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.575+0000 7ff9677fe640 1 -- 172.21.15.16:0/1451659354 learned_addr learned my addr 172.21.15.16:0/1451659354 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff967fff640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff968108a90 0x7ff9681107d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff95effd640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff968108140 0x7ff968110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff9677fe640 1 -- 172.21.15.16:0/1451659354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff968108a90 msgr2=0x7ff9681107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff9677fe640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff968108a90 0x7ff9681107d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff9677fe640 1 -- 172.21.15.16:0/1451659354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff968108140 msgr2=0x7ff968110290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff9677fe640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff968108140 0x7ff968110290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff9677fe640 1 -- 172.21.15.16:0/1451659354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9681a8560 con 0x7ff968102800 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff9677fe640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff968102800 0x7ff96810fd50 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7ff95800d950 tx=0x7ff95800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff9657fa640 1 -- 172.21.15.16:0/1451659354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff958017070 con 0x7ff968102800 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff96e0ad640 1 -- 172.21.15.16:0/1451659354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9681a8850 con 0x7ff968102800 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff9657fa640 1 -- 172.21.15.16:0/1451659354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff958009c30 con 0x7ff968102800 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff9657fa640 1 -- 172.21.15.16:0/1451659354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95800f620 con 0x7ff968102800 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.576+0000 7ff96e0ad640 1 -- 172.21.15.16:0/1451659354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9681a8d60 con 0x7ff968102800 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.579+0000 7ff974622740 1 Processor -- start 2024-08-28T19:28:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.579+0000 7ff974622740 1 -- start start 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.579+0000 7ff974622740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 0x55e0f347d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.579+0000 7ff974622740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 0x55e0f347d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.579+0000 7ff974622740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f347ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.579+0000 7ff974622740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e0f33f4680 con 0x55e0f30d8070 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.579+0000 7ff974622740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e0f33e3c10 con 0x55e0f30b7390 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.579+0000 7ff974622740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e0f33e7bc0 con 0x55e0f30af1b0 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.579+0000 7ff9657fa640 1 -- 172.21.15.16:0/1451659354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff95800f7c0 con 0x7ff968102800 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.579+0000 7ff967fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 0x55e0f347d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.579+0000 7ff967fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 0x55e0f347d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41814/0 (socket says 172.21.15.16:41814) 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff967fff640 1 -- 172.21.15.16:0/514411076 learned_addr learned my addr 172.21.15.16:0/514411076 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff95effd640 1 --2- 172.21.15.16:0/514411076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f347ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff9677fe640 1 --2- 172.21.15.16:0/514411076 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 0x55e0f347d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff967fff640 1 -- 172.21.15.16:0/514411076 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 msgr2=0x55e0f347d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff967fff640 1 --2- 172.21.15.16:0/514411076 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 0x55e0f347d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff967fff640 1 -- 172.21.15.16:0/514411076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 msgr2=0x55e0f347ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff967fff640 1 --2- 172.21.15.16:0/514411076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f347ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff967fff640 1 -- 172.21.15.16:0/514411076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f3143650 con 0x55e0f30d8070 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff967fff640 1 --2- 172.21.15.16:0/514411076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 0x55e0f347d350 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ff960009f50 tx=0x7ff9600118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e18d68b4915b7d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff95dffb640 1 -- 172.21.15.16:0/514411076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960004d20 con 0x55e0f30d8070 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff95dffb640 1 -- 172.21.15.16:0/514411076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff960004050 con 0x55e0f30d8070 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff95dffb640 1 -- 172.21.15.16:0/514411076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960005220 con 0x55e0f30d8070 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.580+0000 7ff9657fa640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9540b4ca0 0x7ff9540b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff9677fe640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9540b4ca0 0x7ff9540b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff9657fa640 1 -- 172.21.15.16:0/1451659354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7ff9580e54f0 con 0x7ff968102800 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff974622740 1 -- 172.21.15.16:0/514411076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 msgr2=0x55e0f347d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff9677fe640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9540b4ca0 0x7ff9540b71b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff9580e2920 tx=0x7ff95801e000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff974622740 1 --2- 172.21.15.16:0/514411076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 0x55e0f347d350 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ff960009f50 tx=0x7ff9600118d0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff974622740 1 -- 172.21.15.16:0/514411076 shutdown_connections 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff974622740 1 --2- 172.21.15.16:0/514411076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f347ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff974622740 1 --2- 172.21.15.16:0/514411076 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 0x55e0f347d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff974622740 1 --2- 172.21.15.16:0/514411076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 0x55e0f347d350 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff974622740 1 -- 172.21.15.16:0/514411076 >> 172.21.15.16:0/514411076 conn(0x55e0f30cc460 msgr2=0x55e0f347b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff974622740 1 -- 172.21.15.16:0/514411076 shutdown_connections 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.581+0000 7ff974622740 1 -- 172.21.15.16:0/514411076 wait complete. 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.582+0000 7ff974622740 1 -- start start 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.582+0000 7ff974622740 1 Processor -- start 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.582+0000 7ff974622740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30af1b0 0x55e0f3488ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.582+0000 7ff974622740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30b7390 0x55e0f3489000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.582+0000 7ff974622740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 0x55e0f3494e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.582+0000 7ff974622740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e0f3246700 con 0x55e0f30d8070 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.582+0000 7ff974622740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e0f336a750 con 0x55e0f30af1b0 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.582+0000 7ff974622740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e0f348bb30 con 0x55e0f30b7390 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.582+0000 7ff974622740 10 client.0 _ll_register_callbacks cb 0x55e0f3399e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff967fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30af1b0 0x55e0f3488ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff95effd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30b7390 0x55e0f3489000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff967fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30af1b0 0x55e0f3488ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37258/0 (socket says 172.21.15.16:37258) 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff967fff640 1 -- 172.21.15.16:0/1131872176 learned_addr learned my addr 172.21.15.16:0/1131872176 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff967fff640 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 0x55e0f3494e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff95effd640 1 -- 172.21.15.16:0/1131872176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30af1b0 msgr2=0x55e0f3488ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff95effd640 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30af1b0 0x55e0f3488ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff95effd640 1 -- 172.21.15.16:0/1131872176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 msgr2=0x55e0f3494e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff95effd640 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 0x55e0f3494e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff95effd640 1 -- 172.21.15.16:0/1131872176 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0f33a3900 con 0x55e0f30b7390 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff95effd640 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30b7390 0x55e0f3489000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff95002a930 tx=0x7ff9500044b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6861eae447aa04c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff946ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff946ffd640 1 -- 172.21.15.16:0/1131872176 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950035040 con 0x55e0f30b7390 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff974622740 1 -- 172.21.15.16:0/1131872176 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff960004050 con 0x55e0f30b7390 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.583+0000 7ff946ffd640 1 -- 172.21.15.16:0/1131872176 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7ff950005590 con 0x55e0f30b7390 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff946ffd640 1 -- 172.21.15.16:0/1131872176 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff95004f050 con 0x55e0f30b7390 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff974622740 10 client.26764 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff974622740 1 -- 172.21.15.16:0/1131872176 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e0f3497950 con 0x55e0f30b7390 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff946ffd640 1 -- 172.21.15.16:0/1131872176 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff960004050 con 0x55e0f30b7390 2024-08-28T19:28:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff946ffd640 1 -- 172.21.15.16:0/1131872176 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7ff9500039d0 con 0x55e0f30b7390 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff974622740 10 client.26764 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff974622740 10 client.26764 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff974622740 20 client.26764 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff974622740 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e0f30969f0 0x55e0f349adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff974622740 4 client.26764 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff974622740 1 -- 172.21.15.16:0/1131872176 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0f33f65d0 con 0x55e0f30969f0 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.584+0000 7ff967fff640 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e0f30969f0 0x55e0f349adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.585+0000 7ff946ffd640 10 client.26764 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.585+0000 7ff967fff640 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e0f30969f0 0x55e0f349adf0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f973c60ad9d2b231 server_cookie=4ebfa6afd5b39640 in_seq=0 out_seq=0 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff946ffd640 1 -- 172.21.15.16:0/1131872176 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e0f33f65d0 con 0x55e0f30969f0 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff946ffd640 10 client.26764 handle_command_reply: tid=0 2024-08-28T19:28:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff974622740 1 client.26764 shutdown 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff974622740 1 -- 172.21.15.16:0/1131872176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30b7390 msgr2=0x55e0f3489000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff974622740 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30b7390 0x55e0f3489000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff95002a930 tx=0x7ff9500044b0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff974622740 1 -- 172.21.15.16:0/1131872176 shutdown_connections 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff974622740 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e0f30969f0 0x55e0f349adf0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff946ffd640 0 client.26764 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff974622740 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30d8070 0x55e0f3494e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff974622740 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30b7390 0x55e0f3489000 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff974622740 1 --2- 172.21.15.16:0/1131872176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30af1b0 0x55e0f3488ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff974622740 1 -- 172.21.15.16:0/1131872176 >> 172.21.15.16:0/1131872176 conn(0x55e0f30cc460 msgr2=0x55e0f3480530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff974622740 1 -- 172.21.15.16:0/1131872176 shutdown_connections 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.586+0000 7ff974622740 1 -- 172.21.15.16:0/1131872176 wait complete. 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.587+0000 7ff974622740 20 client.26764 trim_cache size 0 max 16384 2024-08-28T19:28:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.601+0000 7ff974622740 1 Processor -- start 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.601+0000 7ff974622740 1 -- start start 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.601+0000 7ff974622740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30969f0 0x55e0f347ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.601+0000 7ff974622740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30af1b0 0x55e0f3494d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.601+0000 7ff974622740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f3485cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.601+0000 7ff974622740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9681b2ea0 con 0x55e0f30af1b0 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.601+0000 7ff974622740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9681b4ad0 con 0x55e0f30b7390 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.601+0000 7ff974622740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff9681b3570 con 0x55e0f30969f0 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.601+0000 7ff95effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30af1b0 0x55e0f3494d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff95effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30af1b0 0x55e0f3494d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41846/0 (socket says 172.21.15.16:41846) 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff967fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f3485cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff95effd640 1 -- 172.21.15.16:0/3594930864 learned_addr learned my addr 172.21.15.16:0/3594930864 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff967fff640 1 --2- 172.21.15.16:0/3594930864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30969f0 0x55e0f347ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff95effd640 1 -- 172.21.15.16:0/3594930864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30969f0 msgr2=0x55e0f347ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff95effd640 1 --2- 172.21.15.16:0/3594930864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30969f0 0x55e0f347ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff95effd640 1 -- 172.21.15.16:0/3594930864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 msgr2=0x55e0f3485cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff95effd640 1 --2- 172.21.15.16:0/3594930864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f3485cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff95effd640 1 -- 172.21.15.16:0/3594930864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f3489d80 con 0x55e0f30af1b0 2024-08-28T19:28:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff967fff640 1 --2- 172.21.15.16:0/3594930864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f3485cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:28:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff95effd640 1 --2- 172.21.15.16:0/3594930864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30af1b0 0x55e0f3494d20 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7ff950006010 tx=0x7ff950031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e38dd0aa58c90ab0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff95d7fa640 1 -- 172.21.15.16:0/3594930864 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950059070 con 0x55e0f30af1b0 2024-08-28T19:28:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff95d7fa640 1 -- 172.21.15.16:0/3594930864 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff950041070 con 0x55e0f30af1b0 2024-08-28T19:28:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.602+0000 7ff95d7fa640 1 -- 172.21.15.16:0/3594930864 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950035030 con 0x55e0f30af1b0 2024-08-28T19:28:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.603+0000 7ff974622740 1 -- 172.21.15.16:0/3594930864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30af1b0 msgr2=0x55e0f3494d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.603+0000 7ff974622740 1 --2- 172.21.15.16:0/3594930864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30af1b0 0x55e0f3494d20 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7ff950006010 tx=0x7ff950031630 comp rx=0 tx=0).stop 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.603+0000 7ff974622740 1 -- 172.21.15.16:0/3594930864 shutdown_connections 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.603+0000 7ff974622740 1 --2- 172.21.15.16:0/3594930864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f3485cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.603+0000 7ff974622740 1 --2- 172.21.15.16:0/3594930864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30af1b0 0x55e0f3494d20 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.603+0000 7ff974622740 1 --2- 172.21.15.16:0/3594930864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30969f0 0x55e0f347ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.603+0000 7ff974622740 1 -- 172.21.15.16:0/3594930864 >> 172.21.15.16:0/3594930864 conn(0x55e0f30cc460 msgr2=0x55e0f347e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.603+0000 7ff974622740 1 -- 172.21.15.16:0/3594930864 shutdown_connections 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.603+0000 7ff974622740 1 -- 172.21.15.16:0/3594930864 wait complete. 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.604+0000 7ff974622740 1 -- start start 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.604+0000 7ff974622740 1 Processor -- start 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.604+0000 7ff974622740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30969f0 0x55e0f348d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff974622740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 0x55e0f348dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff974622740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f34fe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff974622740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9681b4370 con 0x55e0f30969f0 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff974622740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9681b3c40 con 0x55e0f30b7390 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff974622740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff9681b0ea0 con 0x55e0f30af1b0 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff95effd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 0x55e0f348dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff967fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30969f0 0x55e0f348d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff95effd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 0x55e0f348dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56526/0 (socket says 172.21.15.16:56526) 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff974622740 10 client.0 _ll_register_callbacks cb 0x7ff950009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff95effd640 1 -- 172.21.15.16:0/3210001033 learned_addr learned my addr 172.21.15.16:0/3210001033 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff95effd640 1 -- 172.21.15.16:0/3210001033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 msgr2=0x55e0f34fe7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff95effd640 1 --2- 172.21.15.16:0/3210001033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f34fe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff95effd640 1 -- 172.21.15.16:0/3210001033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30969f0 msgr2=0x55e0f348d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff95effd640 1 --2- 172.21.15.16:0/3210001033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30969f0 0x55e0f348d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff95effd640 1 -- 172.21.15.16:0/3210001033 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0f34fee40 con 0x55e0f30af1b0 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.605+0000 7ff95effd640 1 --2- 172.21.15.16:0/3210001033 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 0x55e0f348dc80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff95002a450 tx=0x7ff950004140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf832f4d7d232689 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.606+0000 7ff946ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.606+0000 7ff946ffd640 1 -- 172.21.15.16:0/3210001033 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950059070 con 0x55e0f30af1b0 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.606+0000 7ff946ffd640 1 -- 172.21.15.16:0/3210001033 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7ff950003800 con 0x55e0f30af1b0 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.606+0000 7ff974622740 1 -- 172.21.15.16:0/3210001033 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff950041070 con 0x55e0f30af1b0 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.606+0000 7ff946ffd640 1 -- 172.21.15.16:0/3210001033 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff950003a00 con 0x55e0f30af1b0 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.606+0000 7ff974622740 10 client.26770 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.606+0000 7ff974622740 1 -- 172.21.15.16:0/3210001033 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e0f3506180 con 0x55e0f30af1b0 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.606+0000 7ff946ffd640 1 -- 172.21.15.16:0/3210001033 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff950041070 con 0x55e0f30af1b0 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.606+0000 7ff946ffd640 1 -- 172.21.15.16:0/3210001033 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7ff9500046c0 con 0x55e0f30af1b0 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.607+0000 7ff974622740 10 client.26770 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.607+0000 7ff974622740 10 client.26770 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.607+0000 7ff974622740 20 client.26770 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.607+0000 7ff974622740 1 --2- 172.21.15.16:0/3210001033 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e0f30d8070 0x55e0f3508870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.607+0000 7ff974622740 4 client.26770 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.607+0000 7ff974622740 1 -- 172.21.15.16:0/3210001033 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e0f33f2070 con 0x55e0f30d8070 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.607+0000 7ff967fff640 1 --2- 172.21.15.16:0/3210001033 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e0f30d8070 0x55e0f3508870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.607+0000 7ff946ffd640 10 client.26770 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff967fff640 1 --2- 172.21.15.16:0/3210001033 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e0f30d8070 0x55e0f3508870 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7f8282e2a4b5c9a8 server_cookie=6932a23c51a94d7e in_seq=0 out_seq=0 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff946ffd640 1 -- 172.21.15.16:0/3210001033 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e0f33f2070 con 0x55e0f30d8070 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff946ffd640 10 client.26770 handle_command_reply: tid=0 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff974622740 1 client.26770 shutdown 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff974622740 1 -- 172.21.15.16:0/3210001033 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 msgr2=0x55e0f348dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff974622740 1 --2- 172.21.15.16:0/3210001033 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 0x55e0f348dc80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff95002a450 tx=0x7ff950004140 comp rx=0 tx=0).stop 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff974622740 1 -- 172.21.15.16:0/3210001033 shutdown_connections 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff974622740 1 --2- 172.21.15.16:0/3210001033 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e0f30d8070 0x55e0f3508870 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff974622740 1 --2- 172.21.15.16:0/3210001033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0f30b7390 0x55e0f34fe7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff974622740 1 --2- 172.21.15.16:0/3210001033 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0f30af1b0 0x55e0f348dc80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff974622740 1 --2- 172.21.15.16:0/3210001033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0f30969f0 0x55e0f348d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff974622740 1 -- 172.21.15.16:0/3210001033 >> 172.21.15.16:0/3210001033 conn(0x55e0f30cc460 msgr2=0x55e0f3480be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff946ffd640 0 client.26770 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:28:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff974622740 1 -- 172.21.15.16:0/3210001033 shutdown_connections 2024-08-28T19:28:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.608+0000 7ff974622740 1 -- 172.21.15.16:0/3210001033 wait complete. 2024-08-28T19:28:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.609+0000 7ff974622740 20 client.26770 trim_cache size 0 max 16384 2024-08-28T19:28:22.623 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:28:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.609+0000 7ff96e0ad640 1 -- 172.21.15.16:0/1451659354 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9540b4ca0 msgr2=0x7ff9540b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.609+0000 7ff96e0ad640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9540b4ca0 0x7ff9540b71b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff9580e2920 tx=0x7ff95801e000 comp rx=0 tx=0).stop 2024-08-28T19:28:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.609+0000 7ff96e0ad640 1 -- 172.21.15.16:0/1451659354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff968102800 msgr2=0x7ff96810fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.609+0000 7ff96e0ad640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff968102800 0x7ff96810fd50 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7ff95800d950 tx=0x7ff95800de10 comp rx=0 tx=0).stop 2024-08-28T19:28:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.609+0000 7ff96e0ad640 1 -- 172.21.15.16:0/1451659354 shutdown_connections 2024-08-28T19:28:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.609+0000 7ff96e0ad640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff968108a90 0x7ff9681107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.609+0000 7ff96e0ad640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff9540b4ca0 0x7ff9540b71b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.609+0000 7ff96e0ad640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff968108140 0x7ff968110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.609+0000 7ff96e0ad640 1 --2- 172.21.15.16:0/1451659354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff968102800 0x7ff96810fd50 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.609+0000 7ff96e0ad640 1 -- 172.21.15.16:0/1451659354 >> 172.21.15.16:0/1451659354 conn(0x7ff9680794d0 msgr2=0x7ff968104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.610+0000 7ff96e0ad640 1 -- 172.21.15.16:0/1451659354 shutdown_connections 2024-08-28T19:28:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:22.610+0000 7ff96e0ad640 1 -- 172.21.15.16:0/1451659354 wait complete. 2024-08-28T19:28:22.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 351383 4.85 MB/sec execute 242 sec latency 3137.371 ms 2024-08-28T19:28:23.265 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:28:23.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:23 smithi016 ceph-mon[18556]: pgmap v561: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 194 op/s 2024-08-28T19:28:23.555 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:28:23.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:23 smithi028 ceph-mon[21769]: pgmap v561: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 194 op/s 2024-08-28T19:28:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:23 smithi115 ceph-mon[22184]: pgmap v561: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 194 op/s 2024-08-28T19:28:23.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 352415 4.84 MB/sec execute 243 sec latency 56.446 ms 2024-08-28T19:28:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.051+0000 7f15db4cb640 1 -- 172.21.15.16:0/1034890251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15d4104620 msgr2=0x7f15d4104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.051+0000 7f15db4cb640 1 --2- 172.21.15.16:0/1034890251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15d4104620 0x7f15d4104a30 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f15d0009ca0 tx=0x7f15d002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.052+0000 7f15db4cb640 1 -- 172.21.15.16:0/1034890251 shutdown_connections 2024-08-28T19:28:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.052+0000 7f15db4cb640 1 --2- 172.21.15.16:0/1034890251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d410dbd0 0x7f15d41100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.052+0000 7f15db4cb640 1 --2- 172.21.15.16:0/1034890251 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d4105050 0x7f15d410d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.052+0000 7f15db4cb640 1 --2- 172.21.15.16:0/1034890251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15d4104620 0x7f15d4104a30 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.052+0000 7f15db4cb640 1 -- 172.21.15.16:0/1034890251 >> 172.21.15.16:0/1034890251 conn(0x7f15d40794d0 msgr2=0x7f15d40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.052+0000 7f15db4cb640 1 -- 172.21.15.16:0/1034890251 shutdown_connections 2024-08-28T19:28:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.052+0000 7f15db4cb640 1 -- 172.21.15.16:0/1034890251 wait complete. 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.053+0000 7f15db4cb640 1 Processor -- start 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.053+0000 7f15db4cb640 1 -- start start 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.053+0000 7f15db4cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15d4104620 0x7f15d4103f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.053+0000 7f15db4cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d4105050 0x7f15d4102530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.053+0000 7f15db4cb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d410dbd0 0x7f15d4102a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.053+0000 7f15db4cb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15d4113810 con 0x7f15d4104620 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.053+0000 7f15db4cb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15d4113690 con 0x7f15d4105050 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.053+0000 7f15db4cb640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f15d4113990 con 0x7f15d410dbd0 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.053+0000 7f15d9267640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15d4104620 0x7f15d4103f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.053+0000 7f15d9267640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15d4104620 0x7f15d4103f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41868/0 (socket says 172.21.15.16:41868) 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.053+0000 7f15d9267640 1 -- 172.21.15.16:0/2741344153 learned_addr learned my addr 172.21.15.16:0/2741344153 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15d8a66640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d4105050 0x7f15d4102530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15d9a68640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d410dbd0 0x7f15d4102a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15d9267640 1 -- 172.21.15.16:0/2741344153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d410dbd0 msgr2=0x7f15d4102a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15d9267640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d410dbd0 0x7f15d4102a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15d9267640 1 -- 172.21.15.16:0/2741344153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d4105050 msgr2=0x7f15d4102530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15d9267640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d4105050 0x7f15d4102530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15d9267640 1 -- 172.21.15.16:0/2741344153 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15d4103290 con 0x7f15d4104620 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15d9267640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15d4104620 0x7f15d4103f40 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f15d002ade0 tx=0x7f15d0031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15ca7fc640 1 -- 172.21.15.16:0/2741344153 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0003b70 con 0x7f15d4104620 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15db4cb640 1 -- 172.21.15.16:0/2741344153 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15d41acbe0 con 0x7f15d4104620 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15ca7fc640 1 -- 172.21.15.16:0/2741344153 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15d0003d10 con 0x7f15d4104620 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15ca7fc640 1 -- 172.21.15.16:0/2741344153 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d002fa50 con 0x7f15d4104620 2024-08-28T19:28:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.054+0000 7f15db4cb640 1 -- 172.21.15.16:0/2741344153 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15d41ad170 con 0x7f15d4104620 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.057+0000 7f15e1a40740 1 Processor -- start 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.057+0000 7f15e1a40740 1 -- start start 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.057+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.057+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4ad850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.057+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 0x55b29c4add90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.057+0000 7f15e1a40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b29c424680 con 0x55b29c0e7390 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.057+0000 7f15ca7fc640 1 -- 172.21.15.16:0/2741344153 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f15d002fc30 con 0x7f15d4104620 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.057+0000 7f15e1a40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b29c413c10 con 0x55b29c108070 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.057+0000 7f15e1a40740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b29c417bc0 con 0x55b29c0df1b0 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15d9a68640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15d8a66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 0x55b29c4add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15d9a68640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37290/0 (socket says 172.21.15.16:37290) 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15d9a68640 1 -- 172.21.15.16:0/142217765 learned_addr learned my addr 172.21.15.16:0/142217765 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15d9267640 1 --2- 172.21.15.16:0/142217765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15d9a68640 1 -- 172.21.15.16:0/142217765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 msgr2=0x55b29c4ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15d9a68640 1 --2- 172.21.15.16:0/142217765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4ad850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15d9a68640 1 -- 172.21.15.16:0/142217765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 msgr2=0x55b29c4add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15d9a68640 1 --2- 172.21.15.16:0/142217765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 0x55b29c4add90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15d9a68640 1 -- 172.21.15.16:0/142217765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b29c173650 con 0x55b29c108070 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15d9a68640 1 --2- 172.21.15.16:0/142217765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4ad310 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f15cc009ba0 tx=0x7f15cc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa17ce33d968f2b2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15bf7fe640 1 -- 172.21.15.16:0/142217765 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc004230 con 0x55b29c108070 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15bf7fe640 1 -- 172.21.15.16:0/142217765 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15cc009e10 con 0x55b29c108070 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.058+0000 7f15ca7fc640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c00b4ca0 0x7f15c00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.059+0000 7f15bf7fe640 1 -- 172.21.15.16:0/142217765 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc004bc0 con 0x55b29c108070 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.059+0000 7f15d9267640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c00b4ca0 0x7f15c00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.059+0000 7f15ca7fc640 1 -- 172.21.15.16:0/2741344153 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f15d0106b40 con 0x7f15d4104620 2024-08-28T19:28:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.059+0000 7f15e1a40740 1 -- 172.21.15.16:0/142217765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 msgr2=0x55b29c4ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.059+0000 7f15e1a40740 1 --2- 172.21.15.16:0/142217765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4ad310 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f15cc009ba0 tx=0x7f15cc011810 comp rx=0 tx=0).stop 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.059+0000 7f15d9267640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c00b4ca0 0x7f15c00b71b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f15d0103f70 tx=0x7f15d010a690 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.060+0000 7f15e1a40740 1 -- 172.21.15.16:0/142217765 shutdown_connections 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.060+0000 7f15e1a40740 1 --2- 172.21.15.16:0/142217765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 0x55b29c4add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.060+0000 7f15e1a40740 1 --2- 172.21.15.16:0/142217765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4ad850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.060+0000 7f15e1a40740 1 --2- 172.21.15.16:0/142217765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4ad310 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.060+0000 7f15e1a40740 1 -- 172.21.15.16:0/142217765 >> 172.21.15.16:0/142217765 conn(0x55b29c0fc460 msgr2=0x55b29c4ab200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.060+0000 7f15e1a40740 1 -- 172.21.15.16:0/142217765 shutdown_connections 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.060+0000 7f15e1a40740 1 -- 172.21.15.16:0/142217765 wait complete. 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.060+0000 7f15e1a40740 1 -- start start 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.060+0000 7f15e1a40740 1 Processor -- start 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0df1b0 0x55b29c4b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0e7390 0x55b29c4b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15e1a40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b29c276700 con 0x55b29c0df1b0 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15e1a40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b29c39a750 con 0x55b29c108070 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15e1a40740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b29c4bbaf0 con 0x55b29c0e7390 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15e1a40740 10 client.0 _ll_register_callbacks cb 0x55b29c3c9e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15d9a68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0df1b0 0x55b29c4b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15d9a68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0df1b0 0x55b29c4b8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41882/0 (socket says 172.21.15.16:41882) 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15d9a68640 1 -- 172.21.15.16:0/1264071797 learned_addr learned my addr 172.21.15.16:0/1264071797 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15d9a68640 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15d8a66640 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0e7390 0x55b29c4b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15d9a68640 1 -- 172.21.15.16:0/1264071797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0e7390 msgr2=0x55b29c4b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15d9a68640 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0e7390 0x55b29c4b8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15d9a68640 1 -- 172.21.15.16:0/1264071797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 msgr2=0x55b29c4c4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15d9a68640 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4c4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15d9a68640 1 -- 172.21.15.16:0/1264071797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b29c3d3900 con 0x55b29c0df1b0 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.061+0000 7f15d8a66640 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0e7390 0x55b29c4b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.062+0000 7f15bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.062+0000 7f15d9a68640 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0df1b0 0x55b29c4b8a70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f15cc000c00 tx=0x7f15cc004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129ed50f28f09816 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.062+0000 7f15bcff9640 1 -- 172.21.15.16:0/1264071797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc02e060 con 0x55b29c0df1b0 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.062+0000 7f15bcff9640 1 -- 172.21.15.16:0/1264071797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f15cc028810 con 0x55b29c0df1b0 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.062+0000 7f15e1a40740 1 -- 172.21.15.16:0/1264071797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15cc009e10 con 0x55b29c0df1b0 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.062+0000 7f15bcff9640 1 -- 172.21.15.16:0/1264071797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f15cc035050 con 0x55b29c0df1b0 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.062+0000 7f15e1a40740 10 client.17436 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.062+0000 7f15e1a40740 1 -- 172.21.15.16:0/1264071797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b29c4c78a0 con 0x55b29c0df1b0 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.062+0000 7f15bcff9640 1 -- 172.21.15.16:0/1264071797 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15cc009e10 con 0x55b29c0df1b0 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.062+0000 7f15bcff9640 1 -- 172.21.15.16:0/1264071797 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f15cc018070 con 0x55b29c0df1b0 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.063+0000 7f15e1a40740 10 client.17436 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.063+0000 7f15e1a40740 10 client.17436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:28:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.063+0000 7f15e1a40740 20 client.17436 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.063+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b29c0c69f0 0x55b29c4cad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.063+0000 7f15e1a40740 4 client.17436 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.063+0000 7f15e1a40740 1 -- 172.21.15.16:0/1264071797 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b29c4265d0 con 0x55b29c0c69f0 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.063+0000 7f15d8a66640 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b29c0c69f0 0x55b29c4cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.063+0000 7f15d8a66640 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b29c0c69f0 0x55b29c4cad10 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=594820c3379ee531 server_cookie=8d17014f8e546bbf in_seq=0 out_seq=0 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.063+0000 7f15bcff9640 10 client.17436 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15bcff9640 1 -- 172.21.15.16:0/1264071797 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b29c4265d0 con 0x55b29c0c69f0 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15bcff9640 10 client.17436 handle_command_reply: tid=0 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15e1a40740 1 client.17436 shutdown 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15e1a40740 1 -- 172.21.15.16:0/1264071797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0df1b0 msgr2=0x55b29c4b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0df1b0 0x55b29c4b8a70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f15cc000c00 tx=0x7f15cc004840 comp rx=0 tx=0).stop 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15e1a40740 1 -- 172.21.15.16:0/1264071797 shutdown_connections 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4c4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b29c0c69f0 0x55b29c4cad10 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0e7390 0x55b29c4b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1264071797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0df1b0 0x55b29c4b8a70 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15bcff9640 0 client.17436 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.064+0000 7f15e1a40740 1 -- 172.21.15.16:0/1264071797 >> 172.21.15.16:0/1264071797 conn(0x55b29c0fc460 msgr2=0x55b29c4b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.065+0000 7f15e1a40740 1 -- 172.21.15.16:0/1264071797 shutdown_connections 2024-08-28T19:28:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.065+0000 7f15e1a40740 1 -- 172.21.15.16:0/1264071797 wait complete. 2024-08-28T19:28:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.065+0000 7f15e1a40740 20 client.17436 trim_cache size 0 max 16384 2024-08-28T19:28:24.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15e1a40740 1 Processor -- start 2024-08-28T19:28:24.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15e1a40740 1 -- start start 2024-08-28T19:28:24.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c0e7390 0x55b29c4c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c108070 0x55b29c4aab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15e1a40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41b72f0 con 0x55b29c108070 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15e1a40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41b8f20 con 0x55b29c0e7390 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15e1a40740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41b79c0 con 0x55b29c0df1b0 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15d8a66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c0e7390 0x55b29c4c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15d8a66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c0e7390 0x55b29c4c4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37306/0 (socket says 172.21.15.16:37306) 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15d8a66640 1 -- 172.21.15.16:0/4255717644 learned_addr learned my addr 172.21.15.16:0/4255717644 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.080+0000 7f15d9a68640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.081+0000 7f15d9a68640 1 --2- 172.21.15.16:0/4255717644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c108070 0x55b29c4aab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.081+0000 7f15d8a66640 1 -- 172.21.15.16:0/4255717644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 msgr2=0x55b29c4b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.081+0000 7f15d8a66640 1 --2- 172.21.15.16:0/4255717644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.081+0000 7f15d8a66640 1 -- 172.21.15.16:0/4255717644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c108070 msgr2=0x55b29c4aab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:24.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.081+0000 7f15d8a66640 1 --2- 172.21.15.16:0/4255717644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c108070 0x55b29c4aab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.081+0000 7f15d8a66640 1 -- 172.21.15.16:0/4255717644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b29c4bb620 con 0x55b29c0e7390 2024-08-28T19:28:24.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.081+0000 7f15d8a66640 1 --2- 172.21.15.16:0/4255717644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c0e7390 0x55b29c4c4ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f15c4005fd0 tx=0x7f15c400b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e6f58704672897c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:24.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.081+0000 7f15beffd640 1 -- 172.21.15.16:0/4255717644 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c4005020 con 0x55b29c0e7390 2024-08-28T19:28:24.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.081+0000 7f15beffd640 1 -- 172.21.15.16:0/4255717644 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15c40047a0 con 0x55b29c0e7390 2024-08-28T19:28:24.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.081+0000 7f15beffd640 1 -- 172.21.15.16:0/4255717644 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c40116a0 con 0x55b29c0e7390 2024-08-28T19:28:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.082+0000 7f15e1a40740 1 -- 172.21.15.16:0/4255717644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c0e7390 msgr2=0x55b29c4c4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.082+0000 7f15e1a40740 1 --2- 172.21.15.16:0/4255717644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c0e7390 0x55b29c4c4ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f15c4005fd0 tx=0x7f15c400b390 comp rx=0 tx=0).stop 2024-08-28T19:28:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.082+0000 7f15e1a40740 1 -- 172.21.15.16:0/4255717644 shutdown_connections 2024-08-28T19:28:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.082+0000 7f15e1a40740 1 --2- 172.21.15.16:0/4255717644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c108070 0x55b29c4aab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.082+0000 7f15e1a40740 1 --2- 172.21.15.16:0/4255717644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c0e7390 0x55b29c4c4ce0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.082+0000 7f15e1a40740 1 --2- 172.21.15.16:0/4255717644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.082+0000 7f15e1a40740 1 -- 172.21.15.16:0/4255717644 >> 172.21.15.16:0/4255717644 conn(0x55b29c0fc460 msgr2=0x55b29c4b0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.082+0000 7f15e1a40740 1 -- 172.21.15.16:0/4255717644 shutdown_connections 2024-08-28T19:28:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.082+0000 7f15e1a40740 1 -- 172.21.15.16:0/4255717644 wait complete. 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15e1a40740 1 -- start start 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15e1a40740 1 Processor -- start 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4bce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 0x55b29c4bd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15e1a40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4bd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15e1a40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41b87c0 con 0x55b29c0e7390 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15e1a40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41b8090 con 0x55b29c108070 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15e1a40740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41b52f0 con 0x55b29c0df1b0 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15d8a66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 0x55b29c4bd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15d8a66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 0x55b29c4bd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41902/0 (socket says 172.21.15.16:41902) 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15d9a68640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4bce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15d8a66640 1 -- 172.21.15.16:0/1070400017 learned_addr learned my addr 172.21.15.16:0/1070400017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.083+0000 7f15e1a40740 10 client.0 _ll_register_callbacks cb 0x7f15c4004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15d9a68640 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4bd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15d8a66640 1 -- 172.21.15.16:0/1070400017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 msgr2=0x55b29c4bce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15d8a66640 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4bce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15d8a66640 1 -- 172.21.15.16:0/1070400017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 msgr2=0x55b29c4bd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15d8a66640 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4bd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15d8a66640 1 -- 172.21.15.16:0/1070400017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b29c4b0170 con 0x55b29c0e7390 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15d8a66640 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 0x55b29c4bd390 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f15c4004450 tx=0x7f15c400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ab046c26b3d947 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15bcff9640 1 -- 172.21.15.16:0/1070400017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c4004c60 con 0x55b29c0e7390 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15e1a40740 1 -- 172.21.15.16:0/1070400017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15c40047a0 con 0x55b29c0e7390 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15bcff9640 1 -- 172.21.15.16:0/1070400017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f15c4005020 con 0x55b29c0e7390 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.084+0000 7f15bcff9640 1 -- 172.21.15.16:0/1070400017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f15c4038050 con 0x55b29c0e7390 2024-08-28T19:28:24.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.085+0000 7f15e1a40740 10 client.17442 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.085+0000 7f15e1a40740 1 -- 172.21.15.16:0/1070400017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b29c535750 con 0x55b29c0e7390 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.085+0000 7f15bcff9640 1 -- 172.21.15.16:0/1070400017 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15c40047a0 con 0x55b29c0e7390 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.085+0000 7f15bcff9640 1 -- 172.21.15.16:0/1070400017 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f15c40119e0 con 0x55b29c0e7390 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.085+0000 7f15e1a40740 10 client.17442 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.085+0000 7f15e1a40740 10 client.17442 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.085+0000 7f15e1a40740 20 client.17442 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.085+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b29c09bda0 0x55b29c537ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.085+0000 7f15e1a40740 4 client.17442 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.085+0000 7f15e1a40740 1 -- 172.21.15.16:0/1070400017 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b29c4b93e0 con 0x55b29c09bda0 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.085+0000 7f15d9a68640 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b29c09bda0 0x55b29c537ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15bcff9640 10 client.17442 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15d9a68640 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b29c09bda0 0x55b29c537ee0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db83278791273552 server_cookie=ca5499bdc723c94e in_seq=0 out_seq=0 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15bcff9640 1 -- 172.21.15.16:0/1070400017 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b29c4b93e0 con 0x55b29c09bda0 2024-08-28T19:28:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15bcff9640 10 client.17442 handle_command_reply: tid=0 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15e1a40740 1 client.17442 shutdown 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15e1a40740 1 -- 172.21.15.16:0/1070400017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 msgr2=0x55b29c4bd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 0x55b29c4bd390 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f15c4004450 tx=0x7f15c400be60 comp rx=0 tx=0).stop 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15e1a40740 1 -- 172.21.15.16:0/1070400017 shutdown_connections 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b29c108070 0x55b29c4bd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b29c0e7390 0x55b29c4bd390 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b29c09bda0 0x55b29c537ee0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15e1a40740 1 --2- 172.21.15.16:0/1070400017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b29c0df1b0 0x55b29c4bce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15bcff9640 0 client.17442 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.086+0000 7f15e1a40740 1 -- 172.21.15.16:0/1070400017 >> 172.21.15.16:0/1070400017 conn(0x55b29c0fc460 msgr2=0x55b29c4aaae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15e1a40740 1 -- 172.21.15.16:0/1070400017 shutdown_connections 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15e1a40740 1 -- 172.21.15.16:0/1070400017 wait complete. 2024-08-28T19:28:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15e1a40740 20 client.17442 trim_cache size 0 max 16384 2024-08-28T19:28:24.093 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:28:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 -- 172.21.15.16:0/2741344153 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c00b4ca0 msgr2=0x7f15c00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c00b4ca0 0x7f15c00b71b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f15d0103f70 tx=0x7f15d010a690 comp rx=0 tx=0).stop 2024-08-28T19:28:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 -- 172.21.15.16:0/2741344153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15d4104620 msgr2=0x7f15d4103f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15d4104620 0x7f15d4103f40 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f15d002ade0 tx=0x7f15d0031770 comp rx=0 tx=0).stop 2024-08-28T19:28:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 -- 172.21.15.16:0/2741344153 shutdown_connections 2024-08-28T19:28:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d410dbd0 0x7f15d4102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c00b4ca0 0x7f15c00b71b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d4105050 0x7f15d4102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 --2- 172.21.15.16:0/2741344153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15d4104620 0x7f15d4103f40 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 -- 172.21.15.16:0/2741344153 >> 172.21.15.16:0/2741344153 conn(0x7f15d40794d0 msgr2=0x7f15d41006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 -- 172.21.15.16:0/2741344153 shutdown_connections 2024-08-28T19:28:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:24.087+0000 7f15db4cb640 1 -- 172.21.15.16:0/2741344153 wait complete. 2024-08-28T19:28:24.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 352850 4.83 MB/sec execute 244 sec latency 738.829 ms 2024-08-28T19:28:25.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:25 smithi016 ceph-mon[18556]: pgmap v562: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 157 op/s 2024-08-28T19:28:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:25 smithi028 ceph-mon[21769]: pgmap v562: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 157 op/s 2024-08-28T19:28:25.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:25 smithi115 ceph-mon[22184]: pgmap v562: 129 pgs: 129 active+clean; 227 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 157 op/s 2024-08-28T19:28:25.866 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:28:25.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:28:25.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 352850 4.81 MB/sec execute 245 sec latency 1738.890 ms 2024-08-28T19:28:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:28:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.804+0000 7f5b5caf6640 1 -- 172.21.15.16:0/1769885892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58101fd0 msgr2=0x7f5b581023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.804+0000 7f5b5caf6640 1 --2- 172.21.15.16:0/1769885892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58101fd0 0x7f5b581023e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009900 tx=0x7f5b4802a410 comp rx=0 tx=0).stop 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.804+0000 7f5b5caf6640 1 -- 172.21.15.16:0/1769885892 shutdown_connections 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.804+0000 7f5b5caf6640 1 --2- 172.21.15.16:0/1769885892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b58108260 0x7f5b5810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.804+0000 7f5b5caf6640 1 --2- 172.21.15.16:0/1769885892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b58107910 0x7f5b58107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.804+0000 7f5b5caf6640 1 --2- 172.21.15.16:0/1769885892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58101fd0 0x7f5b581023e0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.804+0000 7f5b5caf6640 1 -- 172.21.15.16:0/1769885892 >> 172.21.15.16:0/1769885892 conn(0x7f5b58079be0 msgr2=0x7f5b58079ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.804+0000 7f5b5caf6640 1 -- 172.21.15.16:0/1769885892 shutdown_connections 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.804+0000 7f5b5caf6640 1 -- 172.21.15.16:0/1769885892 wait complete. 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.805+0000 7f5b5caf6640 1 Processor -- start 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.805+0000 7f5b5caf6640 1 -- start start 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.805+0000 7f5b5caf6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58101fd0 0x7f5b5819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.805+0000 7f5b5caf6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b58107910 0x7f5b5819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.805+0000 7f5b5caf6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b58108260 0x7f5b5819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5caf6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581127a0 con 0x7f5b58101fd0 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5caf6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b58112620 con 0x7f5b58108260 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5caf6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5b58112920 con 0x7f5b58107910 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58101fd0 0x7f5b5819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58101fd0 0x7f5b5819a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57782/0 (socket says 172.21.15.16:57782) 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5659c640 1 -- 172.21.15.16:0/347662874 learned_addr learned my addr 172.21.15.16:0/347662874 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b55d9b640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b58107910 0x7f5b5819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b56d9d640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b58108260 0x7f5b5819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5659c640 1 -- 172.21.15.16:0/347662874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b58107910 msgr2=0x7f5b5819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5659c640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b58107910 0x7f5b5819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5659c640 1 -- 172.21.15.16:0/347662874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b58108260 msgr2=0x7f5b5819fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5659c640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b58108260 0x7f5b5819fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5659c640 1 -- 172.21.15.16:0/347662874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b581a0210 con 0x7f5b58101fd0 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b55d9b640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b58107910 0x7f5b5819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5659c640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58101fd0 0x7f5b5819a2a0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5b48005940 tx=0x7f5b480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b477fe640 1 -- 172.21.15.16:0/347662874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4803f030 con 0x7f5b58101fd0 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5caf6640 1 -- 172.21.15.16:0/347662874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b581a04a0 con 0x7f5b58101fd0 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b477fe640 1 -- 172.21.15.16:0/347662874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b48003cb0 con 0x7f5b58101fd0 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b477fe640 1 -- 172.21.15.16:0/347662874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b48035070 con 0x7f5b58101fd0 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.806+0000 7f5b5caf6640 1 -- 172.21.15.16:0/347662874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b581a6c60 con 0x7f5b58101fd0 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.809+0000 7f5b6306b740 1 Processor -- start 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.809+0000 7f5b477fe640 1 -- 172.21.15.16:0/347662874 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5b48042030 con 0x7f5b58101fd0 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.809+0000 7f5b6306b740 1 -- start start 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb65236350 0x55fb655dc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb6520d490 0x55fb655dccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655dd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b6306b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fb655525f0 con 0x55fb65215670 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b6306b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb65541b80 con 0x55fb6520d490 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b6306b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55fb65545b30 con 0x55fb65236350 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b55d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655dd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b55d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655dd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57798/0 (socket says 172.21.15.16:57798) 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b55d9b640 1 -- 172.21.15.16:0/992879598 learned_addr learned my addr 172.21.15.16:0/992879598 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b56d9d640 1 --2- 172.21.15.16:0/992879598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb65236350 0x55fb655dc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b5659c640 1 --2- 172.21.15.16:0/992879598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb6520d490 0x55fb655dccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b55d9b640 1 -- 172.21.15.16:0/992879598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb65236350 msgr2=0x55fb655dc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b55d9b640 1 --2- 172.21.15.16:0/992879598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb65236350 0x55fb655dc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b55d9b640 1 -- 172.21.15.16:0/992879598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb6520d490 msgr2=0x55fb655dccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b55d9b640 1 --2- 172.21.15.16:0/992879598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb6520d490 0x55fb655dccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b55d9b640 1 -- 172.21.15.16:0/992879598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb652a1930 con 0x55fb65215670 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b477fe640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b400b4ca0 0x7f5b400b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.810+0000 7f5b56d9d640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b400b4ca0 0x7f5b400b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.811+0000 7f5b55d9b640 1 --2- 172.21.15.16:0/992879598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655dd200 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00aea0 tx=0x7f5b4c00af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4085984d046bc3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.811+0000 7f5b477fe640 1 -- 172.21.15.16:0/347662874 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f5b481062a0 con 0x7f5b58101fd0 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.811+0000 7f5b44ff9640 1 -- 172.21.15.16:0/992879598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c004d20 con 0x55fb65215670 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.811+0000 7f5b44ff9640 1 -- 172.21.15.16:0/992879598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b4c011b00 con 0x55fb65215670 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.811+0000 7f5b44ff9640 1 -- 172.21.15.16:0/992879598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c0055f0 con 0x55fb65215670 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b6306b740 1 -- 172.21.15.16:0/992879598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 msgr2=0x55fb655dd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b56d9d640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b400b4ca0 0x7f5b400b71b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000bca0 tx=0x7f5b5000b450 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b6306b740 1 --2- 172.21.15.16:0/992879598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655dd200 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00aea0 tx=0x7f5b4c00af90 comp rx=0 tx=0).stop 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b6306b740 1 -- 172.21.15.16:0/992879598 shutdown_connections 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b6306b740 1 --2- 172.21.15.16:0/992879598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655dd200 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b6306b740 1 --2- 172.21.15.16:0/992879598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb6520d490 0x55fb655dccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b6306b740 1 --2- 172.21.15.16:0/992879598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb65236350 0x55fb655dc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b6306b740 1 -- 172.21.15.16:0/992879598 >> 172.21.15.16:0/992879598 conn(0x55fb6522a740 msgr2=0x55fb655da670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b6306b740 1 -- 172.21.15.16:0/992879598 shutdown_connections 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b6306b740 1 -- 172.21.15.16:0/992879598 wait complete. 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b6306b740 1 -- start start 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.812+0000 7f5b6306b740 1 Processor -- start 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb6520d490 0x55fb655ec300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655ec840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65236350 0x55fb655f85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b6306b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fb653a4890 con 0x55fb65215670 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b6306b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb654c8790 con 0x55fb65236350 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b6306b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55fb655ef290 con 0x55fb6520d490 2024-08-28T19:28:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b6306b740 10 client.0 _ll_register_callbacks cb 0x55fb6524dba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b55d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655ec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b55d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655ec840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57802/0 (socket says 172.21.15.16:57802) 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b55d9b640 1 -- 172.21.15.16:0/2750602507 learned_addr learned my addr 172.21.15.16:0/2750602507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b56d9d640 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65236350 0x55fb655f85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b5659c640 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb6520d490 0x55fb655ec300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b55d9b640 1 -- 172.21.15.16:0/2750602507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb6520d490 msgr2=0x55fb655ec300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b55d9b640 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb6520d490 0x55fb655ec300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b55d9b640 1 -- 172.21.15.16:0/2750602507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65236350 msgr2=0x55fb655f85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b55d9b640 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65236350 0x55fb655f85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b55d9b640 1 -- 172.21.15.16:0/2750602507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb654fff30 con 0x55fb65215670 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b55d9b640 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655ec840 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00ff30 tx=0x7f5b4c011880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f6fbeb525066dc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b3dffb640 1 -- 172.21.15.16:0/2750602507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c004c50 con 0x55fb65215670 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b6306b740 1 -- 172.21.15.16:0/2750602507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b4c011b00 con 0x55fb65215670 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.813+0000 7f5b3dffb640 1 -- 172.21.15.16:0/2750602507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f5b4c004df0 con 0x55fb65215670 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b3dffb640 1 -- 172.21.15.16:0/2750602507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5b4c02f050 con 0x55fb65215670 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b6306b740 10 client.17454 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b6306b740 1 -- 172.21.15.16:0/2750602507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55fb655fb0a0 con 0x55fb65215670 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b3dffb640 1 -- 172.21.15.16:0/2750602507 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b4c011b00 con 0x55fb65215670 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b3dffb640 1 -- 172.21.15.16:0/2750602507 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f5b4c0196a0 con 0x55fb65215670 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b6306b740 10 client.17454 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b6306b740 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b6306b740 20 client.17454 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b6306b740 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb651f4cd0 0x55fb655fe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b6306b740 4 client.17454 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b6306b740 1 -- 172.21.15.16:0/2750602507 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb65554540 con 0x55fb651f4cd0 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.814+0000 7f5b5659c640 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb651f4cd0 0x55fb655fe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.815+0000 7f5b3dffb640 10 client.17454 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.815+0000 7f5b5659c640 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb651f4cd0 0x55fb655fe4b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec2b5a2c73b09ac0 server_cookie=cbf04d576b0d8379 in_seq=0 out_seq=0 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b3dffb640 1 -- 172.21.15.16:0/2750602507 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55fb65554540 con 0x55fb651f4cd0 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b3dffb640 10 client.17454 handle_command_reply: tid=0 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 1 client.17454 shutdown 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 1 -- 172.21.15.16:0/2750602507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 msgr2=0x55fb655ec840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655ec840 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00ff30 tx=0x7f5b4c011880 comp rx=0 tx=0).stop 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 1 -- 172.21.15.16:0/2750602507 shutdown_connections 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65236350 0x55fb655f85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65215670 0x55fb655ec840 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb651f4cd0 0x55fb655fe4b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 1 --2- 172.21.15.16:0/2750602507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb6520d490 0x55fb655ec300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 1 -- 172.21.15.16:0/2750602507 >> 172.21.15.16:0/2750602507 conn(0x55fb6522a740 msgr2=0x55fb655d8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b3dffb640 0 client.17454 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 1 -- 172.21.15.16:0/2750602507 shutdown_connections 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 1 -- 172.21.15.16:0/2750602507 wait complete. 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.816+0000 7f5b6306b740 20 client.17454 trim_cache size 0 max 16384 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b6306b740 1 Processor -- start 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b6306b740 1 -- start start 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb6520d490 0x55fb655ee1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65215670 0x55fb655ddec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb65236350 0x55fb655f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b6306b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581aea50 con 0x55fb6520d490 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b6306b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581b0680 con 0x55fb65215670 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b6306b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581af120 con 0x55fb65236350 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b5659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb6520d490 0x55fb655ee1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b5659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb6520d490 0x55fb655ee1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57812/0 (socket says 172.21.15.16:57812) 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b5659c640 1 -- 172.21.15.16:0/2085434424 learned_addr learned my addr 172.21.15.16:0/2085434424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b55d9b640 1 --2- 172.21.15.16:0/2085434424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65215670 0x55fb655ddec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b56d9d640 1 --2- 172.21.15.16:0/2085434424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb65236350 0x55fb655f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b5659c640 1 -- 172.21.15.16:0/2085434424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb65236350 msgr2=0x55fb655f84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b5659c640 1 --2- 172.21.15.16:0/2085434424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb65236350 0x55fb655f84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b5659c640 1 -- 172.21.15.16:0/2085434424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65215670 msgr2=0x55fb655ddec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b5659c640 1 --2- 172.21.15.16:0/2085434424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65215670 0x55fb655ddec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b5659c640 1 -- 172.21.15.16:0/2085434424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb655daff0 con 0x55fb6520d490 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b55d9b640 1 --2- 172.21.15.16:0/2085434424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65215670 0x55fb655ddec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:28:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.834+0000 7f5b56d9d640 1 --2- 172.21.15.16:0/2085434424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb65236350 0x55fb655f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.835+0000 7f5b5659c640 1 --2- 172.21.15.16:0/2085434424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb6520d490 0x55fb655ee1e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5b480c8640 tx=0x7f5b480ef040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343cf8344ca3558f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.835+0000 7f5b3ffff640 1 -- 172.21.15.16:0/2085434424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b48049040 con 0x55fb6520d490 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.835+0000 7f5b3ffff640 1 -- 172.21.15.16:0/2085434424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b480d5220 con 0x55fb6520d490 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.835+0000 7f5b3ffff640 1 -- 172.21.15.16:0/2085434424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b480d0e30 con 0x55fb6520d490 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.835+0000 7f5b6306b740 1 -- 172.21.15.16:0/2085434424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb6520d490 msgr2=0x55fb655ee1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.835+0000 7f5b6306b740 1 --2- 172.21.15.16:0/2085434424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb6520d490 0x55fb655ee1e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5b480c8640 tx=0x7f5b480ef040 comp rx=0 tx=0).stop 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.835+0000 7f5b6306b740 1 -- 172.21.15.16:0/2085434424 shutdown_connections 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.835+0000 7f5b6306b740 1 --2- 172.21.15.16:0/2085434424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb65236350 0x55fb655f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.835+0000 7f5b6306b740 1 --2- 172.21.15.16:0/2085434424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65215670 0x55fb655ddec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.835+0000 7f5b6306b740 1 --2- 172.21.15.16:0/2085434424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb6520d490 0x55fb655ee1e0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.835+0000 7f5b6306b740 1 -- 172.21.15.16:0/2085434424 >> 172.21.15.16:0/2085434424 conn(0x55fb6522a740 msgr2=0x55fb655dd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b6306b740 1 -- 172.21.15.16:0/2085434424 shutdown_connections 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b6306b740 1 -- 172.21.15.16:0/2085434424 wait complete. 2024-08-28T19:28:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b6306b740 1 -- start start 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b6306b740 1 Processor -- start 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb6520d490 0x55fb655f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65215670 0x55fb655f1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b6306b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65236350 0x55fb65667c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b6306b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581aff20 con 0x55fb65236350 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b6306b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581af7f0 con 0x55fb65215670 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b6306b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581aca50 con 0x55fb6520d490 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b6306b740 10 client.0 _ll_register_callbacks cb 0x7f5b48109cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.836+0000 7f5b56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65236350 0x55fb65667c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65236350 0x55fb65667c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57820/0 (socket says 172.21.15.16:57820) 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b56d9d640 1 -- 172.21.15.16:0/1389661964 learned_addr learned my addr 172.21.15.16:0/1389661964 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b55d9b640 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65215670 0x55fb655f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b5659c640 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb6520d490 0x55fb655f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b56d9d640 1 -- 172.21.15.16:0/1389661964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb6520d490 msgr2=0x55fb655f0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b56d9d640 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb6520d490 0x55fb655f0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b56d9d640 1 -- 172.21.15.16:0/1389661964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65215670 msgr2=0x55fb655f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b56d9d640 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65215670 0x55fb655f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b56d9d640 1 -- 172.21.15.16:0/1389661964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb656681c0 con 0x55fb65236350 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b56d9d640 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65236350 0x55fb65667c80 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5b5001f860 tx=0x7f5b500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffdfe2098776f8b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b3dffb640 1 -- 172.21.15.16:0/1389661964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5002d060 con 0x55fb65236350 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b3dffb640 1 -- 172.21.15.16:0/1389661964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f5b5001d610 con 0x55fb65236350 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b6306b740 1 -- 172.21.15.16:0/1389661964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fb6554ffe0 con 0x55fb65236350 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b3dffb640 1 -- 172.21.15.16:0/1389661964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5b50034050 con 0x55fb65236350 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b6306b740 10 client.17460 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.837+0000 7f5b6306b740 1 -- 172.21.15.16:0/1389661964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55fb656693a0 con 0x55fb65236350 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.838+0000 7f5b3dffb640 1 -- 172.21.15.16:0/1389661964 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55fb6554ffe0 con 0x55fb65236350 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.838+0000 7f5b3dffb640 1 -- 172.21.15.16:0/1389661964 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f5b50027070 con 0x55fb65236350 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.838+0000 7f5b6306b740 10 client.17460 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.838+0000 7f5b6306b740 10 client.17460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.838+0000 7f5b6306b740 20 client.17460 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.838+0000 7f5b6306b740 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb651ca080 0x55fb6566bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.838+0000 7f5b6306b740 4 client.17460 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.838+0000 7f5b6306b740 1 -- 172.21.15.16:0/1389661964 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fb6554d720 con 0x55fb651ca080 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.838+0000 7f5b5659c640 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb651ca080 0x55fb6566bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.839+0000 7f5b5659c640 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb651ca080 0x55fb6566bc50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6429873434326004 server_cookie=551133a36466f59e in_seq=0 out_seq=0 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.839+0000 7f5b3dffb640 10 client.17460 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b3dffb640 1 -- 172.21.15.16:0/1389661964 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fb6554d720 con 0x55fb651ca080 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b3dffb640 10 client.17460 handle_command_reply: tid=0 2024-08-28T19:28:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b6306b740 1 client.17460 shutdown 2024-08-28T19:28:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b6306b740 1 -- 172.21.15.16:0/1389661964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65236350 msgr2=0x55fb65667c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b6306b740 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65236350 0x55fb65667c80 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5b5001f860 tx=0x7f5b500046c0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b6306b740 1 -- 172.21.15.16:0/1389661964 shutdown_connections 2024-08-28T19:28:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b6306b740 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fb65236350 0x55fb65667c80 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b6306b740 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb65215670 0x55fb655f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b6306b740 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55fb651ca080 0x55fb6566bc50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b6306b740 1 --2- 172.21.15.16:0/1389661964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fb6520d490 0x55fb655f0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b6306b740 1 -- 172.21.15.16:0/1389661964 >> 172.21.15.16:0/1389661964 conn(0x55fb6522a740 msgr2=0x55fb655ede60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:26.866 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:28:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:28:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "d4cf8d74-0ea8-4831-90ad-1492faca7486", 2024-08-28T19:28:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:28:26.867 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:28:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.840+0000 7f5b3dffb640 0 client.17460 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b6306b740 1 -- 172.21.15.16:0/1389661964 shutdown_connections 2024-08-28T19:28:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b6306b740 1 -- 172.21.15.16:0/1389661964 wait complete. 2024-08-28T19:28:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b6306b740 20 client.17460 trim_cache size 0 max 16384 2024-08-28T19:28:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 -- 172.21.15.16:0/347662874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b400b4ca0 msgr2=0x7f5b400b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b400b4ca0 0x7f5b400b71b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000bca0 tx=0x7f5b5000b450 comp rx=0 tx=0).stop 2024-08-28T19:28:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 -- 172.21.15.16:0/347662874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58101fd0 msgr2=0x7f5b5819a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58101fd0 0x7f5b5819a2a0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5b48005940 tx=0x7f5b480042f0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 -- 172.21.15.16:0/347662874 shutdown_connections 2024-08-28T19:28:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5b400b4ca0 0x7f5b400b71b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b58108260 0x7f5b5819fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b58107910 0x7f5b5819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 --2- 172.21.15.16:0/347662874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58101fd0 0x7f5b5819a2a0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 -- 172.21.15.16:0/347662874 >> 172.21.15.16:0/347662874 conn(0x7f5b58079be0 msgr2=0x7f5b5810f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 -- 172.21.15.16:0/347662874 shutdown_connections 2024-08-28T19:28:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:26.841+0000 7f5b5caf6640 1 -- 172.21.15.16:0/347662874 wait complete. 2024-08-28T19:28:26.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 352934 4.79 MB/sec execute 246 sec latency 2689.698 ms 2024-08-28T19:28:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:27 smithi016 ceph-mon[18556]: pgmap v563: 129 pgs: 129 active+clean; 231 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 197 op/s 2024-08-28T19:28:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:27 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:28:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:27 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:28:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:27 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:27 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:27 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:28:27.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:27 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:28:27.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:27 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:27 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:27 smithi028 ceph-mon[21769]: pgmap v563: 129 pgs: 129 active+clean; 231 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 197 op/s 2024-08-28T19:28:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:27 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:28:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:27 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:28:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:27 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:27 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:27 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:28:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:27 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:28:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:27 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:27 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:27 smithi115 ceph-mon[22184]: pgmap v563: 129 pgs: 129 active+clean; 231 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 197 op/s 2024-08-28T19:28:27.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:27 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:28:27.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:27 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:28:27.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:27 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:27 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:27 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:28:27.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:27 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:28:27.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:27 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:27 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:27.760 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:28:27.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 354972 4.82 MB/sec execute 247 sec latency 12.636 ms 2024-08-28T19:28:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:28:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:28:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:28:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:28:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:28:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:28:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:28:28.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:28 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.535+0000 7f3ae5622640 1 -- 172.21.15.16:0/1529158096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae01050c0 msgr2=0x7f3ae010d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:28.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.535+0000 7f3ae5622640 1 --2- 172.21.15.16:0/1529158096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae01050c0 0x7f3ae010d610 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009900 tx=0x7f3ad402a410 comp rx=0 tx=0).stop 2024-08-28T19:28:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.535+0000 7f3ae5622640 1 -- 172.21.15.16:0/1529158096 shutdown_connections 2024-08-28T19:28:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.535+0000 7f3ae5622640 1 --2- 172.21.15.16:0/1529158096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae010dbd0 0x7f3ae01100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.535+0000 7f3ae5622640 1 --2- 172.21.15.16:0/1529158096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae01050c0 0x7f3ae010d610 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.535+0000 7f3ae5622640 1 --2- 172.21.15.16:0/1529158096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ae0104770 0x7f3ae0104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.535+0000 7f3ae5622640 1 -- 172.21.15.16:0/1529158096 >> 172.21.15.16:0/1529158096 conn(0x7f3ae00794e0 msgr2=0x7f3ae00798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.535+0000 7f3ae5622640 1 -- 172.21.15.16:0/1529158096 shutdown_connections 2024-08-28T19:28:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.535+0000 7f3ae5622640 1 -- 172.21.15.16:0/1529158096 wait complete. 2024-08-28T19:28:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.535+0000 7f3ae5622640 1 Processor -- start 2024-08-28T19:28:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ae5622640 1 -- start start 2024-08-28T19:28:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ae5622640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae0104770 0x7f3ae019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ae5622640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae01050c0 0x7f3ae019a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ae5622640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ae010dbd0 0x7f3ae019fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ae5622640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01137f0 con 0x7f3ae01050c0 2024-08-28T19:28:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ae5622640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae0113670 con 0x7f3ae0104770 2024-08-28T19:28:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ae5622640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae0113970 con 0x7f3ae010dbd0 2024-08-28T19:28:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ade7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae01050c0 0x7f3ae019a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ade7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae01050c0 0x7f3ae019a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57840/0 (socket says 172.21.15.16:57840) 2024-08-28T19:28:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ade7fc640 1 -- 172.21.15.16:0/817899418 learned_addr learned my addr 172.21.15.16:0/817899418 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae0104770 0x7f3ae019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3adf7fe640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ae010dbd0 0x7f3ae019fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ade7fc640 1 -- 172.21.15.16:0/817899418 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ae010dbd0 msgr2=0x7f3ae019fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ae010dbd0 0x7f3ae019fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.536+0000 7f3ade7fc640 1 -- 172.21.15.16:0/817899418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae0104770 msgr2=0x7f3ae019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.537+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae0104770 0x7f3ae019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.537+0000 7f3ade7fc640 1 -- 172.21.15.16:0/817899418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae01a01f0 con 0x7f3ae01050c0 2024-08-28T19:28:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.537+0000 7f3adf7fe640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ae010dbd0 0x7f3ae019fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:28:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.537+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae01050c0 0x7f3ae019a820 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f3ad402a930 tx=0x7f3ad4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.537+0000 7f3ac7fff640 1 -- 172.21.15.16:0/817899418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad403f030 con 0x7f3ae01050c0 2024-08-28T19:28:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.537+0000 7f3ac7fff640 1 -- 172.21.15.16:0/817899418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ad4003aa0 con 0x7f3ae01050c0 2024-08-28T19:28:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.537+0000 7f3ae5622640 1 -- 172.21.15.16:0/817899418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae01a0480 con 0x7f3ae01050c0 2024-08-28T19:28:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.537+0000 7f3ac7fff640 1 -- 172.21.15.16:0/817899418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4035070 con 0x7f3ae01050c0 2024-08-28T19:28:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.537+0000 7f3ae5622640 1 -- 172.21.15.16:0/817899418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae01a6c40 con 0x7f3ae01050c0 2024-08-28T19:28:28.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3aebb97740 1 Processor -- start 2024-08-28T19:28:28.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3aebb97740 1 -- start start 2024-08-28T19:28:28.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29578070 0x561d2991d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 0x561d2991d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d29557390 0x561d2991dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3aebb97740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561d29894630 con 0x561d2954f1b0 2024-08-28T19:28:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3aebb97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d29883bc0 con 0x561d29578070 2024-08-28T19:28:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3aebb97740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561d29887b70 con 0x561d29557390 2024-08-28T19:28:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 0x561d2991d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 0x561d2991d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57848/0 (socket says 172.21.15.16:57848) 2024-08-28T19:28:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3adeffd640 1 -- 172.21.15.16:0/1392334134 learned_addr learned my addr 172.21.15.16:0/1392334134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.540+0000 7f3adf7fe640 1 --2- 172.21.15.16:0/1392334134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29578070 0x561d2991d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.541+0000 7f3adeffd640 1 -- 172.21.15.16:0/1392334134 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d29557390 msgr2=0x561d2991dd40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:28:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.541+0000 7f3adeffd640 1 --2- 172.21.15.16:0/1392334134 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d29557390 0x561d2991dd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.541+0000 7f3adeffd640 1 -- 172.21.15.16:0/1392334134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29578070 msgr2=0x561d2991d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.541+0000 7f3adeffd640 1 --2- 172.21.15.16:0/1392334134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29578070 0x561d2991d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.541+0000 7f3adeffd640 1 -- 172.21.15.16:0/1392334134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d295e3650 con 0x561d2954f1b0 2024-08-28T19:28:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.541+0000 7f3adf7fe640 1 --2- 172.21.15.16:0/1392334134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29578070 0x561d2991d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:28:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.541+0000 7f3ac7fff640 1 -- 172.21.15.16:0/817899418 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3ad4042030 con 0x7f3ae01050c0 2024-08-28T19:28:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.541+0000 7f3adeffd640 1 --2- 172.21.15.16:0/1392334134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 0x561d2991d800 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0003a70 tx=0x7f3ad0003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268206662adcb648 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.541+0000 7f3ac57fa640 1 -- 172.21.15.16:0/1392334134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0012450 con 0x561d2954f1b0 2024-08-28T19:28:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.541+0000 7f3ac57fa640 1 -- 172.21.15.16:0/1392334134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ad0012aa0 con 0x561d2954f1b0 2024-08-28T19:28:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.541+0000 7f3ac57fa640 1 -- 172.21.15.16:0/1392334134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0011b60 con 0x561d2954f1b0 2024-08-28T19:28:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3ac7fff640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ac80b4ca0 0x7f3ac80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3adf7fe640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ac80b4ca0 0x7f3ac80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3ac7fff640 1 -- 172.21.15.16:0/817899418 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f3ad4105df0 con 0x7f3ae01050c0 2024-08-28T19:28:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3aebb97740 1 -- 172.21.15.16:0/1392334134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 msgr2=0x561d2991d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3aebb97740 1 --2- 172.21.15.16:0/1392334134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 0x561d2991d800 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0003a70 tx=0x7f3ad0003f30 comp rx=0 tx=0).stop 2024-08-28T19:28:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3adf7fe640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ac80b4ca0 0x7f3ac80b71b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009d60 tx=0x7f3ad8005dc0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3aebb97740 1 -- 172.21.15.16:0/1392334134 shutdown_connections 2024-08-28T19:28:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3aebb97740 1 --2- 172.21.15.16:0/1392334134 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d29557390 0x561d2991dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3aebb97740 1 --2- 172.21.15.16:0/1392334134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 0x561d2991d800 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3aebb97740 1 --2- 172.21.15.16:0/1392334134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29578070 0x561d2991d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3aebb97740 1 -- 172.21.15.16:0/1392334134 >> 172.21.15.16:0/1392334134 conn(0x561d2956c460 msgr2=0x561d2991b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3aebb97740 1 -- 172.21.15.16:0/1392334134 shutdown_connections 2024-08-28T19:28:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.542+0000 7f3aebb97740 1 -- 172.21.15.16:0/1392334134 wait complete. 2024-08-28T19:28:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.543+0000 7f3aebb97740 1 -- start start 2024-08-28T19:28:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.543+0000 7f3aebb97740 1 Processor -- start 2024-08-28T19:28:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.543+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2954f1b0 0x561d29928b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.543+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29557390 0x561d29929060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.543+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d29578070 0x561d299391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.543+0000 7f3aebb97740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561d296e6700 con 0x561d29578070 2024-08-28T19:28:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.543+0000 7f3aebb97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d2980a700 con 0x561d29557390 2024-08-28T19:28:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.543+0000 7f3aebb97740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561d2992feb0 con 0x561d2954f1b0 2024-08-28T19:28:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.543+0000 7f3aebb97740 10 client.0 _ll_register_callbacks cb 0x561d29839dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ade7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29557390 0x561d29929060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3adf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d29578070 0x561d299391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ade7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29557390 0x561d29929060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54982/0 (socket says 172.21.15.16:54982) 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ade7fc640 1 -- 172.21.15.16:0/3034398538 learned_addr learned my addr 172.21.15.16:0/3034398538 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3adeffd640 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2954f1b0 0x561d29928b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ade7fc640 1 -- 172.21.15.16:0/3034398538 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2954f1b0 msgr2=0x561d29928b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2954f1b0 0x561d29928b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ade7fc640 1 -- 172.21.15.16:0/3034398538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d29578070 msgr2=0x561d299391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d29578070 0x561d299391d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ade7fc640 1 -- 172.21.15.16:0/3034398538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d298438b0 con 0x561d29557390 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29557390 0x561d29929060 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40c8190 tx=0x7f3ad40cba60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=489f96fea8673228 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ac2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:28:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/3034398538 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4049070 con 0x561d29557390 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3aebb97740 1 -- 172.21.15.16:0/3034398538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ad0012aa0 con 0x561d29557390 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.544+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/3034398538 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f3ad41033f0 con 0x561d29557390 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/3034398538 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f3ad40cc110 con 0x561d29557390 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3aebb97740 10 client.27005 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3aebb97740 1 -- 172.21.15.16:0/3034398538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x561d2993bd20 con 0x561d29557390 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/3034398538 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ad0012aa0 con 0x561d29557390 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/3034398538 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f3ad40cc370 con 0x561d29557390 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3aebb97740 10 client.27005 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3aebb97740 10 client.27005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3aebb97740 20 client.27005 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3aebb97740 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561d295369f0 0x561d2993f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3aebb97740 4 client.27005 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3aebb97740 1 -- 172.21.15.16:0/3034398538 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d29896580 con 0x561d295369f0 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.545+0000 7f3adeffd640 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561d295369f0 0x561d2993f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.546+0000 7f3adeffd640 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561d295369f0 0x561d2993f2a0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1e137c24aa5bb70 server_cookie=25dd717c4315ebfb in_seq=0 out_seq=0 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.546+0000 7f3ac2ff5640 10 client.27005 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/3034398538 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561d29896580 con 0x561d295369f0 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3ac2ff5640 10 client.27005 handle_command_reply: tid=0 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 1 client.27005 shutdown 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 1 -- 172.21.15.16:0/3034398538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29557390 msgr2=0x561d29929060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29557390 0x561d29929060 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40c8190 tx=0x7f3ad40cba60 comp rx=0 tx=0).stop 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 1 -- 172.21.15.16:0/3034398538 shutdown_connections 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d29578070 0x561d299391d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d29557390 0x561d29929060 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561d295369f0 0x561d2993f2a0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 1 --2- 172.21.15.16:0/3034398538 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2954f1b0 0x561d29928b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 1 -- 172.21.15.16:0/3034398538 >> 172.21.15.16:0/3034398538 conn(0x561d2956c460 msgr2=0x561d29920450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:28.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3ac2ff5640 0 client.27005 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:28.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 1 -- 172.21.15.16:0/3034398538 shutdown_connections 2024-08-28T19:28:28.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 1 -- 172.21.15.16:0/3034398538 wait complete. 2024-08-28T19:28:28.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.547+0000 7f3aebb97740 20 client.27005 trim_cache size 0 max 16384 2024-08-28T19:28:28.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:28:28.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:28:28.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:28:28.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:28:28.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:28:28.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:28:28.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:28 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.580+0000 7f3aebb97740 1 Processor -- start 2024-08-28T19:28:28.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.583+0000 7f3aebb97740 1 -- start start 2024-08-28T19:28:28.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.583+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 0x561d29928b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.584+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d29578070 0x561d29920ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.584+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d2950bda0 0x561d29925cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.584+0000 7f3aebb97740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01aeae0 con 0x561d2954f1b0 2024-08-28T19:28:28.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.584+0000 7f3aebb97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01b0710 con 0x561d2950bda0 2024-08-28T19:28:28.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.584+0000 7f3aebb97740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01af1b0 con 0x561d29578070 2024-08-28T19:28:28.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.584+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 0x561d29928b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.584+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 0x561d29928b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57866/0 (socket says 172.21.15.16:57866) 2024-08-28T19:28:28.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.584+0000 7f3adeffd640 1 -- 172.21.15.16:0/2805207702 learned_addr learned my addr 172.21.15.16:0/2805207702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:28.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.584+0000 7f3adf7fe640 1 --2- 172.21.15.16:0/2805207702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d2950bda0 0x561d29925cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.585+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/2805207702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d29578070 0x561d29920ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.585+0000 7f3ade7fc640 1 -- 172.21.15.16:0/2805207702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d2950bda0 msgr2=0x561d29925cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.585+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/2805207702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d2950bda0 0x561d29925cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.585+0000 7f3ade7fc640 1 -- 172.21.15.16:0/2805207702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 msgr2=0x561d29928b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.585+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/2805207702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 0x561d29928b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.585+0000 7f3ade7fc640 1 -- 172.21.15.16:0/2805207702 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d29927a60 con 0x561d29578070 2024-08-28T19:28:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.586+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/2805207702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d29578070 0x561d29920ad0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4032000 tx=0x7f3ad4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b2e3203def4db77 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.586+0000 7f3ac4ff9640 1 -- 172.21.15.16:0/2805207702 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad40cb880 con 0x561d29578070 2024-08-28T19:28:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.586+0000 7f3ac4ff9640 1 -- 172.21.15.16:0/2805207702 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ad40d0980 con 0x561d29578070 2024-08-28T19:28:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.586+0000 7f3ac4ff9640 1 -- 172.21.15.16:0/2805207702 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad41033f0 con 0x561d29578070 2024-08-28T19:28:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.588+0000 7f3aebb97740 1 -- 172.21.15.16:0/2805207702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d29578070 msgr2=0x561d29920ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.588+0000 7f3aebb97740 1 --2- 172.21.15.16:0/2805207702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d29578070 0x561d29920ad0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4032000 tx=0x7f3ad4000bc0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.588+0000 7f3aebb97740 1 -- 172.21.15.16:0/2805207702 shutdown_connections 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.588+0000 7f3aebb97740 1 --2- 172.21.15.16:0/2805207702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d2950bda0 0x561d29925cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.588+0000 7f3aebb97740 1 --2- 172.21.15.16:0/2805207702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d29578070 0x561d29920ad0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.588+0000 7f3aebb97740 1 --2- 172.21.15.16:0/2805207702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d2954f1b0 0x561d29928b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.588+0000 7f3aebb97740 1 -- 172.21.15.16:0/2805207702 >> 172.21.15.16:0/2805207702 conn(0x561d2956c460 msgr2=0x561d299203c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.588+0000 7f3aebb97740 1 -- 172.21.15.16:0/2805207702 shutdown_connections 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.589+0000 7f3aebb97740 1 -- 172.21.15.16:0/2805207702 wait complete. 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.589+0000 7f3aebb97740 1 -- start start 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.589+0000 7f3aebb97740 1 Processor -- start 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2950bda0 0x561d29931a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d2954f1b0 0x561d29932070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3aebb97740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d29578070 0x561d299a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3aebb97740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01affb0 con 0x561d29578070 2024-08-28T19:28:28.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3aebb97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01af880 con 0x561d2954f1b0 2024-08-28T19:28:28.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3aebb97740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01acae0 con 0x561d2950bda0 2024-08-28T19:28:28.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3aebb97740 10 client.0 _ll_register_callbacks cb 0x7f3ad4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:28:28.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2950bda0 0x561d29931a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2950bda0 0x561d29931a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:50962/0 (socket says 172.21.15.16:50962) 2024-08-28T19:28:28.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3adeffd640 1 -- 172.21.15.16:0/390928385 learned_addr learned my addr 172.21.15.16:0/390928385 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:28:28.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d2954f1b0 0x561d29932070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3adf7fe640 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d29578070 0x561d299a2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3adeffd640 1 -- 172.21.15.16:0/390928385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d2954f1b0 msgr2=0x561d29932070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3adeffd640 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d2954f1b0 0x561d29932070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3adeffd640 1 -- 172.21.15.16:0/390928385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d29578070 msgr2=0x561d299a2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:28:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3adeffd640 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d29578070 0x561d299a2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3adeffd640 1 -- 172.21.15.16:0/390928385 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d299a3240 con 0x561d2950bda0 2024-08-28T19:28:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.590+0000 7f3adf7fe640 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d29578070 0x561d299a2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:28:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.591+0000 7f3adeffd640 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2950bda0 0x561d29931a90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0006010 tx=0x7f3ad0004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=721e9f97288623f3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:28:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.591+0000 7f3ac2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:28:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.591+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/390928385 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0004d40 con 0x561d2950bda0 2024-08-28T19:28:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.591+0000 7f3aebb97740 1 -- 172.21.15.16:0/390928385 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ad40d0980 con 0x561d2950bda0 2024-08-28T19:28:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.591+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/390928385 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f3ad0012450 con 0x561d2950bda0 2024-08-28T19:28:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.591+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/390928385 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f3ad0012690 con 0x561d2950bda0 2024-08-28T19:28:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.591+0000 7f3aebb97740 10 client.26809 fetch_fsmap learned FSMap version 47 2024-08-28T19:28:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.591+0000 7f3aebb97740 1 -- 172.21.15.16:0/390928385 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x561d299aa620 con 0x561d2950bda0 2024-08-28T19:28:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.591+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/390928385 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ad40d0980 con 0x561d2950bda0 2024-08-28T19:28:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.592+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/390928385 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f3ad000f040 con 0x561d2950bda0 2024-08-28T19:28:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.592+0000 7f3aebb97740 10 client.26809 fetch_fsmap finished waiting for FSMap version 47 2024-08-28T19:28:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.592+0000 7f3aebb97740 10 client.26809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:28:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.592+0000 7f3aebb97740 20 client.26809 populate_metadata read hostname 'smithi016' 2024-08-28T19:28:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.592+0000 7f3aebb97740 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561d299aaa60 0x561d299ace80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:28:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.592+0000 7f3aebb97740 4 client.26809 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:28:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.592+0000 7f3aebb97740 1 -- 172.21.15.16:0/390928385 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561d29892020 con 0x561d299aaa60 2024-08-28T19:28:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.593+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561d299aaa60 0x561d299ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:28:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.594+0000 7f3ade7fc640 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561d299aaa60 0x561d299ace80 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f5807d93b780a2e server_cookie=3dec99cb79185a24 in_seq=0 out_seq=0 2024-08-28T19:28:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.594+0000 7f3ac2ff5640 10 client.26809 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.594+0000 7f3ac2ff5640 1 -- 172.21.15.16:0/390928385 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3410 (crc 0 0 0) 0x561d29892020 con 0x561d299aaa60 2024-08-28T19:28:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.594+0000 7f3ac2ff5640 10 client.26809 handle_command_reply: tid=0 2024-08-28T19:28:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.594+0000 7f3aebb97740 1 client.26809 shutdown 2024-08-28T19:28:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.594+0000 7f3aebb97740 1 -- 172.21.15.16:0/390928385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2950bda0 msgr2=0x561d29931a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.594+0000 7f3aebb97740 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2950bda0 0x561d29931a90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0006010 tx=0x7f3ad0004a50 comp rx=0 tx=0).stop 2024-08-28T19:28:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.594+0000 7f3aebb97740 1 -- 172.21.15.16:0/390928385 shutdown_connections 2024-08-28T19:28:28.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.595+0000 7f3aebb97740 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561d29578070 0x561d299a2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.595+0000 7f3aebb97740 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d2954f1b0 0x561d29932070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.595+0000 7f3aebb97740 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561d299aaa60 0x561d299ace80 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.595+0000 7f3aebb97740 1 --2- 172.21.15.16:0/390928385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d2950bda0 0x561d29931a90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.595+0000 7f3ac2ff5640 0 client.26809 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:28:28.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.595+0000 7f3aebb97740 1 -- 172.21.15.16:0/390928385 >> 172.21.15.16:0/390928385 conn(0x561d2956c460 msgr2=0x561d29929ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:28.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.595+0000 7f3aebb97740 1 -- 172.21.15.16:0/390928385 shutdown_connections 2024-08-28T19:28:28.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.595+0000 7f3aebb97740 1 -- 172.21.15.16:0/390928385 wait complete. 2024-08-28T19:28:28.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.595+0000 7f3aebb97740 20 client.26809 trim_cache size 0 max 16384 2024-08-28T19:28:28.604 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:28:28.604 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:28:28.604 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:28:28.604 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:28:28.604 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.604 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "d4cf8d74-0ea8-4831-90ad-1492faca7486", 2024-08-28T19:28:28.604 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:28:28.605 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:28:28.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:28.605 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:28:28.605 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:28:28.605 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:28:28.605 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.605 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:28.605 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:28.605 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:28.605 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:28.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:28.606 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:28:28.606 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:28:28.606 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.606 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:28.606 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:28.606 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:28.606 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:28.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:28.606 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:28:28.606 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:28:28.606 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:28:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:28:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:28.608 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:28.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:28.608 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:28:28.608 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:28:28.608 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.608 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:28.608 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:28.608 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:28.608 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:28.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:28.608 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:28:28.609 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:28:28.609 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.609 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:28.609 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:28.609 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:28.609 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:28.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:28.609 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:28:28.609 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:28:28.609 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.610 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:28.610 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:28.610 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:28.610 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:28.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:28.610 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:28:28.610 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:28:28.610 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.610 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:28.610 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:28.611 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:28.611 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:28.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:28.611 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:28:28.611 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:28:28.611 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.611 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:28.611 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:28.611 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:28.611 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:28.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:28:28.612 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:28:28.613 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:28:28.613 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:28:28.613 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:28:28.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:28:28.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:28:28.613 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:28:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 -- 172.21.15.16:0/817899418 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ac80b4ca0 msgr2=0x7f3ac80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ac80b4ca0 0x7f3ac80b71b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009d60 tx=0x7f3ad8005dc0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 -- 172.21.15.16:0/817899418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae01050c0 msgr2=0x7f3ae019a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:28:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae01050c0 0x7f3ae019a820 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f3ad402a930 tx=0x7f3ad4004270 comp rx=0 tx=0).stop 2024-08-28T19:28:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 -- 172.21.15.16:0/817899418 shutdown_connections 2024-08-28T19:28:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3ae010dbd0 0x7f3ae019fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae01050c0 0x7f3ae019a820 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ac80b4ca0 0x7f3ac80b71b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 --2- 172.21.15.16:0/817899418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae0104770 0x7f3ae019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:28:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 -- 172.21.15.16:0/817899418 >> 172.21.15.16:0/817899418 conn(0x7f3ae00794e0 msgr2=0x7f3ae0100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:28:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 -- 172.21.15.16:0/817899418 shutdown_connections 2024-08-28T19:28:28.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:28:28.596+0000 7f3ae5622640 1 -- 172.21.15.16:0/817899418 wait complete. 2024-08-28T19:28:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:28:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:28:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:28:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:28:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:28:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:28:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:28 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:28:28.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 356206 4.83 MB/sec execute 248 sec latency 23.111 ms 2024-08-28T19:28:29.286 INFO:tasks.cephfs.filesystem:scrub status for tag:d4cf8d74-0ea8-4831-90ad-1492faca7486 - {'path': '/', 'tag': 'd4cf8d74-0ea8-4831-90ad-1492faca7486', 'options': 'recursive,force'} 2024-08-28T19:28:29.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:29 smithi016 ceph-mon[18556]: pgmap v564: 129 pgs: 129 active+clean; 231 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 174 op/s 2024-08-28T19:28:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:29 smithi028 ceph-mon[21769]: pgmap v564: 129 pgs: 129 active+clean; 231 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 174 op/s 2024-08-28T19:28:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:29 smithi115 ceph-mon[22184]: pgmap v564: 129 pgs: 129 active+clean; 231 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 174 op/s 2024-08-28T19:28:29.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 356392 4.81 MB/sec execute 249 sec latency 842.897 ms 2024-08-28T19:28:30.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 356392 4.79 MB/sec execute 250 sec latency 1842.997 ms 2024-08-28T19:28:31.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:31 smithi016 ceph-mon[18556]: pgmap v565: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 167 op/s 2024-08-28T19:28:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:31 smithi028 ceph-mon[21769]: pgmap v565: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 167 op/s 2024-08-28T19:28:31.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:31 smithi115 ceph-mon[22184]: pgmap v565: 129 pgs: 129 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 167 op/s 2024-08-28T19:28:31.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 356392 4.77 MB/sec execute 251 sec latency 2843.107 ms 2024-08-28T19:28:32.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 358449 4.80 MB/sec execute 252 sec latency 2868.820 ms 2024-08-28T19:28:33.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:33 smithi016 ceph-mon[18556]: pgmap v566: 129 pgs: 129 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 191 op/s 2024-08-28T19:28:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:33 smithi028 ceph-mon[21769]: pgmap v566: 129 pgs: 129 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 191 op/s 2024-08-28T19:28:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:33 smithi115 ceph-mon[22184]: pgmap v566: 129 pgs: 129 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 191 op/s 2024-08-28T19:28:33.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 359659 4.81 MB/sec execute 253 sec latency 22.495 ms 2024-08-28T19:28:34.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 359941 4.79 MB/sec execute 254 sec latency 772.478 ms 2024-08-28T19:28:35.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:35 smithi016 ceph-mon[18556]: pgmap v567: 129 pgs: 129 active+clean; 232 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-08-28T19:28:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:35 smithi028 ceph-mon[21769]: pgmap v567: 129 pgs: 129 active+clean; 232 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-08-28T19:28:35.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:35 smithi115 ceph-mon[22184]: pgmap v567: 129 pgs: 129 active+clean; 232 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-08-28T19:28:35.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 359941 4.77 MB/sec execute 255 sec latency 1772.621 ms 2024-08-28T19:28:36.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 359941 4.75 MB/sec execute 256 sec latency 2772.696 ms 2024-08-28T19:28:37.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:37 smithi016 ceph-mon[18556]: pgmap v568: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-08-28T19:28:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:37 smithi028 ceph-mon[21769]: pgmap v568: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-08-28T19:28:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:37 smithi115 ceph-mon[22184]: pgmap v568: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-08-28T19:28:37.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 362020 4.79 MB/sec execute 257 sec latency 2790.125 ms 2024-08-28T19:28:38.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:38 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:28:38.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:38 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:28:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:38 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:28:38.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 363201 4.79 MB/sec execute 258 sec latency 27.505 ms 2024-08-28T19:28:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:39 smithi028 ceph-mon[21769]: pgmap v569: 129 pgs: 129 active+clean; 236 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-08-28T19:28:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:39 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:28:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:39 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:28:39.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:39 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:28:39.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:39 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:28:39.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:39 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:28:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:39 smithi115 ceph-mon[22184]: pgmap v569: 129 pgs: 129 active+clean; 236 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-08-28T19:28:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:39 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:28:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:39 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:28:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:39 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:28:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:39 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:28:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:39 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:28:39.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:39 smithi016 ceph-mon[18556]: pgmap v569: 129 pgs: 129 active+clean; 236 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-08-28T19:28:39.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:39 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:28:39.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:39 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:28:39.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:39 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:28:39.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:39 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:28:39.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:39 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:28:39.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 363483 4.77 MB/sec execute 259 sec latency 782.900 ms 2024-08-28T19:28:40.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 363483 4.75 MB/sec execute 260 sec latency 1782.963 ms 2024-08-28T19:28:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:41 smithi028 ceph-mon[21769]: pgmap v570: 129 pgs: 129 active+clean; 236 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-08-28T19:28:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:28:41.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:41 smithi115 ceph-mon[22184]: pgmap v570: 129 pgs: 129 active+clean; 236 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-08-28T19:28:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:28:41.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:41 smithi016 ceph-mon[18556]: pgmap v570: 129 pgs: 129 active+clean; 236 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-08-28T19:28:41.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:28:41.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 363483 4.73 MB/sec execute 261 sec latency 2783.038 ms 2024-08-28T19:28:42.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:28:42.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:28:42.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:28:42.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:28:42.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:28:42.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:28:42.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:28:42.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:28:42.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:28:42.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 365482 4.76 MB/sec execute 262 sec latency 2800.167 ms 2024-08-28T19:28:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:43 smithi028 ceph-mon[21769]: pgmap v571: 129 pgs: 129 active+clean; 238 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-08-28T19:28:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:43 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:28:43.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:43 smithi115 ceph-mon[22184]: pgmap v571: 129 pgs: 129 active+clean; 238 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-08-28T19:28:43.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:43 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:28:43.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:43 smithi016 ceph-mon[18556]: pgmap v571: 129 pgs: 129 active+clean; 238 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-08-28T19:28:43.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:43 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:28:43.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 366591 4.76 MB/sec execute 263 sec latency 20.457 ms 2024-08-28T19:28:44.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 367032 4.75 MB/sec execute 264 sec latency 683.132 ms 2024-08-28T19:28:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:45 smithi028 ceph-mon[21769]: pgmap v572: 129 pgs: 129 active+clean; 237 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-08-28T19:28:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:45 smithi115 ceph-mon[22184]: pgmap v572: 129 pgs: 129 active+clean; 237 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-08-28T19:28:45.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:45 smithi016 ceph-mon[18556]: pgmap v572: 129 pgs: 129 active+clean; 237 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-08-28T19:28:45.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 367032 4.73 MB/sec execute 265 sec latency 1683.231 ms 2024-08-28T19:28:46.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 367775 4.72 MB/sec execute 266 sec latency 2326.467 ms 2024-08-28T19:28:47.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:47 smithi028 ceph-mon[21769]: pgmap v573: 129 pgs: 129 active+clean; 241 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 169 op/s 2024-08-28T19:28:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:47 smithi115 ceph-mon[22184]: pgmap v573: 129 pgs: 129 active+clean; 241 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 169 op/s 2024-08-28T19:28:47.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:47 smithi016 ceph-mon[18556]: pgmap v573: 129 pgs: 129 active+clean; 241 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 169 op/s 2024-08-28T19:28:47.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 369309 4.75 MB/sec execute 267 sec latency 11.038 ms 2024-08-28T19:28:48.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 370575 4.75 MB/sec execute 268 sec latency 23.013 ms 2024-08-28T19:28:49.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:49 smithi028 ceph-mon[21769]: pgmap v574: 129 pgs: 129 active+clean; 241 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-08-28T19:28:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:49 smithi115 ceph-mon[22184]: pgmap v574: 129 pgs: 129 active+clean; 241 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-08-28T19:28:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:49 smithi016 ceph-mon[18556]: pgmap v574: 129 pgs: 129 active+clean; 241 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-08-28T19:28:49.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 370575 4.74 MB/sec execute 269 sec latency 1023.086 ms 2024-08-28T19:28:50.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 370575 4.72 MB/sec execute 270 sec latency 2023.144 ms 2024-08-28T19:28:51.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:51 smithi028 ceph-mon[21769]: pgmap v575: 129 pgs: 129 active+clean; 241 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-08-28T19:28:51.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:51 smithi115 ceph-mon[22184]: pgmap v575: 129 pgs: 129 active+clean; 241 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-08-28T19:28:51.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:51 smithi016 ceph-mon[18556]: pgmap v575: 129 pgs: 129 active+clean; 241 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-08-28T19:28:51.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 370575 4.70 MB/sec execute 271 sec latency 3023.226 ms 2024-08-28T19:28:52.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 372259 4.72 MB/sec execute 272 sec latency 3044.069 ms 2024-08-28T19:28:53.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:53 smithi028 ceph-mon[21769]: pgmap v576: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-08-28T19:28:53.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:53 smithi115 ceph-mon[22184]: pgmap v576: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-08-28T19:28:53.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:53 smithi016 ceph-mon[18556]: pgmap v576: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-08-28T19:28:53.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 373030 4.72 MB/sec execute 273 sec latency 491.264 ms 2024-08-28T19:28:54.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 374123 4.72 MB/sec execute 274 sec latency 335.786 ms 2024-08-28T19:28:55.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:55 smithi028 ceph-mon[21769]: pgmap v577: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-08-28T19:28:55.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:55 smithi115 ceph-mon[22184]: pgmap v577: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-08-28T19:28:55.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:55 smithi016 ceph-mon[18556]: pgmap v577: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-08-28T19:28:55.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 374123 4.70 MB/sec execute 275 sec latency 1335.889 ms 2024-08-28T19:28:56.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 374123 4.68 MB/sec execute 276 sec latency 2336.005 ms 2024-08-28T19:28:57.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:57 smithi028 ceph-mon[21769]: pgmap v578: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-08-28T19:28:57.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:57 smithi115 ceph-mon[22184]: pgmap v578: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-08-28T19:28:57.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:57 smithi016 ceph-mon[18556]: pgmap v578: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-08-28T19:28:57.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376198 4.72 MB/sec execute 277 sec latency 2352.796 ms 2024-08-28T19:28:58.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 377543 4.72 MB/sec execute 278 sec latency 14.494 ms 2024-08-28T19:28:59.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:28:59.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:28:59 smithi028 ceph-mon[21769]: pgmap v579: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-08-28T19:28:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:28:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:28:59 smithi115 ceph-mon[22184]: pgmap v579: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-08-28T19:28:59.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:28:59 smithi016 ceph-mon[18556]: pgmap v579: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-08-28T19:28:59.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 379590 4.75 MB/sec execute 279 sec latency 58.783 ms 2024-08-28T19:29:00.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.270+0000 7f5af7401640 1 -- 172.21.15.16:0/1512837469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5af0108140 msgr2=0x7f5af0108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.270+0000 7f5af7401640 1 --2- 172.21.15.16:0/1512837469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5af0108140 0x7f5af0108550 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009900 tx=0x7f5ae002a410 comp rx=0 tx=0).stop 2024-08-28T19:29:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.270+0000 7f5af7401640 1 -- 172.21.15.16:0/1512837469 shutdown_connections 2024-08-28T19:29:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.270+0000 7f5af7401640 1 --2- 172.21.15.16:0/1512837469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5af0108a90 0x7f5af010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.270+0000 7f5af7401640 1 --2- 172.21.15.16:0/1512837469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5af0108140 0x7f5af0108550 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.270+0000 7f5af7401640 1 --2- 172.21.15.16:0/1512837469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5af0102800 0x7f5af0102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.270+0000 7f5af7401640 1 -- 172.21.15.16:0/1512837469 >> 172.21.15.16:0/1512837469 conn(0x7f5af00794d0 msgr2=0x7f5af00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.270+0000 7f5af7401640 1 -- 172.21.15.16:0/1512837469 shutdown_connections 2024-08-28T19:29:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.270+0000 7f5af7401640 1 -- 172.21.15.16:0/1512837469 wait complete. 2024-08-28T19:29:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.271+0000 7f5af7401640 1 Processor -- start 2024-08-28T19:29:00.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.271+0000 7f5af7401640 1 -- start start 2024-08-28T19:29:00.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af7401640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5af0102800 0x7f5af019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af7401640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5af0108140 0x7f5af019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af7401640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5af0108a90 0x7f5af019fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af7401640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5af0113820 con 0x7f5af0102800 2024-08-28T19:29:00.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af7401640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5af01136a0 con 0x7f5af0108140 2024-08-28T19:29:00.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af7401640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5af01139a0 con 0x7f5af0108a90 2024-08-28T19:29:00.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af499c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5af0108140 0x7f5af019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af519d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5af0102800 0x7f5af019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af499c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5af0108140 0x7f5af019a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:39304/0 (socket says 172.21.15.16:39304) 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af499c640 1 -- 172.21.15.16:0/3466015174 learned_addr learned my addr 172.21.15.16:0/3466015174 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af599e640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5af0108a90 0x7f5af019fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af499c640 1 -- 172.21.15.16:0/3466015174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5af0108a90 msgr2=0x7f5af019fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af499c640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5af0108a90 0x7f5af019fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af499c640 1 -- 172.21.15.16:0/3466015174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5af0102800 msgr2=0x7f5af019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.272+0000 7f5af499c640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5af0102800 0x7f5af019a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.273+0000 7f5af499c640 1 -- 172.21.15.16:0/3466015174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af01a01c0 con 0x7f5af0108140 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.273+0000 7f5af499c640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5af0108140 0x7f5af019a7f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005940 tx=0x7f5ae0005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.273+0000 7f5ae67fc640 1 -- 172.21.15.16:0/3466015174 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae003f030 con 0x7f5af0108140 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.273+0000 7f5af7401640 1 -- 172.21.15.16:0/3466015174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af01a0450 con 0x7f5af0108140 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.273+0000 7f5af7401640 1 -- 172.21.15.16:0/3466015174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af010fc30 con 0x7f5af0108140 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.273+0000 7f5ae67fc640 1 -- 172.21.15.16:0/3466015174 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ae00043d0 con 0x7f5af0108140 2024-08-28T19:29:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.273+0000 7f5ae67fc640 1 -- 172.21.15.16:0/3466015174 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0035070 con 0x7f5af0108140 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.276+0000 7f5ae67fc640 1 -- 172.21.15.16:0/3466015174 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5ae0042030 con 0x7f5af0108140 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.276+0000 7f5afd976740 1 Processor -- start 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.276+0000 7f5afd976740 1 -- start start 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5afd976740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 0x55f26872c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5afd976740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 0x55f26872c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5afd976740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f26872cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5afd976740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f2686a3630 con 0x55f26835e1b0 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5afd976740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f268692bc0 con 0x55f268387070 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5afd976740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f268696b70 con 0x55f268366390 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af519d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 0x55f26872c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af599e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 0x55f26872c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af519d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 0x55f26872c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60378/0 (socket says 172.21.15.16:60378) 2024-08-28T19:29:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af599e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 0x55f26872c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:39310/0 (socket says 172.21.15.16:39310) 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af519d640 1 -- 172.21.15.16:0/3134354781 learned_addr learned my addr 172.21.15.16:0/3134354781 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af499c640 1 --2- 172.21.15.16:0/3134354781 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f26872cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af519d640 1 -- 172.21.15.16:0/3134354781 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 msgr2=0x55f26872cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af519d640 1 --2- 172.21.15.16:0/3134354781 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f26872cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af519d640 1 -- 172.21.15.16:0/3134354781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 msgr2=0x55f26872c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af519d640 1 --2- 172.21.15.16:0/3134354781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 0x55f26872c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af519d640 1 -- 172.21.15.16:0/3134354781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2683f2650 con 0x55f26835e1b0 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.277+0000 7f5af599e640 1 --2- 172.21.15.16:0/3134354781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 0x55f26872c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.278+0000 7f5ae67fc640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5adc0b4d20 0x7f5adc0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.278+0000 7f5af519d640 1 --2- 172.21.15.16:0/3134354781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 0x55f26872c800 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f5aec004610 tx=0x7f5aec005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab999d773628b84 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.278+0000 7f5af599e640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5adc0b4d20 0x7f5adc0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.278+0000 7f5ae67fc640 1 -- 172.21.15.16:0/3466015174 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f5ae0105a70 con 0x7f5af0108140 2024-08-28T19:29:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.278+0000 7f5adb7fe640 1 -- 172.21.15.16:0/3134354781 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec009e60 con 0x55f26835e1b0 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.278+0000 7f5adb7fe640 1 -- 172.21.15.16:0/3134354781 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5aec0125a0 con 0x55f26835e1b0 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.278+0000 7f5adb7fe640 1 -- 172.21.15.16:0/3134354781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec01b440 con 0x55f26835e1b0 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.279+0000 7f5afd976740 1 -- 172.21.15.16:0/3134354781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 msgr2=0x55f26872c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.279+0000 7f5af599e640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5adc0b4d20 0x7f5adc0b7230 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8005fd0 tx=0x7f5ae8009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.279+0000 7f5afd976740 1 --2- 172.21.15.16:0/3134354781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 0x55f26872c800 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f5aec004610 tx=0x7f5aec005f90 comp rx=0 tx=0).stop 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.280+0000 7f5afd976740 1 -- 172.21.15.16:0/3134354781 shutdown_connections 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.280+0000 7f5afd976740 1 --2- 172.21.15.16:0/3134354781 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f26872cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.280+0000 7f5afd976740 1 --2- 172.21.15.16:0/3134354781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 0x55f26872c800 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.280+0000 7f5afd976740 1 --2- 172.21.15.16:0/3134354781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 0x55f26872c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.280+0000 7f5afd976740 1 -- 172.21.15.16:0/3134354781 >> 172.21.15.16:0/3134354781 conn(0x55f26837b460 msgr2=0x55f26872a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.280+0000 7f5afd976740 1 -- 172.21.15.16:0/3134354781 shutdown_connections 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.280+0000 7f5afd976740 1 -- 172.21.15.16:0/3134354781 wait complete. 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.281+0000 7f5afd976740 1 -- start start 2024-08-28T19:29:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.281+0000 7f5afd976740 1 Processor -- start 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.281+0000 7f5afd976740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 0x55f2687379e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.281+0000 7f5afd976740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f268737f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.281+0000 7f5afd976740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268387070 0x55f268743e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.281+0000 7f5afd976740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f2684f5700 con 0x55f268387070 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.281+0000 7f5afd976740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f268619700 con 0x55f26835e1b0 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.281+0000 7f5afd976740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f26873aaa0 con 0x55f268366390 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.281+0000 7f5afd976740 10 client.0 _ll_register_callbacks cb 0x55f268648dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af519d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 0x55f2687379e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af499c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f268737f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af519d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 0x55f2687379e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:39320/0 (socket says 172.21.15.16:39320) 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af499c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f268737f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34738/0 (socket says 172.21.15.16:34738) 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af519d640 1 -- 172.21.15.16:0/4041157401 learned_addr learned my addr 172.21.15.16:0/4041157401 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af599e640 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268387070 0x55f268743e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af519d640 1 -- 172.21.15.16:0/4041157401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 msgr2=0x55f268737f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af519d640 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f268737f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af519d640 1 -- 172.21.15.16:0/4041157401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268387070 msgr2=0x55f268743e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af519d640 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268387070 0x55f268743e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af519d640 1 -- 172.21.15.16:0/4041157401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2686528b0 con 0x55f26835e1b0 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af499c640 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f268737f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5af519d640 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 0x55f2687379e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5aec003b90 tx=0x7f5aec005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b40cb7fa0dd4e224 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.282+0000 7f5ad8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/4041157401 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec02b060 con 0x55f26835e1b0 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5afd976740 1 -- 172.21.15.16:0/4041157401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5aec0125a0 con 0x55f26835e1b0 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/4041157401 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f5aec0035f0 con 0x55f26835e1b0 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/4041157401 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f5aec032050 con 0x55f26835e1b0 2024-08-28T19:29:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5afd976740 10 client.27023 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5afd976740 1 -- 172.21.15.16:0/4041157401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55f268746840 con 0x55f26835e1b0 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/4041157401 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5aec0125a0 con 0x55f26835e1b0 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/4041157401 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f5aec019020 con 0x55f26835e1b0 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5afd976740 10 client.27023 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5afd976740 10 client.27023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5afd976740 20 client.27023 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5afd976740 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f2683459f0 0x55f268749be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5afd976740 4 client.27023 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.283+0000 7f5afd976740 1 -- 172.21.15.16:0/4041157401 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2686a5580 con 0x55f2683459f0 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.284+0000 7f5af499c640 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f2683459f0 0x55f268749be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.284+0000 7f5ad8ff9640 10 client.27023 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.284+0000 7f5af499c640 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f2683459f0 0x55f268749be0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f4409c8daaf1e03 server_cookie=4dd9394ca91ed11 in_seq=0 out_seq=0 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/4041157401 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f2686a5580 con 0x55f2683459f0 2024-08-28T19:29:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5ad8ff9640 10 client.27023 handle_command_reply: tid=0 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5afd976740 1 client.27023 shutdown 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5afd976740 1 -- 172.21.15.16:0/4041157401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 msgr2=0x55f2687379e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5afd976740 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 0x55f2687379e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5aec003b90 tx=0x7f5aec005d30 comp rx=0 tx=0).stop 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5afd976740 1 -- 172.21.15.16:0/4041157401 shutdown_connections 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5afd976740 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268387070 0x55f268743e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5afd976740 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f2683459f0 0x55f268749be0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5afd976740 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f268737f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5afd976740 1 --2- 172.21.15.16:0/4041157401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 0x55f2687379e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5afd976740 1 -- 172.21.15.16:0/4041157401 >> 172.21.15.16:0/4041157401 conn(0x55f26837b460 msgr2=0x55f26872f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5ad8ff9640 0 client.27023 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.285+0000 7f5afd976740 1 -- 172.21.15.16:0/4041157401 shutdown_connections 2024-08-28T19:29:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.286+0000 7f5afd976740 1 -- 172.21.15.16:0/4041157401 wait complete. 2024-08-28T19:29:00.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.286+0000 7f5afd976740 20 client.27023 trim_cache size 0 max 16384 2024-08-28T19:29:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.307+0000 7f5afd976740 1 Processor -- start 2024-08-28T19:29:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.307+0000 7f5afd976740 1 -- start start 2024-08-28T19:29:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5afd976740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 0x55f268737b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5afd976740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268366390 0x55f26872da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5afd976740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268387070 0x55f268729780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5afd976740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad400ffb0 con 0x55f268366390 2024-08-28T19:29:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5afd976740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4011c40 con 0x55f26835e1b0 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5afd976740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad40106e0 con 0x55f268387070 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5af499c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268366390 0x55f26872da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5af499c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268366390 0x55f26872da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60388/0 (socket says 172.21.15.16:60388) 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5af499c640 1 -- 172.21.15.16:0/4249022993 learned_addr learned my addr 172.21.15.16:0/4249022993 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5af519d640 1 --2- 172.21.15.16:0/4249022993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 0x55f268737b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5af599e640 1 --2- 172.21.15.16:0/4249022993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268387070 0x55f268729780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5af499c640 1 -- 172.21.15.16:0/4249022993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268387070 msgr2=0x55f268729780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5af499c640 1 --2- 172.21.15.16:0/4249022993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268387070 0x55f268729780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5af499c640 1 -- 172.21.15.16:0/4249022993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 msgr2=0x55f268737b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5af499c640 1 --2- 172.21.15.16:0/4249022993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 0x55f268737b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5af499c640 1 -- 172.21.15.16:0/4249022993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f268736940 con 0x55f268366390 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.308+0000 7f5af519d640 1 --2- 172.21.15.16:0/4249022993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 0x55f268737b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.309+0000 7f5af499c640 1 --2- 172.21.15.16:0/4249022993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268366390 0x55f26872da00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0000c00 tx=0x7f5ae0114c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192a36515fffc386 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.309+0000 7f5adaffd640 1 -- 172.21.15.16:0/4249022993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0049040 con 0x55f268366390 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.309+0000 7f5adaffd640 1 -- 172.21.15.16:0/4249022993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ae003b460 con 0x55f268366390 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.309+0000 7f5adaffd640 1 -- 172.21.15.16:0/4249022993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae00d0600 con 0x55f268366390 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.310+0000 7f5afd976740 1 -- 172.21.15.16:0/4249022993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268366390 msgr2=0x55f26872da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.310+0000 7f5afd976740 1 --2- 172.21.15.16:0/4249022993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268366390 0x55f26872da00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0000c00 tx=0x7f5ae0114c80 comp rx=0 tx=0).stop 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.310+0000 7f5afd976740 1 -- 172.21.15.16:0/4249022993 shutdown_connections 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.310+0000 7f5afd976740 1 --2- 172.21.15.16:0/4249022993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268387070 0x55f268729780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.310+0000 7f5afd976740 1 --2- 172.21.15.16:0/4249022993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f268366390 0x55f26872da00 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.310+0000 7f5afd976740 1 --2- 172.21.15.16:0/4249022993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f26835e1b0 0x55f268737b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.310+0000 7f5afd976740 1 -- 172.21.15.16:0/4249022993 >> 172.21.15.16:0/4249022993 conn(0x55f26837b460 msgr2=0x55f26872d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.311+0000 7f5afd976740 1 -- 172.21.15.16:0/4249022993 shutdown_connections 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.311+0000 7f5afd976740 1 -- 172.21.15.16:0/4249022993 wait complete. 2024-08-28T19:29:00.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.311+0000 7f5afd976740 1 -- start start 2024-08-28T19:29:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.311+0000 7f5afd976740 1 Processor -- start 2024-08-28T19:29:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5afd976740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 0x55f268740660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5afd976740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f26873bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5afd976740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 0x55f26873c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5afd976740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad40114e0 con 0x55f26835e1b0 2024-08-28T19:29:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5afd976740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4010db0 con 0x55f268387070 2024-08-28T19:29:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5afd976740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad400dfb0 con 0x55f268366390 2024-08-28T19:29:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5af499c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f26873bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5af499c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f26873bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34764/0 (socket says 172.21.15.16:34764) 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5af499c640 1 -- 172.21.15.16:0/1277810400 learned_addr learned my addr 172.21.15.16:0/1277810400 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5af599e640 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 0x55f26873c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5afd976740 10 client.0 _ll_register_callbacks cb 0x7f5ae8009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5af519d640 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 0x55f268740660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5af499c640 1 -- 172.21.15.16:0/1277810400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 msgr2=0x55f26873c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5af499c640 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 0x55f26873c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5af499c640 1 -- 172.21.15.16:0/1277810400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 msgr2=0x55f268740660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5af499c640 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 0x55f268740660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.312+0000 7f5af499c640 1 -- 172.21.15.16:0/1277810400 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f26873c8d0 con 0x55f268366390 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.313+0000 7f5af499c640 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f26873bb10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0102ea0 tx=0x7f5ae0103320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deaad8e8ca65633e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.313+0000 7f5ad8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:29:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.313+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/1277810400 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0049050 con 0x55f268366390 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.313+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/1277810400 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f5ae00c8020 con 0x55f268366390 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.313+0000 7f5afd976740 1 -- 172.21.15.16:0/1277810400 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ae003b460 con 0x55f268366390 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.313+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/1277810400 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f5ae00cbb60 con 0x55f268366390 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.313+0000 7f5afd976740 10 client.26827 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.313+0000 7f5afd976740 1 -- 172.21.15.16:0/1277810400 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55f2687b4610 con 0x55f268366390 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.313+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/1277810400 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ae00cc490 con 0x55f268366390 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.314+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/1277810400 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f5ae00d0240 con 0x55f268366390 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.314+0000 7f5afd976740 10 client.26827 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.314+0000 7f5afd976740 10 client.26827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.314+0000 7f5afd976740 20 client.26827 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.314+0000 7f5afd976740 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f26831ada0 0x55f2687b6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.314+0000 7f5afd976740 4 client.26827 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:29:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.314+0000 7f5afd976740 1 -- 172.21.15.16:0/1277810400 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f5ae003b460 con 0x55f26831ada0 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.314+0000 7f5af519d640 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f26831ada0 0x55f2687b6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5ad8ff9640 10 client.26827 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5af519d640 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f26831ada0 0x55f2687b6e00 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dfe7703a2773ca4 server_cookie=33210edc32299756 in_seq=0 out_seq=0 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5ad8ff9640 1 -- 172.21.15.16:0/1277810400 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3214 (crc 0 0 0) 0x7f5ae003b460 con 0x55f26831ada0 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5ad8ff9640 10 client.26827 handle_command_reply: tid=0 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5afd976740 1 client.26827 shutdown 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5afd976740 1 -- 172.21.15.16:0/1277810400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 msgr2=0x55f26873bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5afd976740 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f26873bb10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0102ea0 tx=0x7f5ae0103320 comp rx=0 tx=0).stop 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5afd976740 1 -- 172.21.15.16:0/1277810400 shutdown_connections 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5afd976740 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f268387070 0x55f26873c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5afd976740 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f26831ada0 0x55f2687b6e00 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5afd976740 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f268366390 0x55f26873bb10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5afd976740 1 --2- 172.21.15.16:0/1277810400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f26835e1b0 0x55f268740660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5ad8ff9640 0 client.26827 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:00.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.315+0000 7f5afd976740 1 -- 172.21.15.16:0/1277810400 >> 172.21.15.16:0/1277810400 conn(0x55f26837b460 msgr2=0x55f268738e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:00.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5afd976740 1 -- 172.21.15.16:0/1277810400 shutdown_connections 2024-08-28T19:29:00.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5afd976740 1 -- 172.21.15.16:0/1277810400 wait complete. 2024-08-28T19:29:00.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5afd976740 20 client.26827 trim_cache size 0 max 16384 2024-08-28T19:29:00.324 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:29:00.324 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:29:00.324 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:29:00.324 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:29:00.324 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:29:00.324 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:00.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:29:00.326 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:29:00.328 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:00.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:00.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:00.331 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:00.332 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:29:00.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5af7401640 1 -- 172.21.15.16:0/3466015174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5adc0b4d20 msgr2=0x7f5adc0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:00.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5af7401640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5adc0b4d20 0x7f5adc0b7230 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8005fd0 tx=0x7f5ae8009500 comp rx=0 tx=0).stop 2024-08-28T19:29:00.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5af7401640 1 -- 172.21.15.16:0/3466015174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5af0108140 msgr2=0x7f5af019a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:00.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5af7401640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5af0108140 0x7f5af019a7f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005940 tx=0x7f5ae0005e70 comp rx=0 tx=0).stop 2024-08-28T19:29:00.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5af7401640 1 -- 172.21.15.16:0/3466015174 shutdown_connections 2024-08-28T19:29:00.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5af7401640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5af0108a90 0x7f5af019fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5af7401640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5adc0b4d20 0x7f5adc0b7230 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5af7401640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5af0108140 0x7f5af019a7f0 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005940 tx=0x7f5ae0005e70 comp rx=0 tx=0).stop 2024-08-28T19:29:00.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5af7401640 1 --2- 172.21.15.16:0/3466015174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5af0102800 0x7f5af019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:00.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.316+0000 7f5af7401640 1 -- 172.21.15.16:0/3466015174 >> 172.21.15.16:0/3466015174 conn(0x7f5af00794d0 msgr2=0x7f5af0104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:00.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.317+0000 7f5af7401640 1 -- 172.21.15.16:0/3466015174 shutdown_connections 2024-08-28T19:29:00.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:00.317+0000 7f5af7401640 1 -- 172.21.15.16:0/3466015174 wait complete. 2024-08-28T19:29:00.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 379941 4.74 MB/sec execute 280 sec latency 60.475 ms 2024-08-28T19:29:01.100 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:29:01.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:29:01.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:01 smithi016 ceph-mon[18556]: pgmap v580: 129 pgs: 129 active+clean; 248 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-08-28T19:29:01.394 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:01.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:01 smithi028 ceph-mon[21769]: pgmap v580: 129 pgs: 129 active+clean; 248 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-08-28T19:29:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:01 smithi115 ceph-mon[22184]: pgmap v580: 129 pgs: 129 active+clean; 248 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-08-28T19:29:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.888+0000 7f5e58437640 1 -- 172.21.15.16:0/2852977980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e50109310 msgr2=0x7f5e50109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.888+0000 7f5e58437640 1 --2- 172.21.15.16:0/2852977980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e50109310 0x7f5e50109720 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009ca0 tx=0x7f5e4402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:29:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.888+0000 7f5e58437640 1 -- 172.21.15.16:0/2852977980 shutdown_connections 2024-08-28T19:29:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.888+0000 7f5e58437640 1 --2- 172.21.15.16:0/2852977980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e50109c60 0x7f5e50110490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.888+0000 7f5e58437640 1 --2- 172.21.15.16:0/2852977980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e50109310 0x7f5e50109720 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.888+0000 7f5e58437640 1 --2- 172.21.15.16:0/2852977980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e501039d0 0x7f5e50103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.888+0000 7f5e58437640 1 -- 172.21.15.16:0/2852977980 >> 172.21.15.16:0/2852977980 conn(0x7f5e5007b670 msgr2=0x7f5e5007ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.888+0000 7f5e58437640 1 -- 172.21.15.16:0/2852977980 shutdown_connections 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.888+0000 7f5e58437640 1 -- 172.21.15.16:0/2852977980 wait complete. 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.889+0000 7f5e58437640 1 Processor -- start 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.889+0000 7f5e58437640 1 -- start start 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e58437640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e501039d0 0x7f5e500754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e58437640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e50109310 0x7f5e500759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e58437640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e50109c60 0x7f5e50075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e58437640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5e50113880 con 0x7f5e50109c60 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e58437640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e50113700 con 0x7f5e501039d0 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e58437640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5e50113a00 con 0x7f5e50109310 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e569d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e50109c60 0x7f5e50075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e561d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e501039d0 0x7f5e500754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e559d2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e50109310 0x7f5e500759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e561d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e501039d0 0x7f5e500754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:39354/0 (socket says 172.21.15.16:39354) 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e569d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e50109c60 0x7f5e50075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60408/0 (socket says 172.21.15.16:60408) 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e561d3640 1 -- 172.21.15.16:0/2608678430 learned_addr learned my addr 172.21.15.16:0/2608678430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e561d3640 1 -- 172.21.15.16:0/2608678430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e50109310 msgr2=0x7f5e500759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e561d3640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e50109310 0x7f5e500759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e561d3640 1 -- 172.21.15.16:0/2608678430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e50109c60 msgr2=0x7f5e50075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e561d3640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e50109c60 0x7f5e50075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e561d3640 1 -- 172.21.15.16:0/2608678430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e501a8570 con 0x7f5e501039d0 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e559d2640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e50109310 0x7f5e500759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e569d4640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e50109c60 0x7f5e50075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.890+0000 7f5e561d3640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e501039d0 0x7f5e500754b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800e9f0 tx=0x7f5e4800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.891+0000 7f5e4f7fe640 1 -- 172.21.15.16:0/2608678430 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4800cd80 con 0x7f5e501039d0 2024-08-28T19:29:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.891+0000 7f5e58437640 1 -- 172.21.15.16:0/2608678430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e501a8860 con 0x7f5e501039d0 2024-08-28T19:29:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.891+0000 7f5e4f7fe640 1 -- 172.21.15.16:0/2608678430 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e480108e0 con 0x7f5e501039d0 2024-08-28T19:29:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.891+0000 7f5e4f7fe640 1 -- 172.21.15.16:0/2608678430 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e48018980 con 0x7f5e501039d0 2024-08-28T19:29:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.891+0000 7f5e58437640 1 -- 172.21.15.16:0/2608678430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e501a8dc0 con 0x7f5e501039d0 2024-08-28T19:29:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.892+0000 7f5e58437640 1 -- 172.21.15.16:0/2608678430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e50103de0 con 0x7f5e501039d0 2024-08-28T19:29:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.893+0000 7f5e4f7fe640 1 -- 172.21.15.16:0/2608678430 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5e48010430 con 0x7f5e501039d0 2024-08-28T19:29:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.894+0000 7f5e4f7fe640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5e380b4ca0 0x7f5e380b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.894+0000 7f5e559d2640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5e380b4ca0 0x7f5e380b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.894+0000 7f5e4f7fe640 1 -- 172.21.15.16:0/2608678430 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9490+0+0 (secure 0 0 0) 0x7f5e480e6100 con 0x7f5e501039d0 2024-08-28T19:29:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.895+0000 7f5e559d2640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5e380b4ca0 0x7f5e380b71b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5e44000f80 tx=0x7f5e44009360 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:01.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:01.897+0000 7f5e4f7fe640 1 -- 172.21.15.16:0/2608678430 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f5e480ac4d0 con 0x7f5e501039d0 2024-08-28T19:29:01.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 381154 4.74 MB/sec execute 281 sec latency 43.610 ms 2024-08-28T19:29:02.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.139+0000 7f5e58437640 1 -- 172.21.15.16:0/2608678430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e500767b0 con 0x7f5e501039d0 2024-08-28T19:29:02.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.140+0000 7f5e4f7fe640 1 -- 172.21.15.16:0/2608678430 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11988 (secure 0 0 0) 0x7f5e500767b0 con 0x7f5e501039d0 2024-08-28T19:29:02.142 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:29:02.143 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":49,"btime":"2024-08-28T19:28:42:259944+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:28:41.590027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:29:02.143 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 49 2024-08-28T19:29:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.142+0000 7f5e58437640 1 -- 172.21.15.16:0/2608678430 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5e380b4ca0 msgr2=0x7f5e380b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.142+0000 7f5e58437640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5e380b4ca0 0x7f5e380b71b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5e44000f80 tx=0x7f5e44009360 comp rx=0 tx=0).stop 2024-08-28T19:29:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.142+0000 7f5e58437640 1 -- 172.21.15.16:0/2608678430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e501039d0 msgr2=0x7f5e500754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.142+0000 7f5e58437640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e501039d0 0x7f5e500754b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800e9f0 tx=0x7f5e4800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:29:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.142+0000 7f5e58437640 1 -- 172.21.15.16:0/2608678430 shutdown_connections 2024-08-28T19:29:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.142+0000 7f5e58437640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e50109c60 0x7f5e50075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.142+0000 7f5e58437640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e50109310 0x7f5e500759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.142+0000 7f5e58437640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5e380b4ca0 0x7f5e380b71b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.142+0000 7f5e58437640 1 --2- 172.21.15.16:0/2608678430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e501039d0 0x7f5e500754b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.142+0000 7f5e58437640 1 -- 172.21.15.16:0/2608678430 >> 172.21.15.16:0/2608678430 conn(0x7f5e5007b670 msgr2=0x7f5e50107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.142+0000 7f5e58437640 1 -- 172.21.15.16:0/2608678430 shutdown_connections 2024-08-28T19:29:02.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:02.143+0000 7f5e58437640 1 -- 172.21.15.16:0/2608678430 wait complete. 2024-08-28T19:29:02.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:02 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2608678430' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:02.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:02 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2608678430' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:02.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:02 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2608678430' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:02.880 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:29:02.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 381255 4.72 MB/sec execute 282 sec latency 938.346 ms 2024-08-28T19:29:03.135 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:03.389 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.5 2024-08-28T19:29:03.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:29:03.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:03 smithi028 ceph-mon[21769]: pgmap v581: 129 pgs: 129 active+clean; 249 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 187 op/s 2024-08-28T19:29:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:03 smithi115 ceph-mon[22184]: pgmap v581: 129 pgs: 129 active+clean; 249 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 187 op/s 2024-08-28T19:29:03.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:03 smithi016 ceph-mon[18556]: pgmap v581: 129 pgs: 129 active+clean; 249 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 187 op/s 2024-08-28T19:29:03.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 381255 4.70 MB/sec execute 283 sec latency 1938.376 ms 2024-08-28T19:29:03.982 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:29:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.897+0000 7f76f292f640 1 -- 172.21.15.16:0/1990263600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76ec108140 msgr2=0x7f76ec108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.897+0000 7f76f292f640 1 --2- 172.21.15.16:0/1990263600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76ec108140 0x7f76ec108550 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009900 tx=0x7f76e002a410 comp rx=0 tx=0).stop 2024-08-28T19:29:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.897+0000 7f76f292f640 1 -- 172.21.15.16:0/1990263600 shutdown_connections 2024-08-28T19:29:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.897+0000 7f76f292f640 1 --2- 172.21.15.16:0/1990263600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76ec108a90 0x7f76ec10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.897+0000 7f76f292f640 1 --2- 172.21.15.16:0/1990263600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76ec108140 0x7f76ec108550 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.897+0000 7f76f292f640 1 --2- 172.21.15.16:0/1990263600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec102800 0x7f76ec102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.897+0000 7f76f292f640 1 -- 172.21.15.16:0/1990263600 >> 172.21.15.16:0/1990263600 conn(0x7f76ec0794d0 msgr2=0x7f76ec0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.897+0000 7f76f292f640 1 -- 172.21.15.16:0/1990263600 shutdown_connections 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.898+0000 7f76f292f640 1 -- 172.21.15.16:0/1990263600 wait complete. 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.898+0000 7f76f292f640 1 Processor -- start 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.898+0000 7f76f292f640 1 -- start start 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f292f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec102800 0x7f76ec104e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f292f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76ec108140 0x7f76ec1053c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f292f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76ec108a90 0x7f76ec105900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f292f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec1126e0 con 0x7f76ec108a90 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f292f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec112560 con 0x7f76ec102800 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f292f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec112860 con 0x7f76ec108140 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f0ecc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76ec108a90 0x7f76ec105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f0ecc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76ec108a90 0x7f76ec105900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60422/0 (socket says 172.21.15.16:60422) 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f0ecc640 1 -- 172.21.15.16:0/900629291 learned_addr learned my addr 172.21.15.16:0/900629291 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76ebfff640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec102800 0x7f76ec104e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76ec108140 0x7f76ec1053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f0ecc640 1 -- 172.21.15.16:0/900629291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76ec108140 msgr2=0x7f76ec1053c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f0ecc640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76ec108140 0x7f76ec1053c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f0ecc640 1 -- 172.21.15.16:0/900629291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec102800 msgr2=0x7f76ec104e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f0ecc640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec102800 0x7f76ec104e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f0ecc640 1 -- 172.21.15.16:0/900629291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76ec1a8550 con 0x7f76ec108a90 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f0ecc640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76ec108a90 0x7f76ec105900 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f76e400cad0 tx=0x7f76e400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76e97fa640 1 -- 172.21.15.16:0/900629291 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e400ae20 con 0x7f76ec108a90 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76e97fa640 1 -- 172.21.15.16:0/900629291 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e4005020 con 0x7f76ec108a90 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76e97fa640 1 -- 172.21.15.16:0/900629291 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e400f6b0 con 0x7f76ec108a90 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.899+0000 7f76f292f640 1 -- 172.21.15.16:0/900629291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76ec1a8840 con 0x7f76ec108a90 2024-08-28T19:29:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.900+0000 7f76f292f640 1 -- 172.21.15.16:0/900629291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76ec1a8da0 con 0x7f76ec108a90 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.902+0000 7f76e97fa640 1 -- 172.21.15.16:0/900629291 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f76e40051d0 con 0x7f76ec108a90 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.902+0000 7f76f8ea4740 1 Processor -- start 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.902+0000 7f76f8ea4740 1 -- start start 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df89070 0x56551e32e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e32e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e32edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56551e2a5680 con 0x56551df601b0 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56551e294c10 con 0x56551df68390 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56551e298bc0 con 0x56551df89070 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e32e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e32e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60436/0 (socket says 172.21.15.16:60436) 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76ebfff640 1 -- 172.21.15.16:0/569514444 learned_addr learned my addr 172.21.15.16:0/569514444 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/569514444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df89070 0x56551e32e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/569514444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e32edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76e97fa640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76d40b4d20 0x7f76d40b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76ebfff640 1 -- 172.21.15.16:0/569514444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df89070 msgr2=0x56551e32e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76ebfff640 1 --2- 172.21.15.16:0/569514444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df89070 0x56551e32e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76ebfff640 1 -- 172.21.15.16:0/569514444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 msgr2=0x56551e32edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76ebfff640 1 --2- 172.21.15.16:0/569514444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e32edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76ebfff640 1 -- 172.21.15.16:0/569514444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56551dff4650 con 0x56551df601b0 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/569514444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e32edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76f0ecc640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76d40b4d20 0x7f76d40b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76ebfff640 1 --2- 172.21.15.16:0/569514444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e32e890 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f76dc00bcb0 tx=0x7f76dc0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3f364f4d11c7ba server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.903+0000 7f76e97fa640 1 -- 172.21.15.16:0/900629291 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f76e40e58d0 con 0x7f76ec108a90 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.904+0000 7f76d27fc640 1 -- 172.21.15.16:0/569514444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76dc01c480 con 0x56551df601b0 2024-08-28T19:29:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.904+0000 7f76d27fc640 1 -- 172.21.15.16:0/569514444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76dc01cad0 con 0x56551df601b0 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.904+0000 7f76d27fc640 1 -- 172.21.15.16:0/569514444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76dc011c30 con 0x56551df601b0 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.904+0000 7f76f8ea4740 1 -- 172.21.15.16:0/569514444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 msgr2=0x56551e32e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.904+0000 7f76f0ecc640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76d40b4d20 0x7f76d40b7230 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f76e40a1020 tx=0x7f76e4075be0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.904+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/569514444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e32e890 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f76dc00bcb0 tx=0x7f76dc0107a0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.904+0000 7f76f8ea4740 1 -- 172.21.15.16:0/569514444 shutdown_connections 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.904+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/569514444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e32edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.904+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/569514444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e32e890 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.904+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/569514444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df89070 0x56551e32e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.905+0000 7f76f8ea4740 1 -- 172.21.15.16:0/569514444 >> 172.21.15.16:0/569514444 conn(0x56551df7d460 msgr2=0x56551e32c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.905+0000 7f76f8ea4740 1 -- 172.21.15.16:0/569514444 shutdown_connections 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.905+0000 7f76f8ea4740 1 -- 172.21.15.16:0/569514444 wait complete. 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.905+0000 7f76f8ea4740 1 -- start start 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.905+0000 7f76f8ea4740 1 Processor -- start 2024-08-28T19:29:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.905+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df601b0 0x56551e339ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.905+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e339ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e345e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56551e0f7700 con 0x56551df89070 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56551e21b750 con 0x56551df68390 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56551e33cb30 con 0x56551df601b0 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76f8ea4740 10 client.0 _ll_register_callbacks cb 0x56551e24ae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76eb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e345e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76eb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e345e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60452/0 (socket says 172.21.15.16:60452) 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76eb7fe640 1 -- 172.21.15.16:0/376493822 learned_addr learned my addr 172.21.15.16:0/376493822 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76ebfff640 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e339ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df601b0 0x56551e339ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76eb7fe640 1 -- 172.21.15.16:0/376493822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df601b0 msgr2=0x56551e339ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df601b0 0x56551e339ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76eb7fe640 1 -- 172.21.15.16:0/376493822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 msgr2=0x56551e339ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e339ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76eb7fe640 1 -- 172.21.15.16:0/376493822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56551e254900 con 0x56551df89070 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e345e20 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f76e0005940 tx=0x7f76e0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b08f587663dba3d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76cb7fe640 1 -- 172.21.15.16:0/376493822 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0046020 con 0x56551df89070 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76cb7fe640 1 -- 172.21.15.16:0/376493822 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f76e00035f0 con 0x56551df89070 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.906+0000 7f76f8ea4740 1 -- 172.21.15.16:0/376493822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76dc01cad0 con 0x56551df89070 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.907+0000 7f76cb7fe640 1 -- 172.21.15.16:0/376493822 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f76e0051050 con 0x56551df89070 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.907+0000 7f76f8ea4740 10 client.17514 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.907+0000 7f76f8ea4740 1 -- 172.21.15.16:0/376493822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56551e3488e0 con 0x56551df89070 2024-08-28T19:29:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.907+0000 7f76cb7fe640 1 -- 172.21.15.16:0/376493822 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76dc01cad0 con 0x56551df89070 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.907+0000 7f76cb7fe640 1 -- 172.21.15.16:0/376493822 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f76e003a040 con 0x56551df89070 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.907+0000 7f76f8ea4740 10 client.17514 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.907+0000 7f76f8ea4740 10 client.17514 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.907+0000 7f76f8ea4740 20 client.17514 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.907+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56551df479f0 0x56551e34be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.907+0000 7f76f8ea4740 4 client.17514 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.907+0000 7f76f8ea4740 1 -- 172.21.15.16:0/376493822 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56551e2a75d0 con 0x56551df479f0 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.908+0000 7f76ebfff640 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56551df479f0 0x56551e34be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.908+0000 7f76ebfff640 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56551df479f0 0x56551e34be40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=31f7a8e4ad04761a server_cookie=4e4f7d55501e5224 in_seq=0 out_seq=0 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.908+0000 7f76cb7fe640 10 client.17514 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76cb7fe640 1 -- 172.21.15.16:0/376493822 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56551e2a75d0 con 0x56551df479f0 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76cb7fe640 10 client.17514 handle_command_reply: tid=0 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 1 client.17514 shutdown 2024-08-28T19:29:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 1 -- 172.21.15.16:0/376493822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 msgr2=0x56551e345e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e345e20 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f76e0005940 tx=0x7f76e0009260 comp rx=0 tx=0).stop 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 1 -- 172.21.15.16:0/376493822 shutdown_connections 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56551df479f0 0x56551e34be40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e345e20 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e339ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76cb7fe640 0 client.17514 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/376493822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df601b0 0x56551e339ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 1 -- 172.21.15.16:0/376493822 >> 172.21.15.16:0/376493822 conn(0x56551df7d460 msgr2=0x56551e3314e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 1 -- 172.21.15.16:0/376493822 shutdown_connections 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 1 -- 172.21.15.16:0/376493822 wait complete. 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.909+0000 7f76f8ea4740 20 client.17514 trim_cache size 0 max 16384 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.926+0000 7f76f8ea4740 1 Processor -- start 2024-08-28T19:29:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.926+0000 7f76f8ea4740 1 -- start start 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.926+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e331b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.926+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e336ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.926+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df89070 0x56551e3371e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.926+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec1b3150 con 0x56551df601b0 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.926+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec1b4de0 con 0x56551df68390 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.926+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec1b3880 con 0x56551df89070 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76eb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e331b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76eb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e331b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60462/0 (socket says 172.21.15.16:60462) 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76eb7fe640 1 -- 172.21.15.16:0/355491964 learned_addr learned my addr 172.21.15.16:0/355491964 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76ebfff640 1 --2- 172.21.15.16:0/355491964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e336ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/355491964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df89070 0x56551e3371e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76eb7fe640 1 -- 172.21.15.16:0/355491964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df89070 msgr2=0x56551e3371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/355491964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df89070 0x56551e3371e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76eb7fe640 1 -- 172.21.15.16:0/355491964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 msgr2=0x56551e336ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/355491964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e336ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76eb7fe640 1 -- 172.21.15.16:0/355491964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56551e33c660 con 0x56551df601b0 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/355491964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e331b60 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f76e002a930 tx=0x7f76e0031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974c75b2dc57057b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76d1ffb640 1 -- 172.21.15.16:0/355491964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0046070 con 0x56551df601b0 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76d1ffb640 1 -- 172.21.15.16:0/355491964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e003da50 con 0x56551df601b0 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.927+0000 7f76d1ffb640 1 -- 172.21.15.16:0/355491964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0035040 con 0x56551df601b0 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.928+0000 7f76f8ea4740 1 -- 172.21.15.16:0/355491964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 msgr2=0x56551e331b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.928+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/355491964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e331b60 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f76e002a930 tx=0x7f76e0031400 comp rx=0 tx=0).stop 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.928+0000 7f76f8ea4740 1 -- 172.21.15.16:0/355491964 shutdown_connections 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.928+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/355491964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df89070 0x56551e3371e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.928+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/355491964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df68390 0x56551e336ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.928+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/355491964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df601b0 0x56551e331b60 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.928+0000 7f76f8ea4740 1 -- 172.21.15.16:0/355491964 >> 172.21.15.16:0/355491964 conn(0x56551df7d460 msgr2=0x56551e331450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.928+0000 7f76f8ea4740 1 -- 172.21.15.16:0/355491964 shutdown_connections 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.928+0000 7f76f8ea4740 1 -- 172.21.15.16:0/355491964 wait complete. 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.929+0000 7f76f8ea4740 1 -- start start 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.929+0000 7f76f8ea4740 1 Processor -- start 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.929+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df601b0 0x56551e33e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.929+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df68390 0x56551e33ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.929+0000 7f76f8ea4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e3af920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.929+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec1b4680 con 0x56551df89070 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.929+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec1b3f50 con 0x56551df601b0 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.929+0000 7f76f8ea4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec1b1150 con 0x56551df68390 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.929+0000 7f76f8ea4740 10 client.0 _ll_register_callbacks cb 0x7f76e0031950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.929+0000 7f76eb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e3af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76eb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e3af920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60468/0 (socket says 172.21.15.16:60468) 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76eb7fe640 1 -- 172.21.15.16:0/1041002395 learned_addr learned my addr 172.21.15.16:0/1041002395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76ebfff640 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df68390 0x56551e33ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df601b0 0x56551e33e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76eb7fe640 1 -- 172.21.15.16:0/1041002395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df68390 msgr2=0x56551e33ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df68390 0x56551e33ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76eb7fe640 1 -- 172.21.15.16:0/1041002395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df601b0 msgr2=0x56551e33e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df601b0 0x56551e33e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76eb7fe640 1 -- 172.21.15.16:0/1041002395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56551e3aff90 con 0x56551df89070 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76eb7fe640 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e3af920 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f76e0031890 tx=0x7f76e00038e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6961338133759aa0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76cb7fe640 1 -- 172.21.15.16:0/1041002395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e005a030 con 0x56551df89070 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76cb7fe640 1 -- 172.21.15.16:0/1041002395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f76e0003a80 con 0x56551df89070 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76f8ea4740 1 -- 172.21.15.16:0/1041002395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76e003da50 con 0x56551df89070 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76cb7fe640 1 -- 172.21.15.16:0/1041002395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f76e0069050 con 0x56551df89070 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76f8ea4740 10 client.17520 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.930+0000 7f76f8ea4740 1 -- 172.21.15.16:0/1041002395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56551e3b72d0 con 0x56551df89070 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.931+0000 7f76cb7fe640 1 -- 172.21.15.16:0/1041002395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e003da50 con 0x56551df89070 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.931+0000 7f76cb7fe640 1 -- 172.21.15.16:0/1041002395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f76e0056a50 con 0x56551df89070 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.931+0000 7f76f8ea4740 10 client.17520 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.931+0000 7f76f8ea4740 10 client.17520 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.931+0000 7f76f8ea4740 20 client.17520 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.931+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56551df1cda0 0x56551e3b9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.931+0000 7f76f8ea4740 4 client.17520 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.931+0000 7f76f8ea4740 1 -- 172.21.15.16:0/1041002395 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56551e33a420 con 0x56551df1cda0 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.931+0000 7f76ebfff640 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56551df1cda0 0x56551e3b9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.931+0000 7f76cb7fe640 10 client.17520 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.931+0000 7f76ebfff640 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56551df1cda0 0x56551e3b9a30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=44738366e5a2bbf0 server_cookie=2e5086911823cbb8 in_seq=0 out_seq=0 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76cb7fe640 1 -- 172.21.15.16:0/1041002395 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56551e33a420 con 0x56551df1cda0 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76cb7fe640 10 client.17520 handle_command_reply: tid=0 2024-08-28T19:29:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 1 client.17520 shutdown 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 1 -- 172.21.15.16:0/1041002395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 msgr2=0x56551e3af920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e3af920 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f76e0031890 tx=0x7f76e00038e0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 1 -- 172.21.15.16:0/1041002395 shutdown_connections 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56551df1cda0 0x56551e3b9a30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56551df89070 0x56551e3af920 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56551df68390 0x56551e33ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76cb7fe640 0 client.17520 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 1 --2- 172.21.15.16:0/1041002395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56551df601b0 0x56551e33e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 1 -- 172.21.15.16:0/1041002395 >> 172.21.15.16:0/1041002395 conn(0x56551df7d460 msgr2=0x56551e32f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 1 -- 172.21.15.16:0/1041002395 shutdown_connections 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 1 -- 172.21.15.16:0/1041002395 wait complete. 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.932+0000 7f76f8ea4740 20 client.17520 trim_cache size 0 max 16384 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 -- 172.21.15.16:0/900629291 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76d40b4d20 msgr2=0x7f76d40b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76d40b4d20 0x7f76d40b7230 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f76e40a1020 tx=0x7f76e4075be0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 -- 172.21.15.16:0/900629291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76ec108a90 msgr2=0x7f76ec105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76ec108a90 0x7f76ec105900 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f76e400cad0 tx=0x7f76e400cf90 comp rx=0 tx=0).stop 2024-08-28T19:29:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 -- 172.21.15.16:0/900629291 shutdown_connections 2024-08-28T19:29:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76ec108a90 0x7f76ec105900 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76ec108140 0x7f76ec1053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76d40b4d20 0x7f76d40b7230 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 --2- 172.21.15.16:0/900629291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec102800 0x7f76ec104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 -- 172.21.15.16:0/900629291 >> 172.21.15.16:0/900629291 conn(0x7f76ec0794d0 msgr2=0x7f76ec06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 -- 172.21.15.16:0/900629291 shutdown_connections 2024-08-28T19:29:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:03.933+0000 7f76f292f640 1 -- 172.21.15.16:0/900629291 wait complete. 2024-08-28T19:29:04.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:04 smithi028 ceph-mon[21769]: osdmap e87: 12 total, 12 up, 12 in 2024-08-28T19:29:04.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:04 smithi115 ceph-mon[22184]: osdmap e87: 12 total, 12 up, 12 in 2024-08-28T19:29:04.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:04 smithi016 ceph-mon[18556]: osdmap e87: 12 total, 12 up, 12 in 2024-08-28T19:29:04.701 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:29:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:04.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 381255 4.69 MB/sec execute 284 sec latency 2938.446 ms 2024-08-28T19:29:05.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.262+0000 7fad2b1e8640 1 -- 172.21.15.16:0/452543747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24108aa0 msgr2=0x7fad2410f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:05.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.262+0000 7fad2b1e8640 1 --2- 172.21.15.16:0/452543747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24108aa0 0x7fad2410f2d0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009ca0 tx=0x7fad1c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.262+0000 7fad2b1e8640 1 -- 172.21.15.16:0/452543747 shutdown_connections 2024-08-28T19:29:05.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.262+0000 7fad2b1e8640 1 --2- 172.21.15.16:0/452543747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24108aa0 0x7fad2410f2d0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.262+0000 7fad2b1e8640 1 --2- 172.21.15.16:0/452543747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad24108150 0x7fad24108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.262+0000 7fad2b1e8640 1 --2- 172.21.15.16:0/452543747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad24102860 0x7fad24102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.262+0000 7fad2b1e8640 1 -- 172.21.15.16:0/452543747 >> 172.21.15.16:0/452543747 conn(0x7fad24079530 msgr2=0x7fad24079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:05.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.262+0000 7fad2b1e8640 1 -- 172.21.15.16:0/452543747 shutdown_connections 2024-08-28T19:29:05.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.262+0000 7fad2b1e8640 1 -- 172.21.15.16:0/452543747 wait complete. 2024-08-28T19:29:05.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad2b1e8640 1 Processor -- start 2024-08-28T19:29:05.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad2b1e8640 1 -- start start 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad2b1e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad24102860 0x7fad2419b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad2b1e8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad24108150 0x7fad2419b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad2b1e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24108aa0 0x7fad2419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad2b1e8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad24112770 con 0x7fad24108aa0 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad2b1e8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad241125f0 con 0x7fad24102860 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad2b1e8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fad241128f0 con 0x7fad24108150 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad29785640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24108aa0 0x7fad2419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad29785640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24108aa0 0x7fad2419bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49608/0 (socket says 172.21.15.16:49608) 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad29785640 1 -- 172.21.15.16:0/532730480 learned_addr learned my addr 172.21.15.16:0/532730480 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.263+0000 7fad28f84640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad24102860 0x7fad2419b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.264+0000 7fad23fff640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad24108150 0x7fad2419b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.264+0000 7fad29785640 1 -- 172.21.15.16:0/532730480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad24108150 msgr2=0x7fad2419b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:05.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.264+0000 7fad29785640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad24108150 0x7fad2419b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.264+0000 7fad29785640 1 -- 172.21.15.16:0/532730480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad24102860 msgr2=0x7fad2419b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:05.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.264+0000 7fad29785640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad24102860 0x7fad2419b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.264+0000 7fad29785640 1 -- 172.21.15.16:0/532730480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad241a6330 con 0x7fad24108aa0 2024-08-28T19:29:05.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.264+0000 7fad29785640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24108aa0 0x7fad2419bb20 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fad1c005cc0 tx=0x7fad1c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:05.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.264+0000 7fad21ffb640 1 -- 172.21.15.16:0/532730480 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c003bf0 con 0x7fad24108aa0 2024-08-28T19:29:05.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.264+0000 7fad2b1e8640 1 -- 172.21.15.16:0/532730480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad24198e90 con 0x7fad24108aa0 2024-08-28T19:29:05.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.264+0000 7fad21ffb640 1 -- 172.21.15.16:0/532730480 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad1c003d90 con 0x7fad24108aa0 2024-08-28T19:29:05.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.264+0000 7fad21ffb640 1 -- 172.21.15.16:0/532730480 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c02fbc0 con 0x7fad24108aa0 2024-08-28T19:29:05.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.265+0000 7fad2b1e8640 1 -- 172.21.15.16:0/532730480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad241993d0 con 0x7fad24108aa0 2024-08-28T19:29:05.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.266+0000 7fad2b1e8640 1 -- 172.21.15.16:0/532730480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad24102c70 con 0x7fad24108aa0 2024-08-28T19:29:05.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.268+0000 7fad21ffb640 1 -- 172.21.15.16:0/532730480 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fad1c02fd60 con 0x7fad24108aa0 2024-08-28T19:29:05.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.268+0000 7fad21ffb640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad140b4cd0 0x7fad140b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.269+0000 7fad21ffb640 1 -- 172.21.15.16:0/532730480 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7fad1c106db0 con 0x7fad24108aa0 2024-08-28T19:29:05.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.269+0000 7fad28f84640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad140b4cd0 0x7fad140b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.269+0000 7fad28f84640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad140b4cd0 0x7fad140b71e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fad2406c120 tx=0x7fad1800b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:05.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.272+0000 7fad21ffb640 1 -- 172.21.15.16:0/532730480 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fad1c0cd780 con 0x7fad24108aa0 2024-08-28T19:29:05.284 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:05 smithi016 ceph-mon[18556]: pgmap v583: 129 pgs: 129 active+clean; 249 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-08-28T19:29:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.504+0000 7fad2b1e8640 1 -- 172.21.15.16:0/532730480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad2406b9d0 con 0x7fad24108aa0 2024-08-28T19:29:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.505+0000 7fad21ffb640 1 -- 172.21.15.16:0/532730480 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11988 (secure 0 0 0) 0x7fad1c0cd080 con 0x7fad24108aa0 2024-08-28T19:29:05.507 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:29:05.508 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":49,"btime":"2024-08-28T19:28:42:259944+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:28:41.590027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:29:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 49 2024-08-28T19:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 -- 172.21.15.16:0/532730480 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad140b4cd0 msgr2=0x7fad140b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad140b4cd0 0x7fad140b71e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fad2406c120 tx=0x7fad1800b390 comp rx=0 tx=0).stop 2024-08-28T19:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 -- 172.21.15.16:0/532730480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24108aa0 msgr2=0x7fad2419bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24108aa0 0x7fad2419bb20 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fad1c005cc0 tx=0x7fad1c02f380 comp rx=0 tx=0).stop 2024-08-28T19:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 -- 172.21.15.16:0/532730480 shutdown_connections 2024-08-28T19:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24108aa0 0x7fad2419bb20 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad24108150 0x7fad2419b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad140b4cd0 0x7fad140b71e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 --2- 172.21.15.16:0/532730480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad24102860 0x7fad2419b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 -- 172.21.15.16:0/532730480 >> 172.21.15.16:0/532730480 conn(0x7fad24079530 msgr2=0x7fad24104180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 -- 172.21.15.16:0/532730480 shutdown_connections 2024-08-28T19:29:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.507+0000 7fad2b1e8640 1 -- 172.21.15.16:0/532730480 wait complete. 2024-08-28T19:29:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:05 smithi115 ceph-mon[22184]: pgmap v583: 129 pgs: 129 active+clean; 249 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-08-28T19:29:05.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:05 smithi028 ceph-mon[21769]: pgmap v583: 129 pgs: 129 active+clean; 249 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-08-28T19:29:05.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 381255 4.67 MB/sec execute 285 sec latency 3938.472 ms 2024-08-28T19:29:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.954+0000 7fd1719cb640 1 -- 172.21.15.16:0/2147807606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c102810 msgr2=0x7fd16c102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.954+0000 7fd1719cb640 1 --2- 172.21.15.16:0/2147807606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c102810 0x7fd16c102c20 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009900 tx=0x7fd15c02a410 comp rx=0 tx=0).stop 2024-08-28T19:29:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.954+0000 7fd1719cb640 1 -- 172.21.15.16:0/2147807606 shutdown_connections 2024-08-28T19:29:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.954+0000 7fd1719cb640 1 --2- 172.21.15.16:0/2147807606 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd16c108aa0 0x7fd16c10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.954+0000 7fd1719cb640 1 --2- 172.21.15.16:0/2147807606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd16c108150 0x7fd16c108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.954+0000 7fd1719cb640 1 --2- 172.21.15.16:0/2147807606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c102810 0x7fd16c102c20 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.954+0000 7fd1719cb640 1 -- 172.21.15.16:0/2147807606 >> 172.21.15.16:0/2147807606 conn(0x7fd16c0794d0 msgr2=0x7fd16c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.954+0000 7fd1719cb640 1 -- 172.21.15.16:0/2147807606 shutdown_connections 2024-08-28T19:29:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.954+0000 7fd1719cb640 1 -- 172.21.15.16:0/2147807606 wait complete. 2024-08-28T19:29:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd1719cb640 1 Processor -- start 2024-08-28T19:29:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd1719cb640 1 -- start start 2024-08-28T19:29:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd1719cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd16c102810 0x7fd16c19b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd1719cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c108150 0x7fd16c19b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd1719cb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd16c108aa0 0x7fd16c19bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd1719cb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c112700 con 0x7fd16c108150 2024-08-28T19:29:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd1719cb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c112580 con 0x7fd16c102810 2024-08-28T19:29:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd1719cb640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c112880 con 0x7fd16c108aa0 2024-08-28T19:29:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd16a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c108150 0x7fd16c19b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd16a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c108150 0x7fd16c19b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49618/0 (socket says 172.21.15.16:49618) 2024-08-28T19:29:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd16a7fc640 1 -- 172.21.15.16:0/4032452485 learned_addr learned my addr 172.21.15.16:0/4032452485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd16affd640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd16c102810 0x7fd16c19b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.955+0000 7fd16b7fe640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd16c108aa0 0x7fd16c19bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.956+0000 7fd16a7fc640 1 -- 172.21.15.16:0/4032452485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd16c108aa0 msgr2=0x7fd16c19bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.956+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd16c108aa0 0x7fd16c19bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.956+0000 7fd16a7fc640 1 -- 172.21.15.16:0/4032452485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd16c102810 msgr2=0x7fd16c19b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.956+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd16c102810 0x7fd16c19b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.956+0000 7fd16a7fc640 1 -- 172.21.15.16:0/4032452485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c1a6350 con 0x7fd16c108150 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.956+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c108150 0x7fd16c19b5d0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd16000e9c0 tx=0x7fd16000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.956+0000 7fd1709c9640 1 -- 172.21.15.16:0/4032452485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16000cd80 con 0x7fd16c108150 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.956+0000 7fd1709c9640 1 -- 172.21.15.16:0/4032452485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1600054d0 con 0x7fd16c108150 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.956+0000 7fd1719cb640 1 -- 172.21.15.16:0/4032452485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16c1a6640 con 0x7fd16c108150 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.956+0000 7fd1709c9640 1 -- 172.21.15.16:0/4032452485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd160010430 con 0x7fd16c108150 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.956+0000 7fd1719cb640 1 -- 172.21.15.16:0/4032452485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16c1a6b80 con 0x7fd16c108150 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.959+0000 7fd177f40740 1 Processor -- start 2024-08-28T19:29:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.959+0000 7fd177f40740 1 -- start start 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.959+0000 7fd1709c9640 1 -- 172.21.15.16:0/4032452485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fd160004fd0 con 0x7fd16c108150 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.959+0000 7fd177f40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6c4070 0x55919ea69310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.959+0000 7fd177f40740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e69b1b0 0x55919ea69850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.959+0000 7fd177f40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea69d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.959+0000 7fd177f40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55919e9e0680 con 0x55919e6a3390 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.959+0000 7fd177f40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55919e9cfc10 con 0x55919e6c4070 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.959+0000 7fd177f40740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55919e9d3bc0 con 0x55919e69b1b0 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.959+0000 7fd16a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea69d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd16a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea69d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49622/0 (socket says 172.21.15.16:49622) 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd16a7fc640 1 -- 172.21.15.16:0/770101635 learned_addr learned my addr 172.21.15.16:0/770101635 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd16b7fe640 1 --2- 172.21.15.16:0/770101635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6c4070 0x55919ea69310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd16affd640 1 --2- 172.21.15.16:0/770101635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e69b1b0 0x55919ea69850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd16a7fc640 1 -- 172.21.15.16:0/770101635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e69b1b0 msgr2=0x55919ea69850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/770101635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e69b1b0 0x55919ea69850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd16a7fc640 1 -- 172.21.15.16:0/770101635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6c4070 msgr2=0x55919ea69310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/770101635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6c4070 0x55919ea69310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd16a7fc640 1 -- 172.21.15.16:0/770101635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55919e72f650 con 0x55919e6a3390 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd1709c9640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1540b4ca0 0x7fd1540b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd16b7fe640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1540b4ca0 0x7fd1540b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/770101635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea69d90 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd1600e3010 tx=0x7fd1600e9e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11361111245cf538 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd1709c9640 1 -- 172.21.15.16:0/4032452485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7fd1600e5d70 con 0x7fd16c108150 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.960+0000 7fd151ffb640 1 -- 172.21.15.16:0/770101635 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1600a7f00 con 0x55919e6a3390 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.961+0000 7fd151ffb640 1 -- 172.21.15.16:0/770101635 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1600046c0 con 0x55919e6a3390 2024-08-28T19:29:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.961+0000 7fd151ffb640 1 -- 172.21.15.16:0/770101635 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1600ab7b0 con 0x55919e6a3390 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.961+0000 7fd177f40740 1 -- 172.21.15.16:0/770101635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 msgr2=0x55919ea69d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.961+0000 7fd16b7fe640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1540b4ca0 0x7fd1540b71b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd164005fd0 tx=0x7fd1640096a0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.961+0000 7fd177f40740 1 --2- 172.21.15.16:0/770101635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea69d90 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd1600e3010 tx=0x7fd1600e9e30 comp rx=0 tx=0).stop 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.961+0000 7fd177f40740 1 -- 172.21.15.16:0/770101635 shutdown_connections 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.961+0000 7fd177f40740 1 --2- 172.21.15.16:0/770101635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea69d90 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.961+0000 7fd177f40740 1 --2- 172.21.15.16:0/770101635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e69b1b0 0x55919ea69850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.961+0000 7fd177f40740 1 --2- 172.21.15.16:0/770101635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6c4070 0x55919ea69310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.961+0000 7fd177f40740 1 -- 172.21.15.16:0/770101635 >> 172.21.15.16:0/770101635 conn(0x55919e6b8460 msgr2=0x55919ea67200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.962+0000 7fd177f40740 1 -- 172.21.15.16:0/770101635 shutdown_connections 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.962+0000 7fd177f40740 1 -- 172.21.15.16:0/770101635 wait complete. 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.962+0000 7fd177f40740 1 -- start start 2024-08-28T19:29:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.962+0000 7fd177f40740 1 Processor -- start 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.962+0000 7fd177f40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e69b1b0 0x55919ea74a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.962+0000 7fd177f40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6a3390 0x55919ea74fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd177f40740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6c4070 0x55919ea80e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd177f40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55919e832700 con 0x55919e69b1b0 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd177f40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55919e956750 con 0x55919e6a3390 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd177f40740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55919ea77b40 con 0x55919e6c4070 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd177f40740 10 client.0 _ll_register_callbacks cb 0x55919e985e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e69b1b0 0x55919ea74a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e69b1b0 0x55919ea74a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49624/0 (socket says 172.21.15.16:49624) 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16affd640 1 -- 172.21.15.16:0/3132367683 learned_addr learned my addr 172.21.15.16:0/3132367683 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6a3390 0x55919ea74fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16b7fe640 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6c4070 0x55919ea80e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16affd640 1 -- 172.21.15.16:0/3132367683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6c4070 msgr2=0x55919ea80e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16affd640 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6c4070 0x55919ea80e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16affd640 1 -- 172.21.15.16:0/3132367683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6a3390 msgr2=0x55919ea74fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16affd640 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6a3390 0x55919ea74fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16affd640 1 -- 172.21.15.16:0/3132367683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55919e98f900 con 0x55919e69b1b0 2024-08-28T19:29:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16b7fe640 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6c4070 0x55919ea80e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd16affd640 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e69b1b0 0x55919ea74a80 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009900 tx=0x7fd15c005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a88692b99415e98 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd14affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd14affd640 1 -- 172.21.15.16:0/3132367683 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c035040 con 0x55919e69b1b0 2024-08-28T19:29:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.963+0000 7fd14affd640 1 -- 172.21.15.16:0/3132367683 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7fd15c0317a0 con 0x55919e69b1b0 2024-08-28T19:29:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd177f40740 1 -- 172.21.15.16:0/3132367683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55919e9e25d0 con 0x55919e69b1b0 2024-08-28T19:29:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd14affd640 1 -- 172.21.15.16:0/3132367683 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fd15c004040 con 0x55919e69b1b0 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd177f40740 10 client.17538 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd177f40740 1 -- 172.21.15.16:0/3132367683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55919ea83910 con 0x55919e69b1b0 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd14affd640 1 -- 172.21.15.16:0/3132367683 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55919e9e25d0 con 0x55919e69b1b0 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd14affd640 1 -- 172.21.15.16:0/3132367683 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7fd15c004770 con 0x55919e69b1b0 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd177f40740 10 client.17538 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd177f40740 10 client.17538 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd177f40740 20 client.17538 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd177f40740 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55919e6829f0 0x55919ea86e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd177f40740 4 client.17538 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd177f40740 1 -- 172.21.15.16:0/3132367683 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55919e9de070 con 0x55919e6829f0 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.964+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55919e6829f0 0x55919ea86e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.965+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55919e6829f0 0x55919ea86e20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=50f3fc58c0a9d65d server_cookie=534985073254650d in_seq=0 out_seq=0 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.965+0000 7fd14affd640 10 client.17538 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd14affd640 1 -- 172.21.15.16:0/3132367683 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55919e9de070 con 0x55919e6829f0 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd14affd640 10 client.17538 handle_command_reply: tid=0 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 1 client.17538 shutdown 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 1 -- 172.21.15.16:0/3132367683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e69b1b0 msgr2=0x55919ea74a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e69b1b0 0x55919ea74a80 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009900 tx=0x7fd15c005f00 comp rx=0 tx=0).stop 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 1 -- 172.21.15.16:0/3132367683 shutdown_connections 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55919e6829f0 0x55919ea86e20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6c4070 0x55919ea80e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6a3390 0x55919ea74fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd14affd640 0 client.17538 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:29:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 1 --2- 172.21.15.16:0/3132367683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e69b1b0 0x55919ea74a80 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 1 -- 172.21.15.16:0/3132367683 >> 172.21.15.16:0/3132367683 conn(0x55919e6b8460 msgr2=0x55919ea6a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 1 -- 172.21.15.16:0/3132367683 shutdown_connections 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 1 -- 172.21.15.16:0/3132367683 wait complete. 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.966+0000 7fd177f40740 20 client.17538 trim_cache size 0 max 16384 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.979+0000 7fd177f40740 1 Processor -- start 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.979+0000 7fd177f40740 1 -- start start 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.979+0000 7fd177f40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e69b1b0 0x55919ea80d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.979+0000 7fd177f40740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6a3390 0x55919ea766b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.979+0000 7fd177f40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6c4070 0x55919ea76bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd177f40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1b0dc0 con 0x55919e6c4070 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd177f40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1b2a50 con 0x55919e69b1b0 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd177f40740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1b14f0 con 0x55919e6a3390 2024-08-28T19:29:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6c4070 0x55919ea76bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6c4070 0x55919ea76bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49636/0 (socket says 172.21.15.16:49636) 2024-08-28T19:29:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16b7fe640 1 -- 172.21.15.16:0/2794339393 learned_addr learned my addr 172.21.15.16:0/2794339393 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16affd640 1 --2- 172.21.15.16:0/2794339393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e69b1b0 0x55919ea80d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/2794339393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6a3390 0x55919ea766b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16b7fe640 1 -- 172.21.15.16:0/2794339393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6a3390 msgr2=0x55919ea766b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16b7fe640 1 --2- 172.21.15.16:0/2794339393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6a3390 0x55919ea766b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16b7fe640 1 -- 172.21.15.16:0/2794339393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e69b1b0 msgr2=0x55919ea80d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16b7fe640 1 --2- 172.21.15.16:0/2794339393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e69b1b0 0x55919ea80d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16b7fe640 1 -- 172.21.15.16:0/2794339393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55919ea6bba0 con 0x55919e6c4070 2024-08-28T19:29:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/2794339393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6a3390 0x55919ea766b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:29:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd16b7fe640 1 --2- 172.21.15.16:0/2794339393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6c4070 0x55919ea76bf0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd164004010 tx=0x7fd164016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8ba33b9ee553b7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd1517fa640 1 -- 172.21.15.16:0/2794339393 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16402d070 con 0x55919e6c4070 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd1517fa640 1 -- 172.21.15.16:0/2794339393 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd16401db00 con 0x55919e6c4070 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.980+0000 7fd1517fa640 1 -- 172.21.15.16:0/2794339393 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164028a30 con 0x55919e6c4070 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.981+0000 7fd177f40740 1 -- 172.21.15.16:0/2794339393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6c4070 msgr2=0x55919ea76bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.981+0000 7fd177f40740 1 --2- 172.21.15.16:0/2794339393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6c4070 0x55919ea76bf0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd164004010 tx=0x7fd164016000 comp rx=0 tx=0).stop 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.981+0000 7fd177f40740 1 -- 172.21.15.16:0/2794339393 shutdown_connections 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.981+0000 7fd177f40740 1 --2- 172.21.15.16:0/2794339393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6c4070 0x55919ea76bf0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.981+0000 7fd177f40740 1 --2- 172.21.15.16:0/2794339393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e6a3390 0x55919ea766b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.981+0000 7fd177f40740 1 --2- 172.21.15.16:0/2794339393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e69b1b0 0x55919ea80d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.981+0000 7fd177f40740 1 -- 172.21.15.16:0/2794339393 >> 172.21.15.16:0/2794339393 conn(0x55919e6b8460 msgr2=0x55919ea75fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.981+0000 7fd177f40740 1 -- 172.21.15.16:0/2794339393 shutdown_connections 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.981+0000 7fd177f40740 1 -- 172.21.15.16:0/2794339393 wait complete. 2024-08-28T19:29:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.981+0000 7fd177f40740 1 -- start start 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd177f40740 1 Processor -- start 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd177f40740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e69b1b0 0x55919ea79690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd177f40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea79c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd177f40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6c4070 0x55919eaea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd177f40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1b22f0 con 0x55919e6a3390 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd177f40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1b1bc0 con 0x55919e6c4070 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd177f40740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1aedc0 con 0x55919e69b1b0 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd177f40740 10 client.0 _ll_register_callbacks cb 0x7fd15c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd16a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea79c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd16a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea79c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49638/0 (socket says 172.21.15.16:49638) 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd16a7fc640 1 -- 172.21.15.16:0/2224704462 learned_addr learned my addr 172.21.15.16:0/2224704462 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd16b7fe640 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6c4070 0x55919eaea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd16affd640 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e69b1b0 0x55919ea79690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd16a7fc640 1 -- 172.21.15.16:0/2224704462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e69b1b0 msgr2=0x55919ea79690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e69b1b0 0x55919ea79690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd16a7fc640 1 -- 172.21.15.16:0/2224704462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6c4070 msgr2=0x55919eaea7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6c4070 0x55919eaea7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.982+0000 7fd16a7fc640 1 -- 172.21.15.16:0/2224704462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55919eaeae20 con 0x55919e6a3390 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd16a7fc640 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea79c70 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd160093040 tx=0x7fd16000efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35e5a12a1614c9e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd14affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd14affd640 1 -- 172.21.15.16:0/2224704462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1600ab990 con 0x55919e6a3390 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd14affd640 1 -- 172.21.15.16:0/2224704462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7fd1600e3320 con 0x55919e6a3390 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd177f40740 1 -- 172.21.15.16:0/2224704462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd16401db00 con 0x55919e6a3390 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd14affd640 1 -- 172.21.15.16:0/2224704462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fd1600abe10 con 0x55919e6a3390 2024-08-28T19:29:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd177f40740 10 client.17544 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd177f40740 1 -- 172.21.15.16:0/2224704462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55919eaf20b0 con 0x55919e6a3390 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd14affd640 1 -- 172.21.15.16:0/2224704462 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd16401db00 con 0x55919e6a3390 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd14affd640 1 -- 172.21.15.16:0/2224704462 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7fd1600ec110 con 0x55919e6a3390 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd177f40740 10 client.17544 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd177f40740 10 client.17544 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd177f40740 20 client.17544 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd177f40740 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55919e657da0 0x55919eaf4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd177f40740 4 client.17544 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.983+0000 7fd177f40740 1 -- 172.21.15.16:0/2224704462 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55919e9db7b0 con 0x55919e657da0 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.984+0000 7fd16affd640 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55919e657da0 0x55919eaf4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.984+0000 7fd14affd640 10 client.17544 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.984+0000 7fd16affd640 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55919e657da0 0x55919eaf4450 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=400b0600a74345ef server_cookie=b71177f6db72db5f in_seq=0 out_seq=0 2024-08-28T19:29:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.984+0000 7fd14affd640 1 -- 172.21.15.16:0/2224704462 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55919e9db7b0 con 0x55919e657da0 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.984+0000 7fd14affd640 10 client.17544 handle_command_reply: tid=0 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.984+0000 7fd177f40740 1 client.17544 shutdown 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd177f40740 1 -- 172.21.15.16:0/2224704462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 msgr2=0x55919ea79c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd177f40740 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea79c70 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd160093040 tx=0x7fd16000efb0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd177f40740 1 -- 172.21.15.16:0/2224704462 shutdown_connections 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd177f40740 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55919e657da0 0x55919eaf4450 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd177f40740 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919e6c4070 0x55919eaea7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd177f40740 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55919e6a3390 0x55919ea79c70 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd177f40740 1 --2- 172.21.15.16:0/2224704462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55919e69b1b0 0x55919ea79690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd177f40740 1 -- 172.21.15.16:0/2224704462 >> 172.21.15.16:0/2224704462 conn(0x55919e6b8460 msgr2=0x55919ea768f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd14affd640 0 client.17544 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:29:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd177f40740 1 -- 172.21.15.16:0/2224704462 shutdown_connections 2024-08-28T19:29:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd177f40740 1 -- 172.21.15.16:0/2224704462 wait complete. 2024-08-28T19:29:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd177f40740 20 client.17544 trim_cache size 0 max 16384 2024-08-28T19:29:06.006 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:29:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd1719cb640 1 -- 172.21.15.16:0/4032452485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1540b4ca0 msgr2=0x7fd1540b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd1719cb640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1540b4ca0 0x7fd1540b71b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd164005fd0 tx=0x7fd1640096a0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd1719cb640 1 -- 172.21.15.16:0/4032452485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c108150 msgr2=0x7fd16c19b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd1719cb640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c108150 0x7fd16c19b5d0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd16000e9c0 tx=0x7fd16000ee80 comp rx=0 tx=0).stop 2024-08-28T19:29:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.985+0000 7fd1719cb640 1 -- 172.21.15.16:0/4032452485 shutdown_connections 2024-08-28T19:29:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.986+0000 7fd1719cb640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd16c108aa0 0x7fd16c19bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.986+0000 7fd1719cb640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c108150 0x7fd16c19b5d0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.986+0000 7fd1719cb640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd1540b4ca0 0x7fd1540b71b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.986+0000 7fd1719cb640 1 --2- 172.21.15.16:0/4032452485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd16c102810 0x7fd16c19b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.986+0000 7fd1719cb640 1 -- 172.21.15.16:0/4032452485 >> 172.21.15.16:0/4032452485 conn(0x7fd16c0794d0 msgr2=0x7fd16c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.986+0000 7fd1719cb640 1 -- 172.21.15.16:0/4032452485 shutdown_connections 2024-08-28T19:29:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:05.986+0000 7fd1719cb640 1 -- 172.21.15.16:0/4032452485 wait complete. 2024-08-28T19:29:06.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:06 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/532730480' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:06.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:06 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/532730480' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:06.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:29:06.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:06 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/532730480' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:06.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 382012 4.66 MB/sec execute 286 sec latency 4583.042 ms 2024-08-28T19:29:07.127 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 139.5 --await-for 45.0' 2024-08-28T19:29:07.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 139.5 --await-for 45.0 2024-08-28T19:29:07.190 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:07.593 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:07 smithi016 ceph-mon[18556]: pgmap v584: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 225 op/s 2024-08-28T19:29:07.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:07 smithi115 ceph-mon[22184]: pgmap v584: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 225 op/s 2024-08-28T19:29:07.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.690+0000 7f57c9feb640 1 -- 172.21.15.16:0/3231779418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c4108140 msgr2=0x7f57c4108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:07.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.690+0000 7f57c9feb640 1 --2- 172.21.15.16:0/3231779418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c4108140 0x7f57c4108550 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009900 tx=0x7f57ac02a410 comp rx=0 tx=0).stop 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.690+0000 7f57c9feb640 1 -- 172.21.15.16:0/3231779418 shutdown_connections 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.690+0000 7f57c9feb640 1 --2- 172.21.15.16:0/3231779418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c4108a90 0x7f57c410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.690+0000 7f57c9feb640 1 --2- 172.21.15.16:0/3231779418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c4108140 0x7f57c4108550 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.690+0000 7f57c9feb640 1 --2- 172.21.15.16:0/3231779418 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57c4102800 0x7f57c4102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.690+0000 7f57c9feb640 1 -- 172.21.15.16:0/3231779418 >> 172.21.15.16:0/3231779418 conn(0x7f57c40794d0 msgr2=0x7f57c40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.690+0000 7f57c9feb640 1 -- 172.21.15.16:0/3231779418 shutdown_connections 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.690+0000 7f57c9feb640 1 -- 172.21.15.16:0/3231779418 wait complete. 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.691+0000 7f57c9feb640 1 Processor -- start 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.691+0000 7f57c9feb640 1 -- start start 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57c9feb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c4102800 0x7f57c419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57c9feb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c4108140 0x7f57c419b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57c9feb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57c4108a90 0x7f57c419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57c9feb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41126c0 con 0x7f57c4102800 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57c9feb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57c4112540 con 0x7f57c4108140 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57c9feb640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f57c4112840 con 0x7f57c4108a90 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57baffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c4108140 0x7f57c419b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c4102800 0x7f57c419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57c4108a90 0x7f57c419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57baffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c4108140 0x7f57c419b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54922/0 (socket says 172.21.15.16:54922) 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57baffd640 1 -- 172.21.15.16:0/2339701533 learned_addr learned my addr 172.21.15.16:0/2339701533 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57baffd640 1 -- 172.21.15.16:0/2339701533 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57c4108a90 msgr2=0x7f57c419bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57baffd640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57c4108a90 0x7f57c419bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57baffd640 1 -- 172.21.15.16:0/2339701533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c4102800 msgr2=0x7f57c419b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57baffd640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c4102800 0x7f57c419b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57baffd640 1 -- 172.21.15.16:0/2339701533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c41a62d0 con 0x7f57c4108140 2024-08-28T19:29:07.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57c3fff640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57c4108a90 0x7f57c419bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:07.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57c37fe640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c4102800 0x7f57c419b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:07.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.692+0000 7f57baffd640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c4108140 0x7f57c419b610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f57ac032fd0 tx=0x7f57ac004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:07.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.693+0000 7f57c17fa640 1 -- 172.21.15.16:0/2339701533 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac03f030 con 0x7f57c4108140 2024-08-28T19:29:07.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.693+0000 7f57c9feb640 1 -- 172.21.15.16:0/2339701533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c41a6560 con 0x7f57c4108140 2024-08-28T19:29:07.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.693+0000 7f57c17fa640 1 -- 172.21.15.16:0/2339701533 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57ac004490 con 0x7f57c4108140 2024-08-28T19:29:07.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.693+0000 7f57c17fa640 1 -- 172.21.15.16:0/2339701533 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac035070 con 0x7f57c4108140 2024-08-28T19:29:07.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.693+0000 7f57c9feb640 1 -- 172.21.15.16:0/2339701533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c41a6a40 con 0x7f57c4108140 2024-08-28T19:29:07.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.696+0000 7f57c17fa640 1 -- 172.21.15.16:0/2339701533 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f57ac042030 con 0x7f57c4108140 2024-08-28T19:29:07.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.696+0000 7f57d0560740 1 Processor -- start 2024-08-28T19:29:07.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.696+0000 7f57d0560740 1 -- start start 2024-08-28T19:29:07.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.696+0000 7f57d0560740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045d0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.696+0000 7f57d0560740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.696+0000 7f57d0560740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559204206390 0x5592045d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.696+0000 7f57d0560740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559204543680 con 0x559204206390 2024-08-28T19:29:07.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.696+0000 7f57d0560740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559204532c10 con 0x5592041fe1b0 2024-08-28T19:29:07.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.696+0000 7f57d0560740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559204536bc0 con 0x559204227070 2024-08-28T19:29:07.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.696+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045d0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54932/0 (socket says 172.21.15.16:54932) 2024-08-28T19:29:07.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c37fe640 1 -- 172.21.15.16:0/3327421524 learned_addr learned my addr 172.21.15.16:0/3327421524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:07.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045d0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34468/0 (socket says 172.21.15.16:34468) 2024-08-28T19:29:07.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57baffd640 1 --2- 172.21.15.16:0/3327421524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559204206390 0x5592045d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c17fa640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57b00b4cd0 0x7f57b00b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c37fe640 1 -- 172.21.15.16:0/3327421524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 msgr2=0x5592045d0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:07.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c37fe640 1 --2- 172.21.15.16:0/3327421524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045d0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c37fe640 1 -- 172.21.15.16:0/3327421524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559204206390 msgr2=0x5592045d11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:07.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c37fe640 1 --2- 172.21.15.16:0/3327421524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559204206390 0x5592045d11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c37fe640 1 -- 172.21.15.16:0/3327421524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559204292650 con 0x5592041fe1b0 2024-08-28T19:29:07.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c17fa640 1 -- 172.21.15.16:0/2339701533 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f57ac105e10 con 0x7f57c4108140 2024-08-28T19:29:07.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c3fff640 1 --2- 172.21.15.16:0/3327421524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045d0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:07.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c3fff640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57b00b4cd0 0x7f57b00b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.697+0000 7f57c37fe640 1 --2- 172.21.15.16:0/3327421524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045d0ca0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57b400f470 tx=0x7f57b400f930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a36a97434839b72e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:07.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.698+0000 7f57b9ffb640 1 -- 172.21.15.16:0/3327421524 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4005220 con 0x5592041fe1b0 2024-08-28T19:29:07.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.698+0000 7f57b9ffb640 1 -- 172.21.15.16:0/3327421524 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57b4004c50 con 0x5592041fe1b0 2024-08-28T19:29:07.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.698+0000 7f57b9ffb640 1 -- 172.21.15.16:0/3327421524 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4010c00 con 0x5592041fe1b0 2024-08-28T19:29:07.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.699+0000 7f57d0560740 1 -- 172.21.15.16:0/3327421524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 msgr2=0x5592045d0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:07.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.699+0000 7f57d0560740 1 --2- 172.21.15.16:0/3327421524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045d0ca0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57b400f470 tx=0x7f57b400f930 comp rx=0 tx=0).stop 2024-08-28T19:29:07.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.699+0000 7f57c3fff640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57b00b4cd0 0x7f57b00b71e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f57bc005fd0 tx=0x7f57bc004a40 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:07.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.699+0000 7f57d0560740 1 -- 172.21.15.16:0/3327421524 shutdown_connections 2024-08-28T19:29:07.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.699+0000 7f57d0560740 1 --2- 172.21.15.16:0/3327421524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559204206390 0x5592045d11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.699+0000 7f57d0560740 1 --2- 172.21.15.16:0/3327421524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045d0ca0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.699+0000 7f57d0560740 1 --2- 172.21.15.16:0/3327421524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045d0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.699+0000 7f57d0560740 1 -- 172.21.15.16:0/3327421524 >> 172.21.15.16:0/3327421524 conn(0x55920421b460 msgr2=0x5592045ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.699+0000 7f57d0560740 1 -- 172.21.15.16:0/3327421524 shutdown_connections 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.699+0000 7f57d0560740 1 -- 172.21.15.16:0/3327421524 wait complete. 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.699+0000 7f57d0560740 1 -- start start 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57d0560740 1 Processor -- start 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57d0560740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045dbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57d0560740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559204206390 0x5592045dc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57d0560740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045e8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57d0560740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559204395700 con 0x5592041fe1b0 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57d0560740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5592044b9750 con 0x559204206390 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57d0560740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5592045def40 con 0x559204227070 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045dbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045dbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49658/0 (socket says 172.21.15.16:49658) 2024-08-28T19:29:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57c37fe640 1 -- 172.21.15.16:0/3025911715 learned_addr learned my addr 172.21.15.16:0/3025911715 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57c3fff640 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045e8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57d0560740 10 client.0 _ll_register_callbacks cb 0x5592044e8e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.700+0000 7f57baffd640 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559204206390 0x5592045dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57c3fff640 1 -- 172.21.15.16:0/3025911715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559204206390 msgr2=0x5592045dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57c3fff640 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559204206390 0x5592045dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57c3fff640 1 -- 172.21.15.16:0/3025911715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 msgr2=0x5592045dbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57c3fff640 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045dbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57c3fff640 1 -- 172.21.15.16:0/3025911715 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592044f2900 con 0x559204227070 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57c37fe640 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045dbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57c3fff640 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045e8230 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009fd0 tx=0x7f57bc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b9480491693201a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3025911715 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc022df0 con 0x559204227070 2024-08-28T19:29:07.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57d0560740 1 -- 172.21.15.16:0/3025911715 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57b4004c50 con 0x559204227070 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3025911715 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f57bc0240d0 con 0x559204227070 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3025911715 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f57bc0242d0 con 0x559204227070 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57d0560740 10 client.26878 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.701+0000 7f57d0560740 1 -- 172.21.15.16:0/3025911715 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5592045c9bc0 con 0x559204227070 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.702+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3025911715 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57b4004c50 con 0x559204227070 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.702+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3025911715 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f57bc011a90 con 0x559204227070 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.702+0000 7f57d0560740 10 client.26878 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.702+0000 7f57d0560740 10 client.26878 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.702+0000 7f57d0560740 20 client.26878 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.702+0000 7f57d0560740 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5592041e59f0 0x5592045cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.702+0000 7f57d0560740 4 client.26878 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.702+0000 7f57d0560740 1 -- 172.21.15.16:0/3025911715 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592045455d0 con 0x5592041e59f0 2024-08-28T19:29:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.702+0000 7f57c37fe640 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5592041e59f0 0x5592045cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.703+0000 7f57c37fe640 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5592041e59f0 0x5592045cd590 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fe869e83760af3ba server_cookie=71e697cf6c58cd8f in_seq=0 out_seq=0 2024-08-28T19:29:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.703+0000 7f57a2ffd640 10 client.26878 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3025911715 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5592045455d0 con 0x5592041e59f0 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57a2ffd640 10 client.26878 handle_command_reply: tid=0 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 1 client.26878 shutdown 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 1 -- 172.21.15.16:0/3025911715 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 msgr2=0x5592045e8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045e8230 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009fd0 tx=0x7f57bc004ab0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 1 -- 172.21.15.16:0/3025911715 shutdown_connections 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5592041e59f0 0x5592045cd590 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045e8230 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559204206390 0x5592045dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57a2ffd640 0 client.26878 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 1 --2- 172.21.15.16:0/3025911715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045dbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 1 -- 172.21.15.16:0/3025911715 >> 172.21.15.16:0/3025911715 conn(0x55920421b460 msgr2=0x5592045d38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 1 -- 172.21.15.16:0/3025911715 shutdown_connections 2024-08-28T19:29:07.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 1 -- 172.21.15.16:0/3025911715 wait complete. 2024-08-28T19:29:07.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.704+0000 7f57d0560740 20 client.26878 trim_cache size 0 max 16384 2024-08-28T19:29:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.728+0000 7f57d0560740 1 Processor -- start 2024-08-28T19:29:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.728+0000 7f57d0560740 1 -- start start 2024-08-28T19:29:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.728+0000 7f57d0560740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045ce470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.728+0000 7f57d0560740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204206390 0x5592045e8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.728+0000 7f57d0560740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559204227070 0x5592045e8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.728+0000 7f57d0560740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41aebe0 con 0x5592041fe1b0 2024-08-28T19:29:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.728+0000 7f57d0560740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41b0870 con 0x559204227070 2024-08-28T19:29:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.728+0000 7f57d0560740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41af310 con 0x559204206390 2024-08-28T19:29:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.728+0000 7f57c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.728+0000 7f57c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045ce470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49662/0 (socket says 172.21.15.16:49662) 2024-08-28T19:29:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.728+0000 7f57c37fe640 1 -- 172.21.15.16:0/122542521 learned_addr learned my addr 172.21.15.16:0/122542521 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57c3fff640 1 --2- 172.21.15.16:0/122542521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559204227070 0x5592045e8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57baffd640 1 --2- 172.21.15.16:0/122542521 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204206390 0x5592045e8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57c37fe640 1 -- 172.21.15.16:0/122542521 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204206390 msgr2=0x5592045e8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57c37fe640 1 --2- 172.21.15.16:0/122542521 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204206390 0x5592045e8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57c37fe640 1 -- 172.21.15.16:0/122542521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559204227070 msgr2=0x5592045e8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57c37fe640 1 --2- 172.21.15.16:0/122542521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559204227070 0x5592045e8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57c37fe640 1 -- 172.21.15.16:0/122542521 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592045cef40 con 0x5592041fe1b0 2024-08-28T19:29:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57c3fff640 1 --2- 172.21.15.16:0/122542521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559204227070 0x5592045e8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57c37fe640 1 --2- 172.21.15.16:0/122542521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045ce470 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f57b40107a0 tx=0x7f57b4012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b0028e4a35134 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57b97fa640 1 -- 172.21.15.16:0/122542521 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4010870 con 0x5592041fe1b0 2024-08-28T19:29:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57b97fa640 1 -- 172.21.15.16:0/122542521 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57b4004bf0 con 0x5592041fe1b0 2024-08-28T19:29:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.729+0000 7f57b97fa640 1 -- 172.21.15.16:0/122542521 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4018440 con 0x5592041fe1b0 2024-08-28T19:29:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.730+0000 7f57d0560740 1 -- 172.21.15.16:0/122542521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 msgr2=0x5592045ce470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.730+0000 7f57d0560740 1 --2- 172.21.15.16:0/122542521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045ce470 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f57b40107a0 tx=0x7f57b4012040 comp rx=0 tx=0).stop 2024-08-28T19:29:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.731+0000 7f57d0560740 1 -- 172.21.15.16:0/122542521 shutdown_connections 2024-08-28T19:29:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.731+0000 7f57d0560740 1 --2- 172.21.15.16:0/122542521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559204227070 0x5592045e8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.731+0000 7f57d0560740 1 --2- 172.21.15.16:0/122542521 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204206390 0x5592045e8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.731+0000 7f57d0560740 1 --2- 172.21.15.16:0/122542521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592041fe1b0 0x5592045ce470 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.731+0000 7f57d0560740 1 -- 172.21.15.16:0/122542521 >> 172.21.15.16:0/122542521 conn(0x55920421b460 msgr2=0x5592045cdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.731+0000 7f57d0560740 1 -- 172.21.15.16:0/122542521 shutdown_connections 2024-08-28T19:29:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.731+0000 7f57d0560740 1 -- 172.21.15.16:0/122542521 wait complete. 2024-08-28T19:29:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.732+0000 7f57d0560740 1 -- start start 2024-08-28T19:29:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.732+0000 7f57d0560740 1 Processor -- start 2024-08-28T19:29:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.732+0000 7f57d0560740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045e4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.732+0000 7f57d0560740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559204206390 0x5592045dfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.732+0000 7f57d0560740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045e0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.732+0000 7f57d0560740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41b0110 con 0x559204206390 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.732+0000 7f57d0560740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41af9e0 con 0x5592041fe1b0 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.732+0000 7f57d0560740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41acbe0 con 0x559204227070 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.732+0000 7f57d0560740 10 client.0 _ll_register_callbacks cb 0x7f57bc0103d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045e4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045e0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045e4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54948/0 (socket says 172.21.15.16:54948) 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57c37fe640 1 -- 172.21.15.16:0/3187557471 learned_addr learned my addr 172.21.15.16:0/3187557471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57baffd640 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559204206390 0x5592045dfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57c37fe640 1 -- 172.21.15.16:0/3187557471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 msgr2=0x5592045e0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57c37fe640 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045e0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57c37fe640 1 -- 172.21.15.16:0/3187557471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559204206390 msgr2=0x5592045dfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57c37fe640 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559204206390 0x5592045dfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57c37fe640 1 -- 172.21.15.16:0/3187557471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592045e0640 con 0x5592041fe1b0 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57c37fe640 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045e4710 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f57b4004690 tx=0x7f57b4005050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b0fd3b48bea45bd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.733+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3187557471 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b400fe60 con 0x5592041fe1b0 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.734+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3187557471 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f57b40107d0 con 0x5592041fe1b0 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.734+0000 7f57d0560740 1 -- 172.21.15.16:0/3187557471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57b4004bf0 con 0x5592041fe1b0 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.734+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3187557471 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f57b4041050 con 0x5592041fe1b0 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.734+0000 7f57d0560740 10 client.27095 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.734+0000 7f57d0560740 1 -- 172.21.15.16:0/3187557471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x559204654ae0 con 0x5592041fe1b0 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.734+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3187557471 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57b4038070 con 0x5592041fe1b0 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.735+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3187557471 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f57b4018830 con 0x5592041fe1b0 2024-08-28T19:29:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.735+0000 7f57d0560740 10 client.27095 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.735+0000 7f57d0560740 10 client.27095 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.735+0000 7f57d0560740 20 client.27095 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.735+0000 7f57d0560740 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5592041bada0 0x559204657040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.735+0000 7f57d0560740 4 client.27095 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.735+0000 7f57d0560740 1 -- 172.21.15.16:0/3187557471 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f57b4004bf0 con 0x5592041bada0 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.735+0000 7f57baffd640 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5592041bada0 0x559204657040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57baffd640 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5592041bada0 0x559204657040 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8fd4726749f6677f server_cookie=57a62c595ebbfee0 in_seq=0 out_seq=0 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57a2ffd640 10 client.27095 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57a2ffd640 1 -- 172.21.15.16:0/3187557471 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f57b4004bf0 con 0x5592041bada0 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57a2ffd640 10 client.27095 handle_command_reply: tid=0 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 1 client.27095 shutdown 2024-08-28T19:29:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 1 -- 172.21.15.16:0/3187557471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 msgr2=0x5592045e4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045e4710 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f57b4004690 tx=0x7f57b4005050 comp rx=0 tx=0).stop 2024-08-28T19:29:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 1 -- 172.21.15.16:0/3187557471 shutdown_connections 2024-08-28T19:29:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5592041bada0 0x559204657040 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559204227070 0x5592045e0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559204206390 0x5592045dfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57a2ffd640 0 client.27095 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:29:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 1 --2- 172.21.15.16:0/3187557471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592041fe1b0 0x5592045e4710 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 1 -- 172.21.15.16:0/3187557471 >> 172.21.15.16:0/3187557471 conn(0x55920421b460 msgr2=0x5592045dd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 1 -- 172.21.15.16:0/3187557471 shutdown_connections 2024-08-28T19:29:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 1 -- 172.21.15.16:0/3187557471 wait complete. 2024-08-28T19:29:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.736+0000 7f57d0560740 20 client.27095 trim_cache size 0 max 16384 2024-08-28T19:29:07.743 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:29:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 -- 172.21.15.16:0/2339701533 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57b00b4cd0 msgr2=0x7f57b00b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57b00b4cd0 0x7f57b00b71e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f57bc005fd0 tx=0x7f57bc004a40 comp rx=0 tx=0).stop 2024-08-28T19:29:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 -- 172.21.15.16:0/2339701533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c4108140 msgr2=0x7f57c419b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c4108140 0x7f57c419b610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f57ac032fd0 tx=0x7f57ac004270 comp rx=0 tx=0).stop 2024-08-28T19:29:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 -- 172.21.15.16:0/2339701533 shutdown_connections 2024-08-28T19:29:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57c4108a90 0x7f57c419bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57b00b4cd0 0x7f57b00b71e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c4108140 0x7f57c419b610 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 --2- 172.21.15.16:0/2339701533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c4102800 0x7f57c419b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 -- 172.21.15.16:0/2339701533 >> 172.21.15.16:0/2339701533 conn(0x7f57c40794d0 msgr2=0x7f57c4104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 -- 172.21.15.16:0/2339701533 shutdown_connections 2024-08-28T19:29:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:07.737+0000 7f57c9feb640 1 -- 172.21.15.16:0/2339701533 wait complete. 2024-08-28T19:29:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:07.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:07 smithi028 ceph-mon[21769]: pgmap v584: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 225 op/s 2024-08-28T19:29:07.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 383515 4.69 MB/sec execute 287 sec latency 15.441 ms 2024-08-28T19:29:08.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.461+0000 7f0e9ef43640 1 -- 172.21.15.16:0/2440606343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e98108160 msgr2=0x7f0e98108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.461+0000 7f0e9ef43640 1 --2- 172.21.15.16:0/2440606343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e98108160 0x7f0e98108570 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009900 tx=0x7f0e8802a410 comp rx=0 tx=0).stop 2024-08-28T19:29:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.461+0000 7f0e9ef43640 1 -- 172.21.15.16:0/2440606343 shutdown_connections 2024-08-28T19:29:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.461+0000 7f0e9ef43640 1 --2- 172.21.15.16:0/2440606343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e98108ab0 0x7f0e9810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.461+0000 7f0e9ef43640 1 --2- 172.21.15.16:0/2440606343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e98108160 0x7f0e98108570 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.461+0000 7f0e9ef43640 1 --2- 172.21.15.16:0/2440606343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e98102820 0x7f0e98102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.461+0000 7f0e9ef43640 1 -- 172.21.15.16:0/2440606343 >> 172.21.15.16:0/2440606343 conn(0x7f0e98079500 msgr2=0x7f0e98079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.461+0000 7f0e9ef43640 1 -- 172.21.15.16:0/2440606343 shutdown_connections 2024-08-28T19:29:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.461+0000 7f0e9ef43640 1 -- 172.21.15.16:0/2440606343 wait complete. 2024-08-28T19:29:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.461+0000 7f0e9ef43640 1 Processor -- start 2024-08-28T19:29:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ef43640 1 -- start start 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ef43640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e98102820 0x7f0e9819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ef43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e98108160 0x7f0e9819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ef43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e98108ab0 0x7f0e9819fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ef43640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981126e0 con 0x7f0e98108160 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ef43640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e98112560 con 0x7f0e98108ab0 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ef43640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0e98112860 con 0x7f0e98102820 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e98108160 0x7f0e9819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ccdf640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e98102820 0x7f0e9819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ccdf640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e98102820 0x7f0e9819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34520/0 (socket says 172.21.15.16:34520) 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9d4e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e98108ab0 0x7f0e9819fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/1283558624 learned_addr learned my addr 172.21.15.16:0/1283558624 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e98108160 0x7f0e9819a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49698/0 (socket says 172.21.15.16:49698) 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/1283558624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e98108ab0 msgr2=0x7f0e9819fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e98108ab0 0x7f0e9819fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/1283558624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e98108160 msgr2=0x7f0e9819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e98108160 0x7f0e9819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/1283558624 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e981a0170 con 0x7f0e98102820 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.462+0000 7f0e8ffff640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e98108160 0x7f0e9819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.463+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e98102820 0x7f0e9819a200 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400e9f0 tx=0x7f0e9400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.463+0000 7f0e8dffb640 1 -- 172.21.15.16:0/1283558624 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9400ccc0 con 0x7f0e98102820 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.463+0000 7f0e8dffb640 1 -- 172.21.15.16:0/1283558624 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e940108e0 con 0x7f0e98102820 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.463+0000 7f0e9ef43640 1 -- 172.21.15.16:0/1283558624 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e981a0460 con 0x7f0e98102820 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.463+0000 7f0e8dffb640 1 -- 172.21.15.16:0/1283558624 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94018880 con 0x7f0e98102820 2024-08-28T19:29:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.463+0000 7f0e9ef43640 1 -- 172.21.15.16:0/1283558624 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e981a6bc0 con 0x7f0e98102820 2024-08-28T19:29:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.466+0000 7f0ea54b8740 1 Processor -- start 2024-08-28T19:29:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.466+0000 7f0ea54b8740 1 -- start start 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.466+0000 7f0e8dffb640 1 -- 172.21.15.16:0/1283558624 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0e94010430 con 0x7f0e98102820 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.466+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e68880a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.466+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e68885e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.466+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6888b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.466+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b6e67f8330 con 0x55b6e64b3980 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.466+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6e67e78c0 con 0x55b6e64dc840 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.466+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b6e67eb870 con 0x55b6e64bbb60 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e9ccdf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e68885e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e8ffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6888b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e8ffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6888b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34524/0 (socket says 172.21.15.16:34524) 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e8ffff640 1 -- 172.21.15.16:0/641765720 learned_addr learned my addr 172.21.15.16:0/641765720 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e9d4e0640 1 --2- 172.21.15.16:0/641765720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e68880a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e8ffff640 1 -- 172.21.15.16:0/641765720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 msgr2=0x55b6e68880a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:08.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e8ffff640 1 --2- 172.21.15.16:0/641765720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e68880a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e8ffff640 1 -- 172.21.15.16:0/641765720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 msgr2=0x55b6e68885e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:08.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e8ffff640 1 --2- 172.21.15.16:0/641765720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e68885e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e8ffff640 1 -- 172.21.15.16:0/641765720 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6e679fdf0 con 0x55b6e64bbb60 2024-08-28T19:29:08.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e8ffff640 1 --2- 172.21.15.16:0/641765720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6888b20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009900 tx=0x7f0e88005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a9c550c50b76f11 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:08.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e8dffb640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e800b4ca0 0x7f0e800b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e7effd640 1 -- 172.21.15.16:0/641765720 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88040070 con 0x55b6e64bbb60 2024-08-28T19:29:08.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e7effd640 1 -- 172.21.15.16:0/641765720 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e88031720 con 0x55b6e64bbb60 2024-08-28T19:29:08.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.467+0000 7f0e9d4e0640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e800b4ca0 0x7f0e800b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.468+0000 7f0e8dffb640 1 -- 172.21.15.16:0/1283558624 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f0e940e5db0 con 0x7f0e98102820 2024-08-28T19:29:08.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.468+0000 7f0e7effd640 1 -- 172.21.15.16:0/641765720 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88035040 con 0x55b6e64bbb60 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.468+0000 7f0ea54b8740 1 -- 172.21.15.16:0/641765720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 msgr2=0x55b6e6888b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.468+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/641765720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6888b20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009900 tx=0x7f0e88005e20 comp rx=0 tx=0).stop 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.468+0000 7f0e9d4e0640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e800b4ca0 0x7f0e800b71b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0e90005fd0 tx=0x7f0e90009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.468+0000 7f0ea54b8740 1 -- 172.21.15.16:0/641765720 shutdown_connections 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.468+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/641765720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6888b20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.468+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/641765720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e68885e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.468+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/641765720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e68880a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.468+0000 7f0ea54b8740 1 -- 172.21.15.16:0/641765720 >> 172.21.15.16:0/641765720 conn(0x55b6e64d0990 msgr2=0x55b6e6885f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.469+0000 7f0ea54b8740 1 -- 172.21.15.16:0/641765720 shutdown_connections 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.469+0000 7f0ea54b8740 1 -- 172.21.15.16:0/641765720 wait complete. 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.469+0000 7f0ea54b8740 1 -- start start 2024-08-28T19:29:08.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.469+0000 7f0ea54b8740 1 Processor -- start 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64b3980 0x55b6e6893760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64bbb60 0x55b6e6893ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e689fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b6e664a7a0 con 0x55b6e64bbb60 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6e676e240 con 0x55b6e64dc840 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b6e68968c0 con 0x55b6e64b3980 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0ea54b8740 10 client.0 _ll_register_callbacks cb 0x55b6e64f4090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e9d4e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e689fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64bbb60 0x55b6e6893ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e9ccdf640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64b3980 0x55b6e6893760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e9ccdf640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64b3980 0x55b6e6893760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34536/0 (socket says 172.21.15.16:34536) 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e9d4e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e689fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54992/0 (socket says 172.21.15.16:54992) 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/2267426991 learned_addr learned my addr 172.21.15.16:0/2267426991 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:08.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e9d4e0640 1 -- 172.21.15.16:0/2267426991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64b3980 msgr2=0x55b6e6893760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e9d4e0640 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64b3980 0x55b6e6893760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e9d4e0640 1 -- 172.21.15.16:0/2267426991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64bbb60 msgr2=0x55b6e6893ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e9d4e0640 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64bbb60 0x55b6e6893ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e9d4e0640 1 -- 172.21.15.16:0/2267426991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6e67a42b0 con 0x55b6e64dc840 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.470+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64b3980 0x55b6e6893760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.471+0000 7f0e9d4e0640 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e689fc00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0e90003f30 tx=0x7f0e90004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c13bf7b38b1eaa7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.471+0000 7f0e77fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.471+0000 7f0e77fff640 1 -- 172.21.15.16:0/2267426991 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9000f8d0 con 0x55b6e64dc840 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.471+0000 7f0ea54b8740 1 -- 172.21.15.16:0/2267426991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e88031720 con 0x55b6e64dc840 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.471+0000 7f0e77fff640 1 -- 172.21.15.16:0/2267426991 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f0e9000fa70 con 0x55b6e64dc840 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.471+0000 7f0e77fff640 1 -- 172.21.15.16:0/2267426991 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f0e9000fc70 con 0x55b6e64dc840 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.471+0000 7f0ea54b8740 10 client.27107 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:08.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.471+0000 7f0ea54b8740 1 -- 172.21.15.16:0/2267426991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55b6e6556b90 con 0x55b6e64dc840 2024-08-28T19:29:08.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.471+0000 7f0e77fff640 1 -- 172.21.15.16:0/2267426991 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e88031720 con 0x55b6e64dc840 2024-08-28T19:29:08.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.472+0000 7f0e77fff640 1 -- 172.21.15.16:0/2267426991 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f0e90027070 con 0x55b6e64dc840 2024-08-28T19:29:08.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.472+0000 7f0ea54b8740 10 client.27107 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:08.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.472+0000 7f0ea54b8740 10 client.27107 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:29:08.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.472+0000 7f0ea54b8740 20 client.27107 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:08.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.472+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6e649b1c0 0x55b6e68a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.472+0000 7f0ea54b8740 4 client.27107 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:08.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.472+0000 7f0ea54b8740 1 -- 172.21.15.16:0/2267426991 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6e67fa280 con 0x55b6e649b1c0 2024-08-28T19:29:08.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.472+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6e649b1c0 0x55b6e68a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.472+0000 7f0e77fff640 10 client.27107 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:08.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.472+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6e649b1c0 0x55b6e68a5aa0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38c30058318c209b server_cookie=b39f3ec3e9293618 in_seq=0 out_seq=0 2024-08-28T19:29:08.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0e77fff640 1 -- 172.21.15.16:0/2267426991 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b6e67fa280 con 0x55b6e649b1c0 2024-08-28T19:29:08.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0e77fff640 10 client.27107 handle_command_reply: tid=0 2024-08-28T19:29:08.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0ea54b8740 1 client.27107 shutdown 2024-08-28T19:29:08.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0ea54b8740 1 -- 172.21.15.16:0/2267426991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 msgr2=0x55b6e689fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:08.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e689fc00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0e90003f30 tx=0x7f0e90004ab0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0ea54b8740 1 -- 172.21.15.16:0/2267426991 shutdown_connections 2024-08-28T19:29:08.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e689fc00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64bbb60 0x55b6e6893ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6e649b1c0 0x55b6e68a5aa0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/2267426991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64b3980 0x55b6e6893760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0e77fff640 0 client.27107 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:08.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0ea54b8740 1 -- 172.21.15.16:0/2267426991 >> 172.21.15.16:0/2267426991 conn(0x55b6e64d0990 msgr2=0x55b6e688b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:08.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0ea54b8740 1 -- 172.21.15.16:0/2267426991 shutdown_connections 2024-08-28T19:29:08.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.473+0000 7f0ea54b8740 1 -- 172.21.15.16:0/2267426991 wait complete. 2024-08-28T19:29:08.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.474+0000 7f0ea54b8740 20 client.27107 trim_cache size 0 max 16384 2024-08-28T19:29:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.495+0000 7f0ea54b8740 1 Processor -- start 2024-08-28T19:29:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.495+0000 7f0ea54b8740 1 -- start start 2024-08-28T19:29:08.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e688b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6890950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e6890e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981aeb90 con 0x55b6e64b3980 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981b0820 con 0x55b6e64dc840 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981af2c0 con 0x55b6e64bbb60 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0e9ccdf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e688b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0e9ccdf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e688b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49718/0 (socket says 172.21.15.16:49718) 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/3863392782 learned_addr learned my addr 172.21.15.16:0/3863392782 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0e9d4e0640 1 --2- 172.21.15.16:0/3863392782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e6890e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0e8ffff640 1 --2- 172.21.15.16:0/3863392782 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6890950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/3863392782 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 msgr2=0x55b6e6890950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/3863392782 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6890950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/3863392782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 msgr2=0x55b6e6890e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/3863392782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e6890e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/3863392782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6e67b7d90 con 0x55b6e64b3980 2024-08-28T19:29:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.496+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/3863392782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e688b8b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0e940e3080 tx=0x7f0e940e9900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2b2f20db9d5a6a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.497+0000 7f0e7e7fc640 1 -- 172.21.15.16:0/3863392782 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e940ab7f0 con 0x55b6e64b3980 2024-08-28T19:29:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.497+0000 7f0e7e7fc640 1 -- 172.21.15.16:0/3863392782 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e94010720 con 0x55b6e64b3980 2024-08-28T19:29:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.497+0000 7f0e7e7fc640 1 -- 172.21.15.16:0/3863392782 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e940b53a0 con 0x55b6e64b3980 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.497+0000 7f0ea54b8740 1 -- 172.21.15.16:0/3863392782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 msgr2=0x55b6e688b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.497+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/3863392782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e688b8b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0e940e3080 tx=0x7f0e940e9900 comp rx=0 tx=0).stop 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.498+0000 7f0ea54b8740 1 -- 172.21.15.16:0/3863392782 shutdown_connections 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.498+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/3863392782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e6890e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.498+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/3863392782 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6890950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.498+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/3863392782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e688b8b0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.498+0000 7f0ea54b8740 1 -- 172.21.15.16:0/3863392782 >> 172.21.15.16:0/3863392782 conn(0x55b6e64d0990 msgr2=0x55b6e688b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.498+0000 7f0ea54b8740 1 -- 172.21.15.16:0/3863392782 shutdown_connections 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.498+0000 7f0ea54b8740 1 -- 172.21.15.16:0/3863392782 wait complete. 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.498+0000 7f0ea54b8740 1 -- start start 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.498+0000 7f0ea54b8740 1 Processor -- start 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e689bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6896b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0ea54b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e6897060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981b00c0 con 0x55b6e64b3980 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981af990 con 0x55b6e64dc840 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0ea54b8740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981acb90 con 0x55b6e64bbb60 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0ea54b8740 10 client.0 _ll_register_callbacks cb 0x7f0e90009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e9ccdf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e689bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e9ccdf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e689bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49724/0 (socket says 172.21.15.16:49724) 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/1020046001 learned_addr learned my addr 172.21.15.16:0/1020046001 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e8ffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6896b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e9d4e0640 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e6897060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/1020046001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 msgr2=0x55b6e6896b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6896b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/1020046001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 msgr2=0x55b6e6897060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e6897060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e9ccdf640 1 -- 172.21.15.16:0/1020046001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6e68975a0 con 0x55b6e64b3980 2024-08-28T19:29:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e8ffff640 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6896b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e9ccdf640 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e689bfd0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0e940abd80 tx=0x7f0e940aba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2b3d8a3b0076b7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.499+0000 7f0e77fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.500+0000 7f0e77fff640 1 -- 172.21.15.16:0/1020046001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e940ac080 con 0x55b6e64b3980 2024-08-28T19:29:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.500+0000 7f0ea54b8740 1 -- 172.21.15.16:0/1020046001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b6e67f5d20 con 0x55b6e64b3980 2024-08-28T19:29:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.500+0000 7f0e77fff640 1 -- 172.21.15.16:0/1020046001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f0e940a8310 con 0x55b6e64b3980 2024-08-28T19:29:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.500+0000 7f0e77fff640 1 -- 172.21.15.16:0/1020046001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f0e940abb90 con 0x55b6e64b3980 2024-08-28T19:29:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.500+0000 7f0ea54b8740 10 client.17580 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.500+0000 7f0ea54b8740 1 -- 172.21.15.16:0/1020046001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55b6e69110e0 con 0x55b6e64b3980 2024-08-28T19:29:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.500+0000 7f0e77fff640 1 -- 172.21.15.16:0/1020046001 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b6e67f5d20 con 0x55b6e64b3980 2024-08-28T19:29:08.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.500+0000 7f0e77fff640 1 -- 172.21.15.16:0/1020046001 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f0e940a7bd0 con 0x55b6e64b3980 2024-08-28T19:29:08.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.501+0000 7f0ea54b8740 10 client.17580 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:08.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.501+0000 7f0ea54b8740 10 client.17580 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:29:08.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.501+0000 7f0ea54b8740 20 client.17580 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:08.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.501+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6e6470570 0x55b6e6913980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:08.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.501+0000 7f0ea54b8740 4 client.17580 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.5, "await_for": 45.0}] 2024-08-28T19:29:08.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.501+0000 7f0ea54b8740 1 -- 172.21.15.16:0/1020046001 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.5, "await_for": 45.0}) -- 0x55b6e67f3460 con 0x55b6e6470570 2024-08-28T19:29:08.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.501+0000 7f0e8ffff640 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6e6470570 0x55b6e6913980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:08.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.502+0000 7f0e8ffff640 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6e6470570 0x55b6e6913980 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a940bff30eff5be server_cookie=d58b2c56f1eec1ae in_seq=0 out_seq=0 2024-08-28T19:29:08.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.502+0000 7f0e77fff640 10 client.17580 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0e77fff640 1 -- 172.21.15.16:0/1020046001 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55b6e67f3460 con 0x55b6e6470570 2024-08-28T19:29:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0e77fff640 10 client.17580 handle_command_reply: tid=0 2024-08-28T19:29:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0ea54b8740 1 client.17580 shutdown 2024-08-28T19:29:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0ea54b8740 1 -- 172.21.15.16:0/1020046001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 msgr2=0x55b6e689bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e689bfd0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0e940abd80 tx=0x7f0e940aba90 comp rx=0 tx=0).stop 2024-08-28T19:29:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0ea54b8740 1 -- 172.21.15.16:0/1020046001 shutdown_connections 2024-08-28T19:29:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e64dc840 0x55b6e6897060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b6e6470570 0x55b6e6913980 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6e64bbb60 0x55b6e6896b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0ea54b8740 1 --2- 172.21.15.16:0/1020046001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6e64b3980 0x55b6e689bfd0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0ea54b8740 1 -- 172.21.15.16:0/1020046001 >> 172.21.15.16:0/1020046001 conn(0x55b6e64d0990 msgr2=0x55b6e68890d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.557+0000 7f0e77fff640 0 client.17580 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0ea54b8740 1 -- 172.21.15.16:0/1020046001 shutdown_connections 2024-08-28T19:29:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0ea54b8740 1 -- 172.21.15.16:0/1020046001 wait complete. 2024-08-28T19:29:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0ea54b8740 20 client.17580 trim_cache size 0 max 16384 2024-08-28T19:29:08.560 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 49, 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 22, 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "55b9eba3": { 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 22, 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:29:08.561 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.5, 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:08.562 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:29:08.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0e9ef43640 1 -- 172.21.15.16:0/1283558624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e800b4ca0 msgr2=0x7f0e800b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:08.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0e9ef43640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e800b4ca0 0x7f0e800b71b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0e90005fd0 tx=0x7f0e90009500 comp rx=0 tx=0).stop 2024-08-28T19:29:08.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0e9ef43640 1 -- 172.21.15.16:0/1283558624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e98102820 msgr2=0x7f0e9819a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:08.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0e9ef43640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e98102820 0x7f0e9819a200 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400e9f0 tx=0x7f0e9400edf0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0e9ef43640 1 -- 172.21.15.16:0/1283558624 shutdown_connections 2024-08-28T19:29:08.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0e9ef43640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e800b4ca0 0x7f0e800b71b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0e9ef43640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e98108ab0 0x7f0e9819fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0e9ef43640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e98108160 0x7f0e9819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0e9ef43640 1 --2- 172.21.15.16:0/1283558624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e98102820 0x7f0e9819a200 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.558+0000 7f0e9ef43640 1 -- 172.21.15.16:0/1283558624 >> 172.21.15.16:0/1283558624 conn(0x7f0e98079500 msgr2=0x7f0e98101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.559+0000 7f0e9ef43640 1 -- 172.21.15.16:0/1283558624 shutdown_connections 2024-08-28T19:29:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:08.559+0000 7f0e9ef43640 1 -- 172.21.15.16:0/1283558624 wait complete. 2024-08-28T19:29:08.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.67 MB/sec execute 288 sec latency 461.127 ms 2024-08-28T19:29:09.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:29:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:09 smithi115 ceph-mon[22184]: pgmap v585: 129 pgs: 129 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-08-28T19:29:09.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:09 smithi016 ceph-mon[18556]: pgmap v585: 129 pgs: 129 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-08-28T19:29:09.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:09 smithi028 ceph-mon[21769]: pgmap v585: 129 pgs: 129 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-08-28T19:29:09.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.66 MB/sec execute 289 sec latency 1461.157 ms 2024-08-28T19:29:10.091 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '55b9eba3', quiesce took 6.7 seconds. Will release after: 12.8 2024-08-28T19:29:10.161 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:10.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.672+0000 7ffa42488640 1 -- 172.21.15.16:0/2837761897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa3c108160 msgr2=0x7ffa3c108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:10.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.672+0000 7ffa42488640 1 --2- 172.21.15.16:0/2837761897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa3c108160 0x7ffa3c108570 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009900 tx=0x7ffa3002a410 comp rx=0 tx=0).stop 2024-08-28T19:29:10.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.672+0000 7ffa42488640 1 -- 172.21.15.16:0/2837761897 shutdown_connections 2024-08-28T19:29:10.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.672+0000 7ffa42488640 1 --2- 172.21.15.16:0/2837761897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa3c108ab0 0x7ffa3c10f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.672+0000 7ffa42488640 1 --2- 172.21.15.16:0/2837761897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa3c108160 0x7ffa3c108570 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.672+0000 7ffa42488640 1 --2- 172.21.15.16:0/2837761897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa3c102820 0x7ffa3c102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.672+0000 7ffa42488640 1 -- 172.21.15.16:0/2837761897 >> 172.21.15.16:0/2837761897 conn(0x7ffa3c0794e0 msgr2=0x7ffa3c0798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.672+0000 7ffa42488640 1 -- 172.21.15.16:0/2837761897 shutdown_connections 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.672+0000 7ffa42488640 1 -- 172.21.15.16:0/2837761897 wait complete. 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa42488640 1 Processor -- start 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa42488640 1 -- start start 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa42488640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa3c102820 0x7ffa3c19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa42488640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa3c108160 0x7ffa3c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa42488640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa3c108ab0 0x7ffa3c1a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa42488640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c112700 con 0x7ffa3c108160 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa42488640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c112580 con 0x7ffa3c108ab0 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa42488640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c112880 con 0x7ffa3c102820 2024-08-28T19:29:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa3b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa3c108160 0x7ffa3c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa3b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa3c108160 0x7ffa3c19ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49746/0 (socket says 172.21.15.16:49746) 2024-08-28T19:29:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa3b7fe640 1 -- 172.21.15.16:0/2065539457 learned_addr learned my addr 172.21.15.16:0/2065539457 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa3bfff640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa3c102820 0x7ffa3c19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.673+0000 7ffa40a25640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa3c108ab0 0x7ffa3c1a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa3b7fe640 1 -- 172.21.15.16:0/2065539457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa3c102820 msgr2=0x7ffa3c19c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa3b7fe640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa3c102820 0x7ffa3c19c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa3b7fe640 1 -- 172.21.15.16:0/2065539457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa3c108ab0 msgr2=0x7ffa3c1a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa3b7fe640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa3c108ab0 0x7ffa3c1a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa3b7fe640 1 -- 172.21.15.16:0/2065539457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c1a23f0 con 0x7ffa3c108160 2024-08-28T19:29:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa40a25640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa3c108ab0 0x7ffa3c1a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:10.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa3b7fe640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa3c108160 0x7ffa3c19ca20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ffa30032fd0 tx=0x7ffa30004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:10.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa397fa640 1 -- 172.21.15.16:0/2065539457 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3003f030 con 0x7ffa3c108160 2024-08-28T19:29:10.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa42488640 1 -- 172.21.15.16:0/2065539457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa3c1a2680 con 0x7ffa3c108160 2024-08-28T19:29:10.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa397fa640 1 -- 172.21.15.16:0/2065539457 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa30004490 con 0x7ffa3c108160 2024-08-28T19:29:10.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa397fa640 1 -- 172.21.15.16:0/2065539457 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa30035070 con 0x7ffa3c108160 2024-08-28T19:29:10.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.674+0000 7ffa42488640 1 -- 172.21.15.16:0/2065539457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa3c1a8e40 con 0x7ffa3c108160 2024-08-28T19:29:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa489fd740 1 Processor -- start 2024-08-28T19:29:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa397fa640 1 -- 172.21.15.16:0/2065539457 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ffa30042030 con 0x7ffa3c108160 2024-08-28T19:29:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa489fd740 1 -- start start 2024-08-28T19:29:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d659070 0x55986d9fe350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d6301b0 0x55986d9fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d638390 0x55986d9fedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa489fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55986d975680 con 0x55986d6301b0 2024-08-28T19:29:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa489fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55986d964c10 con 0x55986d659070 2024-08-28T19:29:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa489fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55986d968bc0 con 0x55986d638390 2024-08-28T19:29:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d6301b0 0x55986d9fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d6301b0 0x55986d9fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49754/0 (socket says 172.21.15.16:49754) 2024-08-28T19:29:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.677+0000 7ffa3bfff640 1 -- 172.21.15.16:0/1328802625 learned_addr learned my addr 172.21.15.16:0/1328802625 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa40a25640 1 --2- 172.21.15.16:0/1328802625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d659070 0x55986d9fe350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa3b7fe640 1 --2- 172.21.15.16:0/1328802625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d638390 0x55986d9fedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa3bfff640 1 -- 172.21.15.16:0/1328802625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d638390 msgr2=0x55986d9fedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa3bfff640 1 --2- 172.21.15.16:0/1328802625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d638390 0x55986d9fedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa3bfff640 1 -- 172.21.15.16:0/1328802625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d659070 msgr2=0x55986d9fe350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa3bfff640 1 --2- 172.21.15.16:0/1328802625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d659070 0x55986d9fe350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa3bfff640 1 -- 172.21.15.16:0/1328802625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55986d6c4650 con 0x55986d6301b0 2024-08-28T19:29:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa397fa640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa240b4ca0 0x7ffa240b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa40a25640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa240b4ca0 0x7ffa240b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa3bfff640 1 --2- 172.21.15.16:0/1328802625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d6301b0 0x55986d9fe890 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00ac10 tx=0x7ffa2c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18149edf8342a575 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa397fa640 1 -- 172.21.15.16:0/2065539457 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7ffa30105f20 con 0x7ffa3c108160 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.678+0000 7ffa227fc640 1 -- 172.21.15.16:0/1328802625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c01d070 con 0x55986d6301b0 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.679+0000 7ffa227fc640 1 -- 172.21.15.16:0/1328802625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa2c018af0 con 0x55986d6301b0 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.679+0000 7ffa227fc640 1 -- 172.21.15.16:0/1328802625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c010a90 con 0x55986d6301b0 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.679+0000 7ffa489fd740 1 -- 172.21.15.16:0/1328802625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d6301b0 msgr2=0x55986d9fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.679+0000 7ffa489fd740 1 --2- 172.21.15.16:0/1328802625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d6301b0 0x55986d9fe890 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00ac10 tx=0x7ffa2c00f790 comp rx=0 tx=0).stop 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.679+0000 7ffa40a25640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa240b4ca0 0x7ffa240b71b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005fd0 tx=0x7ffa340098a0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.679+0000 7ffa489fd740 1 -- 172.21.15.16:0/1328802625 shutdown_connections 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.679+0000 7ffa489fd740 1 --2- 172.21.15.16:0/1328802625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d638390 0x55986d9fedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.679+0000 7ffa489fd740 1 --2- 172.21.15.16:0/1328802625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d6301b0 0x55986d9fe890 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.679+0000 7ffa489fd740 1 --2- 172.21.15.16:0/1328802625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d659070 0x55986d9fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.679+0000 7ffa489fd740 1 -- 172.21.15.16:0/1328802625 >> 172.21.15.16:0/1328802625 conn(0x55986d64d460 msgr2=0x55986d9fc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.680+0000 7ffa489fd740 1 -- 172.21.15.16:0/1328802625 shutdown_connections 2024-08-28T19:29:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.680+0000 7ffa489fd740 1 -- 172.21.15.16:0/1328802625 wait complete. 2024-08-28T19:29:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.680+0000 7ffa489fd740 1 -- start start 2024-08-28T19:29:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.680+0000 7ffa489fd740 1 Processor -- start 2024-08-28T19:29:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d6301b0 0x55986da09ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d638390 0x55986da0a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da15e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa489fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55986d7c7700 con 0x55986d659070 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa489fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55986d8eb750 con 0x55986d638390 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa489fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55986da0cb30 con 0x55986d6301b0 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa489fd740 10 client.0 _ll_register_callbacks cb 0x55986d91ae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa40a25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da15e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa40a25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da15e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49766/0 (socket says 172.21.15.16:49766) 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa40a25640 1 -- 172.21.15.16:0/2186688978 learned_addr learned my addr 172.21.15.16:0/2186688978 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa3bfff640 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d6301b0 0x55986da09ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa3b7fe640 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d638390 0x55986da0a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa40a25640 1 -- 172.21.15.16:0/2186688978 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d6301b0 msgr2=0x55986da09ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa40a25640 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d6301b0 0x55986da09ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa40a25640 1 -- 172.21.15.16:0/2186688978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d638390 msgr2=0x55986da0a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa40a25640 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d638390 0x55986da0a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa40a25640 1 -- 172.21.15.16:0/2186688978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55986d924900 con 0x55986d659070 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa40a25640 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da15e50 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ffa34010a20 tx=0x7ffa34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebaf299ac2f601a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.681+0000 7ffa1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/2186688978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3401ed40 con 0x55986d659070 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/2186688978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7ffa3401f420 con 0x55986d659070 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa489fd740 1 -- 172.21.15.16:0/2186688978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa2c018af0 con 0x55986d659070 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/2186688978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ffa34032050 con 0x55986d659070 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa489fd740 10 client.17592 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa489fd740 1 -- 172.21.15.16:0/2186688978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55986da18930 con 0x55986d659070 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/2186688978 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa2c018af0 con 0x55986d659070 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/2186688978 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7ffa3401c670 con 0x55986d659070 2024-08-28T19:29:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa489fd740 10 client.17592 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa489fd740 10 client.17592 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:29:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa489fd740 20 client.17592 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.682+0000 7ffa489fd740 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55986d6179f0 0x55986da1bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.683+0000 7ffa489fd740 4 client.17592 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.683+0000 7ffa489fd740 1 -- 172.21.15.16:0/2186688978 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55986d9775d0 con 0x55986d6179f0 2024-08-28T19:29:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.683+0000 7ffa3bfff640 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55986d6179f0 0x55986da1bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.683+0000 7ffa1b7fe640 10 client.17592 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:29:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.683+0000 7ffa3bfff640 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55986d6179f0 0x55986da1bf30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ddecabda425df0ca server_cookie=6b17f3193653c652 in_seq=0 out_seq=0 2024-08-28T19:29:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/2186688978 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55986d9775d0 con 0x55986d6179f0 2024-08-28T19:29:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa1b7fe640 10 client.17592 handle_command_reply: tid=0 2024-08-28T19:29:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa489fd740 1 client.17592 shutdown 2024-08-28T19:29:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa489fd740 1 -- 172.21.15.16:0/2186688978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 msgr2=0x55986da15e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa489fd740 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da15e50 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ffa34010a20 tx=0x7ffa34004a40 comp rx=0 tx=0).stop 2024-08-28T19:29:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa489fd740 1 -- 172.21.15.16:0/2186688978 shutdown_connections 2024-08-28T19:29:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa489fd740 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55986d6179f0 0x55986da1bf30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa489fd740 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da15e50 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa489fd740 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d638390 0x55986da0a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa489fd740 1 --2- 172.21.15.16:0/2186688978 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d6301b0 0x55986da09ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa489fd740 1 -- 172.21.15.16:0/2186688978 >> 172.21.15.16:0/2186688978 conn(0x55986d64d460 msgr2=0x55986da014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa1b7fe640 0 client.17592 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:29:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa489fd740 1 -- 172.21.15.16:0/2186688978 shutdown_connections 2024-08-28T19:29:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.684+0000 7ffa489fd740 1 -- 172.21.15.16:0/2186688978 wait complete. 2024-08-28T19:29:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.685+0000 7ffa489fd740 20 client.17592 trim_cache size 0 max 16384 2024-08-28T19:29:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.698+0000 7ffa489fd740 1 Processor -- start 2024-08-28T19:29:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.698+0000 7ffa489fd740 1 -- start start 2024-08-28T19:29:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.698+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d6301b0 0x55986d9fc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.698+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d638390 0x55986da15d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.698+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d659070 0x55986da06cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.698+0000 7ffa489fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c1b2fa0 con 0x55986d638390 2024-08-28T19:29:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.698+0000 7ffa489fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c1b4c30 con 0x55986d659070 2024-08-28T19:29:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.698+0000 7ffa489fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c1b36d0 con 0x55986d6301b0 2024-08-28T19:29:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.698+0000 7ffa3b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d638390 0x55986da15d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.698+0000 7ffa3b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d638390 0x55986da15d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49778/0 (socket says 172.21.15.16:49778) 2024-08-28T19:29:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.698+0000 7ffa3b7fe640 1 -- 172.21.15.16:0/1063438108 learned_addr learned my addr 172.21.15.16:0/1063438108 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa40a25640 1 --2- 172.21.15.16:0/1063438108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d659070 0x55986da06cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa3bfff640 1 --2- 172.21.15.16:0/1063438108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d6301b0 0x55986d9fc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa3b7fe640 1 -- 172.21.15.16:0/1063438108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d6301b0 msgr2=0x55986d9fc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa3b7fe640 1 --2- 172.21.15.16:0/1063438108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d6301b0 0x55986d9fc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa3b7fe640 1 -- 172.21.15.16:0/1063438108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d659070 msgr2=0x55986da06cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa3b7fe640 1 --2- 172.21.15.16:0/1063438108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d659070 0x55986da06cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa3b7fe640 1 -- 172.21.15.16:0/1063438108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55986da00640 con 0x55986d638390 2024-08-28T19:29:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa40a25640 1 --2- 172.21.15.16:0/1063438108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d659070 0x55986da06cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:29:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa3b7fe640 1 --2- 172.21.15.16:0/1063438108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d638390 0x55986da15d50 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ffa300c9400 tx=0x7ffa3003b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b65c79360cb8e0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa21ffb640 1 -- 172.21.15.16:0/1063438108 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa30049040 con 0x55986d638390 2024-08-28T19:29:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa21ffb640 1 -- 172.21.15.16:0/1063438108 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa300cbca0 con 0x55986d638390 2024-08-28T19:29:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.699+0000 7ffa21ffb640 1 -- 172.21.15.16:0/1063438108 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa30103470 con 0x55986d638390 2024-08-28T19:29:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.700+0000 7ffa489fd740 1 -- 172.21.15.16:0/1063438108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d638390 msgr2=0x55986da15d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.700+0000 7ffa489fd740 1 --2- 172.21.15.16:0/1063438108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d638390 0x55986da15d50 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ffa300c9400 tx=0x7ffa3003b720 comp rx=0 tx=0).stop 2024-08-28T19:29:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.700+0000 7ffa489fd740 1 -- 172.21.15.16:0/1063438108 shutdown_connections 2024-08-28T19:29:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.700+0000 7ffa489fd740 1 --2- 172.21.15.16:0/1063438108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d659070 0x55986da06cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.700+0000 7ffa489fd740 1 --2- 172.21.15.16:0/1063438108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d638390 0x55986da15d50 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.700+0000 7ffa489fd740 1 --2- 172.21.15.16:0/1063438108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d6301b0 0x55986d9fc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.700+0000 7ffa489fd740 1 -- 172.21.15.16:0/1063438108 >> 172.21.15.16:0/1063438108 conn(0x55986d64d460 msgr2=0x55986d9fbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.700+0000 7ffa489fd740 1 -- 172.21.15.16:0/1063438108 shutdown_connections 2024-08-28T19:29:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.700+0000 7ffa489fd740 1 -- 172.21.15.16:0/1063438108 wait complete. 2024-08-28T19:29:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.700+0000 7ffa489fd740 1 -- start start 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa489fd740 1 Processor -- start 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d6301b0 0x55986da0df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d638390 0x55986da0e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa489fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da0e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa489fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c1b44d0 con 0x55986d659070 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa489fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c1b3da0 con 0x55986d6301b0 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa489fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c1b0fa0 con 0x55986d638390 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa489fd740 10 client.0 _ll_register_callbacks cb 0x7ffa34009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa40a25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da0e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa40a25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da0e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49794/0 (socket says 172.21.15.16:49794) 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa40a25640 1 -- 172.21.15.16:0/3690678072 learned_addr learned my addr 172.21.15.16:0/3690678072 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa3b7fe640 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d638390 0x55986da0e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa3bfff640 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d6301b0 0x55986da0df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa40a25640 1 -- 172.21.15.16:0/3690678072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d638390 msgr2=0x55986da0e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa40a25640 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d638390 0x55986da0e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa40a25640 1 -- 172.21.15.16:0/3690678072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d6301b0 msgr2=0x55986da0df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa40a25640 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d6301b0 0x55986da0df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.701+0000 7ffa40a25640 1 -- 172.21.15.16:0/3690678072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55986da85a70 con 0x55986d659070 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa40a25640 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da0e980 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ffa34009ed0 tx=0x7ffa34010600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47f7f622b0d62c9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/3690678072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3403a020 con 0x55986d659070 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/3690678072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7ffa34004ab0 con 0x55986d659070 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa489fd740 1 -- 172.21.15.16:0/3690678072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa300cbca0 con 0x55986d659070 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/3690678072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ffa34045050 con 0x55986d659070 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa489fd740 10 client.17598 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa489fd740 1 -- 172.21.15.16:0/3690678072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55986da86c60 con 0x55986d659070 2024-08-28T19:29:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/3690678072 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa300cbca0 con 0x55986d659070 2024-08-28T19:29:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/3690678072 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7ffa340051d0 con 0x55986d659070 2024-08-28T19:29:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa489fd740 10 client.17598 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.702+0000 7ffa489fd740 10 client.17598 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:29:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.703+0000 7ffa489fd740 20 client.17598 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.703+0000 7ffa489fd740 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55986d5ecda0 0x55986da894a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.703+0000 7ffa489fd740 4 client.17598 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:29:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.703+0000 7ffa489fd740 1 -- 172.21.15.16:0/3690678072 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55986d973070 con 0x55986d5ecda0 2024-08-28T19:29:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.703+0000 7ffa3bfff640 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55986d5ecda0 0x55986da894a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.703+0000 7ffa1b7fe640 10 client.17598 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.703+0000 7ffa3bfff640 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55986d5ecda0 0x55986da894a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=358c4f66c7316300 server_cookie=8ea1a60772b23284 in_seq=0 out_seq=0 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.703+0000 7ffa1b7fe640 1 -- 172.21.15.16:0/3690678072 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55986d973070 con 0x55986d5ecda0 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa1b7fe640 10 client.17598 handle_command_reply: tid=0 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 1 client.17598 shutdown 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 1 -- 172.21.15.16:0/3690678072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 msgr2=0x55986da0e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da0e980 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ffa34009ed0 tx=0x7ffa34010600 comp rx=0 tx=0).stop 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 1 -- 172.21.15.16:0/3690678072 shutdown_connections 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55986d5ecda0 0x55986da894a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55986d659070 0x55986da0e980 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa1b7fe640 0 client.17598 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55986d638390 0x55986da0e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 1 --2- 172.21.15.16:0/3690678072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55986d6301b0 0x55986da0df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 1 -- 172.21.15.16:0/3690678072 >> 172.21.15.16:0/3690678072 conn(0x55986d64d460 msgr2=0x55986d9ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 1 -- 172.21.15.16:0/3690678072 shutdown_connections 2024-08-28T19:29:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 1 -- 172.21.15.16:0/3690678072 wait complete. 2024-08-28T19:29:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa489fd740 20 client.17598 trim_cache size 0 max 16384 2024-08-28T19:29:10.711 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:29:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa42488640 1 -- 172.21.15.16:0/2065539457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa240b4ca0 msgr2=0x7ffa240b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa42488640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa240b4ca0 0x7ffa240b71b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005fd0 tx=0x7ffa340098a0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa42488640 1 -- 172.21.15.16:0/2065539457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa3c108160 msgr2=0x7ffa3c19ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.704+0000 7ffa42488640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa3c108160 0x7ffa3c19ca20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ffa30032fd0 tx=0x7ffa30004270 comp rx=0 tx=0).stop 2024-08-28T19:29:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.705+0000 7ffa42488640 1 -- 172.21.15.16:0/2065539457 shutdown_connections 2024-08-28T19:29:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.705+0000 7ffa42488640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa240b4ca0 0x7ffa240b71b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.705+0000 7ffa42488640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa3c108ab0 0x7ffa3c1a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.705+0000 7ffa42488640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa3c108160 0x7ffa3c19ca20 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.705+0000 7ffa42488640 1 --2- 172.21.15.16:0/2065539457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa3c102820 0x7ffa3c19c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.705+0000 7ffa42488640 1 -- 172.21.15.16:0/2065539457 >> 172.21.15.16:0/2065539457 conn(0x7ffa3c0794e0 msgr2=0x7ffa3c06b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.705+0000 7ffa42488640 1 -- 172.21.15.16:0/2065539457 shutdown_connections 2024-08-28T19:29:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:10.705+0000 7ffa42488640 1 -- 172.21.15.16:0/2065539457 wait complete. 2024-08-28T19:29:10.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.64 MB/sec execute 290 sec latency 2461.211 ms 2024-08-28T19:29:11.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:29:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:11 smithi115 ceph-mon[22184]: pgmap v586: 129 pgs: 129 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-08-28T19:29:11.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:11 smithi016 ceph-mon[18556]: pgmap v586: 129 pgs: 129 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-08-28T19:29:11.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:11 smithi028 ceph-mon[21769]: pgmap v586: 129 pgs: 129 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-08-28T19:29:11.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.62 MB/sec execute 291 sec latency 3461.415 ms 2024-08-28T19:29:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.120+0000 7ff353b93640 1 -- 172.21.15.16:0/489405820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff34c108140 msgr2=0x7ff34c108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.120+0000 7ff353b93640 1 --2- 172.21.15.16:0/489405820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff34c108140 0x7ff34c108550 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff340009900 tx=0x7ff34002a410 comp rx=0 tx=0).stop 2024-08-28T19:29:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.120+0000 7ff353b93640 1 -- 172.21.15.16:0/489405820 shutdown_connections 2024-08-28T19:29:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.120+0000 7ff353b93640 1 --2- 172.21.15.16:0/489405820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c108a90 0x7ff34c10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.120+0000 7ff353b93640 1 --2- 172.21.15.16:0/489405820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff34c108140 0x7ff34c108550 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.120+0000 7ff353b93640 1 --2- 172.21.15.16:0/489405820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff34c102800 0x7ff34c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.120+0000 7ff353b93640 1 -- 172.21.15.16:0/489405820 >> 172.21.15.16:0/489405820 conn(0x7ff34c0794d0 msgr2=0x7ff34c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.120+0000 7ff353b93640 1 -- 172.21.15.16:0/489405820 shutdown_connections 2024-08-28T19:29:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.120+0000 7ff353b93640 1 -- 172.21.15.16:0/489405820 wait complete. 2024-08-28T19:29:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.121+0000 7ff353b93640 1 Processor -- start 2024-08-28T19:29:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.121+0000 7ff353b93640 1 -- start start 2024-08-28T19:29:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.121+0000 7ff353b93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff34c102800 0x7ff34c19a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.121+0000 7ff353b93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c108140 0x7ff34c19a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.121+0000 7ff353b93640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff34c108a90 0x7ff34c19fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.121+0000 7ff353b93640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c113820 con 0x7ff34c102800 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.121+0000 7ff353b93640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c1136a0 con 0x7ff34c108140 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff353b93640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c1139a0 con 0x7ff34c108a90 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff35192f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff34c102800 0x7ff34c19a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff35192f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff34c102800 0x7ff34c19a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49802/0 (socket says 172.21.15.16:49802) 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff35192f640 1 -- 172.21.15.16:0/2181815864 learned_addr learned my addr 172.21.15.16:0/2181815864 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff35112e640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c108140 0x7ff34c19a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff352130640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff34c108a90 0x7ff34c19fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff35192f640 1 -- 172.21.15.16:0/2181815864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff34c108a90 msgr2=0x7ff34c19fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff35192f640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff34c108a90 0x7ff34c19fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff35192f640 1 -- 172.21.15.16:0/2181815864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c108140 msgr2=0x7ff34c19a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff35192f640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c108140 0x7ff34c19a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff35192f640 1 -- 172.21.15.16:0/2181815864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34c1a02a0 con 0x7ff34c102800 2024-08-28T19:29:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff352130640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff34c108a90 0x7ff34c19fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:29:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff35192f640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff34c102800 0x7ff34c19a330 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00ea80 tx=0x7ff33c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff34affd640 1 -- 172.21.15.16:0/2181815864 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c00ce40 con 0x7ff34c102800 2024-08-28T19:29:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff34affd640 1 -- 172.21.15.16:0/2181815864 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff33c005020 con 0x7ff34c102800 2024-08-28T19:29:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff353b93640 1 -- 172.21.15.16:0/2181815864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff34c1a0590 con 0x7ff34c102800 2024-08-28T19:29:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff34affd640 1 -- 172.21.15.16:0/2181815864 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c0116d0 con 0x7ff34c102800 2024-08-28T19:29:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.122+0000 7ff353b93640 1 -- 172.21.15.16:0/2181815864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff34c1a6cf0 con 0x7ff34c102800 2024-08-28T19:29:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.125+0000 7ff35a108740 1 Processor -- start 2024-08-28T19:29:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.125+0000 7ff34affd640 1 -- 172.21.15.16:0/2181815864 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff33c0054d0 con 0x7ff34c102800 2024-08-28T19:29:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.125+0000 7ff35a108740 1 -- start start 2024-08-28T19:29:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.125+0000 7ff35a108740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd5191310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.125+0000 7ff35a108740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd5191850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.125+0000 7ff35a108740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dcb390 0x560bd5191d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.125+0000 7ff35a108740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560bd5108680 con 0x560bd4dec070 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.125+0000 7ff35a108740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560bd50f7c10 con 0x560bd4dc31b0 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.125+0000 7ff35a108740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560bd50fbbc0 con 0x560bd4dcb390 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff352130640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd5191310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff352130640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd5191310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49816/0 (socket says 172.21.15.16:49816) 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff352130640 1 -- 172.21.15.16:0/2756984325 learned_addr learned my addr 172.21.15.16:0/2756984325 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff35192f640 1 --2- 172.21.15.16:0/2756984325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd5191850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff35112e640 1 --2- 172.21.15.16:0/2756984325 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dcb390 0x560bd5191d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff352130640 1 -- 172.21.15.16:0/2756984325 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dcb390 msgr2=0x560bd5191d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff352130640 1 --2- 172.21.15.16:0/2756984325 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dcb390 0x560bd5191d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff352130640 1 -- 172.21.15.16:0/2756984325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 msgr2=0x560bd5191850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff352130640 1 --2- 172.21.15.16:0/2756984325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd5191850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff352130640 1 -- 172.21.15.16:0/2756984325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd4e57650 con 0x560bd4dec070 2024-08-28T19:29:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff34affd640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3340b4ca0 0x7ff3340b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff35192f640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3340b4ca0 0x7ff3340b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff352130640 1 --2- 172.21.15.16:0/2756984325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd5191310 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff344009b50 tx=0x7ff34400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec45c8d801f65aee server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.126+0000 7ff34affd640 1 -- 172.21.15.16:0/2181815864 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7ff33c0e5cf0 con 0x7ff34c102800 2024-08-28T19:29:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.127+0000 7ff333fff640 1 -- 172.21.15.16:0/2756984325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff344016070 con 0x560bd4dec070 2024-08-28T19:29:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.127+0000 7ff333fff640 1 -- 172.21.15.16:0/2756984325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff344009e40 con 0x560bd4dec070 2024-08-28T19:29:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.127+0000 7ff333fff640 1 -- 172.21.15.16:0/2756984325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff344010790 con 0x560bd4dec070 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.127+0000 7ff35192f640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3340b4ca0 0x7ff3340b71b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0e30d0 tx=0x7ff33c0e9840 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.127+0000 7ff35a108740 1 -- 172.21.15.16:0/2756984325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 msgr2=0x560bd5191310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.128+0000 7ff35a108740 1 --2- 172.21.15.16:0/2756984325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd5191310 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff344009b50 tx=0x7ff34400e6c0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.128+0000 7ff35a108740 1 -- 172.21.15.16:0/2756984325 shutdown_connections 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.128+0000 7ff35a108740 1 --2- 172.21.15.16:0/2756984325 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dcb390 0x560bd5191d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.128+0000 7ff35a108740 1 --2- 172.21.15.16:0/2756984325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd5191850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.128+0000 7ff35a108740 1 --2- 172.21.15.16:0/2756984325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd5191310 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.128+0000 7ff35a108740 1 -- 172.21.15.16:0/2756984325 >> 172.21.15.16:0/2756984325 conn(0x560bd4de0460 msgr2=0x560bd518f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.128+0000 7ff35a108740 1 -- 172.21.15.16:0/2756984325 shutdown_connections 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.128+0000 7ff35a108740 1 -- 172.21.15.16:0/2756984325 wait complete. 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.128+0000 7ff35a108740 1 -- start start 2024-08-28T19:29:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.128+0000 7ff35a108740 1 Processor -- start 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35a108740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd519cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35a108740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd519d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35a108740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dec070 0x560bd51a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35a108740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560bd4f5a700 con 0x560bd4dcb390 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35a108740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560bd507e750 con 0x560bd4dc31b0 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35a108740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560bd519faf0 con 0x560bd4dec070 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35a108740 10 client.0 _ll_register_callbacks cb 0x560bd50ade10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35112e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd519d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35112e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd519d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49822/0 (socket says 172.21.15.16:49822) 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35112e640 1 -- 172.21.15.16:0/1909207343 learned_addr learned my addr 172.21.15.16:0/1909207343 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff352130640 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd519cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff352130640 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dec070 0x560bd51a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35112e640 1 -- 172.21.15.16:0/1909207343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dec070 msgr2=0x560bd51a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35112e640 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dec070 0x560bd51a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35112e640 1 -- 172.21.15.16:0/1909207343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 msgr2=0x560bd519cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35112e640 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd519cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35112e640 1 -- 172.21.15.16:0/1909207343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd50b7900 con 0x560bd4dcb390 2024-08-28T19:29:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff352130640 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dec070 0x560bd51a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff35112e640 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd519d000 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7ff34c06a840 tx=0x7ff3400316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5437f18bf8c7d374 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.129+0000 7ff3317fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff3317fa640 1 -- 172.21.15.16:0/1909207343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff340035040 con 0x560bd4dcb390 2024-08-28T19:29:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff35a108740 1 -- 172.21.15.16:0/1909207343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560bd510a5d0 con 0x560bd4dcb390 2024-08-28T19:29:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff3317fa640 1 -- 172.21.15.16:0/1909207343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7ff340004240 con 0x560bd4dcb390 2024-08-28T19:29:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff3317fa640 1 -- 172.21.15.16:0/1909207343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff340004440 con 0x560bd4dcb390 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff35a108740 10 client.17610 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff35a108740 1 -- 172.21.15.16:0/1909207343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x560bd51ab8f0 con 0x560bd4dcb390 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff3317fa640 1 -- 172.21.15.16:0/1909207343 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560bd510a5d0 con 0x560bd4dcb390 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff3317fa640 1 -- 172.21.15.16:0/1909207343 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7ff340004ad0 con 0x560bd4dcb390 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff35a108740 10 client.17610 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff35a108740 10 client.17610 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff35a108740 20 client.17610 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff35a108740 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560bd4daa9f0 0x560bd51aed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff35a108740 4 client.17610 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.130+0000 7ff35a108740 1 -- 172.21.15.16:0/1909207343 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bd5106070 con 0x560bd4daa9f0 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.131+0000 7ff352130640 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560bd4daa9f0 0x560bd51aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.131+0000 7ff352130640 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560bd4daa9f0 0x560bd51aed80 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a67e77155780e30a server_cookie=1be53897dc388234 in_seq=0 out_seq=0 2024-08-28T19:29:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.131+0000 7ff3317fa640 10 client.17610 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff3317fa640 1 -- 172.21.15.16:0/1909207343 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560bd5106070 con 0x560bd4daa9f0 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff3317fa640 10 client.17610 handle_command_reply: tid=0 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 1 client.17610 shutdown 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 1 -- 172.21.15.16:0/1909207343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 msgr2=0x560bd519d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd519d000 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7ff34c06a840 tx=0x7ff3400316b0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 1 -- 172.21.15.16:0/1909207343 shutdown_connections 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560bd4daa9f0 0x560bd51aed80 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dec070 0x560bd51a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd519d000 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff3317fa640 0 client.17610 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 1 --2- 172.21.15.16:0/1909207343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd519cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 1 -- 172.21.15.16:0/1909207343 >> 172.21.15.16:0/1909207343 conn(0x560bd4de0460 msgr2=0x560bd51944f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 1 -- 172.21.15.16:0/1909207343 shutdown_connections 2024-08-28T19:29:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 1 -- 172.21.15.16:0/1909207343 wait complete. 2024-08-28T19:29:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.132+0000 7ff35a108740 20 client.17610 trim_cache size 0 max 16384 2024-08-28T19:29:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35a108740 1 Processor -- start 2024-08-28T19:29:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35a108740 1 -- start start 2024-08-28T19:29:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35a108740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dc31b0 0x560bd518f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35a108740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd51a8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35a108740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dec070 0x560bd5194410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35a108740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c1b0e60 con 0x560bd4dcb390 2024-08-28T19:29:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35a108740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c1b2af0 con 0x560bd4dec070 2024-08-28T19:29:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35a108740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c1b1590 con 0x560bd4dc31b0 2024-08-28T19:29:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35112e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd51a8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35112e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd51a8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49826/0 (socket says 172.21.15.16:49826) 2024-08-28T19:29:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35112e640 1 -- 172.21.15.16:0/4270453976 learned_addr learned my addr 172.21.15.16:0/4270453976 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff352130640 1 --2- 172.21.15.16:0/4270453976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dec070 0x560bd5194410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff352130640 1 --2- 172.21.15.16:0/4270453976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dc31b0 0x560bd518f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35112e640 1 -- 172.21.15.16:0/4270453976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dc31b0 msgr2=0x560bd518f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35112e640 1 --2- 172.21.15.16:0/4270453976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dc31b0 0x560bd518f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35112e640 1 -- 172.21.15.16:0/4270453976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dec070 msgr2=0x560bd5194410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35112e640 1 --2- 172.21.15.16:0/4270453976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dec070 0x560bd5194410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff35112e640 1 -- 172.21.15.16:0/4270453976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd519dd80 con 0x560bd4dcb390 2024-08-28T19:29:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.146+0000 7ff352130640 1 --2- 172.21.15.16:0/4270453976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dec070 0x560bd5194410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:12.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.147+0000 7ff35112e640 1 --2- 172.21.15.16:0/4270453976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd51a8ce0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ff34002a930 tx=0x7ff34002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32c625401282d42 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:12.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.147+0000 7ff3337fe640 1 -- 172.21.15.16:0/4270453976 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff340035070 con 0x560bd4dcb390 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.147+0000 7ff3337fe640 1 -- 172.21.15.16:0/4270453976 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3400037c0 con 0x560bd4dcb390 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.147+0000 7ff3337fe640 1 -- 172.21.15.16:0/4270453976 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3400053b0 con 0x560bd4dcb390 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.148+0000 7ff35a108740 1 -- 172.21.15.16:0/4270453976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 msgr2=0x560bd51a8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.148+0000 7ff35a108740 1 --2- 172.21.15.16:0/4270453976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd51a8ce0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ff34002a930 tx=0x7ff34002a410 comp rx=0 tx=0).stop 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.148+0000 7ff35a108740 1 -- 172.21.15.16:0/4270453976 shutdown_connections 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.148+0000 7ff35a108740 1 --2- 172.21.15.16:0/4270453976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dec070 0x560bd5194410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.148+0000 7ff35a108740 1 --2- 172.21.15.16:0/4270453976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dcb390 0x560bd51a8ce0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.148+0000 7ff35a108740 1 --2- 172.21.15.16:0/4270453976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dc31b0 0x560bd518f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.148+0000 7ff35a108740 1 -- 172.21.15.16:0/4270453976 >> 172.21.15.16:0/4270453976 conn(0x560bd4de0460 msgr2=0x560bd518ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.148+0000 7ff35a108740 1 -- 172.21.15.16:0/4270453976 shutdown_connections 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.148+0000 7ff35a108740 1 -- 172.21.15.16:0/4270453976 wait complete. 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff35a108740 1 -- start start 2024-08-28T19:29:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff35a108740 1 Processor -- start 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff35a108740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd51a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff35a108740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dcb390 0x560bd51a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff35a108740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd51a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff35a108740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c1b2390 con 0x560bd4dec070 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff35a108740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c1b1c60 con 0x560bd4dc31b0 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff35a108740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c1aee60 con 0x560bd4dcb390 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff35a108740 10 client.0 _ll_register_callbacks cb 0x7ff340005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff352130640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd51a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff352130640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd51a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49828/0 (socket says 172.21.15.16:49828) 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.149+0000 7ff352130640 1 -- 172.21.15.16:0/3349995767 learned_addr learned my addr 172.21.15.16:0/3349995767 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff352130640 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd51a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff35112e640 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dcb390 0x560bd51a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff352130640 1 -- 172.21.15.16:0/3349995767 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dcb390 msgr2=0x560bd51a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff352130640 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dcb390 0x560bd51a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff352130640 1 -- 172.21.15.16:0/3349995767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 msgr2=0x560bd51a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff352130640 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd51a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff352130640 1 -- 172.21.15.16:0/3349995767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd52188c0 con 0x560bd4dec070 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff35112e640 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dcb390 0x560bd51a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff352130640 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd51a1910 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7ff34400ecc0 tx=0x7ff34401d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440def11cd5e5bf6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff3317fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff3317fa640 1 -- 172.21.15.16:0/3349995767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff344039060 con 0x560bd4dec070 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff3317fa640 1 -- 172.21.15.16:0/3349995767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7ff34402bf90 con 0x560bd4dec070 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff35a108740 1 -- 172.21.15.16:0/3349995767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3400037c0 con 0x560bd4dec070 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff3317fa640 1 -- 172.21.15.16:0/3349995767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff344016070 con 0x560bd4dec070 2024-08-28T19:29:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff35a108740 10 client.17616 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.150+0000 7ff35a108740 1 -- 172.21.15.16:0/3349995767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x560bd5219b40 con 0x560bd4dec070 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.151+0000 7ff3317fa640 1 -- 172.21.15.16:0/3349995767 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3400037c0 con 0x560bd4dec070 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.151+0000 7ff3317fa640 1 -- 172.21.15.16:0/3349995767 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7ff34400f040 con 0x560bd4dec070 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.151+0000 7ff35a108740 10 client.17616 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.151+0000 7ff35a108740 10 client.17616 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.151+0000 7ff35a108740 20 client.17616 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.151+0000 7ff35a108740 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560bd4d7fda0 0x560bd521c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.151+0000 7ff35a108740 4 client.17616 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.151+0000 7ff35a108740 1 -- 172.21.15.16:0/3349995767 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560bd51037b0 con 0x560bd4d7fda0 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.151+0000 7ff35112e640 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560bd4d7fda0 0x560bd521c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35112e640 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560bd4d7fda0 0x560bd521c1c0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fce1c26ca0128b server_cookie=216ae1185a14fe83 in_seq=0 out_seq=0 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff3317fa640 10 client.17616 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff3317fa640 1 -- 172.21.15.16:0/3349995767 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560bd51037b0 con 0x560bd4d7fda0 2024-08-28T19:29:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff3317fa640 10 client.17616 handle_command_reply: tid=0 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 1 client.17616 shutdown 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 1 -- 172.21.15.16:0/3349995767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 msgr2=0x560bd51a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd51a1910 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7ff34400ecc0 tx=0x7ff34401d040 comp rx=0 tx=0).stop 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 1 -- 172.21.15.16:0/3349995767 shutdown_connections 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bd4dec070 0x560bd51a1910 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560bd4d7fda0 0x560bd521c1c0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560bd4dcb390 0x560bd51a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 1 --2- 172.21.15.16:0/3349995767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bd4dc31b0 0x560bd51a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff3317fa640 0 client.17616 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 1 -- 172.21.15.16:0/3349995767 >> 172.21.15.16:0/3349995767 conn(0x560bd4de0460 msgr2=0x560bd519af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 1 -- 172.21.15.16:0/3349995767 shutdown_connections 2024-08-28T19:29:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 1 -- 172.21.15.16:0/3349995767 wait complete. 2024-08-28T19:29:12.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.152+0000 7ff35a108740 20 client.17616 trim_cache size 0 max 16384 2024-08-28T19:29:12.159 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:29:12.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 -- 172.21.15.16:0/2181815864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3340b4ca0 msgr2=0x7ff3340b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3340b4ca0 0x7ff3340b71b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0e30d0 tx=0x7ff33c0e9840 comp rx=0 tx=0).stop 2024-08-28T19:29:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 -- 172.21.15.16:0/2181815864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff34c102800 msgr2=0x7ff34c19a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff34c102800 0x7ff34c19a330 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00ea80 tx=0x7ff33c00ef40 comp rx=0 tx=0).stop 2024-08-28T19:29:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 -- 172.21.15.16:0/2181815864 shutdown_connections 2024-08-28T19:29:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff34c108a90 0x7ff34c19fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3340b4ca0 0x7ff3340b71b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c108140 0x7ff34c19a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 --2- 172.21.15.16:0/2181815864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff34c102800 0x7ff34c19a330 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 -- 172.21.15.16:0/2181815864 >> 172.21.15.16:0/2181815864 conn(0x7ff34c0794d0 msgr2=0x7ff34c104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 -- 172.21.15.16:0/2181815864 shutdown_connections 2024-08-28T19:29:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:12.153+0000 7ff353b93640 1 -- 172.21.15.16:0/2181815864 wait complete. 2024-08-28T19:29:12.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.61 MB/sec execute 292 sec latency 4461.472 ms 2024-08-28T19:29:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:13 smithi115 ceph-mon[22184]: pgmap v587: 129 pgs: 129 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-08-28T19:29:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:13 smithi016 ceph-mon[18556]: pgmap v587: 129 pgs: 129 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-08-28T19:29:13.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:13 smithi028 ceph-mon[21769]: pgmap v587: 129 pgs: 129 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-08-28T19:29:13.841 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:29:13.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:29:13.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.59 MB/sec execute 293 sec latency 5461.522 ms 2024-08-28T19:29:14.114 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.595+0000 7f13efccf640 1 -- 172.21.15.16:0/3928497137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e8105010 msgr2=0x7f13e810d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.595+0000 7f13efccf640 1 --2- 172.21.15.16:0/3928497137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e8105010 0x7f13e810d650 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009900 tx=0x7f13d802a410 comp rx=0 tx=0).stop 2024-08-28T19:29:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.595+0000 7f13efccf640 1 -- 172.21.15.16:0/3928497137 shutdown_connections 2024-08-28T19:29:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.595+0000 7f13efccf640 1 --2- 172.21.15.16:0/3928497137 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13e810db90 0x7f13e81100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.595+0000 7f13efccf640 1 --2- 172.21.15.16:0/3928497137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e8105010 0x7f13e810d650 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.595+0000 7f13efccf640 1 --2- 172.21.15.16:0/3928497137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e81045e0 0x7f13e81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.595+0000 7f13efccf640 1 -- 172.21.15.16:0/3928497137 >> 172.21.15.16:0/3928497137 conn(0x7f13e80794d0 msgr2=0x7f13e80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.595+0000 7f13efccf640 1 -- 172.21.15.16:0/3928497137 shutdown_connections 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.595+0000 7f13efccf640 1 -- 172.21.15.16:0/3928497137 wait complete. 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.596+0000 7f13efccf640 1 Processor -- start 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.596+0000 7f13efccf640 1 -- start start 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13efccf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e81045e0 0x7f13e819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13efccf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e8105010 0x7f13e819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13efccf640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13e810db90 0x7f13e819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13efccf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13e81137d0 con 0x7f13e8105010 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13efccf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13e8113650 con 0x7f13e81045e0 2024-08-28T19:29:14.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13efccf640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f13e8113950 con 0x7f13e810db90 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13ed26a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e8105010 0x7f13e819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13ed26a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e8105010 0x7f13e819a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56150/0 (socket says 172.21.15.16:56150) 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13ed26a640 1 -- 172.21.15.16:0/1676240301 learned_addr learned my addr 172.21.15.16:0/1676240301 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13eda6b640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e81045e0 0x7f13e819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13ee26c640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13e810db90 0x7f13e819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13ed26a640 1 -- 172.21.15.16:0/1676240301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13e810db90 msgr2=0x7f13e819fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13ed26a640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13e810db90 0x7f13e819fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13ed26a640 1 -- 172.21.15.16:0/1676240301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e81045e0 msgr2=0x7f13e819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13ed26a640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e81045e0 0x7f13e819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13ed26a640 1 -- 172.21.15.16:0/1676240301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e81a0210 con 0x7f13e8105010 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.597+0000 7f13ed26a640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e8105010 0x7f13e819a840 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f13d802a930 tx=0x7f13d8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.598+0000 7f13deffd640 1 -- 172.21.15.16:0/1676240301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d803f030 con 0x7f13e8105010 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.598+0000 7f13efccf640 1 -- 172.21.15.16:0/1676240301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e81a04a0 con 0x7f13e8105010 2024-08-28T19:29:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.598+0000 7f13deffd640 1 -- 172.21.15.16:0/1676240301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13d8003aa0 con 0x7f13e8105010 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.598+0000 7f13deffd640 1 -- 172.21.15.16:0/1676240301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8035070 con 0x7f13e8105010 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.598+0000 7f13efccf640 1 -- 172.21.15.16:0/1676240301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e81a6c60 con 0x7f13e8105010 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.600+0000 7f13f6244740 1 Processor -- start 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.600+0000 7f13f6244740 1 -- start start 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13f6244740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bdd350 0x55bb59f83780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13f6244740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f83cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13f6244740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bbc670 0x55bb59f84200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13f6244740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bb59ef95f0 con 0x55bb59bbc670 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13f6244740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb59ee8b80 con 0x55bb59bb4490 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13f6244740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55bb59eecb30 con 0x55bb59bdd350 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13eda6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f83cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13ee26c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bdd350 0x55bb59f83780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13eda6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f83cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:52924/0 (socket says 172.21.15.16:52924) 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13eda6b640 1 -- 172.21.15.16:0/1367340894 learned_addr learned my addr 172.21.15.16:0/1367340894 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13eda6b640 1 -- 172.21.15.16:0/1367340894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bdd350 msgr2=0x55bb59f83780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13eda6b640 1 --2- 172.21.15.16:0/1367340894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bdd350 0x55bb59f83780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13eda6b640 1 -- 172.21.15.16:0/1367340894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bbc670 msgr2=0x55bb59f84200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13eda6b640 1 --2- 172.21.15.16:0/1367340894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bbc670 0x55bb59f84200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13eda6b640 1 -- 172.21.15.16:0/1367340894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb59c48930 con 0x55bb59bb4490 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.601+0000 7f13eda6b640 1 --2- 172.21.15.16:0/1367340894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f83cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f13e400b7d0 tx=0x7f13e4004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=522ffefc515c4564 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.602+0000 7f13deffd640 1 -- 172.21.15.16:0/1676240301 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f13d8042030 con 0x7f13e8105010 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.602+0000 7f13d3fff640 1 -- 172.21.15.16:0/1367340894 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e4003dd0 con 0x55bb59bb4490 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.602+0000 7f13d3fff640 1 -- 172.21.15.16:0/1367340894 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13e4015070 con 0x55bb59bb4490 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.602+0000 7f13d3fff640 1 -- 172.21.15.16:0/1367340894 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e40116a0 con 0x55bb59bb4490 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.602+0000 7f13deffd640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13d40b4ca0 0x7f13d40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13ee26c640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13d40b4ca0 0x7f13d40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13deffd640 1 -- 172.21.15.16:0/1676240301 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f13d8105a50 con 0x7f13e8105010 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13f6244740 1 -- 172.21.15.16:0/1367340894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 msgr2=0x55bb59f83cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13f6244740 1 --2- 172.21.15.16:0/1367340894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f83cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f13e400b7d0 tx=0x7f13e4004610 comp rx=0 tx=0).stop 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13f6244740 1 -- 172.21.15.16:0/1367340894 shutdown_connections 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13f6244740 1 --2- 172.21.15.16:0/1367340894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bbc670 0x55bb59f84200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13f6244740 1 --2- 172.21.15.16:0/1367340894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f83cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13f6244740 1 --2- 172.21.15.16:0/1367340894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bdd350 0x55bb59f83780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13f6244740 1 -- 172.21.15.16:0/1367340894 >> 172.21.15.16:0/1367340894 conn(0x55bb59bd1740 msgr2=0x55bb59f81670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13f6244740 1 -- 172.21.15.16:0/1367340894 shutdown_connections 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13f6244740 1 -- 172.21.15.16:0/1367340894 wait complete. 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13ee26c640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13d40b4ca0 0x7f13d40b71b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f13e0005fd0 tx=0x7f13e0009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.603+0000 7f13f6244740 1 -- start start 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13f6244740 1 Processor -- start 2024-08-28T19:29:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13f6244740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f93360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13f6244740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bbc670 0x55bb59f938a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13f6244740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13f6244740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bb59d4b890 con 0x55bb59bdd350 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13f6244740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb59e6f790 con 0x55bb59bb4490 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13f6244740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55bb59f96260 con 0x55bb59bbc670 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13f6244740 10 client.0 _ll_register_callbacks cb 0x55bb59bf4ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13ee26c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13ee26c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56160/0 (socket says 172.21.15.16:56160) 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13ee26c640 1 -- 172.21.15.16:0/488945193 learned_addr learned my addr 172.21.15.16:0/488945193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13eda6b640 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f93360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.604+0000 7f13ed26a640 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bbc670 0x55bb59f938a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13ee26c640 1 -- 172.21.15.16:0/488945193 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bbc670 msgr2=0x55bb59f938a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13ee26c640 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bbc670 0x55bb59f938a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13ee26c640 1 -- 172.21.15.16:0/488945193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 msgr2=0x55bb59f93360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13ee26c640 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f93360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13ee26c640 1 -- 172.21.15.16:0/488945193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb59ea6f30 con 0x55bb59bdd350 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13ed26a640 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bbc670 0x55bb59f938a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13ee26c640 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f5c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f13e0009fd0 tx=0x7f13e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5317e4f53f136d0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13d17fa640 1 -- 172.21.15.16:0/488945193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e0004d20 con 0x55bb59bdd350 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13d17fa640 1 -- 172.21.15.16:0/488945193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f13e0005020 con 0x55bb59bdd350 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13f6244740 1 -- 172.21.15.16:0/488945193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13e4015070 con 0x55bb59bdd350 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13d17fa640 1 -- 172.21.15.16:0/488945193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f13e0005260 con 0x55bb59bdd350 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13f6244740 10 client.17625 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.605+0000 7f13f6244740 1 -- 172.21.15.16:0/488945193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55bb59fa2130 con 0x55bb59bdd350 2024-08-28T19:29:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.606+0000 7f13d17fa640 1 -- 172.21.15.16:0/488945193 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13e4015070 con 0x55bb59bdd350 2024-08-28T19:29:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.606+0000 7f13d17fa640 1 -- 172.21.15.16:0/488945193 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f13e000f040 con 0x55bb59bdd350 2024-08-28T19:29:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.606+0000 7f13f6244740 10 client.17625 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.606+0000 7f13f6244740 10 client.17625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.606+0000 7f13f6244740 20 client.17625 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.606+0000 7f13f6244740 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bb59b9bcd0 0x55bb59fa5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.606+0000 7f13f6244740 4 client.17625 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.606+0000 7f13f6244740 1 -- 172.21.15.16:0/488945193 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb59efb540 con 0x55bb59b9bcd0 2024-08-28T19:29:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.606+0000 7f13eda6b640 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bb59b9bcd0 0x55bb59fa5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.607+0000 7f13eda6b640 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bb59b9bcd0 0x55bb59fa5750 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b699d6508125ed43 server_cookie=c4f4b6eba85abd17 in_seq=0 out_seq=0 2024-08-28T19:29:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.607+0000 7f13d17fa640 10 client.17625 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:14.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.607+0000 7f13d17fa640 1 -- 172.21.15.16:0/488945193 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55bb59efb540 con 0x55bb59b9bcd0 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.607+0000 7f13d17fa640 10 client.17625 handle_command_reply: tid=0 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 1 client.17625 shutdown 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 1 -- 172.21.15.16:0/488945193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 msgr2=0x55bb59f9f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f5c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f13e0009fd0 tx=0x7f13e0004a40 comp rx=0 tx=0).stop 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 1 -- 172.21.15.16:0/488945193 shutdown_connections 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f5c0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bb59b9bcd0 0x55bb59fa5750 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bbc670 0x55bb59f938a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 1 --2- 172.21.15.16:0/488945193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f93360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13d17fa640 0 client.17625 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 1 -- 172.21.15.16:0/488945193 >> 172.21.15.16:0/488945193 conn(0x55bb59bd1740 msgr2=0x55bb59f7fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 1 -- 172.21.15.16:0/488945193 shutdown_connections 2024-08-28T19:29:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 1 -- 172.21.15.16:0/488945193 wait complete. 2024-08-28T19:29:14.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.608+0000 7f13f6244740 20 client.17625 trim_cache size 0 max 16384 2024-08-28T19:29:14.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.626+0000 7f13f6244740 1 Processor -- start 2024-08-28T19:29:14.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.626+0000 7f13f6244740 1 -- start start 2024-08-28T19:29:14.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.626+0000 7f13f6244740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f8eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.626+0000 7f13f6244740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bbc670 0x55bb59f88f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.626+0000 7f13f6244740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.626+0000 7f13f6244740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13e81aec90 con 0x55bb59bdd350 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.626+0000 7f13f6244740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13e81b0920 con 0x55bb59bb4490 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.626+0000 7f13f6244740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f13e81af3c0 con 0x55bb59bbc670 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13ee26c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13ee26c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56164/0 (socket says 172.21.15.16:56164) 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13ee26c640 1 -- 172.21.15.16:0/578979894 learned_addr learned my addr 172.21.15.16:0/578979894 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13eda6b640 1 --2- 172.21.15.16:0/578979894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f8eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13ed26a640 1 --2- 172.21.15.16:0/578979894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bbc670 0x55bb59f88f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13ee26c640 1 -- 172.21.15.16:0/578979894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bbc670 msgr2=0x55bb59f88f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13ee26c640 1 --2- 172.21.15.16:0/578979894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bbc670 0x55bb59f88f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13ee26c640 1 -- 172.21.15.16:0/578979894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 msgr2=0x55bb59f8eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13ee26c640 1 --2- 172.21.15.16:0/578979894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f8eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13ee26c640 1 -- 172.21.15.16:0/578979894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb59f922c0 con 0x55bb59bdd350 2024-08-28T19:29:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13ee26c640 1 --2- 172.21.15.16:0/578979894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f4c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f13e0020a90 tx=0x7f13e0011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3a7bece0537fbc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13d37fe640 1 -- 172.21.15.16:0/578979894 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e003b070 con 0x55bb59bdd350 2024-08-28T19:29:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13d37fe640 1 -- 172.21.15.16:0/578979894 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13e0020d10 con 0x55bb59bdd350 2024-08-28T19:29:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.627+0000 7f13d37fe640 1 -- 172.21.15.16:0/578979894 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e0005000 con 0x55bb59bdd350 2024-08-28T19:29:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.628+0000 7f13f6244740 1 -- 172.21.15.16:0/578979894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 msgr2=0x55bb59f9f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.628+0000 7f13f6244740 1 --2- 172.21.15.16:0/578979894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f4c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f13e0020a90 tx=0x7f13e0011a90 comp rx=0 tx=0).stop 2024-08-28T19:29:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.628+0000 7f13f6244740 1 -- 172.21.15.16:0/578979894 shutdown_connections 2024-08-28T19:29:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.628+0000 7f13f6244740 1 --2- 172.21.15.16:0/578979894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bdd350 0x55bb59f9f4c0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.628+0000 7f13f6244740 1 --2- 172.21.15.16:0/578979894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bbc670 0x55bb59f88f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.628+0000 7f13f6244740 1 --2- 172.21.15.16:0/578979894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bb4490 0x55bb59f8eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.628+0000 7f13f6244740 1 -- 172.21.15.16:0/578979894 >> 172.21.15.16:0/578979894 conn(0x55bb59bd1740 msgr2=0x55bb59f88860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.628+0000 7f13f6244740 1 -- 172.21.15.16:0/578979894 shutdown_connections 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.628+0000 7f13f6244740 1 -- 172.21.15.16:0/578979894 wait complete. 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.629+0000 7f13f6244740 1 -- start start 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.629+0000 7f13f6244740 1 Processor -- start 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.629+0000 7f13f6244740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bb4490 0x55bb59f98030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.629+0000 7f13f6244740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bbc670 0x55bb59f985d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.629+0000 7f13f6244740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bdd350 0x55bb5a009290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.629+0000 7f13f6244740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13e81b01c0 con 0x55bb59bb4490 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.629+0000 7f13f6244740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13e81afa90 con 0x55bb59bbc670 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.629+0000 7f13f6244740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f13e81acc90 con 0x55bb59bdd350 2024-08-28T19:29:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13f6244740 10 client.0 _ll_register_callbacks cb 0x7f13e0009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13eda6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bb4490 0x55bb59f98030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13eda6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bb4490 0x55bb59f98030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56166/0 (socket says 172.21.15.16:56166) 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13eda6b640 1 -- 172.21.15.16:0/1657517388 learned_addr learned my addr 172.21.15.16:0/1657517388 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13ed26a640 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bbc670 0x55bb59f985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13ee26c640 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bdd350 0x55bb5a009290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13eda6b640 1 -- 172.21.15.16:0/1657517388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bdd350 msgr2=0x55bb5a009290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13eda6b640 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bdd350 0x55bb5a009290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13eda6b640 1 -- 172.21.15.16:0/1657517388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bbc670 msgr2=0x55bb59f985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13eda6b640 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bbc670 0x55bb59f985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13eda6b640 1 -- 172.21.15.16:0/1657517388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb5a009900 con 0x55bb59bb4490 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13ed26a640 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bbc670 0x55bb59f985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13eda6b640 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bb4490 0x55bb59f98030 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x55bb59f82400 tx=0x7f13e4003d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9dc6311406a0785 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13d17fa640 1 -- 172.21.15.16:0/1657517388 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e401a670 con 0x55bb59bb4490 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13f6244740 1 -- 172.21.15.16:0/1657517388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13e0020d10 con 0x55bb59bb4490 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.630+0000 7f13d17fa640 1 -- 172.21.15.16:0/1657517388 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9578+0+0 (secure 0 0 0) 0x7f13e401a810 con 0x55bb59bb4490 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13d17fa640 1 -- 172.21.15.16:0/1657517388 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f13e401aa10 con 0x55bb59bb4490 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13f6244740 10 client.17631 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13f6244740 1 -- 172.21.15.16:0/1657517388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55bb5a010d30 con 0x55bb59bb4490 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13d17fa640 1 -- 172.21.15.16:0/1657517388 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13e0020d10 con 0x55bb59bb4490 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13d17fa640 1 -- 172.21.15.16:0/1657517388 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f13e4011c90 con 0x55bb59bb4490 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13f6244740 10 client.17631 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13f6244740 10 client.17631 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13f6244740 20 client.17631 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13f6244740 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bb59b71080 0x55bb5a0133d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13f6244740 4 client.17631 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:29:14.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13f6244740 1 -- 172.21.15.16:0/1657517388 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bb59ef6fe0 con 0x55bb59b71080 2024-08-28T19:29:14.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.631+0000 7f13ed26a640 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bb59b71080 0x55bb5a0133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:14.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.632+0000 7f13ed26a640 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bb59b71080 0x55bb5a0133d0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70801ef714e3140a server_cookie=e91650df4bd93eaf in_seq=0 out_seq=0 2024-08-28T19:29:14.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.632+0000 7f13d17fa640 10 client.17631 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.633+0000 7f13d17fa640 1 -- 172.21.15.16:0/1657517388 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bb59ef6fe0 con 0x55bb59b71080 2024-08-28T19:29:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.633+0000 7f13d17fa640 10 client.17631 handle_command_reply: tid=0 2024-08-28T19:29:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.633+0000 7f13f6244740 1 client.17631 shutdown 2024-08-28T19:29:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.634+0000 7f13f6244740 1 -- 172.21.15.16:0/1657517388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bb4490 msgr2=0x55bb59f98030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.634+0000 7f13f6244740 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bb4490 0x55bb59f98030 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x55bb59f82400 tx=0x7f13e4003d30 comp rx=0 tx=0).stop 2024-08-28T19:29:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.634+0000 7f13f6244740 1 -- 172.21.15.16:0/1657517388 shutdown_connections 2024-08-28T19:29:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.634+0000 7f13f6244740 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bb59b71080 0x55bb5a0133d0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.634+0000 7f13f6244740 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bb59bdd350 0x55bb5a009290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.634+0000 7f13f6244740 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb59bbc670 0x55bb59f985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.634+0000 7f13f6244740 1 --2- 172.21.15.16:0/1657517388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb59bb4490 0x55bb59f98030 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.634+0000 7f13f6244740 1 -- 172.21.15.16:0/1657517388 >> 172.21.15.16:0/1657517388 conn(0x55bb59bd1740 msgr2=0x55bb59f94810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:14.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.634+0000 7f13d17fa640 0 client.17631 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.639+0000 7f13f6244740 1 -- 172.21.15.16:0/1657517388 shutdown_connections 2024-08-28T19:29:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.639+0000 7f13f6244740 1 -- 172.21.15.16:0/1657517388 wait complete. 2024-08-28T19:29:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.639+0000 7f13f6244740 20 client.17631 trim_cache size 0 max 16384 2024-08-28T19:29:14.641 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:29:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:29:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "d863be82-a773-4056-b493-bb9d7d839e7e", 2024-08-28T19:29:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:29:14.641 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:29:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.640+0000 7f13efccf640 1 -- 172.21.15.16:0/1676240301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13d40b4ca0 msgr2=0x7f13d40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.640+0000 7f13efccf640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13d40b4ca0 0x7f13d40b71b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f13e0005fd0 tx=0x7f13e0009600 comp rx=0 tx=0).stop 2024-08-28T19:29:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.640+0000 7f13efccf640 1 -- 172.21.15.16:0/1676240301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e8105010 msgr2=0x7f13e819a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.641+0000 7f13efccf640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e8105010 0x7f13e819a840 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f13d802a930 tx=0x7f13d8004270 comp rx=0 tx=0).stop 2024-08-28T19:29:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.641+0000 7f13efccf640 1 -- 172.21.15.16:0/1676240301 shutdown_connections 2024-08-28T19:29:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.641+0000 7f13efccf640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13e810db90 0x7f13e819fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.641+0000 7f13efccf640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e8105010 0x7f13e819a840 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.641+0000 7f13efccf640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13d40b4ca0 0x7f13d40b71b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.641+0000 7f13efccf640 1 --2- 172.21.15.16:0/1676240301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e81045e0 0x7f13e819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.641+0000 7f13efccf640 1 -- 172.21.15.16:0/1676240301 >> 172.21.15.16:0/1676240301 conn(0x7f13e80794d0 msgr2=0x7f13e80ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.642+0000 7f13efccf640 1 -- 172.21.15.16:0/1676240301 shutdown_connections 2024-08-28T19:29:14.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:14.642+0000 7f13efccf640 1 -- 172.21.15.16:0/1676240301 wait complete. 2024-08-28T19:29:14.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.58 MB/sec execute 294 sec latency 6461.555 ms 2024-08-28T19:29:15.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:15 smithi016 ceph-mon[18556]: pgmap v588: 129 pgs: 129 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-08-28T19:29:15.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:15 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:29:15.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:15 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:29:15.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:15 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:15.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:15 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:29:15.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:15 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:15.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:15 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:15.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:15 smithi016 ceph-mon[18556]: osdmap e88: 12 total, 12 up, 12 in 2024-08-28T19:29:15.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:29:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:15 smithi115 ceph-mon[22184]: pgmap v588: 129 pgs: 129 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-08-28T19:29:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:15 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:29:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:15 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:29:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:15 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:15 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:29:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:15 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:15 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:15 smithi115 ceph-mon[22184]: osdmap e88: 12 total, 12 up, 12 in 2024-08-28T19:29:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:15 smithi028 ceph-mon[21769]: pgmap v588: 129 pgs: 129 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-08-28T19:29:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:15 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:29:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:15 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:29:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:15 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:15 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:29:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:15 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:15 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:15.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:15 smithi028 ceph-mon[21769]: osdmap e88: 12 total, 12 up, 12 in 2024-08-28T19:29:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:15.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.56 MB/sec execute 295 sec latency 7461.617 ms 2024-08-28T19:29:16.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.395+0000 7f0afa098640 1 -- 172.21.15.16:0/3662777329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af410fee0 msgr2=0x7f0af4112300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:16.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.395+0000 7f0afa098640 1 --2- 172.21.15.16:0/3662777329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af410fee0 0x7f0af4112300 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009ca0 tx=0x7f0aec02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.395+0000 7f0afa098640 1 -- 172.21.15.16:0/3662777329 shutdown_connections 2024-08-28T19:29:16.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.395+0000 7f0afa098640 1 --2- 172.21.15.16:0/3662777329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af410fee0 0x7f0af4112300 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.395+0000 7f0afa098640 1 --2- 172.21.15.16:0/3662777329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af4075e20 0x7f0af410f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.395+0000 7f0afa098640 1 --2- 172.21.15.16:0/3662777329 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af40754d0 0x7f0af40758e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.395+0000 7f0afa098640 1 -- 172.21.15.16:0/3662777329 >> 172.21.15.16:0/3662777329 conn(0x7f0af407b5f0 msgr2=0x7f0af407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.395+0000 7f0afa098640 1 -- 172.21.15.16:0/3662777329 shutdown_connections 2024-08-28T19:29:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.395+0000 7f0afa098640 1 -- 172.21.15.16:0/3662777329 wait complete. 2024-08-28T19:29:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.396+0000 7f0afa098640 1 Processor -- start 2024-08-28T19:29:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.396+0000 7f0afa098640 1 -- start start 2024-08-28T19:29:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.396+0000 7f0afa098640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af40754d0 0x7f0af419c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0afa098640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4075e20 0x7f0af419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0afa098640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af410fee0 0x7f0af41a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0afa098640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0af4115a20 con 0x7f0af4075e20 2024-08-28T19:29:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0afa098640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0af41158a0 con 0x7f0af410fee0 2024-08-28T19:29:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0afa098640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0af4115ba0 con 0x7f0af40754d0 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4075e20 0x7f0af419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4075e20 0x7f0af419c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56178/0 (socket says 172.21.15.16:56178) 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af2ffd640 1 -- 172.21.15.16:0/636723698 learned_addr learned my addr 172.21.15.16:0/636723698 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af37fe640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af40754d0 0x7f0af419c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af3fff640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af410fee0 0x7f0af41a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af2ffd640 1 -- 172.21.15.16:0/636723698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af40754d0 msgr2=0x7f0af419c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af40754d0 0x7f0af419c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af2ffd640 1 -- 172.21.15.16:0/636723698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af410fee0 msgr2=0x7f0af41a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af410fee0 0x7f0af41a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af2ffd640 1 -- 172.21.15.16:0/636723698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af41a22e0 con 0x7f0af4075e20 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4075e20 0x7f0af419c910 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00d740 tx=0x7f0adc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.397+0000 7f0af0ff9640 1 -- 172.21.15.16:0/636723698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc017070 con 0x7f0af4075e20 2024-08-28T19:29:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.398+0000 7f0af0ff9640 1 -- 172.21.15.16:0/636723698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0adc00de50 con 0x7f0af4075e20 2024-08-28T19:29:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.398+0000 7f0afa098640 1 -- 172.21.15.16:0/636723698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af41a25d0 con 0x7f0af4075e20 2024-08-28T19:29:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.398+0000 7f0af0ff9640 1 -- 172.21.15.16:0/636723698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc00f620 con 0x7f0af4075e20 2024-08-28T19:29:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.398+0000 7f0afa098640 1 -- 172.21.15.16:0/636723698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af41a8d30 con 0x7f0af4075e20 2024-08-28T19:29:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.400+0000 7f0b0060d740 1 Processor -- start 2024-08-28T19:29:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.400+0000 7f0af0ff9640 1 -- 172.21.15.16:0/636723698 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0adc01a070 con 0x7f0af4075e20 2024-08-28T19:29:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.400+0000 7f0b0060d740 1 -- start start 2024-08-28T19:29:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc1a00f070 0x55cc1a3b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3b4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0b0060d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cc1a32b630 con 0x55cc19fee390 2024-08-28T19:29:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0b0060d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cc1a31abc0 con 0x55cc19fe61b0 2024-08-28T19:29:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0b0060d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cc1a31eb70 con 0x55cc1a00f070 2024-08-28T19:29:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0af2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0af2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3b4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56190/0 (socket says 172.21.15.16:56190) 2024-08-28T19:29:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0af37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0af2ffd640 1 -- 172.21.15.16:0/3190756800 learned_addr learned my addr 172.21.15.16:0/3190756800 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0af3fff640 1 --2- 172.21.15.16:0/3190756800 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc1a00f070 0x55cc1a3b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0af37fe640 1 -- 172.21.15.16:0/3190756800 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc1a00f070 msgr2=0x55cc1a3b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0af37fe640 1 --2- 172.21.15.16:0/3190756800 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc1a00f070 0x55cc1a3b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0af37fe640 1 -- 172.21.15.16:0/3190756800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 msgr2=0x55cc1a3b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.401+0000 7f0af0ff9640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4ca0 0x7f0ae00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.402+0000 7f0af37fe640 1 --2- 172.21.15.16:0/3190756800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3b4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.402+0000 7f0af37fe640 1 -- 172.21.15.16:0/3190756800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc1a07a650 con 0x55cc19fe61b0 2024-08-28T19:29:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.402+0000 7f0af0ff9640 1 -- 172.21.15.16:0/636723698 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f0adc0e5240 con 0x7f0af4075e20 2024-08-28T19:29:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.402+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/3190756800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.402+0000 7f0af37fe640 1 --2- 172.21.15.16:0/3190756800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3b4800 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4005fd0 tx=0x7f0ae40107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b0b400b7308bffe server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.402+0000 7f0af3fff640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4ca0 0x7f0ae00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.403+0000 7f0ae9ffb640 1 -- 172.21.15.16:0/3190756800 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae401c680 con 0x55cc19fe61b0 2024-08-28T19:29:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.403+0000 7f0ae9ffb640 1 -- 172.21.15.16:0/3190756800 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ae400be20 con 0x55cc19fe61b0 2024-08-28T19:29:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.403+0000 7f0ae9ffb640 1 -- 172.21.15.16:0/3190756800 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4011c00 con 0x55cc19fe61b0 2024-08-28T19:29:16.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.403+0000 7f0b0060d740 1 -- 172.21.15.16:0/3190756800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 msgr2=0x55cc1a3b4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:16.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.403+0000 7f0af3fff640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4ca0 0x7f0ae00b71b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0aec006010 tx=0x7f0aec009440 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:16.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.403+0000 7f0b0060d740 1 --2- 172.21.15.16:0/3190756800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3b4800 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4005fd0 tx=0x7f0ae40107a0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.404+0000 7f0b0060d740 1 -- 172.21.15.16:0/3190756800 shutdown_connections 2024-08-28T19:29:16.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.404+0000 7f0b0060d740 1 --2- 172.21.15.16:0/3190756800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.404+0000 7f0b0060d740 1 --2- 172.21.15.16:0/3190756800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3b4800 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.404+0000 7f0b0060d740 1 --2- 172.21.15.16:0/3190756800 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc1a00f070 0x55cc1a3b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.404+0000 7f0b0060d740 1 -- 172.21.15.16:0/3190756800 >> 172.21.15.16:0/3190756800 conn(0x55cc1a003460 msgr2=0x55cc1a3b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.404+0000 7f0b0060d740 1 -- 172.21.15.16:0/3190756800 shutdown_connections 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.404+0000 7f0b0060d740 1 -- 172.21.15.16:0/3190756800 wait complete. 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.404+0000 7f0b0060d740 1 -- start start 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.404+0000 7f0b0060d740 1 Processor -- start 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3bfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 0x55cc1a3bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc1a00f070 0x55cc1a3cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0b0060d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cc1a17d700 con 0x55cc1a00f070 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0b0060d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cc1a2a1700 con 0x55cc19fe61b0 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0b0060d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cc1a3c2aa0 con 0x55cc19fee390 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0b0060d740 10 client.0 _ll_register_callbacks cb 0x55cc1a2d0dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0af3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc1a00f070 0x55cc1a3cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0af3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc1a00f070 0x55cc1a3cbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56198/0 (socket says 172.21.15.16:56198) 2024-08-28T19:29:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0af3fff640 1 -- 172.21.15.16:0/1284913886 learned_addr learned my addr 172.21.15.16:0/1284913886 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 0x55cc1a3bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0af37fe640 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3bfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0af2ffd640 1 -- 172.21.15.16:0/1284913886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 msgr2=0x55cc1a3bfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3bfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0af2ffd640 1 -- 172.21.15.16:0/1284913886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc1a00f070 msgr2=0x55cc1a3cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc1a00f070 0x55cc1a3cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0af2ffd640 1 -- 172.21.15.16:0/1284913886 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc1a2da8b0 con 0x55cc19fee390 2024-08-28T19:29:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 0x55cc1a3bffa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0ab5a0 tx=0x7f0adc0e2ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dc495dfb38552f0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.405+0000 7f0ad2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/1284913886 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc021070 con 0x55cc19fee390 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0b0060d740 1 -- 172.21.15.16:0/1284913886 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ae400be20 con 0x55cc19fee390 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/1284913886 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f0adc0ce2e0 con 0x55cc19fee390 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/1284913886 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f0adc0a74d0 con 0x55cc19fee390 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0b0060d740 10 client.26956 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0b0060d740 1 -- 172.21.15.16:0/1284913886 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55cc1a3ce920 con 0x55cc19fee390 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/1284913886 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ae400be20 con 0x55cc19fee390 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/1284913886 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f0adc0a6f80 con 0x55cc19fee390 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0b0060d740 10 client.26956 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0b0060d740 10 client.26956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0b0060d740 20 client.26956 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0b0060d740 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cc19fcd9f0 0x55cc1a3d1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0b0060d740 4 client.26956 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.406+0000 7f0b0060d740 1 -- 172.21.15.16:0/1284913886 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc1a32d580 con 0x55cc19fcd9f0 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.407+0000 7f0af37fe640 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cc19fcd9f0 0x55cc1a3d1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.407+0000 7f0ad2ffd640 10 client.26956 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.407+0000 7f0af37fe640 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cc19fcd9f0 0x55cc1a3d1e40 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f822b9e37f2cf6ec server_cookie=5bacd05bee95b9e in_seq=0 out_seq=0 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/1284913886 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cc1a32d580 con 0x55cc19fcd9f0 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0ad2ffd640 10 client.26956 handle_command_reply: tid=0 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 1 client.26956 shutdown 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 1 -- 172.21.15.16:0/1284913886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 msgr2=0x55cc1a3bffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 0x55cc1a3bffa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0ab5a0 tx=0x7f0adc0e2ac0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 1 -- 172.21.15.16:0/1284913886 shutdown_connections 2024-08-28T19:29:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc1a00f070 0x55cc1a3cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cc19fcd9f0 0x55cc1a3d1e40 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 0x55cc1a3bffa0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 1 --2- 172.21.15.16:0/1284913886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3bfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0ad2ffd640 0 client.26956 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:16.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 1 -- 172.21.15.16:0/1284913886 >> 172.21.15.16:0/1284913886 conn(0x55cc1a003460 msgr2=0x55cc1a3b7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:16.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 1 -- 172.21.15.16:0/1284913886 shutdown_connections 2024-08-28T19:29:16.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 1 -- 172.21.15.16:0/1284913886 wait complete. 2024-08-28T19:29:16.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.408+0000 7f0b0060d740 20 client.26956 trim_cache size 0 max 16384 2024-08-28T19:29:16.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.426+0000 7f0b0060d740 1 Processor -- start 2024-08-28T19:29:16.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.426+0000 7f0b0060d740 1 -- start start 2024-08-28T19:29:16.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fe61b0 0x55cc1a3b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 0x55cc1a3cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc1a00f070 0x55cc1a3cc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0b0060d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0af41b2f30 con 0x55cc19fe61b0 2024-08-28T19:29:16.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0b0060d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0af41b4bc0 con 0x55cc1a00f070 2024-08-28T19:29:16.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0b0060d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0af41b3660 con 0x55cc19fee390 2024-08-28T19:29:16.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fe61b0 0x55cc1a3b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fe61b0 0x55cc1a3b7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56212/0 (socket says 172.21.15.16:56212) 2024-08-28T19:29:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af37fe640 1 -- 172.21.15.16:0/623350648 learned_addr learned my addr 172.21.15.16:0/623350648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/623350648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 0x55cc1a3cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af3fff640 1 --2- 172.21.15.16:0/623350648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc1a00f070 0x55cc1a3cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af37fe640 1 -- 172.21.15.16:0/623350648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 msgr2=0x55cc1a3cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af37fe640 1 --2- 172.21.15.16:0/623350648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 0x55cc1a3cbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af37fe640 1 -- 172.21.15.16:0/623350648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc1a00f070 msgr2=0x55cc1a3cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af37fe640 1 --2- 172.21.15.16:0/623350648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc1a00f070 0x55cc1a3cc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af37fe640 1 -- 172.21.15.16:0/623350648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc1a3b7120 con 0x55cc19fe61b0 2024-08-28T19:29:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/623350648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 0x55cc1a3cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af3fff640 1 --2- 172.21.15.16:0/623350648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc1a00f070 0x55cc1a3cc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0af37fe640 1 --2- 172.21.15.16:0/623350648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fe61b0 0x55cc1a3b7ad0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4005fd0 tx=0x7f0ae40117a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3c8a6f1a58d963 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0ae97fa640 1 -- 172.21.15.16:0/623350648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4004020 con 0x55cc19fe61b0 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0ae97fa640 1 -- 172.21.15.16:0/623350648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ae40054e0 con 0x55cc19fe61b0 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.427+0000 7f0ae97fa640 1 -- 172.21.15.16:0/623350648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4011b00 con 0x55cc19fe61b0 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.428+0000 7f0b0060d740 1 -- 172.21.15.16:0/623350648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fe61b0 msgr2=0x55cc1a3b7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.428+0000 7f0b0060d740 1 --2- 172.21.15.16:0/623350648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fe61b0 0x55cc1a3b7ad0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4005fd0 tx=0x7f0ae40117a0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.428+0000 7f0b0060d740 1 -- 172.21.15.16:0/623350648 shutdown_connections 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.428+0000 7f0b0060d740 1 --2- 172.21.15.16:0/623350648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc1a00f070 0x55cc1a3cc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.428+0000 7f0b0060d740 1 --2- 172.21.15.16:0/623350648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc19fee390 0x55cc1a3cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.428+0000 7f0b0060d740 1 --2- 172.21.15.16:0/623350648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fe61b0 0x55cc1a3b7ad0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.428+0000 7f0b0060d740 1 -- 172.21.15.16:0/623350648 >> 172.21.15.16:0/623350648 conn(0x55cc1a003460 msgr2=0x55cc1a3b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.428+0000 7f0b0060d740 1 -- 172.21.15.16:0/623350648 shutdown_connections 2024-08-28T19:29:16.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.428+0000 7f0b0060d740 1 -- 172.21.15.16:0/623350648 wait complete. 2024-08-28T19:29:16.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0b0060d740 1 -- start start 2024-08-28T19:29:16.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0b0060d740 1 Processor -- start 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0b0060d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc1a00f070 0x55cc1a3c4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0b0060d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0af41b4460 con 0x55cc19fee390 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0b0060d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0af41b3d30 con 0x55cc19fe61b0 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0b0060d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0af41b0f30 con 0x55cc1a00f070 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0b0060d740 10 client.0 _ll_register_callbacks cb 0x7f0adc009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0af2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0af2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3c41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56226/0 (socket says 172.21.15.16:56226) 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0af2ffd640 1 -- 172.21.15.16:0/3578716056 learned_addr learned my addr 172.21.15.16:0/3578716056 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.429+0000 7f0af37fe640 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0af3fff640 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc1a00f070 0x55cc1a3c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0af2ffd640 1 -- 172.21.15.16:0/3578716056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc1a00f070 msgr2=0x55cc1a3c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc1a00f070 0x55cc1a3c4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0af2ffd640 1 -- 172.21.15.16:0/3578716056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 msgr2=0x55cc1a3c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3c3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0af2ffd640 1 -- 172.21.15.16:0/3578716056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc1a43b800 con 0x55cc19fee390 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0af2ffd640 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3c41c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0b4a40 tx=0x7f0adc0ce360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fddd45319c879d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0ad2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/3578716056 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc021070 con 0x55cc19fee390 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/3578716056 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f0adc0aa2e0 con 0x55cc19fee390 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0b0060d740 1 -- 172.21.15.16:0/3578716056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ae40054e0 con 0x55cc19fee390 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/3578716056 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f0adc0aad80 con 0x55cc19fee390 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0b0060d740 10 client.17649 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.430+0000 7f0b0060d740 1 -- 172.21.15.16:0/3578716056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55cc1a43c990 con 0x55cc19fee390 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.431+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/3578716056 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ae40054e0 con 0x55cc19fee390 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.431+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/3578716056 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f0adc0aa4e0 con 0x55cc19fee390 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.431+0000 7f0b0060d740 10 client.17649 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.431+0000 7f0b0060d740 10 client.17649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.431+0000 7f0b0060d740 20 client.17649 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.431+0000 7f0b0060d740 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cc19fa2da0 0x55cc1a43f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.431+0000 7f0b0060d740 4 client.17649 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.431+0000 7f0b0060d740 1 -- 172.21.15.16:0/3578716056 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cc1a329020 con 0x55cc19fa2da0 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.431+0000 7f0af37fe640 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cc19fa2da0 0x55cc1a43f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.431+0000 7f0af37fe640 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cc19fa2da0 0x55cc1a43f180 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6d914ce506825e1 server_cookie=a9eddafdd5d716c in_seq=0 out_seq=0 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.431+0000 7f0ad2ffd640 10 client.17649 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0ad2ffd640 1 -- 172.21.15.16:0/3578716056 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3694 (crc 0 0 0) 0x55cc1a329020 con 0x55cc19fa2da0 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0ad2ffd640 10 client.17649 handle_command_reply: tid=0 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 1 client.17649 shutdown 2024-08-28T19:29:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 1 -- 172.21.15.16:0/3578716056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 msgr2=0x55cc1a3c41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3c41c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0b4a40 tx=0x7f0adc0ce360 comp rx=0 tx=0).stop 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 1 -- 172.21.15.16:0/3578716056 shutdown_connections 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cc19fa2da0 0x55cc1a43f180 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cc1a00f070 0x55cc1a3c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc19fee390 0x55cc1a3c41c0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 1 --2- 172.21.15.16:0/3578716056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc19fe61b0 0x55cc1a3c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 1 -- 172.21.15.16:0/3578716056 >> 172.21.15.16:0/3578716056 conn(0x55cc1a003460 msgr2=0x55cc1a3b1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0ad2ffd640 0 client.17649 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 1 -- 172.21.15.16:0/3578716056 shutdown_connections 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 1 -- 172.21.15.16:0/3578716056 wait complete. 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.432+0000 7f0b0060d740 20 client.17649 trim_cache size 0 max 16384 2024-08-28T19:29:16.439 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "d863be82-a773-4056-b493-bb9d7d839e7e", 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:29:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:29:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:29:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:29:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:16.447 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:29:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 -- 172.21.15.16:0/636723698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4ca0 msgr2=0x7f0ae00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4ca0 0x7f0ae00b71b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0aec006010 tx=0x7f0aec009440 comp rx=0 tx=0).stop 2024-08-28T19:29:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 -- 172.21.15.16:0/636723698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4075e20 msgr2=0x7f0af419c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4075e20 0x7f0af419c910 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00d740 tx=0x7f0adc00dc00 comp rx=0 tx=0).stop 2024-08-28T19:29:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 -- 172.21.15.16:0/636723698 shutdown_connections 2024-08-28T19:29:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0ae00b4ca0 0x7f0ae00b71b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af410fee0 0x7f0af41a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0af4075e20 0x7f0af419c910 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 --2- 172.21.15.16:0/636723698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0af40754d0 0x7f0af419c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 -- 172.21.15.16:0/636723698 >> 172.21.15.16:0/636723698 conn(0x7f0af407b5f0 msgr2=0x7f0af41015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 -- 172.21.15.16:0/636723698 shutdown_connections 2024-08-28T19:29:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:16.433+0000 7f0afa098640 1 -- 172.21.15.16:0/636723698 wait complete. 2024-08-28T19:29:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:29:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:29:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:29:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:29:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:29:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:29:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:16 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:29:16.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:29:16.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:29:16.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:29:16.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:29:16.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:29:16.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:16 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:29:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:29:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:29:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:29:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:29:16.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:29:16.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:16 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:29:16.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.55 MB/sec execute 296 sec latency 8461.644 ms 2024-08-28T19:29:17.186 INFO:tasks.cephfs.filesystem:scrub status for tag:d863be82-a773-4056-b493-bb9d7d839e7e - {'path': '/', 'tag': 'd863be82-a773-4056-b493-bb9d7d839e7e', 'options': 'recursive,force'} 2024-08-28T19:29:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:17 smithi016 ceph-mon[18556]: pgmap v590: 129 pgs: 129 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.1 MiB/s wr, 101 op/s 2024-08-28T19:29:17.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:17 smithi115 ceph-mon[22184]: pgmap v590: 129 pgs: 129 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.1 MiB/s wr, 101 op/s 2024-08-28T19:29:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:17 smithi028 ceph-mon[21769]: pgmap v590: 129 pgs: 129 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.1 MiB/s wr, 101 op/s 2024-08-28T19:29:17.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.53 MB/sec execute 297 sec latency 9461.686 ms 2024-08-28T19:29:18.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.52 MB/sec execute 298 sec latency 10461.768 ms 2024-08-28T19:29:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:19 smithi115 ceph-mon[22184]: pgmap v591: 129 pgs: 129 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.2 MiB/s wr, 100 op/s 2024-08-28T19:29:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:19 smithi016 ceph-mon[18556]: pgmap v591: 129 pgs: 129 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.2 MiB/s wr, 100 op/s 2024-08-28T19:29:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:19 smithi028 ceph-mon[21769]: pgmap v591: 129 pgs: 129 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.2 MiB/s wr, 100 op/s 2024-08-28T19:29:19.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.50 MB/sec execute 299 sec latency 11461.823 ms 2024-08-28T19:29:20.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.49 MB/sec execute 300 sec latency 12461.880 ms 2024-08-28T19:29:21.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:21 smithi115 ceph-mon[22184]: pgmap v592: 129 pgs: 129 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-08-28T19:29:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:21 smithi016 ceph-mon[18556]: pgmap v592: 129 pgs: 129 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-08-28T19:29:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:21 smithi028 ceph-mon[21769]: pgmap v592: 129 pgs: 129 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-08-28T19:29:21.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.47 MB/sec execute 301 sec latency 13461.922 ms 2024-08-28T19:29:22.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:29:22.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.46 MB/sec execute 302 sec latency 14461.975 ms 2024-08-28T19:29:23.161 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:23.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:23 smithi016 ceph-mon[18556]: pgmap v593: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 887 KiB/s wr, 55 op/s 2024-08-28T19:29:23.610 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:23 smithi028 ceph-mon[21769]: pgmap v593: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 887 KiB/s wr, 55 op/s 2024-08-28T19:29:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:23 smithi115 ceph-mon[22184]: pgmap v593: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 887 KiB/s wr, 55 op/s 2024-08-28T19:29:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.678+0000 7f0d3a822640 1 -- 172.21.15.16:0/98896928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34102880 msgr2=0x7f0d34102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.678+0000 7f0d3a822640 1 --2- 172.21.15.16:0/98896928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34102880 0x7f0d34102c90 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009900 tx=0x7f0d2402a410 comp rx=0 tx=0).stop 2024-08-28T19:29:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.678+0000 7f0d3a822640 1 -- 172.21.15.16:0/98896928 shutdown_connections 2024-08-28T19:29:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.678+0000 7f0d3a822640 1 --2- 172.21.15.16:0/98896928 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0d34108ac0 0x7f0d3410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.678+0000 7f0d3a822640 1 --2- 172.21.15.16:0/98896928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d34108170 0x7f0d34108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.678+0000 7f0d3a822640 1 --2- 172.21.15.16:0/98896928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34102880 0x7f0d34102c90 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.678+0000 7f0d3a822640 1 -- 172.21.15.16:0/98896928 >> 172.21.15.16:0/98896928 conn(0x7f0d34079500 msgr2=0x7f0d34079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.678+0000 7f0d3a822640 1 -- 172.21.15.16:0/98896928 shutdown_connections 2024-08-28T19:29:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.678+0000 7f0d3a822640 1 -- 172.21.15.16:0/98896928 wait complete. 2024-08-28T19:29:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d3a822640 1 Processor -- start 2024-08-28T19:29:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d3a822640 1 -- start start 2024-08-28T19:29:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d3a822640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0d34102880 0x7f0d3419b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d3a822640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d34108170 0x7f0d3419b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d3a822640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34108ac0 0x7f0d3419ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d3a822640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d34112720 con 0x7f0d34108ac0 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d3a822640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d341125a0 con 0x7f0d34108170 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d3a822640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0d341128a0 con 0x7f0d34102880 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d38dbf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34108ac0 0x7f0d3419ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d38dbf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34108ac0 0x7f0d3419ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56246/0 (socket says 172.21.15.16:56246) 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d38dbf640 1 -- 172.21.15.16:0/2936710515 learned_addr learned my addr 172.21.15.16:0/2936710515 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d2bfff640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d34108170 0x7f0d3419b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d33fff640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0d34102880 0x7f0d3419b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d38dbf640 1 -- 172.21.15.16:0/2936710515 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0d34102880 msgr2=0x7f0d3419b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d38dbf640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0d34102880 0x7f0d3419b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d38dbf640 1 -- 172.21.15.16:0/2936710515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d34108170 msgr2=0x7f0d3419b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d38dbf640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d34108170 0x7f0d3419b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.679+0000 7f0d38dbf640 1 -- 172.21.15.16:0/2936710515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d341a6240 con 0x7f0d34108ac0 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.680+0000 7f0d38dbf640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34108ac0 0x7f0d3419ba90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00cce0 tx=0x7f0d2c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.680+0000 7f0d31ffb640 1 -- 172.21.15.16:0/2936710515 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c00f450 con 0x7f0d34108ac0 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.680+0000 7f0d3a822640 1 -- 172.21.15.16:0/2936710515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d341a6530 con 0x7f0d34108ac0 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.680+0000 7f0d31ffb640 1 -- 172.21.15.16:0/2936710515 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d2c00faa0 con 0x7f0d34108ac0 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.680+0000 7f0d31ffb640 1 -- 172.21.15.16:0/2936710515 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c018b70 con 0x7f0d34108ac0 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.680+0000 7f0d3a822640 1 -- 172.21.15.16:0/2936710515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d341a6a20 con 0x7f0d34108ac0 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.681+0000 7f0d3a822640 1 -- 172.21.15.16:0/2936710515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d34103ab0 con 0x7f0d34108ac0 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.682+0000 7f0d31ffb640 1 -- 172.21.15.16:0/2936710515 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0d2c018d10 con 0x7f0d34108ac0 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.683+0000 7f0d31ffb640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0d200b4ca0 0x7f0d200b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.683+0000 7f0d33fff640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0d200b4ca0 0x7f0d200b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.683+0000 7f0d31ffb640 1 -- 172.21.15.16:0/2936710515 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f0d2c0e5e80 con 0x7f0d34108ac0 2024-08-28T19:29:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.684+0000 7f0d33fff640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0d200b4ca0 0x7f0d200b71b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0d24000c00 tx=0x7f0d24031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:23.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.687+0000 7f0d31ffb640 1 -- 172.21.15.16:0/2936710515 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f0d2c014030 con 0x7f0d34108ac0 2024-08-28T19:29:23.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.914+0000 7f0d3a822640 1 -- 172.21.15.16:0/2936710515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d3406bd10 con 0x7f0d34108ac0 2024-08-28T19:29:23.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.915+0000 7f0d31ffb640 1 -- 172.21.15.16:0/2936710515 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11988 (secure 0 0 0) 0x7f0d3406bd10 con 0x7f0d34108ac0 2024-08-28T19:29:23.916 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:29:23.916 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":49,"btime":"2024-08-28T19:28:42:259944+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:28:41.590027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:29:23.916 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 49 2024-08-28T19:29:23.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 -- 172.21.15.16:0/2936710515 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0d200b4ca0 msgr2=0x7f0d200b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:23.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0d200b4ca0 0x7f0d200b71b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0d24000c00 tx=0x7f0d24031400 comp rx=0 tx=0).stop 2024-08-28T19:29:23.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 -- 172.21.15.16:0/2936710515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34108ac0 msgr2=0x7f0d3419ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34108ac0 0x7f0d3419ba90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00cce0 tx=0x7f0d2c00a620 comp rx=0 tx=0).stop 2024-08-28T19:29:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 -- 172.21.15.16:0/2936710515 shutdown_connections 2024-08-28T19:29:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34108ac0 0x7f0d3419ba90 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0d200b4ca0 0x7f0d200b71b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d34108170 0x7f0d3419b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 --2- 172.21.15.16:0/2936710515 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0d34102880 0x7f0d3419b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 -- 172.21.15.16:0/2936710515 >> 172.21.15.16:0/2936710515 conn(0x7f0d34079500 msgr2=0x7f0d34101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 -- 172.21.15.16:0/2936710515 shutdown_connections 2024-08-28T19:29:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:23.917+0000 7f0d3a822640 1 -- 172.21.15.16:0/2936710515 wait complete. 2024-08-28T19:29:23.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.44 MB/sec execute 303 sec latency 15462.017 ms 2024-08-28T19:29:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:24 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2936710515' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:24.653 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:24 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2936710515' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:24.684 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id 55b9eba3 --release --await' 2024-08-28T19:29:24.685 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id 55b9eba3 --release --await 2024-08-28T19:29:24.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:24 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2936710515' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:24.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384083 4.43 MB/sec execute 304 sec latency 16462.073 ms 2024-08-28T19:29:25.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:25 smithi016 ceph-mon[18556]: pgmap v594: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 867 KiB/s wr, 50 op/s 2024-08-28T19:29:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.422+0000 7f13b2f30640 1 -- 172.21.15.16:0/392300277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ac108140 msgr2=0x7f13ac108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.422+0000 7f13b2f30640 1 --2- 172.21.15.16:0/392300277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ac108140 0x7f13ac108550 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009ca0 tx=0x7f13a002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.422+0000 7f13b2f30640 1 -- 172.21.15.16:0/392300277 shutdown_connections 2024-08-28T19:29:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.422+0000 7f13b2f30640 1 --2- 172.21.15.16:0/392300277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13ac108a90 0x7f13ac10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.422+0000 7f13b2f30640 1 --2- 172.21.15.16:0/392300277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ac108140 0x7f13ac108550 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.422+0000 7f13b2f30640 1 --2- 172.21.15.16:0/392300277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ac102800 0x7f13ac102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.422+0000 7f13b2f30640 1 -- 172.21.15.16:0/392300277 >> 172.21.15.16:0/392300277 conn(0x7f13ac0794d0 msgr2=0x7f13ac0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.422+0000 7f13b2f30640 1 -- 172.21.15.16:0/392300277 shutdown_connections 2024-08-28T19:29:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.422+0000 7f13b2f30640 1 -- 172.21.15.16:0/392300277 wait complete. 2024-08-28T19:29:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.423+0000 7f13b2f30640 1 Processor -- start 2024-08-28T19:29:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.423+0000 7f13b2f30640 1 -- start start 2024-08-28T19:29:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.423+0000 7f13b2f30640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13ac102800 0x7f13ac19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.423+0000 7f13b2f30640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ac108140 0x7f13ac19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.423+0000 7f13b2f30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ac108a90 0x7f13ac19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.423+0000 7f13b2f30640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1126c0 con 0x7f13ac108a90 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.423+0000 7f13b2f30640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac112540 con 0x7f13ac108140 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.423+0000 7f13b2f30640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac112840 con 0x7f13ac102800 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.423+0000 7f13b14cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ac108a90 0x7f13ac19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13b14cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ac108a90 0x7f13ac19fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53440/0 (socket says 172.21.15.16:53440) 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13b14cd640 1 -- 172.21.15.16:0/2187026583 learned_addr learned my addr 172.21.15.16:0/2187026583 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13abfff640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ac108140 0x7f13ac19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13ac102800 0x7f13ac19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13b14cd640 1 -- 172.21.15.16:0/2187026583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13ac102800 msgr2=0x7f13ac19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13b14cd640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13ac102800 0x7f13ac19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13b14cd640 1 -- 172.21.15.16:0/2187026583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ac108140 msgr2=0x7f13ac19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13b14cd640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ac108140 0x7f13ac19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13b14cd640 1 -- 172.21.15.16:0/2187026583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ac1a0190 con 0x7f13ac108a90 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13ac102800 0x7f13ac19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13b14cd640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ac108a90 0x7f13ac19fa60 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f13a400d950 tx=0x7f13a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13a9ffb640 1 -- 172.21.15.16:0/2187026583 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4017030 con 0x7f13ac108a90 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13b2f30640 1 -- 172.21.15.16:0/2187026583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ac1a0480 con 0x7f13ac108a90 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13a9ffb640 1 -- 172.21.15.16:0/2187026583 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a4004020 con 0x7f13ac108a90 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.424+0000 7f13a9ffb640 1 -- 172.21.15.16:0/2187026583 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4003b50 con 0x7f13ac108a90 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.425+0000 7f13b2f30640 1 -- 172.21.15.16:0/2187026583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ac1a6be0 con 0x7f13ac108a90 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.427+0000 7f13a9ffb640 1 -- 172.21.15.16:0/2187026583 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f13a401a070 con 0x7f13ac108a90 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.427+0000 7f13b94a5740 1 Processor -- start 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.427+0000 7f13b94a5740 1 -- start start 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.427+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9226520 0x55e2f95cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.427+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91fd660 0x55e2f95cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.427+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f9205840 0x55e2f95cc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.427+0000 7f13b94a5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e2f9542a70 con 0x55e2f9205840 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.427+0000 7f13b94a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e2f9532000 con 0x55e2f91fd660 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.427+0000 7f13b94a5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e2f9535fb0 con 0x55e2f9226520 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f9205840 0x55e2f95cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f9205840 0x55e2f95cc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53442/0 (socket says 172.21.15.16:53442) 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13b0ccc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91fd660 0x55e2f95cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13abfff640 1 -- 172.21.15.16:0/2362000308 learned_addr learned my addr 172.21.15.16:0/2362000308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13a9ffb640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13940b4ca0 0x7f13940b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13abfff640 1 --2- 172.21.15.16:0/2362000308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9226520 0x55e2f95cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13abfff640 1 -- 172.21.15.16:0/2362000308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9226520 msgr2=0x55e2f95cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13abfff640 1 --2- 172.21.15.16:0/2362000308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9226520 0x55e2f95cbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13b14cd640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13940b4ca0 0x7f13940b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13abfff640 1 -- 172.21.15.16:0/2362000308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91fd660 msgr2=0x55e2f95cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13abfff640 1 --2- 172.21.15.16:0/2362000308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91fd660 0x55e2f95cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13abfff640 1 -- 172.21.15.16:0/2362000308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2f9291b00 con 0x55e2f9205840 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/2362000308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91fd660 0x55e2f95cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13a9ffb640 1 -- 172.21.15.16:0/2187026583 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f13a40e50d0 con 0x7f13ac108a90 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f13abfff640 1 --2- 172.21.15.16:0/2362000308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f9205840 0x55e2f95cc700 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f13ac06a840 tx=0x7f13a0009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a6ffce6b305cf2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.428+0000 7f1392ffd640 1 -- 172.21.15.16:0/2362000308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0032c50 con 0x55e2f9205840 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f1392ffd640 1 -- 172.21.15.16:0/2362000308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a0036070 con 0x55e2f9205840 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f1392ffd640 1 -- 172.21.15.16:0/2362000308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0004040 con 0x55e2f9205840 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f13b94a5740 1 -- 172.21.15.16:0/2362000308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f9205840 msgr2=0x55e2f95cc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f13b14cd640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13940b4ca0 0x7f13940b71b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f13a406abf0 tx=0x7f13a40ce040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2362000308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f9205840 0x55e2f95cc700 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f13ac06a840 tx=0x7f13a0009480 comp rx=0 tx=0).stop 2024-08-28T19:29:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f13b94a5740 1 -- 172.21.15.16:0/2362000308 shutdown_connections 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2362000308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f9205840 0x55e2f95cc700 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2362000308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91fd660 0x55e2f95cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2362000308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9226520 0x55e2f95cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f13b94a5740 1 -- 172.21.15.16:0/2362000308 >> 172.21.15.16:0/2362000308 conn(0x55e2f921a910 msgr2=0x55e2f95c9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f13b94a5740 1 -- 172.21.15.16:0/2362000308 shutdown_connections 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.429+0000 7f13b94a5740 1 -- 172.21.15.16:0/2362000308 wait complete. 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13b94a5740 1 -- start start 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13b94a5740 1 Processor -- start 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95d7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f9226520 0x55e2f95e38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13b94a5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e2f95da400 con 0x55e2f91fd660 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13b94a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e2f94f3520 con 0x55e2f9226520 2024-08-28T19:29:25.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13b94a5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e2f95da580 con 0x55e2f9205840 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13b94a5740 10 client.0 _ll_register_callbacks cb 0x55e2f952bc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53450/0 (socket says 172.21.15.16:53450) 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.430+0000 7f13abfff640 1 -- 172.21.15.16:0/2566360000 learned_addr learned my addr 172.21.15.16:0/2566360000 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f13abfff640 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f9226520 0x55e2f95e38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95d7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f13abfff640 1 -- 172.21.15.16:0/2566360000 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 msgr2=0x55e2f95d7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f13abfff640 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95d7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f13abfff640 1 -- 172.21.15.16:0/2566360000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f9226520 msgr2=0x55e2f95e38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f13abfff640 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f9226520 0x55e2f95e38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f13abfff640 1 -- 172.21.15.16:0/2566360000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2f9308960 con 0x55e2f91fd660 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f13abfff640 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d7840 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009620 tx=0x7f13a0003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1203555c69a5565e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f138bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f138bfff640 1 -- 172.21.15.16:0/2566360000 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0055060 con 0x55e2f91fd660 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f138bfff640 1 -- 172.21.15.16:0/2566360000 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f13a0031860 con 0x55e2f91fd660 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.431+0000 7f13b94a5740 1 -- 172.21.15.16:0/2566360000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e2f94d19a0 con 0x55e2f91fd660 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.432+0000 7f138bfff640 1 -- 172.21.15.16:0/2566360000 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f13a0040070 con 0x55e2f91fd660 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.432+0000 7f13b94a5740 10 client.17667 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.432+0000 7f13b94a5740 1 -- 172.21.15.16:0/2566360000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e2f95da400 con 0x55e2f91fd660 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.432+0000 7f138bfff640 1 -- 172.21.15.16:0/2566360000 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e2f94d19a0 con 0x55e2f91fd660 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.432+0000 7f138bfff640 1 -- 172.21.15.16:0/2566360000 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f13a00040c0 con 0x55e2f91fd660 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.432+0000 7f13b94a5740 10 client.17667 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.432+0000 7f13b94a5740 10 client.17667 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.432+0000 7f13b94a5740 20 client.17667 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.432+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e2f91e4ea0 0x55e2f95e9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.432+0000 7f13b94a5740 4 client.17667 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.432+0000 7f13b94a5740 1 -- 172.21.15.16:0/2566360000 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2f95449c0 con 0x55e2f91e4ea0 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.433+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e2f91e4ea0 0x55e2f95e9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.433+0000 7f138bfff640 10 client.17667 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.433+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e2f91e4ea0 0x55e2f95e9760 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f36abe897c5f3c11 server_cookie=6ece64562502bcc3 in_seq=0 out_seq=0 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f138bfff640 1 -- 172.21.15.16:0/2566360000 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e2f95449c0 con 0x55e2f91e4ea0 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f138bfff640 10 client.17667 handle_command_reply: tid=0 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f13b94a5740 1 client.17667 shutdown 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f13b94a5740 1 -- 172.21.15.16:0/2566360000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 msgr2=0x55e2f95d7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d7840 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009620 tx=0x7f13a0003c30 comp rx=0 tx=0).stop 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f13b94a5740 1 -- 172.21.15.16:0/2566360000 shutdown_connections 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f9226520 0x55e2f95e38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e2f91e4ea0 0x55e2f95e9760 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f138bfff640 0 client.17667 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95d7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2566360000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d7840 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.434+0000 7f13b94a5740 1 -- 172.21.15.16:0/2566360000 >> 172.21.15.16:0/2566360000 conn(0x55e2f921a910 msgr2=0x55e2f95cee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.435+0000 7f13b94a5740 1 -- 172.21.15.16:0/2566360000 shutdown_connections 2024-08-28T19:29:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.435+0000 7f13b94a5740 1 -- 172.21.15.16:0/2566360000 wait complete. 2024-08-28T19:29:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.435+0000 7f13b94a5740 20 client.17667 trim_cache size 0 max 16384 2024-08-28T19:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13b94a5740 1 Processor -- start 2024-08-28T19:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13b94a5740 1 -- start start 2024-08-28T19:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91e4ea0 0x55e2f95cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95d9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13b94a5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1b0d40 con 0x55e2f91fd660 2024-08-28T19:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13b94a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1b29d0 con 0x55e2f91e4ea0 2024-08-28T19:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13b94a5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1b1470 con 0x55e2f9205840 2024-08-28T19:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13b0ccc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13b0ccc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53452/0 (socket says 172.21.15.16:53452) 2024-08-28T19:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13b0ccc640 1 -- 172.21.15.16:0/1169637686 learned_addr learned my addr 172.21.15.16:0/1169637686 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.453+0000 7f13abfff640 1 --2- 172.21.15.16:0/1169637686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91e4ea0 0x55e2f95cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.454+0000 7f13abfff640 1 --2- 172.21.15.16:0/1169637686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95d9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.454+0000 7f13b0ccc640 1 -- 172.21.15.16:0/1169637686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 msgr2=0x55e2f95d9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.454+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/1169637686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95d9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.454+0000 7f13b0ccc640 1 -- 172.21.15.16:0/1169637686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91e4ea0 msgr2=0x55e2f95cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.454+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/1169637686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91e4ea0 0x55e2f95cf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.454+0000 7f13b0ccc640 1 -- 172.21.15.16:0/1169637686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2f95d2ed0 con 0x55e2f91fd660 2024-08-28T19:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.454+0000 7f13abfff640 1 --2- 172.21.15.16:0/1169637686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95d9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.454+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/1169637686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d5ec0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f139c005fd0 tx=0x7f139c00b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2510871e2c138fe server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:25.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.454+0000 7f13927fc640 1 -- 172.21.15.16:0/1169637686 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c003f90 con 0x55e2f91fd660 2024-08-28T19:29:25.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.454+0000 7f13927fc640 1 -- 172.21.15.16:0/1169637686 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f139c0054d0 con 0x55e2f91fd660 2024-08-28T19:29:25.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.454+0000 7f13927fc640 1 -- 172.21.15.16:0/1169637686 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c0126b0 con 0x55e2f91fd660 2024-08-28T19:29:25.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.455+0000 7f13b94a5740 1 -- 172.21.15.16:0/1169637686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 msgr2=0x55e2f95d5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:25.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.455+0000 7f13b94a5740 1 --2- 172.21.15.16:0/1169637686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d5ec0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f139c005fd0 tx=0x7f139c00b5c0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.455+0000 7f13b94a5740 1 -- 172.21.15.16:0/1169637686 shutdown_connections 2024-08-28T19:29:25.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.455+0000 7f13b94a5740 1 --2- 172.21.15.16:0/1169637686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95d9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.455+0000 7f13b94a5740 1 --2- 172.21.15.16:0/1169637686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95d5ec0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.455+0000 7f13b94a5740 1 --2- 172.21.15.16:0/1169637686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91e4ea0 0x55e2f95cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.455+0000 7f13b94a5740 1 -- 172.21.15.16:0/1169637686 >> 172.21.15.16:0/1169637686 conn(0x55e2f921a910 msgr2=0x55e2f95d8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.455+0000 7f13b94a5740 1 -- 172.21.15.16:0/1169637686 shutdown_connections 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.455+0000 7f13b94a5740 1 -- 172.21.15.16:0/1169637686 wait complete. 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.456+0000 7f13b94a5740 1 -- start start 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.456+0000 7f13b94a5740 1 Processor -- start 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.456+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91e4ea0 0x55e2f95dff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.456+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95dc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.456+0000 7f13b94a5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95dc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.456+0000 7f13b94a5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1b2270 con 0x55e2f91fd660 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.456+0000 7f13b94a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1b1b40 con 0x55e2f91e4ea0 2024-08-28T19:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.456+0000 7f13b94a5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1aed40 con 0x55e2f9205840 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.456+0000 7f13b0ccc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95dc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13b94a5740 10 client.0 _ll_register_callbacks cb 0x7f139c00b460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13b0ccc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95dc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53468/0 (socket says 172.21.15.16:53468) 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13b0ccc640 1 -- 172.21.15.16:0/2823226187 learned_addr learned my addr 172.21.15.16:0/2823226187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13abfff640 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91e4ea0 0x55e2f95dff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13abfff640 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95dc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13b0ccc640 1 -- 172.21.15.16:0/2823226187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 msgr2=0x55e2f95dc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95dc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13b0ccc640 1 -- 172.21.15.16:0/2823226187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91e4ea0 msgr2=0x55e2f95dff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91e4ea0 0x55e2f95dff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13b0ccc640 1 -- 172.21.15.16:0/2823226187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2f95d65a0 con 0x55e2f91fd660 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13abfff640 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95dc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13b0ccc640 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95dc480 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f139c00b430 tx=0x7f139c005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce15e1b0c4fa29 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f138bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f138bfff640 1 -- 172.21.15.16:0/2823226187 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c004730 con 0x55e2f91fd660 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f138bfff640 1 -- 172.21.15.16:0/2823226187 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f139c0048d0 con 0x55e2f91fd660 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.457+0000 7f13b94a5740 1 -- 172.21.15.16:0/2823226187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f139c0054d0 con 0x55e2f91fd660 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f138bfff640 1 -- 172.21.15.16:0/2823226187 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f139c004ad0 con 0x55e2f91fd660 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f13b94a5740 10 client.17673 fetch_fsmap learned FSMap version 49 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f13b94a5740 1 -- 172.21.15.16:0/2823226187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e2f9654d10 con 0x55e2f91fd660 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f138bfff640 1 -- 172.21.15.16:0/2823226187 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f139c0054d0 con 0x55e2f91fd660 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f138bfff640 1 -- 172.21.15.16:0/2823226187 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 49) ==== 8398+0+0 (secure 0 0 0) 0x7f139c02e4b0 con 0x55e2f91fd660 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f13b94a5740 10 client.17673 fetch_fsmap finished waiting for FSMap version 49 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f13b94a5740 10 client.17673 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f13b94a5740 20 client.17673 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e2f9226520 0x55e2f96574a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f13b94a5740 4 client.17673 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "55b9eba3", "release": true, "await": true}] 2024-08-28T19:29:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f13b94a5740 1 -- 172.21.15.16:0/2823226187 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "55b9eba3", "release": true, "await": true}) -- 0x55e2f9540460 con 0x55e2f9226520 2024-08-28T19:29:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.458+0000 7f13abfff640 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e2f9226520 0x55e2f96574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.459+0000 7f13abfff640 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e2f9226520 0x55e2f96574a0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a50ee0a753451bc5 server_cookie=175bd0fb3c7b74d0 in_seq=0 out_seq=0 2024-08-28T19:29:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.459+0000 7f138bfff640 10 client.17673 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:25.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f138bfff640 1 -- 172.21.15.16:0/2823226187 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e2f9540460 con 0x55e2f9226520 2024-08-28T19:29:25.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f138bfff640 10 client.17673 handle_command_reply: tid=0 2024-08-28T19:29:25.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 1 client.17673 shutdown 2024-08-28T19:29:25.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 1 -- 172.21.15.16:0/2823226187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 msgr2=0x55e2f95dc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:25.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95dc480 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f139c00b430 tx=0x7f139c005710 comp rx=0 tx=0).stop 2024-08-28T19:29:25.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 1 -- 172.21.15.16:0/2823226187 shutdown_connections 2024-08-28T19:29:25.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e2f9226520 0x55e2f96574a0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f138bfff640 0 client.17673 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:25.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e2f9205840 0x55e2f95dc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2f91fd660 0x55e2f95dc480 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 1 --2- 172.21.15.16:0/2823226187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2f91e4ea0 0x55e2f95dff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 1 -- 172.21.15.16:0/2823226187 >> 172.21.15.16:0/2823226187 conn(0x55e2f921a910 msgr2=0x55e2f95cf4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:25.485 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:29:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 49, 2024-08-28T19:29:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:29:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 24, 2024-08-28T19:29:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:29:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "55b9eba3": { 2024-08-28T19:29:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 24, 2024-08-28T19:29:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:29:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:29:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:29:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.5, 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:25.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:25.487 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:29:25.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 1 -- 172.21.15.16:0/2823226187 shutdown_connections 2024-08-28T19:29:25.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 1 -- 172.21.15.16:0/2823226187 wait complete. 2024-08-28T19:29:25.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.482+0000 7f13b94a5740 20 client.17673 trim_cache size 0 max 16384 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 -- 172.21.15.16:0/2187026583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13940b4ca0 msgr2=0x7f13940b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13940b4ca0 0x7f13940b71b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f13a406abf0 tx=0x7f13a40ce040 comp rx=0 tx=0).stop 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 -- 172.21.15.16:0/2187026583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ac108a90 msgr2=0x7f13ac19fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ac108a90 0x7f13ac19fa60 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f13a400d950 tx=0x7f13a400de10 comp rx=0 tx=0).stop 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 -- 172.21.15.16:0/2187026583 shutdown_connections 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ac108a90 0x7f13ac19fa60 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f13940b4ca0 0x7f13940b71b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ac108140 0x7f13ac19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 --2- 172.21.15.16:0/2187026583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f13ac102800 0x7f13ac19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 -- 172.21.15.16:0/2187026583 >> 172.21.15.16:0/2187026583 conn(0x7f13ac0794d0 msgr2=0x7f13ac06afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 -- 172.21.15.16:0/2187026583 shutdown_connections 2024-08-28T19:29:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:25.483+0000 7f13b2f30640 1 -- 172.21.15.16:0/2187026583 wait complete. 2024-08-28T19:29:25.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:25 smithi115 ceph-mon[22184]: pgmap v594: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 867 KiB/s wr, 50 op/s 2024-08-28T19:29:25.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:25 smithi028 ceph-mon[21769]: pgmap v594: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 867 KiB/s wr, 50 op/s 2024-08-28T19:29:25.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384796 4.43 MB/sec execute 305 sec latency 16984.621 ms 2024-08-28T19:29:26.410 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '55b9eba3', total seconds elapsed: 23.0 2024-08-28T19:29:26.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:26 smithi016 ceph-mon[18556]: pgmap v595: 129 pgs: 129 active+clean; 257 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 827 KiB/s wr, 48 op/s 2024-08-28T19:29:26.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:26 smithi028 ceph-mon[21769]: pgmap v595: 129 pgs: 129 active+clean; 257 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 827 KiB/s wr, 48 op/s 2024-08-28T19:29:26.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:26 smithi115 ceph-mon[22184]: pgmap v595: 129 pgs: 129 active+clean; 257 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 827 KiB/s wr, 48 op/s 2024-08-28T19:29:26.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384796 4.41 MB/sec execute 306 sec latency 1033.653 ms 2024-08-28T19:29:27.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 386806 4.44 MB/sec execute 307 sec latency 1051.028 ms 2024-08-28T19:29:28.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 388077 4.45 MB/sec execute 308 sec latency 17.678 ms 2024-08-28T19:29:29.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:28 smithi016 ceph-mon[18556]: pgmap v596: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 813 KiB/s wr, 46 op/s 2024-08-28T19:29:29.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:28 smithi028 ceph-mon[21769]: pgmap v596: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 813 KiB/s wr, 46 op/s 2024-08-28T19:29:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:28 smithi115 ceph-mon[22184]: pgmap v596: 129 pgs: 129 active+clean; 257 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 813 KiB/s wr, 46 op/s 2024-08-28T19:29:29.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 388368 4.43 MB/sec execute 309 sec latency 674.359 ms 2024-08-28T19:29:30.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 388368 4.42 MB/sec execute 310 sec latency 1674.432 ms 2024-08-28T19:29:31.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:30 smithi016 ceph-mon[18556]: pgmap v597: 129 pgs: 129 active+clean; 259 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 962 KiB/s wr, 66 op/s 2024-08-28T19:29:31.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:30 smithi028 ceph-mon[21769]: pgmap v597: 129 pgs: 129 active+clean; 259 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 962 KiB/s wr, 66 op/s 2024-08-28T19:29:31.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:30 smithi115 ceph-mon[22184]: pgmap v597: 129 pgs: 129 active+clean; 259 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 962 KiB/s wr, 66 op/s 2024-08-28T19:29:31.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 388368 4.40 MB/sec execute 311 sec latency 2674.506 ms 2024-08-28T19:29:32.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 390442 4.43 MB/sec execute 312 sec latency 2698.544 ms 2024-08-28T19:29:33.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:32 smithi016 ceph-mon[18556]: pgmap v598: 129 pgs: 129 active+clean; 261 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-08-28T19:29:33.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:32 smithi028 ceph-mon[21769]: pgmap v598: 129 pgs: 129 active+clean; 261 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-08-28T19:29:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:32 smithi115 ceph-mon[22184]: pgmap v598: 129 pgs: 129 active+clean; 261 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-08-28T19:29:33.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 391654 4.44 MB/sec execute 313 sec latency 18.175 ms 2024-08-28T19:29:34.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:33 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:29:34.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:33 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:29:34.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:33 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:29:34.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:33 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:29:34.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:33 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:29:34.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:33 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:29:34.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:33 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:29:34.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:33 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:29:34.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:33 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:29:34.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 391911 4.42 MB/sec execute 314 sec latency 717.299 ms 2024-08-28T19:29:35.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:34 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:29:35.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:34 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:29:35.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:34 smithi016 ceph-mon[18556]: pgmap v599: 129 pgs: 129 active+clean; 261 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 93 op/s 2024-08-28T19:29:35.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:34 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:29:35.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:34 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:29:35.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:34 smithi028 ceph-mon[21769]: pgmap v599: 129 pgs: 129 active+clean; 261 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 93 op/s 2024-08-28T19:29:35.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:34 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:29:35.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:34 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:29:35.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:34 smithi115 ceph-mon[22184]: pgmap v599: 129 pgs: 129 active+clean; 261 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 93 op/s 2024-08-28T19:29:35.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 391911 4.41 MB/sec execute 315 sec latency 1717.380 ms 2024-08-28T19:29:36.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:35 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:29:36.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:35 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:29:36.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:35 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:29:36.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 392688 4.40 MB/sec execute 316 sec latency 2360.289 ms 2024-08-28T19:29:37.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:36 smithi016 ceph-mon[18556]: pgmap v600: 129 pgs: 129 active+clean; 264 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-08-28T19:29:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:36 smithi028 ceph-mon[21769]: pgmap v600: 129 pgs: 129 active+clean; 264 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-08-28T19:29:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:36 smithi115 ceph-mon[22184]: pgmap v600: 129 pgs: 129 active+clean; 264 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-08-28T19:29:37.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 394309 4.42 MB/sec execute 317 sec latency 11.480 ms 2024-08-28T19:29:38.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 395459 4.43 MB/sec execute 318 sec latency 149.171 ms 2024-08-28T19:29:39.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:38 smithi016 ceph-mon[18556]: pgmap v601: 129 pgs: 129 active+clean; 264 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-08-28T19:29:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:38 smithi028 ceph-mon[21769]: pgmap v601: 129 pgs: 129 active+clean; 264 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-08-28T19:29:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:38 smithi115 ceph-mon[22184]: pgmap v601: 129 pgs: 129 active+clean; 264 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-08-28T19:29:39.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 395459 4.41 MB/sec execute 319 sec latency 1149.281 ms 2024-08-28T19:29:40.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 395459 4.40 MB/sec execute 320 sec latency 2149.405 ms 2024-08-28T19:29:41.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:40 smithi016 ceph-mon[18556]: pgmap v602: 129 pgs: 129 active+clean; 266 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-08-28T19:29:41.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:40 smithi028 ceph-mon[21769]: pgmap v602: 129 pgs: 129 active+clean; 266 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-08-28T19:29:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:40 smithi115 ceph-mon[22184]: pgmap v602: 129 pgs: 129 active+clean; 266 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-08-28T19:29:41.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 395459 4.39 MB/sec execute 321 sec latency 3149.489 ms 2024-08-28T19:29:42.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:41 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:29:42.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:41 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:29:42.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:41 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:29:42.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 397558 4.41 MB/sec execute 322 sec latency 3164.976 ms 2024-08-28T19:29:43.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:42 smithi016 ceph-mon[18556]: pgmap v603: 129 pgs: 129 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-08-28T19:29:43.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:29:43.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:29:43.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:29:43.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:42 smithi028 ceph-mon[21769]: pgmap v603: 129 pgs: 129 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-08-28T19:29:43.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:29:43.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:29:43.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:29:43.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:42 smithi115 ceph-mon[22184]: pgmap v603: 129 pgs: 129 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-08-28T19:29:43.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:29:43.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:29:43.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:29:43.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 398802 4.42 MB/sec execute 323 sec latency 16.095 ms 2024-08-28T19:29:44.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 399001 4.40 MB/sec execute 324 sec latency 793.853 ms 2024-08-28T19:29:45.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:44 smithi016 ceph-mon[18556]: pgmap v604: 129 pgs: 129 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-08-28T19:29:45.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:44 smithi028 ceph-mon[21769]: pgmap v604: 129 pgs: 129 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-08-28T19:29:45.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:44 smithi115 ceph-mon[22184]: pgmap v604: 129 pgs: 129 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-08-28T19:29:45.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 399001 4.39 MB/sec execute 325 sec latency 1793.900 ms 2024-08-28T19:29:46.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 399001 4.38 MB/sec execute 326 sec latency 2793.972 ms 2024-08-28T19:29:47.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:46 smithi016 ceph-mon[18556]: pgmap v605: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 167 op/s 2024-08-28T19:29:47.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:46 smithi028 ceph-mon[21769]: pgmap v605: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 167 op/s 2024-08-28T19:29:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:46 smithi115 ceph-mon[22184]: pgmap v605: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 167 op/s 2024-08-28T19:29:47.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:29:47.463 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:47.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.932+0000 7f7cb747b640 1 -- 172.21.15.16:0/1515496669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0108a90 msgr2=0x7f7cb010f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:47.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.932+0000 7f7cb747b640 1 --2- 172.21.15.16:0/1515496669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0108a90 0x7f7cb010f2c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009900 tx=0x7f7ca802a410 comp rx=0 tx=0).stop 2024-08-28T19:29:47.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.933+0000 7f7cb747b640 1 -- 172.21.15.16:0/1515496669 shutdown_connections 2024-08-28T19:29:47.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.933+0000 7f7cb747b640 1 --2- 172.21.15.16:0/1515496669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0108a90 0x7f7cb010f2c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.933+0000 7f7cb747b640 1 --2- 172.21.15.16:0/1515496669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb0108140 0x7f7cb0108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.933+0000 7f7cb747b640 1 --2- 172.21.15.16:0/1515496669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb0102800 0x7f7cb0102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.933+0000 7f7cb747b640 1 -- 172.21.15.16:0/1515496669 >> 172.21.15.16:0/1515496669 conn(0x7f7cb00794d0 msgr2=0x7f7cb00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.933+0000 7f7cb747b640 1 -- 172.21.15.16:0/1515496669 shutdown_connections 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.933+0000 7f7cb747b640 1 -- 172.21.15.16:0/1515496669 wait complete. 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.933+0000 7f7cb747b640 1 Processor -- start 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb747b640 1 -- start start 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb747b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0102800 0x7f7cb0199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb747b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb0108140 0x7f7cb0199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb747b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb0108a90 0x7f7cb019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb747b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb0112710 con 0x7f7cb0102800 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb747b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb0112590 con 0x7f7cb0108a90 2024-08-28T19:29:47.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb747b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb0112890 con 0x7f7cb0108140 2024-08-28T19:29:47.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb5217640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0102800 0x7f7cb0199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb5217640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0102800 0x7f7cb0199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44886/0 (socket says 172.21.15.16:44886) 2024-08-28T19:29:47.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb5217640 1 -- 172.21.15.16:0/760282187 learned_addr learned my addr 172.21.15.16:0/760282187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:47.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb4a16640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb0108140 0x7f7cb0199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb5a18640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb0108a90 0x7f7cb019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb5217640 1 -- 172.21.15.16:0/760282187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb0108140 msgr2=0x7f7cb0199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:47.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb5217640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb0108140 0x7f7cb0199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb5217640 1 -- 172.21.15.16:0/760282187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb0108a90 msgr2=0x7f7cb019e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:47.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb5217640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb0108a90 0x7f7cb019e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb5217640 1 -- 172.21.15.16:0/760282187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb019f0d0 con 0x7f7cb0102800 2024-08-28T19:29:47.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.934+0000 7f7cb4a16640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb0108140 0x7f7cb0199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.935+0000 7f7cb5217640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0102800 0x7f7cb0199160 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00ebc0 tx=0x7f7cac00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.935+0000 7f7ca67fc640 1 -- 172.21.15.16:0/760282187 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac010630 con 0x7f7cb0102800 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.935+0000 7f7cb747b640 1 -- 172.21.15.16:0/760282187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb019f3c0 con 0x7f7cb0102800 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.935+0000 7f7ca67fc640 1 -- 172.21.15.16:0/760282187 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cac010c80 con 0x7f7cb0102800 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.935+0000 7f7ca67fc640 1 -- 172.21.15.16:0/760282187 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac018b60 con 0x7f7cb0102800 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.935+0000 7f7cb747b640 1 -- 172.21.15.16:0/760282187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb006ac00 con 0x7f7cb0102800 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.937+0000 7f7ca67fc640 1 -- 172.21.15.16:0/760282187 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7cac018d00 con 0x7f7cb0102800 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cbd9f0740 1 Processor -- start 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cbd9f0740 1 -- start start 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e024070 0x561a5e3c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3c9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e003390 0x561a5e3c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561a5e340630 con 0x561a5e024070 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561a5e32fbc0 con 0x561a5e003390 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561a5e333b70 con 0x561a5dffb1b0 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cb5a18640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e024070 0x561a5e3c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7ca67fc640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7c9c0b4ca0 0x7f7c9c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cb5217640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cb4a16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e003390 0x561a5e3c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cb5217640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3c9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:54902/0 (socket says 172.21.15.16:54902) 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cb5217640 1 -- 172.21.15.16:0/797481587 learned_addr learned my addr 172.21.15.16:0/797481587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.938+0000 7f7cb5a18640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7c9c0b4ca0 0x7f7c9c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7ca67fc640 1 -- 172.21.15.16:0/760282187 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f7cac0e5f00 con 0x7f7cb0102800 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7cb4a16640 1 -- 172.21.15.16:0/797481587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 msgr2=0x561a5e3c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7cb4a16640 1 --2- 172.21.15.16:0/797481587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7cb4a16640 1 -- 172.21.15.16:0/797481587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e024070 msgr2=0x561a5e3c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7cb4a16640 1 --2- 172.21.15.16:0/797481587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e024070 0x561a5e3c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7cb4a16640 1 -- 172.21.15.16:0/797481587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a5e08f650 con 0x561a5e003390 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7cb5217640 1 --2- 172.21.15.16:0/797481587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7cb4a16640 1 --2- 172.21.15.16:0/797481587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e003390 0x561a5e3c9d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7cb006a840 tx=0x7f7ca0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=902c13543aa3f1b5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7c9b7fe640 1 -- 172.21.15.16:0/797481587 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca001b020 con 0x561a5e003390 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7cb5a18640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7c9c0b4ca0 0x7f7c9c0b71b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8006010 tx=0x7f7ca802a410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7c9b7fe640 1 -- 172.21.15.16:0/797481587 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ca00054d0 con 0x561a5e003390 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.939+0000 7f7c9b7fe640 1 -- 172.21.15.16:0/797481587 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0004ca0 con 0x561a5e003390 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.940+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/797481587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e003390 msgr2=0x561a5e3c9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.940+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/797481587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e003390 0x561a5e3c9d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7cb006a840 tx=0x7f7ca0005f90 comp rx=0 tx=0).stop 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.940+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/797481587 shutdown_connections 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.940+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/797481587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e003390 0x561a5e3c9d80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.940+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/797481587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.940+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/797481587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e024070 0x561a5e3c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.940+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/797481587 >> 172.21.15.16:0/797481587 conn(0x561a5e018460 msgr2=0x561a5e3c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.940+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/797481587 shutdown_connections 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.940+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/797481587 wait complete. 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cbd9f0740 1 -- start start 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cbd9f0740 1 Processor -- start 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 0x561a5e3d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 0x561a5e3e95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561a5e192700 con 0x561a5e003390 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561a5e2b6700 con 0x561a5e024070 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561a5e3e0270 con 0x561a5dffb1b0 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cbd9f0740 10 client.0 _ll_register_callbacks cb 0x561a5e2e5dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cb4a16640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 0x561a5e3d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cb5217640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cb5a18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 0x561a5e3e95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cb4a16640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 0x561a5e3d0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44902/0 (socket says 172.21.15.16:44902) 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cb5217640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3d0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:54904/0 (socket says 172.21.15.16:54904) 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cb4a16640 1 -- 172.21.15.16:0/2529913725 learned_addr learned my addr 172.21.15.16:0/2529913725 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.941+0000 7f7cb5a18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 0x561a5e3e95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34632/0 (socket says 172.21.15.16:34632) 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7cb5a18640 1 -- 172.21.15.16:0/2529913725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 msgr2=0x561a5e3d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7cb5a18640 1 --2- 172.21.15.16:0/2529913725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3d0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7cb5a18640 1 -- 172.21.15.16:0/2529913725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 msgr2=0x561a5e3d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7cb5a18640 1 --2- 172.21.15.16:0/2529913725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 0x561a5e3d0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7cb5a18640 1 -- 172.21.15.16:0/2529913725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a5e2ef8b0 con 0x561a5e024070 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7cb5a18640 1 --2- 172.21.15.16:0/2529913725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 0x561a5e3e95a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8005940 tx=0x7f7ca8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8011a0880a74e03 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7c98ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7c98ff9640 1 -- 172.21.15.16:0/2529913725 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8050060 con 0x561a5e024070 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/2529913725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ca00054d0 con 0x561a5e024070 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7c98ff9640 1 -- 172.21.15.16:0/2529913725 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f7ca8004250 con 0x561a5e024070 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7c98ff9640 1 -- 172.21.15.16:0/2529913725 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f7ca8057050 con 0x561a5e024070 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7cbd9f0740 10 client.27215 fetch_fsmap learned FSMap version 50 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.942+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/2529913725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x561a5e3ebf60 con 0x561a5e024070 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.943+0000 7f7c98ff9640 1 -- 172.21.15.16:0/2529913725 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ca00054d0 con 0x561a5e024070 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.943+0000 7f7c98ff9640 1 -- 172.21.15.16:0/2529913725 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 50) ==== 8394+0+0 (secure 0 0 0) 0x7f7ca804a070 con 0x561a5e024070 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.943+0000 7f7cbd9f0740 10 client.27215 fetch_fsmap finished waiting for FSMap version 50 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.943+0000 7f7cbd9f0740 10 client.27215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.943+0000 7f7cbd9f0740 20 client.27215 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.943+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/2529913725 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561a5dfe29f0 0x561a5e3ef510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.943+0000 7f7cbd9f0740 4 client.27215 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.943+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/2529913725 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a5e342580 con 0x561a5dfe29f0 2024-08-28T19:29:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.943+0000 7f7cb5217640 1 --2- 172.21.15.16:0/2529913725 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561a5dfe29f0 0x561a5e3ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.943+0000 7f7cb5217640 1 --2- 172.21.15.16:0/2529913725 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561a5dfe29f0 0x561a5e3ef510 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a10eadda11378169 server_cookie=3307d727435e1f90 in_seq=0 out_seq=0 2024-08-28T19:29:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.943+0000 7f7c98ff9640 10 client.27215 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7c98ff9640 1 -- 172.21.15.16:0/2529913725 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561a5e342580 con 0x561a5dfe29f0 2024-08-28T19:29:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7c98ff9640 10 client.27215 handle_command_reply: tid=0 2024-08-28T19:29:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7cbd9f0740 1 client.27215 shutdown 2024-08-28T19:29:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/2529913725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 msgr2=0x561a5e3e95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/2529913725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 0x561a5e3e95a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8005940 tx=0x7f7ca8009260 comp rx=0 tx=0).stop 2024-08-28T19:29:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/2529913725 shutdown_connections 2024-08-28T19:29:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/2529913725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 0x561a5e3e95a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/2529913725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 0x561a5e3d0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/2529913725 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561a5dfe29f0 0x561a5e3ef510 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/2529913725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3d0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7c98ff9640 0 client.27215 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.944+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/2529913725 >> 172.21.15.16:0/2529913725 conn(0x561a5e018460 msgr2=0x561a5e3c57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.945+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/2529913725 shutdown_connections 2024-08-28T19:29:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.945+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/2529913725 wait complete. 2024-08-28T19:29:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.945+0000 7f7cbd9f0740 20 client.27215 trim_cache size 0 max 16384 2024-08-28T19:29:47.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 401066 4.40 MB/sec execute 327 sec latency 2809.434 ms 2024-08-28T19:29:47.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.962+0000 7f7cbd9f0740 1 Processor -- start 2024-08-28T19:29:47.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.962+0000 7f7cbd9f0740 1 -- start start 2024-08-28T19:29:47.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3d2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 0x561a5e3e94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 0x561a5e3cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01aeb20 con 0x561a5e003390 2024-08-28T19:29:47.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01b0780 con 0x561a5e024070 2024-08-28T19:29:47.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01af250 con 0x561a5dffb1b0 2024-08-28T19:29:47.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cb4a16640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 0x561a5e3e94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cb4a16640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 0x561a5e3e94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44912/0 (socket says 172.21.15.16:44912) 2024-08-28T19:29:47.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cb4a16640 1 -- 172.21.15.16:0/641970445 learned_addr learned my addr 172.21.15.16:0/641970445 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:47.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cb5217640 1 --2- 172.21.15.16:0/641970445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3d2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cb5a18640 1 --2- 172.21.15.16:0/641970445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 0x561a5e3cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cb4a16640 1 -- 172.21.15.16:0/641970445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 msgr2=0x561a5e3d2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cb4a16640 1 --2- 172.21.15.16:0/641970445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3d2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cb4a16640 1 -- 172.21.15.16:0/641970445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 msgr2=0x561a5e3cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cb4a16640 1 --2- 172.21.15.16:0/641970445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 0x561a5e3cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cb4a16640 1 -- 172.21.15.16:0/641970445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a5e3c7730 con 0x561a5e003390 2024-08-28T19:29:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.963+0000 7f7cb4a16640 1 --2- 172.21.15.16:0/641970445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 0x561a5e3e94a0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0005b90 tx=0x7f7ca0004070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f6c2c9bf04222c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.964+0000 7f7c9affd640 1 -- 172.21.15.16:0/641970445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca001b030 con 0x561a5e003390 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.964+0000 7f7c9affd640 1 -- 172.21.15.16:0/641970445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ca0027070 con 0x561a5e003390 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.964+0000 7f7c9affd640 1 -- 172.21.15.16:0/641970445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca00236b0 con 0x561a5e003390 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.964+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/641970445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 msgr2=0x561a5e3e94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.964+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/641970445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 0x561a5e3e94a0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0005b90 tx=0x7f7ca0004070 comp rx=0 tx=0).stop 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.964+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/641970445 shutdown_connections 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.964+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/641970445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5e024070 0x561a5e3cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.965+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/641970445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e003390 0x561a5e3e94a0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.965+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/641970445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5dffb1b0 0x561a5e3d2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.965+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/641970445 >> 172.21.15.16:0/641970445 conn(0x561a5e018460 msgr2=0x561a5e3d1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.965+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/641970445 shutdown_connections 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.965+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/641970445 wait complete. 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.965+0000 7f7cbd9f0740 1 -- start start 2024-08-28T19:29:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.965+0000 7f7cbd9f0740 1 Processor -- start 2024-08-28T19:29:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.965+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5dffb1b0 0x561a5e3e5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.965+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5e003390 0x561a5e3e1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cbd9f0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e024070 0x561a5e3e2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01b0020 con 0x561a5e024070 2024-08-28T19:29:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01af920 con 0x561a5dffb1b0 2024-08-28T19:29:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cbd9f0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01acb20 con 0x561a5e003390 2024-08-28T19:29:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cbd9f0740 10 client.0 _ll_register_callbacks cb 0x7f7ca8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb5217640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5dffb1b0 0x561a5e3e5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb4a16640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5e003390 0x561a5e3e1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb5217640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5dffb1b0 0x561a5e3e5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34662/0 (socket says 172.21.15.16:34662) 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb5217640 1 -- 172.21.15.16:0/334765323 learned_addr learned my addr 172.21.15.16:0/334765323 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb5a18640 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e024070 0x561a5e3e2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb5217640 1 -- 172.21.15.16:0/334765323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5e003390 msgr2=0x561a5e3e1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb5217640 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5e003390 0x561a5e3e1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb5217640 1 -- 172.21.15.16:0/334765323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e024070 msgr2=0x561a5e3e2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb5217640 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e024070 0x561a5e3e2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb5217640 1 -- 172.21.15.16:0/334765323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a5e3e29a0 con 0x561a5dffb1b0 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb4a16640 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5e003390 0x561a5e3e1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cb5217640 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5dffb1b0 0x561a5e3e5a60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7cac0a8350 tx=0x7f7cac000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44cb18e3d072e96c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7c98ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7c98ff9640 1 -- 172.21.15.16:0/334765323 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac0abde0 con 0x561a5dffb1b0 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7c98ff9640 1 -- 172.21.15.16:0/334765323 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f7cac0107d0 con 0x561a5dffb1b0 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.966+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/334765323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ca0027070 con 0x561a5dffb1b0 2024-08-28T19:29:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.967+0000 7f7c98ff9640 1 -- 172.21.15.16:0/334765323 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f7cac0ac2d0 con 0x561a5dffb1b0 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.967+0000 7f7cbd9f0740 10 client.27221 fetch_fsmap learned FSMap version 50 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.967+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/334765323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x561a5e45ac40 con 0x561a5dffb1b0 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.968+0000 7f7c98ff9640 1 -- 172.21.15.16:0/334765323 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ca0027070 con 0x561a5dffb1b0 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.968+0000 7f7c98ff9640 1 -- 172.21.15.16:0/334765323 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 50) ==== 8394+0+0 (secure 0 0 0) 0x7f7cac0ab530 con 0x561a5dffb1b0 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.968+0000 7f7cbd9f0740 10 client.27221 fetch_fsmap finished waiting for FSMap version 50 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.968+0000 7f7cbd9f0740 10 client.27221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.968+0000 7f7cbd9f0740 20 client.27221 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.968+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561a5dfb7da0 0x561a5e45d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.968+0000 7f7cbd9f0740 4 client.27221 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.968+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/334765323 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561a5e33e020 con 0x561a5dfb7da0 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.968+0000 7f7cb4a16640 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561a5dfb7da0 0x561a5e45d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.968+0000 7f7c98ff9640 10 client.27221 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.968+0000 7f7cb4a16640 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561a5dfb7da0 0x561a5e45d490 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9d30d6036dd0b32 server_cookie=90b9b2835c094482 in_seq=0 out_seq=0 2024-08-28T19:29:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7c98ff9640 1 -- 172.21.15.16:0/334765323 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3498 (crc 0 0 0) 0x561a5e33e020 con 0x561a5dfb7da0 2024-08-28T19:29:47.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7c98ff9640 10 client.27221 handle_command_reply: tid=0 2024-08-28T19:29:47.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7cbd9f0740 1 client.27221 shutdown 2024-08-28T19:29:47.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/334765323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5dffb1b0 msgr2=0x561a5e3e5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:47.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5dffb1b0 0x561a5e3e5a60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7cac0a8350 tx=0x7f7cac000f40 comp rx=0 tx=0).stop 2024-08-28T19:29:47.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/334765323 shutdown_connections 2024-08-28T19:29:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a5e024070 0x561a5e3e2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561a5dfb7da0 0x561a5e45d490 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561a5e003390 0x561a5e3e1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7cbd9f0740 1 --2- 172.21.15.16:0/334765323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a5dffb1b0 0x561a5e3e5a60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/334765323 >> 172.21.15.16:0/334765323 conn(0x561a5e018460 msgr2=0x561a5e3cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.969+0000 7f7c98ff9640 0 client.27221 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/334765323 shutdown_connections 2024-08-28T19:29:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cbd9f0740 1 -- 172.21.15.16:0/334765323 wait complete. 2024-08-28T19:29:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cbd9f0740 20 client.27221 trim_cache size 0 max 16384 2024-08-28T19:29:47.974 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:29:47.975 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.977 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:29:47.978 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:47.979 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.980 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:29:47.981 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:29:47.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:29:47.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:29:47.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:29:47.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:47.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:29:47.982 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:29:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cb747b640 1 -- 172.21.15.16:0/760282187 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7c9c0b4ca0 msgr2=0x7f7c9c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cb747b640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7c9c0b4ca0 0x7f7c9c0b71b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8006010 tx=0x7f7ca802a410 comp rx=0 tx=0).stop 2024-08-28T19:29:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cb747b640 1 -- 172.21.15.16:0/760282187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0102800 msgr2=0x7f7cb0199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cb747b640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0102800 0x7f7cb0199160 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00ebc0 tx=0x7f7cac00c6a0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cb747b640 1 -- 172.21.15.16:0/760282187 shutdown_connections 2024-08-28T19:29:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cb747b640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7c9c0b4ca0 0x7f7c9c0b71b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cb747b640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb0108a90 0x7f7cb019e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cb747b640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb0108140 0x7f7cb0199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cb747b640 1 --2- 172.21.15.16:0/760282187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0102800 0x7f7cb0199160 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.970+0000 7f7cb747b640 1 -- 172.21.15.16:0/760282187 >> 172.21.15.16:0/760282187 conn(0x7f7cb00794d0 msgr2=0x7f7cb0105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.971+0000 7f7cb747b640 1 -- 172.21.15.16:0/760282187 shutdown_connections 2024-08-28T19:29:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:47.971+0000 7f7cb747b640 1 -- 172.21.15.16:0/760282187 wait complete. 2024-08-28T19:29:48.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:48 smithi016 ceph-mon[18556]: pgmap v606: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 133 op/s 2024-08-28T19:29:48.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 402142 4.40 MB/sec execute 328 sec latency 20.342 ms 2024-08-28T19:29:48.965 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:29:48.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:29:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:48 smithi028 ceph-mon[21769]: pgmap v606: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 133 op/s 2024-08-28T19:29:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:48 smithi115 ceph-mon[22184]: pgmap v606: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 133 op/s 2024-08-28T19:29:49.225 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:49.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.714+0000 7f1ca23d7640 1 -- 172.21.15.16:0/3723787591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c108ab0 msgr2=0x7f1c9c10f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:49.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.714+0000 7f1ca23d7640 1 --2- 172.21.15.16:0/3723787591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c108ab0 0x7f1c9c10f2e0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f1c94009ca0 tx=0x7f1c9402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:29:49.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.715+0000 7f1ca23d7640 1 -- 172.21.15.16:0/3723787591 shutdown_connections 2024-08-28T19:29:49.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.715+0000 7f1ca23d7640 1 --2- 172.21.15.16:0/3723787591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c108ab0 0x7f1c9c10f2e0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:49.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.715+0000 7f1ca23d7640 1 --2- 172.21.15.16:0/3723787591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c9c108160 0x7f1c9c108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:49.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.715+0000 7f1ca23d7640 1 --2- 172.21.15.16:0/3723787591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c102820 0x7f1c9c102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.715+0000 7f1ca23d7640 1 -- 172.21.15.16:0/3723787591 >> 172.21.15.16:0/3723787591 conn(0x7f1c9c0794e0 msgr2=0x7f1c9c0798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.715+0000 7f1ca23d7640 1 -- 172.21.15.16:0/3723787591 shutdown_connections 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.715+0000 7f1ca23d7640 1 -- 172.21.15.16:0/3723787591 wait complete. 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.715+0000 7f1ca23d7640 1 Processor -- start 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.715+0000 7f1ca23d7640 1 -- start start 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1ca23d7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c9c102820 0x7f1c9c19afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1ca23d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c108160 0x7f1c9c19b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1ca23d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c108ab0 0x7f1c9c19ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1ca23d7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c112730 con 0x7f1c9c108160 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1ca23d7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1125b0 con 0x7f1c9c108ab0 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1ca23d7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1128b0 con 0x7f1c9c102820 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1c9b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c108160 0x7f1c9c19b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1c9b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c108160 0x7f1c9c19b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44924/0 (socket says 172.21.15.16:44924) 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/2516342041 learned_addr learned my addr 172.21.15.16:0/2516342041 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1ca0974640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c108ab0 0x7f1c9c19ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1c9bfff640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c9c102820 0x7f1c9c19afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/2516342041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c9c102820 msgr2=0x7f1c9c19afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c9c102820 0x7f1c9c19afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/2516342041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c108ab0 msgr2=0x7f1c9c19ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c108ab0 0x7f1c9c19ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.716+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/2516342041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c1a62a0 con 0x7f1c9c108160 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.717+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c108160 0x7f1c9c19b520 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000d740 tx=0x7f1c9000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.717+0000 7f1c997fa640 1 -- 172.21.15.16:0/2516342041 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90017070 con 0x7f1c9c108160 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.717+0000 7f1ca23d7640 1 -- 172.21.15.16:0/2516342041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9c1a6590 con 0x7f1c9c108160 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.717+0000 7f1c997fa640 1 -- 172.21.15.16:0/2516342041 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c9000de50 con 0x7f1c9c108160 2024-08-28T19:29:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.717+0000 7f1c997fa640 1 -- 172.21.15.16:0/2516342041 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90004ce0 con 0x7f1c9c108160 2024-08-28T19:29:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.717+0000 7f1ca23d7640 1 -- 172.21.15.16:0/2516342041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9c1a6a80 con 0x7f1c9c108160 2024-08-28T19:29:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.718+0000 7f1ca23d7640 1 -- 172.21.15.16:0/2516342041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c9c103aa0 con 0x7f1c9c108160 2024-08-28T19:29:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.719+0000 7f1c997fa640 1 -- 172.21.15.16:0/2516342041 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1c90004e80 con 0x7f1c9c108160 2024-08-28T19:29:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.720+0000 7f1c997fa640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c840b4ca0 0x7f1c840b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.720+0000 7f1c997fa640 1 -- 172.21.15.16:0/2516342041 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f1c900e5610 con 0x7f1c9c108160 2024-08-28T19:29:49.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.720+0000 7f1c9bfff640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c840b4ca0 0x7f1c840b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:49.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.721+0000 7f1c9bfff640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c840b4ca0 0x7f1c840b71b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c06c460 tx=0x7f1c8c005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:49.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.723+0000 7f1c997fa640 1 -- 172.21.15.16:0/2516342041 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f1c900ab9e0 con 0x7f1c9c108160 2024-08-28T19:29:49.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.946+0000 7f1ca23d7640 1 -- 172.21.15.16:0/2516342041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c9c102c30 con 0x7f1c9c108160 2024-08-28T19:29:49.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.946+0000 7f1c997fa640 1 -- 172.21.15.16:0/2516342041 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+11986 (secure 0 0 0) 0x7f1c900b01a0 con 0x7f1c9c108160 2024-08-28T19:29:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:29:49.948 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":50,"btime":"2024-08-28T19:29:34:719269+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":50,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:29:34.537035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:29:49.948 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 50 2024-08-28T19:29:49.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 -- 172.21.15.16:0/2516342041 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c840b4ca0 msgr2=0x7f1c840b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:49.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c840b4ca0 0x7f1c840b71b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c06c460 tx=0x7f1c8c005f90 comp rx=0 tx=0).stop 2024-08-28T19:29:49.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 -- 172.21.15.16:0/2516342041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c108160 msgr2=0x7f1c9c19b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:49.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c108160 0x7f1c9c19b520 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000d740 tx=0x7f1c9000dc00 comp rx=0 tx=0).stop 2024-08-28T19:29:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 -- 172.21.15.16:0/2516342041 shutdown_connections 2024-08-28T19:29:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c840b4ca0 0x7f1c840b71b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c108ab0 0x7f1c9c19ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c108160 0x7f1c9c19b520 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 --2- 172.21.15.16:0/2516342041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c9c102820 0x7f1c9c19afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 -- 172.21.15.16:0/2516342041 >> 172.21.15.16:0/2516342041 conn(0x7f1c9c0794e0 msgr2=0x7f1c9c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 -- 172.21.15.16:0/2516342041 shutdown_connections 2024-08-28T19:29:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:49.949+0000 7f1ca23d7640 1 -- 172.21.15.16:0/2516342041 wait complete. 2024-08-28T19:29:49.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 402551 4.39 MB/sec execute 329 sec latency 645.227 ms 2024-08-28T19:29:50.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:29:50.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 402551 4.38 MB/sec execute 330 sec latency 1645.272 ms 2024-08-28T19:29:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:51.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:50 smithi028 ceph-mon[21769]: pgmap v607: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 126 op/s 2024-08-28T19:29:51.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:50 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2516342041' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:51.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:50 smithi115 ceph-mon[22184]: pgmap v607: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 126 op/s 2024-08-28T19:29:51.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:50 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2516342041' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:51.175 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:50 smithi016 ceph-mon[18556]: pgmap v607: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 126 op/s 2024-08-28T19:29:51.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:50 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2516342041' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:29:51.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.534+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/571167498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2acc109bf0 msgr2=0x7f2acc110420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:51.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.534+0000 7f2ad2a1a640 1 --2- 172.21.15.16:0/571167498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2acc109bf0 0x7f2acc110420 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009ca0 tx=0x7f2ac402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.534+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/571167498 shutdown_connections 2024-08-28T19:29:51.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.534+0000 7f2ad2a1a640 1 --2- 172.21.15.16:0/571167498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2acc109bf0 0x7f2acc110420 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.534+0000 7f2ad2a1a640 1 --2- 172.21.15.16:0/571167498 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc1092a0 0x7f2acc1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.534+0000 7f2ad2a1a640 1 --2- 172.21.15.16:0/571167498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2acc101d60 0x7f2acc102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.534+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/571167498 >> 172.21.15.16:0/571167498 conn(0x7f2acc079d30 msgr2=0x7f2acc07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:51.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.534+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/571167498 shutdown_connections 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.534+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/571167498 wait complete. 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.535+0000 7f2ad2a1a640 1 Processor -- start 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.535+0000 7f2ad2a1a640 1 -- start start 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.535+0000 7f2ad2a1a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc101d60 0x7f2acc102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.535+0000 7f2ad2a1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2acc1092a0 0x7f2acc103240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.535+0000 7f2ad2a1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2acc109bf0 0x7f2acc103780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.535+0000 7f2ad2a1a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc1138b0 con 0x7f2acc1092a0 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.535+0000 7f2ad2a1a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc113730 con 0x7f2acc109bf0 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.535+0000 7f2ad2a1a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc113a30 con 0x7f2acc101d60 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.535+0000 7f2ac3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2acc1092a0 0x7f2acc103240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.535+0000 7f2ac3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2acc1092a0 0x7f2acc103240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44952/0 (socket says 172.21.15.16:44952) 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.535+0000 7f2ac3fff640 1 -- 172.21.15.16:0/1284228883 learned_addr learned my addr 172.21.15.16:0/1284228883 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ad0fb7640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2acc109bf0 0x7f2acc103780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2acbfff640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc101d60 0x7f2acc102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ac3fff640 1 -- 172.21.15.16:0/1284228883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc101d60 msgr2=0x7f2acc102d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ac3fff640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc101d60 0x7f2acc102d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ac3fff640 1 -- 172.21.15.16:0/1284228883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2acc109bf0 msgr2=0x7f2acc103780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ac3fff640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2acc109bf0 0x7f2acc103780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ac3fff640 1 -- 172.21.15.16:0/1284228883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2acc1a8500 con 0x7f2acc1092a0 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2acbfff640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc101d60 0x7f2acc102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ad0fb7640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2acc109bf0 0x7f2acc103780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ac3fff640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2acc1092a0 0x7f2acc103240 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400d750 tx=0x7f2ab400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ac9ffb640 1 -- 172.21.15.16:0/1284228883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4017070 con 0x7f2acc1092a0 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/1284228883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2acc1a87f0 con 0x7f2acc1092a0 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ac9ffb640 1 -- 172.21.15.16:0/1284228883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ab40054d0 con 0x7f2acc1092a0 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ac9ffb640 1 -- 172.21.15.16:0/1284228883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4004c30 con 0x7f2acc1092a0 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.536+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/1284228883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2acc1a8d30 con 0x7f2acc1092a0 2024-08-28T19:29:51.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.538+0000 7f2ad8f8f740 1 Processor -- start 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2ad8f8f740 1 -- start start 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2ac9ffb640 1 -- 172.21.15.16:0/1284228883 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2ab4004dd0 con 0x7f2acc1092a0 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f24070 0x55e4f52c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4efb1b0 0x55e4f52c9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4f03390 0x55e4f52c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e4f5240680 con 0x55e4f4efb1b0 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e4f522fc10 con 0x55e4f4f24070 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e4f5233bc0 con 0x55e4f4f03390 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2acbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4efb1b0 0x55e4f52c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2acbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4efb1b0 0x55e4f52c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44958/0 (socket says 172.21.15.16:44958) 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2acbfff640 1 -- 172.21.15.16:0/3130932494 learned_addr learned my addr 172.21.15.16:0/3130932494 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:51.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2ac3fff640 1 --2- 172.21.15.16:0/3130932494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4f03390 0x55e4f52c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2ad0fb7640 1 --2- 172.21.15.16:0/3130932494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f24070 0x55e4f52c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2acbfff640 1 -- 172.21.15.16:0/3130932494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4f03390 msgr2=0x55e4f52c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2acbfff640 1 --2- 172.21.15.16:0/3130932494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4f03390 0x55e4f52c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2acbfff640 1 -- 172.21.15.16:0/3130932494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f24070 msgr2=0x55e4f52c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2acbfff640 1 --2- 172.21.15.16:0/3130932494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f24070 0x55e4f52c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.539+0000 7f2acbfff640 1 -- 172.21.15.16:0/3130932494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4f4f8f650 con 0x55e4f4efb1b0 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.540+0000 7f2ac9ffb640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ab80b4ca0 0x7f2ab80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.540+0000 7f2acbfff640 1 --2- 172.21.15.16:0/3130932494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4efb1b0 0x55e4f52c9850 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00bb80 tx=0x7f2abc00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87eac26c6b5c0fd3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.540+0000 7f2ad0fb7640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ab80b4ca0 0x7f2ab80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.540+0000 7f2ac9ffb640 1 -- 172.21.15.16:0/1284228883 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f2ab40e5420 con 0x7f2acc1092a0 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.540+0000 7f2ac27fc640 1 -- 172.21.15.16:0/3130932494 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc017620 con 0x55e4f4efb1b0 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.540+0000 7f2ac27fc640 1 -- 172.21.15.16:0/3130932494 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2abc017c70 con 0x55e4f4efb1b0 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.540+0000 7f2ac27fc640 1 -- 172.21.15.16:0/3130932494 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc00fb60 con 0x55e4f4efb1b0 2024-08-28T19:29:51.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.541+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3130932494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4efb1b0 msgr2=0x55e4f52c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.541+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3130932494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4efb1b0 0x55e4f52c9850 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00bb80 tx=0x7f2abc00d710 comp rx=0 tx=0).stop 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.541+0000 7f2ad0fb7640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ab80b4ca0 0x7f2ab80b71b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2ac402f500 tx=0x7f2ac402f310 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.541+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3130932494 shutdown_connections 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.541+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3130932494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4f03390 0x55e4f52c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.541+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3130932494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4efb1b0 0x55e4f52c9850 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.541+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3130932494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f24070 0x55e4f52c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.541+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3130932494 >> 172.21.15.16:0/3130932494 conn(0x55e4f4f18460 msgr2=0x55e4f52c7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.542+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3130932494 shutdown_connections 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.542+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3130932494 wait complete. 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.542+0000 7f2ad8f8f740 1 -- start start 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2ad8f8f740 1 Processor -- start 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52d4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f24070 0x55e4f52d5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4ee29f0 0x55e4f52e95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e4f5092700 con 0x55e4f4f03390 2024-08-28T19:29:51.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e4f51b6750 con 0x55e4f4f24070 2024-08-28T19:29:51.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e4f52d7af0 con 0x55e4f4ee29f0 2024-08-28T19:29:51.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2ad8f8f740 10 client.0 _ll_register_callbacks cb 0x55e4f51e5e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:51.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2acbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52d4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2acbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52d4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44966/0 (socket says 172.21.15.16:44966) 2024-08-28T19:29:51.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2acbfff640 1 -- 172.21.15.16:0/3887125327 learned_addr learned my addr 172.21.15.16:0/3887125327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:51.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2ad0fb7640 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4ee29f0 0x55e4f52e95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.543+0000 7f2ac3fff640 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f24070 0x55e4f52d5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.544+0000 7f2acbfff640 1 -- 172.21.15.16:0/3887125327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4ee29f0 msgr2=0x55e4f52e95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.544+0000 7f2acbfff640 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4ee29f0 0x55e4f52e95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.544+0000 7f2acbfff640 1 -- 172.21.15.16:0/3887125327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f24070 msgr2=0x55e4f52d5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.544+0000 7f2acbfff640 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f24070 0x55e4f52d5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.544+0000 7f2acbfff640 1 -- 172.21.15.16:0/3887125327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4f51ef900 con 0x55e4f4f03390 2024-08-28T19:29:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.544+0000 7f2acbfff640 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52d4df0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00f780 tx=0x7f2abc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4637ca5dd8afa626 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.544+0000 7f2aab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.544+0000 7f2aab7fe640 1 -- 172.21.15.16:0/3887125327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc026070 con 0x55e4f4f03390 2024-08-28T19:29:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.544+0000 7f2aab7fe640 1 -- 172.21.15.16:0/3887125327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f2abc017830 con 0x55e4f4f03390 2024-08-28T19:29:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.544+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3887125327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2abc017c70 con 0x55e4f4f03390 2024-08-28T19:29:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.544+0000 7f2aab7fe640 1 -- 172.21.15.16:0/3887125327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f2abc032050 con 0x55e4f4f03390 2024-08-28T19:29:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.545+0000 7f2ad8f8f740 10 client.17709 fetch_fsmap learned FSMap version 50 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.545+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3887125327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55e4f52ec1d0 con 0x55e4f4f03390 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.545+0000 7f2aab7fe640 1 -- 172.21.15.16:0/3887125327 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2abc017e20 con 0x55e4f4f03390 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.545+0000 7f2aab7fe640 1 -- 172.21.15.16:0/3887125327 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 50) ==== 8394+0+0 (secure 0 0 0) 0x7f2abc022b90 con 0x55e4f4f03390 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.545+0000 7f2ad8f8f740 10 client.17709 fetch_fsmap finished waiting for FSMap version 50 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.545+0000 7f2ad8f8f740 10 client.17709 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.545+0000 7f2ad8f8f740 20 client.17709 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.545+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e4f4eb7da0 0x55e4f52ef590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.545+0000 7f2ad8f8f740 4 client.17709 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.545+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3887125327 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2abc017c70 con 0x55e4f4eb7da0 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.545+0000 7f2ac3fff640 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e4f4eb7da0 0x55e4f52ef590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.546+0000 7f2ac3fff640 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e4f4eb7da0 0x55e4f52ef590 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=206ce18b96e43962 server_cookie=f7b578d3f74d3981 in_seq=0 out_seq=0 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.546+0000 7f2aab7fe640 10 client.17709 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:29:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2aab7fe640 1 -- 172.21.15.16:0/3887125327 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f2abc017c70 con 0x55e4f4eb7da0 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2aab7fe640 10 client.17709 handle_command_reply: tid=0 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 1 client.17709 shutdown 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3887125327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 msgr2=0x55e4f52d4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52d4df0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00f780 tx=0x7f2abc0046c0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3887125327 shutdown_connections 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e4f4eb7da0 0x55e4f52ef590 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4ee29f0 0x55e4f52e95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f24070 0x55e4f52d5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2aab7fe640 0 client.17709 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3887125327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52d4df0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3887125327 >> 172.21.15.16:0/3887125327 conn(0x55e4f4f18460 msgr2=0x55e4f52cc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3887125327 shutdown_connections 2024-08-28T19:29:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3887125327 wait complete. 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.547+0000 7f2ad8f8f740 20 client.17709 trim_cache size 0 max 16384 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.560+0000 7f2ad8f8f740 1 Processor -- start 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.560+0000 7f2ad8f8f740 1 -- start start 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4eb7da0 0x55e4f52ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4ee29f0 0x55e4f52d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f03390 0x55e4f52d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc1aeb00 con 0x55e4f4eb7da0 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc1b0760 con 0x55e4f4f03390 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc1af230 con 0x55e4f4ee29f0 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2acbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4eb7da0 0x55e4f52ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2acbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4eb7da0 0x55e4f52ccdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44970/0 (socket says 172.21.15.16:44970) 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2acbfff640 1 -- 172.21.15.16:0/3932196401 learned_addr learned my addr 172.21.15.16:0/3932196401 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2ad0fb7640 1 --2- 172.21.15.16:0/3932196401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f03390 0x55e4f52d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2ac3fff640 1 --2- 172.21.15.16:0/3932196401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4ee29f0 0x55e4f52d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2acbfff640 1 -- 172.21.15.16:0/3932196401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4ee29f0 msgr2=0x55e4f52d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2acbfff640 1 --2- 172.21.15.16:0/3932196401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4ee29f0 0x55e4f52d1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2acbfff640 1 -- 172.21.15.16:0/3932196401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f03390 msgr2=0x55e4f52d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2acbfff640 1 --2- 172.21.15.16:0/3932196401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f03390 0x55e4f52d2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2acbfff640 1 -- 172.21.15.16:0/3932196401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4f52d75a0 con 0x55e4f4eb7da0 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2acbfff640 1 --2- 172.21.15.16:0/3932196401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4eb7da0 0x55e4f52ccdd0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00baa0 tx=0x7f2abc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d88648254dc352b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2ac1ffb640 1 -- 172.21.15.16:0/3932196401 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc02f070 con 0x55e4f4eb7da0 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2ac1ffb640 1 -- 172.21.15.16:0/3932196401 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2abc005b20 con 0x55e4f4eb7da0 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.561+0000 7f2ac1ffb640 1 -- 172.21.15.16:0/3932196401 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc026040 con 0x55e4f4eb7da0 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.562+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3932196401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4eb7da0 msgr2=0x55e4f52ccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.562+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3932196401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4eb7da0 0x55e4f52ccdd0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00baa0 tx=0x7f2abc004730 comp rx=0 tx=0).stop 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.562+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3932196401 shutdown_connections 2024-08-28T19:29:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.562+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3932196401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4f03390 0x55e4f52d2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.562+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3932196401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4ee29f0 0x55e4f52d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.562+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/3932196401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4eb7da0 0x55e4f52ccdd0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.562+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3932196401 >> 172.21.15.16:0/3932196401 conn(0x55e4f4f18460 msgr2=0x55e4f52d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.562+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3932196401 shutdown_connections 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.562+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/3932196401 wait complete. 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.563+0000 7f2ad8f8f740 1 -- start start 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.563+0000 7f2ad8f8f740 1 Processor -- start 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.563+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4eb7da0 0x55e4f52f3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.563+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4ee29f0 0x55e4f52f1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.563+0000 7f2ad8f8f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.563+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc1b0000 con 0x55e4f4f03390 2024-08-28T19:29:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.563+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc1af900 con 0x55e4f4ee29f0 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.563+0000 7f2ad8f8f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc1acb00 con 0x55e4f4eb7da0 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.563+0000 7f2ad8f8f740 10 client.0 _ll_register_callbacks cb 0x7f2abc00b9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.563+0000 7f2ad0fb7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ad0fb7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52f1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44980/0 (socket says 172.21.15.16:44980) 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ad0fb7640 1 -- 172.21.15.16:0/2625337043 learned_addr learned my addr 172.21.15.16:0/2625337043 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ac3fff640 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4ee29f0 0x55e4f52f1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2acbfff640 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4eb7da0 0x55e4f52f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ad0fb7640 1 -- 172.21.15.16:0/2625337043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4eb7da0 msgr2=0x55e4f52f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ad0fb7640 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4eb7da0 0x55e4f52f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ad0fb7640 1 -- 172.21.15.16:0/2625337043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4ee29f0 msgr2=0x55e4f52f1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ad0fb7640 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4ee29f0 0x55e4f52f1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ad0fb7640 1 -- 172.21.15.16:0/2625337043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4f52c13f0 con 0x55e4f4f03390 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2acbfff640 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4eb7da0 0x55e4f52f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ac3fff640 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4ee29f0 0x55e4f52f1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ad0fb7640 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52f1c70 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2ac402f910 tx=0x7f2ac403e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27894ca2b01143c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2aab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2aab7fe640 1 -- 172.21.15.16:0/2625337043 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4055060 con 0x55e4f4f03390 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/2625337043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2abc005b20 con 0x55e4f4f03390 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2aab7fe640 1 -- 172.21.15.16:0/2625337043 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f2ac4003970 con 0x55e4f4f03390 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2aab7fe640 1 -- 172.21.15.16:0/2625337043 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f2ac4030ed0 con 0x55e4f4f03390 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ad8f8f740 10 client.17715 fetch_fsmap learned FSMap version 50 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.564+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/2625337043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55e4f535ad10 con 0x55e4f4f03390 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.565+0000 7f2aab7fe640 1 -- 172.21.15.16:0/2625337043 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2abc005b20 con 0x55e4f4f03390 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.565+0000 7f2aab7fe640 1 -- 172.21.15.16:0/2625337043 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 50) ==== 8394+0+0 (secure 0 0 0) 0x7f2ac4043040 con 0x55e4f4f03390 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.565+0000 7f2ad8f8f740 10 client.17715 fetch_fsmap finished waiting for FSMap version 50 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.565+0000 7f2ad8f8f740 10 client.17715 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.565+0000 7f2ad8f8f740 20 client.17715 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.565+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e4f4f24070 0x55e4f535d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.565+0000 7f2ad8f8f740 4 client.17715 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:29:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.565+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/2625337043 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e4f52425d0 con 0x55e4f4f24070 2024-08-28T19:29:51.617 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:29:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.565+0000 7f2acbfff640 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e4f4f24070 0x55e4f535d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2acbfff640 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e4f4f24070 0x55e4f535d590 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fc1ae346a216dc42 server_cookie=54aba0f1ccf9b967 in_seq=0 out_seq=0 2024-08-28T19:29:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2aab7fe640 10 client.17715 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2aab7fe640 1 -- 172.21.15.16:0/2625337043 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e4f52425d0 con 0x55e4f4f24070 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2aab7fe640 10 client.17715 handle_command_reply: tid=0 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 1 client.17715 shutdown 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/2625337043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 msgr2=0x55e4f52f1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52f1c70 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2ac402f910 tx=0x7f2ac403e040 comp rx=0 tx=0).stop 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/2625337043 shutdown_connections 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e4f4f24070 0x55e4f535d590 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2aab7fe640 0 client.17715 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4f4f03390 0x55e4f52f1c70 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4f4ee29f0 0x55e4f52f1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 1 --2- 172.21.15.16:0/2625337043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4f4eb7da0 0x55e4f52f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/2625337043 >> 172.21.15.16:0/2625337043 conn(0x55e4f4f18460 msgr2=0x55e4f52d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/2625337043 shutdown_connections 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 1 -- 172.21.15.16:0/2625337043 wait complete. 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.566+0000 7f2ad8f8f740 20 client.17715 trim_cache size 0 max 16384 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/1284228883 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ab80b4ca0 msgr2=0x7f2ab80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ab80b4ca0 0x7f2ab80b71b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2ac402f500 tx=0x7f2ac402f310 comp rx=0 tx=0).stop 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/1284228883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2acc1092a0 msgr2=0x7f2acc103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2acc1092a0 0x7f2acc103240 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400d750 tx=0x7f2ab400dc10 comp rx=0 tx=0).stop 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/1284228883 shutdown_connections 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ab80b4ca0 0x7f2ab80b71b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2acc109bf0 0x7f2acc103780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2acc1092a0 0x7f2acc103240 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 --2- 172.21.15.16:0/1284228883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc101d60 0x7f2acc102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/1284228883 >> 172.21.15.16:0/1284228883 conn(0x7f2acc079d30 msgr2=0x7f2acc1072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/1284228883 shutdown_connections 2024-08-28T19:29:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:51.567+0000 7f2ad2a1a640 1 -- 172.21.15.16:0/1284228883 wait complete. 2024-08-28T19:29:51.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 402551 4.37 MB/sec execute 331 sec latency 2645.299 ms 2024-08-28T19:29:52.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:29:52.726 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:52.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 404612 4.40 MB/sec execute 332 sec latency 2656.522 ms 2024-08-28T19:29:53.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:52 smithi028 ceph-mon[21769]: pgmap v608: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-08-28T19:29:53.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:52 smithi115 ceph-mon[22184]: pgmap v608: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-08-28T19:29:53.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:52 smithi016 ceph-mon[18556]: pgmap v608: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-08-28T19:29:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.324+0000 7f28327d8640 1 -- 172.21.15.16:0/2995388519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f282c102820 msgr2=0x7f282c102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.324+0000 7f28327d8640 1 --2- 172.21.15.16:0/2995388519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f282c102820 0x7f282c102c30 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f281c009900 tx=0x7f281c02a3f0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.324+0000 7f28327d8640 1 -- 172.21.15.16:0/2995388519 shutdown_connections 2024-08-28T19:29:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.324+0000 7f28327d8640 1 --2- 172.21.15.16:0/2995388519 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f282c108ab0 0x7f282c10f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.324+0000 7f28327d8640 1 --2- 172.21.15.16:0/2995388519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282c108160 0x7f282c108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.324+0000 7f28327d8640 1 --2- 172.21.15.16:0/2995388519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f282c102820 0x7f282c102c30 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.324+0000 7f28327d8640 1 -- 172.21.15.16:0/2995388519 >> 172.21.15.16:0/2995388519 conn(0x7f282c0794e0 msgr2=0x7f282c0798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.324+0000 7f28327d8640 1 -- 172.21.15.16:0/2995388519 shutdown_connections 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.324+0000 7f28327d8640 1 -- 172.21.15.16:0/2995388519 wait complete. 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.324+0000 7f28327d8640 1 Processor -- start 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.324+0000 7f28327d8640 1 -- start start 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f28327d8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f282c102820 0x7f282c19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f28327d8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282c108160 0x7f282c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f28327d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f282c108ab0 0x7f282c1a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f28327d8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f282c112780 con 0x7f282c108ab0 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f28327d8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f282c112600 con 0x7f282c108160 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f28327d8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f282c112900 con 0x7f282c102820 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f2830d75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f282c108ab0 0x7f282c1a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f28237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282c108160 0x7f282c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f2830d75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f282c108ab0 0x7f282c1a1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45004/0 (socket says 172.21.15.16:45004) 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f282bfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f282c102820 0x7f282c19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f2830d75640 1 -- 172.21.15.16:0/1999381458 learned_addr learned my addr 172.21.15.16:0/1999381458 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f282bfff640 1 -- 172.21.15.16:0/1999381458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282c108160 msgr2=0x7f282c19ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f282bfff640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282c108160 0x7f282c19ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f282bfff640 1 -- 172.21.15.16:0/1999381458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f282c108ab0 msgr2=0x7f282c1a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f282bfff640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f282c108ab0 0x7f282c1a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f282bfff640 1 -- 172.21.15.16:0/1999381458 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282c1a23f0 con 0x7f282c102820 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f2830d75640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f282c108ab0 0x7f282c1a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.325+0000 7f282bfff640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f282c102820 0x7f282c19c480 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f281c02a9e0 tx=0x7f281c003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.326+0000 7f2829ffb640 1 -- 172.21.15.16:0/1999381458 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c03f070 con 0x7f282c102820 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.326+0000 7f28327d8640 1 -- 172.21.15.16:0/1999381458 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f282c1a2680 con 0x7f282c102820 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.326+0000 7f2829ffb640 1 -- 172.21.15.16:0/1999381458 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f281c031b50 con 0x7f282c102820 2024-08-28T19:29:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.326+0000 7f2829ffb640 1 -- 172.21.15.16:0/1999381458 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c035030 con 0x7f282c102820 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.326+0000 7f28327d8640 1 -- 172.21.15.16:0/1999381458 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f282c1a8e40 con 0x7f282c102820 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.329+0000 7f2838d4d740 1 Processor -- start 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.329+0000 7f2838d4d740 1 -- start start 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.329+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 0x55b6bf054310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.329+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf054850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.329+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 0x55b6bf054d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.329+0000 7f2838d4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b6befcb680 con 0x55b6bec8e390 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.329+0000 7f2838d4d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6befbac10 con 0x55b6becaf070 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.329+0000 7f2838d4d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b6befbebc0 con 0x55b6bec861b0 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f2830d75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 0x55b6bf054310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f2830d75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 0x55b6bf054310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34764/0 (socket says 172.21.15.16:34764) 2024-08-28T19:29:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f2830d75640 1 -- 172.21.15.16:0/215896126 learned_addr learned my addr 172.21.15.16:0/215896126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f28237fe640 1 --2- 172.21.15.16:0/215896126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 0x55b6bf054d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f2829ffb640 1 -- 172.21.15.16:0/1999381458 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f281c042070 con 0x7f282c102820 2024-08-28T19:29:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f2830d75640 1 -- 172.21.15.16:0/215896126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 msgr2=0x55b6bf054850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f282bfff640 1 --2- 172.21.15.16:0/215896126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf054850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f2830d75640 1 --2- 172.21.15.16:0/215896126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf054850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f2830d75640 1 -- 172.21.15.16:0/215896126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 msgr2=0x55b6bf054d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f2830d75640 1 --2- 172.21.15.16:0/215896126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 0x55b6bf054d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f2830d75640 1 -- 172.21.15.16:0/215896126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6bed1a650 con 0x55b6becaf070 2024-08-28T19:29:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f28237fe640 1 --2- 172.21.15.16:0/215896126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 0x55b6bf054d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f282bfff640 1 --2- 172.21.15.16:0/215896126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf054850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:29:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f2830d75640 1 --2- 172.21.15.16:0/215896126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 0x55b6bf054310 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2824011850 tx=0x7f2824011d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5bc6d57d46f1f94 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.330+0000 7f28227fc640 1 -- 172.21.15.16:0/215896126 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2824011330 con 0x55b6becaf070 2024-08-28T19:29:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f28227fc640 1 -- 172.21.15.16:0/215896126 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f282400fe50 con 0x55b6becaf070 2024-08-28T19:29:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f28227fc640 1 -- 172.21.15.16:0/215896126 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28240193e0 con 0x55b6becaf070 2024-08-28T19:29:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f2829ffb640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28180b4ca0 0x7f28180b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f282bfff640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28180b4ca0 0x7f28180b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f2829ffb640 1 -- 172.21.15.16:0/1999381458 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f281c105ed0 con 0x7f282c102820 2024-08-28T19:29:53.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f2838d4d740 1 -- 172.21.15.16:0/215896126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 msgr2=0x55b6bf054310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:53.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f282bfff640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28180b4ca0 0x7f28180b71b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f281c103300 tx=0x7f281c03b640 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:53.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f2838d4d740 1 --2- 172.21.15.16:0/215896126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 0x55b6bf054310 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2824011850 tx=0x7f2824011d10 comp rx=0 tx=0).stop 2024-08-28T19:29:53.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f2838d4d740 1 -- 172.21.15.16:0/215896126 shutdown_connections 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f2838d4d740 1 --2- 172.21.15.16:0/215896126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 0x55b6bf054d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f2838d4d740 1 --2- 172.21.15.16:0/215896126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf054850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f2838d4d740 1 --2- 172.21.15.16:0/215896126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 0x55b6bf054310 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.331+0000 7f2838d4d740 1 -- 172.21.15.16:0/215896126 >> 172.21.15.16:0/215896126 conn(0x55b6beca3460 msgr2=0x55b6bf052200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.332+0000 7f2838d4d740 1 -- 172.21.15.16:0/215896126 shutdown_connections 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.332+0000 7f2838d4d740 1 -- 172.21.15.16:0/215896126 wait complete. 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.332+0000 7f2838d4d740 1 -- start start 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.332+0000 7f2838d4d740 1 Processor -- start 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec861b0 0x55b6bf05f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 0x55b6bf05fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6becaf070 0x55b6bf06be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2838d4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b6bee1d700 con 0x55b6bec861b0 2024-08-28T19:29:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2838d4d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6bef41750 con 0x55b6bec8e390 2024-08-28T19:29:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2838d4d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b6bf062af0 con 0x55b6becaf070 2024-08-28T19:29:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2838d4d740 10 client.0 _ll_register_callbacks cb 0x55b6bef70e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2830d75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec861b0 0x55b6bf05f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2830d75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec861b0 0x55b6bf05f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45016/0 (socket says 172.21.15.16:45016) 2024-08-28T19:29:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2830d75640 1 -- 172.21.15.16:0/2820199130 learned_addr learned my addr 172.21.15.16:0/2820199130 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f28237fe640 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 0x55b6bf05fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2830d75640 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6becaf070 0x55b6bf06be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2830d75640 1 -- 172.21.15.16:0/2820199130 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6becaf070 msgr2=0x55b6bf06be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2830d75640 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6becaf070 0x55b6bf06be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2830d75640 1 -- 172.21.15.16:0/2820199130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 msgr2=0x55b6bf05fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2830d75640 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 0x55b6bf05fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2830d75640 1 -- 172.21.15.16:0/2820199130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6bef7a900 con 0x55b6bec861b0 2024-08-28T19:29:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f28237fe640 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 0x55b6bf05fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f2830d75640 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec861b0 0x55b6bf05f960 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2824005c40 tx=0x7f2824011500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6c6f9ac87701da server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.333+0000 7f280b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f280b7fe640 1 -- 172.21.15.16:0/2820199130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282402f060 con 0x55b6bec861b0 2024-08-28T19:29:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f280b7fe640 1 -- 172.21.15.16:0/2820199130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f2824010840 con 0x55b6bec861b0 2024-08-28T19:29:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f2838d4d740 1 -- 172.21.15.16:0/2820199130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f282400fe50 con 0x55b6bec861b0 2024-08-28T19:29:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f280b7fe640 1 -- 172.21.15.16:0/2820199130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f282401f040 con 0x55b6bec861b0 2024-08-28T19:29:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f2838d4d740 10 client.17727 fetch_fsmap learned FSMap version 50 2024-08-28T19:29:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f2838d4d740 1 -- 172.21.15.16:0/2820199130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55b6bf06e740 con 0x55b6bec861b0 2024-08-28T19:29:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f280b7fe640 1 -- 172.21.15.16:0/2820199130 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f282400fe50 con 0x55b6bec861b0 2024-08-28T19:29:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f280b7fe640 1 -- 172.21.15.16:0/2820199130 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 50) ==== 8394+0+0 (secure 0 0 0) 0x7f2824018070 con 0x55b6bec861b0 2024-08-28T19:29:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f2838d4d740 10 client.17727 fetch_fsmap finished waiting for FSMap version 50 2024-08-28T19:29:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f2838d4d740 10 client.17727 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:29:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f2838d4d740 20 client.17727 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f2838d4d740 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b6bec6d9f0 0x55b6bf071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f2838d4d740 4 client.17727 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f2838d4d740 1 -- 172.21.15.16:0/2820199130 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6befcd5d0 con 0x55b6bec6d9f0 2024-08-28T19:29:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.334+0000 7f28237fe640 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b6bec6d9f0 0x55b6bf071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.335+0000 7f280b7fe640 10 client.17727 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:29:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.335+0000 7f28237fe640 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b6bec6d9f0 0x55b6bf071bb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bb35a11a0ef3acbc server_cookie=992c23eadd260b97 in_seq=0 out_seq=0 2024-08-28T19:29:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f280b7fe640 1 -- 172.21.15.16:0/2820199130 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b6befcd5d0 con 0x55b6bec6d9f0 2024-08-28T19:29:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f280b7fe640 10 client.17727 handle_command_reply: tid=0 2024-08-28T19:29:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 1 client.17727 shutdown 2024-08-28T19:29:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 1 -- 172.21.15.16:0/2820199130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec861b0 msgr2=0x55b6bf05f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec861b0 0x55b6bf05f960 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2824005c40 tx=0x7f2824011500 comp rx=0 tx=0).stop 2024-08-28T19:29:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 1 -- 172.21.15.16:0/2820199130 shutdown_connections 2024-08-28T19:29:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b6bec6d9f0 0x55b6bf071bb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6becaf070 0x55b6bf06be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 0x55b6bf05fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 1 --2- 172.21.15.16:0/2820199130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec861b0 0x55b6bf05f960 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 1 -- 172.21.15.16:0/2820199130 >> 172.21.15.16:0/2820199130 conn(0x55b6beca3460 msgr2=0x55b6bf0574a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f280b7fe640 0 client.17727 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:29:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 1 -- 172.21.15.16:0/2820199130 shutdown_connections 2024-08-28T19:29:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 1 -- 172.21.15.16:0/2820199130 wait complete. 2024-08-28T19:29:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.336+0000 7f2838d4d740 20 client.17727 trim_cache size 0 max 16384 2024-08-28T19:29:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f2838d4d740 1 Processor -- start 2024-08-28T19:29:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f2838d4d740 1 -- start start 2024-08-28T19:29:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf061840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 0x55b6bf055a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6becaf070 0x55b6bf0517d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f2838d4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f282c1b2eb0 con 0x55b6becaf070 2024-08-28T19:29:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f2838d4d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f282c1b4b10 con 0x55b6bec8e390 2024-08-28T19:29:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f2838d4d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f282c1b35e0 con 0x55b6bec861b0 2024-08-28T19:29:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f2830d75640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf061840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f28237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 0x55b6bf055a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f2830d75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6becaf070 0x55b6bf0517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f28237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 0x55b6bf055a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34774/0 (socket says 172.21.15.16:34774) 2024-08-28T19:29:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f28237fe640 1 -- 172.21.15.16:0/3793726917 learned_addr learned my addr 172.21.15.16:0/3793726917 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f28237fe640 1 -- 172.21.15.16:0/3793726917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 msgr2=0x55b6bf061840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f28237fe640 1 --2- 172.21.15.16:0/3793726917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf061840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f28237fe640 1 -- 172.21.15.16:0/3793726917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6becaf070 msgr2=0x55b6bf0517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f28237fe640 1 --2- 172.21.15.16:0/3793726917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6becaf070 0x55b6bf0517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f28237fe640 1 -- 172.21.15.16:0/3793726917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6bf057170 con 0x55b6bec8e390 2024-08-28T19:29:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.351+0000 7f2830d75640 1 --2- 172.21.15.16:0/3793726917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6becaf070 0x55b6bf0517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.352+0000 7f28237fe640 1 --2- 172.21.15.16:0/3793726917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 0x55b6bf055a50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f28140129b0 tx=0x7f2814005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4cb2d291ad9bf14 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:53.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.352+0000 7f2821ffb640 1 -- 172.21.15.16:0/3793726917 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28140122d0 con 0x55b6bec8e390 2024-08-28T19:29:53.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.352+0000 7f2821ffb640 1 -- 172.21.15.16:0/3793726917 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f281400fd50 con 0x55b6bec8e390 2024-08-28T19:29:53.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.352+0000 7f2821ffb640 1 -- 172.21.15.16:0/3793726917 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2814027e40 con 0x55b6bec8e390 2024-08-28T19:29:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.353+0000 7f2838d4d740 1 -- 172.21.15.16:0/3793726917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 msgr2=0x55b6bf055a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.353+0000 7f2838d4d740 1 --2- 172.21.15.16:0/3793726917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 0x55b6bf055a50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f28140129b0 tx=0x7f2814005f90 comp rx=0 tx=0).stop 2024-08-28T19:29:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.353+0000 7f2838d4d740 1 -- 172.21.15.16:0/3793726917 shutdown_connections 2024-08-28T19:29:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.353+0000 7f2838d4d740 1 --2- 172.21.15.16:0/3793726917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6becaf070 0x55b6bf0517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.353+0000 7f2838d4d740 1 --2- 172.21.15.16:0/3793726917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6bec8e390 0x55b6bf055a50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.353+0000 7f2838d4d740 1 --2- 172.21.15.16:0/3793726917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf061840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.353+0000 7f2838d4d740 1 -- 172.21.15.16:0/3793726917 >> 172.21.15.16:0/3793726917 conn(0x55b6beca3460 msgr2=0x55b6bf055340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.353+0000 7f2838d4d740 1 -- 172.21.15.16:0/3793726917 shutdown_connections 2024-08-28T19:29:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.353+0000 7f2838d4d740 1 -- 172.21.15.16:0/3793726917 wait complete. 2024-08-28T19:29:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.353+0000 7f2838d4d740 1 -- start start 2024-08-28T19:29:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f2838d4d740 1 Processor -- start 2024-08-28T19:29:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf063540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 0x55b6bf063a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f2838d4d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 0x55b6bf0da370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f2838d4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f282c1b43b0 con 0x55b6bec8e390 2024-08-28T19:29:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f2838d4d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f282c1b3cb0 con 0x55b6becaf070 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f2838d4d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f282c1b0eb0 con 0x55b6bec861b0 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f2838d4d740 10 client.0 _ll_register_callbacks cb 0x7f2824010e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f28237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 0x55b6bf063a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f28237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 0x55b6bf063a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45030/0 (socket says 172.21.15.16:45030) 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f28237fe640 1 -- 172.21.15.16:0/751769449 learned_addr learned my addr 172.21.15.16:0/751769449 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f2830d75640 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 0x55b6bf0da370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.354+0000 7f2830d75640 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf063540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f28237fe640 1 -- 172.21.15.16:0/751769449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 msgr2=0x55b6bf063540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f28237fe640 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf063540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f28237fe640 1 -- 172.21.15.16:0/751769449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 msgr2=0x55b6bf0da370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f28237fe640 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 0x55b6bf0da370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f28237fe640 1 -- 172.21.15.16:0/751769449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6bf0da8e0 con 0x55b6bec8e390 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f28237fe640 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 0x55b6bf063a80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2814028000 tx=0x7f2814012630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a256e236e2e1faf6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f280b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f280b7fe640 1 -- 172.21.15.16:0/751769449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281400faa0 con 0x55b6bec8e390 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f280b7fe640 1 -- 172.21.15.16:0/751769449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f2814027470 con 0x55b6bec8e390 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f2838d4d740 1 -- 172.21.15.16:0/751769449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f281400fd50 con 0x55b6bec8e390 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f280b7fe640 1 -- 172.21.15.16:0/751769449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f2814027670 con 0x55b6bec8e390 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f2838d4d740 10 client.17733 fetch_fsmap learned FSMap version 50 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f2838d4d740 1 -- 172.21.15.16:0/751769449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55b6bf0dbb60 con 0x55b6bec8e390 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.355+0000 7f280b7fe640 1 -- 172.21.15.16:0/751769449 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f281400fd50 con 0x55b6bec8e390 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f280b7fe640 1 -- 172.21.15.16:0/751769449 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 50) ==== 8394+0+0 (secure 0 0 0) 0x7f28140055b0 con 0x55b6bec8e390 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f2838d4d740 10 client.17733 fetch_fsmap finished waiting for FSMap version 50 2024-08-28T19:29:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f2838d4d740 10 client.17733 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:29:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f2838d4d740 20 client.17733 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f2838d4d740 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b6bec42da0 0x55b6bf0de2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f2838d4d740 4 client.17733 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:29:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f2838d4d740 1 -- 172.21.15.16:0/751769449 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b6befc9070 con 0x55b6bec42da0 2024-08-28T19:29:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f2830d75640 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b6bec42da0 0x55b6bf0de2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f280b7fe640 10 client.17733 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:29:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f2830d75640 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b6bec42da0 0x55b6bf0de2f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d30fbf0a1152bfe0 server_cookie=e25b2e2b96ff5565 in_seq=0 out_seq=0 2024-08-28T19:29:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f280b7fe640 1 -- 172.21.15.16:0/751769449 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b6befc9070 con 0x55b6bec42da0 2024-08-28T19:29:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.356+0000 7f280b7fe640 10 client.17733 handle_command_reply: tid=0 2024-08-28T19:29:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 1 client.17733 shutdown 2024-08-28T19:29:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 1 -- 172.21.15.16:0/751769449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 msgr2=0x55b6bf063a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 0x55b6bf063a80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2814028000 tx=0x7f2814012630 comp rx=0 tx=0).stop 2024-08-28T19:29:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 1 -- 172.21.15.16:0/751769449 shutdown_connections 2024-08-28T19:29:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55b6bec42da0 0x55b6bf0de2f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6becaf070 0x55b6bf0da370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6bec8e390 0x55b6bf063a80 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 1 --2- 172.21.15.16:0/751769449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b6bec861b0 0x55b6bf063540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 1 -- 172.21.15.16:0/751769449 >> 172.21.15.16:0/751769449 conn(0x55b6beca3460 msgr2=0x55b6bf0617d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f280b7fe640 0 client.17733 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:29:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 1 -- 172.21.15.16:0/751769449 shutdown_connections 2024-08-28T19:29:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 1 -- 172.21.15.16:0/751769449 wait complete. 2024-08-28T19:29:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.357+0000 7f2838d4d740 20 client.17733 trim_cache size 0 max 16384 2024-08-28T19:29:53.366 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:29:53.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 -- 172.21.15.16:0/1999381458 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28180b4ca0 msgr2=0x7f28180b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:53.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28180b4ca0 0x7f28180b71b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f281c103300 tx=0x7f281c03b640 comp rx=0 tx=0).stop 2024-08-28T19:29:53.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 -- 172.21.15.16:0/1999381458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f282c102820 msgr2=0x7f282c19c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:53.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f282c102820 0x7f282c19c480 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f281c02a9e0 tx=0x7f281c003b90 comp rx=0 tx=0).stop 2024-08-28T19:29:53.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 -- 172.21.15.16:0/1999381458 shutdown_connections 2024-08-28T19:29:53.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f282c108ab0 0x7f282c1a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28180b4ca0 0x7f28180b71b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282c108160 0x7f282c19ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 --2- 172.21.15.16:0/1999381458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f282c102820 0x7f282c19c480 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:53.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 -- 172.21.15.16:0/1999381458 >> 172.21.15.16:0/1999381458 conn(0x7f282c0794e0 msgr2=0x7f282c10e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:53.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 -- 172.21.15.16:0/1999381458 shutdown_connections 2024-08-28T19:29:53.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:53.358+0000 7f28327d8640 1 -- 172.21.15.16:0/1999381458 wait complete. 2024-08-28T19:29:53.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 405607 4.39 MB/sec execute 333 sec latency 45.152 ms 2024-08-28T19:29:54.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:29:54.292 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:54.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.801+0000 7fb4ec385640 1 -- 172.21.15.16:0/3930318334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e4102880 msgr2=0x7fb4e4102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:54.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.801+0000 7fb4ec385640 1 --2- 172.21.15.16:0/3930318334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e4102880 0x7fb4e4102c90 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009900 tx=0x7fb4e002a410 comp rx=0 tx=0).stop 2024-08-28T19:29:54.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.801+0000 7fb4ec385640 1 -- 172.21.15.16:0/3930318334 shutdown_connections 2024-08-28T19:29:54.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.801+0000 7fb4ec385640 1 --2- 172.21.15.16:0/3930318334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4108ac0 0x7fb4e410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.801+0000 7fb4ec385640 1 --2- 172.21.15.16:0/3930318334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4e4108170 0x7fb4e4108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.801+0000 7fb4ec385640 1 --2- 172.21.15.16:0/3930318334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e4102880 0x7fb4e4102c90 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.801+0000 7fb4ec385640 1 -- 172.21.15.16:0/3930318334 >> 172.21.15.16:0/3930318334 conn(0x7fb4e4079500 msgr2=0x7fb4e4079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.801+0000 7fb4ec385640 1 -- 172.21.15.16:0/3930318334 shutdown_connections 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.801+0000 7fb4ec385640 1 -- 172.21.15.16:0/3930318334 wait complete. 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.801+0000 7fb4ec385640 1 Processor -- start 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.801+0000 7fb4ec385640 1 -- start start 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4ec385640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4e4102880 0x7fb4e4104020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4ec385640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e4108170 0x7fb4e4104560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4ec385640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4108ac0 0x7fb4e4104aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4ec385640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e4112740 con 0x7fb4e4108170 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4ec385640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e41125c0 con 0x7fb4e4108ac0 2024-08-28T19:29:54.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4ec385640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e41128c0 con 0x7fb4e4102880 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4e9920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e4108170 0x7fb4e4104560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4ea121640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4e4102880 0x7fb4e4104020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4e9920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e4108170 0x7fb4e4104560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34762/0 (socket says 172.21.15.16:34762) 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4e9920640 1 -- 172.21.15.16:0/767248011 learned_addr learned my addr 172.21.15.16:0/767248011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4ea922640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4108ac0 0x7fb4e4104aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4e9920640 1 -- 172.21.15.16:0/767248011 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4e4102880 msgr2=0x7fb4e4104020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4e9920640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4e4102880 0x7fb4e4104020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4e9920640 1 -- 172.21.15.16:0/767248011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4108ac0 msgr2=0x7fb4e4104aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4e9920640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4108ac0 0x7fb4e4104aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4e9920640 1 -- 172.21.15.16:0/767248011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4e41a84e0 con 0x7fb4e4108170 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.802+0000 7fb4e9920640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e4108170 0x7fb4e4104560 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800d740 tx=0x7fb4d800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.803+0000 7fb4d77fe640 1 -- 172.21.15.16:0/767248011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8017070 con 0x7fb4e4108170 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.803+0000 7fb4ec385640 1 -- 172.21.15.16:0/767248011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4e41a87d0 con 0x7fb4e4108170 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.803+0000 7fb4d77fe640 1 -- 172.21.15.16:0/767248011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4d800de50 con 0x7fb4e4108170 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.803+0000 7fb4d77fe640 1 -- 172.21.15.16:0/767248011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8004ce0 con 0x7fb4e4108170 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.803+0000 7fb4ec385640 1 -- 172.21.15.16:0/767248011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4e41a8cc0 con 0x7fb4e4108170 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.805+0000 7fb4f28fa740 1 Processor -- start 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.805+0000 7fb4f28fa740 1 -- start start 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.805+0000 7fb4d77fe640 1 -- 172.21.15.16:0/767248011 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fb4d8004e80 con 0x7fb4e4108170 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.805+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 0x563b9c7c2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.805+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c3f41b0 0x563b9c7c2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.805+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563b9c739680 con 0x563b9c3fc390 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563b9c728c10 con 0x563b9c41d070 2024-08-28T19:29:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563b9c72cbc0 con 0x563b9c3f41b0 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4e9920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4e9920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34764/0 (socket says 172.21.15.16:34764) 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4e9920640 1 -- 172.21.15.16:0/1685384389 learned_addr learned my addr 172.21.15.16:0/1685384389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4ea922640 1 --2- 172.21.15.16:0/1685384389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 0x563b9c7c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4ea121640 1 --2- 172.21.15.16:0/1685384389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c3f41b0 0x563b9c7c2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4ea922640 1 -- 172.21.15.16:0/1685384389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c3f41b0 msgr2=0x563b9c7c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4ea922640 1 --2- 172.21.15.16:0/1685384389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c3f41b0 0x563b9c7c2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4ea922640 1 -- 172.21.15.16:0/1685384389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 msgr2=0x563b9c7c2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4ea922640 1 --2- 172.21.15.16:0/1685384389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4ea922640 1 -- 172.21.15.16:0/1685384389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b9c488650 con 0x563b9c41d070 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4e9920640 1 --2- 172.21.15.16:0/1685384389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4ea922640 1 --2- 172.21.15.16:0/1685384389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 0x563b9c7c2310 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009ed0 tx=0x7fb4dc0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d48fa68e73c47b37 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.806+0000 7fb4d77fe640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4d00b4ca0 0x7fb4d00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.807+0000 7fb4ea121640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4d00b4ca0 0x7fb4d00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.807+0000 7fb4d4d9a640 1 -- 172.21.15.16:0/1685384389 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc003b70 con 0x563b9c41d070 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.807+0000 7fb4d4d9a640 1 -- 172.21.15.16:0/1685384389 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4dc015070 con 0x563b9c41d070 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.807+0000 7fb4d4d9a640 1 -- 172.21.15.16:0/1685384389 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc011730 con 0x563b9c41d070 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.807+0000 7fb4d77fe640 1 -- 172.21.15.16:0/767248011 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7fb4d80e52f0 con 0x7fb4e4108170 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4f28fa740 1 -- 172.21.15.16:0/1685384389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 msgr2=0x563b9c7c2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4ea121640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4d00b4ca0 0x7fb4d00b71b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0000c00 tx=0x7fb4e0031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/1685384389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 0x563b9c7c2310 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009ed0 tx=0x7fb4dc0107a0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4f28fa740 1 -- 172.21.15.16:0/1685384389 shutdown_connections 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/1685384389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/1685384389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c3f41b0 0x563b9c7c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/1685384389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 0x563b9c7c2310 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4f28fa740 1 -- 172.21.15.16:0/1685384389 >> 172.21.15.16:0/1685384389 conn(0x563b9c411460 msgr2=0x563b9c7c0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4f28fa740 1 -- 172.21.15.16:0/1685384389 shutdown_connections 2024-08-28T19:29:54.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4f28fa740 1 -- 172.21.15.16:0/1685384389 wait complete. 2024-08-28T19:29:54.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4f28fa740 1 -- start start 2024-08-28T19:29:54.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.808+0000 7fb4f28fa740 1 Processor -- start 2024-08-28T19:29:54.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c3f41b0 0x563b9c7cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c41d070 0x563b9c7d9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563b9c58b700 con 0x563b9c3fc390 2024-08-28T19:29:54.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563b9c6af750 con 0x563b9c3f41b0 2024-08-28T19:29:54.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563b9c7d0af0 con 0x563b9c41d070 2024-08-28T19:29:54.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4f28fa740 10 client.0 _ll_register_callbacks cb 0x563b9c6dee10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:54.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4e9920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4e9920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7cdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34774/0 (socket says 172.21.15.16:34774) 2024-08-28T19:29:54.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4e9920640 1 -- 172.21.15.16:0/3113754714 learned_addr learned my addr 172.21.15.16:0/3113754714 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:54.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4ea121640 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c3f41b0 0x563b9c7cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4ea922640 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c41d070 0x563b9c7d9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4e9920640 1 -- 172.21.15.16:0/3113754714 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c41d070 msgr2=0x563b9c7d9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4e9920640 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c41d070 0x563b9c7d9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4e9920640 1 -- 172.21.15.16:0/3113754714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c3f41b0 msgr2=0x563b9c7cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4e9920640 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c3f41b0 0x563b9c7cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4e9920640 1 -- 172.21.15.16:0/3113754714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b9c6e8900 con 0x563b9c3fc390 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4ea121640 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c3f41b0 0x563b9c7cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4e9920640 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7cdfd0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80ab310 tx=0x7fb4d80e2b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327975ccbe64ab55 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4cdffb640 1 -- 172.21.15.16:0/3113754714 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8020050 con 0x563b9c3fc390 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4cdffb640 1 -- 172.21.15.16:0/3113754714 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7fb4d80aa870 con 0x563b9c3fc390 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.809+0000 7fb4f28fa740 1 -- 172.21.15.16:0/3113754714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4dc015070 con 0x563b9c3fc390 2024-08-28T19:29:54.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4cdffb640 1 -- 172.21.15.16:0/3113754714 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fb4d80aaf60 con 0x563b9c3fc390 2024-08-28T19:29:54.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4f28fa740 10 client.17745 fetch_fsmap learned FSMap version 51 2024-08-28T19:29:54.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4f28fa740 1 -- 172.21.15.16:0/3113754714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x563b9c7dc920 con 0x563b9c3fc390 2024-08-28T19:29:54.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4cdffb640 1 -- 172.21.15.16:0/3113754714 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4dc015070 con 0x563b9c3fc390 2024-08-28T19:29:54.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4cdffb640 1 -- 172.21.15.16:0/3113754714 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7fb4d80a6f80 con 0x563b9c3fc390 2024-08-28T19:29:54.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4f28fa740 10 client.17745 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:29:54.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4f28fa740 10 client.17745 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:29:54.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4f28fa740 20 client.17745 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:54.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563b9c3db9f0 0x563b9c7dfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4f28fa740 4 client.17745 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:54.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4f28fa740 1 -- 172.21.15.16:0/3113754714 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b9c73b5d0 con 0x563b9c3db9f0 2024-08-28T19:29:54.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.810+0000 7fb4ea121640 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563b9c3db9f0 0x563b9c7dfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.811+0000 7fb4cdffb640 10 client.17745 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.811+0000 7fb4ea121640 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563b9c3db9f0 0x563b9c7dfdf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1f90333a69e62070 server_cookie=394bfac27148c1cb in_seq=0 out_seq=0 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.811+0000 7fb4cdffb640 1 -- 172.21.15.16:0/3113754714 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563b9c73b5d0 con 0x563b9c3db9f0 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.811+0000 7fb4cdffb640 10 client.17745 handle_command_reply: tid=0 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 1 client.17745 shutdown 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 1 -- 172.21.15.16:0/3113754714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 msgr2=0x563b9c7cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7cdfd0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80ab310 tx=0x7fb4d80e2b70 comp rx=0 tx=0).stop 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 1 -- 172.21.15.16:0/3113754714 shutdown_connections 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563b9c3db9f0 0x563b9c7dfdf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c41d070 0x563b9c7d9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7cdfd0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/3113754714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c3f41b0 0x563b9c7cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 1 -- 172.21.15.16:0/3113754714 >> 172.21.15.16:0/3113754714 conn(0x563b9c411460 msgr2=0x563b9c7c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:54.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4cdffb640 0 client.17745 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:29:54.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 1 -- 172.21.15.16:0/3113754714 shutdown_connections 2024-08-28T19:29:54.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 1 -- 172.21.15.16:0/3113754714 wait complete. 2024-08-28T19:29:54.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.812+0000 7fb4f28fa740 20 client.17745 trim_cache size 0 max 16384 2024-08-28T19:29:54.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.825+0000 7fb4f28fa740 1 Processor -- start 2024-08-28T19:29:54.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.825+0000 7fb4f28fa740 1 -- start start 2024-08-28T19:29:54.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.825+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c3f41b0 0x563b9c7c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.825+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.825+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 0x563b9c7cac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.825+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e41b2ea0 con 0x563b9c3fc390 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.825+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e41b4ad0 con 0x563b9c41d070 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.825+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e41b35d0 con 0x563b9c3f41b0 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.825+0000 7fb4e9920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4e9920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34776/0 (socket says 172.21.15.16:34776) 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4e9920640 1 -- 172.21.15.16:0/1329370301 learned_addr learned my addr 172.21.15.16:0/1329370301 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4ea922640 1 --2- 172.21.15.16:0/1329370301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 0x563b9c7cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4ea121640 1 --2- 172.21.15.16:0/1329370301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c3f41b0 0x563b9c7c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4e9920640 1 -- 172.21.15.16:0/1329370301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c3f41b0 msgr2=0x563b9c7c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4e9920640 1 --2- 172.21.15.16:0/1329370301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c3f41b0 0x563b9c7c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4e9920640 1 -- 172.21.15.16:0/1329370301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 msgr2=0x563b9c7cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4e9920640 1 --2- 172.21.15.16:0/1329370301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 0x563b9c7cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4e9920640 1 -- 172.21.15.16:0/1329370301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b9c7da590 con 0x563b9c3fc390 2024-08-28T19:29:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4ea922640 1 --2- 172.21.15.16:0/1329370301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 0x563b9c7cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4ea121640 1 --2- 172.21.15.16:0/1329370301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c3f41b0 0x563b9c7c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:29:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4e9920640 1 --2- 172.21.15.16:0/1329370301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c5950 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80ae040 tx=0x7fb4d8005c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7316baa8e6182e26 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4cffff640 1 -- 172.21.15.16:0/1329370301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8020070 con 0x563b9c3fc390 2024-08-28T19:29:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4cffff640 1 -- 172.21.15.16:0/1329370301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4d80b4270 con 0x563b9c3fc390 2024-08-28T19:29:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.826+0000 7fb4cffff640 1 -- 172.21.15.16:0/1329370301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d80aad80 con 0x563b9c3fc390 2024-08-28T19:29:54.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.827+0000 7fb4f28fa740 1 -- 172.21.15.16:0/1329370301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 msgr2=0x563b9c7c5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:54.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.827+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/1329370301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c5950 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80ae040 tx=0x7fb4d8005c10 comp rx=0 tx=0).stop 2024-08-28T19:29:54.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.827+0000 7fb4f28fa740 1 -- 172.21.15.16:0/1329370301 shutdown_connections 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.827+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/1329370301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c41d070 0x563b9c7cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.827+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/1329370301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7c5950 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.827+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/1329370301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c3f41b0 0x563b9c7c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.827+0000 7fb4f28fa740 1 -- 172.21.15.16:0/1329370301 >> 172.21.15.16:0/1329370301 conn(0x563b9c411460 msgr2=0x563b9c7cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.827+0000 7fb4f28fa740 1 -- 172.21.15.16:0/1329370301 shutdown_connections 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.827+0000 7fb4f28fa740 1 -- 172.21.15.16:0/1329370301 wait complete. 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.827+0000 7fb4f28fa740 1 -- start start 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.827+0000 7fb4f28fa740 1 Processor -- start 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c3f41b0 0x563b9c7d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7d2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4f28fa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c41d070 0x563b9c7d28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e41b4370 con 0x563b9c3fc390 2024-08-28T19:29:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e41b3ca0 con 0x563b9c3f41b0 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4f28fa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e41b0ea0 con 0x563b9c41d070 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4f28fa740 10 client.0 _ll_register_callbacks cb 0x7fb4d8005d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4e9920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7d2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4e9920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7d2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34778/0 (socket says 172.21.15.16:34778) 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4e9920640 1 -- 172.21.15.16:0/150283859 learned_addr learned my addr 172.21.15.16:0/150283859 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4ea922640 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c41d070 0x563b9c7d28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4ea121640 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c3f41b0 0x563b9c7d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4e9920640 1 -- 172.21.15.16:0/150283859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c41d070 msgr2=0x563b9c7d28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4e9920640 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c41d070 0x563b9c7d28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4e9920640 1 -- 172.21.15.16:0/150283859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c3f41b0 msgr2=0x563b9c7d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4e9920640 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c3f41b0 0x563b9c7d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4e9920640 1 -- 172.21.15.16:0/150283859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b9c849820 con 0x563b9c3fc390 2024-08-28T19:29:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4e9920640 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7d2370 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80ab5a0 tx=0x7fb4d80aaa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a7f2b3bfc1be0a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.828+0000 7fb4cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4cdffb640 1 -- 172.21.15.16:0/150283859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8020070 con 0x563b9c3fc390 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4cdffb640 1 -- 172.21.15.16:0/150283859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7fb4d80ce2c0 con 0x563b9c3fc390 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4f28fa740 1 -- 172.21.15.16:0/150283859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563b9c737070 con 0x563b9c3fc390 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4cdffb640 1 -- 172.21.15.16:0/150283859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fb4d80aad80 con 0x563b9c3fc390 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4f28fa740 10 client.17751 fetch_fsmap learned FSMap version 51 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4f28fa740 1 -- 172.21.15.16:0/150283859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x563b9c84a9f0 con 0x563b9c3fc390 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4cdffb640 1 -- 172.21.15.16:0/150283859 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563b9c737070 con 0x563b9c3fc390 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4cdffb640 1 -- 172.21.15.16:0/150283859 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7fb4d80d2170 con 0x563b9c3fc390 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4f28fa740 10 client.17751 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4f28fa740 10 client.17751 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4f28fa740 20 client.17751 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563b9c3b0da0 0x563b9c84cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4f28fa740 4 client.17751 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4f28fa740 1 -- 172.21.15.16:0/150283859 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563b9c7347b0 con 0x563b9c3b0da0 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.829+0000 7fb4ea121640 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563b9c3b0da0 0x563b9c84cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4cdffb640 10 client.17751 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4ea121640 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563b9c3b0da0 0x563b9c84cd40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e2937ffaf9242a1a server_cookie=1aee9dc963b81606 in_seq=0 out_seq=0 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4cdffb640 1 -- 172.21.15.16:0/150283859 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563b9c7347b0 con 0x563b9c3b0da0 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4cdffb640 10 client.17751 handle_command_reply: tid=0 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 1 client.17751 shutdown 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 1 -- 172.21.15.16:0/150283859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 msgr2=0x563b9c7d2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7d2370 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80ab5a0 tx=0x7fb4d80aaa80 comp rx=0 tx=0).stop 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 1 -- 172.21.15.16:0/150283859 shutdown_connections 2024-08-28T19:29:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x563b9c3b0da0 0x563b9c84cd40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563b9c41d070 0x563b9c7d28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b9c3fc390 0x563b9c7d2370 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 1 --2- 172.21.15.16:0/150283859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b9c3f41b0 0x563b9c7d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 1 -- 172.21.15.16:0/150283859 >> 172.21.15.16:0/150283859 conn(0x563b9c411460 msgr2=0x563b9c7ceed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4cdffb640 0 client.17751 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:29:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 1 -- 172.21.15.16:0/150283859 shutdown_connections 2024-08-28T19:29:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 1 -- 172.21.15.16:0/150283859 wait complete. 2024-08-28T19:29:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.830+0000 7fb4f28fa740 20 client.17751 trim_cache size 0 max 16384 2024-08-28T19:29:54.837 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:29:54.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 -- 172.21.15.16:0/767248011 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4d00b4ca0 msgr2=0x7fb4d00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:54.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4d00b4ca0 0x7fb4d00b71b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0000c00 tx=0x7fb4e0031400 comp rx=0 tx=0).stop 2024-08-28T19:29:54.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 -- 172.21.15.16:0/767248011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e4108170 msgr2=0x7fb4e4104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:54.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e4108170 0x7fb4e4104560 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800d740 tx=0x7fb4d800dc00 comp rx=0 tx=0).stop 2024-08-28T19:29:54.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 -- 172.21.15.16:0/767248011 shutdown_connections 2024-08-28T19:29:54.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4d00b4ca0 0x7fb4d00b71b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4108ac0 0x7fb4e4104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e4108170 0x7fb4e4104560 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 --2- 172.21.15.16:0/767248011 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4e4102880 0x7fb4e4104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 -- 172.21.15.16:0/767248011 >> 172.21.15.16:0/767248011 conn(0x7fb4e4079500 msgr2=0x7fb4e41018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 -- 172.21.15.16:0/767248011 shutdown_connections 2024-08-28T19:29:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:54.831+0000 7fb4ec385640 1 -- 172.21.15.16:0/767248011 wait complete. 2024-08-28T19:29:54.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 406092 4.39 MB/sec execute 334 sec latency 588.094 ms 2024-08-28T19:29:55.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:54 smithi016 ceph-mon[18556]: pgmap v609: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 803 KiB/s wr, 111 op/s 2024-08-28T19:29:55.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:54 smithi028 ceph-mon[21769]: pgmap v609: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 803 KiB/s wr, 111 op/s 2024-08-28T19:29:55.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:54 smithi115 ceph-mon[22184]: pgmap v609: 129 pgs: 129 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 803 KiB/s wr, 111 op/s 2024-08-28T19:29:55.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:29:55.761 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:55.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 406092 4.37 MB/sec execute 335 sec latency 1588.123 ms 2024-08-28T19:29:56.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:55 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:29:56.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:55 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:29:56.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:55 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:29:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.217+0000 7f0617a0d640 1 -- 172.21.15.16:0/705174205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0610109240 msgr2=0x7f0610109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.217+0000 7f0617a0d640 1 --2- 172.21.15.16:0/705174205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0610109240 0x7f0610109650 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f0600009900 tx=0x7f060002a410 comp rx=0 tx=0).stop 2024-08-28T19:29:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.217+0000 7f0617a0d640 1 -- 172.21.15.16:0/705174205 shutdown_connections 2024-08-28T19:29:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.217+0000 7f0617a0d640 1 --2- 172.21.15.16:0/705174205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0610109b90 0x7f06101103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.217+0000 7f0617a0d640 1 --2- 172.21.15.16:0/705174205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0610109240 0x7f0610109650 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.217+0000 7f0617a0d640 1 --2- 172.21.15.16:0/705174205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0610103900 0x7f0610103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.217+0000 7f0617a0d640 1 -- 172.21.15.16:0/705174205 >> 172.21.15.16:0/705174205 conn(0x7f061007b5f0 msgr2=0x7f061007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.217+0000 7f0617a0d640 1 -- 172.21.15.16:0/705174205 shutdown_connections 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.217+0000 7f0617a0d640 1 -- 172.21.15.16:0/705174205 wait complete. 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.218+0000 7f0617a0d640 1 Processor -- start 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.218+0000 7f0617a0d640 1 -- start start 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.218+0000 7f0617a0d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0610103900 0x7f061019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.218+0000 7f0617a0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0610109240 0x7f061019a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.218+0000 7f0617a0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0610109b90 0x7f061019fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.218+0000 7f0617a0d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0610113850 con 0x7f0610109240 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.218+0000 7f0617a0d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06101136d0 con 0x7f0610109b90 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.218+0000 7f0617a0d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f06101139d0 con 0x7f0610103900 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0614fa8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0610109240 0x7f061019a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0614fa8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0610109240 0x7f061019a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34790/0 (socket says 172.21.15.16:34790) 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0614fa8640 1 -- 172.21.15.16:0/179130544 learned_addr learned my addr 172.21.15.16:0/179130544 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f06157a9640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0610103900 0x7f061019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0615faa640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0610109b90 0x7f061019fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0614fa8640 1 -- 172.21.15.16:0/179130544 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0610103900 msgr2=0x7f061019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0614fa8640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0610103900 0x7f061019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0614fa8640 1 -- 172.21.15.16:0/179130544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0610109b90 msgr2=0x7f061019fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0614fa8640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0610109b90 0x7f061019fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0614fa8640 1 -- 172.21.15.16:0/179130544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06101a01f0 con 0x7f0610109240 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0614fa8640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0610109240 0x7f061019a820 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f060002a930 tx=0x7f0600004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f06067fc640 1 -- 172.21.15.16:0/179130544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060003f030 con 0x7f0610109240 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f06067fc640 1 -- 172.21.15.16:0/179130544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0600003aa0 con 0x7f0610109240 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0617a0d640 1 -- 172.21.15.16:0/179130544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06101a0480 con 0x7f0610109240 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f06067fc640 1 -- 172.21.15.16:0/179130544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600035070 con 0x7f0610109240 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.219+0000 7f0617a0d640 1 -- 172.21.15.16:0/179130544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06101a6c40 con 0x7f0610109240 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.222+0000 7f061df82740 1 Processor -- start 2024-08-28T19:29:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.222+0000 7f06067fc640 1 -- 172.21.15.16:0/179130544 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0600042030 con 0x7f0610109240 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.222+0000 7f061df82740 1 -- start start 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.222+0000 7f061df82740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06ae5070 0x562d06e8a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.222+0000 7f061df82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e8a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.222+0000 7f061df82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06e8ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.222+0000 7f061df82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562d06e01680 con 0x562d06ac4390 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.222+0000 7f061df82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d06df0c10 con 0x562d06abc1b0 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.222+0000 7f061df82740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562d06df4bc0 con 0x562d06ae5070 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f0614fa8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06e8ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f0614fa8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06e8ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34798/0 (socket says 172.21.15.16:34798) 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f0614fa8640 1 -- 172.21.15.16:0/3598216359 learned_addr learned my addr 172.21.15.16:0/3598216359 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f06157a9640 1 --2- 172.21.15.16:0/3598216359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e8a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f0615faa640 1 --2- 172.21.15.16:0/3598216359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06ae5070 0x562d06e8a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f0614fa8640 1 -- 172.21.15.16:0/3598216359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06ae5070 msgr2=0x562d06e8a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f0614fa8640 1 --2- 172.21.15.16:0/3598216359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06ae5070 0x562d06e8a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f0614fa8640 1 -- 172.21.15.16:0/3598216359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 msgr2=0x562d06e8a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f0614fa8640 1 --2- 172.21.15.16:0/3598216359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e8a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f0614fa8640 1 -- 172.21.15.16:0/3598216359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d06b50650 con 0x562d06ac4390 2024-08-28T19:29:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f06067fc640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05fc0b4ca0 0x7f05fc0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f0614fa8640 1 --2- 172.21.15.16:0/3598216359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06e8ad90 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0600103270 tx=0x7f060010e2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476379e18a20ef82 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f0615faa640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05fc0b4ca0 0x7f05fc0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f06067fc640 1 -- 172.21.15.16:0/179130544 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f0600105f80 con 0x7f0610109240 2024-08-28T19:29:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.223+0000 7f05fb7fe640 1 -- 172.21.15.16:0/3598216359 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06000c8210 con 0x562d06ac4390 2024-08-28T19:29:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.224+0000 7f05fb7fe640 1 -- 172.21.15.16:0/3598216359 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06000cbe60 con 0x562d06ac4390 2024-08-28T19:29:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.224+0000 7f05fb7fe640 1 -- 172.21.15.16:0/3598216359 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06000cbac0 con 0x562d06ac4390 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.224+0000 7f061df82740 1 -- 172.21.15.16:0/3598216359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 msgr2=0x562d06e8ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.224+0000 7f0615faa640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05fc0b4ca0 0x7f05fc0b71b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0608005fd0 tx=0x7f0608009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.224+0000 7f061df82740 1 --2- 172.21.15.16:0/3598216359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06e8ad90 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0600103270 tx=0x7f060010e2a0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.224+0000 7f061df82740 1 -- 172.21.15.16:0/3598216359 shutdown_connections 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.224+0000 7f061df82740 1 --2- 172.21.15.16:0/3598216359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06e8ad90 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.224+0000 7f061df82740 1 --2- 172.21.15.16:0/3598216359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e8a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.224+0000 7f061df82740 1 --2- 172.21.15.16:0/3598216359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06ae5070 0x562d06e8a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.224+0000 7f061df82740 1 -- 172.21.15.16:0/3598216359 >> 172.21.15.16:0/3598216359 conn(0x562d06ad9460 msgr2=0x562d06e88200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.224+0000 7f061df82740 1 -- 172.21.15.16:0/3598216359 shutdown_connections 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.225+0000 7f061df82740 1 -- 172.21.15.16:0/3598216359 wait complete. 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.225+0000 7f061df82740 1 -- start start 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.225+0000 7f061df82740 1 Processor -- start 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.225+0000 7f061df82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e959f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.225+0000 7f061df82740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06ac4390 0x562d06e95f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.225+0000 7f061df82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ae5070 0x562d06ea1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.225+0000 7f061df82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562d06c53700 con 0x562d06ae5070 2024-08-28T19:29:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.225+0000 7f061df82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d06d77750 con 0x562d06abc1b0 2024-08-28T19:29:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.225+0000 7f061df82740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562d06e988b0 con 0x562d06ac4390 2024-08-28T19:29:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f061df82740 10 client.0 _ll_register_callbacks cb 0x562d06da6e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f0615faa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ae5070 0x562d06ea1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f0615faa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ae5070 0x562d06ea1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34800/0 (socket says 172.21.15.16:34800) 2024-08-28T19:29:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f0615faa640 1 -- 172.21.15.16:0/1001443467 learned_addr learned my addr 172.21.15.16:0/1001443467 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f06157a9640 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f0614fa8640 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06ac4390 0x562d06e95f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f0615faa640 1 -- 172.21.15.16:0/1001443467 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06ac4390 msgr2=0x562d06e95f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f0615faa640 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06ac4390 0x562d06e95f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f0615faa640 1 -- 172.21.15.16:0/1001443467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 msgr2=0x562d06e959f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f0615faa640 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e959f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f0615faa640 1 -- 172.21.15.16:0/1001443467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d06db0900 con 0x562d06ae5070 2024-08-28T19:29:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f0615faa640 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ae5070 0x562d06ea1ad0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f0608005c40 tx=0x7f0608004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a074dd902fcc79 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f05f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f05f8ff9640 1 -- 172.21.15.16:0/1001443467 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608020620 con 0x562d06ae5070 2024-08-28T19:29:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f05f8ff9640 1 -- 172.21.15.16:0/1001443467 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f06080207c0 con 0x562d06ae5070 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f061df82740 1 -- 172.21.15.16:0/1001443467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06000cbe60 con 0x562d06ae5070 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.226+0000 7f05f8ff9640 1 -- 172.21.15.16:0/1001443467 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f0608033050 con 0x562d06ae5070 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.227+0000 7f061df82740 10 client.17763 fetch_fsmap learned FSMap version 51 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.227+0000 7f061df82740 1 -- 172.21.15.16:0/1001443467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x562d06ea4580 con 0x562d06ae5070 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.227+0000 7f05f8ff9640 1 -- 172.21.15.16:0/1001443467 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06000cbe60 con 0x562d06ae5070 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.227+0000 7f05f8ff9640 1 -- 172.21.15.16:0/1001443467 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7f060800f040 con 0x562d06ae5070 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.227+0000 7f061df82740 10 client.17763 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.227+0000 7f061df82740 10 client.17763 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.227+0000 7f061df82740 20 client.17763 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.227+0000 7f061df82740 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562d06aa39f0 0x562d06ea7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.227+0000 7f061df82740 4 client.17763 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.227+0000 7f061df82740 1 -- 172.21.15.16:0/1001443467 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d06e035d0 con 0x562d06aa39f0 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.227+0000 7f06157a9640 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562d06aa39f0 0x562d06ea7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.228+0000 7f05f8ff9640 10 client.17763 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.228+0000 7f06157a9640 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562d06aa39f0 0x562d06ea7b70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=925809dbda80cf40 server_cookie=bbe075242ad657f5 in_seq=0 out_seq=0 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f05f8ff9640 1 -- 172.21.15.16:0/1001443467 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562d06e035d0 con 0x562d06aa39f0 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f05f8ff9640 10 client.17763 handle_command_reply: tid=0 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 1 client.17763 shutdown 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 1 -- 172.21.15.16:0/1001443467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ae5070 msgr2=0x562d06ea1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ae5070 0x562d06ea1ad0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f0608005c40 tx=0x7f0608004a40 comp rx=0 tx=0).stop 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 1 -- 172.21.15.16:0/1001443467 shutdown_connections 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562d06aa39f0 0x562d06ea7b70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ae5070 0x562d06ea1ad0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f05f8ff9640 0 client.17763 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06ac4390 0x562d06e95f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 1 --2- 172.21.15.16:0/1001443467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 1 -- 172.21.15.16:0/1001443467 >> 172.21.15.16:0/1001443467 conn(0x562d06ad9460 msgr2=0x562d06e8d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:56.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 1 -- 172.21.15.16:0/1001443467 shutdown_connections 2024-08-28T19:29:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 1 -- 172.21.15.16:0/1001443467 wait complete. 2024-08-28T19:29:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.229+0000 7f061df82740 20 client.17763 trim_cache size 0 max 16384 2024-08-28T19:29:56.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.242+0000 7f061df82740 1 Processor -- start 2024-08-28T19:29:56.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.242+0000 7f061df82740 1 -- start start 2024-08-28T19:29:56.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f061df82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06aa39f0 0x562d06e8bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f061df82740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06abc1b0 0x562d06e96f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f061df82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06ea1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f061df82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06101b0cf0 con 0x562d06ac4390 2024-08-28T19:29:56.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f061df82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06101b2920 con 0x562d06aa39f0 2024-08-28T19:29:56.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f061df82740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f06101b1420 con 0x562d06abc1b0 2024-08-28T19:29:56.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f0615faa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06ea1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f0615faa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06ea1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34814/0 (socket says 172.21.15.16:34814) 2024-08-28T19:29:56.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f0615faa640 1 -- 172.21.15.16:0/2214990838 learned_addr learned my addr 172.21.15.16:0/2214990838 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:56.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f06157a9640 1 --2- 172.21.15.16:0/2214990838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06aa39f0 0x562d06e8bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f0614fa8640 1 --2- 172.21.15.16:0/2214990838 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06abc1b0 0x562d06e96f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f0615faa640 1 -- 172.21.15.16:0/2214990838 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06abc1b0 msgr2=0x562d06e96f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:56.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f0615faa640 1 --2- 172.21.15.16:0/2214990838 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06abc1b0 0x562d06e96f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f0615faa640 1 -- 172.21.15.16:0/2214990838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06aa39f0 msgr2=0x562d06e8bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:56.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f0615faa640 1 --2- 172.21.15.16:0/2214990838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06aa39f0 0x562d06e8bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f0615faa640 1 -- 172.21.15.16:0/2214990838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d06e97dc0 con 0x562d06ac4390 2024-08-28T19:29:56.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f0614fa8640 1 --2- 172.21.15.16:0/2214990838 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06abc1b0 0x562d06e96f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:29:56.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f06157a9640 1 --2- 172.21.15.16:0/2214990838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06aa39f0 0x562d06e8bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:56.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f0615faa640 1 --2- 172.21.15.16:0/2214990838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06ea1a10 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f060801dab0 tx=0x7f0608005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c4ffa00fc38add server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:56.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f05faffd640 1 -- 172.21.15.16:0/2214990838 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06080119b0 con 0x562d06ac4390 2024-08-28T19:29:56.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f05faffd640 1 -- 172.21.15.16:0/2214990838 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f060803c070 con 0x562d06ac4390 2024-08-28T19:29:56.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.243+0000 7f05faffd640 1 -- 172.21.15.16:0/2214990838 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608003e50 con 0x562d06ac4390 2024-08-28T19:29:56.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.244+0000 7f061df82740 1 -- 172.21.15.16:0/2214990838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 msgr2=0x562d06ea1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:56.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.244+0000 7f061df82740 1 --2- 172.21.15.16:0/2214990838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06ea1a10 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f060801dab0 tx=0x7f0608005c70 comp rx=0 tx=0).stop 2024-08-28T19:29:56.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.244+0000 7f061df82740 1 -- 172.21.15.16:0/2214990838 shutdown_connections 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.244+0000 7f061df82740 1 --2- 172.21.15.16:0/2214990838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06ea1a10 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.244+0000 7f061df82740 1 --2- 172.21.15.16:0/2214990838 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06abc1b0 0x562d06e96f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.244+0000 7f061df82740 1 --2- 172.21.15.16:0/2214990838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06aa39f0 0x562d06e8bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.244+0000 7f061df82740 1 -- 172.21.15.16:0/2214990838 >> 172.21.15.16:0/2214990838 conn(0x562d06ad9460 msgr2=0x562d06e8f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.244+0000 7f061df82740 1 -- 172.21.15.16:0/2214990838 shutdown_connections 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.244+0000 7f061df82740 1 -- 172.21.15.16:0/2214990838 wait complete. 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f061df82740 1 -- start start 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f061df82740 1 Processor -- start 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f061df82740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06aa39f0 0x562d06e9a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f061df82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e9a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f061df82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06f0b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f061df82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06101b21c0 con 0x562d06ac4390 2024-08-28T19:29:56.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f061df82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06101b1af0 con 0x562d06abc1b0 2024-08-28T19:29:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f061df82740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f06101aecf0 con 0x562d06aa39f0 2024-08-28T19:29:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f061df82740 10 client.0 _ll_register_callbacks cb 0x7f0608009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f0615faa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06f0b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f0615faa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06f0b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34820/0 (socket says 172.21.15.16:34820) 2024-08-28T19:29:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f0615faa640 1 -- 172.21.15.16:0/536835324 learned_addr learned my addr 172.21.15.16:0/536835324 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f0614fa8640 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e9a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.245+0000 7f06157a9640 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06aa39f0 0x562d06e9a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f0615faa640 1 -- 172.21.15.16:0/536835324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06aa39f0 msgr2=0x562d06e9a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f0615faa640 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06aa39f0 0x562d06e9a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f0615faa640 1 -- 172.21.15.16:0/536835324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 msgr2=0x562d06e9a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f0615faa640 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e9a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f0615faa640 1 -- 172.21.15.16:0/536835324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d06f0bc20 con 0x562d06ac4390 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f0614fa8640 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e9a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f06157a9640 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06aa39f0 0x562d06e9a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f0615faa640 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06f0b5b0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0608005c40 tx=0x7f060801d6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8de3b02c872b4c0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f05f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f05f8ff9640 1 -- 172.21.15.16:0/536835324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608044070 con 0x562d06ac4390 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f05f8ff9640 1 -- 172.21.15.16:0/536835324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f0608005070 con 0x562d06ac4390 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f061df82740 1 -- 172.21.15.16:0/536835324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f060803c070 con 0x562d06ac4390 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f05f8ff9640 1 -- 172.21.15.16:0/536835324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f0608050050 con 0x562d06ac4390 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f061df82740 10 client.17769 fetch_fsmap learned FSMap version 51 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.246+0000 7f061df82740 1 -- 172.21.15.16:0/536835324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x562d06f12e90 con 0x562d06ac4390 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.247+0000 7f05f8ff9640 1 -- 172.21.15.16:0/536835324 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f060803c070 con 0x562d06ac4390 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.247+0000 7f05f8ff9640 1 -- 172.21.15.16:0/536835324 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7f0608036070 con 0x562d06ac4390 2024-08-28T19:29:56.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.247+0000 7f061df82740 10 client.17769 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.247+0000 7f061df82740 10 client.17769 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.247+0000 7f061df82740 20 client.17769 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.247+0000 7f061df82740 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562d06ae5070 0x562d06f15540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.247+0000 7f061df82740 4 client.17769 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.247+0000 7f061df82740 1 -- 172.21.15.16:0/536835324 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562d06dff070 con 0x562d06ae5070 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.247+0000 7f06157a9640 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562d06ae5070 0x562d06f15540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.247+0000 7f06157a9640 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562d06ae5070 0x562d06f15540 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5a7822052024097d server_cookie=de3f5cf2aae4abc in_seq=0 out_seq=0 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.247+0000 7f05f8ff9640 10 client.17769 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f05f8ff9640 1 -- 172.21.15.16:0/536835324 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562d06dff070 con 0x562d06ae5070 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f05f8ff9640 10 client.17769 handle_command_reply: tid=0 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 1 client.17769 shutdown 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 1 -- 172.21.15.16:0/536835324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 msgr2=0x562d06f0b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06f0b5b0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0608005c40 tx=0x7f060801d6a0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 1 -- 172.21.15.16:0/536835324 shutdown_connections 2024-08-28T19:29:56.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562d06ae5070 0x562d06f15540 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d06ac4390 0x562d06f0b5b0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d06abc1b0 0x562d06e9a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f05f8ff9640 0 client.17769 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:29:56.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 1 --2- 172.21.15.16:0/536835324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d06aa39f0 0x562d06e9a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 1 -- 172.21.15.16:0/536835324 >> 172.21.15.16:0/536835324 conn(0x562d06ad9460 msgr2=0x562d06e8ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:56.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 1 -- 172.21.15.16:0/536835324 shutdown_connections 2024-08-28T19:29:56.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 1 -- 172.21.15.16:0/536835324 wait complete. 2024-08-28T19:29:56.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.248+0000 7f061df82740 20 client.17769 trim_cache size 0 max 16384 2024-08-28T19:29:56.258 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:29:56.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 -- 172.21.15.16:0/179130544 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05fc0b4ca0 msgr2=0x7f05fc0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:56.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05fc0b4ca0 0x7f05fc0b71b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0608005fd0 tx=0x7f0608009600 comp rx=0 tx=0).stop 2024-08-28T19:29:56.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 -- 172.21.15.16:0/179130544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0610109240 msgr2=0x7f061019a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:56.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0610109240 0x7f061019a820 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f060002a930 tx=0x7f0600004270 comp rx=0 tx=0).stop 2024-08-28T19:29:56.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 -- 172.21.15.16:0/179130544 shutdown_connections 2024-08-28T19:29:56.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05fc0b4ca0 0x7f05fc0b71b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0610109b90 0x7f061019fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0610109240 0x7f061019a820 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 --2- 172.21.15.16:0/179130544 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0610103900 0x7f061019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:56.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 -- 172.21.15.16:0/179130544 >> 172.21.15.16:0/179130544 conn(0x7f061007b5f0 msgr2=0x7f0610107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:56.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 -- 172.21.15.16:0/179130544 shutdown_connections 2024-08-28T19:29:56.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:56.249+0000 7f0617a0d640 1 -- 172.21.15.16:0/179130544 wait complete. 2024-08-28T19:29:56.950 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:56 smithi016 ceph-mon[18556]: pgmap v610: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 155 op/s 2024-08-28T19:29:56.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 406841 4.37 MB/sec execute 336 sec latency 2231.911 ms 2024-08-28T19:29:56.977 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:29:57.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:56 smithi028 ceph-mon[21769]: pgmap v610: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 155 op/s 2024-08-28T19:29:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:56 smithi115 ceph-mon[22184]: pgmap v610: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 155 op/s 2024-08-28T19:29:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.766+0000 7efe1b24f640 1 -- 172.21.15.16:0/284311383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14109240 msgr2=0x7efe14109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.766+0000 7efe1b24f640 1 --2- 172.21.15.16:0/284311383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14109240 0x7efe14109650 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7efe08009900 tx=0x7efe0802a410 comp rx=0 tx=0).stop 2024-08-28T19:29:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.766+0000 7efe1b24f640 1 -- 172.21.15.16:0/284311383 shutdown_connections 2024-08-28T19:29:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.766+0000 7efe1b24f640 1 --2- 172.21.15.16:0/284311383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe14109b90 0x7efe141103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.766+0000 7efe1b24f640 1 --2- 172.21.15.16:0/284311383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14109240 0x7efe14109650 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.766+0000 7efe1b24f640 1 --2- 172.21.15.16:0/284311383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe14103900 0x7efe14103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.766+0000 7efe1b24f640 1 -- 172.21.15.16:0/284311383 >> 172.21.15.16:0/284311383 conn(0x7efe1407b5f0 msgr2=0x7efe1407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.766+0000 7efe1b24f640 1 -- 172.21.15.16:0/284311383 shutdown_connections 2024-08-28T19:29:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.766+0000 7efe1b24f640 1 -- 172.21.15.16:0/284311383 wait complete. 2024-08-28T19:29:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe1b24f640 1 Processor -- start 2024-08-28T19:29:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe1b24f640 1 -- start start 2024-08-28T19:29:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe1b24f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe14103900 0x7efe1419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe1b24f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe14109240 0x7efe1419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe1b24f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14109b90 0x7efe1419fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe1b24f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe14113850 con 0x7efe14109b90 2024-08-28T19:29:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe1b24f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe141136d0 con 0x7efe14103900 2024-08-28T19:29:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe1b24f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efe141139d0 con 0x7efe14109240 2024-08-28T19:29:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe197ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14109b90 0x7efe1419fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe18feb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe14103900 0x7efe1419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe197ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14109b90 0x7efe1419fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34826/0 (socket says 172.21.15.16:34826) 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe197ec640 1 -- 172.21.15.16:0/3386487189 learned_addr learned my addr 172.21.15.16:0/3386487189 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.767+0000 7efe13fff640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe14109240 0x7efe1419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe18feb640 1 -- 172.21.15.16:0/3386487189 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe14109240 msgr2=0x7efe1419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe18feb640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe14109240 0x7efe1419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe18feb640 1 -- 172.21.15.16:0/3386487189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14109b90 msgr2=0x7efe1419fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe18feb640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14109b90 0x7efe1419fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe18feb640 1 -- 172.21.15.16:0/3386487189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe141a0190 con 0x7efe14103900 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe197ec640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14109b90 0x7efe1419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe18feb640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe14103900 0x7efe1419a220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00eb00 tx=0x7efe0c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe11ffb640 1 -- 172.21.15.16:0/3386487189 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c010430 con 0x7efe14103900 2024-08-28T19:29:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe1b24f640 1 -- 172.21.15.16:0/3386487189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe141a0480 con 0x7efe14103900 2024-08-28T19:29:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe11ffb640 1 -- 172.21.15.16:0/3386487189 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe0c010a80 con 0x7efe14103900 2024-08-28T19:29:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe11ffb640 1 -- 172.21.15.16:0/3386487189 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c017a40 con 0x7efe14103900 2024-08-28T19:29:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.768+0000 7efe1b24f640 1 -- 172.21.15.16:0/3386487189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe141a6be0 con 0x7efe14103900 2024-08-28T19:29:57.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe217c4740 1 Processor -- start 2024-08-28T19:29:57.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe217c4740 1 -- start start 2024-08-28T19:29:57.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe11ffb640 1 -- 172.21.15.16:0/3386487189 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7efe0c0105d0 con 0x7efe14103900 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe217c4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 0x556b00316310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe217c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b00316850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe217c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff50390 0x556b00316d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe217c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556b0028d680 con 0x556afff481b0 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe217c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b0027cc10 con 0x556afff50390 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe217c4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556b00280bc0 con 0x556afff71070 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe197ec640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 0x556b00316310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe18feb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b00316850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe197ec640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 0x556b00316310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:44484/0 (socket says 172.21.15.16:44484) 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe197ec640 1 -- 172.21.15.16:0/2857021851 learned_addr learned my addr 172.21.15.16:0/2857021851 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.771+0000 7efe13fff640 1 --2- 172.21.15.16:0/2857021851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff50390 0x556b00316d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efe197ec640 1 -- 172.21.15.16:0/2857021851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff50390 msgr2=0x556b00316d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efe197ec640 1 --2- 172.21.15.16:0/2857021851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff50390 0x556b00316d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efe197ec640 1 -- 172.21.15.16:0/2857021851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 msgr2=0x556b00316850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efe197ec640 1 --2- 172.21.15.16:0/2857021851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b00316850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efe197ec640 1 -- 172.21.15.16:0/2857021851 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556afffdc650 con 0x556afff71070 2024-08-28T19:29:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efe11ffb640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdfc0b4d20 0x7efdfc0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efe197ec640 1 --2- 172.21.15.16:0/2857021851 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 0x556b00316310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efe040115b0 tx=0x7efe04011a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d0d098d31a89b44 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efe18feb640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdfc0b4d20 0x7efdfc0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efdfaffd640 1 -- 172.21.15.16:0/2857021851 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0401e070 con 0x556afff71070 2024-08-28T19:29:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efdfaffd640 1 -- 172.21.15.16:0/2857021851 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe0400fb10 con 0x556afff71070 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efe11ffb640 1 -- 172.21.15.16:0/3386487189 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7efe0c0e6320 con 0x7efe14103900 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.772+0000 7efdfaffd640 1 -- 172.21.15.16:0/2857021851 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04019c00 con 0x556afff71070 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.773+0000 7efe217c4740 1 -- 172.21.15.16:0/2857021851 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 msgr2=0x556b00316310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.773+0000 7efe217c4740 1 --2- 172.21.15.16:0/2857021851 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 0x556b00316310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efe040115b0 tx=0x7efe04011a70 comp rx=0 tx=0).stop 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.773+0000 7efe18feb640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdfc0b4d20 0x7efdfc0b7230 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0e3750 tx=0x7efe0c0e9e70 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.773+0000 7efe217c4740 1 -- 172.21.15.16:0/2857021851 shutdown_connections 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.773+0000 7efe217c4740 1 --2- 172.21.15.16:0/2857021851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff50390 0x556b00316d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.773+0000 7efe217c4740 1 --2- 172.21.15.16:0/2857021851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b00316850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.773+0000 7efe217c4740 1 --2- 172.21.15.16:0/2857021851 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 0x556b00316310 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.773+0000 7efe217c4740 1 -- 172.21.15.16:0/2857021851 >> 172.21.15.16:0/2857021851 conn(0x556afff65460 msgr2=0x556b00314200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.773+0000 7efe217c4740 1 -- 172.21.15.16:0/2857021851 shutdown_connections 2024-08-28T19:29:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.773+0000 7efe217c4740 1 -- 172.21.15.16:0/2857021851 wait complete. 2024-08-28T19:29:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe217c4740 1 -- start start 2024-08-28T19:29:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe217c4740 1 Processor -- start 2024-08-28T19:29:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe217c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b00321a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe217c4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff50390 0x556b00321fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe217c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff71070 0x556b0032dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe217c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556b000df700 con 0x556afff481b0 2024-08-28T19:29:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe217c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b00203750 con 0x556afff71070 2024-08-28T19:29:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe217c4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556b00324af0 con 0x556afff50390 2024-08-28T19:29:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe217c4740 10 client.0 _ll_register_callbacks cb 0x556b00232e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe197ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b00321a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe197ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b00321a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34852/0 (socket says 172.21.15.16:34852) 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe197ec640 1 -- 172.21.15.16:0/4099600791 learned_addr learned my addr 172.21.15.16:0/4099600791 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.774+0000 7efe13fff640 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff50390 0x556b00321fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efe197ec640 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff71070 0x556b0032dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efe197ec640 1 -- 172.21.15.16:0/4099600791 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff50390 msgr2=0x556b00321fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efe197ec640 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff50390 0x556b00321fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efe197ec640 1 -- 172.21.15.16:0/4099600791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff71070 msgr2=0x556b0032dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efe197ec640 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff71070 0x556b0032dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efe197ec640 1 -- 172.21.15.16:0/4099600791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b0023c900 con 0x556afff481b0 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efe197ec640 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b00321a90 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7efe04009810 tx=0x7efe040097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5783a6a12a542f8d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efdf3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efdf3fff640 1 -- 172.21.15.16:0/4099600791 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0401e070 con 0x556afff481b0 2024-08-28T19:29:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efdf3fff640 1 -- 172.21.15.16:0/4099600791 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7efe04019b50 con 0x556afff481b0 2024-08-28T19:29:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efe217c4740 1 -- 172.21.15.16:0/4099600791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe0400fb10 con 0x556afff481b0 2024-08-28T19:29:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efdf3fff640 1 -- 172.21.15.16:0/4099600791 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7efe04019d50 con 0x556afff481b0 2024-08-28T19:29:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efe217c4740 10 client.17781 fetch_fsmap learned FSMap version 51 2024-08-28T19:29:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.775+0000 7efe217c4740 1 -- 172.21.15.16:0/4099600791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x556b003308d0 con 0x556afff481b0 2024-08-28T19:29:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.776+0000 7efdf3fff640 1 -- 172.21.15.16:0/4099600791 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe0400fb10 con 0x556afff481b0 2024-08-28T19:29:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.776+0000 7efdf3fff640 1 -- 172.21.15.16:0/4099600791 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7efe040253a0 con 0x556afff481b0 2024-08-28T19:29:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.776+0000 7efe217c4740 10 client.17781 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:29:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.776+0000 7efe217c4740 10 client.17781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.776+0000 7efe217c4740 20 client.17781 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.776+0000 7efe217c4740 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556afff2f9f0 0x556b00333c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.776+0000 7efe217c4740 4 client.17781 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.776+0000 7efe217c4740 1 -- 172.21.15.16:0/4099600791 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b0028f5d0 con 0x556afff2f9f0 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.776+0000 7efe13fff640 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556afff2f9f0 0x556b00333c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.777+0000 7efdf3fff640 10 client.17781 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.777+0000 7efe13fff640 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556afff2f9f0 0x556b00333c30 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c04c178e86125f5 server_cookie=5d36fb3e147701c8 in_seq=0 out_seq=0 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.777+0000 7efdf3fff640 1 -- 172.21.15.16:0/4099600791 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556b0028f5d0 con 0x556afff2f9f0 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.777+0000 7efdf3fff640 10 client.17781 handle_command_reply: tid=0 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.777+0000 7efe217c4740 1 client.17781 shutdown 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efe217c4740 1 -- 172.21.15.16:0/4099600791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 msgr2=0x556b00321a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efe217c4740 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b00321a90 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7efe04009810 tx=0x7efe040097a0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efe217c4740 1 -- 172.21.15.16:0/4099600791 shutdown_connections 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efe217c4740 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff71070 0x556b0032dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efe217c4740 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556afff2f9f0 0x556b00333c30 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efe217c4740 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff50390 0x556b00321fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efe217c4740 1 --2- 172.21.15.16:0/4099600791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b00321a90 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efe217c4740 1 -- 172.21.15.16:0/4099600791 >> 172.21.15.16:0/4099600791 conn(0x556afff65460 msgr2=0x556b003194f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:57.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efdf3fff640 0 client.17781 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:57.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efe217c4740 1 -- 172.21.15.16:0/4099600791 shutdown_connections 2024-08-28T19:29:57.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efe217c4740 1 -- 172.21.15.16:0/4099600791 wait complete. 2024-08-28T19:29:57.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.778+0000 7efe217c4740 20 client.17781 trim_cache size 0 max 16384 2024-08-28T19:29:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.792+0000 7efe217c4740 1 Processor -- start 2024-08-28T19:29:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.792+0000 7efe217c4740 1 -- start start 2024-08-28T19:29:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.792+0000 7efe217c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b0032dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.792+0000 7efe217c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff50390 0x556b0032e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.792+0000 7efe217c4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 0x556b00313bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.792+0000 7efe217c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe141b0d30 con 0x556afff481b0 2024-08-28T19:29:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.792+0000 7efe217c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe141b2960 con 0x556afff50390 2024-08-28T19:29:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.792+0000 7efe217c4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efe141b1460 con 0x556afff71070 2024-08-28T19:29:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe197ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b0032dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe13fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff50390 0x556b0032e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe197ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b0032dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34860/0 (socket says 172.21.15.16:34860) 2024-08-28T19:29:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe197ec640 1 -- 172.21.15.16:0/1680931621 learned_addr learned my addr 172.21.15.16:0/1680931621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe197ec640 1 --2- 172.21.15.16:0/1680931621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 0x556b00313bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe13fff640 1 -- 172.21.15.16:0/1680931621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 msgr2=0x556b00313bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe13fff640 1 --2- 172.21.15.16:0/1680931621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 0x556b00313bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe13fff640 1 -- 172.21.15.16:0/1680931621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 msgr2=0x556b0032dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe13fff640 1 --2- 172.21.15.16:0/1680931621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b0032dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe13fff640 1 -- 172.21.15.16:0/1680931621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b00324640 con 0x556afff50390 2024-08-28T19:29:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe197ec640 1 --2- 172.21.15.16:0/1680931621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b0032dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efe13fff640 1 --2- 172.21.15.16:0/1680931621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff50390 0x556b0032e220 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efe0802a930 tx=0x7efe08031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c25d6f767f0f339d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efdfa7fc640 1 -- 172.21.15.16:0/1680931621 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe08035040 con 0x556afff50390 2024-08-28T19:29:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efdfa7fc640 1 -- 172.21.15.16:0/1680931621 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe080054f0 con 0x556afff50390 2024-08-28T19:29:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.793+0000 7efdfa7fc640 1 -- 172.21.15.16:0/1680931621 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe08004c80 con 0x556afff50390 2024-08-28T19:29:57.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.794+0000 7efe217c4740 1 -- 172.21.15.16:0/1680931621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff50390 msgr2=0x556b0032e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:57.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.794+0000 7efe217c4740 1 --2- 172.21.15.16:0/1680931621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff50390 0x556b0032e220 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efe0802a930 tx=0x7efe08031400 comp rx=0 tx=0).stop 2024-08-28T19:29:57.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.794+0000 7efe217c4740 1 -- 172.21.15.16:0/1680931621 shutdown_connections 2024-08-28T19:29:57.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.794+0000 7efe217c4740 1 --2- 172.21.15.16:0/1680931621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff71070 0x556b00313bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.794+0000 7efe217c4740 1 --2- 172.21.15.16:0/1680931621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff50390 0x556b0032e220 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.794+0000 7efe217c4740 1 --2- 172.21.15.16:0/1680931621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff481b0 0x556b0032dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.794+0000 7efe217c4740 1 -- 172.21.15.16:0/1680931621 >> 172.21.15.16:0/1680931621 conn(0x556afff65460 msgr2=0x556b00320310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.794+0000 7efe217c4740 1 -- 172.21.15.16:0/1680931621 shutdown_connections 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.794+0000 7efe217c4740 1 -- 172.21.15.16:0/1680931621 wait complete. 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.795+0000 7efe217c4740 1 -- start start 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.795+0000 7efe217c4740 1 Processor -- start 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.795+0000 7efe217c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff481b0 0x556b00325d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.795+0000 7efe217c4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff50390 0x556b00326270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.795+0000 7efe217c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff71070 0x556b003267b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.795+0000 7efe217c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe141b2200 con 0x556afff71070 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.795+0000 7efe217c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe141b1b30 con 0x556afff481b0 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.795+0000 7efe217c4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efe141aed30 con 0x556afff50390 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.795+0000 7efe217c4740 10 client.0 _ll_register_callbacks cb 0x7efe0401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.795+0000 7efe197ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff481b0 0x556b00325d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe197ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff481b0 0x556b00325d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:35038/0 (socket says 172.21.15.16:35038) 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe197ec640 1 -- 172.21.15.16:0/1159410420 learned_addr learned my addr 172.21.15.16:0/1159410420 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe13fff640 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff50390 0x556b00326270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe197ec640 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff71070 0x556b003267b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe197ec640 1 -- 172.21.15.16:0/1159410420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff50390 msgr2=0x556b00326270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe197ec640 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff50390 0x556b00326270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe197ec640 1 -- 172.21.15.16:0/1159410420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff71070 msgr2=0x556b003267b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe197ec640 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff71070 0x556b003267b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe197ec640 1 -- 172.21.15.16:0/1159410420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b0039d530 con 0x556afff481b0 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe13fff640 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff50390 0x556b00326270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe197ec640 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff481b0 0x556b00325d30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efe04025ff0 tx=0x7efe04000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=949e4a37f5c0cf1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efdf3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efdf3fff640 1 -- 172.21.15.16:0/1159410420 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0401e040 con 0x556afff481b0 2024-08-28T19:29:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efdf3fff640 1 -- 172.21.15.16:0/1159410420 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7efe04025c30 con 0x556afff481b0 2024-08-28T19:29:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe217c4740 1 -- 172.21.15.16:0/1159410420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe080054f0 con 0x556afff481b0 2024-08-28T19:29:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efdf3fff640 1 -- 172.21.15.16:0/1159410420 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7efe04046050 con 0x556afff481b0 2024-08-28T19:29:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.796+0000 7efe217c4740 10 client.27311 fetch_fsmap learned FSMap version 51 2024-08-28T19:29:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.797+0000 7efe217c4740 1 -- 172.21.15.16:0/1159410420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x556b0039e790 con 0x556afff481b0 2024-08-28T19:29:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.797+0000 7efdf3fff640 1 -- 172.21.15.16:0/1159410420 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe080054f0 con 0x556afff481b0 2024-08-28T19:29:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.797+0000 7efdf3fff640 1 -- 172.21.15.16:0/1159410420 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7efe040387d0 con 0x556afff481b0 2024-08-28T19:29:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.797+0000 7efe217c4740 10 client.27311 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:29:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.797+0000 7efe217c4740 10 client.27311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:29:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.797+0000 7efe217c4740 20 client.27311 populate_metadata read hostname 'smithi016' 2024-08-28T19:29:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.797+0000 7efe217c4740 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556afff04da0 0x556b003a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:29:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.797+0000 7efe217c4740 4 client.27311 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.797+0000 7efe217c4740 1 -- 172.21.15.16:0/1159410420 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b0028b070 con 0x556afff04da0 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.797+0000 7efe13fff640 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556afff04da0 0x556b003a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.797+0000 7efe13fff640 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556afff04da0 0x556b003a0ef0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33621c760eed541d server_cookie=45f4ce95fe7bc7b7 in_seq=0 out_seq=0 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efdf3fff640 10 client.27311 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efdf3fff640 1 -- 172.21.15.16:0/1159410420 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b0028b070 con 0x556afff04da0 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efdf3fff640 10 client.27311 handle_command_reply: tid=0 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 1 client.27311 shutdown 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 1 -- 172.21.15.16:0/1159410420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff481b0 msgr2=0x556b00325d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff481b0 0x556b00325d30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efe04025ff0 tx=0x7efe04000bc0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 1 -- 172.21.15.16:0/1159410420 shutdown_connections 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afff71070 0x556b003267b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556afff04da0 0x556b003a0ef0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afff50390 0x556b00326270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efdf3fff640 0 client.27311 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:29:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 1 --2- 172.21.15.16:0/1159410420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afff481b0 0x556b00325d30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 1 -- 172.21.15.16:0/1159410420 >> 172.21.15.16:0/1159410420 conn(0x556afff65460 msgr2=0x556b00313a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 1 -- 172.21.15.16:0/1159410420 shutdown_connections 2024-08-28T19:29:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 1 -- 172.21.15.16:0/1159410420 wait complete. 2024-08-28T19:29:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.798+0000 7efe217c4740 20 client.27311 trim_cache size 0 max 16384 2024-08-28T19:29:57.805 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:29:57.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 -- 172.21.15.16:0/3386487189 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdfc0b4d20 msgr2=0x7efdfc0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:57.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdfc0b4d20 0x7efdfc0b7230 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0e3750 tx=0x7efe0c0e9e70 comp rx=0 tx=0).stop 2024-08-28T19:29:57.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 -- 172.21.15.16:0/3386487189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe14103900 msgr2=0x7efe1419a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:29:57.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe14103900 0x7efe1419a220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00eb00 tx=0x7efe0c00c2d0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 -- 172.21.15.16:0/3386487189 shutdown_connections 2024-08-28T19:29:57.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14109b90 0x7efe1419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe14109240 0x7efe1419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdfc0b4d20 0x7efdfc0b7230 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 --2- 172.21.15.16:0/3386487189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe14103900 0x7efe1419a220 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:29:57.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 -- 172.21.15.16:0/3386487189 >> 172.21.15.16:0/3386487189 conn(0x7efe1407b5f0 msgr2=0x7efe14107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:29:57.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 -- 172.21.15.16:0/3386487189 shutdown_connections 2024-08-28T19:29:57.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:29:57.799+0000 7efe1b24f640 1 -- 172.21.15.16:0/3386487189 wait complete. 2024-08-28T19:29:57.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 408337 4.39 MB/sec execute 337 sec latency 33.451 ms 2024-08-28T19:29:58.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 409534 4.39 MB/sec execute 338 sec latency 17.582 ms 2024-08-28T19:29:59.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:29:58 smithi028 ceph-mon[21769]: pgmap v611: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 778 KiB/s wr, 118 op/s 2024-08-28T19:29:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:29:58 smithi115 ceph-mon[22184]: pgmap v611: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 778 KiB/s wr, 118 op/s 2024-08-28T19:29:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:29:58 smithi016 ceph-mon[18556]: pgmap v611: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 778 KiB/s wr, 118 op/s 2024-08-28T19:29:59.476 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:29:59.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:29:59.736 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:29:59.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 409641 4.38 MB/sec execute 339 sec latency 926.250 ms 2024-08-28T19:30:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.210+0000 7f87dce0e640 1 -- 172.21.15.16:0/2489579308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d8108ab0 msgr2=0x7f87d810f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.210+0000 7f87dce0e640 1 --2- 172.21.15.16:0/2489579308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d8108ab0 0x7f87d810f2e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f87d0009ca0 tx=0x7f87d002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.210+0000 7f87dce0e640 1 -- 172.21.15.16:0/2489579308 shutdown_connections 2024-08-28T19:30:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.210+0000 7f87dce0e640 1 --2- 172.21.15.16:0/2489579308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d8108ab0 0x7f87d810f2e0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.210+0000 7f87dce0e640 1 --2- 172.21.15.16:0/2489579308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87d8108160 0x7f87d8108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.210+0000 7f87dce0e640 1 --2- 172.21.15.16:0/2489579308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d8102820 0x7f87d8102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.210+0000 7f87dce0e640 1 -- 172.21.15.16:0/2489579308 >> 172.21.15.16:0/2489579308 conn(0x7f87d80794e0 msgr2=0x7f87d80798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.210+0000 7f87dce0e640 1 -- 172.21.15.16:0/2489579308 shutdown_connections 2024-08-28T19:30:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.210+0000 7f87dce0e640 1 -- 172.21.15.16:0/2489579308 wait complete. 2024-08-28T19:30:00.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.211+0000 7f87dce0e640 1 Processor -- start 2024-08-28T19:30:00.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.211+0000 7f87dce0e640 1 -- start start 2024-08-28T19:30:00.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87dce0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d8102820 0x7f87d819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87dce0e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d8108160 0x7f87d819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87dce0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87d8108ab0 0x7f87d819fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87dce0e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87d8112710 con 0x7f87d8102820 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87dce0e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87d8112590 con 0x7f87d8108ab0 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87dce0e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f87d8112890 con 0x7f87d8108160 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87d659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d8102820 0x7f87d819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87d659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d8102820 0x7f87d819a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34872/0 (socket says 172.21.15.16:34872) 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87d659c640 1 -- 172.21.15.16:0/1278557307 learned_addr learned my addr 172.21.15.16:0/1278557307 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87cdd9b640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d8108160 0x7f87d819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87d8108ab0 0x7f87d819fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87d659c640 1 -- 172.21.15.16:0/1278557307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d8108160 msgr2=0x7f87d819a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87d659c640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d8108160 0x7f87d819a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87d659c640 1 -- 172.21.15.16:0/1278557307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87d8108ab0 msgr2=0x7f87d819fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87d659c640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87d8108ab0 0x7f87d819fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87d659c640 1 -- 172.21.15.16:0/1278557307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d81a0140 con 0x7f87d8102820 2024-08-28T19:30:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87d8108ab0 0x7f87d819fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:30:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.212+0000 7f87cdd9b640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d8108160 0x7f87d819a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.213+0000 7f87d659c640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d8102820 0x7f87d819a1d0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f87c800e9f0 tx=0x7f87c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.213+0000 7f87cffff640 1 -- 172.21.15.16:0/1278557307 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c800cd80 con 0x7f87d8102820 2024-08-28T19:30:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.213+0000 7f87cffff640 1 -- 172.21.15.16:0/1278557307 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87c80108e0 con 0x7f87d8102820 2024-08-28T19:30:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.213+0000 7f87dce0e640 1 -- 172.21.15.16:0/1278557307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d81a0430 con 0x7f87d8102820 2024-08-28T19:30:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.213+0000 7f87cffff640 1 -- 172.21.15.16:0/1278557307 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8018980 con 0x7f87d8102820 2024-08-28T19:30:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.213+0000 7f87dce0e640 1 -- 172.21.15.16:0/1278557307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d80ff310 con 0x7f87d8102820 2024-08-28T19:30:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.215+0000 7f87e3383740 1 Processor -- start 2024-08-28T19:30:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.215+0000 7f87cffff640 1 -- 172.21.15.16:0/1278557307 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f87c8018b20 con 0x7f87d8102820 2024-08-28T19:30:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87e3383740 1 -- start start 2024-08-28T19:30:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87e3383740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24e92780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87e3383740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e92cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87e3383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24e93200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87e3383740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557a24e085f0 con 0x557a24aec350 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87e3383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557a24df7b80 con 0x557a24acb670 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87e3383740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557a24dfbb30 con 0x557a24ac3490 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87d6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24e92780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87d6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24e92780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34874/0 (socket says 172.21.15.16:34874) 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87d6d9d640 1 -- 172.21.15.16:0/3784171622 learned_addr learned my addr 172.21.15.16:0/3784171622 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87d659c640 1 --2- 172.21.15.16:0/3784171622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e92cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.216+0000 7f87cdd9b640 1 --2- 172.21.15.16:0/3784171622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24e93200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87d6d9d640 1 -- 172.21.15.16:0/3784171622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 msgr2=0x557a24e92cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3784171622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e92cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87d6d9d640 1 -- 172.21.15.16:0/3784171622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 msgr2=0x557a24e93200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3784171622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24e93200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87d6d9d640 1 -- 172.21.15.16:0/3784171622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a24b57930 con 0x557a24aec350 2024-08-28T19:30:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87cffff640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87c40b4ca0 0x7f87c40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3784171622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24e92780 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f87d0009470 tx=0x7f87d00094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6079564d7a3e0475 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87d659c640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87c40b4ca0 0x7f87c40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87cffff640 1 -- 172.21.15.16:0/1278557307 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f87c80e5d40 con 0x7f87d8102820 2024-08-28T19:30:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87ccd99640 1 -- 172.21.15.16:0/3784171622 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0031cf0 con 0x557a24aec350 2024-08-28T19:30:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87ccd99640 1 -- 172.21.15.16:0/3784171622 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d0036040 con 0x557a24aec350 2024-08-28T19:30:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.217+0000 7f87ccd99640 1 -- 172.21.15.16:0/3784171622 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d00042a0 con 0x557a24aec350 2024-08-28T19:30:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.218+0000 7f87e3383740 1 -- 172.21.15.16:0/3784171622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 msgr2=0x557a24e92780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.218+0000 7f87e3383740 1 --2- 172.21.15.16:0/3784171622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24e92780 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f87d0009470 tx=0x7f87d00094a0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.218+0000 7f87d659c640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87c40b4ca0 0x7f87c40b71b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f87c80e3170 tx=0x7f87c801fad0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.218+0000 7f87e3383740 1 -- 172.21.15.16:0/3784171622 shutdown_connections 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.218+0000 7f87e3383740 1 --2- 172.21.15.16:0/3784171622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24e93200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.218+0000 7f87e3383740 1 --2- 172.21.15.16:0/3784171622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e92cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.218+0000 7f87e3383740 1 --2- 172.21.15.16:0/3784171622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24e92780 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.218+0000 7f87e3383740 1 -- 172.21.15.16:0/3784171622 >> 172.21.15.16:0/3784171622 conn(0x557a24ae0740 msgr2=0x557a24e90670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.218+0000 7f87e3383740 1 -- 172.21.15.16:0/3784171622 shutdown_connections 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.218+0000 7f87e3383740 1 -- 172.21.15.16:0/3784171622 wait complete. 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87e3383740 1 -- start start 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87e3383740 1 Processor -- start 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87e3383740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e9df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87e3383740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24acb670 0x557a24e9e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87e3383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24aec350 0x557a24eaa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87e3383740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557a24c5a890 con 0x557a24acb670 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87e3383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557a24d7e790 con 0x557a24aec350 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87e3383740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557a24ea0f00 con 0x557a24ac3490 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87e3383740 10 client.0 _ll_register_callbacks cb 0x557a24b03ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87cdd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24acb670 0x557a24e9e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87cdd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24acb670 0x557a24e9e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34888/0 (socket says 172.21.15.16:34888) 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87cdd9b640 1 -- 172.21.15.16:0/3040397731 learned_addr learned my addr 172.21.15.16:0/3040397731 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24aec350 0x557a24eaa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e9df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87cdd9b640 1 -- 172.21.15.16:0/3040397731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 msgr2=0x557a24e9df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87cdd9b640 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e9df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87cdd9b640 1 -- 172.21.15.16:0/3040397731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24aec350 msgr2=0x557a24eaa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.219+0000 7f87cdd9b640 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24aec350 0x557a24eaa1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87cdd9b640 1 -- 172.21.15.16:0/3040397731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a24db5f30 con 0x557a24acb670 2024-08-28T19:30:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e9df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:30:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87cdd9b640 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24acb670 0x557a24e9e490 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f87c0012700 tx=0x7f87c0012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817d8bb92ccc5ed8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87bdffb640 1 -- 172.21.15.16:0/3040397731 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c0010d60 con 0x557a24acb670 2024-08-28T19:30:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87e3383740 1 -- 172.21.15.16:0/3040397731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d0036040 con 0x557a24acb670 2024-08-28T19:30:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87bdffb640 1 -- 172.21.15.16:0/3040397731 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f87c0005020 con 0x557a24acb670 2024-08-28T19:30:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87bdffb640 1 -- 172.21.15.16:0/3040397731 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f87c0005220 con 0x557a24acb670 2024-08-28T19:30:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87e3383740 10 client.17799 fetch_fsmap learned FSMap version 51 2024-08-28T19:30:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87e3383740 1 -- 172.21.15.16:0/3040397731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x557a24eacde0 con 0x557a24acb670 2024-08-28T19:30:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87bdffb640 1 -- 172.21.15.16:0/3040397731 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d0036040 con 0x557a24acb670 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.220+0000 7f87bdffb640 1 -- 172.21.15.16:0/3040397731 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7f87c001c470 con 0x557a24acb670 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.221+0000 7f87e3383740 10 client.17799 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.221+0000 7f87e3383740 10 client.17799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.221+0000 7f87e3383740 20 client.17799 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.221+0000 7f87e3383740 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557a24aaacd0 0x557a24eb0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.221+0000 7f87e3383740 4 client.17799 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.221+0000 7f87e3383740 1 -- 172.21.15.16:0/3040397731 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a24e0a540 con 0x557a24aaacd0 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.221+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557a24aaacd0 0x557a24eb0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.221+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557a24aaacd0 0x557a24eb0400 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea997f3ddc7f44b3 server_cookie=6ad53c3dff78d9c0 in_seq=0 out_seq=0 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.221+0000 7f87bdffb640 10 client.17799 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87bdffb640 1 -- 172.21.15.16:0/3040397731 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557a24e0a540 con 0x557a24aaacd0 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87bdffb640 10 client.17799 handle_command_reply: tid=0 2024-08-28T19:30:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87e3383740 1 client.17799 shutdown 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87e3383740 1 -- 172.21.15.16:0/3040397731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24acb670 msgr2=0x557a24e9e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87e3383740 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24acb670 0x557a24e9e490 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f87c0012700 tx=0x7f87c0012bc0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87e3383740 1 -- 172.21.15.16:0/3040397731 shutdown_connections 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87e3383740 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24aec350 0x557a24eaa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87e3383740 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24acb670 0x557a24e9e490 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87e3383740 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557a24aaacd0 0x557a24eb0400 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87e3383740 1 --2- 172.21.15.16:0/3040397731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e9df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87bdffb640 0 client.17799 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87e3383740 1 -- 172.21.15.16:0/3040397731 >> 172.21.15.16:0/3040397731 conn(0x557a24ae0740 msgr2=0x557a24e95960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.222+0000 7f87e3383740 1 -- 172.21.15.16:0/3040397731 shutdown_connections 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.223+0000 7f87e3383740 1 -- 172.21.15.16:0/3040397731 wait complete. 2024-08-28T19:30:00.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.223+0000 7f87e3383740 20 client.17799 trim_cache size 0 max 16384 2024-08-28T19:30:00.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.240+0000 7f87e3383740 1 Processor -- start 2024-08-28T19:30:00.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.240+0000 7f87e3383740 1 -- start start 2024-08-28T19:30:00.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.240+0000 7f87e3383740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e906a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.240+0000 7f87e3383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24eaa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.240+0000 7f87e3383740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24eaa630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.240+0000 7f87e3383740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87d81b2ec0 con 0x557a24aec350 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87e3383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87d81b4af0 con 0x557a24acb670 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87e3383740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f87d81b35f0 con 0x557a24ac3490 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87d6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24eaa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87d6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24eaa630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34904/0 (socket says 172.21.15.16:34904) 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87d6d9d640 1 -- 172.21.15.16:0/1335933089 learned_addr learned my addr 172.21.15.16:0/1335933089 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87cdd9b640 1 --2- 172.21.15.16:0/1335933089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24eaa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/1335933089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e906a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87d6d9d640 1 -- 172.21.15.16:0/1335933089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 msgr2=0x557a24e906a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/1335933089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e906a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87d6d9d640 1 -- 172.21.15.16:0/1335933089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 msgr2=0x557a24eaa0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:00.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/1335933089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24eaa0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87d6d9d640 1 -- 172.21.15.16:0/1335933089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a24aaaf00 con 0x557a24aec350 2024-08-28T19:30:00.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/1335933089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24eaa630 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f87d0009bf0 tx=0x7f87d0005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f3e0bbca7f33876 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:00.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87bffff640 1 -- 172.21.15.16:0/1335933089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0003c10 con 0x557a24aec350 2024-08-28T19:30:00.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87bffff640 1 -- 172.21.15.16:0/1335933089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d0036070 con 0x557a24aec350 2024-08-28T19:30:00.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.241+0000 7f87bffff640 1 -- 172.21.15.16:0/1335933089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d00042a0 con 0x557a24aec350 2024-08-28T19:30:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.242+0000 7f87e3383740 1 -- 172.21.15.16:0/1335933089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 msgr2=0x557a24eaa630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.242+0000 7f87e3383740 1 --2- 172.21.15.16:0/1335933089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24eaa630 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f87d0009bf0 tx=0x7f87d0005c30 comp rx=0 tx=0).stop 2024-08-28T19:30:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.242+0000 7f87e3383740 1 -- 172.21.15.16:0/1335933089 shutdown_connections 2024-08-28T19:30:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.242+0000 7f87e3383740 1 --2- 172.21.15.16:0/1335933089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24aec350 0x557a24eaa630 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.242+0000 7f87e3383740 1 --2- 172.21.15.16:0/1335933089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24eaa0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.242+0000 7f87e3383740 1 --2- 172.21.15.16:0/1335933089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24ac3490 0x557a24e906a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.242+0000 7f87e3383740 1 -- 172.21.15.16:0/1335933089 >> 172.21.15.16:0/1335933089 conn(0x557a24ae0740 msgr2=0x557a24e8ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.242+0000 7f87e3383740 1 -- 172.21.15.16:0/1335933089 shutdown_connections 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.242+0000 7f87e3383740 1 -- 172.21.15.16:0/1335933089 wait complete. 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87e3383740 1 -- start start 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87e3383740 1 Processor -- start 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87e3383740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24ac3490 0x557a24ea6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87e3383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24ea2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87e3383740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24aec350 0x557a24ea2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87e3383740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87d81b4390 con 0x557a24ac3490 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87e3383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87d81b3cc0 con 0x557a24acb670 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87e3383740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f87d81b0ec0 con 0x557a24aec350 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87e3383740 10 client.0 _ll_register_callbacks cb 0x7f87c000bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87d6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24ac3490 0x557a24ea6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87d6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24ac3490 0x557a24ea6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34912/0 (socket says 172.21.15.16:34912) 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87d6d9d640 1 -- 172.21.15.16:0/3984900175 learned_addr learned my addr 172.21.15.16:0/3984900175 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87cdd9b640 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24ea2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24aec350 0x557a24ea2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87d6d9d640 1 -- 172.21.15.16:0/3984900175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24aec350 msgr2=0x557a24ea2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24aec350 0x557a24ea2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87d6d9d640 1 -- 172.21.15.16:0/3984900175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 msgr2=0x557a24ea2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24ea2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.243+0000 7f87d6d9d640 1 -- 172.21.15.16:0/3984900175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a24f19de0 con 0x557a24ac3490 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87d6d9d640 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24ac3490 0x557a24ea6b50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f87d002ade0 tx=0x7f87d0031910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a457ec4aff576472 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87bdffb640 1 -- 172.21.15.16:0/3984900175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d004a040 con 0x557a24ac3490 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87bdffb640 1 -- 172.21.15.16:0/3984900175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f87d0031af0 con 0x557a24ac3490 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87e3383740 1 -- 172.21.15.16:0/3984900175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d0036070 con 0x557a24ac3490 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87bdffb640 1 -- 172.21.15.16:0/3984900175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f87d0031cf0 con 0x557a24ac3490 2024-08-28T19:30:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87e3383740 10 client.17805 fetch_fsmap learned FSMap version 51 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87e3383740 1 -- 172.21.15.16:0/3984900175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x557a24f1af70 con 0x557a24ac3490 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87bdffb640 1 -- 172.21.15.16:0/3984900175 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d0036070 con 0x557a24ac3490 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87bdffb640 1 -- 172.21.15.16:0/3984900175 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7f87d00306c0 con 0x557a24ac3490 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87e3383740 10 client.17805 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87e3383740 10 client.17805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.244+0000 7f87e3383740 20 client.17805 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.245+0000 7f87e3383740 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557a24a80080 0x557a24f1d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.245+0000 7f87e3383740 4 client.17805 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.245+0000 7f87e3383740 1 -- 172.21.15.16:0/3984900175 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557a24e05fe0 con 0x557a24a80080 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.245+0000 7f87cdd9b640 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557a24a80080 0x557a24f1d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.245+0000 7f87cdd9b640 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557a24a80080 0x557a24f1d2c0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53d91609530897da server_cookie=661022fa4291bb33 in_seq=0 out_seq=0 2024-08-28T19:30:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.245+0000 7f87bdffb640 10 client.17805 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:00.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.246+0000 7f87bdffb640 1 -- 172.21.15.16:0/3984900175 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557a24e05fe0 con 0x557a24a80080 2024-08-28T19:30:00.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.246+0000 7f87bdffb640 10 client.17805 handle_command_reply: tid=0 2024-08-28T19:30:00.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 1 client.17805 shutdown 2024-08-28T19:30:00.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 1 -- 172.21.15.16:0/3984900175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24ac3490 msgr2=0x557a24ea6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24ac3490 0x557a24ea6b50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f87d002ade0 tx=0x7f87d0031910 comp rx=0 tx=0).stop 2024-08-28T19:30:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 1 -- 172.21.15.16:0/3984900175 shutdown_connections 2024-08-28T19:30:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557a24a80080 0x557a24f1d2c0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557a24aec350 0x557a24ea2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a24acb670 0x557a24ea2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 1 --2- 172.21.15.16:0/3984900175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a24ac3490 0x557a24ea6b50 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 1 -- 172.21.15.16:0/3984900175 >> 172.21.15.16:0/3984900175 conn(0x557a24ae0740 msgr2=0x557a24e95880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87bdffb640 0 client.17805 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 1 -- 172.21.15.16:0/3984900175 shutdown_connections 2024-08-28T19:30:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 1 -- 172.21.15.16:0/3984900175 wait complete. 2024-08-28T19:30:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.247+0000 7f87e3383740 20 client.17805 trim_cache size 0 max 16384 2024-08-28T19:30:00.254 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:30:00.254 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:30:00.254 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029", 2024-08-28T19:30:00.254 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:30:00.254 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:30:00.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 -- 172.21.15.16:0/1278557307 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87c40b4ca0 msgr2=0x7f87c40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87c40b4ca0 0x7f87c40b71b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f87c80e3170 tx=0x7f87c801fad0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 -- 172.21.15.16:0/1278557307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d8102820 msgr2=0x7f87d819a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d8102820 0x7f87d819a1d0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f87c800e9f0 tx=0x7f87c800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 -- 172.21.15.16:0/1278557307 shutdown_connections 2024-08-28T19:30:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87c40b4ca0 0x7f87c40b71b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87d8108ab0 0x7f87d819fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d8108160 0x7f87d819a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 --2- 172.21.15.16:0/1278557307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d8102820 0x7f87d819a1d0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 -- 172.21.15.16:0/1278557307 >> 172.21.15.16:0/1278557307 conn(0x7f87d80794e0 msgr2=0x7f87d806bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 -- 172.21.15.16:0/1278557307 shutdown_connections 2024-08-28T19:30:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:00.248+0000 7f87dce0e640 1 -- 172.21.15.16:0/1278557307 wait complete. 2024-08-28T19:30:00.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 410235 4.37 MB/sec execute 340 sec latency 1251.148 ms 2024-08-28T19:30:01.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: pgmap v612: 129 pgs: 129 active+clean; 278 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 837 KiB/s wr, 122 op/s 2024-08-28T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: overall HEALTH_OK 2024-08-28T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:00 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: pgmap v612: 129 pgs: 129 active+clean; 278 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 837 KiB/s wr, 122 op/s 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: overall HEALTH_OK 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:00 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: pgmap v612: 129 pgs: 129 active+clean; 278 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 837 KiB/s wr, 122 op/s 2024-08-28T19:30:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: overall HEALTH_OK 2024-08-28T19:30:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:30:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:01.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:01.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:01.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:01.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:01.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:00 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.778+0000 7f644aeef640 1 -- 172.21.15.16:0/3257472068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64441050b0 msgr2=0x7f644410d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.778+0000 7f644aeef640 1 --2- 172.21.15.16:0/3257472068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64441050b0 0x7f644410d600 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f6430009900 tx=0x7f643002a410 comp rx=0 tx=0).stop 2024-08-28T19:30:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.779+0000 7f644aeef640 1 -- 172.21.15.16:0/3257472068 shutdown_connections 2024-08-28T19:30:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.779+0000 7f644aeef640 1 --2- 172.21.15.16:0/3257472068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f644410dbc0 0x7f64441100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.779+0000 7f644aeef640 1 --2- 172.21.15.16:0/3257472068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64441050b0 0x7f644410d600 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.779+0000 7f644aeef640 1 --2- 172.21.15.16:0/3257472068 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6444104760 0x7f6444104b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.779+0000 7f644aeef640 1 -- 172.21.15.16:0/3257472068 >> 172.21.15.16:0/3257472068 conn(0x7f64440794d0 msgr2=0x7f64440798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.779+0000 7f644aeef640 1 -- 172.21.15.16:0/3257472068 shutdown_connections 2024-08-28T19:30:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.779+0000 7f644aeef640 1 -- 172.21.15.16:0/3257472068 wait complete. 2024-08-28T19:30:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.779+0000 7f644aeef640 1 Processor -- start 2024-08-28T19:30:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.779+0000 7f644aeef640 1 -- start start 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644aeef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6444104760 0x7f644419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644aeef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64441050b0 0x7f644419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644aeef640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f644410dbc0 0x7f644419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644aeef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6444113800 con 0x7f6444104760 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644aeef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6444113680 con 0x7f64441050b0 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644aeef640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6444113980 con 0x7f644410dbc0 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f6448c8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6444104760 0x7f644419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f6443fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64441050b0 0x7f644419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644948c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f644410dbc0 0x7f644419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f6448c8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6444104760 0x7f644419a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34932/0 (socket says 172.21.15.16:34932) 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f6448c8b640 1 -- 172.21.15.16:0/463094592 learned_addr learned my addr 172.21.15.16:0/463094592 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:01.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644948c640 1 -- 172.21.15.16:0/463094592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64441050b0 msgr2=0x7f644419a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644948c640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64441050b0 0x7f644419a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644948c640 1 -- 172.21.15.16:0/463094592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6444104760 msgr2=0x7f644419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644948c640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6444104760 0x7f644419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644948c640 1 -- 172.21.15.16:0/463094592 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64441a01c0 con 0x7f644410dbc0 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f6448c8b640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6444104760 0x7f644419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f6443fff640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64441050b0 0x7f644419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.780+0000 7f644948c640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f644410dbc0 0x7f644419fa90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f643c009fd0 tx=0x7f643c00b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.781+0000 7f6441ffb640 1 -- 172.21.15.16:0/463094592 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c004b20 con 0x7f644410dbc0 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.781+0000 7f644aeef640 1 -- 172.21.15.16:0/463094592 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64441a04b0 con 0x7f644410dbc0 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.781+0000 7f6441ffb640 1 -- 172.21.15.16:0/463094592 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f643c004cc0 con 0x7f644410dbc0 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.781+0000 7f6441ffb640 1 -- 172.21.15.16:0/463094592 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c005410 con 0x7f644410dbc0 2024-08-28T19:30:01.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.781+0000 7f644aeef640 1 -- 172.21.15.16:0/463094592 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6444102380 con 0x7f644410dbc0 2024-08-28T19:30:01.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.784+0000 7f6451464740 1 Processor -- start 2024-08-28T19:30:01.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.784+0000 7f6441ffb640 1 -- 172.21.15.16:0/463094592 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f643c005690 con 0x7f644410dbc0 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6451464740 1 -- start start 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6451464740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 0x55bec2d5e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6451464740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 0x55bec2d5e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6451464740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d5ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6451464740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bec2cd5630 con 0x55bec29b9070 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6451464740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bec2cc4bc0 con 0x55bec2998390 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6451464740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55bec2cc8b70 con 0x55bec29901b0 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6443fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 0x55bec2d5e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6448c8b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 0x55bec2d5e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6443fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 0x55bec2d5e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34942/0 (socket says 172.21.15.16:34942) 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6443fff640 1 -- 172.21.15.16:0/1415276972 learned_addr learned my addr 172.21.15.16:0/1415276972 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:01.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6443fff640 1 --2- 172.21.15.16:0/1415276972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d5ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6448c8b640 1 -- 172.21.15.16:0/1415276972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 msgr2=0x55bec2d5ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6448c8b640 1 --2- 172.21.15.16:0/1415276972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d5ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6448c8b640 1 -- 172.21.15.16:0/1415276972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 msgr2=0x55bec2d5e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6448c8b640 1 --2- 172.21.15.16:0/1415276972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 0x55bec2d5e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.785+0000 7f6448c8b640 1 -- 172.21.15.16:0/1415276972 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bec2a24650 con 0x55bec29901b0 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.786+0000 7f6443fff640 1 --2- 172.21.15.16:0/1415276972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 0x55bec2d5e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.786+0000 7f6441ffb640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f64340b4ca0 0x7f64340b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.786+0000 7f6448c8b640 1 --2- 172.21.15.16:0/1415276972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 0x55bec2d5e840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f64380058c0 tx=0x7f6438011950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=597b9d509f3d4ecd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.786+0000 7f6443fff640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f64340b4ca0 0x7f64340b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.786+0000 7f6441ffb640 1 -- 172.21.15.16:0/463094592 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f643c0e98a0 con 0x7f644410dbc0 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.786+0000 7f642effd640 1 -- 172.21.15.16:0/1415276972 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643801e070 con 0x55bec29901b0 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.786+0000 7f642effd640 1 -- 172.21.15.16:0/1415276972 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64380035f0 con 0x55bec29901b0 2024-08-28T19:30:01.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.786+0000 7f642effd640 1 -- 172.21.15.16:0/1415276972 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643800f7a0 con 0x55bec29901b0 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.787+0000 7f6451464740 1 -- 172.21.15.16:0/1415276972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 msgr2=0x55bec2d5e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.787+0000 7f6451464740 1 --2- 172.21.15.16:0/1415276972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 0x55bec2d5e840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f64380058c0 tx=0x7f6438011950 comp rx=0 tx=0).stop 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.787+0000 7f6443fff640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f64340b4ca0 0x7f64340b71b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6430032fd0 tx=0x7f6430040040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.787+0000 7f6451464740 1 -- 172.21.15.16:0/1415276972 shutdown_connections 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.787+0000 7f6451464740 1 --2- 172.21.15.16:0/1415276972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d5ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.787+0000 7f6451464740 1 --2- 172.21.15.16:0/1415276972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 0x55bec2d5e840 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.787+0000 7f6451464740 1 --2- 172.21.15.16:0/1415276972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 0x55bec2d5e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.787+0000 7f6451464740 1 -- 172.21.15.16:0/1415276972 >> 172.21.15.16:0/1415276972 conn(0x55bec29ad460 msgr2=0x55bec2d5c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.787+0000 7f6451464740 1 -- 172.21.15.16:0/1415276972 shutdown_connections 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.787+0000 7f6451464740 1 -- 172.21.15.16:0/1415276972 wait complete. 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.788+0000 7f6451464740 1 -- start start 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.788+0000 7f6451464740 1 Processor -- start 2024-08-28T19:30:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.788+0000 7f6451464740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 0x55bec2d69980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.788+0000 7f6451464740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d69ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.788+0000 7f6451464740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 0x55bec2d75e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.788+0000 7f6451464740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bec2b27700 con 0x55bec29b9070 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.788+0000 7f6451464740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bec2c4b700 con 0x55bec2998390 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.788+0000 7f6451464740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55bec2d6cb80 con 0x55bec29901b0 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.788+0000 7f6451464740 10 client.0 _ll_register_callbacks cb 0x55bec2c7adc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.788+0000 7f6443fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d69ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.788+0000 7f6448c8b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 0x55bec2d69980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6443fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d69ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:35118/0 (socket says 172.21.15.16:35118) 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6443fff640 1 -- 172.21.15.16:0/1508977338 learned_addr learned my addr 172.21.15.16:0/1508977338 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f644948c640 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 0x55bec2d75e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6443fff640 1 -- 172.21.15.16:0/1508977338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 msgr2=0x55bec2d69980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6443fff640 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 0x55bec2d69980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6443fff640 1 -- 172.21.15.16:0/1508977338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 msgr2=0x55bec2d75e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:01.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6443fff640 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 0x55bec2d75e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6443fff640 1 -- 172.21.15.16:0/1508977338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bec2c848b0 con 0x55bec2998390 2024-08-28T19:30:01.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6448c8b640 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 0x55bec2d69980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:01.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6443fff640 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d69ec0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f64300042d0 tx=0x7f6430000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b24142c6ec149ea server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:01.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6423fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:30:01.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6423fff640 1 -- 172.21.15.16:0/1508977338 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6430044030 con 0x55bec2998390 2024-08-28T19:30:01.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.789+0000 7f6423fff640 1 -- 172.21.15.16:0/1508977338 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f6430005250 con 0x55bec2998390 2024-08-28T19:30:01.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.790+0000 7f6451464740 1 -- 172.21.15.16:0/1508977338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64380035f0 con 0x55bec2998390 2024-08-28T19:30:01.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.790+0000 7f6423fff640 1 -- 172.21.15.16:0/1508977338 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f6430059050 con 0x55bec2998390 2024-08-28T19:30:01.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.790+0000 7f6451464740 10 client.27341 fetch_fsmap learned FSMap version 51 2024-08-28T19:30:01.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.790+0000 7f6451464740 1 -- 172.21.15.16:0/1508977338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55bec2d78850 con 0x55bec2998390 2024-08-28T19:30:01.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.790+0000 7f6423fff640 1 -- 172.21.15.16:0/1508977338 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64380035f0 con 0x55bec2998390 2024-08-28T19:30:01.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.790+0000 7f6423fff640 1 -- 172.21.15.16:0/1508977338 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7f643004a070 con 0x55bec2998390 2024-08-28T19:30:01.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.790+0000 7f6451464740 10 client.27341 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:30:01.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.791+0000 7f6451464740 10 client.27341 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.791+0000 7f6451464740 20 client.27341 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.791+0000 7f6451464740 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bec29779f0 0x55bec2d7bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.791+0000 7f6451464740 4 client.27341 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.791+0000 7f6451464740 1 -- 172.21.15.16:0/1508977338 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bec2cd7580 con 0x55bec29779f0 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.791+0000 7f6448c8b640 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bec29779f0 0x55bec2d7bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.791+0000 7f6423fff640 10 client.27341 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.791+0000 7f6448c8b640 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bec29779f0 0x55bec2d7bec0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb2c2f2c36ec358c server_cookie=5acd5d28560a0af2 in_seq=0 out_seq=0 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6423fff640 1 -- 172.21.15.16:0/1508977338 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55bec2cd7580 con 0x55bec29779f0 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6423fff640 10 client.27341 handle_command_reply: tid=0 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6451464740 1 client.27341 shutdown 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6451464740 1 -- 172.21.15.16:0/1508977338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 msgr2=0x55bec2d69ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6451464740 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d69ec0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f64300042d0 tx=0x7f6430000bc0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6451464740 1 -- 172.21.15.16:0/1508977338 shutdown_connections 2024-08-28T19:30:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6451464740 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29b9070 0x55bec2d75e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6451464740 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d69ec0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6451464740 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bec29779f0 0x55bec2d7bec0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6451464740 1 --2- 172.21.15.16:0/1508977338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29901b0 0x55bec2d69980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6451464740 1 -- 172.21.15.16:0/1508977338 >> 172.21.15.16:0/1508977338 conn(0x55bec29ad460 msgr2=0x55bec2d614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.792+0000 7f6423fff640 0 client.27341 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.793+0000 7f6451464740 1 -- 172.21.15.16:0/1508977338 shutdown_connections 2024-08-28T19:30:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.793+0000 7f6451464740 1 -- 172.21.15.16:0/1508977338 wait complete. 2024-08-28T19:30:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.793+0000 7f6451464740 20 client.27341 trim_cache size 0 max 16384 2024-08-28T19:30:01.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6451464740 1 Processor -- start 2024-08-28T19:30:01.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6451464740 1 -- start start 2024-08-28T19:30:01.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6451464740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d698f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6451464740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec2998390 0x55bec2d6b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6451464740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec29b9070 0x55bec2d75d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6451464740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f64441aeb20 con 0x55bec29901b0 2024-08-28T19:30:01.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6451464740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64441b0750 con 0x55bec29b9070 2024-08-28T19:30:01.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6451464740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f64441af250 con 0x55bec2998390 2024-08-28T19:30:01.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6448c8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6448c8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d698f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34952/0 (socket says 172.21.15.16:34952) 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6448c8b640 1 -- 172.21.15.16:0/4099306277 learned_addr learned my addr 172.21.15.16:0/4099306277 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6443fff640 1 --2- 172.21.15.16:0/4099306277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec2998390 0x55bec2d6b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f644948c640 1 --2- 172.21.15.16:0/4099306277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec29b9070 0x55bec2d75d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6448c8b640 1 -- 172.21.15.16:0/4099306277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec2998390 msgr2=0x55bec2d6b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6448c8b640 1 --2- 172.21.15.16:0/4099306277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec2998390 0x55bec2d6b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6448c8b640 1 -- 172.21.15.16:0/4099306277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec29b9070 msgr2=0x55bec2d75d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6448c8b640 1 --2- 172.21.15.16:0/4099306277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec29b9070 0x55bec2d75d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.812+0000 7f6448c8b640 1 -- 172.21.15.16:0/4099306277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bec2d688e0 con 0x55bec29901b0 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.813+0000 7f6448c8b640 1 --2- 172.21.15.16:0/4099306277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d698f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f6438011e50 tx=0x7f6438005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd2db2d5b1f68b6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.813+0000 7f642e7fc640 1 -- 172.21.15.16:0/4099306277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6438029030 con 0x55bec29901b0 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.813+0000 7f642e7fc640 1 -- 172.21.15.16:0/4099306277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f643801f040 con 0x55bec29901b0 2024-08-28T19:30:01.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.813+0000 7f642e7fc640 1 -- 172.21.15.16:0/4099306277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643801e040 con 0x55bec29901b0 2024-08-28T19:30:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.814+0000 7f6451464740 1 -- 172.21.15.16:0/4099306277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 msgr2=0x55bec2d698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.814+0000 7f6451464740 1 --2- 172.21.15.16:0/4099306277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d698f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f6438011e50 tx=0x7f6438005f90 comp rx=0 tx=0).stop 2024-08-28T19:30:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.814+0000 7f6451464740 1 -- 172.21.15.16:0/4099306277 shutdown_connections 2024-08-28T19:30:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.814+0000 7f6451464740 1 --2- 172.21.15.16:0/4099306277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec29b9070 0x55bec2d75d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.814+0000 7f6451464740 1 --2- 172.21.15.16:0/4099306277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec2998390 0x55bec2d6b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.814+0000 7f6451464740 1 --2- 172.21.15.16:0/4099306277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d698f0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.814+0000 7f6451464740 1 -- 172.21.15.16:0/4099306277 >> 172.21.15.16:0/4099306277 conn(0x55bec29ad460 msgr2=0x55bec2d6aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.814+0000 7f6451464740 1 -- 172.21.15.16:0/4099306277 shutdown_connections 2024-08-28T19:30:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.814+0000 7f6451464740 1 -- 172.21.15.16:0/4099306277 wait complete. 2024-08-28T19:30:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.814+0000 7f6451464740 1 -- start start 2024-08-28T19:30:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.814+0000 7f6451464740 1 Processor -- start 2024-08-28T19:30:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6451464740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d6dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6451464740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d6e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6451464740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29b9070 0x55bec2d6e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6451464740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f64441afff0 con 0x55bec29901b0 2024-08-28T19:30:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6451464740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64441af920 con 0x55bec2998390 2024-08-28T19:30:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6451464740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f64441acb20 con 0x55bec29b9070 2024-08-28T19:30:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6451464740 10 client.0 _ll_register_callbacks cb 0x7f6430009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6448c8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6448c8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d6dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34960/0 (socket says 172.21.15.16:34960) 2024-08-28T19:30:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6448c8b640 1 -- 172.21.15.16:0/4073880824 learned_addr learned my addr 172.21.15.16:0/4073880824 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f644948c640 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29b9070 0x55bec2d6e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6443fff640 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d6e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f644948c640 1 -- 172.21.15.16:0/4073880824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 msgr2=0x55bec2d6e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f644948c640 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d6e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f644948c640 1 -- 172.21.15.16:0/4073880824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 msgr2=0x55bec2d6dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f644948c640 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d6dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f644948c640 1 -- 172.21.15.16:0/4073880824 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bec2de5880 con 0x55bec29b9070 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.815+0000 7f6448c8b640 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d6dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.816+0000 7f644948c640 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29b9070 0x55bec2d6e780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f643c0e6bc0 tx=0x7f643c005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce48fba0fa0ca5c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.816+0000 7f6423fff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.816+0000 7f6423fff640 1 -- 172.21.15.16:0/4073880824 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c0b43d0 con 0x55bec29b9070 2024-08-28T19:30:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.816+0000 7f6423fff640 1 -- 172.21.15.16:0/4073880824 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f643c0e9390 con 0x55bec29b9070 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.816+0000 7f6451464740 1 -- 172.21.15.16:0/4073880824 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f643801f040 con 0x55bec29b9070 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.816+0000 7f6423fff640 1 -- 172.21.15.16:0/4073880824 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f643c0d6030 con 0x55bec29b9070 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.816+0000 7f6451464740 10 client.27109 fetch_fsmap learned FSMap version 51 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.816+0000 7f6451464740 1 -- 172.21.15.16:0/4073880824 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55bec2de6a70 con 0x55bec29b9070 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.816+0000 7f6423fff640 1 -- 172.21.15.16:0/4073880824 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f643801f040 con 0x55bec29b9070 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.817+0000 7f6423fff640 1 -- 172.21.15.16:0/4073880824 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7f643c0ab580 con 0x55bec29b9070 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.817+0000 7f6451464740 10 client.27109 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.817+0000 7f6451464740 10 client.27109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.817+0000 7f6451464740 20 client.27109 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.817+0000 7f6451464740 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bec294cda0 0x55bec2de92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.817+0000 7f6451464740 4 client.27109 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.817+0000 7f6451464740 1 -- 172.21.15.16:0/4073880824 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bec2cd3020 con 0x55bec294cda0 2024-08-28T19:30:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.817+0000 7f6448c8b640 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bec294cda0 0x55bec2de92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.817+0000 7f6423fff640 10 client.27109 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.817+0000 7f6448c8b640 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bec294cda0 0x55bec2de92c0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a4eb698a231d256 server_cookie=40aa591ddcd923f4 in_seq=0 out_seq=0 2024-08-28T19:30:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6423fff640 1 -- 172.21.15.16:0/4073880824 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3978 (crc 0 0 0) 0x55bec2cd3020 con 0x55bec294cda0 2024-08-28T19:30:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6423fff640 10 client.27109 handle_command_reply: tid=0 2024-08-28T19:30:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 1 client.27109 shutdown 2024-08-28T19:30:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 1 -- 172.21.15.16:0/4073880824 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29b9070 msgr2=0x55bec2d6e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29b9070 0x55bec2d6e780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f643c0e6bc0 tx=0x7f643c005c00 comp rx=0 tx=0).stop 2024-08-28T19:30:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 1 -- 172.21.15.16:0/4073880824 shutdown_connections 2024-08-28T19:30:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55bec294cda0 0x55bec2de92c0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bec29b9070 0x55bec2d6e780 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bec2998390 0x55bec2d6e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6423fff640 0 client.27109 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 1 --2- 172.21.15.16:0/4073880824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bec29901b0 0x55bec2d6dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 1 -- 172.21.15.16:0/4073880824 >> 172.21.15.16:0/4073880824 conn(0x55bec29ad460 msgr2=0x55bec2d6ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 1 -- 172.21.15.16:0/4073880824 shutdown_connections 2024-08-28T19:30:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 1 -- 172.21.15.16:0/4073880824 wait complete. 2024-08-28T19:30:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.818+0000 7f6451464740 20 client.27109 trim_cache size 0 max 16384 2024-08-28T19:30:01.825 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:30:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:30:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029", 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:30:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:30:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:01.834 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:30:01.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 -- 172.21.15.16:0/463094592 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f64340b4ca0 msgr2=0x7f64340b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:01.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f64340b4ca0 0x7f64340b71b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6430032fd0 tx=0x7f6430040040 comp rx=0 tx=0).stop 2024-08-28T19:30:01.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 -- 172.21.15.16:0/463094592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f644410dbc0 msgr2=0x7f644419fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:01.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f644410dbc0 0x7f644419fa90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f643c009fd0 tx=0x7f643c00b690 comp rx=0 tx=0).stop 2024-08-28T19:30:01.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 -- 172.21.15.16:0/463094592 shutdown_connections 2024-08-28T19:30:01.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f644410dbc0 0x7f644419fa90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f64340b4ca0 0x7f64340b71b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64441050b0 0x7f644419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 --2- 172.21.15.16:0/463094592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6444104760 0x7f644419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:01.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 -- 172.21.15.16:0/463094592 >> 172.21.15.16:0/463094592 conn(0x7f64440794d0 msgr2=0x7f6444100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:01.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 -- 172.21.15.16:0/463094592 shutdown_connections 2024-08-28T19:30:01.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:01.819+0000 7f644aeef640 1 -- 172.21.15.16:0/463094592 wait complete. 2024-08-28T19:30:01.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 411489 4.39 MB/sec execute 341 sec latency 62.681 ms 2024-08-28T19:30:02.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:01 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:02.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:01 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:02.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:01 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:01 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:01 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:02.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:01 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:02.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:01 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:01 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:01 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:02.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:01 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:02.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:01 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:01 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:01 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:02.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:01 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:02.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:01 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:01 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:01 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:01 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:01 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:01 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:01 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:01 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:01 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:01 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:02.451 INFO:tasks.cephfs.filesystem:scrub status for tag:b7a2ba77-8b1b-4e1e-92b8-8a7de0999029 - {'path': '/', 'tag': 'b7a2ba77-8b1b-4e1e-92b8-8a7de0999029', 'options': 'recursive,force'} 2024-08-28T19:30:02.625 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 12.4 2024-08-28T19:30:02.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:30:02.893 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:02.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 412560 4.39 MB/sec execute 342 sec latency 16.315 ms 2024-08-28T19:30:03.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:02 smithi028 ceph-mon[21769]: pgmap v613: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 169 op/s 2024-08-28T19:30:03.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:02 smithi115 ceph-mon[22184]: pgmap v613: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 169 op/s 2024-08-28T19:30:03.136 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:02 smithi016 ceph-mon[18556]: pgmap v613: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 169 op/s 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.371+0000 7f07d83ed640 1 -- 172.21.15.16:0/1069075095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0102370 msgr2=0x7f07d0102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.371+0000 7f07d83ed640 1 --2- 172.21.15.16:0/1069075095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0102370 0x7f07d0102780 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009900 tx=0x7f07cc02a480 comp rx=0 tx=0).stop 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.371+0000 7f07d83ed640 1 -- 172.21.15.16:0/1069075095 shutdown_connections 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.371+0000 7f07d83ed640 1 --2- 172.21.15.16:0/1069075095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d010ff20 0x7f07d0112340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.371+0000 7f07d83ed640 1 --2- 172.21.15.16:0/1069075095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0102cc0 0x7f07d010f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.371+0000 7f07d83ed640 1 --2- 172.21.15.16:0/1069075095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0102370 0x7f07d0102780 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.371+0000 7f07d83ed640 1 -- 172.21.15.16:0/1069075095 >> 172.21.15.16:0/1069075095 conn(0x7f07d00794d0 msgr2=0x7f07d00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.371+0000 7f07d83ed640 1 -- 172.21.15.16:0/1069075095 shutdown_connections 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.371+0000 7f07d83ed640 1 -- 172.21.15.16:0/1069075095 wait complete. 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d83ed640 1 Processor -- start 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d83ed640 1 -- start start 2024-08-28T19:30:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d83ed640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0102370 0x7f07d019c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d83ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102cc0 0x7f07d019c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d83ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d010ff20 0x7f07d01a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d83ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07d0115a10 con 0x7f07d010ff20 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d83ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07d0115890 con 0x7f07d0102cc0 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d83ed640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f07d0115b90 con 0x7f07d0102370 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d5988640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102cc0 0x7f07d019c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d5988640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102cc0 0x7f07d019c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:35166/0 (socket says 172.21.15.16:35166) 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d5988640 1 -- 172.21.15.16:0/1686400802 learned_addr learned my addr 172.21.15.16:0/1686400802 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d6189640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0102370 0x7f07d019c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.372+0000 7f07d698a640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d010ff20 0x7f07d01a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07d5988640 1 -- 172.21.15.16:0/1686400802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0102370 msgr2=0x7f07d019c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07d5988640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0102370 0x7f07d019c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07d5988640 1 -- 172.21.15.16:0/1686400802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d010ff20 msgr2=0x7f07d01a1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07d5988640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d010ff20 0x7f07d01a1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07d5988640 1 -- 172.21.15.16:0/1686400802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d01a2370 con 0x7f07d0102cc0 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07d6189640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0102370 0x7f07d019c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07d698a640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d010ff20 0x7f07d01a1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07d5988640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102cc0 0x7f07d019c9a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f07c000e9f0 tx=0x7f07c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07c77fe640 1 -- 172.21.15.16:0/1686400802 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c000cd80 con 0x7f07d0102cc0 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07d83ed640 1 -- 172.21.15.16:0/1686400802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d01a2660 con 0x7f07d0102cc0 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07c77fe640 1 -- 172.21.15.16:0/1686400802 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07c00108e0 con 0x7f07d0102cc0 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07d83ed640 1 -- 172.21.15.16:0/1686400802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d01a8dc0 con 0x7f07d0102cc0 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.373+0000 7f07c77fe640 1 -- 172.21.15.16:0/1686400802 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c0018980 con 0x7f07d0102cc0 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.374+0000 7f07d83ed640 1 -- 172.21.15.16:0/1686400802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d0102780 con 0x7f07d0102cc0 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.376+0000 7f07c77fe640 1 -- 172.21.15.16:0/1686400802 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f07c0010430 con 0x7f07d0102cc0 2024-08-28T19:30:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.377+0000 7f07c77fe640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4ca0 0x7f07bc0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.377+0000 7f07d6189640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4ca0 0x7f07bc0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.377+0000 7f07c77fe640 1 -- 172.21.15.16:0/1686400802 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9490+0+0 (secure 0 0 0) 0x7f07c00e6130 con 0x7f07d0102cc0 2024-08-28T19:30:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.377+0000 7f07d6189640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4ca0 0x7f07bc0b71b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f07cc031440 tx=0x7f07cc032ed0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.380+0000 7f07c77fe640 1 -- 172.21.15.16:0/1686400802 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f07c00ac4a0 con 0x7f07d0102cc0 2024-08-28T19:30:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.605+0000 7f07d83ed640 1 -- 172.21.15.16:0/1686400802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07d010b0e0 con 0x7f07d0102cc0 2024-08-28T19:30:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.606+0000 7f07c77fe640 1 -- 172.21.15.16:0/1686400802 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+11985 (secure 0 0 0) 0x7f07c00b0c60 con 0x7f07d0102cc0 2024-08-28T19:30:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:30:03.608 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":51,"btime":"2024-08-28T19:29:54:768698+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":51,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:29:54.537542+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:30:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 51 2024-08-28T19:30:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.608+0000 7f07d83ed640 1 -- 172.21.15.16:0/1686400802 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4ca0 msgr2=0x7f07bc0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.608+0000 7f07d83ed640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4ca0 0x7f07bc0b71b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f07cc031440 tx=0x7f07cc032ed0 comp rx=0 tx=0).stop 2024-08-28T19:30:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.608+0000 7f07d83ed640 1 -- 172.21.15.16:0/1686400802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102cc0 msgr2=0x7f07d019c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.608+0000 7f07d83ed640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102cc0 0x7f07d019c9a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f07c000e9f0 tx=0x7f07c000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:30:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.608+0000 7f07d83ed640 1 -- 172.21.15.16:0/1686400802 shutdown_connections 2024-08-28T19:30:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.608+0000 7f07d83ed640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d010ff20 0x7f07d01a1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.608+0000 7f07d83ed640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4ca0 0x7f07bc0b71b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.608+0000 7f07d83ed640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102cc0 0x7f07d019c9a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.608+0000 7f07d83ed640 1 --2- 172.21.15.16:0/1686400802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0102370 0x7f07d019c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.608+0000 7f07d83ed640 1 -- 172.21.15.16:0/1686400802 >> 172.21.15.16:0/1686400802 conn(0x7f07d00794d0 msgr2=0x7f07d01113a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.609+0000 7f07d83ed640 1 -- 172.21.15.16:0/1686400802 shutdown_connections 2024-08-28T19:30:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:03.609+0000 7f07d83ed640 1 -- 172.21.15.16:0/1686400802 wait complete. 2024-08-28T19:30:03.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413183 4.38 MB/sec execute 343 sec latency 578.668 ms 2024-08-28T19:30:04.019 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:03 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1686400802' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:04.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:03 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1686400802' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:04.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:03 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1686400802' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:04.389 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 132.4 --await-for 45.0' 2024-08-28T19:30:04.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 132.4 --await-for 45.0 2024-08-28T19:30:04.642 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:04.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413183 4.37 MB/sec execute 344 sec latency 1578.724 ms 2024-08-28T19:30:05.030 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:04 smithi016 ceph-mon[18556]: pgmap v614: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1017 KiB/s wr, 143 op/s 2024-08-28T19:30:05.030 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:04 smithi016 ceph-mon[18556]: osdmap e89: 12 total, 12 up, 12 in 2024-08-28T19:30:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:04 smithi028 ceph-mon[21769]: pgmap v614: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1017 KiB/s wr, 143 op/s 2024-08-28T19:30:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:04 smithi028 ceph-mon[21769]: osdmap e89: 12 total, 12 up, 12 in 2024-08-28T19:30:05.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:04 smithi115 ceph-mon[22184]: pgmap v614: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1017 KiB/s wr, 143 op/s 2024-08-28T19:30:05.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:04 smithi115 ceph-mon[22184]: osdmap e89: 12 total, 12 up, 12 in 2024-08-28T19:30:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.129+0000 7f15a909c640 1 -- 172.21.15.16:0/4257020184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4102820 msgr2=0x7f15a4102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.129+0000 7f15a909c640 1 --2- 172.21.15.16:0/4257020184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4102820 0x7f15a4102c30 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f1594009900 tx=0x7f159402a410 comp rx=0 tx=0).stop 2024-08-28T19:30:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.129+0000 7f15a909c640 1 -- 172.21.15.16:0/4257020184 shutdown_connections 2024-08-28T19:30:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.129+0000 7f15a909c640 1 --2- 172.21.15.16:0/4257020184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15a4108bc0 0x7f15a410f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.129+0000 7f15a909c640 1 --2- 172.21.15.16:0/4257020184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15a4108270 0x7f15a4108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.129+0000 7f15a909c640 1 --2- 172.21.15.16:0/4257020184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4102820 0x7f15a4102c30 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.129+0000 7f15a909c640 1 -- 172.21.15.16:0/4257020184 >> 172.21.15.16:0/4257020184 conn(0x7f15a40794c0 msgr2=0x7f15a40798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.129+0000 7f15a909c640 1 -- 172.21.15.16:0/4257020184 shutdown_connections 2024-08-28T19:30:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.129+0000 7f15a909c640 1 -- 172.21.15.16:0/4257020184 wait complete. 2024-08-28T19:30:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.130+0000 7f15a909c640 1 Processor -- start 2024-08-28T19:30:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.130+0000 7f15a909c640 1 -- start start 2024-08-28T19:30:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.130+0000 7f15a909c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15a4102820 0x7f15a419a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.130+0000 7f15a909c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4108270 0x7f15a419a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.130+0000 7f15a909c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15a4108bc0 0x7f15a419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a909c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15a41139a0 con 0x7f15a4108270 2024-08-28T19:30:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a909c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15a4113820 con 0x7f15a4108bc0 2024-08-28T19:30:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a909c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f15a4113b20 con 0x7f15a4102820 2024-08-28T19:30:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4108270 0x7f15a419a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4108270 0x7f15a419a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47700/0 (socket says 172.21.15.16:47700) 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a259c640 1 -- 172.21.15.16:0/661754910 learned_addr learned my addr 172.21.15.16:0/661754910 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15a4102820 0x7f15a419a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a359e640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15a4108bc0 0x7f15a419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a259c640 1 -- 172.21.15.16:0/661754910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15a4102820 msgr2=0x7f15a419a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a259c640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15a4102820 0x7f15a419a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a259c640 1 -- 172.21.15.16:0/661754910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15a4108bc0 msgr2=0x7f15a419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a259c640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15a4108bc0 0x7f15a419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a259c640 1 -- 172.21.15.16:0/661754910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a41a0300 con 0x7f15a4108270 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a359e640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15a4108bc0 0x7f15a419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a259c640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4108270 0x7f15a419a930 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f159800d740 tx=0x7f159800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f158bfff640 1 -- 172.21.15.16:0/661754910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598017070 con 0x7f15a4108270 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f15a909c640 1 -- 172.21.15.16:0/661754910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a41a05f0 con 0x7f15a4108270 2024-08-28T19:30:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.131+0000 7f158bfff640 1 -- 172.21.15.16:0/661754910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f159800de50 con 0x7f15a4108270 2024-08-28T19:30:05.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.132+0000 7f158bfff640 1 -- 172.21.15.16:0/661754910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159800f620 con 0x7f15a4108270 2024-08-28T19:30:05.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.132+0000 7f15a909c640 1 -- 172.21.15.16:0/661754910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a41a6d50 con 0x7f15a4108270 2024-08-28T19:30:05.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.134+0000 7f158bfff640 1 -- 172.21.15.16:0/661754910 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f159801a070 con 0x7f15a4108270 2024-08-28T19:30:05.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.134+0000 7f15af611740 1 Processor -- start 2024-08-28T19:30:05.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.134+0000 7f15af611740 1 -- start start 2024-08-28T19:30:05.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15af611740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afcac6840 0x558afce6dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15af611740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afca9d980 0x558afce6e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15af611740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15af611740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558afcde2330 con 0x558afcaa5b60 2024-08-28T19:30:05.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15af611740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558afcdd18c0 con 0x558afcac6840 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15af611740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558afcdd5870 con 0x558afca9d980 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47716/0 (socket says 172.21.15.16:47716) 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a259c640 1 -- 172.21.15.16:0/1745528746 learned_addr learned my addr 172.21.15.16:0/1745528746 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/1745528746 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afca9d980 0x558afce6e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f158bfff640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f158c0b4ca0 0x7f158c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a359e640 1 --2- 172.21.15.16:0/1745528746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afcac6840 0x558afce6dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a359e640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f158c0b4ca0 0x7f158c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f158bfff640 1 -- 172.21.15.16:0/661754910 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9578+0+0 (secure 0 0 0) 0x7f15980e53a0 con 0x7f15a4108270 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a2d9d640 1 -- 172.21.15.16:0/1745528746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afcac6840 msgr2=0x558afce6dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/1745528746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afcac6840 0x558afce6dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a2d9d640 1 -- 172.21.15.16:0/1745528746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 msgr2=0x558afce6e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/1745528746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a2d9d640 1 -- 172.21.15.16:0/1745528746 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558afcd89df0 con 0x558afca9d980 2024-08-28T19:30:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.135+0000 7f15a259c640 1 --2- 172.21.15.16:0/1745528746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.136+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/1745528746 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afca9d980 0x558afce6e1d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1594009230 tx=0x7f1594032cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab77193c8259fe0f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.136+0000 7f15a359e640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f158c0b4ca0 0x7f158c0b71b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f159c005fd0 tx=0x7f159c004ab0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.136+0000 7f15897fa640 1 -- 172.21.15.16:0/1745528746 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1594040070 con 0x558afca9d980 2024-08-28T19:30:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.136+0000 7f15897fa640 1 -- 172.21.15.16:0/1745528746 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1594032e40 con 0x558afca9d980 2024-08-28T19:30:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.136+0000 7f15897fa640 1 -- 172.21.15.16:0/1745528746 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1594035040 con 0x558afca9d980 2024-08-28T19:30:05.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.137+0000 7f15af611740 1 -- 172.21.15.16:0/1745528746 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afca9d980 msgr2=0x558afce6e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.137+0000 7f15af611740 1 --2- 172.21.15.16:0/1745528746 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afca9d980 0x558afce6e1d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1594009230 tx=0x7f1594032cd0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.137+0000 7f15af611740 1 -- 172.21.15.16:0/1745528746 shutdown_connections 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.137+0000 7f15af611740 1 --2- 172.21.15.16:0/1745528746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.137+0000 7f15af611740 1 --2- 172.21.15.16:0/1745528746 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afca9d980 0x558afce6e1d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.137+0000 7f15af611740 1 --2- 172.21.15.16:0/1745528746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afcac6840 0x558afce6dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.137+0000 7f15af611740 1 -- 172.21.15.16:0/1745528746 >> 172.21.15.16:0/1745528746 conn(0x558afcaba990 msgr2=0x558afce6bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.137+0000 7f15af611740 1 -- 172.21.15.16:0/1745528746 shutdown_connections 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.137+0000 7f15af611740 1 -- 172.21.15.16:0/1745528746 wait complete. 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.137+0000 7f15af611740 1 -- start start 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.137+0000 7f15af611740 1 Processor -- start 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15af611740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afca9d980 0x558afce7d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15af611740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce7dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15af611740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afcac6840 0x558afce89aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15af611740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558afcc347a0 con 0x558afcaa5b60 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15af611740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558afcd58240 con 0x558afca9d980 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15af611740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558afce80770 con 0x558afcac6840 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15af611740 10 client.0 _ll_register_callbacks cb 0x558afcade090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce7dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce7dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47726/0 (socket says 172.21.15.16:47726) 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a359e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afcac6840 0x558afce89aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a259c640 1 -- 172.21.15.16:0/2695535124 learned_addr learned my addr 172.21.15.16:0/2695535124 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afca9d980 0x558afce7d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a259c640 1 -- 172.21.15.16:0/2695535124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afcac6840 msgr2=0x558afce89aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a259c640 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afcac6840 0x558afce89aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a259c640 1 -- 172.21.15.16:0/2695535124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afca9d980 msgr2=0x558afce7d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a259c640 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afca9d980 0x558afce7d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a259c640 1 -- 172.21.15.16:0/2695535124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558afcd8e2b0 con 0x558afcaa5b60 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afca9d980 0x558afce7d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f15a259c640 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce7dd00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f15980e2780 tx=0x7f15980a7630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d818481ce526f3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f1586ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.138+0000 7f1586ff5640 1 -- 172.21.15.16:0/2695535124 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598021070 con 0x558afcaa5b60 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f15af611740 1 -- 172.21.15.16:0/2695535124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558afcde4280 con 0x558afcaa5b60 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f1586ff5640 1 -- 172.21.15.16:0/2695535124 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9578+0+0 (secure 0 0 0) 0x7f15980a7900 con 0x558afcaa5b60 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f1586ff5640 1 -- 172.21.15.16:0/2695535124 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f15980aafc0 con 0x558afcaa5b60 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f15af611740 10 client.17841 fetch_fsmap learned FSMap version 51 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f15af611740 1 -- 172.21.15.16:0/2695535124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x558afcb40b90 con 0x558afcaa5b60 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f1586ff5640 1 -- 172.21.15.16:0/2695535124 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558afcde4280 con 0x558afcaa5b60 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f1586ff5640 1 -- 172.21.15.16:0/2695535124 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7f15980a6fe0 con 0x558afcaa5b60 2024-08-28T19:30:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f15af611740 10 client.17841 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f15af611740 10 client.17841 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f15af611740 20 client.17841 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f15af611740 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558afca851c0 0x558afce8f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f15af611740 4 client.17841 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.139+0000 7f15af611740 1 -- 172.21.15.16:0/2695535124 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558afcddfd20 con 0x558afca851c0 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.140+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558afca851c0 0x558afce8f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.140+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558afca851c0 0x558afce8f8e0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b969c6a0b95dfc7 server_cookie=abb1e3be28f8cc3a in_seq=0 out_seq=0 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.140+0000 7f1586ff5640 10 client.17841 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f1586ff5640 1 -- 172.21.15.16:0/2695535124 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558afcddfd20 con 0x558afca851c0 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f1586ff5640 10 client.17841 handle_command_reply: tid=0 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 1 client.17841 shutdown 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 1 -- 172.21.15.16:0/2695535124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 msgr2=0x558afce7dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce7dd00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f15980e2780 tx=0x7f15980a7630 comp rx=0 tx=0).stop 2024-08-28T19:30:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 1 -- 172.21.15.16:0/2695535124 shutdown_connections 2024-08-28T19:30:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558afca851c0 0x558afce8f8e0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afcac6840 0x558afce89aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce7dd00 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 1 --2- 172.21.15.16:0/2695535124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afca9d980 0x558afce7d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 1 -- 172.21.15.16:0/2695535124 >> 172.21.15.16:0/2695535124 conn(0x558afcaba990 msgr2=0x558afce6a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f1586ff5640 0 client.17841 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 1 -- 172.21.15.16:0/2695535124 shutdown_connections 2024-08-28T19:30:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 1 -- 172.21.15.16:0/2695535124 wait complete. 2024-08-28T19:30:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.141+0000 7f15af611740 20 client.17841 trim_cache size 0 max 16384 2024-08-28T19:30:05.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.158+0000 7f15af611740 1 Processor -- start 2024-08-28T19:30:05.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.158+0000 7f15af611740 1 -- start start 2024-08-28T19:30:05.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.158+0000 7f15af611740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afca9d980 0x558afce6f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.158+0000 7f15af611740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.158+0000 7f15af611740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afcac6840 0x558afce6b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.158+0000 7f15af611740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15a41aebf0 con 0x558afcaa5b60 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.158+0000 7f15af611740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15a41b0820 con 0x558afca9d980 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.158+0000 7f15af611740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f15a41af320 con 0x558afcac6840 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f15a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f15a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47730/0 (socket says 172.21.15.16:47730) 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f15a259c640 1 -- 172.21.15.16:0/2747864967 learned_addr learned my addr 172.21.15.16:0/2747864967 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/2747864967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afca9d980 0x558afce6f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f15a359e640 1 --2- 172.21.15.16:0/2747864967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afcac6840 0x558afce6b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f15a259c640 1 -- 172.21.15.16:0/2747864967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afcac6840 msgr2=0x558afce6b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f15a259c640 1 --2- 172.21.15.16:0/2747864967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afcac6840 0x558afce6b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f15a259c640 1 -- 172.21.15.16:0/2747864967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afca9d980 msgr2=0x558afce6f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f15a259c640 1 --2- 172.21.15.16:0/2747864967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afca9d980 0x558afce6f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f15a259c640 1 -- 172.21.15.16:0/2747864967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558afcda1d90 con 0x558afcaa5b60 2024-08-28T19:30:05.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f15a259c640 1 --2- 172.21.15.16:0/2747864967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6b350 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f15980a76f0 tx=0x7f15980e27b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada8c9c7120de7d8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:05.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f1588ff9640 1 -- 172.21.15.16:0/2747864967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598021070 con 0x558afcaa5b60 2024-08-28T19:30:05.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f1588ff9640 1 -- 172.21.15.16:0/2747864967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15980ab180 con 0x558afcaa5b60 2024-08-28T19:30:05.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.159+0000 7f1588ff9640 1 -- 172.21.15.16:0/2747864967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15980aade0 con 0x558afcaa5b60 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.160+0000 7f15af611740 1 -- 172.21.15.16:0/2747864967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 msgr2=0x558afce6b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.160+0000 7f15af611740 1 --2- 172.21.15.16:0/2747864967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6b350 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f15980a76f0 tx=0x7f15980e27b0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.160+0000 7f15af611740 1 -- 172.21.15.16:0/2747864967 shutdown_connections 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.160+0000 7f15af611740 1 --2- 172.21.15.16:0/2747864967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afcac6840 0x558afce6b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.160+0000 7f15af611740 1 --2- 172.21.15.16:0/2747864967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcaa5b60 0x558afce6b350 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.160+0000 7f15af611740 1 --2- 172.21.15.16:0/2747864967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afca9d980 0x558afce6f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.160+0000 7f15af611740 1 -- 172.21.15.16:0/2747864967 >> 172.21.15.16:0/2747864967 conn(0x558afcaba990 msgr2=0x558afce6ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.160+0000 7f15af611740 1 -- 172.21.15.16:0/2747864967 shutdown_connections 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.160+0000 7f15af611740 1 -- 172.21.15.16:0/2747864967 wait complete. 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.160+0000 7f15af611740 1 -- start start 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15af611740 1 Processor -- start 2024-08-28T19:30:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15af611740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afca9d980 0x558afce81dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15af611740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afcaa5b60 0x558afce82300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15af611740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcac6840 0x558afcef92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15af611740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15a41b00c0 con 0x558afcac6840 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15af611740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15a41af9f0 con 0x558afcaa5b60 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15af611740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f15a41acbf0 con 0x558afca9d980 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15af611740 10 client.0 _ll_register_callbacks cb 0x7f15980e8f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15a359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcac6840 0x558afcef92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15a359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcac6840 0x558afcef92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47744/0 (socket says 172.21.15.16:47744) 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15a359e640 1 -- 172.21.15.16:0/868441345 learned_addr learned my addr 172.21.15.16:0/868441345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afca9d980 0x558afce81dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15a259c640 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afcaa5b60 0x558afce82300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15a359e640 1 -- 172.21.15.16:0/868441345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afca9d980 msgr2=0x558afce81dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15a359e640 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afca9d980 0x558afce81dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15a359e640 1 -- 172.21.15.16:0/868441345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afcaa5b60 msgr2=0x558afce82300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15a359e640 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afcaa5b60 0x558afce82300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.161+0000 7f15a359e640 1 -- 172.21.15.16:0/868441345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558afcef97e0 con 0x558afcac6840 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f15a359e640 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcac6840 0x558afcef92a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f159c012e10 tx=0x7f159c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7036c53cf6c4f5fb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f1586ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f1586ff5640 1 -- 172.21.15.16:0/868441345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c01a070 con 0x558afcac6840 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f1586ff5640 1 -- 172.21.15.16:0/868441345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9578+0+0 (secure 0 0 0) 0x7f159c021400 con 0x558afcac6840 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f15af611740 1 -- 172.21.15.16:0/868441345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15980ab180 con 0x558afcac6840 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f1586ff5640 1 -- 172.21.15.16:0/868441345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f159c021600 con 0x558afcac6840 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f15af611740 10 client.17847 fetch_fsmap learned FSMap version 51 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f15af611740 1 -- 172.21.15.16:0/868441345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x558afcefa8f0 con 0x558afcac6840 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f1586ff5640 1 -- 172.21.15.16:0/868441345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15980ab180 con 0x558afcac6840 2024-08-28T19:30:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f1586ff5640 1 -- 172.21.15.16:0/868441345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 8390+0+0 (secure 0 0 0) 0x7f159c01d070 con 0x558afcac6840 2024-08-28T19:30:05.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f15af611740 10 client.17847 fetch_fsmap finished waiting for FSMap version 51 2024-08-28T19:30:05.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f15af611740 10 client.17847 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:30:05.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f15af611740 20 client.17847 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:05.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f15af611740 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558afca5a570 0x558afcefd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:05.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f15af611740 4 client.17847 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 132.4, "await_for": 45.0}] 2024-08-28T19:30:05.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.162+0000 7f15af611740 1 -- 172.21.15.16:0/868441345 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 132.4, "await_for": 45.0}) -- 0x558afcddd460 con 0x558afca5a570 2024-08-28T19:30:05.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.163+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558afca5a570 0x558afcefd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:05.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.163+0000 7f15a2d9d640 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558afca5a570 0x558afcefd260 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc455a1f292f69a server_cookie=203a327118a30c05 in_seq=0 out_seq=0 2024-08-28T19:30:05.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.163+0000 7f1586ff5640 10 client.17847 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:05.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f1586ff5640 1 -- 172.21.15.16:0/868441345 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x558afcddd460 con 0x558afca5a570 2024-08-28T19:30:05.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f1586ff5640 10 client.17847 handle_command_reply: tid=0 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 1 client.17847 shutdown 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 1 -- 172.21.15.16:0/868441345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcac6840 msgr2=0x558afcef92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcac6840 0x558afcef92a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f159c012e10 tx=0x7f159c004a40 comp rx=0 tx=0).stop 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 1 -- 172.21.15.16:0/868441345 shutdown_connections 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558afcac6840 0x558afcef92a0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afcaa5b60 0x558afce82300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558afca5a570 0x558afcefd260 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 1 --2- 172.21.15.16:0/868441345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558afca9d980 0x558afce81dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 1 -- 172.21.15.16:0/868441345 >> 172.21.15.16:0/868441345 conn(0x558afcaba990 msgr2=0x558afce72e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f1586ff5640 0 client.17847 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 1 -- 172.21.15.16:0/868441345 shutdown_connections 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 1 -- 172.21.15.16:0/868441345 wait complete. 2024-08-28T19:30:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.238+0000 7f15af611740 20 client.17847 trim_cache size 0 max 16384 2024-08-28T19:30:05.242 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:30:05.242 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 51, 2024-08-28T19:30:05.242 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:30:05.242 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 26, 2024-08-28T19:30:05.242 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:30:05.242 INFO:teuthology.orchestra.run.smithi016.stdout: "636a4220": { 2024-08-28T19:30:05.242 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 26, 2024-08-28T19:30:05.242 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:30:05.242 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 132.4, 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:05.243 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:05.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:05.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:05.244 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:30:05.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 -- 172.21.15.16:0/661754910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f158c0b4ca0 msgr2=0x7f158c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:05.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f158c0b4ca0 0x7f158c0b71b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f159c005fd0 tx=0x7f159c004ab0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 -- 172.21.15.16:0/661754910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4108270 msgr2=0x7f15a419a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4108270 0x7f15a419a930 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f159800d740 tx=0x7f159800dc00 comp rx=0 tx=0).stop 2024-08-28T19:30:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 -- 172.21.15.16:0/661754910 shutdown_connections 2024-08-28T19:30:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f158c0b4ca0 0x7f158c0b71b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15a4108bc0 0x7f15a419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4108270 0x7f15a419a930 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 --2- 172.21.15.16:0/661754910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15a4102820 0x7f15a419a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 -- 172.21.15.16:0/661754910 >> 172.21.15.16:0/661754910 conn(0x7f15a40794c0 msgr2=0x7f15a41063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 -- 172.21.15.16:0/661754910 shutdown_connections 2024-08-28T19:30:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:05.239+0000 7f15a909c640 1 -- 172.21.15.16:0/661754910 wait complete. 2024-08-28T19:30:05.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.36 MB/sec execute 345 sec latency 1809.820 ms 2024-08-28T19:30:05.997 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '636a4220', quiesce took 3.4 seconds. Will release after: 9.0 2024-08-28T19:30:06.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.34 MB/sec execute 346 sec latency 1769.003 ms 2024-08-28T19:30:07.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:06 smithi028 ceph-mon[21769]: pgmap v616: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 193 op/s 2024-08-28T19:30:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:06 smithi115 ceph-mon[22184]: pgmap v616: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 193 op/s 2024-08-28T19:30:07.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:06 smithi016 ceph-mon[18556]: pgmap v616: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 193 op/s 2024-08-28T19:30:07.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.33 MB/sec execute 347 sec latency 2769.105 ms 2024-08-28T19:30:08.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.32 MB/sec execute 348 sec latency 3769.201 ms 2024-08-28T19:30:09.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:08 smithi028 ceph-mon[21769]: pgmap v617: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-08-28T19:30:09.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:08 smithi115 ceph-mon[22184]: pgmap v617: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-08-28T19:30:09.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:08 smithi016 ceph-mon[18556]: pgmap v617: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-08-28T19:30:09.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.31 MB/sec execute 349 sec latency 4769.265 ms 2024-08-28T19:30:10.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:09 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:10.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:09 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:10.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:09 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:10.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.30 MB/sec execute 350 sec latency 5769.319 ms 2024-08-28T19:30:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:11 smithi016 ceph-mon[18556]: pgmap v618: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-08-28T19:30:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:11 smithi016 ceph-mon[18556]: osdmap e90: 12 total, 12 up, 12 in 2024-08-28T19:30:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:11 smithi028 ceph-mon[21769]: pgmap v618: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-08-28T19:30:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:11 smithi028 ceph-mon[21769]: osdmap e90: 12 total, 12 up, 12 in 2024-08-28T19:30:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:11 smithi115 ceph-mon[22184]: pgmap v618: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-08-28T19:30:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:11 smithi115 ceph-mon[22184]: osdmap e90: 12 total, 12 up, 12 in 2024-08-28T19:30:11.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.28 MB/sec execute 351 sec latency 6769.456 ms 2024-08-28T19:30:12.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.27 MB/sec execute 352 sec latency 7769.571 ms 2024-08-28T19:30:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:13 smithi016 ceph-mon[18556]: pgmap v620: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 158 op/s 2024-08-28T19:30:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:13 smithi028 ceph-mon[21769]: pgmap v620: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 158 op/s 2024-08-28T19:30:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:13 smithi115 ceph-mon[22184]: pgmap v620: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 158 op/s 2024-08-28T19:30:13.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.26 MB/sec execute 353 sec latency 8769.671 ms 2024-08-28T19:30:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:30:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:30:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:30:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:30:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:30:14.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:30:14.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:30:14.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:30:14.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:30:14.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:30:14.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:30:14.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:30:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:30:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:30:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:30:14.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.25 MB/sec execute 354 sec latency 9769.719 ms 2024-08-28T19:30:14.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:30:15.257 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:15.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:15 smithi016 ceph-mon[18556]: pgmap v621: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-08-28T19:30:15.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:15 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:15 smithi028 ceph-mon[21769]: pgmap v621: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-08-28T19:30:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:15 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:15 smithi115 ceph-mon[22184]: pgmap v621: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-08-28T19:30:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:15 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:15.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.726+0000 7fbab6995640 1 -- 172.21.15.16:0/4294168008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab0109240 msgr2=0x7fbab0109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:15.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.726+0000 7fbab6995640 1 --2- 172.21.15.16:0/4294168008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab0109240 0x7fbab0109650 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4009900 tx=0x7fbaa402a410 comp rx=0 tx=0).stop 2024-08-28T19:30:15.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.726+0000 7fbab6995640 1 -- 172.21.15.16:0/4294168008 shutdown_connections 2024-08-28T19:30:15.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.726+0000 7fbab6995640 1 --2- 172.21.15.16:0/4294168008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbab0109b90 0x7fbab01103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:15.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.726+0000 7fbab6995640 1 --2- 172.21.15.16:0/4294168008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab0109240 0x7fbab0109650 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:15.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.726+0000 7fbab6995640 1 --2- 172.21.15.16:0/4294168008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0103900 0x7fbab0103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:15.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.726+0000 7fbab6995640 1 -- 172.21.15.16:0/4294168008 >> 172.21.15.16:0/4294168008 conn(0x7fbab007b5f0 msgr2=0x7fbab007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:15.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.726+0000 7fbab6995640 1 -- 172.21.15.16:0/4294168008 shutdown_connections 2024-08-28T19:30:15.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.726+0000 7fbab6995640 1 -- 172.21.15.16:0/4294168008 wait complete. 2024-08-28T19:30:15.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbab6995640 1 Processor -- start 2024-08-28T19:30:15.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbab6995640 1 -- start start 2024-08-28T19:30:15.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbab6995640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbab0103900 0x7fbab019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:15.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbab6995640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab0109240 0x7fbab019a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:15.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbab6995640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0109b90 0x7fbab019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:15.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbab6995640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbab0113850 con 0x7fbab0109240 2024-08-28T19:30:15.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbab6995640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbab01136d0 con 0x7fbab0109b90 2024-08-28T19:30:15.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbab6995640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbab01139d0 con 0x7fbab0103900 2024-08-28T19:30:15.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbaaf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab0109240 0x7fbab019a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:15.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbaaf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab0109240 0x7fbab019a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46540/0 (socket says 172.21.15.16:46540) 2024-08-28T19:30:15.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbaaf7fe640 1 -- 172.21.15.16:0/428640837 learned_addr learned my addr 172.21.15.16:0/428640837 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:15.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbaaffff640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbab0103900 0x7fbab019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbab4f32640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0109b90 0x7fbab019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbaaf7fe640 1 -- 172.21.15.16:0/428640837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbab0103900 msgr2=0x7fbab019a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbaaf7fe640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbab0103900 0x7fbab019a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbaaf7fe640 1 -- 172.21.15.16:0/428640837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0109b90 msgr2=0x7fbab019fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.727+0000 7fbaaf7fe640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0109b90 0x7fbab019fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.728+0000 7fbaaf7fe640 1 -- 172.21.15.16:0/428640837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab01a0290 con 0x7fbab0109240 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.728+0000 7fbaaf7fe640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab0109240 0x7fbab019a8c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fbaa402a930 tx=0x7fbaa4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.728+0000 7fbaad7fa640 1 -- 172.21.15.16:0/428640837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa403f030 con 0x7fbab0109240 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.728+0000 7fbaad7fa640 1 -- 172.21.15.16:0/428640837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaa4003aa0 con 0x7fbab0109240 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.728+0000 7fbab6995640 1 -- 172.21.15.16:0/428640837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbab01a0520 con 0x7fbab0109240 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.728+0000 7fbaad7fa640 1 -- 172.21.15.16:0/428640837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4035070 con 0x7fbab0109240 2024-08-28T19:30:15.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.728+0000 7fbab6995640 1 -- 172.21.15.16:0/428640837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbab01a6ce0 con 0x7fbab0109240 2024-08-28T19:30:15.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.729+0000 7fbab6995640 1 -- 172.21.15.16:0/428640837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbab0104b80 con 0x7fbab0109240 2024-08-28T19:30:15.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.730+0000 7fbaad7fa640 1 -- 172.21.15.16:0/428640837 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fbaa4042030 con 0x7fbab0109240 2024-08-28T19:30:15.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.731+0000 7fbaad7fa640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba980b4ca0 0x7fba980b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:15.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.731+0000 7fbaaffff640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba980b4ca0 0x7fba980b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:15.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.731+0000 7fbaad7fa640 1 -- 172.21.15.16:0/428640837 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fbaa4106480 con 0x7fbab0109240 2024-08-28T19:30:15.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.732+0000 7fbaaffff640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba980b4ca0 0x7fba980b71b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0005fd0 tx=0x7fbaa0005850 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:15.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.734+0000 7fbaad7fa640 1 -- 172.21.15.16:0/428640837 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fbaa40ccea0 con 0x7fbab0109240 2024-08-28T19:30:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.956+0000 7fbab6995640 1 -- 172.21.15.16:0/428640837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbab0103d10 con 0x7fbab0109240 2024-08-28T19:30:15.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.957+0000 7fbaad7fa640 1 -- 172.21.15.16:0/428640837 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11980 (secure 0 0 0) 0x7fbab0103d10 con 0x7fbab0109240 2024-08-28T19:30:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:30:15.958 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"btime":"2024-08-28T19:30:14:226103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:30:13.285656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:30:15.959 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 53 2024-08-28T19:30:15.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 -- 172.21.15.16:0/428640837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba980b4ca0 msgr2=0x7fba980b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:15.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba980b4ca0 0x7fba980b71b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0005fd0 tx=0x7fbaa0005850 comp rx=0 tx=0).stop 2024-08-28T19:30:15.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 -- 172.21.15.16:0/428640837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab0109240 msgr2=0x7fbab019a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:15.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab0109240 0x7fbab019a8c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fbaa402a930 tx=0x7fbaa4004270 comp rx=0 tx=0).stop 2024-08-28T19:30:15.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 -- 172.21.15.16:0/428640837 shutdown_connections 2024-08-28T19:30:15.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fba980b4ca0 0x7fba980b71b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:15.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0109b90 0x7fbab019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:15.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab0109240 0x7fbab019a8c0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:15.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 --2- 172.21.15.16:0/428640837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbab0103900 0x7fbab019a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:15.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 -- 172.21.15.16:0/428640837 >> 172.21.15.16:0/428640837 conn(0x7fbab007b5f0 msgr2=0x7fbab01053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:15.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 -- 172.21.15.16:0/428640837 shutdown_connections 2024-08-28T19:30:15.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:15.959+0000 7fbab6995640 1 -- 172.21.15.16:0/428640837 wait complete. 2024-08-28T19:30:15.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.23 MB/sec execute 355 sec latency 10769.767 ms 2024-08-28T19:30:16.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:16 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/428640837' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:16.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:16 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/428640837' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:16.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:16 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/428640837' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:16.932 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id 636a4220 --release --await' 2024-08-28T19:30:16.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id 636a4220 --release --await 2024-08-28T19:30:16.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413186 4.22 MB/sec execute 356 sec latency 11769.807 ms 2024-08-28T19:30:17.201 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:17 smithi016 ceph-mon[18556]: pgmap v622: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 834 KiB/s wr, 61 op/s 2024-08-28T19:30:17.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:17 smithi028 ceph-mon[21769]: pgmap v622: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 834 KiB/s wr, 61 op/s 2024-08-28T19:30:17.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:17 smithi115 ceph-mon[22184]: pgmap v622: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 834 KiB/s wr, 61 op/s 2024-08-28T19:30:17.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.710+0000 7fadb3db0640 1 -- 172.21.15.16:0/550375844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadac105120 msgr2=0x7fadac10d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:17.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.710+0000 7fadb3db0640 1 --2- 172.21.15.16:0/550375844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadac105120 0x7fadac10d670 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fad9c009900 tx=0x7fad9c02a410 comp rx=0 tx=0).stop 2024-08-28T19:30:17.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.710+0000 7fadb3db0640 1 -- 172.21.15.16:0/550375844 shutdown_connections 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.710+0000 7fadb3db0640 1 --2- 172.21.15.16:0/550375844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadac10dc00 0x7fadac1100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.710+0000 7fadb3db0640 1 --2- 172.21.15.16:0/550375844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadac105120 0x7fadac10d670 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.710+0000 7fadb3db0640 1 --2- 172.21.15.16:0/550375844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadac1047d0 0x7fadac104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.710+0000 7fadb3db0640 1 -- 172.21.15.16:0/550375844 >> 172.21.15.16:0/550375844 conn(0x7fadac079550 msgr2=0x7fadac079960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.710+0000 7fadb3db0640 1 -- 172.21.15.16:0/550375844 shutdown_connections 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.710+0000 7fadb3db0640 1 -- 172.21.15.16:0/550375844 wait complete. 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.711+0000 7fadb3db0640 1 Processor -- start 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.711+0000 7fadb3db0640 1 -- start start 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.711+0000 7fadb3db0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadac1047d0 0x7fadac19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.711+0000 7fadb3db0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadac105120 0x7fadac19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.711+0000 7fadb3db0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadac10dc00 0x7fadac19fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.711+0000 7fadb3db0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fadac113840 con 0x7fadac1047d0 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.711+0000 7fadb3db0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1136c0 con 0x7fadac105120 2024-08-28T19:30:17.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.711+0000 7fadb3db0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1139c0 con 0x7fadac10dc00 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.711+0000 7fadb1b4c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadac1047d0 0x7fadac19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.711+0000 7fadb1b4c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadac1047d0 0x7fadac19a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46552/0 (socket says 172.21.15.16:46552) 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.711+0000 7fadb1b4c640 1 -- 172.21.15.16:0/1288282991 learned_addr learned my addr 172.21.15.16:0/1288282991 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fadb234d640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadac10dc00 0x7fadac19fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fadb134b640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadac105120 0x7fadac19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fadb1b4c640 1 -- 172.21.15.16:0/1288282991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadac10dc00 msgr2=0x7fadac19fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadac10dc00 0x7fadac19fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fadb1b4c640 1 -- 172.21.15.16:0/1288282991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadac105120 msgr2=0x7fadac19a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadac105120 0x7fadac19a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fadb1b4c640 1 -- 172.21.15.16:0/1288282991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadac1a0240 con 0x7fadac1047d0 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadac1047d0 0x7fadac19a2d0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fada800d770 tx=0x7fada800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fada2ffd640 1 -- 172.21.15.16:0/1288282991 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8017070 con 0x7fadac1047d0 2024-08-28T19:30:17.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fada2ffd640 1 -- 172.21.15.16:0/1288282991 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fada80054d0 con 0x7fadac1047d0 2024-08-28T19:30:17.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fadb3db0640 1 -- 172.21.15.16:0/1288282991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadac1a0530 con 0x7fadac1047d0 2024-08-28T19:30:17.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fada2ffd640 1 -- 172.21.15.16:0/1288282991 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8004c30 con 0x7fadac1047d0 2024-08-28T19:30:17.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.712+0000 7fadb3db0640 1 -- 172.21.15.16:0/1288282991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadac1a6c90 con 0x7fadac1047d0 2024-08-28T19:30:17.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.715+0000 7fada2ffd640 1 -- 172.21.15.16:0/1288282991 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fada8004e10 con 0x7fadac1047d0 2024-08-28T19:30:17.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.715+0000 7fadba325740 1 Processor -- start 2024-08-28T19:30:17.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.715+0000 7fadba325740 1 -- start start 2024-08-28T19:30:17.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.715+0000 7fadba325740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ee2520 0x55ea81287c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.715+0000 7fadba325740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80eb9660 0x55ea812881c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.715+0000 7fadba325740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80ec1840 0x55ea81288700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.715+0000 7fadba325740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea811fea70 con 0x55ea80eb9660 2024-08-28T19:30:17.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.715+0000 7fadba325740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ea811ee000 con 0x55ea80ee2520 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.715+0000 7fadba325740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ea811f1fb0 con 0x55ea80ec1840 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb1b4c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80eb9660 0x55ea812881c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb234d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ee2520 0x55ea81287c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb1b4c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80eb9660 0x55ea812881c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46556/0 (socket says 172.21.15.16:46556) 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb1b4c640 1 -- 172.21.15.16:0/2485289545 learned_addr learned my addr 172.21.15.16:0/2485289545 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb234d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ee2520 0x55ea81287c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45030/0 (socket says 172.21.15.16:45030) 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb134b640 1 --2- 172.21.15.16:0/2485289545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80ec1840 0x55ea81288700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fada2ffd640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad980b4ca0 0x7fad980b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb234d640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad980b4ca0 0x7fad980b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb1b4c640 1 -- 172.21.15.16:0/2485289545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80ec1840 msgr2=0x55ea81288700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/2485289545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80ec1840 0x55ea81288700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb1b4c640 1 -- 172.21.15.16:0/2485289545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ee2520 msgr2=0x55ea81287c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:17.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/2485289545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ee2520 0x55ea81287c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb1b4c640 1 -- 172.21.15.16:0/2485289545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea80f4db00 con 0x55ea80eb9660 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb134b640 1 --2- 172.21.15.16:0/2485289545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80ec1840 0x55ea81288700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fada2ffd640 1 -- 172.21.15.16:0/1288282991 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fada80e5490 con 0x7fadac1047d0 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.716+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/2485289545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80eb9660 0x55ea812881c0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fada80e2720 tx=0x7fada8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca7976c2aeeba52 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.717+0000 7fadb234d640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad980b4ca0 0x7fad980b71b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fada4005fd0 tx=0x7fada4004ab0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.717+0000 7fad97fff640 1 -- 172.21.15.16:0/2485289545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8020070 con 0x55ea80eb9660 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.717+0000 7fad97fff640 1 -- 172.21.15.16:0/2485289545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fada80b43c0 con 0x55ea80eb9660 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.717+0000 7fad97fff640 1 -- 172.21.15.16:0/2485289545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada80affc0 con 0x55ea80eb9660 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.717+0000 7fadba325740 1 -- 172.21.15.16:0/2485289545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80eb9660 msgr2=0x55ea812881c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.717+0000 7fadba325740 1 --2- 172.21.15.16:0/2485289545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80eb9660 0x55ea812881c0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fada80e2720 tx=0x7fada8009f90 comp rx=0 tx=0).stop 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.718+0000 7fadba325740 1 -- 172.21.15.16:0/2485289545 shutdown_connections 2024-08-28T19:30:17.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.718+0000 7fadba325740 1 --2- 172.21.15.16:0/2485289545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80ec1840 0x55ea81288700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.718+0000 7fadba325740 1 --2- 172.21.15.16:0/2485289545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80eb9660 0x55ea812881c0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.718+0000 7fadba325740 1 --2- 172.21.15.16:0/2485289545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ee2520 0x55ea81287c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.718+0000 7fadba325740 1 -- 172.21.15.16:0/2485289545 >> 172.21.15.16:0/2485289545 conn(0x55ea80ed6910 msgr2=0x55ea81285b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.718+0000 7fadba325740 1 -- 172.21.15.16:0/2485289545 shutdown_connections 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.718+0000 7fadba325740 1 -- 172.21.15.16:0/2485289545 wait complete. 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.718+0000 7fadba325740 1 -- start start 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.718+0000 7fadba325740 1 Processor -- start 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadba325740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80eb9660 0x55ea812938f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadba325740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81293e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadba325740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80ee2520 0x55ea8129f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadba325740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea81296400 con 0x55ea80ec1840 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadba325740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ea811af520 con 0x55ea80eb9660 2024-08-28T19:30:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadba325740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ea81296580 con 0x55ea80ee2520 2024-08-28T19:30:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadba325740 10 client.0 _ll_register_callbacks cb 0x55ea811e7c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadb134b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81293e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadb134b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81293e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46560/0 (socket says 172.21.15.16:46560) 2024-08-28T19:30:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadb134b640 1 -- 172.21.15.16:0/1903662406 learned_addr learned my addr 172.21.15.16:0/1903662406 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80eb9660 0x55ea812938f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadb234d640 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80ee2520 0x55ea8129f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadb134b640 1 -- 172.21.15.16:0/1903662406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80ee2520 msgr2=0x55ea8129f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadb134b640 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80ee2520 0x55ea8129f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadb134b640 1 -- 172.21.15.16:0/1903662406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80eb9660 msgr2=0x55ea812938f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadb134b640 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80eb9660 0x55ea812938f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadb134b640 1 -- 172.21.15.16:0/1903662406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea80fc4960 con 0x55ea80ec1840 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fadb134b640 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81293e70 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fad9c005ab0 tx=0x7fad9c0316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7949dd8f5bbc3ea3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.719+0000 7fad957fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fad957fa640 1 -- 172.21.15.16:0/1903662406 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9c035040 con 0x55ea80ec1840 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fad957fa640 1 -- 172.21.15.16:0/1903662406 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fad9c004240 con 0x55ea80ec1840 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fadba325740 1 -- 172.21.15.16:0/1903662406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ea8118d9a0 con 0x55ea80ec1840 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fad957fa640 1 -- 172.21.15.16:0/1903662406 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fad9c004440 con 0x55ea80ec1840 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fadba325740 10 client.17865 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fadba325740 1 -- 172.21.15.16:0/1903662406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55ea812a25e0 con 0x55ea80ec1840 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fad957fa640 1 -- 172.21.15.16:0/1903662406 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ea8118d9a0 con 0x55ea80ec1840 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fad957fa640 1 -- 172.21.15.16:0/1903662406 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fad9c004ad0 con 0x55ea80ec1840 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fadba325740 10 client.17865 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fadba325740 10 client.17865 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fadba325740 20 client.17865 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:17.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fadba325740 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea80ea0ea0 0x55ea812a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fadba325740 4 client.17865 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.720+0000 7fadba325740 1 -- 172.21.15.16:0/1903662406 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea812009c0 con 0x55ea80ea0ea0 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.721+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea80ea0ea0 0x55ea812a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.721+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea80ea0ea0 0x55ea812a5ab0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4755cd6b872a80a server_cookie=53113f2dd652291a in_seq=0 out_seq=0 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.721+0000 7fad957fa640 10 client.17865 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fad957fa640 1 -- 172.21.15.16:0/1903662406 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ea812009c0 con 0x55ea80ea0ea0 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fad957fa640 10 client.17865 handle_command_reply: tid=0 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 1 client.17865 shutdown 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 1 -- 172.21.15.16:0/1903662406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 msgr2=0x55ea81293e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81293e70 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fad9c005ab0 tx=0x7fad9c0316b0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 1 -- 172.21.15.16:0/1903662406 shutdown_connections 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea80ea0ea0 0x55ea812a5ab0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80ee2520 0x55ea8129f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81293e70 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fad957fa640 0 client.17865 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 1 --2- 172.21.15.16:0/1903662406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80eb9660 0x55ea812938f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 1 -- 172.21.15.16:0/1903662406 >> 172.21.15.16:0/1903662406 conn(0x55ea80ed6910 msgr2=0x55ea81288cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 1 -- 172.21.15.16:0/1903662406 shutdown_connections 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 1 -- 172.21.15.16:0/1903662406 wait complete. 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.722+0000 7fadba325740 20 client.17865 trim_cache size 0 max 16384 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.740+0000 7fadba325740 1 Processor -- start 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.740+0000 7fadba325740 1 -- start start 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.740+0000 7fadba325740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ea0ea0 0x55ea8128e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.740+0000 7fadba325740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80eb9660 0x55ea8129f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.740+0000 7fadba325740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea8129fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.740+0000 7fadba325740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1b0d70 con 0x55ea80ec1840 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.740+0000 7fadba325740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1b2910 con 0x55ea80ea0ea0 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.740+0000 7fadba325740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1b14a0 con 0x55ea80eb9660 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.740+0000 7fadb234d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea8129fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.740+0000 7fadb234d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea8129fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46562/0 (socket says 172.21.15.16:46562) 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.740+0000 7fadb234d640 1 -- 172.21.15.16:0/268417237 learned_addr learned my addr 172.21.15.16:0/268417237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.741+0000 7fadb134b640 1 --2- 172.21.15.16:0/268417237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80eb9660 0x55ea8129f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.741+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/268417237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ea0ea0 0x55ea8128e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.741+0000 7fadb234d640 1 -- 172.21.15.16:0/268417237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80eb9660 msgr2=0x55ea8129f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.741+0000 7fadb234d640 1 --2- 172.21.15.16:0/268417237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80eb9660 0x55ea8129f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.741+0000 7fadb234d640 1 -- 172.21.15.16:0/268417237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ea0ea0 msgr2=0x55ea8128e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:17.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.741+0000 7fadb234d640 1 --2- 172.21.15.16:0/268417237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ea0ea0 0x55ea8128e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.741+0000 7fadb234d640 1 -- 172.21.15.16:0/268417237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea81285c40 con 0x55ea80ec1840 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.741+0000 7fadb234d640 1 --2- 172.21.15.16:0/268417237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea8129fcb0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fada4012e90 tx=0x7fada4009ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51909b459342ccda server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.741+0000 7fad977fe640 1 -- 172.21.15.16:0/268417237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada40119b0 con 0x55ea80ec1840 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.741+0000 7fad977fe640 1 -- 172.21.15.16:0/268417237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fada4025070 con 0x55ea80ec1840 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.741+0000 7fad977fe640 1 -- 172.21.15.16:0/268417237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4021d60 con 0x55ea80ec1840 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.742+0000 7fadba325740 1 -- 172.21.15.16:0/268417237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 msgr2=0x55ea8129fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.742+0000 7fadba325740 1 --2- 172.21.15.16:0/268417237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea8129fcb0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fada4012e90 tx=0x7fada4009ab0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.742+0000 7fadba325740 1 -- 172.21.15.16:0/268417237 shutdown_connections 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.742+0000 7fadba325740 1 --2- 172.21.15.16:0/268417237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea8129fcb0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.742+0000 7fadba325740 1 --2- 172.21.15.16:0/268417237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80eb9660 0x55ea8129f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.742+0000 7fadba325740 1 --2- 172.21.15.16:0/268417237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ea0ea0 0x55ea8128e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.742+0000 7fadba325740 1 -- 172.21.15.16:0/268417237 >> 172.21.15.16:0/268417237 conn(0x55ea80ed6910 msgr2=0x55ea81291f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.742+0000 7fadba325740 1 -- 172.21.15.16:0/268417237 shutdown_connections 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.742+0000 7fadba325740 1 -- 172.21.15.16:0/268417237 wait complete. 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.742+0000 7fadba325740 1 -- start start 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.742+0000 7fadba325740 1 Processor -- start 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadba325740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ea0ea0 0x55ea8129c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadba325740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80eb9660 0x55ea812967d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadba325740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81296ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadba325740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1b2240 con 0x55ea80ec1840 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadba325740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1b1b70 con 0x55ea80ea0ea0 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadba325740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1aed70 con 0x55ea80eb9660 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadba325740 10 client.0 _ll_register_callbacks cb 0x7fad9c005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadb234d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81296ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadb234d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81296ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46566/0 (socket says 172.21.15.16:46566) 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadb234d640 1 -- 172.21.15.16:0/49856857 learned_addr learned my addr 172.21.15.16:0/49856857 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadb134b640 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80eb9660 0x55ea812967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ea0ea0 0x55ea8129c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadb234d640 1 -- 172.21.15.16:0/49856857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80eb9660 msgr2=0x55ea812967d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadb234d640 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80eb9660 0x55ea812967d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadb234d640 1 -- 172.21.15.16:0/49856857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ea0ea0 msgr2=0x55ea8129c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadb234d640 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ea0ea0 0x55ea8129c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.743+0000 7fadb234d640 1 -- 172.21.15.16:0/49856857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea81310470 con 0x55ea80ec1840 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fadb234d640 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81296ee0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fada40218a0 tx=0x7fada4021620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4bd32fd6fcc328b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fad957fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fad957fa640 1 -- 172.21.15.16:0/49856857 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4039060 con 0x55ea80ec1840 2024-08-28T19:30:17.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fad957fa640 1 -- 172.21.15.16:0/49856857 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fada400edc0 con 0x55ea80ec1840 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fadba325740 1 -- 172.21.15.16:0/49856857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fada4025070 con 0x55ea80ec1840 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fad957fa640 1 -- 172.21.15.16:0/49856857 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fada4021cb0 con 0x55ea80ec1840 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fadba325740 10 client.17871 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fadba325740 1 -- 172.21.15.16:0/49856857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55ea81311660 con 0x55ea80ec1840 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fad957fa640 1 -- 172.21.15.16:0/49856857 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fada4025070 con 0x55ea80ec1840 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fad957fa640 1 -- 172.21.15.16:0/49856857 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fada401e030 con 0x55ea80ec1840 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fadba325740 10 client.17871 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fadba325740 10 client.17871 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fadba325740 20 client.17871 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fadba325740 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea80ee2520 0x55ea81313d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fadba325740 4 client.17871 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "636a4220", "release": true, "await": true}] 2024-08-28T19:30:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.744+0000 7fadba325740 1 -- 172.21.15.16:0/49856857 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "636a4220", "release": true, "await": true}) -- 0x55ea811fc460 con 0x55ea80ee2520 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.745+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea80ee2520 0x55ea81313d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.745+0000 7fadb1b4c640 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea80ee2520 0x55ea81313d40 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c362c756069590e server_cookie=875ed39ea7b59db7 in_seq=0 out_seq=0 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.745+0000 7fad957fa640 10 client.17871 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fad957fa640 1 -- 172.21.15.16:0/49856857 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ea811fc460 con 0x55ea80ee2520 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fad957fa640 10 client.17871 handle_command_reply: tid=0 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fadba325740 1 client.17871 shutdown 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fadba325740 1 -- 172.21.15.16:0/49856857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 msgr2=0x55ea81296ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fadba325740 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81296ee0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fada40218a0 tx=0x7fada4021620 comp rx=0 tx=0).stop 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fadba325740 1 -- 172.21.15.16:0/49856857 shutdown_connections 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fadba325740 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea80ec1840 0x55ea81296ee0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fadba325740 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ea80ee2520 0x55ea81313d40 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fadba325740 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ea80eb9660 0x55ea812967d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fadba325740 1 --2- 172.21.15.16:0/49856857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea80ea0ea0 0x55ea8129c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fad957fa640 0 client.17871 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:17.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.756+0000 7fadba325740 1 -- 172.21.15.16:0/49856857 >> 172.21.15.16:0/49856857 conn(0x55ea80ed6910 msgr2=0x55ea81284150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:17.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadba325740 1 -- 172.21.15.16:0/49856857 shutdown_connections 2024-08-28T19:30:17.898 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:30:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 53, 2024-08-28T19:30:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:30:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 28, 2024-08-28T19:30:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:30:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "636a4220": { 2024-08-28T19:30:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 28, 2024-08-28T19:30:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 132.4, 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:17.900 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:30:17.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadba325740 1 -- 172.21.15.16:0/49856857 wait complete. 2024-08-28T19:30:17.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadba325740 20 client.17871 trim_cache size 0 max 16384 2024-08-28T19:30:17.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 -- 172.21.15.16:0/1288282991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad980b4ca0 msgr2=0x7fad980b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:17.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad980b4ca0 0x7fad980b71b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fada4005fd0 tx=0x7fada4004ab0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 -- 172.21.15.16:0/1288282991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadac1047d0 msgr2=0x7fadac19a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadac1047d0 0x7fadac19a2d0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fada800d770 tx=0x7fada800dc30 comp rx=0 tx=0).stop 2024-08-28T19:30:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 -- 172.21.15.16:0/1288282991 shutdown_connections 2024-08-28T19:30:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadac10dc00 0x7fadac19fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fad980b4ca0 0x7fad980b71b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadac105120 0x7fadac19a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 --2- 172.21.15.16:0/1288282991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadac1047d0 0x7fadac19a2d0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 -- 172.21.15.16:0/1288282991 >> 172.21.15.16:0/1288282991 conn(0x7fadac079550 msgr2=0x7fadac100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 -- 172.21.15.16:0/1288282991 shutdown_connections 2024-08-28T19:30:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:17.757+0000 7fadb3db0640 1 -- 172.21.15.16:0/1288282991 wait complete. 2024-08-28T19:30:17.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413828 4.22 MB/sec execute 357 sec latency 12564.766 ms 2024-08-28T19:30:18.641 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '636a4220', total seconds elapsed: 16.0 2024-08-28T19:30:18.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 415455 4.24 MB/sec execute 358 sec latency 13.538 ms 2024-08-28T19:30:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:19 smithi016 ceph-mon[18556]: pgmap v623: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 598 KiB/s wr, 47 op/s 2024-08-28T19:30:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:19 smithi028 ceph-mon[21769]: pgmap v623: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 598 KiB/s wr, 47 op/s 2024-08-28T19:30:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:19 smithi115 ceph-mon[22184]: pgmap v623: 129 pgs: 129 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 598 KiB/s wr, 47 op/s 2024-08-28T19:30:19.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 416732 4.24 MB/sec execute 359 sec latency 125.121 ms 2024-08-28T19:30:20.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 416732 4.23 MB/sec execute 360 sec latency 1125.209 ms 2024-08-28T19:30:21.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:21 smithi016 ceph-mon[18556]: pgmap v624: 129 pgs: 129 active+clean; 286 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 837 KiB/s wr, 58 op/s 2024-08-28T19:30:21.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:21 smithi028 ceph-mon[21769]: pgmap v624: 129 pgs: 129 active+clean; 286 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 837 KiB/s wr, 58 op/s 2024-08-28T19:30:21.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:21 smithi115 ceph-mon[22184]: pgmap v624: 129 pgs: 129 active+clean; 286 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 837 KiB/s wr, 58 op/s 2024-08-28T19:30:21.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 416732 4.22 MB/sec execute 361 sec latency 2125.253 ms 2024-08-28T19:30:22.964 INFO:tasks.workunit.client.0.smithi016.stdout: 1 418812 4.24 MB/sec execute 362 sec latency 2144.222 ms 2024-08-28T19:30:23.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:23 smithi016 ceph-mon[18556]: pgmap v625: 129 pgs: 129 active+clean; 286 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 861 KiB/s wr, 80 op/s 2024-08-28T19:30:23.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:23 smithi028 ceph-mon[21769]: pgmap v625: 129 pgs: 129 active+clean; 286 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 861 KiB/s wr, 80 op/s 2024-08-28T19:30:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:23 smithi115 ceph-mon[22184]: pgmap v625: 129 pgs: 129 active+clean; 286 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 861 KiB/s wr, 80 op/s 2024-08-28T19:30:23.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 420054 4.25 MB/sec execute 363 sec latency 15.748 ms 2024-08-28T19:30:24.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 420275 4.23 MB/sec execute 364 sec latency 776.369 ms 2024-08-28T19:30:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:25 smithi028 ceph-mon[21769]: pgmap v626: 129 pgs: 129 active+clean; 287 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 861 KiB/s wr, 81 op/s 2024-08-28T19:30:25.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:25 smithi115 ceph-mon[22184]: pgmap v626: 129 pgs: 129 active+clean; 287 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 861 KiB/s wr, 81 op/s 2024-08-28T19:30:25.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:25 smithi016 ceph-mon[18556]: pgmap v626: 129 pgs: 129 active+clean; 287 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 861 KiB/s wr, 81 op/s 2024-08-28T19:30:25.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 420275 4.22 MB/sec execute 365 sec latency 1776.411 ms 2024-08-28T19:30:26.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 420275 4.21 MB/sec execute 366 sec latency 2776.520 ms 2024-08-28T19:30:27.547 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:27 smithi016 ceph-mon[18556]: pgmap v627: 129 pgs: 129 active+clean; 290 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-08-28T19:30:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:27 smithi028 ceph-mon[21769]: pgmap v627: 129 pgs: 129 active+clean; 290 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-08-28T19:30:27.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:27 smithi115 ceph-mon[22184]: pgmap v627: 129 pgs: 129 active+clean; 290 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-08-28T19:30:27.967 INFO:tasks.workunit.client.0.smithi016.stdout: 1 422356 4.24 MB/sec execute 367 sec latency 2789.335 ms 2024-08-28T19:30:28.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 423601 4.24 MB/sec execute 368 sec latency 17.456 ms 2024-08-28T19:30:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:29 smithi028 ceph-mon[21769]: pgmap v628: 129 pgs: 129 active+clean; 291 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-08-28T19:30:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:29 smithi115 ceph-mon[22184]: pgmap v628: 129 pgs: 129 active+clean; 291 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-08-28T19:30:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:29 smithi016 ceph-mon[18556]: pgmap v628: 129 pgs: 129 active+clean; 291 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-08-28T19:30:29.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 423824 4.23 MB/sec execute 369 sec latency 757.103 ms 2024-08-28T19:30:30.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 423824 4.22 MB/sec execute 370 sec latency 1757.165 ms 2024-08-28T19:30:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:31 smithi028 ceph-mon[21769]: pgmap v629: 129 pgs: 129 active+clean; 292 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-08-28T19:30:31.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:31 smithi115 ceph-mon[22184]: pgmap v629: 129 pgs: 129 active+clean; 292 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-08-28T19:30:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:31 smithi016 ceph-mon[18556]: pgmap v629: 129 pgs: 129 active+clean; 292 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-08-28T19:30:31.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 424562 4.21 MB/sec execute 371 sec latency 2398.099 ms 2024-08-28T19:30:32.452 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:30:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:32.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 426108 4.23 MB/sec execute 372 sec latency 11.406 ms 2024-08-28T19:30:33.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.245+0000 7f915d397640 1 -- 172.21.15.16:0/2089360541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915810dc00 msgr2=0x7f91581100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.245+0000 7f915d397640 1 --2- 172.21.15.16:0/2089360541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915810dc00 0x7f91581100f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f9150009ca0 tx=0x7f915002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.245+0000 7f915d397640 1 -- 172.21.15.16:0/2089360541 shutdown_connections 2024-08-28T19:30:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.245+0000 7f915d397640 1 --2- 172.21.15.16:0/2089360541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915810dc00 0x7f91581100f0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.245+0000 7f915d397640 1 --2- 172.21.15.16:0/2089360541 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91581050f0 0x7f915810d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.245+0000 7f915d397640 1 --2- 172.21.15.16:0/2089360541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581047a0 0x7f9158104bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.245+0000 7f915d397640 1 -- 172.21.15.16:0/2089360541 >> 172.21.15.16:0/2089360541 conn(0x7f9158079450 msgr2=0x7f9158079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.245+0000 7f915d397640 1 -- 172.21.15.16:0/2089360541 shutdown_connections 2024-08-28T19:30:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.245+0000 7f915d397640 1 -- 172.21.15.16:0/2089360541 wait complete. 2024-08-28T19:30:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.246+0000 7f915d397640 1 Processor -- start 2024-08-28T19:30:33.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.246+0000 7f915d397640 1 -- start start 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.246+0000 7f915d397640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91581047a0 0x7f915819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.246+0000 7f915d397640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581050f0 0x7f915819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.246+0000 7f915d397640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915810dc00 0x7f915819fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.246+0000 7f915d397640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9158113880 con 0x7f915810dc00 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f915d397640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9158113700 con 0x7f91581050f0 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f915d397640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9158113a00 con 0x7f91581047a0 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f9156ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91581047a0 0x7f915819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f914ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581050f0 0x7f915819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f9156ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91581047a0 0x7f915819a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55192/0 (socket says 172.21.15.16:55192) 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f914ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581050f0 0x7f915819a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:55964/0 (socket says 172.21.15.16:55964) 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f9156ffd640 1 -- 172.21.15.16:0/3118651756 learned_addr learned my addr 172.21.15.16:0/3118651756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f91577fe640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915810dc00 0x7f915819fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f914ffff640 1 -- 172.21.15.16:0/3118651756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91581047a0 msgr2=0x7f915819a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f914ffff640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91581047a0 0x7f915819a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f914ffff640 1 -- 172.21.15.16:0/3118651756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915810dc00 msgr2=0x7f915819fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f914ffff640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915810dc00 0x7f915819fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f914ffff640 1 -- 172.21.15.16:0/3118651756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91581a0180 con 0x7f91581050f0 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f914ffff640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581050f0 0x7f915819a7b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f915806b720 tx=0x7f914000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f9154ff9640 1 -- 172.21.15.16:0/3118651756 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914000cc40 con 0x7f91581050f0 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f915d397640 1 -- 172.21.15.16:0/3118651756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91581a0470 con 0x7f91581050f0 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f915d397640 1 -- 172.21.15.16:0/3118651756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91581a6bd0 con 0x7f91581050f0 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f9154ff9640 1 -- 172.21.15.16:0/3118651756 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f914000cde0 con 0x7f91581050f0 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.247+0000 7f9154ff9640 1 -- 172.21.15.16:0/3118651756 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140004a00 con 0x7f91581050f0 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.250+0000 7f916390c740 1 Processor -- start 2024-08-28T19:30:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.250+0000 7f916390c740 1 -- start start 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.250+0000 7f916390c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x56056586b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.250+0000 7f916390c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056549d1b0 0x56056586b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f916390c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x56056586bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f916390c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5605657e2630 con 0x5605654c6070 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f916390c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5605657d1bc0 con 0x56056549d1b0 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f916390c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5605657d5b70 con 0x5605654a5390 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f91577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x56056586b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f91577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x56056586b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37218/0 (socket says 172.21.15.16:37218) 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f91577fe640 1 -- 172.21.15.16:0/24899071 learned_addr learned my addr 172.21.15.16:0/24899071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f9156ffd640 1 --2- 172.21.15.16:0/24899071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056549d1b0 0x56056586b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f9154ff9640 1 -- 172.21.15.16:0/3118651756 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f9140004ba0 con 0x7f91581050f0 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f914ffff640 1 --2- 172.21.15.16:0/24899071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x56056586bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f91577fe640 1 -- 172.21.15.16:0/24899071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 msgr2=0x56056586bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f91577fe640 1 --2- 172.21.15.16:0/24899071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x56056586bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f91577fe640 1 -- 172.21.15.16:0/24899071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056549d1b0 msgr2=0x56056586b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f91577fe640 1 --2- 172.21.15.16:0/24899071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056549d1b0 0x56056586b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f91577fe640 1 -- 172.21.15.16:0/24899071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560565531650 con 0x5605654c6070 2024-08-28T19:30:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.251+0000 7f9156ffd640 1 --2- 172.21.15.16:0/24899071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056549d1b0 0x56056586b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.252+0000 7f91577fe640 1 --2- 172.21.15.16:0/24899071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x56056586b300 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f915002f410 tx=0x7f915002f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=513ae8141fcaabb3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.252+0000 7f914d7fa640 1 -- 172.21.15.16:0/24899071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9150031cf0 con 0x5605654c6070 2024-08-28T19:30:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.252+0000 7f914d7fa640 1 -- 172.21.15.16:0/24899071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9150036030 con 0x5605654c6070 2024-08-28T19:30:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.252+0000 7f914d7fa640 1 -- 172.21.15.16:0/24899071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91500042a0 con 0x5605654c6070 2024-08-28T19:30:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.252+0000 7f9154ff9640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91440b4d20 0x7f91440b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.252+0000 7f9156ffd640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91440b4d20 0x7f91440b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.252+0000 7f9154ff9640 1 -- 172.21.15.16:0/3118651756 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7f91400e5880 con 0x7f91581050f0 2024-08-28T19:30:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.253+0000 7f916390c740 1 -- 172.21.15.16:0/24899071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 msgr2=0x56056586b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.253+0000 7f916390c740 1 --2- 172.21.15.16:0/24899071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x56056586b300 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f915002f410 tx=0x7f915002f440 comp rx=0 tx=0).stop 2024-08-28T19:30:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.253+0000 7f9156ffd640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91440b4d20 0x7f91440b7230 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x560565869f80 tx=0x7f9148005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.253+0000 7f916390c740 1 -- 172.21.15.16:0/24899071 shutdown_connections 2024-08-28T19:30:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.253+0000 7f916390c740 1 --2- 172.21.15.16:0/24899071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x56056586bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.253+0000 7f916390c740 1 --2- 172.21.15.16:0/24899071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056549d1b0 0x56056586b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.253+0000 7f916390c740 1 --2- 172.21.15.16:0/24899071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x56056586b300 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.253+0000 7f916390c740 1 -- 172.21.15.16:0/24899071 >> 172.21.15.16:0/24899071 conn(0x5605654ba460 msgr2=0x5605658691f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.253+0000 7f916390c740 1 -- 172.21.15.16:0/24899071 shutdown_connections 2024-08-28T19:30:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.253+0000 7f916390c740 1 -- 172.21.15.16:0/24899071 wait complete. 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f916390c740 1 -- start start 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f916390c740 1 Processor -- start 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f916390c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56056549d1b0 0x560565876b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f916390c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605654a5390 0x560565877040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f916390c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654c6070 0x560565882bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f916390c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560565634700 con 0x56056549d1b0 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f916390c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560565758700 con 0x5605654a5390 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f916390c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560565879900 con 0x5605654c6070 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f916390c740 10 client.0 _ll_register_callbacks cb 0x560565787dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f9156ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56056549d1b0 0x560565876b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f9156ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56056549d1b0 0x560565876b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37234/0 (socket says 172.21.15.16:37234) 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f9156ffd640 1 -- 172.21.15.16:0/1594383825 learned_addr learned my addr 172.21.15.16:0/1594383825 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f91577fe640 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654c6070 0x560565882bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.254+0000 7f914ffff640 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605654a5390 0x560565877040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f9156ffd640 1 -- 172.21.15.16:0/1594383825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654c6070 msgr2=0x560565882bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f9156ffd640 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654c6070 0x560565882bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f9156ffd640 1 -- 172.21.15.16:0/1594383825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605654a5390 msgr2=0x560565877040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f9156ffd640 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605654a5390 0x560565877040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f9156ffd640 1 -- 172.21.15.16:0/1594383825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605657918b0 con 0x56056549d1b0 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f914ffff640 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605654a5390 0x560565877040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f9156ffd640 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56056549d1b0 0x560565876b00 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f9148004610 tx=0x7f9148004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d940ab28e6f819 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f913e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f913e7fc640 1 -- 172.21.15.16:0/1594383825 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148009cb0 con 0x56056549d1b0 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f913e7fc640 1 -- 172.21.15.16:0/1594383825 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7f914800f690 con 0x56056549d1b0 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f916390c740 1 -- 172.21.15.16:0/1594383825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9150036030 con 0x56056549d1b0 2024-08-28T19:30:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f913e7fc640 1 -- 172.21.15.16:0/1594383825 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f914800f890 con 0x56056549d1b0 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f916390c740 10 client.17883 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f916390c740 1 -- 172.21.15.16:0/1594383825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5605658857b0 con 0x56056549d1b0 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.255+0000 7f913e7fc640 1 -- 172.21.15.16:0/1594383825 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9150036030 con 0x56056549d1b0 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.256+0000 7f913e7fc640 1 -- 172.21.15.16:0/1594383825 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7f9148027070 con 0x56056549d1b0 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.256+0000 7f916390c740 10 client.17883 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.256+0000 7f916390c740 10 client.17883 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.256+0000 7f916390c740 20 client.17883 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.256+0000 7f916390c740 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5605654849f0 0x560565888cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.256+0000 7f916390c740 4 client.17883 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.256+0000 7f916390c740 1 -- 172.21.15.16:0/1594383825 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605657e4580 con 0x5605654849f0 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.256+0000 7f914ffff640 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5605654849f0 0x560565888cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.256+0000 7f913e7fc640 10 client.17883 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:33.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.256+0000 7f914ffff640 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5605654849f0 0x560565888cf0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1890315b62f071cc server_cookie=24a99b6ae38f1530 in_seq=0 out_seq=0 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f913e7fc640 1 -- 172.21.15.16:0/1594383825 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5605657e4580 con 0x5605654849f0 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f913e7fc640 10 client.17883 handle_command_reply: tid=0 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f916390c740 1 client.17883 shutdown 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f916390c740 1 -- 172.21.15.16:0/1594383825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56056549d1b0 msgr2=0x560565876b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f916390c740 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56056549d1b0 0x560565876b00 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f9148004610 tx=0x7f9148004730 comp rx=0 tx=0).stop 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f916390c740 1 -- 172.21.15.16:0/1594383825 shutdown_connections 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f916390c740 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5605654849f0 0x560565888cf0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f916390c740 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654c6070 0x560565882bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f913e7fc640 0 client.17883 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f916390c740 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605654a5390 0x560565877040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f916390c740 1 --2- 172.21.15.16:0/1594383825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56056549d1b0 0x560565876b00 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.257+0000 7f916390c740 1 -- 172.21.15.16:0/1594383825 >> 172.21.15.16:0/1594383825 conn(0x5605654ba460 msgr2=0x56056586e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.258+0000 7f916390c740 1 -- 172.21.15.16:0/1594383825 shutdown_connections 2024-08-28T19:30:33.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.258+0000 7f916390c740 1 -- 172.21.15.16:0/1594383825 wait complete. 2024-08-28T19:30:33.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.258+0000 7f916390c740 20 client.17883 trim_cache size 0 max 16384 2024-08-28T19:30:33.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.278+0000 7f916390c740 1 Processor -- start 2024-08-28T19:30:33.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.278+0000 7f916390c740 1 -- start start 2024-08-28T19:30:33.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.278+0000 7f916390c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x560565869220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.278+0000 7f916390c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x560565876ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f916390c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560565459da0 0x560565882af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f916390c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91581b0c20 con 0x5605654c6070 2024-08-28T19:30:33.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f916390c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91581b27c0 con 0x560565459da0 2024-08-28T19:30:33.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f916390c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f91581b1350 con 0x5605654a5390 2024-08-28T19:30:33.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x560565876ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f91577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560565459da0 0x560565882af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x560565876ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37248/0 (socket says 172.21.15.16:37248) 2024-08-28T19:30:33.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914ffff640 1 -- 172.21.15.16:0/3178310015 learned_addr learned my addr 172.21.15.16:0/3178310015 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:33.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f9156ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x560565869220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914ffff640 1 -- 172.21.15.16:0/3178310015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 msgr2=0x560565869220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:33.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914ffff640 1 --2- 172.21.15.16:0/3178310015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x560565869220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914ffff640 1 -- 172.21.15.16:0/3178310015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560565459da0 msgr2=0x560565882af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:33.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914ffff640 1 --2- 172.21.15.16:0/3178310015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560565459da0 0x560565882af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914ffff640 1 -- 172.21.15.16:0/3178310015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560565875a60 con 0x5605654c6070 2024-08-28T19:30:33.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914ffff640 1 --2- 172.21.15.16:0/3178310015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x560565876ca0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f914000c6a0 tx=0x7f91400053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c60a52b0888ccc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914cff9640 1 -- 172.21.15.16:0/3178310015 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914001c040 con 0x5605654c6070 2024-08-28T19:30:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914cff9640 1 -- 172.21.15.16:0/3178310015 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91400ab650 con 0x5605654c6070 2024-08-28T19:30:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.279+0000 7f914cff9640 1 -- 172.21.15.16:0/3178310015 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91400b4e60 con 0x5605654c6070 2024-08-28T19:30:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.280+0000 7f916390c740 1 -- 172.21.15.16:0/3178310015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 msgr2=0x560565876ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.280+0000 7f916390c740 1 --2- 172.21.15.16:0/3178310015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x560565876ca0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f914000c6a0 tx=0x7f91400053f0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.280+0000 7f916390c740 1 -- 172.21.15.16:0/3178310015 shutdown_connections 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.280+0000 7f916390c740 1 --2- 172.21.15.16:0/3178310015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560565459da0 0x560565882af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.280+0000 7f916390c740 1 --2- 172.21.15.16:0/3178310015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x560565876ca0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.280+0000 7f916390c740 1 --2- 172.21.15.16:0/3178310015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x560565869220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.280+0000 7f916390c740 1 -- 172.21.15.16:0/3178310015 >> 172.21.15.16:0/3178310015 conn(0x5605654ba460 msgr2=0x560565868b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.280+0000 7f916390c740 1 -- 172.21.15.16:0/3178310015 shutdown_connections 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.281+0000 7f916390c740 1 -- 172.21.15.16:0/3178310015 wait complete. 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.281+0000 7f916390c740 1 -- start start 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.281+0000 7f916390c740 1 Processor -- start 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.281+0000 7f916390c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560565459da0 0x56056587af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.281+0000 7f916390c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x56056587b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.281+0000 7f916390c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x5605658f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.281+0000 7f916390c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91581b20f0 con 0x5605654c6070 2024-08-28T19:30:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.281+0000 7f916390c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91581b1a20 con 0x560565459da0 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.281+0000 7f916390c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f91581aec20 con 0x5605654a5390 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.281+0000 7f916390c740 10 client.0 _ll_register_callbacks cb 0x7f91480044d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f91577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x5605658f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f91577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x5605658f26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37256/0 (socket says 172.21.15.16:37256) 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f91577fe640 1 -- 172.21.15.16:0/3453405716 learned_addr learned my addr 172.21.15.16:0/3453405716 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f914ffff640 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x56056587b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f9156ffd640 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560565459da0 0x56056587af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f91577fe640 1 -- 172.21.15.16:0/3453405716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 msgr2=0x56056587b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f91577fe640 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x56056587b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f91577fe640 1 -- 172.21.15.16:0/3453405716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560565459da0 msgr2=0x56056587af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f91577fe640 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560565459da0 0x56056587af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f91577fe640 1 -- 172.21.15.16:0/3453405716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605658f2bf0 con 0x5605654c6070 2024-08-28T19:30:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f91577fe640 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x5605658f26b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f915002f4d0 tx=0x7f915002af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea2a1e993ddecb8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f913e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f913e7fc640 1 -- 172.21.15.16:0/3453405716 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915004a040 con 0x5605654c6070 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f913e7fc640 1 -- 172.21.15.16:0/3453405716 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7f9150031650 con 0x5605654c6070 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f916390c740 1 -- 172.21.15.16:0/3453405716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91400ab650 con 0x5605654c6070 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f913e7fc640 1 -- 172.21.15.16:0/3453405716 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f9150009360 con 0x5605654c6070 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f916390c740 10 client.17889 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.282+0000 7f916390c740 1 -- 172.21.15.16:0/3453405716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5605658f3d20 con 0x5605654c6070 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.283+0000 7f913e7fc640 1 -- 172.21.15.16:0/3453405716 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91400ab650 con 0x5605654c6070 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.283+0000 7f913e7fc640 1 -- 172.21.15.16:0/3453405716 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7f9150043070 con 0x5605654c6070 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.283+0000 7f916390c740 10 client.17889 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.283+0000 7f916390c740 10 client.17889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.283+0000 7f916390c740 20 client.17889 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.283+0000 7f916390c740 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5605658f4210 0x5605658f6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:33.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.283+0000 7f916390c740 4 client.17889 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:30:33.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.283+0000 7f916390c740 1 -- 172.21.15.16:0/3453405716 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5605657e0020 con 0x5605658f4210 2024-08-28T19:30:33.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.283+0000 7f9156ffd640 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5605658f4210 0x5605658f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.284+0000 7f9156ffd640 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5605658f4210 0x5605658f6630 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da693a637b4f3bc9 server_cookie=4dbd48abd4e85c0f in_seq=0 out_seq=0 2024-08-28T19:30:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.284+0000 7f913e7fc640 10 client.17889 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.284+0000 7f913e7fc640 1 -- 172.21.15.16:0/3453405716 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3782 (crc 0 0 0) 0x5605657e0020 con 0x5605658f4210 2024-08-28T19:30:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.284+0000 7f913e7fc640 10 client.17889 handle_command_reply: tid=0 2024-08-28T19:30:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.284+0000 7f916390c740 1 client.17889 shutdown 2024-08-28T19:30:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f916390c740 1 -- 172.21.15.16:0/3453405716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 msgr2=0x5605658f26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f916390c740 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x5605658f26b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f915002f4d0 tx=0x7f915002af40 comp rx=0 tx=0).stop 2024-08-28T19:30:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f916390c740 1 -- 172.21.15.16:0/3453405716 shutdown_connections 2024-08-28T19:30:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f916390c740 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605654c6070 0x5605658f26b0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f916390c740 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5605658f4210 0x5605658f6630 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f916390c740 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5605654a5390 0x56056587b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f916390c740 1 --2- 172.21.15.16:0/3453405716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560565459da0 0x56056587af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f916390c740 1 -- 172.21.15.16:0/3453405716 >> 172.21.15.16:0/3453405716 conn(0x5605654ba460 msgr2=0x56056586e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f913e7fc640 0 client.17889 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f916390c740 1 -- 172.21.15.16:0/3453405716 shutdown_connections 2024-08-28T19:30:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f916390c740 1 -- 172.21.15.16:0/3453405716 wait complete. 2024-08-28T19:30:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.285+0000 7f916390c740 20 client.17889 trim_cache size 0 max 16384 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.292 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.293 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:30:33.295 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.296 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:30:33.297 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:33.298 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:33.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:33.299 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:30:33.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.286+0000 7f915d397640 1 -- 172.21.15.16:0/3118651756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91440b4d20 msgr2=0x7f91440b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:33.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.286+0000 7f915d397640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91440b4d20 0x7f91440b7230 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x560565869f80 tx=0x7f9148005f90 comp rx=0 tx=0).stop 2024-08-28T19:30:33.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.287+0000 7f915d397640 1 -- 172.21.15.16:0/3118651756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581050f0 msgr2=0x7f915819a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:33.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.287+0000 7f915d397640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581050f0 0x7f915819a7b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f915806b720 tx=0x7f914000ee50 comp rx=0 tx=0).stop 2024-08-28T19:30:33.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.287+0000 7f915d397640 1 -- 172.21.15.16:0/3118651756 shutdown_connections 2024-08-28T19:30:33.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.287+0000 7f915d397640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915810dc00 0x7f915819fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.287+0000 7f915d397640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91440b4d20 0x7f91440b7230 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.287+0000 7f915d397640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581050f0 0x7f915819a7b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.287+0000 7f915d397640 1 --2- 172.21.15.16:0/3118651756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91581047a0 0x7f915819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:33.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.287+0000 7f915d397640 1 -- 172.21.15.16:0/3118651756 >> 172.21.15.16:0/3118651756 conn(0x7f9158079450 msgr2=0x7f91580ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:33.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.287+0000 7f915d397640 1 -- 172.21.15.16:0/3118651756 shutdown_connections 2024-08-28T19:30:33.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:33.287+0000 7f915d397640 1 -- 172.21.15.16:0/3118651756 wait complete. 2024-08-28T19:30:33.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:33 smithi016 ceph-mon[18556]: pgmap v630: 129 pgs: 129 active+clean; 295 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-08-28T19:30:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:33 smithi028 ceph-mon[21769]: pgmap v630: 129 pgs: 129 active+clean; 295 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-08-28T19:30:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:33 smithi115 ceph-mon[22184]: pgmap v630: 129 pgs: 129 active+clean; 295 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-08-28T19:30:33.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 427313 4.24 MB/sec execute 373 sec latency 42.037 ms 2024-08-28T19:30:34.117 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:30:34.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:30:34.371 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:34.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.862+0000 7f9b9cff1640 1 -- 172.21.15.16:0/498014516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b98109b90 msgr2=0x7f9b981103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:34.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.862+0000 7f9b9cff1640 1 --2- 172.21.15.16:0/498014516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b98109b90 0x7f9b981103c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009ca0 tx=0x7f9b9002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:30:34.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.863+0000 7f9b9cff1640 1 -- 172.21.15.16:0/498014516 shutdown_connections 2024-08-28T19:30:34.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.863+0000 7f9b9cff1640 1 --2- 172.21.15.16:0/498014516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b98109b90 0x7f9b981103c0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:34.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.863+0000 7f9b9cff1640 1 --2- 172.21.15.16:0/498014516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98109240 0x7f9b98109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:34.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.863+0000 7f9b9cff1640 1 --2- 172.21.15.16:0/498014516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b98103900 0x7f9b98103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:34.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.863+0000 7f9b9cff1640 1 -- 172.21.15.16:0/498014516 >> 172.21.15.16:0/498014516 conn(0x7f9b9807b5f0 msgr2=0x7f9b9807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.863+0000 7f9b9cff1640 1 -- 172.21.15.16:0/498014516 shutdown_connections 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.863+0000 7f9b9cff1640 1 -- 172.21.15.16:0/498014516 wait complete. 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.863+0000 7f9b9cff1640 1 Processor -- start 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.863+0000 7f9b9cff1640 1 -- start start 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b9cff1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98103900 0x7f9b9819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b9cff1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b98109240 0x7f9b9819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b9cff1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b98109b90 0x7f9b9819fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b9cff1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9b98113850 con 0x7f9b98109240 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b9cff1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b981136d0 con 0x7f9b98103900 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b9cff1640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9b981139d0 con 0x7f9b98109b90 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b95d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b98109240 0x7f9b9819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b95d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b98109240 0x7f9b9819a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32844/0 (socket says 172.21.15.16:32844) 2024-08-28T19:30:34.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b95d9b640 1 -- 172.21.15.16:0/2141286750 learned_addr learned my addr 172.21.15.16:0/2141286750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b9659c640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98103900 0x7f9b9819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b96d9d640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b98109b90 0x7f9b9819fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b95d9b640 1 -- 172.21.15.16:0/2141286750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b98109b90 msgr2=0x7f9b9819fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b95d9b640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b98109b90 0x7f9b9819fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b95d9b640 1 -- 172.21.15.16:0/2141286750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98103900 msgr2=0x7f9b9819a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b95d9b640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98103900 0x7f9b9819a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b95d9b640 1 -- 172.21.15.16:0/2141286750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b981a0090 con 0x7f9b98109240 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b96d9d640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b98109b90 0x7f9b9819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b95d9b640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b98109240 0x7f9b9819a780 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00c8c0 tx=0x7f9b8c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.864+0000 7f9b877fe640 1 -- 172.21.15.16:0/2141286750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c007bb0 con 0x7f9b98109240 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.865+0000 7f9b9cff1640 1 -- 172.21.15.16:0/2141286750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b981a0380 con 0x7f9b98109240 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.865+0000 7f9b877fe640 1 -- 172.21.15.16:0/2141286750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b8c007d50 con 0x7f9b98109240 2024-08-28T19:30:34.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.865+0000 7f9b877fe640 1 -- 172.21.15.16:0/2141286750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0186a0 con 0x7f9b98109240 2024-08-28T19:30:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.865+0000 7f9b9cff1640 1 -- 172.21.15.16:0/2141286750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b981a6ba0 con 0x7f9b98109240 2024-08-28T19:30:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.866+0000 7f9b9cff1640 1 -- 172.21.15.16:0/2141286750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b98103d10 con 0x7f9b98109240 2024-08-28T19:30:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.867+0000 7f9b877fe640 1 -- 172.21.15.16:0/2141286750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f9b8c00f450 con 0x7f9b98109240 2024-08-28T19:30:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.868+0000 7f9b877fe640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9b800b4cd0 0x7f9b800b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.868+0000 7f9b9659c640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9b800b4cd0 0x7f9b800b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.868+0000 7f9b877fe640 1 -- 172.21.15.16:0/2141286750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7f9b8c0e5ca0 con 0x7f9b98109240 2024-08-28T19:30:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.868+0000 7f9b9659c640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9b800b4cd0 0x7f9b800b71e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800abe0 tx=0x7f9b8800a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:34.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:34.871+0000 7f9b877fe640 1 -- 172.21.15.16:0/2141286750 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f9b8c014030 con 0x7f9b98109240 2024-08-28T19:30:34.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 427365 4.22 MB/sec execute 374 sec latency 951.569 ms 2024-08-28T19:30:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.099+0000 7f9b9cff1640 1 -- 172.21.15.16:0/2141286750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b9819b310 con 0x7f9b98109240 2024-08-28T19:30:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.100+0000 7f9b877fe640 1 -- 172.21.15.16:0/2141286750 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11980 (secure 0 0 0) 0x7f9b9819b310 con 0x7f9b98109240 2024-08-28T19:30:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:30:35.101 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"btime":"2024-08-28T19:30:14:226103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:30:13.285656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:30:35.102 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 53 2024-08-28T19:30:35.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 -- 172.21.15.16:0/2141286750 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9b800b4cd0 msgr2=0x7f9b800b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9b800b4cd0 0x7f9b800b71e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800abe0 tx=0x7f9b8800a2d0 comp rx=0 tx=0).stop 2024-08-28T19:30:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 -- 172.21.15.16:0/2141286750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b98109240 msgr2=0x7f9b9819a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b98109240 0x7f9b9819a780 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00c8c0 tx=0x7f9b8c00ccc0 comp rx=0 tx=0).stop 2024-08-28T19:30:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 -- 172.21.15.16:0/2141286750 shutdown_connections 2024-08-28T19:30:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b98109b90 0x7f9b9819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b98109240 0x7f9b9819a780 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9b800b4cd0 0x7f9b800b71e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 --2- 172.21.15.16:0/2141286750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98103900 0x7f9b9819a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 -- 172.21.15.16:0/2141286750 >> 172.21.15.16:0/2141286750 conn(0x7f9b9807b5f0 msgr2=0x7f9b98107230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 -- 172.21.15.16:0/2141286750 shutdown_connections 2024-08-28T19:30:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:35.102+0000 7f9b9cff1640 1 -- 172.21.15.16:0/2141286750 wait complete. 2024-08-28T19:30:35.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:35 smithi016 ceph-mon[18556]: pgmap v631: 129 pgs: 129 active+clean; 295 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 149 op/s 2024-08-28T19:30:35.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:35 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2141286750' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:35 smithi028 ceph-mon[21769]: pgmap v631: 129 pgs: 129 active+clean; 295 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 149 op/s 2024-08-28T19:30:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:35 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2141286750' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:35 smithi115 ceph-mon[22184]: pgmap v631: 129 pgs: 129 active+clean; 295 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 149 op/s 2024-08-28T19:30:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:35 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2141286750' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:35.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:30:35.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 427365 4.21 MB/sec execute 375 sec latency 1951.616 ms 2024-08-28T19:30:36.045 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.492+0000 7ffa2913b640 1 -- 172.21.15.16:0/575485505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2410dbb0 msgr2=0x7ffa241100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.492+0000 7ffa2913b640 1 --2- 172.21.15.16:0/575485505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2410dbb0 0x7ffa241100a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c009ca0 tx=0x7ffa1c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.493+0000 7ffa2913b640 1 -- 172.21.15.16:0/575485505 shutdown_connections 2024-08-28T19:30:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.493+0000 7ffa2913b640 1 --2- 172.21.15.16:0/575485505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2410dbb0 0x7ffa241100a0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.493+0000 7ffa2913b640 1 --2- 172.21.15.16:0/575485505 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa241050a0 0x7ffa2410d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.493+0000 7ffa2913b640 1 --2- 172.21.15.16:0/575485505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa24104750 0x7ffa24104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.493+0000 7ffa2913b640 1 -- 172.21.15.16:0/575485505 >> 172.21.15.16:0/575485505 conn(0x7ffa240794d0 msgr2=0x7ffa240798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:36.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.493+0000 7ffa2913b640 1 -- 172.21.15.16:0/575485505 shutdown_connections 2024-08-28T19:30:36.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.493+0000 7ffa2913b640 1 -- 172.21.15.16:0/575485505 wait complete. 2024-08-28T19:30:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.493+0000 7ffa2913b640 1 Processor -- start 2024-08-28T19:30:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa2913b640 1 -- start start 2024-08-28T19:30:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa2913b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa24104750 0x7ffa2419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa2913b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa241050a0 0x7ffa2419a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa2913b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2410dbb0 0x7ffa2419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa2913b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa24113830 con 0x7ffa2410dbb0 2024-08-28T19:30:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa2913b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241136b0 con 0x7ffa24104750 2024-08-28T19:30:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa2913b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241139b0 con 0x7ffa241050a0 2024-08-28T19:30:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa2359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2410dbb0 0x7ffa2419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa2359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2410dbb0 0x7ffa2419fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32870/0 (socket says 172.21.15.16:32870) 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa2359e640 1 -- 172.21.15.16:0/1912257907 learned_addr learned my addr 172.21.15.16:0/1912257907 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa24104750 0x7ffa2419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.494+0000 7ffa2259c640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa241050a0 0x7ffa2419a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa2359e640 1 -- 172.21.15.16:0/1912257907 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa241050a0 msgr2=0x7ffa2419a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa2359e640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa241050a0 0x7ffa2419a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa2359e640 1 -- 172.21.15.16:0/1912257907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa24104750 msgr2=0x7ffa2419a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa2359e640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa24104750 0x7ffa2419a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa2359e640 1 -- 172.21.15.16:0/1912257907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa241a01f0 con 0x7ffa2410dbb0 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa24104750 0x7ffa2419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa2359e640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2410dbb0 0x7ffa2419fb80 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c005cc0 tx=0x7ffa1c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa0bfff640 1 -- 172.21.15.16:0/1912257907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c003bf0 con 0x7ffa2410dbb0 2024-08-28T19:30:36.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa0bfff640 1 -- 172.21.15.16:0/1912257907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa1c003d90 con 0x7ffa2410dbb0 2024-08-28T19:30:36.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa2913b640 1 -- 172.21.15.16:0/1912257907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa241a0480 con 0x7ffa2410dbb0 2024-08-28T19:30:36.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa0bfff640 1 -- 172.21.15.16:0/1912257907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c02fbc0 con 0x7ffa2410dbb0 2024-08-28T19:30:36.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.495+0000 7ffa2913b640 1 -- 172.21.15.16:0/1912257907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa241a09c0 con 0x7ffa2410dbb0 2024-08-28T19:30:36.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2f6b0740 1 Processor -- start 2024-08-28T19:30:36.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa0bfff640 1 -- 172.21.15.16:0/1912257907 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ffa1c02fd60 con 0x7ffa2410dbb0 2024-08-28T19:30:36.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2f6b0740 1 -- start start 2024-08-28T19:30:36.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d17a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdac1b0 0x55567d17a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55567d0f1680 con 0x55567cdb4390 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55567d0e0c10 con 0x55567cdd5070 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55567d0e4bc0 con 0x55567cdac1b0 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32884/0 (socket says 172.21.15.16:32884) 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2259c640 1 -- 172.21.15.16:0/3541414411 learned_addr learned my addr 172.21.15.16:0/3541414411 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa2259c640 1 --2- 172.21.15.16:0/3541414411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d17a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/3541414411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdac1b0 0x55567d17a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.498+0000 7ffa0bfff640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa0c0b4cd0 0x7ffa0c0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa2359e640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa0c0b4cd0 0x7ffa0c0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa2259c640 1 -- 172.21.15.16:0/3541414411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdac1b0 msgr2=0x55567d17a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa2259c640 1 --2- 172.21.15.16:0/3541414411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdac1b0 0x55567d17a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa2259c640 1 -- 172.21.15.16:0/3541414411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 msgr2=0x55567d17a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa2259c640 1 --2- 172.21.15.16:0/3541414411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d17a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa2259c640 1 -- 172.21.15.16:0/3541414411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55567ce40650 con 0x55567cdb4390 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa0bfff640 1 -- 172.21.15.16:0/1912257907 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7ffa1c106d40 con 0x7ffa2410dbb0 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa2259c640 1 --2- 172.21.15.16:0/3541414411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17ad90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7ffa180058c0 tx=0x7ffa1800c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c477b14c0ad219 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa2359e640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa0c0b4cd0 0x7ffa0c0b71e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c1046c0 tx=0x7ffa1c091f10 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa097fa640 1 -- 172.21.15.16:0/3541414411 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18019020 con 0x55567cdb4390 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa097fa640 1 -- 172.21.15.16:0/3541414411 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa18011bb0 con 0x55567cdb4390 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.499+0000 7ffa097fa640 1 -- 172.21.15.16:0/3541414411 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1800fb10 con 0x55567cdb4390 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.500+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3541414411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 msgr2=0x55567d17ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.500+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3541414411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17ad90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7ffa180058c0 tx=0x7ffa1800c3f0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.500+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3541414411 shutdown_connections 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.500+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3541414411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17ad90 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.500+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3541414411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdac1b0 0x55567d17a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.500+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3541414411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d17a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.500+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3541414411 >> 172.21.15.16:0/3541414411 conn(0x55567cdc9460 msgr2=0x55567d178200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.500+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3541414411 shutdown_connections 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.500+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3541414411 wait complete. 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2f6b0740 1 -- start start 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2f6b0740 1 Processor -- start 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d185ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdb4390 0x55567d185ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d191de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55567cf43700 con 0x55567cdac1b0 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55567d067750 con 0x55567cdd5070 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55567d188af0 con 0x55567cdb4390 2024-08-28T19:30:36.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2f6b0740 10 client.0 _ll_register_callbacks cb 0x55567d096e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d185ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d185ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32892/0 (socket says 172.21.15.16:32892) 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2259c640 1 -- 172.21.15.16:0/3732606367 learned_addr learned my addr 172.21.15.16:0/3732606367 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2259c640 1 -- 172.21.15.16:0/3732606367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdb4390 msgr2=0x55567d185ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2259c640 1 --2- 172.21.15.16:0/3732606367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdb4390 0x55567d185ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2259c640 1 -- 172.21.15.16:0/3732606367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 msgr2=0x55567d191de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2259c640 1 --2- 172.21.15.16:0/3732606367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d191de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.501+0000 7ffa2259c640 1 -- 172.21.15.16:0/3732606367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55567d0a0900 con 0x55567cdac1b0 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.502+0000 7ffa2259c640 1 --2- 172.21.15.16:0/3732606367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d185ab0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800d010 tx=0x7ffa18014f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfbe02684854dab server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.502+0000 7ffa06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.502+0000 7ffa06ff5640 1 -- 172.21.15.16:0/3732606367 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18019030 con 0x55567cdac1b0 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.502+0000 7ffa06ff5640 1 -- 172.21.15.16:0/3732606367 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7ffa18005240 con 0x55567cdac1b0 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.502+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3732606367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa18011bb0 con 0x55567cdac1b0 2024-08-28T19:30:36.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.502+0000 7ffa06ff5640 1 -- 172.21.15.16:0/3732606367 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7ffa18033050 con 0x55567cdac1b0 2024-08-28T19:30:36.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.502+0000 7ffa2f6b0740 10 client.17907 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:36.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.502+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3732606367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55567d1948f0 con 0x55567cdac1b0 2024-08-28T19:30:36.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.502+0000 7ffa06ff5640 1 -- 172.21.15.16:0/3732606367 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa18011bb0 con 0x55567cdac1b0 2024-08-28T19:30:36.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.502+0000 7ffa06ff5640 1 -- 172.21.15.16:0/3732606367 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7ffa180046c0 con 0x55567cdac1b0 2024-08-28T19:30:36.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.503+0000 7ffa2f6b0740 10 client.17907 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.503+0000 7ffa2f6b0740 10 client.17907 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.503+0000 7ffa2f6b0740 20 client.17907 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.503+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3732606367 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55567cd939f0 0x55567d197c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.503+0000 7ffa2f6b0740 4 client.17907 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.503+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3732606367 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55567d0f35d0 con 0x55567cd939f0 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.503+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/3732606367 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55567cd939f0 0x55567d197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.503+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/3732606367 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55567cd939f0 0x55567d197c50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6146ead4e8ac4af3 server_cookie=e405973ac6046270 in_seq=0 out_seq=0 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.503+0000 7ffa06ff5640 10 client.17907 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa06ff5640 1 -- 172.21.15.16:0/3732606367 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55567d0f35d0 con 0x55567cd939f0 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa06ff5640 10 client.17907 handle_command_reply: tid=0 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa2f6b0740 1 client.17907 shutdown 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3732606367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 msgr2=0x55567d185ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3732606367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d185ab0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800d010 tx=0x7ffa18014f10 comp rx=0 tx=0).stop 2024-08-28T19:30:36.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3732606367 shutdown_connections 2024-08-28T19:30:36.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3732606367 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55567cd939f0 0x55567d197c50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3732606367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d191de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3732606367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdb4390 0x55567d185ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3732606367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d185ab0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa06ff5640 0 client.17907 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:30:36.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.504+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3732606367 >> 172.21.15.16:0/3732606367 conn(0x55567cdc9460 msgr2=0x55567d17d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:36.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.505+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3732606367 shutdown_connections 2024-08-28T19:30:36.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.505+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3732606367 wait complete. 2024-08-28T19:30:36.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.505+0000 7ffa2f6b0740 20 client.17907 trim_cache size 0 max 16384 2024-08-28T19:30:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa2f6b0740 1 Processor -- start 2024-08-28T19:30:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa2f6b0740 1 -- start start 2024-08-28T19:30:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdac1b0 0x55567d187aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d191ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241b0cd0 con 0x55567cdb4390 2024-08-28T19:30:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241b2870 con 0x55567cdd5070 2024-08-28T19:30:36.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241b1400 con 0x55567cdac1b0 2024-08-28T19:30:36.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa22d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa22d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32902/0 (socket says 172.21.15.16:32902) 2024-08-28T19:30:36.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa22d9d640 1 -- 172.21.15.16:0/3166641356 learned_addr learned my addr 172.21.15.16:0/3166641356 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:36.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa2259c640 1 --2- 172.21.15.16:0/3166641356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d191ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa2259c640 1 --2- 172.21.15.16:0/3166641356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdac1b0 0x55567d187aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa22d9d640 1 -- 172.21.15.16:0/3166641356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdac1b0 msgr2=0x55567d187aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:36.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/3166641356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdac1b0 0x55567d187aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa22d9d640 1 -- 172.21.15.16:0/3166641356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 msgr2=0x55567d191ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.518+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/3166641356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d191ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.519+0000 7ffa22d9d640 1 -- 172.21.15.16:0/3166641356 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55567d186930 con 0x55567cdb4390 2024-08-28T19:30:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.519+0000 7ffa2259c640 1 --2- 172.21.15.16:0/3166641356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdac1b0 0x55567d187aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:30:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.519+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/3166641356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17bcb0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ffa14000c00 tx=0x7ffa140087e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb946e926dacbcc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.519+0000 7ffa08ff9640 1 -- 172.21.15.16:0/3166641356 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa14005020 con 0x55567cdb4390 2024-08-28T19:30:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.519+0000 7ffa08ff9640 1 -- 172.21.15.16:0/3166641356 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa1401d030 con 0x55567cdb4390 2024-08-28T19:30:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.519+0000 7ffa08ff9640 1 -- 172.21.15.16:0/3166641356 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa14004e50 con 0x55567cdb4390 2024-08-28T19:30:36.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3166641356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 msgr2=0x55567d17bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:36.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3166641356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17bcb0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ffa14000c00 tx=0x7ffa140087e0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3166641356 shutdown_connections 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3166641356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d191ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3166641356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdb4390 0x55567d17bcb0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/3166641356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdac1b0 0x55567d187aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3166641356 >> 172.21.15.16:0/3166641356 conn(0x55567cdc9460 msgr2=0x55567d17f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3166641356 shutdown_connections 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/3166641356 wait complete. 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 -- start start 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 Processor -- start 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d189ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdb4390 0x55567d18a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d18a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241b21a0 con 0x55567cdac1b0 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241b1ad0 con 0x55567cdd5070 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241aecd0 con 0x55567cdb4390 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.520+0000 7ffa2f6b0740 10 client.0 _ll_register_callbacks cb 0x7ffa14008390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d189ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d189ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32916/0 (socket says 172.21.15.16:32916) 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2259c640 1 -- 172.21.15.16:0/1069494975 learned_addr learned my addr 172.21.15.16:0/1069494975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdb4390 0x55567d18a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2259c640 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d18a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2259c640 1 -- 172.21.15.16:0/1069494975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdb4390 msgr2=0x55567d18a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2259c640 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdb4390 0x55567d18a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2259c640 1 -- 172.21.15.16:0/1069494975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 msgr2=0x55567d18a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2259c640 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d18a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2259c640 1 -- 172.21.15.16:0/1069494975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55567d201680 con 0x55567cdac1b0 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdb4390 0x55567d18a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2259c640 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d189ef0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7ffa18004c00 tx=0x7ffa18014f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20b8a0adefc4c00 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa06ff5640 1 -- 172.21.15.16:0/1069494975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1802f070 con 0x55567cdac1b0 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa06ff5640 1 -- 172.21.15.16:0/1069494975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7ffa18004680 con 0x55567cdac1b0 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/1069494975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa1401d030 con 0x55567cdac1b0 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa06ff5640 1 -- 172.21.15.16:0/1069494975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7ffa18048050 con 0x55567cdac1b0 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2f6b0740 10 client.17913 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.521+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/1069494975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55567d202850 con 0x55567cdac1b0 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.522+0000 7ffa06ff5640 1 -- 172.21.15.16:0/1069494975 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa1401d030 con 0x55567cdac1b0 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.522+0000 7ffa06ff5640 1 -- 172.21.15.16:0/1069494975 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7ffa1801f630 con 0x55567cdac1b0 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.522+0000 7ffa2f6b0740 10 client.17913 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.522+0000 7ffa2f6b0740 10 client.17913 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.522+0000 7ffa2f6b0740 20 client.17913 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.522+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55567cd68da0 0x55567d204ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.522+0000 7ffa2f6b0740 4 client.17913 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.522+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/1069494975 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55567d0ef070 con 0x55567cd68da0 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.522+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55567cd68da0 0x55567d204ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa06ff5640 10 client.17913 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa22d9d640 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55567cd68da0 0x55567d204ef0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=60d66597b3f78fde server_cookie=36b117e4ee6bd298 in_seq=0 out_seq=0 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa06ff5640 1 -- 172.21.15.16:0/1069494975 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55567d0ef070 con 0x55567cd68da0 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa06ff5640 10 client.17913 handle_command_reply: tid=0 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 1 client.17913 shutdown 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/1069494975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 msgr2=0x55567d189ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d189ef0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7ffa18004c00 tx=0x7ffa18014f10 comp rx=0 tx=0).stop 2024-08-28T19:30:36.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/1069494975 shutdown_connections 2024-08-28T19:30:36.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55567cd68da0 0x55567d204ef0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55567cdd5070 0x55567d18a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55567cdb4390 0x55567d18a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa06ff5640 0 client.17913 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:30:36.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 1 --2- 172.21.15.16:0/1069494975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55567cdac1b0 0x55567d189ef0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/1069494975 >> 172.21.15.16:0/1069494975 conn(0x55567cdc9460 msgr2=0x55567d17b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:36.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/1069494975 shutdown_connections 2024-08-28T19:30:36.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 1 -- 172.21.15.16:0/1069494975 wait complete. 2024-08-28T19:30:36.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.523+0000 7ffa2f6b0740 20 client.17913 trim_cache size 0 max 16384 2024-08-28T19:30:36.530 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:30:36.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 -- 172.21.15.16:0/1912257907 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa0c0b4cd0 msgr2=0x7ffa0c0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:36.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa0c0b4cd0 0x7ffa0c0b71e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c1046c0 tx=0x7ffa1c091f10 comp rx=0 tx=0).stop 2024-08-28T19:30:36.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 -- 172.21.15.16:0/1912257907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2410dbb0 msgr2=0x7ffa2419fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:36.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2410dbb0 0x7ffa2419fb80 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c005cc0 tx=0x7ffa1c02f380 comp rx=0 tx=0).stop 2024-08-28T19:30:36.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 -- 172.21.15.16:0/1912257907 shutdown_connections 2024-08-28T19:30:36.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2410dbb0 0x7ffa2419fb80 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffa241050a0 0x7ffa2419a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa0c0b4cd0 0x7ffa0c0b71e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 --2- 172.21.15.16:0/1912257907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa24104750 0x7ffa2419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:36.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 -- 172.21.15.16:0/1912257907 >> 172.21.15.16:0/1912257907 conn(0x7ffa240794d0 msgr2=0x7ffa240ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:36.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 -- 172.21.15.16:0/1912257907 shutdown_connections 2024-08-28T19:30:36.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:36.524+0000 7ffa2913b640 1 -- 172.21.15.16:0/1912257907 wait complete. 2024-08-28T19:30:36.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 427365 4.20 MB/sec execute 376 sec latency 2951.679 ms 2024-08-28T19:30:37.245 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:30:37.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:37 smithi016 ceph-mon[18556]: pgmap v632: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-08-28T19:30:37.507 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:37.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:37 smithi028 ceph-mon[21769]: pgmap v632: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-08-28T19:30:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:37 smithi115 ceph-mon[22184]: pgmap v632: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-08-28T19:30:37.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 429448 4.23 MB/sec execute 377 sec latency 2962.027 ms 2024-08-28T19:30:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.258+0000 7fc73ed5d640 1 -- 172.21.15.16:0/5651475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc738102d20 msgr2=0x7fc73810fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.258+0000 7fc73ed5d640 1 --2- 172.21.15.16:0/5651475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc738102d20 0x7fc73810fa40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc724009900 tx=0x7fc72402a480 comp rx=0 tx=0).stop 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.258+0000 7fc73ed5d640 1 -- 172.21.15.16:0/5651475 shutdown_connections 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.258+0000 7fc73ed5d640 1 --2- 172.21.15.16:0/5651475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc73810ff80 0x7fc7381123a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.258+0000 7fc73ed5d640 1 --2- 172.21.15.16:0/5651475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc738102d20 0x7fc73810fa40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.258+0000 7fc73ed5d640 1 --2- 172.21.15.16:0/5651475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7381023d0 0x7fc7381027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.258+0000 7fc73ed5d640 1 -- 172.21.15.16:0/5651475 >> 172.21.15.16:0/5651475 conn(0x7fc7380794d0 msgr2=0x7fc7380798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.258+0000 7fc73ed5d640 1 -- 172.21.15.16:0/5651475 shutdown_connections 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.258+0000 7fc73ed5d640 1 -- 172.21.15.16:0/5651475 wait complete. 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73ed5d640 1 Processor -- start 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73ed5d640 1 -- start start 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73ed5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7381023d0 0x7fc73819c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73ed5d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc738102d20 0x7fc73819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73ed5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc73810ff80 0x7fc7381a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73ed5d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381159f0 con 0x7fc73810ff80 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73ed5d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc738115870 con 0x7fc7381023d0 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73ed5d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc738115b70 con 0x7fc738102d20 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73caf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7381023d0 0x7fc73819c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73caf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7381023d0 0x7fc73819c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48702/0 (socket says 172.21.15.16:48702) 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73caf9640 1 -- 172.21.15.16:0/3887429799 learned_addr learned my addr 172.21.15.16:0/3887429799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc73d2fa640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc73810ff80 0x7fc7381a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.259+0000 7fc737fff640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc738102d20 0x7fc73819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc73caf9640 1 -- 172.21.15.16:0/3887429799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc738102d20 msgr2=0x7fc73819c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc73caf9640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc738102d20 0x7fc73819c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc73caf9640 1 -- 172.21.15.16:0/3887429799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc73810ff80 msgr2=0x7fc7381a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc73caf9640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc73810ff80 0x7fc7381a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc73caf9640 1 -- 172.21.15.16:0/3887429799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7381a2390 con 0x7fc7381023d0 2024-08-28T19:30:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc737fff640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc738102d20 0x7fc73819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc73d2fa640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc73810ff80 0x7fc7381a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc73caf9640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7381023d0 0x7fc73819c420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00e9f0 tx=0x7fc72c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc735ffb640 1 -- 172.21.15.16:0/3887429799 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00ccc0 con 0x7fc7381023d0 2024-08-28T19:30:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc73ed5d640 1 -- 172.21.15.16:0/3887429799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7381a2680 con 0x7fc7381023d0 2024-08-28T19:30:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc735ffb640 1 -- 172.21.15.16:0/3887429799 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc72c00ce60 con 0x7fc7381023d0 2024-08-28T19:30:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc735ffb640 1 -- 172.21.15.16:0/3887429799 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c010430 con 0x7fc7381023d0 2024-08-28T19:30:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.260+0000 7fc73ed5d640 1 -- 172.21.15.16:0/3887429799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7381a8de0 con 0x7fc7381023d0 2024-08-28T19:30:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.263+0000 7fc735ffb640 1 -- 172.21.15.16:0/3887429799 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc72c004f50 con 0x7fc7381023d0 2024-08-28T19:30:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.263+0000 7fc7452d2740 1 Processor -- start 2024-08-28T19:30:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.263+0000 7fc7452d2740 1 -- start start 2024-08-28T19:30:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.263+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 0x55db6b81f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b81f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b81fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc7452d2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55db6b796680 con 0x55db6b47a070 2024-08-28T19:30:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc7452d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db6b785c10 con 0x55db6b4511b0 2024-08-28T19:30:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc7452d2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55db6b789bc0 con 0x55db6b459390 2024-08-28T19:30:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc73d2fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 0x55db6b81f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc73caf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b81f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc73d2fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 0x55db6b81f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32952/0 (socket says 172.21.15.16:32952) 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc73d2fa640 1 -- 172.21.15.16:0/210581276 learned_addr learned my addr 172.21.15.16:0/210581276 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc735ffb640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7280b4ca0 0x7fc7280b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc737fff640 1 --2- 172.21.15.16:0/210581276 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b81fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc73d2fa640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7280b4ca0 0x7fc7280b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc735ffb640 1 -- 172.21.15.16:0/3887429799 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fc72c0e5d60 con 0x7fc7381023d0 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc737fff640 1 -- 172.21.15.16:0/210581276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 msgr2=0x55db6b81f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc737fff640 1 --2- 172.21.15.16:0/210581276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b81f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc737fff640 1 -- 172.21.15.16:0/210581276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 msgr2=0x55db6b81f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc737fff640 1 --2- 172.21.15.16:0/210581276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 0x55db6b81f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc737fff640 1 -- 172.21.15.16:0/210581276 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db6b4e5650 con 0x55db6b459390 2024-08-28T19:30:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.264+0000 7fc73d2fa640 1 --2- 172.21.15.16:0/210581276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 0x55db6b81f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.265+0000 7fc737fff640 1 --2- 172.21.15.16:0/210581276 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b81fd90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc724009230 tx=0x7fc724005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=744f328da5c4c4f3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.265+0000 7fc73d2fa640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7280b4ca0 0x7fc7280b71b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc73000f480 tx=0x7fc730004ab0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.265+0000 7fc722ffd640 1 -- 172.21.15.16:0/210581276 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724040070 con 0x55db6b459390 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.265+0000 7fc722ffd640 1 -- 172.21.15.16:0/210581276 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc724031720 con 0x55db6b459390 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.265+0000 7fc722ffd640 1 -- 172.21.15.16:0/210581276 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724035040 con 0x55db6b459390 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.266+0000 7fc7452d2740 1 -- 172.21.15.16:0/210581276 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 msgr2=0x55db6b81fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.266+0000 7fc7452d2740 1 --2- 172.21.15.16:0/210581276 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b81fd90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc724009230 tx=0x7fc724005e20 comp rx=0 tx=0).stop 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.266+0000 7fc7452d2740 1 -- 172.21.15.16:0/210581276 shutdown_connections 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.266+0000 7fc7452d2740 1 --2- 172.21.15.16:0/210581276 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b81fd90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.266+0000 7fc7452d2740 1 --2- 172.21.15.16:0/210581276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b81f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.266+0000 7fc7452d2740 1 --2- 172.21.15.16:0/210581276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 0x55db6b81f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.266+0000 7fc7452d2740 1 -- 172.21.15.16:0/210581276 >> 172.21.15.16:0/210581276 conn(0x55db6b46e460 msgr2=0x55db6b81d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.266+0000 7fc7452d2740 1 -- 172.21.15.16:0/210581276 shutdown_connections 2024-08-28T19:30:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.266+0000 7fc7452d2740 1 -- 172.21.15.16:0/210581276 wait complete. 2024-08-28T19:30:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.266+0000 7fc7452d2740 1 -- start start 2024-08-28T19:30:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.266+0000 7fc7452d2740 1 Processor -- start 2024-08-28T19:30:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b82aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b82af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 0x55db6b836d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc7452d2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55db6b5e8700 con 0x55db6b47a070 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc7452d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db6b70c750 con 0x55db6b4511b0 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc7452d2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55db6b82da90 con 0x55db6b459390 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc7452d2740 10 client.0 _ll_register_callbacks cb 0x55db6b73be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc73caf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b82aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc737fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b82af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc73caf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b82aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48728/0 (socket says 172.21.15.16:48728) 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc73d2fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 0x55db6b836d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc73caf9640 1 -- 172.21.15.16:0/2313595879 learned_addr learned my addr 172.21.15.16:0/2313595879 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc73caf9640 1 -- 172.21.15.16:0/2313595879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 msgr2=0x55db6b82af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc73caf9640 1 --2- 172.21.15.16:0/2313595879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b82af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc73caf9640 1 -- 172.21.15.16:0/2313595879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 msgr2=0x55db6b836d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc73caf9640 1 --2- 172.21.15.16:0/2313595879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 0x55db6b836d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.267+0000 7fc73caf9640 1 -- 172.21.15.16:0/2313595879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db6b745900 con 0x55db6b4511b0 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.268+0000 7fc73caf9640 1 --2- 172.21.15.16:0/2313595879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b82aa30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00e9c0 tx=0x7fc72c004980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca30500aaaefe26b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.268+0000 7fc717fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.268+0000 7fc717fff640 1 -- 172.21.15.16:0/2313595879 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c0abb30 con 0x55db6b4511b0 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.268+0000 7fc717fff640 1 -- 172.21.15.16:0/2313595879 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fc72c0e3300 con 0x55db6b4511b0 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.268+0000 7fc7452d2740 1 -- 172.21.15.16:0/2313595879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc724031720 con 0x55db6b4511b0 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc717fff640 1 -- 172.21.15.16:0/2313595879 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc72c0e3500 con 0x55db6b4511b0 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc7452d2740 10 client.27437 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc7452d2740 1 -- 172.21.15.16:0/2313595879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55db6b839800 con 0x55db6b4511b0 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc717fff640 1 -- 172.21.15.16:0/2313595879 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc724031720 con 0x55db6b4511b0 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc717fff640 1 -- 172.21.15.16:0/2313595879 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fc72c097190 con 0x55db6b4511b0 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc7452d2740 10 client.27437 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc7452d2740 10 client.27437 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:30:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc7452d2740 20 client.27437 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2313595879 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55db6b4389f0 0x55db6b83ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc7452d2740 4 client.27437 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc7452d2740 1 -- 172.21.15.16:0/2313595879 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db6b7985d0 con 0x55db6b4389f0 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.269+0000 7fc737fff640 1 --2- 172.21.15.16:0/2313595879 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55db6b4389f0 0x55db6b83ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.270+0000 7fc717fff640 10 client.27437 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.270+0000 7fc737fff640 1 --2- 172.21.15.16:0/2313595879 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55db6b4389f0 0x55db6b83ce00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=57c5ae100737e141 server_cookie=32634633dd9f9e04 in_seq=0 out_seq=0 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.270+0000 7fc717fff640 1 -- 172.21.15.16:0/2313595879 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55db6b7985d0 con 0x55db6b4389f0 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.270+0000 7fc717fff640 10 client.27437 handle_command_reply: tid=0 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 1 client.27437 shutdown 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 1 -- 172.21.15.16:0/2313595879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 msgr2=0x55db6b82aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2313595879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b82aa30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00e9c0 tx=0x7fc72c004980 comp rx=0 tx=0).stop 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 1 -- 172.21.15.16:0/2313595879 shutdown_connections 2024-08-28T19:30:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2313595879 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55db6b4389f0 0x55db6b83ce00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2313595879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b47a070 0x55db6b836d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2313595879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b82af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc717fff640 0 client.27437 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:30:38.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2313595879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b82aa30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 1 -- 172.21.15.16:0/2313595879 >> 172.21.15.16:0/2313595879 conn(0x55db6b46e460 msgr2=0x55db6b8224f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:38.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 1 -- 172.21.15.16:0/2313595879 shutdown_connections 2024-08-28T19:30:38.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 1 -- 172.21.15.16:0/2313595879 wait complete. 2024-08-28T19:30:38.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.271+0000 7fc7452d2740 20 client.27437 trim_cache size 0 max 16384 2024-08-28T19:30:38.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.284+0000 7fc7452d2740 1 Processor -- start 2024-08-28T19:30:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.284+0000 7fc7452d2740 1 -- start start 2024-08-28T19:30:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b822b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b459390 0x55db6b836c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b47a070 0x55db6b8371c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc7452d2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381b2e50 con 0x55db6b459390 2024-08-28T19:30:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc7452d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381b49f0 con 0x55db6b4511b0 2024-08-28T19:30:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc7452d2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381b3580 con 0x55db6b47a070 2024-08-28T19:30:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc737fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b459390 0x55db6b836c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc737fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b459390 0x55db6b836c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32974/0 (socket says 172.21.15.16:32974) 2024-08-28T19:30:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc737fff640 1 -- 172.21.15.16:0/5200352 learned_addr learned my addr 172.21.15.16:0/5200352 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc73caf9640 1 --2- 172.21.15.16:0/5200352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b822b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc73d2fa640 1 --2- 172.21.15.16:0/5200352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b47a070 0x55db6b8371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc737fff640 1 -- 172.21.15.16:0/5200352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b47a070 msgr2=0x55db6b8371c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc737fff640 1 --2- 172.21.15.16:0/5200352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b47a070 0x55db6b8371c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc737fff640 1 -- 172.21.15.16:0/5200352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 msgr2=0x55db6b822b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc737fff640 1 --2- 172.21.15.16:0/5200352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b822b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc737fff640 1 -- 172.21.15.16:0/5200352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db6b823060 con 0x55db6b459390 2024-08-28T19:30:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.285+0000 7fc737fff640 1 --2- 172.21.15.16:0/5200352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b459390 0x55db6b836c80 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc724033000 tx=0x7fc7240054d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84f4367766ad060 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.286+0000 7fc7227fc640 1 -- 172.21.15.16:0/5200352 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724035040 con 0x55db6b459390 2024-08-28T19:30:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.286+0000 7fc7227fc640 1 -- 172.21.15.16:0/5200352 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc724005540 con 0x55db6b459390 2024-08-28T19:30:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.286+0000 7fc7227fc640 1 -- 172.21.15.16:0/5200352 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724040040 con 0x55db6b459390 2024-08-28T19:30:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.287+0000 7fc7452d2740 1 -- 172.21.15.16:0/5200352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b459390 msgr2=0x55db6b836c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.287+0000 7fc7452d2740 1 --2- 172.21.15.16:0/5200352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b459390 0x55db6b836c80 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc724033000 tx=0x7fc7240054d0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.287+0000 7fc7452d2740 1 -- 172.21.15.16:0/5200352 shutdown_connections 2024-08-28T19:30:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.287+0000 7fc7452d2740 1 --2- 172.21.15.16:0/5200352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b47a070 0x55db6b8371c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.287+0000 7fc7452d2740 1 --2- 172.21.15.16:0/5200352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b459390 0x55db6b836c80 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.287+0000 7fc7452d2740 1 --2- 172.21.15.16:0/5200352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b4511b0 0x55db6b822b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.287+0000 7fc7452d2740 1 -- 172.21.15.16:0/5200352 >> 172.21.15.16:0/5200352 conn(0x55db6b46e460 msgr2=0x55db6b822410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.287+0000 7fc7452d2740 1 -- 172.21.15.16:0/5200352 shutdown_connections 2024-08-28T19:30:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.287+0000 7fc7452d2740 1 -- 172.21.15.16:0/5200352 wait complete. 2024-08-28T19:30:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.287+0000 7fc7452d2740 1 -- start start 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.287+0000 7fc7452d2740 1 Processor -- start 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b4511b0 0x55db6b8a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b8a1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc7452d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b47a070 0x55db6b82f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc7452d2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381b4320 con 0x55db6b4511b0 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc7452d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381b3c50 con 0x55db6b47a070 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc7452d2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381b0e50 con 0x55db6b459390 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc73caf9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b4511b0 0x55db6b8a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc737fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b8a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc73caf9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b4511b0 0x55db6b8a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32980/0 (socket says 172.21.15.16:32980) 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc73caf9640 1 -- 172.21.15.16:0/2345984937 learned_addr learned my addr 172.21.15.16:0/2345984937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc73d2fa640 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b47a070 0x55db6b82f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc7452d2740 10 client.0 _ll_register_callbacks cb 0x7fc72c009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc73caf9640 1 -- 172.21.15.16:0/2345984937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 msgr2=0x55db6b8a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc73caf9640 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b8a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc73caf9640 1 -- 172.21.15.16:0/2345984937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b47a070 msgr2=0x55db6b82f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc73caf9640 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b47a070 0x55db6b82f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc73caf9640 1 -- 172.21.15.16:0/2345984937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db6b82fc60 con 0x55db6b4511b0 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc737fff640 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b8a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc73caf9640 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b4511b0 0x55db6b8a0b80 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0e31c0 tx=0x7fc72c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c666e4b300c910be server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.288+0000 7fc717fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc717fff640 1 -- 172.21.15.16:0/2345984937 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c0ab790 con 0x55db6b4511b0 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc717fff640 1 -- 172.21.15.16:0/2345984937 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fc72c0ab280 con 0x55db6b4511b0 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc7452d2740 1 -- 172.21.15.16:0/2345984937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc724005540 con 0x55db6b4511b0 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc717fff640 1 -- 172.21.15.16:0/2345984937 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc72c0e3500 con 0x55db6b4511b0 2024-08-28T19:30:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc7452d2740 10 client.17931 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc7452d2740 1 -- 172.21.15.16:0/2345984937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55db6b8a8310 con 0x55db6b4511b0 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc717fff640 1 -- 172.21.15.16:0/2345984937 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc72c0ac720 con 0x55db6b4511b0 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc717fff640 1 -- 172.21.15.16:0/2345984937 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fc72c0a7b90 con 0x55db6b4511b0 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc7452d2740 10 client.17931 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc7452d2740 10 client.17931 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc7452d2740 20 client.17931 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.289+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55db6b40dda0 0x55db6b8aaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.290+0000 7fc7452d2740 4 client.17931 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.290+0000 7fc7452d2740 1 -- 172.21.15.16:0/2345984937 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc724005540 con 0x55db6b40dda0 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.290+0000 7fc737fff640 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55db6b40dda0 0x55db6b8aaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.290+0000 7fc717fff640 10 client.17931 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.290+0000 7fc737fff640 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55db6b40dda0 0x55db6b8aaad0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1bbe229a50ab6e45 server_cookie=ec96d96d14e16553 in_seq=0 out_seq=0 2024-08-28T19:30:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.290+0000 7fc717fff640 1 -- 172.21.15.16:0/2345984937 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fc724005540 con 0x55db6b40dda0 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.290+0000 7fc717fff640 10 client.17931 handle_command_reply: tid=0 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.290+0000 7fc7452d2740 1 client.17931 shutdown 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.290+0000 7fc7452d2740 1 -- 172.21.15.16:0/2345984937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b4511b0 msgr2=0x55db6b8a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.290+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b4511b0 0x55db6b8a0b80 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0e31c0 tx=0x7fc72c00c840 comp rx=0 tx=0).stop 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc7452d2740 1 -- 172.21.15.16:0/2345984937 shutdown_connections 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55db6b40dda0 0x55db6b8aaad0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db6b47a070 0x55db6b82f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db6b459390 0x55db6b8a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc7452d2740 1 --2- 172.21.15.16:0/2345984937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db6b4511b0 0x55db6b8a0b80 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc7452d2740 1 -- 172.21.15.16:0/2345984937 >> 172.21.15.16:0/2345984937 conn(0x55db6b46e460 msgr2=0x55db6b81cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc717fff640 0 client.17931 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc7452d2740 1 -- 172.21.15.16:0/2345984937 shutdown_connections 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc7452d2740 1 -- 172.21.15.16:0/2345984937 wait complete. 2024-08-28T19:30:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc7452d2740 20 client.17931 trim_cache size 0 max 16384 2024-08-28T19:30:38.297 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:30:38.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc73ed5d640 1 -- 172.21.15.16:0/3887429799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7280b4ca0 msgr2=0x7fc7280b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:38.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc73ed5d640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7280b4ca0 0x7fc7280b71b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc73000f480 tx=0x7fc730004ab0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc73ed5d640 1 -- 172.21.15.16:0/3887429799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7381023d0 msgr2=0x7fc73819c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.291+0000 7fc73ed5d640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7381023d0 0x7fc73819c420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00e9f0 tx=0x7fc72c00edf0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.292+0000 7fc73ed5d640 1 -- 172.21.15.16:0/3887429799 shutdown_connections 2024-08-28T19:30:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.292+0000 7fc73ed5d640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc73810ff80 0x7fc7381a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.292+0000 7fc73ed5d640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc738102d20 0x7fc73819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.292+0000 7fc73ed5d640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7280b4ca0 0x7fc7280b71b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.292+0000 7fc73ed5d640 1 --2- 172.21.15.16:0/3887429799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7381023d0 0x7fc73819c420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.292+0000 7fc73ed5d640 1 -- 172.21.15.16:0/3887429799 >> 172.21.15.16:0/3887429799 conn(0x7fc7380794d0 msgr2=0x7fc738111400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.292+0000 7fc73ed5d640 1 -- 172.21.15.16:0/3887429799 shutdown_connections 2024-08-28T19:30:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:38.292+0000 7fc73ed5d640 1 -- 172.21.15.16:0/3887429799 wait complete. 2024-08-28T19:30:38.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430474 4.22 MB/sec execute 378 sec latency 38.822 ms 2024-08-28T19:30:38.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:30:39.236 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:39 smithi028 ceph-mon[21769]: pgmap v633: 129 pgs: 129 active+clean; 299 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-08-28T19:30:39.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:39 smithi016 ceph-mon[18556]: pgmap v633: 129 pgs: 129 active+clean; 299 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-08-28T19:30:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:39 smithi115 ceph-mon[22184]: pgmap v633: 129 pgs: 129 active+clean; 299 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-08-28T19:30:39.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.697+0000 7fc30575d640 1 -- 172.21.15.16:0/4153667398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300108b10 msgr2=0x7fc30010f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:39.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.697+0000 7fc30575d640 1 --2- 172.21.15.16:0/4153667398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300108b10 0x7fc30010f340 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009900 tx=0x7fc2f802a410 comp rx=0 tx=0).stop 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.697+0000 7fc30575d640 1 -- 172.21.15.16:0/4153667398 shutdown_connections 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.697+0000 7fc30575d640 1 --2- 172.21.15.16:0/4153667398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300108b10 0x7fc30010f340 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.697+0000 7fc30575d640 1 --2- 172.21.15.16:0/4153667398 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3001081c0 0x7fc3001085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.697+0000 7fc30575d640 1 --2- 172.21.15.16:0/4153667398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc300102880 0x7fc300102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.697+0000 7fc30575d640 1 -- 172.21.15.16:0/4153667398 >> 172.21.15.16:0/4153667398 conn(0x7fc300079510 msgr2=0x7fc300079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.697+0000 7fc30575d640 1 -- 172.21.15.16:0/4153667398 shutdown_connections 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.697+0000 7fc30575d640 1 -- 172.21.15.16:0/4153667398 wait complete. 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.697+0000 7fc30575d640 1 Processor -- start 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc30575d640 1 -- start start 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc30575d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102880 0x7fc30019b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc30575d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3001081c0 0x7fc30019b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc30575d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc300108b10 0x7fc30019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc30575d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001127c0 con 0x7fc300102880 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc30575d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc300112640 con 0x7fc3001081c0 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc30575d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc300112940 con 0x7fc300108b10 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102880 0x7fc30019b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102880 0x7fc30019b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33000/0 (socket says 172.21.15.16:33000) 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2feffd640 1 -- 172.21.15.16:0/574576563 learned_addr learned my addr 172.21.15.16:0/574576563 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2ff7fe640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc300108b10 0x7fc30019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3001081c0 0x7fc30019b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2feffd640 1 -- 172.21.15.16:0/574576563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc300108b10 msgr2=0x7fc30019bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2feffd640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc300108b10 0x7fc30019bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2feffd640 1 -- 172.21.15.16:0/574576563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3001081c0 msgr2=0x7fc30019b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2feffd640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3001081c0 0x7fc30019b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2feffd640 1 -- 172.21.15.16:0/574576563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3001a6230 con 0x7fc300102880 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3001081c0 0x7fc30019b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.698+0000 7fc2feffd640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102880 0x7fc30019b030 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000eab0 tx=0x7fc2f000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.699+0000 7fc2f7fff640 1 -- 172.21.15.16:0/574576563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f000ce40 con 0x7fc300102880 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.699+0000 7fc30575d640 1 -- 172.21.15.16:0/574576563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3001a6520 con 0x7fc300102880 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.699+0000 7fc2f7fff640 1 -- 172.21.15.16:0/574576563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2f0011920 con 0x7fc300102880 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.699+0000 7fc2f7fff640 1 -- 172.21.15.16:0/574576563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0010a30 con 0x7fc300102880 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.699+0000 7fc30575d640 1 -- 172.21.15.16:0/574576563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3001a6a60 con 0x7fc300102880 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.701+0000 7fc30bcd2740 1 Processor -- start 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.701+0000 7fc30bcd2740 1 -- start start 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.701+0000 7fc2f7fff640 1 -- 172.21.15.16:0/574576563 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc2f0010bd0 con 0x7fc300102880 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4dc7070 0x564cd516c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4d9e1b0 0x564cd516c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd516cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564cd50e3680 con 0x564cd4da6390 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564cd50d2c10 con 0x564cd4dc7070 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564cd50d6bc0 con 0x564cd4d9e1b0 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2fe7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd516cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2fe7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd516cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33008/0 (socket says 172.21.15.16:33008) 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2fe7fc640 1 -- 172.21.15.16:0/75395388 learned_addr learned my addr 172.21.15.16:0/75395388 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2feffd640 1 --2- 172.21.15.16:0/75395388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4d9e1b0 0x564cd516c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2ff7fe640 1 --2- 172.21.15.16:0/75395388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4dc7070 0x564cd516c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2fe7fc640 1 -- 172.21.15.16:0/75395388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4d9e1b0 msgr2=0x564cd516c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/75395388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4d9e1b0 0x564cd516c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2fe7fc640 1 -- 172.21.15.16:0/75395388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4dc7070 msgr2=0x564cd516c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/75395388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4dc7070 0x564cd516c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2fe7fc640 1 -- 172.21.15.16:0/75395388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cd4e32650 con 0x564cd4da6390 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2f7fff640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ec0b4cd0 0x7fc2ec0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.702+0000 7fc2ff7fe640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ec0b4cd0 0x7fc2ec0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.703+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/75395388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd516cd90 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800bb80 tx=0x7fc2e800d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf520eeaeb5df62 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.703+0000 7fc2f7fff640 1 -- 172.21.15.16:0/574576563 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fc2f00e5fb0 con 0x7fc300102880 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.703+0000 7fc2f57fa640 1 -- 172.21.15.16:0/75395388 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8017620 con 0x564cd4da6390 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.703+0000 7fc2f57fa640 1 -- 172.21.15.16:0/75395388 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2e8017c70 con 0x564cd4da6390 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.703+0000 7fc2f57fa640 1 -- 172.21.15.16:0/75395388 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e800fa50 con 0x564cd4da6390 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.704+0000 7fc30bcd2740 1 -- 172.21.15.16:0/75395388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 msgr2=0x564cd516cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.704+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/75395388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd516cd90 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800bb80 tx=0x7fc2e800d710 comp rx=0 tx=0).stop 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.704+0000 7fc2ff7fe640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ec0b4cd0 0x7fc2ec0b71e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005940 tx=0x7fc2f8009660 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.704+0000 7fc30bcd2740 1 -- 172.21.15.16:0/75395388 shutdown_connections 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.704+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/75395388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd516cd90 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.704+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/75395388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4d9e1b0 0x564cd516c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.704+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/75395388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4dc7070 0x564cd516c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.704+0000 7fc30bcd2740 1 -- 172.21.15.16:0/75395388 >> 172.21.15.16:0/75395388 conn(0x564cd4dbb460 msgr2=0x564cd516a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.704+0000 7fc30bcd2740 1 -- 172.21.15.16:0/75395388 shutdown_connections 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.704+0000 7fc30bcd2740 1 -- 172.21.15.16:0/75395388 wait complete. 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc30bcd2740 1 -- start start 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc30bcd2740 1 Processor -- start 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd517be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4da6390 0x564cd517c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 0x564cd5188180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564cd4f35700 con 0x564cd4dc7070 2024-08-28T19:30:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564cd5059750 con 0x564cd4d9e1b0 2024-08-28T19:30:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564cd517ee50 con 0x564cd4da6390 2024-08-28T19:30:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc30bcd2740 10 client.0 _ll_register_callbacks cb 0x564cd5088e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc2feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd517be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc2feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd517be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48790/0 (socket says 172.21.15.16:48790) 2024-08-28T19:30:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc2ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 0x564cd5188180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc2feffd640 1 -- 172.21.15.16:0/4049523488 learned_addr learned my addr 172.21.15.16:0/4049523488 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.705+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4da6390 0x564cd517c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc2feffd640 1 -- 172.21.15.16:0/4049523488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4da6390 msgr2=0x564cd517c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:39.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc2feffd640 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4da6390 0x564cd517c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc2feffd640 1 -- 172.21.15.16:0/4049523488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 msgr2=0x564cd5188180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc2feffd640 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 0x564cd5188180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc2feffd640 1 -- 172.21.15.16:0/4049523488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cd5092900 con 0x564cd4d9e1b0 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc2feffd640 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd517be40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00a8350 tx=0x7fc2f00ac160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=414afe79bfe70531 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc2e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc2e67fc640 1 -- 172.21.15.16:0/4049523488 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00abde0 con 0x564cd4d9e1b0 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc2e67fc640 1 -- 172.21.15.16:0/4049523488 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fc2f00e9cb0 con 0x564cd4d9e1b0 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc30bcd2740 1 -- 172.21.15.16:0/4049523488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2e8017c70 con 0x564cd4d9e1b0 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc2e67fc640 1 -- 172.21.15.16:0/4049523488 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc2f00ac9d0 con 0x564cd4d9e1b0 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc30bcd2740 10 client.27452 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc30bcd2740 1 -- 172.21.15.16:0/4049523488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x564cd518abd0 con 0x564cd4d9e1b0 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.706+0000 7fc2e67fc640 1 -- 172.21.15.16:0/4049523488 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2e8017c70 con 0x564cd4d9e1b0 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.707+0000 7fc2e67fc640 1 -- 172.21.15.16:0/4049523488 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fc2f000f040 con 0x564cd4d9e1b0 2024-08-28T19:30:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.707+0000 7fc30bcd2740 10 client.27452 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.707+0000 7fc30bcd2740 10 client.27452 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.707+0000 7fc30bcd2740 20 client.27452 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.707+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564cd4d859f0 0x564cd518e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.707+0000 7fc30bcd2740 4 client.27452 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.707+0000 7fc30bcd2740 1 -- 172.21.15.16:0/4049523488 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cd50e55d0 con 0x564cd4d859f0 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.707+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564cd4d859f0 0x564cd518e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.707+0000 7fc2e67fc640 10 client.27452 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.707+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564cd4d859f0 0x564cd518e080 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=81f5f5ce6de3c865 server_cookie=67118932c4483210 in_seq=0 out_seq=0 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc2e67fc640 1 -- 172.21.15.16:0/4049523488 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564cd50e55d0 con 0x564cd4d859f0 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc2e67fc640 10 client.27452 handle_command_reply: tid=0 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc30bcd2740 1 client.27452 shutdown 2024-08-28T19:30:39.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc30bcd2740 1 -- 172.21.15.16:0/4049523488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 msgr2=0x564cd517be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:39.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd517be40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00a8350 tx=0x7fc2f00ac160 comp rx=0 tx=0).stop 2024-08-28T19:30:39.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc30bcd2740 1 -- 172.21.15.16:0/4049523488 shutdown_connections 2024-08-28T19:30:39.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564cd4d859f0 0x564cd518e080 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 0x564cd5188180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4da6390 0x564cd517c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc2e67fc640 0 client.27452 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:30:39.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/4049523488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd517be40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc30bcd2740 1 -- 172.21.15.16:0/4049523488 >> 172.21.15.16:0/4049523488 conn(0x564cd4dbb460 msgr2=0x564cd516dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:39.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc30bcd2740 1 -- 172.21.15.16:0/4049523488 shutdown_connections 2024-08-28T19:30:39.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.708+0000 7fc30bcd2740 1 -- 172.21.15.16:0/4049523488 wait complete. 2024-08-28T19:30:39.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.709+0000 7fc30bcd2740 20 client.27452 trim_cache size 0 max 16384 2024-08-28T19:30:39.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.722+0000 7fc30bcd2740 1 Processor -- start 2024-08-28T19:30:39.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.722+0000 7fc30bcd2740 1 -- start start 2024-08-28T19:30:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.722+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd516a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.722+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4da6390 0x564cd5188080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.722+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 0x564cd5179030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.722+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001ae9c0 con 0x564cd4dc7070 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.722+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001b0560 con 0x564cd4d9e1b0 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.722+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001af0f0 con 0x564cd4da6390 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.722+0000 7fc2ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 0x564cd5179030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 0x564cd5179030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33034/0 (socket says 172.21.15.16:33034) 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2ff7fe640 1 -- 172.21.15.16:0/1330964856 learned_addr learned my addr 172.21.15.16:0/1330964856 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/1330964856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4da6390 0x564cd5188080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2feffd640 1 --2- 172.21.15.16:0/1330964856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd516a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2ff7fe640 1 -- 172.21.15.16:0/1330964856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4da6390 msgr2=0x564cd5188080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2ff7fe640 1 --2- 172.21.15.16:0/1330964856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4da6390 0x564cd5188080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2ff7fe640 1 -- 172.21.15.16:0/1330964856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 msgr2=0x564cd516a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2ff7fe640 1 --2- 172.21.15.16:0/1330964856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd516a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2ff7fe640 1 -- 172.21.15.16:0/1330964856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cd51779d0 con 0x564cd4dc7070 2024-08-28T19:30:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2feffd640 1 --2- 172.21.15.16:0/1330964856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd516a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:30:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2ff7fe640 1 --2- 172.21.15.16:0/1330964856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 0x564cd5179030 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fc2f802a930 tx=0x7fc2f803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b84011840b9a186 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2f4ff9640 1 -- 172.21.15.16:0/1330964856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8035040 con 0x564cd4dc7070 2024-08-28T19:30:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2f4ff9640 1 -- 172.21.15.16:0/1330964856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2f8004440 con 0x564cd4dc7070 2024-08-28T19:30:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.723+0000 7fc2f4ff9640 1 -- 172.21.15.16:0/1330964856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f803c8e0 con 0x564cd4dc7070 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.724+0000 7fc30bcd2740 1 -- 172.21.15.16:0/1330964856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 msgr2=0x564cd5179030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.724+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/1330964856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 0x564cd5179030 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fc2f802a930 tx=0x7fc2f803e000 comp rx=0 tx=0).stop 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.724+0000 7fc30bcd2740 1 -- 172.21.15.16:0/1330964856 shutdown_connections 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.724+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/1330964856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4dc7070 0x564cd5179030 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.724+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/1330964856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4da6390 0x564cd5188080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.724+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/1330964856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd516a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.724+0000 7fc30bcd2740 1 -- 172.21.15.16:0/1330964856 >> 172.21.15.16:0/1330964856 conn(0x564cd4dbb460 msgr2=0x564cd5169b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.724+0000 7fc30bcd2740 1 -- 172.21.15.16:0/1330964856 shutdown_connections 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.724+0000 7fc30bcd2740 1 -- 172.21.15.16:0/1330964856 wait complete. 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc30bcd2740 1 -- start start 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc30bcd2740 1 Processor -- start 2024-08-28T19:30:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd5184780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd5180c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc30bcd2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4dc7070 0x564cd5181180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001afe90 con 0x564cd4da6390 2024-08-28T19:30:39.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001af7c0 con 0x564cd4d9e1b0 2024-08-28T19:30:39.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc30bcd2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001ac9c0 con 0x564cd4dc7070 2024-08-28T19:30:39.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc30bcd2740 10 client.0 _ll_register_callbacks cb 0x7fc2f00a8240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:39.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc2fe7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd5180c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc2fe7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd5180c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33046/0 (socket says 172.21.15.16:33046) 2024-08-28T19:30:39.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc2fe7fc640 1 -- 172.21.15.16:0/627264474 learned_addr learned my addr 172.21.15.16:0/627264474 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:39.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc2ff7fe640 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4dc7070 0x564cd5181180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.725+0000 7fc2feffd640 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd5184780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc2fe7fc640 1 -- 172.21.15.16:0/627264474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4dc7070 msgr2=0x564cd5181180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4dc7070 0x564cd5181180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc2fe7fc640 1 -- 172.21.15.16:0/627264474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 msgr2=0x564cd5184780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd5184780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc2fe7fc640 1 -- 172.21.15.16:0/627264474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cd51817c0 con 0x564cd4da6390 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc2fe7fc640 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd5180c40 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800bb50 tx=0x7fc2e8005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5671b24b2b96e5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc2e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc2e67fc640 1 -- 172.21.15.16:0/627264474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8032070 con 0x564cd4da6390 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc2e67fc640 1 -- 172.21.15.16:0/627264474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fc2e800f420 con 0x564cd4da6390 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc30bcd2740 1 -- 172.21.15.16:0/627264474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2f8004440 con 0x564cd4da6390 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc2e67fc640 1 -- 172.21.15.16:0/627264474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc2e803e050 con 0x564cd4da6390 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc30bcd2740 10 client.17949 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:39.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc30bcd2740 1 -- 172.21.15.16:0/627264474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x564cd51f9840 con 0x564cd4da6390 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.726+0000 7fc2e67fc640 1 -- 172.21.15.16:0/627264474 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2f8004440 con 0x564cd4da6390 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc2e67fc640 1 -- 172.21.15.16:0/627264474 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fc2e800fb20 con 0x564cd4da6390 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc30bcd2740 10 client.17949 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc30bcd2740 10 client.17949 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc30bcd2740 20 client.17949 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564cd4d5ada0 0x564cd51fbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc30bcd2740 4 client.17949 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc30bcd2740 1 -- 172.21.15.16:0/627264474 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564cd50e1070 con 0x564cd4d5ada0 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc2feffd640 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564cd4d5ada0 0x564cd51fbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc2feffd640 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564cd4d5ada0 0x564cd51fbcf0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e72b20e864523bf7 server_cookie=68d0f712bad7969b in_seq=0 out_seq=0 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc2e67fc640 10 client.17949 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc2e67fc640 1 -- 172.21.15.16:0/627264474 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564cd50e1070 con 0x564cd4d5ada0 2024-08-28T19:30:39.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc2e67fc640 10 client.17949 handle_command_reply: tid=0 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.727+0000 7fc30bcd2740 1 client.17949 shutdown 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30bcd2740 1 -- 172.21.15.16:0/627264474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 msgr2=0x564cd5180c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd5180c40 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800bb50 tx=0x7fc2e8005d60 comp rx=0 tx=0).stop 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30bcd2740 1 -- 172.21.15.16:0/627264474 shutdown_connections 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x564cd4d5ada0 0x564cd51fbcf0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cd4dc7070 0x564cd5181180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cd4da6390 0x564cd5180c40 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc2e67fc640 0 client.17949 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30bcd2740 1 --2- 172.21.15.16:0/627264474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cd4d9e1b0 0x564cd5184780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30bcd2740 1 -- 172.21.15.16:0/627264474 >> 172.21.15.16:0/627264474 conn(0x564cd4dbb460 msgr2=0x564cd517d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30bcd2740 1 -- 172.21.15.16:0/627264474 shutdown_connections 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30bcd2740 1 -- 172.21.15.16:0/627264474 wait complete. 2024-08-28T19:30:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30bcd2740 20 client.17949 trim_cache size 0 max 16384 2024-08-28T19:30:39.736 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:30:39.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30575d640 1 -- 172.21.15.16:0/574576563 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ec0b4cd0 msgr2=0x7fc2ec0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:39.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30575d640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ec0b4cd0 0x7fc2ec0b71e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005940 tx=0x7fc2f8009660 comp rx=0 tx=0).stop 2024-08-28T19:30:39.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30575d640 1 -- 172.21.15.16:0/574576563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102880 msgr2=0x7fc30019b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:39.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30575d640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102880 0x7fc30019b030 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000eab0 tx=0x7fc2f000ef70 comp rx=0 tx=0).stop 2024-08-28T19:30:39.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30575d640 1 -- 172.21.15.16:0/574576563 shutdown_connections 2024-08-28T19:30:39.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.728+0000 7fc30575d640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc300108b10 0x7fc30019bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.729+0000 7fc30575d640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc2ec0b4cd0 0x7fc2ec0b71e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.729+0000 7fc30575d640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3001081c0 0x7fc30019b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.729+0000 7fc30575d640 1 --2- 172.21.15.16:0/574576563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102880 0x7fc30019b030 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:39.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.729+0000 7fc30575d640 1 -- 172.21.15.16:0/574576563 >> 172.21.15.16:0/574576563 conn(0x7fc300079510 msgr2=0x7fc300106210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:39.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.729+0000 7fc30575d640 1 -- 172.21.15.16:0/574576563 shutdown_connections 2024-08-28T19:30:39.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:39.729+0000 7fc30575d640 1 -- 172.21.15.16:0/574576563 wait complete. 2024-08-28T19:30:39.965 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430914 4.22 MB/sec execute 379 sec latency 619.658 ms 2024-08-28T19:30:40.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:30:40.726 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:40.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430914 4.21 MB/sec execute 380 sec latency 1619.755 ms 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.239+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2610216731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbec0108ab0 msgr2=0x7fbec010f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.239+0000 7fbec4bf8640 1 --2- 172.21.15.16:0/2610216731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbec0108ab0 0x7fbec010f2e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009ca0 tx=0x7fbeb802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.239+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2610216731 shutdown_connections 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.239+0000 7fbec4bf8640 1 --2- 172.21.15.16:0/2610216731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbec0108ab0 0x7fbec010f2e0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.239+0000 7fbec4bf8640 1 --2- 172.21.15.16:0/2610216731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec0108160 0x7fbec0108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.239+0000 7fbec4bf8640 1 --2- 172.21.15.16:0/2610216731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbec0102820 0x7fbec0102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.239+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2610216731 >> 172.21.15.16:0/2610216731 conn(0x7fbec00794e0 msgr2=0x7fbec00798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.239+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2610216731 shutdown_connections 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.239+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2610216731 wait complete. 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.240+0000 7fbec4bf8640 1 Processor -- start 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbec4bf8640 1 -- start start 2024-08-28T19:30:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbec4bf8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbec0102820 0x7fbec0199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbec4bf8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbec0108160 0x7fbec0199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbec4bf8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec0108ab0 0x7fbec019e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbec4bf8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbec0112730 con 0x7fbec0108160 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbec4bf8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbec01125b0 con 0x7fbec0108ab0 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbec4bf8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbec01128b0 con 0x7fbec0102820 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbebdd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbec0108160 0x7fbec0199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbebdd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbec0108160 0x7fbec0199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33064/0 (socket says 172.21.15.16:33064) 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbebdd9b640 1 -- 172.21.15.16:0/2627642813 learned_addr learned my addr 172.21.15.16:0/2627642813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbebe59c640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbec0102820 0x7fbec0199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbebed9d640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec0108ab0 0x7fbec019e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbebdd9b640 1 -- 172.21.15.16:0/2627642813 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbec0102820 msgr2=0x7fbec0199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbebdd9b640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbec0102820 0x7fbec0199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbebdd9b640 1 -- 172.21.15.16:0/2627642813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec0108ab0 msgr2=0x7fbec019e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbebdd9b640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec0108ab0 0x7fbec019e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbebdd9b640 1 -- 172.21.15.16:0/2627642813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec019f0f0 con 0x7fbec0108160 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.241+0000 7fbebed9d640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec0108ab0 0x7fbec019e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.242+0000 7fbebdd9b640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbec0108160 0x7fbec0199720 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400d770 tx=0x7fbeb400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.242+0000 7fbeaf7fe640 1 -- 172.21.15.16:0/2627642813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4017070 con 0x7fbec0108160 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.242+0000 7fbeaf7fe640 1 -- 172.21.15.16:0/2627642813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb4009c30 con 0x7fbec0108160 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.242+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2627642813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec019f3e0 con 0x7fbec0108160 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.242+0000 7fbeaf7fe640 1 -- 172.21.15.16:0/2627642813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb400f620 con 0x7fbec0108160 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.242+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2627642813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec01a5b40 con 0x7fbec0108160 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.244+0000 7fbecb16d740 1 Processor -- start 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.244+0000 7fbeaf7fe640 1 -- 172.21.15.16:0/2627642813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fbeb401a070 con 0x7fbec0108160 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbecb16d740 1 -- start start 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759be310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f01b0 0x5584759be850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f8390 0x5584759bed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbecb16d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558475935680 con 0x558475619070 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbecb16d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558475924c10 con 0x5584755f01b0 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbecb16d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558475928bc0 con 0x5584755f8390 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbebed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759be310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbebed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759be310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33068/0 (socket says 172.21.15.16:33068) 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbebed9d640 1 -- 172.21.15.16:0/700580138 learned_addr learned my addr 172.21.15.16:0/700580138 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbebe59c640 1 --2- 172.21.15.16:0/700580138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f01b0 0x5584759be850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbebdd9b640 1 --2- 172.21.15.16:0/700580138 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f8390 0x5584759bed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.245+0000 7fbeaf7fe640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbea80b4ca0 0x7fbea80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbebe59c640 1 -- 172.21.15.16:0/700580138 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f8390 msgr2=0x5584759bed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:41.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbebe59c640 1 --2- 172.21.15.16:0/700580138 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f8390 0x5584759bed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbebe59c640 1 -- 172.21.15.16:0/700580138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 msgr2=0x5584759be310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbebe59c640 1 --2- 172.21.15.16:0/700580138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759be310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbebe59c640 1 -- 172.21.15.16:0/700580138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558475684650 con 0x5584755f01b0 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbebed9d640 1 --2- 172.21.15.16:0/700580138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759be310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbebed9d640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbea80b4ca0 0x7fbea80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbeaf7fe640 1 -- 172.21.15.16:0/2627642813 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fbeb40e5400 con 0x7fbec0108160 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbebe59c640 1 --2- 172.21.15.16:0/700580138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f01b0 0x5584759be850 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0004630 tx=0x7fbeb000bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c20422c6331adee server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbeacff9640 1 -- 172.21.15.16:0/700580138 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0003bd0 con 0x5584755f01b0 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbeacff9640 1 -- 172.21.15.16:0/700580138 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb0015070 con 0x5584755f01b0 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbeacff9640 1 -- 172.21.15.16:0/700580138 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0004c80 con 0x5584755f01b0 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.246+0000 7fbebed9d640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbea80b4ca0 0x7fbea80b71b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80094e0 tx=0x7fbeb80095c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.247+0000 7fbecb16d740 1 -- 172.21.15.16:0/700580138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f01b0 msgr2=0x5584759be850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.247+0000 7fbecb16d740 1 --2- 172.21.15.16:0/700580138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f01b0 0x5584759be850 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0004630 tx=0x7fbeb000bb20 comp rx=0 tx=0).stop 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.247+0000 7fbecb16d740 1 -- 172.21.15.16:0/700580138 shutdown_connections 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.247+0000 7fbecb16d740 1 --2- 172.21.15.16:0/700580138 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f8390 0x5584759bed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.247+0000 7fbecb16d740 1 --2- 172.21.15.16:0/700580138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f01b0 0x5584759be850 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.247+0000 7fbecb16d740 1 --2- 172.21.15.16:0/700580138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759be310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.247+0000 7fbecb16d740 1 -- 172.21.15.16:0/700580138 >> 172.21.15.16:0/700580138 conn(0x55847560d460 msgr2=0x5584759bc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.247+0000 7fbecb16d740 1 -- 172.21.15.16:0/700580138 shutdown_connections 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.247+0000 7fbecb16d740 1 -- 172.21.15.16:0/700580138 wait complete. 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbecb16d740 1 -- start start 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbecb16d740 1 Processor -- start 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 0x5584759c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f8390 0x5584759c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759d5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbecb16d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558475787700 con 0x558475619070 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbecb16d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5584758ab750 con 0x5584755f8390 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbecb16d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5584759ccaf0 con 0x5584755f01b0 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbecb16d740 10 client.0 _ll_register_callbacks cb 0x5584758dae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbebed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759d5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbebed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759d5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33076/0 (socket says 172.21.15.16:33076) 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbebed9d640 1 -- 172.21.15.16:0/1379890965 learned_addr learned my addr 172.21.15.16:0/1379890965 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbebdd9b640 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f8390 0x5584759c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbebe59c640 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 0x5584759c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbebed9d640 1 -- 172.21.15.16:0/1379890965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 msgr2=0x5584759c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbebed9d640 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 0x5584759c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbebed9d640 1 -- 172.21.15.16:0/1379890965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f8390 msgr2=0x5584759c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbebed9d640 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f8390 0x5584759c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbebed9d640 1 -- 172.21.15.16:0/1379890965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584758e4900 con 0x558475619070 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.248+0000 7fbebe59c640 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 0x5584759c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbebed9d640 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759d5e10 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009ca0 tx=0x7fbeb802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ecfb474319a932 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbea5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbea5ffb640 1 -- 172.21.15.16:0/1379890965 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb80046c0 con 0x558475619070 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbea5ffb640 1 -- 172.21.15.16:0/1379890965 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fbeb802f660 con 0x558475619070 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbecb16d740 1 -- 172.21.15.16:0/1379890965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbeb0015070 con 0x558475619070 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbea5ffb640 1 -- 172.21.15.16:0/1379890965 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fbeb8004860 con 0x558475619070 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbecb16d740 10 client.17961 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbecb16d740 1 -- 172.21.15.16:0/1379890965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5584759d8830 con 0x558475619070 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbea5ffb640 1 -- 172.21.15.16:0/1379890965 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb0015070 con 0x558475619070 2024-08-28T19:30:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbea5ffb640 1 -- 172.21.15.16:0/1379890965 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fbeb8030890 con 0x558475619070 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbecb16d740 10 client.17961 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbecb16d740 10 client.17961 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbecb16d740 20 client.17961 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbecb16d740 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5584755d79f0 0x5584759dbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbecb16d740 4 client.17961 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.249+0000 7fbecb16d740 1 -- 172.21.15.16:0/1379890965 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584759375d0 con 0x5584755d79f0 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.250+0000 7fbebe59c640 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5584755d79f0 0x5584759dbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.250+0000 7fbea5ffb640 10 client.17961 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.250+0000 7fbebe59c640 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5584755d79f0 0x5584759dbe10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d18e54f868a46a server_cookie=c1362de413284695 in_seq=0 out_seq=0 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbea5ffb640 1 -- 172.21.15.16:0/1379890965 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5584759375d0 con 0x5584755d79f0 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbea5ffb640 10 client.17961 handle_command_reply: tid=0 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbecb16d740 1 client.17961 shutdown 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbecb16d740 1 -- 172.21.15.16:0/1379890965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 msgr2=0x5584759d5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbecb16d740 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759d5e10 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009ca0 tx=0x7fbeb802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbecb16d740 1 -- 172.21.15.16:0/1379890965 shutdown_connections 2024-08-28T19:30:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbecb16d740 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5584755d79f0 0x5584759dbe10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbecb16d740 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558475619070 0x5584759d5e10 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbecb16d740 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f8390 0x5584759c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbea5ffb640 0 client.17961 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:30:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbecb16d740 1 --2- 172.21.15.16:0/1379890965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 0x5584759c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.251+0000 7fbecb16d740 1 -- 172.21.15.16:0/1379890965 >> 172.21.15.16:0/1379890965 conn(0x55847560d460 msgr2=0x5584759c14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.252+0000 7fbecb16d740 1 -- 172.21.15.16:0/1379890965 shutdown_connections 2024-08-28T19:30:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.252+0000 7fbecb16d740 1 -- 172.21.15.16:0/1379890965 wait complete. 2024-08-28T19:30:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.252+0000 7fbecb16d740 20 client.17961 trim_cache size 0 max 16384 2024-08-28T19:30:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbecb16d740 1 Processor -- start 2024-08-28T19:30:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbecb16d740 1 -- start start 2024-08-28T19:30:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755d79f0 0x5584759bc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 0x5584759d5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755f8390 0x5584759c1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbecb16d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbec01aebe0 con 0x5584755f8390 2024-08-28T19:30:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbecb16d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbec01b0780 con 0x5584755d79f0 2024-08-28T19:30:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbecb16d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbec01af310 con 0x5584755f01b0 2024-08-28T19:30:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbebed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755f8390 0x5584759c1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbebed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755f8390 0x5584759c1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33082/0 (socket says 172.21.15.16:33082) 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbebed9d640 1 -- 172.21.15.16:0/2483476514 learned_addr learned my addr 172.21.15.16:0/2483476514 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbebe59c640 1 --2- 172.21.15.16:0/2483476514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755d79f0 0x5584759bc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.265+0000 7fbebdd9b640 1 --2- 172.21.15.16:0/2483476514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 0x5584759d5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.266+0000 7fbebed9d640 1 -- 172.21.15.16:0/2483476514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 msgr2=0x5584759d5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.266+0000 7fbebed9d640 1 --2- 172.21.15.16:0/2483476514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 0x5584759d5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.266+0000 7fbebed9d640 1 -- 172.21.15.16:0/2483476514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755d79f0 msgr2=0x5584759bc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.266+0000 7fbebed9d640 1 --2- 172.21.15.16:0/2483476514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755d79f0 0x5584759bc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.266+0000 7fbebed9d640 1 -- 172.21.15.16:0/2483476514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584759b63d0 con 0x5584755f8390 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.266+0000 7fbebdd9b640 1 --2- 172.21.15.16:0/2483476514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 0x5584759d5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.266+0000 7fbebed9d640 1 --2- 172.21.15.16:0/2483476514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755f8390 0x5584759c1410 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009bf0 tx=0x7fbeb802afc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a5fb4e9f97264a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.266+0000 7fbea7fff640 1 -- 172.21.15.16:0/2483476514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8004610 con 0x5584755f8390 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.266+0000 7fbea7fff640 1 -- 172.21.15.16:0/2483476514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb804d030 con 0x5584755f8390 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.266+0000 7fbea7fff640 1 -- 172.21.15.16:0/2483476514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb802fd00 con 0x5584755f8390 2024-08-28T19:30:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.267+0000 7fbecb16d740 1 -- 172.21.15.16:0/2483476514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755f8390 msgr2=0x5584759c1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.267+0000 7fbecb16d740 1 --2- 172.21.15.16:0/2483476514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755f8390 0x5584759c1410 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009bf0 tx=0x7fbeb802afc0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.267+0000 7fbecb16d740 1 -- 172.21.15.16:0/2483476514 shutdown_connections 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.267+0000 7fbecb16d740 1 --2- 172.21.15.16:0/2483476514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755f8390 0x5584759c1410 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.267+0000 7fbecb16d740 1 --2- 172.21.15.16:0/2483476514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f01b0 0x5584759d5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.267+0000 7fbecb16d740 1 --2- 172.21.15.16:0/2483476514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755d79f0 0x5584759bc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.267+0000 7fbecb16d740 1 -- 172.21.15.16:0/2483476514 >> 172.21.15.16:0/2483476514 conn(0x55847560d460 msgr2=0x5584759bbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.267+0000 7fbecb16d740 1 -- 172.21.15.16:0/2483476514 shutdown_connections 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.267+0000 7fbecb16d740 1 -- 172.21.15.16:0/2483476514 wait complete. 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.267+0000 7fbecb16d740 1 -- start start 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.267+0000 7fbecb16d740 1 Processor -- start 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755d79f0 0x5584759ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f01b0 0x5584759cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbecb16d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f8390 0x558475a457d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbecb16d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbec01b00b0 con 0x5584755d79f0 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbecb16d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbec01af9e0 con 0x5584755f01b0 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbecb16d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbec01acbe0 con 0x5584755f8390 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbecb16d740 10 client.0 _ll_register_callbacks cb 0x7fbeb80099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbebe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755d79f0 0x5584759ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbebe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755d79f0 0x5584759ce4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33088/0 (socket says 172.21.15.16:33088) 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbebe59c640 1 -- 172.21.15.16:0/409499612 learned_addr learned my addr 172.21.15.16:0/409499612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbebdd9b640 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f01b0 0x5584759cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbebed9d640 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f8390 0x558475a457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbebe59c640 1 -- 172.21.15.16:0/409499612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f8390 msgr2=0x558475a457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbebe59c640 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f8390 0x558475a457d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbebe59c640 1 -- 172.21.15.16:0/409499612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f01b0 msgr2=0x5584759cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbebe59c640 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f01b0 0x5584759cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbebe59c640 1 -- 172.21.15.16:0/409499612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558475a45d60 con 0x5584755d79f0 2024-08-28T19:30:41.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbebe59c640 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755d79f0 0x5584759ce4d0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00043f0 tx=0x7fbeb0005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391776c2028e57f0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:41.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.268+0000 7fbea5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:41.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.272+0000 7fbea5ffb640 1 -- 172.21.15.16:0/409499612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0030060 con 0x5584755d79f0 2024-08-28T19:30:41.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.272+0000 7fbea5ffb640 1 -- 172.21.15.16:0/409499612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fbeb0003ee0 con 0x5584755d79f0 2024-08-28T19:30:41.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.272+0000 7fbecb16d740 1 -- 172.21.15.16:0/409499612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbeb804d030 con 0x5584755d79f0 2024-08-28T19:30:41.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.272+0000 7fbea5ffb640 1 -- 172.21.15.16:0/409499612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fbeb0037050 con 0x5584755d79f0 2024-08-28T19:30:41.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.272+0000 7fbecb16d740 10 client.17967 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.272+0000 7fbecb16d740 1 -- 172.21.15.16:0/409499612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x558475a46f50 con 0x5584755d79f0 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.272+0000 7fbea5ffb640 1 -- 172.21.15.16:0/409499612 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb804d030 con 0x5584755d79f0 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.273+0000 7fbea5ffb640 1 -- 172.21.15.16:0/409499612 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fbeb002a070 con 0x5584755d79f0 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.273+0000 7fbecb16d740 10 client.17967 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.273+0000 7fbecb16d740 10 client.17967 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.273+0000 7fbecb16d740 20 client.17967 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.273+0000 7fbecb16d740 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x558475619070 0x558475a49460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.273+0000 7fbecb16d740 4 client.17967 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.273+0000 7fbecb16d740 1 -- 172.21.15.16:0/409499612 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558475933070 con 0x558475619070 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.273+0000 7fbebdd9b640 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x558475619070 0x558475a49460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.273+0000 7fbebdd9b640 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x558475619070 0x558475a49460 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1c41327e87afb2b3 server_cookie=6cddb3706aea27ef in_seq=0 out_seq=0 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.273+0000 7fbea5ffb640 10 client.17967 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:30:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbea5ffb640 1 -- 172.21.15.16:0/409499612 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558475933070 con 0x558475619070 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbea5ffb640 10 client.17967 handle_command_reply: tid=0 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 1 client.17967 shutdown 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 1 -- 172.21.15.16:0/409499612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755d79f0 msgr2=0x5584759ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755d79f0 0x5584759ce4d0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00043f0 tx=0x7fbeb0005cc0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 1 -- 172.21.15.16:0/409499612 shutdown_connections 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x558475619070 0x558475a49460 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5584755f8390 0x558475a457d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5584755f01b0 0x5584759cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbea5ffb640 0 client.17967 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 1 --2- 172.21.15.16:0/409499612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584755d79f0 0x5584759ce4d0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 1 -- 172.21.15.16:0/409499612 >> 172.21.15.16:0/409499612 conn(0x55847560d460 msgr2=0x5584759c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 1 -- 172.21.15.16:0/409499612 shutdown_connections 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 1 -- 172.21.15.16:0/409499612 wait complete. 2024-08-28T19:30:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.274+0000 7fbecb16d740 20 client.17967 trim_cache size 0 max 16384 2024-08-28T19:30:41.279 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:30:41.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2627642813 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbea80b4ca0 msgr2=0x7fbea80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:41.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbea80b4ca0 0x7fbea80b71b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80094e0 tx=0x7fbeb80095c0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2627642813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbec0108160 msgr2=0x7fbec0199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbec0108160 0x7fbec0199720 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400d770 tx=0x7fbeb400dc30 comp rx=0 tx=0).stop 2024-08-28T19:30:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2627642813 shutdown_connections 2024-08-28T19:30:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbea80b4ca0 0x7fbea80b71b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec0108ab0 0x7fbec019e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbec0108160 0x7fbec0199720 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 --2- 172.21.15.16:0/2627642813 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbec0102820 0x7fbec0199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2627642813 >> 172.21.15.16:0/2627642813 conn(0x7fbec00794e0 msgr2=0x7fbec01050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2627642813 shutdown_connections 2024-08-28T19:30:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:41.275+0000 7fbec4bf8640 1 -- 172.21.15.16:0/2627642813 wait complete. 2024-08-28T19:30:41.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:41 smithi016 ceph-mon[18556]: pgmap v634: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-08-28T19:30:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:41 smithi028 ceph-mon[21769]: pgmap v634: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-08-28T19:30:41.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:41 smithi115 ceph-mon[22184]: pgmap v634: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-08-28T19:30:41.670 INFO:tasks.quiescer.fs.[cephfs]:Total duration (54.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-08-28T19:30:41.670 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.05 2024-08-28T19:30:41.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:30:41.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430914 4.20 MB/sec execute 381 sec latency 2619.811 ms 2024-08-28T19:30:41.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:30:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:42.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:42 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:30:42.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:42 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:30:42.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:42 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:30:42.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.727+0000 7f15e2da2640 1 -- 172.21.15.16:0/4049817214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15dc103900 msgr2=0x7f15dc103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:42.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.727+0000 7f15e2da2640 1 --2- 172.21.15.16:0/4049817214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15dc103900 0x7f15dc103d10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f15cc009900 tx=0x7f15cc02a410 comp rx=0 tx=0).stop 2024-08-28T19:30:42.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.727+0000 7f15e2da2640 1 -- 172.21.15.16:0/4049817214 shutdown_connections 2024-08-28T19:30:42.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.727+0000 7f15e2da2640 1 --2- 172.21.15.16:0/4049817214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15dc109b90 0x7f15dc1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:42.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.727+0000 7f15e2da2640 1 --2- 172.21.15.16:0/4049817214 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15dc109240 0x7f15dc109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:42.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.727+0000 7f15e2da2640 1 --2- 172.21.15.16:0/4049817214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15dc103900 0x7f15dc103d10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:42.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.727+0000 7f15e2da2640 1 -- 172.21.15.16:0/4049817214 >> 172.21.15.16:0/4049817214 conn(0x7f15dc07b5f0 msgr2=0x7f15dc07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:42.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.727+0000 7f15e2da2640 1 -- 172.21.15.16:0/4049817214 shutdown_connections 2024-08-28T19:30:42.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.727+0000 7f15e2da2640 1 -- 172.21.15.16:0/4049817214 wait complete. 2024-08-28T19:30:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e2da2640 1 Processor -- start 2024-08-28T19:30:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e2da2640 1 -- start start 2024-08-28T19:30:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e2da2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15dc103900 0x7f15dc19a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e2da2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15dc109240 0x7f15dc19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e2da2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15dc109b90 0x7f15dc19fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e2da2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc113850 con 0x7f15dc103900 2024-08-28T19:30:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e2da2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc1136d0 con 0x7f15dc109b90 2024-08-28T19:30:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e2da2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc1139d0 con 0x7f15dc109240 2024-08-28T19:30:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e0b3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15dc103900 0x7f15dc19a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e0b3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15dc103900 0x7f15dc19a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33110/0 (socket says 172.21.15.16:33110) 2024-08-28T19:30:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e0b3e640 1 -- 172.21.15.16:0/1703041600 learned_addr learned my addr 172.21.15.16:0/1703041600 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15e133f640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15dc109b90 0x7f15dc19fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.728+0000 7f15dbfff640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15dc109240 0x7f15dc19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.729+0000 7f15e0b3e640 1 -- 172.21.15.16:0/1703041600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15dc109240 msgr2=0x7f15dc19a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.729+0000 7f15e0b3e640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15dc109240 0x7f15dc19a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.729+0000 7f15e0b3e640 1 -- 172.21.15.16:0/1703041600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15dc109b90 msgr2=0x7f15dc19fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.729+0000 7f15e0b3e640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15dc109b90 0x7f15dc19fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.729+0000 7f15e0b3e640 1 -- 172.21.15.16:0/1703041600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15dc1a0230 con 0x7f15dc103900 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.729+0000 7f15e0b3e640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15dc103900 0x7f15dc19a2c0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f15cc005940 tx=0x7f15cc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.729+0000 7f15d9ffb640 1 -- 172.21.15.16:0/1703041600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc03f030 con 0x7f15dc103900 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.729+0000 7f15e2da2640 1 -- 172.21.15.16:0/1703041600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15dc1a04c0 con 0x7f15dc103900 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.729+0000 7f15d9ffb640 1 -- 172.21.15.16:0/1703041600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15cc0044c0 con 0x7f15dc103900 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.729+0000 7f15d9ffb640 1 -- 172.21.15.16:0/1703041600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc035070 con 0x7f15dc103900 2024-08-28T19:30:42.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.729+0000 7f15e2da2640 1 -- 172.21.15.16:0/1703041600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15dc1a6c80 con 0x7f15dc103900 2024-08-28T19:30:42.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.730+0000 7f15e2da2640 1 -- 172.21.15.16:0/1703041600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15dc103d10 con 0x7f15dc103900 2024-08-28T19:30:42.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.733+0000 7f15d9ffb640 1 -- 172.21.15.16:0/1703041600 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f15cc042030 con 0x7f15dc103900 2024-08-28T19:30:42.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.734+0000 7f15d9ffb640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c80b4ca0 0x7f15c80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:42.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.734+0000 7f15d9ffb640 1 -- 172.21.15.16:0/1703041600 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7f15cc105f60 con 0x7f15dc103900 2024-08-28T19:30:42.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.734+0000 7f15dbfff640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c80b4ca0 0x7f15c80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:42.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.734+0000 7f15dbfff640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c80b4ca0 0x7f15c80b71b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f15dc106260 tx=0x7f15c4005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:42.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.737+0000 7f15d9ffb640 1 -- 172.21.15.16:0/1703041600 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f15cc0cc920 con 0x7f15dc103900 2024-08-28T19:30:42.783 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:42.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 432763 4.22 MB/sec execute 382 sec latency 2636.549 ms 2024-08-28T19:30:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.987+0000 7f15e2da2640 1 -- 172.21.15.16:0/1703041600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15dc19b3f0 con 0x7f15dc103900 2024-08-28T19:30:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.988+0000 7f15d9ffb640 1 -- 172.21.15.16:0/1703041600 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11980 (secure 0 0 0) 0x7f15cc0cc220 con 0x7f15dc103900 2024-08-28T19:30:42.989 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:30:42.989 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"btime":"2024-08-28T19:30:14:226103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:30:13.285656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:30:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 53 2024-08-28T19:30:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.990+0000 7f15e2da2640 1 -- 172.21.15.16:0/1703041600 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c80b4ca0 msgr2=0x7f15c80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.990+0000 7f15e2da2640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c80b4ca0 0x7f15c80b71b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f15dc106260 tx=0x7f15c4005f90 comp rx=0 tx=0).stop 2024-08-28T19:30:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.990+0000 7f15e2da2640 1 -- 172.21.15.16:0/1703041600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15dc103900 msgr2=0x7f15dc19a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.990+0000 7f15e2da2640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15dc103900 0x7f15dc19a2c0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f15cc005940 tx=0x7f15cc0040c0 comp rx=0 tx=0).stop 2024-08-28T19:30:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.990+0000 7f15e2da2640 1 -- 172.21.15.16:0/1703041600 shutdown_connections 2024-08-28T19:30:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.990+0000 7f15e2da2640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f15c80b4ca0 0x7f15c80b71b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.990+0000 7f15e2da2640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15dc109b90 0x7f15dc19fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.990+0000 7f15e2da2640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15dc109240 0x7f15dc19a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.990+0000 7f15e2da2640 1 --2- 172.21.15.16:0/1703041600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15dc103900 0x7f15dc19a2c0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.990+0000 7f15e2da2640 1 -- 172.21.15.16:0/1703041600 >> 172.21.15.16:0/1703041600 conn(0x7f15dc07b5f0 msgr2=0x7f15dc107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.990+0000 7f15e2da2640 1 -- 172.21.15.16:0/1703041600 shutdown_connections 2024-08-28T19:30:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:42.991+0000 7f15e2da2640 1 -- 172.21.15.16:0/1703041600 wait complete. 2024-08-28T19:30:43.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:43 smithi016 ceph-mon[18556]: pgmap v635: 129 pgs: 129 active+clean; 302 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-08-28T19:30:43.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:43 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1703041600' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:43 smithi028 ceph-mon[21769]: pgmap v635: 129 pgs: 129 active+clean; 302 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-08-28T19:30:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:43 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1703041600' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:43.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:43 smithi115 ceph-mon[22184]: pgmap v635: 129 pgs: 129 active+clean; 302 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-08-28T19:30:43.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:43 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1703041600' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:30:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.622+0000 7fb35812b640 1 -- 172.21.15.16:0/3669442402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb350108140 msgr2=0x7fb350108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.622+0000 7fb35812b640 1 --2- 172.21.15.16:0/3669442402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb350108140 0x7fb350108550 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fb340009900 tx=0x7fb34002a410 comp rx=0 tx=0).stop 2024-08-28T19:30:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.622+0000 7fb35812b640 1 -- 172.21.15.16:0/3669442402 shutdown_connections 2024-08-28T19:30:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.622+0000 7fb35812b640 1 --2- 172.21.15.16:0/3669442402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb350108a90 0x7fb35010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.622+0000 7fb35812b640 1 --2- 172.21.15.16:0/3669442402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb350108140 0x7fb350108550 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.622+0000 7fb35812b640 1 --2- 172.21.15.16:0/3669442402 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb350102800 0x7fb350102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.622+0000 7fb35812b640 1 -- 172.21.15.16:0/3669442402 >> 172.21.15.16:0/3669442402 conn(0x7fb3500794d0 msgr2=0x7fb3500798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.622+0000 7fb35812b640 1 -- 172.21.15.16:0/3669442402 shutdown_connections 2024-08-28T19:30:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.622+0000 7fb35812b640 1 -- 172.21.15.16:0/3669442402 wait complete. 2024-08-28T19:30:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.622+0000 7fb35812b640 1 Processor -- start 2024-08-28T19:30:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb35812b640 1 -- start start 2024-08-28T19:30:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb35812b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb350102800 0x7fb350199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb35812b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb350108140 0x7fb350199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb35812b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb350108a90 0x7fb35019e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb35812b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3501126c0 con 0x7fb350108a90 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb35812b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb350112540 con 0x7fb350102800 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb35812b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb350112840 con 0x7fb350108140 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb3566c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb350108a90 0x7fb35019e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb355ec7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb350102800 0x7fb350199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb3566c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb350108a90 0x7fb35019e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33134/0 (socket says 172.21.15.16:33134) 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb355ec7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb350102800 0x7fb350199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48888/0 (socket says 172.21.15.16:48888) 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb3566c8640 1 -- 172.21.15.16:0/4140597572 learned_addr learned my addr 172.21.15.16:0/4140597572 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb3556c6640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb350108140 0x7fb350199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb355ec7640 1 -- 172.21.15.16:0/4140597572 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb350108140 msgr2=0x7fb350199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb355ec7640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb350108140 0x7fb350199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb355ec7640 1 -- 172.21.15.16:0/4140597572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb350108a90 msgr2=0x7fb35019e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb355ec7640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb350108a90 0x7fb35019e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb355ec7640 1 -- 172.21.15.16:0/4140597572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35019f100 con 0x7fb350102800 2024-08-28T19:30:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.623+0000 7fb3566c8640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb350108a90 0x7fb35019e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.624+0000 7fb355ec7640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb350102800 0x7fb350199190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00d6d0 tx=0x7fb34c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.624+0000 7fb346ffd640 1 -- 172.21.15.16:0/4140597572 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c017070 con 0x7fb350102800 2024-08-28T19:30:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.624+0000 7fb35812b640 1 -- 172.21.15.16:0/4140597572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35019f3f0 con 0x7fb350102800 2024-08-28T19:30:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.624+0000 7fb346ffd640 1 -- 172.21.15.16:0/4140597572 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb34c009820 con 0x7fb350102800 2024-08-28T19:30:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.624+0000 7fb346ffd640 1 -- 172.21.15.16:0/4140597572 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c004c40 con 0x7fb350102800 2024-08-28T19:30:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.624+0000 7fb35812b640 1 -- 172.21.15.16:0/4140597572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35006ac00 con 0x7fb350102800 2024-08-28T19:30:43.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.627+0000 7fb35e6a0740 1 Processor -- start 2024-08-28T19:30:43.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb35e6a0740 1 -- start start 2024-08-28T19:30:43.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb346ffd640 1 -- 172.21.15.16:0/4140597572 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fb34c005220 con 0x7fb350102800 2024-08-28T19:30:43.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 0x55a5ec0f8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec0f8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 0x55a5ec0f8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5ec06f680 con 0x55a5ebd32390 2024-08-28T19:30:43.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5ec05ec10 con 0x55a5ebd53070 2024-08-28T19:30:43.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a5ec062bc0 con 0x55a5ebd2a1b0 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb355ec7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec0f8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb3566c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 0x55a5ec0f8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb3556c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 0x55a5ec0f8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb3566c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 0x55a5ec0f8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48898/0 (socket says 172.21.15.16:48898) 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.628+0000 7fb3566c8640 1 -- 172.21.15.16:0/1866399433 learned_addr learned my addr 172.21.15.16:0/1866399433 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.629+0000 7fb3556c6640 1 -- 172.21.15.16:0/1866399433 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 msgr2=0x55a5ec0f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.629+0000 7fb3556c6640 1 --2- 172.21.15.16:0/1866399433 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec0f8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.629+0000 7fb3556c6640 1 -- 172.21.15.16:0/1866399433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 msgr2=0x55a5ec0f8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.629+0000 7fb3556c6640 1 --2- 172.21.15.16:0/1866399433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 0x55a5ec0f8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.629+0000 7fb3556c6640 1 -- 172.21.15.16:0/1866399433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5ebdbe650 con 0x55a5ebd32390 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.629+0000 7fb3556c6640 1 --2- 172.21.15.16:0/1866399433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 0x55a5ec0f8d90 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fb340006010 tx=0x7fb340032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed258f90c55c5e4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:43.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.629+0000 7fb346ffd640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb33c0b4ca0 0x7fb33c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.629+0000 7fb33bfff640 1 -- 172.21.15.16:0/1866399433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340040070 con 0x55a5ebd32390 2024-08-28T19:30:43.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.629+0000 7fb33bfff640 1 -- 172.21.15.16:0/1866399433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb340004030 con 0x55a5ebd32390 2024-08-28T19:30:43.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.629+0000 7fb33bfff640 1 -- 172.21.15.16:0/1866399433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340035040 con 0x55a5ebd32390 2024-08-28T19:30:43.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.630+0000 7fb3566c8640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb33c0b4ca0 0x7fb33c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.630+0000 7fb346ffd640 1 -- 172.21.15.16:0/4140597572 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fb34c0e5410 con 0x7fb350102800 2024-08-28T19:30:43.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb35e6a0740 1 -- 172.21.15.16:0/1866399433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 msgr2=0x55a5ec0f8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb3566c8640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb33c0b4ca0 0x7fb33c0b71b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb34800f9b0 tx=0x7fb348010040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/1866399433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 0x55a5ec0f8d90 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fb340006010 tx=0x7fb340032bd0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb35e6a0740 1 -- 172.21.15.16:0/1866399433 shutdown_connections 2024-08-28T19:30:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/1866399433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 0x55a5ec0f8d90 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/1866399433 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec0f8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/1866399433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 0x55a5ec0f8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb35e6a0740 1 -- 172.21.15.16:0/1866399433 >> 172.21.15.16:0/1866399433 conn(0x55a5ebd47460 msgr2=0x55a5ec0f6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb35e6a0740 1 -- 172.21.15.16:0/1866399433 shutdown_connections 2024-08-28T19:30:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb35e6a0740 1 -- 172.21.15.16:0/1866399433 wait complete. 2024-08-28T19:30:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb35e6a0740 1 -- start start 2024-08-28T19:30:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.631+0000 7fb35e6a0740 1 Processor -- start 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec107e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd32390 0x55a5ec1083c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec114120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5ebec1700 con 0x55a5ebd53070 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5ebfe5750 con 0x55a5ebd32390 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a5ec10adf0 con 0x55a5ebd2a1b0 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb355ec7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb3566c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec114120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb35e6a0740 10 client.0 _ll_register_callbacks cb 0x55a5ec014e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb3566c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec114120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33146/0 (socket says 172.21.15.16:33146) 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb355ec7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:37598/0 (socket says 172.21.15.16:37598) 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb3566c8640 1 -- 172.21.15.16:0/870550555 learned_addr learned my addr 172.21.15.16:0/870550555 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb3556c6640 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd32390 0x55a5ec1083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb3566c8640 1 -- 172.21.15.16:0/870550555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 msgr2=0x55a5ec107e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb3566c8640 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec107e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb3566c8640 1 -- 172.21.15.16:0/870550555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd32390 msgr2=0x55a5ec1083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb3566c8640 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd32390 0x55a5ec1083c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.632+0000 7fb3566c8640 1 -- 172.21.15.16:0/870550555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5ec01e900 con 0x55a5ebd53070 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.633+0000 7fb355ec7640 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.633+0000 7fb3566c8640 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec114120 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fb34800ed20 tx=0x7fb348004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a580b9df36558e6a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.633+0000 7fb3397fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.633+0000 7fb3397fa640 1 -- 172.21.15.16:0/870550555 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3480245f0 con 0x55a5ebd53070 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.633+0000 7fb3397fa640 1 -- 172.21.15.16:0/870550555 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fb348024790 con 0x55a5ebd53070 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.633+0000 7fb35e6a0740 1 -- 172.21.15.16:0/870550555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb340004030 con 0x55a5ebd53070 2024-08-28T19:30:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.633+0000 7fb3397fa640 1 -- 172.21.15.16:0/870550555 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fb348032050 con 0x55a5ebd53070 2024-08-28T19:30:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.633+0000 7fb35e6a0740 10 client.17985 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.633+0000 7fb35e6a0740 1 -- 172.21.15.16:0/870550555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55a5ec0fe6a0 con 0x55a5ebd53070 2024-08-28T19:30:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.633+0000 7fb3397fa640 1 -- 172.21.15.16:0/870550555 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb340004030 con 0x55a5ebd53070 2024-08-28T19:30:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.634+0000 7fb3397fa640 1 -- 172.21.15.16:0/870550555 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fb34802d7a0 con 0x55a5ebd53070 2024-08-28T19:30:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.634+0000 7fb35e6a0740 10 client.17985 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.634+0000 7fb35e6a0740 10 client.17985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.634+0000 7fb35e6a0740 20 client.17985 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.634+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5ebd119f0 0x55a5ec101c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.634+0000 7fb35e6a0740 4 client.17985 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.634+0000 7fb35e6a0740 1 -- 172.21.15.16:0/870550555 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5ec0715d0 con 0x55a5ebd119f0 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.634+0000 7fb355ec7640 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5ebd119f0 0x55a5ec101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.634+0000 7fb3397fa640 10 client.17985 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.634+0000 7fb355ec7640 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5ebd119f0 0x55a5ec101c60 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd7c612e2bb2569f server_cookie=53cc0adfa28cdfa8 in_seq=0 out_seq=0 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb3397fa640 1 -- 172.21.15.16:0/870550555 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a5ec0715d0 con 0x55a5ebd119f0 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb3397fa640 10 client.17985 handle_command_reply: tid=0 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 1 client.17985 shutdown 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 1 -- 172.21.15.16:0/870550555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 msgr2=0x55a5ec114120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec114120 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fb34800ed20 tx=0x7fb348004ab0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 1 -- 172.21.15.16:0/870550555 shutdown_connections 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec114120 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd32390 0x55a5ec1083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5ebd119f0 0x55a5ec101c60 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/870550555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb3397fa640 0 client.17985 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 1 -- 172.21.15.16:0/870550555 >> 172.21.15.16:0/870550555 conn(0x55a5ebd47460 msgr2=0x55a5ec0f47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 1 -- 172.21.15.16:0/870550555 shutdown_connections 2024-08-28T19:30:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 1 -- 172.21.15.16:0/870550555 wait complete. 2024-08-28T19:30:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.635+0000 7fb35e6a0740 20 client.17985 trim_cache size 0 max 16384 2024-08-28T19:30:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb35e6a0740 1 Processor -- start 2024-08-28T19:30:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb35e6a0740 1 -- start start 2024-08-28T19:30:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec0fdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd32390 0x55a5ec114090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec105070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3501aeab0 con 0x55a5ebd53070 2024-08-28T19:30:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb3501b0650 con 0x55a5ebd32390 2024-08-28T19:30:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb3501af1e0 con 0x55a5ebd2a1b0 2024-08-28T19:30:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb3566c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec105070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb3566c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec105070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33160/0 (socket says 172.21.15.16:33160) 2024-08-28T19:30:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb3566c8640 1 -- 172.21.15.16:0/308599605 learned_addr learned my addr 172.21.15.16:0/308599605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb355ec7640 1 --2- 172.21.15.16:0/308599605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec0fdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.649+0000 7fb3556c6640 1 --2- 172.21.15.16:0/308599605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd32390 0x55a5ec114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.650+0000 7fb3566c8640 1 -- 172.21.15.16:0/308599605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 msgr2=0x55a5ec0fdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.650+0000 7fb3566c8640 1 --2- 172.21.15.16:0/308599605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec0fdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.650+0000 7fb3566c8640 1 -- 172.21.15.16:0/308599605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd32390 msgr2=0x55a5ec114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.650+0000 7fb3566c8640 1 --2- 172.21.15.16:0/308599605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd32390 0x55a5ec114090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.650+0000 7fb3566c8640 1 -- 172.21.15.16:0/308599605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5ec106380 con 0x55a5ebd53070 2024-08-28T19:30:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.650+0000 7fb3566c8640 1 --2- 172.21.15.16:0/308599605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec105070 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fb34802dc40 tx=0x7fb3480049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6b8cea9433c042 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.650+0000 7fb33b7fe640 1 -- 172.21.15.16:0/308599605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348004b20 con 0x55a5ebd53070 2024-08-28T19:30:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.650+0000 7fb33b7fe640 1 -- 172.21.15.16:0/308599605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb348029070 con 0x55a5ebd53070 2024-08-28T19:30:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.650+0000 7fb33b7fe640 1 -- 172.21.15.16:0/308599605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348005450 con 0x55a5ebd53070 2024-08-28T19:30:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.651+0000 7fb35e6a0740 1 -- 172.21.15.16:0/308599605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 msgr2=0x55a5ec105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.651+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/308599605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec105070 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fb34802dc40 tx=0x7fb3480049d0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.651+0000 7fb35e6a0740 1 -- 172.21.15.16:0/308599605 shutdown_connections 2024-08-28T19:30:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.651+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/308599605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd53070 0x55a5ec105070 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.651+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/308599605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd32390 0x55a5ec114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.651+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/308599605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec0fdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.651+0000 7fb35e6a0740 1 -- 172.21.15.16:0/308599605 >> 172.21.15.16:0/308599605 conn(0x55a5ebd47460 msgr2=0x55a5ec0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.651+0000 7fb35e6a0740 1 -- 172.21.15.16:0/308599605 shutdown_connections 2024-08-28T19:30:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.651+0000 7fb35e6a0740 1 -- 172.21.15.16:0/308599605 wait complete. 2024-08-28T19:30:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.651+0000 7fb35e6a0740 1 -- start start 2024-08-28T19:30:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.651+0000 7fb35e6a0740 1 Processor -- start 2024-08-28T19:30:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec119c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 0x55a5ec11a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb35e6a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 0x55a5ec11a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3501aff80 con 0x55a5ebd32390 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb3501af8b0 con 0x55a5ebd53070 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb35e6a0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb3501acab0 con 0x55a5ebd2a1b0 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb35e6a0740 10 client.0 _ll_register_callbacks cb 0x7fb34800ff90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb3556c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 0x55a5ec11a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb355ec7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec119c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb3556c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 0x55a5ec11a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33176/0 (socket says 172.21.15.16:33176) 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb3556c6640 1 -- 172.21.15.16:0/353984951 learned_addr learned my addr 172.21.15.16:0/353984951 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb3566c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 0x55a5ec11a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb3556c6640 1 -- 172.21.15.16:0/353984951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 msgr2=0x55a5ec119c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb3556c6640 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec119c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb3556c6640 1 -- 172.21.15.16:0/353984951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 msgr2=0x55a5ec11a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb3556c6640 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 0x55a5ec11a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb3556c6640 1 -- 172.21.15.16:0/353984951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5ec17fe50 con 0x55a5ebd32390 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb355ec7640 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec119c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.652+0000 7fb3566c8640 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 0x55a5ec11a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb3556c6640 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 0x55a5ec11a150 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fb340009560 tx=0x7fb34002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5933e643595767 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb3397fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb3397fa640 1 -- 172.21.15.16:0/353984951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340035040 con 0x55a5ebd32390 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb3397fa640 1 -- 172.21.15.16:0/353984951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fb340005580 con 0x55a5ebd32390 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb35e6a0740 1 -- 172.21.15.16:0/353984951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb348029070 con 0x55a5ebd32390 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb3397fa640 1 -- 172.21.15.16:0/353984951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fb340040040 con 0x55a5ebd32390 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb35e6a0740 10 client.17991 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb35e6a0740 1 -- 172.21.15.16:0/353984951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55a5ec1810d0 con 0x55a5ebd32390 2024-08-28T19:30:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb3397fa640 1 -- 172.21.15.16:0/353984951 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb348029070 con 0x55a5ebd32390 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb3397fa640 1 -- 172.21.15.16:0/353984951 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fb34004e980 con 0x55a5ebd32390 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb35e6a0740 10 client.17991 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb35e6a0740 10 client.17991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.653+0000 7fb35e6a0740 20 client.17991 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.654+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5ebce6da0 0x55a5ec1833c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.654+0000 7fb35e6a0740 4 client.17991 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.654+0000 7fb35e6a0740 1 -- 172.21.15.16:0/353984951 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5ec06d070 con 0x55a5ebce6da0 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.654+0000 7fb355ec7640 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5ebce6da0 0x55a5ec1833c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.654+0000 7fb3397fa640 10 client.17991 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.654+0000 7fb355ec7640 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5ebce6da0 0x55a5ec1833c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=666e0132666ae169 server_cookie=75c8822ba693a422 in_seq=0 out_seq=0 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.654+0000 7fb3397fa640 1 -- 172.21.15.16:0/353984951 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5ec06d070 con 0x55a5ebce6da0 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.654+0000 7fb3397fa640 10 client.17991 handle_command_reply: tid=0 2024-08-28T19:30:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.654+0000 7fb35e6a0740 1 client.17991 shutdown 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.654+0000 7fb35e6a0740 1 -- 172.21.15.16:0/353984951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 msgr2=0x55a5ec11a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.654+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 0x55a5ec11a150 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fb340009560 tx=0x7fb34002a410 comp rx=0 tx=0).stop 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35e6a0740 1 -- 172.21.15.16:0/353984951 shutdown_connections 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ebd53070 0x55a5ec11a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5ebd32390 0x55a5ec11a150 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5ebce6da0 0x55a5ec1833c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35e6a0740 1 --2- 172.21.15.16:0/353984951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5ebd2a1b0 0x55a5ec119c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35e6a0740 1 -- 172.21.15.16:0/353984951 >> 172.21.15.16:0/353984951 conn(0x55a5ebd47460 msgr2=0x55a5ec0f9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb3397fa640 0 client.17991 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35e6a0740 1 -- 172.21.15.16:0/353984951 shutdown_connections 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35e6a0740 1 -- 172.21.15.16:0/353984951 wait complete. 2024-08-28T19:30:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35e6a0740 20 client.17991 trim_cache size 0 max 16384 2024-08-28T19:30:43.662 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:30:43.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35812b640 1 -- 172.21.15.16:0/4140597572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb33c0b4ca0 msgr2=0x7fb33c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:43.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35812b640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb33c0b4ca0 0x7fb33c0b71b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb34800f9b0 tx=0x7fb348010040 comp rx=0 tx=0).stop 2024-08-28T19:30:43.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35812b640 1 -- 172.21.15.16:0/4140597572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb350102800 msgr2=0x7fb350199190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:43.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35812b640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb350102800 0x7fb350199190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00d6d0 tx=0x7fb34c00db90 comp rx=0 tx=0).stop 2024-08-28T19:30:43.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35812b640 1 -- 172.21.15.16:0/4140597572 shutdown_connections 2024-08-28T19:30:43.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35812b640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb350108a90 0x7fb35019e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35812b640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb350108140 0x7fb350199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35812b640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb33c0b4ca0 0x7fb33c0b71b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35812b640 1 --2- 172.21.15.16:0/4140597572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb350102800 0x7fb350199190 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:43.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.655+0000 7fb35812b640 1 -- 172.21.15.16:0/4140597572 >> 172.21.15.16:0/4140597572 conn(0x7fb3500794d0 msgr2=0x7fb350104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:43.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.656+0000 7fb35812b640 1 -- 172.21.15.16:0/4140597572 shutdown_connections 2024-08-28T19:30:43.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:43.656+0000 7fb35812b640 1 -- 172.21.15.16:0/4140597572 wait complete. 2024-08-28T19:30:43.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 433689 4.21 MB/sec execute 383 sec latency 19.304 ms 2024-08-28T19:30:44.084 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 147.05 --await-for 45.0' 2024-08-28T19:30:44.084 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 147.05 --await-for 45.0 2024-08-28T19:30:44.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:44 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:44 smithi016 ceph-mon[18556]: pgmap v636: 129 pgs: 129 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-08-28T19:30:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:44 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:44 smithi028 ceph-mon[21769]: pgmap v636: 129 pgs: 129 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-08-28T19:30:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:44 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:44 smithi115 ceph-mon[22184]: pgmap v636: 129 pgs: 129 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-08-28T19:30:44.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434458 4.21 MB/sec execute 384 sec latency 405.840 ms 2024-08-28T19:30:45.037 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:45.725 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:30:45.725 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:30:45.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.846+0000 7fa8e522d640 1 -- 172.21.15.16:0/1102559176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e0108ae0 msgr2=0x7fa8e010f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:45.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.846+0000 7fa8e522d640 1 --2- 172.21.15.16:0/1102559176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e0108ae0 0x7fa8e010f310 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009ca0 tx=0x7fa8d802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.846+0000 7fa8e522d640 1 -- 172.21.15.16:0/1102559176 shutdown_connections 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.846+0000 7fa8e522d640 1 --2- 172.21.15.16:0/1102559176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e0108ae0 0x7fa8e010f310 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.846+0000 7fa8e522d640 1 --2- 172.21.15.16:0/1102559176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8e0108190 0x7fa8e01085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.846+0000 7fa8e522d640 1 --2- 172.21.15.16:0/1102559176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e0102850 0x7fa8e0102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.846+0000 7fa8e522d640 1 -- 172.21.15.16:0/1102559176 >> 172.21.15.16:0/1102559176 conn(0x7fa8e0079610 msgr2=0x7fa8e0079a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.846+0000 7fa8e522d640 1 -- 172.21.15.16:0/1102559176 shutdown_connections 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.846+0000 7fa8e522d640 1 -- 172.21.15.16:0/1102559176 wait complete. 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.847+0000 7fa8e522d640 1 Processor -- start 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.847+0000 7fa8e522d640 1 -- start start 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.847+0000 7fa8e522d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e0102850 0x7fa8e019af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.847+0000 7fa8e522d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e0108190 0x7fa8e019b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.847+0000 7fa8e522d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8e0108ae0 0x7fa8e019b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.847+0000 7fa8e522d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01127b0 con 0x7fa8e0102850 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.847+0000 7fa8e522d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e0112630 con 0x7fa8e0108190 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.847+0000 7fa8e522d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e0112930 con 0x7fa8e0108ae0 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8df59e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8e0108ae0 0x7fa8e019b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8ded9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e0102850 0x7fa8e019af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8df59e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8e0108ae0 0x7fa8e019b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:49622/0 (socket says 172.21.15.16:49622) 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8ded9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e0102850 0x7fa8e019af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40246/0 (socket says 172.21.15.16:40246) 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8de59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e0108190 0x7fa8e019b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8df59e640 1 -- 172.21.15.16:0/2333220621 learned_addr learned my addr 172.21.15.16:0/2333220621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8ded9d640 1 -- 172.21.15.16:0/2333220621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8e0108ae0 msgr2=0x7fa8e019b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8ded9d640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8e0108ae0 0x7fa8e019b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8ded9d640 1 -- 172.21.15.16:0/2333220621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e0108190 msgr2=0x7fa8e019b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8ded9d640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e0108190 0x7fa8e019b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8ded9d640 1 -- 172.21.15.16:0/2333220621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e01a6270 con 0x7fa8e0102850 2024-08-28T19:30:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8ded9d640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e0102850 0x7fa8e019af60 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000d6d0 tx=0x7fa8d000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8c7fff640 1 -- 172.21.15.16:0/2333220621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0017070 con 0x7fa8e0102850 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8e522d640 1 -- 172.21.15.16:0/2333220621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e01a6560 con 0x7fa8e0102850 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.848+0000 7fa8c7fff640 1 -- 172.21.15.16:0/2333220621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8d0009820 con 0x7fa8e0102850 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.849+0000 7fa8c7fff640 1 -- 172.21.15.16:0/2333220621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0004c40 con 0x7fa8e0102850 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.849+0000 7fa8e522d640 1 -- 172.21.15.16:0/2333220621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e01a6af0 con 0x7fa8e0102850 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.851+0000 7fa8c7fff640 1 -- 172.21.15.16:0/2333220621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa8d0010460 con 0x7fa8e0102850 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.851+0000 7fa8eb7a2740 1 Processor -- start 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8eb7a2740 1 -- start start 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925af1b850 0x55925b2c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925aef2990 0x55925b2c24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55925b237380 con 0x55925aefab70 2024-08-28T19:30:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55925b226910 con 0x55925af1b850 2024-08-28T19:30:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55925b22a8c0 con 0x55925aef2990 2024-08-28T19:30:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8c7fff640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8c80b4ca0 0x7fa8c80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8de59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8de59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40258/0 (socket says 172.21.15.16:40258) 2024-08-28T19:30:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8de59c640 1 -- 172.21.15.16:0/1277579804 learned_addr learned my addr 172.21.15.16:0/1277579804 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8df59e640 1 --2- 172.21.15.16:0/1277579804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925af1b850 0x55925b2c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8df59e640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8c80b4ca0 0x7fa8c80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8ded9d640 1 --2- 172.21.15.16:0/1277579804 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925aef2990 0x55925b2c24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.852+0000 7fa8c7fff640 1 -- 172.21.15.16:0/2333220621 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fa8d00e5440 con 0x7fa8e0102850 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.853+0000 7fa8de59c640 1 -- 172.21.15.16:0/1277579804 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925aef2990 msgr2=0x55925b2c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.853+0000 7fa8de59c640 1 --2- 172.21.15.16:0/1277579804 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925aef2990 0x55925b2c24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.853+0000 7fa8de59c640 1 -- 172.21.15.16:0/1277579804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925af1b850 msgr2=0x55925b2c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.853+0000 7fa8de59c640 1 --2- 172.21.15.16:0/1277579804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925af1b850 0x55925b2c1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.853+0000 7fa8de59c640 1 -- 172.21.15.16:0/1277579804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55925b1c34c0 con 0x55925aefab70 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.853+0000 7fa8de59c640 1 --2- 172.21.15.16:0/1277579804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2c2a10 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400b690 tx=0x7fa8d400bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361430e4f6dc0bf7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.853+0000 7fa8df59e640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8c80b4ca0 0x7fa8c80b71b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009a60 tx=0x7fa8d8005c50 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.853+0000 7fa8c57fa640 1 -- 172.21.15.16:0/1277579804 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4003bd0 con 0x55925aefab70 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.853+0000 7fa8c57fa640 1 -- 172.21.15.16:0/1277579804 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8d4015070 con 0x55925aefab70 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.853+0000 7fa8c57fa640 1 -- 172.21.15.16:0/1277579804 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4004d10 con 0x55925aefab70 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.854+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1277579804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 msgr2=0x55925b2c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.854+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1277579804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2c2a10 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400b690 tx=0x7fa8d400bb50 comp rx=0 tx=0).stop 2024-08-28T19:30:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.854+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1277579804 shutdown_connections 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.854+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1277579804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2c2a10 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.854+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1277579804 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925aef2990 0x55925b2c24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.854+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1277579804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925af1b850 0x55925b2c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.854+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1277579804 >> 172.21.15.16:0/1277579804 conn(0x55925af0fc40 msgr2=0x55925b2bfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.855+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1277579804 shutdown_connections 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.855+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1277579804 wait complete. 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.855+0000 7fa8eb7a2740 1 -- start start 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.855+0000 7fa8eb7a2740 1 Processor -- start 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.855+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2cd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.855+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.855+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925af1b850 0x55925b2d9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.855+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55925b0897b0 con 0x55925aefab70 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.855+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55925b1ad460 con 0x55925aef2990 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.855+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55925b2d0770 con 0x55925af1b850 2024-08-28T19:30:45.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.855+0000 7fa8eb7a2740 10 client.0 _ll_register_callbacks cb 0x55925b1d2fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8de59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8de59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40262/0 (socket says 172.21.15.16:40262) 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8de59c640 1 -- 172.21.15.16:0/1975849790 learned_addr learned my addr 172.21.15.16:0/1975849790 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8ded9d640 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8df59e640 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925af1b850 0x55925b2d9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8de59c640 1 -- 172.21.15.16:0/1975849790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925af1b850 msgr2=0x55925b2d9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8de59c640 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925af1b850 0x55925b2d9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8de59c640 1 -- 172.21.15.16:0/1975849790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 msgr2=0x55925b2cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8de59c640 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2cd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8de59c640 1 -- 172.21.15.16:0/1975849790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55925af95ba0 con 0x55925aefab70 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8df59e640 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925af1b850 0x55925b2d9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:30:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8de59c640 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2cdba0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40049b0 tx=0x7fa8d400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c434ef60cdceed24 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/1975849790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4003de0 con 0x55925aefab70 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/1975849790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fa8d4011440 con 0x55925aefab70 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.856+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1975849790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8d4015070 con 0x55925aefab70 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/1975849790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fa8d4030050 con 0x55925aefab70 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8eb7a2740 10 client.18003 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1975849790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55925b2dc820 con 0x55925aefab70 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/1975849790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8d4015070 con 0x55925aefab70 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/1975849790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fa8d40257d0 con 0x55925aefab70 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8eb7a2740 10 client.18003 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8eb7a2740 10 client.18003 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8eb7a2740 20 client.18003 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55925aeda1d0 0x55925b2dfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8eb7a2740 4 client.18003 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1975849790 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55925b2392d0 con 0x55925aeda1d0 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.857+0000 7fa8ded9d640 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55925aeda1d0 0x55925b2dfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.858+0000 7fa8c2ff5640 10 client.18003 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.858+0000 7fa8ded9d640 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55925aeda1d0 0x55925b2dfc30 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd7b21ab07d5b01a server_cookie=764ea654f1db552b in_seq=0 out_seq=0 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/1975849790 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55925b2392d0 con 0x55925aeda1d0 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8c2ff5640 10 client.18003 handle_command_reply: tid=0 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 1 client.18003 shutdown 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1975849790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 msgr2=0x55925b2cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2cdba0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40049b0 tx=0x7fa8d400bfd0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1975849790 shutdown_connections 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55925aeda1d0 0x55925b2dfc30 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925af1b850 0x55925b2d9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2cdba0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8c2ff5640 0 client.18003 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1975849790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1975849790 >> 172.21.15.16:0/1975849790 conn(0x55925af0fc40 msgr2=0x55925b2c5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1975849790 shutdown_connections 2024-08-28T19:30:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1975849790 wait complete. 2024-08-28T19:30:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.859+0000 7fa8eb7a2740 20 client.18003 trim_cache size 0 max 16384 2024-08-28T19:30:45.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8eb7a2740 1 Processor -- start 2024-08-28T19:30:45.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8eb7a2740 1 -- start start 2024-08-28T19:30:45.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925aefab70 0x55925b2d9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925af1b850 0x55925b2beee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01b0be0 con 0x55925af1b850 2024-08-28T19:30:45.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01b2780 con 0x55925aef2990 2024-08-28T19:30:45.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01b1310 con 0x55925aefab70 2024-08-28T19:30:45.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8df59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925af1b850 0x55925b2beee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8df59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925af1b850 0x55925b2beee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40270/0 (socket says 172.21.15.16:40270) 2024-08-28T19:30:45.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8df59e640 1 -- 172.21.15.16:0/1205625216 learned_addr learned my addr 172.21.15.16:0/1205625216 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:45.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8ded9d640 1 --2- 172.21.15.16:0/1205625216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8de59c640 1 --2- 172.21.15.16:0/1205625216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925aefab70 0x55925b2d9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8df59e640 1 -- 172.21.15.16:0/1205625216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925aefab70 msgr2=0x55925b2d9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8df59e640 1 --2- 172.21.15.16:0/1205625216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925aefab70 0x55925b2d9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8df59e640 1 -- 172.21.15.16:0/1205625216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 msgr2=0x55925b2cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8df59e640 1 --2- 172.21.15.16:0/1205625216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.877+0000 7fa8df59e640 1 -- 172.21.15.16:0/1205625216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55925b2ce920 con 0x55925af1b850 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.878+0000 7fa8df59e640 1 --2- 172.21.15.16:0/1205625216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925af1b850 0x55925b2beee0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80097b0 tx=0x7fa8d8004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872be9d80b5d76a2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.878+0000 7fa8c4ff9640 1 -- 172.21.15.16:0/1205625216 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8003b70 con 0x55925af1b850 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.878+0000 7fa8c4ff9640 1 -- 172.21.15.16:0/1205625216 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8d8036070 con 0x55925af1b850 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.878+0000 7fa8c4ff9640 1 -- 172.21.15.16:0/1205625216 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d802f8c0 con 0x55925af1b850 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.879+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1205625216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925af1b850 msgr2=0x55925b2beee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.879+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1205625216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925af1b850 0x55925b2beee0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80097b0 tx=0x7fa8d8004a60 comp rx=0 tx=0).stop 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.879+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1205625216 shutdown_connections 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.879+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1205625216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925af1b850 0x55925b2beee0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.879+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1205625216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925aefab70 0x55925b2d9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.879+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/1205625216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.879+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1205625216 >> 172.21.15.16:0/1205625216 conn(0x55925af0fc40 msgr2=0x55925b2bfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:45.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.879+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1205625216 shutdown_connections 2024-08-28T19:30:45.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.879+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/1205625216 wait complete. 2024-08-28T19:30:45.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.879+0000 7fa8eb7a2740 1 -- start start 2024-08-28T19:30:45.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.879+0000 7fa8eb7a2740 1 Processor -- start 2024-08-28T19:30:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2d3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8eb7a2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925af1b850 0x55925b2d36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01b20b0 con 0x55925aefab70 2024-08-28T19:30:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01b19e0 con 0x55925aef2990 2024-08-28T19:30:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8eb7a2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01aebe0 con 0x55925af1b850 2024-08-28T19:30:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8eb7a2740 10 client.0 _ll_register_callbacks cb 0x7fa8d40044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8de59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2d3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8de59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2d3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40284/0 (socket says 172.21.15.16:40284) 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8de59c640 1 -- 172.21.15.16:0/2632993858 learned_addr learned my addr 172.21.15.16:0/2632993858 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8df59e640 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925af1b850 0x55925b2d36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8ded9d640 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8de59c640 1 -- 172.21.15.16:0/2632993858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925af1b850 msgr2=0x55925b2d36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8de59c640 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925af1b850 0x55925b2d36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8de59c640 1 -- 172.21.15.16:0/2632993858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 msgr2=0x55925b2d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8de59c640 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2d2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8de59c640 1 -- 172.21.15.16:0/2632993858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55925b34a6f0 con 0x55925aefab70 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8de59c640 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2d3170 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4025bd0 tx=0x7fa8d4004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6f1fd6ff6bf1a5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.880+0000 7fa8c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/2632993858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4038070 con 0x55925aefab70 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/2632993858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fa8d40047a0 con 0x55925aefab70 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/2632993858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8d8036070 con 0x55925aefab70 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/2632993858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fa8d4044050 con 0x55925aefab70 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8eb7a2740 10 client.18009 fetch_fsmap learned FSMap version 53 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/2632993858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55925b34b950 con 0x55925aefab70 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/2632993858 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8d8036070 con 0x55925aefab70 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/2632993858 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) ==== 8378+0+0 (secure 0 0 0) 0x7fa8d4011bd0 con 0x55925aefab70 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8eb7a2740 10 client.18009 fetch_fsmap finished waiting for FSMap version 53 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8eb7a2740 10 client.18009 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8eb7a2740 20 client.18009 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55925aeaf580 0x55925b34e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8eb7a2740 4 client.18009 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.05, "await_for": 45.0}] 2024-08-28T19:30:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.881+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/2632993858 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.05, "await_for": 45.0}) -- 0x55925b234d70 con 0x55925aeaf580 2024-08-28T19:30:45.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.882+0000 7fa8ded9d640 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55925aeaf580 0x55925b34e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:45.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.882+0000 7fa8c2ff5640 10 client.18009 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:45.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.882+0000 7fa8ded9d640 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55925aeaf580 0x55925b34e090 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2eca53670f5bc0a server_cookie=325f0b20b22b8fdc in_seq=0 out_seq=0 2024-08-28T19:30:45.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8c2ff5640 1 -- 172.21.15.16:0/2632993858 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55925b234d70 con 0x55925aeaf580 2024-08-28T19:30:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8c2ff5640 10 client.18009 handle_command_reply: tid=0 2024-08-28T19:30:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 1 client.18009 shutdown 2024-08-28T19:30:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/2632993858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 msgr2=0x55925b2d3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2d3170 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4025bd0 tx=0x7fa8d4004630 comp rx=0 tx=0).stop 2024-08-28T19:30:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/2632993858 shutdown_connections 2024-08-28T19:30:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55925aeaf580 0x55925b34e090 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55925af1b850 0x55925b2d36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55925aefab70 0x55925b2d3170 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 1 --2- 172.21.15.16:0/2632993858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55925aef2990 0x55925b2d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8c2ff5640 0 client.18009 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/2632993858 >> 172.21.15.16:0/2632993858 conn(0x55925af0fc40 msgr2=0x55925b2bfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:45.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/2632993858 shutdown_connections 2024-08-28T19:30:45.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 1 -- 172.21.15.16:0/2632993858 wait complete. 2024-08-28T19:30:45.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.964+0000 7fa8eb7a2740 20 client.18009 trim_cache size 0 max 16384 2024-08-28T19:30:45.967 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:30:45.967 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 53, 2024-08-28T19:30:45.967 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:30:45.967 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 30, 2024-08-28T19:30:45.967 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "c46102d0": { 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 30, 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.1, 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:30:45.968 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:30:45.969 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:30:45.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:45.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:45.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:45.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:45.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:45.969 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:30:45.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.20 MB/sec execute 385 sec latency 1354.491 ms 2024-08-28T19:30:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 -- 172.21.15.16:0/2333220621 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8c80b4ca0 msgr2=0x7fa8c80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8c80b4ca0 0x7fa8c80b71b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009a60 tx=0x7fa8d8005c50 comp rx=0 tx=0).stop 2024-08-28T19:30:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 -- 172.21.15.16:0/2333220621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e0102850 msgr2=0x7fa8e019af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e0102850 0x7fa8e019af60 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000d6d0 tx=0x7fa8d000db90 comp rx=0 tx=0).stop 2024-08-28T19:30:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 -- 172.21.15.16:0/2333220621 shutdown_connections 2024-08-28T19:30:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8e0108ae0 0x7fa8e019b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8c80b4ca0 0x7fa8c80b71b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e0108190 0x7fa8e019b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 --2- 172.21.15.16:0/2333220621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e0102850 0x7fa8e019af60 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 -- 172.21.15.16:0/2333220621 >> 172.21.15.16:0/2333220621 conn(0x7fa8e0079610 msgr2=0x7fa8e0106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 -- 172.21.15.16:0/2333220621 shutdown_connections 2024-08-28T19:30:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:45.965+0000 7fa8e522d640 1 -- 172.21.15.16:0/2333220621 wait complete. 2024-08-28T19:30:46.330 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:46.757 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c46102d0', quiesce took 5.1 seconds. Will release after: 21.950000000000003 2024-08-28T19:30:46.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.19 MB/sec execute 386 sec latency 1051.390 ms 2024-08-28T19:30:47.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:46 smithi016 ceph-mon[18556]: pgmap v637: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2024-08-28T19:30:47.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:46 smithi028 ceph-mon[21769]: pgmap v637: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2024-08-28T19:30:47.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:46 smithi115 ceph-mon[22184]: pgmap v637: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2024-08-28T19:30:47.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.256+0000 7fb21d0c7640 1 -- 172.21.15.16:0/1358783854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb218108170 msgr2=0x7fb218108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:47.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.256+0000 7fb21d0c7640 1 --2- 172.21.15.16:0/1358783854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb218108170 0x7fb218108580 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009900 tx=0x7fb20c02a410 comp rx=0 tx=0).stop 2024-08-28T19:30:47.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.256+0000 7fb21d0c7640 1 -- 172.21.15.16:0/1358783854 shutdown_connections 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.256+0000 7fb21d0c7640 1 --2- 172.21.15.16:0/1358783854 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb218108ac0 0x7fb21810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.256+0000 7fb21d0c7640 1 --2- 172.21.15.16:0/1358783854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb218108170 0x7fb218108580 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.256+0000 7fb21d0c7640 1 --2- 172.21.15.16:0/1358783854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb218102880 0x7fb218102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.256+0000 7fb21d0c7640 1 -- 172.21.15.16:0/1358783854 >> 172.21.15.16:0/1358783854 conn(0x7fb218079500 msgr2=0x7fb218079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.256+0000 7fb21d0c7640 1 -- 172.21.15.16:0/1358783854 shutdown_connections 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.256+0000 7fb21d0c7640 1 -- 172.21.15.16:0/1358783854 wait complete. 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.257+0000 7fb21d0c7640 1 Processor -- start 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.257+0000 7fb21d0c7640 1 -- start start 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21d0c7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb218102880 0x7fb21819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21d0c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb218108170 0x7fb21819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21d0c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb218108ac0 0x7fb21819fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21d0c7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb218112710 con 0x7fb218108170 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21d0c7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb218112590 con 0x7fb218108ac0 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21d0c7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb218112890 con 0x7fb218102880 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb218108170 0x7fb21819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb218108170 0x7fb21819a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40288/0 (socket says 172.21.15.16:40288) 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21659c640 1 -- 172.21.15.16:0/465154578 learned_addr learned my addr 172.21.15.16:0/465154578 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb216d9d640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb218102880 0x7fb21819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21759e640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb218108ac0 0x7fb21819fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21659c640 1 -- 172.21.15.16:0/465154578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb218102880 msgr2=0x7fb21819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21659c640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb218102880 0x7fb21819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21659c640 1 -- 172.21.15.16:0/465154578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb218108ac0 msgr2=0x7fb21819fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21659c640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb218108ac0 0x7fb21819fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21659c640 1 -- 172.21.15.16:0/465154578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2181a01c0 con 0x7fb218108170 2024-08-28T19:30:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.258+0000 7fb21659c640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb218108170 0x7fb21819a7f0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb20c006010 tx=0x7fb20c004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.259+0000 7fb1fffff640 1 -- 172.21.15.16:0/465154578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c03f030 con 0x7fb218108170 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.259+0000 7fb1fffff640 1 -- 172.21.15.16:0/465154578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb20c003aa0 con 0x7fb218108170 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.259+0000 7fb21d0c7640 1 -- 172.21.15.16:0/465154578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2181a0450 con 0x7fb218108170 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.259+0000 7fb1fffff640 1 -- 172.21.15.16:0/465154578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c035070 con 0x7fb218108170 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.259+0000 7fb21d0c7640 1 -- 172.21.15.16:0/465154578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2181a6c10 con 0x7fb218108170 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.261+0000 7fb22363c740 1 Processor -- start 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.261+0000 7fb22363c740 1 -- start start 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.261+0000 7fb1fffff640 1 -- 172.21.15.16:0/465154578 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fb20c042030 con 0x7fb218108170 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb22363c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe47f350 0x5582fe825780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb22363c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe456490 0x5582fe825cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb22363c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe45e670 0x5582fe826200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb22363c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5582fe79b5f0 con 0x5582fe456490 2024-08-28T19:30:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb22363c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5582fe78ab80 con 0x5582fe45e670 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb22363c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5582fe78eb30 con 0x5582fe47f350 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb216d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe456490 0x5582fe825cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb216d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe456490 0x5582fe825cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40302/0 (socket says 172.21.15.16:40302) 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb216d9d640 1 -- 172.21.15.16:0/652624338 learned_addr learned my addr 172.21.15.16:0/652624338 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb21659c640 1 --2- 172.21.15.16:0/652624338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe45e670 0x5582fe826200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb21759e640 1 --2- 172.21.15.16:0/652624338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe47f350 0x5582fe825780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb216d9d640 1 -- 172.21.15.16:0/652624338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe47f350 msgr2=0x5582fe825780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb216d9d640 1 --2- 172.21.15.16:0/652624338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe47f350 0x5582fe825780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb216d9d640 1 -- 172.21.15.16:0/652624338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe45e670 msgr2=0x5582fe826200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb216d9d640 1 --2- 172.21.15.16:0/652624338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe45e670 0x5582fe826200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb216d9d640 1 -- 172.21.15.16:0/652624338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582fe4ea930 con 0x5582fe456490 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb21759e640 1 --2- 172.21.15.16:0/652624338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe47f350 0x5582fe825780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:30:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.262+0000 7fb216d9d640 1 --2- 172.21.15.16:0/652624338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe456490 0x5582fe825cc0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb208009d20 tx=0x7fb208011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cdd6b98a63c9e6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:47.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.263+0000 7fb1fffff640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 0x7fb2000b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.263+0000 7fb1fd7fa640 1 -- 172.21.15.16:0/652624338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208004230 con 0x5582fe456490 2024-08-28T19:30:47.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.263+0000 7fb1fd7fa640 1 -- 172.21.15.16:0/652624338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2080043d0 con 0x5582fe456490 2024-08-28T19:30:47.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.263+0000 7fb1fd7fa640 1 -- 172.21.15.16:0/652624338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2080197c0 con 0x5582fe456490 2024-08-28T19:30:47.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.263+0000 7fb21759e640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 0x7fb2000b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.263+0000 7fb1fffff640 1 -- 172.21.15.16:0/465154578 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fb20c105ce0 con 0x7fb218108170 2024-08-28T19:30:47.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.263+0000 7fb22363c740 1 -- 172.21.15.16:0/652624338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe456490 msgr2=0x5582fe825cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:47.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.263+0000 7fb21759e640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 0x7fb2000b71b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb210005fd0 tx=0x7fb210009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:47.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.263+0000 7fb22363c740 1 --2- 172.21.15.16:0/652624338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe456490 0x5582fe825cc0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb208009d20 tx=0x7fb208011810 comp rx=0 tx=0).stop 2024-08-28T19:30:47.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.264+0000 7fb22363c740 1 -- 172.21.15.16:0/652624338 shutdown_connections 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.264+0000 7fb22363c740 1 --2- 172.21.15.16:0/652624338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe45e670 0x5582fe826200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.264+0000 7fb22363c740 1 --2- 172.21.15.16:0/652624338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe456490 0x5582fe825cc0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.264+0000 7fb22363c740 1 --2- 172.21.15.16:0/652624338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe47f350 0x5582fe825780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.264+0000 7fb22363c740 1 -- 172.21.15.16:0/652624338 >> 172.21.15.16:0/652624338 conn(0x5582fe473740 msgr2=0x5582fe823670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.264+0000 7fb22363c740 1 -- 172.21.15.16:0/652624338 shutdown_connections 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.264+0000 7fb22363c740 1 -- 172.21.15.16:0/652624338 wait complete. 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.264+0000 7fb22363c740 1 -- start start 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.264+0000 7fb22363c740 1 Processor -- start 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb22363c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe456490 0x5582fe835220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb22363c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe45e670 0x5582fe835760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb22363c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe8415f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb22363c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5582fe5ed890 con 0x5582fe47f350 2024-08-28T19:30:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb22363c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5582fe711790 con 0x5582fe456490 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb22363c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5582fe838290 con 0x5582fe45e670 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb22363c740 10 client.0 _ll_register_callbacks cb 0x5582fe496ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb21759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe8415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb21759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe8415f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40318/0 (socket says 172.21.15.16:40318) 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb21759e640 1 -- 172.21.15.16:0/3453874743 learned_addr learned my addr 172.21.15.16:0/3453874743 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb216d9d640 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe456490 0x5582fe835220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb21659c640 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe45e670 0x5582fe835760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb21759e640 1 -- 172.21.15.16:0/3453874743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe45e670 msgr2=0x5582fe835760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb21759e640 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe45e670 0x5582fe835760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb21759e640 1 -- 172.21.15.16:0/3453874743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe456490 msgr2=0x5582fe835220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb21759e640 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe456490 0x5582fe835220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb21759e640 1 -- 172.21.15.16:0/3453874743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582fe748f30 con 0x5582fe47f350 2024-08-28T19:30:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb21759e640 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe8415f0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb210009a30 tx=0x7fb210004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7733d7ea7c7b2bcd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb1faff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.265+0000 7fb1faff5640 1 -- 172.21.15.16:0/3453874743 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2100104f0 con 0x5582fe47f350 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb1faff5640 1 -- 172.21.15.16:0/3453874743 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fb210010690 con 0x5582fe47f350 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb22363c740 1 -- 172.21.15.16:0/3453874743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2080043d0 con 0x5582fe47f350 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb1faff5640 1 -- 172.21.15.16:0/3453874743 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fb210010890 con 0x5582fe47f350 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb22363c740 10 client.18021 fetch_fsmap learned FSMap version 54 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb22363c740 1 -- 172.21.15.16:0/3453874743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5582fe844040 con 0x5582fe47f350 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb1faff5640 1 -- 172.21.15.16:0/3453874743 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2080043d0 con 0x5582fe47f350 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb1faff5640 1 -- 172.21.15.16:0/3453874743 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 54) ==== 8374+0+0 (secure 0 0 0) 0x7fb21000f040 con 0x5582fe47f350 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb22363c740 10 client.18021 fetch_fsmap finished waiting for FSMap version 54 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb22363c740 10 client.18021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb22363c740 20 client.18021 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb22363c740 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5582fe43dcd0 0x5582fe8476b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb22363c740 4 client.18021 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb22363c740 1 -- 172.21.15.16:0/3453874743 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582fe79d540 con 0x5582fe43dcd0 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.266+0000 7fb216d9d640 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5582fe43dcd0 0x5582fe8476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.267+0000 7fb216d9d640 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5582fe43dcd0 0x5582fe8476b0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ccd1d3dc94293b6 server_cookie=b90d01a0259a64e7 in_seq=0 out_seq=0 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.267+0000 7fb1faff5640 10 client.18021 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb1faff5640 1 -- 172.21.15.16:0/3453874743 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5582fe79d540 con 0x5582fe43dcd0 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb1faff5640 10 client.18021 handle_command_reply: tid=0 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 1 client.18021 shutdown 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 1 -- 172.21.15.16:0/3453874743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 msgr2=0x5582fe8415f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe8415f0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb210009a30 tx=0x7fb210004a40 comp rx=0 tx=0).stop 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 1 -- 172.21.15.16:0/3453874743 shutdown_connections 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe8415f0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5582fe43dcd0 0x5582fe8476b0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe45e670 0x5582fe835760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb1faff5640 0 client.18021 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 1 --2- 172.21.15.16:0/3453874743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe456490 0x5582fe835220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 1 -- 172.21.15.16:0/3453874743 >> 172.21.15.16:0/3453874743 conn(0x5582fe473740 msgr2=0x5582fe8288d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 1 -- 172.21.15.16:0/3453874743 shutdown_connections 2024-08-28T19:30:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 1 -- 172.21.15.16:0/3453874743 wait complete. 2024-08-28T19:30:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.268+0000 7fb22363c740 20 client.18021 trim_cache size 0 max 16384 2024-08-28T19:30:47.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.289+0000 7fb22363c740 1 Processor -- start 2024-08-28T19:30:47.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.289+0000 7fb22363c740 1 -- start start 2024-08-28T19:30:47.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb22363c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe456490 0x5582fe826ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb22363c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe45e670 0x5582fe832410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb22363c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe832950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb22363c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb2181aeab0 con 0x5582fe47f350 2024-08-28T19:30:47.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb22363c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2181b0620 con 0x5582fe45e670 2024-08-28T19:30:47.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb22363c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb2181af1b0 con 0x5582fe456490 2024-08-28T19:30:47.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb21759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe832950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb21659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe45e670 0x5582fe832410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb21759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe832950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40322/0 (socket says 172.21.15.16:40322) 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb216d9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe456490 0x5582fe826ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb21759e640 1 -- 172.21.15.16:0/4211129556 learned_addr learned my addr 172.21.15.16:0/4211129556 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb21759e640 1 -- 172.21.15.16:0/4211129556 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe456490 msgr2=0x5582fe826ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb21759e640 1 --2- 172.21.15.16:0/4211129556 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe456490 0x5582fe826ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb21759e640 1 -- 172.21.15.16:0/4211129556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe45e670 msgr2=0x5582fe832410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb21759e640 1 --2- 172.21.15.16:0/4211129556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe45e670 0x5582fe832410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb21759e640 1 -- 172.21.15.16:0/4211129556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582fe834180 con 0x5582fe47f350 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb216d9d640 1 --2- 172.21.15.16:0/4211129556 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe456490 0x5582fe826ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.290+0000 7fb21659c640 1 --2- 172.21.15.16:0/4211129556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe45e670 0x5582fe832410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.291+0000 7fb21759e640 1 --2- 172.21.15.16:0/4211129556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe832950 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb210005b40 tx=0x7fb2100097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c083c8bd15f9010 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.291+0000 7fb1fcff9640 1 -- 172.21.15.16:0/4211129556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2100104f0 con 0x5582fe47f350 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.291+0000 7fb1fcff9640 1 -- 172.21.15.16:0/4211129556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb21003a070 con 0x5582fe47f350 2024-08-28T19:30:47.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.291+0000 7fb1fcff9640 1 -- 172.21.15.16:0/4211129556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210010e30 con 0x5582fe47f350 2024-08-28T19:30:47.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.292+0000 7fb22363c740 1 -- 172.21.15.16:0/4211129556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 msgr2=0x5582fe832950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:47.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.292+0000 7fb22363c740 1 --2- 172.21.15.16:0/4211129556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe832950 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb210005b40 tx=0x7fb2100097d0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.292+0000 7fb22363c740 1 -- 172.21.15.16:0/4211129556 shutdown_connections 2024-08-28T19:30:47.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.292+0000 7fb22363c740 1 --2- 172.21.15.16:0/4211129556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe47f350 0x5582fe832950 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.292+0000 7fb22363c740 1 --2- 172.21.15.16:0/4211129556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe45e670 0x5582fe832410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.292+0000 7fb22363c740 1 --2- 172.21.15.16:0/4211129556 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe456490 0x5582fe826ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.292+0000 7fb22363c740 1 -- 172.21.15.16:0/4211129556 >> 172.21.15.16:0/4211129556 conn(0x5582fe473740 msgr2=0x5582fe8267b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.292+0000 7fb22363c740 1 -- 172.21.15.16:0/4211129556 shutdown_connections 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.292+0000 7fb22363c740 1 -- 172.21.15.16:0/4211129556 wait complete. 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.293+0000 7fb22363c740 1 -- start start 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.293+0000 7fb22363c740 1 Processor -- start 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.293+0000 7fb22363c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe456490 0x5582fe83bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.293+0000 7fb22363c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe45e670 0x5582fe83c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.293+0000 7fb22363c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe47f350 0x5582fe8aa080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb22363c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb2181aff50 con 0x5582fe45e670 2024-08-28T19:30:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb22363c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2181af880 con 0x5582fe456490 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb22363c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb2181acab0 con 0x5582fe47f350 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb21659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe45e670 0x5582fe83c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb22363c740 10 client.0 _ll_register_callbacks cb 0x7fb210009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb21659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe45e670 0x5582fe83c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40332/0 (socket says 172.21.15.16:40332) 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb21659c640 1 -- 172.21.15.16:0/1406062074 learned_addr learned my addr 172.21.15.16:0/1406062074 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb216d9d640 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe456490 0x5582fe83bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb21759e640 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe47f350 0x5582fe8aa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb21659c640 1 -- 172.21.15.16:0/1406062074 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe47f350 msgr2=0x5582fe8aa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb21659c640 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe47f350 0x5582fe8aa080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb21659c640 1 -- 172.21.15.16:0/1406062074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe456490 msgr2=0x5582fe83bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb21659c640 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe456490 0x5582fe83bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb21659c640 1 -- 172.21.15.16:0/1406062074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582fe8aa6f0 con 0x5582fe45e670 2024-08-28T19:30:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb216d9d640 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe456490 0x5582fe83bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb21659c640 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe45e670 0x5582fe83c5d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0c8130 tx=0x7fb20c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3b46d6dc77e9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb1faff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.294+0000 7fb1faff5640 1 -- 172.21.15.16:0/1406062074 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c049070 con 0x5582fe45e670 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb22363c740 1 -- 172.21.15.16:0/1406062074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb21003a070 con 0x5582fe45e670 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb1faff5640 1 -- 172.21.15.16:0/1406062074 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7fb20c0cc310 con 0x5582fe45e670 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb1faff5640 1 -- 172.21.15.16:0/1406062074 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fb20c0c7f70 con 0x5582fe45e670 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb22363c740 10 client.18027 fetch_fsmap learned FSMap version 54 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb22363c740 1 -- 172.21.15.16:0/1406062074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5582fe8b2ce0 con 0x5582fe45e670 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb1faff5640 1 -- 172.21.15.16:0/1406062074 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb21003a070 con 0x5582fe45e670 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb1faff5640 1 -- 172.21.15.16:0/1406062074 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 54) ==== 8374+0+0 (secure 0 0 0) 0x7fb20c0d4e90 con 0x5582fe45e670 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb22363c740 10 client.18027 fetch_fsmap finished waiting for FSMap version 54 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb22363c740 10 client.18027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:47.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb22363c740 20 client.18027 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:47.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb22363c740 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5582fe413080 0x5582fe8b5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:47.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb22363c740 4 client.18027 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:30:47.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.295+0000 7fb22363c740 1 -- 172.21.15.16:0/1406062074 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5582fe798fe0 con 0x5582fe413080 2024-08-28T19:30:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.296+0000 7fb216d9d640 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5582fe413080 0x5582fe8b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.296+0000 7fb1faff5640 10 client.18027 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.296+0000 7fb216d9d640 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5582fe413080 0x5582fe8b5450 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4d78709906ab287 server_cookie=c18bb622709962d8 in_seq=0 out_seq=0 2024-08-28T19:30:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.298+0000 7fb1faff5640 1 -- 172.21.15.16:0/1406062074 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5582fe798fe0 con 0x5582fe413080 2024-08-28T19:30:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.298+0000 7fb1faff5640 10 client.18027 handle_command_reply: tid=0 2024-08-28T19:30:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.298+0000 7fb22363c740 1 client.18027 shutdown 2024-08-28T19:30:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.298+0000 7fb22363c740 1 -- 172.21.15.16:0/1406062074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe45e670 msgr2=0x5582fe83c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.298+0000 7fb22363c740 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe45e670 0x5582fe83c5d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0c8130 tx=0x7fb20c004610 comp rx=0 tx=0).stop 2024-08-28T19:30:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.299+0000 7fb22363c740 1 -- 172.21.15.16:0/1406062074 shutdown_connections 2024-08-28T19:30:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.299+0000 7fb22363c740 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5582fe413080 0x5582fe8b5450 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.299+0000 7fb22363c740 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5582fe47f350 0x5582fe8aa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.299+0000 7fb22363c740 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5582fe45e670 0x5582fe83c5d0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.299+0000 7fb22363c740 1 --2- 172.21.15.16:0/1406062074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582fe456490 0x5582fe83bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.299+0000 7fb22363c740 1 -- 172.21.15.16:0/1406062074 >> 172.21.15.16:0/1406062074 conn(0x5582fe473740 msgr2=0x5582fe827150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.299+0000 7fb1faff5640 0 client.18027 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.299+0000 7fb22363c740 1 -- 172.21.15.16:0/1406062074 shutdown_connections 2024-08-28T19:30:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.299+0000 7fb22363c740 1 -- 172.21.15.16:0/1406062074 wait complete. 2024-08-28T19:30:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.299+0000 7fb22363c740 20 client.18027 trim_cache size 0 max 16384 2024-08-28T19:30:47.302 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:30:47.302 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:30:47.303 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "217eb860-c45a-4efe-97c5-379fc45c3873", 2024-08-28T19:30:47.303 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:30:47.303 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:30:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 -- 172.21.15.16:0/465154578 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 msgr2=0x7fb2000b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 0x7fb2000b71b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb210005fd0 tx=0x7fb210009600 comp rx=0 tx=0).stop 2024-08-28T19:30:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 -- 172.21.15.16:0/465154578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb218108170 msgr2=0x7fb21819a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb218108170 0x7fb21819a7f0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb20c006010 tx=0x7fb20c004060 comp rx=0 tx=0).stop 2024-08-28T19:30:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 -- 172.21.15.16:0/465154578 shutdown_connections 2024-08-28T19:30:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb2000b4ca0 0x7fb2000b71b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb218108ac0 0x7fb21819fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb218108170 0x7fb21819a7f0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 --2- 172.21.15.16:0/465154578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb218102880 0x7fb21819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 -- 172.21.15.16:0/465154578 >> 172.21.15.16:0/465154578 conn(0x7fb218079500 msgr2=0x7fb21806bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 -- 172.21.15.16:0/465154578 shutdown_connections 2024-08-28T19:30:47.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:47.300+0000 7fb21d0c7640 1 -- 172.21.15.16:0/465154578 wait complete. 2024-08-28T19:30:47.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.18 MB/sec execute 387 sec latency 2051.441 ms 2024-08-28T19:30:48.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:48.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:30:48.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:48.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:48.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:48.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:48.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:48.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:47 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:48.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:30:48.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:48.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:48.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:48.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:48.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:48.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:47 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:30:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:47 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.292 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:30:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:30:48.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:48 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:48.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:48 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:48.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:48 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:48 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:48.950 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:48 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.950 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:48 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.950 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:48 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:30:48.950 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:48 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:48.950 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:48 smithi016 ceph-mon[18556]: pgmap v638: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 120 op/s 2024-08-28T19:30:48.950 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:48.950 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:48 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:48.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.17 MB/sec execute 388 sec latency 3051.498 ms 2024-08-28T19:30:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:48 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:48 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:48 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:48 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:48 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:48 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:48 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:30:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:48 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:48 smithi028 ceph-mon[21769]: pgmap v638: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 120 op/s 2024-08-28T19:30:49.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:49.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:48 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:49.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.044+0000 7f329d1bd640 1 -- 172.21.15.16:0/610396227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32981050a0 msgr2=0x7f329810d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:49.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.044+0000 7f329d1bd640 1 --2- 172.21.15.16:0/610396227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32981050a0 0x7f329810d5f0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f328c009900 tx=0x7f328c02a3f0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.044+0000 7f329d1bd640 1 -- 172.21.15.16:0/610396227 shutdown_connections 2024-08-28T19:30:49.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.044+0000 7f329d1bd640 1 --2- 172.21.15.16:0/610396227 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f329810dbb0 0x7f32981100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.044+0000 7f329d1bd640 1 --2- 172.21.15.16:0/610396227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32981050a0 0x7f329810d5f0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.044+0000 7f329d1bd640 1 --2- 172.21.15.16:0/610396227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3298104750 0x7f3298104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.044+0000 7f329d1bd640 1 -- 172.21.15.16:0/610396227 >> 172.21.15.16:0/610396227 conn(0x7f32980794d0 msgr2=0x7f32980798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:49.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.044+0000 7f329d1bd640 1 -- 172.21.15.16:0/610396227 shutdown_connections 2024-08-28T19:30:49.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.044+0000 7f329d1bd640 1 -- 172.21.15.16:0/610396227 wait complete. 2024-08-28T19:30:49.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.045+0000 7f329d1bd640 1 Processor -- start 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.045+0000 7f329d1bd640 1 -- start start 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.045+0000 7f329d1bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3298104750 0x7f329819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.045+0000 7f329d1bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32981050a0 0x7f329819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.045+0000 7f329d1bd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f329810dbb0 0x7f329819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.045+0000 7f329d1bd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f32981137f0 con 0x7f3298104750 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.045+0000 7f329d1bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3298113670 con 0x7f32981050a0 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.045+0000 7f329d1bd640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3298113970 con 0x7f329810dbb0 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f3296d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3298104750 0x7f329819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f3296d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3298104750 0x7f329819a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40340/0 (socket says 172.21.15.16:40340) 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f3296d9d640 1 -- 172.21.15.16:0/933914464 learned_addr learned my addr 172.21.15.16:0/933914464 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f329659c640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32981050a0 0x7f329819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f329759e640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f329810dbb0 0x7f329819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f3296d9d640 1 -- 172.21.15.16:0/933914464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f329810dbb0 msgr2=0x7f329819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f3296d9d640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f329810dbb0 0x7f329819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f3296d9d640 1 -- 172.21.15.16:0/933914464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32981050a0 msgr2=0x7f329819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f3296d9d640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32981050a0 0x7f329819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f3296d9d640 1 -- 172.21.15.16:0/933914464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32981a01d0 con 0x7f3298104750 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f3296d9d640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3298104750 0x7f329819a260 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f328800d950 tx=0x7f328800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f327ffff640 1 -- 172.21.15.16:0/933914464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3288017020 con 0x7f3298104750 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f329d1bd640 1 -- 172.21.15.16:0/933914464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32981a04c0 con 0x7f3298104750 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f327ffff640 1 -- 172.21.15.16:0/933914464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3288009c30 con 0x7f3298104750 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f329d1bd640 1 -- 172.21.15.16:0/933914464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32981a6c20 con 0x7f3298104750 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.046+0000 7f327ffff640 1 -- 172.21.15.16:0/933914464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328800f6b0 con 0x7f3298104750 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.049+0000 7f32a3732740 1 Processor -- start 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.049+0000 7f32a3732740 1 -- start start 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.049+0000 7f327ffff640 1 -- 172.21.15.16:0/933914464 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f328801a020 con 0x7f3298104750 2024-08-28T19:30:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.049+0000 7f32a3732740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d9397070 0x5654d973c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.049+0000 7f32a3732740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d973c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.049+0000 7f32a3732740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d9376390 0x5654d973cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.049+0000 7f32a3732740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5654d96b3630 con 0x5654d936e1b0 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.049+0000 7f32a3732740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5654d96a2bc0 con 0x5654d9376390 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.049+0000 7f32a3732740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5654d96a6b70 con 0x5654d9397070 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f3296d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d973c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f3296d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d973c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40356/0 (socket says 172.21.15.16:40356) 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f3296d9d640 1 -- 172.21.15.16:0/2511503071 learned_addr learned my addr 172.21.15.16:0/2511503071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f329659c640 1 --2- 172.21.15.16:0/2511503071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d9376390 0x5654d973cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f329759e640 1 --2- 172.21.15.16:0/2511503071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d9397070 0x5654d973c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f3296d9d640 1 -- 172.21.15.16:0/2511503071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d9397070 msgr2=0x5654d973c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:49.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f3296d9d640 1 --2- 172.21.15.16:0/2511503071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d9397070 0x5654d973c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f3296d9d640 1 -- 172.21.15.16:0/2511503071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d9376390 msgr2=0x5654d973cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f3296d9d640 1 --2- 172.21.15.16:0/2511503071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d9376390 0x5654d973cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f3296d9d640 1 -- 172.21.15.16:0/2511503071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654d9402650 con 0x5654d936e1b0 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f329659c640 1 --2- 172.21.15.16:0/2511503071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d9376390 0x5654d973cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f327ffff640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f32800b4ca0 0x7f32800b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f3296d9d640 1 --2- 172.21.15.16:0/2511503071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d973c800 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f32880e27e0 tx=0x7f32880ebba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3c7178d78d8cd8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f329759e640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f32800b4ca0 0x7f32800b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f327d7fa640 1 -- 172.21.15.16:0/2511503071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32880a7700 con 0x5654d936e1b0 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f327ffff640 1 -- 172.21.15.16:0/933914464 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7f32880e5520 con 0x7f3298104750 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.050+0000 7f327d7fa640 1 -- 172.21.15.16:0/2511503071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32880ab350 con 0x5654d936e1b0 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.051+0000 7f327d7fa640 1 -- 172.21.15.16:0/2511503071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32880aafb0 con 0x5654d936e1b0 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.051+0000 7f32a3732740 1 -- 172.21.15.16:0/2511503071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 msgr2=0x5654d973c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.051+0000 7f329759e640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f32800b4ca0 0x7f32800b71b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3290005fd0 tx=0x7f3290009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:49.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.051+0000 7f32a3732740 1 --2- 172.21.15.16:0/2511503071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d973c800 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f32880e27e0 tx=0x7f32880ebba0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.051+0000 7f32a3732740 1 -- 172.21.15.16:0/2511503071 shutdown_connections 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.051+0000 7f32a3732740 1 --2- 172.21.15.16:0/2511503071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d9376390 0x5654d973cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.051+0000 7f32a3732740 1 --2- 172.21.15.16:0/2511503071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d973c800 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.051+0000 7f32a3732740 1 --2- 172.21.15.16:0/2511503071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d9397070 0x5654d973c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.051+0000 7f32a3732740 1 -- 172.21.15.16:0/2511503071 >> 172.21.15.16:0/2511503071 conn(0x5654d938b460 msgr2=0x5654d973a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f32a3732740 1 -- 172.21.15.16:0/2511503071 shutdown_connections 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f32a3732740 1 -- 172.21.15.16:0/2511503071 wait complete. 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f32a3732740 1 -- start start 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f32a3732740 1 Processor -- start 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f32a3732740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 0x5654d974bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f32a3732740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d974c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f32a3732740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d9397070 0x5654d9757f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f32a3732740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5654d9505700 con 0x5654d9376390 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f32a3732740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5654d9629700 con 0x5654d936e1b0 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f32a3732740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5654d974ec50 con 0x5654d9397070 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f32a3732740 10 client.0 _ll_register_callbacks cb 0x5654d9658dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f329659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d974c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f329659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d974c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40364/0 (socket says 172.21.15.16:40364) 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f329659c640 1 -- 172.21.15.16:0/3437514001 learned_addr learned my addr 172.21.15.16:0/3437514001 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.052+0000 7f3296d9d640 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 0x5654d974bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f329759e640 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d9397070 0x5654d9757f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f329659c640 1 -- 172.21.15.16:0/3437514001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d9397070 msgr2=0x5654d9757f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f329659c640 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d9397070 0x5654d9757f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f329659c640 1 -- 172.21.15.16:0/3437514001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 msgr2=0x5654d974bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f329659c640 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 0x5654d974bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f329659c640 1 -- 172.21.15.16:0/3437514001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654d96628b0 con 0x5654d9376390 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f3296d9d640 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 0x5654d974bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f329659c640 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d974c2a0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f328c009900 tx=0x7f328c0314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7a5033c6692f51 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f327aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f327aff5640 1 -- 172.21.15.16:0/3437514001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c035040 con 0x5654d9376390 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f32a3732740 1 -- 172.21.15.16:0/3437514001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32880ab350 con 0x5654d9376390 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f327aff5640 1 -- 172.21.15.16:0/3437514001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7f328c031810 con 0x5654d9376390 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f327aff5640 1 -- 172.21.15.16:0/3437514001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f328c04f050 con 0x5654d9376390 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f32a3732740 10 client.18039 fetch_fsmap learned FSMap version 54 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.053+0000 7f32a3732740 1 -- 172.21.15.16:0/3437514001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5654d975a9c0 con 0x5654d9376390 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.054+0000 7f327aff5640 1 -- 172.21.15.16:0/3437514001 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32880ab350 con 0x5654d9376390 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.054+0000 7f327aff5640 1 -- 172.21.15.16:0/3437514001 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 54) ==== 8374+0+0 (secure 0 0 0) 0x7f328c004750 con 0x5654d9376390 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.054+0000 7f32a3732740 10 client.18039 fetch_fsmap finished waiting for FSMap version 54 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.054+0000 7f32a3732740 10 client.18039 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.054+0000 7f32a3732740 20 client.18039 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.054+0000 7f32a3732740 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5654d93559f0 0x5654d975e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.054+0000 7f32a3732740 4 client.18039 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.054+0000 7f32a3732740 1 -- 172.21.15.16:0/3437514001 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654d96b5580 con 0x5654d93559f0 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.054+0000 7f3296d9d640 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5654d93559f0 0x5654d975e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.055+0000 7f327aff5640 10 client.18039 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.055+0000 7f3296d9d640 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5654d93559f0 0x5654d975e230 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be101e98b053f360 server_cookie=d5ef1ec0b3df12b9 in_seq=0 out_seq=0 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.055+0000 7f327aff5640 1 -- 172.21.15.16:0/3437514001 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5654d96b5580 con 0x5654d93559f0 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.055+0000 7f327aff5640 10 client.18039 handle_command_reply: tid=0 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.055+0000 7f32a3732740 1 client.18039 shutdown 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f32a3732740 1 -- 172.21.15.16:0/3437514001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 msgr2=0x5654d974c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f32a3732740 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d974c2a0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f328c009900 tx=0x7f328c0314a0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f32a3732740 1 -- 172.21.15.16:0/3437514001 shutdown_connections 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f32a3732740 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5654d93559f0 0x5654d975e230 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f32a3732740 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d9397070 0x5654d9757f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f32a3732740 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d974c2a0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f327aff5640 0 client.18039 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:49.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f32a3732740 1 --2- 172.21.15.16:0/3437514001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 0x5654d974bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f32a3732740 1 -- 172.21.15.16:0/3437514001 >> 172.21.15.16:0/3437514001 conn(0x5654d938b460 msgr2=0x5654d973f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:49.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f32a3732740 1 -- 172.21.15.16:0/3437514001 shutdown_connections 2024-08-28T19:30:49.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f32a3732740 1 -- 172.21.15.16:0/3437514001 wait complete. 2024-08-28T19:30:49.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.056+0000 7f32a3732740 20 client.18039 trim_cache size 0 max 16384 2024-08-28T19:30:49.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.074+0000 7f32a3732740 1 Processor -- start 2024-08-28T19:30:49.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.074+0000 7f32a3732740 1 -- start start 2024-08-28T19:30:49.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f32a3732740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 0x5654d974bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f32a3732740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 0x5654d973da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f32a3732740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d9739780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f32a3732740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f32981aeac0 con 0x5654d9376390 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f32a3732740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32981b0630 con 0x5654d936e1b0 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f32a3732740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f32981af1c0 con 0x5654d93559f0 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f329759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d9739780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f329759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d9739780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40374/0 (socket says 172.21.15.16:40374) 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f329759e640 1 -- 172.21.15.16:0/901258910 learned_addr learned my addr 172.21.15.16:0/901258910 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f3296d9d640 1 --2- 172.21.15.16:0/901258910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 0x5654d974bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f329659c640 1 --2- 172.21.15.16:0/901258910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 0x5654d973da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f329759e640 1 -- 172.21.15.16:0/901258910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 msgr2=0x5654d974bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f329759e640 1 --2- 172.21.15.16:0/901258910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 0x5654d974bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f329759e640 1 -- 172.21.15.16:0/901258910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 msgr2=0x5654d973da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f329759e640 1 --2- 172.21.15.16:0/901258910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 0x5654d973da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f329759e640 1 -- 172.21.15.16:0/901258910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654d974acc0 con 0x5654d9376390 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f3296d9d640 1 --2- 172.21.15.16:0/901258910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 0x5654d974bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f329659c640 1 --2- 172.21.15.16:0/901258910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 0x5654d973da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.075+0000 7f329759e640 1 --2- 172.21.15.16:0/901258910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d9739780 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f3290011830 tx=0x7f3290016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3df6a0d5218163 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.076+0000 7f327cff9640 1 -- 172.21.15.16:0/901258910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329001d430 con 0x5654d9376390 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.076+0000 7f327cff9640 1 -- 172.21.15.16:0/901258910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f329001da80 con 0x5654d9376390 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.076+0000 7f327cff9640 1 -- 172.21.15.16:0/901258910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3290020ac0 con 0x5654d9376390 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.077+0000 7f32a3732740 1 -- 172.21.15.16:0/901258910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 msgr2=0x5654d9739780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.077+0000 7f32a3732740 1 --2- 172.21.15.16:0/901258910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d9739780 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f3290011830 tx=0x7f3290016000 comp rx=0 tx=0).stop 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.077+0000 7f32a3732740 1 -- 172.21.15.16:0/901258910 shutdown_connections 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.077+0000 7f32a3732740 1 --2- 172.21.15.16:0/901258910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d9376390 0x5654d9739780 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.077+0000 7f32a3732740 1 --2- 172.21.15.16:0/901258910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d936e1b0 0x5654d973da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.077+0000 7f32a3732740 1 --2- 172.21.15.16:0/901258910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 0x5654d974bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.077+0000 7f32a3732740 1 -- 172.21.15.16:0/901258910 >> 172.21.15.16:0/901258910 conn(0x5654d938b460 msgr2=0x5654d973d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.077+0000 7f32a3732740 1 -- 172.21.15.16:0/901258910 shutdown_connections 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.077+0000 7f32a3732740 1 -- 172.21.15.16:0/901258910 wait complete. 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.077+0000 7f32a3732740 1 -- start start 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.077+0000 7f32a3732740 1 Processor -- start 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f32a3732740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 0x5654d97501e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f32a3732740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d9750720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f32a3732740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d9376390 0x5654d97c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f32a3732740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f32981aff60 con 0x5654d936e1b0 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f32a3732740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32981af890 con 0x5654d9376390 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f32a3732740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f32981acac0 con 0x5654d93559f0 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f329659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d9750720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f32a3732740 10 client.0 _ll_register_callbacks cb 0x7f328c032c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:30:49.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f329659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d9750720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40380/0 (socket says 172.21.15.16:40380) 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f329659c640 1 -- 172.21.15.16:0/1107331122 learned_addr learned my addr 172.21.15.16:0/1107331122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f329759e640 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d9376390 0x5654d97c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f3296d9d640 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 0x5654d97501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f329659c640 1 -- 172.21.15.16:0/1107331122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 msgr2=0x5654d97501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f329659c640 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 0x5654d97501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f329659c640 1 -- 172.21.15.16:0/1107331122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d9376390 msgr2=0x5654d97c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f329659c640 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d9376390 0x5654d97c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f329659c640 1 -- 172.21.15.16:0/1107331122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654d9750fa0 con 0x5654d936e1b0 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f3296d9d640 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 0x5654d97501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f329659c640 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d9750720 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f328c009900 tx=0x7f328c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f563e6630dcdeed server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f327aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f327aff5640 1 -- 172.21.15.16:0/1107331122 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c057030 con 0x5654d936e1b0 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f327aff5640 1 -- 172.21.15.16:0/1107331122 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9490+0+0 (secure 0 0 0) 0x7f328c04c040 con 0x5654d936e1b0 2024-08-28T19:30:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.078+0000 7f32a3732740 1 -- 172.21.15.16:0/1107331122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f329001da80 con 0x5654d936e1b0 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f327aff5640 1 -- 172.21.15.16:0/1107331122 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f328c061050 con 0x5654d936e1b0 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f32a3732740 10 client.18045 fetch_fsmap learned FSMap version 54 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f32a3732740 1 -- 172.21.15.16:0/1107331122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5654d97c9190 con 0x5654d936e1b0 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f327aff5640 1 -- 172.21.15.16:0/1107331122 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f329001da80 con 0x5654d936e1b0 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f327aff5640 1 -- 172.21.15.16:0/1107331122 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 54) ==== 8374+0+0 (secure 0 0 0) 0x7f328c046470 con 0x5654d936e1b0 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f32a3732740 10 client.18045 fetch_fsmap finished waiting for FSMap version 54 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f32a3732740 10 client.18045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f32a3732740 20 client.18045 populate_metadata read hostname 'smithi016' 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f32a3732740 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5654d9397070 0x5654d97cb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f32a3732740 4 client.18045 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f32a3732740 1 -- 172.21.15.16:0/1107331122 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5654d96b1020 con 0x5654d9397070 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.079+0000 7f3296d9d640 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5654d9397070 0x5654d97cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f327aff5640 10 client.18045 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f3296d9d640 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5654d9397070 0x5654d97cb940 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8689d100e9cc42d6 server_cookie=218e71ac9758c715 in_seq=0 out_seq=0 2024-08-28T19:30:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f327aff5640 1 -- 172.21.15.16:0/1107331122 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4262 (crc 0 0 0) 0x5654d96b1020 con 0x5654d9397070 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f327aff5640 10 client.18045 handle_command_reply: tid=0 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f32a3732740 1 client.18045 shutdown 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f32a3732740 1 -- 172.21.15.16:0/1107331122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 msgr2=0x5654d9750720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f32a3732740 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d9750720 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f328c009900 tx=0x7f328c0048f0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f32a3732740 1 -- 172.21.15.16:0/1107331122 shutdown_connections 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f32a3732740 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654d9376390 0x5654d97c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f32a3732740 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5654d936e1b0 0x5654d9750720 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f32a3732740 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5654d9397070 0x5654d97cb940 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f32a3732740 1 --2- 172.21.15.16:0/1107331122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5654d93559f0 0x5654d97501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f327aff5640 0 client.18045 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.080+0000 7f32a3732740 1 -- 172.21.15.16:0/1107331122 >> 172.21.15.16:0/1107331122 conn(0x5654d938b460 msgr2=0x5654d973dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f32a3732740 1 -- 172.21.15.16:0/1107331122 shutdown_connections 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f32a3732740 1 -- 172.21.15.16:0/1107331122 wait complete. 2024-08-28T19:30:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f32a3732740 20 client.18045 trim_cache size 0 max 16384 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "217eb860-c45a-4efe-97c5-379fc45c3873", 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:30:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:30:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:30:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:30:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:30:49.104 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:30:49.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f329d1bd640 1 -- 172.21.15.16:0/933914464 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f32800b4ca0 msgr2=0x7f32800b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:49.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f329d1bd640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f32800b4ca0 0x7f32800b71b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3290005fd0 tx=0x7f3290009600 comp rx=0 tx=0).stop 2024-08-28T19:30:49.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f329d1bd640 1 -- 172.21.15.16:0/933914464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3298104750 msgr2=0x7f329819a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:30:49.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f329d1bd640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3298104750 0x7f329819a260 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f328800d950 tx=0x7f328800de10 comp rx=0 tx=0).stop 2024-08-28T19:30:49.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f329d1bd640 1 -- 172.21.15.16:0/933914464 shutdown_connections 2024-08-28T19:30:49.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f329d1bd640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f329810dbb0 0x7f329819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f329d1bd640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f32800b4ca0 0x7f32800b71b0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f329d1bd640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32981050a0 0x7f329819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f329d1bd640 1 --2- 172.21.15.16:0/933914464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3298104750 0x7f329819a260 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:30:49.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.081+0000 7f329d1bd640 1 -- 172.21.15.16:0/933914464 >> 172.21.15.16:0/933914464 conn(0x7f32980794d0 msgr2=0x7f3298100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:30:49.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.082+0000 7f329d1bd640 1 -- 172.21.15.16:0/933914464 shutdown_connections 2024-08-28T19:30:49.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:30:49.082+0000 7f329d1bd640 1 -- 172.21.15.16:0/933914464 wait complete. 2024-08-28T19:30:49.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:48 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:49.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:48 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:48 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:48 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:48 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:48 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:48 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:30:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:48 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:30:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:48 smithi115 ceph-mon[22184]: pgmap v638: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 120 op/s 2024-08-28T19:30:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:48 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:49.733 INFO:tasks.cephfs.filesystem:scrub status for tag:217eb860-c45a-4efe-97c5-379fc45c3873 - {'path': '/', 'tag': '217eb860-c45a-4efe-97c5-379fc45c3873', 'options': 'recursive,force'} 2024-08-28T19:30:49.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.16 MB/sec execute 389 sec latency 4051.594 ms 2024-08-28T19:30:50.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:30:50.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:30:50.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:50.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:30:50.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:30:50.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:50.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:30:50.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:30:50.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:30:50.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.15 MB/sec execute 390 sec latency 5051.697 ms 2024-08-28T19:30:51.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:50 smithi016 ceph-mon[18556]: pgmap v639: 129 pgs: 129 active+clean; 307 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 999 KiB/s wr, 119 op/s 2024-08-28T19:30:51.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:50 smithi028 ceph-mon[21769]: pgmap v639: 129 pgs: 129 active+clean; 307 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 999 KiB/s wr, 119 op/s 2024-08-28T19:30:51.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:50 smithi115 ceph-mon[22184]: pgmap v639: 129 pgs: 129 active+clean; 307 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 999 KiB/s wr, 119 op/s 2024-08-28T19:30:51.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.14 MB/sec execute 391 sec latency 6051.751 ms 2024-08-28T19:30:52.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.13 MB/sec execute 392 sec latency 7051.861 ms 2024-08-28T19:30:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:52 smithi016 ceph-mon[18556]: pgmap v640: 129 pgs: 129 active+clean; 307 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 855 KiB/s wr, 114 op/s 2024-08-28T19:30:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:52 smithi028 ceph-mon[21769]: pgmap v640: 129 pgs: 129 active+clean; 307 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 855 KiB/s wr, 114 op/s 2024-08-28T19:30:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:52 smithi115 ceph-mon[22184]: pgmap v640: 129 pgs: 129 active+clean; 307 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 855 KiB/s wr, 114 op/s 2024-08-28T19:30:53.966 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.12 MB/sec execute 393 sec latency 8051.918 ms 2024-08-28T19:30:54.967 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.11 MB/sec execute 394 sec latency 9052.042 ms 2024-08-28T19:30:55.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:54 smithi016 ceph-mon[18556]: pgmap v641: 129 pgs: 129 active+clean; 312 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 982 KiB/s wr, 87 op/s 2024-08-28T19:30:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:54 smithi028 ceph-mon[21769]: pgmap v641: 129 pgs: 129 active+clean; 312 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 982 KiB/s wr, 87 op/s 2024-08-28T19:30:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:54 smithi115 ceph-mon[22184]: pgmap v641: 129 pgs: 129 active+clean; 312 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 982 KiB/s wr, 87 op/s 2024-08-28T19:30:55.967 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.10 MB/sec execute 395 sec latency 10052.131 ms 2024-08-28T19:30:56.967 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.09 MB/sec execute 396 sec latency 11052.238 ms 2024-08-28T19:30:57.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:56 smithi016 ceph-mon[18556]: pgmap v642: 129 pgs: 129 active+clean; 316 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-08-28T19:30:57.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:56 smithi028 ceph-mon[21769]: pgmap v642: 129 pgs: 129 active+clean; 316 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-08-28T19:30:57.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:56 smithi115 ceph-mon[22184]: pgmap v642: 129 pgs: 129 active+clean; 316 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-08-28T19:30:57.967 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.08 MB/sec execute 397 sec latency 12052.361 ms 2024-08-28T19:30:58.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:58 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:58.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:58 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:58.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:58 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:30:58.967 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.07 MB/sec execute 398 sec latency 13052.459 ms 2024-08-28T19:30:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:30:59 smithi028 ceph-mon[21769]: pgmap v643: 129 pgs: 129 active+clean; 317 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-08-28T19:30:59.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:30:59 smithi115 ceph-mon[22184]: pgmap v643: 129 pgs: 129 active+clean; 317 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-08-28T19:30:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:30:59 smithi016 ceph-mon[18556]: pgmap v643: 129 pgs: 129 active+clean; 317 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-08-28T19:30:59.967 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.06 MB/sec execute 399 sec latency 14052.534 ms 2024-08-28T19:31:00.967 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.04 MB/sec execute 400 sec latency 15052.646 ms 2024-08-28T19:31:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:01 smithi028 ceph-mon[21769]: pgmap v644: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-08-28T19:31:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:01 smithi115 ceph-mon[22184]: pgmap v644: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-08-28T19:31:01.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:01 smithi016 ceph-mon[18556]: pgmap v644: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-08-28T19:31:01.967 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.03 MB/sec execute 401 sec latency 16052.717 ms 2024-08-28T19:31:02.967 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.02 MB/sec execute 402 sec latency 17052.816 ms 2024-08-28T19:31:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:03 smithi028 ceph-mon[21769]: pgmap v645: 129 pgs: 129 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.6 MiB/s wr, 91 op/s 2024-08-28T19:31:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:03 smithi115 ceph-mon[22184]: pgmap v645: 129 pgs: 129 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.6 MiB/s wr, 91 op/s 2024-08-28T19:31:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:03 smithi016 ceph-mon[18556]: pgmap v645: 129 pgs: 129 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.6 MiB/s wr, 91 op/s 2024-08-28T19:31:03.967 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.01 MB/sec execute 403 sec latency 18052.943 ms 2024-08-28T19:31:04.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 4.00 MB/sec execute 404 sec latency 19053.052 ms 2024-08-28T19:31:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:05 smithi028 ceph-mon[21769]: pgmap v646: 129 pgs: 129 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-08-28T19:31:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:05 smithi028 ceph-mon[21769]: osdmap e91: 12 total, 12 up, 12 in 2024-08-28T19:31:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:05 smithi115 ceph-mon[22184]: pgmap v646: 129 pgs: 129 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-08-28T19:31:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:05 smithi115 ceph-mon[22184]: osdmap e91: 12 total, 12 up, 12 in 2024-08-28T19:31:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:05 smithi016 ceph-mon[18556]: pgmap v646: 129 pgs: 129 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-08-28T19:31:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:05 smithi016 ceph-mon[18556]: osdmap e91: 12 total, 12 up, 12 in 2024-08-28T19:31:05.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 3.99 MB/sec execute 405 sec latency 20053.131 ms 2024-08-28T19:31:06.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 3.99 MB/sec execute 406 sec latency 21053.203 ms 2024-08-28T19:31:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:07 smithi028 ceph-mon[21769]: pgmap v648: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 678 KiB/s wr, 68 op/s 2024-08-28T19:31:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:07 smithi115 ceph-mon[22184]: pgmap v648: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 678 KiB/s wr, 68 op/s 2024-08-28T19:31:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:07 smithi016 ceph-mon[18556]: pgmap v648: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 305 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 678 KiB/s wr, 68 op/s 2024-08-28T19:31:07.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 3.98 MB/sec execute 407 sec latency 22053.316 ms 2024-08-28T19:31:08.708 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:31:08.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 3.97 MB/sec execute 408 sec latency 23053.350 ms 2024-08-28T19:31:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:09 smithi028 ceph-mon[21769]: pgmap v649: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 158 KiB/s wr, 50 op/s 2024-08-28T19:31:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:09 smithi115 ceph-mon[22184]: pgmap v649: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 158 KiB/s wr, 50 op/s 2024-08-28T19:31:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:09 smithi016 ceph-mon[18556]: pgmap v649: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 158 KiB/s wr, 50 op/s 2024-08-28T19:31:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.590+0000 7f9f9fff1640 1 -- 172.21.15.16:0/432924146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98109b90 msgr2=0x7f9f981103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.590+0000 7f9f9fff1640 1 --2- 172.21.15.16:0/432924146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98109b90 0x7f9f981103c0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009900 tx=0x7f9f9002a410 comp rx=0 tx=0).stop 2024-08-28T19:31:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.590+0000 7f9f9fff1640 1 -- 172.21.15.16:0/432924146 shutdown_connections 2024-08-28T19:31:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.590+0000 7f9f9fff1640 1 --2- 172.21.15.16:0/432924146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98109b90 0x7f9f981103c0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.590+0000 7f9f9fff1640 1 --2- 172.21.15.16:0/432924146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f98109240 0x7f9f98109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.590+0000 7f9f9fff1640 1 --2- 172.21.15.16:0/432924146 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f98103900 0x7f9f98103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.590+0000 7f9f9fff1640 1 -- 172.21.15.16:0/432924146 >> 172.21.15.16:0/432924146 conn(0x7f9f9807b5f0 msgr2=0x7f9f9807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.590+0000 7f9f9fff1640 1 -- 172.21.15.16:0/432924146 shutdown_connections 2024-08-28T19:31:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.590+0000 7f9f9fff1640 1 -- 172.21.15.16:0/432924146 wait complete. 2024-08-28T19:31:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.591+0000 7f9f9fff1640 1 Processor -- start 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.591+0000 7f9f9fff1640 1 -- start start 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.591+0000 7f9f9fff1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f98103900 0x7f9f9819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.591+0000 7f9f9fff1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f98109240 0x7f9f9819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.591+0000 7f9f9fff1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98109b90 0x7f9f9819fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.591+0000 7f9f9fff1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f98113850 con 0x7f9f98109b90 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.591+0000 7f9f9fff1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f981136d0 con 0x7f9f98103900 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.591+0000 7f9f9fff1640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9f981139d0 con 0x7f9f98109240 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.591+0000 7f9f9e58e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98109b90 0x7f9f9819fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.591+0000 7f9f9e58e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98109b90 0x7f9f9819fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49560/0 (socket says 172.21.15.16:49560) 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.591+0000 7f9f9e58e640 1 -- 172.21.15.16:0/3175135056 learned_addr learned my addr 172.21.15.16:0/3175135056 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f9d58c640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f98109240 0x7f9f9819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f9dd8d640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f98103900 0x7f9f9819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f9e58e640 1 -- 172.21.15.16:0/3175135056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f98109240 msgr2=0x7f9f9819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f9e58e640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f98109240 0x7f9f9819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f9e58e640 1 -- 172.21.15.16:0/3175135056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f98103900 msgr2=0x7f9f9819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f9e58e640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f98103900 0x7f9f9819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f9e58e640 1 -- 172.21.15.16:0/3175135056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f981a0190 con 0x7f9f98109b90 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f9dd8d640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f98103900 0x7f9f9819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f9e58e640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98109b90 0x7f9f9819fa60 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f9f90004770 tx=0x7f9f90004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f96ffd640 1 -- 172.21.15.16:0/3175135056 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9003f030 con 0x7f9f98109b90 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f9fff1640 1 -- 172.21.15.16:0/3175135056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f981a0420 con 0x7f9f98109b90 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f96ffd640 1 -- 172.21.15.16:0/3175135056 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f90003aa0 con 0x7f9f98109b90 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f96ffd640 1 -- 172.21.15.16:0/3175135056 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90035070 con 0x7f9f98109b90 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.592+0000 7f9f9fff1640 1 -- 172.21.15.16:0/3175135056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f981a6be0 con 0x7f9f98109b90 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.593+0000 7f9f9fff1640 1 -- 172.21.15.16:0/3175135056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f98104b80 con 0x7f9f98109b90 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.596+0000 7f9f96ffd640 1 -- 172.21.15.16:0/3175135056 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f9f90042030 con 0x7f9f98109b90 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.596+0000 7f9f96ffd640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9f800b4ca0 0x7f9f800b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.597+0000 7f9f9dd8d640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9f800b4ca0 0x7f9f800b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.597+0000 7f9f96ffd640 1 -- 172.21.15.16:0/3175135056 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9578+0+0 (secure 0 0 0) 0x7f9f90106360 con 0x7f9f98109b90 2024-08-28T19:31:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.597+0000 7f9f9dd8d640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9f800b4ca0 0x7f9f800b71b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9f88005fd0 tx=0x7f9f88009520 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.600+0000 7f9f96ffd640 1 -- 172.21.15.16:0/3175135056 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f9f900ccd30 con 0x7f9f98109b90 2024-08-28T19:31:09.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.822+0000 7f9f9fff1640 1 -- 172.21.15.16:0/3175135056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f98103d10 con 0x7f9f98109b90 2024-08-28T19:31:09.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.822+0000 7f9f96ffd640 1 -- 172.21.15.16:0/3175135056 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11977 (secure 0 0 0) 0x7f9f98103d10 con 0x7f9f98109b90 2024-08-28T19:31:09.825 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:31:09.825 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":55,"btime":"2024-08-28T19:30:56:995905+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:30:56.593478+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:31:09.825 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 55 2024-08-28T19:31:09.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 -- 172.21.15.16:0/3175135056 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9f800b4ca0 msgr2=0x7f9f800b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:09.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9f800b4ca0 0x7f9f800b71b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9f88005fd0 tx=0x7f9f88009520 comp rx=0 tx=0).stop 2024-08-28T19:31:09.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 -- 172.21.15.16:0/3175135056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98109b90 msgr2=0x7f9f9819fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:09.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98109b90 0x7f9f9819fa60 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f9f90004770 tx=0x7f9f90004270 comp rx=0 tx=0).stop 2024-08-28T19:31:09.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 -- 172.21.15.16:0/3175135056 shutdown_connections 2024-08-28T19:31:09.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98109b90 0x7f9f9819fa60 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:09.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f98109240 0x7f9f9819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:09.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9f800b4ca0 0x7f9f800b71b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:09.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 --2- 172.21.15.16:0/3175135056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f98103900 0x7f9f9819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:09.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 -- 172.21.15.16:0/3175135056 >> 172.21.15.16:0/3175135056 conn(0x7f9f9807b5f0 msgr2=0x7f9f98105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:09.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 -- 172.21.15.16:0/3175135056 shutdown_connections 2024-08-28T19:31:09.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:09.825+0000 7f9f9fff1640 1 -- 172.21.15.16:0/3175135056 wait complete. 2024-08-28T19:31:09.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 3.96 MB/sec execute 409 sec latency 24053.409 ms 2024-08-28T19:31:10.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:10 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3175135056' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:10.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:10 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3175135056' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:10.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:10 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3175135056' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:10.837 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id c46102d0 --release --await' 2024-08-28T19:31:10.837 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id c46102d0 --release --await 2024-08-28T19:31:10.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434461 3.95 MB/sec execute 410 sec latency 25053.454 ms 2024-08-28T19:31:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:11 smithi016 ceph-mon[18556]: pgmap v650: 129 pgs: 129 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 106 KiB/s wr, 41 op/s 2024-08-28T19:31:11.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:11 smithi016 ceph-mon[18556]: osdmap e92: 12 total, 12 up, 12 in 2024-08-28T19:31:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.524+0000 7f4b5edd1640 1 -- 172.21.15.16:0/3112366618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b58109240 msgr2=0x7f4b58109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.524+0000 7f4b5edd1640 1 --2- 172.21.15.16:0/3112366618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b58109240 0x7f4b58109650 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009900 tx=0x7f4b4402a410 comp rx=0 tx=0).stop 2024-08-28T19:31:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.524+0000 7f4b5edd1640 1 -- 172.21.15.16:0/3112366618 shutdown_connections 2024-08-28T19:31:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.524+0000 7f4b5edd1640 1 --2- 172.21.15.16:0/3112366618 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b58109b90 0x7f4b581103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.524+0000 7f4b5edd1640 1 --2- 172.21.15.16:0/3112366618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b58109240 0x7f4b58109650 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.524+0000 7f4b5edd1640 1 --2- 172.21.15.16:0/3112366618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58103900 0x7f4b58103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.524+0000 7f4b5edd1640 1 -- 172.21.15.16:0/3112366618 >> 172.21.15.16:0/3112366618 conn(0x7f4b5807b5f0 msgr2=0x7f4b5807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.524+0000 7f4b5edd1640 1 -- 172.21.15.16:0/3112366618 shutdown_connections 2024-08-28T19:31:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.524+0000 7f4b5edd1640 1 -- 172.21.15.16:0/3112366618 wait complete. 2024-08-28T19:31:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.525+0000 7f4b5edd1640 1 Processor -- start 2024-08-28T19:31:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.525+0000 7f4b5edd1640 1 -- start start 2024-08-28T19:31:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.525+0000 7f4b5edd1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58103900 0x7f4b5819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.525+0000 7f4b5edd1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b58109240 0x7f4b5819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.525+0000 7f4b5edd1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b58109b90 0x7f4b5819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.525+0000 7f4b5edd1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b58113850 con 0x7f4b58109240 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.525+0000 7f4b5edd1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b581136d0 con 0x7f4b58103900 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.525+0000 7f4b5edd1640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4b581139d0 con 0x7f4b58109b90 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.525+0000 7f4b57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b58109240 0x7f4b5819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.525+0000 7f4b57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b58109240 0x7f4b5819a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49576/0 (socket says 172.21.15.16:49576) 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.525+0000 7f4b57fff640 1 -- 172.21.15.16:0/1346263256 learned_addr learned my addr 172.21.15.16:0/1346263256 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b5cb6d640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58103900 0x7f4b5819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b57fff640 1 -- 172.21.15.16:0/1346263256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b58109b90 msgr2=0x7f4b5819fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b57fff640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b58109b90 0x7f4b5819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b57fff640 1 -- 172.21.15.16:0/1346263256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58103900 msgr2=0x7f4b5819a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b57fff640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58103900 0x7f4b5819a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b57fff640 1 -- 172.21.15.16:0/1346263256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b581a02b0 con 0x7f4b58109240 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b57fff640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b58109240 0x7f4b5819a8e0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f4b4402a930 tx=0x7f4b44004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b55ffb640 1 -- 172.21.15.16:0/1346263256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4403f030 con 0x7f4b58109240 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b55ffb640 1 -- 172.21.15.16:0/1346263256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b44003aa0 con 0x7f4b58109240 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b5edd1640 1 -- 172.21.15.16:0/1346263256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b581a0540 con 0x7f4b58109240 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b55ffb640 1 -- 172.21.15.16:0/1346263256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44035070 con 0x7f4b58109240 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.526+0000 7f4b5edd1640 1 -- 172.21.15.16:0/1346263256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b581a6d00 con 0x7f4b58109240 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b65346740 1 Processor -- start 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b65346740 1 -- start start 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b55ffb640 1 -- 172.21.15.16:0/1346263256 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4b44042030 con 0x7f4b58109240 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b65346740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05079520 0x55ac0541ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b65346740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05050660 0x55ac0541f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b65346740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05058840 0x55ac0541f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b65346740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ac05395a70 con 0x55ac05050660 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b65346740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ac05385000 con 0x55ac05079520 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b65346740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ac05388fb0 con 0x55ac05058840 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b5cb6d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05050660 0x55ac0541f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b5cb6d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05050660 0x55ac0541f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49592/0 (socket says 172.21.15.16:49592) 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b5cb6d640 1 -- 172.21.15.16:0/842750175 learned_addr learned my addr 172.21.15.16:0/842750175 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.529+0000 7f4b5d36e640 1 --2- 172.21.15.16:0/842750175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05079520 0x55ac0541ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b57fff640 1 --2- 172.21.15.16:0/842750175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05058840 0x55ac0541f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b5cb6d640 1 -- 172.21.15.16:0/842750175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05058840 msgr2=0x55ac0541f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b5cb6d640 1 --2- 172.21.15.16:0/842750175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05058840 0x55ac0541f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b5cb6d640 1 -- 172.21.15.16:0/842750175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05079520 msgr2=0x55ac0541ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b5cb6d640 1 --2- 172.21.15.16:0/842750175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05079520 0x55ac0541ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b5cb6d640 1 -- 172.21.15.16:0/842750175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac050e4b00 con 0x55ac05050660 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b5d36e640 1 --2- 172.21.15.16:0/842750175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05079520 0x55ac0541ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b55ffb640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b480b4ca0 0x7f4b480b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b5cb6d640 1 --2- 172.21.15.16:0/842750175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05050660 0x55ac0541f1c0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00dcb0 tx=0x7f4b4c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d2cd0d37def088 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b5d36e640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b480b4ca0 0x7f4b480b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b55ffb640 1 -- 172.21.15.16:0/1346263256 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f4b44105ea0 con 0x7f4b58109240 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b42ffd640 1 -- 172.21.15.16:0/842750175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c011440 con 0x55ac05050660 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b42ffd640 1 -- 172.21.15.16:0/842750175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b4c011a90 con 0x55ac05050660 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.530+0000 7f4b42ffd640 1 -- 172.21.15.16:0/842750175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c01ba10 con 0x55ac05050660 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.531+0000 7f4b65346740 1 -- 172.21.15.16:0/842750175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05050660 msgr2=0x55ac0541f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.531+0000 7f4b65346740 1 --2- 172.21.15.16:0/842750175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05050660 0x55ac0541f1c0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00dcb0 tx=0x7f4b4c00f730 comp rx=0 tx=0).stop 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.531+0000 7f4b5d36e640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b480b4ca0 0x7f4b480b71b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000ad60 tx=0x7f4b5000a2f0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.531+0000 7f4b65346740 1 -- 172.21.15.16:0/842750175 shutdown_connections 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.531+0000 7f4b65346740 1 --2- 172.21.15.16:0/842750175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05058840 0x55ac0541f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.531+0000 7f4b65346740 1 --2- 172.21.15.16:0/842750175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05050660 0x55ac0541f1c0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.531+0000 7f4b65346740 1 --2- 172.21.15.16:0/842750175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05079520 0x55ac0541ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.531+0000 7f4b65346740 1 -- 172.21.15.16:0/842750175 >> 172.21.15.16:0/842750175 conn(0x55ac0506d910 msgr2=0x55ac0541cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.531+0000 7f4b65346740 1 -- 172.21.15.16:0/842750175 shutdown_connections 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.531+0000 7f4b65346740 1 -- 172.21.15.16:0/842750175 wait complete. 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.531+0000 7f4b65346740 1 -- start start 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b65346740 1 Processor -- start 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b65346740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac0542a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b65346740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac0542ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b65346740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05079520 0x55ac05436900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b65346740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ac0542d460 con 0x55ac05058840 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b65346740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ac05346520 con 0x55ac05050660 2024-08-28T19:31:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b65346740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ac0542d5e0 con 0x55ac05079520 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b65346740 10 client.0 _ll_register_callbacks cb 0x55ac0537ec20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac0542ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac0542ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49596/0 (socket says 172.21.15.16:49596) 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b57fff640 1 -- 172.21.15.16:0/2575127307 learned_addr learned my addr 172.21.15.16:0/2575127307 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b5cb6d640 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac0542a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b5d36e640 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05079520 0x55ac05436900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b57fff640 1 -- 172.21.15.16:0/2575127307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05079520 msgr2=0x55ac05436900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b57fff640 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05079520 0x55ac05436900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b57fff640 1 -- 172.21.15.16:0/2575127307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 msgr2=0x55ac0542a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b57fff640 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac0542a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.532+0000 7f4b57fff640 1 -- 172.21.15.16:0/2575127307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac0515b960 con 0x55ac05058840 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b5d36e640 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05079520 0x55ac05436900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b57fff640 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac0542ad50 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f4b44031440 tx=0x7f4b44003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654b3012c15896f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b37fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b37fff640 1 -- 172.21.15.16:0/2575127307 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44049070 con 0x55ac05058840 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b37fff640 1 -- 172.21.15.16:0/2575127307 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f4b440c8450 con 0x55ac05058840 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b65346740 1 -- 172.21.15.16:0/2575127307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ac053249a0 con 0x55ac05058840 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b37fff640 1 -- 172.21.15.16:0/2575127307 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f4b440cbcd0 con 0x55ac05058840 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b65346740 10 client.18063 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b65346740 1 -- 172.21.15.16:0/2575127307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ac054395a0 con 0x55ac05058840 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b37fff640 1 -- 172.21.15.16:0/2575127307 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ac053249a0 con 0x55ac05058840 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b37fff640 1 -- 172.21.15.16:0/2575127307 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f4b440cc420 con 0x55ac05058840 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b65346740 10 client.18063 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b65346740 10 client.18063 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b65346740 20 client.18063 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.533+0000 7f4b65346740 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ac05037ea0 0x55ac0543c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.534+0000 7f4b65346740 4 client.18063 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:31:11.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.534+0000 7f4b65346740 1 -- 172.21.15.16:0/2575127307 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac053979c0 con 0x55ac05037ea0 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.534+0000 7f4b5cb6d640 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ac05037ea0 0x55ac0543c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.534+0000 7f4b37fff640 10 client.18063 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.534+0000 7f4b5cb6d640 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ac05037ea0 0x55ac0543c920 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8aaf2e2e3b473d18 server_cookie=12806a4a0f82ec05 in_seq=0 out_seq=0 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b37fff640 1 -- 172.21.15.16:0/2575127307 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ac053979c0 con 0x55ac05037ea0 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b37fff640 10 client.18063 handle_command_reply: tid=0 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b65346740 1 client.18063 shutdown 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b65346740 1 -- 172.21.15.16:0/2575127307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 msgr2=0x55ac0542ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b65346740 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac0542ad50 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f4b44031440 tx=0x7f4b44003f50 comp rx=0 tx=0).stop 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b65346740 1 -- 172.21.15.16:0/2575127307 shutdown_connections 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b65346740 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ac05037ea0 0x55ac0543c920 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b65346740 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05079520 0x55ac05436900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b65346740 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac0542ad50 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b37fff640 0 client.18063 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:11.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b65346740 1 --2- 172.21.15.16:0/2575127307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac0542a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b65346740 1 -- 172.21.15.16:0/2575127307 >> 172.21.15.16:0/2575127307 conn(0x55ac0506d910 msgr2=0x55ac05421e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:11.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b65346740 1 -- 172.21.15.16:0/2575127307 shutdown_connections 2024-08-28T19:31:11.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.535+0000 7f4b65346740 1 -- 172.21.15.16:0/2575127307 wait complete. 2024-08-28T19:31:11.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.536+0000 7f4b65346740 20 client.18063 trim_cache size 0 max 16384 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.554+0000 7f4b65346740 1 Processor -- start 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b65346740 1 -- start start 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b65346740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac05436800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b65346740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac05436d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b65346740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05079520 0x55ac05427800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b65346740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b581b0de0 con 0x55ac05058840 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b65346740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b581b2920 con 0x55ac05050660 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b65346740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4b581b14b0 con 0x55ac05079520 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac05436d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac05436d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49602/0 (socket says 172.21.15.16:49602) 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b5cb6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac05436800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b57fff640 1 -- 172.21.15.16:0/1019172901 learned_addr learned my addr 172.21.15.16:0/1019172901 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:11.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b5d36e640 1 --2- 172.21.15.16:0/1019172901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05079520 0x55ac05427800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b57fff640 1 -- 172.21.15.16:0/1019172901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05079520 msgr2=0x55ac05427800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:11.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b57fff640 1 --2- 172.21.15.16:0/1019172901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05079520 0x55ac05427800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b57fff640 1 -- 172.21.15.16:0/1019172901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 msgr2=0x55ac05436800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:11.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b57fff640 1 --2- 172.21.15.16:0/1019172901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac05436800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.555+0000 7f4b57fff640 1 -- 172.21.15.16:0/1019172901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac05429570 con 0x55ac05058840 2024-08-28T19:31:11.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.556+0000 7f4b57fff640 1 --2- 172.21.15.16:0/1019172901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac05436d40 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f4b440c9430 tx=0x7f4b440ef110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f006fe17760f41a3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:11.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.556+0000 7f4b427fc640 1 -- 172.21.15.16:0/1019172901 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44049070 con 0x55ac05058840 2024-08-28T19:31:11.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.556+0000 7f4b427fc640 1 -- 172.21.15.16:0/1019172901 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b440cc8c0 con 0x55ac05058840 2024-08-28T19:31:11.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.556+0000 7f4b427fc640 1 -- 172.21.15.16:0/1019172901 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b440c8450 con 0x55ac05058840 2024-08-28T19:31:11.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.557+0000 7f4b65346740 1 -- 172.21.15.16:0/1019172901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 msgr2=0x55ac05436d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:11.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.557+0000 7f4b65346740 1 --2- 172.21.15.16:0/1019172901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac05436d40 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f4b440c9430 tx=0x7f4b440ef110 comp rx=0 tx=0).stop 2024-08-28T19:31:11.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.557+0000 7f4b65346740 1 -- 172.21.15.16:0/1019172901 shutdown_connections 2024-08-28T19:31:11.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.557+0000 7f4b65346740 1 --2- 172.21.15.16:0/1019172901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05079520 0x55ac05427800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.557+0000 7f4b65346740 1 --2- 172.21.15.16:0/1019172901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05058840 0x55ac05436d40 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.557+0000 7f4b65346740 1 --2- 172.21.15.16:0/1019172901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac05436800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.557+0000 7f4b65346740 1 -- 172.21.15.16:0/1019172901 >> 172.21.15.16:0/1019172901 conn(0x55ac0506d910 msgr2=0x55ac054257c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.557+0000 7f4b65346740 1 -- 172.21.15.16:0/1019172901 shutdown_connections 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.557+0000 7f4b65346740 1 -- 172.21.15.16:0/1019172901 wait complete. 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.557+0000 7f4b65346740 1 -- start start 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.557+0000 7f4b65346740 1 Processor -- start 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b65346740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac0542f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b65346740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05058840 0x55ac0542fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b65346740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05079520 0x55ac054303a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b65346740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b581b2250 con 0x55ac05079520 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b65346740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b581b1b80 con 0x55ac05050660 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b65346740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4b581aede0 con 0x55ac05058840 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b65346740 10 client.0 _ll_register_callbacks cb 0x7f4b44005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b5d36e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05079520 0x55ac054303a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b5d36e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05079520 0x55ac054303a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49610/0 (socket says 172.21.15.16:49610) 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b5d36e640 1 -- 172.21.15.16:0/3648402424 learned_addr learned my addr 172.21.15.16:0/3648402424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b5cb6d640 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac0542f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b57fff640 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05058840 0x55ac0542fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b5d36e640 1 -- 172.21.15.16:0/3648402424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05058840 msgr2=0x55ac0542fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b5d36e640 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05058840 0x55ac0542fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b5d36e640 1 -- 172.21.15.16:0/3648402424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 msgr2=0x55ac0542f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b5d36e640 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac0542f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b5d36e640 1 -- 172.21.15.16:0/3648402424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac054a73c0 con 0x55ac05079520 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b5d36e640 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05079520 0x55ac054303a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f4b50003fa0 tx=0x7f4b50005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4e22210a102298 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b37fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.558+0000 7f4b37fff640 1 -- 172.21.15.16:0/3648402424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50011640 con 0x55ac05079520 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b37fff640 1 -- 172.21.15.16:0/3648402424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f4b500117e0 con 0x55ac05079520 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b65346740 1 -- 172.21.15.16:0/3648402424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b440cc8c0 con 0x55ac05079520 2024-08-28T19:31:11.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b37fff640 1 -- 172.21.15.16:0/3648402424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f4b50031050 con 0x55ac05079520 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b65346740 10 client.18069 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b65346740 1 -- 172.21.15.16:0/3648402424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ac054a86e0 con 0x55ac05079520 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b37fff640 1 -- 172.21.15.16:0/3648402424 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b440cc8c0 con 0x55ac05079520 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b37fff640 1 -- 172.21.15.16:0/3648402424 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f4b50029ad0 con 0x55ac05079520 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b65346740 10 client.18069 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b65346740 10 client.18069 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b65346740 20 client.18069 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b65346740 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ac0500d250 0x55ac054aaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b65346740 4 client.18069 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c46102d0", "release": true, "await": true}] 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b65346740 1 -- 172.21.15.16:0/3648402424 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c46102d0", "release": true, "await": true}) -- 0x55ac05393460 con 0x55ac0500d250 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.559+0000 7f4b5cb6d640 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ac0500d250 0x55ac054aaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.560+0000 7f4b5cb6d640 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ac0500d250 0x55ac054aaf20 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d85dc42686908d7 server_cookie=2f808370fe6985b0 in_seq=0 out_seq=0 2024-08-28T19:31:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.560+0000 7f4b37fff640 10 client.18069 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:11.565 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:11 smithi028 ceph-mon[21769]: pgmap v650: 129 pgs: 129 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 106 KiB/s wr, 41 op/s 2024-08-28T19:31:11.565 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:11 smithi028 ceph-mon[21769]: osdmap e92: 12 total, 12 up, 12 in 2024-08-28T19:31:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.591+0000 7f4b37fff640 1 -- 172.21.15.16:0/3648402424 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ac05393460 con 0x55ac0500d250 2024-08-28T19:31:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.591+0000 7f4b37fff640 10 client.18069 handle_command_reply: tid=0 2024-08-28T19:31:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.591+0000 7f4b65346740 1 client.18069 shutdown 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.591+0000 7f4b65346740 1 -- 172.21.15.16:0/3648402424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05079520 msgr2=0x55ac054303a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.591+0000 7f4b65346740 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05079520 0x55ac054303a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f4b50003fa0 tx=0x7f4b50005bf0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.591+0000 7f4b65346740 1 -- 172.21.15.16:0/3648402424 shutdown_connections 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.592+0000 7f4b65346740 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac05079520 0x55ac054303a0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.592+0000 7f4b65346740 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ac0500d250 0x55ac054aaf20 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.592+0000 7f4b65346740 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ac05058840 0x55ac0542fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.592+0000 7f4b65346740 1 --2- 172.21.15.16:0/3648402424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac05050660 0x55ac0542f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.592+0000 7f4b65346740 1 -- 172.21.15.16:0/3648402424 >> 172.21.15.16:0/3648402424 conn(0x55ac0506d910 msgr2=0x55ac0541fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.592+0000 7f4b37fff640 0 client.18069 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.592+0000 7f4b65346740 1 -- 172.21.15.16:0/3648402424 shutdown_connections 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.592+0000 7f4b65346740 1 -- 172.21.15.16:0/3648402424 wait complete. 2024-08-28T19:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.592+0000 7f4b65346740 20 client.18069 trim_cache size 0 max 16384 2024-08-28T19:31:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:31:11.595 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 55, 2024-08-28T19:31:11.595 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:31:11.595 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 32, 2024-08-28T19:31:11.595 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:31:11.595 INFO:teuthology.orchestra.run.smithi016.stdout: "c46102d0": { 2024-08-28T19:31:11.595 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 32, 2024-08-28T19:31:11.595 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.1, 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:31:11.596 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:31:11.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:11.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:11.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:11.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:11.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 -- 172.21.15.16:0/1346263256 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b480b4ca0 msgr2=0x7f4b480b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b480b4ca0 0x7f4b480b71b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000ad60 tx=0x7f4b5000a2f0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 -- 172.21.15.16:0/1346263256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b58109240 msgr2=0x7f4b5819a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b58109240 0x7f4b5819a8e0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f4b4402a930 tx=0x7f4b44004270 comp rx=0 tx=0).stop 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 -- 172.21.15.16:0/1346263256 shutdown_connections 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b58109b90 0x7f4b5819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b58109240 0x7f4b5819a8e0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4b480b4ca0 0x7f4b480b71b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 --2- 172.21.15.16:0/1346263256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58103900 0x7f4b5819a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 -- 172.21.15.16:0/1346263256 >> 172.21.15.16:0/1346263256 conn(0x7f4b5807b5f0 msgr2=0x7f4b581073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 -- 172.21.15.16:0/1346263256 shutdown_connections 2024-08-28T19:31:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:11.593+0000 7f4b5edd1640 1 -- 172.21.15.16:0/1346263256 wait complete. 2024-08-28T19:31:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:11 smithi115 ceph-mon[22184]: pgmap v650: 129 pgs: 129 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 106 KiB/s wr, 41 op/s 2024-08-28T19:31:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:11 smithi115 ceph-mon[22184]: osdmap e92: 12 total, 12 up, 12 in 2024-08-28T19:31:11.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 435220 3.94 MB/sec execute 411 sec latency 25685.100 ms 2024-08-28T19:31:12.294 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c46102d0', total seconds elapsed: 30.6 2024-08-28T19:31:12.294 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.05 2024-08-28T19:31:12.294 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:31:12.561 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:12.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 436745 3.96 MB/sec execute 412 sec latency 12.467 ms 2024-08-28T19:31:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.063+0000 7fe8f8397640 1 -- 172.21.15.16:0/3487330263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8f0104750 msgr2=0x7fe8f0104b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.063+0000 7fe8f8397640 1 --2- 172.21.15.16:0/3487330263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8f0104750 0x7fe8f0104b60 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009900 tx=0x7fe8ec02a410 comp rx=0 tx=0).stop 2024-08-28T19:31:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.063+0000 7fe8f8397640 1 -- 172.21.15.16:0/3487330263 shutdown_connections 2024-08-28T19:31:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.063+0000 7fe8f8397640 1 --2- 172.21.15.16:0/3487330263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8f010dbb0 0x7fe8f01100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.063+0000 7fe8f8397640 1 --2- 172.21.15.16:0/3487330263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8f01050a0 0x7fe8f010d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.063+0000 7fe8f8397640 1 --2- 172.21.15.16:0/3487330263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8f0104750 0x7fe8f0104b60 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.063+0000 7fe8f8397640 1 -- 172.21.15.16:0/3487330263 >> 172.21.15.16:0/3487330263 conn(0x7fe8f00794d0 msgr2=0x7fe8f00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.063+0000 7fe8f8397640 1 -- 172.21.15.16:0/3487330263 shutdown_connections 2024-08-28T19:31:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.063+0000 7fe8f8397640 1 -- 172.21.15.16:0/3487330263 wait complete. 2024-08-28T19:31:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.063+0000 7fe8f8397640 1 Processor -- start 2024-08-28T19:31:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.063+0000 7fe8f8397640 1 -- start start 2024-08-28T19:31:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f8397640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8f0104750 0x7fe8f019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f8397640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8f01050a0 0x7fe8f019a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f8397640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8f010dbb0 0x7fe8f019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f8397640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f0113830 con 0x7fe8f0104750 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f8397640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f01136b0 con 0x7fe8f01050a0 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f8397640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f01139b0 con 0x7fe8f010dbb0 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f6133640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8f0104750 0x7fe8f019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f6133640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8f0104750 0x7fe8f019a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49630/0 (socket says 172.21.15.16:49630) 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f6133640 1 -- 172.21.15.16:0/4221437683 learned_addr learned my addr 172.21.15.16:0/4221437683 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f5932640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8f01050a0 0x7fe8f019a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f6934640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8f010dbb0 0x7fe8f019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f6133640 1 -- 172.21.15.16:0/4221437683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8f010dbb0 msgr2=0x7fe8f019fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f6133640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8f010dbb0 0x7fe8f019fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f6133640 1 -- 172.21.15.16:0/4221437683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8f01050a0 msgr2=0x7fe8f019a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f6133640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8f01050a0 0x7fe8f019a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f6133640 1 -- 172.21.15.16:0/4221437683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f01a0270 con 0x7fe8f0104750 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f6934640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8f010dbb0 0x7fe8f019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.064+0000 7fe8f6133640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8f0104750 0x7fe8f019a300 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec032fd0 tx=0x7fe8ec031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.065+0000 7fe8e77fe640 1 -- 172.21.15.16:0/4221437683 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec03f030 con 0x7fe8f0104750 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.065+0000 7fe8e77fe640 1 -- 172.21.15.16:0/4221437683 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8ec004490 con 0x7fe8f0104750 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.065+0000 7fe8f8397640 1 -- 172.21.15.16:0/4221437683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8f01a0500 con 0x7fe8f0104750 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.065+0000 7fe8e77fe640 1 -- 172.21.15.16:0/4221437683 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec035070 con 0x7fe8f0104750 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.065+0000 7fe8f8397640 1 -- 172.21.15.16:0/4221437683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8f01a6cc0 con 0x7fe8f0104750 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.066+0000 7fe8f8397640 1 -- 172.21.15.16:0/4221437683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8f0104b60 con 0x7fe8f0104750 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.067+0000 7fe8e77fe640 1 -- 172.21.15.16:0/4221437683 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe8ec042030 con 0x7fe8f0104750 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.068+0000 7fe8e77fe640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8dc0b4ca0 0x7fe8dc0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.068+0000 7fe8f5932640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8dc0b4ca0 0x7fe8dc0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:13.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.068+0000 7fe8e77fe640 1 -- 172.21.15.16:0/4221437683 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fe8ec105d30 con 0x7fe8f0104750 2024-08-28T19:31:13.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.069+0000 7fe8f5932640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8dc0b4ca0 0x7fe8dc0b71b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe8f01021c0 tx=0x7fe8e0005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:13.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.073+0000 7fe8e77fe640 1 -- 172.21.15.16:0/4221437683 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fe8ec0cc6f0 con 0x7fe8f0104750 2024-08-28T19:31:13.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.311+0000 7fe8f8397640 1 -- 172.21.15.16:0/4221437683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8f0108cb0 con 0x7fe8f0104750 2024-08-28T19:31:13.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.312+0000 7fe8e77fe640 1 -- 172.21.15.16:0/4221437683 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11977 (secure 0 0 0) 0x7fe8ec0cbff0 con 0x7fe8f0104750 2024-08-28T19:31:13.313 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:31:13.314 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":55,"btime":"2024-08-28T19:30:56:995905+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:30:56.593478+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:31:13.314 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 55 2024-08-28T19:31:13.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.314+0000 7fe8f8397640 1 -- 172.21.15.16:0/4221437683 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8dc0b4ca0 msgr2=0x7fe8dc0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:13.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.314+0000 7fe8f8397640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8dc0b4ca0 0x7fe8dc0b71b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe8f01021c0 tx=0x7fe8e0005f90 comp rx=0 tx=0).stop 2024-08-28T19:31:13.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.314+0000 7fe8f8397640 1 -- 172.21.15.16:0/4221437683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8f0104750 msgr2=0x7fe8f019a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:13.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.314+0000 7fe8f8397640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8f0104750 0x7fe8f019a300 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec032fd0 tx=0x7fe8ec031a20 comp rx=0 tx=0).stop 2024-08-28T19:31:13.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.314+0000 7fe8f8397640 1 -- 172.21.15.16:0/4221437683 shutdown_connections 2024-08-28T19:31:13.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.314+0000 7fe8f8397640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8f010dbb0 0x7fe8f019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:13.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.314+0000 7fe8f8397640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8dc0b4ca0 0x7fe8dc0b71b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:13.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.314+0000 7fe8f8397640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8f01050a0 0x7fe8f019a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:13.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.315+0000 7fe8f8397640 1 --2- 172.21.15.16:0/4221437683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8f0104750 0x7fe8f019a300 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:13.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.315+0000 7fe8f8397640 1 -- 172.21.15.16:0/4221437683 >> 172.21.15.16:0/4221437683 conn(0x7fe8f00794d0 msgr2=0x7fe8f00ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:13.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.315+0000 7fe8f8397640 1 -- 172.21.15.16:0/4221437683 shutdown_connections 2024-08-28T19:31:13.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:13.315+0000 7fe8f8397640 1 -- 172.21.15.16:0/4221437683 wait complete. 2024-08-28T19:31:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:13 smithi016 ceph-mon[18556]: pgmap v652: 129 pgs: 129 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-08-28T19:31:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:13 smithi028 ceph-mon[21769]: pgmap v652: 129 pgs: 129 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-08-28T19:31:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:13 smithi115 ceph-mon[22184]: pgmap v652: 129 pgs: 129 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-08-28T19:31:13.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438005 3.96 MB/sec execute 413 sec latency 20.208 ms 2024-08-28T19:31:13.995 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 147.05 --await-for 45.0' 2024-08-28T19:31:13.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 147.05 --await-for 45.0 2024-08-28T19:31:14.250 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:14 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/4221437683' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:14.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:14 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/4221437683' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:14.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:14 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/4221437683' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.773+0000 7f89ba6c8640 1 -- 172.21.15.16:0/3682856350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b406bbf0 msgr2=0x7f89b406c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.773+0000 7f89ba6c8640 1 --2- 172.21.15.16:0/3682856350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b406bbf0 0x7f89b406c000 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009900 tx=0x7f89a402a410 comp rx=0 tx=0).stop 2024-08-28T19:31:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.773+0000 7f89ba6c8640 1 -- 172.21.15.16:0/3682856350 shutdown_connections 2024-08-28T19:31:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.773+0000 7f89ba6c8640 1 --2- 172.21.15.16:0/3682856350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f89b4109bc0 0x7f89b41103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.773+0000 7f89ba6c8640 1 --2- 172.21.15.16:0/3682856350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b4109270 0x7f89b4109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.773+0000 7f89ba6c8640 1 --2- 172.21.15.16:0/3682856350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b406bbf0 0x7f89b406c000 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.773+0000 7f89ba6c8640 1 -- 172.21.15.16:0/3682856350 >> 172.21.15.16:0/3682856350 conn(0x7f89b40795c0 msgr2=0x7f89b40799d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.773+0000 7f89ba6c8640 1 -- 172.21.15.16:0/3682856350 shutdown_connections 2024-08-28T19:31:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.773+0000 7f89ba6c8640 1 -- 172.21.15.16:0/3682856350 wait complete. 2024-08-28T19:31:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89ba6c8640 1 Processor -- start 2024-08-28T19:31:14.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89ba6c8640 1 -- start start 2024-08-28T19:31:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89ba6c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b406bbf0 0x7f89b419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89ba6c8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f89b4109270 0x7f89b419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89ba6c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b4109bc0 0x7f89b419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89ba6c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89b4113880 con 0x7f89b4109bc0 2024-08-28T19:31:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89ba6c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89b4113700 con 0x7f89b406bbf0 2024-08-28T19:31:14.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89ba6c8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f89b4113a00 con 0x7f89b4109270 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89b8c65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b4109bc0 0x7f89b419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89b8c65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b4109bc0 0x7f89b419fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36538/0 (socket says 172.21.15.16:36538) 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89b8c65640 1 -- 172.21.15.16:0/1445166513 learned_addr learned my addr 172.21.15.16:0/1445166513 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89b37fe640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f89b4109270 0x7f89b419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.774+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b406bbf0 0x7f89b419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89b8c65640 1 -- 172.21.15.16:0/1445166513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f89b4109270 msgr2=0x7f89b419a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89b8c65640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f89b4109270 0x7f89b419a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89b8c65640 1 -- 172.21.15.16:0/1445166513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b406bbf0 msgr2=0x7f89b419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89b8c65640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b406bbf0 0x7f89b419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89b8c65640 1 -- 172.21.15.16:0/1445166513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b41a01c0 con 0x7f89b4109bc0 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b406bbf0 0x7f89b419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89b8c65640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b4109bc0 0x7f89b419fa90 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00eb10 tx=0x7f89ac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89b17fa640 1 -- 172.21.15.16:0/1445166513 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac004b20 con 0x7f89b4109bc0 2024-08-28T19:31:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89ba6c8640 1 -- 172.21.15.16:0/1445166513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b41a04b0 con 0x7f89b4109bc0 2024-08-28T19:31:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89b17fa640 1 -- 172.21.15.16:0/1445166513 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89ac004cc0 con 0x7f89b4109bc0 2024-08-28T19:31:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89b17fa640 1 -- 172.21.15.16:0/1445166513 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac010670 con 0x7f89b4109bc0 2024-08-28T19:31:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.775+0000 7f89ba6c8640 1 -- 172.21.15.16:0/1445166513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b41a6c10 con 0x7f89b4109bc0 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89c0c3d740 1 Processor -- start 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89b17fa640 1 -- 172.21.15.16:0/1445166513 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f89ac005020 con 0x7f89b4109bc0 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89c0c3d740 1 -- start start 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6343850 0x55c8b66e9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b631a990 0x55c8b66ea4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6322b70 0x55c8b66eaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c8b665f380 con 0x55c8b6343850 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c8b664e910 con 0x55c8b6322b70 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c8b66528c0 con 0x55c8b631a990 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6343850 0x55c8b66e9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6343850 0x55c8b66e9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36550/0 (socket says 172.21.15.16:36550) 2024-08-28T19:31:14.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.778+0000 7f89b37fe640 1 -- 172.21.15.16:0/2624619859 learned_addr learned my addr 172.21.15.16:0/2624619859 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b37fe640 1 --2- 172.21.15.16:0/2624619859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6322b70 0x55c8b66eaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b37fe640 1 -- 172.21.15.16:0/2624619859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b631a990 msgr2=0x55c8b66ea4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b3fff640 1 --2- 172.21.15.16:0/2624619859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b631a990 0x55c8b66ea4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b37fe640 1 --2- 172.21.15.16:0/2624619859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b631a990 0x55c8b66ea4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b37fe640 1 -- 172.21.15.16:0/2624619859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6322b70 msgr2=0x55c8b66eaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b37fe640 1 --2- 172.21.15.16:0/2624619859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6322b70 0x55c8b66eaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b37fe640 1 -- 172.21.15.16:0/2624619859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8b65eb4c0 con 0x55c8b6343850 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b17fa640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f899c0b4ca0 0x7f899c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b3fff640 1 --2- 172.21.15.16:0/2624619859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b631a990 0x55c8b66ea4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f899c0b4ca0 0x7f899c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b17fa640 1 -- 172.21.15.16:0/1445166513 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f89ac0e5f00 con 0x7f89b4109bc0 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f89b37fe640 1 --2- 172.21.15.16:0/2624619859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6343850 0x55c8b66e9f90 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f89a80135b0 tx=0x7f89a8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5002e9149e565f2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:14.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f899a7fc640 1 -- 172.21.15.16:0/2624619859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a8004050 con 0x55c8b6343850 2024-08-28T19:31:14.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f899a7fc640 1 -- 172.21.15.16:0/2624619859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89a80041f0 con 0x55c8b6343850 2024-08-28T19:31:14.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.779+0000 7f899a7fc640 1 -- 172.21.15.16:0/2624619859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a800f760 con 0x55c8b6343850 2024-08-28T19:31:14.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.780+0000 7f89c0c3d740 1 -- 172.21.15.16:0/2624619859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6343850 msgr2=0x55c8b66e9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:14.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.780+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/2624619859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6343850 0x55c8b66e9f90 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f89a80135b0 tx=0x7f89a8005850 comp rx=0 tx=0).stop 2024-08-28T19:31:14.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.780+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f899c0b4ca0 0x7f899c0b71b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f89a4031440 tx=0x7f89a4032ed0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:14.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.780+0000 7f89c0c3d740 1 -- 172.21.15.16:0/2624619859 shutdown_connections 2024-08-28T19:31:14.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.780+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/2624619859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6322b70 0x55c8b66eaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.780+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/2624619859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b631a990 0x55c8b66ea4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.780+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/2624619859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6343850 0x55c8b66e9f90 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.780+0000 7f89c0c3d740 1 -- 172.21.15.16:0/2624619859 >> 172.21.15.16:0/2624619859 conn(0x55c8b6337c40 msgr2=0x55c8b66e7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:14.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.780+0000 7f89c0c3d740 1 -- 172.21.15.16:0/2624619859 shutdown_connections 2024-08-28T19:31:14.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.780+0000 7f89c0c3d740 1 -- 172.21.15.16:0/2624619859 wait complete. 2024-08-28T19:31:14.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.781+0000 7f89c0c3d740 1 -- start start 2024-08-28T19:31:14.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.781+0000 7f89c0c3d740 1 Processor -- start 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.781+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b66f5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.781+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b6322b70 0x55c8b66f5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.781+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b6701a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.781+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c8b64b17b0 con 0x55c8b631a990 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.781+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c8b65d5460 con 0x55c8b6343850 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.781+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c8b66f8770 con 0x55c8b6322b70 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.781+0000 7f89c0c3d740 10 client.0 _ll_register_callbacks cb 0x55c8b65fafd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.781+0000 7f89b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b66f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b66f5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36560/0 (socket says 172.21.15.16:36560) 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89b3fff640 1 -- 172.21.15.16:0/1733398218 learned_addr learned my addr 172.21.15.16:0/1733398218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89b37fe640 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b6322b70 0x55c8b66f5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89b8c65640 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b6701a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89b3fff640 1 -- 172.21.15.16:0/1733398218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b6322b70 msgr2=0x55c8b66f5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b6322b70 0x55c8b66f5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89b3fff640 1 -- 172.21.15.16:0/1733398218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 msgr2=0x55c8b6701a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b6701a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89b3fff640 1 -- 172.21.15.16:0/1733398218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8b63bdba0 con 0x55c8b631a990 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b66f5620 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009900 tx=0x7f89a402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c77d1ec6b1008e6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89937fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89937fe640 1 -- 172.21.15.16:0/1733398218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a404c020 con 0x55c8b631a990 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89937fe640 1 -- 172.21.15.16:0/1733398218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f89a4004320 con 0x55c8b631a990 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1733398218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89a80041f0 con 0x55c8b631a990 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89937fe640 1 -- 172.21.15.16:0/1733398218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f89a4057050 con 0x55c8b631a990 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89c0c3d740 10 client.18087 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.782+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1733398218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55c8b6704750 con 0x55c8b631a990 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.783+0000 7f89937fe640 1 -- 172.21.15.16:0/1733398218 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89a80041f0 con 0x55c8b631a990 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.783+0000 7f89937fe640 1 -- 172.21.15.16:0/1733398218 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f89a4004bb0 con 0x55c8b631a990 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.783+0000 7f89c0c3d740 10 client.18087 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.783+0000 7f89c0c3d740 10 client.18087 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.783+0000 7f89c0c3d740 20 client.18087 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.783+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c8b63021d0 0x55c8b6707cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.783+0000 7f89c0c3d740 4 client.18087 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.783+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1733398218 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8b66612d0 con 0x55c8b63021d0 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.783+0000 7f89b37fe640 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c8b63021d0 0x55c8b6707cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.784+0000 7f89937fe640 10 client.18087 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.784+0000 7f89b37fe640 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c8b63021d0 0x55c8b6707cb0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7bdc30db7b0ab3 server_cookie=b6a290137aba68aa in_seq=0 out_seq=0 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.784+0000 7f89937fe640 1 -- 172.21.15.16:0/1733398218 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c8b66612d0 con 0x55c8b63021d0 2024-08-28T19:31:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.784+0000 7f89937fe640 10 client.18087 handle_command_reply: tid=0 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.784+0000 7f89c0c3d740 1 client.18087 shutdown 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.784+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1733398218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 msgr2=0x55c8b66f5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.784+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b66f5620 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009900 tx=0x7f89a402a410 comp rx=0 tx=0).stop 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.784+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1733398218 shutdown_connections 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.784+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b6701a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.784+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c8b63021d0 0x55c8b6707cb0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.784+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b6322b70 0x55c8b66f5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.785+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/1733398218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b66f5620 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.785+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1733398218 >> 172.21.15.16:0/1733398218 conn(0x55c8b6337c40 msgr2=0x55c8b66ed170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.785+0000 7f89937fe640 0 client.18087 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.785+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1733398218 shutdown_connections 2024-08-28T19:31:14.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.785+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1733398218 wait complete. 2024-08-28T19:31:14.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.785+0000 7f89c0c3d740 20 client.18087 trim_cache size 0 max 16384 2024-08-28T19:31:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.802+0000 7f89c0c3d740 1 Processor -- start 2024-08-28T19:31:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.802+0000 7f89c0c3d740 1 -- start start 2024-08-28T19:31:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.802+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b6701960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.802+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b6322b70 0x55c8b66e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.802+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b66ef210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.802+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41b0c80 con 0x55c8b631a990 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.802+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41b27c0 con 0x55c8b6343850 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.802+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41b1350 con 0x55c8b6322b70 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.802+0000 7f89b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b6701960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b6701960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36562/0 (socket says 172.21.15.16:36562) 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b3fff640 1 -- 172.21.15.16:0/1535341705 learned_addr learned my addr 172.21.15.16:0/1535341705 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b8c65640 1 --2- 172.21.15.16:0/1535341705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b66ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b37fe640 1 --2- 172.21.15.16:0/1535341705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b6322b70 0x55c8b66e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b3fff640 1 -- 172.21.15.16:0/1535341705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b6322b70 msgr2=0x55c8b66e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1535341705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b6322b70 0x55c8b66e6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b3fff640 1 -- 172.21.15.16:0/1535341705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 msgr2=0x55c8b66ef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1535341705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b66ef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b3fff640 1 -- 172.21.15.16:0/1535341705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8b66e8b60 con 0x55c8b631a990 2024-08-28T19:31:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b37fe640 1 --2- 172.21.15.16:0/1535341705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b6322b70 0x55c8b66e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b8c65640 1 --2- 172.21.15.16:0/1535341705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b66ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1535341705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b6701960 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f89a4033000 tx=0x7f89a4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b6698afcc011f6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f8999ffb640 1 -- 172.21.15.16:0/1535341705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a405f020 con 0x55c8b631a990 2024-08-28T19:31:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f8999ffb640 1 -- 172.21.15.16:0/1535341705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89a4004c20 con 0x55c8b631a990 2024-08-28T19:31:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.803+0000 7f8999ffb640 1 -- 172.21.15.16:0/1535341705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a404c070 con 0x55c8b631a990 2024-08-28T19:31:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.804+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1535341705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 msgr2=0x55c8b6701960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.804+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/1535341705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b6701960 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f89a4033000 tx=0x7f89a4000bc0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.804+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1535341705 shutdown_connections 2024-08-28T19:31:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.804+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/1535341705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b66ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.804+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/1535341705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b6322b70 0x55c8b66e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.804+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/1535341705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b631a990 0x55c8b6701960 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.804+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1535341705 >> 172.21.15.16:0/1535341705 conn(0x55c8b6337c40 msgr2=0x55c8b66e7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.804+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1535341705 shutdown_connections 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.804+0000 7f89c0c3d740 1 -- 172.21.15.16:0/1535341705 wait complete. 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89c0c3d740 1 -- start start 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89c0c3d740 1 Processor -- start 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b631a990 0x55c8b66fdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6322b70 0x55c8b66f8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89c0c3d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b66f8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41b20f0 con 0x55c8b6322b70 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41b1a20 con 0x55c8b6343850 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89c0c3d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41aec80 con 0x55c8b631a990 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89c0c3d740 10 client.0 _ll_register_callbacks cb 0x7f89a4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6322b70 0x55c8b66f8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6322b70 0x55c8b66f8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36566/0 (socket says 172.21.15.16:36566) 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.805+0000 7f89b37fe640 1 -- 172.21.15.16:0/3561042791 learned_addr learned my addr 172.21.15.16:0/3561042791 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89b3fff640 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b631a990 0x55c8b66fdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89b8c65640 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b66f8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89b37fe640 1 -- 172.21.15.16:0/3561042791 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b631a990 msgr2=0x55c8b66fdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89b37fe640 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b631a990 0x55c8b66fdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89b37fe640 1 -- 172.21.15.16:0/3561042791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 msgr2=0x55c8b66f8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89b37fe640 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b66f8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89b37fe640 1 -- 172.21.15.16:0/3561042791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8b66f9400 con 0x55c8b6322b70 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89b8c65640 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b66f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89b37fe640 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6322b70 0x55c8b66f8980 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f89a8003d50 tx=0x7f89a8009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec09b3bf3200852 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89937fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89937fe640 1 -- 172.21.15.16:0/3561042791 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a80035f0 con 0x55c8b6322b70 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89937fe640 1 -- 172.21.15.16:0/3561042791 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f89a8003790 con 0x55c8b6322b70 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.806+0000 7f89c0c3d740 1 -- 172.21.15.16:0/3561042791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89a4004c20 con 0x55c8b6322b70 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89937fe640 1 -- 172.21.15.16:0/3561042791 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f89a8036050 con 0x55c8b6322b70 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89c0c3d740 10 client.18093 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89c0c3d740 1 -- 172.21.15.16:0/3561042791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55c8b6773250 con 0x55c8b6322b70 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89937fe640 1 -- 172.21.15.16:0/3561042791 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89a4004c20 con 0x55c8b6322b70 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89937fe640 1 -- 172.21.15.16:0/3561042791 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f89a801d020 con 0x55c8b6322b70 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89c0c3d740 10 client.18093 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89c0c3d740 10 client.18093 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89c0c3d740 20 client.18093 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c8b62d7580 0x55c8b67758e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89c0c3d740 4 client.18093 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.05, "await_for": 45.0}] 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89c0c3d740 1 -- 172.21.15.16:0/3561042791 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.05, "await_for": 45.0}) -- 0x55c8b665cd70 con 0x55c8b62d7580 2024-08-28T19:31:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.807+0000 7f89b3fff640 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c8b62d7580 0x55c8b67758e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:14.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.808+0000 7f89937fe640 10 client.18093 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:14.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.808+0000 7f89b3fff640 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c8b62d7580 0x55c8b67758e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d3a9a89500764a8 server_cookie=23686d072bfac0a4 in_seq=0 out_seq=0 2024-08-28T19:31:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.878+0000 7f89937fe640 1 -- 172.21.15.16:0/3561042791 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c8b665cd70 con 0x55c8b62d7580 2024-08-28T19:31:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.878+0000 7f89937fe640 10 client.18093 handle_command_reply: tid=0 2024-08-28T19:31:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 1 client.18093 shutdown 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 1 -- 172.21.15.16:0/3561042791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6322b70 msgr2=0x55c8b66f8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6322b70 0x55c8b66f8980 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f89a8003d50 tx=0x7f89a8009810 comp rx=0 tx=0).stop 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 1 -- 172.21.15.16:0/3561042791 shutdown_connections 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8b6343850 0x55c8b66f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8b6322b70 0x55c8b66f8980 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c8b62d7580 0x55c8b67758e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 1 --2- 172.21.15.16:0/3561042791 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c8b631a990 0x55c8b66fdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89937fe640 0 client.18093 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 1 -- 172.21.15.16:0/3561042791 >> 172.21.15.16:0/3561042791 conn(0x55c8b6337c40 msgr2=0x55c8b66f6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 1 -- 172.21.15.16:0/3561042791 shutdown_connections 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 1 -- 172.21.15.16:0/3561042791 wait complete. 2024-08-28T19:31:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.879+0000 7f89c0c3d740 20 client.18093 trim_cache size 0 max 16384 2024-08-28T19:31:14.882 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:31:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 55, 2024-08-28T19:31:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:31:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 34, 2024-08-28T19:31:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:31:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "14a4328a": { 2024-08-28T19:31:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 34, 2024-08-28T19:31:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:31:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:31:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.1, 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:14.884 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:31:14.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 -- 172.21.15.16:0/1445166513 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f899c0b4ca0 msgr2=0x7f899c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:14.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f899c0b4ca0 0x7f899c0b71b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f89a4031440 tx=0x7f89a4032ed0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 -- 172.21.15.16:0/1445166513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b4109bc0 msgr2=0x7f89b419fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:14.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b4109bc0 0x7f89b419fa90 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00eb10 tx=0x7f89ac00c2d0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 -- 172.21.15.16:0/1445166513 shutdown_connections 2024-08-28T19:31:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b4109bc0 0x7f89b419fa90 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f89b4109270 0x7f89b419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f899c0b4ca0 0x7f899c0b71b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 --2- 172.21.15.16:0/1445166513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b406bbf0 0x7f89b419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 -- 172.21.15.16:0/1445166513 >> 172.21.15.16:0/1445166513 conn(0x7f89b40795c0 msgr2=0x7f89b4107360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 -- 172.21.15.16:0/1445166513 shutdown_connections 2024-08-28T19:31:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:14.880+0000 7f89ba6c8640 1 -- 172.21.15.16:0/1445166513 wait complete. 2024-08-28T19:31:14.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.95 MB/sec execute 414 sec latency 868.726 ms 2024-08-28T19:31:15.496 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:15 smithi016 ceph-mon[18556]: pgmap v653: 129 pgs: 129 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 117 KiB/s wr, 34 op/s 2024-08-28T19:31:15.523 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '14a4328a', quiesce took 3.2 seconds. Will release after: 23.85 2024-08-28T19:31:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:15 smithi028 ceph-mon[21769]: pgmap v653: 129 pgs: 129 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 117 KiB/s wr, 34 op/s 2024-08-28T19:31:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:15 smithi115 ceph-mon[22184]: pgmap v653: 129 pgs: 129 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 117 KiB/s wr, 34 op/s 2024-08-28T19:31:15.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.94 MB/sec execute 415 sec latency 1133.912 ms 2024-08-28T19:31:16.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.94 MB/sec execute 416 sec latency 2133.984 ms 2024-08-28T19:31:17.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:17 smithi028 ceph-mon[21769]: pgmap v654: 129 pgs: 129 active+clean; 306 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 230 KiB/s wr, 50 op/s 2024-08-28T19:31:17.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:17 smithi115 ceph-mon[22184]: pgmap v654: 129 pgs: 129 active+clean; 306 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 230 KiB/s wr, 50 op/s 2024-08-28T19:31:17.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:17 smithi016 ceph-mon[18556]: pgmap v654: 129 pgs: 129 active+clean; 306 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 230 KiB/s wr, 50 op/s 2024-08-28T19:31:17.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.93 MB/sec execute 417 sec latency 3134.063 ms 2024-08-28T19:31:18.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.92 MB/sec execute 418 sec latency 4134.159 ms 2024-08-28T19:31:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:19 smithi028 ceph-mon[21769]: pgmap v655: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 497 KiB/s wr, 74 op/s 2024-08-28T19:31:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:31:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:31:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:31:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:31:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:31:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:19 smithi115 ceph-mon[22184]: pgmap v655: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 497 KiB/s wr, 74 op/s 2024-08-28T19:31:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:31:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:31:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:31:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:31:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:31:19.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:31:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:19 smithi016 ceph-mon[18556]: pgmap v655: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 497 KiB/s wr, 74 op/s 2024-08-28T19:31:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:31:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:31:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:31:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:31:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:31:19.968 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.91 MB/sec execute 419 sec latency 5134.193 ms 2024-08-28T19:31:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.526+0000 7f00c3fdc640 1 -- 172.21.15.16:0/1383975260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00bc101e20 msgr2=0x7f00bc102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.526+0000 7f00c3fdc640 1 --2- 172.21.15.16:0/1383975260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00bc101e20 0x7f00bc102230 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009900 tx=0x7f00b802a480 comp rx=0 tx=0).stop 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.526+0000 7f00c3fdc640 1 -- 172.21.15.16:0/1383975260 shutdown_connections 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.526+0000 7f00c3fdc640 1 --2- 172.21.15.16:0/1383975260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc114260 0x7f00bc116770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.527+0000 7f00c3fdc640 1 --2- 172.21.15.16:0/1383975260 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00bc102770 0x7f00bc113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.527+0000 7f00c3fdc640 1 --2- 172.21.15.16:0/1383975260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00bc101e20 0x7f00bc102230 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.527+0000 7f00c3fdc640 1 -- 172.21.15.16:0/1383975260 >> 172.21.15.16:0/1383975260 conn(0x7f00bc079c60 msgr2=0x7f00bc07a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.527+0000 7f00c3fdc640 1 -- 172.21.15.16:0/1383975260 shutdown_connections 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.527+0000 7f00c3fdc640 1 -- 172.21.15.16:0/1383975260 wait complete. 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.527+0000 7f00c3fdc640 1 Processor -- start 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.527+0000 7f00c3fdc640 1 -- start start 2024-08-28T19:31:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c3fdc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00bc101e20 0x7f00bc1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c3fdc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00bc102770 0x7f00bc1a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c3fdc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc114260 0x7f00bc1a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c3fdc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc119ee0 con 0x7f00bc102770 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c3fdc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc119d60 con 0x7f00bc114260 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c3fdc640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc11a060 con 0x7f00bc101e20 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c1577640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00bc102770 0x7f00bc1a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c1577640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00bc102770 0x7f00bc1a0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36592/0 (socket says 172.21.15.16:36592) 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c1577640 1 -- 172.21.15.16:0/948773270 learned_addr learned my addr 172.21.15.16:0/948773270 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c1d78640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00bc101e20 0x7f00bc1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c2579640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc114260 0x7f00bc1a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c1577640 1 -- 172.21.15.16:0/948773270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00bc101e20 msgr2=0x7f00bc1a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c1577640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00bc101e20 0x7f00bc1a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c1577640 1 -- 172.21.15.16:0/948773270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc114260 msgr2=0x7f00bc1a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c1577640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc114260 0x7f00bc1a60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c1577640 1 -- 172.21.15.16:0/948773270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00bc1a6800 con 0x7f00bc102770 2024-08-28T19:31:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c2579640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc114260 0x7f00bc1a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00c1577640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00bc102770 0x7f00bc1a0e30 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00e9f0 tx=0x7f00ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.528+0000 7f00b2ffd640 1 -- 172.21.15.16:0/948773270 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac00ccc0 con 0x7f00bc102770 2024-08-28T19:31:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.529+0000 7f00b2ffd640 1 -- 172.21.15.16:0/948773270 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00ac0054d0 con 0x7f00bc102770 2024-08-28T19:31:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.529+0000 7f00c3fdc640 1 -- 172.21.15.16:0/948773270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00bc1a6af0 con 0x7f00bc102770 2024-08-28T19:31:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.529+0000 7f00b2ffd640 1 -- 172.21.15.16:0/948773270 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac010430 con 0x7f00bc102770 2024-08-28T19:31:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.529+0000 7f00c3fdc640 1 -- 172.21.15.16:0/948773270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00bc1ad250 con 0x7f00bc102770 2024-08-28T19:31:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.531+0000 7f00ca551740 1 Processor -- start 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.531+0000 7f00b2ffd640 1 -- 172.21.15.16:0/948773270 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f00ac0046c0 con 0x7f00bc102770 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.532+0000 7f00ca551740 1 -- start start 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.532+0000 7f00ca551740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 0x55db25de72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.532+0000 7f00ca551740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 0x55db25de7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.532+0000 7f00ca551740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25de7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.532+0000 7f00ca551740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55db25d5e630 con 0x55db25a42070 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.532+0000 7f00ca551740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db25d4dbc0 con 0x55db25a21390 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.532+0000 7f00ca551740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55db25d51b70 con 0x55db25a191b0 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.532+0000 7f00c1d78640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 0x55db25de7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.532+0000 7f00c1d78640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 0x55db25de7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:60042/0 (socket says 172.21.15.16:60042) 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.532+0000 7f00c1d78640 1 -- 172.21.15.16:0/4001312869 learned_addr learned my addr 172.21.15.16:0/4001312869 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.533+0000 7f00c2579640 1 --2- 172.21.15.16:0/4001312869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 0x55db25de72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.533+0000 7f00c1577640 1 --2- 172.21.15.16:0/4001312869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25de7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.533+0000 7f00b2ffd640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00a80b4ca0 0x7f00a80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.533+0000 7f00c2579640 1 -- 172.21.15.16:0/4001312869 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 msgr2=0x55db25de7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.533+0000 7f00c2579640 1 --2- 172.21.15.16:0/4001312869 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 0x55db25de7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.533+0000 7f00c2579640 1 -- 172.21.15.16:0/4001312869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 msgr2=0x55db25de7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.533+0000 7f00c2579640 1 --2- 172.21.15.16:0/4001312869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25de7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.533+0000 7f00c2579640 1 -- 172.21.15.16:0/4001312869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db25aad650 con 0x55db25a42070 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.533+0000 7f00b2ffd640 1 -- 172.21.15.16:0/948773270 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f00ac0e5dd0 con 0x7f00bc102770 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.533+0000 7f00c2579640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00a80b4ca0 0x7f00a80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.533+0000 7f00c2579640 1 --2- 172.21.15.16:0/4001312869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 0x55db25de72c0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f00b4005fd0 tx=0x7f00b400f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18ed596ea99cc90 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.534+0000 7f00a7fff640 1 -- 172.21.15.16:0/4001312869 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4023070 con 0x55db25a42070 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.534+0000 7f00c2579640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00a80b4ca0 0x7f00a80b71b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f00b4020000 tx=0x7f00b401d000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.534+0000 7f00a7fff640 1 -- 172.21.15.16:0/4001312869 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00b4003e70 con 0x55db25a42070 2024-08-28T19:31:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.534+0000 7f00a7fff640 1 -- 172.21.15.16:0/4001312869 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b401d830 con 0x55db25a42070 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.535+0000 7f00ca551740 1 -- 172.21.15.16:0/4001312869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 msgr2=0x55db25de72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.535+0000 7f00ca551740 1 --2- 172.21.15.16:0/4001312869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 0x55db25de72c0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f00b4005fd0 tx=0x7f00b400f840 comp rx=0 tx=0).stop 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.535+0000 7f00ca551740 1 -- 172.21.15.16:0/4001312869 shutdown_connections 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.535+0000 7f00ca551740 1 --2- 172.21.15.16:0/4001312869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25de7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.535+0000 7f00ca551740 1 --2- 172.21.15.16:0/4001312869 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 0x55db25de7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.535+0000 7f00ca551740 1 --2- 172.21.15.16:0/4001312869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 0x55db25de72c0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.535+0000 7f00ca551740 1 -- 172.21.15.16:0/4001312869 >> 172.21.15.16:0/4001312869 conn(0x55db25a36460 msgr2=0x55db25de51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.535+0000 7f00ca551740 1 -- 172.21.15.16:0/4001312869 shutdown_connections 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.535+0000 7f00ca551740 1 -- 172.21.15.16:0/4001312869 wait complete. 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.535+0000 7f00ca551740 1 -- start start 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.535+0000 7f00ca551740 1 Processor -- start 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00ca551740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a191b0 0x55db25df2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00ca551740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a21390 0x55db25df2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00ca551740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a42070 0x55db25dfedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00ca551740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55db25bb0700 con 0x55db25a21390 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00ca551740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db25cd4700 con 0x55db25a191b0 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00ca551740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55db25df5aa0 con 0x55db25a42070 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00ca551740 10 client.0 _ll_register_callbacks cb 0x55db25d03dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c1577640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a21390 0x55db25df2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c1577640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a21390 0x55db25df2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36602/0 (socket says 172.21.15.16:36602) 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c1577640 1 -- 172.21.15.16:0/584586371 learned_addr learned my addr 172.21.15.16:0/584586371 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c2579640 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a42070 0x55db25dfedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c1d78640 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a191b0 0x55db25df2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c1577640 1 -- 172.21.15.16:0/584586371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a42070 msgr2=0x55db25dfedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c1577640 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a42070 0x55db25dfedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c1577640 1 -- 172.21.15.16:0/584586371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a191b0 msgr2=0x55db25df2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c1577640 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a191b0 0x55db25df2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c1577640 1 -- 172.21.15.16:0/584586371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db25d0d8b0 con 0x55db25a21390 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c2579640 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a42070 0x55db25dfedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00c1577640 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a21390 0x55db25df2fe0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f00ac0e31b0 tx=0x7f00ac0ac600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89de3be312e4d194 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.536+0000 7f00a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00a57fa640 1 -- 172.21.15.16:0/584586371 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac0abec0 con 0x55db25a21390 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00a57fa640 1 -- 172.21.15.16:0/584586371 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f00ac0ab350 con 0x55db25a21390 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00ca551740 1 -- 172.21.15.16:0/584586371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00b4003e70 con 0x55db25a21390 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00a57fa640 1 -- 172.21.15.16:0/584586371 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f00ac0ab860 con 0x55db25a21390 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00ca551740 10 client.18105 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00ca551740 1 -- 172.21.15.16:0/584586371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55db25e01900 con 0x55db25a21390 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00a57fa640 1 -- 172.21.15.16:0/584586371 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00b4003e70 con 0x55db25a21390 2024-08-28T19:31:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00a57fa640 1 -- 172.21.15.16:0/584586371 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f00ac0d2020 con 0x55db25a21390 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00ca551740 10 client.18105 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00ca551740 10 client.18105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00ca551740 20 client.18105 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00ca551740 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db25a009f0 0x55db25e04d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00ca551740 4 client.18105 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.537+0000 7f00ca551740 1 -- 172.21.15.16:0/584586371 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db25d60580 con 0x55db25a009f0 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.538+0000 7f00c1d78640 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db25a009f0 0x55db25e04d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.538+0000 7f00a57fa640 10 client.18105 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.538+0000 7f00c1d78640 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db25a009f0 0x55db25e04d20 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18eaeba76a7f3021 server_cookie=f83eeca2f54ba0e4 in_seq=0 out_seq=0 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00a57fa640 1 -- 172.21.15.16:0/584586371 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55db25d60580 con 0x55db25a009f0 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00a57fa640 10 client.18105 handle_command_reply: tid=0 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 1 client.18105 shutdown 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 1 -- 172.21.15.16:0/584586371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a21390 msgr2=0x55db25df2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a21390 0x55db25df2fe0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f00ac0e31b0 tx=0x7f00ac0ac600 comp rx=0 tx=0).stop 2024-08-28T19:31:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 1 -- 172.21.15.16:0/584586371 shutdown_connections 2024-08-28T19:31:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db25a009f0 0x55db25e04d20 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a42070 0x55db25dfedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a21390 0x55db25df2fe0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 1 --2- 172.21.15.16:0/584586371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a191b0 0x55db25df2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00a57fa640 0 client.18105 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 1 -- 172.21.15.16:0/584586371 >> 172.21.15.16:0/584586371 conn(0x55db25a36460 msgr2=0x55db25dea4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 1 -- 172.21.15.16:0/584586371 shutdown_connections 2024-08-28T19:31:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 1 -- 172.21.15.16:0/584586371 wait complete. 2024-08-28T19:31:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.539+0000 7f00ca551740 20 client.18105 trim_cache size 0 max 16384 2024-08-28T19:31:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00ca551740 1 Processor -- start 2024-08-28T19:31:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00ca551740 1 -- start start 2024-08-28T19:31:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00ca551740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 0x55db25df46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00ca551740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25dfecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00ca551740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 0x55db25dff230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00ca551740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1b7270 con 0x55db25a42070 2024-08-28T19:31:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00ca551740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1b8db0 con 0x55db25a21390 2024-08-28T19:31:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00ca551740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1b7940 con 0x55db25a191b0 2024-08-28T19:31:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00c2579640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 0x55db25dff230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00c2579640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 0x55db25dff230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36614/0 (socket says 172.21.15.16:36614) 2024-08-28T19:31:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00c2579640 1 -- 172.21.15.16:0/1356792461 learned_addr learned my addr 172.21.15.16:0/1356792461 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.557+0000 7f00c1d78640 1 --2- 172.21.15.16:0/1356792461 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 0x55db25df46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00c2579640 1 -- 172.21.15.16:0/1356792461 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 msgr2=0x55db25df46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00c1577640 1 --2- 172.21.15.16:0/1356792461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25dfecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00c2579640 1 --2- 172.21.15.16:0/1356792461 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 0x55db25df46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00c2579640 1 -- 172.21.15.16:0/1356792461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 msgr2=0x55db25dfecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00c2579640 1 --2- 172.21.15.16:0/1356792461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25dfecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00c2579640 1 -- 172.21.15.16:0/1356792461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db25dea120 con 0x55db25a42070 2024-08-28T19:31:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00c1d78640 1 --2- 172.21.15.16:0/1356792461 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 0x55db25df46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00c1577640 1 --2- 172.21.15.16:0/1356792461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25dfecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00c2579640 1 --2- 172.21.15.16:0/1356792461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 0x55db25dff230 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f00b400fd40 tx=0x7f00b4005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e77c1ac16f291 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00a77fe640 1 -- 172.21.15.16:0/1356792461 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4023040 con 0x55db25a42070 2024-08-28T19:31:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00a77fe640 1 -- 172.21.15.16:0/1356792461 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00b401d760 con 0x55db25a42070 2024-08-28T19:31:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.558+0000 7f00a77fe640 1 -- 172.21.15.16:0/1356792461 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b40052b0 con 0x55db25a42070 2024-08-28T19:31:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.559+0000 7f00ca551740 1 -- 172.21.15.16:0/1356792461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 msgr2=0x55db25dff230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.559+0000 7f00ca551740 1 --2- 172.21.15.16:0/1356792461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 0x55db25dff230 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f00b400fd40 tx=0x7f00b4005f30 comp rx=0 tx=0).stop 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.559+0000 7f00ca551740 1 -- 172.21.15.16:0/1356792461 shutdown_connections 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.559+0000 7f00ca551740 1 --2- 172.21.15.16:0/1356792461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a42070 0x55db25dff230 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.559+0000 7f00ca551740 1 --2- 172.21.15.16:0/1356792461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25dfecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.559+0000 7f00ca551740 1 --2- 172.21.15.16:0/1356792461 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a191b0 0x55db25df46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.559+0000 7f00ca551740 1 -- 172.21.15.16:0/1356792461 >> 172.21.15.16:0/1356792461 conn(0x55db25a36460 msgr2=0x55db25df3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.559+0000 7f00ca551740 1 -- 172.21.15.16:0/1356792461 shutdown_connections 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.559+0000 7f00ca551740 1 -- 172.21.15.16:0/1356792461 wait complete. 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.559+0000 7f00ca551740 1 -- start start 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.559+0000 7f00ca551740 1 Processor -- start 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00ca551740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a191b0 0x55db25dfb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00ca551740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25df6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00ca551740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a42070 0x55db25df7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00ca551740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1b86e0 con 0x55db25a191b0 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00ca551740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1b8010 con 0x55db25a21390 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00ca551740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1b5270 con 0x55db25a42070 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00ca551740 10 client.0 _ll_register_callbacks cb 0x7f00ac00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00c1d78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a191b0 0x55db25dfb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00c1d78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a191b0 0x55db25dfb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36628/0 (socket says 172.21.15.16:36628) 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00c1d78640 1 -- 172.21.15.16:0/2291842971 learned_addr learned my addr 172.21.15.16:0/2291842971 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00c2579640 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a42070 0x55db25df7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00c1577640 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25df6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00c1d78640 1 -- 172.21.15.16:0/2291842971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a42070 msgr2=0x55db25df7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00c1d78640 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a42070 0x55db25df7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00c1d78640 1 -- 172.21.15.16:0/2291842971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 msgr2=0x55db25df6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00c1d78640 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25df6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.560+0000 7f00c1d78640 1 -- 172.21.15.16:0/2291842971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db25e6e6a0 con 0x55db25a191b0 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00c1d78640 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a191b0 0x55db25dfb790 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f00b8006010 tx=0x7f00b8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af99e461825ca9ff server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00a57fa640 1 -- 172.21.15.16:0/2291842971 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b8035040 con 0x55db25a191b0 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00a57fa640 1 -- 172.21.15.16:0/2291842971 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f00b8005040 con 0x55db25a191b0 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00ca551740 1 -- 172.21.15.16:0/2291842971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00b401d760 con 0x55db25a191b0 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00a57fa640 1 -- 172.21.15.16:0/2291842971 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f00b8054050 con 0x55db25a191b0 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00ca551740 10 client.18111 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00ca551740 1 -- 172.21.15.16:0/2291842971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55db25e6f840 con 0x55db25a191b0 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00a57fa640 1 -- 172.21.15.16:0/2291842971 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00b401d760 con 0x55db25a191b0 2024-08-28T19:31:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00a57fa640 1 -- 172.21.15.16:0/2291842971 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f00b80035f0 con 0x55db25a191b0 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00ca551740 10 client.18111 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00ca551740 10 client.18111 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00ca551740 20 client.18111 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00ca551740 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db259d5da0 0x55db25e72030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00ca551740 4 client.18111 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.561+0000 7f00ca551740 1 -- 172.21.15.16:0/2291842971 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55db25d5c020 con 0x55db259d5da0 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.562+0000 7f00c1577640 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db259d5da0 0x55db25e72030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.562+0000 7f00c1577640 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db259d5da0 0x55db25e72030 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d791a72f58ab0fd7 server_cookie=cc5ebb7d8d56cce6 in_seq=0 out_seq=0 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.562+0000 7f00a57fa640 10 client.18111 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.562+0000 7f00a57fa640 1 -- 172.21.15.16:0/2291842971 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4066 (crc 0 0 0) 0x55db25d5c020 con 0x55db259d5da0 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.562+0000 7f00a57fa640 10 client.18111 handle_command_reply: tid=0 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.562+0000 7f00ca551740 1 client.18111 shutdown 2024-08-28T19:31:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00ca551740 1 -- 172.21.15.16:0/2291842971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a191b0 msgr2=0x55db25dfb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00ca551740 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a191b0 0x55db25dfb790 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f00b8006010 tx=0x7f00b8004040 comp rx=0 tx=0).stop 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00ca551740 1 -- 172.21.15.16:0/2291842971 shutdown_connections 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00ca551740 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55db259d5da0 0x55db25e72030 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00ca551740 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55db25a42070 0x55db25df7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00ca551740 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db25a21390 0x55db25df6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00ca551740 1 --2- 172.21.15.16:0/2291842971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db25a191b0 0x55db25dfb790 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00a57fa640 0 client.18111 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00ca551740 1 -- 172.21.15.16:0/2291842971 >> 172.21.15.16:0/2291842971 conn(0x55db25a36460 msgr2=0x55db25df1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00ca551740 1 -- 172.21.15.16:0/2291842971 shutdown_connections 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00ca551740 1 -- 172.21.15.16:0/2291842971 wait complete. 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.563+0000 7f00ca551740 20 client.18111 trim_cache size 0 max 16384 2024-08-28T19:31:20.569 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:31:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:31:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:20.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:20.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:20.578 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:31:20.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 -- 172.21.15.16:0/948773270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00a80b4ca0 msgr2=0x7f00a80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:20.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00a80b4ca0 0x7f00a80b71b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f00b4020000 tx=0x7f00b401d000 comp rx=0 tx=0).stop 2024-08-28T19:31:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 -- 172.21.15.16:0/948773270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00bc102770 msgr2=0x7f00bc1a0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00bc102770 0x7f00bc1a0e30 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00e9f0 tx=0x7f00ac00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 -- 172.21.15.16:0/948773270 shutdown_connections 2024-08-28T19:31:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f00a80b4ca0 0x7f00a80b71b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc114260 0x7f00bc1a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00bc102770 0x7f00bc1a0e30 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 --2- 172.21.15.16:0/948773270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00bc101e20 0x7f00bc1a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 -- 172.21.15.16:0/948773270 >> 172.21.15.16:0/948773270 conn(0x7f00bc079c60 msgr2=0x7f00bc111b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 -- 172.21.15.16:0/948773270 shutdown_connections 2024-08-28T19:31:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:20.564+0000 7f00c3fdc640 1 -- 172.21.15.16:0/948773270 wait complete. 2024-08-28T19:31:20.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.90 MB/sec execute 420 sec latency 6134.280 ms 2024-08-28T19:31:21.264 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:31:21.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:31:21.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:21 smithi016 ceph-mon[18556]: pgmap v656: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 510 KiB/s wr, 78 op/s 2024-08-28T19:31:21.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:21 smithi028 ceph-mon[21769]: pgmap v656: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 510 KiB/s wr, 78 op/s 2024-08-28T19:31:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:21 smithi115 ceph-mon[22184]: pgmap v656: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 510 KiB/s wr, 78 op/s 2024-08-28T19:31:21.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.89 MB/sec execute 421 sec latency 7134.393 ms 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.035+0000 7ff56fed4640 1 -- 172.21.15.16:0/3660091980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff568102880 msgr2=0x7ff568102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.035+0000 7ff56fed4640 1 --2- 172.21.15.16:0/3660091980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff568102880 0x7ff568102c90 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7ff564009900 tx=0x7ff56402a410 comp rx=0 tx=0).stop 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.035+0000 7ff56fed4640 1 -- 172.21.15.16:0/3660091980 shutdown_connections 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.035+0000 7ff56fed4640 1 --2- 172.21.15.16:0/3660091980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff568108b10 0x7ff56810f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.035+0000 7ff56fed4640 1 --2- 172.21.15.16:0/3660091980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5681081c0 0x7ff5681085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.035+0000 7ff56fed4640 1 --2- 172.21.15.16:0/3660091980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff568102880 0x7ff568102c90 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.035+0000 7ff56fed4640 1 -- 172.21.15.16:0/3660091980 >> 172.21.15.16:0/3660091980 conn(0x7ff568079510 msgr2=0x7ff568079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.035+0000 7ff56fed4640 1 -- 172.21.15.16:0/3660091980 shutdown_connections 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.035+0000 7ff56fed4640 1 -- 172.21.15.16:0/3660091980 wait complete. 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56fed4640 1 Processor -- start 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56fed4640 1 -- start start 2024-08-28T19:31:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56fed4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff568102880 0x7ff56819b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56fed4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5681081c0 0x7ff56819b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56fed4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568108b10 0x7ff56819bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56fed4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff5681127e0 con 0x7ff5681081c0 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56fed4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff568112660 con 0x7ff568108b10 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56fed4640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff568112960 con 0x7ff568102880 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56d46f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5681081c0 0x7ff56819b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56d46f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5681081c0 0x7ff56819b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36656/0 (socket says 172.21.15.16:36656) 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56d46f640 1 -- 172.21.15.16:0/3841710426 learned_addr learned my addr 172.21.15.16:0/3841710426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56e471640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568108b10 0x7ff56819bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.036+0000 7ff56dc70640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff568102880 0x7ff56819b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff56d46f640 1 -- 172.21.15.16:0/3841710426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff568102880 msgr2=0x7ff56819b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff56d46f640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff568102880 0x7ff56819b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff56d46f640 1 -- 172.21.15.16:0/3841710426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568108b10 msgr2=0x7ff56819bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff56d46f640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568108b10 0x7ff56819bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff56d46f640 1 -- 172.21.15.16:0/3841710426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5681a63b0 con 0x7ff5681081c0 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff56dc70640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff568102880 0x7ff56819b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff56d46f640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5681081c0 0x7ff56819b630 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7ff55800e9f0 tx=0x7ff55800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff55effd640 1 -- 172.21.15.16:0/3841710426 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55800ccc0 con 0x7ff5681081c0 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff55effd640 1 -- 172.21.15.16:0/3841710426 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5580108e0 con 0x7ff5681081c0 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff56fed4640 1 -- 172.21.15.16:0/3841710426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5681a66a0 con 0x7ff5681081c0 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff55effd640 1 -- 172.21.15.16:0/3841710426 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558018860 con 0x7ff5681081c0 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.037+0000 7ff56fed4640 1 -- 172.21.15.16:0/3841710426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5681a6be0 con 0x7ff5681081c0 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.038+0000 7ff56fed4640 1 -- 172.21.15.16:0/3841710426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff568103b00 con 0x7ff5681081c0 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.040+0000 7ff55effd640 1 -- 172.21.15.16:0/3841710426 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff558010430 con 0x7ff5681081c0 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.041+0000 7ff55effd640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff5540b4ca0 0x7ff5540b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.041+0000 7ff56dc70640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff5540b4ca0 0x7ff5540b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.041+0000 7ff55effd640 1 -- 172.21.15.16:0/3841710426 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7ff5580e61a0 con 0x7ff5681081c0 2024-08-28T19:31:22.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.042+0000 7ff56dc70640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff5540b4ca0 0x7ff5540b71b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff564000c00 tx=0x7ff564031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:22.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.044+0000 7ff55effd640 1 -- 172.21.15.16:0/3841710426 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ff5580acb60 con 0x7ff5681081c0 2024-08-28T19:31:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.291+0000 7ff56fed4640 1 -- 172.21.15.16:0/3841710426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff56806bd30 con 0x7ff5681081c0 2024-08-28T19:31:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.291+0000 7ff55effd640 1 -- 172.21.15.16:0/3841710426 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11977 (secure 0 0 0) 0x7ff56806bd30 con 0x7ff5681081c0 2024-08-28T19:31:22.293 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:31:22.293 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":55,"btime":"2024-08-28T19:30:56:995905+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:30:56.593478+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:31:22.293 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 55 2024-08-28T19:31:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 -- 172.21.15.16:0/3841710426 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff5540b4ca0 msgr2=0x7ff5540b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff5540b4ca0 0x7ff5540b71b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff564000c00 tx=0x7ff564031400 comp rx=0 tx=0).stop 2024-08-28T19:31:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 -- 172.21.15.16:0/3841710426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5681081c0 msgr2=0x7ff56819b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5681081c0 0x7ff56819b630 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7ff55800e9f0 tx=0x7ff55800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:31:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 -- 172.21.15.16:0/3841710426 shutdown_connections 2024-08-28T19:31:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff5540b4ca0 0x7ff5540b71b0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568108b10 0x7ff56819bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5681081c0 0x7ff56819b630 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 --2- 172.21.15.16:0/3841710426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff568102880 0x7ff56819b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 -- 172.21.15.16:0/3841710426 >> 172.21.15.16:0/3841710426 conn(0x7ff568079510 msgr2=0x7ff5681062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 -- 172.21.15.16:0/3841710426 shutdown_connections 2024-08-28T19:31:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:22.294+0000 7ff56fed4640 1 -- 172.21.15.16:0/3841710426 wait complete. 2024-08-28T19:31:22.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:31:22.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.88 MB/sec execute 422 sec latency 8134.430 ms 2024-08-28T19:31:23.172 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:23.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:23 smithi028 ceph-mon[21769]: pgmap v657: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 470 KiB/s wr, 74 op/s 2024-08-28T19:31:23.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:23 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3841710426' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:23.560 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:23 smithi016 ceph-mon[18556]: pgmap v657: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 470 KiB/s wr, 74 op/s 2024-08-28T19:31:23.560 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:23 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3841710426' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:23 smithi115 ceph-mon[22184]: pgmap v657: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 470 KiB/s wr, 74 op/s 2024-08-28T19:31:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:23 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3841710426' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:23.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.650+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/499720797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc4104800 msgr2=0x7f2fc4104c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.650+0000 7f2fcbc1d640 1 --2- 172.21.15.16:0/499720797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc4104800 0x7f2fc4104c10 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009900 tx=0x7f2fc002a410 comp rx=0 tx=0).stop 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.650+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/499720797 shutdown_connections 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.650+0000 7f2fcbc1d640 1 --2- 172.21.15.16:0/499720797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc410dc60 0x7f2fc4110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.650+0000 7f2fcbc1d640 1 --2- 172.21.15.16:0/499720797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fc4105150 0x7f2fc410d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.650+0000 7f2fcbc1d640 1 --2- 172.21.15.16:0/499720797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc4104800 0x7f2fc4104c10 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.650+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/499720797 >> 172.21.15.16:0/499720797 conn(0x7f2fc4079470 msgr2=0x7f2fc4079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.650+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/499720797 shutdown_connections 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.650+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/499720797 wait complete. 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.651+0000 7f2fcbc1d640 1 Processor -- start 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.651+0000 7f2fcbc1d640 1 -- start start 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.651+0000 7f2fcbc1d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fc4104800 0x7f2fc419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.651+0000 7f2fcbc1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc4105150 0x7f2fc419a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.651+0000 7f2fcbc1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc410dc60 0x7f2fc419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.651+0000 7f2fcbc1d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc41138e0 con 0x7f2fc4105150 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.651+0000 7f2fcbc1d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc4113760 con 0x7f2fc410dc60 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.651+0000 7f2fcbc1d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc4113a60 con 0x7f2fc4104800 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fc91b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc4105150 0x7f2fc419a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fc91b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc4105150 0x7f2fc419a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36668/0 (socket says 172.21.15.16:36668) 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fc91b8640 1 -- 172.21.15.16:0/2676342911 learned_addr learned my addr 172.21.15.16:0/2676342911 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fc99b9640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fc4104800 0x7f2fc419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc410dc60 0x7f2fc419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fc91b8640 1 -- 172.21.15.16:0/2676342911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fc4104800 msgr2=0x7f2fc419a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fc91b8640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fc4104800 0x7f2fc419a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fc91b8640 1 -- 172.21.15.16:0/2676342911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc410dc60 msgr2=0x7f2fc419fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fc91b8640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc410dc60 0x7f2fc419fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fc91b8640 1 -- 172.21.15.16:0/2676342911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc41a01e0 con 0x7f2fc4105150 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fc91b8640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc4105150 0x7f2fc419a8d0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400c970 tx=0x7f2fb400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fbaffd640 1 -- 172.21.15.16:0/2676342911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4004220 con 0x7f2fc4105150 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/2676342911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc41a04d0 con 0x7f2fc4105150 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fbaffd640 1 -- 172.21.15.16:0/2676342911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fb40043c0 con 0x7f2fc4105150 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fbaffd640 1 -- 172.21.15.16:0/2676342911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4004a00 con 0x7f2fc4105150 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.652+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/2676342911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc41a6cf0 con 0x7f2fc4105150 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.655+0000 7f2fbaffd640 1 -- 172.21.15.16:0/2676342911 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2fb40079b0 con 0x7f2fc4105150 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.655+0000 7f2fd2192740 1 Processor -- start 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.655+0000 7f2fd2192740 1 -- start start 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d7a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d7a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d7ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fd2192740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561e62cf1680 con 0x561e629d5070 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fd2192740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e62ce0c10 con 0x561e629b4390 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fd2192740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561e62ce4bc0 con 0x561e629ac1b0 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fca1ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d7a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fca1ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d7a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36676/0 (socket says 172.21.15.16:36676) 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fca1ba640 1 -- 172.21.15.16:0/2580046155 learned_addr learned my addr 172.21.15.16:0/2580046155 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fc91b8640 1 --2- 172.21.15.16:0/2580046155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d7ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fc99b9640 1 --2- 172.21.15.16:0/2580046155 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d7a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fbaffd640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2fb00b4cd0 0x7f2fb00b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fca1ba640 1 -- 172.21.15.16:0/2580046155 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 msgr2=0x561e62d7a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/2580046155 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d7a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fca1ba640 1 -- 172.21.15.16:0/2580046155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 msgr2=0x561e62d7ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/2580046155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d7ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fca1ba640 1 -- 172.21.15.16:0/2580046155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e62a40650 con 0x561e629d5070 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fbaffd640 1 -- 172.21.15.16:0/2676342911 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f2fb40e5590 con 0x7f2fc4105150 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.656+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2fb00b4cd0 0x7f2fb00b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.657+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/2580046155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d7a310 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc007f40 tx=0x7f2fbc00f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf51aeff7a56656 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.657+0000 7f2faffff640 1 -- 172.21.15.16:0/2580046155 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc010b40 con 0x561e629d5070 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.657+0000 7f2faffff640 1 -- 172.21.15.16:0/2580046155 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fbc01add0 con 0x561e629d5070 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.657+0000 7f2faffff640 1 -- 172.21.15.16:0/2580046155 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc019ab0 con 0x561e629d5070 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fd2192740 1 -- 172.21.15.16:0/2580046155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 msgr2=0x561e62d7a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fd2192740 1 --2- 172.21.15.16:0/2580046155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d7a310 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc007f40 tx=0x7f2fbc00f850 comp rx=0 tx=0).stop 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2fb00b4cd0 0x7f2fb00b71e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc007fd0 tx=0x7f2fbc01f000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:23.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fd2192740 1 -- 172.21.15.16:0/2580046155 shutdown_connections 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fd2192740 1 --2- 172.21.15.16:0/2580046155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d7ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fd2192740 1 --2- 172.21.15.16:0/2580046155 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d7a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fd2192740 1 --2- 172.21.15.16:0/2580046155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d7a310 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fd2192740 1 -- 172.21.15.16:0/2580046155 >> 172.21.15.16:0/2580046155 conn(0x561e629c9460 msgr2=0x561e62d78200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fd2192740 1 -- 172.21.15.16:0/2580046155 shutdown_connections 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fd2192740 1 -- 172.21.15.16:0/2580046155 wait complete. 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fd2192740 1 -- start start 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.658+0000 7f2fd2192740 1 Processor -- start 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629ac1b0 0x561e62d85ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629b4390 0x561e62d86000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629d5070 0x561e62d91de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fd2192740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561e62b43700 con 0x561e629ac1b0 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fd2192740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e62c67750 con 0x561e629d5070 2024-08-28T19:31:23.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fd2192740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561e62d88af0 con 0x561e629b4390 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fd2192740 10 client.0 _ll_register_callbacks cb 0x561e62c96e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fc99b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629ac1b0 0x561e62d85ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fc99b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629ac1b0 0x561e62d85ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36690/0 (socket says 172.21.15.16:36690) 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fc99b9640 1 -- 172.21.15.16:0/1392336334 learned_addr learned my addr 172.21.15.16:0/1392336334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629d5070 0x561e62d91de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fc91b8640 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629b4390 0x561e62d86000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fc99b9640 1 -- 172.21.15.16:0/1392336334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629b4390 msgr2=0x561e62d86000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fc99b9640 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629b4390 0x561e62d86000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fc99b9640 1 -- 172.21.15.16:0/1392336334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629d5070 msgr2=0x561e62d91de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fc99b9640 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629d5070 0x561e62d91de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fc99b9640 1 -- 172.21.15.16:0/1392336334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e62ca0900 con 0x561e629ac1b0 2024-08-28T19:31:23.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.659+0000 7f2fc91b8640 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629b4390 0x561e62d86000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fc99b9640 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629ac1b0 0x561e62d85ac0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009900 tx=0x7f2fc00316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fc2918cfee0f91 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fad7fa640 1 -- 172.21.15.16:0/1392336334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0035040 con 0x561e629ac1b0 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fad7fa640 1 -- 172.21.15.16:0/1392336334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f2fc0004040 con 0x561e629ac1b0 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fd2192740 1 -- 172.21.15.16:0/1392336334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fbc01add0 con 0x561e629ac1b0 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fad7fa640 1 -- 172.21.15.16:0/1392336334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f2fc004d050 con 0x561e629ac1b0 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fd2192740 10 client.18129 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fd2192740 1 -- 172.21.15.16:0/1392336334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x561e62d948f0 con 0x561e629ac1b0 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fad7fa640 1 -- 172.21.15.16:0/1392336334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fbc01add0 con 0x561e629ac1b0 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fad7fa640 1 -- 172.21.15.16:0/1392336334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f2fc003c440 con 0x561e629ac1b0 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fd2192740 10 client.18129 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fd2192740 10 client.18129 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:31:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fd2192740 20 client.18129 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.660+0000 7f2fd2192740 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561e629939f0 0x561e62d97e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.661+0000 7f2fd2192740 4 client.18129 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.661+0000 7f2fd2192740 1 -- 172.21.15.16:0/1392336334 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e62cf35d0 con 0x561e629939f0 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.661+0000 7f2fc91b8640 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561e629939f0 0x561e62d97e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.661+0000 7f2fad7fa640 10 client.18129 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.661+0000 7f2fc91b8640 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561e629939f0 0x561e62d97e70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bd33932c953b6146 server_cookie=a62cdfacff70e6a7 in_seq=0 out_seq=0 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fad7fa640 1 -- 172.21.15.16:0/1392336334 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561e62cf35d0 con 0x561e629939f0 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fad7fa640 10 client.18129 handle_command_reply: tid=0 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fd2192740 1 client.18129 shutdown 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fd2192740 1 -- 172.21.15.16:0/1392336334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629ac1b0 msgr2=0x561e62d85ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fd2192740 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629ac1b0 0x561e62d85ac0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009900 tx=0x7f2fc00316d0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fd2192740 1 -- 172.21.15.16:0/1392336334 shutdown_connections 2024-08-28T19:31:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fd2192740 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561e629939f0 0x561e62d97e70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fd2192740 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629d5070 0x561e62d91de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fd2192740 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629b4390 0x561e62d86000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fad7fa640 0 client.18129 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fd2192740 1 --2- 172.21.15.16:0/1392336334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629ac1b0 0x561e62d85ac0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.662+0000 7f2fd2192740 1 -- 172.21.15.16:0/1392336334 >> 172.21.15.16:0/1392336334 conn(0x561e629c9460 msgr2=0x561e62d7d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.663+0000 7f2fd2192740 1 -- 172.21.15.16:0/1392336334 shutdown_connections 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.663+0000 7f2fd2192740 1 -- 172.21.15.16:0/1392336334 wait complete. 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.663+0000 7f2fd2192740 20 client.18129 trim_cache size 0 max 16384 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fd2192740 1 Processor -- start 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fd2192740 1 -- start start 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d7db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d82cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d831f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fd2192740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc41b0d40 con 0x561e629d5070 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fd2192740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc41b2880 con 0x561e629b4390 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fd2192740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc41b1410 con 0x561e629ac1b0 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fca1ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d831f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fca1ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d831f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36700/0 (socket says 172.21.15.16:36700) 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fca1ba640 1 -- 172.21.15.16:0/3698588465 learned_addr learned my addr 172.21.15.16:0/3698588465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fc91b8640 1 --2- 172.21.15.16:0/3698588465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d82cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fca1ba640 1 -- 172.21.15.16:0/3698588465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 msgr2=0x561e62d7db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fc99b9640 1 --2- 172.21.15.16:0/3698588465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d7db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/3698588465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d7db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fca1ba640 1 -- 172.21.15.16:0/3698588465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 msgr2=0x561e62d82cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/3698588465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d82cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fca1ba640 1 -- 172.21.15.16:0/3698588465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e62d7e060 con 0x561e629d5070 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.676+0000 7f2fc99b9640 1 --2- 172.21.15.16:0/3698588465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d7db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.677+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/3698588465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d831f0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc007f10 tx=0x7f2fbc025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab4ba4a7395cc55 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.677+0000 7f2faf7fe640 1 -- 172.21.15.16:0/3698588465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc019ab0 con 0x561e629d5070 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.677+0000 7f2faf7fe640 1 -- 172.21.15.16:0/3698588465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fbc035070 con 0x561e629d5070 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.677+0000 7f2faf7fe640 1 -- 172.21.15.16:0/3698588465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc019490 con 0x561e629d5070 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.677+0000 7f2fd2192740 1 -- 172.21.15.16:0/3698588465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 msgr2=0x561e62d831f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.677+0000 7f2fd2192740 1 --2- 172.21.15.16:0/3698588465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d831f0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc007f10 tx=0x7f2fbc025000 comp rx=0 tx=0).stop 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.678+0000 7f2fd2192740 1 -- 172.21.15.16:0/3698588465 shutdown_connections 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.678+0000 7f2fd2192740 1 --2- 172.21.15.16:0/3698588465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62d831f0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.678+0000 7f2fd2192740 1 --2- 172.21.15.16:0/3698588465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d82cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.678+0000 7f2fd2192740 1 --2- 172.21.15.16:0/3698588465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d7db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.678+0000 7f2fd2192740 1 -- 172.21.15.16:0/3698588465 >> 172.21.15.16:0/3698588465 conn(0x561e629c9460 msgr2=0x561e62d7d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.678+0000 7f2fd2192740 1 -- 172.21.15.16:0/3698588465 shutdown_connections 2024-08-28T19:31:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.678+0000 7f2fd2192740 1 -- 172.21.15.16:0/3698588465 wait complete. 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.678+0000 7f2fd2192740 1 -- start start 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.678+0000 7f2fd2192740 1 Processor -- start 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d8a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d8ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fd2192740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62e01870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fd2192740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc41b21b0 con 0x561e629d5070 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fd2192740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc41b1ae0 con 0x561e629b4390 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fd2192740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc41aed40 con 0x561e629ac1b0 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fd2192740 10 client.0 _ll_register_callbacks cb 0x7f2fc0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fca1ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62e01870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fca1ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62e01870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36702/0 (socket says 172.21.15.16:36702) 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fca1ba640 1 -- 172.21.15.16:0/3165221507 learned_addr learned my addr 172.21.15.16:0/3165221507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fc91b8640 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d8ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fc99b9640 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d8a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fca1ba640 1 -- 172.21.15.16:0/3165221507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 msgr2=0x561e62d8a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d8a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fca1ba640 1 -- 172.21.15.16:0/3165221507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 msgr2=0x561e62d8ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d8ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fca1ba640 1 -- 172.21.15.16:0/3165221507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e62e01ee0 con 0x561e629d5070 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fc91b8640 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d8ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fca1ba640 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62e01870 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc01a000 tx=0x7f2fbc007c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89ea4374be74d3b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fad7fa640 1 -- 172.21.15.16:0/3165221507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc003970 con 0x561e629d5070 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fad7fa640 1 -- 172.21.15.16:0/3165221507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f2fbc003b10 con 0x561e629d5070 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.679+0000 7f2fd2192740 1 -- 172.21.15.16:0/3165221507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fbc035070 con 0x561e629d5070 2024-08-28T19:31:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fad7fa640 1 -- 172.21.15.16:0/3165221507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f2fbc003d30 con 0x561e629d5070 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fd2192740 10 client.18135 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fd2192740 1 -- 172.21.15.16:0/3165221507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x561e62e030b0 con 0x561e629d5070 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fad7fa640 1 -- 172.21.15.16:0/3165221507 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fbc035070 con 0x561e629d5070 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fad7fa640 1 -- 172.21.15.16:0/3165221507 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f2fbc019860 con 0x561e629d5070 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fd2192740 10 client.18135 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fd2192740 10 client.18135 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fd2192740 20 client.18135 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fd2192740 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561e62968da0 0x561e62e05590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fd2192740 4 client.18135 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fd2192740 1 -- 172.21.15.16:0/3165221507 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561e62cef070 con 0x561e62968da0 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.680+0000 7f2fc99b9640 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561e62968da0 0x561e62e05590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fad7fa640 10 client.18135 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:31:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fc99b9640 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561e62968da0 0x561e62e05590 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cd1478fec1d897ea server_cookie=e41a811bffbe6235 in_seq=0 out_seq=0 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fad7fa640 1 -- 172.21.15.16:0/3165221507 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561e62cef070 con 0x561e62968da0 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fad7fa640 10 client.18135 handle_command_reply: tid=0 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 1 client.18135 shutdown 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 1 -- 172.21.15.16:0/3165221507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 msgr2=0x561e62e01870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62e01870 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc01a000 tx=0x7f2fbc007c60 comp rx=0 tx=0).stop 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 1 -- 172.21.15.16:0/3165221507 shutdown_connections 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561e62968da0 0x561e62e05590 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e629d5070 0x561e62e01870 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e629b4390 0x561e62d8ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fad7fa640 0 client.18135 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 1 --2- 172.21.15.16:0/3165221507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e629ac1b0 0x561e62d8a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 1 -- 172.21.15.16:0/3165221507 >> 172.21.15.16:0/3165221507 conn(0x561e629c9460 msgr2=0x561e62d7b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:31:23.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 1 -- 172.21.15.16:0/3165221507 shutdown_connections 2024-08-28T19:31:23.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 1 -- 172.21.15.16:0/3165221507 wait complete. 2024-08-28T19:31:23.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.681+0000 7f2fd2192740 20 client.18135 trim_cache size 0 max 16384 2024-08-28T19:31:23.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/2676342911 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2fb00b4cd0 msgr2=0x7f2fb00b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:23.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2fb00b4cd0 0x7f2fb00b71e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc007fd0 tx=0x7f2fbc01f000 comp rx=0 tx=0).stop 2024-08-28T19:31:23.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/2676342911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc4105150 msgr2=0x7f2fc419a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:23.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc4105150 0x7f2fc419a8d0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400c970 tx=0x7f2fb400ce30 comp rx=0 tx=0).stop 2024-08-28T19:31:23.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/2676342911 shutdown_connections 2024-08-28T19:31:23.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2fb00b4cd0 0x7f2fb00b71e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc410dc60 0x7f2fc419fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc4105150 0x7f2fc419a8d0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 --2- 172.21.15.16:0/2676342911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fc4104800 0x7f2fc419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:23.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/2676342911 >> 172.21.15.16:0/2676342911 conn(0x7f2fc4079470 msgr2=0x7f2fc41077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:23.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/2676342911 shutdown_connections 2024-08-28T19:31:23.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:23.682+0000 7f2fcbc1d640 1 -- 172.21.15.16:0/2676342911 wait complete. 2024-08-28T19:31:23.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.87 MB/sec execute 423 sec latency 9134.495 ms 2024-08-28T19:31:24.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:31:24.768 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:24.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.86 MB/sec execute 424 sec latency 10134.588 ms 2024-08-28T19:31:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.386+0000 7fafdfeaa640 1 -- 172.21.15.16:0/2548517510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8105880 msgr2=0x7fafd810f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.386+0000 7fafdfeaa640 1 --2- 172.21.15.16:0/2548517510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8105880 0x7fafd810f2d0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009ca0 tx=0x7fafd002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.386+0000 7fafdfeaa640 1 -- 172.21.15.16:0/2548517510 shutdown_connections 2024-08-28T19:31:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.386+0000 7fafdfeaa640 1 --2- 172.21.15.16:0/2548517510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8105880 0x7fafd810f2d0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.386+0000 7fafdfeaa640 1 --2- 172.21.15.16:0/2548517510 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fafd8104f30 0x7fafd8105340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.386+0000 7fafdfeaa640 1 --2- 172.21.15.16:0/2548517510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd8102f60 0x7fafd8103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.386+0000 7fafdfeaa640 1 -- 172.21.15.16:0/2548517510 >> 172.21.15.16:0/2548517510 conn(0x7fafd8077a00 msgr2=0x7fafd8077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.386+0000 7fafdfeaa640 1 -- 172.21.15.16:0/2548517510 shutdown_connections 2024-08-28T19:31:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.386+0000 7fafdfeaa640 1 -- 172.21.15.16:0/2548517510 wait complete. 2024-08-28T19:31:25.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafdfeaa640 1 Processor -- start 2024-08-28T19:31:25.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafdfeaa640 1 -- start start 2024-08-28T19:31:25.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafdfeaa640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fafd8102f60 0x7fafd819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafdfeaa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd8104f30 0x7fafd819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafdfeaa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8105880 0x7fafd819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafdfeaa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8113840 con 0x7fafd8105880 2024-08-28T19:31:25.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafdfeaa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81136c0 con 0x7fafd8104f30 2024-08-28T19:31:25.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafdfeaa640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81139c0 con 0x7fafd8102f60 2024-08-28T19:31:25.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafde447640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8105880 0x7fafd819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafde447640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8105880 0x7fafd819fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32830/0 (socket says 172.21.15.16:32830) 2024-08-28T19:31:25.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafde447640 1 -- 172.21.15.16:0/1566808115 learned_addr learned my addr 172.21.15.16:0/1566808115 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:25.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafddc46640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fafd8102f60 0x7fafd819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.387+0000 7fafdd445640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd8104f30 0x7fafd819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.388+0000 7fafde447640 1 -- 172.21.15.16:0/1566808115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fafd8102f60 msgr2=0x7fafd819a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.388+0000 7fafde447640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fafd8102f60 0x7fafd819a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.388+0000 7fafde447640 1 -- 172.21.15.16:0/1566808115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd8104f30 msgr2=0x7fafd819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.388+0000 7fafde447640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd8104f30 0x7fafd819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.388+0000 7fafde447640 1 -- 172.21.15.16:0/1566808115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd81a0120 con 0x7fafd8105880 2024-08-28T19:31:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.388+0000 7fafde447640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8105880 0x7fafd819fab0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fafd0005cc0 tx=0x7fafd002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.388+0000 7fafceffd640 1 -- 172.21.15.16:0/1566808115 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0003bf0 con 0x7fafd8105880 2024-08-28T19:31:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.388+0000 7fafdfeaa640 1 -- 172.21.15.16:0/1566808115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd81a03b0 con 0x7fafd8105880 2024-08-28T19:31:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.388+0000 7fafceffd640 1 -- 172.21.15.16:0/1566808115 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafd0003d90 con 0x7fafd8105880 2024-08-28T19:31:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.388+0000 7fafceffd640 1 -- 172.21.15.16:0/1566808115 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd002fb20 con 0x7fafd8105880 2024-08-28T19:31:25.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.388+0000 7fafdfeaa640 1 -- 172.21.15.16:0/1566808115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd81a08f0 con 0x7fafd8105880 2024-08-28T19:31:25.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.391+0000 7fafe641f740 1 Processor -- start 2024-08-28T19:31:25.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.391+0000 7fafceffd640 1 -- 172.21.15.16:0/1566808115 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fafd0004580 con 0x7fafd8105880 2024-08-28T19:31:25.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafe641f740 1 -- start start 2024-08-28T19:31:25.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafe641f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d78070 0x561ee811d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafe641f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d4f1b0 0x561ee811d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafe641f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d57390 0x561ee811ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafe641f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ee8094680 con 0x561ee7d78070 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafe641f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ee8083c10 con 0x561ee7d57390 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafe641f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561ee8087bc0 con 0x561ee7d4f1b0 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafdd445640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d78070 0x561ee811d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafdd445640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d78070 0x561ee811d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32834/0 (socket says 172.21.15.16:32834) 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafdd445640 1 -- 172.21.15.16:0/1306627743 learned_addr learned my addr 172.21.15.16:0/1306627743 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafddc46640 1 --2- 172.21.15.16:0/1306627743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d4f1b0 0x561ee811d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafdd445640 1 --2- 172.21.15.16:0/1306627743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d57390 0x561ee811ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafceffd640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fafc40b4cd0 0x7fafc40b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafdd445640 1 -- 172.21.15.16:0/1306627743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d4f1b0 msgr2=0x561ee811d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafdd445640 1 --2- 172.21.15.16:0/1306627743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d4f1b0 0x561ee811d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafdd445640 1 -- 172.21.15.16:0/1306627743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d57390 msgr2=0x561ee811ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.392+0000 7fafdd445640 1 --2- 172.21.15.16:0/1306627743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d57390 0x561ee811ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.393+0000 7fafdd445640 1 -- 172.21.15.16:0/1306627743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ee7de3650 con 0x561ee7d78070 2024-08-28T19:31:25.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.393+0000 7fafddc46640 1 --2- 172.21.15.16:0/1306627743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d4f1b0 0x561ee811d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.393+0000 7fafde447640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fafc40b4cd0 0x7fafc40b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.393+0000 7fafceffd640 1 -- 172.21.15.16:0/1566808115 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fafd0106c90 con 0x7fafd8105880 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.393+0000 7fafdd445640 1 --2- 172.21.15.16:0/1306627743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d78070 0x561ee811d350 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fafc800be80 tx=0x7fafc800b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95516306cf3599bb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.393+0000 7fafc3fff640 1 -- 172.21.15.16:0/1306627743 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8011e50 con 0x561ee7d78070 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.393+0000 7fafc3fff640 1 -- 172.21.15.16:0/1306627743 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc8005bf0 con 0x561ee7d78070 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.393+0000 7fafc3fff640 1 -- 172.21.15.16:0/1306627743 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8012810 con 0x561ee7d78070 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.394+0000 7fafe641f740 1 -- 172.21.15.16:0/1306627743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d78070 msgr2=0x561ee811d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.394+0000 7fafe641f740 1 --2- 172.21.15.16:0/1306627743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d78070 0x561ee811d350 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fafc800be80 tx=0x7fafc800b5c0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.394+0000 7fafde447640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fafc40b4cd0 0x7fafc40b71e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fafd01040c0 tx=0x7fafd002a8c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.394+0000 7fafe641f740 1 -- 172.21.15.16:0/1306627743 shutdown_connections 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.394+0000 7fafe641f740 1 --2- 172.21.15.16:0/1306627743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d57390 0x561ee811ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.394+0000 7fafe641f740 1 --2- 172.21.15.16:0/1306627743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d4f1b0 0x561ee811d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.394+0000 7fafe641f740 1 --2- 172.21.15.16:0/1306627743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d78070 0x561ee811d350 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.394+0000 7fafe641f740 1 -- 172.21.15.16:0/1306627743 >> 172.21.15.16:0/1306627743 conn(0x561ee7d6c460 msgr2=0x561ee811b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.394+0000 7fafe641f740 1 -- 172.21.15.16:0/1306627743 shutdown_connections 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.394+0000 7fafe641f740 1 -- 172.21.15.16:0/1306627743 wait complete. 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.394+0000 7fafe641f740 1 -- start start 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafe641f740 1 Processor -- start 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafe641f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d4f1b0 0x561ee8128b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafe641f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8129040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafe641f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d78070 0x561ee8134dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafe641f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ee7ee6700 con 0x561ee7d57390 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafe641f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ee800a750 con 0x561ee7d78070 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafe641f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561ee812bad0 con 0x561ee7d4f1b0 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafe641f740 10 client.0 _ll_register_callbacks cb 0x561ee8039e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafddc46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8129040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafddc46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8129040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32840/0 (socket says 172.21.15.16:32840) 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafddc46640 1 -- 172.21.15.16:0/488065977 learned_addr learned my addr 172.21.15.16:0/488065977 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafdd445640 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d78070 0x561ee8134dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.395+0000 7fafdd445640 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d4f1b0 0x561ee8128b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafddc46640 1 -- 172.21.15.16:0/488065977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d4f1b0 msgr2=0x561ee8128b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafddc46640 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d4f1b0 0x561ee8128b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafddc46640 1 -- 172.21.15.16:0/488065977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d78070 msgr2=0x561ee8134dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafddc46640 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d78070 0x561ee8134dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafddc46640 1 -- 172.21.15.16:0/488065977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ee8043900 con 0x561ee7d57390 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafddc46640 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8129040 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fafd40127d0 tx=0x7fafd4012c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0697d5e39e2b20 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafc17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafc17fa640 1 -- 172.21.15.16:0/488065977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd401f060 con 0x561ee7d57390 2024-08-28T19:31:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafc17fa640 1 -- 172.21.15.16:0/488065977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fafd4010dc0 con 0x561ee7d57390 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafe641f740 1 -- 172.21.15.16:0/488065977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561ee80965d0 con 0x561ee7d57390 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafc17fa640 1 -- 172.21.15.16:0/488065977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fafd4026050 con 0x561ee7d57390 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafe641f740 10 client.18147 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafe641f740 1 -- 172.21.15.16:0/488065977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x561ee8137930 con 0x561ee7d57390 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.396+0000 7fafc17fa640 1 -- 172.21.15.16:0/488065977 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561ee80965d0 con 0x561ee7d57390 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.397+0000 7fafc17fa640 1 -- 172.21.15.16:0/488065977 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7fafd400d500 con 0x561ee7d57390 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.397+0000 7fafe641f740 10 client.18147 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.397+0000 7fafe641f740 10 client.18147 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.397+0000 7fafe641f740 20 client.18147 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.397+0000 7fafe641f740 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561ee7d369f0 0x561ee813ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.397+0000 7fafe641f740 4 client.18147 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.397+0000 7fafe641f740 1 -- 172.21.15.16:0/488065977 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ee8092070 con 0x561ee7d369f0 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.397+0000 7fafdd445640 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561ee7d369f0 0x561ee813ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.397+0000 7fafc17fa640 10 client.18147 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.397+0000 7fafdd445640 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561ee7d369f0 0x561ee813ae80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4edd1de47237e90b server_cookie=8542d3b437a8a244 in_seq=0 out_seq=0 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.398+0000 7fafc17fa640 1 -- 172.21.15.16:0/488065977 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561ee8092070 con 0x561ee7d369f0 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.398+0000 7fafc17fa640 10 client.18147 handle_command_reply: tid=0 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.398+0000 7fafe641f740 1 client.18147 shutdown 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.398+0000 7fafe641f740 1 -- 172.21.15.16:0/488065977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 msgr2=0x561ee8129040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.398+0000 7fafe641f740 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8129040 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fafd40127d0 tx=0x7fafd4012c90 comp rx=0 tx=0).stop 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.399+0000 7fafe641f740 1 -- 172.21.15.16:0/488065977 shutdown_connections 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.399+0000 7fafe641f740 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561ee7d369f0 0x561ee813ae80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.399+0000 7fafe641f740 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d78070 0x561ee8134dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.399+0000 7fafe641f740 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8129040 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.399+0000 7fafe641f740 1 --2- 172.21.15.16:0/488065977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d4f1b0 0x561ee8128b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.399+0000 7fafe641f740 1 -- 172.21.15.16:0/488065977 >> 172.21.15.16:0/488065977 conn(0x561ee7d6c460 msgr2=0x561ee8120530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.399+0000 7fafc17fa640 0 client.18147 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:31:25.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.399+0000 7fafe641f740 1 -- 172.21.15.16:0/488065977 shutdown_connections 2024-08-28T19:31:25.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.399+0000 7fafe641f740 1 -- 172.21.15.16:0/488065977 wait complete. 2024-08-28T19:31:25.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.399+0000 7fafe641f740 20 client.18147 trim_cache size 0 max 16384 2024-08-28T19:31:25.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.412+0000 7fafe641f740 1 Processor -- start 2024-08-28T19:31:25.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.412+0000 7fafe641f740 1 -- start start 2024-08-28T19:31:25.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:25 smithi016 ceph-mon[18556]: pgmap v658: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 480 KiB/s wr, 70 op/s 2024-08-28T19:31:25.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.412+0000 7fafe641f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d4f1b0 0x561ee8120b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.412+0000 7fafe641f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8134cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.412+0000 7fafe641f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d78070 0x561ee8135200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.412+0000 7fafe641f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81b0c40 con 0x561ee7d57390 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.412+0000 7fafe641f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81b2780 con 0x561ee7d4f1b0 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.412+0000 7fafe641f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81b1310 con 0x561ee7d78070 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafddc46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8134cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafddc46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8134cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32846/0 (socket says 172.21.15.16:32846) 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafddc46640 1 -- 172.21.15.16:0/6594456 learned_addr learned my addr 172.21.15.16:0/6594456 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafdd445640 1 --2- 172.21.15.16:0/6594456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d4f1b0 0x561ee8120b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafddc46640 1 -- 172.21.15.16:0/6594456 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d78070 msgr2=0x561ee8135200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafdd445640 1 --2- 172.21.15.16:0/6594456 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d78070 0x561ee8135200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafddc46640 1 --2- 172.21.15.16:0/6594456 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d78070 0x561ee8135200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafddc46640 1 -- 172.21.15.16:0/6594456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d4f1b0 msgr2=0x561ee8120b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafddc46640 1 --2- 172.21.15.16:0/6594456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d4f1b0 0x561ee8120b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafddc46640 1 -- 172.21.15.16:0/6594456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ee811f640 con 0x561ee7d57390 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafdd445640 1 --2- 172.21.15.16:0/6594456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d4f1b0 0x561ee8120b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafddc46640 1 --2- 172.21.15.16:0/6594456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8134cc0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fafd400ac40 tx=0x7fafd4005870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf82b11e5c6d33a9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafc37fe640 1 -- 172.21.15.16:0/6594456 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd402e020 con 0x561ee7d57390 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafc37fe640 1 -- 172.21.15.16:0/6594456 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafd400dd40 con 0x561ee7d57390 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.413+0000 7fafc37fe640 1 -- 172.21.15.16:0/6594456 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd40055f0 con 0x561ee7d57390 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.414+0000 7fafe641f740 1 -- 172.21.15.16:0/6594456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 msgr2=0x561ee8134cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.414+0000 7fafe641f740 1 --2- 172.21.15.16:0/6594456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8134cc0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fafd400ac40 tx=0x7fafd4005870 comp rx=0 tx=0).stop 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.414+0000 7fafe641f740 1 -- 172.21.15.16:0/6594456 shutdown_connections 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.414+0000 7fafe641f740 1 --2- 172.21.15.16:0/6594456 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d78070 0x561ee8135200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.414+0000 7fafe641f740 1 --2- 172.21.15.16:0/6594456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d57390 0x561ee8134cc0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.414+0000 7fafe641f740 1 --2- 172.21.15.16:0/6594456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d4f1b0 0x561ee8120b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.414+0000 7fafe641f740 1 -- 172.21.15.16:0/6594456 >> 172.21.15.16:0/6594456 conn(0x561ee7d6c460 msgr2=0x561ee8120450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:25.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.414+0000 7fafe641f740 1 -- 172.21.15.16:0/6594456 shutdown_connections 2024-08-28T19:31:25.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.414+0000 7fafe641f740 1 -- 172.21.15.16:0/6594456 wait complete. 2024-08-28T19:31:25.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafe641f740 1 -- start start 2024-08-28T19:31:25.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafe641f740 1 Processor -- start 2024-08-28T19:31:25.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafe641f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d4f1b0 0x561ee812d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafe641f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d57390 0x561ee812d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafe641f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d78070 0x561ee81a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafe641f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81b20b0 con 0x561ee7d4f1b0 2024-08-28T19:31:25.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafe641f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81b19e0 con 0x561ee7d57390 2024-08-28T19:31:25.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafe641f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81aec40 con 0x561ee7d78070 2024-08-28T19:31:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafe641f740 10 client.0 _ll_register_callbacks cb 0x7fafd400a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafdd445640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d4f1b0 0x561ee812d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafdd445640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d4f1b0 0x561ee812d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32856/0 (socket says 172.21.15.16:32856) 2024-08-28T19:31:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.415+0000 7fafdd445640 1 -- 172.21.15.16:0/2636490977 learned_addr learned my addr 172.21.15.16:0/2636490977 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafddc46640 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d57390 0x561ee812d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafdd445640 1 -- 172.21.15.16:0/2636490977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d78070 msgr2=0x561ee81a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafdd445640 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d78070 0x561ee81a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafdd445640 1 -- 172.21.15.16:0/2636490977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d57390 msgr2=0x561ee812d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafdd445640 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d57390 0x561ee812d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafdd445640 1 -- 172.21.15.16:0/2636490977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ee81a4d20 con 0x561ee7d4f1b0 2024-08-28T19:31:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafdd445640 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d4f1b0 0x561ee812d0d0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fafc800fee0 tx=0x7fafc800b520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92af30260c84788 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafc17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafc17fa640 1 -- 172.21.15.16:0/2636490977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8025040 con 0x561ee7d4f1b0 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafc17fa640 1 -- 172.21.15.16:0/2636490977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fafc8005050 con 0x561ee7d4f1b0 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafe641f740 1 -- 172.21.15.16:0/2636490977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafd400dd40 con 0x561ee7d4f1b0 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafc17fa640 1 -- 172.21.15.16:0/2636490977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fafc8005250 con 0x561ee7d4f1b0 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafe641f740 10 client.18153 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.416+0000 7fafe641f740 1 -- 172.21.15.16:0/2636490977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x561ee81a5e40 con 0x561ee7d4f1b0 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.417+0000 7fafc17fa640 1 -- 172.21.15.16:0/2636490977 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafd400dd40 con 0x561ee7d4f1b0 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.417+0000 7fafc17fa640 1 -- 172.21.15.16:0/2636490977 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7fafc80046d0 con 0x561ee7d4f1b0 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.417+0000 7fafe641f740 10 client.18153 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.417+0000 7fafe641f740 10 client.18153 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.417+0000 7fafe641f740 20 client.18153 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.417+0000 7fafe641f740 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561ee7d0bda0 0x561ee81a8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.417+0000 7fafe641f740 4 client.18153 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.417+0000 7fafe641f740 1 -- 172.21.15.16:0/2636490977 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ee808f7b0 con 0x561ee7d0bda0 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.417+0000 7fafddc46640 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561ee7d0bda0 0x561ee81a8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.417+0000 7fafc17fa640 10 client.18153 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.418+0000 7fafddc46640 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561ee7d0bda0 0x561ee81a8470 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=81f84affa028bd37 server_cookie=a8c298cf5c74d321 in_seq=0 out_seq=0 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.418+0000 7fafc17fa640 1 -- 172.21.15.16:0/2636490977 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ee808f7b0 con 0x561ee7d0bda0 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.418+0000 7fafc17fa640 10 client.18153 handle_command_reply: tid=0 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.418+0000 7fafe641f740 1 client.18153 shutdown 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.418+0000 7fafe641f740 1 -- 172.21.15.16:0/2636490977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d4f1b0 msgr2=0x561ee812d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.418+0000 7fafe641f740 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d4f1b0 0x561ee812d0d0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fafc800fee0 tx=0x7fafc800b520 comp rx=0 tx=0).stop 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafe641f740 1 -- 172.21.15.16:0/2636490977 shutdown_connections 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafe641f740 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561ee7d0bda0 0x561ee81a8470 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafe641f740 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ee7d78070 0x561ee81a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafe641f740 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ee7d57390 0x561ee812d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafe641f740 1 --2- 172.21.15.16:0/2636490977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee7d4f1b0 0x561ee812d0d0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafe641f740 1 -- 172.21.15.16:0/2636490977 >> 172.21.15.16:0/2636490977 conn(0x561ee7d6c460 msgr2=0x561ee811aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:25.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafc17fa640 0 client.18153 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:31:25.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafe641f740 1 -- 172.21.15.16:0/2636490977 shutdown_connections 2024-08-28T19:31:25.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafe641f740 1 -- 172.21.15.16:0/2636490977 wait complete. 2024-08-28T19:31:25.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafe641f740 20 client.18153 trim_cache size 0 max 16384 2024-08-28T19:31:25.425 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:31:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafdfeaa640 1 -- 172.21.15.16:0/1566808115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fafc40b4cd0 msgr2=0x7fafc40b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafdfeaa640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fafc40b4cd0 0x7fafc40b71e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fafd01040c0 tx=0x7fafd002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafdfeaa640 1 -- 172.21.15.16:0/1566808115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8105880 msgr2=0x7fafd819fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.419+0000 7fafdfeaa640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8105880 0x7fafd819fab0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fafd0005cc0 tx=0x7fafd002f380 comp rx=0 tx=0).stop 2024-08-28T19:31:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.420+0000 7fafdfeaa640 1 -- 172.21.15.16:0/1566808115 shutdown_connections 2024-08-28T19:31:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.420+0000 7fafdfeaa640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8105880 0x7fafd819fab0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.420+0000 7fafdfeaa640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fafc40b4cd0 0x7fafc40b71e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.420+0000 7fafdfeaa640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd8104f30 0x7fafd819a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.420+0000 7fafdfeaa640 1 --2- 172.21.15.16:0/1566808115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fafd8102f60 0x7fafd819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.420+0000 7fafdfeaa640 1 -- 172.21.15.16:0/1566808115 >> 172.21.15.16:0/1566808115 conn(0x7fafd8077a00 msgr2=0x7fafd8106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.420+0000 7fafdfeaa640 1 -- 172.21.15.16:0/1566808115 shutdown_connections 2024-08-28T19:31:25.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:25.420+0000 7fafdfeaa640 1 -- 172.21.15.16:0/1566808115 wait complete. 2024-08-28T19:31:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:25 smithi028 ceph-mon[21769]: pgmap v658: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 480 KiB/s wr, 70 op/s 2024-08-28T19:31:25.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:25 smithi115 ceph-mon[22184]: pgmap v658: 129 pgs: 129 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 480 KiB/s wr, 70 op/s 2024-08-28T19:31:25.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.85 MB/sec execute 425 sec latency 11134.638 ms 2024-08-28T19:31:26.098 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:31:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.838+0000 7fe22cb23640 1 -- 172.21.15.16:0/1113769467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe228109c50 msgr2=0x7fe228110480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.838+0000 7fe22cb23640 1 --2- 172.21.15.16:0/1113769467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe228109c50 0x7fe228110480 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fe220009ca0 tx=0x7fe22002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.838+0000 7fe22cb23640 1 -- 172.21.15.16:0/1113769467 shutdown_connections 2024-08-28T19:31:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.838+0000 7fe22cb23640 1 --2- 172.21.15.16:0/1113769467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe228109c50 0x7fe228110480 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.838+0000 7fe22cb23640 1 --2- 172.21.15.16:0/1113769467 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe228109300 0x7fe228109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.838+0000 7fe22cb23640 1 --2- 172.21.15.16:0/1113769467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe228101e20 0x7fe228102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.838+0000 7fe22cb23640 1 -- 172.21.15.16:0/1113769467 >> 172.21.15.16:0/1113769467 conn(0x7fe228079c60 msgr2=0x7fe22807a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.838+0000 7fe22cb23640 1 -- 172.21.15.16:0/1113769467 shutdown_connections 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.838+0000 7fe22cb23640 1 -- 172.21.15.16:0/1113769467 wait complete. 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe22cb23640 1 Processor -- start 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe22cb23640 1 -- start start 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe22cb23640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe228101e20 0x7fe22819a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe22cb23640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe228109300 0x7fe22819a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe22cb23640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe228109c50 0x7fe22819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe22cb23640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe228113910 con 0x7fe228109c50 2024-08-28T19:31:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe22cb23640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe228113790 con 0x7fe228101e20 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe22cb23640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe228113a90 con 0x7fe228109300 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe226d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe228109c50 0x7fe22819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe225d9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe228109300 0x7fe22819a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe226d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe228109c50 0x7fe22819fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32878/0 (socket says 172.21.15.16:32878) 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe226d9d640 1 -- 172.21.15.16:0/2545839332 learned_addr learned my addr 172.21.15.16:0/2545839332 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe22659c640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe228101e20 0x7fe22819a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe226d9d640 1 -- 172.21.15.16:0/2545839332 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe228109300 msgr2=0x7fe22819a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe226d9d640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe228109300 0x7fe22819a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe226d9d640 1 -- 172.21.15.16:0/2545839332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe228101e20 msgr2=0x7fe22819a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.839+0000 7fe226d9d640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe228101e20 0x7fe22819a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.840+0000 7fe226d9d640 1 -- 172.21.15.16:0/2545839332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2281a0230 con 0x7fe228109c50 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.840+0000 7fe226d9d640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe228109c50 0x7fe22819fb00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe220005cc0 tx=0x7fe22002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.840+0000 7fe2177fe640 1 -- 172.21.15.16:0/2545839332 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220003bf0 con 0x7fe228109c50 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.840+0000 7fe22cb23640 1 -- 172.21.15.16:0/2545839332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2281a04c0 con 0x7fe228109c50 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.840+0000 7fe2177fe640 1 -- 172.21.15.16:0/2545839332 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe220003d90 con 0x7fe228109c50 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.840+0000 7fe2177fe640 1 -- 172.21.15.16:0/2545839332 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe22002fbc0 con 0x7fe228109c50 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.840+0000 7fe22cb23640 1 -- 172.21.15.16:0/2545839332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2281a6c80 con 0x7fe228109c50 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.843+0000 7fe233098740 1 Processor -- start 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.843+0000 7fe2177fe640 1 -- 172.21.15.16:0/2545839332 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe22002fd60 con 0x7fe228109c50 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.843+0000 7fe233098740 1 -- start start 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.843+0000 7fe233098740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee83191070 0x55ee83536310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.843+0000 7fe233098740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee831681b0 0x55ee83536850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.843+0000 7fe233098740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83536d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.843+0000 7fe233098740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ee834ad680 con 0x55ee83170390 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.843+0000 7fe233098740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ee8349cc10 con 0x55ee83191070 2024-08-28T19:31:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.843+0000 7fe233098740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ee834a0bc0 con 0x55ee831681b0 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe225d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83536d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe225d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83536d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32886/0 (socket says 172.21.15.16:32886) 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe225d9b640 1 -- 172.21.15.16:0/1696855196 learned_addr learned my addr 172.21.15.16:0/1696855196 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe225d9b640 1 --2- 172.21.15.16:0/1696855196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee83191070 0x55ee83536310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe22659c640 1 --2- 172.21.15.16:0/1696855196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee831681b0 0x55ee83536850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe2177fe640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2100b4ca0 0x7fe2100b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe225d9b640 1 -- 172.21.15.16:0/1696855196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee831681b0 msgr2=0x55ee83536850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe225d9b640 1 --2- 172.21.15.16:0/1696855196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee831681b0 0x55ee83536850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe225d9b640 1 -- 172.21.15.16:0/1696855196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee83191070 msgr2=0x55ee83536310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe225d9b640 1 --2- 172.21.15.16:0/1696855196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee83191070 0x55ee83536310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe226d9d640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2100b4ca0 0x7fe2100b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe225d9b640 1 -- 172.21.15.16:0/1696855196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee831fc650 con 0x55ee83170390 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe2177fe640 1 -- 172.21.15.16:0/2545839332 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fe220106dc0 con 0x7fe228109c50 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe225d9b640 1 --2- 172.21.15.16:0/1696855196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83536d90 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fe21c011630 tx=0x7fe21c00a5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf72955439b3134 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.844+0000 7fe214ff9640 1 -- 172.21.15.16:0/1696855196 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c004020 con 0x55ee83170390 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.845+0000 7fe214ff9640 1 -- 172.21.15.16:0/1696855196 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe21c0054d0 con 0x55ee83170390 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.845+0000 7fe214ff9640 1 -- 172.21.15.16:0/1696855196 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c004bd0 con 0x55ee83170390 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.845+0000 7fe233098740 1 -- 172.21.15.16:0/1696855196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 msgr2=0x55ee83536d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.845+0000 7fe226d9d640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2100b4ca0 0x7fe2100b71b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe2201041f0 tx=0x7fe2200b4040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.845+0000 7fe233098740 1 --2- 172.21.15.16:0/1696855196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83536d90 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fe21c011630 tx=0x7fe21c00a5e0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.846+0000 7fe233098740 1 -- 172.21.15.16:0/1696855196 shutdown_connections 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.846+0000 7fe233098740 1 --2- 172.21.15.16:0/1696855196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83536d90 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.846+0000 7fe233098740 1 --2- 172.21.15.16:0/1696855196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee831681b0 0x55ee83536850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.846+0000 7fe233098740 1 --2- 172.21.15.16:0/1696855196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee83191070 0x55ee83536310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.846+0000 7fe233098740 1 -- 172.21.15.16:0/1696855196 >> 172.21.15.16:0/1696855196 conn(0x55ee83185460 msgr2=0x55ee83534200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.846+0000 7fe233098740 1 -- 172.21.15.16:0/1696855196 shutdown_connections 2024-08-28T19:31:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.846+0000 7fe233098740 1 -- 172.21.15.16:0/1696855196 wait complete. 2024-08-28T19:31:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.846+0000 7fe233098740 1 -- start start 2024-08-28T19:31:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.846+0000 7fe233098740 1 Processor -- start 2024-08-28T19:31:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe233098740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee83541a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe233098740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83541f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe233098740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83191070 0x55ee8354de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe233098740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ee832ff700 con 0x55ee83170390 2024-08-28T19:31:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe233098740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ee83423750 con 0x55ee831681b0 2024-08-28T19:31:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe233098740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ee83544af0 con 0x55ee83191070 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe233098740 10 client.0 _ll_register_callbacks cb 0x55ee83452e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe22659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83541f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe22659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83541f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32894/0 (socket says 172.21.15.16:32894) 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe22659c640 1 -- 172.21.15.16:0/2773364231 learned_addr learned my addr 172.21.15.16:0/2773364231 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe225d9b640 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83191070 0x55ee8354de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe225d9b640 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee83541a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe22659c640 1 -- 172.21.15.16:0/2773364231 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83191070 msgr2=0x55ee8354de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe22659c640 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83191070 0x55ee8354de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe22659c640 1 -- 172.21.15.16:0/2773364231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 msgr2=0x55ee83541a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe22659c640 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee83541a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe22659c640 1 -- 172.21.15.16:0/2773364231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee8345c900 con 0x55ee83170390 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe225d9b640 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83191070 0x55ee8354de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe22659c640 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83541f40 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fe21800bad0 tx=0x7fe21800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205e609526b4ce92 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe20dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe20dffb640 1 -- 172.21.15.16:0/2773364231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218005020 con 0x55ee83170390 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe20dffb640 1 -- 172.21.15.16:0/2773364231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fe2180051c0 con 0x55ee83170390 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.847+0000 7fe233098740 1 -- 172.21.15.16:0/2773364231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe21c0054d0 con 0x55ee83170390 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe20dffb640 1 -- 172.21.15.16:0/2773364231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fe2180053c0 con 0x55ee83170390 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe233098740 10 client.18165 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe233098740 1 -- 172.21.15.16:0/2773364231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ee835508f0 con 0x55ee83170390 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe20dffb640 1 -- 172.21.15.16:0/2773364231 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe21c0054d0 con 0x55ee83170390 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe20dffb640 1 -- 172.21.15.16:0/2773364231 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7fe21801b6a0 con 0x55ee83170390 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe233098740 10 client.18165 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe233098740 10 client.18165 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe233098740 20 client.18165 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe233098740 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55ee8314f9f0 0x55ee83553e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe233098740 4 client.18165 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe233098740 1 -- 172.21.15.16:0/2773364231 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee834af5d0 con 0x55ee8314f9f0 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.848+0000 7fe225d9b640 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55ee8314f9f0 0x55ee83553e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.849+0000 7fe225d9b640 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55ee8314f9f0 0x55ee83553e00 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=70c97e2cb5d0fa4c server_cookie=ce59ae628a2c3420 in_seq=0 out_seq=0 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.849+0000 7fe20dffb640 10 client.18165 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe20dffb640 1 -- 172.21.15.16:0/2773364231 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ee834af5d0 con 0x55ee8314f9f0 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe20dffb640 10 client.18165 handle_command_reply: tid=0 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 1 client.18165 shutdown 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 1 -- 172.21.15.16:0/2773364231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 msgr2=0x55ee83541f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83541f40 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fe21800bad0 tx=0x7fe21800bed0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 1 -- 172.21.15.16:0/2773364231 shutdown_connections 2024-08-28T19:31:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55ee8314f9f0 0x55ee83553e00 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83191070 0x55ee8354de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83170390 0x55ee83541f40 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 1 --2- 172.21.15.16:0/2773364231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee83541a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 1 -- 172.21.15.16:0/2773364231 >> 172.21.15.16:0/2773364231 conn(0x55ee83185460 msgr2=0x55ee835394a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe20dffb640 0 client.18165 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:31:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 1 -- 172.21.15.16:0/2773364231 shutdown_connections 2024-08-28T19:31:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 1 -- 172.21.15.16:0/2773364231 wait complete. 2024-08-28T19:31:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.850+0000 7fe233098740 20 client.18165 trim_cache size 0 max 16384 2024-08-28T19:31:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.863+0000 7fe233098740 1 Processor -- start 2024-08-28T19:31:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.863+0000 7fe233098740 1 -- start start 2024-08-28T19:31:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe233098740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee8354dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe233098740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83170390 0x55ee8353ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe233098740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee8353f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe233098740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2281b0cd0 con 0x55ee83191070 2024-08-28T19:31:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe233098740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2281b2810 con 0x55ee831681b0 2024-08-28T19:31:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe233098740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe2281b13a0 con 0x55ee83170390 2024-08-28T19:31:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe225d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee8353f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe225d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee8353f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32898/0 (socket says 172.21.15.16:32898) 2024-08-28T19:31:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe225d9b640 1 -- 172.21.15.16:0/2339239788 learned_addr learned my addr 172.21.15.16:0/2339239788 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe22659c640 1 --2- 172.21.15.16:0/2339239788 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83170390 0x55ee8353ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe225d9b640 1 --2- 172.21.15.16:0/2339239788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee8354dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe225d9b640 1 -- 172.21.15.16:0/2339239788 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83170390 msgr2=0x55ee8353ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe225d9b640 1 --2- 172.21.15.16:0/2339239788 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83170390 0x55ee8353ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe225d9b640 1 -- 172.21.15.16:0/2339239788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 msgr2=0x55ee8354dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe225d9b640 1 --2- 172.21.15.16:0/2339239788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee8354dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe225d9b640 1 -- 172.21.15.16:0/2339239788 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee83542aa0 con 0x55ee83191070 2024-08-28T19:31:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe22659c640 1 --2- 172.21.15.16:0/2339239788 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83170390 0x55ee8353ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.864+0000 7fe225d9b640 1 --2- 172.21.15.16:0/2339239788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee8353f130 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fe21c011fd0 tx=0x7fe21c016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4aab88475c8e07e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe20ffff640 1 -- 172.21.15.16:0/2339239788 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c00f870 con 0x55ee83191070 2024-08-28T19:31:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe20ffff640 1 -- 172.21.15.16:0/2339239788 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe21c01e470 con 0x55ee83191070 2024-08-28T19:31:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe20ffff640 1 -- 172.21.15.16:0/2339239788 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c02c5c0 con 0x55ee83191070 2024-08-28T19:31:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe233098740 1 -- 172.21.15.16:0/2339239788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 msgr2=0x55ee8353f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe233098740 1 --2- 172.21.15.16:0/2339239788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee8353f130 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fe21c011fd0 tx=0x7fe21c016040 comp rx=0 tx=0).stop 2024-08-28T19:31:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe233098740 1 -- 172.21.15.16:0/2339239788 shutdown_connections 2024-08-28T19:31:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe233098740 1 --2- 172.21.15.16:0/2339239788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee8353f130 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe233098740 1 --2- 172.21.15.16:0/2339239788 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83170390 0x55ee8353ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe233098740 1 --2- 172.21.15.16:0/2339239788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee8354dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe233098740 1 -- 172.21.15.16:0/2339239788 >> 172.21.15.16:0/2339239788 conn(0x55ee83185460 msgr2=0x55ee83540280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe233098740 1 -- 172.21.15.16:0/2339239788 shutdown_connections 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.865+0000 7fe233098740 1 -- 172.21.15.16:0/2339239788 wait complete. 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe233098740 1 -- start start 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe233098740 1 Processor -- start 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe233098740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee83545ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe233098740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83170390 0x55ee83546530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe233098740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee835bd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe233098740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2281b2140 con 0x55ee83191070 2024-08-28T19:31:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe233098740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2281b1a70 con 0x55ee831681b0 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe233098740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe2281aecd0 con 0x55ee83170390 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe233098740 10 client.0 _ll_register_callbacks cb 0x7fe2180040a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe225d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee835bd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe225d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee835bd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32904/0 (socket says 172.21.15.16:32904) 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe225d9b640 1 -- 172.21.15.16:0/1649305531 learned_addr learned my addr 172.21.15.16:0/1649305531 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.866+0000 7fe22659c640 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83170390 0x55ee83546530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe225d9b640 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee83545ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe225d9b640 1 -- 172.21.15.16:0/1649305531 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83170390 msgr2=0x55ee83546530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe225d9b640 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83170390 0x55ee83546530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe225d9b640 1 -- 172.21.15.16:0/1649305531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 msgr2=0x55ee83545ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe225d9b640 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee83545ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe225d9b640 1 -- 172.21.15.16:0/1649305531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee83546d40 con 0x55ee83191070 2024-08-28T19:31:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe225d9b640 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee835bd7c0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fe21c005940 tx=0x7fe21c01ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8dd230738e4485 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe20dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe20dffb640 1 -- 172.21.15.16:0/1649305531 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c011220 con 0x55ee83191070 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe20dffb640 1 -- 172.21.15.16:0/1649305531 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fe21c01e620 con 0x55ee83191070 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe233098740 1 -- 172.21.15.16:0/1649305531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe21c01e470 con 0x55ee83191070 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe20dffb640 1 -- 172.21.15.16:0/1649305531 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fe21c00a8f0 con 0x55ee83191070 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe233098740 10 client.18171 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.867+0000 7fe233098740 1 -- 172.21.15.16:0/1649305531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ee835becc0 con 0x55ee83191070 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.868+0000 7fe20dffb640 1 -- 172.21.15.16:0/1649305531 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe21c01e470 con 0x55ee83191070 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.868+0000 7fe20dffb640 1 -- 172.21.15.16:0/1649305531 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7fe21c02c570 con 0x55ee83191070 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.868+0000 7fe233098740 10 client.18171 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.868+0000 7fe233098740 10 client.18171 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.868+0000 7fe233098740 20 client.18171 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.868+0000 7fe233098740 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55ee83124da0 0x55ee835c12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.868+0000 7fe233098740 4 client.18171 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.868+0000 7fe233098740 1 -- 172.21.15.16:0/1649305531 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee834ab070 con 0x55ee83124da0 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.868+0000 7fe22659c640 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55ee83124da0 0x55ee835c12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe20dffb640 10 client.18171 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe22659c640 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55ee83124da0 0x55ee835c12f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f306f162cd9b62f server_cookie=c31a3d61d301fb4c in_seq=0 out_seq=0 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe20dffb640 1 -- 172.21.15.16:0/1649305531 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee834ab070 con 0x55ee83124da0 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe20dffb640 10 client.18171 handle_command_reply: tid=0 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 1 client.18171 shutdown 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 1 -- 172.21.15.16:0/1649305531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 msgr2=0x55ee835bd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee835bd7c0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fe21c005940 tx=0x7fe21c01ee40 comp rx=0 tx=0).stop 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 1 -- 172.21.15.16:0/1649305531 shutdown_connections 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55ee83124da0 0x55ee835c12f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee83191070 0x55ee835bd7c0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee83170390 0x55ee83546530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 1 --2- 172.21.15.16:0/1649305531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ee831681b0 0x55ee83545ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 1 -- 172.21.15.16:0/1649305531 >> 172.21.15.16:0/1649305531 conn(0x55ee83185460 msgr2=0x55ee83537340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe20dffb640 0 client.18171 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:31:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 1 -- 172.21.15.16:0/1649305531 shutdown_connections 2024-08-28T19:31:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 1 -- 172.21.15.16:0/1649305531 wait complete. 2024-08-28T19:31:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.869+0000 7fe233098740 20 client.18171 trim_cache size 0 max 16384 2024-08-28T19:31:26.876 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:31:26.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 -- 172.21.15.16:0/2545839332 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2100b4ca0 msgr2=0x7fe2100b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:26.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2100b4ca0 0x7fe2100b71b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe2201041f0 tx=0x7fe2200b4040 comp rx=0 tx=0).stop 2024-08-28T19:31:26.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 -- 172.21.15.16:0/2545839332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe228109c50 msgr2=0x7fe22819fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:26.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe228109c50 0x7fe22819fb00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe220005cc0 tx=0x7fe22002f380 comp rx=0 tx=0).stop 2024-08-28T19:31:26.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 -- 172.21.15.16:0/2545839332 shutdown_connections 2024-08-28T19:31:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe228109c50 0x7fe22819fb00 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe228109300 0x7fe22819a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2100b4ca0 0x7fe2100b71b0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 --2- 172.21.15.16:0/2545839332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe228101e20 0x7fe22819a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 -- 172.21.15.16:0/2545839332 >> 172.21.15.16:0/2545839332 conn(0x7fe228079c60 msgr2=0x7fe2281052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 -- 172.21.15.16:0/2545839332 shutdown_connections 2024-08-28T19:31:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:26.870+0000 7fe22cb23640 1 -- 172.21.15.16:0/2545839332 wait complete. 2024-08-28T19:31:26.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.84 MB/sec execute 426 sec latency 12134.690 ms 2024-08-28T19:31:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:27 smithi016 ceph-mon[18556]: pgmap v659: 129 pgs: 129 active+clean; 312 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 587 KiB/s wr, 59 op/s 2024-08-28T19:31:27.530 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:31:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:27 smithi028 ceph-mon[21769]: pgmap v659: 129 pgs: 129 active+clean; 312 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 587 KiB/s wr, 59 op/s 2024-08-28T19:31:27.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:27 smithi115 ceph-mon[22184]: pgmap v659: 129 pgs: 129 active+clean; 312 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 587 KiB/s wr, 59 op/s 2024-08-28T19:31:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:27.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.83 MB/sec execute 427 sec latency 13134.765 ms 2024-08-28T19:31:28.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.230+0000 7fede3b08640 1 -- 172.21.15.16:0/1832385822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc109240 msgr2=0x7feddc109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:28.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.230+0000 7fede3b08640 1 --2- 172.21.15.16:0/1832385822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc109240 0x7feddc109650 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009900 tx=0x7fedcc02a410 comp rx=0 tx=0).stop 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.230+0000 7fede3b08640 1 -- 172.21.15.16:0/1832385822 shutdown_connections 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.230+0000 7fede3b08640 1 --2- 172.21.15.16:0/1832385822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc109b90 0x7feddc1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.230+0000 7fede3b08640 1 --2- 172.21.15.16:0/1832385822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc109240 0x7feddc109650 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.230+0000 7fede3b08640 1 --2- 172.21.15.16:0/1832385822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feddc103900 0x7feddc103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.230+0000 7fede3b08640 1 -- 172.21.15.16:0/1832385822 >> 172.21.15.16:0/1832385822 conn(0x7feddc07b5f0 msgr2=0x7feddc07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.230+0000 7fede3b08640 1 -- 172.21.15.16:0/1832385822 shutdown_connections 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.230+0000 7fede3b08640 1 -- 172.21.15.16:0/1832385822 wait complete. 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.231+0000 7fede3b08640 1 Processor -- start 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.231+0000 7fede3b08640 1 -- start start 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.231+0000 7fede3b08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc103900 0x7feddc19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.231+0000 7fede3b08640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feddc109240 0x7feddc19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede3b08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc109b90 0x7feddc19fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede3b08640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feddc113850 con 0x7feddc109b90 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede3b08640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feddc1136d0 con 0x7feddc103900 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede3b08640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7feddc1139d0 con 0x7feddc109240 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede20a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc109b90 0x7feddc19fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede20a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc109b90 0x7feddc19fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32924/0 (socket says 172.21.15.16:32924) 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede20a5640 1 -- 172.21.15.16:0/2327530769 learned_addr learned my addr 172.21.15.16:0/2327530769 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede18a4640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc103900 0x7feddc19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede10a3640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feddc109240 0x7feddc19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede20a5640 1 -- 172.21.15.16:0/2327530769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feddc109240 msgr2=0x7feddc19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede20a5640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feddc109240 0x7feddc19a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede20a5640 1 -- 172.21.15.16:0/2327530769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc103900 msgr2=0x7feddc19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede20a5640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc103900 0x7feddc19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede20a5640 1 -- 172.21.15.16:0/2327530769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feddc1a01b0 con 0x7feddc109b90 2024-08-28T19:31:28.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede10a3640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feddc109240 0x7feddc19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede20a5640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc109b90 0x7feddc19fa80 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fedd400eb10 tx=0x7fedd400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fedd2ffd640 1 -- 172.21.15.16:0/2327530769 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4004b20 con 0x7feddc109b90 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fedd2ffd640 1 -- 172.21.15.16:0/2327530769 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd4004cc0 con 0x7feddc109b90 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede3b08640 1 -- 172.21.15.16:0/2327530769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feddc1a04a0 con 0x7feddc109b90 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fedd2ffd640 1 -- 172.21.15.16:0/2327530769 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4010670 con 0x7feddc109b90 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.232+0000 7fede3b08640 1 -- 172.21.15.16:0/2327530769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feddc1a6c00 con 0x7feddc109b90 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.235+0000 7fedea07d740 1 Processor -- start 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.235+0000 7fedd2ffd640 1 -- 172.21.15.16:0/2327530769 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fedd4005020 con 0x7feddc109b90 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.235+0000 7fedea07d740 1 -- start start 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fedea07d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81aec5070 0x55e81b26a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fedea07d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81ae9c1b0 0x55e81b26a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fedea07d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b26add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fedea07d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e81b1e1680 con 0x55e81aea4390 2024-08-28T19:31:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fedea07d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e81b1d0c10 con 0x55e81ae9c1b0 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fedea07d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e81b1d4bc0 con 0x55e81aec5070 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede10a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b26add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede10a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b26add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32938/0 (socket says 172.21.15.16:32938) 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede10a3640 1 -- 172.21.15.16:0/3937382345 learned_addr learned my addr 172.21.15.16:0/3937382345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede18a4640 1 --2- 172.21.15.16:0/3937382345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81ae9c1b0 0x55e81b26a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede10a3640 1 --2- 172.21.15.16:0/3937382345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81aec5070 0x55e81b26a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fedd2ffd640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fedc80b4ca0 0x7fedc80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede18a4640 1 -- 172.21.15.16:0/3937382345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81aec5070 msgr2=0x55e81b26a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede18a4640 1 --2- 172.21.15.16:0/3937382345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81aec5070 0x55e81b26a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede18a4640 1 -- 172.21.15.16:0/3937382345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 msgr2=0x55e81b26add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede18a4640 1 --2- 172.21.15.16:0/3937382345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b26add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede18a4640 1 -- 172.21.15.16:0/3937382345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e81af30650 con 0x55e81ae9c1b0 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede10a3640 1 --2- 172.21.15.16:0/3937382345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b26add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:31:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fede20a5640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fedc80b4ca0 0x7fedc80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.236+0000 7fedd2ffd640 1 -- 172.21.15.16:0/2327530769 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fedd40e5ff0 con 0x7feddc109b90 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.237+0000 7fede18a4640 1 --2- 172.21.15.16:0/3937382345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81ae9c1b0 0x55e81b26a890 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fedd8004630 tx=0x7fedd800bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=884f3047ec9d731 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.237+0000 7fedc7fff640 1 -- 172.21.15.16:0/3937382345 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8003bd0 con 0x55e81ae9c1b0 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.237+0000 7fedc7fff640 1 -- 172.21.15.16:0/3937382345 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd8015070 con 0x55e81ae9c1b0 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.237+0000 7fedc7fff640 1 -- 172.21.15.16:0/3937382345 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8004ca0 con 0x55e81ae9c1b0 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fedea07d740 1 -- 172.21.15.16:0/3937382345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81ae9c1b0 msgr2=0x55e81b26a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fede20a5640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fedc80b4ca0 0x7fedc80b71b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fedd40a9520 tx=0x7fedd4070bb0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fedea07d740 1 --2- 172.21.15.16:0/3937382345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81ae9c1b0 0x55e81b26a890 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fedd8004630 tx=0x7fedd800bb20 comp rx=0 tx=0).stop 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fedea07d740 1 -- 172.21.15.16:0/3937382345 shutdown_connections 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fedea07d740 1 --2- 172.21.15.16:0/3937382345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b26add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fedea07d740 1 --2- 172.21.15.16:0/3937382345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81ae9c1b0 0x55e81b26a890 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fedea07d740 1 --2- 172.21.15.16:0/3937382345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81aec5070 0x55e81b26a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fedea07d740 1 -- 172.21.15.16:0/3937382345 >> 172.21.15.16:0/3937382345 conn(0x55e81aeb9460 msgr2=0x55e81b268240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fedea07d740 1 -- 172.21.15.16:0/3937382345 shutdown_connections 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fedea07d740 1 -- 172.21.15.16:0/3937382345 wait complete. 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fedea07d740 1 -- start start 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.238+0000 7fedea07d740 1 Processor -- start 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fedea07d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b279e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fedea07d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aea4390 0x55e81b27a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fedea07d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aec5070 0x55e81b27a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fedea07d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e81b033700 con 0x55e81aec5070 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fedea07d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e81b157750 con 0x55e81aea4390 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fedea07d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e81b27ce30 con 0x55e81ae9c1b0 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fedea07d740 10 client.0 _ll_register_callbacks cb 0x55e81b186e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fede10a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aec5070 0x55e81b27a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fede10a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aec5070 0x55e81b27a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32954/0 (socket says 172.21.15.16:32954) 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fede10a3640 1 -- 172.21.15.16:0/235199337 learned_addr learned my addr 172.21.15.16:0/235199337 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fede18a4640 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aea4390 0x55e81b27a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fede10a3640 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b279e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fede10a3640 1 -- 172.21.15.16:0/235199337 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 msgr2=0x55e81b279e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fede10a3640 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b279e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fede10a3640 1 -- 172.21.15.16:0/235199337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aea4390 msgr2=0x55e81b27a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fede10a3640 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aea4390 0x55e81b27a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.239+0000 7fede10a3640 1 -- 172.21.15.16:0/235199337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e81b190900 con 0x55e81aec5070 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fede10a3640 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aec5070 0x55e81b27a8e0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fedcc006010 tx=0x7fedcc02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452f9245579ae482 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedc57fa640 1 -- 172.21.15.16:0/235199337 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc035040 con 0x55e81aec5070 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedc57fa640 1 -- 172.21.15.16:0/235199337 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fedcc004450 con 0x55e81aec5070 2024-08-28T19:31:28.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedea07d740 1 -- 172.21.15.16:0/235199337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedd8015070 con 0x55e81aec5070 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedc57fa640 1 -- 172.21.15.16:0/235199337 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fedcc052050 con 0x55e81aec5070 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedea07d740 10 client.18183 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedea07d740 1 -- 172.21.15.16:0/235199337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55e81b288b70 con 0x55e81aec5070 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedc57fa640 1 -- 172.21.15.16:0/235199337 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd8015070 con 0x55e81aec5070 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedc57fa640 1 -- 172.21.15.16:0/235199337 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7fedcc03a040 con 0x55e81aec5070 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedea07d740 10 client.18183 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedea07d740 10 client.18183 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedea07d740 20 client.18183 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedea07d740 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e81ae839f0 0x55e81b28bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedea07d740 4 client.18183 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.240+0000 7fedea07d740 1 -- 172.21.15.16:0/235199337 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e81b1e35d0 con 0x55e81ae839f0 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.241+0000 7fede18a4640 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e81ae839f0 0x55e81b28bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.241+0000 7fedc57fa640 10 client.18183 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:31:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.241+0000 7fede18a4640 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e81ae839f0 0x55e81b28bfd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=37b5ddc0ab0c993c server_cookie=a2df65a8e59b5076 in_seq=0 out_seq=0 2024-08-28T19:31:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedc57fa640 1 -- 172.21.15.16:0/235199337 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e81b1e35d0 con 0x55e81ae839f0 2024-08-28T19:31:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedc57fa640 10 client.18183 handle_command_reply: tid=0 2024-08-28T19:31:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 1 client.18183 shutdown 2024-08-28T19:31:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 1 -- 172.21.15.16:0/235199337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aec5070 msgr2=0x55e81b27a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aec5070 0x55e81b27a8e0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fedcc006010 tx=0x7fedcc02a410 comp rx=0 tx=0).stop 2024-08-28T19:31:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 1 -- 172.21.15.16:0/235199337 shutdown_connections 2024-08-28T19:31:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e81ae839f0 0x55e81b28bfd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aec5070 0x55e81b27a8e0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aea4390 0x55e81b27a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 1 --2- 172.21.15.16:0/235199337 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b279e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedc57fa640 0 client.18183 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:31:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 1 -- 172.21.15.16:0/235199337 >> 172.21.15.16:0/235199337 conn(0x55e81aeb9460 msgr2=0x55e81b266800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 1 -- 172.21.15.16:0/235199337 shutdown_connections 2024-08-28T19:31:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 1 -- 172.21.15.16:0/235199337 wait complete. 2024-08-28T19:31:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.242+0000 7fedea07d740 20 client.18183 trim_cache size 0 max 16384 2024-08-28T19:31:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.256+0000 7fedea07d740 1 Processor -- start 2024-08-28T19:31:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.256+0000 7fedea07d740 1 -- start start 2024-08-28T19:31:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.256+0000 7fedea07d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b26fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.256+0000 7fedea07d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b291820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.256+0000 7fedea07d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aec5070 0x55e81b291d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.256+0000 7fedea07d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feddc1aeab0 con 0x55e81aea4390 2024-08-28T19:31:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.256+0000 7fedea07d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feddc1b05f0 con 0x55e81aec5070 2024-08-28T19:31:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.256+0000 7fedea07d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7feddc1af180 con 0x55e81ae9c1b0 2024-08-28T19:31:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.256+0000 7fede18a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b291820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.256+0000 7fede18a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b291820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32964/0 (socket says 172.21.15.16:32964) 2024-08-28T19:31:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.256+0000 7fede18a4640 1 -- 172.21.15.16:0/4275798886 learned_addr learned my addr 172.21.15.16:0/4275798886 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.257+0000 7fede18a4640 1 -- 172.21.15.16:0/4275798886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 msgr2=0x55e81b26fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.257+0000 7fede10a3640 1 --2- 172.21.15.16:0/4275798886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b26fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.257+0000 7fede18a4640 1 --2- 172.21.15.16:0/4275798886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b26fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.257+0000 7fede18a4640 1 -- 172.21.15.16:0/4275798886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aec5070 msgr2=0x55e81b291d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.257+0000 7fede18a4640 1 --2- 172.21.15.16:0/4275798886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aec5070 0x55e81b291d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.257+0000 7fede18a4640 1 -- 172.21.15.16:0/4275798886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e81b26b140 con 0x55e81aea4390 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.257+0000 7fede10a3640 1 --2- 172.21.15.16:0/4275798886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b26fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.257+0000 7fede18a4640 1 --2- 172.21.15.16:0/4275798886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b291820 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fedd80048d0 tx=0x7fedd8004900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bab37cc9d0e737b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.257+0000 7fedc77fe640 1 -- 172.21.15.16:0/4275798886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8032070 con 0x55e81aea4390 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.257+0000 7fedc77fe640 1 -- 172.21.15.16:0/4275798886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd8015040 con 0x55e81aea4390 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.257+0000 7fedc77fe640 1 -- 172.21.15.16:0/4275798886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8011440 con 0x55e81aea4390 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.258+0000 7fedea07d740 1 -- 172.21.15.16:0/4275798886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 msgr2=0x55e81b291820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.258+0000 7fedea07d740 1 --2- 172.21.15.16:0/4275798886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b291820 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fedd80048d0 tx=0x7fedd8004900 comp rx=0 tx=0).stop 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.258+0000 7fedea07d740 1 -- 172.21.15.16:0/4275798886 shutdown_connections 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.258+0000 7fedea07d740 1 --2- 172.21.15.16:0/4275798886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aec5070 0x55e81b291d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.258+0000 7fedea07d740 1 --2- 172.21.15.16:0/4275798886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b291820 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.258+0000 7fedea07d740 1 --2- 172.21.15.16:0/4275798886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b26fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.258+0000 7fedea07d740 1 -- 172.21.15.16:0/4275798886 >> 172.21.15.16:0/4275798886 conn(0x55e81aeb9460 msgr2=0x55e81b26f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:28.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.258+0000 7fedea07d740 1 -- 172.21.15.16:0/4275798886 shutdown_connections 2024-08-28T19:31:28.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.258+0000 7fedea07d740 1 -- 172.21.15.16:0/4275798886 wait complete. 2024-08-28T19:31:28.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.258+0000 7fedea07d740 1 -- start start 2024-08-28T19:31:28.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.258+0000 7fedea07d740 1 Processor -- start 2024-08-28T19:31:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fedea07d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b27e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fedea07d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b27e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fedea07d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aec5070 0x55e81b2f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fedea07d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feddc1aff20 con 0x55e81aea4390 2024-08-28T19:31:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fedea07d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feddc1af850 con 0x55e81aec5070 2024-08-28T19:31:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fedea07d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7feddc1acab0 con 0x55e81ae9c1b0 2024-08-28T19:31:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fedea07d740 10 client.0 _ll_register_callbacks cb 0x7fedcc031600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fede18a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b27e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fede18a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b27e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32966/0 (socket says 172.21.15.16:32966) 2024-08-28T19:31:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fede18a4640 1 -- 172.21.15.16:0/3453061133 learned_addr learned my addr 172.21.15.16:0/3453061133 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fede10a3640 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b27e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fede10a3640 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aec5070 0x55e81b2f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fede18a4640 1 -- 172.21.15.16:0/3453061133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 msgr2=0x55e81b27e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fede18a4640 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b27e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fede18a4640 1 -- 172.21.15.16:0/3453061133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aec5070 msgr2=0x55e81b2f5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fede18a4640 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aec5070 0x55e81b2f5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.259+0000 7fede18a4640 1 -- 172.21.15.16:0/3453061133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e81b27f140 con 0x55e81aea4390 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fede18a4640 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b27e920 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fedd80041a0 tx=0x7fedd8004660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a43f22ffc94471f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedc57fa640 1 -- 172.21.15.16:0/3453061133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8032070 con 0x55e81aea4390 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedc57fa640 1 -- 172.21.15.16:0/3453061133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fedd802d650 con 0x55e81aea4390 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedea07d740 1 -- 172.21.15.16:0/3453061133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedd8015040 con 0x55e81aea4390 2024-08-28T19:31:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedc57fa640 1 -- 172.21.15.16:0/3453061133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fedd8041050 con 0x55e81aea4390 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedea07d740 10 client.18189 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedea07d740 1 -- 172.21.15.16:0/3453061133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55e81b2f6ea0 con 0x55e81aea4390 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedc57fa640 1 -- 172.21.15.16:0/3453061133 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd8015040 con 0x55e81aea4390 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedc57fa640 1 -- 172.21.15.16:0/3453061133 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7fedd8011c20 con 0x55e81aea4390 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedea07d740 10 client.18189 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedea07d740 10 client.18189 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedea07d740 20 client.18189 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedea07d740 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e81ae58da0 0x55e81b2f9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedea07d740 4 client.18189 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.260+0000 7fedea07d740 1 -- 172.21.15.16:0/3453061133 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e81b1df070 con 0x55e81ae58da0 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.261+0000 7fede10a3640 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e81ae58da0 0x55e81b2f9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.261+0000 7fedc57fa640 10 client.18189 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:31:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.261+0000 7fede10a3640 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e81ae58da0 0x55e81b2f9450 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=16bb6ece7be20c29 server_cookie=410dd8cc63f2584b in_seq=0 out_seq=0 2024-08-28T19:31:28.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.261+0000 7fedc57fa640 1 -- 172.21.15.16:0/3453061133 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e81b1df070 con 0x55e81ae58da0 2024-08-28T19:31:28.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.261+0000 7fedc57fa640 10 client.18189 handle_command_reply: tid=0 2024-08-28T19:31:28.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.261+0000 7fedea07d740 1 client.18189 shutdown 2024-08-28T19:31:28.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.261+0000 7fedea07d740 1 -- 172.21.15.16:0/3453061133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 msgr2=0x55e81b27e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:28.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.261+0000 7fedea07d740 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b27e920 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fedd80041a0 tx=0x7fedd8004660 comp rx=0 tx=0).stop 2024-08-28T19:31:28.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fedea07d740 1 -- 172.21.15.16:0/3453061133 shutdown_connections 2024-08-28T19:31:28.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fedea07d740 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55e81ae58da0 0x55e81b2f9450 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fedea07d740 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81aec5070 0x55e81b2f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fedea07d740 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e81aea4390 0x55e81b27e920 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fedc57fa640 0 client.18189 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:31:28.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fedea07d740 1 --2- 172.21.15.16:0/3453061133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e81ae9c1b0 0x55e81b27e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fedea07d740 1 -- 172.21.15.16:0/3453061133 >> 172.21.15.16:0/3453061133 conn(0x55e81aeb9460 msgr2=0x55e81b26b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:28.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fedea07d740 1 -- 172.21.15.16:0/3453061133 shutdown_connections 2024-08-28T19:31:28.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fedea07d740 1 -- 172.21.15.16:0/3453061133 wait complete. 2024-08-28T19:31:28.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fedea07d740 20 client.18189 trim_cache size 0 max 16384 2024-08-28T19:31:28.279 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:31:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fede3b08640 1 -- 172.21.15.16:0/2327530769 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fedc80b4ca0 msgr2=0x7fedc80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fede3b08640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fedc80b4ca0 0x7fedc80b71b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fedd40a9520 tx=0x7fedd4070bb0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fede3b08640 1 -- 172.21.15.16:0/2327530769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc109b90 msgr2=0x7feddc19fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fede3b08640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc109b90 0x7feddc19fa80 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fedd400eb10 tx=0x7fedd400c2d0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fede3b08640 1 -- 172.21.15.16:0/2327530769 shutdown_connections 2024-08-28T19:31:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fede3b08640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc109b90 0x7feddc19fa80 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fede3b08640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feddc109240 0x7feddc19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fede3b08640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fedc80b4ca0 0x7fedc80b71b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fede3b08640 1 --2- 172.21.15.16:0/2327530769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc103900 0x7feddc19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.262+0000 7fede3b08640 1 -- 172.21.15.16:0/2327530769 >> 172.21.15.16:0/2327530769 conn(0x7feddc07b5f0 msgr2=0x7feddc107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.263+0000 7fede3b08640 1 -- 172.21.15.16:0/2327530769 shutdown_connections 2024-08-28T19:31:28.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:28.263+0000 7fede3b08640 1 -- 172.21.15.16:0/2327530769 wait complete. 2024-08-28T19:31:28.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.82 MB/sec execute 428 sec latency 14134.827 ms 2024-08-28T19:31:28.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:31:29.226 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:29.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:29 smithi016 ceph-mon[18556]: pgmap v660: 129 pgs: 129 active+clean; 313 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 601 KiB/s wr, 45 op/s 2024-08-28T19:31:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:29 smithi028 ceph-mon[21769]: pgmap v660: 129 pgs: 129 active+clean; 313 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 601 KiB/s wr, 45 op/s 2024-08-28T19:31:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:29 smithi115 ceph-mon[22184]: pgmap v660: 129 pgs: 129 active+clean; 313 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 601 KiB/s wr, 45 op/s 2024-08-28T19:31:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.725+0000 7f4893a5b640 1 -- 172.21.15.16:0/2411055612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bc0 msgr2=0x7f488c1103f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.725+0000 7f4893a5b640 1 --2- 172.21.15.16:0/2411055612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bc0 0x7f488c1103f0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f4884009ca0 tx=0x7f488402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.725+0000 7f4893a5b640 1 -- 172.21.15.16:0/2411055612 shutdown_connections 2024-08-28T19:31:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.725+0000 7f4893a5b640 1 --2- 172.21.15.16:0/2411055612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bc0 0x7f488c1103f0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.725+0000 7f4893a5b640 1 --2- 172.21.15.16:0/2411055612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c109270 0x7f488c109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.725+0000 7f4893a5b640 1 --2- 172.21.15.16:0/2411055612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c103930 0x7f488c103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.725+0000 7f4893a5b640 1 -- 172.21.15.16:0/2411055612 >> 172.21.15.16:0/2411055612 conn(0x7f488c0ff3a0 msgr2=0x7f488c1017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.725+0000 7f4893a5b640 1 -- 172.21.15.16:0/2411055612 shutdown_connections 2024-08-28T19:31:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.725+0000 7f4893a5b640 1 -- 172.21.15.16:0/2411055612 wait complete. 2024-08-28T19:31:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.726+0000 7f4893a5b640 1 Processor -- start 2024-08-28T19:31:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.726+0000 7f4893a5b640 1 -- start start 2024-08-28T19:31:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4893a5b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c103930 0x7f488c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4893a5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c109270 0x7f488c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4893a5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bc0 0x7f488c19fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4893a5b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f488c113880 con 0x7f488c109bc0 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4893a5b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f488c113700 con 0x7f488c109270 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4893a5b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f488c113a00 con 0x7f488c103930 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4891ff8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bc0 0x7f488c19fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4891ff8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bc0 0x7f488c19fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32994/0 (socket says 172.21.15.16:32994) 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4891ff8640 1 -- 172.21.15.16:0/51952743 learned_addr learned my addr 172.21.15.16:0/51952743 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f48917f7640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c103930 0x7f488c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4890ff6640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c109270 0x7f488c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4891ff8640 1 -- 172.21.15.16:0/51952743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c103930 msgr2=0x7f488c19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4891ff8640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c103930 0x7f488c19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4891ff8640 1 -- 172.21.15.16:0/51952743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c109270 msgr2=0x7f488c19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4891ff8640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c109270 0x7f488c19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4891ff8640 1 -- 172.21.15.16:0/51952743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c1a00a0 con 0x7f488c109bc0 2024-08-28T19:31:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.727+0000 7f4891ff8640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bc0 0x7f488c19fa30 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f4884005cc0 tx=0x7f488402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.728+0000 7f48827fc640 1 -- 172.21.15.16:0/51952743 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4884003bf0 con 0x7f488c109bc0 2024-08-28T19:31:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.728+0000 7f4893a5b640 1 -- 172.21.15.16:0/51952743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488c1a0330 con 0x7f488c109bc0 2024-08-28T19:31:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.728+0000 7f48827fc640 1 -- 172.21.15.16:0/51952743 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4884003d90 con 0x7f488c109bc0 2024-08-28T19:31:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.728+0000 7f48827fc640 1 -- 172.21.15.16:0/51952743 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488402fb20 con 0x7f488c109bc0 2024-08-28T19:31:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.728+0000 7f4893a5b640 1 -- 172.21.15.16:0/51952743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488c1a0870 con 0x7f488c109bc0 2024-08-28T19:31:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.730+0000 7f48827fc640 1 -- 172.21.15.16:0/51952743 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4884004580 con 0x7f488c109bc0 2024-08-28T19:31:29.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.731+0000 7f4899fd0740 1 Processor -- start 2024-08-28T19:31:29.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.731+0000 7f4899fd0740 1 -- start start 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.731+0000 7f48827fc640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4cd0 0x7f48780b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.731+0000 7f4890ff6640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4cd0 0x7f48780b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.731+0000 7f48827fc640 1 -- 172.21.15.16:0/51952743 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f4884106c50 con 0x7f488c109bc0 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.731+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556757708070 0x556757aad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.731+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757aad850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.731+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757aadd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.731+0000 7f4899fd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556757a24680 con 0x5567576e7390 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.731+0000 7f4899fd0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556757a13c10 con 0x556757708070 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.731+0000 7f4899fd0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556757a17bc0 con 0x5567576df1b0 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f4891ff8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757aadd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f4891ff8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757aadd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33010/0 (socket says 172.21.15.16:33010) 2024-08-28T19:31:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f4891ff8640 1 -- 172.21.15.16:0/345614126 learned_addr learned my addr 172.21.15.16:0/345614126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f48917f7640 1 --2- 172.21.15.16:0/345614126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556757708070 0x556757aad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f4890ff6640 1 --2- 172.21.15.16:0/345614126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757aad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f4891ff8640 1 -- 172.21.15.16:0/345614126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 msgr2=0x556757aad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f4891ff8640 1 --2- 172.21.15.16:0/345614126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757aad850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f4890ff6640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4cd0 0x7f48780b71e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f488c102bf0 tx=0x7f487c00a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f4891ff8640 1 -- 172.21.15.16:0/345614126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556757708070 msgr2=0x556757aad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f4891ff8640 1 --2- 172.21.15.16:0/345614126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556757708070 0x556757aad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f4891ff8640 1 -- 172.21.15.16:0/345614126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556757773650 con 0x5567576e7390 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f48917f7640 1 --2- 172.21.15.16:0/345614126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556757708070 0x556757aad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f4891ff8640 1 --2- 172.21.15.16:0/345614126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757aadd90 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f4884009bf0 tx=0x7f48840cc740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368ba4b7b9a88070 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f48777fe640 1 -- 172.21.15.16:0/345614126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48840cd610 con 0x5567576e7390 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f48777fe640 1 -- 172.21.15.16:0/345614126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4884036040 con 0x5567576e7390 2024-08-28T19:31:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.732+0000 7f48777fe640 1 -- 172.21.15.16:0/345614126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48840cd170 con 0x5567576e7390 2024-08-28T19:31:29.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.733+0000 7f4899fd0740 1 -- 172.21.15.16:0/345614126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 msgr2=0x556757aadd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:29.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.733+0000 7f4899fd0740 1 --2- 172.21.15.16:0/345614126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757aadd90 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f4884009bf0 tx=0x7f48840cc740 comp rx=0 tx=0).stop 2024-08-28T19:31:29.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.733+0000 7f4899fd0740 1 -- 172.21.15.16:0/345614126 shutdown_connections 2024-08-28T19:31:29.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.733+0000 7f4899fd0740 1 --2- 172.21.15.16:0/345614126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757aadd90 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.733+0000 7f4899fd0740 1 --2- 172.21.15.16:0/345614126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757aad850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.733+0000 7f4899fd0740 1 --2- 172.21.15.16:0/345614126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556757708070 0x556757aad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.733+0000 7f4899fd0740 1 -- 172.21.15.16:0/345614126 >> 172.21.15.16:0/345614126 conn(0x5567576fc460 msgr2=0x556757aab200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.733+0000 7f4899fd0740 1 -- 172.21.15.16:0/345614126 shutdown_connections 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.733+0000 7f4899fd0740 1 -- 172.21.15.16:0/345614126 wait complete. 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4899fd0740 1 -- start start 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4899fd0740 1 Processor -- start 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757ab8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567576e7390 0x556757ab8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556757708070 0x556757ac4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4899fd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556757876700 con 0x556757708070 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4899fd0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55675799a750 con 0x5567576e7390 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4899fd0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556757a24680 con 0x5567576df1b0 2024-08-28T19:31:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4899fd0740 10 client.0 _ll_register_callbacks cb 0x5567579c9e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4891ff8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556757708070 0x556757ac4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4891ff8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556757708070 0x556757ac4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33016/0 (socket says 172.21.15.16:33016) 2024-08-28T19:31:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.734+0000 7f4891ff8640 1 -- 172.21.15.16:0/1545600394 learned_addr learned my addr 172.21.15.16:0/1545600394 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f48917f7640 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757ab8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4890ff6640 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567576e7390 0x556757ab8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4891ff8640 1 -- 172.21.15.16:0/1545600394 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 msgr2=0x556757ab8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4891ff8640 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757ab8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4891ff8640 1 -- 172.21.15.16:0/1545600394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567576e7390 msgr2=0x556757ab8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4891ff8640 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567576e7390 0x556757ab8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4891ff8640 1 -- 172.21.15.16:0/1545600394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567579d3900 con 0x556757708070 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4891ff8640 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556757708070 0x556757ac4990 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f4884006010 tx=0x7f4884004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba5b253e8283ef2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4874ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4874ff9640 1 -- 172.21.15.16:0/1545600394 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48840d1780 con 0x556757708070 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4874ff9640 1 -- 172.21.15.16:0/1545600394 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f48840cd170 con 0x556757708070 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4899fd0740 1 -- 172.21.15.16:0/1545600394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4884036040 con 0x556757708070 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4874ff9640 1 -- 172.21.15.16:0/1545600394 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f48840cd610 con 0x556757708070 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4899fd0740 10 client.18201 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.735+0000 7f4899fd0740 1 -- 172.21.15.16:0/1545600394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x556757ac7410 con 0x556757708070 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.736+0000 7f4874ff9640 1 -- 172.21.15.16:0/1545600394 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4884036040 con 0x556757708070 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.736+0000 7f4874ff9640 1 -- 172.21.15.16:0/1545600394 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f48840d5da0 con 0x556757708070 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.736+0000 7f4899fd0740 10 client.18201 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.736+0000 7f4899fd0740 10 client.18201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.736+0000 7f4899fd0740 20 client.18201 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.736+0000 7f4899fd0740 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5567576c69f0 0x556757aca860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.736+0000 7f4899fd0740 4 client.18201 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.736+0000 7f4899fd0740 1 -- 172.21.15.16:0/1545600394 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556757a265d0 con 0x5567576c69f0 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.736+0000 7f48917f7640 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5567576c69f0 0x556757aca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.737+0000 7f48917f7640 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5567576c69f0 0x556757aca860 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ce9c880fd32306 server_cookie=b6fe6fa618b100e1 in_seq=0 out_seq=0 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.737+0000 7f4874ff9640 10 client.18201 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.737+0000 7f4874ff9640 1 -- 172.21.15.16:0/1545600394 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556757a265d0 con 0x5567576c69f0 2024-08-28T19:31:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.737+0000 7f4874ff9640 10 client.18201 handle_command_reply: tid=0 2024-08-28T19:31:29.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.737+0000 7f4899fd0740 1 client.18201 shutdown 2024-08-28T19:31:29.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.737+0000 7f4899fd0740 1 -- 172.21.15.16:0/1545600394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556757708070 msgr2=0x556757ac4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:29.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.737+0000 7f4899fd0740 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556757708070 0x556757ac4990 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f4884006010 tx=0x7f4884004920 comp rx=0 tx=0).stop 2024-08-28T19:31:29.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.737+0000 7f4899fd0740 1 -- 172.21.15.16:0/1545600394 shutdown_connections 2024-08-28T19:31:29.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.737+0000 7f4899fd0740 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556757708070 0x556757ac4990 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.737+0000 7f4899fd0740 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567576e7390 0x556757ab8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.737+0000 7f4899fd0740 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5567576c69f0 0x556757aca860 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.738+0000 7f4899fd0740 1 --2- 172.21.15.16:0/1545600394 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757ab8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.738+0000 7f4874ff9640 0 client.18201 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:29.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.738+0000 7f4899fd0740 1 -- 172.21.15.16:0/1545600394 >> 172.21.15.16:0/1545600394 conn(0x5567576fc460 msgr2=0x556757ab04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:29.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.738+0000 7f4899fd0740 1 -- 172.21.15.16:0/1545600394 shutdown_connections 2024-08-28T19:31:29.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.738+0000 7f4899fd0740 1 -- 172.21.15.16:0/1545600394 wait complete. 2024-08-28T19:31:29.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.738+0000 7f4899fd0740 20 client.18201 trim_cache size 0 max 16384 2024-08-28T19:31:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4899fd0740 1 Processor -- start 2024-08-28T19:31:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4899fd0740 1 -- start start 2024-08-28T19:31:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757ab0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757ac4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556757708070 0x556757aaab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4899fd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1b0c90 con 0x5567576e7390 2024-08-28T19:31:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4899fd0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1b27d0 con 0x556757708070 2024-08-28T19:31:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4899fd0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1b1360 con 0x5567576df1b0 2024-08-28T19:31:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4890ff6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757ac4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4890ff6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757ac4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33028/0 (socket says 172.21.15.16:33028) 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4890ff6640 1 -- 172.21.15.16:0/3014858842 learned_addr learned my addr 172.21.15.16:0/3014858842 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f48917f7640 1 --2- 172.21.15.16:0/3014858842 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757ab0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4891ff8640 1 --2- 172.21.15.16:0/3014858842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556757708070 0x556757aaab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4890ff6640 1 -- 172.21.15.16:0/3014858842 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 msgr2=0x556757ab0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4890ff6640 1 --2- 172.21.15.16:0/3014858842 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757ab0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4890ff6640 1 -- 172.21.15.16:0/3014858842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556757708070 msgr2=0x556757aaab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4890ff6640 1 --2- 172.21.15.16:0/3014858842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556757708070 0x556757aaab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.752+0000 7f4890ff6640 1 -- 172.21.15.16:0/3014858842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556757aaf600 con 0x5567576e7390 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.753+0000 7f4890ff6640 1 --2- 172.21.15.16:0/3014858842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757ac4830 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f487c01d510 tx=0x7f487c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88ac1c0d9448f46 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.753+0000 7f4876ffd640 1 -- 172.21.15.16:0/3014858842 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c02b070 con 0x5567576e7390 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.753+0000 7f4876ffd640 1 -- 172.21.15.16:0/3014858842 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f487c01ddd0 con 0x5567576e7390 2024-08-28T19:31:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.753+0000 7f4876ffd640 1 -- 172.21.15.16:0/3014858842 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c016070 con 0x5567576e7390 2024-08-28T19:31:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.754+0000 7f4899fd0740 1 -- 172.21.15.16:0/3014858842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 msgr2=0x556757ac4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.754+0000 7f4899fd0740 1 --2- 172.21.15.16:0/3014858842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757ac4830 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f487c01d510 tx=0x7f487c012000 comp rx=0 tx=0).stop 2024-08-28T19:31:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.754+0000 7f4899fd0740 1 -- 172.21.15.16:0/3014858842 shutdown_connections 2024-08-28T19:31:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.754+0000 7f4899fd0740 1 --2- 172.21.15.16:0/3014858842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556757708070 0x556757aaab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.754+0000 7f4899fd0740 1 --2- 172.21.15.16:0/3014858842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576e7390 0x556757ac4830 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.754+0000 7f4899fd0740 1 --2- 172.21.15.16:0/3014858842 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5567576df1b0 0x556757ab0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.754+0000 7f4899fd0740 1 -- 172.21.15.16:0/3014858842 >> 172.21.15.16:0/3014858842 conn(0x5567576fc460 msgr2=0x556757ab0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.754+0000 7f4899fd0740 1 -- 172.21.15.16:0/3014858842 shutdown_connections 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.754+0000 7f4899fd0740 1 -- 172.21.15.16:0/3014858842 wait complete. 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.754+0000 7f4899fd0740 1 -- start start 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.754+0000 7f4899fd0740 1 Processor -- start 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576df1b0 0x556757abcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567576e7390 0x556757abd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4899fd0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556757708070 0x556757b341d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4899fd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1b2100 con 0x5567576df1b0 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4899fd0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1b1a30 con 0x5567576e7390 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4899fd0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1aec90 con 0x556757708070 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4899fd0740 10 client.0 _ll_register_callbacks cb 0x7f4884104400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f48917f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576df1b0 0x556757abcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f48917f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576df1b0 0x556757abcb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33030/0 (socket says 172.21.15.16:33030) 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f48917f7640 1 -- 172.21.15.16:0/270371525 learned_addr learned my addr 172.21.15.16:0/270371525 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4891ff8640 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556757708070 0x556757b341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4890ff6640 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567576e7390 0x556757abd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f48917f7640 1 -- 172.21.15.16:0/270371525 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556757708070 msgr2=0x556757b341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f48917f7640 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556757708070 0x556757b341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f48917f7640 1 -- 172.21.15.16:0/270371525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567576e7390 msgr2=0x556757abd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f48917f7640 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567576e7390 0x556757abd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f48917f7640 1 -- 172.21.15.16:0/270371525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556757abd8c0 con 0x5567576df1b0 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4890ff6640 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567576e7390 0x556757abd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f48917f7640 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576df1b0 0x556757abcb30 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f4888005d00 tx=0x7f4888012a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70afa3c073bd24a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4874ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.755+0000 7f4874ff9640 1 -- 172.21.15.16:0/270371525 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888028070 con 0x5567576df1b0 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4874ff9640 1 -- 172.21.15.16:0/270371525 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f4888004220 con 0x5567576df1b0 2024-08-28T19:31:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4899fd0740 1 -- 172.21.15.16:0/270371525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f487c01ddd0 con 0x5567576df1b0 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4874ff9640 1 -- 172.21.15.16:0/270371525 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f4888032050 con 0x5567576df1b0 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4899fd0740 10 client.18207 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4899fd0740 1 -- 172.21.15.16:0/270371525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x556757b357b0 con 0x5567576df1b0 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4874ff9640 1 -- 172.21.15.16:0/270371525 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f487c01ddd0 con 0x5567576df1b0 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4874ff9640 1 -- 172.21.15.16:0/270371525 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f4888005460 con 0x5567576df1b0 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4899fd0740 10 client.18207 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4899fd0740 10 client.18207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4899fd0740 20 client.18207 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4899fd0740 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55675769bda0 0x556757b38020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4899fd0740 4 client.18207 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.756+0000 7f4899fd0740 1 -- 172.21.15.16:0/270371525 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556757a22070 con 0x55675769bda0 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4890ff6640 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55675769bda0 0x556757b38020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4890ff6640 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55675769bda0 0x556757b38020 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57127ce3a141e42c server_cookie=c3d0c8cc711e09f6 in_seq=0 out_seq=0 2024-08-28T19:31:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4874ff9640 10 client.18207 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4874ff9640 1 -- 172.21.15.16:0/270371525 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556757a22070 con 0x55675769bda0 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4874ff9640 10 client.18207 handle_command_reply: tid=0 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4899fd0740 1 client.18207 shutdown 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4899fd0740 1 -- 172.21.15.16:0/270371525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576df1b0 msgr2=0x556757abcb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4899fd0740 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576df1b0 0x556757abcb30 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f4888005d00 tx=0x7f4888012a00 comp rx=0 tx=0).stop 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4899fd0740 1 -- 172.21.15.16:0/270371525 shutdown_connections 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4899fd0740 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55675769bda0 0x556757b38020 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4899fd0740 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556757708070 0x556757b341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4899fd0740 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567576e7390 0x556757abd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4899fd0740 1 --2- 172.21.15.16:0/270371525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567576df1b0 0x556757abcb30 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4899fd0740 1 -- 172.21.15.16:0/270371525 >> 172.21.15.16:0/270371525 conn(0x5567576fc460 msgr2=0x556757aaab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.757+0000 7f4874ff9640 0 client.18207 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.758+0000 7f4899fd0740 1 -- 172.21.15.16:0/270371525 shutdown_connections 2024-08-28T19:31:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.758+0000 7f4899fd0740 1 -- 172.21.15.16:0/270371525 wait complete. 2024-08-28T19:31:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.758+0000 7f4899fd0740 20 client.18207 trim_cache size 0 max 16384 2024-08-28T19:31:29.764 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:31:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.758+0000 7f4893a5b640 1 -- 172.21.15.16:0/51952743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4cd0 msgr2=0x7f48780b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.758+0000 7f4893a5b640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4cd0 0x7f48780b71e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f488c102bf0 tx=0x7f487c00a2d0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.758+0000 7f4893a5b640 1 -- 172.21.15.16:0/51952743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bc0 msgr2=0x7f488c19fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.758+0000 7f4893a5b640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bc0 0x7f488c19fa30 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f4884005cc0 tx=0x7f488402f380 comp rx=0 tx=0).stop 2024-08-28T19:31:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.759+0000 7f4893a5b640 1 -- 172.21.15.16:0/51952743 shutdown_connections 2024-08-28T19:31:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.759+0000 7f4893a5b640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c109bc0 0x7f488c19fa30 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.759+0000 7f4893a5b640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48780b4cd0 0x7f48780b71e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.759+0000 7f4893a5b640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f488c109270 0x7f488c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.759+0000 7f4893a5b640 1 --2- 172.21.15.16:0/51952743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f488c103930 0x7f488c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.759+0000 7f4893a5b640 1 -- 172.21.15.16:0/51952743 >> 172.21.15.16:0/51952743 conn(0x7f488c0ff3a0 msgr2=0x7f488c1088d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.759+0000 7f4893a5b640 1 -- 172.21.15.16:0/51952743 shutdown_connections 2024-08-28T19:31:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:29.759+0000 7f4893a5b640 1 -- 172.21.15.16:0/51952743 wait complete. 2024-08-28T19:31:29.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.82 MB/sec execute 429 sec latency 15134.854 ms 2024-08-28T19:31:30.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.81 MB/sec execute 430 sec latency 16134.949 ms 2024-08-28T19:31:31.326 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:31:31.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:31:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:31 smithi028 ceph-mon[21769]: pgmap v661: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 381 KiB/s wr, 27 op/s 2024-08-28T19:31:31.594 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:31.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:31 smithi115 ceph-mon[22184]: pgmap v661: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 381 KiB/s wr, 27 op/s 2024-08-28T19:31:31.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:31 smithi016 ceph-mon[18556]: pgmap v661: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 381 KiB/s wr, 27 op/s 2024-08-28T19:31:31.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.80 MB/sec execute 431 sec latency 17134.993 ms 2024-08-28T19:31:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.102+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/1757854351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a4108aa0 msgr2=0x7fd5a410f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.102+0000 7fd5a9a1c640 1 --2- 172.21.15.16:0/1757854351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a4108aa0 0x7fd5a410f2d0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009ca0 tx=0x7fd59c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.102+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/1757854351 shutdown_connections 2024-08-28T19:31:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.102+0000 7fd5a9a1c640 1 --2- 172.21.15.16:0/1757854351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a4108aa0 0x7fd5a410f2d0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.102+0000 7fd5a9a1c640 1 --2- 172.21.15.16:0/1757854351 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd5a4108150 0x7fd5a4108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.102+0000 7fd5a9a1c640 1 --2- 172.21.15.16:0/1757854351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a4102810 0x7fd5a4102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.102+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/1757854351 >> 172.21.15.16:0/1757854351 conn(0x7fd5a40794b0 msgr2=0x7fd5a40798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.102+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/1757854351 shutdown_connections 2024-08-28T19:31:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.102+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/1757854351 wait complete. 2024-08-28T19:31:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a9a1c640 1 Processor -- start 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a9a1c640 1 -- start start 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a9a1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a4102810 0x7fd5a419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a9a1c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a4108150 0x7fd5a419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a9a1c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd5a4108aa0 0x7fd5a419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a9a1c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a4112770 con 0x7fd5a4108150 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a9a1c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41125f0 con 0x7fd5a4102810 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a9a1c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41128f0 con 0x7fd5a4108aa0 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a4108150 0x7fd5a419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a4108150 0x7fd5a419a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33038/0 (socket says 172.21.15.16:33038) 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a27fc640 1 -- 172.21.15.16:0/985368860 learned_addr learned my addr 172.21.15.16:0/985368860 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.103+0000 7fd5a2ffd640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a4102810 0x7fd5a419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a37fe640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd5a4108aa0 0x7fd5a419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a27fc640 1 -- 172.21.15.16:0/985368860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd5a4108aa0 msgr2=0x7fd5a419fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd5a4108aa0 0x7fd5a419fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a27fc640 1 -- 172.21.15.16:0/985368860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a4102810 msgr2=0x7fd5a419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a4102810 0x7fd5a419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a27fc640 1 -- 172.21.15.16:0/985368860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a41a0100 con 0x7fd5a4108150 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a4108150 0x7fd5a419a7f0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fd59800e9c0 tx=0x7fd59800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a8a1a640 1 -- 172.21.15.16:0/985368860 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59800ccc0 con 0x7fd5a4108150 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/985368860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a41a03f0 con 0x7fd5a4108150 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a8a1a640 1 -- 172.21.15.16:0/985368860 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5980054d0 con 0x7fd5a4108150 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a8a1a640 1 -- 172.21.15.16:0/985368860 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598010430 con 0x7fd5a4108150 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.104+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/985368860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a41a6c10 con 0x7fd5a4108150 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.107+0000 7fd5aff91740 1 Processor -- start 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.107+0000 7fd5a8a1a640 1 -- 172.21.15.16:0/985368860 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fd598004f20 con 0x7fd5a4108150 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.107+0000 7fd5aff91740 1 -- start start 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.107+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf81e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.107+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf44f490 0x5625cf81ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.107+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf457670 0x5625cf81f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.107+0000 7fd5aff91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5625cf7945f0 con 0x5625cf478350 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.107+0000 7fd5aff91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625cf783b80 con 0x5625cf44f490 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.107+0000 7fd5aff91740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5625cf787b30 con 0x5625cf457670 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.107+0000 7fd5a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf81e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf81e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33044/0 (socket says 172.21.15.16:33044) 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a37fe640 1 -- 172.21.15.16:0/1220981795 learned_addr learned my addr 172.21.15.16:0/1220981795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a2ffd640 1 --2- 172.21.15.16:0/1220981795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf44f490 0x5625cf81ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/1220981795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf457670 0x5625cf81f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a37fe640 1 -- 172.21.15.16:0/1220981795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf457670 msgr2=0x5625cf81f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a37fe640 1 --2- 172.21.15.16:0/1220981795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf457670 0x5625cf81f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a8a1a640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd58c0b4cd0 0x7fd58c0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a37fe640 1 -- 172.21.15.16:0/1220981795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf44f490 msgr2=0x5625cf81ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a37fe640 1 --2- 172.21.15.16:0/1220981795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf44f490 0x5625cf81ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a37fe640 1 -- 172.21.15.16:0/1220981795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625cf4e3930 con 0x5625cf478350 2024-08-28T19:31:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/1220981795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf457670 0x5625cf81f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:32.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a8a1a640 1 -- 172.21.15.16:0/985368860 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fd5980e5cc0 con 0x7fd5a4108150 2024-08-28T19:31:32.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a37fe640 1 --2- 172.21.15.16:0/1220981795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf81e780 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0098b0 tx=0x7fd59c0098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b2c2c081567aa1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:32.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd5a37fe640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd58c0b4cd0 0x7fd58c0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd589ffb640 1 -- 172.21.15.16:0/1220981795 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c04a040 con 0x5625cf478350 2024-08-28T19:31:32.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd589ffb640 1 -- 172.21.15.16:0/1220981795 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd59c036030 con 0x5625cf478350 2024-08-28T19:31:32.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.108+0000 7fd589ffb640 1 -- 172.21.15.16:0/1220981795 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c004780 con 0x5625cf478350 2024-08-28T19:31:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.109+0000 7fd5aff91740 1 -- 172.21.15.16:0/1220981795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 msgr2=0x5625cf81e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.109+0000 7fd5a37fe640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd58c0b4cd0 0x7fd58c0b71e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd59c045000 tx=0x7fd59c02f720 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.109+0000 7fd5aff91740 1 --2- 172.21.15.16:0/1220981795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf81e780 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0098b0 tx=0x7fd59c0098e0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.109+0000 7fd5aff91740 1 -- 172.21.15.16:0/1220981795 shutdown_connections 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.109+0000 7fd5aff91740 1 --2- 172.21.15.16:0/1220981795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf457670 0x5625cf81f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.109+0000 7fd5aff91740 1 --2- 172.21.15.16:0/1220981795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf44f490 0x5625cf81ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.109+0000 7fd5aff91740 1 --2- 172.21.15.16:0/1220981795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf81e780 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.109+0000 7fd5aff91740 1 -- 172.21.15.16:0/1220981795 >> 172.21.15.16:0/1220981795 conn(0x5625cf46c740 msgr2=0x5625cf81c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.110+0000 7fd5aff91740 1 -- 172.21.15.16:0/1220981795 shutdown_connections 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.110+0000 7fd5aff91740 1 -- 172.21.15.16:0/1220981795 wait complete. 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.110+0000 7fd5aff91740 1 -- start start 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.110+0000 7fd5aff91740 1 Processor -- start 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf44f490 0x5625cf82e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf82e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf478350 0x5625cf83a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5aff91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5625cf5e6890 con 0x5625cf457670 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5aff91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625cf70a790 con 0x5625cf478350 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5aff91740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5625cf8310e0 con 0x5625cf44f490 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5aff91740 10 client.0 _ll_register_callbacks cb 0x5625cf48fba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf82e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf82e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33050/0 (socket says 172.21.15.16:33050) 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a27fc640 1 -- 172.21.15.16:0/105638565 learned_addr learned my addr 172.21.15.16:0/105638565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a37fe640 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf478350 0x5625cf83a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a2ffd640 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf44f490 0x5625cf82e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a27fc640 1 -- 172.21.15.16:0/105638565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf44f490 msgr2=0x5625cf82e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf44f490 0x5625cf82e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a27fc640 1 -- 172.21.15.16:0/105638565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf478350 msgr2=0x5625cf83a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf478350 0x5625cf83a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a27fc640 1 -- 172.21.15.16:0/105638565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625cf741f30 con 0x5625cf457670 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a2ffd640 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf44f490 0x5625cf82e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf82e8a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fd598004650 tx=0x7fd5980abe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f77505ee69e9cdc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd582ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd582ffd640 1 -- 172.21.15.16:0/105638565 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5980e7070 con 0x5625cf457670 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd582ffd640 1 -- 172.21.15.16:0/105638565 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fd5980a8270 con 0x5625cf457670 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.111+0000 7fd5aff91740 1 -- 172.21.15.16:0/105638565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd59c036030 con 0x5625cf457670 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd582ffd640 1 -- 172.21.15.16:0/105638565 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fd5980abaf0 con 0x5625cf457670 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd5aff91740 10 client.18219 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd5aff91740 1 -- 172.21.15.16:0/105638565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5625cf83ce90 con 0x5625cf457670 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd582ffd640 1 -- 172.21.15.16:0/105638565 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd59c036030 con 0x5625cf457670 2024-08-28T19:31:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd582ffd640 1 -- 172.21.15.16:0/105638565 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7fd5980b4e70 con 0x5625cf457670 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd5aff91740 10 client.18219 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd5aff91740 10 client.18219 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd5aff91740 20 client.18219 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd5aff91740 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5625cf436cd0 0x5625cf840380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd5aff91740 4 client.18219 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd5aff91740 1 -- 172.21.15.16:0/105638565 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625cf796540 con 0x5625cf436cd0 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.112+0000 7fd5a2ffd640 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5625cf436cd0 0x5625cf840380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.113+0000 7fd582ffd640 10 client.18219 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.113+0000 7fd5a2ffd640 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5625cf436cd0 0x5625cf840380 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3b3df01fd2048b1 server_cookie=65db6fd8bbabcece in_seq=0 out_seq=0 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd582ffd640 1 -- 172.21.15.16:0/105638565 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5625cf796540 con 0x5625cf436cd0 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd582ffd640 10 client.18219 handle_command_reply: tid=0 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 1 client.18219 shutdown 2024-08-28T19:31:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 1 -- 172.21.15.16:0/105638565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 msgr2=0x5625cf82e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf82e8a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fd598004650 tx=0x7fd5980abe70 comp rx=0 tx=0).stop 2024-08-28T19:31:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 1 -- 172.21.15.16:0/105638565 shutdown_connections 2024-08-28T19:31:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf478350 0x5625cf83a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf82e8a0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5625cf436cd0 0x5625cf840380 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 1 --2- 172.21.15.16:0/105638565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf44f490 0x5625cf82e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 1 -- 172.21.15.16:0/105638565 >> 172.21.15.16:0/105638565 conn(0x5625cf46c740 msgr2=0x5625cf820170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd582ffd640 0 client.18219 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 1 -- 172.21.15.16:0/105638565 shutdown_connections 2024-08-28T19:31:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 1 -- 172.21.15.16:0/105638565 wait complete. 2024-08-28T19:31:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.114+0000 7fd5aff91740 20 client.18219 trim_cache size 0 max 16384 2024-08-28T19:31:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.132+0000 7fd5aff91740 1 Processor -- start 2024-08-28T19:31:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.132+0000 7fd5aff91740 1 -- start start 2024-08-28T19:31:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.132+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf44f490 0x5625cf81c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.132+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf829b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.132+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf478350 0x5625cf82ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.132+0000 7fd5aff91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41ae960 con 0x5625cf457670 2024-08-28T19:31:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.132+0000 7fd5aff91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41b04a0 con 0x5625cf44f490 2024-08-28T19:31:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.132+0000 7fd5aff91740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41af030 con 0x5625cf478350 2024-08-28T19:31:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf829b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf829b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33056/0 (socket says 172.21.15.16:33056) 2024-08-28T19:31:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a27fc640 1 -- 172.21.15.16:0/354376356 learned_addr learned my addr 172.21.15.16:0/354376356 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a2ffd640 1 --2- 172.21.15.16:0/354376356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf44f490 0x5625cf81c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a37fe640 1 --2- 172.21.15.16:0/354376356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf478350 0x5625cf82ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a27fc640 1 -- 172.21.15.16:0/354376356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf478350 msgr2=0x5625cf82ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/354376356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf478350 0x5625cf82ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a27fc640 1 -- 172.21.15.16:0/354376356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf44f490 msgr2=0x5625cf81c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/354376356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf44f490 0x5625cf81c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a27fc640 1 -- 172.21.15.16:0/354376356 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625cf82f620 con 0x5625cf457670 2024-08-28T19:31:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a2ffd640 1 --2- 172.21.15.16:0/354376356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf44f490 0x5625cf81c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/354376356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf829b10 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fd5980abce0 tx=0x7fd598009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fce0aa62102a9e8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5897fa640 1 -- 172.21.15.16:0/354376356 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5980ab750 con 0x5625cf457670 2024-08-28T19:31:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5897fa640 1 -- 172.21.15.16:0/354376356 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd598097360 con 0x5625cf457670 2024-08-28T19:31:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.133+0000 7fd5897fa640 1 -- 172.21.15.16:0/354376356 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5980a8270 con 0x5625cf457670 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.134+0000 7fd5aff91740 1 -- 172.21.15.16:0/354376356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 msgr2=0x5625cf829b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.134+0000 7fd5aff91740 1 --2- 172.21.15.16:0/354376356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf829b10 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fd5980abce0 tx=0x7fd598009310 comp rx=0 tx=0).stop 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.134+0000 7fd5aff91740 1 -- 172.21.15.16:0/354376356 shutdown_connections 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.134+0000 7fd5aff91740 1 --2- 172.21.15.16:0/354376356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf478350 0x5625cf82ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.134+0000 7fd5aff91740 1 --2- 172.21.15.16:0/354376356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf457670 0x5625cf829b10 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.134+0000 7fd5aff91740 1 --2- 172.21.15.16:0/354376356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf44f490 0x5625cf81c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.134+0000 7fd5aff91740 1 -- 172.21.15.16:0/354376356 >> 172.21.15.16:0/354376356 conn(0x5625cf46c740 msgr2=0x5625cf82f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.134+0000 7fd5aff91740 1 -- 172.21.15.16:0/354376356 shutdown_connections 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.134+0000 7fd5aff91740 1 -- 172.21.15.16:0/354376356 wait complete. 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.135+0000 7fd5aff91740 1 -- start start 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.135+0000 7fd5aff91740 1 Processor -- start 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.135+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf44f490 0x5625cf832e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.135+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf457670 0x5625cf8333e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.135+0000 7fd5aff91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf8a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.135+0000 7fd5aff91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41afdd0 con 0x5625cf478350 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.135+0000 7fd5aff91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41af700 con 0x5625cf457670 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.135+0000 7fd5aff91740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41ac960 con 0x5625cf44f490 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.135+0000 7fd5aff91740 10 client.0 _ll_register_callbacks cb 0x7fd59800eef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.135+0000 7fd5a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf8a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.135+0000 7fd5a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf8a3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33070/0 (socket says 172.21.15.16:33070) 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5a37fe640 1 -- 172.21.15.16:0/2453438122 learned_addr learned my addr 172.21.15.16:0/2453438122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5a2ffd640 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf44f490 0x5625cf832e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5a27fc640 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf457670 0x5625cf8333e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5a37fe640 1 -- 172.21.15.16:0/2453438122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf44f490 msgr2=0x5625cf832e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5a37fe640 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf44f490 0x5625cf832e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5a37fe640 1 -- 172.21.15.16:0/2453438122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf457670 msgr2=0x5625cf8333e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5a37fe640 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf457670 0x5625cf8333e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5a37fe640 1 -- 172.21.15.16:0/2453438122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625cf8a4450 con 0x5625cf478350 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5a37fe640 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf8a3de0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd59c02fc20 tx=0x7fd59c043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336346b8a7e392d7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd582ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd582ffd640 1 -- 172.21.15.16:0/2453438122 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c061030 con 0x5625cf478350 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd582ffd640 1 -- 172.21.15.16:0/2453438122 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7fd59c005bf0 con 0x5625cf478350 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5aff91740 1 -- 172.21.15.16:0/2453438122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd598097360 con 0x5625cf478350 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd582ffd640 1 -- 172.21.15.16:0/2453438122 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fd59c057040 con 0x5625cf478350 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5aff91740 10 client.18225 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.136+0000 7fd5aff91740 1 -- 172.21.15.16:0/2453438122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5625cf8ab8b0 con 0x5625cf478350 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.137+0000 7fd582ffd640 1 -- 172.21.15.16:0/2453438122 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd598097360 con 0x5625cf478350 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.137+0000 7fd582ffd640 1 -- 172.21.15.16:0/2453438122 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7fd59c043070 con 0x5625cf478350 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.137+0000 7fd5aff91740 10 client.18225 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.137+0000 7fd5aff91740 10 client.18225 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.137+0000 7fd5aff91740 20 client.18225 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.137+0000 7fd5aff91740 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5625cf40c080 0x5625cf8addb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:32.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.137+0000 7fd5aff91740 4 client.18225 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:31:32.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.137+0000 7fd5aff91740 1 -- 172.21.15.16:0/2453438122 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5625cf791fe0 con 0x5625cf40c080 2024-08-28T19:31:32.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.137+0000 7fd5a2ffd640 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5625cf40c080 0x5625cf8addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:32.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.138+0000 7fd582ffd640 10 client.18225 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:32.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.138+0000 7fd5a2ffd640 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5625cf40c080 0x5625cf8addb0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddadb33b76f0b593 server_cookie=68acb1ca20e77601 in_seq=0 out_seq=0 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd582ffd640 1 -- 172.21.15.16:0/2453438122 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5625cf791fe0 con 0x5625cf40c080 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd582ffd640 10 client.18225 handle_command_reply: tid=0 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 1 client.18225 shutdown 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 1 -- 172.21.15.16:0/2453438122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 msgr2=0x5625cf8a3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf8a3de0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd59c02fc20 tx=0x7fd59c043000 comp rx=0 tx=0).stop 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 1 -- 172.21.15.16:0/2453438122 shutdown_connections 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625cf478350 0x5625cf8a3de0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625cf457670 0x5625cf8333e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5625cf40c080 0x5625cf8addb0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 1 --2- 172.21.15.16:0/2453438122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625cf44f490 0x5625cf832e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 1 -- 172.21.15.16:0/2453438122 >> 172.21.15.16:0/2453438122 conn(0x5625cf46c740 msgr2=0x5625cf820150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd582ffd640 0 client.18225 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:32.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 1 -- 172.21.15.16:0/2453438122 shutdown_connections 2024-08-28T19:31:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 1 -- 172.21.15.16:0/2453438122 wait complete. 2024-08-28T19:31:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.140+0000 7fd5aff91740 20 client.18225 trim_cache size 0 max 16384 2024-08-28T19:31:32.144 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:31:32.144 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:31:32.144 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "af6ece82-89e0-4ca5-a70b-36769e0fcc98", 2024-08-28T19:31:32.144 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:31:32.144 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:31:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/985368860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd58c0b4cd0 msgr2=0x7fd58c0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd58c0b4cd0 0x7fd58c0b71e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd59c045000 tx=0x7fd59c02f720 comp rx=0 tx=0).stop 2024-08-28T19:31:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/985368860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a4108150 msgr2=0x7fd5a419a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a4108150 0x7fd5a419a7f0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fd59800e9c0 tx=0x7fd59800edc0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/985368860 shutdown_connections 2024-08-28T19:31:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd5a4108aa0 0x7fd5a419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a4108150 0x7fd5a419a7f0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd58c0b4cd0 0x7fd58c0b71e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 --2- 172.21.15.16:0/985368860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a4102810 0x7fd5a419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/985368860 >> 172.21.15.16:0/985368860 conn(0x7fd5a40794b0 msgr2=0x7fd5a41017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:32.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/985368860 shutdown_connections 2024-08-28T19:31:32.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:32.141+0000 7fd5a9a1c640 1 -- 172.21.15.16:0/985368860 wait complete. 2024-08-28T19:31:32.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:32 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:31:32.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:32 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:31:32.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:32 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:31:32.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:31:32.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.79 MB/sec execute 432 sec latency 18135.015 ms 2024-08-28T19:31:33.001 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:33.402 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: pgmap v662: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 390 KiB/s wr, 29 op/s 2024-08-28T19:31:33.402 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:31:33.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:31:33.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:33 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.494+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/2149352636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f4102d00 msgr2=0x7ff3f410fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.494+0000 7ff3fbdcd640 1 --2- 172.21.15.16:0/2149352636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f4102d00 0x7ff3f410fa20 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8009900 tx=0x7ff3e802a480 comp rx=0 tx=0).stop 2024-08-28T19:31:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.494+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/2149352636 shutdown_connections 2024-08-28T19:31:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.494+0000 7ff3fbdcd640 1 --2- 172.21.15.16:0/2149352636 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f410ff60 0x7ff3f4112380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.494+0000 7ff3fbdcd640 1 --2- 172.21.15.16:0/2149352636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f4102d00 0x7ff3f410fa20 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.494+0000 7ff3fbdcd640 1 --2- 172.21.15.16:0/2149352636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f41023b0 0x7ff3f41027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.494+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/2149352636 >> 172.21.15.16:0/2149352636 conn(0x7ff3f40794f0 msgr2=0x7ff3f4079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.494+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/2149352636 shutdown_connections 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.494+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/2149352636 wait complete. 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.495+0000 7ff3fbdcd640 1 Processor -- start 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.495+0000 7ff3fbdcd640 1 -- start start 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.495+0000 7ff3fbdcd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f41023b0 0x7ff3f419c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.495+0000 7ff3fbdcd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f4102d00 0x7ff3f419ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.495+0000 7ff3fbdcd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f410ff60 0x7ff3f41a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.495+0000 7ff3fbdcd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f4115a50 con 0x7ff3f41023b0 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.495+0000 7ff3fbdcd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f41158d0 con 0x7ff3f410ff60 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.495+0000 7ff3fbdcd640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f4115bd0 con 0x7ff3f4102d00 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.495+0000 7ff3f9b69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f41023b0 0x7ff3f419c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.495+0000 7ff3f9b69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f41023b0 0x7ff3f419c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33078/0 (socket says 172.21.15.16:33078) 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.495+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1650787875 learned_addr learned my addr 172.21.15.16:0/1650787875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3fa36a640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f410ff60 0x7ff3f41a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3f9368640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f4102d00 0x7ff3f419ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1650787875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f4102d00 msgr2=0x7ff3f419ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f4102d00 0x7ff3f419ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1650787875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f410ff60 msgr2=0x7ff3f41a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f410ff60 0x7ff3f41a1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1650787875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f41a23d0 con 0x7ff3f41023b0 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3f9368640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f4102d00 0x7ff3f419ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f41023b0 0x7ff3f419c460 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000d950 tx=0x7ff3f000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3e6ffd640 1 -- 172.21.15.16:0/1650787875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0017070 con 0x7ff3f41023b0 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3e6ffd640 1 -- 172.21.15.16:0/1650787875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f0009c30 con 0x7ff3f41023b0 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3e6ffd640 1 -- 172.21.15.16:0/1650787875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f000f620 con 0x7ff3f41023b0 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/1650787875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f41a26c0 con 0x7ff3f41023b0 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.496+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/1650787875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f41a8e20 con 0x7ff3f41023b0 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.499+0000 7ff402342740 1 Processor -- start 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.499+0000 7ff402342740 1 -- start start 2024-08-28T19:31:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.499+0000 7ff3e6ffd640 1 -- 172.21.15.16:0/1650787875 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff3f0010460 con 0x7ff3f41023b0 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.499+0000 7ff402342740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a1a6070 0x55ca5a54b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.499+0000 7ff402342740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a17d1b0 0x55ca5a54b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.499+0000 7ff402342740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a185390 0x55ca5a54bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.499+0000 7ff402342740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ca5a4c2630 con 0x55ca5a185390 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.499+0000 7ff402342740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca5a4b1bc0 con 0x55ca5a1a6070 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.499+0000 7ff402342740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ca5a4b5b70 con 0x55ca5a17d1b0 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.499+0000 7ff3f9368640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a185390 0x55ca5a54bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3f9368640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a185390 0x55ca5a54bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33090/0 (socket says 172.21.15.16:33090) 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3f9368640 1 -- 172.21.15.16:0/1293254818 learned_addr learned my addr 172.21.15.16:0/1293254818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1293254818 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a17d1b0 0x55ca5a54b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3fa36a640 1 --2- 172.21.15.16:0/1293254818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a1a6070 0x55ca5a54b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3f9368640 1 -- 172.21.15.16:0/1293254818 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a17d1b0 msgr2=0x55ca5a54b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3f9368640 1 --2- 172.21.15.16:0/1293254818 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a17d1b0 0x55ca5a54b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3f9368640 1 -- 172.21.15.16:0/1293254818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a1a6070 msgr2=0x55ca5a54b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3f9368640 1 --2- 172.21.15.16:0/1293254818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a1a6070 0x55ca5a54b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3f9368640 1 -- 172.21.15.16:0/1293254818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca5a211650 con 0x55ca5a185390 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3f9368640 1 --2- 172.21.15.16:0/1293254818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a185390 0x55ca5a54bd40 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8000c00 tx=0x7ff3e8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76661342755e5a6a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3e6ffd640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3e00b4ca0 0x7ff3e00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3dffff640 1 -- 172.21.15.16:0/1293254818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e803f070 con 0x55ca5a185390 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3fa36a640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3e00b4ca0 0x7ff3e00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3dffff640 1 -- 172.21.15.16:0/1293254818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3e80316a0 con 0x55ca5a185390 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3dffff640 1 -- 172.21.15.16:0/1293254818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8035070 con 0x55ca5a185390 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.500+0000 7ff3e6ffd640 1 -- 172.21.15.16:0/1650787875 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7ff3f00e5620 con 0x7ff3f41023b0 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.501+0000 7ff402342740 1 -- 172.21.15.16:0/1293254818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a185390 msgr2=0x55ca5a54bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.501+0000 7ff402342740 1 --2- 172.21.15.16:0/1293254818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a185390 0x55ca5a54bd40 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8000c00 tx=0x7ff3e8032ce0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.501+0000 7ff3fa36a640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3e00b4ca0 0x7ff3e00b71b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00bd90 tx=0x7ff3ec00b320 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.502+0000 7ff402342740 1 -- 172.21.15.16:0/1293254818 shutdown_connections 2024-08-28T19:31:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.502+0000 7ff402342740 1 --2- 172.21.15.16:0/1293254818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a185390 0x55ca5a54bd40 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.502+0000 7ff402342740 1 --2- 172.21.15.16:0/1293254818 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a17d1b0 0x55ca5a54b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.502+0000 7ff402342740 1 --2- 172.21.15.16:0/1293254818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a1a6070 0x55ca5a54b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.502+0000 7ff402342740 1 -- 172.21.15.16:0/1293254818 >> 172.21.15.16:0/1293254818 conn(0x55ca5a19a460 msgr2=0x55ca5a5491b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.502+0000 7ff402342740 1 -- 172.21.15.16:0/1293254818 shutdown_connections 2024-08-28T19:31:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.502+0000 7ff402342740 1 -- 172.21.15.16:0/1293254818 wait complete. 2024-08-28T19:31:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.502+0000 7ff402342740 1 -- start start 2024-08-28T19:31:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff402342740 1 Processor -- start 2024-08-28T19:31:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff402342740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a556980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff402342740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a556ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff402342740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a1a6070 0x55ca5a562d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff402342740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ca5a314700 con 0x55ca5a17d1b0 2024-08-28T19:31:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff402342740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca5a438700 con 0x55ca5a185390 2024-08-28T19:31:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff402342740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ca5a559a40 con 0x55ca5a1a6070 2024-08-28T19:31:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff402342740 10 client.0 _ll_register_callbacks cb 0x55ca5a467dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff3f9b69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a556980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff3f9b69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a556980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33094/0 (socket says 172.21.15.16:33094) 2024-08-28T19:31:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1363053311 learned_addr learned my addr 172.21.15.16:0/1363053311 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff3f9368640 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a556ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.503+0000 7ff3fa36a640 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a1a6070 0x55ca5a562d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1363053311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a1a6070 msgr2=0x55ca5a562d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a1a6070 0x55ca5a562d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1363053311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 msgr2=0x55ca5a556ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a556ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1363053311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca5a4718b0 con 0x55ca5a17d1b0 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a556980 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7ff3f00e2a00 tx=0x7ff3f00ab7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f531b5f35083a4d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff3dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1363053311 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0020050 con 0x55ca5a17d1b0 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1363053311 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7ff3f00b47d0 con 0x55ca5a17d1b0 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff402342740 1 -- 172.21.15.16:0/1363053311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3e80316a0 con 0x55ca5a17d1b0 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1363053311 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7ff3f00a70d0 con 0x55ca5a17d1b0 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff402342740 10 client.18237 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff402342740 1 -- 172.21.15.16:0/1363053311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ca5a565830 con 0x55ca5a17d1b0 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.504+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1363053311 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3e80316a0 con 0x55ca5a17d1b0 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.505+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1363053311 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7ff3f00e51a0 con 0x55ca5a17d1b0 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.505+0000 7ff402342740 10 client.18237 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.505+0000 7ff402342740 10 client.18237 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:31:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.505+0000 7ff402342740 20 client.18237 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.505+0000 7ff402342740 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ca5a1649f0 0x55ca5a568e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.505+0000 7ff402342740 4 client.18237 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.505+0000 7ff402342740 1 -- 172.21.15.16:0/1363053311 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca5a4c4580 con 0x55ca5a1649f0 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.505+0000 7ff3f9368640 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ca5a1649f0 0x55ca5a568e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.505+0000 7ff3dd7fa640 10 client.18237 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.505+0000 7ff3f9368640 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ca5a1649f0 0x55ca5a568e30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85b27b427aacfd68 server_cookie=9a1f55ed9609c8c9 in_seq=0 out_seq=0 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1363053311 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ca5a4c4580 con 0x55ca5a1649f0 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff3dd7fa640 10 client.18237 handle_command_reply: tid=0 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff402342740 1 client.18237 shutdown 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff402342740 1 -- 172.21.15.16:0/1363053311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 msgr2=0x55ca5a556980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff402342740 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a556980 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7ff3f00e2a00 tx=0x7ff3f00ab7d0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff402342740 1 -- 172.21.15.16:0/1363053311 shutdown_connections 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff402342740 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ca5a1649f0 0x55ca5a568e30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff402342740 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a1a6070 0x55ca5a562d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff402342740 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a556ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff402342740 1 --2- 172.21.15.16:0/1363053311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a556980 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff3dd7fa640 0 client.18237 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.506+0000 7ff402342740 1 -- 172.21.15.16:0/1363053311 >> 172.21.15.16:0/1363053311 conn(0x55ca5a19a460 msgr2=0x55ca5a54e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.507+0000 7ff402342740 1 -- 172.21.15.16:0/1363053311 shutdown_connections 2024-08-28T19:31:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.507+0000 7ff402342740 1 -- 172.21.15.16:0/1363053311 wait complete. 2024-08-28T19:31:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.507+0000 7ff402342740 20 client.18237 trim_cache size 0 max 16384 2024-08-28T19:31:33.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff402342740 1 Processor -- start 2024-08-28T19:31:33.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff402342740 1 -- start start 2024-08-28T19:31:33.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff402342740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a5568f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff402342740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a54ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff402342740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a1a6070 0x55ca5a562c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff402342740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f41b2e90 con 0x55ca5a17d1b0 2024-08-28T19:31:33.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff402342740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f41b49d0 con 0x55ca5a185390 2024-08-28T19:31:33.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff402342740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f41b3560 con 0x55ca5a1a6070 2024-08-28T19:31:33.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff3f9b69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a5568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff3f9b69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a5568f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33104/0 (socket says 172.21.15.16:33104) 2024-08-28T19:31:33.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1489344114 learned_addr learned my addr 172.21.15.16:0/1489344114 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:33.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff3fa36a640 1 --2- 172.21.15.16:0/1489344114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a1a6070 0x55ca5a562c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff3f9368640 1 --2- 172.21.15.16:0/1489344114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a54ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1489344114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a1a6070 msgr2=0x55ca5a562c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:33.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1489344114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a1a6070 0x55ca5a562c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1489344114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 msgr2=0x55ca5a54ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:33.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1489344114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a54ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.525+0000 7ff3f9b69640 1 -- 172.21.15.16:0/1489344114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca5a5558e0 con 0x55ca5a17d1b0 2024-08-28T19:31:33.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.526+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1489344114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a5568f0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7ff3f00a8bb0 tx=0x7ff3f00ab840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd9d9c47169319b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:33.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.526+0000 7ff3df7fe640 1 -- 172.21.15.16:0/1489344114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0014030 con 0x55ca5a17d1b0 2024-08-28T19:31:33.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.526+0000 7ff3df7fe640 1 -- 172.21.15.16:0/1489344114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f00ab450 con 0x55ca5a17d1b0 2024-08-28T19:31:33.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.526+0000 7ff3df7fe640 1 -- 172.21.15.16:0/1489344114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0020050 con 0x55ca5a17d1b0 2024-08-28T19:31:33.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.527+0000 7ff402342740 1 -- 172.21.15.16:0/1489344114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 msgr2=0x55ca5a5568f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:33.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.527+0000 7ff402342740 1 --2- 172.21.15.16:0/1489344114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a5568f0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7ff3f00a8bb0 tx=0x7ff3f00ab840 comp rx=0 tx=0).stop 2024-08-28T19:31:33.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.527+0000 7ff402342740 1 -- 172.21.15.16:0/1489344114 shutdown_connections 2024-08-28T19:31:33.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.527+0000 7ff402342740 1 --2- 172.21.15.16:0/1489344114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a1a6070 0x55ca5a562c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.527+0000 7ff402342740 1 --2- 172.21.15.16:0/1489344114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a54ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.527+0000 7ff402342740 1 --2- 172.21.15.16:0/1489344114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a17d1b0 0x55ca5a5568f0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.527+0000 7ff402342740 1 -- 172.21.15.16:0/1489344114 >> 172.21.15.16:0/1489344114 conn(0x55ca5a19a460 msgr2=0x55ca5a54c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:33.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.527+0000 7ff402342740 1 -- 172.21.15.16:0/1489344114 shutdown_connections 2024-08-28T19:31:33.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.527+0000 7ff402342740 1 -- 172.21.15.16:0/1489344114 wait complete. 2024-08-28T19:31:33.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.527+0000 7ff402342740 1 -- start start 2024-08-28T19:31:33.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.527+0000 7ff402342740 1 Processor -- start 2024-08-28T19:31:33.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff402342740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a17d1b0 0x55ca5a55b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff402342740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a55b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff402342740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a1a6070 0x55ca5a5d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff402342740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f41b4300 con 0x55ca5a1a6070 2024-08-28T19:31:33.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff402342740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f41b3c30 con 0x55ca5a185390 2024-08-28T19:31:33.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff402342740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f41b0e90 con 0x55ca5a17d1b0 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff402342740 10 client.0 _ll_register_callbacks cb 0x7ff3f000df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3fa36a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a1a6070 0x55ca5a5d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3fa36a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a1a6070 0x55ca5a5d2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33106/0 (socket says 172.21.15.16:33106) 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3fa36a640 1 -- 172.21.15.16:0/1255449792 learned_addr learned my addr 172.21.15.16:0/1255449792 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a17d1b0 0x55ca5a55b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3f9368640 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a55b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3fa36a640 1 -- 172.21.15.16:0/1255449792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a17d1b0 msgr2=0x55ca5a55b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3fa36a640 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a17d1b0 0x55ca5a55b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3fa36a640 1 -- 172.21.15.16:0/1255449792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 msgr2=0x55ca5a55b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3fa36a640 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a55b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3fa36a640 1 -- 172.21.15.16:0/1255449792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca5a5d2d70 con 0x55ca5a1a6070 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3f9368640 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a55b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3fa36a640 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a1a6070 0x55ca5a5d2830 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec011f40 tx=0x7ff3ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d8f32da526cefd2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:33.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.528+0000 7ff3dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1255449792 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec02d060 con 0x55ca5a1a6070 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1255449792 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7ff3ec01d5c0 con 0x55ca5a1a6070 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff402342740 1 -- 172.21.15.16:0/1255449792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3f00ab450 con 0x55ca5a1a6070 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1255449792 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7ff3ec034050 con 0x55ca5a1a6070 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff402342740 10 client.18243 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff402342740 1 -- 172.21.15.16:0/1255449792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55ca5a5d3ee0 con 0x55ca5a1a6070 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1255449792 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f00ab450 con 0x55ca5a1a6070 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1255449792 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7ff3ec0288e0 con 0x55ca5a1a6070 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff402342740 10 client.18243 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff402342740 10 client.18243 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff402342740 20 client.18243 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff402342740 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ca5a139da0 0x55ca5a5d6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff402342740 4 client.18243 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.529+0000 7ff402342740 1 -- 172.21.15.16:0/1255449792 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ca5a4c0020 con 0x55ca5a139da0 2024-08-28T19:31:33.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.530+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ca5a139da0 0x55ca5a5d6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.530+0000 7ff3dd7fa640 10 client.18243 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.530+0000 7ff3f9b69640 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ca5a139da0 0x55ca5a5d6700 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdff6c28a0310eef server_cookie=f6096ea3725b7445 in_seq=0 out_seq=0 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.530+0000 7ff3dd7fa640 1 -- 172.21.15.16:0/1255449792 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4546 (crc 0 0 0) 0x55ca5a4c0020 con 0x55ca5a139da0 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.530+0000 7ff3dd7fa640 10 client.18243 handle_command_reply: tid=0 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.530+0000 7ff402342740 1 client.18243 shutdown 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.530+0000 7ff402342740 1 -- 172.21.15.16:0/1255449792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a1a6070 msgr2=0x55ca5a5d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.530+0000 7ff402342740 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a1a6070 0x55ca5a5d2830 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec011f40 tx=0x7ff3ec004a40 comp rx=0 tx=0).stop 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff402342740 1 -- 172.21.15.16:0/1255449792 shutdown_connections 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff402342740 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca5a1a6070 0x55ca5a5d2830 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff402342740 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca5a185390 0x55ca5a55b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff402342740 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ca5a139da0 0x55ca5a5d6700 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff402342740 1 --2- 172.21.15.16:0/1255449792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca5a17d1b0 0x55ca5a55b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff402342740 1 -- 172.21.15.16:0/1255449792 >> 172.21.15.16:0/1255449792 conn(0x55ca5a19a460 msgr2=0x55ca5a557c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:33.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff3dd7fa640 0 client.18243 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:33.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff402342740 1 -- 172.21.15.16:0/1255449792 shutdown_connections 2024-08-28T19:31:33.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff402342740 1 -- 172.21.15.16:0/1255449792 wait complete. 2024-08-28T19:31:33.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff402342740 20 client.18243 trim_cache size 0 max 16384 2024-08-28T19:31:33.537 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:31:33.537 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:31:33.537 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "af6ece82-89e0-4ca5-a70b-36769e0fcc98", 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:31:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:31:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:33.546 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:31:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/1650787875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3e00b4ca0 msgr2=0x7ff3e00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff3fbdcd640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3e00b4ca0 0x7ff3e00b71b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00bd90 tx=0x7ff3ec00b320 comp rx=0 tx=0).stop 2024-08-28T19:31:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/1650787875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f41023b0 msgr2=0x7ff3f419c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff3fbdcd640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f41023b0 0x7ff3f419c460 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000d950 tx=0x7ff3f000de10 comp rx=0 tx=0).stop 2024-08-28T19:31:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/1650787875 shutdown_connections 2024-08-28T19:31:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff3fbdcd640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff3e00b4ca0 0x7ff3e00b71b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff3fbdcd640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f410ff60 0x7ff3f41a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff3fbdcd640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f4102d00 0x7ff3f419ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff3fbdcd640 1 --2- 172.21.15.16:0/1650787875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f41023b0 0x7ff3f419c460 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.531+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/1650787875 >> 172.21.15.16:0/1650787875 conn(0x7ff3f40794f0 msgr2=0x7ff3f4109b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.532+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/1650787875 shutdown_connections 2024-08-28T19:31:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:33.532+0000 7ff3fbdcd640 1 -- 172.21.15.16:0/1650787875 wait complete. 2024-08-28T19:31:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: pgmap v662: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 390 KiB/s wr, 29 op/s 2024-08-28T19:31:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:31:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:31:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:31:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:31:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:33 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: pgmap v662: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 390 KiB/s wr, 29 op/s 2024-08-28T19:31:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:31:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:31:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:31:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:31:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:31:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:31:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:31:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:31:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:31:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:33 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:31:33.969 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.78 MB/sec execute 433 sec latency 19135.092 ms 2024-08-28T19:31:34.320 INFO:tasks.cephfs.filesystem:scrub status for tag:af6ece82-89e0-4ca5-a70b-36769e0fcc98 - {'path': '/', 'tag': 'af6ece82-89e0-4ca5-a70b-36769e0fcc98', 'options': 'recursive,force'} 2024-08-28T19:31:34.970 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.77 MB/sec execute 434 sec latency 20135.203 ms 2024-08-28T19:31:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:35 smithi028 ceph-mon[21769]: pgmap v663: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 370 KiB/s wr, 24 op/s 2024-08-28T19:31:35.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:35 smithi115 ceph-mon[22184]: pgmap v663: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 370 KiB/s wr, 24 op/s 2024-08-28T19:31:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:35 smithi016 ceph-mon[18556]: pgmap v663: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 370 KiB/s wr, 24 op/s 2024-08-28T19:31:35.970 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.76 MB/sec execute 435 sec latency 21135.286 ms 2024-08-28T19:31:36.970 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.75 MB/sec execute 436 sec latency 22135.353 ms 2024-08-28T19:31:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:37 smithi028 ceph-mon[21769]: pgmap v664: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 340 KiB/s wr, 25 op/s 2024-08-28T19:31:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:37 smithi115 ceph-mon[22184]: pgmap v664: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 340 KiB/s wr, 25 op/s 2024-08-28T19:31:37.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:37 smithi016 ceph-mon[18556]: pgmap v664: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 340 KiB/s wr, 25 op/s 2024-08-28T19:31:37.970 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.75 MB/sec execute 437 sec latency 23135.410 ms 2024-08-28T19:31:38.970 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.74 MB/sec execute 438 sec latency 24135.466 ms 2024-08-28T19:31:39.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:31:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:39 smithi028 ceph-mon[21769]: pgmap v665: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 160 KiB/s wr, 19 op/s 2024-08-28T19:31:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:39 smithi115 ceph-mon[22184]: pgmap v665: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 160 KiB/s wr, 19 op/s 2024-08-28T19:31:39.643 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:39.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:39 smithi016 ceph-mon[18556]: pgmap v665: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 160 KiB/s wr, 19 op/s 2024-08-28T19:31:39.970 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.73 MB/sec execute 439 sec latency 25135.499 ms 2024-08-28T19:31:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.230+0000 7f2a89258640 1 -- 172.21.15.16:0/1673465676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a84104750 msgr2=0x7f2a84104b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.230+0000 7f2a89258640 1 --2- 172.21.15.16:0/1673465676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a84104750 0x7f2a84104b60 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009900 tx=0x7f2a7402a3f0 comp rx=0 tx=0).stop 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.230+0000 7f2a89258640 1 -- 172.21.15.16:0/1673465676 shutdown_connections 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.230+0000 7f2a89258640 1 --2- 172.21.15.16:0/1673465676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a8410dbb0 0x7f2a841100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.230+0000 7f2a89258640 1 --2- 172.21.15.16:0/1673465676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a841050a0 0x7f2a8410d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.230+0000 7f2a89258640 1 --2- 172.21.15.16:0/1673465676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a84104750 0x7f2a84104b60 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.230+0000 7f2a89258640 1 -- 172.21.15.16:0/1673465676 >> 172.21.15.16:0/1673465676 conn(0x7f2a840794d0 msgr2=0x7f2a840798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.230+0000 7f2a89258640 1 -- 172.21.15.16:0/1673465676 shutdown_connections 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.230+0000 7f2a89258640 1 -- 172.21.15.16:0/1673465676 wait complete. 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.230+0000 7f2a89258640 1 Processor -- start 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a89258640 1 -- start start 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a89258640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a84104750 0x7f2a8419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a89258640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a841050a0 0x7f2a8419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a89258640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a8410dbb0 0x7f2a8419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a89258640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84113810 con 0x7f2a84104750 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a89258640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84113690 con 0x7f2a841050a0 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a89258640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84113990 con 0x7f2a8410dbb0 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a84104750 0x7f2a8419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a84104750 0x7f2a8419a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44976/0 (socket says 172.21.15.16:44976) 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a82d9d640 1 -- 172.21.15.16:0/3833469709 learned_addr learned my addr 172.21.15.16:0/3833469709 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a8359e640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a8410dbb0 0x7f2a8419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a8259c640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a841050a0 0x7f2a8419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a82d9d640 1 -- 172.21.15.16:0/3833469709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a8410dbb0 msgr2=0x7f2a8419faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a82d9d640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a8410dbb0 0x7f2a8419faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a82d9d640 1 -- 172.21.15.16:0/3833469709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a841050a0 msgr2=0x7f2a8419a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a82d9d640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a841050a0 0x7f2a8419a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.231+0000 7f2a82d9d640 1 -- 172.21.15.16:0/3833469709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a841a0110 con 0x7f2a84104750 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.232+0000 7f2a82d9d640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a84104750 0x7f2a8419a260 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f2a74032fd0 tx=0x7f2a74004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.232+0000 7f2a6bfff640 1 -- 172.21.15.16:0/3833469709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7403f030 con 0x7f2a84104750 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.232+0000 7f2a89258640 1 -- 172.21.15.16:0/3833469709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a841a03a0 con 0x7f2a84104750 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.232+0000 7f2a6bfff640 1 -- 172.21.15.16:0/3833469709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a74004490 con 0x7f2a84104750 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.232+0000 7f2a6bfff640 1 -- 172.21.15.16:0/3833469709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74035070 con 0x7f2a84104750 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.232+0000 7f2a89258640 1 -- 172.21.15.16:0/3833469709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a841a08d0 con 0x7f2a84104750 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.233+0000 7f2a89258640 1 -- 172.21.15.16:0/3833469709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a84104b60 con 0x7f2a84104750 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.235+0000 7f2a6bfff640 1 -- 172.21.15.16:0/3833469709 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2a74042030 con 0x7f2a84104750 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.235+0000 7f2a6bfff640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a6c0b4cd0 0x7f2a6c0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.235+0000 7f2a8259c640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a6c0b4cd0 0x7f2a6c0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.235+0000 7f2a6bfff640 1 -- 172.21.15.16:0/3833469709 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f2a74106310 con 0x7f2a84104750 2024-08-28T19:31:40.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.236+0000 7f2a8259c640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a6c0b4cd0 0x7f2a6c0b71e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2a78005fd0 tx=0x7f2a78005850 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:40.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.239+0000 7f2a6bfff640 1 -- 172.21.15.16:0/3833469709 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f2a841a08d0 con 0x7f2a84104750 2024-08-28T19:31:40.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.459+0000 7f2a89258640 1 -- 172.21.15.16:0/3833469709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a8419b390 con 0x7f2a84104750 2024-08-28T19:31:40.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.460+0000 7f2a6bfff640 1 -- 172.21.15.16:0/3833469709 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11977 (secure 0 0 0) 0x7f2a740ccd30 con 0x7f2a84104750 2024-08-28T19:31:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:31:40.462 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":55,"btime":"2024-08-28T19:30:56:995905+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:30:56.593478+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:31:40.462 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 55 2024-08-28T19:31:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.462+0000 7f2a89258640 1 -- 172.21.15.16:0/3833469709 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a6c0b4cd0 msgr2=0x7f2a6c0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.462+0000 7f2a89258640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a6c0b4cd0 0x7f2a6c0b71e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2a78005fd0 tx=0x7f2a78005850 comp rx=0 tx=0).stop 2024-08-28T19:31:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.462+0000 7f2a89258640 1 -- 172.21.15.16:0/3833469709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a84104750 msgr2=0x7f2a8419a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.462+0000 7f2a89258640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a84104750 0x7f2a8419a260 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f2a74032fd0 tx=0x7f2a74004270 comp rx=0 tx=0).stop 2024-08-28T19:31:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.462+0000 7f2a89258640 1 -- 172.21.15.16:0/3833469709 shutdown_connections 2024-08-28T19:31:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.462+0000 7f2a89258640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a8410dbb0 0x7f2a8419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.462+0000 7f2a89258640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a6c0b4cd0 0x7f2a6c0b71e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.462+0000 7f2a89258640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a841050a0 0x7f2a8419a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.462+0000 7f2a89258640 1 --2- 172.21.15.16:0/3833469709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a84104750 0x7f2a8419a260 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.462+0000 7f2a89258640 1 -- 172.21.15.16:0/3833469709 >> 172.21.15.16:0/3833469709 conn(0x7f2a840794d0 msgr2=0x7f2a84100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:40.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.463+0000 7f2a89258640 1 -- 172.21.15.16:0/3833469709 shutdown_connections 2024-08-28T19:31:40.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:40.463+0000 7f2a89258640 1 -- 172.21.15.16:0/3833469709 wait complete. 2024-08-28T19:31:40.970 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.72 MB/sec execute 440 sec latency 26135.782 ms 2024-08-28T19:31:41.421 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id 14a4328a --release --await' 2024-08-28T19:31:41.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id 14a4328a --release --await 2024-08-28T19:31:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:41 smithi028 ceph-mon[21769]: pgmap v666: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 16 op/s 2024-08-28T19:31:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:41 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3833469709' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:41.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:41 smithi115 ceph-mon[22184]: pgmap v666: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 16 op/s 2024-08-28T19:31:41.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:41 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3833469709' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:41.683 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:31:41.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:41 smithi016 ceph-mon[18556]: pgmap v666: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 16 op/s 2024-08-28T19:31:41.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:41 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3833469709' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:31:41.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438008 3.71 MB/sec execute 441 sec latency 27144.278 ms 2024-08-28T19:31:42.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.255+0000 7f36793d6640 1 -- 172.21.15.16:0/4278484656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3674107920 msgr2=0x7f3674107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:42.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.255+0000 7f36793d6640 1 --2- 172.21.15.16:0/4278484656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3674107920 0x7f3674107d30 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f3668009900 tx=0x7f366802a410 comp rx=0 tx=0).stop 2024-08-28T19:31:42.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.255+0000 7f36793d6640 1 -- 172.21.15.16:0/4278484656 shutdown_connections 2024-08-28T19:31:42.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.255+0000 7f36793d6640 1 --2- 172.21.15.16:0/4278484656 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3674108270 0x7f367410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.255+0000 7f36793d6640 1 --2- 172.21.15.16:0/4278484656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3674107920 0x7f3674107d30 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.255+0000 7f36793d6640 1 --2- 172.21.15.16:0/4278484656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3674101fe0 0x7f36741023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.255+0000 7f36793d6640 1 -- 172.21.15.16:0/4278484656 >> 172.21.15.16:0/4278484656 conn(0x7f3674079d30 msgr2=0x7f367407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.255+0000 7f36793d6640 1 -- 172.21.15.16:0/4278484656 shutdown_connections 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.255+0000 7f36793d6640 1 -- 172.21.15.16:0/4278484656 wait complete. 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.255+0000 7f36793d6640 1 Processor -- start 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.255+0000 7f36793d6640 1 -- start start 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36793d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3674101fe0 0x7f367419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36793d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3674107920 0x7f367419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36793d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3674108270 0x7f367419faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36793d6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36741116b0 con 0x7f3674107920 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36793d6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3674111530 con 0x7f3674108270 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36793d6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3674111830 con 0x7f3674101fe0 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36727fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3674107920 0x7f367419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36727fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3674107920 0x7f367419a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45006/0 (socket says 172.21.15.16:45006) 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f3672ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3674101fe0 0x7f367419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36727fc640 1 -- 172.21.15.16:0/1485473195 learned_addr learned my addr 172.21.15.16:0/1485473195 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:42.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36737fe640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3674108270 0x7f367419faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36727fc640 1 -- 172.21.15.16:0/1485473195 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3674101fe0 msgr2=0x7f367419a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:42.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36727fc640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3674101fe0 0x7f367419a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36727fc640 1 -- 172.21.15.16:0/1485473195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3674108270 msgr2=0x7f367419faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:42.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36727fc640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3674108270 0x7f367419faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36727fc640 1 -- 172.21.15.16:0/1485473195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36741a0160 con 0x7f3674107920 2024-08-28T19:31:42.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f36737fe640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3674108270 0x7f367419faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:42.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.256+0000 7f3672ffd640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3674101fe0 0x7f367419a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:31:42.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.257+0000 7f36727fc640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3674107920 0x7f367419a850 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f366802a930 tx=0x7f3668004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:42.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.257+0000 7f365bfff640 1 -- 172.21.15.16:0/1485473195 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366803f030 con 0x7f3674107920 2024-08-28T19:31:42.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.257+0000 7f365bfff640 1 -- 172.21.15.16:0/1485473195 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3668003aa0 con 0x7f3674107920 2024-08-28T19:31:42.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.257+0000 7f36793d6640 1 -- 172.21.15.16:0/1485473195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36741a03f0 con 0x7f3674107920 2024-08-28T19:31:42.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.257+0000 7f365bfff640 1 -- 172.21.15.16:0/1485473195 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3668035070 con 0x7f3674107920 2024-08-28T19:31:42.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.257+0000 7f36793d6640 1 -- 172.21.15.16:0/1485473195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36741a08d0 con 0x7f3674107920 2024-08-28T19:31:42.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.259+0000 7f367f94b740 1 Processor -- start 2024-08-28T19:31:42.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.259+0000 7f367f94b740 1 -- start start 2024-08-28T19:31:42.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f365bfff640 1 -- 172.21.15.16:0/1485473195 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3668042030 con 0x7f3674107920 2024-08-28T19:31:42.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f367f94b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c8566520 0x5603c890bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f367f94b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c890c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f367f94b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 0x5603c890c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f367f94b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5603c8882a70 con 0x5603c853d660 2024-08-28T19:31:42.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f367f94b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5603c8872000 con 0x5603c8545840 2024-08-28T19:31:42.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f367f94b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5603c8875fb0 con 0x5603c8566520 2024-08-28T19:31:42.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f3672ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c890c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f3672ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c890c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45012/0 (socket says 172.21.15.16:45012) 2024-08-28T19:31:42.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f3672ffd640 1 -- 172.21.15.16:0/2645487167 learned_addr learned my addr 172.21.15.16:0/2645487167 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:42.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f36737fe640 1 --2- 172.21.15.16:0/2645487167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c8566520 0x5603c890bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f36727fc640 1 --2- 172.21.15.16:0/2645487167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 0x5603c890c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f3672ffd640 1 -- 172.21.15.16:0/2645487167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c8566520 msgr2=0x5603c890bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:42.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f3672ffd640 1 --2- 172.21.15.16:0/2645487167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c8566520 0x5603c890bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f3672ffd640 1 -- 172.21.15.16:0/2645487167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 msgr2=0x5603c890c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:42.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f3672ffd640 1 --2- 172.21.15.16:0/2645487167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 0x5603c890c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f3672ffd640 1 -- 172.21.15.16:0/2645487167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603c85d1b00 con 0x5603c853d660 2024-08-28T19:31:42.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f36727fc640 1 --2- 172.21.15.16:0/2645487167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 0x5603c890c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:42.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.260+0000 7f365bfff640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f365c0b4cd0 0x7f365c0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.261+0000 7f3672ffd640 1 --2- 172.21.15.16:0/2645487167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c890c1c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f366400e610 tx=0x7f366400ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457ff924937dc4f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:42.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.261+0000 7f36737fe640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f365c0b4cd0 0x7f365c0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.261+0000 7f365bfff640 1 -- 172.21.15.16:0/1485473195 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f3668105e40 con 0x7f3674107920 2024-08-28T19:31:42.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.261+0000 7f36597fa640 1 -- 172.21.15.16:0/2645487167 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664004020 con 0x5603c853d660 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.261+0000 7f36597fa640 1 -- 172.21.15.16:0/2645487167 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f366400de00 con 0x5603c853d660 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.261+0000 7f36597fa640 1 -- 172.21.15.16:0/2645487167 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664003c20 con 0x5603c853d660 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f367f94b740 1 -- 172.21.15.16:0/2645487167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 msgr2=0x5603c890c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f36737fe640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f365c0b4cd0 0x7f365c0b71e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f366c007f90 tx=0x7f366c007720 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f367f94b740 1 --2- 172.21.15.16:0/2645487167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c890c1c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f366400e610 tx=0x7f366400ead0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f367f94b740 1 -- 172.21.15.16:0/2645487167 shutdown_connections 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f367f94b740 1 --2- 172.21.15.16:0/2645487167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 0x5603c890c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f367f94b740 1 --2- 172.21.15.16:0/2645487167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c890c1c0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f367f94b740 1 --2- 172.21.15.16:0/2645487167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c8566520 0x5603c890bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f367f94b740 1 -- 172.21.15.16:0/2645487167 >> 172.21.15.16:0/2645487167 conn(0x5603c855a910 msgr2=0x5603c8909b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f367f94b740 1 -- 172.21.15.16:0/2645487167 shutdown_connections 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f367f94b740 1 -- 172.21.15.16:0/2645487167 wait complete. 2024-08-28T19:31:42.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f367f94b740 1 -- start start 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.262+0000 7f367f94b740 1 Processor -- start 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f367f94b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 0x5603c8917880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f367f94b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c8545840 0x5603c8917dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f367f94b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8566520 0x5603c8923930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f367f94b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5603c891a460 con 0x5603c8566520 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f367f94b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5603c8833520 con 0x5603c853d660 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f367f94b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5603c891a5e0 con 0x5603c8545840 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f367f94b740 10 client.0 _ll_register_callbacks cb 0x5603c886bc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f3672ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 0x5603c8917880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f36737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8566520 0x5603c8923930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f3672ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 0x5603c8917880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34688/0 (socket says 172.21.15.16:34688) 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f3672ffd640 1 -- 172.21.15.16:0/3447540548 learned_addr learned my addr 172.21.15.16:0/3447540548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:42.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f36727fc640 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c8545840 0x5603c8917dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f36737fe640 1 -- 172.21.15.16:0/3447540548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c8545840 msgr2=0x5603c8917dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f36737fe640 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c8545840 0x5603c8917dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f36737fe640 1 -- 172.21.15.16:0/3447540548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 msgr2=0x5603c8917880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f36737fe640 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 0x5603c8917880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f36737fe640 1 -- 172.21.15.16:0/3447540548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603c8648960 con 0x5603c8566520 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.263+0000 7f3672ffd640 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 0x5603c8917880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.264+0000 7f36737fe640 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8566520 0x5603c8923930 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f366c00fe80 tx=0x7f366c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbc80247a111cbf server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.264+0000 7f3656ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.264+0000 7f3656ff5640 1 -- 172.21.15.16:0/3447540548 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c00eb30 con 0x5603c8566520 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.264+0000 7f367f94b740 1 -- 172.21.15.16:0/3447540548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5603c88119a0 con 0x5603c8566520 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.264+0000 7f3656ff5640 1 -- 172.21.15.16:0/3447540548 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f366c00ecd0 con 0x5603c8566520 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.264+0000 7f3656ff5640 1 -- 172.21.15.16:0/3447540548 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f366c02f050 con 0x5603c8566520 2024-08-28T19:31:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.264+0000 7f367f94b740 10 client.18261 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.264+0000 7f367f94b740 1 -- 172.21.15.16:0/3447540548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5603c892ab10 con 0x5603c8566520 2024-08-28T19:31:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.264+0000 7f3656ff5640 1 -- 172.21.15.16:0/3447540548 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f366c01d8c0 con 0x5603c8566520 2024-08-28T19:31:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.264+0000 7f3656ff5640 1 -- 172.21.15.16:0/3447540548 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f366c01c610 con 0x5603c8566520 2024-08-28T19:31:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.265+0000 7f367f94b740 10 client.18261 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.265+0000 7f367f94b740 10 client.18261 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:31:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.265+0000 7f367f94b740 20 client.18261 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.265+0000 7f367f94b740 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5603c8524ea0 0x5603c892e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.265+0000 7f367f94b740 4 client.18261 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:31:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.265+0000 7f367f94b740 1 -- 172.21.15.16:0/3447540548 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603c88119a0 con 0x5603c8524ea0 2024-08-28T19:31:42.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.265+0000 7f3672ffd640 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5603c8524ea0 0x5603c892e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.266+0000 7f3656ff5640 10 client.18261 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.266+0000 7f3672ffd640 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5603c8524ea0 0x5603c892e060 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff4b1f4fbcc0d521 server_cookie=45293f40194e69e in_seq=0 out_seq=0 2024-08-28T19:31:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.266+0000 7f3656ff5640 1 -- 172.21.15.16:0/3447540548 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5603c88119a0 con 0x5603c8524ea0 2024-08-28T19:31:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.266+0000 7f3656ff5640 10 client.18261 handle_command_reply: tid=0 2024-08-28T19:31:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.266+0000 7f367f94b740 1 client.18261 shutdown 2024-08-28T19:31:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f367f94b740 1 -- 172.21.15.16:0/3447540548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8566520 msgr2=0x5603c8923930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f367f94b740 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8566520 0x5603c8923930 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f366c00fe80 tx=0x7f366c004a40 comp rx=0 tx=0).stop 2024-08-28T19:31:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f367f94b740 1 -- 172.21.15.16:0/3447540548 shutdown_connections 2024-08-28T19:31:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f367f94b740 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8566520 0x5603c8923930 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f367f94b740 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5603c8524ea0 0x5603c892e060 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f367f94b740 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c8545840 0x5603c8917dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f367f94b740 1 --2- 172.21.15.16:0/3447540548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 0x5603c8917880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f367f94b740 1 -- 172.21.15.16:0/3447540548 >> 172.21.15.16:0/3447540548 conn(0x5603c855a910 msgr2=0x5603c890ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f3656ff5640 0 client.18261 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f367f94b740 1 -- 172.21.15.16:0/3447540548 shutdown_connections 2024-08-28T19:31:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f367f94b740 1 -- 172.21.15.16:0/3447540548 wait complete. 2024-08-28T19:31:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.267+0000 7f367f94b740 20 client.18261 trim_cache size 0 max 16384 2024-08-28T19:31:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.284+0000 7f367f94b740 1 Processor -- start 2024-08-28T19:31:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.284+0000 7f367f94b740 1 -- start start 2024-08-28T19:31:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f367f94b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c855a910 0x5603c8923830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f367f94b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 0x5603c89094d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f367f94b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8545840 0x5603c8910d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f367f94b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36741aea50 con 0x5603c8545840 2024-08-28T19:31:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f367f94b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36741b0590 con 0x5603c853d660 2024-08-28T19:31:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f367f94b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f36741af120 con 0x5603c855a910 2024-08-28T19:31:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f36737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8545840 0x5603c8910d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f36737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8545840 0x5603c8910d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45022/0 (socket says 172.21.15.16:45022) 2024-08-28T19:31:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f36737fe640 1 -- 172.21.15.16:0/992773667 learned_addr learned my addr 172.21.15.16:0/992773667 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f36727fc640 1 --2- 172.21.15.16:0/992773667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 0x5603c89094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f36737fe640 1 -- 172.21.15.16:0/992773667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c855a910 msgr2=0x5603c8923830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f36737fe640 1 --2- 172.21.15.16:0/992773667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c855a910 0x5603c8923830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f36737fe640 1 -- 172.21.15.16:0/992773667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 msgr2=0x5603c89094d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f36737fe640 1 --2- 172.21.15.16:0/992773667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 0x5603c89094d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f36737fe640 1 -- 172.21.15.16:0/992773667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603c8912ea0 con 0x5603c8545840 2024-08-28T19:31:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f36727fc640 1 --2- 172.21.15.16:0/992773667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 0x5603c89094d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:31:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f36737fe640 1 --2- 172.21.15.16:0/992773667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8545840 0x5603c8910d90 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f366c000f80 tx=0x7f366c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b620cbd56dd677d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f3658ff9640 1 -- 172.21.15.16:0/992773667 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c037030 con 0x5603c8545840 2024-08-28T19:31:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f3658ff9640 1 -- 172.21.15.16:0/992773667 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f366c01bd40 con 0x5603c8545840 2024-08-28T19:31:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.285+0000 7f3658ff9640 1 -- 172.21.15.16:0/992773667 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c01b5f0 con 0x5603c8545840 2024-08-28T19:31:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.286+0000 7f367f94b740 1 -- 172.21.15.16:0/992773667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8545840 msgr2=0x5603c8910d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.286+0000 7f367f94b740 1 --2- 172.21.15.16:0/992773667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8545840 0x5603c8910d90 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f366c000f80 tx=0x7f366c007a40 comp rx=0 tx=0).stop 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.286+0000 7f367f94b740 1 -- 172.21.15.16:0/992773667 shutdown_connections 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.286+0000 7f367f94b740 1 --2- 172.21.15.16:0/992773667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c8545840 0x5603c8910d90 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.286+0000 7f367f94b740 1 --2- 172.21.15.16:0/992773667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c853d660 0x5603c89094d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.286+0000 7f367f94b740 1 --2- 172.21.15.16:0/992773667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c855a910 0x5603c8923830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.286+0000 7f367f94b740 1 -- 172.21.15.16:0/992773667 >> 172.21.15.16:0/992773667 conn(0x5603c8909bb0 msgr2=0x5603c8917880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.286+0000 7f367f94b740 1 -- 172.21.15.16:0/992773667 shutdown_connections 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.286+0000 7f367f94b740 1 -- 172.21.15.16:0/992773667 wait complete. 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f367f94b740 1 -- start start 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f367f94b740 1 Processor -- start 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f367f94b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c891c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f367f94b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 0x5603c891c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f367f94b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c855a910 0x5603c8997a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f367f94b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36741afec0 con 0x5603c853d660 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f367f94b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36741af7f0 con 0x5603c8545840 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f367f94b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f36741aca50 con 0x5603c855a910 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f367f94b740 10 client.0 _ll_register_callbacks cb 0x7f366c00fb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f3672ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c891c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f3672ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c891c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45030/0 (socket says 172.21.15.16:45030) 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f3672ffd640 1 -- 172.21.15.16:0/3693310511 learned_addr learned my addr 172.21.15.16:0/3693310511 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f36737fe640 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c855a910 0x5603c8997a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f36727fc640 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 0x5603c891c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f3672ffd640 1 -- 172.21.15.16:0/3693310511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c855a910 msgr2=0x5603c8997a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f3672ffd640 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c855a910 0x5603c8997a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f3672ffd640 1 -- 172.21.15.16:0/3693310511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 msgr2=0x5603c891c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.287+0000 7f3672ffd640 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 0x5603c891c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f3672ffd640 1 -- 172.21.15.16:0/3693310511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603c8998070 con 0x5603c853d660 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f36727fc640 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 0x5603c891c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f36737fe640 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c855a910 0x5603c8997a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f3672ffd640 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c891c080 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f3664003790 tx=0x7f366400b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7946648413ca5ecc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f3656ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f3656ff5640 1 -- 172.21.15.16:0/3693310511 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366402f060 con 0x5603c853d660 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f3656ff5640 1 -- 172.21.15.16:0/3693310511 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9490+0+0 (secure 0 0 0) 0x7f366400be00 con 0x5603c853d660 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f367f94b740 1 -- 172.21.15.16:0/3693310511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f366c01bd40 con 0x5603c853d660 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f3656ff5640 1 -- 172.21.15.16:0/3693310511 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f3664036050 con 0x5603c853d660 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f367f94b740 10 client.18267 fetch_fsmap learned FSMap version 55 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f367f94b740 1 -- 172.21.15.16:0/3693310511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5603c89992a0 con 0x5603c853d660 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.288+0000 7f3656ff5640 1 -- 172.21.15.16:0/3693310511 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f366c01bd40 con 0x5603c853d660 2024-08-28T19:31:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.289+0000 7f3656ff5640 1 -- 172.21.15.16:0/3693310511 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 55) ==== 8370+0+0 (secure 0 0 0) 0x7f366400b040 con 0x5603c853d660 2024-08-28T19:31:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.289+0000 7f367f94b740 10 client.18267 fetch_fsmap finished waiting for FSMap version 55 2024-08-28T19:31:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.289+0000 7f367f94b740 10 client.18267 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:31:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.289+0000 7f367f94b740 20 client.18267 populate_metadata read hostname 'smithi016' 2024-08-28T19:31:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.289+0000 7f367f94b740 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5603c8566520 0x5603c899b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:31:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.289+0000 7f367f94b740 4 client.18267 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "14a4328a", "release": true, "await": true}] 2024-08-28T19:31:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.289+0000 7f367f94b740 1 -- 172.21.15.16:0/3693310511 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "14a4328a", "release": true, "await": true}) -- 0x5603c88849c0 con 0x5603c8566520 2024-08-28T19:31:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.289+0000 7f36727fc640 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5603c8566520 0x5603c899b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:31:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.289+0000 7f3656ff5640 10 client.18267 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.289+0000 7f36727fc640 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5603c8566520 0x5603c899b740 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf653d99c1ccc018 server_cookie=7fccf60e532a5707 in_seq=0 out_seq=0 2024-08-28T19:31:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f3656ff5640 1 -- 172.21.15.16:0/3693310511 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5603c88849c0 con 0x5603c8566520 2024-08-28T19:31:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f3656ff5640 10 client.18267 handle_command_reply: tid=0 2024-08-28T19:31:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f367f94b740 1 client.18267 shutdown 2024-08-28T19:31:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f367f94b740 1 -- 172.21.15.16:0/3693310511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 msgr2=0x5603c891c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f367f94b740 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c891c080 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f3664003790 tx=0x7f366400b6b0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f367f94b740 1 -- 172.21.15.16:0/3693310511 shutdown_connections 2024-08-28T19:31:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f367f94b740 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5603c8566520 0x5603c899b740 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f367f94b740 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5603c855a910 0x5603c8997a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f367f94b740 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5603c8545840 0x5603c891c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f367f94b740 1 --2- 172.21.15.16:0/3693310511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603c853d660 0x5603c891c080 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f367f94b740 1 -- 172.21.15.16:0/3693310511 >> 172.21.15.16:0/3693310511 conn(0x5603c8909bb0 msgr2=0x5603c8917860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:42.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f3656ff5640 0 client.18267 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:31:42.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f367f94b740 1 -- 172.21.15.16:0/3693310511 shutdown_connections 2024-08-28T19:31:42.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.319+0000 7f367f94b740 1 -- 172.21.15.16:0/3693310511 wait complete. 2024-08-28T19:31:42.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.320+0000 7f367f94b740 20 client.18267 trim_cache size 0 max 16384 2024-08-28T19:31:42.322 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:31:42.322 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 55, 2024-08-28T19:31:42.322 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 36, 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "14a4328a": { 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 36, 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.1, 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:31:42.323 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:31:42.324 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:31:42.324 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:31:42.324 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:42.324 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:42.324 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:42.324 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:42.324 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:31:42.324 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:31:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.320+0000 7f36793d6640 1 -- 172.21.15.16:0/1485473195 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f365c0b4cd0 msgr2=0x7f365c0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.320+0000 7f36793d6640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f365c0b4cd0 0x7f365c0b71e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f366c007f90 tx=0x7f366c007720 comp rx=0 tx=0).stop 2024-08-28T19:31:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.320+0000 7f36793d6640 1 -- 172.21.15.16:0/1485473195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3674107920 msgr2=0x7f367419a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:31:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.320+0000 7f36793d6640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3674107920 0x7f367419a850 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f366802a930 tx=0x7f3668004270 comp rx=0 tx=0).stop 2024-08-28T19:31:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.320+0000 7f36793d6640 1 -- 172.21.15.16:0/1485473195 shutdown_connections 2024-08-28T19:31:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.320+0000 7f36793d6640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f365c0b4cd0 0x7f365c0b71e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.320+0000 7f36793d6640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3674108270 0x7f367419faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.320+0000 7f36793d6640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3674107920 0x7f367419a850 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.320+0000 7f36793d6640 1 --2- 172.21.15.16:0/1485473195 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3674101fe0 0x7f367419a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:31:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.320+0000 7f36793d6640 1 -- 172.21.15.16:0/1485473195 >> 172.21.15.16:0/1485473195 conn(0x7f3674079d30 msgr2=0x7f36741039f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:31:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.321+0000 7f36793d6640 1 -- 172.21.15.16:0/1485473195 shutdown_connections 2024-08-28T19:31:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:31:42.321+0000 7f36793d6640 1 -- 172.21.15.16:0/1485473195 wait complete. 2024-08-28T19:31:42.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 439293 3.72 MB/sec execute 442 sec latency 27492.671 ms 2024-08-28T19:31:43.127 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '14a4328a', total seconds elapsed: 30.8 2024-08-28T19:31:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:43 smithi028 ceph-mon[21769]: pgmap v667: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 38 KiB/s wr, 17 op/s 2024-08-28T19:31:43.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:43 smithi115 ceph-mon[22184]: pgmap v667: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 38 KiB/s wr, 17 op/s 2024-08-28T19:31:43.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:43 smithi016 ceph-mon[18556]: pgmap v667: 129 pgs: 129 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 38 KiB/s wr, 17 op/s 2024-08-28T19:31:43.978 INFO:tasks.workunit.client.0.smithi016.stdout: 1 440663 3.73 MB/sec execute 443 sec latency 20.546 ms 2024-08-28T19:31:44.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 441547 3.73 MB/sec execute 444 sec latency 333.984 ms 2024-08-28T19:31:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:45 smithi028 ceph-mon[21769]: pgmap v668: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-08-28T19:31:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:45 smithi115 ceph-mon[22184]: pgmap v668: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-08-28T19:31:45.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:45 smithi016 ceph-mon[18556]: pgmap v668: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-08-28T19:31:45.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 441547 3.72 MB/sec execute 445 sec latency 1334.101 ms 2024-08-28T19:31:46.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 441550 3.71 MB/sec execute 446 sec latency 2331.838 ms 2024-08-28T19:31:47.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:47 smithi028 ceph-mon[21769]: pgmap v669: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 125 KiB/s wr, 34 op/s 2024-08-28T19:31:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:47 smithi115 ceph-mon[22184]: pgmap v669: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 125 KiB/s wr, 34 op/s 2024-08-28T19:31:47.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:47 smithi016 ceph-mon[18556]: pgmap v669: 129 pgs: 129 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 125 KiB/s wr, 34 op/s 2024-08-28T19:31:47.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 443611 3.74 MB/sec execute 447 sec latency 16.420 ms 2024-08-28T19:31:48.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:48 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:31:48.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:48 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:31:48.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:48 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:31:48.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 444852 3.74 MB/sec execute 448 sec latency 20.528 ms 2024-08-28T19:31:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:49 smithi115 ceph-mon[22184]: pgmap v670: 129 pgs: 129 active+clean; 314 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 390 KiB/s wr, 55 op/s 2024-08-28T19:31:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:31:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:31:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:31:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:31:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:31:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:49 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:31:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:49 smithi016 ceph-mon[18556]: pgmap v670: 129 pgs: 129 active+clean; 314 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 390 KiB/s wr, 55 op/s 2024-08-28T19:31:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:31:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:31:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:31:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:31:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:31:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:49 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:31:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:49 smithi028 ceph-mon[21769]: pgmap v670: 129 pgs: 129 active+clean; 314 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 390 KiB/s wr, 55 op/s 2024-08-28T19:31:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:31:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:31:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:31:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:31:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:31:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:49 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:31:49.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 445096 3.73 MB/sec execute 449 sec latency 824.013 ms 2024-08-28T19:31:50.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:50 smithi016 ceph-mon[18556]: pgmap v671: 129 pgs: 129 active+clean; 315 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 489 KiB/s wr, 72 op/s 2024-08-28T19:31:50.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:31:50.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:31:50.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:50 smithi028 ceph-mon[21769]: pgmap v671: 129 pgs: 129 active+clean; 315 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 489 KiB/s wr, 72 op/s 2024-08-28T19:31:50.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:31:50.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:31:50.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:50 smithi115 ceph-mon[22184]: pgmap v671: 129 pgs: 129 active+clean; 315 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 489 KiB/s wr, 72 op/s 2024-08-28T19:31:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:31:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:31:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:31:50.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 445096 3.72 MB/sec execute 450 sec latency 1824.120 ms 2024-08-28T19:31:51.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 445099 3.71 MB/sec execute 451 sec latency 2823.316 ms 2024-08-28T19:31:52.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 447191 3.74 MB/sec execute 452 sec latency 15.492 ms 2024-08-28T19:31:53.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:52 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:31:53.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:52 smithi016 ceph-mon[18556]: pgmap v672: 129 pgs: 129 active+clean; 317 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 637 KiB/s wr, 94 op/s 2024-08-28T19:31:53.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:52 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:31:53.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:52 smithi028 ceph-mon[21769]: pgmap v672: 129 pgs: 129 active+clean; 317 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 637 KiB/s wr, 94 op/s 2024-08-28T19:31:53.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:52 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:31:53.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:52 smithi115 ceph-mon[22184]: pgmap v672: 129 pgs: 129 active+clean; 317 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 637 KiB/s wr, 94 op/s 2024-08-28T19:31:53.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 448523 3.74 MB/sec execute 453 sec latency 17.443 ms 2024-08-28T19:31:54.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 449528 3.74 MB/sec execute 454 sec latency 487.236 ms 2024-08-28T19:31:55.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:54 smithi028 ceph-mon[21769]: pgmap v673: 129 pgs: 129 active+clean; 319 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 864 KiB/s wr, 121 op/s 2024-08-28T19:31:55.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:54 smithi115 ceph-mon[22184]: pgmap v673: 129 pgs: 129 active+clean; 319 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 864 KiB/s wr, 121 op/s 2024-08-28T19:31:55.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:54 smithi016 ceph-mon[18556]: pgmap v673: 129 pgs: 129 active+clean; 319 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 864 KiB/s wr, 121 op/s 2024-08-28T19:31:55.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 451092 3.75 MB/sec execute 455 sec latency 10.065 ms 2024-08-28T19:31:56.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 452190 3.76 MB/sec execute 456 sec latency 219.160 ms 2024-08-28T19:31:57.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:56 smithi028 ceph-mon[21769]: pgmap v674: 129 pgs: 129 active+clean; 320 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 913 KiB/s wr, 130 op/s 2024-08-28T19:31:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:56 smithi115 ceph-mon[22184]: pgmap v674: 129 pgs: 129 active+clean; 320 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 913 KiB/s wr, 130 op/s 2024-08-28T19:31:57.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:56 smithi016 ceph-mon[18556]: pgmap v674: 129 pgs: 129 active+clean; 320 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 913 KiB/s wr, 130 op/s 2024-08-28T19:31:57.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 454281 3.78 MB/sec execute 457 sec latency 20.912 ms 2024-08-28T19:31:58.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 455596 3.78 MB/sec execute 458 sec latency 16.936 ms 2024-08-28T19:31:59.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:31:58 smithi028 ceph-mon[21769]: pgmap v675: 129 pgs: 129 active+clean; 324 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-08-28T19:31:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:31:58 smithi115 ceph-mon[22184]: pgmap v675: 129 pgs: 129 active+clean; 324 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-08-28T19:31:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:31:58 smithi016 ceph-mon[18556]: pgmap v675: 129 pgs: 129 active+clean; 324 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-08-28T19:31:59.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 455730 3.77 MB/sec execute 459 sec latency 890.824 ms 2024-08-28T19:32:00.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 455730 3.77 MB/sec execute 460 sec latency 1890.924 ms 2024-08-28T19:32:01.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:00 smithi028 ceph-mon[21769]: pgmap v676: 129 pgs: 129 active+clean; 330 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 175 op/s 2024-08-28T19:32:01.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:00 smithi115 ceph-mon[22184]: pgmap v676: 129 pgs: 129 active+clean; 330 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 175 op/s 2024-08-28T19:32:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:00 smithi016 ceph-mon[18556]: pgmap v676: 129 pgs: 129 active+clean; 330 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 175 op/s 2024-08-28T19:32:01.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 456311 3.76 MB/sec execute 461 sec latency 2520.453 ms 2024-08-28T19:32:02.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 457050 3.77 MB/sec execute 462 sec latency 53.189 ms 2024-08-28T19:32:03.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:02 smithi028 ceph-mon[21769]: pgmap v677: 129 pgs: 129 active+clean; 332 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 194 op/s 2024-08-28T19:32:03.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:02 smithi115 ceph-mon[22184]: pgmap v677: 129 pgs: 129 active+clean; 332 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 194 op/s 2024-08-28T19:32:03.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:02 smithi016 ceph-mon[18556]: pgmap v677: 129 pgs: 129 active+clean; 332 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 194 op/s 2024-08-28T19:32:03.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 458331 3.77 MB/sec execute 463 sec latency 21.421 ms 2024-08-28T19:32:04.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:32:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:04.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 459314 3.81 MB/sec execute 464 sec latency 65.251 ms 2024-08-28T19:32:05.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:04 smithi016 ceph-mon[18556]: pgmap v678: 129 pgs: 129 active+clean; 332 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 185 op/s 2024-08-28T19:32:05.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:04 smithi016 ceph-mon[18556]: osdmap e93: 12 total, 12 up, 12 in 2024-08-28T19:32:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:04 smithi028 ceph-mon[21769]: pgmap v678: 129 pgs: 129 active+clean; 332 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 185 op/s 2024-08-28T19:32:05.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:04 smithi028 ceph-mon[21769]: osdmap e93: 12 total, 12 up, 12 in 2024-08-28T19:32:05.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:04 smithi115 ceph-mon[22184]: pgmap v678: 129 pgs: 129 active+clean; 332 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 185 op/s 2024-08-28T19:32:05.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:04 smithi115 ceph-mon[22184]: osdmap e93: 12 total, 12 up, 12 in 2024-08-28T19:32:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.217+0000 7fa0cdab0640 1 -- 172.21.15.16:0/1087821998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c8109250 msgr2=0x7fa0c8109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.217+0000 7fa0cdab0640 1 --2- 172.21.15.16:0/1087821998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c8109250 0x7fa0c8109660 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009900 tx=0x7fa0bc02a3f0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.217+0000 7fa0cdab0640 1 -- 172.21.15.16:0/1087821998 shutdown_connections 2024-08-28T19:32:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.217+0000 7fa0cdab0640 1 --2- 172.21.15.16:0/1087821998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c8109ba0 0x7fa0c81103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.217+0000 7fa0cdab0640 1 --2- 172.21.15.16:0/1087821998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c8109250 0x7fa0c8109660 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.217+0000 7fa0cdab0640 1 --2- 172.21.15.16:0/1087821998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c8103910 0x7fa0c8103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.217+0000 7fa0cdab0640 1 -- 172.21.15.16:0/1087821998 >> 172.21.15.16:0/1087821998 conn(0x7fa0c80ff380 msgr2=0x7fa0c81017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.218+0000 7fa0cdab0640 1 -- 172.21.15.16:0/1087821998 shutdown_connections 2024-08-28T19:32:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.218+0000 7fa0cdab0640 1 -- 172.21.15.16:0/1087821998 wait complete. 2024-08-28T19:32:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.218+0000 7fa0cdab0640 1 Processor -- start 2024-08-28T19:32:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.218+0000 7fa0cdab0640 1 -- start start 2024-08-28T19:32:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0cdab0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c8103910 0x7fa0c80754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0cdab0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c8109250 0x7fa0c80759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0cdab0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c8109ba0 0x7fa0c8075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0cdab0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c8113860 con 0x7fa0c8103910 2024-08-28T19:32:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0cdab0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c81136e0 con 0x7fa0c8109ba0 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0cdab0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c81139e0 con 0x7fa0c8109250 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c8103910 0x7fa0c80754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c8103910 0x7fa0c80754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46440/0 (socket says 172.21.15.16:46440) 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0c6ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c8109250 0x7fa0c80759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0c77fe640 1 -- 172.21.15.16:0/3887389201 learned_addr learned my addr 172.21.15.16:0/3887389201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0c7fff640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c8109ba0 0x7fa0c8075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0c77fe640 1 -- 172.21.15.16:0/3887389201 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c8109250 msgr2=0x7fa0c80759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c8109250 0x7fa0c80759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0c77fe640 1 -- 172.21.15.16:0/3887389201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c8109ba0 msgr2=0x7fa0c8075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c8109ba0 0x7fa0c8075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0c77fe640 1 -- 172.21.15.16:0/3887389201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c81a84f0 con 0x7fa0c8103910 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.219+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c8103910 0x7fa0c80754b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800e9f0 tx=0x7fa0b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.220+0000 7fa0c4ff9640 1 -- 172.21.15.16:0/3887389201 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800cd80 con 0x7fa0c8103910 2024-08-28T19:32:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.220+0000 7fa0cdab0640 1 -- 172.21.15.16:0/3887389201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c81a87e0 con 0x7fa0c8103910 2024-08-28T19:32:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.220+0000 7fa0c4ff9640 1 -- 172.21.15.16:0/3887389201 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0b80108e0 con 0x7fa0c8103910 2024-08-28T19:32:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.220+0000 7fa0c4ff9640 1 -- 172.21.15.16:0/3887389201 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8018980 con 0x7fa0c8103910 2024-08-28T19:32:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.220+0000 7fa0cdab0640 1 -- 172.21.15.16:0/3887389201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c81a8cf0 con 0x7fa0c8103910 2024-08-28T19:32:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.222+0000 7fa0c4ff9640 1 -- 172.21.15.16:0/3887389201 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa0b8018b20 con 0x7fa0c8103910 2024-08-28T19:32:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.222+0000 7fa0d4025740 1 Processor -- start 2024-08-28T19:32:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0d4025740 1 -- start start 2024-08-28T19:32:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1c482c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c48800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c48d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0d4025740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5579d1bbf630 con 0x5579d18a3070 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0d4025740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5579d1baebc0 con 0x5579d187a1b0 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0d4025740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5579d1bb2b70 con 0x5579d1882390 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1c482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1c482c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46454/0 (socket says 172.21.15.16:46454) 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0c7fff640 1 -- 172.21.15.16:0/3101234837 learned_addr learned my addr 172.21.15.16:0/3101234837 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/3101234837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c48800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0c6ffd640 1 --2- 172.21.15.16:0/3101234837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c48d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0c7fff640 1 -- 172.21.15.16:0/3101234837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 msgr2=0x5579d1c48d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0c7fff640 1 --2- 172.21.15.16:0/3101234837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c48d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0c7fff640 1 -- 172.21.15.16:0/3101234837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 msgr2=0x5579d1c48800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0c7fff640 1 --2- 172.21.15.16:0/3101234837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c48800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0c7fff640 1 -- 172.21.15.16:0/3101234837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579d190e650 con 0x5579d18a3070 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.223+0000 7fa0c4ff9640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0b00b4ca0 0x7fa0b00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.224+0000 7fa0c6ffd640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0b00b4ca0 0x7fa0b00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.224+0000 7fa0c4ff9640 1 -- 172.21.15.16:0/3887389201 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9578+0+0 (secure 0 0 0) 0x7fa0b80e5cc0 con 0x7fa0c8103910 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.224+0000 7fa0c7fff640 1 --2- 172.21.15.16:0/3101234837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1c482c0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009fd0 tx=0x7fa0c00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9426f200c144a78 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.224+0000 7fa0adffb640 1 -- 172.21.15.16:0/3101234837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0004d20 con 0x5579d18a3070 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.224+0000 7fa0adffb640 1 -- 172.21.15.16:0/3101234837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c0004050 con 0x5579d18a3070 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.224+0000 7fa0adffb640 1 -- 172.21.15.16:0/3101234837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0005260 con 0x5579d18a3070 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.225+0000 7fa0d4025740 1 -- 172.21.15.16:0/3101234837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 msgr2=0x5579d1c482c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.225+0000 7fa0c6ffd640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0b00b4ca0 0x7fa0b00b71b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009230 tx=0x7fa0bc031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.225+0000 7fa0d4025740 1 --2- 172.21.15.16:0/3101234837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1c482c0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009fd0 tx=0x7fa0c00118d0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.225+0000 7fa0d4025740 1 -- 172.21.15.16:0/3101234837 shutdown_connections 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.225+0000 7fa0d4025740 1 --2- 172.21.15.16:0/3101234837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c48d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.225+0000 7fa0d4025740 1 --2- 172.21.15.16:0/3101234837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c48800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.225+0000 7fa0d4025740 1 --2- 172.21.15.16:0/3101234837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1c482c0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.225+0000 7fa0d4025740 1 -- 172.21.15.16:0/3101234837 >> 172.21.15.16:0/3101234837 conn(0x5579d1897460 msgr2=0x5579d1c461b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.225+0000 7fa0d4025740 1 -- 172.21.15.16:0/3101234837 shutdown_connections 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.225+0000 7fa0d4025740 1 -- 172.21.15.16:0/3101234837 wait complete. 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0d4025740 1 -- start start 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0d4025740 1 Processor -- start 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c53a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c53fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1c5fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0d4025740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5579d1a11700 con 0x5579d18a3070 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0d4025740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5579d1b35700 con 0x5579d187a1b0 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0d4025740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5579d1c56aa0 con 0x5579d1882390 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0d4025740 10 client.0 _ll_register_callbacks cb 0x5579d1b64dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1c5fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0c77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c53a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0c6ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c53fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0c77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c53a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:42372/0 (socket says 172.21.15.16:42372) 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1c5fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46458/0 (socket says 172.21.15.16:46458) 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0c77fe640 1 -- 172.21.15.16:0/2033732272 learned_addr learned my addr 172.21.15.16:0/2033732272 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.226+0000 7fa0c6ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c53fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:39524/0 (socket says 172.21.15.16:39524) 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0c77fe640 1 -- 172.21.15.16:0/2033732272 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 msgr2=0x5579d1c53fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/2033732272 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c53fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0c77fe640 1 -- 172.21.15.16:0/2033732272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 msgr2=0x5579d1c5fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/2033732272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1c5fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0c77fe640 1 -- 172.21.15.16:0/2033732272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579d1b6e8b0 con 0x5579d187a1b0 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/2033732272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c53a70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80e3050 tx=0x7fa0b80ad2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dd1d231bd3821c2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/2033732272 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b80e7070 con 0x5579d187a1b0 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/2033732272 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9578+0+0 (secure 0 0 0) 0x7fa0b8010430 con 0x5579d187a1b0 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0d4025740 1 -- 172.21.15.16:0/2033732272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0c0004050 con 0x5579d187a1b0 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/2033732272 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fa0b80abb50 con 0x5579d187a1b0 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0d4025740 10 client.27746 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0d4025740 1 -- 172.21.15.16:0/2033732272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5579d1c62900 con 0x5579d187a1b0 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.227+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/2033732272 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c0004050 con 0x5579d187a1b0 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.228+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/2033732272 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7fa0b80a7b70 con 0x5579d187a1b0 2024-08-28T19:32:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.228+0000 7fa0d4025740 10 client.27746 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.228+0000 7fa0d4025740 10 client.27746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.228+0000 7fa0d4025740 20 client.27746 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.228+0000 7fa0d4025740 1 --2- 172.21.15.16:0/2033732272 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5579d18619f0 0x5579d1c65df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.228+0000 7fa0d4025740 4 client.27746 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.228+0000 7fa0d4025740 1 -- 172.21.15.16:0/2033732272 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579d1bc1580 con 0x5579d18619f0 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.228+0000 7fa0c6ffd640 1 --2- 172.21.15.16:0/2033732272 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5579d18619f0 0x5579d1c65df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.228+0000 7fa0a6ffd640 10 client.27746 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.228+0000 7fa0c6ffd640 1 --2- 172.21.15.16:0/2033732272 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5579d18619f0 0x5579d1c65df0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59751d12c9bc82da server_cookie=70c81c8f00772a44 in_seq=0 out_seq=0 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/2033732272 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5579d1bc1580 con 0x5579d18619f0 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0a6ffd640 10 client.27746 handle_command_reply: tid=0 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0d4025740 1 client.27746 shutdown 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0d4025740 1 -- 172.21.15.16:0/2033732272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 msgr2=0x5579d1c53a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0d4025740 1 --2- 172.21.15.16:0/2033732272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c53a70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80e3050 tx=0x7fa0b80ad2d0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0d4025740 1 -- 172.21.15.16:0/2033732272 shutdown_connections 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0d4025740 1 --2- 172.21.15.16:0/2033732272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1c5fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0d4025740 1 --2- 172.21.15.16:0/2033732272 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5579d18619f0 0x5579d1c65df0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0d4025740 1 --2- 172.21.15.16:0/2033732272 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c53fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0d4025740 1 --2- 172.21.15.16:0/2033732272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c53a70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0a6ffd640 0 client.27746 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0d4025740 1 -- 172.21.15.16:0/2033732272 >> 172.21.15.16:0/2033732272 conn(0x5579d1897460 msgr2=0x5579d1c4b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.229+0000 7fa0d4025740 1 -- 172.21.15.16:0/2033732272 shutdown_connections 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.230+0000 7fa0d4025740 1 -- 172.21.15.16:0/2033732272 wait complete. 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.230+0000 7fa0d4025740 20 client.27746 trim_cache size 0 max 16384 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0d4025740 1 Processor -- start 2024-08-28T19:32:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0d4025740 1 -- start start 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d187a1b0 0x5579d1c5fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d1882390 0x5579d1c601d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d18a3070 0x5579d1c4b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0d4025740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c81b3060 con 0x5579d187a1b0 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0d4025740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c81b4c90 con 0x5579d1882390 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0d4025740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c81b37f0 con 0x5579d18a3070 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d187a1b0 0x5579d1c5fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d187a1b0 0x5579d1c5fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46462/0 (socket says 172.21.15.16:46462) 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0c77fe640 1 -- 172.21.15.16:0/1857509677 learned_addr learned my addr 172.21.15.16:0/1857509677 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.249+0000 7fa0c6ffd640 1 --2- 172.21.15.16:0/1857509677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d1882390 0x5579d1c601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0c77fe640 1 -- 172.21.15.16:0/1857509677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d18a3070 msgr2=0x5579d1c4b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0c7fff640 1 --2- 172.21.15.16:0/1857509677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d18a3070 0x5579d1c4b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/1857509677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d18a3070 0x5579d1c4b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0c77fe640 1 -- 172.21.15.16:0/1857509677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d1882390 msgr2=0x5579d1c601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/1857509677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d1882390 0x5579d1c601d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0c77fe640 1 -- 172.21.15.16:0/1857509677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579d1c548f0 con 0x5579d187a1b0 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0c6ffd640 1 --2- 172.21.15.16:0/1857509677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d1882390 0x5579d1c601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0c7fff640 1 --2- 172.21.15.16:0/1857509677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d18a3070 0x5579d1c4b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/1857509677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d187a1b0 0x5579d1c5fc90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80a8040 tx=0x7fa0b8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb490dfd0564bfd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0ad7fa640 1 -- 172.21.15.16:0/1857509677 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b80ab970 con 0x5579d187a1b0 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0ad7fa640 1 -- 172.21.15.16:0/1857509677 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0b8097020 con 0x5579d187a1b0 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.250+0000 7fa0ad7fa640 1 -- 172.21.15.16:0/1857509677 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b80ab460 con 0x5579d187a1b0 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.251+0000 7fa0d4025740 1 -- 172.21.15.16:0/1857509677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d187a1b0 msgr2=0x5579d1c5fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.251+0000 7fa0d4025740 1 --2- 172.21.15.16:0/1857509677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d187a1b0 0x5579d1c5fc90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80a8040 tx=0x7fa0b8000bc0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.251+0000 7fa0d4025740 1 -- 172.21.15.16:0/1857509677 shutdown_connections 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.251+0000 7fa0d4025740 1 --2- 172.21.15.16:0/1857509677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d18a3070 0x5579d1c4b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.251+0000 7fa0d4025740 1 --2- 172.21.15.16:0/1857509677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d1882390 0x5579d1c601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.251+0000 7fa0d4025740 1 --2- 172.21.15.16:0/1857509677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d187a1b0 0x5579d1c5fc90 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.251+0000 7fa0d4025740 1 -- 172.21.15.16:0/1857509677 >> 172.21.15.16:0/1857509677 conn(0x5579d1897460 msgr2=0x5579d1c522f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.251+0000 7fa0d4025740 1 -- 172.21.15.16:0/1857509677 shutdown_connections 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.251+0000 7fa0d4025740 1 -- 172.21.15.16:0/1857509677 wait complete. 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0d4025740 1 -- start start 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0d4025740 1 Processor -- start 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c58570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c58b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0d4025740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1cc9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0d4025740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c81b45c0 con 0x5579d18a3070 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0d4025740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c81b3ec0 con 0x5579d187a1b0 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0d4025740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c81b1060 con 0x5579d1882390 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0d4025740 10 client.0 _ll_register_callbacks cb 0x7fa0b800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1cc9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1cc9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46464/0 (socket says 172.21.15.16:46464) 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0c7fff640 1 -- 172.21.15.16:0/3485192975 learned_addr learned my addr 172.21.15.16:0/3485192975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c58570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.252+0000 7fa0c6ffd640 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c58b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0c7fff640 1 -- 172.21.15.16:0/3485192975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 msgr2=0x5579d1c58b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0c7fff640 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c58b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0c7fff640 1 -- 172.21.15.16:0/3485192975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 msgr2=0x5579d1c58570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0c7fff640 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c58570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0c7fff640 1 -- 172.21.15.16:0/3485192975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579d1cc9df0 con 0x5579d18a3070 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c58570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0c7fff640 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1cc9780 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0005c40 tx=0x7fa0c0003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2b08d4bdf6f3c3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/3485192975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0028070 con 0x5579d18a3070 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/3485192975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9578+0+0 (secure 0 0 0) 0x7fa0c0004d50 con 0x5579d18a3070 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0d4025740 1 -- 172.21.15.16:0/3485192975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0b8097020 con 0x5579d18a3070 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/3485192975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fa0c0034050 con 0x5579d18a3070 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0d4025740 10 client.18285 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0d4025740 1 -- 172.21.15.16:0/3485192975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5579d1cd1070 con 0x5579d18a3070 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.253+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/3485192975 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0b8097020 con 0x5579d18a3070 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.254+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/3485192975 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7fa0c0005360 con 0x5579d18a3070 2024-08-28T19:32:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.254+0000 7fa0d4025740 10 client.18285 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.254+0000 7fa0d4025740 10 client.18285 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.254+0000 7fa0d4025740 20 client.18285 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.254+0000 7fa0d4025740 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5579d1836da0 0x5579d1cd36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.254+0000 7fa0d4025740 4 client.18285 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.254+0000 7fa0d4025740 1 -- 172.21.15.16:0/3485192975 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5579d1bbd020 con 0x5579d1836da0 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.254+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5579d1836da0 0x5579d1cd36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.254+0000 7fa0a6ffd640 10 client.18285 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.254+0000 7fa0c77fe640 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5579d1836da0 0x5579d1cd36c0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=568e0b9434fdb5cf server_cookie=1e0c65278758d464 in_seq=0 out_seq=0 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0a6ffd640 1 -- 172.21.15.16:0/3485192975 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4350 (crc 0 0 0) 0x5579d1bbd020 con 0x5579d1836da0 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0a6ffd640 10 client.18285 handle_command_reply: tid=0 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 1 client.18285 shutdown 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 1 -- 172.21.15.16:0/3485192975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 msgr2=0x5579d1cc9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1cc9780 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0005c40 tx=0x7fa0c0003f60 comp rx=0 tx=0).stop 2024-08-28T19:32:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 1 -- 172.21.15.16:0/3485192975 shutdown_connections 2024-08-28T19:32:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579d18a3070 0x5579d1cc9780 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5579d1836da0 0x5579d1cd36c0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579d1882390 0x5579d1c58b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 1 --2- 172.21.15.16:0/3485192975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579d187a1b0 0x5579d1c58570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0a6ffd640 0 client.18285 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 1 -- 172.21.15.16:0/3485192975 >> 172.21.15.16:0/3485192975 conn(0x5579d1897460 msgr2=0x5579d1c4b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:05.523 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:32:05.523 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:32:05.523 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.524 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:32:05.525 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.526 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.527 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:32:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.530 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:32:05.531 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:05.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:32:05.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:32:05.533 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:32:05.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 1 -- 172.21.15.16:0/3485192975 shutdown_connections 2024-08-28T19:32:05.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 1 -- 172.21.15.16:0/3485192975 wait complete. 2024-08-28T19:32:05.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.255+0000 7fa0d4025740 20 client.18285 trim_cache size 0 max 16384 2024-08-28T19:32:05.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 -- 172.21.15.16:0/3887389201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0b00b4ca0 msgr2=0x7fa0b00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:05.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0b00b4ca0 0x7fa0b00b71b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009230 tx=0x7fa0bc031400 comp rx=0 tx=0).stop 2024-08-28T19:32:05.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 -- 172.21.15.16:0/3887389201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c8103910 msgr2=0x7fa0c80754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:05.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c8103910 0x7fa0c80754b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800e9f0 tx=0x7fa0b800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 -- 172.21.15.16:0/3887389201 shutdown_connections 2024-08-28T19:32:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa0b00b4ca0 0x7fa0b00b71b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c8109ba0 0x7fa0c8075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0c8109250 0x7fa0c80759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 --2- 172.21.15.16:0/3887389201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c8103910 0x7fa0c80754b0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 -- 172.21.15.16:0/3887389201 >> 172.21.15.16:0/3887389201 conn(0x7fa0c80ff380 msgr2=0x7fa0c81017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 -- 172.21.15.16:0/3887389201 shutdown_connections 2024-08-28T19:32:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:05.256+0000 7fa0cdab0640 1 -- 172.21.15.16:0/3887389201 wait complete. 2024-08-28T19:32:05.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460030 3.81 MB/sec execute 465 sec latency 631.999 ms 2024-08-28T19:32:06.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:05 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:32:06.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:05 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:32:06.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:05 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:32:06.391 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:32:06.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:32:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:06.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460033 3.80 MB/sec execute 466 sec latency 1628.883 ms 2024-08-28T19:32:07.019 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:06 smithi016 ceph-mon[18556]: pgmap v680: 129 pgs: 129 active+clean; 335 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 237 op/s 2024-08-28T19:32:07.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:06 smithi028 ceph-mon[21769]: pgmap v680: 129 pgs: 129 active+clean; 335 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 237 op/s 2024-08-28T19:32:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.109+0000 7fb015edc640 1 -- 172.21.15.16:0/2255672040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb01010dbd0 msgr2=0x7fb0101100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.109+0000 7fb015edc640 1 --2- 172.21.15.16:0/2255672040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb01010dbd0 0x7fb0101100a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fb008009ca0 tx=0x7fb00802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:32:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.109+0000 7fb015edc640 1 -- 172.21.15.16:0/2255672040 shutdown_connections 2024-08-28T19:32:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.109+0000 7fb015edc640 1 --2- 172.21.15.16:0/2255672040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb01010dbd0 0x7fb0101100a0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.109+0000 7fb015edc640 1 --2- 172.21.15.16:0/2255672040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb010105050 0x7fb01010d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.109+0000 7fb015edc640 1 --2- 172.21.15.16:0/2255672040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb010104620 0x7fb010104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.109+0000 7fb015edc640 1 -- 172.21.15.16:0/2255672040 >> 172.21.15.16:0/2255672040 conn(0x7fb0100794d0 msgr2=0x7fb0100798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.109+0000 7fb015edc640 1 -- 172.21.15.16:0/2255672040 shutdown_connections 2024-08-28T19:32:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.109+0000 7fb015edc640 1 -- 172.21.15.16:0/2255672040 wait complete. 2024-08-28T19:32:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb015edc640 1 Processor -- start 2024-08-28T19:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb015edc640 1 -- start start 2024-08-28T19:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb015edc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb010104620 0x7fb01019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb015edc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010105050 0x7fb01019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb015edc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb01010dbd0 0x7fb01019fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb015edc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb010113840 con 0x7fb010105050 2024-08-28T19:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb015edc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0101136c0 con 0x7fb01010dbd0 2024-08-28T19:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb015edc640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb0101139c0 con 0x7fb010104620 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb00effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010105050 0x7fb01019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb00f7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb010104620 0x7fb01019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb00effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010105050 0x7fb01019a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46486/0 (socket says 172.21.15.16:46486) 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb00effd640 1 -- 172.21.15.16:0/2231916773 learned_addr learned my addr 172.21.15.16:0/2231916773 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.110+0000 7fb00ffff640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb01010dbd0 0x7fb01019fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb00effd640 1 -- 172.21.15.16:0/2231916773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb010104620 msgr2=0x7fb01019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb00effd640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb010104620 0x7fb01019a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb00effd640 1 -- 172.21.15.16:0/2231916773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb01010dbd0 msgr2=0x7fb01019fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb00effd640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb01010dbd0 0x7fb01019fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb00effd640 1 -- 172.21.15.16:0/2231916773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0101a0140 con 0x7fb010105050 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb00f7fe640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb010104620 0x7fb01019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb00effd640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010105050 0x7fb01019a770 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fb00400e9c0 tx=0x7fb00400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb00cff9640 1 -- 172.21.15.16:0/2231916773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00400cd80 con 0x7fb010105050 2024-08-28T19:32:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb00cff9640 1 -- 172.21.15.16:0/2231916773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0040054d0 con 0x7fb010105050 2024-08-28T19:32:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb015edc640 1 -- 172.21.15.16:0/2231916773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0101a0430 con 0x7fb010105050 2024-08-28T19:32:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb00cff9640 1 -- 172.21.15.16:0/2231916773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004010430 con 0x7fb010105050 2024-08-28T19:32:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.111+0000 7fb015edc640 1 -- 172.21.15.16:0/2231916773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0101a6b90 con 0x7fb010105050 2024-08-28T19:32:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.112+0000 7fb015edc640 1 -- 172.21.15.16:0/2231916773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb010104a30 con 0x7fb010105050 2024-08-28T19:32:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.114+0000 7fb00cff9640 1 -- 172.21.15.16:0/2231916773 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fb004004fb0 con 0x7fb010105050 2024-08-28T19:32:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.115+0000 7fb00cff9640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faff80b4d20 0x7faff80b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.115+0000 7fb00f7fe640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faff80b4d20 0x7faff80b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.115+0000 7fb00cff9640 1 -- 172.21.15.16:0/2231916773 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9578+0+0 (secure 0 0 0) 0x7fb0040e5ea0 con 0x7fb010105050 2024-08-28T19:32:07.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.115+0000 7fb00f7fe640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faff80b4d20 0x7faff80b7230 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb01006b960 tx=0x7fb000005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:06 smithi115 ceph-mon[22184]: pgmap v680: 129 pgs: 129 active+clean; 335 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 237 op/s 2024-08-28T19:32:07.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.118+0000 7fb00cff9640 1 -- 172.21.15.16:0/2231916773 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fb004056130 con 0x7fb010105050 2024-08-28T19:32:07.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.355+0000 7fb015edc640 1 -- 172.21.15.16:0/2231916773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb01019b300 con 0x7fb010105050 2024-08-28T19:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.356+0000 7fb00cff9640 1 -- 172.21.15.16:0/2231916773 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+11985 (secure 0 0 0) 0x7fb0040ac870 con 0x7fb010105050 2024-08-28T19:32:07.359 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:32:07.359 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":58,"btime":"2024-08-28T19:32:04:770041+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":58,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:32:04.540216+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 58 2024-08-28T19:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 -- 172.21.15.16:0/2231916773 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faff80b4d20 msgr2=0x7faff80b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faff80b4d20 0x7faff80b7230 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb01006b960 tx=0x7fb000005f90 comp rx=0 tx=0).stop 2024-08-28T19:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 -- 172.21.15.16:0/2231916773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010105050 msgr2=0x7fb01019a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010105050 0x7fb01019a770 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fb00400e9c0 tx=0x7fb00400ee80 comp rx=0 tx=0).stop 2024-08-28T19:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 -- 172.21.15.16:0/2231916773 shutdown_connections 2024-08-28T19:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faff80b4d20 0x7faff80b7230 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb01010dbd0 0x7fb01019fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010105050 0x7fb01019a770 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 --2- 172.21.15.16:0/2231916773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb010104620 0x7fb01019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 -- 172.21.15.16:0/2231916773 >> 172.21.15.16:0/2231916773 conn(0x7fb0100794d0 msgr2=0x7fb010107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 -- 172.21.15.16:0/2231916773 shutdown_connections 2024-08-28T19:32:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:07.359+0000 7fb015edc640 1 -- 172.21.15.16:0/2231916773 wait complete. 2024-08-28T19:32:07.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462156 3.83 MB/sec execute 467 sec latency 19.143 ms 2024-08-28T19:32:08.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:07 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2231916773' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:32:08.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:32:08.058 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:07 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2231916773' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:32:08.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:07 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2231916773' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:32:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.767+0000 7f883854d640 1 -- 172.21.15.16:0/852097759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8830109b90 msgr2=0x7f88301103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.767+0000 7f883854d640 1 --2- 172.21.15.16:0/852097759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8830109b90 0x7f88301103c0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f8828009ca0 tx=0x7f882802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.767+0000 7f883854d640 1 -- 172.21.15.16:0/852097759 shutdown_connections 2024-08-28T19:32:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.767+0000 7f883854d640 1 --2- 172.21.15.16:0/852097759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8830109b90 0x7f88301103c0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.767+0000 7f883854d640 1 --2- 172.21.15.16:0/852097759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8830109240 0x7f8830109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.767+0000 7f883854d640 1 --2- 172.21.15.16:0/852097759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8830103900 0x7f8830103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.767+0000 7f883854d640 1 -- 172.21.15.16:0/852097759 >> 172.21.15.16:0/852097759 conn(0x7f883007b5f0 msgr2=0x7f883007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.767+0000 7f883854d640 1 -- 172.21.15.16:0/852097759 shutdown_connections 2024-08-28T19:32:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.767+0000 7f883854d640 1 -- 172.21.15.16:0/852097759 wait complete. 2024-08-28T19:32:08.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.768+0000 7f883854d640 1 Processor -- start 2024-08-28T19:32:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.768+0000 7f883854d640 1 -- start start 2024-08-28T19:32:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.768+0000 7f883854d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8830103900 0x7f883019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f883854d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8830109240 0x7f883019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f883854d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8830109b90 0x7f883019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f883854d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8830113850 con 0x7f8830103900 2024-08-28T19:32:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f883854d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88301136d0 con 0x7f8830109b90 2024-08-28T19:32:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f883854d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f88301139d0 con 0x7f8830109240 2024-08-28T19:32:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88362e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8830103900 0x7f883019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88362e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8830103900 0x7f883019a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46500/0 (socket says 172.21.15.16:46500) 2024-08-28T19:32:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88362e9640 1 -- 172.21.15.16:0/1998046442 learned_addr learned my addr 172.21.15.16:0/1998046442 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f8835ae8640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8830109240 0x7f883019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f8836aea640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8830109b90 0x7f883019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88362e9640 1 -- 172.21.15.16:0/1998046442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8830109240 msgr2=0x7f883019a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88362e9640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8830109240 0x7f883019a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88362e9640 1 -- 172.21.15.16:0/1998046442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8830109b90 msgr2=0x7f883019fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88362e9640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8830109b90 0x7f883019fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88362e9640 1 -- 172.21.15.16:0/1998046442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88301a0190 con 0x7f8830103900 2024-08-28T19:32:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88362e9640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8830103900 0x7f883019a220 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f882c00e9f0 tx=0x7f882c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88277fe640 1 -- 172.21.15.16:0/1998046442 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c00cd80 con 0x7f8830103900 2024-08-28T19:32:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f883854d640 1 -- 172.21.15.16:0/1998046442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88301a0480 con 0x7f8830103900 2024-08-28T19:32:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88277fe640 1 -- 172.21.15.16:0/1998046442 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f882c0054d0 con 0x7f8830103900 2024-08-28T19:32:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f88277fe640 1 -- 172.21.15.16:0/1998046442 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c010690 con 0x7f8830103900 2024-08-28T19:32:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.769+0000 7f883854d640 1 -- 172.21.15.16:0/1998046442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88301a6be0 con 0x7f8830103900 2024-08-28T19:32:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.772+0000 7f883eac2740 1 Processor -- start 2024-08-28T19:32:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.772+0000 7f883eac2740 1 -- start start 2024-08-28T19:32:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.772+0000 7f88277fe640 1 -- 172.21.15.16:0/1998046442 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f882c004e90 con 0x7f8830103900 2024-08-28T19:32:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.772+0000 7f883eac2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 0x55856898c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.772+0000 7f883eac2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x55856898c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f883eac2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 0x55856898cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f883eac2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558568903680 con 0x5585685c6390 2024-08-28T19:32:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f883eac2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585688f2c10 con 0x5585685be1b0 2024-08-28T19:32:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f883eac2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5585688f6bc0 con 0x5585685e7070 2024-08-28T19:32:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f8836aea640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 0x55856898c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f8835ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 0x55856898cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f88362e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x55856898c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f8836aea640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 0x55856898c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:39576/0 (socket says 172.21.15.16:39576) 2024-08-28T19:32:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f8835ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 0x55856898cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46510/0 (socket says 172.21.15.16:46510) 2024-08-28T19:32:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f8836aea640 1 -- 172.21.15.16:0/4108476601 learned_addr learned my addr 172.21.15.16:0/4108476601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f8836aea640 1 -- 172.21.15.16:0/4108476601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 msgr2=0x55856898c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f8836aea640 1 --2- 172.21.15.16:0/4108476601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x55856898c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f8836aea640 1 -- 172.21.15.16:0/4108476601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 msgr2=0x55856898cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f8836aea640 1 --2- 172.21.15.16:0/4108476601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 0x55856898cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f8836aea640 1 -- 172.21.15.16:0/4108476601 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558568652650 con 0x5585685e7070 2024-08-28T19:32:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f8836aea640 1 --2- 172.21.15.16:0/4108476601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 0x55856898c310 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8828005cc0 tx=0x7f882802f420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22e53e92e549b4f0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.773+0000 7f88277fe640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f881c0b4ca0 0x7f881c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.774+0000 7f88362e9640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f881c0b4ca0 0x7f881c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.774+0000 7f8824ff9640 1 -- 172.21.15.16:0/4108476601 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828031cf0 con 0x5585685e7070 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.774+0000 7f88277fe640 1 -- 172.21.15.16:0/1998046442 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9578+0+0 (secure 0 0 0) 0x7f882c0e5eb0 con 0x7f8830103900 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.774+0000 7f8824ff9640 1 -- 172.21.15.16:0/4108476601 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8828036040 con 0x5585685e7070 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.774+0000 7f8824ff9640 1 -- 172.21.15.16:0/4108476601 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828004040 con 0x5585685e7070 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f883eac2740 1 -- 172.21.15.16:0/4108476601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 msgr2=0x55856898c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f88362e9640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f881c0b4ca0 0x7f881c0b71b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f882c0e3290 tx=0x7f882c0e9a00 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f883eac2740 1 --2- 172.21.15.16:0/4108476601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 0x55856898c310 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8828005cc0 tx=0x7f882802f420 comp rx=0 tx=0).stop 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f883eac2740 1 -- 172.21.15.16:0/4108476601 shutdown_connections 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f883eac2740 1 --2- 172.21.15.16:0/4108476601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 0x55856898cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f883eac2740 1 --2- 172.21.15.16:0/4108476601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x55856898c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f883eac2740 1 --2- 172.21.15.16:0/4108476601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 0x55856898c310 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f883eac2740 1 -- 172.21.15.16:0/4108476601 >> 172.21.15.16:0/4108476601 conn(0x5585685db460 msgr2=0x55856898a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f883eac2740 1 -- 172.21.15.16:0/4108476601 shutdown_connections 2024-08-28T19:32:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f883eac2740 1 -- 172.21.15.16:0/4108476601 wait complete. 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f883eac2740 1 -- start start 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.775+0000 7f883eac2740 1 Processor -- start 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f883eac2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x558568997a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f883eac2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685c6390 0x558568997fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f883eac2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x5585689a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f883eac2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558568755700 con 0x5585685e7070 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f883eac2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558568879750 con 0x5585685be1b0 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f883eac2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55856899aa90 con 0x5585685c6390 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f883eac2740 10 client.0 _ll_register_callbacks cb 0x5585688a8e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8836aea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x5585689a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8836aea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x5585689a3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46518/0 (socket says 172.21.15.16:46518) 2024-08-28T19:32:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8836aea640 1 -- 172.21.15.16:0/1208054055 learned_addr learned my addr 172.21.15.16:0/1208054055 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8836aea640 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x558568997a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8835ae8640 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685c6390 0x558568997fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8836aea640 1 -- 172.21.15.16:0/1208054055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685c6390 msgr2=0x558568997fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8836aea640 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685c6390 0x558568997fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8836aea640 1 -- 172.21.15.16:0/1208054055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 msgr2=0x558568997a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8836aea640 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x558568997a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8836aea640 1 -- 172.21.15.16:0/1208054055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585688b2900 con 0x5585685e7070 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8836aea640 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x5585689a3db0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f8828003a90 tx=0x7f8828003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4333473aaa46d1f0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8819ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8819ffb640 1 -- 172.21.15.16:0/1208054055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882804a040 con 0x5585685e7070 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f8819ffb640 1 -- 172.21.15.16:0/1208054055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9578+0+0 (secure 0 0 0) 0x7f8828031cf0 con 0x5585685e7070 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.776+0000 7f883eac2740 1 -- 172.21.15.16:0/1208054055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8828036040 con 0x5585685e7070 2024-08-28T19:32:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f8819ffb640 1 -- 172.21.15.16:0/1208054055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f8828003e10 con 0x5585685e7070 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f883eac2740 10 client.18303 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f883eac2740 1 -- 172.21.15.16:0/1208054055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5585689a68a0 con 0x5585685e7070 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f8819ffb640 1 -- 172.21.15.16:0/1208054055 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8828036040 con 0x5585685e7070 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f8819ffb640 1 -- 172.21.15.16:0/1208054055 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7f8828050070 con 0x5585685e7070 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f883eac2740 10 client.18303 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f883eac2740 10 client.18303 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f883eac2740 20 client.18303 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f883eac2740 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5585685a59f0 0x5585689a9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f883eac2740 4 client.18303 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f883eac2740 1 -- 172.21.15.16:0/1208054055 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585689055d0 con 0x5585685a59f0 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.777+0000 7f8835ae8640 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5585685a59f0 0x5585689a9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.778+0000 7f8835ae8640 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5585685a59f0 0x5585689a9c00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fe20d298c1450209 server_cookie=b487aa259b4b25b5 in_seq=0 out_seq=0 2024-08-28T19:32:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.778+0000 7f8819ffb640 10 client.18303 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:32:08.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f8819ffb640 1 -- 172.21.15.16:0/1208054055 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5585689055d0 con 0x5585685a59f0 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f8819ffb640 10 client.18303 handle_command_reply: tid=0 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 1 client.18303 shutdown 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 1 -- 172.21.15.16:0/1208054055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 msgr2=0x5585689a3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x5585689a3db0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f8828003a90 tx=0x7f8828003c30 comp rx=0 tx=0).stop 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 1 -- 172.21.15.16:0/1208054055 shutdown_connections 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5585685a59f0 0x5585689a9c00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x5585689a3db0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685c6390 0x558568997fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 1 --2- 172.21.15.16:0/1208054055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x558568997a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f8819ffb640 0 client.18303 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 1 -- 172.21.15.16:0/1208054055 >> 172.21.15.16:0/1208054055 conn(0x5585685db460 msgr2=0x55856898f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 1 -- 172.21.15.16:0/1208054055 shutdown_connections 2024-08-28T19:32:08.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 1 -- 172.21.15.16:0/1208054055 wait complete. 2024-08-28T19:32:08.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.779+0000 7f883eac2740 20 client.18303 trim_cache size 0 max 16384 2024-08-28T19:32:08.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.792+0000 7f883eac2740 1 Processor -- start 2024-08-28T19:32:08.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.792+0000 7f883eac2740 1 -- start start 2024-08-28T19:32:08.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f883eac2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x55856898f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f883eac2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 0x55856898f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f883eac2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 0x558568994c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f883eac2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88301b0da0 con 0x5585685c6390 2024-08-28T19:32:08.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f883eac2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88301b29d0 con 0x5585685be1b0 2024-08-28T19:32:08.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f883eac2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f88301b1530 con 0x5585685e7070 2024-08-28T19:32:08.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f8835ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 0x55856898f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f8835ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 0x55856898f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46522/0 (socket says 172.21.15.16:46522) 2024-08-28T19:32:08.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f8835ae8640 1 -- 172.21.15.16:0/4179176582 learned_addr learned my addr 172.21.15.16:0/4179176582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:08.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f8836aea640 1 --2- 172.21.15.16:0/4179176582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x55856898f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f8836aea640 1 --2- 172.21.15.16:0/4179176582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 0x558568994c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f8835ae8640 1 -- 172.21.15.16:0/4179176582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 msgr2=0x558568994c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f8835ae8640 1 --2- 172.21.15.16:0/4179176582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 0x558568994c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f8835ae8640 1 -- 172.21.15.16:0/4179176582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 msgr2=0x55856898f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f8835ae8640 1 --2- 172.21.15.16:0/4179176582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x55856898f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f8835ae8640 1 -- 172.21.15.16:0/4179176582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585689926c0 con 0x5585685c6390 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f8835ae8640 1 --2- 172.21.15.16:0/4179176582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 0x55856898f950 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f8820009d20 tx=0x7f8820005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f45d3d54595ddf2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f881bfff640 1 -- 172.21.15.16:0/4179176582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882001f630 con 0x5585685c6390 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.793+0000 7f881bfff640 1 -- 172.21.15.16:0/4179176582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f882001fc80 con 0x5585685c6390 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.794+0000 7f881bfff640 1 -- 172.21.15.16:0/4179176582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8820011d70 con 0x5585685c6390 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.794+0000 7f883eac2740 1 -- 172.21.15.16:0/4179176582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 msgr2=0x55856898f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.794+0000 7f883eac2740 1 --2- 172.21.15.16:0/4179176582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 0x55856898f950 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f8820009d20 tx=0x7f8820005f90 comp rx=0 tx=0).stop 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.794+0000 7f883eac2740 1 -- 172.21.15.16:0/4179176582 shutdown_connections 2024-08-28T19:32:08.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.794+0000 7f883eac2740 1 --2- 172.21.15.16:0/4179176582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685e7070 0x558568994c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.794+0000 7f883eac2740 1 --2- 172.21.15.16:0/4179176582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685c6390 0x55856898f950 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.794+0000 7f883eac2740 1 --2- 172.21.15.16:0/4179176582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685be1b0 0x55856898f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.794+0000 7f883eac2740 1 -- 172.21.15.16:0/4179176582 >> 172.21.15.16:0/4179176582 conn(0x5585685db460 msgr2=0x558568996310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:08.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.794+0000 7f883eac2740 1 -- 172.21.15.16:0/4179176582 shutdown_connections 2024-08-28T19:32:08.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.794+0000 7f883eac2740 1 -- 172.21.15.16:0/4179176582 wait complete. 2024-08-28T19:32:08.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f883eac2740 1 -- start start 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f883eac2740 1 Processor -- start 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f883eac2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685be1b0 0x5585689aaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f883eac2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685c6390 0x5585689ab490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f883eac2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x558568a13380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f883eac2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88301b2300 con 0x5585685e7070 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f883eac2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88301b1c00 con 0x5585685c6390 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f883eac2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f88301aeda0 con 0x5585685be1b0 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f883eac2740 10 client.0 _ll_register_callbacks cb 0x7f8820009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f8836aea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x558568a13380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f8836aea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x558568a13380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46530/0 (socket says 172.21.15.16:46530) 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f8836aea640 1 -- 172.21.15.16:0/3189510874 learned_addr learned my addr 172.21.15.16:0/3189510874 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.795+0000 7f8836aea640 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685be1b0 0x5585689aaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f8835ae8640 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685c6390 0x5585689ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f8836aea640 1 -- 172.21.15.16:0/3189510874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685be1b0 msgr2=0x5585689aaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f8836aea640 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685be1b0 0x5585689aaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f8836aea640 1 -- 172.21.15.16:0/3189510874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685c6390 msgr2=0x5585689ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f8836aea640 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685c6390 0x5585689ab490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f8836aea640 1 -- 172.21.15.16:0/3189510874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585689abd60 con 0x5585685e7070 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f8836aea640 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x558568a13380 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f88280049d0 tx=0x7f8828003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3d13b556bef620 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f8819ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f8819ffb640 1 -- 172.21.15.16:0/3189510874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828031450 con 0x5585685e7070 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f8819ffb640 1 -- 172.21.15.16:0/3189510874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9578+0+0 (secure 0 0 0) 0x7f882805f040 con 0x5585685e7070 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f883eac2740 1 -- 172.21.15.16:0/3189510874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f882001fc80 con 0x5585685e7070 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f8819ffb640 1 -- 172.21.15.16:0/3189510874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f88280315f0 con 0x5585685e7070 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.796+0000 7f883eac2740 10 client.18309 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:08.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.797+0000 7f883eac2740 1 -- 172.21.15.16:0/3189510874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x558568a14920 con 0x5585685e7070 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.797+0000 7f8819ffb640 1 -- 172.21.15.16:0/3189510874 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f882001fc80 con 0x5585685e7070 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.797+0000 7f8819ffb640 1 -- 172.21.15.16:0/3189510874 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7f8828050040 con 0x5585685e7070 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.797+0000 7f883eac2740 10 client.18309 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.797+0000 7f883eac2740 10 client.18309 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.797+0000 7f883eac2740 20 client.18309 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.797+0000 7f883eac2740 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55856857ada0 0x558568a170b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.797+0000 7f883eac2740 4 client.18309 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.797+0000 7f883eac2740 1 -- 172.21.15.16:0/3189510874 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558568901070 con 0x55856857ada0 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.797+0000 7f8835ae8640 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55856857ada0 0x558568a170b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f8835ae8640 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55856857ada0 0x558568a170b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4ba30cd14cb1fb98 server_cookie=a9666c85e126abb0 in_seq=0 out_seq=0 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f8819ffb640 10 client.18309 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f8819ffb640 1 -- 172.21.15.16:0/3189510874 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558568901070 con 0x55856857ada0 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f8819ffb640 10 client.18309 handle_command_reply: tid=0 2024-08-28T19:32:08.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 1 client.18309 shutdown 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 1 -- 172.21.15.16:0/3189510874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 msgr2=0x558568a13380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x558568a13380 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f88280049d0 tx=0x7f8828003b90 comp rx=0 tx=0).stop 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 1 -- 172.21.15.16:0/3189510874 shutdown_connections 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55856857ada0 0x558568a170b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585685e7070 0x558568a13380 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f8819ffb640 0 client.18309 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585685c6390 0x5585689ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 1 --2- 172.21.15.16:0/3189510874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585685be1b0 0x5585689aaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 1 -- 172.21.15.16:0/3189510874 >> 172.21.15.16:0/3189510874 conn(0x5585685db460 msgr2=0x558568995ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 1 -- 172.21.15.16:0/3189510874 shutdown_connections 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 1 -- 172.21.15.16:0/3189510874 wait complete. 2024-08-28T19:32:08.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.798+0000 7f883eac2740 20 client.18309 trim_cache size 0 max 16384 2024-08-28T19:32:08.807 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:32:08.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 -- 172.21.15.16:0/1998046442 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f881c0b4ca0 msgr2=0x7f881c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:08.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f881c0b4ca0 0x7f881c0b71b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f882c0e3290 tx=0x7f882c0e9a00 comp rx=0 tx=0).stop 2024-08-28T19:32:08.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 -- 172.21.15.16:0/1998046442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8830103900 msgr2=0x7f883019a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:08.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8830103900 0x7f883019a220 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f882c00e9f0 tx=0x7f882c00edf0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 -- 172.21.15.16:0/1998046442 shutdown_connections 2024-08-28T19:32:08.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f881c0b4ca0 0x7f881c0b71b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8830109b90 0x7f883019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8830109240 0x7f883019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 --2- 172.21.15.16:0/1998046442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8830103900 0x7f883019a220 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:08.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 -- 172.21.15.16:0/1998046442 >> 172.21.15.16:0/1998046442 conn(0x7f883007b5f0 msgr2=0x7f8830105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:08.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 -- 172.21.15.16:0/1998046442 shutdown_connections 2024-08-28T19:32:08.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:08.799+0000 7f883854d640 1 -- 172.21.15.16:0/1998046442 wait complete. 2024-08-28T19:32:08.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462516 3.82 MB/sec execute 468 sec latency 584.319 ms 2024-08-28T19:32:09.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:08 smithi016 ceph-mon[18556]: pgmap v681: 129 pgs: 129 active+clean; 334 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 226 op/s 2024-08-28T19:32:09.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:08 smithi028 ceph-mon[21769]: pgmap v681: 129 pgs: 129 active+clean; 334 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 226 op/s 2024-08-28T19:32:09.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:08 smithi115 ceph-mon[22184]: pgmap v681: 129 pgs: 129 active+clean; 334 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 226 op/s 2024-08-28T19:32:09.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:32:09.877 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:09.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462516 3.81 MB/sec execute 469 sec latency 1584.477 ms 2024-08-28T19:32:10.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.448+0000 7f11e4296640 1 -- 172.21.15.16:0/302329530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc109c60 msgr2=0x7f11dc110490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:10.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.448+0000 7f11e4296640 1 --2- 172.21.15.16:0/302329530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc109c60 0x7f11dc110490 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009900 tx=0x7f11d402a410 comp rx=0 tx=0).stop 2024-08-28T19:32:10.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.448+0000 7f11e4296640 1 -- 172.21.15.16:0/302329530 shutdown_connections 2024-08-28T19:32:10.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.448+0000 7f11e4296640 1 --2- 172.21.15.16:0/302329530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc109c60 0x7f11dc110490 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.448+0000 7f11e4296640 1 --2- 172.21.15.16:0/302329530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11dc109310 0x7f11dc109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.448+0000 7f11e4296640 1 --2- 172.21.15.16:0/302329530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11dc1039d0 0x7f11dc103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.448+0000 7f11e4296640 1 -- 172.21.15.16:0/302329530 >> 172.21.15.16:0/302329530 conn(0x7f11dc07b670 msgr2=0x7f11dc07ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.448+0000 7f11e4296640 1 -- 172.21.15.16:0/302329530 shutdown_connections 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.448+0000 7f11e4296640 1 -- 172.21.15.16:0/302329530 wait complete. 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e4296640 1 Processor -- start 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e4296640 1 -- start start 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e4296640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11dc1039d0 0x7f11dc19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e4296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc109310 0x7f11dc19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e4296640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11dc109c60 0x7f11dc19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e4296640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc113880 con 0x7f11dc109310 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e4296640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc113700 con 0x7f11dc1039d0 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e4296640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc113a00 con 0x7f11dc109c60 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e1831640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc109310 0x7f11dc19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e1831640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc109310 0x7f11dc19a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46556/0 (socket says 172.21.15.16:46556) 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e1831640 1 -- 172.21.15.16:0/266878682 learned_addr learned my addr 172.21.15.16:0/266878682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e2032640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11dc1039d0 0x7f11dc19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.449+0000 7f11e2833640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11dc109c60 0x7f11dc19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.450+0000 7f11e1831640 1 -- 172.21.15.16:0/266878682 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11dc109c60 msgr2=0x7f11dc19fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.450+0000 7f11e1831640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11dc109c60 0x7f11dc19fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.450+0000 7f11e1831640 1 -- 172.21.15.16:0/266878682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11dc1039d0 msgr2=0x7f11dc19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.450+0000 7f11e1831640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11dc1039d0 0x7f11dc19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.450+0000 7f11e1831640 1 -- 172.21.15.16:0/266878682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc1a0200 con 0x7f11dc109310 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.450+0000 7f11e1831640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc109310 0x7f11dc19a830 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f11d000c970 tx=0x7f11d000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.450+0000 7f11db7fe640 1 -- 172.21.15.16:0/266878682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0004220 con 0x7f11dc109310 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.450+0000 7f11db7fe640 1 -- 172.21.15.16:0/266878682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11d00043c0 con 0x7f11dc109310 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.450+0000 7f11e4296640 1 -- 172.21.15.16:0/266878682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11dc1a04f0 con 0x7f11dc109310 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.450+0000 7f11db7fe640 1 -- 172.21.15.16:0/266878682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0004a00 con 0x7f11dc109310 2024-08-28T19:32:10.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.450+0000 7f11e4296640 1 -- 172.21.15.16:0/266878682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11dc1a6c50 con 0x7f11dc109310 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.453+0000 7f11ea80b740 1 Processor -- start 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.453+0000 7f11ea80b740 1 -- start start 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.453+0000 7f11db7fe640 1 -- 172.21.15.16:0/266878682 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f11d00079b0 con 0x7f11dc109310 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 0x561bed18c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed18c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdc6390 0x561bed18cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11ea80b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561bed103680 con 0x561becde7070 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11ea80b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561bed0f2c10 con 0x561becdc6390 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11ea80b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561bed0f6bc0 con 0x561becdbe1b0 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11e2032640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed18c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11e2833640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 0x561bed18c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11e2032640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed18c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:39626/0 (socket says 172.21.15.16:39626) 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11e2032640 1 -- 172.21.15.16:0/612367907 learned_addr learned my addr 172.21.15.16:0/612367907 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11e1831640 1 --2- 172.21.15.16:0/612367907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdc6390 0x561bed18cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11e2032640 1 -- 172.21.15.16:0/612367907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdc6390 msgr2=0x561bed18cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11e2032640 1 --2- 172.21.15.16:0/612367907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdc6390 0x561bed18cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11e2032640 1 -- 172.21.15.16:0/612367907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 msgr2=0x561bed18c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11e2032640 1 --2- 172.21.15.16:0/612367907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 0x561bed18c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11e2032640 1 -- 172.21.15.16:0/612367907 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bece52650 con 0x561becdbe1b0 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11e2032640 1 --2- 172.21.15.16:0/612367907 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed18c850 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00af40 tx=0x7f11cc00f850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ff9a7c4bf061976 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.454+0000 7f11db7fe640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11c40b4ca0 0x7f11c40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.455+0000 7f11e2833640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11c40b4ca0 0x7f11c40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.455+0000 7f11d8d9a640 1 -- 172.21.15.16:0/612367907 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc01d070 con 0x561becdbe1b0 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.455+0000 7f11d8d9a640 1 -- 172.21.15.16:0/612367907 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11cc010db0 con 0x561becdbe1b0 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.455+0000 7f11d8d9a640 1 -- 172.21.15.16:0/612367907 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc018da0 con 0x561becdbe1b0 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.455+0000 7f11db7fe640 1 -- 172.21.15.16:0/266878682 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f11d00e54d0 con 0x7f11dc109310 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.455+0000 7f11ea80b740 1 -- 172.21.15.16:0/612367907 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 msgr2=0x561bed18c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.455+0000 7f11ea80b740 1 --2- 172.21.15.16:0/612367907 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed18c850 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00af40 tx=0x7f11cc00f850 comp rx=0 tx=0).stop 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.455+0000 7f11e2833640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11c40b4ca0 0x7f11c40b71b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f11d4006010 tx=0x7f11d40058d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 -- 172.21.15.16:0/612367907 shutdown_connections 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 --2- 172.21.15.16:0/612367907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdc6390 0x561bed18cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 --2- 172.21.15.16:0/612367907 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed18c850 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 --2- 172.21.15.16:0/612367907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 0x561bed18c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 -- 172.21.15.16:0/612367907 >> 172.21.15.16:0/612367907 conn(0x561becddb460 msgr2=0x561bed18a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 -- 172.21.15.16:0/612367907 shutdown_connections 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 -- 172.21.15.16:0/612367907 wait complete. 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 -- start start 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 Processor -- start 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed197a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed197f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becde7070 0x561bed1a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.456+0000 7f11ea80b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561becf55700 con 0x561becdc6390 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11ea80b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561bed079750 con 0x561becde7070 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11ea80b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561bed19aaf0 con 0x561becdbe1b0 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e1831640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11ea80b740 10 client.0 _ll_register_callbacks cb 0x561bed0a8e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e1831640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46564/0 (socket says 172.21.15.16:46564) 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e1831640 1 -- 172.21.15.16:0/779303564 learned_addr learned my addr 172.21.15.16:0/779303564 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e2032640 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e2833640 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becde7070 0x561bed1a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e1831640 1 -- 172.21.15.16:0/779303564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 msgr2=0x561bed197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e1831640 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e1831640 1 -- 172.21.15.16:0/779303564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becde7070 msgr2=0x561bed1a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e1831640 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becde7070 0x561bed1a3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e1831640 1 -- 172.21.15.16:0/779303564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bed0b2900 con 0x561becdc6390 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e2032640 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11e1831640 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed197f70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f11d00ab8d0 tx=0x7f11d00ab7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7678e3581d16d7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11c1ffb640 1 -- 172.21.15.16:0/779303564 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d001d030 con 0x561becdc6390 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11c1ffb640 1 -- 172.21.15.16:0/779303564 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f11d00a7bd0 con 0x561becdc6390 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11ea80b740 1 -- 172.21.15.16:0/779303564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11cc010db0 con 0x561becdc6390 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11c1ffb640 1 -- 172.21.15.16:0/779303564 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f11d00056a0 con 0x561becdc6390 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11ea80b740 10 client.18321 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.457+0000 7f11ea80b740 1 -- 172.21.15.16:0/779303564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x561bed1a6830 con 0x561becdc6390 2024-08-28T19:32:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.458+0000 7f11c1ffb640 1 -- 172.21.15.16:0/779303564 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11cc010db0 con 0x561becdc6390 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.458+0000 7f11c1ffb640 1 -- 172.21.15.16:0/779303564 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7f11d00e5170 con 0x561becdc6390 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.458+0000 7f11ea80b740 10 client.18321 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.458+0000 7f11ea80b740 10 client.18321 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.458+0000 7f11ea80b740 20 client.18321 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.458+0000 7f11ea80b740 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561becda59f0 0x561bed1a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.458+0000 7f11ea80b740 4 client.18321 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.458+0000 7f11ea80b740 1 -- 172.21.15.16:0/779303564 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bed1055d0 con 0x561becda59f0 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.458+0000 7f11e2032640 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561becda59f0 0x561bed1a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.459+0000 7f11e2032640 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561becda59f0 0x561bed1a9ce0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=709d6dfd9ccc3f89 server_cookie=8b056f1bf76fcc3d in_seq=0 out_seq=0 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.459+0000 7f11c1ffb640 10 client.18321 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.459+0000 7f11c1ffb640 1 -- 172.21.15.16:0/779303564 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561bed1055d0 con 0x561becda59f0 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.459+0000 7f11c1ffb640 10 client.18321 handle_command_reply: tid=0 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.459+0000 7f11ea80b740 1 client.18321 shutdown 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11ea80b740 1 -- 172.21.15.16:0/779303564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 msgr2=0x561bed197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11ea80b740 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed197f70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f11d00ab8d0 tx=0x7f11d00ab7d0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11ea80b740 1 -- 172.21.15.16:0/779303564 shutdown_connections 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11ea80b740 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561becda59f0 0x561bed1a9ce0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11ea80b740 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becde7070 0x561bed1a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11ea80b740 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed197f70 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11ea80b740 1 --2- 172.21.15.16:0/779303564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11c1ffb640 0 client.18321 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11ea80b740 1 -- 172.21.15.16:0/779303564 >> 172.21.15.16:0/779303564 conn(0x561becddb460 msgr2=0x561bed18f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11ea80b740 1 -- 172.21.15.16:0/779303564 shutdown_connections 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11ea80b740 1 -- 172.21.15.16:0/779303564 wait complete. 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.460+0000 7f11ea80b740 20 client.18321 trim_cache size 0 max 16384 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11ea80b740 1 Processor -- start 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11ea80b740 1 -- start start 2024-08-28T19:32:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdbe1b0 0x561bed18a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed1a3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becde7070 0x561bed18f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11ea80b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1aeb70 con 0x561becdc6390 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11ea80b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1b07a0 con 0x561becdbe1b0 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11ea80b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1af300 con 0x561becde7070 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11e1831640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed1a3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11e1831640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed1a3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46572/0 (socket says 172.21.15.16:46572) 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11e1831640 1 -- 172.21.15.16:0/2260757091 learned_addr learned my addr 172.21.15.16:0/2260757091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11e2032640 1 --2- 172.21.15.16:0/2260757091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdbe1b0 0x561bed18a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.473+0000 7f11e2833640 1 --2- 172.21.15.16:0/2260757091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becde7070 0x561bed18f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.474+0000 7f11e1831640 1 -- 172.21.15.16:0/2260757091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becde7070 msgr2=0x561bed18f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.474+0000 7f11e1831640 1 --2- 172.21.15.16:0/2260757091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becde7070 0x561bed18f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.474+0000 7f11e1831640 1 -- 172.21.15.16:0/2260757091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdbe1b0 msgr2=0x561bed18a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.474+0000 7f11e1831640 1 --2- 172.21.15.16:0/2260757091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdbe1b0 0x561bed18a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.474+0000 7f11e1831640 1 -- 172.21.15.16:0/2260757091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bed19a170 con 0x561becdc6390 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.474+0000 7f11e2833640 1 --2- 172.21.15.16:0/2260757091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becde7070 0x561bed18f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.474+0000 7f11e1831640 1 --2- 172.21.15.16:0/2260757091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed1a3d10 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f11d00a79c0 tx=0x7f11d00076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e32335c4b2cb0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.474+0000 7f11c3fff640 1 -- 172.21.15.16:0/2260757091 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d00056a0 con 0x561becdc6390 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.474+0000 7f11c3fff640 1 -- 172.21.15.16:0/2260757091 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11d00ab450 con 0x561becdc6390 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.474+0000 7f11c3fff640 1 -- 172.21.15.16:0/2260757091 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d00a7d00 con 0x561becdc6390 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 -- 172.21.15.16:0/2260757091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 msgr2=0x561bed1a3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 --2- 172.21.15.16:0/2260757091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed1a3d10 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f11d00a79c0 tx=0x7f11d00076c0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 -- 172.21.15.16:0/2260757091 shutdown_connections 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 --2- 172.21.15.16:0/2260757091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becde7070 0x561bed18f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 --2- 172.21.15.16:0/2260757091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becdc6390 0x561bed1a3d10 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 --2- 172.21.15.16:0/2260757091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdbe1b0 0x561bed18a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.507 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:32:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 -- 172.21.15.16:0/2260757091 >> 172.21.15.16:0/2260757091 conn(0x561becddb460 msgr2=0x561bed189b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 -- 172.21.15.16:0/2260757091 shutdown_connections 2024-08-28T19:32:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 -- 172.21.15.16:0/2260757091 wait complete. 2024-08-28T19:32:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 -- start start 2024-08-28T19:32:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 Processor -- start 2024-08-28T19:32:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed1a0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdc6390 0x561bed19bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 0x561bed19c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1b00d0 con 0x561becde7070 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1af9d0 con 0x561becdc6390 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.475+0000 7f11ea80b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1acb70 con 0x561becdbe1b0 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11ea80b740 10 client.0 _ll_register_callbacks cb 0x7f11d000c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11e2833640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 0x561bed19c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11e2833640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 0x561bed19c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46578/0 (socket says 172.21.15.16:46578) 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11e2833640 1 -- 172.21.15.16:0/1799811439 learned_addr learned my addr 172.21.15.16:0/1799811439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11e1831640 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdc6390 0x561bed19bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11e2032640 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed1a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11e2833640 1 -- 172.21.15.16:0/1799811439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 msgr2=0x561bed1a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11e2833640 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed1a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11e2833640 1 -- 172.21.15.16:0/1799811439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdc6390 msgr2=0x561bed19bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11e2833640 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdc6390 0x561bed19bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11e2833640 1 -- 172.21.15.16:0/1799811439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bed2136e0 con 0x561becde7070 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11e2833640 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 0x561bed19c3b0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f11d402a930 tx=0x7f11d402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc0be8ba73ae23 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11c1ffb640 1 -- 172.21.15.16:0/1799811439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4059060 con 0x561becde7070 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11c1ffb640 1 -- 172.21.15.16:0/1799811439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f11d4043030 con 0x561becde7070 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11ea80b740 1 -- 172.21.15.16:0/1799811439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11d00ab450 con 0x561becde7070 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11c1ffb640 1 -- 172.21.15.16:0/1799811439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f11d4060050 con 0x561becde7070 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11ea80b740 10 client.18327 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.476+0000 7f11ea80b740 1 -- 172.21.15.16:0/1799811439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x561bed2147d0 con 0x561becde7070 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.477+0000 7f11c1ffb640 1 -- 172.21.15.16:0/1799811439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11d00ab450 con 0x561becde7070 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.477+0000 7f11c1ffb640 1 -- 172.21.15.16:0/1799811439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7f11d40037f0 con 0x561becde7070 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.477+0000 7f11ea80b740 10 client.18327 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.477+0000 7f11ea80b740 10 client.18327 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.477+0000 7f11ea80b740 20 client.18327 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.477+0000 7f11ea80b740 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561becd7ada0 0x561bed216fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.477+0000 7f11ea80b740 4 client.18327 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.477+0000 7f11ea80b740 1 -- 172.21.15.16:0/1799811439 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561bed101070 con 0x561becd7ada0 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.477+0000 7f11e2032640 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561becd7ada0 0x561bed216fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11e2032640 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561becd7ada0 0x561bed216fc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=375267eba9008a44 server_cookie=26d46c8a6cf2eaaa in_seq=0 out_seq=0 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11c1ffb640 10 client.18327 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11c1ffb640 1 -- 172.21.15.16:0/1799811439 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561bed101070 con 0x561becd7ada0 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11c1ffb640 10 client.18327 handle_command_reply: tid=0 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 1 client.18327 shutdown 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 1 -- 172.21.15.16:0/1799811439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 msgr2=0x561bed19c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 0x561bed19c3b0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f11d402a930 tx=0x7f11d402a410 comp rx=0 tx=0).stop 2024-08-28T19:32:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 1 -- 172.21.15.16:0/1799811439 shutdown_connections 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x561becd7ada0 0x561bed216fc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561becde7070 0x561bed19c3b0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561becdc6390 0x561bed19bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11c1ffb640 0 client.18327 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 1 --2- 172.21.15.16:0/1799811439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561becdbe1b0 0x561bed1a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 1 -- 172.21.15.16:0/1799811439 >> 172.21.15.16:0/1799811439 conn(0x561becddb460 msgr2=0x561bed18f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 1 -- 172.21.15.16:0/1799811439 shutdown_connections 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 1 -- 172.21.15.16:0/1799811439 wait complete. 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.478+0000 7f11ea80b740 20 client.18327 trim_cache size 0 max 16384 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 -- 172.21.15.16:0/266878682 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11c40b4ca0 msgr2=0x7f11c40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11c40b4ca0 0x7f11c40b71b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f11d4006010 tx=0x7f11d40058d0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 -- 172.21.15.16:0/266878682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc109310 msgr2=0x7f11dc19a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc109310 0x7f11dc19a830 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f11d000c970 tx=0x7f11d000ce30 comp rx=0 tx=0).stop 2024-08-28T19:32:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 -- 172.21.15.16:0/266878682 shutdown_connections 2024-08-28T19:32:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11dc109c60 0x7f11dc19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc109310 0x7f11dc19a830 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11c40b4ca0 0x7f11c40b71b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 --2- 172.21.15.16:0/266878682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11dc1039d0 0x7f11dc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 -- 172.21.15.16:0/266878682 >> 172.21.15.16:0/266878682 conn(0x7f11dc07b670 msgr2=0x7f11dc1073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 -- 172.21.15.16:0/266878682 shutdown_connections 2024-08-28T19:32:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:10.479+0000 7f11e4296640 1 -- 172.21.15.16:0/266878682 wait complete. 2024-08-28T19:32:10.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462516 3.80 MB/sec execute 470 sec latency 2584.573 ms 2024-08-28T19:32:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:11 smithi028 ceph-mon[21769]: pgmap v682: 129 pgs: 129 active+clean; 335 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-08-28T19:32:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:11 smithi028 ceph-mon[21769]: osdmap e94: 12 total, 12 up, 12 in 2024-08-28T19:32:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:11 smithi115 ceph-mon[22184]: pgmap v682: 129 pgs: 129 active+clean; 335 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-08-28T19:32:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:11 smithi115 ceph-mon[22184]: osdmap e94: 12 total, 12 up, 12 in 2024-08-28T19:32:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:11 smithi016 ceph-mon[18556]: pgmap v682: 129 pgs: 129 active+clean; 335 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-08-28T19:32:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:11 smithi016 ceph-mon[18556]: osdmap e94: 12 total, 12 up, 12 in 2024-08-28T19:32:11.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:32:11.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462516 3.79 MB/sec execute 471 sec latency 3584.633 ms 2024-08-28T19:32:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:12.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.630+0000 7fb42de7e640 1 -- 172.21.15.16:0/2047729652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb428109280 msgr2=0x7fb428109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:12.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.630+0000 7fb42de7e640 1 --2- 172.21.15.16:0/2047729652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb428109280 0x7fb428109690 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb410009900 tx=0x7fb41002a410 comp rx=0 tx=0).stop 2024-08-28T19:32:12.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.630+0000 7fb42de7e640 1 -- 172.21.15.16:0/2047729652 shutdown_connections 2024-08-28T19:32:12.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.630+0000 7fb42de7e640 1 --2- 172.21.15.16:0/2047729652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb428109bd0 0x7fb428110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.630+0000 7fb42de7e640 1 --2- 172.21.15.16:0/2047729652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb428109280 0x7fb428109690 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.630+0000 7fb42de7e640 1 --2- 172.21.15.16:0/2047729652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb428103940 0x7fb428103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.630+0000 7fb42de7e640 1 -- 172.21.15.16:0/2047729652 >> 172.21.15.16:0/2047729652 conn(0x7fb4280ff3d0 msgr2=0x7fb428101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.630+0000 7fb42de7e640 1 -- 172.21.15.16:0/2047729652 shutdown_connections 2024-08-28T19:32:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.630+0000 7fb42de7e640 1 -- 172.21.15.16:0/2047729652 wait complete. 2024-08-28T19:32:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.631+0000 7fb42de7e640 1 Processor -- start 2024-08-28T19:32:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.631+0000 7fb42de7e640 1 -- start start 2024-08-28T19:32:12.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.631+0000 7fb42de7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb428103940 0x7fb42819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.631+0000 7fb42de7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb428109280 0x7fb42819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.631+0000 7fb42de7e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb428109bd0 0x7fb42819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.631+0000 7fb42de7e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb428113890 con 0x7fb428103940 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.631+0000 7fb42de7e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb428113710 con 0x7fb428109280 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.631+0000 7fb42de7e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb428113a10 con 0x7fb428109bd0 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb428103940 0x7fb42819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb428103940 0x7fb42819a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46602/0 (socket says 172.21.15.16:46602) 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4277fe640 1 -- 172.21.15.16:0/3589549205 learned_addr learned my addr 172.21.15.16:0/3589549205 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb41effd640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb428109280 0x7fb42819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb427fff640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb428109bd0 0x7fb42819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4277fe640 1 -- 172.21.15.16:0/3589549205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb428109bd0 msgr2=0x7fb42819fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4277fe640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb428109bd0 0x7fb42819fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4277fe640 1 -- 172.21.15.16:0/3589549205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb428109280 msgr2=0x7fb42819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4277fe640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb428109280 0x7fb42819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4277fe640 1 -- 172.21.15.16:0/3589549205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4281a0210 con 0x7fb428103940 2024-08-28T19:32:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4277fe640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb428103940 0x7fb42819a2a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fb41800d950 tx=0x7fb41800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4257fa640 1 -- 172.21.15.16:0/3589549205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb418017070 con 0x7fb428103940 2024-08-28T19:32:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb42de7e640 1 -- 172.21.15.16:0/3589549205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4281a0500 con 0x7fb428103940 2024-08-28T19:32:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4257fa640 1 -- 172.21.15.16:0/3589549205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb418009c30 con 0x7fb428103940 2024-08-28T19:32:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb4257fa640 1 -- 172.21.15.16:0/3589549205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41800f620 con 0x7fb428103940 2024-08-28T19:32:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.632+0000 7fb42de7e640 1 -- 172.21.15.16:0/3589549205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4281a6c60 con 0x7fb428103940 2024-08-28T19:32:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.635+0000 7fb4343f3740 1 Processor -- start 2024-08-28T19:32:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.635+0000 7fb4343f3740 1 -- start start 2024-08-28T19:32:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.635+0000 7fb4257fa640 1 -- 172.21.15.16:0/3589549205 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fb41800f7c0 con 0x7fb428103940 2024-08-28T19:32:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.635+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b47070 0x562306eec310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.635+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 0x562306eec850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.635+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b26390 0x562306eecd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.635+0000 7fb4343f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562306e63680 con 0x562306b1e1b0 2024-08-28T19:32:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.635+0000 7fb4343f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562306e52c10 con 0x562306b47070 2024-08-28T19:32:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.635+0000 7fb4343f3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562306e56bc0 con 0x562306b26390 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb427fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b47070 0x562306eec310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb41effd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b26390 0x562306eecd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb427fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b47070 0x562306eec310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:42542/0 (socket says 172.21.15.16:42542) 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb427fff640 1 -- 172.21.15.16:0/4135285455 learned_addr learned my addr 172.21.15.16:0/4135285455 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb4277fe640 1 --2- 172.21.15.16:0/4135285455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 0x562306eec850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb427fff640 1 -- 172.21.15.16:0/4135285455 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b26390 msgr2=0x562306eecd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb427fff640 1 --2- 172.21.15.16:0/4135285455 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b26390 0x562306eecd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb427fff640 1 -- 172.21.15.16:0/4135285455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 msgr2=0x562306eec850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb427fff640 1 --2- 172.21.15.16:0/4135285455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 0x562306eec850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb427fff640 1 -- 172.21.15.16:0/4135285455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562306bb2650 con 0x562306b47070 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb4257fa640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4140b4ca0 0x7fb4140b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb427fff640 1 --2- 172.21.15.16:0/4135285455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b47070 0x562306eec310 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb420008dc0 tx=0x7fb42000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28b02ab7e8c92e39 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb4277fe640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4140b4ca0 0x7fb4140b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb4257fa640 1 -- 172.21.15.16:0/3589549205 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fb4180e55a0 con 0x7fb428103940 2024-08-28T19:32:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.636+0000 7fb41dffb640 1 -- 172.21.15.16:0/4135285455 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb420014070 con 0x562306b47070 2024-08-28T19:32:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.637+0000 7fb41dffb640 1 -- 172.21.15.16:0/4135285455 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb420003b70 con 0x562306b47070 2024-08-28T19:32:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.637+0000 7fb41dffb640 1 -- 172.21.15.16:0/4135285455 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb420010730 con 0x562306b47070 2024-08-28T19:32:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.637+0000 7fb4343f3740 1 -- 172.21.15.16:0/4135285455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b47070 msgr2=0x562306eec310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.637+0000 7fb4277fe640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4140b4ca0 0x7fb4140b71b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb4180e29d0 tx=0x7fb4180a7040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.637+0000 7fb4343f3740 1 --2- 172.21.15.16:0/4135285455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b47070 0x562306eec310 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb420008dc0 tx=0x7fb42000e720 comp rx=0 tx=0).stop 2024-08-28T19:32:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.637+0000 7fb4343f3740 1 -- 172.21.15.16:0/4135285455 shutdown_connections 2024-08-28T19:32:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.637+0000 7fb4343f3740 1 --2- 172.21.15.16:0/4135285455 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b26390 0x562306eecd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.637+0000 7fb4343f3740 1 --2- 172.21.15.16:0/4135285455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 0x562306eec850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.637+0000 7fb4343f3740 1 --2- 172.21.15.16:0/4135285455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b47070 0x562306eec310 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.637+0000 7fb4343f3740 1 -- 172.21.15.16:0/4135285455 >> 172.21.15.16:0/4135285455 conn(0x562306b3b460 msgr2=0x562306eea200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.638+0000 7fb4343f3740 1 -- 172.21.15.16:0/4135285455 shutdown_connections 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.638+0000 7fb4343f3740 1 -- 172.21.15.16:0/4135285455 wait complete. 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.638+0000 7fb4343f3740 1 -- start start 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.638+0000 7fb4343f3740 1 Processor -- start 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.638+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b1e1b0 0x562306ef7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.638+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b26390 0x562306ef7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.638+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b47070 0x562306f03de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.638+0000 7fb4343f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562306cb5700 con 0x562306b47070 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.638+0000 7fb4343f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562306dd9750 con 0x562306b26390 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.638+0000 7fb4343f3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562306efaaf0 con 0x562306b1e1b0 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb4343f3740 10 client.0 _ll_register_callbacks cb 0x562306e08e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb427fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b47070 0x562306f03de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb427fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b47070 0x562306f03de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46616/0 (socket says 172.21.15.16:46616) 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb427fff640 1 -- 172.21.15.16:0/2849482347 learned_addr learned my addr 172.21.15.16:0/2849482347 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb41effd640 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b26390 0x562306ef7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb427fff640 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b1e1b0 0x562306ef7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb41effd640 1 -- 172.21.15.16:0/2849482347 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b1e1b0 msgr2=0x562306ef7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb41effd640 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b1e1b0 0x562306ef7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb41effd640 1 -- 172.21.15.16:0/2849482347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b47070 msgr2=0x562306f03de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb41effd640 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b47070 0x562306f03de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb41effd640 1 -- 172.21.15.16:0/2849482347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562306e12900 con 0x562306b26390 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb427fff640 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b47070 0x562306f03de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb41effd640 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b26390 0x562306ef7fd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb41002a930 tx=0x7fb4100316d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d1fb11e42cc3e41 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb406ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.639+0000 7fb406ffd640 1 -- 172.21.15.16:0/2849482347 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410035040 con 0x562306b26390 2024-08-28T19:32:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb4343f3740 1 -- 172.21.15.16:0/2849482347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb420003b70 con 0x562306b26390 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb406ffd640 1 -- 172.21.15.16:0/2849482347 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fb410031d10 con 0x562306b26390 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb406ffd640 1 -- 172.21.15.16:0/2849482347 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fb41004e050 con 0x562306b26390 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb4343f3740 10 client.27803 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb4343f3740 1 -- 172.21.15.16:0/2849482347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x562306f068d0 con 0x562306b26390 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb406ffd640 1 -- 172.21.15.16:0/2849482347 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb420003b70 con 0x562306b26390 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb406ffd640 1 -- 172.21.15.16:0/2849482347 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7fb410004750 con 0x562306b26390 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb4343f3740 10 client.27803 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb4343f3740 10 client.27803 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb4343f3740 20 client.27803 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb4343f3740 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x562306b059f0 0x562306f09df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb4343f3740 4 client.27803 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.640+0000 7fb4343f3740 1 -- 172.21.15.16:0/2849482347 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562306e655d0 con 0x562306b059f0 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.641+0000 7fb427fff640 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x562306b059f0 0x562306f09df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.641+0000 7fb427fff640 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x562306b059f0 0x562306f09df0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=245cd8708598ca9e server_cookie=ea300ee3498638f8 in_seq=0 out_seq=0 2024-08-28T19:32:12.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.641+0000 7fb406ffd640 10 client.27803 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:32:12.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb406ffd640 1 -- 172.21.15.16:0/2849482347 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562306e655d0 con 0x562306b059f0 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb406ffd640 10 client.27803 handle_command_reply: tid=0 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 1 client.27803 shutdown 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 1 -- 172.21.15.16:0/2849482347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b26390 msgr2=0x562306ef7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b26390 0x562306ef7fd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb41002a930 tx=0x7fb4100316d0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 1 -- 172.21.15.16:0/2849482347 shutdown_connections 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x562306b059f0 0x562306f09df0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b47070 0x562306f03de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b26390 0x562306ef7fd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb406ffd640 0 client.27803 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 1 --2- 172.21.15.16:0/2849482347 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b1e1b0 0x562306ef7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 1 -- 172.21.15.16:0/2849482347 >> 172.21.15.16:0/2849482347 conn(0x562306b3b460 msgr2=0x562306eef4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 1 -- 172.21.15.16:0/2849482347 shutdown_connections 2024-08-28T19:32:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 1 -- 172.21.15.16:0/2849482347 wait complete. 2024-08-28T19:32:12.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.642+0000 7fb4343f3740 20 client.27803 trim_cache size 0 max 16384 2024-08-28T19:32:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.655+0000 7fb4343f3740 1 Processor -- start 2024-08-28T19:32:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.655+0000 7fb4343f3740 1 -- start start 2024-08-28T19:32:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b059f0 0x562306eea3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 0x562306eefb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b26390 0x562306f03ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb4343f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4281b0d70 con 0x562306b1e1b0 2024-08-28T19:32:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb4343f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4281b29a0 con 0x562306b059f0 2024-08-28T19:32:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb4343f3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb4281b1500 con 0x562306b26390 2024-08-28T19:32:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb41effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 0x562306eefb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb41effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 0x562306eefb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46620/0 (socket says 172.21.15.16:46620) 2024-08-28T19:32:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb41effd640 1 -- 172.21.15.16:0/3874724763 learned_addr learned my addr 172.21.15.16:0/3874724763 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb427fff640 1 --2- 172.21.15.16:0/3874724763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b26390 0x562306f03ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb427fff640 1 --2- 172.21.15.16:0/3874724763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b059f0 0x562306eea3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb41effd640 1 -- 172.21.15.16:0/3874724763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b26390 msgr2=0x562306f03ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb41effd640 1 --2- 172.21.15.16:0/3874724763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b26390 0x562306f03ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb41effd640 1 -- 172.21.15.16:0/3874724763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b059f0 msgr2=0x562306eea3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb41effd640 1 --2- 172.21.15.16:0/3874724763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b059f0 0x562306eea3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb41effd640 1 -- 172.21.15.16:0/3874724763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562306eed120 con 0x562306b1e1b0 2024-08-28T19:32:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb427fff640 1 --2- 172.21.15.16:0/3874724763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b059f0 0x562306eea3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:32:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.656+0000 7fb41effd640 1 --2- 172.21.15.16:0/3874724763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 0x562306eefb20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb410006010 tx=0x7fb410031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12321f92e8a5adcf server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.657+0000 7fb41d7fa640 1 -- 172.21.15.16:0/3874724763 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410035030 con 0x562306b1e1b0 2024-08-28T19:32:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.657+0000 7fb41d7fa640 1 -- 172.21.15.16:0/3874724763 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb410041070 con 0x562306b1e1b0 2024-08-28T19:32:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.657+0000 7fb41d7fa640 1 -- 172.21.15.16:0/3874724763 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410031b10 con 0x562306b1e1b0 2024-08-28T19:32:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.657+0000 7fb4343f3740 1 -- 172.21.15.16:0/3874724763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 msgr2=0x562306eefb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.657+0000 7fb4343f3740 1 --2- 172.21.15.16:0/3874724763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 0x562306eefb20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb410006010 tx=0x7fb410031a50 comp rx=0 tx=0).stop 2024-08-28T19:32:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.657+0000 7fb4343f3740 1 -- 172.21.15.16:0/3874724763 shutdown_connections 2024-08-28T19:32:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.657+0000 7fb4343f3740 1 --2- 172.21.15.16:0/3874724763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b26390 0x562306f03ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.657+0000 7fb4343f3740 1 --2- 172.21.15.16:0/3874724763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b1e1b0 0x562306eefb20 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.657+0000 7fb4343f3740 1 --2- 172.21.15.16:0/3874724763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b059f0 0x562306eea3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.657+0000 7fb4343f3740 1 -- 172.21.15.16:0/3874724763 >> 172.21.15.16:0/3874724763 conn(0x562306b3b460 msgr2=0x562306eef410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.657+0000 7fb4343f3740 1 -- 172.21.15.16:0/3874724763 shutdown_connections 2024-08-28T19:32:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.658+0000 7fb4343f3740 1 -- 172.21.15.16:0/3874724763 wait complete. 2024-08-28T19:32:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.658+0000 7fb4343f3740 1 -- start start 2024-08-28T19:32:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.658+0000 7fb4343f3740 1 Processor -- start 2024-08-28T19:32:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.658+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b059f0 0x562306f005f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.658+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b1e1b0 0x562306efbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.658+0000 7fb4343f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b26390 0x562306efc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.658+0000 7fb4343f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4281b22d0 con 0x562306b059f0 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.658+0000 7fb4343f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4281b1bd0 con 0x562306b26390 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.658+0000 7fb4343f3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb4281aed70 con 0x562306b1e1b0 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.658+0000 7fb4343f3740 10 client.0 _ll_register_callbacks cb 0x7fb410005e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.658+0000 7fb427fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b059f0 0x562306f005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb427fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b059f0 0x562306f005f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46636/0 (socket says 172.21.15.16:46636) 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb427fff640 1 -- 172.21.15.16:0/3547707881 learned_addr learned my addr 172.21.15.16:0/3547707881 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb41effd640 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b1e1b0 0x562306efbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb427fff640 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b26390 0x562306efc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb427fff640 1 -- 172.21.15.16:0/3547707881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b1e1b0 msgr2=0x562306efbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb427fff640 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b1e1b0 0x562306efbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb427fff640 1 -- 172.21.15.16:0/3547707881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b26390 msgr2=0x562306efc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb427fff640 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b26390 0x562306efc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb427fff640 1 -- 172.21.15.16:0/3547707881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562306efc880 con 0x562306b059f0 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb427fff640 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b059f0 0x562306f005f0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb4200104e0 tx=0x7fb410000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d6dd87217bce95 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb406ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb406ffd640 1 -- 172.21.15.16:0/3547707881 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb420014040 con 0x562306b059f0 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb406ffd640 1 -- 172.21.15.16:0/3547707881 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fb420010ca0 con 0x562306b059f0 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb4343f3740 1 -- 172.21.15.16:0/3547707881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb410041070 con 0x562306b059f0 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb406ffd640 1 -- 172.21.15.16:0/3547707881 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fb420021040 con 0x562306b059f0 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.659+0000 7fb4343f3740 10 client.18345 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.660+0000 7fb4343f3740 1 -- 172.21.15.16:0/3547707881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x562306f74710 con 0x562306b059f0 2024-08-28T19:32:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.660+0000 7fb406ffd640 1 -- 172.21.15.16:0/3547707881 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb410041070 con 0x562306b059f0 2024-08-28T19:32:12.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.660+0000 7fb406ffd640 1 -- 172.21.15.16:0/3547707881 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7fb420027040 con 0x562306b059f0 2024-08-28T19:32:12.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.660+0000 7fb4343f3740 10 client.18345 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:12.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.660+0000 7fb4343f3740 10 client.18345 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:32:12.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.660+0000 7fb4343f3740 20 client.18345 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:12.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.660+0000 7fb4343f3740 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x562306b47070 0x562306f76d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:12.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.660+0000 7fb4343f3740 4 client.18345 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:32:12.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.660+0000 7fb4343f3740 1 -- 172.21.15.16:0/3547707881 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562306e61070 con 0x562306b47070 2024-08-28T19:32:12.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.660+0000 7fb41effd640 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x562306b47070 0x562306f76d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:12.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb41effd640 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x562306b47070 0x562306f76d40 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=497d36955121580c server_cookie=6054c39eb89c525f in_seq=0 out_seq=0 2024-08-28T19:32:12.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb406ffd640 10 client.18345 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:32:12.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb406ffd640 1 -- 172.21.15.16:0/3547707881 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562306e61070 con 0x562306b47070 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb406ffd640 10 client.18345 handle_command_reply: tid=0 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 1 client.18345 shutdown 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 1 -- 172.21.15.16:0/3547707881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b059f0 msgr2=0x562306f005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b059f0 0x562306f005f0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb4200104e0 tx=0x7fb410000bc0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 1 -- 172.21.15.16:0/3547707881 shutdown_connections 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x562306b47070 0x562306f76d40 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562306b26390 0x562306efc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562306b1e1b0 0x562306efbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 1 --2- 172.21.15.16:0/3547707881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562306b059f0 0x562306f005f0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb406ffd640 0 client.18345 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 1 -- 172.21.15.16:0/3547707881 >> 172.21.15.16:0/3547707881 conn(0x562306b3b460 msgr2=0x562306eed340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 1 -- 172.21.15.16:0/3547707881 shutdown_connections 2024-08-28T19:32:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 1 -- 172.21.15.16:0/3547707881 wait complete. 2024-08-28T19:32:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.661+0000 7fb4343f3740 20 client.18345 trim_cache size 0 max 16384 2024-08-28T19:32:12.669 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:32:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 -- 172.21.15.16:0/3589549205 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4140b4ca0 msgr2=0x7fb4140b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4140b4ca0 0x7fb4140b71b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb4180e29d0 tx=0x7fb4180a7040 comp rx=0 tx=0).stop 2024-08-28T19:32:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 -- 172.21.15.16:0/3589549205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb428103940 msgr2=0x7fb42819a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb428103940 0x7fb42819a2a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fb41800d950 tx=0x7fb41800de10 comp rx=0 tx=0).stop 2024-08-28T19:32:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 -- 172.21.15.16:0/3589549205 shutdown_connections 2024-08-28T19:32:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb428109bd0 0x7fb42819fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb4140b4ca0 0x7fb4140b71b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb428109280 0x7fb42819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 --2- 172.21.15.16:0/3589549205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb428103940 0x7fb42819a2a0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 -- 172.21.15.16:0/3589549205 >> 172.21.15.16:0/3589549205 conn(0x7fb4280ff3d0 msgr2=0x7fb4281017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 -- 172.21.15.16:0/3589549205 shutdown_connections 2024-08-28T19:32:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:12.662+0000 7fb42de7e640 1 -- 172.21.15.16:0/3589549205 wait complete. 2024-08-28T19:32:12.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462516 3.79 MB/sec execute 472 sec latency 4584.721 ms 2024-08-28T19:32:13.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:32:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:13 smithi028 ceph-mon[21769]: pgmap v684: 129 pgs: 129 active+clean; 338 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-08-28T19:32:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:13 smithi115 ceph-mon[22184]: pgmap v684: 129 pgs: 129 active+clean; 338 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-08-28T19:32:13.735 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:13 smithi016 ceph-mon[18556]: pgmap v684: 129 pgs: 129 active+clean; 338 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-08-28T19:32:13.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 463288 3.79 MB/sec execute 473 sec latency 5143.413 ms 2024-08-28T19:32:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.355+0000 7f2597921640 1 -- 172.21.15.16:0/3017748071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590109b90 msgr2=0x7f25901103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.355+0000 7f2597921640 1 --2- 172.21.15.16:0/3017748071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590109b90 0x7f25901103c0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f2588009ca0 tx=0x7f258802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.355+0000 7f2597921640 1 -- 172.21.15.16:0/3017748071 shutdown_connections 2024-08-28T19:32:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.355+0000 7f2597921640 1 --2- 172.21.15.16:0/3017748071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590109b90 0x7f25901103c0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.355+0000 7f2597921640 1 --2- 172.21.15.16:0/3017748071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2590109240 0x7f2590109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.355+0000 7f2597921640 1 --2- 172.21.15.16:0/3017748071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2590103900 0x7f2590103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.355+0000 7f2597921640 1 -- 172.21.15.16:0/3017748071 >> 172.21.15.16:0/3017748071 conn(0x7f259007b5f0 msgr2=0x7f259007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.355+0000 7f2597921640 1 -- 172.21.15.16:0/3017748071 shutdown_connections 2024-08-28T19:32:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.355+0000 7f2597921640 1 -- 172.21.15.16:0/3017748071 wait complete. 2024-08-28T19:32:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f2597921640 1 Processor -- start 2024-08-28T19:32:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f2597921640 1 -- start start 2024-08-28T19:32:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f2597921640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590103900 0x7f259019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f2597921640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2590109240 0x7f259019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f2597921640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2590109b90 0x7f259019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f2597921640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2590113850 con 0x7f2590103900 2024-08-28T19:32:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f2597921640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25901136d0 con 0x7f2590109240 2024-08-28T19:32:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f2597921640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f25901139d0 con 0x7f2590109b90 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f25956bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590103900 0x7f259019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f25956bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590103900 0x7f259019a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51392/0 (socket says 172.21.15.16:51392) 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f25956bd640 1 -- 172.21.15.16:0/2104929900 learned_addr learned my addr 172.21.15.16:0/2104929900 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.356+0000 7f2594ebc640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2590109240 0x7f259019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f2595ebe640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2590109b90 0x7f259019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f25956bd640 1 -- 172.21.15.16:0/2104929900 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2590109b90 msgr2=0x7f259019fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f25956bd640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2590109b90 0x7f259019fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f25956bd640 1 -- 172.21.15.16:0/2104929900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2590109240 msgr2=0x7f259019a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f25956bd640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2590109240 0x7f259019a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f25956bd640 1 -- 172.21.15.16:0/2104929900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25901a0190 con 0x7f2590103900 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f2594ebc640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2590109240 0x7f259019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f25956bd640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590103900 0x7f259019a220 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e9f0 tx=0x7f258c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f25867fc640 1 -- 172.21.15.16:0/2104929900 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c00cd80 con 0x7f2590103900 2024-08-28T19:32:14.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f25867fc640 1 -- 172.21.15.16:0/2104929900 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f258c0054d0 con 0x7f2590103900 2024-08-28T19:32:14.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f25867fc640 1 -- 172.21.15.16:0/2104929900 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c010690 con 0x7f2590103900 2024-08-28T19:32:14.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f2597921640 1 -- 172.21.15.16:0/2104929900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25901a0480 con 0x7f2590103900 2024-08-28T19:32:14.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.357+0000 7f2597921640 1 -- 172.21.15.16:0/2104929900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25901a6be0 con 0x7f2590103900 2024-08-28T19:32:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.360+0000 7f259de96740 1 Processor -- start 2024-08-28T19:32:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.360+0000 7f259de96740 1 -- start start 2024-08-28T19:32:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.360+0000 7f25867fc640 1 -- 172.21.15.16:0/2104929900 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f258c004e90 con 0x7f2590103900 2024-08-28T19:32:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.360+0000 7f259de96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d229350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f259de96740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d229890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f259de96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 0x55cf6d229dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f259de96740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cf6d1a0680 con 0x55cf6ce84070 2024-08-28T19:32:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f259de96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf6d18fc10 con 0x55cf6ce63390 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f259de96740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cf6d193bc0 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f2595ebe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d229350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f25956bd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d229890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f25956bd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d229890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:54730/0 (socket says 172.21.15.16:54730) 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f25956bd640 1 -- 172.21.15.16:0/3859546338 learned_addr learned my addr 172.21.15.16:0/3859546338 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f2595ebe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d229350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51396/0 (socket says 172.21.15.16:51396) 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f2594ebc640 1 --2- 172.21.15.16:0/3859546338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 0x55cf6d229dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f25956bd640 1 -- 172.21.15.16:0/3859546338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 msgr2=0x55cf6d229dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f25956bd640 1 --2- 172.21.15.16:0/3859546338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 0x55cf6d229dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f25956bd640 1 -- 172.21.15.16:0/3859546338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 msgr2=0x55cf6d229350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f25956bd640 1 --2- 172.21.15.16:0/3859546338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d229350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f25956bd640 1 -- 172.21.15.16:0/3859546338 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf6ceef650 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f2595ebe640 1 --2- 172.21.15.16:0/3859546338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d229350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f2594ebc640 1 --2- 172.21.15.16:0/3859546338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 0x55cf6d229dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:14.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f25956bd640 1 --2- 172.21.15.16:0/3859546338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d229890 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f258c0e3050 tx=0x7f258c0e9e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0099028878bcb17 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:14.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.361+0000 7f25867fc640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f257c0b4ca0 0x7f257c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.362+0000 7f2595ebe640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f257c0b4ca0 0x7f257c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.362+0000 7f257b7fe640 1 -- 172.21.15.16:0/3859546338 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c0a7f40 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.362+0000 7f25867fc640 1 -- 172.21.15.16:0/2104929900 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f258c0e5d60 con 0x7f2590103900 2024-08-28T19:32:14.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.362+0000 7f257b7fe640 1 -- 172.21.15.16:0/3859546338 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f258c0abb90 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.363+0000 7f257b7fe640 1 -- 172.21.15.16:0/3859546338 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c0aab90 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.363+0000 7f259de96740 1 -- 172.21.15.16:0/3859546338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 msgr2=0x55cf6d229890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.363+0000 7f2595ebe640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f257c0b4ca0 0x7f257c0b71b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f258802f500 tx=0x7f258802f490 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.363+0000 7f259de96740 1 --2- 172.21.15.16:0/3859546338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d229890 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f258c0e3050 tx=0x7f258c0e9e20 comp rx=0 tx=0).stop 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.363+0000 7f259de96740 1 -- 172.21.15.16:0/3859546338 shutdown_connections 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.363+0000 7f259de96740 1 --2- 172.21.15.16:0/3859546338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 0x55cf6d229dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.363+0000 7f259de96740 1 --2- 172.21.15.16:0/3859546338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d229890 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.363+0000 7f259de96740 1 --2- 172.21.15.16:0/3859546338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d229350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.363+0000 7f259de96740 1 -- 172.21.15.16:0/3859546338 >> 172.21.15.16:0/3859546338 conn(0x55cf6ce78460 msgr2=0x55cf6d227240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.363+0000 7f259de96740 1 -- 172.21.15.16:0/3859546338 shutdown_connections 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.363+0000 7f259de96740 1 -- 172.21.15.16:0/3859546338 wait complete. 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f259de96740 1 -- start start 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f259de96740 1 Processor -- start 2024-08-28T19:32:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f259de96740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d2349f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f259de96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce63390 0x55cf6d234f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f259de96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce84070 0x55cf6d2354b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f259de96740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cf6cff2700 con 0x55cf6ce63390 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f259de96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf6d116750 con 0x55cf6ce84070 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f259de96740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cf6d237bd0 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f259de96740 10 client.0 _ll_register_callbacks cb 0x55cf6d145e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f2594ebc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce63390 0x55cf6d234f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f25956bd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d2349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f25956bd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d2349f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:54732/0 (socket says 172.21.15.16:54732) 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f25956bd640 1 -- 172.21.15.16:0/2590526787 learned_addr learned my addr 172.21.15.16:0/2590526787 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.364+0000 7f2595ebe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce84070 0x55cf6d2354b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f25956bd640 1 -- 172.21.15.16:0/2590526787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce84070 msgr2=0x55cf6d2354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f25956bd640 1 --2- 172.21.15.16:0/2590526787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce84070 0x55cf6d2354b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f25956bd640 1 -- 172.21.15.16:0/2590526787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce63390 msgr2=0x55cf6d234f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f25956bd640 1 --2- 172.21.15.16:0/2590526787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce63390 0x55cf6d234f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f25956bd640 1 -- 172.21.15.16:0/2590526787 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf6d14f900 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f25956bd640 1 --2- 172.21.15.16:0/2590526787 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d2349f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f258c0e6110 tx=0x7f258c009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7cd4dd9103310a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f2578ff9640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f2578ff9640 1 -- 172.21.15.16:0/2590526787 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c0ab7f0 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f2578ff9640 1 -- 172.21.15.16:0/2590526787 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f258c069240 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f259de96740 1 -- 172.21.15.16:0/2590526787 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f258c0abb90 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f2578ff9640 1 -- 172.21.15.16:0/2590526787 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f258c0ab9d0 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f259de96740 10 client.27547 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.365+0000 7f259de96740 1 -- 172.21.15.16:0/2590526787 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55cf6d243920 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.366+0000 7f2578ff9640 1 -- 172.21.15.16:0/2590526787 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f258c0abb90 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.366+0000 7f2578ff9640 1 -- 172.21.15.16:0/2590526787 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7f258c0a8310 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.366+0000 7f259de96740 10 client.27547 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.366+0000 7f259de96740 10 client.27547 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.366+0000 7f259de96740 20 client.27547 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.366+0000 7f259de96740 1 --2- 172.21.15.16:0/2590526787 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55cf6ce429f0 0x55cf6d246c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.366+0000 7f259de96740 4 client.27547 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.366+0000 7f259de96740 1 -- 172.21.15.16:0/2590526787 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf6d1a25d0 con 0x55cf6ce429f0 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.366+0000 7f2594ebc640 1 --2- 172.21.15.16:0/2590526787 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55cf6ce429f0 0x55cf6d246c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.367+0000 7f2578ff9640 10 client.27547 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.367+0000 7f2594ebc640 1 --2- 172.21.15.16:0/2590526787 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55cf6ce429f0 0x55cf6d246c80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aa3a42af84f9d108 server_cookie=1ed0eadcd0935c25 in_seq=0 out_seq=0 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.367+0000 7f2578ff9640 1 -- 172.21.15.16:0/2590526787 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cf6d1a25d0 con 0x55cf6ce429f0 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.367+0000 7f2578ff9640 10 client.27547 handle_command_reply: tid=0 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.367+0000 7f259de96740 1 client.27547 shutdown 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f259de96740 1 -- 172.21.15.16:0/2590526787 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 msgr2=0x55cf6d2349f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f259de96740 1 --2- 172.21.15.16:0/2590526787 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d2349f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f258c0e6110 tx=0x7f258c009720 comp rx=0 tx=0).stop 2024-08-28T19:32:14.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f259de96740 1 -- 172.21.15.16:0/2590526787 shutdown_connections 2024-08-28T19:32:14.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f259de96740 1 --2- 172.21.15.16:0/2590526787 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55cf6ce429f0 0x55cf6d246c80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f259de96740 1 --2- 172.21.15.16:0/2590526787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce84070 0x55cf6d2354b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f259de96740 1 --2- 172.21.15.16:0/2590526787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce63390 0x55cf6d234f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f259de96740 1 --2- 172.21.15.16:0/2590526787 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d2349f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f2578ff9640 0 client.27547 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:32:14.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f259de96740 1 -- 172.21.15.16:0/2590526787 >> 172.21.15.16:0/2590526787 conn(0x55cf6ce78460 msgr2=0x55cf6d22c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:14.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f259de96740 1 -- 172.21.15.16:0/2590526787 shutdown_connections 2024-08-28T19:32:14.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f259de96740 1 -- 172.21.15.16:0/2590526787 wait complete. 2024-08-28T19:32:14.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.368+0000 7f259de96740 20 client.27547 trim_cache size 0 max 16384 2024-08-28T19:32:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f259de96740 1 Processor -- start 2024-08-28T19:32:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f259de96740 1 -- start start 2024-08-28T19:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f259de96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d231be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f259de96740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce63390 0x55cf6d232120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f259de96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce84070 0x55cf6d235f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f259de96740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25901ae970 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f259de96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25901b05a0 con 0x55cf6ce84070 2024-08-28T19:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f259de96740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f25901af100 con 0x55cf6ce63390 2024-08-28T19:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f25956bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d231be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f25956bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d231be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51410/0 (socket says 172.21.15.16:51410) 2024-08-28T19:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f25956bd640 1 -- 172.21.15.16:0/2174042185 learned_addr learned my addr 172.21.15.16:0/2174042185 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f2595ebe640 1 --2- 172.21.15.16:0/2174042185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce84070 0x55cf6d235f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f2594ebc640 1 --2- 172.21.15.16:0/2174042185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce63390 0x55cf6d232120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f25956bd640 1 -- 172.21.15.16:0/2174042185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce63390 msgr2=0x55cf6d232120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:14.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f25956bd640 1 --2- 172.21.15.16:0/2174042185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce63390 0x55cf6d232120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f25956bd640 1 -- 172.21.15.16:0/2174042185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce84070 msgr2=0x55cf6d235f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:14.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f25956bd640 1 --2- 172.21.15.16:0/2174042185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce84070 0x55cf6d235f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f25956bd640 1 -- 172.21.15.16:0/2174042185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf6d2225e0 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.382+0000 7f2594ebc640 1 --2- 172.21.15.16:0/2174042185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce63390 0x55cf6d232120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:32:14.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.383+0000 7f25956bd640 1 --2- 172.21.15.16:0/2174042185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d231be0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e9c0 tx=0x7f258c0e9e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc2cdbd159883f1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.383+0000 7f257affd640 1 -- 172.21.15.16:0/2174042185 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c0abe50 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.383+0000 7f257affd640 1 -- 172.21.15.16:0/2174042185 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f258c0e9680 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.383+0000 7f257affd640 1 -- 172.21.15.16:0/2174042185 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c0ab7f0 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.384+0000 7f259de96740 1 -- 172.21.15.16:0/2174042185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce5b1b0 msgr2=0x55cf6d231be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.384+0000 7f259de96740 1 --2- 172.21.15.16:0/2174042185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d231be0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e9c0 tx=0x7f258c0e9e20 comp rx=0 tx=0).stop 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.384+0000 7f259de96740 1 -- 172.21.15.16:0/2174042185 shutdown_connections 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.384+0000 7f259de96740 1 --2- 172.21.15.16:0/2174042185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce84070 0x55cf6d235f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.384+0000 7f259de96740 1 --2- 172.21.15.16:0/2174042185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce63390 0x55cf6d232120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.384+0000 7f259de96740 1 --2- 172.21.15.16:0/2174042185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d231be0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.384+0000 7f259de96740 1 -- 172.21.15.16:0/2174042185 >> 172.21.15.16:0/2174042185 conn(0x55cf6ce78460 msgr2=0x55cf6d22fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.384+0000 7f259de96740 1 -- 172.21.15.16:0/2174042185 shutdown_connections 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.384+0000 7f259de96740 1 -- 172.21.15.16:0/2174042185 wait complete. 2024-08-28T19:32:14.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.384+0000 7f259de96740 1 -- start start 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.384+0000 7f259de96740 1 Processor -- start 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f259de96740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d23d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f259de96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 0x55cf6d238a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f259de96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d238f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f259de96740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25901afed0 con 0x55cf6ce84070 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f259de96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25901af7d0 con 0x55cf6ce63390 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f259de96740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f25901ac970 con 0x55cf6ce5b1b0 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f259de96740 10 client.0 _ll_register_callbacks cb 0x7f258c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f2595ebe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d238f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f2594ebc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 0x55cf6d238a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f2595ebe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d238f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51424/0 (socket says 172.21.15.16:51424) 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f2595ebe640 1 -- 172.21.15.16:0/276782942 learned_addr learned my addr 172.21.15.16:0/276782942 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f25956bd640 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d23d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f2594ebc640 1 -- 172.21.15.16:0/276782942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 msgr2=0x55cf6d23d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f2594ebc640 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d23d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f2594ebc640 1 -- 172.21.15.16:0/276782942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 msgr2=0x55cf6d238f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f2594ebc640 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d238f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f2594ebc640 1 -- 172.21.15.16:0/276782942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf6d239580 con 0x55cf6ce63390 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.385+0000 7f2595ebe640 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d238f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f2594ebc640 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 0x55cf6d238a00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55cf6d247360 tx=0x7f2580012600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1cbd37eb227a1a1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f2578ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f2578ff9640 1 -- 172.21.15.16:0/276782942 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258002a060 con 0x55cf6ce63390 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f259de96740 1 -- 172.21.15.16:0/276782942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f258c0e9680 con 0x55cf6ce63390 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f2578ff9640 1 -- 172.21.15.16:0/276782942 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f2580004460 con 0x55cf6ce63390 2024-08-28T19:32:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f2578ff9640 1 -- 172.21.15.16:0/276782942 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f2580012e60 con 0x55cf6ce63390 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f259de96740 10 client.27824 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f259de96740 1 -- 172.21.15.16:0/276782942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55cf6d2b1390 con 0x55cf6ce63390 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f2578ff9640 1 -- 172.21.15.16:0/276782942 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f258c0e9680 con 0x55cf6ce63390 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f2578ff9640 1 -- 172.21.15.16:0/276782942 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7f25800035f0 con 0x55cf6ce63390 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f259de96740 10 client.27824 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f259de96740 10 client.27824 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f259de96740 20 client.27824 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f259de96740 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55cf6ce17da0 0x55cf6d2b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f259de96740 4 client.27824 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.386+0000 7f259de96740 1 -- 172.21.15.16:0/276782942 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf6d19e070 con 0x55cf6ce17da0 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f25956bd640 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55cf6ce17da0 0x55cf6d2b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f2578ff9640 10 client.27824 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:32:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f25956bd640 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55cf6ce17da0 0x55cf6d2b3ad0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=32fb0ddcb4aad07a server_cookie=88350342800156a in_seq=0 out_seq=0 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f2578ff9640 1 -- 172.21.15.16:0/276782942 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf6d19e070 con 0x55cf6ce17da0 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f2578ff9640 10 client.27824 handle_command_reply: tid=0 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f259de96740 1 client.27824 shutdown 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f259de96740 1 -- 172.21.15.16:0/276782942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 msgr2=0x55cf6d238a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f259de96740 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 0x55cf6d238a00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55cf6d247360 tx=0x7f2580012600 comp rx=0 tx=0).stop 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f259de96740 1 -- 172.21.15.16:0/276782942 shutdown_connections 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f259de96740 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55cf6ce17da0 0x55cf6d2b3ad0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f259de96740 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf6ce84070 0x55cf6d238f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f259de96740 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf6ce63390 0x55cf6d238a00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f259de96740 1 --2- 172.21.15.16:0/276782942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cf6ce5b1b0 0x55cf6d23d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f2578ff9640 0 client.27824 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.387+0000 7f259de96740 1 -- 172.21.15.16:0/276782942 >> 172.21.15.16:0/276782942 conn(0x55cf6ce78460 msgr2=0x55cf6d22a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:14.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f259de96740 1 -- 172.21.15.16:0/276782942 shutdown_connections 2024-08-28T19:32:14.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f259de96740 1 -- 172.21.15.16:0/276782942 wait complete. 2024-08-28T19:32:14.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f259de96740 20 client.27824 trim_cache size 0 max 16384 2024-08-28T19:32:14.394 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 -- 172.21.15.16:0/2104929900 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f257c0b4ca0 msgr2=0x7f257c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f257c0b4ca0 0x7f257c0b71b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f258802f500 tx=0x7f258802f490 comp rx=0 tx=0).stop 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 -- 172.21.15.16:0/2104929900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590103900 msgr2=0x7f259019a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590103900 0x7f259019a220 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e9f0 tx=0x7f258c00edf0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 -- 172.21.15.16:0/2104929900 shutdown_connections 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2590109b90 0x7f259019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f257c0b4ca0 0x7f257c0b71b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2590109240 0x7f259019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 --2- 172.21.15.16:0/2104929900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590103900 0x7f259019a220 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 -- 172.21.15.16:0/2104929900 >> 172.21.15.16:0/2104929900 conn(0x7f259007b5f0 msgr2=0x7f2590105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 -- 172.21.15.16:0/2104929900 shutdown_connections 2024-08-28T19:32:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:14.388+0000 7f2597921640 1 -- 172.21.15.16:0/2104929900 wait complete. 2024-08-28T19:32:14.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 463595 3.78 MB/sec execute 474 sec latency 619.226 ms 2024-08-28T19:32:15.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:32:15.433 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:15.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:15 smithi016 ceph-mon[18556]: pgmap v685: 129 pgs: 129 active+clean; 343 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-08-28T19:32:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:15 smithi028 ceph-mon[21769]: pgmap v685: 129 pgs: 129 active+clean; 343 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-08-28T19:32:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:15 smithi115 ceph-mon[22184]: pgmap v685: 129 pgs: 129 active+clean; 343 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-08-28T19:32:15.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.951+0000 7f42a75c6640 1 -- 172.21.15.16:0/2168268909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0108150 msgr2=0x7f42a0108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.951+0000 7f42a75c6640 1 --2- 172.21.15.16:0/2168268909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0108150 0x7f42a0108560 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f4290009900 tx=0x7f429002a410 comp rx=0 tx=0).stop 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.951+0000 7f42a75c6640 1 -- 172.21.15.16:0/2168268909 shutdown_connections 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.951+0000 7f42a75c6640 1 --2- 172.21.15.16:0/2168268909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42a0108aa0 0x7f42a010f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.951+0000 7f42a75c6640 1 --2- 172.21.15.16:0/2168268909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0108150 0x7f42a0108560 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.951+0000 7f42a75c6640 1 --2- 172.21.15.16:0/2168268909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42a0102810 0x7f42a0102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.951+0000 7f42a75c6640 1 -- 172.21.15.16:0/2168268909 >> 172.21.15.16:0/2168268909 conn(0x7f42a00794b0 msgr2=0x7f42a00798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.951+0000 7f42a75c6640 1 -- 172.21.15.16:0/2168268909 shutdown_connections 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.951+0000 7f42a75c6640 1 -- 172.21.15.16:0/2168268909 wait complete. 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.952+0000 7f42a75c6640 1 Processor -- start 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.952+0000 7f42a75c6640 1 -- start start 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.952+0000 7f42a75c6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42a0102810 0x7f42a019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.952+0000 7f42a75c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0108150 0x7f42a019a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.952+0000 7f42a75c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42a0108aa0 0x7f42a019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.952+0000 7f42a75c6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42a0112730 con 0x7f42a0108150 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.952+0000 7f42a75c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42a01125b0 con 0x7f42a0108aa0 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.952+0000 7f42a75c6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f42a01128b0 con 0x7f42a0102810 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a4b61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0108150 0x7f42a019a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a4b61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0108150 0x7f42a019a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51452/0 (socket says 172.21.15.16:51452) 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a4b61640 1 -- 172.21.15.16:0/897701241 learned_addr learned my addr 172.21.15.16:0/897701241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a5362640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42a0102810 0x7f42a019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a5b63640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42a0108aa0 0x7f42a019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a4b61640 1 -- 172.21.15.16:0/897701241 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42a0102810 msgr2=0x7f42a019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a4b61640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42a0102810 0x7f42a019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a4b61640 1 -- 172.21.15.16:0/897701241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42a0108aa0 msgr2=0x7f42a019fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a4b61640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42a0108aa0 0x7f42a019fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a4b61640 1 -- 172.21.15.16:0/897701241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a01a0260 con 0x7f42a0108150 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a5b63640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42a0108aa0 0x7f42a019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a4b61640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0108150 0x7f42a019a890 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f4290006010 tx=0x7f4290004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42967fc640 1 -- 172.21.15.16:0/897701241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429003f030 con 0x7f42a0108150 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42967fc640 1 -- 172.21.15.16:0/897701241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4290003aa0 con 0x7f42a0108150 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a75c6640 1 -- 172.21.15.16:0/897701241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a01a04f0 con 0x7f42a0108150 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42967fc640 1 -- 172.21.15.16:0/897701241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4290035070 con 0x7f42a0108150 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.953+0000 7f42a75c6640 1 -- 172.21.15.16:0/897701241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a01a6cb0 con 0x7f42a0108150 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.956+0000 7f42967fc640 1 -- 172.21.15.16:0/897701241 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4290042030 con 0x7f42a0108150 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.956+0000 7f42adb3b740 1 Processor -- start 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.956+0000 7f42adb3b740 1 -- start start 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.956+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf46310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf46850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb80390 0x55c5cbf46d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42adb3b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c5cbebd680 con 0x55c5cbba1070 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42adb3b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c5cbeacc10 con 0x55c5cbb781b0 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42adb3b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c5cbeb0bc0 con 0x55c5cbb80390 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5b63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf46310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5b63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf46310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51468/0 (socket says 172.21.15.16:51468) 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5b63640 1 -- 172.21.15.16:0/2421456612 learned_addr learned my addr 172.21.15.16:0/2421456612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5362640 1 --2- 172.21.15.16:0/2421456612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf46850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a4b61640 1 --2- 172.21.15.16:0/2421456612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb80390 0x55c5cbf46d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42967fc640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f428c0b4ca0 0x7f428c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5b63640 1 -- 172.21.15.16:0/2421456612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb80390 msgr2=0x55c5cbf46d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5b63640 1 --2- 172.21.15.16:0/2421456612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb80390 0x55c5cbf46d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5b63640 1 -- 172.21.15.16:0/2421456612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 msgr2=0x55c5cbf46850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5b63640 1 --2- 172.21.15.16:0/2421456612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf46850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5b63640 1 -- 172.21.15.16:0/2421456612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5cbc0c650 con 0x55c5cbba1070 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5362640 1 --2- 172.21.15.16:0/2421456612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf46850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:15.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42967fc640 1 -- 172.21.15.16:0/897701241 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f4290105e20 con 0x7f42a0108150 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5b63640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f428c0b4ca0 0x7f428c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.957+0000 7f42a5b63640 1 --2- 172.21.15.16:0/2421456612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf46310 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f429800f3e0 tx=0x7f429800f7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2080d48f534d8f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.958+0000 7f428b7fe640 1 -- 172.21.15.16:0/2421456612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429801c400 con 0x55c5cbba1070 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.958+0000 7f428b7fe640 1 -- 172.21.15.16:0/2421456612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f429801ca50 con 0x55c5cbba1070 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.958+0000 7f428b7fe640 1 -- 172.21.15.16:0/2421456612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4298010630 con 0x55c5cbba1070 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.958+0000 7f42a5b63640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f428c0b4ca0 0x7f428c0b71b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4298020000 tx=0x7f429801d000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.959+0000 7f42adb3b740 1 -- 172.21.15.16:0/2421456612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 msgr2=0x55c5cbf46310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.959+0000 7f42adb3b740 1 --2- 172.21.15.16:0/2421456612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf46310 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f429800f3e0 tx=0x7f429800f7e0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.959+0000 7f42adb3b740 1 -- 172.21.15.16:0/2421456612 shutdown_connections 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.959+0000 7f42adb3b740 1 --2- 172.21.15.16:0/2421456612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb80390 0x55c5cbf46d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.959+0000 7f42adb3b740 1 --2- 172.21.15.16:0/2421456612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf46850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.959+0000 7f42adb3b740 1 --2- 172.21.15.16:0/2421456612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf46310 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.959+0000 7f42adb3b740 1 -- 172.21.15.16:0/2421456612 >> 172.21.15.16:0/2421456612 conn(0x55c5cbb95460 msgr2=0x55c5cbf44200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.959+0000 7f42adb3b740 1 -- 172.21.15.16:0/2421456612 shutdown_connections 2024-08-28T19:32:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.959+0000 7f42adb3b740 1 -- 172.21.15.16:0/2421456612 wait complete. 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.959+0000 7f42adb3b740 1 -- start start 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.959+0000 7f42adb3b740 1 Processor -- start 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf51a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb80390 0x55c5cbf51fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf5de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42adb3b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c5cbd0f700 con 0x55c5cbba1070 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42adb3b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c5cbe33750 con 0x55c5cbb781b0 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42adb3b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c5cbf54af0 con 0x55c5cbb80390 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42adb3b740 10 client.0 _ll_register_callbacks cb 0x55c5cbe62e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a5b63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf5de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a5b63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf5de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51472/0 (socket says 172.21.15.16:51472) 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a5b63640 1 -- 172.21.15.16:0/1497144789 learned_addr learned my addr 172.21.15.16:0/1497144789 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a4b61640 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb80390 0x55c5cbf51fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a5362640 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf51a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a5b63640 1 -- 172.21.15.16:0/1497144789 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb80390 msgr2=0x55c5cbf51fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a5b63640 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb80390 0x55c5cbf51fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a5b63640 1 -- 172.21.15.16:0/1497144789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 msgr2=0x55c5cbf51a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a5b63640 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf51a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a5b63640 1 -- 172.21.15.16:0/1497144789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5cbe6c900 con 0x55c5cbba1070 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a5362640 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf51a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42a5b63640 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf5de70 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f42980092f0 tx=0x7f429801ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25d64f539ac4826 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f4288ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f4288ff9640 1 -- 172.21.15.16:0/1497144789 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4298033060 con 0x55c5cbba1070 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f4288ff9640 1 -- 172.21.15.16:0/1497144789 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f4298010a10 con 0x55c5cbba1070 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.960+0000 7f42adb3b740 1 -- 172.21.15.16:0/1497144789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f429801ca50 con 0x55c5cbba1070 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f4288ff9640 1 -- 172.21.15.16:0/1497144789 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f42980105e0 con 0x55c5cbba1070 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f42adb3b740 10 client.18375 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:15.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f42adb3b740 1 -- 172.21.15.16:0/1497144789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55c5cbf60910 con 0x55c5cbba1070 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f4288ff9640 1 -- 172.21.15.16:0/1497144789 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f429801ca50 con 0x55c5cbba1070 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f4288ff9640 1 -- 172.21.15.16:0/1497144789 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7f4298003a00 con 0x55c5cbba1070 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f42adb3b740 10 client.18375 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f42adb3b740 10 client.18375 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f42adb3b740 20 client.18375 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f42adb3b740 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c5cbb5f9f0 0x55c5cbf63d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f42adb3b740 4 client.18375 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f42adb3b740 1 -- 172.21.15.16:0/1497144789 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5cbebf5d0 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.961+0000 7f42a5362640 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c5cbb5f9f0 0x55c5cbf63d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.962+0000 7f4288ff9640 10 client.18375 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.962+0000 7f42a5362640 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c5cbb5f9f0 0x55c5cbf63d80 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4aa1f8a6751066c8 server_cookie=f2c5090b00990956 in_seq=0 out_seq=0 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f4288ff9640 1 -- 172.21.15.16:0/1497144789 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c5cbebf5d0 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f4288ff9640 10 client.18375 handle_command_reply: tid=0 2024-08-28T19:32:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 1 client.18375 shutdown 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 1 -- 172.21.15.16:0/1497144789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 msgr2=0x55c5cbf5de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf5de70 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f42980092f0 tx=0x7f429801ce30 comp rx=0 tx=0).stop 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 1 -- 172.21.15.16:0/1497144789 shutdown_connections 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbba1070 0x55c5cbf5de70 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c5cbb5f9f0 0x55c5cbf63d80 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb80390 0x55c5cbf51fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f4288ff9640 0 client.18375 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 1 --2- 172.21.15.16:0/1497144789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf51a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 1 -- 172.21.15.16:0/1497144789 >> 172.21.15.16:0/1497144789 conn(0x55c5cbb95460 msgr2=0x55c5cbf494a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 1 -- 172.21.15.16:0/1497144789 shutdown_connections 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 1 -- 172.21.15.16:0/1497144789 wait complete. 2024-08-28T19:32:15.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.963+0000 7f42adb3b740 20 client.18375 trim_cache size 0 max 16384 2024-08-28T19:32:15.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42adb3b740 1 Processor -- start 2024-08-28T19:32:15.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42adb3b740 1 -- start start 2024-08-28T19:32:15.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb5f9f0 0x55c5cbf49b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb781b0 0x55c5cbf4ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbf4f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42adb3b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42a01b0d80 con 0x55c5cbb80390 2024-08-28T19:32:15.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42adb3b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42a01b29b0 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42adb3b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f42a01b1510 con 0x55c5cbb781b0 2024-08-28T19:32:15.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42a5b63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbf4f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42a5b63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbf4f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51480/0 (socket says 172.21.15.16:51480) 2024-08-28T19:32:15.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42a5b63640 1 -- 172.21.15.16:0/2429508619 learned_addr learned my addr 172.21.15.16:0/2429508619 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:15.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42a5362640 1 --2- 172.21.15.16:0/2429508619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb5f9f0 0x55c5cbf49b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42a4b61640 1 --2- 172.21.15.16:0/2429508619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb781b0 0x55c5cbf4ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42a5b63640 1 -- 172.21.15.16:0/2429508619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb781b0 msgr2=0x55c5cbf4ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42a5b63640 1 --2- 172.21.15.16:0/2429508619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb781b0 0x55c5cbf4ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42a5b63640 1 -- 172.21.15.16:0/2429508619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb5f9f0 msgr2=0x55c5cbf49b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42a5b63640 1 --2- 172.21.15.16:0/2429508619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb5f9f0 0x55c5cbf49b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.977+0000 7f42a5b63640 1 -- 172.21.15.16:0/2429508619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5cbf44dc0 con 0x55c5cbb80390 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.978+0000 7f42a5b63640 1 --2- 172.21.15.16:0/2429508619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbf4f1a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f42980107d0 tx=0x7f429800f7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5663c2120526ae6a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.978+0000 7f428affd640 1 -- 172.21.15.16:0/2429508619 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4298043070 con 0x55c5cbb80390 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.978+0000 7f428affd640 1 -- 172.21.15.16:0/2429508619 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f429802a040 con 0x55c5cbb80390 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.978+0000 7f428affd640 1 -- 172.21.15.16:0/2429508619 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4298033070 con 0x55c5cbb80390 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.979+0000 7f42adb3b740 1 -- 172.21.15.16:0/2429508619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 msgr2=0x55c5cbf4f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.979+0000 7f42adb3b740 1 --2- 172.21.15.16:0/2429508619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbf4f1a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f42980107d0 tx=0x7f429800f7e0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.979+0000 7f42adb3b740 1 -- 172.21.15.16:0/2429508619 shutdown_connections 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.979+0000 7f42adb3b740 1 --2- 172.21.15.16:0/2429508619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbf4f1a0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.979+0000 7f42adb3b740 1 --2- 172.21.15.16:0/2429508619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb781b0 0x55c5cbf4ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.979+0000 7f42adb3b740 1 --2- 172.21.15.16:0/2429508619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb5f9f0 0x55c5cbf49b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.979+0000 7f42adb3b740 1 -- 172.21.15.16:0/2429508619 >> 172.21.15.16:0/2429508619 conn(0x55c5cbb95460 msgr2=0x55c5cbf49410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:15.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.979+0000 7f42adb3b740 1 -- 172.21.15.16:0/2429508619 shutdown_connections 2024-08-28T19:32:15.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.979+0000 7f42adb3b740 1 -- 172.21.15.16:0/2429508619 wait complete. 2024-08-28T19:32:15.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.979+0000 7f42adb3b740 1 -- start start 2024-08-28T19:32:15.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.979+0000 7f42adb3b740 1 Processor -- start 2024-08-28T19:32:15.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 463595 3.77 MB/sec execute 475 sec latency 1619.259 ms 2024-08-28T19:32:15.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb5f9f0 0x55c5cbf565f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf56bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42adb3b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbfcd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42adb3b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42a01b22e0 con 0x55c5cbb80390 2024-08-28T19:32:15.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42adb3b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42a01b1be0 con 0x55c5cbb781b0 2024-08-28T19:32:15.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42adb3b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f42a01aed80 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42adb3b740 10 client.0 _ll_register_callbacks cb 0x7f429801f020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:15.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42a5b63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbfcd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42a5b63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbfcd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51492/0 (socket says 172.21.15.16:51492) 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42a5b63640 1 -- 172.21.15.16:0/3223427484 learned_addr learned my addr 172.21.15.16:0/3223427484 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42a5362640 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb5f9f0 0x55c5cbf565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42a4b61640 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf56bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42a5362640 1 -- 172.21.15.16:0/3223427484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 msgr2=0x55c5cbf56bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42a5362640 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf56bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42a5362640 1 -- 172.21.15.16:0/3223427484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 msgr2=0x55c5cbfcd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42a5362640 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbfcd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42a5362640 1 -- 172.21.15.16:0/3223427484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5cbfcdc90 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.980+0000 7f42a5b63640 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbfcd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f42a5362640 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb5f9f0 0x55c5cbf565f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55c5cbf64390 tx=0x7f429c011940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b769de06b13a8c5a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f4288ff9640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f4288ff9640 1 -- 172.21.15.16:0/3223427484 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429c004020 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f42adb3b740 1 -- 172.21.15.16:0/3223427484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f429802a040 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f4288ff9640 1 -- 172.21.15.16:0/3223427484 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f429c020070 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f4288ff9640 1 -- 172.21.15.16:0/3223427484 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f429c02f050 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f42adb3b740 10 client.27568 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f42adb3b740 1 -- 172.21.15.16:0/3223427484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55c5cbfcef50 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f4288ff9640 1 -- 172.21.15.16:0/3223427484 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f429802a040 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f4288ff9640 1 -- 172.21.15.16:0/3223427484 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7f429c004830 con 0x55c5cbb5f9f0 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f42adb3b740 10 client.27568 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f42adb3b740 10 client.27568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f42adb3b740 20 client.27568 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f42adb3b740 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c5cbba1070 0x55c5cbfd16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f42adb3b740 4 client.27568 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.981+0000 7f42adb3b740 1 -- 172.21.15.16:0/3223427484 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c5cbebb070 con 0x55c5cbba1070 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.982+0000 7f42a4b61640 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c5cbba1070 0x55c5cbfd16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.982+0000 7f4288ff9640 10 client.27568 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.982+0000 7f42a4b61640 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c5cbba1070 0x55c5cbfd16c0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdf227bbd257805f server_cookie=cd57694b162933cc in_seq=0 out_seq=0 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.982+0000 7f4288ff9640 1 -- 172.21.15.16:0/3223427484 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c5cbebb070 con 0x55c5cbba1070 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.982+0000 7f4288ff9640 10 client.27568 handle_command_reply: tid=0 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.982+0000 7f42adb3b740 1 client.27568 shutdown 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42adb3b740 1 -- 172.21.15.16:0/3223427484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb5f9f0 msgr2=0x55c5cbf565f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42adb3b740 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb5f9f0 0x55c5cbf565f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55c5cbf64390 tx=0x7f429c011940 comp rx=0 tx=0).stop 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42adb3b740 1 -- 172.21.15.16:0/3223427484 shutdown_connections 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42adb3b740 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5cbb80390 0x55c5cbfcd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42adb3b740 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5cbb781b0 0x55c5cbf56bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42adb3b740 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c5cbba1070 0x55c5cbfd16c0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42adb3b740 1 --2- 172.21.15.16:0/3223427484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c5cbb5f9f0 0x55c5cbf565f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f4288ff9640 0 client.27568 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42adb3b740 1 -- 172.21.15.16:0/3223427484 >> 172.21.15.16:0/3223427484 conn(0x55c5cbb95460 msgr2=0x55c5cbf472e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42adb3b740 1 -- 172.21.15.16:0/3223427484 shutdown_connections 2024-08-28T19:32:15.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42adb3b740 1 -- 172.21.15.16:0/3223427484 wait complete. 2024-08-28T19:32:15.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42adb3b740 20 client.27568 trim_cache size 0 max 16384 2024-08-28T19:32:15.990 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42a75c6640 1 -- 172.21.15.16:0/897701241 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f428c0b4ca0 msgr2=0x7f428c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42a75c6640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f428c0b4ca0 0x7f428c0b71b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4298020000 tx=0x7f429801d000 comp rx=0 tx=0).stop 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42a75c6640 1 -- 172.21.15.16:0/897701241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0108150 msgr2=0x7f42a019a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42a75c6640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0108150 0x7f42a019a890 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f4290006010 tx=0x7f4290004060 comp rx=0 tx=0).stop 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42a75c6640 1 -- 172.21.15.16:0/897701241 shutdown_connections 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42a75c6640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f428c0b4ca0 0x7f428c0b71b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42a75c6640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42a0108aa0 0x7f42a019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42a75c6640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0108150 0x7f42a019a890 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42a75c6640 1 --2- 172.21.15.16:0/897701241 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42a0102810 0x7f42a019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.983+0000 7f42a75c6640 1 -- 172.21.15.16:0/897701241 >> 172.21.15.16:0/897701241 conn(0x7f42a00794b0 msgr2=0x7f42a01018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.984+0000 7f42a75c6640 1 -- 172.21.15.16:0/897701241 shutdown_connections 2024-08-28T19:32:15.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:15.984+0000 7f42a75c6640 1 -- 172.21.15.16:0/897701241 wait complete. 2024-08-28T19:32:16.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 463598 3.76 MB/sec execute 476 sec latency 2615.804 ms 2024-08-28T19:32:17.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:17 smithi028 ceph-mon[21769]: pgmap v686: 129 pgs: 129 active+clean; 353 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 2.7 MiB/s wr, 140 op/s 2024-08-28T19:32:17.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:17 smithi115 ceph-mon[22184]: pgmap v686: 129 pgs: 129 active+clean; 353 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 2.7 MiB/s wr, 140 op/s 2024-08-28T19:32:17.720 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:32:17.720 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:32:17.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:17 smithi016 ceph-mon[18556]: pgmap v686: 129 pgs: 129 active+clean; 353 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 2.7 MiB/s wr, 140 op/s 2024-08-28T19:32:17.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 465680 3.79 MB/sec execute 477 sec latency 16.588 ms 2024-08-28T19:32:17.992 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.453+0000 7fe368c61640 1 -- 172.21.15.16:0/2993025223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe364102800 msgr2=0x7fe364102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.453+0000 7fe368c61640 1 --2- 172.21.15.16:0/2993025223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe364102800 0x7fe364102c10 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe354009900 tx=0x7fe35402a410 comp rx=0 tx=0).stop 2024-08-28T19:32:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.453+0000 7fe368c61640 1 -- 172.21.15.16:0/2993025223 shutdown_connections 2024-08-28T19:32:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.453+0000 7fe368c61640 1 --2- 172.21.15.16:0/2993025223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe364108a90 0x7fe36410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.453+0000 7fe368c61640 1 --2- 172.21.15.16:0/2993025223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe364108140 0x7fe364108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.453+0000 7fe368c61640 1 --2- 172.21.15.16:0/2993025223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe364102800 0x7fe364102c10 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.453+0000 7fe368c61640 1 -- 172.21.15.16:0/2993025223 >> 172.21.15.16:0/2993025223 conn(0x7fe3640794d0 msgr2=0x7fe3640798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.453+0000 7fe368c61640 1 -- 172.21.15.16:0/2993025223 shutdown_connections 2024-08-28T19:32:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.454+0000 7fe368c61640 1 -- 172.21.15.16:0/2993025223 wait complete. 2024-08-28T19:32:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.454+0000 7fe368c61640 1 Processor -- start 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.454+0000 7fe368c61640 1 -- start start 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe368c61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe364102800 0x7fe36419b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe368c61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe364108140 0x7fe36419b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe368c61640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe364108a90 0x7fe36419bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe368c61640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3641126f0 con 0x7fe364102800 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe368c61640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe364112570 con 0x7fe364108140 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe368c61640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe364112870 con 0x7fe364108a90 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe36259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe364102800 0x7fe36419b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe361d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe364108140 0x7fe36419b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe36259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe364102800 0x7fe36419b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51508/0 (socket says 172.21.15.16:51508) 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe36259c640 1 -- 172.21.15.16:0/3020914546 learned_addr learned my addr 172.21.15.16:0/3020914546 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe362d9d640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe364108a90 0x7fe36419bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe36259c640 1 -- 172.21.15.16:0/3020914546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe364108a90 msgr2=0x7fe36419bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe36259c640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe364108a90 0x7fe36419bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe36259c640 1 -- 172.21.15.16:0/3020914546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe364108140 msgr2=0x7fe36419b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe36259c640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe364108140 0x7fe36419b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe36259c640 1 -- 172.21.15.16:0/3020914546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3641a6350 con 0x7fe364102800 2024-08-28T19:32:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe361d9b640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe364108140 0x7fe36419b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.455+0000 7fe36259c640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe364102800 0x7fe36419b0c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe354005940 tx=0x7fe3540040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.456+0000 7fe3537fe640 1 -- 172.21.15.16:0/3020914546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35403f030 con 0x7fe364102800 2024-08-28T19:32:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.456+0000 7fe368c61640 1 -- 172.21.15.16:0/3020914546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3641a65e0 con 0x7fe364102800 2024-08-28T19:32:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.456+0000 7fe3537fe640 1 -- 172.21.15.16:0/3020914546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3540044c0 con 0x7fe364102800 2024-08-28T19:32:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.456+0000 7fe3537fe640 1 -- 172.21.15.16:0/3020914546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe354035070 con 0x7fe364102800 2024-08-28T19:32:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.456+0000 7fe368c61640 1 -- 172.21.15.16:0/3020914546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3641a6ac0 con 0x7fe364102800 2024-08-28T19:32:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.458+0000 7fe3537fe640 1 -- 172.21.15.16:0/3020914546 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe354042030 con 0x7fe364102800 2024-08-28T19:32:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe36f1d6740 1 Processor -- start 2024-08-28T19:32:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe36f1d6740 1 -- start start 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 0x557f9832d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 0x557f9832dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f66670 0x557f9832e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557f982a35f0 con 0x557f97f5e490 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f98292b80 con 0x557f97f87350 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557f98296b30 con 0x557f97f66670 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe36259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 0x557f9832dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe362d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 0x557f9832d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe362d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 0x557f9832d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34378/0 (socket says 172.21.15.16:34378) 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe362d9d640 1 -- 172.21.15.16:0/3891525859 learned_addr learned my addr 172.21.15.16:0/3891525859 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.459+0000 7fe361d9b640 1 --2- 172.21.15.16:0/3891525859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f66670 0x557f9832e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe3537fe640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe34c0b4cd0 0x7fe34c0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe362d9d640 1 -- 172.21.15.16:0/3891525859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f66670 msgr2=0x557f9832e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe362d9d640 1 --2- 172.21.15.16:0/3891525859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f66670 0x557f9832e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe362d9d640 1 -- 172.21.15.16:0/3891525859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 msgr2=0x557f9832dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe362d9d640 1 --2- 172.21.15.16:0/3891525859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 0x557f9832dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe362d9d640 1 -- 172.21.15.16:0/3891525859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f97ff2930 con 0x557f97f87350 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe36259c640 1 --2- 172.21.15.16:0/3891525859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 0x557f9832dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe362d9d640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe34c0b4cd0 0x7fe34c0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe3537fe640 1 -- 172.21.15.16:0/3020914546 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fe354105d20 con 0x7fe364102800 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe362d9d640 1 --2- 172.21.15.16:0/3891525859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 0x557f9832d780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe35c011590 tx=0x7fe35c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2938fabc92c79d92 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe350ff9640 1 -- 172.21.15.16:0/3891525859 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c017050 con 0x557f97f87350 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe362d9d640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe34c0b4cd0 0x7fe34c0b71e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe35c023000 tx=0x7fe35c024000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe350ff9640 1 -- 172.21.15.16:0/3891525859 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe35c01bbf0 con 0x557f97f87350 2024-08-28T19:32:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.460+0000 7fe350ff9640 1 -- 172.21.15.16:0/3891525859 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c00eab0 con 0x557f97f87350 2024-08-28T19:32:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.461+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3891525859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 msgr2=0x557f9832d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.461+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/3891525859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 0x557f9832d780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe35c011590 tx=0x7fe35c009f90 comp rx=0 tx=0).stop 2024-08-28T19:32:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.461+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3891525859 shutdown_connections 2024-08-28T19:32:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.461+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/3891525859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f66670 0x557f9832e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.461+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/3891525859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 0x557f9832dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.461+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/3891525859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 0x557f9832d780 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.461+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3891525859 >> 172.21.15.16:0/3891525859 conn(0x557f97f7b740 msgr2=0x557f9832b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.461+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3891525859 shutdown_connections 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3891525859 wait complete. 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36f1d6740 1 -- start start 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36f1d6740 1 Processor -- start 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 0x557f9833d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f66670 0x557f9833d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 0x557f98349640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557f980f5890 con 0x557f97f5e490 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f98219790 con 0x557f97f87350 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557f983402c0 con 0x557f97f66670 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36f1d6740 10 client.0 _ll_register_callbacks cb 0x557f97f9eba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 0x557f9833d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe362d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 0x557f98349640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 0x557f9833d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51516/0 (socket says 172.21.15.16:51516) 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.462+0000 7fe36259c640 1 -- 172.21.15.16:0/1020389464 learned_addr learned my addr 172.21.15.16:0/1020389464 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe361d9b640 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f66670 0x557f9833d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe36259c640 1 -- 172.21.15.16:0/1020389464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f66670 msgr2=0x557f9833d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe36259c640 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f66670 0x557f9833d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe36259c640 1 -- 172.21.15.16:0/1020389464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 msgr2=0x557f98349640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe36259c640 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 0x557f98349640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe36259c640 1 -- 172.21.15.16:0/1020389464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f98250f30 con 0x557f97f5e490 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe362d9d640 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 0x557f98349640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe36259c640 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 0x557f9833d390 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe354006010 tx=0x7fe354009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976157a7920073bd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe349ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe349ffb640 1 -- 172.21.15.16:0/1020389464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe354049070 con 0x557f97f5e490 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe349ffb640 1 -- 172.21.15.16:0/1020389464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fe3540cc440 con 0x557f97f5e490 2024-08-28T19:32:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe36f1d6740 1 -- 172.21.15.16:0/1020389464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe35c01bbf0 con 0x557f97f5e490 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe349ffb640 1 -- 172.21.15.16:0/1020389464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fe3540cb930 con 0x557f97f5e490 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe36f1d6740 10 client.18393 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.463+0000 7fe36f1d6740 1 -- 172.21.15.16:0/1020389464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x557f9834c180 con 0x557f97f5e490 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.464+0000 7fe349ffb640 1 -- 172.21.15.16:0/1020389464 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe35c01bbf0 con 0x557f97f5e490 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.464+0000 7fe349ffb640 1 -- 172.21.15.16:0/1020389464 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7fe3540d4e70 con 0x557f97f5e490 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.464+0000 7fe36f1d6740 10 client.18393 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.464+0000 7fe36f1d6740 10 client.18393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.464+0000 7fe36f1d6740 20 client.18393 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.464+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557f97f45cd0 0x557f9834f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.464+0000 7fe36f1d6740 4 client.18393 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.464+0000 7fe36f1d6740 1 -- 172.21.15.16:0/1020389464 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f982a5540 con 0x557f97f45cd0 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.464+0000 7fe361d9b640 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557f97f45cd0 0x557f9834f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.464+0000 7fe361d9b640 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557f97f45cd0 0x557f9834f630 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d2c0694b618e81e server_cookie=2d323c0e454137b4 in_seq=0 out_seq=0 2024-08-28T19:32:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.464+0000 7fe349ffb640 10 client.18393 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe349ffb640 1 -- 172.21.15.16:0/1020389464 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557f982a5540 con 0x557f97f45cd0 2024-08-28T19:32:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe349ffb640 10 client.18393 handle_command_reply: tid=0 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe36f1d6740 1 client.18393 shutdown 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe36f1d6740 1 -- 172.21.15.16:0/1020389464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 msgr2=0x557f9833d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 0x557f9833d390 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe354006010 tx=0x7fe354009560 comp rx=0 tx=0).stop 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe36f1d6740 1 -- 172.21.15.16:0/1020389464 shutdown_connections 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f87350 0x557f98349640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557f97f45cd0 0x557f9834f630 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f66670 0x557f9833d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe349ffb640 0 client.18393 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/1020389464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f5e490 0x557f9833d390 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe36f1d6740 1 -- 172.21.15.16:0/1020389464 >> 172.21.15.16:0/1020389464 conn(0x557f97f7b740 msgr2=0x557f98329c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe36f1d6740 1 -- 172.21.15.16:0/1020389464 shutdown_connections 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.466+0000 7fe36f1d6740 1 -- 172.21.15.16:0/1020389464 wait complete. 2024-08-28T19:32:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.467+0000 7fe36f1d6740 20 client.18393 trim_cache size 0 max 16384 2024-08-28T19:32:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.485+0000 7fe36f1d6740 1 Processor -- start 2024-08-28T19:32:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.485+0000 7fe36f1d6740 1 -- start start 2024-08-28T19:32:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.485+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f45cd0 0x557f98338b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.485+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f5e490 0x557f9832afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.485+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f9832b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.485+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3641aeb60 con 0x557f97f66670 2024-08-28T19:32:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.485+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3641b0790 con 0x557f97f5e490 2024-08-28T19:32:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.485+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe3641af2f0 con 0x557f97f45cd0 2024-08-28T19:32:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.485+0000 7fe362d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f9832b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe362d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f9832b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51528/0 (socket says 172.21.15.16:51528) 2024-08-28T19:32:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe361d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f5e490 0x557f9832afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe362d9d640 1 -- 172.21.15.16:0/2200867875 learned_addr learned my addr 172.21.15.16:0/2200867875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe36259c640 1 --2- 172.21.15.16:0/2200867875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f45cd0 0x557f98338b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe362d9d640 1 -- 172.21.15.16:0/2200867875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f45cd0 msgr2=0x557f98338b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe362d9d640 1 --2- 172.21.15.16:0/2200867875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f45cd0 0x557f98338b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe362d9d640 1 -- 172.21.15.16:0/2200867875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f5e490 msgr2=0x557f9832afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe362d9d640 1 --2- 172.21.15.16:0/2200867875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f5e490 0x557f9832afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe362d9d640 1 -- 172.21.15.16:0/2200867875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f98326b80 con 0x557f97f66670 2024-08-28T19:32:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe36259c640 1 --2- 172.21.15.16:0/2200867875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f45cd0 0x557f98338b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe361d9b640 1 --2- 172.21.15.16:0/2200867875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f5e490 0x557f9832afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe362d9d640 1 --2- 172.21.15.16:0/2200867875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f9832b510 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe35c011530 tx=0x7fe35c00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04d7b691c902ffa server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe34bfff640 1 -- 172.21.15.16:0/2200867875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c02b070 con 0x557f97f66670 2024-08-28T19:32:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe34bfff640 1 -- 172.21.15.16:0/2200867875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe35c02c040 con 0x557f97f66670 2024-08-28T19:32:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.486+0000 7fe34bfff640 1 -- 172.21.15.16:0/2200867875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c00eb40 con 0x557f97f66670 2024-08-28T19:32:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.487+0000 7fe36f1d6740 1 -- 172.21.15.16:0/2200867875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 msgr2=0x557f9832b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.487+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/2200867875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f9832b510 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe35c011530 tx=0x7fe35c00f3c0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.487+0000 7fe36f1d6740 1 -- 172.21.15.16:0/2200867875 shutdown_connections 2024-08-28T19:32:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.487+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/2200867875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f9832b510 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.487+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/2200867875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f5e490 0x557f9832afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.487+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/2200867875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f45cd0 0x557f98338b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.487+0000 7fe36f1d6740 1 -- 172.21.15.16:0/2200867875 >> 172.21.15.16:0/2200867875 conn(0x557f97f7b740 msgr2=0x557f9833bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.487+0000 7fe36f1d6740 1 -- 172.21.15.16:0/2200867875 shutdown_connections 2024-08-28T19:32:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.487+0000 7fe36f1d6740 1 -- 172.21.15.16:0/2200867875 wait complete. 2024-08-28T19:32:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe36f1d6740 1 -- start start 2024-08-28T19:32:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe36f1d6740 1 Processor -- start 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f45cd0 0x557f983458f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f5e490 0x557f98345e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe36f1d6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f983405a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3641b00c0 con 0x557f97f66670 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3641af9c0 con 0x557f97f45cd0 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe36f1d6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe3641acb60 con 0x557f97f5e490 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe362d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f983405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe36f1d6740 10 client.0 _ll_register_callbacks cb 0x7fe354005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe36259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f45cd0 0x557f983458f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe362d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f983405a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51542/0 (socket says 172.21.15.16:51542) 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe362d9d640 1 -- 172.21.15.16:0/3143901639 learned_addr learned my addr 172.21.15.16:0/3143901639 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.488+0000 7fe361d9b640 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f5e490 0x557f98345e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe362d9d640 1 -- 172.21.15.16:0/3143901639 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f5e490 msgr2=0x557f98345e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe362d9d640 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f5e490 0x557f98345e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe362d9d640 1 -- 172.21.15.16:0/3143901639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f45cd0 msgr2=0x557f983458f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe362d9d640 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f45cd0 0x557f983458f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe362d9d640 1 -- 172.21.15.16:0/3143901639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f98340d30 con 0x557f97f66670 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe362d9d640 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f983405a0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0098d0 tx=0x7fe35c01bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1231e672cb59ca8f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe349ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe349ffb640 1 -- 172.21.15.16:0/3143901639 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c03d070 con 0x557f97f66670 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe349ffb640 1 -- 172.21.15.16:0/3143901639 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fe35c00ec60 con 0x557f97f66670 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3143901639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe35c02c040 con 0x557f97f66670 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe349ffb640 1 -- 172.21.15.16:0/3143901639 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fe35c02b070 con 0x557f97f66670 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe36f1d6740 10 client.18399 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.489+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3143901639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x557f983baa40 con 0x557f97f66670 2024-08-28T19:32:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.490+0000 7fe349ffb640 1 -- 172.21.15.16:0/3143901639 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe35c02c040 con 0x557f97f66670 2024-08-28T19:32:18.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.490+0000 7fe349ffb640 1 -- 172.21.15.16:0/3143901639 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7fe35c0268f0 con 0x557f97f66670 2024-08-28T19:32:18.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.490+0000 7fe36f1d6740 10 client.18399 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:18.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.490+0000 7fe36f1d6740 10 client.18399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:32:18.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.490+0000 7fe36f1d6740 20 client.18399 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:18.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.490+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557f97f87350 0x557f983bcf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:18.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.490+0000 7fe36f1d6740 4 client.18399 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:32:18.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.490+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3143901639 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557f982a0fe0 con 0x557f97f87350 2024-08-28T19:32:18.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.490+0000 7fe36259c640 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557f97f87350 0x557f983bcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:18.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.490+0000 7fe349ffb640 10 client.18399 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:18.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.490+0000 7fe36259c640 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557f97f87350 0x557f983bcf20 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c64fa6c1155e52b2 server_cookie=40ce849a698e6b96 in_seq=0 out_seq=0 2024-08-28T19:32:18.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe349ffb640 1 -- 172.21.15.16:0/3143901639 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557f982a0fe0 con 0x557f97f87350 2024-08-28T19:32:18.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe349ffb640 10 client.18399 handle_command_reply: tid=0 2024-08-28T19:32:18.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe36f1d6740 1 client.18399 shutdown 2024-08-28T19:32:18.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3143901639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 msgr2=0x557f983405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:18.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f983405a0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0098d0 tx=0x7fe35c01bb00 comp rx=0 tx=0).stop 2024-08-28T19:32:18.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3143901639 shutdown_connections 2024-08-28T19:32:18.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f97f66670 0x557f983405a0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x557f97f87350 0x557f983bcf20 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557f97f5e490 0x557f98345e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe36f1d6740 1 --2- 172.21.15.16:0/3143901639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f97f45cd0 0x557f983458f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3143901639 >> 172.21.15.16:0/3143901639 conn(0x557f97f7b740 msgr2=0x557f9832adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe349ffb640 0 client.18399 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.492+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3143901639 shutdown_connections 2024-08-28T19:32:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe36f1d6740 1 -- 172.21.15.16:0/3143901639 wait complete. 2024-08-28T19:32:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe36f1d6740 20 client.18399 trim_cache size 0 max 16384 2024-08-28T19:32:18.497 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:32:18.497 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:32:18.497 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "3f4b095a-d877-4315-8c85-59f66049db8d", 2024-08-28T19:32:18.497 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:32:18.498 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:32:18.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe368c61640 1 -- 172.21.15.16:0/3020914546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe34c0b4cd0 msgr2=0x7fe34c0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:18.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe368c61640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe34c0b4cd0 0x7fe34c0b71e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe35c023000 tx=0x7fe35c024000 comp rx=0 tx=0).stop 2024-08-28T19:32:18.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe368c61640 1 -- 172.21.15.16:0/3020914546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe364102800 msgr2=0x7fe36419b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:18.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe368c61640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe364102800 0x7fe36419b0c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe354005940 tx=0x7fe3540040c0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe368c61640 1 -- 172.21.15.16:0/3020914546 shutdown_connections 2024-08-28T19:32:18.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe368c61640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe364108a90 0x7fe36419bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe368c61640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe34c0b4cd0 0x7fe34c0b71e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe368c61640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe364108140 0x7fe36419b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe368c61640 1 --2- 172.21.15.16:0/3020914546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe364102800 0x7fe36419b0c0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:18.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.493+0000 7fe368c61640 1 -- 172.21.15.16:0/3020914546 >> 172.21.15.16:0/3020914546 conn(0x7fe3640794d0 msgr2=0x7fe364101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:18.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.494+0000 7fe368c61640 1 -- 172.21.15.16:0/3020914546 shutdown_connections 2024-08-28T19:32:18.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:18.494+0000 7fe368c61640 1 -- 172.21.15.16:0/3020914546 wait complete. 2024-08-28T19:32:18.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 466894 3.79 MB/sec execute 478 sec latency 26.206 ms 2024-08-28T19:32:19.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:32:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: pgmap v687: 129 pgs: 129 active+clean; 365 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 4.6 MiB/s wr, 145 op/s 2024-08-28T19:32:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:32:19.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:32:19.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:19 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: pgmap v687: 129 pgs: 129 active+clean; 365 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 4.6 MiB/s wr, 145 op/s 2024-08-28T19:32:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:32:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:32:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:32:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:32:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:32:19.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:19 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: pgmap v687: 129 pgs: 129 active+clean; 365 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 4.6 MiB/s wr, 145 op/s 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:32:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:32:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:32:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:19 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.924+0000 7f06306e7640 1 -- 172.21.15.16:0/1554113063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0628104750 msgr2=0x7f0628104b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:19.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.924+0000 7f06306e7640 1 --2- 172.21.15.16:0/1554113063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0628104750 0x7f0628104b60 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f0624009900 tx=0x7f062402a410 comp rx=0 tx=0).stop 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.924+0000 7f06306e7640 1 -- 172.21.15.16:0/1554113063 shutdown_connections 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.924+0000 7f06306e7640 1 --2- 172.21.15.16:0/1554113063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f062810dbb0 0x7f06281100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.924+0000 7f06306e7640 1 --2- 172.21.15.16:0/1554113063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06281050a0 0x7f062810d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.924+0000 7f06306e7640 1 --2- 172.21.15.16:0/1554113063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0628104750 0x7f0628104b60 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.924+0000 7f06306e7640 1 -- 172.21.15.16:0/1554113063 >> 172.21.15.16:0/1554113063 conn(0x7f06280794d0 msgr2=0x7f06280798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.924+0000 7f06306e7640 1 -- 172.21.15.16:0/1554113063 shutdown_connections 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.924+0000 7f06306e7640 1 -- 172.21.15.16:0/1554113063 wait complete. 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f06306e7640 1 Processor -- start 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f06306e7640 1 -- start start 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f06306e7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0628104750 0x7f062819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f06306e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06281050a0 0x7f062819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f06306e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810dbb0 0x7f06281a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f06306e7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0628113830 con 0x7f062810dbb0 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f06306e7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06281136b0 con 0x7f06281050a0 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f06306e7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f06281139b0 con 0x7f0628104750 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f062ec84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810dbb0 0x7f06281a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f062ec84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810dbb0 0x7f06281a1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51554/0 (socket says 172.21.15.16:51554) 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f062ec84640 1 -- 172.21.15.16:0/4021712423 learned_addr learned my addr 172.21.15.16:0/4021712423 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f062e483640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0628104750 0x7f062819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f062dc82640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06281050a0 0x7f062819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f062ec84640 1 -- 172.21.15.16:0/4021712423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0628104750 msgr2=0x7f062819c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f062ec84640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0628104750 0x7f062819c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f062ec84640 1 -- 172.21.15.16:0/4021712423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06281050a0 msgr2=0x7f062819ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f062ec84640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06281050a0 0x7f062819ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f062ec84640 1 -- 172.21.15.16:0/4021712423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06281a23d0 con 0x7f062810dbb0 2024-08-28T19:32:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.925+0000 7f062e483640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0628104750 0x7f062819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.926+0000 7f062ec84640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810dbb0 0x7f06281a1ca0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f062000eaa0 tx=0x7f062000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.926+0000 7f061f7fe640 1 -- 172.21.15.16:0/4021712423 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062000cdb0 con 0x7f062810dbb0 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.926+0000 7f06306e7640 1 -- 172.21.15.16:0/4021712423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06281a26c0 con 0x7f062810dbb0 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.926+0000 7f061f7fe640 1 -- 172.21.15.16:0/4021712423 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0620005020 con 0x7f062810dbb0 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.926+0000 7f061f7fe640 1 -- 172.21.15.16:0/4021712423 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06200116d0 con 0x7f062810dbb0 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.926+0000 7f06306e7640 1 -- 172.21.15.16:0/4021712423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06281a8e20 con 0x7f062810dbb0 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.929+0000 7f0636c5c740 1 Processor -- start 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.929+0000 7f061f7fe640 1 -- 172.21.15.16:0/4021712423 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f06200051d0 con 0x7f062810dbb0 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.929+0000 7f0636c5c740 1 -- start start 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.929+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.929+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070af800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.929+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070afd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.929+0000 7f0636c5c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558707026630 con 0x558706ce11b0 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.929+0000 7f0636c5c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558707015bc0 con 0x558706ce9390 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.929+0000 7f0636c5c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558707019b70 con 0x558706d0a070 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062e483640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062dc82640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062e483640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070af800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51560/0 (socket says 172.21.15.16:51560) 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062e483640 1 -- 172.21.15.16:0/3599791366 learned_addr learned my addr 172.21.15.16:0/3599791366 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062dc82640 1 --2- 172.21.15.16:0/3599791366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062e483640 1 -- 172.21.15.16:0/3599791366 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 msgr2=0x5587070af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062e483640 1 --2- 172.21.15.16:0/3599791366 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070af2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062e483640 1 -- 172.21.15.16:0/3599791366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 msgr2=0x5587070afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062e483640 1 --2- 172.21.15.16:0/3599791366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070afd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062e483640 1 -- 172.21.15.16:0/3599791366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558706d75650 con 0x558706ce11b0 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f061f7fe640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06140b4ca0 0x7f06140b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062dc82640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06140b4ca0 0x7f06140b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f062e483640 1 --2- 172.21.15.16:0/3599791366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070af800 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f0624031440 tx=0x7f0624032e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3f1a2f74a64c0b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f061f7fe640 1 -- 172.21.15.16:0/4021712423 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f06200e5b10 con 0x7f062810dbb0 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f061cff9640 1 -- 172.21.15.16:0/3599791366 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062403f070 con 0x558706ce11b0 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f061cff9640 1 -- 172.21.15.16:0/3599791366 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06240316a0 con 0x558706ce11b0 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.930+0000 7f061cff9640 1 -- 172.21.15.16:0/3599791366 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624035070 con 0x558706ce11b0 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.931+0000 7f0636c5c740 1 -- 172.21.15.16:0/3599791366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 msgr2=0x5587070af800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.931+0000 7f062dc82640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06140b4ca0 0x7f06140b71b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f061800afd0 tx=0x7f061800a250 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.931+0000 7f0636c5c740 1 --2- 172.21.15.16:0/3599791366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070af800 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f0624031440 tx=0x7f0624032e60 comp rx=0 tx=0).stop 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.931+0000 7f0636c5c740 1 -- 172.21.15.16:0/3599791366 shutdown_connections 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.931+0000 7f0636c5c740 1 --2- 172.21.15.16:0/3599791366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070afd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.931+0000 7f0636c5c740 1 --2- 172.21.15.16:0/3599791366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070af800 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.931+0000 7f0636c5c740 1 --2- 172.21.15.16:0/3599791366 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070af2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.931+0000 7f0636c5c740 1 -- 172.21.15.16:0/3599791366 >> 172.21.15.16:0/3599791366 conn(0x558706cfe460 msgr2=0x5587070ad1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f0636c5c740 1 -- 172.21.15.16:0/3599791366 shutdown_connections 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f0636c5c740 1 -- 172.21.15.16:0/3599791366 wait complete. 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f0636c5c740 1 -- start start 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f0636c5c740 1 Processor -- start 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce11b0 0x5587070baa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706ce9390 0x5587070baf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706d0a070 0x5587070c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f0636c5c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558706e78700 con 0x558706d0a070 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f0636c5c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558706f9c700 con 0x558706ce11b0 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f0636c5c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5587070bda40 con 0x558706ce9390 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f0636c5c740 10 client.0 _ll_register_callbacks cb 0x558706fcbdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f062ec84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706d0a070 0x5587070c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.932+0000 7f062ec84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706d0a070 0x5587070c6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51576/0 (socket says 172.21.15.16:51576) 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f062ec84640 1 -- 172.21.15.16:0/1400355886 learned_addr learned my addr 172.21.15.16:0/1400355886 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f062e483640 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce11b0 0x5587070baa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f062dc82640 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706ce9390 0x5587070baf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f062ec84640 1 -- 172.21.15.16:0/1400355886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706ce9390 msgr2=0x5587070baf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f062ec84640 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706ce9390 0x5587070baf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f062ec84640 1 -- 172.21.15.16:0/1400355886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce11b0 msgr2=0x5587070baa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f062ec84640 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce11b0 0x5587070baa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f062ec84640 1 -- 172.21.15.16:0/1400355886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558706fd58b0 con 0x558706d0a070 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f062dc82640 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706ce9390 0x5587070baf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f062ec84640 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706d0a070 0x5587070c6d30 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f06200e2ef0 tx=0x7f06200ac5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47dfb164be7104df server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f0611ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f0611ffb640 1 -- 172.21.15.16:0/1400355886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062001f040 con 0x558706d0a070 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f0611ffb640 1 -- 172.21.15.16:0/1400355886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f06200b02f0 con 0x558706d0a070 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f0636c5c740 1 -- 172.21.15.16:0/1400355886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06240316a0 con 0x558706d0a070 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f0611ffb640 1 -- 172.21.15.16:0/1400355886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f06200971a0 con 0x558706d0a070 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f0636c5c740 10 client.18411 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f0636c5c740 1 -- 172.21.15.16:0/1400355886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5587070c98a0 con 0x558706d0a070 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.933+0000 7f0611ffb640 1 -- 172.21.15.16:0/1400355886 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06240316a0 con 0x558706d0a070 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.934+0000 7f0611ffb640 1 -- 172.21.15.16:0/1400355886 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7f062000f040 con 0x558706d0a070 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.934+0000 7f0636c5c740 10 client.18411 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.934+0000 7f0636c5c740 10 client.18411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.934+0000 7f0636c5c740 20 client.18411 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.934+0000 7f0636c5c740 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558706cc89f0 0x5587070ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.934+0000 7f0636c5c740 4 client.18411 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.934+0000 7f0636c5c740 1 -- 172.21.15.16:0/1400355886 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558707028580 con 0x558706cc89f0 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.934+0000 7f062e483640 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558706cc89f0 0x5587070ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.934+0000 7f0611ffb640 10 client.18411 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.934+0000 7f062e483640 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558706cc89f0 0x5587070ccd70 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61041195202dffd4 server_cookie=eca31a6eddb035f4 in_seq=0 out_seq=0 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0611ffb640 1 -- 172.21.15.16:0/1400355886 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558707028580 con 0x558706cc89f0 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0611ffb640 10 client.18411 handle_command_reply: tid=0 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0636c5c740 1 client.18411 shutdown 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0636c5c740 1 -- 172.21.15.16:0/1400355886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706d0a070 msgr2=0x5587070c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0636c5c740 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706d0a070 0x5587070c6d30 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f06200e2ef0 tx=0x7f06200ac5f0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0636c5c740 1 -- 172.21.15.16:0/1400355886 shutdown_connections 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0636c5c740 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706d0a070 0x5587070c6d30 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0636c5c740 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558706cc89f0 0x5587070ccd70 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0636c5c740 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706ce9390 0x5587070baf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0636c5c740 1 --2- 172.21.15.16:0/1400355886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce11b0 0x5587070baa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0611ffb640 0 client.18411 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.935+0000 7f0636c5c740 1 -- 172.21.15.16:0/1400355886 >> 172.21.15.16:0/1400355886 conn(0x558706cfe460 msgr2=0x5587070b2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.936+0000 7f0636c5c740 1 -- 172.21.15.16:0/1400355886 shutdown_connections 2024-08-28T19:32:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.936+0000 7f0636c5c740 1 -- 172.21.15.16:0/1400355886 wait complete. 2024-08-28T19:32:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.936+0000 7f0636c5c740 20 client.18411 trim_cache size 0 max 16384 2024-08-28T19:32:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.953+0000 7f0636c5c740 1 Processor -- start 2024-08-28T19:32:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.953+0000 7f0636c5c740 1 -- start start 2024-08-28T19:32:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f0636c5c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06281b2dd0 con 0x558706ce11b0 2024-08-28T19:32:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f0636c5c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06281b4a00 con 0x558706ce9390 2024-08-28T19:32:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f0636c5c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f06281b3560 con 0x558706d0a070 2024-08-28T19:32:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062e483640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062e483640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070ad1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51580/0 (socket says 172.21.15.16:51580) 2024-08-28T19:32:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062e483640 1 -- 172.21.15.16:0/4117865568 learned_addr learned my addr 172.21.15.16:0/4117865568 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062dc82640 1 --2- 172.21.15.16:0/4117865568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062ec84640 1 --2- 172.21.15.16:0/4117865568 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062e483640 1 -- 172.21.15.16:0/4117865568 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 msgr2=0x5587070b7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062e483640 1 --2- 172.21.15.16:0/4117865568 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070b7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062e483640 1 -- 172.21.15.16:0/4117865568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 msgr2=0x5587070c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062e483640 1 --2- 172.21.15.16:0/4117865568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062e483640 1 -- 172.21.15.16:0/4117865568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587070a8550 con 0x558706ce11b0 2024-08-28T19:32:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062dc82640 1 --2- 172.21.15.16:0/4117865568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:32:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f062e483640 1 --2- 172.21.15.16:0/4117865568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070ad1e0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x5587070adf40 tx=0x7f0624031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a099c7d51db7f14 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f0613fff640 1 -- 172.21.15.16:0/4117865568 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062403f040 con 0x558706ce11b0 2024-08-28T19:32:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f0613fff640 1 -- 172.21.15.16:0/4117865568 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0624005030 con 0x558706ce11b0 2024-08-28T19:32:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.954+0000 7f0613fff640 1 -- 172.21.15.16:0/4117865568 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624035040 con 0x558706ce11b0 2024-08-28T19:32:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.955+0000 7f0636c5c740 1 -- 172.21.15.16:0/4117865568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 msgr2=0x5587070ad1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.955+0000 7f0636c5c740 1 --2- 172.21.15.16:0/4117865568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070ad1e0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x5587070adf40 tx=0x7f0624031a30 comp rx=0 tx=0).stop 2024-08-28T19:32:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.955+0000 7f0636c5c740 1 -- 172.21.15.16:0/4117865568 shutdown_connections 2024-08-28T19:32:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.955+0000 7f0636c5c740 1 --2- 172.21.15.16:0/4117865568 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070b7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.955+0000 7f0636c5c740 1 --2- 172.21.15.16:0/4117865568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.955+0000 7f0636c5c740 1 --2- 172.21.15.16:0/4117865568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070ad1e0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.955+0000 7f0636c5c740 1 -- 172.21.15.16:0/4117865568 >> 172.21.15.16:0/4117865568 conn(0x558706cfe460 msgr2=0x5587070acad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.955+0000 7f0636c5c740 1 -- 172.21.15.16:0/4117865568 shutdown_connections 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.955+0000 7f0636c5c740 1 -- 172.21.15.16:0/4117865568 wait complete. 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f0636c5c740 1 -- start start 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f0636c5c740 1 Processor -- start 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070c3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070beac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f0636c5c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070bf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f0636c5c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06281b4330 con 0x558706ce11b0 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f0636c5c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06281b3c30 con 0x558706ce9390 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f0636c5c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f06281b0dd0 con 0x558706d0a070 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f0636c5c740 10 client.0 _ll_register_callbacks cb 0x7f06200a7db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f062e483640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070c3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f062e483640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070c3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51590/0 (socket says 172.21.15.16:51590) 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f062e483640 1 -- 172.21.15.16:0/4145488859 learned_addr learned my addr 172.21.15.16:0/4145488859 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f062dc82640 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070beac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f062ec84640 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070bf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f062e483640 1 -- 172.21.15.16:0/4145488859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 msgr2=0x5587070bf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f062e483640 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070bf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f062e483640 1 -- 172.21.15.16:0/4145488859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 msgr2=0x5587070beac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f062e483640 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070beac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.956+0000 7f062e483640 1 -- 172.21.15.16:0/4145488859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558707136780 con 0x558706ce11b0 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f062e483640 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070c3610 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f0624009900 tx=0x7f0624031620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de8c4e162090013 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0611ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0611ffb640 1 -- 172.21.15.16:0/4145488859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062403f030 con 0x558706ce11b0 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0611ffb640 1 -- 172.21.15.16:0/4145488859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f06240051e0 con 0x558706ce11b0 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0636c5c740 1 -- 172.21.15.16:0/4145488859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0624005030 con 0x558706ce11b0 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0611ffb640 1 -- 172.21.15.16:0/4145488859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f0624035040 con 0x558706ce11b0 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0636c5c740 10 client.18417 fetch_fsmap learned FSMap version 58 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0636c5c740 1 -- 172.21.15.16:0/4145488859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x558707137860 con 0x558706ce11b0 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0611ffb640 1 -- 172.21.15.16:0/4145488859 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0624005030 con 0x558706ce11b0 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0611ffb640 1 -- 172.21.15.16:0/4145488859 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) ==== 8390+0+0 (secure 0 0 0) 0x7f0624060500 con 0x558706ce11b0 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0636c5c740 10 client.18417 fetch_fsmap finished waiting for FSMap version 58 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0636c5c740 10 client.18417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0636c5c740 20 client.18417 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0636c5c740 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558706c9dda0 0x558707139c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0636c5c740 4 client.18417 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:32:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.957+0000 7f0636c5c740 1 -- 172.21.15.16:0/4145488859 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558707024020 con 0x558706c9dda0 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.958+0000 7f062dc82640 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558706c9dda0 0x558707139c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.958+0000 7f0611ffb640 10 client.18417 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.958+0000 7f062dc82640 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558706c9dda0 0x558707139c20 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa065f96e8c4a23e server_cookie=6cb35cf9b46e445d in_seq=0 out_seq=0 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.958+0000 7f0611ffb640 1 -- 172.21.15.16:0/4145488859 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4830 (crc 0 0 0) 0x558707024020 con 0x558706c9dda0 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.958+0000 7f0611ffb640 10 client.18417 handle_command_reply: tid=0 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.958+0000 7f0636c5c740 1 client.18417 shutdown 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0636c5c740 1 -- 172.21.15.16:0/4145488859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 msgr2=0x5587070c3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0636c5c740 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070c3610 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f0624009900 tx=0x7f0624031620 comp rx=0 tx=0).stop 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0636c5c740 1 -- 172.21.15.16:0/4145488859 shutdown_connections 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0636c5c740 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558706c9dda0 0x558707139c20 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0636c5c740 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558706d0a070 0x5587070bf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0636c5c740 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558706ce9390 0x5587070beac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0636c5c740 1 --2- 172.21.15.16:0/4145488859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558706ce11b0 0x5587070c3610 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0611ffb640 0 client.18417 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0636c5c740 1 -- 172.21.15.16:0/4145488859 >> 172.21.15.16:0/4145488859 conn(0x558706cfe460 msgr2=0x5587070b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0636c5c740 1 -- 172.21.15.16:0/4145488859 shutdown_connections 2024-08-28T19:32:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0636c5c740 1 -- 172.21.15.16:0/4145488859 wait complete. 2024-08-28T19:32:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f0636c5c740 20 client.18417 trim_cache size 0 max 16384 2024-08-28T19:32:19.966 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:32:19.966 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:32:19.966 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:32:19.966 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:32:19.966 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "3f4b095a-d877-4315-8c85-59f66049db8d", 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:32:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:32:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:32:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:32:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:32:19.977 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:32:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 -- 172.21.15.16:0/4021712423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06140b4ca0 msgr2=0x7f06140b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06140b4ca0 0x7f06140b71b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f061800afd0 tx=0x7f061800a250 comp rx=0 tx=0).stop 2024-08-28T19:32:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 -- 172.21.15.16:0/4021712423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810dbb0 msgr2=0x7f06281a1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810dbb0 0x7f06281a1ca0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f062000eaa0 tx=0x7f062000ef60 comp rx=0 tx=0).stop 2024-08-28T19:32:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 -- 172.21.15.16:0/4021712423 shutdown_connections 2024-08-28T19:32:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810dbb0 0x7f06281a1ca0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f06140b4ca0 0x7f06140b71b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06281050a0 0x7f062819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 --2- 172.21.15.16:0/4021712423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0628104750 0x7f062819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 -- 172.21.15.16:0/4021712423 >> 172.21.15.16:0/4021712423 conn(0x7f06280794d0 msgr2=0x7f0628100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 -- 172.21.15.16:0/4021712423 shutdown_connections 2024-08-28T19:32:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:19.959+0000 7f06306e7640 1 -- 172.21.15.16:0/4021712423 wait complete. 2024-08-28T19:32:19.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 467138 3.78 MB/sec execute 479 sec latency 828.978 ms 2024-08-28T19:32:20.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:20 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:32:20.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:20 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:20.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:20 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:32:20.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:20 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:20.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:20 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:32:20.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:20 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:20.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:20 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:32:20.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:20 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:20.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:20 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:32:20.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:20 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:20.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:20 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:32:20.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:20 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:32:20.794 INFO:tasks.cephfs.filesystem:scrub status for tag:3f4b095a-d877-4315-8c85-59f66049db8d - {'path': '/', 'tag': '3f4b095a-d877-4315-8c85-59f66049db8d', 'options': 'recursive,force'} 2024-08-28T19:32:20.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 467138 3.77 MB/sec execute 480 sec latency 1829.047 ms 2024-08-28T19:32:21.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:21 smithi028 ceph-mon[21769]: pgmap v688: 129 pgs: 129 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 4.8 MiB/s wr, 153 op/s 2024-08-28T19:32:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:21 smithi115 ceph-mon[22184]: pgmap v688: 129 pgs: 129 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 4.8 MiB/s wr, 153 op/s 2024-08-28T19:32:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:21 smithi016 ceph-mon[18556]: pgmap v688: 129 pgs: 129 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 4.8 MiB/s wr, 153 op/s 2024-08-28T19:32:21.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 467138 3.77 MB/sec execute 481 sec latency 2829.136 ms 2024-08-28T19:32:22.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469262 3.79 MB/sec execute 482 sec latency 2832.457 ms 2024-08-28T19:32:23.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:23 smithi028 ceph-mon[21769]: pgmap v689: 129 pgs: 129 active+clean; 362 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 4.5 MiB/s wr, 169 op/s 2024-08-28T19:32:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:23 smithi115 ceph-mon[22184]: pgmap v689: 129 pgs: 129 active+clean; 362 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 4.5 MiB/s wr, 169 op/s 2024-08-28T19:32:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:23 smithi016 ceph-mon[18556]: pgmap v689: 129 pgs: 129 active+clean; 362 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 4.5 MiB/s wr, 169 op/s 2024-08-28T19:32:23.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 470685 3.79 MB/sec execute 483 sec latency 79.026 ms 2024-08-28T19:32:24.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472345 3.80 MB/sec execute 484 sec latency 221.080 ms 2024-08-28T19:32:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:25 smithi028 ceph-mon[21769]: pgmap v690: 129 pgs: 129 active+clean; 364 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 4.2 MiB/s wr, 150 op/s 2024-08-28T19:32:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:25 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:32:25.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:25 smithi115 ceph-mon[22184]: pgmap v690: 129 pgs: 129 active+clean; 364 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 4.2 MiB/s wr, 150 op/s 2024-08-28T19:32:25.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:25 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:32:25.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:25 smithi016 ceph-mon[18556]: pgmap v690: 129 pgs: 129 active+clean; 364 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 4.2 MiB/s wr, 150 op/s 2024-08-28T19:32:25.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:25 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:32:25.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472345 3.79 MB/sec execute 485 sec latency 1221.184 ms 2024-08-28T19:32:26.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472358 3.79 MB/sec execute 486 sec latency 1847.721 ms 2024-08-28T19:32:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:27 smithi028 ceph-mon[21769]: pgmap v691: 129 pgs: 129 active+clean; 366 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 4.1 MiB/s wr, 179 op/s 2024-08-28T19:32:27.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:27 smithi115 ceph-mon[22184]: pgmap v691: 129 pgs: 129 active+clean; 366 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 4.1 MiB/s wr, 179 op/s 2024-08-28T19:32:27.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:27 smithi016 ceph-mon[18556]: pgmap v691: 129 pgs: 129 active+clean; 366 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 4.1 MiB/s wr, 179 op/s 2024-08-28T19:32:27.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472358 3.78 MB/sec execute 487 sec latency 1357.559 ms 2024-08-28T19:32:28.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472358 3.77 MB/sec execute 488 sec latency 2357.661 ms 2024-08-28T19:32:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:29 smithi028 ceph-mon[21769]: pgmap v692: 129 pgs: 129 active+clean; 367 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 2.7 MiB/s wr, 137 op/s 2024-08-28T19:32:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:29 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:32:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:29 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:32:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:29 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:32:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:29 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:32:29.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:29 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:32:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:29 smithi115 ceph-mon[22184]: pgmap v692: 129 pgs: 129 active+clean; 367 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 2.7 MiB/s wr, 137 op/s 2024-08-28T19:32:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:29 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:32:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:29 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:32:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:29 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:32:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:29 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:32:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:29 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:32:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:29 smithi016 ceph-mon[18556]: pgmap v692: 129 pgs: 129 active+clean; 367 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 2.7 MiB/s wr, 137 op/s 2024-08-28T19:32:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:29 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:32:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:29 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:32:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:29 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:32:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:29 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:32:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:29 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:32:29.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472358 3.76 MB/sec execute 489 sec latency 3357.805 ms 2024-08-28T19:32:30.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472358 3.75 MB/sec execute 490 sec latency 4357.876 ms 2024-08-28T19:32:31.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:31 smithi028 ceph-mon[21769]: pgmap v693: 129 pgs: 129 active+clean; 368 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-08-28T19:32:31.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:31 smithi115 ceph-mon[22184]: pgmap v693: 129 pgs: 129 active+clean; 368 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-08-28T19:32:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:31 smithi016 ceph-mon[18556]: pgmap v693: 129 pgs: 129 active+clean; 368 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-08-28T19:32:31.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472371 3.75 MB/sec execute 491 sec latency 4987.204 ms 2024-08-28T19:32:32.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472371 3.74 MB/sec execute 492 sec latency 1355.072 ms 2024-08-28T19:32:33.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:33 smithi028 ceph-mon[21769]: pgmap v694: 129 pgs: 129 active+clean; 368 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 949 KiB/s wr, 110 op/s 2024-08-28T19:32:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:33 smithi115 ceph-mon[22184]: pgmap v694: 129 pgs: 129 active+clean; 368 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 949 KiB/s wr, 110 op/s 2024-08-28T19:32:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:33 smithi016 ceph-mon[18556]: pgmap v694: 129 pgs: 129 active+clean; 368 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 949 KiB/s wr, 110 op/s 2024-08-28T19:32:33.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472371 3.73 MB/sec execute 493 sec latency 2355.123 ms 2024-08-28T19:32:34.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472371 3.72 MB/sec execute 494 sec latency 3355.173 ms 2024-08-28T19:32:35.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:35 smithi028 ceph-mon[21769]: pgmap v695: 129 pgs: 129 active+clean; 368 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 612 KiB/s wr, 70 op/s 2024-08-28T19:32:35.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:35 smithi115 ceph-mon[22184]: pgmap v695: 129 pgs: 129 active+clean; 368 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 612 KiB/s wr, 70 op/s 2024-08-28T19:32:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:35 smithi016 ceph-mon[18556]: pgmap v695: 129 pgs: 129 active+clean; 368 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 612 KiB/s wr, 70 op/s 2024-08-28T19:32:35.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472371 3.72 MB/sec execute 495 sec latency 4355.227 ms 2024-08-28T19:32:36.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472910 3.72 MB/sec execute 496 sec latency 4982.081 ms 2024-08-28T19:32:37.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:37 smithi028 ceph-mon[21769]: pgmap v696: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 440 KiB/s wr, 64 op/s 2024-08-28T19:32:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:37 smithi115 ceph-mon[22184]: pgmap v696: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 440 KiB/s wr, 64 op/s 2024-08-28T19:32:37.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:37 smithi016 ceph-mon[18556]: pgmap v696: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 440 KiB/s wr, 64 op/s 2024-08-28T19:32:37.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 474231 3.72 MB/sec execute 497 sec latency 138.581 ms 2024-08-28T19:32:38.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 474231 3.71 MB/sec execute 498 sec latency 1138.673 ms 2024-08-28T19:32:39.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:39 smithi028 ceph-mon[21769]: pgmap v697: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 225 KiB/s wr, 29 op/s 2024-08-28T19:32:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:39 smithi115 ceph-mon[22184]: pgmap v697: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 225 KiB/s wr, 29 op/s 2024-08-28T19:32:39.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:39 smithi016 ceph-mon[18556]: pgmap v697: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 225 KiB/s wr, 29 op/s 2024-08-28T19:32:39.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 474231 3.71 MB/sec execute 499 sec latency 2138.771 ms 2024-08-28T19:32:40.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 474231 3.70 MB/sec execute 500 sec latency 3138.874 ms 2024-08-28T19:32:41.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:41 smithi028 ceph-mon[21769]: pgmap v698: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 285 KiB/s wr, 34 op/s 2024-08-28T19:32:41.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:41 smithi115 ceph-mon[22184]: pgmap v698: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 285 KiB/s wr, 34 op/s 2024-08-28T19:32:41.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:41 smithi016 ceph-mon[18556]: pgmap v698: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 285 KiB/s wr, 34 op/s 2024-08-28T19:32:41.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 474234 3.69 MB/sec execute 501 sec latency 4135.747 ms 2024-08-28T19:32:42.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 476328 3.71 MB/sec execute 502 sec latency 17.499 ms 2024-08-28T19:32:43.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:43 smithi028 ceph-mon[21769]: pgmap v699: 129 pgs: 129 active+clean; 371 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 289 KiB/s wr, 46 op/s 2024-08-28T19:32:43.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:43 smithi115 ceph-mon[22184]: pgmap v699: 129 pgs: 129 active+clean; 371 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 289 KiB/s wr, 46 op/s 2024-08-28T19:32:43.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:43 smithi016 ceph-mon[18556]: pgmap v699: 129 pgs: 129 active+clean; 371 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 289 KiB/s wr, 46 op/s 2024-08-28T19:32:43.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 477756 3.71 MB/sec execute 503 sec latency 15.072 ms 2024-08-28T19:32:44.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 479873 3.73 MB/sec execute 504 sec latency 103.686 ms 2024-08-28T19:32:45.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:45 smithi028 ceph-mon[21769]: pgmap v700: 129 pgs: 129 active+clean; 371 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 296 KiB/s wr, 39 op/s 2024-08-28T19:32:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:45 smithi115 ceph-mon[22184]: pgmap v700: 129 pgs: 129 active+clean; 371 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 296 KiB/s wr, 39 op/s 2024-08-28T19:32:45.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:45 smithi016 ceph-mon[18556]: pgmap v700: 129 pgs: 129 active+clean; 371 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 296 KiB/s wr, 39 op/s 2024-08-28T19:32:45.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 481174 3.74 MB/sec execute 505 sec latency 16.262 ms 2024-08-28T19:32:46.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 481363 3.73 MB/sec execute 506 sec latency 888.571 ms 2024-08-28T19:32:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:47 smithi115 ceph-mon[22184]: pgmap v701: 129 pgs: 129 active+clean; 378 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 846 KiB/s wr, 96 op/s 2024-08-28T19:32:47.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:47 smithi016 ceph-mon[18556]: pgmap v701: 129 pgs: 129 active+clean; 378 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 846 KiB/s wr, 96 op/s 2024-08-28T19:32:47.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:47 smithi028 ceph-mon[21769]: pgmap v701: 129 pgs: 129 active+clean; 378 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 846 KiB/s wr, 96 op/s 2024-08-28T19:32:47.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 483464 3.75 MB/sec execute 507 sec latency 19.350 ms 2024-08-28T19:32:48.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 484878 3.75 MB/sec execute 508 sec latency 23.138 ms 2024-08-28T19:32:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:49 smithi115 ceph-mon[22184]: pgmap v702: 129 pgs: 129 active+clean; 378 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 950 KiB/s wr, 97 op/s 2024-08-28T19:32:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:49 smithi016 ceph-mon[18556]: pgmap v702: 129 pgs: 129 active+clean; 378 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 950 KiB/s wr, 97 op/s 2024-08-28T19:32:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:49 smithi028 ceph-mon[21769]: pgmap v702: 129 pgs: 129 active+clean; 378 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 950 KiB/s wr, 97 op/s 2024-08-28T19:32:49.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 484911 3.75 MB/sec execute 509 sec latency 972.775 ms 2024-08-28T19:32:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:32:50.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:32:50.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:32:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:32:50.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 484911 3.74 MB/sec execute 510 sec latency 1972.817 ms 2024-08-28T19:32:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:51.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:51 smithi016 ceph-mon[18556]: pgmap v703: 129 pgs: 129 active+clean; 380 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-08-28T19:32:51.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:51 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:32:51.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:51 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:32:51.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:51 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:32:51.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:51 smithi115 ceph-mon[22184]: pgmap v703: 129 pgs: 129 active+clean; 380 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-08-28T19:32:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:51 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:32:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:51 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:32:51.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:51 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:32:51.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:51 smithi028 ceph-mon[21769]: pgmap v703: 129 pgs: 129 active+clean; 380 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-08-28T19:32:51.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:51 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:32:51.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:51 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:32:51.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:51 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:32:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.853+0000 7fa88778b640 1 -- 172.21.15.16:0/2913948134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8801039d0 msgr2=0x7fa880103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.853+0000 7fa88778b640 1 --2- 172.21.15.16:0/2913948134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8801039d0 0x7fa880103de0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009900 tx=0x7fa87c02a410 comp rx=0 tx=0).stop 2024-08-28T19:32:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.853+0000 7fa88778b640 1 -- 172.21.15.16:0/2913948134 shutdown_connections 2024-08-28T19:32:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.853+0000 7fa88778b640 1 --2- 172.21.15.16:0/2913948134 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa880109c60 0x7fa880110490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.853+0000 7fa88778b640 1 --2- 172.21.15.16:0/2913948134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa880109310 0x7fa880109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.853+0000 7fa88778b640 1 --2- 172.21.15.16:0/2913948134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8801039d0 0x7fa880103de0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.853+0000 7fa88778b640 1 -- 172.21.15.16:0/2913948134 >> 172.21.15.16:0/2913948134 conn(0x7fa88007b670 msgr2=0x7fa88007ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.853+0000 7fa88778b640 1 -- 172.21.15.16:0/2913948134 shutdown_connections 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.853+0000 7fa88778b640 1 -- 172.21.15.16:0/2913948134 wait complete. 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.854+0000 7fa88778b640 1 Processor -- start 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.854+0000 7fa88778b640 1 -- start start 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.854+0000 7fa88778b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8801039d0 0x7fa88019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa88778b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa880109310 0x7fa88019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa88778b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa880109c60 0x7fa88019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa88778b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa880113880 con 0x7fa880109c60 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa88778b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa880113700 con 0x7fa880109310 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa88778b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa880113a00 con 0x7fa8801039d0 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa884d26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa880109310 0x7fa88019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa884d26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa880109310 0x7fa88019a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:56112/0 (socket says 172.21.15.16:56112) 2024-08-28T19:32:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa884d26640 1 -- 172.21.15.16:0/2398855036 learned_addr learned my addr 172.21.15.16:0/2398855036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa885527640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8801039d0 0x7fa88019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa885d28640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa880109c60 0x7fa88019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa884d26640 1 -- 172.21.15.16:0/2398855036 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8801039d0 msgr2=0x7fa88019a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa884d26640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8801039d0 0x7fa88019a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa884d26640 1 -- 172.21.15.16:0/2398855036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa880109c60 msgr2=0x7fa88019fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa884d26640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa880109c60 0x7fa88019fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa884d26640 1 -- 172.21.15.16:0/2398855036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8801a02e0 con 0x7fa880109310 2024-08-28T19:32:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.855+0000 7fa884d26640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa880109310 0x7fa88019a910 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa87000e9f0 tx=0x7fa87000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.856+0000 7fa8767fc640 1 -- 172.21.15.16:0/2398855036 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87000ccc0 con 0x7fa880109310 2024-08-28T19:32:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.856+0000 7fa88778b640 1 -- 172.21.15.16:0/2398855036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8801a05d0 con 0x7fa880109310 2024-08-28T19:32:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.856+0000 7fa8767fc640 1 -- 172.21.15.16:0/2398855036 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8700108e0 con 0x7fa880109310 2024-08-28T19:32:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.856+0000 7fa8767fc640 1 -- 172.21.15.16:0/2398855036 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870018860 con 0x7fa880109310 2024-08-28T19:32:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.856+0000 7fa88778b640 1 -- 172.21.15.16:0/2398855036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8801a6d30 con 0x7fa880109310 2024-08-28T19:32:51.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.858+0000 7fa88dd00740 1 Processor -- start 2024-08-28T19:32:51.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.858+0000 7fa88dd00740 1 -- start start 2024-08-28T19:32:51.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8658c070 0x560b869312c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 0x560b86931800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8656b390 0x560b86931d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa88dd00740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560b868a8630 con 0x560b865631b0 2024-08-28T19:32:51.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa88dd00740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560b86897bc0 con 0x560b8658c070 2024-08-28T19:32:51.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa88dd00740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560b8689bb70 con 0x560b8656b390 2024-08-28T19:32:51.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa885d28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8658c070 0x560b869312c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa885d28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8658c070 0x560b869312c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:56116/0 (socket says 172.21.15.16:56116) 2024-08-28T19:32:51.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa885d28640 1 -- 172.21.15.16:0/4030575837 learned_addr learned my addr 172.21.15.16:0/4030575837 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:51.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa885527640 1 --2- 172.21.15.16:0/4030575837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 0x560b86931800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa8767fc640 1 -- 172.21.15.16:0/2398855036 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa870010430 con 0x7fa880109310 2024-08-28T19:32:51.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa885d28640 1 -- 172.21.15.16:0/4030575837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8656b390 msgr2=0x560b86931d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:51.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa885d28640 1 --2- 172.21.15.16:0/4030575837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8656b390 0x560b86931d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa885d28640 1 -- 172.21.15.16:0/4030575837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 msgr2=0x560b86931800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa885d28640 1 --2- 172.21.15.16:0/4030575837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 0x560b86931800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa885d28640 1 -- 172.21.15.16:0/4030575837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b865f7650 con 0x560b8658c070 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.859+0000 7fa885d28640 1 --2- 172.21.15.16:0/4030575837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8658c070 0x560b869312c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa878009f50 tx=0x7fa8780118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=578f5918aa08eca4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.860+0000 7fa86b7fe640 1 -- 172.21.15.16:0/4030575837 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878004d20 con 0x560b8658c070 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.860+0000 7fa86b7fe640 1 -- 172.21.15.16:0/4030575837 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa878004050 con 0x560b8658c070 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.860+0000 7fa86b7fe640 1 -- 172.21.15.16:0/4030575837 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878005280 con 0x560b8658c070 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.860+0000 7fa8767fc640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa86c0b4ca0 0x7fa86c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.860+0000 7fa885527640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa86c0b4ca0 0x7fa86c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.860+0000 7fa8767fc640 1 -- 172.21.15.16:0/2398855036 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fa8700e5bb0 con 0x7fa880109310 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa88dd00740 1 -- 172.21.15.16:0/4030575837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8658c070 msgr2=0x560b869312c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa885527640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa86c0b4ca0 0x7fa86c0b71b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa87c031440 tx=0x7fa87c009560 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa88dd00740 1 --2- 172.21.15.16:0/4030575837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8658c070 0x560b869312c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa878009f50 tx=0x7fa8780118d0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa88dd00740 1 -- 172.21.15.16:0/4030575837 shutdown_connections 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa88dd00740 1 --2- 172.21.15.16:0/4030575837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8656b390 0x560b86931d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa88dd00740 1 --2- 172.21.15.16:0/4030575837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 0x560b86931800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa88dd00740 1 --2- 172.21.15.16:0/4030575837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8658c070 0x560b869312c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa88dd00740 1 -- 172.21.15.16:0/4030575837 >> 172.21.15.16:0/4030575837 conn(0x560b86580460 msgr2=0x560b8692f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa88dd00740 1 -- 172.21.15.16:0/4030575837 shutdown_connections 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa88dd00740 1 -- 172.21.15.16:0/4030575837 wait complete. 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa88dd00740 1 -- start start 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.861+0000 7fa88dd00740 1 Processor -- start 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b865631b0 0x560b8693cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8656b390 0x560b8693d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b8658c070 0x560b86948d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa88dd00740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560b866fa700 con 0x560b8658c070 2024-08-28T19:32:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa88dd00740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560b8681e700 con 0x560b865631b0 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa88dd00740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560b8693faa0 con 0x560b8656b390 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa885527640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b865631b0 0x560b8693cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa885d28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b8658c070 0x560b86948d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa885527640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b865631b0 0x560b8693cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:56124/0 (socket says 172.21.15.16:56124) 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa885527640 1 -- 172.21.15.16:0/559269089 learned_addr learned my addr 172.21.15.16:0/559269089 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa884d26640 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8656b390 0x560b8693d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa88dd00740 10 client.0 _ll_register_callbacks cb 0x560b8684ddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa885527640 1 -- 172.21.15.16:0/559269089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8656b390 msgr2=0x560b8693d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa885527640 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8656b390 0x560b8693d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa885527640 1 -- 172.21.15.16:0/559269089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b8658c070 msgr2=0x560b86948d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa885527640 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b8658c070 0x560b86948d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa885527640 1 -- 172.21.15.16:0/559269089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b868578b0 con 0x560b865631b0 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.862+0000 7fa884d26640 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8656b390 0x560b8693d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:51.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.863+0000 7fa885527640 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b865631b0 0x560b8693cac0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009900 tx=0x7fa87c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77d49c9aef9a58f7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:51.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.863+0000 7fa868ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:32:51.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.863+0000 7fa868ff9640 1 -- 172.21.15.16:0/559269089 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c04d070 con 0x560b865631b0 2024-08-28T19:32:51.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.863+0000 7fa88dd00740 1 -- 172.21.15.16:0/559269089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa878004050 con 0x560b865631b0 2024-08-28T19:32:51.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.863+0000 7fa868ff9640 1 -- 172.21.15.16:0/559269089 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fa87c004450 con 0x560b865631b0 2024-08-28T19:32:51.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.863+0000 7fa868ff9640 1 -- 172.21.15.16:0/559269089 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fa87c059050 con 0x560b865631b0 2024-08-28T19:32:51.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.863+0000 7fa88dd00740 10 client.27884 fetch_fsmap learned FSMap version 59 2024-08-28T19:32:51.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.863+0000 7fa88dd00740 1 -- 172.21.15.16:0/559269089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x560b8694b900 con 0x560b865631b0 2024-08-28T19:32:51.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.863+0000 7fa868ff9640 1 -- 172.21.15.16:0/559269089 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa878004050 con 0x560b865631b0 2024-08-28T19:32:51.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.863+0000 7fa868ff9640 1 -- 172.21.15.16:0/559269089 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7fa87c004c10 con 0x560b865631b0 2024-08-28T19:32:51.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.864+0000 7fa88dd00740 10 client.27884 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:32:51.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.864+0000 7fa88dd00740 10 client.27884 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:32:51.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.864+0000 7fa88dd00740 20 client.27884 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:51.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.864+0000 7fa88dd00740 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b8654a9f0 0x560b8694f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.864+0000 7fa88dd00740 4 client.27884 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:51.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.864+0000 7fa88dd00740 1 -- 172.21.15.16:0/559269089 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b868aa580 con 0x560b8654a9f0 2024-08-28T19:32:51.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.864+0000 7fa884d26640 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b8654a9f0 0x560b8694f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.864+0000 7fa868ff9640 10 client.27884 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.864+0000 7fa884d26640 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b8654a9f0 0x560b8694f230 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98af2cc2c68d2187 server_cookie=30c9a2904879a49d in_seq=0 out_seq=0 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.865+0000 7fa868ff9640 1 -- 172.21.15.16:0/559269089 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560b868aa580 con 0x560b8654a9f0 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.865+0000 7fa868ff9640 10 client.27884 handle_command_reply: tid=0 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.865+0000 7fa88dd00740 1 client.27884 shutdown 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa88dd00740 1 -- 172.21.15.16:0/559269089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b865631b0 msgr2=0x560b8693cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa88dd00740 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b865631b0 0x560b8693cac0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009900 tx=0x7fa87c009260 comp rx=0 tx=0).stop 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa88dd00740 1 -- 172.21.15.16:0/559269089 shutdown_connections 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa88dd00740 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b8658c070 0x560b86948d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa88dd00740 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b8654a9f0 0x560b8694f230 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa88dd00740 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8656b390 0x560b8693d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa88dd00740 1 --2- 172.21.15.16:0/559269089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b865631b0 0x560b8693cac0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa88dd00740 1 -- 172.21.15.16:0/559269089 >> 172.21.15.16:0/559269089 conn(0x560b86580460 msgr2=0x560b869344a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:51.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa868ff9640 0 client.27884 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:51.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa88dd00740 1 -- 172.21.15.16:0/559269089 shutdown_connections 2024-08-28T19:32:51.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa88dd00740 1 -- 172.21.15.16:0/559269089 wait complete. 2024-08-28T19:32:51.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.866+0000 7fa88dd00740 20 client.27884 trim_cache size 0 max 16384 2024-08-28T19:32:51.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa88dd00740 1 Processor -- start 2024-08-28T19:32:51.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa88dd00740 1 -- start start 2024-08-28T19:32:51.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 0x560b86948c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8656b390 0x560b869491d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8658c070 0x560b8693c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa88dd00740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8801aeb20 con 0x560b865631b0 2024-08-28T19:32:51.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa88dd00740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8801b0780 con 0x560b8656b390 2024-08-28T19:32:51.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa88dd00740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa8801af2b0 con 0x560b8658c070 2024-08-28T19:32:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa885527640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 0x560b86948c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa885527640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 0x560b86948c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59864/0 (socket says 172.21.15.16:59864) 2024-08-28T19:32:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa885527640 1 -- 172.21.15.16:0/783184132 learned_addr learned my addr 172.21.15.16:0/783184132 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa884d26640 1 --2- 172.21.15.16:0/783184132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8656b390 0x560b869491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa885d28640 1 --2- 172.21.15.16:0/783184132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8658c070 0x560b8693c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa885527640 1 -- 172.21.15.16:0/783184132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8658c070 msgr2=0x560b8693c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa885527640 1 --2- 172.21.15.16:0/783184132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8658c070 0x560b8693c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa885527640 1 -- 172.21.15.16:0/783184132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8656b390 msgr2=0x560b869491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa885527640 1 --2- 172.21.15.16:0/783184132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8656b390 0x560b869491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa885527640 1 -- 172.21.15.16:0/783184132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b869335b0 con 0x560b865631b0 2024-08-28T19:32:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.886+0000 7fa884d26640 1 --2- 172.21.15.16:0/783184132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8656b390 0x560b869491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.887+0000 7fa885527640 1 --2- 172.21.15.16:0/783184132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 0x560b86948c90 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa87c003760 tx=0x7fa87c032c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab666711f25abe35 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.887+0000 7fa86affd640 1 -- 172.21.15.16:0/783184132 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c061070 con 0x560b865631b0 2024-08-28T19:32:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.887+0000 7fa86affd640 1 -- 172.21.15.16:0/783184132 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa87c005850 con 0x560b865631b0 2024-08-28T19:32:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.887+0000 7fa86affd640 1 -- 172.21.15.16:0/783184132 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c04d050 con 0x560b865631b0 2024-08-28T19:32:51.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.888+0000 7fa88dd00740 1 -- 172.21.15.16:0/783184132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 msgr2=0x560b86948c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:51.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.888+0000 7fa88dd00740 1 --2- 172.21.15.16:0/783184132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 0x560b86948c90 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa87c003760 tx=0x7fa87c032c70 comp rx=0 tx=0).stop 2024-08-28T19:32:51.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.888+0000 7fa88dd00740 1 -- 172.21.15.16:0/783184132 shutdown_connections 2024-08-28T19:32:51.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.888+0000 7fa88dd00740 1 --2- 172.21.15.16:0/783184132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8658c070 0x560b8693c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.888+0000 7fa88dd00740 1 --2- 172.21.15.16:0/783184132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8656b390 0x560b869491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.888+0000 7fa88dd00740 1 --2- 172.21.15.16:0/783184132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b865631b0 0x560b86948c90 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.888+0000 7fa88dd00740 1 -- 172.21.15.16:0/783184132 >> 172.21.15.16:0/783184132 conn(0x560b86580460 msgr2=0x560b8693b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.888+0000 7fa88dd00740 1 -- 172.21.15.16:0/783184132 shutdown_connections 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.888+0000 7fa88dd00740 1 -- 172.21.15.16:0/783184132 wait complete. 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.889+0000 7fa88dd00740 1 -- start start 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.889+0000 7fa88dd00740 1 Processor -- start 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.889+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b8656b390 0x560b86940d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.889+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8658c070 0x560b86941290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.889+0000 7fa88dd00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8651fda0 0x560b869417d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.889+0000 7fa88dd00740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8801b0080 con 0x560b8656b390 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.889+0000 7fa88dd00740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8801af980 con 0x560b8651fda0 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.889+0000 7fa88dd00740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa8801acb20 con 0x560b8658c070 2024-08-28T19:32:51.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa88dd00740 10 client.0 _ll_register_callbacks cb 0x7fa87c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa885d28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8651fda0 0x560b869417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa884d26640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8658c070 0x560b86941290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa885d28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8651fda0 0x560b869417d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:56134/0 (socket says 172.21.15.16:56134) 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa885d28640 1 -- 172.21.15.16:0/1759697339 learned_addr learned my addr 172.21.15.16:0/1759697339 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa885527640 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b8656b390 0x560b86940d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa884d26640 1 -- 172.21.15.16:0/1759697339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8651fda0 msgr2=0x560b869417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa884d26640 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8651fda0 0x560b869417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa884d26640 1 -- 172.21.15.16:0/1759697339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b8656b390 msgr2=0x560b86940d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa884d26640 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b8656b390 0x560b86940d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa884d26640 1 -- 172.21.15.16:0/1759697339 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b869b8cb0 con 0x560b8658c070 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa884d26640 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8658c070 0x560b86941290 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa8700e2f30 tx=0x7fa8700107c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3634707780b663f1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa868ff9640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa868ff9640 1 -- 172.21.15.16:0/1759697339 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8700b51f0 con 0x560b8658c070 2024-08-28T19:32:51.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa88dd00740 1 -- 172.21.15.16:0/1759697339 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa87c005850 con 0x560b8658c070 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.890+0000 7fa868ff9640 1 -- 172.21.15.16:0/1759697339 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fa8700e31b0 con 0x560b8658c070 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa868ff9640 1 -- 172.21.15.16:0/1759697339 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fa8700e33b0 con 0x560b8658c070 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa88dd00740 10 client.27610 fetch_fsmap learned FSMap version 59 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa88dd00740 1 -- 172.21.15.16:0/1759697339 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x560b869b9f90 con 0x560b8658c070 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa868ff9640 1 -- 172.21.15.16:0/1759697339 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa87c005850 con 0x560b8658c070 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa868ff9640 1 -- 172.21.15.16:0/1759697339 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7fa8700a78f0 con 0x560b8658c070 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa88dd00740 10 client.27610 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa88dd00740 10 client.27610 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa88dd00740 20 client.27610 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa88dd00740 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b869ba670 0x560b869bca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa88dd00740 4 client.27610 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa88dd00740 1 -- 172.21.15.16:0/1759697339 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560b868a6020 con 0x560b869ba670 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.891+0000 7fa885527640 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b869ba670 0x560b869bca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:51.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.892+0000 7fa868ff9640 10 client.27610 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:51.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.892+0000 7fa885527640 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b869ba670 0x560b869bca90 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3d4d979a6d37365 server_cookie=bf044b6569d58cf7 in_seq=0 out_seq=0 2024-08-28T19:32:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.892+0000 7fa868ff9640 1 -- 172.21.15.16:0/1759697339 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4634 (crc 0 0 0) 0x560b868a6020 con 0x560b869ba670 2024-08-28T19:32:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.892+0000 7fa868ff9640 10 client.27610 handle_command_reply: tid=0 2024-08-28T19:32:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.892+0000 7fa88dd00740 1 client.27610 shutdown 2024-08-28T19:32:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa88dd00740 1 -- 172.21.15.16:0/1759697339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8658c070 msgr2=0x560b86941290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa88dd00740 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8658c070 0x560b86941290 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa8700e2f30 tx=0x7fa8700107c0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa88dd00740 1 -- 172.21.15.16:0/1759697339 shutdown_connections 2024-08-28T19:32:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa88dd00740 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b8651fda0 0x560b869417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa88dd00740 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560b869ba670 0x560b869bca90 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa868ff9640 0 client.27610 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:32:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa88dd00740 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560b8658c070 0x560b86941290 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa88dd00740 1 --2- 172.21.15.16:0/1759697339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560b8656b390 0x560b86940d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa88dd00740 1 -- 172.21.15.16:0/1759697339 >> 172.21.15.16:0/1759697339 conn(0x560b86580460 msgr2=0x560b8692e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa88dd00740 1 -- 172.21.15.16:0/1759697339 shutdown_connections 2024-08-28T19:32:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa88dd00740 1 -- 172.21.15.16:0/1759697339 wait complete. 2024-08-28T19:32:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.893+0000 7fa88dd00740 20 client.27610 trim_cache size 0 max 16384 2024-08-28T19:32:51.899 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:32:51.899 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:32:51.899 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:32:51.900 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.902 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.903 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.903 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.903 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:32:51.903 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:32:51.903 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.903 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.903 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.903 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.903 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:32:51.905 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:32:51.906 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:32:51.908 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:32:51.909 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:32:51.909 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:32:51.909 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:32:51.909 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:32:51.909 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:32:51.909 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:32:51.909 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:32:51.909 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:32:51.909 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 -- 172.21.15.16:0/2398855036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa86c0b4ca0 msgr2=0x7fa86c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa86c0b4ca0 0x7fa86c0b71b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa87c031440 tx=0x7fa87c009560 comp rx=0 tx=0).stop 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 -- 172.21.15.16:0/2398855036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa880109310 msgr2=0x7fa88019a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa880109310 0x7fa88019a910 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa87000e9f0 tx=0x7fa87000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 -- 172.21.15.16:0/2398855036 shutdown_connections 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa880109c60 0x7fa88019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa86c0b4ca0 0x7fa86c0b71b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa880109310 0x7fa88019a910 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 --2- 172.21.15.16:0/2398855036 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8801039d0 0x7fa88019a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 -- 172.21.15.16:0/2398855036 >> 172.21.15.16:0/2398855036 conn(0x7fa88007b670 msgr2=0x7fa8801074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 -- 172.21.15.16:0/2398855036 shutdown_connections 2024-08-28T19:32:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:51.894+0000 7fa88778b640 1 -- 172.21.15.16:0/2398855036 wait complete. 2024-08-28T19:32:51.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 485659 3.74 MB/sec execute 511 sec latency 2602.118 ms 2024-08-28T19:32:52.864 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:32:52.865 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:32:52.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 487217 3.75 MB/sec execute 512 sec latency 12.906 ms 2024-08-28T19:32:53.133 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:53.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:53 smithi016 ceph-mon[18556]: pgmap v704: 129 pgs: 129 active+clean; 386 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 159 op/s 2024-08-28T19:32:53.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:53 smithi115 ceph-mon[22184]: pgmap v704: 129 pgs: 129 active+clean; 386 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 159 op/s 2024-08-28T19:32:53.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.662+0000 7f8c815a9640 1 -- 172.21.15.16:0/2090846672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c7c109240 msgr2=0x7f8c7c109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.662+0000 7f8c815a9640 1 --2- 172.21.15.16:0/2090846672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c7c109240 0x7f8c7c109650 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009900 tx=0x7f8c7002a410 comp rx=0 tx=0).stop 2024-08-28T19:32:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.662+0000 7f8c815a9640 1 -- 172.21.15.16:0/2090846672 shutdown_connections 2024-08-28T19:32:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.662+0000 7f8c815a9640 1 --2- 172.21.15.16:0/2090846672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c7c109b90 0x7f8c7c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.662+0000 7f8c815a9640 1 --2- 172.21.15.16:0/2090846672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c7c109240 0x7f8c7c109650 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.662+0000 7f8c815a9640 1 --2- 172.21.15.16:0/2090846672 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c7c103900 0x7f8c7c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.662+0000 7f8c815a9640 1 -- 172.21.15.16:0/2090846672 >> 172.21.15.16:0/2090846672 conn(0x7f8c7c07b5f0 msgr2=0x7f8c7c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.663+0000 7f8c815a9640 1 -- 172.21.15.16:0/2090846672 shutdown_connections 2024-08-28T19:32:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.663+0000 7f8c815a9640 1 -- 172.21.15.16:0/2090846672 wait complete. 2024-08-28T19:32:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.663+0000 7f8c815a9640 1 Processor -- start 2024-08-28T19:32:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.663+0000 7f8c815a9640 1 -- start start 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c815a9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c7c103900 0x7f8c7c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c815a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c7c109240 0x7f8c7c19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c815a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c7c109b90 0x7f8c7c19fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c815a9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c113850 con 0x7f8c7c109240 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c815a9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c1136d0 con 0x7f8c7c109b90 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c815a9640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c1139d0 con 0x7f8c7c103900 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c7affd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c7c103900 0x7f8c7c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c7affd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c7c103900 0x7f8c7c19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:35736/0 (socket says 172.21.15.16:35736) 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c7affd640 1 -- 172.21.15.16:0/3604547974 learned_addr learned my addr 172.21.15.16:0/3604547974 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c7b7fe640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c7c109b90 0x7f8c7c19fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c7a7fc640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c7c109240 0x7f8c7c19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c7affd640 1 -- 172.21.15.16:0/3604547974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c7c109b90 msgr2=0x7f8c7c19fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c7affd640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c7c109b90 0x7f8c7c19fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c7affd640 1 -- 172.21.15.16:0/3604547974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c7c109240 msgr2=0x7f8c7c19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c7affd640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c7c109240 0x7f8c7c19a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c7affd640 1 -- 172.21.15.16:0/3604547974 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7c1a01b0 con 0x7f8c7c103900 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.664+0000 7f8c7b7fe640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c7c109b90 0x7f8c7c19fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.665+0000 7f8c7affd640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c7c103900 0x7f8c7c19a240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00eb90 tx=0x7f8c6c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.665+0000 7f8c63fff640 1 -- 172.21.15.16:0/3604547974 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c004d20 con 0x7f8c7c103900 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.665+0000 7f8c815a9640 1 -- 172.21.15.16:0/3604547974 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7c1a04a0 con 0x7f8c7c103900 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.665+0000 7f8c63fff640 1 -- 172.21.15.16:0/3604547974 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c6c00ee90 con 0x7f8c7c103900 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.665+0000 7f8c63fff640 1 -- 172.21.15.16:0/3604547974 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c010670 con 0x7f8c7c103900 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.665+0000 7f8c815a9640 1 -- 172.21.15.16:0/3604547974 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c7c1a6c00 con 0x7f8c7c103900 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.666+0000 7f8c815a9640 1 -- 172.21.15.16:0/3604547974 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c7c103d10 con 0x7f8c7c103900 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.668+0000 7f8c63fff640 1 -- 172.21.15.16:0/3604547974 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8c6c010830 con 0x7f8c7c103900 2024-08-28T19:32:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.669+0000 7f8c63fff640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8c640b4ca0 0x7f8c640b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.669+0000 7f8c7a7fc640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8c640b4ca0 0x7f8c640b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.669+0000 7f8c63fff640 1 -- 172.21.15.16:0/3604547974 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f8c6c0e6160 con 0x7f8c7c103900 2024-08-28T19:32:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.669+0000 7f8c7a7fc640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8c640b4ca0 0x7f8c640b71b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009230 tx=0x7f8c70031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.672+0000 7f8c63fff640 1 -- 172.21.15.16:0/3604547974 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8c6c056130 con 0x7f8c7c103900 2024-08-28T19:32:53.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:53 smithi028 ceph-mon[21769]: pgmap v704: 129 pgs: 129 active+clean; 386 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 159 op/s 2024-08-28T19:32:53.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.906+0000 7f8c815a9640 1 -- 172.21.15.16:0/3604547974 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c7c19b370 con 0x7f8c7c103900 2024-08-28T19:32:53.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.907+0000 7f8c63fff640 1 -- 172.21.15.16:0/3604547974 <== mon.1 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+11986 (secure 0 0 0) 0x7f8c7c19b370 con 0x7f8c7c103900 2024-08-28T19:32:53.908 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:32:53.908 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":59,"btime":"2024-08-28T19:32:24:272249+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":59,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:32:23.285071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:32:53.908 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 59 2024-08-28T19:32:53.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 -- 172.21.15.16:0/3604547974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8c640b4ca0 msgr2=0x7f8c640b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:53.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8c640b4ca0 0x7f8c640b71b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009230 tx=0x7f8c70031400 comp rx=0 tx=0).stop 2024-08-28T19:32:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 -- 172.21.15.16:0/3604547974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c7c103900 msgr2=0x7f8c7c19a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c7c103900 0x7f8c7c19a240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00eb90 tx=0x7f8c6c00c6a0 comp rx=0 tx=0).stop 2024-08-28T19:32:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 -- 172.21.15.16:0/3604547974 shutdown_connections 2024-08-28T19:32:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8c640b4ca0 0x7f8c640b71b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c7c109b90 0x7f8c7c19fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c7c109240 0x7f8c7c19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 --2- 172.21.15.16:0/3604547974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8c7c103900 0x7f8c7c19a240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 -- 172.21.15.16:0/3604547974 >> 172.21.15.16:0/3604547974 conn(0x7f8c7c07b5f0 msgr2=0x7f8c7c107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 -- 172.21.15.16:0/3604547974 shutdown_connections 2024-08-28T19:32:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:53.909+0000 7f8c815a9640 1 -- 172.21.15.16:0/3604547974 wait complete. 2024-08-28T19:32:53.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 488453 3.75 MB/sec execute 513 sec latency 232.054 ms 2024-08-28T19:32:54.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:54 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3604547974' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:32:54.581 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:32:54.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:54 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3604547974' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:32:54.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:54 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3604547974' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:32:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:54.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 488453 3.75 MB/sec execute 514 sec latency 1232.102 ms 2024-08-28T19:32:55.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.283+0000 7fae5161e640 1 -- 172.21.15.16:0/1603798603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4c109b90 msgr2=0x7fae4c1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.283+0000 7fae5161e640 1 --2- 172.21.15.16:0/1603798603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4c109b90 0x7fae4c1103c0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fae44009ca0 tx=0x7fae4402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.283+0000 7fae5161e640 1 -- 172.21.15.16:0/1603798603 shutdown_connections 2024-08-28T19:32:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.283+0000 7fae5161e640 1 --2- 172.21.15.16:0/1603798603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4c109b90 0x7fae4c1103c0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.283+0000 7fae5161e640 1 --2- 172.21.15.16:0/1603798603 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae4c109240 0x7fae4c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.283+0000 7fae5161e640 1 --2- 172.21.15.16:0/1603798603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c103900 0x7fae4c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.283+0000 7fae5161e640 1 -- 172.21.15.16:0/1603798603 >> 172.21.15.16:0/1603798603 conn(0x7fae4c07b5f0 msgr2=0x7fae4c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.283+0000 7fae5161e640 1 -- 172.21.15.16:0/1603798603 shutdown_connections 2024-08-28T19:32:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.283+0000 7fae5161e640 1 -- 172.21.15.16:0/1603798603 wait complete. 2024-08-28T19:32:55.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.284+0000 7fae5161e640 1 Processor -- start 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.284+0000 7fae5161e640 1 -- start start 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.284+0000 7fae5161e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae4c103900 0x7fae4c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.284+0000 7fae5161e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c109240 0x7fae4c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.284+0000 7fae5161e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4c109b90 0x7fae4c19fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.284+0000 7fae5161e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c113850 con 0x7fae4c109b90 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.284+0000 7fae5161e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1136d0 con 0x7fae4c109240 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.284+0000 7fae5161e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1139d0 con 0x7fae4c103900 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.284+0000 7fae4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4c109b90 0x7fae4c19fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4c109b90 0x7fae4c19fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56436/0 (socket says 172.21.15.16:56436) 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae4b7fe640 1 -- 172.21.15.16:0/1336008107 learned_addr learned my addr 172.21.15.16:0/1336008107 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c109240 0x7fae4c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae4affd640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae4c103900 0x7fae4c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae4b7fe640 1 -- 172.21.15.16:0/1336008107 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae4c103900 msgr2=0x7fae4c19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:55.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae4b7fe640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae4c103900 0x7fae4c19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae4b7fe640 1 -- 172.21.15.16:0/1336008107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c109240 msgr2=0x7fae4c19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:55.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae4b7fe640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c109240 0x7fae4c19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae4b7fe640 1 -- 172.21.15.16:0/1336008107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae4c1a0170 con 0x7fae4c109b90 2024-08-28T19:32:55.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c109240 0x7fae4c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:32:55.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae4b7fe640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4c109b90 0x7fae4c19fa40 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fae440098d0 tx=0x7fae440317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:55.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae33fff640 1 -- 172.21.15.16:0/1336008107 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae44003b70 con 0x7fae4c109b90 2024-08-28T19:32:55.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae5161e640 1 -- 172.21.15.16:0/1336008107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae4c1a0400 con 0x7fae4c109b90 2024-08-28T19:32:55.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae33fff640 1 -- 172.21.15.16:0/1336008107 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae44003d10 con 0x7fae4c109b90 2024-08-28T19:32:55.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae33fff640 1 -- 172.21.15.16:0/1336008107 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4402fa20 con 0x7fae4c109b90 2024-08-28T19:32:55.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.285+0000 7fae5161e640 1 -- 172.21.15.16:0/1336008107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae4c1a6bc0 con 0x7fae4c109b90 2024-08-28T19:32:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.288+0000 7fae57b93740 1 Processor -- start 2024-08-28T19:32:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.288+0000 7fae33fff640 1 -- 172.21.15.16:0/1336008107 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fae4402fc40 con 0x7fae4c109b90 2024-08-28T19:32:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.288+0000 7fae57b93740 1 -- start start 2024-08-28T19:32:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.288+0000 7fae57b93740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d7a070 0x55d90811f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae57b93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90811f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae57b93740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d59390 0x55d90811fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae57b93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d908096680 con 0x55d907d511b0 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae57b93740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d908085c10 con 0x55d907d7a070 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae57b93740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d908089bc0 con 0x55d907d59390 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90811f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90811f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56438/0 (socket says 172.21.15.16:56438) 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4affd640 1 -- 172.21.15.16:0/1980858301 learned_addr learned my addr 172.21.15.16:0/1980858301 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/1980858301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d59390 0x55d90811fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/1980858301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d7a070 0x55d90811f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae33fff640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae340b4ca0 0x7fae340b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4affd640 1 -- 172.21.15.16:0/1980858301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d59390 msgr2=0x55d90811fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4affd640 1 --2- 172.21.15.16:0/1980858301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d59390 0x55d90811fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4affd640 1 -- 172.21.15.16:0/1980858301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d7a070 msgr2=0x55d90811f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4affd640 1 --2- 172.21.15.16:0/1980858301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d7a070 0x55d90811f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4affd640 1 -- 172.21.15.16:0/1980858301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d907de5650 con 0x55d907d511b0 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4b7fe640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae340b4ca0 0x7fae340b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae33fff640 1 -- 172.21.15.16:0/1336008107 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fae44106d00 con 0x7fae4c109b90 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.289+0000 7fae4affd640 1 --2- 172.21.15.16:0/1980858301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90811f850 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00bcb0 tx=0x7fae3c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d68532a22b78c13 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.290+0000 7fae317fa640 1 -- 172.21.15.16:0/1980858301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c015070 con 0x55d907d511b0 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.290+0000 7fae317fa640 1 -- 172.21.15.16:0/1980858301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae3c0118f0 con 0x55d907d511b0 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.290+0000 7fae317fa640 1 -- 172.21.15.16:0/1980858301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c0108b0 con 0x55d907d511b0 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.290+0000 7fae57b93740 1 -- 172.21.15.16:0/1980858301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 msgr2=0x55d90811f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.290+0000 7fae4b7fe640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae340b4ca0 0x7fae340b71b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fae44104130 tx=0x7fae440b4700 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.290+0000 7fae57b93740 1 --2- 172.21.15.16:0/1980858301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90811f850 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00bcb0 tx=0x7fae3c00f730 comp rx=0 tx=0).stop 2024-08-28T19:32:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 -- 172.21.15.16:0/1980858301 shutdown_connections 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 --2- 172.21.15.16:0/1980858301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d59390 0x55d90811fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 --2- 172.21.15.16:0/1980858301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90811f850 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 --2- 172.21.15.16:0/1980858301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d7a070 0x55d90811f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 -- 172.21.15.16:0/1980858301 >> 172.21.15.16:0/1980858301 conn(0x55d907d6e460 msgr2=0x55d90811d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 -- 172.21.15.16:0/1980858301 shutdown_connections 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 -- 172.21.15.16:0/1980858301 wait complete. 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 -- start start 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 Processor -- start 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90812ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d59390 0x55d90812f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d7a070 0x55d90813b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d907ee8700 con 0x55d907d511b0 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d90800c750 con 0x55d907d59390 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d908131e20 con 0x55d907d7a070 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.291+0000 7fae57b93740 10 client.0 _ll_register_callbacks cb 0x55d90803be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae4a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90812ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae4a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90812ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56454/0 (socket says 172.21.15.16:56454) 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae4a7fc640 1 -- 172.21.15.16:0/2659874258 learned_addr learned my addr 172.21.15.16:0/2659874258 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae4affd640 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d59390 0x55d90812f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d7a070 0x55d90813b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae4a7fc640 1 -- 172.21.15.16:0/2659874258 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d7a070 msgr2=0x55d90813b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d7a070 0x55d90813b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae4a7fc640 1 -- 172.21.15.16:0/2659874258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d59390 msgr2=0x55d90812f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d59390 0x55d90812f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae4a7fc640 1 -- 172.21.15.16:0/2659874258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d908045900 con 0x55d907d511b0 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90812ee30 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fae40013b10 tx=0x7fae400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=111c4ef66e9749f5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae2eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae2eff5640 1 -- 172.21.15.16:0/2659874258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4000ec10 con 0x55d907d511b0 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae2eff5640 1 -- 172.21.15.16:0/2659874258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fae4000edb0 con 0x55d907d511b0 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.292+0000 7fae57b93740 1 -- 172.21.15.16:0/2659874258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae3c0118f0 con 0x55d907d511b0 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae2eff5640 1 -- 172.21.15.16:0/2659874258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fae40005020 con 0x55d907d511b0 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae57b93740 10 client.18453 fetch_fsmap learned FSMap version 59 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae57b93740 1 -- 172.21.15.16:0/2659874258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55d90813dc30 con 0x55d907d511b0 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae2eff5640 1 -- 172.21.15.16:0/2659874258 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae3c0118f0 con 0x55d907d511b0 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae2eff5640 1 -- 172.21.15.16:0/2659874258 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7fae40020450 con 0x55d907d511b0 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae57b93740 10 client.18453 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:32:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae57b93740 10 client.18453 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:32:55.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae57b93740 20 client.18453 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:55.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae57b93740 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d907d389f0 0x55d908141190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae57b93740 4 client.18453 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:55.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae57b93740 1 -- 172.21.15.16:0/2659874258 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9080985d0 con 0x55d907d389f0 2024-08-28T19:32:55.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.293+0000 7fae4affd640 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d907d389f0 0x55d908141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.294+0000 7fae2eff5640 10 client.18453 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:32:55.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.294+0000 7fae4affd640 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d907d389f0 0x55d908141190 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d3715f1ca2747d1d server_cookie=bf649f8cd0be1745 in_seq=0 out_seq=0 2024-08-28T19:32:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae2eff5640 1 -- 172.21.15.16:0/2659874258 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d9080985d0 con 0x55d907d389f0 2024-08-28T19:32:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae2eff5640 10 client.18453 handle_command_reply: tid=0 2024-08-28T19:32:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 1 client.18453 shutdown 2024-08-28T19:32:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 1 -- 172.21.15.16:0/2659874258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 msgr2=0x55d90812ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90812ee30 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fae40013b10 tx=0x7fae400046c0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 1 -- 172.21.15.16:0/2659874258 shutdown_connections 2024-08-28T19:32:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d907d389f0 0x55d908141190 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d7a070 0x55d90813b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d59390 0x55d90812f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 1 --2- 172.21.15.16:0/2659874258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d90812ee30 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 1 -- 172.21.15.16:0/2659874258 >> 172.21.15.16:0/2659874258 conn(0x55d907d6e460 msgr2=0x55d90811b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae2eff5640 0 client.18453 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:32:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 1 -- 172.21.15.16:0/2659874258 shutdown_connections 2024-08-28T19:32:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 1 -- 172.21.15.16:0/2659874258 wait complete. 2024-08-28T19:32:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.295+0000 7fae57b93740 20 client.18453 trim_cache size 0 max 16384 2024-08-28T19:32:55.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.308+0000 7fae57b93740 1 Processor -- start 2024-08-28T19:32:55.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.308+0000 7fae57b93740 1 -- start start 2024-08-28T19:32:55.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae57b93740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d511b0 0x55d908118230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae57b93740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d59390 0x55d90812c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae57b93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d7a070 0x55d90812c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae57b93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1aeb10 con 0x55d907d7a070 2024-08-28T19:32:55.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae57b93740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1b0770 con 0x55d907d511b0 2024-08-28T19:32:55.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae57b93740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1af2a0 con 0x55d907d59390 2024-08-28T19:32:55.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae4a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d7a070 0x55d90812c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae4a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d7a070 0x55d90812c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56468/0 (socket says 172.21.15.16:56468) 2024-08-28T19:32:55.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae4a7fc640 1 -- 172.21.15.16:0/816656047 learned_addr learned my addr 172.21.15.16:0/816656047 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:55.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae4affd640 1 --2- 172.21.15.16:0/816656047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d59390 0x55d90812c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/816656047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d511b0 0x55d908118230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae4a7fc640 1 -- 172.21.15.16:0/816656047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d59390 msgr2=0x55d90812c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:55.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/816656047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d59390 0x55d90812c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae4a7fc640 1 -- 172.21.15.16:0/816656047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d511b0 msgr2=0x55d908118230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:55.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/816656047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d511b0 0x55d908118230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae4a7fc640 1 -- 172.21.15.16:0/816656047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d90812a990 con 0x55d907d7a070 2024-08-28T19:32:55.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/816656047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d7a070 0x55d90812c560 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fae40020820 tx=0x7fae40005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf9da86650f6226 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:55.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae30ff9640 1 -- 172.21.15.16:0/816656047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae40020450 con 0x55d907d7a070 2024-08-28T19:32:55.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae30ff9640 1 -- 172.21.15.16:0/816656047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae400205f0 con 0x55d907d7a070 2024-08-28T19:32:55.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.309+0000 7fae30ff9640 1 -- 172.21.15.16:0/816656047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae400055c0 con 0x55d907d7a070 2024-08-28T19:32:55.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.310+0000 7fae57b93740 1 -- 172.21.15.16:0/816656047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d7a070 msgr2=0x55d90812c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:55.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.310+0000 7fae57b93740 1 --2- 172.21.15.16:0/816656047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d7a070 0x55d90812c560 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fae40020820 tx=0x7fae40005f90 comp rx=0 tx=0).stop 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.310+0000 7fae57b93740 1 -- 172.21.15.16:0/816656047 shutdown_connections 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.310+0000 7fae57b93740 1 --2- 172.21.15.16:0/816656047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d7a070 0x55d90812c560 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.310+0000 7fae57b93740 1 --2- 172.21.15.16:0/816656047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d59390 0x55d90812c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.310+0000 7fae57b93740 1 --2- 172.21.15.16:0/816656047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d511b0 0x55d908118230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.310+0000 7fae57b93740 1 -- 172.21.15.16:0/816656047 >> 172.21.15.16:0/816656047 conn(0x55d907d6e460 msgr2=0x55d90812d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.310+0000 7fae57b93740 1 -- 172.21.15.16:0/816656047 shutdown_connections 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.310+0000 7fae57b93740 1 -- 172.21.15.16:0/816656047 wait complete. 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae57b93740 1 -- start start 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae57b93740 1 Processor -- start 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae57b93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d908133730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae57b93740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d59390 0x55d908133c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae57b93740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d7a070 0x55d9081aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae57b93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1b0070 con 0x55d907d511b0 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae57b93740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1af970 con 0x55d907d59390 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae57b93740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1acb10 con 0x55d907d7a070 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae57b93740 10 client.0 _ll_register_callbacks cb 0x7fae40013930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae4a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d908133730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae4a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d908133730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56476/0 (socket says 172.21.15.16:56476) 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae4a7fc640 1 -- 172.21.15.16:0/1888774577 learned_addr learned my addr 172.21.15.16:0/1888774577 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.311+0000 7fae4affd640 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d59390 0x55d908133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae4a7fc640 1 -- 172.21.15.16:0/1888774577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d7a070 msgr2=0x55d9081aabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d7a070 0x55d9081aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae4a7fc640 1 -- 172.21.15.16:0/1888774577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d59390 msgr2=0x55d908133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d59390 0x55d908133c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae4a7fc640 1 -- 172.21.15.16:0/1888774577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9081ab120 con 0x55d907d511b0 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae4a7fc640 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d908133730 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fae40009590 tx=0x7fae40005980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dac378ef436fee5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae2eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae2eff5640 1 -- 172.21.15.16:0/1888774577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4003d060 con 0x55d907d511b0 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae2eff5640 1 -- 172.21.15.16:0/1888774577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fae40020a90 con 0x55d907d511b0 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae57b93740 1 -- 172.21.15.16:0/1888774577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae400205f0 con 0x55d907d511b0 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae2eff5640 1 -- 172.21.15.16:0/1888774577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fae40020450 con 0x55d907d511b0 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae57b93740 10 client.18459 fetch_fsmap learned FSMap version 59 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae57b93740 1 -- 172.21.15.16:0/1888774577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55d9081ac330 con 0x55d907d511b0 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.312+0000 7fae2eff5640 1 -- 172.21.15.16:0/1888774577 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae400205f0 con 0x55d907d511b0 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae2eff5640 1 -- 172.21.15.16:0/1888774577 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7fae40011440 con 0x55d907d511b0 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae57b93740 10 client.18459 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae57b93740 10 client.18459 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae57b93740 20 client.18459 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:55.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae57b93740 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d907d0dda0 0x55d9081aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae57b93740 4 client.18459 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae57b93740 1 -- 172.21.15.16:0/1888774577 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d908094070 con 0x55d907d0dda0 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae4affd640 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d907d0dda0 0x55d9081aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae2eff5640 10 client.18459 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae4affd640 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d907d0dda0 0x55d9081aebb0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f88f8315c993f775 server_cookie=18c5c9e5df7d698c in_seq=0 out_seq=0 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae2eff5640 1 -- 172.21.15.16:0/1888774577 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d908094070 con 0x55d907d0dda0 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.313+0000 7fae2eff5640 10 client.18459 handle_command_reply: tid=0 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 1 client.18459 shutdown 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 1 -- 172.21.15.16:0/1888774577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 msgr2=0x55d908133730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d908133730 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fae40009590 tx=0x7fae40005980 comp rx=0 tx=0).stop 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 1 -- 172.21.15.16:0/1888774577 shutdown_connections 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55d907d0dda0 0x55d9081aebb0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d907d7a070 0x55d9081aabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d907d59390 0x55d908133c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae2eff5640 0 client.18459 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:32:55.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 1 --2- 172.21.15.16:0/1888774577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d907d511b0 0x55d908133730 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 1 -- 172.21.15.16:0/1888774577 >> 172.21.15.16:0/1888774577 conn(0x55d907d6e460 msgr2=0x55d9081243d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:55.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 1 -- 172.21.15.16:0/1888774577 shutdown_connections 2024-08-28T19:32:55.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 1 -- 172.21.15.16:0/1888774577 wait complete. 2024-08-28T19:32:55.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae57b93740 20 client.18459 trim_cache size 0 max 16384 2024-08-28T19:32:55.321 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:32:55.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae5161e640 1 -- 172.21.15.16:0/1336008107 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae340b4ca0 msgr2=0x7fae340b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:55.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae5161e640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae340b4ca0 0x7fae340b71b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fae44104130 tx=0x7fae440b4700 comp rx=0 tx=0).stop 2024-08-28T19:32:55.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae5161e640 1 -- 172.21.15.16:0/1336008107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4c109b90 msgr2=0x7fae4c19fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:55.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae5161e640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4c109b90 0x7fae4c19fa40 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fae440098d0 tx=0x7fae440317b0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae5161e640 1 -- 172.21.15.16:0/1336008107 shutdown_connections 2024-08-28T19:32:55.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae5161e640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4c109b90 0x7fae4c19fa40 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae5161e640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae340b4ca0 0x7fae340b71b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae5161e640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c109240 0x7fae4c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae5161e640 1 --2- 172.21.15.16:0/1336008107 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae4c103900 0x7fae4c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:55.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.314+0000 7fae5161e640 1 -- 172.21.15.16:0/1336008107 >> 172.21.15.16:0/1336008107 conn(0x7fae4c07b5f0 msgr2=0x7fae4c107290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:55.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.315+0000 7fae5161e640 1 -- 172.21.15.16:0/1336008107 shutdown_connections 2024-08-28T19:32:55.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:55.315+0000 7fae5161e640 1 -- 172.21.15.16:0/1336008107 wait complete. 2024-08-28T19:32:55.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:55 smithi115 ceph-mon[22184]: pgmap v705: 129 pgs: 129 active+clean; 385 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-08-28T19:32:55.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:55 smithi016 ceph-mon[18556]: pgmap v705: 129 pgs: 129 active+clean; 385 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-08-28T19:32:55.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:55 smithi028 ceph-mon[21769]: pgmap v705: 129 pgs: 129 active+clean; 385 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-08-28T19:32:55.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 488453 3.74 MB/sec execute 515 sec latency 2232.174 ms 2024-08-28T19:32:56.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:32:56.255 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.886+0000 7f8de0f02640 1 -- 172.21.15.16:0/2812331307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ddc103920 msgr2=0x7f8ddc103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.886+0000 7f8de0f02640 1 --2- 172.21.15.16:0/2812331307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ddc103920 0x7f8ddc103d30 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009900 tx=0x7f8dcc02a410 comp rx=0 tx=0).stop 2024-08-28T19:32:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.887+0000 7f8de0f02640 1 -- 172.21.15.16:0/2812331307 shutdown_connections 2024-08-28T19:32:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.887+0000 7f8de0f02640 1 --2- 172.21.15.16:0/2812331307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc109bb0 0x7f8ddc1103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.887+0000 7f8de0f02640 1 --2- 172.21.15.16:0/2812331307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ddc109260 0x7f8ddc109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.887+0000 7f8de0f02640 1 --2- 172.21.15.16:0/2812331307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ddc103920 0x7f8ddc103d30 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.887+0000 7f8de0f02640 1 -- 172.21.15.16:0/2812331307 >> 172.21.15.16:0/2812331307 conn(0x7f8ddc0ff370 msgr2=0x7f8ddc1017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.887+0000 7f8de0f02640 1 -- 172.21.15.16:0/2812331307 shutdown_connections 2024-08-28T19:32:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.887+0000 7f8de0f02640 1 -- 172.21.15.16:0/2812331307 wait complete. 2024-08-28T19:32:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.887+0000 7f8de0f02640 1 Processor -- start 2024-08-28T19:32:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8de0f02640 1 -- start start 2024-08-28T19:32:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8de0f02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc103920 0x7f8ddc0754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8de0f02640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ddc109260 0x7f8ddc0759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8de0f02640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ddc109bb0 0x7f8ddc075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8de0f02640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc113870 con 0x7f8ddc109bb0 2024-08-28T19:32:56.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8de0f02640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1136f0 con 0x7f8ddc103920 2024-08-28T19:32:56.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8de0f02640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1139f0 con 0x7f8ddc109260 2024-08-28T19:32:56.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8ddad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ddc109bb0 0x7f8ddc075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8ddad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ddc109bb0 0x7f8ddc075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56496/0 (socket says 172.21.15.16:56496) 2024-08-28T19:32:56.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8ddad9d640 1 -- 172.21.15.16:0/3841549069 learned_addr learned my addr 172.21.15.16:0/3841549069 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:56.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ddc109260 0x7f8ddc0759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8dda59c640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc103920 0x7f8ddc0754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8ddad9d640 1 -- 172.21.15.16:0/3841549069 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ddc109260 msgr2=0x7f8ddc0759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8ddad9d640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ddc109260 0x7f8ddc0759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8ddad9d640 1 -- 172.21.15.16:0/3841549069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc103920 msgr2=0x7f8ddc0754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8ddad9d640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc103920 0x7f8ddc0754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.888+0000 7f8ddad9d640 1 -- 172.21.15.16:0/3841549069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ddc1a8500 con 0x7f8ddc109bb0 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.889+0000 7f8ddad9d640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ddc109bb0 0x7f8ddc075f30 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400eaa0 tx=0x7f8dd400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.889+0000 7f8dcb7fe640 1 -- 172.21.15.16:0/3841549069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd400cdb0 con 0x7f8ddc109bb0 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.889+0000 7f8de0f02640 1 -- 172.21.15.16:0/3841549069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ddc1a87f0 con 0x7f8ddc109bb0 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.889+0000 7f8dcb7fe640 1 -- 172.21.15.16:0/3841549069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dd4005020 con 0x7f8ddc109bb0 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.889+0000 7f8dcb7fe640 1 -- 172.21.15.16:0/3841549069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40116d0 con 0x7f8ddc109bb0 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.889+0000 7f8de0f02640 1 -- 172.21.15.16:0/3841549069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ddc1a8d30 con 0x7f8ddc109bb0 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8de7477740 1 Processor -- start 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8dcb7fe640 1 -- 172.21.15.16:0/3841549069 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8dd40051d0 con 0x7f8ddc109bb0 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8de7477740 1 -- start start 2024-08-28T19:32:56.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8de7477740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab37310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8de7477740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a7691b0 0x55e94ab37850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8de7477740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a771390 0x55e94ab37d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8de7477740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e94aaae680 con 0x55e94a7691b0 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8de7477740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e94aa9dc10 con 0x55e94a771390 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8de7477740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e94aaa1bc0 con 0x55e94a792070 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a7691b0 0x55e94ab37850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a7691b0 0x55e94ab37850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56500/0 (socket says 172.21.15.16:56500) 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8dda59c640 1 -- 172.21.15.16:0/1162624495 learned_addr learned my addr 172.21.15.16:0/1162624495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.892+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/1162624495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab37310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/1162624495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a771390 0x55e94ab37d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8dcb7fe640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8dc40b4ca0 0x7f8dc40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8ddad9d640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8dc40b4ca0 0x7f8dc40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8dda59c640 1 -- 172.21.15.16:0/1162624495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 msgr2=0x55e94ab37310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8dda59c640 1 --2- 172.21.15.16:0/1162624495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab37310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8dda59c640 1 -- 172.21.15.16:0/1162624495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a771390 msgr2=0x55e94ab37d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8dda59c640 1 --2- 172.21.15.16:0/1162624495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a771390 0x55e94ab37d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8dda59c640 1 -- 172.21.15.16:0/1162624495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e94a7fd650 con 0x55e94a7691b0 2024-08-28T19:32:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/1162624495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab37310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8dcb7fe640 1 -- 172.21.15.16:0/3841549069 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f8dd40e5a80 con 0x7f8ddc109bb0 2024-08-28T19:32:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8dda59c640 1 --2- 172.21.15.16:0/1162624495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a7691b0 0x55e94ab37850 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009900 tx=0x7f8dcc032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a548e7b215c27ed server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.893+0000 7f8dc8ff9640 1 -- 172.21.15.16:0/1162624495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc040070 con 0x55e94a7691b0 2024-08-28T19:32:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.894+0000 7f8dc8ff9640 1 -- 172.21.15.16:0/1162624495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dcc032e50 con 0x55e94a7691b0 2024-08-28T19:32:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.894+0000 7f8dc8ff9640 1 -- 172.21.15.16:0/1162624495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc035040 con 0x55e94a7691b0 2024-08-28T19:32:56.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.894+0000 7f8de7477740 1 -- 172.21.15.16:0/1162624495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a7691b0 msgr2=0x55e94ab37850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:56.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.894+0000 7f8ddad9d640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8dc40b4ca0 0x7f8dc40b71b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40a7ed0 tx=0x7f8dd40e0040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:56.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.894+0000 7f8de7477740 1 --2- 172.21.15.16:0/1162624495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a7691b0 0x55e94ab37850 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009900 tx=0x7f8dcc032c60 comp rx=0 tx=0).stop 2024-08-28T19:32:56.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.894+0000 7f8de7477740 1 -- 172.21.15.16:0/1162624495 shutdown_connections 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.894+0000 7f8de7477740 1 --2- 172.21.15.16:0/1162624495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a771390 0x55e94ab37d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.894+0000 7f8de7477740 1 --2- 172.21.15.16:0/1162624495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a7691b0 0x55e94ab37850 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.894+0000 7f8de7477740 1 --2- 172.21.15.16:0/1162624495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab37310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.894+0000 7f8de7477740 1 -- 172.21.15.16:0/1162624495 >> 172.21.15.16:0/1162624495 conn(0x55e94a786460 msgr2=0x55e94ab35200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8de7477740 1 -- 172.21.15.16:0/1162624495 shutdown_connections 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8de7477740 1 -- 172.21.15.16:0/1162624495 wait complete. 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8de7477740 1 -- start start 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8de7477740 1 Processor -- start 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8de7477740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a7691b0 0x55e94ab42a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8de7477740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab42fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8de7477740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab4ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8de7477740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e94a900700 con 0x55e94a771390 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8de7477740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e94aa24750 con 0x55e94a7691b0 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8de7477740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e94ab45a90 con 0x55e94a792070 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8de7477740 10 client.0 _ll_register_callbacks cb 0x55e94aa53e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.895+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab42fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab42fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56502/0 (socket says 172.21.15.16:56502) 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dda59c640 1 -- 172.21.15.16:0/3489880301 learned_addr learned my addr 172.21.15.16:0/3489880301 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a7691b0 0x55e94ab42a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab4ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dda59c640 1 -- 172.21.15.16:0/3489880301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 msgr2=0x55e94ab4ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dda59c640 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab4ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dda59c640 1 -- 172.21.15.16:0/3489880301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a7691b0 msgr2=0x55e94ab42a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dda59c640 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a7691b0 0x55e94ab42a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dda59c640 1 -- 172.21.15.16:0/3489880301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e94aa5d900 con 0x55e94a771390 2024-08-28T19:32:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab4ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dda59c640 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab42fb0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc004600 tx=0x7f8dcc031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d23aaa70ca41eff server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/3489880301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc04b020 con 0x55e94a771390 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/3489880301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f8dcc004750 con 0x55e94a771390 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8de7477740 1 -- 172.21.15.16:0/3489880301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dcc032e50 con 0x55e94a771390 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/3489880301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f8dcc056050 con 0x55e94a771390 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8de7477740 10 client.18471 fetch_fsmap learned FSMap version 59 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.896+0000 7f8de7477740 1 -- 172.21.15.16:0/3489880301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55e94ab51840 con 0x55e94a771390 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.897+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/3489880301 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dcc032e50 con 0x55e94a771390 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.897+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/3489880301 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7f8dcc03c640 con 0x55e94a771390 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.897+0000 7f8de7477740 10 client.18471 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.897+0000 7f8de7477740 10 client.18471 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:32:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.897+0000 7f8de7477740 20 client.18471 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.897+0000 7f8de7477740 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e94a7509f0 0x55e94ab54ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.897+0000 7f8de7477740 4 client.18471 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.897+0000 7f8de7477740 1 -- 172.21.15.16:0/3489880301 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e94aab05d0 con 0x55e94a7509f0 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.897+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e94a7509f0 0x55e94ab54ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.898+0000 7f8dc1ffb640 10 client.18471 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.898+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e94a7509f0 0x55e94ab54ba0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=47d00d9ab91b4e6b server_cookie=e80d45c3de8cf52c in_seq=0 out_seq=0 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/3489880301 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e94aab05d0 con 0x55e94a7509f0 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8dc1ffb640 10 client.18471 handle_command_reply: tid=0 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 1 client.18471 shutdown 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 1 -- 172.21.15.16:0/3489880301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 msgr2=0x55e94ab42fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab42fb0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc004600 tx=0x7f8dcc031750 comp rx=0 tx=0).stop 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 1 -- 172.21.15.16:0/3489880301 shutdown_connections 2024-08-28T19:32:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e94a7509f0 0x55e94ab54ba0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab4ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab42fb0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8dc1ffb640 0 client.18471 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:32:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 1 --2- 172.21.15.16:0/3489880301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a7691b0 0x55e94ab42a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 1 -- 172.21.15.16:0/3489880301 >> 172.21.15.16:0/3489880301 conn(0x55e94a786460 msgr2=0x55e94ab3a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 1 -- 172.21.15.16:0/3489880301 shutdown_connections 2024-08-28T19:32:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 1 -- 172.21.15.16:0/3489880301 wait complete. 2024-08-28T19:32:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.899+0000 7f8de7477740 20 client.18471 trim_cache size 0 max 16384 2024-08-28T19:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.912+0000 7f8de7477740 1 Processor -- start 2024-08-28T19:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.912+0000 7f8de7477740 1 -- start start 2024-08-28T19:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.912+0000 7f8de7477740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a7691b0 0x55e94ab43050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.912+0000 7f8de7477740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab3a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.912+0000 7f8de7477740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab3a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.912+0000 7f8de7477740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1b2e90 con 0x55e94a771390 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.912+0000 7f8de7477740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1b4af0 con 0x55e94a7691b0 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.912+0000 7f8de7477740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1b3620 con 0x55e94a792070 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab3a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab3a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56506/0 (socket says 172.21.15.16:56506) 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dda59c640 1 -- 172.21.15.16:0/4278621895 learned_addr learned my addr 172.21.15.16:0/4278621895 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/4278621895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a7691b0 0x55e94ab43050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dda59c640 1 -- 172.21.15.16:0/4278621895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 msgr2=0x55e94ab3a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/4278621895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab3a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dda59c640 1 --2- 172.21.15.16:0/4278621895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab3a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dda59c640 1 -- 172.21.15.16:0/4278621895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a7691b0 msgr2=0x55e94ab43050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dda59c640 1 --2- 172.21.15.16:0/4278621895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a7691b0 0x55e94ab43050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dda59c640 1 -- 172.21.15.16:0/4278621895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e94ab3a1a0 con 0x55e94a771390 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/4278621895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab3a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dda59c640 1 --2- 172.21.15.16:0/4278621895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab3a410 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc02a930 tx=0x7f8dcc031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edee4325b33f5f88 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dc3fff640 1 -- 172.21.15.16:0/4278621895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc04b070 con 0x55e94a771390 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dc3fff640 1 -- 172.21.15.16:0/4278621895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dcc005470 con 0x55e94a771390 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.913+0000 7f8dc3fff640 1 -- 172.21.15.16:0/4278621895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc035070 con 0x55e94a771390 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.914+0000 7f8de7477740 1 -- 172.21.15.16:0/4278621895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 msgr2=0x55e94ab3a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.914+0000 7f8de7477740 1 --2- 172.21.15.16:0/4278621895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab3a410 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc02a930 tx=0x7f8dcc031400 comp rx=0 tx=0).stop 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.914+0000 7f8de7477740 1 -- 172.21.15.16:0/4278621895 shutdown_connections 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.914+0000 7f8de7477740 1 --2- 172.21.15.16:0/4278621895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a792070 0x55e94ab3a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.914+0000 7f8de7477740 1 --2- 172.21.15.16:0/4278621895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab3a410 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.914+0000 7f8de7477740 1 --2- 172.21.15.16:0/4278621895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a7691b0 0x55e94ab43050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.914+0000 7f8de7477740 1 -- 172.21.15.16:0/4278621895 >> 172.21.15.16:0/4278621895 conn(0x55e94a786460 msgr2=0x55e94ab412f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.914+0000 7f8de7477740 1 -- 172.21.15.16:0/4278621895 shutdown_connections 2024-08-28T19:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.914+0000 7f8de7477740 1 -- 172.21.15.16:0/4278621895 wait complete. 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.914+0000 7f8de7477740 1 -- start start 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.914+0000 7f8de7477740 1 Processor -- start 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8de7477740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a7691b0 0x55e94ab55dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8de7477740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab56310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8de7477740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a792070 0x55e94ab56850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8de7477740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1b43f0 con 0x55e94a771390 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8de7477740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1b3cf0 con 0x55e94a792070 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8de7477740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1b0e90 con 0x55e94a7691b0 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8de7477740 10 client.0 _ll_register_callbacks cb 0x7f8dd000be40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab56310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab56310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56516/0 (socket says 172.21.15.16:56516) 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dda59c640 1 -- 172.21.15.16:0/928815620 learned_addr learned my addr 172.21.15.16:0/928815620 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:56.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a792070 0x55e94ab56850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a7691b0 0x55e94ab55dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dda59c640 1 -- 172.21.15.16:0/928815620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a7691b0 msgr2=0x55e94ab55dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dda59c640 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a7691b0 0x55e94ab55dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dda59c640 1 -- 172.21.15.16:0/928815620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a792070 msgr2=0x55e94ab56850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dda59c640 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a792070 0x55e94ab56850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dda59c640 1 -- 172.21.15.16:0/928815620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e94abbe4d0 con 0x55e94a771390 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a7691b0 0x55e94ab55dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dda59c640 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab56310 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc000c00 tx=0x7f8dcc031b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c271f41c04de3c09 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.915+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/928815620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc04b070 con 0x55e94a771390 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8de7477740 1 -- 172.21.15.16:0/928815620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dcc005470 con 0x55e94a771390 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/928815620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f8dcc03c440 con 0x55e94a771390 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/928815620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f8dcc035040 con 0x55e94a771390 2024-08-28T19:32:56.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8de7477740 10 client.18477 fetch_fsmap learned FSMap version 59 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8de7477740 1 -- 172.21.15.16:0/928815620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55e94abbf6c0 con 0x55e94a771390 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/928815620 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dcc005470 con 0x55e94a771390 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/928815620 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7f8dcc05b440 con 0x55e94a771390 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8de7477740 10 client.18477 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8de7477740 10 client.18477 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8de7477740 20 client.18477 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8de7477740 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e94a725da0 0x55e94abc1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8de7477740 4 client.18477 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8de7477740 1 -- 172.21.15.16:0/928815620 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e94aaac070 con 0x55e94a725da0 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.916+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e94a725da0 0x55e94abc1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8dd9d9b640 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e94a725da0 0x55e94abc1eb0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=58642bd4cf648f3 server_cookie=c5c01251f2e89d09 in_seq=0 out_seq=0 2024-08-28T19:32:56.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8dc1ffb640 10 client.18477 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8dc1ffb640 1 -- 172.21.15.16:0/928815620 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e94aaac070 con 0x55e94a725da0 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8dc1ffb640 10 client.18477 handle_command_reply: tid=0 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8de7477740 1 client.18477 shutdown 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8de7477740 1 -- 172.21.15.16:0/928815620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 msgr2=0x55e94ab56310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8de7477740 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab56310 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc000c00 tx=0x7f8dcc031b00 comp rx=0 tx=0).stop 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8de7477740 1 -- 172.21.15.16:0/928815620 shutdown_connections 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8de7477740 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55e94a725da0 0x55e94abc1eb0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8de7477740 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e94a792070 0x55e94ab56850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8dc1ffb640 0 client.18477 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8de7477740 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e94a771390 0x55e94ab56310 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8de7477740 1 --2- 172.21.15.16:0/928815620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e94a7691b0 0x55e94ab55dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8de7477740 1 -- 172.21.15.16:0/928815620 >> 172.21.15.16:0/928815620 conn(0x55e94a786460 msgr2=0x55e94ab3a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:56.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8de7477740 1 -- 172.21.15.16:0/928815620 shutdown_connections 2024-08-28T19:32:56.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.917+0000 7f8de7477740 1 -- 172.21.15.16:0/928815620 wait complete. 2024-08-28T19:32:56.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de7477740 20 client.18477 trim_cache size 0 max 16384 2024-08-28T19:32:56.927 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:32:56.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 -- 172.21.15.16:0/3841549069 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8dc40b4ca0 msgr2=0x7f8dc40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8dc40b4ca0 0x7f8dc40b71b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40a7ed0 tx=0x7f8dd40e0040 comp rx=0 tx=0).stop 2024-08-28T19:32:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 -- 172.21.15.16:0/3841549069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ddc109bb0 msgr2=0x7f8ddc075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ddc109bb0 0x7f8ddc075f30 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400eaa0 tx=0x7f8dd400ef60 comp rx=0 tx=0).stop 2024-08-28T19:32:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 -- 172.21.15.16:0/3841549069 shutdown_connections 2024-08-28T19:32:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ddc109bb0 0x7f8ddc075f30 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ddc109260 0x7f8ddc0759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8dc40b4ca0 0x7f8dc40b71b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 --2- 172.21.15.16:0/3841549069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc103920 0x7f8ddc0754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 -- 172.21.15.16:0/3841549069 >> 172.21.15.16:0/3841549069 conn(0x7f8ddc0ff370 msgr2=0x7f8ddc101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 -- 172.21.15.16:0/3841549069 shutdown_connections 2024-08-28T19:32:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:56.918+0000 7f8de0f02640 1 -- 172.21.15.16:0/3841549069 wait complete. 2024-08-28T19:32:56.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 488453 3.73 MB/sec execute 516 sec latency 3232.213 ms 2024-08-28T19:32:57.611 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:57 smithi028 ceph-mon[21769]: pgmap v706: 129 pgs: 129 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-08-28T19:32:57.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:57 smithi115 ceph-mon[22184]: pgmap v706: 129 pgs: 129 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-08-28T19:32:57.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:57 smithi016 ceph-mon[18556]: pgmap v706: 129 pgs: 129 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-08-28T19:32:57.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:32:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:57.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 490518 3.75 MB/sec execute 517 sec latency 3233.363 ms 2024-08-28T19:32:58.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.509+0000 7fa12d052640 1 -- 172.21.15.16:0/3142097680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa128109260 msgr2=0x7fa128109670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:58.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.509+0000 7fa12d052640 1 --2- 172.21.15.16:0/3142097680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa128109260 0x7fa128109670 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009900 tx=0x7fa11c02a410 comp rx=0 tx=0).stop 2024-08-28T19:32:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.509+0000 7fa12d052640 1 -- 172.21.15.16:0/3142097680 shutdown_connections 2024-08-28T19:32:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.509+0000 7fa12d052640 1 --2- 172.21.15.16:0/3142097680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa128109bb0 0x7fa1281103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.509+0000 7fa12d052640 1 --2- 172.21.15.16:0/3142097680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa128109260 0x7fa128109670 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.509+0000 7fa12d052640 1 --2- 172.21.15.16:0/3142097680 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa128103920 0x7fa128103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.509+0000 7fa12d052640 1 -- 172.21.15.16:0/3142097680 >> 172.21.15.16:0/3142097680 conn(0x7fa1280ff3b0 msgr2=0x7fa128101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.509+0000 7fa12d052640 1 -- 172.21.15.16:0/3142097680 shutdown_connections 2024-08-28T19:32:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.509+0000 7fa12d052640 1 -- 172.21.15.16:0/3142097680 wait complete. 2024-08-28T19:32:58.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.510+0000 7fa12d052640 1 Processor -- start 2024-08-28T19:32:58.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.510+0000 7fa12d052640 1 -- start start 2024-08-28T19:32:58.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12d052640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa128103920 0x7fa1280754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12d052640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa128109260 0x7fa1280759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12d052640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa128109bb0 0x7fa128075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12d052640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281138c0 con 0x7fa128109260 2024-08-28T19:32:58.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12d052640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa128113740 con 0x7fa128109bb0 2024-08-28T19:32:58.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12d052640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa128113a40 con 0x7fa128103920 2024-08-28T19:32:58.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa128109260 0x7fa1280759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa128109bb0 0x7fa128075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa128109260 0x7fa1280759f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56542/0 (socket says 172.21.15.16:56542) 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12659c640 1 -- 172.21.15.16:0/3383208355 learned_addr learned my addr 172.21.15.16:0/3383208355 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa126d9d640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa128103920 0x7fa1280754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12659c640 1 -- 172.21.15.16:0/3383208355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa128103920 msgr2=0x7fa1280754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12659c640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa128103920 0x7fa1280754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12659c640 1 -- 172.21.15.16:0/3383208355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa128109bb0 msgr2=0x7fa128075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12659c640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa128109bb0 0x7fa128075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12659c640 1 -- 172.21.15.16:0/3383208355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1281a8530 con 0x7fa128109260 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12759e640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa128109bb0 0x7fa128075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.511+0000 7fa12659c640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa128109260 0x7fa1280759f0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fa11c005ab0 tx=0x7fa11c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.512+0000 7fa10ffff640 1 -- 172.21.15.16:0/3383208355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c03f070 con 0x7fa128109260 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.512+0000 7fa12d052640 1 -- 172.21.15.16:0/3383208355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1281a87c0 con 0x7fa128109260 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.512+0000 7fa10ffff640 1 -- 172.21.15.16:0/3383208355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11c031b50 con 0x7fa128109260 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.512+0000 7fa12d052640 1 -- 172.21.15.16:0/3383208355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1281a8cf0 con 0x7fa128109260 2024-08-28T19:32:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.512+0000 7fa10ffff640 1 -- 172.21.15.16:0/3383208355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c035030 con 0x7fa128109260 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.514+0000 7fa1335c7740 1 Processor -- start 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.514+0000 7fa1335c7740 1 -- start start 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.514+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 0x55d82e25c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de8e1b0 0x55d82e25c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de96390 0x55d82e25cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa1335c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d82e1d3680 con 0x55d82deb7070 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa1335c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d82e1c2c10 con 0x55d82de96390 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa1335c7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d82e1c6bc0 con 0x55d82de8e1b0 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa12759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 0x55d82e25c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa12759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 0x55d82e25c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56556/0 (socket says 172.21.15.16:56556) 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa12759e640 1 -- 172.21.15.16:0/80447501 learned_addr learned my addr 172.21.15.16:0/80447501 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa126d9d640 1 --2- 172.21.15.16:0/80447501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de8e1b0 0x55d82e25c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa10ffff640 1 -- 172.21.15.16:0/3383208355 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa11c042070 con 0x7fa128109260 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa12759e640 1 -- 172.21.15.16:0/80447501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de8e1b0 msgr2=0x55d82e25c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa12759e640 1 --2- 172.21.15.16:0/80447501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de8e1b0 0x55d82e25c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa12659c640 1 --2- 172.21.15.16:0/80447501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de96390 0x55d82e25cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa12759e640 1 -- 172.21.15.16:0/80447501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de96390 msgr2=0x55d82e25cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa12759e640 1 --2- 172.21.15.16:0/80447501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de96390 0x55d82e25cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa12759e640 1 -- 172.21.15.16:0/80447501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d82df22650 con 0x55d82deb7070 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa126d9d640 1 --2- 172.21.15.16:0/80447501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de8e1b0 0x55d82e25c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa12659c640 1 --2- 172.21.15.16:0/80447501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de96390 0x55d82e25cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa12759e640 1 --2- 172.21.15.16:0/80447501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 0x55d82e25c310 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fa120011810 tx=0x7fa120011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f99135a574ade8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa10d7fa640 1 -- 172.21.15.16:0/80447501 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa120011310 con 0x55d82deb7070 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa10d7fa640 1 -- 172.21.15.16:0/80447501 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa12000f9a0 con 0x55d82deb7070 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.515+0000 7fa10d7fa640 1 -- 172.21.15.16:0/80447501 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1200055f0 con 0x55d82deb7070 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.516+0000 7fa10ffff640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1100b4ca0 0x7fa1100b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.516+0000 7fa126d9d640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1100b4ca0 0x7fa1100b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.516+0000 7fa10ffff640 1 -- 172.21.15.16:0/3383208355 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fa11c105e60 con 0x7fa128109260 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.516+0000 7fa1335c7740 1 -- 172.21.15.16:0/80447501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 msgr2=0x55d82e25c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.516+0000 7fa1335c7740 1 --2- 172.21.15.16:0/80447501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 0x55d82e25c310 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fa120011810 tx=0x7fa120011cd0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.516+0000 7fa126d9d640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1100b4ca0 0x7fa1100b71b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x55d82e25af90 tx=0x7fa118005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.516+0000 7fa1335c7740 1 -- 172.21.15.16:0/80447501 shutdown_connections 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.516+0000 7fa1335c7740 1 --2- 172.21.15.16:0/80447501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de96390 0x55d82e25cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.516+0000 7fa1335c7740 1 --2- 172.21.15.16:0/80447501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de8e1b0 0x55d82e25c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.516+0000 7fa1335c7740 1 --2- 172.21.15.16:0/80447501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 0x55d82e25c310 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.517+0000 7fa1335c7740 1 -- 172.21.15.16:0/80447501 >> 172.21.15.16:0/80447501 conn(0x55d82deab460 msgr2=0x55d82e25a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.517+0000 7fa1335c7740 1 -- 172.21.15.16:0/80447501 shutdown_connections 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.517+0000 7fa1335c7740 1 -- 172.21.15.16:0/80447501 wait complete. 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.517+0000 7fa1335c7740 1 -- start start 2024-08-28T19:32:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.517+0000 7fa1335c7740 1 Processor -- start 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.517+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 0x55d82e267ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.517+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de96390 0x55d82e268020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 0x55d82e273e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa1335c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d82e025700 con 0x55d82deb7070 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa1335c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d82e149750 con 0x55d82de8e1b0 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa1335c7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d82e26aaf0 con 0x55d82de96390 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa1335c7740 10 client.0 _ll_register_callbacks cb 0x55d82e178e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa126d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 0x55d82e267ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa12759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 0x55d82e273e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa126d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 0x55d82e267ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:53312/0 (socket says 172.21.15.16:53312) 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa126d9d640 1 -- 172.21.15.16:0/814451065 learned_addr learned my addr 172.21.15.16:0/814451065 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa12659c640 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de96390 0x55d82e268020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa126d9d640 1 -- 172.21.15.16:0/814451065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de96390 msgr2=0x55d82e268020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa126d9d640 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de96390 0x55d82e268020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa126d9d640 1 -- 172.21.15.16:0/814451065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 msgr2=0x55d82e273e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa126d9d640 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 0x55d82e273e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa126d9d640 1 -- 172.21.15.16:0/814451065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d82e182900 con 0x55d82de8e1b0 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa12759e640 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 0x55d82e273e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa12659c640 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de96390 0x55d82e268020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa126d9d640 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 0x55d82e267ae0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa118004610 tx=0x7fa118004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5eaf9b8b3b5cc08c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa10aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.518+0000 7fa10aff5640 1 -- 172.21.15.16:0/814451065 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11801f700 con 0x55d82de8e1b0 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa1335c7740 1 -- 172.21.15.16:0/814451065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa12000f9a0 con 0x55d82de8e1b0 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa10aff5640 1 -- 172.21.15.16:0/814451065 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fa11801f8a0 con 0x55d82de8e1b0 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa10aff5640 1 -- 172.21.15.16:0/814451065 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fa11801faa0 con 0x55d82de8e1b0 2024-08-28T19:32:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa1335c7740 10 client.27941 fetch_fsmap learned FSMap version 59 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa1335c7740 1 -- 172.21.15.16:0/814451065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55d82e276910 con 0x55d82de8e1b0 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa10aff5640 1 -- 172.21.15.16:0/814451065 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa12000f9a0 con 0x55d82de8e1b0 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa10aff5640 1 -- 172.21.15.16:0/814451065 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7fa118027070 con 0x55d82de8e1b0 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa1335c7740 10 client.27941 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa1335c7740 10 client.27941 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa1335c7740 20 client.27941 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa1335c7740 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d82de759f0 0x55d82e279f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa1335c7740 4 client.27941 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa1335c7740 1 -- 172.21.15.16:0/814451065 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d82e1d55d0 con 0x55d82de759f0 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.519+0000 7fa12659c640 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d82de759f0 0x55d82e279f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.520+0000 7fa10aff5640 10 client.27941 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.520+0000 7fa12659c640 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d82de759f0 0x55d82e279f40 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=86bd269ebd729b11 server_cookie=1c2acd768b83664c in_seq=0 out_seq=0 2024-08-28T19:32:58.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa10aff5640 1 -- 172.21.15.16:0/814451065 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d82e1d55d0 con 0x55d82de759f0 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa10aff5640 10 client.27941 handle_command_reply: tid=0 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa1335c7740 1 client.27941 shutdown 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa1335c7740 1 -- 172.21.15.16:0/814451065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 msgr2=0x55d82e267ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa1335c7740 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 0x55d82e267ae0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa118004610 tx=0x7fa118004730 comp rx=0 tx=0).stop 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa1335c7740 1 -- 172.21.15.16:0/814451065 shutdown_connections 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa1335c7740 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d82de759f0 0x55d82e279f40 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa1335c7740 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82deb7070 0x55d82e273e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa1335c7740 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82de96390 0x55d82e268020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa1335c7740 1 --2- 172.21.15.16:0/814451065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 0x55d82e267ae0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa10aff5640 0 client.27941 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.521+0000 7fa1335c7740 1 -- 172.21.15.16:0/814451065 >> 172.21.15.16:0/814451065 conn(0x55d82deab460 msgr2=0x55d82e25f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.522+0000 7fa1335c7740 1 -- 172.21.15.16:0/814451065 shutdown_connections 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.522+0000 7fa1335c7740 1 -- 172.21.15.16:0/814451065 wait complete. 2024-08-28T19:32:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.522+0000 7fa1335c7740 20 client.27941 trim_cache size 0 max 16384 2024-08-28T19:32:58.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa1335c7740 1 Processor -- start 2024-08-28T19:32:58.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa1335c7740 1 -- start start 2024-08-28T19:32:58.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de8e1b0 0x55d82e273d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de96390 0x55d82e259b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82deb7070 0x55d82e25a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa1335c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281b2f10 con 0x55d82de8e1b0 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa1335c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281b4b70 con 0x55d82de96390 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa1335c7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281b36a0 con 0x55d82deb7070 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa126d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de8e1b0 0x55d82e273d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa126d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de8e1b0 0x55d82e273d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56574/0 (socket says 172.21.15.16:56574) 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa12759e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82deb7070 0x55d82e25a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.536+0000 7fa12659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de96390 0x55d82e259b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.537+0000 7fa126d9d640 1 -- 172.21.15.16:0/2249130747 learned_addr learned my addr 172.21.15.16:0/2249130747 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.537+0000 7fa126d9d640 1 -- 172.21.15.16:0/2249130747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82deb7070 msgr2=0x55d82e25a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.537+0000 7fa126d9d640 1 --2- 172.21.15.16:0/2249130747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82deb7070 0x55d82e25a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.537+0000 7fa126d9d640 1 -- 172.21.15.16:0/2249130747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de96390 msgr2=0x55d82e259b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.537+0000 7fa126d9d640 1 --2- 172.21.15.16:0/2249130747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de96390 0x55d82e259b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.537+0000 7fa126d9d640 1 -- 172.21.15.16:0/2249130747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d82e25e600 con 0x55d82de8e1b0 2024-08-28T19:32:58.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.537+0000 7fa12759e640 1 --2- 172.21.15.16:0/2249130747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82deb7070 0x55d82e25a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:58.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.537+0000 7fa126d9d640 1 --2- 172.21.15.16:0/2249130747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de8e1b0 0x55d82e273d10 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fa118004570 tx=0x7fa118003e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b4819df88521e4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:58.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.537+0000 7fa10cff9640 1 -- 172.21.15.16:0/2249130747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118005020 con 0x55d82de8e1b0 2024-08-28T19:32:58.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.537+0000 7fa10cff9640 1 -- 172.21.15.16:0/2249130747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1180047a0 con 0x55d82de8e1b0 2024-08-28T19:32:58.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.537+0000 7fa10cff9640 1 -- 172.21.15.16:0/2249130747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11801d7f0 con 0x55d82de8e1b0 2024-08-28T19:32:58.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.538+0000 7fa1335c7740 1 -- 172.21.15.16:0/2249130747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de8e1b0 msgr2=0x55d82e273d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:58.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.538+0000 7fa1335c7740 1 --2- 172.21.15.16:0/2249130747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de8e1b0 0x55d82e273d10 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fa118004570 tx=0x7fa118003e50 comp rx=0 tx=0).stop 2024-08-28T19:32:58.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.538+0000 7fa1335c7740 1 -- 172.21.15.16:0/2249130747 shutdown_connections 2024-08-28T19:32:58.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.538+0000 7fa1335c7740 1 --2- 172.21.15.16:0/2249130747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82deb7070 0x55d82e25a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.538+0000 7fa1335c7740 1 --2- 172.21.15.16:0/2249130747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de96390 0x55d82e259b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.538+0000 7fa1335c7740 1 --2- 172.21.15.16:0/2249130747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de8e1b0 0x55d82e273d10 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.538+0000 7fa1335c7740 1 -- 172.21.15.16:0/2249130747 >> 172.21.15.16:0/2249130747 conn(0x55d82deab460 msgr2=0x55d82e266360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.538+0000 7fa1335c7740 1 -- 172.21.15.16:0/2249130747 shutdown_connections 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.538+0000 7fa1335c7740 1 -- 172.21.15.16:0/2249130747 wait complete. 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa1335c7740 1 -- start start 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa1335c7740 1 Processor -- start 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 0x55d82e26bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de96390 0x55d82e26c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa1335c7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82deb7070 0x55d82e26c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa1335c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281b4470 con 0x55d82de96390 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa1335c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281b3d70 con 0x55d82de8e1b0 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa1335c7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281b0f10 con 0x55d82deb7070 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa1335c7740 10 client.0 _ll_register_callbacks cb 0x7fa118009c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:58.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa12659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de96390 0x55d82e26c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa12659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de96390 0x55d82e26c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56580/0 (socket says 172.21.15.16:56580) 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.539+0000 7fa12659c640 1 -- 172.21.15.16:0/990752507 learned_addr learned my addr 172.21.15.16:0/990752507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.540+0000 7fa126d9d640 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 0x55d82e26bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.540+0000 7fa12759e640 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82deb7070 0x55d82e26c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.540+0000 7fa126d9d640 1 -- 172.21.15.16:0/990752507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82deb7070 msgr2=0x55d82e26c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.540+0000 7fa126d9d640 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82deb7070 0x55d82e26c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.540+0000 7fa126d9d640 1 -- 172.21.15.16:0/990752507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de96390 msgr2=0x55d82e26c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.540+0000 7fa126d9d640 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de96390 0x55d82e26c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.540+0000 7fa126d9d640 1 -- 172.21.15.16:0/990752507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d82e2e3b00 con 0x55d82de8e1b0 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.540+0000 7fa10aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.540+0000 7fa126d9d640 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 0x55d82e26bf40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa118012d20 tx=0x7fa118017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d7c42ed9c2c894a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.540+0000 7fa10aff5640 1 -- 172.21.15.16:0/990752507 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118005430 con 0x55d82de8e1b0 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.540+0000 7fa1335c7740 1 -- 172.21.15.16:0/990752507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1180047a0 con 0x55d82de8e1b0 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa10aff5640 1 -- 172.21.15.16:0/990752507 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fa118029440 con 0x55d82de8e1b0 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa10aff5640 1 -- 172.21.15.16:0/990752507 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fa118029640 con 0x55d82de8e1b0 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa1335c7740 10 client.27947 fetch_fsmap learned FSMap version 59 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa1335c7740 1 -- 172.21.15.16:0/990752507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55d82e2e4d80 con 0x55d82de8e1b0 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa10aff5640 1 -- 172.21.15.16:0/990752507 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1180047a0 con 0x55d82de8e1b0 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa10aff5640 1 -- 172.21.15.16:0/990752507 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7fa118027070 con 0x55d82de8e1b0 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa1335c7740 10 client.27947 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa1335c7740 10 client.27947 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa1335c7740 20 client.27947 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa1335c7740 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d82de4ada0 0x55d82e2e72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa1335c7740 4 client.27947 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:32:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa1335c7740 1 -- 172.21.15.16:0/990752507 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d82e1d1070 con 0x55d82de4ada0 2024-08-28T19:32:58.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.541+0000 7fa12659c640 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d82de4ada0 0x55d82e2e72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:58.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.542+0000 7fa10aff5640 10 client.27947 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:32:58.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.542+0000 7fa12659c640 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d82de4ada0 0x55d82e2e72c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f0e5ed2b4ad79a50 server_cookie=f7ada32bf2b47ee9 in_seq=0 out_seq=0 2024-08-28T19:32:58.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.542+0000 7fa10aff5640 1 -- 172.21.15.16:0/990752507 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d82e1d1070 con 0x55d82de4ada0 2024-08-28T19:32:58.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.542+0000 7fa10aff5640 10 client.27947 handle_command_reply: tid=0 2024-08-28T19:32:58.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.542+0000 7fa1335c7740 1 client.27947 shutdown 2024-08-28T19:32:58.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa1335c7740 1 -- 172.21.15.16:0/990752507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 msgr2=0x55d82e26bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:58.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa1335c7740 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 0x55d82e26bf40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa118012d20 tx=0x7fa118017000 comp rx=0 tx=0).stop 2024-08-28T19:32:58.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa1335c7740 1 -- 172.21.15.16:0/990752507 shutdown_connections 2024-08-28T19:32:58.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa1335c7740 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d82de4ada0 0x55d82e2e72c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa1335c7740 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d82deb7070 0x55d82e26c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa1335c7740 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d82de96390 0x55d82e26c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa10aff5640 0 client.27947 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:32:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa1335c7740 1 --2- 172.21.15.16:0/990752507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d82de8e1b0 0x55d82e26bf40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa1335c7740 1 -- 172.21.15.16:0/990752507 >> 172.21.15.16:0/990752507 conn(0x55d82deab460 msgr2=0x55d82e259a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa1335c7740 1 -- 172.21.15.16:0/990752507 shutdown_connections 2024-08-28T19:32:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa1335c7740 1 -- 172.21.15.16:0/990752507 wait complete. 2024-08-28T19:32:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa1335c7740 20 client.27947 trim_cache size 0 max 16384 2024-08-28T19:32:58.552 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:32:58.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa12d052640 1 -- 172.21.15.16:0/3383208355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1100b4ca0 msgr2=0x7fa1100b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:58.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa12d052640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1100b4ca0 0x7fa1100b71b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x55d82e25af90 tx=0x7fa118005f90 comp rx=0 tx=0).stop 2024-08-28T19:32:58.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa12d052640 1 -- 172.21.15.16:0/3383208355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa128109260 msgr2=0x7fa1280759f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:58.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa12d052640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa128109260 0x7fa1280759f0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fa11c005ab0 tx=0x7fa11c003b90 comp rx=0 tx=0).stop 2024-08-28T19:32:58.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa12d052640 1 -- 172.21.15.16:0/3383208355 shutdown_connections 2024-08-28T19:32:58.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa12d052640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1100b4ca0 0x7fa1100b71b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa12d052640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa128109bb0 0x7fa128075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa12d052640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa128109260 0x7fa1280759f0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa12d052640 1 --2- 172.21.15.16:0/3383208355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa128103920 0x7fa1280754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:58.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.543+0000 7fa12d052640 1 -- 172.21.15.16:0/3383208355 >> 172.21.15.16:0/3383208355 conn(0x7fa1280ff3b0 msgr2=0x7fa1281017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:58.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.544+0000 7fa12d052640 1 -- 172.21.15.16:0/3383208355 shutdown_connections 2024-08-28T19:32:58.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:58.544+0000 7fa12d052640 1 -- 172.21.15.16:0/3383208355 wait complete. 2024-08-28T19:32:58.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 491733 3.75 MB/sec execute 518 sec latency 15.818 ms 2024-08-28T19:32:59.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:32:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:32:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:32:59 smithi016 ceph-mon[18556]: pgmap v707: 129 pgs: 129 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-08-28T19:32:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:32:59 smithi115 ceph-mon[22184]: pgmap v707: 129 pgs: 129 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-08-28T19:32:59.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:32:59 smithi028 ceph-mon[21769]: pgmap v707: 129 pgs: 129 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-08-28T19:32:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.940+0000 7f45bd45a640 1 -- 172.21.15.16:0/1870358551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b8103900 msgr2=0x7f45b8103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.940+0000 7f45bd45a640 1 --2- 172.21.15.16:0/1870358551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b8103900 0x7f45b8103d10 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009900 tx=0x7f45a802a410 comp rx=0 tx=0).stop 2024-08-28T19:32:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.941+0000 7f45bd45a640 1 -- 172.21.15.16:0/1870358551 shutdown_connections 2024-08-28T19:32:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.941+0000 7f45bd45a640 1 --2- 172.21.15.16:0/1870358551 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b8109b90 0x7f45b81103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.941+0000 7f45bd45a640 1 --2- 172.21.15.16:0/1870358551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b8109240 0x7f45b8109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.941+0000 7f45bd45a640 1 --2- 172.21.15.16:0/1870358551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b8103900 0x7f45b8103d10 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.941+0000 7f45bd45a640 1 -- 172.21.15.16:0/1870358551 >> 172.21.15.16:0/1870358551 conn(0x7f45b807b5f0 msgr2=0x7f45b807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.941+0000 7f45bd45a640 1 -- 172.21.15.16:0/1870358551 shutdown_connections 2024-08-28T19:32:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.941+0000 7f45bd45a640 1 -- 172.21.15.16:0/1870358551 wait complete. 2024-08-28T19:32:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.941+0000 7f45bd45a640 1 Processor -- start 2024-08-28T19:32:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.941+0000 7f45bd45a640 1 -- start start 2024-08-28T19:32:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45bd45a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b8103900 0x7f45b80754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45bd45a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b8109240 0x7f45b80759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45bd45a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b8109b90 0x7f45b8075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45bd45a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f45b8113850 con 0x7f45b8109b90 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45bd45a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45b81136d0 con 0x7f45b8109240 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45bd45a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f45b81139d0 con 0x7f45b8103900 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b8109b90 0x7f45b8075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b8109b90 0x7f45b8075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56602/0 (socket says 172.21.15.16:56602) 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b77fe640 1 -- 172.21.15.16:0/2656291121 learned_addr learned my addr 172.21.15.16:0/2656291121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b8103900 0x7f45b80754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45affff640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b8109240 0x7f45b80759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b77fe640 1 -- 172.21.15.16:0/2656291121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b8103900 msgr2=0x7f45b80754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b77fe640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b8103900 0x7f45b80754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b77fe640 1 -- 172.21.15.16:0/2656291121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b8109240 msgr2=0x7f45b80759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b77fe640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b8109240 0x7f45b80759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b77fe640 1 -- 172.21.15.16:0/2656291121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b81a8560 con 0x7f45b8109b90 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b77fe640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b8109b90 0x7f45b8075f30 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009fd0 tx=0x7f45b000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:59.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b4ff9640 1 -- 172.21.15.16:0/2656291121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b0017070 con 0x7f45b8109b90 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b4ff9640 1 -- 172.21.15.16:0/2656291121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45b0003e20 con 0x7f45b8109b90 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45bd45a640 1 -- 172.21.15.16:0/2656291121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b81a8850 con 0x7f45b8109b90 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45b4ff9640 1 -- 172.21.15.16:0/2656291121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b0010460 con 0x7f45b8109b90 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.942+0000 7f45bd45a640 1 -- 172.21.15.16:0/2656291121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b81a8d40 con 0x7f45b8109b90 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.945+0000 7f45b4ff9640 1 -- 172.21.15.16:0/2656291121 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f45b0003970 con 0x7f45b8109b90 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.945+0000 7f45c39cf740 1 Processor -- start 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.945+0000 7f45c39cf740 1 -- start start 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f48217310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e491b0 0x562f48217850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e51390 0x562f48217d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45c39cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562f4818e680 con 0x562f47e72070 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45c39cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f4817dc10 con 0x562f47e51390 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45c39cf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562f48181bc0 con 0x562f47e491b0 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f48217310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f48217310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56616/0 (socket says 172.21.15.16:56616) 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45affff640 1 -- 172.21.15.16:0/3179166912 learned_addr learned my addr 172.21.15.16:0/3179166912 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/3179166912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e491b0 0x562f48217850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45affff640 1 --2- 172.21.15.16:0/3179166912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e51390 0x562f48217d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45b4ff9640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45a40b4ca0 0x7f45a40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45b6ffd640 1 -- 172.21.15.16:0/3179166912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e51390 msgr2=0x562f48217d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/3179166912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e51390 0x562f48217d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45b6ffd640 1 -- 172.21.15.16:0/3179166912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 msgr2=0x562f48217310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/3179166912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f48217310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45b6ffd640 1 -- 172.21.15.16:0/3179166912 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f47edd650 con 0x562f47e491b0 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45affff640 1 --2- 172.21.15.16:0/3179166912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f48217310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45b77fe640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45a40b4ca0 0x7f45a40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.946+0000 7f45b4ff9640 1 -- 172.21.15.16:0/2656291121 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f45b00e5600 con 0x7f45b8109b90 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.947+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/3179166912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e491b0 0x562f48217850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f45b8076f00 tx=0x7f45a8032bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b21ffae9daa8103 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.947+0000 7f45ad7fa640 1 -- 172.21.15.16:0/3179166912 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8040070 con 0x562f47e491b0 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.947+0000 7f45b77fe640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45a40b4ca0 0x7f45a40b71b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f45b00e2a30 tx=0x7f45b0004020 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.947+0000 7f45ad7fa640 1 -- 172.21.15.16:0/3179166912 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a8004030 con 0x562f47e491b0 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.947+0000 7f45ad7fa640 1 -- 172.21.15.16:0/3179166912 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8035040 con 0x562f47e491b0 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.948+0000 7f45c39cf740 1 -- 172.21.15.16:0/3179166912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e491b0 msgr2=0x562f48217850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.948+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3179166912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e491b0 0x562f48217850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f45b8076f00 tx=0x7f45a8032bd0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.948+0000 7f45c39cf740 1 -- 172.21.15.16:0/3179166912 shutdown_connections 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.948+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3179166912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e51390 0x562f48217d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.948+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3179166912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e491b0 0x562f48217850 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.948+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3179166912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f48217310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.948+0000 7f45c39cf740 1 -- 172.21.15.16:0/3179166912 >> 172.21.15.16:0/3179166912 conn(0x562f47e66460 msgr2=0x562f48215200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.948+0000 7f45c39cf740 1 -- 172.21.15.16:0/3179166912 shutdown_connections 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.948+0000 7f45c39cf740 1 -- 172.21.15.16:0/3179166912 wait complete. 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.948+0000 7f45c39cf740 1 -- start start 2024-08-28T19:32:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.948+0000 7f45c39cf740 1 Processor -- start 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e491b0 0x562f48222b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e51390 0x562f48223080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f4822ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45c39cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562f47fe0700 con 0x562f47e72070 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45c39cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f48104750 con 0x562f47e491b0 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45c39cf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562f48225a90 con 0x562f47e51390 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45c39cf740 10 client.0 _ll_register_callbacks cb 0x562f48133e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f4822ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f4822ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56628/0 (socket says 172.21.15.16:56628) 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45b6ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e51390 0x562f48223080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45affff640 1 -- 172.21.15.16:0/3787868448 learned_addr learned my addr 172.21.15.16:0/3787868448 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45affff640 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e491b0 0x562f48222b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45affff640 1 -- 172.21.15.16:0/3787868448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e51390 msgr2=0x562f48223080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:59.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45affff640 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e51390 0x562f48223080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45affff640 1 -- 172.21.15.16:0/3787868448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e491b0 msgr2=0x562f48222b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:59.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45affff640 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e491b0 0x562f48222b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45affff640 1 -- 172.21.15.16:0/3787868448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f4813d900 con 0x562f47e72070 2024-08-28T19:32:59.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e51390 0x562f48223080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:32:59.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f45affff640 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f4822ed80 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f45a0012e90 tx=0x7f45a0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf237af9083eab1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:59.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.949+0000 7f459e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:59.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f459e7fc640 1 -- 172.21.15.16:0/3787868448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a0012260 con 0x562f47e72070 2024-08-28T19:32:59.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f45c39cf740 1 -- 172.21.15.16:0/3787868448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45a8004030 con 0x562f47e72070 2024-08-28T19:32:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f459e7fc640 1 -- 172.21.15.16:0/3787868448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f45a0004030 con 0x562f47e72070 2024-08-28T19:32:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f459e7fc640 1 -- 172.21.15.16:0/3787868448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f45a0004230 con 0x562f47e72070 2024-08-28T19:32:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f45c39cf740 10 client.18507 fetch_fsmap learned FSMap version 59 2024-08-28T19:32:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f45c39cf740 1 -- 172.21.15.16:0/3787868448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x562f482319a0 con 0x562f47e72070 2024-08-28T19:32:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f459e7fc640 1 -- 172.21.15.16:0/3787868448 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a8004030 con 0x562f47e72070 2024-08-28T19:32:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f459e7fc640 1 -- 172.21.15.16:0/3787868448 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7f45a0011440 con 0x562f47e72070 2024-08-28T19:32:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f45c39cf740 10 client.18507 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:32:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f45c39cf740 10 client.18507 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:32:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f45c39cf740 20 client.18507 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562f47e309f0 0x562f48234e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f45c39cf740 4 client.18507 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:32:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.950+0000 7f45c39cf740 1 -- 172.21.15.16:0/3787868448 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f481905d0 con 0x562f47e309f0 2024-08-28T19:32:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.951+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562f47e309f0 0x562f48234e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.951+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562f47e309f0 0x562f48234e70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cad4cf3f079485c1 server_cookie=7d2875dfe2057b60 in_seq=0 out_seq=0 2024-08-28T19:32:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.951+0000 7f459e7fc640 10 client.18507 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:32:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f459e7fc640 1 -- 172.21.15.16:0/3787868448 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562f481905d0 con 0x562f47e309f0 2024-08-28T19:32:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f459e7fc640 10 client.18507 handle_command_reply: tid=0 2024-08-28T19:32:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 1 client.18507 shutdown 2024-08-28T19:32:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 1 -- 172.21.15.16:0/3787868448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 msgr2=0x562f4822ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f4822ed80 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f45a0012e90 tx=0x7f45a0005f90 comp rx=0 tx=0).stop 2024-08-28T19:32:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 1 -- 172.21.15.16:0/3787868448 shutdown_connections 2024-08-28T19:32:59.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562f47e309f0 0x562f48234e70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e72070 0x562f4822ed80 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e51390 0x562f48223080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f459e7fc640 0 client.18507 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:32:59.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3787868448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e491b0 0x562f48222b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 1 -- 172.21.15.16:0/3787868448 >> 172.21.15.16:0/3787868448 conn(0x562f47e66460 msgr2=0x562f4821a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:59.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 1 -- 172.21.15.16:0/3787868448 shutdown_connections 2024-08-28T19:32:59.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 1 -- 172.21.15.16:0/3787868448 wait complete. 2024-08-28T19:32:59.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.952+0000 7f45c39cf740 20 client.18507 trim_cache size 0 max 16384 2024-08-28T19:32:59.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.965+0000 7f45c39cf740 1 Processor -- start 2024-08-28T19:32:59.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.965+0000 7f45c39cf740 1 -- start start 2024-08-28T19:32:59.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e66460 0x562f48222fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e491b0 0x562f4821a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f4821a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45c39cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f45b81b2e40 con 0x562f47e51390 2024-08-28T19:32:59.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45c39cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45b81b4aa0 con 0x562f47e66460 2024-08-28T19:32:59.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45c39cf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f45b81b35d0 con 0x562f47e491b0 2024-08-28T19:32:59.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f4821a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f4821a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56644/0 (socket says 172.21.15.16:56644) 2024-08-28T19:32:59.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45affff640 1 -- 172.21.15.16:0/3756723354 learned_addr learned my addr 172.21.15.16:0/3756723354 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:59.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45affff640 1 --2- 172.21.15.16:0/3756723354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e66460 0x562f48222fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/3756723354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e491b0 0x562f4821a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45affff640 1 -- 172.21.15.16:0/3756723354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e491b0 msgr2=0x562f4821a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45affff640 1 --2- 172.21.15.16:0/3756723354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e491b0 0x562f4821a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45affff640 1 -- 172.21.15.16:0/3756723354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e66460 msgr2=0x562f48222fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45affff640 1 --2- 172.21.15.16:0/3756723354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e66460 0x562f48222fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45affff640 1 -- 172.21.15.16:0/3756723354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f48223500 con 0x562f47e51390 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.966+0000 7f45affff640 1 --2- 172.21.15.16:0/3756723354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f4821a950 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f45a00059f0 tx=0x7f45a0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8079a0d760b308b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.967+0000 7f45acff9640 1 -- 172.21.15.16:0/3756723354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a0012260 con 0x562f47e51390 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.967+0000 7f45acff9640 1 -- 172.21.15.16:0/3756723354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a0016070 con 0x562f47e51390 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.967+0000 7f45acff9640 1 -- 172.21.15.16:0/3756723354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a0022810 con 0x562f47e51390 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 -- 172.21.15.16:0/3756723354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 msgr2=0x562f4821a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3756723354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f4821a950 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f45a00059f0 tx=0x7f45a0000bc0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 -- 172.21.15.16:0/3756723354 shutdown_connections 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3756723354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f4821a950 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3756723354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e491b0 0x562f4821a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 --2- 172.21.15.16:0/3756723354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e66460 0x562f48222fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 -- 172.21.15.16:0/3756723354 >> 172.21.15.16:0/3756723354 conn(0x562f4821e450 msgr2=0x562f482228b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 -- 172.21.15.16:0/3756723354 shutdown_connections 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 -- 172.21.15.16:0/3756723354 wait complete. 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 -- start start 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 Processor -- start 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e491b0 0x562f48226df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f48227330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e66460 0x562f48227870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f45b81b43a0 con 0x562f47e51390 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45b81b3ca0 con 0x562f47e491b0 2024-08-28T19:32:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f45b81b0e40 con 0x562f47e66460 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.968+0000 7f45c39cf740 10 client.0 _ll_register_callbacks cb 0x7f45a0009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f48227330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f48227330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56660/0 (socket says 172.21.15.16:56660) 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45b6ffd640 1 -- 172.21.15.16:0/2163672708 learned_addr learned my addr 172.21.15.16:0/2163672708 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45affff640 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e491b0 0x562f48226df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45affff640 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e66460 0x562f48227870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45b6ffd640 1 -- 172.21.15.16:0/2163672708 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e66460 msgr2=0x562f48227870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e66460 0x562f48227870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45b6ffd640 1 -- 172.21.15.16:0/2163672708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e491b0 msgr2=0x562f48226df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e491b0 0x562f48226df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45b6ffd640 1 -- 172.21.15.16:0/2163672708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f4829e950 con 0x562f47e51390 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45b6ffd640 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f48227330 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f45a8005f70 tx=0x7f45a8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71410f4507a0b71 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:32:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f459e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f459e7fc640 1 -- 172.21.15.16:0/2163672708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8035040 con 0x562f47e51390 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f459e7fc640 1 -- 172.21.15.16:0/2163672708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7f45a8004470 con 0x562f47e51390 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45c39cf740 1 -- 172.21.15.16:0/2163672708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45a0016070 con 0x562f47e51390 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f459e7fc640 1 -- 172.21.15.16:0/2163672708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f45a8064050 con 0x562f47e51390 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45c39cf740 10 client.18513 fetch_fsmap learned FSMap version 59 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.969+0000 7f45c39cf740 1 -- 172.21.15.16:0/2163672708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x562f4829fc30 con 0x562f47e51390 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.970+0000 7f459e7fc640 1 -- 172.21.15.16:0/2163672708 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a0016070 con 0x562f47e51390 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.970+0000 7f459e7fc640 1 -- 172.21.15.16:0/2163672708 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7f45a805f030 con 0x562f47e51390 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.970+0000 7f45c39cf740 10 client.18513 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.970+0000 7f45c39cf740 10 client.18513 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.970+0000 7f45c39cf740 20 client.18513 populate_metadata read hostname 'smithi016' 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.970+0000 7f45c39cf740 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562f47e72070 0x562f482a2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.970+0000 7f45c39cf740 4 client.18513 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:32:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.970+0000 7f45c39cf740 1 -- 172.21.15.16:0/2163672708 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562f4818c070 con 0x562f47e72070 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.970+0000 7f45affff640 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562f47e72070 0x562f482a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45affff640 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562f47e72070 0x562f482a2000 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5eedbf9699c50e9f server_cookie=19fb8c433b7d7251 in_seq=0 out_seq=0 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f459e7fc640 10 client.18513 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f459e7fc640 1 -- 172.21.15.16:0/2163672708 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562f4818c070 con 0x562f47e72070 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f459e7fc640 10 client.18513 handle_command_reply: tid=0 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 1 client.18513 shutdown 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 1 -- 172.21.15.16:0/2163672708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 msgr2=0x562f48227330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f48227330 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f45a8005f70 tx=0x7f45a8009260 comp rx=0 tx=0).stop 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 1 -- 172.21.15.16:0/2163672708 shutdown_connections 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562f47e72070 0x562f482a2000 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562f47e66460 0x562f48227870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f47e51390 0x562f48227330 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 1 --2- 172.21.15.16:0/2163672708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f47e491b0 0x562f48226df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f459e7fc640 0 client.18513 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:32:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 1 -- 172.21.15.16:0/2163672708 >> 172.21.15.16:0/2163672708 conn(0x562f4821e450 msgr2=0x562f48218340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 1 -- 172.21.15.16:0/2163672708 shutdown_connections 2024-08-28T19:32:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 1 -- 172.21.15.16:0/2163672708 wait complete. 2024-08-28T19:32:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.971+0000 7f45c39cf740 20 client.18513 trim_cache size 0 max 16384 2024-08-28T19:32:59.982 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:32:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 -- 172.21.15.16:0/2656291121 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45a40b4ca0 msgr2=0x7f45a40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45a40b4ca0 0x7f45a40b71b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f45b00e2a30 tx=0x7f45b0004020 comp rx=0 tx=0).stop 2024-08-28T19:32:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 -- 172.21.15.16:0/2656291121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b8109b90 msgr2=0x7f45b8075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:32:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b8109b90 0x7f45b8075f30 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009fd0 tx=0x7f45b000b260 comp rx=0 tx=0).stop 2024-08-28T19:32:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 -- 172.21.15.16:0/2656291121 shutdown_connections 2024-08-28T19:32:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b8109b90 0x7f45b8075f30 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45a40b4ca0 0x7f45a40b71b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b8109240 0x7f45b80759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 --2- 172.21.15.16:0/2656291121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b8103900 0x7f45b80754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:32:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 -- 172.21.15.16:0/2656291121 >> 172.21.15.16:0/2656291121 conn(0x7f45b807b5f0 msgr2=0x7f45b8107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:32:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 -- 172.21.15.16:0/2656291121 shutdown_connections 2024-08-28T19:32:59.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:32:59.972+0000 7f45bd45a640 1 -- 172.21.15.16:0/2656291121 wait complete. 2024-08-28T19:32:59.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 492000 3.75 MB/sec execute 519 sec latency 802.520 ms 2024-08-28T19:33:00.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:33:00.972 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:00.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 492000 3.74 MB/sec execute 520 sec latency 1802.557 ms 2024-08-28T19:33:01.429 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:01 smithi016 ceph-mon[18556]: pgmap v708: 129 pgs: 129 active+clean; 392 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 153 op/s 2024-08-28T19:33:01.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.427+0000 7fa81f3a5640 1 -- 172.21.15.16:0/716801690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa81810dbc0 msgr2=0x7fa8181100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:01.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.427+0000 7fa81f3a5640 1 --2- 172.21.15.16:0/716801690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa81810dbc0 0x7fa8181100b0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa810009ca0 tx=0x7fa81002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.427+0000 7fa81f3a5640 1 -- 172.21.15.16:0/716801690 shutdown_connections 2024-08-28T19:33:01.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.427+0000 7fa81f3a5640 1 --2- 172.21.15.16:0/716801690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa81810dbc0 0x7fa8181100b0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.427+0000 7fa81f3a5640 1 --2- 172.21.15.16:0/716801690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8181050b0 0x7fa81810d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.427+0000 7fa81f3a5640 1 --2- 172.21.15.16:0/716801690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa818104760 0x7fa818104b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.427+0000 7fa81f3a5640 1 -- 172.21.15.16:0/716801690 >> 172.21.15.16:0/716801690 conn(0x7fa8180794b0 msgr2=0x7fa8180798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.428+0000 7fa81f3a5640 1 -- 172.21.15.16:0/716801690 shutdown_connections 2024-08-28T19:33:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.428+0000 7fa81f3a5640 1 -- 172.21.15.16:0/716801690 wait complete. 2024-08-28T19:33:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.428+0000 7fa81f3a5640 1 Processor -- start 2024-08-28T19:33:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.428+0000 7fa81f3a5640 1 -- start start 2024-08-28T19:33:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81f3a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818104760 0x7fa8181024a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81f3a5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8181050b0 0x7fa8181029e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81f3a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa81810dbc0 0x7fa818102f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81f3a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa818113830 con 0x7fa818104760 2024-08-28T19:33:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81f3a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8181136b0 con 0x7fa81810dbc0 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81f3a5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa8181139b0 con 0x7fa8181050b0 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81d141640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818104760 0x7fa8181024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81d141640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818104760 0x7fa8181024a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56682/0 (socket says 172.21.15.16:56682) 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81d141640 1 -- 172.21.15.16:0/2636465756 learned_addr learned my addr 172.21.15.16:0/2636465756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81d942640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa81810dbc0 0x7fa818102f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81c940640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8181050b0 0x7fa8181029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81d141640 1 -- 172.21.15.16:0/2636465756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8181050b0 msgr2=0x7fa8181029e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81d141640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8181050b0 0x7fa8181029e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81d141640 1 -- 172.21.15.16:0/2636465756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa81810dbc0 msgr2=0x7fa818102f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81d141640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa81810dbc0 0x7fa818102f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81d141640 1 -- 172.21.15.16:0/2636465756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8181a84a0 con 0x7fa818104760 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.429+0000 7fa81d141640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818104760 0x7fa8181024a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c890 tx=0x7fa81400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.430+0000 7fa80e7fc640 1 -- 172.21.15.16:0/2636465756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814004220 con 0x7fa818104760 2024-08-28T19:33:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.430+0000 7fa80e7fc640 1 -- 172.21.15.16:0/2636465756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8140043c0 con 0x7fa818104760 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.430+0000 7fa81f3a5640 1 -- 172.21.15.16:0/2636465756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8181a8790 con 0x7fa818104760 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.430+0000 7fa80e7fc640 1 -- 172.21.15.16:0/2636465756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814004a00 con 0x7fa818104760 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.430+0000 7fa81f3a5640 1 -- 172.21.15.16:0/2636465756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8181a8cd0 con 0x7fa818104760 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.432+0000 7fa82591a740 1 Processor -- start 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.432+0000 7fa82591a740 1 -- start start 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa82591a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56c1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa82591a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa82591a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52fb390 0x55a5c56c1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa80e7fc640 1 -- 172.21.15.16:0/2636465756 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa8140079b0 con 0x7fa818104760 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa82591a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5c5638680 con 0x55a5c531c070 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa82591a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5c5627c10 con 0x55a5c52fb390 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa82591a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a5c562bbc0 con 0x55a5c52f31b0 2024-08-28T19:33:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81d942640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56c1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81d942640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56c1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56684/0 (socket says 172.21.15.16:56684) 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81d942640 1 -- 172.21.15.16:0/267679917 learned_addr learned my addr 172.21.15.16:0/267679917 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81c940640 1 --2- 172.21.15.16:0/267679917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52fb390 0x55a5c56c1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81d141640 1 --2- 172.21.15.16:0/267679917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81d942640 1 -- 172.21.15.16:0/267679917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 msgr2=0x55a5c56c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81d942640 1 --2- 172.21.15.16:0/267679917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81d942640 1 -- 172.21.15.16:0/267679917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52fb390 msgr2=0x55a5c56c1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81d942640 1 --2- 172.21.15.16:0/267679917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52fb390 0x55a5c56c1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81d942640 1 -- 172.21.15.16:0/267679917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5c5387650 con 0x55a5c531c070 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81d141640 1 --2- 172.21.15.16:0/267679917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.433+0000 7fa81d942640 1 --2- 172.21.15.16:0/267679917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56c1310 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fa810009920 tx=0x7fa810009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e8308245f23c5a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.434+0000 7fa8037fe640 1 -- 172.21.15.16:0/267679917 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810031e30 con 0x55a5c531c070 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.434+0000 7fa80e7fc640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8040b4ca0 0x7fa8040b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.434+0000 7fa80e7fc640 1 -- 172.21.15.16:0/2636465756 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fa8140e5620 con 0x7fa818104760 2024-08-28T19:33:01.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.434+0000 7fa8037fe640 1 -- 172.21.15.16:0/267679917 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa810036030 con 0x55a5c531c070 2024-08-28T19:33:01.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.434+0000 7fa8037fe640 1 -- 172.21.15.16:0/267679917 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810004280 con 0x55a5c531c070 2024-08-28T19:33:01.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.434+0000 7fa81d141640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8040b4ca0 0x7fa8040b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.435+0000 7fa82591a740 1 -- 172.21.15.16:0/267679917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 msgr2=0x55a5c56c1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:01.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.435+0000 7fa81d141640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8040b4ca0 0x7fa8040b71b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa8140e2a50 tx=0x7fa814000bc0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:01.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.435+0000 7fa82591a740 1 --2- 172.21.15.16:0/267679917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56c1310 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fa810009920 tx=0x7fa810009950 comp rx=0 tx=0).stop 2024-08-28T19:33:01.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.435+0000 7fa82591a740 1 -- 172.21.15.16:0/267679917 shutdown_connections 2024-08-28T19:33:01.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.435+0000 7fa82591a740 1 --2- 172.21.15.16:0/267679917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52fb390 0x55a5c56c1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.435+0000 7fa82591a740 1 --2- 172.21.15.16:0/267679917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.435+0000 7fa82591a740 1 --2- 172.21.15.16:0/267679917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56c1310 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.435+0000 7fa82591a740 1 -- 172.21.15.16:0/267679917 >> 172.21.15.16:0/267679917 conn(0x55a5c5310460 msgr2=0x55a5c56bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:01.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.435+0000 7fa82591a740 1 -- 172.21.15.16:0/267679917 shutdown_connections 2024-08-28T19:33:01.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.435+0000 7fa82591a740 1 -- 172.21.15.16:0/267679917 wait complete. 2024-08-28T19:33:01.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.436+0000 7fa82591a740 1 -- start start 2024-08-28T19:33:01.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.436+0000 7fa82591a740 1 Processor -- start 2024-08-28T19:33:01.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.436+0000 7fa82591a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.436+0000 7fa82591a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c52fb390 0x55a5c56ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.436+0000 7fa82591a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c531c070 0x55a5c56d8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.436+0000 7fa82591a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5c548a700 con 0x55a5c52fb390 2024-08-28T19:33:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.436+0000 7fa82591a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5c55ae750 con 0x55a5c531c070 2024-08-28T19:33:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.436+0000 7fa82591a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a5c56cf910 con 0x55a5c52f31b0 2024-08-28T19:33:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.436+0000 7fa82591a740 10 client.0 _ll_register_callbacks cb 0x55a5c55dde10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.436+0000 7fa81c940640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c52fb390 0x55a5c56ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa81c940640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c52fb390 0x55a5c56ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56694/0 (socket says 172.21.15.16:56694) 2024-08-28T19:33:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa81c940640 1 -- 172.21.15.16:0/2902217299 learned_addr learned my addr 172.21.15.16:0/2902217299 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa81d942640 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa81d942640 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c531c070 0x55a5c56d8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa81c940640 1 -- 172.21.15.16:0/2902217299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 msgr2=0x55a5c56ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa81c940640 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa81c940640 1 -- 172.21.15.16:0/2902217299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c531c070 msgr2=0x55a5c56d8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa81c940640 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c531c070 0x55a5c56d8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa81c940640 1 -- 172.21.15.16:0/2902217299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5c55e7900 con 0x55a5c52fb390 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa81c940640 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c52fb390 0x55a5c56ccff0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fa80800be80 tx=0x7fa808012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6785a8b5522572f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa800ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa800ff9640 1 -- 172.21.15.16:0/2902217299 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808005220 con 0x55a5c52fb390 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa800ff9640 1 -- 172.21.15.16:0/2902217299 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fa8080053c0 con 0x55a5c52fb390 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.437+0000 7fa82591a740 1 -- 172.21.15.16:0/2902217299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa810036030 con 0x55a5c52fb390 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.438+0000 7fa800ff9640 1 -- 172.21.15.16:0/2902217299 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fa8080055c0 con 0x55a5c52fb390 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.438+0000 7fa82591a740 10 client.18525 fetch_fsmap learned FSMap version 59 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.438+0000 7fa82591a740 1 -- 172.21.15.16:0/2902217299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55a5c56db6c0 con 0x55a5c52fb390 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.438+0000 7fa800ff9640 1 -- 172.21.15.16:0/2902217299 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa810036030 con 0x55a5c52fb390 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.438+0000 7fa800ff9640 1 -- 172.21.15.16:0/2902217299 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7fa808019070 con 0x55a5c52fb390 2024-08-28T19:33:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.438+0000 7fa82591a740 10 client.18525 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.438+0000 7fa82591a740 10 client.18525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.438+0000 7fa82591a740 20 client.18525 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.438+0000 7fa82591a740 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5c52da9f0 0x55a5c56df110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.438+0000 7fa82591a740 4 client.18525 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.438+0000 7fa82591a740 1 -- 172.21.15.16:0/2902217299 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5c563a5d0 con 0x55a5c52da9f0 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.439+0000 7fa81d942640 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5c52da9f0 0x55a5c56df110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.439+0000 7fa81d942640 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5c52da9f0 0x55a5c56df110 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4da35aa1889a2d6b server_cookie=160ea1125ca75f20 in_seq=0 out_seq=0 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.439+0000 7fa800ff9640 10 client.18525 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa800ff9640 1 -- 172.21.15.16:0/2902217299 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a5c563a5d0 con 0x55a5c52da9f0 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa800ff9640 10 client.18525 handle_command_reply: tid=0 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 1 client.18525 shutdown 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 1 -- 172.21.15.16:0/2902217299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c52fb390 msgr2=0x55a5c56ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c52fb390 0x55a5c56ccff0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fa80800be80 tx=0x7fa808012820 comp rx=0 tx=0).stop 2024-08-28T19:33:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 1 -- 172.21.15.16:0/2902217299 shutdown_connections 2024-08-28T19:33:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c531c070 0x55a5c56d8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c52fb390 0x55a5c56ccff0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5c52da9f0 0x55a5c56df110 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 1 --2- 172.21.15.16:0/2902217299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 1 -- 172.21.15.16:0/2902217299 >> 172.21.15.16:0/2902217299 conn(0x55a5c5310460 msgr2=0x55a5c56c44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa800ff9640 0 client.18525 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 1 -- 172.21.15.16:0/2902217299 shutdown_connections 2024-08-28T19:33:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 1 -- 172.21.15.16:0/2902217299 wait complete. 2024-08-28T19:33:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.440+0000 7fa82591a740 20 client.18525 trim_cache size 0 max 16384 2024-08-28T19:33:01.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.454+0000 7fa82591a740 1 Processor -- start 2024-08-28T19:33:01.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.454+0000 7fa82591a740 1 -- start start 2024-08-28T19:33:01.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.454+0000 7fa82591a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.454+0000 7fa82591a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52fb390 0x55a5c56d8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.454+0000 7fa82591a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56d9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.454+0000 7fa82591a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8181b2e80 con 0x55a5c531c070 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.454+0000 7fa82591a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8181b4ae0 con 0x55a5c52fb390 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.454+0000 7fa82591a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa8181b3610 con 0x55a5c52f31b0 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.454+0000 7fa81d942640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56d9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.454+0000 7fa81d942640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56d9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56700/0 (socket says 172.21.15.16:56700) 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.454+0000 7fa81d942640 1 -- 172.21.15.16:0/3353154162 learned_addr learned my addr 172.21.15.16:0/3353154162 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.455+0000 7fa81c940640 1 --2- 172.21.15.16:0/3353154162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52fb390 0x55a5c56d8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.455+0000 7fa81d942640 1 --2- 172.21.15.16:0/3353154162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.455+0000 7fa81d942640 1 -- 172.21.15.16:0/3353154162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 msgr2=0x55a5c56ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.455+0000 7fa81d942640 1 --2- 172.21.15.16:0/3353154162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.455+0000 7fa81d942640 1 -- 172.21.15.16:0/3353154162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52fb390 msgr2=0x55a5c56d8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.455+0000 7fa81d942640 1 --2- 172.21.15.16:0/3353154162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52fb390 0x55a5c56d8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.455+0000 7fa81d942640 1 -- 172.21.15.16:0/3353154162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5c56c3ba0 con 0x55a5c531c070 2024-08-28T19:33:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.455+0000 7fa81d942640 1 --2- 172.21.15.16:0/3353154162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56d9040 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fa81002f470 tx=0x7fa810056040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a2eee7215ba8ce server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.455+0000 7fa802ffd640 1 -- 172.21.15.16:0/3353154162 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810060070 con 0x55a5c531c070 2024-08-28T19:33:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.455+0000 7fa802ffd640 1 -- 172.21.15.16:0/3353154162 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa810036070 con 0x55a5c531c070 2024-08-28T19:33:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.455+0000 7fa802ffd640 1 -- 172.21.15.16:0/3353154162 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810004730 con 0x55a5c531c070 2024-08-28T19:33:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.456+0000 7fa82591a740 1 -- 172.21.15.16:0/3353154162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 msgr2=0x55a5c56d9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.456+0000 7fa82591a740 1 --2- 172.21.15.16:0/3353154162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56d9040 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fa81002f470 tx=0x7fa810056040 comp rx=0 tx=0).stop 2024-08-28T19:33:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.456+0000 7fa82591a740 1 -- 172.21.15.16:0/3353154162 shutdown_connections 2024-08-28T19:33:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.456+0000 7fa82591a740 1 --2- 172.21.15.16:0/3353154162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c56d9040 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.456+0000 7fa82591a740 1 --2- 172.21.15.16:0/3353154162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52fb390 0x55a5c56d8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.456+0000 7fa82591a740 1 --2- 172.21.15.16:0/3353154162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52f31b0 0x55a5c56ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.456+0000 7fa82591a740 1 -- 172.21.15.16:0/3353154162 >> 172.21.15.16:0/3353154162 conn(0x55a5c5310460 msgr2=0x55a5c56cc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.456+0000 7fa82591a740 1 -- 172.21.15.16:0/3353154162 shutdown_connections 2024-08-28T19:33:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.456+0000 7fa82591a740 1 -- 172.21.15.16:0/3353154162 wait complete. 2024-08-28T19:33:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.456+0000 7fa82591a740 1 -- start start 2024-08-28T19:33:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa82591a740 1 Processor -- start 2024-08-28T19:33:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa82591a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52f31b0 0x55a5c56d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa82591a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52fb390 0x55a5c56d0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa82591a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c5741ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa82591a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8181b43e0 con 0x55a5c531c070 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa82591a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8181b3ce0 con 0x55a5c52f31b0 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa82591a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa8181b0e80 con 0x55a5c52fb390 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa82591a740 10 client.0 _ll_register_callbacks cb 0x7fa80800bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa81d942640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c5741ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa81d942640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c5741ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56706/0 (socket says 172.21.15.16:56706) 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa81d942640 1 -- 172.21.15.16:0/1938714025 learned_addr learned my addr 172.21.15.16:0/1938714025 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa81d942640 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52f31b0 0x55a5c56d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.457+0000 7fa81c940640 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52fb390 0x55a5c56d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa81d942640 1 -- 172.21.15.16:0/1938714025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52fb390 msgr2=0x55a5c56d0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa81d942640 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52fb390 0x55a5c56d0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa81d942640 1 -- 172.21.15.16:0/1938714025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52f31b0 msgr2=0x55a5c56d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa81d942640 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52f31b0 0x55a5c56d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa81d942640 1 -- 172.21.15.16:0/1938714025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5c5742140 con 0x55a5c531c070 2024-08-28T19:33:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa81d942640 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c5741ad0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fa81002ade0 tx=0x7fa810009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa2e10ecc438e90 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa800ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa800ff9640 1 -- 172.21.15.16:0/1938714025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810060040 con 0x55a5c531c070 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa800ff9640 1 -- 172.21.15.16:0/1938714025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9490+0+0 (secure 0 0 0) 0x7fa810004b40 con 0x55a5c531c070 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa82591a740 1 -- 172.21.15.16:0/1938714025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa810036070 con 0x55a5c531c070 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa800ff9640 1 -- 172.21.15.16:0/1938714025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fa8100041e0 con 0x55a5c531c070 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa82591a740 10 client.18531 fetch_fsmap learned FSMap version 59 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa82591a740 1 -- 172.21.15.16:0/1938714025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55a5c5749460 con 0x55a5c531c070 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa800ff9640 1 -- 172.21.15.16:0/1938714025 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa810036070 con 0x55a5c531c070 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.458+0000 7fa800ff9640 1 -- 172.21.15.16:0/1938714025 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7fa810066070 con 0x55a5c531c070 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.459+0000 7fa82591a740 10 client.18531 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.459+0000 7fa82591a740 10 client.18531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.459+0000 7fa82591a740 20 client.18531 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.459+0000 7fa82591a740 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5c52afda0 0x55a5c574b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.459+0000 7fa82591a740 4 client.18531 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.459+0000 7fa82591a740 1 -- 172.21.15.16:0/1938714025 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5c5636070 con 0x55a5c52afda0 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.459+0000 7fa81c940640 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5c52afda0 0x55a5c574b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.459+0000 7fa800ff9640 10 client.18531 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.459+0000 7fa81c940640 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5c52afda0 0x55a5c574b830 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d30a420fc0c8b197 server_cookie=dbdeec95a1849762 in_seq=0 out_seq=0 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa800ff9640 1 -- 172.21.15.16:0/1938714025 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5c5636070 con 0x55a5c52afda0 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa800ff9640 10 client.18531 handle_command_reply: tid=0 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 1 client.18531 shutdown 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 1 -- 172.21.15.16:0/1938714025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 msgr2=0x55a5c5741ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c5741ad0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fa81002ade0 tx=0x7fa810009800 comp rx=0 tx=0).stop 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 1 -- 172.21.15.16:0/1938714025 shutdown_connections 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5c531c070 0x55a5c5741ad0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a5c52afda0 0x55a5c574b830 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5c52fb390 0x55a5c56d0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa800ff9640 0 client.18531 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 1 --2- 172.21.15.16:0/1938714025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5c52f31b0 0x55a5c56d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 1 -- 172.21.15.16:0/1938714025 >> 172.21.15.16:0/1938714025 conn(0x55a5c5310460 msgr2=0x55a5c56be9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 1 -- 172.21.15.16:0/1938714025 shutdown_connections 2024-08-28T19:33:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 1 -- 172.21.15.16:0/1938714025 wait complete. 2024-08-28T19:33:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.460+0000 7fa82591a740 20 client.18531 trim_cache size 0 max 16384 2024-08-28T19:33:01.466 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:33:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 -- 172.21.15.16:0/2636465756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8040b4ca0 msgr2=0x7fa8040b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8040b4ca0 0x7fa8040b71b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa8140e2a50 tx=0x7fa814000bc0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 -- 172.21.15.16:0/2636465756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818104760 msgr2=0x7fa8181024a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818104760 0x7fa8181024a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c890 tx=0x7fa81400cd50 comp rx=0 tx=0).stop 2024-08-28T19:33:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 -- 172.21.15.16:0/2636465756 shutdown_connections 2024-08-28T19:33:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa8040b4ca0 0x7fa8040b71b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa81810dbc0 0x7fa818102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa8181050b0 0x7fa8181029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 --2- 172.21.15.16:0/2636465756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818104760 0x7fa8181024a0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 -- 172.21.15.16:0/2636465756 >> 172.21.15.16:0/2636465756 conn(0x7fa8180794b0 msgr2=0x7fa8181055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 -- 172.21.15.16:0/2636465756 shutdown_connections 2024-08-28T19:33:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:01.461+0000 7fa81f3a5640 1 -- 172.21.15.16:0/2636465756 wait complete. 2024-08-28T19:33:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:01 smithi115 ceph-mon[22184]: pgmap v708: 129 pgs: 129 active+clean; 392 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 153 op/s 2024-08-28T19:33:01.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:01 smithi028 ceph-mon[21769]: pgmap v708: 129 pgs: 129 active+clean; 392 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 153 op/s 2024-08-28T19:33:01.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 492003 3.73 MB/sec execute 521 sec latency 2799.804 ms 2024-08-28T19:33:02.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 492637 3.73 MB/sec execute 522 sec latency 63.721 ms 2024-08-28T19:33:03.488 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:33:03.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:33:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:03 smithi115 ceph-mon[22184]: pgmap v709: 129 pgs: 129 active+clean; 393 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 162 op/s 2024-08-28T19:33:03.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:03 smithi016 ceph-mon[18556]: pgmap v709: 129 pgs: 129 active+clean; 393 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 162 op/s 2024-08-28T19:33:03.758 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:03.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:03 smithi028 ceph-mon[21769]: pgmap v709: 129 pgs: 129 active+clean; 393 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 162 op/s 2024-08-28T19:33:03.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 494179 3.75 MB/sec execute 523 sec latency 37.170 ms 2024-08-28T19:33:04.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.255+0000 7f4379c8d640 1 -- 172.21.15.16:0/2977086820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43741081c0 msgr2=0x7f43741085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:04.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.255+0000 7f4379c8d640 1 --2- 172.21.15.16:0/2977086820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43741081c0 0x7f43741085d0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f4368009900 tx=0x7f436802a410 comp rx=0 tx=0).stop 2024-08-28T19:33:04.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.255+0000 7f4379c8d640 1 -- 172.21.15.16:0/2977086820 shutdown_connections 2024-08-28T19:33:04.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.255+0000 7f4379c8d640 1 --2- 172.21.15.16:0/2977086820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4374108b10 0x7f437410f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.255+0000 7f4379c8d640 1 --2- 172.21.15.16:0/2977086820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43741081c0 0x7f43741085d0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.255+0000 7f4379c8d640 1 --2- 172.21.15.16:0/2977086820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4374102880 0x7f4374102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.255+0000 7f4379c8d640 1 -- 172.21.15.16:0/2977086820 >> 172.21.15.16:0/2977086820 conn(0x7f4374079510 msgr2=0x7f4374079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.256+0000 7f4379c8d640 1 -- 172.21.15.16:0/2977086820 shutdown_connections 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.256+0000 7f4379c8d640 1 -- 172.21.15.16:0/2977086820 wait complete. 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.256+0000 7f4379c8d640 1 Processor -- start 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.256+0000 7f4379c8d640 1 -- start start 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4379c8d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4374102880 0x7f437419b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4379c8d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f43741081c0 0x7f437419b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4379c8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4374108b10 0x7f437419bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4379c8d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f43741127f0 con 0x7f4374108b10 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4379c8d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4374112670 con 0x7f4374102880 2024-08-28T19:33:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4379c8d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4374112970 con 0x7f43741081c0 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4373fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4374108b10 0x7f437419bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f43737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4374102880 0x7f437419b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4373fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4374108b10 0x7f437419bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56728/0 (socket says 172.21.15.16:56728) 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f43737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4374102880 0x7f437419b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:53440/0 (socket says 172.21.15.16:53440) 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4373fff640 1 -- 172.21.15.16:0/1206608626 learned_addr learned my addr 172.21.15.16:0/1206608626 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4372ffd640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f43741081c0 0x7f437419b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4373fff640 1 -- 172.21.15.16:0/1206608626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f43741081c0 msgr2=0x7f437419b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4373fff640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f43741081c0 0x7f437419b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4373fff640 1 -- 172.21.15.16:0/1206608626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4374102880 msgr2=0x7f437419b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4373fff640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4374102880 0x7f437419b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4373fff640 1 -- 172.21.15.16:0/1206608626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43741a6410 con 0x7f4374108b10 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4373fff640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4374108b10 0x7f437419bc00 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f436c00b970 tx=0x7f436c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.257+0000 7f4370ff9640 1 -- 172.21.15.16:0/1206608626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c019070 con 0x7f4374108b10 2024-08-28T19:33:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.258+0000 7f4379c8d640 1 -- 172.21.15.16:0/1206608626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43741a6700 con 0x7f4374108b10 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.258+0000 7f4370ff9640 1 -- 172.21.15.16:0/1206608626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f436c014440 con 0x7f4374108b10 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.258+0000 7f4379c8d640 1 -- 172.21.15.16:0/1206608626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43741a6c60 con 0x7f4374108b10 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.258+0000 7f4370ff9640 1 -- 172.21.15.16:0/1206608626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c0075d0 con 0x7f4374108b10 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.260+0000 7f4380202740 1 Processor -- start 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.260+0000 7f4370ff9640 1 -- 172.21.15.16:0/1206608626 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f436c01c070 con 0x7f4374108b10 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.260+0000 7f4380202740 1 -- start start 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f4380202740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38396350 0x556d3873c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f4380202740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3873ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f4380202740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38375670 0x556d3873d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f4380202740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556d386b25f0 con 0x556d3836d490 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f4380202740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d386a1b80 con 0x556d38375670 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f4380202740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556d386a5b30 con 0x556d38396350 2024-08-28T19:33:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f43737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3873ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f43737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3873ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56742/0 (socket says 172.21.15.16:56742) 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f43737fe640 1 -- 172.21.15.16:0/1420609653 learned_addr learned my addr 172.21.15.16:0/1420609653 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f4372ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38396350 0x556d3873c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f4372ffd640 1 --2- 172.21.15.16:0/1420609653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38375670 0x556d3873d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f4370ff9640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f435c0b4cd0 0x7f435c0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f43737fe640 1 -- 172.21.15.16:0/1420609653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38396350 msgr2=0x556d3873c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f43737fe640 1 --2- 172.21.15.16:0/1420609653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38396350 0x556d3873c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f43737fe640 1 -- 172.21.15.16:0/1420609653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38375670 msgr2=0x556d3873d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f43737fe640 1 --2- 172.21.15.16:0/1420609653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38375670 0x556d3873d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f43737fe640 1 -- 172.21.15.16:0/1420609653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d38401930 con 0x556d3836d490 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f4373fff640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f435c0b4cd0 0x7f435c0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f4370ff9640 1 -- 172.21.15.16:0/1206608626 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9578+0+0 (secure 0 0 0) 0x7f436c0e4af0 con 0x7f4374108b10 2024-08-28T19:33:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.261+0000 7f43737fe640 1 --2- 172.21.15.16:0/1420609653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3873ccc0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f4364009c90 tx=0x7f4364011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ba1d833aea3d35 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.262+0000 7f4359ffb640 1 -- 172.21.15.16:0/1420609653 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364009e40 con 0x556d3836d490 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.262+0000 7f4359ffb640 1 -- 172.21.15.16:0/1420609653 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43640054d0 con 0x556d3836d490 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.262+0000 7f4359ffb640 1 -- 172.21.15.16:0/1420609653 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364019670 con 0x556d3836d490 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.263+0000 7f4380202740 1 -- 172.21.15.16:0/1420609653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 msgr2=0x556d3873ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.263+0000 7f4373fff640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f435c0b4cd0 0x7f435c0b71e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f436c0e1ed0 tx=0x7f436c00e280 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.263+0000 7f4380202740 1 --2- 172.21.15.16:0/1420609653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3873ccc0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f4364009c90 tx=0x7f4364011810 comp rx=0 tx=0).stop 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.263+0000 7f4380202740 1 -- 172.21.15.16:0/1420609653 shutdown_connections 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.263+0000 7f4380202740 1 --2- 172.21.15.16:0/1420609653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38375670 0x556d3873d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.263+0000 7f4380202740 1 --2- 172.21.15.16:0/1420609653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3873ccc0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.263+0000 7f4380202740 1 --2- 172.21.15.16:0/1420609653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38396350 0x556d3873c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.263+0000 7f4380202740 1 -- 172.21.15.16:0/1420609653 >> 172.21.15.16:0/1420609653 conn(0x556d3838a740 msgr2=0x556d3873a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.263+0000 7f4380202740 1 -- 172.21.15.16:0/1420609653 shutdown_connections 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.263+0000 7f4380202740 1 -- 172.21.15.16:0/1420609653 wait complete. 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.263+0000 7f4380202740 1 -- start start 2024-08-28T19:33:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4380202740 1 Processor -- start 2024-08-28T19:33:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4380202740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d3836d490 0x556d38747ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4380202740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38375670 0x556d38748410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4380202740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38396350 0x556d387542e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4380202740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556d38504890 con 0x556d38375670 2024-08-28T19:33:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4380202740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d38628790 con 0x556d38396350 2024-08-28T19:33:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4380202740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556d3874af60 con 0x556d3836d490 2024-08-28T19:33:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4380202740 10 client.0 _ll_register_callbacks cb 0x556d383adba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f43737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38375670 0x556d38748410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4372ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d3836d490 0x556d38747ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f43737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38375670 0x556d38748410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36132/0 (socket says 172.21.15.16:36132) 2024-08-28T19:33:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f43737fe640 1 -- 172.21.15.16:0/2304823140 learned_addr learned my addr 172.21.15.16:0/2304823140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4372ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d3836d490 0x556d38747ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:40510/0 (socket says 172.21.15.16:40510) 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4372ffd640 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38396350 0x556d387542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f43737fe640 1 -- 172.21.15.16:0/2304823140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d3836d490 msgr2=0x556d38747ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f43737fe640 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d3836d490 0x556d38747ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f43737fe640 1 -- 172.21.15.16:0/2304823140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38396350 msgr2=0x556d387542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f43737fe640 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38396350 0x556d387542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f43737fe640 1 -- 172.21.15.16:0/2304823140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d3865ff30 con 0x556d38375670 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.264+0000 7f4372ffd640 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d3836d490 0x556d38747ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.265+0000 7f43737fe640 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38375670 0x556d38748410 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f4364012fd0 tx=0x7f4364005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc546a6522dc86ce server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.265+0000 7f4352ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.265+0000 7f4352ffd640 1 -- 172.21.15.16:0/2304823140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364009e40 con 0x556d38375670 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.265+0000 7f4380202740 1 -- 172.21.15.16:0/2304823140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43640054d0 con 0x556d38375670 2024-08-28T19:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.265+0000 7f4352ffd640 1 -- 172.21.15.16:0/2304823140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9578+0+0 (secure 0 0 0) 0x7f4364004030 con 0x556d38375670 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.265+0000 7f4352ffd640 1 -- 172.21.15.16:0/2304823140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f4364033050 con 0x556d38375670 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.265+0000 7f4380202740 10 client.18543 fetch_fsmap learned FSMap version 59 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.265+0000 7f4380202740 1 -- 172.21.15.16:0/2304823140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x556d38756d30 con 0x556d38375670 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.265+0000 7f4352ffd640 1 -- 172.21.15.16:0/2304823140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43640054d0 con 0x556d38375670 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.266+0000 7f4352ffd640 1 -- 172.21.15.16:0/2304823140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7f4364003aa0 con 0x556d38375670 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.266+0000 7f4380202740 10 client.18543 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.266+0000 7f4380202740 10 client.18543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.266+0000 7f4380202740 20 client.18543 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.266+0000 7f4380202740 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d38354cd0 0x556d3875a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.266+0000 7f4380202740 4 client.18543 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.266+0000 7f4380202740 1 -- 172.21.15.16:0/2304823140 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d386b4540 con 0x556d38354cd0 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.266+0000 7f4372ffd640 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d38354cd0 0x556d3875a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.266+0000 7f4372ffd640 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d38354cd0 0x556d3875a1a0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3106b13c98a32ee server_cookie=4fde667464b893b3 in_seq=0 out_seq=0 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.266+0000 7f4352ffd640 10 client.18543 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4352ffd640 1 -- 172.21.15.16:0/2304823140 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556d386b4540 con 0x556d38354cd0 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4352ffd640 10 client.18543 handle_command_reply: tid=0 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4380202740 1 client.18543 shutdown 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4380202740 1 -- 172.21.15.16:0/2304823140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38375670 msgr2=0x556d38748410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4380202740 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38375670 0x556d38748410 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f4364012fd0 tx=0x7f4364005060 comp rx=0 tx=0).stop 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4380202740 1 -- 172.21.15.16:0/2304823140 shutdown_connections 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4380202740 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38396350 0x556d387542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4380202740 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38375670 0x556d38748410 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4380202740 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d38354cd0 0x556d3875a1a0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4380202740 1 --2- 172.21.15.16:0/2304823140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d3836d490 0x556d38747ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4352ffd640 0 client.18543 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.267+0000 7f4380202740 1 -- 172.21.15.16:0/2304823140 >> 172.21.15.16:0/2304823140 conn(0x556d3838a740 msgr2=0x556d3873f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:04.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.268+0000 7f4380202740 1 -- 172.21.15.16:0/2304823140 shutdown_connections 2024-08-28T19:33:04.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.268+0000 7f4380202740 1 -- 172.21.15.16:0/2304823140 wait complete. 2024-08-28T19:33:04.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.268+0000 7f4380202740 20 client.18543 trim_cache size 0 max 16384 2024-08-28T19:33:04.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4380202740 1 Processor -- start 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4380202740 1 -- start start 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4380202740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d3836d490 0x556d387436b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4380202740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38375670 0x556d3873dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4380202740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38396350 0x556d387541e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4380202740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4360010100 con 0x556d38396350 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4380202740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4360011d60 con 0x556d3836d490 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4380202740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4360010890 con 0x556d38375670 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4372ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d3836d490 0x556d387436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4372ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d3836d490 0x556d387436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:56342/0 (socket says 172.21.15.16:56342) 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4372ffd640 1 -- 172.21.15.16:0/1698783124 learned_addr learned my addr 172.21.15.16:0/1698783124 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4372ffd640 1 --2- 172.21.15.16:0/1698783124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38396350 0x556d387541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f43737fe640 1 --2- 172.21.15.16:0/1698783124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38375670 0x556d3873dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4372ffd640 1 -- 172.21.15.16:0/1698783124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38375670 msgr2=0x556d3873dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4372ffd640 1 --2- 172.21.15.16:0/1698783124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38375670 0x556d3873dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4372ffd640 1 -- 172.21.15.16:0/1698783124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38396350 msgr2=0x556d387541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.288+0000 7f4372ffd640 1 --2- 172.21.15.16:0/1698783124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38396350 0x556d387541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.289+0000 7f4372ffd640 1 -- 172.21.15.16:0/1698783124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d38748a40 con 0x556d3836d490 2024-08-28T19:33:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.289+0000 7f4372ffd640 1 --2- 172.21.15.16:0/1698783124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d3836d490 0x556d387436b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4368009900 tx=0x7f4368000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af058c74ee7d398a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:04.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.289+0000 7f43597fa640 1 -- 172.21.15.16:0/1698783124 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4368035040 con 0x556d3836d490 2024-08-28T19:33:04.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.289+0000 7f43597fa640 1 -- 172.21.15.16:0/1698783124 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4368042020 con 0x556d3836d490 2024-08-28T19:33:04.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.289+0000 7f43597fa640 1 -- 172.21.15.16:0/1698783124 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436804e3c0 con 0x556d3836d490 2024-08-28T19:33:04.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.290+0000 7f4380202740 1 -- 172.21.15.16:0/1698783124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d3836d490 msgr2=0x556d387436b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:04.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.290+0000 7f4380202740 1 --2- 172.21.15.16:0/1698783124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d3836d490 0x556d387436b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4368009900 tx=0x7f4368000bc0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.290+0000 7f4380202740 1 -- 172.21.15.16:0/1698783124 shutdown_connections 2024-08-28T19:33:04.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.290+0000 7f4380202740 1 --2- 172.21.15.16:0/1698783124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d38396350 0x556d387541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.290+0000 7f4380202740 1 --2- 172.21.15.16:0/1698783124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38375670 0x556d3873dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.290+0000 7f4380202740 1 --2- 172.21.15.16:0/1698783124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d3836d490 0x556d387436b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.290+0000 7f4380202740 1 -- 172.21.15.16:0/1698783124 >> 172.21.15.16:0/1698783124 conn(0x556d3838a740 msgr2=0x556d3873d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.290+0000 7f4380202740 1 -- 172.21.15.16:0/1698783124 shutdown_connections 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.290+0000 7f4380202740 1 -- 172.21.15.16:0/1698783124 wait complete. 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.290+0000 7f4380202740 1 -- start start 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4380202740 1 Processor -- start 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4380202740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3874c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4380202740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38375670 0x556d3874ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4380202740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38396350 0x556d387c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4380202740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4360011660 con 0x556d3836d490 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4380202740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4360010f60 con 0x556d38396350 2024-08-28T19:33:04.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4380202740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f436000e100 con 0x556d38375670 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4380202740 10 client.0 _ll_register_callbacks cb 0x7f4368032d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4372ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3874c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4372ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3874c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36148/0 (socket says 172.21.15.16:36148) 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4372ffd640 1 -- 172.21.15.16:0/1989935432 learned_addr learned my addr 172.21.15.16:0/1989935432 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f43737fe640 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38375670 0x556d3874ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f4372ffd640 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38396350 0x556d387c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f43737fe640 1 -- 172.21.15.16:0/1989935432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38396350 msgr2=0x556d387c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f43737fe640 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38396350 0x556d387c3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f43737fe640 1 -- 172.21.15.16:0/1989935432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 msgr2=0x556d3874c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.291+0000 7f43737fe640 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3874c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.292+0000 7f43737fe640 1 -- 172.21.15.16:0/1989935432 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d3873ed90 con 0x556d38375670 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.292+0000 7f4372ffd640 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3874c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.292+0000 7f43737fe640 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38375670 0x556d3874ca70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4364000be0 tx=0x7f43640099d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa9a9be9ee7c8bd4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:04.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.292+0000 7f4352ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.292+0000 7f4352ffd640 1 -- 172.21.15.16:0/1989935432 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364005b90 con 0x556d38375670 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.292+0000 7f4352ffd640 1 -- 172.21.15.16:0/1989935432 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9578+0+0 (secure 0 0 0) 0x7f4364005460 con 0x556d38375670 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.292+0000 7f4380202740 1 -- 172.21.15.16:0/1989935432 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4368042020 con 0x556d38375670 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.292+0000 7f4352ffd640 1 -- 172.21.15.16:0/1989935432 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f4364024490 con 0x556d38375670 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.292+0000 7f4380202740 10 client.27703 fetch_fsmap learned FSMap version 59 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.292+0000 7f4380202740 1 -- 172.21.15.16:0/1989935432 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x556d387c4ee0 con 0x556d38375670 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.292+0000 7f4352ffd640 1 -- 172.21.15.16:0/1989935432 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4368042020 con 0x556d38375670 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.293+0000 7f4352ffd640 1 -- 172.21.15.16:0/1989935432 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7f4364024b60 con 0x556d38375670 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.293+0000 7f4380202740 10 client.27703 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.293+0000 7f4380202740 10 client.27703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.293+0000 7f4380202740 20 client.27703 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.293+0000 7f4380202740 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d3832a080 0x556d387c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.293+0000 7f4380202740 4 client.27703 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:33:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.293+0000 7f4380202740 1 -- 172.21.15.16:0/1989935432 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556d386affe0 con 0x556d3832a080 2024-08-28T19:33:04.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.293+0000 7f4372ffd640 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d3832a080 0x556d387c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:04.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.293+0000 7f4352ffd640 10 client.27703 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:04.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.293+0000 7f4372ffd640 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d3832a080 0x556d387c75b0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5384533adb0641c server_cookie=cb6b61163fbbb381 in_seq=0 out_seq=0 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4352ffd640 1 -- 172.21.15.16:0/1989935432 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556d386affe0 con 0x556d3832a080 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4352ffd640 10 client.27703 handle_command_reply: tid=0 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 1 client.27703 shutdown 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 1 -- 172.21.15.16:0/1989935432 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38375670 msgr2=0x556d3874ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38375670 0x556d3874ca70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4364000be0 tx=0x7f43640099d0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 1 -- 172.21.15.16:0/1989935432 shutdown_connections 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d38396350 0x556d387c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556d3832a080 0x556d387c75b0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556d38375670 0x556d3874ca70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 1 --2- 172.21.15.16:0/1989935432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d3836d490 0x556d3874c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4352ffd640 0 client.27703 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 1 -- 172.21.15.16:0/1989935432 >> 172.21.15.16:0/1989935432 conn(0x556d3838a740 msgr2=0x556d38749380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 1 -- 172.21.15.16:0/1989935432 shutdown_connections 2024-08-28T19:33:04.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 1 -- 172.21.15.16:0/1989935432 wait complete. 2024-08-28T19:33:04.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.295+0000 7f4380202740 20 client.27703 trim_cache size 0 max 16384 2024-08-28T19:33:04.301 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:33:04.301 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:33:04.301 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "44650602-6cd3-4083-8dac-11bd45aa6f43", 2024-08-28T19:33:04.301 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:33:04.301 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:33:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 -- 172.21.15.16:0/1206608626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f435c0b4cd0 msgr2=0x7f435c0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f435c0b4cd0 0x7f435c0b71e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f436c0e1ed0 tx=0x7f436c00e280 comp rx=0 tx=0).stop 2024-08-28T19:33:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 -- 172.21.15.16:0/1206608626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4374108b10 msgr2=0x7f437419bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4374108b10 0x7f437419bc00 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f436c00b970 tx=0x7f436c00be30 comp rx=0 tx=0).stop 2024-08-28T19:33:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 -- 172.21.15.16:0/1206608626 shutdown_connections 2024-08-28T19:33:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4374108b10 0x7f437419bc00 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f43741081c0 0x7f437419b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f435c0b4cd0 0x7f435c0b71e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 --2- 172.21.15.16:0/1206608626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4374102880 0x7f437419b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 -- 172.21.15.16:0/1206608626 >> 172.21.15.16:0/1206608626 conn(0x7f4374079510 msgr2=0x7f43741062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:04.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 -- 172.21.15.16:0/1206608626 shutdown_connections 2024-08-28T19:33:04.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:04.296+0000 7f4379c8d640 1 -- 172.21.15.16:0/1206608626 wait complete. 2024-08-28T19:33:04.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:04 smithi016 ceph-mon[18556]: osdmap e95: 12 total, 12 up, 12 in 2024-08-28T19:33:04.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:04 smithi115 ceph-mon[22184]: osdmap e95: 12 total, 12 up, 12 in 2024-08-28T19:33:04.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:04 smithi028 ceph-mon[21769]: osdmap e95: 12 total, 12 up, 12 in 2024-08-28T19:33:04.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:33:04.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 495544 3.75 MB/sec execute 524 sec latency 23.730 ms 2024-08-28T19:33:05.207 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: pgmap v711: 129 pgs: 129 active+clean; 394 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:05.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:05 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: pgmap v711: 129 pgs: 129 active+clean; 394 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:05 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.705+0000 7f275c2b6640 1 -- 172.21.15.16:0/3471818298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2754102820 msgr2=0x7f2754102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:05.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.705+0000 7f275c2b6640 1 --2- 172.21.15.16:0/3471818298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2754102820 0x7f2754102c30 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f2750009900 tx=0x7f275002a410 comp rx=0 tx=0).stop 2024-08-28T19:33:05.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.705+0000 7f275c2b6640 1 -- 172.21.15.16:0/3471818298 shutdown_connections 2024-08-28T19:33:05.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.705+0000 7f275c2b6640 1 --2- 172.21.15.16:0/3471818298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2754108ab0 0x7f275410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.705+0000 7f275c2b6640 1 --2- 172.21.15.16:0/3471818298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2754108160 0x7f2754108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.705+0000 7f275c2b6640 1 --2- 172.21.15.16:0/3471818298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2754102820 0x7f2754102c30 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.705+0000 7f275c2b6640 1 -- 172.21.15.16:0/3471818298 >> 172.21.15.16:0/3471818298 conn(0x7f27540794e0 msgr2=0x7f27540798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.705+0000 7f275c2b6640 1 -- 172.21.15.16:0/3471818298 shutdown_connections 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.705+0000 7f275c2b6640 1 -- 172.21.15.16:0/3471818298 wait complete. 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.706+0000 7f275c2b6640 1 Processor -- start 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.706+0000 7f275c2b6640 1 -- start start 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.706+0000 7f275c2b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2754102820 0x7f275419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.706+0000 7f275c2b6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2754108160 0x7f275419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.706+0000 7f275c2b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2754108ab0 0x7f275419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.706+0000 7f275c2b6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2754113840 con 0x7f2754102820 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.706+0000 7f275c2b6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27541136c0 con 0x7f2754108ab0 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.706+0000 7f275c2b6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f27541139c0 con 0x7f2754108160 2024-08-28T19:33:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275a052640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2754102820 0x7f275419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275a052640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2754102820 0x7f275419a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36160/0 (socket says 172.21.15.16:36160) 2024-08-28T19:33:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275a052640 1 -- 172.21.15.16:0/3426606555 learned_addr learned my addr 172.21.15.16:0/3426606555 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275a853640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2754108ab0 0x7f275419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f2759851640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2754108160 0x7f275419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275a052640 1 -- 172.21.15.16:0/3426606555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2754108160 msgr2=0x7f275419a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275a052640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2754108160 0x7f275419a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275a052640 1 -- 172.21.15.16:0/3426606555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2754108ab0 msgr2=0x7f275419fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275a052640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2754108ab0 0x7f275419fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275a052640 1 -- 172.21.15.16:0/3426606555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27541a01c0 con 0x7f2754102820 2024-08-28T19:33:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f2759851640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2754108160 0x7f275419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275a052640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2754102820 0x7f275419a250 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f2750005940 tx=0x7f27500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f274b7fe640 1 -- 172.21.15.16:0/3426606555 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275003f030 con 0x7f2754102820 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275c2b6640 1 -- 172.21.15.16:0/3426606555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27541a0450 con 0x7f2754102820 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f274b7fe640 1 -- 172.21.15.16:0/3426606555 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2750003cb0 con 0x7f2754102820 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f274b7fe640 1 -- 172.21.15.16:0/3426606555 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750035070 con 0x7f2754102820 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.707+0000 7f275c2b6640 1 -- 172.21.15.16:0/3426606555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27541a6c10 con 0x7f2754102820 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.710+0000 7f274b7fe640 1 -- 172.21.15.16:0/3426606555 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2750042030 con 0x7f2754102820 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.710+0000 7f276282b740 1 Processor -- start 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.710+0000 7f276282b740 1 -- start start 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f276282b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e4f070 0x55a61a1f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f276282b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e261b0 0x55a61a1f4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f276282b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1f4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f276282b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a61a16b630 con 0x55a619e2e390 2024-08-28T19:33:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f276282b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a61a15abc0 con 0x55a619e261b0 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f276282b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a61a15eb70 con 0x55a619e4f070 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f2759851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1f4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f2759851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1f4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36170/0 (socket says 172.21.15.16:36170) 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f2759851640 1 -- 172.21.15.16:0/905348032 learned_addr learned my addr 172.21.15.16:0/905348032 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f275a052640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e261b0 0x55a61a1f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f275a853640 1 --2- 172.21.15.16:0/905348032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e4f070 0x55a61a1f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f274b7fe640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27400b4ca0 0x7f27400b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f2759851640 1 -- 172.21.15.16:0/905348032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e4f070 msgr2=0x55a61a1f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f2759851640 1 --2- 172.21.15.16:0/905348032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e4f070 0x55a61a1f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f2759851640 1 -- 172.21.15.16:0/905348032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e261b0 msgr2=0x55a61a1f4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f2759851640 1 --2- 172.21.15.16:0/905348032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e261b0 0x55a61a1f4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f2759851640 1 -- 172.21.15.16:0/905348032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a619eba650 con 0x55a619e2e390 2024-08-28T19:33:05.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f275a853640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27400b4ca0 0x7f27400b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f274b7fe640 1 -- 172.21.15.16:0/3426606555 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9578+0+0 (secure 0 0 0) 0x7f2750105f00 con 0x7f2754102820 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.711+0000 7f2759851640 1 --2- 172.21.15.16:0/905348032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1f4d40 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f274400bd00 tx=0x7f274400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88efa54bb6ddc254 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.712+0000 7f2748d9a640 1 -- 172.21.15.16:0/905348032 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2744005020 con 0x55a619e2e390 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.712+0000 7f2748d9a640 1 -- 172.21.15.16:0/905348032 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27440047a0 con 0x55a619e2e390 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.712+0000 7f2748d9a640 1 -- 172.21.15.16:0/905348032 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27440116a0 con 0x55a619e2e390 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.712+0000 7f275a853640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27400b4ca0 0x7f27400b71b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f274c005fd0 tx=0x7f274c004a40 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.713+0000 7f276282b740 1 -- 172.21.15.16:0/905348032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 msgr2=0x55a61a1f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.713+0000 7f276282b740 1 --2- 172.21.15.16:0/905348032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1f4d40 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f274400bd00 tx=0x7f274400f730 comp rx=0 tx=0).stop 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.713+0000 7f276282b740 1 -- 172.21.15.16:0/905348032 shutdown_connections 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.713+0000 7f276282b740 1 --2- 172.21.15.16:0/905348032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1f4d40 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.713+0000 7f276282b740 1 --2- 172.21.15.16:0/905348032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e261b0 0x55a61a1f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.713+0000 7f276282b740 1 --2- 172.21.15.16:0/905348032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e4f070 0x55a61a1f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.713+0000 7f276282b740 1 -- 172.21.15.16:0/905348032 >> 172.21.15.16:0/905348032 conn(0x55a619e43460 msgr2=0x55a61a1f21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.713+0000 7f276282b740 1 -- 172.21.15.16:0/905348032 shutdown_connections 2024-08-28T19:33:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.713+0000 7f276282b740 1 -- 172.21.15.16:0/905348032 wait complete. 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.713+0000 7f276282b740 1 -- start start 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.713+0000 7f276282b740 1 Processor -- start 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f276282b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e261b0 0x55a61a1ffa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f276282b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1fff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f276282b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e4f070 0x55a61a20bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f276282b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a619fbd700 con 0x55a619e2e390 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f276282b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a61a0e1700 con 0x55a619e4f070 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f276282b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a61a202aa0 con 0x55a619e261b0 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f276282b740 10 client.0 _ll_register_callbacks cb 0x55a61a110dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f2759851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1fff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f2759851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1fff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36174/0 (socket says 172.21.15.16:36174) 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f2759851640 1 -- 172.21.15.16:0/2327699232 learned_addr learned my addr 172.21.15.16:0/2327699232 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f275a853640 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e4f070 0x55a61a20bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f275a052640 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e261b0 0x55a61a1ffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f2759851640 1 -- 172.21.15.16:0/2327699232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e261b0 msgr2=0x55a61a1ffa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f2759851640 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e261b0 0x55a61a1ffa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f2759851640 1 -- 172.21.15.16:0/2327699232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e4f070 msgr2=0x55a61a20bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f2759851640 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e4f070 0x55a61a20bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f2759851640 1 -- 172.21.15.16:0/2327699232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a61a11a8b0 con 0x55a619e2e390 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f2759851640 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1fff50 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2744006010 tx=0x7f274400ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd42d29e6724080 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.714+0000 7f273dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f273dffb640 1 -- 172.21.15.16:0/2327699232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274402b060 con 0x55a619e2e390 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f273dffb640 1 -- 172.21.15.16:0/2327699232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9578+0+0 (secure 0 0 0) 0x7f2744004ee0 con 0x55a619e2e390 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f276282b740 1 -- 172.21.15.16:0/2327699232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27440047a0 con 0x55a619e2e390 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f273dffb640 1 -- 172.21.15.16:0/2327699232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f2744032050 con 0x55a619e2e390 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f276282b740 10 client.18561 fetch_fsmap learned FSMap version 59 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f276282b740 1 -- 172.21.15.16:0/2327699232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55a61a212c60 con 0x55a619e2e390 2024-08-28T19:33:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f273dffb640 1 -- 172.21.15.16:0/2327699232 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27440047a0 con 0x55a619e2e390 2024-08-28T19:33:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f273dffb640 1 -- 172.21.15.16:0/2327699232 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7f27440119b0 con 0x55a619e2e390 2024-08-28T19:33:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f276282b740 10 client.18561 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:33:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f276282b740 10 client.18561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f276282b740 20 client.18561 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f276282b740 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a619e0d9f0 0x55a61a215fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f276282b740 4 client.18561 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.715+0000 7f276282b740 1 -- 172.21.15.16:0/2327699232 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a61a16d580 con 0x55a619e0d9f0 2024-08-28T19:33:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.716+0000 7f275a052640 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a619e0d9f0 0x55a61a215fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.716+0000 7f275a052640 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a619e0d9f0 0x55a61a215fc0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5e321308da6d567 server_cookie=3744e2ecba099b80 in_seq=0 out_seq=0 2024-08-28T19:33:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.716+0000 7f273dffb640 10 client.18561 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f273dffb640 1 -- 172.21.15.16:0/2327699232 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a61a16d580 con 0x55a619e0d9f0 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f273dffb640 10 client.18561 handle_command_reply: tid=0 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 1 client.18561 shutdown 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 1 -- 172.21.15.16:0/2327699232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 msgr2=0x55a61a1fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1fff50 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2744006010 tx=0x7f274400ba30 comp rx=0 tx=0).stop 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 1 -- 172.21.15.16:0/2327699232 shutdown_connections 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e4f070 0x55a61a20bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e2e390 0x55a61a1fff50 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a619e0d9f0 0x55a61a215fc0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 1 --2- 172.21.15.16:0/2327699232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e261b0 0x55a61a1ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f273dffb640 0 client.18561 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 1 -- 172.21.15.16:0/2327699232 >> 172.21.15.16:0/2327699232 conn(0x55a619e43460 msgr2=0x55a61a1f7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 1 -- 172.21.15.16:0/2327699232 shutdown_connections 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 1 -- 172.21.15.16:0/2327699232 wait complete. 2024-08-28T19:33:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.717+0000 7f276282b740 20 client.18561 trim_cache size 0 max 16384 2024-08-28T19:33:05.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f276282b740 1 Processor -- start 2024-08-28T19:33:05.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f276282b740 1 -- start start 2024-08-28T19:33:05.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f276282b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e261b0 0x55a61a20bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f276282b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 0x55a61a1f7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f276282b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e4f070 0x55a61a1fcc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f276282b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27541aec60 con 0x55a619e261b0 2024-08-28T19:33:05.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f276282b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27541b08c0 con 0x55a619e4f070 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f276282b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f27541af3f0 con 0x55a619e2e390 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f275a052640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e261b0 0x55a61a20bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f275a052640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e261b0 0x55a61a20bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36190/0 (socket says 172.21.15.16:36190) 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f275a052640 1 -- 172.21.15.16:0/313683179 learned_addr learned my addr 172.21.15.16:0/313683179 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f2759851640 1 --2- 172.21.15.16:0/313683179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 0x55a61a1f7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f275a052640 1 -- 172.21.15.16:0/313683179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 msgr2=0x55a61a1f7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f275a853640 1 --2- 172.21.15.16:0/313683179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e4f070 0x55a61a1fcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f275a052640 1 --2- 172.21.15.16:0/313683179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 0x55a61a1f7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f275a052640 1 -- 172.21.15.16:0/313683179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e4f070 msgr2=0x55a61a1fcc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f275a052640 1 --2- 172.21.15.16:0/313683179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e4f070 0x55a61a1fcc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.735+0000 7f275a052640 1 -- 172.21.15.16:0/313683179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a61a1f8010 con 0x55a619e261b0 2024-08-28T19:33:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.736+0000 7f275a052640 1 --2- 172.21.15.16:0/313683179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e261b0 0x55a61a20bd00 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f27500c9440 tx=0x7f275003b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b294a7ecb0e9543b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:05.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.736+0000 7f273ffff640 1 -- 172.21.15.16:0/313683179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750049040 con 0x55a619e261b0 2024-08-28T19:33:05.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.736+0000 7f273ffff640 1 -- 172.21.15.16:0/313683179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27500cbce0 con 0x55a619e261b0 2024-08-28T19:33:05.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.736+0000 7f273ffff640 1 -- 172.21.15.16:0/313683179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27500d0a40 con 0x55a619e261b0 2024-08-28T19:33:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.737+0000 7f276282b740 1 -- 172.21.15.16:0/313683179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e261b0 msgr2=0x55a61a20bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.737+0000 7f276282b740 1 --2- 172.21.15.16:0/313683179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e261b0 0x55a61a20bd00 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f27500c9440 tx=0x7f275003b6b0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.737+0000 7f276282b740 1 -- 172.21.15.16:0/313683179 shutdown_connections 2024-08-28T19:33:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.737+0000 7f276282b740 1 --2- 172.21.15.16:0/313683179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e4f070 0x55a61a1fcc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.737+0000 7f276282b740 1 --2- 172.21.15.16:0/313683179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 0x55a61a1f7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.737+0000 7f276282b740 1 --2- 172.21.15.16:0/313683179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e261b0 0x55a61a20bd00 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.737+0000 7f276282b740 1 -- 172.21.15.16:0/313683179 >> 172.21.15.16:0/313683179 conn(0x55a619e43460 msgr2=0x55a61a1f73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.737+0000 7f276282b740 1 -- 172.21.15.16:0/313683179 shutdown_connections 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.737+0000 7f276282b740 1 -- 172.21.15.16:0/313683179 wait complete. 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.737+0000 7f276282b740 1 -- start start 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.737+0000 7f276282b740 1 Processor -- start 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f276282b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e261b0 0x55a61a217440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f276282b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 0x55a61a2179c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f276282b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e4f070 0x55a61a27f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f276282b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27541b01c0 con 0x55a619e4f070 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f276282b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27541afac0 con 0x55a619e261b0 2024-08-28T19:33:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f276282b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f27541acc60 con 0x55a619e2e390 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f276282b740 10 client.0 _ll_register_callbacks cb 0x7f2744015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f2759851640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 0x55a61a2179c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f2759851640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 0x55a61a2179c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:40580/0 (socket says 172.21.15.16:40580) 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f2759851640 1 -- 172.21.15.16:0/4250737863 learned_addr learned my addr 172.21.15.16:0/4250737863 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f275a052640 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e261b0 0x55a61a217440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f275a853640 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e4f070 0x55a61a27f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f2759851640 1 -- 172.21.15.16:0/4250737863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e261b0 msgr2=0x55a61a217440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f2759851640 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e261b0 0x55a61a217440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f2759851640 1 -- 172.21.15.16:0/4250737863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e4f070 msgr2=0x55a61a27f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f2759851640 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e4f070 0x55a61a27f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f2759851640 1 -- 172.21.15.16:0/4250737863 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a61a27fc80 con 0x55a619e2e390 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f275a853640 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e4f070 0x55a61a27f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f2759851640 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 0x55a61a2179c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f274400bac0 tx=0x7f2744004d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab21240b6d090a8d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.738+0000 7f273dffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f273dffb640 1 -- 172.21.15.16:0/4250737863 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274402b040 con 0x55a619e2e390 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f276282b740 1 -- 172.21.15.16:0/4250737863 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27500cbce0 con 0x55a619e2e390 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f273dffb640 1 -- 172.21.15.16:0/4250737863 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9578+0+0 (secure 0 0 0) 0x7f2744004730 con 0x55a619e2e390 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f273dffb640 1 -- 172.21.15.16:0/4250737863 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f2744004930 con 0x55a619e2e390 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f276282b740 10 client.27721 fetch_fsmap learned FSMap version 59 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f276282b740 1 -- 172.21.15.16:0/4250737863 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55a61a280de0 con 0x55a619e2e390 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f273dffb640 1 -- 172.21.15.16:0/4250737863 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27500cbce0 con 0x55a619e2e390 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f273dffb640 1 -- 172.21.15.16:0/4250737863 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 59) ==== 8394+0+0 (secure 0 0 0) 0x7f274401bbd0 con 0x55a619e2e390 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f276282b740 10 client.27721 fetch_fsmap finished waiting for FSMap version 59 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f276282b740 10 client.27721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f276282b740 20 client.27721 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f276282b740 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a619de2da0 0x55a61a2834d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f276282b740 4 client.27721 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.739+0000 7f276282b740 1 -- 172.21.15.16:0/4250737863 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a61a169020 con 0x55a619de2da0 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.740+0000 7f275a052640 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a619de2da0 0x55a61a2834d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.740+0000 7f273dffb640 10 client.27721 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.740+0000 7f275a052640 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a619de2da0 0x55a61a2834d0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfe315cf2bd04e80 server_cookie=14336f05a8ae3e76 in_seq=0 out_seq=0 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.740+0000 7f273dffb640 1 -- 172.21.15.16:0/4250737863 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+5114 (crc 0 0 0) 0x55a61a169020 con 0x55a619de2da0 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.740+0000 7f273dffb640 10 client.27721 handle_command_reply: tid=0 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.740+0000 7f276282b740 1 client.27721 shutdown 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.740+0000 7f276282b740 1 -- 172.21.15.16:0/4250737863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 msgr2=0x55a61a2179c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.740+0000 7f276282b740 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 0x55a61a2179c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f274400bac0 tx=0x7f2744004d70 comp rx=0 tx=0).stop 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.740+0000 7f276282b740 1 -- 172.21.15.16:0/4250737863 shutdown_connections 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f276282b740 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a619e4f070 0x55a61a27f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f276282b740 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a619de2da0 0x55a61a2834d0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f276282b740 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a619e2e390 0x55a61a2179c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f273dffb640 0 client.27721 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f276282b740 1 --2- 172.21.15.16:0/4250737863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a619e261b0 0x55a61a217440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f276282b740 1 -- 172.21.15.16:0/4250737863 >> 172.21.15.16:0/4250737863 conn(0x55a619e43460 msgr2=0x55a61a1f52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f276282b740 1 -- 172.21.15.16:0/4250737863 shutdown_connections 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f276282b740 1 -- 172.21.15.16:0/4250737863 wait complete. 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f276282b740 20 client.27721 trim_cache size 0 max 16384 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "44650602-6cd3-4083-8dac-11bd45aa6f43", 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:33:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:33:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:33:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:05.759 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:05.759 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:05.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:05.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:05.759 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:33:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 -- 172.21.15.16:0/3426606555 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27400b4ca0 msgr2=0x7f27400b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27400b4ca0 0x7f27400b71b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f274c005fd0 tx=0x7f274c004a40 comp rx=0 tx=0).stop 2024-08-28T19:33:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 -- 172.21.15.16:0/3426606555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2754102820 msgr2=0x7f275419a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2754102820 0x7f275419a250 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f2750005940 tx=0x7f27500042f0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 -- 172.21.15.16:0/3426606555 shutdown_connections 2024-08-28T19:33:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27400b4ca0 0x7f27400b71b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2754108ab0 0x7f275419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2754108160 0x7f275419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 --2- 172.21.15.16:0/3426606555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2754102820 0x7f275419a250 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 -- 172.21.15.16:0/3426606555 >> 172.21.15.16:0/3426606555 conn(0x7f27540794e0 msgr2=0x7f2754101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 -- 172.21.15.16:0/3426606555 shutdown_connections 2024-08-28T19:33:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:05.741+0000 7f275c2b6640 1 -- 172.21.15.16:0/3426606555 wait complete. 2024-08-28T19:33:05.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: pgmap v711: 129 pgs: 129 active+clean; 394 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-08-28T19:33:05.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:33:05.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:05.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:05.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:05.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:05 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:05.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 495544 3.74 MB/sec execute 525 sec latency 1014.316 ms 2024-08-28T19:33:06.479 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:06 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:06.508 INFO:tasks.cephfs.filesystem:scrub status for tag:44650602-6cd3-4083-8dac-11bd45aa6f43 - {'path': '/', 'tag': '44650602-6cd3-4083-8dac-11bd45aa6f43', 'options': 'recursive,force'} 2024-08-28T19:33:06.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:06 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:06.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:06 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:06.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 495547 3.73 MB/sec execute 526 sec latency 2013.977 ms 2024-08-28T19:33:07.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:07 smithi115 ceph-mon[22184]: pgmap v712: 129 pgs: 129 active+clean; 393 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-08-28T19:33:07.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:07 smithi016 ceph-mon[18556]: pgmap v712: 129 pgs: 129 active+clean; 393 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-08-28T19:33:07.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:07 smithi028 ceph-mon[21769]: pgmap v712: 129 pgs: 129 active+clean; 393 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-08-28T19:33:07.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 497624 3.75 MB/sec execute 527 sec latency 49.890 ms 2024-08-28T19:33:08.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:08 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:08.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:08 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:08.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:08 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:08.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499009 3.76 MB/sec execute 528 sec latency 16.958 ms 2024-08-28T19:33:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:09 smithi115 ceph-mon[22184]: pgmap v713: 129 pgs: 129 active+clean; 394 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-08-28T19:33:09.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:09 smithi016 ceph-mon[18556]: pgmap v713: 129 pgs: 129 active+clean; 394 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-08-28T19:33:09.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:09 smithi028 ceph-mon[21769]: pgmap v713: 129 pgs: 129 active+clean; 394 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-08-28T19:33:09.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499092 3.75 MB/sec execute 529 sec latency 948.269 ms 2024-08-28T19:33:10.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499092 3.74 MB/sec execute 530 sec latency 1948.387 ms 2024-08-28T19:33:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:11 smithi115 ceph-mon[22184]: pgmap v714: 129 pgs: 129 active+clean; 395 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-08-28T19:33:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:11 smithi016 ceph-mon[18556]: pgmap v714: 129 pgs: 129 active+clean; 395 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-08-28T19:33:11.929 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:11 smithi028 ceph-mon[21769]: pgmap v714: 129 pgs: 129 active+clean; 395 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-08-28T19:33:11.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499095 3.74 MB/sec execute 531 sec latency 2943.845 ms 2024-08-28T19:33:12.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499247 3.73 MB/sec execute 532 sec latency 930.319 ms 2024-08-28T19:33:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:13 smithi016 ceph-mon[18556]: pgmap v715: 129 pgs: 129 active+clean; 398 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-08-28T19:33:13.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:13 smithi028 ceph-mon[21769]: pgmap v715: 129 pgs: 129 active+clean; 398 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-08-28T19:33:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:13 smithi115 ceph-mon[22184]: pgmap v715: 129 pgs: 129 active+clean; 398 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-08-28T19:33:13.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499247 3.72 MB/sec execute 533 sec latency 1930.424 ms 2024-08-28T19:33:14.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:33:14.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:33:14.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:33:14.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:33:14.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:33:14.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:33:14.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:33:14.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:33:14.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:33:14.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:33:14.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:33:14.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:33:14.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:33:14.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:33:14.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:33:14.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499247 3.71 MB/sec execute 534 sec latency 2930.521 ms 2024-08-28T19:33:15.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:15 smithi016 ceph-mon[18556]: pgmap v716: 129 pgs: 129 active+clean; 399 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 165 op/s 2024-08-28T19:33:15.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:15 smithi016 ceph-mon[18556]: osdmap e96: 12 total, 12 up, 12 in 2024-08-28T19:33:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:15 smithi028 ceph-mon[21769]: pgmap v716: 129 pgs: 129 active+clean; 399 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 165 op/s 2024-08-28T19:33:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:15 smithi028 ceph-mon[21769]: osdmap e96: 12 total, 12 up, 12 in 2024-08-28T19:33:15.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:15 smithi115 ceph-mon[22184]: pgmap v716: 129 pgs: 129 active+clean; 399 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 165 op/s 2024-08-28T19:33:15.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:15 smithi115 ceph-mon[22184]: osdmap e96: 12 total, 12 up, 12 in 2024-08-28T19:33:15.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499247 3.71 MB/sec execute 535 sec latency 3930.624 ms 2024-08-28T19:33:16.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499247 3.70 MB/sec execute 536 sec latency 4930.691 ms 2024-08-28T19:33:17.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:17 smithi016 ceph-mon[18556]: pgmap v718: 129 pgs: 129 active+clean; 400 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-08-28T19:33:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:17 smithi028 ceph-mon[21769]: pgmap v718: 129 pgs: 129 active+clean; 400 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-08-28T19:33:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:17 smithi115 ceph-mon[22184]: pgmap v718: 129 pgs: 129 active+clean; 400 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-08-28T19:33:17.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499247 3.69 MB/sec execute 537 sec latency 5930.779 ms 2024-08-28T19:33:18.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499247 3.69 MB/sec execute 538 sec latency 6930.882 ms 2024-08-28T19:33:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:19 smithi016 ceph-mon[18556]: pgmap v719: 129 pgs: 129 active+clean; 402 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1004 KiB/s wr, 81 op/s 2024-08-28T19:33:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:19 smithi028 ceph-mon[21769]: pgmap v719: 129 pgs: 129 active+clean; 402 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1004 KiB/s wr, 81 op/s 2024-08-28T19:33:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:19 smithi115 ceph-mon[22184]: pgmap v719: 129 pgs: 129 active+clean; 402 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1004 KiB/s wr, 81 op/s 2024-08-28T19:33:19.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499247 3.68 MB/sec execute 539 sec latency 7930.989 ms 2024-08-28T19:33:20.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499247 3.67 MB/sec execute 540 sec latency 8931.073 ms 2024-08-28T19:33:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:21 smithi016 ceph-mon[18556]: pgmap v720: 129 pgs: 129 active+clean; 402 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-08-28T19:33:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:21 smithi028 ceph-mon[21769]: pgmap v720: 129 pgs: 129 active+clean; 402 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-08-28T19:33:21.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:21 smithi115 ceph-mon[22184]: pgmap v720: 129 pgs: 129 active+clean; 402 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-08-28T19:33:21.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 500009 3.67 MB/sec execute 541 sec latency 9560.416 ms 2024-08-28T19:33:22.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 501500 3.68 MB/sec execute 542 sec latency 17.438 ms 2024-08-28T19:33:23.599 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 20.5 2024-08-28T19:33:23.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:33:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:23 smithi016 ceph-mon[18556]: pgmap v721: 129 pgs: 129 active+clean; 403 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 1.1 MiB/s wr, 57 op/s 2024-08-28T19:33:23.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:23 smithi028 ceph-mon[21769]: pgmap v721: 129 pgs: 129 active+clean; 403 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 1.1 MiB/s wr, 57 op/s 2024-08-28T19:33:23.858 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:23 smithi115 ceph-mon[22184]: pgmap v721: 129 pgs: 129 active+clean; 403 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 1.1 MiB/s wr, 57 op/s 2024-08-28T19:33:23.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502633 3.69 MB/sec execute 543 sec latency 287.575 ms 2024-08-28T19:33:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.382+0000 7f2722a68640 1 -- 172.21.15.16:0/384143886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f271c10ff20 msgr2=0x7f271c112340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.382+0000 7f2722a68640 1 --2- 172.21.15.16:0/384143886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f271c10ff20 0x7f271c112340 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f2714009ca0 tx=0x7f271402a950 comp rx=0 tx=0).stop 2024-08-28T19:33:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.382+0000 7f2722a68640 1 -- 172.21.15.16:0/384143886 shutdown_connections 2024-08-28T19:33:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.382+0000 7f2722a68640 1 --2- 172.21.15.16:0/384143886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f271c10ff20 0x7f271c112340 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.382+0000 7f2722a68640 1 --2- 172.21.15.16:0/384143886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f271c102cc0 0x7f271c10f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.382+0000 7f2722a68640 1 --2- 172.21.15.16:0/384143886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f271c102370 0x7f271c102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.382+0000 7f2722a68640 1 -- 172.21.15.16:0/384143886 >> 172.21.15.16:0/384143886 conn(0x7f271c0794d0 msgr2=0x7f271c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.382+0000 7f2722a68640 1 -- 172.21.15.16:0/384143886 shutdown_connections 2024-08-28T19:33:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.382+0000 7f2722a68640 1 -- 172.21.15.16:0/384143886 wait complete. 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.383+0000 7f2722a68640 1 Processor -- start 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.383+0000 7f2722a68640 1 -- start start 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.383+0000 7f2722a68640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f271c102370 0x7f271c19c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.383+0000 7f2722a68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f271c102cc0 0x7f271c19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.383+0000 7f2722a68640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f271c10ff20 0x7f271c1a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.383+0000 7f2722a68640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f271c115a10 con 0x7f271c102cc0 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.383+0000 7f2722a68640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f271c115890 con 0x7f271c102370 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.383+0000 7f2722a68640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f271c115b90 con 0x7f271c10ff20 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f271bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f271c102cc0 0x7f271c19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f271bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f271c102cc0 0x7f271c19c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36408/0 (socket says 172.21.15.16:36408) 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f271bfff640 1 -- 172.21.15.16:0/4187237096 learned_addr learned my addr 172.21.15.16:0/4187237096 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f2720804640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f271c102370 0x7f271c19c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f2721005640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f271c10ff20 0x7f271c1a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f271bfff640 1 -- 172.21.15.16:0/4187237096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f271c10ff20 msgr2=0x7f271c1a1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f271bfff640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f271c10ff20 0x7f271c1a1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f271bfff640 1 -- 172.21.15.16:0/4187237096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f271c102370 msgr2=0x7f271c19c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f271bfff640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f271c102370 0x7f271c19c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f271bfff640 1 -- 172.21.15.16:0/4187237096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271c1a2270 con 0x7f271c102cc0 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f271bfff640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f271c102cc0 0x7f271c19c960 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f271000c920 tx=0x7f271000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f2719ffb640 1 -- 172.21.15.16:0/4187237096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2710007c70 con 0x7f271c102cc0 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f2722a68640 1 -- 172.21.15.16:0/4187237096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f271c1a2560 con 0x7f271c102cc0 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f2719ffb640 1 -- 172.21.15.16:0/4187237096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2710007e10 con 0x7f271c102cc0 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.384+0000 7f2719ffb640 1 -- 172.21.15.16:0/4187237096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2710018820 con 0x7f271c102cc0 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.385+0000 7f2722a68640 1 -- 172.21.15.16:0/4187237096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f271c1a8d80 con 0x7f271c102cc0 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.386+0000 7f2722a68640 1 -- 172.21.15.16:0/4187237096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f271c102780 con 0x7f271c102cc0 2024-08-28T19:33:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.387+0000 7f2719ffb640 1 -- 172.21.15.16:0/4187237096 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f271000f450 con 0x7f271c102cc0 2024-08-28T19:33:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.388+0000 7f2719ffb640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27040b4cd0 0x7f27040b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.388+0000 7f2720804640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27040b4cd0 0x7f27040b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.388+0000 7f2719ffb640 1 -- 172.21.15.16:0/4187237096 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f27100e5c80 con 0x7f271c102cc0 2024-08-28T19:33:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.389+0000 7f2720804640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27040b4cd0 0x7f27040b71e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f270c00bbe0 tx=0x7f270c00b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.392+0000 7f2719ffb640 1 -- 172.21.15.16:0/4187237096 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f2710014030 con 0x7f271c102cc0 2024-08-28T19:33:24.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.622+0000 7f2722a68640 1 -- 172.21.15.16:0/4187237096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f271c19d4f0 con 0x7f271c102cc0 2024-08-28T19:33:24.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.623+0000 7f2719ffb640 1 -- 172.21.15.16:0/4187237096 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+11988 (secure 0 0 0) 0x7f271c19d4f0 con 0x7f271c102cc0 2024-08-28T19:33:24.625 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:33:24.625 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"btime":"2024-08-28T19:33:24:392954+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":61,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:33:24.086016+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:33:24.625 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2024-08-28T19:33:24.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 -- 172.21.15.16:0/4187237096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27040b4cd0 msgr2=0x7f27040b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:24.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27040b4cd0 0x7f27040b71e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f270c00bbe0 tx=0x7f270c00b390 comp rx=0 tx=0).stop 2024-08-28T19:33:24.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 -- 172.21.15.16:0/4187237096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f271c102cc0 msgr2=0x7f271c19c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:24.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f271c102cc0 0x7f271c19c960 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f271000c920 tx=0x7f271000cde0 comp rx=0 tx=0).stop 2024-08-28T19:33:24.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 -- 172.21.15.16:0/4187237096 shutdown_connections 2024-08-28T19:33:24.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f271c10ff20 0x7f271c1a1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:24.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f271c102cc0 0x7f271c19c960 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:24.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f27040b4cd0 0x7f27040b71e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:24.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 --2- 172.21.15.16:0/4187237096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f271c102370 0x7f271c19c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:24.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 -- 172.21.15.16:0/4187237096 >> 172.21.15.16:0/4187237096 conn(0x7f271c0794d0 msgr2=0x7f271c109a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:24.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 -- 172.21.15.16:0/4187237096 shutdown_connections 2024-08-28T19:33:24.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:24.626+0000 7f2722a68640 1 -- 172.21.15.16:0/4187237096 wait complete. 2024-08-28T19:33:24.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502633 3.68 MB/sec execute 544 sec latency 1287.600 ms 2024-08-28T19:33:25.374 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 140.5 --await-for 45.0' 2024-08-28T19:33:25.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 140.5 --await-for 45.0 2024-08-28T19:33:25.625 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:25.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:25 smithi016 ceph-mon[18556]: pgmap v722: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 1.2 MiB/s wr, 65 op/s 2024-08-28T19:33:25.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:25 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:25.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:25 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/4187237096' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:33:25.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:25 smithi028 ceph-mon[21769]: pgmap v722: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 1.2 MiB/s wr, 65 op/s 2024-08-28T19:33:25.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:25 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:25.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:25 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/4187237096' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:33:25.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:25 smithi115 ceph-mon[22184]: pgmap v722: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 1.2 MiB/s wr, 65 op/s 2024-08-28T19:33:25.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:25 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:25.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:25 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/4187237096' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:33:25.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502633 3.67 MB/sec execute 545 sec latency 2287.633 ms 2024-08-28T19:33:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.121+0000 7fe312bc5640 1 -- 172.21.15.16:0/3904582550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe30c0754d0 msgr2=0x7fe30c0758e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.121+0000 7fe312bc5640 1 --2- 172.21.15.16:0/3904582550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe30c0754d0 0x7fe30c0758e0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fe308009900 tx=0x7fe30802a410 comp rx=0 tx=0).stop 2024-08-28T19:33:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.122+0000 7fe312bc5640 1 -- 172.21.15.16:0/3904582550 shutdown_connections 2024-08-28T19:33:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.122+0000 7fe312bc5640 1 --2- 172.21.15.16:0/3904582550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe30c10fef0 0x7fe30c112310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.122+0000 7fe312bc5640 1 --2- 172.21.15.16:0/3904582550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe30c075e20 0x7fe30c10f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.122+0000 7fe312bc5640 1 --2- 172.21.15.16:0/3904582550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe30c0754d0 0x7fe30c0758e0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.122+0000 7fe312bc5640 1 -- 172.21.15.16:0/3904582550 >> 172.21.15.16:0/3904582550 conn(0x7fe30c0ff340 msgr2=0x7fe30c101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.122+0000 7fe312bc5640 1 -- 172.21.15.16:0/3904582550 shutdown_connections 2024-08-28T19:33:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.122+0000 7fe312bc5640 1 -- 172.21.15.16:0/3904582550 wait complete. 2024-08-28T19:33:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.122+0000 7fe312bc5640 1 Processor -- start 2024-08-28T19:33:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.122+0000 7fe312bc5640 1 -- start start 2024-08-28T19:33:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe312bc5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe30c0754d0 0x7fe30c19c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe312bc5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe30c075e20 0x7fe30c19ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe312bc5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe30c10fef0 0x7fe30c1a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe312bc5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c115a30 con 0x7fe30c0754d0 2024-08-28T19:33:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe312bc5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c1158b0 con 0x7fe30c10fef0 2024-08-28T19:33:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe312bc5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c115bb0 con 0x7fe30c075e20 2024-08-28T19:33:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe310961640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe30c0754d0 0x7fe30c19c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe310961640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe30c0754d0 0x7fe30c19c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36440/0 (socket says 172.21.15.16:36440) 2024-08-28T19:33:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe310961640 1 -- 172.21.15.16:0/1519201117 learned_addr learned my addr 172.21.15.16:0/1519201117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe303fff640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe30c075e20 0x7fe30c19ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe311162640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe30c10fef0 0x7fe30c1a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe310961640 1 -- 172.21.15.16:0/1519201117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe30c075e20 msgr2=0x7fe30c19ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe310961640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe30c075e20 0x7fe30c19ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe310961640 1 -- 172.21.15.16:0/1519201117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe30c10fef0 msgr2=0x7fe30c1a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe310961640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe30c10fef0 0x7fe30c1a1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe310961640 1 -- 172.21.15.16:0/1519201117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30c1a2310 con 0x7fe30c0754d0 2024-08-28T19:33:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe310961640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe30c0754d0 0x7fe30c19c460 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fe308005940 tx=0x7fe3080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe301ffb640 1 -- 172.21.15.16:0/1519201117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30803f030 con 0x7fe30c0754d0 2024-08-28T19:33:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe301ffb640 1 -- 172.21.15.16:0/1519201117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe308003cb0 con 0x7fe30c0754d0 2024-08-28T19:33:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.123+0000 7fe301ffb640 1 -- 172.21.15.16:0/1519201117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308035070 con 0x7fe30c0754d0 2024-08-28T19:33:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.124+0000 7fe312bc5640 1 -- 172.21.15.16:0/1519201117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30c1a25a0 con 0x7fe30c0754d0 2024-08-28T19:33:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.124+0000 7fe312bc5640 1 -- 172.21.15.16:0/1519201117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe30c1a2a30 con 0x7fe30c0754d0 2024-08-28T19:33:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.126+0000 7fe31913a740 1 Processor -- start 2024-08-28T19:33:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe31913a740 1 -- start start 2024-08-28T19:33:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe301ffb640 1 -- 172.21.15.16:0/1519201117 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe308042030 con 0x7fe30c0754d0 2024-08-28T19:33:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe31913a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa7fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe31913a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6af980 0x56109fa801d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe31913a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6b7b60 0x56109fa80710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe31913a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56109f9f4330 con 0x56109f6d8840 2024-08-28T19:33:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe31913a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56109f9e38c0 con 0x56109f6b7b60 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe31913a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56109f9e7870 con 0x56109f6af980 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe311162640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa7fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe311162640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa7fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36442/0 (socket says 172.21.15.16:36442) 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe311162640 1 -- 172.21.15.16:0/732459485 learned_addr learned my addr 172.21.15.16:0/732459485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe303fff640 1 --2- 172.21.15.16:0/732459485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6b7b60 0x56109fa80710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.127+0000 7fe310961640 1 --2- 172.21.15.16:0/732459485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6af980 0x56109fa801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe311162640 1 -- 172.21.15.16:0/732459485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6af980 msgr2=0x56109fa801d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe311162640 1 --2- 172.21.15.16:0/732459485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6af980 0x56109fa801d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe311162640 1 -- 172.21.15.16:0/732459485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6b7b60 msgr2=0x56109fa80710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe311162640 1 --2- 172.21.15.16:0/732459485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6b7b60 0x56109fa80710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe311162640 1 -- 172.21.15.16:0/732459485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56109f99bdf0 con 0x56109f6d8840 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe311162640 1 --2- 172.21.15.16:0/732459485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa7fc90 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fe304007f40 tx=0x7fe30400f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50fb26226d1e0723 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe301ffb640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2f40b4d50 0x7fe2f40b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe310961640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2f40b4d50 0x7fe2f40b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe2f2ffd640 1 -- 172.21.15.16:0/732459485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304010900 con 0x56109f6d8840 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe2f2ffd640 1 -- 172.21.15.16:0/732459485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3040183f0 con 0x56109f6d8840 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe2f2ffd640 1 -- 172.21.15.16:0/732459485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3040175c0 con 0x56109f6d8840 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.128+0000 7fe301ffb640 1 -- 172.21.15.16:0/1519201117 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fe308105d10 con 0x7fe30c0754d0 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.129+0000 7fe31913a740 1 -- 172.21.15.16:0/732459485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 msgr2=0x56109fa7fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.129+0000 7fe310961640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2f40b4d50 0x7fe2f40b7260 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe308103140 tx=0x7fe3080b3040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.129+0000 7fe31913a740 1 --2- 172.21.15.16:0/732459485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa7fc90 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fe304007f40 tx=0x7fe30400f850 comp rx=0 tx=0).stop 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.129+0000 7fe31913a740 1 -- 172.21.15.16:0/732459485 shutdown_connections 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.129+0000 7fe31913a740 1 --2- 172.21.15.16:0/732459485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6b7b60 0x56109fa80710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.129+0000 7fe31913a740 1 --2- 172.21.15.16:0/732459485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6af980 0x56109fa801d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.129+0000 7fe31913a740 1 --2- 172.21.15.16:0/732459485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa7fc90 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.129+0000 7fe31913a740 1 -- 172.21.15.16:0/732459485 >> 172.21.15.16:0/732459485 conn(0x56109f6cc990 msgr2=0x56109fa7db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.129+0000 7fe31913a740 1 -- 172.21.15.16:0/732459485 shutdown_connections 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.129+0000 7fe31913a740 1 -- 172.21.15.16:0/732459485 wait complete. 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.129+0000 7fe31913a740 1 -- start start 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe31913a740 1 Processor -- start 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe31913a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6af980 0x56109fa8b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe31913a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6b7b60 0x56109fa8b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe31913a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6d8840 0x56109fa97760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe31913a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56109f8467a0 con 0x56109f6b7b60 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe31913a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56109f96a240 con 0x56109f6d8840 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe31913a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56109fa8e470 con 0x56109f6af980 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe31913a740 10 client.0 _ll_register_callbacks cb 0x56109f6f0090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe303fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6b7b60 0x56109fa8b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe303fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6b7b60 0x56109fa8b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36456/0 (socket says 172.21.15.16:36456) 2024-08-28T19:33:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe303fff640 1 -- 172.21.15.16:0/3012120246 learned_addr learned my addr 172.21.15.16:0/3012120246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.130+0000 7fe311162640 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6af980 0x56109fa8b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe311162640 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6d8840 0x56109fa97760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe303fff640 1 -- 172.21.15.16:0/3012120246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6af980 msgr2=0x56109fa8b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe303fff640 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6af980 0x56109fa8b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe303fff640 1 -- 172.21.15.16:0/3012120246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6d8840 msgr2=0x56109fa97760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe303fff640 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6d8840 0x56109fa97760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe303fff640 1 -- 172.21.15.16:0/3012120246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56109f9a02b0 con 0x56109f6b7b60 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe311162640 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6af980 0x56109fa8b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe303fff640 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6b7b60 0x56109fa8b8f0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00bad0 tx=0x7fe2fc00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10be91c75baf8cff server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe2ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe2ebfff640 1 -- 172.21.15.16:0/3012120246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc003e20 con 0x56109f6b7b60 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe2ebfff640 1 -- 172.21.15.16:0/3012120246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fe2fc011440 con 0x56109f6b7b60 2024-08-28T19:33:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe31913a740 1 -- 172.21.15.16:0/3012120246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3040183f0 con 0x56109f6b7b60 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe2ebfff640 1 -- 172.21.15.16:0/3012120246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fe2fc020680 con 0x56109f6b7b60 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe31913a740 10 client.18585 fetch_fsmap learned FSMap version 61 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe31913a740 1 -- 172.21.15.16:0/3012120246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x56109f752b90 con 0x56109f6b7b60 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.131+0000 7fe2ebfff640 1 -- 172.21.15.16:0/3012120246 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3040183f0 con 0x56109f6b7b60 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.132+0000 7fe2ebfff640 1 -- 172.21.15.16:0/3012120246 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 61) ==== 8402+0+0 (secure 0 0 0) 0x7fe2fc0116d0 con 0x56109f6b7b60 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.132+0000 7fe31913a740 10 client.18585 fetch_fsmap finished waiting for FSMap version 61 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.132+0000 7fe31913a740 10 client.18585 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.132+0000 7fe31913a740 20 client.18585 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.132+0000 7fe31913a740 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56109f6971c0 0x56109fa9d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.132+0000 7fe31913a740 4 client.18585 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.132+0000 7fe31913a740 1 -- 172.21.15.16:0/3012120246 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56109f9f6280 con 0x56109f6971c0 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.132+0000 7fe311162640 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56109f6971c0 0x56109fa9d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.132+0000 7fe2ebfff640 10 client.18585 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.132+0000 7fe311162640 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56109f6971c0 0x56109fa9d690 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53cbebf512b1f05f server_cookie=9abeb425d151ef42 in_seq=0 out_seq=0 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe2ebfff640 1 -- 172.21.15.16:0/3012120246 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56109f9f6280 con 0x56109f6971c0 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe2ebfff640 10 client.18585 handle_command_reply: tid=0 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe31913a740 1 client.18585 shutdown 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe31913a740 1 -- 172.21.15.16:0/3012120246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6b7b60 msgr2=0x56109fa8b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe31913a740 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6b7b60 0x56109fa8b8f0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00bad0 tx=0x7fe2fc00bf90 comp rx=0 tx=0).stop 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe31913a740 1 -- 172.21.15.16:0/3012120246 shutdown_connections 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe31913a740 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6d8840 0x56109fa97760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe31913a740 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6b7b60 0x56109fa8b8f0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe31913a740 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56109f6971c0 0x56109fa9d690 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe31913a740 1 --2- 172.21.15.16:0/3012120246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6af980 0x56109fa8b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe2ebfff640 0 client.18585 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.133+0000 7fe31913a740 1 -- 172.21.15.16:0/3012120246 >> 172.21.15.16:0/3012120246 conn(0x56109f6cc990 msgr2=0x56109fa7c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.134+0000 7fe31913a740 1 -- 172.21.15.16:0/3012120246 shutdown_connections 2024-08-28T19:33:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.134+0000 7fe31913a740 1 -- 172.21.15.16:0/3012120246 wait complete. 2024-08-28T19:33:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.134+0000 7fe31913a740 20 client.18585 trim_cache size 0 max 16384 2024-08-28T19:33:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe31913a740 1 Processor -- start 2024-08-28T19:33:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe31913a740 1 -- start start 2024-08-28T19:33:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe31913a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6af980 0x56109fa8cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe31913a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6b7b60 0x56109fa885a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe31913a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6d8840 0x56109fa88ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe31913a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c1b2f00 con 0x56109f6af980 2024-08-28T19:33:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe31913a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c1b4b90 con 0x56109f6b7b60 2024-08-28T19:33:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe31913a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c1b3690 con 0x56109f6d8840 2024-08-28T19:33:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe311162640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6af980 0x56109fa8cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe311162640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6af980 0x56109fa8cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36462/0 (socket says 172.21.15.16:36462) 2024-08-28T19:33:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe311162640 1 -- 172.21.15.16:0/3510991703 learned_addr learned my addr 172.21.15.16:0/3510991703 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe303fff640 1 --2- 172.21.15.16:0/3510991703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6b7b60 0x56109fa885a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe311162640 1 -- 172.21.15.16:0/3510991703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6d8840 msgr2=0x56109fa88ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe311162640 1 --2- 172.21.15.16:0/3510991703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6d8840 0x56109fa88ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe311162640 1 -- 172.21.15.16:0/3510991703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6b7b60 msgr2=0x56109fa885a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe311162640 1 --2- 172.21.15.16:0/3510991703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6b7b60 0x56109fa885a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe311162640 1 -- 172.21.15.16:0/3510991703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56109f9b3d90 con 0x56109f6af980 2024-08-28T19:33:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.151+0000 7fe303fff640 1 --2- 172.21.15.16:0/3510991703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6b7b60 0x56109fa885a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:33:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.152+0000 7fe311162640 1 --2- 172.21.15.16:0/3510991703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6af980 0x56109fa8cfe0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe304018d30 tx=0x7fe304018d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c63c75e03dd7c6a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.152+0000 7fe2f27fc640 1 -- 172.21.15.16:0/3510991703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3040183f0 con 0x56109f6af980 2024-08-28T19:33:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.152+0000 7fe2f27fc640 1 -- 172.21.15.16:0/3510991703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe304018a40 con 0x56109f6af980 2024-08-28T19:33:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.152+0000 7fe2f27fc640 1 -- 172.21.15.16:0/3510991703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304005110 con 0x56109f6af980 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.153+0000 7fe31913a740 1 -- 172.21.15.16:0/3510991703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6af980 msgr2=0x56109fa8cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.153+0000 7fe31913a740 1 --2- 172.21.15.16:0/3510991703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6af980 0x56109fa8cfe0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe304018d30 tx=0x7fe304018d60 comp rx=0 tx=0).stop 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.153+0000 7fe31913a740 1 -- 172.21.15.16:0/3510991703 shutdown_connections 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.153+0000 7fe31913a740 1 --2- 172.21.15.16:0/3510991703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6d8840 0x56109fa88ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.153+0000 7fe31913a740 1 --2- 172.21.15.16:0/3510991703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6b7b60 0x56109fa885a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.153+0000 7fe31913a740 1 --2- 172.21.15.16:0/3510991703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6af980 0x56109fa8cfe0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.153+0000 7fe31913a740 1 -- 172.21.15.16:0/3510991703 >> 172.21.15.16:0/3510991703 conn(0x56109f6cc990 msgr2=0x56109fa8c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.153+0000 7fe31913a740 1 -- 172.21.15.16:0/3510991703 shutdown_connections 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.153+0000 7fe31913a740 1 -- 172.21.15.16:0/3510991703 wait complete. 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.153+0000 7fe31913a740 1 -- start start 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.153+0000 7fe31913a740 1 Processor -- start 2024-08-28T19:33:26.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe31913a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6af980 0x56109fa93a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe31913a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6b7b60 0x56109fa93fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe31913a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa8e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe31913a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c1b4490 con 0x56109f6d8840 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe31913a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c1b3d90 con 0x56109f6af980 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe31913a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c1b0f00 con 0x56109f6b7b60 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe31913a740 10 client.0 _ll_register_callbacks cb 0x7fe2fc004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe311162640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa8e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe311162640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa8e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36466/0 (socket says 172.21.15.16:36466) 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe311162640 1 -- 172.21.15.16:0/1937329929 learned_addr learned my addr 172.21.15.16:0/1937329929 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe311162640 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6af980 0x56109fa93a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe303fff640 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6b7b60 0x56109fa93fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe311162640 1 -- 172.21.15.16:0/1937329929 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6b7b60 msgr2=0x56109fa93fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe311162640 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6b7b60 0x56109fa93fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe311162640 1 -- 172.21.15.16:0/1937329929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6af980 msgr2=0x56109fa93a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe311162640 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6af980 0x56109fa93a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe311162640 1 -- 172.21.15.16:0/1937329929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56109fa8eeb0 con 0x56109f6d8840 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe311162640 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa8e720 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x56109fa93730 tx=0x7fe304007520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ea64865f339f79 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe2ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.154+0000 7fe2ebfff640 1 -- 172.21.15.16:0/1937329929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304018870 con 0x56109f6d8840 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe2ebfff640 1 -- 172.21.15.16:0/1937329929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fe3040052f0 con 0x56109f6d8840 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe31913a740 1 -- 172.21.15.16:0/1937329929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe304018a40 con 0x56109f6d8840 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe2ebfff640 1 -- 172.21.15.16:0/1937329929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fe3040054f0 con 0x56109f6d8840 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe31913a740 10 client.18591 fetch_fsmap learned FSMap version 61 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe31913a740 1 -- 172.21.15.16:0/1937329929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x56109fb08a40 con 0x56109f6d8840 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe2ebfff640 1 -- 172.21.15.16:0/1937329929 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe304018a40 con 0x56109f6d8840 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe2ebfff640 1 -- 172.21.15.16:0/1937329929 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 61) ==== 8402+0+0 (secure 0 0 0) 0x7fe304024040 con 0x56109f6d8840 2024-08-28T19:33:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe31913a740 10 client.18591 fetch_fsmap finished waiting for FSMap version 61 2024-08-28T19:33:26.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe31913a740 10 client.18591 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:33:26.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe31913a740 20 client.18591 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:26.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe31913a740 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56109f66c570 0x56109fb0afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:26.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe31913a740 4 client.18591 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.5, "await_for": 45.0}] 2024-08-28T19:33:26.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.155+0000 7fe31913a740 1 -- 172.21.15.16:0/1937329929 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.5, "await_for": 45.0}) -- 0x56109f9f1d20 con 0x56109f66c570 2024-08-28T19:33:26.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.156+0000 7fe303fff640 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56109f66c570 0x56109fb0afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:26.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.156+0000 7fe2ebfff640 10 client.18591 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:26.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.156+0000 7fe303fff640 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56109f66c570 0x56109fb0afc0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b60c9b011300cab server_cookie=db4a2de2d8efde0b in_seq=0 out_seq=0 2024-08-28T19:33:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.238+0000 7fe2ebfff640 1 -- 172.21.15.16:0/1937329929 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56109f9f1d20 con 0x56109f66c570 2024-08-28T19:33:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.238+0000 7fe2ebfff640 10 client.18591 handle_command_reply: tid=0 2024-08-28T19:33:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.238+0000 7fe31913a740 1 client.18591 shutdown 2024-08-28T19:33:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.238+0000 7fe31913a740 1 -- 172.21.15.16:0/1937329929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 msgr2=0x56109fa8e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.238+0000 7fe31913a740 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa8e720 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x56109fa93730 tx=0x7fe304007520 comp rx=0 tx=0).stop 2024-08-28T19:33:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe31913a740 1 -- 172.21.15.16:0/1937329929 shutdown_connections 2024-08-28T19:33:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe31913a740 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109f6d8840 0x56109fa8e720 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe31913a740 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56109f66c570 0x56109fb0afc0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe31913a740 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56109f6b7b60 0x56109fa93fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe2ebfff640 0 client.18591 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe31913a740 1 --2- 172.21.15.16:0/1937329929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56109f6af980 0x56109fa93a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe31913a740 1 -- 172.21.15.16:0/1937329929 >> 172.21.15.16:0/1937329929 conn(0x56109f6cc990 msgr2=0x56109fa7c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe31913a740 1 -- 172.21.15.16:0/1937329929 shutdown_connections 2024-08-28T19:33:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe31913a740 1 -- 172.21.15.16:0/1937329929 wait complete. 2024-08-28T19:33:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe31913a740 20 client.18591 trim_cache size 0 max 16384 2024-08-28T19:33:26.242 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:33:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 61, 2024-08-28T19:33:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:33:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 38, 2024-08-28T19:33:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:33:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "b52084a": { 2024-08-28T19:33:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 38, 2024-08-28T19:33:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:33:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 140.5, 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:33:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:26.244 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:33:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe312bc5640 1 -- 172.21.15.16:0/1519201117 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2f40b4d50 msgr2=0x7fe2f40b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe312bc5640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2f40b4d50 0x7fe2f40b7260 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe308103140 tx=0x7fe3080b3040 comp rx=0 tx=0).stop 2024-08-28T19:33:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe312bc5640 1 -- 172.21.15.16:0/1519201117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe30c0754d0 msgr2=0x7fe30c19c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.239+0000 7fe312bc5640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe30c0754d0 0x7fe30c19c460 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fe308005940 tx=0x7fe3080042f0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.240+0000 7fe312bc5640 1 -- 172.21.15.16:0/1519201117 shutdown_connections 2024-08-28T19:33:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.240+0000 7fe312bc5640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2f40b4d50 0x7fe2f40b7260 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.240+0000 7fe312bc5640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe30c10fef0 0x7fe30c1a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.240+0000 7fe312bc5640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe30c075e20 0x7fe30c19ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.240+0000 7fe312bc5640 1 --2- 172.21.15.16:0/1519201117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe30c0754d0 0x7fe30c19c460 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.240+0000 7fe312bc5640 1 -- 172.21.15.16:0/1519201117 >> 172.21.15.16:0/1519201117 conn(0x7fe30c0ff340 msgr2=0x7fe30c109a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.240+0000 7fe312bc5640 1 -- 172.21.15.16:0/1519201117 shutdown_connections 2024-08-28T19:33:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:26.240+0000 7fe312bc5640 1 -- 172.21.15.16:0/1519201117 wait complete. 2024-08-28T19:33:26.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.67 MB/sec execute 546 sec latency 2498.854 ms 2024-08-28T19:33:27.053 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b52084a', quiesce took 3.5 seconds. Will release after: 17.0 2024-08-28T19:33:27.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:27 smithi016 ceph-mon[18556]: pgmap v723: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-08-28T19:33:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:27 smithi028 ceph-mon[21769]: pgmap v723: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-08-28T19:33:27.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:27 smithi115 ceph-mon[22184]: pgmap v723: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-08-28T19:33:27.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.66 MB/sec execute 547 sec latency 1787.271 ms 2024-08-28T19:33:28.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.65 MB/sec execute 548 sec latency 2787.367 ms 2024-08-28T19:33:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:29 smithi016 ceph-mon[18556]: pgmap v724: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 95 op/s 2024-08-28T19:33:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:29 smithi028 ceph-mon[21769]: pgmap v724: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 95 op/s 2024-08-28T19:33:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:29 smithi115 ceph-mon[22184]: pgmap v724: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 95 op/s 2024-08-28T19:33:29.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.65 MB/sec execute 549 sec latency 3787.464 ms 2024-08-28T19:33:30.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:30 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:30.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:30 smithi016 ceph-mon[18556]: pgmap v725: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 1.3 MiB/s wr, 96 op/s 2024-08-28T19:33:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:30 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:30 smithi028 ceph-mon[21769]: pgmap v725: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 1.3 MiB/s wr, 96 op/s 2024-08-28T19:33:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:30 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:30 smithi115 ceph-mon[22184]: pgmap v725: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 1.3 MiB/s wr, 96 op/s 2024-08-28T19:33:30.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.64 MB/sec execute 550 sec latency 4787.576 ms 2024-08-28T19:33:31.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.63 MB/sec execute 551 sec latency 5787.692 ms 2024-08-28T19:33:32.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.63 MB/sec execute 552 sec latency 6787.807 ms 2024-08-28T19:33:33.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:32 smithi028 ceph-mon[21769]: pgmap v726: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-08-28T19:33:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:32 smithi115 ceph-mon[22184]: pgmap v726: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-08-28T19:33:33.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:32 smithi016 ceph-mon[18556]: pgmap v726: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-08-28T19:33:33.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.62 MB/sec execute 553 sec latency 7787.902 ms 2024-08-28T19:33:34.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.61 MB/sec execute 554 sec latency 8787.997 ms 2024-08-28T19:33:35.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:34 smithi028 ceph-mon[21769]: pgmap v727: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.0 MiB/s wr, 89 op/s 2024-08-28T19:33:35.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:34 smithi115 ceph-mon[22184]: pgmap v727: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.0 MiB/s wr, 89 op/s 2024-08-28T19:33:35.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:34 smithi016 ceph-mon[18556]: pgmap v727: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.0 MiB/s wr, 89 op/s 2024-08-28T19:33:35.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.61 MB/sec execute 555 sec latency 9788.103 ms 2024-08-28T19:33:36.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:33:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:36.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.60 MB/sec execute 556 sec latency 10788.165 ms 2024-08-28T19:33:37.048 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:36 smithi016 ceph-mon[18556]: pgmap v728: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 997 KiB/s wr, 84 op/s 2024-08-28T19:33:37.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:36 smithi028 ceph-mon[21769]: pgmap v728: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 997 KiB/s wr, 84 op/s 2024-08-28T19:33:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:36 smithi115 ceph-mon[22184]: pgmap v728: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 997 KiB/s wr, 84 op/s 2024-08-28T19:33:37.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.280+0000 7f872d9c2640 1 -- 172.21.15.16:0/4202256049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728108ab0 msgr2=0x7f872810f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.280+0000 7f872d9c2640 1 --2- 172.21.15.16:0/4202256049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728108ab0 0x7f872810f2e0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f8720009ca0 tx=0x7f872002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.280+0000 7f872d9c2640 1 -- 172.21.15.16:0/4202256049 shutdown_connections 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.280+0000 7f872d9c2640 1 --2- 172.21.15.16:0/4202256049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728108ab0 0x7f872810f2e0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.280+0000 7f872d9c2640 1 --2- 172.21.15.16:0/4202256049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8728108160 0x7f8728108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.280+0000 7f872d9c2640 1 --2- 172.21.15.16:0/4202256049 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8728102820 0x7f8728102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.280+0000 7f872d9c2640 1 -- 172.21.15.16:0/4202256049 >> 172.21.15.16:0/4202256049 conn(0x7f87280794e0 msgr2=0x7f87280798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.280+0000 7f872d9c2640 1 -- 172.21.15.16:0/4202256049 shutdown_connections 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.280+0000 7f872d9c2640 1 -- 172.21.15.16:0/4202256049 wait complete. 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.280+0000 7f872d9c2640 1 Processor -- start 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.280+0000 7f872d9c2640 1 -- start start 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f872d9c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728102820 0x7f872819b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f872d9c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8728108160 0x7f872819b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f872d9c2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8728108ab0 0x7f872819bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f872d9c2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8728112730 con 0x7f8728102820 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f872d9c2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87281125b0 con 0x7f8728108160 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f872d9c2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f87281128b0 con 0x7f8728108ab0 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728102820 0x7f872819b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728102820 0x7f872819b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42952/0 (socket says 172.21.15.16:42952) 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f8726ffd640 1 -- 172.21.15.16:0/1682356889 learned_addr learned my addr 172.21.15.16:0/1682356889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f87277fe640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8728108ab0 0x7f872819bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f871ffff640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8728108160 0x7f872819b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f8726ffd640 1 -- 172.21.15.16:0/1682356889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8728108ab0 msgr2=0x7f872819bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f8726ffd640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8728108ab0 0x7f872819bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f8726ffd640 1 -- 172.21.15.16:0/1682356889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8728108160 msgr2=0x7f872819b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f8726ffd640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8728108160 0x7f872819b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f8726ffd640 1 -- 172.21.15.16:0/1682356889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87281a6310 con 0x7f8728102820 2024-08-28T19:33:37.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f8726ffd640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728102820 0x7f872819b080 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f871800d770 tx=0x7f871800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:37.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.281+0000 7f872c9c0640 1 -- 172.21.15.16:0/1682356889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718017070 con 0x7f8728102820 2024-08-28T19:33:37.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.282+0000 7f872c9c0640 1 -- 172.21.15.16:0/1682356889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87180054d0 con 0x7f8728102820 2024-08-28T19:33:37.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.282+0000 7f872d9c2640 1 -- 172.21.15.16:0/1682356889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87281a6600 con 0x7f8728102820 2024-08-28T19:33:37.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.282+0000 7f872c9c0640 1 -- 172.21.15.16:0/1682356889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718004c30 con 0x7f8728102820 2024-08-28T19:33:37.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.282+0000 7f872d9c2640 1 -- 172.21.15.16:0/1682356889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87281a6b40 con 0x7f8728102820 2024-08-28T19:33:37.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.284+0000 7f872c9c0640 1 -- 172.21.15.16:0/1682356889 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8718004dd0 con 0x7f8728102820 2024-08-28T19:33:37.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.284+0000 7f8733f37740 1 Processor -- start 2024-08-28T19:33:37.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.284+0000 7f8733f37740 1 -- start start 2024-08-28T19:33:37.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8733f37740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296b0070 0x562529a552c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8733f37740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a55800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8733f37740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a55d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8733f37740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5625299cc630 con 0x5625296871b0 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8733f37740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625299bbbc0 con 0x5625296b0070 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8733f37740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5625299bfb70 con 0x56252968f390 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a55800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a55800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42956/0 (socket says 172.21.15.16:42956) 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8726ffd640 1 -- 172.21.15.16:0/2680910494 learned_addr learned my addr 172.21.15.16:0/2680910494 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f87277fe640 1 --2- 172.21.15.16:0/2680910494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296b0070 0x562529a552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f871ffff640 1 --2- 172.21.15.16:0/2680910494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a55d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8726ffd640 1 -- 172.21.15.16:0/2680910494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 msgr2=0x562529a55d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8726ffd640 1 --2- 172.21.15.16:0/2680910494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a55d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8726ffd640 1 -- 172.21.15.16:0/2680910494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296b0070 msgr2=0x562529a552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8726ffd640 1 --2- 172.21.15.16:0/2680910494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296b0070 0x562529a552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8726ffd640 1 -- 172.21.15.16:0/2680910494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56252971b650 con 0x5625296871b0 2024-08-28T19:33:37.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.285+0000 7f8726ffd640 1 --2- 172.21.15.16:0/2680910494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a55800 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f87180e25f0 tx=0x7f87180ed9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0eeb9ccae191379 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:37.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.286+0000 7f872c9c0640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87140b4d50 0x7f87140b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.286+0000 7f871dffb640 1 -- 172.21.15.16:0/2680910494 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87180f9070 con 0x5625296871b0 2024-08-28T19:33:37.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.286+0000 7f87277fe640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87140b4d50 0x7f87140b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.286+0000 7f871dffb640 1 -- 172.21.15.16:0/2680910494 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87180ab1b0 con 0x5625296871b0 2024-08-28T19:33:37.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.286+0000 7f871dffb640 1 -- 172.21.15.16:0/2680910494 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87180aae10 con 0x5625296871b0 2024-08-28T19:33:37.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.286+0000 7f872c9c0640 1 -- 172.21.15.16:0/1682356889 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f87180e52d0 con 0x7f8728102820 2024-08-28T19:33:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.287+0000 7f8733f37740 1 -- 172.21.15.16:0/2680910494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 msgr2=0x562529a55800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.287+0000 7f87277fe640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87140b4d50 0x7f87140b7260 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f872002f500 tx=0x7f872002f370 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.287+0000 7f8733f37740 1 --2- 172.21.15.16:0/2680910494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a55800 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f87180e25f0 tx=0x7f87180ed9d0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.287+0000 7f8733f37740 1 -- 172.21.15.16:0/2680910494 shutdown_connections 2024-08-28T19:33:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.287+0000 7f8733f37740 1 --2- 172.21.15.16:0/2680910494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a55d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.287+0000 7f8733f37740 1 --2- 172.21.15.16:0/2680910494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a55800 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.287+0000 7f8733f37740 1 --2- 172.21.15.16:0/2680910494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296b0070 0x562529a552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.287+0000 7f8733f37740 1 -- 172.21.15.16:0/2680910494 >> 172.21.15.16:0/2680910494 conn(0x5625296a4460 msgr2=0x562529a531b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.287+0000 7f8733f37740 1 -- 172.21.15.16:0/2680910494 shutdown_connections 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.287+0000 7f8733f37740 1 -- 172.21.15.16:0/2680910494 wait complete. 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.287+0000 7f8733f37740 1 -- start start 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8733f37740 1 Processor -- start 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8733f37740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a64d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8733f37740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56252968f390 0x562529a652d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8733f37740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625296b0070 0x562529a71160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8733f37740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56252981e700 con 0x5625296871b0 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8733f37740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562529942700 con 0x56252968f390 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8733f37740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562529a67e00 con 0x5625296b0070 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8733f37740 10 client.0 _ll_register_callbacks cb 0x562529971dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a64d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a64d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42968/0 (socket says 172.21.15.16:42968) 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8726ffd640 1 -- 172.21.15.16:0/2852072452 learned_addr learned my addr 172.21.15.16:0/2852072452 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f87277fe640 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625296b0070 0x562529a71160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f871ffff640 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56252968f390 0x562529a652d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8726ffd640 1 -- 172.21.15.16:0/2852072452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625296b0070 msgr2=0x562529a71160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8726ffd640 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625296b0070 0x562529a71160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8726ffd640 1 -- 172.21.15.16:0/2852072452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56252968f390 msgr2=0x562529a652d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8726ffd640 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56252968f390 0x562529a652d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f8726ffd640 1 -- 172.21.15.16:0/2852072452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56252997b8b0 con 0x5625296871b0 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.288+0000 7f871ffff640 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56252968f390 0x562529a652d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.289+0000 7f8726ffd640 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a64d90 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f8718009610 tx=0x7f8718005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ea445674b36315 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.289+0000 7f8706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.289+0000 7f8706ffd640 1 -- 172.21.15.16:0/2852072452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718001650 con 0x5625296871b0 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.289+0000 7f8706ffd640 1 -- 172.21.15.16:0/2852072452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f8718014470 con 0x5625296871b0 2024-08-28T19:33:37.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.289+0000 7f8733f37740 1 -- 172.21.15.16:0/2852072452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87180ab1b0 con 0x5625296871b0 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.289+0000 7f8706ffd640 1 -- 172.21.15.16:0/2852072452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f8718020070 con 0x5625296871b0 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.289+0000 7f8733f37740 10 client.18603 fetch_fsmap learned FSMap version 62 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.289+0000 7f8733f37740 1 -- 172.21.15.16:0/2852072452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x562529a73b50 con 0x5625296871b0 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.289+0000 7f8706ffd640 1 -- 172.21.15.16:0/2852072452 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87180ab1b0 con 0x5625296871b0 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.289+0000 7f8706ffd640 1 -- 172.21.15.16:0/2852072452 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 62) ==== 8406+0+0 (secure 0 0 0) 0x7f87180e28d0 con 0x5625296871b0 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.290+0000 7f8733f37740 10 client.18603 fetch_fsmap finished waiting for FSMap version 62 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.290+0000 7f8733f37740 10 client.18603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.290+0000 7f8733f37740 20 client.18603 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.290+0000 7f8733f37740 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56252966e9f0 0x562529a76eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.290+0000 7f8733f37740 4 client.18603 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.290+0000 7f8733f37740 1 -- 172.21.15.16:0/2852072452 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625299ce580 con 0x56252966e9f0 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.290+0000 7f871ffff640 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56252966e9f0 0x562529a76eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.290+0000 7f871ffff640 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56252966e9f0 0x562529a76eb0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78c9cc9aae0a39f3 server_cookie=80b8bcf83e1bd744 in_seq=0 out_seq=0 2024-08-28T19:33:37.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.290+0000 7f8706ffd640 10 client.18603 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8706ffd640 1 -- 172.21.15.16:0/2852072452 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5625299ce580 con 0x56252966e9f0 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8706ffd640 10 client.18603 handle_command_reply: tid=0 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8733f37740 1 client.18603 shutdown 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8733f37740 1 -- 172.21.15.16:0/2852072452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 msgr2=0x562529a64d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8733f37740 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a64d90 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f8718009610 tx=0x7f8718005bb0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8733f37740 1 -- 172.21.15.16:0/2852072452 shutdown_connections 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8733f37740 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56252966e9f0 0x562529a76eb0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8733f37740 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625296b0070 0x562529a71160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8733f37740 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56252968f390 0x562529a652d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8706ffd640 0 client.18603 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8733f37740 1 --2- 172.21.15.16:0/2852072452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a64d90 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8733f37740 1 -- 172.21.15.16:0/2852072452 >> 172.21.15.16:0/2852072452 conn(0x5625296a4460 msgr2=0x562529a51770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.291+0000 7f8733f37740 1 -- 172.21.15.16:0/2852072452 shutdown_connections 2024-08-28T19:33:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.292+0000 7f8733f37740 1 -- 172.21.15.16:0/2852072452 wait complete. 2024-08-28T19:33:37.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.292+0000 7f8733f37740 20 client.18603 trim_cache size 0 max 16384 2024-08-28T19:33:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.309+0000 7f8733f37740 1 Processor -- start 2024-08-28T19:33:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.309+0000 7f8733f37740 1 -- start start 2024-08-28T19:33:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8733f37740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a56cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8733f37740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a669c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8733f37740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296b0070 0x562529a710d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8733f37740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87281aea30 con 0x5625296871b0 2024-08-28T19:33:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8733f37740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87281b06c0 con 0x5625296b0070 2024-08-28T19:33:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8733f37740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f87281af1c0 con 0x56252968f390 2024-08-28T19:33:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a56cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a56cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42972/0 (socket says 172.21.15.16:42972) 2024-08-28T19:33:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8726ffd640 1 -- 172.21.15.16:0/3710424797 learned_addr learned my addr 172.21.15.16:0/3710424797 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f87277fe640 1 --2- 172.21.15.16:0/3710424797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296b0070 0x562529a710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f871ffff640 1 --2- 172.21.15.16:0/3710424797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a669c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8726ffd640 1 -- 172.21.15.16:0/3710424797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 msgr2=0x562529a669c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8726ffd640 1 --2- 172.21.15.16:0/3710424797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a669c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8726ffd640 1 -- 172.21.15.16:0/3710424797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296b0070 msgr2=0x562529a710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8726ffd640 1 --2- 172.21.15.16:0/3710424797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296b0070 0x562529a710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8726ffd640 1 -- 172.21.15.16:0/3710424797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562529a57bd0 con 0x5625296871b0 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f87277fe640 1 --2- 172.21.15.16:0/3710424797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296b0070 0x562529a710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f8726ffd640 1 --2- 172.21.15.16:0/3710424797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a56cb0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f87180a7720 tx=0x7f87180ed9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5e53d30229f63d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f871d7fa640 1 -- 172.21.15.16:0/3710424797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718001850 con 0x5625296871b0 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f871d7fa640 1 -- 172.21.15.16:0/3710424797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87180b4250 con 0x5625296871b0 2024-08-28T19:33:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.310+0000 7f871d7fa640 1 -- 172.21.15.16:0/3710424797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718001a50 con 0x5625296871b0 2024-08-28T19:33:37.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.311+0000 7f8733f37740 1 -- 172.21.15.16:0/3710424797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 msgr2=0x562529a56cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:37.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.311+0000 7f8733f37740 1 --2- 172.21.15.16:0/3710424797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a56cb0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f87180a7720 tx=0x7f87180ed9d0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.311+0000 7f8733f37740 1 -- 172.21.15.16:0/3710424797 shutdown_connections 2024-08-28T19:33:37.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.311+0000 7f8733f37740 1 --2- 172.21.15.16:0/3710424797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296b0070 0x562529a710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.311+0000 7f8733f37740 1 --2- 172.21.15.16:0/3710424797 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a669c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.311+0000 7f8733f37740 1 --2- 172.21.15.16:0/3710424797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296871b0 0x562529a56cb0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.311+0000 7f8733f37740 1 -- 172.21.15.16:0/3710424797 >> 172.21.15.16:0/3710424797 conn(0x5625296a4460 msgr2=0x562529a662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:37.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.311+0000 7f8733f37740 1 -- 172.21.15.16:0/3710424797 shutdown_connections 2024-08-28T19:33:37.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.311+0000 7f8733f37740 1 -- 172.21.15.16:0/3710424797 wait complete. 2024-08-28T19:33:37.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.312+0000 7f8733f37740 1 -- start start 2024-08-28T19:33:37.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.312+0000 7f8733f37740 1 Processor -- start 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.312+0000 7f8733f37740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296871b0 0x562529a787f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.312+0000 7f8733f37740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a78dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.312+0000 7f8733f37740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296b0070 0x562529ada600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.312+0000 7f8733f37740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87281affc0 con 0x5625296b0070 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.312+0000 7f8733f37740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87281af8c0 con 0x5625296871b0 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.312+0000 7f8733f37740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f87281aca30 con 0x56252968f390 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8733f37740 10 client.0 _ll_register_callbacks cb 0x7f87180f7070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f87277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296b0070 0x562529ada600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296871b0 0x562529a787f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f87277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296b0070 0x562529ada600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42978/0 (socket says 172.21.15.16:42978) 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f87277fe640 1 -- 172.21.15.16:0/1945549783 learned_addr learned my addr 172.21.15.16:0/1945549783 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f871ffff640 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a78dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8726ffd640 1 -- 172.21.15.16:0/1945549783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 msgr2=0x562529a78dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:37.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8726ffd640 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a78dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8726ffd640 1 -- 172.21.15.16:0/1945549783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296b0070 msgr2=0x562529ada600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8726ffd640 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296b0070 0x562529ada600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8726ffd640 1 -- 172.21.15.16:0/1945549783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562529adac70 con 0x5625296871b0 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f87277fe640 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296b0070 0x562529ada600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8726ffd640 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296871b0 0x562529a787f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87180e2c50 tx=0x7f87180e2b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=797bf1438a4b0cde server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8706ffd640 1 -- 172.21.15.16:0/1945549783 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871805b070 con 0x5625296871b0 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8733f37740 1 -- 172.21.15.16:0/1945549783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5625299ca020 con 0x5625296871b0 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.313+0000 7f8706ffd640 1 -- 172.21.15.16:0/1945549783 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f87180e8930 con 0x5625296871b0 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.314+0000 7f8706ffd640 1 -- 172.21.15.16:0/1945549783 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f8718001c40 con 0x5625296871b0 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.314+0000 7f8733f37740 10 client.28052 fetch_fsmap learned FSMap version 62 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.314+0000 7f8733f37740 1 -- 172.21.15.16:0/1945549783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x562529adac70 con 0x5625296871b0 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.314+0000 7f8706ffd640 1 -- 172.21.15.16:0/1945549783 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5625299ca020 con 0x5625296871b0 2024-08-28T19:33:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.314+0000 7f8706ffd640 1 -- 172.21.15.16:0/1945549783 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 62) ==== 8406+0+0 (secure 0 0 0) 0x7f87180e8bf0 con 0x5625296871b0 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.314+0000 7f8733f37740 10 client.28052 fetch_fsmap finished waiting for FSMap version 62 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.314+0000 7f8733f37740 10 client.28052 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.314+0000 7f8733f37740 20 client.28052 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.314+0000 7f8733f37740 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562529643da0 0x562529ae4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.314+0000 7f8733f37740 4 client.28052 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.314+0000 7f8733f37740 1 -- 172.21.15.16:0/1945549783 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5625299c7760 con 0x562529643da0 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f871ffff640 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562529643da0 0x562529ae4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f8706ffd640 10 client.28052 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f871ffff640 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562529643da0 0x562529ae4270 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=560996b5c6e61176 server_cookie=49ceedd23986944 in_seq=0 out_seq=0 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f8706ffd640 1 -- 172.21.15.16:0/1945549783 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4918 (crc 0 0 0) 0x5625299c7760 con 0x562529643da0 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f8706ffd640 10 client.28052 handle_command_reply: tid=0 2024-08-28T19:33:37.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f8733f37740 1 client.28052 shutdown 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f8733f37740 1 -- 172.21.15.16:0/1945549783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296871b0 msgr2=0x562529a787f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f8733f37740 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296871b0 0x562529a787f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87180e2c50 tx=0x7f87180e2b00 comp rx=0 tx=0).stop 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f8733f37740 1 -- 172.21.15.16:0/1945549783 shutdown_connections 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f8733f37740 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625296b0070 0x562529ada600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f8733f37740 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562529643da0 0x562529ae4270 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f8733f37740 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252968f390 0x562529a78dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.315+0000 7f8733f37740 1 --2- 172.21.15.16:0/1945549783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625296871b0 0x562529a787f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f8706ffd640 0 client.28052 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f8733f37740 1 -- 172.21.15.16:0/1945549783 >> 172.21.15.16:0/1945549783 conn(0x5625296a4460 msgr2=0x562529a66bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f8733f37740 1 -- 172.21.15.16:0/1945549783 shutdown_connections 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f8733f37740 1 -- 172.21.15.16:0/1945549783 wait complete. 2024-08-28T19:33:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f8733f37740 20 client.28052 trim_cache size 0 max 16384 2024-08-28T19:33:37.322 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:33:37.322 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:33:37.322 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:33:37.322 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:33:37.322 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:33:37.322 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:33:37.322 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.322 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:33:37.323 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:33:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:33:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:33:37.325 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:33:37.326 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.329 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:33:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:37.332 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:37.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:37.333 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:33:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 -- 172.21.15.16:0/1682356889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87140b4d50 msgr2=0x7f87140b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87140b4d50 0x7f87140b7260 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f872002f500 tx=0x7f872002f370 comp rx=0 tx=0).stop 2024-08-28T19:33:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 -- 172.21.15.16:0/1682356889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728102820 msgr2=0x7f872819b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728102820 0x7f872819b080 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f871800d770 tx=0x7f871800dc30 comp rx=0 tx=0).stop 2024-08-28T19:33:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 -- 172.21.15.16:0/1682356889 shutdown_connections 2024-08-28T19:33:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8728108ab0 0x7f872819bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f87140b4d50 0x7f87140b7260 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8728108160 0x7f872819b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 --2- 172.21.15.16:0/1682356889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728102820 0x7f872819b080 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 -- 172.21.15.16:0/1682356889 >> 172.21.15.16:0/1682356889 conn(0x7f87280794e0 msgr2=0x7f87281041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 -- 172.21.15.16:0/1682356889 shutdown_connections 2024-08-28T19:33:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:37.316+0000 7f872d9c2640 1 -- 172.21.15.16:0/1682356889 wait complete. 2024-08-28T19:33:37.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.60 MB/sec execute 557 sec latency 11788.253 ms 2024-08-28T19:33:38.259 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:33:38.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:33:38.513 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:38.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:38 smithi016 ceph-mon[18556]: pgmap v729: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 721 KiB/s wr, 52 op/s 2024-08-28T19:33:38.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.59 MB/sec execute 558 sec latency 12788.312 ms 2024-08-28T19:33:39.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.005+0000 7f2e01c81640 1 -- 172.21.15.16:0/3645541139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc102820 msgr2=0x7f2dfc102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.005+0000 7f2e01c81640 1 --2- 172.21.15.16:0/3645541139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc102820 0x7f2dfc102c30 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009900 tx=0x7f2dec02a410 comp rx=0 tx=0).stop 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.005+0000 7f2e01c81640 1 -- 172.21.15.16:0/3645541139 shutdown_connections 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.005+0000 7f2e01c81640 1 --2- 172.21.15.16:0/3645541139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dfc108ab0 0x7f2dfc10f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.005+0000 7f2e01c81640 1 --2- 172.21.15.16:0/3645541139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2dfc108160 0x7f2dfc108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.005+0000 7f2e01c81640 1 --2- 172.21.15.16:0/3645541139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc102820 0x7f2dfc102c30 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.005+0000 7f2e01c81640 1 -- 172.21.15.16:0/3645541139 >> 172.21.15.16:0/3645541139 conn(0x7f2dfc0794e0 msgr2=0x7f2dfc0798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.005+0000 7f2e01c81640 1 -- 172.21.15.16:0/3645541139 shutdown_connections 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.005+0000 7f2e01c81640 1 -- 172.21.15.16:0/3645541139 wait complete. 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.006+0000 7f2e01c81640 1 Processor -- start 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2e01c81640 1 -- start start 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2e01c81640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc102820 0x7f2dfc1991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2e01c81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dfc108160 0x7f2dfc199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2e01c81640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2dfc108ab0 0x7f2dfc19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2e01c81640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc112780 con 0x7f2dfc102820 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2e01c81640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc112600 con 0x7f2dfc108160 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2e01c81640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc112900 con 0x7f2dfc108ab0 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2dfb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc102820 0x7f2dfc1991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2dfb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc102820 0x7f2dfc1991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42996/0 (socket says 172.21.15.16:42996) 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2dfb7fe640 1 -- 172.21.15.16:0/1331667843 learned_addr learned my addr 172.21.15.16:0/1331667843 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2dfaffd640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dfc108160 0x7f2dfc199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2dfb7fe640 1 -- 172.21.15.16:0/1331667843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2dfc108ab0 msgr2=0x7f2dfc19ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.007+0000 7f2dfbfff640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2dfc108ab0 0x7f2dfc19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2dfb7fe640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2dfc108ab0 0x7f2dfc19ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2dfb7fe640 1 -- 172.21.15.16:0/1331667843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dfc108160 msgr2=0x7f2dfc199770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2dfb7fe640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dfc108160 0x7f2dfc199770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2dfb7fe640 1 -- 172.21.15.16:0/1331667843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dfc19f140 con 0x7f2dfc102820 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2dfaffd640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dfc108160 0x7f2dfc199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2dfbfff640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2dfc108ab0 0x7f2dfc19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2dfb7fe640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc102820 0x7f2dfc1991d0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f2dec005940 tx=0x7f2dec004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2df8ff9640 1 -- 172.21.15.16:0/1331667843 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec046070 con 0x7f2dfc102820 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2df8ff9640 1 -- 172.21.15.16:0/1331667843 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dec003aa0 con 0x7f2dfc102820 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2e01c81640 1 -- 172.21.15.16:0/1331667843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dfc19f3d0 con 0x7f2dfc102820 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2df8ff9640 1 -- 172.21.15.16:0/1331667843 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec035050 con 0x7f2dfc102820 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.008+0000 7f2e01c81640 1 -- 172.21.15.16:0/1331667843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dfc06b8e0 con 0x7f2dfc102820 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.009+0000 7f2e01c81640 1 -- 172.21.15.16:0/1331667843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dfc103aa0 con 0x7f2dfc102820 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.011+0000 7f2df8ff9640 1 -- 172.21.15.16:0/1331667843 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2dec049070 con 0x7f2dfc102820 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.011+0000 7f2df8ff9640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2de40b4ca0 0x7f2de40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.012+0000 7f2dfaffd640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2de40b4ca0 0x7f2de40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.012+0000 7f2df8ff9640 1 -- 172.21.15.16:0/1331667843 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f2dec107a30 con 0x7f2dfc102820 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.012+0000 7f2dfaffd640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2de40b4ca0 0x7f2de40b71b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2df0006fd0 tx=0x7f2df0008040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.015+0000 7f2df8ff9640 1 -- 172.21.15.16:0/1331667843 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f2dec0ce450 con 0x7f2dfc102820 2024-08-28T19:33:39.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:38 smithi028 ceph-mon[21769]: pgmap v729: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 721 KiB/s wr, 52 op/s 2024-08-28T19:33:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:38 smithi115 ceph-mon[22184]: pgmap v729: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 721 KiB/s wr, 52 op/s 2024-08-28T19:33:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.241+0000 7f2e01c81640 1 -- 172.21.15.16:0/1331667843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dfc102c30 con 0x7f2dfc102820 2024-08-28T19:33:39.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.242+0000 7f2df8ff9640 1 -- 172.21.15.16:0/1331667843 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11990 (secure 0 0 0) 0x7f2dec0cdd50 con 0x7f2dfc102820 2024-08-28T19:33:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:33:39.244 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":62,"btime":"2024-08-28T19:33:29:404877+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":62,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:33:29.086264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:33:39.244 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 62 2024-08-28T19:33:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 -- 172.21.15.16:0/1331667843 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2de40b4ca0 msgr2=0x7f2de40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2de40b4ca0 0x7f2de40b71b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2df0006fd0 tx=0x7f2df0008040 comp rx=0 tx=0).stop 2024-08-28T19:33:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 -- 172.21.15.16:0/1331667843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc102820 msgr2=0x7f2dfc1991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc102820 0x7f2dfc1991d0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f2dec005940 tx=0x7f2dec004270 comp rx=0 tx=0).stop 2024-08-28T19:33:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 -- 172.21.15.16:0/1331667843 shutdown_connections 2024-08-28T19:33:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2dfc108ab0 0x7f2dfc19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2de40b4ca0 0x7f2de40b71b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dfc108160 0x7f2dfc199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:39.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 --2- 172.21.15.16:0/1331667843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc102820 0x7f2dfc1991d0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:39.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 -- 172.21.15.16:0/1331667843 >> 172.21.15.16:0/1331667843 conn(0x7f2dfc0794e0 msgr2=0x7f2dfc101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:39.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 -- 172.21.15.16:0/1331667843 shutdown_connections 2024-08-28T19:33:39.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:39.245+0000 7f2e01c81640 1 -- 172.21.15.16:0/1331667843 wait complete. 2024-08-28T19:33:39.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:33:39.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.58 MB/sec execute 559 sec latency 13788.359 ms 2024-08-28T19:33:40.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:39 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1331667843' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:33:40.132 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:40.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:39 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1331667843' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:33:40.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:39 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1331667843' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:33:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.632+0000 7f349ad2e640 1 -- 172.21.15.16:0/461339634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34941047d0 msgr2=0x7f3494104be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.632+0000 7f349ad2e640 1 --2- 172.21.15.16:0/461339634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34941047d0 0x7f3494104be0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f3488009900 tx=0x7f348802a410 comp rx=0 tx=0).stop 2024-08-28T19:33:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.632+0000 7f349ad2e640 1 -- 172.21.15.16:0/461339634 shutdown_connections 2024-08-28T19:33:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.632+0000 7f349ad2e640 1 --2- 172.21.15.16:0/461339634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349410dc30 0x7f3494110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.632+0000 7f349ad2e640 1 --2- 172.21.15.16:0/461339634 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3494105120 0x7f349410d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.632+0000 7f349ad2e640 1 --2- 172.21.15.16:0/461339634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34941047d0 0x7f3494104be0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.632+0000 7f349ad2e640 1 -- 172.21.15.16:0/461339634 >> 172.21.15.16:0/461339634 conn(0x7f3494079510 msgr2=0x7f3494079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.632+0000 7f349ad2e640 1 -- 172.21.15.16:0/461339634 shutdown_connections 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.633+0000 7f349ad2e640 1 -- 172.21.15.16:0/461339634 wait complete. 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.633+0000 7f349ad2e640 1 Processor -- start 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.633+0000 7f349ad2e640 1 -- start start 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f349ad2e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34941047d0 0x7f349419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f349ad2e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3494105120 0x7f349419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f349ad2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349410dc30 0x7f349419fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f349ad2e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3494113890 con 0x7f34941047d0 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f349ad2e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3494113710 con 0x7f349410dc30 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f349ad2e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3494113a10 con 0x7f3494105120 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f3498aca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34941047d0 0x7f349419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f3498aca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34941047d0 0x7f349419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43018/0 (socket says 172.21.15.16:43018) 2024-08-28T19:33:40.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f3498aca640 1 -- 172.21.15.16:0/550350180 learned_addr learned my addr 172.21.15.16:0/550350180 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f34992cb640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349410dc30 0x7f349419fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f3493fff640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3494105120 0x7f349419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f3498aca640 1 -- 172.21.15.16:0/550350180 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3494105120 msgr2=0x7f349419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f3498aca640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3494105120 0x7f349419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f3498aca640 1 -- 172.21.15.16:0/550350180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349410dc30 msgr2=0x7f349419fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f3498aca640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349410dc30 0x7f349419fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f3498aca640 1 -- 172.21.15.16:0/550350180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34941a0150 con 0x7f34941047d0 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f3493fff640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3494105120 0x7f349419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.634+0000 7f3498aca640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34941047d0 0x7f349419a2a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f3488005940 tx=0x7f34880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.635+0000 7f3491ffb640 1 -- 172.21.15.16:0/550350180 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348803f030 con 0x7f34941047d0 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.635+0000 7f3491ffb640 1 -- 172.21.15.16:0/550350180 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3488003cb0 con 0x7f34941047d0 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.635+0000 7f349ad2e640 1 -- 172.21.15.16:0/550350180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34941a03e0 con 0x7f34941047d0 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.635+0000 7f3491ffb640 1 -- 172.21.15.16:0/550350180 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3488035070 con 0x7f34941047d0 2024-08-28T19:33:40.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.635+0000 7f349ad2e640 1 -- 172.21.15.16:0/550350180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34941a08c0 con 0x7f34941047d0 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.637+0000 7f3491ffb640 1 -- 172.21.15.16:0/550350180 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3488042030 con 0x7f34941047d0 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.637+0000 7f34a12a3740 1 Processor -- start 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f34a12a3740 1 -- start start 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552cb9070 0x55c55305e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c55305e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55305edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f34a12a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c552fd5680 con 0x55c552c98390 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f34a12a3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c552fc4c10 con 0x55c552c901b0 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f34a12a3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c552fc8bc0 con 0x55c552cb9070 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f3493fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55305edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f3493fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55305edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43030/0 (socket says 172.21.15.16:43030) 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f3493fff640 1 -- 172.21.15.16:0/4197693881 learned_addr learned my addr 172.21.15.16:0/4197693881 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f34992cb640 1 --2- 172.21.15.16:0/4197693881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552cb9070 0x55c55305e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f3498aca640 1 --2- 172.21.15.16:0/4197693881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c55305e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f3491ffb640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34840b4cd0 0x7f34840b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f3493fff640 1 -- 172.21.15.16:0/4197693881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552cb9070 msgr2=0x55c55305e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f3493fff640 1 --2- 172.21.15.16:0/4197693881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552cb9070 0x55c55305e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f3493fff640 1 -- 172.21.15.16:0/4197693881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 msgr2=0x55c55305e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f3493fff640 1 --2- 172.21.15.16:0/4197693881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c55305e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f3493fff640 1 -- 172.21.15.16:0/4197693881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c552d24650 con 0x55c552c98390 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f34992cb640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34840b4cd0 0x7f34840b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.638+0000 7f3491ffb640 1 -- 172.21.15.16:0/550350180 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f3488105e70 con 0x7f34941047d0 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.639+0000 7f3493fff640 1 --2- 172.21.15.16:0/4197693881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55305edd0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f3480008bb0 tx=0x7f348000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4a8f045ec07ac0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.639+0000 7f347effd640 1 -- 172.21.15.16:0/4197693881 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480014070 con 0x55c552c98390 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.639+0000 7f347effd640 1 -- 172.21.15.16:0/4197693881 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34800047a0 con 0x55c552c98390 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.639+0000 7f347effd640 1 -- 172.21.15.16:0/4197693881 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480010650 con 0x55c552c98390 2024-08-28T19:33:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34a12a3740 1 -- 172.21.15.16:0/4197693881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 msgr2=0x55c55305edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34a12a3740 1 --2- 172.21.15.16:0/4197693881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55305edd0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f3480008bb0 tx=0x7f348000e720 comp rx=0 tx=0).stop 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34992cb640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34840b4cd0 0x7f34840b71e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f348c005fd0 tx=0x7f348c004a40 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34a12a3740 1 -- 172.21.15.16:0/4197693881 shutdown_connections 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34a12a3740 1 --2- 172.21.15.16:0/4197693881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55305edd0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34a12a3740 1 --2- 172.21.15.16:0/4197693881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c55305e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34a12a3740 1 --2- 172.21.15.16:0/4197693881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552cb9070 0x55c55305e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34a12a3740 1 -- 172.21.15.16:0/4197693881 >> 172.21.15.16:0/4197693881 conn(0x55c552cad460 msgr2=0x55c55305c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34a12a3740 1 -- 172.21.15.16:0/4197693881 shutdown_connections 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34a12a3740 1 -- 172.21.15.16:0/4197693881 wait complete. 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34a12a3740 1 -- start start 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.640+0000 7f34a12a3740 1 Processor -- start 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c553069a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c553069f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552cb9070 0x55c553075e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f34a12a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c552e27700 con 0x55c552c98390 2024-08-28T19:33:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f34a12a3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c552f4b750 con 0x55c552c901b0 2024-08-28T19:33:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f34a12a3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c55306cb30 con 0x55c552cb9070 2024-08-28T19:33:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f3493fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c553069f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f34a12a3740 10 client.0 _ll_register_callbacks cb 0x55c552f7ae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f3493fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c553069f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43040/0 (socket says 172.21.15.16:43040) 2024-08-28T19:33:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f3493fff640 1 -- 172.21.15.16:0/3482998148 learned_addr learned my addr 172.21.15.16:0/3482998148 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f3498aca640 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c553069a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f3493fff640 1 -- 172.21.15.16:0/3482998148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552cb9070 msgr2=0x55c553075e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f34992cb640 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552cb9070 0x55c553075e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f3493fff640 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552cb9070 0x55c553075e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f3493fff640 1 -- 172.21.15.16:0/3482998148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 msgr2=0x55c553069a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f3493fff640 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c553069a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f3493fff640 1 -- 172.21.15.16:0/3482998148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c552f84900 con 0x55c552c98390 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f3493fff640 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c553069f80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f3480000c00 tx=0x7f3480005210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae6992c670cc8db server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.641+0000 7f3473fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f3473fff640 1 -- 172.21.15.16:0/3482998148 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480014030 con 0x55c552c98390 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f3473fff640 1 -- 172.21.15.16:0/3482998148 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f3480004050 con 0x55c552c98390 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f34a12a3740 1 -- 172.21.15.16:0/3482998148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34800047a0 con 0x55c552c98390 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f3473fff640 1 -- 172.21.15.16:0/3482998148 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f3480004250 con 0x55c552c98390 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f34a12a3740 10 client.18627 fetch_fsmap learned FSMap version 62 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f34a12a3740 1 -- 172.21.15.16:0/3482998148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55c553078870 con 0x55c552c98390 2024-08-28T19:33:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f3473fff640 1 -- 172.21.15.16:0/3482998148 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34800047a0 con 0x55c552c98390 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f3473fff640 1 -- 172.21.15.16:0/3482998148 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 62) ==== 8406+0+0 (secure 0 0 0) 0x7f3480003bc0 con 0x55c552c98390 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f34a12a3740 10 client.18627 fetch_fsmap finished waiting for FSMap version 62 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f34a12a3740 10 client.18627 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f34a12a3740 20 client.18627 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55c552c779f0 0x55c55307bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f34a12a3740 4 client.18627 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.642+0000 7f34a12a3740 1 -- 172.21.15.16:0/3482998148 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c552fd75d0 con 0x55c552c779f0 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.643+0000 7f3498aca640 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55c552c779f0 0x55c55307bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.643+0000 7f3498aca640 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55c552c779f0 0x55c55307bc80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cb5341e760cde478 server_cookie=79cc66064d4c968f in_seq=0 out_seq=0 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.643+0000 7f3473fff640 10 client.18627 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f3473fff640 1 -- 172.21.15.16:0/3482998148 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c552fd75d0 con 0x55c552c779f0 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f3473fff640 10 client.18627 handle_command_reply: tid=0 2024-08-28T19:33:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f34a12a3740 1 client.18627 shutdown 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f34a12a3740 1 -- 172.21.15.16:0/3482998148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 msgr2=0x55c553069f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c553069f80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f3480000c00 tx=0x7f3480005210 comp rx=0 tx=0).stop 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f34a12a3740 1 -- 172.21.15.16:0/3482998148 shutdown_connections 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55c552c779f0 0x55c55307bc80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552cb9070 0x55c553075e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c553069f80 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f3473fff640 0 client.18627 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3482998148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c553069a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f34a12a3740 1 -- 172.21.15.16:0/3482998148 >> 172.21.15.16:0/3482998148 conn(0x55c552cad460 msgr2=0x55c5530614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f34a12a3740 1 -- 172.21.15.16:0/3482998148 shutdown_connections 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.644+0000 7f34a12a3740 1 -- 172.21.15.16:0/3482998148 wait complete. 2024-08-28T19:33:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.645+0000 7f34a12a3740 20 client.18627 trim_cache size 0 max 16384 2024-08-28T19:33:40.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.658+0000 7f34a12a3740 1 Processor -- start 2024-08-28T19:33:40.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.658+0000 7f34a12a3740 1 -- start start 2024-08-28T19:33:40.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.658+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c553075d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.658+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552c98390 0x55c553066c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.658+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552cb9070 0x55c553067170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.658+0000 7f34a12a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b0b80 con 0x55c552cb9070 2024-08-28T19:33:40.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.658+0000 7f34a12a3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b2810 con 0x55c552c901b0 2024-08-28T19:33:40.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.658+0000 7f34a12a3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b1310 con 0x55c552c98390 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.658+0000 7f34992cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552cb9070 0x55c553067170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.658+0000 7f34992cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552cb9070 0x55c553067170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43054/0 (socket says 172.21.15.16:43054) 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.658+0000 7f34992cb640 1 -- 172.21.15.16:0/3344551297 learned_addr learned my addr 172.21.15.16:0/3344551297 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.659+0000 7f3493fff640 1 --2- 172.21.15.16:0/3344551297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552c98390 0x55c553066c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.659+0000 7f3498aca640 1 --2- 172.21.15.16:0/3344551297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c553075d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.659+0000 7f34992cb640 1 -- 172.21.15.16:0/3344551297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552c98390 msgr2=0x55c553066c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.659+0000 7f34992cb640 1 --2- 172.21.15.16:0/3344551297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552c98390 0x55c553066c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.659+0000 7f34992cb640 1 -- 172.21.15.16:0/3344551297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 msgr2=0x55c553075d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.659+0000 7f34992cb640 1 --2- 172.21.15.16:0/3344551297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c553075d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.659+0000 7f34992cb640 1 -- 172.21.15.16:0/3344551297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c553060640 con 0x55c552cb9070 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.659+0000 7f34992cb640 1 --2- 172.21.15.16:0/3344551297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552cb9070 0x55c553067170 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f348c012540 tx=0x7f348c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6603ea2981f652 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.659+0000 7f347e7fc640 1 -- 172.21.15.16:0/3344551297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c01d6f0 con 0x55c552cb9070 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.659+0000 7f347e7fc640 1 -- 172.21.15.16:0/3344551297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f348c01dd40 con 0x55c552cb9070 2024-08-28T19:33:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.659+0000 7f347e7fc640 1 -- 172.21.15.16:0/3344551297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c020d90 con 0x55c552cb9070 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.660+0000 7f34a12a3740 1 -- 172.21.15.16:0/3344551297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552cb9070 msgr2=0x55c553067170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.660+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3344551297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552cb9070 0x55c553067170 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f348c012540 tx=0x7f348c016000 comp rx=0 tx=0).stop 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.660+0000 7f34a12a3740 1 -- 172.21.15.16:0/3344551297 shutdown_connections 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.660+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3344551297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552cb9070 0x55c553067170 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.660+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3344551297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552c98390 0x55c553066c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.660+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3344551297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552c901b0 0x55c553075d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.660+0000 7f34a12a3740 1 -- 172.21.15.16:0/3344551297 >> 172.21.15.16:0/3344551297 conn(0x55c552cad460 msgr2=0x55c5530682c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.660+0000 7f34a12a3740 1 -- 172.21.15.16:0/3344551297 shutdown_connections 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.660+0000 7f34a12a3740 1 -- 172.21.15.16:0/3344551297 wait complete. 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.660+0000 7f34a12a3740 1 -- start start 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.660+0000 7f34a12a3740 1 Processor -- start 2024-08-28T19:33:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552c901b0 0x55c55306e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55306e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f34a12a3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552cb9070 0x55c5530e5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f34a12a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b2110 con 0x55c552c98390 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f34a12a3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b1a10 con 0x55c552cb9070 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f34a12a3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f34941aeb80 con 0x55c552c901b0 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f34a12a3740 10 client.0 _ll_register_callbacks cb 0x7f3488109890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f3493fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55306e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f3493fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55306e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43068/0 (socket says 172.21.15.16:43068) 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f3493fff640 1 -- 172.21.15.16:0/3890055066 learned_addr learned my addr 172.21.15.16:0/3890055066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f34992cb640 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552cb9070 0x55c5530e5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f3498aca640 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552c901b0 0x55c55306e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f3493fff640 1 -- 172.21.15.16:0/3890055066 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552c901b0 msgr2=0x55c55306e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f3493fff640 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552c901b0 0x55c55306e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f3493fff640 1 -- 172.21.15.16:0/3890055066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552cb9070 msgr2=0x55c5530e5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f3493fff640 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552cb9070 0x55c5530e5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.661+0000 7f3493fff640 1 -- 172.21.15.16:0/3890055066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5530e5ac0 con 0x55c552c98390 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f3493fff640 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55306e730 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f34800104b0 tx=0x7f3480005310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed29934a54677871 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f3473fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f3473fff640 1 -- 172.21.15.16:0/3890055066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480014040 con 0x55c552c98390 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f3473fff640 1 -- 172.21.15.16:0/3890055066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f3480003fe0 con 0x55c552c98390 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f34a12a3740 1 -- 172.21.15.16:0/3890055066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f348c01dd40 con 0x55c552c98390 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f3473fff640 1 -- 172.21.15.16:0/3890055066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f3480008a30 con 0x55c552c98390 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f34a12a3740 10 client.18633 fetch_fsmap learned FSMap version 62 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f34a12a3740 1 -- 172.21.15.16:0/3890055066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55c5530e6c40 con 0x55c552c98390 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f3473fff640 1 -- 172.21.15.16:0/3890055066 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f348c01dd40 con 0x55c552c98390 2024-08-28T19:33:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f3473fff640 1 -- 172.21.15.16:0/3890055066 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 62) ==== 8406+0+0 (secure 0 0 0) 0x7f34800035f0 con 0x55c552c98390 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f34a12a3740 10 client.18633 fetch_fsmap finished waiting for FSMap version 62 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f34a12a3740 10 client.18633 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f34a12a3740 20 client.18633 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.662+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55c552c4cda0 0x55c5530e92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.663+0000 7f34a12a3740 4 client.18633 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.663+0000 7f34a12a3740 1 -- 172.21.15.16:0/3890055066 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c553060be0 con 0x55c552c4cda0 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.663+0000 7f3498aca640 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55c552c4cda0 0x55c5530e92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.663+0000 7f3498aca640 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55c552c4cda0 0x55c5530e92f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=953eff99e1041526 server_cookie=bf479cd5eff7a941 in_seq=0 out_seq=0 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.663+0000 7f3473fff640 10 client.18633 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.663+0000 7f3473fff640 1 -- 172.21.15.16:0/3890055066 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c553060be0 con 0x55c552c4cda0 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.663+0000 7f3473fff640 10 client.18633 handle_command_reply: tid=0 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.663+0000 7f34a12a3740 1 client.18633 shutdown 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f34a12a3740 1 -- 172.21.15.16:0/3890055066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 msgr2=0x55c55306e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55306e730 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f34800104b0 tx=0x7f3480005310 comp rx=0 tx=0).stop 2024-08-28T19:33:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f34a12a3740 1 -- 172.21.15.16:0/3890055066 shutdown_connections 2024-08-28T19:33:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55c552c4cda0 0x55c5530e92f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c552cb9070 0x55c5530e5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c552c98390 0x55c55306e730 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f34a12a3740 1 --2- 172.21.15.16:0/3890055066 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c552c901b0 0x55c55306e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f3473fff640 0 client.18633 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:33:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f34a12a3740 1 -- 172.21.15.16:0/3890055066 >> 172.21.15.16:0/3890055066 conn(0x55c552cad460 msgr2=0x55c55305f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f34a12a3740 1 -- 172.21.15.16:0/3890055066 shutdown_connections 2024-08-28T19:33:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f34a12a3740 1 -- 172.21.15.16:0/3890055066 wait complete. 2024-08-28T19:33:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f34a12a3740 20 client.18633 trim_cache size 0 max 16384 2024-08-28T19:33:40.672 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:33:40.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f349ad2e640 1 -- 172.21.15.16:0/550350180 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34840b4cd0 msgr2=0x7f34840b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f349ad2e640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34840b4cd0 0x7f34840b71e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f348c005fd0 tx=0x7f348c004a40 comp rx=0 tx=0).stop 2024-08-28T19:33:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f349ad2e640 1 -- 172.21.15.16:0/550350180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34941047d0 msgr2=0x7f349419a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.664+0000 7f349ad2e640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34941047d0 0x7f349419a2a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f3488005940 tx=0x7f34880042f0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.665+0000 7f349ad2e640 1 -- 172.21.15.16:0/550350180 shutdown_connections 2024-08-28T19:33:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.665+0000 7f349ad2e640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34840b4cd0 0x7f34840b71e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.665+0000 7f349ad2e640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349410dc30 0x7f349419fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.665+0000 7f349ad2e640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3494105120 0x7f349419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.665+0000 7f349ad2e640 1 --2- 172.21.15.16:0/550350180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34941047d0 0x7f349419a2a0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.665+0000 7f349ad2e640 1 -- 172.21.15.16:0/550350180 >> 172.21.15.16:0/550350180 conn(0x7f3494079510 msgr2=0x7f34941005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.665+0000 7f349ad2e640 1 -- 172.21.15.16:0/550350180 shutdown_connections 2024-08-28T19:33:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:40.665+0000 7f349ad2e640 1 -- 172.21.15.16:0/550350180 wait complete. 2024-08-28T19:33:40.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.58 MB/sec execute 560 sec latency 14788.386 ms 2024-08-28T19:33:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:40 smithi115 ceph-mon[22184]: pgmap v730: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 704 KiB/s wr, 43 op/s 2024-08-28T19:33:41.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:40 smithi016 ceph-mon[18556]: pgmap v730: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 704 KiB/s wr, 43 op/s 2024-08-28T19:33:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:40 smithi028 ceph-mon[21769]: pgmap v730: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 704 KiB/s wr, 43 op/s 2024-08-28T19:33:41.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:33:41.585 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:41.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.57 MB/sec execute 561 sec latency 15788.411 ms 2024-08-28T19:33:42.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.076+0000 7faa36b4c640 1 -- 172.21.15.16:0/993945107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa301081c0 msgr2=0x7faa301085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:42.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.076+0000 7faa36b4c640 1 --2- 172.21.15.16:0/993945107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa301081c0 0x7faa301085d0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009900 tx=0x7faa1c02a410 comp rx=0 tx=0).stop 2024-08-28T19:33:42.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.077+0000 7faa36b4c640 1 -- 172.21.15.16:0/993945107 shutdown_connections 2024-08-28T19:33:42.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.077+0000 7faa36b4c640 1 --2- 172.21.15.16:0/993945107 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa30108b10 0x7faa3010f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.077+0000 7faa36b4c640 1 --2- 172.21.15.16:0/993945107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa301081c0 0x7faa301085d0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.077+0000 7faa36b4c640 1 --2- 172.21.15.16:0/993945107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa30102880 0x7faa30102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.077+0000 7faa36b4c640 1 -- 172.21.15.16:0/993945107 >> 172.21.15.16:0/993945107 conn(0x7faa30079510 msgr2=0x7faa30079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.077+0000 7faa36b4c640 1 -- 172.21.15.16:0/993945107 shutdown_connections 2024-08-28T19:33:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.077+0000 7faa36b4c640 1 -- 172.21.15.16:0/993945107 wait complete. 2024-08-28T19:33:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.077+0000 7faa36b4c640 1 Processor -- start 2024-08-28T19:33:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.077+0000 7faa36b4c640 1 -- start start 2024-08-28T19:33:42.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa36b4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa30102880 0x7faa3019b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa36b4c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa301081c0 0x7faa3019b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa36b4c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa30108b10 0x7faa3019bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa36b4c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa301127a0 con 0x7faa301081c0 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa36b4c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa30112620 con 0x7faa30102880 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa36b4c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faa30112920 con 0x7faa30108b10 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa301081c0 0x7faa3019b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa301081c0 0x7faa3019b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43082/0 (socket says 172.21.15.16:43082) 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa2ffff640 1 -- 172.21.15.16:0/612633811 learned_addr learned my addr 172.21.15.16:0/612633811 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa348e8640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa30102880 0x7faa3019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa350e9640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa30108b10 0x7faa3019bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa2ffff640 1 -- 172.21.15.16:0/612633811 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa30108b10 msgr2=0x7faa3019bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa2ffff640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa30108b10 0x7faa3019bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa2ffff640 1 -- 172.21.15.16:0/612633811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa30102880 msgr2=0x7faa3019b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa2ffff640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa30102880 0x7faa3019b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa2ffff640 1 -- 172.21.15.16:0/612633811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa301a6310 con 0x7faa301081c0 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.078+0000 7faa350e9640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa30108b10 0x7faa3019bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.079+0000 7faa2ffff640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa301081c0 0x7faa3019b650 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7faa1c02a930 tx=0x7faa1c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.079+0000 7faa2dffb640 1 -- 172.21.15.16:0/612633811 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c03f030 con 0x7faa301081c0 2024-08-28T19:33:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.079+0000 7faa36b4c640 1 -- 172.21.15.16:0/612633811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa301a65a0 con 0x7faa301081c0 2024-08-28T19:33:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.079+0000 7faa2dffb640 1 -- 172.21.15.16:0/612633811 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa1c003aa0 con 0x7faa301081c0 2024-08-28T19:33:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.079+0000 7faa2dffb640 1 -- 172.21.15.16:0/612633811 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c035070 con 0x7faa301081c0 2024-08-28T19:33:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.079+0000 7faa36b4c640 1 -- 172.21.15.16:0/612633811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa301a6a80 con 0x7faa301081c0 2024-08-28T19:33:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.081+0000 7faa3d0c1740 1 Processor -- start 2024-08-28T19:33:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa3d0c1740 1 -- start start 2024-08-28T19:33:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc7a070 0x562b0c01f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc511b0 0x562b0c01f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c01fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562b0bf96680 con 0x562b0bc511b0 2024-08-28T19:33:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b0bf85c10 con 0x562b0bc7a070 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562b0bf89bc0 con 0x562b0bc59390 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa2dffb640 1 -- 172.21.15.16:0/612633811 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7faa1c042030 con 0x7faa301081c0 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa348e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc511b0 0x562b0c01f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa348e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc511b0 0x562b0c01f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43094/0 (socket says 172.21.15.16:43094) 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa348e8640 1 -- 172.21.15.16:0/848248671 learned_addr learned my addr 172.21.15.16:0/848248671 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa350e9640 1 --2- 172.21.15.16:0/848248671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc7a070 0x562b0c01f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa2ffff640 1 --2- 172.21.15.16:0/848248671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c01fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa348e8640 1 -- 172.21.15.16:0/848248671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 msgr2=0x562b0c01fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa348e8640 1 --2- 172.21.15.16:0/848248671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c01fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa348e8640 1 -- 172.21.15.16:0/848248671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc7a070 msgr2=0x562b0c01f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa348e8640 1 --2- 172.21.15.16:0/848248671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc7a070 0x562b0c01f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa348e8640 1 -- 172.21.15.16:0/848248671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b0bce5650 con 0x562b0bc511b0 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.082+0000 7faa350e9640 1 --2- 172.21.15.16:0/848248671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc7a070 0x562b0c01f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.083+0000 7faa348e8640 1 --2- 172.21.15.16:0/848248671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc511b0 0x562b0c01f850 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7faa2400ac80 tx=0x7faa2400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0965bc69c7070ea server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:42.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.083+0000 7faa1affd640 1 -- 172.21.15.16:0/848248671 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2401d070 con 0x562b0bc511b0 2024-08-28T19:33:42.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.083+0000 7faa2dffb640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa200b4cd0 0x7faa200b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.083+0000 7faa1affd640 1 -- 172.21.15.16:0/848248671 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa24018af0 con 0x562b0bc511b0 2024-08-28T19:33:42.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.083+0000 7faa1affd640 1 -- 172.21.15.16:0/848248671 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa24010ae0 con 0x562b0bc511b0 2024-08-28T19:33:42.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.083+0000 7faa2dffb640 1 -- 172.21.15.16:0/612633811 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7faa1c105eb0 con 0x7faa301081c0 2024-08-28T19:33:42.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.083+0000 7faa350e9640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa200b4cd0 0x7faa200b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa3d0c1740 1 -- 172.21.15.16:0/848248671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc511b0 msgr2=0x562b0c01f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa350e9640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa200b4cd0 0x7faa200b71e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7faa2800a520 tx=0x7faa2800a4b0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/848248671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc511b0 0x562b0c01f850 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7faa2400ac80 tx=0x7faa2400f790 comp rx=0 tx=0).stop 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa3d0c1740 1 -- 172.21.15.16:0/848248671 shutdown_connections 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/848248671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c01fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/848248671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc511b0 0x562b0c01f850 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/848248671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc7a070 0x562b0c01f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa3d0c1740 1 -- 172.21.15.16:0/848248671 >> 172.21.15.16:0/848248671 conn(0x562b0bc6e460 msgr2=0x562b0c01d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa3d0c1740 1 -- 172.21.15.16:0/848248671 shutdown_connections 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa3d0c1740 1 -- 172.21.15.16:0/848248671 wait complete. 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa3d0c1740 1 -- start start 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.084+0000 7faa3d0c1740 1 Processor -- start 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc511b0 0x562b0c02aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c02afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c036e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562b0bde8700 con 0x562b0bc7a070 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b0bf0c750 con 0x562b0bc511b0 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562b0c02daf0 con 0x562b0bc59390 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa3d0c1740 10 client.0 _ll_register_callbacks cb 0x562b0bf3be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa350e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c036e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa350e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c036e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43102/0 (socket says 172.21.15.16:43102) 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa350e9640 1 -- 172.21.15.16:0/1137295640 learned_addr learned my addr 172.21.15.16:0/1137295640 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa2ffff640 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c02afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa348e8640 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc511b0 0x562b0c02aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa350e9640 1 -- 172.21.15.16:0/1137295640 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 msgr2=0x562b0c02afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa350e9640 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c02afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa350e9640 1 -- 172.21.15.16:0/1137295640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc511b0 msgr2=0x562b0c02aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa350e9640 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc511b0 0x562b0c02aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.085+0000 7faa350e9640 1 -- 172.21.15.16:0/1137295640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b0bf45900 con 0x562b0bc7a070 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa350e9640 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c036e40 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7faa28011e10 tx=0x7faa28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392a74d5c01005cc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa0ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa0ffff640 1 -- 172.21.15.16:0/1137295640 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2801e620 con 0x562b0bc7a070 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa0ffff640 1 -- 172.21.15.16:0/1137295640 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7faa2801e7c0 con 0x562b0bc7a070 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1137295640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa24018af0 con 0x562b0bc7a070 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa0ffff640 1 -- 172.21.15.16:0/1137295640 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7faa2802f050 con 0x562b0bc7a070 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa3d0c1740 10 client.18645 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1137295640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x562b0c039920 con 0x562b0bc7a070 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa0ffff640 1 -- 172.21.15.16:0/1137295640 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa24018af0 con 0x562b0bc7a070 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa0ffff640 1 -- 172.21.15.16:0/1137295640 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7faa2801c930 con 0x562b0bc7a070 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa3d0c1740 10 client.18645 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa3d0c1740 10 client.18645 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:33:42.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.086+0000 7faa3d0c1740 20 client.18645 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.087+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562b0bc389f0 0x562b0c03cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.087+0000 7faa3d0c1740 4 client.18645 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.087+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1137295640 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b0bf985d0 con 0x562b0bc389f0 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.087+0000 7faa348e8640 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562b0bc389f0 0x562b0c03cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.087+0000 7faa0ffff640 10 client.18645 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.087+0000 7faa348e8640 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562b0bc389f0 0x562b0c03cf70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=52933647b355a277 server_cookie=3dbed1237e2e6165 in_seq=0 out_seq=0 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa0ffff640 1 -- 172.21.15.16:0/1137295640 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562b0bf985d0 con 0x562b0bc389f0 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa0ffff640 10 client.18645 handle_command_reply: tid=0 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa3d0c1740 1 client.18645 shutdown 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1137295640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 msgr2=0x562b0c036e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c036e40 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7faa28011e10 tx=0x7faa28004a40 comp rx=0 tx=0).stop 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1137295640 shutdown_connections 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562b0bc389f0 0x562b0c03cf70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c036e40 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c02afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1137295640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc511b0 0x562b0c02aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa0ffff640 0 client.18645 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:33:42.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1137295640 >> 172.21.15.16:0/1137295640 conn(0x562b0bc6e460 msgr2=0x562b0c0224a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:42.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1137295640 shutdown_connections 2024-08-28T19:33:42.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.088+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1137295640 wait complete. 2024-08-28T19:33:42.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.089+0000 7faa3d0c1740 20 client.18645 trim_cache size 0 max 16384 2024-08-28T19:33:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa3d0c1740 1 Processor -- start 2024-08-28T19:33:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa3d0c1740 1 -- start start 2024-08-28T19:33:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc511b0 0x562b0c022b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c027c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c0281a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa301b0cd0 con 0x562b0bc7a070 2024-08-28T19:33:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa301b28d0 con 0x562b0bc511b0 2024-08-28T19:33:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faa301b13d0 con 0x562b0bc59390 2024-08-28T19:33:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa350e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c0281a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa350e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c0281a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43116/0 (socket says 172.21.15.16:43116) 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa350e9640 1 -- 172.21.15.16:0/4208154393 learned_addr learned my addr 172.21.15.16:0/4208154393 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.108+0000 7faa348e8640 1 --2- 172.21.15.16:0/4208154393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc511b0 0x562b0c022b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa2ffff640 1 --2- 172.21.15.16:0/4208154393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c027c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa350e9640 1 -- 172.21.15.16:0/4208154393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 msgr2=0x562b0c027c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa350e9640 1 --2- 172.21.15.16:0/4208154393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c027c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa350e9640 1 -- 172.21.15.16:0/4208154393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc511b0 msgr2=0x562b0c022b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa350e9640 1 --2- 172.21.15.16:0/4208154393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc511b0 0x562b0c022b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa350e9640 1 -- 172.21.15.16:0/4208154393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b0c021600 con 0x562b0bc7a070 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa348e8640 1 --2- 172.21.15.16:0/4208154393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc511b0 0x562b0c022b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa2ffff640 1 --2- 172.21.15.16:0/4208154393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c027c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa350e9640 1 --2- 172.21.15.16:0/4208154393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c0281a0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7faa28005c40 tx=0x7faa28011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5d4d39277f2a08 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa1a7fc640 1 -- 172.21.15.16:0/4208154393 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa28039020 con 0x562b0bc7a070 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa1a7fc640 1 -- 172.21.15.16:0/4208154393 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa2801ca60 con 0x562b0bc7a070 2024-08-28T19:33:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.109+0000 7faa1a7fc640 1 -- 172.21.15.16:0/4208154393 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa28035d80 con 0x562b0bc7a070 2024-08-28T19:33:42.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.110+0000 7faa3d0c1740 1 -- 172.21.15.16:0/4208154393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 msgr2=0x562b0c0281a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:42.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.110+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/4208154393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c0281a0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7faa28005c40 tx=0x7faa28011aa0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.110+0000 7faa3d0c1740 1 -- 172.21.15.16:0/4208154393 shutdown_connections 2024-08-28T19:33:42.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.110+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/4208154393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc7a070 0x562b0c0281a0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.110+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/4208154393 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc59390 0x562b0c027c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.110+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/4208154393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc511b0 0x562b0c022b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.110+0000 7faa3d0c1740 1 -- 172.21.15.16:0/4208154393 >> 172.21.15.16:0/4208154393 conn(0x562b0bc6e460 msgr2=0x562b0c022410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.110+0000 7faa3d0c1740 1 -- 172.21.15.16:0/4208154393 shutdown_connections 2024-08-28T19:33:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.110+0000 7faa3d0c1740 1 -- 172.21.15.16:0/4208154393 wait complete. 2024-08-28T19:33:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.110+0000 7faa3d0c1740 1 -- start start 2024-08-28T19:33:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa3d0c1740 1 Processor -- start 2024-08-28T19:33:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc511b0 0x562b0c02f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc59390 0x562b0c02fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa3d0c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc7a070 0x562b0c0a0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa301b21d0 con 0x562b0bc59390 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa301b1ad0 con 0x562b0bc7a070 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa3d0c1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faa301aecd0 con 0x562b0bc511b0 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa3d0c1740 10 client.0 _ll_register_callbacks cb 0x7faa2800ad00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc59390 0x562b0c02fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc59390 0x562b0c02fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43128/0 (socket says 172.21.15.16:43128) 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa2ffff640 1 -- 172.21.15.16:0/1363930537 learned_addr learned my addr 172.21.15.16:0/1363930537 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa348e8640 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc511b0 0x562b0c02f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa350e9640 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc7a070 0x562b0c0a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa2ffff640 1 -- 172.21.15.16:0/1363930537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc511b0 msgr2=0x562b0c02f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa2ffff640 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc511b0 0x562b0c02f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa2ffff640 1 -- 172.21.15.16:0/1363930537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc7a070 msgr2=0x562b0c0a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa2ffff640 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc7a070 0x562b0c0a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.111+0000 7faa2ffff640 1 -- 172.21.15.16:0/1363930537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b0c0a0fb0 con 0x562b0bc59390 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa2ffff640 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc59390 0x562b0c02fbc0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7faa1c031440 tx=0x7faa1c032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06d8eecefd20e34 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa0ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa0ffff640 1 -- 172.21.15.16:0/1363930537 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c049040 con 0x562b0bc59390 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa0ffff640 1 -- 172.21.15.16:0/1363930537 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7faa1c0c8460 con 0x562b0bc59390 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1363930537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa2801ca60 con 0x562b0bc59390 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa0ffff640 1 -- 172.21.15.16:0/1363930537 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7faa1c0cb940 con 0x562b0bc59390 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa3d0c1740 10 client.18651 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1363930537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x562b0c0a8240 con 0x562b0bc59390 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa0ffff640 1 -- 172.21.15.16:0/1363930537 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa1c0cbce0 con 0x562b0bc59390 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa0ffff640 1 -- 172.21.15.16:0/1363930537 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7faa1c03baf0 con 0x562b0bc59390 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa3d0c1740 10 client.18651 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.112+0000 7faa3d0c1740 10 client.18651 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:33:42.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.113+0000 7faa3d0c1740 20 client.18651 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.113+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562b0bc0dda0 0x562b0c0aaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.113+0000 7faa3d0c1740 4 client.18651 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.113+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1363930537 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7faa2801ca60 con 0x562b0bc0dda0 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.113+0000 7faa348e8640 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562b0bc0dda0 0x562b0c0aaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.113+0000 7faa0ffff640 10 client.18651 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.113+0000 7faa348e8640 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562b0bc0dda0 0x562b0c0aaa10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=10aa0563d1e9d94 server_cookie=2b0304fafa6c90f9 in_seq=0 out_seq=0 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.113+0000 7faa0ffff640 1 -- 172.21.15.16:0/1363930537 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7faa2801ca60 con 0x562b0bc0dda0 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.113+0000 7faa0ffff640 10 client.18651 handle_command_reply: tid=0 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.113+0000 7faa3d0c1740 1 client.18651 shutdown 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1363930537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc59390 msgr2=0x562b0c02fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc59390 0x562b0c02fbc0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7faa1c031440 tx=0x7faa1c032f60 comp rx=0 tx=0).stop 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1363930537 shutdown_connections 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x562b0bc0dda0 0x562b0c0aaa10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0bc7a070 0x562b0c0a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b0bc59390 0x562b0c02fbc0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa3d0c1740 1 --2- 172.21.15.16:0/1363930537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562b0bc511b0 0x562b0c02f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1363930537 >> 172.21.15.16:0/1363930537 conn(0x562b0bc6e460 msgr2=0x562b0c0202c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:42.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa0ffff640 0 client.18651 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:33:42.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1363930537 shutdown_connections 2024-08-28T19:33:42.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa3d0c1740 1 -- 172.21.15.16:0/1363930537 wait complete. 2024-08-28T19:33:42.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.114+0000 7faa3d0c1740 20 client.18651 trim_cache size 0 max 16384 2024-08-28T19:33:42.120 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:33:42.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 -- 172.21.15.16:0/612633811 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa200b4cd0 msgr2=0x7faa200b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:42.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa200b4cd0 0x7faa200b71e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7faa2800a520 tx=0x7faa2800a4b0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 -- 172.21.15.16:0/612633811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa301081c0 msgr2=0x7faa3019b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:42.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa301081c0 0x7faa3019b650 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7faa1c02a930 tx=0x7faa1c004270 comp rx=0 tx=0).stop 2024-08-28T19:33:42.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 -- 172.21.15.16:0/612633811 shutdown_connections 2024-08-28T19:33:42.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa30108b10 0x7faa3019bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa301081c0 0x7faa3019b650 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa200b4cd0 0x7faa200b71e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 --2- 172.21.15.16:0/612633811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa30102880 0x7faa3019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:42.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 -- 172.21.15.16:0/612633811 >> 172.21.15.16:0/612633811 conn(0x7faa30079510 msgr2=0x7faa301062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:42.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 -- 172.21.15.16:0/612633811 shutdown_connections 2024-08-28T19:33:42.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:42.115+0000 7faa36b4c640 1 -- 172.21.15.16:0/612633811 wait complete. 2024-08-28T19:33:42.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:33:42.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.56 MB/sec execute 562 sec latency 16788.456 ms 2024-08-28T19:33:43.082 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:43.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:42 smithi115 ceph-mon[22184]: pgmap v731: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 732 KiB/s wr, 47 op/s 2024-08-28T19:33:43.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:42 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:43.202 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:42 smithi016 ceph-mon[18556]: pgmap v731: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 732 KiB/s wr, 47 op/s 2024-08-28T19:33:43.202 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:42 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:42 smithi028 ceph-mon[21769]: pgmap v731: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 732 KiB/s wr, 47 op/s 2024-08-28T19:33:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:42 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.710+0000 7f5075aff640 1 -- 172.21.15.16:0/3319777660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5070108ab0 msgr2=0x7f507010f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.710+0000 7f5075aff640 1 --2- 172.21.15.16:0/3319777660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5070108ab0 0x7f507010f2e0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f5068009ca0 tx=0x7f506802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.710+0000 7f5075aff640 1 -- 172.21.15.16:0/3319777660 shutdown_connections 2024-08-28T19:33:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.711+0000 7f5075aff640 1 --2- 172.21.15.16:0/3319777660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5070108ab0 0x7f507010f2e0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.711+0000 7f5075aff640 1 --2- 172.21.15.16:0/3319777660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5070108160 0x7f5070108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.711+0000 7f5075aff640 1 --2- 172.21.15.16:0/3319777660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5070102820 0x7f5070102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.711+0000 7f5075aff640 1 -- 172.21.15.16:0/3319777660 >> 172.21.15.16:0/3319777660 conn(0x7f50700794e0 msgr2=0x7f50700798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.711+0000 7f5075aff640 1 -- 172.21.15.16:0/3319777660 shutdown_connections 2024-08-28T19:33:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.711+0000 7f5075aff640 1 -- 172.21.15.16:0/3319777660 wait complete. 2024-08-28T19:33:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.712+0000 7f5075aff640 1 Processor -- start 2024-08-28T19:33:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.712+0000 7f5075aff640 1 -- start start 2024-08-28T19:33:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.712+0000 7f5075aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5070102820 0x7f507019b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.712+0000 7f5075aff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5070108160 0x7f507019b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.712+0000 7f5075aff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5070108ab0 0x7f507019bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.712+0000 7f5075aff640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5070112730 con 0x7f5070108160 2024-08-28T19:33:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.712+0000 7f5075aff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50701125b0 con 0x7f5070102820 2024-08-28T19:33:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.712+0000 7f5075aff640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f50701128b0 con 0x7f5070108ab0 2024-08-28T19:33:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5070102820 0x7f507019b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5070108160 0x7f507019b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5070102820 0x7f507019b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:52442/0 (socket says 172.21.15.16:52442) 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506f7fe640 1 -- 172.21.15.16:0/763067289 learned_addr learned my addr 172.21.15.16:0/763067289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506effd640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5070108160 0x7f507019b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43150/0 (socket says 172.21.15.16:43150) 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506ffff640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5070108ab0 0x7f507019bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506effd640 1 -- 172.21.15.16:0/763067289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5070108ab0 msgr2=0x7f507019bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506effd640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5070108ab0 0x7f507019bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506effd640 1 -- 172.21.15.16:0/763067289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5070102820 msgr2=0x7f507019b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506effd640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5070102820 0x7f507019b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506effd640 1 -- 172.21.15.16:0/763067289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50701a6320 con 0x7f5070108160 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506f7fe640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5070102820 0x7f507019b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506effd640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5070108160 0x7f507019b5a0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f506400d6a0 tx=0x7f506400db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506cff9640 1 -- 172.21.15.16:0/763067289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5064017070 con 0x7f5070108160 2024-08-28T19:33:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506cff9640 1 -- 172.21.15.16:0/763067289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5064009820 con 0x7f5070108160 2024-08-28T19:33:43.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f5075aff640 1 -- 172.21.15.16:0/763067289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50701a6610 con 0x7f5070108160 2024-08-28T19:33:43.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.713+0000 7f506cff9640 1 -- 172.21.15.16:0/763067289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5064004c40 con 0x7f5070108160 2024-08-28T19:33:43.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.714+0000 7f5075aff640 1 -- 172.21.15.16:0/763067289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50701a6b50 con 0x7f5070108160 2024-08-28T19:33:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.716+0000 7f507c074740 1 Processor -- start 2024-08-28T19:33:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.716+0000 7f507c074740 1 -- start start 2024-08-28T19:33:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506cff9640 1 -- 172.21.15.16:0/763067289 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5064005220 con 0x7f5070108160 2024-08-28T19:33:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f507c074740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ef9070 0x55798829e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f507c074740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed01b0 0x55798829e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f507c074740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987ed8390 0x55798829ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f507c074740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557988215680 con 0x557987ef9070 2024-08-28T19:33:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f507c074740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557988204c10 con 0x557987ed8390 2024-08-28T19:33:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f507c074740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557988208bc0 con 0x557987ed01b0 2024-08-28T19:33:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ef9070 0x55798829e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ef9070 0x55798829e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43162/0 (socket says 172.21.15.16:43162) 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506ffff640 1 -- 172.21.15.16:0/2992614566 learned_addr learned my addr 172.21.15.16:0/2992614566 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506f7fe640 1 --2- 172.21.15.16:0/2992614566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed01b0 0x55798829e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506effd640 1 --2- 172.21.15.16:0/2992614566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987ed8390 0x55798829ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506ffff640 1 -- 172.21.15.16:0/2992614566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed01b0 msgr2=0x55798829e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506ffff640 1 --2- 172.21.15.16:0/2992614566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed01b0 0x55798829e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506ffff640 1 -- 172.21.15.16:0/2992614566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987ed8390 msgr2=0x55798829ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506ffff640 1 --2- 172.21.15.16:0/2992614566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987ed8390 0x55798829ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506ffff640 1 -- 172.21.15.16:0/2992614566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557987f64650 con 0x557987ef9070 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.717+0000 7f506ffff640 1 --2- 172.21.15.16:0/2992614566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ef9070 0x55798829e310 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f50680098b0 tx=0x7f50680098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc289037a5b3766 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.718+0000 7f506cff9640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50580b4ca0 0x7f50580b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.718+0000 7f5055ffb640 1 -- 172.21.15.16:0/2992614566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068031cf0 con 0x557987ef9070 2024-08-28T19:33:43.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.718+0000 7f506f7fe640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50580b4ca0 0x7f50580b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.718+0000 7f506cff9640 1 -- 172.21.15.16:0/763067289 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f50640e5100 con 0x7f5070108160 2024-08-28T19:33:43.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.718+0000 7f5055ffb640 1 -- 172.21.15.16:0/2992614566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5068036030 con 0x557987ef9070 2024-08-28T19:33:43.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.718+0000 7f5055ffb640 1 -- 172.21.15.16:0/2992614566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50680042a0 con 0x557987ef9070 2024-08-28T19:33:43.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.719+0000 7f507c074740 1 -- 172.21.15.16:0/2992614566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ef9070 msgr2=0x55798829e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.719+0000 7f506f7fe640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50580b4ca0 0x7f50580b71b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55798829cf90 tx=0x7f506000b410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.719+0000 7f507c074740 1 --2- 172.21.15.16:0/2992614566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ef9070 0x55798829e310 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f50680098b0 tx=0x7f50680098e0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.719+0000 7f507c074740 1 -- 172.21.15.16:0/2992614566 shutdown_connections 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.719+0000 7f507c074740 1 --2- 172.21.15.16:0/2992614566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987ed8390 0x55798829ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.719+0000 7f507c074740 1 --2- 172.21.15.16:0/2992614566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed01b0 0x55798829e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.719+0000 7f507c074740 1 --2- 172.21.15.16:0/2992614566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ef9070 0x55798829e310 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.719+0000 7f507c074740 1 -- 172.21.15.16:0/2992614566 >> 172.21.15.16:0/2992614566 conn(0x557987eed460 msgr2=0x55798829c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.719+0000 7f507c074740 1 -- 172.21.15.16:0/2992614566 shutdown_connections 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.719+0000 7f507c074740 1 -- 172.21.15.16:0/2992614566 wait complete. 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.719+0000 7f507c074740 1 -- start start 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f507c074740 1 Processor -- start 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f507c074740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882a9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f507c074740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987ed8390 0x5579882a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f507c074740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ef9070 0x5579882b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f507c074740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557988067700 con 0x557987ed01b0 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f507c074740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55798818b750 con 0x557987ed8390 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f507c074740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5579882ac910 con 0x557987ef9070 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f507c074740 10 client.0 _ll_register_callbacks cb 0x5579881bae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f506f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882a9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f506f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882a9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43176/0 (socket says 172.21.15.16:43176) 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f506f7fe640 1 -- 172.21.15.16:0/3848132443 learned_addr learned my addr 172.21.15.16:0/3848132443 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f506effd640 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987ed8390 0x5579882a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f506ffff640 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ef9070 0x5579882b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f506f7fe640 1 -- 172.21.15.16:0/3848132443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ef9070 msgr2=0x5579882b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f506f7fe640 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ef9070 0x5579882b5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f506f7fe640 1 -- 172.21.15.16:0/3848132443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987ed8390 msgr2=0x5579882a9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f506f7fe640 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987ed8390 0x5579882a9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.720+0000 7f506f7fe640 1 -- 172.21.15.16:0/3848132443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579881c4900 con 0x557987ed01b0 2024-08-28T19:33:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.721+0000 7f506f7fe640 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882a9ab0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f5060013fd0 tx=0x7f50600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792fda0b046a6cbb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.721+0000 7f504effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.721+0000 7f504effd640 1 -- 172.21.15.16:0/3848132443 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506001dcc0 con 0x557987ed01b0 2024-08-28T19:33:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.721+0000 7f504effd640 1 -- 172.21.15.16:0/3848132443 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f5060029480 con 0x557987ed01b0 2024-08-28T19:33:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.721+0000 7f507c074740 1 -- 172.21.15.16:0/3848132443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5068036030 con 0x557987ed01b0 2024-08-28T19:33:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.721+0000 7f504effd640 1 -- 172.21.15.16:0/3848132443 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f506001de60 con 0x557987ed01b0 2024-08-28T19:33:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.721+0000 7f507c074740 10 client.18663 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.721+0000 7f507c074740 1 -- 172.21.15.16:0/3848132443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x5579882b8770 con 0x557987ed01b0 2024-08-28T19:33:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.721+0000 7f504effd640 1 -- 172.21.15.16:0/3848132443 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5068036030 con 0x557987ed01b0 2024-08-28T19:33:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.721+0000 7f504effd640 1 -- 172.21.15.16:0/3848132443 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7f506001e6f0 con 0x557987ed01b0 2024-08-28T19:33:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.721+0000 7f507c074740 10 client.18663 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.722+0000 7f507c074740 10 client.18663 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:33:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.722+0000 7f507c074740 20 client.18663 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.722+0000 7f507c074740 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557987eb79f0 0x5579882bbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.722+0000 7f507c074740 4 client.18663 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.722+0000 7f507c074740 1 -- 172.21.15.16:0/3848132443 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579882175d0 con 0x557987eb79f0 2024-08-28T19:33:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.722+0000 7f506effd640 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557987eb79f0 0x5579882bbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.722+0000 7f504effd640 10 client.18663 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:33:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.722+0000 7f506effd640 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557987eb79f0 0x5579882bbcc0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d64ce581cbaf49e3 server_cookie=8f5cba0be4044a11 in_seq=0 out_seq=0 2024-08-28T19:33:43.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f504effd640 1 -- 172.21.15.16:0/3848132443 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5579882175d0 con 0x557987eb79f0 2024-08-28T19:33:43.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f504effd640 10 client.18663 handle_command_reply: tid=0 2024-08-28T19:33:43.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f507c074740 1 client.18663 shutdown 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f507c074740 1 -- 172.21.15.16:0/3848132443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 msgr2=0x5579882a9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f507c074740 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882a9ab0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f5060013fd0 tx=0x7f50600046c0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f507c074740 1 -- 172.21.15.16:0/3848132443 shutdown_connections 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f507c074740 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557987eb79f0 0x5579882bbcc0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f504effd640 0 client.18663 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f507c074740 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ef9070 0x5579882b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f507c074740 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987ed8390 0x5579882a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f507c074740 1 --2- 172.21.15.16:0/3848132443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882a9ab0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f507c074740 1 -- 172.21.15.16:0/3848132443 >> 172.21.15.16:0/3848132443 conn(0x557987eed460 msgr2=0x5579882a14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f507c074740 1 -- 172.21.15.16:0/3848132443 shutdown_connections 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.723+0000 7f507c074740 1 -- 172.21.15.16:0/3848132443 wait complete. 2024-08-28T19:33:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.724+0000 7f507c074740 20 client.18663 trim_cache size 0 max 16384 2024-08-28T19:33:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f507c074740 1 Processor -- start 2024-08-28T19:33:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f507c074740 1 -- start start 2024-08-28T19:33:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f507c074740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987eb79f0 0x5579882a1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f507c074740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed01b0 0x5579882a1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f507c074740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed8390 0x5579882a6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f507c074740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50701b0d00 con 0x557987ed8390 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f507c074740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50701b2900 con 0x557987eb79f0 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f507c074740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f50701b1400 con 0x557987ed01b0 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f506ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed8390 0x5579882a6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f506ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed8390 0x5579882a6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43190/0 (socket says 172.21.15.16:43190) 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f506ffff640 1 -- 172.21.15.16:0/376581386 learned_addr learned my addr 172.21.15.16:0/376581386 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.737+0000 7f506f7fe640 1 --2- 172.21.15.16:0/376581386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987eb79f0 0x5579882a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.738+0000 7f506effd640 1 --2- 172.21.15.16:0/376581386 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed01b0 0x5579882a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.738+0000 7f506ffff640 1 -- 172.21.15.16:0/376581386 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed01b0 msgr2=0x5579882a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.738+0000 7f506ffff640 1 --2- 172.21.15.16:0/376581386 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed01b0 0x5579882a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.738+0000 7f506ffff640 1 -- 172.21.15.16:0/376581386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987eb79f0 msgr2=0x5579882a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.738+0000 7f506ffff640 1 --2- 172.21.15.16:0/376581386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987eb79f0 0x5579882a1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.738+0000 7f506ffff640 1 -- 172.21.15.16:0/376581386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579882aa3e0 con 0x557987ed8390 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.738+0000 7f506effd640 1 --2- 172.21.15.16:0/376581386 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed01b0 0x5579882a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:33:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.738+0000 7f506ffff640 1 --2- 172.21.15.16:0/376581386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed8390 0x5579882a6ca0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f5068006010 tx=0x7f5068000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b88e00ae054f43 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.738+0000 7f50557fa640 1 -- 172.21.15.16:0/376581386 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506804a040 con 0x557987ed8390 2024-08-28T19:33:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.738+0000 7f50557fa640 1 -- 172.21.15.16:0/376581386 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5068036040 con 0x557987ed8390 2024-08-28T19:33:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.738+0000 7f50557fa640 1 -- 172.21.15.16:0/376581386 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50680042a0 con 0x557987ed8390 2024-08-28T19:33:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.739+0000 7f507c074740 1 -- 172.21.15.16:0/376581386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed8390 msgr2=0x5579882a6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.739+0000 7f507c074740 1 --2- 172.21.15.16:0/376581386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed8390 0x5579882a6ca0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f5068006010 tx=0x7f5068000f40 comp rx=0 tx=0).stop 2024-08-28T19:33:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.739+0000 7f507c074740 1 -- 172.21.15.16:0/376581386 shutdown_connections 2024-08-28T19:33:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.739+0000 7f507c074740 1 --2- 172.21.15.16:0/376581386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed8390 0x5579882a6ca0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.739+0000 7f507c074740 1 --2- 172.21.15.16:0/376581386 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed01b0 0x5579882a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.739+0000 7f507c074740 1 --2- 172.21.15.16:0/376581386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987eb79f0 0x5579882a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.739+0000 7f507c074740 1 -- 172.21.15.16:0/376581386 >> 172.21.15.16:0/376581386 conn(0x557987eed460 msgr2=0x5579882a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.739+0000 7f507c074740 1 -- 172.21.15.16:0/376581386 shutdown_connections 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.739+0000 7f507c074740 1 -- 172.21.15.16:0/376581386 wait complete. 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.739+0000 7f507c074740 1 -- start start 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f507c074740 1 Processor -- start 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f507c074740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987eb79f0 0x5579882ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f507c074740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f507c074740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed8390 0x55798831f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f507c074740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50701b2200 con 0x557987ed01b0 2024-08-28T19:33:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f507c074740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50701b1b00 con 0x557987eb79f0 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f507c074740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f50701aed00 con 0x557987ed8390 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f507c074740 10 client.0 _ll_register_callbacks cb 0x7f506000be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f506effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f506effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882aeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43200/0 (socket says 172.21.15.16:43200) 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f506effd640 1 -- 172.21.15.16:0/2807712675 learned_addr learned my addr 172.21.15.16:0/2807712675 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f506ffff640 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed8390 0x55798831f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f506f7fe640 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987eb79f0 0x5579882ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f506effd640 1 -- 172.21.15.16:0/2807712675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed8390 msgr2=0x55798831f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f506effd640 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed8390 0x55798831f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f506effd640 1 -- 172.21.15.16:0/2807712675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987eb79f0 msgr2=0x5579882ae5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f506effd640 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987eb79f0 0x5579882ae5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f506effd640 1 -- 172.21.15.16:0/2807712675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55798831fda0 con 0x557987ed01b0 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.740+0000 7f506ffff640 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed8390 0x55798831f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f506effd640 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882aeb90 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f50640e2a80 tx=0x7f5064005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9e9390e15a9b5e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f504effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f504effd640 1 -- 172.21.15.16:0/2807712675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5064020040 con 0x557987ed01b0 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f504effd640 1 -- 172.21.15.16:0/2807712675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f5064009cd0 con 0x557987ed01b0 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f507c074740 1 -- 172.21.15.16:0/2807712675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5068036040 con 0x557987ed01b0 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f504effd640 1 -- 172.21.15.16:0/2807712675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f50640ab4d0 con 0x557987ed01b0 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f507c074740 10 client.18669 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f507c074740 1 -- 172.21.15.16:0/2807712675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x557988327140 con 0x557987ed01b0 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f504effd640 1 -- 172.21.15.16:0/2807712675 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5068036040 con 0x557987ed01b0 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f504effd640 1 -- 172.21.15.16:0/2807712675 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7f50640e2700 con 0x557987ed01b0 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f507c074740 10 client.18669 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f507c074740 10 client.18669 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f507c074740 20 client.18669 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f507c074740 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557987ef9070 0x5579883298e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f507c074740 4 client.18669 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:33:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.741+0000 7f507c074740 1 -- 172.21.15.16:0/2807712675 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557988213070 con 0x557987ef9070 2024-08-28T19:33:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.742+0000 7f506f7fe640 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557987ef9070 0x5579883298e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.742+0000 7f504effd640 10 client.18669 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:33:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.742+0000 7f506f7fe640 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557987ef9070 0x5579883298e0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5debce9d50e5b8f7 server_cookie=1003e4d91f3b83ce in_seq=0 out_seq=0 2024-08-28T19:33:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f504effd640 1 -- 172.21.15.16:0/2807712675 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557988213070 con 0x557987ef9070 2024-08-28T19:33:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f504effd640 10 client.18669 handle_command_reply: tid=0 2024-08-28T19:33:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 1 client.18669 shutdown 2024-08-28T19:33:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 1 -- 172.21.15.16:0/2807712675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 msgr2=0x5579882aeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882aeb90 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f50640e2a80 tx=0x7f5064005760 comp rx=0 tx=0).stop 2024-08-28T19:33:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 1 -- 172.21.15.16:0/2807712675 shutdown_connections 2024-08-28T19:33:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557987ef9070 0x5579883298e0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557987ed8390 0x55798831f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557987ed01b0 0x5579882aeb90 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 1 --2- 172.21.15.16:0/2807712675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557987eb79f0 0x5579882ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 1 -- 172.21.15.16:0/2807712675 >> 172.21.15.16:0/2807712675 conn(0x557987eed460 msgr2=0x5579882a82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f504effd640 0 client.18669 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:33:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 1 -- 172.21.15.16:0/2807712675 shutdown_connections 2024-08-28T19:33:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 1 -- 172.21.15.16:0/2807712675 wait complete. 2024-08-28T19:33:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.743+0000 7f507c074740 20 client.18669 trim_cache size 0 max 16384 2024-08-28T19:33:43.749 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:33:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 -- 172.21.15.16:0/763067289 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50580b4ca0 msgr2=0x7f50580b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50580b4ca0 0x7f50580b71b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55798829cf90 tx=0x7f506000b410 comp rx=0 tx=0).stop 2024-08-28T19:33:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 -- 172.21.15.16:0/763067289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5070108160 msgr2=0x7f507019b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5070108160 0x7f507019b5a0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f506400d6a0 tx=0x7f506400db60 comp rx=0 tx=0).stop 2024-08-28T19:33:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 -- 172.21.15.16:0/763067289 shutdown_connections 2024-08-28T19:33:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5070108ab0 0x7f507019bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5070108160 0x7f507019b5a0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f50580b4ca0 0x7f50580b71b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 --2- 172.21.15.16:0/763067289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5070102820 0x7f507019b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 -- 172.21.15.16:0/763067289 >> 172.21.15.16:0/763067289 conn(0x7f50700794e0 msgr2=0x7f50701041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 -- 172.21.15.16:0/763067289 shutdown_connections 2024-08-28T19:33:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:43.744+0000 7f5075aff640 1 -- 172.21.15.16:0/763067289 wait complete. 2024-08-28T19:33:43.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.56 MB/sec execute 563 sec latency 17788.542 ms 2024-08-28T19:33:44.054 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:33:44.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:33:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:44.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.55 MB/sec execute 564 sec latency 18788.605 ms 2024-08-28T19:33:45.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:44 smithi016 ceph-mon[18556]: pgmap v732: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 666 KiB/s wr, 35 op/s 2024-08-28T19:33:45.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:44 smithi115 ceph-mon[22184]: pgmap v732: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 666 KiB/s wr, 35 op/s 2024-08-28T19:33:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.146+0000 7ffaa4a68640 1 -- 172.21.15.16:0/1568053029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa01046a0 msgr2=0x7ffaa0104ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.146+0000 7ffaa4a68640 1 --2- 172.21.15.16:0/1568053029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa01046a0 0x7ffaa0104ab0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7ffa90009900 tx=0x7ffa9002a410 comp rx=0 tx=0).stop 2024-08-28T19:33:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.146+0000 7ffaa4a68640 1 -- 172.21.15.16:0/1568053029 shutdown_connections 2024-08-28T19:33:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.146+0000 7ffaa4a68640 1 --2- 172.21.15.16:0/1568053029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa010dc50 0x7ffaa0110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.146+0000 7ffaa4a68640 1 --2- 172.21.15.16:0/1568053029 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffaa01050d0 0x7ffaa010d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.146+0000 7ffaa4a68640 1 --2- 172.21.15.16:0/1568053029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa01046a0 0x7ffaa0104ab0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.146+0000 7ffaa4a68640 1 -- 172.21.15.16:0/1568053029 >> 172.21.15.16:0/1568053029 conn(0x7ffaa0079510 msgr2=0x7ffaa0079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.146+0000 7ffaa4a68640 1 -- 172.21.15.16:0/1568053029 shutdown_connections 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.146+0000 7ffaa4a68640 1 -- 172.21.15.16:0/1568053029 wait complete. 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.147+0000 7ffaa4a68640 1 Processor -- start 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.147+0000 7ffaa4a68640 1 -- start start 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.147+0000 7ffaa4a68640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa01046a0 0x7ffaa01025b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.147+0000 7ffaa4a68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa01050d0 0x7ffaa0102af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.147+0000 7ffaa4a68640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffaa010dc50 0x7ffaa0103030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.147+0000 7ffaa4a68640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa0113890 con 0x7ffaa01050d0 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.147+0000 7ffaa4a68640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa0113710 con 0x7ffaa01046a0 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.147+0000 7ffaa4a68640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa0113a10 con 0x7ffaa010dc50 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.147+0000 7ffa9dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa01050d0 0x7ffaa0102af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.147+0000 7ffa9dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa01050d0 0x7ffaa0102af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41990/0 (socket says 172.21.15.16:41990) 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa9dd9b640 1 -- 172.21.15.16:0/3891527361 learned_addr learned my addr 172.21.15.16:0/3891527361 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa9e59c640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa01046a0 0x7ffaa01025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa9ed9d640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffaa010dc50 0x7ffaa0103030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa9dd9b640 1 -- 172.21.15.16:0/3891527361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffaa010dc50 msgr2=0x7ffaa0103030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa9dd9b640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffaa010dc50 0x7ffaa0103030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa9dd9b640 1 -- 172.21.15.16:0/3891527361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa01046a0 msgr2=0x7ffaa01025b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa9dd9b640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa01046a0 0x7ffaa01025b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa9dd9b640 1 -- 172.21.15.16:0/3891527361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa01a8520 con 0x7ffaa01050d0 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa9dd9b640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa01050d0 0x7ffaa0102af0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7ffa9400d770 tx=0x7ffa9400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa8f7fe640 1 -- 172.21.15.16:0/3891527361 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa94017070 con 0x7ffaa01050d0 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa8f7fe640 1 -- 172.21.15.16:0/3891527361 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa94009c30 con 0x7ffaa01050d0 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffaa4a68640 1 -- 172.21.15.16:0/3891527361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa01a8810 con 0x7ffaa01050d0 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffa8f7fe640 1 -- 172.21.15.16:0/3891527361 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9400f620 con 0x7ffaa01050d0 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.148+0000 7ffaa4a68640 1 -- 172.21.15.16:0/3891527361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa01a8d50 con 0x7ffaa01050d0 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.149+0000 7ffaa4a68640 1 -- 172.21.15.16:0/3891527361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa006bf60 con 0x7ffaa01050d0 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.151+0000 7ffa8f7fe640 1 -- 172.21.15.16:0/3891527361 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ffa9401a070 con 0x7ffaa01050d0 2024-08-28T19:33:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.152+0000 7ffa8f7fe640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa880b4ca0 0x7ffa880b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:45.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.152+0000 7ffa9e59c640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa880b4ca0 0x7ffa880b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:45.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.152+0000 7ffa8f7fe640 1 -- 172.21.15.16:0/3891527361 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7ffa940e53a0 con 0x7ffaa01050d0 2024-08-28T19:33:45.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.152+0000 7ffa9e59c640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa880b4ca0 0x7ffa880b71b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ffa90000c00 tx=0x7ffa90031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:45.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.155+0000 7ffa8f7fe640 1 -- 172.21.15.16:0/3891527361 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ffa94009de0 con 0x7ffaa01050d0 2024-08-28T19:33:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:45.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:44 smithi028 ceph-mon[21769]: pgmap v732: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 666 KiB/s wr, 35 op/s 2024-08-28T19:33:45.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.402+0000 7ffaa4a68640 1 -- 172.21.15.16:0/3891527361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffaa0104ab0 con 0x7ffaa01050d0 2024-08-28T19:33:45.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.403+0000 7ffa8f7fe640 1 -- 172.21.15.16:0/3891527361 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+11984 (secure 0 0 0) 0x7ffaa0104ab0 con 0x7ffaa01050d0 2024-08-28T19:33:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:33:45.405 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":63,"btime":"2024-08-28T19:33:41:813788+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":63,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:33:41.598113+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:33:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 63 2024-08-28T19:33:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 -- 172.21.15.16:0/3891527361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa880b4ca0 msgr2=0x7ffa880b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa880b4ca0 0x7ffa880b71b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ffa90000c00 tx=0x7ffa90031400 comp rx=0 tx=0).stop 2024-08-28T19:33:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 -- 172.21.15.16:0/3891527361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa01050d0 msgr2=0x7ffaa0102af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa01050d0 0x7ffaa0102af0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7ffa9400d770 tx=0x7ffa9400dc30 comp rx=0 tx=0).stop 2024-08-28T19:33:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 -- 172.21.15.16:0/3891527361 shutdown_connections 2024-08-28T19:33:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffaa010dc50 0x7ffaa0103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa01050d0 0x7ffaa0102af0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffa880b4ca0 0x7ffa880b71b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 --2- 172.21.15.16:0/3891527361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa01046a0 0x7ffaa01025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 -- 172.21.15.16:0/3891527361 >> 172.21.15.16:0/3891527361 conn(0x7ffaa0079510 msgr2=0x7ffaa01077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 -- 172.21.15.16:0/3891527361 shutdown_connections 2024-08-28T19:33:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:45.405+0000 7ffaa4a68640 1 -- 172.21.15.16:0/3891527361 wait complete. 2024-08-28T19:33:45.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.54 MB/sec execute 565 sec latency 19788.697 ms 2024-08-28T19:33:46.050 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:45 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3891527361' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:33:46.082 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id b52084a --release --await' 2024-08-28T19:33:46.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id b52084a --release --await 2024-08-28T19:33:46.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:45 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3891527361' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:33:46.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.302+0000 7fc7e727c640 1 -- 172.21.15.16:0/3063619175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e0108ac0 msgr2=0x7fc7e010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.302+0000 7fc7e727c640 1 --2- 172.21.15.16:0/3063619175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e0108ac0 0x7fc7e010f2f0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8009ca0 tx=0x7fc7d802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.302+0000 7fc7e727c640 1 -- 172.21.15.16:0/3063619175 shutdown_connections 2024-08-28T19:33:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.302+0000 7fc7e727c640 1 --2- 172.21.15.16:0/3063619175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e0108ac0 0x7fc7e010f2f0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.302+0000 7fc7e727c640 1 --2- 172.21.15.16:0/3063619175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7e0108170 0x7fc7e0108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.302+0000 7fc7e727c640 1 --2- 172.21.15.16:0/3063619175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e0102880 0x7fc7e0102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.302+0000 7fc7e727c640 1 -- 172.21.15.16:0/3063619175 >> 172.21.15.16:0/3063619175 conn(0x7fc7e0079530 msgr2=0x7fc7e0079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.302+0000 7fc7e727c640 1 -- 172.21.15.16:0/3063619175 shutdown_connections 2024-08-28T19:33:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.302+0000 7fc7e727c640 1 -- 172.21.15.16:0/3063619175 wait complete. 2024-08-28T19:33:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e727c640 1 Processor -- start 2024-08-28T19:33:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e727c640 1 -- start start 2024-08-28T19:33:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e727c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e0102880 0x7fc7e0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e727c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7e0108170 0x7fc7e0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e727c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e0108ac0 0x7fc7e019d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e727c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e0111680 con 0x7fc7e0108ac0 2024-08-28T19:33:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e727c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e0111500 con 0x7fc7e0108170 2024-08-28T19:33:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e727c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e0111800 con 0x7fc7e0102880 2024-08-28T19:33:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e5819640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e0108ac0 0x7fc7e019d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e5018640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e0102880 0x7fc7e0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e5018640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e0102880 0x7fc7e0197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:39510/0 (socket says 172.21.15.16:39510) 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e5819640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e0108ac0 0x7fc7e019d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42006/0 (socket says 172.21.15.16:42006) 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.303+0000 7fc7e5018640 1 -- 172.21.15.16:0/465209497 learned_addr learned my addr 172.21.15.16:0/465209497 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7e4817640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7e0108170 0x7fc7e0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7e5819640 1 -- 172.21.15.16:0/465209497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e0102880 msgr2=0x7fc7e0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7e5819640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e0102880 0x7fc7e0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7e5819640 1 -- 172.21.15.16:0/465209497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7e0108170 msgr2=0x7fc7e0198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7e5819640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7e0108170 0x7fc7e0198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7e5819640 1 -- 172.21.15.16:0/465209497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e019df60 con 0x7fc7e0108ac0 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7e5018640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e0102880 0x7fc7e0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7e5819640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e0108ac0 0x7fc7e019d830 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fc7d802ade0 tx=0x7fc7d80319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7d67fc640 1 -- 172.21.15.16:0/465209497 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8003bf0 con 0x7fc7e0108ac0 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7d67fc640 1 -- 172.21.15.16:0/465209497 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7d8003d90 con 0x7fc7e0108ac0 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7e727c640 1 -- 172.21.15.16:0/465209497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e019e1f0 con 0x7fc7e0108ac0 2024-08-28T19:33:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7d67fc640 1 -- 172.21.15.16:0/465209497 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d802fbd0 con 0x7fc7e0108ac0 2024-08-28T19:33:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.304+0000 7fc7e727c640 1 -- 172.21.15.16:0/465209497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e01a49b0 con 0x7fc7e0108ac0 2024-08-28T19:33:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.307+0000 7fc7ed7f1740 1 Processor -- start 2024-08-28T19:33:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.307+0000 7fc7d67fc640 1 -- 172.21.15.16:0/465209497 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc7d802fd70 con 0x7fc7e0108ac0 2024-08-28T19:33:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.307+0000 7fc7ed7f1740 1 -- start start 2024-08-28T19:33:46.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:45 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3891527361' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:33:46.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207ea070 0x556b20b8f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c11b0 0x556b20b8f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20b8fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556b20b06680 con 0x556b207c9390 2024-08-28T19:33:46.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b20af5c10 con 0x556b207c11b0 2024-08-28T19:33:46.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556b20af9bc0 con 0x556b207ea070 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e4817640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20b8fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e5018640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c11b0 0x556b20b8f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e4817640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20b8fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42014/0 (socket says 172.21.15.16:42014) 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e4817640 1 -- 172.21.15.16:0/53065980 learned_addr learned my addr 172.21.15.16:0/53065980 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e4817640 1 --2- 172.21.15.16:0/53065980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207ea070 0x556b20b8f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7d67fc640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7cc0b4ca0 0x7fc7cc0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e4817640 1 -- 172.21.15.16:0/53065980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207ea070 msgr2=0x556b20b8f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e4817640 1 --2- 172.21.15.16:0/53065980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207ea070 0x556b20b8f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e4817640 1 -- 172.21.15.16:0/53065980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c11b0 msgr2=0x556b20b8f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e4817640 1 --2- 172.21.15.16:0/53065980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c11b0 0x556b20b8f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e4817640 1 -- 172.21.15.16:0/53065980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b20855650 con 0x556b207c9390 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e5018640 1 --2- 172.21.15.16:0/53065980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c11b0 0x556b20b8f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7e5819640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7cc0b4ca0 0x7fc7cc0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.308+0000 7fc7d67fc640 1 -- 172.21.15.16:0/465209497 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fc7d8106ba0 con 0x7fc7e0108ac0 2024-08-28T19:33:46.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.309+0000 7fc7e4817640 1 --2- 172.21.15.16:0/53065980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20b8fdd0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000a670 tx=0x7fc7d0005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5033ced7b1fec04 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:46.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.309+0000 7fc7cb7fe640 1 -- 172.21.15.16:0/53065980 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0004030 con 0x556b207c9390 2024-08-28T19:33:46.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.309+0000 7fc7cb7fe640 1 -- 172.21.15.16:0/53065980 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7d00035f0 con 0x556b207c9390 2024-08-28T19:33:46.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.309+0000 7fc7cb7fe640 1 -- 172.21.15.16:0/53065980 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d00197c0 con 0x556b207c9390 2024-08-28T19:33:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/53065980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 msgr2=0x556b20b8fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/53065980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20b8fdd0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000a670 tx=0x7fc7d0005c90 comp rx=0 tx=0).stop 2024-08-28T19:33:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7e5819640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7cc0b4ca0 0x7fc7cc0b71b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8103fd0 tx=0x7fc7d80044f0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/53065980 shutdown_connections 2024-08-28T19:33:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/53065980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20b8fdd0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/53065980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c11b0 0x556b20b8f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/53065980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207ea070 0x556b20b8f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/53065980 >> 172.21.15.16:0/53065980 conn(0x556b207de460 msgr2=0x556b20b8d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/53065980 shutdown_connections 2024-08-28T19:33:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/53065980 wait complete. 2024-08-28T19:33:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7ed7f1740 1 -- start start 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.310+0000 7fc7ed7f1740 1 Processor -- start 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c11b0 0x556b20b9aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c9390 0x556b20b9afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207ea070 0x556b20ba6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556b20958700 con 0x556b207c11b0 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b20a7c750 con 0x556b207c9390 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556b20b9db30 con 0x556b207ea070 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7ed7f1740 10 client.0 _ll_register_callbacks cb 0x556b20aabe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7e4817640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c11b0 0x556b20b9aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7e4817640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c11b0 0x556b20b9aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42028/0 (socket says 172.21.15.16:42028) 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7e4817640 1 -- 172.21.15.16:0/247062183 learned_addr learned my addr 172.21.15.16:0/247062183 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7e5018640 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c9390 0x556b20b9afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7e4817640 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207ea070 0x556b20ba6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7e4817640 1 -- 172.21.15.16:0/247062183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207ea070 msgr2=0x556b20ba6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7e4817640 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207ea070 0x556b20ba6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7e4817640 1 -- 172.21.15.16:0/247062183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c9390 msgr2=0x556b20b9afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7e4817640 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c9390 0x556b20b9afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.311+0000 7fc7e4817640 1 -- 172.21.15.16:0/247062183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b20ab5900 con 0x556b207c11b0 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.312+0000 7fc7e4817640 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c11b0 0x556b20b9aa70 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0005bc0 tx=0x7fc7d0001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb8aa8522a71d5f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.312+0000 7fc7c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.312+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/247062183 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0024040 con 0x556b207c11b0 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.312+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/247062183 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fc7d00042c0 con 0x556b207c11b0 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.312+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/247062183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7d00035f0 con 0x556b207c11b0 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.312+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/247062183 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fc7d0032050 con 0x556b207c11b0 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.312+0000 7fc7ed7f1740 10 client.18687 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.312+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/247062183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x556b20ba9900 con 0x556b207c11b0 2024-08-28T19:33:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.312+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/247062183 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7d00035f0 con 0x556b207c11b0 2024-08-28T19:33:46.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.313+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/247062183 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7fc7d001a040 con 0x556b207c11b0 2024-08-28T19:33:46.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.313+0000 7fc7ed7f1740 10 client.18687 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:46.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.313+0000 7fc7ed7f1740 10 client.18687 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:33:46.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.313+0000 7fc7ed7f1740 20 client.18687 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:46.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.313+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556b207a89f0 0x556b20bacc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.313+0000 7fc7ed7f1740 4 client.18687 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.313+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/247062183 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b20b085d0 con 0x556b207a89f0 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.314+0000 7fc7e5018640 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556b207a89f0 0x556b20bacc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.314+0000 7fc7e5018640 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556b207a89f0 0x556b20bacc60 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2793d8d418b1d52d server_cookie=f9dd90901b098db5 in_seq=0 out_seq=0 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.314+0000 7fc7c8ff9640 10 client.18687 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.315+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/247062183 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556b20b085d0 con 0x556b207a89f0 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.315+0000 7fc7c8ff9640 10 client.18687 handle_command_reply: tid=0 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.315+0000 7fc7ed7f1740 1 client.18687 shutdown 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.315+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/247062183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c11b0 msgr2=0x556b20b9aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.315+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c11b0 0x556b20b9aa70 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0005bc0 tx=0x7fc7d0001e50 comp rx=0 tx=0).stop 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.316+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/247062183 shutdown_connections 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.316+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556b207a89f0 0x556b20bacc60 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.316+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207ea070 0x556b20ba6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.316+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c9390 0x556b20b9afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.316+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/247062183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c11b0 0x556b20b9aa70 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.316+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/247062183 >> 172.21.15.16:0/247062183 conn(0x556b207de460 msgr2=0x556b20b924e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:46.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.316+0000 7fc7c8ff9640 0 client.18687 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:33:46.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.316+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/247062183 shutdown_connections 2024-08-28T19:33:46.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.316+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/247062183 wait complete. 2024-08-28T19:33:46.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.316+0000 7fc7ed7f1740 20 client.18687 trim_cache size 0 max 16384 2024-08-28T19:33:46.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.329+0000 7fc7ed7f1740 1 Processor -- start 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7ed7f1740 1 -- start start 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207c11b0 0x556b20b8d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20ba6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207ea070 0x556b20b97c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01aea30 con 0x556b207c9390 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01b0630 con 0x556b207ea070 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01af130 con 0x556b207c11b0 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7e5018640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20ba6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7e5018640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20ba6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42038/0 (socket says 172.21.15.16:42038) 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7e5018640 1 -- 172.21.15.16:0/2959785282 learned_addr learned my addr 172.21.15.16:0/2959785282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7e4817640 1 --2- 172.21.15.16:0/2959785282 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207c11b0 0x556b20b8d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.330+0000 7fc7e4817640 1 --2- 172.21.15.16:0/2959785282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207ea070 0x556b20b97c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.331+0000 7fc7e5018640 1 -- 172.21.15.16:0/2959785282 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207c11b0 msgr2=0x556b20b8d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:46.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.331+0000 7fc7e5018640 1 --2- 172.21.15.16:0/2959785282 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207c11b0 0x556b20b8d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.331+0000 7fc7e5018640 1 -- 172.21.15.16:0/2959785282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207ea070 msgr2=0x556b20b97c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:46.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.331+0000 7fc7e5018640 1 --2- 172.21.15.16:0/2959785282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207ea070 0x556b20b97c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.331+0000 7fc7e5018640 1 -- 172.21.15.16:0/2959785282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b20b9bd30 con 0x556b207c9390 2024-08-28T19:33:46.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.331+0000 7fc7e5018640 1 --2- 172.21.15.16:0/2959785282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20ba6d80 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc014c10 tx=0x7fc7dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbcf3e62d6d8860c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:46.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.331+0000 7fc7caffd640 1 -- 172.21.15.16:0/2959785282 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc014dc0 con 0x556b207c9390 2024-08-28T19:33:46.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.331+0000 7fc7caffd640 1 -- 172.21.15.16:0/2959785282 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7dc01be20 con 0x556b207c9390 2024-08-28T19:33:46.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.331+0000 7fc7caffd640 1 -- 172.21.15.16:0/2959785282 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc004a00 con 0x556b207c9390 2024-08-28T19:33:46.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.332+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2959785282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 msgr2=0x556b20ba6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:46.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.332+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/2959785282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20ba6d80 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc014c10 tx=0x7fc7dc004730 comp rx=0 tx=0).stop 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.332+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2959785282 shutdown_connections 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.332+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/2959785282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207ea070 0x556b20b97c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.332+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/2959785282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207c9390 0x556b20ba6d80 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.332+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/2959785282 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207c11b0 0x556b20b8d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.332+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2959785282 >> 172.21.15.16:0/2959785282 conn(0x556b207de460 msgr2=0x556b20b8cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.332+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2959785282 shutdown_connections 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.332+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2959785282 wait complete. 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.332+0000 7fc7ed7f1740 1 -- start start 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7ed7f1740 1 Processor -- start 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207c11b0 0x556b20ba3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c9390 0x556b20b9e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7ed7f1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207ea070 0x556b20b9eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01aff30 con 0x556b207ea070 2024-08-28T19:33:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01af830 con 0x556b207c9390 2024-08-28T19:33:46.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7ed7f1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01aca30 con 0x556b207c11b0 2024-08-28T19:33:46.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7ed7f1740 10 client.0 _ll_register_callbacks cb 0x7fc7dc00beb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:46.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7e4817640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207ea070 0x556b20b9eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7e4817640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207ea070 0x556b20b9eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42048/0 (socket says 172.21.15.16:42048) 2024-08-28T19:33:46.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7e4817640 1 -- 172.21.15.16:0/2410586124 learned_addr learned my addr 172.21.15.16:0/2410586124 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:46.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7e5018640 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c9390 0x556b20b9e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.333+0000 7fc7e4817640 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207c11b0 0x556b20ba3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7e4817640 1 -- 172.21.15.16:0/2410586124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207c11b0 msgr2=0x556b20ba3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7e4817640 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207c11b0 0x556b20ba3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7e4817640 1 -- 172.21.15.16:0/2410586124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c9390 msgr2=0x556b20b9e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7e4817640 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c9390 0x556b20b9e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7e4817640 1 -- 172.21.15.16:0/2410586124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b20b9f540 con 0x556b207ea070 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7e5018640 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c9390 0x556b20b9e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7e4817640 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207ea070 0x556b20b9eed0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000a640 tx=0x7fc7d0003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96cf17e8911b48f1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/2410586124 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0005130 con 0x556b207ea070 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/2410586124 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fc7d00052d0 con 0x556b207ea070 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2410586124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7dc01be20 con 0x556b207ea070 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/2410586124 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fc7d00054d0 con 0x556b207ea070 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7ed7f1740 10 client.18693 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.334+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2410586124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x556b20c17200 con 0x556b207ea070 2024-08-28T19:33:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.335+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/2410586124 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7d001cdc0 con 0x556b207ea070 2024-08-28T19:33:46.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.335+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/2410586124 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7fc7d0019940 con 0x556b207ea070 2024-08-28T19:33:46.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.335+0000 7fc7ed7f1740 10 client.18693 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:46.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.335+0000 7fc7ed7f1740 10 client.18693 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:33:46.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.335+0000 7fc7ed7f1740 20 client.18693 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:46.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.335+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556b2077dda0 0x556b20c19910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:46.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.335+0000 7fc7ed7f1740 4 client.18693 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:33:46.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.335+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2410586124 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc7dc01be20 con 0x556b2077dda0 2024-08-28T19:33:46.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.335+0000 7fc7e5018640 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556b2077dda0 0x556b20c19910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:46.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.336+0000 7fc7c8ff9640 10 client.18693 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:33:46.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.336+0000 7fc7e5018640 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556b2077dda0 0x556b20c19910 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b9261b52051e5f7 server_cookie=1fb0fc58ba4e346 in_seq=0 out_seq=0 2024-08-28T19:33:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.336+0000 7fc7c8ff9640 1 -- 172.21.15.16:0/2410586124 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fc7dc01be20 con 0x556b2077dda0 2024-08-28T19:33:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.336+0000 7fc7c8ff9640 10 client.18693 handle_command_reply: tid=0 2024-08-28T19:33:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.336+0000 7fc7ed7f1740 1 client.18693 shutdown 2024-08-28T19:33:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2410586124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207ea070 msgr2=0x556b20b9eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207ea070 0x556b20b9eed0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000a640 tx=0x7fc7d0003dd0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2410586124 shutdown_connections 2024-08-28T19:33:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556b2077dda0 0x556b20c19910 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b207ea070 0x556b20b9eed0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b207c9390 0x556b20b9e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7ed7f1740 1 --2- 172.21.15.16:0/2410586124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b207c11b0 0x556b20ba3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2410586124 >> 172.21.15.16:0/2410586124 conn(0x556b207de460 msgr2=0x556b20b90280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7c8ff9640 0 client.18693 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:33:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2410586124 shutdown_connections 2024-08-28T19:33:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7ed7f1740 1 -- 172.21.15.16:0/2410586124 wait complete. 2024-08-28T19:33:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.337+0000 7fc7ed7f1740 20 client.18693 trim_cache size 0 max 16384 2024-08-28T19:33:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:33:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 -- 172.21.15.16:0/465209497 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7cc0b4ca0 msgr2=0x7fc7cc0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7cc0b4ca0 0x7fc7cc0b71b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8103fd0 tx=0x7fc7d80044f0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 -- 172.21.15.16:0/465209497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e0108ac0 msgr2=0x7fc7e019d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e0108ac0 0x7fc7e019d830 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fc7d802ade0 tx=0x7fc7d80319f0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 -- 172.21.15.16:0/465209497 shutdown_connections 2024-08-28T19:33:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e0108ac0 0x7fc7e019d830 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc7cc0b4ca0 0x7fc7cc0b71b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7e0108170 0x7fc7e0198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 --2- 172.21.15.16:0/465209497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e0102880 0x7fc7e0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 -- 172.21.15.16:0/465209497 >> 172.21.15.16:0/465209497 conn(0x7fc7e0079530 msgr2=0x7fc7e0104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:46.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 -- 172.21.15.16:0/465209497 shutdown_connections 2024-08-28T19:33:46.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:46.338+0000 7fc7e727c640 1 -- 172.21.15.16:0/465209497 wait complete. 2024-08-28T19:33:46.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502636 3.54 MB/sec execute 566 sec latency 20788.745 ms 2024-08-28T19:33:47.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:46 smithi016 ceph-mon[18556]: pgmap v733: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 645 KiB/s wr, 42 op/s 2024-08-28T19:33:47.072 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:47.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:33:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:46 smithi115 ceph-mon[22184]: pgmap v733: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 645 KiB/s wr, 42 op/s 2024-08-28T19:33:47.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:46 smithi028 ceph-mon[21769]: pgmap v733: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 645 KiB/s wr, 42 op/s 2024-08-28T19:33:47.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.559+0000 7fc406eb5640 1 -- 172.21.15.16:0/1822124450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc400109bb0 msgr2=0x7fc4001103e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:47.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.559+0000 7fc406eb5640 1 --2- 172.21.15.16:0/1822124450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc400109bb0 0x7fc4001103e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009ca0 tx=0x7fc3f802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.559+0000 7fc406eb5640 1 -- 172.21.15.16:0/1822124450 shutdown_connections 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.559+0000 7fc406eb5640 1 --2- 172.21.15.16:0/1822124450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc400109bb0 0x7fc4001103e0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.559+0000 7fc406eb5640 1 --2- 172.21.15.16:0/1822124450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc400109260 0x7fc400109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.559+0000 7fc406eb5640 1 --2- 172.21.15.16:0/1822124450 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc400103920 0x7fc400103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.559+0000 7fc406eb5640 1 -- 172.21.15.16:0/1822124450 >> 172.21.15.16:0/1822124450 conn(0x7fc4000ff3b0 msgr2=0x7fc400101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.559+0000 7fc406eb5640 1 -- 172.21.15.16:0/1822124450 shutdown_connections 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.559+0000 7fc406eb5640 1 -- 172.21.15.16:0/1822124450 wait complete. 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.559+0000 7fc406eb5640 1 Processor -- start 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc406eb5640 1 -- start start 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc406eb5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc400103920 0x7fc4000754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc406eb5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc400109260 0x7fc4000759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc406eb5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc400109bb0 0x7fc400075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc406eb5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc400113870 con 0x7fc400109260 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc406eb5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4001136f0 con 0x7fc400109bb0 2024-08-28T19:33:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc406eb5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc4001139f0 con 0x7fc400103920 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc404c51640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc400103920 0x7fc4000754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc404c51640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc400103920 0x7fc4000754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:39562/0 (socket says 172.21.15.16:39562) 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc404c51640 1 -- 172.21.15.16:0/3362195943 learned_addr learned my addr 172.21.15.16:0/3362195943 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc3fffff640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc400109260 0x7fc4000759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.560+0000 7fc405452640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc400109bb0 0x7fc400075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.561+0000 7fc404c51640 1 -- 172.21.15.16:0/3362195943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc400109bb0 msgr2=0x7fc400075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.561+0000 7fc404c51640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc400109bb0 0x7fc400075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.561+0000 7fc404c51640 1 -- 172.21.15.16:0/3362195943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc400109260 msgr2=0x7fc4000759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.561+0000 7fc404c51640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc400109260 0x7fc4000759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.561+0000 7fc404c51640 1 -- 172.21.15.16:0/3362195943 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4001a8480 con 0x7fc400103920 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.561+0000 7fc404c51640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc400103920 0x7fc4000754b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400d950 tx=0x7fc3f400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.561+0000 7fc3fdffb640 1 -- 172.21.15.16:0/3362195943 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4017070 con 0x7fc400103920 2024-08-28T19:33:47.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.561+0000 7fc3fdffb640 1 -- 172.21.15.16:0/3362195943 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3f4009c30 con 0x7fc400103920 2024-08-28T19:33:47.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.561+0000 7fc406eb5640 1 -- 172.21.15.16:0/3362195943 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4001a8770 con 0x7fc400103920 2024-08-28T19:33:47.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.562+0000 7fc3fdffb640 1 -- 172.21.15.16:0/3362195943 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f400f620 con 0x7fc400103920 2024-08-28T19:33:47.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.562+0000 7fc406eb5640 1 -- 172.21.15.16:0/3362195943 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4001a8cb0 con 0x7fc400103920 2024-08-28T19:33:47.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.564+0000 7fc3fdffb640 1 -- 172.21.15.16:0/3362195943 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc3f4010460 con 0x7fc400103920 2024-08-28T19:33:47.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.565+0000 7fc40d42a740 1 Processor -- start 2024-08-28T19:33:47.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.565+0000 7fc40d42a740 1 -- start start 2024-08-28T19:33:47.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.565+0000 7fc3fdffb640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3f00b4ca0 0x7fc3f00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.565+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a09c2520 0x5577a0d67c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.565+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 0x5577a0d681c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.565+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d68700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.565+0000 7fc40d42a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5577a0cdea70 con 0x5577a09a1840 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.565+0000 7fc40d42a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5577a0cce000 con 0x5577a09c2520 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.565+0000 7fc40d42a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5577a0cd1fb0 con 0x5577a0999660 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.565+0000 7fc405452640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3f00b4ca0 0x7fc3f00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.565+0000 7fc3fdffb640 1 -- 172.21.15.16:0/3362195943 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fc3f40e55b0 con 0x7fc400103920 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc405452640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d68700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc405452640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d68700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42068/0 (socket says 172.21.15.16:42068) 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc405452640 1 -- 172.21.15.16:0/2077076293 learned_addr learned my addr 172.21.15.16:0/2077076293 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc3fffff640 1 --2- 172.21.15.16:0/2077076293 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 0x5577a0d681c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc405452640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3f00b4ca0 0x7fc3f00b71b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8035000 tx=0x7fc3f8031bd0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc405452640 1 -- 172.21.15.16:0/2077076293 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 msgr2=0x5577a0d681c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc405452640 1 --2- 172.21.15.16:0/2077076293 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 0x5577a0d681c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc405452640 1 -- 172.21.15.16:0/2077076293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a09c2520 msgr2=0x5577a0d67c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:47.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc405452640 1 --2- 172.21.15.16:0/2077076293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a09c2520 0x5577a0d67c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc405452640 1 -- 172.21.15.16:0/2077076293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577a0a2db00 con 0x5577a09a1840 2024-08-28T19:33:47.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc3fffff640 1 --2- 172.21.15.16:0/2077076293 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 0x5577a0d681c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:33:47.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.566+0000 7fc405452640 1 --2- 172.21.15.16:0/2077076293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d68700 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8031c40 tx=0x7fc3f804a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71fcea61df0da37f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:47.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.567+0000 7fc3eaffd640 1 -- 172.21.15.16:0/2077076293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8003d70 con 0x5577a09a1840 2024-08-28T19:33:47.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.567+0000 7fc3eaffd640 1 -- 172.21.15.16:0/2077076293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3f804f020 con 0x5577a09a1840 2024-08-28T19:33:47.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.567+0000 7fc3eaffd640 1 -- 172.21.15.16:0/2077076293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f802f8c0 con 0x5577a09a1840 2024-08-28T19:33:47.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.568+0000 7fc40d42a740 1 -- 172.21.15.16:0/2077076293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 msgr2=0x5577a0d68700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:47.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.568+0000 7fc40d42a740 1 --2- 172.21.15.16:0/2077076293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d68700 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8031c40 tx=0x7fc3f804a000 comp rx=0 tx=0).stop 2024-08-28T19:33:47.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.568+0000 7fc40d42a740 1 -- 172.21.15.16:0/2077076293 shutdown_connections 2024-08-28T19:33:47.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.568+0000 7fc40d42a740 1 --2- 172.21.15.16:0/2077076293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d68700 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.568+0000 7fc40d42a740 1 --2- 172.21.15.16:0/2077076293 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 0x5577a0d681c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.568+0000 7fc40d42a740 1 --2- 172.21.15.16:0/2077076293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a09c2520 0x5577a0d67c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.568+0000 7fc40d42a740 1 -- 172.21.15.16:0/2077076293 >> 172.21.15.16:0/2077076293 conn(0x5577a09b6910 msgr2=0x5577a0d65b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.568+0000 7fc40d42a740 1 -- 172.21.15.16:0/2077076293 shutdown_connections 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.568+0000 7fc40d42a740 1 -- 172.21.15.16:0/2077076293 wait complete. 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.568+0000 7fc40d42a740 1 -- start start 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.568+0000 7fc40d42a740 1 Processor -- start 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a0999660 0x5577a0d73810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d73d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09c2520 0x5577a0d7f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc40d42a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5577a0d76280 con 0x5577a09a1840 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc40d42a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5577a0c8f520 con 0x5577a0999660 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc40d42a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5577a0d76400 con 0x5577a09c2520 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc3fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d73d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc3fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d73d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42084/0 (socket says 172.21.15.16:42084) 2024-08-28T19:33:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc3fffff640 1 -- 172.21.15.16:0/2329417592 learned_addr learned my addr 172.21.15.16:0/2329417592 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc40d42a740 10 client.0 _ll_register_callbacks cb 0x5577a0cc7c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc404c51640 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a0999660 0x5577a0d73810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc405452640 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09c2520 0x5577a0d7f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc3fffff640 1 -- 172.21.15.16:0/2329417592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09c2520 msgr2=0x5577a0d7f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc3fffff640 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09c2520 0x5577a0d7f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc3fffff640 1 -- 172.21.15.16:0/2329417592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a0999660 msgr2=0x5577a0d73810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc3fffff640 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a0999660 0x5577a0d73810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc3fffff640 1 -- 172.21.15.16:0/2329417592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577a0aa4960 con 0x5577a09a1840 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc405452640 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09c2520 0x5577a0d7f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.569+0000 7fc404c51640 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a0999660 0x5577a0d73810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.570+0000 7fc3fffff640 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d73d50 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec010470 tx=0x7fc3ec010870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ecebad8315f2d56 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.570+0000 7fc3dffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.570+0000 7fc3dffff640 1 -- 172.21.15.16:0/2329417592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec004440 con 0x5577a09a1840 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.570+0000 7fc3dffff640 1 -- 172.21.15.16:0/2329417592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fc3ec0035f0 con 0x5577a09a1840 2024-08-28T19:33:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.570+0000 7fc40d42a740 1 -- 172.21.15.16:0/2329417592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5577a0c6d9a0 con 0x5577a09a1840 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.570+0000 7fc3dffff640 1 -- 172.21.15.16:0/2329417592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fc3ec023050 con 0x5577a09a1840 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.570+0000 7fc40d42a740 10 client.18705 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.570+0000 7fc40d42a740 1 -- 172.21.15.16:0/2329417592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x5577a0d82400 con 0x5577a09a1840 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.570+0000 7fc3dffff640 1 -- 172.21.15.16:0/2329417592 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5577a0c6d9a0 con 0x5577a09a1840 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.571+0000 7fc3dffff640 1 -- 172.21.15.16:0/2329417592 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7fc3ec00f040 con 0x5577a09a1840 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.571+0000 7fc40d42a740 10 client.18705 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.571+0000 7fc40d42a740 10 client.18705 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.571+0000 7fc40d42a740 20 client.18705 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.571+0000 7fc40d42a740 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577a0980ea0 0x5577a0d85920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.571+0000 7fc40d42a740 4 client.18705 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.571+0000 7fc40d42a740 1 -- 172.21.15.16:0/2329417592 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577a0ce09c0 con 0x5577a0980ea0 2024-08-28T19:33:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.571+0000 7fc404c51640 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577a0980ea0 0x5577a0d85920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.571+0000 7fc404c51640 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577a0980ea0 0x5577a0d85920 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f7dead1298f876d server_cookie=ed561d78abfcf3ea in_seq=0 out_seq=0 2024-08-28T19:33:47.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.571+0000 7fc3dffff640 10 client.18705 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:47.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.572+0000 7fc3dffff640 1 -- 172.21.15.16:0/2329417592 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5577a0ce09c0 con 0x5577a0980ea0 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.572+0000 7fc3dffff640 10 client.18705 handle_command_reply: tid=0 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.572+0000 7fc40d42a740 1 client.18705 shutdown 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.572+0000 7fc40d42a740 1 -- 172.21.15.16:0/2329417592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 msgr2=0x5577a0d73d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.572+0000 7fc40d42a740 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d73d50 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec010470 tx=0x7fc3ec010870 comp rx=0 tx=0).stop 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.572+0000 7fc40d42a740 1 -- 172.21.15.16:0/2329417592 shutdown_connections 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.572+0000 7fc40d42a740 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577a0980ea0 0x5577a0d85920 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.573+0000 7fc40d42a740 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09c2520 0x5577a0d7f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.573+0000 7fc3dffff640 0 client.18705 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.573+0000 7fc40d42a740 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a09a1840 0x5577a0d73d50 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.573+0000 7fc40d42a740 1 --2- 172.21.15.16:0/2329417592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a0999660 0x5577a0d73810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.573+0000 7fc40d42a740 1 -- 172.21.15.16:0/2329417592 >> 172.21.15.16:0/2329417592 conn(0x5577a09b6910 msgr2=0x5577a0d6ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.573+0000 7fc40d42a740 1 -- 172.21.15.16:0/2329417592 shutdown_connections 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.573+0000 7fc40d42a740 1 -- 172.21.15.16:0/2329417592 wait complete. 2024-08-28T19:33:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.573+0000 7fc40d42a740 20 client.18705 trim_cache size 0 max 16384 2024-08-28T19:33:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:47.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc40d42a740 1 Processor -- start 2024-08-28T19:33:47.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc40d42a740 1 -- start start 2024-08-28T19:33:47.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d654b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a0999660 0x5577a0d7f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09a1840 0x5577a0d7fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc40d42a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4001b2e10 con 0x5577a0980ea0 2024-08-28T19:33:47.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc40d42a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4001b4a10 con 0x5577a0999660 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc40d42a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc4001b3510 con 0x5577a09a1840 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc404c51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d654b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc404c51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d654b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42088/0 (socket says 172.21.15.16:42088) 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc404c51640 1 -- 172.21.15.16:0/1328707867 learned_addr learned my addr 172.21.15.16:0/1328707867 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.597+0000 7fc3fffff640 1 --2- 172.21.15.16:0/1328707867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a0999660 0x5577a0d7f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.598+0000 7fc405452640 1 --2- 172.21.15.16:0/1328707867 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09a1840 0x5577a0d7fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.598+0000 7fc404c51640 1 -- 172.21.15.16:0/1328707867 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09a1840 msgr2=0x5577a0d7fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.598+0000 7fc404c51640 1 --2- 172.21.15.16:0/1328707867 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09a1840 0x5577a0d7fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.598+0000 7fc404c51640 1 -- 172.21.15.16:0/1328707867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a0999660 msgr2=0x5577a0d7f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.598+0000 7fc404c51640 1 --2- 172.21.15.16:0/1328707867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a0999660 0x5577a0d7f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.598+0000 7fc404c51640 1 -- 172.21.15.16:0/1328707867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577a0d73ef0 con 0x5577a0980ea0 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.598+0000 7fc405452640 1 --2- 172.21.15.16:0/1328707867 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09a1840 0x5577a0d7fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:47.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.598+0000 7fc404c51640 1 --2- 172.21.15.16:0/1328707867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d654b0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40a8b40 tx=0x7fc3f40e57b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c647f3162233e23c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:47.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.598+0000 7fc3ea7fc640 1 -- 172.21.15.16:0/1328707867 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4020050 con 0x5577a0980ea0 2024-08-28T19:33:47.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.598+0000 7fc3ea7fc640 1 -- 172.21.15.16:0/1328707867 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3f4009de0 con 0x5577a0980ea0 2024-08-28T19:33:47.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.598+0000 7fc3ea7fc640 1 -- 172.21.15.16:0/1328707867 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f40a7b60 con 0x5577a0980ea0 2024-08-28T19:33:47.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.599+0000 7fc40d42a740 1 -- 172.21.15.16:0/1328707867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 msgr2=0x5577a0d654b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:47.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.599+0000 7fc40d42a740 1 --2- 172.21.15.16:0/1328707867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d654b0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40a8b40 tx=0x7fc3f40e57b0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.599+0000 7fc40d42a740 1 -- 172.21.15.16:0/1328707867 shutdown_connections 2024-08-28T19:33:47.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.599+0000 7fc40d42a740 1 --2- 172.21.15.16:0/1328707867 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a09a1840 0x5577a0d7fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.599+0000 7fc40d42a740 1 --2- 172.21.15.16:0/1328707867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a0999660 0x5577a0d7f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.599+0000 7fc40d42a740 1 --2- 172.21.15.16:0/1328707867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d654b0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.599+0000 7fc40d42a740 1 -- 172.21.15.16:0/1328707867 >> 172.21.15.16:0/1328707867 conn(0x5577a09b6910 msgr2=0x5577a0d6e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:47.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.599+0000 7fc40d42a740 1 -- 172.21.15.16:0/1328707867 shutdown_connections 2024-08-28T19:33:47.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.599+0000 7fc40d42a740 1 -- 172.21.15.16:0/1328707867 wait complete. 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc40d42a740 1 -- start start 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc40d42a740 1 Processor -- start 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d77d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 0x5577a0d782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc40d42a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a09a1840 0x5577a0def2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc40d42a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4001b4310 con 0x5577a0980ea0 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc40d42a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4001b3c10 con 0x5577a09a1840 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc40d42a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc4001b0e10 con 0x5577a0999660 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc40d42a740 10 client.0 _ll_register_callbacks cb 0x7fc3ec00bc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc404c51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d77d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc404c51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d77d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42090/0 (socket says 172.21.15.16:42090) 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc405452640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a09a1840 0x5577a0def2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.600+0000 7fc404c51640 1 -- 172.21.15.16:0/267879626 learned_addr learned my addr 172.21.15.16:0/267879626 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:47.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc3fffff640 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 0x5577a0d782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc404c51640 1 -- 172.21.15.16:0/267879626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 msgr2=0x5577a0d782c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc404c51640 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 0x5577a0d782c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc404c51640 1 -- 172.21.15.16:0/267879626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a09a1840 msgr2=0x5577a0def2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc404c51640 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a09a1840 0x5577a0def2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc404c51640 1 -- 172.21.15.16:0/267879626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577a0def850 con 0x5577a0980ea0 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc3fffff640 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 0x5577a0d782c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc404c51640 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d77d80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40e29e0 tx=0x7fc3f40e2e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b14b5020cc1723 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc3dffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc3dffff640 1 -- 172.21.15.16:0/267879626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f40ab8d0 con 0x5577a0980ea0 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc3dffff640 1 -- 172.21.15.16:0/267879626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fc3f40e5180 con 0x5577a0980ea0 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc40d42a740 1 -- 172.21.15.16:0/267879626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3f4009de0 con 0x5577a0980ea0 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc3dffff640 1 -- 172.21.15.16:0/267879626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fc3f4020020 con 0x5577a0980ea0 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.601+0000 7fc40d42a740 10 client.18711 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:47.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.602+0000 7fc40d42a740 1 -- 172.21.15.16:0/267879626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x5577a0df0b40 con 0x5577a0980ea0 2024-08-28T19:33:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.602+0000 7fc3dffff640 1 -- 172.21.15.16:0/267879626 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3f4009de0 con 0x5577a0980ea0 2024-08-28T19:33:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.602+0000 7fc3dffff640 1 -- 172.21.15.16:0/267879626 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7fc3f404fb00 con 0x5577a0980ea0 2024-08-28T19:33:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.602+0000 7fc40d42a740 10 client.18711 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.602+0000 7fc40d42a740 10 client.18711 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:33:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.602+0000 7fc40d42a740 20 client.18711 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.602+0000 7fc40d42a740 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577a09c2520 0x5577a0df3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.602+0000 7fc40d42a740 4 client.18711 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b52084a", "release": true, "await": true}] 2024-08-28T19:33:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.602+0000 7fc40d42a740 1 -- 172.21.15.16:0/267879626 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b52084a", "release": true, "await": true}) -- 0x5577a0cdc460 con 0x5577a09c2520 2024-08-28T19:33:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.602+0000 7fc3fffff640 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577a09c2520 0x5577a0df3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:47.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.603+0000 7fc3dffff640 10 client.18711 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:47.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.603+0000 7fc3fffff640 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577a09c2520 0x5577a0df3240 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8073a2bc4db58da5 server_cookie=5236cf87500bae99 in_seq=0 out_seq=0 2024-08-28T19:33:47.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.612+0000 7fc3dffff640 1 -- 172.21.15.16:0/267879626 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5577a0cdc460 con 0x5577a09c2520 2024-08-28T19:33:47.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc3dffff640 10 client.18711 handle_command_reply: tid=0 2024-08-28T19:33:47.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc40d42a740 1 client.18711 shutdown 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc40d42a740 1 -- 172.21.15.16:0/267879626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 msgr2=0x5577a0d77d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc40d42a740 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d77d80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40e29e0 tx=0x7fc3f40e2e40 comp rx=0 tx=0).stop 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc40d42a740 1 -- 172.21.15.16:0/267879626 shutdown_connections 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc40d42a740 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577a09a1840 0x5577a0def2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc40d42a740 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5577a09c2520 0x5577a0df3240 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc40d42a740 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5577a0999660 0x5577a0d782c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc40d42a740 1 --2- 172.21.15.16:0/267879626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577a0980ea0 0x5577a0d77d80 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc40d42a740 1 -- 172.21.15.16:0/267879626 >> 172.21.15.16:0/267879626 conn(0x5577a09b6910 msgr2=0x5577a0d6e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc3dffff640 0 client.18711 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.613+0000 7fc40d42a740 1 -- 172.21.15.16:0/267879626 shutdown_connections 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.614+0000 7fc40d42a740 1 -- 172.21.15.16:0/267879626 wait complete. 2024-08-28T19:33:47.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.614+0000 7fc40d42a740 20 client.18711 trim_cache size 0 max 16384 2024-08-28T19:33:47.616 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:33:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 63, 2024-08-28T19:33:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:33:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 40, 2024-08-28T19:33:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:33:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "b52084a": { 2024-08-28T19:33:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 40, 2024-08-28T19:33:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:33:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:33:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 140.5, 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:47.618 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:33:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.614+0000 7fc406eb5640 1 -- 172.21.15.16:0/3362195943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3f00b4ca0 msgr2=0x7fc3f00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.614+0000 7fc406eb5640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3f00b4ca0 0x7fc3f00b71b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8035000 tx=0x7fc3f8031bd0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.614+0000 7fc406eb5640 1 -- 172.21.15.16:0/3362195943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc400103920 msgr2=0x7fc4000754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.614+0000 7fc406eb5640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc400103920 0x7fc4000754b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400d950 tx=0x7fc3f400de10 comp rx=0 tx=0).stop 2024-08-28T19:33:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.615+0000 7fc406eb5640 1 -- 172.21.15.16:0/3362195943 shutdown_connections 2024-08-28T19:33:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.615+0000 7fc406eb5640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc3f00b4ca0 0x7fc3f00b71b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.615+0000 7fc406eb5640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc400109bb0 0x7fc400075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.615+0000 7fc406eb5640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc400109260 0x7fc4000759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.615+0000 7fc406eb5640 1 --2- 172.21.15.16:0/3362195943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc400103920 0x7fc4000754b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.615+0000 7fc406eb5640 1 -- 172.21.15.16:0/3362195943 >> 172.21.15.16:0/3362195943 conn(0x7fc4000ff3b0 msgr2=0x7fc4001067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.615+0000 7fc406eb5640 1 -- 172.21.15.16:0/3362195943 shutdown_connections 2024-08-28T19:33:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:47.615+0000 7fc406eb5640 1 -- 172.21.15.16:0/3362195943 wait complete. 2024-08-28T19:33:47.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 503495 3.54 MB/sec execute 567 sec latency 21417.608 ms 2024-08-28T19:33:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.237+0000 7fcfa5655640 1 -- 172.21.15.16:0/3359440437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa0108a90 msgr2=0x7fcfa010f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.237+0000 7fcfa5655640 1 --2- 172.21.15.16:0/3359440437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa0108a90 0x7fcfa010f2c0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009ca0 tx=0x7fcf9802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.237+0000 7fcfa5655640 1 -- 172.21.15.16:0/3359440437 shutdown_connections 2024-08-28T19:33:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.237+0000 7fcfa5655640 1 --2- 172.21.15.16:0/3359440437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa0108a90 0x7fcfa010f2c0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.237+0000 7fcfa5655640 1 --2- 172.21.15.16:0/3359440437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcfa0108140 0x7fcfa0108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.238+0000 7fcfa5655640 1 --2- 172.21.15.16:0/3359440437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0102800 0x7fcfa0102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.238+0000 7fcfa5655640 1 -- 172.21.15.16:0/3359440437 >> 172.21.15.16:0/3359440437 conn(0x7fcfa00794d0 msgr2=0x7fcfa00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:48.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.238+0000 7fcfa5655640 1 -- 172.21.15.16:0/3359440437 shutdown_connections 2024-08-28T19:33:48.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.238+0000 7fcfa5655640 1 -- 172.21.15.16:0/3359440437 wait complete. 2024-08-28T19:33:48.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcfa5655640 1 Processor -- start 2024-08-28T19:33:48.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcfa5655640 1 -- start start 2024-08-28T19:33:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcfa5655640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0102800 0x7fcfa019b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcfa5655640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa0108140 0x7fcfa019b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcfa5655640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcfa0108a90 0x7fcfa019bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcfa5655640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa0112710 con 0x7fcfa0108140 2024-08-28T19:33:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcfa5655640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa0112590 con 0x7fcfa0102800 2024-08-28T19:33:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcfa5655640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa0112890 con 0x7fcfa0108a90 2024-08-28T19:33:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcf9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0102800 0x7fcfa019b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcf9e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa0108140 0x7fcfa019b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcf9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0102800 0x7fcfa019b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45992/0 (socket says 172.21.15.16:45992) 2024-08-28T19:33:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcf9e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa0108140 0x7fcfa019b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42110/0 (socket says 172.21.15.16:42110) 2024-08-28T19:33:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.239+0000 7fcf9effd640 1 -- 172.21.15.16:0/2755645821 learned_addr learned my addr 172.21.15.16:0/2755645821 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcfa0108a90 0x7fcfa019bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcf9effd640 1 -- 172.21.15.16:0/2755645821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcfa0108a90 msgr2=0x7fcfa019bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcf9effd640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcfa0108a90 0x7fcfa019bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcf9effd640 1 -- 172.21.15.16:0/2755645821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa0108140 msgr2=0x7fcfa019b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcf9effd640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa0108140 0x7fcfa019b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcf9effd640 1 -- 172.21.15.16:0/2755645821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa01a6310 con 0x7fcfa0102800 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcf9effd640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0102800 0x7fcfa019b080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000e9f0 tx=0x7fcf9000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcf87fff640 1 -- 172.21.15.16:0/2755645821 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000cd80 con 0x7fcfa0102800 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcfa5655640 1 -- 172.21.15.16:0/2755645821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa01a6600 con 0x7fcfa0102800 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcf87fff640 1 -- 172.21.15.16:0/2755645821 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf900108e0 con 0x7fcfa0102800 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcfa5655640 1 -- 172.21.15.16:0/2755645821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa01a6b90 con 0x7fcfa0102800 2024-08-28T19:33:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.240+0000 7fcf87fff640 1 -- 172.21.15.16:0/2755645821 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90018980 con 0x7fcfa0102800 2024-08-28T19:33:48.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.243+0000 7fcfabbca740 1 Processor -- start 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.243+0000 7fcfabbca740 1 -- start start 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.243+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef19a070 0x55e4ef53f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.243+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 0x55e4ef53f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.243+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 0x55e4ef53fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.243+0000 7fcfabbca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef4b6680 con 0x55e4ef1711b0 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.243+0000 7fcfabbca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef4a5c10 con 0x55e4ef179390 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.243+0000 7fcfabbca740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef4a9bc0 con 0x55e4ef19a070 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf87fff640 1 -- 172.21.15.16:0/2755645821 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fcf90010430 con 0x7fcfa0102800 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 0x55e4ef53fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9f7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef19a070 0x55e4ef53f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 0x55e4ef53fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:46002/0 (socket says 172.21.15.16:46002) 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9e7fc640 1 -- 172.21.15.16:0/606893926 learned_addr learned my addr 172.21.15.16:0/606893926 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9effd640 1 --2- 172.21.15.16:0/606893926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 0x55e4ef53f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9f7fe640 1 -- 172.21.15.16:0/606893926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 msgr2=0x55e4ef53fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/606893926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 0x55e4ef53fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9f7fe640 1 -- 172.21.15.16:0/606893926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 msgr2=0x55e4ef53f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/606893926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 0x55e4ef53f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9f7fe640 1 -- 172.21.15.16:0/606893926 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4ef205650 con 0x55e4ef19a070 2024-08-28T19:33:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9effd640 1 --2- 172.21.15.16:0/606893926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 0x55e4ef53f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/606893926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef19a070 0x55e4ef53f310 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcf98005e30 tx=0x7fcf9802f410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1caecd31bd884587 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.244+0000 7fcf857fa640 1 -- 172.21.15.16:0/606893926 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98031cf0 con 0x55e4ef19a070 2024-08-28T19:33:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.245+0000 7fcf857fa640 1 -- 172.21.15.16:0/606893926 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf98036030 con 0x55e4ef19a070 2024-08-28T19:33:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.245+0000 7fcf857fa640 1 -- 172.21.15.16:0/606893926 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98004040 con 0x55e4ef19a070 2024-08-28T19:33:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.245+0000 7fcf87fff640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf880b4cd0 0x7fcf880b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.245+0000 7fcf9effd640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf880b4cd0 0x7fcf880b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.245+0000 7fcf87fff640 1 -- 172.21.15.16:0/2755645821 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fcf900e5910 con 0x7fcfa0102800 2024-08-28T19:33:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.245+0000 7fcfabbca740 1 -- 172.21.15.16:0/606893926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef19a070 msgr2=0x55e4ef53f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.245+0000 7fcfabbca740 1 --2- 172.21.15.16:0/606893926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef19a070 0x55e4ef53f310 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcf98005e30 tx=0x7fcf9802f410 comp rx=0 tx=0).stop 2024-08-28T19:33:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.245+0000 7fcf9effd640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf880b4cd0 0x7fcf880b71e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcf900e2d40 tx=0x7fcf9001f000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 -- 172.21.15.16:0/606893926 shutdown_connections 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 --2- 172.21.15.16:0/606893926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 0x55e4ef53fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 --2- 172.21.15.16:0/606893926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 0x55e4ef53f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 --2- 172.21.15.16:0/606893926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef19a070 0x55e4ef53f310 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 -- 172.21.15.16:0/606893926 >> 172.21.15.16:0/606893926 conn(0x55e4ef18e460 msgr2=0x55e4ef53d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 -- 172.21.15.16:0/606893926 shutdown_connections 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 -- 172.21.15.16:0/606893926 wait complete. 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 -- start start 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 Processor -- start 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef1711b0 0x55e4ef54aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef179390 0x55e4ef54afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef19a070 0x55e4ef556c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef308700 con 0x55e4ef179390 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef42c750 con 0x55e4ef19a070 2024-08-28T19:33:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.246+0000 7fcfabbca740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef54d910 con 0x55e4ef1711b0 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcfabbca740 10 client.0 _ll_register_callbacks cb 0x55e4ef45be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef179390 0x55e4ef54afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9f7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef1711b0 0x55e4ef54aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef179390 0x55e4ef54afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42120/0 (socket says 172.21.15.16:42120) 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9f7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef1711b0 0x55e4ef54aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:39640/0 (socket says 172.21.15.16:39640) 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9e7fc640 1 -- 172.21.15.16:0/4256802237 learned_addr learned my addr 172.21.15.16:0/4256802237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef19a070 0x55e4ef556c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9e7fc640 1 -- 172.21.15.16:0/4256802237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef1711b0 msgr2=0x55e4ef54aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9e7fc640 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef1711b0 0x55e4ef54aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9e7fc640 1 -- 172.21.15.16:0/4256802237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef19a070 msgr2=0x55e4ef556c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9e7fc640 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef19a070 0x55e4ef556c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9e7fc640 1 -- 172.21.15.16:0/4256802237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4ef465900 con 0x55e4ef179390 2024-08-28T19:33:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef19a070 0x55e4ef556c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf9e7fc640 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef179390 0x55e4ef54afd0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fcf94012740 tx=0x7fcf94012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a4ccf4ca860ebb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf82ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf82ff5640 1 -- 172.21.15.16:0/4256802237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9400f920 con 0x55e4ef179390 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcfabbca740 1 -- 172.21.15.16:0/4256802237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf98036030 con 0x55e4ef179390 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.247+0000 7fcf82ff5640 1 -- 172.21.15.16:0/4256802237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fcf9400fac0 con 0x55e4ef179390 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcf82ff5640 1 -- 172.21.15.16:0/4256802237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fcf9400fd00 con 0x55e4ef179390 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcfabbca740 10 client.18723 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcfabbca740 1 -- 172.21.15.16:0/4256802237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55e4ef559770 con 0x55e4ef179390 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcf82ff5640 1 -- 172.21.15.16:0/4256802237 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf98036030 con 0x55e4ef179390 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcf82ff5640 1 -- 172.21.15.16:0/4256802237 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7fcf9401c420 con 0x55e4ef179390 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcfabbca740 10 client.18723 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcfabbca740 10 client.18723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcfabbca740 20 client.18723 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcfabbca740 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e4ef1589f0 0x55e4ef55cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcfabbca740 4 client.18723 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcfabbca740 1 -- 172.21.15.16:0/4256802237 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4ef4b85d0 con 0x55e4ef1589f0 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.248+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e4ef1589f0 0x55e4ef55cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.249+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e4ef1589f0 0x55e4ef55cc80 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ec4168e3c39d1e7 server_cookie=ed4fe28f7a0ab18a in_seq=0 out_seq=0 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.249+0000 7fcf82ff5640 10 client.18723 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.249+0000 7fcf82ff5640 1 -- 172.21.15.16:0/4256802237 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e4ef4b85d0 con 0x55e4ef1589f0 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.249+0000 7fcf82ff5640 10 client.18723 handle_command_reply: tid=0 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 1 client.18723 shutdown 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 1 -- 172.21.15.16:0/4256802237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef179390 msgr2=0x55e4ef54afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef179390 0x55e4ef54afd0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fcf94012740 tx=0x7fcf94012c00 comp rx=0 tx=0).stop 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 1 -- 172.21.15.16:0/4256802237 shutdown_connections 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef19a070 0x55e4ef556c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef179390 0x55e4ef54afd0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e4ef1589f0 0x55e4ef55cc80 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 1 --2- 172.21.15.16:0/4256802237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef1711b0 0x55e4ef54aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 1 -- 172.21.15.16:0/4256802237 >> 172.21.15.16:0/4256802237 conn(0x55e4ef18e460 msgr2=0x55e4ef5424a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcf82ff5640 0 client.18723 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 1 -- 172.21.15.16:0/4256802237 shutdown_connections 2024-08-28T19:33:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 1 -- 172.21.15.16:0/4256802237 wait complete. 2024-08-28T19:33:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.250+0000 7fcfabbca740 20 client.18723 trim_cache size 0 max 16384 2024-08-28T19:33:48.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.263+0000 7fcfabbca740 1 Processor -- start 2024-08-28T19:33:48.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcfabbca740 1 -- start start 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 0x55e4ef53d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 0x55e4ef556b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef19a070 0x55e4ef547c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcfabbca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c00ffe0 con 0x55e4ef1711b0 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcfabbca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c011be0 con 0x55e4ef179390 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcfabbca740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c0106e0 con 0x55e4ef19a070 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcf9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 0x55e4ef53d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcf9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 0x55e4ef53d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42130/0 (socket says 172.21.15.16:42130) 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcf9f7fe640 1 -- 172.21.15.16:0/1688999982 learned_addr learned my addr 172.21.15.16:0/1688999982 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcf9e7fc640 1 --2- 172.21.15.16:0/1688999982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 0x55e4ef556b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/1688999982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef19a070 0x55e4ef547c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcf9f7fe640 1 -- 172.21.15.16:0/1688999982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef19a070 msgr2=0x55e4ef547c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/1688999982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef19a070 0x55e4ef547c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcf9f7fe640 1 -- 172.21.15.16:0/1688999982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 msgr2=0x55e4ef556b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/1688999982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 0x55e4ef556b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcf9f7fe640 1 -- 172.21.15.16:0/1688999982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4ef54bd50 con 0x55e4ef1711b0 2024-08-28T19:33:48.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.264+0000 7fcf9e7fc640 1 --2- 172.21.15.16:0/1688999982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 0x55e4ef556b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:48.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.265+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/1688999982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 0x55e4ef53d230 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fcf98005cc0 tx=0x7fcf9802f410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce8cbe75ba4c380 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.265+0000 7fcf84ff9640 1 -- 172.21.15.16:0/1688999982 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98003c00 con 0x55e4ef1711b0 2024-08-28T19:33:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.265+0000 7fcf84ff9640 1 -- 172.21.15.16:0/1688999982 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf98036070 con 0x55e4ef1711b0 2024-08-28T19:33:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.265+0000 7fcf84ff9640 1 -- 172.21.15.16:0/1688999982 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98004130 con 0x55e4ef1711b0 2024-08-28T19:33:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.266+0000 7fcfabbca740 1 -- 172.21.15.16:0/1688999982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 msgr2=0x55e4ef53d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.266+0000 7fcfabbca740 1 --2- 172.21.15.16:0/1688999982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 0x55e4ef53d230 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fcf98005cc0 tx=0x7fcf9802f410 comp rx=0 tx=0).stop 2024-08-28T19:33:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.266+0000 7fcfabbca740 1 -- 172.21.15.16:0/1688999982 shutdown_connections 2024-08-28T19:33:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.266+0000 7fcfabbca740 1 --2- 172.21.15.16:0/1688999982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef19a070 0x55e4ef547c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.266+0000 7fcfabbca740 1 --2- 172.21.15.16:0/1688999982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef179390 0x55e4ef556b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.266+0000 7fcfabbca740 1 --2- 172.21.15.16:0/1688999982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef1711b0 0x55e4ef53d230 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.266+0000 7fcfabbca740 1 -- 172.21.15.16:0/1688999982 >> 172.21.15.16:0/1688999982 conn(0x55e4ef18e460 msgr2=0x55e4ef53cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.266+0000 7fcfabbca740 1 -- 172.21.15.16:0/1688999982 shutdown_connections 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.266+0000 7fcfabbca740 1 -- 172.21.15.16:0/1688999982 wait complete. 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.266+0000 7fcfabbca740 1 -- start start 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.266+0000 7fcfabbca740 1 Processor -- start 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef1711b0 0x55e4ef553370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef179390 0x55e4ef54e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcfabbca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef19a070 0x55e4ef54ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcfabbca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c0114e0 con 0x55e4ef19a070 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcfabbca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c010de0 con 0x55e4ef1711b0 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcfabbca740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c00dfe0 con 0x55e4ef179390 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcfabbca740 10 client.0 _ll_register_callbacks cb 0x7fcf94009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9e7fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef179390 0x55e4ef54e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef1711b0 0x55e4ef553370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9e7fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef179390 0x55e4ef54e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:39656/0 (socket says 172.21.15.16:39656) 2024-08-28T19:33:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9e7fc640 1 -- 172.21.15.16:0/2785883434 learned_addr learned my addr 172.21.15.16:0/2785883434 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef1711b0 0x55e4ef553370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:46020/0 (socket says 172.21.15.16:46020) 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef19a070 0x55e4ef54ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9e7fc640 1 -- 172.21.15.16:0/2785883434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef1711b0 msgr2=0x55e4ef553370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9e7fc640 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef1711b0 0x55e4ef553370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9e7fc640 1 -- 172.21.15.16:0/2785883434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef19a070 msgr2=0x55e4ef54ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9e7fc640 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef19a070 0x55e4ef54ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9e7fc640 1 -- 172.21.15.16:0/2785883434 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4ef54f4d0 con 0x55e4ef179390 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef19a070 0x55e4ef54ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf9e7fc640 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef179390 0x55e4ef54e820 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009e60 tx=0x7fcf9400fb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64f35957b5114996 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.267+0000 7fcf82ff5640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcf82ff5640 1 -- 172.21.15.16:0/2785883434 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9402f060 con 0x55e4ef179390 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcfabbca740 1 -- 172.21.15.16:0/2785883434 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf98036070 con 0x55e4ef179390 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcf82ff5640 1 -- 172.21.15.16:0/2785883434 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7fcf940050d0 con 0x55e4ef179390 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcf82ff5640 1 -- 172.21.15.16:0/2785883434 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fcf9400fea0 con 0x55e4ef179390 2024-08-28T19:33:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcfabbca740 10 client.27850 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcfabbca740 1 -- 172.21.15.16:0/2785883434 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55e4ef5c7550 con 0x55e4ef179390 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcf82ff5640 1 -- 172.21.15.16:0/2785883434 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf98036070 con 0x55e4ef179390 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcf82ff5640 1 -- 172.21.15.16:0/2785883434 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7fcf94003900 con 0x55e4ef179390 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcfabbca740 10 client.27850 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcfabbca740 10 client.27850 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcfabbca740 20 client.27850 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcfabbca740 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e4ef12dda0 0x55e4ef5c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcfabbca740 4 client.27850 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcfabbca740 1 -- 172.21.15.16:0/2785883434 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e4ef4b4070 con 0x55e4ef12dda0 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.268+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e4ef12dda0 0x55e4ef5c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcf9f7fe640 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e4ef12dda0 0x55e4ef5c9d70 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=390522a79c38ba5a server_cookie=60c69ee69cbb8da4 in_seq=0 out_seq=0 2024-08-28T19:33:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcf82ff5640 10 client.27850 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcf82ff5640 1 -- 172.21.15.16:0/2785883434 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e4ef4b4070 con 0x55e4ef12dda0 2024-08-28T19:33:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcf82ff5640 10 client.27850 handle_command_reply: tid=0 2024-08-28T19:33:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcfabbca740 1 client.27850 shutdown 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcfabbca740 1 -- 172.21.15.16:0/2785883434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef179390 msgr2=0x55e4ef54e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcfabbca740 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef179390 0x55e4ef54e820 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009e60 tx=0x7fcf9400fb50 comp rx=0 tx=0).stop 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcfabbca740 1 -- 172.21.15.16:0/2785883434 shutdown_connections 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcfabbca740 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4ef19a070 0x55e4ef54ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcfabbca740 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e4ef12dda0 0x55e4ef5c9d70 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcfabbca740 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e4ef179390 0x55e4ef54e820 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcfabbca740 1 --2- 172.21.15.16:0/2785883434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4ef1711b0 0x55e4ef553370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcfabbca740 1 -- 172.21.15.16:0/2785883434 >> 172.21.15.16:0/2785883434 conn(0x55e4ef18e460 msgr2=0x55e4ef540340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcf82ff5640 0 client.27850 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcfabbca740 1 -- 172.21.15.16:0/2785883434 shutdown_connections 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.269+0000 7fcfabbca740 1 -- 172.21.15.16:0/2785883434 wait complete. 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfabbca740 20 client.27850 trim_cache size 0 max 16384 2024-08-28T19:33:48.275 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:33:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 -- 172.21.15.16:0/2755645821 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf880b4cd0 msgr2=0x7fcf880b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf880b4cd0 0x7fcf880b71e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcf900e2d40 tx=0x7fcf9001f000 comp rx=0 tx=0).stop 2024-08-28T19:33:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 -- 172.21.15.16:0/2755645821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0102800 msgr2=0x7fcfa019b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0102800 0x7fcfa019b080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000e9f0 tx=0x7fcf9000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 -- 172.21.15.16:0/2755645821 shutdown_connections 2024-08-28T19:33:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcfa0108a90 0x7fcfa019bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa0108140 0x7fcfa019b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcf880b4cd0 0x7fcf880b71e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 --2- 172.21.15.16:0/2755645821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0102800 0x7fcfa019b080 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 -- 172.21.15.16:0/2755645821 >> 172.21.15.16:0/2755645821 conn(0x7fcfa00794d0 msgr2=0x7fcfa0105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 -- 172.21.15.16:0/2755645821 shutdown_connections 2024-08-28T19:33:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:48.270+0000 7fcfa5655640 1 -- 172.21.15.16:0/2755645821 wait complete. 2024-08-28T19:33:48.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 504471 3.55 MB/sec execute 568 sec latency 542.884 ms 2024-08-28T19:33:49.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:48 smithi016 ceph-mon[18556]: pgmap v734: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 478 KiB/s wr, 36 op/s 2024-08-28T19:33:49.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:48 smithi115 ceph-mon[22184]: pgmap v734: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 478 KiB/s wr, 36 op/s 2024-08-28T19:33:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:48 smithi028 ceph-mon[21769]: pgmap v734: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 478 KiB/s wr, 36 op/s 2024-08-28T19:33:49.520 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b52084a', total seconds elapsed: 25.9 2024-08-28T19:33:49.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 504471 3.54 MB/sec execute 569 sec latency 1542.953 ms 2024-08-28T19:33:50.053 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:33:50.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:33:50.318 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.854+0000 7ff4bd68d640 1 -- 172.21.15.16:0/478950426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b8109bd0 msgr2=0x7ff4b8110400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.854+0000 7ff4bd68d640 1 --2- 172.21.15.16:0/478950426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b8109bd0 0x7ff4b8110400 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0009ca0 tx=0x7ff4b002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.854+0000 7ff4bd68d640 1 -- 172.21.15.16:0/478950426 shutdown_connections 2024-08-28T19:33:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.854+0000 7ff4bd68d640 1 --2- 172.21.15.16:0/478950426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b8109bd0 0x7ff4b8110400 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.854+0000 7ff4bd68d640 1 --2- 172.21.15.16:0/478950426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4b8109280 0x7ff4b8109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.854+0000 7ff4bd68d640 1 --2- 172.21.15.16:0/478950426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4b8103940 0x7ff4b8103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.854+0000 7ff4bd68d640 1 -- 172.21.15.16:0/478950426 >> 172.21.15.16:0/478950426 conn(0x7ff4b80ff3d0 msgr2=0x7ff4b8101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.854+0000 7ff4bd68d640 1 -- 172.21.15.16:0/478950426 shutdown_connections 2024-08-28T19:33:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.854+0000 7ff4bd68d640 1 -- 172.21.15.16:0/478950426 wait complete. 2024-08-28T19:33:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.855+0000 7ff4bd68d640 1 Processor -- start 2024-08-28T19:33:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.855+0000 7ff4bd68d640 1 -- start start 2024-08-28T19:33:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4bd68d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4b8103940 0x7ff4b819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4bd68d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b8109280 0x7ff4b819a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4bd68d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4b8109bd0 0x7ff4b819fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4bd68d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b8113890 con 0x7ff4b8109280 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4bd68d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b8113710 con 0x7ff4b8109bd0 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4bd68d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b8113a10 con 0x7ff4b8103940 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b8109280 0x7ff4b819a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b8109280 0x7ff4b819a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42158/0 (socket says 172.21.15.16:42158) 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4affff640 1 -- 172.21.15.16:0/327986303 learned_addr learned my addr 172.21.15.16:0/327986303 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4b77fe640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4b8109bd0 0x7ff4b819fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4b6ffd640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4b8103940 0x7ff4b819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4b77fe640 1 -- 172.21.15.16:0/327986303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4b8103940 msgr2=0x7ff4b819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.856+0000 7ff4b77fe640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4b8103940 0x7ff4b819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.857+0000 7ff4b77fe640 1 -- 172.21.15.16:0/327986303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b8109280 msgr2=0x7ff4b819a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.857+0000 7ff4b77fe640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b8109280 0x7ff4b819a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.857+0000 7ff4b77fe640 1 -- 172.21.15.16:0/327986303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b81a01f0 con 0x7ff4b8109bd0 2024-08-28T19:33:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.857+0000 7ff4affff640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b8109280 0x7ff4b819a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.857+0000 7ff4b77fe640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4b8109bd0 0x7ff4b819fac0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff4b002ade0 tx=0x7ff4b0031660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.857+0000 7ff4b4ff9640 1 -- 172.21.15.16:0/327986303 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b0003b70 con 0x7ff4b8109bd0 2024-08-28T19:33:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.857+0000 7ff4b4ff9640 1 -- 172.21.15.16:0/327986303 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4b0031910 con 0x7ff4b8109bd0 2024-08-28T19:33:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.857+0000 7ff4b4ff9640 1 -- 172.21.15.16:0/327986303 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b002f950 con 0x7ff4b8109bd0 2024-08-28T19:33:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.857+0000 7ff4bd68d640 1 -- 172.21.15.16:0/327986303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b81a0480 con 0x7ff4b8109bd0 2024-08-28T19:33:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.857+0000 7ff4bd68d640 1 -- 172.21.15.16:0/327986303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b81a08b0 con 0x7ff4b8109bd0 2024-08-28T19:33:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.860+0000 7ff4c3c02740 1 Processor -- start 2024-08-28T19:33:50.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.861+0000 7ff4b4ff9640 1 -- 172.21.15.16:0/327986303 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff4b002faf0 con 0x7ff4b8109bd0 2024-08-28T19:33:50.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.861+0000 7ff4c3c02740 1 -- start start 2024-08-28T19:33:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.861+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36ae6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.861+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b36717490 0x556b36ae6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b3671f670 0x556b36ae7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556b36a5c5f0 con 0x556b36717490 2024-08-28T19:33:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b36a4bb80 con 0x556b36740350 2024-08-28T19:33:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556b36a4fb30 con 0x556b3671f670 2024-08-28T19:33:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36ae6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b36717490 0x556b36ae6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36ae6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:46054/0 (socket says 172.21.15.16:46054) 2024-08-28T19:33:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4affff640 1 -- 172.21.15.16:0/2785452534 learned_addr learned my addr 172.21.15.16:0/2785452534 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4affff640 1 --2- 172.21.15.16:0/2785452534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b3671f670 0x556b36ae7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4b4ff9640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4a40b4d20 0x7ff4a40b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4affff640 1 -- 172.21.15.16:0/2785452534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b3671f670 msgr2=0x556b36ae7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:50.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4affff640 1 --2- 172.21.15.16:0/2785452534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b3671f670 0x556b36ae7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4affff640 1 -- 172.21.15.16:0/2785452534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b36717490 msgr2=0x556b36ae6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:50.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4affff640 1 --2- 172.21.15.16:0/2785452534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b36717490 0x556b36ae6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4affff640 1 -- 172.21.15.16:0/2785452534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b367ab930 con 0x556b36740350 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.862+0000 7ff4b77fe640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4a40b4d20 0x7ff4a40b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.863+0000 7ff4b4ff9640 1 -- 172.21.15.16:0/327986303 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7ff4b0106990 con 0x7ff4b8109bd0 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.863+0000 7ff4affff640 1 --2- 172.21.15.16:0/2785452534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36ae6780 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00058c0 tx=0x7ff4a0009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95b069ea0b33c0ba server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.863+0000 7ff4ad7fa640 1 -- 172.21.15.16:0/2785452534 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a000fc50 con 0x556b36740350 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.863+0000 7ff4ad7fa640 1 -- 172.21.15.16:0/2785452534 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4a000fdf0 con 0x556b36740350 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.863+0000 7ff4ad7fa640 1 -- 172.21.15.16:0/2785452534 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a000e5b0 con 0x556b36740350 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.864+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2785452534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 msgr2=0x556b36ae6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.864+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2785452534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36ae6780 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00058c0 tx=0x7ff4a0009530 comp rx=0 tx=0).stop 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.864+0000 7ff4b77fe640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4a40b4d20 0x7ff4a40b7230 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff4b00c9f00 tx=0x7ff4b0091750 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.864+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2785452534 shutdown_connections 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.864+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2785452534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b3671f670 0x556b36ae7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.864+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2785452534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b36717490 0x556b36ae6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.864+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2785452534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36ae6780 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.864+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2785452534 >> 172.21.15.16:0/2785452534 conn(0x556b36734740 msgr2=0x556b36ae4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.864+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2785452534 shutdown_connections 2024-08-28T19:33:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.864+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2785452534 wait complete. 2024-08-28T19:33:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.865+0000 7ff4c3c02740 1 -- start start 2024-08-28T19:33:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.865+0000 7ff4c3c02740 1 Processor -- start 2024-08-28T19:33:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.865+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36717490 0x556b36af1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.865+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.865+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36740350 0x556b36afe280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.865+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556b368ae890 con 0x556b3671f670 2024-08-28T19:33:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b369d2790 con 0x556b36717490 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556b36af4f60 con 0x556b36740350 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36717490 0x556b36af1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42190/0 (socket says 172.21.15.16:42190) 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4c3c02740 10 client.0 _ll_register_callbacks cb 0x556b36757ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/2314905593 learned_addr learned my addr 172.21.15.16:0/2314905593 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4affff640 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36740350 0x556b36afe280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/2314905593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36740350 msgr2=0x556b36afe280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4b6ffd640 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36740350 0x556b36afe280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/2314905593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36717490 msgr2=0x556b36af1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4b6ffd640 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36717490 0x556b36af1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/2314905593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b36a09f30 con 0x556b3671f670 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4affff640 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36740350 0x556b36afe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:33:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4b6ffd640 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af24d0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009a50 tx=0x7ff4a8009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe82c46e7c780ae server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff49e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff49e7fc640 1 -- 172.21.15.16:0/2314905593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8004020 con 0x556b3671f670 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff49e7fc640 1 -- 172.21.15.16:0/2314905593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7ff4a80041c0 con 0x556b3671f670 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.866+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2314905593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4a000fdf0 con 0x556b3671f670 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.867+0000 7ff49e7fc640 1 -- 172.21.15.16:0/2314905593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7ff4a8029050 con 0x556b3671f670 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.867+0000 7ff4c3c02740 10 client.18741 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.867+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2314905593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x556b36b00df0 con 0x556b3671f670 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.867+0000 7ff49e7fc640 1 -- 172.21.15.16:0/2314905593 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4a000fdf0 con 0x556b3671f670 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.867+0000 7ff49e7fc640 1 -- 172.21.15.16:0/2314905593 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7ff4a801e400 con 0x556b3671f670 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.867+0000 7ff4c3c02740 10 client.18741 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.867+0000 7ff4c3c02740 10 client.18741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.867+0000 7ff4c3c02740 20 client.18741 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.867+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556b366fecd0 0x556b36b043e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.867+0000 7ff4c3c02740 4 client.18741 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:50.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.867+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2314905593 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b36a5e540 con 0x556b366fecd0 2024-08-28T19:33:50.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.868+0000 7ff4affff640 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556b366fecd0 0x556b36b043e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.868+0000 7ff4affff640 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556b366fecd0 0x556b36b043e0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25e9bd71918553b server_cookie=7cd015528f116b44 in_seq=0 out_seq=0 2024-08-28T19:33:50.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.868+0000 7ff49e7fc640 10 client.18741 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff49e7fc640 1 -- 172.21.15.16:0/2314905593 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556b36a5e540 con 0x556b366fecd0 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff49e7fc640 10 client.18741 handle_command_reply: tid=0 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff4c3c02740 1 client.18741 shutdown 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2314905593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 msgr2=0x556b36af24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af24d0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009a50 tx=0x7ff4a8009f10 comp rx=0 tx=0).stop 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2314905593 shutdown_connections 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556b366fecd0 0x556b36b043e0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36740350 0x556b36afe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af24d0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2314905593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36717490 0x556b36af1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2314905593 >> 172.21.15.16:0/2314905593 conn(0x556b36734740 msgr2=0x556b36ae9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:50.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff49e7fc640 0 client.18741 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:50.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2314905593 shutdown_connections 2024-08-28T19:33:50.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.869+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2314905593 wait complete. 2024-08-28T19:33:50.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.870+0000 7ff4c3c02740 20 client.18741 trim_cache size 0 max 16384 2024-08-28T19:33:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.888+0000 7ff4c3c02740 1 Processor -- start 2024-08-28T19:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.888+0000 7ff4c3c02740 1 -- start start 2024-08-28T19:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.888+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 0x556b36af2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.888+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36ae9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.888+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36afe180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.888+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b81aead0 con 0x556b3671f670 2024-08-28T19:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.888+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b81b06d0 con 0x556b36740350 2024-08-28T19:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.888+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b81af1d0 con 0x556b36717490 2024-08-28T19:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36ae9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36ae9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42198/0 (socket says 172.21.15.16:42198) 2024-08-28T19:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/2396097019 learned_addr learned my addr 172.21.15.16:0/2396097019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4affff640 1 --2- 172.21.15.16:0/2396097019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 0x556b36af2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4affff640 1 --2- 172.21.15.16:0/2396097019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36afe180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/2396097019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 msgr2=0x556b36af2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:50.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4b6ffd640 1 --2- 172.21.15.16:0/2396097019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 0x556b36af2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/2396097019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 msgr2=0x556b36afe180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:50.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4b6ffd640 1 --2- 172.21.15.16:0/2396097019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36afe180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/2396097019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b36ae8aa0 con 0x556b3671f670 2024-08-28T19:33:50.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4affff640 1 --2- 172.21.15.16:0/2396097019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 0x556b36af2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:50.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4b6ffd640 1 --2- 172.21.15.16:0/2396097019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36ae9f90 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7ff4a801e8a0 tx=0x7ff4a800efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b25b89b965f28043 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:50.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4acff9640 1 -- 172.21.15.16:0/2396097019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8031020 con 0x556b3671f670 2024-08-28T19:33:50.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4acff9640 1 -- 172.21.15.16:0/2396097019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4a801e600 con 0x556b3671f670 2024-08-28T19:33:50.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.889+0000 7ff4acff9640 1 -- 172.21.15.16:0/2396097019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a801ee60 con 0x556b3671f670 2024-08-28T19:33:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.891+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2396097019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 msgr2=0x556b36ae9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.891+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2396097019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36ae9f90 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7ff4a801e8a0 tx=0x7ff4a800efd0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.891+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2396097019 shutdown_connections 2024-08-28T19:33:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.891+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2396097019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36afe180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.891+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2396097019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36ae9f90 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.891+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/2396097019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 0x556b36af2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.891+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2396097019 >> 172.21.15.16:0/2396097019 conn(0x556b36734740 msgr2=0x556b36ae9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.891+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2396097019 shutdown_connections 2024-08-28T19:33:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.891+0000 7ff4c3c02740 1 -- 172.21.15.16:0/2396097019 wait complete. 2024-08-28T19:33:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.891+0000 7ff4c3c02740 1 -- start start 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4c3c02740 1 Processor -- start 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 0x556b36afabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4c3c02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36af67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b81affd0 con 0x556b3671f670 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b81af8d0 con 0x556b36740350 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4c3c02740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b81acad0 con 0x556b36717490 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4c3c02740 10 client.0 _ll_register_callbacks cb 0x7ff4a8013960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42200/0 (socket says 172.21.15.16:42200) 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/3833768748 learned_addr learned my addr 172.21.15.16:0/3833768748 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:50.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4affff640 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 0x556b36afabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4affff640 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36af67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/3833768748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 msgr2=0x556b36afabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4b6ffd640 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 0x556b36afabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/3833768748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 msgr2=0x556b36af67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.892+0000 7ff4b6ffd640 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36af67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.893+0000 7ff4b6ffd640 1 -- 172.21.15.16:0/3833768748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b36b6dde0 con 0x556b3671f670 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.893+0000 7ff4affff640 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 0x556b36afabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.893+0000 7ff4b6ffd640 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af60b0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8013f80 tx=0x7ff4a801ff80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74279bb9d857f94 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.893+0000 7ff49e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.893+0000 7ff49e7fc640 1 -- 172.21.15.16:0/3833768748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a80052d0 con 0x556b3671f670 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.893+0000 7ff49e7fc640 1 -- 172.21.15.16:0/3833768748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7ff4a8005470 con 0x556b3671f670 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.893+0000 7ff4c3c02740 1 -- 172.21.15.16:0/3833768748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4a801e600 con 0x556b3671f670 2024-08-28T19:33:50.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.893+0000 7ff49e7fc640 1 -- 172.21.15.16:0/3833768748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7ff4a8048050 con 0x556b3671f670 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.893+0000 7ff4c3c02740 10 client.18747 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.893+0000 7ff4c3c02740 1 -- 172.21.15.16:0/3833768748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x556b36b6ef70 con 0x556b3671f670 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.894+0000 7ff49e7fc640 1 -- 172.21.15.16:0/3833768748 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4a801e600 con 0x556b3671f670 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.894+0000 7ff49e7fc640 1 -- 172.21.15.16:0/3833768748 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7ff4a80424e0 con 0x556b3671f670 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.894+0000 7ff4c3c02740 10 client.18747 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.894+0000 7ff4c3c02740 10 client.18747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.894+0000 7ff4c3c02740 20 client.18747 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.894+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556b366d4080 0x556b36b71670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.894+0000 7ff4c3c02740 4 client.18747 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.894+0000 7ff4c3c02740 1 -- 172.21.15.16:0/3833768748 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556b36a59fe0 con 0x556b366d4080 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.894+0000 7ff4affff640 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556b366d4080 0x556b36b71670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.895+0000 7ff49e7fc640 10 client.18747 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.895+0000 7ff4affff640 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556b366d4080 0x556b36b71670 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf3f2fd13dc0312 server_cookie=8925317fad16e593 in_seq=0 out_seq=0 2024-08-28T19:33:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff49e7fc640 1 -- 172.21.15.16:0/3833768748 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556b36a59fe0 con 0x556b366d4080 2024-08-28T19:33:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff49e7fc640 10 client.18747 handle_command_reply: tid=0 2024-08-28T19:33:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff4c3c02740 1 client.18747 shutdown 2024-08-28T19:33:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff4c3c02740 1 -- 172.21.15.16:0/3833768748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 msgr2=0x556b36af60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af60b0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8013f80 tx=0x7ff4a801ff80 comp rx=0 tx=0).stop 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff4c3c02740 1 -- 172.21.15.16:0/3833768748 shutdown_connections 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b36740350 0x556b36af67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556b3671f670 0x556b36af60b0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x556b366d4080 0x556b36b71670 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff4c3c02740 1 --2- 172.21.15.16:0/3833768748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556b36717490 0x556b36afabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff4c3c02740 1 -- 172.21.15.16:0/3833768748 >> 172.21.15.16:0/3833768748 conn(0x556b36734740 msgr2=0x556b36ae4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.896+0000 7ff49e7fc640 0 client.18747 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4c3c02740 1 -- 172.21.15.16:0/3833768748 shutdown_connections 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4c3c02740 1 -- 172.21.15.16:0/3833768748 wait complete. 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4c3c02740 20 client.18747 trim_cache size 0 max 16384 2024-08-28T19:33:50.901 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:33:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:33:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "1fc60423-efa1-4a92-91a1-0815560f5e52", 2024-08-28T19:33:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:33:50.902 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:33:50.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4bd68d640 1 -- 172.21.15.16:0/327986303 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4a40b4d20 msgr2=0x7ff4a40b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4bd68d640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4a40b4d20 0x7ff4a40b7230 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff4b00c9f00 tx=0x7ff4b0091750 comp rx=0 tx=0).stop 2024-08-28T19:33:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4bd68d640 1 -- 172.21.15.16:0/327986303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4b8109bd0 msgr2=0x7ff4b819fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4bd68d640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4b8109bd0 0x7ff4b819fac0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff4b002ade0 tx=0x7ff4b0031660 comp rx=0 tx=0).stop 2024-08-28T19:33:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4bd68d640 1 -- 172.21.15.16:0/327986303 shutdown_connections 2024-08-28T19:33:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4bd68d640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff4a40b4d20 0x7ff4a40b7230 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4bd68d640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4b8109bd0 0x7ff4b819fac0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4bd68d640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b8109280 0x7ff4b819a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4bd68d640 1 --2- 172.21.15.16:0/327986303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4b8103940 0x7ff4b819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.897+0000 7ff4bd68d640 1 -- 172.21.15.16:0/327986303 >> 172.21.15.16:0/327986303 conn(0x7ff4b80ff3d0 msgr2=0x7ff4b8106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.898+0000 7ff4bd68d640 1 -- 172.21.15.16:0/327986303 shutdown_connections 2024-08-28T19:33:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:50.898+0000 7ff4bd68d640 1 -- 172.21.15.16:0/327986303 wait complete. 2024-08-28T19:33:50.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 504489 3.53 MB/sec execute 570 sec latency 2519.635 ms 2024-08-28T19:33:51.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:50 smithi115 ceph-mon[22184]: pgmap v735: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 456 KiB/s wr, 34 op/s 2024-08-28T19:33:51.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:50 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:33:51.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:50 smithi016 ceph-mon[18556]: pgmap v735: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 456 KiB/s wr, 34 op/s 2024-08-28T19:33:51.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:50 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:33:51.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:50 smithi028 ceph-mon[21769]: pgmap v735: 129 pgs: 129 active+clean; 404 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 456 KiB/s wr, 34 op/s 2024-08-28T19:33:51.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:50 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:33:51.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:33:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:33:51.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 504560 3.53 MB/sec execute 571 sec latency 959.703 ms 2024-08-28T19:33:52.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:33:52.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:52.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:52.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:52.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:52.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:52.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:33:52.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:33:52.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:51 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:33:52.142 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:33:52.142 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.142 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:52.142 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:52.142 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:52.142 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:52.142 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:33:52.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:33:52.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:33:52.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:51 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:33:52.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:33:52.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:52.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:52.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:33:52.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:33:52.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:33:52.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:33:52.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:51 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:33:52.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.369+0000 7f2b60ed1640 1 -- 172.21.15.16:0/2850067728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c10dbc0 msgr2=0x7f2b5c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:52.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.369+0000 7f2b60ed1640 1 --2- 172.21.15.16:0/2850067728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c10dbc0 0x7f2b5c1100b0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f2b54009ca0 tx=0x7f2b5402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.369+0000 7f2b60ed1640 1 -- 172.21.15.16:0/2850067728 shutdown_connections 2024-08-28T19:33:52.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.369+0000 7f2b60ed1640 1 --2- 172.21.15.16:0/2850067728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c10dbc0 0x7f2b5c1100b0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.369+0000 7f2b60ed1640 1 --2- 172.21.15.16:0/2850067728 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b5c1050b0 0x7f2b5c10d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.369+0000 7f2b60ed1640 1 --2- 172.21.15.16:0/2850067728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b5c104760 0x7f2b5c104b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.369+0000 7f2b60ed1640 1 -- 172.21.15.16:0/2850067728 >> 172.21.15.16:0/2850067728 conn(0x7f2b5c0794d0 msgr2=0x7f2b5c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:52.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.369+0000 7f2b60ed1640 1 -- 172.21.15.16:0/2850067728 shutdown_connections 2024-08-28T19:33:52.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.369+0000 7f2b60ed1640 1 -- 172.21.15.16:0/2850067728 wait complete. 2024-08-28T19:33:52.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.369+0000 7f2b60ed1640 1 Processor -- start 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b60ed1640 1 -- start start 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b60ed1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b5c104760 0x7f2b5c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b60ed1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b5c1050b0 0x7f2b5c19a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b60ed1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c10dbc0 0x7f2b5c19fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b60ed1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c113820 con 0x7f2b5c10dbc0 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b60ed1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1136a0 con 0x7f2b5c1050b0 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b60ed1640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1139a0 con 0x7f2b5c104760 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b5a59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b5c104760 0x7f2b5c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c10dbc0 0x7f2b5c19fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b5a59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b5c104760 0x7f2b5c19a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:39720/0 (socket says 172.21.15.16:39720) 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b5a59c640 1 -- 172.21.15.16:0/332359757 learned_addr learned my addr 172.21.15.16:0/332359757 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:52.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b53fff640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b5c1050b0 0x7f2b5c19a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b5a59c640 1 -- 172.21.15.16:0/332359757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b5c1050b0 msgr2=0x7f2b5c19a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b5a59c640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b5c1050b0 0x7f2b5c19a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.370+0000 7f2b5a59c640 1 -- 172.21.15.16:0/332359757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c10dbc0 msgr2=0x7f2b5c19fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.371+0000 7f2b5a59c640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c10dbc0 0x7f2b5c19fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.371+0000 7f2b5a59c640 1 -- 172.21.15.16:0/332359757 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b5c1a0260 con 0x7f2b5c104760 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.371+0000 7f2b53fff640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b5c1050b0 0x7f2b5c19a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.371+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c10dbc0 0x7f2b5c19fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.371+0000 7f2b5a59c640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b5c104760 0x7f2b5c19a2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00d770 tx=0x7f2b4c00dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.371+0000 7f2b537fe640 1 -- 172.21.15.16:0/332359757 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c017070 con 0x7f2b5c104760 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.371+0000 7f2b60ed1640 1 -- 172.21.15.16:0/332359757 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b5c1a0550 con 0x7f2b5c104760 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.371+0000 7f2b60ed1640 1 -- 172.21.15.16:0/332359757 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b5c1a6cb0 con 0x7f2b5c104760 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.371+0000 7f2b537fe640 1 -- 172.21.15.16:0/332359757 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b4c009c30 con 0x7f2b5c104760 2024-08-28T19:33:52.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.371+0000 7f2b537fe640 1 -- 172.21.15.16:0/332359757 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c003cc0 con 0x7f2b5c104760 2024-08-28T19:33:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.373+0000 7f2b67446740 1 Processor -- start 2024-08-28T19:33:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b67446740 1 -- start start 2024-08-28T19:33:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b67446740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0db772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b67446740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7a91b0 0x562d0db77800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b67446740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7b1390 0x562d0db77d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b67446740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562d0daee630 con 0x562d0d7d2070 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b67446740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d0daddbc0 con 0x562d0d7b1390 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b67446740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562d0dae1b70 con 0x562d0d7a91b0 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0db772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0db772c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42238/0 (socket says 172.21.15.16:42238) 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b5ad9d640 1 -- 172.21.15.16:0/3887910903 learned_addr learned my addr 172.21.15.16:0/3887910903 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b53fff640 1 --2- 172.21.15.16:0/3887910903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7b1390 0x562d0db77d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b537fe640 1 -- 172.21.15.16:0/332359757 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2b4c003e60 con 0x7f2b5c104760 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b5ad9d640 1 -- 172.21.15.16:0/3887910903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7a91b0 msgr2=0x562d0db77800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/3887910903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7a91b0 0x562d0db77800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b5ad9d640 1 -- 172.21.15.16:0/3887910903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7b1390 msgr2=0x562d0db77d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/3887910903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7b1390 0x562d0db77d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.374+0000 7f2b5ad9d640 1 -- 172.21.15.16:0/3887910903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d0d83d650 con 0x562d0d7d2070 2024-08-28T19:33:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.375+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/3887910903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0db772c0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f2b5402f430 tx=0x7f2b5402f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd0613a55ff6665 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.375+0000 7f2b50ff9640 1 -- 172.21.15.16:0/3887910903 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b54031650 con 0x562d0d7d2070 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.375+0000 7f2b50ff9640 1 -- 172.21.15.16:0/3887910903 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b54036040 con 0x562d0d7d2070 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.375+0000 7f2b50ff9640 1 -- 172.21.15.16:0/3887910903 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b54004620 con 0x562d0d7d2070 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.375+0000 7f2b537fe640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b480b4ca0 0x7f2b480b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.375+0000 7f2b5a59c640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b480b4ca0 0x7f2b480b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b537fe640 1 -- 172.21.15.16:0/332359757 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f2b4c0e5280 con 0x7f2b5c104760 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b67446740 1 -- 172.21.15.16:0/3887910903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 msgr2=0x562d0db772c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b67446740 1 --2- 172.21.15.16:0/3887910903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0db772c0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f2b5402f430 tx=0x7f2b5402f460 comp rx=0 tx=0).stop 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b67446740 1 -- 172.21.15.16:0/3887910903 shutdown_connections 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b67446740 1 --2- 172.21.15.16:0/3887910903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7b1390 0x562d0db77d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b67446740 1 --2- 172.21.15.16:0/3887910903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7a91b0 0x562d0db77800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b67446740 1 --2- 172.21.15.16:0/3887910903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0db772c0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b67446740 1 -- 172.21.15.16:0/3887910903 >> 172.21.15.16:0/3887910903 conn(0x562d0d7c6460 msgr2=0x562d0db751b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:52.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b5a59c640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b480b4ca0 0x7f2b480b71b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c067800 tx=0x7f2b4c003760 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b67446740 1 -- 172.21.15.16:0/3887910903 shutdown_connections 2024-08-28T19:33:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b67446740 1 -- 172.21.15.16:0/3887910903 wait complete. 2024-08-28T19:33:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.376+0000 7f2b67446740 1 -- start start 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b67446740 1 Processor -- start 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b67446740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7a91b0 0x562d0db86df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b67446740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db87330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b67446740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7d2070 0x562d0db92f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b67446740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562d0d940700 con 0x562d0d7b1390 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b67446740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d0da64700 con 0x562d0d7d2070 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b67446740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562d0db89bf0 con 0x562d0d7a91b0 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b67446740 10 client.0 _ll_register_callbacks cb 0x562d0da93dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db87330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db87330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42252/0 (socket says 172.21.15.16:42252) 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b53fff640 1 -- 172.21.15.16:0/3182688885 learned_addr learned my addr 172.21.15.16:0/3182688885 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7a91b0 0x562d0db86df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7d2070 0x562d0db92f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b53fff640 1 -- 172.21.15.16:0/3182688885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7a91b0 msgr2=0x562d0db86df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b53fff640 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7a91b0 0x562d0db86df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b53fff640 1 -- 172.21.15.16:0/3182688885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7d2070 msgr2=0x562d0db92f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b53fff640 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7d2070 0x562d0db92f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.377+0000 7f2b53fff640 1 -- 172.21.15.16:0/3182688885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d0da9d8b0 con 0x562d0d7b1390 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b53fff640 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db87330 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400be90 tx=0x7f2b440117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee92064bafa8156 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b41ffb640 1 -- 172.21.15.16:0/3182688885 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b44012650 con 0x562d0d7b1390 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b41ffb640 1 -- 172.21.15.16:0/3182688885 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f2b440127f0 con 0x562d0d7b1390 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b67446740 1 -- 172.21.15.16:0/3182688885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b54036040 con 0x562d0d7b1390 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b41ffb640 1 -- 172.21.15.16:0/3182688885 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f2b440129f0 con 0x562d0d7b1390 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b67446740 10 client.18759 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b67446740 1 -- 172.21.15.16:0/3182688885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x562d0db95a00 con 0x562d0d7b1390 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b41ffb640 1 -- 172.21.15.16:0/3182688885 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b54036040 con 0x562d0d7b1390 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b41ffb640 1 -- 172.21.15.16:0/3182688885 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7f2b4401bca0 con 0x562d0d7b1390 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b67446740 10 client.18759 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b67446740 10 client.18759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b67446740 20 client.18759 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b67446740 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d0d7909f0 0x562d0db98fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b67446740 4 client.18759 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.378+0000 7f2b67446740 1 -- 172.21.15.16:0/3182688885 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d0daf0580 con 0x562d0d7909f0 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.379+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d0d7909f0 0x562d0db98fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.379+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d0d7909f0 0x562d0db98fa0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cd48ae7388613a6 server_cookie=ea10ad8631a17cf6 in_seq=0 out_seq=0 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.379+0000 7f2b41ffb640 10 client.18759 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b41ffb640 1 -- 172.21.15.16:0/3182688885 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562d0daf0580 con 0x562d0d7909f0 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b41ffb640 10 client.18759 handle_command_reply: tid=0 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 1 client.18759 shutdown 2024-08-28T19:33:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 1 -- 172.21.15.16:0/3182688885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 msgr2=0x562d0db87330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db87330 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400be90 tx=0x7f2b440117b0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 1 -- 172.21.15.16:0/3182688885 shutdown_connections 2024-08-28T19:33:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7d2070 0x562d0db92f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db87330 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d0d7909f0 0x562d0db98fa0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 1 --2- 172.21.15.16:0/3182688885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7a91b0 0x562d0db86df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 1 -- 172.21.15.16:0/3182688885 >> 172.21.15.16:0/3182688885 conn(0x562d0d7c6460 msgr2=0x562d0db7a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b41ffb640 0 client.18759 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 1 -- 172.21.15.16:0/3182688885 shutdown_connections 2024-08-28T19:33:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 1 -- 172.21.15.16:0/3182688885 wait complete. 2024-08-28T19:33:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.380+0000 7f2b67446740 20 client.18759 trim_cache size 0 max 16384 2024-08-28T19:33:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.398+0000 7f2b67446740 1 Processor -- start 2024-08-28T19:33:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.398+0000 7f2b67446740 1 -- start start 2024-08-28T19:33:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.398+0000 7f2b67446740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7a91b0 0x562d0db75200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.398+0000 7f2b67446740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db92e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.398+0000 7f2b67446740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7d2070 0x562d0db93360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.398+0000 7f2b67446740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1aeb70 con 0x562d0d7b1390 2024-08-28T19:33:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.398+0000 7f2b67446740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1b0770 con 0x562d0d7a91b0 2024-08-28T19:33:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.398+0000 7f2b67446740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1af270 con 0x562d0d7d2070 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.398+0000 7f2b53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db92e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.398+0000 7f2b53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db92e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42254/0 (socket says 172.21.15.16:42254) 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.398+0000 7f2b53fff640 1 -- 172.21.15.16:0/2330701619 learned_addr learned my addr 172.21.15.16:0/2330701619 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/2330701619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7a91b0 0x562d0db75200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/2330701619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7d2070 0x562d0db93360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b53fff640 1 -- 172.21.15.16:0/2330701619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7d2070 msgr2=0x562d0db93360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b53fff640 1 --2- 172.21.15.16:0/2330701619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7d2070 0x562d0db93360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b53fff640 1 -- 172.21.15.16:0/2330701619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7a91b0 msgr2=0x562d0db75200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b53fff640 1 --2- 172.21.15.16:0/2330701619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7a91b0 0x562d0db75200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b53fff640 1 -- 172.21.15.16:0/2330701619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d0db7a0c0 con 0x562d0d7b1390 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/2330701619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7d2070 0x562d0db93360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:33:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b53fff640 1 --2- 172.21.15.16:0/2330701619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db92e20 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2b44001eb0 tx=0x7f2b4401a9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70685cebe5f6a5c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b43fff640 1 -- 172.21.15.16:0/2330701619 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4402d070 con 0x562d0d7b1390 2024-08-28T19:33:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b43fff640 1 -- 172.21.15.16:0/2330701619 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b44012d90 con 0x562d0d7b1390 2024-08-28T19:33:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.399+0000 7f2b43fff640 1 -- 172.21.15.16:0/2330701619 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b44005040 con 0x562d0d7b1390 2024-08-28T19:33:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.400+0000 7f2b67446740 1 -- 172.21.15.16:0/2330701619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 msgr2=0x562d0db92e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.400+0000 7f2b67446740 1 --2- 172.21.15.16:0/2330701619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db92e20 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2b44001eb0 tx=0x7f2b4401a9a0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.400+0000 7f2b67446740 1 -- 172.21.15.16:0/2330701619 shutdown_connections 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.400+0000 7f2b67446740 1 --2- 172.21.15.16:0/2330701619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7d2070 0x562d0db93360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.400+0000 7f2b67446740 1 --2- 172.21.15.16:0/2330701619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7b1390 0x562d0db92e20 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.400+0000 7f2b67446740 1 --2- 172.21.15.16:0/2330701619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7a91b0 0x562d0db75200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.400+0000 7f2b67446740 1 -- 172.21.15.16:0/2330701619 >> 172.21.15.16:0/2330701619 conn(0x562d0d7c6460 msgr2=0x562d0db74af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.400+0000 7f2b67446740 1 -- 172.21.15.16:0/2330701619 shutdown_connections 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.400+0000 7f2b67446740 1 -- 172.21.15.16:0/2330701619 wait complete. 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.400+0000 7f2b67446740 1 -- start start 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b67446740 1 Processor -- start 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b67446740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7a91b0 0x562d0db8b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b67446740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7b1390 0x562d0db8b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b67446740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0dc02960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b67446740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1b0070 con 0x562d0d7d2070 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b67446740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1af970 con 0x562d0d7a91b0 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b67446740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c1acb70 con 0x562d0d7b1390 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b67446740 10 client.0 _ll_register_callbacks cb 0x7f2b4400b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0dc02960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0dc02960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42260/0 (socket says 172.21.15.16:42260) 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b5ad9d640 1 -- 172.21.15.16:0/2548002383 learned_addr learned my addr 172.21.15.16:0/2548002383 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7a91b0 0x562d0db8b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b53fff640 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7b1390 0x562d0db8b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b5ad9d640 1 -- 172.21.15.16:0/2548002383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7b1390 msgr2=0x562d0db8b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7b1390 0x562d0db8b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b5ad9d640 1 -- 172.21.15.16:0/2548002383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7a91b0 msgr2=0x562d0db8b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:33:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.401+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7a91b0 0x562d0db8b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.402+0000 7f2b5ad9d640 1 -- 172.21.15.16:0/2548002383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d0db8be40 con 0x562d0d7d2070 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.402+0000 7f2b5ad9d640 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0dc02960 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f2b540096c0 tx=0x7f2b5402f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c21255cd7dc67b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.402+0000 7f2b41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.402+0000 7f2b41ffb640 1 -- 172.21.15.16:0/2548002383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b54031450 con 0x562d0d7d2070 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.402+0000 7f2b41ffb640 1 -- 172.21.15.16:0/2548002383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9490+0+0 (secure 0 0 0) 0x7f2b540315f0 con 0x562d0d7d2070 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.402+0000 7f2b67446740 1 -- 172.21.15.16:0/2548002383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b44012d90 con 0x562d0d7d2070 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.402+0000 7f2b41ffb640 1 -- 172.21.15.16:0/2548002383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f2b540317f0 con 0x562d0d7d2070 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.402+0000 7f2b67446740 10 client.18765 fetch_fsmap learned FSMap version 63 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.402+0000 7f2b67446740 1 -- 172.21.15.16:0/2548002383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x562d0dc03e60 con 0x562d0d7d2070 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.402+0000 7f2b41ffb640 1 -- 172.21.15.16:0/2548002383 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b44012d90 con 0x562d0d7d2070 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.402+0000 7f2b41ffb640 1 -- 172.21.15.16:0/2548002383 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 63) ==== 8394+0+0 (secure 0 0 0) 0x7f2b54051040 con 0x562d0d7d2070 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.403+0000 7f2b67446740 10 client.18765 fetch_fsmap finished waiting for FSMap version 63 2024-08-28T19:33:52.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.403+0000 7f2b67446740 10 client.18765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:33:52.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.403+0000 7f2b67446740 20 client.18765 populate_metadata read hostname 'smithi016' 2024-08-28T19:33:52.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.403+0000 7f2b67446740 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d0d765da0 0x562d0dc06310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:33:52.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.403+0000 7f2b67446740 4 client.18765 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:33:52.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.403+0000 7f2b67446740 1 -- 172.21.15.16:0/2548002383 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562d0daec020 con 0x562d0d765da0 2024-08-28T19:33:52.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.403+0000 7f2b53fff640 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d0d765da0 0x562d0dc06310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:33:52.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.403+0000 7f2b41ffb640 10 client.18765 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:52.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.403+0000 7f2b53fff640 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d0d765da0 0x562d0dc06310 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2212acb567ff9c4 server_cookie=2c51b753a06f6965 in_seq=0 out_seq=0 2024-08-28T19:33:52.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b41ffb640 1 -- 172.21.15.16:0/2548002383 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+5398 (crc 0 0 0) 0x562d0daec020 con 0x562d0d765da0 2024-08-28T19:33:52.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b41ffb640 10 client.18765 handle_command_reply: tid=0 2024-08-28T19:33:52.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 1 client.18765 shutdown 2024-08-28T19:33:52.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 1 -- 172.21.15.16:0/2548002383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 msgr2=0x562d0dc02960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:52.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0dc02960 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f2b540096c0 tx=0x7f2b5402f310 comp rx=0 tx=0).stop 2024-08-28T19:33:52.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 1 -- 172.21.15.16:0/2548002383 shutdown_connections 2024-08-28T19:33:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d0d7d2070 0x562d0dc02960 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d0d765da0 0x562d0dc06310 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b41ffb640 0 client.18765 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:33:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d0d7b1390 0x562d0db8b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 1 --2- 172.21.15.16:0/2548002383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d0d7a91b0 0x562d0db8b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 1 -- 172.21.15.16:0/2548002383 >> 172.21.15.16:0/2548002383 conn(0x562d0d7c6460 msgr2=0x562d0db782f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 1 -- 172.21.15.16:0/2548002383 shutdown_connections 2024-08-28T19:33:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 1 -- 172.21.15.16:0/2548002383 wait complete. 2024-08-28T19:33:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.404+0000 7f2b67446740 20 client.18765 trim_cache size 0 max 16384 2024-08-28T19:33:52.410 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:33:52.410 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "1fc60423-efa1-4a92-91a1-0815560f5e52", 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:33:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:33:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:33:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:33:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:33:52.421 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:33:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 -- 172.21.15.16:0/332359757 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b480b4ca0 msgr2=0x7f2b480b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b480b4ca0 0x7f2b480b71b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c067800 tx=0x7f2b4c003760 comp rx=0 tx=0).stop 2024-08-28T19:33:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 -- 172.21.15.16:0/332359757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b5c104760 msgr2=0x7f2b5c19a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:33:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b5c104760 0x7f2b5c19a2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00d770 tx=0x7f2b4c00dc30 comp rx=0 tx=0).stop 2024-08-28T19:33:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 -- 172.21.15.16:0/332359757 shutdown_connections 2024-08-28T19:33:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c10dbc0 0x7f2b5c19fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b480b4ca0 0x7f2b480b71b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b5c1050b0 0x7f2b5c19a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 --2- 172.21.15.16:0/332359757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b5c104760 0x7f2b5c19a2f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:33:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 -- 172.21.15.16:0/332359757 >> 172.21.15.16:0/332359757 conn(0x7f2b5c0794d0 msgr2=0x7f2b5c100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:33:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 -- 172.21.15.16:0/332359757 shutdown_connections 2024-08-28T19:33:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:33:52.405+0000 7f2b60ed1640 1 -- 172.21.15.16:0/332359757 wait complete. 2024-08-28T19:33:52.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 504560 3.52 MB/sec execute 572 sec latency 1959.747 ms 2024-08-28T19:33:53.245 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:52 smithi016 ceph-mon[18556]: pgmap v736: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 495 KiB/s wr, 47 op/s 2024-08-28T19:33:53.274 INFO:tasks.cephfs.filesystem:scrub status for tag:1fc60423-efa1-4a92-91a1-0815560f5e52 - {'path': '/', 'tag': '1fc60423-efa1-4a92-91a1-0815560f5e52', 'options': 'recursive,force'} 2024-08-28T19:33:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:52 smithi028 ceph-mon[21769]: pgmap v736: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 495 KiB/s wr, 47 op/s 2024-08-28T19:33:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:52 smithi115 ceph-mon[22184]: pgmap v736: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 495 KiB/s wr, 47 op/s 2024-08-28T19:33:53.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 504560 3.52 MB/sec execute 573 sec latency 2959.846 ms 2024-08-28T19:33:54.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 504627 3.51 MB/sec execute 574 sec latency 3070.113 ms 2024-08-28T19:33:55.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:54 smithi016 ceph-mon[18556]: pgmap v737: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 296 KiB/s wr, 36 op/s 2024-08-28T19:33:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:54 smithi028 ceph-mon[21769]: pgmap v737: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 296 KiB/s wr, 36 op/s 2024-08-28T19:33:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:54 smithi115 ceph-mon[22184]: pgmap v737: 129 pgs: 129 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 296 KiB/s wr, 36 op/s 2024-08-28T19:33:55.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 504776 3.51 MB/sec execute 575 sec latency 1576.170 ms 2024-08-28T19:33:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:56 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:56 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:56.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:56 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:33:56.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 506131 3.51 MB/sec execute 576 sec latency 15.968 ms 2024-08-28T19:33:57.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:57 smithi028 ceph-mon[21769]: pgmap v738: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 268 KiB/s wr, 46 op/s 2024-08-28T19:33:57.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:57 smithi115 ceph-mon[22184]: pgmap v738: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 268 KiB/s wr, 46 op/s 2024-08-28T19:33:57.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:57 smithi016 ceph-mon[18556]: pgmap v738: 129 pgs: 129 active+clean; 406 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 268 KiB/s wr, 46 op/s 2024-08-28T19:33:57.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 506183 3.50 MB/sec execute 577 sec latency 959.748 ms 2024-08-28T19:33:58.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 506183 3.50 MB/sec execute 578 sec latency 1959.857 ms 2024-08-28T19:33:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:59 smithi028 ceph-mon[21769]: pgmap v739: 129 pgs: 129 active+clean; 408 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 493 KiB/s wr, 46 op/s 2024-08-28T19:33:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:59 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:33:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:59 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:33:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:33:59 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:33:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:59 smithi115 ceph-mon[22184]: pgmap v739: 129 pgs: 129 active+clean; 408 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 493 KiB/s wr, 46 op/s 2024-08-28T19:33:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:59 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:33:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:59 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:33:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:33:59 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:33:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:59 smithi016 ceph-mon[18556]: pgmap v739: 129 pgs: 129 active+clean; 408 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 493 KiB/s wr, 46 op/s 2024-08-28T19:33:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:59 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:33:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:59 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:33:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:33:59 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:33:59.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508104 3.51 MB/sec execute 579 sec latency 2070.781 ms 2024-08-28T19:34:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:00 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:34:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:00 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:34:00.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:00 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:34:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:00 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:34:00.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:00 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:34:00.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:00 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:34:00.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508104 3.51 MB/sec execute 580 sec latency 1020.437 ms 2024-08-28T19:34:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:01 smithi028 ceph-mon[21769]: pgmap v740: 129 pgs: 129 active+clean; 408 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 771 KiB/s wr, 55 op/s 2024-08-28T19:34:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:01 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:01 smithi115 ceph-mon[22184]: pgmap v740: 129 pgs: 129 active+clean; 408 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 771 KiB/s wr, 55 op/s 2024-08-28T19:34:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:01 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:01.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:01 smithi016 ceph-mon[18556]: pgmap v740: 129 pgs: 129 active+clean; 408 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 771 KiB/s wr, 55 op/s 2024-08-28T19:34:01.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:01 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:01.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508104 3.50 MB/sec execute 581 sec latency 2020.562 ms 2024-08-28T19:34:02.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508104 3.49 MB/sec execute 582 sec latency 3020.678 ms 2024-08-28T19:34:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:03 smithi028 ceph-mon[21769]: pgmap v741: 129 pgs: 129 active+clean; 411 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 1.2 MiB/s wr, 102 op/s 2024-08-28T19:34:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:03 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:03 smithi115 ceph-mon[22184]: pgmap v741: 129 pgs: 129 active+clean; 411 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 1.2 MiB/s wr, 102 op/s 2024-08-28T19:34:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:03 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:03 smithi016 ceph-mon[18556]: pgmap v741: 129 pgs: 129 active+clean; 411 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 1.2 MiB/s wr, 102 op/s 2024-08-28T19:34:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:03 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:03.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508104 3.49 MB/sec execute 583 sec latency 4020.793 ms 2024-08-28T19:34:04.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508165 3.48 MB/sec execute 584 sec latency 4133.221 ms 2024-08-28T19:34:05.312 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:05 smithi028 ceph-mon[21769]: pgmap v742: 129 pgs: 129 active+clean; 410 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.0 MiB/s wr, 84 op/s 2024-08-28T19:34:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:05 smithi115 ceph-mon[22184]: pgmap v742: 129 pgs: 129 active+clean; 410 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.0 MiB/s wr, 84 op/s 2024-08-28T19:34:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:05 smithi016 ceph-mon[18556]: pgmap v742: 129 pgs: 129 active+clean; 410 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.0 MiB/s wr, 84 op/s 2024-08-28T19:34:05.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508165 3.48 MB/sec execute 585 sec latency 1793.291 ms 2024-08-28T19:34:06.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508165 3.47 MB/sec execute 586 sec latency 2793.380 ms 2024-08-28T19:34:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:07 smithi028 ceph-mon[21769]: pgmap v743: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-08-28T19:34:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:07 smithi115 ceph-mon[22184]: pgmap v743: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-08-28T19:34:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:07 smithi016 ceph-mon[18556]: pgmap v743: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-08-28T19:34:07.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508165 3.47 MB/sec execute 587 sec latency 3793.428 ms 2024-08-28T19:34:08.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:08 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:08.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:08 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:08.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:08 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:08.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508165 3.46 MB/sec execute 588 sec latency 4793.486 ms 2024-08-28T19:34:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:09 smithi028 ceph-mon[21769]: pgmap v744: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-08-28T19:34:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:09 smithi115 ceph-mon[22184]: pgmap v744: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-08-28T19:34:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:09 smithi016 ceph-mon[18556]: pgmap v744: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-08-28T19:34:09.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508171 3.45 MB/sec execute 589 sec latency 4915.649 ms 2024-08-28T19:34:10.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508171 3.45 MB/sec execute 590 sec latency 1877.939 ms 2024-08-28T19:34:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:11 smithi028 ceph-mon[21769]: pgmap v745: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 809 KiB/s wr, 89 op/s 2024-08-28T19:34:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:11 smithi115 ceph-mon[22184]: pgmap v745: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 809 KiB/s wr, 89 op/s 2024-08-28T19:34:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:11 smithi016 ceph-mon[18556]: pgmap v745: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 809 KiB/s wr, 89 op/s 2024-08-28T19:34:11.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 509101 3.45 MB/sec execute 591 sec latency 2027.888 ms 2024-08-28T19:34:12.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 509726 3.45 MB/sec execute 592 sec latency 464.034 ms 2024-08-28T19:34:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:13 smithi028 ceph-mon[21769]: pgmap v746: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 574 KiB/s wr, 86 op/s 2024-08-28T19:34:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:13 smithi115 ceph-mon[22184]: pgmap v746: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 574 KiB/s wr, 86 op/s 2024-08-28T19:34:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:13 smithi016 ceph-mon[18556]: pgmap v746: 129 pgs: 129 active+clean; 411 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 574 KiB/s wr, 86 op/s 2024-08-28T19:34:13.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 509726 3.44 MB/sec execute 593 sec latency 1464.149 ms 2024-08-28T19:34:14.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 509726 3.44 MB/sec execute 594 sec latency 2464.270 ms 2024-08-28T19:34:15.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:15 smithi028 ceph-mon[21769]: pgmap v747: 129 pgs: 129 active+clean; 412 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 237 KiB/s wr, 42 op/s 2024-08-28T19:34:15.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:15 smithi028 ceph-mon[21769]: osdmap e97: 12 total, 12 up, 12 in 2024-08-28T19:34:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:15 smithi115 ceph-mon[22184]: pgmap v747: 129 pgs: 129 active+clean; 412 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 237 KiB/s wr, 42 op/s 2024-08-28T19:34:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:15 smithi115 ceph-mon[22184]: osdmap e97: 12 total, 12 up, 12 in 2024-08-28T19:34:15.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:15 smithi016 ceph-mon[18556]: pgmap v747: 129 pgs: 129 active+clean; 412 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 237 KiB/s wr, 42 op/s 2024-08-28T19:34:15.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:15 smithi016 ceph-mon[18556]: osdmap e97: 12 total, 12 up, 12 in 2024-08-28T19:34:15.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 509726 3.43 MB/sec execute 595 sec latency 3464.337 ms 2024-08-28T19:34:16.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 510530 3.43 MB/sec execute 596 sec latency 4086.696 ms 2024-08-28T19:34:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:17 smithi028 ceph-mon[21769]: pgmap v749: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 758 KiB/s wr, 71 op/s 2024-08-28T19:34:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:17 smithi115 ceph-mon[22184]: pgmap v749: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 758 KiB/s wr, 71 op/s 2024-08-28T19:34:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:17 smithi016 ceph-mon[18556]: pgmap v749: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 758 KiB/s wr, 71 op/s 2024-08-28T19:34:17.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 511650 3.44 MB/sec execute 597 sec latency 536.789 ms 2024-08-28T19:34:18.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 511650 3.43 MB/sec execute 598 sec latency 1536.874 ms 2024-08-28T19:34:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:19 smithi115 ceph-mon[22184]: pgmap v750: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 844 KiB/s wr, 65 op/s 2024-08-28T19:34:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:19 smithi016 ceph-mon[18556]: pgmap v750: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 844 KiB/s wr, 65 op/s 2024-08-28T19:34:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:19 smithi028 ceph-mon[21769]: pgmap v750: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 844 KiB/s wr, 65 op/s 2024-08-28T19:34:19.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 511717 3.43 MB/sec execute 599 sec latency 1646.263 ms 2024-08-28T19:34:20.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 cleanup 600 sec 2024-08-28T19:34:21.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:21 smithi028 ceph-mon[21769]: pgmap v751: 129 pgs: 129 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 840 KiB/s wr, 64 op/s 2024-08-28T19:34:21.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:21 smithi028 ceph-mon[21769]: osdmap e98: 12 total, 12 up, 12 in 2024-08-28T19:34:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:21 smithi115 ceph-mon[22184]: pgmap v751: 129 pgs: 129 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 840 KiB/s wr, 64 op/s 2024-08-28T19:34:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:21 smithi115 ceph-mon[22184]: osdmap e98: 12 total, 12 up, 12 in 2024-08-28T19:34:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:21 smithi016 ceph-mon[18556]: pgmap v751: 129 pgs: 129 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 840 KiB/s wr, 64 op/s 2024-08-28T19:34:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:21 smithi016 ceph-mon[18556]: osdmap e98: 12 total, 12 up, 12 in 2024-08-28T19:34:21.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 cleanup 601 sec 2024-08-28T19:34:22.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 cleanup 602 sec 2024-08-28T19:34:23.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:34:23.544 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:23.558 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:23 smithi028 ceph-mon[21769]: pgmap v753: 129 pgs: 129 active+clean; 418 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 97 op/s 2024-08-28T19:34:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:23 smithi115 ceph-mon[22184]: pgmap v753: 129 pgs: 129 active+clean; 418 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 97 op/s 2024-08-28T19:34:23.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:23 smithi016 ceph-mon[18556]: pgmap v753: 129 pgs: 129 active+clean; 418 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 97 op/s 2024-08-28T19:34:23.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 cleanup 603 sec 2024-08-28T19:34:24.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.029+0000 7f2306636640 1 -- 172.21.15.16:0/3941189121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f230010dbd0 msgr2=0x7f23001100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.029+0000 7f2306636640 1 --2- 172.21.15.16:0/3941189121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f230010dbd0 0x7f23001100c0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009ca0 tx=0x7f22f802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.029+0000 7f2306636640 1 -- 172.21.15.16:0/3941189121 shutdown_connections 2024-08-28T19:34:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.029+0000 7f2306636640 1 --2- 172.21.15.16:0/3941189121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f230010dbd0 0x7f23001100c0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.029+0000 7f2306636640 1 --2- 172.21.15.16:0/3941189121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23001050c0 0x7f230010d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.029+0000 7f2306636640 1 --2- 172.21.15.16:0/3941189121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2300104770 0x7f2300104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.029+0000 7f2306636640 1 -- 172.21.15.16:0/3941189121 >> 172.21.15.16:0/3941189121 conn(0x7f23000794e0 msgr2=0x7f23000798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.029+0000 7f2306636640 1 -- 172.21.15.16:0/3941189121 shutdown_connections 2024-08-28T19:34:24.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.029+0000 7f2306636640 1 -- 172.21.15.16:0/3941189121 wait complete. 2024-08-28T19:34:24.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.030+0000 7f2306636640 1 Processor -- start 2024-08-28T19:34:24.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.030+0000 7f2306636640 1 -- start start 2024-08-28T19:34:24.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.030+0000 7f2306636640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300104770 0x7f230019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.030+0000 7f2306636640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23001050c0 0x7f230019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.030+0000 7f2306636640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230010dbd0 0x7f230019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.030+0000 7f2306636640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2300113830 con 0x7f2300104770 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.030+0000 7f2306636640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23001136b0 con 0x7f230010dbd0 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.030+0000 7f2306636640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f23001139b0 con 0x7f23001050c0 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.030+0000 7f22fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300104770 0x7f230019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.030+0000 7f22fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300104770 0x7f230019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49148/0 (socket says 172.21.15.16:49148) 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.030+0000 7f22fffff640 1 -- 172.21.15.16:0/2776924543 learned_addr learned my addr 172.21.15.16:0/2776924543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f22ff7fe640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23001050c0 0x7f230019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f2304bd3640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230010dbd0 0x7f230019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f22fffff640 1 -- 172.21.15.16:0/2776924543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23001050c0 msgr2=0x7f230019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f22fffff640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23001050c0 0x7f230019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f22fffff640 1 -- 172.21.15.16:0/2776924543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230010dbd0 msgr2=0x7f230019faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f22fffff640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230010dbd0 0x7f230019faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f22fffff640 1 -- 172.21.15.16:0/2776924543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23001a01d0 con 0x7f2300104770 2024-08-28T19:34:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f2304bd3640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230010dbd0 0x7f230019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:34:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f22fffff640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300104770 0x7f230019a260 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f22f000d950 tx=0x7f22f000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f22fd7fa640 1 -- 172.21.15.16:0/2776924543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f0017020 con 0x7f2300104770 2024-08-28T19:34:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f2306636640 1 -- 172.21.15.16:0/2776924543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23001a04c0 con 0x7f2300104770 2024-08-28T19:34:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f22fd7fa640 1 -- 172.21.15.16:0/2776924543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22f0009c30 con 0x7f2300104770 2024-08-28T19:34:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f22fd7fa640 1 -- 172.21.15.16:0/2776924543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f000f600 con 0x7f2300104770 2024-08-28T19:34:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.031+0000 7f2306636640 1 -- 172.21.15.16:0/2776924543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23001a6c20 con 0x7f2300104770 2024-08-28T19:34:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.034+0000 7f230cbab740 1 Processor -- start 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.034+0000 7f230cbab740 1 -- start start 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.034+0000 7f22fd7fa640 1 -- 172.21.15.16:0/2776924543 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f22f001a020 con 0x7f2300104770 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.034+0000 7f230cbab740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc703070 0x55e1bcaa82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.034+0000 7f230cbab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6da1b0 0x55e1bcaa8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.034+0000 7f230cbab740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6e2390 0x55e1bcaa8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.034+0000 7f230cbab740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e1bca1f630 con 0x55e1bc6e2390 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.034+0000 7f230cbab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e1bca0ebc0 con 0x55e1bc6da1b0 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.034+0000 7f230cbab740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e1bca12b70 con 0x55e1bc703070 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6e2390 0x55e1bcaa8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6e2390 0x55e1bcaa8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49154/0 (socket says 172.21.15.16:49154) 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6da1b0 0x55e1bcaa8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22ff7fe640 1 -- 172.21.15.16:0/1298812597 learned_addr learned my addr 172.21.15.16:0/1298812597 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f2304bd3640 1 --2- 172.21.15.16:0/1298812597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc703070 0x55e1bcaa82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22ff7fe640 1 -- 172.21.15.16:0/1298812597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc703070 msgr2=0x55e1bcaa82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22ff7fe640 1 --2- 172.21.15.16:0/1298812597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc703070 0x55e1bcaa82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22ff7fe640 1 -- 172.21.15.16:0/1298812597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6da1b0 msgr2=0x55e1bcaa8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22ff7fe640 1 --2- 172.21.15.16:0/1298812597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6da1b0 0x55e1bcaa8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22ff7fe640 1 -- 172.21.15.16:0/1298812597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1bc76e650 con 0x55e1bc6e2390 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f2304bd3640 1 --2- 172.21.15.16:0/1298812597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc703070 0x55e1bcaa82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22ff7fe640 1 --2- 172.21.15.16:0/1298812597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6e2390 0x55e1bcaa8d40 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f22f400bbb0 tx=0x7f22f40107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22da290f36133262 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22fd7fa640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22e80b4ca0 0x7f22e80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22e67fc640 1 -- 172.21.15.16:0/1298812597 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4004030 con 0x55e1bc6e2390 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f2304bd3640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22e80b4ca0 0x7f22e80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22e67fc640 1 -- 172.21.15.16:0/1298812597 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22f4005020 con 0x55e1bc6e2390 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.035+0000 7f22e67fc640 1 -- 172.21.15.16:0/1298812597 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4004b80 con 0x55e1bc6e2390 2024-08-28T19:34:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.036+0000 7f22fd7fa640 1 -- 172.21.15.16:0/2776924543 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f22f00e58e0 con 0x7f2300104770 2024-08-28T19:34:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.036+0000 7f230cbab740 1 -- 172.21.15.16:0/1298812597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6e2390 msgr2=0x55e1bcaa8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.036+0000 7f2304bd3640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22e80b4ca0 0x7f22e80b71b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f22f802f500 tx=0x7f22f802f310 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.036+0000 7f230cbab740 1 --2- 172.21.15.16:0/1298812597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6e2390 0x55e1bcaa8d40 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f22f400bbb0 tx=0x7f22f40107a0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.036+0000 7f230cbab740 1 -- 172.21.15.16:0/1298812597 shutdown_connections 2024-08-28T19:34:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.036+0000 7f230cbab740 1 --2- 172.21.15.16:0/1298812597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6e2390 0x55e1bcaa8d40 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.036+0000 7f230cbab740 1 --2- 172.21.15.16:0/1298812597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6da1b0 0x55e1bcaa8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.036+0000 7f230cbab740 1 --2- 172.21.15.16:0/1298812597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc703070 0x55e1bcaa82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.036+0000 7f230cbab740 1 -- 172.21.15.16:0/1298812597 >> 172.21.15.16:0/1298812597 conn(0x55e1bc6f7460 msgr2=0x55e1bcaa61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.037+0000 7f230cbab740 1 -- 172.21.15.16:0/1298812597 shutdown_connections 2024-08-28T19:34:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.037+0000 7f230cbab740 1 -- 172.21.15.16:0/1298812597 wait complete. 2024-08-28T19:34:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.037+0000 7f230cbab740 1 -- start start 2024-08-28T19:34:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.037+0000 7f230cbab740 1 Processor -- start 2024-08-28T19:34:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f230cbab740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6da1b0 0x55e1bcab3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f230cbab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcab3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f230cbab740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc703070 0x55e1bcabfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f230cbab740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e1bc871700 con 0x55e1bc703070 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f230cbab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e1bc995700 con 0x55e1bc6e2390 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f230cbab740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e1bcab6aa0 con 0x55e1bc6da1b0 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f230cbab740 10 client.0 _ll_register_callbacks cb 0x55e1bc9c4dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f2304bd3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc703070 0x55e1bcabfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f2304bd3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc703070 0x55e1bcabfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49158/0 (socket says 172.21.15.16:49158) 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f2304bd3640 1 -- 172.21.15.16:0/1058866235 learned_addr learned my addr 172.21.15.16:0/1058866235 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f22fffff640 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6da1b0 0x55e1bcab3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f22ff7fe640 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcab3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f2304bd3640 1 -- 172.21.15.16:0/1058866235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6da1b0 msgr2=0x55e1bcab3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f2304bd3640 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6da1b0 0x55e1bcab3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f2304bd3640 1 -- 172.21.15.16:0/1058866235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 msgr2=0x55e1bcab3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f2304bd3640 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcab3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.038+0000 7f2304bd3640 1 -- 172.21.15.16:0/1058866235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1bc9ce8b0 con 0x55e1bc703070 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.039+0000 7f2304bd3640 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc703070 0x55e1bcabfd90 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f22f8031680 tx=0x7f22f80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d3686add8e09b1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.039+0000 7f22df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.039+0000 7f22df7fe640 1 -- 172.21.15.16:0/1058866235 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f804e070 con 0x55e1bc703070 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.039+0000 7f22df7fe640 1 -- 172.21.15.16:0/1058866235 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f22f80045d0 con 0x55e1bc703070 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.039+0000 7f230cbab740 1 -- 172.21.15.16:0/1058866235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22f4005020 con 0x55e1bc703070 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.039+0000 7f22df7fe640 1 -- 172.21.15.16:0/1058866235 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f22f80047d0 con 0x55e1bc703070 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.039+0000 7f230cbab740 10 client.18777 fetch_fsmap learned FSMap version 67 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.039+0000 7f230cbab740 1 -- 172.21.15.16:0/1058866235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55e1bcac27c0 con 0x55e1bc703070 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.039+0000 7f22df7fe640 1 -- 172.21.15.16:0/1058866235 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22f4005020 con 0x55e1bc703070 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.040+0000 7f22df7fe640 1 -- 172.21.15.16:0/1058866235 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 67) ==== 8402+0+0 (secure 0 0 0) 0x7f22f8043040 con 0x55e1bc703070 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.040+0000 7f230cbab740 10 client.18777 fetch_fsmap finished waiting for FSMap version 67 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.040+0000 7f230cbab740 10 client.18777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:34:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.040+0000 7f230cbab740 20 client.18777 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.040+0000 7f230cbab740 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e1bc6c19f0 0x55e1bcac5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.040+0000 7f230cbab740 4 client.18777 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:34:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.040+0000 7f230cbab740 1 -- 172.21.15.16:0/1058866235 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1bca21580 con 0x55e1bc6c19f0 2024-08-28T19:34:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.040+0000 7f22fffff640 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e1bc6c19f0 0x55e1bcac5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.040+0000 7f22df7fe640 10 client.18777 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.040+0000 7f22fffff640 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e1bc6c19f0 0x55e1bcac5cd0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37055b6625529ade server_cookie=17558ecc86ce4633 in_seq=0 out_seq=0 2024-08-28T19:34:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f22df7fe640 1 -- 172.21.15.16:0/1058866235 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e1bca21580 con 0x55e1bc6c19f0 2024-08-28T19:34:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f22df7fe640 10 client.18777 handle_command_reply: tid=0 2024-08-28T19:34:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f230cbab740 1 client.18777 shutdown 2024-08-28T19:34:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f230cbab740 1 -- 172.21.15.16:0/1058866235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc703070 msgr2=0x55e1bcabfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f230cbab740 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc703070 0x55e1bcabfd90 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f22f8031680 tx=0x7f22f80095c0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f230cbab740 1 -- 172.21.15.16:0/1058866235 shutdown_connections 2024-08-28T19:34:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f230cbab740 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc703070 0x55e1bcabfd90 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f230cbab740 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcab3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f230cbab740 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e1bc6c19f0 0x55e1bcac5cd0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f22df7fe640 0 client.18777 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f230cbab740 1 --2- 172.21.15.16:0/1058866235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6da1b0 0x55e1bcab3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.041+0000 7f230cbab740 1 -- 172.21.15.16:0/1058866235 >> 172.21.15.16:0/1058866235 conn(0x55e1bc6f7460 msgr2=0x55e1bcaab450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.042+0000 7f230cbab740 1 -- 172.21.15.16:0/1058866235 shutdown_connections 2024-08-28T19:34:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.042+0000 7f230cbab740 1 -- 172.21.15.16:0/1058866235 wait complete. 2024-08-28T19:34:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.042+0000 7f230cbab740 20 client.18777 trim_cache size 0 max 16384 2024-08-28T19:34:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.059+0000 7f230cbab740 1 Processor -- start 2024-08-28T19:34:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.059+0000 7f230cbab740 1 -- start start 2024-08-28T19:34:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.059+0000 7f230cbab740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6c19f0 0x55e1bcabfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.059+0000 7f230cbab740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcaa5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.059+0000 7f230cbab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcaa6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.059+0000 7f230cbab740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23001b0d60 con 0x55e1bc6da1b0 2024-08-28T19:34:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.059+0000 7f230cbab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23001b2990 con 0x55e1bc6e2390 2024-08-28T19:34:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.059+0000 7f230cbab740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f23001b1430 con 0x55e1bc6c19f0 2024-08-28T19:34:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.059+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcaa5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.059+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcaa5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49166/0 (socket says 172.21.15.16:49166) 2024-08-28T19:34:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.059+0000 7f22ff7fe640 1 -- 172.21.15.16:0/2464035892 learned_addr learned my addr 172.21.15.16:0/2464035892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f2304bd3640 1 --2- 172.21.15.16:0/2464035892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcaa6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f22fffff640 1 --2- 172.21.15.16:0/2464035892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6c19f0 0x55e1bcabfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f22ff7fe640 1 -- 172.21.15.16:0/2464035892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6c19f0 msgr2=0x55e1bcabfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f22ff7fe640 1 --2- 172.21.15.16:0/2464035892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6c19f0 0x55e1bcabfc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f22ff7fe640 1 -- 172.21.15.16:0/2464035892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 msgr2=0x55e1bcaa6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f22ff7fe640 1 --2- 172.21.15.16:0/2464035892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcaa6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f22ff7fe640 1 -- 172.21.15.16:0/2464035892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1bcab4ce0 con 0x55e1bc6da1b0 2024-08-28T19:34:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f2304bd3640 1 --2- 172.21.15.16:0/2464035892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcaa6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:34:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f22ff7fe640 1 --2- 172.21.15.16:0/2464035892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcaa5ad0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f22f400bbb0 tx=0x7f22f400b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7f79c80c2e422d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f22e5ffb640 1 -- 172.21.15.16:0/2464035892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f40052b0 con 0x55e1bc6da1b0 2024-08-28T19:34:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f22e5ffb640 1 -- 172.21.15.16:0/2464035892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22f4005450 con 0x55e1bc6da1b0 2024-08-28T19:34:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.060+0000 7f22e5ffb640 1 -- 172.21.15.16:0/2464035892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4011770 con 0x55e1bc6da1b0 2024-08-28T19:34:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.061+0000 7f230cbab740 1 -- 172.21.15.16:0/2464035892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 msgr2=0x55e1bcaa5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.061+0000 7f230cbab740 1 --2- 172.21.15.16:0/2464035892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcaa5ad0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f22f400bbb0 tx=0x7f22f400b740 comp rx=0 tx=0).stop 2024-08-28T19:34:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.061+0000 7f230cbab740 1 -- 172.21.15.16:0/2464035892 shutdown_connections 2024-08-28T19:34:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.061+0000 7f230cbab740 1 --2- 172.21.15.16:0/2464035892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcaa6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.061+0000 7f230cbab740 1 --2- 172.21.15.16:0/2464035892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcaa5ad0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.061+0000 7f230cbab740 1 --2- 172.21.15.16:0/2464035892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6c19f0 0x55e1bcabfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.061+0000 7f230cbab740 1 -- 172.21.15.16:0/2464035892 >> 172.21.15.16:0/2464035892 conn(0x55e1bc6f7460 msgr2=0x55e1bcab22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.061+0000 7f230cbab740 1 -- 172.21.15.16:0/2464035892 shutdown_connections 2024-08-28T19:34:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.061+0000 7f230cbab740 1 -- 172.21.15.16:0/2464035892 wait complete. 2024-08-28T19:34:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.061+0000 7f230cbab740 1 -- start start 2024-08-28T19:34:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f230cbab740 1 Processor -- start 2024-08-28T19:34:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f230cbab740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6c19f0 0x55e1bcab8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f230cbab740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcab8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f230cbab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcb2f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f230cbab740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23001b2290 con 0x55e1bc6da1b0 2024-08-28T19:34:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f230cbab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23001b1bc0 con 0x55e1bc6e2390 2024-08-28T19:34:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f230cbab740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f23001aed60 con 0x55e1bc6c19f0 2024-08-28T19:34:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f230cbab740 10 client.0 _ll_register_callbacks cb 0x7f22f8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcab8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcab8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49172/0 (socket says 172.21.15.16:49172) 2024-08-28T19:34:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f22ff7fe640 1 -- 172.21.15.16:0/3882586302 learned_addr learned my addr 172.21.15.16:0/3882586302 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f22fffff640 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6c19f0 0x55e1bcab8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f2304bd3640 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcb2f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f22ff7fe640 1 -- 172.21.15.16:0/3882586302 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6c19f0 msgr2=0x55e1bcab8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f22ff7fe640 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6c19f0 0x55e1bcab8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f22ff7fe640 1 -- 172.21.15.16:0/3882586302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 msgr2=0x55e1bcb2f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f22ff7fe640 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcb2f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.062+0000 7f22ff7fe640 1 -- 172.21.15.16:0/3882586302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1bcb2fc00 con 0x55e1bc6da1b0 2024-08-28T19:34:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f22ff7fe640 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcab8970 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f22f4011010 tx=0x7f22f4003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a25aa3f75387f4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f22df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f22df7fe640 1 -- 172.21.15.16:0/3882586302 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f40035f0 con 0x55e1bc6da1b0 2024-08-28T19:34:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f22df7fe640 1 -- 172.21.15.16:0/3882586302 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f22f4003790 con 0x55e1bc6da1b0 2024-08-28T19:34:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f230cbab740 1 -- 172.21.15.16:0/3882586302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22f4005450 con 0x55e1bc6da1b0 2024-08-28T19:34:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f22df7fe640 1 -- 172.21.15.16:0/3882586302 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f22f4003990 con 0x55e1bc6da1b0 2024-08-28T19:34:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f230cbab740 10 client.18783 fetch_fsmap learned FSMap version 67 2024-08-28T19:34:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f230cbab740 1 -- 172.21.15.16:0/3882586302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55e1bcb30e90 con 0x55e1bc6da1b0 2024-08-28T19:34:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f22df7fe640 1 -- 172.21.15.16:0/3882586302 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22f4005450 con 0x55e1bc6da1b0 2024-08-28T19:34:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f22df7fe640 1 -- 172.21.15.16:0/3882586302 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 67) ==== 8402+0+0 (secure 0 0 0) 0x7f22f4026430 con 0x55e1bc6da1b0 2024-08-28T19:34:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f230cbab740 10 client.18783 fetch_fsmap finished waiting for FSMap version 67 2024-08-28T19:34:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f230cbab740 10 client.18783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:34:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f230cbab740 20 client.18783 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f230cbab740 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e1bc703070 0x55e1bcb333f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f230cbab740 4 client.18783 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.063+0000 7f230cbab740 1 -- 172.21.15.16:0/3882586302 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e1bca1d020 con 0x55e1bc703070 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.064+0000 7f22fffff640 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e1bc703070 0x55e1bcb333f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.064+0000 7f22fffff640 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e1bc703070 0x55e1bcb333f0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e7eaa2181aeafe4 server_cookie=9a46eb2ee2de449 in_seq=0 out_seq=0 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.064+0000 7f22df7fe640 10 client.18783 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.064+0000 7f22df7fe640 1 -- 172.21.15.16:0/3882586302 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+5202 (crc 0 0 0) 0x55e1bca1d020 con 0x55e1bc703070 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.064+0000 7f22df7fe640 10 client.18783 handle_command_reply: tid=0 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.064+0000 7f230cbab740 1 client.18783 shutdown 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f230cbab740 1 -- 172.21.15.16:0/3882586302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 msgr2=0x55e1bcab8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f230cbab740 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcab8970 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f22f4011010 tx=0x7f22f4003eb0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f230cbab740 1 -- 172.21.15.16:0/3882586302 shutdown_connections 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f230cbab740 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1bc6e2390 0x55e1bcb2f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f230cbab740 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e1bc6da1b0 0x55e1bcab8970 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f230cbab740 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e1bc703070 0x55e1bcb333f0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f230cbab740 1 --2- 172.21.15.16:0/3882586302 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e1bc6c19f0 0x55e1bcab8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f230cbab740 1 -- 172.21.15.16:0/3882586302 >> 172.21.15.16:0/3882586302 conn(0x55e1bc6f7460 msgr2=0x55e1bcab4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f22df7fe640 0 client.18783 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f230cbab740 1 -- 172.21.15.16:0/3882586302 shutdown_connections 2024-08-28T19:34:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f230cbab740 1 -- 172.21.15.16:0/3882586302 wait complete. 2024-08-28T19:34:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f230cbab740 20 client.18783 trim_cache size 0 max 16384 2024-08-28T19:34:24.074 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:34:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:34:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:34:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:34:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:34:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:34:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:34:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:34:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:34:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:34:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:34:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:34:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:34:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:24.087 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:34:24.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f2306636640 1 -- 172.21.15.16:0/2776924543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22e80b4ca0 msgr2=0x7f22e80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:24.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f2306636640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22e80b4ca0 0x7f22e80b71b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f22f802f500 tx=0x7f22f802f310 comp rx=0 tx=0).stop 2024-08-28T19:34:24.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f2306636640 1 -- 172.21.15.16:0/2776924543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300104770 msgr2=0x7f230019a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:24.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.065+0000 7f2306636640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300104770 0x7f230019a260 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f22f000d950 tx=0x7f22f000de10 comp rx=0 tx=0).stop 2024-08-28T19:34:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.066+0000 7f2306636640 1 -- 172.21.15.16:0/2776924543 shutdown_connections 2024-08-28T19:34:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.066+0000 7f2306636640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22e80b4ca0 0x7f22e80b71b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.066+0000 7f2306636640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230010dbd0 0x7f230019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.066+0000 7f2306636640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23001050c0 0x7f230019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.066+0000 7f2306636640 1 --2- 172.21.15.16:0/2776924543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300104770 0x7f230019a260 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.066+0000 7f2306636640 1 -- 172.21.15.16:0/2776924543 >> 172.21.15.16:0/2776924543 conn(0x7f23000794e0 msgr2=0x7f23000ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.066+0000 7f2306636640 1 -- 172.21.15.16:0/2776924543 shutdown_connections 2024-08-28T19:34:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:24.066+0000 7f2306636640 1 -- 172.21.15.16:0/2776924543 wait complete. 2024-08-28T19:34:24.328 INFO:tasks.workunit.client.0.smithi016.stdout: 0 cleanup 603 sec 2024-08-28T19:34:24.329 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-08-28T19:34:24.329 INFO:tasks.workunit.client.0.smithi016.stdout: Operation Count AvgLat MaxLat 2024-08-28T19:34:24.329 INFO:tasks.workunit.client.0.smithi016.stdout: ---------------------------------------- 2024-08-28T19:34:24.329 INFO:tasks.workunit.client.0.smithi016.stdout: NTCreateX 65190 4.102 29162.024 2024-08-28T19:34:24.329 INFO:tasks.workunit.client.0.smithi016.stdout: Close 47899 0.002 0.073 2024-08-28T19:34:24.329 INFO:tasks.workunit.client.0.smithi016.stdout: Rename 2761 62.872 16984.223 2024-08-28T19:34:24.329 INFO:tasks.workunit.client.0.smithi016.stdout: Unlink 13160 1.513 71.485 2024-08-28T19:34:24.329 INFO:tasks.workunit.client.0.smithi016.stdout: Deltree 2 111.924 158.605 2024-08-28T19:34:24.329 INFO:tasks.workunit.client.0.smithi016.stdout: Mkdir 1 0.006 0.006 2024-08-28T19:34:24.329 INFO:tasks.workunit.client.0.smithi016.stdout: Qpathinfo 59076 1.217 4915.641 2024-08-28T19:34:24.330 INFO:tasks.workunit.client.0.smithi016.stdout: Qfileinfo 10366 0.001 0.026 2024-08-28T19:34:24.330 INFO:tasks.workunit.client.0.smithi016.stdout: Qfsinfo 10856 0.184 38.689 2024-08-28T19:34:24.330 INFO:tasks.workunit.client.0.smithi016.stdout: Sfileinfo 5304 0.111 10.335 2024-08-28T19:34:24.330 INFO:tasks.workunit.client.0.smithi016.stdout: Find 22880 0.720 7055.358 2024-08-28T19:34:24.330 INFO:tasks.workunit.client.0.smithi016.stdout: WriteX 32570 1.014 28018.745 2024-08-28T19:34:24.330 INFO:tasks.workunit.client.0.smithi016.stdout: ReadX 102362 0.050 37.641 2024-08-28T19:34:24.330 INFO:tasks.workunit.client.0.smithi016.stdout: LockX 212 0.587 1.505 2024-08-28T19:34:24.330 INFO:tasks.workunit.client.0.smithi016.stdout: UnlockX 212 0.542 1.383 2024-08-28T19:34:24.330 INFO:tasks.workunit.client.0.smithi016.stdout: Flush 4569 3.536 58.675 2024-08-28T19:34:24.330 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-08-28T19:34:24.330 INFO:tasks.workunit.client.0.smithi016.stdout:Throughput 3.42775 MB/sec 1 clients 1 procs max_latency=29162.029 ms 2024-08-28T19:34:24.331 INFO:tasks.workunit.client.0.smithi016.stderr:+ dbench 10 2024-08-28T19:34:24.331 INFO:tasks.workunit.client.0.smithi016.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-08-28T19:34:24.331 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-08-28T19:34:24.331 INFO:tasks.workunit.client.0.smithi016.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-08-28T19:34:24.331 INFO:tasks.workunit.client.0.smithi016.stdout:8 of 10 processes prepared for launch 0 sec 2024-08-28T19:34:24.430 INFO:tasks.workunit.client.0.smithi016.stdout:10 of 10 processes prepared for launch 0 sec 2024-08-28T19:34:24.430 INFO:tasks.workunit.client.0.smithi016.stdout:releasing clients 2024-08-28T19:34:24.824 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:34:24.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:34:25.087 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:25.448 INFO:tasks.workunit.client.0.smithi016.stdout: 10 392 137.15 MB/sec warmup 1 sec latency 119.809 ms 2024-08-28T19:34:25.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:25 smithi016 ceph-mon[18556]: pgmap v754: 129 pgs: 129 active+clean; 418 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 935 KiB/s wr, 81 op/s 2024-08-28T19:34:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:25 smithi028 ceph-mon[21769]: pgmap v754: 129 pgs: 129 active+clean; 418 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 935 KiB/s wr, 81 op/s 2024-08-28T19:34:25.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:25 smithi115 ceph-mon[22184]: pgmap v754: 129 pgs: 129 active+clean; 418 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 935 KiB/s wr, 81 op/s 2024-08-28T19:34:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.767+0000 7fd41dc0f640 1 -- 172.21.15.16:0/1923993879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4181045e0 msgr2=0x7fd4181049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.767+0000 7fd41dc0f640 1 --2- 172.21.15.16:0/1923993879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4181045e0 0x7fd4181049f0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fd408009900 tx=0x7fd40802a410 comp rx=0 tx=0).stop 2024-08-28T19:34:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.767+0000 7fd41dc0f640 1 -- 172.21.15.16:0/1923993879 shutdown_connections 2024-08-28T19:34:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.767+0000 7fd41dc0f640 1 --2- 172.21.15.16:0/1923993879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd41810db90 0x7fd4181100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.767+0000 7fd41dc0f640 1 --2- 172.21.15.16:0/1923993879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd418105010 0x7fd41810d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.767+0000 7fd41dc0f640 1 --2- 172.21.15.16:0/1923993879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4181045e0 0x7fd4181049f0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.767+0000 7fd41dc0f640 1 -- 172.21.15.16:0/1923993879 >> 172.21.15.16:0/1923993879 conn(0x7fd4180794d0 msgr2=0x7fd4180798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.767+0000 7fd41dc0f640 1 -- 172.21.15.16:0/1923993879 shutdown_connections 2024-08-28T19:34:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.767+0000 7fd41dc0f640 1 -- 172.21.15.16:0/1923993879 wait complete. 2024-08-28T19:34:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.768+0000 7fd41dc0f640 1 Processor -- start 2024-08-28T19:34:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.768+0000 7fd41dc0f640 1 -- start start 2024-08-28T19:34:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd41dc0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4181045e0 0x7fd4181024f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd41dc0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd418105010 0x7fd418102a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd41dc0f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd41810db90 0x7fd418102f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd41dc0f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd418113830 con 0x7fd4181045e0 2024-08-28T19:34:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd41dc0f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4181136b0 con 0x7fd418105010 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd41dc0f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd4181139b0 con 0x7fd41810db90 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd4177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4181045e0 0x7fd4181024f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd4177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4181045e0 0x7fd4181024f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51078/0 (socket says 172.21.15.16:51078) 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd4177fe640 1 -- 172.21.15.16:0/2763840566 learned_addr learned my addr 172.21.15.16:0/2763840566 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd416ffd640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd418105010 0x7fd418102a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd417fff640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd41810db90 0x7fd418102f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd4177fe640 1 -- 172.21.15.16:0/2763840566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd41810db90 msgr2=0x7fd418102f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd4177fe640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd41810db90 0x7fd418102f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd4177fe640 1 -- 172.21.15.16:0/2763840566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd418105010 msgr2=0x7fd418102a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd4177fe640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd418105010 0x7fd418102a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd4177fe640 1 -- 172.21.15.16:0/2763840566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4181a85a0 con 0x7fd4181045e0 2024-08-28T19:34:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.769+0000 7fd4177fe640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4181045e0 0x7fd4181024f0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fd408032fd0 tx=0x7fd408031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.770+0000 7fd414ff9640 1 -- 172.21.15.16:0/2763840566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40803f030 con 0x7fd4181045e0 2024-08-28T19:34:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.770+0000 7fd41dc0f640 1 -- 172.21.15.16:0/2763840566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4181a8830 con 0x7fd4181045e0 2024-08-28T19:34:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.770+0000 7fd414ff9640 1 -- 172.21.15.16:0/2763840566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd408004490 con 0x7fd4181045e0 2024-08-28T19:34:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.770+0000 7fd414ff9640 1 -- 172.21.15.16:0/2763840566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd408035070 con 0x7fd4181045e0 2024-08-28T19:34:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.770+0000 7fd41dc0f640 1 -- 172.21.15.16:0/2763840566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4181a8d60 con 0x7fd4181045e0 2024-08-28T19:34:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.771+0000 7fd41dc0f640 1 -- 172.21.15.16:0/2763840566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4181049f0 con 0x7fd4181045e0 2024-08-28T19:34:25.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.774+0000 7fd414ff9640 1 -- 172.21.15.16:0/2763840566 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fd408042030 con 0x7fd4181045e0 2024-08-28T19:34:25.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.774+0000 7fd414ff9640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4000b4d20 0x7fd4000b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:25.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.774+0000 7fd416ffd640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4000b4d20 0x7fd4000b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:25.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.775+0000 7fd414ff9640 1 -- 172.21.15.16:0/2763840566 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7fd408105fa0 con 0x7fd4181045e0 2024-08-28T19:34:25.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.775+0000 7fd416ffd640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4000b4d20 0x7fd4000b7230 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd418106300 tx=0x7fd40c005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:25.779+0000 7fd414ff9640 1 -- 172.21.15.16:0/2763840566 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fd4080cc370 con 0x7fd4181045e0 2024-08-28T19:34:26.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.083+0000 7fd41dc0f640 1 -- 172.21.15.16:0/2763840566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4181a9120 con 0x7fd4181045e0 2024-08-28T19:34:26.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.084+0000 7fd414ff9640 1 -- 172.21.15.16:0/2763840566 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+11991 (secure 0 0 0) 0x7fd4080d0b30 con 0x7fd4181045e0 2024-08-28T19:34:26.085 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:34:26.085 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":68,"btime":"2024-08-28T19:34:25:292785+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":68,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:34:24.543088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:34:26.085 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 68 2024-08-28T19:34:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 -- 172.21.15.16:0/2763840566 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4000b4d20 msgr2=0x7fd4000b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4000b4d20 0x7fd4000b7230 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd418106300 tx=0x7fd40c005f90 comp rx=0 tx=0).stop 2024-08-28T19:34:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 -- 172.21.15.16:0/2763840566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4181045e0 msgr2=0x7fd4181024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4181045e0 0x7fd4181024f0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fd408032fd0 tx=0x7fd408031a20 comp rx=0 tx=0).stop 2024-08-28T19:34:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 -- 172.21.15.16:0/2763840566 shutdown_connections 2024-08-28T19:34:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd41810db90 0x7fd418102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd4000b4d20 0x7fd4000b7230 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd418105010 0x7fd418102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 --2- 172.21.15.16:0/2763840566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4181045e0 0x7fd4181024f0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 -- 172.21.15.16:0/2763840566 >> 172.21.15.16:0/2763840566 conn(0x7fd4180794d0 msgr2=0x7fd418106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 -- 172.21.15.16:0/2763840566 shutdown_connections 2024-08-28T19:34:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:26.086+0000 7fd41dc0f640 1 -- 172.21.15.16:0/2763840566 wait complete. 2024-08-28T19:34:26.448 INFO:tasks.workunit.client.0.smithi016.stdout: 10 853 101.43 MB/sec warmup 2 sec latency 1114.228 ms 2024-08-28T19:34:26.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:26 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:26.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:26 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2763840566' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:34:26.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:26 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:26.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:26 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2763840566' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:34:26.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:26 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:26.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:26 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2763840566' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:34:26.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:34:27.165 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:27.460 INFO:tasks.workunit.client.0.smithi016.stdout: 10 1448 80.57 MB/sec warmup 3 sec latency 2126.228 ms 2024-08-28T19:34:27.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:27 smithi016 ceph-mon[18556]: pgmap v755: 129 pgs: 129 active+clean; 419 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 505 KiB/s wr, 57 op/s 2024-08-28T19:34:27.593 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.7 2024-08-28T19:34:27.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:34:27.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:27 smithi115 ceph-mon[22184]: pgmap v755: 129 pgs: 129 active+clean; 419 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 505 KiB/s wr, 57 op/s 2024-08-28T19:34:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.673+0000 7f3aa952f640 1 -- 172.21.15.16:0/3314255609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4108140 msgr2=0x7f3aa4108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.673+0000 7f3aa952f640 1 --2- 172.21.15.16:0/3314255609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4108140 0x7f3aa4108550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c009900 tx=0x7f3a8c02a410 comp rx=0 tx=0).stop 2024-08-28T19:34:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.673+0000 7f3aa952f640 1 -- 172.21.15.16:0/3314255609 shutdown_connections 2024-08-28T19:34:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.673+0000 7f3aa952f640 1 --2- 172.21.15.16:0/3314255609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa4108a90 0x7f3aa410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.673+0000 7f3aa952f640 1 --2- 172.21.15.16:0/3314255609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4108140 0x7f3aa4108550 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.673+0000 7f3aa952f640 1 --2- 172.21.15.16:0/3314255609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3aa4102800 0x7f3aa4102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.673+0000 7f3aa952f640 1 -- 172.21.15.16:0/3314255609 >> 172.21.15.16:0/3314255609 conn(0x7f3aa40794d0 msgr2=0x7f3aa40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.673+0000 7f3aa952f640 1 -- 172.21.15.16:0/3314255609 shutdown_connections 2024-08-28T19:34:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.673+0000 7f3aa952f640 1 -- 172.21.15.16:0/3314255609 wait complete. 2024-08-28T19:34:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.674+0000 7f3aa952f640 1 Processor -- start 2024-08-28T19:34:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.674+0000 7f3aa952f640 1 -- start start 2024-08-28T19:34:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.674+0000 7f3aa952f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4102800 0x7f3aa419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.674+0000 7f3aa952f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa4108140 0x7f3aa419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.674+0000 7f3aa952f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3aa4108a90 0x7f3aa419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.674+0000 7f3aa952f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa4112730 con 0x7f3aa4108140 2024-08-28T19:34:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.674+0000 7f3aa952f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa41125b0 con 0x7f3aa4102800 2024-08-28T19:34:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.674+0000 7f3aa952f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa41128b0 con 0x7f3aa4108a90 2024-08-28T19:34:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa4108140 0x7f3aa419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4102800 0x7f3aa419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4102800 0x7f3aa419a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48382/0 (socket says 172.21.15.16:48382) 2024-08-28T19:34:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa2ffd640 1 -- 172.21.15.16:0/3739862390 learned_addr learned my addr 172.21.15.16:0/3739862390 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa4108140 0x7f3aa419a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51090/0 (socket says 172.21.15.16:51090) 2024-08-28T19:34:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa37fe640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3aa4108a90 0x7f3aa419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa2ffd640 1 -- 172.21.15.16:0/3739862390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3aa4108a90 msgr2=0x7f3aa419fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa2ffd640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3aa4108a90 0x7f3aa419fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa2ffd640 1 -- 172.21.15.16:0/3739862390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa4108140 msgr2=0x7f3aa419a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa2ffd640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa4108140 0x7f3aa419a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa2ffd640 1 -- 172.21.15.16:0/3739862390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa41a01e0 con 0x7f3aa4102800 2024-08-28T19:34:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa27fc640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa4108140 0x7f3aa419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:34:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa2ffd640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4102800 0x7f3aa419a270 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400e9f0 tx=0x7f3a9400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3a9bfff640 1 -- 172.21.15.16:0/3739862390 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9400cd80 con 0x7f3aa4102800 2024-08-28T19:34:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3aa952f640 1 -- 172.21.15.16:0/3739862390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa41a04d0 con 0x7f3aa4102800 2024-08-28T19:34:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.675+0000 7f3a9bfff640 1 -- 172.21.15.16:0/3739862390 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a940054d0 con 0x7f3aa4102800 2024-08-28T19:34:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.676+0000 7f3a9bfff640 1 -- 172.21.15.16:0/3739862390 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a94010690 con 0x7f3aa4102800 2024-08-28T19:34:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.676+0000 7f3aa952f640 1 -- 172.21.15.16:0/3739862390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa41a6c30 con 0x7f3aa4102800 2024-08-28T19:34:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.678+0000 7f3a9bfff640 1 -- 172.21.15.16:0/3739862390 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3a94010830 con 0x7f3aa4102800 2024-08-28T19:34:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.678+0000 7f3aafaa4740 1 Processor -- start 2024-08-28T19:34:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.678+0000 7f3aafaa4740 1 -- start start 2024-08-28T19:34:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae7fd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 0x5638ae7fd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae437390 0x5638ae7fdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5638ae774680 con 0x5638ae42f1b0 2024-08-28T19:34:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638ae763c10 con 0x5638ae458070 2024-08-28T19:34:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5638ae767bc0 con 0x5638ae437390 2024-08-28T19:34:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aa2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 0x5638ae7fd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aa37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae7fd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aa37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae7fd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48386/0 (socket says 172.21.15.16:48386) 2024-08-28T19:34:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aa2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 0x5638ae7fd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51098/0 (socket says 172.21.15.16:51098) 2024-08-28T19:34:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aa37fe640 1 -- 172.21.15.16:0/2076443943 learned_addr learned my addr 172.21.15.16:0/2076443943 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aa27fc640 1 --2- 172.21.15.16:0/2076443943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae437390 0x5638ae7fdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3a9bfff640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3a900b4ca0 0x7f3a900b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aa37fe640 1 -- 172.21.15.16:0/2076443943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae437390 msgr2=0x5638ae7fdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aa37fe640 1 --2- 172.21.15.16:0/2076443943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae437390 0x5638ae7fdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aa37fe640 1 -- 172.21.15.16:0/2076443943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 msgr2=0x5638ae7fd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aa37fe640 1 --2- 172.21.15.16:0/2076443943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 0x5638ae7fd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.679+0000 7f3aa37fe640 1 -- 172.21.15.16:0/2076443943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638ae4c3650 con 0x5638ae458070 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.680+0000 7f3a9bfff640 1 -- 172.21.15.16:0/3739862390 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f3a940e5d10 con 0x7f3aa4102800 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.680+0000 7f3aa37fe640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3a900b4ca0 0x7f3a900b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.680+0000 7f3aa37fe640 1 --2- 172.21.15.16:0/2076443943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae7fd310 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009f50 tx=0x7f3a9c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94eec996be30eef6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.680+0000 7f3a997fa640 1 -- 172.21.15.16:0/2076443943 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c004250 con 0x5638ae458070 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.680+0000 7f3a997fa640 1 -- 172.21.15.16:0/2076443943 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a9c00fe50 con 0x5638ae458070 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.680+0000 7f3a997fa640 1 -- 172.21.15.16:0/2076443943 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c01ba80 con 0x5638ae458070 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.681+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2076443943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 msgr2=0x5638ae7fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.681+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/2076443943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae7fd310 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009f50 tx=0x7f3a9c0118d0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.681+0000 7f3aa37fe640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3a900b4ca0 0x7f3a900b71b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c012fd0 tx=0x7f3a9c021000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.681+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2076443943 shutdown_connections 2024-08-28T19:34:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.681+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/2076443943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae437390 0x5638ae7fdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.681+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/2076443943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 0x5638ae7fd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.681+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/2076443943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae7fd310 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.681+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2076443943 >> 172.21.15.16:0/2076443943 conn(0x5638ae44c460 msgr2=0x5638ae7fb200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.681+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2076443943 shutdown_connections 2024-08-28T19:34:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.681+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2076443943 wait complete. 2024-08-28T19:34:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.682+0000 7f3aafaa4740 1 -- start start 2024-08-28T19:34:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.682+0000 7f3aafaa4740 1 Processor -- start 2024-08-28T19:34:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.682+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae42f1b0 0x5638ae808ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.682+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 0x5638ae809000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.682+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae458070 0x5638ae814de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.682+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5638ae5c6700 con 0x5638ae437390 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.682+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638ae6ea750 con 0x5638ae42f1b0 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.682+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5638ae80baf0 con 0x5638ae458070 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.682+0000 7f3aafaa4740 10 client.0 _ll_register_callbacks cb 0x5638ae719e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.682+0000 7f3aa27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 0x5638ae809000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.682+0000 7f3aa27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 0x5638ae809000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51102/0 (socket says 172.21.15.16:51102) 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.683+0000 7f3aa27fc640 1 -- 172.21.15.16:0/79045184 learned_addr learned my addr 172.21.15.16:0/79045184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.683+0000 7f3aa2ffd640 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae42f1b0 0x5638ae808ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.683+0000 7f3aa2ffd640 1 -- 172.21.15.16:0/79045184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae458070 msgr2=0x5638ae814de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.683+0000 7f3aa2ffd640 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae458070 0x5638ae814de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.683+0000 7f3aa2ffd640 1 -- 172.21.15.16:0/79045184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 msgr2=0x5638ae809000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.683+0000 7f3aa2ffd640 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 0x5638ae809000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.683+0000 7f3aa2ffd640 1 -- 172.21.15.16:0/79045184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638ae723900 con 0x5638ae42f1b0 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.683+0000 7f3aa2ffd640 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae42f1b0 0x5638ae808ac0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3a940e3090 tx=0x7f3a94009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfbdefdadea7dc4c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.683+0000 7f3a8a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.683+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/79045184 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a940ab7a0 con 0x5638ae42f1b0 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.684+0000 7f3aafaa4740 1 -- 172.21.15.16:0/79045184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a9c00fe50 con 0x5638ae42f1b0 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.684+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/79045184 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f3a940b4ec0 con 0x5638ae42f1b0 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.684+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/79045184 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f3a940e3520 con 0x5638ae42f1b0 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.684+0000 7f3aafaa4740 10 client.28235 fetch_fsmap learned FSMap version 69 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.684+0000 7f3aafaa4740 1 -- 172.21.15.16:0/79045184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x5638ae81bd60 con 0x5638ae42f1b0 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.684+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/79045184 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a9c00fe50 con 0x5638ae42f1b0 2024-08-28T19:34:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.684+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/79045184 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 69) ==== 8410+0+0 (secure 0 0 0) 0x7f3a94097170 con 0x5638ae42f1b0 2024-08-28T19:34:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.684+0000 7f3aafaa4740 10 client.28235 fetch_fsmap finished waiting for FSMap version 69 2024-08-28T19:34:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.684+0000 7f3aafaa4740 10 client.28235 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:34:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.685+0000 7f3aafaa4740 20 client.28235 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.685+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5638ae4169f0 0x5638ae81f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.685+0000 7f3aafaa4740 4 client.28235 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:34:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.685+0000 7f3aafaa4740 1 -- 172.21.15.16:0/79045184 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638ae7765d0 con 0x5638ae4169f0 2024-08-28T19:34:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.685+0000 7f3aa27fc640 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5638ae4169f0 0x5638ae81f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.685+0000 7f3aa27fc640 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5638ae4169f0 0x5638ae81f350 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6d2179d7ad0b4ad server_cookie=b1749d17f81b7427 in_seq=0 out_seq=0 2024-08-28T19:34:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.685+0000 7f3a8a7fc640 10 client.28235 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:34:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.686+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/79045184 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5638ae7765d0 con 0x5638ae4169f0 2024-08-28T19:34:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.686+0000 7f3a8a7fc640 10 client.28235 handle_command_reply: tid=0 2024-08-28T19:34:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.686+0000 7f3aafaa4740 1 client.28235 shutdown 2024-08-28T19:34:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.686+0000 7f3aafaa4740 1 -- 172.21.15.16:0/79045184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae42f1b0 msgr2=0x5638ae808ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.686+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae42f1b0 0x5638ae808ac0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3a940e3090 tx=0x7f3a94009310 comp rx=0 tx=0).stop 2024-08-28T19:34:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.686+0000 7f3aafaa4740 1 -- 172.21.15.16:0/79045184 shutdown_connections 2024-08-28T19:34:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.686+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5638ae4169f0 0x5638ae81f350 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.686+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae458070 0x5638ae814de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.686+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 0x5638ae809000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.686+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/79045184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae42f1b0 0x5638ae808ac0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.686+0000 7f3aafaa4740 1 -- 172.21.15.16:0/79045184 >> 172.21.15.16:0/79045184 conn(0x5638ae44c460 msgr2=0x5638ae8004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.687+0000 7f3a8a7fc640 0 client.28235 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:34:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.687+0000 7f3aafaa4740 1 -- 172.21.15.16:0/79045184 shutdown_connections 2024-08-28T19:34:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.687+0000 7f3aafaa4740 1 -- 172.21.15.16:0/79045184 wait complete. 2024-08-28T19:34:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.687+0000 7f3aafaa4740 20 client.28235 trim_cache size 0 max 16384 2024-08-28T19:34:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.701+0000 7f3aafaa4740 1 Processor -- start 2024-08-28T19:34:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.701+0000 7f3aafaa4740 1 -- start start 2024-08-28T19:34:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae42f1b0 0x5638ae7fb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 0x5638ae800410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae800950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa41aeb90 con 0x5638ae437390 2024-08-28T19:34:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa41b0820 con 0x5638ae458070 2024-08-28T19:34:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa41af290 con 0x5638ae42f1b0 2024-08-28T19:34:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aa27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 0x5638ae800410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aa2ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae42f1b0 0x5638ae7fb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aa27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 0x5638ae800410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51112/0 (socket says 172.21.15.16:51112) 2024-08-28T19:34:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aa27fc640 1 -- 172.21.15.16:0/4083260844 learned_addr learned my addr 172.21.15.16:0/4083260844 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aa37fe640 1 --2- 172.21.15.16:0/4083260844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae800950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aa27fc640 1 -- 172.21.15.16:0/4083260844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae42f1b0 msgr2=0x5638ae7fb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aa27fc640 1 --2- 172.21.15.16:0/4083260844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae42f1b0 0x5638ae7fb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aa27fc640 1 -- 172.21.15.16:0/4083260844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 msgr2=0x5638ae800950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aa27fc640 1 --2- 172.21.15.16:0/4083260844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae800950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aa27fc640 1 -- 172.21.15.16:0/4083260844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638ae7ff830 con 0x5638ae437390 2024-08-28T19:34:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.702+0000 7f3aa2ffd640 1 --2- 172.21.15.16:0/4083260844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae42f1b0 0x5638ae7fb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:34:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.703+0000 7f3aa27fc640 1 --2- 172.21.15.16:0/4083260844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 0x5638ae800410 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c031440 tx=0x7f3a8c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37152159bc62417 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.703+0000 7f3a98ff9640 1 -- 172.21.15.16:0/4083260844 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c035040 con 0x5638ae437390 2024-08-28T19:34:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.703+0000 7f3a98ff9640 1 -- 172.21.15.16:0/4083260844 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a8c004770 con 0x5638ae437390 2024-08-28T19:34:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.703+0000 7f3a98ff9640 1 -- 172.21.15.16:0/4083260844 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c003830 con 0x5638ae437390 2024-08-28T19:34:27.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.704+0000 7f3aafaa4740 1 -- 172.21.15.16:0/4083260844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 msgr2=0x5638ae800410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:27.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.704+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/4083260844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 0x5638ae800410 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c031440 tx=0x7f3a8c004610 comp rx=0 tx=0).stop 2024-08-28T19:34:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.704+0000 7f3aafaa4740 1 -- 172.21.15.16:0/4083260844 shutdown_connections 2024-08-28T19:34:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.704+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/4083260844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae800950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.704+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/4083260844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae437390 0x5638ae800410 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.704+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/4083260844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae42f1b0 0x5638ae7fb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.704+0000 7f3aafaa4740 1 -- 172.21.15.16:0/4083260844 >> 172.21.15.16:0/4083260844 conn(0x5638ae44c460 msgr2=0x5638ae7fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.704+0000 7f3aafaa4740 1 -- 172.21.15.16:0/4083260844 shutdown_connections 2024-08-28T19:34:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.704+0000 7f3aafaa4740 1 -- 172.21.15.16:0/4083260844 wait complete. 2024-08-28T19:34:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aafaa4740 1 -- start start 2024-08-28T19:34:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aafaa4740 1 Processor -- start 2024-08-28T19:34:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 0x5638ae8116e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae437390 0x5638ae80ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aafaa4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae80f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa41b0120 con 0x5638ae42f1b0 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa41afa20 con 0x5638ae458070 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aafaa4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa41acb90 con 0x5638ae437390 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aafaa4740 10 client.0 _ll_register_callbacks cb 0x7f3a940e9540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aa2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 0x5638ae8116e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aa2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 0x5638ae8116e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51120/0 (socket says 172.21.15.16:51120) 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aa2ffd640 1 -- 172.21.15.16:0/2846113260 learned_addr learned my addr 172.21.15.16:0/2846113260 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.705+0000 7f3aa37fe640 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae80f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.706+0000 7f3aa27fc640 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae437390 0x5638ae80ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.706+0000 7f3aa2ffd640 1 -- 172.21.15.16:0/2846113260 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae437390 msgr2=0x5638ae80ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.706+0000 7f3aa2ffd640 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae437390 0x5638ae80ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.706+0000 7f3aa2ffd640 1 -- 172.21.15.16:0/2846113260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 msgr2=0x5638ae80f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.706+0000 7f3aa2ffd640 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae80f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.706+0000 7f3aa2ffd640 1 -- 172.21.15.16:0/2846113260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638ae888d60 con 0x5638ae42f1b0 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.706+0000 7f3aa37fe640 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae80f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.706+0000 7f3aa2ffd640 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 0x5638ae8116e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f3a940a80b0 tx=0x7f3a940b54d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=775c99fa63917497 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.706+0000 7f3a8a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.706+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/2846113260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a940ab7a0 con 0x5638ae42f1b0 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.707+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/2846113260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f3a940ab4a0 con 0x5638ae42f1b0 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.707+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2846113260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a8c004770 con 0x5638ae42f1b0 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.707+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/2846113260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f3a940e9330 con 0x5638ae42f1b0 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.707+0000 7f3aafaa4740 10 client.18807 fetch_fsmap learned FSMap version 69 2024-08-28T19:34:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.707+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2846113260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x5638ae889e50 con 0x5638ae42f1b0 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.707+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/2846113260 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a8c004770 con 0x5638ae42f1b0 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.707+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/2846113260 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 69) ==== 8410+0+0 (secure 0 0 0) 0x7f3a940a79a0 con 0x5638ae42f1b0 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.707+0000 7f3aafaa4740 10 client.18807 fetch_fsmap finished waiting for FSMap version 69 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.707+0000 7f3aafaa4740 10 client.18807 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.707+0000 7f3aafaa4740 20 client.18807 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.707+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5638ae3ebda0 0x5638ae88c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.708+0000 7f3aafaa4740 4 client.18807 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.708+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2846113260 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5638ae772070 con 0x5638ae3ebda0 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.708+0000 7f3aa27fc640 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5638ae3ebda0 0x5638ae88c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.708+0000 7f3a8a7fc640 10 client.18807 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.708+0000 7f3aa27fc640 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5638ae3ebda0 0x5638ae88c500 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9eee4d90e2960d7b server_cookie=da8216e53b557f77 in_seq=0 out_seq=0 2024-08-28T19:34:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.708+0000 7f3a8a7fc640 1 -- 172.21.15.16:0/2846113260 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5638ae772070 con 0x5638ae3ebda0 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.708+0000 7f3a8a7fc640 10 client.18807 handle_command_reply: tid=0 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.708+0000 7f3aafaa4740 1 client.18807 shutdown 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.708+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2846113260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 msgr2=0x5638ae8116e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 0x5638ae8116e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f3a940a80b0 tx=0x7f3a940b54d0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2846113260 shutdown_connections 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5638ae3ebda0 0x5638ae88c500 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3a8a7fc640 0 client.18807 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638ae458070 0x5638ae80f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5638ae437390 0x5638ae80ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aafaa4740 1 --2- 172.21.15.16:0/2846113260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638ae42f1b0 0x5638ae8116e0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2846113260 >> 172.21.15.16:0/2846113260 conn(0x5638ae44c460 msgr2=0x5638ae806f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2846113260 shutdown_connections 2024-08-28T19:34:27.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aafaa4740 1 -- 172.21.15.16:0/2846113260 wait complete. 2024-08-28T19:34:27.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aafaa4740 20 client.18807 trim_cache size 0 max 16384 2024-08-28T19:34:27.716 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:34:27.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aa952f640 1 -- 172.21.15.16:0/3739862390 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3a900b4ca0 msgr2=0x7f3a900b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:27.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aa952f640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3a900b4ca0 0x7f3a900b71b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c012fd0 tx=0x7f3a9c021000 comp rx=0 tx=0).stop 2024-08-28T19:34:27.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aa952f640 1 -- 172.21.15.16:0/3739862390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4102800 msgr2=0x7f3aa419a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:27.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.709+0000 7f3aa952f640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4102800 0x7f3aa419a270 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400e9f0 tx=0x7f3a9400edf0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.710+0000 7f3aa952f640 1 -- 172.21.15.16:0/3739862390 shutdown_connections 2024-08-28T19:34:27.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.710+0000 7f3aa952f640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3aa4108a90 0x7f3aa419fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.710+0000 7f3aa952f640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa4108140 0x7f3aa419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.710+0000 7f3aa952f640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3a900b4ca0 0x7f3a900b71b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.710+0000 7f3aa952f640 1 --2- 172.21.15.16:0/3739862390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4102800 0x7f3aa419a270 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:27.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.710+0000 7f3aa952f640 1 -- 172.21.15.16:0/3739862390 >> 172.21.15.16:0/3739862390 conn(0x7f3aa40794d0 msgr2=0x7f3aa41016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:27.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.710+0000 7f3aa952f640 1 -- 172.21.15.16:0/3739862390 shutdown_connections 2024-08-28T19:34:27.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:27.710+0000 7f3aa952f640 1 -- 172.21.15.16:0/3739862390 wait complete. 2024-08-28T19:34:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:27 smithi028 ceph-mon[21769]: pgmap v755: 129 pgs: 129 active+clean; 419 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 505 KiB/s wr, 57 op/s 2024-08-28T19:34:28.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:34:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: 10 2229 65.27 MB/sec warmup 4 sec latency 3126.277 ms 2024-08-28T19:34:28.501 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:28 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:28.624 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:28 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:28 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:29.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.116+0000 7efd35c2a640 1 -- 172.21.15.16:0/1870709229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd30109ba0 msgr2=0x7efd301103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:29.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.116+0000 7efd35c2a640 1 --2- 172.21.15.16:0/1870709229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd30109ba0 0x7efd301103d0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7efd28009900 tx=0x7efd2802a410 comp rx=0 tx=0).stop 2024-08-28T19:34:29.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.116+0000 7efd35c2a640 1 -- 172.21.15.16:0/1870709229 shutdown_connections 2024-08-28T19:34:29.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.116+0000 7efd35c2a640 1 --2- 172.21.15.16:0/1870709229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd30109ba0 0x7efd301103d0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:29.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.116+0000 7efd35c2a640 1 --2- 172.21.15.16:0/1870709229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd30109250 0x7efd30109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:29.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.116+0000 7efd35c2a640 1 --2- 172.21.15.16:0/1870709229 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd30101d10 0x7efd30102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:29.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.116+0000 7efd35c2a640 1 -- 172.21.15.16:0/1870709229 >> 172.21.15.16:0/1870709229 conn(0x7efd30077a00 msgr2=0x7efd30077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:29.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.116+0000 7efd35c2a640 1 -- 172.21.15.16:0/1870709229 shutdown_connections 2024-08-28T19:34:29.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.116+0000 7efd35c2a640 1 -- 172.21.15.16:0/1870709229 wait complete. 2024-08-28T19:34:29.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd35c2a640 1 Processor -- start 2024-08-28T19:34:29.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd35c2a640 1 -- start start 2024-08-28T19:34:29.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd35c2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd30101d10 0x7efd3019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:29.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd35c2a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd30109250 0x7efd3019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:29.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd35c2a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd30109ba0 0x7efd3019fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:29.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd35c2a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd30113860 con 0x7efd30109250 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd35c2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd301136e0 con 0x7efd30101d10 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd35c2a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efd301139e0 con 0x7efd30109ba0 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd26ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd30109250 0x7efd3019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd2f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd30101d10 0x7efd3019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd26ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd30109250 0x7efd3019a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51142/0 (socket says 172.21.15.16:51142) 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd26ffd640 1 -- 172.21.15.16:0/334393216 learned_addr learned my addr 172.21.15.16:0/334393216 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.117+0000 7efd2ffff640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd30109ba0 0x7efd3019fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.118+0000 7efd26ffd640 1 -- 172.21.15.16:0/334393216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd30109ba0 msgr2=0x7efd3019fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.118+0000 7efd26ffd640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd30109ba0 0x7efd3019fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.118+0000 7efd26ffd640 1 -- 172.21.15.16:0/334393216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd30101d10 msgr2=0x7efd3019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.118+0000 7efd26ffd640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd30101d10 0x7efd3019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.118+0000 7efd26ffd640 1 -- 172.21.15.16:0/334393216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd301a0140 con 0x7efd30109250 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.118+0000 7efd26ffd640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd30109250 0x7efd3019a830 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7efd1800e9f0 tx=0x7efd1800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.118+0000 7efd2d7fa640 1 -- 172.21.15.16:0/334393216 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1800ccc0 con 0x7efd30109250 2024-08-28T19:34:29.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.118+0000 7efd2d7fa640 1 -- 172.21.15.16:0/334393216 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd180108e0 con 0x7efd30109250 2024-08-28T19:34:29.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.118+0000 7efd35c2a640 1 -- 172.21.15.16:0/334393216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd301a0430 con 0x7efd30109250 2024-08-28T19:34:29.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.118+0000 7efd2d7fa640 1 -- 172.21.15.16:0/334393216 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd18018860 con 0x7efd30109250 2024-08-28T19:34:29.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.118+0000 7efd35c2a640 1 -- 172.21.15.16:0/334393216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd301a6c50 con 0x7efd30109250 2024-08-28T19:34:29.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.119+0000 7efd35c2a640 1 -- 172.21.15.16:0/334393216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd30102120 con 0x7efd30109250 2024-08-28T19:34:29.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.121+0000 7efd2d7fa640 1 -- 172.21.15.16:0/334393216 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7efd18010430 con 0x7efd30109250 2024-08-28T19:34:29.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.121+0000 7efd2d7fa640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd1c0b4cd0 0x7efd1c0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:29.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.121+0000 7efd2f7fe640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd1c0b4cd0 0x7efd1c0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:29.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.122+0000 7efd2d7fa640 1 -- 172.21.15.16:0/334393216 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7efd180e5b20 con 0x7efd30109250 2024-08-28T19:34:29.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.122+0000 7efd2f7fe640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd1c0b4cd0 0x7efd1c0b71e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efd20007ec0 tx=0x7efd200075b0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:29.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.126+0000 7efd2d7fa640 1 -- 172.21.15.16:0/334393216 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7efd180abef0 con 0x7efd30109250 2024-08-28T19:34:29.211 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:29.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.373+0000 7efd35c2a640 1 -- 172.21.15.16:0/334393216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd3010b990 con 0x7efd30109250 2024-08-28T19:34:29.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.373+0000 7efd2d7fa640 1 -- 172.21.15.16:0/334393216 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+11994 (secure 0 0 0) 0x7efd180b06b0 con 0x7efd30109250 2024-08-28T19:34:29.375 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:34:29.375 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"btime":"2024-08-28T19:34:29:315288+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":70,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:34:29.087294+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:34:29.375 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 70 2024-08-28T19:34:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 -- 172.21.15.16:0/334393216 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd1c0b4cd0 msgr2=0x7efd1c0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd1c0b4cd0 0x7efd1c0b71e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efd20007ec0 tx=0x7efd200075b0 comp rx=0 tx=0).stop 2024-08-28T19:34:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 -- 172.21.15.16:0/334393216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd30109250 msgr2=0x7efd3019a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd30109250 0x7efd3019a830 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7efd1800e9f0 tx=0x7efd1800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:34:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 -- 172.21.15.16:0/334393216 shutdown_connections 2024-08-28T19:34:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd30109ba0 0x7efd3019fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd30109250 0x7efd3019a830 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd1c0b4cd0 0x7efd1c0b71e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 --2- 172.21.15.16:0/334393216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd30101d10 0x7efd3019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 -- 172.21.15.16:0/334393216 >> 172.21.15.16:0/334393216 conn(0x7efd30077a00 msgr2=0x7efd301072e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 -- 172.21.15.16:0/334393216 shutdown_connections 2024-08-28T19:34:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:29.376+0000 7efd35c2a640 1 -- 172.21.15.16:0/334393216 wait complete. 2024-08-28T19:34:29.460 INFO:tasks.workunit.client.0.smithi016.stdout: 10 2915 63.30 MB/sec warmup 5 sec latency 4126.318 ms 2024-08-28T19:34:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:29 smithi115 ceph-mon[22184]: pgmap v756: 129 pgs: 129 active+clean; 420 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 500 KiB/s wr, 53 op/s 2024-08-28T19:34:29.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:29 smithi016 ceph-mon[18556]: pgmap v756: 129 pgs: 129 active+clean; 420 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 500 KiB/s wr, 53 op/s 2024-08-28T19:34:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:29 smithi028 ceph-mon[21769]: pgmap v756: 129 pgs: 129 active+clean; 420 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 500 KiB/s wr, 53 op/s 2024-08-28T19:34:30.151 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 145.7 --await-for 45.0' 2024-08-28T19:34:30.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 145.7 --await-for 45.0 2024-08-28T19:34:30.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:30 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:30.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:30 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/334393216' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:34:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.420+0000 7fdc2e079640 1 -- 172.21.15.16:0/3808742632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc28109280 msgr2=0x7fdc28109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.420+0000 7fdc2e079640 1 --2- 172.21.15.16:0/3808742632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc28109280 0x7fdc28109690 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009900 tx=0x7fdc1c02a410 comp rx=0 tx=0).stop 2024-08-28T19:34:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.420+0000 7fdc2e079640 1 -- 172.21.15.16:0/3808742632 shutdown_connections 2024-08-28T19:34:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.420+0000 7fdc2e079640 1 --2- 172.21.15.16:0/3808742632 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc28109bd0 0x7fdc28110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.420+0000 7fdc2e079640 1 --2- 172.21.15.16:0/3808742632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc28109280 0x7fdc28109690 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.420+0000 7fdc2e079640 1 --2- 172.21.15.16:0/3808742632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc28103940 0x7fdc28103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.420+0000 7fdc2e079640 1 -- 172.21.15.16:0/3808742632 >> 172.21.15.16:0/3808742632 conn(0x7fdc280ff3d0 msgr2=0x7fdc28101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.420+0000 7fdc2e079640 1 -- 172.21.15.16:0/3808742632 shutdown_connections 2024-08-28T19:34:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.420+0000 7fdc2e079640 1 -- 172.21.15.16:0/3808742632 wait complete. 2024-08-28T19:34:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.421+0000 7fdc2e079640 1 Processor -- start 2024-08-28T19:34:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.421+0000 7fdc2e079640 1 -- start start 2024-08-28T19:34:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc2e079640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc28103940 0x7fdc2819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc2e079640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc28109280 0x7fdc2819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc2e079640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc28109bd0 0x7fdc2819fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc2e079640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdc28113890 con 0x7fdc28103940 2024-08-28T19:34:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc28103940 0x7fdc2819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc2e079640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc28113710 con 0x7fdc28109bd0 2024-08-28T19:34:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc2e079640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdc28113a10 con 0x7fdc28109280 2024-08-28T19:34:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc28103940 0x7fdc2819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51146/0 (socket says 172.21.15.16:51146) 2024-08-28T19:34:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc277fe640 1 -- 172.21.15.16:0/737666879 learned_addr learned my addr 172.21.15.16:0/737666879 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc26ffd640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc28109280 0x7fdc2819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc27fff640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc28109bd0 0x7fdc2819fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc277fe640 1 -- 172.21.15.16:0/737666879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc28109280 msgr2=0x7fdc2819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc277fe640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc28109280 0x7fdc2819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc277fe640 1 -- 172.21.15.16:0/737666879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc28109bd0 msgr2=0x7fdc2819fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc277fe640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc28109bd0 0x7fdc2819fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc277fe640 1 -- 172.21.15.16:0/737666879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc281a01b0 con 0x7fdc28103940 2024-08-28T19:34:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc26ffd640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc28109280 0x7fdc2819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:34:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.422+0000 7fdc277fe640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc28103940 0x7fdc2819a240 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800e9f0 tx=0x7fdc1800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.423+0000 7fdc24ff9640 1 -- 172.21.15.16:0/737666879 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1800cd80 con 0x7fdc28103940 2024-08-28T19:34:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.423+0000 7fdc24ff9640 1 -- 172.21.15.16:0/737666879 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc180108e0 con 0x7fdc28103940 2024-08-28T19:34:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.423+0000 7fdc2e079640 1 -- 172.21.15.16:0/737666879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc281a04a0 con 0x7fdc28103940 2024-08-28T19:34:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.423+0000 7fdc24ff9640 1 -- 172.21.15.16:0/737666879 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc18018980 con 0x7fdc28103940 2024-08-28T19:34:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.423+0000 7fdc2e079640 1 -- 172.21.15.16:0/737666879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc281a6c00 con 0x7fdc28103940 2024-08-28T19:34:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.425+0000 7fdc345ee740 1 Processor -- start 2024-08-28T19:34:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc345ee740 1 -- start start 2024-08-28T19:34:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 0x55a3d971e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d971e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 0x55a3d971ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc345ee740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a3d9695680 con 0x55a3d93501b0 2024-08-28T19:34:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc345ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a3d9684c10 con 0x55a3d9358390 2024-08-28T19:34:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc345ee740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a3d9688bc0 con 0x55a3d9379070 2024-08-28T19:34:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc26ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 0x55a3d971ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc27fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 0x55a3d971e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc26ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 0x55a3d971ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48474/0 (socket says 172.21.15.16:48474) 2024-08-28T19:34:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc27fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 0x55a3d971e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:41942/0 (socket says 172.21.15.16:41942) 2024-08-28T19:34:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc26ffd640 1 -- 172.21.15.16:0/2646490415 learned_addr learned my addr 172.21.15.16:0/2646490415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc27fff640 1 -- 172.21.15.16:0/2646490415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 msgr2=0x55a3d971ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc27fff640 1 --2- 172.21.15.16:0/2646490415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 0x55a3d971ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc27fff640 1 -- 172.21.15.16:0/2646490415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 msgr2=0x55a3d971e850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:34:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc27fff640 1 --2- 172.21.15.16:0/2646490415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d971e850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.426+0000 7fdc27fff640 1 -- 172.21.15.16:0/2646490415 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3d93e4650 con 0x55a3d9379070 2024-08-28T19:34:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.427+0000 7fdc27fff640 1 --2- 172.21.15.16:0/2646490415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 0x55a3d971e310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009f30 tx=0x7fdc20010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cc20c5ebc713dc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.427+0000 7fdc0dffb640 1 -- 172.21.15.16:0/2646490415 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20016070 con 0x55a3d9379070 2024-08-28T19:34:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.427+0000 7fdc0dffb640 1 -- 172.21.15.16:0/2646490415 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc20004d20 con 0x55a3d9379070 2024-08-28T19:34:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.427+0000 7fdc0dffb640 1 -- 172.21.15.16:0/2646490415 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc200126b0 con 0x55a3d9379070 2024-08-28T19:34:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.427+0000 7fdc24ff9640 1 -- 172.21.15.16:0/737666879 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fdc18010430 con 0x7fdc28103940 2024-08-28T19:34:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.428+0000 7fdc345ee740 1 -- 172.21.15.16:0/2646490415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 msgr2=0x55a3d971e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.428+0000 7fdc345ee740 1 --2- 172.21.15.16:0/2646490415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 0x55a3d971e310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009f30 tx=0x7fdc20010800 comp rx=0 tx=0).stop 2024-08-28T19:34:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.428+0000 7fdc24ff9640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdc100b4d20 0x7fdc100b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.428+0000 7fdc345ee740 1 -- 172.21.15.16:0/2646490415 shutdown_connections 2024-08-28T19:34:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.428+0000 7fdc345ee740 1 --2- 172.21.15.16:0/2646490415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 0x55a3d971ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.428+0000 7fdc24ff9640 1 -- 172.21.15.16:0/737666879 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7fdc180e5bc0 con 0x7fdc28103940 2024-08-28T19:34:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.428+0000 7fdc345ee740 1 --2- 172.21.15.16:0/2646490415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d971e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.428+0000 7fdc345ee740 1 --2- 172.21.15.16:0/2646490415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 0x55a3d971e310 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.428+0000 7fdc345ee740 1 -- 172.21.15.16:0/2646490415 >> 172.21.15.16:0/2646490415 conn(0x55a3d936d460 msgr2=0x55a3d971c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.429+0000 7fdc277fe640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdc100b4d20 0x7fdc100b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.430+0000 7fdc277fe640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdc100b4d20 0x7fdc100b7230 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x55a3d971cf90 tx=0x7fdc1800cf20 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.430+0000 7fdc345ee740 1 -- 172.21.15.16:0/2646490415 shutdown_connections 2024-08-28T19:34:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.430+0000 7fdc345ee740 1 -- 172.21.15.16:0/2646490415 wait complete. 2024-08-28T19:34:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.430+0000 7fdc345ee740 1 -- start start 2024-08-28T19:34:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.430+0000 7fdc345ee740 1 Processor -- start 2024-08-28T19:34:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.431+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d9729a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.431+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 0x55a3d9729fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.431+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 0x55a3d9735e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.431+0000 7fdc345ee740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a3d94e7700 con 0x55a3d93501b0 2024-08-28T19:34:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.431+0000 7fdc26ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 0x55a3d9729fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.431+0000 7fdc26ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 0x55a3d9729fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48476/0 (socket says 172.21.15.16:48476) 2024-08-28T19:34:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.431+0000 7fdc26ffd640 1 -- 172.21.15.16:0/2434532827 learned_addr learned my addr 172.21.15.16:0/2434532827 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.431+0000 7fdc345ee740 1 -- 172.21.15.16:0/2434532827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a3d960b750 con 0x55a3d9358390 2024-08-28T19:34:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.432+0000 7fdc345ee740 1 -- 172.21.15.16:0/2434532827 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a3d972caf0 con 0x55a3d9379070 2024-08-28T19:34:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.432+0000 7fdc345ee740 10 client.0 _ll_register_callbacks cb 0x55a3d963ae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.432+0000 7fdc27fff640 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d9729a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.432+0000 7fdc27fff640 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 0x55a3d9735e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.432+0000 7fdc26ffd640 1 -- 172.21.15.16:0/2434532827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 msgr2=0x55a3d9735e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.432+0000 7fdc26ffd640 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 0x55a3d9735e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.432+0000 7fdc26ffd640 1 -- 172.21.15.16:0/2434532827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 msgr2=0x55a3d9729a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.432+0000 7fdc26ffd640 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d9729a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.432+0000 7fdc26ffd640 1 -- 172.21.15.16:0/2434532827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3d9644900 con 0x55a3d9358390 2024-08-28T19:34:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.432+0000 7fdc27fff640 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 0x55a3d9735e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:34:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.433+0000 7fdc26ffd640 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 0x55a3d9729fd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c032f70 tx=0x7fdc1c031740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7c5b5b3eb247c3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.433+0000 7fdc06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:34:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.433+0000 7fdc06ffd640 1 -- 172.21.15.16:0/2434532827 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c035040 con 0x55a3d9358390 2024-08-28T19:34:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.433+0000 7fdc345ee740 1 -- 172.21.15.16:0/2434532827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc20004d20 con 0x55a3d9358390 2024-08-28T19:34:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.433+0000 7fdc06ffd640 1 -- 172.21.15.16:0/2434532827 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7fdc1c004240 con 0x55a3d9358390 2024-08-28T19:34:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.433+0000 7fdc06ffd640 1 -- 172.21.15.16:0/2434532827 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7fdc1c053050 con 0x55a3d9358390 2024-08-28T19:34:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.433+0000 7fdc345ee740 10 client.28259 fetch_fsmap learned FSMap version 71 2024-08-28T19:34:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.433+0000 7fdc345ee740 1 -- 172.21.15.16:0/2434532827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55a3d97388e0 con 0x55a3d9358390 2024-08-28T19:34:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.433+0000 7fdc06ffd640 1 -- 172.21.15.16:0/2434532827 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc20004d20 con 0x55a3d9358390 2024-08-28T19:34:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.434+0000 7fdc06ffd640 1 -- 172.21.15.16:0/2434532827 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 71) ==== 8426+0+0 (secure 0 0 0) 0x7fdc1c005300 con 0x55a3d9358390 2024-08-28T19:34:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.434+0000 7fdc345ee740 10 client.28259 fetch_fsmap finished waiting for FSMap version 71 2024-08-28T19:34:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.434+0000 7fdc345ee740 10 client.28259 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:34:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.434+0000 7fdc345ee740 20 client.28259 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.434+0000 7fdc345ee740 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a3d93379f0 0x55a3d973be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.434+0000 7fdc345ee740 4 client.28259 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:34:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.434+0000 7fdc345ee740 1 -- 172.21.15.16:0/2434532827 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3d96975d0 con 0x55a3d93379f0 2024-08-28T19:34:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.434+0000 7fdc27fff640 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a3d93379f0 0x55a3d973be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.434+0000 7fdc06ffd640 10 client.28259 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:34:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.434+0000 7fdc27fff640 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a3d93379f0 0x55a3d973be30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=30dd4f315f7fee3b server_cookie=ad8c85ba130f94f9 in_seq=0 out_seq=0 2024-08-28T19:34:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.435+0000 7fdc06ffd640 1 -- 172.21.15.16:0/2434532827 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a3d96975d0 con 0x55a3d93379f0 2024-08-28T19:34:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.435+0000 7fdc06ffd640 10 client.28259 handle_command_reply: tid=0 2024-08-28T19:34:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.435+0000 7fdc345ee740 1 client.28259 shutdown 2024-08-28T19:34:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.435+0000 7fdc345ee740 1 -- 172.21.15.16:0/2434532827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 msgr2=0x55a3d9729fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.435+0000 7fdc345ee740 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 0x55a3d9729fd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c032f70 tx=0x7fdc1c031740 comp rx=0 tx=0).stop 2024-08-28T19:34:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.435+0000 7fdc345ee740 1 -- 172.21.15.16:0/2434532827 shutdown_connections 2024-08-28T19:34:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.435+0000 7fdc345ee740 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a3d93379f0 0x55a3d973be30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.435+0000 7fdc345ee740 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9379070 0x55a3d9735e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.435+0000 7fdc345ee740 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9358390 0x55a3d9729fd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.435+0000 7fdc345ee740 1 --2- 172.21.15.16:0/2434532827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d9729a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.435+0000 7fdc345ee740 1 -- 172.21.15.16:0/2434532827 >> 172.21.15.16:0/2434532827 conn(0x55a3d936d460 msgr2=0x55a3d97214a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.436+0000 7fdc06ffd640 0 client.28259 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:34:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.436+0000 7fdc345ee740 1 -- 172.21.15.16:0/2434532827 shutdown_connections 2024-08-28T19:34:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.436+0000 7fdc345ee740 1 -- 172.21.15.16:0/2434532827 wait complete. 2024-08-28T19:34:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.436+0000 7fdc345ee740 20 client.28259 trim_cache size 0 max 16384 2024-08-28T19:34:30.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.454+0000 7fdc345ee740 1 Processor -- start 2024-08-28T19:34:30.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.454+0000 7fdc345ee740 1 -- start start 2024-08-28T19:34:30.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.454+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d93501b0 0x55a3d972b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.454+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 0x55a3d9726c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.454+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d9379070 0x55a3d97271c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.454+0000 7fdc345ee740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281aeb20 con 0x55a3d9379070 2024-08-28T19:34:30.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.454+0000 7fdc345ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281b0840 con 0x55a3d93501b0 2024-08-28T19:34:30.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.454+0000 7fdc345ee740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281af220 con 0x55a3d9358390 2024-08-28T19:34:30.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.454+0000 7fdc27fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d9379070 0x55a3d97271c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.454+0000 7fdc27fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d9379070 0x55a3d97271c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51162/0 (socket says 172.21.15.16:51162) 2024-08-28T19:34:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.454+0000 7fdc27fff640 1 -- 172.21.15.16:0/3925732287 learned_addr learned my addr 172.21.15.16:0/3925732287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.455+0000 7fdc27fff640 1 --2- 172.21.15.16:0/3925732287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d93501b0 0x55a3d972b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.455+0000 7fdc26ffd640 1 --2- 172.21.15.16:0/3925732287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 0x55a3d9726c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.455+0000 7fdc27fff640 1 -- 172.21.15.16:0/3925732287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 msgr2=0x55a3d9726c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.455+0000 7fdc27fff640 1 --2- 172.21.15.16:0/3925732287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 0x55a3d9726c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.455+0000 7fdc27fff640 1 -- 172.21.15.16:0/3925732287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d93501b0 msgr2=0x55a3d972b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.455+0000 7fdc27fff640 1 --2- 172.21.15.16:0/3925732287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d93501b0 0x55a3d972b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.455+0000 7fdc27fff640 1 -- 172.21.15.16:0/3925732287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3d972c1d0 con 0x55a3d9379070 2024-08-28T19:34:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.455+0000 7fdc27fff640 1 --2- 172.21.15.16:0/3925732287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d9379070 0x55a3d97271c0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fdc20010d00 tx=0x7fdc20004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36015a964dbfedb3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:30.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.455+0000 7fdc0d7fa640 1 -- 172.21.15.16:0/3925732287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20016070 con 0x55a3d9379070 2024-08-28T19:34:30.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.455+0000 7fdc0d7fa640 1 -- 172.21.15.16:0/3925732287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc200055c0 con 0x55a3d9379070 2024-08-28T19:34:30.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.455+0000 7fdc0d7fa640 1 -- 172.21.15.16:0/3925732287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20012890 con 0x55a3d9379070 2024-08-28T19:34:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.456+0000 7fdc345ee740 1 -- 172.21.15.16:0/3925732287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d9379070 msgr2=0x55a3d97271c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.456+0000 7fdc345ee740 1 --2- 172.21.15.16:0/3925732287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d9379070 0x55a3d97271c0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fdc20010d00 tx=0x7fdc20004ab0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.456+0000 7fdc345ee740 1 -- 172.21.15.16:0/3925732287 shutdown_connections 2024-08-28T19:34:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.456+0000 7fdc345ee740 1 --2- 172.21.15.16:0/3925732287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d9379070 0x55a3d97271c0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.456+0000 7fdc345ee740 1 --2- 172.21.15.16:0/3925732287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 0x55a3d9726c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.456+0000 7fdc345ee740 1 --2- 172.21.15.16:0/3925732287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d93501b0 0x55a3d972b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.456+0000 7fdc345ee740 1 -- 172.21.15.16:0/3925732287 >> 172.21.15.16:0/3925732287 conn(0x55a3d936d460 msgr2=0x55a3d972afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.456+0000 7fdc345ee740 1 -- 172.21.15.16:0/3925732287 shutdown_connections 2024-08-28T19:34:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.457+0000 7fdc345ee740 1 -- 172.21.15.16:0/3925732287 wait complete. 2024-08-28T19:34:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.457+0000 7fdc345ee740 1 -- start start 2024-08-28T19:34:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.457+0000 7fdc345ee740 1 Processor -- start 2024-08-28T19:34:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d972fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 0x55a3d9730370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc345ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9379070 0x55a3d97308b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc345ee740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281b0140 con 0x55a3d93501b0 2024-08-28T19:34:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc345ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281af9b0 con 0x55a3d9379070 2024-08-28T19:34:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc345ee740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281acb20 con 0x55a3d9358390 2024-08-28T19:34:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc345ee740 10 client.0 _ll_register_callbacks cb 0x7fdc1c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc27fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d972fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc26ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 0x55a3d9730370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc27fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d972fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51164/0 (socket says 172.21.15.16:51164) 2024-08-28T19:34:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc26ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 0x55a3d9730370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:41972/0 (socket says 172.21.15.16:41972) 2024-08-28T19:34:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc27fff640 1 -- 172.21.15.16:0/1804898247 learned_addr learned my addr 172.21.15.16:0/1804898247 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc27fff640 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9379070 0x55a3d97308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc26ffd640 1 -- 172.21.15.16:0/1804898247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9379070 msgr2=0x55a3d97308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc26ffd640 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9379070 0x55a3d97308b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc26ffd640 1 -- 172.21.15.16:0/1804898247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 msgr2=0x55a3d972fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc26ffd640 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d972fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc26ffd640 1 -- 172.21.15.16:0/1804898247 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3d97a5910 con 0x55a3d9358390 2024-08-28T19:34:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc26ffd640 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 0x55a3d9730370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c02a930 tx=0x7fdc1c004bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fd429a124a2c405 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.458+0000 7fdc06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:34:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc06ffd640 1 -- 172.21.15.16:0/1804898247 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c05b020 con 0x55a3d9358390 2024-08-28T19:34:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc345ee740 1 -- 172.21.15.16:0/1804898247 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc200055c0 con 0x55a3d9358390 2024-08-28T19:34:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc06ffd640 1 -- 172.21.15.16:0/1804898247 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7fdc1c004250 con 0x55a3d9358390 2024-08-28T19:34:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc06ffd640 1 -- 172.21.15.16:0/1804898247 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7fdc1c063050 con 0x55a3d9358390 2024-08-28T19:34:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc345ee740 10 client.27931 fetch_fsmap learned FSMap version 71 2024-08-28T19:34:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc345ee740 1 -- 172.21.15.16:0/1804898247 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55a3d97a6bc0 con 0x55a3d9358390 2024-08-28T19:34:30.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3405 60.50 MB/sec warmup 6 sec latency 5126.346 ms 2024-08-28T19:34:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc06ffd640 1 -- 172.21.15.16:0/1804898247 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc200055c0 con 0x55a3d9358390 2024-08-28T19:34:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc06ffd640 1 -- 172.21.15.16:0/1804898247 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 71) ==== 8426+0+0 (secure 0 0 0) 0x7fdc1c004cc0 con 0x55a3d9358390 2024-08-28T19:34:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc345ee740 10 client.27931 fetch_fsmap finished waiting for FSMap version 71 2024-08-28T19:34:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc345ee740 10 client.27931 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc345ee740 20 client.27931 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc345ee740 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a3d930cda0 0x55a3d97a9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc345ee740 4 client.27931 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.459+0000 7fdc345ee740 1 -- 172.21.15.16:0/1804898247 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3d9693070 con 0x55a3d930cda0 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.460+0000 7fdc27fff640 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a3d930cda0 0x55a3d97a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.460+0000 7fdc06ffd640 10 client.27931 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.460+0000 7fdc27fff640 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a3d930cda0 0x55a3d97a9490 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5593f9897578dca0 server_cookie=79a05bbb374d3bb4 in_seq=0 out_seq=0 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.460+0000 7fdc06ffd640 1 -- 172.21.15.16:0/1804898247 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3d9693070 con 0x55a3d930cda0 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.460+0000 7fdc06ffd640 10 client.27931 handle_command_reply: tid=0 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.460+0000 7fdc345ee740 1 client.27931 shutdown 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc345ee740 1 -- 172.21.15.16:0/1804898247 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 msgr2=0x55a3d9730370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc345ee740 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 0x55a3d9730370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c02a930 tx=0x7fdc1c004bf0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc345ee740 1 -- 172.21.15.16:0/1804898247 shutdown_connections 2024-08-28T19:34:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc345ee740 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a3d930cda0 0x55a3d97a9490 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc345ee740 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3d9379070 0x55a3d97308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc345ee740 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3d9358390 0x55a3d9730370 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc345ee740 1 --2- 172.21.15.16:0/1804898247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d93501b0 0x55a3d972fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc06ffd640 0 client.27931 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:34:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc345ee740 1 -- 172.21.15.16:0/1804898247 >> 172.21.15.16:0/1804898247 conn(0x55a3d936d460 msgr2=0x55a3d972aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc345ee740 1 -- 172.21.15.16:0/1804898247 shutdown_connections 2024-08-28T19:34:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc345ee740 1 -- 172.21.15.16:0/1804898247 wait complete. 2024-08-28T19:34:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.461+0000 7fdc345ee740 20 client.27931 trim_cache size 0 max 16384 2024-08-28T19:34:30.470 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:34:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 -- 172.21.15.16:0/737666879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdc100b4d20 msgr2=0x7fdc100b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdc100b4d20 0x7fdc100b7230 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x55a3d971cf90 tx=0x7fdc1800cf20 comp rx=0 tx=0).stop 2024-08-28T19:34:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 -- 172.21.15.16:0/737666879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc28103940 msgr2=0x7fdc2819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc28103940 0x7fdc2819a240 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800e9f0 tx=0x7fdc1800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 -- 172.21.15.16:0/737666879 shutdown_connections 2024-08-28T19:34:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdc100b4d20 0x7fdc100b7230 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc28109bd0 0x7fdc2819fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc28109280 0x7fdc2819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 --2- 172.21.15.16:0/737666879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc28103940 0x7fdc2819a240 secure :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800e9f0 tx=0x7fdc1800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:34:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 -- 172.21.15.16:0/737666879 >> 172.21.15.16:0/737666879 conn(0x7fdc280ff3d0 msgr2=0x7fdc281017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 -- 172.21.15.16:0/737666879 shutdown_connections 2024-08-28T19:34:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:30.462+0000 7fdc2e079640 1 -- 172.21.15.16:0/737666879 wait complete. 2024-08-28T19:34:30.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:30 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:30.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:30 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/334393216' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:34:30.722 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:30 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:30 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/334393216' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:34:31.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:34:31.460 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3581 52.56 MB/sec warmup 7 sec latency 6126.371 ms 2024-08-28T19:34:31.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:31 smithi115 ceph-mon[22184]: pgmap v757: 129 pgs: 129 active+clean; 427 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-08-28T19:34:31.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:31 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:31.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:31 smithi016 ceph-mon[18556]: pgmap v757: 129 pgs: 129 active+clean; 427 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-08-28T19:34:31.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:31 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:31 smithi028 ceph-mon[21769]: pgmap v757: 129 pgs: 129 active+clean; 427 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-08-28T19:34:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:31 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.883+0000 7fa480fd0640 1 -- 172.21.15.16:0/4289571442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa47c102840 msgr2=0x7fa47c102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.883+0000 7fa480fd0640 1 --2- 172.21.15.16:0/4289571442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa47c102840 0x7fa47c102c50 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009900 tx=0x7fa46c02a410 comp rx=0 tx=0).stop 2024-08-28T19:34:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.883+0000 7fa480fd0640 1 -- 172.21.15.16:0/4289571442 shutdown_connections 2024-08-28T19:34:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.883+0000 7fa480fd0640 1 --2- 172.21.15.16:0/4289571442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c108ad0 0x7fa47c10f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.883+0000 7fa480fd0640 1 --2- 172.21.15.16:0/4289571442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa47c108180 0x7fa47c108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.883+0000 7fa480fd0640 1 --2- 172.21.15.16:0/4289571442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa47c102840 0x7fa47c102c50 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.883+0000 7fa480fd0640 1 -- 172.21.15.16:0/4289571442 >> 172.21.15.16:0/4289571442 conn(0x7fa47c079530 msgr2=0x7fa47c079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:31.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.884+0000 7fa480fd0640 1 -- 172.21.15.16:0/4289571442 shutdown_connections 2024-08-28T19:34:31.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.884+0000 7fa480fd0640 1 -- 172.21.15.16:0/4289571442 wait complete. 2024-08-28T19:34:31.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa480fd0640 1 Processor -- start 2024-08-28T19:34:31.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa480fd0640 1 -- start start 2024-08-28T19:34:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa480fd0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa47c102840 0x7fa47c19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa480fd0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c108180 0x7fa47c19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa480fd0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa47c108ad0 0x7fa47c19fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa480fd0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c113860 con 0x7fa47c108ad0 2024-08-28T19:34:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa480fd0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c1136e0 con 0x7fa47c108180 2024-08-28T19:34:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa480fd0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c1139e0 con 0x7fa47c102840 2024-08-28T19:34:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa479d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c108180 0x7fa47c19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa47ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa47c108ad0 0x7fa47c19fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa479d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c108180 0x7fa47c19a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48512/0 (socket says 172.21.15.16:48512) 2024-08-28T19:34:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa479d9b640 1 -- 172.21.15.16:0/1233379692 learned_addr learned my addr 172.21.15.16:0/1233379692 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.886+0000 7fa47a59c640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa47c102840 0x7fa47c19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.887+0000 7fa479d9b640 1 -- 172.21.15.16:0/1233379692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa47c102840 msgr2=0x7fa47c19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.887+0000 7fa479d9b640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa47c102840 0x7fa47c19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.887+0000 7fa479d9b640 1 -- 172.21.15.16:0/1233379692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa47c108ad0 msgr2=0x7fa47c19fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.887+0000 7fa479d9b640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa47c108ad0 0x7fa47c19fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.887+0000 7fa479d9b640 1 -- 172.21.15.16:0/1233379692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47c1a01a0 con 0x7fa47c108180 2024-08-28T19:34:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.887+0000 7fa479d9b640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c108180 0x7fa47c19a7d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa47000d740 tx=0x7fa47000db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.887+0000 7fa46b7fe640 1 -- 172.21.15.16:0/1233379692 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa470017070 con 0x7fa47c108180 2024-08-28T19:34:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.887+0000 7fa480fd0640 1 -- 172.21.15.16:0/1233379692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa47c1a0490 con 0x7fa47c108180 2024-08-28T19:34:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.887+0000 7fa46b7fe640 1 -- 172.21.15.16:0/1233379692 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa470009820 con 0x7fa47c108180 2024-08-28T19:34:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.887+0000 7fa46b7fe640 1 -- 172.21.15.16:0/1233379692 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa470004ce0 con 0x7fa47c108180 2024-08-28T19:34:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.887+0000 7fa480fd0640 1 -- 172.21.15.16:0/1233379692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa47c10fc70 con 0x7fa47c108180 2024-08-28T19:34:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.890+0000 7fa487545740 1 Processor -- start 2024-08-28T19:34:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.890+0000 7fa487545740 1 -- start start 2024-08-28T19:34:31.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.890+0000 7fa487545740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 0x5626b231ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.890+0000 7fa487545740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b231b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.890+0000 7fa487545740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f52b60 0x5626b231b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.890+0000 7fa487545740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5626b228f330 con 0x5626b1f73840 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.890+0000 7fa487545740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5626b227e8c0 con 0x5626b1f4a980 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.890+0000 7fa487545740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5626b2282870 con 0x5626b1f52b60 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.891+0000 7fa47a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b231b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.891+0000 7fa47a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b231b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48520/0 (socket says 172.21.15.16:48520) 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.891+0000 7fa47a59c640 1 -- 172.21.15.16:0/1396572931 learned_addr learned my addr 172.21.15.16:0/1396572931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.891+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/1396572931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 0x5626b231ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.891+0000 7fa46b7fe640 1 -- 172.21.15.16:0/1233379692 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa470010460 con 0x7fa47c108180 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.891+0000 7fa47a59c640 1 -- 172.21.15.16:0/1396572931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f52b60 msgr2=0x5626b231b710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.891+0000 7fa47a59c640 1 --2- 172.21.15.16:0/1396572931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f52b60 0x5626b231b710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.891+0000 7fa47a59c640 1 -- 172.21.15.16:0/1396572931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 msgr2=0x5626b231ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.891+0000 7fa47a59c640 1 --2- 172.21.15.16:0/1396572931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 0x5626b231ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.891+0000 7fa47a59c640 1 -- 172.21.15.16:0/1396572931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626b2236df0 con 0x5626b1f4a980 2024-08-28T19:34:31.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.891+0000 7fa47a59c640 1 --2- 172.21.15.16:0/1396572931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b231b1d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa46c02a930 tx=0x7fa46c032c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=131f241df67c1ff0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:31.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.892+0000 7fa468ff9640 1 -- 172.21.15.16:0/1396572931 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c040070 con 0x5626b1f4a980 2024-08-28T19:34:31.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.892+0000 7fa468ff9640 1 -- 172.21.15.16:0/1396572931 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa46c004030 con 0x5626b1f4a980 2024-08-28T19:34:31.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.892+0000 7fa468ff9640 1 -- 172.21.15.16:0/1396572931 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c035040 con 0x5626b1f4a980 2024-08-28T19:34:31.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.892+0000 7fa46b7fe640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4640b4ca0 0x7fa4640b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4640b4ca0 0x7fa4640b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa46b7fe640 1 -- 172.21.15.16:0/1233379692 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7fa4700e5280 con 0x7fa47c108180 2024-08-28T19:34:31.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa487545740 1 -- 172.21.15.16:0/1396572931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 msgr2=0x5626b231b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:31.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa487545740 1 --2- 172.21.15.16:0/1396572931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b231b1d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa46c02a930 tx=0x7fa46c032c60 comp rx=0 tx=0).stop 2024-08-28T19:34:31.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa487545740 1 -- 172.21.15.16:0/1396572931 shutdown_connections 2024-08-28T19:34:31.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa487545740 1 --2- 172.21.15.16:0/1396572931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f52b60 0x5626b231b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa487545740 1 --2- 172.21.15.16:0/1396572931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b231b1d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa487545740 1 --2- 172.21.15.16:0/1396572931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 0x5626b231ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa487545740 1 -- 172.21.15.16:0/1396572931 >> 172.21.15.16:0/1396572931 conn(0x5626b1f67990 msgr2=0x5626b2318b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:31.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4640b4ca0 0x7fa4640b71b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa474005fd0 tx=0x7fa474012040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:31.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa487545740 1 -- 172.21.15.16:0/1396572931 shutdown_connections 2024-08-28T19:34:31.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.893+0000 7fa487545740 1 -- 172.21.15.16:0/1396572931 wait complete. 2024-08-28T19:34:31.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.894+0000 7fa487545740 1 -- start start 2024-08-28T19:34:31.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.894+0000 7fa487545740 1 Processor -- start 2024-08-28T19:34:31.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.894+0000 7fa487545740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b2326470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.894+0000 7fa487545740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 0x5626b23269b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.894+0000 7fa487545740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f73840 0x5626b2332700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.894+0000 7fa487545740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5626b20e17a0 con 0x5626b1f52b60 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.894+0000 7fa487545740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5626b2205240 con 0x5626b1f4a980 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.894+0000 7fa487545740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5626b2329410 con 0x5626b1f73840 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.894+0000 7fa487545740 10 client.0 _ll_register_callbacks cb 0x5626b1f8b090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa47ad9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f73840 0x5626b2332700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa47ad9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f73840 0x5626b2332700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:42000/0 (socket says 172.21.15.16:42000) 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa47ad9d640 1 -- 172.21.15.16:0/180364691 learned_addr learned my addr 172.21.15.16:0/180364691 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa47a59c640 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b2326470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa479d9b640 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 0x5626b23269b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa47ad9d640 1 -- 172.21.15.16:0/180364691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 msgr2=0x5626b2326470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b2326470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa47ad9d640 1 -- 172.21.15.16:0/180364691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 msgr2=0x5626b23269b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 0x5626b23269b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa47ad9d640 1 -- 172.21.15.16:0/180364691 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626b223b2b0 con 0x5626b1f73840 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa479d9b640 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 0x5626b23269b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f73840 0x5626b2332700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa474005f40 tx=0x7fa474004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23e24cdd2cbff6fd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa461ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa461ffb640 1 -- 172.21.15.16:0/180364691 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47402d060 con 0x5626b1f73840 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa487545740 1 -- 172.21.15.16:0/180364691 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa46c004030 con 0x5626b1f73840 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.895+0000 7fa461ffb640 1 -- 172.21.15.16:0/180364691 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7fa474004580 con 0x5626b1f73840 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.896+0000 7fa461ffb640 1 -- 172.21.15.16:0/180364691 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7fa474034050 con 0x5626b1f73840 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.896+0000 7fa487545740 10 client.27940 fetch_fsmap learned FSMap version 71 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.896+0000 7fa487545740 1 -- 172.21.15.16:0/180364691 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x5626b1fedb90 con 0x5626b1f73840 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.896+0000 7fa461ffb640 1 -- 172.21.15.16:0/180364691 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa46c004030 con 0x5626b1f73840 2024-08-28T19:34:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.896+0000 7fa461ffb640 1 -- 172.21.15.16:0/180364691 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 71) ==== 8426+0+0 (secure 0 0 0) 0x7fa474028430 con 0x5626b1f73840 2024-08-28T19:34:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.896+0000 7fa487545740 10 client.27940 fetch_fsmap finished waiting for FSMap version 71 2024-08-28T19:34:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.896+0000 7fa487545740 10 client.27940 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:34:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.896+0000 7fa487545740 20 client.27940 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.896+0000 7fa487545740 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5626b1f321c0 0x5626b2338620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.896+0000 7fa487545740 4 client.27940 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:34:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.896+0000 7fa487545740 1 -- 172.21.15.16:0/180364691 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626b2291280 con 0x5626b1f321c0 2024-08-28T19:34:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.897+0000 7fa47a59c640 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5626b1f321c0 0x5626b2338620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.897+0000 7fa47a59c640 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5626b1f321c0 0x5626b2338620 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27fd06de09a8427d server_cookie=605c1946fb900cf6 in_seq=0 out_seq=0 2024-08-28T19:34:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.897+0000 7fa461ffb640 10 client.27940 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa461ffb640 1 -- 172.21.15.16:0/180364691 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5626b2291280 con 0x5626b1f321c0 2024-08-28T19:34:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa461ffb640 10 client.27940 handle_command_reply: tid=0 2024-08-28T19:34:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 1 client.27940 shutdown 2024-08-28T19:34:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 1 -- 172.21.15.16:0/180364691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f73840 msgr2=0x5626b2332700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f73840 0x5626b2332700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa474005f40 tx=0x7fa474004ab0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 1 -- 172.21.15.16:0/180364691 shutdown_connections 2024-08-28T19:34:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5626b1f321c0 0x5626b2338620 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f73840 0x5626b2332700 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 0x5626b23269b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 1 --2- 172.21.15.16:0/180364691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b2326470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 1 -- 172.21.15.16:0/180364691 >> 172.21.15.16:0/180364691 conn(0x5626b1f67990 msgr2=0x5626b2317340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa461ffb640 0 client.27940 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 1 -- 172.21.15.16:0/180364691 shutdown_connections 2024-08-28T19:34:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 1 -- 172.21.15.16:0/180364691 wait complete. 2024-08-28T19:34:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.903+0000 7fa487545740 20 client.27940 trim_cache size 0 max 16384 2024-08-28T19:34:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:31.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.923+0000 7fa487545740 1 Processor -- start 2024-08-28T19:34:31.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.923+0000 7fa487545740 1 -- start start 2024-08-28T19:34:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.923+0000 7fa487545740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b231c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.923+0000 7fa487545740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f52b60 0x5626b2323660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.923+0000 7fa487545740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 0x5626b2323ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.923+0000 7fa487545740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c1aeaa0 con 0x5626b1f73840 2024-08-28T19:34:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.923+0000 7fa487545740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c1b07c0 con 0x5626b1f4a980 2024-08-28T19:34:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.923+0000 7fa487545740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c1af1a0 con 0x5626b1f52b60 2024-08-28T19:34:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.924+0000 7fa47ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 0x5626b2323ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.924+0000 7fa47ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 0x5626b2323ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51214/0 (socket says 172.21.15.16:51214) 2024-08-28T19:34:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.924+0000 7fa47ad9d640 1 -- 172.21.15.16:0/1980045811 learned_addr learned my addr 172.21.15.16:0/1980045811 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.924+0000 7fa47a59c640 1 --2- 172.21.15.16:0/1980045811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b231c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.924+0000 7fa479d9b640 1 --2- 172.21.15.16:0/1980045811 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f52b60 0x5626b2323660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.924+0000 7fa47ad9d640 1 -- 172.21.15.16:0/1980045811 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f52b60 msgr2=0x5626b2323660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:31.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.924+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/1980045811 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f52b60 0x5626b2323660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.924+0000 7fa47ad9d640 1 -- 172.21.15.16:0/1980045811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 msgr2=0x5626b231c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:31.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.924+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/1980045811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b231c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.924+0000 7fa47ad9d640 1 -- 172.21.15.16:0/1980045811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626b224ed90 con 0x5626b1f73840 2024-08-28T19:34:31.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.924+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/1980045811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 0x5626b2323ba0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fa474012010 tx=0x7fa474009320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6606f5608d95004b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:31.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.925+0000 7fa463fff640 1 -- 172.21.15.16:0/1980045811 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47403d070 con 0x5626b1f73840 2024-08-28T19:34:31.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.925+0000 7fa463fff640 1 -- 172.21.15.16:0/1980045811 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa474024040 con 0x5626b1f73840 2024-08-28T19:34:31.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.925+0000 7fa463fff640 1 -- 172.21.15.16:0/1980045811 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47401b730 con 0x5626b1f73840 2024-08-28T19:34:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.926+0000 7fa487545740 1 -- 172.21.15.16:0/1980045811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 msgr2=0x5626b2323ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.926+0000 7fa487545740 1 --2- 172.21.15.16:0/1980045811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 0x5626b2323ba0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fa474012010 tx=0x7fa474009320 comp rx=0 tx=0).stop 2024-08-28T19:34:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.926+0000 7fa487545740 1 -- 172.21.15.16:0/1980045811 shutdown_connections 2024-08-28T19:34:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.926+0000 7fa487545740 1 --2- 172.21.15.16:0/1980045811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f73840 0x5626b2323ba0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.926+0000 7fa487545740 1 --2- 172.21.15.16:0/1980045811 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f52b60 0x5626b2323660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.926+0000 7fa487545740 1 --2- 172.21.15.16:0/1980045811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f4a980 0x5626b231c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.926+0000 7fa487545740 1 -- 172.21.15.16:0/1980045811 >> 172.21.15.16:0/1980045811 conn(0x5626b1f67990 msgr2=0x5626b231bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.926+0000 7fa487545740 1 -- 172.21.15.16:0/1980045811 shutdown_connections 2024-08-28T19:34:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.926+0000 7fa487545740 1 -- 172.21.15.16:0/1980045811 wait complete. 2024-08-28T19:34:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa487545740 1 -- start start 2024-08-28T19:34:31.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa487545740 1 Processor -- start 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa487545740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f4a980 0x5626b232ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa487545740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 0x5626b2329620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa487545740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f73840 0x5626b2329bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa487545740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c1b00c0 con 0x5626b1f52b60 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa487545740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c1af930 con 0x5626b1f73840 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa487545740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c1acaa0 con 0x5626b1f4a980 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa487545740 10 client.0 _ll_register_callbacks cb 0x7fa474003f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa479d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 0x5626b2329620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa47ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f73840 0x5626b2329bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa47ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f73840 0x5626b2329bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48546/0 (socket says 172.21.15.16:48546) 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa479d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 0x5626b2329620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51226/0 (socket says 172.21.15.16:51226) 2024-08-28T19:34:31.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa47ad9d640 1 -- 172.21.15.16:0/3586130889 learned_addr learned my addr 172.21.15.16:0/3586130889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.927+0000 7fa47a59c640 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f4a980 0x5626b232ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa47ad9d640 1 -- 172.21.15.16:0/3586130889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f4a980 msgr2=0x5626b232ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f4a980 0x5626b232ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa47ad9d640 1 -- 172.21.15.16:0/3586130889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 msgr2=0x5626b2329620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 0x5626b2329620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa47ad9d640 1 -- 172.21.15.16:0/3586130889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626b232a350 con 0x5626b1f73840 2024-08-28T19:34:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa47ad9d640 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f73840 0x5626b2329bc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa474028660 tx=0x7fa47400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=425a3080d4e5114d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:31.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa461ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:34:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa461ffb640 1 -- 172.21.15.16:0/3586130889 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa474046020 con 0x5626b1f73840 2024-08-28T19:34:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa461ffb640 1 -- 172.21.15.16:0/3586130889 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7fa474005800 con 0x5626b1f73840 2024-08-28T19:34:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa487545740 1 -- 172.21.15.16:0/3586130889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa474024040 con 0x5626b1f73840 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa461ffb640 1 -- 172.21.15.16:0/3586130889 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7fa474051050 con 0x5626b1f73840 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa487545740 10 client.28283 fetch_fsmap learned FSMap version 71 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.928+0000 7fa487545740 1 -- 172.21.15.16:0/3586130889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x5626b23a3f90 con 0x5626b1f73840 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.929+0000 7fa461ffb640 1 -- 172.21.15.16:0/3586130889 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa474024040 con 0x5626b1f73840 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.929+0000 7fa461ffb640 1 -- 172.21.15.16:0/3586130889 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 71) ==== 8426+0+0 (secure 0 0 0) 0x7fa474042450 con 0x5626b1f73840 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.929+0000 7fa487545740 10 client.28283 fetch_fsmap finished waiting for FSMap version 71 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.929+0000 7fa487545740 10 client.28283 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.929+0000 7fa487545740 20 client.28283 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.929+0000 7fa487545740 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5626b1f07570 0x5626b23a6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.929+0000 7fa487545740 4 client.28283 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.7, "await_for": 45.0}] 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.929+0000 7fa487545740 1 -- 172.21.15.16:0/3586130889 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.7, "await_for": 45.0}) -- 0x5626b228cd20 con 0x5626b1f07570 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.929+0000 7fa47a59c640 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5626b1f07570 0x5626b23a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.929+0000 7fa461ffb640 10 client.28283 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:31.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:31.930+0000 7fa47a59c640 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5626b1f07570 0x5626b23a6780 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfd0682c3790065f server_cookie=4b0cd3a14201e31e in_seq=0 out_seq=0 2024-08-28T19:34:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.427+0000 7f35c0a35640 1 -- 172.21.15.16:0/2237423398 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc1024d0 msgr2=0x7f35bc112720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.427+0000 7f35c0a35640 1 --2- 172.21.15.16:0/2237423398 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc1024d0 0x7f35bc112720 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009900 tx=0x7f35ac02a3f0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.428+0000 7f35c0a35640 1 -- 172.21.15.16:0/2237423398 shutdown_connections 2024-08-28T19:34:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.428+0000 7f35c0a35640 1 --2- 172.21.15.16:0/2237423398 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc1024d0 0x7f35bc112720 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.428+0000 7f35c0a35640 1 --2- 172.21.15.16:0/2237423398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35bc101b80 0x7f35bc101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.428+0000 7f35c0a35640 1 --2- 172.21.15.16:0/2237423398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35bc108590 0x7f35bc1089a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.428+0000 7f35c0a35640 1 -- 172.21.15.16:0/2237423398 >> 172.21.15.16:0/2237423398 conn(0x7f35bc077a00 msgr2=0x7f35bc077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:32.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.428+0000 7f35c0a35640 1 -- 172.21.15.16:0/2237423398 shutdown_connections 2024-08-28T19:34:32.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.428+0000 7f35c0a35640 1 -- 172.21.15.16:0/2237423398 wait complete. 2024-08-28T19:34:32.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.429+0000 7f35c0a35640 1 Processor -- start 2024-08-28T19:34:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.430+0000 7f35c0a35640 1 -- start start 2024-08-28T19:34:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.430+0000 7f35c0a35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35bc101b80 0x7f35bc1a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.430+0000 7f35c0a35640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35bc1024d0 0x7f35bc1a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.430+0000 7f35c0a35640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc108590 0x7f35bc1a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.430+0000 7f35c0a35640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc115ab0 con 0x7f35bc101b80 2024-08-28T19:34:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.430+0000 7f35c0a35640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc115930 con 0x7f35bc1024d0 2024-08-28T19:34:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.430+0000 7f35c0a35640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc115c30 con 0x7f35bc108590 2024-08-28T19:34:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.430+0000 7f35ba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35bc101b80 0x7f35bc1a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.430+0000 7f35ba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35bc101b80 0x7f35bc1a07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51246/0 (socket says 172.21.15.16:51246) 2024-08-28T19:34:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.430+0000 7f35ba59c640 1 -- 172.21.15.16:0/1465227718 learned_addr learned my addr 172.21.15.16:0/1465227718 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.431+0000 7f35b9d9b640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35bc1024d0 0x7f35bc1a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.431+0000 7f35b9d9b640 1 -- 172.21.15.16:0/1465227718 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc108590 msgr2=0x7f35bc1a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.431+0000 7f35b9d9b640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc108590 0x7f35bc1a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.431+0000 7f35b9d9b640 1 -- 172.21.15.16:0/1465227718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35bc101b80 msgr2=0x7f35bc1a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.431+0000 7f35b9d9b640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35bc101b80 0x7f35bc1a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.431+0000 7f35b9d9b640 1 -- 172.21.15.16:0/1465227718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35bc111580 con 0x7f35bc1024d0 2024-08-28T19:34:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.432+0000 7f35b9d9b640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35bc1024d0 0x7f35bc1a0ce0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f35bc075850 tx=0x7f35b400ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:32.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.432+0000 7f35ab7fe640 1 -- 172.21.15.16:0/1465227718 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4010430 con 0x7f35bc1024d0 2024-08-28T19:34:32.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.432+0000 7f35c0a35640 1 -- 172.21.15.16:0/1465227718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35bc111870 con 0x7f35bc1024d0 2024-08-28T19:34:32.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.432+0000 7f35ab7fe640 1 -- 172.21.15.16:0/1465227718 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35b4010a80 con 0x7f35bc1024d0 2024-08-28T19:34:32.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.432+0000 7f35ab7fe640 1 -- 172.21.15.16:0/1465227718 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4017a20 con 0x7f35bc1024d0 2024-08-28T19:34:32.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.432+0000 7f35c0a35640 1 -- 172.21.15.16:0/1465227718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35bc111e80 con 0x7f35bc1024d0 2024-08-28T19:34:32.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.436+0000 7f35c6faa740 1 Processor -- start 2024-08-28T19:34:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.436+0000 7f35c6faa740 1 -- start start 2024-08-28T19:34:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.437+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 0x55a1d2ff7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.437+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d2ff7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.437+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d2ff7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.437+0000 7f35c6faa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a1d2f6e680 con 0x55a1d2c31390 2024-08-28T19:34:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.437+0000 7f35c6faa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a1d2f5dc10 con 0x55a1d2c291b0 2024-08-28T19:34:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.437+0000 7f35c6faa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a1d2f61bc0 con 0x55a1d2c52070 2024-08-28T19:34:32.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.437+0000 7f35bad9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 0x55a1d2ff7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.437+0000 7f35bad9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 0x55a1d2ff7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:42044/0 (socket says 172.21.15.16:42044) 2024-08-28T19:34:32.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.437+0000 7f35bad9d640 1 -- 172.21.15.16:0/146038438 learned_addr learned my addr 172.21.15.16:0/146038438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:32.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.437+0000 7f35ba59c640 1 --2- 172.21.15.16:0/146038438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d2ff7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.440+0000 7f35bad9d640 1 -- 172.21.15.16:0/146038438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 msgr2=0x55a1d2ff7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:32.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.440+0000 7f35bad9d640 1 --2- 172.21.15.16:0/146038438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d2ff7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.440+0000 7f35bad9d640 1 -- 172.21.15.16:0/146038438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 msgr2=0x55a1d2ff7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:32.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.440+0000 7f35bad9d640 1 --2- 172.21.15.16:0/146038438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d2ff7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.440+0000 7f35bad9d640 1 -- 172.21.15.16:0/146038438 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1d2cbd650 con 0x55a1d2c52070 2024-08-28T19:34:32.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.441+0000 7f35bad9d640 1 --2- 172.21.15.16:0/146038438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 0x55a1d2ff7310 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009a20 tx=0x7f35ac0095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2038af2dd2079000 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:32.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.441+0000 7f35ab7fe640 1 -- 172.21.15.16:0/1465227718 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f35b4017be0 con 0x7f35bc1024d0 2024-08-28T19:34:32.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.442+0000 7f35a8ff9640 1 -- 172.21.15.16:0/146038438 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac040070 con 0x55a1d2c52070 2024-08-28T19:34:32.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.442+0000 7f35a8ff9640 1 -- 172.21.15.16:0/146038438 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35ac004040 con 0x55a1d2c52070 2024-08-28T19:34:32.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.442+0000 7f35a8ff9640 1 -- 172.21.15.16:0/146038438 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac035040 con 0x55a1d2c52070 2024-08-28T19:34:32.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.442+0000 7f35ab7fe640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f35a40b4da0 0x7f35a40b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.443+0000 7f35ab7fe640 1 -- 172.21.15.16:0/1465227718 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f35b40e8530 con 0x7f35bc1024d0 2024-08-28T19:34:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.443+0000 7f35ba59c640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f35a40b4da0 0x7f35a40b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.443+0000 7f35c6faa740 1 -- 172.21.15.16:0/146038438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 msgr2=0x55a1d2ff7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.444+0000 7f35c6faa740 1 --2- 172.21.15.16:0/146038438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 0x55a1d2ff7310 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009a20 tx=0x7f35ac0095e0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.444+0000 7f35c6faa740 1 -- 172.21.15.16:0/146038438 shutdown_connections 2024-08-28T19:34:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.444+0000 7f35ba59c640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f35a40b4da0 0x7f35a40b72b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x55a1d2ff5f90 tx=0x7f35b0008040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.444+0000 7f35c6faa740 1 --2- 172.21.15.16:0/146038438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d2ff7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.444+0000 7f35c6faa740 1 --2- 172.21.15.16:0/146038438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d2ff7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.444+0000 7f35c6faa740 1 --2- 172.21.15.16:0/146038438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 0x55a1d2ff7310 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.444+0000 7f35c6faa740 1 -- 172.21.15.16:0/146038438 >> 172.21.15.16:0/146038438 conn(0x55a1d2c46460 msgr2=0x55a1d2ff5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.445+0000 7f35c6faa740 1 -- 172.21.15.16:0/146038438 shutdown_connections 2024-08-28T19:34:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.445+0000 7f35c6faa740 1 -- 172.21.15.16:0/146038438 wait complete. 2024-08-28T19:34:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.446+0000 7f35c6faa740 1 -- start start 2024-08-28T19:34:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.446+0000 7f35c6faa740 1 Processor -- start 2024-08-28T19:34:32.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.447+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d3002a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.447+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d3002f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.447+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 0x55a1d300edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.447+0000 7f35c6faa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a1d2dc0700 con 0x55a1d2c31390 2024-08-28T19:34:32.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.447+0000 7f35c6faa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a1d2ee4750 con 0x55a1d2c291b0 2024-08-28T19:34:32.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.447+0000 7f35c6faa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a1d3005a90 con 0x55a1d2c52070 2024-08-28T19:34:32.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.448+0000 7f35c6faa740 10 client.0 _ll_register_callbacks cb 0x55a1d2f13e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:32.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.448+0000 7f35b9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d3002f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.448+0000 7f35b9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d3002f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51258/0 (socket says 172.21.15.16:51258) 2024-08-28T19:34:32.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.448+0000 7f35b9d9b640 1 -- 172.21.15.16:0/2571400116 learned_addr learned my addr 172.21.15.16:0/2571400116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:32.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.448+0000 7f35ba59c640 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d3002a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.449+0000 7f35bad9d640 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 0x55a1d300edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.449+0000 7f35b9d9b640 1 -- 172.21.15.16:0/2571400116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 msgr2=0x55a1d300edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:32.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.449+0000 7f35b9d9b640 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 0x55a1d300edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.449+0000 7f35b9d9b640 1 -- 172.21.15.16:0/2571400116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 msgr2=0x55a1d3002a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:32.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.449+0000 7f35b9d9b640 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d3002a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.449+0000 7f35b9d9b640 1 -- 172.21.15.16:0/2571400116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1d2f1d900 con 0x55a1d2c31390 2024-08-28T19:34:32.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.449+0000 7f35b9d9b640 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d3002f90 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f35b40aba40 tx=0x7f35b40ae030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203315b972e296cd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:32.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.449+0000 7f35a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.450+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2571400116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b40ae7d0 con 0x55a1d2c31390 2024-08-28T19:34:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.450+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2571400116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f35b40aaa60 con 0x55a1d2c31390 2024-08-28T19:34:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.450+0000 7f35c6faa740 1 -- 172.21.15.16:0/2571400116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35ac004040 con 0x55a1d2c31390 2024-08-28T19:34:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.451+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2571400116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f35b40b3040 con 0x55a1d2c31390 2024-08-28T19:34:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.451+0000 7f35c6faa740 10 client.18852 fetch_fsmap learned FSMap version 72 2024-08-28T19:34:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.451+0000 7f35c6faa740 1 -- 172.21.15.16:0/2571400116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55a1d30118c0 con 0x55a1d2c31390 2024-08-28T19:34:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.451+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2571400116 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35b40b7af0 con 0x55a1d2c31390 2024-08-28T19:34:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.451+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2571400116 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 72) ==== 8434+0+0 (secure 0 0 0) 0x7f35b40ada30 con 0x55a1d2c31390 2024-08-28T19:34:32.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.452+0000 7f35c6faa740 10 client.18852 fetch_fsmap finished waiting for FSMap version 72 2024-08-28T19:34:32.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.452+0000 7f35c6faa740 10 client.18852 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:34:32.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.452+0000 7f35c6faa740 20 client.18852 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:32.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.452+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a1d2c109f0 0x55a1d3014d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.452+0000 7f35c6faa740 4 client.18852 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:34:32.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.452+0000 7f35c6faa740 1 -- 172.21.15.16:0/2571400116 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f35ac004040 con 0x55a1d2c109f0 2024-08-28T19:34:32.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.452+0000 7f35ba59c640 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a1d2c109f0 0x55a1d3014d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.453+0000 7f35a1ffb640 10 client.18852 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:34:32.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.453+0000 7f35ba59c640 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a1d2c109f0 0x55a1d3014d90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=58dbb65e9dd6b635 server_cookie=a4a3852d446d3c13 in_seq=0 out_seq=0 2024-08-28T19:34:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.455+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2571400116 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f35ac004040 con 0x55a1d2c109f0 2024-08-28T19:34:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.455+0000 7f35a1ffb640 10 client.18852 handle_command_reply: tid=0 2024-08-28T19:34:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.455+0000 7f35c6faa740 1 client.18852 shutdown 2024-08-28T19:34:32.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.456+0000 7f35c6faa740 1 -- 172.21.15.16:0/2571400116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 msgr2=0x55a1d3002f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.456+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d3002f90 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f35b40aba40 tx=0x7f35b40ae030 comp rx=0 tx=0).stop 2024-08-28T19:34:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.457+0000 7f35c6faa740 1 -- 172.21.15.16:0/2571400116 shutdown_connections 2024-08-28T19:34:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.457+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a1d2c109f0 0x55a1d3014d90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.457+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c52070 0x55a1d300edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.457+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d3002f90 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.457+0000 7f35a1ffb640 0 client.18852 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:34:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.457+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2571400116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d3002a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.457+0000 7f35c6faa740 1 -- 172.21.15.16:0/2571400116 >> 172.21.15.16:0/2571400116 conn(0x55a1d2c46460 msgr2=0x55a1d2ffa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.457+0000 7f35c6faa740 1 -- 172.21.15.16:0/2571400116 shutdown_connections 2024-08-28T19:34:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.457+0000 7f35c6faa740 1 -- 172.21.15.16:0/2571400116 wait complete. 2024-08-28T19:34:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.457+0000 7f35c6faa740 20 client.18852 trim_cache size 0 max 16384 2024-08-28T19:34:32.460 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3699 46.04 MB/sec warmup 8 sec latency 7126.403 ms 2024-08-28T19:34:32.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.483+0000 7f35c6faa740 1 Processor -- start 2024-08-28T19:34:32.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.483+0000 7f35c6faa740 1 -- start start 2024-08-28T19:34:32.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c109f0 0x55a1d2ffab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c291b0 0x55a1d3002800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c31390 0x55a1d3002d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35c6faa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc1bfb50 con 0x55a1d2c291b0 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35c6faa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc1c18d0 con 0x55a1d2c109f0 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35c6faa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc1c02b0 con 0x55a1d2c31390 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c109f0 0x55a1d2ffab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c109f0 0x55a1d2ffab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48612/0 (socket says 172.21.15.16:48612) 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35ba59c640 1 -- 172.21.15.16:0/1259440305 learned_addr learned my addr 172.21.15.16:0/1259440305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35bad9d640 1 --2- 172.21.15.16:0/1259440305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c31390 0x55a1d3002d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35b9d9b640 1 --2- 172.21.15.16:0/1259440305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c291b0 0x55a1d3002800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35ba59c640 1 -- 172.21.15.16:0/1259440305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c31390 msgr2=0x55a1d3002d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35ba59c640 1 --2- 172.21.15.16:0/1259440305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c31390 0x55a1d3002d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35ba59c640 1 -- 172.21.15.16:0/1259440305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c291b0 msgr2=0x55a1d3002800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:32.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35ba59c640 1 --2- 172.21.15.16:0/1259440305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c291b0 0x55a1d3002800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.484+0000 7f35ba59c640 1 -- 172.21.15.16:0/1259440305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1d3005190 con 0x55a1d2c109f0 2024-08-28T19:34:32.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.485+0000 7f35ba59c640 1 --2- 172.21.15.16:0/1259440305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c109f0 0x55a1d2ffab20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f35b0010b40 tx=0x7f35b0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46d5d2f764c4f14d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:32.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.485+0000 7f35a3fff640 1 -- 172.21.15.16:0/1259440305 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b002f070 con 0x55a1d2c109f0 2024-08-28T19:34:32.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.485+0000 7f35a3fff640 1 -- 172.21.15.16:0/1259440305 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35b001d040 con 0x55a1d2c109f0 2024-08-28T19:34:32.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.485+0000 7f35a3fff640 1 -- 172.21.15.16:0/1259440305 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b001c040 con 0x55a1d2c109f0 2024-08-28T19:34:32.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.486+0000 7f35c6faa740 1 -- 172.21.15.16:0/1259440305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c109f0 msgr2=0x55a1d2ffab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:32.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.486+0000 7f35c6faa740 1 --2- 172.21.15.16:0/1259440305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c109f0 0x55a1d2ffab20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f35b0010b40 tx=0x7f35b0016000 comp rx=0 tx=0).stop 2024-08-28T19:34:32.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.487+0000 7f35c6faa740 1 -- 172.21.15.16:0/1259440305 shutdown_connections 2024-08-28T19:34:32.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.487+0000 7f35c6faa740 1 --2- 172.21.15.16:0/1259440305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c31390 0x55a1d3002d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.487+0000 7f35c6faa740 1 --2- 172.21.15.16:0/1259440305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c291b0 0x55a1d3002800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.487+0000 7f35c6faa740 1 --2- 172.21.15.16:0/1259440305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c109f0 0x55a1d2ffab20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.487+0000 7f35c6faa740 1 -- 172.21.15.16:0/1259440305 >> 172.21.15.16:0/1259440305 conn(0x55a1d2c46460 msgr2=0x55a1d2ffa410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:32.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.487+0000 7f35c6faa740 1 -- 172.21.15.16:0/1259440305 shutdown_connections 2024-08-28T19:34:32.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.487+0000 7f35c6faa740 1 -- 172.21.15.16:0/1259440305 wait complete. 2024-08-28T19:34:32.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.487+0000 7f35c6faa740 1 -- start start 2024-08-28T19:34:32.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.488+0000 7f35c6faa740 1 Processor -- start 2024-08-28T19:34:32.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.488+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c109f0 0x55a1d30070b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.488+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d30075f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.488+0000 7f35c6faa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d307e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.488+0000 7f35c6faa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc1c11d0 con 0x55a1d2c31390 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.488+0000 7f35c6faa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc1c0a40 con 0x55a1d2c291b0 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.488+0000 7f35c6faa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc1bdb30 con 0x55a1d2c109f0 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.488+0000 7f35c6faa740 10 client.0 _ll_register_callbacks cb 0x7f35b40ae190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35bad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d307e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35ba59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c109f0 0x55a1d30070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35bad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d307e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51282/0 (socket says 172.21.15.16:51282) 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35bad9d640 1 -- 172.21.15.16:0/2005159765 learned_addr learned my addr 172.21.15.16:0/2005159765 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35b9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d30075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35bad9d640 1 -- 172.21.15.16:0/2005159765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c109f0 msgr2=0x55a1d30070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35bad9d640 1 --2- 172.21.15.16:0/2005159765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c109f0 0x55a1d30070b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35bad9d640 1 -- 172.21.15.16:0/2005159765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 msgr2=0x55a1d30075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35bad9d640 1 --2- 172.21.15.16:0/2005159765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d30075f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35bad9d640 1 -- 172.21.15.16:0/2005159765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1d307ed00 con 0x55a1d2c31390 2024-08-28T19:34:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35bad9d640 1 --2- 172.21.15.16:0/2005159765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d307e7c0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f35ac005f70 tx=0x7f35ac0316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95062fdc3e4a0fa5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.489+0000 7f35a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.490+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2005159765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac035040 con 0x55a1d2c31390 2024-08-28T19:34:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.490+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2005159765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f35ac031cd0 con 0x55a1d2c31390 2024-08-28T19:34:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.490+0000 7f35c6faa740 1 -- 172.21.15.16:0/2005159765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35b001d040 con 0x55a1d2c31390 2024-08-28T19:34:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.490+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2005159765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f35ac057050 con 0x55a1d2c31390 2024-08-28T19:34:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.490+0000 7f35c6faa740 10 client.18855 fetch_fsmap learned FSMap version 72 2024-08-28T19:34:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.490+0000 7f35c6faa740 1 -- 172.21.15.16:0/2005159765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55a1d307fdf0 con 0x55a1d2c31390 2024-08-28T19:34:32.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.491+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2005159765 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35b001d040 con 0x55a1d2c31390 2024-08-28T19:34:32.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.492+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2005159765 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 72) ==== 8434+0+0 (secure 0 0 0) 0x7f35ac045450 con 0x55a1d2c31390 2024-08-28T19:34:32.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.492+0000 7f35c6faa740 10 client.18855 fetch_fsmap finished waiting for FSMap version 72 2024-08-28T19:34:32.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.492+0000 7f35c6faa740 10 client.18855 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:34:32.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.492+0000 7f35c6faa740 20 client.18855 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:32.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.492+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2005159765 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a1d2c52070 0x55a1d3082180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:32.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.492+0000 7f35c6faa740 4 client.18855 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:34:32.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.492+0000 7f35c6faa740 1 -- 172.21.15.16:0/2005159765 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a1d2f705d0 con 0x55a1d2c52070 2024-08-28T19:34:32.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.494+0000 7f35ba59c640 1 --2- 172.21.15.16:0/2005159765 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a1d2c52070 0x55a1d3082180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:32.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.497+0000 7f35ba59c640 1 --2- 172.21.15.16:0/2005159765 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a1d2c52070 0x55a1d3082180 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8adffb0fc239daf9 server_cookie=3e86b68882596064 in_seq=0 out_seq=0 2024-08-28T19:34:32.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.497+0000 7f35a1ffb640 10 client.18855 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.498+0000 7f35a1ffb640 1 -- 172.21.15.16:0/2005159765 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a1d2f705d0 con 0x55a1d2c52070 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.498+0000 7f35a1ffb640 10 client.18855 handle_command_reply: tid=0 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.498+0000 7f35c6faa740 1 client.18855 shutdown 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.499+0000 7f35c6faa740 1 -- 172.21.15.16:0/2005159765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 msgr2=0x55a1d307e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.499+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2005159765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d307e7c0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f35ac005f70 tx=0x7f35ac0316d0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.499+0000 7f35c6faa740 1 -- 172.21.15.16:0/2005159765 shutdown_connections 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.499+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2005159765 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55a1d2c52070 0x55a1d3082180 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.499+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2005159765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1d2c31390 0x55a1d307e7c0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.499+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2005159765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1d2c291b0 0x55a1d30075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.499+0000 7f35c6faa740 1 --2- 172.21.15.16:0/2005159765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a1d2c109f0 0x55a1d30070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.499+0000 7f35c6faa740 1 -- 172.21.15.16:0/2005159765 >> 172.21.15.16:0/2005159765 conn(0x55a1d2c46460 msgr2=0x55a1d3002800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.499+0000 7f35a1ffb640 0 client.18855 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:34:32.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.499+0000 7f35c6faa740 1 -- 172.21.15.16:0/2005159765 shutdown_connections 2024-08-28T19:34:32.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.499+0000 7f35c6faa740 1 -- 172.21.15.16:0/2005159765 wait complete. 2024-08-28T19:34:32.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.500+0000 7f35c6faa740 20 client.18855 trim_cache size 0 max 16384 2024-08-28T19:34:32.503 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:34:32.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.500+0000 7f35c0a35640 1 -- 172.21.15.16:0/1465227718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f35a40b4da0 msgr2=0x7f35a40b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:32.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.500+0000 7f35c0a35640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f35a40b4da0 0x7f35a40b72b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x55a1d2ff5f90 tx=0x7f35b0008040 comp rx=0 tx=0).stop 2024-08-28T19:34:32.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.500+0000 7f35c0a35640 1 -- 172.21.15.16:0/1465227718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35bc1024d0 msgr2=0x7f35bc1a0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:32.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.500+0000 7f35c0a35640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35bc1024d0 0x7f35bc1a0ce0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f35bc075850 tx=0x7f35b400ef90 comp rx=0 tx=0).stop 2024-08-28T19:34:32.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.501+0000 7f35c0a35640 1 -- 172.21.15.16:0/1465227718 shutdown_connections 2024-08-28T19:34:32.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.501+0000 7f35c0a35640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc108590 0x7f35bc1a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.501+0000 7f35c0a35640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f35a40b4da0 0x7f35a40b72b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.501+0000 7f35c0a35640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35bc1024d0 0x7f35bc1a0ce0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.501+0000 7f35c0a35640 1 --2- 172.21.15.16:0/1465227718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35bc101b80 0x7f35bc1a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:32.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.501+0000 7f35c0a35640 1 -- 172.21.15.16:0/1465227718 >> 172.21.15.16:0/1465227718 conn(0x7f35bc077a00 msgr2=0x7f35bc107130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:32.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.501+0000 7f35c0a35640 1 -- 172.21.15.16:0/1465227718 shutdown_connections 2024-08-28T19:34:32.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:32.501+0000 7f35c0a35640 1 -- 172.21.15.16:0/1465227718 wait complete. 2024-08-28T19:34:33.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:34:33.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa461ffb640 1 -- 172.21.15.16:0/3586130889 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5626b228cd20 con 0x5626b1f07570 2024-08-28T19:34:33.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa461ffb640 10 client.28283 handle_command_reply: tid=0 2024-08-28T19:34:33.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 1 client.28283 shutdown 2024-08-28T19:34:33.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 1 -- 172.21.15.16:0/3586130889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f73840 msgr2=0x5626b2329bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:33.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f73840 0x5626b2329bc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa474028660 tx=0x7fa47400bfd0 comp rx=0 tx=0).stop 2024-08-28T19:34:33.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 1 -- 172.21.15.16:0/3586130889 shutdown_connections 2024-08-28T19:34:33.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626b1f73840 0x5626b2329bc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:33.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626b1f52b60 0x5626b2329620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:33.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5626b1f07570 0x5626b23a6780 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:33.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 1 --2- 172.21.15.16:0/3586130889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626b1f4a980 0x5626b232ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa461ffb640 0 client.28283 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 1 -- 172.21.15.16:0/3586130889 >> 172.21.15.16:0/3586130889 conn(0x5626b1f67990 msgr2=0x5626b2324930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 1 -- 172.21.15.16:0/3586130889 shutdown_connections 2024-08-28T19:34:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 1 -- 172.21.15.16:0/3586130889 wait complete. 2024-08-28T19:34:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.379+0000 7fa487545740 20 client.28283 trim_cache size 0 max 16384 2024-08-28T19:34:33.382 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:34:33.382 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 73, 2024-08-28T19:34:33.382 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:34:33.382 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 42, 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "293e425a": { 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 42, 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 145.7, 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:34:33.383 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:34:33.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:34:33.384 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:34:33.384 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:33.384 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:33.384 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:33.384 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:33.384 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:33.384 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 -- 172.21.15.16:0/1233379692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4640b4ca0 msgr2=0x7fa4640b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4640b4ca0 0x7fa4640b71b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa474005fd0 tx=0x7fa474012040 comp rx=0 tx=0).stop 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 -- 172.21.15.16:0/1233379692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c108180 msgr2=0x7fa47c19a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c108180 0x7fa47c19a7d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa47000d740 tx=0x7fa47000db40 comp rx=0 tx=0).stop 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 -- 172.21.15.16:0/1233379692 shutdown_connections 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa47c108ad0 0x7fa47c19fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4640b4ca0 0x7fa4640b71b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c108180 0x7fa47c19a7d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 --2- 172.21.15.16:0/1233379692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa47c102840 0x7fa47c19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 -- 172.21.15.16:0/1233379692 >> 172.21.15.16:0/1233379692 conn(0x7fa47c079530 msgr2=0x7fa47c1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 -- 172.21.15.16:0/1233379692 shutdown_connections 2024-08-28T19:34:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:33.380+0000 7fa480fd0640 1 -- 172.21.15.16:0/1233379692 wait complete. 2024-08-28T19:34:33.460 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 40.94 MB/sec warmup 9 sec latency 8126.456 ms 2024-08-28T19:34:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:33 smithi115 ceph-mon[22184]: pgmap v758: 129 pgs: 129 active+clean; 431 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.3 MiB/s wr, 109 op/s 2024-08-28T19:34:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:33 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:33 smithi016 ceph-mon[18556]: pgmap v758: 129 pgs: 129 active+clean; 431 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.3 MiB/s wr, 109 op/s 2024-08-28T19:34:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:33 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:33 smithi028 ceph-mon[21769]: pgmap v758: 129 pgs: 129 active+clean; 431 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.3 MiB/s wr, 109 op/s 2024-08-28T19:34:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:33 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:34.385 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '293e425a', quiesce took 6.8 seconds. Will release after: 18.9 2024-08-28T19:34:34.422 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:34.460 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 36.86 MB/sec warmup 10 sec latency 9126.523 ms 2024-08-28T19:34:34.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:34 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:34 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:34 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.042+0000 7f7c127ce640 1 -- 172.21.15.16:0/3699540113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c10dbd0 msgr2=0x7f7c0c1100e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.042+0000 7f7c127ce640 1 --2- 172.21.15.16:0/3699540113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c10dbd0 0x7f7c0c1100e0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009ca0 tx=0x7f7c0402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.042+0000 7f7c127ce640 1 -- 172.21.15.16:0/3699540113 shutdown_connections 2024-08-28T19:34:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.042+0000 7f7c127ce640 1 --2- 172.21.15.16:0/3699540113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c10dbd0 0x7f7c0c1100e0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.042+0000 7f7c127ce640 1 --2- 172.21.15.16:0/3699540113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c105050 0x7f7c0c10d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.042+0000 7f7c127ce640 1 --2- 172.21.15.16:0/3699540113 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c104620 0x7f7c0c104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.042+0000 7f7c127ce640 1 -- 172.21.15.16:0/3699540113 >> 172.21.15.16:0/3699540113 conn(0x7f7c0c0794f0 msgr2=0x7f7c0c079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.043+0000 7f7c127ce640 1 -- 172.21.15.16:0/3699540113 shutdown_connections 2024-08-28T19:34:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.043+0000 7f7c127ce640 1 -- 172.21.15.16:0/3699540113 wait complete. 2024-08-28T19:34:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.043+0000 7f7c127ce640 1 Processor -- start 2024-08-28T19:34:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.043+0000 7f7c127ce640 1 -- start start 2024-08-28T19:34:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c127ce640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c104620 0x7f7c0c102530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c127ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c105050 0x7f7c0c102a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c127ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c10dbd0 0x7f7c0c102fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c127ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c113870 con 0x7f7c0c10dbd0 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c127ce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1136f0 con 0x7f7c0c105050 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c127ce640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1139f0 con 0x7f7c0c104620 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c10d6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c10dbd0 0x7f7c0c102fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c0bfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c104620 0x7f7c0c102530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c10d6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c10dbd0 0x7f7c0c102fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35568/0 (socket says 172.21.15.16:35568) 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c10d6b640 1 -- 172.21.15.16:0/4228703875 learned_addr learned my addr 172.21.15.16:0/4228703875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c0bfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c104620 0x7f7c0c102530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:44178/0 (socket says 172.21.15.16:44178) 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c105050 0x7f7c0c102a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c10d6b640 1 -- 172.21.15.16:0/4228703875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c104620 msgr2=0x7f7c0c102530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c10d6b640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c104620 0x7f7c0c102530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c10d6b640 1 -- 172.21.15.16:0/4228703875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c105050 msgr2=0x7f7c0c102a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c10d6b640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c105050 0x7f7c0c102a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.044+0000 7f7c10d6b640 1 -- 172.21.15.16:0/4228703875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c1a8450 con 0x7f7c0c10dbd0 2024-08-28T19:34:35.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.045+0000 7f7c10d6b640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c10dbd0 0x7f7c0c102fb0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7c04005cc0 tx=0x7f7c0402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:35.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.045+0000 7f7c097fa640 1 -- 172.21.15.16:0/4228703875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04003bf0 con 0x7f7c0c10dbd0 2024-08-28T19:34:35.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.045+0000 7f7c127ce640 1 -- 172.21.15.16:0/4228703875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0c1a86e0 con 0x7f7c0c10dbd0 2024-08-28T19:34:35.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.045+0000 7f7c097fa640 1 -- 172.21.15.16:0/4228703875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c04003d90 con 0x7f7c0c10dbd0 2024-08-28T19:34:35.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.045+0000 7f7c097fa640 1 -- 172.21.15.16:0/4228703875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0402fb20 con 0x7f7c0c10dbd0 2024-08-28T19:34:35.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.045+0000 7f7c127ce640 1 -- 172.21.15.16:0/4228703875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c0c1a8c20 con 0x7f7c0c10dbd0 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.047+0000 7f7c18d43740 1 Processor -- start 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.047+0000 7f7c097fa640 1 -- 172.21.15.16:0/4228703875 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7c04004580 con 0x7f7c0c10dbd0 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.047+0000 7f7c18d43740 1 -- start start 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9dab4070 0x556e9de59350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de59890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da93390 0x556e9de59dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c18d43740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556e9ddd0680 con 0x556e9da8b1b0 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c18d43740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e9ddbfc10 con 0x556e9da93390 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c18d43740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556e9ddc3bc0 con 0x556e9dab4070 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de59890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de59890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35582/0 (socket says 172.21.15.16:35582) 2024-08-28T19:34:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0bfff640 1 -- 172.21.15.16:0/1897800087 learned_addr learned my addr 172.21.15.16:0/1897800087 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/1897800087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da93390 0x556e9de59dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/1897800087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9dab4070 0x556e9de59350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0bfff640 1 -- 172.21.15.16:0/1897800087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9dab4070 msgr2=0x556e9de59350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0bfff640 1 --2- 172.21.15.16:0/1897800087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9dab4070 0x556e9de59350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0bfff640 1 -- 172.21.15.16:0/1897800087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da93390 msgr2=0x556e9de59dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0bfff640 1 --2- 172.21.15.16:0/1897800087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da93390 0x556e9de59dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0bfff640 1 -- 172.21.15.16:0/1897800087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e9db1f650 con 0x556e9da8b1b0 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/1897800087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9dab4070 0x556e9de59350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c097fa640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf40b4ca0 0x7f7bf40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.048+0000 7f7c0bfff640 1 --2- 172.21.15.16:0/1897800087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de59890 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc004650 tx=0x7f7bfc00bba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529a44c2d1c904b2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.049+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf40b4ca0 0x7f7bf40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.049+0000 7f7c097fa640 1 -- 172.21.15.16:0/4228703875 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f7c04106df0 con 0x7f7c0c10dbd0 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.049+0000 7f7bf27fc640 1 -- 172.21.15.16:0/1897800087 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc003bd0 con 0x556e9da8b1b0 2024-08-28T19:34:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.049+0000 7f7bf27fc640 1 -- 172.21.15.16:0/1897800087 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bfc015070 con 0x556e9da8b1b0 2024-08-28T19:34:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.049+0000 7f7bf27fc640 1 -- 172.21.15.16:0/1897800087 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc004d80 con 0x556e9da8b1b0 2024-08-28T19:34:35.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c18d43740 1 -- 172.21.15.16:0/1897800087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 msgr2=0x556e9de59890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf40b4ca0 0x7f7bf40b71b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c10f9d0 tx=0x7f7c0000a680 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1897800087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de59890 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc004650 tx=0x7f7bfc00bba0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c18d43740 1 -- 172.21.15.16:0/1897800087 shutdown_connections 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1897800087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da93390 0x556e9de59dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1897800087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de59890 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1897800087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9dab4070 0x556e9de59350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c18d43740 1 -- 172.21.15.16:0/1897800087 >> 172.21.15.16:0/1897800087 conn(0x556e9daa8460 msgr2=0x556e9de57240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c18d43740 1 -- 172.21.15.16:0/1897800087 shutdown_connections 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c18d43740 1 -- 172.21.15.16:0/1897800087 wait complete. 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c18d43740 1 -- start start 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.050+0000 7f7c18d43740 1 Processor -- start 2024-08-28T19:34:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de64ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da93390 0x556e9de65000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9dab4070 0x556e9de70e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c18d43740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556e9dc22700 con 0x556e9da8b1b0 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c18d43740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e9dd46750 con 0x556e9da93390 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c18d43740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556e9de67b30 con 0x556e9dab4070 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de64ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c18d43740 10 client.0 _ll_register_callbacks cb 0x556e9dd75e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de64ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35594/0 (socket says 172.21.15.16:35594) 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c0bfff640 1 -- 172.21.15.16:0/1429719788 learned_addr learned my addr 172.21.15.16:0/1429719788 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da93390 0x556e9de65000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c10d6b640 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9dab4070 0x556e9de70e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c0bfff640 1 -- 172.21.15.16:0/1429719788 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9dab4070 msgr2=0x556e9de70e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c0bfff640 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9dab4070 0x556e9de70e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c0bfff640 1 -- 172.21.15.16:0/1429719788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da93390 msgr2=0x556e9de65000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c0bfff640 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da93390 0x556e9de65000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c0bfff640 1 -- 172.21.15.16:0/1429719788 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e9dd7f900 con 0x556e9da8b1b0 2024-08-28T19:34:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7c0bfff640 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de64ac0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc005880 tx=0x7f7bfc0043a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c9a75974ef9dde server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.051+0000 7f7beb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7beb7fe640 1 -- 172.21.15.16:0/1429719788 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc011440 con 0x556e9da8b1b0 2024-08-28T19:34:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7beb7fe640 1 -- 172.21.15.16:0/1429719788 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f7bfc0115e0 con 0x556e9da8b1b0 2024-08-28T19:34:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7c18d43740 1 -- 172.21.15.16:0/1429719788 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bfc015070 con 0x556e9da8b1b0 2024-08-28T19:34:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7beb7fe640 1 -- 172.21.15.16:0/1429719788 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f7bfc030050 con 0x556e9da8b1b0 2024-08-28T19:34:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7c18d43740 10 client.18867 fetch_fsmap learned FSMap version 73 2024-08-28T19:34:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7c18d43740 1 -- 172.21.15.16:0/1429719788 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x556e9de73870 con 0x556e9da8b1b0 2024-08-28T19:34:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7beb7fe640 1 -- 172.21.15.16:0/1429719788 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bfc015070 con 0x556e9da8b1b0 2024-08-28T19:34:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7beb7fe640 1 -- 172.21.15.16:0/1429719788 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) ==== 8434+0+0 (secure 0 0 0) 0x7f7bfc0257a0 con 0x556e9da8b1b0 2024-08-28T19:34:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7c18d43740 10 client.18867 fetch_fsmap finished waiting for FSMap version 73 2024-08-28T19:34:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7c18d43740 10 client.18867 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:34:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7c18d43740 20 client.18867 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e9da729f0 0x556e9de76c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7c18d43740 4 client.18867 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:34:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.052+0000 7f7c18d43740 1 -- 172.21.15.16:0/1429719788 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e9ddd25d0 con 0x556e9da729f0 2024-08-28T19:34:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.053+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e9da729f0 0x556e9de76c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.053+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e9da729f0 0x556e9de76c80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c9bb549b2c752538 server_cookie=64728a44a5deb494 in_seq=0 out_seq=0 2024-08-28T19:34:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.053+0000 7f7beb7fe640 10 client.18867 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:34:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7beb7fe640 1 -- 172.21.15.16:0/1429719788 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556e9ddd25d0 con 0x556e9da729f0 2024-08-28T19:34:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7beb7fe640 10 client.18867 handle_command_reply: tid=0 2024-08-28T19:34:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 1 client.18867 shutdown 2024-08-28T19:34:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 1 -- 172.21.15.16:0/1429719788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 msgr2=0x556e9de64ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de64ac0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc005880 tx=0x7f7bfc0043a0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 1 -- 172.21.15.16:0/1429719788 shutdown_connections 2024-08-28T19:34:35.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e9da729f0 0x556e9de76c80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9dab4070 0x556e9de70e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da93390 0x556e9de65000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1429719788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da8b1b0 0x556e9de64ac0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 1 -- 172.21.15.16:0/1429719788 >> 172.21.15.16:0/1429719788 conn(0x556e9daa8460 msgr2=0x556e9de5c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:35.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7beb7fe640 0 client.18867 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:34:35.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 1 -- 172.21.15.16:0/1429719788 shutdown_connections 2024-08-28T19:34:35.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 1 -- 172.21.15.16:0/1429719788 wait complete. 2024-08-28T19:34:35.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.054+0000 7f7c18d43740 20 client.18867 trim_cache size 0 max 16384 2024-08-28T19:34:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c18d43740 1 Processor -- start 2024-08-28T19:34:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c18d43740 1 -- start start 2024-08-28T19:34:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da8b1b0 0x556e9de63340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9da93390 0x556e9de63880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9dab4070 0x556e9de5c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c18d43740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b2ea0 con 0x556e9dab4070 2024-08-28T19:34:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c18d43740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b4c20 con 0x556e9da8b1b0 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c18d43740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b3600 con 0x556e9da93390 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c10d6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9dab4070 0x556e9de5c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c10d6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9dab4070 0x556e9de5c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35598/0 (socket says 172.21.15.16:35598) 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c10d6b640 1 -- 172.21.15.16:0/1179277667 learned_addr learned my addr 172.21.15.16:0/1179277667 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/1179277667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9da93390 0x556e9de63880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c0bfff640 1 --2- 172.21.15.16:0/1179277667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da8b1b0 0x556e9de63340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c10d6b640 1 -- 172.21.15.16:0/1179277667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9da93390 msgr2=0x556e9de63880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c10d6b640 1 --2- 172.21.15.16:0/1179277667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9da93390 0x556e9de63880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c10d6b640 1 -- 172.21.15.16:0/1179277667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da8b1b0 msgr2=0x556e9de63340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c10d6b640 1 --2- 172.21.15.16:0/1179277667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da8b1b0 0x556e9de63340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c10d6b640 1 -- 172.21.15.16:0/1179277667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e9de715c0 con 0x556e9dab4070 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c0bfff640 1 --2- 172.21.15.16:0/1179277667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da8b1b0 0x556e9de63340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.068+0000 7f7c10d6b640 1 --2- 172.21.15.16:0/1179277667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9dab4070 0x556e9de5c450 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f7c040c9190 tx=0x7f7c040f0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe3dd0911fd99dd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.069+0000 7f7bf1ffb640 1 -- 172.21.15.16:0/1179277667 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040ccc20 con 0x556e9dab4070 2024-08-28T19:34:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.069+0000 7f7bf1ffb640 1 -- 172.21.15.16:0/1179277667 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c04036030 con 0x556e9dab4070 2024-08-28T19:34:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.069+0000 7f7bf1ffb640 1 -- 172.21.15.16:0/1179277667 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0402fcc0 con 0x556e9dab4070 2024-08-28T19:34:35.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.069+0000 7f7c18d43740 1 -- 172.21.15.16:0/1179277667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9dab4070 msgr2=0x556e9de5c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:35.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.069+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1179277667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9dab4070 0x556e9de5c450 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f7c040c9190 tx=0x7f7c040f0040 comp rx=0 tx=0).stop 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.069+0000 7f7c18d43740 1 -- 172.21.15.16:0/1179277667 shutdown_connections 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.069+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1179277667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9dab4070 0x556e9de5c450 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.069+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1179277667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9da93390 0x556e9de63880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.069+0000 7f7c18d43740 1 --2- 172.21.15.16:0/1179277667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9da8b1b0 0x556e9de63340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.069+0000 7f7c18d43740 1 -- 172.21.15.16:0/1179277667 >> 172.21.15.16:0/1179277667 conn(0x556e9daa8460 msgr2=0x556e9de5fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.070+0000 7f7c18d43740 1 -- 172.21.15.16:0/1179277667 shutdown_connections 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.070+0000 7f7c18d43740 1 -- 172.21.15.16:0/1179277667 wait complete. 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.070+0000 7f7c18d43740 1 -- start start 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.070+0000 7f7c18d43740 1 Processor -- start 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.070+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9da8b1b0 0x556e9de6d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.070+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da93390 0x556e9de68900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.070+0000 7f7c18d43740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9dab4070 0x556e9de68e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.070+0000 7f7c18d43740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b4520 con 0x556e9da93390 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.070+0000 7f7c18d43740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b3d90 con 0x556e9dab4070 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.070+0000 7f7c18d43740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b0ea0 con 0x556e9da8b1b0 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.070+0000 7f7c18d43740 10 client.0 _ll_register_callbacks cb 0x7f7bfc004550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c0b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da93390 0x556e9de68900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c0b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da93390 0x556e9de68900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35606/0 (socket says 172.21.15.16:35606) 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c0b7fe640 1 -- 172.21.15.16:0/42372990 learned_addr learned my addr 172.21.15.16:0/42372990 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c0bfff640 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9da8b1b0 0x556e9de6d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c10d6b640 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9dab4070 0x556e9de68e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c0b7fe640 1 -- 172.21.15.16:0/42372990 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9da8b1b0 msgr2=0x556e9de6d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9da8b1b0 0x556e9de6d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c0b7fe640 1 -- 172.21.15.16:0/42372990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9dab4070 msgr2=0x556e9de68e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9dab4070 0x556e9de68e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c0b7fe640 1 -- 172.21.15.16:0/42372990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e9de694b0 con 0x556e9da93390 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da93390 0x556e9de68900 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x556e9de77320 tx=0x7f7c00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ea6c7db3499bb9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7beb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7beb7fe640 1 -- 172.21.15.16:0/42372990 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00005020 con 0x556e9da93390 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7beb7fe640 1 -- 172.21.15.16:0/42372990 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f7c00017030 con 0x556e9da93390 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.071+0000 7f7c18d43740 1 -- 172.21.15.16:0/42372990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c04036030 con 0x556e9da93390 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7beb7fe640 1 -- 172.21.15.16:0/42372990 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f7c000051c0 con 0x556e9da93390 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7c18d43740 10 client.18873 fetch_fsmap learned FSMap version 73 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7c18d43740 1 -- 172.21.15.16:0/42372990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x556e9dee1380 con 0x556e9da93390 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7beb7fe640 1 -- 172.21.15.16:0/42372990 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c04036030 con 0x556e9da93390 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7beb7fe640 1 -- 172.21.15.16:0/42372990 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) ==== 8434+0+0 (secure 0 0 0) 0x7f7c000305a0 con 0x556e9da93390 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7c18d43740 10 client.18873 fetch_fsmap finished waiting for FSMap version 73 2024-08-28T19:34:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7c18d43740 10 client.18873 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7c18d43740 20 client.18873 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7c18d43740 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e9da47da0 0x556e9dee3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7c18d43740 4 client.18873 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7c18d43740 1 -- 172.21.15.16:0/42372990 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e9ddce070 con 0x556e9da47da0 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.072+0000 7f7c0bfff640 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e9da47da0 0x556e9dee3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7beb7fe640 10 client.18873 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c0bfff640 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e9da47da0 0x556e9dee3c30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=99aab93bc825a883 server_cookie=5ea5f547c87e267f in_seq=0 out_seq=0 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7beb7fe640 1 -- 172.21.15.16:0/42372990 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e9ddce070 con 0x556e9da47da0 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7beb7fe640 10 client.18873 handle_command_reply: tid=0 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 1 client.18873 shutdown 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 1 -- 172.21.15.16:0/42372990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da93390 msgr2=0x556e9de68900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da93390 0x556e9de68900 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x556e9de77320 tx=0x7f7c00004730 comp rx=0 tx=0).stop 2024-08-28T19:34:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 1 -- 172.21.15.16:0/42372990 shutdown_connections 2024-08-28T19:34:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x556e9da47da0 0x556e9dee3c30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9dab4070 0x556e9de68e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e9da93390 0x556e9de68900 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 1 --2- 172.21.15.16:0/42372990 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556e9da8b1b0 0x556e9de6d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7beb7fe640 0 client.18873 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:34:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 1 -- 172.21.15.16:0/42372990 >> 172.21.15.16:0/42372990 conn(0x556e9daa8460 msgr2=0x556e9de5c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 1 -- 172.21.15.16:0/42372990 shutdown_connections 2024-08-28T19:34:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 1 -- 172.21.15.16:0/42372990 wait complete. 2024-08-28T19:34:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.073+0000 7f7c18d43740 20 client.18873 trim_cache size 0 max 16384 2024-08-28T19:34:35.080 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:34:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 -- 172.21.15.16:0/4228703875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf40b4ca0 msgr2=0x7f7bf40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf40b4ca0 0x7f7bf40b71b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c10f9d0 tx=0x7f7c0000a680 comp rx=0 tx=0).stop 2024-08-28T19:34:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 -- 172.21.15.16:0/4228703875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c10dbd0 msgr2=0x7f7c0c102fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c10dbd0 0x7f7c0c102fb0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7c04005cc0 tx=0x7f7c0402f380 comp rx=0 tx=0).stop 2024-08-28T19:34:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 -- 172.21.15.16:0/4228703875 shutdown_connections 2024-08-28T19:34:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c10dbd0 0x7f7c0c102fb0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf40b4ca0 0x7f7bf40b71b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c105050 0x7f7c0c102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 --2- 172.21.15.16:0/4228703875 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c104620 0x7f7c0c102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 -- 172.21.15.16:0/4228703875 >> 172.21.15.16:0/4228703875 conn(0x7f7c0c0794f0 msgr2=0x7f7c0c0ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 -- 172.21.15.16:0/4228703875 shutdown_connections 2024-08-28T19:34:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:35.074+0000 7f7c127ce640 1 -- 172.21.15.16:0/4228703875 wait complete. 2024-08-28T19:34:35.460 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 33.52 MB/sec warmup 11 sec latency 10126.606 ms 2024-08-28T19:34:35.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:35 smithi115 ceph-mon[22184]: pgmap v759: 129 pgs: 129 active+clean; 459 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 172 op/s 2024-08-28T19:34:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:35 smithi016 ceph-mon[18556]: pgmap v759: 129 pgs: 129 active+clean; 459 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 172 op/s 2024-08-28T19:34:35.791 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:34:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:35 smithi028 ceph-mon[21769]: pgmap v759: 129 pgs: 129 active+clean; 459 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 172 op/s 2024-08-28T19:34:36.085 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:36.460 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 30.73 MB/sec warmup 12 sec latency 11126.699 ms 2024-08-28T19:34:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.804+0000 7f445bdb6640 1 -- 172.21.15.16:0/3041883079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4454108a90 msgr2=0x7f445410f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.804+0000 7f445bdb6640 1 --2- 172.21.15.16:0/3041883079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4454108a90 0x7f445410f2c0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f444c009ca0 tx=0x7f444c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.804+0000 7f445bdb6640 1 -- 172.21.15.16:0/3041883079 shutdown_connections 2024-08-28T19:34:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.804+0000 7f445bdb6640 1 --2- 172.21.15.16:0/3041883079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4454108a90 0x7f445410f2c0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.804+0000 7f445bdb6640 1 --2- 172.21.15.16:0/3041883079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4454108140 0x7f4454108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.804+0000 7f445bdb6640 1 --2- 172.21.15.16:0/3041883079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4454102800 0x7f4454102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.804+0000 7f445bdb6640 1 -- 172.21.15.16:0/3041883079 >> 172.21.15.16:0/3041883079 conn(0x7f44540794d0 msgr2=0x7f44540798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.804+0000 7f445bdb6640 1 -- 172.21.15.16:0/3041883079 shutdown_connections 2024-08-28T19:34:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.804+0000 7f445bdb6640 1 -- 172.21.15.16:0/3041883079 wait complete. 2024-08-28T19:34:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f445bdb6640 1 Processor -- start 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f445bdb6640 1 -- start start 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f445bdb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4454102800 0x7f445419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f445bdb6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4454108140 0x7f445419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f445bdb6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4454108a90 0x7f445419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f445bdb6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4454113820 con 0x7f4454108a90 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f445bdb6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44541136a0 con 0x7f4454102800 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f445bdb6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f44541139a0 con 0x7f4454108140 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f445a353640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4454108a90 0x7f445419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f445a353640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4454108a90 0x7f445419fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35626/0 (socket says 172.21.15.16:35626) 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f445a353640 1 -- 172.21.15.16:0/2299765968 learned_addr learned my addr 172.21.15.16:0/2299765968 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f4459351640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4454108140 0x7f445419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.805+0000 7f4459b52640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4454102800 0x7f445419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.806+0000 7f445a353640 1 -- 172.21.15.16:0/2299765968 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4454108140 msgr2=0x7f445419a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.806+0000 7f445a353640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4454108140 0x7f445419a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.806+0000 7f445a353640 1 -- 172.21.15.16:0/2299765968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4454102800 msgr2=0x7f445419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.806+0000 7f445a353640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4454102800 0x7f445419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.806+0000 7f445a353640 1 -- 172.21.15.16:0/2299765968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44541a01e0 con 0x7f4454108a90 2024-08-28T19:34:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.806+0000 7f445a353640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4454108a90 0x7f445419fab0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f444c005cc0 tx=0x7f444c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.806+0000 7f444affd640 1 -- 172.21.15.16:0/2299765968 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c003bf0 con 0x7f4454108a90 2024-08-28T19:34:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.806+0000 7f445bdb6640 1 -- 172.21.15.16:0/2299765968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44541a0470 con 0x7f4454108a90 2024-08-28T19:34:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.806+0000 7f444affd640 1 -- 172.21.15.16:0/2299765968 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f444c003d90 con 0x7f4454108a90 2024-08-28T19:34:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.806+0000 7f444affd640 1 -- 172.21.15.16:0/2299765968 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c02fbc0 con 0x7f4454108a90 2024-08-28T19:34:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.806+0000 7f445bdb6640 1 -- 172.21.15.16:0/2299765968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44541a6c30 con 0x7f4454108a90 2024-08-28T19:34:36.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.809+0000 7f446232b740 1 Processor -- start 2024-08-28T19:34:36.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.810+0000 7f444affd640 1 -- 172.21.15.16:0/2299765968 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f444c02fd60 con 0x7f4454108a90 2024-08-28T19:34:36.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.810+0000 7f446232b740 1 -- start start 2024-08-28T19:34:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.810+0000 7f446232b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb2d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.810+0000 7f446232b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 0x55d0fb2d3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f446232b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf0d390 0x55d0fb2d3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f446232b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d0fb24a680 con 0x55d0faf2e070 2024-08-28T19:34:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f446232b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d0fb239c10 con 0x55d0faf0d390 2024-08-28T19:34:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f446232b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d0fb23dbc0 con 0x55d0faf051b0 2024-08-28T19:34:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f4459351640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb2d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f4459351640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb2d3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35630/0 (socket says 172.21.15.16:35630) 2024-08-28T19:34:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f4459351640 1 -- 172.21.15.16:0/558317160 learned_addr learned my addr 172.21.15.16:0/558317160 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f4459b52640 1 --2- 172.21.15.16:0/558317160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 0x55d0fb2d3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f444affd640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44400b4ca0 0x7f44400b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f4459351640 1 --2- 172.21.15.16:0/558317160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf0d390 0x55d0fb2d3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f4459351640 1 -- 172.21.15.16:0/558317160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 msgr2=0x55d0fb2d3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f4459351640 1 --2- 172.21.15.16:0/558317160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 0x55d0fb2d3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f4459351640 1 -- 172.21.15.16:0/558317160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf0d390 msgr2=0x55d0fb2d3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f4459351640 1 --2- 172.21.15.16:0/558317160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf0d390 0x55d0fb2d3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f4459351640 1 -- 172.21.15.16:0/558317160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0faf99650 con 0x55d0faf2e070 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f444affd640 1 -- 172.21.15.16:0/2299765968 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f444c106e20 con 0x7f4454108a90 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f445a353640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44400b4ca0 0x7f44400b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.811+0000 7f4459351640 1 --2- 172.21.15.16:0/558317160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb2d3310 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f4444005d70 tx=0x7f4444004190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=119b185a744bad5f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.812+0000 7f443ffff640 1 -- 172.21.15.16:0/558317160 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444003e40 con 0x55d0faf2e070 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.812+0000 7f443ffff640 1 -- 172.21.15.16:0/558317160 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44440054d0 con 0x55d0faf2e070 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.812+0000 7f443ffff640 1 -- 172.21.15.16:0/558317160 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444010660 con 0x55d0faf2e070 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f446232b740 1 -- 172.21.15.16:0/558317160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 msgr2=0x55d0fb2d3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f446232b740 1 --2- 172.21.15.16:0/558317160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb2d3310 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f4444005d70 tx=0x7f4444004190 comp rx=0 tx=0).stop 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f445a353640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44400b4ca0 0x7f44400b71b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f444c0cce40 tx=0x7f444c0b4040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f446232b740 1 -- 172.21.15.16:0/558317160 shutdown_connections 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f446232b740 1 --2- 172.21.15.16:0/558317160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf0d390 0x55d0fb2d3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f446232b740 1 --2- 172.21.15.16:0/558317160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 0x55d0fb2d3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f446232b740 1 --2- 172.21.15.16:0/558317160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb2d3310 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f446232b740 1 -- 172.21.15.16:0/558317160 >> 172.21.15.16:0/558317160 conn(0x55d0faf22460 msgr2=0x55d0fb2d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f446232b740 1 -- 172.21.15.16:0/558317160 shutdown_connections 2024-08-28T19:34:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f446232b740 1 -- 172.21.15.16:0/558317160 wait complete. 2024-08-28T19:34:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f446232b740 1 -- start start 2024-08-28T19:34:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.813+0000 7f446232b740 1 Processor -- start 2024-08-28T19:34:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f446232b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 0x55d0fb2dea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f446232b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf0d390 0x55d0fb2defb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f446232b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb2eadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f446232b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d0fb09c700 con 0x55d0faf2e070 2024-08-28T19:34:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f446232b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d0fb1c0750 con 0x55d0faf0d390 2024-08-28T19:34:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f446232b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d0fb2e1a90 con 0x55d0faf051b0 2024-08-28T19:34:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f446232b740 10 client.0 _ll_register_callbacks cb 0x55d0fb1efe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f4459351640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb2eadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f4459b52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf0d390 0x55d0fb2defb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f4459351640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb2eadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35632/0 (socket says 172.21.15.16:35632) 2024-08-28T19:34:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f4459351640 1 -- 172.21.15.16:0/4274225634 learned_addr learned my addr 172.21.15.16:0/4274225634 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f4459351640 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 0x55d0fb2dea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f4459b52640 1 -- 172.21.15.16:0/4274225634 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 msgr2=0x55d0fb2dea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f4459b52640 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 0x55d0fb2dea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f4459b52640 1 -- 172.21.15.16:0/4274225634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 msgr2=0x55d0fb2eadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f4459b52640 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb2eadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.814+0000 7f4459b52640 1 -- 172.21.15.16:0/4274225634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0fb1f9900 con 0x55d0faf0d390 2024-08-28T19:34:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.815+0000 7f4459b52640 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf0d390 0x55d0fb2defb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f445000baa0 tx=0x7f445000bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=629dc5e43de08d4e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.815+0000 7f443d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:34:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.815+0000 7f443d7fa640 1 -- 172.21.15.16:0/4274225634 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4450003df0 con 0x55d0faf0d390 2024-08-28T19:34:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.815+0000 7f446232b740 1 -- 172.21.15.16:0/4274225634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44440054d0 con 0x55d0faf0d390 2024-08-28T19:34:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.815+0000 7f443d7fa640 1 -- 172.21.15.16:0/4274225634 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f4450005020 con 0x55d0faf0d390 2024-08-28T19:34:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.815+0000 7f443d7fa640 1 -- 172.21.15.16:0/4274225634 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f4450024050 con 0x55d0faf0d390 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.815+0000 7f446232b740 10 client.28325 fetch_fsmap learned FSMap version 73 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.815+0000 7f446232b740 1 -- 172.21.15.16:0/4274225634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55d0fb2ed8e0 con 0x55d0faf0d390 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.816+0000 7f443d7fa640 1 -- 172.21.15.16:0/4274225634 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44440054d0 con 0x55d0faf0d390 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.816+0000 7f443d7fa640 1 -- 172.21.15.16:0/4274225634 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 73) ==== 8434+0+0 (secure 0 0 0) 0x7f445001b470 con 0x55d0faf0d390 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.816+0000 7f446232b740 10 client.28325 fetch_fsmap finished waiting for FSMap version 73 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.816+0000 7f446232b740 10 client.28325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.816+0000 7f446232b740 20 client.28325 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.816+0000 7f446232b740 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0faeec9f0 0x55d0fb2f0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.816+0000 7f446232b740 4 client.28325 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.816+0000 7f446232b740 1 -- 172.21.15.16:0/4274225634 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0fb24c5d0 con 0x55d0faeec9f0 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.816+0000 7f4459351640 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0faeec9f0 0x55d0fb2f0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.816+0000 7f4459351640 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0faeec9f0 0x55d0fb2f0e40 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5166bc1c19f60e30 server_cookie=de4b398dc035c18e in_seq=0 out_seq=0 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.817+0000 7f443d7fa640 10 client.28325 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.817+0000 7f443d7fa640 1 -- 172.21.15.16:0/4274225634 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d0fb24c5d0 con 0x55d0faeec9f0 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.817+0000 7f443d7fa640 10 client.28325 handle_command_reply: tid=0 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.817+0000 7f446232b740 1 client.28325 shutdown 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.817+0000 7f446232b740 1 -- 172.21.15.16:0/4274225634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf0d390 msgr2=0x55d0fb2defb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.817+0000 7f446232b740 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf0d390 0x55d0fb2defb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f445000baa0 tx=0x7f445000bf60 comp rx=0 tx=0).stop 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.818+0000 7f446232b740 1 -- 172.21.15.16:0/4274225634 shutdown_connections 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.818+0000 7f446232b740 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb2eadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.818+0000 7f446232b740 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf0d390 0x55d0fb2defb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.818+0000 7f446232b740 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0faeec9f0 0x55d0fb2f0e40 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.818+0000 7f446232b740 1 --2- 172.21.15.16:0/4274225634 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 0x55d0fb2dea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.818+0000 7f443d7fa640 0 client.28325 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.818+0000 7f446232b740 1 -- 172.21.15.16:0/4274225634 >> 172.21.15.16:0/4274225634 conn(0x55d0faf22460 msgr2=0x55d0fb2d64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.818+0000 7f446232b740 1 -- 172.21.15.16:0/4274225634 shutdown_connections 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.818+0000 7f446232b740 1 -- 172.21.15.16:0/4274225634 wait complete. 2024-08-28T19:34:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.818+0000 7f446232b740 20 client.28325 trim_cache size 0 max 16384 2024-08-28T19:34:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.831+0000 7f446232b740 1 Processor -- start 2024-08-28T19:34:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.831+0000 7f446232b740 1 -- start start 2024-08-28T19:34:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.831+0000 7f446232b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 0x55d0fb2d6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.831+0000 7f446232b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf0d390 0x55d0fb2eacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.831+0000 7f446232b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf2e070 0x55d0fb2eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.831+0000 7f446232b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44541aeaf0 con 0x55d0faf0d390 2024-08-28T19:34:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.831+0000 7f446232b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44541b0870 con 0x55d0faf2e070 2024-08-28T19:34:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.831+0000 7f446232b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f44541af250 con 0x55d0faf051b0 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.831+0000 7f4459b52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf0d390 0x55d0fb2eacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.831+0000 7f4459b52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf0d390 0x55d0fb2eacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35644/0 (socket says 172.21.15.16:35644) 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.831+0000 7f4459b52640 1 -- 172.21.15.16:0/2667213124 learned_addr learned my addr 172.21.15.16:0/2667213124 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.832+0000 7f4459351640 1 --2- 172.21.15.16:0/2667213124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf2e070 0x55d0fb2eb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.832+0000 7f4459b52640 1 -- 172.21.15.16:0/2667213124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 msgr2=0x55d0fb2d6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.832+0000 7f4459b52640 1 --2- 172.21.15.16:0/2667213124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 0x55d0fb2d6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.832+0000 7f4459b52640 1 -- 172.21.15.16:0/2667213124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf2e070 msgr2=0x55d0fb2eb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.832+0000 7f4459b52640 1 --2- 172.21.15.16:0/2667213124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf2e070 0x55d0fb2eb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.832+0000 7f4459b52640 1 -- 172.21.15.16:0/2667213124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0fb2d5ba0 con 0x55d0faf0d390 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.832+0000 7f4459351640 1 --2- 172.21.15.16:0/2667213124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf2e070 0x55d0fb2eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.832+0000 7f4459b52640 1 --2- 172.21.15.16:0/2667213124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf0d390 0x55d0fb2eacb0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f445000c010 tx=0x7f445001ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da26c202e4842f2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.832+0000 7f443f7fe640 1 -- 172.21.15.16:0/2667213124 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445002c030 con 0x55d0faf0d390 2024-08-28T19:34:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.832+0000 7f443f7fe640 1 -- 172.21.15.16:0/2667213124 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f445000b400 con 0x55d0faf0d390 2024-08-28T19:34:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.832+0000 7f443f7fe640 1 -- 172.21.15.16:0/2667213124 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445001bbe0 con 0x55d0faf0d390 2024-08-28T19:34:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.833+0000 7f446232b740 1 -- 172.21.15.16:0/2667213124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf0d390 msgr2=0x55d0fb2eacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.833+0000 7f446232b740 1 --2- 172.21.15.16:0/2667213124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf0d390 0x55d0fb2eacb0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f445000c010 tx=0x7f445001ce90 comp rx=0 tx=0).stop 2024-08-28T19:34:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.833+0000 7f446232b740 1 -- 172.21.15.16:0/2667213124 shutdown_connections 2024-08-28T19:34:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.833+0000 7f446232b740 1 --2- 172.21.15.16:0/2667213124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf2e070 0x55d0fb2eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.833+0000 7f446232b740 1 --2- 172.21.15.16:0/2667213124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf0d390 0x55d0fb2eacb0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.833+0000 7f446232b740 1 --2- 172.21.15.16:0/2667213124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf051b0 0x55d0fb2d6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.833+0000 7f446232b740 1 -- 172.21.15.16:0/2667213124 >> 172.21.15.16:0/2667213124 conn(0x55d0faf22460 msgr2=0x55d0fb2d6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.833+0000 7f446232b740 1 -- 172.21.15.16:0/2667213124 shutdown_connections 2024-08-28T19:34:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.833+0000 7f446232b740 1 -- 172.21.15.16:0/2667213124 wait complete. 2024-08-28T19:34:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.833+0000 7f446232b740 1 -- start start 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.833+0000 7f446232b740 1 Processor -- start 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f446232b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf051b0 0x55d0fb2e3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f446232b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf0d390 0x55d0fb2e36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f446232b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb35a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f446232b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44541b0170 con 0x55d0faf2e070 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f446232b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44541af9e0 con 0x55d0faf051b0 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f446232b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f44541acaf0 con 0x55d0faf0d390 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f446232b740 10 client.0 _ll_register_callbacks cb 0x7f4450004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459351640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb35a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459351640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb35a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35658/0 (socket says 172.21.15.16:35658) 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459351640 1 -- 172.21.15.16:0/835820354 learned_addr learned my addr 172.21.15.16:0/835820354 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459b52640 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf0d390 0x55d0fb2e36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459351640 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf051b0 0x55d0fb2e3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459351640 1 -- 172.21.15.16:0/835820354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf0d390 msgr2=0x55d0fb2e36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459351640 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf0d390 0x55d0fb2e36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459351640 1 -- 172.21.15.16:0/835820354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf051b0 msgr2=0x55d0fb2e3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459351640 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf051b0 0x55d0fb2e3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459351640 1 -- 172.21.15.16:0/835820354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0fb35ad40 con 0x55d0faf2e070 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459b52640 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf0d390 0x55d0fb2e36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f4459351640 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb35a800 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f4444004c50 tx=0x7f4444004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6fb5882c2f60be0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.834+0000 7f443d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f443d7fa640 1 -- 172.21.15.16:0/835820354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444021040 con 0x55d0faf2e070 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f443d7fa640 1 -- 172.21.15.16:0/835820354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f4444003df0 con 0x55d0faf2e070 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f446232b740 1 -- 172.21.15.16:0/835820354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f445000b400 con 0x55d0faf2e070 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f443d7fa640 1 -- 172.21.15.16:0/835820354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f444403d050 con 0x55d0faf2e070 2024-08-28T19:34:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f446232b740 10 client.18891 fetch_fsmap learned FSMap version 73 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f446232b740 1 -- 172.21.15.16:0/835820354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55d0fb35be30 con 0x55d0faf2e070 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f443d7fa640 1 -- 172.21.15.16:0/835820354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f445000b400 con 0x55d0faf2e070 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f443d7fa640 1 -- 172.21.15.16:0/835820354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) ==== 8434+0+0 (secure 0 0 0) 0x7f4444027040 con 0x55d0faf2e070 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f446232b740 10 client.18891 fetch_fsmap finished waiting for FSMap version 73 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f446232b740 10 client.18891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f446232b740 20 client.18891 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f446232b740 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0faec1da0 0x55d0fb35e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f446232b740 4 client.18891 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.835+0000 7f446232b740 1 -- 172.21.15.16:0/835820354 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0fb248070 con 0x55d0faec1da0 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.836+0000 7f4459b52640 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0faec1da0 0x55d0fb35e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.836+0000 7f4459b52640 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0faec1da0 0x55d0fb35e290 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c14f2b46e40aeca7 server_cookie=efec9fb2f0228afb in_seq=0 out_seq=0 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.836+0000 7f443d7fa640 10 client.18891 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.836+0000 7f443d7fa640 1 -- 172.21.15.16:0/835820354 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0fb248070 con 0x55d0faec1da0 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.836+0000 7f443d7fa640 10 client.18891 handle_command_reply: tid=0 2024-08-28T19:34:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.836+0000 7f446232b740 1 client.18891 shutdown 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f446232b740 1 -- 172.21.15.16:0/835820354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 msgr2=0x55d0fb35a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f446232b740 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb35a800 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f4444004c50 tx=0x7f4444004be0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f446232b740 1 -- 172.21.15.16:0/835820354 shutdown_connections 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f446232b740 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0faf2e070 0x55d0fb35a800 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f446232b740 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0faec1da0 0x55d0fb35e290 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f446232b740 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0faf0d390 0x55d0fb2e36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f446232b740 1 --2- 172.21.15.16:0/835820354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0faf051b0 0x55d0fb2e3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f443d7fa640 0 client.18891 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f446232b740 1 -- 172.21.15.16:0/835820354 >> 172.21.15.16:0/835820354 conn(0x55d0faf22460 msgr2=0x55d0fb2d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f446232b740 1 -- 172.21.15.16:0/835820354 shutdown_connections 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f446232b740 1 -- 172.21.15.16:0/835820354 wait complete. 2024-08-28T19:34:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f446232b740 20 client.18891 trim_cache size 0 max 16384 2024-08-28T19:34:36.844 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:34:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f445bdb6640 1 -- 172.21.15.16:0/2299765968 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44400b4ca0 msgr2=0x7f44400b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f445bdb6640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44400b4ca0 0x7f44400b71b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f444c0cce40 tx=0x7f444c0b4040 comp rx=0 tx=0).stop 2024-08-28T19:34:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f445bdb6640 1 -- 172.21.15.16:0/2299765968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4454108a90 msgr2=0x7f445419fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.837+0000 7f445bdb6640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4454108a90 0x7f445419fab0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f444c005cc0 tx=0x7f444c02f380 comp rx=0 tx=0).stop 2024-08-28T19:34:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.838+0000 7f445bdb6640 1 -- 172.21.15.16:0/2299765968 shutdown_connections 2024-08-28T19:34:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.838+0000 7f445bdb6640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4454108a90 0x7f445419fab0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.838+0000 7f445bdb6640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4454108140 0x7f445419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.838+0000 7f445bdb6640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44400b4ca0 0x7f44400b71b0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.838+0000 7f445bdb6640 1 --2- 172.21.15.16:0/2299765968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4454102800 0x7f445419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.838+0000 7f445bdb6640 1 -- 172.21.15.16:0/2299765968 >> 172.21.15.16:0/2299765968 conn(0x7f44540794d0 msgr2=0x7f4454104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.838+0000 7f445bdb6640 1 -- 172.21.15.16:0/2299765968 shutdown_connections 2024-08-28T19:34:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:36.838+0000 7f445bdb6640 1 -- 172.21.15.16:0/2299765968 wait complete. 2024-08-28T19:34:37.460 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 28.37 MB/sec warmup 13 sec latency 12126.767 ms 2024-08-28T19:34:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:37 smithi115 ceph-mon[22184]: pgmap v760: 129 pgs: 129 active+clean; 490 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.3 MiB/s wr, 230 op/s 2024-08-28T19:34:37.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:37 smithi016 ceph-mon[18556]: pgmap v760: 129 pgs: 129 active+clean; 490 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.3 MiB/s wr, 230 op/s 2024-08-28T19:34:37.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:37 smithi028 ceph-mon[21769]: pgmap v760: 129 pgs: 129 active+clean; 490 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.3 MiB/s wr, 230 op/s 2024-08-28T19:34:38.460 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 26.35 MB/sec warmup 14 sec latency 13126.816 ms 2024-08-28T19:34:38.502 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:34:38.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:34:38.768 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:39.461 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 24.60 MB/sec warmup 15 sec latency 14126.906 ms 2024-08-28T19:34:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.278+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1021085198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d80102910 msgr2=0x7f3d80107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.278+0000 7f3d87eb6640 1 --2- 172.21.15.16:0/1021085198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d80102910 0x7f3d80107e40 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009ca0 tx=0x7f3d7802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.278+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1021085198 shutdown_connections 2024-08-28T19:34:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.278+0000 7f3d87eb6640 1 --2- 172.21.15.16:0/1021085198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d80102910 0x7f3d80107e40 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.278+0000 7f3d87eb6640 1 --2- 172.21.15.16:0/1021085198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d80102500 0x7f3d80107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.278+0000 7f3d87eb6640 1 --2- 172.21.15.16:0/1021085198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8006b0b0 0x7f3d80101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.278+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1021085198 >> 172.21.15.16:0/1021085198 conn(0x7f3d80079bd0 msgr2=0x7f3d80079fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.278+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1021085198 shutdown_connections 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.278+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1021085198 wait complete. 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.279+0000 7f3d87eb6640 1 Processor -- start 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.279+0000 7f3d87eb6640 1 -- start start 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.279+0000 7f3d87eb6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d8006b0b0 0x7f3d80198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.279+0000 7f3d87eb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d80102500 0x7f3d80199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.279+0000 7f3d87eb6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d80102910 0x7f3d80199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.279+0000 7f3d87eb6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3d80112770 con 0x7f3d8006b0b0 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.279+0000 7f3d87eb6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d801125f0 con 0x7f3d80102500 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.279+0000 7f3d87eb6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3d801128f0 con 0x7f3d80102910 2024-08-28T19:34:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.279+0000 7f3d85c52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d8006b0b0 0x7f3d80198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.279+0000 7f3d85c52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d8006b0b0 0x7f3d80198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35676/0 (socket says 172.21.15.16:35676) 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.279+0000 7f3d85c52640 1 -- 172.21.15.16:0/1099852334 learned_addr learned my addr 172.21.15.16:0/1099852334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d86453640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d80102910 0x7f3d80199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d85451640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d80102500 0x7f3d80199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d85c52640 1 -- 172.21.15.16:0/1099852334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d80102910 msgr2=0x7f3d80199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d85c52640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d80102910 0x7f3d80199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d85c52640 1 -- 172.21.15.16:0/1099852334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d80102500 msgr2=0x7f3d80199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d85c52640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d80102500 0x7f3d80199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d85c52640 1 -- 172.21.15.16:0/1099852334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d801a8480 con 0x7f3d8006b0b0 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d85c52640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d8006b0b0 0x7f3d80198dd0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c00e9f0 tx=0x7f3d7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d76ffd640 1 -- 172.21.15.16:0/1099852334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c00cd80 con 0x7f3d8006b0b0 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d76ffd640 1 -- 172.21.15.16:0/1099852334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d7c0108e0 con 0x7f3d8006b0b0 2024-08-28T19:34:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d76ffd640 1 -- 172.21.15.16:0/1099852334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c018980 con 0x7f3d8006b0b0 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1099852334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d801a8770 con 0x7f3d8006b0b0 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.280+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1099852334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d801a8cb0 con 0x7f3d8006b0b0 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.283+0000 7f3d8e42b740 1 Processor -- start 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.283+0000 7f3d8e42b740 1 -- start start 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1379780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0faa490 0x55d5b1379cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0fb2670 0x55d5b137a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d5b12ef5f0 con 0x55d5b0fd3350 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d5b12deb80 con 0x55d5b0faa490 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d5b12e2b30 con 0x55d5b0fb2670 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d76ffd640 1 -- 172.21.15.16:0/1099852334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3d7c010430 con 0x7f3d8006b0b0 2024-08-28T19:34:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d86453640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1379780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d86453640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1379780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35688/0 (socket says 172.21.15.16:35688) 2024-08-28T19:34:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d86453640 1 -- 172.21.15.16:0/1052598740 learned_addr learned my addr 172.21.15.16:0/1052598740 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d85451640 1 --2- 172.21.15.16:0/1052598740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0fb2670 0x55d5b137a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d85c52640 1 --2- 172.21.15.16:0/1052598740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0faa490 0x55d5b1379cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d86453640 1 -- 172.21.15.16:0/1052598740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0fb2670 msgr2=0x55d5b137a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d86453640 1 --2- 172.21.15.16:0/1052598740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0fb2670 0x55d5b137a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d86453640 1 -- 172.21.15.16:0/1052598740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0faa490 msgr2=0x55d5b1379cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d86453640 1 --2- 172.21.15.16:0/1052598740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0faa490 0x55d5b1379cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d86453640 1 -- 172.21.15.16:0/1052598740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5b103e930 con 0x55d5b0fd3350 2024-08-28T19:34:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.284+0000 7f3d85c52640 1 --2- 172.21.15.16:0/1052598740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0faa490 0x55d5b1379cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.285+0000 7f3d86453640 1 --2- 172.21.15.16:0/1052598740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1379780 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f3d7802f4c0 tx=0x7f3d78009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6034bd48cc514f0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.285+0000 7f3d6bfff640 1 -- 172.21.15.16:0/1052598740 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78031cf0 con 0x55d5b0fd3350 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.285+0000 7f3d76ffd640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3d6c0b4ca0 0x7f3d6c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.285+0000 7f3d6bfff640 1 -- 172.21.15.16:0/1052598740 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d78036030 con 0x55d5b0fd3350 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.285+0000 7f3d6bfff640 1 -- 172.21.15.16:0/1052598740 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d780042a0 con 0x55d5b0fd3350 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.285+0000 7f3d85c52640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3d6c0b4ca0 0x7f3d6c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.285+0000 7f3d76ffd640 1 -- 172.21.15.16:0/1099852334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f3d7c0e5cd0 con 0x7f3d8006b0b0 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d8e42b740 1 -- 172.21.15.16:0/1052598740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 msgr2=0x55d5b1379780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d85c52640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3d6c0b4ca0 0x7f3d6c0b71b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c0e3100 tx=0x7f3d7c000bc0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/1052598740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1379780 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f3d7802f4c0 tx=0x7f3d78009800 comp rx=0 tx=0).stop 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d8e42b740 1 -- 172.21.15.16:0/1052598740 shutdown_connections 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/1052598740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0fb2670 0x55d5b137a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/1052598740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0faa490 0x55d5b1379cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/1052598740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1379780 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d8e42b740 1 -- 172.21.15.16:0/1052598740 >> 172.21.15.16:0/1052598740 conn(0x55d5b0fc7740 msgr2=0x55d5b1377670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d8e42b740 1 -- 172.21.15.16:0/1052598740 shutdown_connections 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d8e42b740 1 -- 172.21.15.16:0/1052598740 wait complete. 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d8e42b740 1 -- start start 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.286+0000 7f3d8e42b740 1 Processor -- start 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0faa490 0x55d5b1384f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fb2670 0x55d5b1385440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1391070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d5b1141890 con 0x55d5b0fd3350 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d5b1265790 con 0x55d5b0fb2670 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d5b1387d80 con 0x55d5b0faa490 2024-08-28T19:34:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d8e42b740 10 client.0 _ll_register_callbacks cb 0x55d5b0feaba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d86453640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1391070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d86453640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1391070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35704/0 (socket says 172.21.15.16:35704) 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d86453640 1 -- 172.21.15.16:0/2699208321 learned_addr learned my addr 172.21.15.16:0/2699208321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d86453640 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0faa490 0x55d5b1384f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d85451640 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fb2670 0x55d5b1385440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d86453640 1 -- 172.21.15.16:0/2699208321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0faa490 msgr2=0x55d5b1384f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d86453640 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0faa490 0x55d5b1384f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d86453640 1 -- 172.21.15.16:0/2699208321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fb2670 msgr2=0x55d5b1385440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d86453640 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fb2670 0x55d5b1385440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.287+0000 7f3d86453640 1 -- 172.21.15.16:0/2699208321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5b129cf30 con 0x55d5b0fd3350 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d86453640 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1391070 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009620 tx=0x7f3d78003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11fa77705d89034 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d697fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d697fa640 1 -- 172.21.15.16:0/2699208321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78058060 con 0x55d5b0fd3350 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d697fa640 1 -- 172.21.15.16:0/2699208321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f3d78004630 con 0x55d5b0fd3350 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2699208321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d78036030 con 0x55d5b0fd3350 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d697fa640 1 -- 172.21.15.16:0/2699208321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f3d78004830 con 0x55d5b0fd3350 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d8e42b740 10 client.18903 fetch_fsmap learned FSMap version 73 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2699208321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55d5b1393a60 con 0x55d5b0fd3350 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d697fa640 1 -- 172.21.15.16:0/2699208321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d78036030 con 0x55d5b0fd3350 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d697fa640 1 -- 172.21.15.16:0/2699208321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) ==== 8434+0+0 (secure 0 0 0) 0x7f3d78050070 con 0x55d5b0fd3350 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d8e42b740 10 client.18903 fetch_fsmap finished waiting for FSMap version 73 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.288+0000 7f3d8e42b740 10 client.18903 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.289+0000 7f3d8e42b740 20 client.18903 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.289+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d5b0f91cd0 0x55d5b1396dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.289+0000 7f3d8e42b740 4 client.18903 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:34:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.289+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2699208321 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5b12f1540 con 0x55d5b0f91cd0 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.289+0000 7f3d85451640 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d5b0f91cd0 0x55d5b1396dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.289+0000 7f3d85451640 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d5b0f91cd0 0x55d5b1396dc0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42af4de9fb8c8ea4 server_cookie=ad1099740555763e in_seq=0 out_seq=0 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.289+0000 7f3d697fa640 10 client.18903 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d697fa640 1 -- 172.21.15.16:0/2699208321 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d5b12f1540 con 0x55d5b0f91cd0 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d697fa640 10 client.18903 handle_command_reply: tid=0 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d8e42b740 1 client.18903 shutdown 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2699208321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 msgr2=0x55d5b1391070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1391070 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009620 tx=0x7f3d78003d70 comp rx=0 tx=0).stop 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2699208321 shutdown_connections 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fd3350 0x55d5b1391070 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fb2670 0x55d5b1385440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d5b0f91cd0 0x55d5b1396dc0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2699208321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0faa490 0x55d5b1384f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d697fa640 0 client.18903 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.290+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2699208321 >> 172.21.15.16:0/2699208321 conn(0x55d5b0fc7740 msgr2=0x55d5b137c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.291+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2699208321 shutdown_connections 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.291+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2699208321 wait complete. 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.291+0000 7f3d8e42b740 20 client.18903 trim_cache size 0 max 16384 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.308+0000 7f3d8e42b740 1 Processor -- start 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.308+0000 7f3d8e42b740 1 -- start start 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0faa490 0x55d5b137aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fb2670 0x55d5b1390fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fd3350 0x55d5b13820f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3d801b2e50 con 0x55d5b0fb2670 2024-08-28T19:34:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d801b4bd0 con 0x55d5b0fd3350 2024-08-28T19:34:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3d801b35b0 con 0x55d5b0faa490 2024-08-28T19:34:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d85451640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fb2670 0x55d5b1390fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d85451640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fb2670 0x55d5b1390fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35714/0 (socket says 172.21.15.16:35714) 2024-08-28T19:34:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d85451640 1 -- 172.21.15.16:0/1704895230 learned_addr learned my addr 172.21.15.16:0/1704895230 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d86453640 1 --2- 172.21.15.16:0/1704895230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fd3350 0x55d5b13820f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d86453640 1 --2- 172.21.15.16:0/1704895230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0faa490 0x55d5b137aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d85451640 1 -- 172.21.15.16:0/1704895230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0faa490 msgr2=0x55d5b137aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d85451640 1 --2- 172.21.15.16:0/1704895230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0faa490 0x55d5b137aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:39 smithi016 ceph-mon[18556]: pgmap v761: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 8.5 MiB/s wr, 227 op/s 2024-08-28T19:34:39.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d85451640 1 -- 172.21.15.16:0/1704895230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fd3350 msgr2=0x55d5b13820f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:39.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d85451640 1 --2- 172.21.15.16:0/1704895230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fd3350 0x55d5b13820f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d85451640 1 -- 172.21.15.16:0/1704895230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5b13861c0 con 0x55d5b0fb2670 2024-08-28T19:34:39.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d86453640 1 --2- 172.21.15.16:0/1704895230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fd3350 0x55d5b13820f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:34:39.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.309+0000 7f3d85451640 1 --2- 172.21.15.16:0/1704895230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fb2670 0x55d5b1390fe0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f3d80075800 tx=0x7f3d7000bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d371c39755fc79ac server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:39.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d6b7fe640 1 -- 172.21.15.16:0/1704895230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70003d20 con 0x55d5b0fb2670 2024-08-28T19:34:39.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d6b7fe640 1 -- 172.21.15.16:0/1704895230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d700118f0 con 0x55d5b0fb2670 2024-08-28T19:34:39.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d6b7fe640 1 -- 172.21.15.16:0/1704895230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7001c7e0 con 0x55d5b0fb2670 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d8e42b740 1 -- 172.21.15.16:0/1704895230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fb2670 msgr2=0x55d5b1390fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/1704895230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fb2670 0x55d5b1390fe0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f3d80075800 tx=0x7f3d7000bd10 comp rx=0 tx=0).stop 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d8e42b740 1 -- 172.21.15.16:0/1704895230 shutdown_connections 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/1704895230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fd3350 0x55d5b13820f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/1704895230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0fb2670 0x55d5b1390fe0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/1704895230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0faa490 0x55d5b137aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d8e42b740 1 -- 172.21.15.16:0/1704895230 >> 172.21.15.16:0/1704895230 conn(0x55d5b0fc7740 msgr2=0x55d5b137a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d8e42b740 1 -- 172.21.15.16:0/1704895230 shutdown_connections 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.310+0000 7f3d8e42b740 1 -- 172.21.15.16:0/1704895230 wait complete. 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d8e42b740 1 -- start start 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d8e42b740 1 Processor -- start 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0faa490 0x55d5b139a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0fb2670 0x55d5b13988b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d8e42b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fd3350 0x55d5b1398df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3d801b44d0 con 0x55d5b0faa490 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d801b3d40 con 0x55d5b0fd3350 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d8e42b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3d801b0e50 con 0x55d5b0fb2670 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d8e42b740 10 client.0 _ll_register_callbacks cb 0x7f3d7000b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d86453640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0faa490 0x55d5b139a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d86453640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0faa490 0x55d5b139a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35716/0 (socket says 172.21.15.16:35716) 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.311+0000 7f3d86453640 1 -- 172.21.15.16:0/2231865471 learned_addr learned my addr 172.21.15.16:0/2231865471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d85451640 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0fb2670 0x55d5b13988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d86453640 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fd3350 0x55d5b1398df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d86453640 1 -- 172.21.15.16:0/2231865471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0fb2670 msgr2=0x55d5b13988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d86453640 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0fb2670 0x55d5b13988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d86453640 1 -- 172.21.15.16:0/2231865471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fd3350 msgr2=0x55d5b1398df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d86453640 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fd3350 0x55d5b1398df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d86453640 1 -- 172.21.15.16:0/2231865471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5b13993b0 con 0x55d5b0faa490 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d85451640 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0fb2670 0x55d5b13988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d86453640 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0faa490 0x55d5b139a3d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3d780049d0 tx=0x7f3d78003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a903d789d5cf78 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d697fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d697fa640 1 -- 172.21.15.16:0/2231865471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78058040 con 0x55d5b0faa490 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2231865471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d700118f0 con 0x55d5b0faa490 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d697fa640 1 -- 172.21.15.16:0/2231865471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f3d7802f870 con 0x55d5b0faa490 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d697fa640 1 -- 172.21.15.16:0/2231865471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f3d78009800 con 0x55d5b0faa490 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d8e42b740 10 client.18909 fetch_fsmap learned FSMap version 73 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.312+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2231865471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55d5b14022c0 con 0x55d5b0faa490 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.313+0000 7f3d697fa640 1 -- 172.21.15.16:0/2231865471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d700118f0 con 0x55d5b0faa490 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.313+0000 7f3d697fa640 1 -- 172.21.15.16:0/2231865471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) ==== 8434+0+0 (secure 0 0 0) 0x7f3d78050040 con 0x55d5b0faa490 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.313+0000 7f3d8e42b740 10 client.18909 fetch_fsmap finished waiting for FSMap version 73 2024-08-28T19:34:39.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.313+0000 7f3d8e42b740 10 client.18909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:34:39.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.313+0000 7f3d8e42b740 20 client.18909 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:39.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.313+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d5b0f67080 0x55d5b1404a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:39.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.313+0000 7f3d8e42b740 4 client.18909 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:34:39.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.313+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2231865471 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d5b12ecfe0 con 0x55d5b0f67080 2024-08-28T19:34:39.510 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:34:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:34:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea", 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.313+0000 7f3d85451640 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d5b0f67080 0x55d5b1404a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.313+0000 7f3d697fa640 10 client.18909 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.313+0000 7f3d85451640 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d5b0f67080 0x55d5b1404a30 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e635458e4e018351 server_cookie=51bb813f9358354a in_seq=0 out_seq=0 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d697fa640 1 -- 172.21.15.16:0/2231865471 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d5b12ecfe0 con 0x55d5b0f67080 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d697fa640 10 client.18909 handle_command_reply: tid=0 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 1 client.18909 shutdown 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2231865471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0faa490 msgr2=0x55d5b139a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0faa490 0x55d5b139a3d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3d780049d0 tx=0x7f3d78003d70 comp rx=0 tx=0).stop 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2231865471 shutdown_connections 2024-08-28T19:34:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d5b0fd3350 0x55d5b1398df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d5b0f67080 0x55d5b1404a30 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d5b0fb2670 0x55d5b13988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 1 --2- 172.21.15.16:0/2231865471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5b0faa490 0x55d5b139a3d0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d697fa640 0 client.18909 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2231865471 >> 172.21.15.16:0/2231865471 conn(0x55d5b0fc7740 msgr2=0x55d5b1375c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2231865471 shutdown_connections 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 1 -- 172.21.15.16:0/2231865471 wait complete. 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.315+0000 7f3d8e42b740 20 client.18909 trim_cache size 0 max 16384 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1099852334 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3d6c0b4ca0 msgr2=0x7f3d6c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3d6c0b4ca0 0x7f3d6c0b71b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c0e3100 tx=0x7f3d7c000bc0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1099852334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d8006b0b0 msgr2=0x7f3d80198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d8006b0b0 0x7f3d80198dd0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c00e9f0 tx=0x7f3d7c00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1099852334 shutdown_connections 2024-08-28T19:34:39.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d80102910 0x7f3d80199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3d6c0b4ca0 0x7f3d6c0b71b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d80102500 0x7f3d80199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 --2- 172.21.15.16:0/1099852334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d8006b0b0 0x7f3d80198dd0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:39.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1099852334 >> 172.21.15.16:0/1099852334 conn(0x7f3d80079bd0 msgr2=0x7f3d801046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:39.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1099852334 shutdown_connections 2024-08-28T19:34:39.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:39.316+0000 7f3d87eb6640 1 -- 172.21.15.16:0/1099852334 wait complete. 2024-08-28T19:34:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:39 smithi115 ceph-mon[22184]: pgmap v761: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 8.5 MiB/s wr, 227 op/s 2024-08-28T19:34:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:39 smithi028 ceph-mon[21769]: pgmap v761: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 8.5 MiB/s wr, 227 op/s 2024-08-28T19:34:40.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:34:40.461 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 23.06 MB/sec warmup 16 sec latency 15127.189 ms 2024-08-28T19:34:40.492 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:40 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:34:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:40 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:40 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:34:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:40 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:34:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:40 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:34:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:40 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:40 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:40 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:34:40.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:40 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:40.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:40 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:34:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:40 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:40 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:34:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:40 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:34:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:40 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:34:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:40 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:40 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:40 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:34:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:40 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:40 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:34:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:40 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:40 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:34:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:40 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:34:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:40 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:34:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:40 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:40 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:40.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:40 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:34:40.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:40 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.168+0000 7f7765b6e640 1 -- 172.21.15.16:0/30539699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f776010dc00 msgr2=0x7f77601100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.168+0000 7f7765b6e640 1 --2- 172.21.15.16:0/30539699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f776010dc00 0x7f77601100f0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f7758009900 tx=0x7f775802a410 comp rx=0 tx=0).stop 2024-08-28T19:34:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.168+0000 7f7765b6e640 1 -- 172.21.15.16:0/30539699 shutdown_connections 2024-08-28T19:34:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.168+0000 7f7765b6e640 1 --2- 172.21.15.16:0/30539699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f776010dc00 0x7f77601100f0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.168+0000 7f7765b6e640 1 --2- 172.21.15.16:0/30539699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77601050f0 0x7f776010d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.168+0000 7f7765b6e640 1 --2- 172.21.15.16:0/30539699 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f77601047a0 0x7f7760104bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.168+0000 7f7765b6e640 1 -- 172.21.15.16:0/30539699 >> 172.21.15.16:0/30539699 conn(0x7f7760079450 msgr2=0x7f7760079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.168+0000 7f7765b6e640 1 -- 172.21.15.16:0/30539699 shutdown_connections 2024-08-28T19:34:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.168+0000 7f7765b6e640 1 -- 172.21.15.16:0/30539699 wait complete. 2024-08-28T19:34:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f7765b6e640 1 Processor -- start 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f7765b6e640 1 -- start start 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f7765b6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77601047a0 0x7f776019a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f7765b6e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601050f0 0x7f776019a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f7765b6e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f776010dc00 0x7f776019fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f7765b6e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7760113880 con 0x7f77601050f0 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f7765b6e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7760113700 con 0x7f77601047a0 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f7765b6e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7760113a00 con 0x7f776010dc00 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f775effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601050f0 0x7f776019a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f775effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601050f0 0x7f776019a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35722/0 (socket says 172.21.15.16:35722) 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f775effd640 1 -- 172.21.15.16:0/3417556356 learned_addr learned my addr 172.21.15.16:0/3417556356 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f775ffff640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f776010dc00 0x7f776019fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.169+0000 7f775f7fe640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77601047a0 0x7f776019a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f775effd640 1 -- 172.21.15.16:0/3417556356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f776010dc00 msgr2=0x7f776019fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f775effd640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f776010dc00 0x7f776019fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f775effd640 1 -- 172.21.15.16:0/3417556356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77601047a0 msgr2=0x7f776019a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f775effd640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77601047a0 0x7f776019a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f775effd640 1 -- 172.21.15.16:0/3417556356 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77601a0240 con 0x7f77601050f0 2024-08-28T19:34:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f775f7fe640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77601047a0 0x7f776019a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:34:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f775effd640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601050f0 0x7f776019a870 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f774800d740 tx=0x7f774800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f775cff9640 1 -- 172.21.15.16:0/3417556356 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7748017070 con 0x7f77601050f0 2024-08-28T19:34:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f775cff9640 1 -- 172.21.15.16:0/3417556356 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f774800de50 con 0x7f77601050f0 2024-08-28T19:34:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f775cff9640 1 -- 172.21.15.16:0/3417556356 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774800f620 con 0x7f77601050f0 2024-08-28T19:34:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f7765b6e640 1 -- 172.21.15.16:0/3417556356 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77601a0530 con 0x7f77601050f0 2024-08-28T19:34:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.170+0000 7f7765b6e640 1 -- 172.21.15.16:0/3417556356 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77601a6c90 con 0x7f77601050f0 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f776c0e3740 1 Processor -- start 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f775cff9640 1 -- 172.21.15.16:0/3417556356 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f774801a070 con 0x7f77601050f0 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f776c0e3740 1 -- start start 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca0f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca0f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c649390 0x558f2ca0fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f776c0e3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558f2c986630 con 0x558f2c6411b0 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f776c0e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558f2c975bc0 con 0x558f2c66a070 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f776c0e3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558f2c979b70 con 0x558f2c649390 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f775f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca0f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f775f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca0f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35734/0 (socket says 172.21.15.16:35734) 2024-08-28T19:34:41.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f775f7fe640 1 -- 172.21.15.16:0/2749911430 learned_addr learned my addr 172.21.15.16:0/2749911430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.173+0000 7f775ffff640 1 --2- 172.21.15.16:0/2749911430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca0f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f775effd640 1 --2- 172.21.15.16:0/2749911430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c649390 0x558f2ca0fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f775f7fe640 1 -- 172.21.15.16:0/2749911430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c649390 msgr2=0x558f2ca0fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f775f7fe640 1 --2- 172.21.15.16:0/2749911430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c649390 0x558f2ca0fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f775f7fe640 1 -- 172.21.15.16:0/2749911430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 msgr2=0x558f2ca0f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f775f7fe640 1 --2- 172.21.15.16:0/2749911430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca0f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f775f7fe640 1 -- 172.21.15.16:0/2749911430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f2c6d5650 con 0x558f2c6411b0 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f775f7fe640 1 --2- 172.21.15.16:0/2749911430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca0f800 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f7750010370 tx=0x7f7750010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4031438cc3cfb48e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f775cff9640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f774c0b4d20 0x7f774c0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f7755ffb640 1 -- 172.21.15.16:0/2749911430 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7750016070 con 0x558f2c6411b0 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f7755ffb640 1 -- 172.21.15.16:0/2749911430 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7750012bc0 con 0x558f2c6411b0 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f7755ffb640 1 -- 172.21.15.16:0/2749911430 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775001bcb0 con 0x558f2c6411b0 2024-08-28T19:34:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f775ffff640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f774c0b4d20 0x7f774c0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.174+0000 7f775cff9640 1 -- 172.21.15.16:0/3417556356 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f77480e5270 con 0x7f77601050f0 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.175+0000 7f776c0e3740 1 -- 172.21.15.16:0/2749911430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 msgr2=0x558f2ca0f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.175+0000 7f775ffff640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f774c0b4d20 0x7f774c0b7230 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7758006010 tx=0x7f77580058d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.175+0000 7f776c0e3740 1 --2- 172.21.15.16:0/2749911430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca0f800 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f7750010370 tx=0x7f7750010830 comp rx=0 tx=0).stop 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.175+0000 7f776c0e3740 1 -- 172.21.15.16:0/2749911430 shutdown_connections 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.175+0000 7f776c0e3740 1 --2- 172.21.15.16:0/2749911430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c649390 0x558f2ca0fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.175+0000 7f776c0e3740 1 --2- 172.21.15.16:0/2749911430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca0f800 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.175+0000 7f776c0e3740 1 --2- 172.21.15.16:0/2749911430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca0f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.175+0000 7f776c0e3740 1 -- 172.21.15.16:0/2749911430 >> 172.21.15.16:0/2749911430 conn(0x558f2c65e460 msgr2=0x558f2ca0d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.175+0000 7f776c0e3740 1 -- 172.21.15.16:0/2749911430 shutdown_connections 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.175+0000 7f776c0e3740 1 -- 172.21.15.16:0/2749911430 wait complete. 2024-08-28T19:34:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.176+0000 7f776c0e3740 1 -- start start 2024-08-28T19:34:41.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.176+0000 7f776c0e3740 1 Processor -- start 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.176+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c6411b0 0x558f2ca1aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.176+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c649390 0x558f2ca1af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.176+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca26dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.176+0000 7f776c0e3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558f2c7d8700 con 0x558f2c649390 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.176+0000 7f776c0e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558f2c8fc700 con 0x558f2c66a070 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.176+0000 7f776c0e3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558f2ca1daa0 con 0x558f2c6411b0 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f776c0e3740 10 client.0 _ll_register_callbacks cb 0x558f2c92bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f775effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c649390 0x558f2ca1af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f775effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c649390 0x558f2ca1af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35742/0 (socket says 172.21.15.16:35742) 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f775effd640 1 -- 172.21.15.16:0/1061887831 learned_addr learned my addr 172.21.15.16:0/1061887831 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f775f7fe640 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c6411b0 0x558f2ca1aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f775ffff640 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca26dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f775effd640 1 -- 172.21.15.16:0/1061887831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c6411b0 msgr2=0x558f2ca1aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f775effd640 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c6411b0 0x558f2ca1aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f775effd640 1 -- 172.21.15.16:0/1061887831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 msgr2=0x558f2ca26dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f775effd640 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca26dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f775effd640 1 -- 172.21.15.16:0/1061887831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f2c9358b0 con 0x558f2c649390 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f775effd640 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c649390 0x558f2ca1af80 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f77480e2650 tx=0x7f77480a7580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af990b941348882 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f773effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f773effd640 1 -- 172.21.15.16:0/1061887831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7748021070 con 0x558f2c649390 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f773effd640 1 -- 172.21.15.16:0/1061887831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f77480ce0e0 con 0x558f2c649390 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f776c0e3740 1 -- 172.21.15.16:0/1061887831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7750012bc0 con 0x558f2c649390 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.177+0000 7f773effd640 1 -- 172.21.15.16:0/1061887831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f77480b41f0 con 0x558f2c649390 2024-08-28T19:34:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.178+0000 7f776c0e3740 10 client.18921 fetch_fsmap learned FSMap version 73 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.178+0000 7f776c0e3740 1 -- 172.21.15.16:0/1061887831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x558f2ca29890 con 0x558f2c649390 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.178+0000 7f773effd640 1 -- 172.21.15.16:0/1061887831 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7750012bc0 con 0x558f2c649390 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.178+0000 7f773effd640 1 -- 172.21.15.16:0/1061887831 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) ==== 8434+0+0 (secure 0 0 0) 0x7f77480a6f00 con 0x558f2c649390 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.178+0000 7f776c0e3740 10 client.18921 fetch_fsmap finished waiting for FSMap version 73 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.178+0000 7f776c0e3740 10 client.18921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.178+0000 7f776c0e3740 20 client.18921 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.178+0000 7f776c0e3740 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f2c6289f0 0x558f2ca2cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.178+0000 7f776c0e3740 4 client.18921 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.178+0000 7f776c0e3740 1 -- 172.21.15.16:0/1061887831 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f2c988580 con 0x558f2c6289f0 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.178+0000 7f775f7fe640 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f2c6289f0 0x558f2ca2cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.179+0000 7f773effd640 10 client.18921 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.179+0000 7f775f7fe640 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f2c6289f0 0x558f2ca2cd60 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e54d8065711ecb2f server_cookie=b65cfbe2cbc8c0de in_seq=0 out_seq=0 2024-08-28T19:34:41.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.179+0000 7f773effd640 1 -- 172.21.15.16:0/1061887831 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558f2c988580 con 0x558f2c6289f0 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.179+0000 7f773effd640 10 client.18921 handle_command_reply: tid=0 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.179+0000 7f776c0e3740 1 client.18921 shutdown 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f776c0e3740 1 -- 172.21.15.16:0/1061887831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c649390 msgr2=0x558f2ca1af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f776c0e3740 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c649390 0x558f2ca1af80 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f77480e2650 tx=0x7f77480a7580 comp rx=0 tx=0).stop 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f776c0e3740 1 -- 172.21.15.16:0/1061887831 shutdown_connections 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f776c0e3740 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca26dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f776c0e3740 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c649390 0x558f2ca1af80 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f776c0e3740 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f2c6289f0 0x558f2ca2cd60 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f776c0e3740 1 --2- 172.21.15.16:0/1061887831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c6411b0 0x558f2ca1aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f773effd640 0 client.18921 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f776c0e3740 1 -- 172.21.15.16:0/1061887831 >> 172.21.15.16:0/1061887831 conn(0x558f2c65e460 msgr2=0x558f2ca12450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f776c0e3740 1 -- 172.21.15.16:0/1061887831 shutdown_connections 2024-08-28T19:34:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f776c0e3740 1 -- 172.21.15.16:0/1061887831 wait complete. 2024-08-28T19:34:41.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.180+0000 7f776c0e3740 20 client.18921 trim_cache size 0 max 16384 2024-08-28T19:34:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f776c0e3740 1 Processor -- start 2024-08-28T19:34:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f776c0e3740 1 -- start start 2024-08-28T19:34:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c6411b0 0x558f2ca12ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c649390 0x558f2ca26cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c66a070 0x558f2ca0cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f776c0e3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77601b0e10 con 0x558f2c66a070 2024-08-28T19:34:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f776c0e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77601b2b90 con 0x558f2c649390 2024-08-28T19:34:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f776c0e3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f77601b1570 con 0x558f2c6411b0 2024-08-28T19:34:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f775ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c66a070 0x558f2ca0cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f775ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c66a070 0x558f2ca0cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35756/0 (socket says 172.21.15.16:35756) 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f775ffff640 1 -- 172.21.15.16:0/261320352 learned_addr learned my addr 172.21.15.16:0/261320352 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f775f7fe640 1 --2- 172.21.15.16:0/261320352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c6411b0 0x558f2ca12ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f775effd640 1 --2- 172.21.15.16:0/261320352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c649390 0x558f2ca26cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f775ffff640 1 -- 172.21.15.16:0/261320352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c6411b0 msgr2=0x558f2ca12ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f775ffff640 1 --2- 172.21.15.16:0/261320352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c6411b0 0x558f2ca12ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f775ffff640 1 -- 172.21.15.16:0/261320352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c649390 msgr2=0x558f2ca26cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f775ffff640 1 --2- 172.21.15.16:0/261320352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c649390 0x558f2ca26cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.198+0000 7f775ffff640 1 -- 172.21.15.16:0/261320352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f2ca199a0 con 0x558f2c66a070 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.199+0000 7f775f7fe640 1 --2- 172.21.15.16:0/261320352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c6411b0 0x558f2ca12ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.199+0000 7f775ffff640 1 --2- 172.21.15.16:0/261320352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c66a070 0x558f2ca0cad0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f7758032fd0 tx=0x7f775803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907e97b20c3a4f36 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.199+0000 7f77557fa640 1 -- 172.21.15.16:0/261320352 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758035040 con 0x558f2c66a070 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.199+0000 7f77557fa640 1 -- 172.21.15.16:0/261320352 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f775804e070 con 0x558f2c66a070 2024-08-28T19:34:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.199+0000 7f77557fa640 1 -- 172.21.15.16:0/261320352 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758004cd0 con 0x558f2c66a070 2024-08-28T19:34:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.200+0000 7f776c0e3740 1 -- 172.21.15.16:0/261320352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c66a070 msgr2=0x558f2ca0cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.200+0000 7f776c0e3740 1 --2- 172.21.15.16:0/261320352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c66a070 0x558f2ca0cad0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f7758032fd0 tx=0x7f775803f000 comp rx=0 tx=0).stop 2024-08-28T19:34:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.200+0000 7f776c0e3740 1 -- 172.21.15.16:0/261320352 shutdown_connections 2024-08-28T19:34:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.200+0000 7f776c0e3740 1 --2- 172.21.15.16:0/261320352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c66a070 0x558f2ca0cad0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.200+0000 7f776c0e3740 1 --2- 172.21.15.16:0/261320352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c649390 0x558f2ca26cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.200+0000 7f776c0e3740 1 --2- 172.21.15.16:0/261320352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c6411b0 0x558f2ca12ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.200+0000 7f776c0e3740 1 -- 172.21.15.16:0/261320352 >> 172.21.15.16:0/261320352 conn(0x558f2c65e460 msgr2=0x558f2ca123c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.200+0000 7f776c0e3740 1 -- 172.21.15.16:0/261320352 shutdown_connections 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.200+0000 7f776c0e3740 1 -- 172.21.15.16:0/261320352 wait complete. 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.200+0000 7f776c0e3740 1 -- start start 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f776c0e3740 1 Processor -- start 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca1f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c649390 0x558f2ca1fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f776c0e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca907d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f776c0e3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77601b2490 con 0x558f2c6411b0 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f776c0e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77601b1d00 con 0x558f2c66a070 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f776c0e3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f77601aee10 con 0x558f2c649390 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f776c0e3740 10 client.0 _ll_register_callbacks cb 0x7f7748009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca1f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca1f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35768/0 (socket says 172.21.15.16:35768) 2024-08-28T19:34:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775f7fe640 1 -- 172.21.15.16:0/2701252799 learned_addr learned my addr 172.21.15.16:0/2701252799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775ffff640 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca907d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775effd640 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c649390 0x558f2ca1fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775f7fe640 1 -- 172.21.15.16:0/2701252799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c649390 msgr2=0x558f2ca1fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775f7fe640 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c649390 0x558f2ca1fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775f7fe640 1 -- 172.21.15.16:0/2701252799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 msgr2=0x558f2ca907d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775f7fe640 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca907d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775f7fe640 1 -- 172.21.15.16:0/2701252799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f2ca90e40 con 0x558f2c6411b0 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775effd640 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c649390 0x558f2ca1fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.201+0000 7f775f7fe640 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca1f7c0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f77500099e0 tx=0x7f7750009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9ead3d5885e178 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f773effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f773effd640 1 -- 172.21.15.16:0/2701252799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7750032030 con 0x558f2c6411b0 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f773effd640 1 -- 172.21.15.16:0/2701252799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7f77500049e0 con 0x558f2c6411b0 2024-08-28T19:34:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f776c0e3740 1 -- 172.21.15.16:0/2701252799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f775804e070 con 0x558f2c6411b0 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f773effd640 1 -- 172.21.15.16:0/2701252799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f775003c050 con 0x558f2c6411b0 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f776c0e3740 10 client.18927 fetch_fsmap learned FSMap version 73 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f776c0e3740 1 -- 172.21.15.16:0/2701252799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x558f2ca98240 con 0x558f2c6411b0 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f773effd640 1 -- 172.21.15.16:0/2701252799 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f775804e070 con 0x558f2c6411b0 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f773effd640 1 -- 172.21.15.16:0/2701252799 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) ==== 8434+0+0 (secure 0 0 0) 0x7f775000f040 con 0x558f2c6411b0 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f776c0e3740 10 client.18927 fetch_fsmap finished waiting for FSMap version 73 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f776c0e3740 10 client.18927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f776c0e3740 20 client.18927 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f776c0e3740 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f2c5fdda0 0x558f2ca9a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f776c0e3740 4 client.18927 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.202+0000 7f776c0e3740 1 -- 172.21.15.16:0/2701252799 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558f2c984020 con 0x558f2c5fdda0 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.203+0000 7f775effd640 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f2c5fdda0 0x558f2ca9a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.203+0000 7f775effd640 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f2c5fdda0 0x558f2ca9a780 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be19a2c091b49efa server_cookie=89fd26eaee19da04 in_seq=0 out_seq=0 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.203+0000 7f773effd640 10 client.18927 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.203+0000 7f773effd640 1 -- 172.21.15.16:0/2701252799 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+5682 (crc 0 0 0) 0x558f2c984020 con 0x558f2c5fdda0 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.203+0000 7f773effd640 10 client.18927 handle_command_reply: tid=0 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 1 client.18927 shutdown 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 1 -- 172.21.15.16:0/2701252799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 msgr2=0x558f2ca1f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca1f7c0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f77500099e0 tx=0x7f7750009520 comp rx=0 tx=0).stop 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 1 -- 172.21.15.16:0/2701252799 shutdown_connections 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f2c66a070 0x558f2ca907d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f2c5fdda0 0x558f2ca9a780 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f773effd640 0 client.18927 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f2c649390 0x558f2ca1fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 1 --2- 172.21.15.16:0/2701252799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f2c6411b0 0x558f2ca1f7c0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 1 -- 172.21.15.16:0/2701252799 >> 172.21.15.16:0/2701252799 conn(0x558f2c65e460 msgr2=0x558f2ca0ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 1 -- 172.21.15.16:0/2701252799 shutdown_connections 2024-08-28T19:34:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 1 -- 172.21.15.16:0/2701252799 wait complete. 2024-08-28T19:34:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f776c0e3740 20 client.18927 trim_cache size 0 max 16384 2024-08-28T19:34:41.210 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:34:41.210 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:34:41.210 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:34:41.210 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:34:41.210 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea", 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.212 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:34:41.213 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.214 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:34:41.217 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:34:41.218 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.220 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:34:41.221 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f7765b6e640 1 -- 172.21.15.16:0/3417556356 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f774c0b4d20 msgr2=0x7f774c0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f7765b6e640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f774c0b4d20 0x7f774c0b7230 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7758006010 tx=0x7f77580058d0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f7765b6e640 1 -- 172.21.15.16:0/3417556356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601050f0 msgr2=0x7f776019a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f7765b6e640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601050f0 0x7f776019a870 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f774800d740 tx=0x7f774800dc00 comp rx=0 tx=0).stop 2024-08-28T19:34:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f7765b6e640 1 -- 172.21.15.16:0/3417556356 shutdown_connections 2024-08-28T19:34:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f7765b6e640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f776010dc00 0x7f776019fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f7765b6e640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601050f0 0x7f776019a870 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.204+0000 7f7765b6e640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f774c0b4d20 0x7f774c0b7230 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.205+0000 7f7765b6e640 1 --2- 172.21.15.16:0/3417556356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77601047a0 0x7f776019a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.205+0000 7f7765b6e640 1 -- 172.21.15.16:0/3417556356 >> 172.21.15.16:0/3417556356 conn(0x7f7760079450 msgr2=0x7f77600ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.205+0000 7f7765b6e640 1 -- 172.21.15.16:0/3417556356 shutdown_connections 2024-08-28T19:34:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:41.205+0000 7f7765b6e640 1 -- 172.21.15.16:0/3417556356 wait complete. 2024-08-28T19:34:41.461 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 21.71 MB/sec warmup 17 sec latency 16127.223 ms 2024-08-28T19:34:41.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:41 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:41.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:41 smithi115 ceph-mon[22184]: pgmap v762: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 8.4 MiB/s wr, 223 op/s 2024-08-28T19:34:41.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:41 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:41.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:41 smithi016 ceph-mon[18556]: pgmap v762: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 8.4 MiB/s wr, 223 op/s 2024-08-28T19:34:41.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:41 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:34:41.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:41 smithi028 ceph-mon[21769]: pgmap v762: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 8.4 MiB/s wr, 223 op/s 2024-08-28T19:34:42.118 INFO:tasks.cephfs.filesystem:scrub status for tag:d8a48fd9-cf32-460f-8f4e-8dd4b3061fea - {'path': '/', 'tag': 'd8a48fd9-cf32-460f-8f4e-8dd4b3061fea', 'options': 'recursive,force'} 2024-08-28T19:34:42.461 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 20.50 MB/sec warmup 18 sec latency 17127.299 ms 2024-08-28T19:34:43.461 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 19.43 MB/sec warmup 19 sec latency 18127.341 ms 2024-08-28T19:34:43.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:43 smithi016 ceph-mon[18556]: pgmap v763: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 7.8 MiB/s wr, 178 op/s 2024-08-28T19:34:43.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:43 smithi028 ceph-mon[21769]: pgmap v763: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 7.8 MiB/s wr, 178 op/s 2024-08-28T19:34:43.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:43 smithi115 ceph-mon[22184]: pgmap v763: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 7.8 MiB/s wr, 178 op/s 2024-08-28T19:34:44.461 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 18.46 MB/sec warmup 20 sec latency 19127.449 ms 2024-08-28T19:34:45.461 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 17.58 MB/sec warmup 21 sec latency 20127.542 ms 2024-08-28T19:34:45.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:45 smithi016 ceph-mon[18556]: pgmap v764: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.4 MiB/s wr, 154 op/s 2024-08-28T19:34:45.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:45 smithi028 ceph-mon[21769]: pgmap v764: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.4 MiB/s wr, 154 op/s 2024-08-28T19:34:45.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:45 smithi115 ceph-mon[22184]: pgmap v764: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.4 MiB/s wr, 154 op/s 2024-08-28T19:34:46.461 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 16.78 MB/sec warmup 22 sec latency 21127.612 ms 2024-08-28T19:34:47.461 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 16.05 MB/sec warmup 23 sec latency 22127.658 ms 2024-08-28T19:34:47.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:47 smithi016 ceph-mon[18556]: pgmap v765: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 5.0 MiB/s wr, 79 op/s 2024-08-28T19:34:47.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:47 smithi028 ceph-mon[21769]: pgmap v765: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 5.0 MiB/s wr, 79 op/s 2024-08-28T19:34:47.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:47 smithi115 ceph-mon[22184]: pgmap v765: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 5.0 MiB/s wr, 79 op/s 2024-08-28T19:34:48.461 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 15.38 MB/sec warmup 24 sec latency 23127.749 ms 2024-08-28T19:34:49.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 14.77 MB/sec warmup 25 sec latency 24127.824 ms 2024-08-28T19:34:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:49 smithi016 ceph-mon[18556]: pgmap v766: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 2.5 MiB/s wr, 24 op/s 2024-08-28T19:34:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:49 smithi028 ceph-mon[21769]: pgmap v766: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 2.5 MiB/s wr, 24 op/s 2024-08-28T19:34:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:49 smithi115 ceph-mon[22184]: pgmap v766: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 2.5 MiB/s wr, 24 op/s 2024-08-28T19:34:50.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 14.20 MB/sec warmup 26 sec latency 25127.924 ms 2024-08-28T19:34:51.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 13.68 MB/sec warmup 27 sec latency 26127.973 ms 2024-08-28T19:34:51.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:51 smithi016 ceph-mon[18556]: pgmap v767: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 155 KiB/s wr, 20 op/s 2024-08-28T19:34:51.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:51 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:34:51.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:51 smithi028 ceph-mon[21769]: pgmap v767: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 155 KiB/s wr, 20 op/s 2024-08-28T19:34:51.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:51 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:34:51.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:51 smithi115 ceph-mon[22184]: pgmap v767: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 155 KiB/s wr, 20 op/s 2024-08-28T19:34:51.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:51 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:34:52.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 13.19 MB/sec warmup 28 sec latency 27128.049 ms 2024-08-28T19:34:52.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:34:52.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:34:52.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:34:52.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:34:52.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:34:52.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:34:52.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:34:52.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:34:52.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:34:53.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:34:53.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 12.73 MB/sec warmup 29 sec latency 28128.115 ms 2024-08-28T19:34:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:53.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:53 smithi016 ceph-mon[18556]: pgmap v768: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 168 KiB/s wr, 26 op/s 2024-08-28T19:34:53.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:53 smithi028 ceph-mon[21769]: pgmap v768: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 168 KiB/s wr, 26 op/s 2024-08-28T19:34:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:53 smithi115 ceph-mon[22184]: pgmap v768: 129 pgs: 129 active+clean; 518 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 168 KiB/s wr, 26 op/s 2024-08-28T19:34:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.081+0000 7fde9f684640 1 -- 172.21.15.16:0/2067678375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde98109c60 msgr2=0x7fde98110490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.081+0000 7fde9f684640 1 --2- 172.21.15.16:0/2067678375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde98109c60 0x7fde98110490 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fde90009ca0 tx=0x7fde9002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.081+0000 7fde9f684640 1 -- 172.21.15.16:0/2067678375 shutdown_connections 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.081+0000 7fde9f684640 1 --2- 172.21.15.16:0/2067678375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde98109c60 0x7fde98110490 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.081+0000 7fde9f684640 1 --2- 172.21.15.16:0/2067678375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98109310 0x7fde98109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.081+0000 7fde9f684640 1 --2- 172.21.15.16:0/2067678375 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde981039d0 0x7fde98103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.081+0000 7fde9f684640 1 -- 172.21.15.16:0/2067678375 >> 172.21.15.16:0/2067678375 conn(0x7fde9807b670 msgr2=0x7fde9807ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.081+0000 7fde9f684640 1 -- 172.21.15.16:0/2067678375 shutdown_connections 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.081+0000 7fde9f684640 1 -- 172.21.15.16:0/2067678375 wait complete. 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.082+0000 7fde9f684640 1 Processor -- start 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.082+0000 7fde9f684640 1 -- start start 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.082+0000 7fde9f684640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde981039d0 0x7fde9819a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.082+0000 7fde9f684640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde98109310 0x7fde9819a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.082+0000 7fde9f684640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde98109c60 0x7fde9819fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.082+0000 7fde9f684640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde98113880 con 0x7fde98109c60 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.082+0000 7fde9f684640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde98113700 con 0x7fde981039d0 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.082+0000 7fde9f684640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fde98113a00 con 0x7fde98109310 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9dc21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde98109c60 0x7fde9819fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9d420640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde981039d0 0x7fde9819a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9dc21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde98109c60 0x7fde9819fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38974/0 (socket says 172.21.15.16:38974) 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9dc21640 1 -- 172.21.15.16:0/3240616897 learned_addr learned my addr 172.21.15.16:0/3240616897 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9d420640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde981039d0 0x7fde9819a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54376/0 (socket says 172.21.15.16:54376) 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9cc1f640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde98109310 0x7fde9819a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9d420640 1 -- 172.21.15.16:0/3240616897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde98109310 msgr2=0x7fde9819a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9d420640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde98109310 0x7fde9819a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9d420640 1 -- 172.21.15.16:0/3240616897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde98109c60 msgr2=0x7fde9819fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9d420640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde98109c60 0x7fde9819fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9d420640 1 -- 172.21.15.16:0/3240616897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde981a0280 con 0x7fde981039d0 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9cc1f640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde98109310 0x7fde9819a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde9d420640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde981039d0 0x7fde9819a310 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fde9400d720 tx=0x7fde9400dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.083+0000 7fde8e7fc640 1 -- 172.21.15.16:0/3240616897 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde94017070 con 0x7fde981039d0 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.084+0000 7fde9f684640 1 -- 172.21.15.16:0/3240616897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde981a0570 con 0x7fde981039d0 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.084+0000 7fde8e7fc640 1 -- 172.21.15.16:0/3240616897 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde9400de30 con 0x7fde981039d0 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.084+0000 7fde8e7fc640 1 -- 172.21.15.16:0/3240616897 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde94004d00 con 0x7fde981039d0 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.084+0000 7fde9f684640 1 -- 172.21.15.16:0/3240616897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde981a6cd0 con 0x7fde981039d0 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.085+0000 7fde9f684640 1 -- 172.21.15.16:0/3240616897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde98103de0 con 0x7fde981039d0 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.087+0000 7fde8e7fc640 1 -- 172.21.15.16:0/3240616897 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fde94049880 con 0x7fde981039d0 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.088+0000 7fde8e7fc640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde840b4d20 0x7fde840b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.088+0000 7fde9cc1f640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde840b4d20 0x7fde840b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.088+0000 7fde8e7fc640 1 -- 172.21.15.16:0/3240616897 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7fde940e5220 con 0x7fde981039d0 2024-08-28T19:34:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.089+0000 7fde9cc1f640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde840b4d20 0x7fde840b7230 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fde88005fd0 tx=0x7fde88005850 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:54.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.091+0000 7fde8e7fc640 1 -- 172.21.15.16:0/3240616897 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fde940abc60 con 0x7fde981039d0 2024-08-28T19:34:54.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.315+0000 7fde9f684640 1 -- 172.21.15.16:0/3240616897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde9802e430 con 0x7fde981039d0 2024-08-28T19:34:54.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.316+0000 7fde8e7fc640 1 -- 172.21.15.16:0/3240616897 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+12004 (secure 0 0 0) 0x7fde940ab560 con 0x7fde981039d0 2024-08-28T19:34:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:34:54.318 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"btime":"2024-08-28T19:34:33:338245+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":73,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:34:33.292943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:34:54.318 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2024-08-28T19:34:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.318+0000 7fde9f684640 1 -- 172.21.15.16:0/3240616897 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde840b4d20 msgr2=0x7fde840b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.318+0000 7fde9f684640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde840b4d20 0x7fde840b7230 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fde88005fd0 tx=0x7fde88005850 comp rx=0 tx=0).stop 2024-08-28T19:34:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.319+0000 7fde9f684640 1 -- 172.21.15.16:0/3240616897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde981039d0 msgr2=0x7fde9819a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.319+0000 7fde9f684640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde981039d0 0x7fde9819a310 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fde9400d720 tx=0x7fde9400dbe0 comp rx=0 tx=0).stop 2024-08-28T19:34:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.319+0000 7fde9f684640 1 -- 172.21.15.16:0/3240616897 shutdown_connections 2024-08-28T19:34:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.319+0000 7fde9f684640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde98109c60 0x7fde9819fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.319+0000 7fde9f684640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde98109310 0x7fde9819a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.319+0000 7fde9f684640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fde840b4d20 0x7fde840b7230 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.319+0000 7fde9f684640 1 --2- 172.21.15.16:0/3240616897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde981039d0 0x7fde9819a310 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.319+0000 7fde9f684640 1 -- 172.21.15.16:0/3240616897 >> 172.21.15.16:0/3240616897 conn(0x7fde9807b670 msgr2=0x7fde98107450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:54.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.319+0000 7fde9f684640 1 -- 172.21.15.16:0/3240616897 shutdown_connections 2024-08-28T19:34:54.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:54.319+0000 7fde9f684640 1 -- 172.21.15.16:0/3240616897 wait complete. 2024-08-28T19:34:54.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 12.31 MB/sec warmup 30 sec latency 29128.203 ms 2024-08-28T19:34:54.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:54 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3240616897' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:34:54.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:54 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3240616897' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:34:54.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:54 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3240616897' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:34:55.068 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id 293e425a --release --await' 2024-08-28T19:34:55.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id 293e425a --release --await 2024-08-28T19:34:55.327 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:34:55.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3712 11.91 MB/sec warmup 31 sec latency 30128.304 ms 2024-08-28T19:34:55.717 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:55 smithi016 ceph-mon[18556]: pgmap v769: 129 pgs: 129 active+clean; 520 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 288 KiB/s wr, 23 op/s 2024-08-28T19:34:55.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:55 smithi028 ceph-mon[21769]: pgmap v769: 129 pgs: 129 active+clean; 520 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 288 KiB/s wr, 23 op/s 2024-08-28T19:34:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.815+0000 7fa951d25640 1 -- 172.21.15.16:0/2222140317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa94c1082d0 msgr2=0x7fa94c10eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.815+0000 7fa951d25640 1 --2- 172.21.15.16:0/2222140317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa94c1082d0 0x7fa94c10eb00 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fa944009ca0 tx=0x7fa94402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.815+0000 7fa951d25640 1 -- 172.21.15.16:0/2222140317 shutdown_connections 2024-08-28T19:34:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.815+0000 7fa951d25640 1 --2- 172.21.15.16:0/2222140317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa94c1082d0 0x7fa94c10eb00 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.815+0000 7fa951d25640 1 --2- 172.21.15.16:0/2222140317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa94c107980 0x7fa94c107d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.815+0000 7fa951d25640 1 --2- 172.21.15.16:0/2222140317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c102040 0x7fa94c102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.815+0000 7fa951d25640 1 -- 172.21.15.16:0/2222140317 >> 172.21.15.16:0/2222140317 conn(0x7fa94c079c60 msgr2=0x7fa94c07a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:55.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.815+0000 7fa951d25640 1 -- 172.21.15.16:0/2222140317 shutdown_connections 2024-08-28T19:34:55.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.815+0000 7fa951d25640 1 -- 172.21.15.16:0/2222140317 wait complete. 2024-08-28T19:34:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa951d25640 1 Processor -- start 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa951d25640 1 -- start start 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa951d25640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa94c102040 0x7fa94c111ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa951d25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa94c107980 0x7fa94c112530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa951d25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c1082d0 0x7fa94c112a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa951d25640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c111710 con 0x7fa94c107980 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa951d25640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c111590 con 0x7fa94c1082d0 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa951d25640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c111890 con 0x7fa94c102040 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa94affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa94c107980 0x7fa94c112530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa94affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa94c107980 0x7fa94c112530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53754/0 (socket says 172.21.15.16:53754) 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa94affd640 1 -- 172.21.15.16:0/3984180115 learned_addr learned my addr 172.21.15.16:0/3984180115 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.816+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa94c102040 0x7fa94c111ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa94bfff640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c1082d0 0x7fa94c112a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa94affd640 1 -- 172.21.15.16:0/3984180115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa94c102040 msgr2=0x7fa94c111ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa94affd640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa94c102040 0x7fa94c111ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa94affd640 1 -- 172.21.15.16:0/3984180115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c1082d0 msgr2=0x7fa94c112a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa94affd640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c1082d0 0x7fa94c112a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa94affd640 1 -- 172.21.15.16:0/3984180115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c1a8690 con 0x7fa94c107980 2024-08-28T19:34:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa94c102040 0x7fa94c111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:34:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa94bfff640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c1082d0 0x7fa94c112a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:34:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa94affd640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa94c107980 0x7fa94c112530 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fa94000d750 tx=0x7fa94000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa948ff9640 1 -- 172.21.15.16:0/3984180115 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940017070 con 0x7fa94c107980 2024-08-28T19:34:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa948ff9640 1 -- 172.21.15.16:0/3984180115 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9400054d0 con 0x7fa94c107980 2024-08-28T19:34:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa951d25640 1 -- 172.21.15.16:0/3984180115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c1a8980 con 0x7fa94c107980 2024-08-28T19:34:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa948ff9640 1 -- 172.21.15.16:0/3984180115 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940004c30 con 0x7fa94c107980 2024-08-28T19:34:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.817+0000 7fa951d25640 1 -- 172.21.15.16:0/3984180115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c1a8e70 con 0x7fa94c107980 2024-08-28T19:34:55.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa95829a740 1 Processor -- start 2024-08-28T19:34:55.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa95829a740 1 -- start start 2024-08-28T19:34:55.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa948ff9640 1 -- 172.21.15.16:0/3984180115 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa940010460 con 0x7fa94c107980 2024-08-28T19:34:55.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa95829a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd646c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa95829a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd278660 0x5631dd6471c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa95829a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd280840 0x5631dd647700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa95829a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5631dd5bda70 con 0x5631dd2a1520 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa95829a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5631dd5ad000 con 0x5631dd280840 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa95829a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5631dd5b0fb0 con 0x5631dd278660 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa94bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd646c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa94bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd646c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53770/0 (socket says 172.21.15.16:53770) 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa94bfff640 1 -- 172.21.15.16:0/1114438901 learned_addr learned my addr 172.21.15.16:0/1114438901 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa94affd640 1 --2- 172.21.15.16:0/1114438901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd280840 0x5631dd647700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.820+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/1114438901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd278660 0x5631dd6471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa94bfff640 1 -- 172.21.15.16:0/1114438901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd278660 msgr2=0x5631dd6471c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa94bfff640 1 --2- 172.21.15.16:0/1114438901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd278660 0x5631dd6471c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa94bfff640 1 -- 172.21.15.16:0/1114438901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd280840 msgr2=0x5631dd647700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa94bfff640 1 --2- 172.21.15.16:0/1114438901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd280840 0x5631dd647700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa94bfff640 1 -- 172.21.15.16:0/1114438901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631dd30cb00 con 0x5631dd2a1520 2024-08-28T19:34:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa94bfff640 1 --2- 172.21.15.16:0/1114438901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd646c80 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fa944006010 tx=0x7fa944009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=403ad589ac485eba server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa948ff9640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa9340b4ca0 0x7fa9340b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa9340b4ca0 0x7fa9340b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa931ffb640 1 -- 172.21.15.16:0/1114438901 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944031cf0 con 0x5631dd2a1520 2024-08-28T19:34:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa931ffb640 1 -- 172.21.15.16:0/1114438901 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa944036070 con 0x5631dd2a1520 2024-08-28T19:34:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa931ffb640 1 -- 172.21.15.16:0/1114438901 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9440042a0 con 0x5631dd2a1520 2024-08-28T19:34:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.821+0000 7fa948ff9640 1 -- 172.21.15.16:0/3984180115 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7fa9400e53b0 con 0x7fa94c107980 2024-08-28T19:34:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.822+0000 7fa95829a740 1 -- 172.21.15.16:0/1114438901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 msgr2=0x5631dd646c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.822+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa9340b4ca0 0x7fa9340b71b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x5631dd645900 tx=0x7fa93c005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.822+0000 7fa95829a740 1 --2- 172.21.15.16:0/1114438901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd646c80 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fa944006010 tx=0x7fa944009800 comp rx=0 tx=0).stop 2024-08-28T19:34:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.822+0000 7fa95829a740 1 -- 172.21.15.16:0/1114438901 shutdown_connections 2024-08-28T19:34:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.822+0000 7fa95829a740 1 --2- 172.21.15.16:0/1114438901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd280840 0x5631dd647700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.822+0000 7fa95829a740 1 --2- 172.21.15.16:0/1114438901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd278660 0x5631dd6471c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.822+0000 7fa95829a740 1 --2- 172.21.15.16:0/1114438901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd646c80 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.822+0000 7fa95829a740 1 -- 172.21.15.16:0/1114438901 >> 172.21.15.16:0/1114438901 conn(0x5631dd295910 msgr2=0x5631dd644b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.822+0000 7fa95829a740 1 -- 172.21.15.16:0/1114438901 shutdown_connections 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.822+0000 7fa95829a740 1 -- 172.21.15.16:0/1114438901 wait complete. 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.823+0000 7fa95829a740 1 -- start start 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.823+0000 7fa95829a740 1 Processor -- start 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.823+0000 7fa95829a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd652830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.823+0000 7fa95829a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd280840 0x5631dd652d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.823+0000 7fa95829a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd2a1520 0x5631dd65e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.823+0000 7fa95829a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5631dd655400 con 0x5631dd278660 2024-08-28T19:34:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.823+0000 7fa95829a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5631dd56e520 con 0x5631dd2a1520 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.823+0000 7fa95829a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5631dd655580 con 0x5631dd280840 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.823+0000 7fa95829a740 10 client.0 _ll_register_callbacks cb 0x5631dd5a6c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.823+0000 7fa94b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd652830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa94b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd652830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53786/0 (socket says 172.21.15.16:53786) 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa94b7fe640 1 -- 172.21.15.16:0/72830945 learned_addr learned my addr 172.21.15.16:0/72830945 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa94bfff640 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd2a1520 0x5631dd65e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa94affd640 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd280840 0x5631dd652d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa94b7fe640 1 -- 172.21.15.16:0/72830945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd280840 msgr2=0x5631dd652d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd280840 0x5631dd652d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa94b7fe640 1 -- 172.21.15.16:0/72830945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd2a1520 msgr2=0x5631dd65e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd2a1520 0x5631dd65e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa94b7fe640 1 -- 172.21.15.16:0/72830945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631dd383960 con 0x5631dd278660 2024-08-28T19:34:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa94affd640 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd280840 0x5631dd652d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd652830 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fa93c01cc40 tx=0x7fa93c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0801ccbf3c72725 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa92affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa92affd640 1 -- 172.21.15.16:0/72830945 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c01cd30 con 0x5631dd278660 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa92affd640 1 -- 172.21.15.16:0/72830945 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7fa93c016030 con 0x5631dd278660 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.824+0000 7fa95829a740 1 -- 172.21.15.16:0/72830945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5631dd54c9a0 con 0x5631dd278660 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.825+0000 7fa92affd640 1 -- 172.21.15.16:0/72830945 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fa93c032050 con 0x5631dd278660 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.825+0000 7fa95829a740 10 client.18945 fetch_fsmap learned FSMap version 73 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.825+0000 7fa95829a740 1 -- 172.21.15.16:0/72830945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x5631dd6615f0 con 0x5631dd278660 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.825+0000 7fa92affd640 1 -- 172.21.15.16:0/72830945 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5631dd54c9a0 con 0x5631dd278660 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.825+0000 7fa92affd640 1 -- 172.21.15.16:0/72830945 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) ==== 8434+0+0 (secure 0 0 0) 0x7fa93c00b040 con 0x5631dd278660 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.825+0000 7fa95829a740 10 client.18945 fetch_fsmap finished waiting for FSMap version 73 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.825+0000 7fa95829a740 10 client.18945 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.825+0000 7fa95829a740 20 client.18945 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.825+0000 7fa95829a740 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5631dd25fea0 0x5631dd664bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.825+0000 7fa95829a740 4 client.18945 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:34:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.825+0000 7fa95829a740 1 -- 172.21.15.16:0/72830945 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631dd5bf9c0 con 0x5631dd25fea0 2024-08-28T19:34:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.826+0000 7fa94affd640 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5631dd25fea0 0x5631dd664bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.826+0000 7fa94affd640 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5631dd25fea0 0x5631dd664bc0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96016f1998468f48 server_cookie=3a7f9da0368d7133 in_seq=0 out_seq=0 2024-08-28T19:34:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.826+0000 7fa92affd640 10 client.18945 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa92affd640 1 -- 172.21.15.16:0/72830945 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5631dd5bf9c0 con 0x5631dd25fea0 2024-08-28T19:34:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa92affd640 10 client.18945 handle_command_reply: tid=0 2024-08-28T19:34:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 1 client.18945 shutdown 2024-08-28T19:34:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 1 -- 172.21.15.16:0/72830945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 msgr2=0x5631dd652830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd652830 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fa93c01cc40 tx=0x7fa93c004730 comp rx=0 tx=0).stop 2024-08-28T19:34:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 1 -- 172.21.15.16:0/72830945 shutdown_connections 2024-08-28T19:34:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd2a1520 0x5631dd65e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5631dd25fea0 0x5631dd664bc0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd280840 0x5631dd652d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 1 --2- 172.21.15.16:0/72830945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd652830 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa92affd640 0 client.18945 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 1 -- 172.21.15.16:0/72830945 >> 172.21.15.16:0/72830945 conn(0x5631dd295910 msgr2=0x5631dd649e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 1 -- 172.21.15.16:0/72830945 shutdown_connections 2024-08-28T19:34:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 1 -- 172.21.15.16:0/72830945 wait complete. 2024-08-28T19:34:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.827+0000 7fa95829a740 20 client.18945 trim_cache size 0 max 16384 2024-08-28T19:34:55.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.844+0000 7fa95829a740 1 Processor -- start 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.844+0000 7fa95829a740 1 -- start start 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa95829a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd65e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa95829a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd280840 0x5631dd650eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa95829a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd2a1520 0x5631dd64f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa95829a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1b2f50 con 0x5631dd278660 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa95829a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1b4cd0 con 0x5631dd280840 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa95829a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1b36b0 con 0x5631dd2a1520 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa94b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd65e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa94b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd65e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53790/0 (socket says 172.21.15.16:53790) 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa94b7fe640 1 -- 172.21.15.16:0/276728635 learned_addr learned my addr 172.21.15.16:0/276728635 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa94affd640 1 --2- 172.21.15.16:0/276728635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd280840 0x5631dd650eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa94bfff640 1 --2- 172.21.15.16:0/276728635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd2a1520 0x5631dd64f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa94b7fe640 1 -- 172.21.15.16:0/276728635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd2a1520 msgr2=0x5631dd64f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/276728635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd2a1520 0x5631dd64f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa94b7fe640 1 -- 172.21.15.16:0/276728635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd280840 msgr2=0x5631dd650eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/276728635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd280840 0x5631dd650eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa94b7fe640 1 -- 172.21.15.16:0/276728635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631dd64dec0 con 0x5631dd278660 2024-08-28T19:34:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/276728635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd65e770 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fa93c017040 tx=0x7fa93c027b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea0d3bc66d8741f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.845+0000 7fa9317fa640 1 -- 172.21.15.16:0/276728635 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c027c90 con 0x5631dd278660 2024-08-28T19:34:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.846+0000 7fa9317fa640 1 -- 172.21.15.16:0/276728635 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa93c004520 con 0x5631dd278660 2024-08-28T19:34:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.846+0000 7fa9317fa640 1 -- 172.21.15.16:0/276728635 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c0035f0 con 0x5631dd278660 2024-08-28T19:34:55.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.846+0000 7fa95829a740 1 -- 172.21.15.16:0/276728635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 msgr2=0x5631dd65e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:55.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.846+0000 7fa95829a740 1 --2- 172.21.15.16:0/276728635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd65e770 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fa93c017040 tx=0x7fa93c027b20 comp rx=0 tx=0).stop 2024-08-28T19:34:55.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.847+0000 7fa95829a740 1 -- 172.21.15.16:0/276728635 shutdown_connections 2024-08-28T19:34:55.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.847+0000 7fa95829a740 1 --2- 172.21.15.16:0/276728635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd2a1520 0x5631dd64f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.847+0000 7fa95829a740 1 --2- 172.21.15.16:0/276728635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd280840 0x5631dd650eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.847+0000 7fa95829a740 1 --2- 172.21.15.16:0/276728635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd278660 0x5631dd65e770 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.847+0000 7fa95829a740 1 -- 172.21.15.16:0/276728635 >> 172.21.15.16:0/276728635 conn(0x5631dd295910 msgr2=0x5631dd652810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:55.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.847+0000 7fa95829a740 1 -- 172.21.15.16:0/276728635 shutdown_connections 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.847+0000 7fa95829a740 1 -- 172.21.15.16:0/276728635 wait complete. 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.847+0000 7fa95829a740 1 -- start start 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.847+0000 7fa95829a740 1 Processor -- start 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.847+0000 7fa95829a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd278660 0x5631dd65aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.847+0000 7fa95829a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd280840 0x5631dd6573c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa95829a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd657900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa95829a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1b45d0 con 0x5631dd2a1520 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa95829a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1b3e40 con 0x5631dd278660 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa95829a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1b0f50 con 0x5631dd280840 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa95829a740 10 client.0 _ll_register_callbacks cb 0x7fa93c00ac80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa94bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd657900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa94bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd657900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53798/0 (socket says 172.21.15.16:53798) 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa94bfff640 1 -- 172.21.15.16:0/2454468116 learned_addr learned my addr 172.21.15.16:0/2454468116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:34:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd278660 0x5631dd65aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa94affd640 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd280840 0x5631dd6573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa94bfff640 1 -- 172.21.15.16:0/2454468116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd280840 msgr2=0x5631dd6573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa94bfff640 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd280840 0x5631dd6573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa94bfff640 1 -- 172.21.15.16:0/2454468116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd278660 msgr2=0x5631dd65aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa94bfff640 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd278660 0x5631dd65aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa94bfff640 1 -- 172.21.15.16:0/2454468116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631dd657fa0 con 0x5631dd2a1520 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa94bfff640 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd657900 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fa944005e30 tx=0x7fa94402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a45bda7c836272a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa92affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa92affd640 1 -- 172.21.15.16:0/2454468116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944049040 con 0x5631dd2a1520 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa95829a740 1 -- 172.21.15.16:0/2454468116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa93c004520 con 0x5631dd2a1520 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.848+0000 7fa92affd640 1 -- 172.21.15.16:0/2454468116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9490+0+0 (secure 0 0 0) 0x7fa944032e00 con 0x5631dd2a1520 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa92affd640 1 -- 172.21.15.16:0/2454468116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fa94405b050 con 0x5631dd2a1520 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa95829a740 10 client.18951 fetch_fsmap learned FSMap version 73 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa95829a740 1 -- 172.21.15.16:0/2454468116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x5631dd6d0450 con 0x5631dd2a1520 2024-08-28T19:34:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa92affd640 1 -- 172.21.15.16:0/2454468116 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa93c004520 con 0x5631dd2a1520 2024-08-28T19:34:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa92affd640 1 -- 172.21.15.16:0/2454468116 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) ==== 8434+0+0 (secure 0 0 0) 0x7fa94404c030 con 0x5631dd2a1520 2024-08-28T19:34:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa95829a740 10 client.18951 fetch_fsmap finished waiting for FSMap version 73 2024-08-28T19:34:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa95829a740 10 client.18951 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:34:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa95829a740 20 client.18951 populate_metadata read hostname 'smithi016' 2024-08-28T19:34:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa95829a740 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5631dd235250 0x5631dd6d28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:34:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa95829a740 4 client.18951 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "293e425a", "release": true, "await": true}] 2024-08-28T19:34:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa95829a740 1 -- 172.21.15.16:0/2454468116 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "293e425a", "release": true, "await": true}) -- 0x5631dd5bb460 con 0x5631dd235250 2024-08-28T19:34:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.849+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5631dd235250 0x5631dd6d28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:34:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.850+0000 7fa94b7fe640 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5631dd235250 0x5631dd6d28b0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=182a0982ff05234e server_cookie=bebb478b77c1e1a8 in_seq=0 out_seq=0 2024-08-28T19:34:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:55.850+0000 7fa92affd640 10 client.18951 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:55.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:55 smithi115 ceph-mon[22184]: pgmap v769: 129 pgs: 129 active+clean; 520 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 288 KiB/s wr, 23 op/s 2024-08-28T19:34:56.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.147+0000 7fa92affd640 1 -- 172.21.15.16:0/2454468116 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5631dd5bb460 con 0x5631dd235250 2024-08-28T19:34:56.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.147+0000 7fa92affd640 10 client.18951 handle_command_reply: tid=0 2024-08-28T19:34:56.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.147+0000 7fa95829a740 1 client.18951 shutdown 2024-08-28T19:34:56.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.147+0000 7fa95829a740 1 -- 172.21.15.16:0/2454468116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 msgr2=0x5631dd657900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.147+0000 7fa95829a740 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd657900 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fa944005e30 tx=0x7fa94402f380 comp rx=0 tx=0).stop 2024-08-28T19:34:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.147+0000 7fa95829a740 1 -- 172.21.15.16:0/2454468116 shutdown_connections 2024-08-28T19:34:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.147+0000 7fa95829a740 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631dd2a1520 0x5631dd657900 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.147+0000 7fa95829a740 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5631dd235250 0x5631dd6d28b0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.147+0000 7fa95829a740 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5631dd280840 0x5631dd6573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.147+0000 7fa95829a740 1 --2- 172.21.15.16:0/2454468116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631dd278660 0x5631dd65aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.147+0000 7fa95829a740 1 -- 172.21.15.16:0/2454468116 >> 172.21.15.16:0/2454468116 conn(0x5631dd295910 msgr2=0x5631dd650e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa92affd640 0 client.18951 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:34:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa95829a740 1 -- 172.21.15.16:0/2454468116 shutdown_connections 2024-08-28T19:34:56.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa95829a740 1 -- 172.21.15.16:0/2454468116 wait complete. 2024-08-28T19:34:56.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa95829a740 20 client.18951 trim_cache size 0 max 16384 2024-08-28T19:34:56.151 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:34:56.151 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 73, 2024-08-28T19:34:56.151 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:34:56.151 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 44, 2024-08-28T19:34:56.151 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:34:56.151 INFO:teuthology.orchestra.run.smithi016.stdout: "293e425a": { 2024-08-28T19:34:56.151 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 44, 2024-08-28T19:34:56.152 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:34:56.152 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:34:56.152 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:34:56.152 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:34:56.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:34:56.152 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:34:56.152 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 145.7, 2024-08-28T19:34:56.152 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:34:56.152 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:34:56.152 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:34:56.152 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:34:56.153 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:34:56.153 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:34:56.153 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:56.153 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:56.153 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:56.153 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:56.153 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:34:56.153 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:34:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa951d25640 1 -- 172.21.15.16:0/3984180115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa9340b4ca0 msgr2=0x7fa9340b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa951d25640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa9340b4ca0 0x7fa9340b71b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x5631dd645900 tx=0x7fa93c005f90 comp rx=0 tx=0).stop 2024-08-28T19:34:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa951d25640 1 -- 172.21.15.16:0/3984180115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa94c107980 msgr2=0x7fa94c112530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:34:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa951d25640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa94c107980 0x7fa94c112530 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fa94000d750 tx=0x7fa94000dc10 comp rx=0 tx=0).stop 2024-08-28T19:34:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa951d25640 1 -- 172.21.15.16:0/3984180115 shutdown_connections 2024-08-28T19:34:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa951d25640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa9340b4ca0 0x7fa9340b71b0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa951d25640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c1082d0 0x7fa94c112a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa951d25640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa94c107980 0x7fa94c112530 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa951d25640 1 --2- 172.21.15.16:0/3984180115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa94c102040 0x7fa94c111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:34:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.148+0000 7fa951d25640 1 -- 172.21.15.16:0/3984180115 >> 172.21.15.16:0/3984180115 conn(0x7fa94c079c60 msgr2=0x7fa94c105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:34:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.149+0000 7fa951d25640 1 -- 172.21.15.16:0/3984180115 shutdown_connections 2024-08-28T19:34:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:34:56.149+0000 7fa951d25640 1 -- 172.21.15.16:0/3984180115 wait complete. 2024-08-28T19:34:56.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3713 11.54 MB/sec warmup 32 sec latency 31128.341 ms 2024-08-28T19:34:57.093 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '293e425a', total seconds elapsed: 29.5 2024-08-28T19:34:57.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3733 11.20 MB/sec warmup 33 sec latency 31181.979 ms 2024-08-28T19:34:57.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:57 smithi016 ceph-mon[18556]: pgmap v770: 129 pgs: 129 active+clean; 520 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 327 KiB/s wr, 27 op/s 2024-08-28T19:34:57.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:57 smithi028 ceph-mon[21769]: pgmap v770: 129 pgs: 129 active+clean; 520 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 327 KiB/s wr, 27 op/s 2024-08-28T19:34:57.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:57 smithi115 ceph-mon[22184]: pgmap v770: 129 pgs: 129 active+clean; 520 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 327 KiB/s wr, 27 op/s 2024-08-28T19:34:58.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3947 11.05 MB/sec warmup 34 sec latency 1053.342 ms 2024-08-28T19:34:58.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:58 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:58.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:58 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:58.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:58 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:34:59.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4297 11.24 MB/sec warmup 35 sec latency 143.072 ms 2024-08-28T19:34:59.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:34:59 smithi016 ceph-mon[18556]: pgmap v771: 129 pgs: 129 active+clean; 535 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 1.5 MiB/s wr, 30 op/s 2024-08-28T19:34:59.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:34:59 smithi028 ceph-mon[21769]: pgmap v771: 129 pgs: 129 active+clean; 535 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 1.5 MiB/s wr, 30 op/s 2024-08-28T19:34:59.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:34:59 smithi115 ceph-mon[22184]: pgmap v771: 129 pgs: 129 active+clean; 535 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 1.5 MiB/s wr, 30 op/s 2024-08-28T19:35:00.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4738 11.78 MB/sec warmup 36 sec latency 275.495 ms 2024-08-28T19:35:01.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 5336 11.79 MB/sec warmup 37 sec latency 168.454 ms 2024-08-28T19:35:01.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:01 smithi016 ceph-mon[18556]: pgmap v772: 129 pgs: 129 active+clean; 589 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 7.1 MiB/s wr, 73 op/s 2024-08-28T19:35:01.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:01 smithi028 ceph-mon[21769]: pgmap v772: 129 pgs: 129 active+clean; 589 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 7.1 MiB/s wr, 73 op/s 2024-08-28T19:35:01.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:01 smithi115 ceph-mon[22184]: pgmap v772: 129 pgs: 129 active+clean; 589 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 7.1 MiB/s wr, 73 op/s 2024-08-28T19:35:02.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 5427 11.73 MB/sec warmup 38 sec latency 230.165 ms 2024-08-28T19:35:02.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:02 smithi016 ceph-mon[18556]: pgmap v773: 129 pgs: 129 active+clean; 642 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 13 MiB/s wr, 182 op/s 2024-08-28T19:35:02.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:02 smithi028 ceph-mon[21769]: pgmap v773: 129 pgs: 129 active+clean; 642 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 13 MiB/s wr, 182 op/s 2024-08-28T19:35:02.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:02 smithi115 ceph-mon[22184]: pgmap v773: 129 pgs: 129 active+clean; 642 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 13 MiB/s wr, 182 op/s 2024-08-28T19:35:03.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 5665 11.69 MB/sec warmup 39 sec latency 162.713 ms 2024-08-28T19:35:03.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:03 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:35:03.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:03 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:35:03.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:03 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:35:04.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6217 12.55 MB/sec warmup 40 sec latency 1127.132 ms 2024-08-28T19:35:04.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:04 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:35:04.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:04 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:35:04.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:04 smithi016 ceph-mon[18556]: osdmap e99: 12 total, 12 up, 12 in 2024-08-28T19:35:04.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:04 smithi016 ceph-mon[18556]: pgmap v775: 129 pgs: 129 active+clean; 645 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 16 MiB/s wr, 303 op/s 2024-08-28T19:35:04.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:04 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:35:04.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:04 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:35:04.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:04 smithi028 ceph-mon[21769]: osdmap e99: 12 total, 12 up, 12 in 2024-08-28T19:35:04.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:04 smithi028 ceph-mon[21769]: pgmap v775: 129 pgs: 129 active+clean; 645 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 16 MiB/s wr, 303 op/s 2024-08-28T19:35:04.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:04 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:35:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:04 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:35:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:04 smithi115 ceph-mon[22184]: osdmap e99: 12 total, 12 up, 12 in 2024-08-28T19:35:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:04 smithi115 ceph-mon[22184]: pgmap v775: 129 pgs: 129 active+clean; 645 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 16 MiB/s wr, 303 op/s 2024-08-28T19:35:05.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6635 13.14 MB/sec warmup 41 sec latency 2127.158 ms 2024-08-28T19:35:06.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6824 13.05 MB/sec warmup 42 sec latency 3127.229 ms 2024-08-28T19:35:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:06 smithi115 ceph-mon[22184]: pgmap v776: 129 pgs: 129 active+clean; 647 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 17 MiB/s wr, 452 op/s 2024-08-28T19:35:07.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:06 smithi016 ceph-mon[18556]: pgmap v776: 129 pgs: 129 active+clean; 647 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 17 MiB/s wr, 452 op/s 2024-08-28T19:35:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:06 smithi028 ceph-mon[21769]: pgmap v776: 129 pgs: 129 active+clean; 647 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 17 MiB/s wr, 452 op/s 2024-08-28T19:35:07.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 7009 12.78 MB/sec warmup 43 sec latency 4127.269 ms 2024-08-28T19:35:08.462 INFO:tasks.workunit.client.0.smithi016.stdout: 10 7320 12.78 MB/sec warmup 44 sec latency 5127.312 ms 2024-08-28T19:35:09.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:08 smithi115 ceph-mon[22184]: pgmap v777: 129 pgs: 129 active+clean; 650 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 16 MiB/s wr, 553 op/s 2024-08-28T19:35:09.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:08 smithi016 ceph-mon[18556]: pgmap v777: 129 pgs: 129 active+clean; 650 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 16 MiB/s wr, 553 op/s 2024-08-28T19:35:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:08 smithi028 ceph-mon[21769]: pgmap v777: 129 pgs: 129 active+clean; 650 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 16 MiB/s wr, 553 op/s 2024-08-28T19:35:09.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 8008 13.13 MB/sec warmup 45 sec latency 6127.347 ms 2024-08-28T19:35:10.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 8596 13.71 MB/sec warmup 46 sec latency 7127.378 ms 2024-08-28T19:35:11.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 9076 14.25 MB/sec warmup 47 sec latency 8127.404 ms 2024-08-28T19:35:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:11 smithi028 ceph-mon[21769]: pgmap v778: 129 pgs: 129 active+clean; 659 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 10 MiB/s wr, 553 op/s 2024-08-28T19:35:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:11 smithi028 ceph-mon[21769]: osdmap e100: 12 total, 12 up, 12 in 2024-08-28T19:35:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:11 smithi115 ceph-mon[22184]: pgmap v778: 129 pgs: 129 active+clean; 659 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 10 MiB/s wr, 553 op/s 2024-08-28T19:35:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:11 smithi115 ceph-mon[22184]: osdmap e100: 12 total, 12 up, 12 in 2024-08-28T19:35:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:11 smithi016 ceph-mon[18556]: pgmap v778: 129 pgs: 129 active+clean; 659 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 10 MiB/s wr, 553 op/s 2024-08-28T19:35:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:11 smithi016 ceph-mon[18556]: osdmap e100: 12 total, 12 up, 12 in 2024-08-28T19:35:12.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:35:12.430 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:35:12.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 9319 14.22 MB/sec warmup 48 sec latency 9127.430 ms 2024-08-28T19:35:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.898+0000 7f030c188640 1 -- 172.21.15.16:0/150863232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030410dc50 msgr2=0x7f03041100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.898+0000 7f030c188640 1 --2- 172.21.15.16:0/150863232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030410dc50 0x7f03041100d0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009ca0 tx=0x7f02fc02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.898+0000 7f030c188640 1 -- 172.21.15.16:0/150863232 shutdown_connections 2024-08-28T19:35:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.898+0000 7f030c188640 1 --2- 172.21.15.16:0/150863232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030410dc50 0x7f03041100d0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.898+0000 7f030c188640 1 --2- 172.21.15.16:0/150863232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0304104ff0 0x7f030410d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.898+0000 7f030c188640 1 --2- 172.21.15.16:0/150863232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03041046a0 0x7f0304104ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.898+0000 7f030c188640 1 -- 172.21.15.16:0/150863232 >> 172.21.15.16:0/150863232 conn(0x7f0304079500 msgr2=0x7f0304079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.898+0000 7f030c188640 1 -- 172.21.15.16:0/150863232 shutdown_connections 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.898+0000 7f030c188640 1 -- 172.21.15.16:0/150863232 wait complete. 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f030c188640 1 Processor -- start 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f030c188640 1 -- start start 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f030c188640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041046a0 0x7f030419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f030c188640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0304104ff0 0x7f030419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f030c188640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030410dc50 0x7f030419f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f030c188640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0304113840 con 0x7f03041046a0 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f030c188640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03041136c0 con 0x7f030410dc50 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f030c188640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f03041139c0 con 0x7f0304104ff0 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f0309f24640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041046a0 0x7f030419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f0309f24640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041046a0 0x7f030419a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38746/0 (socket says 172.21.15.16:38746) 2024-08-28T19:35:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f0309f24640 1 -- 172.21.15.16:0/588789967 learned_addr learned my addr 172.21.15.16:0/588789967 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f0309723640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0304104ff0 0x7f030419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.899+0000 7f030a725640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030410dc50 0x7f030419f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.900+0000 7f0309f24640 1 -- 172.21.15.16:0/588789967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0304104ff0 msgr2=0x7f030419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.900+0000 7f0309f24640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0304104ff0 0x7f030419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.900+0000 7f0309f24640 1 -- 172.21.15.16:0/588789967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030410dc50 msgr2=0x7f030419f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.900+0000 7f0309f24640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030410dc50 0x7f030419f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.900+0000 7f0309f24640 1 -- 172.21.15.16:0/588789967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03041a0100 con 0x7f03041046a0 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.900+0000 7f0309f24640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041046a0 0x7f030419a190 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f030000e9c0 tx=0x7f030000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.900+0000 7f02faffd640 1 -- 172.21.15.16:0/588789967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030000cd80 con 0x7f03041046a0 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.900+0000 7f02faffd640 1 -- 172.21.15.16:0/588789967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03000035f0 con 0x7f03041046a0 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.900+0000 7f02faffd640 1 -- 172.21.15.16:0/588789967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0300010690 con 0x7f03041046a0 2024-08-28T19:35:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.900+0000 7f030c188640 1 -- 172.21.15.16:0/588789967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03041a03f0 con 0x7f03041046a0 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.900+0000 7f030c188640 1 -- 172.21.15.16:0/588789967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03041a6b50 con 0x7f03041046a0 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.903+0000 7f02faffd640 1 -- 172.21.15.16:0/588789967 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f03000037a0 con 0x7f03041046a0 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.903+0000 7f03126fd740 1 Processor -- start 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.903+0000 7f03126fd740 1 -- start start 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.903+0000 7f03126fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c63e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.903+0000 7f03126fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c2701b0 0x55dd2c63e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.903+0000 7f03126fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 0x55dd2c63ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.903+0000 7f03126fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd2c5b5630 con 0x55dd2c278390 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.903+0000 7f03126fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dd2c5a4bc0 con 0x55dd2c299070 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.903+0000 7f03126fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55dd2c5a8b70 con 0x55dd2c2701b0 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f030a725640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c63e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f0309723640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 0x55dd2c63ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f0309f24640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c2701b0 0x55dd2c63e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f030a725640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c63e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:32956/0 (socket says 172.21.15.16:32956) 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f030a725640 1 -- 172.21.15.16:0/418541497 learned_addr learned my addr 172.21.15.16:0/418541497 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f030a725640 1 -- 172.21.15.16:0/418541497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c2701b0 msgr2=0x55dd2c63e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f030a725640 1 --2- 172.21.15.16:0/418541497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c2701b0 0x55dd2c63e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f030a725640 1 -- 172.21.15.16:0/418541497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 msgr2=0x55dd2c63ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f030a725640 1 --2- 172.21.15.16:0/418541497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 0x55dd2c63ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f02faffd640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f02f00b4d20 0x7f02f00b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f030a725640 1 -- 172.21.15.16:0/418541497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd2c304650 con 0x55dd2c299070 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f0309723640 1 --2- 172.21.15.16:0/418541497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 0x55dd2c63ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f0309f24640 1 --2- 172.21.15.16:0/418541497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c2701b0 0x55dd2c63e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f0309f24640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f02f00b4d20 0x7f02f00b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f030a725640 1 --2- 172.21.15.16:0/418541497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c63e2c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f02fc035000 tx=0x7f02fc02f410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aea830728d6b58f1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f02faffd640 1 -- 172.21.15.16:0/588789967 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f03000e5c70 con 0x7f03041046a0 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f02effff640 1 -- 172.21.15.16:0/418541497 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc031cf0 con 0x55dd2c299070 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f02effff640 1 -- 172.21.15.16:0/418541497 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02fc036030 con 0x55dd2c299070 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.904+0000 7f02effff640 1 -- 172.21.15.16:0/418541497 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0042a0 con 0x55dd2c299070 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.905+0000 7f03126fd740 1 -- 172.21.15.16:0/418541497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 msgr2=0x55dd2c63e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.905+0000 7f0309f24640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f02f00b4d20 0x7f02f00b7230 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f03000e30a0 tx=0x7f0300093040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.905+0000 7f03126fd740 1 --2- 172.21.15.16:0/418541497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c63e2c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f02fc035000 tx=0x7f02fc02f410 comp rx=0 tx=0).stop 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.905+0000 7f03126fd740 1 -- 172.21.15.16:0/418541497 shutdown_connections 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.905+0000 7f03126fd740 1 --2- 172.21.15.16:0/418541497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 0x55dd2c63ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.905+0000 7f03126fd740 1 --2- 172.21.15.16:0/418541497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c2701b0 0x55dd2c63e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.905+0000 7f03126fd740 1 --2- 172.21.15.16:0/418541497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c63e2c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.905+0000 7f03126fd740 1 -- 172.21.15.16:0/418541497 >> 172.21.15.16:0/418541497 conn(0x55dd2c28d460 msgr2=0x55dd2c63c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.905+0000 7f03126fd740 1 -- 172.21.15.16:0/418541497 shutdown_connections 2024-08-28T19:35:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.905+0000 7f03126fd740 1 -- 172.21.15.16:0/418541497 wait complete. 2024-08-28T19:35:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.906+0000 7f03126fd740 1 -- start start 2024-08-28T19:35:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.906+0000 7f03126fd740 1 Processor -- start 2024-08-28T19:35:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.906+0000 7f03126fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c2701b0 0x55dd2c649a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.906+0000 7f03126fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c278390 0x55dd2c649f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.906+0000 7f03126fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c655bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f03126fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd2c407700 con 0x55dd2c2701b0 2024-08-28T19:35:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f03126fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dd2c52b700 con 0x55dd2c299070 2024-08-28T19:35:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f03126fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55dd2c64c8c0 con 0x55dd2c278390 2024-08-28T19:35:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f03126fd740 10 client.0 _ll_register_callbacks cb 0x55dd2c55adc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f030a725640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c2701b0 0x55dd2c649a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f030a725640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c2701b0 0x55dd2c649a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38762/0 (socket says 172.21.15.16:38762) 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f030a725640 1 -- 172.21.15.16:0/1964195752 learned_addr learned my addr 172.21.15.16:0/1964195752 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f030a725640 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c655bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f0309723640 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c278390 0x55dd2c649f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f030a725640 1 -- 172.21.15.16:0/1964195752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c278390 msgr2=0x55dd2c649f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f030a725640 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c278390 0x55dd2c649f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f030a725640 1 -- 172.21.15.16:0/1964195752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 msgr2=0x55dd2c655bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f030a725640 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c655bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f030a725640 1 -- 172.21.15.16:0/1964195752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd2c5648b0 con 0x55dd2c2701b0 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f0309723640 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c278390 0x55dd2c649f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f030a725640 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c2701b0 0x55dd2c649a40 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f02fc02a900 tx=0x7f02fc003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0ac4da67da1cb2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f02ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f02ed7fa640 1 -- 172.21.15.16:0/1964195752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc059060 con 0x55dd2c2701b0 2024-08-28T19:35:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.907+0000 7f02ed7fa640 1 -- 172.21.15.16:0/1964195752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f02fc02f660 con 0x55dd2c2701b0 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f03126fd740 1 -- 172.21.15.16:0/1964195752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02fc036030 con 0x55dd2c2701b0 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f02ed7fa640 1 -- 172.21.15.16:0/1964195752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f02fc004b40 con 0x55dd2c2701b0 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f03126fd740 10 client.18963 fetch_fsmap learned FSMap version 74 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f03126fd740 1 -- 172.21.15.16:0/1964195752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=4+}) -- 0x55dd2c6586a0 con 0x55dd2c2701b0 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f02ed7fa640 1 -- 172.21.15.16:0/1964195752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02fc036030 con 0x55dd2c2701b0 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f02ed7fa640 1 -- 172.21.15.16:0/1964195752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 8430+0+0 (secure 0 0 0) 0x7f02fc043070 con 0x55dd2c2701b0 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f03126fd740 10 client.18963 fetch_fsmap finished waiting for FSMap version 74 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f03126fd740 10 client.18963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f03126fd740 20 client.18963 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f03126fd740 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dd2c2579f0 0x55dd2c65ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f03126fd740 4 client.18963 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f03126fd740 1 -- 172.21.15.16:0/1964195752 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd2c5b7580 con 0x55dd2c2579f0 2024-08-28T19:35:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.908+0000 7f0309723640 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dd2c2579f0 0x55dd2c65ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.909+0000 7f02ed7fa640 10 client.18963 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.909+0000 7f0309723640 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dd2c2579f0 0x55dd2c65ba00 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35c8a0a6b6349fbe server_cookie=8886b99ae7ee4f64 in_seq=0 out_seq=0 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f02ed7fa640 1 -- 172.21.15.16:0/1964195752 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55dd2c5b7580 con 0x55dd2c2579f0 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f02ed7fa640 10 client.18963 handle_command_reply: tid=0 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 1 client.18963 shutdown 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 1 -- 172.21.15.16:0/1964195752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c2701b0 msgr2=0x55dd2c649a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c2701b0 0x55dd2c649a40 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f02fc02a900 tx=0x7f02fc003d10 comp rx=0 tx=0).stop 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 1 -- 172.21.15.16:0/1964195752 shutdown_connections 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c655bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dd2c2579f0 0x55dd2c65ba00 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c278390 0x55dd2c649f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 1 --2- 172.21.15.16:0/1964195752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c2701b0 0x55dd2c649a40 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f02ed7fa640 0 client.18963 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 1 -- 172.21.15.16:0/1964195752 >> 172.21.15.16:0/1964195752 conn(0x55dd2c28d460 msgr2=0x55dd2c6414a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 1 -- 172.21.15.16:0/1964195752 shutdown_connections 2024-08-28T19:35:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 1 -- 172.21.15.16:0/1964195752 wait complete. 2024-08-28T19:35:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.914+0000 7f03126fd740 20 client.18963 trim_cache size 0 max 16384 2024-08-28T19:35:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.932+0000 7f03126fd740 1 Processor -- start 2024-08-28T19:35:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.932+0000 7f03126fd740 1 -- start start 2024-08-28T19:35:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f03126fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c2701b0 0x55dd2c6451f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f03126fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 0x55dd2c63fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f03126fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c646c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f03126fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03041b0c70 con 0x55dd2c278390 2024-08-28T19:35:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f03126fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03041b29c0 con 0x55dd2c299070 2024-08-28T19:35:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f03126fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f03041b13d0 con 0x55dd2c2701b0 2024-08-28T19:35:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f0309723640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 0x55dd2c63fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f0309723640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 0x55dd2c63fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38770/0 (socket says 172.21.15.16:38770) 2024-08-28T19:35:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f0309723640 1 -- 172.21.15.16:0/996912992 learned_addr learned my addr 172.21.15.16:0/996912992 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f030a725640 1 --2- 172.21.15.16:0/996912992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c2701b0 0x55dd2c6451f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.042 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:35:13.042 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:35:13.042 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:35:13.042 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:35:13.042 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:35:13.042 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:35:13.042 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.042 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.042 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.042 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.042 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:35:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:35:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:35:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:35:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:35:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:35:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.046 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.047 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.048 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.049 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:35:13.050 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.051 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:35:13.052 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.053 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:13.054 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:35:13.054 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:35:13.054 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:13.054 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:13.054 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:13.054 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:13.054 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:13.054 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:35:13.054 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:35:13.054 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:35:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f030a725640 1 --2- 172.21.15.16:0/996912992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c646c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f0309723640 1 -- 172.21.15.16:0/996912992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c2701b0 msgr2=0x55dd2c6451f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f0309723640 1 --2- 172.21.15.16:0/996912992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c2701b0 0x55dd2c6451f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f0309723640 1 -- 172.21.15.16:0/996912992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 msgr2=0x55dd2c646c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f0309723640 1 --2- 172.21.15.16:0/996912992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c646c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f0309723640 1 -- 172.21.15.16:0/996912992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd2c647f40 con 0x55dd2c278390 2024-08-28T19:35:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f030a725640 1 --2- 172.21.15.16:0/996912992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c646c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:35:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f0309723640 1 --2- 172.21.15.16:0/996912992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 0x55dd2c63fa00 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f030410f9e0 tx=0x7f02f4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358057c224025e41 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f02ef7fe640 1 -- 172.21.15.16:0/996912992 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4004310 con 0x55dd2c278390 2024-08-28T19:35:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f02ef7fe640 1 -- 172.21.15.16:0/996912992 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02f40044b0 con 0x55dd2c278390 2024-08-28T19:35:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.933+0000 7f02ef7fe640 1 -- 172.21.15.16:0/996912992 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f40035f0 con 0x55dd2c278390 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.934+0000 7f03126fd740 1 -- 172.21.15.16:0/996912992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 msgr2=0x55dd2c63fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.934+0000 7f03126fd740 1 --2- 172.21.15.16:0/996912992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 0x55dd2c63fa00 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f030410f9e0 tx=0x7f02f4005f90 comp rx=0 tx=0).stop 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.934+0000 7f03126fd740 1 -- 172.21.15.16:0/996912992 shutdown_connections 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.934+0000 7f03126fd740 1 --2- 172.21.15.16:0/996912992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c299070 0x55dd2c646c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.934+0000 7f03126fd740 1 --2- 172.21.15.16:0/996912992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c278390 0x55dd2c63fa00 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.934+0000 7f03126fd740 1 --2- 172.21.15.16:0/996912992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c2701b0 0x55dd2c6451f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.934+0000 7f03126fd740 1 -- 172.21.15.16:0/996912992 >> 172.21.15.16:0/996912992 conn(0x55dd2c28d460 msgr2=0x55dd2c63f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.934+0000 7f03126fd740 1 -- 172.21.15.16:0/996912992 shutdown_connections 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.934+0000 7f03126fd740 1 -- 172.21.15.16:0/996912992 wait complete. 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.935+0000 7f03126fd740 1 -- start start 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.935+0000 7f03126fd740 1 Processor -- start 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.935+0000 7f03126fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c2701b0 0x55dd2c65e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.935+0000 7f03126fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c278390 0x55dd2c65ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.935+0000 7f03126fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c299070 0x55dd2c6bb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.935+0000 7f03126fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03041b22c0 con 0x55dd2c299070 2024-08-28T19:35:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.935+0000 7f03126fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03041b1b30 con 0x55dd2c2701b0 2024-08-28T19:35:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.935+0000 7f03126fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f03041aec70 con 0x55dd2c278390 2024-08-28T19:35:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.935+0000 7f03126fd740 10 client.0 _ll_register_callbacks cb 0x7f02f40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.935+0000 7f030a725640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c299070 0x55dd2c6bb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f030a725640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c299070 0x55dd2c6bb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38782/0 (socket says 172.21.15.16:38782) 2024-08-28T19:35:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f030a725640 1 -- 172.21.15.16:0/3645794586 learned_addr learned my addr 172.21.15.16:0/3645794586 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f030a725640 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c2701b0 0x55dd2c65e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f0309723640 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c278390 0x55dd2c65ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f030a725640 1 -- 172.21.15.16:0/3645794586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c278390 msgr2=0x55dd2c65ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f030a725640 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c278390 0x55dd2c65ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f030a725640 1 -- 172.21.15.16:0/3645794586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c2701b0 msgr2=0x55dd2c65e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f030a725640 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c2701b0 0x55dd2c65e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f030a725640 1 -- 172.21.15.16:0/3645794586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd2c6bb800 con 0x55dd2c299070 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f030a725640 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c299070 0x55dd2c6bb190 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0309d0 tx=0x7f02fc003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced8e6fb48e8dbdd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f02ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f02ed7fa640 1 -- 172.21.15.16:0/3645794586 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc059040 con 0x55dd2c299070 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f03126fd740 1 -- 172.21.15.16:0/3645794586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02f40044b0 con 0x55dd2c299070 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f02ed7fa640 1 -- 172.21.15.16:0/3645794586 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f02fc030a50 con 0x55dd2c299070 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f02ed7fa640 1 -- 172.21.15.16:0/3645794586 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f02fc009b20 con 0x55dd2c299070 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f03126fd740 10 client.18969 fetch_fsmap learned FSMap version 74 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.936+0000 7f03126fd740 1 -- 172.21.15.16:0/3645794586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=4+}) -- 0x55dd2c6c5e40 con 0x55dd2c299070 2024-08-28T19:35:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.937+0000 7f02ed7fa640 1 -- 172.21.15.16:0/3645794586 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02f40044b0 con 0x55dd2c299070 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.937+0000 7f02ed7fa640 1 -- 172.21.15.16:0/3645794586 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 8430+0+0 (secure 0 0 0) 0x7f02fc043040 con 0x55dd2c299070 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.937+0000 7f03126fd740 10 client.18969 fetch_fsmap finished waiting for FSMap version 74 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.937+0000 7f03126fd740 10 client.18969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.937+0000 7f03126fd740 20 client.18969 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.937+0000 7f03126fd740 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dd2c22cda0 0x55dd2c6c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.937+0000 7f03126fd740 4 client.18969 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.937+0000 7f03126fd740 1 -- 172.21.15.16:0/3645794586 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dd2c5b3020 con 0x55dd2c22cda0 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.937+0000 7f0309723640 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dd2c22cda0 0x55dd2c6c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.937+0000 7f0309723640 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dd2c22cda0 0x55dd2c6c8320 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85498efcab70dec0 server_cookie=c66049859d77ed10 in_seq=0 out_seq=0 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.937+0000 7f02ed7fa640 10 client.18969 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f02ed7fa640 1 -- 172.21.15.16:0/3645794586 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+5486 (crc 0 0 0) 0x55dd2c5b3020 con 0x55dd2c22cda0 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f02ed7fa640 10 client.18969 handle_command_reply: tid=0 2024-08-28T19:35:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 1 client.18969 shutdown 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 1 -- 172.21.15.16:0/3645794586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c299070 msgr2=0x55dd2c6bb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c299070 0x55dd2c6bb190 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0309d0 tx=0x7f02fc003d10 comp rx=0 tx=0).stop 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 1 -- 172.21.15.16:0/3645794586 shutdown_connections 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd2c299070 0x55dd2c6bb190 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dd2c22cda0 0x55dd2c6c8320 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dd2c278390 0x55dd2c65ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 1 --2- 172.21.15.16:0/3645794586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd2c2701b0 0x55dd2c65e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f02ed7fa640 0 client.18969 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 1 -- 172.21.15.16:0/3645794586 >> 172.21.15.16:0/3645794586 conn(0x55dd2c28d460 msgr2=0x55dd2c64ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 1 -- 172.21.15.16:0/3645794586 shutdown_connections 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 1 -- 172.21.15.16:0/3645794586 wait complete. 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.940+0000 7f03126fd740 20 client.18969 trim_cache size 0 max 16384 2024-08-28T19:35:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 -- 172.21.15.16:0/588789967 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f02f00b4d20 msgr2=0x7f02f00b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f02f00b4d20 0x7f02f00b7230 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f03000e30a0 tx=0x7f0300093040 comp rx=0 tx=0).stop 2024-08-28T19:35:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 -- 172.21.15.16:0/588789967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041046a0 msgr2=0x7f030419a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041046a0 0x7f030419a190 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f030000e9c0 tx=0x7f030000ee80 comp rx=0 tx=0).stop 2024-08-28T19:35:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 -- 172.21.15.16:0/588789967 shutdown_connections 2024-08-28T19:35:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f02f00b4d20 0x7f02f00b7230 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030410dc50 0x7f030419f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0304104ff0 0x7f030419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 --2- 172.21.15.16:0/588789967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041046a0 0x7f030419a190 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 -- 172.21.15.16:0/588789967 >> 172.21.15.16:0/588789967 conn(0x7f0304079500 msgr2=0x7f03041056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 -- 172.21.15.16:0/588789967 shutdown_connections 2024-08-28T19:35:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:12.941+0000 7f030c188640 1 -- 172.21.15.16:0/588789967 wait complete. 2024-08-28T19:35:13.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 9505 14.00 MB/sec warmup 49 sec latency 10127.463 ms 2024-08-28T19:35:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:13 smithi028 ceph-mon[21769]: pgmap v780: 129 pgs: 129 active+clean; 669 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.7 MiB/s wr, 544 op/s 2024-08-28T19:35:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:13 smithi115 ceph-mon[22184]: pgmap v780: 129 pgs: 129 active+clean; 669 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.7 MiB/s wr, 544 op/s 2024-08-28T19:35:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:13 smithi016 ceph-mon[18556]: pgmap v780: 129 pgs: 129 active+clean; 669 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.7 MiB/s wr, 544 op/s 2024-08-28T19:35:13.812 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:35:13.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:35:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:35:14.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 9962 14.28 MB/sec warmup 50 sec latency 10450.502 ms 2024-08-28T19:35:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.718+0000 7f794a41b640 1 -- 172.21.15.16:0/3030293065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7944108140 msgr2=0x7f7944108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.718+0000 7f794a41b640 1 --2- 172.21.15.16:0/3030293065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7944108140 0x7f7944108550 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f7938009900 tx=0x7f793802a410 comp rx=0 tx=0).stop 2024-08-28T19:35:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.718+0000 7f794a41b640 1 -- 172.21.15.16:0/3030293065 shutdown_connections 2024-08-28T19:35:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.718+0000 7f794a41b640 1 --2- 172.21.15.16:0/3030293065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7944108a90 0x7f794410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.718+0000 7f794a41b640 1 --2- 172.21.15.16:0/3030293065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7944108140 0x7f7944108550 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.718+0000 7f794a41b640 1 --2- 172.21.15.16:0/3030293065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944102800 0x7f7944102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.718+0000 7f794a41b640 1 -- 172.21.15.16:0/3030293065 >> 172.21.15.16:0/3030293065 conn(0x7f79440794d0 msgr2=0x7f79440798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.718+0000 7f794a41b640 1 -- 172.21.15.16:0/3030293065 shutdown_connections 2024-08-28T19:35:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.718+0000 7f794a41b640 1 -- 172.21.15.16:0/3030293065 wait complete. 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.718+0000 7f794a41b640 1 Processor -- start 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.719+0000 7f794a41b640 1 -- start start 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.719+0000 7f794a41b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7944102800 0x7f794419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.719+0000 7f794a41b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944108140 0x7f794419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.719+0000 7f794a41b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7944108a90 0x7f794419fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.719+0000 7f794a41b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7944113820 con 0x7f7944102800 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.719+0000 7f794a41b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79441136a0 con 0x7f7944108140 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.719+0000 7f794a41b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f79441139a0 con 0x7f7944108a90 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.719+0000 7f79437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944108140 0x7f794419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.719+0000 7f79437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944108140 0x7f794419a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:46922/0 (socket says 172.21.15.16:46922) 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.719+0000 7f79437fe640 1 -- 172.21.15.16:0/1459982201 learned_addr learned my addr 172.21.15.16:0/1459982201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.719+0000 7f79489b8640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7944108a90 0x7f794419fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.720+0000 7f7943fff640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7944102800 0x7f794419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.720+0000 7f79437fe640 1 -- 172.21.15.16:0/1459982201 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7944108a90 msgr2=0x7f794419fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.720+0000 7f79437fe640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7944108a90 0x7f794419fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.720+0000 7f79437fe640 1 -- 172.21.15.16:0/1459982201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7944102800 msgr2=0x7f794419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.720+0000 7f79437fe640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7944102800 0x7f794419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.720+0000 7f79437fe640 1 -- 172.21.15.16:0/1459982201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79441a0180 con 0x7f7944108140 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.720+0000 7f7943fff640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7944102800 0x7f794419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.720+0000 7f79437fe640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944108140 0x7f794419a7b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7938032fd0 tx=0x7f7938031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.720+0000 7f79417fa640 1 -- 172.21.15.16:0/1459982201 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793803f030 con 0x7f7944108140 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.720+0000 7f79417fa640 1 -- 172.21.15.16:0/1459982201 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79380043d0 con 0x7f7944108140 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.720+0000 7f794a41b640 1 -- 172.21.15.16:0/1459982201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79441a0410 con 0x7f7944108140 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.721+0000 7f79417fa640 1 -- 172.21.15.16:0/1459982201 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938035070 con 0x7f7944108140 2024-08-28T19:35:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.721+0000 7f794a41b640 1 -- 172.21.15.16:0/1459982201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f794410fc30 con 0x7f7944108140 2024-08-28T19:35:14.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.722+0000 7f794a41b640 1 -- 172.21.15.16:0/1459982201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7944103a80 con 0x7f7944108140 2024-08-28T19:35:14.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.724+0000 7f79417fa640 1 -- 172.21.15.16:0/1459982201 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7938042030 con 0x7f7944108140 2024-08-28T19:35:14.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.724+0000 7f79417fa640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f792c0b4ca0 0x7f792c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:14.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.725+0000 7f7943fff640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f792c0b4ca0 0x7f792c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:14.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.725+0000 7f79417fa640 1 -- 172.21.15.16:0/1459982201 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f7938106130 con 0x7f7944108140 2024-08-28T19:35:14.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.725+0000 7f7943fff640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f792c0b4ca0 0x7f792c0b71b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7934009c20 tx=0x7f7934005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:14.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.730+0000 7f79417fa640 1 -- 172.21.15.16:0/1459982201 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f79380ccb50 con 0x7f7944108140 2024-08-28T19:35:14.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.976+0000 7f794a41b640 1 -- 172.21.15.16:0/1459982201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f794410a860 con 0x7f7944108140 2024-08-28T19:35:14.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.977+0000 7f79417fa640 1 -- 172.21.15.16:0/1459982201 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+12002 (secure 0 0 0) 0x7f79380cc450 con 0x7f7944108140 2024-08-28T19:35:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:35:14.980 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":74,"btime":"2024-08-28T19:34:57:400363+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":74,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:34:56.971796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:35:14.980 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 74 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.980+0000 7f794a41b640 1 -- 172.21.15.16:0/1459982201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f792c0b4ca0 msgr2=0x7f792c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.980+0000 7f794a41b640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f792c0b4ca0 0x7f792c0b71b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7934009c20 tx=0x7f7934005f90 comp rx=0 tx=0).stop 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.980+0000 7f794a41b640 1 -- 172.21.15.16:0/1459982201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944108140 msgr2=0x7f794419a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.980+0000 7f794a41b640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944108140 0x7f794419a7b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7938032fd0 tx=0x7f7938031a20 comp rx=0 tx=0).stop 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.980+0000 7f794a41b640 1 -- 172.21.15.16:0/1459982201 shutdown_connections 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.981+0000 7f794a41b640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7944108a90 0x7f794419fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.981+0000 7f794a41b640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f792c0b4ca0 0x7f792c0b71b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.981+0000 7f794a41b640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944108140 0x7f794419a7b0 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7938032fd0 tx=0x7f7938031a20 comp rx=0 tx=0).stop 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.981+0000 7f794a41b640 1 --2- 172.21.15.16:0/1459982201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7944102800 0x7f794419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.981+0000 7f794a41b640 1 -- 172.21.15.16:0/1459982201 >> 172.21.15.16:0/1459982201 conn(0x7f79440794d0 msgr2=0x7f7944104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.981+0000 7f794a41b640 1 -- 172.21.15.16:0/1459982201 shutdown_connections 2024-08-28T19:35:14.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:14.981+0000 7f794a41b640 1 -- 172.21.15.16:0/1459982201 wait complete. 2024-08-28T19:35:15.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10535 14.52 MB/sec warmup 51 sec latency 1811.403 ms 2024-08-28T19:35:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:15 smithi115 ceph-mon[22184]: pgmap v781: 129 pgs: 129 active+clean; 674 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.5 MiB/s wr, 413 op/s 2024-08-28T19:35:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:15 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1459982201' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:35:15.705 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:15 smithi016 ceph-mon[18556]: pgmap v781: 129 pgs: 129 active+clean; 674 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.5 MiB/s wr, 413 op/s 2024-08-28T19:35:15.705 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:15 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1459982201' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:35:15.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:35:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:15 smithi028 ceph-mon[21769]: pgmap v781: 129 pgs: 129 active+clean; 674 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.5 MiB/s wr, 413 op/s 2024-08-28T19:35:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:15 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1459982201' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:35:15.994 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:35:16.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 11322 14.80 MB/sec warmup 52 sec latency 2811.433 ms 2024-08-28T19:35:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.504+0000 7f7c13a97640 1 -- 172.21.15.16:0/188066275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c102800 msgr2=0x7f7c0c102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.504+0000 7f7c13a97640 1 --2- 172.21.15.16:0/188066275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c102800 0x7f7c0c102c10 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009900 tx=0x7f7c0802a410 comp rx=0 tx=0).stop 2024-08-28T19:35:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.504+0000 7f7c13a97640 1 -- 172.21.15.16:0/188066275 shutdown_connections 2024-08-28T19:35:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.504+0000 7f7c13a97640 1 --2- 172.21.15.16:0/188066275 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c108a90 0x7f7c0c10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.504+0000 7f7c13a97640 1 --2- 172.21.15.16:0/188066275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c108140 0x7f7c0c108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.504+0000 7f7c13a97640 1 --2- 172.21.15.16:0/188066275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c102800 0x7f7c0c102c10 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.504+0000 7f7c13a97640 1 -- 172.21.15.16:0/188066275 >> 172.21.15.16:0/188066275 conn(0x7f7c0c0794d0 msgr2=0x7f7c0c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.504+0000 7f7c13a97640 1 -- 172.21.15.16:0/188066275 shutdown_connections 2024-08-28T19:35:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.504+0000 7f7c13a97640 1 -- 172.21.15.16:0/188066275 wait complete. 2024-08-28T19:35:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.505+0000 7f7c13a97640 1 Processor -- start 2024-08-28T19:35:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.505+0000 7f7c13a97640 1 -- start start 2024-08-28T19:35:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c13a97640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c102800 0x7f7c0c19c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c13a97640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c108140 0x7f7c0c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c13a97640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c108a90 0x7f7c0c1a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c13a97640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c112740 con 0x7f7c0c102800 2024-08-28T19:35:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c13a97640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1125c0 con 0x7f7c0c108a90 2024-08-28T19:35:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c13a97640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1128c0 con 0x7f7c0c108140 2024-08-28T19:35:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c11032640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c108140 0x7f7c0c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c11032640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c108140 0x7f7c0c19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56648/0 (socket says 172.21.15.16:56648) 2024-08-28T19:35:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c11032640 1 -- 172.21.15.16:0/3412628892 learned_addr learned my addr 172.21.15.16:0/3412628892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c11833640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c102800 0x7f7c0c19c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c12034640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c108a90 0x7f7c0c1a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c11032640 1 -- 172.21.15.16:0/3412628892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c108a90 msgr2=0x7f7c0c1a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c11032640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c108a90 0x7f7c0c1a1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c11032640 1 -- 172.21.15.16:0/3412628892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c102800 msgr2=0x7f7c0c19c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c11032640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c102800 0x7f7c0c19c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.506+0000 7f7c11032640 1 -- 172.21.15.16:0/3412628892 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c1a2410 con 0x7f7c0c108140 2024-08-28T19:35:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.507+0000 7f7c11032640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c108140 0x7f7c0c19ca40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00e9c0 tx=0x7f7bfc00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.507+0000 7f7c02ffd640 1 -- 172.21.15.16:0/3412628892 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc00ccc0 con 0x7f7c0c108140 2024-08-28T19:35:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.507+0000 7f7c02ffd640 1 -- 172.21.15.16:0/3412628892 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bfc0054d0 con 0x7f7c0c108140 2024-08-28T19:35:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.507+0000 7f7c13a97640 1 -- 172.21.15.16:0/3412628892 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0c1a2700 con 0x7f7c0c108140 2024-08-28T19:35:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.507+0000 7f7c02ffd640 1 -- 172.21.15.16:0/3412628892 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc010430 con 0x7f7c0c108140 2024-08-28T19:35:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.507+0000 7f7c13a97640 1 -- 172.21.15.16:0/3412628892 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c0c1a8e60 con 0x7f7c0c108140 2024-08-28T19:35:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c1a00c740 1 Processor -- start 2024-08-28T19:35:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c1a00c740 1 -- start start 2024-08-28T19:35:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c02ffd640 1 -- 172.21.15.16:0/3412628892 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7bfc005020 con 0x7f7c0c108140 2024-08-28T19:35:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff224d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff224d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 0x561ff224dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ff21c4680 con 0x561ff1ea8070 2024-08-28T19:35:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ff21b3c10 con 0x561ff1e7f1b0 2024-08-28T19:35:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561ff21b7bc0 con 0x561ff1e87390 2024-08-28T19:35:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c12034640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff224d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c12034640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff224d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37076/0 (socket says 172.21.15.16:37076) 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c11833640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff224d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c12034640 1 -- 172.21.15.16:0/3438457829 learned_addr learned my addr 172.21.15.16:0/3438457829 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.511+0000 7f7c11032640 1 --2- 172.21.15.16:0/3438457829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 0x561ff224dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.512+0000 7f7c12034640 1 -- 172.21.15.16:0/3438457829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 msgr2=0x561ff224dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.512+0000 7f7c12034640 1 --2- 172.21.15.16:0/3438457829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 0x561ff224dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.512+0000 7f7c12034640 1 -- 172.21.15.16:0/3438457829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 msgr2=0x561ff224d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.512+0000 7f7c12034640 1 --2- 172.21.15.16:0/3438457829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff224d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.512+0000 7f7c12034640 1 -- 172.21.15.16:0/3438457829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ff1f13650 con 0x561ff1ea8070 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.512+0000 7f7c11032640 1 --2- 172.21.15.16:0/3438457829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 0x561ff224dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.512+0000 7f7c12034640 1 --2- 172.21.15.16:0/3438457829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff224d310 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009f60 tx=0x7f7c04009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa5ebc3db9d38b5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.512+0000 7f7c02ffd640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf80b4d20 0x7f7bf80b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.512+0000 7f7bf7fff640 1 -- 172.21.15.16:0/3438457829 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0401e070 con 0x561ff1ea8070 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.513+0000 7f7c11833640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf80b4d20 0x7f7bf80b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.513+0000 7f7c02ffd640 1 -- 172.21.15.16:0/3412628892 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f7bfc0e5c20 con 0x7f7c0c108140 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.513+0000 7f7bf7fff640 1 -- 172.21.15.16:0/3438457829 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c04003c30 con 0x561ff1ea8070 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.513+0000 7f7bf7fff640 1 -- 172.21.15.16:0/3438457829 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400f820 con 0x561ff1ea8070 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.514+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3438457829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 msgr2=0x561ff224d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.514+0000 7f7c11833640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf80b4d20 0x7f7bf80b7230 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7c08005ab0 tx=0x7f7c08005a20 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.514+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/3438457829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff224d310 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009f60 tx=0x7f7c04009f90 comp rx=0 tx=0).stop 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.514+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3438457829 shutdown_connections 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.514+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/3438457829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 0x561ff224dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.514+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/3438457829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff224d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.514+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/3438457829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff224d310 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.514+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3438457829 >> 172.21.15.16:0/3438457829 conn(0x561ff1e9c460 msgr2=0x561ff224b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.514+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3438457829 shutdown_connections 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.514+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3438457829 wait complete. 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.514+0000 7f7c1a00c740 1 -- start start 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c1a00c740 1 Processor -- start 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff2258af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1e87390 0x561ff2259050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1ea8070 0x561ff2264e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ff2016700 con 0x561ff1e87390 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ff213a750 con 0x561ff1e7f1b0 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561ff225bb90 con 0x561ff1ea8070 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c1a00c740 10 client.0 _ll_register_callbacks cb 0x561ff2169e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c11833640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff2258af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c11032640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1e87390 0x561ff2259050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c11833640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff2258af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:46960/0 (socket says 172.21.15.16:46960) 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c11833640 1 -- 172.21.15.16:0/3057846894 learned_addr learned my addr 172.21.15.16:0/3057846894 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.515+0000 7f7c12034640 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1ea8070 0x561ff2264e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7c11032640 1 -- 172.21.15.16:0/3057846894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1ea8070 msgr2=0x561ff2264e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7c11032640 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1ea8070 0x561ff2264e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7c11032640 1 -- 172.21.15.16:0/3057846894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 msgr2=0x561ff2258af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7c11032640 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff2258af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7c11032640 1 -- 172.21.15.16:0/3057846894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ff2173900 con 0x561ff1e87390 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7c11833640 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff2258af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7c11032640 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1e87390 0x561ff2259050 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0e2fa0 tx=0x7f7bfc0a7f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a458aa7460dd574c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7bf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7bf57fa640 1 -- 172.21.15.16:0/3057846894 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc00ce60 con 0x561ff1e87390 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7bf57fa640 1 -- 172.21.15.16:0/3057846894 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f7bfc0b4fe0 con 0x561ff1e87390 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3057846894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c04003c30 con 0x561ff1e87390 2024-08-28T19:35:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7bf57fa640 1 -- 172.21.15.16:0/3057846894 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f7bfc005310 con 0x561ff1e87390 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7c1a00c740 10 client.18987 fetch_fsmap learned FSMap version 74 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.516+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3057846894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=4+}) -- 0x561ff2267a30 con 0x561ff1e87390 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.517+0000 7f7bf57fa640 1 -- 172.21.15.16:0/3057846894 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c04003c30 con 0x561ff1e87390 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.517+0000 7f7bf57fa640 1 -- 172.21.15.16:0/3057846894 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 8430+0+0 (secure 0 0 0) 0x7f7bfc0eb030 con 0x561ff1e87390 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.517+0000 7f7c1a00c740 10 client.18987 fetch_fsmap finished waiting for FSMap version 74 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.517+0000 7f7c1a00c740 10 client.18987 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.517+0000 7f7c1a00c740 20 client.18987 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.517+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561ff1e669f0 0x561ff226af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.517+0000 7f7c1a00c740 4 client.18987 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.517+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3057846894 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ff21c65d0 con 0x561ff1e669f0 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.517+0000 7f7c11833640 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561ff1e669f0 0x561ff226af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.518+0000 7f7c11833640 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561ff1e669f0 0x561ff226af60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d3790cd69292e2e9 server_cookie=96679900fd40231c in_seq=0 out_seq=0 2024-08-28T19:35:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.518+0000 7f7bf57fa640 10 client.18987 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:35:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.518+0000 7f7bf57fa640 1 -- 172.21.15.16:0/3057846894 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561ff21c65d0 con 0x561ff1e669f0 2024-08-28T19:35:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.518+0000 7f7bf57fa640 10 client.18987 handle_command_reply: tid=0 2024-08-28T19:35:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.518+0000 7f7c1a00c740 1 client.18987 shutdown 2024-08-28T19:35:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3057846894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1e87390 msgr2=0x561ff2259050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1e87390 0x561ff2259050 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0e2fa0 tx=0x7f7bfc0a7f30 comp rx=0 tx=0).stop 2024-08-28T19:35:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3057846894 shutdown_connections 2024-08-28T19:35:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561ff1e669f0 0x561ff226af60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1ea8070 0x561ff2264e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1e87390 0x561ff2259050 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/3057846894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff2258af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3057846894 >> 172.21.15.16:0/3057846894 conn(0x561ff1e9c460 msgr2=0x561ff224e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7bf57fa640 0 client.18987 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:35:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3057846894 shutdown_connections 2024-08-28T19:35:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7c1a00c740 1 -- 172.21.15.16:0/3057846894 wait complete. 2024-08-28T19:35:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.519+0000 7f7c1a00c740 20 client.18987 trim_cache size 0 max 16384 2024-08-28T19:35:16.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.538+0000 7f7c1a00c740 1 Processor -- start 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.538+0000 7f7c1a00c740 1 -- start start 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff224e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 0x561ff224e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff2255ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b2f20 con 0x561ff1ea8070 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b4c70 con 0x561ff1e7f1b0 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b3680 con 0x561ff1e87390 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c11833640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff224e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c11032640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 0x561ff224e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c11833640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff224e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:46968/0 (socket says 172.21.15.16:46968) 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c11833640 1 -- 172.21.15.16:0/2319771495 learned_addr learned my addr 172.21.15.16:0/2319771495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c12034640 1 --2- 172.21.15.16:0/2319771495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff2255ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c11032640 1 -- 172.21.15.16:0/2319771495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 msgr2=0x561ff224e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c11032640 1 --2- 172.21.15.16:0/2319771495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff224e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c11032640 1 -- 172.21.15.16:0/2319771495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 msgr2=0x561ff2255ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c11032640 1 --2- 172.21.15.16:0/2319771495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff2255ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c11032640 1 -- 172.21.15.16:0/2319771495 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ff226b460 con 0x561ff1e87390 2024-08-28T19:35:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.539+0000 7f7c11032640 1 --2- 172.21.15.16:0/2319771495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 0x561ff224e880 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0e3050 tx=0x7f7bfc0b0460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e93b9012e5f51fde server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.540+0000 7f7bf77fe640 1 -- 172.21.15.16:0/2319771495 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc0e8030 con 0x561ff1e87390 2024-08-28T19:35:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.540+0000 7f7bf77fe640 1 -- 172.21.15.16:0/2319771495 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bfc0046c0 con 0x561ff1e87390 2024-08-28T19:35:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.540+0000 7f7bf77fe640 1 -- 172.21.15.16:0/2319771495 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc0aba50 con 0x561ff1e87390 2024-08-28T19:35:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.540+0000 7f7c1a00c740 1 -- 172.21.15.16:0/2319771495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 msgr2=0x561ff224e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.540+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/2319771495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 0x561ff224e880 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0e3050 tx=0x7f7bfc0b0460 comp rx=0 tx=0).stop 2024-08-28T19:35:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.541+0000 7f7c1a00c740 1 -- 172.21.15.16:0/2319771495 shutdown_connections 2024-08-28T19:35:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.541+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/2319771495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff2255ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.541+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/2319771495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e87390 0x561ff224e880 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.541+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/2319771495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e7f1b0 0x561ff224e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.541+0000 7f7c1a00c740 1 -- 172.21.15.16:0/2319771495 >> 172.21.15.16:0/2319771495 conn(0x561ff1e9c460 msgr2=0x561ff2257370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.541+0000 7f7c1a00c740 1 -- 172.21.15.16:0/2319771495 shutdown_connections 2024-08-28T19:35:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.541+0000 7f7c1a00c740 1 -- 172.21.15.16:0/2319771495 wait complete. 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.541+0000 7f7c1a00c740 1 -- start start 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.541+0000 7f7c1a00c740 1 Processor -- start 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e7f1b0 0x561ff2261530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e87390 0x561ff225c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c1a00c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff225cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b4570 con 0x561ff1ea8070 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b3de0 con 0x561ff1e87390 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c1a00c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c1b0f20 con 0x561ff1e7f1b0 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c1a00c740 10 client.0 _ll_register_callbacks cb 0x7f7bfc004f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c11032640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e87390 0x561ff225c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c12034640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff225cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c11032640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e87390 0x561ff225c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:46980/0 (socket says 172.21.15.16:46980) 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c11032640 1 -- 172.21.15.16:0/1211715678 learned_addr learned my addr 172.21.15.16:0/1211715678 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c11833640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e7f1b0 0x561ff2261530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c12034640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff225cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37100/0 (socket says 172.21.15.16:37100) 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c11833640 1 -- 172.21.15.16:0/1211715678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e87390 msgr2=0x561ff225c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c11833640 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e87390 0x561ff225c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c11833640 1 -- 172.21.15.16:0/1211715678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 msgr2=0x561ff225cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c11833640 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff225cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c11833640 1 -- 172.21.15.16:0/1211715678 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ff225d610 con 0x561ff1e7f1b0 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c11032640 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e87390 0x561ff225c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.542+0000 7f7c12034640 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff225cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.543+0000 7f7c11833640 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e7f1b0 0x561ff2261530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7c08031440 tx=0x7f7c08009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=930c4475c8968e7d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.543+0000 7f7bf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:35:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.543+0000 7f7bf57fa640 1 -- 172.21.15.16:0/1211715678 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c08060060 con 0x561ff1e7f1b0 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.543+0000 7f7bf57fa640 1 -- 172.21.15.16:0/1211715678 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f7c08004b80 con 0x561ff1e7f1b0 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.543+0000 7f7c1a00c740 1 -- 172.21.15.16:0/1211715678 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561ff21c2070 con 0x561ff1e7f1b0 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.543+0000 7f7bf57fa640 1 -- 172.21.15.16:0/1211715678 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f7c08067050 con 0x561ff1e7f1b0 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.543+0000 7f7c1a00c740 10 client.28081 fetch_fsmap learned FSMap version 74 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.543+0000 7f7c1a00c740 1 -- 172.21.15.16:0/1211715678 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=4+}) -- 0x561ff22d5820 con 0x561ff1e7f1b0 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.543+0000 7f7bf57fa640 1 -- 172.21.15.16:0/1211715678 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c08004d80 con 0x561ff1e7f1b0 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7bf57fa640 1 -- 172.21.15.16:0/1211715678 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 74) ==== 8430+0+0 (secure 0 0 0) 0x7f7c08032720 con 0x561ff1e7f1b0 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7c1a00c740 10 client.28081 fetch_fsmap finished waiting for FSMap version 74 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7c1a00c740 10 client.28081 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7c1a00c740 20 client.28081 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561ff1e3bda0 0x561ff22d8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7c1a00c740 4 client.28081 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:35:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7c1a00c740 1 -- 172.21.15.16:0/1211715678 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ff21c2070 con 0x561ff1e3bda0 2024-08-28T19:35:16.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7c11032640 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561ff1e3bda0 0x561ff22d8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7bf57fa640 10 client.28081 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:35:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7c11032640 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561ff1e3bda0 0x561ff22d8050 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=70e1f516c1bd59af server_cookie=5c7cc0c892b7fa92 in_seq=0 out_seq=0 2024-08-28T19:35:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7bf57fa640 1 -- 172.21.15.16:0/1211715678 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ff21c2070 con 0x561ff1e3bda0 2024-08-28T19:35:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7bf57fa640 10 client.28081 handle_command_reply: tid=0 2024-08-28T19:35:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.544+0000 7f7c1a00c740 1 client.28081 shutdown 2024-08-28T19:35:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7c1a00c740 1 -- 172.21.15.16:0/1211715678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e7f1b0 msgr2=0x561ff2261530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e7f1b0 0x561ff2261530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7c08031440 tx=0x7f7c08009260 comp rx=0 tx=0).stop 2024-08-28T19:35:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7c1a00c740 1 -- 172.21.15.16:0/1211715678 shutdown_connections 2024-08-28T19:35:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x561ff1e3bda0 0x561ff22d8050 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ff1ea8070 0x561ff225cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7bf57fa640 0 client.28081 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:35:16.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff1e87390 0x561ff225c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7c1a00c740 1 --2- 172.21.15.16:0/1211715678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ff1e7f1b0 0x561ff2261530 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7c1a00c740 1 -- 172.21.15.16:0/1211715678 >> 172.21.15.16:0/1211715678 conn(0x561ff1e9c460 msgr2=0x561ff224e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:16.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7c1a00c740 1 -- 172.21.15.16:0/1211715678 shutdown_connections 2024-08-28T19:35:16.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7c1a00c740 1 -- 172.21.15.16:0/1211715678 wait complete. 2024-08-28T19:35:16.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.545+0000 7f7c1a00c740 20 client.28081 trim_cache size 0 max 16384 2024-08-28T19:35:16.551 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:35:16.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 -- 172.21.15.16:0/3412628892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf80b4d20 msgr2=0x7f7bf80b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:16.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf80b4d20 0x7f7bf80b7230 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7c08005ab0 tx=0x7f7c08005a20 comp rx=0 tx=0).stop 2024-08-28T19:35:16.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 -- 172.21.15.16:0/3412628892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c108140 msgr2=0x7f7c0c19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:16.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c108140 0x7f7c0c19ca40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00e9c0 tx=0x7f7bfc00edc0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 -- 172.21.15.16:0/3412628892 shutdown_connections 2024-08-28T19:35:16.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7bf80b4d20 0x7f7bf80b7230 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c0c108a90 0x7f7c0c1a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c0c108140 0x7f7c0c19ca40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 --2- 172.21.15.16:0/3412628892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c0c102800 0x7f7c0c19c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:16.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 -- 172.21.15.16:0/3412628892 >> 172.21.15.16:0/3412628892 conn(0x7f7c0c0794d0 msgr2=0x7f7c0c06ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:16.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 -- 172.21.15.16:0/3412628892 shutdown_connections 2024-08-28T19:35:16.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:16.546+0000 7f7c13a97640 1 -- 172.21.15.16:0/3412628892 wait complete. 2024-08-28T19:35:17.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 12131 15.74 MB/sec warmup 53 sec latency 3811.481 ms 2024-08-28T19:35:17.483 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:17 smithi016 ceph-mon[18556]: pgmap v782: 129 pgs: 129 active+clean; 684 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 336 op/s 2024-08-28T19:35:17.519 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:35:17.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:17 smithi115 ceph-mon[22184]: pgmap v782: 129 pgs: 129 active+clean; 684 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 336 op/s 2024-08-28T19:35:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:17 smithi028 ceph-mon[21769]: pgmap v782: 129 pgs: 129 active+clean; 684 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 336 op/s 2024-08-28T19:35:17.810 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:35:18.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.373+0000 7f3f0bea4640 1 -- 172.21.15.16:0/3039026877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f04102810 msgr2=0x7f3f04102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.373+0000 7f3f0bea4640 1 --2- 172.21.15.16:0/3039026877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f04102810 0x7f3f04102c20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009860 tx=0x7f3f0002a340 comp rx=0 tx=0).stop 2024-08-28T19:35:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.373+0000 7f3f0bea4640 1 -- 172.21.15.16:0/3039026877 shutdown_connections 2024-08-28T19:35:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.373+0000 7f3f0bea4640 1 --2- 172.21.15.16:0/3039026877 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f04108aa0 0x7f3f0410f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.373+0000 7f3f0bea4640 1 --2- 172.21.15.16:0/3039026877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f04108150 0x7f3f04108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.373+0000 7f3f0bea4640 1 --2- 172.21.15.16:0/3039026877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f04102810 0x7f3f04102c20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.373+0000 7f3f0bea4640 1 -- 172.21.15.16:0/3039026877 >> 172.21.15.16:0/3039026877 conn(0x7f3f040794d0 msgr2=0x7f3f040798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.374+0000 7f3f0bea4640 1 -- 172.21.15.16:0/3039026877 shutdown_connections 2024-08-28T19:35:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.374+0000 7f3f0bea4640 1 -- 172.21.15.16:0/3039026877 wait complete. 2024-08-28T19:35:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.376+0000 7f3f0bea4640 1 Processor -- start 2024-08-28T19:35:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.376+0000 7f3f0bea4640 1 -- start start 2024-08-28T19:35:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.376+0000 7f3f0bea4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f04102810 0x7f3f041991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.376+0000 7f3f0bea4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f04108150 0x7f3f04199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.376+0000 7f3f0bea4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f04108aa0 0x7f3f0419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.376+0000 7f3f0bea4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f04112770 con 0x7f3f04108150 2024-08-28T19:35:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.376+0000 7f3f0bea4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f041125f0 con 0x7f3f04102810 2024-08-28T19:35:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.376+0000 7f3f0bea4640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3f041128f0 con 0x7f3f04108aa0 2024-08-28T19:35:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.376+0000 7f3f0943f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f04108150 0x7f3f04199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.377+0000 7f3f09c40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f04102810 0x7f3f041991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.377+0000 7f3f0943f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f04108150 0x7f3f04199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37116/0 (socket says 172.21.15.16:37116) 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.377+0000 7f3f0943f640 1 -- 172.21.15.16:0/570615096 learned_addr learned my addr 172.21.15.16:0/570615096 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.377+0000 7f3f0a441640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f04108aa0 0x7f3f0419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.377+0000 7f3f09c40640 1 -- 172.21.15.16:0/570615096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f04108aa0 msgr2=0x7f3f0419ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.377+0000 7f3f09c40640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f04108aa0 0x7f3f0419ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.377+0000 7f3f09c40640 1 -- 172.21.15.16:0/570615096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f04108150 msgr2=0x7f3f04199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.377+0000 7f3f09c40640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f04108150 0x7f3f04199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.377+0000 7f3f09c40640 1 -- 172.21.15.16:0/570615096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0419f120 con 0x7f3f04102810 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.378+0000 7f3f09c40640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f04102810 0x7f3f041991e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3f00035040 tx=0x7f3f00004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.378+0000 7f3efaffd640 1 -- 172.21.15.16:0/570615096 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00037040 con 0x7f3f04102810 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.378+0000 7f3efaffd640 1 -- 172.21.15.16:0/570615096 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f00004460 con 0x7f3f04102810 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.378+0000 7f3f0bea4640 1 -- 172.21.15.16:0/570615096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f0419f3b0 con 0x7f3f04102810 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.378+0000 7f3f0bea4640 1 -- 172.21.15.16:0/570615096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0406b880 con 0x7f3f04102810 2024-08-28T19:35:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.378+0000 7f3efaffd640 1 -- 172.21.15.16:0/570615096 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00004d70 con 0x7f3f04102810 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.381+0000 7f3f12419740 1 Processor -- start 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.382+0000 7f3efaffd640 1 -- 172.21.15.16:0/570615096 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3f00043020 con 0x7f3f04102810 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.382+0000 7f3f12419740 1 -- start start 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.382+0000 7f3f12419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd30070 0x564b8e0d5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.382+0000 7f3f12419740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd071b0 0x564b8e0d5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.382+0000 7f3f12419740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd0f390 0x564b8e0d5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f12419740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564b8e04c680 con 0x564b8dd071b0 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f12419740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b8e03bc10 con 0x564b8dd30070 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f12419740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564b8e03fbc0 con 0x564b8dd0f390 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f0a441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd30070 0x564b8e0d5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f0a441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd30070 0x564b8e0d5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:46998/0 (socket says 172.21.15.16:46998) 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f0a441640 1 -- 172.21.15.16:0/736306413 learned_addr learned my addr 172.21.15.16:0/736306413 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:18.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f09c40640 1 --2- 172.21.15.16:0/736306413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd071b0 0x564b8e0d5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3efaffd640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ef00b4dd0 0x7f3ef00b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f0a441640 1 -- 172.21.15.16:0/736306413 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd0f390 msgr2=0x564b8e0d5d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f0a441640 1 --2- 172.21.15.16:0/736306413 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd0f390 0x564b8e0d5d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f0a441640 1 -- 172.21.15.16:0/736306413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd071b0 msgr2=0x564b8e0d5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f0a441640 1 --2- 172.21.15.16:0/736306413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd071b0 0x564b8e0d5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f0a441640 1 -- 172.21.15.16:0/736306413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b8dd9b650 con 0x564b8dd30070 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3f09c40640 1 --2- 172.21.15.16:0/736306413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd071b0 0x564b8e0d5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.383+0000 7f3efaffd640 1 -- 172.21.15.16:0/570615096 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f3f00104bc0 con 0x7f3f04102810 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.384+0000 7f3f0a441640 1 --2- 172.21.15.16:0/736306413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd30070 0x564b8e0d5310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009fd0 tx=0x7f3efc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4be1a9d76925b33a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.384+0000 7f3f0a441640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ef00b4dd0 0x7f3ef00b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.384+0000 7f3eeffff640 1 -- 172.21.15.16:0/736306413 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc01e070 con 0x564b8dd30070 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.384+0000 7f3eeffff640 1 -- 172.21.15.16:0/736306413 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3efc004600 con 0x564b8dd30070 2024-08-28T19:35:18.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.384+0000 7f3eeffff640 1 -- 172.21.15.16:0/736306413 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc019430 con 0x564b8dd30070 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.385+0000 7f3f12419740 1 -- 172.21.15.16:0/736306413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd30070 msgr2=0x564b8e0d5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.385+0000 7f3f12419740 1 --2- 172.21.15.16:0/736306413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd30070 0x564b8e0d5310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009fd0 tx=0x7f3efc0118d0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.385+0000 7f3f0a441640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ef00b4dd0 0x7f3ef00b72e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3efc01b000 tx=0x7f3efc019b60 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.385+0000 7f3f12419740 1 -- 172.21.15.16:0/736306413 shutdown_connections 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.385+0000 7f3f12419740 1 --2- 172.21.15.16:0/736306413 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd0f390 0x564b8e0d5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.385+0000 7f3f12419740 1 --2- 172.21.15.16:0/736306413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd071b0 0x564b8e0d5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.385+0000 7f3f12419740 1 --2- 172.21.15.16:0/736306413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd30070 0x564b8e0d5310 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.385+0000 7f3f12419740 1 -- 172.21.15.16:0/736306413 >> 172.21.15.16:0/736306413 conn(0x564b8dd24460 msgr2=0x564b8e0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.385+0000 7f3f12419740 1 -- 172.21.15.16:0/736306413 shutdown_connections 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.385+0000 7f3f12419740 1 -- 172.21.15.16:0/736306413 wait complete. 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.386+0000 7f3f12419740 1 -- start start 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.386+0000 7f3f12419740 1 Processor -- start 2024-08-28T19:35:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.386+0000 7f3f12419740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd071b0 0x564b8e0e91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.386+0000 7f3f12419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd0f390 0x564b8e0e96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.386+0000 7f3f12419740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd30070 0x564b8e0f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.386+0000 7f3f12419740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564b8de9e700 con 0x564b8dd30070 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.386+0000 7f3f12419740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b8dfc2750 con 0x564b8dd0f390 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.386+0000 7f3f12419740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564b8e0ec220 con 0x564b8dd071b0 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.386+0000 7f3f12419740 10 client.0 _ll_register_callbacks cb 0x564b8dff1e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.387+0000 7f3f0943f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd0f390 0x564b8e0e96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.387+0000 7f3f09c40640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd071b0 0x564b8e0e91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.387+0000 7f3f0943f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd0f390 0x564b8e0e96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47006/0 (socket says 172.21.15.16:47006) 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.387+0000 7f3f0943f640 1 -- 172.21.15.16:0/3593875905 learned_addr learned my addr 172.21.15.16:0/3593875905 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.387+0000 7f3f0a441640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd30070 0x564b8e0f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.387+0000 7f3f0a441640 1 -- 172.21.15.16:0/3593875905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd071b0 msgr2=0x564b8e0e91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.387+0000 7f3f0a441640 1 --2- 172.21.15.16:0/3593875905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd071b0 0x564b8e0e91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.387+0000 7f3f0a441640 1 -- 172.21.15.16:0/3593875905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd0f390 msgr2=0x564b8e0e96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.387+0000 7f3f0a441640 1 --2- 172.21.15.16:0/3593875905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd0f390 0x564b8e0e96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.387+0000 7f3f0a441640 1 -- 172.21.15.16:0/3593875905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b8dffb900 con 0x564b8dd30070 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.388+0000 7f3f0a441640 1 --2- 172.21.15.16:0/3593875905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd30070 0x564b8e0f55e0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f3efc01c040 tx=0x7f3efc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6565a1e454e8a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.388+0000 7f3eed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.388+0000 7f3eed7fa640 1 -- 172.21.15.16:0/3593875905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc01e040 con 0x564b8dd30070 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.388+0000 7f3f12419740 1 -- 172.21.15.16:0/3593875905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3efc004600 con 0x564b8dd30070 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.388+0000 7f3eed7fa640 1 -- 172.21.15.16:0/3593875905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f3efc0047b0 con 0x564b8dd30070 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.388+0000 7f3eed7fa640 1 -- 172.21.15.16:0/3593875905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f3efc03e050 con 0x564b8dd30070 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.388+0000 7f3f12419740 10 client.19005 fetch_fsmap learned FSMap version 75 2024-08-28T19:35:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.388+0000 7f3f12419740 1 -- 172.21.15.16:0/3593875905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x564b8e0f8020 con 0x564b8dd30070 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.389+0000 7f3eed7fa640 1 -- 172.21.15.16:0/3593875905 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3efc004600 con 0x564b8dd30070 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.389+0000 7f3eed7fa640 1 -- 172.21.15.16:0/3593875905 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 75) ==== 8434+0+0 (secure 0 0 0) 0x7f3efc00e950 con 0x564b8dd30070 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.389+0000 7f3f12419740 10 client.19005 fetch_fsmap finished waiting for FSMap version 75 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.389+0000 7f3f12419740 10 client.19005 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.389+0000 7f3f12419740 20 client.19005 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.389+0000 7f3f12419740 1 --2- 172.21.15.16:0/3593875905 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564b8dcee9f0 0x564b8e0fb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.389+0000 7f3f12419740 4 client.19005 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.389+0000 7f3f12419740 1 -- 172.21.15.16:0/3593875905 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b8e04e5d0 con 0x564b8dcee9f0 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.389+0000 7f3f09c40640 1 --2- 172.21.15.16:0/3593875905 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564b8dcee9f0 0x564b8e0fb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.390+0000 7f3eed7fa640 10 client.19005 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.390+0000 7f3f09c40640 1 --2- 172.21.15.16:0/3593875905 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564b8dcee9f0 0x564b8e0fb480 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=25dbf96e023a0dee server_cookie=ad2dd6f60442c3f0 in_seq=0 out_seq=0 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3eed7fa640 1 -- 172.21.15.16:0/3593875905 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564b8e04e5d0 con 0x564b8dcee9f0 2024-08-28T19:35:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3eed7fa640 10 client.19005 handle_command_reply: tid=0 2024-08-28T19:35:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3f12419740 1 client.19005 shutdown 2024-08-28T19:35:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3f12419740 1 -- 172.21.15.16:0/3593875905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd30070 msgr2=0x564b8e0f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3f12419740 1 --2- 172.21.15.16:0/3593875905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd30070 0x564b8e0f55e0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f3efc01c040 tx=0x7f3efc019000 comp rx=0 tx=0).stop 2024-08-28T19:35:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3f12419740 1 -- 172.21.15.16:0/3593875905 shutdown_connections 2024-08-28T19:35:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3f12419740 1 --2- 172.21.15.16:0/3593875905 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564b8dcee9f0 0x564b8e0fb480 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3eed7fa640 0 client.19005 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:35:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3f12419740 1 --2- 172.21.15.16:0/3593875905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd30070 0x564b8e0f55e0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3f12419740 1 --2- 172.21.15.16:0/3593875905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd0f390 0x564b8e0e96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3f12419740 1 --2- 172.21.15.16:0/3593875905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd071b0 0x564b8e0e91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3f12419740 1 -- 172.21.15.16:0/3593875905 >> 172.21.15.16:0/3593875905 conn(0x564b8dd24460 msgr2=0x564b8e0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3f12419740 1 -- 172.21.15.16:0/3593875905 shutdown_connections 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.391+0000 7f3f12419740 1 -- 172.21.15.16:0/3593875905 wait complete. 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.392+0000 7f3f12419740 20 client.19005 trim_cache size 0 max 16384 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.415+0000 7f3f12419740 1 Processor -- start 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.415+0000 7f3f12419740 1 -- start start 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.415+0000 7f3f12419740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd24460 0x564b8e0e9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.415+0000 7f3f12419740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dcee9f0 0x564b8e0f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.415+0000 7f3f12419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 0x564b8e0d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.415+0000 7f3f12419740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee800ffb0 con 0x564b8dd24460 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.415+0000 7f3f12419740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee8011d30 con 0x564b8dd071b0 2024-08-28T19:35:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.415+0000 7f3f12419740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee8010710 con 0x564b8dcee9f0 2024-08-28T19:35:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.416+0000 7f3f0a441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 0x564b8e0d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.416+0000 7f3f0a441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 0x564b8e0d2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47020/0 (socket says 172.21.15.16:47020) 2024-08-28T19:35:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.416+0000 7f3f0a441640 1 -- 172.21.15.16:0/3993623070 learned_addr learned my addr 172.21.15.16:0/3993623070 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.416+0000 7f3f09c40640 1 --2- 172.21.15.16:0/3993623070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd24460 0x564b8e0e9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.416+0000 7f3f0943f640 1 --2- 172.21.15.16:0/3993623070 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dcee9f0 0x564b8e0f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.416+0000 7f3f0a441640 1 -- 172.21.15.16:0/3993623070 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dcee9f0 msgr2=0x564b8e0f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.416+0000 7f3f0a441640 1 --2- 172.21.15.16:0/3993623070 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dcee9f0 0x564b8e0f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.416+0000 7f3f0a441640 1 -- 172.21.15.16:0/3993623070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd24460 msgr2=0x564b8e0e9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.416+0000 7f3f0a441640 1 --2- 172.21.15.16:0/3993623070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd24460 0x564b8e0e9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.416+0000 7f3f0a441640 1 -- 172.21.15.16:0/3993623070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b8e0d2910 con 0x564b8dd071b0 2024-08-28T19:35:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.417+0000 7f3f0a441640 1 --2- 172.21.15.16:0/3993623070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 0x564b8e0d2bf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3efc002230 tx=0x7f3efc003f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95806127e8d5f275 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.417+0000 7f3eef7fe640 1 -- 172.21.15.16:0/3993623070 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc00e8d0 con 0x564b8dd071b0 2024-08-28T19:35:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.417+0000 7f3eef7fe640 1 -- 172.21.15.16:0/3993623070 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3efc035070 con 0x564b8dd071b0 2024-08-28T19:35:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.417+0000 7f3eef7fe640 1 -- 172.21.15.16:0/3993623070 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc004410 con 0x564b8dd071b0 2024-08-28T19:35:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.418+0000 7f3f12419740 1 -- 172.21.15.16:0/3993623070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 msgr2=0x564b8e0d2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.418+0000 7f3f12419740 1 --2- 172.21.15.16:0/3993623070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 0x564b8e0d2bf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3efc002230 tx=0x7f3efc003f30 comp rx=0 tx=0).stop 2024-08-28T19:35:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.419+0000 7f3f12419740 1 -- 172.21.15.16:0/3993623070 shutdown_connections 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.419+0000 7f3f12419740 1 --2- 172.21.15.16:0/3993623070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 0x564b8e0d2bf0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.419+0000 7f3f12419740 1 --2- 172.21.15.16:0/3993623070 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dcee9f0 0x564b8e0f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.419+0000 7f3f12419740 1 --2- 172.21.15.16:0/3993623070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dd24460 0x564b8e0e9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.419+0000 7f3f12419740 1 -- 172.21.15.16:0/3993623070 >> 172.21.15.16:0/3993623070 conn(0x564b8e0fb7f0 msgr2=0x564b8e0e8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.419+0000 7f3f12419740 1 -- 172.21.15.16:0/3993623070 shutdown_connections 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.419+0000 7f3f12419740 1 -- 172.21.15.16:0/3993623070 wait complete. 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.420+0000 7f3f12419740 1 -- start start 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.420+0000 7f3f12419740 1 Processor -- start 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f12419740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dcee9f0 0x564b8e0f1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f12419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 0x564b8e0f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f12419740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd24460 0x564b8e0ec500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f12419740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee8011600 con 0x564b8dcee9f0 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f12419740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee8010e70 con 0x564b8dd071b0 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f12419740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee800dfb0 con 0x564b8dd24460 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f12419740 10 client.0 _ll_register_callbacks cb 0x7f3efc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f0a441640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd24460 0x564b8e0ec500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f09c40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dcee9f0 0x564b8e0f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f09c40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dcee9f0 0x564b8e0f1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37166/0 (socket says 172.21.15.16:37166) 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f09c40640 1 -- 172.21.15.16:0/1441185709 learned_addr learned my addr 172.21.15.16:0/1441185709 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f0a441640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd24460 0x564b8e0ec500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56724/0 (socket says 172.21.15.16:56724) 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.421+0000 7f3f0943f640 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 0x564b8e0f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.422+0000 7f3f0a441640 1 -- 172.21.15.16:0/1441185709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 msgr2=0x564b8e0f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.422+0000 7f3f0a441640 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 0x564b8e0f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.422+0000 7f3f0a441640 1 -- 172.21.15.16:0/1441185709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dcee9f0 msgr2=0x564b8e0f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.422+0000 7f3f0a441640 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dcee9f0 0x564b8e0f1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.422+0000 7f3f0a441640 1 -- 172.21.15.16:0/1441185709 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b8e0eca40 con 0x564b8dd24460 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.422+0000 7f3eed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.422+0000 7f3f0a441640 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd24460 0x564b8e0ec500 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3efc011f50 tx=0x7f3efc00eb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8cc0bc3f504111d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.423+0000 7f3eed7fa640 1 -- 172.21.15.16:0/1441185709 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc01e070 con 0x564b8dd24460 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.423+0000 7f3eed7fa640 1 -- 172.21.15.16:0/1441185709 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f3efc00fa40 con 0x564b8dd24460 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.423+0000 7f3f12419740 1 -- 172.21.15.16:0/1441185709 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3efc035070 con 0x564b8dd24460 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.423+0000 7f3eed7fa640 1 -- 172.21.15.16:0/1441185709 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f3efc00fc40 con 0x564b8dd24460 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.423+0000 7f3f12419740 10 client.28096 fetch_fsmap learned FSMap version 75 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.423+0000 7f3f12419740 1 -- 172.21.15.16:0/1441185709 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x564b8e1664a0 con 0x564b8dd24460 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.424+0000 7f3eed7fa640 1 -- 172.21.15.16:0/1441185709 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3efc035070 con 0x564b8dd24460 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.424+0000 7f3eed7fa640 1 -- 172.21.15.16:0/1441185709 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 75) ==== 8434+0+0 (secure 0 0 0) 0x7f3efc057480 con 0x564b8dd24460 2024-08-28T19:35:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.424+0000 7f3f12419740 10 client.28096 fetch_fsmap finished waiting for FSMap version 75 2024-08-28T19:35:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.424+0000 7f3f12419740 10 client.28096 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:35:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.424+0000 7f3f12419740 20 client.28096 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.424+0000 7f3f12419740 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564b8dd0f390 0x564b8e168c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.424+0000 7f3f12419740 4 client.28096 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:35:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.424+0000 7f3f12419740 1 -- 172.21.15.16:0/1441185709 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564b8e04a070 con 0x564b8dd0f390 2024-08-28T19:35:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.425+0000 7f3f09c40640 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564b8dd0f390 0x564b8e168c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:18.470 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:35:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.426+0000 7f3f09c40640 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564b8dd0f390 0x564b8e168c00 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=73726236675405eb server_cookie=47dc148864211944 in_seq=0 out_seq=0 2024-08-28T19:35:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.426+0000 7f3eed7fa640 10 client.28096 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:35:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.426+0000 7f3eed7fa640 1 -- 172.21.15.16:0/1441185709 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564b8e04a070 con 0x564b8dd0f390 2024-08-28T19:35:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.426+0000 7f3eed7fa640 10 client.28096 handle_command_reply: tid=0 2024-08-28T19:35:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.426+0000 7f3f12419740 1 client.28096 shutdown 2024-08-28T19:35:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.426+0000 7f3f12419740 1 -- 172.21.15.16:0/1441185709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd24460 msgr2=0x564b8e0ec500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.426+0000 7f3f12419740 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd24460 0x564b8e0ec500 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3efc011f50 tx=0x7f3efc00eb60 comp rx=0 tx=0).stop 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.427+0000 7f3f12419740 1 -- 172.21.15.16:0/1441185709 shutdown_connections 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.427+0000 7f3f12419740 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564b8dd0f390 0x564b8e168c00 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.427+0000 7f3f12419740 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564b8dd24460 0x564b8e0ec500 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.427+0000 7f3f12419740 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dd071b0 0x564b8e0f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.427+0000 7f3f12419740 1 --2- 172.21.15.16:0/1441185709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564b8dcee9f0 0x564b8e0f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.427+0000 7f3f12419740 1 -- 172.21.15.16:0/1441185709 >> 172.21.15.16:0/1441185709 conn(0x564b8e0fb7f0 msgr2=0x564b8e0d2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.427+0000 7f3eed7fa640 0 client.28096 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.427+0000 7f3f12419740 1 -- 172.21.15.16:0/1441185709 shutdown_connections 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.427+0000 7f3f12419740 1 -- 172.21.15.16:0/1441185709 wait complete. 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.427+0000 7f3f12419740 20 client.28096 trim_cache size 0 max 16384 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 -- 172.21.15.16:0/570615096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ef00b4dd0 msgr2=0x7f3ef00b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ef00b4dd0 0x7f3ef00b72e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3efc01b000 tx=0x7f3efc019b60 comp rx=0 tx=0).stop 2024-08-28T19:35:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 -- 172.21.15.16:0/570615096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f04102810 msgr2=0x7f3f041991e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f04102810 0x7f3f041991e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3f00035040 tx=0x7f3f00004270 comp rx=0 tx=0).stop 2024-08-28T19:35:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 -- 172.21.15.16:0/570615096 shutdown_connections 2024-08-28T19:35:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f04108aa0 0x7f3f0419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f04108150 0x7f3f04199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3ef00b4dd0 0x7f3ef00b72e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 --2- 172.21.15.16:0/570615096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f04102810 0x7f3f041991e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 -- 172.21.15.16:0/570615096 >> 172.21.15.16:0/570615096 conn(0x7f3f040794d0 msgr2=0x7f3f04101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 -- 172.21.15.16:0/570615096 shutdown_connections 2024-08-28T19:35:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:18.428+0000 7f3f0bea4640 1 -- 172.21.15.16:0/570615096 wait complete. 2024-08-28T19:35:18.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 12731 16.16 MB/sec warmup 54 sec latency 4811.513 ms 2024-08-28T19:35:19.237 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:35:19.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 13241 16.12 MB/sec warmup 55 sec latency 4981.256 ms 2024-08-28T19:35:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:19 smithi016 ceph-mon[18556]: pgmap v783: 129 pgs: 129 active+clean; 686 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.0 MiB/s wr, 248 op/s 2024-08-28T19:35:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:19 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:19.584 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:35:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:19 smithi115 ceph-mon[22184]: pgmap v783: 129 pgs: 129 active+clean; 686 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.0 MiB/s wr, 248 op/s 2024-08-28T19:35:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:19 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:19 smithi028 ceph-mon[21769]: pgmap v783: 129 pgs: 129 active+clean; 686 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.0 MiB/s wr, 248 op/s 2024-08-28T19:35:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:19 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:20.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.245+0000 7f7dc95c5640 1 -- 172.21.15.16:0/340025504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc4102540 msgr2=0x7f7dc4102950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.245+0000 7f7dc95c5640 1 --2- 172.21.15.16:0/340025504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc4102540 0x7f7dc4102950 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009890 tx=0x7f7dbc02a360 comp rx=0 tx=0).stop 2024-08-28T19:35:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.245+0000 7f7dc95c5640 1 -- 172.21.15.16:0/340025504 shutdown_connections 2024-08-28T19:35:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.245+0000 7f7dc95c5640 1 --2- 172.21.15.16:0/340025504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc4107980 0x7f7dc4107e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.245+0000 7f7dc95c5640 1 --2- 172.21.15.16:0/340025504 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7dc4102e90 0x7f7dc41073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.245+0000 7f7dc95c5640 1 --2- 172.21.15.16:0/340025504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc4102540 0x7f7dc4102950 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.245+0000 7f7dc95c5640 1 -- 172.21.15.16:0/340025504 >> 172.21.15.16:0/340025504 conn(0x7f7dc40794d0 msgr2=0x7f7dc40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.245+0000 7f7dc95c5640 1 -- 172.21.15.16:0/340025504 shutdown_connections 2024-08-28T19:35:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.245+0000 7f7dc95c5640 1 -- 172.21.15.16:0/340025504 wait complete. 2024-08-28T19:35:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.246+0000 7f7dc95c5640 1 Processor -- start 2024-08-28T19:35:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.246+0000 7f7dc95c5640 1 -- start start 2024-08-28T19:35:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc95c5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7dc4102540 0x7f7dc41a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc95c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc4102e90 0x7f7dc41a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc95c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc4107980 0x7f7dc41a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc95c5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4119e20 con 0x7f7dc4102e90 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc95c5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4119ca0 con 0x7f7dc4107980 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc95c5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4119fa0 con 0x7f7dc4102540 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc4102e90 0x7f7dc41a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc2ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7dc4102540 0x7f7dc41a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc4102e90 0x7f7dc41a0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37190/0 (socket says 172.21.15.16:37190) 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc27fc640 1 -- 172.21.15.16:0/1988514141 learned_addr learned my addr 172.21.15.16:0/1988514141 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc37fe640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc4107980 0x7f7dc41a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc2ffd640 1 -- 172.21.15.16:0/1988514141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc4107980 msgr2=0x7f7dc41a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc4107980 0x7f7dc41a60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc2ffd640 1 -- 172.21.15.16:0/1988514141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc4102e90 msgr2=0x7f7dc41a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc4102e90 0x7f7dc41a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.247+0000 7f7dc2ffd640 1 -- 172.21.15.16:0/1988514141 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc41a6820 con 0x7f7dc4102540 2024-08-28T19:35:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.248+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7dc4102540 0x7f7dc41a08b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc02a880 tx=0x7f7dbc032d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.248+0000 7f7dbbfff640 1 -- 172.21.15.16:0/1988514141 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc040020 con 0x7f7dc4102540 2024-08-28T19:35:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.248+0000 7f7dbbfff640 1 -- 172.21.15.16:0/1988514141 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dbc030d20 con 0x7f7dc4102540 2024-08-28T19:35:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.248+0000 7f7dbbfff640 1 -- 172.21.15.16:0/1988514141 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc037040 con 0x7f7dc4102540 2024-08-28T19:35:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.248+0000 7f7dc95c5640 1 -- 172.21.15.16:0/1988514141 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dc41a6ab0 con 0x7f7dc4102540 2024-08-28T19:35:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.248+0000 7f7dc95c5640 1 -- 172.21.15.16:0/1988514141 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dc41ad270 con 0x7f7dc4102540 2024-08-28T19:35:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.252+0000 7f7dbbfff640 1 -- 172.21.15.16:0/1988514141 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7dbc043020 con 0x7f7dc4102540 2024-08-28T19:35:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.252+0000 7f7dcfb3a740 1 Processor -- start 2024-08-28T19:35:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.252+0000 7f7dcfb3a740 1 -- start start 2024-08-28T19:35:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.253+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561299490070 0x561299835310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.253+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612994671b0 0x561299835850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.253+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56129946f390 0x561299835d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.253+0000 7f7dcfb3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5612997ac680 con 0x56129946f390 2024-08-28T19:35:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.253+0000 7f7dcfb3a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56129979bc10 con 0x5612994671b0 2024-08-28T19:35:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.253+0000 7f7dcfb3a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56129979fbc0 con 0x561299490070 2024-08-28T19:35:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.254+0000 7f7dc37fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561299490070 0x561299835310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.254+0000 7f7dc2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612994671b0 0x561299835850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.254+0000 7f7dc37fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561299490070 0x561299835310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56762/0 (socket says 172.21.15.16:56762) 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.254+0000 7f7dc37fe640 1 -- 172.21.15.16:0/3078286431 learned_addr learned my addr 172.21.15.16:0/3078286431 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.254+0000 7f7dbbfff640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7da40b4ca0 0x7f7da40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.254+0000 7f7dc2ffd640 1 -- 172.21.15.16:0/3078286431 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561299490070 msgr2=0x561299835310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.254+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/3078286431 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561299490070 0x561299835310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.254+0000 7f7dc2ffd640 1 -- 172.21.15.16:0/3078286431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56129946f390 msgr2=0x561299835d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.254+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/3078286431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56129946f390 0x561299835d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.254+0000 7f7dc2ffd640 1 -- 172.21.15.16:0/3078286431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612994fb650 con 0x5612994671b0 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.255+0000 7f7dbbfff640 1 -- 172.21.15.16:0/1988514141 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f7dbc1054b0 con 0x7f7dc4102540 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.255+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/3078286431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612994671b0 0x561299835850 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc1028d0 tx=0x7f7dbc04b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ccb2fcfcc34a7e8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.255+0000 7f7dc37fe640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7da40b4ca0 0x7f7da40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.255+0000 7f7db97fa640 1 -- 172.21.15.16:0/3078286431 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc0cb640 con 0x5612994671b0 2024-08-28T19:35:20.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.256+0000 7f7db97fa640 1 -- 172.21.15.16:0/3078286431 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dbc0b9100 con 0x5612994671b0 2024-08-28T19:35:20.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.256+0000 7f7db97fa640 1 -- 172.21.15.16:0/3078286431 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc0cafe0 con 0x5612994671b0 2024-08-28T19:35:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.257+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/3078286431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612994671b0 msgr2=0x561299835850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.257+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/3078286431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612994671b0 0x561299835850 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc1028d0 tx=0x7f7dbc04b000 comp rx=0 tx=0).stop 2024-08-28T19:35:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.257+0000 7f7dc37fe640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7da40b4ca0 0x7f7da40b71b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7db40058c0 tx=0x7f7db40046c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.257+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/3078286431 shutdown_connections 2024-08-28T19:35:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.257+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/3078286431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56129946f390 0x561299835d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.257+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/3078286431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612994671b0 0x561299835850 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.257+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/3078286431 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561299490070 0x561299835310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.257+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/3078286431 >> 172.21.15.16:0/3078286431 conn(0x561299484460 msgr2=0x561299833200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.257+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/3078286431 shutdown_connections 2024-08-28T19:35:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.257+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/3078286431 wait complete. 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.258+0000 7f7dcfb3a740 1 -- start start 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.258+0000 7f7dcfb3a740 1 Processor -- start 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612994671b0 0x561299840a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 0x561299840f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x56129984ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dcfb3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5612995fe700 con 0x5612994671b0 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dcfb3a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561299722750 con 0x561299490070 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dcfb3a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5612998438b0 con 0x56129946f390 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dc27fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 0x561299840f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dc27fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 0x561299840f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56772/0 (socket says 172.21.15.16:56772) 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dc27fc640 1 -- 172.21.15.16:0/4183943897 learned_addr learned my addr 172.21.15.16:0/4183943897 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612994671b0 0x561299840a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dcfb3a740 10 client.0 _ll_register_callbacks cb 0x561299751e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dc27fc640 1 -- 172.21.15.16:0/4183943897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 msgr2=0x56129984ca90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dc37fe640 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x56129984ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.259+0000 7f7dc27fc640 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x56129984ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.260+0000 7f7dc27fc640 1 -- 172.21.15.16:0/4183943897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612994671b0 msgr2=0x561299840a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.260+0000 7f7dc27fc640 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612994671b0 0x561299840a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.260+0000 7f7dc27fc640 1 -- 172.21.15.16:0/4183943897 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56129975b900 con 0x56129946f390 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.260+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612994671b0 0x561299840a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.260+0000 7f7dc27fc640 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 0x561299840f50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7dac005f40 tx=0x7f7dac00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c5a64bc93af4b60 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.260+0000 7f7db27fc640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.260+0000 7f7db27fc640 1 -- 172.21.15.16:0/4183943897 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dac019420 con 0x56129946f390 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.260+0000 7f7db27fc640 1 -- 172.21.15.16:0/4183943897 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f7dac0195c0 con 0x56129946f390 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.260+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/4183943897 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dbc0b9100 con 0x56129946f390 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.261+0000 7f7db27fc640 1 -- 172.21.15.16:0/4183943897 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f7dac024050 con 0x56129946f390 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.261+0000 7f7dcfb3a740 10 client.28108 fetch_fsmap learned FSMap version 75 2024-08-28T19:35:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.261+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/4183943897 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x56129984f550 con 0x56129946f390 2024-08-28T19:35:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.261+0000 7f7db27fc640 1 -- 172.21.15.16:0/4183943897 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dac019c70 con 0x56129946f390 2024-08-28T19:35:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.262+0000 7f7db27fc640 1 -- 172.21.15.16:0/4183943897 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 75) ==== 8434+0+0 (secure 0 0 0) 0x7f7dac018b70 con 0x56129946f390 2024-08-28T19:35:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.262+0000 7f7dcfb3a740 10 client.28108 fetch_fsmap finished waiting for FSMap version 75 2024-08-28T19:35:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.262+0000 7f7dcfb3a740 10 client.28108 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:35:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.262+0000 7f7dcfb3a740 20 client.28108 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.262+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56129944e9f0 0x561299852ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.262+0000 7f7dcfb3a740 4 client.28108 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:35:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.262+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/4183943897 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7dbc0b9100 con 0x56129944e9f0 2024-08-28T19:35:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.263+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56129944e9f0 0x561299852ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.265+0000 7f7db27fc640 10 client.28108 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:35:20.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.265+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56129944e9f0 0x561299852ba0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=21982cab94d951d5 server_cookie=7e5e5e7951a1457d in_seq=0 out_seq=0 2024-08-28T19:35:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7db27fc640 1 -- 172.21.15.16:0/4183943897 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f7dbc0b9100 con 0x56129944e9f0 2024-08-28T19:35:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7db27fc640 10 client.28108 handle_command_reply: tid=0 2024-08-28T19:35:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7dcfb3a740 1 client.28108 shutdown 2024-08-28T19:35:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/4183943897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 msgr2=0x561299840f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 0x561299840f50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7dac005f40 tx=0x7f7dac00f790 comp rx=0 tx=0).stop 2024-08-28T19:35:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/4183943897 shutdown_connections 2024-08-28T19:35:20.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56129944e9f0 0x561299852ba0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x56129984ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 0x561299840f50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/4183943897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612994671b0 0x561299840a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/4183943897 >> 172.21.15.16:0/4183943897 conn(0x561299484460 msgr2=0x561299836340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:20.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.268+0000 7f7db27fc640 0 client.28108 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:35:20.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.269+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/4183943897 shutdown_connections 2024-08-28T19:35:20.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.269+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/4183943897 wait complete. 2024-08-28T19:35:20.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.269+0000 7f7dcfb3a740 20 client.28108 trim_cache size 0 max 16384 2024-08-28T19:35:20.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.292+0000 7f7dcfb3a740 1 Processor -- start 2024-08-28T19:35:20.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.292+0000 7f7dcfb3a740 1 -- start start 2024-08-28T19:35:20.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.292+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5612994671b0 0x561299840ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.292+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56129946f390 0x56129983dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.292+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x56129983e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.293+0000 7f7dcfb3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc41b7370 con 0x56129946f390 2024-08-28T19:35:20.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.293+0000 7f7dc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x56129983e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.293+0000 7f7dc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x56129983e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47060/0 (socket says 172.21.15.16:47060) 2024-08-28T19:35:20.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.293+0000 7f7dc37fe640 1 -- 172.21.15.16:0/57916287 learned_addr learned my addr 172.21.15.16:0/57916287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:20.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.293+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/57916287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc41b90f0 con 0x561299490070 2024-08-28T19:35:20.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.293+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/57916287 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc41b7ad0 con 0x5612994671b0 2024-08-28T19:35:20.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.293+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/57916287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5612994671b0 0x561299840ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.293+0000 7f7dc27fc640 1 --2- 172.21.15.16:0/57916287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56129946f390 0x56129983dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.294+0000 7f7dc2ffd640 1 -- 172.21.15.16:0/57916287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 msgr2=0x56129983e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:20.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.294+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/57916287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x56129983e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.294+0000 7f7dc2ffd640 1 -- 172.21.15.16:0/57916287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56129946f390 msgr2=0x56129983dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:20.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.294+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/57916287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56129946f390 0x56129983dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.294+0000 7f7dc2ffd640 1 -- 172.21.15.16:0/57916287 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612998435e0 con 0x5612994671b0 2024-08-28T19:35:20.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.294+0000 7f7dc27fc640 1 --2- 172.21.15.16:0/57916287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56129946f390 0x56129983dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:20.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.294+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/57916287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5612994671b0 0x561299840ed0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc0c78f0 tx=0x7f7dbc10b110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeab3e831e80e1d1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:20.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.295+0000 7f7db8ff9640 1 -- 172.21.15.16:0/57916287 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc0cafe0 con 0x5612994671b0 2024-08-28T19:35:20.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.296+0000 7f7db8ff9640 1 -- 172.21.15.16:0/57916287 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dbc0cb380 con 0x5612994671b0 2024-08-28T19:35:20.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.296+0000 7f7db8ff9640 1 -- 172.21.15.16:0/57916287 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc0cb1c0 con 0x5612994671b0 2024-08-28T19:35:20.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.297+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/57916287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5612994671b0 msgr2=0x561299840ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:20.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.297+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/57916287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5612994671b0 0x561299840ed0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc0c78f0 tx=0x7f7dbc10b110 comp rx=0 tx=0).stop 2024-08-28T19:35:20.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.297+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/57916287 shutdown_connections 2024-08-28T19:35:20.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.297+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/57916287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x56129983e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.297+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/57916287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56129946f390 0x56129983dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.297+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/57916287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5612994671b0 0x561299840ed0 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc0c78f0 tx=0x7f7dbc10b110 comp rx=0 tx=0).stop 2024-08-28T19:35:20.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.297+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/57916287 >> 172.21.15.16:0/57916287 conn(0x561299484460 msgr2=0x5612998407c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:20.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.298+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/57916287 shutdown_connections 2024-08-28T19:35:20.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.298+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/57916287 wait complete. 2024-08-28T19:35:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.298+0000 7f7dcfb3a740 1 -- start start 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.298+0000 7f7dcfb3a740 1 Processor -- start 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 0x561299845220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x561299845760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dcfb3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561299423da0 0x5612998bc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dcfb3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc41b89c0 con 0x561299423da0 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dcfb3a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc41b8230 con 0x561299490070 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dcfb3a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc41b5370 con 0x56129946f390 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dcfb3a740 10 client.0 _ll_register_callbacks cb 0x7f7dac011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dc37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561299423da0 0x5612998bc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dc37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561299423da0 0x5612998bc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37224/0 (socket says 172.21.15.16:37224) 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dc37fe640 1 -- 172.21.15.16:0/2704240167 learned_addr learned my addr 172.21.15.16:0/2704240167 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dc27fc640 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x561299845760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.299+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 0x561299845220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.300+0000 7f7dc27fc640 1 -- 172.21.15.16:0/2704240167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 msgr2=0x561299845220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:20.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.300+0000 7f7dc27fc640 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 0x561299845220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.300+0000 7f7dc27fc640 1 -- 172.21.15.16:0/2704240167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561299423da0 msgr2=0x5612998bc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:20.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.300+0000 7f7dc27fc640 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561299423da0 0x5612998bc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.300+0000 7f7dc27fc640 1 -- 172.21.15.16:0/2704240167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612998bcaa0 con 0x561299490070 2024-08-28T19:35:20.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.301+0000 7f7dc27fc640 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x561299845760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7dac011860 tx=0x7f7dac00e7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90f064ace903bfa server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:20.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.302+0000 7f7db27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:35:20.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.302+0000 7f7db27fc640 1 -- 172.21.15.16:0/2704240167 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dac003970 con 0x561299490070 2024-08-28T19:35:20.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.302+0000 7f7db27fc640 1 -- 172.21.15.16:0/2704240167 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f7dac003b10 con 0x561299490070 2024-08-28T19:35:20.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.302+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/2704240167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dbc0cb380 con 0x561299490070 2024-08-28T19:35:20.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.302+0000 7f7db27fc640 1 -- 172.21.15.16:0/2704240167 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f7dac003d10 con 0x561299490070 2024-08-28T19:35:20.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.303+0000 7f7dcfb3a740 10 client.28457 fetch_fsmap learned FSMap version 75 2024-08-28T19:35:20.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.303+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/2704240167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x5612998bdd20 con 0x561299490070 2024-08-28T19:35:20.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.303+0000 7f7db27fc640 1 -- 172.21.15.16:0/2704240167 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dbc0cb380 con 0x561299490070 2024-08-28T19:35:20.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.303+0000 7f7db27fc640 1 -- 172.21.15.16:0/2704240167 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 75) ==== 8434+0+0 (secure 0 0 0) 0x7f7dac0183d0 con 0x561299490070 2024-08-28T19:35:20.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.303+0000 7f7dcfb3a740 10 client.28457 fetch_fsmap finished waiting for FSMap version 75 2024-08-28T19:35:20.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.303+0000 7f7dcfb3a740 10 client.28457 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:35:20.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.303+0000 7f7dcfb3a740 20 client.28457 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:20.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.303+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5612998be480 0x5612998c08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:20.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.303+0000 7f7dcfb3a740 4 client.28457 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:35:20.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.304+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/2704240167 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612997ae5d0 con 0x5612998be480 2024-08-28T19:35:20.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.304+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5612998be480 0x5612998c08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:20.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.305+0000 7f7db27fc640 10 client.28457 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:35:20.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.305+0000 7f7dc2ffd640 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5612998be480 0x5612998c08a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=627e55469e296727 server_cookie=af99838450af931d in_seq=0 out_seq=0 2024-08-28T19:35:20.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.308+0000 7f7db27fc640 1 -- 172.21.15.16:0/2704240167 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612997ae5d0 con 0x5612998be480 2024-08-28T19:35:20.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.308+0000 7f7db27fc640 10 client.28457 handle_command_reply: tid=0 2024-08-28T19:35:20.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.308+0000 7f7dcfb3a740 1 client.28457 shutdown 2024-08-28T19:35:20.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.308+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/2704240167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 msgr2=0x561299845760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:20.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.308+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x561299845760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7dac011860 tx=0x7f7dac00e7a0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.310+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/2704240167 shutdown_connections 2024-08-28T19:35:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.310+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x5612998be480 0x5612998c08a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.310+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561299423da0 0x5612998bc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.310+0000 7f7db27fc640 0 client.28457 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:35:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.310+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561299490070 0x561299845760 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.310+0000 7f7dcfb3a740 1 --2- 172.21.15.16:0/2704240167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56129946f390 0x561299845220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.310+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/2704240167 >> 172.21.15.16:0/2704240167 conn(0x561299484460 msgr2=0x561299841ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.310+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/2704240167 shutdown_connections 2024-08-28T19:35:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.311+0000 7f7dcfb3a740 1 -- 172.21.15.16:0/2704240167 wait complete. 2024-08-28T19:35:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.311+0000 7f7dcfb3a740 20 client.28457 trim_cache size 0 max 16384 2024-08-28T19:35:20.313 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:35:20.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.313+0000 7f7dc95c5640 1 -- 172.21.15.16:0/1988514141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7da40b4ca0 msgr2=0x7f7da40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:20.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.313+0000 7f7dc95c5640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7da40b4ca0 0x7f7da40b71b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7db40058c0 tx=0x7f7db40046c0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.313+0000 7f7dc95c5640 1 -- 172.21.15.16:0/1988514141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7dc4102540 msgr2=0x7f7dc41a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.313+0000 7f7dc95c5640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7dc4102540 0x7f7dc41a08b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc02a880 tx=0x7f7dbc032d30 comp rx=0 tx=0).stop 2024-08-28T19:35:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.313+0000 7f7dc95c5640 1 -- 172.21.15.16:0/1988514141 shutdown_connections 2024-08-28T19:35:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.313+0000 7f7dc95c5640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7da40b4ca0 0x7f7da40b71b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.313+0000 7f7dc95c5640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc4107980 0x7f7dc41a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.313+0000 7f7dc95c5640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc4102e90 0x7f7dc41a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.313+0000 7f7dc95c5640 1 --2- 172.21.15.16:0/1988514141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7dc4102540 0x7f7dc41a08b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.313+0000 7f7dc95c5640 1 -- 172.21.15.16:0/1988514141 >> 172.21.15.16:0/1988514141 conn(0x7f7dc40794d0 msgr2=0x7f7dc4105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.314+0000 7f7dc95c5640 1 -- 172.21.15.16:0/1988514141 shutdown_connections 2024-08-28T19:35:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:20.314+0000 7f7dc95c5640 1 -- 172.21.15.16:0/1988514141 wait complete. 2024-08-28T19:35:20.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 14266 16.65 MB/sec warmup 56 sec latency 1337.820 ms 2024-08-28T19:35:21.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:35:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:35:21.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 15414 17.68 MB/sec warmup 57 sec latency 2337.846 ms 2024-08-28T19:35:21.599 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:21 smithi016 ceph-mon[18556]: pgmap v784: 129 pgs: 129 active+clean; 692 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.7 MiB/s wr, 266 op/s 2024-08-28T19:35:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:21 smithi115 ceph-mon[22184]: pgmap v784: 129 pgs: 129 active+clean; 692 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.7 MiB/s wr, 266 op/s 2024-08-28T19:35:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:21 smithi028 ceph-mon[21769]: pgmap v784: 129 pgs: 129 active+clean; 692 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.7 MiB/s wr, 266 op/s 2024-08-28T19:35:21.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.980+0000 7f0848c8b640 1 -- 172.21.15.16:0/3662453033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0844108a90 msgr2=0x7f084410f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:21.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.980+0000 7f0848c8b640 1 --2- 172.21.15.16:0/3662453033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0844108a90 0x7f084410f2c0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f0834009900 tx=0x7f083402a410 comp rx=0 tx=0).stop 2024-08-28T19:35:21.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.980+0000 7f0848c8b640 1 -- 172.21.15.16:0/3662453033 shutdown_connections 2024-08-28T19:35:21.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.980+0000 7f0848c8b640 1 --2- 172.21.15.16:0/3662453033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0844108a90 0x7f084410f2c0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.980+0000 7f0848c8b640 1 --2- 172.21.15.16:0/3662453033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844108140 0x7f0844108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.980+0000 7f0848c8b640 1 --2- 172.21.15.16:0/3662453033 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0844102800 0x7f0844102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.980+0000 7f0848c8b640 1 -- 172.21.15.16:0/3662453033 >> 172.21.15.16:0/3662453033 conn(0x7f08440794d0 msgr2=0x7f08440798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:21.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.980+0000 7f0848c8b640 1 -- 172.21.15.16:0/3662453033 shutdown_connections 2024-08-28T19:35:21.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.980+0000 7f0848c8b640 1 -- 172.21.15.16:0/3662453033 wait complete. 2024-08-28T19:35:21.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.981+0000 7f0848c8b640 1 Processor -- start 2024-08-28T19:35:21.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.981+0000 7f0848c8b640 1 -- start start 2024-08-28T19:35:21.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f0848c8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0844102800 0x7f084419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:21.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f0848c8b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0844108140 0x7f084419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:21.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f0848c8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844108a90 0x7f084419fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f0848c8b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0844112710 con 0x7f0844102800 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f0848c8b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0844112590 con 0x7f0844108a90 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f0848c8b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0844112890 con 0x7f0844108140 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f084259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0844102800 0x7f084419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f084259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0844102800 0x7f084419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37246/0 (socket says 172.21.15.16:37246) 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f084259c640 1 -- 172.21.15.16:0/1626344557 learned_addr learned my addr 172.21.15.16:0/1626344557 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f0842d9d640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844108a90 0x7f084419fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f0841d9b640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0844108140 0x7f084419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f084259c640 1 -- 172.21.15.16:0/1626344557 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0844108140 msgr2=0x7f084419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f084259c640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0844108140 0x7f084419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f084259c640 1 -- 172.21.15.16:0/1626344557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844108a90 msgr2=0x7f084419fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f084259c640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844108a90 0x7f084419fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.982+0000 7f084259c640 1 -- 172.21.15.16:0/1626344557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08441a0140 con 0x7f0844102800 2024-08-28T19:35:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.983+0000 7f084259c640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0844102800 0x7f084419a1d0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f083c00eb90 tx=0x7f083c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.983+0000 7f083b7fe640 1 -- 172.21.15.16:0/1626344557 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c004d20 con 0x7f0844102800 2024-08-28T19:35:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.983+0000 7f0848c8b640 1 -- 172.21.15.16:0/1626344557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08441a0430 con 0x7f0844102800 2024-08-28T19:35:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.983+0000 7f083b7fe640 1 -- 172.21.15.16:0/1626344557 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f083c00ee90 con 0x7f0844102800 2024-08-28T19:35:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.983+0000 7f083b7fe640 1 -- 172.21.15.16:0/1626344557 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c010670 con 0x7f0844102800 2024-08-28T19:35:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.983+0000 7f0848c8b640 1 -- 172.21.15.16:0/1626344557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08441a6b90 con 0x7f0844102800 2024-08-28T19:35:21.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.986+0000 7f084f200740 1 Processor -- start 2024-08-28T19:35:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.986+0000 7f084f200740 1 -- start start 2024-08-28T19:35:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.986+0000 7f084f200740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e449310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.986+0000 7f084f200740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e449850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.986+0000 7f084f200740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e449d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.986+0000 7f084f200740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a97e3c0680 con 0x55a97e083390 2024-08-28T19:35:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.986+0000 7f084f200740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a97e3afc10 con 0x55a97e0a4070 2024-08-28T19:35:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.986+0000 7f084f200740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a97e3b3bc0 con 0x55a97e07b1b0 2024-08-28T19:35:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f083b7fe640 1 -- 172.21.15.16:0/1626344557 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f083c005020 con 0x7f0844102800 2024-08-28T19:35:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f0842d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e449310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f0842d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e449310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47104/0 (socket says 172.21.15.16:47104) 2024-08-28T19:35:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f0842d9d640 1 -- 172.21.15.16:0/1757119676 learned_addr learned my addr 172.21.15.16:0/1757119676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f0841d9b640 1 --2- 172.21.15.16:0/1757119676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e449d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f084259c640 1 --2- 172.21.15.16:0/1757119676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e449850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f0842d9d640 1 -- 172.21.15.16:0/1757119676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 msgr2=0x55a97e449850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f0842d9d640 1 --2- 172.21.15.16:0/1757119676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e449850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f0842d9d640 1 -- 172.21.15.16:0/1757119676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 msgr2=0x55a97e449d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f0842d9d640 1 --2- 172.21.15.16:0/1757119676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e449d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f0842d9d640 1 -- 172.21.15.16:0/1757119676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a97e10f650 con 0x55a97e0a4070 2024-08-28T19:35:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.987+0000 7f084259c640 1 --2- 172.21.15.16:0/1757119676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e449850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:35:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.988+0000 7f083b7fe640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08240b4ca0 0x7f08240b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.988+0000 7f084259c640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08240b4ca0 0x7f08240b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.988+0000 7f083b7fe640 1 -- 172.21.15.16:0/1626344557 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f083c0e5f80 con 0x7f0844102800 2024-08-28T19:35:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.988+0000 7f084259c640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08240b4ca0 0x7f08240b71b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f083c0e33b0 tx=0x7f083c0eb040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.989+0000 7f0842d9d640 1 --2- 172.21.15.16:0/1757119676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e449310 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0834009900 tx=0x7f0834032b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf96dcdd34a4544e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.989+0000 7f0838ff9640 1 -- 172.21.15.16:0/1757119676 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834040070 con 0x55a97e0a4070 2024-08-28T19:35:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.989+0000 7f0838ff9640 1 -- 172.21.15.16:0/1757119676 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0834004030 con 0x55a97e0a4070 2024-08-28T19:35:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.989+0000 7f0838ff9640 1 -- 172.21.15.16:0/1757119676 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834035040 con 0x55a97e0a4070 2024-08-28T19:35:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.990+0000 7f084f200740 1 -- 172.21.15.16:0/1757119676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 msgr2=0x55a97e449310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.990+0000 7f084f200740 1 --2- 172.21.15.16:0/1757119676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e449310 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0834009900 tx=0x7f0834032b20 comp rx=0 tx=0).stop 2024-08-28T19:35:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.990+0000 7f084f200740 1 -- 172.21.15.16:0/1757119676 shutdown_connections 2024-08-28T19:35:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.990+0000 7f084f200740 1 --2- 172.21.15.16:0/1757119676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e449d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.990+0000 7f084f200740 1 --2- 172.21.15.16:0/1757119676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e449850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.990+0000 7f084f200740 1 --2- 172.21.15.16:0/1757119676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e449310 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.990+0000 7f084f200740 1 -- 172.21.15.16:0/1757119676 >> 172.21.15.16:0/1757119676 conn(0x55a97e098460 msgr2=0x55a97e447200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.991+0000 7f084f200740 1 -- 172.21.15.16:0/1757119676 shutdown_connections 2024-08-28T19:35:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.991+0000 7f084f200740 1 -- 172.21.15.16:0/1757119676 wait complete. 2024-08-28T19:35:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.991+0000 7f084f200740 1 -- start start 2024-08-28T19:35:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.991+0000 7f084f200740 1 Processor -- start 2024-08-28T19:35:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f084f200740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e07b1b0 0x55a97e458e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f084f200740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e083390 0x55a97e459390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f084f200740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e4650c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f084f200740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a97e212700 con 0x55a97e07b1b0 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f0842d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e07b1b0 0x55a97e458e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f084f200740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a97e336750 con 0x55a97e0a4070 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f084f200740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a97e45bd90 con 0x55a97e083390 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f0842d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e07b1b0 0x55a97e458e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37276/0 (socket says 172.21.15.16:37276) 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f0842d9d640 1 -- 172.21.15.16:0/779878390 learned_addr learned my addr 172.21.15.16:0/779878390 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f084f200740 10 client.0 _ll_register_callbacks cb 0x55a97e365e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f0841d9b640 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e083390 0x55a97e459390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f0842d9d640 1 -- 172.21.15.16:0/779878390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e083390 msgr2=0x55a97e459390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f0842d9d640 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e083390 0x55a97e459390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f0842d9d640 1 -- 172.21.15.16:0/779878390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 msgr2=0x55a97e4650c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f0842d9d640 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e4650c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f0842d9d640 1 -- 172.21.15.16:0/779878390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a97e36f900 con 0x55a97e07b1b0 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.992+0000 7f0841d9b640 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e083390 0x55a97e459390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.993+0000 7f0842d9d640 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e07b1b0 0x55a97e458e50 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f0834006010 tx=0x7f0834005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca4399350f291f3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.993+0000 7f0831ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.993+0000 7f0831ffb640 1 -- 172.21.15.16:0/779878390 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834035070 con 0x55a97e07b1b0 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.993+0000 7f0831ffb640 1 -- 172.21.15.16:0/779878390 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f0834005460 con 0x55a97e07b1b0 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.993+0000 7f084f200740 1 -- 172.21.15.16:0/779878390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0834004030 con 0x55a97e07b1b0 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.993+0000 7f0831ffb640 1 -- 172.21.15.16:0/779878390 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f0834058050 con 0x55a97e07b1b0 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.993+0000 7f084f200740 10 client.19038 fetch_fsmap learned FSMap version 75 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.993+0000 7f084f200740 1 -- 172.21.15.16:0/779878390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x55a97e467b40 con 0x55a97e07b1b0 2024-08-28T19:35:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.994+0000 7f0831ffb640 1 -- 172.21.15.16:0/779878390 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0834004030 con 0x55a97e07b1b0 2024-08-28T19:35:21.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.994+0000 7f0831ffb640 1 -- 172.21.15.16:0/779878390 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 75) ==== 8434+0+0 (secure 0 0 0) 0x7f0834004c20 con 0x55a97e07b1b0 2024-08-28T19:35:21.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.994+0000 7f084f200740 10 client.19038 fetch_fsmap finished waiting for FSMap version 75 2024-08-28T19:35:21.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.994+0000 7f084f200740 10 client.19038 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:35:21.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.994+0000 7f084f200740 20 client.19038 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:21.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.994+0000 7f084f200740 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55a97e0629f0 0x55a97e46b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.994+0000 7f084f200740 4 client.19038 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.994+0000 7f084f200740 1 -- 172.21.15.16:0/779878390 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a97e3c25d0 con 0x55a97e0629f0 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.994+0000 7f0841d9b640 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55a97e0629f0 0x55a97e46b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.995+0000 7f0831ffb640 10 client.19038 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.995+0000 7f0841d9b640 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55a97e0629f0 0x55a97e46b220 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ff433676ae507b5f server_cookie=dae4ecbf6b55dee0 in_seq=0 out_seq=0 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f0831ffb640 1 -- 172.21.15.16:0/779878390 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a97e3c25d0 con 0x55a97e0629f0 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f0831ffb640 10 client.19038 handle_command_reply: tid=0 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f084f200740 1 client.19038 shutdown 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f084f200740 1 -- 172.21.15.16:0/779878390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e07b1b0 msgr2=0x55a97e458e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f084f200740 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e07b1b0 0x55a97e458e50 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f0834006010 tx=0x7f0834005eb0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f084f200740 1 -- 172.21.15.16:0/779878390 shutdown_connections 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f084f200740 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55a97e0629f0 0x55a97e46b220 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f084f200740 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e4650c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f084f200740 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e083390 0x55a97e459390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f084f200740 1 --2- 172.21.15.16:0/779878390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e07b1b0 0x55a97e458e50 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f084f200740 1 -- 172.21.15.16:0/779878390 >> 172.21.15.16:0/779878390 conn(0x55a97e098460 msgr2=0x55a97e44ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f0831ffb640 0 client.19038 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:35:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.996+0000 7f084f200740 1 -- 172.21.15.16:0/779878390 shutdown_connections 2024-08-28T19:35:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.997+0000 7f084f200740 1 -- 172.21.15.16:0/779878390 wait complete. 2024-08-28T19:35:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:21.997+0000 7f084f200740 20 client.19038 trim_cache size 0 max 16384 2024-08-28T19:35:22.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f084f200740 1 Processor -- start 2024-08-28T19:35:22.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f084f200740 1 -- start start 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f084f200740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e464fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f084f200740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e4576d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f084f200740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e457c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f084f200740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08441ae9e0 con 0x55a97e083390 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f084f200740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08441b0760 con 0x55a97e0a4070 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f084f200740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f08441af140 con 0x55a97e07b1b0 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f0841d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e4576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f0841d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e4576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37284/0 (socket says 172.21.15.16:37284) 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f0841d9b640 1 -- 172.21.15.16:0/748281830 learned_addr learned my addr 172.21.15.16:0/748281830 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f0842d9d640 1 --2- 172.21.15.16:0/748281830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e464fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f0842d9d640 1 --2- 172.21.15.16:0/748281830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e457c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f0841d9b640 1 -- 172.21.15.16:0/748281830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 msgr2=0x55a97e464fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f0841d9b640 1 --2- 172.21.15.16:0/748281830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e464fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f0841d9b640 1 -- 172.21.15.16:0/748281830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 msgr2=0x55a97e457c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:22.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f0841d9b640 1 --2- 172.21.15.16:0/748281830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e457c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.015+0000 7f0841d9b640 1 -- 172.21.15.16:0/748281830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a97e45a110 con 0x55a97e083390 2024-08-28T19:35:22.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.016+0000 7f0841d9b640 1 --2- 172.21.15.16:0/748281830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e4576d0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f082c011470 tx=0x7f082c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89eae8c913087e1d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:22.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.016+0000 7f0833fff640 1 -- 172.21.15.16:0/748281830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c029070 con 0x55a97e083390 2024-08-28T19:35:22.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.016+0000 7f0833fff640 1 -- 172.21.15.16:0/748281830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f082c01dbb0 con 0x55a97e083390 2024-08-28T19:35:22.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.016+0000 7f0833fff640 1 -- 172.21.15.16:0/748281830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c01ec80 con 0x55a97e083390 2024-08-28T19:35:22.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.017+0000 7f084f200740 1 -- 172.21.15.16:0/748281830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 msgr2=0x55a97e4576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:22.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.017+0000 7f084f200740 1 --2- 172.21.15.16:0/748281830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e4576d0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f082c011470 tx=0x7f082c005f90 comp rx=0 tx=0).stop 2024-08-28T19:35:22.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.017+0000 7f084f200740 1 -- 172.21.15.16:0/748281830 shutdown_connections 2024-08-28T19:35:22.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.017+0000 7f084f200740 1 --2- 172.21.15.16:0/748281830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e457c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.017+0000 7f084f200740 1 --2- 172.21.15.16:0/748281830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e4576d0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.017+0000 7f084f200740 1 --2- 172.21.15.16:0/748281830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e464fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.017+0000 7f084f200740 1 -- 172.21.15.16:0/748281830 >> 172.21.15.16:0/748281830 conn(0x55a97e098460 msgr2=0x55a97e453f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.017+0000 7f084f200740 1 -- 172.21.15.16:0/748281830 shutdown_connections 2024-08-28T19:35:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.017+0000 7f084f200740 1 -- 172.21.15.16:0/748281830 wait complete. 2024-08-28T19:35:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.018+0000 7f084f200740 1 -- start start 2024-08-28T19:35:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.018+0000 7f084f200740 1 Processor -- start 2024-08-28T19:35:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.018+0000 7f084f200740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e45ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.018+0000 7f084f200740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e45cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.018+0000 7f084f200740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e4cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.018+0000 7f084f200740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08441b0030 con 0x55a97e083390 2024-08-28T19:35:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.018+0000 7f084f200740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08441af8a0 con 0x55a97e0a4070 2024-08-28T19:35:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.018+0000 7f084f200740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f08441ac9e0 con 0x55a97e07b1b0 2024-08-28T19:35:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.018+0000 7f084f200740 10 client.0 _ll_register_callbacks cb 0x7f082c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.018+0000 7f0842d9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e45ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0842d9d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e45ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56844/0 (socket says 172.21.15.16:56844) 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0842d9d640 1 -- 172.21.15.16:0/2811674523 learned_addr learned my addr 172.21.15.16:0/2811674523 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.018+0000 7f0841d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e45cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0842d9d640 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e4cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0841d9b640 1 -- 172.21.15.16:0/2811674523 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 msgr2=0x55a97e45ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0841d9b640 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e45ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0841d9b640 1 -- 172.21.15.16:0/2811674523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 msgr2=0x55a97e4cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0841d9b640 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e4cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0841d9b640 1 -- 172.21.15.16:0/2811674523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a97e4ce2c0 con 0x55a97e083390 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0841d9b640 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e45cff0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x55a97e447eb0 tx=0x7f082c01dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac1ca69a69b8d66 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0831ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0831ffb640 1 -- 172.21.15.16:0/2811674523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c029070 con 0x55a97e083390 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f084f200740 1 -- 172.21.15.16:0/2811674523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f082c01dbb0 con 0x55a97e083390 2024-08-28T19:35:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.019+0000 7f0831ffb640 1 -- 172.21.15.16:0/2811674523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f082c01ea00 con 0x55a97e083390 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.020+0000 7f0831ffb640 1 -- 172.21.15.16:0/2811674523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f082c01ec20 con 0x55a97e083390 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.020+0000 7f084f200740 10 client.19044 fetch_fsmap learned FSMap version 75 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.020+0000 7f084f200740 1 -- 172.21.15.16:0/2811674523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) -- 0x55a97e4d56b0 con 0x55a97e083390 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.020+0000 7f0831ffb640 1 -- 172.21.15.16:0/2811674523 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f082c01dbb0 con 0x55a97e083390 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.020+0000 7f0831ffb640 1 -- 172.21.15.16:0/2811674523 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 75) ==== 8434+0+0 (secure 0 0 0) 0x7f082c0035f0 con 0x55a97e083390 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.020+0000 7f084f200740 10 client.19044 fetch_fsmap finished waiting for FSMap version 75 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.020+0000 7f084f200740 10 client.19044 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.020+0000 7f084f200740 20 client.19044 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.020+0000 7f084f200740 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55a97e037da0 0x55a97e4d7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.020+0000 7f084f200740 4 client.19044 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.020+0000 7f084f200740 1 -- 172.21.15.16:0/2811674523 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a97e3be070 con 0x55a97e037da0 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.021+0000 7f0842d9d640 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55a97e037da0 0x55a97e4d7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.021+0000 7f0831ffb640 10 client.19044 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:35:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.021+0000 7f0842d9d640 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55a97e037da0 0x55a97e4d7e70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=64cbcb9eec1ca6f3 server_cookie=30fe050525ffb7ff in_seq=0 out_seq=0 2024-08-28T19:35:22.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f0831ffb640 1 -- 172.21.15.16:0/2811674523 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a97e3be070 con 0x55a97e037da0 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f0831ffb640 10 client.19044 handle_command_reply: tid=0 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 1 client.19044 shutdown 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 1 -- 172.21.15.16:0/2811674523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 msgr2=0x55a97e45cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e45cff0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x55a97e447eb0 tx=0x7f082c01dd60 comp rx=0 tx=0).stop 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 1 -- 172.21.15.16:0/2811674523 shutdown_connections 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55a97e037da0 0x55a97e4d7e70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a97e0a4070 0x55a97e4cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a97e083390 0x55a97e45cff0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 1 --2- 172.21.15.16:0/2811674523 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a97e07b1b0 0x55a97e45ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 1 -- 172.21.15.16:0/2811674523 >> 172.21.15.16:0/2811674523 conn(0x55a97e098460 msgr2=0x55a97e457690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f0831ffb640 0 client.19044 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 1 -- 172.21.15.16:0/2811674523 shutdown_connections 2024-08-28T19:35:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 1 -- 172.21.15.16:0/2811674523 wait complete. 2024-08-28T19:35:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.025+0000 7f084f200740 20 client.19044 trim_cache size 0 max 16384 2024-08-28T19:35:22.028 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:35:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 -- 172.21.15.16:0/1626344557 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08240b4ca0 msgr2=0x7f08240b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08240b4ca0 0x7f08240b71b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f083c0e33b0 tx=0x7f083c0eb040 comp rx=0 tx=0).stop 2024-08-28T19:35:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 -- 172.21.15.16:0/1626344557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0844102800 msgr2=0x7f084419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0844102800 0x7f084419a1d0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f083c00eb90 tx=0x7f083c00c6a0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 -- 172.21.15.16:0/1626344557 shutdown_connections 2024-08-28T19:35:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f08240b4ca0 0x7f08240b71b0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844108a90 0x7f084419fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0844108140 0x7f084419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 --2- 172.21.15.16:0/1626344557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0844102800 0x7f084419a1d0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 -- 172.21.15.16:0/1626344557 >> 172.21.15.16:0/1626344557 conn(0x7f08440794d0 msgr2=0x7f084406aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 -- 172.21.15.16:0/1626344557 shutdown_connections 2024-08-28T19:35:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:22.026+0000 7f0848c8b640 1 -- 172.21.15.16:0/1626344557 wait complete. 2024-08-28T19:35:22.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 16103 18.07 MB/sec warmup 58 sec latency 3337.893 ms 2024-08-28T19:35:22.753 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:35:23.042 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:35:23.460 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:23 smithi016 ceph-mon[18556]: pgmap v785: 129 pgs: 129 active+clean; 715 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 4.7 MiB/s wr, 349 op/s 2024-08-28T19:35:23.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 16887 18.34 MB/sec warmup 59 sec latency 4337.923 ms 2024-08-28T19:35:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.604+0000 7fd297c44640 1 -- 172.21.15.16:0/2809465098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd290102800 msgr2=0x7fd290102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.604+0000 7fd297c44640 1 --2- 172.21.15.16:0/2809465098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd290102800 0x7fd290102c10 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009900 tx=0x7fd28c02a410 comp rx=0 tx=0).stop 2024-08-28T19:35:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.604+0000 7fd297c44640 1 -- 172.21.15.16:0/2809465098 shutdown_connections 2024-08-28T19:35:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.604+0000 7fd297c44640 1 --2- 172.21.15.16:0/2809465098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd290108a90 0x7fd29010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.604+0000 7fd297c44640 1 --2- 172.21.15.16:0/2809465098 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd290108140 0x7fd290108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.604+0000 7fd297c44640 1 --2- 172.21.15.16:0/2809465098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd290102800 0x7fd290102c10 secure :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009900 tx=0x7fd28c02a410 comp rx=0 tx=0).stop 2024-08-28T19:35:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.604+0000 7fd297c44640 1 -- 172.21.15.16:0/2809465098 >> 172.21.15.16:0/2809465098 conn(0x7fd2900794d0 msgr2=0x7fd2900798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.604+0000 7fd297c44640 1 -- 172.21.15.16:0/2809465098 shutdown_connections 2024-08-28T19:35:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.604+0000 7fd297c44640 1 -- 172.21.15.16:0/2809465098 wait complete. 2024-08-28T19:35:23.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.605+0000 7fd297c44640 1 Processor -- start 2024-08-28T19:35:23.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.606+0000 7fd297c44640 1 -- start start 2024-08-28T19:35:23.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.606+0000 7fd297c44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd290108140 0x7fd290105a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.606+0000 7fd297c44640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd290108a90 0x7fd290103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.606+0000 7fd297c44640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd290104530 0x7fd2901049f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.606+0000 7fd297c44640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd290112710 con 0x7fd290108140 2024-08-28T19:35:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.606+0000 7fd297c44640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd290112590 con 0x7fd290104530 2024-08-28T19:35:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.606+0000 7fd297c44640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd290112890 con 0x7fd290108a90 2024-08-28T19:35:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.606+0000 7fd2959e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd290108140 0x7fd290105a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.606+0000 7fd2959e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd290108140 0x7fd290105a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37302/0 (socket says 172.21.15.16:37302) 2024-08-28T19:35:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.606+0000 7fd2959e0640 1 -- 172.21.15.16:0/554763876 learned_addr learned my addr 172.21.15.16:0/554763876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.606+0000 7fd2961e1640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd290104530 0x7fd2901049f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.607+0000 7fd2951df640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd290108a90 0x7fd290103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.607+0000 7fd2959e0640 1 -- 172.21.15.16:0/554763876 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd290108a90 msgr2=0x7fd290103ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.607+0000 7fd2959e0640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd290108a90 0x7fd290103ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.607+0000 7fd2959e0640 1 -- 172.21.15.16:0/554763876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd290104530 msgr2=0x7fd2901049f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.607+0000 7fd2959e0640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd290104530 0x7fd2901049f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.607+0000 7fd2959e0640 1 -- 172.21.15.16:0/554763876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2901ace20 con 0x7fd290108140 2024-08-28T19:35:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.607+0000 7fd2961e1640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd290104530 0x7fd2901049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.608+0000 7fd2959e0640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd290108140 0x7fd290105a00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fd28c006010 tx=0x7fd28c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.608+0000 7fd286ffd640 1 -- 172.21.15.16:0/554763876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c03f030 con 0x7fd290108140 2024-08-28T19:35:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.608+0000 7fd286ffd640 1 -- 172.21.15.16:0/554763876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd28c003cb0 con 0x7fd290108140 2024-08-28T19:35:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.609+0000 7fd286ffd640 1 -- 172.21.15.16:0/554763876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c035070 con 0x7fd290108140 2024-08-28T19:35:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.609+0000 7fd297c44640 1 -- 172.21.15.16:0/554763876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2901ad0b0 con 0x7fd290108140 2024-08-28T19:35:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.609+0000 7fd297c44640 1 -- 172.21.15.16:0/554763876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2901ad540 con 0x7fd290108140 2024-08-28T19:35:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.613+0000 7fd29e1b9740 1 Processor -- start 2024-08-28T19:35:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.614+0000 7fd29e1b9740 1 -- start start 2024-08-28T19:35:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:23 smithi115 ceph-mon[22184]: pgmap v785: 129 pgs: 129 active+clean; 715 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 4.7 MiB/s wr, 349 op/s 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.614+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3782d070 0x55cd37bd2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.614+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37bd2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.614+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd3780c390 0x55cd37bd2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.614+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cd37b49680 con 0x55cd378041b0 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.614+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd37b38c10 con 0x55cd3782d070 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.614+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cd37b3cbc0 con 0x55cd3780c390 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.615+0000 7fd2961e1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3782d070 0x55cd37bd2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.615+0000 7fd2961e1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3782d070 0x55cd37bd2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47160/0 (socket says 172.21.15.16:47160) 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.615+0000 7fd2961e1640 1 -- 172.21.15.16:0/3338575986 learned_addr learned my addr 172.21.15.16:0/3338575986 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.615+0000 7fd286ffd640 1 -- 172.21.15.16:0/554763876 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fd28c042030 con 0x7fd290108140 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.615+0000 7fd2961e1640 1 -- 172.21.15.16:0/3338575986 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd3780c390 msgr2=0x55cd37bd2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.615+0000 7fd2961e1640 1 --2- 172.21.15.16:0/3338575986 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd3780c390 0x55cd37bd2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.615+0000 7fd2961e1640 1 -- 172.21.15.16:0/3338575986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 msgr2=0x55cd37bd2850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.616+0000 7fd2961e1640 1 --2- 172.21.15.16:0/3338575986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37bd2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.616+0000 7fd2961e1640 1 -- 172.21.15.16:0/3338575986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd37898650 con 0x55cd3782d070 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.616+0000 7fd2961e1640 1 --2- 172.21.15.16:0/3338575986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3782d070 0x55cd37bd2310 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd288010750 tx=0x7fd288010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=917a3a7799426dcf server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.617+0000 7fd27bfff640 1 -- 172.21.15.16:0/3338575986 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28800ead0 con 0x55cd3782d070 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.617+0000 7fd27bfff640 1 -- 172.21.15.16:0/3338575986 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd288010e60 con 0x55cd3782d070 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.617+0000 7fd286ffd640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd27c0b4df0 0x7fd27c0b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.617+0000 7fd27bfff640 1 -- 172.21.15.16:0/3338575986 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd288011640 con 0x55cd3782d070 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.617+0000 7fd286ffd640 1 -- 172.21.15.16:0/554763876 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7fd28c105ed0 con 0x7fd290108140 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.618+0000 7fd29e1b9740 1 -- 172.21.15.16:0/3338575986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3782d070 msgr2=0x55cd37bd2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.618+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/3338575986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3782d070 0x55cd37bd2310 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd288010750 tx=0x7fd288010c10 comp rx=0 tx=0).stop 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.618+0000 7fd29e1b9740 1 -- 172.21.15.16:0/3338575986 shutdown_connections 2024-08-28T19:35:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.618+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/3338575986 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd3780c390 0x55cd37bd2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.618+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/3338575986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37bd2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.618+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/3338575986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3782d070 0x55cd37bd2310 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.618+0000 7fd29e1b9740 1 -- 172.21.15.16:0/3338575986 >> 172.21.15.16:0/3338575986 conn(0x55cd37821460 msgr2=0x55cd37bd0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.618+0000 7fd2959e0640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd27c0b4df0 0x7fd27c0b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.619+0000 7fd29e1b9740 1 -- 172.21.15.16:0/3338575986 shutdown_connections 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.619+0000 7fd29e1b9740 1 -- 172.21.15.16:0/3338575986 wait complete. 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.620+0000 7fd29e1b9740 1 -- start start 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.620+0000 7fd29e1b9740 1 Processor -- start 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.620+0000 7fd2959e0640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd27c0b4df0 0x7fd27c0b7300 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x55cd37bd0f90 tx=0x7fd28c03b6b0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.620+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd378041b0 0x55cd37bddb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.621+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd3780c390 0x55cd37bde060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.621+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3782d070 0x55cd37be9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.621+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cd3799b700 con 0x55cd3782d070 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.621+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd37abf750 con 0x55cd378041b0 2024-08-28T19:35:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.621+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cd37be0b40 con 0x55cd3780c390 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.621+0000 7fd29e1b9740 10 client.0 _ll_register_callbacks cb 0x55cd37aeee10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.621+0000 7fd2951df640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd3780c390 0x55cd37bde060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.621+0000 7fd2951df640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd3780c390 0x55cd37bde060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56876/0 (socket says 172.21.15.16:56876) 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.621+0000 7fd2951df640 1 -- 172.21.15.16:0/86970533 learned_addr learned my addr 172.21.15.16:0/86970533 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.621+0000 7fd2961e1640 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd378041b0 0x55cd37bddb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.621+0000 7fd2961e1640 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3782d070 0x55cd37be9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.622+0000 7fd2951df640 1 -- 172.21.15.16:0/86970533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd378041b0 msgr2=0x55cd37bddb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.622+0000 7fd2951df640 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd378041b0 0x55cd37bddb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.622+0000 7fd2951df640 1 -- 172.21.15.16:0/86970533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3782d070 msgr2=0x55cd37be9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.622+0000 7fd2951df640 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3782d070 0x55cd37be9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.622+0000 7fd2951df640 1 -- 172.21.15.16:0/86970533 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd37af8900 con 0x55cd3780c390 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.622+0000 7fd2961e1640 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd378041b0 0x55cd37bddb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.622+0000 7fd2951df640 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd3780c390 0x55cd37bde060 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd28000bdb0 tx=0x7fd2800107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f5be6dad7b39098 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.622+0000 7fd2797fa640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:35:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.622+0000 7fd2797fa640 1 -- 172.21.15.16:0/86970533 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28001c680 con 0x55cd3780c390 2024-08-28T19:35:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.623+0000 7fd29e1b9740 1 -- 172.21.15.16:0/86970533 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd288010e60 con 0x55cd3780c390 2024-08-28T19:35:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.623+0000 7fd2797fa640 1 -- 172.21.15.16:0/86970533 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7fd28001c820 con 0x55cd3780c390 2024-08-28T19:35:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.623+0000 7fd2797fa640 1 -- 172.21.15.16:0/86970533 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fd280025050 con 0x55cd3780c390 2024-08-28T19:35:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.623+0000 7fd29e1b9740 10 client.28141 fetch_fsmap learned FSMap version 76 2024-08-28T19:35:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.623+0000 7fd29e1b9740 1 -- 172.21.15.16:0/86970533 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55cd37bec960 con 0x55cd3780c390 2024-08-28T19:35:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.624+0000 7fd2797fa640 1 -- 172.21.15.16:0/86970533 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd288010e60 con 0x55cd3780c390 2024-08-28T19:35:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.624+0000 7fd2797fa640 1 -- 172.21.15.16:0/86970533 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 76) ==== 8438+0+0 (secure 0 0 0) 0x7fd28001a600 con 0x55cd3780c390 2024-08-28T19:35:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.625+0000 7fd29e1b9740 10 client.28141 fetch_fsmap finished waiting for FSMap version 76 2024-08-28T19:35:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.625+0000 7fd29e1b9740 10 client.28141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:35:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.625+0000 7fd29e1b9740 20 client.28141 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.625+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cd377eb9f0 0x55cd37bf0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.625+0000 7fd29e1b9740 4 client.28141 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:35:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.625+0000 7fd29e1b9740 1 -- 172.21.15.16:0/86970533 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd37b4b5d0 con 0x55cd377eb9f0 2024-08-28T19:35:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.626+0000 7fd2961e1640 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cd377eb9f0 0x55cd37bf0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.628+0000 7fd2961e1640 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cd377eb9f0 0x55cd37bf0300 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f006f53df1856644 server_cookie=70edc516d09779bb in_seq=0 out_seq=0 2024-08-28T19:35:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.628+0000 7fd2797fa640 10 client.28141 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.630+0000 7fd2797fa640 1 -- 172.21.15.16:0/86970533 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cd37b4b5d0 con 0x55cd377eb9f0 2024-08-28T19:35:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.630+0000 7fd2797fa640 10 client.28141 handle_command_reply: tid=0 2024-08-28T19:35:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.630+0000 7fd29e1b9740 1 client.28141 shutdown 2024-08-28T19:35:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.630+0000 7fd29e1b9740 1 -- 172.21.15.16:0/86970533 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd3780c390 msgr2=0x55cd37bde060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.630+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd3780c390 0x55cd37bde060 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd28000bdb0 tx=0x7fd2800107a0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.631+0000 7fd29e1b9740 1 -- 172.21.15.16:0/86970533 shutdown_connections 2024-08-28T19:35:23.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.631+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3782d070 0x55cd37be9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.631+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cd377eb9f0 0x55cd37bf0300 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.631+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd3780c390 0x55cd37bde060 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.631+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/86970533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd378041b0 0x55cd37bddb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.631+0000 7fd29e1b9740 1 -- 172.21.15.16:0/86970533 >> 172.21.15.16:0/86970533 conn(0x55cd37821460 msgr2=0x55cd37bd12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.631+0000 7fd2797fa640 0 client.28141 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.633+0000 7fd29e1b9740 1 -- 172.21.15.16:0/86970533 shutdown_connections 2024-08-28T19:35:23.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.633+0000 7fd29e1b9740 1 -- 172.21.15.16:0/86970533 wait complete. 2024-08-28T19:35:23.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.633+0000 7fd29e1b9740 20 client.28141 trim_cache size 0 max 16384 2024-08-28T19:35:23.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.654+0000 7fd29e1b9740 1 Processor -- start 2024-08-28T19:35:23.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.654+0000 7fd29e1b9740 1 -- start start 2024-08-28T19:35:23.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.654+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd37821460 0x55cd37bd0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.654+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37be9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.654+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3780c390 0x55cd37bdadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.654+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901b7990 con 0x55cd378041b0 2024-08-28T19:35:23.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.654+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901b9740 con 0x55cd3780c390 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.654+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901b80f0 con 0x55cd37821460 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2951df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37be9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2961e1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd37821460 0x55cd37bd0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2951df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37be9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37330/0 (socket says 172.21.15.16:37330) 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2951df640 1 -- 172.21.15.16:0/2309203849 learned_addr learned my addr 172.21.15.16:0/2309203849 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2961e1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd37821460 0x55cd37bd0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56884/0 (socket says 172.21.15.16:56884) 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2961e1640 1 --2- 172.21.15.16:0/2309203849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3780c390 0x55cd37bdadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2951df640 1 -- 172.21.15.16:0/2309203849 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd37821460 msgr2=0x55cd37bd0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2951df640 1 --2- 172.21.15.16:0/2309203849 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd37821460 0x55cd37bd0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2951df640 1 -- 172.21.15.16:0/2309203849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3780c390 msgr2=0x55cd37bdadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2951df640 1 --2- 172.21.15.16:0/2309203849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3780c390 0x55cd37bdadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2951df640 1 -- 172.21.15.16:0/2309203849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd37bd4600 con 0x55cd378041b0 2024-08-28T19:35:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.655+0000 7fd2951df640 1 --2- 172.21.15.16:0/2309203849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37be9d60 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fd28001b000 tx=0x7fd280001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1be8bc1c2c3a20 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.656+0000 7fd27b7fe640 1 -- 172.21.15.16:0/2309203849 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28002d070 con 0x55cd378041b0 2024-08-28T19:35:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.656+0000 7fd27b7fe640 1 -- 172.21.15.16:0/2309203849 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd280005040 con 0x55cd378041b0 2024-08-28T19:35:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.656+0000 7fd27b7fe640 1 -- 172.21.15.16:0/2309203849 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2800048e0 con 0x55cd378041b0 2024-08-28T19:35:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.657+0000 7fd29e1b9740 1 -- 172.21.15.16:0/2309203849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 msgr2=0x55cd37be9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.657+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/2309203849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37be9d60 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fd28001b000 tx=0x7fd280001e50 comp rx=0 tx=0).stop 2024-08-28T19:35:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.657+0000 7fd29e1b9740 1 -- 172.21.15.16:0/2309203849 shutdown_connections 2024-08-28T19:35:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.657+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/2309203849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3780c390 0x55cd37bdadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.657+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/2309203849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37be9d60 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.657+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/2309203849 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd37821460 0x55cd37bd0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.657+0000 7fd29e1b9740 1 -- 172.21.15.16:0/2309203849 >> 172.21.15.16:0/2309203849 conn(0x55cd37bd5780 msgr2=0x55cd37bcfa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.658+0000 7fd29e1b9740 1 -- 172.21.15.16:0/2309203849 shutdown_connections 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.658+0000 7fd29e1b9740 1 -- 172.21.15.16:0/2309203849 wait complete. 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.658+0000 7fd29e1b9740 1 -- start start 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.658+0000 7fd29e1b9740 1 Processor -- start 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.659+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37bdb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.659+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3780c390 0x55cd37bd8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.659+0000 7fd29e1b9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd37821460 0x55cd37bd9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.659+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901b8fe0 con 0x55cd378041b0 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.659+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901b8850 con 0x55cd3780c390 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.659+0000 7fd29e1b9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901b5990 con 0x55cd37821460 2024-08-28T19:35:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.659+0000 7fd29e1b9740 10 client.0 _ll_register_callbacks cb 0x7fd28000b9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.659+0000 7fd2961e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37bdb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.659+0000 7fd2961e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37bdb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37342/0 (socket says 172.21.15.16:37342) 2024-08-28T19:35:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.659+0000 7fd2961e1640 1 -- 172.21.15.16:0/1774777582 learned_addr learned my addr 172.21.15.16:0/1774777582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.660+0000 7fd2951df640 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3780c390 0x55cd37bd8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.660+0000 7fd2961e1640 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd37821460 0x55cd37bd9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.660+0000 7fd2961e1640 1 -- 172.21.15.16:0/1774777582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd37821460 msgr2=0x55cd37bd9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.660+0000 7fd2961e1640 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd37821460 0x55cd37bd9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.660+0000 7fd2961e1640 1 -- 172.21.15.16:0/1774777582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3780c390 msgr2=0x55cd37bd8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.660+0000 7fd2961e1640 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3780c390 0x55cd37bd8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.660+0000 7fd2961e1640 1 -- 172.21.15.16:0/1774777582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd37bd9840 con 0x55cd378041b0 2024-08-28T19:35:23.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.661+0000 7fd2961e1640 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37bdb530 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fd288013d80 tx=0x7fd28000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad002716160e4d01 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.661+0000 7fd2797fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.661+0000 7fd2797fa640 1 -- 172.21.15.16:0/1774777582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd288037020 con 0x55cd378041b0 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.662+0000 7fd2797fa640 1 -- 172.21.15.16:0/1774777582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7fd2880127a0 con 0x55cd378041b0 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.662+0000 7fd29e1b9740 1 -- 172.21.15.16:0/1774777582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd280005040 con 0x55cd378041b0 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.662+0000 7fd2797fa640 1 -- 172.21.15.16:0/1774777582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fd288025040 con 0x55cd378041b0 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.662+0000 7fd29e1b9740 10 client.19059 fetch_fsmap learned FSMap version 76 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.662+0000 7fd29e1b9740 1 -- 172.21.15.16:0/1774777582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55cd37be4db0 con 0x55cd378041b0 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.663+0000 7fd2797fa640 1 -- 172.21.15.16:0/1774777582 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd280005040 con 0x55cd378041b0 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.663+0000 7fd2797fa640 1 -- 172.21.15.16:0/1774777582 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 76) ==== 8438+0+0 (secure 0 0 0) 0x7fd28800f040 con 0x55cd378041b0 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.663+0000 7fd29e1b9740 10 client.19059 fetch_fsmap finished waiting for FSMap version 76 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.663+0000 7fd29e1b9740 10 client.19059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.663+0000 7fd29e1b9740 20 client.19059 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.663+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cd3782d070 0x55cd37be53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.663+0000 7fd29e1b9740 4 client.19059 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:35:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.663+0000 7fd29e1b9740 1 -- 172.21.15.16:0/1774777582 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cd37b47070 con 0x55cd3782d070 2024-08-28T19:35:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.663+0000 7fd2951df640 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cd3782d070 0x55cd37be53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.664+0000 7fd2797fa640 10 client.19059 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.664+0000 7fd2951df640 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cd3782d070 0x55cd37be53d0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24a0c3cfdaac7159 server_cookie=448814dd7204ad6c in_seq=0 out_seq=0 2024-08-28T19:35:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.665+0000 7fd2797fa640 1 -- 172.21.15.16:0/1774777582 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cd37b47070 con 0x55cd3782d070 2024-08-28T19:35:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.665+0000 7fd2797fa640 10 client.19059 handle_command_reply: tid=0 2024-08-28T19:35:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.665+0000 7fd29e1b9740 1 client.19059 shutdown 2024-08-28T19:35:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.665+0000 7fd29e1b9740 1 -- 172.21.15.16:0/1774777582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 msgr2=0x55cd37bdb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.665+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37bdb530 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fd288013d80 tx=0x7fd28000b960 comp rx=0 tx=0).stop 2024-08-28T19:35:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.666+0000 7fd29e1b9740 1 -- 172.21.15.16:0/1774777582 shutdown_connections 2024-08-28T19:35:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.666+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cd3782d070 0x55cd37be53d0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.666+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cd37821460 0x55cd37bd9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.666+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd3780c390 0x55cd37bd8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.666+0000 7fd29e1b9740 1 --2- 172.21.15.16:0/1774777582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd378041b0 0x55cd37bdb530 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.666+0000 7fd29e1b9740 1 -- 172.21.15.16:0/1774777582 >> 172.21.15.16:0/1774777582 conn(0x55cd37bd5780 msgr2=0x55cd37bd3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.666+0000 7fd2797fa640 0 client.19059 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.666+0000 7fd29e1b9740 1 -- 172.21.15.16:0/1774777582 shutdown_connections 2024-08-28T19:35:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.666+0000 7fd29e1b9740 1 -- 172.21.15.16:0/1774777582 wait complete. 2024-08-28T19:35:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.666+0000 7fd29e1b9740 20 client.19059 trim_cache size 0 max 16384 2024-08-28T19:35:23.668 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:35:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 -- 172.21.15.16:0/554763876 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd27c0b4df0 msgr2=0x7fd27c0b7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd27c0b4df0 0x7fd27c0b7300 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x55cd37bd0f90 tx=0x7fd28c03b6b0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 -- 172.21.15.16:0/554763876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd290108140 msgr2=0x7fd290105a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd290108140 0x7fd290105a00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fd28c006010 tx=0x7fd28c0042f0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 -- 172.21.15.16:0/554763876 shutdown_connections 2024-08-28T19:35:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd27c0b4df0 0x7fd27c0b7300 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd290104530 0x7fd2901049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd290108a90 0x7fd290103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 --2- 172.21.15.16:0/554763876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd290108140 0x7fd290105a00 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 -- 172.21.15.16:0/554763876 >> 172.21.15.16:0/554763876 conn(0x7fd2900794d0 msgr2=0x7fd2901018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 -- 172.21.15.16:0/554763876 shutdown_connections 2024-08-28T19:35:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:23.667+0000 7fd297c44640 1 -- 172.21.15.16:0/554763876 wait complete. 2024-08-28T19:35:23.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:23 smithi028 ceph-mon[21769]: pgmap v785: 129 pgs: 129 active+clean; 715 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 4.7 MiB/s wr, 349 op/s 2024-08-28T19:35:24.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 17972 18.92 MB/sec warmup 60 sec latency 4376.471 ms 2024-08-28T19:35:24.593 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:24 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:24 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:24.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:24 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:25.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 18995 19.76 MB/sec warmup 61 sec latency 72.173 ms 2024-08-28T19:35:25.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:25 smithi115 ceph-mon[22184]: pgmap v786: 129 pgs: 129 active+clean; 717 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 4.6 MiB/s wr, 340 op/s 2024-08-28T19:35:25.629 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:35:25.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:35:25.655 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:25 smithi016 ceph-mon[18556]: pgmap v786: 129 pgs: 129 active+clean; 717 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 4.6 MiB/s wr, 340 op/s 2024-08-28T19:35:25.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:25 smithi028 ceph-mon[21769]: pgmap v786: 129 pgs: 129 active+clean; 717 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 4.6 MiB/s wr, 340 op/s 2024-08-28T19:35:25.927 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:35:26.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 19803 20.05 MB/sec warmup 62 sec latency 73.975 ms 2024-08-28T19:35:26.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.485+0000 7f11e8765640 1 -- 172.21.15.16:0/1231014111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11e010b490 msgr2=0x7f11e010b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:26.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.485+0000 7f11e8765640 1 --2- 172.21.15.16:0/1231014111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11e010b490 0x7f11e010b8a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009860 tx=0x7f11d002a340 comp rx=0 tx=0).stop 2024-08-28T19:35:26.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.485+0000 7f11e8765640 1 -- 172.21.15.16:0/1231014111 shutdown_connections 2024-08-28T19:35:26.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.485+0000 7f11e8765640 1 --2- 172.21.15.16:0/1231014111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e010bde0 0x7f11e0112610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.485+0000 7f11e8765640 1 --2- 172.21.15.16:0/1231014111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11e010b490 0x7f11e010b8a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.485+0000 7f11e8765640 1 --2- 172.21.15.16:0/1231014111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e0105b50 0x7f11e0105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.485+0000 7f11e8765640 1 -- 172.21.15.16:0/1231014111 >> 172.21.15.16:0/1231014111 conn(0x7f11e0101560 msgr2=0x7f11e01039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.485+0000 7f11e8765640 1 -- 172.21.15.16:0/1231014111 shutdown_connections 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.485+0000 7f11e8765640 1 -- 172.21.15.16:0/1231014111 wait complete. 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.486+0000 7f11e8765640 1 Processor -- start 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.486+0000 7f11e8765640 1 -- start start 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.486+0000 7f11e8765640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11e0105b50 0x7f11e019c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.486+0000 7f11e8765640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e010b490 0x7f11e019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.486+0000 7f11e8765640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e010bde0 0x7f11e01a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.486+0000 7f11e8765640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11e0115aa0 con 0x7f11e010bde0 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.486+0000 7f11e8765640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11e0115920 con 0x7f11e010b490 2024-08-28T19:35:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.486+0000 7f11e8765640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f11e0115c20 con 0x7f11e0105b50 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e6501640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11e0105b50 0x7f11e019c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e6d02640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e010bde0 0x7f11e01a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e6501640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11e0105b50 0x7f11e019c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:46982/0 (socket says 172.21.15.16:46982) 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e6501640 1 -- 172.21.15.16:0/3033823545 learned_addr learned my addr 172.21.15.16:0/3033823545 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e5d00640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e010b490 0x7f11e019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e6501640 1 -- 172.21.15.16:0/3033823545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e010b490 msgr2=0x7f11e019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e6501640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e010b490 0x7f11e019ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e6501640 1 -- 172.21.15.16:0/3033823545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e010bde0 msgr2=0x7f11e01a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e6501640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e010bde0 0x7f11e01a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e6501640 1 -- 172.21.15.16:0/3033823545 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e01a2420 con 0x7f11e0105b50 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e6501640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11e0105b50 0x7f11e019c4b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00e9c0 tx=0x7f11dc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11d77fe640 1 -- 172.21.15.16:0/3033823545 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc00ccc0 con 0x7f11e0105b50 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11e8765640 1 -- 172.21.15.16:0/3033823545 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e01a2710 con 0x7f11e0105b50 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.487+0000 7f11d77fe640 1 -- 172.21.15.16:0/3033823545 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11dc0054d0 con 0x7f11e0105b50 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.488+0000 7f11d77fe640 1 -- 172.21.15.16:0/3033823545 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc010430 con 0x7f11e0105b50 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.488+0000 7f11e8765640 1 -- 172.21.15.16:0/3033823545 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e00775a0 con 0x7f11e0105b50 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.491+0000 7f11d77fe640 1 -- 172.21.15.16:0/3033823545 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f11dc0105d0 con 0x7f11e0105b50 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.491+0000 7f11eecda740 1 Processor -- start 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.492+0000 7f11eecda740 1 -- start start 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.492+0000 7f11eecda740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 0x55e68801f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.492+0000 7f11eecda740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 0x55e68801fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.492+0000 7f11eecda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c58670 0x55e688020200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.492+0000 7f11eecda740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e687f955f0 con 0x55e687c79350 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.492+0000 7f11eecda740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e687f84b80 con 0x55e687c58670 2024-08-28T19:35:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.492+0000 7f11eecda740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e687f88b30 con 0x55e687c50490 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.492+0000 7f11e6d02640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 0x55e68801f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.492+0000 7f11e6d02640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 0x55e68801f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43818/0 (socket says 172.21.15.16:43818) 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.492+0000 7f11e6d02640 1 -- 172.21.15.16:0/4251272953 learned_addr learned my addr 172.21.15.16:0/4251272953 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.492+0000 7f11e6501640 1 --2- 172.21.15.16:0/4251272953 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 0x55e68801fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.493+0000 7f11e5d00640 1 --2- 172.21.15.16:0/4251272953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c58670 0x55e688020200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.493+0000 7f11d77fe640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11cc0b4ca0 0x7f11cc0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.493+0000 7f11e6d02640 1 -- 172.21.15.16:0/4251272953 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 msgr2=0x55e68801fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.493+0000 7f11e6d02640 1 --2- 172.21.15.16:0/4251272953 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 0x55e68801fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.493+0000 7f11e6d02640 1 -- 172.21.15.16:0/4251272953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c58670 msgr2=0x55e688020200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.493+0000 7f11e6d02640 1 --2- 172.21.15.16:0/4251272953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c58670 0x55e688020200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.493+0000 7f11e6d02640 1 -- 172.21.15.16:0/4251272953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e687ce4930 con 0x55e687c79350 2024-08-28T19:35:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.493+0000 7f11e6d02640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11cc0b4ca0 0x7f11cc0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.493+0000 7f11d77fe640 1 -- 172.21.15.16:0/3033823545 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f11dc0e5d40 con 0x7f11e0105b50 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.493+0000 7f11e6d02640 1 --2- 172.21.15.16:0/4251272953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 0x55e68801f780 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f11d8009b30 tx=0x7f11d8008230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742343609dad9ac4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.493+0000 7f11d4ff9640 1 -- 172.21.15.16:0/4251272953 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8008a00 con 0x55e687c79350 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.494+0000 7f11d4ff9640 1 -- 172.21.15.16:0/4251272953 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11d8014660 con 0x55e687c79350 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.494+0000 7f11d4ff9640 1 -- 172.21.15.16:0/4251272953 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d801f410 con 0x55e687c79350 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.495+0000 7f11eecda740 1 -- 172.21.15.16:0/4251272953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 msgr2=0x55e68801f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.495+0000 7f11e6d02640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11cc0b4ca0 0x7f11cc0b71b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f11d8027000 tx=0x7f11d8021040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.495+0000 7f11eecda740 1 --2- 172.21.15.16:0/4251272953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 0x55e68801f780 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f11d8009b30 tx=0x7f11d8008230 comp rx=0 tx=0).stop 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.495+0000 7f11eecda740 1 -- 172.21.15.16:0/4251272953 shutdown_connections 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.495+0000 7f11eecda740 1 --2- 172.21.15.16:0/4251272953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c58670 0x55e688020200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.495+0000 7f11eecda740 1 --2- 172.21.15.16:0/4251272953 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 0x55e68801fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.495+0000 7f11eecda740 1 --2- 172.21.15.16:0/4251272953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 0x55e68801f780 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.495+0000 7f11eecda740 1 -- 172.21.15.16:0/4251272953 >> 172.21.15.16:0/4251272953 conn(0x55e687c6d740 msgr2=0x55e68801d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.495+0000 7f11eecda740 1 -- 172.21.15.16:0/4251272953 shutdown_connections 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.495+0000 7f11eecda740 1 -- 172.21.15.16:0/4251272953 wait complete. 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.495+0000 7f11eecda740 1 -- start start 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11eecda740 1 Processor -- start 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11eecda740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 0x55e68802af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11eecda740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c58670 0x55e68802b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11eecda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 0x55e688037250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11eecda740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e687de7890 con 0x55e687c58670 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11eecda740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e687f0b790 con 0x55e687c79350 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11eecda740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e68802df60 con 0x55e687c50490 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11e5d00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c58670 0x55e68802b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11e6501640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 0x55e68802af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11e5d00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c58670 0x55e68802b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43822/0 (socket says 172.21.15.16:43822) 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11e6501640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 0x55e68802af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:46992/0 (socket says 172.21.15.16:46992) 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11e5d00640 1 -- 172.21.15.16:0/2268849802 learned_addr learned my addr 172.21.15.16:0/2268849802 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11eecda740 10 client.0 _ll_register_callbacks cb 0x55e687c90ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.496+0000 7f11e6d02640 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 0x55e688037250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.497+0000 7f11e5d00640 1 -- 172.21.15.16:0/2268849802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 msgr2=0x55e68802af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.497+0000 7f11e5d00640 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 0x55e68802af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.497+0000 7f11e5d00640 1 -- 172.21.15.16:0/2268849802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 msgr2=0x55e688037250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.497+0000 7f11e5d00640 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 0x55e688037250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.497+0000 7f11e5d00640 1 -- 172.21.15.16:0/2268849802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e687f42f30 con 0x55e687c58670 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.497+0000 7f11e6501640 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 0x55e68802af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.497+0000 7f11e6d02640 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 0x55e688037250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.497+0000 7f11e5d00640 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c58670 0x55e68802b470 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009860 tx=0x7f11d0030df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41108232418f899 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.497+0000 7f11c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.497+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2268849802 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0042070 con 0x55e687c58670 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.497+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2268849802 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f11d0004240 con 0x55e687c58670 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.498+0000 7f11eecda740 1 -- 172.21.15.16:0/2268849802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11d8014660 con 0x55e687c58670 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.498+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2268849802 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f11d004c050 con 0x55e687c58670 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.498+0000 7f11eecda740 10 client.19071 fetch_fsmap learned FSMap version 76 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.498+0000 7f11eecda740 1 -- 172.21.15.16:0/2268849802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55e688039d40 con 0x55e687c58670 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.498+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2268849802 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11d8014660 con 0x55e687c58670 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.499+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2268849802 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 76) ==== 8438+0+0 (secure 0 0 0) 0x7f11d0003b50 con 0x55e687c58670 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.499+0000 7f11eecda740 10 client.19071 fetch_fsmap finished waiting for FSMap version 76 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.499+0000 7f11eecda740 10 client.19071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:35:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.499+0000 7f11eecda740 20 client.19071 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.499+0000 7f11eecda740 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e687c37cd0 0x55e68803d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.499+0000 7f11eecda740 4 client.19071 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.499+0000 7f11eecda740 1 -- 172.21.15.16:0/2268849802 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e687f97540 con 0x55e687c37cd0 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.499+0000 7f11e6501640 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e687c37cd0 0x55e68803d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.500+0000 7f11c9ffb640 10 client.19071 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.500+0000 7f11e6501640 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e687c37cd0 0x55e68803d340 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7faffa1df6138dc server_cookie=34ecc232f3d0f554 in_seq=0 out_seq=0 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2268849802 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e687f97540 con 0x55e687c37cd0 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11c9ffb640 10 client.19071 handle_command_reply: tid=0 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11eecda740 1 client.19071 shutdown 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11eecda740 1 -- 172.21.15.16:0/2268849802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c58670 msgr2=0x55e68802b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11eecda740 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c58670 0x55e68802b470 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009860 tx=0x7f11d0030df0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11eecda740 1 -- 172.21.15.16:0/2268849802 shutdown_connections 2024-08-28T19:35:26.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11eecda740 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 0x55e688037250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11eecda740 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c58670 0x55e68802b470 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11eecda740 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e687c37cd0 0x55e68803d340 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11eecda740 1 --2- 172.21.15.16:0/2268849802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c50490 0x55e68802af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11eecda740 1 -- 172.21.15.16:0/2268849802 >> 172.21.15.16:0/2268849802 conn(0x55e687c6d740 msgr2=0x55e688022960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.501+0000 7f11c9ffb640 0 client.19071 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.502+0000 7f11eecda740 1 -- 172.21.15.16:0/2268849802 shutdown_connections 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.502+0000 7f11eecda740 1 -- 172.21.15.16:0/2268849802 wait complete. 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.502+0000 7f11eecda740 20 client.19071 trim_cache size 0 max 16384 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.523+0000 7f11eecda740 1 Processor -- start 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.523+0000 7f11eecda740 1 -- start start 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11eecda740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c50490 0x55e68802ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11eecda740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c58670 0x55e68802b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11eecda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 0x55e688022880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11eecda740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01aeb80 con 0x55e687c50490 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11eecda740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01b0930 con 0x55e687c79350 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11eecda740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01af2e0 con 0x55e687c58670 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11e6501640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c50490 0x55e68802ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11e6501640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c50490 0x55e68802ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43828/0 (socket says 172.21.15.16:43828) 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11e6501640 1 -- 172.21.15.16:0/2927256636 learned_addr learned my addr 172.21.15.16:0/2927256636 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11e6d02640 1 --2- 172.21.15.16:0/2927256636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 0x55e688022880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11e5d00640 1 --2- 172.21.15.16:0/2927256636 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c58670 0x55e68802b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11e6d02640 1 -- 172.21.15.16:0/2927256636 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c58670 msgr2=0x55e68802b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11e6d02640 1 --2- 172.21.15.16:0/2927256636 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c58670 0x55e68802b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11e6d02640 1 -- 172.21.15.16:0/2927256636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c50490 msgr2=0x55e68802ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11e6d02640 1 --2- 172.21.15.16:0/2927256636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c50490 0x55e68802ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11e6d02640 1 -- 172.21.15.16:0/2927256636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e688029430 con 0x55e687c79350 2024-08-28T19:35:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.524+0000 7f11e6501640 1 --2- 172.21.15.16:0/2927256636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c50490 0x55e68802ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.525+0000 7f11e6d02640 1 --2- 172.21.15.16:0/2927256636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 0x55e688022880 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f11d8008730 tx=0x7f11d802e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66772700976838fe server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.525+0000 7f11cbfff640 1 -- 172.21.15.16:0/2927256636 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d803a070 con 0x55e687c79350 2024-08-28T19:35:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.525+0000 7f11cbfff640 1 -- 172.21.15.16:0/2927256636 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11d8012d20 con 0x55e687c79350 2024-08-28T19:35:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.525+0000 7f11cbfff640 1 -- 172.21.15.16:0/2927256636 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8014880 con 0x55e687c79350 2024-08-28T19:35:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.526+0000 7f11eecda740 1 -- 172.21.15.16:0/2927256636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 msgr2=0x55e688022880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.526+0000 7f11eecda740 1 --2- 172.21.15.16:0/2927256636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 0x55e688022880 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f11d8008730 tx=0x7f11d802e000 comp rx=0 tx=0).stop 2024-08-28T19:35:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.526+0000 7f11eecda740 1 -- 172.21.15.16:0/2927256636 shutdown_connections 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.526+0000 7f11eecda740 1 --2- 172.21.15.16:0/2927256636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c79350 0x55e688022880 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.526+0000 7f11eecda740 1 --2- 172.21.15.16:0/2927256636 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c58670 0x55e68802b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.526+0000 7f11eecda740 1 --2- 172.21.15.16:0/2927256636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c50490 0x55e68802ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.526+0000 7f11eecda740 1 -- 172.21.15.16:0/2927256636 >> 172.21.15.16:0/2927256636 conn(0x55e687c6d740 msgr2=0x55e6880297b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.526+0000 7f11eecda740 1 -- 172.21.15.16:0/2927256636 shutdown_connections 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.526+0000 7f11eecda740 1 -- 172.21.15.16:0/2927256636 wait complete. 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.527+0000 7f11eecda740 1 -- start start 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.527+0000 7f11eecda740 1 Processor -- start 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.527+0000 7f11eecda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c50490 0x55e688033ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.527+0000 7f11eecda740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c58670 0x55e68802f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11eecda740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 0x55e68802f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11eecda740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01b01d0 con 0x55e687c79350 2024-08-28T19:35:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11eecda740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01afa40 con 0x55e687c50490 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11eecda740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01acbc0 con 0x55e687c58670 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11eecda740 10 client.0 _ll_register_callbacks cb 0x7f11d00096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11e6501640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c50490 0x55e688033ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11e5d00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c58670 0x55e68802f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11e6501640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c50490 0x55e688033ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34828/0 (socket says 172.21.15.16:34828) 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11e6501640 1 -- 172.21.15.16:0/2177236158 learned_addr learned my addr 172.21.15.16:0/2177236158 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11e6d02640 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 0x55e68802f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11e5d00640 1 -- 172.21.15.16:0/2177236158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c50490 msgr2=0x55e688033ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11e5d00640 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c50490 0x55e688033ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11e5d00640 1 -- 172.21.15.16:0/2177236158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 msgr2=0x55e68802f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11e5d00640 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 0x55e68802f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11e5d00640 1 -- 172.21.15.16:0/2177236158 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6880a6d00 con 0x55e687c58670 2024-08-28T19:35:26.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11e5d00640 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c58670 0x55e68802f090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f11d004a040 tx=0x7f11d00056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70397ac85a73d2d1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.528+0000 7f11c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.529+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2177236158 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0042040 con 0x55e687c58670 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.529+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2177236158 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f11d0030860 con 0x55e687c58670 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.529+0000 7f11eecda740 1 -- 172.21.15.16:0/2177236158 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11d8012d20 con 0x55e687c58670 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.529+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2177236158 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f11d005f050 con 0x55e687c58670 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.529+0000 7f11eecda740 10 client.28165 fetch_fsmap learned FSMap version 76 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.529+0000 7f11eecda740 1 -- 172.21.15.16:0/2177236158 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55e6880a7e90 con 0x55e687c58670 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.529+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2177236158 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11d0005d30 con 0x55e687c58670 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.529+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2177236158 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 76) ==== 8438+0+0 (secure 0 0 0) 0x7f11d0003860 con 0x55e687c58670 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.530+0000 7f11eecda740 10 client.28165 fetch_fsmap finished waiting for FSMap version 76 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.530+0000 7f11eecda740 10 client.28165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.530+0000 7f11eecda740 20 client.28165 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.530+0000 7f11eecda740 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e687c0d080 0x55e6880aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:26.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.530+0000 7f11eecda740 4 client.28165 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.530+0000 7f11eecda740 1 -- 172.21.15.16:0/2177236158 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f11d8012d20 con 0x55e687c0d080 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.530+0000 7f11e6501640 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e687c0d080 0x55e6880aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.531+0000 7f11c9ffb640 10 client.28165 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.531+0000 7f11e6501640 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e687c0d080 0x55e6880aa6a0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=470129a29ea57fe7 server_cookie=6a5511adc4a499f7 in_seq=0 out_seq=0 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11c9ffb640 1 -- 172.21.15.16:0/2177236158 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f11d8012d20 con 0x55e687c0d080 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11c9ffb640 10 client.28165 handle_command_reply: tid=0 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11eecda740 1 client.28165 shutdown 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11eecda740 1 -- 172.21.15.16:0/2177236158 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c58670 msgr2=0x55e68802f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11eecda740 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c58670 0x55e68802f090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f11d004a040 tx=0x7f11d00056d0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11eecda740 1 -- 172.21.15.16:0/2177236158 shutdown_connections 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11eecda740 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687c79350 0x55e68802f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11eecda740 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e687c0d080 0x55e6880aa6a0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11eecda740 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687c58670 0x55e68802f090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11eecda740 1 --2- 172.21.15.16:0/2177236158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687c50490 0x55e688033ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11eecda740 1 -- 172.21.15.16:0/2177236158 >> 172.21.15.16:0/2177236158 conn(0x55e687c6d740 msgr2=0x55e68802ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:26.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11c9ffb640 0 client.28165 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:26.701 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:35:26.701 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:35:26.701 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "840047cc-208d-4758-921b-8f80ba8e251f", 2024-08-28T19:35:26.701 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:35:26.701 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.535+0000 7f11eecda740 1 -- 172.21.15.16:0/2177236158 shutdown_connections 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.536+0000 7f11eecda740 1 -- 172.21.15.16:0/2177236158 wait complete. 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.536+0000 7f11eecda740 20 client.28165 trim_cache size 0 max 16384 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.536+0000 7f11e8765640 1 -- 172.21.15.16:0/3033823545 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11cc0b4ca0 msgr2=0x7f11cc0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.536+0000 7f11e8765640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11cc0b4ca0 0x7f11cc0b71b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f11d8027000 tx=0x7f11d8021040 comp rx=0 tx=0).stop 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.536+0000 7f11e8765640 1 -- 172.21.15.16:0/3033823545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11e0105b50 msgr2=0x7f11e019c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.536+0000 7f11e8765640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11e0105b50 0x7f11e019c4b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00e9c0 tx=0x7f11dc00ee80 comp rx=0 tx=0).stop 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.537+0000 7f11e8765640 1 -- 172.21.15.16:0/3033823545 shutdown_connections 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.537+0000 7f11e8765640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e010bde0 0x7f11e01a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.537+0000 7f11e8765640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f11cc0b4ca0 0x7f11cc0b71b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.537+0000 7f11e8765640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e010b490 0x7f11e019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.537+0000 7f11e8765640 1 --2- 172.21.15.16:0/3033823545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11e0105b50 0x7f11e019c4b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:26.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.537+0000 7f11e8765640 1 -- 172.21.15.16:0/3033823545 >> 172.21.15.16:0/3033823545 conn(0x7f11e0101560 msgr2=0x7f11e0107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:26.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.537+0000 7f11e8765640 1 -- 172.21.15.16:0/3033823545 shutdown_connections 2024-08-28T19:35:26.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:26.537+0000 7f11e8765640 1 -- 172.21.15.16:0/3033823545 wait complete. 2024-08-28T19:35:27.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 20279 20.08 MB/sec warmup 63 sec latency 282.002 ms 2024-08-28T19:35:27.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:27 smithi016 ceph-mon[18556]: pgmap v787: 129 pgs: 129 active+clean; 737 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 6.0 MiB/s wr, 458 op/s 2024-08-28T19:35:27.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:27 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:35:27.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:27 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:35:27.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:27 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:35:27.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:27 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:27.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:27 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:27.715 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:35:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:27 smithi028 ceph-mon[21769]: pgmap v787: 129 pgs: 129 active+clean; 737 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 6.0 MiB/s wr, 458 op/s 2024-08-28T19:35:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:27 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:35:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:27 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:35:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:27 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:35:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:27 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:27 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:27.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:27 smithi115 ceph-mon[22184]: pgmap v787: 129 pgs: 129 active+clean; 737 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 6.0 MiB/s wr, 458 op/s 2024-08-28T19:35:27.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:27 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:35:27.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:27 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:35:27.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:27 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:35:27.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:27 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:27.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:27 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:28.069 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:35:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 21241 20.60 MB/sec warmup 64 sec latency 154.587 ms 2024-08-28T19:35:28.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:28 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:35:28.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:28 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:28.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:28 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:35:28.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:28 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:28.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:28 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:28 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:35:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:28 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:28 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:35:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:28 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:28 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:28 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:35:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:28 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:28 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:35:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:28 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:28 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:35:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.924+0000 7f1e8f5da640 1 -- 172.21.15.16:0/2429996789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e88105120 msgr2=0x7f1e8810d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.924+0000 7f1e8f5da640 1 --2- 172.21.15.16:0/2429996789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e88105120 0x7f1e8810d670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1e78009860 tx=0x7f1e7802a340 comp rx=0 tx=0).stop 2024-08-28T19:35:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.924+0000 7f1e8f5da640 1 -- 172.21.15.16:0/2429996789 shutdown_connections 2024-08-28T19:35:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.924+0000 7f1e8f5da640 1 --2- 172.21.15.16:0/2429996789 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e8810dc30 0x7f1e88110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.924+0000 7f1e8f5da640 1 --2- 172.21.15.16:0/2429996789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e88105120 0x7f1e8810d670 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.924+0000 7f1e8f5da640 1 --2- 172.21.15.16:0/2429996789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e881047d0 0x7f1e88104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.924+0000 7f1e8f5da640 1 -- 172.21.15.16:0/2429996789 >> 172.21.15.16:0/2429996789 conn(0x7f1e88079510 msgr2=0x7f1e88079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.924+0000 7f1e8f5da640 1 -- 172.21.15.16:0/2429996789 shutdown_connections 2024-08-28T19:35:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.924+0000 7f1e8f5da640 1 -- 172.21.15.16:0/2429996789 wait complete. 2024-08-28T19:35:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.925+0000 7f1e8f5da640 1 Processor -- start 2024-08-28T19:35:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.925+0000 7f1e8f5da640 1 -- start start 2024-08-28T19:35:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.925+0000 7f1e8f5da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e881047d0 0x7f1e8819c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.925+0000 7f1e8f5da640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e88105120 0x7f1e8819cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.925+0000 7f1e8f5da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e8810dc30 0x7f1e881a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.925+0000 7f1e8f5da640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e88113890 con 0x7f1e8810dc30 2024-08-28T19:35:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.925+0000 7f1e8f5da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e88113710 con 0x7f1e881047d0 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.925+0000 7f1e8f5da640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1e88113a10 con 0x7f1e88105120 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.925+0000 7f1e8db77640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e8810dc30 0x7f1e881a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e8d376640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e881047d0 0x7f1e8819c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e8db77640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e8810dc30 0x7f1e881a1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43850/0 (socket says 172.21.15.16:43850) 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e8db77640 1 -- 172.21.15.16:0/277429019 learned_addr learned my addr 172.21.15.16:0/277429019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e88105120 0x7f1e8819cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e8db77640 1 -- 172.21.15.16:0/277429019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e88105120 msgr2=0x7f1e8819cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e8db77640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e88105120 0x7f1e8819cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e8db77640 1 -- 172.21.15.16:0/277429019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e881047d0 msgr2=0x7f1e8819c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e8db77640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e881047d0 0x7f1e8819c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e8db77640 1 -- 172.21.15.16:0/277429019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e881a2510 con 0x7f1e8810dc30 2024-08-28T19:35:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e8db77640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e8810dc30 0x7f1e881a1de0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000eb10 tx=0x7f1e8000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e7e7fc640 1 -- 172.21.15.16:0/277429019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e80004b20 con 0x7f1e8810dc30 2024-08-28T19:35:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e8f5da640 1 -- 172.21.15.16:0/277429019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e881a2800 con 0x7f1e8810dc30 2024-08-28T19:35:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e7e7fc640 1 -- 172.21.15.16:0/277429019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e80004cc0 con 0x7f1e8810dc30 2024-08-28T19:35:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.926+0000 7f1e7e7fc640 1 -- 172.21.15.16:0/277429019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e80010670 con 0x7f1e8810dc30 2024-08-28T19:35:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.927+0000 7f1e8f5da640 1 -- 172.21.15.16:0/277429019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e881a8f60 con 0x7f1e8810dc30 2024-08-28T19:35:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.930+0000 7f1e95b4f740 1 Processor -- start 2024-08-28T19:35:28.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.932+0000 7f1e7e7fc640 1 -- 172.21.15.16:0/277429019 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1e80005020 con 0x7f1e8810dc30 2024-08-28T19:35:28.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.932+0000 7f1e95b4f740 1 -- start start 2024-08-28T19:35:28.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.932+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba90070 0x561b8be352c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.932+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba671b0 0x561b8be35800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.932+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 0x561b8be35d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.932+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561b8bdac630 con 0x561b8ba90070 2024-08-28T19:35:28.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.932+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561b8bd9bbc0 con 0x561b8ba6f390 2024-08-28T19:35:28.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.932+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561b8bd9fb70 con 0x561b8ba671b0 2024-08-28T19:35:28.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.933+0000 7f1e8cb75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 0x561b8be35d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.933+0000 7f1e8cb75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba90070 0x561b8be352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.933+0000 7f1e8cb75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba90070 0x561b8be352c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43854/0 (socket says 172.21.15.16:43854) 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.933+0000 7f1e8cb75640 1 -- 172.21.15.16:0/579931330 learned_addr learned my addr 172.21.15.16:0/579931330 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.933+0000 7f1e7e7fc640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1e740b4d20 0x7f1e740b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.933+0000 7f1e8cb75640 1 -- 172.21.15.16:0/579931330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba671b0 msgr2=0x561b8be35800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.933+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/579931330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba671b0 0x561b8be35800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.933+0000 7f1e8cb75640 1 -- 172.21.15.16:0/579931330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 msgr2=0x561b8be35d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.933+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/579931330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 0x561b8be35d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.933+0000 7f1e8cb75640 1 -- 172.21.15.16:0/579931330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b8bafb650 con 0x561b8ba90070 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.934+0000 7f1e7e7fc640 1 -- 172.21.15.16:0/277429019 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f1e800e5dc0 con 0x7f1e8810dc30 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.934+0000 7f1e8db77640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1e740b4d20 0x7f1e740b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.934+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/579931330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba90070 0x561b8be352c0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1e7802a930 tx=0x7f1e780058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a77480a7072136 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.934+0000 7f1e737fe640 1 -- 172.21.15.16:0/579931330 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e78037040 con 0x561b8ba90070 2024-08-28T19:35:28.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.934+0000 7f1e737fe640 1 -- 172.21.15.16:0/579931330 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e78048070 con 0x561b8ba90070 2024-08-28T19:35:28.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.934+0000 7f1e737fe640 1 -- 172.21.15.16:0/579931330 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e78003830 con 0x561b8ba90070 2024-08-28T19:35:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.935+0000 7f1e95b4f740 1 -- 172.21.15.16:0/579931330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba90070 msgr2=0x561b8be352c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.935+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/579931330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba90070 0x561b8be352c0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1e7802a930 tx=0x7f1e780058d0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.935+0000 7f1e8db77640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1e740b4d20 0x7f1e740b7230 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1e800a8160 tx=0x7f1e8000cf70 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.936+0000 7f1e95b4f740 1 -- 172.21.15.16:0/579931330 shutdown_connections 2024-08-28T19:35:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.936+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/579931330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 0x561b8be35d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.936+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/579931330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba671b0 0x561b8be35800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.936+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/579931330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba90070 0x561b8be352c0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.936+0000 7f1e95b4f740 1 -- 172.21.15.16:0/579931330 >> 172.21.15.16:0/579931330 conn(0x561b8ba84460 msgr2=0x561b8be331b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.936+0000 7f1e95b4f740 1 -- 172.21.15.16:0/579931330 shutdown_connections 2024-08-28T19:35:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.936+0000 7f1e95b4f740 1 -- 172.21.15.16:0/579931330 wait complete. 2024-08-28T19:35:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.937+0000 7f1e95b4f740 1 -- start start 2024-08-28T19:35:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.937+0000 7f1e95b4f740 1 Processor -- start 2024-08-28T19:35:28.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be40d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 0x561b8be41280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba90070 0x561b8be4cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561b8bbfe700 con 0x561b8ba671b0 2024-08-28T19:35:28.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561b8bd22700 con 0x561b8ba6f390 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561b8be43a40 con 0x561b8ba90070 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e95b4f740 10 client.0 _ll_register_callbacks cb 0x561b8bd51dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e8d376640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 0x561b8be41280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e8d376640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 0x561b8be41280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34848/0 (socket says 172.21.15.16:34848) 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e8d376640 1 -- 172.21.15.16:0/1455172574 learned_addr learned my addr 172.21.15.16:0/1455172574 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba90070 0x561b8be4cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be40d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e8d376640 1 -- 172.21.15.16:0/1455172574 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba90070 msgr2=0x561b8be4cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e8d376640 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba90070 0x561b8be4cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e8d376640 1 -- 172.21.15.16:0/1455172574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 msgr2=0x561b8be40d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e8d376640 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be40d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e8d376640 1 -- 172.21.15.16:0/1455172574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b8bd5b8b0 con 0x561b8ba6f390 2024-08-28T19:35:28.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.938+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba90070 0x561b8be4cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.939+0000 7f1e8d376640 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 0x561b8be41280 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1e840116c0 tx=0x7f1e84011b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13c8bd2c70d4ca4f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.939+0000 7f1e70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.940+0000 7f1e70ff9640 1 -- 172.21.15.16:0/1455172574 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400fdf0 con 0x561b8ba6f390 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.940+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1455172574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e78048070 con 0x561b8ba6f390 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.940+0000 7f1e70ff9640 1 -- 172.21.15.16:0/1455172574 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f1e84004020 con 0x561b8ba6f390 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.940+0000 7f1e70ff9640 1 -- 172.21.15.16:0/1455172574 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f1e84024050 con 0x561b8ba6f390 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.940+0000 7f1e95b4f740 10 client.28520 fetch_fsmap learned FSMap version 77 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.940+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1455172574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=4+}) -- 0x561b8be4f9c0 con 0x561b8ba6f390 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.940+0000 7f1e70ff9640 1 -- 172.21.15.16:0/1455172574 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e78048070 con 0x561b8ba6f390 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.941+0000 7f1e70ff9640 1 -- 172.21.15.16:0/1455172574 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 77) ==== 8442+0+0 (secure 0 0 0) 0x7f1e84019430 con 0x561b8ba6f390 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.941+0000 7f1e95b4f740 10 client.28520 fetch_fsmap finished waiting for FSMap version 77 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.941+0000 7f1e95b4f740 10 client.28520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:35:28.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.941+0000 7f1e95b4f740 20 client.28520 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:28.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.941+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561b8ba4e9f0 0x561b8be52e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.941+0000 7f1e95b4f740 4 client.28520 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:35:28.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.941+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1455172574 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b8bdae580 con 0x561b8ba4e9f0 2024-08-28T19:35:28.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.941+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561b8ba4e9f0 0x561b8be52e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.942+0000 7f1e70ff9640 10 client.28520 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:28.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.942+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561b8ba4e9f0 0x561b8be52e70 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7311bc8556d0eae server_cookie=2a2e49c6e122d717 in_seq=0 out_seq=0 2024-08-28T19:35:28.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e70ff9640 1 -- 172.21.15.16:0/1455172574 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561b8bdae580 con 0x561b8ba4e9f0 2024-08-28T19:35:28.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e70ff9640 10 client.28520 handle_command_reply: tid=0 2024-08-28T19:35:28.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 1 client.28520 shutdown 2024-08-28T19:35:28.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1455172574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 msgr2=0x561b8be41280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:28.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 0x561b8be41280 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1e840116c0 tx=0x7f1e84011b80 comp rx=0 tx=0).stop 2024-08-28T19:35:28.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1455172574 shutdown_connections 2024-08-28T19:35:28.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561b8ba4e9f0 0x561b8be52e70 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba90070 0x561b8be4cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba6f390 0x561b8be41280 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e70ff9640 0 client.28520 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:28.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/1455172574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be40d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1455172574 >> 172.21.15.16:0/1455172574 conn(0x561b8ba84460 msgr2=0x561b8be384a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:28.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1455172574 shutdown_connections 2024-08-28T19:35:28.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1455172574 wait complete. 2024-08-28T19:35:28.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.943+0000 7f1e95b4f740 20 client.28520 trim_cache size 0 max 16384 2024-08-28T19:35:28.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e95b4f740 1 Processor -- start 2024-08-28T19:35:28.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e95b4f740 1 -- start start 2024-08-28T19:35:28.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be38d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba6f390 0x561b8be2e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba90070 0x561b8be42770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e881aeac0 con 0x561b8ba671b0 2024-08-28T19:35:28.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e881b08a0 con 0x561b8ba90070 2024-08-28T19:35:28.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1e881af220 con 0x561b8ba6f390 2024-08-28T19:35:28.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e8d376640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba6f390 0x561b8be2e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e8d376640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba6f390 0x561b8be2e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:47060/0 (socket says 172.21.15.16:47060) 2024-08-28T19:35:28.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e8cb75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be38d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e8d376640 1 -- 172.21.15.16:0/1608882007 learned_addr learned my addr 172.21.15.16:0/1608882007 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:28.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.968+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/1608882007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba90070 0x561b8be42770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.969+0000 7f1e8d376640 1 -- 172.21.15.16:0/1608882007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba90070 msgr2=0x561b8be42770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:28.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.969+0000 7f1e8d376640 1 --2- 172.21.15.16:0/1608882007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba90070 0x561b8be42770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.969+0000 7f1e8d376640 1 -- 172.21.15.16:0/1608882007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 msgr2=0x561b8be38d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:28.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.969+0000 7f1e8d376640 1 --2- 172.21.15.16:0/1608882007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be38d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.969+0000 7f1e8d376640 1 -- 172.21.15.16:0/1608882007 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b8be3c680 con 0x561b8ba6f390 2024-08-28T19:35:28.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.969+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/1608882007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be38d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:35:28.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.969+0000 7f1e8d376640 1 --2- 172.21.15.16:0/1608882007 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba6f390 0x561b8be2e1e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1e84005f70 tx=0x7f1e84005760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14818d6d38d240d7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:28.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.969+0000 7f1e72ffd640 1 -- 172.21.15.16:0/1608882007 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8402c070 con 0x561b8ba6f390 2024-08-28T19:35:28.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.969+0000 7f1e72ffd640 1 -- 172.21.15.16:0/1608882007 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e84019430 con 0x561b8ba6f390 2024-08-28T19:35:28.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.969+0000 7f1e72ffd640 1 -- 172.21.15.16:0/1608882007 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e84031b10 con 0x561b8ba6f390 2024-08-28T19:35:28.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.970+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1608882007 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba6f390 msgr2=0x561b8be2e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:28.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.970+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/1608882007 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba6f390 0x561b8be2e1e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1e84005f70 tx=0x7f1e84005760 comp rx=0 tx=0).stop 2024-08-28T19:35:28.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.970+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1608882007 shutdown_connections 2024-08-28T19:35:28.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.970+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/1608882007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba90070 0x561b8be42770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.970+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/1608882007 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba6f390 0x561b8be2e1e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.970+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/1608882007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be38d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.970+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1608882007 >> 172.21.15.16:0/1608882007 conn(0x561b8ba84460 msgr2=0x561b8be42060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:28.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.971+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1608882007 shutdown_connections 2024-08-28T19:35:28.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.971+0000 7f1e95b4f740 1 -- 172.21.15.16:0/1608882007 wait complete. 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.971+0000 7f1e95b4f740 1 -- start start 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.971+0000 7f1e95b4f740 1 Processor -- start 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be457c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba90070 0x561b8be45d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e95b4f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba23da0 0x561b8beb69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e881b0110 con 0x561b8ba671b0 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e881af980 con 0x561b8ba23da0 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e95b4f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1e881acb40 con 0x561b8ba90070 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e95b4f740 10 client.0 _ll_register_callbacks cb 0x7f1e84009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e8cb75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e8cb75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be457c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43882/0 (socket says 172.21.15.16:43882) 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e8cb75640 1 -- 172.21.15.16:0/2016373062 learned_addr learned my addr 172.21.15.16:0/2016373062 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:28.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba23da0 0x561b8beb69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.972+0000 7f1e8d376640 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba90070 0x561b8be45d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.973+0000 7f1e8cb75640 1 -- 172.21.15.16:0/2016373062 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba90070 msgr2=0x561b8be45d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.973+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba90070 0x561b8be45d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.973+0000 7f1e8cb75640 1 -- 172.21.15.16:0/2016373062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba23da0 msgr2=0x561b8beb69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.973+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba23da0 0x561b8beb69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.973+0000 7f1e8cb75640 1 -- 172.21.15.16:0/2016373062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b8beb7040 con 0x561b8ba671b0 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.973+0000 7f1e8cb75640 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be457c0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f1e7802a900 tx=0x7f1e78000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59891427071bbc6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.973+0000 7f1e70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.973+0000 7f1e70ff9640 1 -- 172.21.15.16:0/2016373062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e78037040 con 0x561b8ba671b0 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.973+0000 7f1e70ff9640 1 -- 172.21.15.16:0/2016373062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f1e78052d20 con 0x561b8ba671b0 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.973+0000 7f1e95b4f740 1 -- 172.21.15.16:0/2016373062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e84019430 con 0x561b8ba671b0 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e70ff9640 1 -- 172.21.15.16:0/2016373062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f1e7806c050 con 0x561b8ba671b0 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e95b4f740 10 client.19095 fetch_fsmap learned FSMap version 77 2024-08-28T19:35:28.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e95b4f740 1 -- 172.21.15.16:0/2016373062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=4+}) -- 0x561b8bebe440 con 0x561b8ba671b0 2024-08-28T19:35:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e70ff9640 1 -- 172.21.15.16:0/2016373062 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e84019430 con 0x561b8ba671b0 2024-08-28T19:35:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e70ff9640 1 -- 172.21.15.16:0/2016373062 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 77) ==== 8442+0+0 (secure 0 0 0) 0x7f1e7803d070 con 0x561b8ba671b0 2024-08-28T19:35:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e95b4f740 10 client.19095 fetch_fsmap finished waiting for FSMap version 77 2024-08-28T19:35:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e95b4f740 10 client.19095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:35:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e95b4f740 20 client.19095 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561b8bebe6d0 0x561b8bec0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e95b4f740 4 client.19095 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:35:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e95b4f740 1 -- 172.21.15.16:0/2016373062 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561b8be41610 con 0x561b8bebe6d0 2024-08-28T19:35:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.974+0000 7f1e8d376640 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561b8bebe6d0 0x561b8bec0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e70ff9640 10 client.19095 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e8d376640 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561b8bebe6d0 0x561b8bec0af0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4b799f9b0e036d4 server_cookie=b8a628fbe7039770 in_seq=0 out_seq=0 2024-08-28T19:35:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e70ff9640 1 -- 172.21.15.16:0/2016373062 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+5966 (crc 0 0 0) 0x561b8be41610 con 0x561b8bebe6d0 2024-08-28T19:35:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e70ff9640 10 client.19095 handle_command_reply: tid=0 2024-08-28T19:35:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e95b4f740 1 client.19095 shutdown 2024-08-28T19:35:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e95b4f740 1 -- 172.21.15.16:0/2016373062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 msgr2=0x561b8be457c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be457c0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f1e7802a900 tx=0x7f1e78000bc0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e95b4f740 1 -- 172.21.15.16:0/2016373062 shutdown_connections 2024-08-28T19:35:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b8ba23da0 0x561b8beb69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561b8bebe6d0 0x561b8bec0af0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561b8ba90070 0x561b8be45d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e95b4f740 1 --2- 172.21.15.16:0/2016373062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b8ba671b0 0x561b8be457c0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.975+0000 7f1e70ff9640 0 client.19095 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e95b4f740 1 -- 172.21.15.16:0/2016373062 >> 172.21.15.16:0/2016373062 conn(0x561b8ba84460 msgr2=0x561b8be36cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e95b4f740 1 -- 172.21.15.16:0/2016373062 shutdown_connections 2024-08-28T19:35:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e95b4f740 1 -- 172.21.15.16:0/2016373062 wait complete. 2024-08-28T19:35:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e95b4f740 20 client.19095 trim_cache size 0 max 16384 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "840047cc-208d-4758-921b-8f80ba8e251f", 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:35:28.981 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:35:28.982 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.984 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.985 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.986 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:35:28.987 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.989 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.990 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:35:28.991 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:35:28.992 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:35:28.993 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:35:28.993 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:35:28.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e8f5da640 1 -- 172.21.15.16:0/277429019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1e740b4d20 msgr2=0x7f1e740b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:28.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e8f5da640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1e740b4d20 0x7f1e740b7230 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1e800a8160 tx=0x7f1e8000cf70 comp rx=0 tx=0).stop 2024-08-28T19:35:28.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e8f5da640 1 -- 172.21.15.16:0/277429019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e8810dc30 msgr2=0x7f1e881a1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:28.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e8f5da640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e8810dc30 0x7f1e881a1de0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000eb10 tx=0x7f1e8000c2d0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e8f5da640 1 -- 172.21.15.16:0/277429019 shutdown_connections 2024-08-28T19:35:28.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e8f5da640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e8810dc30 0x7f1e881a1de0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e8f5da640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e88105120 0x7f1e8819cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e8f5da640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1e740b4d20 0x7f1e740b7230 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e8f5da640 1 --2- 172.21.15.16:0/277429019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e881047d0 0x7f1e8819c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:28.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.976+0000 7f1e8f5da640 1 -- 172.21.15.16:0/277429019 >> 172.21.15.16:0/277429019 conn(0x7f1e88079510 msgr2=0x7f1e88100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:28.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.977+0000 7f1e8f5da640 1 -- 172.21.15.16:0/277429019 shutdown_connections 2024-08-28T19:35:28.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:28.977+0000 7f1e8f5da640 1 -- 172.21.15.16:0/277429019 wait complete. 2024-08-28T19:35:29.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 21974 20.80 MB/sec warmup 65 sec latency 1154.624 ms 2024-08-28T19:35:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:29 smithi016 ceph-mon[18556]: pgmap v788: 129 pgs: 129 active+clean; 753 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 6.6 MiB/s wr, 521 op/s 2024-08-28T19:35:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:29 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:29 smithi028 ceph-mon[21769]: pgmap v788: 129 pgs: 129 active+clean; 753 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 6.6 MiB/s wr, 521 op/s 2024-08-28T19:35:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:29 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:29 smithi115 ceph-mon[22184]: pgmap v788: 129 pgs: 129 active+clean; 753 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 6.6 MiB/s wr, 521 op/s 2024-08-28T19:35:29.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:29 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:29.914 INFO:tasks.cephfs.filesystem:scrub status for tag:840047cc-208d-4758-921b-8f80ba8e251f - {'path': '/', 'tag': '840047cc-208d-4758-921b-8f80ba8e251f', 'options': 'recursive,force'} 2024-08-28T19:35:30.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 22819 21.23 MB/sec warmup 66 sec latency 2154.662 ms 2024-08-28T19:35:31.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 23594 21.67 MB/sec warmup 67 sec latency 3154.696 ms 2024-08-28T19:35:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:31 smithi016 ceph-mon[18556]: pgmap v789: 129 pgs: 129 active+clean; 757 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 536 op/s 2024-08-28T19:35:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:31 smithi028 ceph-mon[21769]: pgmap v789: 129 pgs: 129 active+clean; 757 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 536 op/s 2024-08-28T19:35:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:31 smithi115 ceph-mon[22184]: pgmap v789: 129 pgs: 129 active+clean; 757 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 536 op/s 2024-08-28T19:35:32.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 24325 21.78 MB/sec warmup 68 sec latency 4154.728 ms 2024-08-28T19:35:33.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 24871 21.93 MB/sec warmup 69 sec latency 5154.772 ms 2024-08-28T19:35:33.672 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:33 smithi115 ceph-mon[22184]: pgmap v790: 129 pgs: 129 active+clean; 780 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.4 MiB/s wr, 650 op/s 2024-08-28T19:35:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:33 smithi016 ceph-mon[18556]: pgmap v790: 129 pgs: 129 active+clean; 780 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.4 MiB/s wr, 650 op/s 2024-08-28T19:35:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:33 smithi028 ceph-mon[21769]: pgmap v790: 129 pgs: 129 active+clean; 780 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.4 MiB/s wr, 650 op/s 2024-08-28T19:35:34.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 24972 21.64 MB/sec warmup 70 sec latency 6071.966 ms 2024-08-28T19:35:34.696 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:34 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:35:34.696 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:34 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:35:34.696 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:34 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:35:34.696 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:34 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:35:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:34 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:35:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:34 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:35:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:34 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:35:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:34 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:35:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:34 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:35:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:34 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:35:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:34 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:35:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:34 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:35:35.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 25366 21.63 MB/sec warmup 71 sec latency 7072.006 ms 2024-08-28T19:35:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:35 smithi016 ceph-mon[18556]: pgmap v791: 129 pgs: 129 active+clean; 784 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.7 MiB/s wr, 536 op/s 2024-08-28T19:35:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:35 smithi028 ceph-mon[21769]: pgmap v791: 129 pgs: 129 active+clean; 784 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.7 MiB/s wr, 536 op/s 2024-08-28T19:35:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:35 smithi115 ceph-mon[22184]: pgmap v791: 129 pgs: 129 active+clean; 784 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.7 MiB/s wr, 536 op/s 2024-08-28T19:35:36.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 26081 21.98 MB/sec warmup 72 sec latency 8072.045 ms 2024-08-28T19:35:37.463 INFO:tasks.workunit.client.0.smithi016.stdout: 10 27028 22.35 MB/sec warmup 73 sec latency 9072.084 ms 2024-08-28T19:35:37.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:37 smithi016 ceph-mon[18556]: pgmap v792: 129 pgs: 129 active+clean; 847 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 745 op/s 2024-08-28T19:35:37.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:37 smithi028 ceph-mon[21769]: pgmap v792: 129 pgs: 129 active+clean; 847 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 745 op/s 2024-08-28T19:35:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:37 smithi115 ceph-mon[22184]: pgmap v792: 129 pgs: 129 active+clean; 847 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 745 op/s 2024-08-28T19:35:38.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 27691 22.60 MB/sec warmup 74 sec latency 10072.121 ms 2024-08-28T19:35:39.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 27886 22.43 MB/sec warmup 75 sec latency 10595.778 ms 2024-08-28T19:35:39.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:39 smithi016 ceph-mon[18556]: pgmap v793: 129 pgs: 129 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 771 op/s 2024-08-28T19:35:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:39 smithi028 ceph-mon[21769]: pgmap v793: 129 pgs: 129 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 771 op/s 2024-08-28T19:35:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:39 smithi115 ceph-mon[22184]: pgmap v793: 129 pgs: 129 active+clean; 860 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 771 op/s 2024-08-28T19:35:40.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 28372 22.43 MB/sec warmup 76 sec latency 1091.553 ms 2024-08-28T19:35:41.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 29111 22.66 MB/sec warmup 77 sec latency 78.714 ms 2024-08-28T19:35:41.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:41 smithi016 ceph-mon[18556]: pgmap v794: 129 pgs: 129 active+clean; 854 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 837 op/s 2024-08-28T19:35:41.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:41 smithi028 ceph-mon[21769]: pgmap v794: 129 pgs: 129 active+clean; 854 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 837 op/s 2024-08-28T19:35:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:41 smithi115 ceph-mon[22184]: pgmap v794: 129 pgs: 129 active+clean; 854 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 837 op/s 2024-08-28T19:35:42.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 30007 22.93 MB/sec warmup 78 sec latency 59.822 ms 2024-08-28T19:35:43.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 30962 23.28 MB/sec warmup 79 sec latency 68.120 ms 2024-08-28T19:35:43.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:43 smithi016 ceph-mon[18556]: pgmap v795: 129 pgs: 129 active+clean; 854 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 22 MiB/s wr, 1.04k op/s 2024-08-28T19:35:43.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:43 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:43.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:43 smithi028 ceph-mon[21769]: pgmap v795: 129 pgs: 129 active+clean; 854 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 22 MiB/s wr, 1.04k op/s 2024-08-28T19:35:43.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:43 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:43.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:43 smithi115 ceph-mon[22184]: pgmap v795: 129 pgs: 129 active+clean; 854 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 22 MiB/s wr, 1.04k op/s 2024-08-28T19:35:43.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:43 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:35:44.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 31782 23.61 MB/sec warmup 80 sec latency 85.649 ms 2024-08-28T19:35:44.822 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-08-28T19:35:44.822 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-08-28T19:35:44.823 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:35:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:35:45.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 32613 23.83 MB/sec warmup 81 sec latency 72.211 ms 2024-08-28T19:35:45.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:45 smithi016 ceph-mon[18556]: pgmap v796: 129 pgs: 129 active+clean; 855 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 919 op/s 2024-08-28T19:35:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.702+0000 7f44db770640 1 -- 172.21.15.16:0/2892683183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44cc0a4e70 msgr2=0x7f44cc0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.702+0000 7f44db770640 1 --2- 172.21.15.16:0/2892683183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44cc0a4e70 0x7f44cc0a5280 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f44d4069c20 tx=0x7f44d408f5b0 comp rx=0 tx=0).stop 2024-08-28T19:35:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.703+0000 7f44db770640 1 -- 172.21.15.16:0/2892683183 shutdown_connections 2024-08-28T19:35:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.703+0000 7f44db770640 1 --2- 172.21.15.16:0/2892683183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44cc0b7b90 0x7f44cc0b9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.703+0000 7f44db770640 1 --2- 172.21.15.16:0/2892683183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44cc0a57c0 0x7f44cc0b7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.703+0000 7f44db770640 1 --2- 172.21.15.16:0/2892683183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44cc0a4e70 0x7f44cc0a5280 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.703+0000 7f44db770640 1 -- 172.21.15.16:0/2892683183 >> 172.21.15.16:0/2892683183 conn(0x7f44cc01cc50 msgr2=0x7f44cc01d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.703+0000 7f44db770640 1 -- 172.21.15.16:0/2892683183 shutdown_connections 2024-08-28T19:35:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.703+0000 7f44db770640 1 -- 172.21.15.16:0/2892683183 wait complete. 2024-08-28T19:35:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.704+0000 7f44db770640 1 Processor -- start 2024-08-28T19:35:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.704+0000 7f44db770640 1 -- start start 2024-08-28T19:35:45.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.705+0000 7f44db770640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44cc0a4e70 0x7f44cc1440e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:45.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.705+0000 7f44db770640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44cc0a57c0 0x7f44cc144680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.705+0000 7f44db770640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44cc0b7b90 0x7f44cc149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.705+0000 7f44db770640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc0bd610 con 0x7f44cc0a57c0 2024-08-28T19:35:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.705+0000 7f44db770640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc0bd490 con 0x7f44cc0a4e70 2024-08-28T19:35:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.705+0000 7f44db770640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc0bd790 con 0x7f44cc0b7b90 2024-08-28T19:35:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.705+0000 7f44d950c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44cc0a4e70 0x7f44cc1440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.705+0000 7f44d950c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44cc0a4e70 0x7f44cc1440e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48320/0 (socket says 172.21.15.16:48320) 2024-08-28T19:35:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.705+0000 7f44d950c640 1 -- 172.21.15.16:0/3839438713 learned_addr learned my addr 172.21.15.16:0/3839438713 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.705+0000 7f44d9d0d640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44cc0b7b90 0x7f44cc149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.706+0000 7f44d8d0b640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44cc0a57c0 0x7f44cc144680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.706+0000 7f44d9d0d640 1 -- 172.21.15.16:0/3839438713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44cc0a4e70 msgr2=0x7f44cc1440e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.706+0000 7f44d9d0d640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44cc0a4e70 0x7f44cc1440e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.706+0000 7f44d9d0d640 1 -- 172.21.15.16:0/3839438713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44cc0a57c0 msgr2=0x7f44cc144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:45.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.706+0000 7f44d9d0d640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44cc0a57c0 0x7f44cc144680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:45.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.706+0000 7f44d9d0d640 1 -- 172.21.15.16:0/3839438713 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44cc14a050 con 0x7f44cc0b7b90 2024-08-28T19:35:45.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.706+0000 7f44d8d0b640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44cc0a57c0 0x7f44cc144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:35:45.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.707+0000 7f44d9d0d640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44cc0b7b90 0x7f44cc149920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009fd0 tx=0x7f44d000eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.708+0000 7f44ca7fc640 1 -- 172.21.15.16:0/3839438713 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d0004220 con 0x7f44cc0b7b90 2024-08-28T19:35:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.708+0000 7f44ca7fc640 1 -- 172.21.15.16:0/3839438713 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d00043c0 con 0x7f44cc0b7b90 2024-08-28T19:35:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.708+0000 7f44ca7fc640 1 -- 172.21.15.16:0/3839438713 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d0005650 con 0x7f44cc0b7b90 2024-08-28T19:35:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.709+0000 7f44db770640 1 -- 172.21.15.16:0/3839438713 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44cc14a340 con 0x7f44cc0b7b90 2024-08-28T19:35:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.709+0000 7f44db770640 1 -- 172.21.15.16:0/3839438713 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44cc150aa0 con 0x7f44cc0b7b90 2024-08-28T19:35:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.710+0000 7f44db770640 1 -- 172.21.15.16:0/3839438713 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44cc0a5280 con 0x7f44cc0b7b90 2024-08-28T19:35:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.713+0000 7f44ca7fc640 1 -- 172.21.15.16:0/3839438713 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f44d0004a00 con 0x7f44cc0b7b90 2024-08-28T19:35:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.714+0000 7f44ca7fc640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44c00b4e30 0x7f44c00b7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.715+0000 7f44d950c640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44c00b4e30 0x7f44c00b7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:45.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.716+0000 7f44ca7fc640 1 -- 172.21.15.16:0/3839438713 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f44d00e57a0 con 0x7f44cc0b7b90 2024-08-28T19:35:45.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.716+0000 7f44d950c640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44c00b4e30 0x7f44c00b7340 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f44d402e9c0 tx=0x7f44d4069310 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:45.719+0000 7f44ca7fc640 1 -- 172.21.15.16:0/3839438713 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f44d00ac2d0 con 0x7f44cc0b7b90 2024-08-28T19:35:45.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:45 smithi028 ceph-mon[21769]: pgmap v796: 129 pgs: 129 active+clean; 855 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 919 op/s 2024-08-28T19:35:45.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:45 smithi115 ceph-mon[22184]: pgmap v796: 129 pgs: 129 active+clean; 855 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 919 op/s 2024-08-28T19:35:46.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.044+0000 7f44db770640 1 -- 172.21.15.16:0/3839438713 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44cc145210 con 0x7f44cc0b7b90 2024-08-28T19:35:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.045+0000 7f44ca7fc640 1 -- 172.21.15.16:0/3839438713 <== mon.1 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) ==== 94+0+12010 (secure 0 0 0) 0x7f44cc145210 con 0x7f44cc0b7b90 2024-08-28T19:35:46.111 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:35:46.111 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"btime":"2024-08-28T19:35:42:414805+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":78,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:35:41.973464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:35:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 78 2024-08-28T19:35:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 -- 172.21.15.16:0/3839438713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44c00b4e30 msgr2=0x7f44c00b7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44c00b4e30 0x7f44c00b7340 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f44d402e9c0 tx=0x7f44d4069310 comp rx=0 tx=0).stop 2024-08-28T19:35:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 -- 172.21.15.16:0/3839438713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44cc0b7b90 msgr2=0x7f44cc149920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44cc0b7b90 0x7f44cc149920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009fd0 tx=0x7f44d000eba0 comp rx=0 tx=0).stop 2024-08-28T19:35:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 -- 172.21.15.16:0/3839438713 shutdown_connections 2024-08-28T19:35:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44cc0b7b90 0x7f44cc149920 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44cc0a57c0 0x7f44cc144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44c00b4e30 0x7f44c00b7340 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 --2- 172.21.15.16:0/3839438713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44cc0a4e70 0x7f44cc1440e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 -- 172.21.15.16:0/3839438713 >> 172.21.15.16:0/3839438713 conn(0x7f44cc01cc50 msgr2=0x7f44cc0a7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:46.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 -- 172.21.15.16:0/3839438713 shutdown_connections 2024-08-28T19:35:46.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:46.048+0000 7f44db770640 1 -- 172.21.15.16:0/3839438713 wait complete. 2024-08-28T19:35:46.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 33378 24.12 MB/sec warmup 82 sec latency 105.228 ms 2024-08-28T19:35:46.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:46 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3839438713' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:35:46.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:46 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3839438713' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:35:46.865 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-08-28T19:35:46.865 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-08-28T19:35:46.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:46 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3839438713' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:35:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:35:47.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34236 24.25 MB/sec warmup 83 sec latency 129.236 ms 2024-08-28T19:35:47.696 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:47 smithi016 ceph-mon[18556]: pgmap v797: 129 pgs: 129 active+clean; 840 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 22 MiB/s wr, 1.07k op/s 2024-08-28T19:35:47.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:47 smithi028 ceph-mon[21769]: pgmap v797: 129 pgs: 129 active+clean; 840 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 22 MiB/s wr, 1.07k op/s 2024-08-28T19:35:47.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.833+0000 7f0f984b4640 1 -- 172.21.15.16:0/4264975426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f90102560 msgr2=0x7f0f90102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:47.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.833+0000 7f0f984b4640 1 --2- 172.21.15.16:0/4264975426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f90102560 0x7f0f90102970 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009930 tx=0x7f0f8c02a440 comp rx=0 tx=0).stop 2024-08-28T19:35:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.833+0000 7f0f984b4640 1 -- 172.21.15.16:0/4264975426 shutdown_connections 2024-08-28T19:35:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.833+0000 7f0f984b4640 1 --2- 172.21.15.16:0/4264975426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f901079a0 0x7f0f90107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.833+0000 7f0f984b4640 1 --2- 172.21.15.16:0/4264975426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f90102eb0 0x7f0f901073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.833+0000 7f0f984b4640 1 --2- 172.21.15.16:0/4264975426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f90102560 0x7f0f90102970 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.833+0000 7f0f984b4640 1 -- 172.21.15.16:0/4264975426 >> 172.21.15.16:0/4264975426 conn(0x7f0f90079560 msgr2=0x7f0f90079970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.833+0000 7f0f984b4640 1 -- 172.21.15.16:0/4264975426 shutdown_connections 2024-08-28T19:35:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.833+0000 7f0f984b4640 1 -- 172.21.15.16:0/4264975426 wait complete. 2024-08-28T19:35:47.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.834+0000 7f0f984b4640 1 Processor -- start 2024-08-28T19:35:47.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.834+0000 7f0f984b4640 1 -- start start 2024-08-28T19:35:47.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.834+0000 7f0f984b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f90102560 0x7f0f901a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.834+0000 7f0f984b4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f90102eb0 0x7f0f901a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.834+0000 7f0f984b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901079a0 0x7f0f901a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.834+0000 7f0f984b4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f90119ea0 con 0x7f0f901079a0 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.834+0000 7f0f984b4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f90119d20 con 0x7f0f90102560 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.834+0000 7f0f984b4640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9011a020 con 0x7f0f90102eb0 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.835+0000 7f0f96250640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f90102560 0x7f0f901a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.835+0000 7f0f96a51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901079a0 0x7f0f901a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.835+0000 7f0f96250640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f90102560 0x7f0f901a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48338/0 (socket says 172.21.15.16:48338) 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.835+0000 7f0f96250640 1 -- 172.21.15.16:0/3923332243 learned_addr learned my addr 172.21.15.16:0/3923332243 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.835+0000 7f0f95a4f640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f90102eb0 0x7f0f901a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.835+0000 7f0f96250640 1 -- 172.21.15.16:0/3923332243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f90102eb0 msgr2=0x7f0f901a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.835+0000 7f0f96250640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f90102eb0 0x7f0f901a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.835+0000 7f0f96250640 1 -- 172.21.15.16:0/3923332243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901079a0 msgr2=0x7f0f901a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.835+0000 7f0f96250640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901079a0 0x7f0f901a60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.835+0000 7f0f96250640 1 -- 172.21.15.16:0/3923332243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f901a6820 con 0x7f0f90102560 2024-08-28T19:35:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.835+0000 7f0f96250640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f90102560 0x7f0f901a08b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c005940 tx=0x7f0f8c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:47.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.836+0000 7f0f877fe640 1 -- 172.21.15.16:0/3923332243 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c03f030 con 0x7f0f90102560 2024-08-28T19:35:47.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.836+0000 7f0f984b4640 1 -- 172.21.15.16:0/3923332243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f901a6ab0 con 0x7f0f90102560 2024-08-28T19:35:47.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.836+0000 7f0f877fe640 1 -- 172.21.15.16:0/3923332243 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f8c003cb0 con 0x7f0f90102560 2024-08-28T19:35:47.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.836+0000 7f0f984b4640 1 -- 172.21.15.16:0/3923332243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f901ad270 con 0x7f0f90102560 2024-08-28T19:35:47.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.836+0000 7f0f877fe640 1 -- 172.21.15.16:0/3923332243 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c035070 con 0x7f0f90102560 2024-08-28T19:35:47.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.840+0000 7f0f9ea29740 1 Processor -- start 2024-08-28T19:35:47.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.840+0000 7f0f9ea29740 1 -- start start 2024-08-28T19:35:47.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669475840 0x55e66981cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 0x55e66981d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 0x55e66981d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e669791330 con 0x55e669475840 2024-08-28T19:35:47.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e6697808c0 con 0x55e669454b60 2024-08-28T19:35:47.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e669784870 con 0x55e66944c980 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f95a4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 0x55e66981d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f95a4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 0x55e66981d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48344/0 (socket says 172.21.15.16:48344) 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f95a4f640 1 -- 172.21.15.16:0/2329519161 learned_addr learned my addr 172.21.15.16:0/2329519161 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f96a51640 1 --2- 172.21.15.16:0/2329519161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669475840 0x55e66981cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f95a4f640 1 -- 172.21.15.16:0/2329519161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 msgr2=0x55e66981d1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f95a4f640 1 --2- 172.21.15.16:0/2329519161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 0x55e66981d1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f95a4f640 1 -- 172.21.15.16:0/2329519161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669475840 msgr2=0x55e66981cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f95a4f640 1 --2- 172.21.15.16:0/2329519161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669475840 0x55e66981cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f95a4f640 1 -- 172.21.15.16:0/2329519161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e669738df0 con 0x55e669454b60 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.841+0000 7f0f877fe640 1 -- 172.21.15.16:0/3923332243 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0f8c042030 con 0x7f0f90102560 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.842+0000 7f0f95a4f640 1 --2- 172.21.15.16:0/2329519161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 0x55e66981d710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0f800099d0 tx=0x7f0f80009e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d09cb5407e27a5f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.842+0000 7f0f84d9a640 1 -- 172.21.15.16:0/2329519161 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f80017070 con 0x55e669454b60 2024-08-28T19:35:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.842+0000 7f0f84d9a640 1 -- 172.21.15.16:0/2329519161 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f80013470 con 0x55e669454b60 2024-08-28T19:35:47.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.842+0000 7f0f84d9a640 1 -- 172.21.15.16:0/2329519161 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f80012630 con 0x55e669454b60 2024-08-28T19:35:47.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f877fe640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f7c0b4d20 0x7f0f7c0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f96a51640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f7c0b4d20 0x7f0f7c0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f877fe640 1 -- 172.21.15.16:0/3923332243 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f0f8c106060 con 0x7f0f90102560 2024-08-28T19:35:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f9ea29740 1 -- 172.21.15.16:0/2329519161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 msgr2=0x55e66981d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/2329519161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 0x55e66981d710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0f800099d0 tx=0x7f0f80009e90 comp rx=0 tx=0).stop 2024-08-28T19:35:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f9ea29740 1 -- 172.21.15.16:0/2329519161 shutdown_connections 2024-08-28T19:35:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/2329519161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 0x55e66981d710 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/2329519161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 0x55e66981d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/2329519161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669475840 0x55e66981cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f9ea29740 1 -- 172.21.15.16:0/2329519161 >> 172.21.15.16:0/2329519161 conn(0x55e669469990 msgr2=0x55e66981ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f96a51640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f7c0b4d20 0x7f0f7c0b7230 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0f88005fd0 tx=0x7f0f88009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f9ea29740 1 -- 172.21.15.16:0/2329519161 shutdown_connections 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.843+0000 7f0f9ea29740 1 -- 172.21.15.16:0/2329519161 wait complete. 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.844+0000 7f0f9ea29740 1 -- start start 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.844+0000 7f0f9ea29740 1 Processor -- start 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.844+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 0x55e6698284b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.844+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669454b60 0x55e6698289f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.844+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669475840 0x55e6698347e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.844+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e6695e37a0 con 0x55e669454b60 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e669707240 con 0x55e669475840 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f96a51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669475840 0x55e6698347e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e66982b4c0 con 0x55e66944c980 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f95a4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669454b60 0x55e6698289f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f96a51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669475840 0x55e6698347e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48350/0 (socket says 172.21.15.16:48350) 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f96a51640 1 -- 172.21.15.16:0/1239123578 learned_addr learned my addr 172.21.15.16:0/1239123578 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f95a4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669454b60 0x55e6698289f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55544/0 (socket says 172.21.15.16:55544) 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f9ea29740 10 client.0 _ll_register_callbacks cb 0x55e66948d090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f96a51640 1 -- 172.21.15.16:0/1239123578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 msgr2=0x55e6698284b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f96a51640 1 --2- 172.21.15.16:0/1239123578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 0x55e6698284b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f96a51640 1 -- 172.21.15.16:0/1239123578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669454b60 msgr2=0x55e6698289f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f96a51640 1 --2- 172.21.15.16:0/1239123578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669454b60 0x55e6698289f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f96a51640 1 -- 172.21.15.16:0/1239123578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e66973d2b0 con 0x55e669475840 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f9ea29740 1 -- 172.21.15.16:0/1239123578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f80013470 con 0x55e669475840 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f96a51640 1 --2- 172.21.15.16:0/1239123578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669475840 0x55e6698347e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0f88003fc0 tx=0x7f0f88004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2e502316656457 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.845+0000 7f0f79ffb640 10 client.28544 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f79ffb640 1 -- 172.21.15.16:0/1239123578 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8800f970 con 0x55e669475840 2024-08-28T19:35:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f79ffb640 1 -- 172.21.15.16:0/1239123578 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f0f8800fb10 con 0x55e669475840 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f79ffb640 1 -- 172.21.15.16:0/1239123578 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f0f8800fd10 con 0x55e669475840 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f9ea29740 10 client.28544 fetch_fsmap learned FSMap version 78 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f9ea29740 1 -- 172.21.15.16:0/1239123578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=78}) -- 0x55e6694efb90 con 0x55e669475840 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f79ffb640 1 -- 172.21.15.16:0/1239123578 <== mon.2 v2:172.21.15.28:3300/0 4 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f0f8801e470 con 0x55e669475840 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f9ea29740 10 client.28544 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f9ea29740 10 client.28544 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f9ea29740 20 client.28544 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/1239123578 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6694341c0 0x55e66983a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f9ea29740 4 client.28544 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.846+0000 7f0f9ea29740 1 -- 172.21.15.16:0/1239123578 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e669793280 con 0x55e6694341c0 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.847+0000 7f0f96250640 1 --2- 172.21.15.16:0/1239123578 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6694341c0 0x55e66983a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.847+0000 7f0f79ffb640 10 client.28544 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.847+0000 7f0f96250640 1 --2- 172.21.15.16:0/1239123578 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6694341c0 0x55e66983a530 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b20b2a84adfedb9e server_cookie=42ea56a5c2150793 in_seq=0 out_seq=0 2024-08-28T19:35:47.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.848+0000 7f0f79ffb640 1 -- 172.21.15.16:0/1239123578 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e669793280 con 0x55e6694341c0 2024-08-28T19:35:47.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.848+0000 7f0f79ffb640 10 client.28544 handle_command_reply: tid=0 2024-08-28T19:35:47.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.848+0000 7f0f9ea29740 1 client.28544 shutdown 2024-08-28T19:35:47.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.848+0000 7f0f9ea29740 1 -- 172.21.15.16:0/1239123578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669475840 msgr2=0x55e6698347e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:47.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.848+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/1239123578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669475840 0x55e6698347e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0f88003fc0 tx=0x7f0f88004a40 comp rx=0 tx=0).stop 2024-08-28T19:35:47.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.849+0000 7f0f9ea29740 1 -- 172.21.15.16:0/1239123578 shutdown_connections 2024-08-28T19:35:47.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.849+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/1239123578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669475840 0x55e6698347e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.849+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/1239123578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669454b60 0x55e6698289f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.849+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/1239123578 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6694341c0 0x55e66983a530 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.849+0000 7f0f79ffb640 0 client.28544 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:47.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.849+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/1239123578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 0x55e6698284b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.849+0000 7f0f9ea29740 1 -- 172.21.15.16:0/1239123578 >> 172.21.15.16:0/1239123578 conn(0x55e669469990 msgr2=0x55e669819340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:47.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.849+0000 7f0f9ea29740 1 -- 172.21.15.16:0/1239123578 shutdown_connections 2024-08-28T19:35:47.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.849+0000 7f0f9ea29740 1 -- 172.21.15.16:0/1239123578 wait complete. 2024-08-28T19:35:47.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.849+0000 7f0f9ea29740 20 client.28544 trim_cache size 0 max 16384 2024-08-28T19:35:47.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:47 smithi115 ceph-mon[22184]: pgmap v797: 129 pgs: 129 active+clean; 840 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 22 MiB/s wr, 1.07k op/s 2024-08-28T19:35:47.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.872+0000 7f0f9ea29740 1 Processor -- start 2024-08-28T19:35:47.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.872+0000 7f0f9ea29740 1 -- start start 2024-08-28T19:35:47.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6694341c0 0x55e669819d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e66944c980 0x55e6698346e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669454b60 0x55e6698256a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901aead0 con 0x55e669454b60 2024-08-28T19:35:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901b08e0 con 0x55e66944c980 2024-08-28T19:35:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901af230 con 0x55e6694341c0 2024-08-28T19:35:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f95a4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e66944c980 0x55e6698346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f95a4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e66944c980 0x55e6698346e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:48354/0 (socket says 172.21.15.16:48354) 2024-08-28T19:35:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f95a4f640 1 -- 172.21.15.16:0/854286415 learned_addr learned my addr 172.21.15.16:0/854286415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f96a51640 1 --2- 172.21.15.16:0/854286415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669454b60 0x55e6698256a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f96250640 1 --2- 172.21.15.16:0/854286415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6694341c0 0x55e669819d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f95a4f640 1 -- 172.21.15.16:0/854286415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6694341c0 msgr2=0x55e669819d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f95a4f640 1 --2- 172.21.15.16:0/854286415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6694341c0 0x55e669819d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f95a4f640 1 -- 172.21.15.16:0/854286415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669454b60 msgr2=0x55e6698256a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f95a4f640 1 --2- 172.21.15.16:0/854286415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669454b60 0x55e6698256a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.873+0000 7f0f95a4f640 1 -- 172.21.15.16:0/854286415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e669750d90 con 0x55e66944c980 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.874+0000 7f0f96250640 1 --2- 172.21.15.16:0/854286415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6694341c0 0x55e669819d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.874+0000 7f0f95a4f640 1 --2- 172.21.15.16:0/854286415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e66944c980 0x55e6698346e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0f80013d90 tx=0x7f0f800047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1baf2d9ba4fa6125 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.874+0000 7f0f7bfff640 1 -- 172.21.15.16:0/854286415 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f80017040 con 0x55e66944c980 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.874+0000 7f0f7bfff640 1 -- 172.21.15.16:0/854286415 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f80013ad0 con 0x55e66944c980 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.874+0000 7f0f7bfff640 1 -- 172.21.15.16:0/854286415 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8001cb60 con 0x55e66944c980 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.875+0000 7f0f9ea29740 1 -- 172.21.15.16:0/854286415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e66944c980 msgr2=0x55e6698346e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.875+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/854286415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e66944c980 0x55e6698346e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0f80013d90 tx=0x7f0f800047d0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.875+0000 7f0f9ea29740 1 -- 172.21.15.16:0/854286415 shutdown_connections 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.875+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/854286415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e669454b60 0x55e6698256a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.875+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/854286415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e66944c980 0x55e6698346e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.875+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/854286415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6694341c0 0x55e669819d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.875+0000 7f0f9ea29740 1 -- 172.21.15.16:0/854286415 >> 172.21.15.16:0/854286415 conn(0x55e669469990 msgr2=0x55e6698288d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.875+0000 7f0f9ea29740 1 -- 172.21.15.16:0/854286415 shutdown_connections 2024-08-28T19:35:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.876+0000 7f0f9ea29740 1 -- 172.21.15.16:0/854286415 wait complete. 2024-08-28T19:35:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.876+0000 7f0f9ea29740 1 -- start start 2024-08-28T19:35:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.876+0000 7f0f9ea29740 1 Processor -- start 2024-08-28T19:35:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6694341c0 0x55e669830d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 0x55e66982e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f9ea29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 0x55e66982ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901b0150 con 0x55e6694341c0 2024-08-28T19:35:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901af9c0 con 0x55e669454b60 2024-08-28T19:35:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f9ea29740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9006bf40 con 0x55e66944c980 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f9ea29740 10 client.0 _ll_register_callbacks cb 0x7f0f88009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f96250640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6694341c0 0x55e669830d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f96250640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6694341c0 0x55e669830d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55562/0 (socket says 172.21.15.16:55562) 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f96250640 1 -- 172.21.15.16:0/3607760106 learned_addr learned my addr 172.21.15.16:0/3607760106 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f96a51640 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 0x55e66982ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f95a4f640 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 0x55e66982e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f96250640 1 -- 172.21.15.16:0/3607760106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 msgr2=0x55e66982e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f96250640 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 0x55e66982e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f96250640 1 -- 172.21.15.16:0/3607760106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 msgr2=0x55e66982ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f96250640 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 0x55e66982ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.877+0000 7f0f96250640 1 -- 172.21.15.16:0/3607760106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e66982f290 con 0x55e6694341c0 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.878+0000 7f0f96250640 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6694341c0 0x55e669830d50 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c031a70 tx=0x7f0f8c0095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2071c2399aef7f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:35:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.878+0000 7f0f79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:35:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.878+0000 7f0f79ffb640 1 -- 172.21.15.16:0/3607760106 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c049040 con 0x55e6694341c0 2024-08-28T19:35:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.878+0000 7f0f9ea29740 1 -- 172.21.15.16:0/3607760106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f80013ad0 con 0x55e6694341c0 2024-08-28T19:35:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.878+0000 7f0f79ffb640 1 -- 172.21.15.16:0/3607760106 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f0f8c0c8610 con 0x55e6694341c0 2024-08-28T19:35:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.878+0000 7f0f79ffb640 1 -- 172.21.15.16:0/3607760106 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f0f8c0b3040 con 0x55e6694341c0 2024-08-28T19:35:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.878+0000 7f0f9ea29740 10 client.19119 fetch_fsmap learned FSMap version 78 2024-08-28T19:35:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.878+0000 7f0f9ea29740 1 -- 172.21.15.16:0/3607760106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55e6698a5d30 con 0x55e6694341c0 2024-08-28T19:35:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.879+0000 7f0f79ffb640 1 -- 172.21.15.16:0/3607760106 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f80013ad0 con 0x55e6694341c0 2024-08-28T19:35:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.879+0000 7f0f79ffb640 1 -- 172.21.15.16:0/3607760106 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f0f8c0cc5e0 con 0x55e6694341c0 2024-08-28T19:35:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.879+0000 7f0f9ea29740 10 client.19119 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:35:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.879+0000 7f0f9ea29740 10 client.19119 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:35:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.879+0000 7f0f9ea29740 20 client.19119 populate_metadata read hostname 'smithi016' 2024-08-28T19:35:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.879+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e669475840 0x55e6698a8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:35:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.879+0000 7f0f9ea29740 4 client.19119 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-08-28T19:35:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.879+0000 7f0f9ea29740 1 -- 172.21.15.16:0/3607760106 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x55e66978ed20 con 0x55e669475840 2024-08-28T19:35:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.880+0000 7f0f95a4f640 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e669475840 0x55e6698a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:35:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.880+0000 7f0f79ffb640 10 client.19119 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:47.880+0000 7f0f95a4f640 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e669475840 0x55e6698a8590 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9ed6bacdcbe06a0 server_cookie=1f28445c9c9fbb7f in_seq=0 out_seq=0 2024-08-28T19:35:48.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.331+0000 7f0f79ffb640 1 -- 172.21.15.16:0/3607760106 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e66978ed20 con 0x55e669475840 2024-08-28T19:35:48.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.331+0000 7f0f79ffb640 10 client.19119 handle_command_reply: tid=0 2024-08-28T19:35:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.331+0000 7f0f9ea29740 1 client.19119 shutdown 2024-08-28T19:35:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.331+0000 7f0f9ea29740 1 -- 172.21.15.16:0/3607760106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6694341c0 msgr2=0x55e669830d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.331+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6694341c0 0x55e669830d50 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c031a70 tx=0x7f0f8c0095d0 comp rx=0 tx=0).stop 2024-08-28T19:35:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.331+0000 7f0f9ea29740 1 -- 172.21.15.16:0/3607760106 shutdown_connections 2024-08-28T19:35:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.331+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e669454b60 0x55e66982ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.332+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e669475840 0x55e6698a8590 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.332+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e66944c980 0x55e66982e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.332+0000 7f0f79ffb640 0 client.19119 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:35:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.332+0000 7f0f9ea29740 1 --2- 172.21.15.16:0/3607760106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6694341c0 0x55e669830d50 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.332+0000 7f0f9ea29740 1 -- 172.21.15.16:0/3607760106 >> 172.21.15.16:0/3607760106 conn(0x55e669469990 msgr2=0x55e66981dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:48.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.332+0000 7f0f9ea29740 1 -- 172.21.15.16:0/3607760106 shutdown_connections 2024-08-28T19:35:48.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.332+0000 7f0f9ea29740 1 -- 172.21.15.16:0/3607760106 wait complete. 2024-08-28T19:35:48.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.332+0000 7f0f9ea29740 20 client.19119 trim_cache size 0 max 16384 2024-08-28T19:35:48.335 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:35:48.335 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 78, 2024-08-28T19:35:48.335 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:35:48.335 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 46, 2024-08-28T19:35:48.335 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a4beb8": { 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 46, 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:35:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:35:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:35:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:35:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:35:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:35:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:35:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:35:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:35:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:35:48.337 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:35:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.333+0000 7f0f984b4640 1 -- 172.21.15.16:0/3923332243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f7c0b4d20 msgr2=0x7f0f7c0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.333+0000 7f0f984b4640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f7c0b4d20 0x7f0f7c0b7230 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0f88005fd0 tx=0x7f0f88009500 comp rx=0 tx=0).stop 2024-08-28T19:35:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.333+0000 7f0f984b4640 1 -- 172.21.15.16:0/3923332243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f90102560 msgr2=0x7f0f901a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:35:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.333+0000 7f0f984b4640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f90102560 0x7f0f901a08b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c005940 tx=0x7f0f8c0042f0 comp rx=0 tx=0).stop 2024-08-28T19:35:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.334+0000 7f0f984b4640 1 -- 172.21.15.16:0/3923332243 shutdown_connections 2024-08-28T19:35:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.334+0000 7f0f984b4640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901079a0 0x7f0f901a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.334+0000 7f0f984b4640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0f90102eb0 0x7f0f901a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:48.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.334+0000 7f0f984b4640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0f7c0b4d20 0x7f0f7c0b7230 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:48.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.334+0000 7f0f984b4640 1 --2- 172.21.15.16:0/3923332243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f90102560 0x7f0f901a08b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:35:48.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.334+0000 7f0f984b4640 1 -- 172.21.15.16:0/3923332243 >> 172.21.15.16:0/3923332243 conn(0x7f0f90079560 msgr2=0x7f0f901053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:35:48.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.334+0000 7f0f984b4640 1 -- 172.21.15.16:0/3923332243 shutdown_connections 2024-08-28T19:35:48.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:35:48.334+0000 7f0f984b4640 1 -- 172.21.15.16:0/3923332243 wait complete. 2024-08-28T19:35:48.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 24.33 MB/sec warmup 84 sec latency 554.758 ms 2024-08-28T19:35:48.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:48 smithi016 ceph-mon[18556]: pgmap v798: 129 pgs: 129 active+clean; 853 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 960 op/s 2024-08-28T19:35:48.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:48 smithi028 ceph-mon[21769]: pgmap v798: 129 pgs: 129 active+clean; 853 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 960 op/s 2024-08-28T19:35:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:48 smithi115 ceph-mon[22184]: pgmap v798: 129 pgs: 129 active+clean; 853 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 960 op/s 2024-08-28T19:35:49.310 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'd8a4beb8', quiesce took 4.5 seconds. Will release after: 25.5 2024-08-28T19:35:49.464 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 24.04 MB/sec warmup 85 sec latency 1554.810 ms 2024-08-28T19:35:50.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 23.76 MB/sec warmup 86 sec latency 2554.892 ms 2024-08-28T19:35:51.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 23.49 MB/sec warmup 87 sec latency 3554.984 ms 2024-08-28T19:35:51.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:51 smithi115 ceph-mon[22184]: pgmap v799: 129 pgs: 129 active+clean; 863 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 877 op/s 2024-08-28T19:35:51.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:51 smithi016 ceph-mon[18556]: pgmap v799: 129 pgs: 129 active+clean; 863 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 877 op/s 2024-08-28T19:35:51.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:51 smithi028 ceph-mon[21769]: pgmap v799: 129 pgs: 129 active+clean; 863 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 877 op/s 2024-08-28T19:35:52.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 23.22 MB/sec warmup 88 sec latency 4555.080 ms 2024-08-28T19:35:52.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:35:52.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:35:52.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:35:52.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:35:52.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:35:52.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:35:52.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:35:52.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:35:52.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:35:52.812 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:35:52.812 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:35:52.812 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:35:53.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 22.96 MB/sec warmup 89 sec latency 5555.173 ms 2024-08-28T19:35:53.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:53 smithi115 ceph-mon[22184]: pgmap v800: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 9.8 MiB/s wr, 766 op/s 2024-08-28T19:35:53.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:53 smithi016 ceph-mon[18556]: pgmap v800: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 9.8 MiB/s wr, 766 op/s 2024-08-28T19:35:53.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:53 smithi028 ceph-mon[21769]: pgmap v800: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 9.8 MiB/s wr, 766 op/s 2024-08-28T19:35:54.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 22.71 MB/sec warmup 90 sec latency 6555.223 ms 2024-08-28T19:35:55.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 22.46 MB/sec warmup 91 sec latency 7555.299 ms 2024-08-28T19:35:55.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:55 smithi115 ceph-mon[22184]: pgmap v801: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 535 op/s 2024-08-28T19:35:55.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:55 smithi016 ceph-mon[18556]: pgmap v801: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 535 op/s 2024-08-28T19:35:55.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:55 smithi028 ceph-mon[21769]: pgmap v801: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 535 op/s 2024-08-28T19:35:56.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 22.21 MB/sec warmup 92 sec latency 8555.395 ms 2024-08-28T19:35:57.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 21.97 MB/sec warmup 93 sec latency 9555.464 ms 2024-08-28T19:35:57.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:57 smithi016 ceph-mon[18556]: pgmap v802: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.8 MiB/s wr, 486 op/s 2024-08-28T19:35:57.611 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:57 smithi028 ceph-mon[21769]: pgmap v802: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.8 MiB/s wr, 486 op/s 2024-08-28T19:35:57.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:57 smithi115 ceph-mon[22184]: pgmap v802: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.8 MiB/s wr, 486 op/s 2024-08-28T19:35:58.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 21.74 MB/sec warmup 94 sec latency 10555.502 ms 2024-08-28T19:35:59.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 21.51 MB/sec warmup 95 sec latency 11555.542 ms 2024-08-28T19:35:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:35:59 smithi115 ceph-mon[22184]: pgmap v803: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 298 op/s 2024-08-28T19:35:59.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:35:59 smithi016 ceph-mon[18556]: pgmap v803: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 298 op/s 2024-08-28T19:35:59.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:35:59 smithi028 ceph-mon[21769]: pgmap v803: 129 pgs: 129 active+clean; 871 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 298 op/s 2024-08-28T19:35:59.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:36:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:00.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 21.29 MB/sec warmup 96 sec latency 12555.585 ms 2024-08-28T19:36:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.749+0000 7f36b306f640 1 -- 172.21.15.16:0/980404587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36ac108150 msgr2=0x7f36ac108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.749+0000 7f36b306f640 1 --2- 172.21.15.16:0/980404587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36ac108150 0x7f36ac108560 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3698009860 tx=0x7f369802a340 comp rx=0 tx=0).stop 2024-08-28T19:36:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.749+0000 7f36b306f640 1 -- 172.21.15.16:0/980404587 shutdown_connections 2024-08-28T19:36:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.749+0000 7f36b306f640 1 --2- 172.21.15.16:0/980404587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f36ac108aa0 0x7f36ac10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.749+0000 7f36b306f640 1 --2- 172.21.15.16:0/980404587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36ac108150 0x7f36ac108560 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.749+0000 7f36b306f640 1 --2- 172.21.15.16:0/980404587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ac102f20 0x7f36ac103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.749+0000 7f36b306f640 1 -- 172.21.15.16:0/980404587 >> 172.21.15.16:0/980404587 conn(0x7f36ac077a00 msgr2=0x7f36ac077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.750+0000 7f36b306f640 1 -- 172.21.15.16:0/980404587 shutdown_connections 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.750+0000 7f36b306f640 1 -- 172.21.15.16:0/980404587 wait complete. 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.750+0000 7f36b306f640 1 Processor -- start 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.750+0000 7f36b306f640 1 -- start start 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.751+0000 7f36b306f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ac102f20 0x7f36ac19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.751+0000 7f36b306f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36ac108150 0x7f36ac19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.751+0000 7f36b306f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f36ac108aa0 0x7f36ac19fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.751+0000 7f36b0e0b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ac102f20 0x7f36ac19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.751+0000 7f36b306f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac113880 con 0x7f36ac102f20 2024-08-28T19:36:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.751+0000 7f36b306f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac113700 con 0x7f36ac108150 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.751+0000 7f36b306f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac113a00 con 0x7f36ac108aa0 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.751+0000 7f36b0e0b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ac102f20 0x7f36ac19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56586/0 (socket says 172.21.15.16:56586) 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.751+0000 7f36b0e0b640 1 -- 172.21.15.16:0/709556184 learned_addr learned my addr 172.21.15.16:0/709556184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.751+0000 7f36abfff640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36ac108150 0x7f36ac19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.751+0000 7f36b160c640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f36ac108aa0 0x7f36ac19fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.752+0000 7f36abfff640 1 -- 172.21.15.16:0/709556184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f36ac108aa0 msgr2=0x7f36ac19fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.752+0000 7f36abfff640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f36ac108aa0 0x7f36ac19fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.752+0000 7f36abfff640 1 -- 172.21.15.16:0/709556184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ac102f20 msgr2=0x7f36ac19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.752+0000 7f36abfff640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ac102f20 0x7f36ac19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.752+0000 7f36abfff640 1 -- 172.21.15.16:0/709556184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36ac1a01a0 con 0x7f36ac108150 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.752+0000 7f36b160c640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f36ac108aa0 0x7f36ac19fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.752+0000 7f36abfff640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36ac108150 0x7f36ac19a7d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f369802a860 tx=0x7f3698032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.752+0000 7f36a9ffb640 1 -- 172.21.15.16:0/709556184 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3698040020 con 0x7f36ac108150 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.752+0000 7f36b306f640 1 -- 172.21.15.16:0/709556184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36ac1a0430 con 0x7f36ac108150 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.752+0000 7f36a9ffb640 1 -- 172.21.15.16:0/709556184 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3698030e10 con 0x7f36ac108150 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.752+0000 7f36a9ffb640 1 -- 172.21.15.16:0/709556184 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3698037040 con 0x7f36ac108150 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.753+0000 7f36b306f640 1 -- 172.21.15.16:0/709556184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36ac1a6bf0 con 0x7f36ac108150 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.755+0000 7f36b95e4740 1 Processor -- start 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.756+0000 7f36b95e4740 1 -- start start 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.757+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d7709070 0x55b4d7aae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.757+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e01b0 0x55b4d7aae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.757+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e8390 0x55b4d7aaed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.757+0000 7f36b95e4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b4d7a25630 con 0x55b4d76e8390 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.757+0000 7f36b95e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b4d7a14bc0 con 0x55b4d76e01b0 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.757+0000 7f36b95e4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b4d7a18b70 con 0x55b4d7709070 2024-08-28T19:36:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.757+0000 7f36b160c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d7709070 0x55b4d7aae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.757+0000 7f36b160c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d7709070 0x55b4d7aae2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:50192/0 (socket says 172.21.15.16:50192) 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.757+0000 7f36b160c640 1 -- 172.21.15.16:0/1362655749 learned_addr learned my addr 172.21.15.16:0/1362655749 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.757+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/1362655749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e01b0 0x55b4d7aae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.758+0000 7f36b160c640 1 -- 172.21.15.16:0/1362655749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e01b0 msgr2=0x55b4d7aae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.758+0000 7f36b160c640 1 --2- 172.21.15.16:0/1362655749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e01b0 0x55b4d7aae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.758+0000 7f36b160c640 1 -- 172.21.15.16:0/1362655749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e8390 msgr2=0x55b4d7aaed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.758+0000 7f36b160c640 1 --2- 172.21.15.16:0/1362655749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e8390 0x55b4d7aaed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.758+0000 7f36b160c640 1 -- 172.21.15.16:0/1362655749 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4d7774650 con 0x55b4d7709070 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.758+0000 7f36b160c640 1 --2- 172.21.15.16:0/1362655749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d7709070 0x55b4d7aae2c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f36a40117e0 tx=0x7f36a4011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9559f99836f0de1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.758+0000 7f36a9ffb640 1 -- 172.21.15.16:0/709556184 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3698043020 con 0x7f36ac108150 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.758+0000 7f3696ffd640 1 -- 172.21.15.16:0/1362655749 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a4011310 con 0x55b4d7709070 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.758+0000 7f3696ffd640 1 -- 172.21.15.16:0/1362655749 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36a4003e20 con 0x55b4d7709070 2024-08-28T19:36:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.758+0000 7f3696ffd640 1 -- 172.21.15.16:0/1362655749 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a4004560 con 0x55b4d7709070 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.759+0000 7f36a9ffb640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f369c0b4ca0 0x7f369c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.759+0000 7f36b95e4740 1 -- 172.21.15.16:0/1362655749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d7709070 msgr2=0x55b4d7aae2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.759+0000 7f36b95e4740 1 --2- 172.21.15.16:0/1362655749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d7709070 0x55b4d7aae2c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f36a40117e0 tx=0x7f36a4011ca0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.759+0000 7f36a9ffb640 1 -- 172.21.15.16:0/709556184 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f36981056d0 con 0x7f36ac108150 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.759+0000 7f36b95e4740 1 -- 172.21.15.16:0/1362655749 shutdown_connections 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.759+0000 7f36b95e4740 1 --2- 172.21.15.16:0/1362655749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e8390 0x55b4d7aaed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.759+0000 7f36b95e4740 1 --2- 172.21.15.16:0/1362655749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e01b0 0x55b4d7aae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.759+0000 7f36b95e4740 1 --2- 172.21.15.16:0/1362655749 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d7709070 0x55b4d7aae2c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.759+0000 7f36b95e4740 1 -- 172.21.15.16:0/1362655749 >> 172.21.15.16:0/1362655749 conn(0x55b4d76fd460 msgr2=0x55b4d7aac1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.759+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f369c0b4ca0 0x7f369c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.760+0000 7f36b95e4740 1 -- 172.21.15.16:0/1362655749 shutdown_connections 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.760+0000 7f36b95e4740 1 -- 172.21.15.16:0/1362655749 wait complete. 2024-08-28T19:36:01.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.760+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f369c0b4ca0 0x7f369c0b71b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x55b4d7aacf40 tx=0x7f36a0015040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.760+0000 7f36b95e4740 1 -- start start 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.761+0000 7f36b95e4740 1 Processor -- start 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.761+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e01b0 0x55b4d7ab9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e8390 0x55b4d7ab9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d7709070 0x55b4d7ac5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b95e4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b4d7877700 con 0x55b4d76e8390 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b95e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b4d799b700 con 0x55b4d7709070 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b95e4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b4d7abcb40 con 0x55b4d76e01b0 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e8390 0x55b4d7ab9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e8390 0x55b4d7ab9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56602/0 (socket says 172.21.15.16:56602) 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36abfff640 1 -- 172.21.15.16:0/2045265108 learned_addr learned my addr 172.21.15.16:0/2045265108 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e01b0 0x55b4d7ab9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b160c640 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d7709070 0x55b4d7ac5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b95e4740 10 client.0 _ll_register_callbacks cb 0x55b4d79cadc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b0e0b640 1 -- 172.21.15.16:0/2045265108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d7709070 msgr2=0x55b4d7ac5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d7709070 0x55b4d7ac5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b0e0b640 1 -- 172.21.15.16:0/2045265108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e8390 msgr2=0x55b4d7ab9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e8390 0x55b4d7ab9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.762+0000 7f36b0e0b640 1 -- 172.21.15.16:0/2045265108 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4d79d48b0 con 0x55b4d76e01b0 2024-08-28T19:36:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.763+0000 7f36abfff640 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e8390 0x55b4d7ab9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.763+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e01b0 0x55b4d7ab9a20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f36a0013fd0 tx=0x7f36a0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c219caa1ce288ec5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.764+0000 7f368bfff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:36:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.764+0000 7f368bfff640 1 -- 172.21.15.16:0/2045265108 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0009e50 con 0x55b4d76e01b0 2024-08-28T19:36:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.764+0000 7f368bfff640 1 -- 172.21.15.16:0/2045265108 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f36a0003a40 con 0x55b4d76e01b0 2024-08-28T19:36:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.764+0000 7f36b95e4740 1 -- 172.21.15.16:0/2045265108 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36a4003e20 con 0x55b4d76e01b0 2024-08-28T19:36:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.764+0000 7f368bfff640 1 -- 172.21.15.16:0/2045265108 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f36a0003c40 con 0x55b4d76e01b0 2024-08-28T19:36:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.764+0000 7f36b95e4740 10 client.28204 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.764+0000 7f36b95e4740 1 -- 172.21.15.16:0/2045265108 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55b4d7ac8930 con 0x55b4d76e01b0 2024-08-28T19:36:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.765+0000 7f368bfff640 1 -- 172.21.15.16:0/2045265108 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36a0003de0 con 0x55b4d76e01b0 2024-08-28T19:36:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.766+0000 7f368bfff640 1 -- 172.21.15.16:0/2045265108 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f36a001d520 con 0x55b4d76e01b0 2024-08-28T19:36:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.766+0000 7f36b95e4740 10 client.28204 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.766+0000 7f36b95e4740 10 client.28204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:36:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.766+0000 7f36b95e4740 20 client.28204 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.766+0000 7f36b95e4740 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b4d76c79f0 0x55b4d7acbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.766+0000 7f36b95e4740 4 client.28204 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.766+0000 7f36b95e4740 1 -- 172.21.15.16:0/2045265108 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f36a4003e20 con 0x55b4d76c79f0 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.766+0000 7f36abfff640 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b4d76c79f0 0x55b4d7acbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.767+0000 7f36abfff640 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b4d76c79f0 0x55b4d7acbe70 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0b8b4e2d9f7d934 server_cookie=57d08c5f9a1ed2c7 in_seq=0 out_seq=0 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.767+0000 7f368bfff640 10 client.28204 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.768+0000 7f368bfff640 1 -- 172.21.15.16:0/2045265108 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f36a4003e20 con 0x55b4d76c79f0 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.768+0000 7f368bfff640 10 client.28204 handle_command_reply: tid=0 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.768+0000 7f36b95e4740 1 client.28204 shutdown 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.768+0000 7f36b95e4740 1 -- 172.21.15.16:0/2045265108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e01b0 msgr2=0x55b4d7ab9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.768+0000 7f36b95e4740 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e01b0 0x55b4d7ab9a20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f36a0013fd0 tx=0x7f36a0004730 comp rx=0 tx=0).stop 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.768+0000 7f36b95e4740 1 -- 172.21.15.16:0/2045265108 shutdown_connections 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.768+0000 7f36b95e4740 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d7709070 0x55b4d7ac5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.768+0000 7f36b95e4740 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e8390 0x55b4d7ab9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.768+0000 7f36b95e4740 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b4d76c79f0 0x55b4d7acbe70 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.769+0000 7f36b95e4740 1 --2- 172.21.15.16:0/2045265108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e01b0 0x55b4d7ab9a20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.769+0000 7f36b95e4740 1 -- 172.21.15.16:0/2045265108 >> 172.21.15.16:0/2045265108 conn(0x55b4d76fd460 msgr2=0x55b4d7ab1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.769+0000 7f368bfff640 0 client.28204 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.769+0000 7f36b95e4740 1 -- 172.21.15.16:0/2045265108 shutdown_connections 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.769+0000 7f36b95e4740 1 -- 172.21.15.16:0/2045265108 wait complete. 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.769+0000 7f36b95e4740 20 client.28204 trim_cache size 0 max 16384 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.791+0000 7f36b95e4740 1 Processor -- start 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.791+0000 7f36b95e4740 1 -- start start 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.791+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e01b0 0x55b4d7abb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.791+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e8390 0x55b4d7ac5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d7709070 0x55b4d7ab6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b95e4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1aeaf0 con 0x55b4d76e01b0 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b95e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1b0900 con 0x55b4d7709070 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b95e4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1af250 con 0x55b4d76e8390 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b0e0b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e01b0 0x55b4d7abb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b0e0b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e01b0 0x55b4d7abb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56614/0 (socket says 172.21.15.16:56614) 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b0e0b640 1 -- 172.21.15.16:0/1369629573 learned_addr learned my addr 172.21.15.16:0/1369629573 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b160c640 1 --2- 172.21.15.16:0/1369629573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d7709070 0x55b4d7ab6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36abfff640 1 --2- 172.21.15.16:0/1369629573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e8390 0x55b4d7ac5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b0e0b640 1 -- 172.21.15.16:0/1369629573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e8390 msgr2=0x55b4d7ac5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/1369629573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e8390 0x55b4d7ac5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b0e0b640 1 -- 172.21.15.16:0/1369629573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d7709070 msgr2=0x55b4d7ab6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/1369629573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d7709070 0x55b4d7ab6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b0e0b640 1 -- 172.21.15.16:0/1369629573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4d7ab05b0 con 0x55b4d76e01b0 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/1369629573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e01b0 0x55b4d7abb650 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009740 tx=0x7f36a00096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7683400a6bb7a458 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36967fc640 1 -- 172.21.15.16:0/1369629573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0009e50 con 0x55b4d76e01b0 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36967fc640 1 -- 172.21.15.16:0/1369629573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36a0003de0 con 0x55b4d76e01b0 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.792+0000 7f36967fc640 1 -- 172.21.15.16:0/1369629573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a001d400 con 0x55b4d76e01b0 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.793+0000 7f36b95e4740 1 -- 172.21.15.16:0/1369629573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e01b0 msgr2=0x55b4d7abb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.793+0000 7f36b95e4740 1 --2- 172.21.15.16:0/1369629573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e01b0 0x55b4d7abb650 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009740 tx=0x7f36a00096d0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.793+0000 7f36b95e4740 1 -- 172.21.15.16:0/1369629573 shutdown_connections 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.793+0000 7f36b95e4740 1 --2- 172.21.15.16:0/1369629573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d7709070 0x55b4d7ab6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.793+0000 7f36b95e4740 1 --2- 172.21.15.16:0/1369629573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e8390 0x55b4d7ac5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.793+0000 7f36b95e4740 1 --2- 172.21.15.16:0/1369629573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d76e01b0 0x55b4d7abb650 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.793+0000 7f36b95e4740 1 -- 172.21.15.16:0/1369629573 >> 172.21.15.16:0/1369629573 conn(0x55b4d76fd460 msgr2=0x55b4d7abaf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.793+0000 7f36b95e4740 1 -- 172.21.15.16:0/1369629573 shutdown_connections 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.793+0000 7f36b95e4740 1 -- 172.21.15.16:0/1369629573 wait complete. 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b95e4740 1 -- start start 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b95e4740 1 Processor -- start 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e01b0 0x55b4d7ac26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e8390 0x55b4d7aae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b95e4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d7709070 0x55b4d7aae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b95e4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1b0170 con 0x55b4d7709070 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b95e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1af9e0 con 0x55b4d76e8390 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b95e4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1acaf0 con 0x55b4d76e01b0 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b95e4740 10 client.0 _ll_register_callbacks cb 0x7f36a0013880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b160c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d7709070 0x55b4d7aae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e8390 0x55b4d7aae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e8390 0x55b4d7aae0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:40200/0 (socket says 172.21.15.16:40200) 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b160c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d7709070 0x55b4d7aae7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56618/0 (socket says 172.21.15.16:56618) 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36abfff640 1 -- 172.21.15.16:0/463934372 learned_addr learned my addr 172.21.15.16:0/463934372 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.794+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e01b0 0x55b4d7ac26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f36b160c640 1 -- 172.21.15.16:0/463934372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e01b0 msgr2=0x55b4d7ac26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f36b160c640 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e01b0 0x55b4d7ac26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f36b160c640 1 -- 172.21.15.16:0/463934372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e8390 msgr2=0x55b4d7aae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f36b160c640 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e8390 0x55b4d7aae0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f36b160c640 1 -- 172.21.15.16:0/463934372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4d7ac1310 con 0x55b4d7709070 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f36abfff640 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e8390 0x55b4d7aae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f36b160c640 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d7709070 0x55b4d7aae7a0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f36a4004180 tx=0x7f36a400fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ee6a1b4d387ea3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f368bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f368bfff640 1 -- 172.21.15.16:0/463934372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a4003970 con 0x55b4d7709070 2024-08-28T19:36:01.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f368bfff640 1 -- 172.21.15.16:0/463934372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f36a4003b10 con 0x55b4d7709070 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f36b95e4740 1 -- 172.21.15.16:0/463934372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36a0003de0 con 0x55b4d7709070 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f368bfff640 1 -- 172.21.15.16:0/463934372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f36a4035050 con 0x55b4d7709070 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f36b95e4740 10 client.19134 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.795+0000 7f36b95e4740 1 -- 172.21.15.16:0/463934372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55b4d7b36900 con 0x55b4d7709070 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.796+0000 7f368bfff640 1 -- 172.21.15.16:0/463934372 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36a0003de0 con 0x55b4d7709070 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.796+0000 7f368bfff640 1 -- 172.21.15.16:0/463934372 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f36a4018020 con 0x55b4d7709070 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.796+0000 7f36b95e4740 10 client.19134 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.796+0000 7f36b95e4740 10 client.19134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.796+0000 7f36b95e4740 20 client.19134 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.796+0000 7f36b95e4740 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b4d769cda0 0x55b4d7b39000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.796+0000 7f36b95e4740 4 client.19134 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.796+0000 7f36b95e4740 1 -- 172.21.15.16:0/463934372 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b4d7a27580 con 0x55b4d769cda0 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.796+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b4d769cda0 0x55b4d7b39000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:01.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.796+0000 7f368bfff640 10 client.19134 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.796+0000 7f36b0e0b640 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b4d769cda0 0x55b4d7b39000 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad8f40faf9abf8c6 server_cookie=fa256ca5756deb10 in_seq=0 out_seq=0 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f368bfff640 1 -- 172.21.15.16:0/463934372 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+5770 (crc 0 0 0) 0x55b4d7a27580 con 0x55b4d769cda0 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f368bfff640 10 client.19134 handle_command_reply: tid=0 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 1 client.19134 shutdown 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 1 -- 172.21.15.16:0/463934372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d7709070 msgr2=0x55b4d7aae7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d7709070 0x55b4d7aae7a0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f36a4004180 tx=0x7f36a400fbb0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 1 -- 172.21.15.16:0/463934372 shutdown_connections 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4d7709070 0x55b4d7aae7a0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d76e8390 0x55b4d7aae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55b4d769cda0 0x55b4d7b39000 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 1 --2- 172.21.15.16:0/463934372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b4d76e01b0 0x55b4d7ac26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f368bfff640 0 client.19134 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:01.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 1 -- 172.21.15.16:0/463934372 >> 172.21.15.16:0/463934372 conn(0x55b4d76fd460 msgr2=0x55b4d7aaa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:01.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 1 -- 172.21.15.16:0/463934372 shutdown_connections 2024-08-28T19:36:01.185 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:36:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:36:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:36:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:36:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:36:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:36:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:36:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:36:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:36:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:36:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:36:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:36:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:36:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:01.197 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:36:01.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 1 -- 172.21.15.16:0/463934372 wait complete. 2024-08-28T19:36:01.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.797+0000 7f36b95e4740 20 client.19134 trim_cache size 0 max 16384 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 -- 172.21.15.16:0/709556184 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f369c0b4ca0 msgr2=0x7f369c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f369c0b4ca0 0x7f369c0b71b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x55b4d7aacf40 tx=0x7f36a0015040 comp rx=0 tx=0).stop 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 -- 172.21.15.16:0/709556184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36ac108150 msgr2=0x7f36ac19a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36ac108150 0x7f36ac19a7d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f369802a860 tx=0x7f3698032d50 comp rx=0 tx=0).stop 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 -- 172.21.15.16:0/709556184 shutdown_connections 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f36ac108aa0 0x7f36ac19fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f369c0b4ca0 0x7f369c0b71b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36ac108150 0x7f36ac19a7d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 --2- 172.21.15.16:0/709556184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ac102f20 0x7f36ac19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 -- 172.21.15.16:0/709556184 >> 172.21.15.16:0/709556184 conn(0x7f36ac077a00 msgr2=0x7f36ac104180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 -- 172.21.15.16:0/709556184 shutdown_connections 2024-08-28T19:36:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:00.798+0000 7f36b306f640 1 -- 172.21.15.16:0/709556184 wait complete. 2024-08-28T19:36:01.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 21.07 MB/sec warmup 97 sec latency 13555.624 ms 2024-08-28T19:36:01.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:01 smithi016 ceph-mon[18556]: pgmap v804: 129 pgs: 129 active+clean; 895 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 224 op/s 2024-08-28T19:36:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:01 smithi115 ceph-mon[22184]: pgmap v804: 129 pgs: 129 active+clean; 895 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 224 op/s 2024-08-28T19:36:01.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:01 smithi028 ceph-mon[21769]: pgmap v804: 129 pgs: 129 active+clean; 895 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 224 op/s 2024-08-28T19:36:02.193 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:36:02.193 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:36:02.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 20.85 MB/sec warmup 98 sec latency 14555.675 ms 2024-08-28T19:36:02.600 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:03.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.124+0000 7f3f8f939640 1 -- 172.21.15.16:0/321160021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f88109b90 msgr2=0x7f3f881103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:03.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.124+0000 7f3f8f939640 1 --2- 172.21.15.16:0/321160021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f88109b90 0x7f3f881103c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f3f80009ca0 tx=0x7f3f8002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:36:03.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.125+0000 7f3f8f939640 1 -- 172.21.15.16:0/321160021 shutdown_connections 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.125+0000 7f3f8f939640 1 --2- 172.21.15.16:0/321160021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f88109b90 0x7f3f881103c0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.125+0000 7f3f8f939640 1 --2- 172.21.15.16:0/321160021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f88109240 0x7f3f88109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.125+0000 7f3f8f939640 1 --2- 172.21.15.16:0/321160021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f88103900 0x7f3f88103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.125+0000 7f3f8f939640 1 -- 172.21.15.16:0/321160021 >> 172.21.15.16:0/321160021 conn(0x7f3f8807b5f0 msgr2=0x7f3f8807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.125+0000 7f3f8f939640 1 -- 172.21.15.16:0/321160021 shutdown_connections 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.125+0000 7f3f8f939640 1 -- 172.21.15.16:0/321160021 wait complete. 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.125+0000 7f3f8f939640 1 Processor -- start 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.125+0000 7f3f8f939640 1 -- start start 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8f939640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f88103900 0x7f3f8819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8f939640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f88109240 0x7f3f8819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8f939640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f88109b90 0x7f3f8819fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8f939640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f88113850 con 0x7f3f88103900 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8f939640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f881136d0 con 0x7f3f88109b90 2024-08-28T19:36:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8f939640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3f881139d0 con 0x7f3f88109240 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8d6d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f88103900 0x7f3f8819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8d6d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f88103900 0x7f3f8819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56638/0 (socket says 172.21.15.16:56638) 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8d6d5640 1 -- 172.21.15.16:0/2981272771 learned_addr learned my addr 172.21.15.16:0/2981272771 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8ced4640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f88109240 0x7f3f8819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8ded6640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f88109b90 0x7f3f8819fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8d6d5640 1 -- 172.21.15.16:0/2981272771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f88109240 msgr2=0x7f3f8819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8d6d5640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f88109240 0x7f3f8819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8d6d5640 1 -- 172.21.15.16:0/2981272771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f88109b90 msgr2=0x7f3f8819fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8d6d5640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f88109b90 0x7f3f8819fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8d6d5640 1 -- 172.21.15.16:0/2981272771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f881a0150 con 0x7f3f88103900 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f8d6d5640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f88103900 0x7f3f8819a1e0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400e9f0 tx=0x7f3f8400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.126+0000 7f3f7e7fc640 1 -- 172.21.15.16:0/2981272771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8400cd80 con 0x7f3f88103900 2024-08-28T19:36:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.127+0000 7f3f8f939640 1 -- 172.21.15.16:0/2981272771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f881a0440 con 0x7f3f88103900 2024-08-28T19:36:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.127+0000 7f3f7e7fc640 1 -- 172.21.15.16:0/2981272771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f840108e0 con 0x7f3f88103900 2024-08-28T19:36:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.127+0000 7f3f7e7fc640 1 -- 172.21.15.16:0/2981272771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84018980 con 0x7f3f88103900 2024-08-28T19:36:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.127+0000 7f3f8f939640 1 -- 172.21.15.16:0/2981272771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f881a6ba0 con 0x7f3f88103900 2024-08-28T19:36:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.128+0000 7f3f8f939640 1 -- 172.21.15.16:0/2981272771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f88103d10 con 0x7f3f88103900 2024-08-28T19:36:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.129+0000 7f3f7e7fc640 1 -- 172.21.15.16:0/2981272771 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3f84010430 con 0x7f3f88103900 2024-08-28T19:36:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.130+0000 7f3f7e7fc640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f740b4ca0 0x7f3f740b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.130+0000 7f3f7e7fc640 1 -- 172.21.15.16:0/2981272771 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9490+0+0 (secure 0 0 0) 0x7f3f840e6110 con 0x7f3f88103900 2024-08-28T19:36:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.130+0000 7f3f8ced4640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f740b4ca0 0x7f3f740b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.130+0000 7f3f8ced4640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f740b4ca0 0x7f3f740b71b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3f88106d60 tx=0x7f3f7800b410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.133+0000 7f3f7e7fc640 1 -- 172.21.15.16:0/2981272771 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f3f840ac4e0 con 0x7f3f88103900 2024-08-28T19:36:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.387+0000 7f3f8f939640 1 -- 172.21.15.16:0/2981272771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f8819b310 con 0x7f3f88103900 2024-08-28T19:36:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.391+0000 7f3f7e7fc640 1 -- 172.21.15.16:0/2981272771 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) ==== 94+0+12010 (secure 0 0 0) 0x7f3f840b5070 con 0x7f3f88103900 2024-08-28T19:36:03.393 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:36:03.393 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"btime":"2024-08-28T19:35:42:414805+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":78,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:35:41.973464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:36:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 78 2024-08-28T19:36:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.397+0000 7f3f8f939640 1 -- 172.21.15.16:0/2981272771 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f740b4ca0 msgr2=0x7f3f740b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.397+0000 7f3f8f939640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f740b4ca0 0x7f3f740b71b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3f88106d60 tx=0x7f3f7800b410 comp rx=0 tx=0).stop 2024-08-28T19:36:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.398+0000 7f3f8f939640 1 -- 172.21.15.16:0/2981272771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f88103900 msgr2=0x7f3f8819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.398+0000 7f3f8f939640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f88103900 0x7f3f8819a1e0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400e9f0 tx=0x7f3f8400eeb0 comp rx=0 tx=0).stop 2024-08-28T19:36:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.400+0000 7f3f8f939640 1 -- 172.21.15.16:0/2981272771 shutdown_connections 2024-08-28T19:36:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.400+0000 7f3f8f939640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3f740b4ca0 0x7f3f740b71b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.400+0000 7f3f8f939640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f88109b90 0x7f3f8819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.400+0000 7f3f8f939640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f88109240 0x7f3f8819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.400+0000 7f3f8f939640 1 --2- 172.21.15.16:0/2981272771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f88103900 0x7f3f8819a1e0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.400+0000 7f3f8f939640 1 -- 172.21.15.16:0/2981272771 >> 172.21.15.16:0/2981272771 conn(0x7f3f8807b5f0 msgr2=0x7f3f88105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:03.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.402+0000 7f3f8f939640 1 -- 172.21.15.16:0/2981272771 shutdown_connections 2024-08-28T19:36:03.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:03.402+0000 7f3f8f939640 1 -- 172.21.15.16:0/2981272771 wait complete. 2024-08-28T19:36:03.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 20.64 MB/sec warmup 99 sec latency 15555.760 ms 2024-08-28T19:36:03.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:03 smithi016 ceph-mon[18556]: pgmap v805: 129 pgs: 129 active+clean; 920 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.7 MiB/s wr, 227 op/s 2024-08-28T19:36:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:03 smithi115 ceph-mon[22184]: pgmap v805: 129 pgs: 129 active+clean; 920 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.7 MiB/s wr, 227 op/s 2024-08-28T19:36:03.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:03 smithi028 ceph-mon[21769]: pgmap v805: 129 pgs: 129 active+clean; 920 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.7 MiB/s wr, 227 op/s 2024-08-28T19:36:04.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 20.44 MB/sec warmup 100 sec latency 16555.803 ms 2024-08-28T19:36:04.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:04 smithi115 ceph-mon[22184]: osdmap e101: 12 total, 12 up, 12 in 2024-08-28T19:36:04.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:04 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2981272771' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:04.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:04 smithi016 ceph-mon[18556]: osdmap e101: 12 total, 12 up, 12 in 2024-08-28T19:36:04.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:04 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2981272771' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:04.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:04 smithi028 ceph-mon[21769]: osdmap e101: 12 total, 12 up, 12 in 2024-08-28T19:36:04.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:04 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2981272771' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:05.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 20.23 MB/sec warmup 101 sec latency 17555.908 ms 2024-08-28T19:36:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:05 smithi115 ceph-mon[22184]: pgmap v807: 129 pgs: 129 active+clean; 924 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 8.4 MiB/s wr, 209 op/s 2024-08-28T19:36:05.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:05 smithi016 ceph-mon[18556]: pgmap v807: 129 pgs: 129 active+clean; 924 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 8.4 MiB/s wr, 209 op/s 2024-08-28T19:36:05.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:05 smithi028 ceph-mon[21769]: pgmap v807: 129 pgs: 129 active+clean; 924 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 8.4 MiB/s wr, 209 op/s 2024-08-28T19:36:06.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:36:06.465 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 20.04 MB/sec warmup 102 sec latency 18555.967 ms 2024-08-28T19:36:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.051+0000 7ff05a905640 1 -- 172.21.15.16:0/566532598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0541050d0 msgr2=0x7ff05410d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.051+0000 7ff05a905640 1 --2- 172.21.15.16:0/566532598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0541050d0 0x7ff05410d620 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ff048009900 tx=0x7ff04802a410 comp rx=0 tx=0).stop 2024-08-28T19:36:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.051+0000 7ff05a905640 1 -- 172.21.15.16:0/566532598 shutdown_connections 2024-08-28T19:36:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.051+0000 7ff05a905640 1 --2- 172.21.15.16:0/566532598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff05410dbe0 0x7ff0541100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.051+0000 7ff05a905640 1 --2- 172.21.15.16:0/566532598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0541050d0 0x7ff05410d620 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.051+0000 7ff05a905640 1 --2- 172.21.15.16:0/566532598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff054104780 0x7ff054104b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.051+0000 7ff05a905640 1 -- 172.21.15.16:0/566532598 >> 172.21.15.16:0/566532598 conn(0x7ff0540794f0 msgr2=0x7ff054079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.051+0000 7ff05a905640 1 -- 172.21.15.16:0/566532598 shutdown_connections 2024-08-28T19:36:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.051+0000 7ff05a905640 1 -- 172.21.15.16:0/566532598 wait complete. 2024-08-28T19:36:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.051+0000 7ff05a905640 1 Processor -- start 2024-08-28T19:36:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff05a905640 1 -- start start 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff05a905640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff054104780 0x7ff05419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff05a905640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0541050d0 0x7ff05419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff05a905640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05410dbe0 0x7ff05419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff05a905640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff054113840 con 0x7ff05410dbe0 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff05a905640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541136c0 con 0x7ff054104780 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff05a905640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541139c0 con 0x7ff0541050d0 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff058ea2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05410dbe0 0x7ff05419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff058ea2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05410dbe0 0x7ff05419fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55840/0 (socket says 172.21.15.16:55840) 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff058ea2640 1 -- 172.21.15.16:0/3724957698 learned_addr learned my addr 172.21.15.16:0/3724957698 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff0537fe640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0541050d0 0x7ff05419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff053fff640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff054104780 0x7ff05419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff058ea2640 1 -- 172.21.15.16:0/3724957698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0541050d0 msgr2=0x7ff05419a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff058ea2640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0541050d0 0x7ff05419a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff058ea2640 1 -- 172.21.15.16:0/3724957698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff054104780 msgr2=0x7ff05419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff058ea2640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff054104780 0x7ff05419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.052+0000 7ff058ea2640 1 -- 172.21.15.16:0/3724957698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0541a01c0 con 0x7ff05410dbe0 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.053+0000 7ff058ea2640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05410dbe0 0x7ff05419fa90 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00eaa0 tx=0x7ff04c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.053+0000 7ff0517fa640 1 -- 172.21.15.16:0/3724957698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c00cdb0 con 0x7ff05410dbe0 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.053+0000 7ff05a905640 1 -- 172.21.15.16:0/3724957698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0541a04b0 con 0x7ff05410dbe0 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.053+0000 7ff0517fa640 1 -- 172.21.15.16:0/3724957698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff04c005020 con 0x7ff05410dbe0 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.053+0000 7ff0517fa640 1 -- 172.21.15.16:0/3724957698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c0116d0 con 0x7ff05410dbe0 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.053+0000 7ff05a905640 1 -- 172.21.15.16:0/3724957698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0541a6c10 con 0x7ff05410dbe0 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.055+0000 7ff0517fa640 1 -- 172.21.15.16:0/3724957698 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff04c0054a0 con 0x7ff05410dbe0 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff060e7a740 1 Processor -- start 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff060e7a740 1 -- start start 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61cc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61cc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff060e7a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aeb6143680 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff060e7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aeb6132c10 con 0x55aeb5e27070 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff060e7a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55aeb6136bc0 con 0x55aeb5e06390 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff053fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff0517fa640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff03c0b4ca0 0x7ff03c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff053fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61cc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55848/0 (socket says 172.21.15.16:55848) 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff053fff640 1 -- 172.21.15.16:0/682905643 learned_addr learned my addr 172.21.15.16:0/682905643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff0537fe640 1 --2- 172.21.15.16:0/682905643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff058ea2640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff03c0b4ca0 0x7ff03c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.056+0000 7ff0537fe640 1 --2- 172.21.15.16:0/682905643 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff0517fa640 1 -- 172.21.15.16:0/3724957698 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7ff04c0e5b60 con 0x7ff05410dbe0 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff053fff640 1 -- 172.21.15.16:0/682905643 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 msgr2=0x55aeb61ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff053fff640 1 --2- 172.21.15.16:0/682905643 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff053fff640 1 -- 172.21.15.16:0/682905643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 msgr2=0x55aeb61cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff053fff640 1 --2- 172.21.15.16:0/682905643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61cc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff053fff640 1 -- 172.21.15.16:0/682905643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aeb5e92650 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff0537fe640 1 --2- 172.21.15.16:0/682905643 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff058ea2640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff03c0b4ca0 0x7ff03c0b71b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0a7e50 tx=0x7ff04c005790 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff053fff640 1 --2- 172.21.15.16:0/682905643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61cc850 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7ff044009c50 tx=0x7ff0440117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8f0c9fded34c10 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff03a7fc640 1 -- 172.21.15.16:0/682905643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044012450 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff03a7fc640 1 -- 172.21.15.16:0/682905643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff044012aa0 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.057+0000 7ff03a7fc640 1 -- 172.21.15.16:0/682905643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04400fac0 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.058+0000 7ff060e7a740 1 -- 172.21.15.16:0/682905643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 msgr2=0x55aeb61cc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.058+0000 7ff060e7a740 1 --2- 172.21.15.16:0/682905643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61cc850 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7ff044009c50 tx=0x7ff0440117b0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.058+0000 7ff060e7a740 1 -- 172.21.15.16:0/682905643 shutdown_connections 2024-08-28T19:36:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.058+0000 7ff060e7a740 1 --2- 172.21.15.16:0/682905643 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.058+0000 7ff060e7a740 1 --2- 172.21.15.16:0/682905643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61cc850 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.058+0000 7ff060e7a740 1 --2- 172.21.15.16:0/682905643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.058+0000 7ff060e7a740 1 -- 172.21.15.16:0/682905643 >> 172.21.15.16:0/682905643 conn(0x55aeb5e1b460 msgr2=0x55aeb61ca200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.058+0000 7ff060e7a740 1 -- 172.21.15.16:0/682905643 shutdown_connections 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.058+0000 7ff060e7a740 1 -- 172.21.15.16:0/682905643 wait complete. 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff060e7a740 1 -- start start 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff060e7a740 1 Processor -- start 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61dbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e06390 0x55aeb61dc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e27070 0x55aeb61e8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff060e7a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aeb5f95700 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff060e7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aeb60b9750 con 0x55aeb5e06390 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff060e7a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55aeb61dee20 con 0x55aeb5e27070 2024-08-28T19:36:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff060e7a740 10 client.0 _ll_register_callbacks cb 0x55aeb60e8e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff0537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61dbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff0537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61dbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55850/0 (socket says 172.21.15.16:55850) 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff0537fe640 1 -- 172.21.15.16:0/3957345327 learned_addr learned my addr 172.21.15.16:0/3957345327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.059+0000 7ff053fff640 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e06390 0x55aeb61dc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff0537fe640 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e27070 0x55aeb61e8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff0537fe640 1 -- 172.21.15.16:0/3957345327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e27070 msgr2=0x55aeb61e8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff0537fe640 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e27070 0x55aeb61e8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff0537fe640 1 -- 172.21.15.16:0/3957345327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e06390 msgr2=0x55aeb61dc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff0537fe640 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e06390 0x55aeb61dc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff0537fe640 1 -- 172.21.15.16:0/3957345327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aeb60f2900 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff0537fe640 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61dbe90 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7ff05406b6b0 tx=0x7ff04802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e959df877160b52a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff0337fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff0337fe640 1 -- 172.21.15.16:0/3957345327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048035070 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff0337fe640 1 -- 172.21.15.16:0/3957345327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7ff048004250 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff060e7a740 1 -- 172.21.15.16:0/3957345327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff044012aa0 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff0337fe640 1 -- 172.21.15.16:0/3957345327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7ff048051050 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff060e7a740 10 client.19152 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.060+0000 7ff060e7a740 1 -- 172.21.15.16:0/3957345327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55aeb61eac60 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.061+0000 7ff0337fe640 1 -- 172.21.15.16:0/3957345327 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff044012aa0 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.061+0000 7ff0337fe640 1 -- 172.21.15.16:0/3957345327 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7ff048004870 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.061+0000 7ff060e7a740 10 client.19152 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.061+0000 7ff060e7a740 10 client.19152 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.061+0000 7ff060e7a740 20 client.19152 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.061+0000 7ff060e7a740 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aeb5de59f0 0x55aeb61ee180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.061+0000 7ff060e7a740 4 client.19152 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.061+0000 7ff060e7a740 1 -- 172.21.15.16:0/3957345327 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aeb61455d0 con 0x55aeb5de59f0 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.061+0000 7ff053fff640 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aeb5de59f0 0x55aeb61ee180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.062+0000 7ff053fff640 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aeb5de59f0 0x55aeb61ee180 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=26bd3cfe08eac9d server_cookie=6842266ffccef1c in_seq=0 out_seq=0 2024-08-28T19:36:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.062+0000 7ff0337fe640 10 client.19152 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.062+0000 7ff0337fe640 1 -- 172.21.15.16:0/3957345327 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55aeb61455d0 con 0x55aeb5de59f0 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.062+0000 7ff0337fe640 10 client.19152 handle_command_reply: tid=0 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.062+0000 7ff060e7a740 1 client.19152 shutdown 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff060e7a740 1 -- 172.21.15.16:0/3957345327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 msgr2=0x55aeb61dbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff060e7a740 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61dbe90 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7ff05406b6b0 tx=0x7ff04802a410 comp rx=0 tx=0).stop 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff060e7a740 1 -- 172.21.15.16:0/3957345327 shutdown_connections 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff060e7a740 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aeb5de59f0 0x55aeb61ee180 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff060e7a740 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e27070 0x55aeb61e8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff060e7a740 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e06390 0x55aeb61dc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff060e7a740 1 --2- 172.21.15.16:0/3957345327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61dbe90 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff060e7a740 1 -- 172.21.15.16:0/3957345327 >> 172.21.15.16:0/3957345327 conn(0x55aeb5e1b460 msgr2=0x55aeb61cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff0337fe640 0 client.19152 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff060e7a740 1 -- 172.21.15.16:0/3957345327 shutdown_connections 2024-08-28T19:36:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff060e7a740 1 -- 172.21.15.16:0/3957345327 wait complete. 2024-08-28T19:36:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.063+0000 7ff060e7a740 20 client.19152 trim_cache size 0 max 16384 2024-08-28T19:36:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.076+0000 7ff060e7a740 1 Processor -- start 2024-08-28T19:36:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.076+0000 7ff060e7a740 1 -- start start 2024-08-28T19:36:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.076+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.076+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.076+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61d9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.076+0000 7ff060e7a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541aebe0 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.076+0000 7ff060e7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541b09f0 con 0x55aeb5e27070 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.076+0000 7ff060e7a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541af340 con 0x55aeb5e06390 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff0537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff0537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55858/0 (socket says 172.21.15.16:55858) 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff0537fe640 1 -- 172.21.15.16:0/1717990253 learned_addr learned my addr 172.21.15.16:0/1717990253 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff0537fe640 1 --2- 172.21.15.16:0/1717990253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61d9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff053fff640 1 --2- 172.21.15.16:0/1717990253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff0537fe640 1 -- 172.21.15.16:0/1717990253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 msgr2=0x55aeb61c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff0537fe640 1 --2- 172.21.15.16:0/1717990253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff0537fe640 1 -- 172.21.15.16:0/1717990253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 msgr2=0x55aeb61d9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff0537fe640 1 --2- 172.21.15.16:0/1717990253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61d9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff0537fe640 1 -- 172.21.15.16:0/1717990253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aeb61dd150 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff053fff640 1 --2- 172.21.15.16:0/1717990253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff0537fe640 1 --2- 172.21.15.16:0/1717990253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61c97d0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ff048031540 tx=0x7ff048000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ff45543a4764f1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff039ffb640 1 -- 172.21.15.16:0/1717990253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048035070 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff039ffb640 1 -- 172.21.15.16:0/1717990253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff048004250 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.077+0000 7ff039ffb640 1 -- 172.21.15.16:0/1717990253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048003780 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.078+0000 7ff060e7a740 1 -- 172.21.15.16:0/1717990253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 msgr2=0x55aeb61c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:07.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.078+0000 7ff060e7a740 1 --2- 172.21.15.16:0/1717990253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61c97d0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ff048031540 tx=0x7ff048000bc0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.078+0000 7ff060e7a740 1 -- 172.21.15.16:0/1717990253 shutdown_connections 2024-08-28T19:36:07.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.078+0000 7ff060e7a740 1 --2- 172.21.15.16:0/1717990253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61d9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.078+0000 7ff060e7a740 1 --2- 172.21.15.16:0/1717990253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.078+0000 7ff060e7a740 1 --2- 172.21.15.16:0/1717990253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61c97d0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.078+0000 7ff060e7a740 1 -- 172.21.15.16:0/1717990253 >> 172.21.15.16:0/1717990253 conn(0x55aeb5e1b460 msgr2=0x55aeb61da710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.078+0000 7ff060e7a740 1 -- 172.21.15.16:0/1717990253 shutdown_connections 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.078+0000 7ff060e7a740 1 -- 172.21.15.16:0/1717990253 wait complete. 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff060e7a740 1 -- start start 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff060e7a740 1 Processor -- start 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61e48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61dfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff060e7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61e0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff060e7a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541b0260 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff060e7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541afad0 con 0x55aeb5e27070 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff060e7a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541acbe0 con 0x55aeb5e06390 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff060e7a740 10 client.0 _ll_register_callbacks cb 0x7ff048009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:07.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff0537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61e48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff0537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61e48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55860/0 (socket says 172.21.15.16:55860) 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff0537fe640 1 -- 172.21.15.16:0/144991965 learned_addr learned my addr 172.21.15.16:0/144991965 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.079+0000 7ff0537fe640 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61e0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff053fff640 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61dfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff0537fe640 1 -- 172.21.15.16:0/144991965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 msgr2=0x55aeb61dfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff0537fe640 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61dfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff0537fe640 1 -- 172.21.15.16:0/144991965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 msgr2=0x55aeb61e0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff0537fe640 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61e0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff0537fe640 1 -- 172.21.15.16:0/144991965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aeb61e0ad0 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff0537fe640 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61e48e0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7ff048004d60 tx=0x7ff0480318a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b094923291c4dc6a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff0337fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff0337fe640 1 -- 172.21.15.16:0/144991965 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048035070 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff0337fe640 1 -- 172.21.15.16:0/144991965 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7ff048004950 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff060e7a740 1 -- 172.21.15.16:0/144991965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff048004250 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff0337fe640 1 -- 172.21.15.16:0/144991965 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7ff048004eb0 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff060e7a740 10 client.19158 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff060e7a740 1 -- 172.21.15.16:0/144991965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55aeb6258ac0 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.080+0000 7ff0337fe640 1 -- 172.21.15.16:0/144991965 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff048004250 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.081+0000 7ff0337fe640 1 -- 172.21.15.16:0/144991965 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7ff0480039b0 con 0x55aeb5dfe1b0 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.081+0000 7ff060e7a740 10 client.19158 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.081+0000 7ff060e7a740 10 client.19158 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.081+0000 7ff060e7a740 20 client.19158 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.081+0000 7ff060e7a740 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aeb5dbada0 0x55aeb625b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.081+0000 7ff060e7a740 4 client.19158 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:36:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.081+0000 7ff060e7a740 1 -- 172.21.15.16:0/144991965 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aeb6141070 con 0x55aeb5dbada0 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.081+0000 7ff053fff640 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aeb5dbada0 0x55aeb625b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff053fff640 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aeb5dbada0 0x55aeb625b280 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2b3aa39880ab0339 server_cookie=7a07a73b30cb8f1e in_seq=0 out_seq=0 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff0337fe640 10 client.19158 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff0337fe640 1 -- 172.21.15.16:0/144991965 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aeb6141070 con 0x55aeb5dbada0 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff0337fe640 10 client.19158 handle_command_reply: tid=0 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 1 client.19158 shutdown 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 1 -- 172.21.15.16:0/144991965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 msgr2=0x55aeb61e48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61e48e0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7ff048004d60 tx=0x7ff0480318a0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 1 -- 172.21.15.16:0/144991965 shutdown_connections 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aeb5dbada0 0x55aeb625b280 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aeb5e27070 0x55aeb61e0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aeb5e06390 0x55aeb61dfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 1 --2- 172.21.15.16:0/144991965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeb5dfe1b0 0x55aeb61e48e0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 1 -- 172.21.15.16:0/144991965 >> 172.21.15.16:0/144991965 conn(0x55aeb5e1b460 msgr2=0x55aeb61dd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff0337fe640 0 client.19158 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:36:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 1 -- 172.21.15.16:0/144991965 shutdown_connections 2024-08-28T19:36:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 1 -- 172.21.15.16:0/144991965 wait complete. 2024-08-28T19:36:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.082+0000 7ff060e7a740 20 client.19158 trim_cache size 0 max 16384 2024-08-28T19:36:07.090 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:36:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 -- 172.21.15.16:0/3724957698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff03c0b4ca0 msgr2=0x7ff03c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff03c0b4ca0 0x7ff03c0b71b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0a7e50 tx=0x7ff04c005790 comp rx=0 tx=0).stop 2024-08-28T19:36:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 -- 172.21.15.16:0/3724957698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05410dbe0 msgr2=0x7ff05419fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05410dbe0 0x7ff05419fa90 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00eaa0 tx=0x7ff04c00ef60 comp rx=0 tx=0).stop 2024-08-28T19:36:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 -- 172.21.15.16:0/3724957698 shutdown_connections 2024-08-28T19:36:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05410dbe0 0x7ff05419fa90 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0541050d0 0x7ff05419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff03c0b4ca0 0x7ff03c0b71b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 --2- 172.21.15.16:0/3724957698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff054104780 0x7ff05419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 -- 172.21.15.16:0/3724957698 >> 172.21.15.16:0/3724957698 conn(0x7ff0540794f0 msgr2=0x7ff0541065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 -- 172.21.15.16:0/3724957698 shutdown_connections 2024-08-28T19:36:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:07.083+0000 7ff05a905640 1 -- 172.21.15.16:0/3724957698 wait complete. 2024-08-28T19:36:07.466 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 19.84 MB/sec warmup 103 sec latency 19556.009 ms 2024-08-28T19:36:07.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:07 smithi115 ceph-mon[22184]: pgmap v808: 129 pgs: 129 active+clean; 893 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 9.4 MiB/s wr, 290 op/s 2024-08-28T19:36:07.692 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:07 smithi016 ceph-mon[18556]: pgmap v808: 129 pgs: 129 active+clean; 893 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 9.4 MiB/s wr, 290 op/s 2024-08-28T19:36:07.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:36:07.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:07 smithi028 ceph-mon[21769]: pgmap v808: 129 pgs: 129 active+clean; 893 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 9.4 MiB/s wr, 290 op/s 2024-08-28T19:36:08.010 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:08.466 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 19.65 MB/sec warmup 104 sec latency 20556.067 ms 2024-08-28T19:36:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.502+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/3506519046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e14108170 msgr2=0x7f4e14108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.502+0000 7f4e1c3a8640 1 --2- 172.21.15.16:0/3506519046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e14108170 0x7f4e14108580 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009900 tx=0x7f4e0402a410 comp rx=0 tx=0).stop 2024-08-28T19:36:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.502+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/3506519046 shutdown_connections 2024-08-28T19:36:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.502+0000 7f4e1c3a8640 1 --2- 172.21.15.16:0/3506519046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e14108ac0 0x7f4e1410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.502+0000 7f4e1c3a8640 1 --2- 172.21.15.16:0/3506519046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e14108170 0x7f4e14108580 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.502+0000 7f4e1c3a8640 1 --2- 172.21.15.16:0/3506519046 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e14102880 0x7f4e14102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.502+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/3506519046 >> 172.21.15.16:0/3506519046 conn(0x7f4e14079500 msgr2=0x7f4e14079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.502+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/3506519046 shutdown_connections 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.502+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/3506519046 wait complete. 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1c3a8640 1 Processor -- start 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1c3a8640 1 -- start start 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1c3a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e14102880 0x7f4e1419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1c3a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e14108170 0x7f4e1419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1c3a8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e14108ac0 0x7f4e1419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1c3a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e14112710 con 0x7f4e14102880 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1c3a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e14112590 con 0x7f4e14108170 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1c3a8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4e14112890 con 0x7f4e14108ac0 2024-08-28T19:36:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1a144640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e14102880 0x7f4e1419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1a144640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e14102880 0x7f4e1419a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55874/0 (socket says 172.21.15.16:55874) 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1a144640 1 -- 172.21.15.16:0/234572640 learned_addr learned my addr 172.21.15.16:0/234572640 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e19943640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e14108170 0x7f4e1419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1a945640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e14108ac0 0x7f4e1419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1a144640 1 -- 172.21.15.16:0/234572640 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e14108ac0 msgr2=0x7f4e1419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1a144640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e14108ac0 0x7f4e1419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1a144640 1 -- 172.21.15.16:0/234572640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e14108170 msgr2=0x7f4e1419a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1a144640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e14108170 0x7f4e1419a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.503+0000 7f4e1a144640 1 -- 172.21.15.16:0/234572640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e141a0280 con 0x7f4e14102880 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.504+0000 7f4e1a144640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e14102880 0x7f4e1419a310 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000d770 tx=0x7f4e1000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.504+0000 7f4e0b7fe640 1 -- 172.21.15.16:0/234572640 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10017070 con 0x7f4e14102880 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.504+0000 7f4e0b7fe640 1 -- 172.21.15.16:0/234572640 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e100054d0 con 0x7f4e14102880 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.504+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/234572640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e141a0570 con 0x7f4e14102880 2024-08-28T19:36:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.504+0000 7f4e0b7fe640 1 -- 172.21.15.16:0/234572640 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10004c30 con 0x7f4e14102880 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.504+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/234572640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e141a6cd0 con 0x7f4e14102880 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.506+0000 7f4e2291d740 1 Processor -- start 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e2291d740 1 -- start start 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e0b7fe640 1 -- 172.21.15.16:0/234572640 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4e10004e10 con 0x7f4e14102880 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b27a070 0x563a6b61f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b2511b0 0x563a6b61f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b259390 0x563a6b61fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e2291d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563a6b596680 con 0x563a6b259390 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e2291d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563a6b585c10 con 0x563a6b2511b0 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e2291d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563a6b589bc0 con 0x563a6b27a070 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e19943640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b259390 0x563a6b61fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e19943640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b259390 0x563a6b61fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55880/0 (socket says 172.21.15.16:55880) 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e19943640 1 -- 172.21.15.16:0/213531964 learned_addr learned my addr 172.21.15.16:0/213531964 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e1a144640 1 --2- 172.21.15.16:0/213531964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b2511b0 0x563a6b61f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.507+0000 7f4e1a945640 1 --2- 172.21.15.16:0/213531964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b27a070 0x563a6b61f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e19943640 1 -- 172.21.15.16:0/213531964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b27a070 msgr2=0x563a6b61f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e19943640 1 --2- 172.21.15.16:0/213531964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b27a070 0x563a6b61f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e19943640 1 -- 172.21.15.16:0/213531964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b2511b0 msgr2=0x563a6b61f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e19943640 1 --2- 172.21.15.16:0/213531964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b2511b0 0x563a6b61f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e19943640 1 -- 172.21.15.16:0/213531964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6b2e5650 con 0x563a6b259390 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e0b7fe640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4e000b4d20 0x7f4e000b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e19943640 1 --2- 172.21.15.16:0/213531964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b259390 0x563a6b61fdd0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f4e1406b420 tx=0x7f4e04032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cbe5451743e530 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e1a945640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4e000b4d20 0x7f4e000b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e08d9a640 1 -- 172.21.15.16:0/213531964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0403f070 con 0x563a6b259390 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e0b7fe640 1 -- 172.21.15.16:0/234572640 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7f4e100e54b0 con 0x7f4e14102880 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e08d9a640 1 -- 172.21.15.16:0/213531964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e040316a0 con 0x563a6b259390 2024-08-28T19:36:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.508+0000 7f4e08d9a640 1 -- 172.21.15.16:0/213531964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04035070 con 0x563a6b259390 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.509+0000 7f4e2291d740 1 -- 172.21.15.16:0/213531964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b259390 msgr2=0x563a6b61fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.509+0000 7f4e1a945640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4e000b4d20 0x7f4e000b7230 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c008dc0 tx=0x7f4e0c008570 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.509+0000 7f4e2291d740 1 --2- 172.21.15.16:0/213531964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b259390 0x563a6b61fdd0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f4e1406b420 tx=0x7f4e04032ce0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.509+0000 7f4e2291d740 1 -- 172.21.15.16:0/213531964 shutdown_connections 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.509+0000 7f4e2291d740 1 --2- 172.21.15.16:0/213531964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b259390 0x563a6b61fdd0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.509+0000 7f4e2291d740 1 --2- 172.21.15.16:0/213531964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b2511b0 0x563a6b61f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.509+0000 7f4e2291d740 1 --2- 172.21.15.16:0/213531964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b27a070 0x563a6b61f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.509+0000 7f4e2291d740 1 -- 172.21.15.16:0/213531964 >> 172.21.15.16:0/213531964 conn(0x563a6b26e460 msgr2=0x563a6b61d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.509+0000 7f4e2291d740 1 -- 172.21.15.16:0/213531964 shutdown_connections 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.509+0000 7f4e2291d740 1 -- 172.21.15.16:0/213531964 wait complete. 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e2291d740 1 -- start start 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e2291d740 1 Processor -- start 2024-08-28T19:36:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b62aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b259390 0x563a6b62af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b27a070 0x563a6b636dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e2291d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563a6b3e8700 con 0x563a6b2511b0 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e2291d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563a6b50c750 con 0x563a6b27a070 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e2291d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563a6b62dad0 con 0x563a6b259390 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e2291d740 10 client.0 _ll_register_callbacks cb 0x563a6b53be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e1a144640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b62aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e1a144640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b62aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55888/0 (socket says 172.21.15.16:55888) 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e1a144640 1 -- 172.21.15.16:0/4034442589 learned_addr learned my addr 172.21.15.16:0/4034442589 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e1a945640 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b27a070 0x563a6b636dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.510+0000 7f4e19943640 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b259390 0x563a6b62af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4e1a144640 1 -- 172.21.15.16:0/4034442589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b259390 msgr2=0x563a6b62af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4e1a144640 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b259390 0x563a6b62af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4e1a144640 1 -- 172.21.15.16:0/4034442589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b27a070 msgr2=0x563a6b636dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4e1a144640 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b27a070 0x563a6b636dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4e1a144640 1 -- 172.21.15.16:0/4034442589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6b545900 con 0x563a6b2511b0 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4e1a945640 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b27a070 0x563a6b636dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4e1a144640 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b62aa40 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f4e100e2840 tx=0x7f4e100e8bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2b86d7fe97f039 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4dfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4dfdffb640 1 -- 172.21.15.16:0/4034442589 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10020070 con 0x563a6b2511b0 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4dfdffb640 1 -- 172.21.15.16:0/4034442589 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7f4e100e5150 con 0x563a6b2511b0 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4e2291d740 1 -- 172.21.15.16:0/4034442589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e040316a0 con 0x563a6b2511b0 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4dfdffb640 1 -- 172.21.15.16:0/4034442589 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f4e100ab0d0 con 0x563a6b2511b0 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4e2291d740 10 client.19170 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4e2291d740 1 -- 172.21.15.16:0/4034442589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x563a6b6398c0 con 0x563a6b2511b0 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4dfdffb640 1 -- 172.21.15.16:0/4034442589 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e040316a0 con 0x563a6b2511b0 2024-08-28T19:36:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.511+0000 7f4dfdffb640 1 -- 172.21.15.16:0/4034442589 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f4e100b4610 con 0x563a6b2511b0 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.512+0000 7f4e2291d740 10 client.19170 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.512+0000 7f4e2291d740 10 client.19170 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.512+0000 7f4e2291d740 20 client.19170 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.512+0000 7f4e2291d740 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x563a6b2389f0 0x563a6b63cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.512+0000 7f4e2291d740 4 client.19170 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.512+0000 7f4e2291d740 1 -- 172.21.15.16:0/4034442589 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a6b5985d0 con 0x563a6b2389f0 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.512+0000 7f4e19943640 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x563a6b2389f0 0x563a6b63cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.512+0000 7f4dfdffb640 10 client.19170 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.512+0000 7f4e19943640 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x563a6b2389f0 0x563a6b63cda0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d5e1ff93791fc704 server_cookie=9ca82478d346e236 in_seq=0 out_seq=0 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4dfdffb640 1 -- 172.21.15.16:0/4034442589 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563a6b5985d0 con 0x563a6b2389f0 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4dfdffb640 10 client.19170 handle_command_reply: tid=0 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 1 client.19170 shutdown 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 1 -- 172.21.15.16:0/4034442589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 msgr2=0x563a6b62aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b62aa40 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f4e100e2840 tx=0x7f4e100e8bf0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 1 -- 172.21.15.16:0/4034442589 shutdown_connections 2024-08-28T19:36:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x563a6b2389f0 0x563a6b63cda0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b27a070 0x563a6b636dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b259390 0x563a6b62af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 1 --2- 172.21.15.16:0/4034442589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b62aa40 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 1 -- 172.21.15.16:0/4034442589 >> 172.21.15.16:0/4034442589 conn(0x563a6b26e460 msgr2=0x563a6b6224e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4dfdffb640 0 client.19170 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:36:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 1 -- 172.21.15.16:0/4034442589 shutdown_connections 2024-08-28T19:36:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 1 -- 172.21.15.16:0/4034442589 wait complete. 2024-08-28T19:36:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.513+0000 7f4e2291d740 20 client.19170 trim_cache size 0 max 16384 2024-08-28T19:36:08.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.529+0000 7f4e2291d740 1 Processor -- start 2024-08-28T19:36:08.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.529+0000 7f4e2291d740 1 -- start start 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.529+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b2511b0 0x563a6b636cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.529+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b259390 0x563a6b627c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.529+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b27a070 0x563a6b628170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.529+0000 7f4e2291d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e141b0db0 con 0x563a6b27a070 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.529+0000 7f4e2291d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e141b2bc0 con 0x563a6b259390 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.529+0000 7f4e2291d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4e141b1510 con 0x563a6b2511b0 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e1a945640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b27a070 0x563a6b628170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e1a945640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b27a070 0x563a6b628170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55894/0 (socket says 172.21.15.16:55894) 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e1a945640 1 -- 172.21.15.16:0/1673083132 learned_addr learned my addr 172.21.15.16:0/1673083132 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e1a144640 1 --2- 172.21.15.16:0/1673083132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b2511b0 0x563a6b636cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e19943640 1 --2- 172.21.15.16:0/1673083132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b259390 0x563a6b627c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e1a945640 1 -- 172.21.15.16:0/1673083132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b2511b0 msgr2=0x563a6b636cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e1a945640 1 --2- 172.21.15.16:0/1673083132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b2511b0 0x563a6b636cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e1a945640 1 -- 172.21.15.16:0/1673083132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b259390 msgr2=0x563a6b627c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e1a945640 1 --2- 172.21.15.16:0/1673083132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b259390 0x563a6b627c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e1a945640 1 -- 172.21.15.16:0/1673083132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6b621640 con 0x563a6b27a070 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e19943640 1 --2- 172.21.15.16:0/1673083132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b259390 0x563a6b627c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4e1a945640 1 --2- 172.21.15.16:0/1673083132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b27a070 0x563a6b628170 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c01e5a0 tx=0x7f4e0c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1e7460425e8608 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4dfffff640 1 -- 172.21.15.16:0/1673083132 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c01c840 con 0x563a6b27a070 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4dfffff640 1 -- 172.21.15.16:0/1673083132 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e0c01ce90 con 0x563a6b27a070 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.530+0000 7f4dfffff640 1 -- 172.21.15.16:0/1673083132 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c00e3b0 con 0x563a6b27a070 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.531+0000 7f4e2291d740 1 -- 172.21.15.16:0/1673083132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b27a070 msgr2=0x563a6b628170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.531+0000 7f4e2291d740 1 --2- 172.21.15.16:0/1673083132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b27a070 0x563a6b628170 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c01e5a0 tx=0x7f4e0c014000 comp rx=0 tx=0).stop 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.531+0000 7f4e2291d740 1 -- 172.21.15.16:0/1673083132 shutdown_connections 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.531+0000 7f4e2291d740 1 --2- 172.21.15.16:0/1673083132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b27a070 0x563a6b628170 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.531+0000 7f4e2291d740 1 --2- 172.21.15.16:0/1673083132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b259390 0x563a6b627c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.531+0000 7f4e2291d740 1 --2- 172.21.15.16:0/1673083132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b2511b0 0x563a6b636cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.531+0000 7f4e2291d740 1 -- 172.21.15.16:0/1673083132 >> 172.21.15.16:0/1673083132 conn(0x563a6b26e460 msgr2=0x563a6b6292c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.531+0000 7f4e2291d740 1 -- 172.21.15.16:0/1673083132 shutdown_connections 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.531+0000 7f4e2291d740 1 -- 172.21.15.16:0/1673083132 wait complete. 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e2291d740 1 -- start start 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e2291d740 1 Processor -- start 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b633300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b259390 0x563a6b62f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e2291d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b27a070 0x563a6b62fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e2291d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e141b2430 con 0x563a6b2511b0 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e2291d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e141b1ca0 con 0x563a6b259390 2024-08-28T19:36:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e2291d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4e141aedb0 con 0x563a6b27a070 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e2291d740 10 client.0 _ll_register_callbacks cb 0x7f4e1000d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e1a144640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b633300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e1a144640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b633300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55910/0 (socket says 172.21.15.16:55910) 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.532+0000 7f4e1a144640 1 -- 172.21.15.16:0/935144949 learned_addr learned my addr 172.21.15.16:0/935144949 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.533+0000 7f4e1a945640 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b27a070 0x563a6b62fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.533+0000 7f4e19943640 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b259390 0x563a6b62f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.533+0000 7f4e1a144640 1 -- 172.21.15.16:0/935144949 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b27a070 msgr2=0x563a6b62fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.533+0000 7f4e1a144640 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b27a070 0x563a6b62fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.533+0000 7f4e1a144640 1 -- 172.21.15.16:0/935144949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b259390 msgr2=0x563a6b62f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.533+0000 7f4e1a144640 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b259390 0x563a6b62f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.533+0000 7f4e1a144640 1 -- 172.21.15.16:0/935144949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6b6302c0 con 0x563a6b2511b0 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.533+0000 7f4e1a945640 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b27a070 0x563a6b62fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4e1a144640 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b633300 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f4e100e2de0 tx=0x7f4e100097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2d796cc84e69c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4dfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4dfdffb640 1 -- 172.21.15.16:0/935144949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10014020 con 0x563a6b2511b0 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4dfdffb640 1 -- 172.21.15.16:0/935144949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7f4e100aabf0 con 0x563a6b2511b0 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4e2291d740 1 -- 172.21.15.16:0/935144949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e0c01ce90 con 0x563a6b2511b0 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4dfdffb640 1 -- 172.21.15.16:0/935144949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f4e10020040 con 0x563a6b2511b0 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4e2291d740 10 client.19176 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4e2291d740 1 -- 172.21.15.16:0/935144949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x563a6b6a84d0 con 0x563a6b2511b0 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4dfdffb640 1 -- 172.21.15.16:0/935144949 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e0c01ce90 con 0x563a6b2511b0 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4dfdffb640 1 -- 172.21.15.16:0/935144949 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f4e1005b1f0 con 0x563a6b2511b0 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4e2291d740 10 client.19176 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4e2291d740 10 client.19176 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4e2291d740 20 client.19176 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4e2291d740 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x563a6b20dda0 0x563a6b6aad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4e2291d740 4 client.19176 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.534+0000 7f4e2291d740 1 -- 172.21.15.16:0/935144949 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a6b594070 con 0x563a6b20dda0 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4e19943640 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x563a6b20dda0 0x563a6b6aad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4dfdffb640 10 client.19176 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4e19943640 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x563a6b20dda0 0x563a6b6aad50 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=621e962603ae85e0 server_cookie=fe68facec2b443ef in_seq=0 out_seq=0 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4dfdffb640 1 -- 172.21.15.16:0/935144949 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a6b594070 con 0x563a6b20dda0 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4dfdffb640 10 client.19176 handle_command_reply: tid=0 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4e2291d740 1 client.19176 shutdown 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4e2291d740 1 -- 172.21.15.16:0/935144949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 msgr2=0x563a6b633300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4e2291d740 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b633300 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f4e100e2de0 tx=0x7f4e100097b0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4e2291d740 1 -- 172.21.15.16:0/935144949 shutdown_connections 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4e2291d740 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x563a6b20dda0 0x563a6b6aad50 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4e2291d740 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563a6b27a070 0x563a6b62fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4e2291d740 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a6b259390 0x563a6b62f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4e2291d740 1 --2- 172.21.15.16:0/935144949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563a6b2511b0 0x563a6b633300 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4dfdffb640 0 client.19176 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:36:08.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.535+0000 7f4e2291d740 1 -- 172.21.15.16:0/935144949 >> 172.21.15.16:0/935144949 conn(0x563a6b26e460 msgr2=0x563a6b620380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:08.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e2291d740 1 -- 172.21.15.16:0/935144949 shutdown_connections 2024-08-28T19:36:08.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e2291d740 1 -- 172.21.15.16:0/935144949 wait complete. 2024-08-28T19:36:08.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e2291d740 20 client.19176 trim_cache size 0 max 16384 2024-08-28T19:36:08.548 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:36:08.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/234572640 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4e000b4d20 msgr2=0x7f4e000b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:08.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4e000b4d20 0x7f4e000b7230 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c008dc0 tx=0x7f4e0c008570 comp rx=0 tx=0).stop 2024-08-28T19:36:08.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/234572640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e14102880 msgr2=0x7f4e1419a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:08.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e14102880 0x7f4e1419a310 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000d770 tx=0x7f4e1000dc30 comp rx=0 tx=0).stop 2024-08-28T19:36:08.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/234572640 shutdown_connections 2024-08-28T19:36:08.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e14108ac0 0x7f4e1419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4e000b4d20 0x7f4e000b7230 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e14108170 0x7f4e1419a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 --2- 172.21.15.16:0/234572640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e14102880 0x7f4e1419a310 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/234572640 >> 172.21.15.16:0/234572640 conn(0x7f4e14079500 msgr2=0x7f4e14101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/234572640 shutdown_connections 2024-08-28T19:36:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:08.536+0000 7f4e1c3a8640 1 -- 172.21.15.16:0/234572640 wait complete. 2024-08-28T19:36:09.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:36:09.430 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:09.466 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 19.46 MB/sec warmup 105 sec latency 21556.381 ms 2024-08-28T19:36:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:09 smithi016 ceph-mon[18556]: pgmap v809: 129 pgs: 129 active+clean; 875 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 9.6 MiB/s wr, 321 op/s 2024-08-28T19:36:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:09 smithi115 ceph-mon[22184]: pgmap v809: 129 pgs: 129 active+clean; 875 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 9.6 MiB/s wr, 321 op/s 2024-08-28T19:36:09.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:09 smithi028 ceph-mon[21769]: pgmap v809: 129 pgs: 129 active+clean; 875 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 9.6 MiB/s wr, 321 op/s 2024-08-28T19:36:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.902+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/3412167251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b0107920 msgr2=0x7fa1b0107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.902+0000 7fa1b6c1b640 1 --2- 172.21.15.16:0/3412167251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b0107920 0x7fa1b0107d30 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0009900 tx=0x7fa1a002a410 comp rx=0 tx=0).stop 2024-08-28T19:36:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.902+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/3412167251 shutdown_connections 2024-08-28T19:36:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.902+0000 7fa1b6c1b640 1 --2- 172.21.15.16:0/3412167251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b0108270 0x7fa1b010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.902+0000 7fa1b6c1b640 1 --2- 172.21.15.16:0/3412167251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b0107920 0x7fa1b0107d30 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.902+0000 7fa1b6c1b640 1 --2- 172.21.15.16:0/3412167251 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b0101fe0 0x7fa1b01023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.902+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/3412167251 >> 172.21.15.16:0/3412167251 conn(0x7fa1b0079d30 msgr2=0x7fa1b007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.903+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/3412167251 shutdown_connections 2024-08-28T19:36:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.903+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/3412167251 wait complete. 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.903+0000 7fa1b6c1b640 1 Processor -- start 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.903+0000 7fa1b6c1b640 1 -- start start 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.903+0000 7fa1b6c1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b0101fe0 0x7fa1b0198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.903+0000 7fa1b6c1b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b0107920 0x7fa1b01986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.903+0000 7fa1b6c1b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b0108270 0x7fa1b019d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.903+0000 7fa1b6c1b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b01116d0 con 0x7fa1b0108270 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.903+0000 7fa1b6c1b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b0111550 con 0x7fa1b0101fe0 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.903+0000 7fa1b6c1b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b0111850 con 0x7fa1b0107920 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b51b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b0108270 0x7fa1b019d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b51b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b0108270 0x7fa1b019d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55938/0 (socket says 172.21.15.16:55938) 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b49b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b0101fe0 0x7fa1b0198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b51b8640 1 -- 172.21.15.16:0/1591534454 learned_addr learned my addr 172.21.15.16:0/1591534454 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b0107920 0x7fa1b01986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b51b8640 1 -- 172.21.15.16:0/1591534454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b0107920 msgr2=0x7fa1b01986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b51b8640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b0107920 0x7fa1b01986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b51b8640 1 -- 172.21.15.16:0/1591534454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b0101fe0 msgr2=0x7fa1b0198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b51b8640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b0101fe0 0x7fa1b0198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b51b8640 1 -- 172.21.15.16:0/1591534454 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b019e070 con 0x7fa1b0108270 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b49b7640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b0101fe0 0x7fa1b0198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b51b8640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b0108270 0x7fa1b019d940 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800eed0 tx=0x7fa1a800d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1a5ffb640 1 -- 172.21.15.16:0/1591534454 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8004b20 con 0x7fa1b0108270 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1a5ffb640 1 -- 172.21.15.16:0/1591534454 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1a8004cc0 con 0x7fa1b0108270 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/1591534454 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b019e360 con 0x7fa1b0108270 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.904+0000 7fa1a5ffb640 1 -- 172.21.15.16:0/1591534454 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8005480 con 0x7fa1b0108270 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.905+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/1591534454 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1b01a4ac0 con 0x7fa1b0108270 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.907+0000 7fa1a5ffb640 1 -- 172.21.15.16:0/1591534454 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa1a8005620 con 0x7fa1b0108270 2024-08-28T19:36:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.907+0000 7fa1bd190740 1 Processor -- start 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.907+0000 7fa1bd190740 1 -- start start 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 0x55b0b3266310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2e981b0 0x55b0b3266850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2ea0390 0x55b0b3266d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1bd190740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b0b31dd680 con 0x55b0b2ec1070 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1bd190740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b0b31ccc10 con 0x55b0b2e981b0 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1bd190740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b0b31d0bc0 con 0x55b0b2ea0390 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 0x55b0b3266310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 0x55b0b3266310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55948/0 (socket says 172.21.15.16:55948) 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a7fff640 1 -- 172.21.15.16:0/215312880 learned_addr learned my addr 172.21.15.16:0/215312880 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1b49b7640 1 --2- 172.21.15.16:0/215312880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2e981b0 0x55b0b3266850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a5ffb640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1980b4ca0 0x7fa1980b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/215312880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2ea0390 0x55b0b3266d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1b51b8640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1980b4ca0 0x7fa1980b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a7fff640 1 -- 172.21.15.16:0/215312880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2ea0390 msgr2=0x55b0b3266d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:09.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/215312880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2ea0390 0x55b0b3266d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a7fff640 1 -- 172.21.15.16:0/215312880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2e981b0 msgr2=0x55b0b3266850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:09.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/215312880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2e981b0 0x55b0b3266850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a7fff640 1 -- 172.21.15.16:0/215312880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0b2f2c650 con 0x55b0b2ec1070 2024-08-28T19:36:09.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a5ffb640 1 -- 172.21.15.16:0/1591534454 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7fa1a80e6080 con 0x7fa1b0108270 2024-08-28T19:36:09.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.908+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/215312880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 0x55b0b3266310 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0005ab0 tx=0x7fa1a0005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e504b7ba9d654876 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:09.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.909+0000 7fa196ffd640 1 -- 172.21.15.16:0/215312880 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0035050 con 0x55b0b2ec1070 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.909+0000 7fa196ffd640 1 -- 172.21.15.16:0/215312880 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1a0031620 con 0x55b0b2ec1070 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.909+0000 7fa1b51b8640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1980b4ca0 0x7fa1980b71b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80e3460 tx=0x7fa1a8078550 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.909+0000 7fa196ffd640 1 -- 172.21.15.16:0/215312880 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a00055f0 con 0x55b0b2ec1070 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.909+0000 7fa1bd190740 1 -- 172.21.15.16:0/215312880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 msgr2=0x55b0b3266310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.909+0000 7fa1bd190740 1 --2- 172.21.15.16:0/215312880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 0x55b0b3266310 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0005ab0 tx=0x7fa1a0005a20 comp rx=0 tx=0).stop 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.910+0000 7fa1bd190740 1 -- 172.21.15.16:0/215312880 shutdown_connections 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.910+0000 7fa1bd190740 1 --2- 172.21.15.16:0/215312880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2ea0390 0x55b0b3266d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.910+0000 7fa1bd190740 1 --2- 172.21.15.16:0/215312880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2e981b0 0x55b0b3266850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.910+0000 7fa1bd190740 1 --2- 172.21.15.16:0/215312880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 0x55b0b3266310 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.910+0000 7fa1bd190740 1 -- 172.21.15.16:0/215312880 >> 172.21.15.16:0/215312880 conn(0x55b0b2eb5460 msgr2=0x55b0b3264200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.910+0000 7fa1bd190740 1 -- 172.21.15.16:0/215312880 shutdown_connections 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.910+0000 7fa1bd190740 1 -- 172.21.15.16:0/215312880 wait complete. 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.910+0000 7fa1bd190740 1 -- start start 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.910+0000 7fa1bd190740 1 Processor -- start 2024-08-28T19:36:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.910+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2e981b0 0x55b0b3275ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2ea0390 0x55b0b3276400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 0x55b0b3282150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1bd190740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b0b302f700 con 0x55b0b2ec1070 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1bd190740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b0b3153750 con 0x55b0b2ea0390 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1bd190740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b0b3278df0 con 0x55b0b2e981b0 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1bd190740 10 client.0 _ll_register_callbacks cb 0x55b0b3182e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 0x55b0b3282150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 0x55b0b3282150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55956/0 (socket says 172.21.15.16:55956) 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1b49b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2ea0390 0x55b0b3276400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1a7fff640 1 -- 172.21.15.16:0/3765573887 learned_addr learned my addr 172.21.15.16:0/3765573887 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2e981b0 0x55b0b3275ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1b49b7640 1 -- 172.21.15.16:0/3765573887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2e981b0 msgr2=0x55b0b3275ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1b49b7640 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2e981b0 0x55b0b3275ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1b49b7640 1 -- 172.21.15.16:0/3765573887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 msgr2=0x55b0b3282150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1b49b7640 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 0x55b0b3282150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1b49b7640 1 -- 172.21.15.16:0/3765573887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0b318c900 con 0x55b0b2ea0390 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa1b49b7640 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2ea0390 0x55b0b3276400 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55b0b3264f90 tx=0x7fa1ac011bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eae22f9fa3c6b63 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.911+0000 7fa18ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa18ffff640 1 -- 172.21.15.16:0/3765573887 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac01e060 con 0x55b0b2ea0390 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa1bd190740 1 -- 172.21.15.16:0/3765573887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1a0031620 con 0x55b0b2ea0390 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa18ffff640 1 -- 172.21.15.16:0/3765573887 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7fa1ac0193f0 con 0x55b0b2ea0390 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa18ffff640 1 -- 172.21.15.16:0/3765573887 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7fa1ac025050 con 0x55b0b2ea0390 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa1bd190740 10 client.28619 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa1bd190740 1 -- 172.21.15.16:0/3765573887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55b0b3284ce0 con 0x55b0b2ea0390 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa18ffff640 1 -- 172.21.15.16:0/3765573887 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1a0031620 con 0x55b0b2ea0390 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa18ffff640 1 -- 172.21.15.16:0/3765573887 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7fa1ac018030 con 0x55b0b2ea0390 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa1bd190740 10 client.28619 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:09.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa1bd190740 10 client.28619 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa1bd190740 20 client.28619 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa1bd190740 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55b0b2e7f9f0 0x55b0b32881e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa1bd190740 4 client.28619 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.912+0000 7fa1bd190740 1 -- 172.21.15.16:0/3765573887 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0b31df5d0 con 0x55b0b2e7f9f0 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.913+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55b0b2e7f9f0 0x55b0b32881e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.913+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55b0b2e7f9f0 0x55b0b32881e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=94ca9e35b189c1c7 server_cookie=f8c2d1ee99c18bbe in_seq=0 out_seq=0 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.913+0000 7fa18ffff640 10 client.28619 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa18ffff640 1 -- 172.21.15.16:0/3765573887 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b0b31df5d0 con 0x55b0b2e7f9f0 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa18ffff640 10 client.28619 handle_command_reply: tid=0 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 1 client.28619 shutdown 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 1 -- 172.21.15.16:0/3765573887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2ea0390 msgr2=0x55b0b3276400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2ea0390 0x55b0b3276400 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55b0b3264f90 tx=0x7fa1ac011bd0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 1 -- 172.21.15.16:0/3765573887 shutdown_connections 2024-08-28T19:36:09.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55b0b2e7f9f0 0x55b0b32881e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2ec1070 0x55b0b3282150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2ea0390 0x55b0b3276400 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa18ffff640 0 client.28619 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:36:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 1 --2- 172.21.15.16:0/3765573887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2e981b0 0x55b0b3275ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 1 -- 172.21.15.16:0/3765573887 >> 172.21.15.16:0/3765573887 conn(0x55b0b2eb5460 msgr2=0x55b0b32627c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 1 -- 172.21.15.16:0/3765573887 shutdown_connections 2024-08-28T19:36:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 1 -- 172.21.15.16:0/3765573887 wait complete. 2024-08-28T19:36:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.914+0000 7fa1bd190740 20 client.28619 trim_cache size 0 max 16384 2024-08-28T19:36:09.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.928+0000 7fa1bd190740 1 Processor -- start 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.928+0000 7fa1bd190740 1 -- start start 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.928+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2eb5460 0x55b0b326be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.928+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e7f9f0 0x55b0b3267a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.928+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2e981b0 0x55b0b3282050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.928+0000 7fa1bd190740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b01aebd0 con 0x55b0b2e7f9f0 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.928+0000 7fa1bd190740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b01b09e0 con 0x55b0b2eb5460 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.928+0000 7fa1bd190740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b01af330 con 0x55b0b2e981b0 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.928+0000 7fa1b49b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e7f9f0 0x55b0b3267a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.928+0000 7fa1b49b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e7f9f0 0x55b0b3267a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55968/0 (socket says 172.21.15.16:55968) 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.928+0000 7fa1b49b7640 1 -- 172.21.15.16:0/1293229073 learned_addr learned my addr 172.21.15.16:0/1293229073 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/1293229073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2eb5460 0x55b0b326be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/1293229073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2e981b0 0x55b0b3282050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1b49b7640 1 -- 172.21.15.16:0/1293229073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2e981b0 msgr2=0x55b0b3282050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1b49b7640 1 --2- 172.21.15.16:0/1293229073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2e981b0 0x55b0b3282050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1b49b7640 1 -- 172.21.15.16:0/1293229073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2eb5460 msgr2=0x55b0b326be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1b49b7640 1 --2- 172.21.15.16:0/1293229073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2eb5460 0x55b0b326be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1b49b7640 1 -- 172.21.15.16:0/1293229073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0b3284af0 con 0x55b0b2e7f9f0 2024-08-28T19:36:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/1293229073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2eb5460 0x55b0b326be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1b49b7640 1 --2- 172.21.15.16:0/1293229073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e7f9f0 0x55b0b3267a50 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac004810 tx=0x7fa1ac00b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e993a3fb119a9948 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1967fc640 1 -- 172.21.15.16:0/1293229073 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac00fe80 con 0x55b0b2e7f9f0 2024-08-28T19:36:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1967fc640 1 -- 172.21.15.16:0/1293229073 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1ac00e810 con 0x55b0b2e7f9f0 2024-08-28T19:36:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.929+0000 7fa1967fc640 1 -- 172.21.15.16:0/1293229073 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac00da60 con 0x55b0b2e7f9f0 2024-08-28T19:36:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.930+0000 7fa1bd190740 1 -- 172.21.15.16:0/1293229073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e7f9f0 msgr2=0x55b0b3267a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.930+0000 7fa1bd190740 1 --2- 172.21.15.16:0/1293229073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e7f9f0 0x55b0b3267a50 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac004810 tx=0x7fa1ac00b4b0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.930+0000 7fa1bd190740 1 -- 172.21.15.16:0/1293229073 shutdown_connections 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.930+0000 7fa1bd190740 1 --2- 172.21.15.16:0/1293229073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2e981b0 0x55b0b3282050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.930+0000 7fa1bd190740 1 --2- 172.21.15.16:0/1293229073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e7f9f0 0x55b0b3267a50 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.930+0000 7fa1bd190740 1 --2- 172.21.15.16:0/1293229073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2eb5460 0x55b0b326be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.930+0000 7fa1bd190740 1 -- 172.21.15.16:0/1293229073 >> 172.21.15.16:0/1293229073 conn(0x55b0b3288530 msgr2=0x55b0b3267340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.930+0000 7fa1bd190740 1 -- 172.21.15.16:0/1293229073 shutdown_connections 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.930+0000 7fa1bd190740 1 -- 172.21.15.16:0/1293229073 wait complete. 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1bd190740 1 -- start start 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1bd190740 1 Processor -- start 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2e7f9f0 0x55b0b327a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e981b0 0x55b0b327af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1bd190740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2eb5460 0x55b0b32ebb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1bd190740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b01b0250 con 0x55b0b2e981b0 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1bd190740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b01afac0 con 0x55b0b2e7f9f0 2024-08-28T19:36:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1bd190740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b01acbd0 con 0x55b0b2eb5460 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1bd190740 10 client.0 _ll_register_callbacks cb 0x7fa1ac00bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1b49b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e981b0 0x55b0b327af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1b49b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e981b0 0x55b0b327af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55970/0 (socket says 172.21.15.16:55970) 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1b49b7640 1 -- 172.21.15.16:0/90343171 learned_addr learned my addr 172.21.15.16:0/90343171 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2e7f9f0 0x55b0b327a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.931+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2eb5460 0x55b0b32ebb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa1b49b7640 1 -- 172.21.15.16:0/90343171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2eb5460 msgr2=0x55b0b32ebb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa1b49b7640 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2eb5460 0x55b0b32ebb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa1b49b7640 1 -- 172.21.15.16:0/90343171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2e7f9f0 msgr2=0x55b0b327a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa1b49b7640 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2e7f9f0 0x55b0b327a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa1b49b7640 1 -- 172.21.15.16:0/90343171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0b32ec1e0 con 0x55b0b2e981b0 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa1b49b7640 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e981b0 0x55b0b327af20 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00d370 tx=0x7fa1ac00feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce285b663e730170 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:09.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa18ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa18ffff640 1 -- 172.21.15.16:0/90343171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac038020 con 0x55b0b2e981b0 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa18ffff640 1 -- 172.21.15.16:0/90343171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7fa1ac00e360 con 0x55b0b2e981b0 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa1bd190740 1 -- 172.21.15.16:0/90343171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1ac00e810 con 0x55b0b2e981b0 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa18ffff640 1 -- 172.21.15.16:0/90343171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7fa1ac043050 con 0x55b0b2e981b0 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa1bd190740 10 client.19194 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa1bd190740 1 -- 172.21.15.16:0/90343171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55b0b32f3500 con 0x55b0b2e981b0 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.932+0000 7fa18ffff640 1 -- 172.21.15.16:0/90343171 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1ac00e810 con 0x55b0b2e981b0 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.933+0000 7fa18ffff640 1 -- 172.21.15.16:0/90343171 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7fa1ac018040 con 0x55b0b2e981b0 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.933+0000 7fa1bd190740 10 client.19194 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.933+0000 7fa1bd190740 10 client.19194 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.933+0000 7fa1bd190740 20 client.19194 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.933+0000 7fa1bd190740 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55b0b2ea0390 0x55b0b32f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.933+0000 7fa1bd190740 4 client.19194 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:36:09.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.933+0000 7fa1bd190740 1 -- 172.21.15.16:0/90343171 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b0b31db070 con 0x55b0b2ea0390 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.933+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55b0b2ea0390 0x55b0b32f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.933+0000 7fa1a7fff640 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55b0b2ea0390 0x55b0b32f5bf0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=abb4973f0b9e431 server_cookie=9b4028a5355adb22 in_seq=0 out_seq=0 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.933+0000 7fa18ffff640 10 client.19194 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa18ffff640 1 -- 172.21.15.16:0/90343171 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b0b31db070 con 0x55b0b2ea0390 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa18ffff640 10 client.19194 handle_command_reply: tid=0 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 1 client.19194 shutdown 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 1 -- 172.21.15.16:0/90343171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e981b0 msgr2=0x55b0b327af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e981b0 0x55b0b327af20 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00d370 tx=0x7fa1ac00feb0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 1 -- 172.21.15.16:0/90343171 shutdown_connections 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55b0b2ea0390 0x55b0b32f5bf0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b0b2eb5460 0x55b0b32ebb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0b2e981b0 0x55b0b327af20 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 1 --2- 172.21.15.16:0/90343171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0b2e7f9f0 0x55b0b327a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 1 -- 172.21.15.16:0/90343171 >> 172.21.15.16:0/90343171 conn(0x55b0b3288530 msgr2=0x55b0b326bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:09.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa18ffff640 0 client.19194 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:36:09.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 1 -- 172.21.15.16:0/90343171 shutdown_connections 2024-08-28T19:36:09.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 1 -- 172.21.15.16:0/90343171 wait complete. 2024-08-28T19:36:09.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.934+0000 7fa1bd190740 20 client.19194 trim_cache size 0 max 16384 2024-08-28T19:36:09.943 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:36:09.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/1591534454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1980b4ca0 msgr2=0x7fa1980b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:09.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1980b4ca0 0x7fa1980b71b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80e3460 tx=0x7fa1a8078550 comp rx=0 tx=0).stop 2024-08-28T19:36:09.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/1591534454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b0108270 msgr2=0x7fa1b019d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:09.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b0108270 0x7fa1b019d940 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800eed0 tx=0x7fa1a800d7b0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/1591534454 shutdown_connections 2024-08-28T19:36:09.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b0108270 0x7fa1b019d940 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b0107920 0x7fa1b01986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa1980b4ca0 0x7fa1980b71b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 --2- 172.21.15.16:0/1591534454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b0101fe0 0x7fa1b0198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:09.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/1591534454 >> 172.21.15.16:0/1591534454 conn(0x7fa1b0079d30 msgr2=0x7fa1b0105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:09.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/1591534454 shutdown_connections 2024-08-28T19:36:09.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:09.935+0000 7fa1b6c1b640 1 -- 172.21.15.16:0/1591534454 wait complete. 2024-08-28T19:36:10.466 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 19.28 MB/sec warmup 106 sec latency 22556.480 ms 2024-08-28T19:36:10.723 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:36:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:11.466 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 19.10 MB/sec warmup 107 sec latency 23556.541 ms 2024-08-28T19:36:11.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.491+0000 7f4a42ff5640 1 -- 172.21.15.16:0/398266913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c1036f0 msgr2=0x7f4a3c10f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.491+0000 7f4a42ff5640 1 --2- 172.21.15.16:0/398266913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c1036f0 0x7f4a3c10f2a0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f4a34009ca0 tx=0x7f4a3402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.491+0000 7f4a42ff5640 1 -- 172.21.15.16:0/398266913 shutdown_connections 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.491+0000 7f4a42ff5640 1 --2- 172.21.15.16:0/398266913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c1036f0 0x7f4a3c10f2a0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.491+0000 7f4a42ff5640 1 --2- 172.21.15.16:0/398266913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a3c102da0 0x7f4a3c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.491+0000 7f4a42ff5640 1 --2- 172.21.15.16:0/398266913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a3c107430 0x7f4a3c107840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.491+0000 7f4a42ff5640 1 -- 172.21.15.16:0/398266913 >> 172.21.15.16:0/398266913 conn(0x7f4a3c079c40 msgr2=0x7f4a3c07a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.491+0000 7f4a42ff5640 1 -- 172.21.15.16:0/398266913 shutdown_connections 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.491+0000 7f4a42ff5640 1 -- 172.21.15.16:0/398266913 wait complete. 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a42ff5640 1 Processor -- start 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a42ff5640 1 -- start start 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a42ff5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c102da0 0x7f4a3c10fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a42ff5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a3c1036f0 0x7f4a3c110370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a42ff5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a3c107430 0x7f4a3c1108b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a42ff5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c113900 con 0x7f4a3c102da0 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a42ff5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c113780 con 0x7f4a3c1036f0 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a42ff5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c113a80 con 0x7f4a3c107430 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a40d91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c102da0 0x7f4a3c10fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a40d91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c102da0 0x7f4a3c10fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56000/0 (socket says 172.21.15.16:56000) 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a40d91640 1 -- 172.21.15.16:0/3352078196 learned_addr learned my addr 172.21.15.16:0/3352078196 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a41592640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a3c107430 0x7f4a3c1108b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.492+0000 7f4a3bfff640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a3c1036f0 0x7f4a3c110370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.493+0000 7f4a40d91640 1 -- 172.21.15.16:0/3352078196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a3c107430 msgr2=0x7f4a3c1108b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.493+0000 7f4a40d91640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a3c107430 0x7f4a3c1108b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.493+0000 7f4a40d91640 1 -- 172.21.15.16:0/3352078196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a3c1036f0 msgr2=0x7f4a3c110370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.493+0000 7f4a40d91640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a3c1036f0 0x7f4a3c110370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.493+0000 7f4a40d91640 1 -- 172.21.15.16:0/3352078196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3c1a8510 con 0x7f4a3c102da0 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.493+0000 7f4a40d91640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c102da0 0x7f4a3c10fe30 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000d950 tx=0x7f4a3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.493+0000 7f4a39ffb640 1 -- 172.21.15.16:0/3352078196 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30017030 con 0x7f4a3c102da0 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.493+0000 7f4a39ffb640 1 -- 172.21.15.16:0/3352078196 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a30004020 con 0x7f4a3c102da0 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.493+0000 7f4a42ff5640 1 -- 172.21.15.16:0/3352078196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3c1a8800 con 0x7f4a3c102da0 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.493+0000 7f4a39ffb640 1 -- 172.21.15.16:0/3352078196 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a300047a0 con 0x7f4a3c102da0 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.493+0000 7f4a42ff5640 1 -- 172.21.15.16:0/3352078196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3c1a8d40 con 0x7f4a3c102da0 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a39ffb640 1 -- 172.21.15.16:0/3352078196 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4a30009a30 con 0x7f4a3c102da0 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a4956a740 1 Processor -- start 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a4956a740 1 -- start start 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0267a4070 0x55f026b49310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b49850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f026783390 0x55f026b49d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a4956a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f026ac0680 con 0x55f02677b1b0 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a4956a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f026aafc10 con 0x55f026783390 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a4956a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f026ab3bc0 con 0x55f0267a4070 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a39ffb640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a2c0b4ca0 0x7f4a2c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a40d91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b49850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.496+0000 7f4a40d91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b49850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56016/0 (socket says 172.21.15.16:56016) 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a40d91640 1 -- 172.21.15.16:0/4103919025 learned_addr learned my addr 172.21.15.16:0/4103919025 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a41592640 1 --2- 172.21.15.16:0/4103919025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0267a4070 0x55f026b49310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a3bfff640 1 --2- 172.21.15.16:0/4103919025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f026783390 0x55f026b49d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a41592640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a2c0b4ca0 0x7f4a2c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a39ffb640 1 -- 172.21.15.16:0/3352078196 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7f4a300e53b0 con 0x7f4a3c102da0 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a40d91640 1 -- 172.21.15.16:0/4103919025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0267a4070 msgr2=0x55f026b49310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a40d91640 1 --2- 172.21.15.16:0/4103919025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0267a4070 0x55f026b49310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a40d91640 1 -- 172.21.15.16:0/4103919025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f026783390 msgr2=0x55f026b49d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a40d91640 1 --2- 172.21.15.16:0/4103919025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f026783390 0x55f026b49d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a40d91640 1 -- 172.21.15.16:0/4103919025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f02680f650 con 0x55f02677b1b0 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a40d91640 1 --2- 172.21.15.16:0/4103919025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b49850 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f4a300e26e0 tx=0x7f4a30005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d18c91698264c6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a41592640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a2c0b4ca0 0x7f4a2c0b71b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4a34006010 tx=0x7f4a340095c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a26ffd640 1 -- 172.21.15.16:0/4103919025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a300ab130 con 0x55f02677b1b0 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a26ffd640 1 -- 172.21.15.16:0/4103919025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a300b4270 con 0x55f02677b1b0 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.497+0000 7f4a26ffd640 1 -- 172.21.15.16:0/4103919025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a300a78b0 con 0x55f02677b1b0 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.498+0000 7f4a4956a740 1 -- 172.21.15.16:0/4103919025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 msgr2=0x55f026b49850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.498+0000 7f4a4956a740 1 --2- 172.21.15.16:0/4103919025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b49850 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f4a300e26e0 tx=0x7f4a30005db0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.498+0000 7f4a4956a740 1 -- 172.21.15.16:0/4103919025 shutdown_connections 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.498+0000 7f4a4956a740 1 --2- 172.21.15.16:0/4103919025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f026783390 0x55f026b49d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.498+0000 7f4a4956a740 1 --2- 172.21.15.16:0/4103919025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b49850 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.498+0000 7f4a4956a740 1 --2- 172.21.15.16:0/4103919025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f0267a4070 0x55f026b49310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.498+0000 7f4a4956a740 1 -- 172.21.15.16:0/4103919025 >> 172.21.15.16:0/4103919025 conn(0x55f026798460 msgr2=0x55f026b47200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.498+0000 7f4a4956a740 1 -- 172.21.15.16:0/4103919025 shutdown_connections 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.498+0000 7f4a4956a740 1 -- 172.21.15.16:0/4103919025 wait complete. 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a4956a740 1 -- start start 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a4956a740 1 Processor -- start 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f02677b1b0 0x55f026b54a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f026783390 0x55f026b54fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0267a4070 0x55f026b60d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a4956a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f026912700 con 0x55f026783390 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a4956a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f026a36750 con 0x55f0267a4070 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a4956a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f026b57a30 con 0x55f02677b1b0 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a4956a740 10 client.0 _ll_register_callbacks cb 0x55f026a65e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f026783390 0x55f026b54fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f026783390 0x55f026b54fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56026/0 (socket says 172.21.15.16:56026) 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.499+0000 7f4a3bfff640 1 -- 172.21.15.16:0/4038634318 learned_addr learned my addr 172.21.15.16:0/4038634318 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a40d91640 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f02677b1b0 0x55f026b54a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a41592640 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0267a4070 0x55f026b60d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a3bfff640 1 -- 172.21.15.16:0/4038634318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f02677b1b0 msgr2=0x55f026b54a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a3bfff640 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f02677b1b0 0x55f026b54a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a3bfff640 1 -- 172.21.15.16:0/4038634318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0267a4070 msgr2=0x55f026b60d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a3bfff640 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0267a4070 0x55f026b60d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a3bfff640 1 -- 172.21.15.16:0/4038634318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f026a6f900 con 0x55f026783390 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a41592640 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0267a4070 0x55f026b60d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a3bfff640 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f026783390 0x55f026b54fd0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800bf50 tx=0x7f4a280118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c626b437157e0f6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a1bfff640 1 -- 172.21.15.16:0/4038634318 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28005020 con 0x55f026783390 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a1bfff640 1 -- 172.21.15.16:0/4038634318 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7f4a280051c0 con 0x55f026783390 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a4956a740 1 -- 172.21.15.16:0/4038634318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a300b4270 con 0x55f026783390 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a1bfff640 1 -- 172.21.15.16:0/4038634318 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f4a28025050 con 0x55f026783390 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a4956a740 10 client.19206 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.500+0000 7f4a4956a740 1 -- 172.21.15.16:0/4038634318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55f026b63810 con 0x55f026783390 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.501+0000 7f4a1bfff640 1 -- 172.21.15.16:0/4038634318 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a300b4270 con 0x55f026783390 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.501+0000 7f4a1bfff640 1 -- 172.21.15.16:0/4038634318 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f4a28010710 con 0x55f026783390 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.501+0000 7f4a4956a740 10 client.19206 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.501+0000 7f4a4956a740 10 client.19206 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.501+0000 7f4a4956a740 20 client.19206 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.501+0000 7f4a4956a740 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f0267629f0 0x55f026b66df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.501+0000 7f4a4956a740 4 client.19206 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.501+0000 7f4a4956a740 1 -- 172.21.15.16:0/4038634318 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f026ac25d0 con 0x55f0267629f0 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.501+0000 7f4a40d91640 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f0267629f0 0x55f026b66df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.502+0000 7f4a1bfff640 10 client.19206 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.502+0000 7f4a40d91640 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f0267629f0 0x55f026b66df0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ab16cf2120987f39 server_cookie=2f011b2d273a9585 in_seq=0 out_seq=0 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a1bfff640 1 -- 172.21.15.16:0/4038634318 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f026ac25d0 con 0x55f0267629f0 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a1bfff640 10 client.19206 handle_command_reply: tid=0 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 1 client.19206 shutdown 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 1 -- 172.21.15.16:0/4038634318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f026783390 msgr2=0x55f026b54fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f026783390 0x55f026b54fd0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800bf50 tx=0x7f4a280118d0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 1 -- 172.21.15.16:0/4038634318 shutdown_connections 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f0267629f0 0x55f026b66df0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0267a4070 0x55f026b60d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f026783390 0x55f026b54fd0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 1 --2- 172.21.15.16:0/4038634318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f02677b1b0 0x55f026b54a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a1bfff640 0 client.19206 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 1 -- 172.21.15.16:0/4038634318 >> 172.21.15.16:0/4038634318 conn(0x55f026798460 msgr2=0x55f026b4a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 1 -- 172.21.15.16:0/4038634318 shutdown_connections 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 1 -- 172.21.15.16:0/4038634318 wait complete. 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.503+0000 7f4a4956a740 20 client.19206 trim_cache size 0 max 16384 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.516+0000 7f4a4956a740 1 Processor -- start 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.516+0000 7f4a4956a740 1 -- start start 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.516+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0267629f0 0x55f026b566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.516+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f02677b1b0 0x55f026b60c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.516+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f026783390 0x55f026b51c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a4956a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1b3040 con 0x55f0267629f0 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a4956a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1b4e50 con 0x55f026783390 2024-08-28T19:36:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a4956a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1b37a0 con 0x55f02677b1b0 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a40d91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0267629f0 0x55f026b566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a40d91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0267629f0 0x55f026b566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56036/0 (socket says 172.21.15.16:56036) 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a40d91640 1 -- 172.21.15.16:0/1976841195 learned_addr learned my addr 172.21.15.16:0/1976841195 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a3bfff640 1 --2- 172.21.15.16:0/1976841195 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f02677b1b0 0x55f026b60c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a40d91640 1 -- 172.21.15.16:0/1976841195 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f02677b1b0 msgr2=0x55f026b60c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a40d91640 1 --2- 172.21.15.16:0/1976841195 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f02677b1b0 0x55f026b60c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a40d91640 1 -- 172.21.15.16:0/1976841195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f026783390 msgr2=0x55f026b51c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a40d91640 1 --2- 172.21.15.16:0/1976841195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f026783390 0x55f026b51c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a40d91640 1 -- 172.21.15.16:0/1976841195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f026b52f90 con 0x55f0267629f0 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a3bfff640 1 --2- 172.21.15.16:0/1976841195 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f02677b1b0 0x55f026b60c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a40d91640 1 --2- 172.21.15.16:0/1976841195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0267629f0 0x55f026b566c0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4a30004770 tx=0x7f4a300ab230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228b7db75da7a2a1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a267fc640 1 -- 172.21.15.16:0/1976841195 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a300afe90 con 0x55f0267629f0 2024-08-28T19:36:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a267fc640 1 -- 172.21.15.16:0/1976841195 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a300b4270 con 0x55f0267629f0 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.517+0000 7f4a267fc640 1 -- 172.21.15.16:0/1976841195 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a300ab940 con 0x55f0267629f0 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.518+0000 7f4a4956a740 1 -- 172.21.15.16:0/1976841195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0267629f0 msgr2=0x55f026b566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.518+0000 7f4a4956a740 1 --2- 172.21.15.16:0/1976841195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0267629f0 0x55f026b566c0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4a30004770 tx=0x7f4a300ab230 comp rx=0 tx=0).stop 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.518+0000 7f4a4956a740 1 -- 172.21.15.16:0/1976841195 shutdown_connections 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.518+0000 7f4a4956a740 1 --2- 172.21.15.16:0/1976841195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f026783390 0x55f026b51c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.518+0000 7f4a4956a740 1 --2- 172.21.15.16:0/1976841195 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f02677b1b0 0x55f026b60c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.518+0000 7f4a4956a740 1 --2- 172.21.15.16:0/1976841195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f0267629f0 0x55f026b566c0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.518+0000 7f4a4956a740 1 -- 172.21.15.16:0/1976841195 >> 172.21.15.16:0/1976841195 conn(0x55f026798460 msgr2=0x55f026b55fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.518+0000 7f4a4956a740 1 -- 172.21.15.16:0/1976841195 shutdown_connections 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.518+0000 7f4a4956a740 1 -- 172.21.15.16:0/1976841195 wait complete. 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.518+0000 7f4a4956a740 1 -- start start 2024-08-28T19:36:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.518+0000 7f4a4956a740 1 Processor -- start 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b5d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f026783390 0x55f026b598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a4956a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0267a4070 0x55f026b59e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a4956a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1b46c0 con 0x55f02677b1b0 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a4956a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1b3f30 con 0x55f0267a4070 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a4956a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1b1040 con 0x55f026783390 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a4956a740 10 client.0 _ll_register_callbacks cb 0x7f4a2800b820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a40d91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b5d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a40d91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b5d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56052/0 (socket says 172.21.15.16:56052) 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a40d91640 1 -- 172.21.15.16:0/3837880253 learned_addr learned my addr 172.21.15.16:0/3837880253 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a41592640 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0267a4070 0x55f026b59e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a3bfff640 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f026783390 0x55f026b598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a40d91640 1 -- 172.21.15.16:0/3837880253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f026783390 msgr2=0x55f026b598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a40d91640 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f026783390 0x55f026b598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a40d91640 1 -- 172.21.15.16:0/3837880253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0267a4070 msgr2=0x55f026b59e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a40d91640 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0267a4070 0x55f026b59e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a40d91640 1 -- 172.21.15.16:0/3837880253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f026b5a550 con 0x55f02677b1b0 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a40d91640 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b5d410 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f4a300a8890 tx=0x7f4a300b49d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=addbd489bf51d41 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.519+0000 7f4a1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a1bfff640 1 -- 172.21.15.16:0/3837880253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a300afe90 con 0x55f02677b1b0 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a1bfff640 1 -- 172.21.15.16:0/3837880253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7f4a300041d0 con 0x55f02677b1b0 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a4956a740 1 -- 172.21.15.16:0/3837880253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a300b4270 con 0x55f02677b1b0 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a1bfff640 1 -- 172.21.15.16:0/3837880253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f4a300c60d0 con 0x55f02677b1b0 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a4956a740 10 client.19212 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a4956a740 1 -- 172.21.15.16:0/3837880253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55f026bd25e0 con 0x55f02677b1b0 2024-08-28T19:36:11.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a1bfff640 1 -- 172.21.15.16:0/3837880253 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a300b4270 con 0x55f02677b1b0 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a1bfff640 1 -- 172.21.15.16:0/3837880253 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f4a300d2170 con 0x55f02677b1b0 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a4956a740 10 client.19212 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a4956a740 10 client.19212 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a4956a740 20 client.19212 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a4956a740 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f026737da0 0x55f026bd4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a4956a740 4 client.19212 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.520+0000 7f4a4956a740 1 -- 172.21.15.16:0/3837880253 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f026abe070 con 0x55f026737da0 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a3bfff640 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f026737da0 0x55f026bd4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a1bfff640 10 client.19212 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a3bfff640 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f026737da0 0x55f026bd4810 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=17e58c385f1bcbff server_cookie=a7993969daae4e19 in_seq=0 out_seq=0 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a1bfff640 1 -- 172.21.15.16:0/3837880253 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f026abe070 con 0x55f026737da0 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a1bfff640 10 client.19212 handle_command_reply: tid=0 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a4956a740 1 client.19212 shutdown 2024-08-28T19:36:11.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a4956a740 1 -- 172.21.15.16:0/3837880253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 msgr2=0x55f026b5d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a4956a740 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b5d410 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f4a300a8890 tx=0x7f4a300b49d0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a4956a740 1 -- 172.21.15.16:0/3837880253 shutdown_connections 2024-08-28T19:36:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a4956a740 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f026737da0 0x55f026bd4810 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a1bfff640 0 client.19212 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:36:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a4956a740 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0267a4070 0x55f026b59e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a4956a740 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f026783390 0x55f026b598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a4956a740 1 --2- 172.21.15.16:0/3837880253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f02677b1b0 0x55f026b5d410 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.521+0000 7f4a4956a740 1 -- 172.21.15.16:0/3837880253 >> 172.21.15.16:0/3837880253 conn(0x55f026798460 msgr2=0x55f026b47170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:11.617 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a4956a740 1 -- 172.21.15.16:0/3837880253 shutdown_connections 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a4956a740 1 -- 172.21.15.16:0/3837880253 wait complete. 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a4956a740 20 client.19212 trim_cache size 0 max 16384 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 -- 172.21.15.16:0/3352078196 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a2c0b4ca0 msgr2=0x7f4a2c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a2c0b4ca0 0x7f4a2c0b71b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4a34006010 tx=0x7f4a340095c0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 -- 172.21.15.16:0/3352078196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c102da0 msgr2=0x7f4a3c10fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c102da0 0x7f4a3c10fe30 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000d950 tx=0x7f4a3000de10 comp rx=0 tx=0).stop 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 -- 172.21.15.16:0/3352078196 shutdown_connections 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a3c107430 0x7f4a3c1108b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a2c0b4ca0 0x7f4a2c0b71b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a3c1036f0 0x7f4a3c110370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 --2- 172.21.15.16:0/3352078196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c102da0 0x7f4a3c10fe30 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:11.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 -- 172.21.15.16:0/3352078196 >> 172.21.15.16:0/3352078196 conn(0x7f4a3c079c40 msgr2=0x7f4a3c103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:11.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 -- 172.21.15.16:0/3352078196 shutdown_connections 2024-08-28T19:36:11.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:11.522+0000 7f4a42ff5640 1 -- 172.21.15.16:0/3352078196 wait complete. 2024-08-28T19:36:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:11 smithi115 ceph-mon[22184]: pgmap v810: 129 pgs: 129 active+clean; 873 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 6.8 MiB/s wr, 278 op/s 2024-08-28T19:36:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:11 smithi016 ceph-mon[18556]: pgmap v810: 129 pgs: 129 active+clean; 873 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 6.8 MiB/s wr, 278 op/s 2024-08-28T19:36:11.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:11 smithi028 ceph-mon[21769]: pgmap v810: 129 pgs: 129 active+clean; 873 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 6.8 MiB/s wr, 278 op/s 2024-08-28T19:36:12.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:36:12.466 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 18.92 MB/sec warmup 108 sec latency 24556.590 ms 2024-08-28T19:36:12.578 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:13.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.060+0000 7f52155ad640 1 -- 172.21.15.16:0/2084511819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5210109240 msgr2=0x7f5210109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:13.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.060+0000 7f52155ad640 1 --2- 172.21.15.16:0/2084511819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5210109240 0x7f5210109650 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f5204009900 tx=0x7f520402a410 comp rx=0 tx=0).stop 2024-08-28T19:36:13.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.060+0000 7f52155ad640 1 -- 172.21.15.16:0/2084511819 shutdown_connections 2024-08-28T19:36:13.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.060+0000 7f52155ad640 1 --2- 172.21.15.16:0/2084511819 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5210109b90 0x7f52101103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.060+0000 7f52155ad640 1 --2- 172.21.15.16:0/2084511819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5210109240 0x7f5210109650 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.060+0000 7f52155ad640 1 --2- 172.21.15.16:0/2084511819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5210103900 0x7f5210103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.060+0000 7f52155ad640 1 -- 172.21.15.16:0/2084511819 >> 172.21.15.16:0/2084511819 conn(0x7f521007b5f0 msgr2=0x7f521007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:13.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.060+0000 7f52155ad640 1 -- 172.21.15.16:0/2084511819 shutdown_connections 2024-08-28T19:36:13.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.060+0000 7f52155ad640 1 -- 172.21.15.16:0/2084511819 wait complete. 2024-08-28T19:36:13.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f52155ad640 1 Processor -- start 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f52155ad640 1 -- start start 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f52155ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5210103900 0x7f521019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f52155ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5210109240 0x7f521019a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f52155ad640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5210109b90 0x7f521019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f52155ad640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5210113850 con 0x7f5210109240 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f52155ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52101136d0 con 0x7f5210103900 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f52155ad640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f52101139d0 con 0x7f5210109b90 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f520e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5210109240 0x7f521019a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f520e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5210109240 0x7f521019a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56078/0 (socket says 172.21.15.16:56078) 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f520e7fc640 1 -- 172.21.15.16:0/2846731244 learned_addr learned my addr 172.21.15.16:0/2846731244 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f520f7fe640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5210109b90 0x7f521019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.061+0000 7f520effd640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5210103900 0x7f521019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.062+0000 7f520e7fc640 1 -- 172.21.15.16:0/2846731244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5210109b90 msgr2=0x7f521019fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.062+0000 7f520e7fc640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5210109b90 0x7f521019fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.062+0000 7f520e7fc640 1 -- 172.21.15.16:0/2846731244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5210103900 msgr2=0x7f521019a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.062+0000 7f520e7fc640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5210103900 0x7f521019a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.062+0000 7f520e7fc640 1 -- 172.21.15.16:0/2846731244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52101a0270 con 0x7f5210109240 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.062+0000 7f520e7fc640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5210109240 0x7f521019a8a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f5204006010 tx=0x7f5204004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.062+0000 7f51f7fff640 1 -- 172.21.15.16:0/2846731244 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520403f030 con 0x7f5210109240 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.062+0000 7f52155ad640 1 -- 172.21.15.16:0/2846731244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52101a0500 con 0x7f5210109240 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.062+0000 7f51f7fff640 1 -- 172.21.15.16:0/2846731244 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5204003aa0 con 0x7f5210109240 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.062+0000 7f51f7fff640 1 -- 172.21.15.16:0/2846731244 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204035070 con 0x7f5210109240 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.062+0000 7f52155ad640 1 -- 172.21.15.16:0/2846731244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52101a6cc0 con 0x7f5210109240 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.065+0000 7f521bb22740 1 Processor -- start 2024-08-28T19:36:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.065+0000 7f521bb22740 1 -- start start 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.065+0000 7f521bb22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e0907070 0x55e3e0cac310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.065+0000 7f521bb22740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.065+0000 7f521bb22740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cacd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f521bb22740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e3e0c23680 con 0x55e3e08de1b0 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f521bb22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e3e0c12c10 con 0x55e3e0907070 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f521bb22740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e3e0c16bc0 con 0x55e3e08e6390 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f520effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f520effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56084/0 (socket says 172.21.15.16:56084) 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f520effd640 1 -- 172.21.15.16:0/3289374562 learned_addr learned my addr 172.21.15.16:0/3289374562 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f520f7fe640 1 --2- 172.21.15.16:0/3289374562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e0907070 0x55e3e0cac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f51f7fff640 1 -- 172.21.15.16:0/2846731244 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5204042030 con 0x7f5210109240 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f520effd640 1 -- 172.21.15.16:0/3289374562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 msgr2=0x55e3e0cacd90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:36:13.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f520effd640 1 --2- 172.21.15.16:0/3289374562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cacd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f520effd640 1 -- 172.21.15.16:0/3289374562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e0907070 msgr2=0x55e3e0cac310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f520effd640 1 --2- 172.21.15.16:0/3289374562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e0907070 0x55e3e0cac310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f520effd640 1 -- 172.21.15.16:0/3289374562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3e0972650 con 0x55e3e08de1b0 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f520effd640 1 --2- 172.21.15.16:0/3289374562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cac850 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f5200009c80 tx=0x7f52000117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570f80e51971d118 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f51f57fa640 1 -- 172.21.15.16:0/3289374562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5200012450 con 0x55e3e08de1b0 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f51f57fa640 1 -- 172.21.15.16:0/3289374562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5200012aa0 con 0x55e3e08de1b0 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.066+0000 7f51f57fa640 1 -- 172.21.15.16:0/3289374562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520000fb80 con 0x55e3e08de1b0 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f51f7fff640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51f80b4ca0 0x7f51f80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f520f7fe640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51f80b4ca0 0x7f51f80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f51f7fff640 1 -- 172.21.15.16:0/2846731244 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7f5204105e40 con 0x7f5210109240 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f520f7fe640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51f80b4ca0 0x7f51f80b71b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5208005fd0 tx=0x7f5208009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f521bb22740 1 -- 172.21.15.16:0/3289374562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 msgr2=0x55e3e0cac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f521bb22740 1 --2- 172.21.15.16:0/3289374562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cac850 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f5200009c80 tx=0x7f52000117b0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f521bb22740 1 -- 172.21.15.16:0/3289374562 shutdown_connections 2024-08-28T19:36:13.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f521bb22740 1 --2- 172.21.15.16:0/3289374562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cacd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f521bb22740 1 --2- 172.21.15.16:0/3289374562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cac850 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f521bb22740 1 --2- 172.21.15.16:0/3289374562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e0907070 0x55e3e0cac310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f521bb22740 1 -- 172.21.15.16:0/3289374562 >> 172.21.15.16:0/3289374562 conn(0x55e3e08fb460 msgr2=0x55e3e0caa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:13.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f521bb22740 1 -- 172.21.15.16:0/3289374562 shutdown_connections 2024-08-28T19:36:13.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.067+0000 7f521bb22740 1 -- 172.21.15.16:0/3289374562 wait complete. 2024-08-28T19:36:13.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f521bb22740 1 -- start start 2024-08-28T19:36:13.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f521bb22740 1 Processor -- start 2024-08-28T19:36:13.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f521bb22740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cb7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f521bb22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08e6390 0x55e3e0cb8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f521bb22740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e0907070 0x55e3e0cc3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f521bb22740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e3e0a75700 con 0x55e3e08de1b0 2024-08-28T19:36:13.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f521bb22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e3e0b99750 con 0x55e3e08e6390 2024-08-28T19:36:13.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f521bb22740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e3e0cbaaf0 con 0x55e3e0907070 2024-08-28T19:36:13.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f521bb22740 10 client.0 _ll_register_callbacks cb 0x55e3e0bc8e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:13.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f520effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cb7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f520effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cb7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56090/0 (socket says 172.21.15.16:56090) 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.068+0000 7f520effd640 1 -- 172.21.15.16:0/3067675586 learned_addr learned my addr 172.21.15.16:0/3067675586 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f520e7fc640 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08e6390 0x55e3e0cb8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f520effd640 1 -- 172.21.15.16:0/3067675586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e0907070 msgr2=0x55e3e0cc3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f520effd640 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e0907070 0x55e3e0cc3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f520effd640 1 -- 172.21.15.16:0/3067675586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08e6390 msgr2=0x55e3e0cb8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f520effd640 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08e6390 0x55e3e0cb8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f520effd640 1 -- 172.21.15.16:0/3067675586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3e0bd2900 con 0x55e3e08de1b0 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f520effd640 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cb7b20 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f5200009c50 tx=0x7f5200009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a7375561a711f3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f51f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f51f2ff5640 1 -- 172.21.15.16:0/3067675586 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5200029020 con 0x55e3e08de1b0 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f51f2ff5640 1 -- 172.21.15.16:0/3067675586 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7f5200012660 con 0x55e3e08de1b0 2024-08-28T19:36:13.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f521bb22740 1 -- 172.21.15.16:0/3067675586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5200012aa0 con 0x55e3e08de1b0 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f51f2ff5640 1 -- 172.21.15.16:0/3067675586 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f5200034050 con 0x55e3e08de1b0 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.069+0000 7f521bb22740 10 client.19224 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.070+0000 7f521bb22740 1 -- 172.21.15.16:0/3067675586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55e3e0cc6920 con 0x55e3e08de1b0 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.070+0000 7f51f2ff5640 1 -- 172.21.15.16:0/3067675586 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5200012aa0 con 0x55e3e08de1b0 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.070+0000 7f51f2ff5640 1 -- 172.21.15.16:0/3067675586 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f52000035f0 con 0x55e3e08de1b0 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.070+0000 7f521bb22740 10 client.19224 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.070+0000 7f521bb22740 10 client.19224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.070+0000 7f521bb22740 20 client.19224 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.070+0000 7f521bb22740 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e3e08c59f0 0x55e3e0cc9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.070+0000 7f521bb22740 4 client.19224 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.070+0000 7f521bb22740 1 -- 172.21.15.16:0/3067675586 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3e0c255d0 con 0x55e3e08c59f0 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.070+0000 7f520e7fc640 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e3e08c59f0 0x55e3e0cc9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.071+0000 7f520e7fc640 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e3e08c59f0 0x55e3e0cc9d90 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8ec7a4676f13d59 server_cookie=b9fc05bab9cb2505 in_seq=0 out_seq=0 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.071+0000 7f51f2ff5640 10 client.19224 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f51f2ff5640 1 -- 172.21.15.16:0/3067675586 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e3e0c255d0 con 0x55e3e08c59f0 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f51f2ff5640 10 client.19224 handle_command_reply: tid=0 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 1 client.19224 shutdown 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 1 -- 172.21.15.16:0/3067675586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 msgr2=0x55e3e0cb7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cb7b20 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f5200009c50 tx=0x7f5200009fa0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 1 -- 172.21.15.16:0/3067675586 shutdown_connections 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e3e08c59f0 0x55e3e0cc9d90 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e0907070 0x55e3e0cc3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08e6390 0x55e3e0cb8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f51f2ff5640 0 client.19224 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 1 --2- 172.21.15.16:0/3067675586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e08de1b0 0x55e3e0cb7b20 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 1 -- 172.21.15.16:0/3067675586 >> 172.21.15.16:0/3067675586 conn(0x55e3e08fb460 msgr2=0x55e3e0caf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:13.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 1 -- 172.21.15.16:0/3067675586 shutdown_connections 2024-08-28T19:36:13.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 1 -- 172.21.15.16:0/3067675586 wait complete. 2024-08-28T19:36:13.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.072+0000 7f521bb22740 20 client.19224 trim_cache size 0 max 16384 2024-08-28T19:36:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.085+0000 7f521bb22740 1 Processor -- start 2024-08-28T19:36:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.085+0000 7f521bb22740 1 -- start start 2024-08-28T19:36:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f521bb22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08de1b0 0x55e3e0caa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f521bb22740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cc3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f521bb22740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cc4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f521bb22740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f52101b0eb0 con 0x55e3e0907070 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f521bb22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52101b2cc0 con 0x55e3e08de1b0 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f521bb22740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f52101b1610 con 0x55e3e08e6390 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f520f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cc4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f520f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cc4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56096/0 (socket says 172.21.15.16:56096) 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f520f7fe640 1 -- 172.21.15.16:0/462326449 learned_addr learned my addr 172.21.15.16:0/462326449 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f520e7fc640 1 --2- 172.21.15.16:0/462326449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cc3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f520effd640 1 --2- 172.21.15.16:0/462326449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08de1b0 0x55e3e0caa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f520f7fe640 1 -- 172.21.15.16:0/462326449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 msgr2=0x55e3e0cc3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f520f7fe640 1 --2- 172.21.15.16:0/462326449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cc3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f520f7fe640 1 -- 172.21.15.16:0/462326449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08de1b0 msgr2=0x55e3e0caa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f520f7fe640 1 --2- 172.21.15.16:0/462326449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08de1b0 0x55e3e0caa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f520f7fe640 1 -- 172.21.15.16:0/462326449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3e0cae600 con 0x55e3e0907070 2024-08-28T19:36:13.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.086+0000 7f520f7fe640 1 --2- 172.21.15.16:0/462326449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cc4250 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f5208011830 tx=0x7f5208015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b11ae54ff52930c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f51f4ff9640 1 -- 172.21.15.16:0/462326449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520801f8b0 con 0x55e3e0907070 2024-08-28T19:36:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f51f4ff9640 1 -- 172.21.15.16:0/462326449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5208020420 con 0x55e3e0907070 2024-08-28T19:36:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f51f4ff9640 1 -- 172.21.15.16:0/462326449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520801d650 con 0x55e3e0907070 2024-08-28T19:36:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f521bb22740 1 -- 172.21.15.16:0/462326449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 msgr2=0x55e3e0cc4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f521bb22740 1 --2- 172.21.15.16:0/462326449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cc4250 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f5208011830 tx=0x7f5208015040 comp rx=0 tx=0).stop 2024-08-28T19:36:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f521bb22740 1 -- 172.21.15.16:0/462326449 shutdown_connections 2024-08-28T19:36:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f521bb22740 1 --2- 172.21.15.16:0/462326449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cc4250 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f521bb22740 1 --2- 172.21.15.16:0/462326449 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cc3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f521bb22740 1 --2- 172.21.15.16:0/462326449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08de1b0 0x55e3e0caa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f521bb22740 1 -- 172.21.15.16:0/462326449 >> 172.21.15.16:0/462326449 conn(0x55e3e08fb460 msgr2=0x55e3e0ca9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f521bb22740 1 -- 172.21.15.16:0/462326449 shutdown_connections 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.087+0000 7f521bb22740 1 -- 172.21.15.16:0/462326449 wait complete. 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f521bb22740 1 -- start start 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f521bb22740 1 Processor -- start 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f521bb22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08de1b0 0x55e3e0cbbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f521bb22740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cbc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f521bb22740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cbc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f521bb22740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f52101b2530 con 0x55e3e0907070 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f521bb22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52101b1da0 con 0x55e3e08de1b0 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f521bb22740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f52101aeeb0 con 0x55e3e08e6390 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f521bb22740 10 client.0 _ll_register_callbacks cb 0x7f5208009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f520f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cbc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f520f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cbc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56104/0 (socket says 172.21.15.16:56104) 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f520f7fe640 1 -- 172.21.15.16:0/2028770664 learned_addr learned my addr 172.21.15.16:0/2028770664 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.088+0000 7f520effd640 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08de1b0 0x55e3e0cbbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f520e7fc640 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cbc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f520f7fe640 1 -- 172.21.15.16:0/2028770664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 msgr2=0x55e3e0cbc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f520f7fe640 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cbc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f520f7fe640 1 -- 172.21.15.16:0/2028770664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08de1b0 msgr2=0x55e3e0cbbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f520f7fe640 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08de1b0 0x55e3e0cbbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f520f7fe640 1 -- 172.21.15.16:0/2028770664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3e0d33720 con 0x55e3e0907070 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f520e7fc640 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cbc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f520f7fe640 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cbc8e0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f5208020d70 tx=0x7f5208004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb79bc0b7c88ff0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f51f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f51f2ff5640 1 -- 172.21.15.16:0/2028770664 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520801f8b0 con 0x55e3e0907070 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f51f2ff5640 1 -- 172.21.15.16:0/2028770664 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9578+0+0 (secure 0 0 0) 0x7f520801fa50 con 0x55e3e0907070 2024-08-28T19:36:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f521bb22740 1 -- 172.21.15.16:0/2028770664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5208020420 con 0x55e3e0907070 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f51f2ff5640 1 -- 172.21.15.16:0/2028770664 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f520803f050 con 0x55e3e0907070 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f521bb22740 10 client.19230 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.089+0000 7f521bb22740 1 -- 172.21.15.16:0/2028770664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55e3e0d34980 con 0x55e3e0907070 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.090+0000 7f51f2ff5640 1 -- 172.21.15.16:0/2028770664 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5208020420 con 0x55e3e0907070 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.090+0000 7f51f2ff5640 1 -- 172.21.15.16:0/2028770664 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f520800f040 con 0x55e3e0907070 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.090+0000 7f521bb22740 10 client.19230 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.090+0000 7f521bb22740 10 client.19230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.090+0000 7f521bb22740 20 client.19230 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.090+0000 7f521bb22740 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e3e089ada0 0x55e3e0d37190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.090+0000 7f521bb22740 4 client.19230 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.090+0000 7f521bb22740 1 -- 172.21.15.16:0/2028770664 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e3e0c21070 con 0x55e3e089ada0 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.090+0000 7f520effd640 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e3e089ada0 0x55e3e0d37190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.090+0000 7f520effd640 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e3e089ada0 0x55e3e0d37190 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcdcd43e5874c89b server_cookie=28e18ad73b3ed47 in_seq=0 out_seq=0 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.090+0000 7f51f2ff5640 10 client.19230 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f51f2ff5640 1 -- 172.21.15.16:0/2028770664 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e3e0c21070 con 0x55e3e089ada0 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f51f2ff5640 10 client.19230 handle_command_reply: tid=0 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 1 client.19230 shutdown 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 1 -- 172.21.15.16:0/2028770664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 msgr2=0x55e3e0cbc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cbc8e0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f5208020d70 tx=0x7f5208004f50 comp rx=0 tx=0).stop 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 1 -- 172.21.15.16:0/2028770664 shutdown_connections 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e3e0907070 0x55e3e0cbc8e0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e3e089ada0 0x55e3e0d37190 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e3e08e6390 0x55e3e0cbc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 1 --2- 172.21.15.16:0/2028770664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e08de1b0 0x55e3e0cbbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f51f2ff5640 0 client.19230 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 1 -- 172.21.15.16:0/2028770664 >> 172.21.15.16:0/2028770664 conn(0x55e3e08fb460 msgr2=0x55e3e0caf410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 1 -- 172.21.15.16:0/2028770664 shutdown_connections 2024-08-28T19:36:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 1 -- 172.21.15.16:0/2028770664 wait complete. 2024-08-28T19:36:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.091+0000 7f521bb22740 20 client.19230 trim_cache size 0 max 16384 2024-08-28T19:36:13.099 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:36:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 -- 172.21.15.16:0/2846731244 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51f80b4ca0 msgr2=0x7f51f80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51f80b4ca0 0x7f51f80b71b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5208005fd0 tx=0x7f5208009600 comp rx=0 tx=0).stop 2024-08-28T19:36:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 -- 172.21.15.16:0/2846731244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5210109240 msgr2=0x7f521019a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5210109240 0x7f521019a8a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f5204006010 tx=0x7f5204004060 comp rx=0 tx=0).stop 2024-08-28T19:36:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 -- 172.21.15.16:0/2846731244 shutdown_connections 2024-08-28T19:36:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5210109b90 0x7f521019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5210109240 0x7f521019a8a0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51f80b4ca0 0x7f51f80b71b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 --2- 172.21.15.16:0/2846731244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5210103900 0x7f521019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 -- 172.21.15.16:0/2846731244 >> 172.21.15.16:0/2846731244 conn(0x7f521007b5f0 msgr2=0x7f5210105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 -- 172.21.15.16:0/2846731244 shutdown_connections 2024-08-28T19:36:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:13.092+0000 7f52155ad640 1 -- 172.21.15.16:0/2846731244 wait complete. 2024-08-28T19:36:13.466 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 18.75 MB/sec warmup 109 sec latency 25556.672 ms 2024-08-28T19:36:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:13 smithi115 ceph-mon[22184]: pgmap v811: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-08-28T19:36:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:13 smithi016 ceph-mon[18556]: pgmap v811: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-08-28T19:36:13.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:13 smithi028 ceph-mon[21769]: pgmap v811: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-08-28T19:36:14.466 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 18.58 MB/sec warmup 110 sec latency 26556.799 ms 2024-08-28T19:36:14.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:36:14.871 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:36:14.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:36:15.092 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:15.466 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 18.41 MB/sec warmup 111 sec latency 27556.847 ms 2024-08-28T19:36:15.472 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:15 smithi016 ceph-mon[18556]: pgmap v812: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-08-28T19:36:15.472 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:15 smithi016 ceph-mon[18556]: osdmap e102: 12 total, 12 up, 12 in 2024-08-28T19:36:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.564+0000 7f4a32af8640 1 -- 172.21.15.16:0/171114738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a2c102800 msgr2=0x7f4a2c102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.564+0000 7f4a32af8640 1 --2- 172.21.15.16:0/171114738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a2c102800 0x7f4a2c102c10 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009ca0 tx=0x7f4a1c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:36:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.564+0000 7f4a32af8640 1 -- 172.21.15.16:0/171114738 shutdown_connections 2024-08-28T19:36:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.564+0000 7f4a32af8640 1 --2- 172.21.15.16:0/171114738 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a2c108a90 0x7f4a2c10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.564+0000 7f4a32af8640 1 --2- 172.21.15.16:0/171114738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c108140 0x7f4a2c108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.564+0000 7f4a32af8640 1 --2- 172.21.15.16:0/171114738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a2c102800 0x7f4a2c102c10 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.564+0000 7f4a32af8640 1 -- 172.21.15.16:0/171114738 >> 172.21.15.16:0/171114738 conn(0x7f4a2c0794d0 msgr2=0x7f4a2c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.564+0000 7f4a32af8640 1 -- 172.21.15.16:0/171114738 shutdown_connections 2024-08-28T19:36:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.564+0000 7f4a32af8640 1 -- 172.21.15.16:0/171114738 wait complete. 2024-08-28T19:36:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.565+0000 7f4a32af8640 1 Processor -- start 2024-08-28T19:36:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.565+0000 7f4a32af8640 1 -- start start 2024-08-28T19:36:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.565+0000 7f4a32af8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a2c102800 0x7f4a2c1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.565+0000 7f4a32af8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c108140 0x7f4a2c199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.565+0000 7f4a32af8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a2c108a90 0x7f4a2c19e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.565+0000 7f4a32af8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c1126f0 con 0x7f4a2c102800 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.565+0000 7f4a32af8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c112570 con 0x7f4a2c108140 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.565+0000 7f4a32af8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c112870 con 0x7f4a2c108a90 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.565+0000 7f4a30894640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a2c102800 0x7f4a2c1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.565+0000 7f4a30894640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a2c102800 0x7f4a2c1991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42562/0 (socket says 172.21.15.16:42562) 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.565+0000 7f4a30894640 1 -- 172.21.15.16:0/488763106 learned_addr learned my addr 172.21.15.16:0/488763106 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a31095640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a2c108a90 0x7f4a2c19e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a2bfff640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c108140 0x7f4a2c199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a30894640 1 -- 172.21.15.16:0/488763106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a2c108a90 msgr2=0x7f4a2c19e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a30894640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a2c108a90 0x7f4a2c19e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a30894640 1 -- 172.21.15.16:0/488763106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c108140 msgr2=0x7f4a2c199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a30894640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c108140 0x7f4a2c199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a30894640 1 -- 172.21.15.16:0/488763106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2c19f110 con 0x7f4a2c102800 2024-08-28T19:36:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a30894640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a2c102800 0x7f4a2c1991a0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c02ade0 tx=0x7f4a1c031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a29ffb640 1 -- 172.21.15.16:0/488763106 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c003b70 con 0x7f4a2c102800 2024-08-28T19:36:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a29ffb640 1 -- 172.21.15.16:0/488763106 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a1c003d10 con 0x7f4a2c102800 2024-08-28T19:36:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a32af8640 1 -- 172.21.15.16:0/488763106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2c19f3a0 con 0x7f4a2c102800 2024-08-28T19:36:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.566+0000 7f4a29ffb640 1 -- 172.21.15.16:0/488763106 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c02fa50 con 0x7f4a2c102800 2024-08-28T19:36:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.567+0000 7f4a32af8640 1 -- 172.21.15.16:0/488763106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2c1a5b60 con 0x7f4a2c102800 2024-08-28T19:36:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.568+0000 7f4a32af8640 1 -- 172.21.15.16:0/488763106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a2c102c10 con 0x7f4a2c102800 2024-08-28T19:36:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.569+0000 7f4a29ffb640 1 -- 172.21.15.16:0/488763106 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4a1c004580 con 0x7f4a2c102800 2024-08-28T19:36:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.570+0000 7f4a29ffb640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a180b4d20 0x7f4a180b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.571+0000 7f4a2bfff640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a180b4d20 0x7f4a180b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.571+0000 7f4a29ffb640 1 -- 172.21.15.16:0/488763106 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f4a1c106c20 con 0x7f4a2c102800 2024-08-28T19:36:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.571+0000 7f4a2bfff640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a180b4d20 0x7f4a180b7230 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c100a60 tx=0x7f4a14005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.574+0000 7f4a29ffb640 1 -- 172.21.15.16:0/488763106 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f4a1c0cd640 con 0x7f4a2c102800 2024-08-28T19:36:15.578 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:15 smithi115 ceph-mon[22184]: pgmap v812: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-08-28T19:36:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:15 smithi115 ceph-mon[22184]: osdmap e102: 12 total, 12 up, 12 in 2024-08-28T19:36:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:15 smithi028 ceph-mon[21769]: pgmap v812: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-08-28T19:36:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:15 smithi028 ceph-mon[21769]: osdmap e102: 12 total, 12 up, 12 in 2024-08-28T19:36:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.808+0000 7f4a32af8640 1 -- 172.21.15.16:0/488763106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a2c19a2d0 con 0x7f4a2c102800 2024-08-28T19:36:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.809+0000 7f4a29ffb640 1 -- 172.21.15.16:0/488763106 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) ==== 94+0+12010 (secure 0 0 0) 0x7f4a1c0ccf40 con 0x7f4a2c102800 2024-08-28T19:36:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:36:15.811 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"btime":"2024-08-28T19:35:42:414805+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":78,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:35:41.973464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:36:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 78 2024-08-28T19:36:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.811+0000 7f4a32af8640 1 -- 172.21.15.16:0/488763106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a180b4d20 msgr2=0x7f4a180b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.811+0000 7f4a32af8640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a180b4d20 0x7f4a180b7230 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c100a60 tx=0x7f4a14005f90 comp rx=0 tx=0).stop 2024-08-28T19:36:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.811+0000 7f4a32af8640 1 -- 172.21.15.16:0/488763106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a2c102800 msgr2=0x7f4a2c1991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.811+0000 7f4a32af8640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a2c102800 0x7f4a2c1991a0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c02ade0 tx=0x7f4a1c031770 comp rx=0 tx=0).stop 2024-08-28T19:36:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.811+0000 7f4a32af8640 1 -- 172.21.15.16:0/488763106 shutdown_connections 2024-08-28T19:36:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.811+0000 7f4a32af8640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a2c108a90 0x7f4a2c19e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.811+0000 7f4a32af8640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4a180b4d20 0x7f4a180b7230 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.811+0000 7f4a32af8640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c108140 0x7f4a2c199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.811+0000 7f4a32af8640 1 --2- 172.21.15.16:0/488763106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a2c102800 0x7f4a2c1991a0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.811+0000 7f4a32af8640 1 -- 172.21.15.16:0/488763106 >> 172.21.15.16:0/488763106 conn(0x7f4a2c0794d0 msgr2=0x7f4a2c101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.812+0000 7f4a32af8640 1 -- 172.21.15.16:0/488763106 shutdown_connections 2024-08-28T19:36:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:15.812+0000 7f4a32af8640 1 -- 172.21.15.16:0/488763106 wait complete. 2024-08-28T19:36:16.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.265+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2014725705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a0109270 msgr2=0x7f51a0109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:16.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.265+0000 7f51a4cbe640 1 --2- 172.21.15.16:0/2014725705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a0109270 0x7f51a0109680 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f5188009900 tx=0x7f518802a410 comp rx=0 tx=0).stop 2024-08-28T19:36:16.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.265+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2014725705 shutdown_connections 2024-08-28T19:36:16.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.265+0000 7f51a4cbe640 1 --2- 172.21.15.16:0/2014725705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a0109bc0 0x7f51a01103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.265+0000 7f51a4cbe640 1 --2- 172.21.15.16:0/2014725705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a0109270 0x7f51a0109680 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.265+0000 7f51a4cbe640 1 --2- 172.21.15.16:0/2014725705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51a0103930 0x7f51a0103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.265+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2014725705 >> 172.21.15.16:0/2014725705 conn(0x7f51a00ff380 msgr2=0x7f51a01017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:16.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.265+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2014725705 shutdown_connections 2024-08-28T19:36:16.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.265+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2014725705 wait complete. 2024-08-28T19:36:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.266+0000 7f51a4cbe640 1 Processor -- start 2024-08-28T19:36:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.266+0000 7f51a4cbe640 1 -- start start 2024-08-28T19:36:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.266+0000 7f51a4cbe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a0103930 0x7f51a00754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.266+0000 7f51a4cbe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a0109270 0x7f51a00759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.266+0000 7f51a4cbe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51a0109bc0 0x7f51a0075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.266+0000 7f51a4cbe640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51a0113880 con 0x7f51a0109270 2024-08-28T19:36:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.266+0000 7f51a4cbe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51a0113700 con 0x7f51a0103930 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.266+0000 7f51a4cbe640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f51a0113a00 con 0x7f51a0109bc0 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a0109270 0x7f51a00759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a0109270 0x7f51a00759f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42574/0 (socket says 172.21.15.16:42574) 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519dd9b640 1 -- 172.21.15.16:0/2468493633 learned_addr learned my addr 172.21.15.16:0/2468493633 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519e59c640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a0103930 0x7f51a00754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519ed9d640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51a0109bc0 0x7f51a0075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519dd9b640 1 -- 172.21.15.16:0/2468493633 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51a0109bc0 msgr2=0x7f51a0075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519dd9b640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51a0109bc0 0x7f51a0075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519dd9b640 1 -- 172.21.15.16:0/2468493633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a0103930 msgr2=0x7f51a00754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519dd9b640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a0103930 0x7f51a00754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519dd9b640 1 -- 172.21.15.16:0/2468493633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a01a8510 con 0x7f51a0109270 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519ed9d640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51a0109bc0 0x7f51a0075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f519dd9b640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a0109270 0x7f51a00759f0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f518802a930 tx=0x7f5188031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f51977fe640 1 -- 172.21.15.16:0/2468493633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518803f030 con 0x7f51a0109270 2024-08-28T19:36:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f51977fe640 1 -- 172.21.15.16:0/2468493633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5188004490 con 0x7f51a0109270 2024-08-28T19:36:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2468493633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a01a87a0 con 0x7f51a0109270 2024-08-28T19:36:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.267+0000 7f51977fe640 1 -- 172.21.15.16:0/2468493633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188035070 con 0x7f51a0109270 2024-08-28T19:36:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.268+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2468493633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a01a8cd0 con 0x7f51a0109270 2024-08-28T19:36:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.270+0000 7f51977fe640 1 -- 172.21.15.16:0/2468493633 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5188042030 con 0x7f51a0109270 2024-08-28T19:36:16.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.271+0000 7f51ab233740 1 Processor -- start 2024-08-28T19:36:16.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.271+0000 7f51ab233740 1 -- start start 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.271+0000 7f51977fe640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f518c0b4d20 0x7f518c0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.271+0000 7f51ab233740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5624313ab350 0x562431751780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.271+0000 7f51ab233740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 0x562431751cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.271+0000 7f51ab233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x562431752200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.271+0000 7f51ab233740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5624316c75f0 con 0x562431382490 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f51ab233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624316b6b80 con 0x56243138a670 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f51ab233740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5624316bab30 con 0x5624313ab350 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519ed9d640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f518c0b4d20 0x7f518c0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 0x562431751cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x562431752200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 0x562431751cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42588/0 (socket says 172.21.15.16:42588) 2024-08-28T19:36:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519dd9b640 1 -- 172.21.15.16:0/3289493308 learned_addr learned my addr 172.21.15.16:0/3289493308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519e59c640 1 --2- 172.21.15.16:0/3289493308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5624313ab350 0x562431751780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f51977fe640 1 -- 172.21.15.16:0/2468493633 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f51881058a0 con 0x7f51a0109270 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519ed9d640 1 -- 172.21.15.16:0/3289493308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5624313ab350 msgr2=0x562431751780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519ed9d640 1 --2- 172.21.15.16:0/3289493308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5624313ab350 0x562431751780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519ed9d640 1 -- 172.21.15.16:0/3289493308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 msgr2=0x562431751cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519ed9d640 1 --2- 172.21.15.16:0/3289493308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 0x562431751cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519ed9d640 1 -- 172.21.15.16:0/3289493308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562431416930 con 0x56243138a670 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.272+0000 7f519dd9b640 1 --2- 172.21.15.16:0/3289493308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 0x562431751cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.273+0000 7f519ed9d640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f518c0b4d20 0x7f518c0b7230 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5198005fd0 tx=0x7f5198009eb0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.273+0000 7f519ed9d640 1 --2- 172.21.15.16:0/3289493308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x562431752200 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5198010bd0 tx=0x7f519801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d31a0387556f215 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.273+0000 7f5194ff9640 1 -- 172.21.15.16:0/3289493308 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5198024070 con 0x56243138a670 2024-08-28T19:36:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.273+0000 7f5194ff9640 1 -- 172.21.15.16:0/3289493308 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f519801fbd0 con 0x56243138a670 2024-08-28T19:36:16.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.273+0000 7f5194ff9640 1 -- 172.21.15.16:0/3289493308 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519801e3f0 con 0x56243138a670 2024-08-28T19:36:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.274+0000 7f51ab233740 1 -- 172.21.15.16:0/3289493308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 msgr2=0x562431752200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.274+0000 7f51ab233740 1 --2- 172.21.15.16:0/3289493308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x562431752200 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5198010bd0 tx=0x7f519801c000 comp rx=0 tx=0).stop 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.274+0000 7f51ab233740 1 -- 172.21.15.16:0/3289493308 shutdown_connections 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.274+0000 7f51ab233740 1 --2- 172.21.15.16:0/3289493308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x562431752200 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.274+0000 7f51ab233740 1 --2- 172.21.15.16:0/3289493308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 0x562431751cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.274+0000 7f51ab233740 1 --2- 172.21.15.16:0/3289493308 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5624313ab350 0x562431751780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.274+0000 7f51ab233740 1 -- 172.21.15.16:0/3289493308 >> 172.21.15.16:0/3289493308 conn(0x56243139f740 msgr2=0x56243174f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.274+0000 7f51ab233740 1 -- 172.21.15.16:0/3289493308 shutdown_connections 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.274+0000 7f51ab233740 1 -- 172.21.15.16:0/3289493308 wait complete. 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.274+0000 7f51ab233740 1 -- start start 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.274+0000 7f51ab233740 1 Processor -- start 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f51ab233740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 0x56243175cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f51ab233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x56243175d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f51ab233740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5624313ab350 0x562431769240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f51ab233740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562431519890 con 0x562431382490 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f51ab233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56243163d790 con 0x56243138a670 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f51ab233740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56243175ff00 con 0x5624313ab350 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f51ab233740 10 client.0 _ll_register_callbacks cb 0x5624313c2ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f519e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 0x56243175cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f519e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 0x56243175cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42604/0 (socket says 172.21.15.16:42604) 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f519e59c640 1 -- 172.21.15.16:0/3254867617 learned_addr learned my addr 172.21.15.16:0/3254867617 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f519dd9b640 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x56243175d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f519ed9d640 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5624313ab350 0x562431769240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f519dd9b640 1 -- 172.21.15.16:0/3254867617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5624313ab350 msgr2=0x562431769240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f519dd9b640 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5624313ab350 0x562431769240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f519dd9b640 1 -- 172.21.15.16:0/3254867617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 msgr2=0x56243175cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f519dd9b640 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 0x56243175cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.275+0000 7f519dd9b640 1 -- 172.21.15.16:0/3254867617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562431674f30 con 0x56243138a670 2024-08-28T19:36:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.276+0000 7f519dd9b640 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x56243175d420 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5188102c80 tx=0x7f5188000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bc19b47f86e0f1e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.276+0000 7f5185ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:36:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.276+0000 7f5185ffb640 1 -- 172.21.15.16:0/3254867617 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188049070 con 0x56243138a670 2024-08-28T19:36:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.276+0000 7f5185ffb640 1 -- 172.21.15.16:0/3254867617 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f5188102ea0 con 0x56243138a670 2024-08-28T19:36:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.276+0000 7f51ab233740 1 -- 172.21.15.16:0/3254867617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f519801fbd0 con 0x56243138a670 2024-08-28T19:36:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.277+0000 7f5185ffb640 1 -- 172.21.15.16:0/3254867617 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f51880cc2c0 con 0x56243138a670 2024-08-28T19:36:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.277+0000 7f51ab233740 10 client.28676 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.277+0000 7f51ab233740 1 -- 172.21.15.16:0/3254867617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x56243176bd00 con 0x56243138a670 2024-08-28T19:36:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.277+0000 7f5185ffb640 1 -- 172.21.15.16:0/3254867617 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51880dc020 con 0x56243138a670 2024-08-28T19:36:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.277+0000 7f5185ffb640 1 -- 172.21.15.16:0/3254867617 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f518804c060 con 0x56243138a670 2024-08-28T19:36:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.277+0000 7f51ab233740 10 client.28676 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.277+0000 7f51ab233740 10 client.28676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:36:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.278+0000 7f51ab233740 20 client.28676 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.278+0000 7f51ab233740 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562431369cd0 0x56243176f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.278+0000 7f51ab233740 4 client.28676 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.278+0000 7f51ab233740 1 -- 172.21.15.16:0/3254867617 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624316c9540 con 0x562431369cd0 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.278+0000 7f519e59c640 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562431369cd0 0x56243176f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.278+0000 7f519e59c640 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562431369cd0 0x56243176f260 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63d57052942e5dff server_cookie=58fc3a2b4aae74f3 in_seq=0 out_seq=0 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.278+0000 7f5185ffb640 10 client.28676 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f5185ffb640 1 -- 172.21.15.16:0/3254867617 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5624316c9540 con 0x562431369cd0 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f5185ffb640 10 client.28676 handle_command_reply: tid=0 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f51ab233740 1 client.28676 shutdown 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f51ab233740 1 -- 172.21.15.16:0/3254867617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 msgr2=0x56243175d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f51ab233740 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x56243175d420 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5188102c80 tx=0x7f5188000bc0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f51ab233740 1 -- 172.21.15.16:0/3254867617 shutdown_connections 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f51ab233740 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562431369cd0 0x56243176f260 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f51ab233740 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5624313ab350 0x562431769240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f51ab233740 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x56243175d420 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f5185ffb640 0 client.28676 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f51ab233740 1 --2- 172.21.15.16:0/3254867617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562431382490 0x56243175cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f51ab233740 1 -- 172.21.15.16:0/3254867617 >> 172.21.15.16:0/3254867617 conn(0x56243139f740 msgr2=0x562431754960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f51ab233740 1 -- 172.21.15.16:0/3254867617 shutdown_connections 2024-08-28T19:36:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.279+0000 7f51ab233740 1 -- 172.21.15.16:0/3254867617 wait complete. 2024-08-28T19:36:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.280+0000 7f51ab233740 20 client.28676 trim_cache size 0 max 16384 2024-08-28T19:36:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.298+0000 7f51ab233740 1 Processor -- start 2024-08-28T19:36:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.298+0000 7f51ab233740 1 -- start start 2024-08-28T19:36:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.298+0000 7f51ab233740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562431382490 0x56243174f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f51ab233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x562431754f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f51ab233740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431769140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f51ab233740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51a01b2fc0 con 0x5624313ab350 2024-08-28T19:36:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f51ab233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51a01b4dd0 con 0x56243138a670 2024-08-28T19:36:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f51ab233740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f51a01b3720 con 0x562431382490 2024-08-28T19:36:16.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f519ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431769140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f519ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431769140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42608/0 (socket says 172.21.15.16:42608) 2024-08-28T19:36:16.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f519ed9d640 1 -- 172.21.15.16:0/4009331576 learned_addr learned my addr 172.21.15.16:0/4009331576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:16.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f519e59c640 1 --2- 172.21.15.16:0/4009331576 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562431382490 0x56243174f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f519dd9b640 1 --2- 172.21.15.16:0/4009331576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x562431754f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f519ed9d640 1 -- 172.21.15.16:0/4009331576 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562431382490 msgr2=0x56243174f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:16.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f519ed9d640 1 --2- 172.21.15.16:0/4009331576 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562431382490 0x56243174f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f519ed9d640 1 -- 172.21.15.16:0/4009331576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 msgr2=0x562431754f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:16.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f519ed9d640 1 --2- 172.21.15.16:0/4009331576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x562431754f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f519ed9d640 1 -- 172.21.15.16:0/4009331576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56243175df80 con 0x5624313ab350 2024-08-28T19:36:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.299+0000 7f519ed9d640 1 --2- 172.21.15.16:0/4009331576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431769140 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f5198005c40 tx=0x7f51980099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=468c7110c2756f3b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.300+0000 7f5187fff640 1 -- 172.21.15.16:0/4009331576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5198024040 con 0x5624313ab350 2024-08-28T19:36:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.300+0000 7f5187fff640 1 -- 172.21.15.16:0/4009331576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5198004bc0 con 0x5624313ab350 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.300+0000 7f5187fff640 1 -- 172.21.15.16:0/4009331576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5198005040 con 0x5624313ab350 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.301+0000 7f51ab233740 1 -- 172.21.15.16:0/4009331576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 msgr2=0x562431769140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.301+0000 7f51ab233740 1 --2- 172.21.15.16:0/4009331576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431769140 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f5198005c40 tx=0x7f51980099b0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.301+0000 7f51ab233740 1 -- 172.21.15.16:0/4009331576 shutdown_connections 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.301+0000 7f51ab233740 1 --2- 172.21.15.16:0/4009331576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431769140 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.301+0000 7f51ab233740 1 --2- 172.21.15.16:0/4009331576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56243138a670 0x562431754f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.301+0000 7f51ab233740 1 --2- 172.21.15.16:0/4009331576 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562431382490 0x56243174f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.301+0000 7f51ab233740 1 -- 172.21.15.16:0/4009331576 >> 172.21.15.16:0/4009331576 conn(0x56243139f740 msgr2=0x562431754880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.301+0000 7f51ab233740 1 -- 172.21.15.16:0/4009331576 shutdown_connections 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.301+0000 7f51ab233740 1 -- 172.21.15.16:0/4009331576 wait complete. 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.301+0000 7f51ab233740 1 -- start start 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.301+0000 7f51ab233740 1 Processor -- start 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f51ab233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562431382490 0x562431762120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f51ab233740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56243138a670 0x562431762660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f51ab233740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431762ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f51ab233740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51a01b4640 con 0x5624313ab350 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f51ab233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51a01b3eb0 con 0x562431382490 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f51ab233740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f51a01b0fc0 con 0x56243138a670 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f51ab233740 10 client.0 _ll_register_callbacks cb 0x7f51880cb330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f519ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431762ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f519dd9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56243138a670 0x562431762660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f519ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431762ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42620/0 (socket says 172.21.15.16:42620) 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f519ed9d640 1 -- 172.21.15.16:0/2058452856 learned_addr learned my addr 172.21.15.16:0/2058452856 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f519e59c640 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562431382490 0x562431762120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f519dd9b640 1 -- 172.21.15.16:0/2058452856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562431382490 msgr2=0x562431762120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f519dd9b640 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562431382490 0x562431762120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f519dd9b640 1 -- 172.21.15.16:0/2058452856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 msgr2=0x562431762ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f519dd9b640 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431762ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f519dd9b640 1 -- 172.21.15.16:0/2058452856 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624317d9c70 con 0x56243138a670 2024-08-28T19:36:16.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.302+0000 7f519ed9d640 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431762ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:16.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.303+0000 7f519dd9b640 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56243138a670 0x562431762660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5188006010 tx=0x7f51880e1360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5e23c331836eb9f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:16.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.303+0000 7f5185ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.303+0000 7f5185ffb640 1 -- 172.21.15.16:0/2058452856 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188049040 con 0x56243138a670 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.303+0000 7f51ab233740 1 -- 172.21.15.16:0/2058452856 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5198004bc0 con 0x56243138a670 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.303+0000 7f5185ffb640 1 -- 172.21.15.16:0/2058452856 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f51880d4a50 con 0x56243138a670 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.303+0000 7f5185ffb640 1 -- 172.21.15.16:0/2058452856 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f51880c7a80 con 0x56243138a670 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.303+0000 7f51ab233740 10 client.28318 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.303+0000 7f51ab233740 1 -- 172.21.15.16:0/2058452856 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x5624317daec0 con 0x56243138a670 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.303+0000 7f5185ffb640 1 -- 172.21.15.16:0/2058452856 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5198004bc0 con 0x56243138a670 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.304+0000 7f5185ffb640 1 -- 172.21.15.16:0/2058452856 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f5188105470 con 0x56243138a670 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.304+0000 7f51ab233740 10 client.28318 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.304+0000 7f51ab233740 10 client.28318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.304+0000 7f51ab233740 20 client.28318 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.304+0000 7f51ab233740 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56243133f080 0x5624317dd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.304+0000 7f51ab233740 4 client.28318 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.304+0000 7f51ab233740 1 -- 172.21.15.16:0/2058452856 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5624316c4fe0 con 0x56243133f080 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.304+0000 7f519e59c640 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56243133f080 0x5624317dd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.304+0000 7f519e59c640 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56243133f080 0x5624317dd530 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82fc14c45b809e8e server_cookie=cea73ef25191ae10 in_seq=0 out_seq=0 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.304+0000 7f5185ffb640 10 client.28318 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f5185ffb640 1 -- 172.21.15.16:0/2058452856 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5624316c4fe0 con 0x56243133f080 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f5185ffb640 10 client.28318 handle_command_reply: tid=0 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 1 client.28318 shutdown 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 1 -- 172.21.15.16:0/2058452856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56243138a670 msgr2=0x562431762660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56243138a670 0x562431762660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5188006010 tx=0x7f51880e1360 comp rx=0 tx=0).stop 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 1 -- 172.21.15.16:0/2058452856 shutdown_connections 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624313ab350 0x562431762ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56243133f080 0x5624317dd530 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56243138a670 0x562431762660 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 1 --2- 172.21.15.16:0/2058452856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562431382490 0x562431762120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f5185ffb640 0 client.28318 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:16.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 1 -- 172.21.15.16:0/2058452856 >> 172.21.15.16:0/2058452856 conn(0x56243139f740 msgr2=0x562431752760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:16.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 1 -- 172.21.15.16:0/2058452856 shutdown_connections 2024-08-28T19:36:16.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 1 -- 172.21.15.16:0/2058452856 wait complete. 2024-08-28T19:36:16.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.306+0000 7f51ab233740 20 client.28318 trim_cache size 0 max 16384 2024-08-28T19:36:16.311 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:36:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:36:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "4d705fea-6bef-4150-87f6-bc36721d0baa", 2024-08-28T19:36:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:36:16.311 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:36:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2468493633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f518c0b4d20 msgr2=0x7f518c0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f518c0b4d20 0x7f518c0b7230 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5198005fd0 tx=0x7f5198009eb0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2468493633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a0109270 msgr2=0x7f51a00759f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a0109270 0x7f51a00759f0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f518802a930 tx=0x7f5188031a20 comp rx=0 tx=0).stop 2024-08-28T19:36:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2468493633 shutdown_connections 2024-08-28T19:36:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51a0109bc0 0x7f51a0075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a0109270 0x7f51a00759f0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f518c0b4d20 0x7f518c0b7230 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 --2- 172.21.15.16:0/2468493633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a0103930 0x7f51a00754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2468493633 >> 172.21.15.16:0/2468493633 conn(0x7f51a00ff380 msgr2=0x7f51a01017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2468493633 shutdown_connections 2024-08-28T19:36:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:16.307+0000 7f51a4cbe640 1 -- 172.21.15.16:0/2468493633 wait complete. 2024-08-28T19:36:16.466 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 18.25 MB/sec warmup 112 sec latency 28556.880 ms 2024-08-28T19:36:16.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:16 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/488763106' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:16.628 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:16 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/488763106' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:16 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/488763106' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:16.871 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id d8a4beb8 --release --await' 2024-08-28T19:36:16.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id d8a4beb8 --release --await 2024-08-28T19:36:17.255 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:17.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 18.09 MB/sec warmup 113 sec latency 29556.960 ms 2024-08-28T19:36:17.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: pgmap v814: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 335 KiB/s wr, 52 op/s 2024-08-28T19:36:17.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:36:17.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:36:17.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:36:17.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:36:17.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:36:17.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:36:17.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:17 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:36:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: pgmap v814: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 335 KiB/s wr, 52 op/s 2024-08-28T19:36:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:36:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:36:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:36:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:36:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:36:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:36:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:17 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: pgmap v814: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 335 KiB/s wr, 52 op/s 2024-08-28T19:36:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:36:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:36:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:36:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:36:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:36:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:36:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:17 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.284+0000 7fea641bc640 1 -- 172.21.15.16:0/1600036148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea5c1028b0 msgr2=0x7fea5c102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.284+0000 7fea641bc640 1 --2- 172.21.15.16:0/1600036148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea5c1028b0 0x7fea5c102cc0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fea58009900 tx=0x7fea5802a410 comp rx=0 tx=0).stop 2024-08-28T19:36:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.284+0000 7fea641bc640 1 -- 172.21.15.16:0/1600036148 shutdown_connections 2024-08-28T19:36:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.284+0000 7fea641bc640 1 --2- 172.21.15.16:0/1600036148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea5c108b40 0x7fea5c10f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.284+0000 7fea641bc640 1 --2- 172.21.15.16:0/1600036148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea5c1081f0 0x7fea5c108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.284+0000 7fea641bc640 1 --2- 172.21.15.16:0/1600036148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea5c1028b0 0x7fea5c102cc0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.284+0000 7fea641bc640 1 -- 172.21.15.16:0/1600036148 >> 172.21.15.16:0/1600036148 conn(0x7fea5c079470 msgr2=0x7fea5c079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.284+0000 7fea641bc640 1 -- 172.21.15.16:0/1600036148 shutdown_connections 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.284+0000 7fea641bc640 1 -- 172.21.15.16:0/1600036148 wait complete. 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.285+0000 7fea641bc640 1 Processor -- start 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.285+0000 7fea641bc640 1 -- start start 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea641bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea5c1028b0 0x7fea5c199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea61f58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea5c1028b0 0x7fea5c199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea61f58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea5c1028b0 0x7fea5c199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42642/0 (socket says 172.21.15.16:42642) 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea61f58640 1 -- 172.21.15.16:0/89160435 learned_addr learned my addr 172.21.15.16:0/89160435 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea641bc640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea5c1081f0 0x7fea5c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea641bc640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea5c108b40 0x7fea5c19ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea641bc640 1 -- 172.21.15.16:0/89160435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c112800 con 0x7fea5c1028b0 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea641bc640 1 -- 172.21.15.16:0/89160435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c112680 con 0x7fea5c1081f0 2024-08-28T19:36:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea641bc640 1 -- 172.21.15.16:0/89160435 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c112980 con 0x7fea5c108b40 2024-08-28T19:36:18.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea62759640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea5c108b40 0x7fea5c19ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea61757640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea5c1081f0 0x7fea5c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea61f58640 1 -- 172.21.15.16:0/89160435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea5c108b40 msgr2=0x7fea5c19ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea61f58640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea5c108b40 0x7fea5c19ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea61f58640 1 -- 172.21.15.16:0/89160435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea5c1081f0 msgr2=0x7fea5c1997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea61f58640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea5c1081f0 0x7fea5c1997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea61f58640 1 -- 172.21.15.16:0/89160435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea5c19f1c0 con 0x7fea5c1028b0 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea62759640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea5c108b40 0x7fea5c19ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.286+0000 7fea61757640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea5c1081f0 0x7fea5c1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.287+0000 7fea61f58640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea5c1028b0 0x7fea5c199250 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fea58005940 tx=0x7fea580042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.287+0000 7fea52ffd640 1 -- 172.21.15.16:0/89160435 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5803f030 con 0x7fea5c1028b0 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.287+0000 7fea641bc640 1 -- 172.21.15.16:0/89160435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea5c19f450 con 0x7fea5c1028b0 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.287+0000 7fea52ffd640 1 -- 172.21.15.16:0/89160435 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea58003cb0 con 0x7fea5c1028b0 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.287+0000 7fea52ffd640 1 -- 172.21.15.16:0/89160435 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea58035070 con 0x7fea5c1028b0 2024-08-28T19:36:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.287+0000 7fea641bc640 1 -- 172.21.15.16:0/89160435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea5c06b900 con 0x7fea5c1028b0 2024-08-28T19:36:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.290+0000 7fea6a731740 1 Processor -- start 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.290+0000 7fea52ffd640 1 -- 172.21.15.16:0/89160435 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fea58042030 con 0x7fea5c1028b0 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.290+0000 7fea6a731740 1 -- start start 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.290+0000 7fea6a731740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f20c520 0x55a20f5b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.290+0000 7fea6a731740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1e3660 0x55a20f5b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.290+0000 7fea6a731740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.290+0000 7fea6a731740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a20f528a70 con 0x55a20f1eb840 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.290+0000 7fea6a731740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a20f518000 con 0x55a20f20c520 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.290+0000 7fea6a731740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a20f51bfb0 con 0x55a20f1e3660 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea61757640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea62759640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f20c520 0x55a20f5b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea62759640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f20c520 0x55a20f5b1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:53118/0 (socket says 172.21.15.16:53118) 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea62759640 1 -- 172.21.15.16:0/3737465242 learned_addr learned my addr 172.21.15.16:0/3737465242 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea61757640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42650/0 (socket says 172.21.15.16:42650) 2024-08-28T19:36:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea61f58640 1 --2- 172.21.15.16:0/3737465242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1e3660 0x55a20f5b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea52ffd640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fea480b4ca0 0x7fea480b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea61757640 1 -- 172.21.15.16:0/3737465242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1e3660 msgr2=0x55a20f5b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea61757640 1 --2- 172.21.15.16:0/3737465242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1e3660 0x55a20f5b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea61757640 1 -- 172.21.15.16:0/3737465242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f20c520 msgr2=0x55a20f5b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea61757640 1 --2- 172.21.15.16:0/3737465242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f20c520 0x55a20f5b1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea61757640 1 -- 172.21.15.16:0/3737465242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a20f277b00 con 0x55a20f1eb840 2024-08-28T19:36:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea62759640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fea480b4ca0 0x7fea480b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.291+0000 7fea52ffd640 1 -- 172.21.15.16:0/89160435 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7fea58105fa0 con 0x7fea5c1028b0 2024-08-28T19:36:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.292+0000 7fea62759640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fea480b4ca0 0x7fea480b71b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fea540092f0 tx=0x7fea54004a40 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.292+0000 7fea61757640 1 --2- 172.21.15.16:0/3737465242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5b2700 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fea5c06b420 tx=0x7fea4c00e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b546be519c6830 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.292+0000 7fea47fff640 1 -- 172.21.15.16:0/3737465242 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c016070 con 0x55a20f1eb840 2024-08-28T19:36:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.292+0000 7fea47fff640 1 -- 172.21.15.16:0/3737465242 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea4c0054d0 con 0x55a20f1eb840 2024-08-28T19:36:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.292+0000 7fea47fff640 1 -- 172.21.15.16:0/3737465242 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c011470 con 0x55a20f1eb840 2024-08-28T19:36:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.293+0000 7fea6a731740 1 -- 172.21.15.16:0/3737465242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 msgr2=0x55a20f5b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.293+0000 7fea6a731740 1 --2- 172.21.15.16:0/3737465242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5b2700 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fea5c06b420 tx=0x7fea4c00e2f0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.293+0000 7fea6a731740 1 -- 172.21.15.16:0/3737465242 shutdown_connections 2024-08-28T19:36:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.293+0000 7fea6a731740 1 --2- 172.21.15.16:0/3737465242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5b2700 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.293+0000 7fea6a731740 1 --2- 172.21.15.16:0/3737465242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1e3660 0x55a20f5b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.293+0000 7fea6a731740 1 --2- 172.21.15.16:0/3737465242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f20c520 0x55a20f5b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.293+0000 7fea6a731740 1 -- 172.21.15.16:0/3737465242 >> 172.21.15.16:0/3737465242 conn(0x55a20f200910 msgr2=0x55a20f5afb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.293+0000 7fea6a731740 1 -- 172.21.15.16:0/3737465242 shutdown_connections 2024-08-28T19:36:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.293+0000 7fea6a731740 1 -- 172.21.15.16:0/3737465242 wait complete. 2024-08-28T19:36:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.294+0000 7fea6a731740 1 -- start start 2024-08-28T19:36:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.294+0000 7fea6a731740 1 Processor -- start 2024-08-28T19:36:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.294+0000 7fea6a731740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 0x55a20f5bd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.294+0000 7fea6a731740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.294+0000 7fea6a731740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f20c520 0x55a20f5cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.294+0000 7fea6a731740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a20f5c4870 con 0x55a20f1eb840 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.294+0000 7fea6a731740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a20f4d9520 con 0x55a20f1e3660 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.294+0000 7fea6a731740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a20f5c49f0 con 0x55a20f20c520 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.294+0000 7fea6a731740 10 client.0 _ll_register_callbacks cb 0x55a20f511c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.294+0000 7fea61757640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea61f58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 0x55a20f5bd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea61757640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42660/0 (socket says 172.21.15.16:42660) 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea61757640 1 -- 172.21.15.16:0/2618216946 learned_addr learned my addr 172.21.15.16:0/2618216946 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea62759640 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f20c520 0x55a20f5cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea61f58640 1 -- 172.21.15.16:0/2618216946 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f20c520 msgr2=0x55a20f5cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea61f58640 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f20c520 0x55a20f5cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea61f58640 1 -- 172.21.15.16:0/2618216946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 msgr2=0x55a20f5bddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea61f58640 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea61f58640 1 -- 172.21.15.16:0/2618216946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a20f2ee960 con 0x55a20f1e3660 2024-08-28T19:36:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea62759640 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f20c520 0x55a20f5cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea61757640 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea61f58640 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 0x55a20f5bd870 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fea58103320 tx=0x7fea581096e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8cd85a7d6d2936 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea457fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:36:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea457fa640 1 -- 172.21.15.16:0/2618216946 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea58049070 con 0x55a20f1e3660 2024-08-28T19:36:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea457fa640 1 -- 172.21.15.16:0/2618216946 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7fea580c8550 con 0x55a20f1e3660 2024-08-28T19:36:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.295+0000 7fea6a731740 1 -- 172.21.15.16:0/2618216946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a20f4b79a0 con 0x55a20f1e3660 2024-08-28T19:36:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.296+0000 7fea457fa640 1 -- 172.21.15.16:0/2618216946 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7fea580cba30 con 0x55a20f1e3660 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.296+0000 7fea6a731740 10 client.28691 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.296+0000 7fea6a731740 1 -- 172.21.15.16:0/2618216946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55a20f5d0a50 con 0x55a20f1e3660 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.296+0000 7fea457fa640 1 -- 172.21.15.16:0/2618216946 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a20f4b79a0 con 0x55a20f1e3660 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.296+0000 7fea457fa640 1 -- 172.21.15.16:0/2618216946 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7fea58109ca0 con 0x55a20f1e3660 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.296+0000 7fea6a731740 10 client.28691 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.296+0000 7fea6a731740 10 client.28691 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.296+0000 7fea6a731740 20 client.28691 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.296+0000 7fea6a731740 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a20f1caea0 0x55a20f5d3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.296+0000 7fea6a731740 4 client.28691 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.296+0000 7fea6a731740 1 -- 172.21.15.16:0/2618216946 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a20f52a9c0 con 0x55a20f1caea0 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.297+0000 7fea61757640 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a20f1caea0 0x55a20f5d3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.297+0000 7fea61757640 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a20f1caea0 0x55a20f5d3fc0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9953994819d7492 server_cookie=b925232c05be79d2 in_seq=0 out_seq=0 2024-08-28T19:36:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.297+0000 7fea457fa640 10 client.28691 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:18.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea457fa640 1 -- 172.21.15.16:0/2618216946 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a20f52a9c0 con 0x55a20f1caea0 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea457fa640 10 client.28691 handle_command_reply: tid=0 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 1 client.28691 shutdown 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 1 -- 172.21.15.16:0/2618216946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 msgr2=0x55a20f5bd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 0x55a20f5bd870 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fea58103320 tx=0x7fea581096e0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 1 -- 172.21.15.16:0/2618216946 shutdown_connections 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a20f1caea0 0x55a20f5d3fc0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f20c520 0x55a20f5cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 1 --2- 172.21.15.16:0/2618216946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 0x55a20f5bd870 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea457fa640 0 client.28691 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 1 -- 172.21.15.16:0/2618216946 >> 172.21.15.16:0/2618216946 conn(0x55a20f200910 msgr2=0x55a20f5b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 1 -- 172.21.15.16:0/2618216946 shutdown_connections 2024-08-28T19:36:18.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 1 -- 172.21.15.16:0/2618216946 wait complete. 2024-08-28T19:36:18.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.298+0000 7fea6a731740 20 client.28691 trim_cache size 0 max 16384 2024-08-28T19:36:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.318+0000 7fea6a731740 1 Processor -- start 2024-08-28T19:36:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.318+0000 7fea6a731740 1 -- start start 2024-08-28T19:36:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea6a731740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1caea0 0x55a20f5cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea6a731740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 0x55a20f5ba860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea6a731740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea6a731740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c1aebb0 con 0x55a20f1eb840 2024-08-28T19:36:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea6a731740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c1b09c0 con 0x55a20f1e3660 2024-08-28T19:36:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea6a731740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c1af310 con 0x55a20f1caea0 2024-08-28T19:36:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea62759640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea62759640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42676/0 (socket says 172.21.15.16:42676) 2024-08-28T19:36:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea62759640 1 -- 172.21.15.16:0/1079696136 learned_addr learned my addr 172.21.15.16:0/1079696136 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea61757640 1 --2- 172.21.15.16:0/1079696136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 0x55a20f5ba860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea61f58640 1 --2- 172.21.15.16:0/1079696136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1caea0 0x55a20f5cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea62759640 1 -- 172.21.15.16:0/1079696136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1caea0 msgr2=0x55a20f5cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea62759640 1 --2- 172.21.15.16:0/1079696136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1caea0 0x55a20f5cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea62759640 1 -- 172.21.15.16:0/1079696136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 msgr2=0x55a20f5ba860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea62759640 1 --2- 172.21.15.16:0/1079696136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 0x55a20f5ba860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea62759640 1 -- 172.21.15.16:0/1079696136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a20f5b8f00 con 0x55a20f1eb840 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.319+0000 7fea61757640 1 --2- 172.21.15.16:0/1079696136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 0x55a20f5ba860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.320+0000 7fea62759640 1 --2- 172.21.15.16:0/1079696136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bada0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fea5400f8b0 tx=0x7fea54015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d45df74979856e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.320+0000 7fea477fe640 1 -- 172.21.15.16:0/1079696136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5400fe30 con 0x55a20f1eb840 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.320+0000 7fea477fe640 1 -- 172.21.15.16:0/1079696136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea5401a420 con 0x55a20f1eb840 2024-08-28T19:36:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.320+0000 7fea477fe640 1 -- 172.21.15.16:0/1079696136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5401ee60 con 0x55a20f1eb840 2024-08-28T19:36:18.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.321+0000 7fea6a731740 1 -- 172.21.15.16:0/1079696136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 msgr2=0x55a20f5bada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:18.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.321+0000 7fea6a731740 1 --2- 172.21.15.16:0/1079696136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bada0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fea5400f8b0 tx=0x7fea54015000 comp rx=0 tx=0).stop 2024-08-28T19:36:18.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.321+0000 7fea6a731740 1 -- 172.21.15.16:0/1079696136 shutdown_connections 2024-08-28T19:36:18.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.321+0000 7fea6a731740 1 --2- 172.21.15.16:0/1079696136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1eb840 0x55a20f5bada0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.321+0000 7fea6a731740 1 --2- 172.21.15.16:0/1079696136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1e3660 0x55a20f5ba860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.321+0000 7fea6a731740 1 --2- 172.21.15.16:0/1079696136 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1caea0 0x55a20f5cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.321+0000 7fea6a731740 1 -- 172.21.15.16:0/1079696136 >> 172.21.15.16:0/1079696136 conn(0x55a20f200910 msgr2=0x55a20f5af3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:18.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.321+0000 7fea6a731740 1 -- 172.21.15.16:0/1079696136 shutdown_connections 2024-08-28T19:36:18.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.321+0000 7fea6a731740 1 -- 172.21.15.16:0/1079696136 wait complete. 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.322+0000 7fea6a731740 1 -- start start 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.322+0000 7fea6a731740 1 Processor -- start 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.322+0000 7fea6a731740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1caea0 0x55a20f5c6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.322+0000 7fea6a731740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1e3660 0x55a20f5c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea6a731740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1eb840 0x55a20f637b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea6a731740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c1b0230 con 0x55a20f1caea0 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea6a731740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c1afaa0 con 0x55a20f1eb840 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea6a731740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c1acbb0 con 0x55a20f1e3660 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea61f58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1caea0 0x55a20f5c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea6a731740 10 client.0 _ll_register_callbacks cb 0x7fea58003e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea61f58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1caea0 0x55a20f5c6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42680/0 (socket says 172.21.15.16:42680) 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea61f58640 1 -- 172.21.15.16:0/4159645351 learned_addr learned my addr 172.21.15.16:0/4159645351 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea61757640 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1e3660 0x55a20f5c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea62759640 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1eb840 0x55a20f637b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea61f58640 1 -- 172.21.15.16:0/4159645351 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1e3660 msgr2=0x55a20f5c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea61f58640 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1e3660 0x55a20f5c6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea61f58640 1 -- 172.21.15.16:0/4159645351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1eb840 msgr2=0x55a20f637b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea61f58640 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1eb840 0x55a20f637b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea61f58640 1 -- 172.21.15.16:0/4159645351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a20f638170 con 0x55a20f1caea0 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea61f58640 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1caea0 0x55a20f5c6750 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fea58003fd0 tx=0x7fea580d5360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f3bdaa9a86726d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.323+0000 7fea457fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea457fa640 1 -- 172.21.15.16:0/4159645351 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea58049040 con 0x55a20f1caea0 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea457fa640 1 -- 172.21.15.16:0/4159645351 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7fea580cc730 con 0x55a20f1caea0 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea6a731740 1 -- 172.21.15.16:0/4159645351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea5401a420 con 0x55a20f1caea0 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea457fa640 1 -- 172.21.15.16:0/4159645351 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7fea5803b460 con 0x55a20f1caea0 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea6a731740 10 client.19272 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:18.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea6a731740 1 -- 172.21.15.16:0/4159645351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55a20f63f5a0 con 0x55a20f1caea0 2024-08-28T19:36:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea457fa640 1 -- 172.21.15.16:0/4159645351 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea5401a420 con 0x55a20f1caea0 2024-08-28T19:36:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea457fa640 1 -- 172.21.15.16:0/4159645351 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7fea5803bb50 con 0x55a20f1caea0 2024-08-28T19:36:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea6a731740 10 client.19272 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea6a731740 10 client.19272 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:36:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea6a731740 20 client.19272 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea6a731740 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a20f20c520 0x55a20f641c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea6a731740 4 client.19272 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "d8a4beb8", "release": true, "await": true}] 2024-08-28T19:36:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.324+0000 7fea6a731740 1 -- 172.21.15.16:0/4159645351 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "d8a4beb8", "release": true, "await": true}) -- 0x55a20f526460 con 0x55a20f20c520 2024-08-28T19:36:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.325+0000 7fea61757640 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a20f20c520 0x55a20f641c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.325+0000 7fea457fa640 10 client.19272 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.325+0000 7fea61757640 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a20f20c520 0x55a20f641c20 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7e4267b67a71c04 server_cookie=298018360b6b4f7a in_seq=0 out_seq=0 2024-08-28T19:36:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.460+0000 7fea457fa640 1 -- 172.21.15.16:0/4159645351 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55a20f526460 con 0x55a20f20c520 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea457fa640 10 client.19272 handle_command_reply: tid=0 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 1 client.19272 shutdown 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 1 -- 172.21.15.16:0/4159645351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1caea0 msgr2=0x55a20f5c6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1caea0 0x55a20f5c6750 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fea58003fd0 tx=0x7fea580d5360 comp rx=0 tx=0).stop 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 1 -- 172.21.15.16:0/4159645351 shutdown_connections 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a20f1eb840 0x55a20f637b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a20f20c520 0x55a20f641c20 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a20f1e3660 0x55a20f5c6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 1 --2- 172.21.15.16:0/4159645351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a20f1caea0 0x55a20f5c6750 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea457fa640 0 client.19272 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 1 -- 172.21.15.16:0/4159645351 >> 172.21.15.16:0/4159645351 conn(0x55a20f200910 msgr2=0x55a20f5ae150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 1 -- 172.21.15.16:0/4159645351 shutdown_connections 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 1 -- 172.21.15.16:0/4159645351 wait complete. 2024-08-28T19:36:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.461+0000 7fea6a731740 20 client.19272 trim_cache size 0 max 16384 2024-08-28T19:36:18.464 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:36:18.464 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 78, 2024-08-28T19:36:18.464 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:36:18.464 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 48, 2024-08-28T19:36:18.464 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:36:18.464 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a4beb8": { 2024-08-28T19:36:18.464 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 48, 2024-08-28T19:36:18.464 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:36:18.464 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:36:18.464 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:18.465 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:18.466 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:36:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 -- 172.21.15.16:0/89160435 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fea480b4ca0 msgr2=0x7fea480b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fea480b4ca0 0x7fea480b71b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fea540092f0 tx=0x7fea54004a40 comp rx=0 tx=0).stop 2024-08-28T19:36:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 -- 172.21.15.16:0/89160435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea5c1028b0 msgr2=0x7fea5c199250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea5c1028b0 0x7fea5c199250 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fea58005940 tx=0x7fea580042f0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 -- 172.21.15.16:0/89160435 shutdown_connections 2024-08-28T19:36:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea5c108b40 0x7fea5c19ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fea480b4ca0 0x7fea480b71b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea5c1081f0 0x7fea5c1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 --2- 172.21.15.16:0/89160435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea5c1028b0 0x7fea5c199250 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 -- 172.21.15.16:0/89160435 >> 172.21.15.16:0/89160435 conn(0x7fea5c079470 msgr2=0x7fea5c1062f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 -- 172.21.15.16:0/89160435 shutdown_connections 2024-08-28T19:36:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:18.462+0000 7fea641bc640 1 -- 172.21.15.16:0/89160435 wait complete. 2024-08-28T19:36:18.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34708 17.93 MB/sec warmup 114 sec latency 30536.864 ms 2024-08-28T19:36:18.578 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:18 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:18.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:18 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:18.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:18 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:36:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.069+0000 7f4112894640 1 -- 172.21.15.16:0/1855053255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c108a90 msgr2=0x7f410c10f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.069+0000 7f4112894640 1 --2- 172.21.15.16:0/1855053255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c108a90 0x7f410c10f2c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f4104009ca0 tx=0x7f410402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.069+0000 7f4112894640 1 -- 172.21.15.16:0/1855053255 shutdown_connections 2024-08-28T19:36:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.069+0000 7f4112894640 1 --2- 172.21.15.16:0/1855053255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c108a90 0x7f410c10f2c0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.069+0000 7f4112894640 1 --2- 172.21.15.16:0/1855053255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f410c108140 0x7f410c108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.069+0000 7f4112894640 1 --2- 172.21.15.16:0/1855053255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f410c102800 0x7f410c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.069+0000 7f4112894640 1 -- 172.21.15.16:0/1855053255 >> 172.21.15.16:0/1855053255 conn(0x7f410c0794d0 msgr2=0x7f410c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.070+0000 7f4112894640 1 -- 172.21.15.16:0/1855053255 shutdown_connections 2024-08-28T19:36:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.070+0000 7f4112894640 1 -- 172.21.15.16:0/1855053255 wait complete. 2024-08-28T19:36:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.070+0000 7f4112894640 1 Processor -- start 2024-08-28T19:36:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.070+0000 7f4112894640 1 -- start start 2024-08-28T19:36:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.071+0000 7f4112894640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f410c102800 0x7f410c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.071+0000 7f4112894640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c108140 0x7f410c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.071+0000 7f4112894640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f410c108a90 0x7f410c19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.071+0000 7f4112894640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f410c113820 con 0x7f410c108140 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.071+0000 7f4112894640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f410c1136a0 con 0x7f410c108a90 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.071+0000 7f4112894640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f410c1139a0 con 0x7f410c102800 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.071+0000 7f4110e31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f410c108a90 0x7f410c19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.071+0000 7f410b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c108140 0x7f410c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.071+0000 7f4110e31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f410c108a90 0x7f410c19fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:53154/0 (socket says 172.21.15.16:53154) 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.071+0000 7f4110e31640 1 -- 172.21.15.16:0/1835552253 learned_addr learned my addr 172.21.15.16:0/1835552253 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.071+0000 7f410bfff640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f410c102800 0x7f410c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.072+0000 7f410b7fe640 1 -- 172.21.15.16:0/1835552253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f410c102800 msgr2=0x7f410c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.072+0000 7f410b7fe640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f410c102800 0x7f410c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.072+0000 7f410b7fe640 1 -- 172.21.15.16:0/1835552253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f410c108a90 msgr2=0x7f410c19fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.072+0000 7f410b7fe640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f410c108a90 0x7f410c19fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.072+0000 7f410b7fe640 1 -- 172.21.15.16:0/1835552253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f410c1a0200 con 0x7f410c108140 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.072+0000 7f410bfff640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f410c102800 0x7f410c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.072+0000 7f410b7fe640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c108140 0x7f410c19a830 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f410000e9c0 tx=0x7f410000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.072+0000 7f41097fa640 1 -- 172.21.15.16:0/1835552253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410000ccc0 con 0x7f410c108140 2024-08-28T19:36:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.073+0000 7f41097fa640 1 -- 172.21.15.16:0/1835552253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41000108e0 con 0x7f410c108140 2024-08-28T19:36:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.073+0000 7f41097fa640 1 -- 172.21.15.16:0/1835552253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41000187c0 con 0x7f410c108140 2024-08-28T19:36:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.073+0000 7f4112894640 1 -- 172.21.15.16:0/1835552253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f410c1a04f0 con 0x7f410c108140 2024-08-28T19:36:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.073+0000 7f4112894640 1 -- 172.21.15.16:0/1835552253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f410c1a6c50 con 0x7f410c108140 2024-08-28T19:36:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.076+0000 7f4118e09740 1 Processor -- start 2024-08-28T19:36:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f4118e09740 1 -- start start 2024-08-28T19:36:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f41097fa640 1 -- 172.21.15.16:0/1835552253 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4100010430 con 0x7f410c108140 2024-08-28T19:36:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f4118e09740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe8a070 0x55817022f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f4118e09740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe611b0 0x55817022f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f4118e09740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe69390 0x55817022fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f4118e09740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5581701a6630 con 0x55816fe611b0 2024-08-28T19:36:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f4118e09740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558170195bc0 con 0x55816fe8a070 2024-08-28T19:36:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f4118e09740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558170199b70 con 0x55816fe69390 2024-08-28T19:36:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f410bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe611b0 0x55817022f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f410bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe611b0 0x55817022f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42720/0 (socket says 172.21.15.16:42720) 2024-08-28T19:36:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f410bfff640 1 -- 172.21.15.16:0/4289908330 learned_addr learned my addr 172.21.15.16:0/4289908330 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f4110e31640 1 --2- 172.21.15.16:0/4289908330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe8a070 0x55817022f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f410b7fe640 1 --2- 172.21.15.16:0/4289908330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe69390 0x55817022fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f410bfff640 1 -- 172.21.15.16:0/4289908330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe69390 msgr2=0x55817022fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f410bfff640 1 --2- 172.21.15.16:0/4289908330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe69390 0x55817022fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f410bfff640 1 -- 172.21.15.16:0/4289908330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe8a070 msgr2=0x55817022f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f410bfff640 1 --2- 172.21.15.16:0/4289908330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe8a070 0x55817022f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.077+0000 7f410bfff640 1 -- 172.21.15.16:0/4289908330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55816fef5650 con 0x55816fe611b0 2024-08-28T19:36:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.078+0000 7f410b7fe640 1 --2- 172.21.15.16:0/4289908330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe69390 0x55817022fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.078+0000 7f410bfff640 1 --2- 172.21.15.16:0/4289908330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe611b0 0x55817022f800 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0115f0 tx=0x7f40fc011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af62972a25ee616 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.078+0000 7f40f27fc640 1 -- 172.21.15.16:0/4289908330 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc01e070 con 0x55816fe611b0 2024-08-28T19:36:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.078+0000 7f41097fa640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f40f40b4da0 0x7f40f40b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.078+0000 7f40f27fc640 1 -- 172.21.15.16:0/4289908330 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40fc00faf0 con 0x55816fe611b0 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.078+0000 7f40f27fc640 1 -- 172.21.15.16:0/4289908330 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc019bc0 con 0x55816fe611b0 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.079+0000 7f41097fa640 1 -- 172.21.15.16:0/1835552253 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f41000e5bb0 con 0x7f410c108140 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.079+0000 7f4110e31640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f40f40b4da0 0x7f40f40b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.080+0000 7f4118e09740 1 -- 172.21.15.16:0/4289908330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe611b0 msgr2=0x55817022f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.080+0000 7f4118e09740 1 --2- 172.21.15.16:0/4289908330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe611b0 0x55817022f800 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0115f0 tx=0x7f40fc011ab0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.080+0000 7f4110e31640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f40f40b4da0 0x7f40f40b72b0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4104005e30 tx=0x7f4104005c50 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.080+0000 7f4118e09740 1 -- 172.21.15.16:0/4289908330 shutdown_connections 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.080+0000 7f4118e09740 1 --2- 172.21.15.16:0/4289908330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe69390 0x55817022fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.080+0000 7f4118e09740 1 --2- 172.21.15.16:0/4289908330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe611b0 0x55817022f800 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.080+0000 7f4118e09740 1 --2- 172.21.15.16:0/4289908330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe8a070 0x55817022f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.080+0000 7f4118e09740 1 -- 172.21.15.16:0/4289908330 >> 172.21.15.16:0/4289908330 conn(0x55816fe7e460 msgr2=0x55817022d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.080+0000 7f4118e09740 1 -- 172.21.15.16:0/4289908330 shutdown_connections 2024-08-28T19:36:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.080+0000 7f4118e09740 1 -- 172.21.15.16:0/4289908330 wait complete. 2024-08-28T19:36:19.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.081+0000 7f4118e09740 1 -- start start 2024-08-28T19:36:19.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.081+0000 7f4118e09740 1 Processor -- start 2024-08-28T19:36:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f4118e09740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe611b0 0x558170236a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f4118e09740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 0x558170236f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f4118e09740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe8a070 0x55817024f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f4118e09740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55816fff8700 con 0x55816fe8a070 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f4118e09740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55817011c700 con 0x55816fe69390 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f4118e09740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558170246230 con 0x55816fe611b0 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f410b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 0x558170236f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f410b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 0x558170236f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:53162/0 (socket says 172.21.15.16:53162) 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f410b7fe640 1 -- 172.21.15.16:0/3934113495 learned_addr learned my addr 172.21.15.16:0/3934113495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f4110e31640 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe8a070 0x55817024f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f410bfff640 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe611b0 0x558170236a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f4118e09740 10 client.0 _ll_register_callbacks cb 0x55817014bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f410b7fe640 1 -- 172.21.15.16:0/3934113495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe611b0 msgr2=0x558170236a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f410b7fe640 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe611b0 0x558170236a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f410b7fe640 1 -- 172.21.15.16:0/3934113495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe8a070 msgr2=0x55817024f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f410b7fe640 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe8a070 0x55817024f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f410b7fe640 1 -- 172.21.15.16:0/3934113495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581701558b0 con 0x55816fe69390 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.082+0000 7f4110e31640 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe8a070 0x55817024f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.083+0000 7f410b7fe640 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 0x558170236f90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f41000e5e90 tx=0x7f41000107e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d0576f72f7a073 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.083+0000 7f40eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.083+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3934113495 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41000b06e0 con 0x55816fe69390 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.083+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3934113495 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f41000930b0 con 0x55816fe69390 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.083+0000 7f4118e09740 1 -- 172.21.15.16:0/3934113495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40fc00faf0 con 0x55816fe69390 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.083+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3934113495 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f41000b5190 con 0x55816fe69390 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.083+0000 7f4118e09740 10 client.28709 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.083+0000 7f4118e09740 1 -- 172.21.15.16:0/3934113495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x558170251fc0 con 0x55816fe69390 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.083+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3934113495 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41000932b0 con 0x55816fe69390 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.084+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3934113495 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f41000aabf0 con 0x55816fe69390 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.084+0000 7f4118e09740 10 client.28709 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.084+0000 7f4118e09740 10 client.28709 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.084+0000 7f4118e09740 20 client.28709 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.084+0000 7f4118e09740 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55816fe489f0 0x558170255500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.084+0000 7f4118e09740 4 client.28709 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.084+0000 7f4118e09740 1 -- 172.21.15.16:0/3934113495 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f40fc00faf0 con 0x55816fe489f0 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.085+0000 7f410bfff640 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55816fe489f0 0x558170255500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.085+0000 7f410bfff640 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55816fe489f0 0x558170255500 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=796f19fdf46c423a server_cookie=c8619f53e0c2e51 in_seq=0 out_seq=0 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.085+0000 7f40eb7fe640 10 client.28709 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.086+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3934113495 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f40fc00faf0 con 0x55816fe489f0 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.086+0000 7f40eb7fe640 10 client.28709 handle_command_reply: tid=0 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.086+0000 7f4118e09740 1 client.28709 shutdown 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.086+0000 7f4118e09740 1 -- 172.21.15.16:0/3934113495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 msgr2=0x558170236f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.086+0000 7f4118e09740 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 0x558170236f90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f41000e5e90 tx=0x7f41000107e0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.086+0000 7f4118e09740 1 -- 172.21.15.16:0/3934113495 shutdown_connections 2024-08-28T19:36:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.086+0000 7f4118e09740 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe8a070 0x55817024f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.086+0000 7f4118e09740 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 0x558170236f90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.087+0000 7f4118e09740 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55816fe489f0 0x558170255500 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.087+0000 7f40eb7fe640 0 client.28709 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.087+0000 7f4118e09740 1 --2- 172.21.15.16:0/3934113495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe611b0 0x558170236a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.087+0000 7f4118e09740 1 -- 172.21.15.16:0/3934113495 >> 172.21.15.16:0/3934113495 conn(0x55816fe7e460 msgr2=0x55817022b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.087+0000 7f4118e09740 1 -- 172.21.15.16:0/3934113495 shutdown_connections 2024-08-28T19:36:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.087+0000 7f4118e09740 1 -- 172.21.15.16:0/3934113495 wait complete. 2024-08-28T19:36:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.087+0000 7f4118e09740 20 client.28709 trim_cache size 0 max 16384 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f4118e09740 1 Processor -- start 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f4118e09740 1 -- start start 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f4118e09740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe611b0 0x55817024f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f4118e09740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe69390 0x55817024f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f4118e09740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe8a070 0x55817022caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f4118e09740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f410c1aebb0 con 0x55816fe8a070 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f4118e09740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f410c1b09c0 con 0x55816fe611b0 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f4118e09740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f410c1af310 con 0x55816fe69390 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f410bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe611b0 0x55817024f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f4110e31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe8a070 0x55817022caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f410bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe611b0 0x55817024f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:53168/0 (socket says 172.21.15.16:53168) 2024-08-28T19:36:19.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f410bfff640 1 -- 172.21.15.16:0/1304002349 learned_addr learned my addr 172.21.15.16:0/1304002349 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:19.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.108+0000 7f410b7fe640 1 --2- 172.21.15.16:0/1304002349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe69390 0x55817024f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.109+0000 7f410bfff640 1 -- 172.21.15.16:0/1304002349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe69390 msgr2=0x55817024f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:19.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.109+0000 7f410bfff640 1 --2- 172.21.15.16:0/1304002349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe69390 0x55817024f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.109+0000 7f410bfff640 1 -- 172.21.15.16:0/1304002349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe8a070 msgr2=0x55817022caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:19.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.109+0000 7f410bfff640 1 --2- 172.21.15.16:0/1304002349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe8a070 0x55817022caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.109+0000 7f410bfff640 1 -- 172.21.15.16:0/1304002349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558170237d10 con 0x55816fe611b0 2024-08-28T19:36:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.109+0000 7f410bfff640 1 --2- 172.21.15.16:0/1304002349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe611b0 0x55817024f460 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f40fc011510 tx=0x7f40fc005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff8a8bd0c39c35ac server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.109+0000 7f40f1ffb640 1 -- 172.21.15.16:0/1304002349 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc01e040 con 0x55816fe611b0 2024-08-28T19:36:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.109+0000 7f40f1ffb640 1 -- 172.21.15.16:0/1304002349 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40fc00faf0 con 0x55816fe611b0 2024-08-28T19:36:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.109+0000 7f40f1ffb640 1 -- 172.21.15.16:0/1304002349 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc019810 con 0x55816fe611b0 2024-08-28T19:36:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.111+0000 7f4118e09740 1 -- 172.21.15.16:0/1304002349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe611b0 msgr2=0x55817024f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.111+0000 7f4118e09740 1 --2- 172.21.15.16:0/1304002349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe611b0 0x55817024f460 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f40fc011510 tx=0x7f40fc005850 comp rx=0 tx=0).stop 2024-08-28T19:36:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.111+0000 7f4118e09740 1 -- 172.21.15.16:0/1304002349 shutdown_connections 2024-08-28T19:36:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.111+0000 7f4118e09740 1 --2- 172.21.15.16:0/1304002349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe8a070 0x55817022caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.111+0000 7f4118e09740 1 --2- 172.21.15.16:0/1304002349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe69390 0x55817024f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.111+0000 7f4118e09740 1 --2- 172.21.15.16:0/1304002349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe611b0 0x55817024f460 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f40fc011510 tx=0x7f40fc005850 comp rx=0 tx=0).stop 2024-08-28T19:36:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.111+0000 7f4118e09740 1 -- 172.21.15.16:0/1304002349 >> 172.21.15.16:0/1304002349 conn(0x55816fe7e460 msgr2=0x5581702352d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.111+0000 7f4118e09740 1 -- 172.21.15.16:0/1304002349 shutdown_connections 2024-08-28T19:36:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.111+0000 7f4118e09740 1 -- 172.21.15.16:0/1304002349 wait complete. 2024-08-28T19:36:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.112+0000 7f4118e09740 1 -- start start 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.112+0000 7f4118e09740 1 Processor -- start 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.112+0000 7f4118e09740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 0x55817024bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.112+0000 7f4118e09740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe8a070 0x558170247fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.112+0000 7f4118e09740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe1dda0 0x558170248510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f4118e09740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f410c1b0230 con 0x55816fe1dda0 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f4118e09740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f410c1afaa0 con 0x55816fe69390 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f4118e09740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f410c1acbb0 con 0x55816fe8a070 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f4110e31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe1dda0 0x558170248510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f410bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 0x55817024bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f4110e31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe1dda0 0x558170248510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42740/0 (socket says 172.21.15.16:42740) 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f4110e31640 1 -- 172.21.15.16:0/3710931143 learned_addr learned my addr 172.21.15.16:0/3710931143 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f4118e09740 10 client.0 _ll_register_callbacks cb 0x7f410000c7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f410b7fe640 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe8a070 0x558170247fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f410bfff640 1 -- 172.21.15.16:0/3710931143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe8a070 msgr2=0x558170247fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f410bfff640 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe8a070 0x558170247fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f410bfff640 1 -- 172.21.15.16:0/3710931143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe1dda0 msgr2=0x558170248510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f410bfff640 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe1dda0 0x558170248510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f410bfff640 1 -- 172.21.15.16:0/3710931143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558170248b00 con 0x55816fe69390 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.113+0000 7f410b7fe640 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe8a070 0x558170247fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.114+0000 7f410bfff640 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 0x55817024bb10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x558170231c60 tx=0x7f40fc00f440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=544ce66e38268483 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.114+0000 7f40eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.114+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3710931143 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc033070 con 0x55816fe69390 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.114+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3710931143 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f40fc00fdc0 con 0x55816fe69390 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.114+0000 7f4118e09740 1 -- 172.21.15.16:0/3710931143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40fc00faf0 con 0x55816fe69390 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.115+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3710931143 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f40fc01e040 con 0x55816fe69390 2024-08-28T19:36:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.115+0000 7f4118e09740 10 client.28715 fetch_fsmap learned FSMap version 78 2024-08-28T19:36:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.115+0000 7f4118e09740 1 -- 172.21.15.16:0/3710931143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x5581702c0c40 con 0x55816fe69390 2024-08-28T19:36:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.115+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3710931143 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40fc00faf0 con 0x55816fe69390 2024-08-28T19:36:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.115+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3710931143 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 78) ==== 8446+0+0 (secure 0 0 0) 0x7f40fc019770 con 0x55816fe69390 2024-08-28T19:36:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.115+0000 7f4118e09740 10 client.28715 fetch_fsmap finished waiting for FSMap version 78 2024-08-28T19:36:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.115+0000 7f4118e09740 10 client.28715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:36:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.115+0000 7f4118e09740 20 client.28715 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.115+0000 7f4118e09740 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5581702c1070 0x5581702c3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.115+0000 7f4118e09740 4 client.28715 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:36:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.115+0000 7f4118e09740 1 -- 172.21.15.16:0/3710931143 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5581701a8580 con 0x5581702c1070 2024-08-28T19:36:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.116+0000 7f410b7fe640 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5581702c1070 0x5581702c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.116+0000 7f40eb7fe640 10 client.28715 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.116+0000 7f410b7fe640 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5581702c1070 0x5581702c3490 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3ffc923cbe10ac4 server_cookie=6d51483cb1778f5d in_seq=0 out_seq=0 2024-08-28T19:36:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f40eb7fe640 1 -- 172.21.15.16:0/3710931143 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+6250 (crc 0 0 0) 0x5581701a8580 con 0x5581702c1070 2024-08-28T19:36:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f40eb7fe640 10 client.28715 handle_command_reply: tid=0 2024-08-28T19:36:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f4118e09740 1 client.28715 shutdown 2024-08-28T19:36:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f4118e09740 1 -- 172.21.15.16:0/3710931143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 msgr2=0x55817024bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f4118e09740 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 0x55817024bb10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x558170231c60 tx=0x7f40fc00f440 comp rx=0 tx=0).stop 2024-08-28T19:36:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f4118e09740 1 -- 172.21.15.16:0/3710931143 shutdown_connections 2024-08-28T19:36:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f4118e09740 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55816fe1dda0 0x558170248510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f4118e09740 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5581702c1070 0x5581702c3490 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f4118e09740 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55816fe8a070 0x558170247fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f4118e09740 1 --2- 172.21.15.16:0/3710931143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55816fe69390 0x55817024bb10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f4118e09740 1 -- 172.21.15.16:0/3710931143 >> 172.21.15.16:0/3710931143 conn(0x55816fe7e460 msgr2=0x558170234e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f40eb7fe640 0 client.28715 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f4118e09740 1 -- 172.21.15.16:0/3710931143 shutdown_connections 2024-08-28T19:36:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.117+0000 7f4118e09740 1 -- 172.21.15.16:0/3710931143 wait complete. 2024-08-28T19:36:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.118+0000 7f4118e09740 20 client.28715 trim_cache size 0 max 16384 2024-08-28T19:36:19.122 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "4d705fea-6bef-4150-87f6-bc36721d0baa", 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:36:19.123 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.125 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.126 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.126 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.126 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:36:19.126 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:36:19.126 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.126 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.126 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.126 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.126 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.127 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.128 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:36:19.129 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:36:19.132 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:36:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.134 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:19.135 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:19.136 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:19.136 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:19.136 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:19.136 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:19.136 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:19.136 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:36:19.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.118+0000 7f4112894640 1 -- 172.21.15.16:0/1835552253 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f40f40b4da0 msgr2=0x7f40f40b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:19.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.118+0000 7f4112894640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f40f40b4da0 0x7f40f40b72b0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4104005e30 tx=0x7f4104005c50 comp rx=0 tx=0).stop 2024-08-28T19:36:19.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.118+0000 7f4112894640 1 -- 172.21.15.16:0/1835552253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c108140 msgr2=0x7f410c19a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:19.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.118+0000 7f4112894640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c108140 0x7f410c19a830 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f410000e9c0 tx=0x7f410000ee80 comp rx=0 tx=0).stop 2024-08-28T19:36:19.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.118+0000 7f4112894640 1 -- 172.21.15.16:0/1835552253 shutdown_connections 2024-08-28T19:36:19.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.118+0000 7f4112894640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f40f40b4da0 0x7f40f40b72b0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.118+0000 7f4112894640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f410c108a90 0x7f410c19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.118+0000 7f4112894640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c108140 0x7f410c19a830 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.118+0000 7f4112894640 1 --2- 172.21.15.16:0/1835552253 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f410c102800 0x7f410c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:19.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.118+0000 7f4112894640 1 -- 172.21.15.16:0/1835552253 >> 172.21.15.16:0/1835552253 conn(0x7f410c0794d0 msgr2=0x7f410c1050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:19.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.119+0000 7f4112894640 1 -- 172.21.15.16:0/1835552253 shutdown_connections 2024-08-28T19:36:19.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:19.119+0000 7f4112894640 1 -- 172.21.15.16:0/1835552253 wait complete. 2024-08-28T19:36:19.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 35443 18.13 MB/sec warmup 115 sec latency 30628.958 ms 2024-08-28T19:36:19.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:19 smithi016 ceph-mon[18556]: pgmap v815: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 21 op/s 2024-08-28T19:36:19.620 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'd8a4beb8', total seconds elapsed: 34.8 2024-08-28T19:36:19.621 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-08-28T19:36:19.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:36:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:19 smithi028 ceph-mon[21769]: pgmap v815: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 21 op/s 2024-08-28T19:36:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:19 smithi115 ceph-mon[22184]: pgmap v815: 129 pgs: 129 active+clean; 874 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 21 op/s 2024-08-28T19:36:20.430 INFO:tasks.cephfs.filesystem:scrub status for tag:4d705fea-6bef-4150-87f6-bc36721d0baa - {'path': '/', 'tag': '4d705fea-6bef-4150-87f6-bc36721d0baa', 'options': 'recursive,force'} 2024-08-28T19:36:20.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 36189 18.31 MB/sec warmup 116 sec latency 57.052 ms 2024-08-28T19:36:20.511 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:20.647 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:20 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:20.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:20 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:20.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:20 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.085+0000 7fe3db806640 1 -- 172.21.15.16:0/2324544984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3d4103960 msgr2=0x7fe3d4103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.085+0000 7fe3db806640 1 --2- 172.21.15.16:0/2324544984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3d4103960 0x7fe3d4103d70 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009900 tx=0x7fe3d002a410 comp rx=0 tx=0).stop 2024-08-28T19:36:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.085+0000 7fe3db806640 1 -- 172.21.15.16:0/2324544984 shutdown_connections 2024-08-28T19:36:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.085+0000 7fe3db806640 1 --2- 172.21.15.16:0/2324544984 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3d4109ba0 0x7fe3d41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.085+0000 7fe3db806640 1 --2- 172.21.15.16:0/2324544984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3d4109250 0x7fe3d4109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.085+0000 7fe3db806640 1 --2- 172.21.15.16:0/2324544984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3d4103960 0x7fe3d4103d70 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.085+0000 7fe3db806640 1 -- 172.21.15.16:0/2324544984 >> 172.21.15.16:0/2324544984 conn(0x7fe3d407b6b0 msgr2=0x7fe3d407bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.085+0000 7fe3db806640 1 -- 172.21.15.16:0/2324544984 shutdown_connections 2024-08-28T19:36:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.085+0000 7fe3db806640 1 -- 172.21.15.16:0/2324544984 wait complete. 2024-08-28T19:36:21.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3db806640 1 Processor -- start 2024-08-28T19:36:21.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3db806640 1 -- start start 2024-08-28T19:36:21.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3db806640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3d4103960 0x7fe3d419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3db806640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3d4109250 0x7fe3d419c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3db806640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3d4109ba0 0x7fe3d41a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3db806640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d4113860 con 0x7fe3d4103960 2024-08-28T19:36:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3db806640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d41136e0 con 0x7fe3d4109250 2024-08-28T19:36:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3db806640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d41139e0 con 0x7fe3d4109ba0 2024-08-28T19:36:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3d95a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3d4103960 0x7fe3d419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3d8da1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3d4109250 0x7fe3d419c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3d95a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3d4103960 0x7fe3d419c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42762/0 (socket says 172.21.15.16:42762) 2024-08-28T19:36:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3d95a2640 1 -- 172.21.15.16:0/2225040313 learned_addr learned my addr 172.21.15.16:0/2225040313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.086+0000 7fe3d9da3640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3d4109ba0 0x7fe3d41a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.087+0000 7fe3d8da1640 1 -- 172.21.15.16:0/2225040313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3d4109ba0 msgr2=0x7fe3d41a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.087+0000 7fe3d8da1640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3d4109ba0 0x7fe3d41a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.087+0000 7fe3d8da1640 1 -- 172.21.15.16:0/2225040313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3d4103960 msgr2=0x7fe3d419c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.087+0000 7fe3d8da1640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3d4103960 0x7fe3d419c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.087+0000 7fe3d8da1640 1 -- 172.21.15.16:0/2225040313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3d41a23a0 con 0x7fe3d4109250 2024-08-28T19:36:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.087+0000 7fe3d8da1640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3d4109250 0x7fe3d419c9d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe3d406b8d0 tx=0x7fe3c400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.087+0000 7fe3ca7fc640 1 -- 172.21.15.16:0/2225040313 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400cc40 con 0x7fe3d4109250 2024-08-28T19:36:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.087+0000 7fe3ca7fc640 1 -- 172.21.15.16:0/2225040313 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3c400cde0 con 0x7fe3d4109250 2024-08-28T19:36:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.087+0000 7fe3db806640 1 -- 172.21.15.16:0/2225040313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3d41a2690 con 0x7fe3d4109250 2024-08-28T19:36:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.087+0000 7fe3ca7fc640 1 -- 172.21.15.16:0/2225040313 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4018780 con 0x7fe3d4109250 2024-08-28T19:36:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.087+0000 7fe3db806640 1 -- 172.21.15.16:0/2225040313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3d41a8df0 con 0x7fe3d4109250 2024-08-28T19:36:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.088+0000 7fe3db806640 1 -- 172.21.15.16:0/2225040313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3d4103d70 con 0x7fe3d4109250 2024-08-28T19:36:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.090+0000 7fe3ca7fc640 1 -- 172.21.15.16:0/2225040313 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe3c4010a90 con 0x7fe3d4109250 2024-08-28T19:36:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.091+0000 7fe3ca7fc640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3c00b4ca0 0x7fe3c00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.092+0000 7fe3d95a2640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3c00b4ca0 0x7fe3c00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.092+0000 7fe3ca7fc640 1 -- 172.21.15.16:0/2225040313 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7fe3c40e60d0 con 0x7fe3d4109250 2024-08-28T19:36:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.092+0000 7fe3d95a2640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3c00b4ca0 0x7fe3c00b71b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0031440 tx=0x7fe3d00058d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:21.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.095+0000 7fe3ca7fc640 1 -- 172.21.15.16:0/2225040313 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fe3c40ac440 con 0x7fe3d4109250 2024-08-28T19:36:21.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.425+0000 7fe3db806640 1 -- 172.21.15.16:0/2225040313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3d41a9090 con 0x7fe3d4109250 2024-08-28T19:36:21.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.426+0000 7fe3ca7fc640 1 -- 172.21.15.16:0/2225040313 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+12008 (secure 0 0 0) 0x7fe3c40b0c00 con 0x7fe3d4109250 2024-08-28T19:36:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:36:21.428 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":79,"btime":"2024-08-28T19:36:19:378060+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":79,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:36:19.096106+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:36:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 79 2024-08-28T19:36:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 -- 172.21.15.16:0/2225040313 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3c00b4ca0 msgr2=0x7fe3c00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3c00b4ca0 0x7fe3c00b71b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0031440 tx=0x7fe3d00058d0 comp rx=0 tx=0).stop 2024-08-28T19:36:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 -- 172.21.15.16:0/2225040313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3d4109250 msgr2=0x7fe3d419c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3d4109250 0x7fe3d419c9d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe3d406b8d0 tx=0x7fe3c400ee50 comp rx=0 tx=0).stop 2024-08-28T19:36:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 -- 172.21.15.16:0/2225040313 shutdown_connections 2024-08-28T19:36:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3d4109ba0 0x7fe3d41a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3c00b4ca0 0x7fe3c00b71b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3d4109250 0x7fe3d419c9d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 --2- 172.21.15.16:0/2225040313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3d4103960 0x7fe3d419c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:21.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 -- 172.21.15.16:0/2225040313 >> 172.21.15.16:0/2225040313 conn(0x7fe3d407b6b0 msgr2=0x7fe3d41070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:21.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 -- 172.21.15.16:0/2225040313 shutdown_connections 2024-08-28T19:36:21.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:21.430+0000 7fe3db806640 1 -- 172.21.15.16:0/2225040313 wait complete. 2024-08-28T19:36:21.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 36966 18.51 MB/sec warmup 117 sec latency 67.645 ms 2024-08-28T19:36:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:21 smithi016 ceph-mon[18556]: pgmap v816: 129 pgs: 129 active+clean; 875 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 236 KiB/s wr, 30 op/s 2024-08-28T19:36:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:21 smithi028 ceph-mon[21769]: pgmap v816: 129 pgs: 129 active+clean; 875 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 236 KiB/s wr, 30 op/s 2024-08-28T19:36:21.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:21 smithi115 ceph-mon[22184]: pgmap v816: 129 pgs: 129 active+clean; 875 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 236 KiB/s wr, 30 op/s 2024-08-28T19:36:22.129 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-08-28T19:36:22.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-08-28T19:36:22.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 37766 18.68 MB/sec warmup 118 sec latency 99.880 ms 2024-08-28T19:36:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:22.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:22 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2225040313' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:22.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:22 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2225040313' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:22 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2225040313' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:23.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.134+0000 7f28d7cbf640 1 -- 172.21.15.16:0/2974831695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d010dbd0 msgr2=0x7f28d01100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:23.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.134+0000 7f28d7cbf640 1 --2- 172.21.15.16:0/2974831695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d010dbd0 0x7f28d01100c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009be0 tx=0x7f28c802a6c0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.134+0000 7f28d7cbf640 1 -- 172.21.15.16:0/2974831695 shutdown_connections 2024-08-28T19:36:23.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.134+0000 7f28d7cbf640 1 --2- 172.21.15.16:0/2974831695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d010dbd0 0x7f28d01100c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.134+0000 7f28d7cbf640 1 --2- 172.21.15.16:0/2974831695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d01050c0 0x7f28d010d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.134+0000 7f28d7cbf640 1 --2- 172.21.15.16:0/2974831695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28d0104770 0x7f28d0104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.134+0000 7f28d7cbf640 1 -- 172.21.15.16:0/2974831695 >> 172.21.15.16:0/2974831695 conn(0x7f28d0079500 msgr2=0x7f28d0079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:23.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.134+0000 7f28d7cbf640 1 -- 172.21.15.16:0/2974831695 shutdown_connections 2024-08-28T19:36:23.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.134+0000 7f28d7cbf640 1 -- 172.21.15.16:0/2974831695 wait complete. 2024-08-28T19:36:23.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.135+0000 7f28d7cbf640 1 Processor -- start 2024-08-28T19:36:23.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.135+0000 7f28d7cbf640 1 -- start start 2024-08-28T19:36:23.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.135+0000 7f28d7cbf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d0104770 0x7f28d0197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.135+0000 7f28d7cbf640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28d01050c0 0x7f28d0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.135+0000 7f28d7cbf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d010dbd0 0x7f28d019d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.135+0000 7f28d7cbf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28d0103b60 con 0x7f28d0104770 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.135+0000 7f28d7cbf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28d01039e0 con 0x7f28d010dbd0 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.135+0000 7f28d7cbf640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f28d0103ce0 con 0x7f28d01050c0 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.136+0000 7f28d5a5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d0104770 0x7f28d0197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.136+0000 7f28d5a5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d0104770 0x7f28d0197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42782/0 (socket says 172.21.15.16:42782) 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.136+0000 7f28d5a5b640 1 -- 172.21.15.16:0/1858134218 learned_addr learned my addr 172.21.15.16:0/1858134218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.136+0000 7f28d525a640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28d01050c0 0x7f28d0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.136+0000 7f28d625c640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d010dbd0 0x7f28d019d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.136+0000 7f28d5a5b640 1 -- 172.21.15.16:0/1858134218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28d01050c0 msgr2=0x7f28d0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.136+0000 7f28d5a5b640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28d01050c0 0x7f28d0198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.136+0000 7f28d5a5b640 1 -- 172.21.15.16:0/1858134218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d010dbd0 msgr2=0x7f28d019d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:23.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.136+0000 7f28d5a5b640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d010dbd0 0x7f28d019d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.136+0000 7f28d5a5b640 1 -- 172.21.15.16:0/1858134218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d019df20 con 0x7f28d0104770 2024-08-28T19:36:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.136+0000 7f28d5a5b640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d0104770 0x7f28d0197fb0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f28cc00e9c0 tx=0x7f28cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.137+0000 7f28c6ffd640 1 -- 172.21.15.16:0/1858134218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc00cd80 con 0x7f28d0104770 2024-08-28T19:36:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.137+0000 7f28c6ffd640 1 -- 172.21.15.16:0/1858134218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28cc0035f0 con 0x7f28d0104770 2024-08-28T19:36:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.137+0000 7f28d7cbf640 1 -- 172.21.15.16:0/1858134218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d019e210 con 0x7f28d0104770 2024-08-28T19:36:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.137+0000 7f28c6ffd640 1 -- 172.21.15.16:0/1858134218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc010690 con 0x7f28d0104770 2024-08-28T19:36:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.137+0000 7f28d7cbf640 1 -- 172.21.15.16:0/1858134218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d01a4970 con 0x7f28d0104770 2024-08-28T19:36:23.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.141+0000 7f28de234740 1 Processor -- start 2024-08-28T19:36:23.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.141+0000 7f28de234740 1 -- start start 2024-08-28T19:36:23.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.141+0000 7f28c6ffd640 1 -- 172.21.15.16:0/1858134218 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f28cc010830 con 0x7f28d0104770 2024-08-28T19:36:23.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28de234740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da27fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28de234740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da2801d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28de234740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eb7b60 0x55f6da280710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28de234740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6da1f4330 con 0x55f6d9eaf980 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28de234740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6da1e38c0 con 0x55f6d9ed8840 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28de234740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f6da1e7870 con 0x55f6d9eb7b60 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28d625c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da27fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28d625c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da27fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:53238/0 (socket says 172.21.15.16:53238) 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28d625c640 1 -- 172.21.15.16:0/964814678 learned_addr learned my addr 172.21.15.16:0/964814678 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28d525a640 1 --2- 172.21.15.16:0/964814678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eb7b60 0x55f6da280710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28d5a5b640 1 --2- 172.21.15.16:0/964814678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da2801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28d625c640 1 -- 172.21.15.16:0/964814678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eb7b60 msgr2=0x55f6da280710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28d625c640 1 --2- 172.21.15.16:0/964814678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eb7b60 0x55f6da280710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28d625c640 1 -- 172.21.15.16:0/964814678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 msgr2=0x55f6da2801d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:23.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.142+0000 7f28d625c640 1 --2- 172.21.15.16:0/964814678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da2801d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.143+0000 7f28d625c640 1 -- 172.21.15.16:0/964814678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6da19bdf0 con 0x55f6d9ed8840 2024-08-28T19:36:23.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.143+0000 7f28d5a5b640 1 --2- 172.21.15.16:0/964814678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da2801d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:23.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.143+0000 7f28c6ffd640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28bc0b4d20 0x7f28bc0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.143+0000 7f28d625c640 1 --2- 172.21.15.16:0/964814678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da27fc90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f28c802acb0 tx=0x7f28c8030420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e40f926ee4729b82 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.143+0000 7f28d5a5b640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28bc0b4d20 0x7f28bc0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.143+0000 7f28c6ffd640 1 -- 172.21.15.16:0/1858134218 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f28cc0e5d90 con 0x7f28d0104770 2024-08-28T19:36:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.143+0000 7f28bbfff640 1 -- 172.21.15.16:0/964814678 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8041030 con 0x55f6d9ed8840 2024-08-28T19:36:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.143+0000 7f28bbfff640 1 -- 172.21.15.16:0/964814678 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c8003e20 con 0x55f6d9ed8840 2024-08-28T19:36:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.143+0000 7f28bbfff640 1 -- 172.21.15.16:0/964814678 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8036040 con 0x55f6d9ed8840 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.144+0000 7f28de234740 1 -- 172.21.15.16:0/964814678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 msgr2=0x55f6da27fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.144+0000 7f28de234740 1 --2- 172.21.15.16:0/964814678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da27fc90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f28c802acb0 tx=0x7f28c8030420 comp rx=0 tx=0).stop 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.144+0000 7f28d5a5b640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28bc0b4d20 0x7f28bc0b7230 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0e31c0 tx=0x7f28cc003e70 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.144+0000 7f28de234740 1 -- 172.21.15.16:0/964814678 shutdown_connections 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.144+0000 7f28de234740 1 --2- 172.21.15.16:0/964814678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eb7b60 0x55f6da280710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.144+0000 7f28de234740 1 --2- 172.21.15.16:0/964814678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da2801d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.144+0000 7f28de234740 1 --2- 172.21.15.16:0/964814678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da27fc90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.144+0000 7f28de234740 1 -- 172.21.15.16:0/964814678 >> 172.21.15.16:0/964814678 conn(0x55f6d9ecc990 msgr2=0x55f6da27daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.145+0000 7f28de234740 1 -- 172.21.15.16:0/964814678 shutdown_connections 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.145+0000 7f28de234740 1 -- 172.21.15.16:0/964814678 wait complete. 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.145+0000 7f28de234740 1 -- start start 2024-08-28T19:36:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.145+0000 7f28de234740 1 Processor -- start 2024-08-28T19:36:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28de234740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da293c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28de234740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9eb7b60 0x55f6da294190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28de234740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9ed8840 0x55f6da29fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28de234740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6da0467a0 con 0x55f6d9eaf980 2024-08-28T19:36:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28de234740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6da16a240 con 0x55f6d9eb7b60 2024-08-28T19:36:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28de234740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f6da296b40 con 0x55f6d9ed8840 2024-08-28T19:36:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28de234740 10 client.0 _ll_register_callbacks cb 0x55f6d9ef0090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28d625c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da293c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28d625c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da293c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42804/0 (socket says 172.21.15.16:42804) 2024-08-28T19:36:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28d625c640 1 -- 172.21.15.16:0/3974263436 learned_addr learned my addr 172.21.15.16:0/3974263436 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28d525a640 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9eb7b60 0x55f6da294190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28d625c640 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9ed8840 0x55f6da29fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28d625c640 1 -- 172.21.15.16:0/3974263436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9ed8840 msgr2=0x55f6da29fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28d625c640 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9ed8840 0x55f6da29fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28d625c640 1 -- 172.21.15.16:0/3974263436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9eb7b60 msgr2=0x55f6da294190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28d625c640 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9eb7b60 0x55f6da294190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28d625c640 1 -- 172.21.15.16:0/3974263436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6da1a02b0 con 0x55f6d9eaf980 2024-08-28T19:36:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28d625c640 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da293c50 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f28c80308e0 tx=0x7f28c802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2fe608148eda343 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.146+0000 7f28b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.147+0000 7f28b97fa640 1 -- 172.21.15.16:0/3974263436 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8036070 con 0x55f6d9eaf980 2024-08-28T19:36:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.147+0000 7f28b97fa640 1 -- 172.21.15.16:0/3974263436 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f28c8003ad0 con 0x55f6d9eaf980 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.147+0000 7f28de234740 1 -- 172.21.15.16:0/3974263436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28c8003e20 con 0x55f6d9eaf980 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.147+0000 7f28b97fa640 1 -- 172.21.15.16:0/3974263436 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f28c805a050 con 0x55f6d9eaf980 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.147+0000 7f28de234740 10 client.19308 fetch_fsmap learned FSMap version 80 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.147+0000 7f28de234740 1 -- 172.21.15.16:0/3974263436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=4+}) -- 0x55f6d9f52b90 con 0x55f6d9eaf980 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.147+0000 7f28b97fa640 1 -- 172.21.15.16:0/3974263436 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c8003e20 con 0x55f6d9eaf980 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.148+0000 7f28b97fa640 1 -- 172.21.15.16:0/3974263436 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 80) ==== 8438+0+0 (secure 0 0 0) 0x7f28c80046a0 con 0x55f6d9eaf980 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.148+0000 7f28de234740 10 client.19308 fetch_fsmap finished waiting for FSMap version 80 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.148+0000 7f28de234740 10 client.19308 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.148+0000 7f28de234740 20 client.19308 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.148+0000 7f28de234740 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f6d9e971c0 0x55f6da2a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.148+0000 7f28de234740 4 client.19308 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.148+0000 7f28de234740 1 -- 172.21.15.16:0/3974263436 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6da1f6280 con 0x55f6d9e971c0 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.148+0000 7f28d525a640 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f6d9e971c0 0x55f6da2a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.149+0000 7f28b97fa640 10 client.19308 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.149+0000 7f28d525a640 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f6d9e971c0 0x55f6da2a5b10 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f2ee7ac841eedd6 server_cookie=28385d3577fdfe85 in_seq=0 out_seq=0 2024-08-28T19:36:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28b97fa640 1 -- 172.21.15.16:0/3974263436 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f6da1f6280 con 0x55f6d9e971c0 2024-08-28T19:36:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28b97fa640 10 client.19308 handle_command_reply: tid=0 2024-08-28T19:36:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 1 client.19308 shutdown 2024-08-28T19:36:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 1 -- 172.21.15.16:0/3974263436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 msgr2=0x55f6da293c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da293c50 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f28c80308e0 tx=0x7f28c802a6c0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 1 -- 172.21.15.16:0/3974263436 shutdown_connections 2024-08-28T19:36:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f6d9e971c0 0x55f6da2a5b10 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9ed8840 0x55f6da29fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9eb7b60 0x55f6da294190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 1 --2- 172.21.15.16:0/3974263436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da293c50 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28b97fa640 0 client.19308 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 1 -- 172.21.15.16:0/3974263436 >> 172.21.15.16:0/3974263436 conn(0x55f6d9ecc990 msgr2=0x55f6da282eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 1 -- 172.21.15.16:0/3974263436 shutdown_connections 2024-08-28T19:36:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 1 -- 172.21.15.16:0/3974263436 wait complete. 2024-08-28T19:36:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.150+0000 7f28de234740 20 client.19308 trim_cache size 0 max 16384 2024-08-28T19:36:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.173+0000 7f28de234740 1 Processor -- start 2024-08-28T19:36:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.173+0000 7f28de234740 1 -- start start 2024-08-28T19:36:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28de234740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da295880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28de234740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eb7b60 0x55f6da29fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28de234740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da290e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28de234740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28d01aeac0 con 0x55f6d9eaf980 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28de234740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28d01b08a0 con 0x55f6d9ed8840 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28de234740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f28d01af220 con 0x55f6d9eb7b60 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28d625c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da295880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28d625c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da295880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42812/0 (socket says 172.21.15.16:42812) 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28d625c640 1 -- 172.21.15.16:0/1626681420 learned_addr learned my addr 172.21.15.16:0/1626681420 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28d525a640 1 --2- 172.21.15.16:0/1626681420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eb7b60 0x55f6da29fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28d625c640 1 --2- 172.21.15.16:0/1626681420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da290e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28d625c640 1 -- 172.21.15.16:0/1626681420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eb7b60 msgr2=0x55f6da29fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28d625c640 1 --2- 172.21.15.16:0/1626681420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eb7b60 0x55f6da29fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28d625c640 1 -- 172.21.15.16:0/1626681420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 msgr2=0x55f6da290e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28d625c640 1 --2- 172.21.15.16:0/1626681420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da290e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.174+0000 7f28d625c640 1 -- 172.21.15.16:0/1626681420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6da1b3d90 con 0x55f6d9eaf980 2024-08-28T19:36:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.175+0000 7f28d625c640 1 --2- 172.21.15.16:0/1626681420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da295880 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009300 tx=0x7f28c8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529a04a4b0e7a56 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.175+0000 7f28bb7fe640 1 -- 172.21.15.16:0/1626681420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8036040 con 0x55f6d9eaf980 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.175+0000 7f28bb7fe640 1 -- 172.21.15.16:0/1626681420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c803d5f0 con 0x55f6d9eaf980 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.175+0000 7f28bb7fe640 1 -- 172.21.15.16:0/1626681420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8041040 con 0x55f6d9eaf980 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.176+0000 7f28de234740 1 -- 172.21.15.16:0/1626681420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 msgr2=0x55f6da295880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.176+0000 7f28de234740 1 --2- 172.21.15.16:0/1626681420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da295880 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009300 tx=0x7f28c8000f40 comp rx=0 tx=0).stop 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.176+0000 7f28de234740 1 -- 172.21.15.16:0/1626681420 shutdown_connections 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.176+0000 7f28de234740 1 --2- 172.21.15.16:0/1626681420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da290e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.176+0000 7f28de234740 1 --2- 172.21.15.16:0/1626681420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eb7b60 0x55f6da29fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.176+0000 7f28de234740 1 --2- 172.21.15.16:0/1626681420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eaf980 0x55f6da295880 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.176+0000 7f28de234740 1 -- 172.21.15.16:0/1626681420 >> 172.21.15.16:0/1626681420 conn(0x55f6d9ecc990 msgr2=0x55f6da295170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.177+0000 7f28de234740 1 -- 172.21.15.16:0/1626681420 shutdown_connections 2024-08-28T19:36:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.177+0000 7f28de234740 1 -- 172.21.15.16:0/1626681420 wait complete. 2024-08-28T19:36:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.177+0000 7f28de234740 1 -- start start 2024-08-28T19:36:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.177+0000 7f28de234740 1 Processor -- start 2024-08-28T19:36:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28de234740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eaf980 0x55f6da2a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28de234740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eb7b60 0x55f6da2a7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28de234740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da30f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28de234740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28d01b0110 con 0x55f6d9eb7b60 2024-08-28T19:36:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28de234740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28d01af980 con 0x55f6d9ed8840 2024-08-28T19:36:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28de234740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f28d01acac0 con 0x55f6d9eaf980 2024-08-28T19:36:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28de234740 10 client.0 _ll_register_callbacks cb 0x7f28c0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28d525a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eb7b60 0x55f6da2a7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28d625c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da30f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28d525a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eb7b60 0x55f6da2a7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42822/0 (socket says 172.21.15.16:42822) 2024-08-28T19:36:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28d525a640 1 -- 172.21.15.16:0/1048400266 learned_addr learned my addr 172.21.15.16:0/1048400266 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28d625c640 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eaf980 0x55f6da2a71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28d525a640 1 -- 172.21.15.16:0/1048400266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eaf980 msgr2=0x55f6da2a71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28d525a640 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eaf980 0x55f6da2a71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28d525a640 1 -- 172.21.15.16:0/1048400266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 msgr2=0x55f6da30f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.178+0000 7f28d525a640 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da30f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.179+0000 7f28d525a640 1 -- 172.21.15.16:0/1048400266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6da30f7d0 con 0x55f6d9eb7b60 2024-08-28T19:36:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.179+0000 7f28d625c640 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da30f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.179+0000 7f28d525a640 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eb7b60 0x55f6da2a7730 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f28d006a630 tx=0x7f28c00109b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697b7c2c2e3e1d29 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.179+0000 7f28b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.179+0000 7f28b97fa640 1 -- 172.21.15.16:0/1048400266 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c0004220 con 0x55f6d9eb7b60 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.179+0000 7f28b97fa640 1 -- 172.21.15.16:0/1048400266 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f28c00043c0 con 0x55f6d9eb7b60 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.179+0000 7f28de234740 1 -- 172.21.15.16:0/1048400266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28c803d5f0 con 0x55f6d9eb7b60 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.179+0000 7f28b97fa640 1 -- 172.21.15.16:0/1048400266 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f28c001f790 con 0x55f6d9eb7b60 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.179+0000 7f28de234740 10 client.19314 fetch_fsmap learned FSMap version 80 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.179+0000 7f28de234740 1 -- 172.21.15.16:0/1048400266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=4+}) -- 0x55f6da310a00 con 0x55f6d9eb7b60 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.180+0000 7f28b97fa640 1 -- 172.21.15.16:0/1048400266 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c803d5f0 con 0x55f6d9eb7b60 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.180+0000 7f28b97fa640 1 -- 172.21.15.16:0/1048400266 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 80) ==== 8438+0+0 (secure 0 0 0) 0x7f28c0003bd0 con 0x55f6d9eb7b60 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.180+0000 7f28de234740 10 client.19314 fetch_fsmap finished waiting for FSMap version 80 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.180+0000 7f28de234740 10 client.19314 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.180+0000 7f28de234740 20 client.19314 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.180+0000 7f28de234740 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f6d9e6c570 0x55f6da313370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.180+0000 7f28de234740 4 client.19314 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-08-28T19:36:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.180+0000 7f28de234740 1 -- 172.21.15.16:0/1048400266 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x55f6da1f1d20 con 0x55f6d9e6c570 2024-08-28T19:36:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.180+0000 7f28d625c640 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f6d9e6c570 0x55f6da313370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.181+0000 7f28b97fa640 10 client.19314 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.181+0000 7f28d625c640 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f6d9e6c570 0x55f6da313370 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3e50d0595e24965 server_cookie=5e1d5bec3a87ff00 in_seq=0 out_seq=0 2024-08-28T19:36:23.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38399 18.81 MB/sec warmup 119 sec latency 290.214 ms 2024-08-28T19:36:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28b97fa640 1 -- 172.21.15.16:0/1048400266 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f6da1f1d20 con 0x55f6d9e6c570 2024-08-28T19:36:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28b97fa640 10 client.19314 handle_command_reply: tid=0 2024-08-28T19:36:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28de234740 1 client.19314 shutdown 2024-08-28T19:36:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28de234740 1 -- 172.21.15.16:0/1048400266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eb7b60 msgr2=0x55f6da2a7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28de234740 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eb7b60 0x55f6da2a7730 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f28d006a630 tx=0x7f28c00109b0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28de234740 1 -- 172.21.15.16:0/1048400266 shutdown_connections 2024-08-28T19:36:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28de234740 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6d9ed8840 0x55f6da30f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28de234740 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d9eb7b60 0x55f6da2a7730 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28de234740 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f6d9e6c570 0x55f6da313370 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28de234740 1 --2- 172.21.15.16:0/1048400266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6d9eaf980 0x55f6da2a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28b97fa640 0 client.19314 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28de234740 1 -- 172.21.15.16:0/1048400266 >> 172.21.15.16:0/1048400266 conn(0x55f6d9ecc990 msgr2=0x55f6da27c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28de234740 1 -- 172.21.15.16:0/1048400266 shutdown_connections 2024-08-28T19:36:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.624+0000 7f28de234740 1 -- 172.21.15.16:0/1048400266 wait complete. 2024-08-28T19:36:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28de234740 20 client.19314 trim_cache size 0 max 16384 2024-08-28T19:36:23.627 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:36:23.627 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 80, 2024-08-28T19:36:23.627 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:36:23.627 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 50, 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "e447cd33": { 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 50, 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:36:23.628 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:36:23.629 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:36:23.629 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:36:23.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:23.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:23.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:23.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:23.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:23.629 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 -- 172.21.15.16:0/1858134218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28bc0b4d20 msgr2=0x7f28bc0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28bc0b4d20 0x7f28bc0b7230 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0e31c0 tx=0x7f28cc003e70 comp rx=0 tx=0).stop 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 -- 172.21.15.16:0/1858134218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d0104770 msgr2=0x7f28d0197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d0104770 0x7f28d0197fb0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f28cc00e9c0 tx=0x7f28cc00ee80 comp rx=0 tx=0).stop 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 -- 172.21.15.16:0/1858134218 shutdown_connections 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f28bc0b4d20 0x7f28bc0b7230 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d010dbd0 0x7f28d019d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28d01050c0 0x7f28d0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 --2- 172.21.15.16:0/1858134218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d0104770 0x7f28d0197fb0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 -- 172.21.15.16:0/1858134218 >> 172.21.15.16:0/1858134218 conn(0x7f28d0079500 msgr2=0x7f28d01055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 -- 172.21.15.16:0/1858134218 shutdown_connections 2024-08-28T19:36:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:23.625+0000 7f28d7cbf640 1 -- 172.21.15.16:0/1858134218 wait complete. 2024-08-28T19:36:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:23 smithi016 ceph-mon[18556]: pgmap v817: 129 pgs: 129 active+clean; 887 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-08-28T19:36:23.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:23 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:23.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:23 smithi028 ceph-mon[21769]: pgmap v817: 129 pgs: 129 active+clean; 887 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-08-28T19:36:23.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:23 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:23 smithi115 ceph-mon[22184]: pgmap v817: 129 pgs: 129 active+clean; 887 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-08-28T19:36:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:23 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:24.342 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'e447cd33', quiesce took 4.7 seconds. Will release after: 25.3 2024-08-28T19:36:25.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 1 sec latency 2290.365 ms 2024-08-28T19:36:25.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:25 smithi016 ceph-mon[18556]: pgmap v818: 129 pgs: 129 active+clean; 889 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 159 op/s 2024-08-28T19:36:25.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:25 smithi028 ceph-mon[21769]: pgmap v818: 129 pgs: 129 active+clean; 889 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 159 op/s 2024-08-28T19:36:25.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:25 smithi115 ceph-mon[22184]: pgmap v818: 129 pgs: 129 active+clean; 889 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 159 op/s 2024-08-28T19:36:26.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 2 sec latency 3290.476 ms 2024-08-28T19:36:27.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 3 sec latency 4290.527 ms 2024-08-28T19:36:27.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:27 smithi016 ceph-mon[18556]: pgmap v819: 129 pgs: 129 active+clean; 903 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.1 MiB/s wr, 318 op/s 2024-08-28T19:36:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:27 smithi028 ceph-mon[21769]: pgmap v819: 129 pgs: 129 active+clean; 903 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.1 MiB/s wr, 318 op/s 2024-08-28T19:36:27.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:27 smithi115 ceph-mon[22184]: pgmap v819: 129 pgs: 129 active+clean; 903 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.1 MiB/s wr, 318 op/s 2024-08-28T19:36:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 4 sec latency 5290.568 ms 2024-08-28T19:36:28.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:28 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:36:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:28 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:36:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:28 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:36:29.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 5 sec latency 6290.657 ms 2024-08-28T19:36:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:29 smithi016 ceph-mon[18556]: pgmap v820: 129 pgs: 129 active+clean; 910 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.3 MiB/s wr, 336 op/s 2024-08-28T19:36:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:29 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:36:29.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:29 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:36:29.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:29 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:36:29.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:29 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:36:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:29 smithi028 ceph-mon[21769]: pgmap v820: 129 pgs: 129 active+clean; 910 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.3 MiB/s wr, 336 op/s 2024-08-28T19:36:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:29 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:36:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:29 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:36:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:29 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:36:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:29 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:36:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:29 smithi115 ceph-mon[22184]: pgmap v820: 129 pgs: 129 active+clean; 910 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.3 MiB/s wr, 336 op/s 2024-08-28T19:36:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:29 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:36:29.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:29 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:36:29.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:29 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:36:29.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:29 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:36:30.467 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 6 sec latency 7290.760 ms 2024-08-28T19:36:31.468 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 7 sec latency 8290.846 ms 2024-08-28T19:36:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:31 smithi016 ceph-mon[18556]: pgmap v821: 129 pgs: 129 active+clean; 910 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-08-28T19:36:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:31 smithi028 ceph-mon[21769]: pgmap v821: 129 pgs: 129 active+clean; 910 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-08-28T19:36:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:31 smithi115 ceph-mon[22184]: pgmap v821: 129 pgs: 129 active+clean; 910 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-08-28T19:36:32.468 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 8 sec latency 9290.944 ms 2024-08-28T19:36:32.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:32 smithi016 ceph-mon[18556]: pgmap v822: 129 pgs: 129 active+clean; 912 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.4 MiB/s wr, 335 op/s 2024-08-28T19:36:32.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:32 smithi028 ceph-mon[21769]: pgmap v822: 129 pgs: 129 active+clean; 912 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.4 MiB/s wr, 335 op/s 2024-08-28T19:36:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:32 smithi115 ceph-mon[22184]: pgmap v822: 129 pgs: 129 active+clean; 912 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.4 MiB/s wr, 335 op/s 2024-08-28T19:36:33.468 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 9 sec latency 10291.054 ms 2024-08-28T19:36:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:33 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:33.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:33 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:33 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:34.468 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 10 sec latency 11291.160 ms 2024-08-28T19:36:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:34 smithi016 ceph-mon[18556]: pgmap v823: 129 pgs: 129 active+clean; 913 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.4 MiB/s wr, 232 op/s 2024-08-28T19:36:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:34 smithi028 ceph-mon[21769]: pgmap v823: 129 pgs: 129 active+clean; 913 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.4 MiB/s wr, 232 op/s 2024-08-28T19:36:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:34 smithi115 ceph-mon[22184]: pgmap v823: 129 pgs: 129 active+clean; 913 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.4 MiB/s wr, 232 op/s 2024-08-28T19:36:35.468 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 11 sec latency 12291.278 ms 2024-08-28T19:36:36.468 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 12 sec latency 13291.353 ms 2024-08-28T19:36:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:36 smithi115 ceph-mon[22184]: pgmap v824: 129 pgs: 129 active+clean; 916 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.6 MiB/s wr, 231 op/s 2024-08-28T19:36:37.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:36 smithi016 ceph-mon[18556]: pgmap v824: 129 pgs: 129 active+clean; 916 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.6 MiB/s wr, 231 op/s 2024-08-28T19:36:37.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:36 smithi028 ceph-mon[21769]: pgmap v824: 129 pgs: 129 active+clean; 916 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.6 MiB/s wr, 231 op/s 2024-08-28T19:36:37.468 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 13 sec latency 14291.462 ms 2024-08-28T19:36:38.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:37 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:38.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:37 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:38.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:37 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:36:38.468 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 14 sec latency 15291.590 ms 2024-08-28T19:36:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:38 smithi115 ceph-mon[22184]: pgmap v825: 129 pgs: 129 active+clean; 917 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 90 op/s 2024-08-28T19:36:39.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:38 smithi016 ceph-mon[18556]: pgmap v825: 129 pgs: 129 active+clean; 917 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 90 op/s 2024-08-28T19:36:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:38 smithi028 ceph-mon[21769]: pgmap v825: 129 pgs: 129 active+clean; 917 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 90 op/s 2024-08-28T19:36:39.468 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 15 sec latency 16291.710 ms 2024-08-28T19:36:40.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 16 sec latency 17291.843 ms 2024-08-28T19:36:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:40 smithi115 ceph-mon[22184]: pgmap v826: 129 pgs: 129 active+clean; 917 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 681 KiB/s wr, 30 op/s 2024-08-28T19:36:41.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:40 smithi016 ceph-mon[18556]: pgmap v826: 129 pgs: 129 active+clean; 917 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 681 KiB/s wr, 30 op/s 2024-08-28T19:36:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:40 smithi028 ceph-mon[21769]: pgmap v826: 129 pgs: 129 active+clean; 917 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 681 KiB/s wr, 30 op/s 2024-08-28T19:36:41.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 17 sec latency 18291.908 ms 2024-08-28T19:36:42.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 18 sec latency 19292.019 ms 2024-08-28T19:36:43.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:42 smithi016 ceph-mon[18556]: pgmap v827: 129 pgs: 129 active+clean; 918 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 695 KiB/s wr, 36 op/s 2024-08-28T19:36:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:42 smithi028 ceph-mon[21769]: pgmap v827: 129 pgs: 129 active+clean; 918 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 695 KiB/s wr, 36 op/s 2024-08-28T19:36:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:42 smithi115 ceph-mon[22184]: pgmap v827: 129 pgs: 129 active+clean; 918 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 695 KiB/s wr, 36 op/s 2024-08-28T19:36:43.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 19 sec latency 20292.105 ms 2024-08-28T19:36:44.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 20 sec latency 21292.206 ms 2024-08-28T19:36:45.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:44 smithi016 ceph-mon[18556]: pgmap v828: 129 pgs: 129 active+clean; 918 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 554 KiB/s wr, 24 op/s 2024-08-28T19:36:45.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:44 smithi028 ceph-mon[21769]: pgmap v828: 129 pgs: 129 active+clean; 918 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 554 KiB/s wr, 24 op/s 2024-08-28T19:36:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:44 smithi115 ceph-mon[22184]: pgmap v828: 129 pgs: 129 active+clean; 918 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 554 KiB/s wr, 24 op/s 2024-08-28T19:36:45.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 21 sec latency 22292.259 ms 2024-08-28T19:36:46.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 22 sec latency 23292.351 ms 2024-08-28T19:36:47.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:46 smithi016 ceph-mon[18556]: pgmap v829: 129 pgs: 129 active+clean; 919 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 529 KiB/s wr, 28 op/s 2024-08-28T19:36:47.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:46 smithi028 ceph-mon[21769]: pgmap v829: 129 pgs: 129 active+clean; 919 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 529 KiB/s wr, 28 op/s 2024-08-28T19:36:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:46 smithi115 ceph-mon[22184]: pgmap v829: 129 pgs: 129 active+clean; 919 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 529 KiB/s wr, 28 op/s 2024-08-28T19:36:47.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 23 sec latency 24292.445 ms 2024-08-28T19:36:48.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 24 sec latency 25292.554 ms 2024-08-28T19:36:49.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:48 smithi016 ceph-mon[18556]: pgmap v830: 129 pgs: 129 active+clean; 919 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 252 KiB/s wr, 21 op/s 2024-08-28T19:36:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:48 smithi028 ceph-mon[21769]: pgmap v830: 129 pgs: 129 active+clean; 919 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 252 KiB/s wr, 21 op/s 2024-08-28T19:36:49.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:48 smithi115 ceph-mon[22184]: pgmap v830: 129 pgs: 129 active+clean; 919 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 252 KiB/s wr, 21 op/s 2024-08-28T19:36:49.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 25 sec latency 26292.605 ms 2024-08-28T19:36:49.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:36:49.899 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.409+0000 7f8aa7919640 1 -- 172.21.15.16:0/375484541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8aa01050a0 msgr2=0x7f8aa010d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.409+0000 7f8aa7919640 1 --2- 172.21.15.16:0/375484541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8aa01050a0 0x7f8aa010d5f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009860 tx=0x7f8a9002a340 comp rx=0 tx=0).stop 2024-08-28T19:36:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.409+0000 7f8aa7919640 1 -- 172.21.15.16:0/375484541 shutdown_connections 2024-08-28T19:36:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.409+0000 7f8aa7919640 1 --2- 172.21.15.16:0/375484541 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8aa010dbb0 0x7f8aa01100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.409+0000 7f8aa7919640 1 --2- 172.21.15.16:0/375484541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8aa01050a0 0x7f8aa010d5f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.409+0000 7f8aa7919640 1 --2- 172.21.15.16:0/375484541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa0104750 0x7f8aa0104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.409+0000 7f8aa7919640 1 -- 172.21.15.16:0/375484541 >> 172.21.15.16:0/375484541 conn(0x7f8aa00794d0 msgr2=0x7f8aa00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.409+0000 7f8aa7919640 1 -- 172.21.15.16:0/375484541 shutdown_connections 2024-08-28T19:36:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.409+0000 7f8aa7919640 1 -- 172.21.15.16:0/375484541 wait complete. 2024-08-28T19:36:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.410+0000 7f8aa7919640 1 Processor -- start 2024-08-28T19:36:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.410+0000 7f8aa7919640 1 -- start start 2024-08-28T19:36:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.410+0000 7f8aa7919640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8aa0104750 0x7f8aa019c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.410+0000 7f8aa7919640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa01050a0 0x7f8aa019ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.410+0000 7f8aa7919640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8aa010dbb0 0x7f8aa01a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.410+0000 7f8aa7919640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa01137d0 con 0x7f8aa01050a0 2024-08-28T19:36:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.410+0000 7f8aa7919640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa0113650 con 0x7f8aa010dbb0 2024-08-28T19:36:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.410+0000 7f8aa7919640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa0113950 con 0x7f8aa0104750 2024-08-28T19:36:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.410+0000 7f8aa4eb4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa01050a0 0x7f8aa019ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.410+0000 7f8aa4eb4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa01050a0 0x7f8aa019ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59852/0 (socket says 172.21.15.16:59852) 2024-08-28T19:36:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.410+0000 7f8aa4eb4640 1 -- 172.21.15.16:0/1773722374 learned_addr learned my addr 172.21.15.16:0/1773722374 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.411+0000 7f8aa56b5640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8aa0104750 0x7f8aa019c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.411+0000 7f8aa4eb4640 1 -- 172.21.15.16:0/1773722374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8aa0104750 msgr2=0x7f8aa019c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.411+0000 7f8aa5eb6640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8aa010dbb0 0x7f8aa01a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.411+0000 7f8aa4eb4640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8aa0104750 0x7f8aa019c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.411+0000 7f8aa4eb4640 1 -- 172.21.15.16:0/1773722374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8aa010dbb0 msgr2=0x7f8aa01a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.411+0000 7f8aa4eb4640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8aa010dbb0 0x7f8aa01a1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.411+0000 7f8aa4eb4640 1 -- 172.21.15.16:0/1773722374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa01a23a0 con 0x7f8aa01050a0 2024-08-28T19:36:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.411+0000 7f8aa5eb6640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8aa010dbb0 0x7f8aa01a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.411+0000 7f8aa56b5640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8aa0104750 0x7f8aa019c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.412+0000 7f8aa4eb4640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa01050a0 0x7f8aa019ca00 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f8a90035040 tx=0x7f8a90032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:50.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.413+0000 7f8a967fc640 1 -- 172.21.15.16:0/1773722374 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90040020 con 0x7f8aa01050a0 2024-08-28T19:36:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.413+0000 7f8a967fc640 1 -- 172.21.15.16:0/1773722374 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a90030e10 con 0x7f8aa01050a0 2024-08-28T19:36:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.413+0000 7f8aa7919640 1 -- 172.21.15.16:0/1773722374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa01a2630 con 0x7f8aa01050a0 2024-08-28T19:36:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.413+0000 7f8aa7919640 1 -- 172.21.15.16:0/1773722374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa0105590 con 0x7f8aa01050a0 2024-08-28T19:36:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.413+0000 7f8a967fc640 1 -- 172.21.15.16:0/1773722374 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90037040 con 0x7f8aa01050a0 2024-08-28T19:36:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.415+0000 7f8aa7919640 1 -- 172.21.15.16:0/1773722374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a840052c0 con 0x7f8aa01050a0 2024-08-28T19:36:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.420+0000 7f8a967fc640 1 -- 172.21.15.16:0/1773722374 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8a90043020 con 0x7f8aa01050a0 2024-08-28T19:36:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.422+0000 7f8a967fc640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a8c0b4ea0 0x7f8a8c0b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.422+0000 7f8a967fc640 1 -- 172.21.15.16:0/1773722374 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f8a90105a80 con 0x7f8aa01050a0 2024-08-28T19:36:50.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.423+0000 7f8aa56b5640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a8c0b4ea0 0x7f8a8c0b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:50.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.424+0000 7f8aa56b5640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a8c0b4ea0 0x7f8a8c0b73b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8aa006ba00 tx=0x7f8a9c009260 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.426+0000 7f8a967fc640 1 -- 172.21.15.16:0/1773722374 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8a9004a020 con 0x7f8aa01050a0 2024-08-28T19:36:50.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:36:50.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 26 sec latency 27292.648 ms 2024-08-28T19:36:50.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.727+0000 7f8aa7919640 1 -- 172.21.15.16:0/1773722374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a840055b0 con 0x7f8aa01050a0 2024-08-28T19:36:50.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.728+0000 7f8a967fc640 1 -- 172.21.15.16:0/1773722374 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+12002 (secure 0 0 0) 0x7f8a900cc5d0 con 0x7f8aa01050a0 2024-08-28T19:36:50.729 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:36:50.729 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"btime":"2024-08-28T19:36:36:851162+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":82,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:36:36.606659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:36:50.730 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 82 2024-08-28T19:36:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.730+0000 7f8aa7919640 1 -- 172.21.15.16:0/1773722374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a8c0b4ea0 msgr2=0x7f8a8c0b73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.730+0000 7f8aa7919640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a8c0b4ea0 0x7f8a8c0b73b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8aa006ba00 tx=0x7f8a9c009260 comp rx=0 tx=0).stop 2024-08-28T19:36:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.731+0000 7f8aa7919640 1 -- 172.21.15.16:0/1773722374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa01050a0 msgr2=0x7f8aa019ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.731+0000 7f8aa7919640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa01050a0 0x7f8aa019ca00 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f8a90035040 tx=0x7f8a90032d50 comp rx=0 tx=0).stop 2024-08-28T19:36:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.731+0000 7f8aa7919640 1 -- 172.21.15.16:0/1773722374 shutdown_connections 2024-08-28T19:36:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.731+0000 7f8aa7919640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8a8c0b4ea0 0x7f8a8c0b73b0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.731+0000 7f8aa7919640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8aa010dbb0 0x7f8aa01a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.731+0000 7f8aa7919640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa01050a0 0x7f8aa019ca00 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:50.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.731+0000 7f8aa7919640 1 --2- 172.21.15.16:0/1773722374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8aa0104750 0x7f8aa019c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:50.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.731+0000 7f8aa7919640 1 -- 172.21.15.16:0/1773722374 >> 172.21.15.16:0/1773722374 conn(0x7f8aa00794d0 msgr2=0x7f8aa010b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:50.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.731+0000 7f8aa7919640 1 -- 172.21.15.16:0/1773722374 shutdown_connections 2024-08-28T19:36:50.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:50.731+0000 7f8aa7919640 1 -- 172.21.15.16:0/1773722374 wait complete. 2024-08-28T19:36:50.918 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:51.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:50 smithi016 ceph-mon[18556]: pgmap v831: 129 pgs: 129 active+clean; 921 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 303 KiB/s wr, 20 op/s 2024-08-28T19:36:51.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:50 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1773722374' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:51.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:50 smithi028 ceph-mon[21769]: pgmap v831: 129 pgs: 129 active+clean; 921 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 303 KiB/s wr, 20 op/s 2024-08-28T19:36:51.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:50 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1773722374' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:51.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:50 smithi115 ceph-mon[22184]: pgmap v831: 129 pgs: 129 active+clean; 921 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 303 KiB/s wr, 20 op/s 2024-08-28T19:36:51.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:50 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1773722374' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:51.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 27 sec latency 28292.697 ms 2024-08-28T19:36:51.732 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id e447cd33 --release --await' 2024-08-28T19:36:51.733 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id e447cd33 --release --await 2024-08-28T19:36:51.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.899+0000 7ffb7f146640 1 -- 172.21.15.16:0/3310800104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb78109260 msgr2=0x7ffb78109670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:51.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.899+0000 7ffb7f146640 1 --2- 172.21.15.16:0/3310800104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb78109260 0x7ffb78109670 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009900 tx=0x7ffb6402a410 comp rx=0 tx=0).stop 2024-08-28T19:36:51.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.899+0000 7ffb7f146640 1 -- 172.21.15.16:0/3310800104 shutdown_connections 2024-08-28T19:36:51.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.899+0000 7ffb7f146640 1 --2- 172.21.15.16:0/3310800104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb78109bb0 0x7ffb781103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.899+0000 7ffb7f146640 1 --2- 172.21.15.16:0/3310800104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb78109260 0x7ffb78109670 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.899+0000 7ffb7f146640 1 --2- 172.21.15.16:0/3310800104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb78103920 0x7ffb78103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.899+0000 7ffb7f146640 1 -- 172.21.15.16:0/3310800104 >> 172.21.15.16:0/3310800104 conn(0x7ffb780ff3b0 msgr2=0x7ffb78101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.899+0000 7ffb7f146640 1 -- 172.21.15.16:0/3310800104 shutdown_connections 2024-08-28T19:36:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.899+0000 7ffb7f146640 1 -- 172.21.15.16:0/3310800104 wait complete. 2024-08-28T19:36:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.900+0000 7ffb7f146640 1 Processor -- start 2024-08-28T19:36:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.900+0000 7ffb7f146640 1 -- start start 2024-08-28T19:36:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.900+0000 7ffb7f146640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb78103920 0x7ffb7819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.900+0000 7ffb7f146640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb78109260 0x7ffb7819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.900+0000 7ffb7f146640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb78109bb0 0x7ffb7819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.900+0000 7ffb7f146640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffb78113870 con 0x7ffb78109260 2024-08-28T19:36:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.900+0000 7ffb7f146640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffb781136f0 con 0x7ffb78103920 2024-08-28T19:36:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.900+0000 7ffb7f146640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ffb781139f0 con 0x7ffb78109bb0 2024-08-28T19:36:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.900+0000 7ffb77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb78109260 0x7ffb7819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.900+0000 7ffb77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb78109260 0x7ffb7819a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59862/0 (socket says 172.21.15.16:59862) 2024-08-28T19:36:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.900+0000 7ffb77fff640 1 -- 172.21.15.16:0/513260727 learned_addr learned my addr 172.21.15.16:0/513260727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb78103920 0x7ffb7819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb7d6e3640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb78109bb0 0x7ffb7819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb77fff640 1 -- 172.21.15.16:0/513260727 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb78109bb0 msgr2=0x7ffb7819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb77fff640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb78109bb0 0x7ffb7819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb77fff640 1 -- 172.21.15.16:0/513260727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb78103920 msgr2=0x7ffb7819a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb77fff640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb78103920 0x7ffb7819a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb77fff640 1 -- 172.21.15.16:0/513260727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb781a0310 con 0x7ffb78109260 2024-08-28T19:36:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb78103920 0x7ffb7819a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb77fff640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb78109260 0x7ffb7819a940 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ffb64006010 tx=0x7ffb64004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb75ffb640 1 -- 172.21.15.16:0/513260727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6403f030 con 0x7ffb78109260 2024-08-28T19:36:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb75ffb640 1 -- 172.21.15.16:0/513260727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb64003aa0 con 0x7ffb78109260 2024-08-28T19:36:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb7f146640 1 -- 172.21.15.16:0/513260727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb781a05a0 con 0x7ffb78109260 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb75ffb640 1 -- 172.21.15.16:0/513260727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb64035070 con 0x7ffb78109260 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.901+0000 7ffb7f146640 1 -- 172.21.15.16:0/513260727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb781a6d60 con 0x7ffb78109260 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.904+0000 7ffb856bb740 1 Processor -- start 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.904+0000 7ffb856bb740 1 -- start start 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.904+0000 7ffb75ffb640 1 -- 172.21.15.16:0/513260727 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ffb64042030 con 0x7ffb78109260 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.904+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 0x55d4aeb212c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae7531b0 0x55d4aeb21800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb21d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb856bb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d4aea98630 con 0x55d4ae75b390 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb856bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d4aea87bc0 con 0x55d4ae77c070 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb856bb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d4aea8bb70 con 0x55d4ae7531b0 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb21d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb7d6e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 0x55d4aeb212c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb7d6e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 0x55d4aeb212c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60472/0 (socket says 172.21.15.16:60472) 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb7d6e3640 1 -- 172.21.15.16:0/3929071963 learned_addr learned my addr 172.21.15.16:0/3929071963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/3929071963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae7531b0 0x55d4aeb21800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb7d6e3640 1 -- 172.21.15.16:0/3929071963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae7531b0 msgr2=0x55d4aeb21800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb7d6e3640 1 --2- 172.21.15.16:0/3929071963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae7531b0 0x55d4aeb21800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb7d6e3640 1 -- 172.21.15.16:0/3929071963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 msgr2=0x55d4aeb21d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb7d6e3640 1 --2- 172.21.15.16:0/3929071963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb21d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb7d6e3640 1 -- 172.21.15.16:0/3929071963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4ae7e7650 con 0x55d4ae77c070 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/3929071963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae7531b0 0x55d4aeb21800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb7d6e3640 1 --2- 172.21.15.16:0/3929071963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 0x55d4aeb212c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009f20 tx=0x7ffb7000f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62d9e98a1582992 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.905+0000 7ffb75ffb640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffb680b4ca0 0x7ffb680b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.906+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffb680b4ca0 0x7ffb680b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.906+0000 7ffb62ffd640 1 -- 172.21.15.16:0/3929071963 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb70015070 con 0x55d4ae77c070 2024-08-28T19:36:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.906+0000 7ffb62ffd640 1 -- 172.21.15.16:0/3929071963 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb7001ab10 con 0x55d4ae77c070 2024-08-28T19:36:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.906+0000 7ffb62ffd640 1 -- 172.21.15.16:0/3929071963 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb70011c20 con 0x55d4ae77c070 2024-08-28T19:36:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.906+0000 7ffb75ffb640 1 -- 172.21.15.16:0/513260727 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7ffb64105ed0 con 0x7ffb78109260 2024-08-28T19:36:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.907+0000 7ffb856bb740 1 -- 172.21.15.16:0/3929071963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 msgr2=0x55d4aeb212c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.907+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffb680b4ca0 0x7ffb680b71b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x55d4aeb1ff40 tx=0x7ffb6c005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.907+0000 7ffb856bb740 1 --2- 172.21.15.16:0/3929071963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 0x55d4aeb212c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009f20 tx=0x7ffb7000f7f0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.907+0000 7ffb856bb740 1 -- 172.21.15.16:0/3929071963 shutdown_connections 2024-08-28T19:36:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.907+0000 7ffb856bb740 1 --2- 172.21.15.16:0/3929071963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb21d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.907+0000 7ffb856bb740 1 --2- 172.21.15.16:0/3929071963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae7531b0 0x55d4aeb21800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.907+0000 7ffb856bb740 1 --2- 172.21.15.16:0/3929071963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 0x55d4aeb212c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.907+0000 7ffb856bb740 1 -- 172.21.15.16:0/3929071963 >> 172.21.15.16:0/3929071963 conn(0x55d4ae770460 msgr2=0x55d4aeb1f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.907+0000 7ffb856bb740 1 -- 172.21.15.16:0/3929071963 shutdown_connections 2024-08-28T19:36:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.907+0000 7ffb856bb740 1 -- 172.21.15.16:0/3929071963 wait complete. 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.908+0000 7ffb856bb740 1 -- start start 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.908+0000 7ffb856bb740 1 Processor -- start 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.908+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 0x55d4aeb30de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.908+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb31320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.908+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae77c070 0x55d4aeb3d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.908+0000 7ffb856bb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d4ae8ea700 con 0x55d4ae75b390 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.908+0000 7ffb856bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d4aea0e700 con 0x55d4ae7531b0 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.908+0000 7ffb856bb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d4aeb33da0 con 0x55d4ae77c070 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.908+0000 7ffb856bb740 10 client.0 _ll_register_callbacks cb 0x55d4aea3ddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.908+0000 7ffb77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb31320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.908+0000 7ffb77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb31320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59892/0 (socket says 172.21.15.16:59892) 2024-08-28T19:36:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb77fff640 1 -- 172.21.15.16:0/2568405132 learned_addr learned my addr 172.21.15.16:0/2568405132 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 0x55d4aeb30de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb7d6e3640 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae77c070 0x55d4aeb3d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb77fff640 1 -- 172.21.15.16:0/2568405132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae77c070 msgr2=0x55d4aeb3d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb77fff640 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae77c070 0x55d4aeb3d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb77fff640 1 -- 172.21.15.16:0/2568405132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 msgr2=0x55d4aeb30de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb77fff640 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 0x55d4aeb30de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb77fff640 1 -- 172.21.15.16:0/2568405132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4aea478b0 con 0x55d4ae75b390 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb7d6e3640 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae77c070 0x55d4aeb3d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb77fff640 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb31320 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7ffb641032b0 tx=0x7ffb6403b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69f7d1e4724435e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb57fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb57fff640 1 -- 172.21.15.16:0/2568405132 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb64049070 con 0x55d4ae75b390 2024-08-28T19:36:51.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb57fff640 1 -- 172.21.15.16:0/2568405132 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7ffb640cc450 con 0x55d4ae75b390 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb856bb740 1 -- 172.21.15.16:0/2568405132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb7001ab10 con 0x55d4ae75b390 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb57fff640 1 -- 172.21.15.16:0/2568405132 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7ffb640c80b0 con 0x55d4ae75b390 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.909+0000 7ffb856bb740 10 client.19332 fetch_fsmap learned FSMap version 82 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.910+0000 7ffb856bb740 1 -- 172.21.15.16:0/2568405132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55d4aeb3fb50 con 0x55d4ae75b390 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.910+0000 7ffb57fff640 1 -- 172.21.15.16:0/2568405132 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb7001ab10 con 0x55d4ae75b390 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.910+0000 7ffb57fff640 1 -- 172.21.15.16:0/2568405132 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7ffb64109bd0 con 0x55d4ae75b390 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.910+0000 7ffb856bb740 10 client.19332 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.910+0000 7ffb856bb740 10 client.19332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.910+0000 7ffb856bb740 20 client.19332 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.910+0000 7ffb856bb740 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d4ae73a9f0 0x55d4aeb430f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.910+0000 7ffb856bb740 4 client.19332 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.910+0000 7ffb856bb740 1 -- 172.21.15.16:0/2568405132 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4aea9a580 con 0x55d4ae73a9f0 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.910+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d4ae73a9f0 0x55d4aeb430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.911+0000 7ffb57fff640 10 client.19332 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.911+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d4ae73a9f0 0x55d4aeb430f0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=948ce52489fe4df8 server_cookie=6f41bd2acccf69cc in_seq=0 out_seq=0 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb57fff640 1 -- 172.21.15.16:0/2568405132 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d4aea9a580 con 0x55d4ae73a9f0 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb57fff640 10 client.19332 handle_command_reply: tid=0 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 1 client.19332 shutdown 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 1 -- 172.21.15.16:0/2568405132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 msgr2=0x55d4aeb31320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb31320 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7ffb641032b0 tx=0x7ffb6403b5d0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 1 -- 172.21.15.16:0/2568405132 shutdown_connections 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d4ae73a9f0 0x55d4aeb430f0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae77c070 0x55d4aeb3d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb31320 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 1 --2- 172.21.15.16:0/2568405132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 0x55d4aeb30de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 1 -- 172.21.15.16:0/2568405132 >> 172.21.15.16:0/2568405132 conn(0x55d4ae770460 msgr2=0x55d4aeb22cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb57fff640 0 client.19332 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 1 -- 172.21.15.16:0/2568405132 shutdown_connections 2024-08-28T19:36:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 1 -- 172.21.15.16:0/2568405132 wait complete. 2024-08-28T19:36:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.912+0000 7ffb856bb740 20 client.19332 trim_cache size 0 max 16384 2024-08-28T19:36:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb856bb740 1 Processor -- start 2024-08-28T19:36:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb856bb740 1 -- start start 2024-08-28T19:36:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae7531b0 0x55d4aeb1f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb30d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 0x55d4aeb3d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb856bb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffb781aeb60 con 0x55d4ae75b390 2024-08-28T19:36:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb856bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffb781b08e0 con 0x55d4ae77c070 2024-08-28T19:36:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb856bb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ffb781af260 con 0x55d4ae7531b0 2024-08-28T19:36:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb30d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb30d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59898/0 (socket says 172.21.15.16:59898) 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb77fff640 1 -- 172.21.15.16:0/936279008 learned_addr learned my addr 172.21.15.16:0/936279008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/936279008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae7531b0 0x55d4aeb1f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.930+0000 7ffb7d6e3640 1 --2- 172.21.15.16:0/936279008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 0x55d4aeb3d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.931+0000 7ffb77fff640 1 -- 172.21.15.16:0/936279008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae7531b0 msgr2=0x55d4aeb1f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.931+0000 7ffb77fff640 1 --2- 172.21.15.16:0/936279008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae7531b0 0x55d4aeb1f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.931+0000 7ffb77fff640 1 -- 172.21.15.16:0/936279008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 msgr2=0x55d4aeb3d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.931+0000 7ffb77fff640 1 --2- 172.21.15.16:0/936279008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 0x55d4aeb3d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.931+0000 7ffb77fff640 1 -- 172.21.15.16:0/936279008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4aeb2fd40 con 0x55d4ae75b390 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.931+0000 7ffb7d6e3640 1 --2- 172.21.15.16:0/936279008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 0x55d4aeb3d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.931+0000 7ffb77fff640 1 --2- 172.21.15.16:0/936279008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb30d50 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7ffb6403b580 tx=0x7ffb64106260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f5bc1fcc4f3914 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:51.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.931+0000 7ffb627fc640 1 -- 172.21.15.16:0/936279008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb64049070 con 0x55d4ae75b390 2024-08-28T19:36:51.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.931+0000 7ffb627fc640 1 -- 172.21.15.16:0/936279008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb64109710 con 0x55d4ae75b390 2024-08-28T19:36:51.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.931+0000 7ffb627fc640 1 -- 172.21.15.16:0/936279008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb640cb960 con 0x55d4ae75b390 2024-08-28T19:36:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.932+0000 7ffb856bb740 1 -- 172.21.15.16:0/936279008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 msgr2=0x55d4aeb30d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.932+0000 7ffb856bb740 1 --2- 172.21.15.16:0/936279008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb30d50 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7ffb6403b580 tx=0x7ffb64106260 comp rx=0 tx=0).stop 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.932+0000 7ffb856bb740 1 -- 172.21.15.16:0/936279008 shutdown_connections 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.932+0000 7ffb856bb740 1 --2- 172.21.15.16:0/936279008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae77c070 0x55d4aeb3d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.932+0000 7ffb856bb740 1 --2- 172.21.15.16:0/936279008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb30d50 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.932+0000 7ffb856bb740 1 --2- 172.21.15.16:0/936279008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae7531b0 0x55d4aeb1f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.932+0000 7ffb856bb740 1 -- 172.21.15.16:0/936279008 >> 172.21.15.16:0/936279008 conn(0x55d4ae770460 msgr2=0x55d4aeb1ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.932+0000 7ffb856bb740 1 -- 172.21.15.16:0/936279008 shutdown_connections 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.932+0000 7ffb856bb740 1 -- 172.21.15.16:0/936279008 wait complete. 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.932+0000 7ffb856bb740 1 -- start start 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb856bb740 1 Processor -- start 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 0x55d4aeb39540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb35a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb856bb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae77c070 0x55d4aeb35f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb856bb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffb781b0150 con 0x55d4ae75b390 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb856bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffb781af9c0 con 0x55d4ae7531b0 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb856bb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ffb781acb60 con 0x55d4ae77c070 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb856bb740 10 client.0 _ll_register_callbacks cb 0x7ffb64005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb7cee2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 0x55d4aeb39540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb35a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb7cee2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 0x55d4aeb39540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:60488/0 (socket says 172.21.15.16:60488) 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb7cee2640 1 -- 172.21.15.16:0/365515160 learned_addr learned my addr 172.21.15.16:0/365515160 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.933+0000 7ffb77fff640 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb35a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59906/0 (socket says 172.21.15.16:59906) 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb7cee2640 1 -- 172.21.15.16:0/365515160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae77c070 msgr2=0x55d4aeb35f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae77c070 0x55d4aeb35f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb7cee2640 1 -- 172.21.15.16:0/365515160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 msgr2=0x55d4aeb35a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb35a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb7cee2640 1 -- 172.21.15.16:0/365515160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4aeb36480 con 0x55d4ae7531b0 2024-08-28T19:36:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb7cee2640 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 0x55d4aeb39540 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c009680 tx=0x7ffb6c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8d335bd4190f02d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb57fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:36:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb57fff640 1 -- 172.21.15.16:0/365515160 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c020830 con 0x55d4ae7531b0 2024-08-28T19:36:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb856bb740 1 -- 172.21.15.16:0/365515160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb64109710 con 0x55d4ae7531b0 2024-08-28T19:36:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb57fff640 1 -- 172.21.15.16:0/365515160 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7ffb6c0209d0 con 0x55d4ae7531b0 2024-08-28T19:36:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb57fff640 1 -- 172.21.15.16:0/365515160 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7ffb6c020bd0 con 0x55d4ae7531b0 2024-08-28T19:36:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb856bb740 10 client.28757 fetch_fsmap learned FSMap version 82 2024-08-28T19:36:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.934+0000 7ffb856bb740 1 -- 172.21.15.16:0/365515160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55d4aebae6f0 con 0x55d4ae7531b0 2024-08-28T19:36:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.935+0000 7ffb57fff640 1 -- 172.21.15.16:0/365515160 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb64109710 con 0x55d4ae7531b0 2024-08-28T19:36:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.935+0000 7ffb57fff640 1 -- 172.21.15.16:0/365515160 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7ffb6c031cb0 con 0x55d4ae7531b0 2024-08-28T19:36:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.935+0000 7ffb856bb740 10 client.28757 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.935+0000 7ffb856bb740 10 client.28757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.935+0000 7ffb856bb740 20 client.28757 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.935+0000 7ffb856bb740 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d4ae70fda0 0x55d4aebb0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.935+0000 7ffb856bb740 4 client.28757 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.935+0000 7ffb856bb740 1 -- 172.21.15.16:0/365515160 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d4aea96020 con 0x55d4ae70fda0 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.935+0000 7ffb77fff640 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d4ae70fda0 0x55d4aebb0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.935+0000 7ffb77fff640 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d4ae70fda0 0x55d4aebb0fa0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74350fdd3e76adf9 server_cookie=d96b61a6cc061ec8 in_seq=0 out_seq=0 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.935+0000 7ffb57fff640 10 client.28757 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb57fff640 1 -- 172.21.15.16:0/365515160 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+6054 (crc 0 0 0) 0x55d4aea96020 con 0x55d4ae70fda0 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb57fff640 10 client.28757 handle_command_reply: tid=0 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 1 client.28757 shutdown 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 1 -- 172.21.15.16:0/365515160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 msgr2=0x55d4aeb39540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:51.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 0x55d4aeb39540 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c009680 tx=0x7ffb6c016000 comp rx=0 tx=0).stop 2024-08-28T19:36:51.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 1 -- 172.21.15.16:0/365515160 shutdown_connections 2024-08-28T19:36:51.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d4ae70fda0 0x55d4aebb0fa0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d4ae77c070 0x55d4aeb35f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d4ae75b390 0x55d4aeb35a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb57fff640 0 client.28757 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:51.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 1 --2- 172.21.15.16:0/365515160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4ae7531b0 0x55d4aeb39540 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 1 -- 172.21.15.16:0/365515160 >> 172.21.15.16:0/365515160 conn(0x55d4ae770460 msgr2=0x55d4aeb222f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:51.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 1 -- 172.21.15.16:0/365515160 shutdown_connections 2024-08-28T19:36:51.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 1 -- 172.21.15.16:0/365515160 wait complete. 2024-08-28T19:36:51.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.936+0000 7ffb856bb740 20 client.28757 trim_cache size 0 max 16384 2024-08-28T19:36:51.943 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:36:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:36:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:36:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:36:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:36:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:36:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:36:51.955 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:36:51.956 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:36:51.956 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:36:51.956 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:36:51.956 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:36:51.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:51.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:51.956 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:36:51.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 -- 172.21.15.16:0/513260727 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffb680b4ca0 msgr2=0x7ffb680b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:51.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffb680b4ca0 0x7ffb680b71b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x55d4aeb1ff40 tx=0x7ffb6c005f90 comp rx=0 tx=0).stop 2024-08-28T19:36:51.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 -- 172.21.15.16:0/513260727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb78109260 msgr2=0x7ffb7819a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:51.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb78109260 0x7ffb7819a940 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ffb64006010 tx=0x7ffb64004060 comp rx=0 tx=0).stop 2024-08-28T19:36:51.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 -- 172.21.15.16:0/513260727 shutdown_connections 2024-08-28T19:36:51.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb78109bb0 0x7ffb7819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb78109260 0x7ffb7819a940 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ffb680b4ca0 0x7ffb680b71b0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 --2- 172.21.15.16:0/513260727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb78103920 0x7ffb7819a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:51.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 -- 172.21.15.16:0/513260727 >> 172.21.15.16:0/513260727 conn(0x7ffb780ff3b0 msgr2=0x7ffb781017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:51.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 -- 172.21.15.16:0/513260727 shutdown_connections 2024-08-28T19:36:51.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:51.937+0000 7ffb7f146640 1 -- 172.21.15.16:0/513260727 wait complete. 2024-08-28T19:36:52.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 28 sec latency 29292.749 ms 2024-08-28T19:36:52.667 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:36:52.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:36:52.793 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:52 smithi016 ceph-mon[18556]: pgmap v832: 129 pgs: 129 active+clean; 932 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 1.4 MiB/s wr, 50 op/s 2024-08-28T19:36:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:36:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:36:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:36:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:36:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:52 smithi028 ceph-mon[21769]: pgmap v832: 129 pgs: 129 active+clean; 932 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 1.4 MiB/s wr, 50 op/s 2024-08-28T19:36:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:36:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:36:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:36:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:36:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:52 smithi115 ceph-mon[22184]: pgmap v832: 129 pgs: 129 active+clean; 932 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 1.4 MiB/s wr, 50 op/s 2024-08-28T19:36:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:36:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:36:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:36:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:36:53.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38401 0.00 MB/sec execute 29 sec latency 30292.803 ms 2024-08-28T19:36:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.626+0000 7faa3fb2f640 1 -- 172.21.15.16:0/2748281292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa38105050 msgr2=0x7faa3810d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.626+0000 7faa3fb2f640 1 --2- 172.21.15.16:0/2748281292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa38105050 0x7faa3810d690 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7faa28009900 tx=0x7faa2802a410 comp rx=0 tx=0).stop 2024-08-28T19:36:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.626+0000 7faa3fb2f640 1 -- 172.21.15.16:0/2748281292 shutdown_connections 2024-08-28T19:36:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.626+0000 7faa3fb2f640 1 --2- 172.21.15.16:0/2748281292 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa3810dbd0 0x7faa381100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.626+0000 7faa3fb2f640 1 --2- 172.21.15.16:0/2748281292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa38105050 0x7faa3810d690 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.627+0000 7faa3fb2f640 1 --2- 172.21.15.16:0/2748281292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa38104620 0x7faa38104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.627+0000 7faa3fb2f640 1 -- 172.21.15.16:0/2748281292 >> 172.21.15.16:0/2748281292 conn(0x7faa380794e0 msgr2=0x7faa380798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.627+0000 7faa3fb2f640 1 -- 172.21.15.16:0/2748281292 shutdown_connections 2024-08-28T19:36:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.627+0000 7faa3fb2f640 1 -- 172.21.15.16:0/2748281292 wait complete. 2024-08-28T19:36:53.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.627+0000 7faa3fb2f640 1 Processor -- start 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.627+0000 7faa3fb2f640 1 -- start start 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3fb2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa38104620 0x7faa3819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3fb2f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa38105050 0x7faa3819a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3fb2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3810dbd0 0x7faa3819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3fb2f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa38113800 con 0x7faa38104620 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3fb2f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa38113680 con 0x7faa3810dbd0 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3fb2f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faa38113980 con 0x7faa38105050 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3d8cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa38104620 0x7faa3819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3d8cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa38104620 0x7faa3819a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59926/0 (socket says 172.21.15.16:59926) 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3d8cb640 1 -- 172.21.15.16:0/3463958889 learned_addr learned my addr 172.21.15.16:0/3463958889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa38105050 0x7faa3819a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3e0cc640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3810dbd0 0x7faa3819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3d8cb640 1 -- 172.21.15.16:0/3463958889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa38105050 msgr2=0x7faa3819a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:53.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3d8cb640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa38105050 0x7faa3819a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3d8cb640 1 -- 172.21.15.16:0/3463958889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3810dbd0 msgr2=0x7faa3819faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:53.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3d8cb640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3810dbd0 0x7faa3819faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3d8cb640 1 -- 172.21.15.16:0/3463958889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa381a0220 con 0x7faa38104620 2024-08-28T19:36:53.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa38105050 0x7faa3819a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:53.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3e0cc640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3810dbd0 0x7faa3819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa3d8cb640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa38104620 0x7faa3819a2b0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7faa3400e9f0 tx=0x7faa3400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa2effd640 1 -- 172.21.15.16:0/3463958889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3400cd80 con 0x7faa38104620 2024-08-28T19:36:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa2effd640 1 -- 172.21.15.16:0/3463958889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa340108e0 con 0x7faa38104620 2024-08-28T19:36:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.628+0000 7faa2effd640 1 -- 172.21.15.16:0/3463958889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34018980 con 0x7faa38104620 2024-08-28T19:36:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.629+0000 7faa3fb2f640 1 -- 172.21.15.16:0/3463958889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa381a0510 con 0x7faa38104620 2024-08-28T19:36:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.629+0000 7faa3fb2f640 1 -- 172.21.15.16:0/3463958889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa381a6c70 con 0x7faa38104620 2024-08-28T19:36:53.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa460a4740 1 Processor -- start 2024-08-28T19:36:53.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa460a4740 1 -- start start 2024-08-28T19:36:53.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa460a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b43e520 0x55ae7b7e3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa460a4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b415660 0x55ae7b7e41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa460a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7e4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa460a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ae7b75aa70 con 0x55ae7b41d840 2024-08-28T19:36:53.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa460a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae7b74a000 con 0x55ae7b43e520 2024-08-28T19:36:53.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa2effd640 1 -- 172.21.15.16:0/3463958889 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7faa34010430 con 0x7faa38104620 2024-08-28T19:36:53.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa460a4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ae7b74dfb0 con 0x55ae7b415660 2024-08-28T19:36:53.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa3d0ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7e4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa3d0ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7e4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59942/0 (socket says 172.21.15.16:59942) 2024-08-28T19:36:53.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.632+0000 7faa3d0ca640 1 -- 172.21.15.16:0/461370289 learned_addr learned my addr 172.21.15.16:0/461370289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa3d8cb640 1 --2- 172.21.15.16:0/461370289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b415660 0x55ae7b7e41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa3e0cc640 1 --2- 172.21.15.16:0/461370289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b43e520 0x55ae7b7e3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa3d0ca640 1 -- 172.21.15.16:0/461370289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b415660 msgr2=0x55ae7b7e41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/461370289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b415660 0x55ae7b7e41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa3d0ca640 1 -- 172.21.15.16:0/461370289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b43e520 msgr2=0x55ae7b7e3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/461370289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b43e520 0x55ae7b7e3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa3d0ca640 1 -- 172.21.15.16:0/461370289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae7b4a9b00 con 0x55ae7b41d840 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa3e0cc640 1 --2- 172.21.15.16:0/461370289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b43e520 0x55ae7b7e3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/461370289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7e4700 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7faa28000c00 tx=0x7faa28032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d06cb709e27292 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa2effd640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa240b4ca0 0x7faa240b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa23fff640 1 -- 172.21.15.16:0/461370289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2803f070 con 0x55ae7b41d840 2024-08-28T19:36:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa3e0cc640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa240b4ca0 0x7faa240b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa2effd640 1 -- 172.21.15.16:0/3463958889 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7faa340e5d80 con 0x7faa38104620 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa23fff640 1 -- 172.21.15.16:0/461370289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa280316a0 con 0x55ae7b41d840 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.633+0000 7faa23fff640 1 -- 172.21.15.16:0/461370289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa28035070 con 0x55ae7b41d840 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.634+0000 7faa460a4740 1 -- 172.21.15.16:0/461370289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 msgr2=0x55ae7b7e4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.634+0000 7faa3e0cc640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa240b4ca0 0x7faa240b71b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7faa30005fd0 tx=0x7faa30009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.634+0000 7faa460a4740 1 --2- 172.21.15.16:0/461370289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7e4700 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7faa28000c00 tx=0x7faa28032c30 comp rx=0 tx=0).stop 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.634+0000 7faa460a4740 1 -- 172.21.15.16:0/461370289 shutdown_connections 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.634+0000 7faa460a4740 1 --2- 172.21.15.16:0/461370289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7e4700 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.634+0000 7faa460a4740 1 --2- 172.21.15.16:0/461370289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b415660 0x55ae7b7e41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.634+0000 7faa460a4740 1 --2- 172.21.15.16:0/461370289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b43e520 0x55ae7b7e3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.634+0000 7faa460a4740 1 -- 172.21.15.16:0/461370289 >> 172.21.15.16:0/461370289 conn(0x55ae7b432910 msgr2=0x55ae7b7e1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:53.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.635+0000 7faa460a4740 1 -- 172.21.15.16:0/461370289 shutdown_connections 2024-08-28T19:36:53.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.635+0000 7faa460a4740 1 -- 172.21.15.16:0/461370289 wait complete. 2024-08-28T19:36:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.635+0000 7faa460a4740 1 -- start start 2024-08-28T19:36:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.635+0000 7faa460a4740 1 Processor -- start 2024-08-28T19:36:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.635+0000 7faa460a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b415660 0x55ae7b7ef860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.635+0000 7faa460a4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 0x55ae7b7efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.635+0000 7faa460a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b43e520 0x55ae7b7fb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.635+0000 7faa460a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ae7b7f2400 con 0x55ae7b415660 2024-08-28T19:36:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.635+0000 7faa460a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae7b70b520 con 0x55ae7b43e520 2024-08-28T19:36:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.635+0000 7faa460a4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ae7b7f2580 con 0x55ae7b41d840 2024-08-28T19:36:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa460a4740 10 client.0 _ll_register_callbacks cb 0x55ae7b743c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3d8cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b415660 0x55ae7b7ef860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3d8cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b415660 0x55ae7b7ef860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59944/0 (socket says 172.21.15.16:59944) 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3d8cb640 1 -- 172.21.15.16:0/3942442411 learned_addr learned my addr 172.21.15.16:0/3942442411 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3e0cc640 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b43e520 0x55ae7b7fb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 0x55ae7b7efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3d8cb640 1 -- 172.21.15.16:0/3942442411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 msgr2=0x55ae7b7efdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3d8cb640 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 0x55ae7b7efdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3d8cb640 1 -- 172.21.15.16:0/3942442411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b43e520 msgr2=0x55ae7b7fb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3d8cb640 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b43e520 0x55ae7b7fb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3d8cb640 1 -- 172.21.15.16:0/3942442411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae7b520960 con 0x55ae7b415660 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 0x55ae7b7efdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa3d8cb640 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b415660 0x55ae7b7ef860 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7faa34000c00 tx=0x7faa34011fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f8344fc8846543 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa217fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa217fa640 1 -- 172.21.15.16:0/3942442411 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa340ab810 con 0x55ae7b415660 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa217fa640 1 -- 172.21.15.16:0/3942442411 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7faa340ab300 con 0x55ae7b415660 2024-08-28T19:36:53.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.636+0000 7faa460a4740 1 -- 172.21.15.16:0/3942442411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ae7b6e99a0 con 0x55ae7b415660 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa217fa640 1 -- 172.21.15.16:0/3942442411 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7faa34010720 con 0x55ae7b415660 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa460a4740 10 client.19350 fetch_fsmap learned FSMap version 82 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa460a4740 1 -- 172.21.15.16:0/3942442411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55ae7b7fe5c0 con 0x55ae7b415660 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa217fa640 1 -- 172.21.15.16:0/3942442411 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ae7b6e99a0 con 0x55ae7b415660 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa217fa640 1 -- 172.21.15.16:0/3942442411 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7faa340e9b10 con 0x55ae7b415660 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa460a4740 10 client.19350 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa460a4740 10 client.19350 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa460a4740 20 client.19350 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa460a4740 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae7b3fcea0 0x55ae7b801a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa460a4740 4 client.19350 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa460a4740 1 -- 172.21.15.16:0/3942442411 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae7b75c9c0 con 0x55ae7b3fcea0 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.637+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae7b3fcea0 0x55ae7b801a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.638+0000 7faa217fa640 10 client.19350 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.638+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae7b3fcea0 0x55ae7b801a90 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9351c8e7ef752db8 server_cookie=27e1e12b9d850548 in_seq=0 out_seq=0 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa217fa640 1 -- 172.21.15.16:0/3942442411 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ae7b75c9c0 con 0x55ae7b3fcea0 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa217fa640 10 client.19350 handle_command_reply: tid=0 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 1 client.19350 shutdown 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 1 -- 172.21.15.16:0/3942442411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b415660 msgr2=0x55ae7b7ef860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b415660 0x55ae7b7ef860 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7faa34000c00 tx=0x7faa34011fe0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 1 -- 172.21.15.16:0/3942442411 shutdown_connections 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b43e520 0x55ae7b7fb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae7b3fcea0 0x55ae7b801a90 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 0x55ae7b7efdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 1 --2- 172.21.15.16:0/3942442411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b415660 0x55ae7b7ef860 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa217fa640 0 client.19350 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:53.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 1 -- 172.21.15.16:0/3942442411 >> 172.21.15.16:0/3942442411 conn(0x55ae7b432910 msgr2=0x55ae7b7e6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:53.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 1 -- 172.21.15.16:0/3942442411 shutdown_connections 2024-08-28T19:36:53.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 1 -- 172.21.15.16:0/3942442411 wait complete. 2024-08-28T19:36:53.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.639+0000 7faa460a4740 20 client.19350 trim_cache size 0 max 16384 2024-08-28T19:36:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.656+0000 7faa460a4740 1 Processor -- start 2024-08-28T19:36:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.656+0000 7faa460a4740 1 -- start start 2024-08-28T19:36:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa460a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b415660 0x55ae7b7fb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa460a4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 0x55ae7b7f1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa460a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b43e520 0x55ae7b7e53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa460a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa381b0db0 con 0x55ae7b43e520 2024-08-28T19:36:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa460a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa381b2b30 con 0x55ae7b415660 2024-08-28T19:36:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa460a4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faa381b14b0 con 0x55ae7b41d840 2024-08-28T19:36:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3e0cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b43e520 0x55ae7b7e53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3e0cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b43e520 0x55ae7b7e53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59960/0 (socket says 172.21.15.16:59960) 2024-08-28T19:36:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3e0cc640 1 -- 172.21.15.16:0/234791813 learned_addr learned my addr 172.21.15.16:0/234791813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3d8cb640 1 --2- 172.21.15.16:0/234791813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b415660 0x55ae7b7fb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/234791813 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 0x55ae7b7f1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3e0cc640 1 -- 172.21.15.16:0/234791813 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 msgr2=0x55ae7b7f1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3e0cc640 1 --2- 172.21.15.16:0/234791813 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 0x55ae7b7f1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3e0cc640 1 -- 172.21.15.16:0/234791813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b415660 msgr2=0x55ae7b7fb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3e0cc640 1 --2- 172.21.15.16:0/234791813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b415660 0x55ae7b7fb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3e0cc640 1 -- 172.21.15.16:0/234791813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae7b7edb60 con 0x55ae7b43e520 2024-08-28T19:36:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/234791813 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 0x55ae7b7f1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa3e0cc640 1 --2- 172.21.15.16:0/234791813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b43e520 0x55ae7b7e53c0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7faa300094b0 tx=0x7faa30015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8945a830d335c7de server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.657+0000 7faa237fe640 1 -- 172.21.15.16:0/234791813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa30010e50 con 0x55ae7b43e520 2024-08-28T19:36:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.658+0000 7faa237fe640 1 -- 172.21.15.16:0/234791813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa30003e20 con 0x55ae7b43e520 2024-08-28T19:36:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.658+0000 7faa237fe640 1 -- 172.21.15.16:0/234791813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3001f680 con 0x55ae7b43e520 2024-08-28T19:36:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.658+0000 7faa460a4740 1 -- 172.21.15.16:0/234791813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b43e520 msgr2=0x55ae7b7e53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.658+0000 7faa460a4740 1 --2- 172.21.15.16:0/234791813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b43e520 0x55ae7b7e53c0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7faa300094b0 tx=0x7faa30015000 comp rx=0 tx=0).stop 2024-08-28T19:36:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.658+0000 7faa460a4740 1 -- 172.21.15.16:0/234791813 shutdown_connections 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.658+0000 7faa460a4740 1 --2- 172.21.15.16:0/234791813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b43e520 0x55ae7b7e53c0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.658+0000 7faa460a4740 1 --2- 172.21.15.16:0/234791813 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b41d840 0x55ae7b7f1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.658+0000 7faa460a4740 1 --2- 172.21.15.16:0/234791813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b415660 0x55ae7b7fb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.658+0000 7faa460a4740 1 -- 172.21.15.16:0/234791813 >> 172.21.15.16:0/234791813 conn(0x55ae7b432910 msgr2=0x55ae7b7e4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.658+0000 7faa460a4740 1 -- 172.21.15.16:0/234791813 shutdown_connections 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.658+0000 7faa460a4740 1 -- 172.21.15.16:0/234791813 wait complete. 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.659+0000 7faa460a4740 1 -- start start 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.659+0000 7faa460a4740 1 Processor -- start 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.659+0000 7faa460a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b415660 0x55ae7b7f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.659+0000 7faa460a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7f41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.659+0000 7faa460a4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b43e520 0x55ae7b7f4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa460a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa381b23a0 con 0x55ae7b41d840 2024-08-28T19:36:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa460a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa381b1c10 con 0x55ae7b415660 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa460a4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faa381aedb0 con 0x55ae7b43e520 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa460a4740 10 client.0 _ll_register_callbacks cb 0x7faa340a7f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa3d0ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7f41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa3d0ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7f41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59962/0 (socket says 172.21.15.16:59962) 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa3d0ca640 1 -- 172.21.15.16:0/4197402333 learned_addr learned my addr 172.21.15.16:0/4197402333 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa3d8cb640 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b415660 0x55ae7b7f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa3e0cc640 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b43e520 0x55ae7b7f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa3d0ca640 1 -- 172.21.15.16:0/4197402333 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b43e520 msgr2=0x55ae7b7f4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b43e520 0x55ae7b7f4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa3d0ca640 1 -- 172.21.15.16:0/4197402333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b415660 msgr2=0x55ae7b7f7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b415660 0x55ae7b7f7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa3d0ca640 1 -- 172.21.15.16:0/4197402333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae7b7f4c40 con 0x55ae7b41d840 2024-08-28T19:36:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa3d0ca640 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7f41c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7faa28005dc0 tx=0x7faa28005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e1aff4fa4991b8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa217fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa217fa640 1 -- 172.21.15.16:0/4197402333 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2803f040 con 0x55ae7b41d840 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa217fa640 1 -- 172.21.15.16:0/4197402333 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7faa280316a0 con 0x55ae7b41d840 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.660+0000 7faa460a4740 1 -- 172.21.15.16:0/4197402333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa30003e20 con 0x55ae7b41d840 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa217fa640 1 -- 172.21.15.16:0/4197402333 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7faa2805c050 con 0x55ae7b41d840 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa460a4740 10 client.19356 fetch_fsmap learned FSMap version 82 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa460a4740 1 -- 172.21.15.16:0/4197402333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55ae7b86d0f0 con 0x55ae7b41d840 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa217fa640 1 -- 172.21.15.16:0/4197402333 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa30003e20 con 0x55ae7b41d840 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa217fa640 1 -- 172.21.15.16:0/4197402333 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7faa28003850 con 0x55ae7b41d840 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa460a4740 10 client.19356 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa460a4740 10 client.19356 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa460a4740 20 client.19356 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa460a4740 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae7b3d2250 0x55ae7b86f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa460a4740 4 client.19356 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e447cd33", "release": true, "await": true}] 2024-08-28T19:36:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa460a4740 1 -- 172.21.15.16:0/4197402333 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e447cd33", "release": true, "await": true}) -- 0x55ae7b758460 con 0x55ae7b3d2250 2024-08-28T19:36:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.661+0000 7faa3d8cb640 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae7b3d2250 0x55ae7b86f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.662+0000 7faa3d8cb640 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae7b3d2250 0x55ae7b86f5a0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc8aac44163ff9f4 server_cookie=16b3e8649fecf37d in_seq=0 out_seq=0 2024-08-28T19:36:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.662+0000 7faa217fa640 10 client.19356 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa217fa640 1 -- 172.21.15.16:0/4197402333 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ae7b758460 con 0x55ae7b3d2250 2024-08-28T19:36:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa217fa640 10 client.19356 handle_command_reply: tid=0 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 1 client.19356 shutdown 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 1 -- 172.21.15.16:0/4197402333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 msgr2=0x55ae7b7f41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7f41c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7faa28005dc0 tx=0x7faa28005f10 comp rx=0 tx=0).stop 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 1 -- 172.21.15.16:0/4197402333 shutdown_connections 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ae7b3d2250 0x55ae7b86f5a0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae7b43e520 0x55ae7b7f4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae7b41d840 0x55ae7b7f41c0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa217fa640 0 client.19356 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 1 --2- 172.21.15.16:0/4197402333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7b415660 0x55ae7b7f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 1 -- 172.21.15.16:0/4197402333 >> 172.21.15.16:0/4197402333 conn(0x55ae7b432910 msgr2=0x55ae7b7e5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 1 -- 172.21.15.16:0/4197402333 shutdown_connections 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 1 -- 172.21.15.16:0/4197402333 wait complete. 2024-08-28T19:36:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.693+0000 7faa460a4740 20 client.19356 trim_cache size 0 max 16384 2024-08-28T19:36:53.696 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:36:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 82, 2024-08-28T19:36:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:36:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 52, 2024-08-28T19:36:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:36:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "e447cd33": { 2024-08-28T19:36:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 52, 2024-08-28T19:36:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:36:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:36:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:36:53.698 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:36:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 -- 172.21.15.16:0/3463958889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa240b4ca0 msgr2=0x7faa240b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa240b4ca0 0x7faa240b71b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7faa30005fd0 tx=0x7faa30009500 comp rx=0 tx=0).stop 2024-08-28T19:36:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 -- 172.21.15.16:0/3463958889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa38104620 msgr2=0x7faa3819a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa38104620 0x7faa3819a2b0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7faa3400e9f0 tx=0x7faa3400eeb0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 -- 172.21.15.16:0/3463958889 shutdown_connections 2024-08-28T19:36:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa240b4ca0 0x7faa240b71b0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3810dbd0 0x7faa3819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa38105050 0x7faa3819a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 --2- 172.21.15.16:0/3463958889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa38104620 0x7faa3819a2b0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 -- 172.21.15.16:0/3463958889 >> 172.21.15.16:0/3463958889 conn(0x7faa380794e0 msgr2=0x7faa380ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 -- 172.21.15.16:0/3463958889 shutdown_connections 2024-08-28T19:36:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:53.694+0000 7faa3fb2f640 1 -- 172.21.15.16:0/3463958889 wait complete. 2024-08-28T19:36:54.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38994 1.21 MB/sec execute 30 sec latency 30511.954 ms 2024-08-28T19:36:54.484 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'e447cd33', total seconds elapsed: 34.9 2024-08-28T19:36:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:54.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:54 smithi016 ceph-mon[18556]: pgmap v833: 129 pgs: 129 active+clean; 935 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.8 MiB/s wr, 54 op/s 2024-08-28T19:36:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.058+0000 7ff296f11640 1 -- 172.21.15.16:0/4213354616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff290103920 msgr2=0x7ff290103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.058+0000 7ff296f11640 1 --2- 172.21.15.16:0/4213354616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff290103920 0x7ff290103d30 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff284009900 tx=0x7ff28402a3f0 comp rx=0 tx=0).stop 2024-08-28T19:36:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.058+0000 7ff296f11640 1 -- 172.21.15.16:0/4213354616 shutdown_connections 2024-08-28T19:36:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.058+0000 7ff296f11640 1 --2- 172.21.15.16:0/4213354616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff290109bb0 0x7ff2901103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.058+0000 7ff296f11640 1 --2- 172.21.15.16:0/4213354616 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff290109260 0x7ff290109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.058+0000 7ff296f11640 1 --2- 172.21.15.16:0/4213354616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff290103920 0x7ff290103d30 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.058+0000 7ff296f11640 1 -- 172.21.15.16:0/4213354616 >> 172.21.15.16:0/4213354616 conn(0x7ff2900ff3b0 msgr2=0x7ff290101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.058+0000 7ff296f11640 1 -- 172.21.15.16:0/4213354616 shutdown_connections 2024-08-28T19:36:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.058+0000 7ff296f11640 1 -- 172.21.15.16:0/4213354616 wait complete. 2024-08-28T19:36:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.059+0000 7ff296f11640 1 Processor -- start 2024-08-28T19:36:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.059+0000 7ff296f11640 1 -- start start 2024-08-28T19:36:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.059+0000 7ff296f11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff290103920 0x7ff29019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.059+0000 7ff296f11640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff290109260 0x7ff29019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.059+0000 7ff296f11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff290109bb0 0x7ff29019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff296f11640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff290113870 con 0x7ff290103920 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff296f11640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901136f0 con 0x7ff290109bb0 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff296f11640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901139f0 con 0x7ff290109260 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff294cad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff290103920 0x7ff29019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff294cad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff290103920 0x7ff29019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42742/0 (socket says 172.21.15.16:42742) 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff294cad640 1 -- 172.21.15.16:0/3888085015 learned_addr learned my addr 172.21.15.16:0/3888085015 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff2954ae640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff290109bb0 0x7ff29019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff28ffff640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff290109260 0x7ff29019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff2954ae640 1 -- 172.21.15.16:0/3888085015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff290109260 msgr2=0x7ff29019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff2954ae640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff290109260 0x7ff29019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff2954ae640 1 -- 172.21.15.16:0/3888085015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff290103920 msgr2=0x7ff29019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff2954ae640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff290103920 0x7ff29019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff2954ae640 1 -- 172.21.15.16:0/3888085015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2901a01d0 con 0x7ff290109bb0 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff294cad640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff290103920 0x7ff29019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff2954ae640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff290109bb0 0x7ff29019faa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff288009fd0 tx=0x7ff28800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.060+0000 7ff28dffb640 1 -- 172.21.15.16:0/3888085015 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28800ce20 con 0x7ff290109bb0 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.061+0000 7ff296f11640 1 -- 172.21.15.16:0/3888085015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2901a04c0 con 0x7ff290109bb0 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.061+0000 7ff28dffb640 1 -- 172.21.15.16:0/3888085015 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff288005020 con 0x7ff290109bb0 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.061+0000 7ff28dffb640 1 -- 172.21.15.16:0/3888085015 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288010690 con 0x7ff290109bb0 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.061+0000 7ff296f11640 1 -- 172.21.15.16:0/3888085015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2901a6c20 con 0x7ff290109bb0 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.062+0000 7ff296f11640 1 -- 172.21.15.16:0/3888085015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff290103d30 con 0x7ff290109bb0 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.063+0000 7ff28dffb640 1 -- 172.21.15.16:0/3888085015 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff2880054d0 con 0x7ff290109bb0 2024-08-28T19:36:55.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.064+0000 7ff28dffb640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2800b4ca0 0x7ff2800b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:55.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.064+0000 7ff28dffb640 1 -- 172.21.15.16:0/3888085015 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7ff2880e5fc0 con 0x7ff290109bb0 2024-08-28T19:36:55.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.064+0000 7ff294cad640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2800b4ca0 0x7ff2800b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:55.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.065+0000 7ff294cad640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2800b4ca0 0x7ff2800b71b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff284005ec0 tx=0x7ff284005a40 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:55.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.072+0000 7ff28dffb640 1 -- 172.21.15.16:0/3888085015 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7ff2880ac9e0 con 0x7ff290109bb0 2024-08-28T19:36:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:54 smithi028 ceph-mon[21769]: pgmap v833: 129 pgs: 129 active+clean; 935 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.8 MiB/s wr, 54 op/s 2024-08-28T19:36:55.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.364+0000 7ff296f11640 1 -- 172.21.15.16:0/3888085015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff29010b9e0 con 0x7ff290109bb0 2024-08-28T19:36:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.365+0000 7ff28dffb640 1 -- 172.21.15.16:0/3888085015 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+12002 (secure 0 0 0) 0x7ff2880ac2e0 con 0x7ff290109bb0 2024-08-28T19:36:55.366 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:36:55.366 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"btime":"2024-08-28T19:36:36:851162+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":82,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:36:36.606659+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:36:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 82 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 -- 172.21.15.16:0/3888085015 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2800b4ca0 msgr2=0x7ff2800b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2800b4ca0 0x7ff2800b71b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff284005ec0 tx=0x7ff284005a40 comp rx=0 tx=0).stop 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 -- 172.21.15.16:0/3888085015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff290109bb0 msgr2=0x7ff29019faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff290109bb0 0x7ff29019faa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff288009fd0 tx=0x7ff28800ef10 comp rx=0 tx=0).stop 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 -- 172.21.15.16:0/3888085015 shutdown_connections 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff2800b4ca0 0x7ff2800b71b0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff290109bb0 0x7ff29019faa0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff290109260 0x7ff29019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 --2- 172.21.15.16:0/3888085015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff290103920 0x7ff29019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 -- 172.21.15.16:0/3888085015 >> 172.21.15.16:0/3888085015 conn(0x7ff2900ff3b0 msgr2=0x7ff2901017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 -- 172.21.15.16:0/3888085015 shutdown_connections 2024-08-28T19:36:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:55.367+0000 7ff296f11640 1 -- 172.21.15.16:0/3888085015 wait complete. 2024-08-28T19:36:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:54 smithi115 ceph-mon[22184]: pgmap v833: 129 pgs: 129 active+clean; 935 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.8 MiB/s wr, 54 op/s 2024-08-28T19:36:55.469 INFO:tasks.workunit.client.0.smithi016.stdout: 10 39743 2.48 MB/sec execute 31 sec latency 81.691 ms 2024-08-28T19:36:56.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:55 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3888085015' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:56.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:36:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:55 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3888085015' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:55 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3888085015' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:36:56.384 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:56.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 40513 3.70 MB/sec execute 32 sec latency 209.049 ms 2024-08-28T19:36:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.007+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1793390079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4108140 msgr2=0x7f19e4108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.007+0000 7f19e8a6a640 1 --2- 172.21.15.16:0/1793390079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4108140 0x7f19e4108550 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f19d8009900 tx=0x7f19d802a410 comp rx=0 tx=0).stop 2024-08-28T19:36:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.007+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1793390079 shutdown_connections 2024-08-28T19:36:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.007+0000 7f19e8a6a640 1 --2- 172.21.15.16:0/1793390079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f19e4108a90 0x7f19e410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.007+0000 7f19e8a6a640 1 --2- 172.21.15.16:0/1793390079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4108140 0x7f19e4108550 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.007+0000 7f19e8a6a640 1 --2- 172.21.15.16:0/1793390079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e4102800 0x7f19e4102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.007+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1793390079 >> 172.21.15.16:0/1793390079 conn(0x7f19e40794d0 msgr2=0x7f19e40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.007+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1793390079 shutdown_connections 2024-08-28T19:36:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.007+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1793390079 wait complete. 2024-08-28T19:36:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.008+0000 7f19e8a6a640 1 Processor -- start 2024-08-28T19:36:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.008+0000 7f19e8a6a640 1 -- start start 2024-08-28T19:36:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.008+0000 7f19e8a6a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f19e4102800 0x7f19e4199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.008+0000 7f19e8a6a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4108140 0x7f19e41996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.008+0000 7f19e8a6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e4108a90 0x7f19e419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.008+0000 7f19e8a6a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19e4112730 con 0x7f19e4108140 2024-08-28T19:36:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.008+0000 7f19e8a6a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41125b0 con 0x7f19e4108a90 2024-08-28T19:36:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.008+0000 7f19e8a6a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41128b0 con 0x7f19e4102800 2024-08-28T19:36:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.008+0000 7f19e1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4108140 0x7f19e41996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.008+0000 7f19e1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4108140 0x7f19e41996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42760/0 (socket says 172.21.15.16:42760) 2024-08-28T19:36:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.008+0000 7f19e1d9b640 1 -- 172.21.15.16:0/1902018936 learned_addr learned my addr 172.21.15.16:0/1902018936 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19e2d9d640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e4108a90 0x7f19e419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19e259c640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f19e4102800 0x7f19e4199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19e1d9b640 1 -- 172.21.15.16:0/1902018936 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f19e4102800 msgr2=0x7f19e4199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19e1d9b640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f19e4102800 0x7f19e4199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19e1d9b640 1 -- 172.21.15.16:0/1902018936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e4108a90 msgr2=0x7f19e419e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19e1d9b640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e4108a90 0x7f19e419e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19e1d9b640 1 -- 172.21.15.16:0/1902018936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e419f090 con 0x7f19e4108140 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19e2d9d640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e4108a90 0x7f19e419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19e1d9b640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4108140 0x7f19e41996c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f19d8005ab0 tx=0x7f19d8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19d37fe640 1 -- 172.21.15.16:0/1902018936 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d803f070 con 0x7f19e4108140 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19d37fe640 1 -- 172.21.15.16:0/1902018936 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d8031b50 con 0x7f19e4108140 2024-08-28T19:36:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19d37fe640 1 -- 172.21.15.16:0/1902018936 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d8035030 con 0x7f19e4108140 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1902018936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e419f320 con 0x7f19e4108140 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.009+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1902018936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e41a5ae0 con 0x7f19e4108140 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.013+0000 7f19eefdf740 1 Processor -- start 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.014+0000 7f19eefdf740 1 -- start start 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.014+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b5d070 0x55fa67f02350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.014+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f02890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.014+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f02dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.014+0000 7f19eefdf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fa67e79680 con 0x55fa67b3c390 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.014+0000 7f19eefdf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fa67e68c10 con 0x55fa67b341b0 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.014+0000 7f19eefdf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55fa67e6cbc0 con 0x55fa67b5d070 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.014+0000 7f19e259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f02890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.014+0000 7f19e259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f02890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49810/0 (socket says 172.21.15.16:49810) 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.014+0000 7f19e259c640 1 -- 172.21.15.16:0/3169042479 learned_addr learned my addr 172.21.15.16:0/3169042479 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.015+0000 7f19e259c640 1 -- 172.21.15.16:0/3169042479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b5d070 msgr2=0x55fa67f02350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.015+0000 7f19e259c640 1 --2- 172.21.15.16:0/3169042479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b5d070 0x55fa67f02350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.015+0000 7f19e259c640 1 -- 172.21.15.16:0/3169042479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 msgr2=0x55fa67f02dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.015+0000 7f19e259c640 1 --2- 172.21.15.16:0/3169042479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f02dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.015+0000 7f19e259c640 1 -- 172.21.15.16:0/3169042479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa67bc8650 con 0x55fa67b341b0 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.015+0000 7f19d37fe640 1 -- 172.21.15.16:0/1902018936 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f19d8042070 con 0x7f19e4108140 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.015+0000 7f19e259c640 1 --2- 172.21.15.16:0/3169042479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f02890 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009cf0 tx=0x7f19d4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8534549c48a6430 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.015+0000 7f19d0ff9640 1 -- 172.21.15.16:0/3169042479 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d401e070 con 0x55fa67b341b0 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.016+0000 7f19d0ff9640 1 -- 172.21.15.16:0/3169042479 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d4019ae0 con 0x55fa67b341b0 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.016+0000 7f19d0ff9640 1 -- 172.21.15.16:0/3169042479 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d400e970 con 0x55fa67b341b0 2024-08-28T19:36:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.016+0000 7f19d37fe640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19cc0b4da0 0x7f19cc0b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.016+0000 7f19e2d9d640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19cc0b4da0 0x7f19cc0b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.016+0000 7f19d37fe640 1 -- 172.21.15.16:0/1902018936 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f19d8105ee0 con 0x7f19e4108140 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19eefdf740 1 -- 172.21.15.16:0/3169042479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 msgr2=0x55fa67f02890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19eefdf740 1 --2- 172.21.15.16:0/3169042479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f02890 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009cf0 tx=0x7f19d4011810 comp rx=0 tx=0).stop 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19e2d9d640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19cc0b4da0 0x7f19cc0b72b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f19dc005fd0 tx=0x7f19dc009f00 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19eefdf740 1 -- 172.21.15.16:0/3169042479 shutdown_connections 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19eefdf740 1 --2- 172.21.15.16:0/3169042479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f02dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19eefdf740 1 --2- 172.21.15.16:0/3169042479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f02890 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19eefdf740 1 --2- 172.21.15.16:0/3169042479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b5d070 0x55fa67f02350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19eefdf740 1 -- 172.21.15.16:0/3169042479 >> 172.21.15.16:0/3169042479 conn(0x55fa67b51460 msgr2=0x55fa67f00240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19eefdf740 1 -- 172.21.15.16:0/3169042479 shutdown_connections 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19eefdf740 1 -- 172.21.15.16:0/3169042479 wait complete. 2024-08-28T19:36:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19eefdf740 1 -- start start 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.017+0000 7f19eefdf740 1 Processor -- start 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f09b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f0a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b5d070 0x55fa67f1c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19eefdf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fa67ccb700 con 0x55fa67b3c390 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19eefdf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fa67def750 con 0x55fa67b341b0 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19eefdf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55fa67f12de0 con 0x55fa67b5d070 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19eefdf740 10 client.0 _ll_register_callbacks cb 0x55fa67e1ee10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19e259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f09b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19e1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f0a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19e1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f0a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42774/0 (socket says 172.21.15.16:42774) 2024-08-28T19:36:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19e1d9b640 1 -- 172.21.15.16:0/3996273558 learned_addr learned my addr 172.21.15.16:0/3996273558 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.018+0000 7f19e2d9d640 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b5d070 0x55fa67f1c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19e259c640 1 -- 172.21.15.16:0/3996273558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b5d070 msgr2=0x55fa67f1c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19e259c640 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b5d070 0x55fa67f1c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19e259c640 1 -- 172.21.15.16:0/3996273558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 msgr2=0x55fa67f0a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19e259c640 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f0a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19e259c640 1 -- 172.21.15.16:0/3996273558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa67e28900 con 0x55fa67b341b0 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19e2d9d640 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b5d070 0x55fa67f1c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19e1d9b640 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f0a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19e259c640 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f09b40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009cf0 tx=0x7f19d4019960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26e431b203651ea5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19c9ffb640 1 -- 172.21.15.16:0/3996273558 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d401e050 con 0x55fa67b341b0 2024-08-28T19:36:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19c9ffb640 1 -- 172.21.15.16:0/3996273558 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f19d4005450 con 0x55fa67b341b0 2024-08-28T19:36:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19eefdf740 1 -- 172.21.15.16:0/3996273558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19d4019ae0 con 0x55fa67b341b0 2024-08-28T19:36:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.019+0000 7f19c9ffb640 1 -- 172.21.15.16:0/3996273558 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f19d4033050 con 0x55fa67b341b0 2024-08-28T19:36:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.020+0000 7f19eefdf740 10 client.28790 fetch_fsmap learned FSMap version 82 2024-08-28T19:36:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.020+0000 7f19eefdf740 1 -- 172.21.15.16:0/3996273558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55fa67f0e270 con 0x55fa67b341b0 2024-08-28T19:36:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.020+0000 7f19c9ffb640 1 -- 172.21.15.16:0/3996273558 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d4019ae0 con 0x55fa67b341b0 2024-08-28T19:36:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.020+0000 7f19c9ffb640 1 -- 172.21.15.16:0/3996273558 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7f19d400e5f0 con 0x55fa67b341b0 2024-08-28T19:36:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.020+0000 7f19eefdf740 10 client.28790 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:36:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.020+0000 7f19eefdf740 10 client.28790 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:36:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.020+0000 7f19eefdf740 20 client.28790 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.020+0000 7f19eefdf740 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55fa67b1b9f0 0x55fa67f24620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.020+0000 7f19eefdf740 4 client.28790 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.020+0000 7f19eefdf740 1 -- 172.21.15.16:0/3996273558 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa67e7b5d0 con 0x55fa67b1b9f0 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.021+0000 7f19e1d9b640 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55fa67b1b9f0 0x55fa67f24620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.024+0000 7f19c9ffb640 10 client.28790 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.024+0000 7f19e1d9b640 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55fa67b1b9f0 0x55fa67f24620 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c13782cbc1fd4801 server_cookie=be6144b1ce47c1ed in_seq=0 out_seq=0 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19c9ffb640 1 -- 172.21.15.16:0/3996273558 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55fa67e7b5d0 con 0x55fa67b1b9f0 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19c9ffb640 10 client.28790 handle_command_reply: tid=0 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 1 client.28790 shutdown 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 1 -- 172.21.15.16:0/3996273558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 msgr2=0x55fa67f09b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f09b40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009cf0 tx=0x7f19d4019960 comp rx=0 tx=0).stop 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 1 -- 172.21.15.16:0/3996273558 shutdown_connections 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55fa67b1b9f0 0x55fa67f24620 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b5d070 0x55fa67f1c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f0a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19c9ffb640 0 client.28790 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 1 --2- 172.21.15.16:0/3996273558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f09b40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 1 -- 172.21.15.16:0/3996273558 >> 172.21.15.16:0/3996273558 conn(0x55fa67b51460 msgr2=0x55fa67f054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 1 -- 172.21.15.16:0/3996273558 shutdown_connections 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 1 -- 172.21.15.16:0/3996273558 wait complete. 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.027+0000 7f19eefdf740 20 client.28790 trim_cache size 0 max 16384 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19eefdf740 1 Processor -- start 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19eefdf740 1 -- start start 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b1b9f0 0x55fa67f00270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b341b0 0x55fa67f072a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b3c390 0x55fa67f1c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19eefdf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41aea90 con 0x55fa67b341b0 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19eefdf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b0810 con 0x55fa67b1b9f0 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19eefdf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41af190 con 0x55fa67b3c390 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19e1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b341b0 0x55fa67f072a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19e259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b1b9f0 0x55fa67f00270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19e1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b341b0 0x55fa67f072a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42776/0 (socket says 172.21.15.16:42776) 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19e1d9b640 1 -- 172.21.15.16:0/379723021 learned_addr learned my addr 172.21.15.16:0/379723021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.046+0000 7f19e2d9d640 1 --2- 172.21.15.16:0/379723021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b3c390 0x55fa67f1c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.047+0000 7f19e259c640 1 -- 172.21.15.16:0/379723021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b3c390 msgr2=0x55fa67f1c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.047+0000 7f19e259c640 1 --2- 172.21.15.16:0/379723021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b3c390 0x55fa67f1c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.047+0000 7f19e259c640 1 -- 172.21.15.16:0/379723021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b341b0 msgr2=0x55fa67f072a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.047+0000 7f19e259c640 1 --2- 172.21.15.16:0/379723021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b341b0 0x55fa67f072a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.047+0000 7f19e259c640 1 -- 172.21.15.16:0/379723021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa67f04640 con 0x55fa67b1b9f0 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.047+0000 7f19e1d9b640 1 --2- 172.21.15.16:0/379723021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b341b0 0x55fa67f072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.047+0000 7f19e259c640 1 --2- 172.21.15.16:0/379723021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b1b9f0 0x55fa67f00270 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009460 tx=0x7f19d4009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b4f41e133a02fd2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.047+0000 7f19cbfff640 1 -- 172.21.15.16:0/379723021 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d401e040 con 0x55fa67b1b9f0 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.047+0000 7f19cbfff640 1 -- 172.21.15.16:0/379723021 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d402a070 con 0x55fa67b1b9f0 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.047+0000 7f19cbfff640 1 -- 172.21.15.16:0/379723021 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d40199d0 con 0x55fa67b1b9f0 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.048+0000 7f19eefdf740 1 -- 172.21.15.16:0/379723021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b1b9f0 msgr2=0x55fa67f00270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.048+0000 7f19eefdf740 1 --2- 172.21.15.16:0/379723021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b1b9f0 0x55fa67f00270 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009460 tx=0x7f19d4009f60 comp rx=0 tx=0).stop 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.048+0000 7f19eefdf740 1 -- 172.21.15.16:0/379723021 shutdown_connections 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.048+0000 7f19eefdf740 1 --2- 172.21.15.16:0/379723021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b3c390 0x55fa67f1c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.048+0000 7f19eefdf740 1 --2- 172.21.15.16:0/379723021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b341b0 0x55fa67f072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.048+0000 7f19eefdf740 1 --2- 172.21.15.16:0/379723021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b1b9f0 0x55fa67f00270 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.048+0000 7f19eefdf740 1 -- 172.21.15.16:0/379723021 >> 172.21.15.16:0/379723021 conn(0x55fa67b51460 msgr2=0x55fa67effb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.048+0000 7f19eefdf740 1 -- 172.21.15.16:0/379723021 shutdown_connections 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.048+0000 7f19eefdf740 1 -- 172.21.15.16:0/379723021 wait complete. 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19eefdf740 1 -- start start 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19eefdf740 1 Processor -- start 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b1b9f0 0x55fa67f24d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f25260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19eefdf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f8cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19eefdf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b0080 con 0x55fa67b3c390 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19eefdf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41af8f0 con 0x55fa67b341b0 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19eefdf740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41aca90 con 0x55fa67b1b9f0 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19eefdf740 10 client.0 _ll_register_callbacks cb 0x7f19d401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19e2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f8cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19e1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f25260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19e2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f8cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42784/0 (socket says 172.21.15.16:42784) 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19e2d9d640 1 -- 172.21.15.16:0/2431221289 learned_addr learned my addr 172.21.15.16:0/2431221289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.049+0000 7f19e259c640 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b1b9f0 0x55fa67f24d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19e1d9b640 1 -- 172.21.15.16:0/2431221289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b1b9f0 msgr2=0x55fa67f24d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19e1d9b640 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b1b9f0 0x55fa67f24d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19e1d9b640 1 -- 172.21.15.16:0/2431221289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 msgr2=0x55fa67f8cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19e1d9b640 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f8cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19e1d9b640 1 -- 172.21.15.16:0/2431221289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa67f8d2d0 con 0x55fa67b341b0 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19e1d9b640 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f25260 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f19d80c82d0 tx=0x7f19d80ef040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df25d3bb5c4b9c15 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19c9ffb640 1 -- 172.21.15.16:0/2431221289 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d80b7070 con 0x55fa67b341b0 2024-08-28T19:36:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19c9ffb640 1 -- 172.21.15.16:0/2431221289 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f19d803b460 con 0x55fa67b341b0 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19eefdf740 1 -- 172.21.15.16:0/2431221289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19d402a070 con 0x55fa67b341b0 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19c9ffb640 1 -- 172.21.15.16:0/2431221289 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f19d8049040 con 0x55fa67b341b0 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19eefdf740 10 client.28796 fetch_fsmap learned FSMap version 82 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.050+0000 7f19eefdf740 1 -- 172.21.15.16:0/2431221289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55fa67f8e560 con 0x55fa67b341b0 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.051+0000 7f19c9ffb640 1 -- 172.21.15.16:0/2431221289 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d402a070 con 0x55fa67b341b0 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.051+0000 7f19c9ffb640 1 -- 172.21.15.16:0/2431221289 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7f19d8103480 con 0x55fa67b341b0 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.051+0000 7f19eefdf740 10 client.28796 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.051+0000 7f19eefdf740 10 client.28796 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.051+0000 7f19eefdf740 20 client.28796 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.051+0000 7f19eefdf740 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55fa67b5d070 0x55fa67f90ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.051+0000 7f19eefdf740 4 client.28796 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.051+0000 7f19eefdf740 1 -- 172.21.15.16:0/2431221289 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fa67e77070 con 0x55fa67b5d070 2024-08-28T19:36:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.052+0000 7f19e259c640 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55fa67b5d070 0x55fa67f90ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.052+0000 7f19e259c640 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55fa67b5d070 0x55fa67f90ca0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f251dd6ceffc9334 server_cookie=e891a9ee4413a97d in_seq=0 out_seq=0 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.052+0000 7f19c9ffb640 10 client.28796 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.054+0000 7f19c9ffb640 1 -- 172.21.15.16:0/2431221289 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fa67e77070 con 0x55fa67b5d070 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.054+0000 7f19c9ffb640 10 client.28796 handle_command_reply: tid=0 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.054+0000 7f19eefdf740 1 client.28796 shutdown 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19eefdf740 1 -- 172.21.15.16:0/2431221289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 msgr2=0x55fa67f25260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19eefdf740 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f25260 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f19d80c82d0 tx=0x7f19d80ef040 comp rx=0 tx=0).stop 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19eefdf740 1 -- 172.21.15.16:0/2431221289 shutdown_connections 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19eefdf740 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55fa67b5d070 0x55fa67f90ca0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19eefdf740 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa67b3c390 0x55fa67f8cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19eefdf740 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa67b341b0 0x55fa67f25260 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19eefdf740 1 --2- 172.21.15.16:0/2431221289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fa67b1b9f0 0x55fa67f24d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19eefdf740 1 -- 172.21.15.16:0/2431221289 >> 172.21.15.16:0/2431221289 conn(0x55fa67b51460 msgr2=0x55fa67f03d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19c9ffb640 0 client.28796 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:36:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19eefdf740 1 -- 172.21.15.16:0/2431221289 shutdown_connections 2024-08-28T19:36:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19eefdf740 1 -- 172.21.15.16:0/2431221289 wait complete. 2024-08-28T19:36:57.109 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:36:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.055+0000 7f19eefdf740 20 client.28796 trim_cache size 0 max 16384 2024-08-28T19:36:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1902018936 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19cc0b4da0 msgr2=0x7f19cc0b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19cc0b4da0 0x7f19cc0b72b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f19dc005fd0 tx=0x7f19dc009f00 comp rx=0 tx=0).stop 2024-08-28T19:36:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1902018936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4108140 msgr2=0x7f19e41996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4108140 0x7f19e41996c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f19d8005ab0 tx=0x7f19d8003b90 comp rx=0 tx=0).stop 2024-08-28T19:36:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1902018936 shutdown_connections 2024-08-28T19:36:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f19cc0b4da0 0x7f19cc0b72b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e4108a90 0x7f19e419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4108140 0x7f19e41996c0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 --2- 172.21.15.16:0/1902018936 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f19e4102800 0x7f19e4199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1902018936 >> 172.21.15.16:0/1902018936 conn(0x7f19e40794d0 msgr2=0x7f19e4104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1902018936 shutdown_connections 2024-08-28T19:36:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:57.056+0000 7f19e8a6a640 1 -- 172.21.15.16:0/1902018936 wait complete. 2024-08-28T19:36:57.210 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:56 smithi028 ceph-mon[21769]: pgmap v834: 129 pgs: 129 active+clean; 935 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 134 op/s 2024-08-28T19:36:57.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:56 smithi016 ceph-mon[18556]: pgmap v834: 129 pgs: 129 active+clean; 935 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 134 op/s 2024-08-28T19:36:57.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:56 smithi115 ceph-mon[22184]: pgmap v834: 129 pgs: 129 active+clean; 935 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 134 op/s 2024-08-28T19:36:57.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 41141 4.75 MB/sec execute 33 sec latency 67.543 ms 2024-08-28T19:36:58.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:36:58.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 41879 5.60 MB/sec execute 34 sec latency 118.764 ms 2024-08-28T19:36:58.512 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:36:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.056+0000 7f05bfb67640 1 -- 172.21.15.16:0/2243613206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b810dc30 msgr2=0x7f05b81100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.056+0000 7f05bfb67640 1 --2- 172.21.15.16:0/2243613206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b810dc30 0x7f05b81100b0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009ca0 tx=0x7f05b002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.056+0000 7f05bfb67640 1 -- 172.21.15.16:0/2243613206 shutdown_connections 2024-08-28T19:36:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.056+0000 7f05bfb67640 1 --2- 172.21.15.16:0/2243613206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b810dc30 0x7f05b81100b0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.056+0000 7f05bfb67640 1 --2- 172.21.15.16:0/2243613206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05b8104fd0 0x7f05b810d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.056+0000 7f05bfb67640 1 --2- 172.21.15.16:0/2243613206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b8104680 0x7f05b8104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.056+0000 7f05bfb67640 1 -- 172.21.15.16:0/2243613206 >> 172.21.15.16:0/2243613206 conn(0x7f05b8079530 msgr2=0x7f05b8079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.057+0000 7f05bfb67640 1 -- 172.21.15.16:0/2243613206 shutdown_connections 2024-08-28T19:36:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.057+0000 7f05bfb67640 1 -- 172.21.15.16:0/2243613206 wait complete. 2024-08-28T19:36:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.058+0000 7f05bfb67640 1 Processor -- start 2024-08-28T19:36:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.058+0000 7f05bfb67640 1 -- start start 2024-08-28T19:36:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.058+0000 7f05bfb67640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b8104680 0x7f05b81a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.058+0000 7f05bfb67640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b8104fd0 0x7f05b81a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.058+0000 7f05bfb67640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05b810dc30 0x7f05b81a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.058+0000 7f05bfb67640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05b8113830 con 0x7f05b8104fd0 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.058+0000 7f05bfb67640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05b81136b0 con 0x7f05b8104680 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.058+0000 7f05bfb67640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f05b81139b0 con 0x7f05b810dc30 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.059+0000 7f05be104640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05b810dc30 0x7f05b81a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.059+0000 7f05be104640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05b810dc30 0x7f05b81a1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34712/0 (socket says 172.21.15.16:34712) 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.059+0000 7f05bd903640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b8104680 0x7f05b81a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.059+0000 7f05be104640 1 -- 172.21.15.16:0/2338615947 learned_addr learned my addr 172.21.15.16:0/2338615947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.059+0000 7f05be104640 1 -- 172.21.15.16:0/2338615947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b8104680 msgr2=0x7f05b81a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.059+0000 7f05be104640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b8104680 0x7f05b81a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.059+0000 7f05be104640 1 -- 172.21.15.16:0/2338615947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b8104fd0 msgr2=0x7f05b81a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.059+0000 7f05be104640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b8104fd0 0x7f05b81a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.059+0000 7f05be104640 1 -- 172.21.15.16:0/2338615947 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b81ac820 con 0x7f05b810dc30 2024-08-28T19:36:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.059+0000 7f05be104640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05b810dc30 0x7f05b81a1f90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f05b0005cc0 tx=0x7f05b0031700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.060+0000 7f05aeffd640 1 -- 172.21.15.16:0/2338615947 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0003b70 con 0x7f05b810dc30 2024-08-28T19:36:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.060+0000 7f05aeffd640 1 -- 172.21.15.16:0/2338615947 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05b00318f0 con 0x7f05b810dc30 2024-08-28T19:36:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.060+0000 7f05aeffd640 1 -- 172.21.15.16:0/2338615947 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b002f990 con 0x7f05b810dc30 2024-08-28T19:36:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.060+0000 7f05bfb67640 1 -- 172.21.15.16:0/2338615947 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b81acab0 con 0x7f05b810dc30 2024-08-28T19:36:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.060+0000 7f05bfb67640 1 -- 172.21.15.16:0/2338615947 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b81acee0 con 0x7f05b810dc30 2024-08-28T19:36:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.064+0000 7f05aeffd640 1 -- 172.21.15.16:0/2338615947 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f05b002fb70 con 0x7f05b810dc30 2024-08-28T19:36:59.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.066+0000 7f05aeffd640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05a40b4da0 0x7f05a40b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.066+0000 7f05c60dc740 1 Processor -- start 2024-08-28T19:36:59.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.066+0000 7f05bd903640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05a40b4da0 0x7f05a40b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.066+0000 7f05aeffd640 1 -- 172.21.15.16:0/2338615947 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f05b0106ca0 con 0x7f05b810dc30 2024-08-28T19:36:59.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.066+0000 7f05c60dc740 1 -- start start 2024-08-28T19:36:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.066+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5717070 0x564ed5abc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.066+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56ee1b0 0x564ed5abc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.066+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56f6390 0x564ed5abcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05c60dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564ed5a33680 con 0x564ed5717070 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05c60dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ed5a22c10 con 0x564ed56f6390 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05c60dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564ed5a26bc0 con 0x564ed56ee1b0 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05bd102640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56ee1b0 0x564ed5abc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05bd102640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56ee1b0 0x564ed5abc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34720/0 (socket says 172.21.15.16:34720) 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05bd102640 1 -- 172.21.15.16:0/1589384662 learned_addr learned my addr 172.21.15.16:0/1589384662 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05be104640 1 --2- 172.21.15.16:0/1589384662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56f6390 0x564ed5abcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05bd903640 1 --2- 172.21.15.16:0/1589384662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5717070 0x564ed5abc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05bd102640 1 -- 172.21.15.16:0/1589384662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56f6390 msgr2=0x564ed5abcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05bd102640 1 --2- 172.21.15.16:0/1589384662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56f6390 0x564ed5abcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05bd102640 1 -- 172.21.15.16:0/1589384662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5717070 msgr2=0x564ed5abc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05bd102640 1 --2- 172.21.15.16:0/1589384662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5717070 0x564ed5abc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.067+0000 7f05bd102640 1 -- 172.21.15.16:0/1589384662 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed5782650 con 0x564ed56ee1b0 2024-08-28T19:36:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.068+0000 7f05bd102640 1 --2- 172.21.15.16:0/1589384662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56ee1b0 0x564ed5abc850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009bb0 tx=0x7f05a800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f232a59fed8bacb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.068+0000 7f05bd903640 1 --2- 172.21.15.16:0/1589384662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5717070 0x564ed5abc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:36:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.068+0000 7f05a3fff640 1 -- 172.21.15.16:0/1589384662 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8005020 con 0x564ed56ee1b0 2024-08-28T19:36:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.068+0000 7f05bd903640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05a40b4da0 0x7f05a40b72b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f05b810d700 tx=0x7f05b4009b10 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.068+0000 7f05a3fff640 1 -- 172.21.15.16:0/1589384662 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05a80047a0 con 0x564ed56ee1b0 2024-08-28T19:36:59.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.068+0000 7f05a3fff640 1 -- 172.21.15.16:0/1589384662 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8010690 con 0x564ed56ee1b0 2024-08-28T19:36:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.070+0000 7f05c60dc740 1 -- 172.21.15.16:0/1589384662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56ee1b0 msgr2=0x564ed5abc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.070+0000 7f05c60dc740 1 --2- 172.21.15.16:0/1589384662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56ee1b0 0x564ed5abc850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009bb0 tx=0x7f05a800e720 comp rx=0 tx=0).stop 2024-08-28T19:36:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.070+0000 7f05c60dc740 1 -- 172.21.15.16:0/1589384662 shutdown_connections 2024-08-28T19:36:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.070+0000 7f05c60dc740 1 --2- 172.21.15.16:0/1589384662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56f6390 0x564ed5abcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.070+0000 7f05c60dc740 1 --2- 172.21.15.16:0/1589384662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56ee1b0 0x564ed5abc850 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.070+0000 7f05c60dc740 1 --2- 172.21.15.16:0/1589384662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5717070 0x564ed5abc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.070+0000 7f05c60dc740 1 -- 172.21.15.16:0/1589384662 >> 172.21.15.16:0/1589384662 conn(0x564ed570b460 msgr2=0x564ed5aba200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.070+0000 7f05c60dc740 1 -- 172.21.15.16:0/1589384662 shutdown_connections 2024-08-28T19:36:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.071+0000 7f05c60dc740 1 -- 172.21.15.16:0/1589384662 wait complete. 2024-08-28T19:36:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.071+0000 7f05c60dc740 1 -- start start 2024-08-28T19:36:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.071+0000 7f05c60dc740 1 Processor -- start 2024-08-28T19:36:59.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.072+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56ee1b0 0x564ed5ac7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.072+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56f6390 0x564ed5ac7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.072+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed5717070 0x564ed5ad3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.072+0000 7f05c60dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564ed5885700 con 0x564ed56f6390 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.072+0000 7f05c60dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ed59a9750 con 0x564ed56ee1b0 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.072+0000 7f05c60dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564ed5acaaf0 con 0x564ed5717070 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.072+0000 7f05c60dc740 10 client.0 _ll_register_callbacks cb 0x564ed59d8e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.073+0000 7f05be104640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed5717070 0x564ed5ad3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.073+0000 7f05be104640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed5717070 0x564ed5ad3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34722/0 (socket says 172.21.15.16:34722) 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.073+0000 7f05be104640 1 -- 172.21.15.16:0/2338012815 learned_addr learned my addr 172.21.15.16:0/2338012815 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.073+0000 7f05bd102640 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56f6390 0x564ed5ac7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.073+0000 7f05be104640 1 -- 172.21.15.16:0/2338012815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56ee1b0 msgr2=0x564ed5ac7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.073+0000 7f05be104640 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56ee1b0 0x564ed5ac7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.073+0000 7f05be104640 1 -- 172.21.15.16:0/2338012815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56f6390 msgr2=0x564ed5ac7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.073+0000 7f05be104640 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56f6390 0x564ed5ac7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.073+0000 7f05be104640 1 -- 172.21.15.16:0/2338012815 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ed59e2900 con 0x564ed5717070 2024-08-28T19:36:59.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.074+0000 7f05be104640 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed5717070 0x564ed5ad3de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f05b00ccc60 tx=0x7f05b00cd720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd9b74d3dbbc178e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:59.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.074+0000 7f05a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:36:59.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.074+0000 7f05a17fa640 1 -- 172.21.15.16:0/2338012815 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0104450 con 0x564ed5717070 2024-08-28T19:36:59.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.074+0000 7f05a17fa640 1 -- 172.21.15.16:0/2338012815 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f05b00d5df0 con 0x564ed5717070 2024-08-28T19:36:59.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.074+0000 7f05c60dc740 1 -- 172.21.15.16:0/2338012815 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05a80047a0 con 0x564ed5717070 2024-08-28T19:36:59.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.075+0000 7f05a17fa640 1 -- 172.21.15.16:0/2338012815 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f05b00e7050 con 0x564ed5717070 2024-08-28T19:36:59.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.075+0000 7f05c60dc740 10 client.28432 fetch_fsmap learned FSMap version 82 2024-08-28T19:36:59.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.075+0000 7f05c60dc740 1 -- 172.21.15.16:0/2338012815 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x564ed5ad6840 con 0x564ed5717070 2024-08-28T19:36:59.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.076+0000 7f05a17fa640 1 -- 172.21.15.16:0/2338012815 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05b0036070 con 0x564ed5717070 2024-08-28T19:36:59.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.076+0000 7f05a17fa640 1 -- 172.21.15.16:0/2338012815 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7f05b00d0070 con 0x564ed5717070 2024-08-28T19:36:59.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.076+0000 7f05c60dc740 10 client.28432 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:36:59.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.076+0000 7f05c60dc740 10 client.28432 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:36:59.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.076+0000 7f05c60dc740 20 client.28432 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:59.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.077+0000 7f05c60dc740 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564ed56d59f0 0x564ed5ad9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.077+0000 7f05c60dc740 4 client.28432 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:36:59.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.077+0000 7f05c60dc740 1 -- 172.21.15.16:0/2338012815 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f05a80047a0 con 0x564ed56d59f0 2024-08-28T19:36:59.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.077+0000 7f05bd903640 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564ed56d59f0 0x564ed5ad9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.077+0000 7f05a17fa640 10 client.28432 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:36:59.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.077+0000 7f05bd903640 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564ed56d59f0 0x564ed5ad9e80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b5d863ac4b90dd15 server_cookie=ada6b1deff3746c6 in_seq=0 out_seq=0 2024-08-28T19:36:59.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.080+0000 7f05a17fa640 1 -- 172.21.15.16:0/2338012815 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f05a80047a0 con 0x564ed56d59f0 2024-08-28T19:36:59.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.080+0000 7f05a17fa640 10 client.28432 handle_command_reply: tid=0 2024-08-28T19:36:59.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.080+0000 7f05c60dc740 1 client.28432 shutdown 2024-08-28T19:36:59.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.080+0000 7f05c60dc740 1 -- 172.21.15.16:0/2338012815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed5717070 msgr2=0x564ed5ad3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:59.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.080+0000 7f05c60dc740 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed5717070 0x564ed5ad3de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f05b00ccc60 tx=0x7f05b00cd720 comp rx=0 tx=0).stop 2024-08-28T19:36:59.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.081+0000 7f05c60dc740 1 -- 172.21.15.16:0/2338012815 shutdown_connections 2024-08-28T19:36:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.081+0000 7f05c60dc740 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564ed56d59f0 0x564ed5ad9e80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.081+0000 7f05c60dc740 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed5717070 0x564ed5ad3de0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.081+0000 7f05a17fa640 0 client.28432 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:36:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.081+0000 7f05c60dc740 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56f6390 0x564ed5ac7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.081+0000 7f05c60dc740 1 --2- 172.21.15.16:0/2338012815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56ee1b0 0x564ed5ac7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.081+0000 7f05c60dc740 1 -- 172.21.15.16:0/2338012815 >> 172.21.15.16:0/2338012815 conn(0x564ed570b460 msgr2=0x564ed5abf4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.081+0000 7f05c60dc740 1 -- 172.21.15.16:0/2338012815 shutdown_connections 2024-08-28T19:36:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.081+0000 7f05c60dc740 1 -- 172.21.15.16:0/2338012815 wait complete. 2024-08-28T19:36:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.081+0000 7f05c60dc740 20 client.28432 trim_cache size 0 max 16384 2024-08-28T19:36:59.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.097+0000 7f05c60dc740 1 Processor -- start 2024-08-28T19:36:59.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.097+0000 7f05c60dc740 1 -- start start 2024-08-28T19:36:59.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.097+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 0x564ed5ac7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.097+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56ee1b0 0x564ed5abf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.097+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56f6390 0x564ed5abf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.097+0000 7f05c60dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05b81b71e0 con 0x564ed56d59f0 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.097+0000 7f05c60dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05b81b8f60 con 0x564ed56f6390 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.097+0000 7f05c60dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f05b81b78e0 con 0x564ed56ee1b0 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.098+0000 7f05be104640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56f6390 0x564ed5abf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.098+0000 7f05be104640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56f6390 0x564ed5abf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49868/0 (socket says 172.21.15.16:49868) 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.098+0000 7f05be104640 1 -- 172.21.15.16:0/86139404 learned_addr learned my addr 172.21.15.16:0/86139404 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.098+0000 7f05bd903640 1 --2- 172.21.15.16:0/86139404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 0x564ed5ac7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.098+0000 7f05bd903640 1 -- 172.21.15.16:0/86139404 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56ee1b0 msgr2=0x564ed5abf410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.098+0000 7f05bd903640 1 --2- 172.21.15.16:0/86139404 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56ee1b0 0x564ed5abf410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.098+0000 7f05bd903640 1 -- 172.21.15.16:0/86139404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56f6390 msgr2=0x564ed5abf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.098+0000 7f05bd903640 1 --2- 172.21.15.16:0/86139404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56f6390 0x564ed5abf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.098+0000 7f05bd903640 1 -- 172.21.15.16:0/86139404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed5ad4590 con 0x564ed56d59f0 2024-08-28T19:36:59.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.099+0000 7f05bd903640 1 --2- 172.21.15.16:0/86139404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 0x564ed5ac7f50 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x564ed5ada5c0 tx=0x7f05b4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e641a320281560 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:59.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.099+0000 7f05a37fe640 1 -- 172.21.15.16:0/86139404 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b4005220 con 0x564ed56d59f0 2024-08-28T19:36:59.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.099+0000 7f05a37fe640 1 -- 172.21.15.16:0/86139404 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05b4004c50 con 0x564ed56d59f0 2024-08-28T19:36:59.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.099+0000 7f05a37fe640 1 -- 172.21.15.16:0/86139404 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b401e970 con 0x564ed56d59f0 2024-08-28T19:36:59.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.100+0000 7f05c60dc740 1 -- 172.21.15.16:0/86139404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 msgr2=0x564ed5ac7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:59.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.100+0000 7f05c60dc740 1 --2- 172.21.15.16:0/86139404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 0x564ed5ac7f50 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x564ed5ada5c0 tx=0x7f05b4015000 comp rx=0 tx=0).stop 2024-08-28T19:36:59.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.101+0000 7f05c60dc740 1 -- 172.21.15.16:0/86139404 shutdown_connections 2024-08-28T19:36:59.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.101+0000 7f05c60dc740 1 --2- 172.21.15.16:0/86139404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56f6390 0x564ed5abf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.101+0000 7f05c60dc740 1 --2- 172.21.15.16:0/86139404 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56ee1b0 0x564ed5abf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.101+0000 7f05c60dc740 1 --2- 172.21.15.16:0/86139404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 0x564ed5ac7f50 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.101+0000 7f05c60dc740 1 -- 172.21.15.16:0/86139404 >> 172.21.15.16:0/86139404 conn(0x564ed570b460 msgr2=0x564ed5ac7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:59.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.101+0000 7f05c60dc740 1 -- 172.21.15.16:0/86139404 shutdown_connections 2024-08-28T19:36:59.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.101+0000 7f05c60dc740 1 -- 172.21.15.16:0/86139404 wait complete. 2024-08-28T19:36:59.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.101+0000 7f05c60dc740 1 -- start start 2024-08-28T19:36:59.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.102+0000 7f05c60dc740 1 Processor -- start 2024-08-28T19:36:59.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 0x564ed5acc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56ee1b0 0x564ed5acca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05c60dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56f6390 0x564ed5b43880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05c60dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05b81b87d0 con 0x564ed56d59f0 2024-08-28T19:36:59.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05c60dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05b81b8040 con 0x564ed56ee1b0 2024-08-28T19:36:59.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05c60dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f05b81b51e0 con 0x564ed56f6390 2024-08-28T19:36:59.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05bd102640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56ee1b0 0x564ed5acca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05c60dc740 10 client.0 _ll_register_callbacks cb 0x7f05b0009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:36:59.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05bd102640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56ee1b0 0x564ed5acca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49882/0 (socket says 172.21.15.16:49882) 2024-08-28T19:36:59.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05bd102640 1 -- 172.21.15.16:0/671432742 learned_addr learned my addr 172.21.15.16:0/671432742 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:36:59.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05bd903640 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 0x564ed5acc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.103+0000 7f05be104640 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56f6390 0x564ed5b43880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05bd903640 1 -- 172.21.15.16:0/671432742 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56f6390 msgr2=0x564ed5b43880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:59.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05bd903640 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56f6390 0x564ed5b43880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05bd903640 1 -- 172.21.15.16:0/671432742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56ee1b0 msgr2=0x564ed5acca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:36:59.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05bd903640 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56ee1b0 0x564ed5acca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05bd903640 1 -- 172.21.15.16:0/671432742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ed5b43e40 con 0x564ed56d59f0 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05bd903640 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 0x564ed5acc4d0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f05b40058c0 tx=0x7f05b4004e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0f696289629fe7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05a17fa640 1 -- 172.21.15.16:0/671432742 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b403b020 con 0x564ed56d59f0 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05a17fa640 1 -- 172.21.15.16:0/671432742 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f05b4004fa0 con 0x564ed56d59f0 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05c60dc740 1 -- 172.21.15.16:0/671432742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05b4004c50 con 0x564ed56d59f0 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05a17fa640 1 -- 172.21.15.16:0/671432742 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f05b4046050 con 0x564ed56d59f0 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05c60dc740 10 client.19389 fetch_fsmap learned FSMap version 82 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.104+0000 7f05c60dc740 1 -- 172.21.15.16:0/671432742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x564ed5b450f0 con 0x564ed56d59f0 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.105+0000 7f05a17fa640 1 -- 172.21.15.16:0/671432742 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05b4004c50 con 0x564ed56d59f0 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.105+0000 7f05a17fa640 1 -- 172.21.15.16:0/671432742 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7f05b4036030 con 0x564ed56d59f0 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.105+0000 7f05c60dc740 10 client.19389 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:36:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.105+0000 7f05c60dc740 10 client.19389 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:36:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.105+0000 7f05c60dc740 20 client.19389 populate_metadata read hostname 'smithi016' 2024-08-28T19:36:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.105+0000 7f05c60dc740 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564ed5717070 0x564ed5b47960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:36:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.105+0000 7f05c60dc740 4 client.19389 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:36:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.105+0000 7f05c60dc740 1 -- 172.21.15.16:0/671432742 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564ed5a355d0 con 0x564ed5717070 2024-08-28T19:36:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.105+0000 7f05bd102640 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564ed5717070 0x564ed5b47960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:36:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.110+0000 7f05a17fa640 10 client.19389 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:36:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.110+0000 7f05bd102640 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564ed5717070 0x564ed5b47960 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5c0d340fdd76a1a5 server_cookie=4cc69b895301a799 in_seq=0 out_seq=0 2024-08-28T19:36:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05a17fa640 1 -- 172.21.15.16:0/671432742 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564ed5a355d0 con 0x564ed5717070 2024-08-28T19:36:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05a17fa640 10 client.19389 handle_command_reply: tid=0 2024-08-28T19:36:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 1 client.19389 shutdown 2024-08-28T19:36:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 1 -- 172.21.15.16:0/671432742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 msgr2=0x564ed5acc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 0x564ed5acc4d0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f05b40058c0 tx=0x7f05b4004e80 comp rx=0 tx=0).stop 2024-08-28T19:36:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 1 -- 172.21.15.16:0/671432742 shutdown_connections 2024-08-28T19:36:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x564ed5717070 0x564ed5b47960 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05a17fa640 0 client.19389 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:36:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564ed56f6390 0x564ed5b43880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ed56ee1b0 0x564ed5acca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 1 --2- 172.21.15.16:0/671432742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed56d59f0 0x564ed5acc4d0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 1 -- 172.21.15.16:0/671432742 >> 172.21.15.16:0/671432742 conn(0x564ed570b460 msgr2=0x564ed5abf410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 1 -- 172.21.15.16:0/671432742 shutdown_connections 2024-08-28T19:36:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 1 -- 172.21.15.16:0/671432742 wait complete. 2024-08-28T19:36:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.112+0000 7f05c60dc740 20 client.19389 trim_cache size 0 max 16384 2024-08-28T19:36:59.115 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:36:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 -- 172.21.15.16:0/2338615947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05a40b4da0 msgr2=0x7f05a40b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05a40b4da0 0x7f05a40b72b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f05b810d700 tx=0x7f05b4009b10 comp rx=0 tx=0).stop 2024-08-28T19:36:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 -- 172.21.15.16:0/2338615947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05b810dc30 msgr2=0x7f05b81a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:36:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05b810dc30 0x7f05b81a1f90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f05b0005cc0 tx=0x7f05b0031700 comp rx=0 tx=0).stop 2024-08-28T19:36:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 -- 172.21.15.16:0/2338615947 shutdown_connections 2024-08-28T19:36:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05b810dc30 0x7f05b81a1f90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b8104fd0 0x7f05b81a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f05a40b4da0 0x7f05a40b72b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 --2- 172.21.15.16:0/2338615947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b8104680 0x7f05b81a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:36:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 -- 172.21.15.16:0/2338615947 >> 172.21.15.16:0/2338615947 conn(0x7f05b8079530 msgr2=0x7f05b8100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:36:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 -- 172.21.15.16:0/2338615947 shutdown_connections 2024-08-28T19:36:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:36:59.113+0000 7f05bfb67640 1 -- 172.21.15.16:0/2338615947 wait complete. 2024-08-28T19:36:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:36:58 smithi016 ceph-mon[18556]: pgmap v835: 129 pgs: 129 active+clean; 937 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.8 MiB/s wr, 232 op/s 2024-08-28T19:36:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:36:58 smithi028 ceph-mon[21769]: pgmap v835: 129 pgs: 129 active+clean; 937 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.8 MiB/s wr, 232 op/s 2024-08-28T19:36:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:36:58 smithi115 ceph-mon[22184]: pgmap v835: 129 pgs: 129 active+clean; 937 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.8 MiB/s wr, 232 op/s 2024-08-28T19:36:59.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 42810 6.74 MB/sec execute 35 sec latency 1118.800 ms 2024-08-28T19:36:59.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:37:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:00.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 43652 8.06 MB/sec execute 36 sec latency 2118.832 ms 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.782+0000 7fe41652c640 1 -- 172.21.15.16:0/2177525931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4101081c0 msgr2=0x7fe4101085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.782+0000 7fe41652c640 1 --2- 172.21.15.16:0/2177525931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4101081c0 0x7fe4101085d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fe404009900 tx=0x7fe40402a410 comp rx=0 tx=0).stop 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.782+0000 7fe41652c640 1 -- 172.21.15.16:0/2177525931 shutdown_connections 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.782+0000 7fe41652c640 1 --2- 172.21.15.16:0/2177525931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe410108b10 0x7fe41010f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.782+0000 7fe41652c640 1 --2- 172.21.15.16:0/2177525931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4101081c0 0x7fe4101085d0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.782+0000 7fe41652c640 1 --2- 172.21.15.16:0/2177525931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe410102880 0x7fe410102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.782+0000 7fe41652c640 1 -- 172.21.15.16:0/2177525931 >> 172.21.15.16:0/2177525931 conn(0x7fe410079510 msgr2=0x7fe410079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.782+0000 7fe41652c640 1 -- 172.21.15.16:0/2177525931 shutdown_connections 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.782+0000 7fe41652c640 1 -- 172.21.15.16:0/2177525931 wait complete. 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.783+0000 7fe41652c640 1 Processor -- start 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.783+0000 7fe41652c640 1 -- start start 2024-08-28T19:37:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe41652c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410102880 0x7fe41019b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe41652c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4101081c0 0x7fe41019b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe41652c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe410108b10 0x7fe41019bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe41652c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe410112780 con 0x7fe410102880 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe41652c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe410112600 con 0x7fe410108b10 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe41652c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe410112900 con 0x7fe4101081c0 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe40ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410102880 0x7fe41019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe40ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410102880 0x7fe41019b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42858/0 (socket says 172.21.15.16:42858) 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe40ffff640 1 -- 172.21.15.16:0/841936943 learned_addr learned my addr 172.21.15.16:0/841936943 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe40f7fe640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4101081c0 0x7fe41019b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe414ac9640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe410108b10 0x7fe41019bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe40ffff640 1 -- 172.21.15.16:0/841936943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4101081c0 msgr2=0x7fe41019b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe40ffff640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4101081c0 0x7fe41019b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe40ffff640 1 -- 172.21.15.16:0/841936943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe410108b10 msgr2=0x7fe41019bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe40ffff640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe410108b10 0x7fe41019bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe40ffff640 1 -- 172.21.15.16:0/841936943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4101a6310 con 0x7fe410102880 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.784+0000 7fe40ffff640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410102880 0x7fe41019b110 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fe40000c920 tx=0x7fe40000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.785+0000 7fe40d7fa640 1 -- 172.21.15.16:0/841936943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe400007c70 con 0x7fe410102880 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.785+0000 7fe41652c640 1 -- 172.21.15.16:0/841936943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4101a6600 con 0x7fe410102880 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.785+0000 7fe40d7fa640 1 -- 172.21.15.16:0/841936943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe400007e10 con 0x7fe410102880 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.785+0000 7fe40d7fa640 1 -- 172.21.15.16:0/841936943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe400018820 con 0x7fe410102880 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.785+0000 7fe41652c640 1 -- 172.21.15.16:0/841936943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4101a6b10 con 0x7fe410102880 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.787+0000 7fe41caa1740 1 Processor -- start 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.787+0000 7fe41caa1740 1 -- start start 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.787+0000 7fe40d7fa640 1 -- 172.21.15.16:0/841936943 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe40000f450 con 0x7fe410102880 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 0x561c0498c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 0x561c0498c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045c6390 0x561c0498cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe41caa1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561c04903680 con 0x561c045e7070 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe41caa1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561c048f2c10 con 0x561c045be1b0 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe41caa1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561c048f6bc0 con 0x561c045c6390 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe414ac9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 0x561c0498c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe414ac9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 0x561c0498c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42868/0 (socket says 172.21.15.16:42868) 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe414ac9640 1 -- 172.21.15.16:0/3152667567 learned_addr learned my addr 172.21.15.16:0/3152667567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe40ffff640 1 --2- 172.21.15.16:0/3152667567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 0x561c0498c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe414ac9640 1 -- 172.21.15.16:0/3152667567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045c6390 msgr2=0x561c0498cdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe414ac9640 1 --2- 172.21.15.16:0/3152667567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045c6390 0x561c0498cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe414ac9640 1 -- 172.21.15.16:0/3152667567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 msgr2=0x561c0498c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe414ac9640 1 --2- 172.21.15.16:0/3152667567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 0x561c0498c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe414ac9640 1 -- 172.21.15.16:0/3152667567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c04652650 con 0x561c045e7070 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe40d7fa640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3f80b4cd0 0x7fe3f80b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.788+0000 7fe414ac9640 1 --2- 172.21.15.16:0/3152667567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 0x561c0498c350 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fe408007ec0 tx=0x7fe40800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67fc57a82ff7ce3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.789+0000 7fe40ffff640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3f80b4cd0 0x7fe3f80b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.789+0000 7fe40d7fa640 1 -- 172.21.15.16:0/841936943 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7fe4000e5980 con 0x7fe410102880 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.789+0000 7fe3f67fc640 1 -- 172.21.15.16:0/3152667567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe408010840 con 0x561c045e7070 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.789+0000 7fe3f67fc640 1 -- 172.21.15.16:0/3152667567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4080183f0 con 0x561c045e7070 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.789+0000 7fe3f67fc640 1 -- 172.21.15.16:0/3152667567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe408017600 con 0x561c045e7070 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.790+0000 7fe41caa1740 1 -- 172.21.15.16:0/3152667567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 msgr2=0x561c0498c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.790+0000 7fe41caa1740 1 --2- 172.21.15.16:0/3152667567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 0x561c0498c350 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fe408007ec0 tx=0x7fe40800f790 comp rx=0 tx=0).stop 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.790+0000 7fe40ffff640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3f80b4cd0 0x7fe3f80b71e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe4000e2db0 tx=0x7fe4000eb530 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.790+0000 7fe41caa1740 1 -- 172.21.15.16:0/3152667567 shutdown_connections 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.790+0000 7fe41caa1740 1 --2- 172.21.15.16:0/3152667567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045c6390 0x561c0498cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.790+0000 7fe41caa1740 1 --2- 172.21.15.16:0/3152667567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 0x561c0498c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.790+0000 7fe41caa1740 1 --2- 172.21.15.16:0/3152667567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 0x561c0498c350 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.790+0000 7fe41caa1740 1 -- 172.21.15.16:0/3152667567 >> 172.21.15.16:0/3152667567 conn(0x561c045db460 msgr2=0x561c0498a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:00.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.790+0000 7fe41caa1740 1 -- 172.21.15.16:0/3152667567 shutdown_connections 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.790+0000 7fe41caa1740 1 -- 172.21.15.16:0/3152667567 wait complete. 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.790+0000 7fe41caa1740 1 -- start start 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe41caa1740 1 Processor -- start 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045be1b0 0x561c0499bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c0499c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045e7070 0x561c049a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe41caa1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561c04755700 con 0x561c045c6390 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe41caa1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561c04879750 con 0x561c045e7070 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe41caa1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561c0499ee90 con 0x561c045be1b0 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe41caa1740 10 client.0 _ll_register_callbacks cb 0x561c048a8e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe40f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c0499c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe40f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c0499c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42876/0 (socket says 172.21.15.16:42876) 2024-08-28T19:37:00.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe40f7fe640 1 -- 172.21.15.16:0/961276497 learned_addr learned my addr 172.21.15.16:0/961276497 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe414ac9640 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045e7070 0x561c049a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe414ac9640 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045be1b0 0x561c0499bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe40f7fe640 1 -- 172.21.15.16:0/961276497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045be1b0 msgr2=0x561c0499bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe40f7fe640 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045be1b0 0x561c0499bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe40f7fe640 1 -- 172.21.15.16:0/961276497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045e7070 msgr2=0x561c049a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe40f7fe640 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045e7070 0x561c049a81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.791+0000 7fe40f7fe640 1 -- 172.21.15.16:0/961276497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c048b2900 con 0x561c045c6390 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe40f7fe640 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c0499c450 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fe404009900 tx=0x7fe404031f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c3754e39877d9b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe3ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/961276497 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe404040070 con 0x561c045c6390 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/961276497 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7fe404004040 con 0x561c045c6390 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe41caa1740 1 -- 172.21.15.16:0/961276497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4080183f0 con 0x561c045c6390 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/961276497 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fe40404d050 con 0x561c045c6390 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe41caa1740 10 client.19401 fetch_fsmap learned FSMap version 82 2024-08-28T19:37:00.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe41caa1740 1 -- 172.21.15.16:0/961276497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x561c049aacb0 con 0x561c045c6390 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/961276497 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4080183f0 con 0x561c045c6390 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/961276497 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7fe40403c440 con 0x561c045c6390 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe41caa1740 10 client.19401 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe41caa1740 10 client.19401 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe41caa1740 20 client.19401 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.792+0000 7fe41caa1740 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561c045a59f0 0x561c049ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.793+0000 7fe41caa1740 4 client.19401 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.793+0000 7fe41caa1740 1 -- 172.21.15.16:0/961276497 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c049055d0 con 0x561c045a59f0 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.793+0000 7fe414ac9640 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561c045a59f0 0x561c049ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.793+0000 7fe3ef7fe640 10 client.19401 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.793+0000 7fe414ac9640 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561c045a59f0 0x561c049ae2c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2fe09656c3daae8f server_cookie=5ad1e91d2af8dec9 in_seq=0 out_seq=0 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/961276497 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561c049055d0 con 0x561c045a59f0 2024-08-28T19:37:00.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe3ef7fe640 10 client.19401 handle_command_reply: tid=0 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 1 client.19401 shutdown 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 1 -- 172.21.15.16:0/961276497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 msgr2=0x561c0499c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c0499c450 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fe404009900 tx=0x7fe404031f50 comp rx=0 tx=0).stop 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 1 -- 172.21.15.16:0/961276497 shutdown_connections 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561c045a59f0 0x561c049ae2c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045e7070 0x561c049a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c0499c450 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 1 --2- 172.21.15.16:0/961276497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045be1b0 0x561c0499bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 1 -- 172.21.15.16:0/961276497 >> 172.21.15.16:0/961276497 conn(0x561c045db460 msgr2=0x561c04988800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe3ef7fe640 0 client.19401 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 1 -- 172.21.15.16:0/961276497 shutdown_connections 2024-08-28T19:37:00.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 1 -- 172.21.15.16:0/961276497 wait complete. 2024-08-28T19:37:00.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.794+0000 7fe41caa1740 20 client.19401 trim_cache size 0 max 16384 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.807+0000 7fe41caa1740 1 Processor -- start 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.807+0000 7fe41caa1740 1 -- start start 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045be1b0 0x561c0499c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c049a80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045e7070 0x561c049a8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe41caa1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4101aea80 con 0x561c045c6390 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe41caa1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4101b0800 con 0x561c045e7070 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe41caa1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe4101af180 con 0x561c045be1b0 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe40f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c049a80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe40f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c049a80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42890/0 (socket says 172.21.15.16:42890) 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe40f7fe640 1 -- 172.21.15.16:0/3651937601 learned_addr learned my addr 172.21.15.16:0/3651937601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe414ac9640 1 --2- 172.21.15.16:0/3651937601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045e7070 0x561c049a8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe414ac9640 1 --2- 172.21.15.16:0/3651937601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045be1b0 0x561c0499c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe40f7fe640 1 -- 172.21.15.16:0/3651937601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045be1b0 msgr2=0x561c0499c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe40f7fe640 1 --2- 172.21.15.16:0/3651937601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045be1b0 0x561c0499c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe40f7fe640 1 -- 172.21.15.16:0/3651937601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045e7070 msgr2=0x561c049a8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe40f7fe640 1 --2- 172.21.15.16:0/3651937601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045e7070 0x561c049a8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe40f7fe640 1 -- 172.21.15.16:0/3651937601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c0499a410 con 0x561c045c6390 2024-08-28T19:37:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe414ac9640 1 --2- 172.21.15.16:0/3651937601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045be1b0 0x561c0499c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:37:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.808+0000 7fe40f7fe640 1 --2- 172.21.15.16:0/3651937601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c049a80c0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fe40402a930 tx=0x7fe40402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6efc0c4f0f646f7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.809+0000 7fe3f5ffb640 1 -- 172.21.15.16:0/3651937601 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe404055020 con 0x561c045c6390 2024-08-28T19:37:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.809+0000 7fe3f5ffb640 1 -- 172.21.15.16:0/3651937601 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe404031870 con 0x561c045c6390 2024-08-28T19:37:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.809+0000 7fe3f5ffb640 1 -- 172.21.15.16:0/3651937601 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40404a040 con 0x561c045c6390 2024-08-28T19:37:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.809+0000 7fe41caa1740 1 -- 172.21.15.16:0/3651937601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 msgr2=0x561c049a80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.809+0000 7fe41caa1740 1 --2- 172.21.15.16:0/3651937601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c049a80c0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fe40402a930 tx=0x7fe40402a410 comp rx=0 tx=0).stop 2024-08-28T19:37:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.809+0000 7fe41caa1740 1 -- 172.21.15.16:0/3651937601 shutdown_connections 2024-08-28T19:37:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.809+0000 7fe41caa1740 1 --2- 172.21.15.16:0/3651937601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045e7070 0x561c049a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.809+0000 7fe41caa1740 1 --2- 172.21.15.16:0/3651937601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045c6390 0x561c049a80c0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.809+0000 7fe41caa1740 1 --2- 172.21.15.16:0/3651937601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045be1b0 0x561c0499c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.809+0000 7fe41caa1740 1 -- 172.21.15.16:0/3651937601 >> 172.21.15.16:0/3651937601 conn(0x561c045db460 msgr2=0x561c0499bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.810+0000 7fe41caa1740 1 -- 172.21.15.16:0/3651937601 shutdown_connections 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.810+0000 7fe41caa1740 1 -- 172.21.15.16:0/3651937601 wait complete. 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.810+0000 7fe41caa1740 1 -- start start 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.810+0000 7fe41caa1740 1 Processor -- start 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.810+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 0x561c049a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.810+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045c6390 0x561c049a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.810+0000 7fe41caa1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 0x561c049a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.810+0000 7fe41caa1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4101b0070 con 0x561c045e7070 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.810+0000 7fe41caa1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4101af8e0 con 0x561c045be1b0 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.810+0000 7fe41caa1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe4101aca80 con 0x561c045c6390 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe41caa1740 10 client.0 _ll_register_callbacks cb 0x7fe404009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe414ac9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 0x561c049a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe414ac9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 0x561c049a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49942/0 (socket says 172.21.15.16:49942) 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe414ac9640 1 -- 172.21.15.16:0/656446815 learned_addr learned my addr 172.21.15.16:0/656446815 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe40f7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045c6390 0x561c049a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe414ac9640 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 0x561c049a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe414ac9640 1 -- 172.21.15.16:0/656446815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045c6390 msgr2=0x561c049a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe414ac9640 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045c6390 0x561c049a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe414ac9640 1 -- 172.21.15.16:0/656446815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 msgr2=0x561c049a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe414ac9640 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 0x561c049a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe414ac9640 1 -- 172.21.15.16:0/656446815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c04a17d00 con 0x561c045be1b0 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe40f7fe640 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045c6390 0x561c049a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe414ac9640 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 0x561c049a0180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe408018cc0 tx=0x7fe404000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a44cd0db4d172606 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe3ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/656446815 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe408018820 con 0x561c045be1b0 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/656446815 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7fe408003970 con 0x561c045be1b0 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.811+0000 7fe41caa1740 1 -- 172.21.15.16:0/656446815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe404031870 con 0x561c045be1b0 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/656446815 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fe40803f050 con 0x561c045be1b0 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe41caa1740 10 client.28826 fetch_fsmap learned FSMap version 82 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe41caa1740 1 -- 172.21.15.16:0/656446815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x561c04a18f80 con 0x561c045be1b0 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/656446815 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe404031870 con 0x561c045be1b0 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/656446815 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7fe408024040 con 0x561c045be1b0 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe41caa1740 10 client.28826 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe41caa1740 10 client.28826 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe41caa1740 20 client.28826 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe41caa1740 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561c0457ada0 0x561c04a1b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe41caa1740 4 client.28826 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe41caa1740 1 -- 172.21.15.16:0/656446815 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561c04901070 con 0x561c0457ada0 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.812+0000 7fe40f7fe640 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561c0457ada0 0x561c04a1b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe3ef7fe640 10 client.28826 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe40f7fe640 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561c0457ada0 0x561c04a1b590 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=72cadf537eee06a1 server_cookie=feb407433fc57b69 in_seq=0 out_seq=0 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe3ef7fe640 1 -- 172.21.15.16:0/656446815 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561c04901070 con 0x561c0457ada0 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe3ef7fe640 10 client.28826 handle_command_reply: tid=0 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 1 client.28826 shutdown 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 1 -- 172.21.15.16:0/656446815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 msgr2=0x561c049a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 0x561c049a0180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe408018cc0 tx=0x7fe404000bc0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 1 -- 172.21.15.16:0/656446815 shutdown_connections 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x561c0457ada0 0x561c04a1b590 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c045e7070 0x561c049a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c045c6390 0x561c049a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe3ef7fe640 0 client.28826 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 1 --2- 172.21.15.16:0/656446815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c045be1b0 0x561c049a0180 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 1 -- 172.21.15.16:0/656446815 >> 172.21.15.16:0/656446815 conn(0x561c045db460 msgr2=0x561c04989a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 1 -- 172.21.15.16:0/656446815 shutdown_connections 2024-08-28T19:37:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 1 -- 172.21.15.16:0/656446815 wait complete. 2024-08-28T19:37:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.813+0000 7fe41caa1740 20 client.28826 trim_cache size 0 max 16384 2024-08-28T19:37:00.820 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:37:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 -- 172.21.15.16:0/841936943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3f80b4cd0 msgr2=0x7fe3f80b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3f80b4cd0 0x7fe3f80b71e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe4000e2db0 tx=0x7fe4000eb530 comp rx=0 tx=0).stop 2024-08-28T19:37:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 -- 172.21.15.16:0/841936943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410102880 msgr2=0x7fe41019b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:00.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410102880 0x7fe41019b110 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fe40000c920 tx=0x7fe40000cde0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 -- 172.21.15.16:0/841936943 shutdown_connections 2024-08-28T19:37:00.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe3f80b4cd0 0x7fe3f80b71e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe410108b10 0x7fe41019bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4101081c0 0x7fe41019b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 --2- 172.21.15.16:0/841936943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410102880 0x7fe41019b110 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:00.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 -- 172.21.15.16:0/841936943 >> 172.21.15.16:0/841936943 conn(0x7fe410079510 msgr2=0x7fe4101062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:00.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 -- 172.21.15.16:0/841936943 shutdown_connections 2024-08-28T19:37:00.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:00.814+0000 7fe41652c640 1 -- 172.21.15.16:0/841936943 wait complete. 2024-08-28T19:37:01.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:00 smithi016 ceph-mon[18556]: pgmap v836: 129 pgs: 129 active+clean; 938 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.4 MiB/s wr, 294 op/s 2024-08-28T19:37:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:00 smithi028 ceph-mon[21769]: pgmap v836: 129 pgs: 129 active+clean; 938 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.4 MiB/s wr, 294 op/s 2024-08-28T19:37:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:00 smithi115 ceph-mon[22184]: pgmap v836: 129 pgs: 129 active+clean; 938 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.4 MiB/s wr, 294 op/s 2024-08-28T19:37:01.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 43657 7.84 MB/sec execute 37 sec latency 3118.918 ms 2024-08-28T19:37:01.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:37:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:02.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.382+0000 7f1d333fd640 1 -- 172.21.15.16:0/532776665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c107fd0 msgr2=0x7f1d2c1083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:02.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.382+0000 7f1d333fd640 1 --2- 172.21.15.16:0/532776665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c107fd0 0x7f1d2c1083e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009980 tx=0x7f1d1c02a420 comp rx=0 tx=0).stop 2024-08-28T19:37:02.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.382+0000 7f1d333fd640 1 -- 172.21.15.16:0/532776665 shutdown_connections 2024-08-28T19:37:02.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.382+0000 7f1d333fd640 1 --2- 172.21.15.16:0/532776665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c108920 0x7f1d2c10f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.382+0000 7f1d333fd640 1 --2- 172.21.15.16:0/532776665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c107fd0 0x7f1d2c1083e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.382+0000 7f1d333fd640 1 --2- 172.21.15.16:0/532776665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d2c102690 0x7f1d2c102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.382+0000 7f1d333fd640 1 -- 172.21.15.16:0/532776665 >> 172.21.15.16:0/532776665 conn(0x7f1d2c0fe120 msgr2=0x7f1d2c100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.382+0000 7f1d333fd640 1 -- 172.21.15.16:0/532776665 shutdown_connections 2024-08-28T19:37:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.382+0000 7f1d333fd640 1 -- 172.21.15.16:0/532776665 wait complete. 2024-08-28T19:37:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.383+0000 7f1d333fd640 1 Processor -- start 2024-08-28T19:37:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.384+0000 7f1d333fd640 1 -- start start 2024-08-28T19:37:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.384+0000 7f1d333fd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c102690 0x7f1d2c074c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.384+0000 7f1d333fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c107fd0 0x7f1d2c06df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.384+0000 7f1d333fd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d2c108920 0x7f1d2c06e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.384+0000 7f1d333fd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c1114e0 con 0x7f1d2c102690 2024-08-28T19:37:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.384+0000 7f1d333fd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c111360 con 0x7f1d2c107fd0 2024-08-28T19:37:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.384+0000 7f1d333fd640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c111660 con 0x7f1d2c108920 2024-08-28T19:37:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.385+0000 7f1d3199a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d2c108920 0x7f1d2c06e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.385+0000 7f1d31199640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c102690 0x7f1d2c074c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.385+0000 7f1d3199a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d2c108920 0x7f1d2c06e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34826/0 (socket says 172.21.15.16:34826) 2024-08-28T19:37:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.385+0000 7f1d3199a640 1 -- 172.21.15.16:0/79125706 learned_addr learned my addr 172.21.15.16:0/79125706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.385+0000 7f1d30998640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c107fd0 0x7f1d2c06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.386+0000 7f1d31199640 1 -- 172.21.15.16:0/79125706 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d2c108920 msgr2=0x7f1d2c06e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.386+0000 7f1d31199640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d2c108920 0x7f1d2c06e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.386+0000 7f1d31199640 1 -- 172.21.15.16:0/79125706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c107fd0 msgr2=0x7f1d2c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.386+0000 7f1d31199640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c107fd0 0x7f1d2c06df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.386+0000 7f1d31199640 1 -- 172.21.15.16:0/79125706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2c06ecb0 con 0x7f1d2c102690 2024-08-28T19:37:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.386+0000 7f1d31199640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c102690 0x7f1d2c074c60 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800d6d0 tx=0x7f1d2800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.387+0000 7f1d227fc640 1 -- 172.21.15.16:0/79125706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28017070 con 0x7f1d2c102690 2024-08-28T19:37:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.387+0000 7f1d333fd640 1 -- 172.21.15.16:0/79125706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d2c073930 con 0x7f1d2c102690 2024-08-28T19:37:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.387+0000 7f1d333fd640 1 -- 172.21.15.16:0/79125706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d2c073e70 con 0x7f1d2c102690 2024-08-28T19:37:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.387+0000 7f1d227fc640 1 -- 172.21.15.16:0/79125706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d28009820 con 0x7f1d2c102690 2024-08-28T19:37:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.387+0000 7f1d227fc640 1 -- 172.21.15.16:0/79125706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28004c40 con 0x7f1d2c102690 2024-08-28T19:37:02.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.393+0000 7f1d227fc640 1 -- 172.21.15.16:0/79125706 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1d28010460 con 0x7f1d2c102690 2024-08-28T19:37:02.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.394+0000 7f1d39972740 1 Processor -- start 2024-08-28T19:37:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.395+0000 7f1d227fc640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d180b4e70 0x7f1d180b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.395+0000 7f1d227fc640 1 -- 172.21.15.16:0/79125706 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f1d280e6110 con 0x7f1d2c102690 2024-08-28T19:37:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.396+0000 7f1d39972740 1 -- start start 2024-08-28T19:37:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.396+0000 7f1d3199a640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d180b4e70 0x7f1d180b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.396+0000 7f1d39972740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a344f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.396+0000 7f1d39972740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5629986661b0 0x562998a34a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.396+0000 7f1d39972740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56299866e390 0x562998a34f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.396+0000 7f1d39972740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5629989ab680 con 0x56299866e390 2024-08-28T19:37:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.396+0000 7f1d39972740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56299899ac10 con 0x56299868f070 2024-08-28T19:37:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.396+0000 7f1d39972740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56299899ebc0 con 0x5629986661b0 2024-08-28T19:37:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.397+0000 7f1d31199640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.397+0000 7f1d31199640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a344f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49986/0 (socket says 172.21.15.16:49986) 2024-08-28T19:37:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.397+0000 7f1d31199640 1 -- 172.21.15.16:0/2862914464 learned_addr learned my addr 172.21.15.16:0/2862914464 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.397+0000 7f1d30998640 1 --2- 172.21.15.16:0/2862914464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5629986661b0 0x562998a34a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.397+0000 7f1d31199640 1 -- 172.21.15.16:0/2862914464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5629986661b0 msgr2=0x562998a34a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.397+0000 7f1d31199640 1 --2- 172.21.15.16:0/2862914464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5629986661b0 0x562998a34a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.397+0000 7f1d31199640 1 -- 172.21.15.16:0/2862914464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56299866e390 msgr2=0x562998a34f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.397+0000 7f1d31199640 1 --2- 172.21.15.16:0/2862914464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56299866e390 0x562998a34f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.397+0000 7f1d31199640 1 -- 172.21.15.16:0/2862914464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629986fa650 con 0x56299868f070 2024-08-28T19:37:02.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.398+0000 7f1d31199640 1 --2- 172.21.15.16:0/2862914464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a344f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009610 tx=0x7f1d280acad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f91dfa2e59b3ee9c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:02.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.398+0000 7f1d3199a640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d180b4e70 0x7f1d180b7380 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1d24013440 tx=0x7f1d2400b9f0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:02.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.399+0000 7f1d177fe640 1 -- 172.21.15.16:0/2862914464 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28020050 con 0x56299868f070 2024-08-28T19:37:02.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.399+0000 7f1d177fe640 1 -- 172.21.15.16:0/2862914464 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d280b56f0 con 0x56299868f070 2024-08-28T19:37:02.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.400+0000 7f1d177fe640 1 -- 172.21.15.16:0/2862914464 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d280e3b80 con 0x56299868f070 2024-08-28T19:37:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.400+0000 7f1d39972740 1 -- 172.21.15.16:0/2862914464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 msgr2=0x562998a344f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.400+0000 7f1d39972740 1 --2- 172.21.15.16:0/2862914464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a344f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009610 tx=0x7f1d280acad0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.400+0000 7f1d39972740 1 -- 172.21.15.16:0/2862914464 shutdown_connections 2024-08-28T19:37:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.400+0000 7f1d39972740 1 --2- 172.21.15.16:0/2862914464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56299866e390 0x562998a34f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.400+0000 7f1d39972740 1 --2- 172.21.15.16:0/2862914464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5629986661b0 0x562998a34a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.400+0000 7f1d39972740 1 --2- 172.21.15.16:0/2862914464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a344f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.400+0000 7f1d39972740 1 -- 172.21.15.16:0/2862914464 >> 172.21.15.16:0/2862914464 conn(0x562998683460 msgr2=0x562998a323e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.400+0000 7f1d39972740 1 -- 172.21.15.16:0/2862914464 shutdown_connections 2024-08-28T19:37:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.400+0000 7f1d39972740 1 -- 172.21.15.16:0/2862914464 wait complete. 2024-08-28T19:37:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.401+0000 7f1d39972740 1 -- start start 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.401+0000 7f1d39972740 1 Processor -- start 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.402+0000 7f1d39972740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5629986661b0 0x562998a3fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.402+0000 7f1d39972740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56299866e390 0x562998a40120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.402+0000 7f1d39972740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a4c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.402+0000 7f1d39972740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5629987fd700 con 0x56299866e390 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.402+0000 7f1d39972740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562998921750 con 0x56299868f070 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.402+0000 7f1d39972740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562998a42c60 con 0x5629986661b0 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.402+0000 7f1d39972740 10 client.0 _ll_register_callbacks cb 0x562998950e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.402+0000 7f1d3199a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a4c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.402+0000 7f1d30998640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56299866e390 0x562998a40120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.402+0000 7f1d3199a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a4c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49988/0 (socket says 172.21.15.16:49988) 2024-08-28T19:37:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.402+0000 7f1d3199a640 1 -- 172.21.15.16:0/878463290 learned_addr learned my addr 172.21.15.16:0/878463290 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:02.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.403+0000 7f1d31199640 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5629986661b0 0x562998a3fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.403+0000 7f1d30998640 1 -- 172.21.15.16:0/878463290 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5629986661b0 msgr2=0x562998a3fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.403+0000 7f1d30998640 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5629986661b0 0x562998a3fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.403+0000 7f1d30998640 1 -- 172.21.15.16:0/878463290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 msgr2=0x562998a4c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.403+0000 7f1d30998640 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a4c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.403+0000 7f1d30998640 1 -- 172.21.15.16:0/878463290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56299895a900 con 0x56299866e390 2024-08-28T19:37:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.403+0000 7f1d30998640 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56299866e390 0x562998a40120 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009230 tx=0x7f1d1c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b737ff7c9dfa5f8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.403+0000 7f1d14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:37:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.404+0000 7f1d14ff9640 1 -- 172.21.15.16:0/878463290 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c035020 con 0x56299866e390 2024-08-28T19:37:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.404+0000 7f1d14ff9640 1 -- 172.21.15.16:0/878463290 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f1d1c0316d0 con 0x56299866e390 2024-08-28T19:37:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.404+0000 7f1d39972740 1 -- 172.21.15.16:0/878463290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d280b56f0 con 0x56299866e390 2024-08-28T19:37:02.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.404+0000 7f1d14ff9640 1 -- 172.21.15.16:0/878463290 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f1d1c005df0 con 0x56299866e390 2024-08-28T19:37:02.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.404+0000 7f1d39972740 10 client.19413 fetch_fsmap learned FSMap version 82 2024-08-28T19:37:02.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.404+0000 7f1d39972740 1 -- 172.21.15.16:0/878463290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x562998a4e9f0 con 0x56299866e390 2024-08-28T19:37:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.405+0000 7f1d14ff9640 1 -- 172.21.15.16:0/878463290 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d280b56f0 con 0x56299866e390 2024-08-28T19:37:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.405+0000 7f1d14ff9640 1 -- 172.21.15.16:0/878463290 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7f1d1c03a030 con 0x56299866e390 2024-08-28T19:37:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.405+0000 7f1d39972740 10 client.19413 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:37:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.405+0000 7f1d39972740 10 client.19413 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:37:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.405+0000 7f1d39972740 20 client.19413 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.405+0000 7f1d39972740 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56299864d9f0 0x562998a51f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.405+0000 7f1d39972740 4 client.19413 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:37:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.405+0000 7f1d39972740 1 -- 172.21.15.16:0/878463290 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629989ad5d0 con 0x56299864d9f0 2024-08-28T19:37:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.406+0000 7f1d31199640 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56299864d9f0 0x562998a51f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.407+0000 7f1d31199640 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56299864d9f0 0x562998a51f90 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=65ebfafbb7a6effd server_cookie=df612a3fcab86c1f in_seq=0 out_seq=0 2024-08-28T19:37:02.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.407+0000 7f1d14ff9640 10 client.19413 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:37:02.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.408+0000 7f1d14ff9640 1 -- 172.21.15.16:0/878463290 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5629989ad5d0 con 0x56299864d9f0 2024-08-28T19:37:02.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.408+0000 7f1d14ff9640 10 client.19413 handle_command_reply: tid=0 2024-08-28T19:37:02.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.408+0000 7f1d39972740 1 client.19413 shutdown 2024-08-28T19:37:02.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.409+0000 7f1d39972740 1 -- 172.21.15.16:0/878463290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56299866e390 msgr2=0x562998a40120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:02.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.409+0000 7f1d39972740 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56299866e390 0x562998a40120 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009230 tx=0x7f1d1c004270 comp rx=0 tx=0).stop 2024-08-28T19:37:02.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.409+0000 7f1d39972740 1 -- 172.21.15.16:0/878463290 shutdown_connections 2024-08-28T19:37:02.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.409+0000 7f1d39972740 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56299864d9f0 0x562998a51f90 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.409+0000 7f1d39972740 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a4c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.409+0000 7f1d39972740 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56299866e390 0x562998a40120 secure :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009230 tx=0x7f1d1c004270 comp rx=0 tx=0).stop 2024-08-28T19:37:02.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.409+0000 7f1d39972740 1 --2- 172.21.15.16:0/878463290 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5629986661b0 0x562998a3fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.409+0000 7f1d39972740 1 -- 172.21.15.16:0/878463290 >> 172.21.15.16:0/878463290 conn(0x562998683460 msgr2=0x562998a37680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:02.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.409+0000 7f1d14ff9640 0 client.19413 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:37:02.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.410+0000 7f1d39972740 1 -- 172.21.15.16:0/878463290 shutdown_connections 2024-08-28T19:37:02.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.410+0000 7f1d39972740 1 -- 172.21.15.16:0/878463290 wait complete. 2024-08-28T19:37:02.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.410+0000 7f1d39972740 20 client.19413 trim_cache size 0 max 16384 2024-08-28T19:37:02.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.437+0000 7f1d39972740 1 Processor -- start 2024-08-28T19:37:02.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.437+0000 7f1d39972740 1 -- start start 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.438+0000 7f1d39972740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629986661b0 0x562998a3eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.438+0000 7f1d39972740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a4bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.438+0000 7f1d39972740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562998622da0 0x562998a3cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.438+0000 7f1d30998640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a4bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.438+0000 7f1d30998640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a4bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:49998/0 (socket says 172.21.15.16:49998) 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.438+0000 7f1d30998640 1 -- 172.21.15.16:0/1071448969 learned_addr learned my addr 172.21.15.16:0/1071448969 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.438+0000 7f1d39972740 1 -- 172.21.15.16:0/1071448969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1d100100b0 con 0x5629986661b0 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.438+0000 7f1d39972740 1 -- 172.21.15.16:0/1071448969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d10011e30 con 0x56299868f070 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.438+0000 7f1d39972740 1 -- 172.21.15.16:0/1071448969 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1d100107b0 con 0x562998622da0 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.438+0000 7f1d3199a640 1 --2- 172.21.15.16:0/1071448969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562998622da0 0x562998a3cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.439+0000 7f1d31199640 1 --2- 172.21.15.16:0/1071448969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629986661b0 0x562998a3eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.439+0000 7f1d30998640 1 -- 172.21.15.16:0/1071448969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562998622da0 msgr2=0x562998a3cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.439+0000 7f1d30998640 1 --2- 172.21.15.16:0/1071448969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562998622da0 0x562998a3cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.439+0000 7f1d30998640 1 -- 172.21.15.16:0/1071448969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629986661b0 msgr2=0x562998a3eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:02.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.439+0000 7f1d30998640 1 --2- 172.21.15.16:0/1071448969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629986661b0 0x562998a3eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.439+0000 7f1d30998640 1 -- 172.21.15.16:0/1071448969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562998a36d80 con 0x56299868f070 2024-08-28T19:37:02.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.439+0000 7f1d30998640 1 --2- 172.21.15.16:0/1071448969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a4bf00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c004ec0 tx=0x7f1d1c004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5431b54dc626af82 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.440+0000 7f1d16ffd640 1 -- 172.21.15.16:0/1071448969 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c005df0 con 0x56299868f070 2024-08-28T19:37:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.440+0000 7f1d16ffd640 1 -- 172.21.15.16:0/1071448969 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d1c041070 con 0x56299868f070 2024-08-28T19:37:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.440+0000 7f1d16ffd640 1 -- 172.21.15.16:0/1071448969 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c031d40 con 0x56299868f070 2024-08-28T19:37:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.441+0000 7f1d39972740 1 -- 172.21.15.16:0/1071448969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 msgr2=0x562998a4bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.441+0000 7f1d39972740 1 --2- 172.21.15.16:0/1071448969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a4bf00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c004ec0 tx=0x7f1d1c004360 comp rx=0 tx=0).stop 2024-08-28T19:37:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.442+0000 7f1d39972740 1 -- 172.21.15.16:0/1071448969 shutdown_connections 2024-08-28T19:37:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.442+0000 7f1d39972740 1 --2- 172.21.15.16:0/1071448969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562998622da0 0x562998a3cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.442+0000 7f1d39972740 1 --2- 172.21.15.16:0/1071448969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a4bf00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.442+0000 7f1d39972740 1 --2- 172.21.15.16:0/1071448969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629986661b0 0x562998a3eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.442+0000 7f1d39972740 1 -- 172.21.15.16:0/1071448969 >> 172.21.15.16:0/1071448969 conn(0x562998683460 msgr2=0x562998a3e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.442+0000 7f1d39972740 1 -- 172.21.15.16:0/1071448969 shutdown_connections 2024-08-28T19:37:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.442+0000 7f1d39972740 1 -- 172.21.15.16:0/1071448969 wait complete. 2024-08-28T19:37:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.442+0000 7f1d39972740 1 -- start start 2024-08-28T19:37:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.443+0000 7f1d39972740 1 Processor -- start 2024-08-28T19:37:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.443+0000 7f1d39972740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562998622da0 0x562998a488e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.444+0000 7f1d39972740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629986661b0 0x562998a34370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.444+0000 7f1d39972740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a34a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.444+0000 7f1d39972740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1d100116a0 con 0x5629986661b0 2024-08-28T19:37:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.444+0000 7f1d39972740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d10010f10 con 0x56299868f070 2024-08-28T19:37:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.444+0000 7f1d39972740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1000e0b0 con 0x562998622da0 2024-08-28T19:37:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.444+0000 7f1d3199a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a34a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.444+0000 7f1d3199a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a34a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:50012/0 (socket says 172.21.15.16:50012) 2024-08-28T19:37:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.444+0000 7f1d3199a640 1 -- 172.21.15.16:0/220549448 learned_addr learned my addr 172.21.15.16:0/220549448 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.444+0000 7f1d39972740 10 client.0 _ll_register_callbacks cb 0x7f1d1c0097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.445+0000 7f1d30998640 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629986661b0 0x562998a34370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.445+0000 7f1d3199a640 1 -- 172.21.15.16:0/220549448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562998622da0 msgr2=0x562998a488e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:37:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.445+0000 7f1d3199a640 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562998622da0 0x562998a488e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.445+0000 7f1d3199a640 1 -- 172.21.15.16:0/220549448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629986661b0 msgr2=0x562998a34370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.445+0000 7f1d3199a640 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629986661b0 0x562998a34370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.445+0000 7f1d3199a640 1 -- 172.21.15.16:0/220549448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562998a47520 con 0x56299868f070 2024-08-28T19:37:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.445+0000 7f1d3199a640 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a34a60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1d2401ae50 tx=0x7f1d24004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f7cbe3f2f6b47df server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.445+0000 7f1d14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:02.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.447+0000 7f1d14ff9640 1 -- 172.21.15.16:0/220549448 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2402a060 con 0x56299868f070 2024-08-28T19:37:02.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.447+0000 7f1d14ff9640 1 -- 172.21.15.16:0/220549448 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7f1d24014040 con 0x56299868f070 2024-08-28T19:37:02.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.447+0000 7f1d39972740 1 -- 172.21.15.16:0/220549448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d1c041070 con 0x56299868f070 2024-08-28T19:37:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.448+0000 7f1d14ff9640 1 -- 172.21.15.16:0/220549448 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f1d24019e50 con 0x56299868f070 2024-08-28T19:37:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.448+0000 7f1d39972740 10 client.28841 fetch_fsmap learned FSMap version 82 2024-08-28T19:37:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.448+0000 7f1d39972740 1 -- 172.21.15.16:0/220549448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x562998abcae0 con 0x56299868f070 2024-08-28T19:37:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.448+0000 7f1d14ff9640 1 -- 172.21.15.16:0/220549448 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d24018b00 con 0x56299868f070 2024-08-28T19:37:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.448+0000 7f1d14ff9640 1 -- 172.21.15.16:0/220549448 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7f1d240258e0 con 0x56299868f070 2024-08-28T19:37:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.449+0000 7f1d39972740 10 client.28841 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:37:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.449+0000 7f1d39972740 10 client.28841 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:37:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.449+0000 7f1d39972740 20 client.28841 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.449+0000 7f1d39972740 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562998abd380 0x562998abf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.449+0000 7f1d39972740 4 client.28841 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:37:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.449+0000 7f1d39972740 1 -- 172.21.15.16:0/220549448 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f1d1c041070 con 0x562998abd380 2024-08-28T19:37:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.449+0000 7f1d31199640 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562998abd380 0x562998abf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:02.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.451+0000 7f1d31199640 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562998abd380 0x562998abf7a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d45d7695b6b673af server_cookie=61df9abeb6d3e334 in_seq=0 out_seq=0 2024-08-28T19:37:02.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.451+0000 7f1d14ff9640 10 client.28841 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:37:02.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.451+0000 7f1d14ff9640 1 -- 172.21.15.16:0/220549448 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f1d1c041070 con 0x562998abd380 2024-08-28T19:37:02.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.451+0000 7f1d14ff9640 10 client.28841 handle_command_reply: tid=0 2024-08-28T19:37:02.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.451+0000 7f1d39972740 1 client.28841 shutdown 2024-08-28T19:37:02.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d39972740 1 -- 172.21.15.16:0/220549448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 msgr2=0x562998a34a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:02.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d39972740 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a34a60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1d2401ae50 tx=0x7f1d24004a40 comp rx=0 tx=0).stop 2024-08-28T19:37:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d39972740 1 -- 172.21.15.16:0/220549448 shutdown_connections 2024-08-28T19:37:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d39972740 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x562998abd380 0x562998abf7a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d39972740 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56299868f070 0x562998a34a60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d39972740 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629986661b0 0x562998a34370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d14ff9640 0 client.28841 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:37:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d39972740 1 --2- 172.21.15.16:0/220549448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562998622da0 0x562998a488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d39972740 1 -- 172.21.15.16:0/220549448 >> 172.21.15.16:0/220549448 conn(0x562998683460 msgr2=0x562998a35520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d39972740 1 -- 172.21.15.16:0/220549448 shutdown_connections 2024-08-28T19:37:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d39972740 1 -- 172.21.15.16:0/220549448 wait complete. 2024-08-28T19:37:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.452+0000 7f1d39972740 20 client.28841 trim_cache size 0 max 16384 2024-08-28T19:37:02.455 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:37:02.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.453+0000 7f1d333fd640 1 -- 172.21.15.16:0/79125706 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d180b4e70 msgr2=0x7f1d180b7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:02.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.453+0000 7f1d333fd640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d180b4e70 0x7f1d180b7380 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1d24013440 tx=0x7f1d2400b9f0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.453+0000 7f1d333fd640 1 -- 172.21.15.16:0/79125706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c102690 msgr2=0x7f1d2c074c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:02.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.453+0000 7f1d333fd640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c102690 0x7f1d2c074c60 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800d6d0 tx=0x7f1d2800db90 comp rx=0 tx=0).stop 2024-08-28T19:37:02.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.453+0000 7f1d333fd640 1 -- 172.21.15.16:0/79125706 shutdown_connections 2024-08-28T19:37:02.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.453+0000 7f1d333fd640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d2c108920 0x7f1d2c06e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.453+0000 7f1d333fd640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1d180b4e70 0x7f1d180b7380 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.453+0000 7f1d333fd640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c107fd0 0x7f1d2c06df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.453+0000 7f1d333fd640 1 --2- 172.21.15.16:0/79125706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c102690 0x7f1d2c074c60 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:02.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.453+0000 7f1d333fd640 1 -- 172.21.15.16:0/79125706 >> 172.21.15.16:0/79125706 conn(0x7f1d2c0fe120 msgr2=0x7f1d2c100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:02.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.454+0000 7f1d333fd640 1 -- 172.21.15.16:0/79125706 shutdown_connections 2024-08-28T19:37:02.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:02.454+0000 7f1d333fd640 1 -- 172.21.15.16:0/79125706 wait complete. 2024-08-28T19:37:02.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 43936 7.88 MB/sec execute 38 sec latency 4118.973 ms 2024-08-28T19:37:03.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:02 smithi115 ceph-mon[22184]: pgmap v837: 129 pgs: 129 active+clean; 954 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 6.1 MiB/s wr, 474 op/s 2024-08-28T19:37:03.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:02 smithi016 ceph-mon[18556]: pgmap v837: 129 pgs: 129 active+clean; 954 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 6.1 MiB/s wr, 474 op/s 2024-08-28T19:37:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:02 smithi028 ceph-mon[21769]: pgmap v837: 129 pgs: 129 active+clean; 954 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 6.1 MiB/s wr, 474 op/s 2024-08-28T19:37:03.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 44054 7.93 MB/sec execute 39 sec latency 5119.012 ms 2024-08-28T19:37:04.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 44255 8.10 MB/sec execute 40 sec latency 6119.066 ms 2024-08-28T19:37:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:04 smithi016 ceph-mon[18556]: pgmap v838: 129 pgs: 129 active+clean; 958 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5.5 MiB/s wr, 486 op/s 2024-08-28T19:37:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:04 smithi028 ceph-mon[21769]: pgmap v838: 129 pgs: 129 active+clean; 958 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5.5 MiB/s wr, 486 op/s 2024-08-28T19:37:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:04 smithi115 ceph-mon[22184]: pgmap v838: 129 pgs: 129 active+clean; 958 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5.5 MiB/s wr, 486 op/s 2024-08-28T19:37:05.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:37:05.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 44731 8.41 MB/sec execute 41 sec latency 7119.099 ms 2024-08-28T19:37:05.740 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.253+0000 7fbb78671640 1 -- 172.21.15.16:0/731260100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb70102800 msgr2=0x7fbb70102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.253+0000 7fbb78671640 1 --2- 172.21.15.16:0/731260100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb70102800 0x7fbb70102c10 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009900 tx=0x7fbb6c02a3f0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.253+0000 7fbb78671640 1 -- 172.21.15.16:0/731260100 shutdown_connections 2024-08-28T19:37:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.253+0000 7fbb78671640 1 --2- 172.21.15.16:0/731260100 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb70108a90 0x7fbb7010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.253+0000 7fbb78671640 1 --2- 172.21.15.16:0/731260100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb70108140 0x7fbb70108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.253+0000 7fbb78671640 1 --2- 172.21.15.16:0/731260100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb70102800 0x7fbb70102c10 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.253+0000 7fbb78671640 1 -- 172.21.15.16:0/731260100 >> 172.21.15.16:0/731260100 conn(0x7fbb700794d0 msgr2=0x7fbb700798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.253+0000 7fbb78671640 1 -- 172.21.15.16:0/731260100 shutdown_connections 2024-08-28T19:37:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.253+0000 7fbb78671640 1 -- 172.21.15.16:0/731260100 wait complete. 2024-08-28T19:37:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.254+0000 7fbb78671640 1 Processor -- start 2024-08-28T19:37:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.254+0000 7fbb78671640 1 -- start start 2024-08-28T19:37:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.254+0000 7fbb78671640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb70102800 0x7fbb70103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.254+0000 7fbb78671640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb70108140 0x7fbb70104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb78671640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb70108a90 0x7fbb70104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb78671640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbb70112710 con 0x7fbb70108140 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb78671640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb70112590 con 0x7fbb70108a90 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb78671640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbb70112890 con 0x7fbb70102800 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb76c0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb70108a90 0x7fbb70104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb7640d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb70102800 0x7fbb70103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb7640d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb70102800 0x7fbb70103ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55452/0 (socket says 172.21.15.16:55452) 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb76c0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb70108a90 0x7fbb70104a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57596/0 (socket says 172.21.15.16:57596) 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb7640d640 1 -- 172.21.15.16:0/4155896879 learned_addr learned my addr 172.21.15.16:0/4155896879 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb75c0c640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb70108140 0x7fbb70104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb76c0e640 1 -- 172.21.15.16:0/4155896879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb70102800 msgr2=0x7fbb70103ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb76c0e640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb70102800 0x7fbb70103ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb76c0e640 1 -- 172.21.15.16:0/4155896879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb70108140 msgr2=0x7fbb70104530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb76c0e640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb70108140 0x7fbb70104530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.255+0000 7fbb76c0e640 1 -- 172.21.15.16:0/4155896879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb701a84b0 con 0x7fbb70108a90 2024-08-28T19:37:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.256+0000 7fbb76c0e640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb70108a90 0x7fbb70104a70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800ca30 tx=0x7fbb6800cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.256+0000 7fbb677fe640 1 -- 172.21.15.16:0/4155896879 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68007d90 con 0x7fbb70108a90 2024-08-28T19:37:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.256+0000 7fbb78671640 1 -- 172.21.15.16:0/4155896879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb701a87a0 con 0x7fbb70108a90 2024-08-28T19:37:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.256+0000 7fbb677fe640 1 -- 172.21.15.16:0/4155896879 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb68005020 con 0x7fbb70108a90 2024-08-28T19:37:06.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.256+0000 7fbb677fe640 1 -- 172.21.15.16:0/4155896879 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6800f670 con 0x7fbb70108a90 2024-08-28T19:37:06.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.256+0000 7fbb78671640 1 -- 172.21.15.16:0/4155896879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb701a8ce0 con 0x7fbb70108a90 2024-08-28T19:37:06.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.259+0000 7fbb7ebe6740 1 Processor -- start 2024-08-28T19:37:06.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.260+0000 7fbb677fe640 1 -- 172.21.15.16:0/4155896879 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fbb680051d0 con 0x7fbb70108a90 2024-08-28T19:37:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb7ebe6740 1 -- start start 2024-08-28T19:37:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 0x55d53300fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c391b0 0x55d533010060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c41390 0x55d5330105a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb7ebe6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d532f7e680 con 0x55d532c391b0 2024-08-28T19:37:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb7ebe6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d532f6dc10 con 0x55d532c62070 2024-08-28T19:37:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb7ebe6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d532f71bc0 con 0x55d532c41390 2024-08-28T19:37:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb75c0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 0x55d53300fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb75c0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 0x55d53300fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57604/0 (socket says 172.21.15.16:57604) 2024-08-28T19:37:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb75c0c640 1 -- 172.21.15.16:0/974817274 learned_addr learned my addr 172.21.15.16:0/974817274 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb75c0c640 1 --2- 172.21.15.16:0/974817274 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c41390 0x55d5330105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.261+0000 7fbb7640d640 1 --2- 172.21.15.16:0/974817274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c391b0 0x55d533010060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.262+0000 7fbb75c0c640 1 -- 172.21.15.16:0/974817274 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c41390 msgr2=0x55d5330105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.262+0000 7fbb75c0c640 1 --2- 172.21.15.16:0/974817274 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c41390 0x55d5330105a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.262+0000 7fbb75c0c640 1 -- 172.21.15.16:0/974817274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c391b0 msgr2=0x55d533010060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.262+0000 7fbb75c0c640 1 --2- 172.21.15.16:0/974817274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c391b0 0x55d533010060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.262+0000 7fbb75c0c640 1 -- 172.21.15.16:0/974817274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d532ccd650 con 0x55d532c62070 2024-08-28T19:37:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.262+0000 7fbb75c0c640 1 --2- 172.21.15.16:0/974817274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 0x55d53300fb20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbb600093e0 tx=0x7fbb60009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1acf9695fa4df08 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.263+0000 7fbb64ff9640 1 -- 172.21.15.16:0/974817274 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6000f790 con 0x55d532c62070 2024-08-28T19:37:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.263+0000 7fbb64ff9640 1 -- 172.21.15.16:0/974817274 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6000fde0 con 0x55d532c62070 2024-08-28T19:37:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.263+0000 7fbb677fe640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb5c0b4ca0 0x7fbb5c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.264+0000 7fbb677fe640 1 -- 172.21.15.16:0/4155896879 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7fbb680e5b20 con 0x7fbb70108a90 2024-08-28T19:37:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.264+0000 7fbb7640d640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb5c0b4ca0 0x7fbb5c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.264+0000 7fbb64ff9640 1 -- 172.21.15.16:0/974817274 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6001fd90 con 0x55d532c62070 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.264+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/974817274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 msgr2=0x55d53300fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.264+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/974817274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 0x55d53300fb20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbb600093e0 tx=0x7fbb60009370 comp rx=0 tx=0).stop 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.264+0000 7fbb7640d640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb5c0b4ca0 0x7fbb5c0b71b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009230 tx=0x7fbb6c0058d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.264+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/974817274 shutdown_connections 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.265+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/974817274 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c41390 0x55d5330105a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.265+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/974817274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c391b0 0x55d533010060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.265+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/974817274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 0x55d53300fb20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.265+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/974817274 >> 172.21.15.16:0/974817274 conn(0x55d532c56460 msgr2=0x55d53300da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.265+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/974817274 shutdown_connections 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.265+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/974817274 wait complete. 2024-08-28T19:37:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.265+0000 7fbb7ebe6740 1 -- start start 2024-08-28T19:37:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.266+0000 7fbb7ebe6740 1 Processor -- start 2024-08-28T19:37:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.266+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 0x55d53301b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.266+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c41390 0x55d53301b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.266+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 0x55d5330276e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.266+0000 7fbb7640d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 0x55d53301b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.266+0000 7fbb7640d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 0x55d53301b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55478/0 (socket says 172.21.15.16:55478) 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.266+0000 7fbb7640d640 1 -- 172.21.15.16:0/3916107112 learned_addr learned my addr 172.21.15.16:0/3916107112 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.267+0000 7fbb76c0e640 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 0x55d5330276e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.266+0000 7fbb75c0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c41390 0x55d53301b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.267+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/3916107112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d532dd0700 con 0x55d532c62070 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.267+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/3916107112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d532ef4750 con 0x55d532c41390 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.267+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/3916107112 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d53301e3a0 con 0x55d532c391b0 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.267+0000 7fbb7ebe6740 10 client.0 _ll_register_callbacks cb 0x55d532f23e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.268+0000 7fbb7640d640 1 -- 172.21.15.16:0/3916107112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c41390 msgr2=0x55d53301b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.268+0000 7fbb7640d640 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c41390 0x55d53301b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.268+0000 7fbb7640d640 1 -- 172.21.15.16:0/3916107112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 msgr2=0x55d5330276e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.268+0000 7fbb7640d640 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 0x55d5330276e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.268+0000 7fbb7640d640 1 -- 172.21.15.16:0/3916107112 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d532f2d900 con 0x55d532c391b0 2024-08-28T19:37:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.269+0000 7fbb7640d640 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 0x55d53301b1b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009a20 tx=0x7fbb6c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=832d260f85894114 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.269+0000 7fbb59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:37:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.270+0000 7fbb59ffb640 1 -- 172.21.15.16:0/3916107112 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c04d030 con 0x55d532c391b0 2024-08-28T19:37:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.270+0000 7fbb59ffb640 1 -- 172.21.15.16:0/3916107112 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7fbb6c03dbe0 con 0x55d532c391b0 2024-08-28T19:37:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.270+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/3916107112 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb6000fde0 con 0x55d532c391b0 2024-08-28T19:37:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.270+0000 7fbb59ffb640 1 -- 172.21.15.16:0/3916107112 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fbb6c05a050 con 0x55d532c391b0 2024-08-28T19:37:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.270+0000 7fbb7ebe6740 10 client.28471 fetch_fsmap learned FSMap version 82 2024-08-28T19:37:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.270+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/3916107112 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55d53302a100 con 0x55d532c391b0 2024-08-28T19:37:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.271+0000 7fbb59ffb640 1 -- 172.21.15.16:0/3916107112 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6000fde0 con 0x55d532c391b0 2024-08-28T19:37:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.271+0000 7fbb59ffb640 1 -- 172.21.15.16:0/3916107112 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7fbb6c0483a0 con 0x55d532c391b0 2024-08-28T19:37:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.271+0000 7fbb7ebe6740 10 client.28471 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:37:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.271+0000 7fbb7ebe6740 10 client.28471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:37:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.271+0000 7fbb7ebe6740 20 client.28471 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.271+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d532c209f0 0x55d53302dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.271+0000 7fbb7ebe6740 4 client.28471 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:37:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.271+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/3916107112 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d532f805d0 con 0x55d532c209f0 2024-08-28T19:37:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.273+0000 7fbb75c0c640 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d532c209f0 0x55d53302dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.273+0000 7fbb75c0c640 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d532c209f0 0x55d53302dad0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=382b89d1041c8654 server_cookie=bde99b256fedd955 in_seq=0 out_seq=0 2024-08-28T19:37:06.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.273+0000 7fbb59ffb640 10 client.28471 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.274+0000 7fbb59ffb640 1 -- 172.21.15.16:0/3916107112 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d532f805d0 con 0x55d532c209f0 2024-08-28T19:37:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb59ffb640 10 client.28471 handle_command_reply: tid=0 2024-08-28T19:37:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb7ebe6740 1 client.28471 shutdown 2024-08-28T19:37:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/3916107112 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 msgr2=0x55d53301b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 0x55d53301b1b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009a20 tx=0x7fbb6c009260 comp rx=0 tx=0).stop 2024-08-28T19:37:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/3916107112 shutdown_connections 2024-08-28T19:37:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 0x55d5330276e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c41390 0x55d53301b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d532c209f0 0x55d53302dad0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/3916107112 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 0x55d53301b1b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb59ffb640 0 client.28471 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/3916107112 >> 172.21.15.16:0/3916107112 conn(0x55d532c56460 msgr2=0x55d533012d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/3916107112 shutdown_connections 2024-08-28T19:37:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.275+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/3916107112 wait complete. 2024-08-28T19:37:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.276+0000 7fbb7ebe6740 20 client.28471 trim_cache size 0 max 16384 2024-08-28T19:37:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.299+0000 7fbb7ebe6740 1 Processor -- start 2024-08-28T19:37:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.299+0000 7fbb7ebe6740 1 -- start start 2024-08-28T19:37:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.299+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c391b0 0x55d5330275e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.299+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c41390 0x55d5330183a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.299+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 0x55d5330188e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.299+0000 7fbb7ebe6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701aeb00 con 0x55d532c62070 2024-08-28T19:37:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.299+0000 7fbb7ebe6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701b0880 con 0x55d532c391b0 2024-08-28T19:37:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.299+0000 7fbb7ebe6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701af200 con 0x55d532c41390 2024-08-28T19:37:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.300+0000 7fbb76c0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 0x55d5330188e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.300+0000 7fbb76c0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 0x55d5330188e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34294/0 (socket says 172.21.15.16:34294) 2024-08-28T19:37:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.300+0000 7fbb76c0e640 1 -- 172.21.15.16:0/447214472 learned_addr learned my addr 172.21.15.16:0/447214472 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.300+0000 7fbb7640d640 1 --2- 172.21.15.16:0/447214472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c391b0 0x55d5330275e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.300+0000 7fbb76c0e640 1 -- 172.21.15.16:0/447214472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c41390 msgr2=0x55d5330183a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.300+0000 7fbb76c0e640 1 --2- 172.21.15.16:0/447214472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c41390 0x55d5330183a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.300+0000 7fbb76c0e640 1 -- 172.21.15.16:0/447214472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c391b0 msgr2=0x55d5330275e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.300+0000 7fbb76c0e640 1 --2- 172.21.15.16:0/447214472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c391b0 0x55d5330275e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.300+0000 7fbb76c0e640 1 -- 172.21.15.16:0/447214472 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5330196b0 con 0x55d532c62070 2024-08-28T19:37:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.300+0000 7fbb76c0e640 1 --2- 172.21.15.16:0/447214472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 0x55d5330188e0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fbb680e2f50 tx=0x7fbb680b0770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97042882c82326d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.301+0000 7fbb5bfff640 1 -- 172.21.15.16:0/447214472 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb680ae070 con 0x55d532c62070 2024-08-28T19:37:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.301+0000 7fbb5bfff640 1 -- 172.21.15.16:0/447214472 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb680a8180 con 0x55d532c62070 2024-08-28T19:37:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.301+0000 7fbb5bfff640 1 -- 172.21.15.16:0/447214472 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6801d040 con 0x55d532c62070 2024-08-28T19:37:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.302+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/447214472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 msgr2=0x55d5330188e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.302+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/447214472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 0x55d5330188e0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fbb680e2f50 tx=0x7fbb680b0770 comp rx=0 tx=0).stop 2024-08-28T19:37:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.302+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/447214472 shutdown_connections 2024-08-28T19:37:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.302+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/447214472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c62070 0x55d5330188e0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.302+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/447214472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c41390 0x55d5330183a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.302+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/447214472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c391b0 0x55d5330275e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.302+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/447214472 >> 172.21.15.16:0/447214472 conn(0x55d532c56460 msgr2=0x55d533019a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.302+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/447214472 shutdown_connections 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.302+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/447214472 wait complete. 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb7ebe6740 1 -- start start 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb7ebe6740 1 Processor -- start 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 0x55d53301fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c41390 0x55d53301ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb7ebe6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 0x55d5330974d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb7ebe6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701b00f0 con 0x55d532c41390 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb7ebe6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701af960 con 0x55d532c62070 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb7ebe6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701acb20 con 0x55d532c391b0 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb7ebe6740 10 client.0 _ll_register_callbacks cb 0x7fbb6c0314a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb75c0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c41390 0x55d53301ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb75c0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c41390 0x55d53301ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34310/0 (socket says 172.21.15.16:34310) 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb76c0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 0x55d5330974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb75c0c640 1 -- 172.21.15.16:0/822427142 learned_addr learned my addr 172.21.15.16:0/822427142 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.303+0000 7fbb7640d640 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 0x55d53301fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb75c0c640 1 -- 172.21.15.16:0/822427142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 msgr2=0x55d53301fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb75c0c640 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 0x55d53301fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb75c0c640 1 -- 172.21.15.16:0/822427142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 msgr2=0x55d5330974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb75c0c640 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 0x55d5330974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb75c0c640 1 -- 172.21.15.16:0/822427142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d533097a10 con 0x55d532c41390 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb75c0c640 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c41390 0x55d53301ffb0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fbb600093e0 tx=0x7fbb60005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f499083b03765f16 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb59ffb640 1 -- 172.21.15.16:0/822427142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60009a90 con 0x55d532c41390 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb59ffb640 1 -- 172.21.15.16:0/822427142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9490+0+0 (secure 0 0 0) 0x7fbb6001f4c0 con 0x55d532c41390 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/822427142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb680a8180 con 0x55d532c41390 2024-08-28T19:37:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb59ffb640 1 -- 172.21.15.16:0/822427142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fbb6003b050 con 0x55d532c41390 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb7ebe6740 10 client.19437 fetch_fsmap learned FSMap version 82 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.304+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/822427142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55d533098b60 con 0x55d532c41390 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.305+0000 7fbb59ffb640 1 -- 172.21.15.16:0/822427142 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb680a8180 con 0x55d532c41390 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.305+0000 7fbb59ffb640 1 -- 172.21.15.16:0/822427142 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7fbb6001d020 con 0x55d532c41390 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.305+0000 7fbb7ebe6740 10 client.19437 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.305+0000 7fbb7ebe6740 10 client.19437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.305+0000 7fbb7ebe6740 20 client.19437 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.305+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d532bf5da0 0x55d53309b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.305+0000 7fbb7ebe6740 4 client.19437 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.305+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/822427142 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d532f7c070 con 0x55d532bf5da0 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.305+0000 7fbb7640d640 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d532bf5da0 0x55d53309b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb59ffb640 10 client.19437 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb7640d640 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d532bf5da0 0x55d53309b170 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c5a53fb4b181310 server_cookie=b96ea0f7b65c3b16 in_seq=0 out_seq=0 2024-08-28T19:37:06.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb59ffb640 1 -- 172.21.15.16:0/822427142 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d532f7c070 con 0x55d532bf5da0 2024-08-28T19:37:06.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb59ffb640 10 client.19437 handle_command_reply: tid=0 2024-08-28T19:37:06.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb7ebe6740 1 client.19437 shutdown 2024-08-28T19:37:06.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/822427142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c41390 msgr2=0x55d53301ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:06.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c41390 0x55d53301ffb0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fbb600093e0 tx=0x7fbb60005dc0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/822427142 shutdown_connections 2024-08-28T19:37:06.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d532c62070 0x55d5330974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d532c41390 0x55d53301ffb0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d532bf5da0 0x55d53309b170 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb59ffb640 0 client.19437 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:06.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb7ebe6740 1 --2- 172.21.15.16:0/822427142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d532c391b0 0x55d53301fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.306+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/822427142 >> 172.21.15.16:0/822427142 conn(0x55d532c56460 msgr2=0x55d533010b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:06.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.307+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/822427142 shutdown_connections 2024-08-28T19:37:06.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.307+0000 7fbb7ebe6740 1 -- 172.21.15.16:0/822427142 wait complete. 2024-08-28T19:37:06.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.307+0000 7fbb7ebe6740 20 client.19437 trim_cache size 0 max 16384 2024-08-28T19:37:06.312 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:37:06.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 -- 172.21.15.16:0/4155896879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb5c0b4ca0 msgr2=0x7fbb5c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:06.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb5c0b4ca0 0x7fbb5c0b71b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009230 tx=0x7fbb6c0058d0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 -- 172.21.15.16:0/4155896879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb70108a90 msgr2=0x7fbb70104a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:06.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb70108a90 0x7fbb70104a70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800ca30 tx=0x7fbb6800cef0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 -- 172.21.15.16:0/4155896879 shutdown_connections 2024-08-28T19:37:06.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb5c0b4ca0 0x7fbb5c0b71b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb70108a90 0x7fbb70104a70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb70108140 0x7fbb70104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 --2- 172.21.15.16:0/4155896879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb70102800 0x7fbb70103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:06.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 -- 172.21.15.16:0/4155896879 >> 172.21.15.16:0/4155896879 conn(0x7fbb700794d0 msgr2=0x7fbb70101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:06.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 -- 172.21.15.16:0/4155896879 shutdown_connections 2024-08-28T19:37:06.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:06.308+0000 7fbb78671640 1 -- 172.21.15.16:0/4155896879 wait complete. 2024-08-28T19:37:06.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 45590 9.15 MB/sec execute 42 sec latency 8119.139 ms 2024-08-28T19:37:07.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 46370 10.21 MB/sec execute 43 sec latency 8455.094 ms 2024-08-28T19:37:07.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:07 smithi115 ceph-mon[22184]: pgmap v839: 129 pgs: 129 active+clean; 964 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 6.1 MiB/s wr, 644 op/s 2024-08-28T19:37:07.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:07 smithi016 ceph-mon[18556]: pgmap v839: 129 pgs: 129 active+clean; 964 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 6.1 MiB/s wr, 644 op/s 2024-08-28T19:37:07.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:07 smithi028 ceph-mon[21769]: pgmap v839: 129 pgs: 129 active+clean; 964 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 6.1 MiB/s wr, 644 op/s 2024-08-28T19:37:08.044 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:37:08.044 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:37:08.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 47063 10.69 MB/sec execute 44 sec latency 226.457 ms 2024-08-28T19:37:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.105+0000 7fbdde347640 1 -- 172.21.15.16:0/1239016089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd8102ee0 msgr2=0x7fbdd81032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.105+0000 7fbdde347640 1 --2- 172.21.15.16:0/1239016089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd8102ee0 0x7fbdd81032f0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009900 tx=0x7fbdc802a410 comp rx=0 tx=0).stop 2024-08-28T19:37:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.105+0000 7fbdde347640 1 -- 172.21.15.16:0/1239016089 shutdown_connections 2024-08-28T19:37:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.105+0000 7fbdde347640 1 --2- 172.21.15.16:0/1239016089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd8108aa0 0x7fbdd810f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.105+0000 7fbdde347640 1 --2- 172.21.15.16:0/1239016089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd8108150 0x7fbdd8108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.105+0000 7fbdde347640 1 --2- 172.21.15.16:0/1239016089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd8102ee0 0x7fbdd81032f0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.105+0000 7fbdde347640 1 -- 172.21.15.16:0/1239016089 >> 172.21.15.16:0/1239016089 conn(0x7fbdd8077a00 msgr2=0x7fbdd8077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.105+0000 7fbdde347640 1 -- 172.21.15.16:0/1239016089 shutdown_connections 2024-08-28T19:37:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.105+0000 7fbdde347640 1 -- 172.21.15.16:0/1239016089 wait complete. 2024-08-28T19:37:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.106+0000 7fbdde347640 1 Processor -- start 2024-08-28T19:37:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.106+0000 7fbdde347640 1 -- start start 2024-08-28T19:37:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbdde347640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd8102ee0 0x7fbdd81a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbdde347640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd8108150 0x7fbdd81a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbdde347640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd8108aa0 0x7fbdd81a6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbdde347640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd8112750 con 0x7fbdd8108150 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbdde347640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd81125d0 con 0x7fbdd8108aa0 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbdde347640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd81128d0 con 0x7fbdd8102ee0 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbdd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd8108150 0x7fbdd81a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd8102ee0 0x7fbdd81a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd8102ee0 0x7fbdd81a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55504/0 (socket says 172.21.15.16:55504) 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbdd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd8108150 0x7fbdd81a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34322/0 (socket says 172.21.15.16:34322) 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbdd7fff640 1 -- 172.21.15.16:0/2397428692 learned_addr learned my addr 172.21.15.16:0/2397428692 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.107+0000 7fbddc8e4640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd8108aa0 0x7fbdd81a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.108+0000 7fbdd7fff640 1 -- 172.21.15.16:0/2397428692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd8108aa0 msgr2=0x7fbdd81a6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.108+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd8108aa0 0x7fbdd81a6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.108+0000 7fbdd7fff640 1 -- 172.21.15.16:0/2397428692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd8108150 msgr2=0x7fbdd81a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.108+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd8108150 0x7fbdd81a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.108+0000 7fbdd7fff640 1 -- 172.21.15.16:0/2397428692 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd81a67c0 con 0x7fbdd8102ee0 2024-08-28T19:37:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.108+0000 7fbddc8e4640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd8108aa0 0x7fbdd81a6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.108+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd8102ee0 0x7fbdd81a0860 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8005940 tx=0x7fbdc8004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.108+0000 7fbdd57fa640 1 -- 172.21.15.16:0/2397428692 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc803f030 con 0x7fbdd8102ee0 2024-08-28T19:37:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.109+0000 7fbdde347640 1 -- 172.21.15.16:0/2397428692 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd81a6a50 con 0x7fbdd8102ee0 2024-08-28T19:37:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.109+0000 7fbdde347640 1 -- 172.21.15.16:0/2397428692 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd81ad210 con 0x7fbdd8102ee0 2024-08-28T19:37:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.109+0000 7fbdd57fa640 1 -- 172.21.15.16:0/2397428692 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdc8003aa0 con 0x7fbdd8102ee0 2024-08-28T19:37:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.109+0000 7fbdd57fa640 1 -- 172.21.15.16:0/2397428692 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8049060 con 0x7fbdd8102ee0 2024-08-28T19:37:09.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.112+0000 7fbde48bc740 1 Processor -- start 2024-08-28T19:37:09.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.113+0000 7fbde48bc740 1 -- start start 2024-08-28T19:37:09.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.113+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600827bf350 0x560082b65780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.113+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560082796490 0x560082b65cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.113+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 0x560082b66200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.113+0000 7fbde48bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560082adb5f0 con 0x560082796490 2024-08-28T19:37:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.113+0000 7fbde48bc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560082acab80 con 0x56008279e670 2024-08-28T19:37:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.113+0000 7fbde48bc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560082aceb30 con 0x5600827bf350 2024-08-28T19:37:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.113+0000 7fbdd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 0x560082b66200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.113+0000 7fbdd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 0x560082b66200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57666/0 (socket says 172.21.15.16:57666) 2024-08-28T19:37:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.113+0000 7fbdd77fe640 1 -- 172.21.15.16:0/2632357173 learned_addr learned my addr 172.21.15.16:0/2632357173 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.113+0000 7fbdd57fa640 1 -- 172.21.15.16:0/2397428692 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fbdc8042030 con 0x7fbdd8102ee0 2024-08-28T19:37:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.114+0000 7fbdd77fe640 1 -- 172.21.15.16:0/2632357173 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600827bf350 msgr2=0x560082b65780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.114+0000 7fbdd77fe640 1 --2- 172.21.15.16:0/2632357173 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600827bf350 0x560082b65780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.114+0000 7fbdd77fe640 1 -- 172.21.15.16:0/2632357173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560082796490 msgr2=0x560082b65cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.114+0000 7fbdd77fe640 1 --2- 172.21.15.16:0/2632357173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560082796490 0x560082b65cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.114+0000 7fbdd77fe640 1 -- 172.21.15.16:0/2632357173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56008282a930 con 0x56008279e670 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.114+0000 7fbdd77fe640 1 --2- 172.21.15.16:0/2632357173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 0x560082b66200 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009c20 tx=0x7fbdcc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f751dd2261ecf46 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.114+0000 7fbdbe7fc640 1 -- 172.21.15.16:0/2632357173 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc004230 con 0x56008279e670 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.114+0000 7fbdbe7fc640 1 -- 172.21.15.16:0/2632357173 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdcc0043d0 con 0x56008279e670 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.115+0000 7fbdbe7fc640 1 -- 172.21.15.16:0/2632357173 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc004d50 con 0x56008279e670 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.115+0000 7fbdd57fa640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdc00b4ca0 0x7fbdc00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.115+0000 7fbdd57fa640 1 -- 172.21.15.16:0/2397428692 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9578+0+0 (secure 0 0 0) 0x7fbdc8105980 con 0x7fbdd8102ee0 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.116+0000 7fbddc8e4640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdc00b4ca0 0x7fbdc00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.116+0000 7fbde48bc740 1 -- 172.21.15.16:0/2632357173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 msgr2=0x560082b66200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.116+0000 7fbde48bc740 1 --2- 172.21.15.16:0/2632357173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 0x560082b66200 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009c20 tx=0x7fbdcc011810 comp rx=0 tx=0).stop 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.116+0000 7fbde48bc740 1 -- 172.21.15.16:0/2632357173 shutdown_connections 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.116+0000 7fbde48bc740 1 --2- 172.21.15.16:0/2632357173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 0x560082b66200 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.116+0000 7fbde48bc740 1 --2- 172.21.15.16:0/2632357173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560082796490 0x560082b65cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.116+0000 7fbde48bc740 1 --2- 172.21.15.16:0/2632357173 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600827bf350 0x560082b65780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.116+0000 7fbde48bc740 1 -- 172.21.15.16:0/2632357173 >> 172.21.15.16:0/2632357173 conn(0x5600827b3740 msgr2=0x560082b63670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.116+0000 7fbddc8e4640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdc00b4ca0 0x7fbdc00b71b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0005fd0 tx=0x7fbdd0009ed0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.116+0000 7fbde48bc740 1 -- 172.21.15.16:0/2632357173 shutdown_connections 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.117+0000 7fbde48bc740 1 -- 172.21.15.16:0/2632357173 wait complete. 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.117+0000 7fbde48bc740 1 -- start start 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.118+0000 7fbde48bc740 1 Processor -- start 2024-08-28T19:37:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.118+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560082796490 0x560082b70fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.118+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56008279e670 0x560082b71510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.118+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600827bf350 0x560082b7d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560082796490 0x560082b70fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560082796490 0x560082b70fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57682/0 (socket says 172.21.15.16:57682) 2024-08-28T19:37:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdd77fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56008279e670 0x560082b71510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdd7fff640 1 -- 172.21.15.16:0/2702817144 learned_addr learned my addr 172.21.15.16:0/2702817144 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbde48bc740 1 -- 172.21.15.16:0/2702817144 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56008292d890 con 0x5600827bf350 2024-08-28T19:37:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbde48bc740 1 -- 172.21.15.16:0/2702817144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560082a51790 con 0x560082796490 2024-08-28T19:37:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbde48bc740 1 -- 172.21.15.16:0/2702817144 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560082b73f60 con 0x56008279e670 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbde48bc740 10 client.0 _ll_register_callbacks cb 0x5600827d6ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbddc8e4640 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600827bf350 0x560082b7d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdd7fff640 1 -- 172.21.15.16:0/2702817144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56008279e670 msgr2=0x560082b71510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56008279e670 0x560082b71510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdd7fff640 1 -- 172.21.15.16:0/2702817144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600827bf350 msgr2=0x560082b7d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600827bf350 0x560082b7d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdd7fff640 1 -- 172.21.15.16:0/2702817144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560082a88f30 con 0x560082796490 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbddc8e4640 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600827bf350 0x560082b7d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560082796490 0x560082b70fd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbdc81032b0 tx=0x7fbdc803bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6459093e917956be server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.119+0000 7fbdb77fe640 1 -- 172.21.15.16:0/2702817144 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8035070 con 0x560082796490 2024-08-28T19:37:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.120+0000 7fbdb77fe640 1 -- 172.21.15.16:0/2702817144 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9578+0+0 (secure 0 0 0) 0x7fbdc8102f30 con 0x560082796490 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.120+0000 7fbde48bc740 1 -- 172.21.15.16:0/2702817144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdcc0043d0 con 0x560082796490 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.120+0000 7fbdb77fe640 1 -- 172.21.15.16:0/2702817144 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fbdc80cb810 con 0x560082796490 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.120+0000 7fbde48bc740 10 client.28871 fetch_fsmap learned FSMap version 82 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.120+0000 7fbde48bc740 1 -- 172.21.15.16:0/2702817144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x560082b90da0 con 0x560082796490 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.121+0000 7fbdb77fe640 1 -- 172.21.15.16:0/2702817144 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdcc0043d0 con 0x560082796490 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.121+0000 7fbdb77fe640 1 -- 172.21.15.16:0/2702817144 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7fbdc804c060 con 0x560082796490 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.121+0000 7fbde48bc740 10 client.28871 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.122+0000 7fbde48bc740 10 client.28871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.122+0000 7fbde48bc740 20 client.28871 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.122+0000 7fbde48bc740 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56008277dcd0 0x560082b94440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.122+0000 7fbde48bc740 4 client.28871 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:37:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.122+0000 7fbde48bc740 1 -- 172.21.15.16:0/2702817144 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560082add540 con 0x56008277dcd0 2024-08-28T19:37:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.122+0000 7fbdd77fe640 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56008277dcd0 0x560082b94440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.123+0000 7fbdb77fe640 10 client.28871 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.123+0000 7fbdd77fe640 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56008277dcd0 0x560082b94440 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e680fe6ca37ecf70 server_cookie=2330d4ea2911f109 in_seq=0 out_seq=0 2024-08-28T19:37:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.124+0000 7fbdb77fe640 1 -- 172.21.15.16:0/2702817144 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560082add540 con 0x56008277dcd0 2024-08-28T19:37:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.124+0000 7fbdb77fe640 10 client.28871 handle_command_reply: tid=0 2024-08-28T19:37:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.124+0000 7fbde48bc740 1 client.28871 shutdown 2024-08-28T19:37:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.124+0000 7fbde48bc740 1 -- 172.21.15.16:0/2702817144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560082796490 msgr2=0x560082b70fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.124+0000 7fbde48bc740 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560082796490 0x560082b70fd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbdc81032b0 tx=0x7fbdc803bf90 comp rx=0 tx=0).stop 2024-08-28T19:37:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.124+0000 7fbde48bc740 1 -- 172.21.15.16:0/2702817144 shutdown_connections 2024-08-28T19:37:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.124+0000 7fbde48bc740 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600827bf350 0x560082b7d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.124+0000 7fbde48bc740 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56008277dcd0 0x560082b94440 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.124+0000 7fbde48bc740 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56008279e670 0x560082b71510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.124+0000 7fbde48bc740 1 --2- 172.21.15.16:0/2702817144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560082796490 0x560082b70fd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.124+0000 7fbdb77fe640 0 client.28871 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:09.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.125+0000 7fbde48bc740 1 -- 172.21.15.16:0/2702817144 >> 172.21.15.16:0/2702817144 conn(0x5600827b3740 msgr2=0x560082b667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:09.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.125+0000 7fbde48bc740 1 -- 172.21.15.16:0/2702817144 shutdown_connections 2024-08-28T19:37:09.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.125+0000 7fbde48bc740 1 -- 172.21.15.16:0/2702817144 wait complete. 2024-08-28T19:37:09.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.125+0000 7fbde48bc740 20 client.28871 trim_cache size 0 max 16384 2024-08-28T19:37:09.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.149+0000 7fbde48bc740 1 Processor -- start 2024-08-28T19:37:09.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.149+0000 7fbde48bc740 1 -- start start 2024-08-28T19:37:09.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008277dcd0 0x560082b636e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560082796490 0x560082b70d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56008279e670 0x560082b712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbde48bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc40101d0 con 0x560082796490 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbde48bc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc4011f50 con 0x56008277dcd0 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbde48bc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc40108d0 con 0x56008279e670 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbdd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560082796490 0x560082b70d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008277dcd0 0x560082b636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008277dcd0 0x560082b636e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57690/0 (socket says 172.21.15.16:57690) 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbdd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560082796490 0x560082b70d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34352/0 (socket says 172.21.15.16:34352) 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbdd7fff640 1 -- 172.21.15.16:0/3563885604 learned_addr learned my addr 172.21.15.16:0/3563885604 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbddc8e4640 1 --2- 172.21.15.16:0/3563885604 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56008279e670 0x560082b712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbdd7fff640 1 -- 172.21.15.16:0/3563885604 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56008279e670 msgr2=0x560082b712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:09.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/3563885604 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56008279e670 0x560082b712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbdd7fff640 1 -- 172.21.15.16:0/3563885604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560082796490 msgr2=0x560082b70d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:09.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/3563885604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560082796490 0x560082b70d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.150+0000 7fbdd7fff640 1 -- 172.21.15.16:0/3563885604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560082b67a70 con 0x56008277dcd0 2024-08-28T19:37:09.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.151+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/3563885604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008277dcd0 0x560082b636e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbdc80b7220 tx=0x7fbdc8105c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12eed8d5b2004349 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.151+0000 7fbdbdffb640 1 -- 172.21.15.16:0/3563885604 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8035070 con 0x56008277dcd0 2024-08-28T19:37:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.151+0000 7fbdbdffb640 1 -- 172.21.15.16:0/3563885604 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdc80cbd00 con 0x56008277dcd0 2024-08-28T19:37:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.151+0000 7fbdbdffb640 1 -- 172.21.15.16:0/3563885604 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8003ca0 con 0x56008277dcd0 2024-08-28T19:37:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.152+0000 7fbde48bc740 1 -- 172.21.15.16:0/3563885604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008277dcd0 msgr2=0x560082b636e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.152+0000 7fbde48bc740 1 --2- 172.21.15.16:0/3563885604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008277dcd0 0x560082b636e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbdc80b7220 tx=0x7fbdc8105c80 comp rx=0 tx=0).stop 2024-08-28T19:37:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.152+0000 7fbde48bc740 1 -- 172.21.15.16:0/3563885604 shutdown_connections 2024-08-28T19:37:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.152+0000 7fbde48bc740 1 --2- 172.21.15.16:0/3563885604 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56008279e670 0x560082b712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.152+0000 7fbde48bc740 1 --2- 172.21.15.16:0/3563885604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560082796490 0x560082b70d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.152+0000 7fbde48bc740 1 --2- 172.21.15.16:0/3563885604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008277dcd0 0x560082b636e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.152+0000 7fbde48bc740 1 -- 172.21.15.16:0/3563885604 >> 172.21.15.16:0/3563885604 conn(0x5600827b3740 msgr2=0x560082b62fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.153+0000 7fbde48bc740 1 -- 172.21.15.16:0/3563885604 shutdown_connections 2024-08-28T19:37:09.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.153+0000 7fbde48bc740 1 -- 172.21.15.16:0/3563885604 wait complete. 2024-08-28T19:37:09.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.153+0000 7fbde48bc740 1 -- start start 2024-08-28T19:37:09.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.153+0000 7fbde48bc740 1 Processor -- start 2024-08-28T19:37:09.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.154+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56008277dcd0 0x560082b753e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.154+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560082796490 0x560082b75920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.154+0000 7fbde48bc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 0x560082b75e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.154+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56008277dcd0 0x560082b753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.155+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56008277dcd0 0x560082b753e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34358/0 (socket says 172.21.15.16:34358) 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.155+0000 7fbdd77fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560082796490 0x560082b75920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.155+0000 7fbdd77fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560082796490 0x560082b75920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55528/0 (socket says 172.21.15.16:55528) 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.155+0000 7fbdd7fff640 1 -- 172.21.15.16:0/4266997284 learned_addr learned my addr 172.21.15.16:0/4266997284 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.155+0000 7fbde48bc740 1 -- 172.21.15.16:0/4266997284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc40117c0 con 0x56008277dcd0 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.155+0000 7fbde48bc740 1 -- 172.21.15.16:0/4266997284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc4011030 con 0x56008279e670 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.155+0000 7fbde48bc740 1 -- 172.21.15.16:0/4266997284 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc400e1d0 con 0x560082796490 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.155+0000 7fbddc8e4640 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 0x560082b75e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.155+0000 7fbde48bc740 10 client.0 _ll_register_callbacks cb 0x7fbdc80cbad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.156+0000 7fbdd7fff640 1 -- 172.21.15.16:0/4266997284 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560082796490 msgr2=0x560082b75920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.156+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560082796490 0x560082b75920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.156+0000 7fbdd7fff640 1 -- 172.21.15.16:0/4266997284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 msgr2=0x560082b75e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.156+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 0x560082b75e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.156+0000 7fbdd7fff640 1 -- 172.21.15.16:0/4266997284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560082bfdf80 con 0x56008277dcd0 2024-08-28T19:37:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.156+0000 7fbdd7fff640 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56008277dcd0 0x560082b753e0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009a20 tx=0x7fbdc80d03e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6204e09ad459a8f2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.156+0000 7fbdb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:37:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.156+0000 7fbdb77fe640 1 -- 172.21.15.16:0/4266997284 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8035070 con 0x56008277dcd0 2024-08-28T19:37:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.157+0000 7fbde48bc740 1 -- 172.21.15.16:0/4266997284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdc80cbd00 con 0x56008277dcd0 2024-08-28T19:37:09.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.157+0000 7fbdb77fe640 1 -- 172.21.15.16:0/4266997284 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9578+0+0 (secure 0 0 0) 0x7fbdc8003ca0 con 0x56008277dcd0 2024-08-28T19:37:09.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.157+0000 7fbdb77fe640 1 -- 172.21.15.16:0/4266997284 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fbdc80d04c0 con 0x56008277dcd0 2024-08-28T19:37:09.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.157+0000 7fbde48bc740 10 client.19449 fetch_fsmap learned FSMap version 82 2024-08-28T19:37:09.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.157+0000 7fbde48bc740 1 -- 172.21.15.16:0/4266997284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x560082bfdf80 con 0x56008277dcd0 2024-08-28T19:37:09.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.157+0000 7fbdb77fe640 1 -- 172.21.15.16:0/4266997284 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdc80cbd00 con 0x56008277dcd0 2024-08-28T19:37:09.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.157+0000 7fbdb77fe640 1 -- 172.21.15.16:0/4266997284 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7fbdc80caf60 con 0x56008277dcd0 2024-08-28T19:37:09.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.158+0000 7fbde48bc740 10 client.19449 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:37:09.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.158+0000 7fbde48bc740 10 client.19449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:37:09.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.158+0000 7fbde48bc740 20 client.19449 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:09.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.158+0000 7fbde48bc740 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600827bf350 0x560082c01630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:09.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.158+0000 7fbde48bc740 4 client.19449 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:37:09.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.158+0000 7fbde48bc740 1 -- 172.21.15.16:0/4266997284 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560082ad8fe0 con 0x5600827bf350 2024-08-28T19:37:09.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.158+0000 7fbdd77fe640 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600827bf350 0x560082c01630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:09.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.159+0000 7fbdb77fe640 10 client.19449 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:09.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.159+0000 7fbdd77fe640 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600827bf350 0x560082c01630 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8363982af1aa0e19 server_cookie=92067e348877ba01 in_seq=0 out_seq=0 2024-08-28T19:37:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.161+0000 7fbdb77fe640 1 -- 172.21.15.16:0/4266997284 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560082ad8fe0 con 0x5600827bf350 2024-08-28T19:37:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.161+0000 7fbdb77fe640 10 client.19449 handle_command_reply: tid=0 2024-08-28T19:37:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.161+0000 7fbde48bc740 1 client.19449 shutdown 2024-08-28T19:37:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.161+0000 7fbde48bc740 1 -- 172.21.15.16:0/4266997284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56008277dcd0 msgr2=0x560082b753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.161+0000 7fbde48bc740 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56008277dcd0 0x560082b753e0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009a20 tx=0x7fbdc80d03e0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.161+0000 7fbde48bc740 1 -- 172.21.15.16:0/4266997284 shutdown_connections 2024-08-28T19:37:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.161+0000 7fbde48bc740 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56008279e670 0x560082b75e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.162+0000 7fbde48bc740 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5600827bf350 0x560082c01630 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.162+0000 7fbde48bc740 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560082796490 0x560082b75920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.162+0000 7fbdb77fe640 0 client.19449 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.162+0000 7fbde48bc740 1 --2- 172.21.15.16:0/4266997284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56008277dcd0 0x560082b753e0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.162+0000 7fbde48bc740 1 -- 172.21.15.16:0/4266997284 >> 172.21.15.16:0/4266997284 conn(0x5600827b3740 msgr2=0x560082b70d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.162+0000 7fbde48bc740 1 -- 172.21.15.16:0/4266997284 shutdown_connections 2024-08-28T19:37:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.162+0000 7fbde48bc740 1 -- 172.21.15.16:0/4266997284 wait complete. 2024-08-28T19:37:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.162+0000 7fbde48bc740 20 client.19449 trim_cache size 0 max 16384 2024-08-28T19:37:09.166 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:37:09.166 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:37:09.166 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "c36c0080-16dd-4e78-a556-732432a8ae9b", 2024-08-28T19:37:09.167 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:37:09.167 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:37:09.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.163+0000 7fbdde347640 1 -- 172.21.15.16:0/2397428692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdc00b4ca0 msgr2=0x7fbdc00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.163+0000 7fbdde347640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdc00b4ca0 0x7fbdc00b71b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0005fd0 tx=0x7fbdd0009ed0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.163+0000 7fbdde347640 1 -- 172.21.15.16:0/2397428692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd8102ee0 msgr2=0x7fbdd81a0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.163+0000 7fbdde347640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd8102ee0 0x7fbdd81a0860 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8005940 tx=0x7fbdc8004270 comp rx=0 tx=0).stop 2024-08-28T19:37:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.163+0000 7fbdde347640 1 -- 172.21.15.16:0/2397428692 shutdown_connections 2024-08-28T19:37:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.163+0000 7fbdde347640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbdc00b4ca0 0x7fbdc00b71b0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.163+0000 7fbdde347640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd8108aa0 0x7fbdd81a6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.163+0000 7fbdde347640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd8108150 0x7fbdd81a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.163+0000 7fbdde347640 1 --2- 172.21.15.16:0/2397428692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdd8102ee0 0x7fbdd81a0860 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.163+0000 7fbdde347640 1 -- 172.21.15.16:0/2397428692 >> 172.21.15.16:0/2397428692 conn(0x7fbdd8077a00 msgr2=0x7fbdd8106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.165+0000 7fbdde347640 1 -- 172.21.15.16:0/2397428692 shutdown_connections 2024-08-28T19:37:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:09.165+0000 7fbdde347640 1 -- 172.21.15.16:0/2397428692 wait complete. 2024-08-28T19:37:09.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 47565 11.14 MB/sec execute 45 sec latency 328.556 ms 2024-08-28T19:37:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:09 smithi115 ceph-mon[22184]: pgmap v840: 129 pgs: 129 active+clean; 970 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 5.8 MiB/s wr, 677 op/s 2024-08-28T19:37:09.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:09 smithi115 ceph-mon[22184]: osdmap e103: 12 total, 12 up, 12 in 2024-08-28T19:37:09.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:09 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:37:09.701 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:09 smithi016 ceph-mon[18556]: pgmap v840: 129 pgs: 129 active+clean; 970 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 5.8 MiB/s wr, 677 op/s 2024-08-28T19:37:09.701 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:09 smithi016 ceph-mon[18556]: osdmap e103: 12 total, 12 up, 12 in 2024-08-28T19:37:09.701 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:09 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:37:09.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:09 smithi028 ceph-mon[21769]: pgmap v840: 129 pgs: 129 active+clean; 970 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 5.8 MiB/s wr, 677 op/s 2024-08-28T19:37:09.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:09 smithi028 ceph-mon[21769]: osdmap e103: 12 total, 12 up, 12 in 2024-08-28T19:37:09.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:09 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:37:10.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 48440 11.68 MB/sec execute 46 sec latency 282.875 ms 2024-08-28T19:37:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:10 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:37:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:10 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:10 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:37:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:10 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:37:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:10 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:37:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:10 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:37:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:10 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:10 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:10 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:10 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:10 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:37:10.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:10 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:10 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:37:10.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:10 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:37:10.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:10 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:37:10.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:10 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:37:10.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:10 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:10 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:10 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:10 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:37:10.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:10 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:37:10.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:10 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:10 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:37:10.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:10 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:37:10.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:10 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:37:10.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:10 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:37:10.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:10 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:10 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:10 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:10.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:10 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:11.038 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:11.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 49472 12.90 MB/sec execute 47 sec latency 61.559 ms 2024-08-28T19:37:11.483 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:11 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:11.483 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:11 smithi016 ceph-mon[18556]: pgmap v842: 129 pgs: 129 active+clean; 990 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 7.5 MiB/s wr, 804 op/s 2024-08-28T19:37:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.638+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/2342015534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bec102770 msgr2=0x7f4bec113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.638+0000 7f4bf2ce7640 1 --2- 172.21.15.16:0/2342015534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bec102770 0x7f4bec113b20 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc01a910 tx=0x7f4bdc03b4b0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.638+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/2342015534 shutdown_connections 2024-08-28T19:37:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.638+0000 7f4bf2ce7640 1 --2- 172.21.15.16:0/2342015534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bec114260 0x7f4bec116770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.638+0000 7f4bf2ce7640 1 --2- 172.21.15.16:0/2342015534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bec102770 0x7f4bec113b20 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.638+0000 7f4bf2ce7640 1 --2- 172.21.15.16:0/2342015534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bec101e20 0x7f4bec102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.638+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/2342015534 >> 172.21.15.16:0/2342015534 conn(0x7f4bec079c60 msgr2=0x7f4bec07a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.638+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/2342015534 shutdown_connections 2024-08-28T19:37:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.638+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/2342015534 wait complete. 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.639+0000 7f4bf2ce7640 1 Processor -- start 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.639+0000 7f4bf2ce7640 1 -- start start 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.639+0000 7f4bf2ce7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bec101e20 0x7f4bec1138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.639+0000 7f4bf2ce7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bec102770 0x7f4bec10fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.639+0000 7f4bf2ce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bec114260 0x7f4bec1100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.639+0000 7f4bf2ce7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec10bb00 con 0x7f4bec102770 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.639+0000 7f4bf2ce7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec10b980 con 0x7f4bec114260 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.639+0000 7f4bf2ce7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec10bc80 con 0x7f4bec101e20 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.640+0000 7f4bf1284640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bec114260 0x7f4bec1100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.640+0000 7f4bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bec102770 0x7f4bec10fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.640+0000 7f4bf0a83640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bec101e20 0x7f4bec1138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.640+0000 7f4bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bec102770 0x7f4bec10fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34386/0 (socket says 172.21.15.16:34386) 2024-08-28T19:37:11.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.640+0000 7f4bebfff640 1 -- 172.21.15.16:0/1380042037 learned_addr learned my addr 172.21.15.16:0/1380042037 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:11.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.640+0000 7f4bf0a83640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bec101e20 0x7f4bec1138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55532/0 (socket says 172.21.15.16:55532) 2024-08-28T19:37:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.640+0000 7f4bf1284640 1 -- 172.21.15.16:0/1380042037 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bec101e20 msgr2=0x7f4bec1138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.640+0000 7f4bf1284640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bec101e20 0x7f4bec1138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.640+0000 7f4bf1284640 1 -- 172.21.15.16:0/1380042037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bec102770 msgr2=0x7f4bec10fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.640+0000 7f4bf1284640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bec102770 0x7f4bec10fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.640+0000 7f4bf1284640 1 -- 172.21.15.16:0/1380042037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bec1108f0 con 0x7f4bec114260 2024-08-28T19:37:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.641+0000 7f4bf1284640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bec114260 0x7f4bec1100d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009fd0 tx=0x7f4be400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.641+0000 7f4be9ffb640 1 -- 172.21.15.16:0/1380042037 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be400ce20 con 0x7f4bec114260 2024-08-28T19:37:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.641+0000 7f4be9ffb640 1 -- 172.21.15.16:0/1380042037 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4be4005020 con 0x7f4bec114260 2024-08-28T19:37:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.641+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/1380042037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bec1a4460 con 0x7f4bec114260 2024-08-28T19:37:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.642+0000 7f4be9ffb640 1 -- 172.21.15.16:0/1380042037 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4010690 con 0x7f4bec114260 2024-08-28T19:37:11.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.642+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/1380042037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bec1a49a0 con 0x7f4bec114260 2024-08-28T19:37:11.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.645+0000 7f4bf925c740 1 Processor -- start 2024-08-28T19:37:11.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.645+0000 7f4be9ffb640 1 -- 172.21.15.16:0/1380042037 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4be40054d0 con 0x7f4bec114260 2024-08-28T19:37:11.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bf925c740 1 -- start start 2024-08-28T19:37:11.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f2617070 0x55a0f29bc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 0x55a0f29bcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bf925c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a0f2933630 con 0x55a0f2617070 2024-08-28T19:37:11.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bf925c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0f2922bc0 con 0x55a0f25ee1b0 2024-08-28T19:37:11.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bf925c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a0f2926b70 con 0x55a0f25f6390 2024-08-28T19:37:11.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bf0a83640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bf0a83640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57732/0 (socket says 172.21.15.16:57732) 2024-08-28T19:37:11.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bf0a83640 1 -- 172.21.15.16:0/2434630230 learned_addr learned my addr 172.21.15.16:0/2434630230 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bebfff640 1 --2- 172.21.15.16:0/2434630230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f2617070 0x55a0f29bc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.646+0000 7f4bebfff640 1 --2- 172.21.15.16:0/2434630230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 0x55a0f29bcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4bf0a83640 1 -- 172.21.15.16:0/2434630230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 msgr2=0x55a0f29bcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4bf0a83640 1 --2- 172.21.15.16:0/2434630230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 0x55a0f29bcd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4bf0a83640 1 -- 172.21.15.16:0/2434630230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f2617070 msgr2=0x55a0f29bc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4bf0a83640 1 --2- 172.21.15.16:0/2434630230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f2617070 0x55a0f29bc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4bf0a83640 1 -- 172.21.15.16:0/2434630230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0f2682650 con 0x55a0f25ee1b0 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4bebfff640 1 --2- 172.21.15.16:0/2434630230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 0x55a0f29bcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4be9ffb640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4bd80b4d20 0x7f4bd80b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4bebfff640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4bd80b4d20 0x7f4bd80b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4bf0a83640 1 --2- 172.21.15.16:0/2434630230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bc800 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009ca0 tx=0x7f4bd4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=570abf5929077c41 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4be9ffb640 1 -- 172.21.15.16:0/1380042037 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9578+0+0 (secure 0 0 0) 0x7f4be40e5ff0 con 0x7f4bec114260 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4be2ffd640 1 -- 172.21.15.16:0/2434630230 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4005220 con 0x55a0f25ee1b0 2024-08-28T19:37:11.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4be2ffd640 1 -- 172.21.15.16:0/2434630230 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bd4004c50 con 0x55a0f25ee1b0 2024-08-28T19:37:11.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.647+0000 7f4be2ffd640 1 -- 172.21.15.16:0/2434630230 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd40196d0 con 0x55a0f25ee1b0 2024-08-28T19:37:11.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.648+0000 7f4bf925c740 1 -- 172.21.15.16:0/2434630230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25ee1b0 msgr2=0x55a0f29bc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:11.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.648+0000 7f4bf925c740 1 --2- 172.21.15.16:0/2434630230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bc800 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009ca0 tx=0x7f4bd4011810 comp rx=0 tx=0).stop 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.648+0000 7f4bebfff640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4bd80b4d20 0x7f4bd80b7230 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc0057a0 tx=0x7f4bdc005710 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.648+0000 7f4bf925c740 1 -- 172.21.15.16:0/2434630230 shutdown_connections 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.648+0000 7f4bf925c740 1 --2- 172.21.15.16:0/2434630230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 0x55a0f29bcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.648+0000 7f4bf925c740 1 --2- 172.21.15.16:0/2434630230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bc800 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.648+0000 7f4bf925c740 1 --2- 172.21.15.16:0/2434630230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f2617070 0x55a0f29bc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.648+0000 7f4bf925c740 1 -- 172.21.15.16:0/2434630230 >> 172.21.15.16:0/2434630230 conn(0x55a0f260b460 msgr2=0x55a0f29ba1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.648+0000 7f4bf925c740 1 -- 172.21.15.16:0/2434630230 shutdown_connections 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.649+0000 7f4bf925c740 1 -- 172.21.15.16:0/2434630230 wait complete. 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.649+0000 7f4bf925c740 1 -- start start 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.649+0000 7f4bf925c740 1 Processor -- start 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 0x55a0f29c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 0x55a0f29c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f2617070 0x55a0f29d3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bf925c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a0f2785700 con 0x55a0f25ee1b0 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bf925c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0f28a9700 con 0x55a0f2617070 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bf925c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a0f29caaa0 con 0x55a0f25f6390 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bebfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 0x55a0f29c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bebfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 0x55a0f29c7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55558/0 (socket says 172.21.15.16:55558) 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bebfff640 1 -- 172.21.15.16:0/210672205 learned_addr learned my addr 172.21.15.16:0/210672205 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bf1284640 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f2617070 0x55a0f29d3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bf0a83640 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 0x55a0f29c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.650+0000 7f4bf925c740 10 client.0 _ll_register_callbacks cb 0x55a0f28d8dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bf1284640 1 -- 172.21.15.16:0/210672205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 msgr2=0x55a0f29c7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bf1284640 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 0x55a0f29c7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bf1284640 1 -- 172.21.15.16:0/210672205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 msgr2=0x55a0f29c7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:11.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bf1284640 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 0x55a0f29c7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bf1284640 1 -- 172.21.15.16:0/210672205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0f28e28b0 con 0x55a0f2617070 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bebfff640 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 0x55a0f29c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bf1284640 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f2617070 0x55a0f29d3dc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4be40e3320 tx=0x7f4be40aca80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c23eea6a58275860 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bcbfff640 1 -- 172.21.15.16:0/210672205 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be40b55e0 con 0x55a0f2617070 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bcbfff640 1 -- 172.21.15.16:0/210672205 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9578+0+0 (secure 0 0 0) 0x7f4be40a8550 con 0x55a0f2617070 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bf925c740 1 -- 172.21.15.16:0/210672205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bd4004c50 con 0x55a0f2617070 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bcbfff640 1 -- 172.21.15.16:0/210672205 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f4be40b5150 con 0x55a0f2617070 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bf925c740 10 client.28889 fetch_fsmap learned FSMap version 82 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.651+0000 7f4bf925c740 1 -- 172.21.15.16:0/210672205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55a0f29d6830 con 0x55a0f2617070 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.652+0000 7f4bcbfff640 1 -- 172.21.15.16:0/210672205 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bd4004c50 con 0x55a0f2617070 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.652+0000 7f4bcbfff640 1 -- 172.21.15.16:0/210672205 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7f4be40aaf90 con 0x55a0f2617070 2024-08-28T19:37:11.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.652+0000 7f4bf925c740 10 client.28889 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:37:11.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.652+0000 7f4bf925c740 10 client.28889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:37:11.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.652+0000 7f4bf925c740 20 client.28889 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:11.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.652+0000 7f4bf925c740 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0f25d59f0 0x55a0f29d9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.652+0000 7f4bf925c740 4 client.28889 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:37:11.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.652+0000 7f4bf925c740 1 -- 172.21.15.16:0/210672205 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0f2935580 con 0x55a0f25d59f0 2024-08-28T19:37:11.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.652+0000 7f4bf0a83640 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0f25d59f0 0x55a0f29d9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.653+0000 7f4bcbfff640 10 client.28889 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:11.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.653+0000 7f4bf0a83640 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0f25d59f0 0x55a0f29d9d70 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62548cddb087bda2 server_cookie=cad069b34658875 in_seq=0 out_seq=0 2024-08-28T19:37:11.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.656+0000 7f4bcbfff640 1 -- 172.21.15.16:0/210672205 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a0f2935580 con 0x55a0f25d59f0 2024-08-28T19:37:11.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.656+0000 7f4bcbfff640 10 client.28889 handle_command_reply: tid=0 2024-08-28T19:37:11.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.656+0000 7f4bf925c740 1 client.28889 shutdown 2024-08-28T19:37:11.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bf925c740 1 -- 172.21.15.16:0/210672205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f2617070 msgr2=0x55a0f29d3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:11.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bf925c740 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f2617070 0x55a0f29d3dc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4be40e3320 tx=0x7f4be40aca80 comp rx=0 tx=0).stop 2024-08-28T19:37:11.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bf925c740 1 -- 172.21.15.16:0/210672205 shutdown_connections 2024-08-28T19:37:11.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bf925c740 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f2617070 0x55a0f29d3dc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bf925c740 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0f25d59f0 0x55a0f29d9d70 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bf925c740 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25f6390 0x55a0f29c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bcbfff640 0 client.28889 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:11.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bf925c740 1 --2- 172.21.15.16:0/210672205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 0x55a0f29c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bf925c740 1 -- 172.21.15.16:0/210672205 >> 172.21.15.16:0/210672205 conn(0x55a0f260b460 msgr2=0x55a0f29bf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:11.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bf925c740 1 -- 172.21.15.16:0/210672205 shutdown_connections 2024-08-28T19:37:11.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bf925c740 1 -- 172.21.15.16:0/210672205 wait complete. 2024-08-28T19:37:11.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.657+0000 7f4bf925c740 20 client.28889 trim_cache size 0 max 16384 2024-08-28T19:37:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bf925c740 1 Processor -- start 2024-08-28T19:37:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bf925c740 1 -- start start 2024-08-28T19:37:11.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25d59f0 0x55a0f29c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25f6390 0x55a0f29c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bf925c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1aeb20 con 0x55a0f25ee1b0 2024-08-28T19:37:11.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bf925c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1b08a0 con 0x55a0f25f6390 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bf925c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1af220 con 0x55a0f25d59f0 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34418/0 (socket says 172.21.15.16:34418) 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bebfff640 1 -- 172.21.15.16:0/1103149255 learned_addr learned my addr 172.21.15.16:0/1103149255 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bf0a83640 1 --2- 172.21.15.16:0/1103149255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25d59f0 0x55a0f29c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.678+0000 7f4bf1284640 1 --2- 172.21.15.16:0/1103149255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25f6390 0x55a0f29c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.679+0000 7f4bebfff640 1 -- 172.21.15.16:0/1103149255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25d59f0 msgr2=0x55a0f29c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.679+0000 7f4bebfff640 1 --2- 172.21.15.16:0/1103149255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25d59f0 0x55a0f29c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.679+0000 7f4bebfff640 1 -- 172.21.15.16:0/1103149255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25f6390 msgr2=0x55a0f29c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.679+0000 7f4bebfff640 1 --2- 172.21.15.16:0/1103149255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25f6390 0x55a0f29c4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.679+0000 7f4bebfff640 1 -- 172.21.15.16:0/1103149255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0f29c69d0 con 0x55a0f25ee1b0 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.679+0000 7f4bebfff640 1 --2- 172.21.15.16:0/1103149255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bfad0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc040ce0 tx=0x7f4bdc01a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d81ab432232038 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.679+0000 7f4be27fc640 1 -- 172.21.15.16:0/1103149255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc046040 con 0x55a0f25ee1b0 2024-08-28T19:37:11.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.679+0000 7f4be27fc640 1 -- 172.21.15.16:0/1103149255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bdc004ac0 con 0x55a0f25ee1b0 2024-08-28T19:37:11.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.679+0000 7f4be27fc640 1 -- 172.21.15.16:0/1103149255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc05c710 con 0x55a0f25ee1b0 2024-08-28T19:37:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.680+0000 7f4bf925c740 1 -- 172.21.15.16:0/1103149255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 msgr2=0x55a0f29bfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.680+0000 7f4bf925c740 1 --2- 172.21.15.16:0/1103149255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bfad0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc040ce0 tx=0x7f4bdc01a270 comp rx=0 tx=0).stop 2024-08-28T19:37:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.680+0000 7f4bf925c740 1 -- 172.21.15.16:0/1103149255 shutdown_connections 2024-08-28T19:37:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.680+0000 7f4bf925c740 1 --2- 172.21.15.16:0/1103149255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25f6390 0x55a0f29c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.680+0000 7f4bf925c740 1 --2- 172.21.15.16:0/1103149255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25ee1b0 0x55a0f29bfad0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.680+0000 7f4bf925c740 1 --2- 172.21.15.16:0/1103149255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25d59f0 0x55a0f29c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.680+0000 7f4bf925c740 1 -- 172.21.15.16:0/1103149255 >> 172.21.15.16:0/1103149255 conn(0x55a0f260b460 msgr2=0x55a0f29bf3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.681+0000 7f4bf925c740 1 -- 172.21.15.16:0/1103149255 shutdown_connections 2024-08-28T19:37:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.681+0000 7f4bf925c740 1 -- 172.21.15.16:0/1103149255 wait complete. 2024-08-28T19:37:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.681+0000 7f4bf925c740 1 -- start start 2024-08-28T19:37:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.681+0000 7f4bf925c740 1 Processor -- start 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.681+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25d59f0 0x55a0f29cc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.681+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25ee1b0 0x55a0f29ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.681+0000 7f4bf925c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25f6390 0x55a0f2a43730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.681+0000 7f4bf925c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1b0110 con 0x55a0f25f6390 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.681+0000 7f4bf925c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1af980 con 0x55a0f25d59f0 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf925c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1acb20 con 0x55a0f25ee1b0 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bebfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25ee1b0 0x55a0f29ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf0a83640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25d59f0 0x55a0f29cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf1284640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25f6390 0x55a0f2a43730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf0a83640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25d59f0 0x55a0f29cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57758/0 (socket says 172.21.15.16:57758) 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf0a83640 1 -- 172.21.15.16:0/1594023920 learned_addr learned my addr 172.21.15.16:0/1594023920 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf925c740 10 client.0 _ll_register_callbacks cb 0x7f4be40a8180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf0a83640 1 -- 172.21.15.16:0/1594023920 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25ee1b0 msgr2=0x55a0f29ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf0a83640 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25ee1b0 0x55a0f29ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf0a83640 1 -- 172.21.15.16:0/1594023920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25f6390 msgr2=0x55a0f2a43730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf0a83640 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25f6390 0x55a0f2a43730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf0a83640 1 -- 172.21.15.16:0/1594023920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0f2a43d40 con 0x55a0f25d59f0 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf1284640 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25f6390 0x55a0f2a43730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bf0a83640 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25d59f0 0x55a0f29cc540 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009bc0 tx=0x7f4bd40042c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d699e154c6e7b9a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.682+0000 7f4bcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bcbfff640 1 -- 172.21.15.16:0/1594023920 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4005c50 con 0x55a0f25d59f0 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bf925c740 1 -- 172.21.15.16:0/1594023920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bdc004ac0 con 0x55a0f25d59f0 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bcbfff640 1 -- 172.21.15.16:0/1594023920 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9578+0+0 (secure 0 0 0) 0x7f4bd40035f0 con 0x55a0f25d59f0 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bcbfff640 1 -- 172.21.15.16:0/1594023920 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f4bd4039050 con 0x55a0f25d59f0 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bf925c740 10 client.28895 fetch_fsmap learned FSMap version 82 2024-08-28T19:37:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bf925c740 1 -- 172.21.15.16:0/1594023920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55a0f2a44fb0 con 0x55a0f25d59f0 2024-08-28T19:37:11.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bcbfff640 1 -- 172.21.15.16:0/1594023920 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bdc004ac0 con 0x55a0f25d59f0 2024-08-28T19:37:11.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bcbfff640 1 -- 172.21.15.16:0/1594023920 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 82) ==== 8430+0+0 (secure 0 0 0) 0x7f4bd4019810 con 0x55a0f25d59f0 2024-08-28T19:37:11.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bf925c740 10 client.28895 fetch_fsmap finished waiting for FSMap version 82 2024-08-28T19:37:11.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bf925c740 10 client.28895 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:37:11.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bf925c740 20 client.28895 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:11.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bf925c740 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0f2617070 0x55a0f2a47600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:11.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bf925c740 4 client.28895 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:37:11.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.683+0000 7f4bf925c740 1 -- 172.21.15.16:0/1594023920 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a0f2931020 con 0x55a0f2617070 2024-08-28T19:37:11.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.684+0000 7f4bebfff640 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0f2617070 0x55a0f2a47600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.684+0000 7f4bcbfff640 10 client.28895 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.684+0000 7f4bebfff640 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0f2617070 0x55a0f2a47600 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1eb2c21e59021975 server_cookie=91d7893cf38a43c6 in_seq=0 out_seq=0 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bcbfff640 1 -- 172.21.15.16:0/1594023920 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+6534 (crc 0 0 0) 0x55a0f2931020 con 0x55a0f2617070 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bcbfff640 10 client.28895 handle_command_reply: tid=0 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 1 client.28895 shutdown 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 1 -- 172.21.15.16:0/1594023920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25d59f0 msgr2=0x55a0f29cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25d59f0 0x55a0f29cc540 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009bc0 tx=0x7f4bd40042c0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 1 -- 172.21.15.16:0/1594023920 shutdown_connections 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0f25f6390 0x55a0f2a43730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a0f2617070 0x55a0f2a47600 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a0f25ee1b0 0x55a0f29ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 1 --2- 172.21.15.16:0/1594023920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0f25d59f0 0x55a0f29cc540 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 1 -- 172.21.15.16:0/1594023920 >> 172.21.15.16:0/1594023920 conn(0x55a0f260b460 msgr2=0x55a0f29bd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:11.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bcbfff640 0 client.28895 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:11.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 1 -- 172.21.15.16:0/1594023920 shutdown_connections 2024-08-28T19:37:11.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 1 -- 172.21.15.16:0/1594023920 wait complete. 2024-08-28T19:37:11.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.685+0000 7f4bf925c740 20 client.28895 trim_cache size 0 max 16384 2024-08-28T19:37:11.692 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:37:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:37:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:37:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:37:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "c36c0080-16dd-4e78-a556-732432a8ae9b", 2024-08-28T19:37:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:37:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:37:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:37:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:37:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:37:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:37:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:37:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:37:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:37:11.706 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:37:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.686+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/1380042037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4bd80b4d20 msgr2=0x7f4bd80b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.686+0000 7f4bf2ce7640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4bd80b4d20 0x7f4bd80b7230 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc0057a0 tx=0x7f4bdc005710 comp rx=0 tx=0).stop 2024-08-28T19:37:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.686+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/1380042037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bec114260 msgr2=0x7f4bec1100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.686+0000 7f4bf2ce7640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bec114260 0x7f4bec1100d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009fd0 tx=0x7f4be400ef10 comp rx=0 tx=0).stop 2024-08-28T19:37:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.686+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/1380042037 shutdown_connections 2024-08-28T19:37:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.686+0000 7f4bf2ce7640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4bd80b4d20 0x7f4bd80b7230 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.686+0000 7f4bf2ce7640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bec114260 0x7f4bec1100d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.686+0000 7f4bf2ce7640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bec102770 0x7f4bec10fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.686+0000 7f4bf2ce7640 1 --2- 172.21.15.16:0/1380042037 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bec101e20 0x7f4bec1138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.686+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/1380042037 >> 172.21.15.16:0/1380042037 conn(0x7f4bec079c60 msgr2=0x7f4bec115fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.687+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/1380042037 shutdown_connections 2024-08-28T19:37:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:11.687+0000 7f4bf2ce7640 1 -- 172.21.15.16:0/1380042037 wait complete. 2024-08-28T19:37:11.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:11 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:11.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:11 smithi028 ceph-mon[21769]: pgmap v842: 129 pgs: 129 active+clean; 990 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 7.5 MiB/s wr, 804 op/s 2024-08-28T19:37:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:11 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:37:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:11 smithi115 ceph-mon[22184]: pgmap v842: 129 pgs: 129 active+clean; 990 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 7.5 MiB/s wr, 804 op/s 2024-08-28T19:37:12.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 50164 13.27 MB/sec execute 48 sec latency 64.074 ms 2024-08-28T19:37:12.586 INFO:tasks.cephfs.filesystem:scrub status for tag:c36c0080-16dd-4e78-a556-732432a8ae9b - {'path': '/', 'tag': 'c36c0080-16dd-4e78-a556-732432a8ae9b', 'options': 'recursive,force'} 2024-08-28T19:37:13.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 50974 13.90 MB/sec execute 49 sec latency 60.490 ms 2024-08-28T19:37:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:13 smithi016 ceph-mon[18556]: pgmap v843: 129 pgs: 129 active+clean; 998 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.5 MiB/s wr, 720 op/s 2024-08-28T19:37:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:13 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:13.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:13 smithi028 ceph-mon[21769]: pgmap v843: 129 pgs: 129 active+clean; 998 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.5 MiB/s wr, 720 op/s 2024-08-28T19:37:13.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:13 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:13 smithi115 ceph-mon[22184]: pgmap v843: 129 pgs: 129 active+clean; 998 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.5 MiB/s wr, 720 op/s 2024-08-28T19:37:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:13 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:14.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 51890 14.66 MB/sec execute 50 sec latency 969.531 ms 2024-08-28T19:37:15.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 52706 15.33 MB/sec execute 51 sec latency 1969.597 ms 2024-08-28T19:37:15.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:15 smithi016 ceph-mon[18556]: pgmap v844: 129 pgs: 129 active+clean; 1003 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.5 MiB/s wr, 728 op/s 2024-08-28T19:37:15.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:15 smithi016 ceph-mon[18556]: osdmap e104: 12 total, 12 up, 12 in 2024-08-28T19:37:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:15 smithi028 ceph-mon[21769]: pgmap v844: 129 pgs: 129 active+clean; 1003 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.5 MiB/s wr, 728 op/s 2024-08-28T19:37:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:15 smithi028 ceph-mon[21769]: osdmap e104: 12 total, 12 up, 12 in 2024-08-28T19:37:15.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:15 smithi115 ceph-mon[22184]: pgmap v844: 129 pgs: 129 active+clean; 1003 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.5 MiB/s wr, 728 op/s 2024-08-28T19:37:15.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:15 smithi115 ceph-mon[22184]: osdmap e104: 12 total, 12 up, 12 in 2024-08-28T19:37:16.470 INFO:tasks.workunit.client.0.smithi016.stdout: 10 53450 15.66 MB/sec execute 52 sec latency 2969.641 ms 2024-08-28T19:37:17.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 54143 16.15 MB/sec execute 53 sec latency 3969.675 ms 2024-08-28T19:37:17.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:17 smithi016 ceph-mon[18556]: pgmap v846: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 9.6 MiB/s wr, 832 op/s 2024-08-28T19:37:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:17 smithi028 ceph-mon[21769]: pgmap v846: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 9.6 MiB/s wr, 832 op/s 2024-08-28T19:37:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:17 smithi115 ceph-mon[22184]: pgmap v846: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 9.6 MiB/s wr, 832 op/s 2024-08-28T19:37:18.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 55175 16.88 MB/sec execute 54 sec latency 4969.708 ms 2024-08-28T19:37:19.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 55857 17.39 MB/sec execute 55 sec latency 5969.741 ms 2024-08-28T19:37:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:19 smithi016 ceph-mon[18556]: pgmap v847: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 10 MiB/s wr, 841 op/s 2024-08-28T19:37:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:19 smithi028 ceph-mon[21769]: pgmap v847: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 10 MiB/s wr, 841 op/s 2024-08-28T19:37:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:19 smithi115 ceph-mon[22184]: pgmap v847: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 10 MiB/s wr, 841 op/s 2024-08-28T19:37:20.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 56497 17.51 MB/sec execute 56 sec latency 6969.776 ms 2024-08-28T19:37:21.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 57376 18.11 MB/sec execute 57 sec latency 7969.806 ms 2024-08-28T19:37:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:21 smithi016 ceph-mon[18556]: pgmap v848: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 8.7 MiB/s wr, 714 op/s 2024-08-28T19:37:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:21 smithi028 ceph-mon[21769]: pgmap v848: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 8.7 MiB/s wr, 714 op/s 2024-08-28T19:37:21.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:21 smithi115 ceph-mon[22184]: pgmap v848: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 8.7 MiB/s wr, 714 op/s 2024-08-28T19:37:22.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 58015 18.32 MB/sec execute 58 sec latency 8246.748 ms 2024-08-28T19:37:23.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 58721 18.74 MB/sec execute 59 sec latency 195.132 ms 2024-08-28T19:37:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:23 smithi016 ceph-mon[18556]: pgmap v849: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 830 op/s 2024-08-28T19:37:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:23 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:37:23.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:23 smithi028 ceph-mon[21769]: pgmap v849: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 830 op/s 2024-08-28T19:37:23.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:23 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:37:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:23 smithi115 ceph-mon[22184]: pgmap v849: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 830 op/s 2024-08-28T19:37:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:23 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:37:24.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 59338 18.99 MB/sec execute 60 sec latency 186.678 ms 2024-08-28T19:37:24.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:24 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:37:24.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:24 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:37:24.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:24 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:37:24.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:24 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:37:24.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:24 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:37:24.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:24 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:37:24.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:24 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:37:24.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:24 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:37:24.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:24 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:37:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:24 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:37:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:24 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:37:24.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:24 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:37:25.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 60004 19.10 MB/sec execute 61 sec latency 185.719 ms 2024-08-28T19:37:25.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:25 smithi016 ceph-mon[18556]: pgmap v850: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 790 op/s 2024-08-28T19:37:25.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:25 smithi028 ceph-mon[21769]: pgmap v850: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 790 op/s 2024-08-28T19:37:25.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:25 smithi115 ceph-mon[22184]: pgmap v850: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 790 op/s 2024-08-28T19:37:26.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 60968 19.64 MB/sec execute 62 sec latency 122.970 ms 2024-08-28T19:37:26.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:26 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:26.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:26 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:26.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:26 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:27.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 61923 20.24 MB/sec execute 63 sec latency 197.115 ms 2024-08-28T19:37:27.710 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:27 smithi028 ceph-mon[21769]: pgmap v851: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 779 op/s 2024-08-28T19:37:27.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:27 smithi016 ceph-mon[18556]: pgmap v851: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 779 op/s 2024-08-28T19:37:27.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:27 smithi115 ceph-mon[22184]: pgmap v851: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 779 op/s 2024-08-28T19:37:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 62531 20.47 MB/sec execute 64 sec latency 142.167 ms 2024-08-28T19:37:29.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 62896 20.45 MB/sec execute 65 sec latency 208.774 ms 2024-08-28T19:37:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:29 smithi016 ceph-mon[18556]: pgmap v852: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 11 MiB/s wr, 759 op/s 2024-08-28T19:37:29.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:29 smithi028 ceph-mon[21769]: pgmap v852: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 11 MiB/s wr, 759 op/s 2024-08-28T19:37:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:29 smithi115 ceph-mon[22184]: pgmap v852: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 11 MiB/s wr, 759 op/s 2024-08-28T19:37:30.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 63116 20.26 MB/sec execute 66 sec latency 747.600 ms 2024-08-28T19:37:31.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 63136 20.00 MB/sec execute 67 sec latency 1490.231 ms 2024-08-28T19:37:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:31 smithi016 ceph-mon[18556]: pgmap v853: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 11 MiB/s wr, 734 op/s 2024-08-28T19:37:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:31 smithi028 ceph-mon[21769]: pgmap v853: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 11 MiB/s wr, 734 op/s 2024-08-28T19:37:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:31 smithi115 ceph-mon[22184]: pgmap v853: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 11 MiB/s wr, 734 op/s 2024-08-28T19:37:32.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 63279 19.78 MB/sec execute 68 sec latency 1404.840 ms 2024-08-28T19:37:33.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 63596 19.62 MB/sec execute 69 sec latency 1073.017 ms 2024-08-28T19:37:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:33 smithi016 ceph-mon[18556]: pgmap v854: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 15 MiB/s wr, 848 op/s 2024-08-28T19:37:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:33 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:33 smithi028 ceph-mon[21769]: pgmap v854: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 15 MiB/s wr, 848 op/s 2024-08-28T19:37:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:33 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:33.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:33 smithi115 ceph-mon[22184]: pgmap v854: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 15 MiB/s wr, 848 op/s 2024-08-28T19:37:33.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:33 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:34.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 64168 19.80 MB/sec execute 70 sec latency 170.302 ms 2024-08-28T19:37:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:34 smithi016 ceph-mon[18556]: pgmap v855: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 14 MiB/s wr, 698 op/s 2024-08-28T19:37:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:34 smithi028 ceph-mon[21769]: pgmap v855: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 14 MiB/s wr, 698 op/s 2024-08-28T19:37:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:34 smithi115 ceph-mon[22184]: pgmap v855: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 14 MiB/s wr, 698 op/s 2024-08-28T19:37:35.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 64683 19.90 MB/sec execute 71 sec latency 521.775 ms 2024-08-28T19:37:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:35 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:35 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:35 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:36.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 65172 20.14 MB/sec execute 72 sec latency 279.963 ms 2024-08-28T19:37:36.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:36 smithi016 ceph-mon[18556]: pgmap v856: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 24 MiB/s wr, 987 op/s 2024-08-28T19:37:36.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:36 smithi028 ceph-mon[21769]: pgmap v856: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 24 MiB/s wr, 987 op/s 2024-08-28T19:37:36.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:36 smithi115 ceph-mon[22184]: pgmap v856: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 24 MiB/s wr, 987 op/s 2024-08-28T19:37:37.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 65554 20.25 MB/sec execute 73 sec latency 242.267 ms 2024-08-28T19:37:38.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 65940 20.16 MB/sec execute 74 sec latency 557.784 ms 2024-08-28T19:37:39.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:38 smithi016 ceph-mon[18556]: pgmap v857: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 23 MiB/s wr, 896 op/s 2024-08-28T19:37:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:38 smithi028 ceph-mon[21769]: pgmap v857: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 23 MiB/s wr, 896 op/s 2024-08-28T19:37:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:38 smithi115 ceph-mon[22184]: pgmap v857: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 23 MiB/s wr, 896 op/s 2024-08-28T19:37:39.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 66715 20.39 MB/sec execute 75 sec latency 186.199 ms 2024-08-28T19:37:40.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 67475 20.58 MB/sec execute 76 sec latency 479.162 ms 2024-08-28T19:37:41.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:40 smithi016 ceph-mon[18556]: pgmap v858: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 22 MiB/s wr, 945 op/s 2024-08-28T19:37:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:40 smithi028 ceph-mon[21769]: pgmap v858: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 22 MiB/s wr, 945 op/s 2024-08-28T19:37:41.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:40 smithi115 ceph-mon[22184]: pgmap v858: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 22 MiB/s wr, 945 op/s 2024-08-28T19:37:41.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 68500 21.18 MB/sec execute 77 sec latency 74.175 ms 2024-08-28T19:37:42.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 69132 21.36 MB/sec execute 78 sec latency 126.158 ms 2024-08-28T19:37:42.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:37:42.889 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:42 smithi016 ceph-mon[18556]: pgmap v859: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 23 MiB/s wr, 1.05k op/s 2024-08-28T19:37:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:42 smithi028 ceph-mon[21769]: pgmap v859: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 23 MiB/s wr, 1.05k op/s 2024-08-28T19:37:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:42 smithi115 ceph-mon[22184]: pgmap v859: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 23 MiB/s wr, 1.05k op/s 2024-08-28T19:37:43.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 69756 21.42 MB/sec execute 79 sec latency 158.963 ms 2024-08-28T19:37:43.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.477+0000 7f6fd240a640 1 -- 172.21.15.16:0/4037821026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fcc10b490 msgr2=0x7f6fcc10b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:43.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.477+0000 7f6fd240a640 1 --2- 172.21.15.16:0/4037821026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fcc10b490 0x7f6fcc10b8a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009900 tx=0x7f6fb402a400 comp rx=0 tx=0).stop 2024-08-28T19:37:43.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.477+0000 7f6fd240a640 1 -- 172.21.15.16:0/4037821026 shutdown_connections 2024-08-28T19:37:43.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.477+0000 7f6fd240a640 1 --2- 172.21.15.16:0/4037821026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc10bde0 0x7f6fcc112610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.477+0000 7f6fd240a640 1 --2- 172.21.15.16:0/4037821026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fcc10b490 0x7f6fcc10b8a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.477+0000 7f6fd240a640 1 --2- 172.21.15.16:0/4037821026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6fcc105b50 0x7f6fcc105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.477+0000 7f6fd240a640 1 -- 172.21.15.16:0/4037821026 >> 172.21.15.16:0/4037821026 conn(0x7f6fcc101580 msgr2=0x7f6fcc1039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:43.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.478+0000 7f6fd240a640 1 -- 172.21.15.16:0/4037821026 shutdown_connections 2024-08-28T19:37:43.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.478+0000 7f6fd240a640 1 -- 172.21.15.16:0/4037821026 wait complete. 2024-08-28T19:37:43.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fd240a640 1 Processor -- start 2024-08-28T19:37:43.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fd240a640 1 -- start start 2024-08-28T19:37:43.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fd240a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fcc105b50 0x7f6fcc07b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fd240a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6fcc10b490 0x7f6fcc0776c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fd240a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc10bde0 0x7f6fcc077de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fd240a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc115b90 con 0x7f6fcc10bde0 2024-08-28T19:37:43.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fd240a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc115a10 con 0x7f6fcc105b50 2024-08-28T19:37:43.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fd240a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc115d10 con 0x7f6fcc10b490 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fcc105b50 0x7f6fcc07b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fcc105b50 0x7f6fcc07b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:33190/0 (socket says 172.21.15.16:33190) 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fcbfff640 1 -- 172.21.15.16:0/631764670 learned_addr learned my addr 172.21.15.16:0/631764670 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc10bde0 0x7f6fcc077de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.479+0000 7f6fc37fe640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6fcc10b490 0x7f6fcc0776c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.480+0000 7f6fcbfff640 1 -- 172.21.15.16:0/631764670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6fcc10b490 msgr2=0x7f6fcc0776c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.480+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6fcc10b490 0x7f6fcc0776c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.480+0000 7f6fcbfff640 1 -- 172.21.15.16:0/631764670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc10bde0 msgr2=0x7f6fcc077de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.480+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc10bde0 0x7f6fcc077de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.480+0000 7f6fcbfff640 1 -- 172.21.15.16:0/631764670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc1acb70 con 0x7f6fcc105b50 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.480+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc10bde0 0x7f6fcc077de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.480+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fcc105b50 0x7f6fcc07b3e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00ba50 tx=0x7f6fbc00bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:43.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.480+0000 7f6fc9ffb640 1 -- 172.21.15.16:0/631764670 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc019070 con 0x7f6fcc105b50 2024-08-28T19:37:43.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.480+0000 7f6fd240a640 1 -- 172.21.15.16:0/631764670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fcc1acda0 con 0x7f6fcc105b50 2024-08-28T19:37:43.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.480+0000 7f6fc9ffb640 1 -- 172.21.15.16:0/631764670 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fbc00cbe0 con 0x7f6fcc105b50 2024-08-28T19:37:43.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.481+0000 7f6fd240a640 1 -- 172.21.15.16:0/631764670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fcc1ad270 con 0x7f6fcc105b50 2024-08-28T19:37:43.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.481+0000 7f6fc9ffb640 1 -- 172.21.15.16:0/631764670 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0075d0 con 0x7f6fcc105b50 2024-08-28T19:37:43.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.485+0000 7f6fc9ffb640 1 -- 172.21.15.16:0/631764670 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f6fbc01c070 con 0x7f6fcc105b50 2024-08-28T19:37:43.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.486+0000 7f6fd897f740 1 Processor -- start 2024-08-28T19:37:43.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.486+0000 7f6fc9ffb640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6fb80b4cd0 0x7f6fb80b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.487+0000 7f6fc9ffb640 1 -- 172.21.15.16:0/631764670 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7f6fbc0e4e80 con 0x7f6fcc105b50 2024-08-28T19:37:43.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.487+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6fb80b4cd0 0x7f6fb80b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.487+0000 7f6fd897f740 1 -- start start 2024-08-28T19:37:43.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.487+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f8955e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.487+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891901b0 0x558f8955e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.487+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f89198390 0x558f8955ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.487+0000 7f6fd897f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558f894d5630 con 0x558f891b9070 2024-08-28T19:37:43.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.487+0000 7f6fd897f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558f894c4bc0 con 0x558f891901b0 2024-08-28T19:37:43.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.487+0000 7f6fd897f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558f894c8b70 con 0x558f89198390 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.488+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f8955e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.488+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f8955e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53364/0 (socket says 172.21.15.16:53364) 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.488+0000 7f6fcbfff640 1 -- 172.21.15.16:0/3211048605 learned_addr learned my addr 172.21.15.16:0/3211048605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.488+0000 7f6fc37fe640 1 --2- 172.21.15.16:0/3211048605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891901b0 0x558f8955e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.489+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/3211048605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f89198390 0x558f8955ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.489+0000 7f6fcbfff640 1 -- 172.21.15.16:0/3211048605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f89198390 msgr2=0x558f8955ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.489+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/3211048605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f89198390 0x558f8955ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.489+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6fb80b4cd0 0x7f6fb80b71e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009cd0 tx=0x7f6fc40090e0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.489+0000 7f6fcbfff640 1 -- 172.21.15.16:0/3211048605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891901b0 msgr2=0x558f8955e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.489+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/3211048605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891901b0 0x558f8955e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.489+0000 7f6fcbfff640 1 -- 172.21.15.16:0/3211048605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f89224650 con 0x558f891b9070 2024-08-28T19:37:43.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.489+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/3211048605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f8955e2c0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0aaf50 tx=0x7f6fbc004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26be8e9a360eb18e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:43.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.490+0000 7f6fc27fc640 1 -- 172.21.15.16:0/3211048605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0b44c0 con 0x558f891b9070 2024-08-28T19:37:43.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.490+0000 7f6fc27fc640 1 -- 172.21.15.16:0/3211048605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fbc0e2480 con 0x558f891b9070 2024-08-28T19:37:43.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.490+0000 7f6fc27fc640 1 -- 172.21.15.16:0/3211048605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0a6e20 con 0x558f891b9070 2024-08-28T19:37:43.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.491+0000 7f6fd897f740 1 -- 172.21.15.16:0/3211048605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 msgr2=0x558f8955e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:43.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.491+0000 7f6fd897f740 1 --2- 172.21.15.16:0/3211048605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f8955e2c0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0aaf50 tx=0x7f6fbc004b50 comp rx=0 tx=0).stop 2024-08-28T19:37:43.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.491+0000 7f6fd897f740 1 -- 172.21.15.16:0/3211048605 shutdown_connections 2024-08-28T19:37:43.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.491+0000 7f6fd897f740 1 --2- 172.21.15.16:0/3211048605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f89198390 0x558f8955ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.492+0000 7f6fd897f740 1 --2- 172.21.15.16:0/3211048605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891901b0 0x558f8955e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.492+0000 7f6fd897f740 1 --2- 172.21.15.16:0/3211048605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f8955e2c0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.492+0000 7f6fd897f740 1 -- 172.21.15.16:0/3211048605 >> 172.21.15.16:0/3211048605 conn(0x558f891ad460 msgr2=0x558f8955c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:43.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.492+0000 7f6fd897f740 1 -- 172.21.15.16:0/3211048605 shutdown_connections 2024-08-28T19:37:43.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.494+0000 7f6fd897f740 1 -- 172.21.15.16:0/3211048605 wait complete. 2024-08-28T19:37:43.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.495+0000 7f6fd897f740 1 -- start start 2024-08-28T19:37:43.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.495+0000 7f6fd897f740 1 Processor -- start 2024-08-28T19:37:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.496+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f89569a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.496+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f89198390 0x558f89569f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.496+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891b9070 0x558f89575d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.496+0000 7f6fd897f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558f89327700 con 0x558f89198390 2024-08-28T19:37:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.496+0000 7f6fd897f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558f8944b700 con 0x558f891b9070 2024-08-28T19:37:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.496+0000 7f6fd897f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558f8956caa0 con 0x558f891901b0 2024-08-28T19:37:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.496+0000 7f6fd09a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891b9070 0x558f89575d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.496+0000 7f6fd09a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891b9070 0x558f89575d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:33200/0 (socket says 172.21.15.16:33200) 2024-08-28T19:37:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.496+0000 7f6fd09a7640 1 -- 172.21.15.16:0/973408043 learned_addr learned my addr 172.21.15.16:0/973408043 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.496+0000 7f6fc37fe640 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f89198390 0x558f89569f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.497+0000 7f6fd897f740 10 client.0 _ll_register_callbacks cb 0x558f8947adc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.497+0000 7f6fd09a7640 1 -- 172.21.15.16:0/973408043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 msgr2=0x558f89569a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.497+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f89569a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.497+0000 7f6fd09a7640 1 -- 172.21.15.16:0/973408043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f89198390 msgr2=0x558f89569f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.497+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f89198390 0x558f89569f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.497+0000 7f6fd09a7640 1 -- 172.21.15.16:0/973408043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f894848b0 con 0x558f891b9070 2024-08-28T19:37:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.497+0000 7f6fc37fe640 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f89198390 0x558f89569f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.497+0000 7f6fd897f740 1 -- 172.21.15.16:0/973408043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fbc0e2480 con 0x558f891b9070 2024-08-28T19:37:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.498+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891b9070 0x558f89575d90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4015e80 tx=0x7f6fc40047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=591e108925288642 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.498+0000 7f6fab7fe640 10 client.28907 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.498+0000 7f6fab7fe640 1 -- 172.21.15.16:0/973408043 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4024030 con 0x558f891b9070 2024-08-28T19:37:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.498+0000 7f6fab7fe640 1 -- 172.21.15.16:0/973408043 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7f6fc400f040 con 0x558f891b9070 2024-08-28T19:37:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.498+0000 7f6fab7fe640 1 -- 172.21.15.16:0/973408043 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f6fc400ece0 con 0x558f891b9070 2024-08-28T19:37:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.499+0000 7f6fd897f740 10 client.28907 fetch_fsmap learned FSMap version 86 2024-08-28T19:37:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.499+0000 7f6fd897f740 1 -- 172.21.15.16:0/973408043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=86}) -- 0x558f89578790 con 0x558f891b9070 2024-08-28T19:37:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.500+0000 7f6fab7fe640 1 -- 172.21.15.16:0/973408043 <== mon.2 v2:172.21.15.28:3300/0 4 ==== fsmap(e 86) ==== 8426+0+0 (secure 0 0 0) 0x7f6fc4033420 con 0x558f891b9070 2024-08-28T19:37:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.500+0000 7f6fd897f740 10 client.28907 fetch_fsmap finished waiting for FSMap version 86 2024-08-28T19:37:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.500+0000 7f6fd897f740 10 client.28907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:37:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.500+0000 7f6fd897f740 20 client.28907 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.500+0000 7f6fd897f740 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f891779f0 0x558f8957bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.500+0000 7f6fd897f740 4 client.28907 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:37:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.500+0000 7f6fd897f740 1 -- 172.21.15.16:0/973408043 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f894d7580 con 0x558f891779f0 2024-08-28T19:37:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.501+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f891779f0 0x558f8957bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.503+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f891779f0 0x558f8957bd60 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d6a891664f37a03 server_cookie=d79c74bde2fa6693 in_seq=0 out_seq=0 2024-08-28T19:37:43.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.503+0000 7f6fab7fe640 10 client.28907 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:43.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fab7fe640 1 -- 172.21.15.16:0/973408043 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558f894d7580 con 0x558f891779f0 2024-08-28T19:37:43.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fab7fe640 10 client.28907 handle_command_reply: tid=0 2024-08-28T19:37:43.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fd897f740 1 client.28907 shutdown 2024-08-28T19:37:43.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fd897f740 1 -- 172.21.15.16:0/973408043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891b9070 msgr2=0x558f89575d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:43.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fd897f740 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891b9070 0x558f89575d90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4015e80 tx=0x7f6fc40047d0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fd897f740 1 -- 172.21.15.16:0/973408043 shutdown_connections 2024-08-28T19:37:43.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fd897f740 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f891b9070 0x558f89575d90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fd897f740 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f89198390 0x558f89569f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fd897f740 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f891779f0 0x558f8957bd60 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fd897f740 1 --2- 172.21.15.16:0/973408043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f89569a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fd897f740 1 -- 172.21.15.16:0/973408043 >> 172.21.15.16:0/973408043 conn(0x558f891ad460 msgr2=0x558f895614a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:43.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.507+0000 7f6fab7fe640 0 client.28907 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:43.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.508+0000 7f6fd897f740 1 -- 172.21.15.16:0/973408043 shutdown_connections 2024-08-28T19:37:43.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.508+0000 7f6fd897f740 1 -- 172.21.15.16:0/973408043 wait complete. 2024-08-28T19:37:43.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.508+0000 7f6fd897f740 20 client.28907 trim_cache size 0 max 16384 2024-08-28T19:37:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.547+0000 7f6fd897f740 1 Processor -- start 2024-08-28T19:37:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.547+0000 7f6fd897f740 1 -- start start 2024-08-28T19:37:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.547+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f895682b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.548+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f89198390 0x558f895687f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.548+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f895613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.548+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f895682b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.548+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f895682b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:44724/0 (socket says 172.21.15.16:44724) 2024-08-28T19:37:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.548+0000 7f6fcbfff640 1 -- 172.21.15.16:0/714216237 learned_addr learned my addr 172.21.15.16:0/714216237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.548+0000 7f6fd897f740 1 -- 172.21.15.16:0/714216237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1aecd0 con 0x558f891b9070 2024-08-28T19:37:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.548+0000 7f6fc37fe640 1 --2- 172.21.15.16:0/714216237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f89198390 0x558f895687f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.548+0000 7f6fd897f740 1 -- 172.21.15.16:0/714216237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1b0a50 con 0x558f89198390 2024-08-28T19:37:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.548+0000 7f6fd897f740 1 -- 172.21.15.16:0/714216237 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1af460 con 0x558f891901b0 2024-08-28T19:37:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.548+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/714216237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f895613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.549+0000 7f6fd09a7640 1 -- 172.21.15.16:0/714216237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 msgr2=0x558f895682b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.549+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/714216237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f895682b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.549+0000 7f6fd09a7640 1 -- 172.21.15.16:0/714216237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f89198390 msgr2=0x558f895687f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.549+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/714216237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f89198390 0x558f895687f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.549+0000 7f6fd09a7640 1 -- 172.21.15.16:0/714216237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f89556380 con 0x558f891b9070 2024-08-28T19:37:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.549+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/714216237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f895613c0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4013bc0 tx=0x7f6fc4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25c114eab7bf6ed server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.550+0000 7f6fc1ffb640 1 -- 172.21.15.16:0/714216237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4024050 con 0x558f891b9070 2024-08-28T19:37:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.550+0000 7f6fc1ffb640 1 -- 172.21.15.16:0/714216237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fc4025040 con 0x558f891b9070 2024-08-28T19:37:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.550+0000 7f6fc1ffb640 1 -- 172.21.15.16:0/714216237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4029cc0 con 0x558f891b9070 2024-08-28T19:37:43.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.551+0000 7f6fd897f740 1 -- 172.21.15.16:0/714216237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 msgr2=0x558f895613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:43.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.551+0000 7f6fd897f740 1 --2- 172.21.15.16:0/714216237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f895613c0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4013bc0 tx=0x7f6fc4018000 comp rx=0 tx=0).stop 2024-08-28T19:37:43.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.552+0000 7f6fd897f740 1 -- 172.21.15.16:0/714216237 shutdown_connections 2024-08-28T19:37:43.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.552+0000 7f6fd897f740 1 --2- 172.21.15.16:0/714216237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f895613c0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.552+0000 7f6fd897f740 1 --2- 172.21.15.16:0/714216237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f89198390 0x558f895687f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.552+0000 7f6fd897f740 1 --2- 172.21.15.16:0/714216237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f895682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.552+0000 7f6fd897f740 1 -- 172.21.15.16:0/714216237 >> 172.21.15.16:0/714216237 conn(0x558f891ad460 msgr2=0x558f89564b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:43.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.553+0000 7f6fd897f740 1 -- 172.21.15.16:0/714216237 shutdown_connections 2024-08-28T19:37:43.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.553+0000 7f6fd897f740 1 -- 172.21.15.16:0/714216237 wait complete. 2024-08-28T19:37:43.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.554+0000 7f6fd897f740 1 -- start start 2024-08-28T19:37:43.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.555+0000 7f6fd897f740 1 Processor -- start 2024-08-28T19:37:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.555+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f89572190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.555+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f89198390 0x558f8956d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.556+0000 7f6fd897f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f8956db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.556+0000 7f6fd897f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1b02c0 con 0x558f891b9070 2024-08-28T19:37:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.556+0000 7f6fd897f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc1afb90 con 0x558f89198390 2024-08-28T19:37:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.556+0000 7f6fd897f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc074b00 con 0x558f891901b0 2024-08-28T19:37:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.556+0000 7f6fd897f740 10 client.0 _ll_register_callbacks cb 0x7f6fc4009980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.558+0000 7f6fc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f89198390 0x558f8956d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.558+0000 7f6fc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f89198390 0x558f8956d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:33210/0 (socket says 172.21.15.16:33210) 2024-08-28T19:37:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.558+0000 7f6fc37fe640 1 -- 172.21.15.16:0/2784931320 learned_addr learned my addr 172.21.15.16:0/2784931320 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.558+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f89572190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.559+0000 7f6fd09a7640 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f8956db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.559+0000 7f6fcbfff640 1 -- 172.21.15.16:0/2784931320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f89198390 msgr2=0x558f8956d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.559+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f89198390 0x558f8956d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.559+0000 7f6fcbfff640 1 -- 172.21.15.16:0/2784931320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 msgr2=0x558f8956db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.559+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f8956db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.559+0000 7f6fcbfff640 1 -- 172.21.15.16:0/2784931320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f8956e0c0 con 0x558f891901b0 2024-08-28T19:37:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.559+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f89572190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0a84c0 tx=0x7f6fbc014c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3e589d755e59274 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.559+0000 7f6fab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:37:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.560+0000 7f6fab7fe640 1 -- 172.21.15.16:0/2784931320 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0a7110 con 0x558f891901b0 2024-08-28T19:37:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.560+0000 7f6fd897f740 1 -- 172.21.15.16:0/2784931320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fc4025040 con 0x558f891901b0 2024-08-28T19:37:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.560+0000 7f6fab7fe640 1 -- 172.21.15.16:0/2784931320 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7f6fbc0b4030 con 0x558f891901b0 2024-08-28T19:37:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.561+0000 7f6fab7fe640 1 -- 172.21.15.16:0/2784931320 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f6fbc0b44c0 con 0x558f891901b0 2024-08-28T19:37:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.561+0000 7f6fd897f740 10 client.28519 fetch_fsmap learned FSMap version 86 2024-08-28T19:37:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.561+0000 7f6fd897f740 1 -- 172.21.15.16:0/2784931320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=4+}) -- 0x558f895e6330 con 0x558f891901b0 2024-08-28T19:37:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.561+0000 7f6fab7fe640 1 -- 172.21.15.16:0/2784931320 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fc4025040 con 0x558f891901b0 2024-08-28T19:37:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.562+0000 7f6fab7fe640 1 -- 172.21.15.16:0/2784931320 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 86) ==== 8426+0+0 (secure 0 0 0) 0x7f6fbc0d0040 con 0x558f891901b0 2024-08-28T19:37:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.562+0000 7f6fd897f740 10 client.28519 fetch_fsmap finished waiting for FSMap version 86 2024-08-28T19:37:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.562+0000 7f6fd897f740 10 client.28519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:37:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.562+0000 7f6fd897f740 20 client.28519 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.562+0000 7f6fd897f740 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f8914cda0 0x558f895e8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.563+0000 7f6fd897f740 4 client.28519 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:37:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.563+0000 7f6fd897f740 1 -- 172.21.15.16:0/2784931320 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558f894d3020 con 0x558f8914cda0 2024-08-28T19:37:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.563+0000 7f6fc37fe640 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f8914cda0 0x558f895e8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.565+0000 7f6fc37fe640 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f8914cda0 0x558f895e8670 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6af01829c5c5ee6e server_cookie=5c61b9b236476a75 in_seq=0 out_seq=0 2024-08-28T19:37:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.565+0000 7f6fab7fe640 10 client.28519 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.568+0000 7f6fab7fe640 1 -- 172.21.15.16:0/2784931320 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+6338 (crc 0 0 0) 0x558f894d3020 con 0x558f8914cda0 2024-08-28T19:37:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.568+0000 7f6fab7fe640 10 client.28519 handle_command_reply: tid=0 2024-08-28T19:37:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.569+0000 7f6fd897f740 1 client.28519 shutdown 2024-08-28T19:37:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.570+0000 7f6fd897f740 1 -- 172.21.15.16:0/2784931320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 msgr2=0x558f89572190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.570+0000 7f6fd897f740 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f89572190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0a84c0 tx=0x7f6fbc014c90 comp rx=0 tx=0).stop 2024-08-28T19:37:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.570+0000 7f6fd897f740 1 -- 172.21.15.16:0/2784931320 shutdown_connections 2024-08-28T19:37:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.570+0000 7f6fd897f740 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f891b9070 0x558f8956db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.570+0000 7f6fd897f740 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f89198390 0x558f8956d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.570+0000 7f6fd897f740 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x558f8914cda0 0x558f895e8670 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.570+0000 7f6fd897f740 1 --2- 172.21.15.16:0/2784931320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558f891901b0 0x558f89572190 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.570+0000 7f6fd897f740 1 -- 172.21.15.16:0/2784931320 >> 172.21.15.16:0/2784931320 conn(0x558f891ad460 msgr2=0x558f89564b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.570+0000 7f6fab7fe640 0 client.28519 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.571+0000 7f6fd897f740 1 -- 172.21.15.16:0/2784931320 shutdown_connections 2024-08-28T19:37:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.571+0000 7f6fd897f740 1 -- 172.21.15.16:0/2784931320 wait complete. 2024-08-28T19:37:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.571+0000 7f6fd897f740 20 client.28519 trim_cache size 0 max 16384 2024-08-28T19:37:43.573 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:37:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.580 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:37:43.580 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:37:43.580 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:37:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:37:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:37:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:37:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:37:43.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:37:43.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:37:43.588 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:37:43.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.572+0000 7f6fd240a640 1 -- 172.21.15.16:0/631764670 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6fb80b4cd0 msgr2=0x7f6fb80b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:43.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.572+0000 7f6fd240a640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6fb80b4cd0 0x7f6fb80b71e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009cd0 tx=0x7f6fc40090e0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.572+0000 7f6fd240a640 1 -- 172.21.15.16:0/631764670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fcc105b50 msgr2=0x7f6fcc07b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.572+0000 7f6fd240a640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fcc105b50 0x7f6fcc07b3e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00ba50 tx=0x7f6fbc00bf10 comp rx=0 tx=0).stop 2024-08-28T19:37:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.573+0000 7f6fd240a640 1 -- 172.21.15.16:0/631764670 shutdown_connections 2024-08-28T19:37:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.573+0000 7f6fd240a640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc10bde0 0x7f6fcc077de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.573+0000 7f6fd240a640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6fcc10b490 0x7f6fcc0776c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.573+0000 7f6fd240a640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6fb80b4cd0 0x7f6fb80b71e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.573+0000 7f6fd240a640 1 --2- 172.21.15.16:0/631764670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fcc105b50 0x7f6fcc07b3e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.573+0000 7f6fd240a640 1 -- 172.21.15.16:0/631764670 >> 172.21.15.16:0/631764670 conn(0x7f6fcc101580 msgr2=0x7f6fcc10ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.573+0000 7f6fd240a640 1 -- 172.21.15.16:0/631764670 shutdown_connections 2024-08-28T19:37:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:43.573+0000 7f6fd240a640 1 -- 172.21.15.16:0/631764670 wait complete. 2024-08-28T19:37:44.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 70603 21.77 MB/sec execute 80 sec latency 91.588 ms 2024-08-28T19:37:45.150 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:44 smithi016 ceph-mon[18556]: pgmap v860: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 18 MiB/s wr, 862 op/s 2024-08-28T19:37:45.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:44 smithi028 ceph-mon[21769]: pgmap v860: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 18 MiB/s wr, 862 op/s 2024-08-28T19:37:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:44 smithi115 ceph-mon[22184]: pgmap v860: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 18 MiB/s wr, 862 op/s 2024-08-28T19:37:45.428 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:37:45.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:37:45.471 INFO:tasks.workunit.client.0.smithi016.stdout: 10 71459 22.04 MB/sec execute 81 sec latency 167.651 ms 2024-08-28T19:37:45.724 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:46.132 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:45 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:46.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.259+0000 7f9ac9843640 1 -- 172.21.15.16:0/598874728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac4102880 msgr2=0x7f9ac4102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.259+0000 7f9ac9843640 1 --2- 172.21.15.16:0/598874728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac4102880 0x7f9ac4102c90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009900 tx=0x7f9ab402a410 comp rx=0 tx=0).stop 2024-08-28T19:37:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.259+0000 7f9ac9843640 1 -- 172.21.15.16:0/598874728 shutdown_connections 2024-08-28T19:37:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.259+0000 7f9ac9843640 1 --2- 172.21.15.16:0/598874728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4108ac0 0x7f9ac410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.259+0000 7f9ac9843640 1 --2- 172.21.15.16:0/598874728 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ac4108170 0x7f9ac4108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.259+0000 7f9ac9843640 1 --2- 172.21.15.16:0/598874728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac4102880 0x7f9ac4102c90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.259+0000 7f9ac9843640 1 -- 172.21.15.16:0/598874728 >> 172.21.15.16:0/598874728 conn(0x7f9ac4079530 msgr2=0x7f9ac4079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.259+0000 7f9ac9843640 1 -- 172.21.15.16:0/598874728 shutdown_connections 2024-08-28T19:37:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.259+0000 7f9ac9843640 1 -- 172.21.15.16:0/598874728 wait complete. 2024-08-28T19:37:46.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.260+0000 7f9ac9843640 1 Processor -- start 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac9843640 1 -- start start 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac9843640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ac4102880 0x7f9ac419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac9843640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4108170 0x7f9ac419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac9843640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac4108ac0 0x7f9ac419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac9843640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac4112790 con 0x7f9ac4108170 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac9843640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac4112610 con 0x7f9ac4108ac0 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac9843640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac4112910 con 0x7f9ac4102880 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4108170 0x7f9ac419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4108170 0x7f9ac419a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50790/0 (socket says 172.21.15.16:50790) 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac27fc640 1 -- 172.21.15.16:0/2568220465 learned_addr learned my addr 172.21.15.16:0/2568220465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac37fe640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac4108ac0 0x7f9ac419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.261+0000 7f9ac2ffd640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ac4102880 0x7f9ac419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.262+0000 7f9ac37fe640 1 -- 172.21.15.16:0/2568220465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ac4102880 msgr2=0x7f9ac419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.262+0000 7f9ac37fe640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ac4102880 0x7f9ac419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.262+0000 7f9ac37fe640 1 -- 172.21.15.16:0/2568220465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4108170 msgr2=0x7f9ac419a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.262+0000 7f9ac37fe640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4108170 0x7f9ac419a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.262+0000 7f9ac37fe640 1 -- 172.21.15.16:0/2568220465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac41a01a0 con 0x7f9ac4108ac0 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.262+0000 7f9ac2ffd640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ac4102880 0x7f9ac419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.262+0000 7f9ac37fe640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac4108ac0 0x7f9ac419fa70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009fd0 tx=0x7f9abc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.262+0000 7f9ac8841640 1 -- 172.21.15.16:0/2568220465 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc00ce20 con 0x7f9ac4108ac0 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.262+0000 7f9ac9843640 1 -- 172.21.15.16:0/2568220465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac41a0490 con 0x7f9ac4108ac0 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.262+0000 7f9ac8841640 1 -- 172.21.15.16:0/2568220465 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9abc005020 con 0x7f9ac4108ac0 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.262+0000 7f9ac9843640 1 -- 172.21.15.16:0/2568220465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac41a6bf0 con 0x7f9ac4108ac0 2024-08-28T19:37:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.263+0000 7f9ac8841640 1 -- 172.21.15.16:0/2568220465 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc010690 con 0x7f9ac4108ac0 2024-08-28T19:37:46.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.264+0000 7f9ac9843640 1 -- 172.21.15.16:0/2568220465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab00052c0 con 0x7f9ac4108ac0 2024-08-28T19:37:46.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.267+0000 7f9ac8841640 1 -- 172.21.15.16:0/2568220465 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f9abc004a40 con 0x7f9ac4108ac0 2024-08-28T19:37:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.268+0000 7f9ac8841640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9aac0b4d20 0x7f9aac0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.268+0000 7f9ac2ffd640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9aac0b4d20 0x7f9aac0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.268+0000 7f9ac8841640 1 -- 172.21.15.16:0/2568220465 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7f9abc0e5f70 con 0x7f9ac4108ac0 2024-08-28T19:37:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.269+0000 7f9ac2ffd640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9aac0b4d20 0x7f9aac0b7230 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4031440 tx=0x7f9ab4009680 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.272+0000 7f9ac8841640 1 -- 172.21.15.16:0/2568220465 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f9abc0ac990 con 0x7f9ac4108ac0 2024-08-28T19:37:46.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:45 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:45 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:46.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 72205 22.36 MB/sec execute 82 sec latency 155.912 ms 2024-08-28T19:37:46.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.563+0000 7f9ac9843640 1 -- 172.21.15.16:0/2568220465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ab00055b0 con 0x7f9ac4108ac0 2024-08-28T19:37:46.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.564+0000 7f9ac8841640 1 -- 172.21.15.16:0/2568220465 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+11998 (secure 0 0 0) 0x7f9abc0ac290 con 0x7f9ac4108ac0 2024-08-28T19:37:46.566 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:37:46.566 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"btime":"2024-08-28T19:37:44:891460+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":87,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:37:44.102954+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:37:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 87 2024-08-28T19:37:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.567+0000 7f9ac9843640 1 -- 172.21.15.16:0/2568220465 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9aac0b4d20 msgr2=0x7f9aac0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.567+0000 7f9ac9843640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9aac0b4d20 0x7f9aac0b7230 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4031440 tx=0x7f9ab4009680 comp rx=0 tx=0).stop 2024-08-28T19:37:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.567+0000 7f9ac9843640 1 -- 172.21.15.16:0/2568220465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac4108ac0 msgr2=0x7f9ac419fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.567+0000 7f9ac9843640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac4108ac0 0x7f9ac419fa70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009fd0 tx=0x7f9abc00ef10 comp rx=0 tx=0).stop 2024-08-28T19:37:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.567+0000 7f9ac9843640 1 -- 172.21.15.16:0/2568220465 shutdown_connections 2024-08-28T19:37:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.567+0000 7f9ac9843640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9aac0b4d20 0x7f9aac0b7230 secure :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4031440 tx=0x7f9ab4009680 comp rx=0 tx=0).stop 2024-08-28T19:37:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.567+0000 7f9ac9843640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac4108ac0 0x7f9ac419fa70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.567+0000 7f9ac9843640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4108170 0x7f9ac419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.567+0000 7f9ac9843640 1 --2- 172.21.15.16:0/2568220465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ac4102880 0x7f9ac419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.567+0000 7f9ac9843640 1 -- 172.21.15.16:0/2568220465 >> 172.21.15.16:0/2568220465 conn(0x7f9ac4079530 msgr2=0x7f9ac4101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:46.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.568+0000 7f9ac9843640 1 -- 172.21.15.16:0/2568220465 shutdown_connections 2024-08-28T19:37:46.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:46.568+0000 7f9ac9843640 1 -- 172.21.15.16:0/2568220465 wait complete. 2024-08-28T19:37:47.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:46 smithi016 ceph-mon[18556]: pgmap v861: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 18 MiB/s wr, 997 op/s 2024-08-28T19:37:47.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:46 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2568220465' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:37:47.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:37:47.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:46 smithi028 ceph-mon[21769]: pgmap v861: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 18 MiB/s wr, 997 op/s 2024-08-28T19:37:47.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:46 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2568220465' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:37:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:46 smithi115 ceph-mon[22184]: pgmap v861: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 18 MiB/s wr, 997 op/s 2024-08-28T19:37:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:46 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2568220465' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:37:47.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 72890 22.50 MB/sec execute 83 sec latency 67.025 ms 2024-08-28T19:37:47.598 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.253+0000 7f0aa446e640 1 -- 172.21.15.16:0/756992735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c103900 msgr2=0x7f0a9c103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.253+0000 7f0aa446e640 1 --2- 172.21.15.16:0/756992735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c103900 0x7f0a9c103d10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0a94009be0 tx=0x7f0a9402a6c0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.253+0000 7f0aa446e640 1 -- 172.21.15.16:0/756992735 shutdown_connections 2024-08-28T19:37:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.253+0000 7f0aa446e640 1 --2- 172.21.15.16:0/756992735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a9c109b90 0x7f0a9c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.253+0000 7f0aa446e640 1 --2- 172.21.15.16:0/756992735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c109240 0x7f0a9c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.253+0000 7f0aa446e640 1 --2- 172.21.15.16:0/756992735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c103900 0x7f0a9c103d10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.253+0000 7f0aa446e640 1 -- 172.21.15.16:0/756992735 >> 172.21.15.16:0/756992735 conn(0x7f0a9c07b5f0 msgr2=0x7f0a9c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.254+0000 7f0aa446e640 1 -- 172.21.15.16:0/756992735 shutdown_connections 2024-08-28T19:37:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.254+0000 7f0aa446e640 1 -- 172.21.15.16:0/756992735 wait complete. 2024-08-28T19:37:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.255+0000 7f0aa446e640 1 Processor -- start 2024-08-28T19:37:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.256+0000 7f0aa446e640 1 -- start start 2024-08-28T19:37:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.256+0000 7f0aa446e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c103900 0x7f0a9c1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.257+0000 7f0aa446e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c109240 0x7f0a9c1a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.257+0000 7f0aa446e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a9c109b90 0x7f0a9c1a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.257+0000 7f0aa446e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c119ea0 con 0x7f0a9c109240 2024-08-28T19:37:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.257+0000 7f0aa446e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c119d20 con 0x7f0a9c103900 2024-08-28T19:37:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.257+0000 7f0aa446e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c11a020 con 0x7f0a9c109b90 2024-08-28T19:37:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.257+0000 7f0aa2a0b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a9c109b90 0x7f0a9c1a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.257+0000 7f0aa2a0b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a9c109b90 0x7f0a9c1a60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:43762/0 (socket says 172.21.15.16:43762) 2024-08-28T19:37:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.257+0000 7f0aa2a0b640 1 -- 172.21.15.16:0/4070216977 learned_addr learned my addr 172.21.15.16:0/4070216977 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.258+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c109240 0x7f0a9c1a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.258+0000 7f0aa220a640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c103900 0x7f0a9c1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.258+0000 7f0aa2a0b640 1 -- 172.21.15.16:0/4070216977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c103900 msgr2=0x7f0a9c1a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.258+0000 7f0aa2a0b640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c103900 0x7f0a9c1a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.258+0000 7f0aa2a0b640 1 -- 172.21.15.16:0/4070216977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c109240 msgr2=0x7f0a9c1a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.258+0000 7f0aa2a0b640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c109240 0x7f0a9c1a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.258+0000 7f0aa2a0b640 1 -- 172.21.15.16:0/4070216977 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c1a6800 con 0x7f0a9c109b90 2024-08-28T19:37:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.258+0000 7f0aa2a0b640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a9c109b90 0x7f0a9c1a60d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800d750 tx=0x7f0a9800db50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.259+0000 7f0a937fe640 1 -- 172.21.15.16:0/4070216977 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a98017070 con 0x7f0a9c109b90 2024-08-28T19:37:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.259+0000 7f0a937fe640 1 -- 172.21.15.16:0/4070216977 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a980097c0 con 0x7f0a9c109b90 2024-08-28T19:37:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.259+0000 7f0a937fe640 1 -- 172.21.15.16:0/4070216977 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a98017070 con 0x7f0a9c109b90 2024-08-28T19:37:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.259+0000 7f0aa446e640 1 -- 172.21.15.16:0/4070216977 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9c1a6af0 con 0x7f0a9c109b90 2024-08-28T19:37:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.259+0000 7f0aa446e640 1 -- 172.21.15.16:0/4070216977 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a9c1ad250 con 0x7f0a9c109b90 2024-08-28T19:37:48.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.263+0000 7f0aaa9e3740 1 Processor -- start 2024-08-28T19:37:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.264+0000 7f0a937fe640 1 -- 172.21.15.16:0/4070216977 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0a98005220 con 0x7f0a9c109b90 2024-08-28T19:37:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.265+0000 7f0aaa9e3740 1 -- start start 2024-08-28T19:37:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.266+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd448070 0x55e0dd7ed350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.266+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7ed890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.266+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd427390 0x55e0dd7eddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.266+0000 7f0aaa9e3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e0dd764680 con 0x55e0dd448070 2024-08-28T19:37:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.266+0000 7f0a937fe640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a880b50e0 0x7f0a880b75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.266+0000 7f0aaa9e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e0dd753c10 con 0x55e0dd427390 2024-08-28T19:37:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.267+0000 7f0aaa9e3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e0dd757bc0 con 0x55e0dd41f1b0 2024-08-28T19:37:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.267+0000 7f0aa2a0b640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a880b50e0 0x7f0a880b75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.267+0000 7f0aa220a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7ed890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.267+0000 7f0aa220a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7ed890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:43778/0 (socket says 172.21.15.16:43778) 2024-08-28T19:37:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.267+0000 7f0aa220a640 1 -- 172.21.15.16:0/3733564389 learned_addr learned my addr 172.21.15.16:0/3733564389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.267+0000 7f0a937fe640 1 -- 172.21.15.16:0/4070216977 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7f0a98005020 con 0x7f0a9c109b90 2024-08-28T19:37:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.267+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/3733564389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd427390 0x55e0dd7eddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.267+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/3733564389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd448070 0x55e0dd7ed350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.268+0000 7f0aa220a640 1 -- 172.21.15.16:0/3733564389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd427390 msgr2=0x55e0dd7eddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.268+0000 7f0aa220a640 1 --2- 172.21.15.16:0/3733564389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd427390 0x55e0dd7eddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.268+0000 7f0aa220a640 1 -- 172.21.15.16:0/3733564389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd448070 msgr2=0x55e0dd7ed350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.268+0000 7f0aa220a640 1 --2- 172.21.15.16:0/3733564389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd448070 0x55e0dd7ed350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.268+0000 7f0aa220a640 1 -- 172.21.15.16:0/3733564389 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0dd4b3650 con 0x55e0dd41f1b0 2024-08-28T19:37:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.268+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/3733564389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd448070 0x55e0dd7ed350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:37:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.269+0000 7f0aa2a0b640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a880b50e0 0x7f0a880b75f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0a980e26e0 tx=0x7f0a9809c040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.270+0000 7f0aa220a640 1 --2- 172.21.15.16:0/3733564389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7ed890 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0a9402ac30 tx=0x7f0a940093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0c2adb1372b9142 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.271+0000 7f0a90d9a640 1 -- 172.21.15.16:0/3733564389 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94040030 con 0x55e0dd41f1b0 2024-08-28T19:37:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.271+0000 7f0a90d9a640 1 -- 172.21.15.16:0/3733564389 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a94003e20 con 0x55e0dd41f1b0 2024-08-28T19:37:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.272+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3733564389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 msgr2=0x55e0dd7ed890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.272+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/3733564389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7ed890 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0a9402ac30 tx=0x7f0a940093e0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.273+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3733564389 shutdown_connections 2024-08-28T19:37:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.273+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/3733564389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd427390 0x55e0dd7eddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.273+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/3733564389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7ed890 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.273+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/3733564389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd448070 0x55e0dd7ed350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.273+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3733564389 >> 172.21.15.16:0/3733564389 conn(0x55e0dd43c460 msgr2=0x55e0dd7eb240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.275+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3733564389 shutdown_connections 2024-08-28T19:37:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.275+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3733564389 wait complete. 2024-08-28T19:37:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.276+0000 7f0aaa9e3740 1 -- start start 2024-08-28T19:37:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.276+0000 7f0aaa9e3740 1 Processor -- start 2024-08-28T19:37:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.277+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7f8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.277+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd427390 0x55e0dd7f8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.277+0000 7f0aa1a09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7f8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.277+0000 7f0aa1a09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7f8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47610/0 (socket says 172.21.15.16:47610) 2024-08-28T19:37:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.277+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd448070 0x55e0dd804e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.277+0000 7f0aa1a09640 1 -- 172.21.15.16:0/2936778188 learned_addr learned my addr 172.21.15.16:0/2936778188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.278+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2936778188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e0dd5b6700 con 0x55e0dd448070 2024-08-28T19:37:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.278+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2936778188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e0dd6da750 con 0x55e0dd41f1b0 2024-08-28T19:37:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.278+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2936778188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e0dd7fbb20 con 0x55e0dd427390 2024-08-28T19:37:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.278+0000 7f0aaa9e3740 10 client.0 _ll_register_callbacks cb 0x55e0dd709e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.278+0000 7f0aa220a640 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd427390 0x55e0dd7f8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.278+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd448070 0x55e0dd804e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.279+0000 7f0aa1a09640 1 -- 172.21.15.16:0/2936778188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd427390 msgr2=0x55e0dd7f8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.279+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd427390 0x55e0dd7f8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.279+0000 7f0aa1a09640 1 -- 172.21.15.16:0/2936778188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd448070 msgr2=0x55e0dd804e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.279+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd448070 0x55e0dd804e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.279+0000 7f0aa1a09640 1 -- 172.21.15.16:0/2936778188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0dd713900 con 0x55e0dd41f1b0 2024-08-28T19:37:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.279+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7f8a10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c012e40 tx=0x7f0a8c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ace943f7593b61 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.279+0000 7f0a85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.281+0000 7f0a85ffb640 1 -- 172.21.15.16:0/2936778188 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c019040 con 0x55e0dd41f1b0 2024-08-28T19:37:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.281+0000 7f0a85ffb640 1 -- 172.21.15.16:0/2936778188 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7f0a8c012110 con 0x55e0dd41f1b0 2024-08-28T19:37:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.282+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2936778188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a94003e20 con 0x55e0dd41f1b0 2024-08-28T19:37:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.282+0000 7f0a85ffb640 1 -- 172.21.15.16:0/2936778188 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f0a8c005020 con 0x55e0dd41f1b0 2024-08-28T19:37:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.282+0000 7f0aaa9e3740 10 client.28931 fetch_fsmap learned FSMap version 88 2024-08-28T19:37:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.282+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2936778188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=4+}) -- 0x55e0dd807930 con 0x55e0dd41f1b0 2024-08-28T19:37:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.283+0000 7f0a85ffb640 1 -- 172.21.15.16:0/2936778188 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a94003e20 con 0x55e0dd41f1b0 2024-08-28T19:37:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.283+0000 7f0a85ffb640 1 -- 172.21.15.16:0/2936778188 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 88) ==== 8418+0+0 (secure 0 0 0) 0x7f0a8c004b20 con 0x55e0dd41f1b0 2024-08-28T19:37:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.283+0000 7f0aaa9e3740 10 client.28931 fetch_fsmap finished waiting for FSMap version 88 2024-08-28T19:37:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.283+0000 7f0aaa9e3740 10 client.28931 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:37:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.283+0000 7f0aaa9e3740 20 client.28931 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.283+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e0dd4069f0 0x55e0dd80ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.283+0000 7f0aaa9e3740 4 client.28931 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:37:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.283+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2936778188 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0dd7665d0 con 0x55e0dd4069f0 2024-08-28T19:37:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.284+0000 7f0aa220a640 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e0dd4069f0 0x55e0dd80ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.287+0000 7f0a85ffb640 10 client.28931 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:37:48.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.287+0000 7f0aa220a640 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e0dd4069f0 0x55e0dd80ae00 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=351cdb6d9ea12fee server_cookie=be742328fb92f566 in_seq=0 out_seq=0 2024-08-28T19:37:48.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.288+0000 7f0a85ffb640 1 -- 172.21.15.16:0/2936778188 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e0dd7665d0 con 0x55e0dd4069f0 2024-08-28T19:37:48.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.288+0000 7f0a85ffb640 10 client.28931 handle_command_reply: tid=0 2024-08-28T19:37:48.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.288+0000 7f0aaa9e3740 1 client.28931 shutdown 2024-08-28T19:37:48.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2936778188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd41f1b0 msgr2=0x55e0dd7f8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:48.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7f8a10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c012e40 tx=0x7f0a8c009b90 comp rx=0 tx=0).stop 2024-08-28T19:37:48.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2936778188 shutdown_connections 2024-08-28T19:37:48.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e0dd4069f0 0x55e0dd80ae00 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd448070 0x55e0dd804e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd427390 0x55e0dd7f8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/2936778188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7f8a10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2936778188 >> 172.21.15.16:0/2936778188 conn(0x55e0dd43c460 msgr2=0x55e0dd7f0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:48.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0a85ffb640 0 client.28931 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:37:48.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2936778188 shutdown_connections 2024-08-28T19:37:48.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2936778188 wait complete. 2024-08-28T19:37:48.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.290+0000 7f0aaa9e3740 20 client.28931 trim_cache size 0 max 16384 2024-08-28T19:37:48.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.312+0000 7f0aaa9e3740 1 Processor -- start 2024-08-28T19:37:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.313+0000 7f0aaa9e3740 1 -- start start 2024-08-28T19:37:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.313+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7f8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.313+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd427390 0x55e0dd7ea810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.313+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd448070 0x55e0dd7ead50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.313+0000 7f0aaa9e3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1b74b0 con 0x55e0dd41f1b0 2024-08-28T19:37:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.313+0000 7f0aaa9e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1b9200 con 0x55e0dd427390 2024-08-28T19:37:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.313+0000 7f0aaa9e3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1b7c40 con 0x55e0dd448070 2024-08-28T19:37:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.313+0000 7f0aa1a09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7f8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.313+0000 7f0aa1a09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7f8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50816/0 (socket says 172.21.15.16:50816) 2024-08-28T19:37:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.314+0000 7f0aa1a09640 1 -- 172.21.15.16:0/2964840313 learned_addr learned my addr 172.21.15.16:0/2964840313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.314+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/2964840313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd448070 0x55e0dd7ead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.314+0000 7f0aa220a640 1 --2- 172.21.15.16:0/2964840313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd427390 0x55e0dd7ea810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.314+0000 7f0aa1a09640 1 -- 172.21.15.16:0/2964840313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd427390 msgr2=0x55e0dd7ea810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.314+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/2964840313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd427390 0x55e0dd7ea810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.314+0000 7f0aa1a09640 1 -- 172.21.15.16:0/2964840313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd41f1b0 msgr2=0x55e0dd7f8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.314+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/2964840313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7f8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.314+0000 7f0aa1a09640 1 -- 172.21.15.16:0/2964840313 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0dd7ef640 con 0x55e0dd448070 2024-08-28T19:37:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.315+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/2964840313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd448070 0x55e0dd7ead50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c004ff0 tx=0x7f0a8c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8909bfe19a0433cb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.316+0000 7f0a87fff640 1 -- 172.21.15.16:0/2964840313 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c030020 con 0x55e0dd448070 2024-08-28T19:37:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.316+0000 7f0a87fff640 1 -- 172.21.15.16:0/2964840313 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a8c020640 con 0x55e0dd448070 2024-08-28T19:37:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.316+0000 7f0a87fff640 1 -- 172.21.15.16:0/2964840313 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c0209b0 con 0x55e0dd448070 2024-08-28T19:37:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.317+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2964840313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd448070 msgr2=0x55e0dd7ead50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.317+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/2964840313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd448070 0x55e0dd7ead50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c004ff0 tx=0x7f0a8c005fd0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.318+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2964840313 shutdown_connections 2024-08-28T19:37:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.318+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/2964840313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd448070 0x55e0dd7ead50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.318+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/2964840313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd427390 0x55e0dd7ea810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.318+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/2964840313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd41f1b0 0x55e0dd7f8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.318+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2964840313 >> 172.21.15.16:0/2964840313 conn(0x55e0dd43c460 msgr2=0x55e0dd7f87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.318+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2964840313 shutdown_connections 2024-08-28T19:37:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.318+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/2964840313 wait complete. 2024-08-28T19:37:48.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.319+0000 7f0aaa9e3740 1 -- start start 2024-08-28T19:37:48.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.319+0000 7f0aaa9e3740 1 Processor -- start 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.319+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd86ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.319+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd427390 0x55e0dd86f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.319+0000 7f0aaa9e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd448070 0x55e0dd800dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.319+0000 7f0aaa9e3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1b8a70 con 0x55e0dd427390 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.319+0000 7f0aaa9e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1b8340 con 0x55e0dd448070 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.319+0000 7f0aaa9e3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1b54b0 con 0x55e0dd41f1b0 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aa1a09640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd86ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aa1a09640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd86ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:43802/0 (socket says 172.21.15.16:43802) 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aa1a09640 1 -- 172.21.15.16:0/3017504313 learned_addr learned my addr 172.21.15.16:0/3017504313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aa220a640 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd427390 0x55e0dd86f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd448070 0x55e0dd800dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aaa9e3740 10 client.0 _ll_register_callbacks cb 0x7f0a8c009a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aa1a09640 1 -- 172.21.15.16:0/3017504313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd448070 msgr2=0x55e0dd800dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd448070 0x55e0dd800dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aa1a09640 1 -- 172.21.15.16:0/3017504313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd427390 msgr2=0x55e0dd86f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd427390 0x55e0dd86f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aa1a09640 1 -- 172.21.15.16:0/3017504313 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0dd801300 con 0x55e0dd41f1b0 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0aa1a09640 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd86ef00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009d40 tx=0x7f0a8c012370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=193e2bf81c793ae1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.320+0000 7f0a85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.321+0000 7f0a85ffb640 1 -- 172.21.15.16:0/3017504313 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c030070 con 0x55e0dd41f1b0 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.321+0000 7f0a85ffb640 1 -- 172.21.15.16:0/3017504313 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7f0a8c0209b0 con 0x55e0dd41f1b0 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.321+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3017504313 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a8c020640 con 0x55e0dd41f1b0 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.321+0000 7f0a85ffb640 1 -- 172.21.15.16:0/3017504313 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f0a8c020bb0 con 0x55e0dd41f1b0 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.322+0000 7f0aaa9e3740 10 client.28540 fetch_fsmap learned FSMap version 88 2024-08-28T19:37:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.322+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3017504313 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=4+}) -- 0x55e0dd886f50 con 0x55e0dd41f1b0 2024-08-28T19:37:48.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.322+0000 7f0a85ffb640 1 -- 172.21.15.16:0/3017504313 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a8c004b30 con 0x55e0dd41f1b0 2024-08-28T19:37:48.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.323+0000 7f0a85ffb640 1 -- 172.21.15.16:0/3017504313 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 88) ==== 8418+0+0 (secure 0 0 0) 0x7f0a8c010660 con 0x55e0dd41f1b0 2024-08-28T19:37:48.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.323+0000 7f0aaa9e3740 10 client.28540 fetch_fsmap finished waiting for FSMap version 88 2024-08-28T19:37:48.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.323+0000 7f0aaa9e3740 10 client.28540 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:37:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.323+0000 7f0aaa9e3740 20 client.28540 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.323+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e0dd3dbda0 0x55e0dd889690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.323+0000 7f0aaa9e3740 4 client.28540 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:37:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.323+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3017504313 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f0a8c020640 con 0x55e0dd3dbda0 2024-08-28T19:37:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.323+0000 7f0aa220a640 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e0dd3dbda0 0x55e0dd889690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.324+0000 7f0aa220a640 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e0dd3dbda0 0x55e0dd889690 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3fce833d1edb2d07 server_cookie=cd2772ba7d1ffc56 in_seq=0 out_seq=0 2024-08-28T19:37:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.324+0000 7f0a85ffb640 10 client.28540 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:37:48.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.325+0000 7f0a85ffb640 1 -- 172.21.15.16:0/3017504313 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f0a8c020640 con 0x55e0dd3dbda0 2024-08-28T19:37:48.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.325+0000 7f0a85ffb640 10 client.28540 handle_command_reply: tid=0 2024-08-28T19:37:48.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.326+0000 7f0aaa9e3740 1 client.28540 shutdown 2024-08-28T19:37:48.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.326+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3017504313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 msgr2=0x55e0dd86ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:48.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.326+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd86ef00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009d40 tx=0x7f0a8c012370 comp rx=0 tx=0).stop 2024-08-28T19:37:48.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.326+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3017504313 shutdown_connections 2024-08-28T19:37:48.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.326+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e0dd3dbda0 0x55e0dd889690 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.326+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dd448070 0x55e0dd800dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.326+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0dd427390 0x55e0dd86f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.326+0000 7f0aaa9e3740 1 --2- 172.21.15.16:0/3017504313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e0dd41f1b0 0x55e0dd86ef00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.326+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3017504313 >> 172.21.15.16:0/3017504313 conn(0x55e0dd43c460 msgr2=0x55e0dd7f0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.326+0000 7f0a85ffb640 0 client.28540 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:37:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.327+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3017504313 shutdown_connections 2024-08-28T19:37:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.327+0000 7f0aaa9e3740 1 -- 172.21.15.16:0/3017504313 wait complete. 2024-08-28T19:37:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.327+0000 7f0aaa9e3740 20 client.28540 trim_cache size 0 max 16384 2024-08-28T19:37:48.329 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:37:48.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.328+0000 7f0aa446e640 1 -- 172.21.15.16:0/4070216977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a880b50e0 msgr2=0x7f0a880b75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.328+0000 7f0aa446e640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a880b50e0 0x7f0a880b75f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0a980e26e0 tx=0x7f0a9809c040 comp rx=0 tx=0).stop 2024-08-28T19:37:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.328+0000 7f0aa446e640 1 -- 172.21.15.16:0/4070216977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a9c109b90 msgr2=0x7f0a9c1a60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.328+0000 7f0aa446e640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a9c109b90 0x7f0a9c1a60d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800d750 tx=0x7f0a9800db50 comp rx=0 tx=0).stop 2024-08-28T19:37:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.328+0000 7f0aa446e640 1 -- 172.21.15.16:0/4070216977 shutdown_connections 2024-08-28T19:37:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.328+0000 7f0aa446e640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a9c109b90 0x7f0a9c1a60d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.328+0000 7f0aa446e640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c109240 0x7f0a9c1a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.328+0000 7f0aa446e640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a880b50e0 0x7f0a880b75f0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.328+0000 7f0aa446e640 1 --2- 172.21.15.16:0/4070216977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c103900 0x7f0a9c1a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.328+0000 7f0aa446e640 1 -- 172.21.15.16:0/4070216977 >> 172.21.15.16:0/4070216977 conn(0x7f0a9c07b5f0 msgr2=0x7f0a9c107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.329+0000 7f0aa446e640 1 -- 172.21.15.16:0/4070216977 shutdown_connections 2024-08-28T19:37:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:48.329+0000 7f0aa446e640 1 -- 172.21.15.16:0/4070216977 wait complete. 2024-08-28T19:37:48.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 73586 22.62 MB/sec execute 84 sec latency 106.005 ms 2024-08-28T19:37:49.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:48 smithi016 ceph-mon[18556]: pgmap v862: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 9.2 MiB/s wr, 809 op/s 2024-08-28T19:37:49.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:48 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:48 smithi028 ceph-mon[21769]: pgmap v862: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 9.2 MiB/s wr, 809 op/s 2024-08-28T19:37:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:48 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:49.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:37:49.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:48 smithi115 ceph-mon[22184]: pgmap v862: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 9.2 MiB/s wr, 809 op/s 2024-08-28T19:37:49.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:48 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:37:49.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 74265 22.78 MB/sec execute 85 sec latency 293.197 ms 2024-08-28T19:37:49.661 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.433+0000 7fa847ca4640 1 -- 172.21.15.16:0/369176451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa840104750 msgr2=0x7fa840104b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.433+0000 7fa847ca4640 1 --2- 172.21.15.16:0/369176451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa840104750 0x7fa840104b60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009860 tx=0x7fa83c02a340 comp rx=0 tx=0).stop 2024-08-28T19:37:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.433+0000 7fa847ca4640 1 -- 172.21.15.16:0/369176451 shutdown_connections 2024-08-28T19:37:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.433+0000 7fa847ca4640 1 --2- 172.21.15.16:0/369176451 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa84010dbb0 0x7fa8401100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.433+0000 7fa847ca4640 1 --2- 172.21.15.16:0/369176451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8401050a0 0x7fa84010d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.433+0000 7fa847ca4640 1 --2- 172.21.15.16:0/369176451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa840104750 0x7fa840104b60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.433+0000 7fa847ca4640 1 -- 172.21.15.16:0/369176451 >> 172.21.15.16:0/369176451 conn(0x7fa8400794d0 msgr2=0x7fa8400798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.434+0000 7fa847ca4640 1 -- 172.21.15.16:0/369176451 shutdown_connections 2024-08-28T19:37:50.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.434+0000 7fa847ca4640 1 -- 172.21.15.16:0/369176451 wait complete. 2024-08-28T19:37:50.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa847ca4640 1 Processor -- start 2024-08-28T19:37:50.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa847ca4640 1 -- start start 2024-08-28T19:37:50.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa847ca4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa840104750 0x7fa84019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa847ca4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8401050a0 0x7fa84019a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa847ca4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa84010dbb0 0x7fa84019fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa847ca4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa840113830 con 0x7fa8401050a0 2024-08-28T19:37:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa847ca4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8401136b0 con 0x7fa840104750 2024-08-28T19:37:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa847ca4640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa8401139b0 con 0x7fa84010dbb0 2024-08-28T19:37:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa845a40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa840104750 0x7fa84019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa845a40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa840104750 0x7fa84019a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47648/0 (socket says 172.21.15.16:47648) 2024-08-28T19:37:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa845a40640 1 -- 172.21.15.16:0/3588535292 learned_addr learned my addr 172.21.15.16:0/3588535292 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa84523f640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8401050a0 0x7fa84019a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.435+0000 7fa846241640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa84010dbb0 0x7fa84019fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.436+0000 7fa845a40640 1 -- 172.21.15.16:0/3588535292 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa84010dbb0 msgr2=0x7fa84019fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.436+0000 7fa845a40640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa84010dbb0 0x7fa84019fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.436+0000 7fa845a40640 1 -- 172.21.15.16:0/3588535292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8401050a0 msgr2=0x7fa84019a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.436+0000 7fa845a40640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8401050a0 0x7fa84019a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.436+0000 7fa845a40640 1 -- 172.21.15.16:0/3588535292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8401a0210 con 0x7fa840104750 2024-08-28T19:37:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.436+0000 7fa845a40640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa840104750 0x7fa84019a2a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa83c000c00 tx=0x7fa83c032d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.436+0000 7fa836ffd640 1 -- 172.21.15.16:0/3588535292 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c040020 con 0x7fa840104750 2024-08-28T19:37:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.436+0000 7fa836ffd640 1 -- 172.21.15.16:0/3588535292 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c030e10 con 0x7fa840104750 2024-08-28T19:37:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.436+0000 7fa836ffd640 1 -- 172.21.15.16:0/3588535292 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c037040 con 0x7fa840104750 2024-08-28T19:37:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.436+0000 7fa847ca4640 1 -- 172.21.15.16:0/3588535292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8401a04a0 con 0x7fa840104750 2024-08-28T19:37:50.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.436+0000 7fa847ca4640 1 -- 172.21.15.16:0/3588535292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8401a6c60 con 0x7fa840104750 2024-08-28T19:37:50.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.439+0000 7fa84e219740 1 Processor -- start 2024-08-28T19:37:50.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa84e219740 1 -- start start 2024-08-28T19:37:50.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa836ffd640 1 -- 172.21.15.16:0/3588535292 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa83c043020 con 0x7fa840104750 2024-08-28T19:37:50.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa84e219740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c9643a070 0x557c967df310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa84e219740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c964111b0 0x557c967df850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa84e219740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c96419390 0x557c967dfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa84e219740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557c96756680 con 0x557c96419390 2024-08-28T19:37:50.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa84e219740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557c96745c10 con 0x557c964111b0 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa84e219740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557c96749bc0 con 0x557c9643a070 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa845a40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c964111b0 0x557c967df850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa846241640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c9643a070 0x557c967df310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa846241640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c9643a070 0x557c967df310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:43842/0 (socket says 172.21.15.16:43842) 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.441+0000 7fa846241640 1 -- 172.21.15.16:0/2749106961 learned_addr learned my addr 172.21.15.16:0/2749106961 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.440+0000 7fa845a40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c964111b0 0x557c967df850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47662/0 (socket says 172.21.15.16:47662) 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.441+0000 7fa846241640 1 -- 172.21.15.16:0/2749106961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c964111b0 msgr2=0x557c967df850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.441+0000 7fa846241640 1 --2- 172.21.15.16:0/2749106961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c964111b0 0x557c967df850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.441+0000 7fa846241640 1 -- 172.21.15.16:0/2749106961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c96419390 msgr2=0x557c967dfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.441+0000 7fa846241640 1 --2- 172.21.15.16:0/2749106961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c96419390 0x557c967dfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.441+0000 7fa846241640 1 -- 172.21.15.16:0/2749106961 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c964a5650 con 0x557c9643a070 2024-08-28T19:37:50.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.442+0000 7fa846241640 1 --2- 172.21.15.16:0/2749106961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c9643a070 0x557c967df310 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa838009fd0 tx=0x7fa8380118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15b0714d8efbc0cd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:50.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.442+0000 7fa836ffd640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa82c0b4d20 0x7fa82c0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.442+0000 7fa82bfff640 1 -- 172.21.15.16:0/2749106961 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa838004d20 con 0x557c9643a070 2024-08-28T19:37:50.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.442+0000 7fa82bfff640 1 -- 172.21.15.16:0/2749106961 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa838004050 con 0x557c9643a070 2024-08-28T19:37:50.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.442+0000 7fa845a40640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa82c0b4d20 0x7fa82c0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.442+0000 7fa82bfff640 1 -- 172.21.15.16:0/2749106961 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa838005280 con 0x557c9643a070 2024-08-28T19:37:50.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.442+0000 7fa836ffd640 1 -- 172.21.15.16:0/3588535292 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fa83c1057e0 con 0x7fa840104750 2024-08-28T19:37:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.443+0000 7fa84e219740 1 -- 172.21.15.16:0/2749106961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c9643a070 msgr2=0x557c967df310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.443+0000 7fa845a40640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa82c0b4d20 0x7fa82c0b7230 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x557c967ddf90 tx=0x7fa83c10f940 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.444+0000 7fa84e219740 1 --2- 172.21.15.16:0/2749106961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c9643a070 0x557c967df310 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa838009fd0 tx=0x7fa8380118d0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.444+0000 7fa84e219740 1 -- 172.21.15.16:0/2749106961 shutdown_connections 2024-08-28T19:37:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.444+0000 7fa84e219740 1 --2- 172.21.15.16:0/2749106961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c96419390 0x557c967dfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.444+0000 7fa84e219740 1 --2- 172.21.15.16:0/2749106961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c964111b0 0x557c967df850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.444+0000 7fa84e219740 1 --2- 172.21.15.16:0/2749106961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c9643a070 0x557c967df310 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.444+0000 7fa84e219740 1 -- 172.21.15.16:0/2749106961 >> 172.21.15.16:0/2749106961 conn(0x557c9642e460 msgr2=0x557c967dd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.444+0000 7fa84e219740 1 -- 172.21.15.16:0/2749106961 shutdown_connections 2024-08-28T19:37:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.444+0000 7fa84e219740 1 -- 172.21.15.16:0/2749106961 wait complete. 2024-08-28T19:37:50.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.445+0000 7fa84e219740 1 -- start start 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.445+0000 7fa84e219740 1 Processor -- start 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.445+0000 7fa84e219740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c964111b0 0x557c967eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.445+0000 7fa84e219740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 0x557c967eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.445+0000 7fa84e219740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c9643a070 0x557c967f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.445+0000 7fa84e219740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557c965a8700 con 0x557c964111b0 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.445+0000 7fa846241640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c964111b0 0x557c967eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa84e219740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557c966cc750 con 0x557c96419390 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa846241640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c964111b0 0x557c967eaa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50860/0 (socket says 172.21.15.16:50860) 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa84523f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 0x557c967eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa846241640 1 -- 172.21.15.16:0/3084944313 learned_addr learned my addr 172.21.15.16:0/3084944313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa84e219740 1 -- 172.21.15.16:0/3084944313 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557c967edaf0 con 0x557c9643a070 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa84e219740 10 client.0 _ll_register_callbacks cb 0x557c966fbe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa84523f640 1 -- 172.21.15.16:0/3084944313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c9643a070 msgr2=0x557c967f6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:50.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa84523f640 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c9643a070 0x557c967f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa84523f640 1 -- 172.21.15.16:0/3084944313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c964111b0 msgr2=0x557c967eaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa84523f640 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c964111b0 0x557c967eaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa84523f640 1 -- 172.21.15.16:0/3084944313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c96705900 con 0x557c96419390 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa846241640 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c964111b0 0x557c967eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa84e219740 1 -- 172.21.15.16:0/3084944313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa838004050 con 0x557c96419390 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa84523f640 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 0x557c967eaf80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa840079e40 tx=0x7fa830011b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfd5b6f17067dbfc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.446+0000 7fa8297fa640 10 client.28949 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa8297fa640 1 -- 172.21.15.16:0/3084944313 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83001e060 con 0x557c96419390 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa8297fa640 1 -- 172.21.15.16:0/3084944313 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fa83000fcc0 con 0x557c96419390 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa8297fa640 1 -- 172.21.15.16:0/3084944313 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fa830019430 con 0x557c96419390 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa84e219740 10 client.28949 fetch_fsmap learned FSMap version 88 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa84e219740 1 -- 172.21.15.16:0/3084944313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=88}) -- 0x557c967f98a0 con 0x557c96419390 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa8297fa640 1 -- 172.21.15.16:0/3084944313 <== mon.2 v2:172.21.15.28:3300/0 4 ==== fsmap(e 88) ==== 8418+0+0 (secure 0 0 0) 0x7fa8300223d0 con 0x557c96419390 2024-08-28T19:37:50.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa84e219740 10 client.28949 fetch_fsmap finished waiting for FSMap version 88 2024-08-28T19:37:50.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa84e219740 10 client.28949 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:37:50.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa84e219740 20 client.28949 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:50.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa84e219740 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557c963f89f0 0x557c967fcca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa84e219740 4 client.28949 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:37:50.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa84e219740 1 -- 172.21.15.16:0/3084944313 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c967585d0 con 0x557c963f89f0 2024-08-28T19:37:50.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.447+0000 7fa846241640 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557c963f89f0 0x557c967fcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.448+0000 7fa846241640 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557c963f89f0 0x557c967fcca0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b4152092ddc96967 server_cookie=7f97338441137cec in_seq=0 out_seq=0 2024-08-28T19:37:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.448+0000 7fa8297fa640 10 client.28949 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:37:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.449+0000 7fa8297fa640 1 -- 172.21.15.16:0/3084944313 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557c967585d0 con 0x557c963f89f0 2024-08-28T19:37:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.449+0000 7fa8297fa640 10 client.28949 handle_command_reply: tid=0 2024-08-28T19:37:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.449+0000 7fa84e219740 1 client.28949 shutdown 2024-08-28T19:37:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa84e219740 1 -- 172.21.15.16:0/3084944313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 msgr2=0x557c967eaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa84e219740 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 0x557c967eaf80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa840079e40 tx=0x7fa830011b50 comp rx=0 tx=0).stop 2024-08-28T19:37:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa84e219740 1 -- 172.21.15.16:0/3084944313 shutdown_connections 2024-08-28T19:37:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa84e219740 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557c963f89f0 0x557c967fcca0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa84e219740 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c9643a070 0x557c967f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa84e219740 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 0x557c967eaf80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa84e219740 1 --2- 172.21.15.16:0/3084944313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c964111b0 0x557c967eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa84e219740 1 -- 172.21.15.16:0/3084944313 >> 172.21.15.16:0/3084944313 conn(0x557c9642e460 msgr2=0x557c967e24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa8297fa640 0 client.28949 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:37:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa84e219740 1 -- 172.21.15.16:0/3084944313 shutdown_connections 2024-08-28T19:37:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa84e219740 1 -- 172.21.15.16:0/3084944313 wait complete. 2024-08-28T19:37:50.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.450+0000 7fa84e219740 20 client.28949 trim_cache size 0 max 16384 2024-08-28T19:37:50.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.466+0000 7fa84e219740 1 Processor -- start 2024-08-28T19:37:50.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.466+0000 7fa84e219740 1 -- start start 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84e219740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c964111b0 0x557c967ec920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84e219740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 0x557c967f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84e219740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c9643a070 0x557c967e0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84e219740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8401aeae0 con 0x557c9643a070 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84e219740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8401b0830 con 0x557c96419390 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84e219740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa8401af270 con 0x557c964111b0 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84523f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 0x557c967f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa846241640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c9643a070 0x557c967e0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84523f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 0x557c967f6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47686/0 (socket says 172.21.15.16:47686) 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84523f640 1 -- 172.21.15.16:0/463606753 learned_addr learned my addr 172.21.15.16:0/463606753 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa846241640 1 --2- 172.21.15.16:0/463606753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c964111b0 0x557c967ec920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84523f640 1 -- 172.21.15.16:0/463606753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c964111b0 msgr2=0x557c967ec920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84523f640 1 --2- 172.21.15.16:0/463606753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c964111b0 0x557c967ec920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84523f640 1 -- 172.21.15.16:0/463606753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c9643a070 msgr2=0x557c967e0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84523f640 1 --2- 172.21.15.16:0/463606753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c9643a070 0x557c967e0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa84523f640 1 -- 172.21.15.16:0/463606753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c967e18d0 con 0x557c96419390 2024-08-28T19:37:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.467+0000 7fa846241640 1 --2- 172.21.15.16:0/463606753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c9643a070 0x557c967e0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.468+0000 7fa84523f640 1 --2- 172.21.15.16:0/463606753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 0x557c967f6d10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa830000c00 tx=0x7fa830014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=122e9453177481 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.468+0000 7fa82b7fe640 1 -- 172.21.15.16:0/463606753 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83001e070 con 0x557c96419390 2024-08-28T19:37:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.468+0000 7fa82b7fe640 1 -- 172.21.15.16:0/463606753 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa838004050 con 0x557c96419390 2024-08-28T19:37:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.468+0000 7fa82b7fe640 1 -- 172.21.15.16:0/463606753 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa830022b30 con 0x557c96419390 2024-08-28T19:37:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.469+0000 7fa84e219740 1 -- 172.21.15.16:0/463606753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 msgr2=0x557c967f6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.469+0000 7fa84e219740 1 --2- 172.21.15.16:0/463606753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 0x557c967f6d10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa830000c00 tx=0x7fa830014000 comp rx=0 tx=0).stop 2024-08-28T19:37:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.469+0000 7fa84e219740 1 -- 172.21.15.16:0/463606753 shutdown_connections 2024-08-28T19:37:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.469+0000 7fa84e219740 1 --2- 172.21.15.16:0/463606753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c9643a070 0x557c967e0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.469+0000 7fa84e219740 1 --2- 172.21.15.16:0/463606753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c96419390 0x557c967f6d10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.469+0000 7fa84e219740 1 --2- 172.21.15.16:0/463606753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c964111b0 0x557c967ec920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.469+0000 7fa84e219740 1 -- 172.21.15.16:0/463606753 >> 172.21.15.16:0/463606753 conn(0x557c9642e460 msgr2=0x557c967e01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.469+0000 7fa84e219740 1 -- 172.21.15.16:0/463606753 shutdown_connections 2024-08-28T19:37:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.469+0000 7fa84e219740 1 -- 172.21.15.16:0/463606753 wait complete. 2024-08-28T19:37:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.470+0000 7fa84e219740 1 -- start start 2024-08-28T19:37:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.470+0000 7fa84e219740 1 Processor -- start 2024-08-28T19:37:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa84e219740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c964111b0 0x557c967f1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa84e219740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c96419390 0x557c967f1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa84e219740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c9643a070 0x557c96866670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa84e219740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8401b00a0 con 0x557c96419390 2024-08-28T19:37:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa84e219740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8401af970 con 0x557c9643a070 2024-08-28T19:37:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa84e219740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa8401acae0 con 0x557c964111b0 2024-08-28T19:37:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa84e219740 10 client.0 _ll_register_callbacks cb 0x7fa830009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:50.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 74936 22.85 MB/sec execute 86 sec latency 203.415 ms 2024-08-28T19:37:50.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa84523f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c96419390 0x557c967f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa846241640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c964111b0 0x557c967f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa84523f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c96419390 0x557c967f1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50874/0 (socket says 172.21.15.16:50874) 2024-08-28T19:37:50.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa84523f640 1 -- 172.21.15.16:0/3041491763 learned_addr learned my addr 172.21.15.16:0/3041491763 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:50.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.471+0000 7fa846241640 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c9643a070 0x557c96866670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa846241640 1 -- 172.21.15.16:0/3041491763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c964111b0 msgr2=0x557c967f1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa846241640 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c964111b0 0x557c967f1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa846241640 1 -- 172.21.15.16:0/3041491763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c96419390 msgr2=0x557c967f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa846241640 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c96419390 0x557c967f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa846241640 1 -- 172.21.15.16:0/3041491763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c96866bb0 con 0x557c9643a070 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa84523f640 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c96419390 0x557c967f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa846241640 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c9643a070 0x557c96866670 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa838009fa0 tx=0x7fa8380118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc3479d101849bc7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa8297fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa8297fa640 1 -- 172.21.15.16:0/3041491763 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83800f8f0 con 0x557c9643a070 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa8297fa640 1 -- 172.21.15.16:0/3041491763 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fa83800fa90 con 0x557c9643a070 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa84e219740 1 -- 172.21.15.16:0/3041491763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa838004050 con 0x557c9643a070 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa8297fa640 1 -- 172.21.15.16:0/3041491763 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fa83800fc90 con 0x557c9643a070 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa84e219740 10 client.28955 fetch_fsmap learned FSMap version 88 2024-08-28T19:37:50.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa84e219740 1 -- 172.21.15.16:0/3041491763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=4+}) -- 0x557c96867d60 con 0x557c9643a070 2024-08-28T19:37:50.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.472+0000 7fa8297fa640 1 -- 172.21.15.16:0/3041491763 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83800fe30 con 0x557c9643a070 2024-08-28T19:37:50.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.473+0000 7fa8297fa640 1 -- 172.21.15.16:0/3041491763 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 88) ==== 8418+0+0 (secure 0 0 0) 0x7fa838029450 con 0x557c9643a070 2024-08-28T19:37:50.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.473+0000 7fa84e219740 10 client.28955 fetch_fsmap finished waiting for FSMap version 88 2024-08-28T19:37:50.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.473+0000 7fa84e219740 10 client.28955 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:37:50.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.473+0000 7fa84e219740 20 client.28955 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:50.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.473+0000 7fa84e219740 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557c963cdda0 0x557c9686a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:50.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.473+0000 7fa84e219740 4 client.28955 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:37:50.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.473+0000 7fa84e219740 1 -- 172.21.15.16:0/3041491763 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fa838004050 con 0x557c963cdda0 2024-08-28T19:37:50.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.473+0000 7fa84523f640 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557c963cdda0 0x557c9686a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:50.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.474+0000 7fa8297fa640 10 client.28955 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:37:50.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.474+0000 7fa84523f640 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557c963cdda0 0x557c9686a250 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9f5dd42894b2bdde server_cookie=d3b2de8539960cbb in_seq=0 out_seq=0 2024-08-28T19:37:50.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.479+0000 7fa8297fa640 1 -- 172.21.15.16:0/3041491763 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fa838004050 con 0x557c963cdda0 2024-08-28T19:37:50.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.479+0000 7fa8297fa640 10 client.28955 handle_command_reply: tid=0 2024-08-28T19:37:50.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.479+0000 7fa84e219740 1 client.28955 shutdown 2024-08-28T19:37:50.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa84e219740 1 -- 172.21.15.16:0/3041491763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c9643a070 msgr2=0x557c96866670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa84e219740 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c9643a070 0x557c96866670 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa838009fa0 tx=0x7fa8380118d0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa84e219740 1 -- 172.21.15.16:0/3041491763 shutdown_connections 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa84e219740 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x557c963cdda0 0x557c9686a250 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa84e219740 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c9643a070 0x557c96866670 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa84e219740 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557c96419390 0x557c967f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa84e219740 1 --2- 172.21.15.16:0/3041491763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557c964111b0 0x557c967f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa8297fa640 0 client.28955 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa84e219740 1 -- 172.21.15.16:0/3041491763 >> 172.21.15.16:0/3041491763 conn(0x557c9642e460 msgr2=0x557c967ebe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa84e219740 1 -- 172.21.15.16:0/3041491763 shutdown_connections 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa84e219740 1 -- 172.21.15.16:0/3041491763 wait complete. 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.480+0000 7fa84e219740 20 client.28955 trim_cache size 0 max 16384 2024-08-28T19:37:50.482 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:37:50.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 -- 172.21.15.16:0/3588535292 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa82c0b4d20 msgr2=0x7fa82c0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:50.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa82c0b4d20 0x7fa82c0b7230 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x557c967ddf90 tx=0x7fa83c10f940 comp rx=0 tx=0).stop 2024-08-28T19:37:50.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 -- 172.21.15.16:0/3588535292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa840104750 msgr2=0x7fa84019a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:50.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa840104750 0x7fa84019a2a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa83c000c00 tx=0x7fa83c032d40 comp rx=0 tx=0).stop 2024-08-28T19:37:50.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 -- 172.21.15.16:0/3588535292 shutdown_connections 2024-08-28T19:37:50.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa84010dbb0 0x7fa84019fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8401050a0 0x7fa84019a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa82c0b4d20 0x7fa82c0b7230 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 --2- 172.21.15.16:0/3588535292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa840104750 0x7fa84019a2a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:50.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 -- 172.21.15.16:0/3588535292 >> 172.21.15.16:0/3588535292 conn(0x7fa8400794d0 msgr2=0x7fa8400ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:50.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 -- 172.21.15.16:0/3588535292 shutdown_connections 2024-08-28T19:37:50.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:50.481+0000 7fa847ca4640 1 -- 172.21.15.16:0/3588535292 wait complete. 2024-08-28T19:37:51.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:50 smithi016 ceph-mon[18556]: pgmap v863: 129 pgs: 129 active+clean; 1.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 8.4 MiB/s wr, 787 op/s 2024-08-28T19:37:51.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:37:51.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:50 smithi028 ceph-mon[21769]: pgmap v863: 129 pgs: 129 active+clean; 1.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 8.4 MiB/s wr, 787 op/s 2024-08-28T19:37:51.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:50 smithi115 ceph-mon[22184]: pgmap v863: 129 pgs: 129 active+clean; 1.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 8.4 MiB/s wr, 787 op/s 2024-08-28T19:37:51.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 75710 23.10 MB/sec execute 87 sec latency 190.570 ms 2024-08-28T19:37:51.530 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:52.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.043+0000 7fe294620640 1 -- 172.21.15.16:0/1895972144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe28c109240 msgr2=0x7fe28c109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:52.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.043+0000 7fe294620640 1 --2- 172.21.15.16:0/1895972144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe28c109240 0x7fe28c109650 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009900 tx=0x7fe27c02a3f0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.043+0000 7fe294620640 1 -- 172.21.15.16:0/1895972144 shutdown_connections 2024-08-28T19:37:52.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.043+0000 7fe294620640 1 --2- 172.21.15.16:0/1895972144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28c109b90 0x7fe28c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.043+0000 7fe294620640 1 --2- 172.21.15.16:0/1895972144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe28c109240 0x7fe28c109650 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.043+0000 7fe294620640 1 --2- 172.21.15.16:0/1895972144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28c103900 0x7fe28c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.043+0000 7fe294620640 1 -- 172.21.15.16:0/1895972144 >> 172.21.15.16:0/1895972144 conn(0x7fe28c07b5f0 msgr2=0x7fe28c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:52.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.043+0000 7fe294620640 1 -- 172.21.15.16:0/1895972144 shutdown_connections 2024-08-28T19:37:52.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.043+0000 7fe294620640 1 -- 172.21.15.16:0/1895972144 wait complete. 2024-08-28T19:37:52.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.044+0000 7fe294620640 1 Processor -- start 2024-08-28T19:37:52.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.044+0000 7fe294620640 1 -- start start 2024-08-28T19:37:52.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.045+0000 7fe294620640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28c103900 0x7fe28c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.045+0000 7fe294620640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe28c109240 0x7fe28c19c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.045+0000 7fe294620640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28c109b90 0x7fe28c1a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.045+0000 7fe294620640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c115a50 con 0x7fe28c103900 2024-08-28T19:37:52.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.045+0000 7fe294620640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1158d0 con 0x7fe28c109b90 2024-08-28T19:37:52.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.045+0000 7fe294620640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c115bd0 con 0x7fe28c109240 2024-08-28T19:37:52.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.045+0000 7fe2923bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28c103900 0x7fe28c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.045+0000 7fe292bbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28c109b90 0x7fe28c1a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.045+0000 7fe291bbb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe28c109240 0x7fe28c19c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.045+0000 7fe292bbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28c109b90 0x7fe28c1a1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47700/0 (socket says 172.21.15.16:47700) 2024-08-28T19:37:52.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.045+0000 7fe292bbd640 1 -- 172.21.15.16:0/1016000227 learned_addr learned my addr 172.21.15.16:0/1016000227 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:52.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.046+0000 7fe292bbd640 1 -- 172.21.15.16:0/1016000227 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe28c109240 msgr2=0x7fe28c19c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:52.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.046+0000 7fe292bbd640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe28c109240 0x7fe28c19c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.046+0000 7fe292bbd640 1 -- 172.21.15.16:0/1016000227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28c103900 msgr2=0x7fe28c19c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:52.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.046+0000 7fe292bbd640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28c103900 0x7fe28c19c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.046+0000 7fe292bbd640 1 -- 172.21.15.16:0/1016000227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c1a2390 con 0x7fe28c109b90 2024-08-28T19:37:52.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.046+0000 7fe292bbd640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28c109b90 0x7fe28c1a1c60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe28400eb10 tx=0x7fe28400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:52.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.047+0000 7fe2837fe640 1 -- 172.21.15.16:0/1016000227 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284004b20 con 0x7fe28c109b90 2024-08-28T19:37:52.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.047+0000 7fe2837fe640 1 -- 172.21.15.16:0/1016000227 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe284004cc0 con 0x7fe28c109b90 2024-08-28T19:37:52.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.047+0000 7fe2837fe640 1 -- 172.21.15.16:0/1016000227 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284010670 con 0x7fe28c109b90 2024-08-28T19:37:52.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.047+0000 7fe294620640 1 -- 172.21.15.16:0/1016000227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28c1a2680 con 0x7fe28c109b90 2024-08-28T19:37:52.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.047+0000 7fe294620640 1 -- 172.21.15.16:0/1016000227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28c1a8de0 con 0x7fe28c109b90 2024-08-28T19:37:52.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.051+0000 7fe29ab95740 1 Processor -- start 2024-08-28T19:37:52.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.051+0000 7fe2837fe640 1 -- 172.21.15.16:0/1016000227 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe284005020 con 0x7fe28c109b90 2024-08-28T19:37:52.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.051+0000 7fe29ab95740 1 -- start start 2024-08-28T19:37:52.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.051+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6003b070 0x557d603e0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.051+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603e0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.051+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6001a390 0x557d603e0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.051+0000 7fe29ab95740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557d60357680 con 0x557d600121b0 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.051+0000 7fe29ab95740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d60346c10 con 0x557d6003b070 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.051+0000 7fe29ab95740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557d6034abc0 con 0x557d6001a390 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe291bbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6003b070 0x557d603e0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe2923bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603e0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe291bbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6003b070 0x557d603e0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47708/0 (socket says 172.21.15.16:47708) 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe291bbb640 1 -- 172.21.15.16:0/3881767596 learned_addr learned my addr 172.21.15.16:0/3881767596 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe291bbb640 1 --2- 172.21.15.16:0/3881767596 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6001a390 0x557d603e0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe2923bc640 1 -- 172.21.15.16:0/3881767596 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6001a390 msgr2=0x557d603e0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe2923bc640 1 --2- 172.21.15.16:0/3881767596 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6001a390 0x557d603e0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe2923bc640 1 -- 172.21.15.16:0/3881767596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6003b070 msgr2=0x557d603e0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe2923bc640 1 --2- 172.21.15.16:0/3881767596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6003b070 0x557d603e0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe2837fe640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2780b4d20 0x7fe2780b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe2923bc640 1 -- 172.21.15.16:0/3881767596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d600a6650 con 0x557d600121b0 2024-08-28T19:37:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe291bbb640 1 --2- 172.21.15.16:0/3881767596 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6001a390 0x557d603e0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:37:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe292bbd640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2780b4d20 0x7fe2780b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.052+0000 7fe2923bc640 1 --2- 172.21.15.16:0/3881767596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603e0890 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fe288009c20 tx=0x7fe288011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c013332558d5b21 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.053+0000 7fe2837fe640 1 -- 172.21.15.16:0/1016000227 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fe2840e5e80 con 0x7fe28c109b90 2024-08-28T19:37:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.053+0000 7fe280ff9640 1 -- 172.21.15.16:0/3881767596 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288009dd0 con 0x557d600121b0 2024-08-28T19:37:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.053+0000 7fe280ff9640 1 -- 172.21.15.16:0/3881767596 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe288005020 con 0x557d600121b0 2024-08-28T19:37:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.053+0000 7fe280ff9640 1 -- 172.21.15.16:0/3881767596 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288004b20 con 0x557d600121b0 2024-08-28T19:37:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.054+0000 7fe29ab95740 1 -- 172.21.15.16:0/3881767596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 msgr2=0x557d603e0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.054+0000 7fe29ab95740 1 --2- 172.21.15.16:0/3881767596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603e0890 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fe288009c20 tx=0x7fe288011810 comp rx=0 tx=0).stop 2024-08-28T19:37:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.054+0000 7fe292bbd640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2780b4d20 0x7fe2780b7230 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe2840ac130 tx=0x7fe2840a8140 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.054+0000 7fe29ab95740 1 -- 172.21.15.16:0/3881767596 shutdown_connections 2024-08-28T19:37:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.054+0000 7fe29ab95740 1 --2- 172.21.15.16:0/3881767596 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6001a390 0x557d603e0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.054+0000 7fe29ab95740 1 --2- 172.21.15.16:0/3881767596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603e0890 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.054+0000 7fe29ab95740 1 --2- 172.21.15.16:0/3881767596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6003b070 0x557d603e0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.054+0000 7fe29ab95740 1 -- 172.21.15.16:0/3881767596 >> 172.21.15.16:0/3881767596 conn(0x557d6002f460 msgr2=0x557d603de240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.054+0000 7fe29ab95740 1 -- 172.21.15.16:0/3881767596 shutdown_connections 2024-08-28T19:37:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.054+0000 7fe29ab95740 1 -- 172.21.15.16:0/3881767596 wait complete. 2024-08-28T19:37:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.054+0000 7fe29ab95740 1 -- start start 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe29ab95740 1 Processor -- start 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603ebaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 0x557d603ec030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 0x557d603f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe29ab95740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557d601a9700 con 0x557d600121b0 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe29ab95740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d602cd750 con 0x557d6001a390 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe29ab95740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557d603eeb30 con 0x557d6003b070 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe29ab95740 10 client.0 _ll_register_callbacks cb 0x557d602fce10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe291bbb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603ebaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe2923bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 0x557d603ec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe2923bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 0x557d603ec030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47714/0 (socket says 172.21.15.16:47714) 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe2923bc640 1 -- 172.21.15.16:0/36457826 learned_addr learned my addr 172.21.15.16:0/36457826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.055+0000 7fe291bbb640 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 0x557d603f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe2923bc640 1 -- 172.21.15.16:0/36457826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 msgr2=0x557d603f7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe2923bc640 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 0x557d603f7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe2923bc640 1 -- 172.21.15.16:0/36457826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 msgr2=0x557d603ebaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe2923bc640 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603ebaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe2923bc640 1 -- 172.21.15.16:0/36457826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d60306900 con 0x557d6001a390 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe2923bc640 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 0x557d603ec030 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe288009730 tx=0x7fe2880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d6f2f412753f658 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe275ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe275ffb640 1 -- 172.21.15.16:0/36457826 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288029020 con 0x557d6001a390 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe29ab95740 1 -- 172.21.15.16:0/36457826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe288005020 con 0x557d6001a390 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe275ffb640 1 -- 172.21.15.16:0/36457826 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fe288004ab0 con 0x557d6001a390 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe275ffb640 1 -- 172.21.15.16:0/36457826 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fe288034050 con 0x557d6001a390 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe29ab95740 10 client.28967 fetch_fsmap learned FSMap version 88 2024-08-28T19:37:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.056+0000 7fe29ab95740 1 -- 172.21.15.16:0/36457826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=4+}) -- 0x557d603fa8f0 con 0x557d6001a390 2024-08-28T19:37:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.057+0000 7fe275ffb640 1 -- 172.21.15.16:0/36457826 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe288005020 con 0x557d6001a390 2024-08-28T19:37:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.057+0000 7fe275ffb640 1 -- 172.21.15.16:0/36457826 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 88) ==== 8418+0+0 (secure 0 0 0) 0x7fe288023420 con 0x557d6001a390 2024-08-28T19:37:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.057+0000 7fe29ab95740 10 client.28967 fetch_fsmap finished waiting for FSMap version 88 2024-08-28T19:37:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.057+0000 7fe29ab95740 10 client.28967 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:37:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.057+0000 7fe29ab95740 20 client.28967 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.057+0000 7fe29ab95740 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d5fff99f0 0x557d603fdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.057+0000 7fe29ab95740 4 client.28967 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:37:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.057+0000 7fe29ab95740 1 -- 172.21.15.16:0/36457826 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d603595d0 con 0x557d5fff99f0 2024-08-28T19:37:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.057+0000 7fe291bbb640 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d5fff99f0 0x557d603fdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.058+0000 7fe291bbb640 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d5fff99f0 0x557d603fdcb0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9dcff0940639b433 server_cookie=b122bba7fa13f973 in_seq=0 out_seq=0 2024-08-28T19:37:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.058+0000 7fe275ffb640 10 client.28967 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:37:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe275ffb640 1 -- 172.21.15.16:0/36457826 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557d603595d0 con 0x557d5fff99f0 2024-08-28T19:37:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe275ffb640 10 client.28967 handle_command_reply: tid=0 2024-08-28T19:37:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe29ab95740 1 client.28967 shutdown 2024-08-28T19:37:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe29ab95740 1 -- 172.21.15.16:0/36457826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 msgr2=0x557d603ec030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe29ab95740 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 0x557d603ec030 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe288009730 tx=0x7fe2880046c0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe29ab95740 1 -- 172.21.15.16:0/36457826 shutdown_connections 2024-08-28T19:37:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe29ab95740 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d5fff99f0 0x557d603fdcb0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe29ab95740 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 0x557d603f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe29ab95740 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 0x557d603ec030 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe29ab95740 1 --2- 172.21.15.16:0/36457826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603ebaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe29ab95740 1 -- 172.21.15.16:0/36457826 >> 172.21.15.16:0/36457826 conn(0x557d6002f460 msgr2=0x557d603e34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:52.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.061+0000 7fe275ffb640 0 client.28967 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:37:52.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.062+0000 7fe29ab95740 1 -- 172.21.15.16:0/36457826 shutdown_connections 2024-08-28T19:37:52.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.062+0000 7fe29ab95740 1 -- 172.21.15.16:0/36457826 wait complete. 2024-08-28T19:37:52.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.062+0000 7fe29ab95740 20 client.28967 trim_cache size 0 max 16384 2024-08-28T19:37:52.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.083+0000 7fe29ab95740 1 Processor -- start 2024-08-28T19:37:52.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.083+0000 7fe29ab95740 1 -- start start 2024-08-28T19:37:52.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.083+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d600121b0 0x557d603e3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.083+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d6001a390 0x557d603f7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.083+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 0x557d603e8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.083+0000 7fe29ab95740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1b2f00 con 0x557d6001a390 2024-08-28T19:37:52.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.083+0000 7fe29ab95740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1b4c50 con 0x557d600121b0 2024-08-28T19:37:52.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.083+0000 7fe29ab95740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1b3690 con 0x557d6003b070 2024-08-28T19:37:52.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.083+0000 7fe291bbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d600121b0 0x557d603e3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.083+0000 7fe291bbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d600121b0 0x557d603e3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47720/0 (socket says 172.21.15.16:47720) 2024-08-28T19:37:52.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.083+0000 7fe291bbb640 1 -- 172.21.15.16:0/3399968810 learned_addr learned my addr 172.21.15.16:0/3399968810 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:52.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.084+0000 7fe291bbb640 1 --2- 172.21.15.16:0/3399968810 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 0x557d603e8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.084+0000 7fe291bbb640 1 -- 172.21.15.16:0/3399968810 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 msgr2=0x557d603e8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:52.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.084+0000 7fe291bbb640 1 --2- 172.21.15.16:0/3399968810 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 0x557d603e8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.084+0000 7fe291bbb640 1 -- 172.21.15.16:0/3399968810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d6001a390 msgr2=0x557d603f7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:52.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.084+0000 7fe291bbb640 1 --2- 172.21.15.16:0/3399968810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d6001a390 0x557d603f7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.084+0000 7fe291bbb640 1 -- 172.21.15.16:0/3399968810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d603ee620 con 0x557d600121b0 2024-08-28T19:37:52.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.084+0000 7fe291bbb640 1 --2- 172.21.15.16:0/3399968810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d600121b0 0x557d603e3b60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe27c006010 tx=0x7fe27c03f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f56fe971b6d29ed9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:52.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.084+0000 7fe277fff640 1 -- 172.21.15.16:0/3399968810 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c035040 con 0x557d600121b0 2024-08-28T19:37:52.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.084+0000 7fe277fff640 1 -- 172.21.15.16:0/3399968810 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe27c04e030 con 0x557d600121b0 2024-08-28T19:37:52.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.085+0000 7fe277fff640 1 -- 172.21.15.16:0/3399968810 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c005540 con 0x557d600121b0 2024-08-28T19:37:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.086+0000 7fe29ab95740 1 -- 172.21.15.16:0/3399968810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d600121b0 msgr2=0x557d603e3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.086+0000 7fe29ab95740 1 --2- 172.21.15.16:0/3399968810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d600121b0 0x557d603e3b60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe27c006010 tx=0x7fe27c03f040 comp rx=0 tx=0).stop 2024-08-28T19:37:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.086+0000 7fe29ab95740 1 -- 172.21.15.16:0/3399968810 shutdown_connections 2024-08-28T19:37:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.086+0000 7fe29ab95740 1 --2- 172.21.15.16:0/3399968810 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 0x557d603e8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.086+0000 7fe29ab95740 1 --2- 172.21.15.16:0/3399968810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d6001a390 0x557d603f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.086+0000 7fe29ab95740 1 --2- 172.21.15.16:0/3399968810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d600121b0 0x557d603e3b60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.086+0000 7fe29ab95740 1 -- 172.21.15.16:0/3399968810 >> 172.21.15.16:0/3399968810 conn(0x557d6002f460 msgr2=0x557d603e3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.086+0000 7fe29ab95740 1 -- 172.21.15.16:0/3399968810 shutdown_connections 2024-08-28T19:37:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.086+0000 7fe29ab95740 1 -- 172.21.15.16:0/3399968810 wait complete. 2024-08-28T19:37:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.086+0000 7fe29ab95740 1 -- start start 2024-08-28T19:37:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.087+0000 7fe29ab95740 1 Processor -- start 2024-08-28T19:37:52.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.087+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603efe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.087+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 0x557d603f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.087+0000 7fe29ab95740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 0x557d603f08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.087+0000 7fe291bbb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603efe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.087+0000 7fe291bbb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603efe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50918/0 (socket says 172.21.15.16:50918) 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.087+0000 7fe291bbb640 1 -- 172.21.15.16:0/2857873077 learned_addr learned my addr 172.21.15.16:0/2857873077 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe29ab95740 1 -- 172.21.15.16:0/2857873077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1b44c0 con 0x557d600121b0 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe29ab95740 1 -- 172.21.15.16:0/2857873077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1b3d90 con 0x557d6001a390 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe2923bc640 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 0x557d603f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe29ab95740 1 -- 172.21.15.16:0/2857873077 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1b0f00 con 0x557d6003b070 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe29ab95740 10 client.0 _ll_register_callbacks cb 0x7fe2840093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe2923bc640 1 -- 172.21.15.16:0/2857873077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 msgr2=0x557d603f08e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:37:52.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe2923bc640 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 0x557d603f08e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe2923bc640 1 -- 172.21.15.16:0/2857873077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 msgr2=0x557d603efe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe2923bc640 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603efe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe2923bc640 1 -- 172.21.15.16:0/2857873077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d60467630 con 0x557d6001a390 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe291bbb640 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603efe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe29ab95740 1 -- 172.21.15.16:0/2857873077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe27c04e030 con 0x557d6001a390 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe2923bc640 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 0x557d603f03a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe288023840 tx=0x7fe288005cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=142bdf11c7d5078c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.088+0000 7fe275ffb640 10 client.28973 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.089+0000 7fe275ffb640 1 -- 172.21.15.16:0/2857873077 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288029070 con 0x557d6001a390 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.089+0000 7fe275ffb640 1 -- 172.21.15.16:0/2857873077 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fe288023620 con 0x557d6001a390 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.089+0000 7fe275ffb640 1 -- 172.21.15.16:0/2857873077 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fe288009a70 con 0x557d6001a390 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.089+0000 7fe29ab95740 10 client.28973 fetch_fsmap learned FSMap version 88 2024-08-28T19:37:52.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.089+0000 7fe29ab95740 1 -- 172.21.15.16:0/2857873077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=88}) -- 0x557d604687d0 con 0x557d6001a390 2024-08-28T19:37:52.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.090+0000 7fe275ffb640 1 -- 172.21.15.16:0/2857873077 <== mon.2 v2:172.21.15.28:3300/0 4 ==== fsmap(e 88) ==== 8418+0+0 (secure 0 0 0) 0x7fe288023ab0 con 0x557d6001a390 2024-08-28T19:37:52.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.090+0000 7fe29ab95740 10 client.28973 fetch_fsmap finished waiting for FSMap version 88 2024-08-28T19:37:52.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.090+0000 7fe29ab95740 10 client.28973 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:37:52.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.090+0000 7fe29ab95740 20 client.28973 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:52.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.090+0000 7fe29ab95740 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d5ffceda0 0x557d6046ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:52.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.091+0000 7fe29ab95740 4 client.28973 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:37:52.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.091+0000 7fe29ab95740 1 -- 172.21.15.16:0/2857873077 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557d603e76a0 con 0x557d5ffceda0 2024-08-28T19:37:52.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.091+0000 7fe291bbb640 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d5ffceda0 0x557d6046ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:52.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.092+0000 7fe291bbb640 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d5ffceda0 0x557d6046ae30 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4cc8c82cf07a395e server_cookie=6a54e0662b5ec4cc in_seq=0 out_seq=0 2024-08-28T19:37:52.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.092+0000 7fe275ffb640 10 client.28973 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:37:52.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.095+0000 7fe275ffb640 1 -- 172.21.15.16:0/2857873077 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557d603e76a0 con 0x557d5ffceda0 2024-08-28T19:37:52.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.095+0000 7fe275ffb640 10 client.28973 handle_command_reply: tid=0 2024-08-28T19:37:52.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.095+0000 7fe29ab95740 1 client.28973 shutdown 2024-08-28T19:37:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.099+0000 7fe29ab95740 1 -- 172.21.15.16:0/2857873077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 msgr2=0x557d603f03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.099+0000 7fe29ab95740 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 0x557d603f03a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe288023840 tx=0x7fe288005cc0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.099+0000 7fe29ab95740 1 -- 172.21.15.16:0/2857873077 shutdown_connections 2024-08-28T19:37:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.099+0000 7fe29ab95740 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x557d5ffceda0 0x557d6046ae30 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.099+0000 7fe29ab95740 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557d6003b070 0x557d603f08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.099+0000 7fe275ffb640 0 client.28973 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:37:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.099+0000 7fe29ab95740 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d6001a390 0x557d603f03a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.099+0000 7fe29ab95740 1 --2- 172.21.15.16:0/2857873077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d600121b0 0x557d603efe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.099+0000 7fe29ab95740 1 -- 172.21.15.16:0/2857873077 >> 172.21.15.16:0/2857873077 conn(0x557d6002f460 msgr2=0x557d603e1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:52.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.099+0000 7fe29ab95740 1 -- 172.21.15.16:0/2857873077 shutdown_connections 2024-08-28T19:37:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.101+0000 7fe29ab95740 1 -- 172.21.15.16:0/2857873077 wait complete. 2024-08-28T19:37:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.101+0000 7fe29ab95740 20 client.28973 trim_cache size 0 max 16384 2024-08-28T19:37:52.102 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:37:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.102+0000 7fe294620640 1 -- 172.21.15.16:0/1016000227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2780b4d20 msgr2=0x7fe2780b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.102+0000 7fe294620640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2780b4d20 0x7fe2780b7230 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe2840ac130 tx=0x7fe2840a8140 comp rx=0 tx=0).stop 2024-08-28T19:37:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.102+0000 7fe294620640 1 -- 172.21.15.16:0/1016000227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28c109b90 msgr2=0x7fe28c1a1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.102+0000 7fe294620640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28c109b90 0x7fe28c1a1c60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe28400eb10 tx=0x7fe28400c2d0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.103+0000 7fe294620640 1 -- 172.21.15.16:0/1016000227 shutdown_connections 2024-08-28T19:37:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.103+0000 7fe294620640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2780b4d20 0x7fe2780b7230 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.103+0000 7fe294620640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28c109b90 0x7fe28c1a1c60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.103+0000 7fe294620640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe28c109240 0x7fe28c19c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.103+0000 7fe294620640 1 --2- 172.21.15.16:0/1016000227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28c103900 0x7fe28c19c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.103+0000 7fe294620640 1 -- 172.21.15.16:0/1016000227 >> 172.21.15.16:0/1016000227 conn(0x7fe28c07b5f0 msgr2=0x7fe28c1072f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:52.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.104+0000 7fe294620640 1 -- 172.21.15.16:0/1016000227 shutdown_connections 2024-08-28T19:37:52.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:52.104+0000 7fe294620640 1 -- 172.21.15.16:0/1016000227 wait complete. 2024-08-28T19:37:52.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 76562 23.48 MB/sec execute 88 sec latency 330.489 ms 2024-08-28T19:37:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:52 smithi016 ceph-mon[18556]: pgmap v864: 129 pgs: 129 active+clean; 1.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 9.0 MiB/s wr, 789 op/s 2024-08-28T19:37:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:52 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:37:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:52 smithi028 ceph-mon[21769]: pgmap v864: 129 pgs: 129 active+clean; 1.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 9.0 MiB/s wr, 789 op/s 2024-08-28T19:37:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:52 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:37:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:52 smithi115 ceph-mon[22184]: pgmap v864: 129 pgs: 129 active+clean; 1.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 9.0 MiB/s wr, 789 op/s 2024-08-28T19:37:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:52 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:37:53.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 76890 23.42 MB/sec execute 89 sec latency 569.260 ms 2024-08-28T19:37:53.560 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:37:53.914 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:54.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.181 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:53 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:53 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:53 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:54.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 77331 23.41 MB/sec execute 90 sec latency 404.756 ms 2024-08-28T19:37:55.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:54 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-08-28T19:37:55.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:54 smithi016 ceph-mon[18556]: Unable to set osd_memory_target on smithi016 to 185701376: error parsing value: Value '185701376' is below minimum 939524096 2024-08-28T19:37:55.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:54 smithi016 ceph-mon[18556]: pgmap v865: 129 pgs: 129 active+clean; 1.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 7.5 MiB/s wr, 625 op/s 2024-08-28T19:37:55.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:54 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-08-28T19:37:55.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:54 smithi016 ceph-mon[18556]: Unable to set osd_memory_target on smithi028 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-08-28T19:37:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:54 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-08-28T19:37:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:54 smithi028 ceph-mon[21769]: Unable to set osd_memory_target on smithi016 to 185701376: error parsing value: Value '185701376' is below minimum 939524096 2024-08-28T19:37:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:54 smithi028 ceph-mon[21769]: pgmap v865: 129 pgs: 129 active+clean; 1.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 7.5 MiB/s wr, 625 op/s 2024-08-28T19:37:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:54 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-08-28T19:37:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:54 smithi028 ceph-mon[21769]: Unable to set osd_memory_target on smithi028 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-08-28T19:37:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:54 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-08-28T19:37:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:54 smithi115 ceph-mon[22184]: Unable to set osd_memory_target on smithi016 to 185701376: error parsing value: Value '185701376' is below minimum 939524096 2024-08-28T19:37:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:54 smithi115 ceph-mon[22184]: pgmap v865: 129 pgs: 129 active+clean; 1.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 7.5 MiB/s wr, 625 op/s 2024-08-28T19:37:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:54 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-08-28T19:37:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:54 smithi115 ceph-mon[22184]: Unable to set osd_memory_target on smithi028 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-08-28T19:37:55.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.422+0000 7fbba57a5640 1 -- 172.21.15.16:0/210685767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba0108140 msgr2=0x7fbba0108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:55.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.422+0000 7fbba57a5640 1 --2- 172.21.15.16:0/210685767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba0108140 0x7fbba0108550 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009900 tx=0x7fbb9402a410 comp rx=0 tx=0).stop 2024-08-28T19:37:55.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.422+0000 7fbba57a5640 1 -- 172.21.15.16:0/210685767 shutdown_connections 2024-08-28T19:37:55.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.422+0000 7fbba57a5640 1 --2- 172.21.15.16:0/210685767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba0108a90 0x7fbba010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.422+0000 7fbba57a5640 1 --2- 172.21.15.16:0/210685767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba0108140 0x7fbba0108550 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.422+0000 7fbba57a5640 1 --2- 172.21.15.16:0/210685767 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbba0102800 0x7fbba0102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.422+0000 7fbba57a5640 1 -- 172.21.15.16:0/210685767 >> 172.21.15.16:0/210685767 conn(0x7fbba00794d0 msgr2=0x7fbba00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.422+0000 7fbba57a5640 1 -- 172.21.15.16:0/210685767 shutdown_connections 2024-08-28T19:37:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.422+0000 7fbba57a5640 1 -- 172.21.15.16:0/210685767 wait complete. 2024-08-28T19:37:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.423+0000 7fbba57a5640 1 Processor -- start 2024-08-28T19:37:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.423+0000 7fbba57a5640 1 -- start start 2024-08-28T19:37:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.423+0000 7fbba57a5640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbba0102800 0x7fbba019b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.423+0000 7fbba57a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba0108140 0x7fbba019b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.423+0000 7fbba57a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba0108a90 0x7fbba019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.423+0000 7fbba57a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbba01126e0 con 0x7fbba0108a90 2024-08-28T19:37:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.423+0000 7fbba57a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbba0112560 con 0x7fbba0108140 2024-08-28T19:37:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.423+0000 7fbba57a5640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbba0112860 con 0x7fbba0102800 2024-08-28T19:37:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.424+0000 7fbb9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba0108a90 0x7fbba019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.424+0000 7fbb9effd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbba0102800 0x7fbba019b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.424+0000 7fbb9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba0108a90 0x7fbba019bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33474/0 (socket says 172.21.15.16:33474) 2024-08-28T19:37:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.424+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/595858398 learned_addr learned my addr 172.21.15.16:0/595858398 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:55.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.424+0000 7fbb9e7fc640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba0108140 0x7fbba019b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.424+0000 7fbb9effd640 1 -- 172.21.15.16:0/595858398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba0108140 msgr2=0x7fbba019b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:55.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.424+0000 7fbb9effd640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba0108140 0x7fbba019b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.424+0000 7fbb9effd640 1 -- 172.21.15.16:0/595858398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba0108a90 msgr2=0x7fbba019bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:55.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.424+0000 7fbb9effd640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba0108a90 0x7fbba019bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.424+0000 7fbb9effd640 1 -- 172.21.15.16:0/595858398 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba01a62c0 con 0x7fbba0102800 2024-08-28T19:37:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.424+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba0108a90 0x7fbba019bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.425+0000 7fbb9effd640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbba0102800 0x7fbba019b030 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000e9f0 tx=0x7fbb9000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.425+0000 7fbb87fff640 1 -- 172.21.15.16:0/595858398 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9000ccc0 con 0x7fbba0102800 2024-08-28T19:37:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.425+0000 7fbb87fff640 1 -- 172.21.15.16:0/595858398 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb90005020 con 0x7fbba0102800 2024-08-28T19:37:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.425+0000 7fbba57a5640 1 -- 172.21.15.16:0/595858398 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba01a65b0 con 0x7fbba0102800 2024-08-28T19:37:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.425+0000 7fbb87fff640 1 -- 172.21.15.16:0/595858398 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb90004a00 con 0x7fbba0102800 2024-08-28T19:37:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.425+0000 7fbba57a5640 1 -- 172.21.15.16:0/595858398 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba01a6a70 con 0x7fbba0102800 2024-08-28T19:37:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.428+0000 7fbbabd1a740 1 Processor -- start 2024-08-28T19:37:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.428+0000 7fbbabd1a740 1 -- start start 2024-08-28T19:37:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.428+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f536b070 0x5608f5710350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.428+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f5710890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.428+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f534a390 0x5608f5710dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.428+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5608f5687680 con 0x5608f534a390 2024-08-28T19:37:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.428+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5608f5676c10 con 0x5608f536b070 2024-08-28T19:37:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.428+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5608f567abc0 con 0x5608f53421b0 2024-08-28T19:37:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.428+0000 7fbb9e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f534a390 0x5608f5710dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.429+0000 7fbb9e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f534a390 0x5608f5710dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33486/0 (socket says 172.21.15.16:33486) 2024-08-28T19:37:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.429+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/2362236958 learned_addr learned my addr 172.21.15.16:0/2362236958 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.429+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/2362236958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f536b070 0x5608f5710350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.429+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/2362236958 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 msgr2=0x5608f5710890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:37:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.429+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/2362236958 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f5710890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.429+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/2362236958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f534a390 msgr2=0x5608f5710dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.429+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/2362236958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f534a390 0x5608f5710dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.429+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/2362236958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608f53d6650 con 0x5608f536b070 2024-08-28T19:37:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.429+0000 7fbb9e7fc640 1 --2- 172.21.15.16:0/2362236958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f534a390 0x5608f5710dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.429+0000 7fbb87fff640 1 -- 172.21.15.16:0/595858398 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fbb90004ba0 con 0x7fbba0102800 2024-08-28T19:37:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.429+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/2362236958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f536b070 0x5608f5710350 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbb980117e0 tx=0x7fbb98011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d7ff2c6603916f9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.430+0000 7fbb857fa640 1 -- 172.21.15.16:0/2362236958 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb98011310 con 0x5608f536b070 2024-08-28T19:37:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.430+0000 7fbb857fa640 1 -- 172.21.15.16:0/2362236958 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb9800f9a0 con 0x5608f536b070 2024-08-28T19:37:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.430+0000 7fbb857fa640 1 -- 172.21.15.16:0/2362236958 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb980055f0 con 0x5608f536b070 2024-08-28T19:37:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.430+0000 7fbb87fff640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb880b4cd0 0x7fbb880b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.430+0000 7fbb9effd640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb880b4cd0 0x7fbb880b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.431+0000 7fbb87fff640 1 -- 172.21.15.16:0/595858398 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fbb900e5870 con 0x7fbba0102800 2024-08-28T19:37:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.431+0000 7fbbabd1a740 1 -- 172.21.15.16:0/2362236958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f536b070 msgr2=0x5608f5710350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.431+0000 7fbb9effd640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb880b4cd0 0x7fbb880b71e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x5608f570efd0 tx=0x7fbb90013d50 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.431+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/2362236958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f536b070 0x5608f5710350 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbb980117e0 tx=0x7fbb98011ca0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.431+0000 7fbbabd1a740 1 -- 172.21.15.16:0/2362236958 shutdown_connections 2024-08-28T19:37:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.431+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/2362236958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f534a390 0x5608f5710dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.431+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/2362236958 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f5710890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.431+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/2362236958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f536b070 0x5608f5710350 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.431+0000 7fbbabd1a740 1 -- 172.21.15.16:0/2362236958 >> 172.21.15.16:0/2362236958 conn(0x5608f535f460 msgr2=0x5608f570e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:55.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.431+0000 7fbbabd1a740 1 -- 172.21.15.16:0/2362236958 shutdown_connections 2024-08-28T19:37:55.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.431+0000 7fbbabd1a740 1 -- 172.21.15.16:0/2362236958 wait complete. 2024-08-28T19:37:55.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.432+0000 7fbbabd1a740 1 -- start start 2024-08-28T19:37:55.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.432+0000 7fbbabd1a740 1 Processor -- start 2024-08-28T19:37:55.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f571b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f571bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f536b070 0x5608f5727e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5608f54d9700 con 0x5608f536b070 2024-08-28T19:37:55.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5608f55fd750 con 0x5608f534a390 2024-08-28T19:37:55.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5608f571eb30 con 0x5608f53421b0 2024-08-28T19:37:55.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbbabd1a740 10 client.0 _ll_register_callbacks cb 0x5608f562ce10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:55.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f536b070 0x5608f5727e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f571bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f536b070 0x5608f5727e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33492/0 (socket says 172.21.15.16:33492) 2024-08-28T19:37:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/3090114370 learned_addr learned my addr 172.21.15.16:0/3090114370 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f571b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/3090114370 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 msgr2=0x5608f571b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:55.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f571b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/3090114370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 msgr2=0x5608f571bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:55.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f571bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/3090114370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608f5636900 con 0x5608f536b070 2024-08-28T19:37:55.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9e7fc640 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f571bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:55.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f536b070 0x5608f5727e20 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005af0 tx=0x7fbb98005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617313800e3fcf34 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.433+0000 7fbb82ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:37:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.434+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3090114370 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9801e040 con 0x5608f536b070 2024-08-28T19:37:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.434+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3090114370 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fbb98004a40 con 0x5608f536b070 2024-08-28T19:37:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.434+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3090114370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb9800f9a0 con 0x5608f536b070 2024-08-28T19:37:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.434+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3090114370 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fbb98004c40 con 0x5608f536b070 2024-08-28T19:37:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.434+0000 7fbbabd1a740 10 client.19548 fetch_fsmap learned FSMap version 88 2024-08-28T19:37:55.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.434+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3090114370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=4+}) -- 0x5608f572a7c0 con 0x5608f536b070 2024-08-28T19:37:55.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.434+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3090114370 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb9800f9a0 con 0x5608f536b070 2024-08-28T19:37:55.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.434+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3090114370 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 88) ==== 8418+0+0 (secure 0 0 0) 0x7fbb98024030 con 0x5608f536b070 2024-08-28T19:37:55.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.435+0000 7fbbabd1a740 10 client.19548 fetch_fsmap finished waiting for FSMap version 88 2024-08-28T19:37:55.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.435+0000 7fbbabd1a740 10 client.19548 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:37:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.435+0000 7fbbabd1a740 20 client.19548 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.435+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5608f53299f0 0x5608f572dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.435+0000 7fbbabd1a740 4 client.19548 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:37:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.435+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3090114370 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608f56895d0 con 0x5608f53299f0 2024-08-28T19:37:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.435+0000 7fbb9e7fc640 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5608f53299f0 0x5608f572dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.437+0000 7fbb9e7fc640 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5608f53299f0 0x5608f572dc30 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c877372073aeef03 server_cookie=3c6eff8244d66483 in_seq=0 out_seq=0 2024-08-28T19:37:55.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.437+0000 7fbb82ff5640 10 client.19548 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:37:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.438+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3090114370 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5608f56895d0 con 0x5608f53299f0 2024-08-28T19:37:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.438+0000 7fbb82ff5640 10 client.19548 handle_command_reply: tid=0 2024-08-28T19:37:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 1 client.19548 shutdown 2024-08-28T19:37:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3090114370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f536b070 msgr2=0x5608f5727e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f536b070 0x5608f5727e20 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005af0 tx=0x7fbb98005bd0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3090114370 shutdown_connections 2024-08-28T19:37:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5608f53299f0 0x5608f572dc30 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f536b070 0x5608f5727e20 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbb82ff5640 0 client.19548 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:37:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f571bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3090114370 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f571b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3090114370 >> 172.21.15.16:0/3090114370 conn(0x5608f535f460 msgr2=0x5608f5713530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3090114370 shutdown_connections 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3090114370 wait complete. 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.439+0000 7fbbabd1a740 20 client.19548 trim_cache size 0 max 16384 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbbabd1a740 1 Processor -- start 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbbabd1a740 1 -- start start 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f570d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f570dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbba01aeb40 con 0x5608f53299f0 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbba01b0890 con 0x5608f534a390 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbba01af2d0 con 0x5608f53421b0 2024-08-28T19:37:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbb9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbb9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33508/0 (socket says 172.21.15.16:33508) 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/1780848686 learned_addr learned my addr 172.21.15.16:0/1780848686 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbb9e7fc640 1 --2- 172.21.15.16:0/1780848686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f570d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/1780848686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f570dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.457+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/1780848686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 msgr2=0x5608f570d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.458+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/1780848686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f570d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.458+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/1780848686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 msgr2=0x5608f570dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.458+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/1780848686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f570dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.458+0000 7fbb9f7fe640 1 -- 172.21.15.16:0/1780848686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608f570ee00 con 0x5608f53299f0 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.458+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/1780848686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571d5d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800f790 tx=0x7fbb980021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e9c32a4272e5da server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.458+0000 7fbb84ff9640 1 -- 172.21.15.16:0/1780848686 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb98005710 con 0x5608f53299f0 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.458+0000 7fbb84ff9640 1 -- 172.21.15.16:0/1780848686 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb9800f840 con 0x5608f53299f0 2024-08-28T19:37:55.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.458+0000 7fbb84ff9640 1 -- 172.21.15.16:0/1780848686 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb98028a80 con 0x5608f53299f0 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.459+0000 7fbbabd1a740 1 -- 172.21.15.16:0/1780848686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 msgr2=0x5608f571d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.459+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/1780848686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571d5d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800f790 tx=0x7fbb980021f0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.459+0000 7fbbabd1a740 1 -- 172.21.15.16:0/1780848686 shutdown_connections 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.459+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/1780848686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f570dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.459+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/1780848686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f570d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.459+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/1780848686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571d5d0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.459+0000 7fbbabd1a740 1 -- 172.21.15.16:0/1780848686 >> 172.21.15.16:0/1780848686 conn(0x5608f535f460 msgr2=0x5608f571cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.459+0000 7fbbabd1a740 1 -- 172.21.15.16:0/1780848686 shutdown_connections 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.459+0000 7fbbabd1a740 1 -- 172.21.15.16:0/1780848686 wait complete. 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbbabd1a740 1 -- start start 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbbabd1a740 1 Processor -- start 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f5720240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbbabd1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f5720780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbba01b0100 con 0x5608f53299f0 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbba01af9d0 con 0x5608f534a390 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbbabd1a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbba01acb40 con 0x5608f53421b0 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbbabd1a740 10 client.0 _ll_register_callbacks cb 0x7fbb9800fb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbb9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbb9e7fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f5720240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbb9e7fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f5720240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:35490/0 (socket says 172.21.15.16:35490) 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbb9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33516/0 (socket says 172.21.15.16:33516) 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.460+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/3933227387 learned_addr learned my addr 172.21.15.16:0/3933227387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.461+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f5720780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.461+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/3933227387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 msgr2=0x5608f5720780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.461+0000 7fbb9e7fc640 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f5720780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.461+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/3933227387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 msgr2=0x5608f571fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.461+0000 7fbb9e7fc640 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.461+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/3933227387 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608f5797460 con 0x5608f53421b0 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.461+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.461+0000 7fbb9e7fc640 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f5720240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbb9402a930 tx=0x7fbb94004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c72108d377142462 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.461+0000 7fbb82ff5640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.462+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3933227387 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94054060 con 0x5608f53421b0 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.462+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3933227387 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fbb94005040 con 0x5608f53421b0 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.462+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3933227387 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb9800f840 con 0x5608f53421b0 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.462+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3933227387 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fbb9405b050 con 0x5608f53421b0 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.462+0000 7fbbabd1a740 10 client.28582 fetch_fsmap learned FSMap version 88 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.462+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3933227387 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=4+}) -- 0x5608f5798680 con 0x5608f53421b0 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.462+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3933227387 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb9800f840 con 0x5608f53421b0 2024-08-28T19:37:55.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.463+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3933227387 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 88) ==== 8418+0+0 (secure 0 0 0) 0x7fbb94004750 con 0x5608f53421b0 2024-08-28T19:37:55.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.463+0000 7fbbabd1a740 10 client.28582 fetch_fsmap finished waiting for FSMap version 88 2024-08-28T19:37:55.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.463+0000 7fbbabd1a740 10 client.28582 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:37:55.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.463+0000 7fbbabd1a740 20 client.28582 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:55.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.463+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5608f536b070 0x5608f579ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:55.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.463+0000 7fbbabd1a740 4 client.28582 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:37:55.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.463+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3933227387 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5608f5685070 con 0x5608f536b070 2024-08-28T19:37:55.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.463+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5608f536b070 0x5608f579ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:55.450 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:37:55.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.463+0000 7fbb9f7fe640 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5608f536b070 0x5608f579ae80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96bc966d937fea50 server_cookie=b26d6862100d168a in_seq=0 out_seq=0 2024-08-28T19:37:55.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.463+0000 7fbb82ff5640 10 client.28582 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:37:55.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbb82ff5640 1 -- 172.21.15.16:0/3933227387 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5608f5685070 con 0x5608f536b070 2024-08-28T19:37:55.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbb82ff5640 10 client.28582 handle_command_reply: tid=0 2024-08-28T19:37:55.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbbabd1a740 1 client.28582 shutdown 2024-08-28T19:37:55.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3933227387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 msgr2=0x5608f5720240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:55.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f5720240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbb9402a930 tx=0x7fbb94004040 comp rx=0 tx=0).stop 2024-08-28T19:37:55.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3933227387 shutdown_connections 2024-08-28T19:37:55.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5608f536b070 0x5608f579ae80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608f534a390 0x5608f5720780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5608f53421b0 0x5608f5720240 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbb82ff5640 0 client.28582 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:37:55.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbbabd1a740 1 --2- 172.21.15.16:0/3933227387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608f53299f0 0x5608f571fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3933227387 >> 172.21.15.16:0/3933227387 conn(0x5608f535f460 msgr2=0x5608f571cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:55.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3933227387 shutdown_connections 2024-08-28T19:37:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.464+0000 7fbbabd1a740 1 -- 172.21.15.16:0/3933227387 wait complete. 2024-08-28T19:37:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbbabd1a740 20 client.28582 trim_cache size 0 max 16384 2024-08-28T19:37:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 -- 172.21.15.16:0/595858398 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb880b4cd0 msgr2=0x7fbb880b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb880b4cd0 0x7fbb880b71e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x5608f570efd0 tx=0x7fbb90013d50 comp rx=0 tx=0).stop 2024-08-28T19:37:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 -- 172.21.15.16:0/595858398 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbba0102800 msgr2=0x7fbba019b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbba0102800 0x7fbba019b030 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000e9f0 tx=0x7fbb9000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 -- 172.21.15.16:0/595858398 shutdown_connections 2024-08-28T19:37:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba0108a90 0x7fbba019bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbb880b4cd0 0x7fbb880b71e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba0108140 0x7fbba019b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 --2- 172.21.15.16:0/595858398 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbba0102800 0x7fbba019b030 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 -- 172.21.15.16:0/595858398 >> 172.21.15.16:0/595858398 conn(0x7fbba00794d0 msgr2=0x7fbba01041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 -- 172.21.15.16:0/595858398 shutdown_connections 2024-08-28T19:37:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:54.465+0000 7fbba57a5640 1 -- 172.21.15.16:0/595858398 wait complete. 2024-08-28T19:37:55.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 77814 23.40 MB/sec execute 91 sec latency 212.004 ms 2024-08-28T19:37:56.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:37:56.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:37:56.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:55 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:37:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:37:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:37:56.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:55 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:37:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-28T19:37:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:37:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:37:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:55 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:37:56.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 77841 23.15 MB/sec execute 92 sec latency 1059.698 ms 2024-08-28T19:37:56.665 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:37:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:37:57.210 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:56 smithi028 ceph-mon[21769]: Adjusting osd_memory_target on smithi115 to 176.0M 2024-08-28T19:37:57.210 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:56 smithi028 ceph-mon[21769]: Unable to set osd_memory_target on smithi115 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-08-28T19:37:57.210 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:56 smithi028 ceph-mon[21769]: pgmap v866: 129 pgs: 129 active+clean; 1.2 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 8.7 MiB/s wr, 730 op/s 2024-08-28T19:37:57.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:56 smithi016 ceph-mon[18556]: Adjusting osd_memory_target on smithi115 to 176.0M 2024-08-28T19:37:57.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:56 smithi016 ceph-mon[18556]: Unable to set osd_memory_target on smithi115 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-08-28T19:37:57.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:56 smithi016 ceph-mon[18556]: pgmap v866: 129 pgs: 129 active+clean; 1.2 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 8.7 MiB/s wr, 730 op/s 2024-08-28T19:37:57.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:56 smithi115 ceph-mon[22184]: Adjusting osd_memory_target on smithi115 to 176.0M 2024-08-28T19:37:57.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:56 smithi115 ceph-mon[22184]: Unable to set osd_memory_target on smithi115 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-08-28T19:37:57.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:56 smithi115 ceph-mon[22184]: pgmap v866: 129 pgs: 129 active+clean; 1.2 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 8.7 MiB/s wr, 730 op/s 2024-08-28T19:37:57.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 78076 23.01 MB/sec execute 93 sec latency 1393.330 ms 2024-08-28T19:37:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.678+0000 7fb1ed0e6640 1 -- 172.21.15.16:0/639104468 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4031810 con 0x7fb1e8108a90 2024-08-28T19:37:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.678+0000 7fb1f034c640 1 -- 172.21.15.16:0/639104468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e8108a90 msgr2=0x7fb1e810f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.678+0000 7fb1f034c640 1 --2- 172.21.15.16:0/639104468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e8108a90 0x7fb1e810f2c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009900 tx=0x7fb1e402a420 comp rx=0 tx=0).stop 2024-08-28T19:37:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.679+0000 7fb1f034c640 1 -- 172.21.15.16:0/639104468 shutdown_connections 2024-08-28T19:37:57.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.679+0000 7fb1f034c640 1 --2- 172.21.15.16:0/639104468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e8108a90 0x7fb1e810f2c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.679+0000 7fb1f034c640 1 --2- 172.21.15.16:0/639104468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1e8108140 0x7fb1e8108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.679+0000 7fb1f034c640 1 --2- 172.21.15.16:0/639104468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e8102800 0x7fb1e8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.679+0000 7fb1f034c640 1 -- 172.21.15.16:0/639104468 >> 172.21.15.16:0/639104468 conn(0x7fb1e80794d0 msgr2=0x7fb1e80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:57.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.679+0000 7fb1f034c640 1 -- 172.21.15.16:0/639104468 shutdown_connections 2024-08-28T19:37:57.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.679+0000 7fb1f034c640 1 -- 172.21.15.16:0/639104468 wait complete. 2024-08-28T19:37:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.680+0000 7fb1f034c640 1 Processor -- start 2024-08-28T19:37:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.680+0000 7fb1f034c640 1 -- start start 2024-08-28T19:37:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.680+0000 7fb1f034c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1e8102800 0x7fb1e810b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.681+0000 7fb1f034c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e8108140 0x7fb1e810be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.681+0000 7fb1f034c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e8108a90 0x7fb1e81ac550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.681+0000 7fb1f034c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e8111630 con 0x7fb1e8108140 2024-08-28T19:37:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.681+0000 7fb1f034c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81114b0 con 0x7fb1e8108a90 2024-08-28T19:37:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.681+0000 7fb1f034c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81117b0 con 0x7fb1e8102800 2024-08-28T19:37:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.681+0000 7fb1ee0e8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1e8102800 0x7fb1e810b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.681+0000 7fb1ee0e8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1e8102800 0x7fb1e810b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:35512/0 (socket says 172.21.15.16:35512) 2024-08-28T19:37:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.681+0000 7fb1ee0e8640 1 -- 172.21.15.16:0/4087460421 learned_addr learned my addr 172.21.15.16:0/4087460421 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.681+0000 7fb1ee8e9640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e8108a90 0x7fb1e81ac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.681+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e8108140 0x7fb1e810be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.683+0000 7fb1ee8e9640 1 -- 172.21.15.16:0/4087460421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1e8102800 msgr2=0x7fb1e810b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.683+0000 7fb1ee8e9640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1e8102800 0x7fb1e810b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.683+0000 7fb1ee8e9640 1 -- 172.21.15.16:0/4087460421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e8108140 msgr2=0x7fb1e810be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.683+0000 7fb1ee8e9640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e8108140 0x7fb1e810be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.683+0000 7fb1ee8e9640 1 -- 172.21.15.16:0/4087460421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e810c6b0 con 0x7fb1e8108a90 2024-08-28T19:37:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.683+0000 7fb1ee8e9640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e8108a90 0x7fb1e81ac550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009850 tx=0x7fb1e4003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.684+0000 7fb1df7fe640 1 -- 172.21.15.16:0/4087460421 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e403f070 con 0x7fb1e8108a90 2024-08-28T19:37:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.684+0000 7fb1f034c640 1 -- 172.21.15.16:0/4087460421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e81acaf0 con 0x7fb1e8108a90 2024-08-28T19:37:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.684+0000 7fb1df7fe640 1 -- 172.21.15.16:0/4087460421 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1e403bb10 con 0x7fb1e8108a90 2024-08-28T19:37:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.684+0000 7fb1df7fe640 1 -- 172.21.15.16:0/4087460421 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4035030 con 0x7fb1e8108a90 2024-08-28T19:37:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.684+0000 7fb1f034c640 1 -- 172.21.15.16:0/4087460421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e81acff0 con 0x7fb1e8108a90 2024-08-28T19:37:57.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.688+0000 7fb1f68c1740 1 Processor -- start 2024-08-28T19:37:57.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.689+0000 7fb1df7fe640 1 -- 172.21.15.16:0/4087460421 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fb1e4042070 con 0x7fb1e8108a90 2024-08-28T19:37:57.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.691+0000 7fb1f68c1740 1 -- start start 2024-08-28T19:37:57.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.691+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 0x56240ee21350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.691+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea531b0 0x56240ee21890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.691+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea5b390 0x56240ee21dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.691+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56240ed98680 con 0x56240ea7c070 2024-08-28T19:37:57.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.691+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56240ed87c10 con 0x56240ea531b0 2024-08-28T19:37:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.691+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56240ed8bbc0 con 0x56240ea5b390 2024-08-28T19:37:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.691+0000 7fb1ed8e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 0x56240ee21350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.691+0000 7fb1ed8e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 0x56240ee21350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33558/0 (socket says 172.21.15.16:33558) 2024-08-28T19:37:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.691+0000 7fb1ed8e7640 1 -- 172.21.15.16:0/3559694700 learned_addr learned my addr 172.21.15.16:0/3559694700 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.692+0000 7fb1df7fe640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1d40b4d50 0x7fb1d40b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.692+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/3559694700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea5b390 0x56240ee21dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.692+0000 7fb1df7fe640 1 -- 172.21.15.16:0/4087460421 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fb1e4105c10 con 0x7fb1e8108a90 2024-08-28T19:37:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.693+0000 7fb1ee8e9640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1d40b4d50 0x7fb1d40b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.693+0000 7fb1ed8e7640 1 -- 172.21.15.16:0/3559694700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea5b390 msgr2=0x56240ee21dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.693+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/3559694700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea5b390 0x56240ee21dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.693+0000 7fb1ed8e7640 1 -- 172.21.15.16:0/3559694700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea531b0 msgr2=0x56240ee21890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.693+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/3559694700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea531b0 0x56240ee21890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.693+0000 7fb1ed8e7640 1 -- 172.21.15.16:0/3559694700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56240eae7650 con 0x56240ea7c070 2024-08-28T19:37:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.694+0000 7fb1ee8e9640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1d40b4d50 0x7fb1d40b7260 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40c8060 tx=0x7fb1e4003f50 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:57.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.695+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/3559694700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 0x56240ee21350 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8005bc0 tx=0x7fb1d80097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615b3d71d4d2836a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:57.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.696+0000 7fb1dcd9a640 1 -- 172.21.15.16:0/3559694700 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8027070 con 0x56240ea7c070 2024-08-28T19:37:57.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.696+0000 7fb1dcd9a640 1 -- 172.21.15.16:0/3559694700 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1d80054d0 con 0x56240ea7c070 2024-08-28T19:37:57.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.696+0000 7fb1dcd9a640 1 -- 172.21.15.16:0/3559694700 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d802a040 con 0x56240ea7c070 2024-08-28T19:37:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.697+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3559694700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 msgr2=0x56240ee21350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.697+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/3559694700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 0x56240ee21350 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8005bc0 tx=0x7fb1d80097a0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.699+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3559694700 shutdown_connections 2024-08-28T19:37:57.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.699+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/3559694700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea5b390 0x56240ee21dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.699+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/3559694700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea531b0 0x56240ee21890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.699+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/3559694700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 0x56240ee21350 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.699+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3559694700 >> 172.21.15.16:0/3559694700 conn(0x56240ea70460 msgr2=0x56240ee1f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:57.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.700+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3559694700 shutdown_connections 2024-08-28T19:37:57.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.700+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3559694700 wait complete. 2024-08-28T19:37:57.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.701+0000 7fb1f68c1740 1 -- start start 2024-08-28T19:37:57.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.701+0000 7fb1f68c1740 1 Processor -- start 2024-08-28T19:37:57.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.701+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea531b0 0x56240ee2cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.701+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea5b390 0x56240ee2d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.701+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 0x56240ee38e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.701+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56240ebea700 con 0x56240ea7c070 2024-08-28T19:37:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.702+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56240ed0e750 con 0x56240ea5b390 2024-08-28T19:37:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.702+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56240ee2fb20 con 0x56240ea531b0 2024-08-28T19:37:57.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.702+0000 7fb1f68c1740 10 client.0 _ll_register_callbacks cb 0x56240ed3de10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:57.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.702+0000 7fb1ee0e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea5b390 0x56240ee2d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.702+0000 7fb1ee0e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea5b390 0x56240ee2d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:42318/0 (socket says 172.21.15.16:42318) 2024-08-28T19:37:57.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.702+0000 7fb1ee0e8640 1 -- 172.21.15.16:0/3592585552 learned_addr learned my addr 172.21.15.16:0/3592585552 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:57.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.702+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 0x56240ee38e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.703+0000 7fb1ee0e8640 1 -- 172.21.15.16:0/3592585552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea531b0 msgr2=0x56240ee2cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:57.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.703+0000 7fb1ee0e8640 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea531b0 0x56240ee2cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.703+0000 7fb1ee0e8640 1 -- 172.21.15.16:0/3592585552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 msgr2=0x56240ee38e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:57.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.703+0000 7fb1ee0e8640 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 0x56240ee38e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.703+0000 7fb1ee0e8640 1 -- 172.21.15.16:0/3592585552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56240ed47900 con 0x56240ea5b390 2024-08-28T19:37:57.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.703+0000 7fb1ee0e8640 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea5b390 0x56240ee2d070 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0005f40 tx=0x7fb1e0011b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=855a97f3e7f45e9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:57.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.703+0000 7fb1d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:37:57.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.705+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/3592585552 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0003b70 con 0x56240ea5b390 2024-08-28T19:37:57.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.705+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3592585552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1d80054d0 con 0x56240ea5b390 2024-08-28T19:37:57.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.705+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/3592585552 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fb1e0003d10 con 0x56240ea5b390 2024-08-28T19:37:57.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.710+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/3592585552 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fb1e000f810 con 0x56240ea5b390 2024-08-28T19:37:57.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.711+0000 7fb1f68c1740 10 client.29000 fetch_fsmap learned FSMap version 88 2024-08-28T19:37:57.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.711+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3592585552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=4+}) -- 0x56240ee3b990 con 0x56240ea5b390 2024-08-28T19:37:57.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.711+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/3592585552 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1d80054d0 con 0x56240ea5b390 2024-08-28T19:37:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.712+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/3592585552 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 88) ==== 8418+0+0 (secure 0 0 0) 0x7fb1e000f040 con 0x56240ea5b390 2024-08-28T19:37:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.712+0000 7fb1f68c1740 10 client.29000 fetch_fsmap finished waiting for FSMap version 88 2024-08-28T19:37:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.712+0000 7fb1f68c1740 10 client.29000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:37:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.712+0000 7fb1f68c1740 20 client.29000 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.712+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56240ea3a9f0 0x56240ee3f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.712+0000 7fb1f68c1740 4 client.29000 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:37:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.712+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3592585552 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56240ed9a5d0 con 0x56240ea3a9f0 2024-08-28T19:37:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.713+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56240ea3a9f0 0x56240ee3f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.720+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56240ea3a9f0 0x56240ee3f360 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0556a07adf64553 server_cookie=ed710bb87bb09286 in_seq=0 out_seq=0 2024-08-28T19:37:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.720+0000 7fb1d1ffb640 10 client.29000 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.725+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/3592585552 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56240ed9a5d0 con 0x56240ea3a9f0 2024-08-28T19:37:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.725+0000 7fb1d1ffb640 10 client.29000 handle_command_reply: tid=0 2024-08-28T19:37:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.726+0000 7fb1f68c1740 1 client.29000 shutdown 2024-08-28T19:37:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.726+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3592585552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea5b390 msgr2=0x56240ee2d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.726+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea5b390 0x56240ee2d070 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0005f40 tx=0x7fb1e0011b60 comp rx=0 tx=0).stop 2024-08-28T19:37:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.727+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3592585552 shutdown_connections 2024-08-28T19:37:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.727+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea7c070 0x56240ee38e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.727+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea5b390 0x56240ee2d070 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.727+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56240ea3a9f0 0x56240ee3f360 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.727+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/3592585552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea531b0 0x56240ee2cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.727+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3592585552 >> 172.21.15.16:0/3592585552 conn(0x56240ea70460 msgr2=0x56240ee24530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.727+0000 7fb1d1ffb640 0 client.29000 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.727+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3592585552 shutdown_connections 2024-08-28T19:37:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.727+0000 7fb1f68c1740 1 -- 172.21.15.16:0/3592585552 wait complete. 2024-08-28T19:37:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.727+0000 7fb1f68c1740 20 client.29000 trim_cache size 0 max 16384 2024-08-28T19:37:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.754+0000 7fb1f68c1740 1 Processor -- start 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.754+0000 7fb1f68c1740 1 -- start start 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.755+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea531b0 0x56240ee2d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.755+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea5b390 0x56240ee38d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.755+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 0x56240ee39280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.755+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81aeaf0 con 0x56240ea531b0 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.755+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81b0840 con 0x56240ea7c070 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.755+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81af280 con 0x56240ea5b390 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.755+0000 7fb1ed8e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 0x56240ee39280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.755+0000 7fb1ed8e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 0x56240ee39280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:42334/0 (socket says 172.21.15.16:42334) 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.755+0000 7fb1ed8e7640 1 -- 172.21.15.16:0/2717351372 learned_addr learned my addr 172.21.15.16:0/2717351372 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.755+0000 7fb1ee0e8640 1 --2- 172.21.15.16:0/2717351372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea5b390 0x56240ee38d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.756+0000 7fb1ee0e8640 1 -- 172.21.15.16:0/2717351372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 msgr2=0x56240ee39280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.756+0000 7fb1ee0e8640 1 --2- 172.21.15.16:0/2717351372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 0x56240ee39280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.756+0000 7fb1ee0e8640 1 -- 172.21.15.16:0/2717351372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea531b0 msgr2=0x56240ee2d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:57.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.756+0000 7fb1ee0e8640 1 --2- 172.21.15.16:0/2717351372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea531b0 0x56240ee2d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.756+0000 7fb1ee0e8640 1 -- 172.21.15.16:0/2717351372 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56240ee22180 con 0x56240ea5b390 2024-08-28T19:37:57.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.756+0000 7fb1ee0e8640 1 --2- 172.21.15.16:0/2717351372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea5b390 0x56240ee38d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0004990 tx=0x7fb1e0009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22132df1aa0e828f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:57.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.757+0000 7fb1d3fff640 1 -- 172.21.15.16:0/2717351372 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e000f9b0 con 0x56240ea5b390 2024-08-28T19:37:57.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.757+0000 7fb1d3fff640 1 -- 172.21.15.16:0/2717351372 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1e00124a0 con 0x56240ea5b390 2024-08-28T19:37:57.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.758+0000 7fb1f68c1740 1 -- 172.21.15.16:0/2717351372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea5b390 msgr2=0x56240ee38d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:57.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.758+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/2717351372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea5b390 0x56240ee38d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0004990 tx=0x7fb1e0009520 comp rx=0 tx=0).stop 2024-08-28T19:37:57.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.759+0000 7fb1f68c1740 1 -- 172.21.15.16:0/2717351372 shutdown_connections 2024-08-28T19:37:57.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.759+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/2717351372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 0x56240ee39280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.759+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/2717351372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea5b390 0x56240ee38d40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.759+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/2717351372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea531b0 0x56240ee2d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.759+0000 7fb1f68c1740 1 -- 172.21.15.16:0/2717351372 >> 172.21.15.16:0/2717351372 conn(0x56240ea70460 msgr2=0x56240ee2b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:57.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.764+0000 7fb1f68c1740 1 -- 172.21.15.16:0/2717351372 shutdown_connections 2024-08-28T19:37:57.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.764+0000 7fb1f68c1740 1 -- 172.21.15.16:0/2717351372 wait complete. 2024-08-28T19:37:57.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.764+0000 7fb1f68c1740 1 -- start start 2024-08-28T19:37:57.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.765+0000 7fb1f68c1740 1 Processor -- start 2024-08-28T19:37:57.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.765+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea531b0 0x56240ee30190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.765+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 0x56240ee306d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.765+0000 7fb1f68c1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea0fda0 0x56240eea7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.765+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81b00b0 con 0x56240ea531b0 2024-08-28T19:37:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.765+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81af980 con 0x56240ea7c070 2024-08-28T19:37:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.765+0000 7fb1f68c1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e8074c30 con 0x56240ea0fda0 2024-08-28T19:37:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.765+0000 7fb1f68c1740 10 client.0 _ll_register_callbacks cb 0x7fb1e0009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:37:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.766+0000 7fb1ee0e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 0x56240ee306d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.766+0000 7fb1ee0e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 0x56240ee306d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:42348/0 (socket says 172.21.15.16:42348) 2024-08-28T19:37:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.766+0000 7fb1ee0e8640 1 -- 172.21.15.16:0/1338087954 learned_addr learned my addr 172.21.15.16:0/1338087954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:37:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.766+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea0fda0 0x56240eea7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.766+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea531b0 0x56240ee30190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.767+0000 7fb1ed8e7640 1 -- 172.21.15.16:0/1338087954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 msgr2=0x56240ee306d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.767+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 0x56240ee306d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.767+0000 7fb1ed8e7640 1 -- 172.21.15.16:0/1338087954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea531b0 msgr2=0x56240ee30190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:37:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.767+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea531b0 0x56240ee30190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.767+0000 7fb1ed8e7640 1 -- 172.21.15.16:0/1338087954 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56240ee30ec0 con 0x56240ea0fda0 2024-08-28T19:37:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.768+0000 7fb1ed8e7640 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea0fda0 0x56240eea7d30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8011940 tx=0x7fb1d8004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=406c228b74f065db server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:37:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.768+0000 7fb1d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:37:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.769+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/1338087954 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8027040 con 0x56240ea0fda0 2024-08-28T19:37:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.769+0000 7fb1f68c1740 1 -- 172.21.15.16:0/1338087954 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56240ed96070 con 0x56240ea0fda0 2024-08-28T19:37:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.769+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/1338087954 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fb1d800f990 con 0x56240ea0fda0 2024-08-28T19:37:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.770+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/1338087954 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fb1d800fb90 con 0x56240ea0fda0 2024-08-28T19:37:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.771+0000 7fb1f68c1740 10 client.28597 fetch_fsmap learned FSMap version 88 2024-08-28T19:37:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.771+0000 7fb1f68c1740 1 -- 172.21.15.16:0/1338087954 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=4+}) -- 0x56240eea92b0 con 0x56240ea0fda0 2024-08-28T19:37:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.773+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/1338087954 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56240ed96070 con 0x56240ea0fda0 2024-08-28T19:37:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.773+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/1338087954 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 88) ==== 8418+0+0 (secure 0 0 0) 0x7fb1d8045020 con 0x56240ea0fda0 2024-08-28T19:37:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.773+0000 7fb1f68c1740 10 client.28597 fetch_fsmap finished waiting for FSMap version 88 2024-08-28T19:37:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.773+0000 7fb1f68c1740 10 client.28597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:37:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.773+0000 7fb1f68c1740 20 client.28597 populate_metadata read hostname 'smithi016' 2024-08-28T19:37:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.774+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56240eea97b0 0x56240eeabbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:37:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.774+0000 7fb1f68c1740 4 client.28597 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:37:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.774+0000 7fb1f68c1740 1 -- 172.21.15.16:0/1338087954 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56240ed937b0 con 0x56240eea97b0 2024-08-28T19:37:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.774+0000 7fb1ee0e8640 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56240eea97b0 0x56240eeabbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:37:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.777+0000 7fb1ee0e8640 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56240eea97b0 0x56240eeabbd0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15416ba49efea281 server_cookie=18ab10682ad892aa in_seq=0 out_seq=0 2024-08-28T19:37:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.777+0000 7fb1d1ffb640 10 client.28597 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:57.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.778+0000 7fb1d1ffb640 1 -- 172.21.15.16:0/1338087954 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56240ed937b0 con 0x56240eea97b0 2024-08-28T19:37:57.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.778+0000 7fb1d1ffb640 10 client.28597 handle_command_reply: tid=0 2024-08-28T19:37:57.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.778+0000 7fb1f68c1740 1 client.28597 shutdown 2024-08-28T19:37:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.779+0000 7fb1f68c1740 1 -- 172.21.15.16:0/1338087954 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea0fda0 msgr2=0x56240eea7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.779+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea0fda0 0x56240eea7d30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8011940 tx=0x7fb1d8004530 comp rx=0 tx=0).stop 2024-08-28T19:37:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.779+0000 7fb1f68c1740 1 -- 172.21.15.16:0/1338087954 shutdown_connections 2024-08-28T19:37:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.779+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56240eea97b0 0x56240eeabbd0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.779+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56240ea0fda0 0x56240eea7d30 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8011940 tx=0x7fb1d8004530 comp rx=0 tx=0).stop 2024-08-28T19:37:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.779+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56240ea7c070 0x56240ee306d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.779+0000 7fb1f68c1740 1 --2- 172.21.15.16:0/1338087954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56240ea531b0 0x56240ee30190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.779+0000 7fb1f68c1740 1 -- 172.21.15.16:0/1338087954 >> 172.21.15.16:0/1338087954 conn(0x56240ea70460 msgr2=0x56240ee24450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.779+0000 7fb1d1ffb640 0 client.28597 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:37:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.780+0000 7fb1f68c1740 1 -- 172.21.15.16:0/1338087954 shutdown_connections 2024-08-28T19:37:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.781+0000 7fb1f68c1740 1 -- 172.21.15.16:0/1338087954 wait complete. 2024-08-28T19:37:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.781+0000 7fb1f68c1740 20 client.28597 trim_cache size 0 max 16384 2024-08-28T19:37:57.784 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:37:57.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.783+0000 7fb1f034c640 1 -- 172.21.15.16:0/4087460421 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1d40b4d50 msgr2=0x7fb1d40b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:57.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.783+0000 7fb1f034c640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1d40b4d50 0x7fb1d40b7260 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40c8060 tx=0x7fb1e4003f50 comp rx=0 tx=0).stop 2024-08-28T19:37:57.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.784+0000 7fb1f034c640 1 -- 172.21.15.16:0/4087460421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e8108a90 msgr2=0x7fb1e81ac550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:37:57.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.784+0000 7fb1f034c640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e8108a90 0x7fb1e81ac550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009850 tx=0x7fb1e4003b90 comp rx=0 tx=0).stop 2024-08-28T19:37:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.786+0000 7fb1f034c640 1 -- 172.21.15.16:0/4087460421 shutdown_connections 2024-08-28T19:37:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.786+0000 7fb1f034c640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fb1d40b4d50 0x7fb1d40b7260 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.786+0000 7fb1f034c640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e8108a90 0x7fb1e81ac550 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.786+0000 7fb1f034c640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e8108140 0x7fb1e810be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.786+0000 7fb1f034c640 1 --2- 172.21.15.16:0/4087460421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb1e8102800 0x7fb1e810b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:37:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.786+0000 7fb1f034c640 1 -- 172.21.15.16:0/4087460421 >> 172.21.15.16:0/4087460421 conn(0x7fb1e80794d0 msgr2=0x7fb1e8104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:37:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.787+0000 7fb1f034c640 1 -- 172.21.15.16:0/4087460421 shutdown_connections 2024-08-28T19:37:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:37:57.787+0000 7fb1f034c640 1 -- 172.21.15.16:0/4087460421 wait complete. 2024-08-28T19:37:58.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 78475 22.90 MB/sec execute 94 sec latency 472.413 ms 2024-08-28T19:37:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:58 smithi016 ceph-mon[18556]: pgmap v867: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 9.7 MiB/s wr, 652 op/s 2024-08-28T19:37:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:58 smithi028 ceph-mon[21769]: pgmap v867: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 9.7 MiB/s wr, 652 op/s 2024-08-28T19:37:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:58 smithi115 ceph-mon[22184]: pgmap v867: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 9.7 MiB/s wr, 652 op/s 2024-08-28T19:37:59.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 78889 22.92 MB/sec execute 95 sec latency 464.169 ms 2024-08-28T19:38:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:37:59 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:37:59 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:00.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:37:59 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:00.472 INFO:tasks.workunit.client.0.smithi016.stdout: 10 79546 23.23 MB/sec execute 96 sec latency 200.443 ms 2024-08-28T19:38:00.879 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:38:00.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:38:01.237 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:01.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:00 smithi016 ceph-mon[18556]: pgmap v868: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 14 MiB/s wr, 735 op/s 2024-08-28T19:38:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:00 smithi028 ceph-mon[21769]: pgmap v868: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 14 MiB/s wr, 735 op/s 2024-08-28T19:38:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:00 smithi115 ceph-mon[22184]: pgmap v868: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 14 MiB/s wr, 735 op/s 2024-08-28T19:38:01.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 80047 23.24 MB/sec execute 97 sec latency 285.976 ms 2024-08-28T19:38:02.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.130+0000 7fdf4c8c3640 1 -- 172.21.15.16:0/1862776392 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf40030d70 con 0x7fdf4811c940 2024-08-28T19:38:02.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.131+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/1862776392 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf4811c940 msgr2=0x7fdf4811ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:02.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.131+0000 7fdf4e8c7640 1 --2- 172.21.15.16:0/1862776392 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf4811c940 0x7fdf4811ed60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009be0 tx=0x7fdf4002a6f0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.131+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/1862776392 shutdown_connections 2024-08-28T19:38:02.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.131+0000 7fdf4e8c7640 1 --2- 172.21.15.16:0/1862776392 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf4811c940 0x7fdf4811ed60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.131+0000 7fdf4e8c7640 1 --2- 172.21.15.16:0/1862776392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf4810d990 0x7fdf4810dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.131+0000 7fdf4e8c7640 1 --2- 172.21.15.16:0/1862776392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf48111a70 0x7fdf4810d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.131+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/1862776392 >> 172.21.15.16:0/1862776392 conn(0x7fdf4806e790 msgr2=0x7fdf4806ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:02.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.131+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/1862776392 shutdown_connections 2024-08-28T19:38:02.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.131+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/1862776392 wait complete. 2024-08-28T19:38:02.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.132+0000 7fdf4e8c7640 1 Processor -- start 2024-08-28T19:38:02.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.132+0000 7fdf4e8c7640 1 -- start start 2024-08-28T19:38:02.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.133+0000 7fdf4e8c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf4810d990 0x7fdf4811c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.133+0000 7fdf4e8c7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf48111a70 0x7fdf481184e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.133+0000 7fdf4e8c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf4811c940 0x7fdf48118aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.133+0000 7fdf4e8c7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48114a30 con 0x7fdf4810d990 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.133+0000 7fdf4e8c7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf481148b0 con 0x7fdf4811c940 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.133+0000 7fdf4e8c7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48114bb0 con 0x7fdf48111a70 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.133+0000 7fdf4e0c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf4811c940 0x7fdf48118aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.133+0000 7fdf4e0c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf4811c940 0x7fdf48118aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:42356/0 (socket says 172.21.15.16:42356) 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.133+0000 7fdf4e0c6640 1 -- 172.21.15.16:0/640826931 learned_addr learned my addr 172.21.15.16:0/640826931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.133+0000 7fdf4d0c4640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf48111a70 0x7fdf481184e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.133+0000 7fdf4d8c5640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf4810d990 0x7fdf4811c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.134+0000 7fdf4e0c6640 1 -- 172.21.15.16:0/640826931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf48111a70 msgr2=0x7fdf481184e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.134+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf48111a70 0x7fdf481184e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.134+0000 7fdf4e0c6640 1 -- 172.21.15.16:0/640826931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf4810d990 msgr2=0x7fdf4811c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.134+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf4810d990 0x7fdf4811c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.134+0000 7fdf4e0c6640 1 -- 172.21.15.16:0/640826931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf481192c0 con 0x7fdf4811c940 2024-08-28T19:38:02.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.135+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf4811c940 0x7fdf48118aa0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009b30 tx=0x7fdf40003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:02.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.137+0000 7fdf3effd640 1 -- 172.21.15.16:0/640826931 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf40040020 con 0x7fdf4811c940 2024-08-28T19:38:02.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.137+0000 7fdf3effd640 1 -- 172.21.15.16:0/640826931 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf40030e20 con 0x7fdf4811c940 2024-08-28T19:38:02.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.137+0000 7fdf3effd640 1 -- 172.21.15.16:0/640826931 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf40036040 con 0x7fdf4811c940 2024-08-28T19:38:02.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.138+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/640826931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4807d440 con 0x7fdf4811c940 2024-08-28T19:38:02.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.138+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/640826931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf4807d950 con 0x7fdf4811c940 2024-08-28T19:38:02.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.142+0000 7fdf5609e740 1 Processor -- start 2024-08-28T19:38:02.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.143+0000 7fdf3effd640 1 -- 172.21.15.16:0/640826931 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fdf40043020 con 0x7fdf4811c940 2024-08-28T19:38:02.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.146+0000 7fdf3effd640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf340b4df0 0x7fdf340b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.146+0000 7fdf3effd640 1 -- 172.21.15.16:0/640826931 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fdf40106790 con 0x7fdf4811c940 2024-08-28T19:38:02.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.147+0000 7fdf4d0c4640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf340b4df0 0x7fdf340b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.147+0000 7fdf5609e740 1 -- start start 2024-08-28T19:38:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.147+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ecc350 0x560e87272950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.147+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ea3490 0x560e87272e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.147+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 0x560e872733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.147+0000 7fdf5609e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560e871e85f0 con 0x560e86ecc350 2024-08-28T19:38:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf5609e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560e871d7b80 con 0x560e86eab670 2024-08-28T19:38:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf5609e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560e871dbb30 con 0x560e86ea3490 2024-08-28T19:38:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf4d0c4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ea3490 0x560e87272e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf4d0c4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ea3490 0x560e87272e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:35570/0 (socket says 172.21.15.16:35570) 2024-08-28T19:38:02.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf4d0c4640 1 -- 172.21.15.16:0/631618609 learned_addr learned my addr 172.21.15.16:0/631618609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:02.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/631618609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 0x560e872733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf4d8c5640 1 --2- 172.21.15.16:0/631618609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ecc350 0x560e87272950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf4d0c4640 1 -- 172.21.15.16:0/631618609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 msgr2=0x560e872733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf4d0c4640 1 --2- 172.21.15.16:0/631618609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 0x560e872733d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf4d0c4640 1 -- 172.21.15.16:0/631618609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ecc350 msgr2=0x560e87272950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf4d0c4640 1 --2- 172.21.15.16:0/631618609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ecc350 0x560e87272950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.148+0000 7fdf4d0c4640 1 -- 172.21.15.16:0/631618609 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e86f37930 con 0x560e86ea3490 2024-08-28T19:38:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.149+0000 7fdf4d0c4640 1 --2- 172.21.15.16:0/631618609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ea3490 0x560e87272e90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdf44003720 tx=0x7fdf44003be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51df303fe0360d14 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.150+0000 7fdf4d0c4640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf340b4df0 0x7fdf340b7300 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdf44025850 tx=0x7fdf44028000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.150+0000 7fdf33fff640 1 -- 172.21.15.16:0/631618609 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf44035070 con 0x560e86ea3490 2024-08-28T19:38:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.151+0000 7fdf33fff640 1 -- 172.21.15.16:0/631618609 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf44004240 con 0x560e86ea3490 2024-08-28T19:38:02.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.152+0000 7fdf33fff640 1 -- 172.21.15.16:0/631618609 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf44022ab0 con 0x560e86ea3490 2024-08-28T19:38:02.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.152+0000 7fdf5609e740 1 -- 172.21.15.16:0/631618609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ea3490 msgr2=0x560e87272e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:02.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.152+0000 7fdf5609e740 1 --2- 172.21.15.16:0/631618609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ea3490 0x560e87272e90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdf44003720 tx=0x7fdf44003be0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.152+0000 7fdf5609e740 1 -- 172.21.15.16:0/631618609 shutdown_connections 2024-08-28T19:38:02.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.152+0000 7fdf5609e740 1 --2- 172.21.15.16:0/631618609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 0x560e872733d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.152+0000 7fdf5609e740 1 --2- 172.21.15.16:0/631618609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ea3490 0x560e87272e90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.152+0000 7fdf5609e740 1 --2- 172.21.15.16:0/631618609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ecc350 0x560e87272950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.152+0000 7fdf5609e740 1 -- 172.21.15.16:0/631618609 >> 172.21.15.16:0/631618609 conn(0x560e86ec0740 msgr2=0x560e87270840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.153+0000 7fdf5609e740 1 -- 172.21.15.16:0/631618609 shutdown_connections 2024-08-28T19:38:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.153+0000 7fdf5609e740 1 -- 172.21.15.16:0/631618609 wait complete. 2024-08-28T19:38:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.153+0000 7fdf5609e740 1 -- start start 2024-08-28T19:38:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.153+0000 7fdf5609e740 1 Processor -- start 2024-08-28T19:38:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.154+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86ea3490 0x560e8727e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.154+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86eab670 0x560e8727e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.154+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 0x560e8728a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.154+0000 7fdf5609e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560e8703a890 con 0x560e86eab670 2024-08-28T19:38:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.154+0000 7fdf5609e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560e8715e790 con 0x560e86ea3490 2024-08-28T19:38:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.154+0000 7fdf5609e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560e87281170 con 0x560e86ecc350 2024-08-28T19:38:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.154+0000 7fdf5609e740 10 client.0 _ll_register_callbacks cb 0x560e86ee3ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.155+0000 7fdf4e0c6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 0x560e8728a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.155+0000 7fdf4e0c6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 0x560e8728a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:35578/0 (socket says 172.21.15.16:35578) 2024-08-28T19:38:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.155+0000 7fdf4e0c6640 1 -- 172.21.15.16:0/3725786478 learned_addr learned my addr 172.21.15.16:0/3725786478 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.155+0000 7fdf4d8c5640 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86ea3490 0x560e8727e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.156+0000 7fdf4e0c6640 1 -- 172.21.15.16:0/3725786478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86ea3490 msgr2=0x560e8727e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.156+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86ea3490 0x560e8727e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.156+0000 7fdf4e0c6640 1 -- 172.21.15.16:0/3725786478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86eab670 msgr2=0x560e8727e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.156+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86eab670 0x560e8727e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.156+0000 7fdf4e0c6640 1 -- 172.21.15.16:0/3725786478 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e87195f30 con 0x560e86ecc350 2024-08-28T19:38:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.156+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 0x560e8728a4b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdf400cc7b0 tx=0x7fdf400cc2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbb90ce41519c75d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.156+0000 7fdf317fa640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:38:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.158+0000 7fdf317fa640 1 -- 172.21.15.16:0/3725786478 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400d1100 con 0x560e86ecc350 2024-08-28T19:38:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.159+0000 7fdf317fa640 1 -- 172.21.15.16:0/3725786478 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fdf400c90a0 con 0x560e86ecc350 2024-08-28T19:38:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.159+0000 7fdf5609e740 1 -- 172.21.15.16:0/3725786478 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560e871ea540 con 0x560e86ecc350 2024-08-28T19:38:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.161+0000 7fdf317fa640 1 -- 172.21.15.16:0/3725786478 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fdf400cc5c0 con 0x560e86ecc350 2024-08-28T19:38:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.161+0000 7fdf5609e740 10 client.28609 fetch_fsmap learned FSMap version 89 2024-08-28T19:38:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.161+0000 7fdf5609e740 1 -- 172.21.15.16:0/3725786478 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=4+}) -- 0x560e8728cf30 con 0x560e86ecc350 2024-08-28T19:38:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.162+0000 7fdf317fa640 1 -- 172.21.15.16:0/3725786478 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560e871ea540 con 0x560e86ecc350 2024-08-28T19:38:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.162+0000 7fdf317fa640 1 -- 172.21.15.16:0/3725786478 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 89) ==== 8414+0+0 (secure 0 0 0) 0x7fdf40106370 con 0x560e86ecc350 2024-08-28T19:38:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.162+0000 7fdf5609e740 10 client.28609 fetch_fsmap finished waiting for FSMap version 89 2024-08-28T19:38:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.162+0000 7fdf5609e740 10 client.28609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.162+0000 7fdf5609e740 20 client.28609 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.162+0000 7fdf5609e740 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e86e8acd0 0x560e872904d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.162+0000 7fdf5609e740 4 client.28609 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:38:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.162+0000 7fdf5609e740 1 -- 172.21.15.16:0/3725786478 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e871e5fe0 con 0x560e86e8acd0 2024-08-28T19:38:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.163+0000 7fdf4d8c5640 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e86e8acd0 0x560e872904d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.163+0000 7fdf317fa640 10 client.28609 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:02.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.164+0000 7fdf4d8c5640 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e86e8acd0 0x560e872904d0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=835600ce79483974 server_cookie=5ba273069dd6d0c in_seq=0 out_seq=0 2024-08-28T19:38:02.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.168+0000 7fdf317fa640 1 -- 172.21.15.16:0/3725786478 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560e871e5fe0 con 0x560e86e8acd0 2024-08-28T19:38:02.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.168+0000 7fdf317fa640 10 client.28609 handle_command_reply: tid=0 2024-08-28T19:38:02.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.168+0000 7fdf5609e740 1 client.28609 shutdown 2024-08-28T19:38:02.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.168+0000 7fdf5609e740 1 -- 172.21.15.16:0/3725786478 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 msgr2=0x560e8728a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:02.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.168+0000 7fdf5609e740 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 0x560e8728a4b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdf400cc7b0 tx=0x7fdf400cc2e0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.169+0000 7fdf5609e740 1 -- 172.21.15.16:0/3725786478 shutdown_connections 2024-08-28T19:38:02.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.169+0000 7fdf5609e740 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e86e8acd0 0x560e872904d0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.169+0000 7fdf5609e740 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 0x560e8728a4b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.169+0000 7fdf5609e740 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86eab670 0x560e8727e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.169+0000 7fdf5609e740 1 --2- 172.21.15.16:0/3725786478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86ea3490 0x560e8727e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.169+0000 7fdf5609e740 1 -- 172.21.15.16:0/3725786478 >> 172.21.15.16:0/3725786478 conn(0x560e86ec0740 msgr2=0x560e87275b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:02.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.169+0000 7fdf317fa640 0 client.28609 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:02.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.169+0000 7fdf5609e740 1 -- 172.21.15.16:0/3725786478 shutdown_connections 2024-08-28T19:38:02.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.169+0000 7fdf5609e740 1 -- 172.21.15.16:0/3725786478 wait complete. 2024-08-28T19:38:02.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.169+0000 7fdf5609e740 20 client.28609 trim_cache size 0 max 16384 2024-08-28T19:38:02.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.206+0000 7fdf5609e740 1 Processor -- start 2024-08-28T19:38:02.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.206+0000 7fdf5609e740 1 -- start start 2024-08-28T19:38:02.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.206+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ea3490 0x560e8728a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.206+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86eab670 0x560e8727b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.207+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86ecc350 0x560e8727b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.207+0000 7fdf5609e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf481c8100 con 0x560e86ea3490 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.207+0000 7fdf5609e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf481c9e20 con 0x560e86ecc350 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.207+0000 7fdf5609e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdf481c8890 con 0x560e86eab670 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.207+0000 7fdf4d8c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ea3490 0x560e8728a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.207+0000 7fdf4d8c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ea3490 0x560e8728a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33626/0 (socket says 172.21.15.16:33626) 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.207+0000 7fdf4d8c5640 1 -- 172.21.15.16:0/3972524402 learned_addr learned my addr 172.21.15.16:0/3972524402 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.207+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/3972524402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86ecc350 0x560e8727b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.207+0000 7fdf4d0c4640 1 --2- 172.21.15.16:0/3972524402 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86eab670 0x560e8727b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.208+0000 7fdf4e0c6640 1 -- 172.21.15.16:0/3972524402 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86eab670 msgr2=0x560e8727b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.208+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/3972524402 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86eab670 0x560e8727b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.208+0000 7fdf4e0c6640 1 -- 172.21.15.16:0/3972524402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ea3490 msgr2=0x560e8728a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.208+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/3972524402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ea3490 0x560e8728a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.208+0000 7fdf4e0c6640 1 -- 172.21.15.16:0/3972524402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e87278c80 con 0x560e86ecc350 2024-08-28T19:38:02.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.208+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/3972524402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86ecc350 0x560e8727b780 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdf40034040 tx=0x7fdf400cc350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33a1e3bbc455c09b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:02.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.214+0000 7fdf337fe640 1 -- 172.21.15.16:0/3972524402 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400d1100 con 0x560e86ecc350 2024-08-28T19:38:02.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.214+0000 7fdf337fe640 1 -- 172.21.15.16:0/3972524402 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf400cd1b0 con 0x560e86ecc350 2024-08-28T19:38:02.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.214+0000 7fdf337fe640 1 -- 172.21.15.16:0/3972524402 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400d5dd0 con 0x560e86ecc350 2024-08-28T19:38:02.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.215+0000 7fdf5609e740 1 -- 172.21.15.16:0/3972524402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86ecc350 msgr2=0x560e8727b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:02.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.215+0000 7fdf5609e740 1 --2- 172.21.15.16:0/3972524402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86ecc350 0x560e8727b780 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdf40034040 tx=0x7fdf400cc350 comp rx=0 tx=0).stop 2024-08-28T19:38:02.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.216+0000 7fdf5609e740 1 -- 172.21.15.16:0/3972524402 shutdown_connections 2024-08-28T19:38:02.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.216+0000 7fdf5609e740 1 --2- 172.21.15.16:0/3972524402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86ecc350 0x560e8727b780 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.216+0000 7fdf5609e740 1 --2- 172.21.15.16:0/3972524402 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86eab670 0x560e8727b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.216+0000 7fdf5609e740 1 --2- 172.21.15.16:0/3972524402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ea3490 0x560e8728a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.216+0000 7fdf5609e740 1 -- 172.21.15.16:0/3972524402 >> 172.21.15.16:0/3972524402 conn(0x560e86ec0740 msgr2=0x560e8727c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:02.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.218+0000 7fdf5609e740 1 -- 172.21.15.16:0/3972524402 shutdown_connections 2024-08-28T19:38:02.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.218+0000 7fdf5609e740 1 -- 172.21.15.16:0/3972524402 wait complete. 2024-08-28T19:38:02.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.218+0000 7fdf5609e740 1 -- start start 2024-08-28T19:38:02.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.218+0000 7fdf5609e740 1 Processor -- start 2024-08-28T19:38:02.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.219+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ea3490 0x560e87286420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.220+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 0x560e87286960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.220+0000 7fdf5609e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 0x560e872813d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.220+0000 7fdf4d0c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 0x560e87286960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.220+0000 7fdf4d0c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 0x560e87286960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:42404/0 (socket says 172.21.15.16:42404) 2024-08-28T19:38:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.221+0000 7fdf4d0c4640 1 -- 172.21.15.16:0/2410265187 learned_addr learned my addr 172.21.15.16:0/2410265187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.221+0000 7fdf4e0c6640 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 0x560e872813d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.221+0000 7fdf5609e740 1 -- 172.21.15.16:0/2410265187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf481c96c0 con 0x560e86ea3490 2024-08-28T19:38:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.221+0000 7fdf5609e740 1 -- 172.21.15.16:0/2410265187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf481c8f90 con 0x560e86eab670 2024-08-28T19:38:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.221+0000 7fdf5609e740 1 -- 172.21.15.16:0/2410265187 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdf481197e0 con 0x560e86ecc350 2024-08-28T19:38:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.222+0000 7fdf5609e740 10 client.0 _ll_register_callbacks cb 0x7fdf40031940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.222+0000 7fdf4d0c4640 1 -- 172.21.15.16:0/2410265187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 msgr2=0x560e872813d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.222+0000 7fdf4d0c4640 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 0x560e872813d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.222+0000 7fdf4d0c4640 1 -- 172.21.15.16:0/2410265187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ea3490 msgr2=0x560e87286420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.222+0000 7fdf4d0c4640 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ea3490 0x560e87286420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.222+0000 7fdf4d0c4640 1 -- 172.21.15.16:0/2410265187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e87281910 con 0x560e86eab670 2024-08-28T19:38:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.223+0000 7fdf5609e740 1 -- 172.21.15.16:0/2410265187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf400cd1b0 con 0x560e86eab670 2024-08-28T19:38:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.223+0000 7fdf4d0c4640 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 0x560e87286960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdf44032000 tx=0x7fdf44005510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70ef1c8180b63e05 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:02.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.224+0000 7fdf317fa640 10 client.29021 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:38:02.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.230+0000 7fdf317fa640 1 -- 172.21.15.16:0/2410265187 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf44012020 con 0x560e86eab670 2024-08-28T19:38:02.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.230+0000 7fdf317fa640 1 -- 172.21.15.16:0/2410265187 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9490+0+0 (secure 0 0 0) 0x7fdf440045d0 con 0x560e86eab670 2024-08-28T19:38:02.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.230+0000 7fdf317fa640 1 -- 172.21.15.16:0/2410265187 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fdf44035040 con 0x560e86eab670 2024-08-28T19:38:02.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.231+0000 7fdf5609e740 10 client.29021 fetch_fsmap learned FSMap version 89 2024-08-28T19:38:02.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.231+0000 7fdf5609e740 1 -- 172.21.15.16:0/2410265187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=89}) -- 0x560e872fb430 con 0x560e86eab670 2024-08-28T19:38:02.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.231+0000 7fdf317fa640 1 -- 172.21.15.16:0/2410265187 <== mon.2 v2:172.21.15.28:3300/0 4 ==== fsmap(e 89) ==== 8414+0+0 (secure 0 0 0) 0x7fdf4401f040 con 0x560e86eab670 2024-08-28T19:38:02.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.232+0000 7fdf5609e740 10 client.29021 fetch_fsmap finished waiting for FSMap version 89 2024-08-28T19:38:02.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.232+0000 7fdf5609e740 10 client.29021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:02.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.232+0000 7fdf5609e740 20 client.29021 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:02.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.232+0000 7fdf5609e740 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e86e60080 0x560e872fd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:02.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.232+0000 7fdf5609e740 4 client.29021 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:38:02.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.232+0000 7fdf5609e740 1 -- 172.21.15.16:0/2410265187 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fdf400cd1b0 con 0x560e86e60080 2024-08-28T19:38:02.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.232+0000 7fdf4d8c5640 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e86e60080 0x560e872fd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:02.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.234+0000 7fdf4d8c5640 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e86e60080 0x560e872fd9e0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b19150bcb0548f51 server_cookie=957af9c9236073f9 in_seq=0 out_seq=0 2024-08-28T19:38:02.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.234+0000 7fdf317fa640 10 client.29021 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:02.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.237+0000 7fdf317fa640 1 -- 172.21.15.16:0/2410265187 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fdf400cd1b0 con 0x560e86e60080 2024-08-28T19:38:02.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.237+0000 7fdf317fa640 10 client.29021 handle_command_reply: tid=0 2024-08-28T19:38:02.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.237+0000 7fdf5609e740 1 client.29021 shutdown 2024-08-28T19:38:02.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.249+0000 7fdf5609e740 1 -- 172.21.15.16:0/2410265187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 msgr2=0x560e87286960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:02.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.249+0000 7fdf5609e740 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 0x560e87286960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdf44032000 tx=0x7fdf44005510 comp rx=0 tx=0).stop 2024-08-28T19:38:02.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.249+0000 7fdf5609e740 1 -- 172.21.15.16:0/2410265187 shutdown_connections 2024-08-28T19:38:02.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.249+0000 7fdf5609e740 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e86e60080 0x560e872fd9e0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.249+0000 7fdf5609e740 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e86ecc350 0x560e872813d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.249+0000 7fdf317fa640 0 client.29021 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:02.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.249+0000 7fdf5609e740 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e86eab670 0x560e87286960 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.249+0000 7fdf5609e740 1 --2- 172.21.15.16:0/2410265187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e86ea3490 0x560e87286420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.249+0000 7fdf5609e740 1 -- 172.21.15.16:0/2410265187 >> 172.21.15.16:0/2410265187 conn(0x560e86ec0740 msgr2=0x560e8727c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:02.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.250+0000 7fdf5609e740 1 -- 172.21.15.16:0/2410265187 shutdown_connections 2024-08-28T19:38:02.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.250+0000 7fdf5609e740 1 -- 172.21.15.16:0/2410265187 wait complete. 2024-08-28T19:38:02.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.250+0000 7fdf5609e740 20 client.29021 trim_cache size 0 max 16384 2024-08-28T19:38:02.252 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:38:02.253 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:38:02.253 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "f27568c6-5b0e-422a-bf5c-feccae126ec2", 2024-08-28T19:38:02.253 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:38:02.253 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:38:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.251+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/640826931 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf340b4df0 msgr2=0x7fdf340b7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.251+0000 7fdf4e8c7640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf340b4df0 0x7fdf340b7300 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdf44025850 tx=0x7fdf44028000 comp rx=0 tx=0).stop 2024-08-28T19:38:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.251+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/640826931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf4811c940 msgr2=0x7fdf48118aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.251+0000 7fdf4e8c7640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf4811c940 0x7fdf48118aa0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009b30 tx=0x7fdf40003f10 comp rx=0 tx=0).stop 2024-08-28T19:38:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.251+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/640826931 shutdown_connections 2024-08-28T19:38:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.251+0000 7fdf4e8c7640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf340b4df0 0x7fdf340b7300 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.251+0000 7fdf4e8c7640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf4811c940 0x7fdf48118aa0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.251+0000 7fdf4e8c7640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf48111a70 0x7fdf481184e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.251+0000 7fdf4e8c7640 1 --2- 172.21.15.16:0/640826931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf4810d990 0x7fdf4811c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:02.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.251+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/640826931 >> 172.21.15.16:0/640826931 conn(0x7fdf4806e790 msgr2=0x7fdf48110720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:02.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.251+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/640826931 shutdown_connections 2024-08-28T19:38:02.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:02.253+0000 7fdf4e8c7640 1 -- 172.21.15.16:0/640826931 wait complete. 2024-08-28T19:38:02.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 80225 23.06 MB/sec execute 98 sec latency 422.759 ms 2024-08-28T19:38:03.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:38:03.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:02 smithi016 ceph-mon[18556]: pgmap v869: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 17 MiB/s wr, 853 op/s 2024-08-28T19:38:03.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:02 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:38:03.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:02 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:03.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:02 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:03.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:02 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:03.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:02 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:03.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:02 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:03.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:02 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:02 smithi028 ceph-mon[21769]: pgmap v869: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 17 MiB/s wr, 853 op/s 2024-08-28T19:38:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:02 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:38:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:02 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:02 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:02 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:02 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:02 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:03.310 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:02 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:02 smithi115 ceph-mon[22184]: pgmap v869: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 17 MiB/s wr, 853 op/s 2024-08-28T19:38:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:02 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:38:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:02 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:02 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:02 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:02 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:02 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:02 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:03.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 80262 22.90 MB/sec execute 99 sec latency 485.929 ms 2024-08-28T19:38:04.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.060+0000 7fbf193b0640 1 -- 172.21.15.16:0/3721011694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14107f50 msgr2=0x7fbf14108360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:04.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.060+0000 7fbf193b0640 1 --2- 172.21.15.16:0/3721011694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14107f50 0x7fbf14108360 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009900 tx=0x7fbf0402a410 comp rx=0 tx=0).stop 2024-08-28T19:38:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.065+0000 7fbf193b0640 1 -- 172.21.15.16:0/3721011694 shutdown_connections 2024-08-28T19:38:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.066+0000 7fbf193b0640 1 --2- 172.21.15.16:0/3721011694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf141088a0 0x7fbf1410f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.066+0000 7fbf193b0640 1 --2- 172.21.15.16:0/3721011694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14107f50 0x7fbf14108360 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.066+0000 7fbf193b0640 1 --2- 172.21.15.16:0/3721011694 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbf14102610 0x7fbf14102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.066+0000 7fbf193b0640 1 -- 172.21.15.16:0/3721011694 >> 172.21.15.16:0/3721011694 conn(0x7fbf140fe060 msgr2=0x7fbf141004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:04.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.067+0000 7fbf193b0640 1 -- 172.21.15.16:0/3721011694 shutdown_connections 2024-08-28T19:38:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.069+0000 7fbf193b0640 1 -- 172.21.15.16:0/3721011694 wait complete. 2024-08-28T19:38:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.071+0000 7fbf193b0640 1 Processor -- start 2024-08-28T19:38:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.071+0000 7fbf193b0640 1 -- start start 2024-08-28T19:38:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.072+0000 7fbf193b0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbf14102610 0x7fbf14199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.072+0000 7fbf193b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14107f50 0x7fbf141995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.073+0000 7fbf13fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbf14102610 0x7fbf14199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.073+0000 7fbf13fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbf14102610 0x7fbf14199000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:35624/0 (socket says 172.21.15.16:35624) 2024-08-28T19:38:04.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.073+0000 7fbf13fff640 1 -- 172.21.15.16:0/3537161415 learned_addr learned my addr 172.21.15.16:0/3537161415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:04.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.073+0000 7fbf193b0640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf141088a0 0x7fbf1419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.073+0000 7fbf193b0640 1 -- 172.21.15.16:0/3537161415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf141125f0 con 0x7fbf141088a0 2024-08-28T19:38:04.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.073+0000 7fbf193b0640 1 -- 172.21.15.16:0/3537161415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf14112470 con 0x7fbf14107f50 2024-08-28T19:38:04.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.073+0000 7fbf18baf640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf141088a0 0x7fbf1419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.073+0000 7fbf193b0640 1 -- 172.21.15.16:0/3537161415 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbf14112770 con 0x7fbf14102610 2024-08-28T19:38:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.074+0000 7fbf137fe640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14107f50 0x7fbf141995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.075+0000 7fbf137fe640 1 -- 172.21.15.16:0/3537161415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbf14102610 msgr2=0x7fbf14199000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.075+0000 7fbf137fe640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbf14102610 0x7fbf14199000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.075+0000 7fbf137fe640 1 -- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf141088a0 msgr2=0x7fbf1419e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.075+0000 7fbf137fe640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf141088a0 0x7fbf1419e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.075+0000 7fbf137fe640 1 -- 172.21.15.16:0/3537161415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1419ef70 con 0x7fbf14107f50 2024-08-28T19:38:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.075+0000 7fbf18baf640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf141088a0 0x7fbf1419e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:38:04.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.076+0000 7fbf137fe640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14107f50 0x7fbf141995a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009850 tx=0x7fbf04003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:04.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.078+0000 7fbf117fa640 1 -- 172.21.15.16:0/3537161415 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04035070 con 0x7fbf14107f50 2024-08-28T19:38:04.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.078+0000 7fbf117fa640 1 -- 172.21.15.16:0/3537161415 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf04031950 con 0x7fbf14107f50 2024-08-28T19:38:04.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.078+0000 7fbf193b0640 1 -- 172.21.15.16:0/3537161415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf1419f200 con 0x7fbf14107f50 2024-08-28T19:38:04.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.078+0000 7fbf193b0640 1 -- 172.21.15.16:0/3537161415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf1406ac00 con 0x7fbf14107f50 2024-08-28T19:38:04.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.078+0000 7fbf117fa640 1 -- 172.21.15.16:0/3537161415 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0403a9b0 con 0x7fbf14107f50 2024-08-28T19:38:04.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.085+0000 7fbf20b87740 1 Processor -- start 2024-08-28T19:38:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.093+0000 7fbf117fa640 1 -- 172.21.15.16:0/3537161415 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fbf0404e060 con 0x7fbf14107f50 2024-08-28T19:38:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.094+0000 7fbf20b87740 1 -- start start 2024-08-28T19:38:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.094+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 0x55802b50d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.095+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b13f1b0 0x55802b50d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.095+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b147390 0x55802b50dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.095+0000 7fbf20b87740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55802b484630 con 0x55802b168070 2024-08-28T19:38:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.095+0000 7fbf20b87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55802b473bc0 con 0x55802b13f1b0 2024-08-28T19:38:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.095+0000 7fbf20b87740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55802b477b70 con 0x55802b147390 2024-08-28T19:38:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.095+0000 7fbf117fa640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbefc0b4eb0 0x7fbefc0b73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.096+0000 7fbf117fa640 1 -- 172.21.15.16:0/3537161415 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9578+0+0 (secure 0 0 0) 0x7fbf0410e350 con 0x7fbf14107f50 2024-08-28T19:38:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.096+0000 7fbf18baf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 0x55802b50d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.096+0000 7fbf18baf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 0x55802b50d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33650/0 (socket says 172.21.15.16:33650) 2024-08-28T19:38:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.096+0000 7fbf18baf640 1 -- 172.21.15.16:0/1967706134 learned_addr learned my addr 172.21.15.16:0/1967706134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.097+0000 7fbf13fff640 1 --2- 172.21.15.16:0/1967706134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b13f1b0 0x55802b50d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.097+0000 7fbf18baf640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbefc0b4eb0 0x7fbefc0b73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.098+0000 7fbf18baf640 1 -- 172.21.15.16:0/1967706134 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b147390 msgr2=0x55802b50dee0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:38:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.098+0000 7fbf18baf640 1 --2- 172.21.15.16:0/1967706134 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b147390 0x55802b50dee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.098+0000 7fbf18baf640 1 -- 172.21.15.16:0/1967706134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b13f1b0 msgr2=0x55802b50d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.098+0000 7fbf18baf640 1 --2- 172.21.15.16:0/1967706134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b13f1b0 0x55802b50d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.098+0000 7fbf18baf640 1 -- 172.21.15.16:0/1967706134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55802b1d3650 con 0x55802b168070 2024-08-28T19:38:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.099+0000 7fbf18baf640 1 --2- 172.21.15.16:0/1967706134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 0x55802b50d460 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fbf08015860 tx=0x7fbf08015d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9256d78abda61ad server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.099+0000 7fbf18baf640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbefc0b4eb0 0x7fbefc0b73c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009000 tx=0x7fbf0801b000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.099+0000 7fbefa7fc640 1 -- 172.21.15.16:0/1967706134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0800f040 con 0x55802b168070 2024-08-28T19:38:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.099+0000 7fbefa7fc640 1 -- 172.21.15.16:0/1967706134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf08024070 con 0x55802b168070 2024-08-28T19:38:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.100+0000 7fbefa7fc640 1 -- 172.21.15.16:0/1967706134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0801f660 con 0x55802b168070 2024-08-28T19:38:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.101+0000 7fbf20b87740 1 -- 172.21.15.16:0/1967706134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 msgr2=0x55802b50d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.101+0000 7fbf20b87740 1 --2- 172.21.15.16:0/1967706134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 0x55802b50d460 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fbf08015860 tx=0x7fbf08015d20 comp rx=0 tx=0).stop 2024-08-28T19:38:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.103+0000 7fbf20b87740 1 -- 172.21.15.16:0/1967706134 shutdown_connections 2024-08-28T19:38:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.103+0000 7fbf20b87740 1 --2- 172.21.15.16:0/1967706134 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b147390 0x55802b50dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.103+0000 7fbf20b87740 1 --2- 172.21.15.16:0/1967706134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b13f1b0 0x55802b50d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.103+0000 7fbf20b87740 1 --2- 172.21.15.16:0/1967706134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 0x55802b50d460 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.103+0000 7fbf20b87740 1 -- 172.21.15.16:0/1967706134 >> 172.21.15.16:0/1967706134 conn(0x55802b15c460 msgr2=0x55802b50b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:04.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.105+0000 7fbf20b87740 1 -- 172.21.15.16:0/1967706134 shutdown_connections 2024-08-28T19:38:04.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.105+0000 7fbf20b87740 1 -- 172.21.15.16:0/1967706134 wait complete. 2024-08-28T19:38:04.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.105+0000 7fbf20b87740 1 -- start start 2024-08-28T19:38:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.106+0000 7fbf20b87740 1 Processor -- start 2024-08-28T19:38:04.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.110+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b13f1b0 0x55802b518d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.110+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b147390 0x55802b5192c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.110+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 0x55802b524f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.110+0000 7fbf20b87740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55802b2d6700 con 0x55802b168070 2024-08-28T19:38:04.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.110+0000 7fbf20b87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55802b3fa700 con 0x55802b147390 2024-08-28T19:38:04.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.110+0000 7fbf20b87740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55802b51bbe0 con 0x55802b13f1b0 2024-08-28T19:38:04.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.110+0000 7fbf20b87740 10 client.0 _ll_register_callbacks cb 0x55802b429dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.110+0000 7fbf13fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b13f1b0 0x55802b518d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.110+0000 7fbf13fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b13f1b0 0x55802b518d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:35648/0 (socket says 172.21.15.16:35648) 2024-08-28T19:38:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.110+0000 7fbf13fff640 1 -- 172.21.15.16:0/2114466498 learned_addr learned my addr 172.21.15.16:0/2114466498 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.110+0000 7fbf137fe640 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b147390 0x55802b5192c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.111+0000 7fbf18baf640 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 0x55802b524f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.111+0000 7fbf13fff640 1 -- 172.21.15.16:0/2114466498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b147390 msgr2=0x55802b5192c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.111+0000 7fbf13fff640 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b147390 0x55802b5192c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.111+0000 7fbf13fff640 1 -- 172.21.15.16:0/2114466498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 msgr2=0x55802b524f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.111+0000 7fbf13fff640 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 0x55802b524f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.111+0000 7fbf13fff640 1 -- 172.21.15.16:0/2114466498 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55802b4338b0 con 0x55802b13f1b0 2024-08-28T19:38:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.111+0000 7fbf18baf640 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 0x55802b524f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:38:04.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.111+0000 7fbf13fff640 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b13f1b0 0x55802b518d00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0042f0 tx=0x7fbf0c003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=834d8afba28b316d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:04.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.111+0000 7fbef37fe640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:38:04.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.112+0000 7fbef37fe640 1 -- 172.21.15.16:0/2114466498 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c013c40 con 0x55802b13f1b0 2024-08-28T19:38:04.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.112+0000 7fbf20b87740 1 -- 172.21.15.16:0/2114466498 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf08024070 con 0x55802b13f1b0 2024-08-28T19:38:04.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.112+0000 7fbef37fe640 1 -- 172.21.15.16:0/2114466498 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9578+0+0 (secure 0 0 0) 0x7fbf0c013de0 con 0x55802b13f1b0 2024-08-28T19:38:04.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.112+0000 7fbef37fe640 1 -- 172.21.15.16:0/2114466498 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fbf0c01c440 con 0x55802b13f1b0 2024-08-28T19:38:04.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.112+0000 7fbf20b87740 10 client.28624 fetch_fsmap learned FSMap version 90 2024-08-28T19:38:04.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.112+0000 7fbf20b87740 1 -- 172.21.15.16:0/2114466498 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=4+}) -- 0x55802b538b90 con 0x55802b13f1b0 2024-08-28T19:38:04.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.114+0000 7fbef37fe640 1 -- 172.21.15.16:0/2114466498 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf08024070 con 0x55802b13f1b0 2024-08-28T19:38:04.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.114+0000 7fbef37fe640 1 -- 172.21.15.16:0/2114466498 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 90) ==== 8406+0+0 (secure 0 0 0) 0x7fbf0c01a070 con 0x55802b13f1b0 2024-08-28T19:38:04.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.114+0000 7fbf20b87740 10 client.28624 fetch_fsmap finished waiting for FSMap version 90 2024-08-28T19:38:04.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.114+0000 7fbf20b87740 10 client.28624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:04.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.114+0000 7fbf20b87740 20 client.28624 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:04.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.114+0000 7fbf20b87740 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55802b1269f0 0x55802b53c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.114+0000 7fbf20b87740 4 client.28624 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:38:04.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.114+0000 7fbf20b87740 1 -- 172.21.15.16:0/2114466498 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55802b486580 con 0x55802b1269f0 2024-08-28T19:38:04.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.114+0000 7fbf137fe640 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55802b1269f0 0x55802b53c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.115+0000 7fbf137fe640 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55802b1269f0 0x55802b53c070 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2eb6dc6d13a831b9 server_cookie=7355f5de53af438 in_seq=0 out_seq=0 2024-08-28T19:38:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.117+0000 7fbef37fe640 10 client.28624 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbef37fe640 1 -- 172.21.15.16:0/2114466498 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55802b486580 con 0x55802b1269f0 2024-08-28T19:38:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbef37fe640 10 client.28624 handle_command_reply: tid=0 2024-08-28T19:38:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbf20b87740 1 client.28624 shutdown 2024-08-28T19:38:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbf20b87740 1 -- 172.21.15.16:0/2114466498 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b13f1b0 msgr2=0x55802b518d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbf20b87740 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b13f1b0 0x55802b518d00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0042f0 tx=0x7fbf0c003f90 comp rx=0 tx=0).stop 2024-08-28T19:38:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbf20b87740 1 -- 172.21.15.16:0/2114466498 shutdown_connections 2024-08-28T19:38:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbf20b87740 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b168070 0x55802b524f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbf20b87740 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b147390 0x55802b5192c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbf20b87740 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55802b1269f0 0x55802b53c070 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbf20b87740 1 --2- 172.21.15.16:0/2114466498 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b13f1b0 0x55802b518d00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbf20b87740 1 -- 172.21.15.16:0/2114466498 >> 172.21.15.16:0/2114466498 conn(0x55802b15c460 msgr2=0x55802b510640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.118+0000 7fbef37fe640 0 client.28624 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.119+0000 7fbf20b87740 1 -- 172.21.15.16:0/2114466498 shutdown_connections 2024-08-28T19:38:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.119+0000 7fbf20b87740 1 -- 172.21.15.16:0/2114466498 wait complete. 2024-08-28T19:38:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.119+0000 7fbf20b87740 20 client.28624 trim_cache size 0 max 16384 2024-08-28T19:38:04.155 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:03 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:04.155 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:03 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.155 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:03 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.155 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:03 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.155 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:03 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.153+0000 7fbf20b87740 1 Processor -- start 2024-08-28T19:38:04.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.154+0000 7fbf20b87740 1 -- start start 2024-08-28T19:38:04.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.155+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b13f1b0 0x55802b524e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.155+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b147390 0x55802b50b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.155+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b168070 0x55802b515ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.155+0000 7fbf20b87740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf141a6270 con 0x55802b147390 2024-08-28T19:38:04.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.155+0000 7fbf137fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b147390 0x55802b50b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.155+0000 7fbf137fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b147390 0x55802b50b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33668/0 (socket says 172.21.15.16:33668) 2024-08-28T19:38:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.155+0000 7fbf137fe640 1 -- 172.21.15.16:0/2534535443 learned_addr learned my addr 172.21.15.16:0/2534535443 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.155+0000 7fbf13fff640 1 --2- 172.21.15.16:0/2534535443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b13f1b0 0x55802b524e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.155+0000 7fbf18baf640 1 --2- 172.21.15.16:0/2534535443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b168070 0x55802b515ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.156+0000 7fbf20b87740 1 -- 172.21.15.16:0/2534535443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf141a7f30 con 0x55802b13f1b0 2024-08-28T19:38:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.156+0000 7fbf20b87740 1 -- 172.21.15.16:0/2534535443 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbf141a6a00 con 0x55802b168070 2024-08-28T19:38:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.156+0000 7fbf137fe640 1 -- 172.21.15.16:0/2534535443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b168070 msgr2=0x55802b515ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.156+0000 7fbf137fe640 1 --2- 172.21.15.16:0/2534535443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b168070 0x55802b515ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.156+0000 7fbf137fe640 1 -- 172.21.15.16:0/2534535443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b13f1b0 msgr2=0x55802b524e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.156+0000 7fbf137fe640 1 --2- 172.21.15.16:0/2534535443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b13f1b0 0x55802b524e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.156+0000 7fbf137fe640 1 -- 172.21.15.16:0/2534535443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55802b517200 con 0x55802b147390 2024-08-28T19:38:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.157+0000 7fbf137fe640 1 --2- 172.21.15.16:0/2534535443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b147390 0x55802b50b210 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fbf040d0640 tx=0x7fbf0402a910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f360628bac4daa9a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.157+0000 7fbef9ffb640 1 -- 172.21.15.16:0/2534535443 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf040d40d0 con 0x55802b147390 2024-08-28T19:38:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.157+0000 7fbef9ffb640 1 -- 172.21.15.16:0/2534535443 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf040d4cc0 con 0x55802b147390 2024-08-28T19:38:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.157+0000 7fbef9ffb640 1 -- 172.21.15.16:0/2534535443 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf040d0850 con 0x55802b147390 2024-08-28T19:38:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.158+0000 7fbf20b87740 1 -- 172.21.15.16:0/2534535443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b147390 msgr2=0x55802b50b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.158+0000 7fbf20b87740 1 --2- 172.21.15.16:0/2534535443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b147390 0x55802b50b210 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fbf040d0640 tx=0x7fbf0402a910 comp rx=0 tx=0).stop 2024-08-28T19:38:04.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.159+0000 7fbf20b87740 1 -- 172.21.15.16:0/2534535443 shutdown_connections 2024-08-28T19:38:04.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.159+0000 7fbf20b87740 1 --2- 172.21.15.16:0/2534535443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b168070 0x55802b515ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.159+0000 7fbf20b87740 1 --2- 172.21.15.16:0/2534535443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b147390 0x55802b50b210 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.159+0000 7fbf20b87740 1 --2- 172.21.15.16:0/2534535443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b13f1b0 0x55802b524e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.159+0000 7fbf20b87740 1 -- 172.21.15.16:0/2534535443 >> 172.21.15.16:0/2534535443 conn(0x55802b15c460 msgr2=0x55802b50ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:04.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.163+0000 7fbf20b87740 1 -- 172.21.15.16:0/2534535443 shutdown_connections 2024-08-28T19:38:04.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.163+0000 7fbf20b87740 1 -- 172.21.15.16:0/2534535443 wait complete. 2024-08-28T19:38:04.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.163+0000 7fbf20b87740 1 -- start start 2024-08-28T19:38:04.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.164+0000 7fbf20b87740 1 Processor -- start 2024-08-28T19:38:04.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.164+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b13f1b0 0x55802b521590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.164+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b147390 0x55802b51da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.164+0000 7fbf20b87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b168070 0x55802b51df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.164+0000 7fbf20b87740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf141a7830 con 0x55802b13f1b0 2024-08-28T19:38:04.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.164+0000 7fbf20b87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf141a7100 con 0x55802b168070 2024-08-28T19:38:04.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.164+0000 7fbf20b87740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbf14073c40 con 0x55802b147390 2024-08-28T19:38:04.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.164+0000 7fbf20b87740 10 client.0 _ll_register_callbacks cb 0x7fbf0c009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:04.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.165+0000 7fbf18baf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b168070 0x55802b51df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.165+0000 7fbf18baf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b168070 0x55802b51df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:42454/0 (socket says 172.21.15.16:42454) 2024-08-28T19:38:04.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.165+0000 7fbf18baf640 1 -- 172.21.15.16:0/3245998048 learned_addr learned my addr 172.21.15.16:0/3245998048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:04.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.165+0000 7fbf137fe640 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b147390 0x55802b51da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.165+0000 7fbf18baf640 1 -- 172.21.15.16:0/3245998048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b147390 msgr2=0x55802b51da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.165+0000 7fbf18baf640 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b147390 0x55802b51da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.165+0000 7fbf18baf640 1 -- 172.21.15.16:0/3245998048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b13f1b0 msgr2=0x55802b521590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.165+0000 7fbf18baf640 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b13f1b0 0x55802b521590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.165+0000 7fbf18baf640 1 -- 172.21.15.16:0/3245998048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55802b51e680 con 0x55802b168070 2024-08-28T19:38:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.166+0000 7fbf18baf640 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b168070 0x55802b51df90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800afd0 tx=0x7fbf08006d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab37e03495e4697e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.166+0000 7fbef37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:38:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.166+0000 7fbef37fe640 1 -- 172.21.15.16:0/3245998048 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0802e070 con 0x55802b168070 2024-08-28T19:38:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.166+0000 7fbf20b87740 1 -- 172.21.15.16:0/3245998048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf040d4cc0 con 0x55802b168070 2024-08-28T19:38:04.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.167+0000 7fbef37fe640 1 -- 172.21.15.16:0/3245998048 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9578+0+0 (secure 0 0 0) 0x7fbf08018850 con 0x55802b168070 2024-08-28T19:38:04.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.167+0000 7fbef37fe640 1 -- 172.21.15.16:0/3245998048 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fbf08018a50 con 0x55802b168070 2024-08-28T19:38:04.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.168+0000 7fbf20b87740 10 client.29039 fetch_fsmap learned FSMap version 90 2024-08-28T19:38:04.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.168+0000 7fbf20b87740 1 -- 172.21.15.16:0/3245998048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=4+}) -- 0x55802b5b88f0 con 0x55802b168070 2024-08-28T19:38:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.168+0000 7fbef37fe640 1 -- 172.21.15.16:0/3245998048 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf040d4cc0 con 0x55802b168070 2024-08-28T19:38:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.168+0000 7fbef37fe640 1 -- 172.21.15.16:0/3245998048 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 90) ==== 8406+0+0 (secure 0 0 0) 0x7fbf08032b80 con 0x55802b168070 2024-08-28T19:38:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.169+0000 7fbf20b87740 10 client.29039 fetch_fsmap finished waiting for FSMap version 90 2024-08-28T19:38:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.169+0000 7fbf20b87740 10 client.29039 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.169+0000 7fbf20b87740 20 client.29039 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.169+0000 7fbf20b87740 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55802b0fbda0 0x55802b5bae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.169+0000 7fbf20b87740 4 client.29039 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:38:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.169+0000 7fbf20b87740 1 -- 172.21.15.16:0/3245998048 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55802b482020 con 0x55802b0fbda0 2024-08-28T19:38:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.169+0000 7fbf13fff640 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55802b0fbda0 0x55802b5bae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:04.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.172+0000 7fbf13fff640 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55802b0fbda0 0x55802b5bae90 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a10ec5fb1d9d024 server_cookie=1a5134aaa96d1714 in_seq=0 out_seq=0 2024-08-28T19:38:04.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.172+0000 7fbef37fe640 10 client.29039 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:04.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.176+0000 7fbef37fe640 1 -- 172.21.15.16:0/3245998048 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+6818 (crc 0 0 0) 0x55802b482020 con 0x55802b0fbda0 2024-08-28T19:38:04.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.176+0000 7fbef37fe640 10 client.29039 handle_command_reply: tid=0 2024-08-28T19:38:04.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.177+0000 7fbf20b87740 1 client.29039 shutdown 2024-08-28T19:38:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.179+0000 7fbf20b87740 1 -- 172.21.15.16:0/3245998048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b168070 msgr2=0x55802b51df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.179+0000 7fbf20b87740 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b168070 0x55802b51df90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800afd0 tx=0x7fbf08006d20 comp rx=0 tx=0).stop 2024-08-28T19:38:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.180+0000 7fbf20b87740 1 -- 172.21.15.16:0/3245998048 shutdown_connections 2024-08-28T19:38:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.180+0000 7fbf20b87740 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55802b168070 0x55802b51df90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.180+0000 7fbf20b87740 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55802b0fbda0 0x55802b5bae90 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.180+0000 7fbf20b87740 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55802b147390 0x55802b51da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.180+0000 7fbf20b87740 1 --2- 172.21.15.16:0/3245998048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55802b13f1b0 0x55802b521590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.180+0000 7fbef37fe640 0 client.29039 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.180+0000 7fbf20b87740 1 -- 172.21.15.16:0/3245998048 >> 172.21.15.16:0/3245998048 conn(0x55802b15c460 msgr2=0x55802b510560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.180+0000 7fbf20b87740 1 -- 172.21.15.16:0/3245998048 shutdown_connections 2024-08-28T19:38:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.180+0000 7fbf20b87740 1 -- 172.21.15.16:0/3245998048 wait complete. 2024-08-28T19:38:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.180+0000 7fbf20b87740 20 client.29039 trim_cache size 0 max 16384 2024-08-28T19:38:04.183 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:38:04.183 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:38:04.183 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:38:04.183 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:38:04.183 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.183 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "f27568c6-5b0e-422a-bf5c-feccae126ec2", 2024-08-28T19:38:04.183 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:38:04.183 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:04.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.183 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.184 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.186 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.188 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.189 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.190 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:38:04.190 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:38:04.190 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.190 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.191 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.192 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:38:04.193 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.194 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.195 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:04.197 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:04.198 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:04.198 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:04.198 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:04.198 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:38:04.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 -- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbefc0b4eb0 msgr2=0x7fbefc0b73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:04.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbefc0b4eb0 0x7fbefc0b73c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009000 tx=0x7fbf0801b000 comp rx=0 tx=0).stop 2024-08-28T19:38:04.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 -- 172.21.15.16:0/3537161415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14107f50 msgr2=0x7fbf141995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:04.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14107f50 0x7fbf141995a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009850 tx=0x7fbf04003c00 comp rx=0 tx=0).stop 2024-08-28T19:38:04.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 -- 172.21.15.16:0/3537161415 shutdown_connections 2024-08-28T19:38:04.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf141088a0 0x7fbf1419e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fbefc0b4eb0 0x7fbefc0b73c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14107f50 0x7fbf141995a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 --2- 172.21.15.16:0/3537161415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbf14102610 0x7fbf14199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:04.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 -- 172.21.15.16:0/3537161415 >> 172.21.15.16:0/3537161415 conn(0x7fbf140fe060 msgr2=0x7fbf14100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:04.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 -- 172.21.15.16:0/3537161415 shutdown_connections 2024-08-28T19:38:04.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:04.181+0000 7fbf193b0640 1 -- 172.21.15.16:0/3537161415 wait complete. 2024-08-28T19:38:04.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:03 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:04.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:03 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:03 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:03 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:03 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:03 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:03 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:03 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:03 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:03 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:04.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 80415 22.77 MB/sec execute 100 sec latency 373.919 ms 2024-08-28T19:38:05.051 INFO:tasks.cephfs.filesystem:scrub status for tag:f27568c6-5b0e-422a-bf5c-feccae126ec2 - {'path': '/', 'tag': 'f27568c6-5b0e-422a-bf5c-feccae126ec2', 'options': 'recursive,force'} 2024-08-28T19:38:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:04 smithi016 ceph-mon[18556]: pgmap v870: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 16 MiB/s wr, 740 op/s 2024-08-28T19:38:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:04 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:04 smithi016 ceph-mon[18556]: osdmap e105: 12 total, 12 up, 12 in 2024-08-28T19:38:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:04 smithi028 ceph-mon[21769]: pgmap v870: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 16 MiB/s wr, 740 op/s 2024-08-28T19:38:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:04 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:04 smithi028 ceph-mon[21769]: osdmap e105: 12 total, 12 up, 12 in 2024-08-28T19:38:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:04 smithi115 ceph-mon[22184]: pgmap v870: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 16 MiB/s wr, 740 op/s 2024-08-28T19:38:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:04 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:04 smithi115 ceph-mon[22184]: osdmap e105: 12 total, 12 up, 12 in 2024-08-28T19:38:05.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 80858 22.72 MB/sec execute 101 sec latency 234.590 ms 2024-08-28T19:38:06.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:06 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:06.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:06 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:06.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 81441 22.78 MB/sec execute 102 sec latency 721.816 ms 2024-08-28T19:38:06.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:06 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:07 smithi028 ceph-mon[21769]: pgmap v872: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-08-28T19:38:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:07 smithi115 ceph-mon[22184]: pgmap v872: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-08-28T19:38:07.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 82326 23.03 MB/sec execute 103 sec latency 1721.851 ms 2024-08-28T19:38:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:07 smithi016 ceph-mon[18556]: pgmap v872: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-08-28T19:38:08.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 82957 23.21 MB/sec execute 104 sec latency 2721.880 ms 2024-08-28T19:38:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:09 smithi028 ceph-mon[21769]: pgmap v873: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 16 MiB/s wr, 1.01k op/s 2024-08-28T19:38:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:09 smithi115 ceph-mon[22184]: pgmap v873: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 16 MiB/s wr, 1.01k op/s 2024-08-28T19:38:09.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 83298 23.11 MB/sec execute 105 sec latency 3721.909 ms 2024-08-28T19:38:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:09 smithi016 ceph-mon[18556]: pgmap v873: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 16 MiB/s wr, 1.01k op/s 2024-08-28T19:38:10.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 83807 23.18 MB/sec execute 106 sec latency 4721.942 ms 2024-08-28T19:38:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:11 smithi028 ceph-mon[21769]: pgmap v874: 129 pgs: 129 active+clean; 1.3 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.8 MiB/s wr, 867 op/s 2024-08-28T19:38:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:11 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:11 smithi028 ceph-mon[21769]: osdmap e106: 12 total, 12 up, 12 in 2024-08-28T19:38:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:11 smithi115 ceph-mon[22184]: pgmap v874: 129 pgs: 129 active+clean; 1.3 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.8 MiB/s wr, 867 op/s 2024-08-28T19:38:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:11 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:11 smithi115 ceph-mon[22184]: osdmap e106: 12 total, 12 up, 12 in 2024-08-28T19:38:11.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 84117 23.11 MB/sec execute 107 sec latency 5721.994 ms 2024-08-28T19:38:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:11 smithi016 ceph-mon[18556]: pgmap v874: 129 pgs: 129 active+clean; 1.3 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.8 MiB/s wr, 867 op/s 2024-08-28T19:38:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:11 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:11 smithi016 ceph-mon[18556]: osdmap e106: 12 total, 12 up, 12 in 2024-08-28T19:38:12.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 84305 22.99 MB/sec execute 108 sec latency 6722.026 ms 2024-08-28T19:38:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:13 smithi028 ceph-mon[21769]: pgmap v876: 129 pgs: 129 active+clean; 1.3 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 8.4 MiB/s wr, 893 op/s 2024-08-28T19:38:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:13 smithi115 ceph-mon[22184]: pgmap v876: 129 pgs: 129 active+clean; 1.3 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 8.4 MiB/s wr, 893 op/s 2024-08-28T19:38:13.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 84434 22.84 MB/sec execute 109 sec latency 7722.074 ms 2024-08-28T19:38:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:13 smithi016 ceph-mon[18556]: pgmap v876: 129 pgs: 129 active+clean; 1.3 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 8.4 MiB/s wr, 893 op/s 2024-08-28T19:38:14.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:38:14.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:38:14.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:14 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:38:14.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:38:14.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:38:14.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:14 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:38:14.473 INFO:tasks.workunit.client.0.smithi016.stdout: 10 84663 22.73 MB/sec execute 110 sec latency 8722.139 ms 2024-08-28T19:38:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:38:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:38:14.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:14 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:38:15.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:15 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:38:15.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:15 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:38:15.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:15 smithi028 ceph-mon[21769]: pgmap v877: 129 pgs: 129 active+clean; 1.3 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 6.9 MiB/s wr, 732 op/s 2024-08-28T19:38:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:15 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:38:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:15 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:38:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:15 smithi115 ceph-mon[22184]: pgmap v877: 129 pgs: 129 active+clean; 1.3 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 6.9 MiB/s wr, 732 op/s 2024-08-28T19:38:15.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:15 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:38:15.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:15 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:38:15.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:15 smithi016 ceph-mon[18556]: pgmap v877: 129 pgs: 129 active+clean; 1.3 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 6.9 MiB/s wr, 732 op/s 2024-08-28T19:38:15.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85030 22.73 MB/sec execute 111 sec latency 9722.172 ms 2024-08-28T19:38:16.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85251 22.62 MB/sec execute 112 sec latency 10722.599 ms 2024-08-28T19:38:17.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:17 smithi028 ceph-mon[21769]: pgmap v878: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.6 MiB/s wr, 517 op/s 2024-08-28T19:38:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:17 smithi115 ceph-mon[22184]: pgmap v878: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.6 MiB/s wr, 517 op/s 2024-08-28T19:38:17.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85504 22.53 MB/sec execute 113 sec latency 11722.633 ms 2024-08-28T19:38:17.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:17 smithi016 ceph-mon[18556]: pgmap v878: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.6 MiB/s wr, 517 op/s 2024-08-28T19:38:18.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:18 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:18.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:18 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:18.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85827 22.50 MB/sec execute 114 sec latency 12722.675 ms 2024-08-28T19:38:18.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:18 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:19.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:19 smithi028 ceph-mon[21769]: pgmap v879: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 417 op/s 2024-08-28T19:38:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:19 smithi115 ceph-mon[22184]: pgmap v879: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 417 op/s 2024-08-28T19:38:19.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 86328 22.52 MB/sec execute 115 sec latency 13722.705 ms 2024-08-28T19:38:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:19 smithi016 ceph-mon[18556]: pgmap v879: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 417 op/s 2024-08-28T19:38:20.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 86925 22.59 MB/sec execute 116 sec latency 13954.861 ms 2024-08-28T19:38:21.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:21 smithi028 ceph-mon[21769]: pgmap v880: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 6.6 MiB/s wr, 383 op/s 2024-08-28T19:38:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:21 smithi115 ceph-mon[22184]: pgmap v880: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 6.6 MiB/s wr, 383 op/s 2024-08-28T19:38:21.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 87720 22.73 MB/sec execute 117 sec latency 10260.136 ms 2024-08-28T19:38:21.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:21 smithi016 ceph-mon[18556]: pgmap v880: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 6.6 MiB/s wr, 383 op/s 2024-08-28T19:38:22.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88483 22.92 MB/sec execute 118 sec latency 10481.593 ms 2024-08-28T19:38:23.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:23 smithi028 ceph-mon[21769]: pgmap v881: 129 pgs: 129 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.0 MiB/s wr, 482 op/s 2024-08-28T19:38:23.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:23 smithi115 ceph-mon[22184]: pgmap v881: 129 pgs: 129 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.0 MiB/s wr, 482 op/s 2024-08-28T19:38:23.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 89216 23.11 MB/sec execute 119 sec latency 69.359 ms 2024-08-28T19:38:23.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:23 smithi016 ceph-mon[18556]: pgmap v881: 129 pgs: 129 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.0 MiB/s wr, 482 op/s 2024-08-28T19:38:24.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 89873 23.17 MB/sec execute 120 sec latency 91.439 ms 2024-08-28T19:38:25.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:25 smithi115 ceph-mon[22184]: pgmap v882: 129 pgs: 129 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 6.2 MiB/s wr, 343 op/s 2024-08-28T19:38:25.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:25 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:25.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 90607 23.27 MB/sec execute 121 sec latency 79.195 ms 2024-08-28T19:38:25.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:25 smithi016 ceph-mon[18556]: pgmap v882: 129 pgs: 129 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 6.2 MiB/s wr, 343 op/s 2024-08-28T19:38:25.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:25 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:25 smithi028 ceph-mon[21769]: pgmap v882: 129 pgs: 129 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 6.2 MiB/s wr, 343 op/s 2024-08-28T19:38:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:25 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:26.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 91323 23.41 MB/sec execute 122 sec latency 227.449 ms 2024-08-28T19:38:27.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 91986 23.51 MB/sec execute 123 sec latency 97.138 ms 2024-08-28T19:38:27.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:27 smithi115 ceph-mon[22184]: pgmap v883: 129 pgs: 129 active+clean; 1.3 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.0 MiB/s wr, 474 op/s 2024-08-28T19:38:27.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:27 smithi016 ceph-mon[18556]: pgmap v883: 129 pgs: 129 active+clean; 1.3 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.0 MiB/s wr, 474 op/s 2024-08-28T19:38:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:27 smithi028 ceph-mon[21769]: pgmap v883: 129 pgs: 129 active+clean; 1.3 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.0 MiB/s wr, 474 op/s 2024-08-28T19:38:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 92547 23.57 MB/sec execute 124 sec latency 476.245 ms 2024-08-28T19:38:29.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 93190 23.64 MB/sec execute 125 sec latency 182.786 ms 2024-08-28T19:38:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:29 smithi115 ceph-mon[22184]: pgmap v884: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.0 MiB/s wr, 522 op/s 2024-08-28T19:38:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:29 smithi016 ceph-mon[18556]: pgmap v884: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.0 MiB/s wr, 522 op/s 2024-08-28T19:38:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:29 smithi028 ceph-mon[21769]: pgmap v884: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.0 MiB/s wr, 522 op/s 2024-08-28T19:38:30.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 94007 23.88 MB/sec execute 126 sec latency 126.477 ms 2024-08-28T19:38:31.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 94480 23.85 MB/sec execute 127 sec latency 301.725 ms 2024-08-28T19:38:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:31 smithi016 ceph-mon[18556]: pgmap v885: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 8.8 MiB/s wr, 626 op/s 2024-08-28T19:38:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:31 smithi028 ceph-mon[21769]: pgmap v885: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 8.8 MiB/s wr, 626 op/s 2024-08-28T19:38:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:31 smithi115 ceph-mon[22184]: pgmap v885: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 8.8 MiB/s wr, 626 op/s 2024-08-28T19:38:32.474 INFO:tasks.workunit.client.0.smithi016.stdout: 10 94511 23.66 MB/sec execute 128 sec latency 1269.717 ms 2024-08-28T19:38:33.475 INFO:tasks.workunit.client.0.smithi016.stdout: 10 94707 23.54 MB/sec execute 129 sec latency 1522.405 ms 2024-08-28T19:38:33.672 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:33 smithi115 ceph-mon[22184]: pgmap v886: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 9.0 MiB/s wr, 676 op/s 2024-08-28T19:38:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:33 smithi016 ceph-mon[18556]: pgmap v886: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 9.0 MiB/s wr, 676 op/s 2024-08-28T19:38:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:33 smithi028 ceph-mon[21769]: pgmap v886: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 9.0 MiB/s wr, 676 op/s 2024-08-28T19:38:34.475 INFO:tasks.workunit.client.0.smithi016.stdout: 10 94987 23.48 MB/sec execute 130 sec latency 225.351 ms 2024-08-28T19:38:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:34 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:34 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:34 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:35.052 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:38:35.475 INFO:tasks.workunit.client.0.smithi016.stdout: 10 95363 23.47 MB/sec execute 131 sec latency 159.037 ms 2024-08-28T19:38:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:35.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:35 smithi016 ceph-mon[18556]: pgmap v887: 129 pgs: 129 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 8.9 MiB/s wr, 584 op/s 2024-08-28T19:38:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:35 smithi028 ceph-mon[21769]: pgmap v887: 129 pgs: 129 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 8.9 MiB/s wr, 584 op/s 2024-08-28T19:38:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:35 smithi115 ceph-mon[22184]: pgmap v887: 129 pgs: 129 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 8.9 MiB/s wr, 584 op/s 2024-08-28T19:38:36.475 INFO:tasks.workunit.client.0.smithi016.stdout: 10 95833 23.47 MB/sec execute 132 sec latency 149.695 ms 2024-08-28T19:38:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.572+0000 7f61d429d640 1 -- 172.21.15.16:0/2784764422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61cc10a780 msgr2=0x7f61cc10ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.572+0000 7f61d429d640 1 --2- 172.21.15.16:0/2784764422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61cc10a780 0x7f61cc10ab90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f61c8009860 tx=0x7f61c802a340 comp rx=0 tx=0).stop 2024-08-28T19:38:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.573+0000 7f61d429d640 1 -- 172.21.15.16:0/2784764422 shutdown_connections 2024-08-28T19:38:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.573+0000 7f61d429d640 1 --2- 172.21.15.16:0/2784764422 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61cc11c9d0 0x7f61cc11eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.573+0000 7f61d429d640 1 --2- 172.21.15.16:0/2784764422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61cc10a780 0x7f61cc10ab90 secure :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f61c8009860 tx=0x7f61c802a340 comp rx=0 tx=0).stop 2024-08-28T19:38:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.573+0000 7f61d429d640 1 --2- 172.21.15.16:0/2784764422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc111ce0 0x7f61cc10a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.573+0000 7f61d429d640 1 -- 172.21.15.16:0/2784764422 >> 172.21.15.16:0/2784764422 conn(0x7f61cc070c60 msgr2=0x7f61cc071070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:36.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.575+0000 7f61d429d640 1 -- 172.21.15.16:0/2784764422 shutdown_connections 2024-08-28T19:38:36.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.576+0000 7f61d429d640 1 -- 172.21.15.16:0/2784764422 wait complete. 2024-08-28T19:38:36.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.577+0000 7f61d429d640 1 Processor -- start 2024-08-28T19:38:36.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.577+0000 7f61d429d640 1 -- start start 2024-08-28T19:38:36.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.578+0000 7f61d429d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc111ce0 0x7f61cc1a91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.578+0000 7f61d429d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61cc11c9d0 0x7f61cc1a9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.578+0000 7f61d2039640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc111ce0 0x7f61cc1a91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.578+0000 7f61d2039640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc111ce0 0x7f61cc1a91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51356/0 (socket says 172.21.15.16:51356) 2024-08-28T19:38:36.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.578+0000 7f61d2039640 1 -- 172.21.15.16:0/2170927582 learned_addr learned my addr 172.21.15.16:0/2170927582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:36.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.578+0000 7f61d429d640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61cc1ae960 0x7f61cc1aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.578+0000 7f61d429d640 1 -- 172.21.15.16:0/2170927582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc122610 con 0x7f61cc111ce0 2024-08-28T19:38:36.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.578+0000 7f61d429d640 1 -- 172.21.15.16:0/2170927582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc122490 con 0x7f61cc11c9d0 2024-08-28T19:38:36.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.578+0000 7f61d429d640 1 -- 172.21.15.16:0/2170927582 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc122790 con 0x7f61cc1ae960 2024-08-28T19:38:36.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.578+0000 7f61d283a640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61cc1ae960 0x7f61cc1aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.579+0000 7f61d283a640 1 -- 172.21.15.16:0/2170927582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61cc11c9d0 msgr2=0x7f61cc1a9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:36.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.579+0000 7f61d283a640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61cc11c9d0 0x7f61cc1a9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.579+0000 7f61d283a640 1 -- 172.21.15.16:0/2170927582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc111ce0 msgr2=0x7f61cc1a91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:36.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.579+0000 7f61d283a640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc111ce0 0x7f61cc1a91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.579+0000 7f61d283a640 1 -- 172.21.15.16:0/2170927582 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61cc1af540 con 0x7f61cc1ae960 2024-08-28T19:38:36.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.579+0000 7f61d283a640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61cc1ae960 0x7f61cc1aed70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f61c400ef00 tx=0x7f61c400c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:36.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.581+0000 7f61c37fe640 1 -- 172.21.15.16:0/2170927582 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4004b20 con 0x7f61cc1ae960 2024-08-28T19:38:36.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.581+0000 7f61c37fe640 1 -- 172.21.15.16:0/2170927582 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61c4003e20 con 0x7f61cc1ae960 2024-08-28T19:38:36.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.581+0000 7f61d429d640 1 -- 172.21.15.16:0/2170927582 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61cc1b5ae0 con 0x7f61cc1ae960 2024-08-28T19:38:36.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.581+0000 7f61d429d640 1 -- 172.21.15.16:0/2170927582 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61cc1b5e80 con 0x7f61cc1ae960 2024-08-28T19:38:36.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.582+0000 7f61c37fe640 1 -- 172.21.15.16:0/2170927582 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c400b570 con 0x7f61cc1ae960 2024-08-28T19:38:36.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.585+0000 7f61da812740 1 Processor -- start 2024-08-28T19:38:36.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.587+0000 7f61c37fe640 1 -- 172.21.15.16:0/2170927582 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f61c401d020 con 0x7f61cc1ae960 2024-08-28T19:38:36.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.587+0000 7f61da812740 1 -- start start 2024-08-28T19:38:36.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.588+0000 7f61da812740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c65c070 0x56148ca01300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.588+0000 7f61da812740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c6331b0 0x56148ca01840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.588+0000 7f61da812740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c63b390 0x56148ca01d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.588+0000 7f61da812740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56148c978630 con 0x56148c63b390 2024-08-28T19:38:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.588+0000 7f61da812740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56148c967bc0 con 0x56148c6331b0 2024-08-28T19:38:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.588+0000 7f61da812740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56148c96bb70 con 0x56148c65c070 2024-08-28T19:38:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.588+0000 7f61d2039640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c6331b0 0x56148ca01840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.588+0000 7f61d2039640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c6331b0 0x56148ca01840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:40014/0 (socket says 172.21.15.16:40014) 2024-08-28T19:38:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.588+0000 7f61d2039640 1 -- 172.21.15.16:0/1484435543 learned_addr learned my addr 172.21.15.16:0/1484435543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.589+0000 7f61d1838640 1 --2- 172.21.15.16:0/1484435543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c65c070 0x56148ca01300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.589+0000 7f61d2039640 1 -- 172.21.15.16:0/1484435543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c65c070 msgr2=0x56148ca01300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:36.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.589+0000 7f61d1838640 1 --2- 172.21.15.16:0/1484435543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c63b390 0x56148ca01d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.590+0000 7f61c37fe640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61b80b4df0 0x7f61b80b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.590+0000 7f61d2039640 1 --2- 172.21.15.16:0/1484435543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c65c070 0x56148ca01300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.590+0000 7f61d2039640 1 -- 172.21.15.16:0/1484435543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c63b390 msgr2=0x56148ca01d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:36.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.590+0000 7f61d2039640 1 --2- 172.21.15.16:0/1484435543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c63b390 0x56148ca01d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.590+0000 7f61d2039640 1 -- 172.21.15.16:0/1484435543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56148c6c7650 con 0x56148c6331b0 2024-08-28T19:38:36.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.590+0000 7f61c37fe640 1 -- 172.21.15.16:0/2170927582 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f61c40e5730 con 0x7f61cc1ae960 2024-08-28T19:38:36.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.590+0000 7f61d1838640 1 --2- 172.21.15.16:0/1484435543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c63b390 0x56148ca01d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:38:36.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.590+0000 7f61d283a640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61b80b4df0 0x7f61b80b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.591+0000 7f61d2039640 1 --2- 172.21.15.16:0/1484435543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c6331b0 0x56148ca01840 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00bda0 tx=0x7f61bc0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15d7990b23a3363c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:36.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.591+0000 7f61d283a640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61b80b4df0 0x7f61b80b7300 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f61c40ab9e0 tx=0x7f61c400cf70 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:36.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.591+0000 7f61c0d9a640 1 -- 172.21.15.16:0/1484435543 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc01c680 con 0x56148c6331b0 2024-08-28T19:38:36.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.592+0000 7f61c0d9a640 1 -- 172.21.15.16:0/1484435543 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61bc0118f0 con 0x56148c6331b0 2024-08-28T19:38:36.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.593+0000 7f61da812740 1 -- 172.21.15.16:0/1484435543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c6331b0 msgr2=0x56148ca01840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:36.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.593+0000 7f61da812740 1 --2- 172.21.15.16:0/1484435543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c6331b0 0x56148ca01840 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00bda0 tx=0x7f61bc0107a0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.593+0000 7f61c0d9a640 1 -- 172.21.15.16:0/1484435543 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc01a800 con 0x56148c6331b0 2024-08-28T19:38:36.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.593+0000 7f61da812740 1 -- 172.21.15.16:0/1484435543 shutdown_connections 2024-08-28T19:38:36.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.593+0000 7f61da812740 1 --2- 172.21.15.16:0/1484435543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c63b390 0x56148ca01d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.593+0000 7f61da812740 1 --2- 172.21.15.16:0/1484435543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c6331b0 0x56148ca01840 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.593+0000 7f61da812740 1 --2- 172.21.15.16:0/1484435543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c65c070 0x56148ca01300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.593+0000 7f61da812740 1 -- 172.21.15.16:0/1484435543 >> 172.21.15.16:0/1484435543 conn(0x56148c650460 msgr2=0x56148c9ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:36.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.593+0000 7f61da812740 1 -- 172.21.15.16:0/1484435543 shutdown_connections 2024-08-28T19:38:36.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.594+0000 7f61da812740 1 -- 172.21.15.16:0/1484435543 wait complete. 2024-08-28T19:38:36.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.594+0000 7f61da812740 1 -- start start 2024-08-28T19:38:36.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.595+0000 7f61da812740 1 Processor -- start 2024-08-28T19:38:36.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.595+0000 7f61da812740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 0x56148ca0f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.595+0000 7f61da812740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c65c070 0x56148ca0c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.595+0000 7f61da812740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c61a9f0 0x56148ca0cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.595+0000 7f61da812740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56148c7ca700 con 0x56148c65c070 2024-08-28T19:38:36.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.595+0000 7f61da812740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56148c8ee700 con 0x56148c63b390 2024-08-28T19:38:36.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.596+0000 7f61da812740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56148ca0fb80 con 0x56148c61a9f0 2024-08-28T19:38:36.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.596+0000 7f61da812740 10 client.0 _ll_register_callbacks cb 0x56148c91ddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:36.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.596+0000 7f61d1838640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 0x56148ca0f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.596+0000 7f61d1838640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 0x56148ca0f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:40026/0 (socket says 172.21.15.16:40026) 2024-08-28T19:38:36.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.596+0000 7f61d1838640 1 -- 172.21.15.16:0/885180451 learned_addr learned my addr 172.21.15.16:0/885180451 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:36.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.596+0000 7f61d1838640 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c61a9f0 0x56148ca0cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.596+0000 7f61d2039640 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c65c070 0x56148ca0c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.597+0000 7f61d1838640 1 -- 172.21.15.16:0/885180451 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c61a9f0 msgr2=0x56148ca0cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.597+0000 7f61d1838640 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c61a9f0 0x56148ca0cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.597+0000 7f61d1838640 1 -- 172.21.15.16:0/885180451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c65c070 msgr2=0x56148ca0c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.597+0000 7f61d1838640 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c65c070 0x56148ca0c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.597+0000 7f61d1838640 1 -- 172.21.15.16:0/885180451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56148c9278b0 con 0x56148c63b390 2024-08-28T19:38:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.598+0000 7f61d1838640 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 0x56148ca0f130 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f61cc06c480 tx=0x7f61c80097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ee47456b5933c06 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.598+0000 7f61b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:38:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.598+0000 7f61b5ffb640 1 -- 172.21.15.16:0/885180451 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c8037040 con 0x56148c63b390 2024-08-28T19:38:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.598+0000 7f61b5ffb640 1 -- 172.21.15.16:0/885180451 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f61c8030860 con 0x56148c63b390 2024-08-28T19:38:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.598+0000 7f61da812740 1 -- 172.21.15.16:0/885180451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61bc0118f0 con 0x56148c63b390 2024-08-28T19:38:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.599+0000 7f61b5ffb640 1 -- 172.21.15.16:0/885180451 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f61c8030a60 con 0x56148c63b390 2024-08-28T19:38:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.599+0000 7f61da812740 10 client.29048 fetch_fsmap learned FSMap version 95 2024-08-28T19:38:36.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.599+0000 7f61da812740 1 -- 172.21.15.16:0/885180451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) -- 0x56148ca1b4d0 con 0x56148c63b390 2024-08-28T19:38:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.599+0000 7f61b5ffb640 1 -- 172.21.15.16:0/885180451 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61c8038040 con 0x56148c63b390 2024-08-28T19:38:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.599+0000 7f61b5ffb640 1 -- 172.21.15.16:0/885180451 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 95) ==== 8402+0+0 (secure 0 0 0) 0x7f61c8040ad0 con 0x56148c63b390 2024-08-28T19:38:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.599+0000 7f61da812740 10 client.29048 fetch_fsmap finished waiting for FSMap version 95 2024-08-28T19:38:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.599+0000 7f61da812740 10 client.29048 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.599+0000 7f61da812740 20 client.29048 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.599+0000 7f61da812740 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56148c5efda0 0x56148ca1ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.599+0000 7f61da812740 4 client.29048 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:38:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.599+0000 7f61da812740 1 -- 172.21.15.16:0/885180451 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f61bc0118f0 con 0x56148c5efda0 2024-08-28T19:38:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.600+0000 7f61d2039640 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56148c5efda0 0x56148ca1ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.600+0000 7f61b5ffb640 10 client.29048 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.601+0000 7f61d2039640 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56148c5efda0 0x56148ca1ea30 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=664dff407c2b7b54 server_cookie=e8b40a7be56c3cad in_seq=0 out_seq=0 2024-08-28T19:38:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.603+0000 7f61b5ffb640 1 -- 172.21.15.16:0/885180451 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f61bc0118f0 con 0x56148c5efda0 2024-08-28T19:38:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.603+0000 7f61b5ffb640 10 client.29048 handle_command_reply: tid=0 2024-08-28T19:38:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.603+0000 7f61da812740 1 client.29048 shutdown 2024-08-28T19:38:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.604+0000 7f61da812740 1 -- 172.21.15.16:0/885180451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 msgr2=0x56148ca0f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.604+0000 7f61da812740 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 0x56148ca0f130 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f61cc06c480 tx=0x7f61c80097b0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.604+0000 7f61da812740 1 -- 172.21.15.16:0/885180451 shutdown_connections 2024-08-28T19:38:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.604+0000 7f61da812740 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56148c5efda0 0x56148ca1ea30 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.604+0000 7f61b5ffb640 0 client.29048 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.604+0000 7f61da812740 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c61a9f0 0x56148ca0cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.604+0000 7f61da812740 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c65c070 0x56148ca0c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.604+0000 7f61da812740 1 --2- 172.21.15.16:0/885180451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 0x56148ca0f130 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.604+0000 7f61da812740 1 -- 172.21.15.16:0/885180451 >> 172.21.15.16:0/885180451 conn(0x56148c650460 msgr2=0x56148ca044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.605+0000 7f61da812740 1 -- 172.21.15.16:0/885180451 shutdown_connections 2024-08-28T19:38:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.606+0000 7f61da812740 1 -- 172.21.15.16:0/885180451 wait complete. 2024-08-28T19:38:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.606+0000 7f61da812740 20 client.29048 trim_cache size 0 max 16384 2024-08-28T19:38:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.635+0000 7f61da812740 1 Processor -- start 2024-08-28T19:38:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.635+0000 7f61da812740 1 -- start start 2024-08-28T19:38:36.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.635+0000 7f61da812740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca077b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.635+0000 7f61da812740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c61a9f0 0x56148ca097f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.635+0000 7f61da812740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c63b390 0x56148ca09d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.635+0000 7f61da812740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc1c0090 con 0x56148c63b390 2024-08-28T19:38:36.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.635+0000 7f61da812740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc1c1d50 con 0x56148c61a9f0 2024-08-28T19:38:36.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.636+0000 7f61da812740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc1c0820 con 0x56148c5efda0 2024-08-28T19:38:36.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.636+0000 7f61d1838640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca077b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.636+0000 7f61d1838640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca077b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:32832/0 (socket says 172.21.15.16:32832) 2024-08-28T19:38:36.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.636+0000 7f61d1838640 1 -- 172.21.15.16:0/1081841079 learned_addr learned my addr 172.21.15.16:0/1081841079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:36.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.636+0000 7f61d1838640 1 --2- 172.21.15.16:0/1081841079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c63b390 0x56148ca09d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.636+0000 7f61d2039640 1 --2- 172.21.15.16:0/1081841079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c61a9f0 0x56148ca097f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.636+0000 7f61d1838640 1 -- 172.21.15.16:0/1081841079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c61a9f0 msgr2=0x56148ca097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:36.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.636+0000 7f61d1838640 1 --2- 172.21.15.16:0/1081841079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c61a9f0 0x56148ca097f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.636+0000 7f61d1838640 1 -- 172.21.15.16:0/1081841079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c63b390 msgr2=0x56148ca09d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:36.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.636+0000 7f61d1838640 1 --2- 172.21.15.16:0/1081841079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c63b390 0x56148ca09d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.636+0000 7f61d1838640 1 -- 172.21.15.16:0/1081841079 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56148ca035f0 con 0x56148c5efda0 2024-08-28T19:38:36.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.637+0000 7f61d1838640 1 --2- 172.21.15.16:0/1081841079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca077b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f61c8032c50 tx=0x7f61c8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77bf93a7a5f169a4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:36.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.638+0000 7f61b7fff640 1 -- 172.21.15.16:0/1081841079 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c8037040 con 0x56148c5efda0 2024-08-28T19:38:36.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.638+0000 7f61b7fff640 1 -- 172.21.15.16:0/1081841079 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61c8030660 con 0x56148c5efda0 2024-08-28T19:38:36.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.638+0000 7f61b7fff640 1 -- 172.21.15.16:0/1081841079 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c8042670 con 0x56148c5efda0 2024-08-28T19:38:36.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.640+0000 7f61da812740 1 -- 172.21.15.16:0/1081841079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 msgr2=0x56148ca077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:36.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.640+0000 7f61da812740 1 --2- 172.21.15.16:0/1081841079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca077b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f61c8032c50 tx=0x7f61c8000bc0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.640+0000 7f61da812740 1 -- 172.21.15.16:0/1081841079 shutdown_connections 2024-08-28T19:38:36.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.640+0000 7f61da812740 1 --2- 172.21.15.16:0/1081841079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c63b390 0x56148ca09d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.640+0000 7f61da812740 1 --2- 172.21.15.16:0/1081841079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c61a9f0 0x56148ca097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.640+0000 7f61da812740 1 --2- 172.21.15.16:0/1081841079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca077b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.640+0000 7f61da812740 1 -- 172.21.15.16:0/1081841079 >> 172.21.15.16:0/1081841079 conn(0x56148c650460 msgr2=0x56148ca0ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.640+0000 7f61da812740 1 -- 172.21.15.16:0/1081841079 shutdown_connections 2024-08-28T19:38:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.640+0000 7f61da812740 1 -- 172.21.15.16:0/1081841079 wait complete. 2024-08-28T19:38:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.640+0000 7f61da812740 1 -- start start 2024-08-28T19:38:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61da812740 1 Processor -- start 2024-08-28T19:38:36.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61da812740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca11720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61da812740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c61a9f0 0x56148ca11cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61da812740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 0x56148ca823c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61da812740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc1c1620 con 0x56148c61a9f0 2024-08-28T19:38:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61da812740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc1c0ef0 con 0x56148c63b390 2024-08-28T19:38:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61da812740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc1be0b0 con 0x56148c5efda0 2024-08-28T19:38:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61da812740 10 client.0 _ll_register_callbacks cb 0x7f61c8035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61d1838640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca11720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61d1838640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca11720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:32848/0 (socket says 172.21.15.16:32848) 2024-08-28T19:38:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61d1838640 1 -- 172.21.15.16:0/2804639584 learned_addr learned my addr 172.21.15.16:0/2804639584 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61d1838640 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 0x56148ca823c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.641+0000 7f61d2039640 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c61a9f0 0x56148ca11cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.642+0000 7f61d1838640 1 -- 172.21.15.16:0/2804639584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 msgr2=0x56148ca823c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:36.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.642+0000 7f61d1838640 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 0x56148ca823c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.642+0000 7f61d1838640 1 -- 172.21.15.16:0/2804639584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c61a9f0 msgr2=0x56148ca11cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:36.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.642+0000 7f61d1838640 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c61a9f0 0x56148ca11cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.642+0000 7f61d1838640 1 -- 172.21.15.16:0/2804639584 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56148ca82a30 con 0x56148c5efda0 2024-08-28T19:38:36.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.642+0000 7f61d1838640 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca11720 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f61c802a860 tx=0x7f61c8005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a80636fdc0071ce server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:36.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.642+0000 7f61b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:38:36.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.642+0000 7f61b5ffb640 1 -- 172.21.15.16:0/2804639584 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c8042670 con 0x56148c5efda0 2024-08-28T19:38:36.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.643+0000 7f61da812740 1 -- 172.21.15.16:0/2804639584 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61c8030660 con 0x56148c5efda0 2024-08-28T19:38:36.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.643+0000 7f61b5ffb640 1 -- 172.21.15.16:0/2804639584 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f61c8061030 con 0x56148c5efda0 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.643+0000 7f61b5ffb640 1 -- 172.21.15.16:0/2804639584 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f61c8042d90 con 0x56148c5efda0 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.643+0000 7f61da812740 10 client.28648 fetch_fsmap learned FSMap version 95 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.643+0000 7f61da812740 1 -- 172.21.15.16:0/2804639584 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) -- 0x56148ca89d60 con 0x56148c5efda0 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.644+0000 7f61b5ffb640 1 -- 172.21.15.16:0/2804639584 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61c8030660 con 0x56148c5efda0 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.644+0000 7f61b5ffb640 1 -- 172.21.15.16:0/2804639584 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 95) ==== 8402+0+0 (secure 0 0 0) 0x7f61c8065440 con 0x56148c5efda0 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.644+0000 7f61da812740 10 client.28648 fetch_fsmap finished waiting for FSMap version 95 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.644+0000 7f61da812740 10 client.28648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.644+0000 7f61da812740 20 client.28648 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.644+0000 7f61da812740 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56148c65c070 0x56148ca8c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.644+0000 7f61da812740 4 client.28648 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.644+0000 7f61da812740 1 -- 172.21.15.16:0/2804639584 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56148c97a580 con 0x56148c65c070 2024-08-28T19:38:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.645+0000 7f61d2039640 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56148c65c070 0x56148ca8c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:36.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.645+0000 7f61b5ffb640 10 client.28648 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:36.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.645+0000 7f61d2039640 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56148c65c070 0x56148ca8c560 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac5ee0dde081582c server_cookie=ba135d45b256944b in_seq=0 out_seq=0 2024-08-28T19:38:36.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.649+0000 7f61b5ffb640 1 -- 172.21.15.16:0/2804639584 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+6622 (crc 0 0 0) 0x56148c97a580 con 0x56148c65c070 2024-08-28T19:38:36.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.649+0000 7f61b5ffb640 10 client.28648 handle_command_reply: tid=0 2024-08-28T19:38:36.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.649+0000 7f61da812740 1 client.28648 shutdown 2024-08-28T19:38:36.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.649+0000 7f61da812740 1 -- 172.21.15.16:0/2804639584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 msgr2=0x56148ca11720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:36.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.649+0000 7f61da812740 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca11720 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f61c802a860 tx=0x7f61c8005e40 comp rx=0 tx=0).stop 2024-08-28T19:38:36.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.649+0000 7f61da812740 1 -- 172.21.15.16:0/2804639584 shutdown_connections 2024-08-28T19:38:36.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.649+0000 7f61da812740 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148c63b390 0x56148ca823c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.649+0000 7f61da812740 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56148c61a9f0 0x56148ca11cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.649+0000 7f61da812740 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56148c65c070 0x56148ca8c560 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.650+0000 7f61da812740 1 --2- 172.21.15.16:0/2804639584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56148c5efda0 0x56148ca11720 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.650+0000 7f61b5ffb640 0 client.28648 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:36.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.650+0000 7f61da812740 1 -- 172.21.15.16:0/2804639584 >> 172.21.15.16:0/2804639584 conn(0x56148c650460 msgr2=0x56148ca0adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:36.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.650+0000 7f61da812740 1 -- 172.21.15.16:0/2804639584 shutdown_connections 2024-08-28T19:38:36.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.650+0000 7f61da812740 1 -- 172.21.15.16:0/2804639584 wait complete. 2024-08-28T19:38:36.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.650+0000 7f61da812740 20 client.28648 trim_cache size 0 max 16384 2024-08-28T19:38:36.653 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:38:36.653 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:38:36.653 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:38:36.653 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:38:36.653 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:38:36.653 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:38:36.653 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.653 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.653 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.654 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.655 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:38:36.656 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:38:36.662 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:38:36.665 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:36.666 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:38:36.667 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:38:36.667 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:36.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:36.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:36.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:36.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:36.667 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:36.667 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:36.667 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:38:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.652+0000 7f61d429d640 1 -- 172.21.15.16:0/2170927582 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61b80b4df0 msgr2=0x7f61b80b7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.652+0000 7f61d429d640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61b80b4df0 0x7f61b80b7300 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f61c40ab9e0 tx=0x7f61c400cf70 comp rx=0 tx=0).stop 2024-08-28T19:38:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.652+0000 7f61d429d640 1 -- 172.21.15.16:0/2170927582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61cc1ae960 msgr2=0x7f61cc1aed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.652+0000 7f61d429d640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61cc1ae960 0x7f61cc1aed70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f61c400ef00 tx=0x7f61c400c740 comp rx=0 tx=0).stop 2024-08-28T19:38:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.653+0000 7f61d429d640 1 -- 172.21.15.16:0/2170927582 shutdown_connections 2024-08-28T19:38:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.653+0000 7f61d429d640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f61cc1ae960 0x7f61cc1aed70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.653+0000 7f61d429d640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f61b80b4df0 0x7f61b80b7300 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.653+0000 7f61d429d640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61cc11c9d0 0x7f61cc1a9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.653+0000 7f61d429d640 1 --2- 172.21.15.16:0/2170927582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc111ce0 0x7f61cc1a91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:36.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.653+0000 7f61d429d640 1 -- 172.21.15.16:0/2170927582 >> 172.21.15.16:0/2170927582 conn(0x7f61cc070c60 msgr2=0x7f61cc071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:36.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.653+0000 7f61d429d640 1 -- 172.21.15.16:0/2170927582 shutdown_connections 2024-08-28T19:38:36.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:36.653+0000 7f61d429d640 1 -- 172.21.15.16:0/2170927582 wait complete. 2024-08-28T19:38:37.475 INFO:tasks.workunit.client.0.smithi016.stdout: 10 96721 23.64 MB/sec execute 133 sec latency 88.090 ms 2024-08-28T19:38:37.554 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:38:37.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:38:37.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:37 smithi016 ceph-mon[18556]: pgmap v888: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 17 MiB/s wr, 837 op/s 2024-08-28T19:38:37.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:37 smithi028 ceph-mon[21769]: pgmap v888: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 17 MiB/s wr, 837 op/s 2024-08-28T19:38:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:37 smithi115 ceph-mon[22184]: pgmap v888: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 17 MiB/s wr, 837 op/s 2024-08-28T19:38:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.454+0000 7f03fa64d640 1 -- 172.21.15.16:0/1261985117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03f4104600 msgr2=0x7f03f4104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.454+0000 7f03fa64d640 1 --2- 172.21.15.16:0/1261985117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03f4104600 0x7f03f4104a10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f03e4009900 tx=0x7f03e402a410 comp rx=0 tx=0).stop 2024-08-28T19:38:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.454+0000 7f03fa64d640 1 -- 172.21.15.16:0/1261985117 shutdown_connections 2024-08-28T19:38:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.454+0000 7f03fa64d640 1 --2- 172.21.15.16:0/1261985117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03f410dbb0 0x7f03f4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.454+0000 7f03fa64d640 1 --2- 172.21.15.16:0/1261985117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03f4105030 0x7f03f410d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.454+0000 7f03fa64d640 1 --2- 172.21.15.16:0/1261985117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03f4104600 0x7f03f4104a10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:38.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.454+0000 7f03fa64d640 1 -- 172.21.15.16:0/1261985117 >> 172.21.15.16:0/1261985117 conn(0x7f03f40794d0 msgr2=0x7f03f40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.454+0000 7f03fa64d640 1 -- 172.21.15.16:0/1261985117 shutdown_connections 2024-08-28T19:38:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.454+0000 7f03fa64d640 1 -- 172.21.15.16:0/1261985117 wait complete. 2024-08-28T19:38:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.455+0000 7f03fa64d640 1 Processor -- start 2024-08-28T19:38:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.455+0000 7f03fa64d640 1 -- start start 2024-08-28T19:38:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.455+0000 7f03fa64d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03f4104600 0x7f03f4198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03fa64d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03f4105030 0x7f03f41992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03fa64d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03f410dbb0 0x7f03f419fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03fa64d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03f41138d0 con 0x7f03f410dbb0 2024-08-28T19:38:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03fa64d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4113750 con 0x7f03f4105030 2024-08-28T19:38:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03fa64d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4113a50 con 0x7f03f4104600 2024-08-28T19:38:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03ebfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03f4105030 0x7f03f41992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03ebfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03f4105030 0x7f03f41992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:40068/0 (socket says 172.21.15.16:40068) 2024-08-28T19:38:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03ebfff640 1 -- 172.21.15.16:0/1806165693 learned_addr learned my addr 172.21.15.16:0/1806165693 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03f8bea640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03f410dbb0 0x7f03f419fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03ebfff640 1 -- 172.21.15.16:0/1806165693 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03f4104600 msgr2=0x7f03f4198da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03ebfff640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03f4104600 0x7f03f4198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03ebfff640 1 -- 172.21.15.16:0/1806165693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03f410dbb0 msgr2=0x7f03f419fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03ebfff640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03f410dbb0 0x7f03f419fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.456+0000 7f03ebfff640 1 -- 172.21.15.16:0/1806165693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f4199b80 con 0x7f03f4105030 2024-08-28T19:38:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.457+0000 7f03ebfff640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03f4105030 0x7f03f41992e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00e9f0 tx=0x7f03dc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.457+0000 7f03f1ffb640 1 -- 172.21.15.16:0/1806165693 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc00ccc0 con 0x7f03f4105030 2024-08-28T19:38:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.457+0000 7f03fa64d640 1 -- 172.21.15.16:0/1806165693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f41a0400 con 0x7f03f4105030 2024-08-28T19:38:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.458+0000 7f03fa64d640 1 -- 172.21.15.16:0/1806165693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f41a0940 con 0x7f03f4105030 2024-08-28T19:38:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.457+0000 7f03f1ffb640 1 -- 172.21.15.16:0/1806165693 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03dc0108e0 con 0x7f03f4105030 2024-08-28T19:38:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.458+0000 7f03f1ffb640 1 -- 172.21.15.16:0/1806165693 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc018860 con 0x7f03f4105030 2024-08-28T19:38:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.459+0000 7f03fa64d640 1 -- 172.21.15.16:0/1806165693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d40052c0 con 0x7f03f4105030 2024-08-28T19:38:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.460+0000 7f03f1ffb640 1 -- 172.21.15.16:0/1806165693 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f03dc018a00 con 0x7f03f4105030 2024-08-28T19:38:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.461+0000 7f03f1ffb640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03e00b4df0 0x7f03e00b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.461+0000 7f03f3fff640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03e00b4df0 0x7f03e00b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.461+0000 7f03f1ffb640 1 -- 172.21.15.16:0/1806165693 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f03dc0e59e0 con 0x7f03f4105030 2024-08-28T19:38:38.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.462+0000 7f03f3fff640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03e00b4df0 0x7f03e00b7300 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f03e4009230 tx=0x7f03e4031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.470+0000 7f03f1ffb640 1 -- 172.21.15.16:0/1806165693 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f03dc0ac400 con 0x7f03f4105030 2024-08-28T19:38:38.475 INFO:tasks.workunit.client.0.smithi016.stdout: 10 97407 23.82 MB/sec execute 134 sec latency 152.603 ms 2024-08-28T19:38:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.849+0000 7f03fa64d640 1 -- 172.21.15.16:0/1806165693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03d4005880 con 0x7f03f4105030 2024-08-28T19:38:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.851+0000 7f03f1ffb640 1 -- 172.21.15.16:0/1806165693 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+11994 (secure 0 0 0) 0x7f03dc0abd00 con 0x7f03f4105030 2024-08-28T19:38:38.853 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:38:38.853 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":96,"btime":"2024-08-28T19:38:38:408543+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":96,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:38:38.323875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:38:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 96 2024-08-28T19:38:38.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 -- 172.21.15.16:0/1806165693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03e00b4df0 msgr2=0x7f03e00b7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:38.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03e00b4df0 0x7f03e00b7300 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f03e4009230 tx=0x7f03e4031400 comp rx=0 tx=0).stop 2024-08-28T19:38:38.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 -- 172.21.15.16:0/1806165693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03f4105030 msgr2=0x7f03f41992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:38.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03f4105030 0x7f03f41992e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00e9f0 tx=0x7f03dc00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:38:38.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 -- 172.21.15.16:0/1806165693 shutdown_connections 2024-08-28T19:38:38.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03f410dbb0 0x7f03f419fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:38.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f03e00b4df0 0x7f03e00b7300 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:38.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03f4105030 0x7f03f41992e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:38.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 --2- 172.21.15.16:0/1806165693 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03f4104600 0x7f03f4198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:38.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 -- 172.21.15.16:0/1806165693 >> 172.21.15.16:0/1806165693 conn(0x7f03f40794d0 msgr2=0x7f03f4100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:38.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 -- 172.21.15.16:0/1806165693 shutdown_connections 2024-08-28T19:38:38.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:38.855+0000 7f03fa64d640 1 -- 172.21.15.16:0/1806165693 wait complete. 2024-08-28T19:38:39.475 INFO:tasks.workunit.client.0.smithi016.stdout: 10 97872 23.79 MB/sec execute 135 sec latency 214.628 ms 2024-08-28T19:38:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:39 smithi016 ceph-mon[18556]: pgmap v889: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 18 MiB/s wr, 851 op/s 2024-08-28T19:38:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:39 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:39.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:39 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1806165693' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:38:39.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:38:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:39 smithi028 ceph-mon[21769]: pgmap v889: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 18 MiB/s wr, 851 op/s 2024-08-28T19:38:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:39 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:39 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1806165693' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:38:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:39 smithi115 ceph-mon[22184]: pgmap v889: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 18 MiB/s wr, 851 op/s 2024-08-28T19:38:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:39 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:39 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1806165693' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:38:40.026 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:40.475 INFO:tasks.workunit.client.0.smithi016.stdout: 10 98579 23.92 MB/sec execute 136 sec latency 129.458 ms 2024-08-28T19:38:40.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.921+0000 7f58c702f640 1 -- 172.21.15.16:0/2410004974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c0108160 msgr2=0x7f58c0108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:40.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.921+0000 7f58c702f640 1 --2- 172.21.15.16:0/2410004974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c0108160 0x7f58c0108570 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009900 tx=0x7f58ac02a410 comp rx=0 tx=0).stop 2024-08-28T19:38:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.921+0000 7f58c702f640 1 -- 172.21.15.16:0/2410004974 shutdown_connections 2024-08-28T19:38:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.921+0000 7f58c702f640 1 --2- 172.21.15.16:0/2410004974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c0108ab0 0x7f58c010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.921+0000 7f58c702f640 1 --2- 172.21.15.16:0/2410004974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c0108160 0x7f58c0108570 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.921+0000 7f58c702f640 1 --2- 172.21.15.16:0/2410004974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c0102820 0x7f58c0102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.921+0000 7f58c702f640 1 -- 172.21.15.16:0/2410004974 >> 172.21.15.16:0/2410004974 conn(0x7f58c00794e0 msgr2=0x7f58c00798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.921+0000 7f58c702f640 1 -- 172.21.15.16:0/2410004974 shutdown_connections 2024-08-28T19:38:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.921+0000 7f58c702f640 1 -- 172.21.15.16:0/2410004974 wait complete. 2024-08-28T19:38:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.922+0000 7f58c702f640 1 Processor -- start 2024-08-28T19:38:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.922+0000 7f58c702f640 1 -- start start 2024-08-28T19:38:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.922+0000 7f58c702f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c0102820 0x7f58c019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.922+0000 7f58c702f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c0108160 0x7f58c019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.922+0000 7f58c702f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c0108ab0 0x7f58c019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.922+0000 7f58c702f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f58c0112700 con 0x7f58c0102820 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.922+0000 7f58c702f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58c0112580 con 0x7f58c0108160 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.922+0000 7f58c702f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f58c0112880 con 0x7f58c0108ab0 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58c4dcb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c0102820 0x7f58c019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c0108160 0x7f58c019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58c4dcb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c0102820 0x7f58c019a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51434/0 (socket says 172.21.15.16:51434) 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58c4dcb640 1 -- 172.21.15.16:0/3396600637 learned_addr learned my addr 172.21.15.16:0/3396600637 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c0108160 0x7f58c019a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:40074/0 (socket says 172.21.15.16:40074) 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58c55cc640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c0108ab0 0x7f58c019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58c4dcb640 1 -- 172.21.15.16:0/3396600637 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c0108ab0 msgr2=0x7f58c019fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c0108ab0 0x7f58c019fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58c4dcb640 1 -- 172.21.15.16:0/3396600637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c0108160 msgr2=0x7f58c019a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:40.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c0108160 0x7f58c019a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58c4dcb640 1 -- 172.21.15.16:0/3396600637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c01a0190 con 0x7f58c0102820 2024-08-28T19:38:40.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c0102820 0x7f58c019a220 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f58b400e9f0 tx=0x7f58b400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:40.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.923+0000 7f58bdffb640 1 -- 172.21.15.16:0/3396600637 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b400cd80 con 0x7f58c0102820 2024-08-28T19:38:40.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.924+0000 7f58c702f640 1 -- 172.21.15.16:0/3396600637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c01a0480 con 0x7f58c0102820 2024-08-28T19:38:40.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.924+0000 7f58bdffb640 1 -- 172.21.15.16:0/3396600637 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58b40054d0 con 0x7f58c0102820 2024-08-28T19:38:40.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.924+0000 7f58bdffb640 1 -- 172.21.15.16:0/3396600637 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b4010690 con 0x7f58c0102820 2024-08-28T19:38:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.924+0000 7f58c702f640 1 -- 172.21.15.16:0/3396600637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c0104d80 con 0x7f58c0102820 2024-08-28T19:38:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58bdffb640 1 -- 172.21.15.16:0/3396600637 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f58b4004e90 con 0x7f58c0102820 2024-08-28T19:38:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58cd5a4740 1 Processor -- start 2024-08-28T19:38:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58cd5a4740 1 -- start start 2024-08-28T19:38:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cce070 0x559811073310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x559811073850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 0x559811073d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58cd5a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559810fea680 con 0x559810cce070 2024-08-28T19:38:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58cd5a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559810fd9c10 con 0x559810cad390 2024-08-28T19:38:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58cd5a4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559810fddbc0 con 0x559810ca51b0 2024-08-28T19:38:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58c55cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cce070 0x559811073310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58c55cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cce070 0x559811073310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51448/0 (socket says 172.21.15.16:51448) 2024-08-28T19:38:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58c55cc640 1 -- 172.21.15.16:0/1926702141 learned_addr learned my addr 172.21.15.16:0/1926702141 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58bffff640 1 --2- 172.21.15.16:0/1926702141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 0x559811073d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.927+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/1926702141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x559811073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.928+0000 7f58bdffb640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58b00bd4b0 0x7f58b00bf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.928+0000 7f58c55cc640 1 -- 172.21.15.16:0/1926702141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 msgr2=0x559811073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.928+0000 7f58c55cc640 1 --2- 172.21.15.16:0/1926702141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x559811073850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.928+0000 7f58c55cc640 1 -- 172.21.15.16:0/1926702141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 msgr2=0x559811073d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.928+0000 7f58c55cc640 1 --2- 172.21.15.16:0/1926702141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 0x559811073d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.928+0000 7f58c55cc640 1 -- 172.21.15.16:0/1926702141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559810d39650 con 0x559810cce070 2024-08-28T19:38:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.928+0000 7f58bffff640 1 --2- 172.21.15.16:0/1926702141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 0x559811073d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:38:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.928+0000 7f58bdffb640 1 -- 172.21.15.16:0/3396600637 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f58b40e5d60 con 0x7f58c0102820 2024-08-28T19:38:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.928+0000 7f58c55cc640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58b00bd4b0 0x7f58b00bf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.928+0000 7f58c55cc640 1 --2- 172.21.15.16:0/1926702141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cce070 0x559811073310 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f58b8009fd0 tx=0x7f58b80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc548db0550e888 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.928+0000 7f58aaffd640 1 -- 172.21.15.16:0/1926702141 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8004250 con 0x559810cce070 2024-08-28T19:38:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.929+0000 7f58aaffd640 1 -- 172.21.15.16:0/1926702141 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58b8003db0 con 0x559810cce070 2024-08-28T19:38:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.929+0000 7f58aaffd640 1 -- 172.21.15.16:0/1926702141 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b801bb80 con 0x559810cce070 2024-08-28T19:38:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.930+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1926702141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cce070 msgr2=0x559811073310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.930+0000 7f58c55cc640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58b00bd4b0 0x7f58b00bf9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f58b8020000 tx=0x7f58b8021000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.930+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/1926702141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cce070 0x559811073310 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f58b8009fd0 tx=0x7f58b80118d0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.930+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1926702141 shutdown_connections 2024-08-28T19:38:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.930+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/1926702141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 0x559811073d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.930+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/1926702141 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x559811073850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.930+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/1926702141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cce070 0x559811073310 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.930+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1926702141 >> 172.21.15.16:0/1926702141 conn(0x559810cc2460 msgr2=0x559811071200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.930+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1926702141 shutdown_connections 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.930+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1926702141 wait complete. 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.931+0000 7f58cd5a4740 1 -- start start 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.931+0000 7f58cd5a4740 1 Processor -- start 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.931+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x55981107ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.931+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 0x55981107efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 0x55981108ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 0x55981107efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58cd5a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559810e3c700 con 0x559810cad390 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 0x55981107efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51460/0 (socket says 172.21.15.16:51460) 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58bffff640 1 -- 172.21.15.16:0/1626395397 learned_addr learned my addr 172.21.15.16:0/1626395397 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1626395397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559810f60750 con 0x559810cce070 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1626395397 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559811081af0 con 0x559810ca51b0 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58c55cc640 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 0x55981108ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x55981107ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58cd5a4740 10 client.0 _ll_register_callbacks cb 0x559810f8fe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58bffff640 1 -- 172.21.15.16:0/1626395397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 msgr2=0x55981107ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58bffff640 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x55981107ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58bffff640 1 -- 172.21.15.16:0/1626395397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 msgr2=0x55981108ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58bffff640 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 0x55981108ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58bffff640 1 -- 172.21.15.16:0/1626395397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559810f99900 con 0x559810cad390 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x55981107ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1626395397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58b8003db0 con 0x559810cad390 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.932+0000 7f58bffff640 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 0x55981107efc0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009900 tx=0x7f58ac0316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3c3b519bc59153 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.933+0000 7f589ffff640 10 client.19620 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.933+0000 7f589ffff640 1 -- 172.21.15.16:0/1626395397 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac035040 con 0x559810cad390 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.933+0000 7f589ffff640 1 -- 172.21.15.16:0/1626395397 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f58ac004040 con 0x559810cad390 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.933+0000 7f589ffff640 1 -- 172.21.15.16:0/1626395397 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f58ac004280 con 0x559810cad390 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.933+0000 7f58cd5a4740 10 client.19620 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.933+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1626395397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=96}) -- 0x55981108d7c0 con 0x559810cad390 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.934+0000 7f589ffff640 1 -- 172.21.15.16:0/1626395397 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7f58ac005030 con 0x559810cad390 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.934+0000 7f58cd5a4740 10 client.19620 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.934+0000 7f58cd5a4740 10 client.19620 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.934+0000 7f58cd5a4740 20 client.19620 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.934+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x559810c8c9f0 0x559811090c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.934+0000 7f58cd5a4740 4 client.19620 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:38:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.934+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1626395397 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559810fec5d0 con 0x559810c8c9f0 2024-08-28T19:38:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.935+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x559810c8c9f0 0x559811090c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.937+0000 7f589ffff640 10 client.19620 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:38:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.937+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x559810c8c9f0 0x559811090c30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=155337f5bfb166a3 server_cookie=597167d062e9161d in_seq=0 out_seq=0 2024-08-28T19:38:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.941+0000 7f589ffff640 1 -- 172.21.15.16:0/1626395397 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559810fec5d0 con 0x559810c8c9f0 2024-08-28T19:38:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.941+0000 7f589ffff640 10 client.19620 handle_command_reply: tid=0 2024-08-28T19:38:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.941+0000 7f58cd5a4740 1 client.19620 shutdown 2024-08-28T19:38:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.941+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1626395397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 msgr2=0x55981107efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.941+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 0x55981107efc0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009900 tx=0x7f58ac0316d0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.942+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1626395397 shutdown_connections 2024-08-28T19:38:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.942+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x559810c8c9f0 0x559811090c30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.942+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 0x55981108ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.942+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 0x55981107efc0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.942+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/1626395397 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x55981107ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.942+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1626395397 >> 172.21.15.16:0/1626395397 conn(0x559810cc2460 msgr2=0x5598110764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.942+0000 7f589ffff640 0 client.19620 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:38:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.942+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1626395397 shutdown_connections 2024-08-28T19:38:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.942+0000 7f58cd5a4740 1 -- 172.21.15.16:0/1626395397 wait complete. 2024-08-28T19:38:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.942+0000 7f58cd5a4740 20 client.19620 trim_cache size 0 max 16384 2024-08-28T19:38:40.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58cd5a4740 1 Processor -- start 2024-08-28T19:38:40.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58cd5a4740 1 -- start start 2024-08-28T19:38:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810ca51b0 0x559811074d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 0x5598110806b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810cce070 0x55981108ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58cd5a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01aeb10 con 0x559810ca51b0 2024-08-28T19:38:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58cd5a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01b0830 con 0x559810cad390 2024-08-28T19:38:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58cd5a4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01af2a0 con 0x559810cce070 2024-08-28T19:38:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58c4dcb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810ca51b0 0x559811074d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 0x5598110806b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 0x5598110806b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:40100/0 (socket says 172.21.15.16:40100) 2024-08-28T19:38:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.964+0000 7f58bffff640 1 -- 172.21.15.16:0/3077498782 learned_addr learned my addr 172.21.15.16:0/3077498782 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.965+0000 7f58c55cc640 1 --2- 172.21.15.16:0/3077498782 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810cce070 0x55981108ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.965+0000 7f58bffff640 1 -- 172.21.15.16:0/3077498782 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810cce070 msgr2=0x55981108ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.965+0000 7f58bffff640 1 --2- 172.21.15.16:0/3077498782 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810cce070 0x55981108ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.965+0000 7f58bffff640 1 -- 172.21.15.16:0/3077498782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810ca51b0 msgr2=0x559811074d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.965+0000 7f58bffff640 1 --2- 172.21.15.16:0/3077498782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810ca51b0 0x559811074d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.965+0000 7f58bffff640 1 -- 172.21.15.16:0/3077498782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598110811c0 con 0x559810cad390 2024-08-28T19:38:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.965+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/3077498782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810ca51b0 0x559811074d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:38:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.965+0000 7f58bffff640 1 --2- 172.21.15.16:0/3077498782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 0x5598110806b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f58ac032de0 tx=0x7f58ac03e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c53eda66adf0eff1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.966+0000 7f58aa7fc640 1 -- 172.21.15.16:0/3077498782 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac035070 con 0x559810cad390 2024-08-28T19:38:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.966+0000 7f58aa7fc640 1 -- 172.21.15.16:0/3077498782 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58b8003db0 con 0x559810cad390 2024-08-28T19:38:40.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.967+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3077498782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 msgr2=0x5598110806b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:40.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.967+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/3077498782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 0x5598110806b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f58ac032de0 tx=0x7f58ac03e000 comp rx=0 tx=0).stop 2024-08-28T19:38:40.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.967+0000 7f58aa7fc640 1 -- 172.21.15.16:0/3077498782 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac0324f0 con 0x559810cad390 2024-08-28T19:38:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.969+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3077498782 shutdown_connections 2024-08-28T19:38:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.969+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/3077498782 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810cce070 0x55981108ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.969+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/3077498782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cad390 0x5598110806b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.969+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/3077498782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810ca51b0 0x559811074d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.969+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3077498782 >> 172.21.15.16:0/3077498782 conn(0x559810cc2460 msgr2=0x55981107ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.972+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3077498782 shutdown_connections 2024-08-28T19:38:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.972+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3077498782 wait complete. 2024-08-28T19:38:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.973+0000 7f58cd5a4740 1 -- start start 2024-08-28T19:38:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.973+0000 7f58cd5a4740 1 Processor -- start 2024-08-28T19:38:40.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.974+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x559811085190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.974+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 0x559811085750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.974+0000 7f58cd5a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 0x5598110fa640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.974+0000 7f58cd5a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01b00a0 con 0x559810cad390 2024-08-28T19:38:40.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.974+0000 7f58cd5a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01af970 con 0x559810cce070 2024-08-28T19:38:40.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.974+0000 7f58cd5a4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01acb10 con 0x559810ca51b0 2024-08-28T19:38:40.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.974+0000 7f58cd5a4740 10 client.0 _ll_register_callbacks cb 0x7f58b4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:40.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.975+0000 7f58c55cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 0x5598110fa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.975+0000 7f58c55cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 0x5598110fa640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:40116/0 (socket says 172.21.15.16:40116) 2024-08-28T19:38:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.975+0000 7f58c55cc640 1 -- 172.21.15.16:0/3137723485 learned_addr learned my addr 172.21.15.16:0/3137723485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.975+0000 7f58bffff640 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 0x559811085750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.975+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x559811085190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.975+0000 7f58c55cc640 1 -- 172.21.15.16:0/3137723485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 msgr2=0x559811085190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.975+0000 7f58c55cc640 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x559811085190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.975+0000 7f58c55cc640 1 -- 172.21.15.16:0/3137723485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 msgr2=0x559811085750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.975+0000 7f58c55cc640 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 0x559811085750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.975+0000 7f58c55cc640 1 -- 172.21.15.16:0/3137723485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598110fab80 con 0x559810cce070 2024-08-28T19:38:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.976+0000 7f58c55cc640 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 0x5598110fa640 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f58b80039a0 tx=0x7f58b80039d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=722905e1fde016ac server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.976+0000 7f589ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:38:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.976+0000 7f589ffff640 1 -- 172.21.15.16:0/3137723485 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8029040 con 0x559810cce070 2024-08-28T19:38:40.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.976+0000 7f589ffff640 1 -- 172.21.15.16:0/3137723485 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f58b800f980 con 0x559810cce070 2024-08-28T19:38:40.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.976+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3137723485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58b8003db0 con 0x559810cce070 2024-08-28T19:38:40.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.977+0000 7f589ffff640 1 -- 172.21.15.16:0/3137723485 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f58b8043050 con 0x559810cce070 2024-08-28T19:38:40.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.977+0000 7f58cd5a4740 10 client.29078 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:40.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.977+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3137723485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x5598110fbd10 con 0x559810cce070 2024-08-28T19:38:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.977+0000 7f589ffff640 1 -- 172.21.15.16:0/3137723485 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58b8003db0 con 0x559810cce070 2024-08-28T19:38:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.977+0000 7f589ffff640 1 -- 172.21.15.16:0/3137723485 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7f58b8030430 con 0x559810cce070 2024-08-28T19:38:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.977+0000 7f58cd5a4740 10 client.29078 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.977+0000 7f58cd5a4740 10 client.29078 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:38:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.977+0000 7f58cd5a4740 20 client.29078 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.977+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x559810c61da0 0x5598110fe3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.977+0000 7f58cd5a4740 4 client.29078 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:38:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.977+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3137723485 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559810fe8070 con 0x559810c61da0 2024-08-28T19:38:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.978+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x559810c61da0 0x5598110fe3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:40.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.983+0000 7f58c4dcb640 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x559810c61da0 0x5598110fe3c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=511dfa28c9aea4a8 server_cookie=adf1563246e14010 in_seq=0 out_seq=0 2024-08-28T19:38:40.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.983+0000 7f589ffff640 10 client.29078 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:38:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.983+0000 7f589ffff640 1 -- 172.21.15.16:0/3137723485 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559810fe8070 con 0x559810c61da0 2024-08-28T19:38:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.983+0000 7f589ffff640 10 client.29078 handle_command_reply: tid=0 2024-08-28T19:38:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.983+0000 7f58cd5a4740 1 client.29078 shutdown 2024-08-28T19:38:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.984+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3137723485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 msgr2=0x5598110fa640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.984+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 0x5598110fa640 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f58b80039a0 tx=0x7f58b80039d0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.984+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3137723485 shutdown_connections 2024-08-28T19:38:40.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.984+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x559810c61da0 0x5598110fe3c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.984+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559810cce070 0x5598110fa640 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.984+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559810cad390 0x559811085750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.984+0000 7f589ffff640 0 client.29078 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:38:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.984+0000 7f58cd5a4740 1 --2- 172.21.15.16:0/3137723485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559810ca51b0 0x559811085190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.985+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3137723485 >> 172.21.15.16:0/3137723485 conn(0x559810cc2460 msgr2=0x559811074ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.985+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3137723485 shutdown_connections 2024-08-28T19:38:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.985+0000 7f58cd5a4740 1 -- 172.21.15.16:0/3137723485 wait complete. 2024-08-28T19:38:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.986+0000 7f58cd5a4740 20 client.29078 trim_cache size 0 max 16384 2024-08-28T19:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.986+0000 7f58c702f640 1 -- 172.21.15.16:0/3396600637 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58b00bd4b0 msgr2=0x7f58b00bf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.987+0000 7f58c702f640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58b00bd4b0 0x7f58b00bf9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f58b8020000 tx=0x7f58b8021000 comp rx=0 tx=0).stop 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.987+0000 7f58c702f640 1 -- 172.21.15.16:0/3396600637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c0102820 msgr2=0x7f58c019a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.987+0000 7f58c702f640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c0102820 0x7f58c019a220 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f58b400e9f0 tx=0x7f58b400edf0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.987+0000 7f58c702f640 1 -- 172.21.15.16:0/3396600637 shutdown_connections 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.987+0000 7f58c702f640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c0108ab0 0x7f58c019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.987+0000 7f58c702f640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f58b00bd4b0 0x7f58b00bf9c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.987+0000 7f58c702f640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c0108160 0x7f58c019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.987+0000 7f58c702f640 1 --2- 172.21.15.16:0/3396600637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c0102820 0x7f58c019a220 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.987+0000 7f58c702f640 1 -- 172.21.15.16:0/3396600637 >> 172.21.15.16:0/3396600637 conn(0x7f58c00794e0 msgr2=0x7f58c006b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.988+0000 7f58c702f640 1 -- 172.21.15.16:0/3396600637 shutdown_connections 2024-08-28T19:38:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:40.988+0000 7f58c702f640 1 -- 172.21.15.16:0/3396600637 wait complete. 2024-08-28T19:38:41.475 INFO:tasks.workunit.client.0.smithi016.stdout: 10 99211 23.94 MB/sec execute 137 sec latency 131.027 ms 2024-08-28T19:38:41.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:41 smithi016 ceph-mon[18556]: pgmap v890: 129 pgs: 129 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 17 MiB/s wr, 868 op/s 2024-08-28T19:38:41.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:41 smithi028 ceph-mon[21769]: pgmap v890: 129 pgs: 129 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 17 MiB/s wr, 868 op/s 2024-08-28T19:38:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:41 smithi115 ceph-mon[22184]: pgmap v890: 129 pgs: 129 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 17 MiB/s wr, 868 op/s 2024-08-28T19:38:42.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:38:42.421 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:42.476 INFO:tasks.workunit.client.0.smithi016.stdout: 10 100014 24.11 MB/sec execute 138 sec latency 129.946 ms 2024-08-28T19:38:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.985+0000 7efd3370c640 1 -- 172.21.15.16:0/4142200801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd2c108a90 msgr2=0x7efd2c10f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.986+0000 7efd3370c640 1 --2- 172.21.15.16:0/4142200801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd2c108a90 0x7efd2c10f2c0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7efd24009ca0 tx=0x7efd2402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:38:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.986+0000 7efd3370c640 1 -- 172.21.15.16:0/4142200801 shutdown_connections 2024-08-28T19:38:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.986+0000 7efd3370c640 1 --2- 172.21.15.16:0/4142200801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd2c108a90 0x7efd2c10f2c0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.986+0000 7efd3370c640 1 --2- 172.21.15.16:0/4142200801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd2c108140 0x7efd2c108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.986+0000 7efd3370c640 1 --2- 172.21.15.16:0/4142200801 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd2c102800 0x7efd2c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.986+0000 7efd3370c640 1 -- 172.21.15.16:0/4142200801 >> 172.21.15.16:0/4142200801 conn(0x7efd2c0794d0 msgr2=0x7efd2c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.986+0000 7efd3370c640 1 -- 172.21.15.16:0/4142200801 shutdown_connections 2024-08-28T19:38:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.986+0000 7efd3370c640 1 -- 172.21.15.16:0/4142200801 wait complete. 2024-08-28T19:38:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.987+0000 7efd3370c640 1 Processor -- start 2024-08-28T19:38:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.987+0000 7efd3370c640 1 -- start start 2024-08-28T19:38:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.988+0000 7efd3370c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd2c102800 0x7efd2c199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.988+0000 7efd3370c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd2c108140 0x7efd2c1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.988+0000 7efd3370c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd2c108a90 0x7efd2c19e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.988+0000 7efd3370c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c1127c0 con 0x7efd2c102800 2024-08-28T19:38:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.988+0000 7efd3370c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c112640 con 0x7efd2c108140 2024-08-28T19:38:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.988+0000 7efd3370c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c112940 con 0x7efd2c108a90 2024-08-28T19:38:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.988+0000 7efd314a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd2c102800 0x7efd2c199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.988+0000 7efd314a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd2c102800 0x7efd2c199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51484/0 (socket says 172.21.15.16:51484) 2024-08-28T19:38:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.988+0000 7efd314a8640 1 -- 172.21.15.16:0/3846759831 learned_addr learned my addr 172.21.15.16:0/3846759831 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.989+0000 7efd314a8640 1 -- 172.21.15.16:0/3846759831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd2c108a90 msgr2=0x7efd2c19e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.989+0000 7efd314a8640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd2c108a90 0x7efd2c19e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.989+0000 7efd314a8640 1 -- 172.21.15.16:0/3846759831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd2c108140 msgr2=0x7efd2c1996b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-28T19:38:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.989+0000 7efd314a8640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd2c108140 0x7efd2c1996b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.989+0000 7efd314a8640 1 -- 172.21.15.16:0/3846759831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2c19f080 con 0x7efd2c102800 2024-08-28T19:38:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.990+0000 7efd314a8640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd2c102800 0x7efd2c199110 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7efd2800b4a0 tx=0x7efd2800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.990+0000 7efd227fc640 1 -- 172.21.15.16:0/3846759831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28014070 con 0x7efd2c102800 2024-08-28T19:38:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.990+0000 7efd227fc640 1 -- 172.21.15.16:0/3846759831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd28010ac0 con 0x7efd2c102800 2024-08-28T19:38:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.990+0000 7efd3370c640 1 -- 172.21.15.16:0/3846759831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2c19f370 con 0x7efd2c102800 2024-08-28T19:38:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.990+0000 7efd3370c640 1 -- 172.21.15.16:0/3846759831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd2c1a5ad0 con 0x7efd2c102800 2024-08-28T19:38:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.990+0000 7efd227fc640 1 -- 172.21.15.16:0/3846759831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28017c00 con 0x7efd2c102800 2024-08-28T19:38:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.994+0000 7efd39c81740 1 Processor -- start 2024-08-28T19:38:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.996+0000 7efd227fc640 1 -- 172.21.15.16:0/3846759831 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7efd28017070 con 0x7efd2c102800 2024-08-28T19:38:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.996+0000 7efd39c81740 1 -- start start 2024-08-28T19:38:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.996+0000 7efd39c81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bf8070 0x55a5a4f9ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.996+0000 7efd39c81740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4f9d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.996+0000 7efd39c81740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.996+0000 7efd39c81740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5a4f14680 con 0x55a5a4bcf1b0 2024-08-28T19:38:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.996+0000 7efd39c81740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5a4f03c10 con 0x55a5a4bf8070 2024-08-28T19:38:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.996+0000 7efd39c81740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a5a4f07bc0 con 0x55a5a4bd7390 2024-08-28T19:38:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.997+0000 7efd314a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4f9d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.997+0000 7efd314a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4f9d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51496/0 (socket says 172.21.15.16:51496) 2024-08-28T19:38:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.997+0000 7efd314a8640 1 -- 172.21.15.16:0/3928909844 learned_addr learned my addr 172.21.15.16:0/3928909844 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.998+0000 7efd31ca9640 1 --2- 172.21.15.16:0/3928909844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bf8070 0x55a5a4f9ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.998+0000 7efd30ca7640 1 --2- 172.21.15.16:0/3928909844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.998+0000 7efd227fc640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd180b4da0 0x7efd180b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.999+0000 7efd314a8640 1 -- 172.21.15.16:0/3928909844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bd7390 msgr2=0x55a5a4fb38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.999+0000 7efd314a8640 1 --2- 172.21.15.16:0/3928909844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.999+0000 7efd314a8640 1 -- 172.21.15.16:0/3928909844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bf8070 msgr2=0x55a5a4f9ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.999+0000 7efd314a8640 1 --2- 172.21.15.16:0/3928909844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bf8070 0x55a5a4f9ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.999+0000 7efd314a8640 1 -- 172.21.15.16:0/3928909844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a4c63650 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.999+0000 7efd227fc640 1 -- 172.21.15.16:0/3846759831 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7efd280e4cd0 con 0x7efd2c102800 2024-08-28T19:38:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:42.999+0000 7efd314a8640 1 --2- 172.21.15.16:0/3928909844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4f9d380 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7efd280e1f60 tx=0x7efd280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c5ffa7db10f4c1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.000+0000 7efd31ca9640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd180b4da0 0x7efd180b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.000+0000 7efd177fe640 1 -- 172.21.15.16:0/3928909844 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2801e070 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.000+0000 7efd31ca9640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd180b4da0 0x7efd180b72b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7efd2402f540 tx=0x7efd24005fd0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.000+0000 7efd177fe640 1 -- 172.21.15.16:0/3928909844 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd280aaaa0 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.000+0000 7efd177fe640 1 -- 172.21.15.16:0/3928909844 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28010c70 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.001+0000 7efd39c81740 1 -- 172.21.15.16:0/3928909844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 msgr2=0x55a5a4f9d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.001+0000 7efd39c81740 1 --2- 172.21.15.16:0/3928909844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4f9d380 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7efd280e1f60 tx=0x7efd280046c0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.002+0000 7efd39c81740 1 -- 172.21.15.16:0/3928909844 shutdown_connections 2024-08-28T19:38:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.002+0000 7efd39c81740 1 --2- 172.21.15.16:0/3928909844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.002+0000 7efd39c81740 1 --2- 172.21.15.16:0/3928909844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4f9d380 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.002+0000 7efd39c81740 1 --2- 172.21.15.16:0/3928909844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bf8070 0x55a5a4f9ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.002+0000 7efd39c81740 1 -- 172.21.15.16:0/3928909844 >> 172.21.15.16:0/3928909844 conn(0x55a5a4bec460 msgr2=0x55a5a4f9ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.002+0000 7efd39c81740 1 -- 172.21.15.16:0/3928909844 shutdown_connections 2024-08-28T19:38:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.002+0000 7efd39c81740 1 -- 172.21.15.16:0/3928909844 wait complete. 2024-08-28T19:38:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.003+0000 7efd39c81740 1 -- start start 2024-08-28T19:38:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.003+0000 7efd39c81740 1 Processor -- start 2024-08-28T19:38:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.003+0000 7efd39c81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4fb1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.003+0000 7efd39c81740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.004+0000 7efd39c81740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bf8070 0x55a5a4fbd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.004+0000 7efd39c81740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5a4d66700 con 0x55a5a4bd7390 2024-08-28T19:38:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.004+0000 7efd39c81740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5a4e8a750 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.004+0000 7efd39c81740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a5a4fb42a0 con 0x55a5a4bf8070 2024-08-28T19:38:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.004+0000 7efd39c81740 10 client.0 _ll_register_callbacks cb 0x55a5a4eb9e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.004+0000 7efd30ca7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.004+0000 7efd30ca7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51502/0 (socket says 172.21.15.16:51502) 2024-08-28T19:38:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.004+0000 7efd30ca7640 1 -- 172.21.15.16:0/2757458705 learned_addr learned my addr 172.21.15.16:0/2757458705 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.004+0000 7efd31ca9640 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bf8070 0x55a5a4fbd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.005+0000 7efd314a8640 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4fb1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.005+0000 7efd30ca7640 1 -- 172.21.15.16:0/2757458705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bf8070 msgr2=0x55a5a4fbd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.005+0000 7efd30ca7640 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bf8070 0x55a5a4fbd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.005+0000 7efd30ca7640 1 -- 172.21.15.16:0/2757458705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bcf1b0 msgr2=0x55a5a4fb1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.005+0000 7efd30ca7640 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4fb1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.005+0000 7efd30ca7640 1 -- 172.21.15.16:0/2757458705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a4ec3900 con 0x55a5a4bd7390 2024-08-28T19:38:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.005+0000 7efd30ca7640 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb17b0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009fd0 tx=0x7efd1c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2664d60432aa93d6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.005+0000 7efd14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:38:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.006+0000 7efd14ff9640 1 -- 172.21.15.16:0/2757458705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c005220 con 0x55a5a4bd7390 2024-08-28T19:38:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.006+0000 7efd39c81740 1 -- 172.21.15.16:0/2757458705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd280aaaa0 con 0x55a5a4bd7390 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.006+0000 7efd14ff9640 1 -- 172.21.15.16:0/2757458705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7efd1c0053c0 con 0x55a5a4bd7390 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.006+0000 7efd14ff9640 1 -- 172.21.15.16:0/2757458705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7efd1c023050 con 0x55a5a4bd7390 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.006+0000 7efd39c81740 10 client.19638 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.006+0000 7efd39c81740 1 -- 172.21.15.16:0/2757458705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x55a5a4fd11d0 con 0x55a5a4bd7390 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.006+0000 7efd14ff9640 1 -- 172.21.15.16:0/2757458705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd280aaaa0 con 0x55a5a4bd7390 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.007+0000 7efd14ff9640 1 -- 172.21.15.16:0/2757458705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7efd1c00f040 con 0x55a5a4bd7390 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.007+0000 7efd39c81740 10 client.19638 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.007+0000 7efd39c81740 10 client.19638 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.007+0000 7efd39c81740 20 client.19638 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.007+0000 7efd39c81740 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a5a4bb69f0 0x55a5a4fd4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.007+0000 7efd39c81740 4 client.19638 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:38:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.007+0000 7efd39c81740 1 -- 172.21.15.16:0/2757458705 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5a4f165d0 con 0x55a5a4bb69f0 2024-08-28T19:38:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.007+0000 7efd314a8640 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a5a4bb69f0 0x55a5a4fd4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.008+0000 7efd14ff9640 10 client.19638 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:38:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.009+0000 7efd314a8640 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a5a4bb69f0 0x55a5a4fd4640 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=42325f77696ac7e5 server_cookie=851b1fe4db2b79a0 in_seq=0 out_seq=0 2024-08-28T19:38:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.010+0000 7efd14ff9640 1 -- 172.21.15.16:0/2757458705 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a5a4f165d0 con 0x55a5a4bb69f0 2024-08-28T19:38:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.010+0000 7efd14ff9640 10 client.19638 handle_command_reply: tid=0 2024-08-28T19:38:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.010+0000 7efd39c81740 1 client.19638 shutdown 2024-08-28T19:38:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.011+0000 7efd39c81740 1 -- 172.21.15.16:0/2757458705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 msgr2=0x55a5a4fb17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.011+0000 7efd39c81740 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb17b0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009fd0 tx=0x7efd1c010800 comp rx=0 tx=0).stop 2024-08-28T19:38:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.011+0000 7efd39c81740 1 -- 172.21.15.16:0/2757458705 shutdown_connections 2024-08-28T19:38:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.011+0000 7efd39c81740 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a5a4bb69f0 0x55a5a4fd4640 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.011+0000 7efd39c81740 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bf8070 0x55a5a4fbd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.011+0000 7efd39c81740 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb17b0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.011+0000 7efd39c81740 1 --2- 172.21.15.16:0/2757458705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4fb1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.011+0000 7efd14ff9640 0 client.19638 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:38:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.011+0000 7efd39c81740 1 -- 172.21.15.16:0/2757458705 >> 172.21.15.16:0/2757458705 conn(0x55a5a4bec460 msgr2=0x55a5a4f9ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.012+0000 7efd39c81740 1 -- 172.21.15.16:0/2757458705 shutdown_connections 2024-08-28T19:38:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.012+0000 7efd39c81740 1 -- 172.21.15.16:0/2757458705 wait complete. 2024-08-28T19:38:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.012+0000 7efd39c81740 20 client.19638 trim_cache size 0 max 16384 2024-08-28T19:38:43.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.036+0000 7efd39c81740 1 Processor -- start 2024-08-28T19:38:43.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.036+0000 7efd39c81740 1 -- start start 2024-08-28T19:38:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.036+0000 7efd39c81740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4f9a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.036+0000 7efd39c81740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4f9ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.036+0000 7efd39c81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bf8070 0x55a5a4fb1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd39c81740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c1a6380 con 0x55a5a4bd7390 2024-08-28T19:38:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd39c81740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c1a80a0 con 0x55a5a4bf8070 2024-08-28T19:38:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd39c81740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c1a6b10 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd30ca7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4f9ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd30ca7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4f9ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51506/0 (socket says 172.21.15.16:51506) 2024-08-28T19:38:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd30ca7640 1 -- 172.21.15.16:0/4169955463 learned_addr learned my addr 172.21.15.16:0/4169955463 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd314a8640 1 --2- 172.21.15.16:0/4169955463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4f9a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd31ca9640 1 --2- 172.21.15.16:0/4169955463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bf8070 0x55a5a4fb1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd30ca7640 1 -- 172.21.15.16:0/4169955463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bcf1b0 msgr2=0x55a5a4f9a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd30ca7640 1 --2- 172.21.15.16:0/4169955463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4f9a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd30ca7640 1 -- 172.21.15.16:0/4169955463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bf8070 msgr2=0x55a5a4fb1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd30ca7640 1 --2- 172.21.15.16:0/4169955463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bf8070 0x55a5a4fb1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd30ca7640 1 -- 172.21.15.16:0/4169955463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a4fb2310 con 0x55a5a4bd7390 2024-08-28T19:38:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.037+0000 7efd31ca9640 1 --2- 172.21.15.16:0/4169955463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bf8070 0x55a5a4fb1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:38:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.038+0000 7efd30ca7640 1 --2- 172.21.15.16:0/4169955463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4f9ad10 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009c40 tx=0x7efd1c009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1bf92304955ad7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.039+0000 7efd16ffd640 1 -- 172.21.15.16:0/4169955463 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c012660 con 0x55a5a4bd7390 2024-08-28T19:38:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.039+0000 7efd16ffd640 1 -- 172.21.15.16:0/4169955463 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd1c012800 con 0x55a5a4bd7390 2024-08-28T19:38:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.039+0000 7efd16ffd640 1 -- 172.21.15.16:0/4169955463 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c004bc0 con 0x55a5a4bd7390 2024-08-28T19:38:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.040+0000 7efd39c81740 1 -- 172.21.15.16:0/4169955463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 msgr2=0x55a5a4f9ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.040+0000 7efd39c81740 1 --2- 172.21.15.16:0/4169955463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4f9ad10 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009c40 tx=0x7efd1c009c70 comp rx=0 tx=0).stop 2024-08-28T19:38:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.040+0000 7efd39c81740 1 -- 172.21.15.16:0/4169955463 shutdown_connections 2024-08-28T19:38:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.040+0000 7efd39c81740 1 --2- 172.21.15.16:0/4169955463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bf8070 0x55a5a4fb1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.040+0000 7efd39c81740 1 --2- 172.21.15.16:0/4169955463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bd7390 0x55a5a4f9ad10 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.040+0000 7efd39c81740 1 --2- 172.21.15.16:0/4169955463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4f9a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.040+0000 7efd39c81740 1 -- 172.21.15.16:0/4169955463 >> 172.21.15.16:0/4169955463 conn(0x55a5a4bec460 msgr2=0x55a5a4f9ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.040+0000 7efd39c81740 1 -- 172.21.15.16:0/4169955463 shutdown_connections 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.040+0000 7efd39c81740 1 -- 172.21.15.16:0/4169955463 wait complete. 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.041+0000 7efd39c81740 1 -- start start 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.041+0000 7efd39c81740 1 Processor -- start 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.042+0000 7efd39c81740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4fb5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.042+0000 7efd39c81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.042+0000 7efd39c81740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bf8070 0x55a5a5038080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.042+0000 7efd39c81740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c1a7910 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.042+0000 7efd39c81740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c1a71e0 con 0x55a5a4bd7390 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.042+0000 7efd39c81740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c073900 con 0x55a5a4bf8070 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.042+0000 7efd314a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4fb5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.042+0000 7efd39c81740 10 client.0 _ll_register_callbacks cb 0x7efd1c009430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.042+0000 7efd30ca7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.042+0000 7efd314a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4fb5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51512/0 (socket says 172.21.15.16:51512) 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.042+0000 7efd314a8640 1 -- 172.21.15.16:0/3967251446 learned_addr learned my addr 172.21.15.16:0/3967251446 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.043+0000 7efd314a8640 1 -- 172.21.15.16:0/3967251446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bf8070 msgr2=0x55a5a5038080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.043+0000 7efd314a8640 1 --2- 172.21.15.16:0/3967251446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bf8070 0x55a5a5038080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.043+0000 7efd314a8640 1 -- 172.21.15.16:0/3967251446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bd7390 msgr2=0x55a5a4fb65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.043+0000 7efd314a8640 1 --2- 172.21.15.16:0/3967251446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.043+0000 7efd314a8640 1 -- 172.21.15.16:0/3967251446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a50386f0 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.043+0000 7efd314a8640 1 --2- 172.21.15.16:0/3967251446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4fb5fd0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7efd280aaf20 tx=0x7efd28017570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fc30e4f3ef0d30 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.043+0000 7efd14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.044+0000 7efd14ff9640 1 -- 172.21.15.16:0/3967251446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2801e040 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.044+0000 7efd14ff9640 1 -- 172.21.15.16:0/3967251446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7efd28017dc0 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.044+0000 7efd39c81740 1 -- 172.21.15.16:0/3967251446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd1c012800 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.044+0000 7efd14ff9640 1 -- 172.21.15.16:0/3967251446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7efd280aaaa0 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.044+0000 7efd39c81740 10 client.19644 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:43.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.044+0000 7efd39c81740 1 -- 172.21.15.16:0/3967251446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x55a5a503fae0 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.044+0000 7efd14ff9640 1 -- 172.21.15.16:0/3967251446 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd280af800 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.045+0000 7efd14ff9640 1 -- 172.21.15.16:0/3967251446 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7efd280af440 con 0x55a5a4bcf1b0 2024-08-28T19:38:43.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.045+0000 7efd39c81740 10 client.19644 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:43.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.045+0000 7efd39c81740 10 client.19644 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:38:43.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.045+0000 7efd39c81740 20 client.19644 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:43.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.045+0000 7efd39c81740 1 --2- 172.21.15.16:0/3967251446 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a5a4b8bda0 0x55a5a5041ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:43.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.045+0000 7efd39c81740 4 client.19644 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:38:43.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.045+0000 7efd39c81740 1 -- 172.21.15.16:0/3967251446 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7efd1c012800 con 0x55a5a4b8bda0 2024-08-28T19:38:43.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.045+0000 7efd30ca7640 1 --2- 172.21.15.16:0/3967251446 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a5a4b8bda0 0x55a5a5041ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:43.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.046+0000 7efd30ca7640 1 --2- 172.21.15.16:0/3967251446 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a5a4b8bda0 0x55a5a5041ef0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d3196bf9879d4ba5 server_cookie=c0d944166ca846a9 in_seq=0 out_seq=0 2024-08-28T19:38:43.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.046+0000 7efd14ff9640 10 client.19644 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:38:43.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.050+0000 7efd14ff9640 1 -- 172.21.15.16:0/3967251446 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7efd1c012800 con 0x55a5a4b8bda0 2024-08-28T19:38:43.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.050+0000 7efd14ff9640 10 client.19644 handle_command_reply: tid=0 2024-08-28T19:38:43.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.050+0000 7efd39c81740 1 client.19644 shutdown 2024-08-28T19:38:43.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd39c81740 1 -- 172.21.15.16:0/3967251446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 msgr2=0x55a5a4fb5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:43.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd39c81740 1 --2- 172.21.15.16:0/3967251446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4fb5fd0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7efd280aaf20 tx=0x7efd28017570 comp rx=0 tx=0).stop 2024-08-28T19:38:43.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd39c81740 1 -- 172.21.15.16:0/3967251446 shutdown_connections 2024-08-28T19:38:43.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd39c81740 1 --2- 172.21.15.16:0/3967251446 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55a5a4b8bda0 0x55a5a5041ef0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd39c81740 1 --2- 172.21.15.16:0/3967251446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a5a4bf8070 0x55a5a5038080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd39c81740 1 --2- 172.21.15.16:0/3967251446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5a4bd7390 0x55a5a4fb65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd39c81740 1 --2- 172.21.15.16:0/3967251446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a4bcf1b0 0x55a5a4fb5fd0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd14ff9640 0 client.19644 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:38:43.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd39c81740 1 -- 172.21.15.16:0/3967251446 >> 172.21.15.16:0/3967251446 conn(0x55a5a4bec460 msgr2=0x55a5a4fb1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:43.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd39c81740 1 -- 172.21.15.16:0/3967251446 shutdown_connections 2024-08-28T19:38:43.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd39c81740 1 -- 172.21.15.16:0/3967251446 wait complete. 2024-08-28T19:38:43.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.051+0000 7efd39c81740 20 client.19644 trim_cache size 0 max 16384 2024-08-28T19:38:43.053 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:38:43.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 -- 172.21.15.16:0/3846759831 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd180b4da0 msgr2=0x7efd180b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:43.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd180b4da0 0x7efd180b72b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7efd2402f540 tx=0x7efd24005fd0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 -- 172.21.15.16:0/3846759831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd2c102800 msgr2=0x7efd2c199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:43.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd2c102800 0x7efd2c199110 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7efd2800b4a0 tx=0x7efd2800b960 comp rx=0 tx=0).stop 2024-08-28T19:38:43.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 -- 172.21.15.16:0/3846759831 shutdown_connections 2024-08-28T19:38:43.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd2c108a90 0x7efd2c19e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efd180b4da0 0x7efd180b72b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd2c108140 0x7efd2c1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 --2- 172.21.15.16:0/3846759831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd2c102800 0x7efd2c199110 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:43.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 -- 172.21.15.16:0/3846759831 >> 172.21.15.16:0/3846759831 conn(0x7efd2c0794d0 msgr2=0x7efd2c104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:43.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 -- 172.21.15.16:0/3846759831 shutdown_connections 2024-08-28T19:38:43.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:43.052+0000 7efd3370c640 1 -- 172.21.15.16:0/3846759831 wait complete. 2024-08-28T19:38:43.476 INFO:tasks.workunit.client.0.smithi016.stdout: 10 100639 24.14 MB/sec execute 139 sec latency 130.262 ms 2024-08-28T19:38:43.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:43 smithi016 ceph-mon[18556]: pgmap v891: 129 pgs: 129 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 18 MiB/s wr, 881 op/s 2024-08-28T19:38:43.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:43 smithi028 ceph-mon[21769]: pgmap v891: 129 pgs: 129 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 18 MiB/s wr, 881 op/s 2024-08-28T19:38:43.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:38:43.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:43 smithi115 ceph-mon[22184]: pgmap v891: 129 pgs: 129 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 18 MiB/s wr, 881 op/s 2024-08-28T19:38:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:44.476 INFO:tasks.workunit.client.0.smithi016.stdout: 10 101390 24.27 MB/sec execute 140 sec latency 129.495 ms 2024-08-28T19:38:44.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.736+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/398842244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a8810f890 msgr2=0x7f2a8810fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:44.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.736+0000 7f2a8c9e2640 1 --2- 172.21.15.16:0/398842244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a8810f890 0x7f2a8810fca0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2a78009900 tx=0x7f2a7802a400 comp rx=0 tx=0).stop 2024-08-28T19:38:44.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.736+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/398842244 shutdown_connections 2024-08-28T19:38:44.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.736+0000 7f2a8c9e2640 1 --2- 172.21.15.16:0/398842244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a881101e0 0x7f2a88116a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.736+0000 7f2a8c9e2640 1 --2- 172.21.15.16:0/398842244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a8810f890 0x7f2a8810fca0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.736+0000 7f2a8c9e2640 1 --2- 172.21.15.16:0/398842244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a88109f50 0x7f2a8810a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.736+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/398842244 >> 172.21.15.16:0/398842244 conn(0x7f2a88077e80 msgr2=0x7f2a88107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:44.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.737+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/398842244 shutdown_connections 2024-08-28T19:38:44.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.737+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/398842244 wait complete. 2024-08-28T19:38:44.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.738+0000 7f2a8c9e2640 1 Processor -- start 2024-08-28T19:38:44.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.739+0000 7f2a8c9e2640 1 -- start start 2024-08-28T19:38:44.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.739+0000 7f2a8c9e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a88109f50 0x7f2a881a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.739+0000 7f2a8c9e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a8810f890 0x7f2a881a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.740+0000 7f2a8659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a88109f50 0x7f2a881a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.740+0000 7f2a8659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a88109f50 0x7f2a881a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37906/0 (socket says 172.21.15.16:37906) 2024-08-28T19:38:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.740+0000 7f2a7dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a8810f890 0x7f2a881a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.740+0000 7f2a8659c640 1 -- 172.21.15.16:0/1793823457 learned_addr learned my addr 172.21.15.16:0/1793823457 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.740+0000 7f2a8c9e2640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a881101e0 0x7f2a881a5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.740+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/1793823457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a88119ef0 con 0x7f2a8810f890 2024-08-28T19:38:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.740+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/1793823457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a88119d70 con 0x7f2a88109f50 2024-08-28T19:38:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.740+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/1793823457 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8811a070 con 0x7f2a881101e0 2024-08-28T19:38:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.740+0000 7f2a86d9d640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a881101e0 0x7f2a881a5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.741+0000 7f2a86d9d640 1 -- 172.21.15.16:0/1793823457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a88109f50 msgr2=0x7f2a881a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.741+0000 7f2a86d9d640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a88109f50 0x7f2a881a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.741+0000 7f2a86d9d640 1 -- 172.21.15.16:0/1793823457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a8810f890 msgr2=0x7f2a881a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.741+0000 7f2a86d9d640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a8810f890 0x7f2a881a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.741+0000 7f2a86d9d640 1 -- 172.21.15.16:0/1793823457 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a881a65e0 con 0x7f2a881101e0 2024-08-28T19:38:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.741+0000 7f2a86d9d640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a881101e0 0x7f2a881a5eb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2a80009fd0 tx=0x7f2a8000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.744+0000 7f2a7ffff640 1 -- 172.21.15.16:0/1793823457 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8000ce20 con 0x7f2a881101e0 2024-08-28T19:38:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.744+0000 7f2a7ffff640 1 -- 172.21.15.16:0/1793823457 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a80005020 con 0x7f2a881101e0 2024-08-28T19:38:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.744+0000 7f2a7ffff640 1 -- 172.21.15.16:0/1793823457 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a80010690 con 0x7f2a881101e0 2024-08-28T19:38:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.744+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/1793823457 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a881a68d0 con 0x7f2a881101e0 2024-08-28T19:38:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.745+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/1793823457 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a881ad030 con 0x7f2a881101e0 2024-08-28T19:38:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.748+0000 7f2a92f57740 1 Processor -- start 2024-08-28T19:38:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.751+0000 7f2a7ffff640 1 -- 172.21.15.16:0/1793823457 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2a800054d0 con 0x7f2a881101e0 2024-08-28T19:38:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.751+0000 7f2a92f57740 1 -- start start 2024-08-28T19:38:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.753+0000 7f2a7ffff640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a740b4e70 0x7f2a740b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.753+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b817350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.753+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b4491b0 0x56509b817890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.753+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b451390 0x56509b817dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.753+0000 7f2a92f57740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56509b78e680 con 0x56509b472070 2024-08-28T19:38:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.753+0000 7f2a92f57740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56509b77dc10 con 0x56509b451390 2024-08-28T19:38:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.753+0000 7f2a92f57740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56509b781bc0 con 0x56509b4491b0 2024-08-28T19:38:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.753+0000 7f2a7ffff640 1 -- 172.21.15.16:0/1793823457 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f2a800e5f60 con 0x7f2a881101e0 2024-08-28T19:38:44.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.754+0000 7f2a86d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b451390 0x56509b817dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.754+0000 7f2a86d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b451390 0x56509b817dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37914/0 (socket says 172.21.15.16:37914) 2024-08-28T19:38:44.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.754+0000 7f2a86d9d640 1 -- 172.21.15.16:0/2828325945 learned_addr learned my addr 172.21.15.16:0/2828325945 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:44.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.754+0000 7f2a8659c640 1 --2- 172.21.15.16:0/2828325945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b817350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.754+0000 7f2a86d9d640 1 -- 172.21.15.16:0/2828325945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b4491b0 msgr2=0x56509b817890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:38:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.754+0000 7f2a86d9d640 1 --2- 172.21.15.16:0/2828325945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b4491b0 0x56509b817890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.754+0000 7f2a86d9d640 1 -- 172.21.15.16:0/2828325945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 msgr2=0x56509b817350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.754+0000 7f2a86d9d640 1 --2- 172.21.15.16:0/2828325945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b817350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.754+0000 7f2a7dd9b640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a740b4e70 0x7f2a740b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.754+0000 7f2a86d9d640 1 -- 172.21.15.16:0/2828325945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56509b4dd650 con 0x56509b451390 2024-08-28T19:38:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.755+0000 7f2a7dd9b640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a740b4e70 0x7f2a740b7380 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f2a7802a920 tx=0x7f2a7800b040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:44.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.756+0000 7f2a86d9d640 1 --2- 172.21.15.16:0/2828325945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b451390 0x56509b817dd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2a80005820 tx=0x7f2a800abe70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b1ba137f97eacd5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:44.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.759+0000 7f2a7cd99640 1 -- 172.21.15.16:0/2828325945 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a800b0a90 con 0x56509b451390 2024-08-28T19:38:44.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.759+0000 7f2a7cd99640 1 -- 172.21.15.16:0/2828325945 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a800b5540 con 0x56509b451390 2024-08-28T19:38:44.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.760+0000 7f2a92f57740 1 -- 172.21.15.16:0/2828325945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b451390 msgr2=0x56509b817dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:44.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.760+0000 7f2a92f57740 1 --2- 172.21.15.16:0/2828325945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b451390 0x56509b817dd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2a80005820 tx=0x7f2a800abe70 comp rx=0 tx=0).stop 2024-08-28T19:38:44.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.761+0000 7f2a92f57740 1 -- 172.21.15.16:0/2828325945 shutdown_connections 2024-08-28T19:38:44.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.761+0000 7f2a92f57740 1 --2- 172.21.15.16:0/2828325945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b451390 0x56509b817dd0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.761+0000 7f2a92f57740 1 --2- 172.21.15.16:0/2828325945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b4491b0 0x56509b817890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.761+0000 7f2a92f57740 1 --2- 172.21.15.16:0/2828325945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b817350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.761+0000 7f2a92f57740 1 -- 172.21.15.16:0/2828325945 >> 172.21.15.16:0/2828325945 conn(0x56509b466460 msgr2=0x56509b815240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:44.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.761+0000 7f2a92f57740 1 -- 172.21.15.16:0/2828325945 shutdown_connections 2024-08-28T19:38:44.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.761+0000 7f2a92f57740 1 -- 172.21.15.16:0/2828325945 wait complete. 2024-08-28T19:38:44.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.762+0000 7f2a92f57740 1 -- start start 2024-08-28T19:38:44.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.763+0000 7f2a92f57740 1 Processor -- start 2024-08-28T19:38:44.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.765+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 0x56509b822a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.765+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b451390 0x56509b822f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.765+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b472070 0x56509b82eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.765+0000 7f2a92f57740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56509b5e0700 con 0x56509b451390 2024-08-28T19:38:44.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.765+0000 7f2a92f57740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56509b704750 con 0x56509b4491b0 2024-08-28T19:38:44.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.765+0000 7f2a92f57740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56509b825b80 con 0x56509b472070 2024-08-28T19:38:44.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.765+0000 7f2a92f57740 10 client.0 _ll_register_callbacks cb 0x56509b733e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:44.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.766+0000 7f2a7dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b451390 0x56509b822f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.766+0000 7f2a7dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b451390 0x56509b822f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60052/0 (socket says 172.21.15.16:60052) 2024-08-28T19:38:44.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.766+0000 7f2a7dd9b640 1 -- 172.21.15.16:0/1310431029 learned_addr learned my addr 172.21.15.16:0/1310431029 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:44.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.766+0000 7f2a8659c640 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 0x56509b822a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.766+0000 7f2a86d9d640 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b472070 0x56509b82eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.767+0000 7f2a7dd9b640 1 -- 172.21.15.16:0/1310431029 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b472070 msgr2=0x56509b82eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.767+0000 7f2a7dd9b640 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b472070 0x56509b82eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.767+0000 7f2a7dd9b640 1 -- 172.21.15.16:0/1310431029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 msgr2=0x56509b822a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.767+0000 7f2a7dd9b640 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 0x56509b822a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.767+0000 7f2a7dd9b640 1 -- 172.21.15.16:0/1310431029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56509b73d900 con 0x56509b451390 2024-08-28T19:38:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.768+0000 7f2a7dd9b640 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b451390 0x56509b822f50 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f2a78009900 tx=0x7f2a78031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a82c3f39413a253 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.768+0000 7f2a6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:38:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.768+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1310431029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7804f060 con 0x56509b451390 2024-08-28T19:38:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.768+0000 7f2a92f57740 1 -- 172.21.15.16:0/1310431029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a800b5540 con 0x56509b451390 2024-08-28T19:38:44.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.769+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1310431029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f2a78005500 con 0x56509b451390 2024-08-28T19:38:44.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.769+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1310431029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f2a78056050 con 0x56509b451390 2024-08-28T19:38:44.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.770+0000 7f2a92f57740 10 client.19656 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:44.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.770+0000 7f2a92f57740 1 -- 172.21.15.16:0/1310431029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x56509b831930 con 0x56509b451390 2024-08-28T19:38:44.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.771+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1310431029 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a800b5540 con 0x56509b451390 2024-08-28T19:38:44.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.771+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1310431029 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7f2a7803c680 con 0x56509b451390 2024-08-28T19:38:44.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.771+0000 7f2a92f57740 10 client.19656 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:44.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.771+0000 7f2a92f57740 10 client.19656 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:38:44.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.771+0000 7f2a92f57740 20 client.19656 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:44.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.771+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56509b4309f0 0x56509b8352a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.771+0000 7f2a92f57740 4 client.19656 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:38:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.771+0000 7f2a92f57740 1 -- 172.21.15.16:0/1310431029 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56509b7905d0 con 0x56509b4309f0 2024-08-28T19:38:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.771+0000 7f2a8659c640 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56509b4309f0 0x56509b8352a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.773+0000 7f2a8659c640 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56509b4309f0 0x56509b8352a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=551545bc7ad6fbdc server_cookie=13c42bd2192176a1 in_seq=0 out_seq=0 2024-08-28T19:38:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.773+0000 7f2a6dffb640 10 client.19656 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:38:44.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1310431029 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56509b7905d0 con 0x56509b4309f0 2024-08-28T19:38:44.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a6dffb640 10 client.19656 handle_command_reply: tid=0 2024-08-28T19:38:44.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a92f57740 1 client.19656 shutdown 2024-08-28T19:38:44.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a92f57740 1 -- 172.21.15.16:0/1310431029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b451390 msgr2=0x56509b822f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:44.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b451390 0x56509b822f50 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f2a78009900 tx=0x7f2a78031400 comp rx=0 tx=0).stop 2024-08-28T19:38:44.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a92f57740 1 -- 172.21.15.16:0/1310431029 shutdown_connections 2024-08-28T19:38:44.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56509b4309f0 0x56509b8352a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b472070 0x56509b82eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b451390 0x56509b822f50 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1310431029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 0x56509b822a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a92f57740 1 -- 172.21.15.16:0/1310431029 >> 172.21.15.16:0/1310431029 conn(0x56509b466460 msgr2=0x56509b81a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:44.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a6dffb640 0 client.19656 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:38:44.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.774+0000 7f2a92f57740 1 -- 172.21.15.16:0/1310431029 shutdown_connections 2024-08-28T19:38:44.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.775+0000 7f2a92f57740 1 -- 172.21.15.16:0/1310431029 wait complete. 2024-08-28T19:38:44.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.775+0000 7f2a92f57740 20 client.19656 trim_cache size 0 max 16384 2024-08-28T19:38:44.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.803+0000 7f2a92f57740 1 Processor -- start 2024-08-28T19:38:44.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.803+0000 7f2a92f57740 1 -- start start 2024-08-28T19:38:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.803+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 0x56509b822ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b451390 0x56509b81a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a8659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 0x56509b822ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a8659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 0x56509b822ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37922/0 (socket says 172.21.15.16:37922) 2024-08-28T19:38:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b81a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a7dd9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b451390 0x56509b81a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a7dd9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b451390 0x56509b81a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55444/0 (socket says 172.21.15.16:55444) 2024-08-28T19:38:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a8659c640 1 -- 172.21.15.16:0/651631201 learned_addr learned my addr 172.21.15.16:0/651631201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a92f57740 1 -- 172.21.15.16:0/651631201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a881b73a0 con 0x56509b472070 2024-08-28T19:38:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a92f57740 1 -- 172.21.15.16:0/651631201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a881b90c0 con 0x56509b4491b0 2024-08-28T19:38:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a92f57740 1 -- 172.21.15.16:0/651631201 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2a881b7b30 con 0x56509b451390 2024-08-28T19:38:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a86d9d640 1 --2- 172.21.15.16:0/651631201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b81a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a8659c640 1 -- 172.21.15.16:0/651631201 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b451390 msgr2=0x56509b81a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a8659c640 1 --2- 172.21.15.16:0/651631201 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b451390 0x56509b81a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a8659c640 1 -- 172.21.15.16:0/651631201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 msgr2=0x56509b81a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.804+0000 7f2a8659c640 1 --2- 172.21.15.16:0/651631201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b81a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.805+0000 7f2a8659c640 1 -- 172.21.15.16:0/651631201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56509b823ab0 con 0x56509b4491b0 2024-08-28T19:38:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.806+0000 7f2a86d9d640 1 --2- 172.21.15.16:0/651631201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b81a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:38:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.806+0000 7f2a8659c640 1 --2- 172.21.15.16:0/651631201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 0x56509b822ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2a700058c0 tx=0x7f2a70013830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=211aeb81584bbb8b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.807+0000 7f2a6ffff640 1 -- 172.21.15.16:0/651631201 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7000bce0 con 0x56509b4491b0 2024-08-28T19:38:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.807+0000 7f2a6ffff640 1 -- 172.21.15.16:0/651631201 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a700130d0 con 0x56509b4491b0 2024-08-28T19:38:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.807+0000 7f2a6ffff640 1 -- 172.21.15.16:0/651631201 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a70003ca0 con 0x56509b4491b0 2024-08-28T19:38:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.809+0000 7f2a92f57740 1 -- 172.21.15.16:0/651631201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 msgr2=0x56509b822ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:44.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.809+0000 7f2a92f57740 1 --2- 172.21.15.16:0/651631201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 0x56509b822ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2a700058c0 tx=0x7f2a70013830 comp rx=0 tx=0).stop 2024-08-28T19:38:44.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.809+0000 7f2a92f57740 1 -- 172.21.15.16:0/651631201 shutdown_connections 2024-08-28T19:38:44.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.809+0000 7f2a92f57740 1 --2- 172.21.15.16:0/651631201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b81a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.809+0000 7f2a92f57740 1 --2- 172.21.15.16:0/651631201 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b451390 0x56509b81a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.809+0000 7f2a92f57740 1 --2- 172.21.15.16:0/651631201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b4491b0 0x56509b822ed0 secure :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2a700058c0 tx=0x7f2a70013830 comp rx=0 tx=0).stop 2024-08-28T19:38:44.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.809+0000 7f2a92f57740 1 -- 172.21.15.16:0/651631201 >> 172.21.15.16:0/651631201 conn(0x56509b466460 msgr2=0x56509b8227c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:44.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.809+0000 7f2a92f57740 1 -- 172.21.15.16:0/651631201 shutdown_connections 2024-08-28T19:38:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.809+0000 7f2a92f57740 1 -- 172.21.15.16:0/651631201 wait complete. 2024-08-28T19:38:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.810+0000 7f2a92f57740 1 -- start start 2024-08-28T19:38:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.810+0000 7f2a92f57740 1 Processor -- start 2024-08-28T19:38:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.811+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b451390 0x56509b82b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.811+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b826b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.811+0000 7f2a92f57740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b405da0 0x56509b827080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.811+0000 7f2a92f57740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a881b8930 con 0x56509b472070 2024-08-28T19:38:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.811+0000 7f2a92f57740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a881b8200 con 0x56509b405da0 2024-08-28T19:38:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.811+0000 7f2a92f57740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2a881b53a0 con 0x56509b451390 2024-08-28T19:38:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.811+0000 7f2a92f57740 10 client.0 _ll_register_callbacks cb 0x7f2a78040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.811+0000 7f2a86d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b405da0 0x56509b827080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.812+0000 7f2a86d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b405da0 0x56509b827080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37930/0 (socket says 172.21.15.16:37930) 2024-08-28T19:38:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.812+0000 7f2a86d9d640 1 -- 172.21.15.16:0/1682539610 learned_addr learned my addr 172.21.15.16:0/1682539610 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.812+0000 7f2a7dd9b640 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b826b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.812+0000 7f2a8659c640 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b451390 0x56509b82b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.812+0000 7f2a7dd9b640 1 -- 172.21.15.16:0/1682539610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b451390 msgr2=0x56509b82b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.812+0000 7f2a7dd9b640 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b451390 0x56509b82b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.812+0000 7f2a7dd9b640 1 -- 172.21.15.16:0/1682539610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b405da0 msgr2=0x56509b827080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.812+0000 7f2a7dd9b640 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b405da0 0x56509b827080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.812+0000 7f2a7dd9b640 1 -- 172.21.15.16:0/1682539610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56509b827810 con 0x56509b472070 2024-08-28T19:38:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.813+0000 7f2a7dd9b640 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b826b20 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f2a78004f20 tx=0x7f2a7803cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab7ad0dd6a9f2f2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.813+0000 7f2a6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:38:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.814+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1682539610 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7804f040 con 0x56509b472070 2024-08-28T19:38:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.814+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1682539610 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f2a78005500 con 0x56509b472070 2024-08-28T19:38:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.814+0000 7f2a92f57740 1 -- 172.21.15.16:0/1682539610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a700130d0 con 0x56509b472070 2024-08-28T19:38:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.814+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1682539610 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f2a78035040 con 0x56509b472070 2024-08-28T19:38:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.814+0000 7f2a92f57740 10 client.19659 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.814+0000 7f2a92f57740 1 -- 172.21.15.16:0/1682539610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x56509b89fb70 con 0x56509b472070 2024-08-28T19:38:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.815+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1682539610 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a700130d0 con 0x56509b472070 2024-08-28T19:38:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.815+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1682539610 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7f2a78064580 con 0x56509b472070 2024-08-28T19:38:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.815+0000 7f2a92f57740 10 client.19659 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.815+0000 7f2a92f57740 10 client.19659 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:38:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.815+0000 7f2a92f57740 20 client.19659 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.815+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56509b8a02a0 0x56509b8a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.815+0000 7f2a92f57740 4 client.19659 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:38:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.815+0000 7f2a92f57740 1 -- 172.21.15.16:0/1682539610 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56509b78c070 con 0x56509b8a02a0 2024-08-28T19:38:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.816+0000 7f2a8659c640 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56509b8a02a0 0x56509b8a26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:44.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.818+0000 7f2a8659c640 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56509b8a02a0 0x56509b8a26c0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=18feab040b20d672 server_cookie=a123916f47b68990 in_seq=0 out_seq=0 2024-08-28T19:38:44.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.818+0000 7f2a6dffb640 10 client.19659 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:38:44.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.819+0000 7f2a6dffb640 1 -- 172.21.15.16:0/1682539610 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56509b78c070 con 0x56509b8a02a0 2024-08-28T19:38:44.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.819+0000 7f2a6dffb640 10 client.19659 handle_command_reply: tid=0 2024-08-28T19:38:44.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.819+0000 7f2a92f57740 1 client.19659 shutdown 2024-08-28T19:38:44.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.820+0000 7f2a92f57740 1 -- 172.21.15.16:0/1682539610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 msgr2=0x56509b826b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:44.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.820+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b826b20 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f2a78004f20 tx=0x7f2a7803cea0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.820+0000 7f2a92f57740 1 -- 172.21.15.16:0/1682539610 shutdown_connections 2024-08-28T19:38:44.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.820+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56509b8a02a0 0x56509b8a26c0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.820+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56509b405da0 0x56509b827080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.820+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56509b472070 0x56509b826b20 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.820+0000 7f2a92f57740 1 --2- 172.21.15.16:0/1682539610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56509b451390 0x56509b82b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.820+0000 7f2a92f57740 1 -- 172.21.15.16:0/1682539610 >> 172.21.15.16:0/1682539610 conn(0x56509b466460 msgr2=0x56509b81a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:44.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.821+0000 7f2a6dffb640 0 client.19659 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:38:44.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.821+0000 7f2a92f57740 1 -- 172.21.15.16:0/1682539610 shutdown_connections 2024-08-28T19:38:44.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.821+0000 7f2a92f57740 1 -- 172.21.15.16:0/1682539610 wait complete. 2024-08-28T19:38:44.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.821+0000 7f2a92f57740 20 client.19659 trim_cache size 0 max 16384 2024-08-28T19:38:44.824 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:38:44.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/1793823457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a740b4e70 msgr2=0x7f2a740b7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:44.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a740b4e70 0x7f2a740b7380 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f2a7802a920 tx=0x7f2a7800b040 comp rx=0 tx=0).stop 2024-08-28T19:38:44.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/1793823457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a881101e0 msgr2=0x7f2a881a5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:44.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a881101e0 0x7f2a881a5eb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2a80009fd0 tx=0x7f2a8000ef10 comp rx=0 tx=0).stop 2024-08-28T19:38:44.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/1793823457 shutdown_connections 2024-08-28T19:38:44.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a881101e0 0x7f2a881a5eb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a8810f890 0x7f2a881a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2a740b4e70 0x7f2a740b7380 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 --2- 172.21.15.16:0/1793823457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a88109f50 0x7f2a881a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:44.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/1793823457 >> 172.21.15.16:0/1793823457 conn(0x7f2a88077e80 msgr2=0x7f2a8810d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:44.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/1793823457 shutdown_connections 2024-08-28T19:38:44.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:44.824+0000 7f2a8c9e2640 1 -- 172.21.15.16:0/1793823457 wait complete. 2024-08-28T19:38:45.476 INFO:tasks.workunit.client.0.smithi016.stdout: 10 102159 24.46 MB/sec execute 141 sec latency 62.868 ms 2024-08-28T19:38:45.645 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:38:45.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:45 smithi016 ceph-mon[18556]: pgmap v892: 129 pgs: 129 active+clean; 1.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 17 MiB/s wr, 829 op/s 2024-08-28T19:38:45.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:45 smithi028 ceph-mon[21769]: pgmap v892: 129 pgs: 129 active+clean; 1.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 17 MiB/s wr, 829 op/s 2024-08-28T19:38:45.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:45 smithi115 ceph-mon[22184]: pgmap v892: 129 pgs: 129 active+clean; 1.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 17 MiB/s wr, 829 op/s 2024-08-28T19:38:45.913 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.8 2024-08-28T19:38:45.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:38:45.987 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:46.476 INFO:tasks.workunit.client.0.smithi016.stdout: 10 102709 24.48 MB/sec execute 142 sec latency 148.288 ms 2024-08-28T19:38:47.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 103187 24.39 MB/sec execute 143 sec latency 160.212 ms 2024-08-28T19:38:47.634 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:47 smithi016 ceph-mon[18556]: pgmap v893: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 17 MiB/s wr, 931 op/s 2024-08-28T19:38:47.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:47 smithi028 ceph-mon[21769]: pgmap v893: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 17 MiB/s wr, 931 op/s 2024-08-28T19:38:47.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:47 smithi115 ceph-mon[22184]: pgmap v893: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 17 MiB/s wr, 931 op/s 2024-08-28T19:38:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.213+0000 7fabe284e640 1 -- 172.21.15.16:0/1910568318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabdc10dbf0 msgr2=0x7fabdc1100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.213+0000 7fabe284e640 1 --2- 172.21.15.16:0/1910568318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabdc10dbf0 0x7fabdc1100c0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009ca0 tx=0x7fabd402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.214+0000 7fabe284e640 1 -- 172.21.15.16:0/1910568318 shutdown_connections 2024-08-28T19:38:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.214+0000 7fabe284e640 1 --2- 172.21.15.16:0/1910568318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabdc10dbf0 0x7fabdc1100c0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.214+0000 7fabe284e640 1 --2- 172.21.15.16:0/1910568318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabdc105110 0x7fabdc10d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.214+0000 7fabe284e640 1 --2- 172.21.15.16:0/1910568318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabdc1047c0 0x7fabdc104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.214+0000 7fabe284e640 1 -- 172.21.15.16:0/1910568318 >> 172.21.15.16:0/1910568318 conn(0x7fabdc079470 msgr2=0x7fabdc079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.214+0000 7fabe284e640 1 -- 172.21.15.16:0/1910568318 shutdown_connections 2024-08-28T19:38:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.214+0000 7fabe284e640 1 -- 172.21.15.16:0/1910568318 wait complete. 2024-08-28T19:38:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.215+0000 7fabe284e640 1 Processor -- start 2024-08-28T19:38:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabe284e640 1 -- start start 2024-08-28T19:38:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabe284e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabdc1047c0 0x7fabdc19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabe284e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabdc105110 0x7fabdc19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabe284e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabdc10dbf0 0x7fabdc19f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabe284e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc113840 con 0x7fabdc1047c0 2024-08-28T19:38:48.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabe284e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc1136c0 con 0x7fabdc105110 2024-08-28T19:38:48.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabe284e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc1139c0 con 0x7fabdc10dbf0 2024-08-28T19:38:48.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabdbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabdc1047c0 0x7fabdc19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabdbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabdc1047c0 0x7fabdc19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60082/0 (socket says 172.21.15.16:60082) 2024-08-28T19:38:48.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabdb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabdc105110 0x7fabdc19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabdbfff640 1 -- 172.21.15.16:0/2643974863 learned_addr learned my addr 172.21.15.16:0/2643974863 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:48.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.216+0000 7fabe0deb640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabdc10dbf0 0x7fabdc19f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.217+0000 7fabdbfff640 1 -- 172.21.15.16:0/2643974863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabdc10dbf0 msgr2=0x7fabdc19f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:48.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.217+0000 7fabdbfff640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabdc10dbf0 0x7fabdc19f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.217+0000 7fabdbfff640 1 -- 172.21.15.16:0/2643974863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabdc105110 msgr2=0x7fabdc19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:48.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.217+0000 7fabdbfff640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabdc105110 0x7fabdc19a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.217+0000 7fabdbfff640 1 -- 172.21.15.16:0/2643974863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabdc1a00e0 con 0x7fabdc1047c0 2024-08-28T19:38:48.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.217+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabdc105110 0x7fabdc19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:38:48.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.217+0000 7fabdbfff640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabdc1047c0 0x7fabdc19a170 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00e9f0 tx=0x7fabcc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:48.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.217+0000 7fabd97fa640 1 -- 172.21.15.16:0/2643974863 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc00cd80 con 0x7fabdc1047c0 2024-08-28T19:38:48.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.217+0000 7fabe284e640 1 -- 172.21.15.16:0/2643974863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabdc1a03d0 con 0x7fabdc1047c0 2024-08-28T19:38:48.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.217+0000 7fabe284e640 1 -- 172.21.15.16:0/2643974863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabdc1023e0 con 0x7fabdc1047c0 2024-08-28T19:38:48.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.218+0000 7fabd97fa640 1 -- 172.21.15.16:0/2643974863 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabcc0108e0 con 0x7fabdc1047c0 2024-08-28T19:38:48.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.218+0000 7fabd97fa640 1 -- 172.21.15.16:0/2643974863 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc018980 con 0x7fabdc1047c0 2024-08-28T19:38:48.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.221+0000 7fabe8dc3740 1 Processor -- start 2024-08-28T19:38:48.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.221+0000 7fabe8dc3740 1 -- start start 2024-08-28T19:38:48.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabd97fa640 1 -- 172.21.15.16:0/2643974863 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fabcc018b20 con 0x7fabdc1047c0 2024-08-28T19:38:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56be070 0x55f6b5a63350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a63890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 0x55f6b5a63dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6b59da680 con 0x55f6b569d390 2024-08-28T19:38:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6b59c9c10 con 0x55f6b56be070 2024-08-28T19:38:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f6b59cdbc0 con 0x55f6b56951b0 2024-08-28T19:38:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabe0deb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56be070 0x55f6b5a63350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabe0deb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56be070 0x55f6b5a63350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37946/0 (socket says 172.21.15.16:37946) 2024-08-28T19:38:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabe0deb640 1 -- 172.21.15.16:0/2122659804 learned_addr learned my addr 172.21.15.16:0/2122659804 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabdbfff640 1 --2- 172.21.15.16:0/2122659804 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a63890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.222+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/2122659804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 0x55f6b5a63dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.223+0000 7fabe0deb640 1 -- 172.21.15.16:0/2122659804 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 msgr2=0x55f6b5a63890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.223+0000 7fabe0deb640 1 --2- 172.21.15.16:0/2122659804 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a63890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.223+0000 7fabe0deb640 1 -- 172.21.15.16:0/2122659804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 msgr2=0x55f6b5a63dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.223+0000 7fabe0deb640 1 --2- 172.21.15.16:0/2122659804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 0x55f6b5a63dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.223+0000 7fabe0deb640 1 -- 172.21.15.16:0/2122659804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6b5729650 con 0x55f6b56be070 2024-08-28T19:38:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.223+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/2122659804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 0x55f6b5a63dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:38:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.223+0000 7fabe0deb640 1 --2- 172.21.15.16:0/2122659804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56be070 0x55f6b5a63350 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fabd402f4c0 tx=0x7fabd4009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5d7fe52d79768d5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.224+0000 7fabd97fa640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabc40b4da0 0x7fabc40b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.224+0000 7fabdbfff640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabc40b4da0 0x7fabc40b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.224+0000 7fabc27fc640 1 -- 172.21.15.16:0/2122659804 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4031cf0 con 0x55f6b56be070 2024-08-28T19:38:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.224+0000 7fabc27fc640 1 -- 172.21.15.16:0/2122659804 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabd4036030 con 0x55f6b56be070 2024-08-28T19:38:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.224+0000 7fabd97fa640 1 -- 172.21.15.16:0/2643974863 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7fabcc0e5b30 con 0x7fabdc1047c0 2024-08-28T19:38:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.225+0000 7fabc27fc640 1 -- 172.21.15.16:0/2122659804 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd40042a0 con 0x55f6b56be070 2024-08-28T19:38:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.225+0000 7fabe8dc3740 1 -- 172.21.15.16:0/2122659804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56be070 msgr2=0x55f6b5a63350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.225+0000 7fabdbfff640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabc40b4da0 0x7fabc40b72b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x55f6b5a61fd0 tx=0x7fabcc0e9560 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.225+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/2122659804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56be070 0x55f6b5a63350 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fabd402f4c0 tx=0x7fabd4009800 comp rx=0 tx=0).stop 2024-08-28T19:38:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.226+0000 7fabe8dc3740 1 -- 172.21.15.16:0/2122659804 shutdown_connections 2024-08-28T19:38:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.226+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/2122659804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 0x55f6b5a63dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.226+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/2122659804 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a63890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.226+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/2122659804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56be070 0x55f6b5a63350 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.226+0000 7fabe8dc3740 1 -- 172.21.15.16:0/2122659804 >> 172.21.15.16:0/2122659804 conn(0x55f6b56b2460 msgr2=0x55f6b5a61240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.226+0000 7fabe8dc3740 1 -- 172.21.15.16:0/2122659804 shutdown_connections 2024-08-28T19:38:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.226+0000 7fabe8dc3740 1 -- 172.21.15.16:0/2122659804 wait complete. 2024-08-28T19:38:48.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.226+0000 7fabe8dc3740 1 -- start start 2024-08-28T19:38:48.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.227+0000 7fabe8dc3740 1 Processor -- start 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.227+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56951b0 0x55f6b5a771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.227+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b569d390 0x55f6b5a77710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.227+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 0x55f6b5a83530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.227+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6b582c700 con 0x55f6b56be070 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.227+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6b5950750 con 0x55f6b56951b0 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.227+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f6b5a7a0d0 con 0x55f6b569d390 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.227+0000 7fabe8dc3740 10 client.0 _ll_register_callbacks cb 0x55f6b597fe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.228+0000 7fabe0deb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 0x55f6b5a83530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.228+0000 7fabe0deb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 0x55f6b5a83530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60090/0 (socket says 172.21.15.16:60090) 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.228+0000 7fabe0deb640 1 -- 172.21.15.16:0/1766581951 learned_addr learned my addr 172.21.15.16:0/1766581951 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.228+0000 7fabe0deb640 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56951b0 0x55f6b5a771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.228+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b569d390 0x55f6b5a77710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.228+0000 7fabe0deb640 1 -- 172.21.15.16:0/1766581951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b569d390 msgr2=0x55f6b5a77710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.228+0000 7fabe0deb640 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b569d390 0x55f6b5a77710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.228+0000 7fabe0deb640 1 -- 172.21.15.16:0/1766581951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56951b0 msgr2=0x55f6b5a771d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.228+0000 7fabe0deb640 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56951b0 0x55f6b5a771d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.228+0000 7fabe0deb640 1 -- 172.21.15.16:0/1766581951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6b5989900 con 0x55f6b56be070 2024-08-28T19:38:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.229+0000 7fabe0deb640 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 0x55f6b5a83530 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009620 tx=0x7fabd4003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd5b3a8f0b73225 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.229+0000 7fabbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:38:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.229+0000 7fabbb7fe640 1 -- 172.21.15.16:0/1766581951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4059060 con 0x55f6b56be070 2024-08-28T19:38:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.229+0000 7fabbb7fe640 1 -- 172.21.15.16:0/1766581951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7fabd4004440 con 0x55f6b56be070 2024-08-28T19:38:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.229+0000 7fabe8dc3740 1 -- 172.21.15.16:0/1766581951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabd4036030 con 0x55f6b56be070 2024-08-28T19:38:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.230+0000 7fabbb7fe640 1 -- 172.21.15.16:0/1766581951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fabd4004680 con 0x55f6b56be070 2024-08-28T19:38:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.230+0000 7fabe8dc3740 10 client.19671 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.230+0000 7fabe8dc3740 1 -- 172.21.15.16:0/1766581951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x55f6b5a85f10 con 0x55f6b56be070 2024-08-28T19:38:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.231+0000 7fabbb7fe640 1 -- 172.21.15.16:0/1766581951 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabd4036030 con 0x55f6b56be070 2024-08-28T19:38:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.231+0000 7fabbb7fe640 1 -- 172.21.15.16:0/1766581951 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7fabd4043070 con 0x55f6b56be070 2024-08-28T19:38:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.231+0000 7fabe8dc3740 10 client.19671 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.231+0000 7fabe8dc3740 10 client.19671 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:38:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.231+0000 7fabe8dc3740 20 client.19671 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.231+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f6b567c9f0 0x55f6b5a89270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.231+0000 7fabe8dc3740 4 client.19671 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:38:48.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.231+0000 7fabe8dc3740 1 -- 172.21.15.16:0/1766581951 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6b59dc5d0 con 0x55f6b567c9f0 2024-08-28T19:38:48.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.231+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f6b567c9f0 0x55f6b5a89270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.232+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f6b567c9f0 0x55f6b5a89270 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73c4b59a1fcc0b82 server_cookie=83c55aee14966581 in_seq=0 out_seq=0 2024-08-28T19:38:48.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.232+0000 7fabbb7fe640 10 client.19671 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:38:48.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.234+0000 7fabbb7fe640 1 -- 172.21.15.16:0/1766581951 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f6b59dc5d0 con 0x55f6b567c9f0 2024-08-28T19:38:48.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.234+0000 7fabbb7fe640 10 client.19671 handle_command_reply: tid=0 2024-08-28T19:38:48.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.234+0000 7fabe8dc3740 1 client.19671 shutdown 2024-08-28T19:38:48.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabe8dc3740 1 -- 172.21.15.16:0/1766581951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 msgr2=0x55f6b5a83530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:48.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 0x55f6b5a83530 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009620 tx=0x7fabd4003d70 comp rx=0 tx=0).stop 2024-08-28T19:38:48.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabe8dc3740 1 -- 172.21.15.16:0/1766581951 shutdown_connections 2024-08-28T19:38:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f6b567c9f0 0x55f6b5a89270 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 0x55f6b5a83530 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b569d390 0x55f6b5a77710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/1766581951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56951b0 0x55f6b5a771d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabbb7fe640 0 client.19671 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:38:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabe8dc3740 1 -- 172.21.15.16:0/1766581951 >> 172.21.15.16:0/1766581951 conn(0x55f6b56b2460 msgr2=0x55f6b5a5f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabe8dc3740 1 -- 172.21.15.16:0/1766581951 shutdown_connections 2024-08-28T19:38:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabe8dc3740 1 -- 172.21.15.16:0/1766581951 wait complete. 2024-08-28T19:38:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.235+0000 7fabe8dc3740 20 client.19671 trim_cache size 0 max 16384 2024-08-28T19:38:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.255+0000 7fabe8dc3740 1 Processor -- start 2024-08-28T19:38:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.255+0000 7fabe8dc3740 1 -- start start 2024-08-28T19:38:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.255+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a77690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.255+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 0x55f6b5a684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.255+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56be070 0x55f6b5a68a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.255+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabc80100a0 con 0x55f6b569d390 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.255+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabc8011dc0 con 0x55f6b56be070 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.255+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fabc8010830 con 0x55f6b56951b0 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.256+0000 7fabdb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 0x55f6b5a684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.256+0000 7fabdb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 0x55f6b5a684c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60098/0 (socket says 172.21.15.16:60098) 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.256+0000 7fabdb7fe640 1 -- 172.21.15.16:0/5303083 learned_addr learned my addr 172.21.15.16:0/5303083 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.256+0000 7fabe0deb640 1 --2- 172.21.15.16:0/5303083 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a77690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.256+0000 7fabdb7fe640 1 -- 172.21.15.16:0/5303083 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 msgr2=0x55f6b5a77690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.256+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/5303083 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a77690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.256+0000 7fabdb7fe640 1 -- 172.21.15.16:0/5303083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56be070 msgr2=0x55f6b5a68a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.256+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/5303083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56be070 0x55f6b5a68a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.256+0000 7fabdb7fe640 1 -- 172.21.15.16:0/5303083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6b5a83be0 con 0x55f6b569d390 2024-08-28T19:38:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.256+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/5303083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 0x55f6b5a684c0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fabd00124c0 tx=0x7fabd000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e2482c26577b44b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.257+0000 7fabc1ffb640 1 -- 172.21.15.16:0/5303083 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd0021040 con 0x55f6b569d390 2024-08-28T19:38:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.257+0000 7fabc1ffb640 1 -- 172.21.15.16:0/5303083 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabd0010d90 con 0x55f6b569d390 2024-08-28T19:38:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.257+0000 7fabc1ffb640 1 -- 172.21.15.16:0/5303083 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd00054e0 con 0x55f6b569d390 2024-08-28T19:38:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.258+0000 7fabe8dc3740 1 -- 172.21.15.16:0/5303083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 msgr2=0x55f6b5a684c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.258+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/5303083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 0x55f6b5a684c0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fabd00124c0 tx=0x7fabd000b390 comp rx=0 tx=0).stop 2024-08-28T19:38:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.258+0000 7fabe8dc3740 1 -- 172.21.15.16:0/5303083 shutdown_connections 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.258+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/5303083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b56be070 0x55f6b5a68a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.258+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/5303083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b569d390 0x55f6b5a684c0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.258+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/5303083 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a77690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.258+0000 7fabe8dc3740 1 -- 172.21.15.16:0/5303083 >> 172.21.15.16:0/5303083 conn(0x55f6b56b2460 msgr2=0x55f6b5a76f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.258+0000 7fabe8dc3740 1 -- 172.21.15.16:0/5303083 shutdown_connections 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.258+0000 7fabe8dc3740 1 -- 172.21.15.16:0/5303083 wait complete. 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.258+0000 7fabe8dc3740 1 -- start start 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.258+0000 7fabe8dc3740 1 Processor -- start 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a7bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b569d390 0x55f6b5a7c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabe0deb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a7bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabe8dc3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 0x55f6b5aec9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabc8011630 con 0x55f6b56be070 2024-08-28T19:38:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabc8010f00 con 0x55f6b569d390 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabe8dc3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fabc800e0a0 con 0x55f6b56951b0 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabe0deb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a7bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55518/0 (socket says 172.21.15.16:55518) 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabe0deb640 1 -- 172.21.15.16:0/4195546015 learned_addr learned my addr 172.21.15.16:0/4195546015 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabe8dc3740 10 client.0 _ll_register_callbacks cb 0x7fabcc0e9430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b569d390 0x55f6b5a7c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.259+0000 7fabe0deb640 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 0x55f6b5aec9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.260+0000 7fabe0deb640 1 -- 172.21.15.16:0/4195546015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b569d390 msgr2=0x55f6b5a7c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.260+0000 7fabe0deb640 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b569d390 0x55f6b5a7c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.260+0000 7fabe0deb640 1 -- 172.21.15.16:0/4195546015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 msgr2=0x55f6b5aec9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.260+0000 7fabe0deb640 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 0x55f6b5aec9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.260+0000 7fabe0deb640 1 -- 172.21.15.16:0/4195546015 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6b5aed050 con 0x55f6b56951b0 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.260+0000 7fabe0deb640 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a7bd60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fabd402ade0 tx=0x7fabd402f3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=954ed7a88127f4e7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.260+0000 7fabbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:38:48.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.260+0000 7fabbb7fe640 1 -- 172.21.15.16:0/4195546015 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4059040 con 0x55f6b56951b0 2024-08-28T19:38:48.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.260+0000 7fabe8dc3740 1 -- 172.21.15.16:0/4195546015 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabd0010d90 con 0x55f6b56951b0 2024-08-28T19:38:48.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.261+0000 7fabbb7fe640 1 -- 172.21.15.16:0/4195546015 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7fabd4002030 con 0x55f6b56951b0 2024-08-28T19:38:48.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.261+0000 7fabbb7fe640 1 -- 172.21.15.16:0/4195546015 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fabd4009800 con 0x55f6b56951b0 2024-08-28T19:38:48.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.261+0000 7fabe8dc3740 10 client.28702 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:48.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.261+0000 7fabe8dc3740 1 -- 172.21.15.16:0/4195546015 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x55f6b5af4370 con 0x55f6b56951b0 2024-08-28T19:38:48.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.262+0000 7fabbb7fe640 1 -- 172.21.15.16:0/4195546015 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabd0010d90 con 0x55f6b56951b0 2024-08-28T19:38:48.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.262+0000 7fabbb7fe640 1 -- 172.21.15.16:0/4195546015 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7fabd4043070 con 0x55f6b56951b0 2024-08-28T19:38:48.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.263+0000 7fabe8dc3740 10 client.28702 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:48.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.263+0000 7fabe8dc3740 10 client.28702 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:38:48.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.263+0000 7fabe8dc3740 20 client.28702 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:48.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.263+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f6b5651da0 0x55f6b5af6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:48.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.263+0000 7fabe8dc3740 4 client.28702 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:38:48.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.263+0000 7fabe8dc3740 1 -- 172.21.15.16:0/4195546015 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f6b59d8070 con 0x55f6b5651da0 2024-08-28T19:38:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.264+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f6b5651da0 0x55f6b5af6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:48.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.266+0000 7fabdb7fe640 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f6b5651da0 0x55f6b5af6af0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f0bd02d64f1c6bd server_cookie=c197f2fdb7474cd8 in_seq=0 out_seq=0 2024-08-28T19:38:48.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.266+0000 7fabbb7fe640 10 client.28702 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:38:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabbb7fe640 1 -- 172.21.15.16:0/4195546015 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f6b59d8070 con 0x55f6b5651da0 2024-08-28T19:38:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabbb7fe640 10 client.28702 handle_command_reply: tid=0 2024-08-28T19:38:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabe8dc3740 1 client.28702 shutdown 2024-08-28T19:38:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabe8dc3740 1 -- 172.21.15.16:0/4195546015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 msgr2=0x55f6b5a7bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a7bd60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fabd402ade0 tx=0x7fabd402f3a0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabe8dc3740 1 -- 172.21.15.16:0/4195546015 shutdown_connections 2024-08-28T19:38:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55f6b5651da0 0x55f6b5af6af0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b56be070 0x55f6b5aec9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6b569d390 0x55f6b5a7c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabe8dc3740 1 --2- 172.21.15.16:0/4195546015 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f6b56951b0 0x55f6b5a7bd60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabe8dc3740 1 -- 172.21.15.16:0/4195546015 >> 172.21.15.16:0/4195546015 conn(0x55f6b56b2460 msgr2=0x55f6b5a75650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.270+0000 7fabbb7fe640 0 client.28702 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:38:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.271+0000 7fabe8dc3740 1 -- 172.21.15.16:0/4195546015 shutdown_connections 2024-08-28T19:38:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.271+0000 7fabe8dc3740 1 -- 172.21.15.16:0/4195546015 wait complete. 2024-08-28T19:38:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.271+0000 7fabe8dc3740 20 client.28702 trim_cache size 0 max 16384 2024-08-28T19:38:48.273 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:38:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 -- 172.21.15.16:0/2643974863 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabc40b4da0 msgr2=0x7fabc40b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabc40b4da0 0x7fabc40b72b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x55f6b5a61fd0 tx=0x7fabcc0e9560 comp rx=0 tx=0).stop 2024-08-28T19:38:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 -- 172.21.15.16:0/2643974863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabdc1047c0 msgr2=0x7fabdc19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabdc1047c0 0x7fabdc19a170 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00e9f0 tx=0x7fabcc00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 -- 172.21.15.16:0/2643974863 shutdown_connections 2024-08-28T19:38:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabdc10dbf0 0x7fabdc19f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fabc40b4da0 0x7fabc40b72b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabdc105110 0x7fabdc19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 --2- 172.21.15.16:0/2643974863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabdc1047c0 0x7fabdc19a170 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 -- 172.21.15.16:0/2643974863 >> 172.21.15.16:0/2643974863 conn(0x7fabdc079470 msgr2=0x7fabdc100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 -- 172.21.15.16:0/2643974863 shutdown_connections 2024-08-28T19:38:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:48.272+0000 7fabe284e640 1 -- 172.21.15.16:0/2643974863 wait complete. 2024-08-28T19:38:48.476 INFO:tasks.workunit.client.0.smithi016.stdout: 10 103964 24.57 MB/sec execute 144 sec latency 134.714 ms 2024-08-28T19:38:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:49.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:38:49.476 INFO:tasks.workunit.client.0.smithi016.stdout: 10 104758 24.71 MB/sec execute 145 sec latency 122.462 ms 2024-08-28T19:38:49.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:49 smithi016 ceph-mon[18556]: pgmap v894: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 10 MiB/s wr, 751 op/s 2024-08-28T19:38:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:49 smithi028 ceph-mon[21769]: pgmap v894: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 10 MiB/s wr, 751 op/s 2024-08-28T19:38:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:49 smithi115 ceph-mon[22184]: pgmap v894: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 10 MiB/s wr, 751 op/s 2024-08-28T19:38:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.953+0000 7f94052b6640 1 -- 172.21.15.16:0/1226654387 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc0318a0 con 0x7f9400111ce0 2024-08-28T19:38:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.954+0000 7f940851c640 1 -- 172.21.15.16:0/1226654387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9400111ce0 msgr2=0x7f940010a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.954+0000 7f940851c640 1 --2- 172.21.15.16:0/1226654387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9400111ce0 0x7f940010a160 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f93fc009900 tx=0x7f93fc02a400 comp rx=0 tx=0).stop 2024-08-28T19:38:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.955+0000 7f940851c640 1 -- 172.21.15.16:0/1226654387 shutdown_connections 2024-08-28T19:38:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.955+0000 7f940851c640 1 --2- 172.21.15.16:0/1226654387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f940011c9d0 0x7f940011eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.955+0000 7f940851c640 1 --2- 172.21.15.16:0/1226654387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f940010a780 0x7f940010ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.955+0000 7f940851c640 1 --2- 172.21.15.16:0/1226654387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9400111ce0 0x7f940010a160 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.955+0000 7f940851c640 1 -- 172.21.15.16:0/1226654387 >> 172.21.15.16:0/1226654387 conn(0x7f9400070c60 msgr2=0x7f9400071070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.955+0000 7f940851c640 1 -- 172.21.15.16:0/1226654387 shutdown_connections 2024-08-28T19:38:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.955+0000 7f940851c640 1 -- 172.21.15.16:0/1226654387 wait complete. 2024-08-28T19:38:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.956+0000 7f940851c640 1 Processor -- start 2024-08-28T19:38:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.956+0000 7f940851c640 1 -- start start 2024-08-28T19:38:49.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.956+0000 7f940851c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f940010a780 0x7f94001a9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:49.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.956+0000 7f940851c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9400111ce0 0x7f94001a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:49.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.956+0000 7f940851c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f940011c9d0 0x7f94001ae870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:49.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.956+0000 7f940851c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9400122690 con 0x7f9400111ce0 2024-08-28T19:38:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.956+0000 7f940851c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9400122510 con 0x7f940010a780 2024-08-28T19:38:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.956+0000 7f940851c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9400122810 con 0x7f940011c9d0 2024-08-28T19:38:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.957+0000 7f94062b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f940010a780 0x7f94001a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.957+0000 7f94062b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f940010a780 0x7f94001a9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37998/0 (socket says 172.21.15.16:37998) 2024-08-28T19:38:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.957+0000 7f94062b8640 1 -- 172.21.15.16:0/2146254073 learned_addr learned my addr 172.21.15.16:0/2146254073 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.957+0000 7f94062b8640 1 -- 172.21.15.16:0/2146254073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f940011c9d0 msgr2=0x7f94001ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.957+0000 7f94062b8640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f940011c9d0 0x7f94001ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.957+0000 7f94062b8640 1 -- 172.21.15.16:0/2146254073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9400111ce0 msgr2=0x7f94001a95d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-28T19:38:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.957+0000 7f94062b8640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9400111ce0 0x7f94001a95d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.957+0000 7f94062b8640 1 -- 172.21.15.16:0/2146254073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94001aefa0 con 0x7f940010a780 2024-08-28T19:38:49.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.958+0000 7f94062b8640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f940010a780 0x7f94001a9030 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f93fc005940 tx=0x7f93fc004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:49.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.959+0000 7f93f77fe640 1 -- 172.21.15.16:0/2146254073 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc03f030 con 0x7f940010a780 2024-08-28T19:38:49.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.959+0000 7f940851c640 1 -- 172.21.15.16:0/2146254073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94001af230 con 0x7f940010a780 2024-08-28T19:38:49.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.959+0000 7f940851c640 1 -- 172.21.15.16:0/2146254073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94001b59f0 con 0x7f940010a780 2024-08-28T19:38:49.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.960+0000 7f93f77fe640 1 -- 172.21.15.16:0/2146254073 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93fc003aa0 con 0x7f940010a780 2024-08-28T19:38:49.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.960+0000 7f93f77fe640 1 -- 172.21.15.16:0/2146254073 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc035070 con 0x7f940010a780 2024-08-28T19:38:49.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.961+0000 7f940851c640 1 -- 172.21.15.16:0/2146254073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e40052c0 con 0x7f940010a780 2024-08-28T19:38:49.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.964+0000 7f93f77fe640 1 -- 172.21.15.16:0/2146254073 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f93fc042030 con 0x7f940010a780 2024-08-28T19:38:49.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.965+0000 7f93f77fe640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f93ec0b4da0 0x7f93ec0b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:49.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.965+0000 7f93f77fe640 1 -- 172.21.15.16:0/2146254073 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f93fc106470 con 0x7f940010a780 2024-08-28T19:38:49.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.965+0000 7f9405ab7640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f93ec0b4da0 0x7f93ec0b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:49.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.966+0000 7f9405ab7640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f93ec0b4da0 0x7f93ec0b72b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f940011c110 tx=0x7f93f0005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:49.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:49.972+0000 7f93f77fe640 1 -- 172.21.15.16:0/2146254073 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f93fc049030 con 0x7f940010a780 2024-08-28T19:38:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:50.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.273+0000 7f940851c640 1 -- 172.21.15.16:0/2146254073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93e40055b0 con 0x7f940010a780 2024-08-28T19:38:50.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.274+0000 7f93f77fe640 1 -- 172.21.15.16:0/2146254073 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+11994 (secure 0 0 0) 0x7f93fc0cce30 con 0x7f940010a780 2024-08-28T19:38:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:38:50.276 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":96,"btime":"2024-08-28T19:38:38:408543+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":96,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:38:38.323875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:38:50.276 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 96 2024-08-28T19:38:50.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.278+0000 7f940851c640 1 -- 172.21.15.16:0/2146254073 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f93ec0b4da0 msgr2=0x7f93ec0b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:50.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.278+0000 7f940851c640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f93ec0b4da0 0x7f93ec0b72b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f940011c110 tx=0x7f93f0005f90 comp rx=0 tx=0).stop 2024-08-28T19:38:50.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.278+0000 7f940851c640 1 -- 172.21.15.16:0/2146254073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f940010a780 msgr2=0x7f94001a9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:50.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.278+0000 7f940851c640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f940010a780 0x7f94001a9030 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f93fc005940 tx=0x7f93fc004270 comp rx=0 tx=0).stop 2024-08-28T19:38:50.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.278+0000 7f940851c640 1 -- 172.21.15.16:0/2146254073 shutdown_connections 2024-08-28T19:38:50.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.278+0000 7f940851c640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f940011c9d0 0x7f94001ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:50.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.278+0000 7f940851c640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9400111ce0 0x7f94001a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:50.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.278+0000 7f940851c640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f93ec0b4da0 0x7f93ec0b72b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:50.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.278+0000 7f940851c640 1 --2- 172.21.15.16:0/2146254073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f940010a780 0x7f94001a9030 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:50.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.278+0000 7f940851c640 1 -- 172.21.15.16:0/2146254073 >> 172.21.15.16:0/2146254073 conn(0x7f9400070c60 msgr2=0x7f940011e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:50.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.278+0000 7f940851c640 1 -- 172.21.15.16:0/2146254073 shutdown_connections 2024-08-28T19:38:50.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.279+0000 7f940851c640 1 -- 172.21.15.16:0/2146254073 wait complete. 2024-08-28T19:38:50.476 INFO:tasks.workunit.client.0.smithi016.stdout: 10 105440 24.79 MB/sec execute 146 sec latency 236.926 ms 2024-08-28T19:38:50.521 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:50 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2146254073' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:38:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:50 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2146254073' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:38:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:50 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2146254073' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:38:51.263 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 141.8 --await-for 45.0' 2024-08-28T19:38:51.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 141.8 --await-for 45.0 2024-08-28T19:38:51.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.513+0000 7f7b34139640 1 -- 172.21.15.16:0/1480219315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b2c108a90 msgr2=0x7f7b2c10f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:51.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.513+0000 7f7b34139640 1 --2- 172.21.15.16:0/1480219315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b2c108a90 0x7f7b2c10f2c0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009ca0 tx=0x7f7b2402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.514+0000 7f7b34139640 1 -- 172.21.15.16:0/1480219315 shutdown_connections 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.514+0000 7f7b34139640 1 --2- 172.21.15.16:0/1480219315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b2c108a90 0x7f7b2c10f2c0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.514+0000 7f7b34139640 1 --2- 172.21.15.16:0/1480219315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b2c108140 0x7f7b2c108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.514+0000 7f7b34139640 1 --2- 172.21.15.16:0/1480219315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2c102800 0x7f7b2c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.514+0000 7f7b34139640 1 -- 172.21.15.16:0/1480219315 >> 172.21.15.16:0/1480219315 conn(0x7f7b2c0794d0 msgr2=0x7f7b2c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.514+0000 7f7b34139640 1 -- 172.21.15.16:0/1480219315 shutdown_connections 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.514+0000 7f7b34139640 1 -- 172.21.15.16:0/1480219315 wait complete. 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.515+0000 7f7b34139640 1 Processor -- start 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.515+0000 7f7b34139640 1 -- start start 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.515+0000 7f7b34139640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b2c102800 0x7f7b2c19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.515+0000 7f7b34139640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b2c108140 0x7f7b2c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.515+0000 7f7b34139640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2c108a90 0x7f7b2c19fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.515+0000 7f7b34139640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c112760 con 0x7f7b2c108140 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.515+0000 7f7b34139640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1125e0 con 0x7f7b2c108a90 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.515+0000 7f7b34139640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1128e0 con 0x7f7b2c102800 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.515+0000 7f7b316d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b2c108140 0x7f7b2c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b316d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b2c108140 0x7f7b2c19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60136/0 (socket says 172.21.15.16:60136) 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b316d4640 1 -- 172.21.15.16:0/1627668427 learned_addr learned my addr 172.21.15.16:0/1627668427 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b31ed5640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b2c102800 0x7f7b2c19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b326d6640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2c108a90 0x7f7b2c19fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b316d4640 1 -- 172.21.15.16:0/1627668427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b2c102800 msgr2=0x7f7b2c19a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b316d4640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b2c102800 0x7f7b2c19a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b316d4640 1 -- 172.21.15.16:0/1627668427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2c108a90 msgr2=0x7f7b2c19fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b316d4640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2c108a90 0x7f7b2c19fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b316d4640 1 -- 172.21.15.16:0/1627668427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b2c1a0150 con 0x7f7b2c108140 2024-08-28T19:38:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b316d4640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b2c108140 0x7f7b2c19a780 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00e9f0 tx=0x7f7b1c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b22ffd640 1 -- 172.21.15.16:0/1627668427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c00ccc0 con 0x7f7b2c108140 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b34139640 1 -- 172.21.15.16:0/1627668427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b2c1a0440 con 0x7f7b2c108140 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b34139640 1 -- 172.21.15.16:0/1627668427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b2c1a6ba0 con 0x7f7b2c108140 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b22ffd640 1 -- 172.21.15.16:0/1627668427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b1c0054d0 con 0x7f7b2c108140 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.516+0000 7f7b22ffd640 1 -- 172.21.15.16:0/1627668427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c010430 con 0x7f7b2c108140 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.519+0000 7f7b22ffd640 1 -- 172.21.15.16:0/1627668427 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7b1c0105d0 con 0x7f7b2c108140 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.519+0000 7f7b3a6ae740 1 Processor -- start 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.520+0000 7f7b3a6ae740 1 -- start start 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.520+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff788b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.520+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74bd1b0 0x563ff788b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.520+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74c5390 0x563ff788bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.520+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563ff7802680 con 0x563ff74e6070 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.520+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ff77f1c10 con 0x563ff74c5390 2024-08-28T19:38:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.520+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563ff77f5bc0 con 0x563ff74bd1b0 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.520+0000 7f7b326d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff788b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.520+0000 7f7b326d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff788b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60144/0 (socket says 172.21.15.16:60144) 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.520+0000 7f7b326d6640 1 -- 172.21.15.16:0/2587591822 learned_addr learned my addr 172.21.15.16:0/2587591822 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.520+0000 7f7b22ffd640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7b180b4ca0 0x7f7b180b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b31ed5640 1 --2- 172.21.15.16:0/2587591822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74bd1b0 0x563ff788b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b326d6640 1 -- 172.21.15.16:0/2587591822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74bd1b0 msgr2=0x563ff788b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b326d6640 1 --2- 172.21.15.16:0/2587591822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74bd1b0 0x563ff788b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b326d6640 1 -- 172.21.15.16:0/2587591822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74c5390 msgr2=0x563ff788bd90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b326d6640 1 --2- 172.21.15.16:0/2587591822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74c5390 0x563ff788bd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b326d6640 1 -- 172.21.15.16:0/2587591822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ff7551650 con 0x563ff74e6070 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b22ffd640 1 -- 172.21.15.16:0/1627668427 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f7b1c0e5cd0 con 0x7f7b2c108140 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b326d6640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7b180b4ca0 0x7f7b180b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b326d6640 1 --2- 172.21.15.16:0/2587591822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff788b310 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009ca0 tx=0x7f7b2402f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be424068e627e1c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b17fff640 1 -- 172.21.15.16:0/2587591822 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b24042040 con 0x563ff74e6070 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b17fff640 1 -- 172.21.15.16:0/2587591822 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b24036070 con 0x563ff74e6070 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.521+0000 7f7b17fff640 1 -- 172.21.15.16:0/2587591822 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b240047e0 con 0x563ff74e6070 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.522+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/2587591822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 msgr2=0x563ff788b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.522+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/2587591822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff788b310 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009ca0 tx=0x7f7b2402f310 comp rx=0 tx=0).stop 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.523+0000 7f7b326d6640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7b180b4ca0 0x7f7b180b71b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7b2404d000 tx=0x7f7b2404e000 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.523+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/2587591822 shutdown_connections 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.523+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/2587591822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74c5390 0x563ff788bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.523+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/2587591822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74bd1b0 0x563ff788b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.523+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/2587591822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff788b310 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.523+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/2587591822 >> 172.21.15.16:0/2587591822 conn(0x563ff74da460 msgr2=0x563ff7889200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.523+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/2587591822 shutdown_connections 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.523+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/2587591822 wait complete. 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.524+0000 7f7b3a6ae740 1 -- start start 2024-08-28T19:38:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.524+0000 7f7b3a6ae740 1 Processor -- start 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.525+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74bd1b0 0x563ff789ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.525+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74c5390 0x563ff789b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.525+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78a71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.525+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563ff7654700 con 0x563ff74e6070 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.525+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ff7778750 con 0x563ff74bd1b0 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.525+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563ff789de50 con 0x563ff74c5390 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.525+0000 7f7b3a6ae740 10 client.0 _ll_register_callbacks cb 0x563ff77a7e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.525+0000 7f7b326d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78a71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.525+0000 7f7b326d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78a71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60148/0 (socket says 172.21.15.16:60148) 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.525+0000 7f7b326d6640 1 -- 172.21.15.16:0/128601041 learned_addr learned my addr 172.21.15.16:0/128601041 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.525+0000 7f7b31ed5640 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74bd1b0 0x563ff789ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.526+0000 7f7b316d4640 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74c5390 0x563ff789b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.526+0000 7f7b326d6640 1 -- 172.21.15.16:0/128601041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74c5390 msgr2=0x563ff789b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.526+0000 7f7b326d6640 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74c5390 0x563ff789b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.526+0000 7f7b326d6640 1 -- 172.21.15.16:0/128601041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74bd1b0 msgr2=0x563ff789ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.526+0000 7f7b326d6640 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74bd1b0 0x563ff789ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.526+0000 7f7b326d6640 1 -- 172.21.15.16:0/128601041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ff77b1900 con 0x563ff74e6070 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.526+0000 7f7b326d6640 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78a71b0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f7b24005e30 tx=0x7f7b240095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834a5cd50101e53 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.526+0000 7f7b157fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.526+0000 7f7b157fa640 1 -- 172.21.15.16:0/128601041 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b24058070 con 0x563ff74e6070 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.527+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/128601041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b24036070 con 0x563ff74e6070 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.527+0000 7f7b157fa640 1 -- 172.21.15.16:0/128601041 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f7b24003970 con 0x563ff74e6070 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.527+0000 7f7b157fa640 1 -- 172.21.15.16:0/128601041 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f7b24064050 con 0x563ff74e6070 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.527+0000 7f7b3a6ae740 10 client.19692 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.527+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/128601041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x563ff78a9c30 con 0x563ff74e6070 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.527+0000 7f7b157fa640 1 -- 172.21.15.16:0/128601041 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b24036070 con 0x563ff74e6070 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.527+0000 7f7b157fa640 1 -- 172.21.15.16:0/128601041 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7f7b240305c0 con 0x563ff74e6070 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.528+0000 7f7b3a6ae740 10 client.19692 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.528+0000 7f7b3a6ae740 10 client.19692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.528+0000 7f7b3a6ae740 20 client.19692 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.528+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563ff74a49f0 0x563ff78acf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.528+0000 7f7b3a6ae740 4 client.19692 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.528+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/128601041 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ff78045d0 con 0x563ff74a49f0 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.528+0000 7f7b31ed5640 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563ff74a49f0 0x563ff78acf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.528+0000 7f7b31ed5640 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563ff74a49f0 0x563ff78acf90 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdf2d0aea8fcaf9e server_cookie=399754a61610285b in_seq=0 out_seq=0 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.528+0000 7f7b157fa640 10 client.19692 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.533+0000 7f7b157fa640 1 -- 172.21.15.16:0/128601041 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563ff78045d0 con 0x563ff74a49f0 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.533+0000 7f7b157fa640 10 client.19692 handle_command_reply: tid=0 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.533+0000 7f7b3a6ae740 1 client.19692 shutdown 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.533+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/128601041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 msgr2=0x563ff78a71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.533+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78a71b0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f7b24005e30 tx=0x7f7b240095c0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.533+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/128601041 shutdown_connections 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.534+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78a71b0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.534+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563ff74a49f0 0x563ff78acf90 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.534+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74c5390 0x563ff789b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.534+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/128601041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74bd1b0 0x563ff789ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.534+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/128601041 >> 172.21.15.16:0/128601041 conn(0x563ff74da460 msgr2=0x563ff78877c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.534+0000 7f7b157fa640 0 client.19692 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.534+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/128601041 shutdown_connections 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.534+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/128601041 wait complete. 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.534+0000 7f7b3a6ae740 20 client.19692 trim_cache size 0 max 16384 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b3a6ae740 1 Processor -- start 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b3a6ae740 1 -- start start 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74bd1b0 0x563ff7889250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74c5390 0x563ff78a7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78904b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1aea10 con 0x563ff74e6070 2024-08-28T19:38:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1b0730 con 0x563ff74c5390 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1af1a0 con 0x563ff74bd1b0 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b326d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78904b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b326d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78904b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60160/0 (socket says 172.21.15.16:60160) 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b326d6640 1 -- 172.21.15.16:0/312020117 learned_addr learned my addr 172.21.15.16:0/312020117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b316d4640 1 --2- 172.21.15.16:0/312020117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74c5390 0x563ff78a7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.553+0000 7f7b31ed5640 1 --2- 172.21.15.16:0/312020117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74bd1b0 0x563ff7889250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.554+0000 7f7b326d6640 1 -- 172.21.15.16:0/312020117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74bd1b0 msgr2=0x563ff7889250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.554+0000 7f7b326d6640 1 --2- 172.21.15.16:0/312020117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74bd1b0 0x563ff7889250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.554+0000 7f7b326d6640 1 -- 172.21.15.16:0/312020117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74c5390 msgr2=0x563ff78a7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.554+0000 7f7b326d6640 1 --2- 172.21.15.16:0/312020117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74c5390 0x563ff78a7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.554+0000 7f7b326d6640 1 -- 172.21.15.16:0/312020117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ff789d5a0 con 0x563ff74e6070 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.554+0000 7f7b316d4640 1 --2- 172.21.15.16:0/312020117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74c5390 0x563ff78a7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.554+0000 7f7b326d6640 1 --2- 172.21.15.16:0/312020117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78904b0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f7b24030ff0 tx=0x7f7b24009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cef7c4a69d6798e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.554+0000 7f7b177fe640 1 -- 172.21.15.16:0/312020117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b24058070 con 0x563ff74e6070 2024-08-28T19:38:51.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.554+0000 7f7b177fe640 1 -- 172.21.15.16:0/312020117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b24036070 con 0x563ff74e6070 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.554+0000 7f7b177fe640 1 -- 172.21.15.16:0/312020117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b24042040 con 0x563ff74e6070 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.556+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/312020117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 msgr2=0x563ff78904b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.556+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/312020117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78904b0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f7b24030ff0 tx=0x7f7b24009430 comp rx=0 tx=0).stop 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.556+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/312020117 shutdown_connections 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.556+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/312020117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74e6070 0x563ff78904b0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.556+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/312020117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74c5390 0x563ff78a7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.556+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/312020117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74bd1b0 0x563ff7889250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.556+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/312020117 >> 172.21.15.16:0/312020117 conn(0x563ff74da460 msgr2=0x563ff7888b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.556+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/312020117 shutdown_connections 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.556+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/312020117 wait complete. 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.557+0000 7f7b3a6ae740 1 -- start start 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.557+0000 7f7b3a6ae740 1 Processor -- start 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74bd1b0 0x563ff78b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74c5390 0x563ff78944a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b3a6ae740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74e6070 0x563ff78949e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1affa0 con 0x563ff74c5390 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1af870 con 0x563ff74bd1b0 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b3a6ae740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1aca10 con 0x563ff74e6070 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b3a6ae740 10 client.0 _ll_register_callbacks cb 0x7f7b24009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b316d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74c5390 0x563ff78944a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b326d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74e6070 0x563ff78949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b326d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74e6070 0x563ff78949e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55602/0 (socket says 172.21.15.16:55602) 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b326d6640 1 -- 172.21.15.16:0/460093601 learned_addr learned my addr 172.21.15.16:0/460093601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b316d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74c5390 0x563ff78944a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60174/0 (socket says 172.21.15.16:60174) 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.558+0000 7f7b31ed5640 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74bd1b0 0x563ff78b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.559+0000 7f7b326d6640 1 -- 172.21.15.16:0/460093601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74bd1b0 msgr2=0x563ff78b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.559+0000 7f7b326d6640 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74bd1b0 0x563ff78b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.559+0000 7f7b326d6640 1 -- 172.21.15.16:0/460093601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74c5390 msgr2=0x563ff78944a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.559+0000 7f7b326d6640 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74c5390 0x563ff78944a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.559+0000 7f7b326d6640 1 -- 172.21.15.16:0/460093601 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ff7894f20 con 0x563ff74e6070 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.560+0000 7f7b326d6640 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74e6070 0x563ff78949e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009a60 tx=0x7f7b24030920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=314bcb9c891aa69a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.560+0000 7f7b157fa640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.563+0000 7f7b157fa640 1 -- 172.21.15.16:0/460093601 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2407e060 con 0x563ff74e6070 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.563+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/460093601 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b24036070 con 0x563ff74e6070 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.563+0000 7f7b157fa640 1 -- 172.21.15.16:0/460093601 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f7b24003d80 con 0x563ff74e6070 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.564+0000 7f7b157fa640 1 -- 172.21.15.16:0/460093601 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f7b24085050 con 0x563ff74e6070 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.564+0000 7f7b3a6ae740 10 client.28720 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.564+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/460093601 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x563ff78acc60 con 0x563ff74e6070 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.564+0000 7f7b157fa640 1 -- 172.21.15.16:0/460093601 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b24036070 con 0x563ff74e6070 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.564+0000 7f7b157fa640 1 -- 172.21.15.16:0/460093601 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7f7b24051440 con 0x563ff74e6070 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.564+0000 7f7b3a6ae740 10 client.28720 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:51.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.564+0000 7f7b3a6ae740 10 client.28720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.564+0000 7f7b3a6ae740 20 client.28720 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.564+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563ff7479da0 0x563ff791e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.564+0000 7f7b3a6ae740 4 client.28720 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.564+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/460093601 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ff7800070 con 0x563ff7479da0 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.565+0000 7f7b31ed5640 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563ff7479da0 0x563ff791e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.566+0000 7f7b31ed5640 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563ff7479da0 0x563ff791e3a0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47ce812a6dcb660a server_cookie=5b02214365a48bf6 in_seq=0 out_seq=0 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.566+0000 7f7b157fa640 10 client.28720 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.571+0000 7f7b157fa640 1 -- 172.21.15.16:0/460093601 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ff7800070 con 0x563ff7479da0 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.571+0000 7f7b157fa640 10 client.28720 handle_command_reply: tid=0 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.571+0000 7f7b3a6ae740 1 client.28720 shutdown 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.572+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/460093601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74e6070 msgr2=0x563ff78949e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:51.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.572+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74e6070 0x563ff78949e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009a60 tx=0x7f7b24030920 comp rx=0 tx=0).stop 2024-08-28T19:38:51.350 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:38:51.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.572+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/460093601 shutdown_connections 2024-08-28T19:38:51.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.572+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563ff7479da0 0x563ff791e3a0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.572+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563ff74e6070 0x563ff78949e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.572+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563ff74c5390 0x563ff78944a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.572+0000 7f7b3a6ae740 1 --2- 172.21.15.16:0/460093601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ff74bd1b0 0x563ff78b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.572+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/460093601 >> 172.21.15.16:0/460093601 conn(0x563ff74da460 msgr2=0x563ff78904b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:51.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.573+0000 7f7b157fa640 0 client.28720 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.573+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/460093601 shutdown_connections 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.573+0000 7f7b3a6ae740 1 -- 172.21.15.16:0/460093601 wait complete. 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.573+0000 7f7b3a6ae740 20 client.28720 trim_cache size 0 max 16384 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.580+0000 7f7b34139640 1 -- 172.21.15.16:0/1627668427 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7b180b4ca0 msgr2=0x7f7b180b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.580+0000 7f7b34139640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7b180b4ca0 0x7f7b180b71b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7b2404d000 tx=0x7f7b2404e000 comp rx=0 tx=0).stop 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.580+0000 7f7b34139640 1 -- 172.21.15.16:0/1627668427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b2c108140 msgr2=0x7f7b2c19a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.580+0000 7f7b34139640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b2c108140 0x7f7b2c19a780 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00e9f0 tx=0x7f7b1c00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.580+0000 7f7b34139640 1 -- 172.21.15.16:0/1627668427 shutdown_connections 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.580+0000 7f7b34139640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7b180b4ca0 0x7f7b180b71b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.580+0000 7f7b34139640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2c108a90 0x7f7b2c19fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.580+0000 7f7b34139640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b2c108140 0x7f7b2c19a780 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.580+0000 7f7b34139640 1 --2- 172.21.15.16:0/1627668427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b2c102800 0x7f7b2c19a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.580+0000 7f7b34139640 1 -- 172.21.15.16:0/1627668427 >> 172.21.15.16:0/1627668427 conn(0x7f7b2c0794d0 msgr2=0x7f7b2c06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.581+0000 7f7b34139640 1 -- 172.21.15.16:0/1627668427 shutdown_connections 2024-08-28T19:38:51.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:50.581+0000 7f7b34139640 1 -- 172.21.15.16:0/1627668427 wait complete. 2024-08-28T19:38:51.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106033 24.85 MB/sec execute 147 sec latency 139.002 ms 2024-08-28T19:38:51.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:51 smithi016 ceph-mon[18556]: pgmap v895: 129 pgs: 129 active+clean; 1.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 7.5 MiB/s wr, 655 op/s 2024-08-28T19:38:51.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:51 smithi028 ceph-mon[21769]: pgmap v895: 129 pgs: 129 active+clean; 1.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 7.5 MiB/s wr, 655 op/s 2024-08-28T19:38:51.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:51 smithi115 ceph-mon[22184]: pgmap v895: 129 pgs: 129 active+clean; 1.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 7.5 MiB/s wr, 655 op/s 2024-08-28T19:38:52.050 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:52.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106736 24.94 MB/sec execute 148 sec latency 70.393 ms 2024-08-28T19:38:52.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.602+0000 7f88637fe640 1 -- 172.21.15.16:0/2093679941 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850030850 con 0x7f88641078e0 2024-08-28T19:38:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.603+0000 7f886b1c4640 1 -- 172.21.15.16:0/2093679941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88641078e0 msgr2=0x7f8864107cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.603+0000 7f886b1c4640 1 --2- 172.21.15.16:0/2093679941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88641078e0 0x7f8864107cf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8850009860 tx=0x7f885002a340 comp rx=0 tx=0).stop 2024-08-28T19:38:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.604+0000 7f886b1c4640 1 -- 172.21.15.16:0/2093679941 shutdown_connections 2024-08-28T19:38:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.604+0000 7f886b1c4640 1 --2- 172.21.15.16:0/2093679941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8864108230 0x7f886410ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.604+0000 7f886b1c4640 1 --2- 172.21.15.16:0/2093679941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88641078e0 0x7f8864107cf0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.604+0000 7f886b1c4640 1 --2- 172.21.15.16:0/2093679941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8864101fa0 0x7f88641023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.604+0000 7f886b1c4640 1 -- 172.21.15.16:0/2093679941 >> 172.21.15.16:0/2093679941 conn(0x7f8864079cf0 msgr2=0x7f886407a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.605+0000 7f886b1c4640 1 -- 172.21.15.16:0/2093679941 shutdown_connections 2024-08-28T19:38:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.605+0000 7f886b1c4640 1 -- 172.21.15.16:0/2093679941 wait complete. 2024-08-28T19:38:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.606+0000 7f886b1c4640 1 Processor -- start 2024-08-28T19:38:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.606+0000 7f886b1c4640 1 -- start start 2024-08-28T19:38:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.606+0000 7f886b1c4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8864101fa0 0x7f88641130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.606+0000 7f886b1c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641078e0 0x7f886406ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.606+0000 7f886b1c4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8864108230 0x7f886406f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.606+0000 7f886b1c4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8864111f40 con 0x7f8864101fa0 2024-08-28T19:38:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.606+0000 7f886b1c4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8864111dc0 con 0x7f88641078e0 2024-08-28T19:38:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.606+0000 7f886b1c4640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f88641120c0 con 0x7f8864108230 2024-08-28T19:38:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.607+0000 7f8869761640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8864108230 0x7f886406f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.607+0000 7f8869761640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8864108230 0x7f886406f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55626/0 (socket says 172.21.15.16:55626) 2024-08-28T19:38:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.607+0000 7f8869761640 1 -- 172.21.15.16:0/3424471302 learned_addr learned my addr 172.21.15.16:0/3424471302 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.607+0000 7f8869761640 1 -- 172.21.15.16:0/3424471302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641078e0 msgr2=0x7f886406ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.607+0000 7f8869761640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641078e0 0x7f886406ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.607+0000 7f8869761640 1 -- 172.21.15.16:0/3424471302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8864101fa0 msgr2=0x7f88641130c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.607+0000 7f8869761640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8864101fa0 0x7f88641130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.607+0000 7f8869761640 1 -- 172.21.15.16:0/3424471302 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886406fa00 con 0x7f8864108230 2024-08-28T19:38:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.609+0000 7f8869761640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8864108230 0x7f886406f2d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f885c00eaa0 tx=0x7f885c00ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.609+0000 7f8861ffb640 1 -- 172.21.15.16:0/3424471302 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c00cdb0 con 0x7f8864108230 2024-08-28T19:38:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.610+0000 7f886b1c4640 1 -- 172.21.15.16:0/3424471302 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8864073de0 con 0x7f8864108230 2024-08-28T19:38:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.610+0000 7f886b1c4640 1 -- 172.21.15.16:0/3424471302 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8864074320 con 0x7f8864108230 2024-08-28T19:38:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.610+0000 7f8861ffb640 1 -- 172.21.15.16:0/3424471302 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f885c005020 con 0x7f8864108230 2024-08-28T19:38:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.610+0000 7f8861ffb640 1 -- 172.21.15.16:0/3424471302 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c00cdb0 con 0x7f8864108230 2024-08-28T19:38:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.614+0000 7f8871739740 1 Processor -- start 2024-08-28T19:38:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.615+0000 7f8861ffb640 1 -- 172.21.15.16:0/3424471302 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f885c004b20 con 0x7f8864108230 2024-08-28T19:38:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.616+0000 7f8861ffb640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f88540b50a0 0x7f88540b75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.616+0000 7f8871739740 1 -- start start 2024-08-28T19:38:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.616+0000 7f8861ffb640 1 -- 172.21.15.16:0/3424471302 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f885c00cbb0 con 0x7f8864108230 2024-08-28T19:38:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.617+0000 7f8863fff640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f88540b50a0 0x7f88540b75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.618+0000 7f8863fff640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f88540b50a0 0x7f88540b75b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f885002a860 tx=0x7f8850009520 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.619+0000 7f8871739740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d843840 0x56168dbeac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.619+0000 7f8871739740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d81a980 0x56168dbeb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.619+0000 7f8871739740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d822b60 0x56168dbeb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.619+0000 7f8871739740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56168db5f330 con 0x56168d843840 2024-08-28T19:38:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.619+0000 7f8871739740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56168db4e8c0 con 0x56168d822b60 2024-08-28T19:38:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.619+0000 7f8871739740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56168db52870 con 0x56168d81a980 2024-08-28T19:38:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.619+0000 7f8868f60640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d843840 0x56168dbeac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.619+0000 7f8868f60640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d843840 0x56168dbeac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60198/0 (socket says 172.21.15.16:60198) 2024-08-28T19:38:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.619+0000 7f8868f60640 1 -- 172.21.15.16:0/4179141171 learned_addr learned my addr 172.21.15.16:0/4179141171 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.620+0000 7f8868f60640 1 -- 172.21.15.16:0/4179141171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d81a980 msgr2=0x56168dbeb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.620+0000 7f8868f60640 1 --2- 172.21.15.16:0/4179141171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d81a980 0x56168dbeb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.620+0000 7f8868f60640 1 -- 172.21.15.16:0/4179141171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d822b60 msgr2=0x56168dbeb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.620+0000 7f8868f60640 1 --2- 172.21.15.16:0/4179141171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d822b60 0x56168dbeb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.620+0000 7f8868f60640 1 -- 172.21.15.16:0/4179141171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56168db06df0 con 0x56168d843840 2024-08-28T19:38:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.620+0000 7f8868f60640 1 --2- 172.21.15.16:0/4179141171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d843840 0x56168dbeac90 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f8858004650 tx=0x7f885800e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b29658d2921b62 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.621+0000 7f884effd640 1 -- 172.21.15.16:0/4179141171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885800ed80 con 0x56168d843840 2024-08-28T19:38:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.621+0000 7f884effd640 1 -- 172.21.15.16:0/4179141171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88580054d0 con 0x56168d843840 2024-08-28T19:38:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.622+0000 7f8871739740 1 -- 172.21.15.16:0/4179141171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d843840 msgr2=0x56168dbeac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.622+0000 7f8871739740 1 --2- 172.21.15.16:0/4179141171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d843840 0x56168dbeac90 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f8858004650 tx=0x7f885800e350 comp rx=0 tx=0).stop 2024-08-28T19:38:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.624+0000 7f8871739740 1 -- 172.21.15.16:0/4179141171 shutdown_connections 2024-08-28T19:38:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.624+0000 7f8871739740 1 --2- 172.21.15.16:0/4179141171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d822b60 0x56168dbeb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.624+0000 7f8871739740 1 --2- 172.21.15.16:0/4179141171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d81a980 0x56168dbeb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.624+0000 7f8871739740 1 --2- 172.21.15.16:0/4179141171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d843840 0x56168dbeac90 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.624+0000 7f8871739740 1 -- 172.21.15.16:0/4179141171 >> 172.21.15.16:0/4179141171 conn(0x56168d837990 msgr2=0x56168dbe8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.624+0000 7f8871739740 1 -- 172.21.15.16:0/4179141171 shutdown_connections 2024-08-28T19:38:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.624+0000 7f8871739740 1 -- 172.21.15.16:0/4179141171 wait complete. 2024-08-28T19:38:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.625+0000 7f8871739740 1 -- start start 2024-08-28T19:38:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.625+0000 7f8871739740 1 Processor -- start 2024-08-28T19:38:52.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.625+0000 7f8871739740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d81a980 0x56168dbfeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.625+0000 7f8871739740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d822b60 0x56168dbff0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.625+0000 7f8871739740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc0b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.625+0000 7f8871739740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56168d9b17a0 con 0x56168d822b60 2024-08-28T19:38:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.625+0000 7f8871739740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56168dad5240 con 0x56168d81a980 2024-08-28T19:38:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.625+0000 7f8871739740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56168dc01bf0 con 0x56168d843840 2024-08-28T19:38:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.626+0000 7f8871739740 10 client.0 _ll_register_callbacks cb 0x56168d85b090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.626+0000 7f8869761640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc0b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.626+0000 7f8869761640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc0b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55656/0 (socket says 172.21.15.16:55656) 2024-08-28T19:38:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.626+0000 7f8869761640 1 -- 172.21.15.16:0/640076702 learned_addr learned my addr 172.21.15.16:0/640076702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:52.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.628+0000 7f8868f60640 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d81a980 0x56168dbfeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.628+0000 7f8869761640 1 -- 172.21.15.16:0/640076702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d81a980 msgr2=0x56168dbfeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:52.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.628+0000 7f8869761640 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d81a980 0x56168dbfeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.628+0000 7f8869761640 1 -- 172.21.15.16:0/640076702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d822b60 msgr2=0x56168dbff0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:52.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.628+0000 7f8869761640 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d822b60 0x56168dbff0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.628+0000 7f8869761640 1 -- 172.21.15.16:0/640076702 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56168db0b2b0 con 0x56168d843840 2024-08-28T19:38:52.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.629+0000 7f8869761640 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc0b000 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f885c0e2dc0 tx=0x7f885c0abe60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=667d07279acef94a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:52.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.629+0000 7f8843fff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:38:52.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.629+0000 7f8843fff640 1 -- 172.21.15.16:0/640076702 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c01f040 con 0x56168d843840 2024-08-28T19:38:52.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.629+0000 7f8871739740 1 -- 172.21.15.16:0/640076702 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88580054d0 con 0x56168d843840 2024-08-28T19:38:52.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.629+0000 7f8843fff640 1 -- 172.21.15.16:0/640076702 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f885c004e10 con 0x56168d843840 2024-08-28T19:38:52.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.630+0000 7f8843fff640 1 -- 172.21.15.16:0/640076702 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f885c0affd0 con 0x56168d843840 2024-08-28T19:38:52.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.630+0000 7f8871739740 10 client.28729 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:52.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.630+0000 7f8871739740 1 -- 172.21.15.16:0/640076702 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x56168d8bdb90 con 0x56168d843840 2024-08-28T19:38:52.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.631+0000 7f8843fff640 1 -- 172.21.15.16:0/640076702 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88580054d0 con 0x56168d843840 2024-08-28T19:38:52.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.631+0000 7f8843fff640 1 -- 172.21.15.16:0/640076702 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7f885c00f040 con 0x56168d843840 2024-08-28T19:38:52.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.631+0000 7f8871739740 10 client.28729 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:52.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.631+0000 7f8871739740 10 client.28729 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:38:52.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.631+0000 7f8871739740 20 client.28729 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:52.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.632+0000 7f8871739740 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56168d8021c0 0x56168dc10ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.632+0000 7f8871739740 4 client.28729 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:38:52.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.632+0000 7f8871739740 1 -- 172.21.15.16:0/640076702 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56168db61280 con 0x56168d8021c0 2024-08-28T19:38:52.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.632+0000 7f8868f60640 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56168d8021c0 0x56168dc10ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.633+0000 7f8868f60640 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56168d8021c0 0x56168dc10ec0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd61d7a5efa5ab25 server_cookie=b09b96c9ab257517 in_seq=0 out_seq=0 2024-08-28T19:38:52.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.633+0000 7f8843fff640 10 client.28729 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:52.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.636+0000 7f8843fff640 1 -- 172.21.15.16:0/640076702 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56168db61280 con 0x56168d8021c0 2024-08-28T19:38:52.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.636+0000 7f8843fff640 10 client.28729 handle_command_reply: tid=0 2024-08-28T19:38:52.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.636+0000 7f8871739740 1 client.28729 shutdown 2024-08-28T19:38:52.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8871739740 1 -- 172.21.15.16:0/640076702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 msgr2=0x56168dc0b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8871739740 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc0b000 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f885c0e2dc0 tx=0x7f885c0abe60 comp rx=0 tx=0).stop 2024-08-28T19:38:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8871739740 1 -- 172.21.15.16:0/640076702 shutdown_connections 2024-08-28T19:38:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8871739740 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56168d8021c0 0x56168dc10ec0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8871739740 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc0b000 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8871739740 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d822b60 0x56168dbff0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8843fff640 0 client.28729 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8871739740 1 --2- 172.21.15.16:0/640076702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d81a980 0x56168dbfeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8871739740 1 -- 172.21.15.16:0/640076702 >> 172.21.15.16:0/640076702 conn(0x56168d837990 msgr2=0x56168dbe7340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8871739740 1 -- 172.21.15.16:0/640076702 shutdown_connections 2024-08-28T19:38:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8871739740 1 -- 172.21.15.16:0/640076702 wait complete. 2024-08-28T19:38:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.637+0000 7f8871739740 20 client.28729 trim_cache size 0 max 16384 2024-08-28T19:38:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.668+0000 7f8871739740 1 Processor -- start 2024-08-28T19:38:52.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.668+0000 7f8871739740 1 -- start start 2024-08-28T19:38:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.669+0000 7f8871739740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d81a980 0x56168dc0af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.669+0000 7f8871739740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d822b60 0x56168dbfbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.669+0000 7f8871739740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dbfc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8871739740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f884800ffd0 con 0x56168d81a980 2024-08-28T19:38:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8871739740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8848011cf0 con 0x56168d822b60 2024-08-28T19:38:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8871739740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8848010760 con 0x56168d843840 2024-08-28T19:38:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8863fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d822b60 0x56168dbfbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8863fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d822b60 0x56168dbfbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:38112/0 (socket says 172.21.15.16:38112) 2024-08-28T19:38:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8863fff640 1 -- 172.21.15.16:0/4141980380 learned_addr learned my addr 172.21.15.16:0/4141980380 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8869761640 1 --2- 172.21.15.16:0/4141980380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dbfc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8868f60640 1 --2- 172.21.15.16:0/4141980380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d81a980 0x56168dc0af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8863fff640 1 -- 172.21.15.16:0/4141980380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 msgr2=0x56168dbfc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8863fff640 1 --2- 172.21.15.16:0/4141980380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dbfc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8863fff640 1 -- 172.21.15.16:0/4141980380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d81a980 msgr2=0x56168dc0af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8863fff640 1 --2- 172.21.15.16:0/4141980380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d81a980 0x56168dc0af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.670+0000 7f8863fff640 1 -- 172.21.15.16:0/4141980380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56168db1ed90 con 0x56168d822b60 2024-08-28T19:38:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.671+0000 7f8863fff640 1 --2- 172.21.15.16:0/4141980380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d822b60 0x56168dbfbd60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8850009860 tx=0x7f8850005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f1eabcf50ca92b5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.671+0000 7f884e7fc640 1 -- 172.21.15.16:0/4141980380 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885004d070 con 0x56168d822b60 2024-08-28T19:38:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.671+0000 7f884e7fc640 1 -- 172.21.15.16:0/4141980380 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8850030450 con 0x56168d822b60 2024-08-28T19:38:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.671+0000 7f884e7fc640 1 -- 172.21.15.16:0/4141980380 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850037040 con 0x56168d822b60 2024-08-28T19:38:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.673+0000 7f8871739740 1 -- 172.21.15.16:0/4141980380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d822b60 msgr2=0x56168dbfbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.673+0000 7f8871739740 1 --2- 172.21.15.16:0/4141980380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d822b60 0x56168dbfbd60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8850009860 tx=0x7f8850005f90 comp rx=0 tx=0).stop 2024-08-28T19:38:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.673+0000 7f8871739740 1 -- 172.21.15.16:0/4141980380 shutdown_connections 2024-08-28T19:38:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.673+0000 7f8871739740 1 --2- 172.21.15.16:0/4141980380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dbfc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.673+0000 7f8871739740 1 --2- 172.21.15.16:0/4141980380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d822b60 0x56168dbfbd60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.673+0000 7f8871739740 1 --2- 172.21.15.16:0/4141980380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d81a980 0x56168dc0af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.673+0000 7f8871739740 1 -- 172.21.15.16:0/4141980380 >> 172.21.15.16:0/4141980380 conn(0x56168d837990 msgr2=0x56168dbec630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.673+0000 7f8871739740 1 -- 172.21.15.16:0/4141980380 shutdown_connections 2024-08-28T19:38:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.673+0000 7f8871739740 1 -- 172.21.15.16:0/4141980380 wait complete. 2024-08-28T19:38:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.674+0000 7f8871739740 1 -- start start 2024-08-28T19:38:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.674+0000 7f8871739740 1 Processor -- start 2024-08-28T19:38:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.675+0000 7f8871739740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d81a980 0x56168dc03750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.675+0000 7f8871739740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d822b60 0x56168dc03c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.675+0000 7f8871739740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc7a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.675+0000 7f8871739740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8848011560 con 0x56168d822b60 2024-08-28T19:38:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.675+0000 7f8871739740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8848010e30 con 0x56168d81a980 2024-08-28T19:38:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.675+0000 7f8871739740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f884800dfd0 con 0x56168d843840 2024-08-28T19:38:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.675+0000 7f8871739740 10 client.0 _ll_register_callbacks cb 0x7f885c0a7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.675+0000 7f8869761640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc7a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.675+0000 7f8869761640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc7a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55666/0 (socket says 172.21.15.16:55666) 2024-08-28T19:38:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.675+0000 7f8869761640 1 -- 172.21.15.16:0/2767226384 learned_addr learned my addr 172.21.15.16:0/2767226384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:52.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.676+0000 7f8863fff640 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d822b60 0x56168dc03c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.677+0000 7f8869761640 1 -- 172.21.15.16:0/2767226384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d81a980 msgr2=0x56168dc03750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:38:52.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.677+0000 7f8869761640 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d81a980 0x56168dc03750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.677+0000 7f8869761640 1 -- 172.21.15.16:0/2767226384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d822b60 msgr2=0x56168dc03c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:52.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.677+0000 7f8869761640 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d822b60 0x56168dc03c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.677+0000 7f8869761640 1 -- 172.21.15.16:0/2767226384 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56168dc7ae70 con 0x56168d843840 2024-08-28T19:38:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.678+0000 7f8869761640 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc7a880 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f885c0ab920 tx=0x7f885c0e5450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b91b76ff0fd7adc6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.678+0000 7f8843fff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:38:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.678+0000 7f8843fff640 1 -- 172.21.15.16:0/2767226384 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c01f070 con 0x56168d843840 2024-08-28T19:38:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.679+0000 7f8843fff640 1 -- 172.21.15.16:0/2767226384 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f885c0a79f0 con 0x56168d843840 2024-08-28T19:38:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.679+0000 7f8871739740 1 -- 172.21.15.16:0/2767226384 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8850030450 con 0x56168d843840 2024-08-28T19:38:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.679+0000 7f8843fff640 1 -- 172.21.15.16:0/2767226384 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f885c0b4a80 con 0x56168d843840 2024-08-28T19:38:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.681+0000 7f8871739740 10 client.28735 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.681+0000 7f8871739740 1 -- 172.21.15.16:0/2767226384 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x56168dc7c0b0 con 0x56168d843840 2024-08-28T19:38:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.682+0000 7f8843fff640 1 -- 172.21.15.16:0/2767226384 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8850030450 con 0x56168d843840 2024-08-28T19:38:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.682+0000 7f8843fff640 1 -- 172.21.15.16:0/2767226384 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7f885c0d2260 con 0x56168d843840 2024-08-28T19:38:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.682+0000 7f8871739740 10 client.28735 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.682+0000 7f8871739740 10 client.28735 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:38:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.682+0000 7f8871739740 20 client.28735 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.682+0000 7f8871739740 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56168d7d7570 0x56168dc7e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.682+0000 7f8871739740 4 client.28735 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.8, "await_for": 45.0}] 2024-08-28T19:38:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.682+0000 7f8871739740 1 -- 172.21.15.16:0/2767226384 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.8, "await_for": 45.0}) -- 0x56168db5cd20 con 0x56168d7d7570 2024-08-28T19:38:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.683+0000 7f8868f60640 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56168d7d7570 0x56168dc7e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.684+0000 7f8868f60640 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56168d7d7570 0x56168dc7e920 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5128261d0149d41 server_cookie=1f8d8439899248c9 in_seq=0 out_seq=0 2024-08-28T19:38:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:52.684+0000 7f8843fff640 10 client.28735 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:52.999 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:38:52.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:38:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.128+0000 7f8843fff640 1 -- 172.21.15.16:0/2767226384 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x56168db5cd20 con 0x56168d7d7570 2024-08-28T19:38:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.128+0000 7f8843fff640 10 client.28735 handle_command_reply: tid=0 2024-08-28T19:38:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.128+0000 7f8871739740 1 client.28735 shutdown 2024-08-28T19:38:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8871739740 1 -- 172.21.15.16:0/2767226384 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 msgr2=0x56168dc7a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:53.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8871739740 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc7a880 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f885c0ab920 tx=0x7f885c0e5450 comp rx=0 tx=0).stop 2024-08-28T19:38:53.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8871739740 1 -- 172.21.15.16:0/2767226384 shutdown_connections 2024-08-28T19:38:53.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8871739740 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56168d7d7570 0x56168dc7e920 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:53.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8871739740 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56168d843840 0x56168dc7a880 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:53.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8843fff640 0 client.28735 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:53.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8871739740 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56168d822b60 0x56168dc03c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:53.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8871739740 1 --2- 172.21.15.16:0/2767226384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56168d81a980 0x56168dc03750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:53.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8871739740 1 -- 172.21.15.16:0/2767226384 >> 172.21.15.16:0/2767226384 conn(0x56168d837990 msgr2=0x56168dbec610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:53.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8871739740 1 -- 172.21.15.16:0/2767226384 shutdown_connections 2024-08-28T19:38:53.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8871739740 1 -- 172.21.15.16:0/2767226384 wait complete. 2024-08-28T19:38:53.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.129+0000 7f8871739740 20 client.28735 trim_cache size 0 max 16384 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 96, 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 54, 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: "3c5129": { 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 54, 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 141.8, 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:53.133 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:38:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 -- 172.21.15.16:0/3424471302 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f88540b50a0 msgr2=0x7f88540b75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f88540b50a0 0x7f88540b75b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f885002a860 tx=0x7f8850009520 comp rx=0 tx=0).stop 2024-08-28T19:38:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 -- 172.21.15.16:0/3424471302 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8864108230 msgr2=0x7f886406f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8864108230 0x7f886406f2d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f885c00eaa0 tx=0x7f885c00ef60 comp rx=0 tx=0).stop 2024-08-28T19:38:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 -- 172.21.15.16:0/3424471302 shutdown_connections 2024-08-28T19:38:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8864108230 0x7f886406f2d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f88540b50a0 0x7f88540b75b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641078e0 0x7f886406ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 --2- 172.21.15.16:0/3424471302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8864101fa0 0x7f88641130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 -- 172.21.15.16:0/3424471302 >> 172.21.15.16:0/3424471302 conn(0x7f8864079cf0 msgr2=0x7f8864100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 -- 172.21.15.16:0/3424471302 shutdown_connections 2024-08-28T19:38:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:53.130+0000 7f886b1c4640 1 -- 172.21.15.16:0/3424471302 wait complete. 2024-08-28T19:38:53.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 24.80 MB/sec execute 149 sec latency 805.078 ms 2024-08-28T19:38:53.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:53 smithi016 ceph-mon[18556]: pgmap v896: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 8.7 MiB/s wr, 662 op/s 2024-08-28T19:38:53.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:53 smithi028 ceph-mon[21769]: pgmap v896: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 8.7 MiB/s wr, 662 op/s 2024-08-28T19:38:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:53 smithi115 ceph-mon[22184]: pgmap v896: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 8.7 MiB/s wr, 662 op/s 2024-08-28T19:38:53.934 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '3c5129', quiesce took 8.0 seconds. Will release after: 13.8 2024-08-28T19:38:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:54.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 24.64 MB/sec execute 150 sec latency 1805.176 ms 2024-08-28T19:38:55.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 24.47 MB/sec execute 151 sec latency 2805.217 ms 2024-08-28T19:38:55.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.581+0000 7f57fda90640 1 -- 172.21.15.16:0/2552276072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f8107980 msgr2=0x7f57f8107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:55.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.581+0000 7f57fda90640 1 --2- 172.21.15.16:0/2552276072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f8107980 0x7f57f8107d90 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009900 tx=0x7f57ec02a400 comp rx=0 tx=0).stop 2024-08-28T19:38:55.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.581+0000 7f57fda90640 1 -- 172.21.15.16:0/2552276072 shutdown_connections 2024-08-28T19:38:55.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.581+0000 7f57fda90640 1 --2- 172.21.15.16:0/2552276072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f81082d0 0x7f57f810eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.581+0000 7f57fda90640 1 --2- 172.21.15.16:0/2552276072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f8107980 0x7f57f8107d90 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.581+0000 7f57fda90640 1 --2- 172.21.15.16:0/2552276072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f8102040 0x7f57f8102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.581+0000 7f57fda90640 1 -- 172.21.15.16:0/2552276072 >> 172.21.15.16:0/2552276072 conn(0x7f57f8079c60 msgr2=0x7f57f807a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:55.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.581+0000 7f57fda90640 1 -- 172.21.15.16:0/2552276072 shutdown_connections 2024-08-28T19:38:55.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.581+0000 7f57fda90640 1 -- 172.21.15.16:0/2552276072 wait complete. 2024-08-28T19:38:55.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.582+0000 7f57fda90640 1 Processor -- start 2024-08-28T19:38:55.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.582+0000 7f57fda90640 1 -- start start 2024-08-28T19:38:55.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.582+0000 7f57fda90640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f8102040 0x7f57f819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.582+0000 7f57fda90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f8107980 0x7f57f819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.582+0000 7f57fda90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f81082d0 0x7f57f819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.582+0000 7f57fda90640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57f8112810 con 0x7f57f81082d0 2024-08-28T19:38:55.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.582+0000 7f57fda90640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57f8112690 con 0x7f57f8107980 2024-08-28T19:38:55.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.582+0000 7f57fda90640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f57f8112990 con 0x7f57f8102040 2024-08-28T19:38:55.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.583+0000 7f57f7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f81082d0 0x7f57f819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.583+0000 7f57f7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f81082d0 0x7f57f819fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43402/0 (socket says 172.21.15.16:43402) 2024-08-28T19:38:55.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.583+0000 7f57f7fff640 1 -- 172.21.15.16:0/960822536 learned_addr learned my addr 172.21.15.16:0/960822536 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.583+0000 7f57f77fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f8102040 0x7f57f819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.583+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f8107980 0x7f57f819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.583+0000 7f57f7fff640 1 -- 172.21.15.16:0/960822536 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f8102040 msgr2=0x7f57f819a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.583+0000 7f57f7fff640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f8102040 0x7f57f819a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.583+0000 7f57f7fff640 1 -- 172.21.15.16:0/960822536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f8107980 msgr2=0x7f57f819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.583+0000 7f57f7fff640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f8107980 0x7f57f819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.583+0000 7f57f7fff640 1 -- 172.21.15.16:0/960822536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f81a0320 con 0x7f57f81082d0 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.583+0000 7f57f7fff640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f81082d0 0x7f57f819fbf0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f57f000cad0 tx=0x7f57f000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.584+0000 7f57f4ff9640 1 -- 172.21.15.16:0/960822536 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f000ae20 con 0x7f57f81082d0 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.584+0000 7f57f4ff9640 1 -- 172.21.15.16:0/960822536 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57f0005020 con 0x7f57f81082d0 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.584+0000 7f57fda90640 1 -- 172.21.15.16:0/960822536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f81a0610 con 0x7f57f81082d0 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.584+0000 7f57f4ff9640 1 -- 172.21.15.16:0/960822536 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f000f6b0 con 0x7f57f81082d0 2024-08-28T19:38:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.584+0000 7f57fda90640 1 -- 172.21.15.16:0/960822536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f80feb70 con 0x7f57f81082d0 2024-08-28T19:38:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.586+0000 7f5804005740 1 Processor -- start 2024-08-28T19:38:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f57f4ff9640 1 -- 172.21.15.16:0/960822536 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f57f00051d0 con 0x7f57f81082d0 2024-08-28T19:38:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f5804005740 1 -- start start 2024-08-28T19:38:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f5804005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c625350 0x555d3c9cb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f5804005740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 0x555d3c9cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f5804005740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9cc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f5804005740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555d3c9415f0 con 0x555d3c604670 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f5804005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d3c930b80 con 0x555d3c625350 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f5804005740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555d3c934b30 con 0x555d3c5fc490 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f57f6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f57f6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9cc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43412/0 (socket says 172.21.15.16:43412) 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f57f6ffd640 1 -- 172.21.15.16:0/2738452365 learned_addr learned my addr 172.21.15.16:0/2738452365 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f57f77fe640 1 --2- 172.21.15.16:0/2738452365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 0x555d3c9cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.587+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/2738452365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c625350 0x555d3c9cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.588+0000 7f57f4ff9640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57e00b4ca0 0x7f57e00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.588+0000 7f57f6ffd640 1 -- 172.21.15.16:0/2738452365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 msgr2=0x555d3c9cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.588+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/2738452365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 0x555d3c9cbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.588+0000 7f57f6ffd640 1 -- 172.21.15.16:0/2738452365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c625350 msgr2=0x555d3c9cb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.588+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/2738452365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c625350 0x555d3c9cb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.588+0000 7f57f6ffd640 1 -- 172.21.15.16:0/2738452365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d3c690930 con 0x555d3c604670 2024-08-28T19:38:55.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.588+0000 7f57f4ff9640 1 -- 172.21.15.16:0/960822536 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f57f00e5810 con 0x7f57f81082d0 2024-08-28T19:38:55.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.588+0000 7f57f7fff640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57e00b4ca0 0x7f57e00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.588+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/2738452365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9cc200 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f57ec032fd0 tx=0x7f57ec000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fe7261e81797a0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.589+0000 7f57ddffb640 1 -- 172.21.15.16:0/2738452365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec035070 con 0x555d3c604670 2024-08-28T19:38:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.589+0000 7f57ddffb640 1 -- 172.21.15.16:0/2738452365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57ec005030 con 0x555d3c604670 2024-08-28T19:38:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.589+0000 7f57ddffb640 1 -- 172.21.15.16:0/2738452365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec0044e0 con 0x555d3c604670 2024-08-28T19:38:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.589+0000 7f5804005740 1 -- 172.21.15.16:0/2738452365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 msgr2=0x555d3c9cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.589+0000 7f5804005740 1 --2- 172.21.15.16:0/2738452365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9cc200 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f57ec032fd0 tx=0x7f57ec000bc0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.589+0000 7f57f7fff640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57e00b4ca0 0x7f57e00b71b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f57f000caa0 tx=0x7f57f0005c50 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.590+0000 7f5804005740 1 -- 172.21.15.16:0/2738452365 shutdown_connections 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.590+0000 7f5804005740 1 --2- 172.21.15.16:0/2738452365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9cc200 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.590+0000 7f5804005740 1 --2- 172.21.15.16:0/2738452365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 0x555d3c9cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.590+0000 7f5804005740 1 --2- 172.21.15.16:0/2738452365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c625350 0x555d3c9cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.590+0000 7f5804005740 1 -- 172.21.15.16:0/2738452365 >> 172.21.15.16:0/2738452365 conn(0x555d3c619740 msgr2=0x555d3c9c9670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.590+0000 7f5804005740 1 -- 172.21.15.16:0/2738452365 shutdown_connections 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.590+0000 7f5804005740 1 -- 172.21.15.16:0/2738452365 wait complete. 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.590+0000 7f5804005740 1 -- start start 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.590+0000 7f5804005740 1 Processor -- start 2024-08-28T19:38:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f5804005740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 0x555d3c9d6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f5804005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c604670 0x555d3c9d7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f5804005740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c625350 0x555d3c9e31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f5804005740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555d3c793890 con 0x555d3c625350 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f5804005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d3c8b7790 con 0x555d3c604670 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f5804005740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555d3c9d9f00 con 0x555d3c5fc490 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f5804005740 10 client.0 _ll_register_callbacks cb 0x555d3c63cba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f57f6ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 0x555d3c9d6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f57f6ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 0x555d3c9d6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:48090/0 (socket says 172.21.15.16:48090) 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f57f6ffd640 1 -- 172.21.15.16:0/2277116689 learned_addr learned my addr 172.21.15.16:0/2277116689 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c625350 0x555d3c9e31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f57f77fe640 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c604670 0x555d3c9d7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f57f6ffd640 1 -- 172.21.15.16:0/2277116689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c604670 msgr2=0x555d3c9d7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c604670 0x555d3c9d7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f57f6ffd640 1 -- 172.21.15.16:0/2277116689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c625350 msgr2=0x555d3c9e31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c625350 0x555d3c9e31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f57f6ffd640 1 -- 172.21.15.16:0/2277116689 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d3c8eef30 con 0x555d3c5fc490 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.591+0000 7f57f77fe640 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c604670 0x555d3c9d7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.592+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 0x555d3c9d6f10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0048b0 tx=0x7f57ec02a400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf1d66dcca25a41c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.592+0000 7f57d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.592+0000 7f57d6ffd640 1 -- 172.21.15.16:0/2277116689 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec035040 con 0x555d3c5fc490 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.592+0000 7f57d6ffd640 1 -- 172.21.15.16:0/2277116689 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f57ec0053e0 con 0x555d3c5fc490 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.592+0000 7f5804005740 1 -- 172.21.15.16:0/2277116689 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57ec005030 con 0x555d3c5fc490 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.592+0000 7f57d6ffd640 1 -- 172.21.15.16:0/2277116689 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f57ec0055e0 con 0x555d3c5fc490 2024-08-28T19:38:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.592+0000 7f5804005740 10 client.28744 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.592+0000 7f5804005740 1 -- 172.21.15.16:0/2277116689 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x555d3c9e5ce0 con 0x555d3c5fc490 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.593+0000 7f57d6ffd640 1 -- 172.21.15.16:0/2277116689 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57ec005030 con 0x555d3c5fc490 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.593+0000 7f57d6ffd640 1 -- 172.21.15.16:0/2277116689 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7f57ec0035f0 con 0x555d3c5fc490 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.593+0000 7f5804005740 10 client.28744 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.593+0000 7f5804005740 10 client.28744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.593+0000 7f5804005740 20 client.28744 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.593+0000 7f5804005740 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555d3c5e3cd0 0x555d3c9e9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.593+0000 7f5804005740 4 client.28744 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.593+0000 7f5804005740 1 -- 172.21.15.16:0/2277116689 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d3c943540 con 0x555d3c5e3cd0 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.593+0000 7f57f77fe640 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555d3c5e3cd0 0x555d3c9e9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.594+0000 7f57d6ffd640 10 client.28744 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:55.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.594+0000 7f57f77fe640 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555d3c5e3cd0 0x555d3c9e9040 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b806ebe53cb96acc server_cookie=9964e6d8794decf8 in_seq=0 out_seq=0 2024-08-28T19:38:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.594+0000 7f57d6ffd640 1 -- 172.21.15.16:0/2277116689 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555d3c943540 con 0x555d3c5e3cd0 2024-08-28T19:38:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.594+0000 7f57d6ffd640 10 client.28744 handle_command_reply: tid=0 2024-08-28T19:38:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.594+0000 7f5804005740 1 client.28744 shutdown 2024-08-28T19:38:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.594+0000 7f5804005740 1 -- 172.21.15.16:0/2277116689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 msgr2=0x555d3c9d6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.594+0000 7f5804005740 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 0x555d3c9d6f10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0048b0 tx=0x7f57ec02a400 comp rx=0 tx=0).stop 2024-08-28T19:38:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.595+0000 7f5804005740 1 -- 172.21.15.16:0/2277116689 shutdown_connections 2024-08-28T19:38:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.595+0000 7f5804005740 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c625350 0x555d3c9e31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.595+0000 7f5804005740 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c604670 0x555d3c9d7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.595+0000 7f5804005740 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555d3c5e3cd0 0x555d3c9e9040 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.595+0000 7f5804005740 1 --2- 172.21.15.16:0/2277116689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c5fc490 0x555d3c9d6f10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.595+0000 7f5804005740 1 -- 172.21.15.16:0/2277116689 >> 172.21.15.16:0/2277116689 conn(0x555d3c619740 msgr2=0x555d3c9ce960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.595+0000 7f57d6ffd640 0 client.28744 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.595+0000 7f5804005740 1 -- 172.21.15.16:0/2277116689 shutdown_connections 2024-08-28T19:38:55.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.595+0000 7f5804005740 1 -- 172.21.15.16:0/2277116689 wait complete. 2024-08-28T19:38:55.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.595+0000 7f5804005740 20 client.28744 trim_cache size 0 max 16384 2024-08-28T19:38:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.613+0000 7f5804005740 1 Processor -- start 2024-08-28T19:38:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.613+0000 7f5804005740 1 -- start start 2024-08-28T19:38:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.613+0000 7f5804005740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c5fc490 0x555d3c9cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.613+0000 7f5804005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c604670 0x555d3c9d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.613+0000 7f5804005740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c625350 0x555d3c9d4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.613+0000 7f5804005740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81aec10 con 0x555d3c5fc490 2024-08-28T19:38:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.613+0000 7f5804005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81b0930 con 0x555d3c604670 2024-08-28T19:38:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.613+0000 7f5804005740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81af3a0 con 0x555d3c625350 2024-08-28T19:38:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.613+0000 7f57f6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c5fc490 0x555d3c9cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.613+0000 7f57f6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c5fc490 0x555d3c9cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43434/0 (socket says 172.21.15.16:43434) 2024-08-28T19:38:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.613+0000 7f57f6ffd640 1 -- 172.21.15.16:0/431034998 learned_addr learned my addr 172.21.15.16:0/431034998 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.614+0000 7f57f6ffd640 1 -- 172.21.15.16:0/431034998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c625350 msgr2=0x555d3c9d4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.614+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/431034998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c625350 0x555d3c9d4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.614+0000 7f57f6ffd640 1 -- 172.21.15.16:0/431034998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c604670 msgr2=0x555d3c9d4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.614+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/431034998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c604670 0x555d3c9d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.614+0000 7f57f6ffd640 1 -- 172.21.15.16:0/431034998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d3c9d5e70 con 0x555d3c5fc490 2024-08-28T19:38:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.614+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/431034998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c5fc490 0x555d3c9cef50 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f57ec02a920 tx=0x7f57ec005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ccb6df38c34f46 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.614+0000 7f57dd7fa640 1 -- 172.21.15.16:0/431034998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec0035f0 con 0x555d3c5fc490 2024-08-28T19:38:55.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.614+0000 7f57dd7fa640 1 -- 172.21.15.16:0/431034998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57ec003790 con 0x555d3c5fc490 2024-08-28T19:38:55.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.614+0000 7f57dd7fa640 1 -- 172.21.15.16:0/431034998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec04d800 con 0x555d3c5fc490 2024-08-28T19:38:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.615+0000 7f5804005740 1 -- 172.21.15.16:0/431034998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c5fc490 msgr2=0x555d3c9cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.615+0000 7f5804005740 1 --2- 172.21.15.16:0/431034998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c5fc490 0x555d3c9cef50 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f57ec02a920 tx=0x7f57ec005eb0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.615+0000 7f5804005740 1 -- 172.21.15.16:0/431034998 shutdown_connections 2024-08-28T19:38:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.615+0000 7f5804005740 1 --2- 172.21.15.16:0/431034998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c625350 0x555d3c9d4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.615+0000 7f5804005740 1 --2- 172.21.15.16:0/431034998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c604670 0x555d3c9d4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.615+0000 7f5804005740 1 --2- 172.21.15.16:0/431034998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c5fc490 0x555d3c9cef50 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.615+0000 7f5804005740 1 -- 172.21.15.16:0/431034998 >> 172.21.15.16:0/431034998 conn(0x555d3c619740 msgr2=0x555d3c9ce840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.615+0000 7f5804005740 1 -- 172.21.15.16:0/431034998 shutdown_connections 2024-08-28T19:38:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.615+0000 7f5804005740 1 -- 172.21.15.16:0/431034998 wait complete. 2024-08-28T19:38:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.615+0000 7f5804005740 1 -- start start 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f5804005740 1 Processor -- start 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f5804005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c5fc490 0x555d3c9eaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f5804005740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9eb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f5804005740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c625350 0x555d3ca4c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f5804005740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81b01a0 con 0x555d3c604670 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f5804005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81afa70 con 0x555d3c5fc490 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f5804005740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f57f81acc10 con 0x555d3c625350 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f5804005740 10 client.0 _ll_register_callbacks cb 0x7f57ec032d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f57f77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9eb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f57f77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9eb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43438/0 (socket says 172.21.15.16:43438) 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f57f77fe640 1 -- 172.21.15.16:0/1590593042 learned_addr learned my addr 172.21.15.16:0/1590593042 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c625350 0x555d3ca4c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.616+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c5fc490 0x555d3c9eaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f57f77fe640 1 -- 172.21.15.16:0/1590593042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c625350 msgr2=0x555d3ca4c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f57f77fe640 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c625350 0x555d3ca4c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f57f77fe640 1 -- 172.21.15.16:0/1590593042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c5fc490 msgr2=0x555d3c9eaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f57f77fe640 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c5fc490 0x555d3c9eaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f57f77fe640 1 -- 172.21.15.16:0/1590593042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d3ca4ce70 con 0x555d3c604670 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c625350 0x555d3ca4c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f57f77fe640 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9eb0a0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f57e8005e00 tx=0x7f57e8009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddcd99c85324c9c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f57d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f57d6ffd640 1 -- 172.21.15.16:0/1590593042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8003cc0 con 0x555d3c604670 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f5804005740 1 -- 172.21.15.16:0/1590593042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57ec003790 con 0x555d3c604670 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f57d6ffd640 1 -- 172.21.15.16:0/1590593042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f57e801c3f0 con 0x555d3c604670 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f57d6ffd640 1 -- 172.21.15.16:0/1590593042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f57e8032050 con 0x555d3c604670 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f5804005740 10 client.19728 fetch_fsmap learned FSMap version 96 2024-08-28T19:38:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.617+0000 7f5804005740 1 -- 172.21.15.16:0/1590593042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) -- 0x555d3ca54310 con 0x555d3c604670 2024-08-28T19:38:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.618+0000 7f57d6ffd640 1 -- 172.21.15.16:0/1590593042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57ec003790 con 0x555d3c604670 2024-08-28T19:38:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.618+0000 7f57d6ffd640 1 -- 172.21.15.16:0/1590593042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 96) ==== 8410+0+0 (secure 0 0 0) 0x7f57e802a4a0 con 0x555d3c604670 2024-08-28T19:38:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.618+0000 7f5804005740 10 client.19728 fetch_fsmap finished waiting for FSMap version 96 2024-08-28T19:38:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.618+0000 7f5804005740 10 client.19728 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.618+0000 7f5804005740 20 client.19728 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.618+0000 7f5804005740 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555d3c5b9080 0x555d3ca56b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.618+0000 7f5804005740 4 client.19728 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:38:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.618+0000 7f5804005740 1 -- 172.21.15.16:0/1590593042 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555d3c93efe0 con 0x555d3c5b9080 2024-08-28T19:38:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.618+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555d3c5b9080 0x555d3ca56b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.619+0000 7f57d6ffd640 10 client.19728 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.619+0000 7f57f6ffd640 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555d3c5b9080 0x555d3ca56b40 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a02babd925926936 server_cookie=4ccf4b05475093d4 in_seq=0 out_seq=0 2024-08-28T19:38:55.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f57d6ffd640 1 -- 172.21.15.16:0/1590593042 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555d3c93efe0 con 0x555d3c5b9080 2024-08-28T19:38:55.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f57d6ffd640 10 client.19728 handle_command_reply: tid=0 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 1 client.19728 shutdown 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 1 -- 172.21.15.16:0/1590593042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 msgr2=0x555d3c9eb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9eb0a0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f57e8005e00 tx=0x7f57e8009d60 comp rx=0 tx=0).stop 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 1 -- 172.21.15.16:0/1590593042 shutdown_connections 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x555d3c5b9080 0x555d3ca56b40 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555d3c625350 0x555d3ca4c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f57d6ffd640 0 client.19728 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555d3c604670 0x555d3c9eb0a0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 1 --2- 172.21.15.16:0/1590593042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3c5fc490 0x555d3c9eaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 1 -- 172.21.15.16:0/1590593042 >> 172.21.15.16:0/1590593042 conn(0x555d3c619740 msgr2=0x555d3c9cc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 1 -- 172.21.15.16:0/1590593042 shutdown_connections 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 1 -- 172.21.15.16:0/1590593042 wait complete. 2024-08-28T19:38:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.620+0000 7f5804005740 20 client.19728 trim_cache size 0 max 16384 2024-08-28T19:38:55.626 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:38:55.626 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:38:55.626 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "18e2c1b8-628f-4f38-b56c-dcde542d4ddb", 2024-08-28T19:38:55.626 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:38:55.626 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 -- 172.21.15.16:0/960822536 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57e00b4ca0 msgr2=0x7f57e00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57e00b4ca0 0x7f57e00b71b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f57f000caa0 tx=0x7f57f0005c50 comp rx=0 tx=0).stop 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 -- 172.21.15.16:0/960822536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f81082d0 msgr2=0x7f57f819fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f81082d0 0x7f57f819fbf0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f57f000cad0 tx=0x7f57f000cf90 comp rx=0 tx=0).stop 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 -- 172.21.15.16:0/960822536 shutdown_connections 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f81082d0 0x7f57f819fbf0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f57e00b4ca0 0x7f57e00b71b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f8107980 0x7f57f819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 --2- 172.21.15.16:0/960822536 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57f8102040 0x7f57f819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 -- 172.21.15.16:0/960822536 >> 172.21.15.16:0/960822536 conn(0x7f57f8079c60 msgr2=0x7f57f810f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 -- 172.21.15.16:0/960822536 shutdown_connections 2024-08-28T19:38:55.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:55.621+0000 7f57fda90640 1 -- 172.21.15.16:0/960822536 wait complete. 2024-08-28T19:38:55.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:55 smithi016 ceph-mon[18556]: pgmap v897: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 7.1 MiB/s wr, 548 op/s 2024-08-28T19:38:55.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:55 smithi028 ceph-mon[21769]: pgmap v897: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 7.1 MiB/s wr, 548 op/s 2024-08-28T19:38:55.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:55 smithi115 ceph-mon[22184]: pgmap v897: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 7.1 MiB/s wr, 548 op/s 2024-08-28T19:38:56.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 24.31 MB/sec execute 152 sec latency 3805.256 ms 2024-08-28T19:38:56.570 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:56 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:56.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:56 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:56.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:56 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:56.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:56 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:56 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:56.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:56 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:56 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:56.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:56 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:56 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:56 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:38:56.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:38:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:56 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:56 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:56 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:56 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:56 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:56 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:56 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:56 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:56 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:56 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:38:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:56 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:56 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:56 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:56 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:56 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:56 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:56 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:56 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:56 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:56 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:38:57.041 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:38:57.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 24.15 MB/sec execute 153 sec latency 4805.294 ms 2024-08-28T19:38:57.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:57 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:38:57.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:57 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:57 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:57.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:57 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:57.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:57 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:57 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:38:57.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:57 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:57 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:57 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:57 smithi016 ceph-mon[18556]: pgmap v898: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 8.3 MiB/s wr, 649 op/s 2024-08-28T19:38:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.768+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/1806888356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ae4113c50 msgr2=0x7f7ae4114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.768+0000 7f7ae8d7d640 1 --2- 172.21.15.16:0/1806888356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ae4113c50 0x7f7ae4114060 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009ca0 tx=0x7f7adc02a8f0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.769+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/1806888356 shutdown_connections 2024-08-28T19:38:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.769+0000 7f7ae8d7d640 1 --2- 172.21.15.16:0/1806888356 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ae41145a0 0x7f7ae411add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.769+0000 7f7ae8d7d640 1 --2- 172.21.15.16:0/1806888356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ae4113c50 0x7f7ae4114060 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.769+0000 7f7ae8d7d640 1 --2- 172.21.15.16:0/1806888356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae410e360 0x7f7ae410e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.769+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/1806888356 >> 172.21.15.16:0/1806888356 conn(0x7f7ae410a160 msgr2=0x7f7ae410c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.771+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/1806888356 shutdown_connections 2024-08-28T19:38:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.771+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/1806888356 wait complete. 2024-08-28T19:38:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.773+0000 7f7ae8d7d640 1 Processor -- start 2024-08-28T19:38:57.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.774+0000 7f7ae8d7d640 1 -- start start 2024-08-28T19:38:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.775+0000 7f7ae8d7d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ae410e360 0x7f7ae41a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.775+0000 7f7ae8d7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4113c50 0x7f7ae41a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.775+0000 7f7ae8d7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ae41145a0 0x7f7ae41aa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.775+0000 7f7ae8d7d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae411e2f0 con 0x7f7ae41145a0 2024-08-28T19:38:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.775+0000 7f7ae8d7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae411e170 con 0x7f7ae4113c50 2024-08-28T19:38:57.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.776+0000 7f7ae8d7d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae411e470 con 0x7f7ae410e360 2024-08-28T19:38:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.776+0000 7f7ae1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4113c50 0x7f7ae41a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.776+0000 7f7ae1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4113c50 0x7f7ae41a5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:41056/0 (socket says 172.21.15.16:41056) 2024-08-28T19:38:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.776+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/2446022399 learned_addr learned my addr 172.21.15.16:0/2446022399 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.776+0000 7f7ae259c640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ae410e360 0x7f7ae41a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.776+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/2446022399 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ae410e360 msgr2=0x7f7ae41a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.776+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ae410e360 0x7f7ae41a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.776+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/2446022399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ae41145a0 msgr2=0x7f7ae41aa500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.776+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ae41145a0 0x7f7ae41aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.776+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/2446022399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae41aac30 con 0x7f7ae4113c50 2024-08-28T19:38:57.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.777+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4113c50 0x7f7ae41a5260 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7adc02f570 tx=0x7f7adc0317f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.781+0000 7f7ad37fe640 1 -- 172.21.15.16:0/2446022399 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc003b70 con 0x7f7ae4113c50 2024-08-28T19:38:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.781+0000 7f7ad37fe640 1 -- 172.21.15.16:0/2446022399 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7adc02d070 con 0x7f7ae4113c50 2024-08-28T19:38:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.781+0000 7f7ad37fe640 1 -- 172.21.15.16:0/2446022399 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc02f930 con 0x7f7ae4113c50 2024-08-28T19:38:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.781+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/2446022399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae41aaec0 con 0x7f7ae4113c50 2024-08-28T19:38:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.782+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/2446022399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae41b1680 con 0x7f7ae4113c50 2024-08-28T19:38:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.788+0000 7f7aef2f2740 1 Processor -- start 2024-08-28T19:38:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.789+0000 7f7ad37fe640 1 -- 172.21.15.16:0/2446022399 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7adc02fad0 con 0x7f7ae4113c50 2024-08-28T19:38:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.790+0000 7f7aef2f2740 1 -- start start 2024-08-28T19:38:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.790+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b248aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.790+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 0x5570b248b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.790+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b248b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.790+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5570b23f9630 con 0x5570b20b41b0 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.790+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5570b23e8bc0 con 0x5570b20dd070 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.790+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5570b23ecb70 con 0x5570b20bc390 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.791+0000 7f7ae1d9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b248b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.791+0000 7f7ae1d9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b248b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:48126/0 (socket says 172.21.15.16:48126) 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.791+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/1199506830 learned_addr learned my addr 172.21.15.16:0/1199506830 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.791+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/1199506830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 msgr2=0x5570b248aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.791+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/1199506830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b248aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.791+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/1199506830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 msgr2=0x5570b248b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.791+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/1199506830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 0x5570b248b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.791+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/1199506830 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570b2148650 con 0x5570b20bc390 2024-08-28T19:38:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.792+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/1199506830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b248b550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7adc103e20 tx=0x7f7adc10ac70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fffc466e16fac028 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.793+0000 7f7ad37fe640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7acc0b4e30 0x7f7acc0b7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.793+0000 7f7ad0ff9640 1 -- 172.21.15.16:0/1199506830 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc111850 con 0x5570b20bc390 2024-08-28T19:38:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.793+0000 7f7ad0ff9640 1 -- 172.21.15.16:0/1199506830 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7adc112420 con 0x5570b20bc390 2024-08-28T19:38:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.794+0000 7f7ad37fe640 1 -- 172.21.15.16:0/2446022399 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f7adc106b30 con 0x7f7ae4113c50 2024-08-28T19:38:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.795+0000 7f7aef2f2740 1 -- 172.21.15.16:0/1199506830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 msgr2=0x5570b248b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.795+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/1199506830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b248b550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7adc103e20 tx=0x7f7adc10ac70 comp rx=0 tx=0).stop 2024-08-28T19:38:57.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.797+0000 7f7ae2d9d640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7acc0b4e30 0x7f7acc0b7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.798+0000 7f7ae2d9d640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7acc0b4e30 0x7f7acc0b7340 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009650 tx=0x7f7ad4006cb0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.798+0000 7f7aef2f2740 1 -- 172.21.15.16:0/1199506830 shutdown_connections 2024-08-28T19:38:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.798+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/1199506830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b248b550 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.798+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/1199506830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 0x5570b248b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.798+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/1199506830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b248aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.798+0000 7f7aef2f2740 1 -- 172.21.15.16:0/1199506830 >> 172.21.15.16:0/1199506830 conn(0x5570b20d1460 msgr2=0x5570b24889c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.801+0000 7f7aef2f2740 1 -- 172.21.15.16:0/1199506830 shutdown_connections 2024-08-28T19:38:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.801+0000 7f7aef2f2740 1 -- 172.21.15.16:0/1199506830 wait complete. 2024-08-28T19:38:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.801+0000 7f7aef2f2740 1 -- start start 2024-08-28T19:38:57.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.801+0000 7f7aef2f2740 1 Processor -- start 2024-08-28T19:38:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.802+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 0x5570b24961d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.802+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b2496710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.802+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b24a2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.802+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5570b224b700 con 0x5570b20b41b0 2024-08-28T19:38:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.802+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5570b236f700 con 0x5570b20dd070 2024-08-28T19:38:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.802+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5570b2499070 con 0x5570b20bc390 2024-08-28T19:38:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.802+0000 7f7aef2f2740 10 client.0 _ll_register_callbacks cb 0x5570b239edc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.803+0000 7f7ae259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 0x5570b24961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.803+0000 7f7ae259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 0x5570b24961d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43478/0 (socket says 172.21.15.16:43478) 2024-08-28T19:38:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.803+0000 7f7ae259c640 1 -- 172.21.15.16:0/4250442104 learned_addr learned my addr 172.21.15.16:0/4250442104 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.803+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b2496710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.804+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/4250442104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 msgr2=0x5570b24a2370 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:38:57.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.804+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b24a2370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.804+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/4250442104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 msgr2=0x5570b24961d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:57.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.804+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 0x5570b24961d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.804+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/4250442104 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570b23a88b0 con 0x5570b20bc390 2024-08-28T19:38:57.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:57 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:38:57.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:57 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:57 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:57.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:57 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:57.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:57 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:57 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:38:57.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:57 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:57 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:57 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:57 smithi028 ceph-mon[21769]: pgmap v898: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 8.3 MiB/s wr, 649 op/s 2024-08-28T19:38:57.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.808+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b2496710 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7adc103f60 tx=0x7f7adc106d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfdab75fa11297dc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:57.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.808+0000 7f7ac9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:38:57.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.809+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/4250442104 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0c8d10 con 0x5570b20bc390 2024-08-28T19:38:57.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.810+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/4250442104 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f7adc0c90e0 con 0x5570b20bc390 2024-08-28T19:38:57.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.814+0000 7f7aef2f2740 1 -- 172.21.15.16:0/4250442104 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7adc112420 con 0x5570b20bc390 2024-08-28T19:38:57.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.815+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/4250442104 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f7adc0cc5c0 con 0x5570b20bc390 2024-08-28T19:38:57.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.816+0000 7f7aef2f2740 10 client.28756 fetch_fsmap learned FSMap version 97 2024-08-28T19:38:57.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.816+0000 7f7aef2f2740 1 -- 172.21.15.16:0/4250442104 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5570b24a4df0 con 0x5570b20bc390 2024-08-28T19:38:57.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.817+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/4250442104 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7adc112420 con 0x5570b20bc390 2024-08-28T19:38:57.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.817+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/4250442104 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 97) ==== 8402+0+0 (secure 0 0 0) 0x7f7adc0d1300 con 0x5570b20bc390 2024-08-28T19:38:57.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.817+0000 7f7aef2f2740 10 client.28756 fetch_fsmap finished waiting for FSMap version 97 2024-08-28T19:38:57.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.817+0000 7f7aef2f2740 10 client.28756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:57.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.817+0000 7f7aef2f2740 20 client.28756 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:57.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.817+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5570b209b9f0 0x5570b24a8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.817+0000 7f7aef2f2740 4 client.28756 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:38:57.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.817+0000 7f7aef2f2740 1 -- 172.21.15.16:0/4250442104 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570b23fb580 con 0x5570b209b9f0 2024-08-28T19:38:57.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.819+0000 7f7ae259c640 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5570b209b9f0 0x5570b24a8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.829+0000 7f7ae259c640 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5570b209b9f0 0x5570b24a8370 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce4b90c44cefca66 server_cookie=51290fe078ffd23e in_seq=0 out_seq=0 2024-08-28T19:38:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.830+0000 7f7ac9ffb640 10 client.28756 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.832+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/4250442104 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5570b23fb580 con 0x5570b209b9f0 2024-08-28T19:38:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.832+0000 7f7ac9ffb640 10 client.28756 handle_command_reply: tid=0 2024-08-28T19:38:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.832+0000 7f7aef2f2740 1 client.28756 shutdown 2024-08-28T19:38:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.832+0000 7f7aef2f2740 1 -- 172.21.15.16:0/4250442104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 msgr2=0x5570b2496710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.832+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b2496710 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7adc103f60 tx=0x7f7adc106d30 comp rx=0 tx=0).stop 2024-08-28T19:38:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.834+0000 7f7aef2f2740 1 -- 172.21.15.16:0/4250442104 shutdown_connections 2024-08-28T19:38:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.834+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b24a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.834+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5570b209b9f0 0x5570b24a8370 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.834+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b2496710 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.834+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/4250442104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 0x5570b24961d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.834+0000 7f7aef2f2740 1 -- 172.21.15.16:0/4250442104 >> 172.21.15.16:0/4250442104 conn(0x5570b20d1460 msgr2=0x5570b248dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.836+0000 7f7ac9ffb640 0 client.28756 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.837+0000 7f7aef2f2740 1 -- 172.21.15.16:0/4250442104 shutdown_connections 2024-08-28T19:38:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.837+0000 7f7aef2f2740 1 -- 172.21.15.16:0/4250442104 wait complete. 2024-08-28T19:38:57.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.837+0000 7f7aef2f2740 20 client.28756 trim_cache size 0 max 16384 2024-08-28T19:38:57.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:57 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:38:57.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:57 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:57 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:57.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:57 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:57.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:57 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:57 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:38:57.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:57 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:57 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:57 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:38:57.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:57 smithi115 ceph-mon[22184]: pgmap v898: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 8.3 MiB/s wr, 649 op/s 2024-08-28T19:38:57.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.872+0000 7f7aef2f2740 1 Processor -- start 2024-08-28T19:38:57.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.872+0000 7f7aef2f2740 1 -- start start 2024-08-28T19:38:57.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.873+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20b41b0 0x5570b248c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.873+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20bc390 0x5570b2497e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.873+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b24a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.873+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae41b73f0 con 0x5570b20bc390 2024-08-28T19:38:57.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.873+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae41b90b0 con 0x5570b20dd070 2024-08-28T19:38:57.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.873+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae41b7b20 con 0x5570b20b41b0 2024-08-28T19:38:57.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.876+0000 7f7ae1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20bc390 0x5570b2497e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.876+0000 7f7ae1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20bc390 0x5570b2497e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43492/0 (socket says 172.21.15.16:43492) 2024-08-28T19:38:57.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.876+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/3000015464 learned_addr learned my addr 172.21.15.16:0/3000015464 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.878+0000 7f7ae259c640 1 --2- 172.21.15.16:0/3000015464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20b41b0 0x5570b248c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.878+0000 7f7ae2d9d640 1 --2- 172.21.15.16:0/3000015464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b24a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.882+0000 7f7ae2d9d640 1 -- 172.21.15.16:0/3000015464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20b41b0 msgr2=0x5570b248c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.882+0000 7f7ae2d9d640 1 --2- 172.21.15.16:0/3000015464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20b41b0 0x5570b248c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.882+0000 7f7ae2d9d640 1 -- 172.21.15.16:0/3000015464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20bc390 msgr2=0x5570b2497e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.882+0000 7f7ae2d9d640 1 --2- 172.21.15.16:0/3000015464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20bc390 0x5570b2497e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.882+0000 7f7ae2d9d640 1 -- 172.21.15.16:0/3000015464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570b209bc20 con 0x5570b20dd070 2024-08-28T19:38:57.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.883+0000 7f7ae2d9d640 1 --2- 172.21.15.16:0/3000015464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b24a22b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009ee0 tx=0x7f7ad400e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9258f7f6d49d740f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:57.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.883+0000 7f7acbfff640 1 -- 172.21.15.16:0/3000015464 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad400ba60 con 0x5570b20dd070 2024-08-28T19:38:57.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.883+0000 7f7acbfff640 1 -- 172.21.15.16:0/3000015464 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad4022070 con 0x5570b20dd070 2024-08-28T19:38:57.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.883+0000 7f7acbfff640 1 -- 172.21.15.16:0/3000015464 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4012390 con 0x5570b20dd070 2024-08-28T19:38:57.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.885+0000 7f7aef2f2740 1 -- 172.21.15.16:0/3000015464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 msgr2=0x5570b24a22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:57.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.885+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/3000015464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b24a22b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009ee0 tx=0x7f7ad400e000 comp rx=0 tx=0).stop 2024-08-28T19:38:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.885+0000 7f7aef2f2740 1 -- 172.21.15.16:0/3000015464 shutdown_connections 2024-08-28T19:38:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.885+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/3000015464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b24a22b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.885+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/3000015464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20bc390 0x5570b2497e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.885+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/3000015464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20b41b0 0x5570b248c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.885+0000 7f7aef2f2740 1 -- 172.21.15.16:0/3000015464 >> 172.21.15.16:0/3000015464 conn(0x5570b20d1460 msgr2=0x5570b24976f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:57.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.890+0000 7f7aef2f2740 1 -- 172.21.15.16:0/3000015464 shutdown_connections 2024-08-28T19:38:57.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.891+0000 7f7aef2f2740 1 -- 172.21.15.16:0/3000015464 wait complete. 2024-08-28T19:38:57.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.891+0000 7f7aef2f2740 1 -- start start 2024-08-28T19:38:57.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.891+0000 7f7aef2f2740 1 Processor -- start 2024-08-28T19:38:57.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.892+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 0x5570b249e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.892+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b249e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.892+0000 7f7aef2f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b250bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.892+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae41b8920 con 0x5570b20b41b0 2024-08-28T19:38:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.892+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae41b81f0 con 0x5570b20dd070 2024-08-28T19:38:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.892+0000 7f7aef2f2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae41b5410 con 0x5570b20bc390 2024-08-28T19:38:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.892+0000 7f7aef2f2740 10 client.0 _ll_register_callbacks cb 0x7f7adc10aa30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:38:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.893+0000 7f7ae1d9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b249e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.893+0000 7f7ae1d9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b249e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:48158/0 (socket says 172.21.15.16:48158) 2024-08-28T19:38:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.893+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/2900746381 learned_addr learned my addr 172.21.15.16:0/2900746381 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:38:57.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.893+0000 7f7ae2d9d640 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b250bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.894+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/2900746381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 msgr2=0x5570b250bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:57.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.894+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b250bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.894+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/2900746381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 msgr2=0x5570b249e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:38:57.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.894+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 0x5570b249e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.894+0000 7f7ae1d9b640 1 -- 172.21.15.16:0/2900746381 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570b249ee40 con 0x5570b20bc390 2024-08-28T19:38:57.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.895+0000 7f7ae1d9b640 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b249e850 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7adc1044b0 tx=0x7f7adc10ac70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98d8c9829dcab132 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:38:57.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.895+0000 7f7ac9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:38:57.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.898+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/2900746381 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0cc7a0 con 0x5570b20bc390 2024-08-28T19:38:57.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.898+0000 7f7aef2f2740 1 -- 172.21.15.16:0/2900746381 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ad4022070 con 0x5570b20bc390 2024-08-28T19:38:57.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.898+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/2900746381 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9490+0+0 (secure 0 0 0) 0x7f7adc0b81a0 con 0x5570b20bc390 2024-08-28T19:38:57.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.899+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/2900746381 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f7adc0ccc20 con 0x5570b20bc390 2024-08-28T19:38:57.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.899+0000 7f7aef2f2740 10 client.28762 fetch_fsmap learned FSMap version 97 2024-08-28T19:38:57.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.899+0000 7f7aef2f2740 1 -- 172.21.15.16:0/2900746381 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5570b249ee40 con 0x5570b20bc390 2024-08-28T19:38:57.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.902+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/2900746381 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad4022070 con 0x5570b20bc390 2024-08-28T19:38:57.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.902+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/2900746381 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 97) ==== 8402+0+0 (secure 0 0 0) 0x7f7adc0ee170 con 0x5570b20bc390 2024-08-28T19:38:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.902+0000 7f7aef2f2740 10 client.28762 fetch_fsmap finished waiting for FSMap version 97 2024-08-28T19:38:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.902+0000 7f7aef2f2740 10 client.28762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:38:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.902+0000 7f7aef2f2740 20 client.28762 populate_metadata read hostname 'smithi016' 2024-08-28T19:38:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.902+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5570b2070da0 0x5570b2486120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:38:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.902+0000 7f7aef2f2740 4 client.28762 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:38:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.902+0000 7f7aef2f2740 1 -- 172.21.15.16:0/2900746381 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5570b23f7020 con 0x5570b2070da0 2024-08-28T19:38:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.902+0000 7f7ae259c640 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5570b2070da0 0x5570b2486120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:38:57.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.905+0000 7f7ae259c640 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5570b2070da0 0x5570b2486120 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e03948372086bd08 server_cookie=eab7672f32d4a37b in_seq=0 out_seq=0 2024-08-28T19:38:57.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.905+0000 7f7ac9ffb640 10 client.28762 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:57.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.906+0000 7f7ac9ffb640 1 -- 172.21.15.16:0/2900746381 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7102 (crc 0 0 0) 0x5570b23f7020 con 0x5570b2070da0 2024-08-28T19:38:57.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.906+0000 7f7ac9ffb640 10 client.28762 handle_command_reply: tid=0 2024-08-28T19:38:57.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.906+0000 7f7aef2f2740 1 client.28762 shutdown 2024-08-28T19:38:57.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.911+0000 7f7aef2f2740 1 -- 172.21.15.16:0/2900746381 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 msgr2=0x5570b249e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:57.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.911+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b249e850 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7adc1044b0 tx=0x7f7adc10ac70 comp rx=0 tx=0).stop 2024-08-28T19:38:57.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.914+0000 7f7aef2f2740 1 -- 172.21.15.16:0/2900746381 shutdown_connections 2024-08-28T19:38:57.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.914+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570b20dd070 0x5570b250bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.914+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5570b2070da0 0x5570b2486120 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.914+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5570b20bc390 0x5570b249e850 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.914+0000 7f7aef2f2740 1 --2- 172.21.15.16:0/2900746381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570b20b41b0 0x5570b249e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.914+0000 7f7aef2f2740 1 -- 172.21.15.16:0/2900746381 >> 172.21.15.16:0/2900746381 conn(0x5570b20d1460 msgr2=0x5570b2497f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:57.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.914+0000 7f7ac9ffb640 0 client.28762 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:38:57.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.920+0000 7f7aef2f2740 1 -- 172.21.15.16:0/2900746381 shutdown_connections 2024-08-28T19:38:57.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.920+0000 7f7aef2f2740 1 -- 172.21.15.16:0/2900746381 wait complete. 2024-08-28T19:38:57.925 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:38:57.925 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:38:57.925 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:38:57.925 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:38:57.925 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "18e2c1b8-628f-4f38-b56c-dcde542d4ddb", 2024-08-28T19:38:57.925 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:55.619275+0000", 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:38:57.926 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:38:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:38:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:38:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.928 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.929 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.931 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.932 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.934 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.935 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:38:57.937 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:38:57.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:57.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:38:57.940 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:38:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.922+0000 7f7aef2f2740 20 client.28762 trim_cache size 0 max 16384 2024-08-28T19:38:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.923+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/2446022399 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7acc0b4e30 msgr2=0x7f7acc0b7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.923+0000 7f7ae8d7d640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7acc0b4e30 0x7f7acc0b7340 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009650 tx=0x7f7ad4006cb0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.923+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/2446022399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4113c50 msgr2=0x7f7ae41a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:38:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.923+0000 7f7ae8d7d640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4113c50 0x7f7ae41a5260 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7adc02f570 tx=0x7f7adc0317f0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.924+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/2446022399 shutdown_connections 2024-08-28T19:38:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.924+0000 7f7ae8d7d640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ae41145a0 0x7f7ae41aa500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.924+0000 7f7ae8d7d640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f7acc0b4e30 0x7f7acc0b7340 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.924+0000 7f7ae8d7d640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4113c50 0x7f7ae41a5260 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.924+0000 7f7ae8d7d640 1 --2- 172.21.15.16:0/2446022399 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ae410e360 0x7f7ae41a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:38:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.924+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/2446022399 >> 172.21.15.16:0/2446022399 conn(0x7f7ae410a160 msgr2=0x7f7ae4113220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:38:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.924+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/2446022399 shutdown_connections 2024-08-28T19:38:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:38:57.924+0000 7f7ae8d7d640 1 -- 172.21.15.16:0/2446022399 wait complete. 2024-08-28T19:38:58.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 24.00 MB/sec execute 154 sec latency 5805.401 ms 2024-08-28T19:38:58.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:58 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:58.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:58 smithi016 ceph-mon[18556]: pgmap v899: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.9 MiB/s wr, 514 op/s 2024-08-28T19:38:58.727 INFO:tasks.cephfs.filesystem:scrub status for tag:18e2c1b8-628f-4f38-b56c-dcde542d4ddb - {'path': '/', 'tag': '18e2c1b8-628f-4f38-b56c-dcde542d4ddb', 'options': 'recursive,force'} 2024-08-28T19:38:58.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:58 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:58.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:58 smithi028 ceph-mon[21769]: pgmap v899: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.9 MiB/s wr, 514 op/s 2024-08-28T19:38:58.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:58 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:38:58.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:58 smithi115 ceph-mon[22184]: pgmap v899: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.9 MiB/s wr, 514 op/s 2024-08-28T19:38:59.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 23.84 MB/sec execute 155 sec latency 6805.521 ms 2024-08-28T19:38:59.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:38:59.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:38:59.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:38:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:38:59.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:38:59.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:38:59.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:38:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:38:59.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:38:59.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:38:59.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:38:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:39:00.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 23.69 MB/sec execute 156 sec latency 7805.617 ms 2024-08-28T19:39:00.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:00 smithi016 ceph-mon[18556]: pgmap v900: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.5 MiB/s wr, 521 op/s 2024-08-28T19:39:00.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:00 smithi028 ceph-mon[21769]: pgmap v900: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.5 MiB/s wr, 521 op/s 2024-08-28T19:39:00.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:00 smithi115 ceph-mon[22184]: pgmap v900: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.5 MiB/s wr, 521 op/s 2024-08-28T19:39:01.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 23.54 MB/sec execute 157 sec latency 8805.716 ms 2024-08-28T19:39:02.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 23.39 MB/sec execute 158 sec latency 9805.811 ms 2024-08-28T19:39:03.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:02 smithi016 ceph-mon[18556]: pgmap v901: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 555 op/s 2024-08-28T19:39:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:02 smithi028 ceph-mon[21769]: pgmap v901: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 555 op/s 2024-08-28T19:39:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:02 smithi115 ceph-mon[22184]: pgmap v901: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 555 op/s 2024-08-28T19:39:03.477 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 23.24 MB/sec execute 159 sec latency 10805.926 ms 2024-08-28T19:39:04.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 23.10 MB/sec execute 160 sec latency 11806.056 ms 2024-08-28T19:39:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:04 smithi016 ceph-mon[18556]: pgmap v902: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 465 op/s 2024-08-28T19:39:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:04 smithi016 ceph-mon[18556]: osdmap e107: 12 total, 12 up, 12 in 2024-08-28T19:39:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:04 smithi028 ceph-mon[21769]: pgmap v902: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 465 op/s 2024-08-28T19:39:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:04 smithi028 ceph-mon[21769]: osdmap e107: 12 total, 12 up, 12 in 2024-08-28T19:39:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:04 smithi115 ceph-mon[22184]: pgmap v902: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 465 op/s 2024-08-28T19:39:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:04 smithi115 ceph-mon[22184]: osdmap e107: 12 total, 12 up, 12 in 2024-08-28T19:39:05.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 22.95 MB/sec execute 161 sec latency 12806.183 ms 2024-08-28T19:39:06.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 22.81 MB/sec execute 162 sec latency 13806.234 ms 2024-08-28T19:39:07.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:06 smithi016 ceph-mon[18556]: pgmap v904: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 404 op/s 2024-08-28T19:39:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:06 smithi028 ceph-mon[21769]: pgmap v904: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 404 op/s 2024-08-28T19:39:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:06 smithi115 ceph-mon[22184]: pgmap v904: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 404 op/s 2024-08-28T19:39:07.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 22.67 MB/sec execute 163 sec latency 14806.349 ms 2024-08-28T19:39:07.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:39:08.003 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:08.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 22.53 MB/sec execute 164 sec latency 15806.414 ms 2024-08-28T19:39:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.511+0000 7fdaf6814640 1 -- 172.21.15.16:0/329242769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf0108ac0 msgr2=0x7fdaf010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.511+0000 7fdaf6814640 1 --2- 172.21.15.16:0/329242769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf0108ac0 0x7fdaf010f2f0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009ca0 tx=0x7fdae802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:39:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.511+0000 7fdaf6814640 1 -- 172.21.15.16:0/329242769 shutdown_connections 2024-08-28T19:39:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.511+0000 7fdaf6814640 1 --2- 172.21.15.16:0/329242769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf0108ac0 0x7fdaf010f2f0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.511+0000 7fdaf6814640 1 --2- 172.21.15.16:0/329242769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaf0108170 0x7fdaf0108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.511+0000 7fdaf6814640 1 --2- 172.21.15.16:0/329242769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaf0102880 0x7fdaf0102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:08.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.511+0000 7fdaf6814640 1 -- 172.21.15.16:0/329242769 >> 172.21.15.16:0/329242769 conn(0x7fdaf0079530 msgr2=0x7fdaf0079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:08.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.511+0000 7fdaf6814640 1 -- 172.21.15.16:0/329242769 shutdown_connections 2024-08-28T19:39:08.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.511+0000 7fdaf6814640 1 -- 172.21.15.16:0/329242769 wait complete. 2024-08-28T19:39:08.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.511+0000 7fdaf6814640 1 Processor -- start 2024-08-28T19:39:08.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaf6814640 1 -- start start 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaf6814640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf0102880 0x7fdaf019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaf6814640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaf0108170 0x7fdaf019a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaf6814640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaf0108ac0 0x7fdaf019fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaf6814640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf0112790 con 0x7fdaf0102880 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaf6814640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf0112610 con 0x7fdaf0108170 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaf6814640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf0112910 con 0x7fdaf0108ac0 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaeffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf0102880 0x7fdaf019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaeffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf0102880 0x7fdaf019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60390/0 (socket says 172.21.15.16:60390) 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaeffff640 1 -- 172.21.15.16:0/4283320810 learned_addr learned my addr 172.21.15.16:0/4283320810 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaef7fe640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaf0108170 0x7fdaf019a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaf4db1640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaf0108ac0 0x7fdaf019fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaeffff640 1 -- 172.21.15.16:0/4283320810 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaf0108ac0 msgr2=0x7fdaf019fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaeffff640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaf0108ac0 0x7fdaf019fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaeffff640 1 -- 172.21.15.16:0/4283320810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaf0108170 msgr2=0x7fdaf019a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.512+0000 7fdaeffff640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaf0108170 0x7fdaf019a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.513+0000 7fdaeffff640 1 -- 172.21.15.16:0/4283320810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf01a0180 con 0x7fdaf0102880 2024-08-28T19:39:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.513+0000 7fdaeffff640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf0102880 0x7fdaf019a210 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fdae000e9f0 tx=0x7fdae000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.513+0000 7fdaed7fa640 1 -- 172.21.15.16:0/4283320810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae000cd80 con 0x7fdaf0102880 2024-08-28T19:39:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.513+0000 7fdaed7fa640 1 -- 172.21.15.16:0/4283320810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae00108e0 con 0x7fdaf0102880 2024-08-28T19:39:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.513+0000 7fdaf6814640 1 -- 172.21.15.16:0/4283320810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaf01a0470 con 0x7fdaf0102880 2024-08-28T19:39:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.513+0000 7fdaed7fa640 1 -- 172.21.15.16:0/4283320810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0018980 con 0x7fdaf0102880 2024-08-28T19:39:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.513+0000 7fdaf6814640 1 -- 172.21.15.16:0/4283320810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaf01a6bd0 con 0x7fdaf0102880 2024-08-28T19:39:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.514+0000 7fdaf6814640 1 -- 172.21.15.16:0/4283320810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaf0103ab0 con 0x7fdaf0102880 2024-08-28T19:39:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.516+0000 7fdaed7fa640 1 -- 172.21.15.16:0/4283320810 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fdae0010430 con 0x7fdaf0102880 2024-08-28T19:39:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.516+0000 7fdaed7fa640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdad80b4ca0 0x7fdad80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.517+0000 7fdaef7fe640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdad80b4ca0 0x7fdad80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.517+0000 7fdaed7fa640 1 -- 172.21.15.16:0/4283320810 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9578+0+0 (secure 0 0 0) 0x7fdae00e6160 con 0x7fdaf0102880 2024-08-28T19:39:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.517+0000 7fdaef7fe640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdad80b4ca0 0x7fdad80b71b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0079980 tx=0x7fdae400a250 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.520+0000 7fdaed7fa640 1 -- 172.21.15.16:0/4283320810 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fdae00ac4e0 con 0x7fdaf0102880 2024-08-28T19:39:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.741+0000 7fdaf6814640 1 -- 172.21.15.16:0/4283320810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdaf0102c90 con 0x7fdaf0102880 2024-08-28T19:39:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.742+0000 7fdaed7fa640 1 -- 172.21.15.16:0/4283320810 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+11990 (secure 0 0 0) 0x7fdae00b5070 con 0x7fdaf0102880 2024-08-28T19:39:08.743 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:39:08.744 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":97,"btime":"2024-08-28T19:38:57:452099+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":97,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:38:56.621804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:39:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 97 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.744+0000 7fdaf6814640 1 -- 172.21.15.16:0/4283320810 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdad80b4ca0 msgr2=0x7fdad80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.744+0000 7fdaf6814640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdad80b4ca0 0x7fdad80b71b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0079980 tx=0x7fdae400a250 comp rx=0 tx=0).stop 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.744+0000 7fdaf6814640 1 -- 172.21.15.16:0/4283320810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf0102880 msgr2=0x7fdaf019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.744+0000 7fdaf6814640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf0102880 0x7fdaf019a210 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fdae000e9f0 tx=0x7fdae000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.745+0000 7fdaf6814640 1 -- 172.21.15.16:0/4283320810 shutdown_connections 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.745+0000 7fdaf6814640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaf0108ac0 0x7fdaf019fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.745+0000 7fdaf6814640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdad80b4ca0 0x7fdad80b71b0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.745+0000 7fdaf6814640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaf0108170 0x7fdaf019a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.745+0000 7fdaf6814640 1 --2- 172.21.15.16:0/4283320810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf0102880 0x7fdaf019a210 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.745+0000 7fdaf6814640 1 -- 172.21.15.16:0/4283320810 >> 172.21.15.16:0/4283320810 conn(0x7fdaf0079530 msgr2=0x7fdaf01017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.745+0000 7fdaf6814640 1 -- 172.21.15.16:0/4283320810 shutdown_connections 2024-08-28T19:39:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:08.745+0000 7fdaf6814640 1 -- 172.21.15.16:0/4283320810 wait complete. 2024-08-28T19:39:09.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:08 smithi016 ceph-mon[18556]: pgmap v905: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 11 MiB/s wr, 389 op/s 2024-08-28T19:39:09.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:08 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/4283320810' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:39:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:08 smithi028 ceph-mon[21769]: pgmap v905: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 11 MiB/s wr, 389 op/s 2024-08-28T19:39:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:08 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/4283320810' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:39:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:08 smithi115 ceph-mon[22184]: pgmap v905: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 11 MiB/s wr, 389 op/s 2024-08-28T19:39:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:08 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/4283320810' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:39:09.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 22.40 MB/sec execute 165 sec latency 16806.705 ms 2024-08-28T19:39:10.448 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id 3c5129 --release --await' 2024-08-28T19:39:10.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id 3c5129 --release --await 2024-08-28T19:39:10.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106839 22.26 MB/sec execute 166 sec latency 17806.757 ms 2024-08-28T19:39:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.169+0000 7fe69bbc2640 1 -- 172.21.15.16:0/2034626760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694108ab0 msgr2=0x7fe69410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.169+0000 7fe69bbc2640 1 --2- 172.21.15.16:0/2034626760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694108ab0 0x7fe69410f2e0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009ca0 tx=0x7fe68c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.169+0000 7fe69bbc2640 1 -- 172.21.15.16:0/2034626760 shutdown_connections 2024-08-28T19:39:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.169+0000 7fe69bbc2640 1 --2- 172.21.15.16:0/2034626760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694108ab0 0x7fe69410f2e0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.169+0000 7fe69bbc2640 1 --2- 172.21.15.16:0/2034626760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe694108160 0x7fe694108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.169+0000 7fe69bbc2640 1 --2- 172.21.15.16:0/2034626760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe694102820 0x7fe694102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.169+0000 7fe69bbc2640 1 -- 172.21.15.16:0/2034626760 >> 172.21.15.16:0/2034626760 conn(0x7fe6940794e0 msgr2=0x7fe6940798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.169+0000 7fe69bbc2640 1 -- 172.21.15.16:0/2034626760 shutdown_connections 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.169+0000 7fe69bbc2640 1 -- 172.21.15.16:0/2034626760 wait complete. 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.170+0000 7fe69bbc2640 1 Processor -- start 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.170+0000 7fe69bbc2640 1 -- start start 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69bbc2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe694102820 0x7fe69419afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69bbc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe694108160 0x7fe69419b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69bbc2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694108ab0 0x7fe69419ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69bbc2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe694112710 con 0x7fe694108ab0 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69bbc2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe694112590 con 0x7fe694108160 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69bbc2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe694112890 con 0x7fe694102820 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69a15f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694108ab0 0x7fe69419ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69a15f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694108ab0 0x7fe69419ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60414/0 (socket says 172.21.15.16:60414) 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69a15f640 1 -- 172.21.15.16:0/54172219 learned_addr learned my addr 172.21.15.16:0/54172219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69915d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe694108160 0x7fe69419b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69995e640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe694102820 0x7fe69419afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69a15f640 1 -- 172.21.15.16:0/54172219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe694102820 msgr2=0x7fe69419afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69a15f640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe694102820 0x7fe69419afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69a15f640 1 -- 172.21.15.16:0/54172219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe694108160 msgr2=0x7fe69419b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69a15f640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe694108160 0x7fe69419b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69a15f640 1 -- 172.21.15.16:0/54172219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6941a6270 con 0x7fe694108ab0 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.171+0000 7fe69a15f640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694108ab0 0x7fe69419ba60 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0098d0 tx=0x7fe68c0317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.172+0000 7fe68affd640 1 -- 172.21.15.16:0/54172219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c003b70 con 0x7fe694108ab0 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.172+0000 7fe68affd640 1 -- 172.21.15.16:0/54172219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe68c003d10 con 0x7fe694108ab0 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.172+0000 7fe68affd640 1 -- 172.21.15.16:0/54172219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c02fa20 con 0x7fe694108ab0 2024-08-28T19:39:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.172+0000 7fe69bbc2640 1 -- 172.21.15.16:0/54172219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6941a6500 con 0x7fe694108ab0 2024-08-28T19:39:11.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.172+0000 7fe69bbc2640 1 -- 172.21.15.16:0/54172219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6941a69f0 con 0x7fe694108ab0 2024-08-28T19:39:11.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe68affd640 1 -- 172.21.15.16:0/54172219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe68c02fc00 con 0x7fe694108ab0 2024-08-28T19:39:11.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe6a2137740 1 Processor -- start 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe6a2137740 1 -- start start 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd2d520 0x5594cc0d2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd04660 0x5594cc0d31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd0c840 0x5594cc0d3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe6a2137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5594cc049a70 con 0x5594cbd0c840 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe6a2137740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5594cc039000 con 0x5594cbd2d520 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe6a2137740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5594cc03cfb0 con 0x5594cbd04660 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe69915d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd0c840 0x5594cc0d3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe69915d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd0c840 0x5594cc0d3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60430/0 (socket says 172.21.15.16:60430) 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.175+0000 7fe69915d640 1 -- 172.21.15.16:0/3284946926 learned_addr learned my addr 172.21.15.16:0/3284946926 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe69915d640 1 --2- 172.21.15.16:0/3284946926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd2d520 0x5594cc0d2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe68affd640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6800b4d50 0x7fe6800b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe69995e640 1 --2- 172.21.15.16:0/3284946926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd04660 0x5594cc0d31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe69a15f640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6800b4d50 0x7fe6800b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe69915d640 1 -- 172.21.15.16:0/3284946926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd04660 msgr2=0x5594cc0d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe69915d640 1 --2- 172.21.15.16:0/3284946926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd04660 0x5594cc0d31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe69915d640 1 -- 172.21.15.16:0/3284946926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd2d520 msgr2=0x5594cc0d2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe69915d640 1 --2- 172.21.15.16:0/3284946926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd2d520 0x5594cc0d2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe69915d640 1 -- 172.21.15.16:0/3284946926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594cbd98b00 con 0x5594cbd0c840 2024-08-28T19:39:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe68affd640 1 -- 172.21.15.16:0/54172219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fe68c106af0 con 0x7fe694108ab0 2024-08-28T19:39:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe69915d640 1 --2- 172.21.15.16:0/3284946926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd0c840 0x5594cc0d3700 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fe684012750 tx=0x7fe684004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17642400815f034a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe67ffff640 1 -- 172.21.15.16:0/3284946926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684012cd0 con 0x5594cbd0c840 2024-08-28T19:39:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe67ffff640 1 -- 172.21.15.16:0/3284946926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe684005020 con 0x5594cbd0c840 2024-08-28T19:39:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.176+0000 7fe67ffff640 1 -- 172.21.15.16:0/3284946926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6840049a0 con 0x5594cbd0c840 2024-08-28T19:39:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.177+0000 7fe6a2137740 1 -- 172.21.15.16:0/3284946926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd0c840 msgr2=0x5594cc0d3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.177+0000 7fe69a15f640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6800b4d50 0x7fe6800b7260 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe68c103f20 tx=0x7fe68c0b4040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.177+0000 7fe6a2137740 1 --2- 172.21.15.16:0/3284946926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd0c840 0x5594cc0d3700 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fe684012750 tx=0x7fe684004730 comp rx=0 tx=0).stop 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.177+0000 7fe6a2137740 1 -- 172.21.15.16:0/3284946926 shutdown_connections 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.177+0000 7fe6a2137740 1 --2- 172.21.15.16:0/3284946926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd0c840 0x5594cc0d3700 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.177+0000 7fe6a2137740 1 --2- 172.21.15.16:0/3284946926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd04660 0x5594cc0d31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.177+0000 7fe6a2137740 1 --2- 172.21.15.16:0/3284946926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd2d520 0x5594cc0d2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.177+0000 7fe6a2137740 1 -- 172.21.15.16:0/3284946926 >> 172.21.15.16:0/3284946926 conn(0x5594cbd21910 msgr2=0x5594cc0d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.177+0000 7fe6a2137740 1 -- 172.21.15.16:0/3284946926 shutdown_connections 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.177+0000 7fe6a2137740 1 -- 172.21.15.16:0/3284946926 wait complete. 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.178+0000 7fe6a2137740 1 -- start start 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.178+0000 7fe6a2137740 1 Processor -- start 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.178+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd04660 0x5594cc0e2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.178+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd0c840 0x5594cc0e3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.178+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0eec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.178+0000 7fe6a2137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5594cc0e5760 con 0x5594cbd2d520 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.178+0000 7fe6a2137740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5594cbffa520 con 0x5594cbd04660 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.178+0000 7fe6a2137740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5594cc0e58e0 con 0x5594cbd0c840 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.178+0000 7fe6a2137740 10 client.0 _ll_register_callbacks cb 0x5594cc032c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe69915d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe69915d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0eec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60434/0 (socket says 172.21.15.16:60434) 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe69915d640 1 -- 172.21.15.16:0/2939843689 learned_addr learned my addr 172.21.15.16:0/2939843689 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe69915d640 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd04660 0x5594cc0e2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe69995e640 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd0c840 0x5594cc0e3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe69915d640 1 -- 172.21.15.16:0/2939843689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd0c840 msgr2=0x5594cc0e3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe69915d640 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd0c840 0x5594cc0e3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe69915d640 1 -- 172.21.15.16:0/2939843689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd04660 msgr2=0x5594cc0e2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe69915d640 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd04660 0x5594cc0e2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe69915d640 1 -- 172.21.15.16:0/2939843689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594cbe0f960 con 0x5594cbd2d520 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe69915d640 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0eec10 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fe684004610 tx=0x7fe684014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a854309bbe84f4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe67d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe67d7fa640 1 -- 172.21.15.16:0/2939843689 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684010980 con 0x5594cbd2d520 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe67d7fa640 1 -- 172.21.15.16:0/2939843689 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fe684010b20 con 0x5594cbd2d520 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe6a2137740 1 -- 172.21.15.16:0/2939843689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5594cbfd89a0 con 0x5594cbd2d520 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe67d7fa640 1 -- 172.21.15.16:0/2939843689 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fe684033050 con 0x5594cbd2d520 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe6a2137740 10 client.19755 fetch_fsmap learned FSMap version 97 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.179+0000 7fe6a2137740 1 -- 172.21.15.16:0/2939843689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5594cc0f1960 con 0x5594cbd2d520 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.180+0000 7fe67d7fa640 1 -- 172.21.15.16:0/2939843689 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5594cbfd89a0 con 0x5594cbd2d520 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.180+0000 7fe67d7fa640 1 -- 172.21.15.16:0/2939843689 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 97) ==== 8402+0+0 (secure 0 0 0) 0x7fe6840053a0 con 0x5594cbd2d520 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.180+0000 7fe6a2137740 10 client.19755 fetch_fsmap finished waiting for FSMap version 97 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.180+0000 7fe6a2137740 10 client.19755 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:39:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.180+0000 7fe6a2137740 20 client.19755 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.180+0000 7fe6a2137740 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5594cbcebea0 0x5594cc0f4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.180+0000 7fe6a2137740 4 client.19755 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:39:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.180+0000 7fe6a2137740 1 -- 172.21.15.16:0/2939843689 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594cc04b9c0 con 0x5594cbcebea0 2024-08-28T19:39:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.180+0000 7fe69995e640 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5594cbcebea0 0x5594cc0f4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.181+0000 7fe67d7fa640 10 client.19755 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.181+0000 7fe69995e640 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5594cbcebea0 0x5594cc0f4cc0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=704586fb082d30f6 server_cookie=d6af9e0c6ca781d1 in_seq=0 out_seq=0 2024-08-28T19:39:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.181+0000 7fe67d7fa640 1 -- 172.21.15.16:0/2939843689 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5594cc04b9c0 con 0x5594cbcebea0 2024-08-28T19:39:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.181+0000 7fe67d7fa640 10 client.19755 handle_command_reply: tid=0 2024-08-28T19:39:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 1 client.19755 shutdown 2024-08-28T19:39:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 1 -- 172.21.15.16:0/2939843689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 msgr2=0x5594cc0eec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0eec10 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fe684004610 tx=0x7fe684014040 comp rx=0 tx=0).stop 2024-08-28T19:39:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 1 -- 172.21.15.16:0/2939843689 shutdown_connections 2024-08-28T19:39:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0eec10 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5594cbcebea0 0x5594cc0f4cc0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd0c840 0x5594cc0e3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 1 --2- 172.21.15.16:0/2939843689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd04660 0x5594cc0e2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe67d7fa640 0 client.19755 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 1 -- 172.21.15.16:0/2939843689 >> 172.21.15.16:0/2939843689 conn(0x5594cbd21910 msgr2=0x5594cc0d4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 1 -- 172.21.15.16:0/2939843689 shutdown_connections 2024-08-28T19:39:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 1 -- 172.21.15.16:0/2939843689 wait complete. 2024-08-28T19:39:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.182+0000 7fe6a2137740 20 client.19755 trim_cache size 0 max 16384 2024-08-28T19:39:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.199+0000 7fe6a2137740 1 Processor -- start 2024-08-28T19:39:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.199+0000 7fe6a2137740 1 -- start start 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd04660 0x5594cc0e46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd0c840 0x5594cc0dfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe6a2137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6941aeae0 con 0x5594cbd2d520 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe6a2137740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6941b07a0 con 0x5594cbd04660 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe6a2137740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe6941af210 con 0x5594cbd0c840 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69915d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69915d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0e01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60436/0 (socket says 172.21.15.16:60436) 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69915d640 1 -- 172.21.15.16:0/2783671663 learned_addr learned my addr 172.21.15.16:0/2783671663 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69915d640 1 --2- 172.21.15.16:0/2783671663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd04660 0x5594cc0e46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69995e640 1 --2- 172.21.15.16:0/2783671663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd0c840 0x5594cc0dfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69915d640 1 -- 172.21.15.16:0/2783671663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd0c840 msgr2=0x5594cc0dfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69915d640 1 --2- 172.21.15.16:0/2783671663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd0c840 0x5594cc0dfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69915d640 1 -- 172.21.15.16:0/2783671663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd04660 msgr2=0x5594cc0e46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69915d640 1 --2- 172.21.15.16:0/2783671663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd04660 0x5594cc0e46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69915d640 1 -- 172.21.15.16:0/2783671663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594cc0cade0 con 0x5594cbd2d520 2024-08-28T19:39:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69995e640 1 --2- 172.21.15.16:0/2783671663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd0c840 0x5594cc0dfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:39:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe69915d640 1 --2- 172.21.15.16:0/2783671663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0e01d0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fe68400bb00 tx=0x7fe684017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf4be1472073398 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.200+0000 7fe67f7fe640 1 -- 172.21.15.16:0/2783671663 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684012e10 con 0x5594cbd2d520 2024-08-28T19:39:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.201+0000 7fe67f7fe640 1 -- 172.21.15.16:0/2783671663 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe684010e30 con 0x5594cbd2d520 2024-08-28T19:39:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.201+0000 7fe67f7fe640 1 -- 172.21.15.16:0/2783671663 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68402ecd0 con 0x5594cbd2d520 2024-08-28T19:39:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.201+0000 7fe6a2137740 1 -- 172.21.15.16:0/2783671663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 msgr2=0x5594cc0e01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.201+0000 7fe6a2137740 1 --2- 172.21.15.16:0/2783671663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0e01d0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fe68400bb00 tx=0x7fe684017040 comp rx=0 tx=0).stop 2024-08-28T19:39:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.201+0000 7fe6a2137740 1 -- 172.21.15.16:0/2783671663 shutdown_connections 2024-08-28T19:39:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.201+0000 7fe6a2137740 1 --2- 172.21.15.16:0/2783671663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc0e01d0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.201+0000 7fe6a2137740 1 --2- 172.21.15.16:0/2783671663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd0c840 0x5594cc0dfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.201+0000 7fe6a2137740 1 --2- 172.21.15.16:0/2783671663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd04660 0x5594cc0e46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.201+0000 7fe6a2137740 1 -- 172.21.15.16:0/2783671663 >> 172.21.15.16:0/2783671663 conn(0x5594cbd21910 msgr2=0x5594cc0e3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.201+0000 7fe6a2137740 1 -- 172.21.15.16:0/2783671663 shutdown_connections 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.201+0000 7fe6a2137740 1 -- 172.21.15.16:0/2783671663 wait complete. 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.202+0000 7fe6a2137740 1 -- start start 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.202+0000 7fe6a2137740 1 Processor -- start 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.202+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd04660 0x5594cc0e6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.202+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd0c840 0x5594cc0e68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.202+0000 7fe6a2137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc15f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.202+0000 7fe6a2137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6941b0010 con 0x5594cbd2d520 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.202+0000 7fe6a2137740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6941af8e0 con 0x5594cbd0c840 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.202+0000 7fe6a2137740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe6941acae0 con 0x5594cbd04660 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.202+0000 7fe6a2137740 10 client.0 _ll_register_callbacks cb 0x7fe68400be40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe69915d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc15f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe69915d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc15f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60446/0 (socket says 172.21.15.16:60446) 2024-08-28T19:39:11.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe69915d640 1 -- 172.21.15.16:0/1635215126 learned_addr learned my addr 172.21.15.16:0/1635215126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe69995e640 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd0c840 0x5594cc0e68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe69915d640 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd04660 0x5594cc0e6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe69915d640 1 -- 172.21.15.16:0/1635215126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd04660 msgr2=0x5594cc0e6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe69915d640 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd04660 0x5594cc0e6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe69915d640 1 -- 172.21.15.16:0/1635215126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd0c840 msgr2=0x5594cc0e68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe69915d640 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd0c840 0x5594cc0e68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe69915d640 1 -- 172.21.15.16:0/1635215126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594cc15fc70 con 0x5594cbd2d520 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe69915d640 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc15f660 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fe68402e550 tx=0x7fe684014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f86cf44d925f7b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe67d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe67d7fa640 1 -- 172.21.15.16:0/1635215126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684012e10 con 0x5594cbd2d520 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe67d7fa640 1 -- 172.21.15.16:0/1635215126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fe68402a040 con 0x5594cbd2d520 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe6a2137740 1 -- 172.21.15.16:0/1635215126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe684010e30 con 0x5594cbd2d520 2024-08-28T19:39:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.203+0000 7fe67d7fa640 1 -- 172.21.15.16:0/1635215126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fe684010b90 con 0x5594cbd2d520 2024-08-28T19:39:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.204+0000 7fe6a2137740 10 client.19761 fetch_fsmap learned FSMap version 97 2024-08-28T19:39:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.204+0000 7fe6a2137740 1 -- 172.21.15.16:0/1635215126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5594cc160ea0 con 0x5594cbd2d520 2024-08-28T19:39:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.204+0000 7fe67d7fa640 1 -- 172.21.15.16:0/1635215126 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe684010e30 con 0x5594cbd2d520 2024-08-28T19:39:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.204+0000 7fe67d7fa640 1 -- 172.21.15.16:0/1635215126 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 97) ==== 8402+0+0 (secure 0 0 0) 0x7fe6840035f0 con 0x5594cbd2d520 2024-08-28T19:39:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.204+0000 7fe6a2137740 10 client.19761 fetch_fsmap finished waiting for FSMap version 97 2024-08-28T19:39:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.204+0000 7fe6a2137740 10 client.19761 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:39:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.204+0000 7fe6a2137740 20 client.19761 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.204+0000 7fe6a2137740 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5594cbcc1250 0x5594cc1636c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.204+0000 7fe6a2137740 4 client.19761 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3c5129", "release": true, "await": true}] 2024-08-28T19:39:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.204+0000 7fe6a2137740 1 -- 172.21.15.16:0/1635215126 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3c5129", "release": true, "await": true}) -- 0x5594cc047460 con 0x5594cbcc1250 2024-08-28T19:39:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.204+0000 7fe69995e640 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5594cbcc1250 0x5594cc1636c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.205+0000 7fe67d7fa640 10 client.19761 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:11.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.205+0000 7fe69995e640 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5594cbcc1250 0x5594cc1636c0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7cb8db6aa5e138f server_cookie=68e6f65d82d6c360 in_seq=0 out_seq=0 2024-08-28T19:39:11.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.323+0000 7fe67d7fa640 1 -- 172.21.15.16:0/1635215126 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5594cc047460 con 0x5594cbcc1250 2024-08-28T19:39:11.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.323+0000 7fe67d7fa640 10 client.19761 handle_command_reply: tid=0 2024-08-28T19:39:11.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 1 client.19761 shutdown 2024-08-28T19:39:11.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 1 -- 172.21.15.16:0/1635215126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 msgr2=0x5594cc15f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:11.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc15f660 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fe68402e550 tx=0x7fe684014040 comp rx=0 tx=0).stop 2024-08-28T19:39:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 1 -- 172.21.15.16:0/1635215126 shutdown_connections 2024-08-28T19:39:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594cbd2d520 0x5594cc15f660 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594cbd0c840 0x5594cc0e68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5594cbcc1250 0x5594cc1636c0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 1 --2- 172.21.15.16:0/1635215126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594cbd04660 0x5594cc0e6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe67d7fa640 0 client.19761 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 1 -- 172.21.15.16:0/1635215126 >> 172.21.15.16:0/1635215126 conn(0x5594cbd21910 msgr2=0x5594cc0e0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 1 -- 172.21.15.16:0/1635215126 shutdown_connections 2024-08-28T19:39:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 1 -- 172.21.15.16:0/1635215126 wait complete. 2024-08-28T19:39:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.324+0000 7fe6a2137740 20 client.19761 trim_cache size 0 max 16384 2024-08-28T19:39:11.327 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 97, 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 56, 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: "3c5129": { 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 56, 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:11.328 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 141.8, 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:11.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:11.330 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:39:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 -- 172.21.15.16:0/54172219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6800b4d50 msgr2=0x7fe6800b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6800b4d50 0x7fe6800b7260 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe68c103f20 tx=0x7fe68c0b4040 comp rx=0 tx=0).stop 2024-08-28T19:39:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 -- 172.21.15.16:0/54172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694108ab0 msgr2=0x7fe69419ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694108ab0 0x7fe69419ba60 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0098d0 tx=0x7fe68c0317b0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 -- 172.21.15.16:0/54172219 shutdown_connections 2024-08-28T19:39:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694108ab0 0x7fe69419ba60 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6800b4d50 0x7fe6800b7260 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe694108160 0x7fe69419b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 --2- 172.21.15.16:0/54172219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe694102820 0x7fe69419afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 -- 172.21.15.16:0/54172219 >> 172.21.15.16:0/54172219 conn(0x7fe6940794e0 msgr2=0x7fe694104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 -- 172.21.15.16:0/54172219 shutdown_connections 2024-08-28T19:39:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:11.325+0000 7fe69bbc2640 1 -- 172.21.15.16:0/54172219 wait complete. 2024-08-28T19:39:11.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106891 22.14 MB/sec execute 167 sec latency 18693.957 ms 2024-08-28T19:39:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:11 smithi016 ceph-mon[18556]: pgmap v906: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 280 op/s 2024-08-28T19:39:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:11 smithi016 ceph-mon[18556]: osdmap e108: 12 total, 12 up, 12 in 2024-08-28T19:39:11.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:11 smithi028 ceph-mon[21769]: pgmap v906: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 280 op/s 2024-08-28T19:39:11.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:11 smithi028 ceph-mon[21769]: osdmap e108: 12 total, 12 up, 12 in 2024-08-28T19:39:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:11 smithi115 ceph-mon[22184]: pgmap v906: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 280 op/s 2024-08-28T19:39:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:11 smithi115 ceph-mon[22184]: osdmap e108: 12 total, 12 up, 12 in 2024-08-28T19:39:11.993 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '3c5129', total seconds elapsed: 26.1 2024-08-28T19:39:12.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 107608 22.18 MB/sec execute 168 sec latency 157.221 ms 2024-08-28T19:39:13.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 108438 22.39 MB/sec execute 169 sec latency 89.739 ms 2024-08-28T19:39:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:13 smithi016 ceph-mon[18556]: pgmap v908: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-08-28T19:39:13.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:13 smithi028 ceph-mon[21769]: pgmap v908: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-08-28T19:39:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:13 smithi115 ceph-mon[22184]: pgmap v908: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-08-28T19:39:14.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 109052 22.46 MB/sec execute 170 sec latency 94.434 ms 2024-08-28T19:39:15.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 109762 22.51 MB/sec execute 171 sec latency 128.684 ms 2024-08-28T19:39:15.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:15 smithi016 ceph-mon[18556]: pgmap v909: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-08-28T19:39:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:15 smithi028 ceph-mon[21769]: pgmap v909: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-08-28T19:39:15.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:15 smithi115 ceph-mon[22184]: pgmap v909: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-08-28T19:39:16.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 110368 22.57 MB/sec execute 172 sec latency 98.064 ms 2024-08-28T19:39:17.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 111109 22.71 MB/sec execute 173 sec latency 79.151 ms 2024-08-28T19:39:17.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:17 smithi016 ceph-mon[18556]: pgmap v910: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 246 op/s 2024-08-28T19:39:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:17 smithi028 ceph-mon[21769]: pgmap v910: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 246 op/s 2024-08-28T19:39:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:17 smithi115 ceph-mon[22184]: pgmap v910: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 246 op/s 2024-08-28T19:39:18.478 INFO:tasks.workunit.client.0.smithi016.stdout: 10 111778 22.72 MB/sec execute 174 sec latency 102.096 ms 2024-08-28T19:39:19.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 112634 22.93 MB/sec execute 175 sec latency 411.190 ms 2024-08-28T19:39:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:19 smithi016 ceph-mon[18556]: pgmap v911: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.7 MiB/s wr, 281 op/s 2024-08-28T19:39:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:39:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:39:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:39:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:39:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:19 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:39:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:19 smithi028 ceph-mon[21769]: pgmap v911: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.7 MiB/s wr, 281 op/s 2024-08-28T19:39:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:39:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:39:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:39:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:39:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:19 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:39:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:19 smithi115 ceph-mon[22184]: pgmap v911: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.7 MiB/s wr, 281 op/s 2024-08-28T19:39:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:39:19.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:39:19.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:39:19.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:39:19.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:19 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:39:20.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 113233 22.96 MB/sec execute 176 sec latency 1411.231 ms 2024-08-28T19:39:21.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 113975 23.04 MB/sec execute 177 sec latency 2411.266 ms 2024-08-28T19:39:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:21 smithi016 ceph-mon[18556]: pgmap v912: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5.4 MiB/s wr, 396 op/s 2024-08-28T19:39:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:21 smithi028 ceph-mon[21769]: pgmap v912: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5.4 MiB/s wr, 396 op/s 2024-08-28T19:39:21.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:21 smithi115 ceph-mon[22184]: pgmap v912: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5.4 MiB/s wr, 396 op/s 2024-08-28T19:39:22.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 114600 23.12 MB/sec execute 178 sec latency 3411.294 ms 2024-08-28T19:39:23.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 115238 23.18 MB/sec execute 179 sec latency 4411.321 ms 2024-08-28T19:39:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:23 smithi016 ceph-mon[18556]: pgmap v913: 129 pgs: 129 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.5 MiB/s wr, 478 op/s 2024-08-28T19:39:23.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:23 smithi028 ceph-mon[21769]: pgmap v913: 129 pgs: 129 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.5 MiB/s wr, 478 op/s 2024-08-28T19:39:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:23 smithi115 ceph-mon[22184]: pgmap v913: 129 pgs: 129 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.5 MiB/s wr, 478 op/s 2024-08-28T19:39:24.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 115860 23.21 MB/sec execute 180 sec latency 5411.349 ms 2024-08-28T19:39:25.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 116492 23.29 MB/sec execute 181 sec latency 6411.378 ms 2024-08-28T19:39:25.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:25 smithi016 ceph-mon[18556]: pgmap v914: 129 pgs: 129 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.7 MiB/s wr, 493 op/s 2024-08-28T19:39:25.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:25 smithi028 ceph-mon[21769]: pgmap v914: 129 pgs: 129 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.7 MiB/s wr, 493 op/s 2024-08-28T19:39:25.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:25 smithi115 ceph-mon[22184]: pgmap v914: 129 pgs: 129 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.7 MiB/s wr, 493 op/s 2024-08-28T19:39:26.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 117202 23.36 MB/sec execute 182 sec latency 7411.409 ms 2024-08-28T19:39:27.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 117797 23.43 MB/sec execute 183 sec latency 7745.079 ms 2024-08-28T19:39:27.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:27 smithi016 ceph-mon[18556]: pgmap v915: 129 pgs: 129 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 8.5 MiB/s wr, 611 op/s 2024-08-28T19:39:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:27 smithi028 ceph-mon[21769]: pgmap v915: 129 pgs: 129 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 8.5 MiB/s wr, 611 op/s 2024-08-28T19:39:27.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:27 smithi115 ceph-mon[22184]: pgmap v915: 129 pgs: 129 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 8.5 MiB/s wr, 611 op/s 2024-08-28T19:39:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 118403 23.49 MB/sec execute 184 sec latency 82.159 ms 2024-08-28T19:39:28.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:39:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:29.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 118952 23.49 MB/sec execute 185 sec latency 164.721 ms 2024-08-28T19:39:29.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:29 smithi016 ceph-mon[18556]: pgmap v916: 129 pgs: 129 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.1 MiB/s wr, 567 op/s 2024-08-28T19:39:29.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.625+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1606364355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7081050a0 msgr2=0x7fc70810d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:29.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.625+0000 7fc70fd0e640 1 --2- 172.21.15.16:0/1606364355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7081050a0 0x7fc70810d5f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009900 tx=0x7fc6f802a410 comp rx=0 tx=0).stop 2024-08-28T19:39:29.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.625+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1606364355 shutdown_connections 2024-08-28T19:39:29.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.625+0000 7fc70fd0e640 1 --2- 172.21.15.16:0/1606364355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc70810dbb0 0x7fc7081100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.625+0000 7fc70fd0e640 1 --2- 172.21.15.16:0/1606364355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7081050a0 0x7fc70810d5f0 secure :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009900 tx=0x7fc6f802a410 comp rx=0 tx=0).stop 2024-08-28T19:39:29.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.625+0000 7fc70fd0e640 1 --2- 172.21.15.16:0/1606364355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc708104750 0x7fc708104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.625+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1606364355 >> 172.21.15.16:0/1606364355 conn(0x7fc7080794d0 msgr2=0x7fc7080798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:29.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.626+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1606364355 shutdown_connections 2024-08-28T19:39:29.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.627+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1606364355 wait complete. 2024-08-28T19:39:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.628+0000 7fc70fd0e640 1 Processor -- start 2024-08-28T19:39:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.628+0000 7fc70fd0e640 1 -- start start 2024-08-28T19:39:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.628+0000 7fc70fd0e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc708104750 0x7fc70819a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.628+0000 7fc70fd0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc70810dbb0 0x7fc70819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.628+0000 7fc70fd0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc70819fb70 0x7fc70819ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.628+0000 7fc70daaa640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc708104750 0x7fc70819a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70e2ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc70819fb70 0x7fc70819ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70d2a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc70810dbb0 0x7fc70819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70daaa640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc708104750 0x7fc70819a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34918/0 (socket says 172.21.15.16:34918) 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70daaa640 1 -- 172.21.15.16:0/1923771572 learned_addr learned my addr 172.21.15.16:0/1923771572 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70e2ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc70819fb70 0x7fc70819ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57182/0 (socket says 172.21.15.16:57182) 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70fd0e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc708113840 con 0x7fc70810dbb0 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1923771572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7081136c0 con 0x7fc70819fb70 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1923771572 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc7081139c0 con 0x7fc708104750 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70e2ab640 1 -- 172.21.15.16:0/1923771572 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc708104750 msgr2=0x7fc70819a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70e2ab640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc708104750 0x7fc70819a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70e2ab640 1 -- 172.21.15.16:0/1923771572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc70810dbb0 msgr2=0x7fc70819a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70e2ab640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc70810dbb0 0x7fc70819a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70e2ab640 1 -- 172.21.15.16:0/1923771572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7081a0750 con 0x7fc70819fb70 2024-08-28T19:39:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70daaa640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc708104750 0x7fc70819a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70d2a9640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc70810dbb0 0x7fc70819a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.629+0000 7fc70e2ab640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc70819fb70 0x7fc70819ff80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc700009fa0 tx=0x7fc70000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.630+0000 7fc6feffd640 1 -- 172.21.15.16:0/1923771572 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc700017dd0 con 0x7fc70819fb70 2024-08-28T19:39:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.630+0000 7fc6feffd640 1 -- 172.21.15.16:0/1923771572 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc700003e20 con 0x7fc70819fb70 2024-08-28T19:39:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.630+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1923771572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc708102370 con 0x7fc70819fb70 2024-08-28T19:39:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.630+0000 7fc6feffd640 1 -- 172.21.15.16:0/1923771572 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70001f440 con 0x7fc70819fb70 2024-08-28T19:39:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.630+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1923771572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7081027c0 con 0x7fc70819fb70 2024-08-28T19:39:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.634+0000 7fc716283740 1 Processor -- start 2024-08-28T19:39:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.635+0000 7fc6feffd640 1 -- 172.21.15.16:0/1923771572 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc700026020 con 0x7fc70819fb70 2024-08-28T19:39:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.636+0000 7fc716283740 1 -- start start 2024-08-28T19:39:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.636+0000 7fc716283740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2223070 0x55dfa25c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc70d2a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2223070 0x55dfa25c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc70d2a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2223070 0x55dfa25c82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55856/0 (socket says 172.21.15.16:55856) 2024-08-28T19:39:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc70d2a9640 1 -- 172.21.15.16:0/780028827 learned_addr learned my addr 172.21.15.16:0/780028827 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc716283740 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa21fa1b0 0x55dfa25c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc716283740 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa2202390 0x55dfa25c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc716283740 1 -- 172.21.15.16:0/780028827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dfa253f630 con 0x55dfa2223070 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc716283740 1 -- 172.21.15.16:0/780028827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dfa252ebc0 con 0x55dfa21fa1b0 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc6feffd640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6f40b4df0 0x7fc6f40b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc716283740 1 -- 172.21.15.16:0/780028827 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55dfa2532b70 con 0x55dfa2202390 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc70daaa640 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa21fa1b0 0x55dfa25c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc6feffd640 1 -- 172.21.15.16:0/1923771572 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fc7000eddc0 con 0x7fc70819fb70 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc70d2a9640 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa2202390 0x55dfa25c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc70daaa640 1 -- 172.21.15.16:0/780028827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa2202390 msgr2=0x55dfa25c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc70daaa640 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa2202390 0x55dfa25c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc70daaa640 1 -- 172.21.15.16:0/780028827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2223070 msgr2=0x55dfa25c82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc70daaa640 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2223070 0x55dfa25c82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc70daaa640 1 -- 172.21.15.16:0/780028827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa228e650 con 0x55dfa21fa1b0 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.637+0000 7fc70d2a9640 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa2202390 0x55dfa25c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:39:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.638+0000 7fc70e2ab640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6f40b4df0 0x7fc6f40b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.638+0000 7fc70daaa640 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa21fa1b0 0x55dfa25c8800 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc7040058c0 tx=0x7fc704009af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a38d38aa0ad41418 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.638+0000 7fc6f3fff640 1 -- 172.21.15.16:0/780028827 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704004030 con 0x55dfa21fa1b0 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.638+0000 7fc6f3fff640 1 -- 172.21.15.16:0/780028827 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7040041d0 con 0x55dfa21fa1b0 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.638+0000 7fc6f3fff640 1 -- 172.21.15.16:0/780028827 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7040118f0 con 0x55dfa21fa1b0 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.639+0000 7fc716283740 1 -- 172.21.15.16:0/780028827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa21fa1b0 msgr2=0x55dfa25c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.639+0000 7fc716283740 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa21fa1b0 0x55dfa25c8800 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc7040058c0 tx=0x7fc704009af0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.639+0000 7fc70e2ab640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6f40b4df0 0x7fc6f40b7300 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc7000eb1a0 tx=0x7fc700017f70 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.639+0000 7fc716283740 1 -- 172.21.15.16:0/780028827 shutdown_connections 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.639+0000 7fc716283740 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa2202390 0x55dfa25c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.639+0000 7fc716283740 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa21fa1b0 0x55dfa25c8800 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.639+0000 7fc716283740 1 --2- 172.21.15.16:0/780028827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2223070 0x55dfa25c82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.639+0000 7fc716283740 1 -- 172.21.15.16:0/780028827 >> 172.21.15.16:0/780028827 conn(0x55dfa2217460 msgr2=0x55dfa25c61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.639+0000 7fc716283740 1 -- 172.21.15.16:0/780028827 shutdown_connections 2024-08-28T19:39:29.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.640+0000 7fc716283740 1 -- 172.21.15.16:0/780028827 wait complete. 2024-08-28T19:39:29.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.640+0000 7fc716283740 1 -- start start 2024-08-28T19:39:29.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.640+0000 7fc716283740 1 Processor -- start 2024-08-28T19:39:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.641+0000 7fc716283740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa21fa1b0 0x55dfa25d3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.641+0000 7fc716283740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.641+0000 7fc716283740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa2223070 0x55dfa25dfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.641+0000 7fc716283740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dfa2391700 con 0x55dfa2202390 2024-08-28T19:39:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.641+0000 7fc716283740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dfa24b5700 con 0x55dfa2223070 2024-08-28T19:39:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.641+0000 7fc716283740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55dfa25d6aa0 con 0x55dfa21fa1b0 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.641+0000 7fc716283740 10 client.0 _ll_register_callbacks cb 0x55dfa24e4dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.641+0000 7fc70daaa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.641+0000 7fc70d2a9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa21fa1b0 0x55dfa25d3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.642+0000 7fc70d2a9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa21fa1b0 0x55dfa25d3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34934/0 (socket says 172.21.15.16:34934) 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.642+0000 7fc70d2a9640 1 -- 172.21.15.16:0/1268509168 learned_addr learned my addr 172.21.15.16:0/1268509168 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.643+0000 7fc70daaa640 1 -- 172.21.15.16:0/1268509168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa21fa1b0 msgr2=0x55dfa25d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.643+0000 7fc70daaa640 1 --2- 172.21.15.16:0/1268509168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa21fa1b0 0x55dfa25d3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.643+0000 7fc70daaa640 1 -- 172.21.15.16:0/1268509168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa2223070 msgr2=0x55dfa25dfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.643+0000 7fc70daaa640 1 --2- 172.21.15.16:0/1268509168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa2223070 0x55dfa25dfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.643+0000 7fc70daaa640 1 -- 172.21.15.16:0/1268509168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa24ee8b0 con 0x55dfa2202390 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.643+0000 7fc70daaa640 1 --2- 172.21.15.16:0/1268509168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d3ff0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fc70401c040 tx=0x7fc704012630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87c1a7a756fb50c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.644+0000 7fc6f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.644+0000 7fc6f17fa640 1 -- 172.21.15.16:0/1268509168 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704028070 con 0x55dfa2202390 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.644+0000 7fc6f17fa640 1 -- 172.21.15.16:0/1268509168 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fc7040035f0 con 0x55dfa2202390 2024-08-28T19:39:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.644+0000 7fc716283740 1 -- 172.21.15.16:0/1268509168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7040041d0 con 0x55dfa2202390 2024-08-28T19:39:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.644+0000 7fc6f17fa640 1 -- 172.21.15.16:0/1268509168 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fc704032050 con 0x55dfa2202390 2024-08-28T19:39:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.644+0000 7fc716283740 10 client.19773 fetch_fsmap learned FSMap version 97 2024-08-28T19:39:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.644+0000 7fc716283740 1 -- 172.21.15.16:0/1268509168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55dfa25e28b0 con 0x55dfa2202390 2024-08-28T19:39:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.645+0000 7fc6f17fa640 1 -- 172.21.15.16:0/1268509168 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7040046c0 con 0x55dfa2202390 2024-08-28T19:39:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.646+0000 7fc6f17fa640 1 -- 172.21.15.16:0/1268509168 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 97) ==== 8402+0+0 (secure 0 0 0) 0x7fc7040115f0 con 0x55dfa2202390 2024-08-28T19:39:29.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.646+0000 7fc716283740 10 client.19773 fetch_fsmap finished waiting for FSMap version 97 2024-08-28T19:39:29.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.646+0000 7fc716283740 10 client.19773 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:39:29.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.646+0000 7fc716283740 20 client.19773 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:29.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.646+0000 7fc716283740 1 --2- 172.21.15.16:0/1268509168 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dfa21e19f0 0x55dfa25e5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.646+0000 7fc716283740 4 client.19773 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:39:29.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.646+0000 7fc716283740 1 -- 172.21.15.16:0/1268509168 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc7040041d0 con 0x55dfa21e19f0 2024-08-28T19:39:29.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.647+0000 7fc70d2a9640 1 --2- 172.21.15.16:0/1268509168 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dfa21e19f0 0x55dfa25e5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.647+0000 7fc6f17fa640 10 client.19773 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:29.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.647+0000 7fc70d2a9640 1 --2- 172.21.15.16:0/1268509168 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dfa21e19f0 0x55dfa25e5d80 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fa03febc509f8f7 server_cookie=47331802ba0f87da in_seq=0 out_seq=0 2024-08-28T19:39:29.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.649+0000 7fc6f17fa640 1 -- 172.21.15.16:0/1268509168 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fc7040041d0 con 0x55dfa21e19f0 2024-08-28T19:39:29.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.649+0000 7fc6f17fa640 10 client.19773 handle_command_reply: tid=0 2024-08-28T19:39:29.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.649+0000 7fc716283740 1 client.19773 shutdown 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc716283740 1 -- 172.21.15.16:0/1268509168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 msgr2=0x55dfa25d3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc716283740 1 --2- 172.21.15.16:0/1268509168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d3ff0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fc70401c040 tx=0x7fc704012630 comp rx=0 tx=0).stop 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc716283740 1 -- 172.21.15.16:0/1268509168 shutdown_connections 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc716283740 1 --2- 172.21.15.16:0/1268509168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa2223070 0x55dfa25dfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc716283740 1 --2- 172.21.15.16:0/1268509168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d3ff0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc716283740 1 --2- 172.21.15.16:0/1268509168 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dfa21e19f0 0x55dfa25e5d80 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc716283740 1 --2- 172.21.15.16:0/1268509168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa21fa1b0 0x55dfa25d3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc716283740 1 -- 172.21.15.16:0/1268509168 >> 172.21.15.16:0/1268509168 conn(0x55dfa2217460 msgr2=0x55dfa25cb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc6f17fa640 0 client.19773 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc716283740 1 -- 172.21.15.16:0/1268509168 shutdown_connections 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc716283740 1 -- 172.21.15.16:0/1268509168 wait complete. 2024-08-28T19:39:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.650+0000 7fc716283740 20 client.19773 trim_cache size 0 max 16384 2024-08-28T19:39:29.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.686+0000 7fc716283740 1 Processor -- start 2024-08-28T19:39:29.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.686+0000 7fc716283740 1 -- start start 2024-08-28T19:39:29.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.687+0000 7fc716283740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa21fa1b0 0x55dfa25d3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.687+0000 7fc716283740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.687+0000 7fc716283740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa2223070 0x55dfa25cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.687+0000 7fc716283740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc708074db0 con 0x55dfa2202390 2024-08-28T19:39:29.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.687+0000 7fc716283740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7081b93e0 con 0x55dfa21fa1b0 2024-08-28T19:39:29.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.687+0000 7fc716283740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc7081b7e50 con 0x55dfa2223070 2024-08-28T19:39:29.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.687+0000 7fc70daaa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.688+0000 7fc70daaa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55862/0 (socket says 172.21.15.16:55862) 2024-08-28T19:39:29.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.688+0000 7fc70daaa640 1 -- 172.21.15.16:0/2325258548 learned_addr learned my addr 172.21.15.16:0/2325258548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:29.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.688+0000 7fc70d2a9640 1 --2- 172.21.15.16:0/2325258548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa2223070 0x55dfa25cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.688+0000 7fc70d2a9640 1 --2- 172.21.15.16:0/2325258548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa21fa1b0 0x55dfa25d3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.689+0000 7fc70daaa640 1 -- 172.21.15.16:0/2325258548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa2223070 msgr2=0x55dfa25cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:29.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.689+0000 7fc70daaa640 1 --2- 172.21.15.16:0/2325258548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa2223070 0x55dfa25cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.689+0000 7fc70daaa640 1 -- 172.21.15.16:0/2325258548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa21fa1b0 msgr2=0x55dfa25d3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:29.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.689+0000 7fc70daaa640 1 --2- 172.21.15.16:0/2325258548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa21fa1b0 0x55dfa25d3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.689+0000 7fc70daaa640 1 -- 172.21.15.16:0/2325258548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa25ca5b0 con 0x55dfa2202390 2024-08-28T19:39:29.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.689+0000 7fc70daaa640 1 --2- 172.21.15.16:0/2325258548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d3da0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fc704005f90 tx=0x7fc704005fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6828e163099140 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:29.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.690+0000 7fc6f37fe640 1 -- 172.21.15.16:0/2325258548 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704028070 con 0x55dfa2202390 2024-08-28T19:39:29.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.690+0000 7fc6f37fe640 1 -- 172.21.15.16:0/2325258548 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc704003aa0 con 0x55dfa2202390 2024-08-28T19:39:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.690+0000 7fc6f37fe640 1 -- 172.21.15.16:0/2325258548 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704004590 con 0x55dfa2202390 2024-08-28T19:39:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.692+0000 7fc716283740 1 -- 172.21.15.16:0/2325258548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 msgr2=0x55dfa25d3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.692+0000 7fc716283740 1 --2- 172.21.15.16:0/2325258548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d3da0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fc704005f90 tx=0x7fc704005fc0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.693+0000 7fc716283740 1 -- 172.21.15.16:0/2325258548 shutdown_connections 2024-08-28T19:39:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.693+0000 7fc716283740 1 --2- 172.21.15.16:0/2325258548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa2223070 0x55dfa25cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.693+0000 7fc716283740 1 --2- 172.21.15.16:0/2325258548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d3da0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.693+0000 7fc716283740 1 --2- 172.21.15.16:0/2325258548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa21fa1b0 0x55dfa25d3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.693+0000 7fc716283740 1 -- 172.21.15.16:0/2325258548 >> 172.21.15.16:0/2325258548 conn(0x55dfa2217460 msgr2=0x55dfa25d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.694+0000 7fc716283740 1 -- 172.21.15.16:0/2325258548 shutdown_connections 2024-08-28T19:39:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.695+0000 7fc716283740 1 -- 172.21.15.16:0/2325258548 wait complete. 2024-08-28T19:39:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.695+0000 7fc716283740 1 -- start start 2024-08-28T19:39:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.696+0000 7fc716283740 1 Processor -- start 2024-08-28T19:39:29.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.696+0000 7fc716283740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa21fa1b0 0x55dfa25dc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.696+0000 7fc716283740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.696+0000 7fc716283740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa2223070 0x55dfa25d7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.696+0000 7fc716283740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7081b8c50 con 0x55dfa2202390 2024-08-28T19:39:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.696+0000 7fc716283740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7081b8520 con 0x55dfa2223070 2024-08-28T19:39:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.696+0000 7fc716283740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc708072e40 con 0x55dfa21fa1b0 2024-08-28T19:39:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.696+0000 7fc716283740 10 client.0 _ll_register_callbacks cb 0x7fc704012c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.697+0000 7fc70d2a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa2223070 0x55dfa25d7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.697+0000 7fc70d2a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa2223070 0x55dfa25d7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57216/0 (socket says 172.21.15.16:57216) 2024-08-28T19:39:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.697+0000 7fc70d2a9640 1 -- 172.21.15.16:0/3945108913 learned_addr learned my addr 172.21.15.16:0/3945108913 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.697+0000 7fc70daaa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.697+0000 7fc70d2a9640 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa21fa1b0 0x55dfa25dc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.697+0000 7fc70d2a9640 1 -- 172.21.15.16:0/3945108913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa21fa1b0 msgr2=0x55dfa25dc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.697+0000 7fc70d2a9640 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa21fa1b0 0x55dfa25dc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.697+0000 7fc70d2a9640 1 -- 172.21.15.16:0/3945108913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 msgr2=0x55dfa25d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.697+0000 7fc70d2a9640 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.697+0000 7fc70d2a9640 1 -- 172.21.15.16:0/3945108913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa25d85d0 con 0x55dfa2223070 2024-08-28T19:39:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.697+0000 7fc70d2a9640 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa2223070 0x55dfa25d7e90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc6f802aa00 tx=0x7fc6f8040040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ce99eafa7bdb740 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.698+0000 7fc6f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:39:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.698+0000 7fc6f17fa640 1 -- 172.21.15.16:0/3945108913 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f804b040 con 0x55dfa2223070 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.698+0000 7fc716283740 1 -- 172.21.15.16:0/3945108913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc704003aa0 con 0x55dfa2223070 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.698+0000 7fc6f17fa640 1 -- 172.21.15.16:0/3945108913 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fc6f8053440 con 0x55dfa2223070 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.698+0000 7fc6f17fa640 1 -- 172.21.15.16:0/3945108913 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fc6f8000e60 con 0x55dfa2223070 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.698+0000 7fc716283740 10 client.29216 fetch_fsmap learned FSMap version 97 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.698+0000 7fc716283740 1 -- 172.21.15.16:0/3945108913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55dfa26505a0 con 0x55dfa2223070 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.699+0000 7fc6f17fa640 1 -- 172.21.15.16:0/3945108913 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc704003aa0 con 0x55dfa2223070 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.699+0000 7fc6f17fa640 1 -- 172.21.15.16:0/3945108913 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 97) ==== 8402+0+0 (secure 0 0 0) 0x7fc6f8051020 con 0x55dfa2223070 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.699+0000 7fc716283740 10 client.29216 fetch_fsmap finished waiting for FSMap version 97 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.699+0000 7fc716283740 10 client.29216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.699+0000 7fc716283740 20 client.29216 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.699+0000 7fc716283740 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dfa21b6da0 0x55dfa2652d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.699+0000 7fc716283740 4 client.29216 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:39:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.699+0000 7fc716283740 1 -- 172.21.15.16:0/3945108913 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dfa2541580 con 0x55dfa21b6da0 2024-08-28T19:39:29.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.699+0000 7fc70daaa640 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dfa21b6da0 0x55dfa2652d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:29.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.700+0000 7fc70daaa640 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dfa21b6da0 0x55dfa2652d50 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daf2d8f90f3d940d server_cookie=debf5fbc5172a6 in_seq=0 out_seq=0 2024-08-28T19:39:29.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.700+0000 7fc6f17fa640 10 client.29216 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:29.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.701+0000 7fc6f17fa640 1 -- 172.21.15.16:0/3945108913 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+6906 (crc 0 0 0) 0x55dfa2541580 con 0x55dfa21b6da0 2024-08-28T19:39:29.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.701+0000 7fc6f17fa640 10 client.29216 handle_command_reply: tid=0 2024-08-28T19:39:29.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.701+0000 7fc716283740 1 client.29216 shutdown 2024-08-28T19:39:29.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.701+0000 7fc716283740 1 -- 172.21.15.16:0/3945108913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa2223070 msgr2=0x55dfa25d7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:29.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.701+0000 7fc716283740 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa2223070 0x55dfa25d7e90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc6f802aa00 tx=0x7fc6f8040040 comp rx=0 tx=0).stop 2024-08-28T19:39:29.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.702+0000 7fc716283740 1 -- 172.21.15.16:0/3945108913 shutdown_connections 2024-08-28T19:39:29.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.702+0000 7fc716283740 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfa2223070 0x55dfa25d7e90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.702+0000 7fc716283740 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfa2202390 0x55dfa25d7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.702+0000 7fc716283740 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55dfa21b6da0 0x55dfa2652d50 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.702+0000 7fc6f17fa640 0 client.29216 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:29.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.702+0000 7fc716283740 1 --2- 172.21.15.16:0/3945108913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55dfa21fa1b0 0x55dfa25dc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.702+0000 7fc716283740 1 -- 172.21.15.16:0/3945108913 >> 172.21.15.16:0/3945108913 conn(0x55dfa2217460 msgr2=0x55dfa25d3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:29.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.702+0000 7fc716283740 1 -- 172.21.15.16:0/3945108913 shutdown_connections 2024-08-28T19:39:29.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.702+0000 7fc716283740 1 -- 172.21.15.16:0/3945108913 wait complete. 2024-08-28T19:39:29.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.703+0000 7fc716283740 20 client.29216 trim_cache size 0 max 16384 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:55.619275+0000", 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.705 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:39:29.708 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.709 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.711 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.713 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:39:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:39:29.715 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:39:29.716 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:29.717 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:39:29.718 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:39:29.718 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:29.718 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:29.718 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:29.718 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:29.718 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:29.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:29.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:29.718 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.704+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1923771572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6f40b4df0 msgr2=0x7fc6f40b7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.704+0000 7fc70fd0e640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6f40b4df0 0x7fc6f40b7300 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc7000eb1a0 tx=0x7fc700017f70 comp rx=0 tx=0).stop 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.704+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1923771572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc70819fb70 msgr2=0x7fc70819ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.704+0000 7fc70fd0e640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc70819fb70 0x7fc70819ff80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc700009fa0 tx=0x7fc70000c6a0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.704+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1923771572 shutdown_connections 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.704+0000 7fc70fd0e640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc6f40b4df0 0x7fc6f40b7300 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.704+0000 7fc70fd0e640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc70819fb70 0x7fc70819ff80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.704+0000 7fc70fd0e640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc70810dbb0 0x7fc70819a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.704+0000 7fc70fd0e640 1 --2- 172.21.15.16:0/1923771572 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc708104750 0x7fc70819a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.704+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1923771572 >> 172.21.15.16:0/1923771572 conn(0x7fc7080794d0 msgr2=0x7fc7081006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.705+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1923771572 shutdown_connections 2024-08-28T19:39:29.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:29.705+0000 7fc70fd0e640 1 -- 172.21.15.16:0/1923771572 wait complete. 2024-08-28T19:39:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:29 smithi028 ceph-mon[21769]: pgmap v916: 129 pgs: 129 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.1 MiB/s wr, 567 op/s 2024-08-28T19:39:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:29 smithi115 ceph-mon[22184]: pgmap v916: 129 pgs: 129 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.1 MiB/s wr, 567 op/s 2024-08-28T19:39:30.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 119599 23.54 MB/sec execute 186 sec latency 168.999 ms 2024-08-28T19:39:30.640 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:39:30.640 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:39:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:31.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 120344 23.62 MB/sec execute 187 sec latency 112.377 ms 2024-08-28T19:39:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:31 smithi016 ceph-mon[18556]: pgmap v917: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 8.6 MiB/s wr, 578 op/s 2024-08-28T19:39:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:31 smithi028 ceph-mon[21769]: pgmap v917: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 8.6 MiB/s wr, 578 op/s 2024-08-28T19:39:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:31 smithi115 ceph-mon[22184]: pgmap v917: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 8.6 MiB/s wr, 578 op/s 2024-08-28T19:39:32.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 120822 23.65 MB/sec execute 188 sec latency 395.068 ms 2024-08-28T19:39:33.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 120850 23.53 MB/sec execute 189 sec latency 1371.823 ms 2024-08-28T19:39:33.672 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:33 smithi115 ceph-mon[22184]: pgmap v918: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.3 MiB/s wr, 582 op/s 2024-08-28T19:39:33.672 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:33 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:39:33.682 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:33 smithi016 ceph-mon[18556]: pgmap v918: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.3 MiB/s wr, 582 op/s 2024-08-28T19:39:33.682 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:33 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:39:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:33 smithi028 ceph-mon[21769]: pgmap v918: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.3 MiB/s wr, 582 op/s 2024-08-28T19:39:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:33 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.846+0000 7fcddf59e640 1 -- 172.21.15.16:0/3529496727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde011c940 msgr2=0x7fcde011ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.846+0000 7fcddf59e640 1 --2- 172.21.15.16:0/3529496727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde011c940 0x7fcde011ed60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8009ca0 tx=0x7fcdd802a8d0 comp rx=0 tx=0).stop 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.847+0000 7fcddf59e640 1 -- 172.21.15.16:0/3529496727 shutdown_connections 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.847+0000 7fcddf59e640 1 --2- 172.21.15.16:0/3529496727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde011c940 0x7fcde011ed60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.847+0000 7fcddf59e640 1 --2- 172.21.15.16:0/3529496727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcde010d990 0x7fcde010dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.847+0000 7fcddf59e640 1 --2- 172.21.15.16:0/3529496727 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcde0111a70 0x7fcde010d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.847+0000 7fcddf59e640 1 -- 172.21.15.16:0/3529496727 >> 172.21.15.16:0/3529496727 conn(0x7fcde006e790 msgr2=0x7fcde006ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.856+0000 7fcddf59e640 1 -- 172.21.15.16:0/3529496727 shutdown_connections 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.858+0000 7fcddf59e640 1 -- 172.21.15.16:0/3529496727 wait complete. 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.859+0000 7fcddf59e640 1 Processor -- start 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.861+0000 7fcddf59e640 1 -- start start 2024-08-28T19:39:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.862+0000 7fcddf59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcde010d990 0x7fcde011c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.862+0000 7fcddf59e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcde0111a70 0x7fcde0115440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.862+0000 7fcddf59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde011c940 0x7fcde0115980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.862+0000 7fcddf59e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcde0122500 con 0x7fcde010d990 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.862+0000 7fcddf59e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcde0122380 con 0x7fcde011c940 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.862+0000 7fcddf59e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcde0122680 con 0x7fcde0111a70 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.863+0000 7fcdddd9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcde0111a70 0x7fcde0115440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.863+0000 7fcdddd9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcde0111a70 0x7fcde0115440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34948/0 (socket says 172.21.15.16:34948) 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.863+0000 7fcdddd9b640 1 -- 172.21.15.16:0/75370255 learned_addr learned my addr 172.21.15.16:0/75370255 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.863+0000 7fcdde59c640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcde010d990 0x7fcde011c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.864+0000 7fcdde59c640 1 -- 172.21.15.16:0/75370255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcde0111a70 msgr2=0x7fcde0115440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.864+0000 7fcdde59c640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcde0111a70 0x7fcde0115440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.864+0000 7fcdde59c640 1 -- 172.21.15.16:0/75370255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde011c940 msgr2=0x7fcde0115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.864+0000 7fcdde59c640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde011c940 0x7fcde0115980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.864+0000 7fcdde59c640 1 -- 172.21.15.16:0/75370255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde01161a0 con 0x7fcde010d990 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.865+0000 7fcdde59c640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcde010d990 0x7fcde011c170 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400b3c0 tx=0x7fcdd400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.870+0000 7fcdcf7fe640 1 -- 172.21.15.16:0/75370255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4011830 con 0x7fcde010d990 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.870+0000 7fcddf59e640 1 -- 172.21.15.16:0/75370255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde011ae40 con 0x7fcde010d990 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.870+0000 7fcddf59e640 1 -- 172.21.15.16:0/75370255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde011b400 con 0x7fcde010d990 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.870+0000 7fcdcf7fe640 1 -- 172.21.15.16:0/75370255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd4016070 con 0x7fcde010d990 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.871+0000 7fcdcf7fe640 1 -- 172.21.15.16:0/75370255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4010e20 con 0x7fcde010d990 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.874+0000 7fcddf59e640 1 -- 172.21.15.16:0/75370255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc00052c0 con 0x7fcde010d990 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.876+0000 7fcdcf7fe640 1 -- 172.21.15.16:0/75370255 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fcdd4010380 con 0x7fcde010d990 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.877+0000 7fcdcf7fe640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcdc80b4d00 0x7fcdc80b7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.878+0000 7fcdcf7fe640 1 -- 172.21.15.16:0/75370255 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fcdd40e4e10 con 0x7fcde010d990 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.878+0000 7fcdddd9b640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcdc80b4d00 0x7fcdc80b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.883+0000 7fcdddd9b640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcdc80b4d00 0x7fcdc80b7210 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fcde010fb60 tx=0x7fcdd0005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:33.890+0000 7fcdcf7fe640 1 -- 172.21.15.16:0/75370255 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fcdd40ab7d0 con 0x7fcde010d990 2024-08-28T19:39:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.327+0000 7fcddf59e640 1 -- 172.21.15.16:0/75370255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdc0005880 con 0x7fcde010d990 2024-08-28T19:39:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.329+0000 7fcdcf7fe640 1 -- 172.21.15.16:0/75370255 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) ==== 94+0+11992 (secure 0 0 0) 0x7fcdd40ab0d0 con 0x7fcde010d990 2024-08-28T19:39:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:39:34.330 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":98,"btime":"2024-08-28T19:39:32:407292+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":98,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:39:31.629237+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:39:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 98 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.332+0000 7fcddf59e640 1 -- 172.21.15.16:0/75370255 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcdc80b4d00 msgr2=0x7fcdc80b7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.332+0000 7fcddf59e640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcdc80b4d00 0x7fcdc80b7210 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fcde010fb60 tx=0x7fcdd0005f90 comp rx=0 tx=0).stop 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.332+0000 7fcddf59e640 1 -- 172.21.15.16:0/75370255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcde010d990 msgr2=0x7fcde011c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.332+0000 7fcddf59e640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcde010d990 0x7fcde011c170 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400b3c0 tx=0x7fcdd400b880 comp rx=0 tx=0).stop 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.332+0000 7fcddf59e640 1 -- 172.21.15.16:0/75370255 shutdown_connections 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.332+0000 7fcddf59e640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcdc80b4d00 0x7fcdc80b7210 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.332+0000 7fcddf59e640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde011c940 0x7fcde0115980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.332+0000 7fcddf59e640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcde0111a70 0x7fcde0115440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.332+0000 7fcddf59e640 1 --2- 172.21.15.16:0/75370255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcde010d990 0x7fcde011c170 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.332+0000 7fcddf59e640 1 -- 172.21.15.16:0/75370255 >> 172.21.15.16:0/75370255 conn(0x7fcde006e790 msgr2=0x7fcde0110720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.332+0000 7fcddf59e640 1 -- 172.21.15.16:0/75370255 shutdown_connections 2024-08-28T19:39:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:34.333+0000 7fcddf59e640 1 -- 172.21.15.16:0/75370255 wait complete. 2024-08-28T19:39:34.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 120981 23.47 MB/sec execute 190 sec latency 2174.432 ms 2024-08-28T19:39:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:34 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/75370255' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:39:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:34 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/75370255' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:39:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:34 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/75370255' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:39:35.265 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:39:35.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 121278 23.42 MB/sec execute 191 sec latency 243.595 ms 2024-08-28T19:39:35.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:35 smithi016 ceph-mon[18556]: pgmap v919: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.8 MiB/s wr, 546 op/s 2024-08-28T19:39:35.713 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:35 smithi028 ceph-mon[21769]: pgmap v919: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.8 MiB/s wr, 546 op/s 2024-08-28T19:39:35.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:35 smithi115 ceph-mon[22184]: pgmap v919: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.8 MiB/s wr, 546 op/s 2024-08-28T19:39:36.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.334+0000 7f349959a640 1 -- 172.21.15.16:0/4137704516 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34840318a0 con 0x7f349c10f870 2024-08-28T19:39:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.335+0000 7f34a0c52640 1 -- 172.21.15.16:0/4137704516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349c10f870 msgr2=0x7f349c10fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.335+0000 7f34a0c52640 1 --2- 172.21.15.16:0/4137704516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349c10f870 0x7f349c10fc80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3484009900 tx=0x7f348402a410 comp rx=0 tx=0).stop 2024-08-28T19:39:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.335+0000 7f34a0c52640 1 -- 172.21.15.16:0/4137704516 shutdown_connections 2024-08-28T19:39:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.335+0000 7f34a0c52640 1 --2- 172.21.15.16:0/4137704516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f349c1101c0 0x7f349c1169f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.335+0000 7f34a0c52640 1 --2- 172.21.15.16:0/4137704516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349c10f870 0x7f349c10fc80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.335+0000 7f34a0c52640 1 --2- 172.21.15.16:0/4137704516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f349c109f30 0x7f349c10a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.336+0000 7f34a0c52640 1 -- 172.21.15.16:0/4137704516 >> 172.21.15.16:0/4137704516 conn(0x7f349c077e90 msgr2=0x7f349c107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.336+0000 7f34a0c52640 1 -- 172.21.15.16:0/4137704516 shutdown_connections 2024-08-28T19:39:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.336+0000 7f34a0c52640 1 -- 172.21.15.16:0/4137704516 wait complete. 2024-08-28T19:39:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.337+0000 7f34a0c52640 1 Processor -- start 2024-08-28T19:39:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.337+0000 7f34a0c52640 1 -- start start 2024-08-28T19:39:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.337+0000 7f34a0c52640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f349c109f30 0x7f349c1a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.337+0000 7f34a0c52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f349c10f870 0x7f349c1a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.337+0000 7f34a0c52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349c1101c0 0x7f349c1a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.337+0000 7f34a0c52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f349c119ea0 con 0x7f349c10f870 2024-08-28T19:39:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.337+0000 7f34a0c52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f349c119d20 con 0x7f349c1101c0 2024-08-28T19:39:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.337+0000 7f34a0c52640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f349c11a020 con 0x7f349c109f30 2024-08-28T19:39:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.338+0000 7f349ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349c1101c0 0x7f349c1a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.338+0000 7f349a59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f349c109f30 0x7f349c1a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.338+0000 7f349ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349c1101c0 0x7f349c1a60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51398/0 (socket says 172.21.15.16:51398) 2024-08-28T19:39:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.338+0000 7f349ad9d640 1 -- 172.21.15.16:0/3636136603 learned_addr learned my addr 172.21.15.16:0/3636136603 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.338+0000 7f349a59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f349c109f30 0x7f349c1a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34300/0 (socket says 172.21.15.16:34300) 2024-08-28T19:39:36.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.338+0000 7f349a59c640 1 -- 172.21.15.16:0/3636136603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349c1101c0 msgr2=0x7f349c1a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:36.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.338+0000 7f349a59c640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349c1101c0 0x7f349c1a60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.338+0000 7f349a59c640 1 -- 172.21.15.16:0/3636136603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f349c10f870 msgr2=0x7f349c1a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:36.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.338+0000 7f349a59c640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f349c10f870 0x7f349c1a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.338+0000 7f349a59c640 1 -- 172.21.15.16:0/3636136603 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349c1a6820 con 0x7f349c109f30 2024-08-28T19:39:36.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.338+0000 7f349ad9d640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349c1101c0 0x7f349c1a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:39:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.339+0000 7f349a59c640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f349c109f30 0x7f349c1a08b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f349400b3f0 tx=0x7f349400b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.339+0000 7f34937fe640 1 -- 172.21.15.16:0/3636136603 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3494015020 con 0x7f349c109f30 2024-08-28T19:39:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.339+0000 7f34937fe640 1 -- 172.21.15.16:0/3636136603 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3494011ce0 con 0x7f349c109f30 2024-08-28T19:39:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.339+0000 7f34937fe640 1 -- 172.21.15.16:0/3636136603 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3494010d80 con 0x7f349c109f30 2024-08-28T19:39:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.340+0000 7f34a0c52640 1 -- 172.21.15.16:0/3636136603 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f349c1a6b10 con 0x7f349c109f30 2024-08-28T19:39:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.340+0000 7f34a0c52640 1 -- 172.21.15.16:0/3636136603 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f349c1ad270 con 0x7f349c109f30 2024-08-28T19:39:36.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.343+0000 7f34937fe640 1 -- 172.21.15.16:0/3636136603 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3494018020 con 0x7f349c109f30 2024-08-28T19:39:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.343+0000 7f34a71c7740 1 Processor -- start 2024-08-28T19:39:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.344+0000 7f34a71c7740 1 -- start start 2024-08-28T19:39:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.344+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa02094070 0x55aa02439310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa0206b1b0 0x55aa02439850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02073390 0x55aa02439d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f34a71c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aa023b0680 con 0x55aa0206b1b0 2024-08-28T19:39:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f34a71c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aa0239fc10 con 0x55aa02073390 2024-08-28T19:39:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f34a71c7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55aa023a3bc0 con 0x55aa02094070 2024-08-28T19:39:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f349a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa0206b1b0 0x55aa02439850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f349a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa0206b1b0 0x55aa02439850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47650/0 (socket says 172.21.15.16:47650) 2024-08-28T19:39:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f349a59c640 1 -- 172.21.15.16:0/1601685116 learned_addr learned my addr 172.21.15.16:0/1601685116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f3499d9b640 1 --2- 172.21.15.16:0/1601685116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02073390 0x55aa02439d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f349ad9d640 1 --2- 172.21.15.16:0/1601685116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa02094070 0x55aa02439310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f349a59c640 1 -- 172.21.15.16:0/1601685116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa02094070 msgr2=0x55aa02439310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f349a59c640 1 --2- 172.21.15.16:0/1601685116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa02094070 0x55aa02439310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f349a59c640 1 -- 172.21.15.16:0/1601685116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02073390 msgr2=0x55aa02439d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f349a59c640 1 --2- 172.21.15.16:0/1601685116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02073390 0x55aa02439d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.345+0000 7f349a59c640 1 -- 172.21.15.16:0/1601685116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa020ff650 con 0x55aa0206b1b0 2024-08-28T19:39:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.346+0000 7f349a59c640 1 --2- 172.21.15.16:0/1601685116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa0206b1b0 0x55aa02439850 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f34940e23b0 tx=0x7f34940ed910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78187c253ec8f25 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.346+0000 7f34937fe640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34880b4d20 0x7f34880b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.346+0000 7f349ad9d640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34880b4d20 0x7f34880b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.346+0000 7f34937fe640 1 -- 172.21.15.16:0/3636136603 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f34940e5150 con 0x7f349c109f30 2024-08-28T19:39:36.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.346+0000 7f3490ff9640 1 -- 172.21.15.16:0/1601685116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34940f8070 con 0x55aa0206b1b0 2024-08-28T19:39:36.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.347+0000 7f3490ff9640 1 -- 172.21.15.16:0/1601685116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34940604e0 con 0x55aa0206b1b0 2024-08-28T19:39:36.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.347+0000 7f3490ff9640 1 -- 172.21.15.16:0/1601685116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34940aad60 con 0x55aa0206b1b0 2024-08-28T19:39:36.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.348+0000 7f349ad9d640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34880b4d20 0x7f34880b7230 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f348c0049d0 tx=0x7f348c009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:36.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.348+0000 7f34a71c7740 1 -- 172.21.15.16:0/1601685116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa0206b1b0 msgr2=0x55aa02439850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:36.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.348+0000 7f34a71c7740 1 --2- 172.21.15.16:0/1601685116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa0206b1b0 0x55aa02439850 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f34940e23b0 tx=0x7f34940ed910 comp rx=0 tx=0).stop 2024-08-28T19:39:36.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.348+0000 7f34a71c7740 1 -- 172.21.15.16:0/1601685116 shutdown_connections 2024-08-28T19:39:36.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.348+0000 7f34a71c7740 1 --2- 172.21.15.16:0/1601685116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02073390 0x55aa02439d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.348+0000 7f34a71c7740 1 --2- 172.21.15.16:0/1601685116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa0206b1b0 0x55aa02439850 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.348+0000 7f34a71c7740 1 --2- 172.21.15.16:0/1601685116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa02094070 0x55aa02439310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.348+0000 7f34a71c7740 1 -- 172.21.15.16:0/1601685116 >> 172.21.15.16:0/1601685116 conn(0x55aa02088460 msgr2=0x55aa02437200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:36.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.348+0000 7f34a71c7740 1 -- 172.21.15.16:0/1601685116 shutdown_connections 2024-08-28T19:39:36.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.348+0000 7f34a71c7740 1 -- 172.21.15.16:0/1601685116 wait complete. 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.349+0000 7f34a71c7740 1 -- start start 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.349+0000 7f34a71c7740 1 Processor -- start 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.349+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa0206b1b0 0x55aa02444ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.349+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02073390 0x55aa02445010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.349+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02094070 0x55aa02450c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.349+0000 7f34a71c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aa02202700 con 0x55aa02073390 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f34a71c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aa02326750 con 0x55aa02094070 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f34a71c7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55aa02447970 con 0x55aa0206b1b0 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f3499d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02073390 0x55aa02445010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f349a59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa0206b1b0 0x55aa02444ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f3499d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02073390 0x55aa02445010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47660/0 (socket says 172.21.15.16:47660) 2024-08-28T19:39:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f3499d9b640 1 -- 172.21.15.16:0/1854036886 learned_addr learned my addr 172.21.15.16:0/1854036886 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f34a71c7740 10 client.0 _ll_register_callbacks cb 0x55aa02355e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f3499d9b640 1 -- 172.21.15.16:0/1854036886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa0206b1b0 msgr2=0x55aa02444ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f3499d9b640 1 --2- 172.21.15.16:0/1854036886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa0206b1b0 0x55aa02444ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f3499d9b640 1 -- 172.21.15.16:0/1854036886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02094070 msgr2=0x55aa02450c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f3499d9b640 1 --2- 172.21.15.16:0/1854036886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02094070 0x55aa02450c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f3499d9b640 1 -- 172.21.15.16:0/1854036886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa0235f900 con 0x55aa02073390 2024-08-28T19:39:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f3499d9b640 1 --2- 172.21.15.16:0/1854036886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02073390 0x55aa02445010 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f349c06b8d0 tx=0x7f34840316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98dad6d4ab3380e8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.350+0000 7f3481ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.351+0000 7f3481ffb640 1 -- 172.21.15.16:0/1854036886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484051070 con 0x55aa02073390 2024-08-28T19:39:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.351+0000 7f34a71c7740 1 -- 172.21.15.16:0/1854036886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34940604e0 con 0x55aa02073390 2024-08-28T19:39:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.351+0000 7f3481ffb640 1 -- 172.21.15.16:0/1854036886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f3484004250 con 0x55aa02073390 2024-08-28T19:39:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.351+0000 7f3481ffb640 1 -- 172.21.15.16:0/1854036886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7f348405f050 con 0x55aa02073390 2024-08-28T19:39:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.351+0000 7f34a71c7740 10 client.19791 fetch_fsmap learned FSMap version 98 2024-08-28T19:39:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.351+0000 7f34a71c7740 1 -- 172.21.15.16:0/1854036886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=4+}) -- 0x55aa024537b0 con 0x55aa02073390 2024-08-28T19:39:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.351+0000 7f3481ffb640 1 -- 172.21.15.16:0/1854036886 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34940604e0 con 0x55aa02073390 2024-08-28T19:39:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.352+0000 7f3481ffb640 1 -- 172.21.15.16:0/1854036886 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 98) ==== 8406+0+0 (secure 0 0 0) 0x7f348404a040 con 0x55aa02073390 2024-08-28T19:39:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.352+0000 7f34a71c7740 10 client.19791 fetch_fsmap finished waiting for FSMap version 98 2024-08-28T19:39:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.352+0000 7f34a71c7740 10 client.19791 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:39:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.352+0000 7f34a71c7740 20 client.19791 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.352+0000 7f34a71c7740 1 --2- 172.21.15.16:0/1854036886 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aa020529f0 0x55aa024571b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.352+0000 7f34a71c7740 4 client.19791 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:39:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.352+0000 7f34a71c7740 1 -- 172.21.15.16:0/1854036886 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa023b25d0 con 0x55aa020529f0 2024-08-28T19:39:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.353+0000 7f349a59c640 1 --2- 172.21.15.16:0/1854036886 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aa020529f0 0x55aa024571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.354+0000 7f349a59c640 1 --2- 172.21.15.16:0/1854036886 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aa020529f0 0x55aa024571b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b7a6457baba1536a server_cookie=4e3026f9bdde476b in_seq=0 out_seq=0 2024-08-28T19:39:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.354+0000 7f3481ffb640 10 client.19791 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:39:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.355+0000 7f3481ffb640 1 -- 172.21.15.16:0/1854036886 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55aa023b25d0 con 0x55aa020529f0 2024-08-28T19:39:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.355+0000 7f3481ffb640 10 client.19791 handle_command_reply: tid=0 2024-08-28T19:39:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.355+0000 7f34a71c7740 1 client.19791 shutdown 2024-08-28T19:39:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.356+0000 7f34a71c7740 1 -- 172.21.15.16:0/1854036886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02073390 msgr2=0x55aa02445010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.356+0000 7f34a71c7740 1 --2- 172.21.15.16:0/1854036886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02073390 0x55aa02445010 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f349c06b8d0 tx=0x7f34840316d0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.357+0000 7f34a71c7740 1 -- 172.21.15.16:0/1854036886 shutdown_connections 2024-08-28T19:39:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.357+0000 7f34a71c7740 1 --2- 172.21.15.16:0/1854036886 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aa020529f0 0x55aa024571b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.357+0000 7f34a71c7740 1 --2- 172.21.15.16:0/1854036886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02094070 0x55aa02450c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.358+0000 7f34a71c7740 1 --2- 172.21.15.16:0/1854036886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02073390 0x55aa02445010 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.358+0000 7f34a71c7740 1 --2- 172.21.15.16:0/1854036886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa0206b1b0 0x55aa02444ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.358+0000 7f34a71c7740 1 -- 172.21.15.16:0/1854036886 >> 172.21.15.16:0/1854036886 conn(0x55aa02088460 msgr2=0x55aa0243c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.358+0000 7f3481ffb640 0 client.19791 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:39:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.358+0000 7f34a71c7740 1 -- 172.21.15.16:0/1854036886 shutdown_connections 2024-08-28T19:39:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.358+0000 7f34a71c7740 1 -- 172.21.15.16:0/1854036886 wait complete. 2024-08-28T19:39:36.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.358+0000 7f34a71c7740 20 client.19791 trim_cache size 0 max 16384 2024-08-28T19:39:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.389+0000 7f34a71c7740 1 Processor -- start 2024-08-28T19:39:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.389+0000 7f34a71c7740 1 -- start start 2024-08-28T19:39:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.389+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa0206b1b0 0x55aa02450b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.389+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02073390 0x55aa024510a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.389+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02094070 0x55aa02436b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.389+0000 7f34a71c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f349c1aec60 con 0x55aa02073390 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.389+0000 7f34a71c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f349c1b0950 con 0x55aa02094070 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.389+0000 7f34a71c7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f349c1af3c0 con 0x55aa0206b1b0 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.389+0000 7f349ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02094070 0x55aa02436b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.389+0000 7f349ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02094070 0x55aa02436b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51420/0 (socket says 172.21.15.16:51420) 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.389+0000 7f349ad9d640 1 -- 172.21.15.16:0/2354276554 learned_addr learned my addr 172.21.15.16:0/2354276554 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.390+0000 7f349ad9d640 1 -- 172.21.15.16:0/2354276554 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa0206b1b0 msgr2=0x55aa02450b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.390+0000 7f349ad9d640 1 --2- 172.21.15.16:0/2354276554 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa0206b1b0 0x55aa02450b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.390+0000 7f349ad9d640 1 -- 172.21.15.16:0/2354276554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02073390 msgr2=0x55aa024510a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.390+0000 7f349ad9d640 1 --2- 172.21.15.16:0/2354276554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02073390 0x55aa024510a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.390+0000 7f349ad9d640 1 -- 172.21.15.16:0/2354276554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa02457780 con 0x55aa02094070 2024-08-28T19:39:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.390+0000 7f349ad9d640 1 --2- 172.21.15.16:0/2354276554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02094070 0x55aa02436b20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f348c011960 tx=0x7f348c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f6926aea3317fee server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:36.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.396+0000 7f3483fff640 1 -- 172.21.15.16:0/2354276554 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c02d070 con 0x55aa02094070 2024-08-28T19:39:36.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.396+0000 7f3483fff640 1 -- 172.21.15.16:0/2354276554 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f348c020de0 con 0x55aa02094070 2024-08-28T19:39:36.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.396+0000 7f3483fff640 1 -- 172.21.15.16:0/2354276554 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c028ba0 con 0x55aa02094070 2024-08-28T19:39:36.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.398+0000 7f34a71c7740 1 -- 172.21.15.16:0/2354276554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02094070 msgr2=0x55aa02436b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:36.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.398+0000 7f34a71c7740 1 --2- 172.21.15.16:0/2354276554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02094070 0x55aa02436b20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f348c011960 tx=0x7f348c004a40 comp rx=0 tx=0).stop 2024-08-28T19:39:36.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.398+0000 7f34a71c7740 1 -- 172.21.15.16:0/2354276554 shutdown_connections 2024-08-28T19:39:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.398+0000 7f34a71c7740 1 --2- 172.21.15.16:0/2354276554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa02094070 0x55aa02436b20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.398+0000 7f34a71c7740 1 --2- 172.21.15.16:0/2354276554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02073390 0x55aa024510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.398+0000 7f34a71c7740 1 --2- 172.21.15.16:0/2354276554 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa0206b1b0 0x55aa02450b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.398+0000 7f34a71c7740 1 -- 172.21.15.16:0/2354276554 >> 172.21.15.16:0/2354276554 conn(0x55aa02088460 msgr2=0x55aa02443350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.399+0000 7f34a71c7740 1 -- 172.21.15.16:0/2354276554 shutdown_connections 2024-08-28T19:39:36.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.400+0000 7f34a71c7740 1 -- 172.21.15.16:0/2354276554 wait complete. 2024-08-28T19:39:36.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.400+0000 7f34a71c7740 1 -- start start 2024-08-28T19:39:36.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.401+0000 7f34a71c7740 1 Processor -- start 2024-08-28T19:39:36.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.401+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa0206b1b0 0x55aa0244d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.401+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa02073390 0x55aa02448690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.401+0000 7f34a71c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02094070 0x55aa02448bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.401+0000 7f34a71c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f349c1b01c0 con 0x55aa02094070 2024-08-28T19:39:36.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.401+0000 7f34a71c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f349c1afa90 con 0x55aa0206b1b0 2024-08-28T19:39:36.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.401+0000 7f34a71c7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f349c083910 con 0x55aa02073390 2024-08-28T19:39:36.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.401+0000 7f34a71c7740 10 client.0 _ll_register_callbacks cb 0x7f3484009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:36.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.402+0000 7f349a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa0206b1b0 0x55aa0244d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.402+0000 7f349a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa0206b1b0 0x55aa0244d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51434/0 (socket says 172.21.15.16:51434) 2024-08-28T19:39:36.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.402+0000 7f349a59c640 1 -- 172.21.15.16:0/3313369773 learned_addr learned my addr 172.21.15.16:0/3313369773 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:36.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.402+0000 7f349ad9d640 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02094070 0x55aa02448bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.404+0000 7f3499d9b640 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa02073390 0x55aa02448690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.404+0000 7f349ad9d640 1 -- 172.21.15.16:0/3313369773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa02073390 msgr2=0x55aa02448690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:36.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.404+0000 7f349ad9d640 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa02073390 0x55aa02448690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.404+0000 7f349ad9d640 1 -- 172.21.15.16:0/3313369773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa0206b1b0 msgr2=0x55aa0244d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:36.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.404+0000 7f349ad9d640 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa0206b1b0 0x55aa0244d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.404+0000 7f349ad9d640 1 -- 172.21.15.16:0/3313369773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa02449190 con 0x55aa02094070 2024-08-28T19:39:36.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.405+0000 7f349ad9d640 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02094070 0x55aa02448bd0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f348c0289c0 tx=0x7f348c009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1a7f4044c9e952 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:36.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.407+0000 7f3481ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:36.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.407+0000 7f3481ffb640 1 -- 172.21.15.16:0/3313369773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c02d070 con 0x55aa02094070 2024-08-28T19:39:36.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.407+0000 7f3481ffb640 1 -- 172.21.15.16:0/3313369773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f348c020b40 con 0x55aa02094070 2024-08-28T19:39:36.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.407+0000 7f34a71c7740 1 -- 172.21.15.16:0/3313369773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f348c020de0 con 0x55aa02094070 2024-08-28T19:39:36.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.407+0000 7f3481ffb640 1 -- 172.21.15.16:0/3313369773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7f348c028b20 con 0x55aa02094070 2024-08-28T19:39:36.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.407+0000 7f34a71c7740 10 client.19794 fetch_fsmap learned FSMap version 98 2024-08-28T19:39:36.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.407+0000 7f34a71c7740 1 -- 172.21.15.16:0/3313369773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=4+}) -- 0x55aa024d2930 con 0x55aa02094070 2024-08-28T19:39:36.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.408+0000 7f3481ffb640 1 -- 172.21.15.16:0/3313369773 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f348c020de0 con 0x55aa02094070 2024-08-28T19:39:36.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.408+0000 7f3481ffb640 1 -- 172.21.15.16:0/3313369773 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 98) ==== 8406+0+0 (secure 0 0 0) 0x7f348c005470 con 0x55aa02094070 2024-08-28T19:39:36.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.408+0000 7f34a71c7740 10 client.19794 fetch_fsmap finished waiting for FSMap version 98 2024-08-28T19:39:36.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.408+0000 7f34a71c7740 10 client.19794 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:39:36.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.408+0000 7f34a71c7740 20 client.19794 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:36.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.408+0000 7f34a71c7740 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aa02027da0 0x55aa024d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:36.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.408+0000 7f34a71c7740 4 client.19794 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:39:36.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.408+0000 7f34a71c7740 1 -- 172.21.15.16:0/3313369773 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aa023ae070 con 0x55aa02027da0 2024-08-28T19:39:36.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.409+0000 7f349a59c640 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aa02027da0 0x55aa024d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:36.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.410+0000 7f349a59c640 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aa02027da0 0x55aa024d5040 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5a4b80e2bcc8cdd6 server_cookie=89ded4d4e154dd89 in_seq=0 out_seq=0 2024-08-28T19:39:36.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.410+0000 7f3481ffb640 10 client.19794 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:39:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.414+0000 7f3481ffb640 1 -- 172.21.15.16:0/3313369773 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aa023ae070 con 0x55aa02027da0 2024-08-28T19:39:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.414+0000 7f3481ffb640 10 client.19794 handle_command_reply: tid=0 2024-08-28T19:39:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.414+0000 7f34a71c7740 1 client.19794 shutdown 2024-08-28T19:39:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.414+0000 7f34a71c7740 1 -- 172.21.15.16:0/3313369773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02094070 msgr2=0x55aa02448bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.414+0000 7f34a71c7740 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02094070 0x55aa02448bd0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f348c0289c0 tx=0x7f348c009fc0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.417+0000 7f34a71c7740 1 -- 172.21.15.16:0/3313369773 shutdown_connections 2024-08-28T19:39:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.417+0000 7f34a71c7740 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55aa02027da0 0x55aa024d5040 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.417+0000 7f34a71c7740 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa02094070 0x55aa02448bd0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.417+0000 7f34a71c7740 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa02073390 0x55aa02448690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.417+0000 7f34a71c7740 1 --2- 172.21.15.16:0/3313369773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa0206b1b0 0x55aa0244d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.417+0000 7f34a71c7740 1 -- 172.21.15.16:0/3313369773 >> 172.21.15.16:0/3313369773 conn(0x55aa02088460 msgr2=0x55aa0243c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.418+0000 7f3481ffb640 0 client.19794 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:39:36.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.418+0000 7f34a71c7740 1 -- 172.21.15.16:0/3313369773 shutdown_connections 2024-08-28T19:39:36.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.421+0000 7f34a71c7740 1 -- 172.21.15.16:0/3313369773 wait complete. 2024-08-28T19:39:36.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.421+0000 7f34a71c7740 20 client.19794 trim_cache size 0 max 16384 2024-08-28T19:39:36.422 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:39:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.422+0000 7f34a0c52640 1 -- 172.21.15.16:0/3636136603 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34880b4d20 msgr2=0x7f34880b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.422+0000 7f34a0c52640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34880b4d20 0x7f34880b7230 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f348c0049d0 tx=0x7f348c009500 comp rx=0 tx=0).stop 2024-08-28T19:39:36.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.423+0000 7f34a0c52640 1 -- 172.21.15.16:0/3636136603 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f349c109f30 msgr2=0x7f349c1a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:36.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.423+0000 7f34a0c52640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f349c109f30 0x7f349c1a08b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f349400b3f0 tx=0x7f349400b8b0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.423+0000 7f34a0c52640 1 -- 172.21.15.16:0/3636136603 shutdown_connections 2024-08-28T19:39:36.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.423+0000 7f34a0c52640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34880b4d20 0x7f34880b7230 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.423+0000 7f34a0c52640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f349c1101c0 0x7f349c1a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.423+0000 7f34a0c52640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f349c10f870 0x7f349c1a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.423+0000 7f34a0c52640 1 --2- 172.21.15.16:0/3636136603 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f349c109f30 0x7f349c1a08b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:36.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.423+0000 7f34a0c52640 1 -- 172.21.15.16:0/3636136603 >> 172.21.15.16:0/3636136603 conn(0x7f349c077e90 msgr2=0x7f349c10dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:36.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.423+0000 7f34a0c52640 1 -- 172.21.15.16:0/3636136603 shutdown_connections 2024-08-28T19:39:36.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:36.423+0000 7f34a0c52640 1 -- 172.21.15.16:0/3636136603 wait complete. 2024-08-28T19:39:36.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 121586 23.40 MB/sec execute 192 sec latency 125.067 ms 2024-08-28T19:39:37.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:39:37.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 121933 23.38 MB/sec execute 193 sec latency 99.163 ms 2024-08-28T19:39:37.652 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:37.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:37 smithi016 ceph-mon[18556]: pgmap v920: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 16 MiB/s wr, 682 op/s 2024-08-28T19:39:37.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:37 smithi028 ceph-mon[21769]: pgmap v920: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 16 MiB/s wr, 682 op/s 2024-08-28T19:39:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:37 smithi115 ceph-mon[22184]: pgmap v920: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 16 MiB/s wr, 682 op/s 2024-08-28T19:39:38.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.211+0000 7f09d82e9640 1 -- 172.21.15.16:0/4055354939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d0108a90 msgr2=0x7f09d010f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:38.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.211+0000 7f09d82e9640 1 --2- 172.21.15.16:0/4055354939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d0108a90 0x7f09d010f2c0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009900 tx=0x7f09c802a410 comp rx=0 tx=0).stop 2024-08-28T19:39:38.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.211+0000 7f09d82e9640 1 -- 172.21.15.16:0/4055354939 shutdown_connections 2024-08-28T19:39:38.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.211+0000 7f09d82e9640 1 --2- 172.21.15.16:0/4055354939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d0108a90 0x7f09d010f2c0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.211+0000 7f09d82e9640 1 --2- 172.21.15.16:0/4055354939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09d0108140 0x7f09d0108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.211+0000 7f09d82e9640 1 --2- 172.21.15.16:0/4055354939 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09d0102800 0x7f09d0102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.211+0000 7f09d82e9640 1 -- 172.21.15.16:0/4055354939 >> 172.21.15.16:0/4055354939 conn(0x7f09d00794d0 msgr2=0x7f09d00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:38.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.211+0000 7f09d82e9640 1 -- 172.21.15.16:0/4055354939 shutdown_connections 2024-08-28T19:39:38.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.211+0000 7f09d82e9640 1 -- 172.21.15.16:0/4055354939 wait complete. 2024-08-28T19:39:38.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.212+0000 7f09d82e9640 1 Processor -- start 2024-08-28T19:39:38.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.212+0000 7f09d82e9640 1 -- start start 2024-08-28T19:39:38.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.213+0000 7f09d82e9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09d0102800 0x7f09d00ff410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.213+0000 7f09d82e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d0108140 0x7f09d00ff950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.213+0000 7f09d82e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09d0108a90 0x7f09d00ffe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.213+0000 7f09d82e9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09d0112710 con 0x7f09d0108140 2024-08-28T19:39:38.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.213+0000 7f09d82e9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09d0112590 con 0x7f09d0108a90 2024-08-28T19:39:38.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.213+0000 7f09d82e9640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f09d0112890 con 0x7f09d0102800 2024-08-28T19:39:38.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.213+0000 7f09d6085640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09d0102800 0x7f09d00ff410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.213+0000 7f09d6085640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09d0102800 0x7f09d00ff410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34352/0 (socket says 172.21.15.16:34352) 2024-08-28T19:39:38.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.213+0000 7f09d6085640 1 -- 172.21.15.16:0/544326425 learned_addr learned my addr 172.21.15.16:0/544326425 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.213+0000 7f09d5884640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d0108140 0x7f09d00ff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.213+0000 7f09d6886640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09d0108a90 0x7f09d00ffe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.214+0000 7f09d6085640 1 -- 172.21.15.16:0/544326425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09d0108a90 msgr2=0x7f09d00ffe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.214+0000 7f09d6085640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09d0108a90 0x7f09d00ffe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.214+0000 7f09d6085640 1 -- 172.21.15.16:0/544326425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d0108140 msgr2=0x7f09d00ff950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.214+0000 7f09d6085640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d0108140 0x7f09d00ff950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.214+0000 7f09d6085640 1 -- 172.21.15.16:0/544326425 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d01a8510 con 0x7f09d0102800 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.214+0000 7f09d5884640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d0108140 0x7f09d00ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.214+0000 7f09d6886640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09d0108a90 0x7f09d00ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.214+0000 7f09d6085640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09d0102800 0x7f09d00ff410 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00eb90 tx=0x7f09cc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.214+0000 7f09c77fe640 1 -- 172.21.15.16:0/544326425 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc004d20 con 0x7f09d0102800 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.214+0000 7f09d82e9640 1 -- 172.21.15.16:0/544326425 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d01a8800 con 0x7f09d0102800 2024-08-28T19:39:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.215+0000 7f09d82e9640 1 -- 172.21.15.16:0/544326425 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d01a8d40 con 0x7f09d0102800 2024-08-28T19:39:38.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.215+0000 7f09c77fe640 1 -- 172.21.15.16:0/544326425 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09cc00ee90 con 0x7f09d0102800 2024-08-28T19:39:38.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.215+0000 7f09c77fe640 1 -- 172.21.15.16:0/544326425 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc0106b0 con 0x7f09d0102800 2024-08-28T19:39:38.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.218+0000 7f09de85e740 1 Processor -- start 2024-08-28T19:39:38.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.218+0000 7f09de85e740 1 -- start start 2024-08-28T19:39:38.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.218+0000 7f09de85e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d2e070 0x55c4ea0d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.218+0000 7f09de85e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0d3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.218+0000 7f09de85e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 0x55c4ea0d3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.218+0000 7f09de85e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c4ea04a680 con 0x55c4e9d2e070 2024-08-28T19:39:38.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.218+0000 7f09de85e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4ea039c10 con 0x55c4e9d0d390 2024-08-28T19:39:38.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.218+0000 7f09de85e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c4ea03dbc0 con 0x55c4e9d051b0 2024-08-28T19:39:38.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.218+0000 7f09d6886640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d2e070 0x55c4ea0d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.218+0000 7f09d6886640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d2e070 0x55c4ea0d3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47686/0 (socket says 172.21.15.16:47686) 2024-08-28T19:39:38.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.218+0000 7f09d6886640 1 -- 172.21.15.16:0/2711885516 learned_addr learned my addr 172.21.15.16:0/2711885516 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:38.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.219+0000 7f09d5884640 1 --2- 172.21.15.16:0/2711885516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 0x55c4ea0d3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.219+0000 7f09d6886640 1 -- 172.21.15.16:0/2711885516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 msgr2=0x55c4ea0d3850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:39:38.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.219+0000 7f09d6886640 1 --2- 172.21.15.16:0/2711885516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0d3850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.219+0000 7f09d6886640 1 -- 172.21.15.16:0/2711885516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 msgr2=0x55c4ea0d3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:38.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.219+0000 7f09d6886640 1 --2- 172.21.15.16:0/2711885516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 0x55c4ea0d3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.219+0000 7f09d6886640 1 -- 172.21.15.16:0/2711885516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4e9d99650 con 0x55c4e9d2e070 2024-08-28T19:39:38.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.219+0000 7f09d5884640 1 --2- 172.21.15.16:0/2711885516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 0x55c4ea0d3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.220+0000 7f09d6886640 1 --2- 172.21.15.16:0/2711885516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d2e070 0x55c4ea0d3310 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f09c8031440 tx=0x7f09c8009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a33c48070a9c83 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.220+0000 7f09c77fe640 1 -- 172.21.15.16:0/544326425 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f09cc021440 con 0x7f09d0102800 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.220+0000 7f09c4d9a640 1 -- 172.21.15.16:0/2711885516 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8040070 con 0x55c4e9d2e070 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.220+0000 7f09c4d9a640 1 -- 172.21.15.16:0/2711885516 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09c80317a0 con 0x55c4e9d2e070 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.220+0000 7f09c4d9a640 1 -- 172.21.15.16:0/2711885516 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8035040 con 0x55c4e9d2e070 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.221+0000 7f09c77fe640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09bc0b4d20 0x7f09bc0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.221+0000 7f09c77fe640 1 -- 172.21.15.16:0/544326425 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f09cc0e5d10 con 0x7f09d0102800 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.221+0000 7f09de85e740 1 -- 172.21.15.16:0/2711885516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d2e070 msgr2=0x55c4ea0d3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.221+0000 7f09de85e740 1 --2- 172.21.15.16:0/2711885516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d2e070 0x55c4ea0d3310 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f09c8031440 tx=0x7f09c8009600 comp rx=0 tx=0).stop 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.221+0000 7f09d6085640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09bc0b4d20 0x7f09bc0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.222+0000 7f09de85e740 1 -- 172.21.15.16:0/2711885516 shutdown_connections 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.222+0000 7f09de85e740 1 --2- 172.21.15.16:0/2711885516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 0x55c4ea0d3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.222+0000 7f09de85e740 1 --2- 172.21.15.16:0/2711885516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0d3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.222+0000 7f09de85e740 1 --2- 172.21.15.16:0/2711885516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d2e070 0x55c4ea0d3310 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.222+0000 7f09de85e740 1 -- 172.21.15.16:0/2711885516 >> 172.21.15.16:0/2711885516 conn(0x55c4e9d22460 msgr2=0x55c4ea0d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.222+0000 7f09de85e740 1 -- 172.21.15.16:0/2711885516 shutdown_connections 2024-08-28T19:39:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.222+0000 7f09d6085640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09bc0b4d20 0x7f09bc0b7230 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x55c4ea0d1f90 tx=0x7f09cc0a7f80 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.222+0000 7f09de85e740 1 -- 172.21.15.16:0/2711885516 wait complete. 2024-08-28T19:39:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.223+0000 7f09de85e740 1 -- start start 2024-08-28T19:39:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.223+0000 7f09de85e740 1 Processor -- start 2024-08-28T19:39:38.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.223+0000 7f09de85e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d051b0 0x55c4ea0e7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.223+0000 7f09de85e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 0x55c4ea0e7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09d6886640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d051b0 0x55c4ea0e7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09d6886640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d051b0 0x55c4ea0e7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47696/0 (socket says 172.21.15.16:47696) 2024-08-28T19:39:38.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09de85e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d2e070 0x55c4ea0f3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09de85e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c4e9e9c700 con 0x55c4e9d051b0 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09de85e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4e9fc0750 con 0x55c4e9d0d390 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09de85e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c4ea0ea1c0 con 0x55c4e9d2e070 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09d6886640 1 -- 172.21.15.16:0/3831082234 learned_addr learned my addr 172.21.15.16:0/3831082234 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09d5884640 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 0x55c4ea0e7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09de85e740 10 client.0 _ll_register_callbacks cb 0x55c4e9fefe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09d6886640 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d2e070 0x55c4ea0f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09d5884640 1 -- 172.21.15.16:0/3831082234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d2e070 msgr2=0x55c4ea0f3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09d5884640 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d2e070 0x55c4ea0f3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09d5884640 1 -- 172.21.15.16:0/3831082234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d051b0 msgr2=0x55c4ea0e7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09d5884640 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d051b0 0x55c4ea0e7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.224+0000 7f09d5884640 1 -- 172.21.15.16:0/3831082234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4e9ff9900 con 0x55c4e9d0d390 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.225+0000 7f09d5884640 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 0x55c4ea0e7750 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f09d006a840 tx=0x7f09c0012e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1cd2db0e3b49254 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.225+0000 7f09b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.225+0000 7f09b9ffb640 1 -- 172.21.15.16:0/3831082234 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c000fa60 con 0x55c4e9d0d390 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.225+0000 7f09de85e740 1 -- 172.21.15.16:0/3831082234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09c80317a0 con 0x55c4e9d0d390 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.225+0000 7f09b9ffb640 1 -- 172.21.15.16:0/3831082234 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f09c000fc00 con 0x55c4e9d0d390 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.226+0000 7f09b9ffb640 1 -- 172.21.15.16:0/3831082234 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7f09c0025050 con 0x55c4e9d0d390 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.226+0000 7f09de85e740 10 client.29246 fetch_fsmap learned FSMap version 98 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.226+0000 7f09de85e740 1 -- 172.21.15.16:0/3831082234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=4+}) -- 0x55c4ea0f6070 con 0x55c4e9d0d390 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.226+0000 7f09b9ffb640 1 -- 172.21.15.16:0/3831082234 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09c000fe00 con 0x55c4e9d0d390 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.226+0000 7f09b9ffb640 1 -- 172.21.15.16:0/3831082234 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 98) ==== 8406+0+0 (secure 0 0 0) 0x7f09c001c470 con 0x55c4e9d0d390 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.226+0000 7f09de85e740 10 client.29246 fetch_fsmap finished waiting for FSMap version 98 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.226+0000 7f09de85e740 10 client.29246 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.226+0000 7f09de85e740 20 client.29246 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.226+0000 7f09de85e740 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55c4e9cec9f0 0x55c4ea0f9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.226+0000 7f09de85e740 4 client.29246 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:39:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.226+0000 7f09de85e740 1 -- 172.21.15.16:0/3831082234 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f09c80317a0 con 0x55c4e9cec9f0 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.227+0000 7f09d6886640 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55c4e9cec9f0 0x55c4ea0f9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.227+0000 7f09b9ffb640 10 client.29246 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.227+0000 7f09d6886640 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55c4e9cec9f0 0x55c4ea0f9610 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7860259d7f0c3bbf server_cookie=d9ed3684e17b4179 in_seq=0 out_seq=0 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09b9ffb640 1 -- 172.21.15.16:0/3831082234 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f09c80317a0 con 0x55c4e9cec9f0 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09b9ffb640 10 client.29246 handle_command_reply: tid=0 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 1 client.29246 shutdown 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 1 -- 172.21.15.16:0/3831082234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 msgr2=0x55c4ea0e7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 0x55c4ea0e7750 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f09d006a840 tx=0x7f09c0012e70 comp rx=0 tx=0).stop 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 1 -- 172.21.15.16:0/3831082234 shutdown_connections 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55c4e9cec9f0 0x55c4ea0f9610 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d2e070 0x55c4ea0f3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9d0d390 0x55c4ea0e7750 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 1 --2- 172.21.15.16:0/3831082234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d051b0 0x55c4ea0e7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09b9ffb640 0 client.29246 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:39:38.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 1 -- 172.21.15.16:0/3831082234 >> 172.21.15.16:0/3831082234 conn(0x55c4e9d22460 msgr2=0x55c4ea0cf7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:38.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 1 -- 172.21.15.16:0/3831082234 shutdown_connections 2024-08-28T19:39:38.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 1 -- 172.21.15.16:0/3831082234 wait complete. 2024-08-28T19:39:38.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.229+0000 7f09de85e740 20 client.29246 trim_cache size 0 max 16384 2024-08-28T19:39:38.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.248+0000 7f09de85e740 1 Processor -- start 2024-08-28T19:39:38.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.248+0000 7f09de85e740 1 -- start start 2024-08-28T19:39:38.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.248+0000 7f09de85e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 0x55c4ea0d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.248+0000 7f09de85e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0f3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.248+0000 7f09de85e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d0d390 0x55c4ea0d83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.248+0000 7f09de85e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09b400ffb0 con 0x55c4e9d0d390 2024-08-28T19:39:38.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09de85e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09b4011ca0 con 0x55c4e9cec9f0 2024-08-28T19:39:38.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09de85e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f09b4010710 con 0x55c4e9d051b0 2024-08-28T19:39:38.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09d5884640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09d5884640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0f3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34374/0 (socket says 172.21.15.16:34374) 2024-08-28T19:39:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09d5884640 1 -- 172.21.15.16:0/3147307234 learned_addr learned my addr 172.21.15.16:0/3147307234 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09d6886640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 0x55c4ea0d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09d6886640 1 --2- 172.21.15.16:0/3147307234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d0d390 0x55c4ea0d83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09d5884640 1 -- 172.21.15.16:0/3147307234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 msgr2=0x55c4ea0d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09d5884640 1 --2- 172.21.15.16:0/3147307234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 0x55c4ea0d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09d5884640 1 -- 172.21.15.16:0/3147307234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d0d390 msgr2=0x55c4ea0d83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09d5884640 1 --2- 172.21.15.16:0/3147307234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d0d390 0x55c4ea0d83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09d5884640 1 -- 172.21.15.16:0/3147307234 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4ea0e5710 con 0x55c4e9d051b0 2024-08-28T19:39:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.249+0000 7f09d6886640 1 --2- 172.21.15.16:0/3147307234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d0d390 0x55c4ea0d83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.250+0000 7f09d5884640 1 --2- 172.21.15.16:0/3147307234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0f3480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f09c00126c0 tx=0x7f09c00092e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de71f4769d0d0410 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:38.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.250+0000 7f09bbfff640 1 -- 172.21.15.16:0/3147307234 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c002e070 con 0x55c4e9d051b0 2024-08-28T19:39:38.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.250+0000 7f09bbfff640 1 -- 172.21.15.16:0/3147307234 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09c001ca00 con 0x55c4e9d051b0 2024-08-28T19:39:38.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.250+0000 7f09bbfff640 1 -- 172.21.15.16:0/3147307234 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0022040 con 0x55c4e9d051b0 2024-08-28T19:39:38.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.251+0000 7f09de85e740 1 -- 172.21.15.16:0/3147307234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 msgr2=0x55c4ea0f3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:38.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.251+0000 7f09de85e740 1 --2- 172.21.15.16:0/3147307234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0f3480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f09c00126c0 tx=0x7f09c00092e0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.251+0000 7f09de85e740 1 -- 172.21.15.16:0/3147307234 shutdown_connections 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.251+0000 7f09de85e740 1 --2- 172.21.15.16:0/3147307234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d0d390 0x55c4ea0d83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.251+0000 7f09de85e740 1 --2- 172.21.15.16:0/3147307234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0f3480 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.251+0000 7f09de85e740 1 --2- 172.21.15.16:0/3147307234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 0x55c4ea0d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.251+0000 7f09de85e740 1 -- 172.21.15.16:0/3147307234 >> 172.21.15.16:0/3147307234 conn(0x55c4e9d22460 msgr2=0x55c4ea0d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.251+0000 7f09de85e740 1 -- 172.21.15.16:0/3147307234 shutdown_connections 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.252+0000 7f09de85e740 1 -- 172.21.15.16:0/3147307234 wait complete. 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.252+0000 7f09de85e740 1 -- start start 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.252+0000 7f09de85e740 1 Processor -- start 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09de85e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 0x55c4ea0efeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09de85e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0ec3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09de85e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d0d390 0x55c4ea0ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09de85e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09b4011510 con 0x55c4e9d0d390 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09de85e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09b4010de0 con 0x55c4e9cec9f0 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09de85e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f09b400dfb0 con 0x55c4e9d051b0 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09de85e740 10 client.0 _ll_register_callbacks cb 0x7f09c0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09d6886640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 0x55c4ea0efeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09d6886640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 0x55c4ea0efeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51492/0 (socket says 172.21.15.16:51492) 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09d6886640 1 -- 172.21.15.16:0/653031696 learned_addr learned my addr 172.21.15.16:0/653031696 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09d5884640 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0ec3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09d6886640 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d0d390 0x55c4ea0ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09d6886640 1 -- 172.21.15.16:0/653031696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 msgr2=0x55c4ea0ec3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09d6886640 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0ec3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09d6886640 1 -- 172.21.15.16:0/653031696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d0d390 msgr2=0x55c4ea0ecaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09d6886640 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d0d390 0x55c4ea0ecaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.253+0000 7f09d6886640 1 -- 172.21.15.16:0/653031696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4ea163fe0 con 0x55c4e9cec9f0 2024-08-28T19:39:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.254+0000 7f09d6886640 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 0x55c4ea0efeb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009850 tx=0x7f09c0012590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78c46764e9b05651 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.254+0000 7f09b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:39:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.254+0000 7f09b9ffb640 1 -- 172.21.15.16:0/653031696 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c805d070 con 0x55c4e9cec9f0 2024-08-28T19:39:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.254+0000 7f09b9ffb640 1 -- 172.21.15.16:0/653031696 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f09c803c8f0 con 0x55c4e9cec9f0 2024-08-28T19:39:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.254+0000 7f09de85e740 1 -- 172.21.15.16:0/653031696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09c001ca00 con 0x55c4e9cec9f0 2024-08-28T19:39:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.254+0000 7f09b9ffb640 1 -- 172.21.15.16:0/653031696 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7f09c8069050 con 0x55c4e9cec9f0 2024-08-28T19:39:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.254+0000 7f09de85e740 10 client.29252 fetch_fsmap learned FSMap version 98 2024-08-28T19:39:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.254+0000 7f09de85e740 1 -- 172.21.15.16:0/653031696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=4+}) -- 0x55c4ea165170 con 0x55c4e9cec9f0 2024-08-28T19:39:38.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.255+0000 7f09b9ffb640 1 -- 172.21.15.16:0/653031696 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09c001ca00 con 0x55c4e9cec9f0 2024-08-28T19:39:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.255+0000 7f09b9ffb640 1 -- 172.21.15.16:0/653031696 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 98) ==== 8406+0+0 (secure 0 0 0) 0x7f09c0009ce0 con 0x55c4e9cec9f0 2024-08-28T19:39:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.255+0000 7f09de85e740 10 client.29252 fetch_fsmap finished waiting for FSMap version 98 2024-08-28T19:39:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.255+0000 7f09de85e740 10 client.29252 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:39:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.255+0000 7f09de85e740 20 client.29252 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.256+0000 7f09de85e740 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55c4e9d2e070 0x55c4ea1675d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.256+0000 7f09de85e740 4 client.29252 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:39:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.256+0000 7f09de85e740 1 -- 172.21.15.16:0/653031696 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4ea04c5d0 con 0x55c4e9d2e070 2024-08-28T19:39:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.256+0000 7f09d5884640 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55c4e9d2e070 0x55c4ea1675d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.256+0000 7f09b9ffb640 10 client.29252 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.257+0000 7f09d5884640 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55c4e9d2e070 0x55c4ea1675d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a897888446abbc47 server_cookie=48f70248fd93558b in_seq=0 out_seq=0 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.258+0000 7f09b9ffb640 1 -- 172.21.15.16:0/653031696 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4ea04c5d0 con 0x55c4e9d2e070 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.258+0000 7f09b9ffb640 10 client.29252 handle_command_reply: tid=0 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.258+0000 7f09de85e740 1 client.29252 shutdown 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.258+0000 7f09de85e740 1 -- 172.21.15.16:0/653031696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 msgr2=0x55c4ea0efeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.258+0000 7f09de85e740 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 0x55c4ea0efeb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009850 tx=0x7f09c0012590 comp rx=0 tx=0).stop 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.259+0000 7f09de85e740 1 -- 172.21.15.16:0/653031696 shutdown_connections 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.259+0000 7f09de85e740 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55c4e9d2e070 0x55c4ea1675d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.259+0000 7f09de85e740 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4e9d0d390 0x55c4ea0ecaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.259+0000 7f09de85e740 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c4e9d051b0 0x55c4ea0ec3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.259+0000 7f09b9ffb640 0 client.29252 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.259+0000 7f09de85e740 1 --2- 172.21.15.16:0/653031696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e9cec9f0 0x55c4ea0efeb0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.259+0000 7f09de85e740 1 -- 172.21.15.16:0/653031696 >> 172.21.15.16:0/653031696 conn(0x55c4e9d22460 msgr2=0x55c4ea0d83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.259+0000 7f09de85e740 1 -- 172.21.15.16:0/653031696 shutdown_connections 2024-08-28T19:39:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.259+0000 7f09de85e740 1 -- 172.21.15.16:0/653031696 wait complete. 2024-08-28T19:39:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.260+0000 7f09de85e740 20 client.29252 trim_cache size 0 max 16384 2024-08-28T19:39:38.263 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:39:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.260+0000 7f09d82e9640 1 -- 172.21.15.16:0/544326425 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09bc0b4d20 msgr2=0x7f09bc0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.260+0000 7f09d82e9640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09bc0b4d20 0x7f09bc0b7230 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x55c4ea0d1f90 tx=0x7f09cc0a7f80 comp rx=0 tx=0).stop 2024-08-28T19:39:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.260+0000 7f09d82e9640 1 -- 172.21.15.16:0/544326425 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09d0102800 msgr2=0x7f09d00ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.260+0000 7f09d82e9640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09d0102800 0x7f09d00ff410 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00eb90 tx=0x7f09cc00c6a0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.260+0000 7f09d82e9640 1 -- 172.21.15.16:0/544326425 shutdown_connections 2024-08-28T19:39:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.260+0000 7f09d82e9640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f09bc0b4d20 0x7f09bc0b7230 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.260+0000 7f09d82e9640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09d0108a90 0x7f09d00ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.260+0000 7f09d82e9640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d0108140 0x7f09d00ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.260+0000 7f09d82e9640 1 --2- 172.21.15.16:0/544326425 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09d0102800 0x7f09d00ff410 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.260+0000 7f09d82e9640 1 -- 172.21.15.16:0/544326425 >> 172.21.15.16:0/544326425 conn(0x7f09d00794d0 msgr2=0x7f09d01017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.261+0000 7f09d82e9640 1 -- 172.21.15.16:0/544326425 shutdown_connections 2024-08-28T19:39:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:38.261+0000 7f09d82e9640 1 -- 172.21.15.16:0/544326425 wait complete. 2024-08-28T19:39:38.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 122460 23.38 MB/sec execute 194 sec latency 284.046 ms 2024-08-28T19:39:38.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:39:39.414 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:39.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 122999 23.38 MB/sec execute 195 sec latency 159.994 ms 2024-08-28T19:39:39.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:39 smithi016 ceph-mon[18556]: pgmap v921: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 701 op/s 2024-08-28T19:39:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:39 smithi028 ceph-mon[21769]: pgmap v921: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 701 op/s 2024-08-28T19:39:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:39 smithi115 ceph-mon[22184]: pgmap v921: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 701 op/s 2024-08-28T19:39:40.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 123783 23.51 MB/sec execute 196 sec latency 93.935 ms 2024-08-28T19:39:41.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.107+0000 7f9ae17f7640 1 -- 172.21.15.16:0/2372968092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc108a90 msgr2=0x7f9adc10f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:41.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.107+0000 7f9ae17f7640 1 --2- 172.21.15.16:0/2372968092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc108a90 0x7f9adc10f2c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009ca0 tx=0x7f9ad402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.107+0000 7f9ae17f7640 1 -- 172.21.15.16:0/2372968092 shutdown_connections 2024-08-28T19:39:41.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.107+0000 7f9ae17f7640 1 --2- 172.21.15.16:0/2372968092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc108a90 0x7f9adc10f2c0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.107+0000 7f9ae17f7640 1 --2- 172.21.15.16:0/2372968092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9adc108140 0x7f9adc108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.107+0000 7f9ae17f7640 1 --2- 172.21.15.16:0/2372968092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9adc102800 0x7f9adc102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.107+0000 7f9ae17f7640 1 -- 172.21.15.16:0/2372968092 >> 172.21.15.16:0/2372968092 conn(0x7f9adc0794d0 msgr2=0x7f9adc0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:41.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.107+0000 7f9ae17f7640 1 -- 172.21.15.16:0/2372968092 shutdown_connections 2024-08-28T19:39:41.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.107+0000 7f9ae17f7640 1 -- 172.21.15.16:0/2372968092 wait complete. 2024-08-28T19:39:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.109+0000 7f9ae17f7640 1 Processor -- start 2024-08-28T19:39:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.109+0000 7f9ae17f7640 1 -- start start 2024-08-28T19:39:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.109+0000 7f9ae17f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9adc102800 0x7f9adc104e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.109+0000 7f9ae17f7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9adc108140 0x7f9adc1053c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.109+0000 7f9ae17f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc108a90 0x7f9adc105900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.109+0000 7f9ae17f7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc112740 con 0x7f9adc108a90 2024-08-28T19:39:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.109+0000 7f9ae17f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1125c0 con 0x7f9adc102800 2024-08-28T19:39:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.109+0000 7f9ae17f7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1128c0 con 0x7f9adc108140 2024-08-28T19:39:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.110+0000 7f9adb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc108a90 0x7f9adc105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.110+0000 7f9adb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc108a90 0x7f9adc105900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47722/0 (socket says 172.21.15.16:47722) 2024-08-28T19:39:41.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.110+0000 7f9adb7fe640 1 -- 172.21.15.16:0/816287346 learned_addr learned my addr 172.21.15.16:0/816287346 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:41.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.110+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9adc108140 0x7f9adc1053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.110+0000 7f9adb7fe640 1 -- 172.21.15.16:0/816287346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9adc108140 msgr2=0x7f9adc1053c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.110+0000 7f9adb7fe640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9adc108140 0x7f9adc1053c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.110+0000 7f9adb7fe640 1 -- 172.21.15.16:0/816287346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9adc102800 msgr2=0x7f9adc104e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-28T19:39:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.110+0000 7f9adb7fe640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9adc102800 0x7f9adc104e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.110+0000 7f9adb7fe640 1 -- 172.21.15.16:0/816287346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9adc1a8470 con 0x7f9adc108a90 2024-08-28T19:39:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.111+0000 7f9adb7fe640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc108a90 0x7f9adc105900 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f9ad402f540 tx=0x7f9ad40316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.111+0000 7f9ac3fff640 1 -- 172.21.15.16:0/816287346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4003b70 con 0x7f9adc108a90 2024-08-28T19:39:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.111+0000 7f9ac3fff640 1 -- 172.21.15.16:0/816287346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ad4003d10 con 0x7f9adc108a90 2024-08-28T19:39:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.111+0000 7f9ac3fff640 1 -- 172.21.15.16:0/816287346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad402fa30 con 0x7f9adc108a90 2024-08-28T19:39:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.112+0000 7f9ae17f7640 1 -- 172.21.15.16:0/816287346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9adc1a8700 con 0x7f9adc108a90 2024-08-28T19:39:41.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.112+0000 7f9ae17f7640 1 -- 172.21.15.16:0/816287346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9adc1a8bf0 con 0x7f9adc108a90 2024-08-28T19:39:41.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.115+0000 7f9ae7d6c740 1 Processor -- start 2024-08-28T19:39:41.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.115+0000 7f9ac3fff640 1 -- 172.21.15.16:0/816287346 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f9ad402fbd0 con 0x7f9adc108a90 2024-08-28T19:39:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.115+0000 7f9ae7d6c740 1 -- start start 2024-08-28T19:39:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.115+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdf03070 0x556afe2a8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.115+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.115+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdee2390 0x556afe2a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.115+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556afe21f680 con 0x556afdee2390 2024-08-28T19:39:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.115+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556afe20ec10 con 0x556afdeda1b0 2024-08-28T19:39:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.115+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556afe212bc0 con 0x556afdf03070 2024-08-28T19:39:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9adaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9adaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51516/0 (socket says 172.21.15.16:51516) 2024-08-28T19:39:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9adaffd640 1 -- 172.21.15.16:0/1908541550 learned_addr learned my addr 172.21.15.16:0/1908541550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/1908541550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdee2390 0x556afe2a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/1908541550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdf03070 0x556afe2a8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9adaffd640 1 -- 172.21.15.16:0/1908541550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdf03070 msgr2=0x556afe2a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9adaffd640 1 --2- 172.21.15.16:0/1908541550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdf03070 0x556afe2a8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9ac3fff640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ac40b4df0 0x7f9ac40b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9adaffd640 1 -- 172.21.15.16:0/1908541550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdee2390 msgr2=0x556afe2a8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9adaffd640 1 --2- 172.21.15.16:0/1908541550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdee2390 0x556afe2a8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9adaffd640 1 -- 172.21.15.16:0/1908541550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556afdf6e650 con 0x556afdeda1b0 2024-08-28T19:39:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9adaffd640 1 --2- 172.21.15.16:0/1908541550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2a8850 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009ab0 tx=0x7f9acc009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14582711fd79d7b9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.116+0000 7f9ac3fff640 1 -- 172.21.15.16:0/816287346 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f9ad4106ca0 con 0x7f9adc108a90 2024-08-28T19:39:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.117+0000 7f9adb7fe640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ac40b4df0 0x7f9ac40b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.117+0000 7f9ac17fa640 1 -- 172.21.15.16:0/1908541550 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9acc00ed10 con 0x556afdeda1b0 2024-08-28T19:39:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.117+0000 7f9ac17fa640 1 -- 172.21.15.16:0/1908541550 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9acc0035f0 con 0x556afdeda1b0 2024-08-28T19:39:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.117+0000 7f9ac17fa640 1 -- 172.21.15.16:0/1908541550 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9acc010690 con 0x556afdeda1b0 2024-08-28T19:39:41.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.118+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/1908541550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 msgr2=0x556afe2a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.118+0000 7f9adb7fe640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ac40b4df0 0x7f9ac40b7300 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40ccf50 tx=0x7f9ad403e040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.118+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/1908541550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2a8850 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009ab0 tx=0x7f9acc009f70 comp rx=0 tx=0).stop 2024-08-28T19:39:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.118+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/1908541550 shutdown_connections 2024-08-28T19:39:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.118+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/1908541550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdee2390 0x556afe2a8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.118+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/1908541550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2a8850 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.118+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/1908541550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdf03070 0x556afe2a8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.118+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/1908541550 >> 172.21.15.16:0/1908541550 conn(0x556afdef7460 msgr2=0x556afe2a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.119+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/1908541550 shutdown_connections 2024-08-28T19:39:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.119+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/1908541550 wait complete. 2024-08-28T19:39:41.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.120+0000 7f9ae7d6c740 1 -- start start 2024-08-28T19:39:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.120+0000 7f9ae7d6c740 1 Processor -- start 2024-08-28T19:39:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.120+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdeda1b0 0x556afe2b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.120+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 0x556afe2b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.120+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdf03070 0x556afe2c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.120+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556afe071700 con 0x556afdeda1b0 2024-08-28T19:39:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.120+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556afe195750 con 0x556afdee2390 2024-08-28T19:39:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.120+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556afe2b6af0 con 0x556afdf03070 2024-08-28T19:39:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.120+0000 7f9ae7d6c740 10 client.0 _ll_register_callbacks cb 0x556afe1c4e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.121+0000 7f9ada7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdeda1b0 0x556afe2b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.121+0000 7f9ada7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdeda1b0 0x556afe2b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47734/0 (socket says 172.21.15.16:47734) 2024-08-28T19:39:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.121+0000 7f9ada7fc640 1 -- 172.21.15.16:0/2741748609 learned_addr learned my addr 172.21.15.16:0/2741748609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.121+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdf03070 0x556afe2c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.121+0000 7f9adaffd640 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 0x556afe2b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.121+0000 7f9ada7fc640 1 -- 172.21.15.16:0/2741748609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdf03070 msgr2=0x556afe2c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.121+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdf03070 0x556afe2c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.121+0000 7f9ada7fc640 1 -- 172.21.15.16:0/2741748609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 msgr2=0x556afe2b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.121+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 0x556afe2b3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.121+0000 7f9ada7fc640 1 -- 172.21.15.16:0/2741748609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556afe1ce900 con 0x556afdeda1b0 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.122+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdeda1b0 0x556afe2b3a70 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0012bd0 tx=0x7f9ad0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11340ede9ca87b1d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.122+0000 7f9abeff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.122+0000 7f9abeff5640 1 -- 172.21.15.16:0/2741748609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad000ed00 con 0x556afdeda1b0 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.122+0000 7f9abeff5640 1 -- 172.21.15.16:0/2741748609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f9ad0004040 con 0x556afdeda1b0 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.122+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2741748609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9acc0035f0 con 0x556afdeda1b0 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9abeff5640 1 -- 172.21.15.16:0/2741748609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f9ad000eea0 con 0x556afdeda1b0 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9ae7d6c740 10 client.19821 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2741748609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x556afe2cb110 con 0x556afdeda1b0 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9abeff5640 1 -- 172.21.15.16:0/2741748609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9acc0035f0 con 0x556afdeda1b0 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9abeff5640 1 -- 172.21.15.16:0/2741748609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7f9ad000f420 con 0x556afdeda1b0 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9ae7d6c740 10 client.19821 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9ae7d6c740 10 client.19821 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:39:41.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9ae7d6c740 20 client.19821 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:41.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x556afdec19f0 0x556afe2ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9ae7d6c740 4 client.19821 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:39:41.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2741748609 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556afe2215d0 con 0x556afdec19f0 2024-08-28T19:39:41.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.123+0000 7f9adaffd640 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x556afdec19f0 0x556afe2ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.124+0000 7f9abeff5640 10 client.19821 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:39:41.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.124+0000 7f9adaffd640 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x556afdec19f0 0x556afe2ce600 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1328c67b675cf4c server_cookie=8fd676005262bcbb in_seq=0 out_seq=0 2024-08-28T19:39:41.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9abeff5640 1 -- 172.21.15.16:0/2741748609 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556afe2215d0 con 0x556afdec19f0 2024-08-28T19:39:41.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9abeff5640 10 client.19821 handle_command_reply: tid=0 2024-08-28T19:39:41.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9ae7d6c740 1 client.19821 shutdown 2024-08-28T19:39:41.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2741748609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdeda1b0 msgr2=0x556afe2b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdeda1b0 0x556afe2b3a70 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0012bd0 tx=0x7f9ad0004730 comp rx=0 tx=0).stop 2024-08-28T19:39:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2741748609 shutdown_connections 2024-08-28T19:39:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x556afdec19f0 0x556afe2ce600 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdf03070 0x556afe2c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 0x556afe2b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2741748609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdeda1b0 0x556afe2b3a70 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2741748609 >> 172.21.15.16:0/2741748609 conn(0x556afdef7460 msgr2=0x556afe2ab4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.125+0000 7f9abeff5640 0 client.19821 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:39:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.126+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2741748609 shutdown_connections 2024-08-28T19:39:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.126+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2741748609 wait complete. 2024-08-28T19:39:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.126+0000 7f9ae7d6c740 20 client.19821 trim_cache size 0 max 16384 2024-08-28T19:39:41.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.143+0000 7f9ae7d6c740 1 Processor -- start 2024-08-28T19:39:41.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.143+0000 7f9ae7d6c740 1 -- start start 2024-08-28T19:39:41.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.143+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdeda1b0 0x556afe2b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.143+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 0x556afe2ab410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.143+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdf03070 0x556afe2ab950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.143+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1ae9d0 con 0x556afdf03070 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.143+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1b06f0 con 0x556afdee2390 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.143+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1af130 con 0x556afdeda1b0 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9adaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 0x556afe2ab410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9ada7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdf03070 0x556afe2ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9adaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 0x556afe2ab410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51540/0 (socket says 172.21.15.16:51540) 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9adaffd640 1 -- 172.21.15.16:0/3323439789 learned_addr learned my addr 172.21.15.16:0/3323439789 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/3323439789 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdeda1b0 0x556afe2b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9adaffd640 1 -- 172.21.15.16:0/3323439789 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdeda1b0 msgr2=0x556afe2b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9adaffd640 1 --2- 172.21.15.16:0/3323439789 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdeda1b0 0x556afe2b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9adaffd640 1 -- 172.21.15.16:0/3323439789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdf03070 msgr2=0x556afe2ab950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9adaffd640 1 --2- 172.21.15.16:0/3323439789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdf03070 0x556afe2ab950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9adaffd640 1 -- 172.21.15.16:0/3323439789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556afe2a6fc0 con 0x556afdee2390 2024-08-28T19:39:41.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9adaffd640 1 --2- 172.21.15.16:0/3323439789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 0x556afe2ab410 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x556afe2a6f90 tx=0x7f9acc00eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7224629b9cfbd0c6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:41.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9ac0ff9640 1 -- 172.21.15.16:0/3323439789 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9acc00ed90 con 0x556afdee2390 2024-08-28T19:39:41.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9ac0ff9640 1 -- 172.21.15.16:0/3323439789 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9acc005040 con 0x556afdee2390 2024-08-28T19:39:41.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.144+0000 7f9ac0ff9640 1 -- 172.21.15.16:0/3323439789 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9acc003830 con 0x556afdee2390 2024-08-28T19:39:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.145+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/3323439789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 msgr2=0x556afe2ab410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:41.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.145+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/3323439789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 0x556afe2ab410 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x556afe2a6f90 tx=0x7f9acc00eca0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.145+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/3323439789 shutdown_connections 2024-08-28T19:39:41.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.145+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/3323439789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdf03070 0x556afe2ab950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.145+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/3323439789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdee2390 0x556afe2ab410 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.145+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/3323439789 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdeda1b0 0x556afe2b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.145+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/3323439789 >> 172.21.15.16:0/3323439789 conn(0x556afdef7460 msgr2=0x556afe2b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:41.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.146+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/3323439789 shutdown_connections 2024-08-28T19:39:41.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.146+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/3323439789 wait complete. 2024-08-28T19:39:41.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.146+0000 7f9ae7d6c740 1 -- start start 2024-08-28T19:39:41.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.146+0000 7f9ae7d6c740 1 Processor -- start 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.146+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2c0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.146+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdee2390 0x556afe2c0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.146+0000 7f9ae7d6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdf03070 0x556afe2c1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.146+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1aff60 con 0x556afdf03070 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.146+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1af800 con 0x556afdeda1b0 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.146+0000 7f9ae7d6c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1aca10 con 0x556afdee2390 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9ae7d6c740 10 client.0 _ll_register_callbacks cb 0x7f9ad0012530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9ada7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2c0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9ada7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2c0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51542/0 (socket says 172.21.15.16:51542) 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9adaffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdee2390 0x556afe2c0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9ada7fc640 1 -- 172.21.15.16:0/2639382849 learned_addr learned my addr 172.21.15.16:0/2639382849 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdf03070 0x556afe2c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9ada7fc640 1 -- 172.21.15.16:0/2639382849 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdee2390 msgr2=0x556afe2c0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:41.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdee2390 0x556afe2c0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9ada7fc640 1 -- 172.21.15.16:0/2639382849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdf03070 msgr2=0x556afe2c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:41.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdf03070 0x556afe2c1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9ada7fc640 1 -- 172.21.15.16:0/2639382849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556afe338050 con 0x556afdeda1b0 2024-08-28T19:39:41.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9ada7fc640 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2c0600 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000ee70 tx=0x7f9ad0012f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33a0c4ef313ff257 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:41.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.147+0000 7f9abeff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:39:41.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.148+0000 7f9abeff5640 1 -- 172.21.15.16:0/2639382849 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad000fe60 con 0x556afdeda1b0 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.148+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2639382849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9acc005040 con 0x556afdeda1b0 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.148+0000 7f9abeff5640 1 -- 172.21.15.16:0/2639382849 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f9ad0004250 con 0x556afdeda1b0 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.148+0000 7f9abeff5640 1 -- 172.21.15.16:0/2639382849 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f9ad0044050 con 0x556afdeda1b0 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.148+0000 7f9ae7d6c740 10 client.29267 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.148+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2639382849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x556afe3392d0 con 0x556afdeda1b0 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.149+0000 7f9abeff5640 1 -- 172.21.15.16:0/2639382849 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9acc005040 con 0x556afdeda1b0 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.149+0000 7f9abeff5640 1 -- 172.21.15.16:0/2639382849 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7f9ad0003ca0 con 0x556afdeda1b0 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.149+0000 7f9ae7d6c740 10 client.29267 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.149+0000 7f9ae7d6c740 10 client.29267 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.149+0000 7f9ae7d6c740 20 client.29267 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.149+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x556afde96da0 0x556afe33ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.149+0000 7f9ae7d6c740 4 client.29267 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.149+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2639382849 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556afe21d070 con 0x556afde96da0 2024-08-28T19:39:41.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.149+0000 7f9adaffd640 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x556afde96da0 0x556afe33ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9abeff5640 10 client.29267 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9adaffd640 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x556afde96da0 0x556afe33ba90 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5919a9cce3505e40 server_cookie=caa9709f49a1589 in_seq=0 out_seq=0 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9abeff5640 1 -- 172.21.15.16:0/2639382849 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556afe21d070 con 0x556afde96da0 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9abeff5640 10 client.29267 handle_command_reply: tid=0 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9ae7d6c740 1 client.29267 shutdown 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2639382849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 msgr2=0x556afe2c0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2c0600 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000ee70 tx=0x7f9ad0012f90 comp rx=0 tx=0).stop 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2639382849 shutdown_connections 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x556afde96da0 0x556afe33ba90 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556afdf03070 0x556afe2c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556afdee2390 0x556afe2c0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9ae7d6c740 1 --2- 172.21.15.16:0/2639382849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556afdeda1b0 0x556afe2c0600 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2639382849 >> 172.21.15.16:0/2639382849 conn(0x556afdef7460 msgr2=0x556afe2a92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.150+0000 7f9abeff5640 0 client.29267 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:39:41.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.151+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2639382849 shutdown_connections 2024-08-28T19:39:41.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.151+0000 7f9ae7d6c740 1 -- 172.21.15.16:0/2639382849 wait complete. 2024-08-28T19:39:41.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.151+0000 7f9ae7d6c740 20 client.29267 trim_cache size 0 max 16384 2024-08-28T19:39:41.156 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:39:41.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.151+0000 7f9ae17f7640 1 -- 172.21.15.16:0/816287346 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ac40b4df0 msgr2=0x7f9ac40b7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.151+0000 7f9ae17f7640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ac40b4df0 0x7f9ac40b7300 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40ccf50 tx=0x7f9ad403e040 comp rx=0 tx=0).stop 2024-08-28T19:39:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.151+0000 7f9ae17f7640 1 -- 172.21.15.16:0/816287346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc108a90 msgr2=0x7f9adc105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.151+0000 7f9ae17f7640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc108a90 0x7f9adc105900 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f9ad402f540 tx=0x7f9ad40316f0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.152+0000 7f9ae17f7640 1 -- 172.21.15.16:0/816287346 shutdown_connections 2024-08-28T19:39:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.152+0000 7f9ae17f7640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc108a90 0x7f9adc105900 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.152+0000 7f9ae17f7640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9adc108140 0x7f9adc1053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.152+0000 7f9ae17f7640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9ac40b4df0 0x7f9ac40b7300 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.152+0000 7f9ae17f7640 1 --2- 172.21.15.16:0/816287346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9adc102800 0x7f9adc104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.152+0000 7f9ae17f7640 1 -- 172.21.15.16:0/816287346 >> 172.21.15.16:0/816287346 conn(0x7f9adc0794d0 msgr2=0x7f9adc0ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.152+0000 7f9ae17f7640 1 -- 172.21.15.16:0/816287346 shutdown_connections 2024-08-28T19:39:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:41.152+0000 7f9ae17f7640 1 -- 172.21.15.16:0/816287346 wait complete. 2024-08-28T19:39:41.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 124457 23.55 MB/sec execute 197 sec latency 126.333 ms 2024-08-28T19:39:41.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:41 smithi016 ceph-mon[18556]: pgmap v922: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 17 MiB/s wr, 727 op/s 2024-08-28T19:39:41.713 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:41 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:39:41.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:41 smithi028 ceph-mon[21769]: pgmap v922: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 17 MiB/s wr, 727 op/s 2024-08-28T19:39:41.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:41 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:39:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:41 smithi115 ceph-mon[22184]: pgmap v922: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 17 MiB/s wr, 727 op/s 2024-08-28T19:39:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:41 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:39:42.089 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:39:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:42.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 125227 23.67 MB/sec execute 198 sec latency 139.661 ms 2024-08-28T19:39:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.919+0000 7fec42df7640 1 -- 172.21.15.16:0/434268453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c109280 msgr2=0x7fec3c109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.919+0000 7fec42df7640 1 --2- 172.21.15.16:0/434268453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c109280 0x7fec3c109690 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fec28009900 tx=0x7fec2802a410 comp rx=0 tx=0).stop 2024-08-28T19:39:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.920+0000 7fec42df7640 1 -- 172.21.15.16:0/434268453 shutdown_connections 2024-08-28T19:39:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.920+0000 7fec42df7640 1 --2- 172.21.15.16:0/434268453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec3c109bd0 0x7fec3c110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.920+0000 7fec42df7640 1 --2- 172.21.15.16:0/434268453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c109280 0x7fec3c109690 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.920+0000 7fec42df7640 1 --2- 172.21.15.16:0/434268453 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec3c103940 0x7fec3c103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.920+0000 7fec42df7640 1 -- 172.21.15.16:0/434268453 >> 172.21.15.16:0/434268453 conn(0x7fec3c0ff3d0 msgr2=0x7fec3c101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:42.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.922+0000 7fec42df7640 1 -- 172.21.15.16:0/434268453 shutdown_connections 2024-08-28T19:39:42.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.922+0000 7fec42df7640 1 -- 172.21.15.16:0/434268453 wait complete. 2024-08-28T19:39:42.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.923+0000 7fec42df7640 1 Processor -- start 2024-08-28T19:39:42.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.923+0000 7fec42df7640 1 -- start start 2024-08-28T19:39:42.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec42df7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec3c103940 0x7fec3c10c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec42df7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c109280 0x7fec3c10ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec42df7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec3c109bd0 0x7fec3c10d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec42df7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c076130 con 0x7fec3c109280 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec42df7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c075fb0 con 0x7fec3c103940 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec42df7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c0762b0 con 0x7fec3c109bd0 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c109280 0x7fec3c10ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec40b93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec3c103940 0x7fec3c10c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec40b93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec3c103940 0x7fec3c10c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51570/0 (socket says 172.21.15.16:51570) 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c109280 0x7fec3c10ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47776/0 (socket says 172.21.15.16:47776) 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec40b93640 1 -- 172.21.15.16:0/3944912437 learned_addr learned my addr 172.21.15.16:0/3944912437 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec41394640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec3c109bd0 0x7fec3c10d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec40b93640 1 -- 172.21.15.16:0/3944912437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec3c109bd0 msgr2=0x7fec3c10d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec40b93640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec3c109bd0 0x7fec3c10d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec40b93640 1 -- 172.21.15.16:0/3944912437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c109280 msgr2=0x7fec3c10ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec40b93640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c109280 0x7fec3c10ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.924+0000 7fec40b93640 1 -- 172.21.15.16:0/3944912437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3c1ac750 con 0x7fec3c103940 2024-08-28T19:39:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.925+0000 7fec41394640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec3c109bd0 0x7fec3c10d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:39:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.925+0000 7fec3bfff640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c109280 0x7fec3c10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.925+0000 7fec40b93640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec3c103940 0x7fec3c10c970 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fec3000e9f0 tx=0x7fec3000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.925+0000 7fec39ffb640 1 -- 172.21.15.16:0/3944912437 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3000cd80 con 0x7fec3c103940 2024-08-28T19:39:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.925+0000 7fec42df7640 1 -- 172.21.15.16:0/3944912437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec3c1aca40 con 0x7fec3c103940 2024-08-28T19:39:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.925+0000 7fec39ffb640 1 -- 172.21.15.16:0/3944912437 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec300108e0 con 0x7fec3c103940 2024-08-28T19:39:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.925+0000 7fec39ffb640 1 -- 172.21.15.16:0/3944912437 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec30018980 con 0x7fec3c103940 2024-08-28T19:39:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.925+0000 7fec42df7640 1 -- 172.21.15.16:0/3944912437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec3c1acfa0 con 0x7fec3c103940 2024-08-28T19:39:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.931+0000 7fec4936c740 1 Processor -- start 2024-08-28T19:39:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.931+0000 7fec4936c740 1 -- start start 2024-08-28T19:39:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.932+0000 7fec4936c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab046070 0x55fcab3eb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.932+0000 7fec4936c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab01d1b0 0x55fcab3eb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.932+0000 7fec4936c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab025390 0x55fcab3ebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.932+0000 7fec4936c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fcab362680 con 0x55fcab01d1b0 2024-08-28T19:39:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.932+0000 7fec4936c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fcab351c10 con 0x55fcab025390 2024-08-28T19:39:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.932+0000 7fec4936c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55fcab355bc0 con 0x55fcab046070 2024-08-28T19:39:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.932+0000 7fec3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab025390 0x55fcab3ebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.932+0000 7fec3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab025390 0x55fcab3ebdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51572/0 (socket says 172.21.15.16:51572) 2024-08-28T19:39:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.932+0000 7fec3bfff640 1 -- 172.21.15.16:0/1943584472 learned_addr learned my addr 172.21.15.16:0/1943584472 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.933+0000 7fec3bfff640 1 -- 172.21.15.16:0/1943584472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab046070 msgr2=0x55fcab3eb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.933+0000 7fec3bfff640 1 --2- 172.21.15.16:0/1943584472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab046070 0x55fcab3eb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.933+0000 7fec3bfff640 1 -- 172.21.15.16:0/1943584472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab01d1b0 msgr2=0x55fcab3eb890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:39:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.933+0000 7fec3bfff640 1 --2- 172.21.15.16:0/1943584472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab01d1b0 0x55fcab3eb890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.933+0000 7fec3bfff640 1 -- 172.21.15.16:0/1943584472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcab0b1650 con 0x55fcab025390 2024-08-28T19:39:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.934+0000 7fec3bfff640 1 --2- 172.21.15.16:0/1943584472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab025390 0x55fcab3ebdd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fec28009900 tx=0x7fec28000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba6befa29ed67a88 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.934+0000 7fec39ffb640 1 -- 172.21.15.16:0/3944912437 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fec30010430 con 0x7fec3c103940 2024-08-28T19:39:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.934+0000 7fec26ffd640 1 -- 172.21.15.16:0/1943584472 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec28035040 con 0x55fcab025390 2024-08-28T19:39:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.934+0000 7fec26ffd640 1 -- 172.21.15.16:0/1943584472 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec280054e0 con 0x55fcab025390 2024-08-28T19:39:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.934+0000 7fec26ffd640 1 -- 172.21.15.16:0/1943584472 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec280049b0 con 0x55fcab025390 2024-08-28T19:39:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.935+0000 7fec39ffb640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec2c0b4da0 0x7fec2c0b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.936+0000 7fec39ffb640 1 -- 172.21.15.16:0/3944912437 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fec300e5cb0 con 0x7fec3c103940 2024-08-28T19:39:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.936+0000 7fec41394640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec2c0b4da0 0x7fec2c0b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.936+0000 7fec4936c740 1 -- 172.21.15.16:0/1943584472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab025390 msgr2=0x55fcab3ebdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.936+0000 7fec4936c740 1 --2- 172.21.15.16:0/1943584472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab025390 0x55fcab3ebdd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fec28009900 tx=0x7fec28000bc0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.937+0000 7fec41394640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec2c0b4da0 0x7fec2c0b72b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fec34005c40 tx=0x7fec34005bd0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.938+0000 7fec4936c740 1 -- 172.21.15.16:0/1943584472 shutdown_connections 2024-08-28T19:39:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.938+0000 7fec4936c740 1 --2- 172.21.15.16:0/1943584472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab025390 0x55fcab3ebdd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.938+0000 7fec4936c740 1 --2- 172.21.15.16:0/1943584472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab01d1b0 0x55fcab3eb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.938+0000 7fec4936c740 1 --2- 172.21.15.16:0/1943584472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab046070 0x55fcab3eb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.938+0000 7fec4936c740 1 -- 172.21.15.16:0/1943584472 >> 172.21.15.16:0/1943584472 conn(0x55fcab03a460 msgr2=0x55fcab3e9240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.938+0000 7fec4936c740 1 -- 172.21.15.16:0/1943584472 shutdown_connections 2024-08-28T19:39:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.938+0000 7fec4936c740 1 -- 172.21.15.16:0/1943584472 wait complete. 2024-08-28T19:39:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.939+0000 7fec4936c740 1 -- start start 2024-08-28T19:39:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.939+0000 7fec4936c740 1 Processor -- start 2024-08-28T19:39:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.940+0000 7fec4936c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 0x55fcab3f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.940+0000 7fec4936c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab3f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.940+0000 7fec4936c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 0x55fcab402df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.940+0000 7fec4936c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fcab1b4700 con 0x55fcab025390 2024-08-28T19:39:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.940+0000 7fec4936c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fcab2d8750 con 0x55fcab046070 2024-08-28T19:39:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.940+0000 7fec4936c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55fcab3f9ad0 con 0x55fcab01d1b0 2024-08-28T19:39:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.940+0000 7fec4936c740 10 client.0 _ll_register_callbacks cb 0x55fcab307e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.940+0000 7fec41394640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 0x55fcab402df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.940+0000 7fec41394640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 0x55fcab402df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51586/0 (socket says 172.21.15.16:51586) 2024-08-28T19:39:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.940+0000 7fec41394640 1 -- 172.21.15.16:0/1385538422 learned_addr learned my addr 172.21.15.16:0/1385538422 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.940+0000 7fec40b93640 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 0x55fcab3f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.941+0000 7fec41394640 1 -- 172.21.15.16:0/1385538422 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 msgr2=0x55fcab3f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.941+0000 7fec41394640 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 0x55fcab3f6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.941+0000 7fec41394640 1 -- 172.21.15.16:0/1385538422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 msgr2=0x55fcab3f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.941+0000 7fec41394640 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab3f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.941+0000 7fec41394640 1 -- 172.21.15.16:0/1385538422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcab311900 con 0x55fcab046070 2024-08-28T19:39:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.941+0000 7fec41394640 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 0x55fcab402df0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fec340117c0 tx=0x7fec34004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa0591861f17b6fa server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.942+0000 7fec1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:39:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.942+0000 7fec1bfff640 1 -- 172.21.15.16:0/1385538422 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec34022070 con 0x55fcab046070 2024-08-28T19:39:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.942+0000 7fec1bfff640 1 -- 172.21.15.16:0/1385538422 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fec34010ad0 con 0x55fcab046070 2024-08-28T19:39:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.942+0000 7fec4936c740 1 -- 172.21.15.16:0/1385538422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec280054e0 con 0x55fcab046070 2024-08-28T19:39:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.943+0000 7fec1bfff640 1 -- 172.21.15.16:0/1385538422 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fec3402c050 con 0x55fcab046070 2024-08-28T19:39:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.943+0000 7fec4936c740 10 client.29279 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.943+0000 7fec4936c740 1 -- 172.21.15.16:0/1385538422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x55fcab4059a0 con 0x55fcab046070 2024-08-28T19:39:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.944+0000 7fec1bfff640 1 -- 172.21.15.16:0/1385538422 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec280054e0 con 0x55fcab046070 2024-08-28T19:39:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.945+0000 7fec1bfff640 1 -- 172.21.15.16:0/1385538422 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7fec34027440 con 0x55fcab046070 2024-08-28T19:39:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.945+0000 7fec4936c740 10 client.29279 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.945+0000 7fec4936c740 10 client.29279 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:39:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.945+0000 7fec4936c740 20 client.29279 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.945+0000 7fec4936c740 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55fcab0049f0 0x55fcab409420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.945+0000 7fec4936c740 4 client.29279 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:39:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.945+0000 7fec4936c740 1 -- 172.21.15.16:0/1385538422 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcab3645d0 con 0x55fcab0049f0 2024-08-28T19:39:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.945+0000 7fec40b93640 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55fcab0049f0 0x55fcab409420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.949+0000 7fec40b93640 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55fcab0049f0 0x55fcab409420 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e676aca4d8a9d395 server_cookie=6a23ee0811329b6e in_seq=0 out_seq=0 2024-08-28T19:39:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.949+0000 7fec1bfff640 10 client.29279 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:39:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.952+0000 7fec1bfff640 1 -- 172.21.15.16:0/1385538422 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55fcab3645d0 con 0x55fcab0049f0 2024-08-28T19:39:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.952+0000 7fec1bfff640 10 client.29279 handle_command_reply: tid=0 2024-08-28T19:39:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.952+0000 7fec4936c740 1 client.29279 shutdown 2024-08-28T19:39:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec4936c740 1 -- 172.21.15.16:0/1385538422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 msgr2=0x55fcab402df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec4936c740 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 0x55fcab402df0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fec340117c0 tx=0x7fec34004ab0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec4936c740 1 -- 172.21.15.16:0/1385538422 shutdown_connections 2024-08-28T19:39:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec4936c740 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55fcab0049f0 0x55fcab409420 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec4936c740 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 0x55fcab402df0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec4936c740 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab3f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec4936c740 1 --2- 172.21.15.16:0/1385538422 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 0x55fcab3f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec1bfff640 0 client.29279 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:39:42.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec4936c740 1 -- 172.21.15.16:0/1385538422 >> 172.21.15.16:0/1385538422 conn(0x55fcab03a460 msgr2=0x55fcab3ee4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:42.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec4936c740 1 -- 172.21.15.16:0/1385538422 shutdown_connections 2024-08-28T19:39:42.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec4936c740 1 -- 172.21.15.16:0/1385538422 wait complete. 2024-08-28T19:39:42.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.953+0000 7fec4936c740 20 client.29279 trim_cache size 0 max 16384 2024-08-28T19:39:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec4936c740 1 Processor -- start 2024-08-28T19:39:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec4936c740 1 -- start start 2024-08-28T19:39:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec4936c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab01d1b0 0x55fcab402cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec4936c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab403230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec4936c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab046070 0x55fcab3e8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec4936c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1b7300 con 0x55fcab025390 2024-08-28T19:39:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec4936c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1b9020 con 0x55fcab01d1b0 2024-08-28T19:39:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec4936c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1b7a60 con 0x55fcab046070 2024-08-28T19:39:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec40b93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab01d1b0 0x55fcab402cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec40b93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab01d1b0 0x55fcab402cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51600/0 (socket says 172.21.15.16:51600) 2024-08-28T19:39:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec40b93640 1 -- 172.21.15.16:0/2652734979 learned_addr learned my addr 172.21.15.16:0/2652734979 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.974+0000 7fec3bfff640 1 --2- 172.21.15.16:0/2652734979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab403230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.975+0000 7fec40b93640 1 -- 172.21.15.16:0/2652734979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab046070 msgr2=0x55fcab3e8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.975+0000 7fec41394640 1 --2- 172.21.15.16:0/2652734979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab046070 0x55fcab3e8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.975+0000 7fec40b93640 1 --2- 172.21.15.16:0/2652734979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab046070 0x55fcab3e8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.975+0000 7fec40b93640 1 -- 172.21.15.16:0/2652734979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 msgr2=0x55fcab403230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.975+0000 7fec40b93640 1 --2- 172.21.15.16:0/2652734979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab403230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.975+0000 7fec40b93640 1 -- 172.21.15.16:0/2652734979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcab3eb140 con 0x55fcab01d1b0 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.975+0000 7fec41394640 1 --2- 172.21.15.16:0/2652734979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab046070 0x55fcab3e8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.976+0000 7fec40b93640 1 --2- 172.21.15.16:0/2652734979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab01d1b0 0x55fcab402cf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fec3000e9c0 tx=0x7fec300a7eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7231f60c935b427e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.976+0000 7fec267fc640 1 -- 172.21.15.16:0/2652734979 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec300b07e0 con 0x55fcab01d1b0 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.976+0000 7fec267fc640 1 -- 172.21.15.16:0/2652734979 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec300b5290 con 0x55fcab01d1b0 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.976+0000 7fec267fc640 1 -- 172.21.15.16:0/2652734979 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec300e3250 con 0x55fcab01d1b0 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.977+0000 7fec4936c740 1 -- 172.21.15.16:0/2652734979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab01d1b0 msgr2=0x55fcab402cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.977+0000 7fec4936c740 1 --2- 172.21.15.16:0/2652734979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab01d1b0 0x55fcab402cf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fec3000e9c0 tx=0x7fec300a7eb0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.977+0000 7fec4936c740 1 -- 172.21.15.16:0/2652734979 shutdown_connections 2024-08-28T19:39:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.977+0000 7fec4936c740 1 --2- 172.21.15.16:0/2652734979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab046070 0x55fcab3e8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.977+0000 7fec4936c740 1 --2- 172.21.15.16:0/2652734979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab403230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.977+0000 7fec4936c740 1 --2- 172.21.15.16:0/2652734979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab01d1b0 0x55fcab402cf0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.977+0000 7fec4936c740 1 -- 172.21.15.16:0/2652734979 >> 172.21.15.16:0/2652734979 conn(0x55fcab03a460 msgr2=0x55fcab3f5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.977+0000 7fec4936c740 1 -- 172.21.15.16:0/2652734979 shutdown_connections 2024-08-28T19:39:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.977+0000 7fec4936c740 1 -- 172.21.15.16:0/2652734979 wait complete. 2024-08-28T19:39:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.978+0000 7fec4936c740 1 -- start start 2024-08-28T19:39:42.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.978+0000 7fec4936c740 1 Processor -- start 2024-08-28T19:39:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.978+0000 7fec4936c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 0x55fcab3fed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.978+0000 7fec4936c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab3ff290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.978+0000 7fec4936c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 0x55fcab3fbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.978+0000 7fec4936c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1b8890 con 0x55fcab025390 2024-08-28T19:39:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.978+0000 7fec4936c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1b8130 con 0x55fcab046070 2024-08-28T19:39:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.978+0000 7fec4936c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1b5300 con 0x55fcab01d1b0 2024-08-28T19:39:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.978+0000 7fec4936c740 10 client.0 _ll_register_callbacks cb 0x7fec34015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab3ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec40b93640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 0x55fcab3fed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec40b93640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 0x55fcab3fed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:34454/0 (socket says 172.21.15.16:34454) 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec40b93640 1 -- 172.21.15.16:0/415905653 learned_addr learned my addr 172.21.15.16:0/415905653 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab3ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47796/0 (socket says 172.21.15.16:47796) 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec41394640 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 0x55fcab3fbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec40b93640 1 -- 172.21.15.16:0/415905653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 msgr2=0x55fcab3fbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec40b93640 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 0x55fcab3fbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec40b93640 1 -- 172.21.15.16:0/415905653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 msgr2=0x55fcab3ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec40b93640 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab3ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec40b93640 1 -- 172.21.15.16:0/415905653 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcab3ff980 con 0x55fcab01d1b0 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec3bfff640 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab3ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec40b93640 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 0x55fcab3fed50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fec300e3000 tx=0x7fec3000c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ec0d126cec42e0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.979+0000 7fec1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.980+0000 7fec1bfff640 1 -- 172.21.15.16:0/415905653 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec30097030 con 0x55fcab01d1b0 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.980+0000 7fec4936c740 1 -- 172.21.15.16:0/415905653 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec300b5290 con 0x55fcab01d1b0 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.980+0000 7fec1bfff640 1 -- 172.21.15.16:0/415905653 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fec300a8200 con 0x55fcab01d1b0 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.980+0000 7fec1bfff640 1 -- 172.21.15.16:0/415905653 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fec300e3250 con 0x55fcab01d1b0 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.980+0000 7fec4936c740 10 client.28858 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.980+0000 7fec4936c740 1 -- 172.21.15.16:0/415905653 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x55fcab474200 con 0x55fcab01d1b0 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.981+0000 7fec1bfff640 1 -- 172.21.15.16:0/415905653 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec300b5290 con 0x55fcab01d1b0 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.981+0000 7fec1bfff640 1 -- 172.21.15.16:0/415905653 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7fec300cd2e0 con 0x55fcab01d1b0 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.981+0000 7fec4936c740 10 client.28858 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.981+0000 7fec4936c740 10 client.28858 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.981+0000 7fec4936c740 20 client.28858 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.981+0000 7fec4936c740 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55fcaafd9da0 0x55fcab476ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.981+0000 7fec4936c740 4 client.28858 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:39:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.981+0000 7fec4936c740 1 -- 172.21.15.16:0/415905653 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fcab360070 con 0x55fcaafd9da0 2024-08-28T19:39:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.981+0000 7fec3bfff640 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55fcaafd9da0 0x55fcab476ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.982+0000 7fec3bfff640 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55fcaafd9da0 0x55fcab476ab0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d39adae768de62a server_cookie=618962157ee9071 in_seq=0 out_seq=0 2024-08-28T19:39:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.982+0000 7fec1bfff640 10 client.28858 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:39:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.982+0000 7fec1bfff640 1 -- 172.21.15.16:0/415905653 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fcab360070 con 0x55fcaafd9da0 2024-08-28T19:39:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.982+0000 7fec1bfff640 10 client.28858 handle_command_reply: tid=0 2024-08-28T19:39:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.982+0000 7fec4936c740 1 client.28858 shutdown 2024-08-28T19:39:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec4936c740 1 -- 172.21.15.16:0/415905653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 msgr2=0x55fcab3fed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec4936c740 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 0x55fcab3fed50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fec300e3000 tx=0x7fec3000c840 comp rx=0 tx=0).stop 2024-08-28T19:39:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec4936c740 1 -- 172.21.15.16:0/415905653 shutdown_connections 2024-08-28T19:39:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec4936c740 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55fcaafd9da0 0x55fcab476ab0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec1bfff640 0 client.28858 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:39:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec4936c740 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fcab046070 0x55fcab3fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec4936c740 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcab025390 0x55fcab3ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec4936c740 1 --2- 172.21.15.16:0/415905653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55fcab01d1b0 0x55fcab3fed50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec4936c740 1 -- 172.21.15.16:0/415905653 >> 172.21.15.16:0/415905653 conn(0x55fcab03a460 msgr2=0x55fcab3e8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec4936c740 1 -- 172.21.15.16:0/415905653 shutdown_connections 2024-08-28T19:39:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec4936c740 1 -- 172.21.15.16:0/415905653 wait complete. 2024-08-28T19:39:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.983+0000 7fec4936c740 20 client.28858 trim_cache size 0 max 16384 2024-08-28T19:39:42.988 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:39:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 -- 172.21.15.16:0/3944912437 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec2c0b4da0 msgr2=0x7fec2c0b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec2c0b4da0 0x7fec2c0b72b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fec34005c40 tx=0x7fec34005bd0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 -- 172.21.15.16:0/3944912437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec3c103940 msgr2=0x7fec3c10c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec3c103940 0x7fec3c10c970 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fec3000e9f0 tx=0x7fec3000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 -- 172.21.15.16:0/3944912437 shutdown_connections 2024-08-28T19:39:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec3c109bd0 0x7fec3c10d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c109280 0x7fec3c10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec2c0b4da0 0x7fec2c0b72b0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 --2- 172.21.15.16:0/3944912437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec3c103940 0x7fec3c10c970 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 -- 172.21.15.16:0/3944912437 >> 172.21.15.16:0/3944912437 conn(0x7fec3c0ff3d0 msgr2=0x7fec3c105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 -- 172.21.15.16:0/3944912437 shutdown_connections 2024-08-28T19:39:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:42.984+0000 7fec42df7640 1 -- 172.21.15.16:0/3944912437 wait complete. 2024-08-28T19:39:43.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 125853 23.72 MB/sec execute 199 sec latency 265.814 ms 2024-08-28T19:39:43.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:43 smithi016 ceph-mon[18556]: pgmap v923: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 17 MiB/s wr, 773 op/s 2024-08-28T19:39:43.747 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:39:43.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:43 smithi028 ceph-mon[21769]: pgmap v923: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 17 MiB/s wr, 773 op/s 2024-08-28T19:39:43.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:43 smithi115 ceph-mon[22184]: pgmap v923: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 17 MiB/s wr, 773 op/s 2024-08-28T19:39:44.154 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:44.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 126454 23.77 MB/sec execute 200 sec latency 125.133 ms 2024-08-28T19:39:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.458+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1763443761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b78108170 msgr2=0x7f8b78108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.458+0000 7f8b7e56b640 1 --2- 172.21.15.16:0/1763443761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b78108170 0x7f8b78108580 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c009890 tx=0x7f8b6c02a380 comp rx=0 tx=0).stop 2024-08-28T19:39:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.459+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1763443761 shutdown_connections 2024-08-28T19:39:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.459+0000 7f8b7e56b640 1 --2- 172.21.15.16:0/1763443761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b78108ac0 0x7f8b7810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.459+0000 7f8b7e56b640 1 --2- 172.21.15.16:0/1763443761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b78108170 0x7f8b78108580 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.459+0000 7f8b7e56b640 1 --2- 172.21.15.16:0/1763443761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b78103340 0x7f8b78103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.459+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1763443761 >> 172.21.15.16:0/1763443761 conn(0x7f8b780775a0 msgr2=0x7f8b780779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.459+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1763443761 shutdown_connections 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.459+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1763443761 wait complete. 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.459+0000 7f8b7e56b640 1 Processor -- start 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.460+0000 7f8b7e56b640 1 -- start start 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.460+0000 7f8b7e56b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b78103340 0x7f8b7819a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.460+0000 7f8b7e56b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b78108170 0x7f8b7819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.460+0000 7f8b7e56b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b78108ac0 0x7f8b7819fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.460+0000 7f8b77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b78103340 0x7f8b7819a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.460+0000 7f8b77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b78103340 0x7f8b7819a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51658/0 (socket says 172.21.15.16:51658) 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.460+0000 7f8b77fff640 1 -- 172.21.15.16:0/1962776261 learned_addr learned my addr 172.21.15.16:0/1962776261 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.460+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1962776261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b78113840 con 0x7f8b78103340 2024-08-28T19:39:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.460+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1962776261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b781136c0 con 0x7f8b78108170 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.460+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1962776261 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8b781139c0 con 0x7f8b78108ac0 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.461+0000 7f8b777fe640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b78108170 0x7f8b7819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.461+0000 7f8b7cb08640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b78108ac0 0x7f8b7819fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.461+0000 7f8b77fff640 1 -- 172.21.15.16:0/1962776261 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b78108ac0 msgr2=0x7f8b7819fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.461+0000 7f8b77fff640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b78108ac0 0x7f8b7819fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.461+0000 7f8b77fff640 1 -- 172.21.15.16:0/1962776261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b78108170 msgr2=0x7f8b7819a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.461+0000 7f8b77fff640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b78108170 0x7f8b7819a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.461+0000 7f8b77fff640 1 -- 172.21.15.16:0/1962776261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b781a02a0 con 0x7f8b78103340 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.461+0000 7f8b777fe640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b78108170 0x7f8b7819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.461+0000 7f8b77fff640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b78103340 0x7f8b7819a330 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f8b6800e9a0 tx=0x7f8b6800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.461+0000 7f8b757fa640 1 -- 172.21.15.16:0/1962776261 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6800cd80 con 0x7f8b78103340 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.462+0000 7f8b757fa640 1 -- 172.21.15.16:0/1962776261 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b680108e0 con 0x7f8b78103340 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.462+0000 7f8b757fa640 1 -- 172.21.15.16:0/1962776261 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b68018960 con 0x7f8b78103340 2024-08-28T19:39:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.462+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1962776261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b781a0590 con 0x7f8b78103340 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.462+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1962776261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b781a6cf0 con 0x7f8b78103340 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.465+0000 7f8b84ae0740 1 Processor -- start 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.465+0000 7f8b84ae0740 1 -- start start 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.465+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 0x55ba92e03310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.465+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a351b0 0x55ba92e03850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.465+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 0x55ba92e03d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.465+0000 7f8b84ae0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ba92d7a680 con 0x55ba92a3d390 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.465+0000 7f8b84ae0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ba92d69c10 con 0x55ba92a5e070 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.465+0000 7f8b84ae0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ba92d6dbc0 con 0x55ba92a351b0 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.466+0000 7f8b7cb08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 0x55ba92e03310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.466+0000 7f8b7cb08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 0x55ba92e03310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57864/0 (socket says 172.21.15.16:57864) 2024-08-28T19:39:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.466+0000 7f8b7cb08640 1 -- 172.21.15.16:0/3354603186 learned_addr learned my addr 172.21.15.16:0/3354603186 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.466+0000 7f8b777fe640 1 --2- 172.21.15.16:0/3354603186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 0x55ba92e03d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.466+0000 7f8b757fa640 1 -- 172.21.15.16:0/1962776261 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8b68010a90 con 0x7f8b78103340 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.466+0000 7f8b777fe640 1 -- 172.21.15.16:0/3354603186 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a351b0 msgr2=0x55ba92e03850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.466+0000 7f8b777fe640 1 --2- 172.21.15.16:0/3354603186 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a351b0 0x55ba92e03850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.466+0000 7f8b777fe640 1 -- 172.21.15.16:0/3354603186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 msgr2=0x55ba92e03310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.466+0000 7f8b777fe640 1 --2- 172.21.15.16:0/3354603186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 0x55ba92e03310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.466+0000 7f8b777fe640 1 -- 172.21.15.16:0/3354603186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba92ac9650 con 0x55ba92a3d390 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.466+0000 7f8b7cb08640 1 --2- 172.21.15.16:0/3354603186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 0x55ba92e03310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.467+0000 7f8b777fe640 1 --2- 172.21.15.16:0/3354603186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 0x55ba92e03d90 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c02a8a0 tx=0x7f8b6c005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e404c11f58366bfc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.467+0000 7f8b5e7fc640 1 -- 172.21.15.16:0/3354603186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c037040 con 0x55ba92a3d390 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.467+0000 7f8b5e7fc640 1 -- 172.21.15.16:0/3354603186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b6c030450 con 0x55ba92a3d390 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.467+0000 7f8b5e7fc640 1 -- 172.21.15.16:0/3354603186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c032ae0 con 0x55ba92a3d390 2024-08-28T19:39:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.467+0000 7f8b757fa640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b600b4d20 0x7f8b600b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.467+0000 7f8b7cb08640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b600b4d20 0x7f8b600b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.467+0000 7f8b757fa640 1 -- 172.21.15.16:0/1962776261 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f8b680e5b80 con 0x7f8b78103340 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.468+0000 7f8b84ae0740 1 -- 172.21.15.16:0/3354603186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 msgr2=0x55ba92e03d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.468+0000 7f8b7cb08640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b600b4d20 0x7f8b600b7230 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009fd0 tx=0x7f8b70009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.468+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/3354603186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 0x55ba92e03d90 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c02a8a0 tx=0x7f8b6c005d10 comp rx=0 tx=0).stop 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.468+0000 7f8b84ae0740 1 -- 172.21.15.16:0/3354603186 shutdown_connections 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.468+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/3354603186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 0x55ba92e03d90 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.468+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/3354603186 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a351b0 0x55ba92e03850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.468+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/3354603186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 0x55ba92e03310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.468+0000 7f8b84ae0740 1 -- 172.21.15.16:0/3354603186 >> 172.21.15.16:0/3354603186 conn(0x55ba92a52460 msgr2=0x55ba92e01200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.468+0000 7f8b84ae0740 1 -- 172.21.15.16:0/3354603186 shutdown_connections 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.468+0000 7f8b84ae0740 1 -- 172.21.15.16:0/3354603186 wait complete. 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.469+0000 7f8b84ae0740 1 -- start start 2024-08-28T19:39:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.469+0000 7f8b84ae0740 1 Processor -- start 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.470+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 0x55ba92e0aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.470+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a3d390 0x55ba92e0af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.470+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a5e070 0x55ba92e23530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.470+0000 7f8b77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 0x55ba92e0aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.470+0000 7f8b84ae0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ba92bcc700 con 0x55ba92a351b0 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.470+0000 7f8b77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 0x55ba92e0aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51684/0 (socket says 172.21.15.16:51684) 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.470+0000 7f8b77fff640 1 -- 172.21.15.16:0/1370012450 learned_addr learned my addr 172.21.15.16:0/1370012450 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.470+0000 7f8b777fe640 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a3d390 0x55ba92e0af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.470+0000 7f8b7cb08640 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a5e070 0x55ba92e23530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.470+0000 7f8b84ae0740 1 -- 172.21.15.16:0/1370012450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ba92cf0750 con 0x55ba92a3d390 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.470+0000 7f8b84ae0740 1 -- 172.21.15.16:0/1370012450 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ba92e1a280 con 0x55ba92a5e070 2024-08-28T19:39:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.471+0000 7f8b84ae0740 10 client.0 _ll_register_callbacks cb 0x55ba92d1fe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.472+0000 7f8b77fff640 1 -- 172.21.15.16:0/1370012450 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a5e070 msgr2=0x55ba92e23530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.472+0000 7f8b77fff640 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a5e070 0x55ba92e23530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.472+0000 7f8b77fff640 1 -- 172.21.15.16:0/1370012450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a3d390 msgr2=0x55ba92e0af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.472+0000 7f8b77fff640 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a3d390 0x55ba92e0af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.472+0000 7f8b77fff640 1 -- 172.21.15.16:0/1370012450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba92d29900 con 0x55ba92a351b0 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.472+0000 7f8b77fff640 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 0x55ba92e0aa50 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f8b680e3500 tx=0x7f8b680ab610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72d8eb1a332cc7b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.472+0000 7f8b577fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.472+0000 7f8b577fe640 1 -- 172.21.15.16:0/1370012450 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b680abea0 con 0x55ba92a351b0 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.472+0000 7f8b577fe640 1 -- 172.21.15.16:0/1370012450 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f8b680a8490 con 0x55ba92a351b0 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.472+0000 7f8b84ae0740 1 -- 172.21.15.16:0/1370012450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b6c030450 con 0x55ba92a351b0 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.473+0000 7f8b577fe640 1 -- 172.21.15.16:0/1370012450 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f8b680b0710 con 0x55ba92a351b0 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.473+0000 7f8b84ae0740 10 client.19851 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.473+0000 7f8b84ae0740 1 -- 172.21.15.16:0/1370012450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x55ba92e25ff0 con 0x55ba92a351b0 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.473+0000 7f8b577fe640 1 -- 172.21.15.16:0/1370012450 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b6c030450 con 0x55ba92a351b0 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.473+0000 7f8b577fe640 1 -- 172.21.15.16:0/1370012450 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7f8b680930d0 con 0x55ba92a351b0 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.473+0000 7f8b84ae0740 10 client.19851 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.473+0000 7f8b84ae0740 10 client.19851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.474+0000 7f8b84ae0740 20 client.19851 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.474+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ba92a1c9f0 0x55ba92e294c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.474+0000 7f8b84ae0740 4 client.19851 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.474+0000 7f8b84ae0740 1 -- 172.21.15.16:0/1370012450 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba92d7c5d0 con 0x55ba92a1c9f0 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.474+0000 7f8b777fe640 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ba92a1c9f0 0x55ba92e294c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.475+0000 7f8b777fe640 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ba92a1c9f0 0x55ba92e294c0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=672e817f1fb71dcb server_cookie=33ad0788fae000e8 in_seq=0 out_seq=0 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.475+0000 7f8b577fe640 10 client.19851 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.476+0000 7f8b577fe640 1 -- 172.21.15.16:0/1370012450 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ba92d7c5d0 con 0x55ba92a1c9f0 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.476+0000 7f8b577fe640 10 client.19851 handle_command_reply: tid=0 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.476+0000 7f8b84ae0740 1 client.19851 shutdown 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.476+0000 7f8b84ae0740 1 -- 172.21.15.16:0/1370012450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 msgr2=0x55ba92e0aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.476+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 0x55ba92e0aa50 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f8b680e3500 tx=0x7f8b680ab610 comp rx=0 tx=0).stop 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.477+0000 7f8b84ae0740 1 -- 172.21.15.16:0/1370012450 shutdown_connections 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.477+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ba92a1c9f0 0x55ba92e294c0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.477+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a5e070 0x55ba92e23530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.477+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a3d390 0x55ba92e0af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.477+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/1370012450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 0x55ba92e0aa50 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.477+0000 7f8b84ae0740 1 -- 172.21.15.16:0/1370012450 >> 172.21.15.16:0/1370012450 conn(0x55ba92a52460 msgr2=0x55ba92e04d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.477+0000 7f8b577fe640 0 client.19851 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.477+0000 7f8b84ae0740 1 -- 172.21.15.16:0/1370012450 shutdown_connections 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.477+0000 7f8b84ae0740 1 -- 172.21.15.16:0/1370012450 wait complete. 2024-08-28T19:39:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.477+0000 7f8b84ae0740 20 client.19851 trim_cache size 0 max 16384 2024-08-28T19:39:45.486 INFO:tasks.workunit.client.0.smithi016.stdout: 10 126957 23.78 MB/sec execute 201 sec latency 141.781 ms 2024-08-28T19:39:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.496+0000 7f8b84ae0740 1 Processor -- start 2024-08-28T19:39:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.496+0000 7f8b84ae0740 1 -- start start 2024-08-28T19:39:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.497+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a351b0 0x55ba92e23470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.497+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 0x55ba92e0a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.497+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 0x55ba92e0ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.497+0000 7f8b84ae0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b781aeb30 con 0x55ba92a3d390 2024-08-28T19:39:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.497+0000 7f8b84ae0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b781b0850 con 0x55ba92a5e070 2024-08-28T19:39:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.497+0000 7f8b84ae0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8b781af290 con 0x55ba92a351b0 2024-08-28T19:39:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.497+0000 7f8b77fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a351b0 0x55ba92e23470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.497+0000 7f8b77fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a351b0 0x55ba92e23470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:58928/0 (socket says 172.21.15.16:58928) 2024-08-28T19:39:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.497+0000 7f8b77fff640 1 -- 172.21.15.16:0/603020994 learned_addr learned my addr 172.21.15.16:0/603020994 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.497+0000 7f8b777fe640 1 --2- 172.21.15.16:0/603020994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 0x55ba92e0a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.497+0000 7f8b7cb08640 1 --2- 172.21.15.16:0/603020994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 0x55ba92e0ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.498+0000 7f8b777fe640 1 -- 172.21.15.16:0/603020994 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a351b0 msgr2=0x55ba92e23470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.498+0000 7f8b777fe640 1 --2- 172.21.15.16:0/603020994 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a351b0 0x55ba92e23470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.498+0000 7f8b777fe640 1 -- 172.21.15.16:0/603020994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 msgr2=0x55ba92e0ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.498+0000 7f8b777fe640 1 --2- 172.21.15.16:0/603020994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 0x55ba92e0ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.498+0000 7f8b777fe640 1 -- 172.21.15.16:0/603020994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba92e29a20 con 0x55ba92a3d390 2024-08-28T19:39:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.498+0000 7f8b7cb08640 1 --2- 172.21.15.16:0/603020994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 0x55ba92e0ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.498+0000 7f8b777fe640 1 --2- 172.21.15.16:0/603020994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 0x55ba92e0a800 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c02a970 tx=0x7f8b6c005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e270adff1ea6f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.499+0000 7f8b5dffb640 1 -- 172.21.15.16:0/603020994 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c037070 con 0x55ba92a3d390 2024-08-28T19:39:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.499+0000 7f8b5dffb640 1 -- 172.21.15.16:0/603020994 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b6c030450 con 0x55ba92a3d390 2024-08-28T19:39:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.499+0000 7f8b5dffb640 1 -- 172.21.15.16:0/603020994 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c0327f0 con 0x55ba92a3d390 2024-08-28T19:39:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.500+0000 7f8b84ae0740 1 -- 172.21.15.16:0/603020994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 msgr2=0x55ba92e0a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.501+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/603020994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 0x55ba92e0a800 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c02a970 tx=0x7f8b6c005d10 comp rx=0 tx=0).stop 2024-08-28T19:39:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.502+0000 7f8b84ae0740 1 -- 172.21.15.16:0/603020994 shutdown_connections 2024-08-28T19:39:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.502+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/603020994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a5e070 0x55ba92e0ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.502+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/603020994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a3d390 0x55ba92e0a800 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.502+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/603020994 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a351b0 0x55ba92e23470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.502+0000 7f8b84ae0740 1 -- 172.21.15.16:0/603020994 >> 172.21.15.16:0/603020994 conn(0x55ba92a52460 msgr2=0x55ba92e18c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.503+0000 7f8b84ae0740 1 -- 172.21.15.16:0/603020994 shutdown_connections 2024-08-28T19:39:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.503+0000 7f8b84ae0740 1 -- 172.21.15.16:0/603020994 wait complete. 2024-08-28T19:39:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.503+0000 7f8b84ae0740 1 -- start start 2024-08-28T19:39:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.504+0000 7f8b84ae0740 1 Processor -- start 2024-08-28T19:39:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.504+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 0x55ba92e1b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.504+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a3d390 0x55ba92e1bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.504+0000 7f8b84ae0740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a5e070 0x55ba92e1c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b84ae0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b781b00c0 con 0x55ba92a351b0 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b84ae0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b781af960 con 0x55ba92a3d390 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b84ae0740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8b781acb30 con 0x55ba92a5e070 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b84ae0740 10 client.0 _ll_register_callbacks cb 0x7f8b6800ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a3d390 0x55ba92e1bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a3d390 0x55ba92e1bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57886/0 (socket says 172.21.15.16:57886) 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b777fe640 1 -- 172.21.15.16:0/460593723 learned_addr learned my addr 172.21.15.16:0/460593723 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b7cb08640 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a5e070 0x55ba92e1c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b77fff640 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 0x55ba92e1b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b77fff640 1 -- 172.21.15.16:0/460593723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a5e070 msgr2=0x55ba92e1c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b77fff640 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a5e070 0x55ba92e1c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.505+0000 7f8b77fff640 1 -- 172.21.15.16:0/460593723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a3d390 msgr2=0x55ba92e1bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:45.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.506+0000 7f8b77fff640 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a3d390 0x55ba92e1bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.506+0000 7f8b77fff640 1 -- 172.21.15.16:0/460593723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba92e930f0 con 0x55ba92a351b0 2024-08-28T19:39:45.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.506+0000 7f8b77fff640 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 0x55ba92e1b6b0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f8b680abe30 tx=0x7f8b6800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef86618d00d121e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:45.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.506+0000 7f8b577fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.506+0000 7f8b577fe640 1 -- 172.21.15.16:0/460593723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b680a7d60 con 0x55ba92a351b0 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.506+0000 7f8b577fe640 1 -- 172.21.15.16:0/460593723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f8b680a8490 con 0x55ba92a351b0 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.506+0000 7f8b84ae0740 1 -- 172.21.15.16:0/460593723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b6c030450 con 0x55ba92a351b0 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.507+0000 7f8b577fe640 1 -- 172.21.15.16:0/460593723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f8b680ab9b0 con 0x55ba92a351b0 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.507+0000 7f8b84ae0740 10 client.19857 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.507+0000 7f8b84ae0740 1 -- 172.21.15.16:0/460593723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x55ba92e0da50 con 0x55ba92a351b0 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.508+0000 7f8b577fe640 1 -- 172.21.15.16:0/460593723 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b680abea0 con 0x55ba92a351b0 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.508+0000 7f8b577fe640 1 -- 172.21.15.16:0/460593723 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7f8b68018b00 con 0x55ba92a351b0 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.508+0000 7f8b84ae0740 10 client.19857 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.508+0000 7f8b84ae0740 10 client.19857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.508+0000 7f8b84ae0740 20 client.19857 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.508+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ba929f1da0 0x55ba92e10100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:45.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.508+0000 7f8b84ae0740 4 client.19857 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:39:45.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.508+0000 7f8b84ae0740 1 -- 172.21.15.16:0/460593723 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f8b6c030450 con 0x55ba929f1da0 2024-08-28T19:39:45.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.509+0000 7f8b777fe640 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ba929f1da0 0x55ba92e10100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:45.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.509+0000 7f8b577fe640 10 client.19857 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:45.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.509+0000 7f8b777fe640 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ba929f1da0 0x55ba92e10100 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bff672c519fb0fb0 server_cookie=596e08c7a84e4425 in_seq=0 out_seq=0 2024-08-28T19:39:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b577fe640 1 -- 172.21.15.16:0/460593723 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f8b6c030450 con 0x55ba929f1da0 2024-08-28T19:39:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b577fe640 10 client.19857 handle_command_reply: tid=0 2024-08-28T19:39:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b84ae0740 1 client.19857 shutdown 2024-08-28T19:39:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b84ae0740 1 -- 172.21.15.16:0/460593723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 msgr2=0x55ba92e1b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 0x55ba92e1b6b0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f8b680abe30 tx=0x7f8b6800cf20 comp rx=0 tx=0).stop 2024-08-28T19:39:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b84ae0740 1 -- 172.21.15.16:0/460593723 shutdown_connections 2024-08-28T19:39:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55ba929f1da0 0x55ba92e10100 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ba92a5e070 0x55ba92e1c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba92a3d390 0x55ba92e1bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b84ae0740 1 --2- 172.21.15.16:0/460593723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ba92a351b0 0x55ba92e1b6b0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b84ae0740 1 -- 172.21.15.16:0/460593723 >> 172.21.15.16:0/460593723 conn(0x55ba92a52460 msgr2=0x55ba92e0a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.512+0000 7f8b577fe640 0 client.19857 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.513+0000 7f8b84ae0740 1 -- 172.21.15.16:0/460593723 shutdown_connections 2024-08-28T19:39:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.513+0000 7f8b84ae0740 1 -- 172.21.15.16:0/460593723 wait complete. 2024-08-28T19:39:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.513+0000 7f8b84ae0740 20 client.19857 trim_cache size 0 max 16384 2024-08-28T19:39:45.515 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:39:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1962776261 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b600b4d20 msgr2=0x7f8b600b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b600b4d20 0x7f8b600b7230 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009fd0 tx=0x7f8b70009500 comp rx=0 tx=0).stop 2024-08-28T19:39:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1962776261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b78103340 msgr2=0x7f8b7819a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b78103340 0x7f8b7819a330 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f8b6800e9a0 tx=0x7f8b6800ee60 comp rx=0 tx=0).stop 2024-08-28T19:39:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1962776261 shutdown_connections 2024-08-28T19:39:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b78108ac0 0x7f8b7819fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b600b4d20 0x7f8b600b7230 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b78108170 0x7f8b7819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 --2- 172.21.15.16:0/1962776261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b78103340 0x7f8b7819a330 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1962776261 >> 172.21.15.16:0/1962776261 conn(0x7f8b780775a0 msgr2=0x7f8b781004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1962776261 shutdown_connections 2024-08-28T19:39:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:45.514+0000 7f8b7e56b640 1 -- 172.21.15.16:0/1962776261 wait complete. 2024-08-28T19:39:45.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:45 smithi016 ceph-mon[18556]: pgmap v924: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 16 MiB/s wr, 718 op/s 2024-08-28T19:39:45.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:45 smithi028 ceph-mon[21769]: pgmap v924: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 16 MiB/s wr, 718 op/s 2024-08-28T19:39:45.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:45 smithi115 ceph-mon[22184]: pgmap v924: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 16 MiB/s wr, 718 op/s 2024-08-28T19:39:46.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 127675 23.83 MB/sec execute 202 sec latency 110.316 ms 2024-08-28T19:39:47.224 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:39:47.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:39:47.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 128413 23.88 MB/sec execute 203 sec latency 91.217 ms 2024-08-28T19:39:47.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:47 smithi016 ceph-mon[18556]: pgmap v925: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 MiB/s wr, 790 op/s 2024-08-28T19:39:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:47.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:47 smithi028 ceph-mon[21769]: pgmap v925: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 MiB/s wr, 790 op/s 2024-08-28T19:39:47.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:47 smithi115 ceph-mon[22184]: pgmap v925: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 MiB/s wr, 790 op/s 2024-08-28T19:39:48.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.052+0000 7fed0a590640 1 -- 172.21.15.16:0/1390264269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed04102880 msgr2=0x7fed04102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:48.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.052+0000 7fed0a590640 1 --2- 172.21.15.16:0/1390264269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed04102880 0x7fed04102c90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fecf400d910 tx=0x7fecf402c370 comp rx=0 tx=0).stop 2024-08-28T19:39:48.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.052+0000 7fed0a590640 1 -- 172.21.15.16:0/1390264269 shutdown_connections 2024-08-28T19:39:48.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.052+0000 7fed0a590640 1 --2- 172.21.15.16:0/1390264269 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed04108ac0 0x7fed0410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.052+0000 7fed0a590640 1 --2- 172.21.15.16:0/1390264269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed04108170 0x7fed04108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.052+0000 7fed0a590640 1 --2- 172.21.15.16:0/1390264269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed04102880 0x7fed04102c90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.052+0000 7fed0a590640 1 -- 172.21.15.16:0/1390264269 >> 172.21.15.16:0/1390264269 conn(0x7fed04079550 msgr2=0x7fed04079960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:48.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.052+0000 7fed0a590640 1 -- 172.21.15.16:0/1390264269 shutdown_connections 2024-08-28T19:39:48.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.052+0000 7fed0a590640 1 -- 172.21.15.16:0/1390264269 wait complete. 2024-08-28T19:39:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.053+0000 7fed0a590640 1 Processor -- start 2024-08-28T19:39:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.053+0000 7fed0a590640 1 -- start start 2024-08-28T19:39:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.053+0000 7fed0a590640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed04102880 0x7fed041980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.053+0000 7fed0a590640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed04108170 0x7fed04198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.053+0000 7fed0a590640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed04108ac0 0x7fed0419d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.053+0000 7fed0a590640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed04111680 con 0x7fed04102880 2024-08-28T19:39:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.053+0000 7fed0a590640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed04111500 con 0x7fed04108ac0 2024-08-28T19:39:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.053+0000 7fed0a590640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fed04111800 con 0x7fed04108170 2024-08-28T19:39:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.053+0000 7fed03fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed04102880 0x7fed041980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.053+0000 7fed03fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed04102880 0x7fed041980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51708/0 (socket says 172.21.15.16:51708) 2024-08-28T19:39:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.053+0000 7fed03fff640 1 -- 172.21.15.16:0/864956388 learned_addr learned my addr 172.21.15.16:0/864956388 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.054+0000 7fed08b2d640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed04108ac0 0x7fed0419d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.054+0000 7fed037fe640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed04108170 0x7fed04198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.054+0000 7fed03fff640 1 -- 172.21.15.16:0/864956388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed04108170 msgr2=0x7fed04198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.054+0000 7fed03fff640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed04108170 0x7fed04198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.054+0000 7fed03fff640 1 -- 172.21.15.16:0/864956388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed04108ac0 msgr2=0x7fed0419d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.054+0000 7fed03fff640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed04108ac0 0x7fed0419d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.054+0000 7fed03fff640 1 -- 172.21.15.16:0/864956388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0419e020 con 0x7fed04102880 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.054+0000 7fed08b2d640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed04108ac0 0x7fed0419d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.054+0000 7fed03fff640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed04102880 0x7fed041980b0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fecf4005ab0 tx=0x7fecf4004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.055+0000 7fed017fa640 1 -- 172.21.15.16:0/864956388 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf4004030 con 0x7fed04102880 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.055+0000 7fed017fa640 1 -- 172.21.15.16:0/864956388 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf40053a0 con 0x7fed04102880 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.055+0000 7fed0a590640 1 -- 172.21.15.16:0/864956388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0419e2b0 con 0x7fed04102880 2024-08-28T19:39:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.055+0000 7fed017fa640 1 -- 172.21.15.16:0/864956388 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf4003850 con 0x7fed04102880 2024-08-28T19:39:48.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.055+0000 7fed0a590640 1 -- 172.21.15.16:0/864956388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed041a4a70 con 0x7fed04102880 2024-08-28T19:39:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.058+0000 7fed10b05740 1 Processor -- start 2024-08-28T19:39:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.058+0000 7fed10b05740 1 -- start start 2024-08-28T19:39:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.058+0000 7fed017fa640 1 -- 172.21.15.16:0/864956388 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fecf403c070 con 0x7fed04102880 2024-08-28T19:39:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.058+0000 7fed10b05740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 0x561e212b6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.058+0000 7fed10b05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212b6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.058+0000 7fed10b05740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20eef670 0x561e212b7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.058+0000 7fed10b05740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561e2122c5f0 con 0x561e20eef670 2024-08-28T19:39:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.058+0000 7fed10b05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e2121bb80 con 0x561e20ee7490 2024-08-28T19:39:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.058+0000 7fed10b05740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561e2121fb30 con 0x561e20f10350 2024-08-28T19:39:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.058+0000 7fed03fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212b6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed08b2d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 0x561e212b6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed03fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212b6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57910/0 (socket says 172.21.15.16:57910) 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed03fff640 1 -- 172.21.15.16:0/1266066368 learned_addr learned my addr 172.21.15.16:0/1266066368 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed037fe640 1 --2- 172.21.15.16:0/1266066368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20eef670 0x561e212b7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed03fff640 1 -- 172.21.15.16:0/1266066368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 msgr2=0x561e212b6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed03fff640 1 --2- 172.21.15.16:0/1266066368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 0x561e212b6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed03fff640 1 -- 172.21.15.16:0/1266066368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20eef670 msgr2=0x561e212b7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed03fff640 1 --2- 172.21.15.16:0/1266066368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20eef670 0x561e212b7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed03fff640 1 -- 172.21.15.16:0/1266066368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e20f7b930 con 0x561e20ee7490 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed017fa640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fecec0b4ca0 0x7fecec0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed03fff640 1 --2- 172.21.15.16:0/1266066368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212b6cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fecf4104a30 tx=0x7fecf410b640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccf29b51639efcc6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.059+0000 7fed08b2d640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fecec0b4ca0 0x7fecec0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.060+0000 7fed017fa640 1 -- 172.21.15.16:0/864956388 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fecf4107710 con 0x7fed04102880 2024-08-28T19:39:48.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.060+0000 7fecea7fc640 1 -- 172.21.15.16:0/1266066368 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf40c98f0 con 0x561e20ee7490 2024-08-28T19:39:48.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.060+0000 7fecea7fc640 1 -- 172.21.15.16:0/1266066368 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf410be40 con 0x561e20ee7490 2024-08-28T19:39:48.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.060+0000 7fecea7fc640 1 -- 172.21.15.16:0/1266066368 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf40041d0 con 0x561e20ee7490 2024-08-28T19:39:48.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.061+0000 7fed10b05740 1 -- 172.21.15.16:0/1266066368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 msgr2=0x561e212b6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:48.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.061+0000 7fed08b2d640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fecec0b4ca0 0x7fecec0b71b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009fd0 tx=0x7fecfc009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:48.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.061+0000 7fed10b05740 1 --2- 172.21.15.16:0/1266066368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212b6cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fecf4104a30 tx=0x7fecf410b640 comp rx=0 tx=0).stop 2024-08-28T19:39:48.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.061+0000 7fed10b05740 1 -- 172.21.15.16:0/1266066368 shutdown_connections 2024-08-28T19:39:48.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.061+0000 7fed10b05740 1 --2- 172.21.15.16:0/1266066368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20eef670 0x561e212b7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.061+0000 7fed10b05740 1 --2- 172.21.15.16:0/1266066368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212b6cc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.061+0000 7fed10b05740 1 --2- 172.21.15.16:0/1266066368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 0x561e212b6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.061+0000 7fed10b05740 1 -- 172.21.15.16:0/1266066368 >> 172.21.15.16:0/1266066368 conn(0x561e20f04740 msgr2=0x561e212b4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:48.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.061+0000 7fed10b05740 1 -- 172.21.15.16:0/1266066368 shutdown_connections 2024-08-28T19:39:48.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.061+0000 7fed10b05740 1 -- 172.21.15.16:0/1266066368 wait complete. 2024-08-28T19:39:48.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.061+0000 7fed10b05740 1 -- start start 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.062+0000 7fed10b05740 1 Processor -- start 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.062+0000 7fed10b05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212c1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.062+0000 7fed10b05740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20eef670 0x561e212c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.062+0000 7fed10b05740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20f10350 0x561e212d2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.062+0000 7fed10b05740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561e2107e890 con 0x561e20f10350 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.062+0000 7fed10b05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e211a2790 con 0x561e20ee7490 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.062+0000 7fed10b05740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561e2121bb80 con 0x561e20eef670 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.062+0000 7fed03fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212c1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.062+0000 7fed03fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212c1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57926/0 (socket says 172.21.15.16:57926) 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.062+0000 7fed03fff640 1 -- 172.21.15.16:0/718249530 learned_addr learned my addr 172.21.15.16:0/718249530 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fed037fe640 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20eef670 0x561e212c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fed08b2d640 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20f10350 0x561e212d2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fed10b05740 10 client.0 _ll_register_callbacks cb 0x561e20f27ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fed03fff640 1 -- 172.21.15.16:0/718249530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20eef670 msgr2=0x561e212c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fed03fff640 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20eef670 0x561e212c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fed03fff640 1 -- 172.21.15.16:0/718249530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20f10350 msgr2=0x561e212d2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fed03fff640 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20f10350 0x561e212d2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fed03fff640 1 -- 172.21.15.16:0/718249530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e211d9f30 con 0x561e20ee7490 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fed08b2d640 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20f10350 0x561e212d2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fed03fff640 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212c1e60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fecf4105090 tx=0x7fecf4104a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcb810363d836206 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fece37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fece37fe640 1 -- 172.21.15.16:0/718249530 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf40cd1a0 con 0x561e20ee7490 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fece37fe640 1 -- 172.21.15.16:0/718249530 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fecf40cdc90 con 0x561e20ee7490 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.063+0000 7fed10b05740 1 -- 172.21.15.16:0/718249530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561e2122e540 con 0x561e20ee7490 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.064+0000 7fece37fe640 1 -- 172.21.15.16:0/718249530 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fecf40cd380 con 0x561e20ee7490 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.064+0000 7fed10b05740 10 client.29315 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.064+0000 7fed10b05740 1 -- 172.21.15.16:0/718249530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x561e211d9f30 con 0x561e20ee7490 2024-08-28T19:39:48.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.064+0000 7fece37fe640 1 -- 172.21.15.16:0/718249530 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561e2122e540 con 0x561e20ee7490 2024-08-28T19:39:48.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.064+0000 7fece37fe640 1 -- 172.21.15.16:0/718249530 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7fecf400c040 con 0x561e20ee7490 2024-08-28T19:39:48.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.064+0000 7fed10b05740 10 client.29315 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:48.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.064+0000 7fed10b05740 10 client.29315 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:39:48.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.064+0000 7fed10b05740 20 client.29315 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:48.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.064+0000 7fed10b05740 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561e20ececd0 0x561e212d7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.064+0000 7fed10b05740 4 client.29315 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:39:48.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.064+0000 7fed10b05740 1 -- 172.21.15.16:0/718249530 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e21229fe0 con 0x561e20ececd0 2024-08-28T19:39:48.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.065+0000 7fed037fe640 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561e20ececd0 0x561e212d7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.065+0000 7fece37fe640 10 client.29315 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.065+0000 7fed037fe640 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561e20ececd0 0x561e212d7e00 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5122c80c406e57f server_cookie=882605e1d0026ee1 in_seq=0 out_seq=0 2024-08-28T19:39:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fece37fe640 1 -- 172.21.15.16:0/718249530 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561e21229fe0 con 0x561e20ececd0 2024-08-28T19:39:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fece37fe640 10 client.29315 handle_command_reply: tid=0 2024-08-28T19:39:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fed10b05740 1 client.29315 shutdown 2024-08-28T19:39:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fed10b05740 1 -- 172.21.15.16:0/718249530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 msgr2=0x561e212c1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fed10b05740 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212c1e60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fecf4105090 tx=0x7fecf4104a30 comp rx=0 tx=0).stop 2024-08-28T19:39:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fed10b05740 1 -- 172.21.15.16:0/718249530 shutdown_connections 2024-08-28T19:39:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fed10b05740 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20f10350 0x561e212d2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fed10b05740 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561e20ececd0 0x561e212d7e00 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fed10b05740 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20eef670 0x561e212c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fed10b05740 1 --2- 172.21.15.16:0/718249530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20ee7490 0x561e212c1e60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fed10b05740 1 -- 172.21.15.16:0/718249530 >> 172.21.15.16:0/718249530 conn(0x561e20f04740 msgr2=0x561e212b9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:48.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fece37fe640 0 client.29315 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:48.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fed10b05740 1 -- 172.21.15.16:0/718249530 shutdown_connections 2024-08-28T19:39:48.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.066+0000 7fed10b05740 1 -- 172.21.15.16:0/718249530 wait complete. 2024-08-28T19:39:48.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.067+0000 7fed10b05740 20 client.29315 trim_cache size 0 max 16384 2024-08-28T19:39:48.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed10b05740 1 Processor -- start 2024-08-28T19:39:48.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed10b05740 1 -- start start 2024-08-28T19:39:48.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed10b05740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed10b05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20eef670 0x561e212d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed10b05740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 0x561e212bf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed10b05740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed041aea50 con 0x561e20ee7490 2024-08-28T19:39:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed10b05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed041b0770 con 0x561e20eef670 2024-08-28T19:39:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed10b05740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fed041af1b0 con 0x561e20f10350 2024-08-28T19:39:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed03fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed03fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212c3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51746/0 (socket says 172.21.15.16:51746) 2024-08-28T19:39:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed03fff640 1 -- 172.21.15.16:0/3670504824 learned_addr learned my addr 172.21.15.16:0/3670504824 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.090+0000 7fed08b2d640 1 --2- 172.21.15.16:0/3670504824 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 0x561e212bf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.091+0000 7fed03fff640 1 -- 172.21.15.16:0/3670504824 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 msgr2=0x561e212bf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.091+0000 7fed03fff640 1 --2- 172.21.15.16:0/3670504824 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 0x561e212bf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.091+0000 7fed03fff640 1 -- 172.21.15.16:0/3670504824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20eef670 msgr2=0x561e212d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.091+0000 7fed037fe640 1 --2- 172.21.15.16:0/3670504824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20eef670 0x561e212d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.091+0000 7fed03fff640 1 --2- 172.21.15.16:0/3670504824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20eef670 0x561e212d21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.091+0000 7fed03fff640 1 -- 172.21.15.16:0/3670504824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e212c0dc0 con 0x561e20ee7490 2024-08-28T19:39:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.091+0000 7fed03fff640 1 --2- 172.21.15.16:0/3670504824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212c3a90 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fecf4000c00 tx=0x7fecf410b0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4fdd6b2167781a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.091+0000 7fece9ffb640 1 -- 172.21.15.16:0/3670504824 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf40cd800 con 0x561e20ee7490 2024-08-28T19:39:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.092+0000 7fece9ffb640 1 -- 172.21.15.16:0/3670504824 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf4030070 con 0x561e20ee7490 2024-08-28T19:39:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.092+0000 7fece9ffb640 1 -- 172.21.15.16:0/3670504824 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf40cd1a0 con 0x561e20ee7490 2024-08-28T19:39:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.093+0000 7fed10b05740 1 -- 172.21.15.16:0/3670504824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 msgr2=0x561e212c3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.093+0000 7fed10b05740 1 --2- 172.21.15.16:0/3670504824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212c3a90 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fecf4000c00 tx=0x7fecf410b0e0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.093+0000 7fed10b05740 1 -- 172.21.15.16:0/3670504824 shutdown_connections 2024-08-28T19:39:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.093+0000 7fed10b05740 1 --2- 172.21.15.16:0/3670504824 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 0x561e212bf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.093+0000 7fed10b05740 1 --2- 172.21.15.16:0/3670504824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20eef670 0x561e212d21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.093+0000 7fed10b05740 1 --2- 172.21.15.16:0/3670504824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212c3a90 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.093+0000 7fed10b05740 1 -- 172.21.15.16:0/3670504824 >> 172.21.15.16:0/3670504824 conn(0x561e20f04740 msgr2=0x561e212c3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.093+0000 7fed10b05740 1 -- 172.21.15.16:0/3670504824 shutdown_connections 2024-08-28T19:39:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.093+0000 7fed10b05740 1 -- 172.21.15.16:0/3670504824 wait complete. 2024-08-28T19:39:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.093+0000 7fed10b05740 1 -- start start 2024-08-28T19:39:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed10b05740 1 Processor -- start 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed10b05740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed10b05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20eef670 0x561e212caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed10b05740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 0x561e213416e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed10b05740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed041affe0 con 0x561e20ee7490 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed10b05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed041af880 con 0x561e20eef670 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed10b05740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fed041aca50 con 0x561e20f10350 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed10b05740 10 client.0 _ll_register_callbacks cb 0x7fecf40d1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed03fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed03fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212ca5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51762/0 (socket says 172.21.15.16:51762) 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed03fff640 1 -- 172.21.15.16:0/1767086768 learned_addr learned my addr 172.21.15.16:0/1767086768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.094+0000 7fed037fe640 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20eef670 0x561e212caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.095+0000 7fed03fff640 1 -- 172.21.15.16:0/1767086768 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 msgr2=0x561e213416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.095+0000 7fed03fff640 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 0x561e213416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.095+0000 7fed03fff640 1 -- 172.21.15.16:0/1767086768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20eef670 msgr2=0x561e212caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.095+0000 7fed03fff640 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20eef670 0x561e212caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.095+0000 7fed03fff640 1 -- 172.21.15.16:0/1767086768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e212cb3c0 con 0x561e20ee7490 2024-08-28T19:39:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.095+0000 7fed03fff640 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212ca5b0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fecf410b4d0 tx=0x7fecf410bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99949f1ff265c854 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.095+0000 7fece37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.095+0000 7fece37fe640 1 -- 172.21.15.16:0/1767086768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf40c98f0 con 0x561e20ee7490 2024-08-28T19:39:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fece37fe640 1 -- 172.21.15.16:0/1767086768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7fecf40d6ad0 con 0x561e20ee7490 2024-08-28T19:39:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fed10b05740 1 -- 172.21.15.16:0/1767086768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecf4030070 con 0x561e20ee7490 2024-08-28T19:39:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fece37fe640 1 -- 172.21.15.16:0/1767086768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fecf40cd1a0 con 0x561e20ee7490 2024-08-28T19:39:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fed10b05740 10 client.19875 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fed10b05740 1 -- 172.21.15.16:0/1767086768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x561e212cb3c0 con 0x561e20ee7490 2024-08-28T19:39:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fece37fe640 1 -- 172.21.15.16:0/1767086768 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf4030070 con 0x561e20ee7490 2024-08-28T19:39:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fece37fe640 1 -- 172.21.15.16:0/1767086768 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7fecf400c050 con 0x561e20ee7490 2024-08-28T19:39:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fed10b05740 10 client.19875 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fed10b05740 10 client.19875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:39:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fed10b05740 20 client.19875 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fed10b05740 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561e20ea4080 0x561e21345000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fed10b05740 4 client.19875 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:39:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.096+0000 7fed10b05740 1 -- 172.21.15.16:0/1767086768 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561e21227720 con 0x561e20ea4080 2024-08-28T19:39:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.097+0000 7fed037fe640 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561e20ea4080 0x561e21345000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.097+0000 7fed037fe640 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561e20ea4080 0x561e21345000 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=983ade0420a2e27f server_cookie=71c3bdbc8cae0aed in_seq=0 out_seq=0 2024-08-28T19:39:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.097+0000 7fece37fe640 10 client.19875 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.100+0000 7fece37fe640 1 -- 172.21.15.16:0/1767086768 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561e21227720 con 0x561e20ea4080 2024-08-28T19:39:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.100+0000 7fece37fe640 10 client.19875 handle_command_reply: tid=0 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.100+0000 7fed10b05740 1 client.19875 shutdown 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.100+0000 7fed10b05740 1 -- 172.21.15.16:0/1767086768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 msgr2=0x561e212ca5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.100+0000 7fed10b05740 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212ca5b0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fecf410b4d0 tx=0x7fecf410bb40 comp rx=0 tx=0).stop 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.100+0000 7fed10b05740 1 -- 172.21.15.16:0/1767086768 shutdown_connections 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.100+0000 7fed10b05740 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561e20ea4080 0x561e21345000 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.100+0000 7fed10b05740 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561e20f10350 0x561e213416e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.101+0000 7fece37fe640 0 client.19875 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.101+0000 7fed10b05740 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e20eef670 0x561e212caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.101+0000 7fed10b05740 1 --2- 172.21.15.16:0/1767086768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e20ee7490 0x561e212ca5b0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.101+0000 7fed10b05740 1 -- 172.21.15.16:0/1767086768 >> 172.21.15.16:0/1767086768 conn(0x561e20f04740 msgr2=0x561e212c3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.101+0000 7fed10b05740 1 -- 172.21.15.16:0/1767086768 shutdown_connections 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.101+0000 7fed10b05740 1 -- 172.21.15.16:0/1767086768 wait complete. 2024-08-28T19:39:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.101+0000 7fed10b05740 20 client.19875 trim_cache size 0 max 16384 2024-08-28T19:39:48.104 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:39:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:39:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "8de90bf3-5743-4711-98c2-a56ca0757291", 2024-08-28T19:39:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:39:48.104 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:39:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.101+0000 7fed0a590640 1 -- 172.21.15.16:0/864956388 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fecec0b4ca0 msgr2=0x7fecec0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.101+0000 7fed0a590640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fecec0b4ca0 0x7fecec0b71b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009fd0 tx=0x7fecfc009500 comp rx=0 tx=0).stop 2024-08-28T19:39:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.102+0000 7fed0a590640 1 -- 172.21.15.16:0/864956388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed04102880 msgr2=0x7fed041980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.102+0000 7fed0a590640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed04102880 0x7fed041980b0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fecf4005ab0 tx=0x7fecf4004970 comp rx=0 tx=0).stop 2024-08-28T19:39:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.102+0000 7fed0a590640 1 -- 172.21.15.16:0/864956388 shutdown_connections 2024-08-28T19:39:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.102+0000 7fed0a590640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fecec0b4ca0 0x7fecec0b71b0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.102+0000 7fed0a590640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed04108ac0 0x7fed0419d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.102+0000 7fed0a590640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed04108170 0x7fed04198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.102+0000 7fed0a590640 1 --2- 172.21.15.16:0/864956388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed04102880 0x7fed041980b0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.102+0000 7fed0a590640 1 -- 172.21.15.16:0/864956388 >> 172.21.15.16:0/864956388 conn(0x7fed04079550 msgr2=0x7fed04106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.102+0000 7fed0a590640 1 -- 172.21.15.16:0/864956388 shutdown_connections 2024-08-28T19:39:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:48.102+0000 7fed0a590640 1 -- 172.21.15.16:0/864956388 wait complete. 2024-08-28T19:39:48.479 INFO:tasks.workunit.client.0.smithi016.stdout: 10 129033 24.00 MB/sec execute 204 sec latency 333.994 ms 2024-08-28T19:39:48.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:48 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:39:48.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:48 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:39:48.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:48 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:39:48.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:48 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:39:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:48 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:39:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:48 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:39:49.061 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:39:49.376 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:49.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 129553 24.05 MB/sec execute 205 sec latency 542.940 ms 2024-08-28T19:39:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:49 smithi016 ceph-mon[18556]: pgmap v926: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 MiB/s wr, 698 op/s 2024-08-28T19:39:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:49 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:39:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:49 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:49 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:39:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:49 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:39:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:49 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:39:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:49 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:49 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:49 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:49 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:49 smithi028 ceph-mon[21769]: pgmap v926: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 MiB/s wr, 698 op/s 2024-08-28T19:39:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:49 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:39:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:49 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:49 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:39:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:49 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:39:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:49 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:39:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:49 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:49 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:49 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:49 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:49 smithi115 ceph-mon[22184]: pgmap v926: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 MiB/s wr, 698 op/s 2024-08-28T19:39:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:49 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:39:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:49 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:49 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:39:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:49 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:39:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:49 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:39:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:49 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:49 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:49 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:49 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:50.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 130021 24.03 MB/sec execute 206 sec latency 79.194 ms 2024-08-28T19:39:50.587 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:50 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:50.587 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:50 smithi016 ceph-mon[18556]: pgmap v927: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.2 MiB/s wr, 598 op/s 2024-08-28T19:39:50.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.586+0000 7f6e00900640 1 -- 172.21.15.16:0/553326717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dfc108aa0 msgr2=0x7f6dfc10f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.586+0000 7f6e00900640 1 --2- 172.21.15.16:0/553326717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dfc108aa0 0x7f6dfc10f2d0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009ca0 tx=0x7f6df402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.586+0000 7f6e00900640 1 -- 172.21.15.16:0/553326717 shutdown_connections 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.586+0000 7f6e00900640 1 --2- 172.21.15.16:0/553326717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dfc108aa0 0x7f6dfc10f2d0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.586+0000 7f6e00900640 1 --2- 172.21.15.16:0/553326717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dfc108150 0x7f6dfc108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.586+0000 7f6e00900640 1 --2- 172.21.15.16:0/553326717 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6dfc102810 0x7f6dfc102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.586+0000 7f6e00900640 1 -- 172.21.15.16:0/553326717 >> 172.21.15.16:0/553326717 conn(0x7f6dfc0794d0 msgr2=0x7f6dfc0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.586+0000 7f6e00900640 1 -- 172.21.15.16:0/553326717 shutdown_connections 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.586+0000 7f6e00900640 1 -- 172.21.15.16:0/553326717 wait complete. 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.587+0000 7f6e00900640 1 Processor -- start 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6e00900640 1 -- start start 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6e00900640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6dfc102810 0x7f6dfc19a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6e00900640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dfc108150 0x7f6dfc19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6e00900640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dfc108aa0 0x7f6dfc19fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6e00900640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc113830 con 0x7f6dfc108aa0 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6e00900640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc1136b0 con 0x7f6dfc108150 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6e00900640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc1139b0 con 0x7f6dfc102810 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6dfa59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6dfc102810 0x7f6dfc19a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6dfad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dfc108aa0 0x7f6dfc19fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6df3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dfc108150 0x7f6dfc19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6dfad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dfc108aa0 0x7f6dfc19fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51782/0 (socket says 172.21.15.16:51782) 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6dfa59c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6dfc102810 0x7f6dfc19a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:59012/0 (socket says 172.21.15.16:59012) 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6df3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dfc108150 0x7f6dfc19a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57964/0 (socket says 172.21.15.16:57964) 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.588+0000 7f6dfad9d640 1 -- 172.21.15.16:0/3220964249 learned_addr learned my addr 172.21.15.16:0/3220964249 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.589+0000 7f6df3fff640 1 -- 172.21.15.16:0/3220964249 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6dfc102810 msgr2=0x7f6dfc19a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.589+0000 7f6df3fff640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6dfc102810 0x7f6dfc19a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.589+0000 7f6df3fff640 1 -- 172.21.15.16:0/3220964249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dfc108aa0 msgr2=0x7f6dfc19fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.589+0000 7f6df3fff640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dfc108aa0 0x7f6dfc19fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.589+0000 7f6df3fff640 1 -- 172.21.15.16:0/3220964249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dfc1a0230 con 0x7f6dfc108150 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.589+0000 7f6dfad9d640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dfc108aa0 0x7f6dfc19fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.589+0000 7f6df3fff640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dfc108150 0x7f6dfc19a860 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009b90 tx=0x7f6de400b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.589+0000 7f6df37fe640 1 -- 172.21.15.16:0/3220964249 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de401e070 con 0x7f6dfc108150 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.589+0000 7f6e00900640 1 -- 172.21.15.16:0/3220964249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dfc1a0520 con 0x7f6dfc108150 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.589+0000 7f6df37fe640 1 -- 172.21.15.16:0/3220964249 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de40198c0 con 0x7f6dfc108150 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.590+0000 7f6e00900640 1 -- 172.21.15.16:0/3220964249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dfc10fc40 con 0x7f6dfc108150 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.590+0000 7f6df37fe640 1 -- 172.21.15.16:0/3220964249 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4018730 con 0x7f6dfc108150 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.593+0000 7f6e06e75740 1 Processor -- start 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.593+0000 7f6e06e75740 1 -- start start 2024-08-28T19:39:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.593+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66aae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.593+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66aae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 0x559c66aaed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6e06e75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559c66a25630 con 0x559c66709070 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6dfa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66aae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6dfad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66aae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6dfa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66aae800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57968/0 (socket says 172.21.15.16:57968) 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6dfa59c640 1 -- 172.21.15.16:0/4098884663 learned_addr learned my addr 172.21.15.16:0/4098884663 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6dfad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66aae2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51796/0 (socket says 172.21.15.16:51796) 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6e06e75740 1 -- 172.21.15.16:0/4098884663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559c66a14bc0 con 0x559c666e01b0 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6e06e75740 1 -- 172.21.15.16:0/4098884663 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559c66a18b70 con 0x559c666e8390 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6df37fe640 1 -- 172.21.15.16:0/3220964249 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f6de4021070 con 0x7f6dfc108150 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6dfad9d640 1 -- 172.21.15.16:0/4098884663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 msgr2=0x559c66aaed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6dfad9d640 1 --2- 172.21.15.16:0/4098884663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 0x559c66aaed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6dfad9d640 1 -- 172.21.15.16:0/4098884663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 msgr2=0x559c66aae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6dfad9d640 1 --2- 172.21.15.16:0/4098884663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66aae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.594+0000 7f6dfad9d640 1 -- 172.21.15.16:0/4098884663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c66774650 con 0x559c66709070 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.595+0000 7f6dfad9d640 1 --2- 172.21.15.16:0/4098884663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66aae2c0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f6df4005cc0 tx=0x7f6df402f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2aea564e52b329 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.595+0000 7f6df0ff9640 1 -- 172.21.15.16:0/4098884663 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4031450 con 0x559c66709070 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.595+0000 7f6df0ff9640 1 -- 172.21.15.16:0/4098884663 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6df4036040 con 0x559c66709070 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.595+0000 7f6df0ff9640 1 -- 172.21.15.16:0/4098884663 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df40043c0 con 0x559c66709070 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.596+0000 7f6df37fe640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6de80b4d20 0x7f6de80b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.596+0000 7f6df37fe640 1 -- 172.21.15.16:0/3220964249 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f6de40e90b0 con 0x7f6dfc108150 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.596+0000 7f6dfa59c640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6de80b4d20 0x7f6de80b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.596+0000 7f6e06e75740 1 -- 172.21.15.16:0/4098884663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 msgr2=0x559c66aae2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.596+0000 7f6e06e75740 1 --2- 172.21.15.16:0/4098884663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66aae2c0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f6df4005cc0 tx=0x7f6df402f310 comp rx=0 tx=0).stop 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.597+0000 7f6e06e75740 1 -- 172.21.15.16:0/4098884663 shutdown_connections 2024-08-28T19:39:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.597+0000 7f6e06e75740 1 --2- 172.21.15.16:0/4098884663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 0x559c66aaed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.597+0000 7f6e06e75740 1 --2- 172.21.15.16:0/4098884663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66aae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.597+0000 7f6e06e75740 1 --2- 172.21.15.16:0/4098884663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66aae2c0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.597+0000 7f6e06e75740 1 -- 172.21.15.16:0/4098884663 >> 172.21.15.16:0/4098884663 conn(0x559c666fd460 msgr2=0x559c66aac1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.597+0000 7f6dfa59c640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6de80b4d20 0x7f6de80b7230 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x559c66aacf40 tx=0x7f6dec005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.597+0000 7f6e06e75740 1 -- 172.21.15.16:0/4098884663 shutdown_connections 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.597+0000 7f6e06e75740 1 -- 172.21.15.16:0/4098884663 wait complete. 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.597+0000 7f6e06e75740 1 -- start start 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6e06e75740 1 Processor -- start 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66ab9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 0x559c66aba020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66ac5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6e06e75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559c66877700 con 0x559c66709070 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6e06e75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559c6699b700 con 0x559c666e01b0 2024-08-28T19:39:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6e06e75740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559c66abc8c0 con 0x559c666e8390 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6e06e75740 10 client.0 _ll_register_callbacks cb 0x559c669cadc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6dfad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66ac5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6dfad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66ac5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51808/0 (socket says 172.21.15.16:51808) 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6dfad9d640 1 -- 172.21.15.16:0/819377997 learned_addr learned my addr 172.21.15.16:0/819377997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6df3fff640 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 0x559c66aba020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6dfa59c640 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66ab9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6dfad9d640 1 -- 172.21.15.16:0/819377997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 msgr2=0x559c66aba020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6dfad9d640 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 0x559c66aba020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6dfad9d640 1 -- 172.21.15.16:0/819377997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 msgr2=0x559c66ab9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6dfad9d640 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66ab9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.598+0000 7f6dfad9d640 1 -- 172.21.15.16:0/819377997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c669d48b0 con 0x559c66709070 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6dfad9d640 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66ac5bb0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f6df402a900 tx=0x7f6df40098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773e7387ed2dbfdf server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6de1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6de1ffb640 1 -- 172.21.15.16:0/819377997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4054060 con 0x559c66709070 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6e06e75740 1 -- 172.21.15.16:0/819377997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6df4036040 con 0x559c66709070 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6de1ffb640 1 -- 172.21.15.16:0/819377997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f6df4003b80 con 0x559c66709070 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6de1ffb640 1 -- 172.21.15.16:0/819377997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f6df405b050 con 0x559c66709070 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6e06e75740 10 client.19887 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6e06e75740 1 -- 172.21.15.16:0/819377997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x559c66ac86f0 con 0x559c66709070 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6de1ffb640 1 -- 172.21.15.16:0/819377997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6df4036040 con 0x559c66709070 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6de1ffb640 1 -- 172.21.15.16:0/819377997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7f6df4044070 con 0x559c66709070 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6e06e75740 10 client.19887 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.599+0000 7f6e06e75740 10 client.19887 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.600+0000 7f6e06e75740 20 client.19887 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.600+0000 7f6e06e75740 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559c666c79f0 0x559c66acba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.600+0000 7f6e06e75740 4 client.19887 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.600+0000 7f6e06e75740 1 -- 172.21.15.16:0/819377997 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c66a27580 con 0x559c666c79f0 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.600+0000 7f6dfa59c640 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559c666c79f0 0x559c66acba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.600+0000 7f6de1ffb640 10 client.19887 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.600+0000 7f6dfa59c640 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559c666c79f0 0x559c66acba50 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36b7dccf6724bde8 server_cookie=4be738adc2d87f1b in_seq=0 out_seq=0 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6de1ffb640 1 -- 172.21.15.16:0/819377997 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559c66a27580 con 0x559c666c79f0 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6de1ffb640 10 client.19887 handle_command_reply: tid=0 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 1 client.19887 shutdown 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 1 -- 172.21.15.16:0/819377997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 msgr2=0x559c66ac5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66ac5bb0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f6df402a900 tx=0x7f6df40098a0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 1 -- 172.21.15.16:0/819377997 shutdown_connections 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66ac5bb0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559c666c79f0 0x559c66acba50 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6de1ffb640 0 client.19887 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 0x559c66aba020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 1 --2- 172.21.15.16:0/819377997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66ab9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 1 -- 172.21.15.16:0/819377997 >> 172.21.15.16:0/819377997 conn(0x559c666fd460 msgr2=0x559c66ab14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 1 -- 172.21.15.16:0/819377997 shutdown_connections 2024-08-28T19:39:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 1 -- 172.21.15.16:0/819377997 wait complete. 2024-08-28T19:39:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.603+0000 7f6e06e75740 20 client.19887 trim_cache size 0 max 16384 2024-08-28T19:39:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.627+0000 7f6e06e75740 1 Processor -- start 2024-08-28T19:39:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.627+0000 7f6e06e75740 1 -- start start 2024-08-28T19:39:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.627+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66ab89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.627+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c666e8390 0x559c66aac200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "8de90bf3-5743-4711-98c2-a56ca0757291": { 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "8de90bf3-5743-4711-98c2-a56ca0757291", 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:39:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:55.619275+0000", 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:39:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:39:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:39:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "8de90bf3-5743-4711-98c2-a56ca0757291": { 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:39:48.098772+0000", 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:39:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:39:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:39:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:39:50.670 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:39:50.670 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:39:50.670 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:39:50.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:50.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:50.670 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:39:50.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.627+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c66709070 0x559c66ac5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.627+0000 7f6e06e75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc1aeab0 con 0x559c666e8390 2024-08-28T19:39:50.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.627+0000 7f6e06e75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc1b07d0 con 0x559c666e01b0 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.627+0000 7f6e06e75740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc1af210 con 0x559c66709070 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6dfa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66ab89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6dfa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66ab89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57996/0 (socket says 172.21.15.16:57996) 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6dfa59c640 1 -- 172.21.15.16:0/3849304750 learned_addr learned my addr 172.21.15.16:0/3849304750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6dfad9d640 1 --2- 172.21.15.16:0/3849304750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c66709070 0x559c66ac5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6df3fff640 1 --2- 172.21.15.16:0/3849304750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c666e8390 0x559c66aac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6dfa59c640 1 -- 172.21.15.16:0/3849304750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c66709070 msgr2=0x559c66ac5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6dfa59c640 1 --2- 172.21.15.16:0/3849304750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c66709070 0x559c66ac5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6dfa59c640 1 -- 172.21.15.16:0/3849304750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c666e8390 msgr2=0x559c66aac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6dfa59c640 1 --2- 172.21.15.16:0/3849304750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c666e8390 0x559c66aac200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6dfa59c640 1 -- 172.21.15.16:0/3849304750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c66aacb20 con 0x559c666e01b0 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6df3fff640 1 --2- 172.21.15.16:0/3849304750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c666e8390 0x559c66aac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:39:50.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.628+0000 7f6dfa59c640 1 --2- 172.21.15.16:0/3849304750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66ab89d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6dec012cb0 tx=0x7f6dec019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7d85cec5c1aaba2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.629+0000 7f6de3fff640 1 -- 172.21.15.16:0/3849304750 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec01dd90 con 0x559c666e01b0 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.629+0000 7f6de3fff640 1 -- 172.21.15.16:0/3849304750 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dec0054d0 con 0x559c666e01b0 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.629+0000 7f6de3fff640 1 -- 172.21.15.16:0/3849304750 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec02a670 con 0x559c666e01b0 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.630+0000 7f6e06e75740 1 -- 172.21.15.16:0/3849304750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 msgr2=0x559c66ab89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.630+0000 7f6e06e75740 1 --2- 172.21.15.16:0/3849304750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66ab89d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6dec012cb0 tx=0x7f6dec019000 comp rx=0 tx=0).stop 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.630+0000 7f6e06e75740 1 -- 172.21.15.16:0/3849304750 shutdown_connections 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.630+0000 7f6e06e75740 1 --2- 172.21.15.16:0/3849304750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c66709070 0x559c66ac5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.630+0000 7f6e06e75740 1 --2- 172.21.15.16:0/3849304750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c666e8390 0x559c66aac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.630+0000 7f6e06e75740 1 --2- 172.21.15.16:0/3849304750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66ab89d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.630+0000 7f6e06e75740 1 -- 172.21.15.16:0/3849304750 >> 172.21.15.16:0/3849304750 conn(0x559c666fd460 msgr2=0x559c66aabaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.630+0000 7f6e06e75740 1 -- 172.21.15.16:0/3849304750 shutdown_connections 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.630+0000 7f6e06e75740 1 -- 172.21.15.16:0/3849304750 wait complete. 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.630+0000 7f6e06e75740 1 -- start start 2024-08-28T19:39:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.630+0000 7f6e06e75740 1 Processor -- start 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66abddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 0x559c66abe2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6e06e75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66b352e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6e06e75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc1b0040 con 0x559c66709070 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6e06e75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc1af8e0 con 0x559c666e01b0 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6e06e75740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc1acab0 con 0x559c666e8390 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6e06e75740 10 client.0 _ll_register_callbacks cb 0x7f6dec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6dfa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66abddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6dfad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66b352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6dfa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66abddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:57998/0 (socket says 172.21.15.16:57998) 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6dfa59c640 1 -- 172.21.15.16:0/3401197203 learned_addr learned my addr 172.21.15.16:0/3401197203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6df3fff640 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 0x559c66abe2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6dfad9d640 1 -- 172.21.15.16:0/3401197203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 msgr2=0x559c66abe2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6dfad9d640 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 0x559c66abe2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6dfad9d640 1 -- 172.21.15.16:0/3401197203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 msgr2=0x559c66abddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6dfad9d640 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66abddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6dfad9d640 1 -- 172.21.15.16:0/3401197203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c66abeb40 con 0x559c66709070 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6dfa59c640 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66abddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6dfad9d640 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66b352e0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f6df4005e30 tx=0x7f6df4009910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd0cb534a6a592d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.631+0000 7f6de1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6de1ffb640 1 -- 172.21.15.16:0/3401197203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4054040 con 0x559c66709070 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6de1ffb640 1 -- 172.21.15.16:0/3401197203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f6df402fcf0 con 0x559c66709070 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6e06e75740 1 -- 172.21.15.16:0/3401197203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dec0054d0 con 0x559c66709070 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6de1ffb640 1 -- 172.21.15.16:0/3401197203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f6df4032eb0 con 0x559c66709070 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6e06e75740 10 client.19890 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6e06e75740 1 -- 172.21.15.16:0/3401197203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x559c66b367b0 con 0x559c66709070 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6de1ffb640 1 -- 172.21.15.16:0/3401197203 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dec0054d0 con 0x559c66709070 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6de1ffb640 1 -- 172.21.15.16:0/3401197203 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7f6df4044040 con 0x559c66709070 2024-08-28T19:39:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6e06e75740 10 client.19890 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6e06e75740 10 client.19890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6e06e75740 20 client.19890 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6e06e75740 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559c6669cda0 0x559c66b38e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6e06e75740 4 client.19890 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.632+0000 7f6e06e75740 1 -- 172.21.15.16:0/3401197203 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559c66a23020 con 0x559c6669cda0 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.633+0000 7f6dfa59c640 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559c6669cda0 0x559c66b38e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.633+0000 7f6de1ffb640 10 client.19890 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.633+0000 7f6dfa59c640 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559c6669cda0 0x559c66b38e10 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44119596aca28d6c server_cookie=34f80f103f6a85e7 in_seq=0 out_seq=0 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6de1ffb640 1 -- 172.21.15.16:0/3401197203 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7386 (crc 0 0 0) 0x559c66a23020 con 0x559c6669cda0 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6de1ffb640 10 client.19890 handle_command_reply: tid=0 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6e06e75740 1 client.19890 shutdown 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6e06e75740 1 -- 172.21.15.16:0/3401197203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 msgr2=0x559c66b352e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6e06e75740 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66b352e0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f6df4005e30 tx=0x7f6df4009910 comp rx=0 tx=0).stop 2024-08-28T19:39:50.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6e06e75740 1 -- 172.21.15.16:0/3401197203 shutdown_connections 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6e06e75740 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c66709070 0x559c66b352e0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6e06e75740 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x559c6669cda0 0x559c66b38e10 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6e06e75740 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559c666e8390 0x559c66abe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6e06e75740 1 --2- 172.21.15.16:0/3401197203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c666e01b0 0x559c66abddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6e06e75740 1 -- 172.21.15.16:0/3401197203 >> 172.21.15.16:0/3401197203 conn(0x559c666fd460 msgr2=0x559c66ab13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6de1ffb640 0 client.19890 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6e06e75740 1 -- 172.21.15.16:0/3401197203 shutdown_connections 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.635+0000 7f6e06e75740 1 -- 172.21.15.16:0/3401197203 wait complete. 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.636+0000 7f6e06e75740 20 client.19890 trim_cache size 0 max 16384 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.636+0000 7f6e00900640 1 -- 172.21.15.16:0/3220964249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6de80b4d20 msgr2=0x7f6de80b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.636+0000 7f6e00900640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6de80b4d20 0x7f6de80b7230 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x559c66aacf40 tx=0x7f6dec005f90 comp rx=0 tx=0).stop 2024-08-28T19:39:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.636+0000 7f6e00900640 1 -- 172.21.15.16:0/3220964249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dfc108150 msgr2=0x7f6dfc19a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:50.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.636+0000 7f6e00900640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dfc108150 0x7f6dfc19a860 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009b90 tx=0x7f6de400b690 comp rx=0 tx=0).stop 2024-08-28T19:39:50.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.636+0000 7f6e00900640 1 -- 172.21.15.16:0/3220964249 shutdown_connections 2024-08-28T19:39:50.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.636+0000 7f6e00900640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dfc108aa0 0x7f6dfc19fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.636+0000 7f6e00900640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f6de80b4d20 0x7f6de80b7230 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.636+0000 7f6e00900640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dfc108150 0x7f6dfc19a860 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.636+0000 7f6e00900640 1 --2- 172.21.15.16:0/3220964249 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6dfc102810 0x7f6dfc19a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:50.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.636+0000 7f6e00900640 1 -- 172.21.15.16:0/3220964249 >> 172.21.15.16:0/3220964249 conn(0x7f6dfc0794d0 msgr2=0x7f6dfc104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:50.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.637+0000 7f6e00900640 1 -- 172.21.15.16:0/3220964249 shutdown_connections 2024-08-28T19:39:50.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:50.637+0000 7f6e00900640 1 -- 172.21.15.16:0/3220964249 wait complete. 2024-08-28T19:39:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:50 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:50 smithi028 ceph-mon[21769]: pgmap v927: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.2 MiB/s wr, 598 op/s 2024-08-28T19:39:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:50 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:39:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:50 smithi115 ceph-mon[22184]: pgmap v927: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.2 MiB/s wr, 598 op/s 2024-08-28T19:39:51.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 130565 24.02 MB/sec execute 207 sec latency 82.390 ms 2024-08-28T19:39:51.577 INFO:tasks.cephfs.filesystem:scrub status for tag:8de90bf3-5743-4711-98c2-a56ca0757291 - {'path': '/', 'tag': '8de90bf3-5743-4711-98c2-a56ca0757291', 'options': 'recursive,force'} 2024-08-28T19:39:52.479 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.7 2024-08-28T19:39:52.480 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:39:52.481 INFO:tasks.workunit.client.0.smithi016.stdout: 10 131293 24.05 MB/sec execute 208 sec latency 172.487 ms 2024-08-28T19:39:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:52 smithi016 ceph-mon[18556]: pgmap v928: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.0 MiB/s wr, 608 op/s 2024-08-28T19:39:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:52 smithi028 ceph-mon[21769]: pgmap v928: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.0 MiB/s wr, 608 op/s 2024-08-28T19:39:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:52 smithi115 ceph-mon[22184]: pgmap v928: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.0 MiB/s wr, 608 op/s 2024-08-28T19:39:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.415+0000 7f494a90d640 1 -- 172.21.15.16:0/727029361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4944108140 msgr2=0x7f4944108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.415+0000 7f494a90d640 1 --2- 172.21.15.16:0/727029361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4944108140 0x7f4944108550 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f4938009900 tx=0x7f493802a400 comp rx=0 tx=0).stop 2024-08-28T19:39:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.415+0000 7f494a90d640 1 -- 172.21.15.16:0/727029361 shutdown_connections 2024-08-28T19:39:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.415+0000 7f494a90d640 1 --2- 172.21.15.16:0/727029361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4944108a90 0x7f494410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.415+0000 7f494a90d640 1 --2- 172.21.15.16:0/727029361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4944108140 0x7f4944108550 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.415+0000 7f494a90d640 1 --2- 172.21.15.16:0/727029361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4944102800 0x7f4944102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.415+0000 7f494a90d640 1 -- 172.21.15.16:0/727029361 >> 172.21.15.16:0/727029361 conn(0x7f49440794d0 msgr2=0x7f49440798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.415+0000 7f494a90d640 1 -- 172.21.15.16:0/727029361 shutdown_connections 2024-08-28T19:39:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.415+0000 7f494a90d640 1 -- 172.21.15.16:0/727029361 wait complete. 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.416+0000 7f494a90d640 1 Processor -- start 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.416+0000 7f494a90d640 1 -- start start 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.416+0000 7f494a90d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4944102800 0x7f494419aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.416+0000 7f494a90d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4944108140 0x7f494419b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.416+0000 7f494a90d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4944108a90 0x7f494419ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f494a90d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49441126c0 con 0x7f4944108140 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f494a90d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4944112540 con 0x7f4944108a90 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f494a90d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4944112840 con 0x7f4944102800 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f49437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4944108140 0x7f494419b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f49437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4944108140 0x7f494419b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51834/0 (socket says 172.21.15.16:51834) 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f49437fe640 1 -- 172.21.15.16:0/2615341895 learned_addr learned my addr 172.21.15.16:0/2615341895 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f4943fff640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4944102800 0x7f494419aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:53.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f4948eaa640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4944108a90 0x7f494419ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f49437fe640 1 -- 172.21.15.16:0/2615341895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4944102800 msgr2=0x7f494419aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f49437fe640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4944102800 0x7f494419aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f49437fe640 1 -- 172.21.15.16:0/2615341895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4944108a90 msgr2=0x7f494419ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f49437fe640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4944108a90 0x7f494419ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.417+0000 7f49437fe640 1 -- 172.21.15.16:0/2615341895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49441a6280 con 0x7f4944108140 2024-08-28T19:39:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.418+0000 7f49437fe640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4944108140 0x7f494419b530 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f4938032fd0 tx=0x7f4938031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.418+0000 7f49417fa640 1 -- 172.21.15.16:0/2615341895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493803f030 con 0x7f4944108140 2024-08-28T19:39:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.418+0000 7f494a90d640 1 -- 172.21.15.16:0/2615341895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49441a6510 con 0x7f4944108140 2024-08-28T19:39:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.418+0000 7f494a90d640 1 -- 172.21.15.16:0/2615341895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49441a6a40 con 0x7f4944108140 2024-08-28T19:39:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.418+0000 7f49417fa640 1 -- 172.21.15.16:0/2615341895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4938004490 con 0x7f4944108140 2024-08-28T19:39:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.418+0000 7f49417fa640 1 -- 172.21.15.16:0/2615341895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938035070 con 0x7f4944108140 2024-08-28T19:39:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.419+0000 7f494a90d640 1 -- 172.21.15.16:0/2615341895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4930005290 con 0x7f4944108140 2024-08-28T19:39:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.421+0000 7f49417fa640 1 -- 172.21.15.16:0/2615341895 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4938042030 con 0x7f4944108140 2024-08-28T19:39:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.422+0000 7f49417fa640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f492c0b4d50 0x7f492c0b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.422+0000 7f4943fff640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f492c0b4d50 0x7f492c0b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.422+0000 7f49417fa640 1 -- 172.21.15.16:0/2615341895 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f4938105720 con 0x7f4944108140 2024-08-28T19:39:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.423+0000 7f4943fff640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f492c0b4d50 0x7f492c0b7260 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f494406b760 tx=0x7f4934005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:53.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.428+0000 7f49417fa640 1 -- 172.21.15.16:0/2615341895 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f49380cc0e0 con 0x7f4944108140 2024-08-28T19:39:53.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 131992 24.17 MB/sec execute 209 sec latency 340.554 ms 2024-08-28T19:39:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.703+0000 7f494a90d640 1 -- 172.21.15.16:0/2615341895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4930005850 con 0x7f4944108140 2024-08-28T19:39:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.704+0000 7f49417fa640 1 -- 172.21.15.16:0/2615341895 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+11994 (secure 0 0 0) 0x7f49380cb9e0 con 0x7f4944108140 2024-08-28T19:39:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:39:53.706 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":99,"btime":"2024-08-28T19:39:40:439110+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":99,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:39:39.583625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:39:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 99 2024-08-28T19:39:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.707+0000 7f494a90d640 1 -- 172.21.15.16:0/2615341895 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f492c0b4d50 msgr2=0x7f492c0b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.707+0000 7f494a90d640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f492c0b4d50 0x7f492c0b7260 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f494406b760 tx=0x7f4934005f90 comp rx=0 tx=0).stop 2024-08-28T19:39:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.707+0000 7f494a90d640 1 -- 172.21.15.16:0/2615341895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4944108140 msgr2=0x7f494419b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.707+0000 7f494a90d640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4944108140 0x7f494419b530 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f4938032fd0 tx=0x7f4938031a20 comp rx=0 tx=0).stop 2024-08-28T19:39:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.708+0000 7f494a90d640 1 -- 172.21.15.16:0/2615341895 shutdown_connections 2024-08-28T19:39:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.708+0000 7f494a90d640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f492c0b4d50 0x7f492c0b7260 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.708+0000 7f494a90d640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4944108a90 0x7f494419ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.708+0000 7f494a90d640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4944108140 0x7f494419b530 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.708+0000 7f494a90d640 1 --2- 172.21.15.16:0/2615341895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4944102800 0x7f494419aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:53.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.708+0000 7f494a90d640 1 -- 172.21.15.16:0/2615341895 >> 172.21.15.16:0/2615341895 conn(0x7f49440794d0 msgr2=0x7f4944104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:53.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.708+0000 7f494a90d640 1 -- 172.21.15.16:0/2615341895 shutdown_connections 2024-08-28T19:39:53.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:53.708+0000 7f494a90d640 1 -- 172.21.15.16:0/2615341895 wait complete. 2024-08-28T19:39:54.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:53 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2615341895' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:39:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:53 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2615341895' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:39:54.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:53 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2615341895' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:39:54.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 132756 24.24 MB/sec execute 210 sec latency 112.424 ms 2024-08-28T19:39:55.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:54 smithi016 ceph-mon[18556]: pgmap v929: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.1 MiB/s wr, 515 op/s 2024-08-28T19:39:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:54 smithi028 ceph-mon[21769]: pgmap v929: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.1 MiB/s wr, 515 op/s 2024-08-28T19:39:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:54 smithi115 ceph-mon[22184]: pgmap v929: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.1 MiB/s wr, 515 op/s 2024-08-28T19:39:55.445 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 146.7 --await-for 45.0' 2024-08-28T19:39:55.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 146.7 --await-for 45.0 2024-08-28T19:39:55.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 133368 24.30 MB/sec execute 211 sec latency 110.948 ms 2024-08-28T19:39:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:39:56.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 133919 24.34 MB/sec execute 212 sec latency 67.455 ms 2024-08-28T19:39:57.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.605+0000 7f07d5847640 1 -- 172.21.15.16:0/3938821380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0108160 msgr2=0x7f07d0108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:57.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.605+0000 7f07d5847640 1 --2- 172.21.15.16:0/3938821380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0108160 0x7f07d0108570 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009900 tx=0x7f07b802a410 comp rx=0 tx=0).stop 2024-08-28T19:39:57.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.606+0000 7f07d5847640 1 -- 172.21.15.16:0/3938821380 shutdown_connections 2024-08-28T19:39:57.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.606+0000 7f07d5847640 1 --2- 172.21.15.16:0/3938821380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0108ab0 0x7f07d010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.606+0000 7f07d5847640 1 --2- 172.21.15.16:0/3938821380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0108160 0x7f07d0108570 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.606+0000 7f07d5847640 1 --2- 172.21.15.16:0/3938821380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0102820 0x7f07d0102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.606+0000 7f07d5847640 1 -- 172.21.15.16:0/3938821380 >> 172.21.15.16:0/3938821380 conn(0x7f07d0079500 msgr2=0x7f07d0079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.606+0000 7f07d5847640 1 -- 172.21.15.16:0/3938821380 shutdown_connections 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.606+0000 7f07d5847640 1 -- 172.21.15.16:0/3938821380 wait complete. 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.606+0000 7f07d5847640 1 Processor -- start 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07d5847640 1 -- start start 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07d5847640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102820 0x7f07d019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07d5847640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0108160 0x7f07d019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07d5847640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0108ab0 0x7f07d019fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07d5847640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01126e0 con 0x7f07d0108160 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07d5847640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07d0112560 con 0x7f07d0102820 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07d5847640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f07d0112860 con 0x7f07d0108ab0 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0108160 0x7f07d019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07ceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102820 0x7f07d019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07ceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102820 0x7f07d019a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54708/0 (socket says 172.21.15.16:54708) 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07ceffd640 1 -- 172.21.15.16:0/2845653991 learned_addr learned my addr 172.21.15.16:0/2845653991 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07cf7fe640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0108ab0 0x7f07d019fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07c7fff640 1 -- 172.21.15.16:0/2845653991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0108ab0 msgr2=0x7f07d019fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07c7fff640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0108ab0 0x7f07d019fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07c7fff640 1 -- 172.21.15.16:0/2845653991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102820 msgr2=0x7f07d019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.607+0000 7f07c7fff640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102820 0x7f07d019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.608+0000 7f07c7fff640 1 -- 172.21.15.16:0/2845653991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d01a0170 con 0x7f07d0108160 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.608+0000 7f07ceffd640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102820 0x7f07d019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.608+0000 7f07cf7fe640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0108ab0 0x7f07d019fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.608+0000 7f07c7fff640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0108160 0x7f07d019a7a0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f07b802a930 tx=0x7f07b8031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.608+0000 7f07d4845640 1 -- 172.21.15.16:0/2845653991 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b803f070 con 0x7f07d0108160 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.608+0000 7f07d4845640 1 -- 172.21.15.16:0/2845653991 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b8004490 con 0x7f07d0108160 2024-08-28T19:39:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.608+0000 7f07d4845640 1 -- 172.21.15.16:0/2845653991 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8035070 con 0x7f07d0108160 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.608+0000 7f07d5847640 1 -- 172.21.15.16:0/2845653991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d01a0400 con 0x7f07d0108160 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.608+0000 7f07d5847640 1 -- 172.21.15.16:0/2845653991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d01a6bc0 con 0x7f07d0108160 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.611+0000 7f07d4845640 1 -- 172.21.15.16:0/2845653991 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f07b803b460 con 0x7f07d0108160 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.611+0000 7f07dbdbc740 1 Processor -- start 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.611+0000 7f07dbdbc740 1 -- start start 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.611+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f9bc840 0x55954fd63c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.611+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f993980 0x55954fd641d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.611+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f99bb60 0x55954fd64710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55954fcd8330 con 0x55954f9bc840 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55954fcc78c0 con 0x55954f993980 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55954fccb870 con 0x55954f99bb60 2024-08-28T19:39:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07cf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f9bc840 0x55954fd63c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07cf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f9bc840 0x55954fd63c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37214/0 (socket says 172.21.15.16:37214) 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07cf7fe640 1 -- 172.21.15.16:0/3303058388 learned_addr learned my addr 172.21.15.16:0/3303058388 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07c7fff640 1 --2- 172.21.15.16:0/3303058388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f99bb60 0x55954fd64710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07ceffd640 1 --2- 172.21.15.16:0/3303058388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f993980 0x55954fd641d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07cf7fe640 1 -- 172.21.15.16:0/3303058388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f99bb60 msgr2=0x55954fd64710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07cf7fe640 1 --2- 172.21.15.16:0/3303058388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f99bb60 0x55954fd64710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07cf7fe640 1 -- 172.21.15.16:0/3303058388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f993980 msgr2=0x55954fd641d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07cf7fe640 1 --2- 172.21.15.16:0/3303058388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f993980 0x55954fd641d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07cf7fe640 1 -- 172.21.15.16:0/3303058388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55954fc7fdf0 con 0x55954f9bc840 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07d4845640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4d20 0x7f07bc0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07cf7fe640 1 --2- 172.21.15.16:0/3303058388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f9bc840 0x55954fd63c90 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f07c80117e0 tx=0x7f07c8011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afcb4f3c5a539e80 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.612+0000 7f07ceffd640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4d20 0x7f07bc0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.613+0000 7f07d4845640 1 -- 172.21.15.16:0/2845653991 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f07b8105d70 con 0x7f07d0108160 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.613+0000 7f07c5ffb640 1 -- 172.21.15.16:0/3303058388 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c8011310 con 0x55954f9bc840 2024-08-28T19:39:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.613+0000 7f07c5ffb640 1 -- 172.21.15.16:0/3303058388 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07c8003e20 con 0x55954f9bc840 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.613+0000 7f07c5ffb640 1 -- 172.21.15.16:0/3303058388 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c80044f0 con 0x55954f9bc840 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.614+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3303058388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f9bc840 msgr2=0x55954fd63c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.614+0000 7f07ceffd640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4d20 0x7f07bc0b7230 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55954fd62910 tx=0x7f07c0005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.614+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/3303058388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f9bc840 0x55954fd63c90 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f07c80117e0 tx=0x7f07c8011ca0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.614+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3303058388 shutdown_connections 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.614+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/3303058388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f99bb60 0x55954fd64710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.614+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/3303058388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f993980 0x55954fd641d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.614+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/3303058388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f9bc840 0x55954fd63c90 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.614+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3303058388 >> 172.21.15.16:0/3303058388 conn(0x55954f9b0990 msgr2=0x55954fd61b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.615+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3303058388 shutdown_connections 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.615+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3303058388 wait complete. 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.615+0000 7f07dbdbc740 1 -- start start 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07dbdbc740 1 Processor -- start 2024-08-28T19:39:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 0x55954fd738d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f99bb60 0x55954fd73e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f9bc840 0x55954fd7fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55954fb2a7a0 con 0x55954f993980 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55954fc4e240 con 0x55954f9bc840 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55954fd767d0 con 0x55954f99bb60 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07dbdbc740 10 client.0 _ll_register_callbacks cb 0x55954f9d4090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07c7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f99bb60 0x55954fd73e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07ceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 0x55954fd738d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f9bc840 0x55954fd7fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07c7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f99bb60 0x55954fd73e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:54490/0 (socket says 172.21.15.16:54490) 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.616+0000 7f07c7fff640 1 -- 172.21.15.16:0/4161165861 learned_addr learned my addr 172.21.15.16:0/4161165861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.617+0000 7f07cf7fe640 1 -- 172.21.15.16:0/4161165861 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f99bb60 msgr2=0x55954fd73e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.617+0000 7f07cf7fe640 1 --2- 172.21.15.16:0/4161165861 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f99bb60 0x55954fd73e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.617+0000 7f07cf7fe640 1 -- 172.21.15.16:0/4161165861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 msgr2=0x55954fd738d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.617+0000 7f07cf7fe640 1 --2- 172.21.15.16:0/4161165861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 0x55954fd738d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.617+0000 7f07cf7fe640 1 -- 172.21.15.16:0/4161165861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55954fc842b0 con 0x55954f9bc840 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.617+0000 7f07cf7fe640 1 --2- 172.21.15.16:0/4161165861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f9bc840 0x55954fd7fb00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f07c8019000 tx=0x7f07c8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=853babac0c1b39a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.617+0000 7f07aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.617+0000 7f07aeffd640 1 -- 172.21.15.16:0/4161165861 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c802a060 con 0x55954f9bc840 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.617+0000 7f07dbdbc740 1 -- 172.21.15.16:0/4161165861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07c8003e20 con 0x55954f9bc840 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.617+0000 7f07aeffd640 1 -- 172.21.15.16:0/4161165861 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f07c800f970 con 0x55954f9bc840 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.618+0000 7f07aeffd640 1 -- 172.21.15.16:0/4161165861 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f07c8031050 con 0x55954f9bc840 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.618+0000 7f07dbdbc740 10 client.29354 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.618+0000 7f07dbdbc740 1 -- 172.21.15.16:0/4161165861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x55954fa36b90 con 0x55954f9bc840 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.618+0000 7f07aeffd640 1 -- 172.21.15.16:0/4161165861 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07c8003e20 con 0x55954f9bc840 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.618+0000 7f07aeffd640 1 -- 172.21.15.16:0/4161165861 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7f07c8018020 con 0x55954f9bc840 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.618+0000 7f07dbdbc740 10 client.29354 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.618+0000 7f07dbdbc740 10 client.29354 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.618+0000 7f07dbdbc740 20 client.29354 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.618+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/4161165861 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55954f97b1c0 0x55954fd857e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.618+0000 7f07dbdbc740 4 client.29354 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.618+0000 7f07dbdbc740 1 -- 172.21.15.16:0/4161165861 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55954fcda280 con 0x55954f97b1c0 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.619+0000 7f07ceffd640 1 --2- 172.21.15.16:0/4161165861 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55954f97b1c0 0x55954fd857e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.619+0000 7f07aeffd640 10 client.29354 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.619+0000 7f07ceffd640 1 --2- 172.21.15.16:0/4161165861 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55954f97b1c0 0x55954fd857e0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c83f992aedc6a553 server_cookie=ded154ed86fe5c6 in_seq=0 out_seq=0 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.623+0000 7f07aeffd640 1 -- 172.21.15.16:0/4161165861 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55954fcda280 con 0x55954f97b1c0 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.623+0000 7f07aeffd640 10 client.29354 handle_command_reply: tid=0 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.623+0000 7f07dbdbc740 1 client.29354 shutdown 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07dbdbc740 1 -- 172.21.15.16:0/4161165861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f9bc840 msgr2=0x55954fd7fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/4161165861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f9bc840 0x55954fd7fb00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f07c8019000 tx=0x7f07c8005fd0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07dbdbc740 1 -- 172.21.15.16:0/4161165861 shutdown_connections 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/4161165861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f9bc840 0x55954fd7fb00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/4161165861 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55954f97b1c0 0x55954fd857e0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/4161165861 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f99bb60 0x55954fd73e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07aeffd640 0 client.29354 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/4161165861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 0x55954fd738d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07dbdbc740 1 -- 172.21.15.16:0/4161165861 >> 172.21.15.16:0/4161165861 conn(0x55954f9b0990 msgr2=0x55954fd65680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07dbdbc740 1 -- 172.21.15.16:0/4161165861 shutdown_connections 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07dbdbc740 1 -- 172.21.15.16:0/4161165861 wait complete. 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.624+0000 7f07dbdbc740 20 client.29354 trim_cache size 0 max 16384 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.644+0000 7f07dbdbc740 1 Processor -- start 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.644+0000 7f07dbdbc740 1 -- start start 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 0x55954fd6f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f99bb60 0x55954fd65380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f9bc840 0x55954fd70ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01aea60 con 0x55954f993980 2024-08-28T19:39:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01b0780 con 0x55954f99bb60 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01af1c0 con 0x55954f9bc840 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07ceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 0x55954fd6f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07ceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 0x55954fd6f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37234/0 (socket says 172.21.15.16:37234) 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07ceffd640 1 -- 172.21.15.16:0/1806166645 learned_addr learned my addr 172.21.15.16:0/1806166645 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07c7fff640 1 --2- 172.21.15.16:0/1806166645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f99bb60 0x55954fd65380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07cf7fe640 1 --2- 172.21.15.16:0/1806166645 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f9bc840 0x55954fd70ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07ceffd640 1 -- 172.21.15.16:0/1806166645 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f9bc840 msgr2=0x55954fd70ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07ceffd640 1 --2- 172.21.15.16:0/1806166645 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f9bc840 0x55954fd70ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07ceffd640 1 -- 172.21.15.16:0/1806166645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f99bb60 msgr2=0x55954fd65380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07ceffd640 1 --2- 172.21.15.16:0/1806166645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f99bb60 0x55954fd65380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.645+0000 7f07ceffd640 1 -- 172.21.15.16:0/1806166645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55954fc97d90 con 0x55954f993980 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.646+0000 7f07ceffd640 1 --2- 172.21.15.16:0/1806166645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 0x55954fd6f080 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x55954fd85e80 tx=0x7f07c001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1b303e538e5a43 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.646+0000 7f07c57fa640 1 -- 172.21.15.16:0/1806166645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c000fb10 con 0x55954f993980 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.646+0000 7f07c57fa640 1 -- 172.21.15.16:0/1806166645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07c00054d0 con 0x55954f993980 2024-08-28T19:39:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.646+0000 7f07c57fa640 1 -- 172.21.15.16:0/1806166645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c0004c60 con 0x55954f993980 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.647+0000 7f07dbdbc740 1 -- 172.21.15.16:0/1806166645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 msgr2=0x55954fd6f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.647+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/1806166645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 0x55954fd6f080 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x55954fd85e80 tx=0x7f07c001a000 comp rx=0 tx=0).stop 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.647+0000 7f07dbdbc740 1 -- 172.21.15.16:0/1806166645 shutdown_connections 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.647+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/1806166645 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f9bc840 0x55954fd70ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.647+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/1806166645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f99bb60 0x55954fd65380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.647+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/1806166645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f993980 0x55954fd6f080 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.647+0000 7f07dbdbc740 1 -- 172.21.15.16:0/1806166645 >> 172.21.15.16:0/1806166645 conn(0x55954f9b0990 msgr2=0x55954fd64c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.647+0000 7f07dbdbc740 1 -- 172.21.15.16:0/1806166645 shutdown_connections 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.647+0000 7f07dbdbc740 1 -- 172.21.15.16:0/1806166645 wait complete. 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.647+0000 7f07dbdbc740 1 -- start start 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07dbdbc740 1 Processor -- start 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f993980 0x55954fd78a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f99bb60 0x55954fd78f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07dbdbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f9bc840 0x55954fd794c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01afff0 con 0x55954f99bb60 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01af890 con 0x55954f9bc840 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07dbdbc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01aca60 con 0x55954f993980 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07dbdbc740 10 client.0 _ll_register_callbacks cb 0x7f07c0009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f99bb60 0x55954fd78f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f99bb60 0x55954fd78f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37246/0 (socket says 172.21.15.16:37246) 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07c7fff640 1 -- 172.21.15.16:0/3100773004 learned_addr learned my addr 172.21.15.16:0/3100773004 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.648+0000 7f07cf7fe640 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f9bc840 0x55954fd794c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.649+0000 7f07ceffd640 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f993980 0x55954fd78a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.649+0000 7f07c7fff640 1 -- 172.21.15.16:0/3100773004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f993980 msgr2=0x55954fd78a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.649+0000 7f07c7fff640 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f993980 0x55954fd78a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.649+0000 7f07c7fff640 1 -- 172.21.15.16:0/3100773004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f9bc840 msgr2=0x55954fd794c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.649+0000 7f07c7fff640 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f9bc840 0x55954fd794c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.649+0000 7f07c7fff640 1 -- 172.21.15.16:0/3100773004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55954fdf00f0 con 0x55954f99bb60 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.649+0000 7f07c7fff640 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f99bb60 0x55954fd78f80 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f07b80cc0d0 tx=0x7f07b8105f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de18a3b6cf0c4c7 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.649+0000 7f07aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.649+0000 7f07ceffd640 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f993980 0x55954fd78a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.650+0000 7f07aeffd640 1 -- 172.21.15.16:0/3100773004 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8048040 con 0x55954f99bb60 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.650+0000 7f07aeffd640 1 -- 172.21.15.16:0/3100773004 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9490+0+0 (secure 0 0 0) 0x7f07b80cc5b0 con 0x55954f99bb60 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.650+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3100773004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07c00054d0 con 0x55954f99bb60 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.650+0000 7f07aeffd640 1 -- 172.21.15.16:0/3100773004 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f07b80c8000 con 0x55954f99bb60 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.650+0000 7f07dbdbc740 10 client.19914 fetch_fsmap learned FSMap version 99 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.650+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3100773004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x55954fdf1380 con 0x55954f99bb60 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.650+0000 7f07aeffd640 1 -- 172.21.15.16:0/3100773004 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07c00054d0 con 0x55954f99bb60 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.651+0000 7f07aeffd640 1 -- 172.21.15.16:0/3100773004 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 99) ==== 8410+0+0 (secure 0 0 0) 0x7f07b804b060 con 0x55954f99bb60 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.651+0000 7f07dbdbc740 10 client.19914 fetch_fsmap finished waiting for FSMap version 99 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.651+0000 7f07dbdbc740 10 client.19914 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:39:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.651+0000 7f07dbdbc740 20 client.19914 populate_metadata read hostname 'smithi016' 2024-08-28T19:39:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.651+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55954f950570 0x55954fdf3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:39:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.651+0000 7f07dbdbc740 4 client.19914 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.7, "await_for": 45.0}] 2024-08-28T19:39:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.651+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3100773004 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.7, "await_for": 45.0}) -- 0x55954fcd5d20 con 0x55954f950570 2024-08-28T19:39:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.651+0000 7f07ceffd640 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55954f950570 0x55954fdf3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:39:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.652+0000 7f07aeffd640 10 client.19914 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:56.652+0000 7f07ceffd640 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55954f950570 0x55954fdf3c20 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a65c15253572c387 server_cookie=aea2c61c46709a10 in_seq=0 out_seq=0 2024-08-28T19:39:57.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:56 smithi016 ceph-mon[18556]: pgmap v930: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.1 MiB/s wr, 654 op/s 2024-08-28T19:39:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07aeffd640 1 -- 172.21.15.16:0/3100773004 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55954fcd5d20 con 0x55954f950570 2024-08-28T19:39:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07aeffd640 10 client.19914 handle_command_reply: tid=0 2024-08-28T19:39:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 1 client.19914 shutdown 2024-08-28T19:39:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3100773004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f99bb60 msgr2=0x55954fd78f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f99bb60 0x55954fd78f80 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f07b80cc0d0 tx=0x7f07b8105f70 comp rx=0 tx=0).stop 2024-08-28T19:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3100773004 shutdown_connections 2024-08-28T19:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55954f9bc840 0x55954fd794c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55954f99bb60 0x55954fd78f80 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55954f950570 0x55954fdf3c20 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 1 --2- 172.21.15.16:0/3100773004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55954f993980 0x55954fd78a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3100773004 >> 172.21.15.16:0/3100773004 conn(0x55954f9b0990 msgr2=0x55954fd65630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07aeffd640 0 client.19914 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3100773004 shutdown_connections 2024-08-28T19:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 1 -- 172.21.15.16:0/3100773004 wait complete. 2024-08-28T19:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.254+0000 7f07dbdbc740 20 client.19914 trim_cache size 0 max 16384 2024-08-28T19:39:57.257 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:39:57.257 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 99, 2024-08-28T19:39:57.257 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:39:57.257 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 58, 2024-08-28T19:39:57.257 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:39:57.257 INFO:teuthology.orchestra.run.smithi016.stdout: "a87c215e": { 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 58, 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 146.7, 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:39:57.258 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:39:57.259 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:57.259 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:57.259 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:57.259 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:57.259 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:39:57.259 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:39:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 -- 172.21.15.16:0/2845653991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4d20 msgr2=0x7f07bc0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4d20 0x7f07bc0b7230 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55954fd62910 tx=0x7f07c0005f90 comp rx=0 tx=0).stop 2024-08-28T19:39:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 -- 172.21.15.16:0/2845653991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0108160 msgr2=0x7f07d019a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:39:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0108160 0x7f07d019a7a0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f07b802a930 tx=0x7f07b8031a20 comp rx=0 tx=0).stop 2024-08-28T19:39:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 -- 172.21.15.16:0/2845653991 shutdown_connections 2024-08-28T19:39:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07d0108ab0 0x7f07d019fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d0108160 0x7f07d019a7a0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f07bc0b4d20 0x7f07bc0b7230 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 --2- 172.21.15.16:0/2845653991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d0102820 0x7f07d019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:39:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 -- 172.21.15.16:0/2845653991 >> 172.21.15.16:0/2845653991 conn(0x7f07d0079500 msgr2=0x7f07d0101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:39:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 -- 172.21.15.16:0/2845653991 shutdown_connections 2024-08-28T19:39:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:39:57.255+0000 7f07d5847640 1 -- 172.21.15.16:0/2845653991 wait complete. 2024-08-28T19:39:57.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:56 smithi028 ceph-mon[21769]: pgmap v930: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.1 MiB/s wr, 654 op/s 2024-08-28T19:39:57.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:56 smithi115 ceph-mon[22184]: pgmap v930: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.1 MiB/s wr, 654 op/s 2024-08-28T19:39:57.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 24.26 MB/sec execute 213 sec latency 833.003 ms 2024-08-28T19:39:58.076 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a87c215e', quiesce took 5.6 seconds. Will release after: 21.1 2024-08-28T19:39:58.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 24.14 MB/sec execute 214 sec latency 1833.057 ms 2024-08-28T19:39:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:58 smithi016 ceph-mon[18556]: pgmap v931: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 523 op/s 2024-08-28T19:39:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:58 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:39:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:58 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:39:59.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:58 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:39:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:58 smithi028 ceph-mon[21769]: pgmap v931: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 523 op/s 2024-08-28T19:39:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:58 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:39:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:58 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:39:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:58 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:39:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:58 smithi115 ceph-mon[22184]: pgmap v931: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 523 op/s 2024-08-28T19:39:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:58 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:39:59.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:58 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:39:59.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:58 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:39:59.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 24.03 MB/sec execute 215 sec latency 2833.179 ms 2024-08-28T19:40:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:59 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:40:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:59 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:40:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:40:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:40:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:40:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:39:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:40:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:59 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:40:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:59 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:40:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:40:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:40:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:40:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:39:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:40:00.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:59 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:40:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:59 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:40:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:40:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:40:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:40:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:39:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:40:00.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 23.92 MB/sec execute 216 sec latency 3833.220 ms 2024-08-28T19:40:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:00 smithi016 ceph-mon[18556]: pgmap v932: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 MiB/s wr, 573 op/s 2024-08-28T19:40:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:00 smithi016 ceph-mon[18556]: overall HEALTH_OK 2024-08-28T19:40:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:00 smithi028 ceph-mon[21769]: pgmap v932: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 MiB/s wr, 573 op/s 2024-08-28T19:40:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:00 smithi028 ceph-mon[21769]: overall HEALTH_OK 2024-08-28T19:40:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:00 smithi115 ceph-mon[22184]: pgmap v932: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 MiB/s wr, 573 op/s 2024-08-28T19:40:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:00 smithi115 ceph-mon[22184]: overall HEALTH_OK 2024-08-28T19:40:01.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 23.81 MB/sec execute 217 sec latency 4833.297 ms 2024-08-28T19:40:02.481 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 23.70 MB/sec execute 218 sec latency 5833.392 ms 2024-08-28T19:40:03.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:02 smithi016 ceph-mon[18556]: pgmap v933: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 658 op/s 2024-08-28T19:40:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:02 smithi028 ceph-mon[21769]: pgmap v933: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 658 op/s 2024-08-28T19:40:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:02 smithi115 ceph-mon[22184]: pgmap v933: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 658 op/s 2024-08-28T19:40:03.480 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 23.59 MB/sec execute 219 sec latency 6833.455 ms 2024-08-28T19:40:04.481 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 23.49 MB/sec execute 220 sec latency 7833.526 ms 2024-08-28T19:40:05.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:04 smithi016 ceph-mon[18556]: pgmap v934: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 15 MiB/s wr, 605 op/s 2024-08-28T19:40:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:04 smithi028 ceph-mon[21769]: pgmap v934: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 15 MiB/s wr, 605 op/s 2024-08-28T19:40:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:04 smithi115 ceph-mon[22184]: pgmap v934: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 15 MiB/s wr, 605 op/s 2024-08-28T19:40:05.481 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 23.38 MB/sec execute 221 sec latency 8833.651 ms 2024-08-28T19:40:06.481 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 23.27 MB/sec execute 222 sec latency 9833.756 ms 2024-08-28T19:40:07.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:06 smithi016 ceph-mon[18556]: pgmap v935: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 15 MiB/s wr, 664 op/s 2024-08-28T19:40:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:06 smithi028 ceph-mon[21769]: pgmap v935: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 15 MiB/s wr, 664 op/s 2024-08-28T19:40:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:06 smithi115 ceph-mon[22184]: pgmap v935: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 15 MiB/s wr, 664 op/s 2024-08-28T19:40:07.481 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 23.17 MB/sec execute 223 sec latency 10833.858 ms 2024-08-28T19:40:08.481 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 23.07 MB/sec execute 224 sec latency 11833.929 ms 2024-08-28T19:40:09.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:08 smithi016 ceph-mon[18556]: pgmap v936: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 12 MiB/s wr, 485 op/s 2024-08-28T19:40:09.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:08 smithi028 ceph-mon[21769]: pgmap v936: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 12 MiB/s wr, 485 op/s 2024-08-28T19:40:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:08 smithi115 ceph-mon[22184]: pgmap v936: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 12 MiB/s wr, 485 op/s 2024-08-28T19:40:09.481 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 22.96 MB/sec execute 225 sec latency 12834.031 ms 2024-08-28T19:40:10.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:09 smithi016 ceph-mon[18556]: osdmap e109: 12 total, 12 up, 12 in 2024-08-28T19:40:10.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:09 smithi028 ceph-mon[21769]: osdmap e109: 12 total, 12 up, 12 in 2024-08-28T19:40:10.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:09 smithi115 ceph-mon[22184]: osdmap e109: 12 total, 12 up, 12 in 2024-08-28T19:40:10.481 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 22.86 MB/sec execute 226 sec latency 13834.146 ms 2024-08-28T19:40:11.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:10 smithi016 ceph-mon[18556]: pgmap v938: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.1 MiB/s wr, 387 op/s 2024-08-28T19:40:11.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:10 smithi028 ceph-mon[21769]: pgmap v938: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.1 MiB/s wr, 387 op/s 2024-08-28T19:40:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:10 smithi115 ceph-mon[22184]: pgmap v938: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.1 MiB/s wr, 387 op/s 2024-08-28T19:40:11.481 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 22.76 MB/sec execute 227 sec latency 14834.226 ms 2024-08-28T19:40:12.481 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 22.66 MB/sec execute 228 sec latency 15834.335 ms 2024-08-28T19:40:13.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:12 smithi016 ceph-mon[18556]: pgmap v939: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 228 op/s 2024-08-28T19:40:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:12 smithi028 ceph-mon[21769]: pgmap v939: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 228 op/s 2024-08-28T19:40:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:12 smithi115 ceph-mon[22184]: pgmap v939: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 228 op/s 2024-08-28T19:40:13.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 22.56 MB/sec execute 229 sec latency 16834.443 ms 2024-08-28T19:40:14.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 22.47 MB/sec execute 230 sec latency 17834.549 ms 2024-08-28T19:40:15.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:14 smithi016 ceph-mon[18556]: pgmap v940: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 459 KiB/s wr, 119 op/s 2024-08-28T19:40:15.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:14 smithi028 ceph-mon[21769]: pgmap v940: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 459 KiB/s wr, 119 op/s 2024-08-28T19:40:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:14 smithi115 ceph-mon[22184]: pgmap v940: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 459 KiB/s wr, 119 op/s 2024-08-28T19:40:15.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 22.37 MB/sec execute 231 sec latency 18834.605 ms 2024-08-28T19:40:16.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:15 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:40:16.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:15 smithi016 ceph-mon[18556]: osdmap e110: 12 total, 12 up, 12 in 2024-08-28T19:40:16.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:15 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:40:16.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:15 smithi028 ceph-mon[21769]: osdmap e110: 12 total, 12 up, 12 in 2024-08-28T19:40:16.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:15 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:40:16.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:15 smithi115 ceph-mon[22184]: osdmap e110: 12 total, 12 up, 12 in 2024-08-28T19:40:16.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 22.27 MB/sec execute 232 sec latency 19834.650 ms 2024-08-28T19:40:17.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 22.18 MB/sec execute 233 sec latency 20834.699 ms 2024-08-28T19:40:17.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:17 smithi016 ceph-mon[18556]: pgmap v942: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 34 op/s 2024-08-28T19:40:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:17 smithi028 ceph-mon[21769]: pgmap v942: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 34 op/s 2024-08-28T19:40:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:17 smithi115 ceph-mon[22184]: pgmap v942: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 34 op/s 2024-08-28T19:40:18.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 22.08 MB/sec execute 234 sec latency 21834.800 ms 2024-08-28T19:40:19.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:40:19.444 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:40:19.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 21.99 MB/sec execute 235 sec latency 22834.921 ms 2024-08-28T19:40:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:19 smithi016 ceph-mon[18556]: pgmap v943: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 58 KiB/s wr, 22 op/s 2024-08-28T19:40:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:19 smithi028 ceph-mon[21769]: pgmap v943: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 58 KiB/s wr, 22 op/s 2024-08-28T19:40:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:19 smithi115 ceph-mon[22184]: pgmap v943: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 58 KiB/s wr, 22 op/s 2024-08-28T19:40:19.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.913+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1741988552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe268102880 msgr2=0x7fe268102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.913+0000 7fe26e7d0640 1 --2- 172.21.15.16:0/1741988552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe268102880 0x7fe268102c90 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fe258009900 tx=0x7fe25802a410 comp rx=0 tx=0).stop 2024-08-28T19:40:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.913+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1741988552 shutdown_connections 2024-08-28T19:40:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.913+0000 7fe26e7d0640 1 --2- 172.21.15.16:0/1741988552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe268108b10 0x7fe26810f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.913+0000 7fe26e7d0640 1 --2- 172.21.15.16:0/1741988552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe2681081c0 0x7fe2681085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.913+0000 7fe26e7d0640 1 --2- 172.21.15.16:0/1741988552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe268102880 0x7fe268102c90 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.913+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1741988552 >> 172.21.15.16:0/1741988552 conn(0x7fe268079510 msgr2=0x7fe268079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.913+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1741988552 shutdown_connections 2024-08-28T19:40:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.913+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1741988552 wait complete. 2024-08-28T19:40:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.914+0000 7fe26e7d0640 1 Processor -- start 2024-08-28T19:40:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.914+0000 7fe26e7d0640 1 -- start start 2024-08-28T19:40:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe26e7d0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe268102880 0x7fe268199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe26e7d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2681081c0 0x7fe268199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe26e7d0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe268108b10 0x7fe26819e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe26e7d0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2681127c0 con 0x7fe268102880 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe26e7d0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe268112640 con 0x7fe2681081c0 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe26e7d0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe268112940 con 0x7fe268108b10 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe267fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe268102880 0x7fe268199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe267fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe268102880 0x7fe268199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43392/0 (socket says 172.21.15.16:43392) 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe267fff640 1 -- 172.21.15.16:0/1155273568 learned_addr learned my addr 172.21.15.16:0/1155273568 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe2677fe640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2681081c0 0x7fe268199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe26cd6d640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe268108b10 0x7fe26819e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe267fff640 1 -- 172.21.15.16:0/1155273568 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe268108b10 msgr2=0x7fe26819e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe267fff640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe268108b10 0x7fe26819e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe267fff640 1 -- 172.21.15.16:0/1155273568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2681081c0 msgr2=0x7fe268199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe267fff640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2681081c0 0x7fe268199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe267fff640 1 -- 172.21.15.16:0/1155273568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26819f100 con 0x7fe268102880 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe267fff640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe268102880 0x7fe268199190 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fe258005940 tx=0x7fe258004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe2657fa640 1 -- 172.21.15.16:0/1155273568 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25803f030 con 0x7fe268102880 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1155273568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe26819f390 con 0x7fe268102880 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe2657fa640 1 -- 172.21.15.16:0/1155273568 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe258003aa0 con 0x7fe268102880 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.915+0000 7fe2657fa640 1 -- 172.21.15.16:0/1155273568 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258035070 con 0x7fe268102880 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.916+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1155273568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2681a5b50 con 0x7fe268102880 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.917+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1155273568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe268103b00 con 0x7fe268102880 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.918+0000 7fe2657fa640 1 -- 172.21.15.16:0/1155273568 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe258042030 con 0x7fe268102880 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.919+0000 7fe2657fa640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2500b4ca0 0x7fe2500b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.919+0000 7fe2657fa640 1 -- 172.21.15.16:0/1155273568 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fe2581062d0 con 0x7fe268102880 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.919+0000 7fe2677fe640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2500b4ca0 0x7fe2500b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.920+0000 7fe2677fe640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2500b4ca0 0x7fe2500b71b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe268105250 tx=0x7fe25c00b410 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:19.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:19.922+0000 7fe2657fa640 1 -- 172.21.15.16:0/1155273568 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fe2580ccc90 con 0x7fe268102880 2024-08-28T19:40:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.143+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1155273568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe268102c90 con 0x7fe268102880 2024-08-28T19:40:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.144+0000 7fe2657fa640 1 -- 172.21.15.16:0/1155273568 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+11994 (secure 0 0 0) 0x7fe2580cc590 con 0x7fe268102880 2024-08-28T19:40:20.146 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:40:20.146 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":100,"btime":"2024-08-28T19:40:14:986386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":100,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:40:14.585232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:40:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 100 2024-08-28T19:40:20.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1155273568 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2500b4ca0 msgr2=0x7fe2500b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2500b4ca0 0x7fe2500b71b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe268105250 tx=0x7fe25c00b410 comp rx=0 tx=0).stop 2024-08-28T19:40:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1155273568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe268102880 msgr2=0x7fe268199190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe268102880 0x7fe268199190 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fe258005940 tx=0x7fe258004270 comp rx=0 tx=0).stop 2024-08-28T19:40:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1155273568 shutdown_connections 2024-08-28T19:40:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe268108b10 0x7fe26819e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe2500b4ca0 0x7fe2500b71b0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2681081c0 0x7fe268199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 --2- 172.21.15.16:0/1155273568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe268102880 0x7fe268199190 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1155273568 >> 172.21.15.16:0/1155273568 conn(0x7fe268079510 msgr2=0x7fe268106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1155273568 shutdown_connections 2024-08-28T19:40:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:20.146+0000 7fe26e7d0640 1 -- 172.21.15.16:0/1155273568 wait complete. 2024-08-28T19:40:20.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 21.89 MB/sec execute 236 sec latency 23834.971 ms 2024-08-28T19:40:20.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:20 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1155273568' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:40:20.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:20 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1155273568' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:40:20.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:20 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1155273568' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:40:20.918 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id a87c215e --release --await' 2024-08-28T19:40:20.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id a87c215e --release --await 2024-08-28T19:40:21.172 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:40:21.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134049 21.80 MB/sec execute 237 sec latency 24835.061 ms 2024-08-28T19:40:21.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:40:21.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:21 smithi016 ceph-mon[18556]: pgmap v944: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 18 op/s 2024-08-28T19:40:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:21 smithi028 ceph-mon[21769]: pgmap v944: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 18 op/s 2024-08-28T19:40:21.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:21 smithi115 ceph-mon[22184]: pgmap v944: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 18 op/s 2024-08-28T19:40:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.023+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1650592879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc0108140 msgr2=0x7f2bc0108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.023+0000 7f2bc6a76640 1 --2- 172.21.15.16:0/1650592879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc0108140 0x7f2bc0108550 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0009900 tx=0x7f2bb002a410 comp rx=0 tx=0).stop 2024-08-28T19:40:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.023+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1650592879 shutdown_connections 2024-08-28T19:40:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.023+0000 7f2bc6a76640 1 --2- 172.21.15.16:0/1650592879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0108a90 0x7f2bc010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.023+0000 7f2bc6a76640 1 --2- 172.21.15.16:0/1650592879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc0108140 0x7f2bc0108550 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.023+0000 7f2bc6a76640 1 --2- 172.21.15.16:0/1650592879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bc0102800 0x7f2bc0102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.023+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1650592879 >> 172.21.15.16:0/1650592879 conn(0x7f2bc00794d0 msgr2=0x7f2bc00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.024+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1650592879 shutdown_connections 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.024+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1650592879 wait complete. 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.024+0000 7f2bc6a76640 1 Processor -- start 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.024+0000 7f2bc6a76640 1 -- start start 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.024+0000 7f2bc6a76640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bc0102800 0x7f2bc0104e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bc6a76640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc0108140 0x7f2bc01053c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bc6a76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0108a90 0x7f2bc0105900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bc6a76640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc01126e0 con 0x7f2bc0108140 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bc6a76640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc0112560 con 0x7f2bc0108a90 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bc6a76640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc0112860 con 0x7f2bc0102800 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc0108140 0x7f2bc01053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc0108140 0x7f2bc01053c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43410/0 (socket says 172.21.15.16:43410) 2024-08-28T19:40:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb7fff640 1 -- 172.21.15.16:0/1274680314 learned_addr learned my addr 172.21.15.16:0/1274680314 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bc4812640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bc0102800 0x7f2bc0104e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bc5013640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0108a90 0x7f2bc0105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb7fff640 1 -- 172.21.15.16:0/1274680314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bc0102800 msgr2=0x7f2bc0104e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb7fff640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bc0102800 0x7f2bc0104e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb7fff640 1 -- 172.21.15.16:0/1274680314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0108a90 msgr2=0x7f2bc0105900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb7fff640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0108a90 0x7f2bc0105900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb7fff640 1 -- 172.21.15.16:0/1274680314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc01a84d0 con 0x7f2bc0108140 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb7fff640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc0108140 0x7f2bc01053c0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0006010 tx=0x7f2bb0004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb5ffb640 1 -- 172.21.15.16:0/1274680314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb003f030 con 0x7f2bc0108140 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1274680314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc01a8760 con 0x7f2bc0108140 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb5ffb640 1 -- 172.21.15.16:0/1274680314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bb0003aa0 con 0x7f2bc0108140 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.025+0000 7f2bb5ffb640 1 -- 172.21.15.16:0/1274680314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0035070 con 0x7f2bc0108140 2024-08-28T19:40:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.026+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1274680314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc01a8c90 con 0x7f2bc0108140 2024-08-28T19:40:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.029+0000 7f2bb5ffb640 1 -- 172.21.15.16:0/1274680314 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2bb0042030 con 0x7f2bc0108140 2024-08-28T19:40:22.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.030+0000 7f2bccfeb740 1 Processor -- start 2024-08-28T19:40:22.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.030+0000 7f2bccfeb740 1 -- start start 2024-08-28T19:40:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.030+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2cdb9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.030+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 0x560e2cdba1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.030+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 0x560e2cdba700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560e2cd30a70 con 0x560e2c9f3840 2024-08-28T19:40:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560e2cd20000 con 0x560e2ca14520 2024-08-28T19:40:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560e2cd23fb0 con 0x560e2c9eb660 2024-08-28T19:40:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bc5013640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2cdb9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bc4812640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 0x560e2cdba1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bc5013640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2cdb9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34872/0 (socket says 172.21.15.16:34872) 2024-08-28T19:40:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bc5013640 1 -- 172.21.15.16:0/1082611660 learned_addr learned my addr 172.21.15.16:0/1082611660 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bb5ffb640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ba80b4d20 0x7f2ba80b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bb5ffb640 1 -- 172.21.15.16:0/1274680314 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7f2bb0105c90 con 0x7f2bc0108140 2024-08-28T19:40:22.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bb7fff640 1 --2- 172.21.15.16:0/1082611660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 0x560e2cdba700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bc4812640 1 -- 172.21.15.16:0/1082611660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 msgr2=0x560e2cdb9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bc4812640 1 --2- 172.21.15.16:0/1082611660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2cdb9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bc5013640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ba80b4d20 0x7f2ba80b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bc4812640 1 -- 172.21.15.16:0/1082611660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 msgr2=0x560e2cdba700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bc4812640 1 --2- 172.21.15.16:0/1082611660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 0x560e2cdba700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bc4812640 1 -- 172.21.15.16:0/1082611660 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e2ca7fb00 con 0x560e2c9eb660 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.031+0000 7f2bb7fff640 1 --2- 172.21.15.16:0/1082611660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 0x560e2cdba700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.032+0000 7f2bc4812640 1 --2- 172.21.15.16:0/1082611660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 0x560e2cdba1c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005fd0 tx=0x7f2bbc00bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66bcb4b943089a1a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.032+0000 7f2bc5013640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ba80b4d20 0x7f2ba80b7230 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8003b10 tx=0x7f2bb8004ab0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.032+0000 7f2ba6ffd640 1 -- 172.21.15.16:0/1082611660 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc003bd0 con 0x560e2c9eb660 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.032+0000 7f2ba6ffd640 1 -- 172.21.15.16:0/1082611660 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bbc015070 con 0x560e2c9eb660 2024-08-28T19:40:22.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.032+0000 7f2ba6ffd640 1 -- 172.21.15.16:0/1082611660 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc004d60 con 0x560e2c9eb660 2024-08-28T19:40:22.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.033+0000 7f2bccfeb740 1 -- 172.21.15.16:0/1082611660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 msgr2=0x560e2cdba1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.033+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/1082611660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 0x560e2cdba1c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005fd0 tx=0x7f2bbc00bb20 comp rx=0 tx=0).stop 2024-08-28T19:40:22.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.033+0000 7f2bccfeb740 1 -- 172.21.15.16:0/1082611660 shutdown_connections 2024-08-28T19:40:22.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.033+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/1082611660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 0x560e2cdba700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.033+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/1082611660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 0x560e2cdba1c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.033+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/1082611660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2cdb9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.033+0000 7f2bccfeb740 1 -- 172.21.15.16:0/1082611660 >> 172.21.15.16:0/1082611660 conn(0x560e2ca08910 msgr2=0x560e2cdb7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.033+0000 7f2bccfeb740 1 -- 172.21.15.16:0/1082611660 shutdown_connections 2024-08-28T19:40:22.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.033+0000 7f2bccfeb740 1 -- 172.21.15.16:0/1082611660 wait complete. 2024-08-28T19:40:22.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.033+0000 7f2bccfeb740 1 -- start start 2024-08-28T19:40:22.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bccfeb740 1 Processor -- start 2024-08-28T19:40:22.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 0x560e2cdc5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 0x560e2cdc5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2cdd18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560e2cdc8460 con 0x560e2c9f3840 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560e2cce1520 con 0x560e2ca14520 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560e2cdc85e0 con 0x560e2c9eb660 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bccfeb740 10 client.0 _ll_register_callbacks cb 0x560e2cd19c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bc4812640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 0x560e2cdc5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bb7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 0x560e2cdc5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bb7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 0x560e2cdc5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43428/0 (socket says 172.21.15.16:43428) 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.034+0000 7f2bb7fff640 1 -- 172.21.15.16:0/834025009 learned_addr learned my addr 172.21.15.16:0/834025009 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.035+0000 7f2bc5013640 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2cdd18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.035+0000 7f2bc4812640 1 -- 172.21.15.16:0/834025009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 msgr2=0x560e2cdd18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.035+0000 7f2bc4812640 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2cdd18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.035+0000 7f2bc4812640 1 -- 172.21.15.16:0/834025009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 msgr2=0x560e2cdc5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.035+0000 7f2bc4812640 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 0x560e2cdc5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.035+0000 7f2bc4812640 1 -- 172.21.15.16:0/834025009 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e2caf6960 con 0x560e2c9eb660 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.035+0000 7f2bc4812640 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 0x560e2cdc5840 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc004650 tx=0x7f2bbc00bfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca38a570f10c0595 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.035+0000 7f2b9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.035+0000 7f2b9ffff640 1 -- 172.21.15.16:0/834025009 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc003e60 con 0x560e2c9eb660 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.035+0000 7f2bccfeb740 1 -- 172.21.15.16:0/834025009 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560e2ccbf9a0 con 0x560e2c9eb660 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2b9ffff640 1 -- 172.21.15.16:0/834025009 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7f2bbc011440 con 0x560e2c9eb660 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2b9ffff640 1 -- 172.21.15.16:0/834025009 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f2bbc011640 con 0x560e2c9eb660 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2bccfeb740 10 client.28945 fetch_fsmap learned FSMap version 100 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2bccfeb740 1 -- 172.21.15.16:0/834025009 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x560e2cdd4570 con 0x560e2c9eb660 2024-08-28T19:40:22.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2b9ffff640 1 -- 172.21.15.16:0/834025009 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560e2ccbf9a0 con 0x560e2c9eb660 2024-08-28T19:40:22.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2b9ffff640 1 -- 172.21.15.16:0/834025009 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 100) ==== 8406+0+0 (secure 0 0 0) 0x7f2bbc0258a0 con 0x560e2c9eb660 2024-08-28T19:40:22.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2bccfeb740 10 client.28945 fetch_fsmap finished waiting for FSMap version 100 2024-08-28T19:40:22.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2bccfeb740 10 client.28945 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:40:22.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2bccfeb740 20 client.28945 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:22.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e2c9d2ea0 0x560e2cdd7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2bccfeb740 4 client.28945 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:40:22.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.036+0000 7f2bccfeb740 1 -- 172.21.15.16:0/834025009 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e2cd329c0 con 0x560e2c9d2ea0 2024-08-28T19:40:22.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.037+0000 7f2bb7fff640 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e2c9d2ea0 0x560e2cdd7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.037+0000 7f2bb7fff640 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e2c9d2ea0 0x560e2cdd7930 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f57a9cb033a36a75 server_cookie=3a47f4ae2b02c506 in_seq=0 out_seq=0 2024-08-28T19:40:22.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.037+0000 7f2b9ffff640 10 client.28945 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:22.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2b9ffff640 1 -- 172.21.15.16:0/834025009 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560e2cd329c0 con 0x560e2c9d2ea0 2024-08-28T19:40:22.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2b9ffff640 10 client.28945 handle_command_reply: tid=0 2024-08-28T19:40:22.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 1 client.28945 shutdown 2024-08-28T19:40:22.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 1 -- 172.21.15.16:0/834025009 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 msgr2=0x560e2cdc5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 0x560e2cdc5840 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc004650 tx=0x7f2bbc00bfa0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 1 -- 172.21.15.16:0/834025009 shutdown_connections 2024-08-28T19:40:22.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2cdd18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9f3840 0x560e2cdc5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e2c9d2ea0 0x560e2cdd7930 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/834025009 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9eb660 0x560e2cdc5840 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2b9ffff640 0 client.28945 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:22.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 1 -- 172.21.15.16:0/834025009 >> 172.21.15.16:0/834025009 conn(0x560e2ca08910 msgr2=0x560e2cdbce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 1 -- 172.21.15.16:0/834025009 shutdown_connections 2024-08-28T19:40:22.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 1 -- 172.21.15.16:0/834025009 wait complete. 2024-08-28T19:40:22.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.038+0000 7f2bccfeb740 20 client.28945 trim_cache size 0 max 16384 2024-08-28T19:40:22.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.065+0000 7f2bccfeb740 1 Processor -- start 2024-08-28T19:40:22.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.065+0000 7f2bccfeb740 1 -- start start 2024-08-28T19:40:22.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.065+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdbb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.065+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c9f3840 0x560e2cdc3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2ca14520 0x560e2cdb7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc01b2ec0 con 0x560e2c9eb660 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc01b4bb0 con 0x560e2c9f3840 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc01b3620 con 0x560e2ca14520 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bc4812640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdbb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bc4812640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdbb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43436/0 (socket says 172.21.15.16:43436) 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bc4812640 1 -- 172.21.15.16:0/189208916 learned_addr learned my addr 172.21.15.16:0/189208916 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bb7fff640 1 --2- 172.21.15.16:0/189208916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c9f3840 0x560e2cdc3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bc5013640 1 --2- 172.21.15.16:0/189208916 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2ca14520 0x560e2cdb7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bc4812640 1 -- 172.21.15.16:0/189208916 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2ca14520 msgr2=0x560e2cdb7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bc4812640 1 --2- 172.21.15.16:0/189208916 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2ca14520 0x560e2cdb7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bc4812640 1 -- 172.21.15.16:0/189208916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c9f3840 msgr2=0x560e2cdc3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bc4812640 1 --2- 172.21.15.16:0/189208916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c9f3840 0x560e2cdc3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bc4812640 1 -- 172.21.15.16:0/189208916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e2cdc5f00 con 0x560e2c9eb660 2024-08-28T19:40:22.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2bc4812640 1 --2- 172.21.15.16:0/189208916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdbb670 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00b620 tx=0x7f2bbc00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae32f8a8effd1bba server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.066+0000 7f2ba67fc640 1 -- 172.21.15.16:0/189208916 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc011440 con 0x560e2c9eb660 2024-08-28T19:40:22.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.067+0000 7f2ba67fc640 1 -- 172.21.15.16:0/189208916 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bbc015050 con 0x560e2c9eb660 2024-08-28T19:40:22.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.067+0000 7f2ba67fc640 1 -- 172.21.15.16:0/189208916 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc025950 con 0x560e2c9eb660 2024-08-28T19:40:22.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.067+0000 7f2bccfeb740 1 -- 172.21.15.16:0/189208916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 msgr2=0x560e2cdbb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.067+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/189208916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdbb670 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00b620 tx=0x7f2bbc00b5b0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.068+0000 7f2bccfeb740 1 -- 172.21.15.16:0/189208916 shutdown_connections 2024-08-28T19:40:22.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.068+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/189208916 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2ca14520 0x560e2cdb7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.068+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/189208916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c9f3840 0x560e2cdc3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.068+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/189208916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdbb670 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.068+0000 7f2bccfeb740 1 -- 172.21.15.16:0/189208916 >> 172.21.15.16:0/189208916 conn(0x560e2ca08910 msgr2=0x560e2cdc07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.068+0000 7f2bccfeb740 1 -- 172.21.15.16:0/189208916 shutdown_connections 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.068+0000 7f2bccfeb740 1 -- 172.21.15.16:0/189208916 wait complete. 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.068+0000 7f2bccfeb740 1 -- start start 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.068+0000 7f2bccfeb740 1 Processor -- start 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9f3840 0x560e2cdca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bccfeb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2ce43b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc01b4420 con 0x560e2c9eb660 2024-08-28T19:40:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc01b3cf0 con 0x560e2ca14520 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bccfeb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc01b0ec0 con 0x560e2c9f3840 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bccfeb740 10 client.0 _ll_register_callbacks cb 0x7f2bb8009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bc4812640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bc4812640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43446/0 (socket says 172.21.15.16:43446) 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bc4812640 1 -- 172.21.15.16:0/340003288 learned_addr learned my addr 172.21.15.16:0/340003288 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bb7fff640 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9f3840 0x560e2cdca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.069+0000 7f2bc5013640 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2ce43b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2bc4812640 1 -- 172.21.15.16:0/340003288 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9f3840 msgr2=0x560e2cdca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2bc4812640 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9f3840 0x560e2cdca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2bc4812640 1 -- 172.21.15.16:0/340003288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 msgr2=0x560e2ce43b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2bc4812640 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2ce43b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2bc4812640 1 -- 172.21.15.16:0/340003288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e2ce44180 con 0x560e2c9eb660 2024-08-28T19:40:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2bc4812640 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdca290 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005880 tx=0x7f2bbc003be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c813adcde9294c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2b9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2b9ffff640 1 -- 172.21.15.16:0/340003288 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc03f020 con 0x560e2c9eb660 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2b9ffff640 1 -- 172.21.15.16:0/340003288 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7f2bbc011440 con 0x560e2c9eb660 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2bccfeb740 1 -- 172.21.15.16:0/340003288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bbc015050 con 0x560e2c9eb660 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2b9ffff640 1 -- 172.21.15.16:0/340003288 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f2bbc004a20 con 0x560e2c9eb660 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2bccfeb740 10 client.19935 fetch_fsmap learned FSMap version 100 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.070+0000 7f2bccfeb740 1 -- 172.21.15.16:0/340003288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x560e2ce4b590 con 0x560e2c9eb660 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.071+0000 7f2b9ffff640 1 -- 172.21.15.16:0/340003288 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bbc015050 con 0x560e2c9eb660 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.071+0000 7f2b9ffff640 1 -- 172.21.15.16:0/340003288 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 100) ==== 8406+0+0 (secure 0 0 0) 0x7f2bbc005030 con 0x560e2c9eb660 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.071+0000 7f2bccfeb740 10 client.19935 fetch_fsmap finished waiting for FSMap version 100 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.071+0000 7f2bccfeb740 10 client.19935 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.071+0000 7f2bccfeb740 20 client.19935 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.071+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e2c9a8250 0x560e2ce4db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.071+0000 7f2bccfeb740 4 client.19935 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a87c215e", "release": true, "await": true}] 2024-08-28T19:40:22.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.071+0000 7f2bccfeb740 1 -- 172.21.15.16:0/340003288 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a87c215e", "release": true, "await": true}) -- 0x560e2cd2e460 con 0x560e2c9a8250 2024-08-28T19:40:22.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.071+0000 7f2bb7fff640 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e2c9a8250 0x560e2ce4db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.072+0000 7f2b9ffff640 10 client.19935 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:22.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.072+0000 7f2bb7fff640 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e2c9a8250 0x560e2ce4db60 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d5f305413404a3e server_cookie=5b1812b32688a66d in_seq=0 out_seq=0 2024-08-28T19:40:22.077 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:40:22.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2b9ffff640 1 -- 172.21.15.16:0/340003288 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+592 (crc 0 0 0) 0x560e2cd2e460 con 0x560e2c9a8250 2024-08-28T19:40:22.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2b9ffff640 10 client.19935 handle_command_reply: tid=0 2024-08-28T19:40:22.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 1 client.19935 shutdown 2024-08-28T19:40:22.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 1 -- 172.21.15.16:0/340003288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 msgr2=0x560e2cdca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdca290 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005880 tx=0x7f2bbc003be0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 1 -- 172.21.15.16:0/340003288 shutdown_connections 2024-08-28T19:40:22.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2ca14520 0x560e2ce43b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560e2c9a8250 0x560e2ce4db60 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e2c9f3840 0x560e2cdca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 1 --2- 172.21.15.16:0/340003288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e2c9eb660 0x560e2cdca290 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 1 -- 172.21.15.16:0/340003288 >> 172.21.15.16:0/340003288 conn(0x560e2ca08910 msgr2=0x560e2cdc3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2b9ffff640 0 client.19935 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:22.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 1 -- 172.21.15.16:0/340003288 shutdown_connections 2024-08-28T19:40:22.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 1 -- 172.21.15.16:0/340003288 wait complete. 2024-08-28T19:40:22.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.111+0000 7f2bccfeb740 20 client.19935 trim_cache size 0 max 16384 2024-08-28T19:40:22.114 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:40:22.114 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 100, 2024-08-28T19:40:22.114 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 60, 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "a87c215e": { 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 60, 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 146.7, 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:40:22.115 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:40:22.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:40:22.116 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:40:22.116 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:40:22.116 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:40:22.116 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:40:22.116 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:40:22.116 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:40:22.116 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:40:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1274680314 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ba80b4d20 msgr2=0x7f2ba80b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ba80b4d20 0x7f2ba80b7230 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8003b10 tx=0x7f2bb8004ab0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1274680314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc0108140 msgr2=0x7f2bc01053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc0108140 0x7f2bc01053c0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0006010 tx=0x7f2bb0004060 comp rx=0 tx=0).stop 2024-08-28T19:40:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1274680314 shutdown_connections 2024-08-28T19:40:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2ba80b4d20 0x7f2ba80b7230 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0108a90 0x7f2bc0105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc0108140 0x7f2bc01053c0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 --2- 172.21.15.16:0/1274680314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bc0102800 0x7f2bc0104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1274680314 >> 172.21.15.16:0/1274680314 conn(0x7f2bc00794d0 msgr2=0x7f2bc006b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1274680314 shutdown_connections 2024-08-28T19:40:22.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.112+0000 7f2bc6a76640 1 -- 172.21.15.16:0/1274680314 wait complete. 2024-08-28T19:40:22.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134317 21.73 MB/sec execute 238 sec latency 25441.191 ms 2024-08-28T19:40:22.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.725+0000 7f346d68b640 1 -- 172.21.15.16:0/2951685824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f346810dbb0 msgr2=0x7f34681100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.725+0000 7f346d68b640 1 --2- 172.21.15.16:0/2951685824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f346810dbb0 0x7f34681100a0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f3460009ca0 tx=0x7f346002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.725+0000 7f346d68b640 1 -- 172.21.15.16:0/2951685824 shutdown_connections 2024-08-28T19:40:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.725+0000 7f346d68b640 1 --2- 172.21.15.16:0/2951685824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f346810dbb0 0x7f34681100a0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.725+0000 7f346d68b640 1 --2- 172.21.15.16:0/2951685824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34681050a0 0x7f346810d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.725+0000 7f346d68b640 1 --2- 172.21.15.16:0/2951685824 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3468104750 0x7f3468104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.725+0000 7f346d68b640 1 -- 172.21.15.16:0/2951685824 >> 172.21.15.16:0/2951685824 conn(0x7f34680794d0 msgr2=0x7f34680798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.725+0000 7f346d68b640 1 -- 172.21.15.16:0/2951685824 shutdown_connections 2024-08-28T19:40:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.725+0000 7f346d68b640 1 -- 172.21.15.16:0/2951685824 wait complete. 2024-08-28T19:40:22.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.726+0000 7f346d68b640 1 Processor -- start 2024-08-28T19:40:22.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.726+0000 7f346d68b640 1 -- start start 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f346d68b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3468104750 0x7f346819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f346d68b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34681050a0 0x7f346819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f346d68b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f346810dbb0 0x7f346819fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f346d68b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34681138b0 con 0x7f34681050a0 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f346d68b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3468113730 con 0x7f346810dbb0 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f346d68b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3468113a30 con 0x7f3468104750 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f3466ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3468104750 0x7f346819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f34667fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34681050a0 0x7f346819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f3466ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3468104750 0x7f346819a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:51180/0 (socket says 172.21.15.16:51180) 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f3466ffd640 1 -- 172.21.15.16:0/1707713735 learned_addr learned my addr 172.21.15.16:0/1707713735 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f34667fc640 1 -- 172.21.15.16:0/1707713735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3468104750 msgr2=0x7f346819a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f34667fc640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3468104750 0x7f346819a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f34667fc640 1 -- 172.21.15.16:0/1707713735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f346810dbb0 msgr2=0x7f346819fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f34667fc640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f346810dbb0 0x7f346819fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.727+0000 7f34667fc640 1 -- 172.21.15.16:0/1707713735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34681a0270 con 0x7f34681050a0 2024-08-28T19:40:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.728+0000 7f3466ffd640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3468104750 0x7f346819a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:40:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.728+0000 7f34667fc640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34681050a0 0x7f346819a8a0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f345c005fd0 tx=0x7f345c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.728+0000 7f344ffff640 1 -- 172.21.15.16:0/1707713735 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c009c00 con 0x7f34681050a0 2024-08-28T19:40:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.728+0000 7f344ffff640 1 -- 172.21.15.16:0/1707713735 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f345c009da0 con 0x7f34681050a0 2024-08-28T19:40:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.728+0000 7f344ffff640 1 -- 172.21.15.16:0/1707713735 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c003e10 con 0x7f34681050a0 2024-08-28T19:40:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.728+0000 7f346d68b640 1 -- 172.21.15.16:0/1707713735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34681a0560 con 0x7f34681050a0 2024-08-28T19:40:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.728+0000 7f346d68b640 1 -- 172.21.15.16:0/1707713735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3468102370 con 0x7f34681050a0 2024-08-28T19:40:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.731+0000 7f3473c00740 1 Processor -- start 2024-08-28T19:40:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.731+0000 7f3473c00740 1 -- start start 2024-08-28T19:40:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.731+0000 7f344ffff640 1 -- 172.21.15.16:0/1707713735 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f345c004030 con 0x7f34681050a0 2024-08-28T19:40:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.732+0000 7f3473c00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687330070 0x55e6876d52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.732+0000 7f3473c00740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 0x55e6876d5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.732+0000 7f3473c00740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e68730f390 0x55e6876d5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.732+0000 7f3473c00740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e68764c630 con 0x55e6873071b0 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.732+0000 7f3473c00740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e68763bbc0 con 0x55e687330070 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.732+0000 7f3473c00740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e68763fb70 con 0x55e68730f390 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.732+0000 7f34677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687330070 0x55e6876d52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.732+0000 7f34677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687330070 0x55e6876d52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34928/0 (socket says 172.21.15.16:34928) 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.732+0000 7f34677fe640 1 -- 172.21.15.16:0/2213785674 learned_addr learned my addr 172.21.15.16:0/2213785674 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.732+0000 7f3466ffd640 1 --2- 172.21.15.16:0/2213785674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 0x55e6876d5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.732+0000 7f34667fc640 1 --2- 172.21.15.16:0/2213785674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e68730f390 0x55e6876d5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f34677fe640 1 -- 172.21.15.16:0/2213785674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e68730f390 msgr2=0x55e6876d5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f34677fe640 1 --2- 172.21.15.16:0/2213785674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e68730f390 0x55e6876d5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f34677fe640 1 -- 172.21.15.16:0/2213785674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 msgr2=0x55e6876d5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f34677fe640 1 --2- 172.21.15.16:0/2213785674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 0x55e6876d5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f34677fe640 1 -- 172.21.15.16:0/2213785674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e68739b650 con 0x55e687330070 2024-08-28T19:40:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f34667fc640 1 --2- 172.21.15.16:0/2213785674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e68730f390 0x55e6876d5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:40:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f3466ffd640 1 --2- 172.21.15.16:0/2213785674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 0x55e6876d5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:40:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f344ffff640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34500b4da0 0x7f34500b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f34677fe640 1 --2- 172.21.15.16:0/2213785674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687330070 0x55e6876d52c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f34600097b0 tx=0x7f34600098b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dca1c98290fc4cc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f344ffff640 1 -- 172.21.15.16:0/1707713735 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7f345c0e54c0 con 0x7f34681050a0 2024-08-28T19:40:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f344d7fa640 1 -- 172.21.15.16:0/2213785674 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3460031cf0 con 0x55e687330070 2024-08-28T19:40:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f344d7fa640 1 -- 172.21.15.16:0/2213785674 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f346003f070 con 0x55e687330070 2024-08-28T19:40:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f344d7fa640 1 -- 172.21.15.16:0/2213785674 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34600042a0 con 0x55e687330070 2024-08-28T19:40:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.733+0000 7f3466ffd640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34500b4da0 0x7f34500b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.734+0000 7f3473c00740 1 -- 172.21.15.16:0/2213785674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687330070 msgr2=0x55e6876d52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.734+0000 7f3466ffd640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34500b4da0 0x7f34500b72b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x55e6876d3f40 tx=0x7f3458009190 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.734+0000 7f3473c00740 1 --2- 172.21.15.16:0/2213785674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687330070 0x55e6876d52c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f34600097b0 tx=0x7f34600098b0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.734+0000 7f3473c00740 1 -- 172.21.15.16:0/2213785674 shutdown_connections 2024-08-28T19:40:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.734+0000 7f3473c00740 1 --2- 172.21.15.16:0/2213785674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e68730f390 0x55e6876d5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.734+0000 7f3473c00740 1 --2- 172.21.15.16:0/2213785674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 0x55e6876d5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.734+0000 7f3473c00740 1 --2- 172.21.15.16:0/2213785674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e687330070 0x55e6876d52c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.734+0000 7f3473c00740 1 -- 172.21.15.16:0/2213785674 >> 172.21.15.16:0/2213785674 conn(0x55e687324460 msgr2=0x55e6876d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.735+0000 7f3473c00740 1 -- 172.21.15.16:0/2213785674 shutdown_connections 2024-08-28T19:40:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.735+0000 7f3473c00740 1 -- 172.21.15.16:0/2213785674 wait complete. 2024-08-28T19:40:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.735+0000 7f3473c00740 1 -- start start 2024-08-28T19:40:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.735+0000 7f3473c00740 1 Processor -- start 2024-08-28T19:40:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f3473c00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6873071b0 0x55e6876e0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f3473c00740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e68730f390 0x55e6876e0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f3473c00740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687330070 0x55e6876ecd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f3473c00740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e68749e700 con 0x55e68730f390 2024-08-28T19:40:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f3473c00740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e6875c2700 con 0x55e6873071b0 2024-08-28T19:40:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f3473c00740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e6876e3a40 con 0x55e687330070 2024-08-28T19:40:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f3466ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6873071b0 0x55e6876e0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f34667fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e68730f390 0x55e6876e0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f3466ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6873071b0 0x55e6876e0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:34936/0 (socket says 172.21.15.16:34936) 2024-08-28T19:40:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f3466ffd640 1 -- 172.21.15.16:0/1922092965 learned_addr learned my addr 172.21.15.16:0/1922092965 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f3473c00740 10 client.0 _ll_register_callbacks cb 0x55e6875f1dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f34677fe640 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687330070 0x55e6876ecd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f34667fc640 1 -- 172.21.15.16:0/1922092965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687330070 msgr2=0x55e6876ecd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f34667fc640 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687330070 0x55e6876ecd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f34667fc640 1 -- 172.21.15.16:0/1922092965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6873071b0 msgr2=0x55e6876e0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f34667fc640 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6873071b0 0x55e6876e0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f34667fc640 1 -- 172.21.15.16:0/1922092965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6875fb8b0 con 0x55e68730f390 2024-08-28T19:40:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.736+0000 7f34677fe640 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687330070 0x55e6876ecd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:40:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.737+0000 7f34667fc640 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e68730f390 0x55e6876e0fc0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f345c0e27e0 tx=0x7f345c0ab550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67120c6f0796170e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.737+0000 7f344aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:40:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.737+0000 7f344aff5640 1 -- 172.21.15.16:0/1922092965 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c020070 con 0x55e68730f390 2024-08-28T19:40:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.737+0000 7f3473c00740 1 -- 172.21.15.16:0/1922092965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f346003f070 con 0x55e68730f390 2024-08-28T19:40:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.737+0000 7f344aff5640 1 -- 172.21.15.16:0/1922092965 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7f345c0a7a10 con 0x55e68730f390 2024-08-28T19:40:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.737+0000 7f344aff5640 1 -- 172.21.15.16:0/1922092965 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f345c0b43e0 con 0x55e68730f390 2024-08-28T19:40:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.737+0000 7f3473c00740 10 client.19947 fetch_fsmap learned FSMap version 100 2024-08-28T19:40:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.737+0000 7f3473c00740 1 -- 172.21.15.16:0/1922092965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55e6876ef880 con 0x55e68730f390 2024-08-28T19:40:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.737+0000 7f344aff5640 1 -- 172.21.15.16:0/1922092965 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f346003f070 con 0x55e68730f390 2024-08-28T19:40:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.738+0000 7f344aff5640 1 -- 172.21.15.16:0/1922092965 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 100) ==== 8406+0+0 (secure 0 0 0) 0x7f345c0e5120 con 0x55e68730f390 2024-08-28T19:40:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.738+0000 7f3473c00740 10 client.19947 fetch_fsmap finished waiting for FSMap version 100 2024-08-28T19:40:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.738+0000 7f3473c00740 10 client.19947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:40:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.738+0000 7f3473c00740 20 client.19947 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.738+0000 7f3473c00740 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6872ee9f0 0x55e6876f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.738+0000 7f3473c00740 4 client.19947 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:40:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.738+0000 7f3473c00740 1 -- 172.21.15.16:0/1922092965 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e68764e580 con 0x55e6872ee9f0 2024-08-28T19:40:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.738+0000 7f3466ffd640 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6872ee9f0 0x55e6876f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.741+0000 7f344aff5640 10 client.19947 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.741+0000 7f3466ffd640 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6872ee9f0 0x55e6876f2d90 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f18e36d2992d5a42 server_cookie=72deea684664f9c8 in_seq=0 out_seq=0 2024-08-28T19:40:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f344aff5640 1 -- 172.21.15.16:0/1922092965 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e68764e580 con 0x55e6872ee9f0 2024-08-28T19:40:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f344aff5640 10 client.19947 handle_command_reply: tid=0 2024-08-28T19:40:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f3473c00740 1 client.19947 shutdown 2024-08-28T19:40:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f3473c00740 1 -- 172.21.15.16:0/1922092965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e68730f390 msgr2=0x55e6876e0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f3473c00740 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e68730f390 0x55e6876e0fc0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f345c0e27e0 tx=0x7f345c0ab550 comp rx=0 tx=0).stop 2024-08-28T19:40:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f3473c00740 1 -- 172.21.15.16:0/1922092965 shutdown_connections 2024-08-28T19:40:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f3473c00740 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6872ee9f0 0x55e6876f2d90 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f3473c00740 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687330070 0x55e6876ecd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f3473c00740 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e68730f390 0x55e6876e0fc0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f3473c00740 1 --2- 172.21.15.16:0/1922092965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6873071b0 0x55e6876e0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f344aff5640 0 client.19947 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.744+0000 7f3473c00740 1 -- 172.21.15.16:0/1922092965 >> 172.21.15.16:0/1922092965 conn(0x55e687324460 msgr2=0x55e6876d84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.745+0000 7f3473c00740 1 -- 172.21.15.16:0/1922092965 shutdown_connections 2024-08-28T19:40:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.745+0000 7f3473c00740 1 -- 172.21.15.16:0/1922092965 wait complete. 2024-08-28T19:40:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.745+0000 7f3473c00740 20 client.19947 trim_cache size 0 max 16384 2024-08-28T19:40:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f3473c00740 1 Processor -- start 2024-08-28T19:40:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f3473c00740 1 -- start start 2024-08-28T19:40:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f3473c00740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 0x55e6876d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f3473c00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e68730f390 0x55e6876ecc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f3473c00740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687330070 0x55e6876ed170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f3473c00740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b2f90 con 0x55e6873071b0 2024-08-28T19:40:22.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f3473c00740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b4c80 con 0x55e68730f390 2024-08-28T19:40:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f3473c00740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b36f0 con 0x55e687330070 2024-08-28T19:40:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f3466ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 0x55e6876d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f3466ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 0x55e6876d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43496/0 (socket says 172.21.15.16:43496) 2024-08-28T19:40:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f3466ffd640 1 -- 172.21.15.16:0/1757488740 learned_addr learned my addr 172.21.15.16:0/1757488740 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.767+0000 7f34667fc640 1 --2- 172.21.15.16:0/1757488740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e68730f390 0x55e6876ecc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.768+0000 7f34677fe640 1 --2- 172.21.15.16:0/1757488740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687330070 0x55e6876ed170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.768+0000 7f3466ffd640 1 -- 172.21.15.16:0/1757488740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687330070 msgr2=0x55e6876ed170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.768+0000 7f3466ffd640 1 --2- 172.21.15.16:0/1757488740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687330070 0x55e6876ed170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.768+0000 7f3466ffd640 1 -- 172.21.15.16:0/1757488740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e68730f390 msgr2=0x55e6876ecc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.768+0000 7f3466ffd640 1 --2- 172.21.15.16:0/1757488740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e68730f390 0x55e6876ecc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.768+0000 7f3466ffd640 1 -- 172.21.15.16:0/1757488740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6876df9e0 con 0x55e6873071b0 2024-08-28T19:40:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.768+0000 7f3466ffd640 1 --2- 172.21.15.16:0/1757488740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 0x55e6876d3220 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f3458011d10 tx=0x7f3458019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3bca6e75e1cf5a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.768+0000 7f344cff9640 1 -- 172.21.15.16:0/1757488740 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345802e030 con 0x55e6873071b0 2024-08-28T19:40:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.768+0000 7f344cff9640 1 -- 172.21.15.16:0/1757488740 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f345801b040 con 0x55e6873071b0 2024-08-28T19:40:22.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.768+0000 7f344cff9640 1 -- 172.21.15.16:0/1757488740 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3458014440 con 0x55e6873071b0 2024-08-28T19:40:22.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.769+0000 7f3473c00740 1 -- 172.21.15.16:0/1757488740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 msgr2=0x55e6876d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.769+0000 7f3473c00740 1 --2- 172.21.15.16:0/1757488740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 0x55e6876d3220 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f3458011d10 tx=0x7f3458019000 comp rx=0 tx=0).stop 2024-08-28T19:40:22.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.769+0000 7f3473c00740 1 -- 172.21.15.16:0/1757488740 shutdown_connections 2024-08-28T19:40:22.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.769+0000 7f3473c00740 1 --2- 172.21.15.16:0/1757488740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e687330070 0x55e6876ed170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.769+0000 7f3473c00740 1 --2- 172.21.15.16:0/1757488740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e68730f390 0x55e6876ecc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.769+0000 7f3473c00740 1 --2- 172.21.15.16:0/1757488740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6873071b0 0x55e6876d3220 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.769+0000 7f3473c00740 1 -- 172.21.15.16:0/1757488740 >> 172.21.15.16:0/1757488740 conn(0x55e687324460 msgr2=0x55e6876d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.769+0000 7f3473c00740 1 -- 172.21.15.16:0/1757488740 shutdown_connections 2024-08-28T19:40:22.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.769+0000 7f3473c00740 1 -- 172.21.15.16:0/1757488740 wait complete. 2024-08-28T19:40:22.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.770+0000 7f3473c00740 1 -- start start 2024-08-28T19:40:22.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.770+0000 7f3473c00740 1 Processor -- start 2024-08-28T19:40:22.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.770+0000 7f3473c00740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6873071b0 0x55e6876e4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.770+0000 7f3473c00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e68730f390 0x55e6876e5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.770+0000 7f3473c00740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687330070 0x55e68775c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.770+0000 7f3473c00740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b44f0 con 0x55e687330070 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.770+0000 7f3473c00740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b3dc0 con 0x55e68730f390 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.770+0000 7f3473c00740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b0f90 con 0x55e6873071b0 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.771+0000 7f3473c00740 10 client.0 _ll_register_callbacks cb 0x7f345c005d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.771+0000 7f3466ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6873071b0 0x55e6876e4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.771+0000 7f3466ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6873071b0 0x55e6876e4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:51212/0 (socket says 172.21.15.16:51212) 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.771+0000 7f3466ffd640 1 -- 172.21.15.16:0/280530820 learned_addr learned my addr 172.21.15.16:0/280530820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.771+0000 7f34677fe640 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687330070 0x55e68775c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.771+0000 7f3466ffd640 1 -- 172.21.15.16:0/280530820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e68730f390 msgr2=0x55e6876e5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.771+0000 7f34667fc640 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e68730f390 0x55e6876e5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.771+0000 7f3466ffd640 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e68730f390 0x55e6876e5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.771+0000 7f3466ffd640 1 -- 172.21.15.16:0/280530820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687330070 msgr2=0x55e68775c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:22.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.771+0000 7f3466ffd640 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687330070 0x55e68775c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.771+0000 7f3466ffd640 1 -- 172.21.15.16:0/280530820 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6876e5bf0 con 0x55e6873071b0 2024-08-28T19:40:22.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.772+0000 7f3466ffd640 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6873071b0 0x55e6876e4ec0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3458004810 tx=0x7f3458023700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f2554af5b2213e5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:22.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.772+0000 7f344aff5640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:40:22.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.772+0000 7f344aff5640 1 -- 172.21.15.16:0/280530820 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345802e040 con 0x55e6873071b0 2024-08-28T19:40:22.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.772+0000 7f344aff5640 1 -- 172.21.15.16:0/280530820 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7f3458023af0 con 0x55e6873071b0 2024-08-28T19:40:22.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.772+0000 7f3473c00740 1 -- 172.21.15.16:0/280530820 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f345801b040 con 0x55e6873071b0 2024-08-28T19:40:22.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.772+0000 7f344aff5640 1 -- 172.21.15.16:0/280530820 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f3458023cf0 con 0x55e6873071b0 2024-08-28T19:40:22.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.772+0000 7f3473c00740 10 client.28966 fetch_fsmap learned FSMap version 100 2024-08-28T19:40:22.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.772+0000 7f3473c00740 1 -- 172.21.15.16:0/280530820 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55e68775dc90 con 0x55e6873071b0 2024-08-28T19:40:22.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.773+0000 7f344aff5640 1 -- 172.21.15.16:0/280530820 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f345801b040 con 0x55e6873071b0 2024-08-28T19:40:22.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.773+0000 7f344aff5640 1 -- 172.21.15.16:0/280530820 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 100) ==== 8406+0+0 (secure 0 0 0) 0x7f3458020040 con 0x55e6873071b0 2024-08-28T19:40:22.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.773+0000 7f3473c00740 10 client.28966 fetch_fsmap finished waiting for FSMap version 100 2024-08-28T19:40:22.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.773+0000 7f3473c00740 10 client.28966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:40:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.773+0000 7f3473c00740 20 client.28966 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.773+0000 7f3473c00740 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6872c3da0 0x55e6877604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.773+0000 7f3473c00740 4 client.28966 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:40:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.773+0000 7f3473c00740 1 -- 172.21.15.16:0/280530820 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e68764a020 con 0x55e6872c3da0 2024-08-28T19:40:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.773+0000 7f34667fc640 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6872c3da0 0x55e6877604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.774+0000 7f344aff5640 10 client.28966 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.774+0000 7f34667fc640 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6872c3da0 0x55e6877604b0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aed864ca46238293 server_cookie=6dbd9e5240d57283 in_seq=0 out_seq=0 2024-08-28T19:40:22.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f344aff5640 1 -- 172.21.15.16:0/280530820 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7190 (crc 0 0 0) 0x55e68764a020 con 0x55e6872c3da0 2024-08-28T19:40:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f344aff5640 10 client.28966 handle_command_reply: tid=0 2024-08-28T19:40:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f3473c00740 1 client.28966 shutdown 2024-08-28T19:40:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f3473c00740 1 -- 172.21.15.16:0/280530820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6873071b0 msgr2=0x55e6876e4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f3473c00740 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6873071b0 0x55e6876e4ec0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3458004810 tx=0x7f3458023700 comp rx=0 tx=0).stop 2024-08-28T19:40:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f3473c00740 1 -- 172.21.15.16:0/280530820 shutdown_connections 2024-08-28T19:40:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f3473c00740 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e687330070 0x55e68775c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f3473c00740 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e68730f390 0x55e6876e5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f3473c00740 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55e6872c3da0 0x55e6877604b0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f3473c00740 1 --2- 172.21.15.16:0/280530820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e6873071b0 0x55e6876e4ec0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f3473c00740 1 -- 172.21.15.16:0/280530820 >> 172.21.15.16:0/280530820 conn(0x55e687324460 msgr2=0x55e6876d83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f344aff5640 0 client.28966 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.779+0000 7f3473c00740 1 -- 172.21.15.16:0/280530820 shutdown_connections 2024-08-28T19:40:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.780+0000 7f3473c00740 1 -- 172.21.15.16:0/280530820 wait complete. 2024-08-28T19:40:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.780+0000 7f3473c00740 20 client.28966 trim_cache size 0 max 16384 2024-08-28T19:40:22.782 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:40:22.782 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:55.619275+0000", 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:40:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.786 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:40:22.787 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.788 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.788 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.788 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.788 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.788 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:40:22.788 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:40:22.788 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.788 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.790 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: "8de90bf3-5743-4711-98c2-a56ca0757291": { 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:39:48.098772+0000", 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:40:22.791 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:40:22.794 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.796 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:22.797 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:22.798 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:22.798 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:22.798 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:22.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:40:22.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:40:22.798 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:40:22.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.780+0000 7f346d68b640 1 -- 172.21.15.16:0/1707713735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34500b4da0 msgr2=0x7f34500b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.781+0000 7f346d68b640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34500b4da0 0x7f34500b72b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x55e6876d3f40 tx=0x7f3458009190 comp rx=0 tx=0).stop 2024-08-28T19:40:22.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.781+0000 7f346d68b640 1 -- 172.21.15.16:0/1707713735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34681050a0 msgr2=0x7f346819a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:22.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.781+0000 7f346d68b640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34681050a0 0x7f346819a8a0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f345c005fd0 tx=0x7f345c00dde0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.781+0000 7f346d68b640 1 -- 172.21.15.16:0/1707713735 shutdown_connections 2024-08-28T19:40:22.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.781+0000 7f346d68b640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f34500b4da0 0x7f34500b72b0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.781+0000 7f346d68b640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f346810dbb0 0x7f346819fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.781+0000 7f346d68b640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34681050a0 0x7f346819a8a0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.781+0000 7f346d68b640 1 --2- 172.21.15.16:0/1707713735 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3468104750 0x7f346819a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:22.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.781+0000 7f346d68b640 1 -- 172.21.15.16:0/1707713735 >> 172.21.15.16:0/1707713735 conn(0x7f34680794d0 msgr2=0x7f34680ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:22.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.781+0000 7f346d68b640 1 -- 172.21.15.16:0/1707713735 shutdown_connections 2024-08-28T19:40:22.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:22.781+0000 7f346d68b640 1 -- 172.21.15.16:0/1707713735 wait complete. 2024-08-28T19:40:23.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134987 21.80 MB/sec execute 239 sec latency 70.936 ms 2024-08-28T19:40:23.617 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a87c215e', total seconds elapsed: 31.1 2024-08-28T19:40:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:23 smithi016 ceph-mon[18556]: pgmap v945: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 16 op/s 2024-08-28T19:40:23.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:23 smithi028 ceph-mon[21769]: pgmap v945: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 16 op/s 2024-08-28T19:40:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:23 smithi115 ceph-mon[22184]: pgmap v945: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 16 op/s 2024-08-28T19:40:24.298 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:40:24.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:40:24.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 135537 21.84 MB/sec execute 240 sec latency 106.182 ms 2024-08-28T19:40:24.680 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:40:25.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.252+0000 7fdddbb5f640 1 -- 172.21.15.16:0/1868609219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd41050c0 msgr2=0x7fddd410d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:25.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.252+0000 7fdddbb5f640 1 --2- 172.21.15.16:0/1868609219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd41050c0 0x7fddd410d610 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009900 tx=0x7fddc402a410 comp rx=0 tx=0).stop 2024-08-28T19:40:25.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.252+0000 7fdddbb5f640 1 -- 172.21.15.16:0/1868609219 shutdown_connections 2024-08-28T19:40:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.252+0000 7fdddbb5f640 1 --2- 172.21.15.16:0/1868609219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd410dbd0 0x7fddd41100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.252+0000 7fdddbb5f640 1 --2- 172.21.15.16:0/1868609219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd41050c0 0x7fddd410d610 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.252+0000 7fdddbb5f640 1 --2- 172.21.15.16:0/1868609219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd4104770 0x7fddd4104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.252+0000 7fdddbb5f640 1 -- 172.21.15.16:0/1868609219 >> 172.21.15.16:0/1868609219 conn(0x7fddd40794e0 msgr2=0x7fddd40798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.253+0000 7fdddbb5f640 1 -- 172.21.15.16:0/1868609219 shutdown_connections 2024-08-28T19:40:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.253+0000 7fdddbb5f640 1 -- 172.21.15.16:0/1868609219 wait complete. 2024-08-28T19:40:25.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.254+0000 7fdddbb5f640 1 Processor -- start 2024-08-28T19:40:25.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.254+0000 7fdddbb5f640 1 -- start start 2024-08-28T19:40:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.255+0000 7fdddbb5f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd4104770 0x7fddd419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.255+0000 7fdddbb5f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd41050c0 0x7fddd419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.255+0000 7fdddbb5f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd410dbd0 0x7fddd419fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.255+0000 7fdddbb5f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fddd41137f0 con 0x7fddd410dbd0 2024-08-28T19:40:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.255+0000 7fdddbb5f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fddd4113670 con 0x7fddd41050c0 2024-08-28T19:40:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.255+0000 7fdddbb5f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fddd4113970 con 0x7fddd4104770 2024-08-28T19:40:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.255+0000 7fddd98fb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd4104770 0x7fddd419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.255+0000 7fddd90fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd41050c0 0x7fddd419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.255+0000 7fddd90fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd41050c0 0x7fddd419a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:36924/0 (socket says 172.21.15.16:36924) 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.255+0000 7fddd90fa640 1 -- 172.21.15.16:0/908495506 learned_addr learned my addr 172.21.15.16:0/908495506 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.256+0000 7fddda0fc640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd410dbd0 0x7fddd419fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.256+0000 7fddd98fb640 1 -- 172.21.15.16:0/908495506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd41050c0 msgr2=0x7fddd419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.256+0000 7fddd98fb640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd41050c0 0x7fddd419a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.256+0000 7fddd98fb640 1 -- 172.21.15.16:0/908495506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd410dbd0 msgr2=0x7fddd419fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.256+0000 7fddd98fb640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd410dbd0 0x7fddd419fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.256+0000 7fddd98fb640 1 -- 172.21.15.16:0/908495506 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd41a01b0 con 0x7fddd4104770 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.256+0000 7fddd90fa640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd41050c0 0x7fddd419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.256+0000 7fddd98fb640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd4104770 0x7fddd419a240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fddd000e9c0 tx=0x7fddd000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.257+0000 7fddcaffd640 1 -- 172.21.15.16:0/908495506 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd000cd80 con 0x7fddd4104770 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.257+0000 7fddcaffd640 1 -- 172.21.15.16:0/908495506 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddd00108e0 con 0x7fddd4104770 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.257+0000 7fddcaffd640 1 -- 172.21.15.16:0/908495506 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd0018900 con 0x7fddd4104770 2024-08-28T19:40:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.257+0000 7fdddbb5f640 1 -- 172.21.15.16:0/908495506 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd41a04a0 con 0x7fddd4104770 2024-08-28T19:40:25.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.257+0000 7fdddbb5f640 1 -- 172.21.15.16:0/908495506 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd41a6c00 con 0x7fddd4104770 2024-08-28T19:40:25.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.258+0000 7fdddbb5f640 1 -- 172.21.15.16:0/908495506 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd4104b80 con 0x7fddd4104770 2024-08-28T19:40:25.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.261+0000 7fddcaffd640 1 -- 172.21.15.16:0/908495506 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fddd001f020 con 0x7fddd4104770 2024-08-28T19:40:25.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.262+0000 7fddcaffd640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc00b4da0 0x7fddc00b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.262+0000 7fddcaffd640 1 -- 172.21.15.16:0/908495506 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fddd00e5e90 con 0x7fddd4104770 2024-08-28T19:40:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.263+0000 7fddd90fa640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc00b4da0 0x7fddc00b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.264+0000 7fddd90fa640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc00b4da0 0x7fddc00b72b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fddd4102250 tx=0x7fddc4032ed0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.268+0000 7fddcaffd640 1 -- 172.21.15.16:0/908495506 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fddd00ac850 con 0x7fddd4104770 2024-08-28T19:40:25.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:25 smithi016 ceph-mon[18556]: pgmap v946: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 144 KiB/s wr, 20 op/s 2024-08-28T19:40:25.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 136233 21.90 MB/sec execute 241 sec latency 99.168 ms 2024-08-28T19:40:25.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.543+0000 7fdddbb5f640 1 -- 172.21.15.16:0/908495506 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddd419b370 con 0x7fddd4104770 2024-08-28T19:40:25.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.544+0000 7fddcaffd640 1 -- 172.21.15.16:0/908495506 <== mon.1 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+11992 (secure 0 0 0) 0x7fddd00ac150 con 0x7fddd4104770 2024-08-28T19:40:25.546 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:40:25.546 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":101,"btime":"2024-08-28T19:40:25:411669+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":101,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:40:24.586397+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:40:25.546 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 101 2024-08-28T19:40:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.546+0000 7fdddbb5f640 1 -- 172.21.15.16:0/908495506 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc00b4da0 msgr2=0x7fddc00b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.546+0000 7fdddbb5f640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc00b4da0 0x7fddc00b72b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fddd4102250 tx=0x7fddc4032ed0 comp rx=0 tx=0).stop 2024-08-28T19:40:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.546+0000 7fdddbb5f640 1 -- 172.21.15.16:0/908495506 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd4104770 msgr2=0x7fddd419a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.546+0000 7fdddbb5f640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd4104770 0x7fddd419a240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fddd000e9c0 tx=0x7fddd000ee80 comp rx=0 tx=0).stop 2024-08-28T19:40:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.546+0000 7fdddbb5f640 1 -- 172.21.15.16:0/908495506 shutdown_connections 2024-08-28T19:40:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.546+0000 7fdddbb5f640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddd410dbd0 0x7fddd419fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.546+0000 7fdddbb5f640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fddc00b4da0 0x7fddc00b72b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.546+0000 7fdddbb5f640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd41050c0 0x7fddd419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.546+0000 7fdddbb5f640 1 --2- 172.21.15.16:0/908495506 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fddd4104770 0x7fddd419a240 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.546+0000 7fdddbb5f640 1 -- 172.21.15.16:0/908495506 >> 172.21.15.16:0/908495506 conn(0x7fddd40794e0 msgr2=0x7fddd41005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.547+0000 7fdddbb5f640 1 -- 172.21.15.16:0/908495506 shutdown_connections 2024-08-28T19:40:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:25.547+0000 7fdddbb5f640 1 -- 172.21.15.16:0/908495506 wait complete. 2024-08-28T19:40:25.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:25 smithi028 ceph-mon[21769]: pgmap v946: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 144 KiB/s wr, 20 op/s 2024-08-28T19:40:25.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:25 smithi115 ceph-mon[22184]: pgmap v946: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 144 KiB/s wr, 20 op/s 2024-08-28T19:40:26.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:40:26.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 136993 21.98 MB/sec execute 242 sec latency 74.053 ms 2024-08-28T19:40:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:40:26.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:26 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:40:26.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:26 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/908495506' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:40:26.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:26 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:40:26.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:26 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/908495506' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:40:26.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:26 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:40:26.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:26 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/908495506' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:40:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.267+0000 7ff837bbd640 1 -- 172.21.15.16:0/425467254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff830101d80 msgr2=0x7ff830102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.267+0000 7ff837bbd640 1 --2- 172.21.15.16:0/425467254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff830101d80 0x7ff830102190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009900 tx=0x7ff82c02a4a0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.267+0000 7ff837bbd640 1 -- 172.21.15.16:0/425467254 shutdown_connections 2024-08-28T19:40:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.267+0000 7ff837bbd640 1 --2- 172.21.15.16:0/425467254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8301141c0 0x7ff8301166d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.267+0000 7ff837bbd640 1 --2- 172.21.15.16:0/425467254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8301026d0 0x7ff830113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.267+0000 7ff837bbd640 1 --2- 172.21.15.16:0/425467254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff830101d80 0x7ff830102190 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.267+0000 7ff837bbd640 1 -- 172.21.15.16:0/425467254 >> 172.21.15.16:0/425467254 conn(0x7ff830077960 msgr2=0x7ff830077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.270+0000 7ff837bbd640 1 -- 172.21.15.16:0/425467254 shutdown_connections 2024-08-28T19:40:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.270+0000 7ff837bbd640 1 -- 172.21.15.16:0/425467254 wait complete. 2024-08-28T19:40:27.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.271+0000 7ff837bbd640 1 Processor -- start 2024-08-28T19:40:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.271+0000 7ff837bbd640 1 -- start start 2024-08-28T19:40:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.271+0000 7ff837bbd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff830101d80 0x7ff8301a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.271+0000 7ff837bbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8301026d0 0x7ff8301a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.271+0000 7ff837bbd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8301141c0 0x7ff8301a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.271+0000 7ff837bbd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff830119ef0 con 0x7ff830101d80 2024-08-28T19:40:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.271+0000 7ff837bbd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff830119d70 con 0x7ff8301026d0 2024-08-28T19:40:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.271+0000 7ff837bbd640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff83011a070 con 0x7ff8301141c0 2024-08-28T19:40:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.272+0000 7ff835158640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8301026d0 0x7ff8301a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.272+0000 7ff835158640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8301026d0 0x7ff8301a0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:36936/0 (socket says 172.21.15.16:36936) 2024-08-28T19:40:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.272+0000 7ff835959640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff830101d80 0x7ff8301a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.272+0000 7ff835158640 1 -- 172.21.15.16:0/4265015959 learned_addr learned my addr 172.21.15.16:0/4265015959 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.272+0000 7ff83615a640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8301141c0 0x7ff8301a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.272+0000 7ff835959640 1 -- 172.21.15.16:0/4265015959 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8301141c0 msgr2=0x7ff8301a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.272+0000 7ff835959640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8301141c0 0x7ff8301a60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.272+0000 7ff835959640 1 -- 172.21.15.16:0/4265015959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8301026d0 msgr2=0x7ff8301a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.272+0000 7ff835959640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8301026d0 0x7ff8301a0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.272+0000 7ff835959640 1 -- 172.21.15.16:0/4265015959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8301a67e0 con 0x7ff830101d80 2024-08-28T19:40:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.273+0000 7ff835959640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff830101d80 0x7ff8301a08a0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009850 tx=0x7ff82c031bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.276+0000 7ff826ffd640 1 -- 172.21.15.16:0/4265015959 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c040030 con 0x7ff830101d80 2024-08-28T19:40:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.276+0000 7ff826ffd640 1 -- 172.21.15.16:0/4265015959 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c03c8f0 con 0x7ff830101d80 2024-08-28T19:40:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.276+0000 7ff826ffd640 1 -- 172.21.15.16:0/4265015959 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c044990 con 0x7ff830101d80 2024-08-28T19:40:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.277+0000 7ff837bbd640 1 -- 172.21.15.16:0/4265015959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8301a6a70 con 0x7ff830101d80 2024-08-28T19:40:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.277+0000 7ff837bbd640 1 -- 172.21.15.16:0/4265015959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff83006cb80 con 0x7ff830101d80 2024-08-28T19:40:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.281+0000 7ff83e132740 1 Processor -- start 2024-08-28T19:40:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.282+0000 7ff83e132740 1 -- start start 2024-08-28T19:40:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.283+0000 7ff83e132740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 0x563c10eca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.283+0000 7ff83e132740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10afc1b0 0x563c10eca850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.283+0000 7ff83e132740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 0x563c10ecad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.283+0000 7ff83e132740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563c10e41680 con 0x563c10b25070 2024-08-28T19:40:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.283+0000 7ff83e132740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c10e30c10 con 0x563c10afc1b0 2024-08-28T19:40:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.283+0000 7ff83e132740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563c10e34bc0 con 0x563c10b04390 2024-08-28T19:40:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.283+0000 7ff83615a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 0x563c10eca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.283+0000 7ff83615a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 0x563c10eca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45758/0 (socket says 172.21.15.16:45758) 2024-08-28T19:40:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.283+0000 7ff83615a640 1 -- 172.21.15.16:0/3539159497 learned_addr learned my addr 172.21.15.16:0/3539159497 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.283+0000 7ff826ffd640 1 -- 172.21.15.16:0/4265015959 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7ff82c043030 con 0x7ff830101d80 2024-08-28T19:40:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.283+0000 7ff835959640 1 --2- 172.21.15.16:0/3539159497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10afc1b0 0x563c10eca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.284+0000 7ff835158640 1 --2- 172.21.15.16:0/3539159497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 0x563c10ecad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.284+0000 7ff835959640 1 -- 172.21.15.16:0/3539159497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 msgr2=0x563c10ecad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.284+0000 7ff835959640 1 --2- 172.21.15.16:0/3539159497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 0x563c10ecad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.284+0000 7ff835959640 1 -- 172.21.15.16:0/3539159497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 msgr2=0x563c10eca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.284+0000 7ff835959640 1 --2- 172.21.15.16:0/3539159497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 0x563c10eca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.284+0000 7ff835959640 1 -- 172.21.15.16:0/3539159497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c10b90650 con 0x563c10afc1b0 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.285+0000 7ff83615a640 1 --2- 172.21.15.16:0/3539159497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 0x563c10eca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.285+0000 7ff835158640 1 --2- 172.21.15.16:0/3539159497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 0x563c10ecad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.285+0000 7ff835959640 1 --2- 172.21.15.16:0/3539159497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10afc1b0 0x563c10eca850 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff82c103470 tx=0x7ff82c10e3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee1679f8d2ed7355 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.286+0000 7ff81bfff640 1 -- 172.21.15.16:0/3539159497 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c109050 con 0x563c10afc1b0 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.286+0000 7ff826ffd640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff81c0b4eb0 0x7ff81c0b73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.286+0000 7ff81bfff640 1 -- 172.21.15.16:0/3539159497 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c10e780 con 0x563c10afc1b0 2024-08-28T19:40:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.286+0000 7ff83615a640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff81c0b4eb0 0x7ff81c0b73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.286+0000 7ff826ffd640 1 -- 172.21.15.16:0/4265015959 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7ff82c106150 con 0x7ff830101d80 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.287+0000 7ff81bfff640 1 -- 172.21.15.16:0/3539159497 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c108ac0 con 0x563c10afc1b0 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.287+0000 7ff83e132740 1 -- 172.21.15.16:0/3539159497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10afc1b0 msgr2=0x563c10eca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.287+0000 7ff83e132740 1 --2- 172.21.15.16:0/3539159497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10afc1b0 0x563c10eca850 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff82c103470 tx=0x7ff82c10e3b0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.288+0000 7ff83615a640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff81c0b4eb0 0x7ff81c0b73c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff82800b760 tx=0x7ff82800e040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.288+0000 7ff83e132740 1 -- 172.21.15.16:0/3539159497 shutdown_connections 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.288+0000 7ff83e132740 1 --2- 172.21.15.16:0/3539159497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 0x563c10ecad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.288+0000 7ff83e132740 1 --2- 172.21.15.16:0/3539159497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10afc1b0 0x563c10eca850 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.288+0000 7ff83e132740 1 --2- 172.21.15.16:0/3539159497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 0x563c10eca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.288+0000 7ff83e132740 1 -- 172.21.15.16:0/3539159497 >> 172.21.15.16:0/3539159497 conn(0x563c10b19460 msgr2=0x563c10ec8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.288+0000 7ff83e132740 1 -- 172.21.15.16:0/3539159497 shutdown_connections 2024-08-28T19:40:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.288+0000 7ff83e132740 1 -- 172.21.15.16:0/3539159497 wait complete. 2024-08-28T19:40:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.289+0000 7ff83e132740 1 -- start start 2024-08-28T19:40:27.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.289+0000 7ff83e132740 1 Processor -- start 2024-08-28T19:40:27.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff83e132740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 0x563c10ed5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff83e132740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b04390 0x563c10ed5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff83e132740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 0x563c10ee1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff83e132740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563c10c93700 con 0x563c10b04390 2024-08-28T19:40:27.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff83e132740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c10db7750 con 0x563c10b25070 2024-08-28T19:40:27.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff83e132740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563c10ed88b0 con 0x563c10afc1b0 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff83e132740 10 client.0 _ll_register_callbacks cb 0x563c10de6e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff835959640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 0x563c10ed5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff835959640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 0x563c10ed5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:53150/0 (socket says 172.21.15.16:53150) 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff835959640 1 -- 172.21.15.16:0/3117764482 learned_addr learned my addr 172.21.15.16:0/3117764482 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff83615a640 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 0x563c10ee1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.290+0000 7ff835158640 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b04390 0x563c10ed5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.291+0000 7ff83615a640 1 -- 172.21.15.16:0/3117764482 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 msgr2=0x563c10ed5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.291+0000 7ff83615a640 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 0x563c10ed5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.291+0000 7ff83615a640 1 -- 172.21.15.16:0/3117764482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b04390 msgr2=0x563c10ed5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.291+0000 7ff83615a640 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b04390 0x563c10ed5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.291+0000 7ff83615a640 1 -- 172.21.15.16:0/3117764482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c10df0900 con 0x563c10b25070 2024-08-28T19:40:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.291+0000 7ff83615a640 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 0x563c10ee1bb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff8280072b0 tx=0x7ff828004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3a3ae7c6f9d8f11 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.291+0000 7ff8197fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:40:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.292+0000 7ff8197fa640 1 -- 172.21.15.16:0/3117764482 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828029060 con 0x563c10b25070 2024-08-28T19:40:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.292+0000 7ff83e132740 1 -- 172.21.15.16:0/3117764482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff82c10e780 con 0x563c10b25070 2024-08-28T19:40:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.292+0000 7ff8197fa640 1 -- 172.21.15.16:0/3117764482 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7ff8280194e0 con 0x563c10b25070 2024-08-28T19:40:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.294+0000 7ff8197fa640 1 -- 172.21.15.16:0/3117764482 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7ff828030050 con 0x563c10b25070 2024-08-28T19:40:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.294+0000 7ff83e132740 10 client.29414 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.294+0000 7ff83e132740 1 -- 172.21.15.16:0/3117764482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x563c10ee4670 con 0x563c10b25070 2024-08-28T19:40:27.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.298+0000 7ff8197fa640 1 -- 172.21.15.16:0/3117764482 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c10e780 con 0x563c10b25070 2024-08-28T19:40:27.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.298+0000 7ff8197fa640 1 -- 172.21.15.16:0/3117764482 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7ff828024840 con 0x563c10b25070 2024-08-28T19:40:27.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.298+0000 7ff83e132740 10 client.29414 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:27.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.298+0000 7ff83e132740 10 client.29414 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:40:27.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.298+0000 7ff83e132740 20 client.29414 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:27.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.298+0000 7ff83e132740 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563c10ae39f0 0x563c10ee7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.298+0000 7ff83e132740 4 client.29414 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:40:27.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.298+0000 7ff83e132740 1 -- 172.21.15.16:0/3117764482 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c10e435d0 con 0x563c10ae39f0 2024-08-28T19:40:27.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.299+0000 7ff835959640 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563c10ae39f0 0x563c10ee7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.300+0000 7ff8197fa640 10 client.29414 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:40:27.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.300+0000 7ff835959640 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563c10ae39f0 0x563c10ee7bd0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3ecdfe0590894375 server_cookie=53d457e6645db8b7 in_seq=0 out_seq=0 2024-08-28T19:40:27.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff8197fa640 1 -- 172.21.15.16:0/3117764482 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563c10e435d0 con 0x563c10ae39f0 2024-08-28T19:40:27.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff8197fa640 10 client.29414 handle_command_reply: tid=0 2024-08-28T19:40:27.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff83e132740 1 client.29414 shutdown 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff83e132740 1 -- 172.21.15.16:0/3117764482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 msgr2=0x563c10ee1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff83e132740 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 0x563c10ee1bb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff8280072b0 tx=0x7ff828004a40 comp rx=0 tx=0).stop 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff83e132740 1 -- 172.21.15.16:0/3117764482 shutdown_connections 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff83e132740 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563c10ae39f0 0x563c10ee7bd0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff83e132740 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 0x563c10ee1bb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff83e132740 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b04390 0x563c10ed5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff83e132740 1 --2- 172.21.15.16:0/3117764482 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 0x563c10ed5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff83e132740 1 -- 172.21.15.16:0/3117764482 >> 172.21.15.16:0/3117764482 conn(0x563c10b19460 msgr2=0x563c10ecd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff8197fa640 0 client.29414 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff83e132740 1 -- 172.21.15.16:0/3117764482 shutdown_connections 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.301+0000 7ff83e132740 1 -- 172.21.15.16:0/3117764482 wait complete. 2024-08-28T19:40:27.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.302+0000 7ff83e132740 20 client.29414 trim_cache size 0 max 16384 2024-08-28T19:40:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.319+0000 7ff83e132740 1 Processor -- start 2024-08-28T19:40:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff83e132740 1 -- start start 2024-08-28T19:40:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff83e132740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10afc1b0 0x563c10ed5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff83e132740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 0x563c10ecd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff83e132740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 0x563c10ecd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff83e132740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8301bfb90 con 0x563c10afc1b0 2024-08-28T19:40:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff83e132740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8301c1850 con 0x563c10b25070 2024-08-28T19:40:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff835959640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10afc1b0 0x563c10ed5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff83e132740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff8301c02f0 con 0x563c10b04390 2024-08-28T19:40:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff835158640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 0x563c10ecd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff835158640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 0x563c10ecd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:53164/0 (socket says 172.21.15.16:53164) 2024-08-28T19:40:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff835158640 1 -- 172.21.15.16:0/1466784831 learned_addr learned my addr 172.21.15.16:0/1466784831 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.320+0000 7ff83615a640 1 --2- 172.21.15.16:0/1466784831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 0x563c10ecd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.321+0000 7ff835959640 1 -- 172.21.15.16:0/1466784831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 msgr2=0x563c10ecd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.321+0000 7ff835959640 1 --2- 172.21.15.16:0/1466784831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 0x563c10ecd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.321+0000 7ff835959640 1 -- 172.21.15.16:0/1466784831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 msgr2=0x563c10ecd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.321+0000 7ff835959640 1 --2- 172.21.15.16:0/1466784831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 0x563c10ecd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.321+0000 7ff835959640 1 -- 172.21.15.16:0/1466784831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c10ee4480 con 0x563c10afc1b0 2024-08-28T19:40:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.321+0000 7ff83615a640 1 --2- 172.21.15.16:0/1466784831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 0x563c10ecd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:40:27.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.321+0000 7ff835959640 1 --2- 172.21.15.16:0/1466784831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10afc1b0 0x563c10ed5f70 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7ff82c103440 tx=0x7ff82c109b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ad80a42efd3bd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:27.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.322+0000 7ff81b7fe640 1 -- 172.21.15.16:0/1466784831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c04a040 con 0x563c10afc1b0 2024-08-28T19:40:27.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.322+0000 7ff81b7fe640 1 -- 172.21.15.16:0/1466784831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c1099d0 con 0x563c10afc1b0 2024-08-28T19:40:27.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.322+0000 7ff81b7fe640 1 -- 172.21.15.16:0/1466784831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c0cbf20 con 0x563c10afc1b0 2024-08-28T19:40:27.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.323+0000 7ff83e132740 1 -- 172.21.15.16:0/1466784831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10afc1b0 msgr2=0x563c10ed5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:27.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.323+0000 7ff83e132740 1 --2- 172.21.15.16:0/1466784831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10afc1b0 0x563c10ed5f70 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7ff82c103440 tx=0x7ff82c109b80 comp rx=0 tx=0).stop 2024-08-28T19:40:27.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.324+0000 7ff83e132740 1 -- 172.21.15.16:0/1466784831 shutdown_connections 2024-08-28T19:40:27.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.324+0000 7ff83e132740 1 --2- 172.21.15.16:0/1466784831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b25070 0x563c10ecd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.324+0000 7ff83e132740 1 --2- 172.21.15.16:0/1466784831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10b04390 0x563c10ecd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.324+0000 7ff83e132740 1 --2- 172.21.15.16:0/1466784831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10afc1b0 0x563c10ed5f70 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.324+0000 7ff83e132740 1 -- 172.21.15.16:0/1466784831 >> 172.21.15.16:0/1466784831 conn(0x563c10b19460 msgr2=0x563c10ed5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:27.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.325+0000 7ff83e132740 1 -- 172.21.15.16:0/1466784831 shutdown_connections 2024-08-28T19:40:27.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.326+0000 7ff83e132740 1 -- 172.21.15.16:0/1466784831 wait complete. 2024-08-28T19:40:27.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.327+0000 7ff83e132740 1 -- start start 2024-08-28T19:40:27.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.327+0000 7ff83e132740 1 Processor -- start 2024-08-28T19:40:27.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.328+0000 7ff83e132740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 0x563c10ed9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.328+0000 7ff83e132740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b04390 0x563c10eda2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.328+0000 7ff83e132740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 0x563c10f51590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.328+0000 7ff83e132740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8301c10c0 con 0x563c10b25070 2024-08-28T19:40:27.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.328+0000 7ff83e132740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8301c09c0 con 0x563c10b04390 2024-08-28T19:40:27.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.328+0000 7ff83e132740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff830102b00 con 0x563c10afc1b0 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.328+0000 7ff83e132740 10 client.0 _ll_register_callbacks cb 0x7ff828009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff835158640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b04390 0x563c10eda2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff835959640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 0x563c10ed9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff835158640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b04390 0x563c10eda2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:36988/0 (socket says 172.21.15.16:36988) 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff83615a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 0x563c10f51590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff835158640 1 -- 172.21.15.16:0/1040375359 learned_addr learned my addr 172.21.15.16:0/1040375359 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff83615a640 1 -- 172.21.15.16:0/1040375359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 msgr2=0x563c10ed9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff83615a640 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 0x563c10ed9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff83615a640 1 -- 172.21.15.16:0/1040375359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b04390 msgr2=0x563c10eda2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff83615a640 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b04390 0x563c10eda2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff83615a640 1 -- 172.21.15.16:0/1040375359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c10edaab0 con 0x563c10b25070 2024-08-28T19:40:27.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff835959640 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 0x563c10ed9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:40:27.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff83615a640 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 0x563c10f51590 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7ff828024c10 tx=0x7ff828009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b094b57b23e3d3c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:27.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.329+0000 7ff8197fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:40:27.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.330+0000 7ff8197fa640 1 -- 172.21.15.16:0/1040375359 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828024630 con 0x563c10b25070 2024-08-28T19:40:27.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.330+0000 7ff8197fa640 1 -- 172.21.15.16:0/1040375359 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7ff8280247d0 con 0x563c10b25070 2024-08-28T19:40:27.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.330+0000 7ff83e132740 1 -- 172.21.15.16:0/1040375359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff82c1099d0 con 0x563c10b25070 2024-08-28T19:40:27.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.330+0000 7ff8197fa640 1 -- 172.21.15.16:0/1040375359 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7ff8280249d0 con 0x563c10b25070 2024-08-28T19:40:27.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.330+0000 7ff83e132740 10 client.19974 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:27.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.330+0000 7ff83e132740 1 -- 172.21.15.16:0/1040375359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x563c10f52a80 con 0x563c10b25070 2024-08-28T19:40:27.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.331+0000 7ff8197fa640 1 -- 172.21.15.16:0/1040375359 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c1099d0 con 0x563c10b25070 2024-08-28T19:40:27.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.331+0000 7ff8197fa640 1 -- 172.21.15.16:0/1040375359 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7ff828019760 con 0x563c10b25070 2024-08-28T19:40:27.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.332+0000 7ff83e132740 10 client.19974 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:27.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.332+0000 7ff83e132740 10 client.19974 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:40:27.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.332+0000 7ff83e132740 20 client.19974 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:27.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.332+0000 7ff83e132740 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563c10ab8da0 0x563c10f552c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:27.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.332+0000 7ff83e132740 4 client.19974 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:40:27.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.332+0000 7ff83e132740 1 -- 172.21.15.16:0/1040375359 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563c10e3f070 con 0x563c10ab8da0 2024-08-28T19:40:27.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.334+0000 7ff835959640 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563c10ab8da0 0x563c10f552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.338+0000 7ff835959640 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563c10ab8da0 0x563c10f552c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=66db7dbf8ccd099b server_cookie=4099c6ca5d2e1d40 in_seq=0 out_seq=0 2024-08-28T19:40:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.339+0000 7ff8197fa640 10 client.19974 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:40:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.339+0000 7ff8197fa640 1 -- 172.21.15.16:0/1040375359 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563c10e3f070 con 0x563c10ab8da0 2024-08-28T19:40:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.339+0000 7ff8197fa640 10 client.19974 handle_command_reply: tid=0 2024-08-28T19:40:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.339+0000 7ff83e132740 1 client.19974 shutdown 2024-08-28T19:40:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.339+0000 7ff83e132740 1 -- 172.21.15.16:0/1040375359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 msgr2=0x563c10f51590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.339+0000 7ff83e132740 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 0x563c10f51590 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7ff828024c10 tx=0x7ff828009870 comp rx=0 tx=0).stop 2024-08-28T19:40:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.340+0000 7ff83e132740 1 -- 172.21.15.16:0/1040375359 shutdown_connections 2024-08-28T19:40:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.340+0000 7ff83e132740 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x563c10ab8da0 0x563c10f552c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.340+0000 7ff83e132740 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c10b25070 0x563c10f51590 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.340+0000 7ff83e132740 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c10b04390 0x563c10eda2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.340+0000 7ff8197fa640 0 client.19974 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:40:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.340+0000 7ff83e132740 1 --2- 172.21.15.16:0/1040375359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563c10afc1b0 0x563c10ed9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.340+0000 7ff83e132740 1 -- 172.21.15.16:0/1040375359 >> 172.21.15.16:0/1040375359 conn(0x563c10b19460 msgr2=0x563c10ecd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.340+0000 7ff83e132740 1 -- 172.21.15.16:0/1040375359 shutdown_connections 2024-08-28T19:40:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.340+0000 7ff83e132740 1 -- 172.21.15.16:0/1040375359 wait complete. 2024-08-28T19:40:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.340+0000 7ff83e132740 20 client.19974 trim_cache size 0 max 16384 2024-08-28T19:40:27.343 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:40:27.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.341+0000 7ff837bbd640 1 -- 172.21.15.16:0/4265015959 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff81c0b4eb0 msgr2=0x7ff81c0b73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:27.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.341+0000 7ff837bbd640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff81c0b4eb0 0x7ff81c0b73c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff82800b760 tx=0x7ff82800e040 comp rx=0 tx=0).stop 2024-08-28T19:40:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.341+0000 7ff837bbd640 1 -- 172.21.15.16:0/4265015959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff830101d80 msgr2=0x7ff8301a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.341+0000 7ff837bbd640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff830101d80 0x7ff8301a08a0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009850 tx=0x7ff82c031bb0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.341+0000 7ff837bbd640 1 -- 172.21.15.16:0/4265015959 shutdown_connections 2024-08-28T19:40:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.341+0000 7ff837bbd640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8301141c0 0x7ff8301a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.341+0000 7ff837bbd640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7ff81c0b4eb0 0x7ff81c0b73c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.341+0000 7ff837bbd640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8301026d0 0x7ff8301a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.341+0000 7ff837bbd640 1 --2- 172.21.15.16:0/4265015959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff830101d80 0x7ff8301a08a0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.341+0000 7ff837bbd640 1 -- 172.21.15.16:0/4265015959 >> 172.21.15.16:0/4265015959 conn(0x7ff830077960 msgr2=0x7ff830112fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.342+0000 7ff837bbd640 1 -- 172.21.15.16:0/4265015959 shutdown_connections 2024-08-28T19:40:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:27.342+0000 7ff837bbd640 1 -- 172.21.15.16:0/4265015959 wait complete. 2024-08-28T19:40:27.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 137556 22.03 MB/sec execute 243 sec latency 139.992 ms 2024-08-28T19:40:27.710 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:27 smithi028 ceph-mon[21769]: pgmap v947: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-08-28T19:40:27.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:27 smithi016 ceph-mon[18556]: pgmap v947: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-08-28T19:40:27.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:27 smithi115 ceph-mon[22184]: pgmap v947: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-08-28T19:40:28.468 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:40:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 138192 22.08 MB/sec execute 244 sec latency 96.561 ms 2024-08-28T19:40:28.799 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:40:29.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.384+0000 7f432dfce640 1 -- 172.21.15.16:0/1513912736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4328111a20 msgr2=0x7f4328113e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:29.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.384+0000 7f432dfce640 1 --2- 172.21.15.16:0/1513912736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4328111a20 0x7f4328113e40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4310009900 tx=0x7f431002a410 comp rx=0 tx=0).stop 2024-08-28T19:40:29.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.384+0000 7f432dfce640 1 -- 172.21.15.16:0/1513912736 shutdown_connections 2024-08-28T19:40:29.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.384+0000 7f432dfce640 1 --2- 172.21.15.16:0/1513912736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4328114380 0x7f43281167a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.384+0000 7f432dfce640 1 --2- 172.21.15.16:0/1513912736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4328111a20 0x7f4328113e40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.384+0000 7f432dfce640 1 --2- 172.21.15.16:0/1513912736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4328101d70 0x7f4328102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.384+0000 7f432dfce640 1 -- 172.21.15.16:0/1513912736 >> 172.21.15.16:0/1513912736 conn(0x7f4328079bb0 msgr2=0x7f4328079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:29.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.384+0000 7f432dfce640 1 -- 172.21.15.16:0/1513912736 shutdown_connections 2024-08-28T19:40:29.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.385+0000 7f432dfce640 1 -- 172.21.15.16:0/1513912736 wait complete. 2024-08-28T19:40:29.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.386+0000 7f432dfce640 1 Processor -- start 2024-08-28T19:40:29.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.386+0000 7f432dfce640 1 -- start start 2024-08-28T19:40:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.386+0000 7f432dfce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4328101d70 0x7f4328111230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.386+0000 7f432dfce640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4328111a20 0x7f432810a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.386+0000 7f432dfce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4328114380 0x7f432810aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.386+0000 7f432dfce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4328119ee0 con 0x7f4328114380 2024-08-28T19:40:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.386+0000 7f432dfce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4328119d60 con 0x7f4328101d70 2024-08-28T19:40:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.386+0000 7f432dfce640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f432811a060 con 0x7f4328111a20 2024-08-28T19:40:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.387+0000 7f43277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4328101d70 0x7f4328111230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.387+0000 7f43277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4328101d70 0x7f4328111230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37020/0 (socket says 172.21.15.16:37020) 2024-08-28T19:40:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.387+0000 7f43277fe640 1 -- 172.21.15.16:0/2371661740 learned_addr learned my addr 172.21.15.16:0/2371661740 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.387+0000 7f4327fff640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4328114380 0x7f432810aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.387+0000 7f4326ffd640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4328111a20 0x7f432810a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.388+0000 7f43277fe640 1 -- 172.21.15.16:0/2371661740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4328111a20 msgr2=0x7f432810a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.388+0000 7f43277fe640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4328111a20 0x7f432810a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.388+0000 7f43277fe640 1 -- 172.21.15.16:0/2371661740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4328114380 msgr2=0x7f432810aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.388+0000 7f43277fe640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4328114380 0x7f432810aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.388+0000 7f43277fe640 1 -- 172.21.15.16:0/2371661740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432810b290 con 0x7f4328101d70 2024-08-28T19:40:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.388+0000 7f43277fe640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4328101d70 0x7f4328111230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f432000d950 tx=0x7f432000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.388+0000 7f4324ff9640 1 -- 172.21.15.16:0/2371661740 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320017030 con 0x7f4328101d70 2024-08-28T19:40:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.388+0000 7f432dfce640 1 -- 172.21.15.16:0/2371661740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f432810ff00 con 0x7f4328101d70 2024-08-28T19:40:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.389+0000 7f432dfce640 1 -- 172.21.15.16:0/2371661740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4328110460 con 0x7f4328101d70 2024-08-28T19:40:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.389+0000 7f4324ff9640 1 -- 172.21.15.16:0/2371661740 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4320004020 con 0x7f4328101d70 2024-08-28T19:40:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.389+0000 7f4324ff9640 1 -- 172.21.15.16:0/2371661740 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43200047a0 con 0x7f4328101d70 2024-08-28T19:40:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.393+0000 7f4334543740 1 Processor -- start 2024-08-28T19:40:29.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.393+0000 7f4334543740 1 -- start start 2024-08-28T19:40:29.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.393+0000 7f4334543740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559483516070 0x5594838bb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.393+0000 7f4334543740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838bb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.393+0000 7f4334543740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 0x5594838bbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.393+0000 7f4334543740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559483832680 con 0x5594834ed1b0 2024-08-28T19:40:29.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.393+0000 7f4334543740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559483821c10 con 0x559483516070 2024-08-28T19:40:29.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.393+0000 7f4334543740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559483825bc0 con 0x5594834f5390 2024-08-28T19:40:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.394+0000 7f4326ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 0x5594838bbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.394+0000 7f4326ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 0x5594838bbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:53202/0 (socket says 172.21.15.16:53202) 2024-08-28T19:40:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.394+0000 7f4326ffd640 1 -- 172.21.15.16:0/3962192204 learned_addr learned my addr 172.21.15.16:0/3962192204 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.394+0000 7f4327fff640 1 --2- 172.21.15.16:0/3962192204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559483516070 0x5594838bb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.395+0000 7f4326ffd640 1 -- 172.21.15.16:0/3962192204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559483516070 msgr2=0x5594838bb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.395+0000 7f4326ffd640 1 --2- 172.21.15.16:0/3962192204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559483516070 0x5594838bb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.395+0000 7f4326ffd640 1 -- 172.21.15.16:0/3962192204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 msgr2=0x5594838bb850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:40:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.395+0000 7f4326ffd640 1 --2- 172.21.15.16:0/3962192204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838bb850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.395+0000 7f4326ffd640 1 -- 172.21.15.16:0/3962192204 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559483581650 con 0x5594834f5390 2024-08-28T19:40:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.395+0000 7f4324ff9640 1 -- 172.21.15.16:0/2371661740 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4320009a30 con 0x7f4328101d70 2024-08-28T19:40:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.395+0000 7f4327fff640 1 --2- 172.21.15.16:0/3962192204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559483516070 0x5594838bb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:40:29.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.395+0000 7f4326ffd640 1 --2- 172.21.15.16:0/3962192204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 0x5594838bbd90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4310005ab0 tx=0x7f4310004180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6afd198bf13dbc75 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:29.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.395+0000 7f431dffb640 1 -- 172.21.15.16:0/3962192204 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310035040 con 0x5594834f5390 2024-08-28T19:40:29.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.396+0000 7f431dffb640 1 -- 172.21.15.16:0/3962192204 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43100054f0 con 0x5594834f5390 2024-08-28T19:40:29.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.396+0000 7f431dffb640 1 -- 172.21.15.16:0/3962192204 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310003850 con 0x5594834f5390 2024-08-28T19:40:29.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.396+0000 7f4324ff9640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43140b4d20 0x7f43140b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.396+0000 7f4327fff640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43140b4d20 0x7f43140b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.396+0000 7f4324ff9640 1 -- 172.21.15.16:0/2371661740 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7f43200e5150 con 0x7f4328101d70 2024-08-28T19:40:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.397+0000 7f4327fff640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43140b4d20 0x7f43140b7230 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4318005fd0 tx=0x7f4318009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.397+0000 7f4334543740 1 -- 172.21.15.16:0/3962192204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 msgr2=0x5594838bbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.397+0000 7f4334543740 1 --2- 172.21.15.16:0/3962192204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 0x5594838bbd90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4310005ab0 tx=0x7f4310004180 comp rx=0 tx=0).stop 2024-08-28T19:40:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.397+0000 7f4334543740 1 -- 172.21.15.16:0/3962192204 shutdown_connections 2024-08-28T19:40:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.397+0000 7f4334543740 1 --2- 172.21.15.16:0/3962192204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 0x5594838bbd90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.397+0000 7f4334543740 1 --2- 172.21.15.16:0/3962192204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838bb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.397+0000 7f4334543740 1 --2- 172.21.15.16:0/3962192204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559483516070 0x5594838bb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.397+0000 7f4334543740 1 -- 172.21.15.16:0/3962192204 >> 172.21.15.16:0/3962192204 conn(0x55948350a460 msgr2=0x5594838b9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.397+0000 7f4334543740 1 -- 172.21.15.16:0/3962192204 shutdown_connections 2024-08-28T19:40:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.397+0000 7f4334543740 1 -- 172.21.15.16:0/3962192204 wait complete. 2024-08-28T19:40:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.398+0000 7f4334543740 1 -- start start 2024-08-28T19:40:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.398+0000 7f4334543740 1 Processor -- start 2024-08-28T19:40:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.398+0000 7f4334543740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f4334543740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834f5390 0x5594838c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f4334543740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559483516070 0x5594838d2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f4334543740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559483684700 con 0x5594834ed1b0 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f4334543740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5594837a8750 con 0x5594834f5390 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f4334543740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5594838c9b30 con 0x559483516070 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f43277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f4334543740 10 client.0 _ll_register_callbacks cb 0x5594837d7e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f43277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838c6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45818/0 (socket says 172.21.15.16:45818) 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f43277fe640 1 -- 172.21.15.16:0/3511678778 learned_addr learned my addr 172.21.15.16:0/3511678778 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f4326ffd640 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834f5390 0x5594838c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f4327fff640 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559483516070 0x5594838d2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f43277fe640 1 -- 172.21.15.16:0/3511678778 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559483516070 msgr2=0x5594838d2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f43277fe640 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559483516070 0x5594838d2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f43277fe640 1 -- 172.21.15.16:0/3511678778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834f5390 msgr2=0x5594838c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f43277fe640 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834f5390 0x5594838c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f43277fe640 1 -- 172.21.15.16:0/3511678778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594837e1900 con 0x5594834ed1b0 2024-08-28T19:40:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f43277fe640 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838c6a00 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f43200e2ad0 tx=0x7f4320009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecdc5be73af7c11 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.399+0000 7f4306ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.400+0000 7f4306ffd640 1 -- 172.21.15.16:0/3511678778 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43200b4790 con 0x5594834ed1b0 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.400+0000 7f4306ffd640 1 -- 172.21.15.16:0/3511678778 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7f43200a7700 con 0x5594834ed1b0 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.400+0000 7f4334543740 1 -- 172.21.15.16:0/3511678778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43100054f0 con 0x5594834ed1b0 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.400+0000 7f4306ffd640 1 -- 172.21.15.16:0/3511678778 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f43200d20c0 con 0x5594834ed1b0 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.400+0000 7f4334543740 10 client.19983 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.400+0000 7f4334543740 1 -- 172.21.15.16:0/3511678778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5594838d5870 con 0x5594834ed1b0 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.400+0000 7f4306ffd640 1 -- 172.21.15.16:0/3511678778 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f432005b1a0 con 0x5594834ed1b0 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.400+0000 7f4306ffd640 1 -- 172.21.15.16:0/3511678778 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f43200ccbe0 con 0x5594834ed1b0 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.400+0000 7f4334543740 10 client.19983 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.400+0000 7f4334543740 10 client.19983 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.401+0000 7f4334543740 20 client.19983 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.401+0000 7f4334543740 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5594834d49f0 0x5594838d92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.401+0000 7f4334543740 4 client.19983 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:40:29.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.401+0000 7f4334543740 1 -- 172.21.15.16:0/3511678778 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f43100054f0 con 0x5594834d49f0 2024-08-28T19:40:29.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.401+0000 7f4326ffd640 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5594834d49f0 0x5594838d92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.402+0000 7f4326ffd640 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5594834d49f0 0x5594838d92a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=111f5c387f41e50c server_cookie=9aa4af7ef96df41 in_seq=0 out_seq=0 2024-08-28T19:40:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.402+0000 7f4306ffd640 10 client.19983 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:40:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.403+0000 7f4306ffd640 1 -- 172.21.15.16:0/3511678778 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f43100054f0 con 0x5594834d49f0 2024-08-28T19:40:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.403+0000 7f4306ffd640 10 client.19983 handle_command_reply: tid=0 2024-08-28T19:40:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.403+0000 7f4334543740 1 client.19983 shutdown 2024-08-28T19:40:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4334543740 1 -- 172.21.15.16:0/3511678778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 msgr2=0x5594838c6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4334543740 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838c6a00 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f43200e2ad0 tx=0x7f4320009f50 comp rx=0 tx=0).stop 2024-08-28T19:40:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4334543740 1 -- 172.21.15.16:0/3511678778 shutdown_connections 2024-08-28T19:40:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4334543740 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5594834d49f0 0x5594838d92a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4334543740 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559483516070 0x5594838d2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4334543740 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834f5390 0x5594838c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4334543740 1 --2- 172.21.15.16:0/3511678778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838c6a00 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4334543740 1 -- 172.21.15.16:0/3511678778 >> 172.21.15.16:0/3511678778 conn(0x55948350a460 msgr2=0x5594838be4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4306ffd640 0 client.19983 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:40:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4334543740 1 -- 172.21.15.16:0/3511678778 shutdown_connections 2024-08-28T19:40:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4334543740 1 -- 172.21.15.16:0/3511678778 wait complete. 2024-08-28T19:40:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.404+0000 7f4334543740 20 client.19983 trim_cache size 0 max 16384 2024-08-28T19:40:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.421+0000 7f4334543740 1 Processor -- start 2024-08-28T19:40:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.421+0000 7f4334543740 1 -- start start 2024-08-28T19:40:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f4334543740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834ed1b0 0x5594838c5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f4334543740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 0x5594838c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f4334543740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559483516070 0x5594838be410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f43277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834ed1b0 0x5594838c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f43277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834ed1b0 0x5594838c5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37052/0 (socket says 172.21.15.16:37052) 2024-08-28T19:40:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f4326ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 0x5594838c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f43277fe640 1 -- 172.21.15.16:0/872877823 learned_addr learned my addr 172.21.15.16:0/872877823 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f4334543740 1 -- 172.21.15.16:0/872877823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4308010050 con 0x559483516070 2024-08-28T19:40:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f4334543740 1 -- 172.21.15.16:0/872877823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4308011d10 con 0x5594834ed1b0 2024-08-28T19:40:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f4334543740 1 -- 172.21.15.16:0/872877823 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f43080107b0 con 0x5594834f5390 2024-08-28T19:40:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f43277fe640 1 -- 172.21.15.16:0/872877823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 msgr2=0x5594838c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f43277fe640 1 --2- 172.21.15.16:0/872877823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 0x5594838c57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f43277fe640 1 -- 172.21.15.16:0/872877823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559483516070 msgr2=0x5594838be410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f43277fe640 1 --2- 172.21.15.16:0/872877823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559483516070 0x5594838be410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.422+0000 7f43277fe640 1 -- 172.21.15.16:0/872877823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594838c7560 con 0x5594834ed1b0 2024-08-28T19:40:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.423+0000 7f43277fe640 1 --2- 172.21.15.16:0/872877823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834ed1b0 0x5594838c5280 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4320092d20 tx=0x7f43200afe60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d58e11866e0c4fec server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:29.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.423+0000 7f431d7fa640 1 -- 172.21.15.16:0/872877823 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43200aaf80 con 0x5594834ed1b0 2024-08-28T19:40:29.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.423+0000 7f431d7fa640 1 -- 172.21.15.16:0/872877823 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43200b4790 con 0x5594834ed1b0 2024-08-28T19:40:29.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.423+0000 7f431d7fa640 1 -- 172.21.15.16:0/872877823 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43200a7700 con 0x5594834ed1b0 2024-08-28T19:40:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.424+0000 7f4334543740 1 -- 172.21.15.16:0/872877823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834ed1b0 msgr2=0x5594838c5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.424+0000 7f4334543740 1 --2- 172.21.15.16:0/872877823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834ed1b0 0x5594838c5280 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4320092d20 tx=0x7f43200afe60 comp rx=0 tx=0).stop 2024-08-28T19:40:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.424+0000 7f4334543740 1 -- 172.21.15.16:0/872877823 shutdown_connections 2024-08-28T19:40:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.424+0000 7f4334543740 1 --2- 172.21.15.16:0/872877823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559483516070 0x5594838be410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.424+0000 7f4334543740 1 --2- 172.21.15.16:0/872877823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5594834f5390 0x5594838c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.424+0000 7f4334543740 1 --2- 172.21.15.16:0/872877823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834ed1b0 0x5594838c5280 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.424+0000 7f4334543740 1 -- 172.21.15.16:0/872877823 >> 172.21.15.16:0/872877823 conn(0x55948350a460 msgr2=0x5594838c6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.424+0000 7f4334543740 1 -- 172.21.15.16:0/872877823 shutdown_connections 2024-08-28T19:40:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.424+0000 7f4334543740 1 -- 172.21.15.16:0/872877823 wait complete. 2024-08-28T19:40:29.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.425+0000 7f4334543740 1 -- start start 2024-08-28T19:40:29.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.425+0000 7f4334543740 1 Processor -- start 2024-08-28T19:40:29.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f4334543740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838cf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f4334543740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834f5390 0x5594838caa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f4334543740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559483516070 0x5594838caf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f4334543740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4308011580 con 0x5594834ed1b0 2024-08-28T19:40:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f4334543740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4308010e80 con 0x5594834f5390 2024-08-28T19:40:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f4334543740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f430800e050 con 0x559483516070 2024-08-28T19:40:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f4334543740 10 client.0 _ll_register_callbacks cb 0x7f43200a7330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f43277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838cf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f43277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838cf560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45840/0 (socket says 172.21.15.16:45840) 2024-08-28T19:40:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f43277fe640 1 -- 172.21.15.16:0/1068628263 learned_addr learned my addr 172.21.15.16:0/1068628263 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f4326ffd640 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834f5390 0x5594838caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.426+0000 7f4327fff640 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559483516070 0x5594838caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.427+0000 7f4326ffd640 1 -- 172.21.15.16:0/1068628263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559483516070 msgr2=0x5594838caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.427+0000 7f4326ffd640 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559483516070 0x5594838caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.427+0000 7f4326ffd640 1 -- 172.21.15.16:0/1068628263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 msgr2=0x5594838cf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.427+0000 7f4326ffd640 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838cf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.427+0000 7f4326ffd640 1 -- 172.21.15.16:0/1068628263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594838cb640 con 0x5594834f5390 2024-08-28T19:40:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.427+0000 7f4326ffd640 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834f5390 0x5594838caa10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f431002a930 tx=0x7f4310004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38e033bb2b48a457 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.427+0000 7f4306ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.428+0000 7f4306ffd640 1 -- 172.21.15.16:0/1068628263 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310035040 con 0x5594834f5390 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.428+0000 7f4306ffd640 1 -- 172.21.15.16:0/1068628263 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7f4310005060 con 0x5594834f5390 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.428+0000 7f4334543740 1 -- 172.21.15.16:0/1068628263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43200b4790 con 0x5594834f5390 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.428+0000 7f4306ffd640 1 -- 172.21.15.16:0/1068628263 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f4310060050 con 0x5594834f5390 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.428+0000 7f4334543740 10 client.29435 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.428+0000 7f4334543740 1 -- 172.21.15.16:0/1068628263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x559483943ba0 con 0x5594834f5390 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.429+0000 7f4306ffd640 1 -- 172.21.15.16:0/1068628263 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43200b4790 con 0x5594834f5390 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.429+0000 7f4306ffd640 1 -- 172.21.15.16:0/1068628263 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f4310003c30 con 0x5594834f5390 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.429+0000 7f4334543740 10 client.29435 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.429+0000 7f4334543740 10 client.29435 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.429+0000 7f4334543740 20 client.29435 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.429+0000 7f4334543740 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5594834a9da0 0x559483945ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.429+0000 7f4334543740 4 client.29435 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:40:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.429+0000 7f4334543740 1 -- 172.21.15.16:0/1068628263 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5594838345d0 con 0x5594834a9da0 2024-08-28T19:40:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.429+0000 7f43277fe640 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5594834a9da0 0x559483945ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.430+0000 7f43277fe640 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5594834a9da0 0x559483945ef0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=81566486986c3297 server_cookie=aaa9995e1e842b6b in_seq=0 out_seq=0 2024-08-28T19:40:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.430+0000 7f4306ffd640 10 client.29435 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:40:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4306ffd640 1 -- 172.21.15.16:0/1068628263 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5594838345d0 con 0x5594834a9da0 2024-08-28T19:40:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4306ffd640 10 client.29435 handle_command_reply: tid=0 2024-08-28T19:40:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4334543740 1 client.29435 shutdown 2024-08-28T19:40:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4334543740 1 -- 172.21.15.16:0/1068628263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834f5390 msgr2=0x5594838caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4334543740 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834f5390 0x5594838caa10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f431002a930 tx=0x7f4310004960 comp rx=0 tx=0).stop 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4334543740 1 -- 172.21.15.16:0/1068628263 shutdown_connections 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4334543740 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x5594834a9da0 0x559483945ef0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4334543740 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559483516070 0x5594838caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4334543740 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594834f5390 0x5594838caa10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4334543740 1 --2- 172.21.15.16:0/1068628263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594834ed1b0 0x5594838cf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4334543740 1 -- 172.21.15.16:0/1068628263 >> 172.21.15.16:0/1068628263 conn(0x55948350a460 msgr2=0x5594838c6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4306ffd640 0 client.29435 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4334543740 1 -- 172.21.15.16:0/1068628263 shutdown_connections 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.431+0000 7f4334543740 1 -- 172.21.15.16:0/1068628263 wait complete. 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.432+0000 7f4334543740 20 client.29435 trim_cache size 0 max 16384 2024-08-28T19:40:29.434 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:40:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.432+0000 7f432dfce640 1 -- 172.21.15.16:0/2371661740 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43140b4d20 msgr2=0x7f43140b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.432+0000 7f432dfce640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43140b4d20 0x7f43140b7230 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4318005fd0 tx=0x7f4318009600 comp rx=0 tx=0).stop 2024-08-28T19:40:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.432+0000 7f432dfce640 1 -- 172.21.15.16:0/2371661740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4328101d70 msgr2=0x7f4328111230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.432+0000 7f432dfce640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4328101d70 0x7f4328111230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f432000d950 tx=0x7f432000de10 comp rx=0 tx=0).stop 2024-08-28T19:40:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.432+0000 7f432dfce640 1 -- 172.21.15.16:0/2371661740 shutdown_connections 2024-08-28T19:40:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.432+0000 7f432dfce640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4328114380 0x7f432810aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.432+0000 7f432dfce640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4328111a20 0x7f432810a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.432+0000 7f432dfce640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f43140b4d20 0x7f43140b7230 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.432+0000 7f432dfce640 1 --2- 172.21.15.16:0/2371661740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4328101d70 0x7f4328111230 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.432+0000 7f432dfce640 1 -- 172.21.15.16:0/2371661740 >> 172.21.15.16:0/2371661740 conn(0x7f4328079bb0 msgr2=0x7f4328111e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.433+0000 7f432dfce640 1 -- 172.21.15.16:0/2371661740 shutdown_connections 2024-08-28T19:40:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:29.433+0000 7f432dfce640 1 -- 172.21.15.16:0/2371661740 wait complete. 2024-08-28T19:40:29.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 138687 22.07 MB/sec execute 245 sec latency 127.384 ms 2024-08-28T19:40:29.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:29 smithi016 ceph-mon[18556]: pgmap v948: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 2.5 MiB/s wr, 179 op/s 2024-08-28T19:40:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:29 smithi028 ceph-mon[21769]: pgmap v948: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 2.5 MiB/s wr, 179 op/s 2024-08-28T19:40:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:29 smithi115 ceph-mon[22184]: pgmap v948: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 2.5 MiB/s wr, 179 op/s 2024-08-28T19:40:30.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:40:30.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 139559 22.17 MB/sec execute 246 sec latency 86.791 ms 2024-08-28T19:40:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:40:31.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.219+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1045157734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8a810db90 msgr2=0x7fe8a81100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:31.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.219+0000 7fe8aed6c640 1 --2- 172.21.15.16:0/1045157734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8a810db90 0x7fe8a81100a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009ca0 tx=0x7fe8a002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.219+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1045157734 shutdown_connections 2024-08-28T19:40:31.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.219+0000 7fe8aed6c640 1 --2- 172.21.15.16:0/1045157734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8a810db90 0x7fe8a81100a0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.219+0000 7fe8aed6c640 1 --2- 172.21.15.16:0/1045157734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a8105010 0x7fe8a810d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.219+0000 7fe8aed6c640 1 --2- 172.21.15.16:0/1045157734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8a81045e0 0x7fe8a81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.219+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1045157734 >> 172.21.15.16:0/1045157734 conn(0x7fe8a80794d0 msgr2=0x7fe8a80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:31.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.219+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1045157734 shutdown_connections 2024-08-28T19:40:31.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.219+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1045157734 wait complete. 2024-08-28T19:40:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.220+0000 7fe8aed6c640 1 Processor -- start 2024-08-28T19:40:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.220+0000 7fe8aed6c640 1 -- start start 2024-08-28T19:40:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.220+0000 7fe8aed6c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8a81045e0 0x7fe8a819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.220+0000 7fe8aed6c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8a8105010 0x7fe8a819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.220+0000 7fe8aed6c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a810db90 0x7fe8a81a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.220+0000 7fe8aed6c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a8113810 con 0x7fe8a8105010 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.220+0000 7fe8aed6c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a8113690 con 0x7fe8a810db90 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.220+0000 7fe8aed6c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a8113990 con 0x7fe8a81045e0 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8a8105010 0x7fe8a819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8a8105010 0x7fe8a819ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45852/0 (socket says 172.21.15.16:45852) 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8a7fff640 1 -- 172.21.15.16:0/1787759083 learned_addr learned my addr 172.21.15.16:0/1787759083 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8ad309640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a810db90 0x7fe8a81a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8acb08640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8a81045e0 0x7fe8a819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8ad309640 1 -- 172.21.15.16:0/1787759083 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8a81045e0 msgr2=0x7fe8a819c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8ad309640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8a81045e0 0x7fe8a819c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8ad309640 1 -- 172.21.15.16:0/1787759083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8a8105010 msgr2=0x7fe8a819ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8ad309640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8a8105010 0x7fe8a819ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8ad309640 1 -- 172.21.15.16:0/1787759083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a81a23f0 con 0x7fe8a810db90 2024-08-28T19:40:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8ad309640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a810db90 0x7fe8a81a1cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe8a002ade0 tx=0x7fe8a00316f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8a5ffb640 1 -- 172.21.15.16:0/1787759083 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0003b70 con 0x7fe8a810db90 2024-08-28T19:40:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8a5ffb640 1 -- 172.21.15.16:0/1787759083 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8a0003d10 con 0x7fe8a810db90 2024-08-28T19:40:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.221+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1787759083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a81a2680 con 0x7fe8a810db90 2024-08-28T19:40:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.222+0000 7fe8a5ffb640 1 -- 172.21.15.16:0/1787759083 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a002f9e0 con 0x7fe8a810db90 2024-08-28T19:40:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.222+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1787759083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a81a8e40 con 0x7fe8a810db90 2024-08-28T19:40:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.224+0000 7fe8b52e1740 1 Processor -- start 2024-08-28T19:40:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.225+0000 7fe8a5ffb640 1 -- 172.21.15.16:0/1787759083 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe8a0004040 con 0x7fe8a810db90 2024-08-28T19:40:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.225+0000 7fe8b52e1740 1 -- start start 2024-08-28T19:40:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.226+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd2ac070 0x55d9cd651310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.226+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 0x55d9cd651850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.226+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd28b390 0x55d9cd651d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.226+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d9cd5c8680 con 0x55d9cd2831b0 2024-08-28T19:40:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.226+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d9cd5b7c10 con 0x55d9cd28b390 2024-08-28T19:40:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.226+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d9cd5bbbc0 con 0x55d9cd2ac070 2024-08-28T19:40:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.226+0000 7fe8acb08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 0x55d9cd651850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.226+0000 7fe8a7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd2ac070 0x55d9cd651310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.226+0000 7fe8acb08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 0x55d9cd651850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45862/0 (socket says 172.21.15.16:45862) 2024-08-28T19:40:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.226+0000 7fe8acb08640 1 -- 172.21.15.16:0/3308453504 learned_addr learned my addr 172.21.15.16:0/3308453504 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.226+0000 7fe8a7fff640 1 --2- 172.21.15.16:0/3308453504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd28b390 0x55d9cd651d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.227+0000 7fe8acb08640 1 -- 172.21.15.16:0/3308453504 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd2ac070 msgr2=0x55d9cd651310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.227+0000 7fe8acb08640 1 --2- 172.21.15.16:0/3308453504 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd2ac070 0x55d9cd651310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.227+0000 7fe8acb08640 1 -- 172.21.15.16:0/3308453504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd28b390 msgr2=0x55d9cd651d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.227+0000 7fe8acb08640 1 --2- 172.21.15.16:0/3308453504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd28b390 0x55d9cd651d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.227+0000 7fe8a5ffb640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8980b4d20 0x7fe8980b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.227+0000 7fe8acb08640 1 -- 172.21.15.16:0/3308453504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9cd317650 con 0x55d9cd2831b0 2024-08-28T19:40:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.227+0000 7fe8a7fff640 1 --2- 172.21.15.16:0/3308453504 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd2ac070 0x55d9cd651310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:40:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.227+0000 7fe8a5ffb640 1 -- 172.21.15.16:0/1787759083 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fe8a0106a40 con 0x7fe8a810db90 2024-08-28T19:40:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.227+0000 7fe8a7fff640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8980b4d20 0x7fe8980b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.228+0000 7fe8acb08640 1 --2- 172.21.15.16:0/3308453504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 0x55d9cd651850 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fe89c00bcb0 tx=0x7fe89c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b86f36063a2936 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.228+0000 7fe892ffd640 1 -- 172.21.15.16:0/3308453504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89c004020 con 0x55d9cd2831b0 2024-08-28T19:40:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.228+0000 7fe8a7fff640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8980b4d20 0x7fe8980b7230 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe894009fd0 tx=0x7fe894004730 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.228+0000 7fe892ffd640 1 -- 172.21.15.16:0/3308453504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe89c0054d0 con 0x55d9cd2831b0 2024-08-28T19:40:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.228+0000 7fe892ffd640 1 -- 172.21.15.16:0/3308453504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89c004cb0 con 0x55d9cd2831b0 2024-08-28T19:40:31.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.229+0000 7fe8b52e1740 1 -- 172.21.15.16:0/3308453504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 msgr2=0x55d9cd651850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:31.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.229+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/3308453504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 0x55d9cd651850 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fe89c00bcb0 tx=0x7fe89c0107a0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.229+0000 7fe8b52e1740 1 -- 172.21.15.16:0/3308453504 shutdown_connections 2024-08-28T19:40:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.229+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/3308453504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd28b390 0x55d9cd651d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.229+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/3308453504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 0x55d9cd651850 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.229+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/3308453504 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd2ac070 0x55d9cd651310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.229+0000 7fe8b52e1740 1 -- 172.21.15.16:0/3308453504 >> 172.21.15.16:0/3308453504 conn(0x55d9cd2a0460 msgr2=0x55d9cd64f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.229+0000 7fe8b52e1740 1 -- 172.21.15.16:0/3308453504 shutdown_connections 2024-08-28T19:40:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.229+0000 7fe8b52e1740 1 -- 172.21.15.16:0/3308453504 wait complete. 2024-08-28T19:40:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.230+0000 7fe8b52e1740 1 -- start start 2024-08-28T19:40:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.230+0000 7fe8b52e1740 1 Processor -- start 2024-08-28T19:40:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 0x55d9cd665b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd666140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 0x55d9cd66b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d9cd41a700 con 0x55d9cd2ac070 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d9cd53e750 con 0x55d9cd2831b0 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d9cd65faf0 con 0x55d9cd28b390 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8a7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd666140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8b52e1740 10 client.0 _ll_register_callbacks cb 0x55d9cd56de10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8a7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd666140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:53258/0 (socket says 172.21.15.16:53258) 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8a7fff640 1 -- 172.21.15.16:0/1696389219 learned_addr learned my addr 172.21.15.16:0/1696389219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8acb08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 0x55d9cd665b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8ad309640 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 0x55d9cd66b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8a7fff640 1 -- 172.21.15.16:0/1696389219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 msgr2=0x55d9cd665b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8a7fff640 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 0x55d9cd665b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8a7fff640 1 -- 172.21.15.16:0/1696389219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 msgr2=0x55d9cd66b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8a7fff640 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 0x55d9cd66b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.231+0000 7fe8a7fff640 1 -- 172.21.15.16:0/1696389219 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9cd577900 con 0x55d9cd28b390 2024-08-28T19:40:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.232+0000 7fe8acb08640 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 0x55d9cd665b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:40:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.232+0000 7fe8a7fff640 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd666140 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe894009d90 tx=0x7fe8940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d281a6dd31e3cf2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.232+0000 7fe887fff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:40:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.232+0000 7fe887fff640 1 -- 172.21.15.16:0/1696389219 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894030060 con 0x55d9cd28b390 2024-08-28T19:40:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.232+0000 7fe887fff640 1 -- 172.21.15.16:0/1696389219 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fe89401a050 con 0x55d9cd28b390 2024-08-28T19:40:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.232+0000 7fe8b52e1740 1 -- 172.21.15.16:0/1696389219 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe89c0054d0 con 0x55d9cd28b390 2024-08-28T19:40:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.233+0000 7fe887fff640 1 -- 172.21.15.16:0/1696389219 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fe894037050 con 0x55d9cd28b390 2024-08-28T19:40:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.233+0000 7fe8b52e1740 10 client.29017 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.233+0000 7fe8b52e1740 1 -- 172.21.15.16:0/1696389219 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55d9cd674110 con 0x55d9cd28b390 2024-08-28T19:40:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.233+0000 7fe887fff640 1 -- 172.21.15.16:0/1696389219 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe89c0054d0 con 0x55d9cd28b390 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.233+0000 7fe887fff640 1 -- 172.21.15.16:0/1696389219 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fe89402b9e0 con 0x55d9cd28b390 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.233+0000 7fe8b52e1740 10 client.29017 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.233+0000 7fe8b52e1740 10 client.29017 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.233+0000 7fe8b52e1740 20 client.29017 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.233+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d9cd26a9f0 0x55d9cd6776c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.233+0000 7fe8b52e1740 4 client.29017 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.233+0000 7fe8b52e1740 1 -- 172.21.15.16:0/1696389219 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9cd5ca5d0 con 0x55d9cd26a9f0 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.234+0000 7fe8acb08640 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d9cd26a9f0 0x55d9cd6776c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.234+0000 7fe887fff640 10 client.29017 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.234+0000 7fe8acb08640 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d9cd26a9f0 0x55d9cd6776c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cc39fd27b551c5e4 server_cookie=1a5b40885066dda5 in_seq=0 out_seq=0 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.235+0000 7fe887fff640 1 -- 172.21.15.16:0/1696389219 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d9cd5ca5d0 con 0x55d9cd26a9f0 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.235+0000 7fe887fff640 10 client.29017 handle_command_reply: tid=0 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.235+0000 7fe8b52e1740 1 client.29017 shutdown 2024-08-28T19:40:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.235+0000 7fe8b52e1740 1 -- 172.21.15.16:0/1696389219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 msgr2=0x55d9cd666140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.235+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd666140 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe894009d90 tx=0x7fe8940046c0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.235+0000 7fe8b52e1740 1 -- 172.21.15.16:0/1696389219 shutdown_connections 2024-08-28T19:40:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.235+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d9cd26a9f0 0x55d9cd6776c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.235+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 0x55d9cd66b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.235+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd666140 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.235+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/1696389219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 0x55d9cd665b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.236+0000 7fe8b52e1740 1 -- 172.21.15.16:0/1696389219 >> 172.21.15.16:0/1696389219 conn(0x55d9cd2a0460 msgr2=0x55d9cd6544a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.236+0000 7fe887fff640 0 client.29017 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:40:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.236+0000 7fe8b52e1740 1 -- 172.21.15.16:0/1696389219 shutdown_connections 2024-08-28T19:40:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.236+0000 7fe8b52e1740 1 -- 172.21.15.16:0/1696389219 wait complete. 2024-08-28T19:40:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.236+0000 7fe8b52e1740 20 client.29017 trim_cache size 0 max 16384 2024-08-28T19:40:31.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.251+0000 7fe8b52e1740 1 Processor -- start 2024-08-28T19:40:31.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.251+0000 7fe8b52e1740 1 -- start start 2024-08-28T19:40:31.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.251+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 0x55d9cd64f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.251+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd65ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.251+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 0x55d9cd652340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.251+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a81aeb90 con 0x55d9cd2ac070 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.251+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a81b0850 con 0x55d9cd2831b0 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.251+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a81af2f0 con 0x55d9cd28b390 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8ad309640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 0x55d9cd652340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8a7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd65ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8ad309640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 0x55d9cd652340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45870/0 (socket says 172.21.15.16:45870) 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8ad309640 1 -- 172.21.15.16:0/3463137730 learned_addr learned my addr 172.21.15.16:0/3463137730 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8acb08640 1 --2- 172.21.15.16:0/3463137730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 0x55d9cd64f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8ad309640 1 -- 172.21.15.16:0/3463137730 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 msgr2=0x55d9cd65ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8ad309640 1 --2- 172.21.15.16:0/3463137730 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd65ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8ad309640 1 -- 172.21.15.16:0/3463137730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 msgr2=0x55d9cd64f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8ad309640 1 --2- 172.21.15.16:0/3463137730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 0x55d9cd64f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8ad309640 1 -- 172.21.15.16:0/3463137730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9cd64a960 con 0x55d9cd2ac070 2024-08-28T19:40:31.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8acb08640 1 --2- 172.21.15.16:0/3463137730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 0x55d9cd64f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:40:31.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8ad309640 1 --2- 172.21.15.16:0/3463137730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 0x55d9cd652340 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0006010 tx=0x7fe8a002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb4ada6a117bae8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:31.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8927fc640 1 -- 172.21.15.16:0/3463137730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a00cd400 con 0x55d9cd2ac070 2024-08-28T19:40:31.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8927fc640 1 -- 172.21.15.16:0/3463137730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8a0036070 con 0x55d9cd2ac070 2024-08-28T19:40:31.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.252+0000 7fe8927fc640 1 -- 172.21.15.16:0/3463137730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a00b4290 con 0x55d9cd2ac070 2024-08-28T19:40:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.253+0000 7fe8b52e1740 1 -- 172.21.15.16:0/3463137730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 msgr2=0x55d9cd652340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.253+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/3463137730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 0x55d9cd652340 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0006010 tx=0x7fe8a002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.254+0000 7fe8b52e1740 1 -- 172.21.15.16:0/3463137730 shutdown_connections 2024-08-28T19:40:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.254+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/3463137730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2ac070 0x55d9cd652340 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.254+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/3463137730 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd65ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.254+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/3463137730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2831b0 0x55d9cd64f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.254+0000 7fe8b52e1740 1 -- 172.21.15.16:0/3463137730 >> 172.21.15.16:0/3463137730 conn(0x55d9cd2a0460 msgr2=0x55d9cd64eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.254+0000 7fe8b52e1740 1 -- 172.21.15.16:0/3463137730 shutdown_connections 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.254+0000 7fe8b52e1740 1 -- 172.21.15.16:0/3463137730 wait complete. 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.254+0000 7fe8b52e1740 1 -- start start 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.254+0000 7fe8b52e1740 1 Processor -- start 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 0x55d9cd670eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd6713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8b52e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2ac070 0x55d9cd671970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a81b00c0 con 0x55d9cd2831b0 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a81af9c0 con 0x55d9cd2ac070 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8b52e1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a81acbd0 con 0x55d9cd28b390 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8b52e1740 10 client.0 _ll_register_callbacks cb 0x7fe894009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8a7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd6713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8a7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd6713f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:53276/0 (socket says 172.21.15.16:53276) 2024-08-28T19:40:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8a7fff640 1 -- 172.21.15.16:0/4280340211 learned_addr learned my addr 172.21.15.16:0/4280340211 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8acb08640 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 0x55d9cd670eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8ad309640 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2ac070 0x55d9cd671970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8a7fff640 1 -- 172.21.15.16:0/4280340211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2ac070 msgr2=0x55d9cd671970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8a7fff640 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2ac070 0x55d9cd671970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8a7fff640 1 -- 172.21.15.16:0/4280340211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 msgr2=0x55d9cd670eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8a7fff640 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 0x55d9cd670eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.255+0000 7fe8a7fff640 1 -- 172.21.15.16:0/4280340211 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9cd655420 con 0x55d9cd28b390 2024-08-28T19:40:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.256+0000 7fe8a7fff640 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd6713f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe8940058c0 tx=0x7fe894023900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69dd805972b69b35 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.256+0000 7fe887fff640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:40:31.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.256+0000 7fe887fff640 1 -- 172.21.15.16:0/4280340211 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894023da0 con 0x55d9cd28b390 2024-08-28T19:40:31.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.256+0000 7fe887fff640 1 -- 172.21.15.16:0/4280340211 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fe894011040 con 0x55d9cd28b390 2024-08-28T19:40:31.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.256+0000 7fe8b52e1740 1 -- 172.21.15.16:0/4280340211 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8a0036070 con 0x55d9cd28b390 2024-08-28T19:40:31.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.256+0000 7fe887fff640 1 -- 172.21.15.16:0/4280340211 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fe894011240 con 0x55d9cd28b390 2024-08-28T19:40:31.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.256+0000 7fe8b52e1740 10 client.29023 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:31.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.256+0000 7fe8b52e1740 1 -- 172.21.15.16:0/4280340211 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55d9cd6e23b0 con 0x55d9cd28b390 2024-08-28T19:40:31.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.257+0000 7fe887fff640 1 -- 172.21.15.16:0/4280340211 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8a0036070 con 0x55d9cd28b390 2024-08-28T19:40:31.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.257+0000 7fe887fff640 1 -- 172.21.15.16:0/4280340211 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fe8940209c0 con 0x55d9cd28b390 2024-08-28T19:40:31.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.257+0000 7fe8b52e1740 10 client.29023 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:31.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.257+0000 7fe8b52e1740 10 client.29023 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:40:31.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.257+0000 7fe8b52e1740 20 client.29023 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:31.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.257+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d9cd23fda0 0x55d9cd6e4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:31.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.257+0000 7fe8b52e1740 4 client.29023 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.257+0000 7fe8b52e1740 1 -- 172.21.15.16:0/4280340211 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9cd5c6070 con 0x55d9cd23fda0 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.257+0000 7fe8acb08640 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d9cd23fda0 0x55d9cd6e4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe887fff640 10 client.29023 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe8acb08640 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d9cd23fda0 0x55d9cd6e4b40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a8a84eea610b897a server_cookie=3c8f0293afa93c33 in_seq=0 out_seq=0 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe887fff640 1 -- 172.21.15.16:0/4280340211 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9cd5c6070 con 0x55d9cd23fda0 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe887fff640 10 client.29023 handle_command_reply: tid=0 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe8b52e1740 1 client.29023 shutdown 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe8b52e1740 1 -- 172.21.15.16:0/4280340211 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 msgr2=0x55d9cd6713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd6713f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe8940058c0 tx=0x7fe894023900 comp rx=0 tx=0).stop 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe8b52e1740 1 -- 172.21.15.16:0/4280340211 shutdown_connections 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55d9cd23fda0 0x55d9cd6e4b40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9cd2ac070 0x55d9cd671970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d9cd28b390 0x55d9cd6713f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe8b52e1740 1 --2- 172.21.15.16:0/4280340211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9cd2831b0 0x55d9cd670eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe887fff640 0 client.29023 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:40:31.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe8b52e1740 1 -- 172.21.15.16:0/4280340211 >> 172.21.15.16:0/4280340211 conn(0x55d9cd2a0460 msgr2=0x55d9cd64d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:31.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.258+0000 7fe8b52e1740 1 -- 172.21.15.16:0/4280340211 shutdown_connections 2024-08-28T19:40:31.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8b52e1740 1 -- 172.21.15.16:0/4280340211 wait complete. 2024-08-28T19:40:31.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8b52e1740 20 client.29023 trim_cache size 0 max 16384 2024-08-28T19:40:31.264 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:40:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1787759083 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8980b4d20 msgr2=0x7fe8980b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8aed6c640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8980b4d20 0x7fe8980b7230 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe894009fd0 tx=0x7fe894004730 comp rx=0 tx=0).stop 2024-08-28T19:40:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1787759083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a810db90 msgr2=0x7fe8a81a1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8aed6c640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a810db90 0x7fe8a81a1cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe8a002ade0 tx=0x7fe8a00316f0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1787759083 shutdown_connections 2024-08-28T19:40:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8aed6c640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe8980b4d20 0x7fe8980b7230 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8aed6c640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a810db90 0x7fe8a81a1cc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8aed6c640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8a8105010 0x7fe8a819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8aed6c640 1 --2- 172.21.15.16:0/1787759083 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8a81045e0 0x7fe8a819c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.259+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1787759083 >> 172.21.15.16:0/1787759083 conn(0x7fe8a80794d0 msgr2=0x7fe8a8100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.260+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1787759083 shutdown_connections 2024-08-28T19:40:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:31.260+0000 7fe8aed6c640 1 -- 172.21.15.16:0/1787759083 wait complete. 2024-08-28T19:40:31.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 140299 22.28 MB/sec execute 247 sec latency 109.241 ms 2024-08-28T19:40:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:31 smithi016 ceph-mon[18556]: pgmap v949: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.9 MiB/s wr, 291 op/s 2024-08-28T19:40:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:31 smithi028 ceph-mon[21769]: pgmap v949: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.9 MiB/s wr, 291 op/s 2024-08-28T19:40:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:31 smithi115 ceph-mon[22184]: pgmap v949: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.9 MiB/s wr, 291 op/s 2024-08-28T19:40:32.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:40:32.409 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:40:32.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 140913 22.31 MB/sec execute 248 sec latency 95.367 ms 2024-08-28T19:40:33.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.277+0000 7efdda784640 1 -- 172.21.15.16:0/674413954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdd4108220 msgr2=0x7efdd410ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:33.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.277+0000 7efdda784640 1 --2- 172.21.15.16:0/674413954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdd4108220 0x7efdd410ea50 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7efdcc009ca0 tx=0x7efdcc02a8b0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.277+0000 7efdda784640 1 -- 172.21.15.16:0/674413954 shutdown_connections 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.277+0000 7efdda784640 1 --2- 172.21.15.16:0/674413954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdd4108220 0x7efdd410ea50 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.277+0000 7efdda784640 1 --2- 172.21.15.16:0/674413954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd41078d0 0x7efdd4107ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.277+0000 7efdda784640 1 --2- 172.21.15.16:0/674413954 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efdd4101f90 0x7efdd41023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.277+0000 7efdda784640 1 -- 172.21.15.16:0/674413954 >> 172.21.15.16:0/674413954 conn(0x7efdd4077a00 msgr2=0x7efdd4077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.277+0000 7efdda784640 1 -- 172.21.15.16:0/674413954 shutdown_connections 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.277+0000 7efdda784640 1 -- 172.21.15.16:0/674413954 wait complete. 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.278+0000 7efdda784640 1 Processor -- start 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.278+0000 7efdda784640 1 -- start start 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdda784640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdd4101f90 0x7efdd4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdda784640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efdd41078d0 0x7efdd4198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdd4101f90 0x7efdd4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdda784640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4108220 0x7efdd419d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdd4101f90 0x7efdd4197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45894/0 (socket says 172.21.15.16:45894) 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdd3fff640 1 -- 172.21.15.16:0/2538598693 learned_addr learned my addr 172.21.15.16:0/2538598693 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdda784640 1 -- 172.21.15.16:0/2538598693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efdd4111660 con 0x7efdd4101f90 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdda784640 1 -- 172.21.15.16:0/2538598693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdd41114e0 con 0x7efdd4108220 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdda784640 1 -- 172.21.15.16:0/2538598693 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efdd41117e0 con 0x7efdd41078d0 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdd37fe640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efdd41078d0 0x7efdd4198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdd8d21640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4108220 0x7efdd419d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdd3fff640 1 -- 172.21.15.16:0/2538598693 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efdd41078d0 msgr2=0x7efdd4198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdd3fff640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efdd41078d0 0x7efdd4198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdd3fff640 1 -- 172.21.15.16:0/2538598693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4108220 msgr2=0x7efdd419d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdd3fff640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4108220 0x7efdd419d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.279+0000 7efdd3fff640 1 -- 172.21.15.16:0/2538598693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd419df20 con 0x7efdd4101f90 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.280+0000 7efdd3fff640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdd4101f90 0x7efdd4197fb0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7efdc400fb00 tx=0x7efdc400ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.280+0000 7efdd17fa640 1 -- 172.21.15.16:0/2538598693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc401c070 con 0x7efdd4101f90 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.280+0000 7efdd17fa640 1 -- 172.21.15.16:0/2538598693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdc400dc90 con 0x7efdd4101f90 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.281+0000 7efdd17fa640 1 -- 172.21.15.16:0/2538598693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc4016610 con 0x7efdd4101f90 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.281+0000 7efdda784640 1 -- 172.21.15.16:0/2538598693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd419e210 con 0x7efdd4101f90 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.281+0000 7efdda784640 1 -- 172.21.15.16:0/2538598693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd41a4970 con 0x7efdd4101f90 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.288+0000 7efde0cf9740 1 Processor -- start 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.289+0000 7efde0cf9740 1 -- start start 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.289+0000 7efde0cf9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c107070 0x56089c4ac310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.289+0000 7efde0cf9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 0x56089c4ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.289+0000 7efde0cf9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4acd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.289+0000 7efde0cf9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56089c423680 con 0x56089c0de1b0 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.289+0000 7efde0cf9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56089c412c10 con 0x56089c0e6390 2024-08-28T19:40:33.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.289+0000 7efde0cf9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56089c416bc0 con 0x56089c107070 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.290+0000 7efdd37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.290+0000 7efdd37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4acd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37128/0 (socket says 172.21.15.16:37128) 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.290+0000 7efdd37fe640 1 -- 172.21.15.16:0/1781225176 learned_addr learned my addr 172.21.15.16:0/1781225176 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.290+0000 7efdd17fa640 1 -- 172.21.15.16:0/2538598693 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7efdc4017410 con 0x7efdd4101f90 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.291+0000 7efdd8d21640 1 --2- 172.21.15.16:0/1781225176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c107070 0x56089c4ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.292+0000 7efdd3fff640 1 --2- 172.21.15.16:0/1781225176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 0x56089c4ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.292+0000 7efdd37fe640 1 -- 172.21.15.16:0/1781225176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c107070 msgr2=0x56089c4ac310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.292+0000 7efdd37fe640 1 --2- 172.21.15.16:0/1781225176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c107070 0x56089c4ac310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.292+0000 7efdd37fe640 1 -- 172.21.15.16:0/1781225176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 msgr2=0x56089c4ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.292+0000 7efdd37fe640 1 --2- 172.21.15.16:0/1781225176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 0x56089c4ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.292+0000 7efdd37fe640 1 -- 172.21.15.16:0/1781225176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56089c172650 con 0x56089c0e6390 2024-08-28T19:40:33.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.292+0000 7efdd3fff640 1 --2- 172.21.15.16:0/1781225176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 0x56089c4ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.292+0000 7efdd17fa640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdbc0b4cd0 0x7efdbc0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.293+0000 7efdd8d21640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdbc0b4cd0 0x7efdbc0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.293+0000 7efdd17fa640 1 -- 172.21.15.16:0/2538598693 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7efdc40a74a0 con 0x7efdd4101f90 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.293+0000 7efdd37fe640 1 --2- 172.21.15.16:0/1781225176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4acd90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7efdc800bda0 tx=0x7efdc8009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=261296458445aaf1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.293+0000 7efdba7fc640 1 -- 172.21.15.16:0/1781225176 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc8004020 con 0x56089c0e6390 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.294+0000 7efdba7fc640 1 -- 172.21.15.16:0/1781225176 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdc80041c0 con 0x56089c0e6390 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.295+0000 7efde0cf9740 1 -- 172.21.15.16:0/1781225176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 msgr2=0x56089c4acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.295+0000 7efde0cf9740 1 --2- 172.21.15.16:0/1781225176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4acd90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7efdc800bda0 tx=0x7efdc8009610 comp rx=0 tx=0).stop 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.295+0000 7efdd8d21640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdbc0b4cd0 0x7efdbc0b71e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7efdcc006010 tx=0x7efdcc009360 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.295+0000 7efde0cf9740 1 -- 172.21.15.16:0/1781225176 shutdown_connections 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.295+0000 7efde0cf9740 1 --2- 172.21.15.16:0/1781225176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4acd90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.295+0000 7efde0cf9740 1 --2- 172.21.15.16:0/1781225176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 0x56089c4ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.295+0000 7efde0cf9740 1 --2- 172.21.15.16:0/1781225176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c107070 0x56089c4ac310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.295+0000 7efde0cf9740 1 -- 172.21.15.16:0/1781225176 >> 172.21.15.16:0/1781225176 conn(0x56089c0fb460 msgr2=0x56089c4aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.296+0000 7efde0cf9740 1 -- 172.21.15.16:0/1781225176 shutdown_connections 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.297+0000 7efde0cf9740 1 -- 172.21.15.16:0/1781225176 wait complete. 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.297+0000 7efde0cf9740 1 -- start start 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.298+0000 7efde0cf9740 1 Processor -- start 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.299+0000 7efde0cf9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0de1b0 0x56089c4bbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.299+0000 7efde0cf9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0e6390 0x56089c4bc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.299+0000 7efde0cf9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 0x56089c4bc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.299+0000 7efde0cf9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56089c275700 con 0x56089c0e6390 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.299+0000 7efde0cf9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56089c399750 con 0x56089c107070 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.299+0000 7efde0cf9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56089c4beef0 con 0x56089c0de1b0 2024-08-28T19:40:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.300+0000 7efde0cf9740 10 client.0 _ll_register_callbacks cb 0x56089c3c8e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.300+0000 7efdd8d21640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 0x56089c4bc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.300+0000 7efdd8d21640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 0x56089c4bc7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37140/0 (socket says 172.21.15.16:37140) 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.300+0000 7efdd8d21640 1 -- 172.21.15.16:0/3750727947 learned_addr learned my addr 172.21.15.16:0/3750727947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.300+0000 7efdd37fe640 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0e6390 0x56089c4bc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.301+0000 7efdd3fff640 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0de1b0 0x56089c4bbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.301+0000 7efdd37fe640 1 -- 172.21.15.16:0/3750727947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0de1b0 msgr2=0x56089c4bbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.301+0000 7efdd37fe640 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0de1b0 0x56089c4bbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.301+0000 7efdd37fe640 1 -- 172.21.15.16:0/3750727947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 msgr2=0x56089c4bc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.301+0000 7efdd37fe640 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 0x56089c4bc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.301+0000 7efdd37fe640 1 -- 172.21.15.16:0/3750727947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56089c3d2900 con 0x56089c0e6390 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.302+0000 7efdd37fe640 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0e6390 0x56089c4bc280 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7efdc800bd70 tx=0x7efdc8018d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67a8adeb47fadf4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.302+0000 7efdb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:40:33.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.303+0000 7efdb37fe640 1 -- 172.21.15.16:0/3750727947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc8026070 con 0x56089c0e6390 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.303+0000 7efdb37fe640 1 -- 172.21.15.16:0/3750727947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7efdc8005250 con 0x56089c0e6390 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.303+0000 7efde0cf9740 1 -- 172.21.15.16:0/3750727947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdc80041c0 con 0x56089c0e6390 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.304+0000 7efdb37fe640 1 -- 172.21.15.16:0/3750727947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7efdc8030050 con 0x56089c0e6390 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.305+0000 7efde0cf9740 10 client.20013 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.305+0000 7efde0cf9740 1 -- 172.21.15.16:0/3750727947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x56089c4cab40 con 0x56089c0e6390 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.305+0000 7efdb37fe640 1 -- 172.21.15.16:0/3750727947 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdc8003aa0 con 0x56089c0e6390 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.306+0000 7efdb37fe640 1 -- 172.21.15.16:0/3750727947 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7efdc8033020 con 0x56089c0e6390 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.306+0000 7efde0cf9740 10 client.20013 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.306+0000 7efde0cf9740 10 client.20013 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.306+0000 7efde0cf9740 20 client.20013 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.306+0000 7efde0cf9740 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56089c0c59f0 0x56089c4cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.306+0000 7efde0cf9740 4 client.20013 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.306+0000 7efde0cf9740 1 -- 172.21.15.16:0/3750727947 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7efdc80041c0 con 0x56089c0c59f0 2024-08-28T19:40:33.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.306+0000 7efdd3fff640 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56089c0c59f0 0x56089c4cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.312+0000 7efdd3fff640 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56089c0c59f0 0x56089c4cdf50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3ee5236fef5f5230 server_cookie=d6437459f40d1c49 in_seq=0 out_seq=0 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.313+0000 7efdb37fe640 10 client.20013 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.314+0000 7efdb37fe640 1 -- 172.21.15.16:0/3750727947 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7efdc80041c0 con 0x56089c0c59f0 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.314+0000 7efdb37fe640 10 client.20013 handle_command_reply: tid=0 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.315+0000 7efde0cf9740 1 client.20013 shutdown 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.315+0000 7efde0cf9740 1 -- 172.21.15.16:0/3750727947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0e6390 msgr2=0x56089c4bc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.315+0000 7efde0cf9740 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0e6390 0x56089c4bc280 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7efdc800bd70 tx=0x7efdc8018d90 comp rx=0 tx=0).stop 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.316+0000 7efde0cf9740 1 -- 172.21.15.16:0/3750727947 shutdown_connections 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.316+0000 7efde0cf9740 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56089c0c59f0 0x56089c4cdf50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.316+0000 7efde0cf9740 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 0x56089c4bc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.316+0000 7efde0cf9740 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0e6390 0x56089c4bc280 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.316+0000 7efde0cf9740 1 --2- 172.21.15.16:0/3750727947 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0de1b0 0x56089c4bbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.316+0000 7efde0cf9740 1 -- 172.21.15.16:0/3750727947 >> 172.21.15.16:0/3750727947 conn(0x56089c0fb460 msgr2=0x56089c4a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.316+0000 7efdb37fe640 0 client.20013 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.316+0000 7efde0cf9740 1 -- 172.21.15.16:0/3750727947 shutdown_connections 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.316+0000 7efde0cf9740 1 -- 172.21.15.16:0/3750727947 wait complete. 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.316+0000 7efde0cf9740 20 client.20013 trim_cache size 0 max 16384 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.343+0000 7efde0cf9740 1 Processor -- start 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.343+0000 7efde0cf9740 1 -- start start 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.344+0000 7efde0cf9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 0x56089c4b9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.344+0000 7efde0cf9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0e6390 0x56089c4b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.344+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 0x56089c4b9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.344+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 0x56089c4b9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45924/0 (socket says 172.21.15.16:45924) 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.344+0000 7efdd3fff640 1 -- 172.21.15.16:0/3322569459 learned_addr learned my addr 172.21.15.16:0/3322569459 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:33.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.345+0000 7efde0cf9740 1 --2- 172.21.15.16:0/3322569459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 0x56089c4b1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.345+0000 7efde0cf9740 1 -- 172.21.15.16:0/3322569459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efdd41ae8d0 con 0x56089c0de1b0 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.345+0000 7efde0cf9740 1 -- 172.21.15.16:0/3322569459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdd41b0590 con 0x56089c107070 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.345+0000 7efde0cf9740 1 -- 172.21.15.16:0/3322569459 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efdd41af030 con 0x56089c0e6390 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.345+0000 7efdd8d21640 1 --2- 172.21.15.16:0/3322569459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 0x56089c4b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.345+0000 7efdd37fe640 1 --2- 172.21.15.16:0/3322569459 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0e6390 0x56089c4b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.346+0000 7efdd8d21640 1 -- 172.21.15.16:0/3322569459 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0e6390 msgr2=0x56089c4b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.346+0000 7efdd8d21640 1 --2- 172.21.15.16:0/3322569459 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0e6390 0x56089c4b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.346+0000 7efdd8d21640 1 -- 172.21.15.16:0/3322569459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 msgr2=0x56089c4b9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.346+0000 7efdd8d21640 1 --2- 172.21.15.16:0/3322569459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 0x56089c4b9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.346+0000 7efdd8d21640 1 -- 172.21.15.16:0/3322569459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56089c4ba240 con 0x56089c107070 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.347+0000 7efdd8d21640 1 --2- 172.21.15.16:0/3322569459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 0x56089c4b1920 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efdcc030610 tx=0x7efdcc041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fba303b2a8077eff server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.349+0000 7efdb9ffb640 1 -- 172.21.15.16:0/3322569459 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc030a00 con 0x56089c107070 2024-08-28T19:40:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.349+0000 7efdb9ffb640 1 -- 172.21.15.16:0/3322569459 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdcc036050 con 0x56089c107070 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.349+0000 7efdb9ffb640 1 -- 172.21.15.16:0/3322569459 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc004430 con 0x56089c107070 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.351+0000 7efde0cf9740 1 -- 172.21.15.16:0/3322569459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 msgr2=0x56089c4b1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.351+0000 7efde0cf9740 1 --2- 172.21.15.16:0/3322569459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 0x56089c4b1920 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efdcc030610 tx=0x7efdcc041000 comp rx=0 tx=0).stop 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.353+0000 7efde0cf9740 1 -- 172.21.15.16:0/3322569459 shutdown_connections 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.353+0000 7efde0cf9740 1 --2- 172.21.15.16:0/3322569459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c107070 0x56089c4b1920 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.353+0000 7efde0cf9740 1 --2- 172.21.15.16:0/3322569459 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0e6390 0x56089c4b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.353+0000 7efde0cf9740 1 --2- 172.21.15.16:0/3322569459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c0de1b0 0x56089c4b9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.353+0000 7efde0cf9740 1 -- 172.21.15.16:0/3322569459 >> 172.21.15.16:0/3322569459 conn(0x56089c0fb460 msgr2=0x56089c4b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.354+0000 7efde0cf9740 1 -- 172.21.15.16:0/3322569459 shutdown_connections 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.354+0000 7efde0cf9740 1 -- 172.21.15.16:0/3322569459 wait complete. 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.354+0000 7efde0cf9740 1 -- start start 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.355+0000 7efde0cf9740 1 Processor -- start 2024-08-28T19:40:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.361+0000 7efde0cf9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0de1b0 0x56089c4cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.361+0000 7efde0cf9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.361+0000 7efde0cf9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c107070 0x56089c4cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.361+0000 7efde0cf9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efdd41afe00 con 0x56089c107070 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.361+0000 7efde0cf9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdd41af700 con 0x56089c0e6390 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.361+0000 7efde0cf9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efdd41ac8d0 con 0x56089c0de1b0 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.361+0000 7efde0cf9740 10 client.0 _ll_register_callbacks cb 0x7efdc800bbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.362+0000 7efdd37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.362+0000 7efdd37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:37158/0 (socket says 172.21.15.16:37158) 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.362+0000 7efdd37fe640 1 -- 172.21.15.16:0/2157357211 learned_addr learned my addr 172.21.15.16:0/2157357211 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.362+0000 7efdd8d21640 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c107070 0x56089c4cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.363+0000 7efdd37fe640 1 -- 172.21.15.16:0/2157357211 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0de1b0 msgr2=0x56089c4cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.363+0000 7efdd37fe640 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0de1b0 0x56089c4cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.363+0000 7efdd37fe640 1 -- 172.21.15.16:0/2157357211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c107070 msgr2=0x56089c4cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.363+0000 7efdd37fe640 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c107070 0x56089c4cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.363+0000 7efdd37fe640 1 -- 172.21.15.16:0/2157357211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56089c5486e0 con 0x56089c0e6390 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.363+0000 7efdd37fe640 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4cf440 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efdc8004ca0 tx=0x7efdc8004910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a50f4d3e8b19ab4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.367+0000 7efdb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.367+0000 7efdb37fe640 1 -- 172.21.15.16:0/2157357211 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc8026070 con 0x56089c0e6390 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.367+0000 7efdb37fe640 1 -- 172.21.15.16:0/2157357211 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7efdc8005460 con 0x56089c0e6390 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.367+0000 7efde0cf9740 1 -- 172.21.15.16:0/2157357211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdcc036050 con 0x56089c0e6390 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.367+0000 7efdb37fe640 1 -- 172.21.15.16:0/2157357211 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7efdc803d050 con 0x56089c0e6390 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.368+0000 7efde0cf9740 10 client.29468 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.368+0000 7efde0cf9740 1 -- 172.21.15.16:0/2157357211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x56089c5498f0 con 0x56089c0e6390 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.368+0000 7efdb37fe640 1 -- 172.21.15.16:0/2157357211 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdcc036050 con 0x56089c0e6390 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.368+0000 7efdb37fe640 1 -- 172.21.15.16:0/2157357211 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7efdc8033040 con 0x56089c0e6390 2024-08-28T19:40:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.368+0000 7efde0cf9740 10 client.29468 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.368+0000 7efde0cf9740 10 client.29468 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:40:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.368+0000 7efde0cf9740 20 client.29468 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.368+0000 7efde0cf9740 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56089c09ada0 0x56089c54c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.369+0000 7efde0cf9740 4 client.29468 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:40:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.369+0000 7efde0cf9740 1 -- 172.21.15.16:0/2157357211 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56089c4255d0 con 0x56089c09ada0 2024-08-28T19:40:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.369+0000 7efdd3fff640 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56089c09ada0 0x56089c54c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.370+0000 7efdd3fff640 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56089c09ada0 0x56089c54c040 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c19d684083dc7b3c server_cookie=632c8592f33a41d1 in_seq=0 out_seq=0 2024-08-28T19:40:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efdb37fe640 10 client.29468 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:40:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efdb37fe640 1 -- 172.21.15.16:0/2157357211 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56089c4255d0 con 0x56089c09ada0 2024-08-28T19:40:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efdb37fe640 10 client.29468 handle_command_reply: tid=0 2024-08-28T19:40:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efde0cf9740 1 client.29468 shutdown 2024-08-28T19:40:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efde0cf9740 1 -- 172.21.15.16:0/2157357211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 msgr2=0x56089c4cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efde0cf9740 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4cf440 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efdc8004ca0 tx=0x7efdc8004910 comp rx=0 tx=0).stop 2024-08-28T19:40:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efde0cf9740 1 -- 172.21.15.16:0/2157357211 shutdown_connections 2024-08-28T19:40:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efde0cf9740 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x56089c09ada0 0x56089c54c040 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efde0cf9740 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56089c107070 0x56089c4cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efde0cf9740 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56089c0e6390 0x56089c4cf440 secure :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efdc8004ca0 tx=0x7efdc8004910 comp rx=0 tx=0).stop 2024-08-28T19:40:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efde0cf9740 1 --2- 172.21.15.16:0/2157357211 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56089c0de1b0 0x56089c4cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.372+0000 7efde0cf9740 1 -- 172.21.15.16:0/2157357211 >> 172.21.15.16:0/2157357211 conn(0x56089c0fb460 msgr2=0x56089c4b13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:33.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.375+0000 7efdb37fe640 0 client.29468 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:40:33.461 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:40:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.375+0000 7efde0cf9740 1 -- 172.21.15.16:0/2157357211 shutdown_connections 2024-08-28T19:40:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.375+0000 7efde0cf9740 1 -- 172.21.15.16:0/2157357211 wait complete. 2024-08-28T19:40:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.375+0000 7efde0cf9740 20 client.29468 trim_cache size 0 max 16384 2024-08-28T19:40:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 -- 172.21.15.16:0/2538598693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdbc0b4cd0 msgr2=0x7efdbc0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdbc0b4cd0 0x7efdbc0b71e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7efdcc006010 tx=0x7efdcc009360 comp rx=0 tx=0).stop 2024-08-28T19:40:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 -- 172.21.15.16:0/2538598693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdd4101f90 msgr2=0x7efdd4197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdd4101f90 0x7efdd4197fb0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7efdc400fb00 tx=0x7efdc400ce20 comp rx=0 tx=0).stop 2024-08-28T19:40:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 -- 172.21.15.16:0/2538598693 shutdown_connections 2024-08-28T19:40:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7efdbc0b4cd0 0x7efdbc0b71e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4108220 0x7efdd419d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efdd41078d0 0x7efdd4198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 --2- 172.21.15.16:0/2538598693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdd4101f90 0x7efdd4197fb0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 -- 172.21.15.16:0/2538598693 >> 172.21.15.16:0/2538598693 conn(0x7efdd4077a00 msgr2=0x7efdd41038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 -- 172.21.15.16:0/2538598693 shutdown_connections 2024-08-28T19:40:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:33.379+0000 7efdda784640 1 -- 172.21.15.16:0/2538598693 wait complete. 2024-08-28T19:40:33.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 141481 22.36 MB/sec execute 249 sec latency 86.648 ms 2024-08-28T19:40:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:33 smithi016 ceph-mon[18556]: pgmap v950: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5.5 MiB/s wr, 406 op/s 2024-08-28T19:40:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:33 smithi028 ceph-mon[21769]: pgmap v950: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5.5 MiB/s wr, 406 op/s 2024-08-28T19:40:33.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:33 smithi115 ceph-mon[22184]: pgmap v950: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5.5 MiB/s wr, 406 op/s 2024-08-28T19:40:34.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:40:34.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 141964 22.33 MB/sec execute 250 sec latency 205.226 ms 2024-08-28T19:40:34.581 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:40:35.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.293+0000 7fc33804f640 1 -- 172.21.15.16:0/3221706140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc330103940 msgr2=0x7fc330103d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:35.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.293+0000 7fc33804f640 1 --2- 172.21.15.16:0/3221706140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc330103940 0x7fc330103d50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009900 tx=0x7fc32c02a410 comp rx=0 tx=0).stop 2024-08-28T19:40:35.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.294+0000 7fc33804f640 1 -- 172.21.15.16:0/3221706140 shutdown_connections 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.294+0000 7fc33804f640 1 --2- 172.21.15.16:0/3221706140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc330109bd0 0x7fc330110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.294+0000 7fc33804f640 1 --2- 172.21.15.16:0/3221706140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc330109280 0x7fc330109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.294+0000 7fc33804f640 1 --2- 172.21.15.16:0/3221706140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc330103940 0x7fc330103d50 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.294+0000 7fc33804f640 1 -- 172.21.15.16:0/3221706140 >> 172.21.15.16:0/3221706140 conn(0x7fc3300ff3d0 msgr2=0x7fc330101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.294+0000 7fc33804f640 1 -- 172.21.15.16:0/3221706140 shutdown_connections 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.294+0000 7fc33804f640 1 -- 172.21.15.16:0/3221706140 wait complete. 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc33804f640 1 Processor -- start 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc33804f640 1 -- start start 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc33804f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc330103940 0x7fc33019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc33804f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc330109280 0x7fc33019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc33804f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc330109bd0 0x7fc33019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc33804f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc330113890 con 0x7fc330109280 2024-08-28T19:40:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc33804f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc330113710 con 0x7fc330109bd0 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc33804f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc330113a10 con 0x7fc330103940 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc3365ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc330109bd0 0x7fc33019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc3355ea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc330109280 0x7fc33019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc3365ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc330109bd0 0x7fc33019faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54858/0 (socket says 172.21.15.16:54858) 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc3355ea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc330109280 0x7fc33019a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50684/0 (socket says 172.21.15.16:50684) 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.295+0000 7fc3365ec640 1 -- 172.21.15.16:0/1381217622 learned_addr learned my addr 172.21.15.16:0/1381217622 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.296+0000 7fc335deb640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc330103940 0x7fc33019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.296+0000 7fc3365ec640 1 -- 172.21.15.16:0/1381217622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc330103940 msgr2=0x7fc33019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.296+0000 7fc3365ec640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc330103940 0x7fc33019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.296+0000 7fc3365ec640 1 -- 172.21.15.16:0/1381217622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc330109280 msgr2=0x7fc33019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.296+0000 7fc3365ec640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc330109280 0x7fc33019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.296+0000 7fc3365ec640 1 -- 172.21.15.16:0/1381217622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3301a01d0 con 0x7fc330109bd0 2024-08-28T19:40:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.296+0000 7fc3365ec640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc330109bd0 0x7fc33019faa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc32800eb10 tx=0x7fc32800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.296+0000 7fc326ffd640 1 -- 172.21.15.16:0/1381217622 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328004b20 con 0x7fc330109bd0 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.296+0000 7fc33804f640 1 -- 172.21.15.16:0/1381217622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3301a04c0 con 0x7fc330109bd0 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.296+0000 7fc326ffd640 1 -- 172.21.15.16:0/1381217622 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc328004cc0 con 0x7fc330109bd0 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.297+0000 7fc33804f640 1 -- 172.21.15.16:0/1381217622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc330075390 con 0x7fc330109bd0 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.297+0000 7fc326ffd640 1 -- 172.21.15.16:0/1381217622 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328010690 con 0x7fc330109bd0 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.300+0000 7fc326ffd640 1 -- 172.21.15.16:0/1381217622 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc3280108f0 con 0x7fc330109bd0 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.300+0000 7fc33e5c4740 1 Processor -- start 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.300+0000 7fc33e5c4740 1 -- start start 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.300+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c238070 0x55763c5dd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.300+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c20f1b0 0x55763c5dd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.300+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c217390 0x55763c5ddd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55763c554680 con 0x55763c217390 2024-08-28T19:40:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55763c543c10 con 0x55763c238070 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55763c547bc0 con 0x55763c20f1b0 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc3355ea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c238070 0x55763c5dd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc335deb640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c20f1b0 0x55763c5dd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc3355ea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c238070 0x55763c5dd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54860/0 (socket says 172.21.15.16:54860) 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc3355ea640 1 -- 172.21.15.16:0/3138519979 learned_addr learned my addr 172.21.15.16:0/3138519979 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc3355ea640 1 --2- 172.21.15.16:0/3138519979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c217390 0x55763c5ddd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc326ffd640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc31c0b4d20 0x7fc31c0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc335deb640 1 -- 172.21.15.16:0/3138519979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c238070 msgr2=0x55763c5dd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc335deb640 1 --2- 172.21.15.16:0/3138519979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c238070 0x55763c5dd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc335deb640 1 -- 172.21.15.16:0/3138519979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c217390 msgr2=0x55763c5ddd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc335deb640 1 --2- 172.21.15.16:0/3138519979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c217390 0x55763c5ddd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc335deb640 1 -- 172.21.15.16:0/3138519979 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55763c2a3650 con 0x55763c20f1b0 2024-08-28T19:40:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc3355ea640 1 --2- 172.21.15.16:0/3138519979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c238070 0x55763c5dd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.301+0000 7fc326ffd640 1 -- 172.21.15.16:0/1381217622 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fc3280e5ee0 con 0x7fc330109bd0 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.302+0000 7fc335deb640 1 --2- 172.21.15.16:0/3138519979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c20f1b0 0x55763c5dd850 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009230 tx=0x7fc32c032b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cae4169f5da92e6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.302+0000 7fc3365ec640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc31c0b4d20 0x7fc31c0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.302+0000 7fc31bfff640 1 -- 172.21.15.16:0/3138519979 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c040070 con 0x55763c20f1b0 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.302+0000 7fc31bfff640 1 -- 172.21.15.16:0/3138519979 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc32c004030 con 0x55763c20f1b0 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.302+0000 7fc31bfff640 1 -- 172.21.15.16:0/3138519979 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c035040 con 0x55763c20f1b0 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.303+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3138519979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c20f1b0 msgr2=0x55763c5dd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.303+0000 7fc3365ec640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc31c0b4d20 0x7fc31c0b7230 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc3280ac130 tx=0x7fc32800cee0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.303+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3138519979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c20f1b0 0x55763c5dd850 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009230 tx=0x7fc32c032b90 comp rx=0 tx=0).stop 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.303+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3138519979 shutdown_connections 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.303+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3138519979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c217390 0x55763c5ddd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.303+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3138519979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c20f1b0 0x55763c5dd850 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.303+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3138519979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c238070 0x55763c5dd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.303+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3138519979 >> 172.21.15.16:0/3138519979 conn(0x55763c22c460 msgr2=0x55763c5db200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.304+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3138519979 shutdown_connections 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.304+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3138519979 wait complete. 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.304+0000 7fc33e5c4740 1 -- start start 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.304+0000 7fc33e5c4740 1 Processor -- start 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.305+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 0x55763c5f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.305+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c217390 0x55763c5f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.305+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c238070 0x55763c5f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.305+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55763c3a6700 con 0x55763c20f1b0 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.305+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55763c4ca750 con 0x55763c217390 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.305+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55763c5eba90 con 0x55763c238070 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.305+0000 7fc335deb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c217390 0x55763c5f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.305+0000 7fc3355ea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 0x55763c5f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.305+0000 7fc335deb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c217390 0x55763c5f21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54870/0 (socket says 172.21.15.16:54870) 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.305+0000 7fc335deb640 1 -- 172.21.15.16:0/3513681345 learned_addr learned my addr 172.21.15.16:0/3513681345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.305+0000 7fc33e5c4740 10 client.0 _ll_register_callbacks cb 0x55763c4f9e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc3355ea640 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c238070 0x55763c5f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc335deb640 1 -- 172.21.15.16:0/3513681345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c238070 msgr2=0x55763c5f71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc335deb640 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c238070 0x55763c5f71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc335deb640 1 -- 172.21.15.16:0/3513681345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 msgr2=0x55763c5f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc335deb640 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 0x55763c5f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc335deb640 1 -- 172.21.15.16:0/3513681345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55763c503900 con 0x55763c217390 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc3355ea640 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 0x55763c5f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc335deb640 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c217390 0x55763c5f21a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc32c02a930 tx=0x7fc32c032f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70b9ded1b2c158ee server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc3197fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:40:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc3197fa640 1 -- 172.21.15.16:0/3513681345 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c040070 con 0x55763c217390 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc3197fa640 1 -- 172.21.15.16:0/3513681345 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fc32c0314a0 con 0x55763c217390 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.306+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3513681345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc32c004030 con 0x55763c217390 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc3197fa640 1 -- 172.21.15.16:0/3513681345 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fc32c056050 con 0x55763c217390 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc33e5c4740 10 client.29480 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3513681345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55763c6000d0 con 0x55763c217390 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc3197fa640 1 -- 172.21.15.16:0/3513681345 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc32c004030 con 0x55763c217390 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc3197fa640 1 -- 172.21.15.16:0/3513681345 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fc32c004980 con 0x55763c217390 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc33e5c4740 10 client.29480 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc33e5c4740 10 client.29480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc33e5c4740 20 client.29480 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55763c1f69f0 0x55763c603430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc33e5c4740 4 client.29480 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:40:35.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3513681345 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55763c5565d0 con 0x55763c1f69f0 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.307+0000 7fc3355ea640 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55763c1f69f0 0x55763c603430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.308+0000 7fc3197fa640 10 client.29480 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.309+0000 7fc3355ea640 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55763c1f69f0 0x55763c603430 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e73a7d05c50916f server_cookie=c11caa6afd815a81 in_seq=0 out_seq=0 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc3197fa640 1 -- 172.21.15.16:0/3513681345 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55763c5565d0 con 0x55763c1f69f0 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc3197fa640 10 client.29480 handle_command_reply: tid=0 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc33e5c4740 1 client.29480 shutdown 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3513681345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c217390 msgr2=0x55763c5f21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c217390 0x55763c5f21a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc32c02a930 tx=0x7fc32c032f90 comp rx=0 tx=0).stop 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3513681345 shutdown_connections 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55763c1f69f0 0x55763c603430 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c238070 0x55763c5f71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc3197fa640 0 client.29480 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c217390 0x55763c5f21a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3513681345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 0x55763c5f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3513681345 >> 172.21.15.16:0/3513681345 conn(0x55763c22c460 msgr2=0x55763c5e04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.312+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3513681345 shutdown_connections 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.313+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3513681345 wait complete. 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.313+0000 7fc33e5c4740 20 client.29480 trim_cache size 0 max 16384 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.330+0000 7fc33e5c4740 1 Processor -- start 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.330+0000 7fc33e5c4740 1 -- start start 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c22c460 0x55763c5f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c1f69f0 0x55763c5f7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c20f1b0 0x55763c5dab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3301aeb10 con 0x55763c1f69f0 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3301b07d0 con 0x55763c20f1b0 2024-08-28T19:40:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc3301af270 con 0x55763c22c460 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc335deb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c1f69f0 0x55763c5f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc335deb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c1f69f0 0x55763c5f7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50700/0 (socket says 172.21.15.16:50700) 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc335deb640 1 -- 172.21.15.16:0/3151956584 learned_addr learned my addr 172.21.15.16:0/3151956584 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc3355ea640 1 --2- 172.21.15.16:0/3151956584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c22c460 0x55763c5f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc3355ea640 1 --2- 172.21.15.16:0/3151956584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c20f1b0 0x55763c5dab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc335deb640 1 -- 172.21.15.16:0/3151956584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c22c460 msgr2=0x55763c5f71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc335deb640 1 --2- 172.21.15.16:0/3151956584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c22c460 0x55763c5f71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc335deb640 1 -- 172.21.15.16:0/3151956584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c20f1b0 msgr2=0x55763c5dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc335deb640 1 --2- 172.21.15.16:0/3151956584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c20f1b0 0x55763c5dab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc335deb640 1 -- 172.21.15.16:0/3151956584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55763c5f7c40 con 0x55763c1f69f0 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc3355ea640 1 --2- 172.21.15.16:0/3151956584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c20f1b0 0x55763c5dab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.331+0000 7fc335deb640 1 --2- 172.21.15.16:0/3151956584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c1f69f0 0x55763c5f7700 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fc32c006010 tx=0x7fc32c0098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9d4fbeb1492dc2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.332+0000 7fc31b7fe640 1 -- 172.21.15.16:0/3151956584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c05f070 con 0x55763c1f69f0 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.332+0000 7fc31b7fe640 1 -- 172.21.15.16:0/3151956584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc32c04d070 con 0x55763c1f69f0 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.332+0000 7fc31b7fe640 1 -- 172.21.15.16:0/3151956584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c040040 con 0x55763c1f69f0 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.333+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3151956584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c1f69f0 msgr2=0x55763c5f7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.333+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3151956584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c1f69f0 0x55763c5f7700 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fc32c006010 tx=0x7fc32c0098c0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.333+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3151956584 shutdown_connections 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.333+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3151956584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c20f1b0 0x55763c5dab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.333+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3151956584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c1f69f0 0x55763c5f7700 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.333+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/3151956584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c22c460 0x55763c5f71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.333+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3151956584 >> 172.21.15.16:0/3151956584 conn(0x55763c5f2390 msgr2=0x55763c5f0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.333+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3151956584 shutdown_connections 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.333+0000 7fc33e5c4740 1 -- 172.21.15.16:0/3151956584 wait complete. 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.333+0000 7fc33e5c4740 1 -- start start 2024-08-28T19:40:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc33e5c4740 1 Processor -- start 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c1f69f0 0x55763c5f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 0x55763c5f8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc33e5c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c22c460 0x55763c5fd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3301b0040 con 0x55763c20f1b0 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3301af940 con 0x55763c22c460 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc33e5c4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc3301acb10 con 0x55763c1f69f0 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc33e5c4740 10 client.0 _ll_register_callbacks cb 0x7fc320010e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc335deb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 0x55763c5f8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc335deb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 0x55763c5f8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50702/0 (socket says 172.21.15.16:50702) 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc335deb640 1 -- 172.21.15.16:0/1951717019 learned_addr learned my addr 172.21.15.16:0/1951717019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc3355ea640 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c22c460 0x55763c5fd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.334+0000 7fc3355ea640 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c1f69f0 0x55763c5f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.335+0000 7fc3355ea640 1 -- 172.21.15.16:0/1951717019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c1f69f0 msgr2=0x55763c5f81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.335+0000 7fc3355ea640 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c1f69f0 0x55763c5f81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.335+0000 7fc3355ea640 1 -- 172.21.15.16:0/1951717019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 msgr2=0x55763c5f8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.335+0000 7fc3355ea640 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 0x55763c5f8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.335+0000 7fc3355ea640 1 -- 172.21.15.16:0/1951717019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55763c5fd780 con 0x55763c22c460 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.335+0000 7fc335deb640 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 0x55763c5f8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.335+0000 7fc3355ea640 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c22c460 0x55763c5fd240 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55763c5547d0 tx=0x7fc320029040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=475b0462b38efbc3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.335+0000 7fc3197fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.335+0000 7fc3197fa640 1 -- 172.21.15.16:0/1951717019 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320021850 con 0x55763c22c460 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.336+0000 7fc3197fa640 1 -- 172.21.15.16:0/1951717019 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fc3200219f0 con 0x55763c22c460 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.336+0000 7fc33e5c4740 1 -- 172.21.15.16:0/1951717019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc32c04d070 con 0x55763c22c460 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.336+0000 7fc3197fa640 1 -- 172.21.15.16:0/1951717019 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fc320003e50 con 0x55763c22c460 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.336+0000 7fc33e5c4740 10 client.29486 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.336+0000 7fc33e5c4740 1 -- 172.21.15.16:0/1951717019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55763c66e280 con 0x55763c22c460 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.336+0000 7fc3197fa640 1 -- 172.21.15.16:0/1951717019 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc32c04d070 con 0x55763c22c460 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.336+0000 7fc3197fa640 1 -- 172.21.15.16:0/1951717019 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fc320020430 con 0x55763c22c460 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.336+0000 7fc33e5c4740 10 client.29486 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.336+0000 7fc33e5c4740 10 client.29486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.336+0000 7fc33e5c4740 20 client.29486 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.337+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55763c217390 0x55763c670ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.337+0000 7fc33e5c4740 4 client.29486 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.337+0000 7fc33e5c4740 1 -- 172.21.15.16:0/1951717019 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55763c5dfba0 con 0x55763c217390 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.337+0000 7fc335deb640 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55763c217390 0x55763c670ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.337+0000 7fc3197fa640 10 client.29486 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.338+0000 7fc335deb640 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55763c217390 0x55763c670ae0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b0bc094d4f34e0f server_cookie=9bca548191bd5eed in_seq=0 out_seq=0 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc3197fa640 1 -- 172.21.15.16:0/1951717019 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55763c5dfba0 con 0x55763c217390 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc3197fa640 10 client.29486 handle_command_reply: tid=0 2024-08-28T19:40:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc33e5c4740 1 client.29486 shutdown 2024-08-28T19:40:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc33e5c4740 1 -- 172.21.15.16:0/1951717019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c22c460 msgr2=0x55763c5fd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c22c460 0x55763c5fd240 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55763c5547d0 tx=0x7fc320029040 comp rx=0 tx=0).stop 2024-08-28T19:40:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc33e5c4740 1 -- 172.21.15.16:0/1951717019 shutdown_connections 2024-08-28T19:40:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55763c22c460 0x55763c5fd240 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55763c20f1b0 0x55763c5f8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55763c217390 0x55763c670ae0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:40:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc33e5c4740 1 --2- 172.21.15.16:0/1951717019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55763c1f69f0 0x55763c5f81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc33e5c4740 1 -- 172.21.15.16:0/1951717019 >> 172.21.15.16:0/1951717019 conn(0x55763c5f2390 msgr2=0x55763c5dab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc3197fa640 0 client.29486 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc33e5c4740 1 -- 172.21.15.16:0/1951717019 shutdown_connections 2024-08-28T19:40:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.339+0000 7fc33e5c4740 1 -- 172.21.15.16:0/1951717019 wait complete. 2024-08-28T19:40:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.340+0000 7fc33e5c4740 20 client.29486 trim_cache size 0 max 16384 2024-08-28T19:40:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.340+0000 7fc33804f640 1 -- 172.21.15.16:0/1381217622 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc31c0b4d20 msgr2=0x7fc31c0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.340+0000 7fc33804f640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc31c0b4d20 0x7fc31c0b7230 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc3280ac130 tx=0x7fc32800cee0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.340+0000 7fc33804f640 1 -- 172.21.15.16:0/1381217622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc330109bd0 msgr2=0x7fc33019faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.340+0000 7fc33804f640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc330109bd0 0x7fc33019faa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc32800eb10 tx=0x7fc32800c2d0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.340+0000 7fc33804f640 1 -- 172.21.15.16:0/1381217622 shutdown_connections 2024-08-28T19:40:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.340+0000 7fc33804f640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc31c0b4d20 0x7fc31c0b7230 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.340+0000 7fc33804f640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc330109bd0 0x7fc33019faa0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.340+0000 7fc33804f640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc330109280 0x7fc33019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.340+0000 7fc33804f640 1 --2- 172.21.15.16:0/1381217622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc330103940 0x7fc33019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.340+0000 7fc33804f640 1 -- 172.21.15.16:0/1381217622 >> 172.21.15.16:0/1381217622 conn(0x7fc3300ff3d0 msgr2=0x7fc3301017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.341+0000 7fc33804f640 1 -- 172.21.15.16:0/1381217622 shutdown_connections 2024-08-28T19:40:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:35.341+0000 7fc33804f640 1 -- 172.21.15.16:0/1381217622 wait complete. 2024-08-28T19:40:35.482 INFO:tasks.workunit.client.0.smithi016.stdout: 10 142747 22.42 MB/sec execute 251 sec latency 72.259 ms 2024-08-28T19:40:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:35 smithi016 ceph-mon[18556]: pgmap v951: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 6.6 MiB/s wr, 465 op/s 2024-08-28T19:40:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:35 smithi028 ceph-mon[21769]: pgmap v951: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 6.6 MiB/s wr, 465 op/s 2024-08-28T19:40:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:35 smithi115 ceph-mon[22184]: pgmap v951: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 6.6 MiB/s wr, 465 op/s 2024-08-28T19:40:36.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 143601 22.52 MB/sec execute 252 sec latency 64.432 ms 2024-08-28T19:40:37.298 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:40:37.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:40:37.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 144212 22.57 MB/sec execute 253 sec latency 276.277 ms 2024-08-28T19:40:37.650 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:40:37.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:37 smithi016 ceph-mon[18556]: pgmap v952: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 9.1 MiB/s wr, 631 op/s 2024-08-28T19:40:37.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:37 smithi028 ceph-mon[21769]: pgmap v952: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 9.1 MiB/s wr, 631 op/s 2024-08-28T19:40:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:37 smithi115 ceph-mon[22184]: pgmap v952: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 9.1 MiB/s wr, 631 op/s 2024-08-28T19:40:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.218+0000 7fdf969a2640 1 -- 172.21.15.16:0/596700357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf90078030 msgr2=0x7fdf90113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.218+0000 7fdf969a2640 1 --2- 172.21.15.16:0/596700357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf90078030 0x7fdf90113fd0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009ca0 tx=0x7fdf8802a8f0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.218+0000 7fdf969a2640 1 -- 172.21.15.16:0/596700357 shutdown_connections 2024-08-28T19:40:38.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.218+0000 7fdf969a2640 1 --2- 172.21.15.16:0/596700357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf90114510 0x7fdf90116930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.218+0000 7fdf969a2640 1 --2- 172.21.15.16:0/596700357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf90078030 0x7fdf90113fd0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.218+0000 7fdf969a2640 1 --2- 172.21.15.16:0/596700357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf900776e0 0x7fdf90077af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.218+0000 7fdf969a2640 1 -- 172.21.15.16:0/596700357 >> 172.21.15.16:0/596700357 conn(0x7fdf90101560 msgr2=0x7fdf901039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.218+0000 7fdf969a2640 1 -- 172.21.15.16:0/596700357 shutdown_connections 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.218+0000 7fdf969a2640 1 -- 172.21.15.16:0/596700357 wait complete. 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.220+0000 7fdf969a2640 1 Processor -- start 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.220+0000 7fdf969a2640 1 -- start start 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.220+0000 7fdf969a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf900776e0 0x7fdf901a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.220+0000 7fdf969a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf90078030 0x7fdf901a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.220+0000 7fdf969a2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf90114510 0x7fdf901a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.220+0000 7fdf969a2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90119fc0 con 0x7fdf900776e0 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.221+0000 7fdf969a2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90119e40 con 0x7fdf90078030 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.221+0000 7fdf969a2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9011a140 con 0x7fdf90114510 2024-08-28T19:40:38.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.221+0000 7fdf8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf90078030 0x7fdf901a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.221+0000 7fdf8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf90078030 0x7fdf901a0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54908/0 (socket says 172.21.15.16:54908) 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.221+0000 7fdf8f7fe640 1 -- 172.21.15.16:0/2514266382 learned_addr learned my addr 172.21.15.16:0/2514266382 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.221+0000 7fdf8f7fe640 1 -- 172.21.15.16:0/2514266382 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf90114510 msgr2=0x7fdf901a6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.221+0000 7fdf8f7fe640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf90114510 0x7fdf901a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.221+0000 7fdf8f7fe640 1 -- 172.21.15.16:0/2514266382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf900776e0 msgr2=0x7fdf901a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.221+0000 7fdf8f7fe640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf900776e0 0x7fdf901a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.221+0000 7fdf8f7fe640 1 -- 172.21.15.16:0/2514266382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf901a69c0 con 0x7fdf90078030 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.223+0000 7fdf8f7fe640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf90078030 0x7fdf901a0ff0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdf8802aee0 tx=0x7fdf88031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.223+0000 7fdf8d7fa640 1 -- 172.21.15.16:0/2514266382 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88003cf0 con 0x7fdf90078030 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.223+0000 7fdf969a2640 1 -- 172.21.15.16:0/2514266382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf901a6c50 con 0x7fdf90078030 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.223+0000 7fdf8d7fa640 1 -- 172.21.15.16:0/2514266382 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf8802d070 con 0x7fdf90078030 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.223+0000 7fdf969a2640 1 -- 172.21.15.16:0/2514266382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf901ad410 con 0x7fdf90078030 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.223+0000 7fdf8d7fa640 1 -- 172.21.15.16:0/2514266382 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88003cf0 con 0x7fdf90078030 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.227+0000 7fdf9cf17740 1 Processor -- start 2024-08-28T19:40:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.228+0000 7fdf9cf17740 1 -- start start 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.228+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f63350 0x55cb11309780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.228+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11309cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.228+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f42670 0x55cb1130a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.228+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cb1127f5f0 con 0x55cb10f63350 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.228+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cb1126eb80 con 0x55cb10f3a490 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.228+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cb11272b30 con 0x55cb10f42670 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.229+0000 7fdf8d7fa640 1 -- 172.21.15.16:0/2514266382 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fdf880045f0 con 0x7fdf90078030 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.229+0000 7fdf94f3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f63350 0x55cb11309780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.230+0000 7fdf8f7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f42670 0x55cb1130a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.231+0000 7fdf8d7fa640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf780b50a0 0x7fdf780b75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.229+0000 7fdf8ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11309cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.231+0000 7fdf8ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11309cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54922/0 (socket says 172.21.15.16:54922) 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.231+0000 7fdf8ffff640 1 -- 172.21.15.16:0/2447469115 learned_addr learned my addr 172.21.15.16:0/2447469115 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.231+0000 7fdf8d7fa640 1 -- 172.21.15.16:0/2514266382 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fdf88003af0 con 0x7fdf90078030 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.232+0000 7fdf94f3f640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf780b50a0 0x7fdf780b75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.232+0000 7fdf8ffff640 1 -- 172.21.15.16:0/2447469115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f42670 msgr2=0x55cb1130a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.232+0000 7fdf8ffff640 1 --2- 172.21.15.16:0/2447469115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f42670 0x55cb1130a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.232+0000 7fdf8ffff640 1 -- 172.21.15.16:0/2447469115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f63350 msgr2=0x55cb11309780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.232+0000 7fdf8ffff640 1 --2- 172.21.15.16:0/2447469115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f63350 0x55cb11309780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.232+0000 7fdf8ffff640 1 -- 172.21.15.16:0/2447469115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb10fce930 con 0x55cb10f3a490 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.233+0000 7fdf8ffff640 1 --2- 172.21.15.16:0/2447469115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11309cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdf840097d0 tx=0x7fdf84009c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b8f8210470b1cba server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.233+0000 7fdf767fc640 1 -- 172.21.15.16:0/2447469115 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84015020 con 0x55cb10f3a490 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.233+0000 7fdf767fc640 1 -- 172.21.15.16:0/2447469115 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf84011420 con 0x55cb10f3a490 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.234+0000 7fdf767fc640 1 -- 172.21.15.16:0/2447469115 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84015020 con 0x55cb10f3a490 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.234+0000 7fdf9cf17740 1 -- 172.21.15.16:0/2447469115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 msgr2=0x55cb11309cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.234+0000 7fdf94f3f640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf780b50a0 0x7fdf780b75b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000dab0 tx=0x7fdf80008040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:38.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.234+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/2447469115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11309cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdf840097d0 tx=0x7fdf84009c90 comp rx=0 tx=0).stop 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.235+0000 7fdf9cf17740 1 -- 172.21.15.16:0/2447469115 shutdown_connections 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.235+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/2447469115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f42670 0x55cb1130a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.235+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/2447469115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11309cc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.235+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/2447469115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f63350 0x55cb11309780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.235+0000 7fdf9cf17740 1 -- 172.21.15.16:0/2447469115 >> 172.21.15.16:0/2447469115 conn(0x55cb10f57740 msgr2=0x55cb11307670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.235+0000 7fdf9cf17740 1 -- 172.21.15.16:0/2447469115 shutdown_connections 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.235+0000 7fdf9cf17740 1 -- 172.21.15.16:0/2447469115 wait complete. 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.235+0000 7fdf9cf17740 1 -- start start 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.236+0000 7fdf9cf17740 1 Processor -- start 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.236+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11314e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.236+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 0x55cb11315340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.236+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f63350 0x55cb113158a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.236+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cb110d1890 con 0x55cb10f42670 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.236+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cb111f5790 con 0x55cb10f3a490 2024-08-28T19:40:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.236+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55cb11317fb0 con 0x55cb10f63350 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.237+0000 7fdf9cf17740 10 client.0 _ll_register_callbacks cb 0x55cb10f7aba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.237+0000 7fdf8ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11314e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.237+0000 7fdf8ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11314e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54938/0 (socket says 172.21.15.16:54938) 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.237+0000 7fdf8ffff640 1 -- 172.21.15.16:0/1387091068 learned_addr learned my addr 172.21.15.16:0/1387091068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.237+0000 7fdf94f3f640 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f63350 0x55cb113158a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.238+0000 7fdf8f7fe640 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 0x55cb11315340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.238+0000 7fdf8ffff640 1 -- 172.21.15.16:0/1387091068 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f63350 msgr2=0x55cb113158a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.238+0000 7fdf8ffff640 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f63350 0x55cb113158a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.238+0000 7fdf8ffff640 1 -- 172.21.15.16:0/1387091068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 msgr2=0x55cb11315340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.238+0000 7fdf8ffff640 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 0x55cb11315340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.238+0000 7fdf8ffff640 1 -- 172.21.15.16:0/1387091068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb1122cf30 con 0x55cb10f3a490 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.238+0000 7fdf8f7fe640 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 0x55cb11315340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.240+0000 7fdf8ffff640 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11314e00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdf8400bd80 tx=0x7fdf8400bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11d86263e2e5f0dd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.240+0000 7fdf6f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.242+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/1387091068 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84015020 con 0x55cb10f3a490 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.242+0000 7fdf9cf17740 1 -- 172.21.15.16:0/1387091068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf84011420 con 0x55cb10f3a490 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.242+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/1387091068 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fdf840117e0 con 0x55cb10f3a490 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.242+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/1387091068 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fdf840093e0 con 0x55cb10f3a490 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.243+0000 7fdf9cf17740 10 client.29495 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.243+0000 7fdf9cf17740 1 -- 172.21.15.16:0/1387091068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55cb11323c80 con 0x55cb10f3a490 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.243+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/1387091068 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf84011420 con 0x55cb10f3a490 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.243+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/1387091068 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fdf8401b430 con 0x55cb10f3a490 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.243+0000 7fdf9cf17740 10 client.29495 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.244+0000 7fdf9cf17740 10 client.29495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.244+0000 7fdf9cf17740 20 client.29495 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.244+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cb10f21cd0 0x55cb113275c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.244+0000 7fdf9cf17740 4 client.29495 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.244+0000 7fdf9cf17740 1 -- 172.21.15.16:0/1387091068 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb11281540 con 0x55cb10f21cd0 2024-08-28T19:40:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.244+0000 7fdf8f7fe640 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cb10f21cd0 0x55cb113275c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.245+0000 7fdf8f7fe640 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cb10f21cd0 0x55cb113275c0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2b4a8027a348c4f server_cookie=7071242fe63ac175 in_seq=0 out_seq=0 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.245+0000 7fdf6f7fe640 10 client.29495 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.248+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/1387091068 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cb11281540 con 0x55cb10f21cd0 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.248+0000 7fdf6f7fe640 10 client.29495 handle_command_reply: tid=0 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.248+0000 7fdf9cf17740 1 client.29495 shutdown 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.249+0000 7fdf9cf17740 1 -- 172.21.15.16:0/1387091068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 msgr2=0x55cb11314e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.249+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11314e00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdf8400bd80 tx=0x7fdf8400bdb0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.249+0000 7fdf9cf17740 1 -- 172.21.15.16:0/1387091068 shutdown_connections 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.249+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cb10f21cd0 0x55cb113275c0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.249+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f63350 0x55cb113158a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.249+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 0x55cb11315340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.250+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/1387091068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f3a490 0x55cb11314e00 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdf8400bd80 tx=0x7fdf8400bdb0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.250+0000 7fdf9cf17740 1 -- 172.21.15.16:0/1387091068 >> 172.21.15.16:0/1387091068 conn(0x55cb10f57740 msgr2=0x55cb1130c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.250+0000 7fdf6f7fe640 0 client.29495 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.251+0000 7fdf9cf17740 1 -- 172.21.15.16:0/1387091068 shutdown_connections 2024-08-28T19:40:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.251+0000 7fdf9cf17740 1 -- 172.21.15.16:0/1387091068 wait complete. 2024-08-28T19:40:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.251+0000 7fdf9cf17740 20 client.29495 trim_cache size 0 max 16384 2024-08-28T19:40:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.287+0000 7fdf9cf17740 1 Processor -- start 2024-08-28T19:40:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.287+0000 7fdf9cf17740 1 -- start start 2024-08-28T19:40:38.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.288+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f42670 0x55cb1130aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.288+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f63350 0x55cb11306ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.288+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10ef7080 0x55cb113071e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.288+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901b7620 con 0x55cb10f63350 2024-08-28T19:40:38.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.288+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901b92e0 con 0x55cb10ef7080 2024-08-28T19:40:38.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.288+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901b7d80 con 0x55cb10f42670 2024-08-28T19:40:38.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.290+0000 7fdf94f3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10ef7080 0x55cb113071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.290+0000 7fdf94f3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10ef7080 0x55cb113071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54942/0 (socket says 172.21.15.16:54942) 2024-08-28T19:40:38.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.290+0000 7fdf94f3f640 1 -- 172.21.15.16:0/3852455802 learned_addr learned my addr 172.21.15.16:0/3852455802 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.291+0000 7fdf8ffff640 1 --2- 172.21.15.16:0/3852455802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f42670 0x55cb1130aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.291+0000 7fdf94f3f640 1 -- 172.21.15.16:0/3852455802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f42670 msgr2=0x55cb1130aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.291+0000 7fdf94f3f640 1 --2- 172.21.15.16:0/3852455802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f42670 0x55cb1130aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.291+0000 7fdf94f3f640 1 -- 172.21.15.16:0/3852455802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f63350 msgr2=0x55cb11306ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.292+0000 7fdf94f3f640 1 --2- 172.21.15.16:0/3852455802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f63350 0x55cb11306ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.292+0000 7fdf94f3f640 1 -- 172.21.15.16:0/3852455802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb1130c5e0 con 0x55cb10ef7080 2024-08-28T19:40:38.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.292+0000 7fdf94f3f640 1 --2- 172.21.15.16:0/3852455802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10ef7080 0x55cb113071e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdf80010ac0 tx=0x7fdf80009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=115e58ac4ca63ad5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.293+0000 7fdf75ffb640 1 -- 172.21.15.16:0/3852455802 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80015bc0 con 0x55cb10ef7080 2024-08-28T19:40:38.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.293+0000 7fdf75ffb640 1 -- 172.21.15.16:0/3852455802 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf80026070 con 0x55cb10ef7080 2024-08-28T19:40:38.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.293+0000 7fdf75ffb640 1 -- 172.21.15.16:0/3852455802 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf800145f0 con 0x55cb10ef7080 2024-08-28T19:40:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.294+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3852455802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10ef7080 msgr2=0x55cb113071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.294+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/3852455802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10ef7080 0x55cb113071e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdf80010ac0 tx=0x7fdf80009060 comp rx=0 tx=0).stop 2024-08-28T19:40:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.295+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3852455802 shutdown_connections 2024-08-28T19:40:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.295+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/3852455802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10ef7080 0x55cb113071e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.295+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/3852455802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f63350 0x55cb11306ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.295+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/3852455802 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10f42670 0x55cb1130aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.295+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3852455802 >> 172.21.15.16:0/3852455802 conn(0x55cb10f57740 msgr2=0x55cb1130a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:38.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.295+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3852455802 shutdown_connections 2024-08-28T19:40:38.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.295+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3852455802 wait complete. 2024-08-28T19:40:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.296+0000 7fdf9cf17740 1 -- start start 2024-08-28T19:40:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.296+0000 7fdf9cf17740 1 Processor -- start 2024-08-28T19:40:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10ef7080 0x55cb11319150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 0x55cb11319690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf9cf17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f63350 0x55cb11319bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901b8b50 con 0x55cb10f42670 2024-08-28T19:40:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901b8450 con 0x55cb10f63350 2024-08-28T19:40:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf9cf17740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901b5620 con 0x55cb10ef7080 2024-08-28T19:40:38.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf9cf17740 10 client.0 _ll_register_callbacks cb 0x7fdf88009430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:38.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf94f3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f63350 0x55cb11319bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf94f3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f63350 0x55cb11319bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54958/0 (socket says 172.21.15.16:54958) 2024-08-28T19:40:38.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf94f3f640 1 -- 172.21.15.16:0/3055117563 learned_addr learned my addr 172.21.15.16:0/3055117563 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:38.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf8ffff640 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10ef7080 0x55cb11319150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.297+0000 7fdf94f3f640 1 -- 172.21.15.16:0/3055117563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10ef7080 msgr2=0x55cb11319150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:38.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.298+0000 7fdf94f3f640 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10ef7080 0x55cb11319150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.298+0000 7fdf8f7fe640 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 0x55cb11319690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.298+0000 7fdf94f3f640 1 -- 172.21.15.16:0/3055117563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 msgr2=0x55cb11319690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:38.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.298+0000 7fdf94f3f640 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 0x55cb11319690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.298+0000 7fdf94f3f640 1 -- 172.21.15.16:0/3055117563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb11390f80 con 0x55cb10f63350 2024-08-28T19:40:38.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.298+0000 7fdf8f7fe640 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 0x55cb11319690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:40:38.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.298+0000 7fdf94f3f640 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f63350 0x55cb11319bd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdf80006f70 tx=0x7fdf80015f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=945885fc2e29544 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:38.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.298+0000 7fdf6f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:40:38.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.299+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/3055117563 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80031070 con 0x55cb10f63350 2024-08-28T19:40:38.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.299+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/3055117563 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fdf80021830 con 0x55cb10f63350 2024-08-28T19:40:38.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.299+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3055117563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf80026070 con 0x55cb10f63350 2024-08-28T19:40:38.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.299+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/3055117563 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fdf8003d050 con 0x55cb10f63350 2024-08-28T19:40:38.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.300+0000 7fdf9cf17740 10 client.29501 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:38.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.300+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3055117563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55cb11392160 con 0x55cb10f63350 2024-08-28T19:40:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.302+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/3055117563 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf80026070 con 0x55cb10f63350 2024-08-28T19:40:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.302+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/3055117563 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fdf80029380 con 0x55cb10f63350 2024-08-28T19:40:38.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.302+0000 7fdf9cf17740 10 client.29501 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:38.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.302+0000 7fdf9cf17740 10 client.29501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:40:38.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.302+0000 7fdf9cf17740 20 client.29501 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:38.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.302+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cb113928c0 0x55cb11394ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:38.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.302+0000 7fdf9cf17740 4 client.29501 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:40:38.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.302+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3055117563 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cb1127cfe0 con 0x55cb113928c0 2024-08-28T19:40:38.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.304+0000 7fdf8ffff640 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cb113928c0 0x55cb11394ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:38.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.305+0000 7fdf8ffff640 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cb113928c0 0x55cb11394ce0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4519bdaca58166d7 server_cookie=cad13d2eb2cddb78 in_seq=0 out_seq=0 2024-08-28T19:40:38.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.305+0000 7fdf6f7fe640 10 client.29501 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:38.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.309+0000 7fdf6f7fe640 1 -- 172.21.15.16:0/3055117563 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cb1127cfe0 con 0x55cb113928c0 2024-08-28T19:40:38.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.309+0000 7fdf6f7fe640 10 client.29501 handle_command_reply: tid=0 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.309+0000 7fdf9cf17740 1 client.29501 shutdown 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.310+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3055117563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f63350 msgr2=0x55cb11319bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.310+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f63350 0x55cb11319bd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdf80006f70 tx=0x7fdf80015f90 comp rx=0 tx=0).stop 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.310+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3055117563 shutdown_connections 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.310+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb10f63350 0x55cb11319bd0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.310+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb10f42670 0x55cb11319690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.310+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55cb113928c0 0x55cb11394ce0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.310+0000 7fdf9cf17740 1 --2- 172.21.15.16:0/3055117563 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb10ef7080 0x55cb11319150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.310+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3055117563 >> 172.21.15.16:0/3055117563 conn(0x55cb10f57740 msgr2=0x55cb1130b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.310+0000 7fdf6f7fe640 0 client.29501 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.310+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3055117563 shutdown_connections 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.311+0000 7fdf9cf17740 1 -- 172.21.15.16:0/3055117563 wait complete. 2024-08-28T19:40:38.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.311+0000 7fdf9cf17740 20 client.29501 trim_cache size 0 max 16384 2024-08-28T19:40:38.313 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:40:38.313 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:40:38.313 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "a466ab70-970a-4b3e-927e-47c65e508132", 2024-08-28T19:40:38.313 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:40:38.313 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:40:38.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 -- 172.21.15.16:0/2514266382 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf780b50a0 msgr2=0x7fdf780b75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:38.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf780b50a0 0x7fdf780b75b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000dab0 tx=0x7fdf80008040 comp rx=0 tx=0).stop 2024-08-28T19:40:38.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 -- 172.21.15.16:0/2514266382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf90078030 msgr2=0x7fdf901a0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:38.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf90078030 0x7fdf901a0ff0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdf8802aee0 tx=0x7fdf88031a20 comp rx=0 tx=0).stop 2024-08-28T19:40:38.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 -- 172.21.15.16:0/2514266382 shutdown_connections 2024-08-28T19:40:38.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf90114510 0x7fdf901a6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fdf780b50a0 0x7fdf780b75b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf90078030 0x7fdf901a0ff0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 --2- 172.21.15.16:0/2514266382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf900776e0 0x7fdf901a0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:38.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 -- 172.21.15.16:0/2514266382 >> 172.21.15.16:0/2514266382 conn(0x7fdf90101560 msgr2=0x7fdf90103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:38.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 -- 172.21.15.16:0/2514266382 shutdown_connections 2024-08-28T19:40:38.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:38.312+0000 7fdf969a2640 1 -- 172.21.15.16:0/2514266382 wait complete. 2024-08-28T19:40:38.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 144682 22.59 MB/sec execute 254 sec latency 244.045 ms 2024-08-28T19:40:39.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 144926 22.58 MB/sec execute 255 sec latency 506.565 ms 2024-08-28T19:40:39.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: pgmap v953: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 8.0 MiB/s wr, 540 op/s 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:39 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:40:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: pgmap v953: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 8.0 MiB/s wr, 540 op/s 2024-08-28T19:40:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:40:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:40:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:40:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:40:39.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:40:39.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:40:39.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:40:39.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:39 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: pgmap v953: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 8.0 MiB/s wr, 540 op/s 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:39 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:39.929 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:40:40.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 145255 22.54 MB/sec execute 256 sec latency 782.965 ms 2024-08-28T19:40:40.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.511+0000 7fa3889c8640 1 -- 172.21.15.16:0/3135937892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa384104f10 msgr2=0x7fa384105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:40.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.511+0000 7fa3889c8640 1 --2- 172.21.15.16:0/3135937892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa384104f10 0x7fa384105320 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fa378009900 tx=0x7fa37802a410 comp rx=0 tx=0).stop 2024-08-28T19:40:40.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.511+0000 7fa3889c8640 1 -- 172.21.15.16:0/3135937892 shutdown_connections 2024-08-28T19:40:40.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.511+0000 7fa3889c8640 1 --2- 172.21.15.16:0/3135937892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa384105860 0x7fa38410f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.511+0000 7fa3889c8640 1 --2- 172.21.15.16:0/3135937892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa384104f10 0x7fa384105320 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.511+0000 7fa3889c8640 1 --2- 172.21.15.16:0/3135937892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa384102e20 0x7fa384103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.511+0000 7fa3889c8640 1 -- 172.21.15.16:0/3135937892 >> 172.21.15.16:0/3135937892 conn(0x7fa384077a00 msgr2=0x7fa384077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:40.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.511+0000 7fa3889c8640 1 -- 172.21.15.16:0/3135937892 shutdown_connections 2024-08-28T19:40:40.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.511+0000 7fa3889c8640 1 -- 172.21.15.16:0/3135937892 wait complete. 2024-08-28T19:40:40.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.512+0000 7fa3889c8640 1 Processor -- start 2024-08-28T19:40:40.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.512+0000 7fa3889c8640 1 -- start start 2024-08-28T19:40:40.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.512+0000 7fa3889c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa384102e20 0x7fa38419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa3889c8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa384104f10 0x7fa38419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa3889c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa384105860 0x7fa38419fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa3889c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa384112710 con 0x7fa384102e20 2024-08-28T19:40:40.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa3889c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa384112590 con 0x7fa384105860 2024-08-28T19:40:40.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa3889c8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa384112890 con 0x7fa384104f10 2024-08-28T19:40:40.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa38259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa384102e20 0x7fa38419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa38259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa384102e20 0x7fa38419a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50780/0 (socket says 172.21.15.16:50780) 2024-08-28T19:40:40.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa38259c640 1 -- 172.21.15.16:0/3807479905 learned_addr learned my addr 172.21.15.16:0/3807479905 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:40.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa382d9d640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa384105860 0x7fa38419fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa381d9b640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa384104f10 0x7fa38419a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa38259c640 1 -- 172.21.15.16:0/3807479905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa384104f10 msgr2=0x7fa38419a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa38259c640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa384104f10 0x7fa38419a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa38259c640 1 -- 172.21.15.16:0/3807479905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa384105860 msgr2=0x7fa38419fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa38259c640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa384105860 0x7fa38419fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.513+0000 7fa38259c640 1 -- 172.21.15.16:0/3807479905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3841a0260 con 0x7fa384102e20 2024-08-28T19:40:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.514+0000 7fa38259c640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa384102e20 0x7fa38419a2f0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fa37400e9f0 tx=0x7fa37400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:40.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.514+0000 7fa3737fe640 1 -- 172.21.15.16:0/3807479905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37400ccc0 con 0x7fa384102e20 2024-08-28T19:40:40.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.514+0000 7fa3889c8640 1 -- 172.21.15.16:0/3807479905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3841a0550 con 0x7fa384102e20 2024-08-28T19:40:40.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.514+0000 7fa3889c8640 1 -- 172.21.15.16:0/3807479905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3841a6cb0 con 0x7fa384102e20 2024-08-28T19:40:40.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.514+0000 7fa3737fe640 1 -- 172.21.15.16:0/3807479905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3740108e0 con 0x7fa384102e20 2024-08-28T19:40:40.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.514+0000 7fa3737fe640 1 -- 172.21.15.16:0/3807479905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa374018880 con 0x7fa384102e20 2024-08-28T19:40:40.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.517+0000 7fa38ef3d740 1 Processor -- start 2024-08-28T19:40:40.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.517+0000 7fa38ef3d740 1 -- start start 2024-08-28T19:40:40.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.517+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2c0070 0x564cee665300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.517+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee2971b0 0x564cee665840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.517+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee665d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.517+0000 7fa38ef3d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564cee5dc630 con 0x564cee2c0070 2024-08-28T19:40:40.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.517+0000 7fa38ef3d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564cee5cbbc0 con 0x564cee29f390 2024-08-28T19:40:40.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.517+0000 7fa38ef3d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564cee5cfb70 con 0x564cee2971b0 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa382d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2c0070 0x564cee665300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.517+0000 7fa381d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee665d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa381d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee665d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54990/0 (socket says 172.21.15.16:54990) 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa381d9b640 1 -- 172.21.15.16:0/1078971719 learned_addr learned my addr 172.21.15.16:0/1078971719 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa3737fe640 1 -- 172.21.15.16:0/3807479905 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa374010430 con 0x7fa384102e20 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa38259c640 1 --2- 172.21.15.16:0/1078971719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee2971b0 0x564cee665840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa381d9b640 1 -- 172.21.15.16:0/1078971719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee2971b0 msgr2=0x564cee665840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa381d9b640 1 --2- 172.21.15.16:0/1078971719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee2971b0 0x564cee665840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa381d9b640 1 -- 172.21.15.16:0/1078971719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2c0070 msgr2=0x564cee665300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa381d9b640 1 --2- 172.21.15.16:0/1078971719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2c0070 0x564cee665300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa381d9b640 1 -- 172.21.15.16:0/1078971719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cee32b650 con 0x564cee29f390 2024-08-28T19:40:40.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa38259c640 1 --2- 172.21.15.16:0/1078971719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee2971b0 0x564cee665840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:40:40.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.518+0000 7fa381d9b640 1 --2- 172.21.15.16:0/1078971719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee665d80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa378009900 tx=0x7fa378000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9850f9b0f73027f2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:40.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.519+0000 7fa370ff9640 1 -- 172.21.15.16:0/1078971719 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa378040070 con 0x564cee29f390 2024-08-28T19:40:40.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.519+0000 7fa370ff9640 1 -- 172.21.15.16:0/1078971719 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa378004030 con 0x564cee29f390 2024-08-28T19:40:40.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.519+0000 7fa3737fe640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa36c0b4d20 0x7fa36c0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.519+0000 7fa370ff9640 1 -- 172.21.15.16:0/1078971719 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa378035040 con 0x564cee29f390 2024-08-28T19:40:40.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.519+0000 7fa382d9d640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa36c0b4d20 0x7fa36c0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.519+0000 7fa3737fe640 1 -- 172.21.15.16:0/3807479905 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fa3740e5b60 con 0x7fa384102e20 2024-08-28T19:40:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.521+0000 7fa38ef3d740 1 -- 172.21.15.16:0/1078971719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 msgr2=0x564cee665d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.521+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/1078971719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee665d80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa378009900 tx=0x7fa378000bc0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.521+0000 7fa382d9d640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa36c0b4d20 0x7fa36c0b7230 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa37c005fd0 tx=0x7fa37c009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.521+0000 7fa38ef3d740 1 -- 172.21.15.16:0/1078971719 shutdown_connections 2024-08-28T19:40:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.521+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/1078971719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee665d80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.521+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/1078971719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee2971b0 0x564cee665840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.521+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/1078971719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2c0070 0x564cee665300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.521+0000 7fa38ef3d740 1 -- 172.21.15.16:0/1078971719 >> 172.21.15.16:0/1078971719 conn(0x564cee2b4460 msgr2=0x564cee6631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.522+0000 7fa38ef3d740 1 -- 172.21.15.16:0/1078971719 shutdown_connections 2024-08-28T19:40:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.522+0000 7fa38ef3d740 1 -- 172.21.15.16:0/1078971719 wait complete. 2024-08-28T19:40:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.522+0000 7fa38ef3d740 1 -- start start 2024-08-28T19:40:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.523+0000 7fa38ef3d740 1 Processor -- start 2024-08-28T19:40:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.523+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee674e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.523+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee675340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee2c0070 0x564cee681190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38ef3d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564cee42e700 con 0x564cee2971b0 2024-08-28T19:40:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38ef3d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564cee552700 con 0x564cee29f390 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38ef3d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564cee677dd0 con 0x564cee2c0070 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee674e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38ef3d740 10 client.0 _ll_register_callbacks cb 0x564cee581dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee674e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50790/0 (socket says 172.21.15.16:50790) 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38259c640 1 -- 172.21.15.16:0/3379872890 learned_addr learned my addr 172.21.15.16:0/3379872890 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38259c640 1 -- 172.21.15.16:0/3379872890 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee2c0070 msgr2=0x564cee681190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38259c640 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee2c0070 0x564cee681190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38259c640 1 -- 172.21.15.16:0/3379872890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 msgr2=0x564cee675340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa381d9b640 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee675340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38259c640 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee675340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa38259c640 1 -- 172.21.15.16:0/3379872890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cee58b8b0 con 0x564cee2971b0 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.524+0000 7fa381d9b640 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee675340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:40:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.525+0000 7fa38259c640 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee674e00 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fa3740e2ee0 tx=0x7fa3740ab600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a156c6ae0ae711d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.525+0000 7fa369ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:40:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.525+0000 7fa369ffb640 1 -- 172.21.15.16:0/3379872890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3740ac520 con 0x564cee2971b0 2024-08-28T19:40:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.525+0000 7fa369ffb640 1 -- 172.21.15.16:0/3379872890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fa3740ab080 con 0x564cee2971b0 2024-08-28T19:40:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.525+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3379872890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa378004030 con 0x564cee2971b0 2024-08-28T19:40:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.525+0000 7fa369ffb640 1 -- 172.21.15.16:0/3379872890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fa3740abe20 con 0x564cee2971b0 2024-08-28T19:40:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.525+0000 7fa38ef3d740 10 client.20058 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.525+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3379872890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x564cee683b20 con 0x564cee2971b0 2024-08-28T19:40:40.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.526+0000 7fa369ffb640 1 -- 172.21.15.16:0/3379872890 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3740b0690 con 0x564cee2971b0 2024-08-28T19:40:40.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.526+0000 7fa369ffb640 1 -- 172.21.15.16:0/3379872890 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fa3740b0280 con 0x564cee2971b0 2024-08-28T19:40:40.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.526+0000 7fa38ef3d740 10 client.20058 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:40.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.526+0000 7fa38ef3d740 10 client.20058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:40:40.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.526+0000 7fa38ef3d740 20 client.20058 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:40.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.526+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x564cee27e9f0 0x564cee6871d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.526+0000 7fa38ef3d740 4 client.20058 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:40:40.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.526+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3379872890 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa378004030 con 0x564cee27e9f0 2024-08-28T19:40:40.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.526+0000 7fa381d9b640 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x564cee27e9f0 0x564cee6871d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.527+0000 7fa369ffb640 10 client.20058 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:40.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.527+0000 7fa381d9b640 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x564cee27e9f0 0x564cee6871d0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6787bcb962d12c90 server_cookie=698bf1929f13805b in_seq=0 out_seq=0 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.534+0000 7fa369ffb640 1 -- 172.21.15.16:0/3379872890 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fa378004030 con 0x564cee27e9f0 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.534+0000 7fa369ffb640 10 client.20058 handle_command_reply: tid=0 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.534+0000 7fa38ef3d740 1 client.20058 shutdown 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3379872890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 msgr2=0x564cee674e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee674e00 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fa3740e2ee0 tx=0x7fa3740ab600 comp rx=0 tx=0).stop 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3379872890 shutdown_connections 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x564cee27e9f0 0x564cee6871d0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee2c0070 0x564cee681190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee675340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3379872890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee674e00 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa369ffb640 0 client.20058 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3379872890 >> 172.21.15.16:0/3379872890 conn(0x564cee2b4460 msgr2=0x564cee666cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3379872890 shutdown_connections 2024-08-28T19:40:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3379872890 wait complete. 2024-08-28T19:40:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.535+0000 7fa38ef3d740 20 client.20058 trim_cache size 0 max 16384 2024-08-28T19:40:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa38ef3d740 1 Processor -- start 2024-08-28T19:40:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa38ef3d740 1 -- start start 2024-08-28T19:40:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee27e9f0 0x564cee65e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee66f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee29f390 0x564cee671ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa38ef3d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa36400ffe0 con 0x564cee2971b0 2024-08-28T19:40:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa38ef3d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa364011ca0 con 0x564cee27e9f0 2024-08-28T19:40:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa38ef3d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa364010740 con 0x564cee29f390 2024-08-28T19:40:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa38259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee27e9f0 0x564cee65e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa38259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee27e9f0 0x564cee65e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:55008/0 (socket says 172.21.15.16:55008) 2024-08-28T19:40:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa38259c640 1 -- 172.21.15.16:0/3045294901 learned_addr learned my addr 172.21.15.16:0/3045294901 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.564+0000 7fa382d9d640 1 --2- 172.21.15.16:0/3045294901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee29f390 0x564cee671ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.565+0000 7fa381d9b640 1 --2- 172.21.15.16:0/3045294901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee66f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.565+0000 7fa382d9d640 1 -- 172.21.15.16:0/3045294901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee27e9f0 msgr2=0x564cee65e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.565+0000 7fa382d9d640 1 --2- 172.21.15.16:0/3045294901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee27e9f0 0x564cee65e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.565+0000 7fa382d9d640 1 -- 172.21.15.16:0/3045294901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 msgr2=0x564cee66f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.565+0000 7fa382d9d640 1 --2- 172.21.15.16:0/3045294901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee66f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.565+0000 7fa382d9d640 1 -- 172.21.15.16:0/3045294901 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cee66aa00 con 0x564cee29f390 2024-08-28T19:40:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.566+0000 7fa382d9d640 1 --2- 172.21.15.16:0/3045294901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee29f390 0x564cee671ff0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa37c00f7a0 tx=0x7fa37c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef89d8fec7e6fe8e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.566+0000 7fa36bfff640 1 -- 172.21.15.16:0/3045294901 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c02d070 con 0x564cee29f390 2024-08-28T19:40:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.566+0000 7fa36bfff640 1 -- 172.21.15.16:0/3045294901 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa37c00fdd0 con 0x564cee29f390 2024-08-28T19:40:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.566+0000 7fa36bfff640 1 -- 172.21.15.16:0/3045294901 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c01d6b0 con 0x564cee29f390 2024-08-28T19:40:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.567+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3045294901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee29f390 msgr2=0x564cee671ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.567+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3045294901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee29f390 0x564cee671ff0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa37c00f7a0 tx=0x7fa37c004a40 comp rx=0 tx=0).stop 2024-08-28T19:40:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.567+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3045294901 shutdown_connections 2024-08-28T19:40:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.567+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3045294901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee29f390 0x564cee671ff0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.567+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3045294901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee66f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.567+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3045294901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee27e9f0 0x564cee65e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.567+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3045294901 >> 172.21.15.16:0/3045294901 conn(0x564cee2b4460 msgr2=0x564cee673680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.568+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3045294901 shutdown_connections 2024-08-28T19:40:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.568+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3045294901 wait complete. 2024-08-28T19:40:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.568+0000 7fa38ef3d740 1 -- start start 2024-08-28T19:40:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.568+0000 7fa38ef3d740 1 Processor -- start 2024-08-28T19:40:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.569+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee27e9f0 0x564cee67d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.569+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee67d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.569+0000 7fa38259c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee27e9f0 0x564cee67d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.569+0000 7fa38ef3d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee66cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.569+0000 7fa38259c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee27e9f0 0x564cee67d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:45128/0 (socket says 172.21.15.16:45128) 2024-08-28T19:40:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.569+0000 7fa38ef3d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa364011510 con 0x564cee2971b0 2024-08-28T19:40:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.569+0000 7fa38259c640 1 -- 172.21.15.16:0/3104404270 learned_addr learned my addr 172.21.15.16:0/3104404270 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:40:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.569+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3104404270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa364010e10 con 0x564cee29f390 2024-08-28T19:40:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.569+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3104404270 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa36400dfe0 con 0x564cee27e9f0 2024-08-28T19:40:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.569+0000 7fa38ef3d740 10 client.0 _ll_register_callbacks cb 0x7fa374009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:40:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.570+0000 7fa382d9d640 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee66cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.570+0000 7fa381d9b640 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee67d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.570+0000 7fa38259c640 1 -- 172.21.15.16:0/3104404270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 msgr2=0x564cee66cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.570+0000 7fa38259c640 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee66cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.570+0000 7fa38259c640 1 -- 172.21.15.16:0/3104404270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 msgr2=0x564cee67d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.570+0000 7fa38259c640 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee67d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.570+0000 7fa38259c640 1 -- 172.21.15.16:0/3104404270 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cee67dbf0 con 0x564cee27e9f0 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.570+0000 7fa38259c640 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee27e9f0 0x564cee67d0f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa3740a83c0 tx=0x7fa3740abcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd39b6c67b39dc4a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.570+0000 7fa369ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.570+0000 7fa369ffb640 1 -- 172.21.15.16:0/3104404270 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa374010ca0 con 0x564cee27e9f0 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.571+0000 7fa369ffb640 1 -- 172.21.15.16:0/3104404270 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9490+0+0 (secure 0 0 0) 0x7fa3740ab290 con 0x564cee27e9f0 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.571+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3104404270 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa37c00fdd0 con 0x564cee27e9f0 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.571+0000 7fa369ffb640 1 -- 172.21.15.16:0/3104404270 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fa3740ac520 con 0x564cee27e9f0 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.571+0000 7fa38ef3d740 10 client.29077 fetch_fsmap learned FSMap version 101 2024-08-28T19:40:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.571+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3104404270 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x564cee6f66e0 con 0x564cee27e9f0 2024-08-28T19:40:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.571+0000 7fa369ffb640 1 -- 172.21.15.16:0/3104404270 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa37c00fdd0 con 0x564cee27e9f0 2024-08-28T19:40:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.571+0000 7fa369ffb640 1 -- 172.21.15.16:0/3104404270 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fa3740c6bd0 con 0x564cee27e9f0 2024-08-28T19:40:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.571+0000 7fa38ef3d740 10 client.29077 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:40:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.572+0000 7fa38ef3d740 10 client.29077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:40:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.572+0000 7fa38ef3d740 20 client.29077 populate_metadata read hostname 'smithi016' 2024-08-28T19:40:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.572+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x564cee2c0070 0x564cee6f8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:40:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.572+0000 7fa38ef3d740 4 client.29077 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:40:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.572+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3104404270 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564cee5de580 con 0x564cee2c0070 2024-08-28T19:40:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.573+0000 7fa381d9b640 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x564cee2c0070 0x564cee6f8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:40:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.573+0000 7fa369ffb640 10 client.29077 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.573+0000 7fa381d9b640 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x564cee2c0070 0x564cee6f8de0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bd6c7765429a6bd server_cookie=9f334d12c5476f56 in_seq=0 out_seq=0 2024-08-28T19:40:40.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa369ffb640 1 -- 172.21.15.16:0/3104404270 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7670 (crc 0 0 0) 0x564cee5de580 con 0x564cee2c0070 2024-08-28T19:40:40.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa369ffb640 10 client.29077 handle_command_reply: tid=0 2024-08-28T19:40:40.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa38ef3d740 1 client.29077 shutdown 2024-08-28T19:40:40.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3104404270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee27e9f0 msgr2=0x564cee67d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:40.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee27e9f0 0x564cee67d0f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa3740a83c0 tx=0x7fa3740abcf0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3104404270 shutdown_connections 2024-08-28T19:40:40.578 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:40:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "a466ab70-970a-4b3e-927e-47c65e508132": { 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "a466ab70-970a-4b3e-927e-47c65e508132", 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "a466ab70-970a-4b3e-927e-47c65e508132": { 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:40:38.307336+0000", 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:55.619275+0000", 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:40:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:40:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:40:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:40:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:40:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:40:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:40:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:40:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:40:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "8de90bf3-5743-4711-98c2-a56ca0757291": { 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:39:48.098772+0000", 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:40:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:40:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:40:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:40:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:40:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:40:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:40:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:40:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:40:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:40:40.596 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564cee29f390 0x564cee66cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cee2971b0 0x564cee67d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x564cee2c0070 0x564cee6f8de0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa38ef3d740 1 --2- 172.21.15.16:0/3104404270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564cee27e9f0 0x564cee67d0f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3104404270 >> 172.21.15.16:0/3104404270 conn(0x564cee2b4460 msgr2=0x564cee6662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa369ffb640 0 client.29077 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3104404270 shutdown_connections 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.576+0000 7fa38ef3d740 1 -- 172.21.15.16:0/3104404270 wait complete. 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.577+0000 7fa38ef3d740 20 client.29077 trim_cache size 0 max 16384 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.577+0000 7fa3889c8640 1 -- 172.21.15.16:0/3807479905 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa36c0b4d20 msgr2=0x7fa36c0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.577+0000 7fa3889c8640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa36c0b4d20 0x7fa36c0b7230 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa37c005fd0 tx=0x7fa37c009500 comp rx=0 tx=0).stop 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.577+0000 7fa3889c8640 1 -- 172.21.15.16:0/3807479905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa384102e20 msgr2=0x7fa38419a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:40:40.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.577+0000 7fa3889c8640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa384102e20 0x7fa38419a2f0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fa37400e9f0 tx=0x7fa37400edf0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.577+0000 7fa3889c8640 1 -- 172.21.15.16:0/3807479905 shutdown_connections 2024-08-28T19:40:40.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.577+0000 7fa3889c8640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa36c0b4d20 0x7fa36c0b7230 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.577+0000 7fa3889c8640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa384105860 0x7fa38419fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.577+0000 7fa3889c8640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa384104f10 0x7fa38419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.577+0000 7fa3889c8640 1 --2- 172.21.15.16:0/3807479905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa384102e20 0x7fa38419a2f0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:40:40.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.577+0000 7fa3889c8640 1 -- 172.21.15.16:0/3807479905 >> 172.21.15.16:0/3807479905 conn(0x7fa384077a00 msgr2=0x7fa384100c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:40:40.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.578+0000 7fa3889c8640 1 -- 172.21.15.16:0/3807479905 shutdown_connections 2024-08-28T19:40:40.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:40:40.578+0000 7fa3889c8640 1 -- 172.21.15.16:0/3807479905 wait complete. 2024-08-28T19:40:40.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:40 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:40.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:40 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:40 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:40.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:40 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:40.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:40 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:40.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:40 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:40:41.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 145928 22.55 MB/sec execute 257 sec latency 140.734 ms 2024-08-28T19:40:41.556 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:41 smithi016 ceph-mon[18556]: pgmap v954: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 8.8 MiB/s wr, 627 op/s 2024-08-28T19:40:41.591 INFO:tasks.cephfs.filesystem:scrub status for tag:a466ab70-970a-4b3e-927e-47c65e508132 - {'path': '/', 'tag': 'a466ab70-970a-4b3e-927e-47c65e508132', 'options': 'recursive,force'} 2024-08-28T19:40:41.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:41 smithi028 ceph-mon[21769]: pgmap v954: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 8.8 MiB/s wr, 627 op/s 2024-08-28T19:40:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:41 smithi115 ceph-mon[22184]: pgmap v954: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 8.8 MiB/s wr, 627 op/s 2024-08-28T19:40:42.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 146811 22.65 MB/sec execute 258 sec latency 129.466 ms 2024-08-28T19:40:43.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 147496 22.74 MB/sec execute 259 sec latency 186.213 ms 2024-08-28T19:40:43.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:43 smithi016 ceph-mon[18556]: pgmap v955: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 9.3 MiB/s wr, 651 op/s 2024-08-28T19:40:43.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:43 smithi028 ceph-mon[21769]: pgmap v955: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 9.3 MiB/s wr, 651 op/s 2024-08-28T19:40:43.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:43 smithi115 ceph-mon[22184]: pgmap v955: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 9.3 MiB/s wr, 651 op/s 2024-08-28T19:40:44.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148107 22.78 MB/sec execute 260 sec latency 74.232 ms 2024-08-28T19:40:45.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148695 22.82 MB/sec execute 261 sec latency 87.791 ms 2024-08-28T19:40:45.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:45 smithi016 ceph-mon[18556]: pgmap v956: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 551 op/s 2024-08-28T19:40:45.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:45 smithi028 ceph-mon[21769]: pgmap v956: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 551 op/s 2024-08-28T19:40:45.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:45 smithi115 ceph-mon[22184]: pgmap v956: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 551 op/s 2024-08-28T19:40:46.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 149284 22.81 MB/sec execute 262 sec latency 123.039 ms 2024-08-28T19:40:47.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150097 22.90 MB/sec execute 263 sec latency 52.311 ms 2024-08-28T19:40:47.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:47 smithi016 ceph-mon[18556]: pgmap v957: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 9.2 MiB/s wr, 653 op/s 2024-08-28T19:40:47.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:47 smithi028 ceph-mon[21769]: pgmap v957: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 9.2 MiB/s wr, 653 op/s 2024-08-28T19:40:47.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:47 smithi115 ceph-mon[22184]: pgmap v957: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 9.2 MiB/s wr, 653 op/s 2024-08-28T19:40:48.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150810 23.00 MB/sec execute 264 sec latency 85.156 ms 2024-08-28T19:40:49.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 151267 22.96 MB/sec execute 265 sec latency 145.183 ms 2024-08-28T19:40:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:49 smithi016 ceph-mon[18556]: pgmap v958: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 7.1 MiB/s wr, 517 op/s 2024-08-28T19:40:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:49 smithi028 ceph-mon[21769]: pgmap v958: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 7.1 MiB/s wr, 517 op/s 2024-08-28T19:40:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:49 smithi115 ceph-mon[22184]: pgmap v958: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 7.1 MiB/s wr, 517 op/s 2024-08-28T19:40:50.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 151986 23.04 MB/sec execute 266 sec latency 94.762 ms 2024-08-28T19:40:50.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:50 smithi028 ceph-mon[21769]: pgmap v959: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 9.1 MiB/s wr, 645 op/s 2024-08-28T19:40:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:50 smithi115 ceph-mon[22184]: pgmap v959: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 9.1 MiB/s wr, 645 op/s 2024-08-28T19:40:51.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:50 smithi016 ceph-mon[18556]: pgmap v959: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 9.1 MiB/s wr, 645 op/s 2024-08-28T19:40:51.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 152751 23.13 MB/sec execute 267 sec latency 84.880 ms 2024-08-28T19:40:52.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 153381 23.14 MB/sec execute 268 sec latency 75.215 ms 2024-08-28T19:40:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:52 smithi016 ceph-mon[18556]: pgmap v960: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 9.0 MiB/s wr, 617 op/s 2024-08-28T19:40:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:52 smithi028 ceph-mon[21769]: pgmap v960: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 9.0 MiB/s wr, 617 op/s 2024-08-28T19:40:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:52 smithi115 ceph-mon[22184]: pgmap v960: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 9.0 MiB/s wr, 617 op/s 2024-08-28T19:40:53.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 153818 23.14 MB/sec execute 269 sec latency 474.801 ms 2024-08-28T19:40:54.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:53 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:40:54.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:53 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:40:54.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:53 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:40:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:53 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:40:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:53 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:40:54.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:53 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:40:54.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:53 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:40:54.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:53 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:40:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:53 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:40:54.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 153841 23.06 MB/sec execute 270 sec latency 1451.735 ms 2024-08-28T19:40:55.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:54 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:40:55.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:54 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:40:55.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:54 smithi016 ceph-mon[18556]: pgmap v961: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 7.5 MiB/s wr, 503 op/s 2024-08-28T19:40:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:54 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:40:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:54 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:40:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:54 smithi028 ceph-mon[21769]: pgmap v961: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 7.5 MiB/s wr, 503 op/s 2024-08-28T19:40:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:54 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:40:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:54 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:40:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:54 smithi115 ceph-mon[22184]: pgmap v961: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 7.5 MiB/s wr, 503 op/s 2024-08-28T19:40:55.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 153921 22.98 MB/sec execute 271 sec latency 1648.190 ms 2024-08-28T19:40:56.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 154207 22.98 MB/sec execute 272 sec latency 347.803 ms 2024-08-28T19:40:57.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:56 smithi016 ceph-mon[18556]: pgmap v962: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 16 MiB/s wr, 740 op/s 2024-08-28T19:40:57.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:56 smithi028 ceph-mon[21769]: pgmap v962: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 16 MiB/s wr, 740 op/s 2024-08-28T19:40:57.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:56 smithi115 ceph-mon[22184]: pgmap v962: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 16 MiB/s wr, 740 op/s 2024-08-28T19:40:57.483 INFO:tasks.workunit.client.0.smithi016.stdout: 10 154478 22.94 MB/sec execute 273 sec latency 134.602 ms 2024-08-28T19:40:58.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 154767 22.91 MB/sec execute 274 sec latency 289.303 ms 2024-08-28T19:40:59.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:58 smithi016 ceph-mon[18556]: pgmap v963: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 16 MiB/s wr, 631 op/s 2024-08-28T19:40:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:58 smithi028 ceph-mon[21769]: pgmap v963: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 16 MiB/s wr, 631 op/s 2024-08-28T19:40:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:58 smithi115 ceph-mon[22184]: pgmap v963: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 16 MiB/s wr, 631 op/s 2024-08-28T19:40:59.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 155032 22.86 MB/sec execute 275 sec latency 395.836 ms 2024-08-28T19:41:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:40:59 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:41:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:40:59 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:41:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:40:59 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:41:00.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 155404 22.84 MB/sec execute 276 sec latency 257.978 ms 2024-08-28T19:41:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:00 smithi016 ceph-mon[18556]: pgmap v964: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 21 MiB/s wr, 799 op/s 2024-08-28T19:41:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:00 smithi028 ceph-mon[21769]: pgmap v964: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 21 MiB/s wr, 799 op/s 2024-08-28T19:41:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:00 smithi115 ceph-mon[22184]: pgmap v964: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 21 MiB/s wr, 799 op/s 2024-08-28T19:41:01.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 155683 22.83 MB/sec execute 277 sec latency 254.666 ms 2024-08-28T19:41:02.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 155909 22.80 MB/sec execute 278 sec latency 443.648 ms 2024-08-28T19:41:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:03 smithi028 ceph-mon[21769]: pgmap v965: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 21 MiB/s wr, 822 op/s 2024-08-28T19:41:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:03 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:03 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:03 smithi115 ceph-mon[22184]: pgmap v965: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 21 MiB/s wr, 822 op/s 2024-08-28T19:41:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:03 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:03 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:03.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 156083 22.73 MB/sec execute 279 sec latency 475.303 ms 2024-08-28T19:41:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:03 smithi016 ceph-mon[18556]: pgmap v965: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 21 MiB/s wr, 822 op/s 2024-08-28T19:41:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:03 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:03 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:04.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 156090 22.66 MB/sec execute 280 sec latency 612.666 ms 2024-08-28T19:41:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:05 smithi028 ceph-mon[21769]: pgmap v966: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 20 MiB/s wr, 784 op/s 2024-08-28T19:41:05.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:05 smithi028 ceph-mon[21769]: osdmap e111: 12 total, 12 up, 12 in 2024-08-28T19:41:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:05 smithi115 ceph-mon[22184]: pgmap v966: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 20 MiB/s wr, 784 op/s 2024-08-28T19:41:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:05 smithi115 ceph-mon[22184]: osdmap e111: 12 total, 12 up, 12 in 2024-08-28T19:41:05.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 156159 22.60 MB/sec execute 281 sec latency 549.999 ms 2024-08-28T19:41:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:05 smithi016 ceph-mon[18556]: pgmap v966: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 20 MiB/s wr, 784 op/s 2024-08-28T19:41:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:05 smithi016 ceph-mon[18556]: osdmap e111: 12 total, 12 up, 12 in 2024-08-28T19:41:06.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:06 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:06.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:06 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:06.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:06 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:06.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:06 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:06.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:06 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:06.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:06 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:06.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:06 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:06.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:06 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:06.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 156383 22.56 MB/sec execute 282 sec latency 421.047 ms 2024-08-28T19:41:06.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:06 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:06.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:06 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:06.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:06 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:06.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:06 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:07 smithi028 ceph-mon[21769]: pgmap v968: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 14 MiB/s wr, 791 op/s 2024-08-28T19:41:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:41:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:41:07.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:07 smithi115 ceph-mon[22184]: pgmap v968: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 14 MiB/s wr, 791 op/s 2024-08-28T19:41:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:41:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:41:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:07.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 156812 22.58 MB/sec execute 283 sec latency 339.038 ms 2024-08-28T19:41:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:07 smithi016 ceph-mon[18556]: pgmap v968: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 14 MiB/s wr, 791 op/s 2024-08-28T19:41:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:41:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:41:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:41:08.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 157487 22.63 MB/sec execute 284 sec latency 58.523 ms 2024-08-28T19:41:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:09 smithi115 ceph-mon[22184]: pgmap v969: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 11 MiB/s wr, 775 op/s 2024-08-28T19:41:09.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 158048 22.66 MB/sec execute 285 sec latency 70.634 ms 2024-08-28T19:41:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:09 smithi016 ceph-mon[18556]: pgmap v969: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 11 MiB/s wr, 775 op/s 2024-08-28T19:41:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:09 smithi028 ceph-mon[21769]: pgmap v969: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 11 MiB/s wr, 775 op/s 2024-08-28T19:41:10.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 158603 22.68 MB/sec execute 286 sec latency 1032.118 ms 2024-08-28T19:41:11.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 159213 22.71 MB/sec execute 287 sec latency 2032.150 ms 2024-08-28T19:41:11.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:41:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:11 smithi016 ceph-mon[18556]: pgmap v970: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 6.0 MiB/s wr, 654 op/s 2024-08-28T19:41:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:11 smithi016 ceph-mon[18556]: osdmap e112: 12 total, 12 up, 12 in 2024-08-28T19:41:11.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:11 smithi028 ceph-mon[21769]: pgmap v970: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 6.0 MiB/s wr, 654 op/s 2024-08-28T19:41:11.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:11 smithi028 ceph-mon[21769]: osdmap e112: 12 total, 12 up, 12 in 2024-08-28T19:41:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:11 smithi115 ceph-mon[22184]: pgmap v970: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 6.0 MiB/s wr, 654 op/s 2024-08-28T19:41:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:11 smithi115 ceph-mon[22184]: osdmap e112: 12 total, 12 up, 12 in 2024-08-28T19:41:11.917 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:12.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.477+0000 7fe6caffd640 1 -- 172.21.15.16:0/1190733600 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc030850 con 0x7fe6cc10bf50 2024-08-28T19:41:12.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.478+0000 7fe6d244d640 1 -- 172.21.15.16:0/1190733600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc10bf50 msgr2=0x7fe6cc1109c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:12.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.478+0000 7fe6d244d640 1 --2- 172.21.15.16:0/1190733600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc10bf50 0x7fe6cc1109c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009860 tx=0x7fe6bc02a340 comp rx=0 tx=0).stop 2024-08-28T19:41:12.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.478+0000 7fe6d244d640 1 -- 172.21.15.16:0/1190733600 shutdown_connections 2024-08-28T19:41:12.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.478+0000 7fe6d244d640 1 --2- 172.21.15.16:0/1190733600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc10bf50 0x7fe6cc1109c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.478+0000 7fe6d244d640 1 --2- 172.21.15.16:0/1190733600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6cc10b600 0x7fe6cc10ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.478+0000 7fe6d244d640 1 --2- 172.21.15.16:0/1190733600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6cc10a160 0x7fe6cc10a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.478+0000 7fe6d244d640 1 -- 172.21.15.16:0/1190733600 >> 172.21.15.16:0/1190733600 conn(0x7fe6cc070c60 msgr2=0x7fe6cc071070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:12.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.478+0000 7fe6d244d640 1 -- 172.21.15.16:0/1190733600 shutdown_connections 2024-08-28T19:41:12.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.479+0000 7fe6d244d640 1 -- 172.21.15.16:0/1190733600 wait complete. 2024-08-28T19:41:12.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.480+0000 7fe6d244d640 1 Processor -- start 2024-08-28T19:41:12.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.480+0000 7fe6d244d640 1 -- start start 2024-08-28T19:41:12.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.480+0000 7fe6d244d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc10a160 0x7fe6cc11db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.480+0000 7fe6d244d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6cc10b600 0x7fe6cc119e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.480+0000 7fe6d244d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6cc10bf50 0x7fe6cc11a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.480+0000 7fe6d244d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc11e390 con 0x7fe6cc10bf50 2024-08-28T19:41:12.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.480+0000 7fe6d244d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc11e210 con 0x7fe6cc10a160 2024-08-28T19:41:12.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.480+0000 7fe6d244d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc11e510 con 0x7fe6cc10b600 2024-08-28T19:41:12.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc10a160 0x7fe6cc11db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc10a160 0x7fe6cc11db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:53506/0 (socket says 172.21.15.16:53506) 2024-08-28T19:41:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6cbfff640 1 -- 172.21.15.16:0/2025991362 learned_addr learned my addr 172.21.15.16:0/2025991362 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6cc10b600 0x7fe6cc119e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6d09ea640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6cc10bf50 0x7fe6cc11a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6cbfff640 1 -- 172.21.15.16:0/2025991362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6cc10b600 msgr2=0x7fe6cc119e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6cc10b600 0x7fe6cc119e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6cbfff640 1 -- 172.21.15.16:0/2025991362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6cc10bf50 msgr2=0x7fe6cc11a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6cc10bf50 0x7fe6cc11a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6cbfff640 1 -- 172.21.15.16:0/2025991362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6cc1b51b0 con 0x7fe6cc10a160 2024-08-28T19:41:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6cc10b600 0x7fe6cc119e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:41:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.481+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc10a160 0x7fe6cc11db50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400efb0 tx=0x7fe6c400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:12.486 INFO:tasks.workunit.client.0.smithi016.stdout: 10 159810 22.75 MB/sec execute 288 sec latency 3032.182 ms 2024-08-28T19:41:12.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.482+0000 7fe6c97fa640 1 -- 172.21.15.16:0/2025991362 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c401b070 con 0x7fe6cc10a160 2024-08-28T19:41:12.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.482+0000 7fe6d244d640 1 -- 172.21.15.16:0/2025991362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6cc1b53e0 con 0x7fe6cc10a160 2024-08-28T19:41:12.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.482+0000 7fe6d244d640 1 -- 172.21.15.16:0/2025991362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6cc1b5970 con 0x7fe6cc10a160 2024-08-28T19:41:12.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.482+0000 7fe6c97fa640 1 -- 172.21.15.16:0/2025991362 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c400ba20 con 0x7fe6cc10a160 2024-08-28T19:41:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.482+0000 7fe6c97fa640 1 -- 172.21.15.16:0/2025991362 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4016b30 con 0x7fe6cc10a160 2024-08-28T19:41:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.485+0000 7fe6d89c2740 1 Processor -- start 2024-08-28T19:41:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.486+0000 7fe6d89c2740 1 -- start start 2024-08-28T19:41:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.487+0000 7fe6c97fa640 1 -- 172.21.15.16:0/2025991362 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe6c401e070 con 0x7fe6cc10a160 2024-08-28T19:41:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.487+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3c34070 0x5576b3fd92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.487+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c0b1b0 0x5576b3fd9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.487+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c13390 0x5576b3fd9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.487+0000 7fe6d89c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5576b3f50630 con 0x5576b3c0b1b0 2024-08-28T19:41:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.487+0000 7fe6d89c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576b3f3fbc0 con 0x5576b3c34070 2024-08-28T19:41:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.487+0000 7fe6d89c2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5576b3f43b70 con 0x5576b3c13390 2024-08-28T19:41:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.487+0000 7fe6cb7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c13390 0x5576b3fd9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.487+0000 7fe6cb7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c13390 0x5576b3fd9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:51220/0 (socket says 172.21.15.16:51220) 2024-08-28T19:41:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.487+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/3244199419 learned_addr learned my addr 172.21.15.16:0/3244199419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.488+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/3244199419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c0b1b0 0x5576b3fd9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.488+0000 7fe6d09ea640 1 --2- 172.21.15.16:0/3244199419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3c34070 0x5576b3fd92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.488+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/3244199419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3c34070 msgr2=0x5576b3fd92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.488+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/3244199419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3c34070 0x5576b3fd92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.488+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/3244199419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c0b1b0 msgr2=0x5576b3fd9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.488+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/3244199419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c0b1b0 0x5576b3fd9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.488+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/3244199419 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576b3c9f650 con 0x5576b3c13390 2024-08-28T19:41:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.488+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/3244199419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c0b1b0 0x5576b3fd9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:41:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.488+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/3244199419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c13390 0x5576b3fd9d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009c90 tx=0x7fe6c00117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8320cecfb154d91 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.488+0000 7fe6c97fa640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b40b4ca0 0x7fe6b40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.489+0000 7fe6c97fa640 1 -- 172.21.15.16:0/2025991362 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7fe6c40e57e0 con 0x7fe6cc10a160 2024-08-28T19:41:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.489+0000 7fe6d09ea640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b40b4ca0 0x7fe6b40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.489+0000 7fe6b27fc640 1 -- 172.21.15.16:0/3244199419 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0012450 con 0x5576b3c13390 2024-08-28T19:41:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.489+0000 7fe6b27fc640 1 -- 172.21.15.16:0/3244199419 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c0012aa0 con 0x5576b3c13390 2024-08-28T19:41:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.490+0000 7fe6b27fc640 1 -- 172.21.15.16:0/3244199419 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c000fa50 con 0x5576b3c13390 2024-08-28T19:41:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.490+0000 7fe6d89c2740 1 -- 172.21.15.16:0/3244199419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c13390 msgr2=0x5576b3fd9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.490+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/3244199419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c13390 0x5576b3fd9d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009c90 tx=0x7fe6c00117b0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.491+0000 7fe6d09ea640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b40b4ca0 0x7fe6b40b71b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc02a930 tx=0x7fe6bc030340 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.491+0000 7fe6d89c2740 1 -- 172.21.15.16:0/3244199419 shutdown_connections 2024-08-28T19:41:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.491+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/3244199419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c13390 0x5576b3fd9d40 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009c90 tx=0x7fe6c00117b0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.491+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/3244199419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c0b1b0 0x5576b3fd9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.491+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/3244199419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3c34070 0x5576b3fd92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.491+0000 7fe6d89c2740 1 -- 172.21.15.16:0/3244199419 >> 172.21.15.16:0/3244199419 conn(0x5576b3c28460 msgr2=0x5576b3fd71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.493+0000 7fe6d89c2740 1 -- 172.21.15.16:0/3244199419 shutdown_connections 2024-08-28T19:41:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.493+0000 7fe6d89c2740 1 -- 172.21.15.16:0/3244199419 wait complete. 2024-08-28T19:41:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.493+0000 7fe6d89c2740 1 -- start start 2024-08-28T19:41:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.494+0000 7fe6d89c2740 1 Processor -- start 2024-08-28T19:41:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.494+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c0b1b0 0x5576b3ff5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.494+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c34070 0x5576b3ff5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.494+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3bf29f0 0x5576b4001e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.494+0000 7fe6d89c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5576b3da2700 con 0x5576b3c34070 2024-08-28T19:41:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.494+0000 7fe6d89c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576b3ec6700 con 0x5576b3bf29f0 2024-08-28T19:41:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.494+0000 7fe6d89c2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5576b3ff8a50 con 0x5576b3c0b1b0 2024-08-28T19:41:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.494+0000 7fe6d89c2740 10 client.0 _ll_register_callbacks cb 0x5576b3ef5dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.495+0000 7fe6cb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c34070 0x5576b3ff5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.495+0000 7fe6cb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c34070 0x5576b3ff5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46364/0 (socket says 172.21.15.16:46364) 2024-08-28T19:41:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.495+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/4276968463 learned_addr learned my addr 172.21.15.16:0/4276968463 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.496+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/4276968463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c0b1b0 msgr2=0x5576b3ff5a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:41:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.496+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/4276968463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c0b1b0 0x5576b3ff5a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.496+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/4276968463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3bf29f0 msgr2=0x5576b4001e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:41:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.496+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/4276968463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3bf29f0 0x5576b4001e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.496+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/4276968463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576b3eff8b0 con 0x5576b3c34070 2024-08-28T19:41:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.497+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/4276968463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c34070 0x5576b3ff5f80 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000f590 tx=0x7fe6c0009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f590a1d3fea2114e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.497+0000 7fe6ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.498+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/4276968463 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0029020 con 0x5576b3c34070 2024-08-28T19:41:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.498+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/4276968463 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7fe6c0012660 con 0x5576b3c34070 2024-08-28T19:41:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.498+0000 7fe6d89c2740 1 -- 172.21.15.16:0/4276968463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6c0012aa0 con 0x5576b3c34070 2024-08-28T19:41:12.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.499+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/4276968463 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fe6c0034050 con 0x5576b3c34070 2024-08-28T19:41:12.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.499+0000 7fe6d89c2740 10 client.20073 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:12.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.499+0000 7fe6d89c2740 1 -- 172.21.15.16:0/4276968463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5576b40048d0 con 0x5576b3c34070 2024-08-28T19:41:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.499+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/4276968463 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c0012aa0 con 0x5576b3c34070 2024-08-28T19:41:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.500+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/4276968463 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fe6c000f9e0 con 0x5576b3c34070 2024-08-28T19:41:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.500+0000 7fe6d89c2740 10 client.20073 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.500+0000 7fe6d89c2740 10 client.20073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:41:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.500+0000 7fe6d89c2740 20 client.20073 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.500+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/4276968463 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5576b3bc7da0 0x5576b4007eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.500+0000 7fe6d89c2740 4 client.20073 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:41:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.500+0000 7fe6d89c2740 1 -- 172.21.15.16:0/4276968463 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576b3f52580 con 0x5576b3bc7da0 2024-08-28T19:41:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.500+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/4276968463 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5576b3bc7da0 0x5576b4007eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.501+0000 7fe6ab7fe640 10 client.20073 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.501+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/4276968463 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5576b3bc7da0 0x5576b4007eb0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97e7f6e79c6acd5 server_cookie=eda7e2301b49b7c9 in_seq=0 out_seq=0 2024-08-28T19:41:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.501+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/4276968463 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5576b3f52580 con 0x5576b3bc7da0 2024-08-28T19:41:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.501+0000 7fe6ab7fe640 10 client.20073 handle_command_reply: tid=0 2024-08-28T19:41:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.501+0000 7fe6d89c2740 1 client.20073 shutdown 2024-08-28T19:41:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6d89c2740 1 -- 172.21.15.16:0/4276968463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c34070 msgr2=0x5576b3ff5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/4276968463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c34070 0x5576b3ff5f80 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000f590 tx=0x7fe6c0009df0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6d89c2740 1 -- 172.21.15.16:0/4276968463 shutdown_connections 2024-08-28T19:41:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/4276968463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3bf29f0 0x5576b4001e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/4276968463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c34070 0x5576b3ff5f80 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/4276968463 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5576b3bc7da0 0x5576b4007eb0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/4276968463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c0b1b0 0x5576b3ff5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6d89c2740 1 -- 172.21.15.16:0/4276968463 >> 172.21.15.16:0/4276968463 conn(0x5576b3c28460 msgr2=0x5576b3fdac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6ab7fe640 0 client.20073 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6d89c2740 1 -- 172.21.15.16:0/4276968463 shutdown_connections 2024-08-28T19:41:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6d89c2740 1 -- 172.21.15.16:0/4276968463 wait complete. 2024-08-28T19:41:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.502+0000 7fe6d89c2740 20 client.20073 trim_cache size 0 max 16384 2024-08-28T19:41:12.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.533+0000 7fe6d89c2740 1 Processor -- start 2024-08-28T19:41:12.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.534+0000 7fe6d89c2740 1 -- start start 2024-08-28T19:41:12.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.534+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3bf29f0 0x5576b3fd1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.534+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c0b1b0 0x5576b4001d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.534+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c34070 0x5576b3ff42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.534+0000 7fe6d89c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b800ff80 con 0x5576b3c0b1b0 2024-08-28T19:41:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.534+0000 7fe6d89c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b8011c40 con 0x5576b3bf29f0 2024-08-28T19:41:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.534+0000 7fe6cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3bf29f0 0x5576b3fd1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.534+0000 7fe6cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3bf29f0 0x5576b3fd1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:53534/0 (socket says 172.21.15.16:53534) 2024-08-28T19:41:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.534+0000 7fe6cbfff640 1 -- 172.21.15.16:0/887825349 learned_addr learned my addr 172.21.15.16:0/887825349 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:12.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.534+0000 7fe6d89c2740 1 -- 172.21.15.16:0/887825349 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b80106e0 con 0x5576b3c34070 2024-08-28T19:41:12.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.534+0000 7fe6d09ea640 1 --2- 172.21.15.16:0/887825349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c34070 0x5576b3ff42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.535+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/887825349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c0b1b0 0x5576b4001d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.535+0000 7fe6cbfff640 1 -- 172.21.15.16:0/887825349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c34070 msgr2=0x5576b3ff42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:12.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.535+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/887825349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c34070 0x5576b3ff42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.535+0000 7fe6cbfff640 1 -- 172.21.15.16:0/887825349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c0b1b0 msgr2=0x5576b4001d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:12.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.535+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/887825349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c0b1b0 0x5576b4001d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.535+0000 7fe6cbfff640 1 -- 172.21.15.16:0/887825349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576b3ff6990 con 0x5576b3bf29f0 2024-08-28T19:41:12.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.535+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/887825349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3bf29f0 0x5576b3fd1e10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4000f80 tx=0x7fe6c4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=860642bbf1f7d35d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:12.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.536+0000 7fe6b1ffb640 1 -- 172.21.15.16:0/887825349 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40b0170 con 0x5576b3bf29f0 2024-08-28T19:41:12.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.536+0000 7fe6b1ffb640 1 -- 172.21.15.16:0/887825349 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c40eb030 con 0x5576b3bf29f0 2024-08-28T19:41:12.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.536+0000 7fe6b1ffb640 1 -- 172.21.15.16:0/887825349 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40a7d90 con 0x5576b3bf29f0 2024-08-28T19:41:12.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.537+0000 7fe6d89c2740 1 -- 172.21.15.16:0/887825349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3bf29f0 msgr2=0x5576b3fd1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:12.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.537+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/887825349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3bf29f0 0x5576b3fd1e10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4000f80 tx=0x7fe6c4009b50 comp rx=0 tx=0).stop 2024-08-28T19:41:12.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.537+0000 7fe6d89c2740 1 -- 172.21.15.16:0/887825349 shutdown_connections 2024-08-28T19:41:12.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.537+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/887825349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c34070 0x5576b3ff42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.537+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/887825349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3c0b1b0 0x5576b4001d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.537+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/887825349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3bf29f0 0x5576b3fd1e10 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.537+0000 7fe6d89c2740 1 -- 172.21.15.16:0/887825349 >> 172.21.15.16:0/887825349 conn(0x5576b3c28460 msgr2=0x5576b3ff0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:12.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.538+0000 7fe6d89c2740 1 -- 172.21.15.16:0/887825349 shutdown_connections 2024-08-28T19:41:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.539+0000 7fe6d89c2740 1 -- 172.21.15.16:0/887825349 wait complete. 2024-08-28T19:41:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.539+0000 7fe6d89c2740 1 -- start start 2024-08-28T19:41:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.539+0000 7fe6d89c2740 1 Processor -- start 2024-08-28T19:41:12.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.540+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3bf29f0 0x5576b3ffe650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.540+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c0b1b0 0x5576b3ff9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.541+0000 7fe6d89c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3c34070 0x5576b3ffa150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.541+0000 7fe6cb7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c0b1b0 0x5576b3ff9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.541+0000 7fe6cb7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c0b1b0 0x5576b3ff9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:51238/0 (socket says 172.21.15.16:51238) 2024-08-28T19:41:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.541+0000 7fe6cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3bf29f0 0x5576b3ffe650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.541+0000 7fe6cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3bf29f0 0x5576b3ffe650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46390/0 (socket says 172.21.15.16:46390) 2024-08-28T19:41:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.541+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/2375194676 learned_addr learned my addr 172.21.15.16:0/2375194676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.541+0000 7fe6d09ea640 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3c34070 0x5576b3ffa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.541+0000 7fe6d89c2740 1 -- 172.21.15.16:0/2375194676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b80114b0 con 0x5576b3bf29f0 2024-08-28T19:41:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.541+0000 7fe6d89c2740 1 -- 172.21.15.16:0/2375194676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b8010db0 con 0x5576b3c34070 2024-08-28T19:41:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.541+0000 7fe6d89c2740 1 -- 172.21.15.16:0/2375194676 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b800df80 con 0x5576b3c0b1b0 2024-08-28T19:41:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.542+0000 7fe6d89c2740 10 client.0 _ll_register_callbacks cb 0x7fe6bc035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:12.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.542+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/2375194676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3c34070 msgr2=0x5576b3ffa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:12.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.542+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3c34070 0x5576b3ffa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.542+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/2375194676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3bf29f0 msgr2=0x5576b3ffe650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:12.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.542+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3bf29f0 0x5576b3ffe650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.542+0000 7fe6cb7fe640 1 -- 172.21.15.16:0/2375194676 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576b3ffa8c0 con 0x5576b3c0b1b0 2024-08-28T19:41:12.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.543+0000 7fe6cb7fe640 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c0b1b0 0x5576b3ff9b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0012fd0 tx=0x7fe6c0009c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce65346e731917a4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:12.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.543+0000 7fe6ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:41:12.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.543+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/2375194676 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c003c030 con 0x5576b3c0b1b0 2024-08-28T19:41:12.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.543+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/2375194676 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7fe6c0012660 con 0x5576b3c0b1b0 2024-08-28T19:41:12.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.543+0000 7fe6d89c2740 1 -- 172.21.15.16:0/2375194676 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6c40eb030 con 0x5576b3c0b1b0 2024-08-28T19:41:12.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.543+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/2375194676 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fe6c0046050 con 0x5576b3c0b1b0 2024-08-28T19:41:12.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.544+0000 7fe6d89c2740 10 client.29092 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:12.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.544+0000 7fe6d89c2740 1 -- 172.21.15.16:0/2375194676 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5576b40728b0 con 0x5576b3c0b1b0 2024-08-28T19:41:12.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.544+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/2375194676 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c40eb030 con 0x5576b3c0b1b0 2024-08-28T19:41:12.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.545+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/2375194676 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fe6c000f8c0 con 0x5576b3c0b1b0 2024-08-28T19:41:12.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.546+0000 7fe6d89c2740 10 client.29092 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:12.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.546+0000 7fe6d89c2740 10 client.29092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:41:12.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.546+0000 7fe6d89c2740 20 client.29092 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:12.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.546+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5576b4073050 0x5576b4075470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:12.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.546+0000 7fe6d89c2740 4 client.29092 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:41:12.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.546+0000 7fe6d89c2740 1 -- 172.21.15.16:0/2375194676 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5576b3f4e020 con 0x5576b4073050 2024-08-28T19:41:12.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.546+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5576b4073050 0x5576b4075470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:12.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.547+0000 7fe6cbfff640 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5576b4073050 0x5576b4075470 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97d87bff2c09fda8 server_cookie=771c21f1538226bb in_seq=0 out_seq=0 2024-08-28T19:41:12.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.547+0000 7fe6ab7fe640 10 client.29092 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6ab7fe640 1 -- 172.21.15.16:0/2375194676 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7474 (crc 0 0 0) 0x5576b3f4e020 con 0x5576b4073050 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6ab7fe640 10 client.29092 handle_command_reply: tid=0 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 1 client.29092 shutdown 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 1 -- 172.21.15.16:0/2375194676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c0b1b0 msgr2=0x5576b3ff9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c0b1b0 0x5576b3ff9b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0012fd0 tx=0x7fe6c0009c50 comp rx=0 tx=0).stop 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 1 -- 172.21.15.16:0/2375194676 shutdown_connections 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b3c34070 0x5576b3ffa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5576b4073050 0x5576b4075470 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5576b3c0b1b0 0x5576b3ff9b00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 1 --2- 172.21.15.16:0/2375194676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576b3bf29f0 0x5576b3ffe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6ab7fe640 0 client.29092 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 1 -- 172.21.15.16:0/2375194676 >> 172.21.15.16:0/2375194676 conn(0x5576b3c28460 msgr2=0x5576b3ff4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 1 -- 172.21.15.16:0/2375194676 shutdown_connections 2024-08-28T19:41:12.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 1 -- 172.21.15.16:0/2375194676 wait complete. 2024-08-28T19:41:12.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.548+0000 7fe6d89c2740 20 client.29092 trim_cache size 0 max 16384 2024-08-28T19:41:12.552 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:41:12.552 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:41:12.552 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:41:12.552 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "a466ab70-970a-4b3e-927e-47c65e508132": { 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:40:38.307336+0000", 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:55.619275+0000", 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.553 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:41:12.554 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.555 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:41:12.556 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.557 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.558 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.559 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.560 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "8de90bf3-5743-4711-98c2-a56ca0757291": { 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:39:48.098772+0000", 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:41:12.561 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:41:12.562 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.563 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.564 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.565 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.566 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:12.567 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:41:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.549+0000 7fe6d244d640 1 -- 172.21.15.16:0/2025991362 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b40b4ca0 msgr2=0x7fe6b40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.549+0000 7fe6d244d640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b40b4ca0 0x7fe6b40b71b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc02a930 tx=0x7fe6bc030340 comp rx=0 tx=0).stop 2024-08-28T19:41:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.549+0000 7fe6d244d640 1 -- 172.21.15.16:0/2025991362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc10a160 msgr2=0x7fe6cc11db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.549+0000 7fe6d244d640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc10a160 0x7fe6cc11db50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400efb0 tx=0x7fe6c400c800 comp rx=0 tx=0).stop 2024-08-28T19:41:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.550+0000 7fe6d244d640 1 -- 172.21.15.16:0/2025991362 shutdown_connections 2024-08-28T19:41:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.550+0000 7fe6d244d640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6cc10bf50 0x7fe6cc11a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.550+0000 7fe6d244d640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe6cc10b600 0x7fe6cc119e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.550+0000 7fe6d244d640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe6b40b4ca0 0x7fe6b40b71b0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.550+0000 7fe6d244d640 1 --2- 172.21.15.16:0/2025991362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc10a160 0x7fe6cc11db50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.550+0000 7fe6d244d640 1 -- 172.21.15.16:0/2025991362 >> 172.21.15.16:0/2025991362 conn(0x7fe6cc070c60 msgr2=0x7fe6cc10f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.550+0000 7fe6d244d640 1 -- 172.21.15.16:0/2025991362 shutdown_connections 2024-08-28T19:41:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:12.550+0000 7fe6d244d640 1 -- 172.21.15.16:0/2025991362 wait complete. 2024-08-28T19:41:13.203 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-08-28T19:41:13.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:41:13.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 160398 22.80 MB/sec execute 289 sec latency 4032.364 ms 2024-08-28T19:41:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:13 smithi016 ceph-mon[18556]: pgmap v972: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 668 op/s 2024-08-28T19:41:13.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:13 smithi028 ceph-mon[21769]: pgmap v972: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 668 op/s 2024-08-28T19:41:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:13 smithi115 ceph-mon[22184]: pgmap v972: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 668 op/s 2024-08-28T19:41:14.484 INFO:tasks.workunit.client.0.smithi016.stdout: 10 160920 22.78 MB/sec execute 290 sec latency 4065.684 ms 2024-08-28T19:41:14.642 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:41:14.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:41:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:15.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.301+0000 7f8088aa7640 1 -- 172.21.15.16:0/4210942454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8084111ce0 msgr2=0x7f808410a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.301+0000 7f8088aa7640 1 --2- 172.21.15.16:0/4210942454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8084111ce0 0x7f808410a160 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8078009860 tx=0x7f807802a340 comp rx=0 tx=0).stop 2024-08-28T19:41:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.301+0000 7f8088aa7640 1 -- 172.21.15.16:0/4210942454 shutdown_connections 2024-08-28T19:41:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.301+0000 7f8088aa7640 1 --2- 172.21.15.16:0/4210942454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f808411c9d0 0x7f808411eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.301+0000 7f8088aa7640 1 --2- 172.21.15.16:0/4210942454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808410a780 0x7f808410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.301+0000 7f8088aa7640 1 --2- 172.21.15.16:0/4210942454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8084111ce0 0x7f808410a160 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.301+0000 7f8088aa7640 1 -- 172.21.15.16:0/4210942454 >> 172.21.15.16:0/4210942454 conn(0x7f8084070c60 msgr2=0x7f8084071070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.302+0000 7f8088aa7640 1 -- 172.21.15.16:0/4210942454 shutdown_connections 2024-08-28T19:41:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.302+0000 7f8088aa7640 1 -- 172.21.15.16:0/4210942454 wait complete. 2024-08-28T19:41:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.303+0000 7f8088aa7640 1 Processor -- start 2024-08-28T19:41:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.303+0000 7f8088aa7640 1 -- start start 2024-08-28T19:41:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.304+0000 7f8088aa7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808410a780 0x7f808411c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.304+0000 7f8088aa7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8084111ce0 0x7f8084118470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.304+0000 7f8088aa7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808411c9d0 0x7f80841189b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.304+0000 7f8088aa7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8084114960 con 0x7f808410a780 2024-08-28T19:41:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.304+0000 7f8088aa7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80841147e0 con 0x7f808411c9d0 2024-08-28T19:41:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.304+0000 7f8088aa7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8084114ae0 con 0x7f8084111ce0 2024-08-28T19:41:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.304+0000 7f8081d9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8084111ce0 0x7f8084118470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.304+0000 7f8081d9b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8084111ce0 0x7f8084118470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:57910/0 (socket says 172.21.15.16:57910) 2024-08-28T19:41:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.304+0000 7f8081d9b640 1 -- 172.21.15.16:0/1309382484 learned_addr learned my addr 172.21.15.16:0/1309382484 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.304+0000 7f8082d9d640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808411c9d0 0x7f80841189b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.306+0000 7f8081d9b640 1 -- 172.21.15.16:0/1309382484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808411c9d0 msgr2=0x7f80841189b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:15.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.306+0000 7f8081d9b640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808411c9d0 0x7f80841189b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:15.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.306+0000 7f8081d9b640 1 -- 172.21.15.16:0/1309382484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808410a780 msgr2=0x7f808411c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:15.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.306+0000 7f8081d9b640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808410a780 0x7f808411c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:15.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.306+0000 7f8081d9b640 1 -- 172.21.15.16:0/1309382484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80841191d0 con 0x7f8084111ce0 2024-08-28T19:41:15.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.308+0000 7f8081d9b640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8084111ce0 0x7f8084118470 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f807401fd70 tx=0x7f807401d6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:15.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.309+0000 7f80737fe640 1 -- 172.21.15.16:0/1309382484 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807401ae40 con 0x7f8084111ce0 2024-08-28T19:41:15.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.309+0000 7f80737fe640 1 -- 172.21.15.16:0/1309382484 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80740218f0 con 0x7f8084111ce0 2024-08-28T19:41:15.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.309+0000 7f80737fe640 1 -- 172.21.15.16:0/1309382484 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074029980 con 0x7f8084111ce0 2024-08-28T19:41:15.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.309+0000 7f8088aa7640 1 -- 172.21.15.16:0/1309382484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80841acbd0 con 0x7f8084111ce0 2024-08-28T19:41:15.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.309+0000 7f8088aa7640 1 -- 172.21.15.16:0/1309382484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80841ad0c0 con 0x7f8084111ce0 2024-08-28T19:41:15.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.311+0000 7f8088aa7640 1 -- 172.21.15.16:0/1309382484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f808410ab90 con 0x7f8084111ce0 2024-08-28T19:41:15.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.314+0000 7f80737fe640 1 -- 172.21.15.16:0/1309382484 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8074030020 con 0x7f8084111ce0 2024-08-28T19:41:15.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.315+0000 7f80737fe640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f806c0b4e70 0x7f806c0b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:15.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.316+0000 7f80737fe640 1 -- 172.21.15.16:0/1309382484 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f80740f7150 con 0x7f8084111ce0 2024-08-28T19:41:15.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.316+0000 7f808259c640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f806c0b4e70 0x7f806c0b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:15.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.317+0000 7f808259c640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f806c0b4e70 0x7f806c0b7380 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f807802afa0 tx=0x7f8078005c90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:15.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.321+0000 7f80737fe640 1 -- 172.21.15.16:0/1309382484 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f80740bdb90 con 0x7f8084111ce0 2024-08-28T19:41:15.336 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:15.485 INFO:tasks.workunit.client.0.smithi016.stdout: 10 161472 22.82 MB/sec execute 291 sec latency 105.371 ms 2024-08-28T19:41:15.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:15 smithi016 ceph-mon[18556]: pgmap v973: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 595 op/s 2024-08-28T19:41:15.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.715+0000 7f8088aa7640 1 -- 172.21.15.16:0/1309382484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8084119770 con 0x7f8084111ce0 2024-08-28T19:41:15.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.716+0000 7f80737fe640 1 -- 172.21.15.16:0/1309382484 <== mon.1 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+11992 (secure 0 0 0) 0x7f80740bd490 con 0x7f8084111ce0 2024-08-28T19:41:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:41:15.719 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":101,"btime":"2024-08-28T19:40:25:411669+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":101,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:40:24.586397+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:41:15.719 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 101 2024-08-28T19:41:15.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.720+0000 7f8088aa7640 1 -- 172.21.15.16:0/1309382484 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f806c0b4e70 msgr2=0x7f806c0b7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:15.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.720+0000 7f8088aa7640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f806c0b4e70 0x7f806c0b7380 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f807802afa0 tx=0x7f8078005c90 comp rx=0 tx=0).stop 2024-08-28T19:41:15.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.720+0000 7f8088aa7640 1 -- 172.21.15.16:0/1309382484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8084111ce0 msgr2=0x7f8084118470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:15.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.720+0000 7f8088aa7640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8084111ce0 0x7f8084118470 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f807401fd70 tx=0x7f807401d6b0 comp rx=0 tx=0).stop 2024-08-28T19:41:15.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.720+0000 7f8088aa7640 1 -- 172.21.15.16:0/1309382484 shutdown_connections 2024-08-28T19:41:15.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.720+0000 7f8088aa7640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f806c0b4e70 0x7f806c0b7380 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:15.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.720+0000 7f8088aa7640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808411c9d0 0x7f80841189b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:15.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.720+0000 7f8088aa7640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8084111ce0 0x7f8084118470 secure :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f807401fd70 tx=0x7f807401d6b0 comp rx=0 tx=0).stop 2024-08-28T19:41:15.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.720+0000 7f8088aa7640 1 --2- 172.21.15.16:0/1309382484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808410a780 0x7f808411c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:15.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.720+0000 7f8088aa7640 1 -- 172.21.15.16:0/1309382484 >> 172.21.15.16:0/1309382484 conn(0x7f8084070c60 msgr2=0x7f808410b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:15.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.721+0000 7f8088aa7640 1 -- 172.21.15.16:0/1309382484 shutdown_connections 2024-08-28T19:41:15.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.721+0000 7f8088aa7640 1 -- 172.21.15.16:0/1309382484 wait complete. 2024-08-28T19:41:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:15 smithi028 ceph-mon[21769]: pgmap v973: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 595 op/s 2024-08-28T19:41:15.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:15 smithi115 ceph-mon[22184]: pgmap v973: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 595 op/s 2024-08-28T19:41:16.485 INFO:tasks.workunit.client.0.smithi016.stdout: 10 162067 22.85 MB/sec execute 292 sec latency 151.476 ms 2024-08-28T19:41:16.547 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-08-28T19:41:16.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-08-28T19:41:16.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:16 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1309382484' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:16.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:16 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/601580602' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:16 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1309382484' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:16 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/601580602' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:16.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:16 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1309382484' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:16.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:16 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/601580602' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:17.065 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:41:17.065 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":101,"btime":"2024-08-28T19:40:25:411669+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":101,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:40:24.586397+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:41:17.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.906+0000 7f5dbec22640 1 -- 172.21.15.16:0/3467473039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5db81047d0 msgr2=0x7f5db8104be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:17.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.906+0000 7f5dbec22640 1 --2- 172.21.15.16:0/3467473039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5db81047d0 0x7f5db8104be0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009900 tx=0x7f5da802a410 comp rx=0 tx=0).stop 2024-08-28T19:41:17.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.906+0000 7f5dbec22640 1 -- 172.21.15.16:0/3467473039 shutdown_connections 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.906+0000 7f5dbec22640 1 --2- 172.21.15.16:0/3467473039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5db810dc00 0x7f5db81100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.906+0000 7f5dbec22640 1 --2- 172.21.15.16:0/3467473039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5db8105120 0x7f5db810d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.906+0000 7f5dbec22640 1 --2- 172.21.15.16:0/3467473039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5db81047d0 0x7f5db8104be0 secure :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009900 tx=0x7f5da802a410 comp rx=0 tx=0).stop 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.906+0000 7f5dbec22640 1 -- 172.21.15.16:0/3467473039 >> 172.21.15.16:0/3467473039 conn(0x7f5db8079530 msgr2=0x7f5db8079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.907+0000 7f5dbec22640 1 -- 172.21.15.16:0/3467473039 shutdown_connections 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.907+0000 7f5dbec22640 1 -- 172.21.15.16:0/3467473039 wait complete. 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.908+0000 7f5dbec22640 1 Processor -- start 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.908+0000 7f5dbec22640 1 -- start start 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.908+0000 7f5dbec22640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5db8105120 0x7f5db8074d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.908+0000 7f5dbec22640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5db810dc00 0x7f5db806df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.908+0000 7f5dbec22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5db806e5a0 0x7f5db8073930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.908+0000 7f5dbec22640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5db8113890 con 0x7f5db810dc00 2024-08-28T19:41:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.908+0000 7f5dbec22640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5db8113710 con 0x7f5db806e5a0 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.908+0000 7f5dbec22640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5db8113a10 con 0x7f5db8105120 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.909+0000 7f5dbc9be640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5db8105120 0x7f5db8074d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.909+0000 7f5dbc9be640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5db8105120 0x7f5db8074d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:57916/0 (socket says 172.21.15.16:57916) 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.909+0000 7f5db7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5db810dc00 0x7f5db806df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.909+0000 7f5dbc9be640 1 -- 172.21.15.16:0/601580602 learned_addr learned my addr 172.21.15.16:0/601580602 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.909+0000 7f5dbd1bf640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5db806e5a0 0x7f5db8073930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.910+0000 7f5db7fff640 1 -- 172.21.15.16:0/601580602 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5db8105120 msgr2=0x7f5db8074d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.910+0000 7f5db7fff640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5db8105120 0x7f5db8074d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.910+0000 7f5db7fff640 1 -- 172.21.15.16:0/601580602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5db806e5a0 msgr2=0x7f5db8073930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.910+0000 7f5db7fff640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5db806e5a0 0x7f5db8073930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.910+0000 7f5db7fff640 1 -- 172.21.15.16:0/601580602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db806ed80 con 0x7f5db810dc00 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.910+0000 7f5db7fff640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5db810dc00 0x7f5db806df90 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f5db806a8c0 tx=0x7f5dac00ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.910+0000 7f5db5ffb640 1 -- 172.21.15.16:0/601580602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac020070 con 0x7f5db810dc00 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.911+0000 7f5dbec22640 1 -- 172.21.15.16:0/601580602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db8073f90 con 0x7f5db810dc00 2024-08-28T19:41:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.911+0000 7f5db5ffb640 1 -- 172.21.15.16:0/601580602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5dac00ccc0 con 0x7f5db810dc00 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.911+0000 7f5db5ffb640 1 -- 172.21.15.16:0/601580602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac01ba30 con 0x7f5db810dc00 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.911+0000 7f5dbec22640 1 -- 172.21.15.16:0/601580602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db80744d0 con 0x7f5db810dc00 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.913+0000 7f5dbec22640 1 -- 172.21.15.16:0/601580602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db806bc70 con 0x7f5db810dc00 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.916+0000 7f5db5ffb640 1 -- 172.21.15.16:0/601580602 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5dac023070 con 0x7f5db810dc00 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.918+0000 7f5db5ffb640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5da00b4dd0 0x7f5da00b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.918+0000 7f5dbc9be640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5da00b4dd0 0x7f5da00b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.918+0000 7f5db5ffb640 1 -- 172.21.15.16:0/601580602 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f5dac0e9890 con 0x7f5db810dc00 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.919+0000 7f5dbc9be640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5da00b4dd0 0x7f5da00b72e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f5da8005ab0 tx=0x7f5da8031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:15.922+0000 7f5db5ffb640 1 -- 172.21.15.16:0/601580602 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f5dac0afc00 con 0x7f5db810dc00 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.278+0000 7f5dbec22640 1 -- 172.21.15.16:0/601580602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5db8104be0 con 0x7f5db810dc00 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.279+0000 7f5db5ffb640 1 -- 172.21.15.16:0/601580602 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+11992 (secure 0 0 0) 0x7f5db8104be0 con 0x7f5db810dc00 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 101 2024-08-28T19:41:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.281+0000 7f5dbec22640 1 -- 172.21.15.16:0/601580602 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5da00b4dd0 msgr2=0x7f5da00b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.281+0000 7f5dbec22640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5da00b4dd0 0x7f5da00b72e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f5da8005ab0 tx=0x7f5da8031400 comp rx=0 tx=0).stop 2024-08-28T19:41:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.281+0000 7f5dbec22640 1 -- 172.21.15.16:0/601580602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5db810dc00 msgr2=0x7f5db806df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.281+0000 7f5dbec22640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5db810dc00 0x7f5db806df90 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f5db806a8c0 tx=0x7f5dac00ed90 comp rx=0 tx=0).stop 2024-08-28T19:41:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.282+0000 7f5dbec22640 1 -- 172.21.15.16:0/601580602 shutdown_connections 2024-08-28T19:41:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.282+0000 7f5dbec22640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5da00b4dd0 0x7f5da00b72e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.282+0000 7f5dbec22640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5db806e5a0 0x7f5db8073930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.282+0000 7f5dbec22640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5db810dc00 0x7f5db806df90 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.282+0000 7f5dbec22640 1 --2- 172.21.15.16:0/601580602 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5db8105120 0x7f5db8074d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.282+0000 7f5dbec22640 1 -- 172.21.15.16:0/601580602 >> 172.21.15.16:0/601580602 conn(0x7f5db8079530 msgr2=0x7f5db80ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.282+0000 7f5dbec22640 1 -- 172.21.15.16:0/601580602 shutdown_connections 2024-08-28T19:41:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:16.282+0000 7f5dbec22640 1 -- 172.21.15.16:0/601580602 wait complete. 2024-08-28T19:41:17.485 INFO:tasks.workunit.client.0.smithi016.stdout: 10 162774 22.87 MB/sec execute 293 sec latency 104.240 ms 2024-08-28T19:41:17.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:17 smithi016 ceph-mon[18556]: pgmap v974: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.0 MiB/s wr, 598 op/s 2024-08-28T19:41:17.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:41:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:17 smithi028 ceph-mon[21769]: pgmap v974: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.0 MiB/s wr, 598 op/s 2024-08-28T19:41:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:17 smithi115 ceph-mon[22184]: pgmap v974: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.0 MiB/s wr, 598 op/s 2024-08-28T19:41:18.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.446+0000 7f8feffff640 1 -- 172.21.15.16:0/2044458371 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8040770 con 0x7f8ff8111a80 2024-08-28T19:41:18.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.447+0000 7f8fff465640 1 -- 172.21.15.16:0/2044458371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8111a80 msgr2=0x7f8ff810a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:18.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.447+0000 7f8fff465640 1 --2- 172.21.15.16:0/2044458371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8111a80 0x7f8ff810a160 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f8fe801a910 tx=0x7f8fe803b420 comp rx=0 tx=0).stop 2024-08-28T19:41:18.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.449+0000 7f8fff465640 1 -- 172.21.15.16:0/2044458371 shutdown_connections 2024-08-28T19:41:18.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.449+0000 7f8fff465640 1 --2- 172.21.15.16:0/2044458371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff811c940 0x7f8ff811ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.449+0000 7f8fff465640 1 --2- 172.21.15.16:0/2044458371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff810a6a0 0x7f8ff810aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.449+0000 7f8fff465640 1 --2- 172.21.15.16:0/2044458371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8111a80 0x7f8ff810a160 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.449+0000 7f8fff465640 1 -- 172.21.15.16:0/2044458371 >> 172.21.15.16:0/2044458371 conn(0x7f8ff806e7b0 msgr2=0x7f8ff806ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:18.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.449+0000 7f8fff465640 1 -- 172.21.15.16:0/2044458371 shutdown_connections 2024-08-28T19:41:18.452 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:18.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.449+0000 7f8fff465640 1 -- 172.21.15.16:0/2044458371 wait complete. 2024-08-28T19:41:18.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.450+0000 7f8fff465640 1 Processor -- start 2024-08-28T19:41:18.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.450+0000 7f8fff465640 1 -- start start 2024-08-28T19:41:18.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.450+0000 7f8fff465640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff810a6a0 0x7f8ff811c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8fff465640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8111a80 0x7f8ff81184b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8fff465640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff811c940 0x7f8ff8118a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8fff465640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81149b0 con 0x7f8ff8111a80 2024-08-28T19:41:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8fff465640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8114830 con 0x7f8ff811c940 2024-08-28T19:41:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8fff465640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8114b30 con 0x7f8ff810a6a0 2024-08-28T19:41:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8ffda02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff811c940 0x7f8ff8118a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8ffda02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff811c940 0x7f8ff8118a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45042/0 (socket says 172.21.15.16:45042) 2024-08-28T19:41:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8ffda02640 1 -- 172.21.15.16:0/2697925092 learned_addr learned my addr 172.21.15.16:0/2697925092 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8ffd201640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff810a6a0 0x7f8ff811c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8ffda02640 1 -- 172.21.15.16:0/2697925092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff810a6a0 msgr2=0x7f8ff811c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8ffda02640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff810a6a0 0x7f8ff811c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8ffda02640 1 -- 172.21.15.16:0/2697925092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8111a80 msgr2=0x7f8ff81184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8ffda02640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8111a80 0x7f8ff81184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.451+0000 7f8ffda02640 1 -- 172.21.15.16:0/2697925092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff8119250 con 0x7f8ff811c940 2024-08-28T19:41:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.452+0000 7f8ffda02640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff811c940 0x7f8ff8118a30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000eaa0 tx=0x7f8ff000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.452+0000 7f8fee7fc640 1 -- 172.21.15.16:0/2697925092 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff000cdb0 con 0x7f8ff811c940 2024-08-28T19:41:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.452+0000 7f8fee7fc640 1 -- 172.21.15.16:0/2697925092 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ff0005020 con 0x7f8ff811c940 2024-08-28T19:41:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.452+0000 7f8fee7fc640 1 -- 172.21.15.16:0/2697925092 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00116d0 con 0x7f8ff811c940 2024-08-28T19:41:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.452+0000 7f8fff465640 1 -- 172.21.15.16:0/2697925092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff81aca90 con 0x7f8ff811c940 2024-08-28T19:41:18.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.452+0000 7f8fff465640 1 -- 172.21.15.16:0/2697925092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81acf50 con 0x7f8ff811c940 2024-08-28T19:41:18.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.457+0000 7f90059da740 1 Processor -- start 2024-08-28T19:41:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.459+0000 7f8fee7fc640 1 -- 172.21.15.16:0/2697925092 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8ff0005490 con 0x7f8ff811c940 2024-08-28T19:41:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.459+0000 7f90059da740 1 -- start start 2024-08-28T19:41:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.460+0000 7f90059da740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560439409840 0x5604397b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.460+0000 7f90059da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 0x5604397b13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.460+0000 7f90059da740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x5604397b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.460+0000 7f90059da740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560439725330 con 0x560439409840 2024-08-28T19:41:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.460+0000 7f90059da740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5604397148c0 con 0x5604393e0980 2024-08-28T19:41:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.460+0000 7f90059da740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560439718870 con 0x5604393e8b60 2024-08-28T19:41:18.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.460+0000 7f8ffd201640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 0x5604397b13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.460+0000 7f8ffd201640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 0x5604397b13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45058/0 (socket says 172.21.15.16:45058) 2024-08-28T19:41:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.460+0000 7f8ffd201640 1 -- 172.21.15.16:0/3390533488 learned_addr learned my addr 172.21.15.16:0/3390533488 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.461+0000 7f8fee7fc640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe40b4d90 0x7f8fe40b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.461+0000 7f8fee7fc640 1 -- 172.21.15.16:0/2697925092 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f8ff00e5a60 con 0x7f8ff811c940 2024-08-28T19:41:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.461+0000 7f8ffd201640 1 -- 172.21.15.16:0/3390533488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 msgr2=0x5604397b18f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:41:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.461+0000 7f8ffd201640 1 --2- 172.21.15.16:0/3390533488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x5604397b18f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.461+0000 7f8ffd201640 1 -- 172.21.15.16:0/3390533488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560439409840 msgr2=0x5604397b0e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:41:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.461+0000 7f8ffd201640 1 --2- 172.21.15.16:0/3390533488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560439409840 0x5604397b0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.461+0000 7f8ffd201640 1 -- 172.21.15.16:0/3390533488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604396ccdf0 con 0x5604393e0980 2024-08-28T19:41:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.462+0000 7f8ffda02640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe40b4d90 0x7f8fe40b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.462+0000 7f8ffd201640 1 --2- 172.21.15.16:0/3390533488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 0x5604397b13b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8fe803b940 tx=0x7f8fe80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dbf8c90be4d457e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.462+0000 7f8fe37fe640 1 -- 172.21.15.16:0/3390533488 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8046030 con 0x5604393e0980 2024-08-28T19:41:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.463+0000 7f8fe37fe640 1 -- 172.21.15.16:0/3390533488 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe8004030 con 0x5604393e0980 2024-08-28T19:41:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.463+0000 7f8fe37fe640 1 -- 172.21.15.16:0/3390533488 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe80035f0 con 0x5604393e0980 2024-08-28T19:41:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.464+0000 7f90059da740 1 -- 172.21.15.16:0/3390533488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 msgr2=0x5604397b13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.464+0000 7f8ffda02640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe40b4d90 0x7f8fe40b72a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00a7da0 tx=0x7f8ff0005780 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.464+0000 7f90059da740 1 --2- 172.21.15.16:0/3390533488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 0x5604397b13b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8fe803b940 tx=0x7f8fe80046c0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.464+0000 7f90059da740 1 -- 172.21.15.16:0/3390533488 shutdown_connections 2024-08-28T19:41:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.464+0000 7f90059da740 1 --2- 172.21.15.16:0/3390533488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x5604397b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.464+0000 7f90059da740 1 --2- 172.21.15.16:0/3390533488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 0x5604397b13b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.464+0000 7f90059da740 1 --2- 172.21.15.16:0/3390533488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560439409840 0x5604397b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.464+0000 7f90059da740 1 -- 172.21.15.16:0/3390533488 >> 172.21.15.16:0/3390533488 conn(0x5604393fd990 msgr2=0x5604397aed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.465+0000 7f90059da740 1 -- 172.21.15.16:0/3390533488 shutdown_connections 2024-08-28T19:41:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.465+0000 7f90059da740 1 -- 172.21.15.16:0/3390533488 wait complete. 2024-08-28T19:41:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.465+0000 7f90059da740 1 -- start start 2024-08-28T19:41:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.466+0000 7f90059da740 1 Processor -- start 2024-08-28T19:41:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.466+0000 7f90059da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 0x5604397bc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.466+0000 7f90059da740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e8b60 0x5604397bcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.466+0000 7f90059da740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560439409840 0x5604397c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.466+0000 7f90059da740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5604395777a0 con 0x5604393e8b60 2024-08-28T19:41:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.466+0000 7f90059da740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56043969b240 con 0x5604393e0980 2024-08-28T19:41:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.466+0000 7f90059da740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5604397bf690 con 0x560439409840 2024-08-28T19:41:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.466+0000 7f90059da740 10 client.0 _ll_register_callbacks cb 0x560439421090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.467+0000 7f8ffca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 0x5604397bc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.467+0000 7f8ffca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 0x5604397bc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45070/0 (socket says 172.21.15.16:45070) 2024-08-28T19:41:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.467+0000 7f8ffca00640 1 -- 172.21.15.16:0/2318547672 learned_addr learned my addr 172.21.15.16:0/2318547672 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.467+0000 7f8ffd201640 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e8b60 0x5604397bcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.467+0000 7f8ffca00640 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560439409840 0x5604397c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.467+0000 7f8ffd201640 1 -- 172.21.15.16:0/2318547672 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560439409840 msgr2=0x5604397c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.467+0000 7f8ffd201640 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560439409840 0x5604397c89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.467+0000 7f8ffd201640 1 -- 172.21.15.16:0/2318547672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 msgr2=0x5604397bc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.467+0000 7f8ffd201640 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 0x5604397bc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.467+0000 7f8ffd201640 1 -- 172.21.15.16:0/2318547672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604396d12b0 con 0x5604393e8b60 2024-08-28T19:41:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.468+0000 7f8ffd201640 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e8b60 0x5604397bcb30 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f8fe803ba10 tx=0x7f8fe8005010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac712951c0f436e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.469+0000 7f8fe0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.469+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/2318547672 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8046040 con 0x5604393e8b60 2024-08-28T19:41:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.469+0000 7f90059da740 1 -- 172.21.15.16:0/2318547672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fe8004030 con 0x5604393e8b60 2024-08-28T19:41:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.469+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/2318547672 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f8fe80041e0 con 0x5604393e8b60 2024-08-28T19:41:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.470+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/2318547672 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f8fe80043e0 con 0x5604393e8b60 2024-08-28T19:41:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.470+0000 7f90059da740 10 client.20094 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.471+0000 7f90059da740 1 -- 172.21.15.16:0/2318547672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x560439483b90 con 0x5604393e8b60 2024-08-28T19:41:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.472+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/2318547672 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe805e070 con 0x5604393e8b60 2024-08-28T19:41:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.472+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/2318547672 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f8fe8003c90 con 0x5604393e8b60 2024-08-28T19:41:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.472+0000 7f90059da740 10 client.20094 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.472+0000 7f90059da740 10 client.20094 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:41:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.472+0000 7f90059da740 20 client.20094 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.472+0000 7f90059da740 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5604393c81c0 0x5604397ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.472+0000 7f90059da740 4 client.20094 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:41:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.472+0000 7f90059da740 1 -- 172.21.15.16:0/2318547672 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8fe8004030 con 0x5604393c81c0 2024-08-28T19:41:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.472+0000 7f8ffca00640 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5604393c81c0 0x5604397ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.473+0000 7f8fe0ff9640 10 client.20094 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:18.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.473+0000 7f8ffca00640 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5604393c81c0 0x5604397ce530 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57e1e80b810c5691 server_cookie=fd14bc54446183a9 in_seq=0 out_seq=0 2024-08-28T19:41:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.475+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/2318547672 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f8fe8004030 con 0x5604393c81c0 2024-08-28T19:41:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.475+0000 7f8fe0ff9640 10 client.20094 handle_command_reply: tid=0 2024-08-28T19:41:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.475+0000 7f90059da740 1 client.20094 shutdown 2024-08-28T19:41:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.475+0000 7f90059da740 1 -- 172.21.15.16:0/2318547672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e8b60 msgr2=0x5604397bcb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.475+0000 7f90059da740 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e8b60 0x5604397bcb30 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f8fe803ba10 tx=0x7f8fe8005010 comp rx=0 tx=0).stop 2024-08-28T19:41:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.475+0000 7f90059da740 1 -- 172.21.15.16:0/2318547672 shutdown_connections 2024-08-28T19:41:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.476+0000 7f90059da740 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5604393c81c0 0x5604397ce530 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.476+0000 7f90059da740 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560439409840 0x5604397c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.476+0000 7f8fe0ff9640 0 client.20094 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.476+0000 7f90059da740 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e8b60 0x5604397bcb30 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.476+0000 7f90059da740 1 --2- 172.21.15.16:0/2318547672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393e0980 0x5604397bc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.476+0000 7f90059da740 1 -- 172.21.15.16:0/2318547672 >> 172.21.15.16:0/2318547672 conn(0x5604393fd990 msgr2=0x5604397b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.476+0000 7f90059da740 1 -- 172.21.15.16:0/2318547672 shutdown_connections 2024-08-28T19:41:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.476+0000 7f90059da740 1 -- 172.21.15.16:0/2318547672 wait complete. 2024-08-28T19:41:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.476+0000 7f90059da740 20 client.20094 trim_cache size 0 max 16384 2024-08-28T19:41:18.487 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163422 22.94 MB/sec execute 294 sec latency 111.499 ms 2024-08-28T19:41:18.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.507+0000 7f90059da740 1 Processor -- start 2024-08-28T19:41:18.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.507+0000 7f90059da740 1 -- start start 2024-08-28T19:41:18.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.507+0000 7f90059da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 0x5604397b1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.507+0000 7f90059da740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 0x5604397ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.507+0000 7f90059da740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x5604397aec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.507+0000 7f90059da740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81b71c0 con 0x5604393e0980 2024-08-28T19:41:18.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.507+0000 7f90059da740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81b8e80 con 0x5604393c81c0 2024-08-28T19:41:18.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.507+0000 7f90059da740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81b7920 con 0x5604393e8b60 2024-08-28T19:41:18.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.508+0000 7f8ffca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 0x5604397b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.508+0000 7f8ffca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 0x5604397b1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45076/0 (socket says 172.21.15.16:45076) 2024-08-28T19:41:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.508+0000 7f8ffca00640 1 -- 172.21.15.16:0/938164665 learned_addr learned my addr 172.21.15.16:0/938164665 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.508+0000 7f8ffd201640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 0x5604397ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.508+0000 7f8ffca00640 1 --2- 172.21.15.16:0/938164665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x5604397aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.508+0000 7f8ffd201640 1 -- 172.21.15.16:0/938164665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 msgr2=0x5604397aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.508+0000 7f8ffd201640 1 --2- 172.21.15.16:0/938164665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x5604397aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.508+0000 7f8ffd201640 1 -- 172.21.15.16:0/938164665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 msgr2=0x5604397b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.508+0000 7f8ffd201640 1 --2- 172.21.15.16:0/938164665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 0x5604397b1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.508+0000 7f8ffd201640 1 -- 172.21.15.16:0/938164665 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604396e4d90 con 0x5604393e0980 2024-08-28T19:41:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.508+0000 7f8ffca00640 1 --2- 172.21.15.16:0/938164665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x5604397aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:41:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.509+0000 7f8ffd201640 1 --2- 172.21.15.16:0/938164665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 0x5604397ae6c0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f8fe803b940 tx=0x7f8fe803b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c8f1e6c9e135ed server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:18.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.509+0000 7f8fe2ffd640 1 -- 172.21.15.16:0/938164665 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8046070 con 0x5604393e0980 2024-08-28T19:41:18.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.509+0000 7f8fe2ffd640 1 -- 172.21.15.16:0/938164665 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe805e040 con 0x5604393e0980 2024-08-28T19:41:18.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.509+0000 7f8fe2ffd640 1 -- 172.21.15.16:0/938164665 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8063450 con 0x5604393e0980 2024-08-28T19:41:18.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.511+0000 7f90059da740 1 -- 172.21.15.16:0/938164665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 msgr2=0x5604397ae6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:18.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.511+0000 7f90059da740 1 --2- 172.21.15.16:0/938164665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 0x5604397ae6c0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f8fe803b940 tx=0x7f8fe803b420 comp rx=0 tx=0).stop 2024-08-28T19:41:18.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.511+0000 7f90059da740 1 -- 172.21.15.16:0/938164665 shutdown_connections 2024-08-28T19:41:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.511+0000 7f90059da740 1 --2- 172.21.15.16:0/938164665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x5604397aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.511+0000 7f90059da740 1 --2- 172.21.15.16:0/938164665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 0x5604397ae6c0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.511+0000 7f90059da740 1 --2- 172.21.15.16:0/938164665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 0x5604397b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.511+0000 7f90059da740 1 -- 172.21.15.16:0/938164665 >> 172.21.15.16:0/938164665 conn(0x5604393fd990 msgr2=0x5604397bc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.511+0000 7f90059da740 1 -- 172.21.15.16:0/938164665 shutdown_connections 2024-08-28T19:41:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.511+0000 7f90059da740 1 -- 172.21.15.16:0/938164665 wait complete. 2024-08-28T19:41:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.512+0000 7f90059da740 1 -- start start 2024-08-28T19:41:18.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.512+0000 7f90059da740 1 Processor -- start 2024-08-28T19:41:18.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.513+0000 7f90059da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 0x5604397d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.513+0000 7f90059da740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 0x5604397d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.513+0000 7f8ffca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 0x5604397d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.513+0000 7f8ffca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 0x5604397d0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45088/0 (socket says 172.21.15.16:45088) 2024-08-28T19:41:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.513+0000 7f8ffca00640 1 -- 172.21.15.16:0/410800312 learned_addr learned my addr 172.21.15.16:0/410800312 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.513+0000 7f90059da740 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x560439831e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.513+0000 7f90059da740 1 -- 172.21.15.16:0/410800312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81b86f0 con 0x5604393e0980 2024-08-28T19:41:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.513+0000 7f8ffd201640 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 0x5604397d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.513+0000 7f90059da740 1 -- 172.21.15.16:0/410800312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81b7ff0 con 0x5604393c81c0 2024-08-28T19:41:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.514+0000 7f90059da740 1 -- 172.21.15.16:0/410800312 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81b51c0 con 0x5604393e8b60 2024-08-28T19:41:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.514+0000 7f90059da740 10 client.0 _ll_register_callbacks cb 0x7f8fe801a590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.514+0000 7f8ffca00640 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x560439831e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.514+0000 7f8ffd201640 1 -- 172.21.15.16:0/410800312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 msgr2=0x560439831e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.514+0000 7f8ffd201640 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x560439831e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.514+0000 7f8ffd201640 1 -- 172.21.15.16:0/410800312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 msgr2=0x5604397d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.514+0000 7f8ffd201640 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 0x5604397d0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.514+0000 7f8ffd201640 1 -- 172.21.15.16:0/410800312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560439832500 con 0x5604393e0980 2024-08-28T19:41:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.515+0000 7f8ffd201640 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 0x5604397d0700 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x5604397b3bb0 tx=0x7f8fe8040ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfd6f8527549ffd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.515+0000 7f8fe0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.515+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/410800312 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8046070 con 0x5604393e0980 2024-08-28T19:41:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.515+0000 7f90059da740 1 -- 172.21.15.16:0/410800312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fe805e040 con 0x5604393e0980 2024-08-28T19:41:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.515+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/410800312 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f8fe8004940 con 0x5604393e0980 2024-08-28T19:41:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.516+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/410800312 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f8fe8004b80 con 0x5604393e0980 2024-08-28T19:41:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.516+0000 7f90059da740 10 client.20100 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.516+0000 7f90059da740 1 -- 172.21.15.16:0/410800312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x560439839820 con 0x5604393e0980 2024-08-28T19:41:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.517+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/410800312 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe8040990 con 0x5604393e0980 2024-08-28T19:41:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.517+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/410800312 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f8fe804d440 con 0x5604393e0980 2024-08-28T19:41:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.517+0000 7f90059da740 10 client.20100 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.517+0000 7f90059da740 10 client.20100 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:41:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.517+0000 7f90059da740 20 client.20100 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.517+0000 7f90059da740 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560439409840 0x56043983be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.517+0000 7f90059da740 4 client.20100 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-08-28T19:41:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.517+0000 7f90059da740 1 -- 172.21.15.16:0/410800312 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x7f8fe805e040 con 0x560439409840 2024-08-28T19:41:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.517+0000 7f8ffca00640 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560439409840 0x56043983be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.518+0000 7f8fe0ff9640 10 client.20100 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:18.518+0000 7f8ffca00640 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560439409840 0x56043983be60 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=325ee7761fe10113 server_cookie=d8b0d6ee211d9b34 in_seq=0 out_seq=0 2024-08-28T19:41:19.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.042+0000 7fab1e19d640 1 -- 172.21.15.16:0/779653441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab18102640 msgr2=0x7fab18102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.042+0000 7fab1e19d640 1 --2- 172.21.15.16:0/779653441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab18102640 0x7fab18102a50 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fab14009900 tx=0x7fab1402a410 comp rx=0 tx=0).stop 2024-08-28T19:41:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.042+0000 7fab1e19d640 1 -- 172.21.15.16:0/779653441 shutdown_connections 2024-08-28T19:41:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.042+0000 7fab1e19d640 1 --2- 172.21.15.16:0/779653441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab181088d0 0x7fab1810f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.042+0000 7fab1e19d640 1 --2- 172.21.15.16:0/779653441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab18107f80 0x7fab18108390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.042+0000 7fab1e19d640 1 --2- 172.21.15.16:0/779653441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab18102640 0x7fab18102a50 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.042+0000 7fab1e19d640 1 -- 172.21.15.16:0/779653441 >> 172.21.15.16:0/779653441 conn(0x7fab180fe0b0 msgr2=0x7fab18100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.042+0000 7fab1e19d640 1 -- 172.21.15.16:0/779653441 shutdown_connections 2024-08-28T19:41:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.042+0000 7fab1e19d640 1 -- 172.21.15.16:0/779653441 wait complete. 2024-08-28T19:41:19.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1e19d640 1 Processor -- start 2024-08-28T19:41:19.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1e19d640 1 -- start start 2024-08-28T19:41:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1e19d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab18102640 0x7fab1819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1e19d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab18107f80 0x7fab1819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1e19d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab181088d0 0x7fab1819f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1e19d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab181125a0 con 0x7fab181088d0 2024-08-28T19:41:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1e19d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab18112420 con 0x7fab18102640 2024-08-28T19:41:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1e19d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fab18112720 con 0x7fab18107f80 2024-08-28T19:41:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1d19b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab18102640 0x7fab1819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1d19b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab18102640 0x7fab1819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45104/0 (socket says 172.21.15.16:45104) 2024-08-28T19:41:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1d99c640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab181088d0 0x7fab1819f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.043+0000 7fab1d19b640 1 -- 172.21.15.16:0/3659007722 learned_addr learned my addr 172.21.15.16:0/3659007722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.044+0000 7fab1c99a640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab18107f80 0x7fab1819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.044+0000 7fab1d99c640 1 -- 172.21.15.16:0/3659007722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab18107f80 msgr2=0x7fab1819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.044+0000 7fab1d99c640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab18107f80 0x7fab1819a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.044+0000 7fab1d99c640 1 -- 172.21.15.16:0/3659007722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab18102640 msgr2=0x7fab1819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.044+0000 7fab1d99c640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab18102640 0x7fab1819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.044+0000 7fab1d99c640 1 -- 172.21.15.16:0/3659007722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab181a00d0 con 0x7fab181088d0 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.044+0000 7fab1d19b640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab18102640 0x7fab1819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.044+0000 7fab1d99c640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab181088d0 0x7fab1819f9a0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fab10009fd0 tx=0x7fab1000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.044+0000 7fab0e7fc640 1 -- 172.21.15.16:0/3659007722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1000ce20 con 0x7fab181088d0 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.044+0000 7fab0e7fc640 1 -- 172.21.15.16:0/3659007722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab10005020 con 0x7fab181088d0 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.044+0000 7fab1e19d640 1 -- 172.21.15.16:0/3659007722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab181a03c0 con 0x7fab181088d0 2024-08-28T19:41:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.045+0000 7fab1e19d640 1 -- 172.21.15.16:0/3659007722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab180fe4c0 con 0x7fab181088d0 2024-08-28T19:41:19.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.045+0000 7fab0e7fc640 1 -- 172.21.15.16:0/3659007722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab10010690 con 0x7fab181088d0 2024-08-28T19:41:19.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab25974740 1 Processor -- start 2024-08-28T19:41:19.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab0e7fc640 1 -- 172.21.15.16:0/3659007722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fab10010830 con 0x7fab181088d0 2024-08-28T19:41:19.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab25974740 1 -- start start 2024-08-28T19:41:19.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab25974740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59a9070 0x5591d5d4e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab25974740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 0x5591d5d4e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab25974740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d4ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab25974740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591d5cc5680 con 0x5591d5988390 2024-08-28T19:41:19.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab25974740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591d5cb4c10 con 0x5591d59801b0 2024-08-28T19:41:19.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab25974740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5591d5cb8bc0 con 0x5591d59a9070 2024-08-28T19:41:19.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab1d19b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 0x5591d5d4e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab1c99a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d4ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab1d19b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 0x5591d5d4e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45120/0 (socket says 172.21.15.16:45120) 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab1d19b640 1 -- 172.21.15.16:0/1565800550 learned_addr learned my addr 172.21.15.16:0/1565800550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.048+0000 7fab1c99a640 1 --2- 172.21.15.16:0/1565800550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59a9070 0x5591d5d4e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab0e7fc640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fab040b4d20 0x7fab040b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab1d19b640 1 -- 172.21.15.16:0/1565800550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59a9070 msgr2=0x5591d5d4e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab1d19b640 1 --2- 172.21.15.16:0/1565800550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59a9070 0x5591d5d4e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab1d19b640 1 -- 172.21.15.16:0/1565800550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 msgr2=0x5591d5d4ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab1d99c640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fab040b4d20 0x7fab040b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab1d19b640 1 --2- 172.21.15.16:0/1565800550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d4ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab1d19b640 1 -- 172.21.15.16:0/1565800550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d5a14650 con 0x5591d59801b0 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab1c99a640 1 --2- 172.21.15.16:0/1565800550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d4ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab0e7fc640 1 -- 172.21.15.16:0/3659007722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7fab100e5f30 con 0x7fab181088d0 2024-08-28T19:41:19.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab1d19b640 1 --2- 172.21.15.16:0/1565800550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 0x5591d5d4e9f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fab14000c00 tx=0x7fab14005ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87244848a17c549f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:19.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab1d99c640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fab040b4d20 0x7fab040b7230 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fab100e3360 tx=0x7fab10078450 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:19.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab037fe640 1 -- 172.21.15.16:0/1565800550 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14035040 con 0x5591d59801b0 2024-08-28T19:41:19.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab037fe640 1 -- 172.21.15.16:0/1565800550 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab14031820 con 0x5591d59801b0 2024-08-28T19:41:19.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.049+0000 7fab037fe640 1 -- 172.21.15.16:0/1565800550 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14004a00 con 0x5591d59801b0 2024-08-28T19:41:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.050+0000 7fab25974740 1 -- 172.21.15.16:0/1565800550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 msgr2=0x5591d5d4e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.050+0000 7fab25974740 1 --2- 172.21.15.16:0/1565800550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 0x5591d5d4e9f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fab14000c00 tx=0x7fab14005ed0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.050+0000 7fab25974740 1 -- 172.21.15.16:0/1565800550 shutdown_connections 2024-08-28T19:41:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.050+0000 7fab25974740 1 --2- 172.21.15.16:0/1565800550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d4ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.050+0000 7fab25974740 1 --2- 172.21.15.16:0/1565800550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 0x5591d5d4e9f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.050+0000 7fab25974740 1 --2- 172.21.15.16:0/1565800550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59a9070 0x5591d5d4e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.050+0000 7fab25974740 1 -- 172.21.15.16:0/1565800550 >> 172.21.15.16:0/1565800550 conn(0x5591d599d460 msgr2=0x5591d5d4c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.051+0000 7fab25974740 1 -- 172.21.15.16:0/1565800550 shutdown_connections 2024-08-28T19:41:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.051+0000 7fab25974740 1 -- 172.21.15.16:0/1565800550 wait complete. 2024-08-28T19:41:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.051+0000 7fab25974740 1 -- start start 2024-08-28T19:41:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.051+0000 7fab25974740 1 Processor -- start 2024-08-28T19:41:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.051+0000 7fab25974740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 0x5591d5d59c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.051+0000 7fab25974740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d5a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab25974740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59a9070 0x5591d5d65fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab25974740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591d5b17700 con 0x5591d5988390 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab25974740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591d5c3b750 con 0x5591d59801b0 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab25974740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5591d5d5cc30 con 0x5591d59a9070 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab25974740 10 client.0 _ll_register_callbacks cb 0x5591d5c6ae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab1d19b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d5a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab1d19b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d5a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49022/0 (socket says 172.21.15.16:49022) 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab1c99a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 0x5591d5d59c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab1d19b640 1 -- 172.21.15.16:0/1641218021 learned_addr learned my addr 172.21.15.16:0/1641218021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab1c99a640 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59a9070 0x5591d5d65fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab1d19b640 1 -- 172.21.15.16:0/1641218021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59a9070 msgr2=0x5591d5d65fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab1d19b640 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59a9070 0x5591d5d65fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab1d19b640 1 -- 172.21.15.16:0/1641218021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 msgr2=0x5591d5d59c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab1d19b640 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 0x5591d5d59c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab1d19b640 1 -- 172.21.15.16:0/1641218021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591d5c74900 con 0x5591d5988390 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab1d19b640 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d5a1d0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fab140042a0 tx=0x7fab1402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99485fdc57fbf403 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab00ff9640 1 -- 172.21.15.16:0/1641218021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1404e030 con 0x5591d5988390 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.052+0000 7fab00ff9640 1 -- 172.21.15.16:0/1641218021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7fab14031620 con 0x5591d5988390 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab25974740 1 -- 172.21.15.16:0/1641218021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab14031820 con 0x5591d5988390 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab00ff9640 1 -- 172.21.15.16:0/1641218021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fab14058050 con 0x5591d5988390 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab25974740 10 client.20112 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab25974740 1 -- 172.21.15.16:0/1641218021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5591d5d68aa0 con 0x5591d5988390 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab00ff9640 1 -- 172.21.15.16:0/1641218021 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab14031820 con 0x5591d5988390 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab00ff9640 1 -- 172.21.15.16:0/1641218021 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fab14004aa0 con 0x5591d5988390 2024-08-28T19:41:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab25974740 10 client.20112 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab25974740 10 client.20112 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:41:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab25974740 20 client.20112 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab25974740 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5591d59679f0 0x5591d5d6be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab25974740 4 client.20112 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:41:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.053+0000 7fab25974740 1 -- 172.21.15.16:0/1641218021 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591d5cc75d0 con 0x5591d59679f0 2024-08-28T19:41:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.054+0000 7fab1c99a640 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5591d59679f0 0x5591d5d6be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.054+0000 7fab1c99a640 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5591d59679f0 0x5591d5d6be00 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ea5ebbb1b971d859 server_cookie=a08e9b16e29809e2 in_seq=0 out_seq=0 2024-08-28T19:41:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.054+0000 7fab00ff9640 10 client.20112 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:41:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab00ff9640 1 -- 172.21.15.16:0/1641218021 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5591d5cc75d0 con 0x5591d59679f0 2024-08-28T19:41:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab00ff9640 10 client.20112 handle_command_reply: tid=0 2024-08-28T19:41:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab25974740 1 client.20112 shutdown 2024-08-28T19:41:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab25974740 1 -- 172.21.15.16:0/1641218021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 msgr2=0x5591d5d5a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab25974740 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d5a1d0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fab140042a0 tx=0x7fab1402a410 comp rx=0 tx=0).stop 2024-08-28T19:41:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab25974740 1 -- 172.21.15.16:0/1641218021 shutdown_connections 2024-08-28T19:41:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab25974740 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5591d59679f0 0x5591d5d6be00 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab25974740 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59a9070 0x5591d5d65fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab00ff9640 0 client.20112 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:41:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab25974740 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d5a1d0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab25974740 1 --2- 172.21.15.16:0/1641218021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59801b0 0x5591d5d59c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.055+0000 7fab25974740 1 -- 172.21.15.16:0/1641218021 >> 172.21.15.16:0/1641218021 conn(0x5591d599d460 msgr2=0x5591d5d51640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.056+0000 7fab25974740 1 -- 172.21.15.16:0/1641218021 shutdown_connections 2024-08-28T19:41:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.056+0000 7fab25974740 1 -- 172.21.15.16:0/1641218021 wait complete. 2024-08-28T19:41:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.056+0000 7fab25974740 20 client.20112 trim_cache size 0 max 16384 2024-08-28T19:41:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab25974740 1 Processor -- start 2024-08-28T19:41:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab25974740 1 -- start start 2024-08-28T19:41:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab25974740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59679f0 0x5591d5d4c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab25974740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d59801b0 0x5591d5d65f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab25974740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d5988390 0x5591d5d56e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab25974740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b2d20 con 0x5591d59801b0 2024-08-28T19:41:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab25974740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b49e0 con 0x5591d5988390 2024-08-28T19:41:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab25974740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b3480 con 0x5591d59679f0 2024-08-28T19:41:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab1d19b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d59801b0 0x5591d5d65f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab1d19b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d59801b0 0x5591d5d65f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49038/0 (socket says 172.21.15.16:49038) 2024-08-28T19:41:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab1d19b640 1 -- 172.21.15.16:0/1246606116 learned_addr learned my addr 172.21.15.16:0/1246606116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab1c99a640 1 --2- 172.21.15.16:0/1246606116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d5988390 0x5591d5d56e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab1c99a640 1 --2- 172.21.15.16:0/1246606116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59679f0 0x5591d5d4c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab1d19b640 1 -- 172.21.15.16:0/1246606116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59679f0 msgr2=0x5591d5d4c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab1d19b640 1 --2- 172.21.15.16:0/1246606116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59679f0 0x5591d5d4c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab1d19b640 1 -- 172.21.15.16:0/1246606116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d5988390 msgr2=0x5591d5d56e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab1d19b640 1 --2- 172.21.15.16:0/1246606116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d5988390 0x5591d5d56e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.069+0000 7fab1d19b640 1 -- 172.21.15.16:0/1246606116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d5d51310 con 0x5591d59801b0 2024-08-28T19:41:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.070+0000 7fab1d19b640 1 --2- 172.21.15.16:0/1246606116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d59801b0 0x5591d5d65f20 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fab14004270 tx=0x7fab14005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9ce81970227fa6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.070+0000 7fab02ffd640 1 -- 172.21.15.16:0/1246606116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1404e040 con 0x5591d59801b0 2024-08-28T19:41:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.070+0000 7fab02ffd640 1 -- 172.21.15.16:0/1246606116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab14009520 con 0x5591d59801b0 2024-08-28T19:41:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.070+0000 7fab02ffd640 1 -- 172.21.15.16:0/1246606116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14035040 con 0x5591d59801b0 2024-08-28T19:41:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.071+0000 7fab25974740 1 -- 172.21.15.16:0/1246606116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d59801b0 msgr2=0x5591d5d65f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.071+0000 7fab25974740 1 --2- 172.21.15.16:0/1246606116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d59801b0 0x5591d5d65f20 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fab14004270 tx=0x7fab14005fd0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.071+0000 7fab25974740 1 -- 172.21.15.16:0/1246606116 shutdown_connections 2024-08-28T19:41:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.071+0000 7fab25974740 1 --2- 172.21.15.16:0/1246606116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d5988390 0x5591d5d56e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.071+0000 7fab25974740 1 --2- 172.21.15.16:0/1246606116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d59801b0 0x5591d5d65f20 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.071+0000 7fab25974740 1 --2- 172.21.15.16:0/1246606116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59679f0 0x5591d5d4c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.071+0000 7fab25974740 1 -- 172.21.15.16:0/1246606116 >> 172.21.15.16:0/1246606116 conn(0x5591d599d460 msgr2=0x5591d5d4bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.071+0000 7fab25974740 1 -- 172.21.15.16:0/1246606116 shutdown_connections 2024-08-28T19:41:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.071+0000 7fab25974740 1 -- 172.21.15.16:0/1246606116 wait complete. 2024-08-28T19:41:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.071+0000 7fab25974740 1 -- start start 2024-08-28T19:41:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.071+0000 7fab25974740 1 Processor -- start 2024-08-28T19:41:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab25974740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59679f0 0x5591d5d6b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab25974740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59801b0 0x5591d5d6b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab25974740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d6ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab25974740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b4250 con 0x5591d5988390 2024-08-28T19:41:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab25974740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b3b50 con 0x5591d59679f0 2024-08-28T19:41:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab25974740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b0d20 con 0x5591d59801b0 2024-08-28T19:41:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab25974740 10 client.0 _ll_register_callbacks cb 0x7fab14040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab1c99a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d6ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab1d19b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59801b0 0x5591d5d6b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab1c99a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d6ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49046/0 (socket says 172.21.15.16:49046) 2024-08-28T19:41:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab1c99a640 1 -- 172.21.15.16:0/2079921368 learned_addr learned my addr 172.21.15.16:0/2079921368 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab1c99a640 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59679f0 0x5591d5d6b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab1c99a640 1 -- 172.21.15.16:0/2079921368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59801b0 msgr2=0x5591d5d6b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab1c99a640 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59801b0 0x5591d5d6b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab1c99a640 1 -- 172.21.15.16:0/2079921368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59679f0 msgr2=0x5591d5d6b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab1c99a640 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59679f0 0x5591d5d6b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab1c99a640 1 -- 172.21.15.16:0/2079921368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591d5dd5770 con 0x5591d5988390 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab1c99a640 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d6ba80 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fab18108740 tx=0x7fab14005470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e96e54d6efb692 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.072+0000 7fab00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab00ff9640 1 -- 172.21.15.16:0/2079921368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08004040 con 0x5591d5988390 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab00ff9640 1 -- 172.21.15.16:0/2079921368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7fab0801a040 con 0x5591d5988390 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab25974740 1 -- 172.21.15.16:0/2079921368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab14009520 con 0x5591d5988390 2024-08-28T19:41:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab00ff9640 1 -- 172.21.15.16:0/2079921368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fab080041e0 con 0x5591d5988390 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab25974740 10 client.20118 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab25974740 1 -- 172.21.15.16:0/2079921368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5591d5dd6a20 con 0x5591d5988390 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab00ff9640 1 -- 172.21.15.16:0/2079921368 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab14009520 con 0x5591d5988390 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab00ff9640 1 -- 172.21.15.16:0/2079921368 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fab080048c0 con 0x5591d5988390 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab25974740 10 client.20118 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab25974740 10 client.20118 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab25974740 20 client.20118 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab25974740 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5591d59a9070 0x5591d5dd9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab25974740 4 client.20118 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.073+0000 7fab25974740 1 -- 172.21.15.16:0/2079921368 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5591d5cc3070 con 0x5591d59a9070 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.074+0000 7fab1d19b640 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5591d59a9070 0x5591d5dd9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.074+0000 7fab1d19b640 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5591d59a9070 0x5591d5dd9270 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c73d5964fc7de30e server_cookie=b25a2077acad4d2d in_seq=0 out_seq=0 2024-08-28T19:41:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.074+0000 7fab00ff9640 10 client.20118 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:41:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.074+0000 7fab00ff9640 1 -- 172.21.15.16:0/2079921368 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5591d5cc3070 con 0x5591d59a9070 2024-08-28T19:41:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.074+0000 7fab00ff9640 10 client.20118 handle_command_reply: tid=0 2024-08-28T19:41:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.074+0000 7fab25974740 1 client.20118 shutdown 2024-08-28T19:41:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab25974740 1 -- 172.21.15.16:0/2079921368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 msgr2=0x5591d5d6ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab25974740 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d6ba80 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fab18108740 tx=0x7fab14005470 comp rx=0 tx=0).stop 2024-08-28T19:41:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab25974740 1 -- 172.21.15.16:0/2079921368 shutdown_connections 2024-08-28T19:41:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab25974740 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x5591d59a9070 0x5591d5dd9270 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab25974740 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591d5988390 0x5591d5d6ba80 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab25974740 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5591d59801b0 0x5591d5d6b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab25974740 1 --2- 172.21.15.16:0/2079921368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d59679f0 0x5591d5d6b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab25974740 1 -- 172.21.15.16:0/2079921368 >> 172.21.15.16:0/2079921368 conn(0x5591d599d460 msgr2=0x5591d5d4a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:19.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab00ff9640 0 client.20118 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:41:19.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab25974740 1 -- 172.21.15.16:0/2079921368 shutdown_connections 2024-08-28T19:41:19.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab25974740 1 -- 172.21.15.16:0/2079921368 wait complete. 2024-08-28T19:41:19.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.075+0000 7fab25974740 20 client.20118 trim_cache size 0 max 16384 2024-08-28T19:41:19.084 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:41:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 -- 172.21.15.16:0/3659007722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fab040b4d20 msgr2=0x7fab040b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fab040b4d20 0x7fab040b7230 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fab100e3360 tx=0x7fab10078450 comp rx=0 tx=0).stop 2024-08-28T19:41:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 -- 172.21.15.16:0/3659007722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab181088d0 msgr2=0x7fab1819f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab181088d0 0x7fab1819f9a0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fab10009fd0 tx=0x7fab1000ef10 comp rx=0 tx=0).stop 2024-08-28T19:41:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 -- 172.21.15.16:0/3659007722 shutdown_connections 2024-08-28T19:41:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab181088d0 0x7fab1819f9a0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab18107f80 0x7fab1819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fab040b4d20 0x7fab040b7230 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 --2- 172.21.15.16:0/3659007722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab18102640 0x7fab1819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 -- 172.21.15.16:0/3659007722 >> 172.21.15.16:0/3659007722 conn(0x7fab180fe0b0 msgr2=0x7fab18101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 -- 172.21.15.16:0/3659007722 shutdown_connections 2024-08-28T19:41:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.076+0000 7fab1e19d640 1 -- 172.21.15.16:0/3659007722 wait complete. 2024-08-28T19:41:19.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.099+0000 7f8fe0ff9640 1 -- 172.21.15.16:0/410800312 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+592 (crc 0 0 0) 0x7f8fe805e040 con 0x560439409840 2024-08-28T19:41:19.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.099+0000 7f8fe0ff9640 10 client.20100 handle_command_reply: tid=0 2024-08-28T19:41:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.099+0000 7f90059da740 1 client.20100 shutdown 2024-08-28T19:41:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.099+0000 7f90059da740 1 -- 172.21.15.16:0/410800312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 msgr2=0x5604397d0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.099+0000 7f90059da740 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 0x5604397d0700 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x5604397b3bb0 tx=0x7f8fe8040ec0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.100+0000 7f90059da740 1 -- 172.21.15.16:0/410800312 shutdown_connections 2024-08-28T19:41:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.100+0000 7f90059da740 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560439409840 0x56043983be60 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.100+0000 7f90059da740 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5604393e8b60 0x560439831e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.100+0000 7f90059da740 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604393e0980 0x5604397d0700 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.100+0000 7f90059da740 1 --2- 172.21.15.16:0/410800312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604393c81c0 0x5604397d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.100+0000 7f90059da740 1 -- 172.21.15.16:0/410800312 >> 172.21.15.16:0/410800312 conn(0x5604393fd990 msgr2=0x5604397ae510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.100+0000 7f8fe0ff9640 0 client.20100 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:19.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.100+0000 7f90059da740 1 -- 172.21.15.16:0/410800312 shutdown_connections 2024-08-28T19:41:19.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.100+0000 7f90059da740 1 -- 172.21.15.16:0/410800312 wait complete. 2024-08-28T19:41:19.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.100+0000 7f90059da740 20 client.20100 trim_cache size 0 max 16384 2024-08-28T19:41:19.104 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:41:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 101, 2024-08-28T19:41:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:41:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 62, 2024-08-28T19:41:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:41:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "9e1b6317": { 2024-08-28T19:41:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 62, 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 130.0, 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:41:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:41:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:41:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:19.106 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 -- 172.21.15.16:0/2697925092 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe40b4d90 msgr2=0x7f8fe40b72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe40b4d90 0x7f8fe40b72a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00a7da0 tx=0x7f8ff0005780 comp rx=0 tx=0).stop 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 -- 172.21.15.16:0/2697925092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff811c940 msgr2=0x7f8ff8118a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff811c940 0x7f8ff8118a30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000eaa0 tx=0x7f8ff000ef60 comp rx=0 tx=0).stop 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 -- 172.21.15.16:0/2697925092 shutdown_connections 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8fe40b4d90 0x7f8fe40b72a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff811c940 0x7f8ff8118a30 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8111a80 0x7f8ff81184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 --2- 172.21.15.16:0/2697925092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ff810a6a0 0x7f8ff811c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 -- 172.21.15.16:0/2697925092 >> 172.21.15.16:0/2697925092 conn(0x7f8ff806e7b0 msgr2=0x7f8ff811e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 -- 172.21.15.16:0/2697925092 shutdown_connections 2024-08-28T19:41:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:19.101+0000 7f8fff465640 1 -- 172.21.15.16:0/2697925092 wait complete. 2024-08-28T19:41:19.487 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163498 22.89 MB/sec execute 295 sec latency 881.647 ms 2024-08-28T19:41:19.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:19 smithi016 ceph-mon[18556]: pgmap v975: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.4 MiB/s wr, 591 op/s 2024-08-28T19:41:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:19 smithi028 ceph-mon[21769]: pgmap v975: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.4 MiB/s wr, 591 op/s 2024-08-28T19:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:19 smithi115 ceph-mon[22184]: pgmap v975: 129 pgs: 129 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.4 MiB/s wr, 591 op/s 2024-08-28T19:41:20.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:41:20.487 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163498 22.81 MB/sec execute 296 sec latency 1881.754 ms 2024-08-28T19:41:20.606 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '9e1b6317', quiesce took 7.4 seconds. Will release after: 2.5999999999999996 2024-08-28T19:41:20.703 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.173+0000 7f44c5624640 1 -- 172.21.15.16:0/4167793150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44c0109240 msgr2=0x7f44c0109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.173+0000 7f44c5624640 1 --2- 172.21.15.16:0/4167793150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44c0109240 0x7f44c0109650 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009900 tx=0x7f44b402a410 comp rx=0 tx=0).stop 2024-08-28T19:41:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.173+0000 7f44c5624640 1 -- 172.21.15.16:0/4167793150 shutdown_connections 2024-08-28T19:41:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.173+0000 7f44c5624640 1 --2- 172.21.15.16:0/4167793150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44c0109b90 0x7f44c01103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.173+0000 7f44c5624640 1 --2- 172.21.15.16:0/4167793150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44c0109240 0x7f44c0109650 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.173+0000 7f44c5624640 1 --2- 172.21.15.16:0/4167793150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c0103900 0x7f44c0103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.173+0000 7f44c5624640 1 -- 172.21.15.16:0/4167793150 >> 172.21.15.16:0/4167793150 conn(0x7f44c007b5f0 msgr2=0x7f44c007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.173+0000 7f44c5624640 1 -- 172.21.15.16:0/4167793150 shutdown_connections 2024-08-28T19:41:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.173+0000 7f44c5624640 1 -- 172.21.15.16:0/4167793150 wait complete. 2024-08-28T19:41:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.174+0000 7f44c5624640 1 Processor -- start 2024-08-28T19:41:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.174+0000 7f44c5624640 1 -- start start 2024-08-28T19:41:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.174+0000 7f44c5624640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44c0103900 0x7f44c019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.174+0000 7f44c5624640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44c0109240 0x7f44c019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.174+0000 7f44c5624640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c0109b90 0x7f44c019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.174+0000 7f44c5624640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44c0113850 con 0x7f44c0103900 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.174+0000 7f44c5624640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01136d0 con 0x7f44c0109b90 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.174+0000 7f44c5624640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01139d0 con 0x7f44c0109240 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44beffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44c0103900 0x7f44c019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44beffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44c0103900 0x7f44c019a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49074/0 (socket says 172.21.15.16:49074) 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44beffd640 1 -- 172.21.15.16:0/4038855352 learned_addr learned my addr 172.21.15.16:0/4038855352 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c0109b90 0x7f44c019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44be7fc640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44c0109240 0x7f44c019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44beffd640 1 -- 172.21.15.16:0/4038855352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44c0109240 msgr2=0x7f44c019a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44beffd640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44c0109240 0x7f44c019a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44beffd640 1 -- 172.21.15.16:0/4038855352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c0109b90 msgr2=0x7f44c019fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44beffd640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c0109b90 0x7f44c019fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44beffd640 1 -- 172.21.15.16:0/4038855352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c01a00d0 con 0x7f44c0103900 2024-08-28T19:41:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44beffd640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44c0103900 0x7f44c019a220 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f44b000c920 tx=0x7f44b000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:21.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44a7fff640 1 -- 172.21.15.16:0/4038855352 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b0007c70 con 0x7f44c0103900 2024-08-28T19:41:21.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44c5624640 1 -- 172.21.15.16:0/4038855352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44c01a03c0 con 0x7f44c0103900 2024-08-28T19:41:21.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44a7fff640 1 -- 172.21.15.16:0/4038855352 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b0007e10 con 0x7f44c0103900 2024-08-28T19:41:21.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44a7fff640 1 -- 172.21.15.16:0/4038855352 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b0018820 con 0x7f44c0103900 2024-08-28T19:41:21.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.175+0000 7f44c5624640 1 -- 172.21.15.16:0/4038855352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44c01a6be0 con 0x7f44c0103900 2024-08-28T19:41:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.178+0000 7f44cbb99740 1 Processor -- start 2024-08-28T19:41:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.178+0000 7f44cbb99740 1 -- start start 2024-08-28T19:41:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.178+0000 7f44a7fff640 1 -- 172.21.15.16:0/4038855352 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f44b000f450 con 0x7f44c0103900 2024-08-28T19:41:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.178+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19947070 0x556c19cec350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.178+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 0x556c19cec890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.178+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19926390 0x556c19cecdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.178+0000 7f44cbb99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556c19c63680 con 0x556c19926390 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.178+0000 7f44cbb99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c19c52c10 con 0x556c1991e1b0 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.178+0000 7f44cbb99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556c19c56bc0 con 0x556c19947070 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44be7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19926390 0x556c19cecdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44be7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19926390 0x556c19cecdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49076/0 (socket says 172.21.15.16:49076) 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44be7fc640 1 -- 172.21.15.16:0/1111998327 learned_addr learned my addr 172.21.15.16:0/1111998327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44beffd640 1 --2- 172.21.15.16:0/1111998327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 0x556c19cec890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/1111998327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19947070 0x556c19cec350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44be7fc640 1 -- 172.21.15.16:0/1111998327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19947070 msgr2=0x556c19cec350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44be7fc640 1 --2- 172.21.15.16:0/1111998327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19947070 0x556c19cec350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44be7fc640 1 -- 172.21.15.16:0/1111998327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 msgr2=0x556c19cec890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44be7fc640 1 --2- 172.21.15.16:0/1111998327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 0x556c19cec890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44be7fc640 1 -- 172.21.15.16:0/1111998327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c199b2650 con 0x556c19926390 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/1111998327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19947070 0x556c19cec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44beffd640 1 --2- 172.21.15.16:0/1111998327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 0x556c19cec890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44a7fff640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44a80b4cd0 0x7f44a80b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44be7fc640 1 --2- 172.21.15.16:0/1111998327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19926390 0x556c19cecdd0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f44b4000c00 tx=0x7f44b4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39fc31f4d0a9d1c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44a80b4cd0 0x7f44a80b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44a57fa640 1 -- 172.21.15.16:0/1111998327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b403f070 con 0x556c19926390 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44a7fff640 1 -- 172.21.15.16:0/4038855352 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f44b00e5a50 con 0x7f44c0103900 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.179+0000 7f44a57fa640 1 -- 172.21.15.16:0/1111998327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b40316a0 con 0x556c19926390 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.180+0000 7f44a57fa640 1 -- 172.21.15.16:0/1111998327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b4035070 con 0x556c19926390 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.180+0000 7f44cbb99740 1 -- 172.21.15.16:0/1111998327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19926390 msgr2=0x556c19cecdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.180+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44a80b4cd0 0x7f44a80b71e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f44b8005fd0 tx=0x7f44b8009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.180+0000 7f44cbb99740 1 --2- 172.21.15.16:0/1111998327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19926390 0x556c19cecdd0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f44b4000c00 tx=0x7f44b4032ce0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.180+0000 7f44cbb99740 1 -- 172.21.15.16:0/1111998327 shutdown_connections 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.180+0000 7f44cbb99740 1 --2- 172.21.15.16:0/1111998327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19926390 0x556c19cecdd0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.180+0000 7f44cbb99740 1 --2- 172.21.15.16:0/1111998327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 0x556c19cec890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.180+0000 7f44cbb99740 1 --2- 172.21.15.16:0/1111998327 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19947070 0x556c19cec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.180+0000 7f44cbb99740 1 -- 172.21.15.16:0/1111998327 >> 172.21.15.16:0/1111998327 conn(0x556c1993b460 msgr2=0x556c19cea240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.180+0000 7f44cbb99740 1 -- 172.21.15.16:0/1111998327 shutdown_connections 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.180+0000 7f44cbb99740 1 -- 172.21.15.16:0/1111998327 wait complete. 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44cbb99740 1 -- start start 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44cbb99740 1 Processor -- start 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c1991e1b0 0x556c19cfbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19926390 0x556c19cfc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19947070 0x556c19d08160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44cbb99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556c19ab5700 con 0x556c1991e1b0 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44cbb99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c19bd9750 con 0x556c19947070 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44cbb99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556c19cfee30 con 0x556c19926390 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44cbb99740 10 client.0 _ll_register_callbacks cb 0x556c19c08e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44beffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c1991e1b0 0x556c19cfbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44beffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c1991e1b0 0x556c19cfbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49086/0 (socket says 172.21.15.16:49086) 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.181+0000 7f44beffd640 1 -- 172.21.15.16:0/1758989004 learned_addr learned my addr 172.21.15.16:0/1758989004 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19947070 0x556c19d08160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44be7fc640 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19926390 0x556c19cfc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44beffd640 1 -- 172.21.15.16:0/1758989004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19926390 msgr2=0x556c19cfc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44beffd640 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19926390 0x556c19cfc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44beffd640 1 -- 172.21.15.16:0/1758989004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19947070 msgr2=0x556c19d08160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44beffd640 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19947070 0x556c19d08160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44beffd640 1 -- 172.21.15.16:0/1758989004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c19c12900 con 0x556c1991e1b0 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44be7fc640 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19926390 0x556c19cfc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19947070 0x556c19d08160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44beffd640 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c1991e1b0 0x556c19cfbe30 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f44b00a8fe0 tx=0x7f44b000fcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2ec86ec2177c11 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44a2ff5640 1 -- 172.21.15.16:0/1758989004 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b00ac470 con 0x556c1991e1b0 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44a2ff5640 1 -- 172.21.15.16:0/1758989004 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f44b000f740 con 0x556c1991e1b0 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44cbb99740 1 -- 172.21.15.16:0/1758989004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44b40316a0 con 0x556c1991e1b0 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44a2ff5640 1 -- 172.21.15.16:0/1758989004 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f44b00abd70 con 0x556c1991e1b0 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44cbb99740 10 client.20130 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44cbb99740 1 -- 172.21.15.16:0/1758989004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x556c19d0ab60 con 0x556c1991e1b0 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.182+0000 7f44a2ff5640 1 -- 172.21.15.16:0/1758989004 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b40316a0 con 0x556c1991e1b0 2024-08-28T19:41:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.183+0000 7f44a2ff5640 1 -- 172.21.15.16:0/1758989004 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f44b00b0220 con 0x556c1991e1b0 2024-08-28T19:41:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.183+0000 7f44cbb99740 10 client.20130 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.183+0000 7f44cbb99740 10 client.20130 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:41:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.183+0000 7f44cbb99740 20 client.20130 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.183+0000 7f44cbb99740 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x556c199059f0 0x556c19d0e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.183+0000 7f44cbb99740 4 client.20130 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:41:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.183+0000 7f44cbb99740 1 -- 172.21.15.16:0/1758989004 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c19c655d0 con 0x556c199059f0 2024-08-28T19:41:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.183+0000 7f44be7fc640 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x556c199059f0 0x556c19d0e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.183+0000 7f44be7fc640 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x556c199059f0 0x556c19d0e030 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7c15226faebd0be6 server_cookie=2d2f40ac3dc2f15 in_seq=0 out_seq=0 2024-08-28T19:41:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.183+0000 7f44a2ff5640 10 client.20130 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:41:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44a2ff5640 1 -- 172.21.15.16:0/1758989004 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556c19c655d0 con 0x556c199059f0 2024-08-28T19:41:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44a2ff5640 10 client.20130 handle_command_reply: tid=0 2024-08-28T19:41:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44cbb99740 1 client.20130 shutdown 2024-08-28T19:41:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44cbb99740 1 -- 172.21.15.16:0/1758989004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c1991e1b0 msgr2=0x556c19cfbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44cbb99740 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c1991e1b0 0x556c19cfbe30 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f44b00a8fe0 tx=0x7f44b000fcc0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44cbb99740 1 -- 172.21.15.16:0/1758989004 shutdown_connections 2024-08-28T19:41:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44cbb99740 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x556c199059f0 0x556c19d0e030 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44cbb99740 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19947070 0x556c19d08160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44cbb99740 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19926390 0x556c19cfc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44cbb99740 1 --2- 172.21.15.16:0/1758989004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c1991e1b0 0x556c19cfbe30 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44a2ff5640 0 client.20130 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:41:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44cbb99740 1 -- 172.21.15.16:0/1758989004 >> 172.21.15.16:0/1758989004 conn(0x556c1993b460 msgr2=0x556c19ce8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.184+0000 7f44cbb99740 1 -- 172.21.15.16:0/1758989004 shutdown_connections 2024-08-28T19:41:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.185+0000 7f44cbb99740 1 -- 172.21.15.16:0/1758989004 wait complete. 2024-08-28T19:41:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.185+0000 7f44cbb99740 20 client.20130 trim_cache size 0 max 16384 2024-08-28T19:41:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44cbb99740 1 Processor -- start 2024-08-28T19:41:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44cbb99740 1 -- start start 2024-08-28T19:41:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c1991e1b0 0x556c19d08060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19926390 0x556c19cf9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19cf9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44cbb99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01ae930 con 0x556c19947070 2024-08-28T19:41:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44cbb99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01b05f0 con 0x556c19926390 2024-08-28T19:41:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44cbb99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01af090 con 0x556c1991e1b0 2024-08-28T19:41:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19cf9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19cf9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49098/0 (socket says 172.21.15.16:49098) 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44bf7fe640 1 -- 172.21.15.16:0/502357849 learned_addr learned my addr 172.21.15.16:0/502357849 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44be7fc640 1 --2- 172.21.15.16:0/502357849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19926390 0x556c19cf9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44beffd640 1 --2- 172.21.15.16:0/502357849 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c1991e1b0 0x556c19d08060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44bf7fe640 1 -- 172.21.15.16:0/502357849 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c1991e1b0 msgr2=0x556c19d08060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/502357849 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c1991e1b0 0x556c19d08060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44bf7fe640 1 -- 172.21.15.16:0/502357849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19926390 msgr2=0x556c19cf9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.198+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/502357849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19926390 0x556c19cf9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.199+0000 7f44bf7fe640 1 -- 172.21.15.16:0/502357849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c19cfa330 con 0x556c19947070 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.199+0000 7f44be7fc640 1 --2- 172.21.15.16:0/502357849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19926390 0x556c19cf9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.199+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/502357849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19cf9560 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f44b8010700 tx=0x7f44b8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8feff4aee594a6c6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.199+0000 7f44a4ff9640 1 -- 172.21.15.16:0/502357849 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b8004d20 con 0x556c19947070 2024-08-28T19:41:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.199+0000 7f44a4ff9640 1 -- 172.21.15.16:0/502357849 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b801cb30 con 0x556c19947070 2024-08-28T19:41:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.199+0000 7f44a4ff9640 1 -- 172.21.15.16:0/502357849 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b8005720 con 0x556c19947070 2024-08-28T19:41:21.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.200+0000 7f44cbb99740 1 -- 172.21.15.16:0/502357849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 msgr2=0x556c19cf9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.200+0000 7f44cbb99740 1 --2- 172.21.15.16:0/502357849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19cf9560 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f44b8010700 tx=0x7f44b8019000 comp rx=0 tx=0).stop 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.200+0000 7f44cbb99740 1 -- 172.21.15.16:0/502357849 shutdown_connections 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.200+0000 7f44cbb99740 1 --2- 172.21.15.16:0/502357849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19cf9560 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.200+0000 7f44cbb99740 1 --2- 172.21.15.16:0/502357849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c19926390 0x556c19cf9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.200+0000 7f44cbb99740 1 --2- 172.21.15.16:0/502357849 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c1991e1b0 0x556c19d08060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.200+0000 7f44cbb99740 1 -- 172.21.15.16:0/502357849 >> 172.21.15.16:0/502357849 conn(0x556c1993b460 msgr2=0x556c19cfa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.200+0000 7f44cbb99740 1 -- 172.21.15.16:0/502357849 shutdown_connections 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.200+0000 7f44cbb99740 1 -- 172.21.15.16:0/502357849 wait complete. 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.200+0000 7f44cbb99740 1 -- start start 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.200+0000 7f44cbb99740 1 Processor -- start 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 0x556c19d048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19926390 0x556c19d00d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44cbb99740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19d012e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44cbb99740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01afe60 con 0x556c19947070 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44cbb99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01af760 con 0x556c1991e1b0 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44cbb99740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01ac930 con 0x556c19926390 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44cbb99740 10 client.0 _ll_register_callbacks cb 0x7f44b00a7c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19d012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19d012e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49104/0 (socket says 172.21.15.16:49104) 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44bf7fe640 1 -- 172.21.15.16:0/2592971044 learned_addr learned my addr 172.21.15.16:0/2592971044 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44beffd640 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 0x556c19d048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44be7fc640 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19926390 0x556c19d00d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44bf7fe640 1 -- 172.21.15.16:0/2592971044 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19926390 msgr2=0x556c19d00d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19926390 0x556c19d00d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44bf7fe640 1 -- 172.21.15.16:0/2592971044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 msgr2=0x556c19d048c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 0x556c19d048c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44bf7fe640 1 -- 172.21.15.16:0/2592971044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c19d01a70 con 0x556c19947070 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44beffd640 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 0x556c19d048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44bf7fe640 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19d012e0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f44b8005340 tx=0x7f44b801c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8a5f38e4bd7538 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.201+0000 7f44a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44a2ff5640 1 -- 172.21.15.16:0/2592971044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b8036030 con 0x556c19947070 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44a2ff5640 1 -- 172.21.15.16:0/2592971044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f44b8004c50 con 0x556c19947070 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44cbb99740 1 -- 172.21.15.16:0/2592971044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44b801cb30 con 0x556c19947070 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44a2ff5640 1 -- 172.21.15.16:0/2592971044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f44b8004e50 con 0x556c19947070 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44cbb99740 10 client.20136 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44cbb99740 1 -- 172.21.15.16:0/2592971044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x556c19d79930 con 0x556c19947070 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44a2ff5640 1 -- 172.21.15.16:0/2592971044 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b801cb30 con 0x556c19947070 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44a2ff5640 1 -- 172.21.15.16:0/2592971044 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f44b801cce0 con 0x556c19947070 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44cbb99740 10 client.20136 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44cbb99740 10 client.20136 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44cbb99740 20 client.20136 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44cbb99740 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x556c198dada0 0x556c19d7c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44cbb99740 4 client.20136 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.202+0000 7f44cbb99740 1 -- 172.21.15.16:0/2592971044 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c19c61070 con 0x556c198dada0 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44beffd640 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x556c198dada0 0x556c19d7c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44beffd640 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x556c198dada0 0x556c19d7c040 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=73265a0569289473 server_cookie=b6c785beef202ee3 in_seq=0 out_seq=0 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44a2ff5640 10 client.20136 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44a2ff5640 1 -- 172.21.15.16:0/2592971044 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c19c61070 con 0x556c198dada0 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44a2ff5640 10 client.20136 handle_command_reply: tid=0 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44cbb99740 1 client.20136 shutdown 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44cbb99740 1 -- 172.21.15.16:0/2592971044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 msgr2=0x556c19d012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44cbb99740 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19d012e0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f44b8005340 tx=0x7f44b801c7e0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44cbb99740 1 -- 172.21.15.16:0/2592971044 shutdown_connections 2024-08-28T19:41:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44cbb99740 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x556c198dada0 0x556c19d7c040 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44cbb99740 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556c19947070 0x556c19d012e0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44cbb99740 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556c19926390 0x556c19d00d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44cbb99740 1 --2- 172.21.15.16:0/2592971044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c1991e1b0 0x556c19d048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44a2ff5640 0 client.20136 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:41:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44cbb99740 1 -- 172.21.15.16:0/2592971044 >> 172.21.15.16:0/2592971044 conn(0x556c1993b460 msgr2=0x556c19ced380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.203+0000 7f44cbb99740 1 -- 172.21.15.16:0/2592971044 shutdown_connections 2024-08-28T19:41:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44cbb99740 1 -- 172.21.15.16:0/2592971044 wait complete. 2024-08-28T19:41:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44cbb99740 20 client.20136 trim_cache size 0 max 16384 2024-08-28T19:41:21.211 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:41:21.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44c5624640 1 -- 172.21.15.16:0/4038855352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44a80b4cd0 msgr2=0x7f44a80b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:21.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44c5624640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44a80b4cd0 0x7f44a80b71e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f44b8005fd0 tx=0x7f44b8009600 comp rx=0 tx=0).stop 2024-08-28T19:41:21.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44c5624640 1 -- 172.21.15.16:0/4038855352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44c0103900 msgr2=0x7f44c019a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:21.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44c5624640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44c0103900 0x7f44c019a220 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f44b000c920 tx=0x7f44b000cde0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44c5624640 1 -- 172.21.15.16:0/4038855352 shutdown_connections 2024-08-28T19:41:21.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44c5624640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f44a80b4cd0 0x7f44a80b71e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44c5624640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c0109b90 0x7f44c019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44c5624640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f44c0109240 0x7f44c019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44c5624640 1 --2- 172.21.15.16:0/4038855352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44c0103900 0x7f44c019a220 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:21.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.204+0000 7f44c5624640 1 -- 172.21.15.16:0/4038855352 >> 172.21.15.16:0/4038855352 conn(0x7f44c007b5f0 msgr2=0x7f44c0107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:21.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.205+0000 7f44c5624640 1 -- 172.21.15.16:0/4038855352 shutdown_connections 2024-08-28T19:41:21.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:21.205+0000 7f44c5624640 1 -- 172.21.15.16:0/4038855352 wait complete. 2024-08-28T19:41:21.487 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163498 22.73 MB/sec execute 297 sec latency 2881.829 ms 2024-08-28T19:41:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:21 smithi016 ceph-mon[18556]: pgmap v976: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.9 MiB/s wr, 575 op/s 2024-08-28T19:41:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:21 smithi028 ceph-mon[21769]: pgmap v976: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.9 MiB/s wr, 575 op/s 2024-08-28T19:41:21.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:21 smithi115 ceph-mon[22184]: pgmap v976: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.9 MiB/s wr, 575 op/s 2024-08-28T19:41:22.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:41:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:22.487 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163498 22.66 MB/sec execute 298 sec latency 3881.927 ms 2024-08-28T19:41:23.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:41:23.487 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163498 22.58 MB/sec execute 299 sec latency 4881.998 ms 2024-08-28T19:41:23.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:23 smithi016 ceph-mon[18556]: pgmap v977: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.4 MiB/s wr, 548 op/s 2024-08-28T19:41:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.595+0000 7f4efff7e640 1 -- 172.21.15.16:0/3584166006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef8102830 msgr2=0x7f4ef8102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.595+0000 7f4efff7e640 1 --2- 172.21.15.16:0/3584166006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef8102830 0x7f4ef8102c40 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009900 tx=0x7f4ef402a410 comp rx=0 tx=0).stop 2024-08-28T19:41:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.595+0000 7f4efff7e640 1 -- 172.21.15.16:0/3584166006 shutdown_connections 2024-08-28T19:41:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.595+0000 7f4efff7e640 1 --2- 172.21.15.16:0/3584166006 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ef8108ac0 0x7f4ef810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.595+0000 7f4efff7e640 1 --2- 172.21.15.16:0/3584166006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8108170 0x7f4ef8108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.595+0000 7f4efff7e640 1 --2- 172.21.15.16:0/3584166006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef8102830 0x7f4ef8102c40 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.595+0000 7f4efff7e640 1 -- 172.21.15.16:0/3584166006 >> 172.21.15.16:0/3584166006 conn(0x7f4ef80794f0 msgr2=0x7f4ef8079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.595+0000 7f4efff7e640 1 -- 172.21.15.16:0/3584166006 shutdown_connections 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.595+0000 7f4efff7e640 1 -- 172.21.15.16:0/3584166006 wait complete. 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.595+0000 7f4efff7e640 1 Processor -- start 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.595+0000 7f4efff7e640 1 -- start start 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efff7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102830 0x7f4ef819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efff7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef8108170 0x7f4ef819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efff7e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ef8108ac0 0x7f4ef819fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efff7e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef8113850 con 0x7f4ef8108170 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efff7e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81136d0 con 0x7f4ef8102830 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efff7e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81139d0 con 0x7f4ef8108ac0 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efd519640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef8108170 0x7f4ef819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efd519640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef8108170 0x7f4ef819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49122/0 (socket says 172.21.15.16:49122) 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efd519640 1 -- 172.21.15.16:0/1019793040 learned_addr learned my addr 172.21.15.16:0/1019793040 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102830 0x7f4ef819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efe51b640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ef8108ac0 0x7f4ef819fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efd519640 1 -- 172.21.15.16:0/1019793040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ef8108ac0 msgr2=0x7f4ef819fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efd519640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ef8108ac0 0x7f4ef819fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efd519640 1 -- 172.21.15.16:0/1019793040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102830 msgr2=0x7f4ef819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efd519640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102830 0x7f4ef819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efd519640 1 -- 172.21.15.16:0/1019793040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef81a0130 con 0x7f4ef8108170 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efe51b640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ef8108ac0 0x7f4ef819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efd519640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef8108170 0x7f4ef819a760 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800d770 tx=0x7f4ee800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4eeeffd640 1 -- 172.21.15.16:0/1019793040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8017070 con 0x7f4ef8108170 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4eeeffd640 1 -- 172.21.15.16:0/1019793040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ee8009c30 con 0x7f4ef8108170 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.596+0000 7f4efff7e640 1 -- 172.21.15.16:0/1019793040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef81a0420 con 0x7f4ef8108170 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.597+0000 7f4eeeffd640 1 -- 172.21.15.16:0/1019793040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee800f620 con 0x7f4ef8108170 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.597+0000 7f4efff7e640 1 -- 172.21.15.16:0/1019793040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef81a6b80 con 0x7f4ef8108170 2024-08-28T19:41:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.599+0000 7f4f064f3740 1 Processor -- start 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.599+0000 7f4f064f3740 1 -- start start 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.599+0000 7f4eeeffd640 1 -- 172.21.15.16:0/1019793040 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4ee801a070 con 0x7f4ef8108170 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.599+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cedb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.599+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 0x55c58cedb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cedbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4f064f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c58ce52680 con 0x55c58cb36070 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4f064f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c58ce41c10 con 0x55c58cb0d1b0 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4f064f3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c58ce45bc0 con 0x55c58cb15390 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efe51b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cedb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efe51b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cedb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49136/0 (socket says 172.21.15.16:49136) 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efe51b640 1 -- 172.21.15.16:0/2226522380 learned_addr learned my addr 172.21.15.16:0/2226522380 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/2226522380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 0x55c58cedb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efd519640 1 --2- 172.21.15.16:0/2226522380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cedbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efe51b640 1 -- 172.21.15.16:0/2226522380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 msgr2=0x55c58cedbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efe51b640 1 --2- 172.21.15.16:0/2226522380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cedbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efe51b640 1 -- 172.21.15.16:0/2226522380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 msgr2=0x55c58cedb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efe51b640 1 --2- 172.21.15.16:0/2226522380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 0x55c58cedb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efe51b640 1 -- 172.21.15.16:0/2226522380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c58cba1650 con 0x55c58cb36070 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/2226522380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 0x55c58cedb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4efe51b640 1 --2- 172.21.15.16:0/2226522380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cedb310 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009fb0 tx=0x7f4ef0010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77b0992df7a7b6c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4eeeffd640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ee40b4ca0 0x7f4ee40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.600+0000 7f4ee3fff640 1 -- 172.21.15.16:0/2226522380 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0016070 con 0x55c58cb36070 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.601+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ee40b4ca0 0x7f4ee40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.601+0000 7f4ee3fff640 1 -- 172.21.15.16:0/2226522380 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ef0012bc0 con 0x55c58cb36070 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.601+0000 7f4ee3fff640 1 -- 172.21.15.16:0/2226522380 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef001bcb0 con 0x55c58cb36070 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.601+0000 7f4eeeffd640 1 -- 172.21.15.16:0/1019793040 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f4ee80e54a0 con 0x7f4ef8108170 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.601+0000 7f4f064f3740 1 -- 172.21.15.16:0/2226522380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 msgr2=0x55c58cedb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.601+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ee40b4ca0 0x7f4ee40b71b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4031440 tx=0x7f4ef4009610 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.601+0000 7f4f064f3740 1 --2- 172.21.15.16:0/2226522380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cedb310 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009fb0 tx=0x7f4ef0010800 comp rx=0 tx=0).stop 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.602+0000 7f4f064f3740 1 -- 172.21.15.16:0/2226522380 shutdown_connections 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.602+0000 7f4f064f3740 1 --2- 172.21.15.16:0/2226522380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cedbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.602+0000 7f4f064f3740 1 --2- 172.21.15.16:0/2226522380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 0x55c58cedb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.602+0000 7f4f064f3740 1 --2- 172.21.15.16:0/2226522380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cedb310 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.602+0000 7f4f064f3740 1 -- 172.21.15.16:0/2226522380 >> 172.21.15.16:0/2226522380 conn(0x55c58cb2a460 msgr2=0x55c58ced9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.602+0000 7f4f064f3740 1 -- 172.21.15.16:0/2226522380 shutdown_connections 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.602+0000 7f4f064f3740 1 -- 172.21.15.16:0/2226522380 wait complete. 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.602+0000 7f4f064f3740 1 -- start start 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.602+0000 7f4f064f3740 1 Processor -- start 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 0x55c58cee6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cee6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cef2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4f064f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c58cca4700 con 0x55c58cb36070 2024-08-28T19:41:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4f064f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c58cdc8750 con 0x55c58cb0d1b0 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4f064f3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c58cee9af0 con 0x55c58cb15390 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4f064f3740 10 client.0 _ll_register_callbacks cb 0x55c58cdf7e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efe51b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cef2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efdd1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 0x55c58cee6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efdd1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 0x55c58cee6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:45206/0 (socket says 172.21.15.16:45206) 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efe51b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cef2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49144/0 (socket says 172.21.15.16:49144) 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efdd1a640 1 -- 172.21.15.16:0/1676584019 learned_addr learned my addr 172.21.15.16:0/1676584019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efd519640 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cee6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efdd1a640 1 -- 172.21.15.16:0/1676584019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 msgr2=0x55c58cee6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cee6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efdd1a640 1 -- 172.21.15.16:0/1676584019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 msgr2=0x55c58cef2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cef2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efdd1a640 1 -- 172.21.15.16:0/1676584019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c58ce01900 con 0x55c58cb0d1b0 2024-08-28T19:41:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efe51b640 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cef2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 0x55c58cee6a90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009900 tx=0x7f4ef4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e512395a8ab20a5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.603+0000 7f4ee17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4ee17fa640 1 -- 172.21.15.16:0/1676584019 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef404d070 con 0x55c58cb0d1b0 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4f064f3740 1 -- 172.21.15.16:0/1676584019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ef0012bc0 con 0x55c58cb0d1b0 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4ee17fa640 1 -- 172.21.15.16:0/1676584019 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f4ef4005030 con 0x55c58cb0d1b0 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4ee17fa640 1 -- 172.21.15.16:0/1676584019 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f4ef4059050 con 0x55c58cb0d1b0 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4f064f3740 10 client.29606 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4f064f3740 1 -- 172.21.15.16:0/1676584019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55c58cef58b0 con 0x55c58cb0d1b0 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4ee17fa640 1 -- 172.21.15.16:0/1676584019 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ef0012bc0 con 0x55c58cb0d1b0 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4ee17fa640 1 -- 172.21.15.16:0/1676584019 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f4ef40035f0 con 0x55c58cb0d1b0 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4f064f3740 10 client.29606 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4f064f3740 10 client.29606 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4f064f3740 20 client.29606 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4f064f3740 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55c58caf49f0 0x55c58cef8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4f064f3740 4 client.29606 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.604+0000 7f4f064f3740 1 -- 172.21.15.16:0/1676584019 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c58ce545d0 con 0x55c58caf49f0 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.605+0000 7f4efd519640 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55c58caf49f0 0x55c58cef8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.605+0000 7f4efd519640 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55c58caf49f0 0x55c58cef8d10 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7470fc2179afc5de server_cookie=d13a7472e47fe959 in_seq=0 out_seq=0 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.605+0000 7f4ee17fa640 10 client.29606 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4ee17fa640 1 -- 172.21.15.16:0/1676584019 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c58ce545d0 con 0x55c58caf49f0 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4ee17fa640 10 client.29606 handle_command_reply: tid=0 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 1 client.29606 shutdown 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 1 -- 172.21.15.16:0/1676584019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 msgr2=0x55c58cee6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 0x55c58cee6a90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009900 tx=0x7f4ef4009260 comp rx=0 tx=0).stop 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 1 -- 172.21.15.16:0/1676584019 shutdown_connections 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55c58caf49f0 0x55c58cef8d10 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb36070 0x55c58cef2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cee6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 1 --2- 172.21.15.16:0/1676584019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb0d1b0 0x55c58cee6a90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 1 -- 172.21.15.16:0/1676584019 >> 172.21.15.16:0/1676584019 conn(0x55c58cb2a460 msgr2=0x55c58cede4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4ee17fa640 0 client.29606 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:41:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 1 -- 172.21.15.16:0/1676584019 shutdown_connections 2024-08-28T19:41:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 1 -- 172.21.15.16:0/1676584019 wait complete. 2024-08-28T19:41:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.606+0000 7f4f064f3740 20 client.29606 trim_cache size 0 max 16384 2024-08-28T19:41:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4f064f3740 1 Processor -- start 2024-08-28T19:41:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4f064f3740 1 -- start start 2024-08-28T19:41:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ced9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cef2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb36070 0x55c58cede410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4f064f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81b0c10 con 0x55c58cb0d1b0 2024-08-28T19:41:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4f064f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81b28d0 con 0x55c58cb36070 2024-08-28T19:41:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4f064f3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81b1370 con 0x55c58cb15390 2024-08-28T19:41:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4efdd1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ced9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4efdd1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ced9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49156/0 (socket says 172.21.15.16:49156) 2024-08-28T19:41:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4efdd1a640 1 -- 172.21.15.16:0/756618761 learned_addr learned my addr 172.21.15.16:0/756618761 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4efd519640 1 --2- 172.21.15.16:0/756618761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cef2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4efe51b640 1 --2- 172.21.15.16:0/756618761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb36070 0x55c58cede410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4efdd1a640 1 -- 172.21.15.16:0/756618761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 msgr2=0x55c58cef2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/756618761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cef2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4efdd1a640 1 -- 172.21.15.16:0/756618761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb36070 msgr2=0x55c58cede410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/756618761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb36070 0x55c58cede410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.620+0000 7f4efdd1a640 1 -- 172.21.15.16:0/756618761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c58cee7d50 con 0x55c58cb0d1b0 2024-08-28T19:41:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.621+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/756618761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ced9300 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f4ef402a930 tx=0x7f4ef402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f71cafe11c6b40a server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.621+0000 7f4ee37fe640 1 -- 172.21.15.16:0/756618761 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef404d070 con 0x55c58cb0d1b0 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.621+0000 7f4ee37fe640 1 -- 172.21.15.16:0/756618761 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ef40054f0 con 0x55c58cb0d1b0 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.621+0000 7f4ee37fe640 1 -- 172.21.15.16:0/756618761 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4035040 con 0x55c58cb0d1b0 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.622+0000 7f4f064f3740 1 -- 172.21.15.16:0/756618761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 msgr2=0x55c58ced9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.622+0000 7f4f064f3740 1 --2- 172.21.15.16:0/756618761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ced9300 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f4ef402a930 tx=0x7f4ef402a410 comp rx=0 tx=0).stop 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.622+0000 7f4f064f3740 1 -- 172.21.15.16:0/756618761 shutdown_connections 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.622+0000 7f4f064f3740 1 --2- 172.21.15.16:0/756618761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb36070 0x55c58cede410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.622+0000 7f4f064f3740 1 --2- 172.21.15.16:0/756618761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58cef2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.622+0000 7f4f064f3740 1 --2- 172.21.15.16:0/756618761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ced9300 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.622+0000 7f4f064f3740 1 -- 172.21.15.16:0/756618761 >> 172.21.15.16:0/756618761 conn(0x55c58cb2a460 msgr2=0x55c58ced8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.622+0000 7f4f064f3740 1 -- 172.21.15.16:0/756618761 shutdown_connections 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.622+0000 7f4f064f3740 1 -- 172.21.15.16:0/756618761 wait complete. 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.622+0000 7f4f064f3740 1 -- start start 2024-08-28T19:41:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4f064f3740 1 Processor -- start 2024-08-28T19:41:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ceeed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58ceef2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4f064f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb36070 0x55c58cf5cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4f064f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81b2140 con 0x55c58cb0d1b0 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4f064f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81b1a40 con 0x55c58cb36070 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4f064f3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81aec10 con 0x55c58cb15390 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4efdd1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ceeed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4efdd1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ceeed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49162/0 (socket says 172.21.15.16:49162) 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4f064f3740 10 client.0 _ll_register_callbacks cb 0x7f4ef4005e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4efdd1a640 1 -- 172.21.15.16:0/371517254 learned_addr learned my addr 172.21.15.16:0/371517254 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4efe51b640 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb36070 0x55c58cf5cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.623+0000 7f4efd519640 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58ceef2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4efdd1a640 1 -- 172.21.15.16:0/371517254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 msgr2=0x55c58ceef2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58ceef2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4efdd1a640 1 -- 172.21.15.16:0/371517254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb36070 msgr2=0x55c58cf5cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb36070 0x55c58cf5cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4efdd1a640 1 -- 172.21.15.16:0/371517254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c58cf5d170 con 0x55c58cb0d1b0 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4efd519640 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58ceef2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4efdd1a640 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ceeed40 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4004010 tx=0x7f4ef4003bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b32a538c76c0b2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4ee17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4ee17fa640 1 -- 172.21.15.16:0/371517254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef406a070 con 0x55c58cb0d1b0 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4ee17fa640 1 -- 172.21.15.16:0/371517254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f4ef4066660 con 0x55c58cb0d1b0 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4f064f3740 1 -- 172.21.15.16:0/371517254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ef40054f0 con 0x55c58cb0d1b0 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4ee17fa640 1 -- 172.21.15.16:0/371517254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f4ef4076050 con 0x55c58cb0d1b0 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4f064f3740 10 client.20154 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4f064f3740 1 -- 172.21.15.16:0/371517254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55c58cf682b0 con 0x55c58cb0d1b0 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.624+0000 7f4ee17fa640 1 -- 172.21.15.16:0/371517254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ef40054f0 con 0x55c58cb0d1b0 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.625+0000 7f4ee17fa640 1 -- 172.21.15.16:0/371517254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f4ef403d680 con 0x55c58cb0d1b0 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.625+0000 7f4f064f3740 10 client.20154 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.625+0000 7f4f064f3740 10 client.20154 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.625+0000 7f4f064f3740 20 client.20154 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.625+0000 7f4f064f3740 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55c58cac9da0 0x55c58cf6a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.625+0000 7f4f064f3740 4 client.20154 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.625+0000 7f4f064f3740 1 -- 172.21.15.16:0/371517254 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c58ce50070 con 0x55c58cac9da0 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.625+0000 7f4efd519640 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55c58cac9da0 0x55c58cf6a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4ee17fa640 10 client.20154 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4efd519640 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55c58cac9da0 0x55c58cf6a960 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a110f35fbb72ea82 server_cookie=c0ad6d28f1009d55 in_seq=0 out_seq=0 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4ee17fa640 1 -- 172.21.15.16:0/371517254 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c58ce50070 con 0x55c58cac9da0 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4ee17fa640 10 client.20154 handle_command_reply: tid=0 2024-08-28T19:41:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 1 client.20154 shutdown 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 1 -- 172.21.15.16:0/371517254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 msgr2=0x55c58ceeed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ceeed40 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4004010 tx=0x7f4ef4003bb0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 1 -- 172.21.15.16:0/371517254 shutdown_connections 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55c58cac9da0 0x55c58cf6a960 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c58cb36070 0x55c58cf5cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c58cb15390 0x55c58ceef2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4ee17fa640 0 client.20154 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 1 --2- 172.21.15.16:0/371517254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58cb0d1b0 0x55c58ceeed40 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 1 -- 172.21.15.16:0/371517254 >> 172.21.15.16:0/371517254 conn(0x55c58cb2a460 msgr2=0x55c58cede410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 1 -- 172.21.15.16:0/371517254 shutdown_connections 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 1 -- 172.21.15.16:0/371517254 wait complete. 2024-08-28T19:41:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.626+0000 7f4f064f3740 20 client.20154 trim_cache size 0 max 16384 2024-08-28T19:41:23.632 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:41:23.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 -- 172.21.15.16:0/1019793040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ee40b4ca0 msgr2=0x7f4ee40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:23.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ee40b4ca0 0x7f4ee40b71b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4031440 tx=0x7f4ef4009610 comp rx=0 tx=0).stop 2024-08-28T19:41:23.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 -- 172.21.15.16:0/1019793040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef8108170 msgr2=0x7f4ef819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:23.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef8108170 0x7f4ef819a760 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800d770 tx=0x7f4ee800dc30 comp rx=0 tx=0).stop 2024-08-28T19:41:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 -- 172.21.15.16:0/1019793040 shutdown_connections 2024-08-28T19:41:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ef8108ac0 0x7f4ef819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef8108170 0x7f4ef819a760 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4ee40b4ca0 0x7f4ee40b71b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 --2- 172.21.15.16:0/1019793040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102830 0x7f4ef819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 -- 172.21.15.16:0/1019793040 >> 172.21.15.16:0/1019793040 conn(0x7f4ef80794f0 msgr2=0x7f4ef8101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 -- 172.21.15.16:0/1019793040 shutdown_connections 2024-08-28T19:41:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:23.627+0000 7f4efff7e640 1 -- 172.21.15.16:0/1019793040 wait complete. 2024-08-28T19:41:23.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:23 smithi028 ceph-mon[21769]: pgmap v977: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.4 MiB/s wr, 548 op/s 2024-08-28T19:41:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:23 smithi115 ceph-mon[22184]: pgmap v977: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.4 MiB/s wr, 548 op/s 2024-08-28T19:41:24.488 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163498 22.51 MB/sec execute 300 sec latency 5882.034 ms 2024-08-28T19:41:24.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:41:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:25.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.419+0000 7f5f66655640 1 -- 172.21.15.16:0/1898081018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6011c9d0 msgr2=0x7f5f6011eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:25.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.419+0000 7f5f66655640 1 --2- 172.21.15.16:0/1898081018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6011c9d0 0x7f5f6011eee0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009ca0 tx=0x7f5f5802a8d0 comp rx=0 tx=0).stop 2024-08-28T19:41:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.419+0000 7f5f66655640 1 -- 172.21.15.16:0/1898081018 shutdown_connections 2024-08-28T19:41:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.419+0000 7f5f66655640 1 --2- 172.21.15.16:0/1898081018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6011c9d0 0x7f5f6011eee0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.419+0000 7f5f66655640 1 --2- 172.21.15.16:0/1898081018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5f6010a780 0x7f5f6010ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.419+0000 7f5f66655640 1 --2- 172.21.15.16:0/1898081018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f60111ce0 0x7f5f6010a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.419+0000 7f5f66655640 1 -- 172.21.15.16:0/1898081018 >> 172.21.15.16:0/1898081018 conn(0x7f5f60070c60 msgr2=0x7f5f60071070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.422+0000 7f5f66655640 1 -- 172.21.15.16:0/1898081018 shutdown_connections 2024-08-28T19:41:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.422+0000 7f5f66655640 1 -- 172.21.15.16:0/1898081018 wait complete. 2024-08-28T19:41:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.423+0000 7f5f66655640 1 Processor -- start 2024-08-28T19:41:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.423+0000 7f5f66655640 1 -- start start 2024-08-28T19:41:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.424+0000 7f5f66655640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5f6010a780 0x7f5f601a9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.424+0000 7f5f66655640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f60111ce0 0x7f5f601a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.424+0000 7f5f66655640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6011c9d0 0x7f5f601aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.424+0000 7f5f66655640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5f60122730 con 0x7f5f60111ce0 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.424+0000 7f5f66655640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f601225b0 con 0x7f5f6011c9d0 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.424+0000 7f5f66655640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5f601228b0 con 0x7f5f6010a780 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.424+0000 7f5f64bf2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6011c9d0 0x7f5f601aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.424+0000 7f5f64bf2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6011c9d0 0x7f5f601aea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:50974/0 (socket says 172.21.15.16:50974) 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.424+0000 7f5f64bf2640 1 -- 172.21.15.16:0/1293028770 learned_addr learned my addr 172.21.15.16:0/1293028770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.426+0000 7f5f64bf2640 1 -- 172.21.15.16:0/1293028770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5f6010a780 msgr2=0x7f5f601a9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.426+0000 7f5f64bf2640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5f6010a780 0x7f5f601a9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.426+0000 7f5f64bf2640 1 -- 172.21.15.16:0/1293028770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f60111ce0 msgr2=0x7f5f601a97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.426+0000 7f5f64bf2640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f60111ce0 0x7f5f601a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.426+0000 7f5f64bf2640 1 -- 172.21.15.16:0/1293028770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f601af180 con 0x7f5f6011c9d0 2024-08-28T19:41:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.432+0000 7f5f64bf2640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6011c9d0 0x7f5f601aea80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5f580097b0 tx=0x7f5f58031a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.433+0000 7f5f5d7fa640 1 -- 172.21.15.16:0/1293028770 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f58003db0 con 0x7f5f6011c9d0 2024-08-28T19:41:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.433+0000 7f5f66655640 1 -- 172.21.15.16:0/1293028770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f601af410 con 0x7f5f6011c9d0 2024-08-28T19:41:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.433+0000 7f5f66655640 1 -- 172.21.15.16:0/1293028770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f60112120 con 0x7f5f6011c9d0 2024-08-28T19:41:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.434+0000 7f5f5d7fa640 1 -- 172.21.15.16:0/1293028770 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f580044f0 con 0x7f5f6011c9d0 2024-08-28T19:41:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.434+0000 7f5f5d7fa640 1 -- 172.21.15.16:0/1293028770 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5802fd70 con 0x7f5f6011c9d0 2024-08-28T19:41:25.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.435+0000 7f5f66655640 1 -- 172.21.15.16:0/1293028770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f6010ac10 con 0x7f5f6011c9d0 2024-08-28T19:41:25.437 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:25.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.437+0000 7f5f5d7fa640 1 -- 172.21.15.16:0/1293028770 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5f580305d0 con 0x7f5f6011c9d0 2024-08-28T19:41:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.438+0000 7f5f5d7fa640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5f480b4dd0 0x7f5f480b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.439+0000 7f5f5d7fa640 1 -- 172.21.15.16:0/1293028770 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f5f58107f80 con 0x7f5f6011c9d0 2024-08-28T19:41:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.439+0000 7f5f5ffff640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5f480b4dd0 0x7f5f480b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.440+0000 7f5f5ffff640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5f480b4dd0 0x7f5f480b72e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009710 tx=0x7f5f50006c90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:25.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.444+0000 7f5f5d7fa640 1 -- 172.21.15.16:0/1293028770 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f5f580ce940 con 0x7f5f6011c9d0 2024-08-28T19:41:25.487 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163498 22.43 MB/sec execute 301 sec latency 6882.366 ms 2024-08-28T19:41:25.626 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:25 smithi016 ceph-mon[18556]: pgmap v978: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 451 op/s 2024-08-28T19:41:25.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:25 smithi028 ceph-mon[21769]: pgmap v978: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 451 op/s 2024-08-28T19:41:25.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:25 smithi115 ceph-mon[22184]: pgmap v978: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.0 MiB/s wr, 451 op/s 2024-08-28T19:41:25.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.873+0000 7f5f66655640 1 -- 172.21.15.16:0/1293028770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f60112440 con 0x7f5f6011c9d0 2024-08-28T19:41:25.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.874+0000 7f5f5d7fa640 1 -- 172.21.15.16:0/1293028770 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+11992 (secure 0 0 0) 0x7f5f580ce240 con 0x7f5f6011c9d0 2024-08-28T19:41:25.876 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:41:25.876 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":101,"btime":"2024-08-28T19:40:25:411669+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":101,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:40:24.586397+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:41:25.876 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 101 2024-08-28T19:41:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.878+0000 7f5f66655640 1 -- 172.21.15.16:0/1293028770 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5f480b4dd0 msgr2=0x7f5f480b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:25.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.878+0000 7f5f66655640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5f480b4dd0 0x7f5f480b72e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009710 tx=0x7f5f50006c90 comp rx=0 tx=0).stop 2024-08-28T19:41:25.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.878+0000 7f5f66655640 1 -- 172.21.15.16:0/1293028770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6011c9d0 msgr2=0x7f5f601aea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:25.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.878+0000 7f5f66655640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6011c9d0 0x7f5f601aea80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5f580097b0 tx=0x7f5f58031a00 comp rx=0 tx=0).stop 2024-08-28T19:41:25.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.880+0000 7f5f66655640 1 -- 172.21.15.16:0/1293028770 shutdown_connections 2024-08-28T19:41:25.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.880+0000 7f5f66655640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5f480b4dd0 0x7f5f480b72e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:25.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.880+0000 7f5f66655640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6011c9d0 0x7f5f601aea80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:25.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.880+0000 7f5f66655640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f60111ce0 0x7f5f601a97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:25.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.880+0000 7f5f66655640 1 --2- 172.21.15.16:0/1293028770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5f6010a780 0x7f5f601a9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:25.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.880+0000 7f5f66655640 1 -- 172.21.15.16:0/1293028770 >> 172.21.15.16:0/1293028770 conn(0x7f5f60070c60 msgr2=0x7f5f6011e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:25.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.882+0000 7f5f66655640 1 -- 172.21.15.16:0/1293028770 shutdown_connections 2024-08-28T19:41:25.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:25.882+0000 7f5f66655640 1 -- 172.21.15.16:0/1293028770 wait complete. 2024-08-28T19:41:26.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.062+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/500950589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9dc8102800 msgr2=0x7f9dc8102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.062+0000 7f9dcdd4d640 1 --2- 172.21.15.16:0/500950589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9dc8102800 0x7f9dc8102c10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9db8009900 tx=0x7f9db802a410 comp rx=0 tx=0).stop 2024-08-28T19:41:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.062+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/500950589 shutdown_connections 2024-08-28T19:41:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.062+0000 7f9dcdd4d640 1 --2- 172.21.15.16:0/500950589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc8108a90 0x7f9dc810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.062+0000 7f9dcdd4d640 1 --2- 172.21.15.16:0/500950589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dc8108140 0x7f9dc8108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.062+0000 7f9dcdd4d640 1 --2- 172.21.15.16:0/500950589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9dc8102800 0x7f9dc8102c10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.062+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/500950589 >> 172.21.15.16:0/500950589 conn(0x7f9dc80794d0 msgr2=0x7f9dc80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.063+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/500950589 shutdown_connections 2024-08-28T19:41:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.063+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/500950589 wait complete. 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.064+0000 7f9dcdd4d640 1 Processor -- start 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.064+0000 7f9dcdd4d640 1 -- start start 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.065+0000 7f9dcdd4d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9dc8102800 0x7f9dc81a1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.065+0000 7f9dcdd4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dc8108140 0x7f9dc81a2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.065+0000 7f9dcdd4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc8108a90 0x7f9dc81a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.065+0000 7f9dcdd4d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc8111640 con 0x7f9dc8108a90 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.065+0000 7f9dcdd4d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81114c0 con 0x7f9dc8108140 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.065+0000 7f9dcdd4d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81117c0 con 0x7f9dc8102800 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.065+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc8108a90 0x7f9dc81a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.065+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc8108a90 0x7f9dc81a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44498/0 (socket says 172.21.15.16:44498) 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.065+0000 7f9dc7fff640 1 -- 172.21.15.16:0/3829451462 learned_addr learned my addr 172.21.15.16:0/3829451462 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.065+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9dc8102800 0x7f9dc81a1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.065+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dc8108140 0x7f9dc81a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.066+0000 7f9dc77fe640 1 -- 172.21.15.16:0/3829451462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dc8108140 msgr2=0x7f9dc81a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:26.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.066+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dc8108140 0x7f9dc81a2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.066+0000 7f9dc77fe640 1 -- 172.21.15.16:0/3829451462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc8108a90 msgr2=0x7f9dc81a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:26.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.066+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc8108a90 0x7f9dc81a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.066+0000 7f9dc77fe640 1 -- 172.21.15.16:0/3829451462 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc81a67b0 con 0x7f9dc8102800 2024-08-28T19:41:26.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.066+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9dc8102800 0x7f9dc81a1fe0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9db802a930 tx=0x7f9db8005080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:26.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.067+0000 7f9dc4ff9640 1 -- 172.21.15.16:0/3829451462 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db803f070 con 0x7f9dc8102800 2024-08-28T19:41:26.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.067+0000 7f9dc4ff9640 1 -- 172.21.15.16:0/3829451462 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9db8003aa0 con 0x7f9dc8102800 2024-08-28T19:41:26.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.067+0000 7f9dc4ff9640 1 -- 172.21.15.16:0/3829451462 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db803f070 con 0x7f9dc8102800 2024-08-28T19:41:26.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.067+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/3829451462 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc81a69e0 con 0x7f9dc8102800 2024-08-28T19:41:26.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.067+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/3829451462 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc81a6e40 con 0x7f9dc8102800 2024-08-28T19:41:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.072+0000 7f9dc4ff9640 1 -- 172.21.15.16:0/3829451462 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f9db8042070 con 0x7f9dc8102800 2024-08-28T19:41:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.072+0000 7f9dd42c2740 1 Processor -- start 2024-08-28T19:41:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.073+0000 7f9dd42c2740 1 -- start start 2024-08-28T19:41:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.073+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00184070 0x55ca00529310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.073+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca0015b1b0 0x55ca00529850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.073+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00163390 0x55ca00529d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.073+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ca004a0680 con 0x55ca0015b1b0 2024-08-28T19:41:26.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.073+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca0048fc10 con 0x55ca00163390 2024-08-28T19:41:26.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.074+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ca00493bc0 con 0x55ca00184070 2024-08-28T19:41:26.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.074+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00184070 0x55ca00529310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.074+0000 7f9dc6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00163390 0x55ca00529d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.074+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00184070 0x55ca00529310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:52974/0 (socket says 172.21.15.16:52974) 2024-08-28T19:41:26.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.074+0000 7f9dc7fff640 1 -- 172.21.15.16:0/1600008097 learned_addr learned my addr 172.21.15.16:0/1600008097 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:26.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.074+0000 7f9dc4ff9640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9db00b5100 0x7f9db00b7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.075+0000 7f9dc6ffd640 1 -- 172.21.15.16:0/1600008097 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00184070 msgr2=0x55ca00529310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:26.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.075+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/1600008097 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00184070 0x55ca00529310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.075+0000 7f9dc6ffd640 1 -- 172.21.15.16:0/1600008097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca0015b1b0 msgr2=0x55ca00529850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:26.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.075+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/1600008097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca0015b1b0 0x55ca00529850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.075+0000 7f9dc6ffd640 1 -- 172.21.15.16:0/1600008097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca001ef650 con 0x55ca00163390 2024-08-28T19:41:26.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.075+0000 7f9dc4ff9640 1 -- 172.21.15.16:0/3829451462 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f9db8005480 con 0x7f9dc8102800 2024-08-28T19:41:26.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.075+0000 7f9dc7fff640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9db00b5100 0x7f9db00b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.076+0000 7f9dc7fff640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9db00b5100 0x7f9db00b7610 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0010670 tx=0x7f9dc0004a40 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:26.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.076+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/1600008097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00163390 0x55ca00529d90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0058c0 tx=0x7f9dbc00aef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3a9c55e4f152e4c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:26.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.076+0000 7f9dadffb640 1 -- 172.21.15.16:0/1600008097 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc004020 con 0x55ca00163390 2024-08-28T19:41:26.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.077+0000 7f9dadffb640 1 -- 172.21.15.16:0/1600008097 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dbc0041c0 con 0x55ca00163390 2024-08-28T19:41:26.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.077+0000 7f9dadffb640 1 -- 172.21.15.16:0/1600008097 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc018680 con 0x55ca00163390 2024-08-28T19:41:26.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.078+0000 7f9dd42c2740 1 -- 172.21.15.16:0/1600008097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00163390 msgr2=0x55ca00529d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:26.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.078+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/1600008097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00163390 0x55ca00529d90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0058c0 tx=0x7f9dbc00aef0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.079+0000 7f9dd42c2740 1 -- 172.21.15.16:0/1600008097 shutdown_connections 2024-08-28T19:41:26.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.079+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/1600008097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00163390 0x55ca00529d90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.079+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/1600008097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca0015b1b0 0x55ca00529850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.079+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/1600008097 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00184070 0x55ca00529310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.079+0000 7f9dd42c2740 1 -- 172.21.15.16:0/1600008097 >> 172.21.15.16:0/1600008097 conn(0x55ca00178460 msgr2=0x55ca00527200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:26.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.079+0000 7f9dd42c2740 1 -- 172.21.15.16:0/1600008097 shutdown_connections 2024-08-28T19:41:26.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.079+0000 7f9dd42c2740 1 -- 172.21.15.16:0/1600008097 wait complete. 2024-08-28T19:41:26.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.080+0000 7f9dd42c2740 1 -- start start 2024-08-28T19:41:26.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.080+0000 7f9dd42c2740 1 Processor -- start 2024-08-28T19:41:26.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca0015b1b0 0x55ca00534b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca00535070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00540eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ca002f2700 con 0x55ca00163390 2024-08-28T19:41:26.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca00416750 con 0x55ca00184070 2024-08-28T19:41:26.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ca00537b40 con 0x55ca0015b1b0 2024-08-28T19:41:26.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dd42c2740 10 client.0 _ll_register_callbacks cb 0x55ca00445e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:26.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dc6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca00535070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dc6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca00535070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44528/0 (socket says 172.21.15.16:44528) 2024-08-28T19:41:26.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dc6ffd640 1 -- 172.21.15.16:0/3616971324 learned_addr learned my addr 172.21.15.16:0/3616971324 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:26.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca0015b1b0 0x55ca00534b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dc7fff640 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00540eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dc77fe640 1 -- 172.21.15.16:0/3616971324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 msgr2=0x55ca00540eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:26.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00540eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dc77fe640 1 -- 172.21.15.16:0/3616971324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 msgr2=0x55ca00535070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:26.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca00535070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.081+0000 7f9dc77fe640 1 -- 172.21.15.16:0/3616971324 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca0044f900 con 0x55ca0015b1b0 2024-08-28T19:41:26.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.082+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca0015b1b0 0x55ca00534b30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9db8102cd0 tx=0x7f9db80cb670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cda9fe76f05dd6d9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.082+0000 7f9da6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.082+0000 7f9da6ffd640 1 -- 172.21.15.16:0/3616971324 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db8049030 con 0x55ca0015b1b0 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.082+0000 7f9dd42c2740 1 -- 172.21.15.16:0/3616971324 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dbc0041c0 con 0x55ca0015b1b0 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.082+0000 7f9da6ffd640 1 -- 172.21.15.16:0/3616971324 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f9db8102f50 con 0x55ca0015b1b0 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.082+0000 7f9da6ffd640 1 -- 172.21.15.16:0/3616971324 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f9db803f070 con 0x55ca0015b1b0 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.082+0000 7f9dd42c2740 10 client.29179 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.082+0000 7f9dd42c2740 1 -- 172.21.15.16:0/3616971324 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55ca005439f0 con 0x55ca0015b1b0 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.083+0000 7f9da6ffd640 1 -- 172.21.15.16:0/3616971324 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dbc0041c0 con 0x55ca0015b1b0 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.083+0000 7f9da6ffd640 1 -- 172.21.15.16:0/3616971324 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f9db80c7fb0 con 0x55ca0015b1b0 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.083+0000 7f9dd42c2740 10 client.29179 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.083+0000 7f9dd42c2740 10 client.29179 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.083+0000 7f9dd42c2740 20 client.29179 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.083+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55ca001429f0 0x55ca00546e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.083+0000 7f9dd42c2740 4 client.29179 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:41:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.083+0000 7f9dd42c2740 1 -- 172.21.15.16:0/3616971324 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca004a25d0 con 0x55ca001429f0 2024-08-28T19:41:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.083+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55ca001429f0 0x55ca00546e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.084+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55ca001429f0 0x55ca00546e50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fc5dd82578bf8057 server_cookie=fe3cc05d14034a73 in_seq=0 out_seq=0 2024-08-28T19:41:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.084+0000 7f9da6ffd640 10 client.29179 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:41:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.085+0000 7f9da6ffd640 1 -- 172.21.15.16:0/3616971324 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ca004a25d0 con 0x55ca001429f0 2024-08-28T19:41:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.085+0000 7f9da6ffd640 10 client.29179 handle_command_reply: tid=0 2024-08-28T19:41:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.085+0000 7f9dd42c2740 1 client.29179 shutdown 2024-08-28T19:41:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.086+0000 7f9dd42c2740 1 -- 172.21.15.16:0/3616971324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca0015b1b0 msgr2=0x55ca00534b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.086+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca0015b1b0 0x55ca00534b30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9db8102cd0 tx=0x7f9db80cb670 comp rx=0 tx=0).stop 2024-08-28T19:41:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.086+0000 7f9dd42c2740 1 -- 172.21.15.16:0/3616971324 shutdown_connections 2024-08-28T19:41:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.086+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55ca001429f0 0x55ca00546e50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.086+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00540eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.086+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca00535070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.086+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/3616971324 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca0015b1b0 0x55ca00534b30 secure :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9db8102cd0 tx=0x7f9db80cb670 comp rx=0 tx=0).stop 2024-08-28T19:41:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.086+0000 7f9dd42c2740 1 -- 172.21.15.16:0/3616971324 >> 172.21.15.16:0/3616971324 conn(0x55ca00178460 msgr2=0x55ca0052c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:26.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.086+0000 7f9da6ffd640 0 client.29179 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:41:26.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.086+0000 7f9dd42c2740 1 -- 172.21.15.16:0/3616971324 shutdown_connections 2024-08-28T19:41:26.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.087+0000 7f9dd42c2740 1 -- 172.21.15.16:0/3616971324 wait complete. 2024-08-28T19:41:26.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.087+0000 7f9dd42c2740 20 client.29179 trim_cache size 0 max 16384 2024-08-28T19:41:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dd42c2740 1 Processor -- start 2024-08-28T19:41:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dd42c2740 1 -- start start 2024-08-28T19:41:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca00533ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00531d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 0x55ca00532260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81bf9b0 con 0x55ca00163390 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81c1670 con 0x55ca00184070 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81c0110 con 0x55ca00117da0 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dc77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca00533ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dc6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00531d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dc77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca00533ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44530/0 (socket says 172.21.15.16:44530) 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dc77fe640 1 -- 172.21.15.16:0/1232552647 learned_addr learned my addr 172.21.15.16:0/1232552647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.105+0000 7f9dc7fff640 1 --2- 172.21.15.16:0/1232552647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 0x55ca00532260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.106+0000 7f9dc6ffd640 1 -- 172.21.15.16:0/1232552647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 msgr2=0x55ca00532260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.106+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/1232552647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 0x55ca00532260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.106+0000 7f9dc6ffd640 1 -- 172.21.15.16:0/1232552647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 msgr2=0x55ca00533ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.106+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/1232552647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca00533ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.106+0000 7f9dc6ffd640 1 -- 172.21.15.16:0/1232552647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca0052b600 con 0x55ca00184070 2024-08-28T19:41:26.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.106+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/1232552647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca00533ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:26.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.106+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/1232552647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00531d20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x55ca005475f0 tx=0x7f9dbc00aef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fa98a9f48f308eb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:26.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.106+0000 7f9dad7fa640 1 -- 172.21.15.16:0/1232552647 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc01c070 con 0x55ca00184070 2024-08-28T19:41:26.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.106+0000 7f9dad7fa640 1 -- 172.21.15.16:0/1232552647 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dbc01d040 con 0x55ca00184070 2024-08-28T19:41:26.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.106+0000 7f9dad7fa640 1 -- 172.21.15.16:0/1232552647 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc005290 con 0x55ca00184070 2024-08-28T19:41:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.107+0000 7f9dd42c2740 1 -- 172.21.15.16:0/1232552647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 msgr2=0x55ca00531d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.107+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/1232552647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00531d20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x55ca005475f0 tx=0x7f9dbc00aef0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.107+0000 7f9dd42c2740 1 -- 172.21.15.16:0/1232552647 shutdown_connections 2024-08-28T19:41:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.107+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/1232552647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 0x55ca00532260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.107+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/1232552647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00531d20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.107+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/1232552647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca00533ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.107+0000 7f9dd42c2740 1 -- 172.21.15.16:0/1232552647 >> 172.21.15.16:0/1232552647 conn(0x55ca00178460 msgr2=0x55ca005333b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.107+0000 7f9dd42c2740 1 -- 172.21.15.16:0/1232552647 shutdown_connections 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.107+0000 7f9dd42c2740 1 -- 172.21.15.16:0/1232552647 wait complete. 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.107+0000 7f9dd42c2740 1 -- start start 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dd42c2740 1 Processor -- start 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 0x55ca00538f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca005394d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dd42c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00539a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81c0ee0 con 0x55ca00163390 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81c07e0 con 0x55ca00184070 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dd42c2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81bd9b0 con 0x55ca00117da0 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dd42c2740 10 client.0 _ll_register_callbacks cb 0x7f9db80c7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dc77fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 0x55ca00538f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dc77fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 0x55ca00538f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:53008/0 (socket says 172.21.15.16:53008) 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dc77fe640 1 -- 172.21.15.16:0/961139409 learned_addr learned my addr 172.21.15.16:0/961139409 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca005394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dc7fff640 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00539a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dc77fe640 1 -- 172.21.15.16:0/961139409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 msgr2=0x55ca00539a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00539a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dc77fe640 1 -- 172.21.15.16:0/961139409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 msgr2=0x55ca005394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca005394d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dc77fe640 1 -- 172.21.15.16:0/961139409 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca005b09b0 con 0x55ca00117da0 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.108+0000 7f9dc7fff640 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00539a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.109+0000 7f9dc77fe640 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 0x55ca00538f90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9db80c7da0 tx=0x7f9db80cb7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22ce35de952d14d1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.109+0000 7f9da6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:41:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.109+0000 7f9da6ffd640 1 -- 172.21.15.16:0/961139409 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db80d0020 con 0x55ca00117da0 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.109+0000 7f9da6ffd640 1 -- 172.21.15.16:0/961139409 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f9db80d4b00 con 0x55ca00117da0 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.109+0000 7f9dd42c2740 1 -- 172.21.15.16:0/961139409 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dbc01d040 con 0x55ca00117da0 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.109+0000 7f9da6ffd640 1 -- 172.21.15.16:0/961139409 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f9db8049040 con 0x55ca00117da0 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.109+0000 7f9dd42c2740 10 client.29185 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.109+0000 7f9dd42c2740 1 -- 172.21.15.16:0/961139409 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55ca005b1c60 con 0x55ca00117da0 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.109+0000 7f9da6ffd640 1 -- 172.21.15.16:0/961139409 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dbc01d040 con 0x55ca00117da0 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.109+0000 7f9da6ffd640 1 -- 172.21.15.16:0/961139409 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f9db80cbf80 con 0x55ca00117da0 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.110+0000 7f9dd42c2740 10 client.29185 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.110+0000 7f9dd42c2740 10 client.29185 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.110+0000 7f9dd42c2740 20 client.29185 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.110+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55ca005b2220 0x55ca005b4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.110+0000 7f9dd42c2740 4 client.29185 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.110+0000 7f9dd42c2740 1 -- 172.21.15.16:0/961139409 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ca0049e070 con 0x55ca005b2220 2024-08-28T19:41:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.110+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55ca005b2220 0x55ca005b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.110+0000 7f9da6ffd640 10 client.29185 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.110+0000 7f9dc6ffd640 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55ca005b2220 0x55ca005b4640 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=39a32e6a29eca8b5 server_cookie=d5de209dfce9b389 in_seq=0 out_seq=0 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.110+0000 7f9da6ffd640 1 -- 172.21.15.16:0/961139409 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ca0049e070 con 0x55ca005b2220 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.110+0000 7f9da6ffd640 10 client.29185 handle_command_reply: tid=0 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 1 client.29185 shutdown 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 1 -- 172.21.15.16:0/961139409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 msgr2=0x55ca00538f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 0x55ca00538f90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9db80c7da0 tx=0x7f9db80cb7a0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 1 -- 172.21.15.16:0/961139409 shutdown_connections 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x55ca005b2220 0x55ca005b4640 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca00184070 0x55ca00539a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca00163390 0x55ca005394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 1 --2- 172.21.15.16:0/961139409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ca00117da0 0x55ca00538f90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 1 -- 172.21.15.16:0/961139409 >> 172.21.15.16:0/961139409 conn(0x55ca00178460 msgr2=0x55ca005353b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9da6ffd640 0 client.29185 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:41:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 1 -- 172.21.15.16:0/961139409 shutdown_connections 2024-08-28T19:41:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 1 -- 172.21.15.16:0/961139409 wait complete. 2024-08-28T19:41:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dd42c2740 20 client.29185 trim_cache size 0 max 16384 2024-08-28T19:41:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:41:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/3829451462 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9db00b5100 msgr2=0x7f9db00b7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dcdd4d640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9db00b5100 0x7f9db00b7610 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0010670 tx=0x7f9dc0004a40 comp rx=0 tx=0).stop 2024-08-28T19:41:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/3829451462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9dc8102800 msgr2=0x7f9dc81a1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.111+0000 7f9dcdd4d640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9dc8102800 0x7f9dc81a1fe0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9db802a930 tx=0x7f9db8005080 comp rx=0 tx=0).stop 2024-08-28T19:41:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.112+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/3829451462 shutdown_connections 2024-08-28T19:41:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.112+0000 7f9dcdd4d640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc8108a90 0x7f9dc81a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.112+0000 7f9dcdd4d640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9db00b5100 0x7f9db00b7610 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.112+0000 7f9dcdd4d640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dc8108140 0x7f9dc81a2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.112+0000 7f9dcdd4d640 1 --2- 172.21.15.16:0/3829451462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9dc8102800 0x7f9dc81a1fe0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.112+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/3829451462 >> 172.21.15.16:0/3829451462 conn(0x7f9dc80794d0 msgr2=0x7f9dc8101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.112+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/3829451462 shutdown_connections 2024-08-28T19:41:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:26.112+0000 7f9dcdd4d640 1 -- 172.21.15.16:0/3829451462 wait complete. 2024-08-28T19:41:26.488 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163498 22.36 MB/sec execute 302 sec latency 7882.466 ms 2024-08-28T19:41:26.712 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:26 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1293028770' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:26.738 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id 9e1b6317 --release --await' 2024-08-28T19:41:26.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id 9e1b6317 --release --await 2024-08-28T19:41:26.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:26 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1293028770' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:26.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:26 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1293028770' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:27.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163498 22.28 MB/sec execute 303 sec latency 8884.824 ms 2024-08-28T19:41:27.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:27 smithi016 ceph-mon[18556]: pgmap v979: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 489 op/s 2024-08-28T19:41:27.710 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:27 smithi028 ceph-mon[21769]: pgmap v979: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 489 op/s 2024-08-28T19:41:27.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:27 smithi115 ceph-mon[22184]: pgmap v979: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 489 op/s 2024-08-28T19:41:28.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:41:28.079 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163498 22.21 MB/sec execute 304 sec latency 9884.907 ms 2024-08-28T19:41:28.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.625+0000 7f1c6630a640 1 -- 172.21.15.16:0/1992025289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c601024d0 msgr2=0x7f1c60110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:28.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.625+0000 7f1c6630a640 1 --2- 172.21.15.16:0/1992025289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c601024d0 0x7f1c60110300 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009ca0 tx=0x7f1c5802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.625+0000 7f1c6630a640 1 -- 172.21.15.16:0/1992025289 shutdown_connections 2024-08-28T19:41:28.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.625+0000 7f1c6630a640 1 --2- 172.21.15.16:0/1992025289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c601024d0 0x7f1c60110300 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.625+0000 7f1c6630a640 1 --2- 172.21.15.16:0/1992025289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c60101b80 0x7f1c60101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.625+0000 7f1c6630a640 1 --2- 172.21.15.16:0/1992025289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c60108490 0x7f1c601088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.625+0000 7f1c6630a640 1 -- 172.21.15.16:0/1992025289 >> 172.21.15.16:0/1992025289 conn(0x7f1c60077a00 msgr2=0x7f1c60077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:28.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.625+0000 7f1c6630a640 1 -- 172.21.15.16:0/1992025289 shutdown_connections 2024-08-28T19:41:28.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.625+0000 7f1c6630a640 1 -- 172.21.15.16:0/1992025289 wait complete. 2024-08-28T19:41:28.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c6630a640 1 Processor -- start 2024-08-28T19:41:28.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c6630a640 1 -- start start 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c6630a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c60101b80 0x7f1c6019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c6630a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c601024d0 0x7f1c6019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c6630a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c60108490 0x7f1c6019fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c6630a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c60113840 con 0x7f1c60101b80 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c6630a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c601136c0 con 0x7f1c601024d0 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c6630a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1c601139c0 con 0x7f1c60108490 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c60101b80 0x7f1c6019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c60101b80 0x7f1c6019a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44562/0 (socket says 172.21.15.16:44562) 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c5ffff640 1 -- 172.21.15.16:0/3826530775 learned_addr learned my addr 172.21.15.16:0/3826530775 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c601024d0 0x7f1c6019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.626+0000 7f1c648a7640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c60108490 0x7f1c6019fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.627+0000 7f1c5ffff640 1 -- 172.21.15.16:0/3826530775 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c60108490 msgr2=0x7f1c6019fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:28.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.627+0000 7f1c5ffff640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c60108490 0x7f1c6019fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.627+0000 7f1c5ffff640 1 -- 172.21.15.16:0/3826530775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c601024d0 msgr2=0x7f1c6019a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:28.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.627+0000 7f1c5ffff640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c601024d0 0x7f1c6019a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.627+0000 7f1c5ffff640 1 -- 172.21.15.16:0/3826530775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c601a01c0 con 0x7f1c60101b80 2024-08-28T19:41:28.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.627+0000 7f1c5ffff640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c60101b80 0x7f1c6019a250 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f1c5000e9f0 tx=0x7f1c5000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:28.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.627+0000 7f1c5d7fa640 1 -- 172.21.15.16:0/3826530775 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5000cd80 con 0x7f1c60101b80 2024-08-28T19:41:28.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.627+0000 7f1c6630a640 1 -- 172.21.15.16:0/3826530775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c601a04b0 con 0x7f1c60101b80 2024-08-28T19:41:28.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.627+0000 7f1c5d7fa640 1 -- 172.21.15.16:0/3826530775 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c500054d0 con 0x7f1c60101b80 2024-08-28T19:41:28.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.627+0000 7f1c6630a640 1 -- 172.21.15.16:0/3826530775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c601a6c10 con 0x7f1c60101b80 2024-08-28T19:41:28.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.627+0000 7f1c5d7fa640 1 -- 172.21.15.16:0/3826530775 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c50010690 con 0x7f1c60101b80 2024-08-28T19:41:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.630+0000 7f1c5d7fa640 1 -- 172.21.15.16:0/3826530775 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1c50010830 con 0x7f1c60101b80 2024-08-28T19:41:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.630+0000 7f1c6c87f740 1 Processor -- start 2024-08-28T19:41:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.630+0000 7f1c6c87f740 1 -- start start 2024-08-28T19:41:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a75a520 0x562d5aaffc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a731660 0x562d5ab001c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5ab00700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562d5aa76a70 con 0x562d5a739840 2024-08-28T19:41:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d5aa66000 con 0x562d5a75a520 2024-08-28T19:41:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562d5aa69fb0 con 0x562d5a731660 2024-08-28T19:41:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c5f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5ab00700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c648a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a75a520 0x562d5aaffc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c648a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a75a520 0x562d5aaffc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51062/0 (socket says 172.21.15.16:51062) 2024-08-28T19:41:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c5f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5ab00700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44568/0 (socket says 172.21.15.16:44568) 2024-08-28T19:41:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c648a7640 1 -- 172.21.15.16:0/728093943 learned_addr learned my addr 172.21.15.16:0/728093943 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c5ffff640 1 --2- 172.21.15.16:0/728093943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a731660 0x562d5ab001c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c648a7640 1 -- 172.21.15.16:0/728093943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a731660 msgr2=0x562d5ab001c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c648a7640 1 --2- 172.21.15.16:0/728093943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a731660 0x562d5ab001c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c648a7640 1 -- 172.21.15.16:0/728093943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 msgr2=0x562d5ab00700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c648a7640 1 --2- 172.21.15.16:0/728093943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5ab00700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c648a7640 1 -- 172.21.15.16:0/728093943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d5a7c5b00 con 0x562d5a75a520 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c5d7fa640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c480b4ca0 0x7f1c480b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c5ffff640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c480b4ca0 0x7f1c480b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.631+0000 7f1c648a7640 1 --2- 172.21.15.16:0/728093943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a75a520 0x562d5aaffc80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1c5802f410 tx=0x7f1c5802f440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f244573d80697aeb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.632+0000 7f1c5d7fa640 1 -- 172.21.15.16:0/3826530775 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f1c500e5cc0 con 0x7f1c60101b80 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.632+0000 7f1c467fc640 1 -- 172.21.15.16:0/728093943 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58031cf0 con 0x562d5a75a520 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.632+0000 7f1c467fc640 1 -- 172.21.15.16:0/728093943 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c58036030 con 0x562d5a75a520 2024-08-28T19:41:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.632+0000 7f1c467fc640 1 -- 172.21.15.16:0/728093943 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c580042a0 con 0x562d5a75a520 2024-08-28T19:41:28.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.633+0000 7f1c6c87f740 1 -- 172.21.15.16:0/728093943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a75a520 msgr2=0x562d5aaffc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:28.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.633+0000 7f1c5ffff640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c480b4ca0 0x7f1c480b71b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1c500e30f0 tx=0x7f1c50004a90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:28.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.633+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/728093943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a75a520 0x562d5aaffc80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1c5802f410 tx=0x7f1c5802f440 comp rx=0 tx=0).stop 2024-08-28T19:41:28.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.633+0000 7f1c6c87f740 1 -- 172.21.15.16:0/728093943 shutdown_connections 2024-08-28T19:41:28.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.633+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/728093943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5ab00700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.633+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/728093943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a731660 0x562d5ab001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.633+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/728093943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a75a520 0x562d5aaffc80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.633+0000 7f1c6c87f740 1 -- 172.21.15.16:0/728093943 >> 172.21.15.16:0/728093943 conn(0x562d5a74e910 msgr2=0x562d5aafdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.633+0000 7f1c6c87f740 1 -- 172.21.15.16:0/728093943 shutdown_connections 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.633+0000 7f1c6c87f740 1 -- 172.21.15.16:0/728093943 wait complete. 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.634+0000 7f1c6c87f740 1 -- start start 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.634+0000 7f1c6c87f740 1 Processor -- start 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.634+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a731660 0x562d5ab0fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.634+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a739840 0x562d5ab10120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.634+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a75a520 0x562d5ab1ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.634+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562d5ab125b0 con 0x562d5a731660 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.634+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d5aa27520 con 0x562d5a739840 2024-08-28T19:41:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.634+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562d5ab12730 con 0x562d5a75a520 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.634+0000 7f1c6c87f740 10 client.0 _ll_register_callbacks cb 0x562d5aa5fc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.634+0000 7f1c648a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a731660 0x562d5ab0fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.634+0000 7f1c648a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a731660 0x562d5ab0fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44580/0 (socket says 172.21.15.16:44580) 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c648a7640 1 -- 172.21.15.16:0/4032021924 learned_addr learned my addr 172.21.15.16:0/4032021924 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a739840 0x562d5ab10120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c648a7640 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a75a520 0x562d5ab1ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c648a7640 1 -- 172.21.15.16:0/4032021924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a75a520 msgr2=0x562d5ab1ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c648a7640 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a75a520 0x562d5ab1ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c648a7640 1 -- 172.21.15.16:0/4032021924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a739840 msgr2=0x562d5ab10120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c648a7640 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a739840 0x562d5ab10120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c648a7640 1 -- 172.21.15.16:0/4032021924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d5a83c960 con 0x562d5a731660 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a739840 0x562d5ab10120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c648a7640 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a731660 0x562d5ab0fbe0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f1c58005c90 tx=0x7f1c58003c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c5126b596466c70 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c3f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/4032021924 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58057060 con 0x562d5a731660 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c6c87f740 1 -- 172.21.15.16:0/4032021924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562d5aa059a0 con 0x562d5a731660 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.635+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/4032021924 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f1c58031890 con 0x562d5a731660 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/4032021924 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f1c58059730 con 0x562d5a731660 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c6c87f740 10 client.20181 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c6c87f740 1 -- 172.21.15.16:0/4032021924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x562d5ab125b0 con 0x562d5a731660 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/4032021924 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562d5aa059a0 con 0x562d5a731660 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/4032021924 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f1c58043070 con 0x562d5a731660 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c6c87f740 10 client.20181 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c6c87f740 10 client.20181 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c6c87f740 20 client.20181 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d5a718ea0 0x562d5ab21980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c6c87f740 4 client.20181 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:41:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c6c87f740 1 -- 172.21.15.16:0/4032021924 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d5aa789c0 con 0x562d5a718ea0 2024-08-28T19:41:28.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.636+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d5a718ea0 0x562d5ab21980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.637+0000 7f1c3f7fe640 10 client.20181 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.637+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d5a718ea0 0x562d5ab21980 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4408c560278a7a22 server_cookie=5a1de10cd6ce17cb in_seq=0 out_seq=0 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/4032021924 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562d5aa789c0 con 0x562d5a718ea0 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c3f7fe640 10 client.20181 handle_command_reply: tid=0 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c6c87f740 1 client.20181 shutdown 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c6c87f740 1 -- 172.21.15.16:0/4032021924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a731660 msgr2=0x562d5ab0fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a731660 0x562d5ab0fbe0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f1c58005c90 tx=0x7f1c58003c90 comp rx=0 tx=0).stop 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c6c87f740 1 -- 172.21.15.16:0/4032021924 shutdown_connections 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d5a718ea0 0x562d5ab21980 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a75a520 0x562d5ab1ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c3f7fe640 0 client.20181 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a739840 0x562d5ab10120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/4032021924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a731660 0x562d5ab0fbe0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c6c87f740 1 -- 172.21.15.16:0/4032021924 >> 172.21.15.16:0/4032021924 conn(0x562d5a74e910 msgr2=0x562d5aafc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c6c87f740 1 -- 172.21.15.16:0/4032021924 shutdown_connections 2024-08-28T19:41:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.638+0000 7f1c6c87f740 1 -- 172.21.15.16:0/4032021924 wait complete. 2024-08-28T19:41:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.639+0000 7f1c6c87f740 20 client.20181 trim_cache size 0 max 16384 2024-08-28T19:41:28.654 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c6c87f740 1 Processor -- start 2024-08-28T19:41:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c6c87f740 1 -- start start 2024-08-28T19:41:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a718ea0 0x562d5ab1b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a731660 0x562d5ab1bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5aafc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c601aeab0 con 0x562d5a739840 2024-08-28T19:41:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c601b0770 con 0x562d5a731660 2024-08-28T19:41:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1c601af210 con 0x562d5a718ea0 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c648a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5aafc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c648a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5aafc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44592/0 (socket says 172.21.15.16:44592) 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c648a7640 1 -- 172.21.15.16:0/1915067123 learned_addr learned my addr 172.21.15.16:0/1915067123 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c648a7640 1 --2- 172.21.15.16:0/1915067123 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a718ea0 0x562d5ab1b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.658+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/1915067123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a731660 0x562d5ab1bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.659+0000 7f1c648a7640 1 -- 172.21.15.16:0/1915067123 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a718ea0 msgr2=0x562d5ab1b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.659+0000 7f1c648a7640 1 --2- 172.21.15.16:0/1915067123 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a718ea0 0x562d5ab1b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.659+0000 7f1c648a7640 1 -- 172.21.15.16:0/1915067123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a731660 msgr2=0x562d5ab1bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.659+0000 7f1c648a7640 1 --2- 172.21.15.16:0/1915067123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a731660 0x562d5ab1bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.659+0000 7f1c648a7640 1 -- 172.21.15.16:0/1915067123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d5ab1c410 con 0x562d5a739840 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.659+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/1915067123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a731660 0x562d5ab1bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.659+0000 7f1c648a7640 1 --2- 172.21.15.16:0/1915067123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5aafc150 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006010 tx=0x7f1c5802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e2abe3697bd0bb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.659+0000 7f1c45ffb640 1 -- 172.21.15.16:0/1915067123 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58057040 con 0x562d5a739840 2024-08-28T19:41:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.659+0000 7f1c45ffb640 1 -- 172.21.15.16:0/1915067123 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c58036070 con 0x562d5a739840 2024-08-28T19:41:28.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.659+0000 7f1c45ffb640 1 -- 172.21.15.16:0/1915067123 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58031690 con 0x562d5a739840 2024-08-28T19:41:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.660+0000 7f1c6c87f740 1 -- 172.21.15.16:0/1915067123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 msgr2=0x562d5aafc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.660+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/1915067123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5aafc150 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006010 tx=0x7f1c5802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.660+0000 7f1c6c87f740 1 -- 172.21.15.16:0/1915067123 shutdown_connections 2024-08-28T19:41:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.660+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/1915067123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5aafc150 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.660+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/1915067123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a731660 0x562d5ab1bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.660+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/1915067123 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a718ea0 0x562d5ab1b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.660+0000 7f1c6c87f740 1 -- 172.21.15.16:0/1915067123 >> 172.21.15.16:0/1915067123 conn(0x562d5a74e910 msgr2=0x562d5ab0e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.660+0000 7f1c6c87f740 1 -- 172.21.15.16:0/1915067123 shutdown_connections 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.660+0000 7f1c6c87f740 1 -- 172.21.15.16:0/1915067123 wait complete. 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.661+0000 7f1c6c87f740 1 -- start start 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.661+0000 7f1c6c87f740 1 Processor -- start 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.661+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a718ea0 0x562d5ab17a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.661+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a731660 0x562d5ab17f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.661+0000 7f1c6c87f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5ab129b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.661+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c601affe0 con 0x562d5a739840 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.661+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c601af8e0 con 0x562d5a731660 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.661+0000 7f1c6c87f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1c601acab0 con 0x562d5a718ea0 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.661+0000 7f1c6c87f740 10 client.0 _ll_register_callbacks cb 0x7f1c540097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c648a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5ab129b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c648a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5ab129b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44596/0 (socket says 172.21.15.16:44596) 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c648a7640 1 -- 172.21.15.16:0/2647233938 learned_addr learned my addr 172.21.15.16:0/2647233938 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a731660 0x562d5ab17f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c648a7640 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a718ea0 0x562d5ab17a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c648a7640 1 -- 172.21.15.16:0/2647233938 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a718ea0 msgr2=0x562d5ab17a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c648a7640 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a718ea0 0x562d5ab17a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c648a7640 1 -- 172.21.15.16:0/2647233938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a731660 msgr2=0x562d5ab17f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c648a7640 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a731660 0x562d5ab17f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c648a7640 1 -- 172.21.15.16:0/2647233938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d5ab18530 con 0x562d5a739840 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c648a7640 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5ab129b0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f1c580308f0 tx=0x7f1c5802f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbe20ff57d9a0c0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c3f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/2647233938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58057070 con 0x562d5a739840 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/2647233938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f1c58030b70 con 0x562d5a739840 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.662+0000 7f1c6c87f740 1 -- 172.21.15.16:0/2647233938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c58036070 con 0x562d5a739840 2024-08-28T19:41:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/2647233938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f1c58030d70 con 0x562d5a739840 2024-08-28T19:41:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c6c87f740 10 client.20187 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c6c87f740 1 -- 172.21.15.16:0/2647233938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x562d5ab8c6a0 con 0x562d5a739840 2024-08-28T19:41:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/2647233938 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c5804b040 con 0x562d5a739840 2024-08-28T19:41:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/2647233938 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7f1c58043070 con 0x562d5a739840 2024-08-28T19:41:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c6c87f740 10 client.20187 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c6c87f740 10 client.20187 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:41:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c6c87f740 20 client.20187 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d5a75a520 0x562d5ab8eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c6c87f740 4 client.20187 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9e1b6317", "release": true, "await": true}] 2024-08-28T19:41:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c6c87f740 1 -- 172.21.15.16:0/2647233938 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9e1b6317", "release": true, "await": true}) -- 0x7f1c58036070 con 0x562d5a75a520 2024-08-28T19:41:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.663+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d5a75a520 0x562d5ab8eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.664+0000 7f1c3f7fe640 10 client.20187 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.664+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d5a75a520 0x562d5ab8eee0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aceac0579c4b1e5a server_cookie=9eefe2cbb268618b in_seq=0 out_seq=0 2024-08-28T19:41:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c3f7fe640 1 -- 172.21.15.16:0/2647233938 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+592 (crc 0 0 0) 0x7f1c58036070 con 0x562d5a75a520 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c3f7fe640 10 client.20187 handle_command_reply: tid=0 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 1 client.20187 shutdown 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 1 -- 172.21.15.16:0/2647233938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 msgr2=0x562d5ab129b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5ab129b0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f1c580308f0 tx=0x7f1c5802f510 comp rx=0 tx=0).stop 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 1 -- 172.21.15.16:0/2647233938 shutdown_connections 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d5a739840 0x562d5ab129b0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d5a731660 0x562d5ab17f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562d5a75a520 0x562d5ab8eee0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 1 --2- 172.21.15.16:0/2647233938 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d5a718ea0 0x562d5ab17a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 1 -- 172.21.15.16:0/2647233938 >> 172.21.15.16:0/2647233938 conn(0x562d5a74e910 msgr2=0x562d5aafc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c3f7fe640 0 client.20187 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 1 -- 172.21.15.16:0/2647233938 shutdown_connections 2024-08-28T19:41:28.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 1 -- 172.21.15.16:0/2647233938 wait complete. 2024-08-28T19:41:28.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.699+0000 7f1c6c87f740 20 client.20187 trim_cache size 0 max 16384 2024-08-28T19:41:28.702 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:41:28.702 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 101, 2024-08-28T19:41:28.702 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:41:28.702 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 64, 2024-08-28T19:41:28.702 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:41:28.702 INFO:teuthology.orchestra.run.smithi016.stdout: "9e1b6317": { 2024-08-28T19:41:28.702 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 64, 2024-08-28T19:41:28.702 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 130.0, 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:28.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:28.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:28.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:28.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:28.704 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:41:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 -- 172.21.15.16:0/3826530775 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c480b4ca0 msgr2=0x7f1c480b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c480b4ca0 0x7f1c480b71b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1c500e30f0 tx=0x7f1c50004a90 comp rx=0 tx=0).stop 2024-08-28T19:41:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 -- 172.21.15.16:0/3826530775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c60101b80 msgr2=0x7f1c6019a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c60101b80 0x7f1c6019a250 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f1c5000e9f0 tx=0x7f1c5000edf0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 -- 172.21.15.16:0/3826530775 shutdown_connections 2024-08-28T19:41:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c60108490 0x7f1c6019fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c480b4ca0 0x7f1c480b71b0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c601024d0 0x7f1c6019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 --2- 172.21.15.16:0/3826530775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c60101b80 0x7f1c6019a250 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 -- 172.21.15.16:0/3826530775 >> 172.21.15.16:0/3826530775 conn(0x7f1c60077a00 msgr2=0x7f1c60106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 -- 172.21.15.16:0/3826530775 shutdown_connections 2024-08-28T19:41:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:28.700+0000 7f1c6630a640 1 -- 172.21.15.16:0/3826530775 wait complete. 2024-08-28T19:41:29.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163912 22.21 MB/sec execute 305 sec latency 10092.594 ms 2024-08-28T19:41:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:29 smithi016 ceph-mon[18556]: pgmap v980: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.0 MiB/s wr, 361 op/s 2024-08-28T19:41:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:29 smithi028 ceph-mon[21769]: pgmap v980: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.0 MiB/s wr, 361 op/s 2024-08-28T19:41:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:29 smithi115 ceph-mon[22184]: pgmap v980: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.0 MiB/s wr, 361 op/s 2024-08-28T19:41:29.900 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '9e1b6317', total seconds elapsed: 16.7 2024-08-28T19:41:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.086+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/2701779092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c8102800 msgr2=0x7fd6c8102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.086+0000 7fd6cd3a6640 1 --2- 172.21.15.16:0/2701779092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c8102800 0x7fd6c8102c10 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009900 tx=0x7fd6b802a410 comp rx=0 tx=0).stop 2024-08-28T19:41:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.086+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/2701779092 shutdown_connections 2024-08-28T19:41:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.086+0000 7fd6cd3a6640 1 --2- 172.21.15.16:0/2701779092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6c8108a90 0x7fd6c810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.086+0000 7fd6cd3a6640 1 --2- 172.21.15.16:0/2701779092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c8108140 0x7fd6c8108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.086+0000 7fd6cd3a6640 1 --2- 172.21.15.16:0/2701779092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c8102800 0x7fd6c8102c10 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.086+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/2701779092 >> 172.21.15.16:0/2701779092 conn(0x7fd6c80794d0 msgr2=0x7fd6c80798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.086+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/2701779092 shutdown_connections 2024-08-28T19:41:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.086+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/2701779092 wait complete. 2024-08-28T19:41:30.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.087+0000 7fd6cd3a6640 1 Processor -- start 2024-08-28T19:41:30.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.087+0000 7fd6cd3a6640 1 -- start start 2024-08-28T19:41:30.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6cd3a6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6c8102800 0x7fd6c819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6cd3a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c8108140 0x7fd6c819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6cd3a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c8108a90 0x7fd6c819fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6cd3a6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c8112710 con 0x7fd6c8108140 2024-08-28T19:41:30.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6cd3a6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c8112590 con 0x7fd6c8108a90 2024-08-28T19:41:30.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6cd3a6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c8112890 con 0x7fd6c8102800 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c8108140 0x7fd6c819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c8108140 0x7fd6c819a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44610/0 (socket says 172.21.15.16:44610) 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6bffff640 1 -- 172.21.15.16:0/93734129 learned_addr learned my addr 172.21.15.16:0/93734129 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6c77fe640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c8108a90 0x7fd6c819fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6c6ffd640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6c8102800 0x7fd6c819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6bffff640 1 -- 172.21.15.16:0/93734129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6c8102800 msgr2=0x7fd6c819a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6bffff640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6c8102800 0x7fd6c819a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6bffff640 1 -- 172.21.15.16:0/93734129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c8108a90 msgr2=0x7fd6c819fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6bffff640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c8108a90 0x7fd6c819fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.088+0000 7fd6bffff640 1 -- 172.21.15.16:0/93734129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c81a01a0 con 0x7fd6c8108140 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.089+0000 7fd6bffff640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c8108140 0x7fd6c819a7d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000e9c0 tx=0x7fd6b000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.089+0000 7fd6c4ff9640 1 -- 172.21.15.16:0/93734129 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b000cd80 con 0x7fd6c8108140 2024-08-28T19:41:30.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.089+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/93734129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c81a0490 con 0x7fd6c8108140 2024-08-28T19:41:30.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.089+0000 7fd6c4ff9640 1 -- 172.21.15.16:0/93734129 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6b00054d0 con 0x7fd6c8108140 2024-08-28T19:41:30.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.089+0000 7fd6c4ff9640 1 -- 172.21.15.16:0/93734129 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0010430 con 0x7fd6c8108140 2024-08-28T19:41:30.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.089+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/93734129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c81a6bf0 con 0x7fd6c8108140 2024-08-28T19:41:30.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.092+0000 7fd6d391b740 1 Processor -- start 2024-08-28T19:41:30.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.092+0000 7fd6d391b740 1 -- start start 2024-08-28T19:41:30.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.092+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 0x564254ac8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.092+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5642546fa1b0 0x564254ac8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.092+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254702390 0x564254ac8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.092+0000 7fd6d391b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564254a3f680 con 0x564254723070 2024-08-28T19:41:30.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.092+0000 7fd6d391b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564254a2ec10 con 0x564254702390 2024-08-28T19:41:30.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.092+0000 7fd6d391b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564254a32bc0 con 0x5642546fa1b0 2024-08-28T19:41:30.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.092+0000 7fd6c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 0x564254ac8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.092+0000 7fd6c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 0x564254ac8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44624/0 (socket says 172.21.15.16:44624) 2024-08-28T19:41:30.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.092+0000 7fd6c77fe640 1 -- 172.21.15.16:0/2384484871 learned_addr learned my addr 172.21.15.16:0/2384484871 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:30.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.093+0000 7fd6c6ffd640 1 --2- 172.21.15.16:0/2384484871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5642546fa1b0 0x564254ac8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.093+0000 7fd6c77fe640 1 -- 172.21.15.16:0/2384484871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5642546fa1b0 msgr2=0x564254ac8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:30.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.093+0000 7fd6c77fe640 1 --2- 172.21.15.16:0/2384484871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5642546fa1b0 0x564254ac8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.093+0000 7fd6c77fe640 1 -- 172.21.15.16:0/2384484871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254702390 msgr2=0x564254ac8d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:41:30.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.093+0000 7fd6c77fe640 1 --2- 172.21.15.16:0/2384484871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254702390 0x564254ac8d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.093+0000 7fd6c77fe640 1 -- 172.21.15.16:0/2384484871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56425478e650 con 0x564254723070 2024-08-28T19:41:30.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.093+0000 7fd6c4ff9640 1 -- 172.21.15.16:0/93734129 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fd6b00105d0 con 0x7fd6c8108140 2024-08-28T19:41:30.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.093+0000 7fd6c77fe640 1 --2- 172.21.15.16:0/2384484871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 0x564254ac8310 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009fd0 tx=0x7fd6c00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f085f9121542a94 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:30.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.093+0000 7fd6bd7fa640 1 -- 172.21.15.16:0/2384484871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0004d20 con 0x564254723070 2024-08-28T19:41:30.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.093+0000 7fd6bd7fa640 1 -- 172.21.15.16:0/2384484871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c0004050 con 0x564254723070 2024-08-28T19:41:30.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.093+0000 7fd6bd7fa640 1 -- 172.21.15.16:0/2384484871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0005280 con 0x564254723070 2024-08-28T19:41:30.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.094+0000 7fd6c4ff9640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd6b40b4ca0 0x7fd6b40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.094+0000 7fd6c6ffd640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd6b40b4ca0 0x7fd6b40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.094+0000 7fd6c4ff9640 1 -- 172.21.15.16:0/93734129 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7fd6b00e5d80 con 0x7fd6c8108140 2024-08-28T19:41:30.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.095+0000 7fd6d391b740 1 -- 172.21.15.16:0/2384484871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 msgr2=0x564254ac8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:30.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.095+0000 7fd6d391b740 1 --2- 172.21.15.16:0/2384484871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 0x564254ac8310 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009fd0 tx=0x7fd6c00118d0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.095+0000 7fd6d391b740 1 -- 172.21.15.16:0/2384484871 shutdown_connections 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.095+0000 7fd6d391b740 1 --2- 172.21.15.16:0/2384484871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254702390 0x564254ac8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.095+0000 7fd6d391b740 1 --2- 172.21.15.16:0/2384484871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5642546fa1b0 0x564254ac8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.095+0000 7fd6d391b740 1 --2- 172.21.15.16:0/2384484871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 0x564254ac8310 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.095+0000 7fd6d391b740 1 -- 172.21.15.16:0/2384484871 >> 172.21.15.16:0/2384484871 conn(0x564254717460 msgr2=0x564254ac6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.095+0000 7fd6d391b740 1 -- 172.21.15.16:0/2384484871 shutdown_connections 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.095+0000 7fd6d391b740 1 -- 172.21.15.16:0/2384484871 wait complete. 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.095+0000 7fd6c6ffd640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd6b40b4ca0 0x7fd6b40b71b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8031440 tx=0x7fd6b8009560 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.095+0000 7fd6d391b740 1 -- start start 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6d391b740 1 Processor -- start 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5642546fa1b0 0x564254acfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254702390 0x564254acffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254723070 0x564254ae85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6d391b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564254891700 con 0x564254702390 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6d391b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5642549b5750 con 0x564254723070 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6d391b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564254adf280 con 0x5642546fa1b0 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6d391b740 10 client.0 _ll_register_callbacks cb 0x5642549e4e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6c6ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5642546fa1b0 0x564254acfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254702390 0x564254acffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6c6ffd640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5642546fa1b0 0x564254acfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:53098/0 (socket says 172.21.15.16:53098) 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6c77fe640 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254723070 0x564254ae85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.096+0000 7fd6c6ffd640 1 -- 172.21.15.16:0/4052424941 learned_addr learned my addr 172.21.15.16:0/4052424941 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.097+0000 7fd6bffff640 1 -- 172.21.15.16:0/4052424941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5642546fa1b0 msgr2=0x564254acfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.097+0000 7fd6bffff640 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5642546fa1b0 0x564254acfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.097+0000 7fd6bffff640 1 -- 172.21.15.16:0/4052424941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254723070 msgr2=0x564254ae85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.097+0000 7fd6bffff640 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254723070 0x564254ae85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.097+0000 7fd6bffff640 1 -- 172.21.15.16:0/4052424941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642549ee900 con 0x564254702390 2024-08-28T19:41:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.097+0000 7fd6bffff640 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254702390 0x564254acffb0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00e3100 tx=0x7fd6b00ab9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2cacbfd7504745d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.097+0000 7fd6ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.097+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/4052424941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b00abbb0 con 0x564254702390 2024-08-28T19:41:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.097+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/4052424941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7fd6b00a8690 con 0x564254702390 2024-08-28T19:41:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.097+0000 7fd6d391b740 1 -- 172.21.15.16:0/4052424941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6c0004050 con 0x564254702390 2024-08-28T19:41:30.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.097+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/4052424941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fd6b00ac7a0 con 0x564254702390 2024-08-28T19:41:30.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.098+0000 7fd6d391b740 10 client.20199 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:30.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.098+0000 7fd6d391b740 1 -- 172.21.15.16:0/4052424941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x564254aeafd0 con 0x564254702390 2024-08-28T19:41:30.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.098+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/4052424941 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c0004050 con 0x564254702390 2024-08-28T19:41:30.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.098+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/4052424941 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fd6b00a7ba0 con 0x564254702390 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.098+0000 7fd6d391b740 10 client.20199 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.098+0000 7fd6d391b740 10 client.20199 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.098+0000 7fd6d391b740 20 client.20199 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.098+0000 7fd6d391b740 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5642546e19f0 0x564254aee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.098+0000 7fd6d391b740 4 client.20199 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.098+0000 7fd6d391b740 1 -- 172.21.15.16:0/4052424941 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564254a415d0 con 0x5642546e19f0 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.098+0000 7fd6c6ffd640 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5642546e19f0 0x564254aee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.099+0000 7fd6ae7fc640 10 client.20199 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.099+0000 7fd6c6ffd640 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5642546e19f0 0x564254aee5b0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1b4978507c0430a server_cookie=90272791b49343f7 in_seq=0 out_seq=0 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.100+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/4052424941 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564254a415d0 con 0x5642546e19f0 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.100+0000 7fd6ae7fc640 10 client.20199 handle_command_reply: tid=0 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.100+0000 7fd6d391b740 1 client.20199 shutdown 2024-08-28T19:41:30.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.100+0000 7fd6d391b740 1 -- 172.21.15.16:0/4052424941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254702390 msgr2=0x564254acffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.100+0000 7fd6d391b740 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254702390 0x564254acffb0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00e3100 tx=0x7fd6b00ab9f0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.100+0000 7fd6d391b740 1 -- 172.21.15.16:0/4052424941 shutdown_connections 2024-08-28T19:41:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.101+0000 7fd6d391b740 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254723070 0x564254ae85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.101+0000 7fd6d391b740 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254702390 0x564254acffb0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.101+0000 7fd6d391b740 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5642546e19f0 0x564254aee5b0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.101+0000 7fd6d391b740 1 --2- 172.21.15.16:0/4052424941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5642546fa1b0 0x564254acfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.101+0000 7fd6ae7fc640 0 client.20199 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.101+0000 7fd6d391b740 1 -- 172.21.15.16:0/4052424941 >> 172.21.15.16:0/4052424941 conn(0x564254717460 msgr2=0x564254ac47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.101+0000 7fd6d391b740 1 -- 172.21.15.16:0/4052424941 shutdown_connections 2024-08-28T19:41:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.101+0000 7fd6d391b740 1 -- 172.21.15.16:0/4052424941 wait complete. 2024-08-28T19:41:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.101+0000 7fd6d391b740 20 client.20199 trim_cache size 0 max 16384 2024-08-28T19:41:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.119+0000 7fd6d391b740 1 Processor -- start 2024-08-28T19:41:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.119+0000 7fd6d391b740 1 -- start start 2024-08-28T19:41:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.119+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642546fa1b0 0x564254ad16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.119+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254702390 0x564254accc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.119+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254723070 0x564254acd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.119+0000 7fd6d391b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81aeb00 con 0x5642546fa1b0 2024-08-28T19:41:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.119+0000 7fd6d391b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81b07c0 con 0x564254702390 2024-08-28T19:41:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.119+0000 7fd6d391b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81af260 con 0x564254723070 2024-08-28T19:41:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.120+0000 7fd6bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254702390 0x564254accc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.120+0000 7fd6c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642546fa1b0 0x564254ad16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.120+0000 7fd6bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254702390 0x564254accc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51138/0 (socket says 172.21.15.16:51138) 2024-08-28T19:41:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.120+0000 7fd6c77fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254723070 0x564254acd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.120+0000 7fd6bffff640 1 -- 172.21.15.16:0/4040305121 learned_addr learned my addr 172.21.15.16:0/4040305121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.120+0000 7fd6bffff640 1 -- 172.21.15.16:0/4040305121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254723070 msgr2=0x564254acd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.120+0000 7fd6bffff640 1 --2- 172.21.15.16:0/4040305121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254723070 0x564254acd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.120+0000 7fd6bffff640 1 -- 172.21.15.16:0/4040305121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642546fa1b0 msgr2=0x564254ad16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.120+0000 7fd6bffff640 1 --2- 172.21.15.16:0/4040305121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642546fa1b0 0x564254ad16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.120+0000 7fd6bffff640 1 -- 172.21.15.16:0/4040305121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564254acdf70 con 0x564254702390 2024-08-28T19:41:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.120+0000 7fd6bffff640 1 --2- 172.21.15.16:0/4040305121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254702390 0x564254accc60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000cfd0 tx=0x7fd6b00abb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=701e47ba6776e6a9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.121+0000 7fd6bcff9640 1 -- 172.21.15.16:0/4040305121 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b00abbb0 con 0x564254702390 2024-08-28T19:41:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.121+0000 7fd6bcff9640 1 -- 172.21.15.16:0/4040305121 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6b00d0020 con 0x564254702390 2024-08-28T19:41:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.121+0000 7fd6bcff9640 1 -- 172.21.15.16:0/4040305121 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b00b5140 con 0x564254702390 2024-08-28T19:41:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.122+0000 7fd6d391b740 1 -- 172.21.15.16:0/4040305121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254702390 msgr2=0x564254accc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.122+0000 7fd6d391b740 1 --2- 172.21.15.16:0/4040305121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254702390 0x564254accc60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000cfd0 tx=0x7fd6b00abb20 comp rx=0 tx=0).stop 2024-08-28T19:41:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.122+0000 7fd6d391b740 1 -- 172.21.15.16:0/4040305121 shutdown_connections 2024-08-28T19:41:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.122+0000 7fd6d391b740 1 --2- 172.21.15.16:0/4040305121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254723070 0x564254acd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.122+0000 7fd6d391b740 1 --2- 172.21.15.16:0/4040305121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564254702390 0x564254accc60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.122+0000 7fd6d391b740 1 --2- 172.21.15.16:0/4040305121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642546fa1b0 0x564254ad16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.122+0000 7fd6d391b740 1 -- 172.21.15.16:0/4040305121 >> 172.21.15.16:0/4040305121 conn(0x564254717460 msgr2=0x564254ad0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.122+0000 7fd6d391b740 1 -- 172.21.15.16:0/4040305121 shutdown_connections 2024-08-28T19:41:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.122+0000 7fd6d391b740 1 -- 172.21.15.16:0/4040305121 wait complete. 2024-08-28T19:41:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.123+0000 7fd6d391b740 1 -- start start 2024-08-28T19:41:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.123+0000 7fd6d391b740 1 Processor -- start 2024-08-28T19:41:30.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.123+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642546fa1b0 0x564254ae2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.123+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254702390 0x564254ae3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.123+0000 7fd6d391b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 0x564254b51fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.123+0000 7fd6d391b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81b0030 con 0x564254723070 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.123+0000 7fd6d391b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81af930 con 0x5642546fa1b0 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.123+0000 7fd6d391b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81acb00 con 0x564254702390 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6d391b740 10 client.0 _ll_register_callbacks cb 0x7fd6b000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6c6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642546fa1b0 0x564254ae2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6bffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254702390 0x564254ae3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 0x564254b51fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6bffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254702390 0x564254ae3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:53124/0 (socket says 172.21.15.16:53124) 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6bffff640 1 -- 172.21.15.16:0/972019269 learned_addr learned my addr 172.21.15.16:0/972019269 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6bffff640 1 -- 172.21.15.16:0/972019269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642546fa1b0 msgr2=0x564254ae2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6bffff640 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642546fa1b0 0x564254ae2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6bffff640 1 -- 172.21.15.16:0/972019269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 msgr2=0x564254b51fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6bffff640 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 0x564254b51fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6bffff640 1 -- 172.21.15.16:0/972019269 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564254b52610 con 0x564254702390 2024-08-28T19:41:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6c77fe640 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 0x564254b51fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6bffff640 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254702390 0x564254ae3460 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00a9310 tx=0x7fd6b000efb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b1a591a713700ac server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.124+0000 7fd6ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:41:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.125+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/972019269 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b00abbb0 con 0x564254702390 2024-08-28T19:41:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.125+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/972019269 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7fd6b00a8690 con 0x564254702390 2024-08-28T19:41:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.125+0000 7fd6d391b740 1 -- 172.21.15.16:0/972019269 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6b00d0020 con 0x564254702390 2024-08-28T19:41:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.126+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/972019269 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fd6b00b0910 con 0x564254702390 2024-08-28T19:41:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.126+0000 7fd6d391b740 10 client.29221 fetch_fsmap learned FSMap version 101 2024-08-28T19:41:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.126+0000 7fd6d391b740 1 -- 172.21.15.16:0/972019269 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x564254b59950 con 0x564254702390 2024-08-28T19:41:30.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.126+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/972019269 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6b00d0020 con 0x564254702390 2024-08-28T19:41:30.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.127+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/972019269 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 101) ==== 8402+0+0 (secure 0 0 0) 0x7fd6b0097080 con 0x564254702390 2024-08-28T19:41:30.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.127+0000 7fd6d391b740 10 client.29221 fetch_fsmap finished waiting for FSMap version 101 2024-08-28T19:41:30.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.127+0000 7fd6d391b740 10 client.29221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:41:30.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.127+0000 7fd6d391b740 20 client.29221 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:30.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.127+0000 7fd6d391b740 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5642546b6da0 0x564254b5bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:30.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.127+0000 7fd6d391b740 4 client.29221 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:41:30.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.127+0000 7fd6d391b740 1 -- 172.21.15.16:0/972019269 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564254a3d070 con 0x5642546b6da0 2024-08-28T19:41:30.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.127+0000 7fd6c6ffd640 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5642546b6da0 0x564254b5bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:30.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.133+0000 7fd6ae7fc640 10 client.29221 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:30.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.133+0000 7fd6c6ffd640 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5642546b6da0 0x564254b5bec0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd7c10f7428e5d99 server_cookie=5837e19e989cb467 in_seq=0 out_seq=0 2024-08-28T19:41:30.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6ae7fc640 1 -- 172.21.15.16:0/972019269 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564254a3d070 con 0x5642546b6da0 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6ae7fc640 10 client.29221 handle_command_reply: tid=0 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6d391b740 1 client.29221 shutdown 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6d391b740 1 -- 172.21.15.16:0/972019269 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254702390 msgr2=0x564254ae3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6d391b740 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254702390 0x564254ae3460 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00a9310 tx=0x7fd6b000efb0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6d391b740 1 -- 172.21.15.16:0/972019269 shutdown_connections 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6d391b740 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564254723070 0x564254b51fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6d391b740 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5642546b6da0 0x564254b5bec0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6d391b740 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564254702390 0x564254ae3460 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6d391b740 1 --2- 172.21.15.16:0/972019269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642546fa1b0 0x564254ae2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6ae7fc640 0 client.29221 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.136+0000 7fd6d391b740 1 -- 172.21.15.16:0/972019269 >> 172.21.15.16:0/972019269 conn(0x564254717460 msgr2=0x564254ad16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:30.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6d391b740 1 -- 172.21.15.16:0/972019269 shutdown_connections 2024-08-28T19:41:30.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6d391b740 1 -- 172.21.15.16:0/972019269 wait complete. 2024-08-28T19:41:30.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6d391b740 20 client.29221 trim_cache size 0 max 16384 2024-08-28T19:41:30.139 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/93734129 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd6b40b4ca0 msgr2=0x7fd6b40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6cd3a6640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd6b40b4ca0 0x7fd6b40b71b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8031440 tx=0x7fd6b8009560 comp rx=0 tx=0).stop 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/93734129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c8108140 msgr2=0x7fd6c819a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6cd3a6640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c8108140 0x7fd6c819a7d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000e9c0 tx=0x7fd6b000ee80 comp rx=0 tx=0).stop 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/93734129 shutdown_connections 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6cd3a6640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fd6b40b4ca0 0x7fd6b40b71b0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6cd3a6640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c8108a90 0x7fd6c819fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6cd3a6640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c8108140 0x7fd6c819a7d0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6cd3a6640 1 --2- 172.21.15.16:0/93734129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6c8102800 0x7fd6c819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.137+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/93734129 >> 172.21.15.16:0/93734129 conn(0x7fd6c80794d0 msgr2=0x7fd6c8101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.138+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/93734129 shutdown_connections 2024-08-28T19:41:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:30.138+0000 7fd6cd3a6640 1 -- 172.21.15.16:0/93734129 wait complete. 2024-08-28T19:41:30.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 164403 22.23 MB/sec execute 306 sec latency 96.549 ms 2024-08-28T19:41:31.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 164849 22.22 MB/sec execute 307 sec latency 100.399 ms 2024-08-28T19:41:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:31 smithi016 ceph-mon[18556]: pgmap v981: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 431 op/s 2024-08-28T19:41:31.795 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:41:31.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:41:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:31 smithi028 ceph-mon[21769]: pgmap v981: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 431 op/s 2024-08-28T19:41:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:31 smithi115 ceph-mon[22184]: pgmap v981: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 431 op/s 2024-08-28T19:41:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:32.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 165328 22.22 MB/sec execute 308 sec latency 118.757 ms 2024-08-28T19:41:33.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 165885 22.26 MB/sec execute 309 sec latency 137.332 ms 2024-08-28T19:41:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:33 smithi016 ceph-mon[18556]: pgmap v982: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 10 MiB/s wr, 438 op/s 2024-08-28T19:41:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:33 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:41:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:33 smithi028 ceph-mon[21769]: pgmap v982: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 10 MiB/s wr, 438 op/s 2024-08-28T19:41:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:33 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:41:33.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:33 smithi115 ceph-mon[22184]: pgmap v982: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 10 MiB/s wr, 438 op/s 2024-08-28T19:41:33.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:33 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:41:34.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.570+0000 7f97c8bc6640 1 -- 172.21.15.16:0/3177225232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4109bb0 msgr2=0x7f97c41103e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:34.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.570+0000 7f97c8bc6640 1 --2- 172.21.15.16:0/3177225232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4109bb0 0x7f97c41103e0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009ca0 tx=0x7f97bc02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.570+0000 7f97c8bc6640 1 -- 172.21.15.16:0/3177225232 shutdown_connections 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.570+0000 7f97c8bc6640 1 --2- 172.21.15.16:0/3177225232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4109bb0 0x7f97c41103e0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.570+0000 7f97c8bc6640 1 --2- 172.21.15.16:0/3177225232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c4109260 0x7f97c4109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.570+0000 7f97c8bc6640 1 --2- 172.21.15.16:0/3177225232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97c4103920 0x7f97c4103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.570+0000 7f97c8bc6640 1 -- 172.21.15.16:0/3177225232 >> 172.21.15.16:0/3177225232 conn(0x7f97c40ff350 msgr2=0x7f97c41017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.570+0000 7f97c8bc6640 1 -- 172.21.15.16:0/3177225232 shutdown_connections 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.570+0000 7f97c8bc6640 1 -- 172.21.15.16:0/3177225232 wait complete. 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c8bc6640 1 Processor -- start 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c8bc6640 1 -- start start 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c8bc6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97c4103920 0x7f97c419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c8bc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c4109260 0x7f97c419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c8bc6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4109bb0 0x7f97c419fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c8bc6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41138c0 con 0x7f97c4109bb0 2024-08-28T19:41:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c8bc6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4113740 con 0x7f97c4109260 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c8bc6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4113a40 con 0x7f97c4103920 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4109bb0 0x7f97c419fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c4109260 0x7f97c419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c259c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97c4103920 0x7f97c419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4109bb0 0x7f97c419fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44680/0 (socket says 172.21.15.16:44680) 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c4109260 0x7f97c419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:51160/0 (socket says 172.21.15.16:51160) 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c2d9d640 1 -- 172.21.15.16:0/143251371 learned_addr learned my addr 172.21.15.16:0/143251371 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c2d9d640 1 -- 172.21.15.16:0/143251371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97c4103920 msgr2=0x7f97c419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c2d9d640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97c4103920 0x7f97c419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c2d9d640 1 -- 172.21.15.16:0/143251371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c4109260 msgr2=0x7f97c419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c2d9d640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c4109260 0x7f97c419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97c2d9d640 1 -- 172.21.15.16:0/143251371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c41a0160 con 0x7f97c4109bb0 2024-08-28T19:41:34.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.571+0000 7f97bbfff640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c4109260 0x7f97c419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.572+0000 7f97c2d9d640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4109bb0 0x7f97c419fa30 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f97bc02f540 tx=0x7f97bc0316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.572+0000 7f97bb7fe640 1 -- 172.21.15.16:0/143251371 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc003b70 con 0x7f97c4109bb0 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.572+0000 7f97bb7fe640 1 -- 172.21.15.16:0/143251371 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97bc003d10 con 0x7f97c4109bb0 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.572+0000 7f97c8bc6640 1 -- 172.21.15.16:0/143251371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c41a03f0 con 0x7f97c4109bb0 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.572+0000 7f97bb7fe640 1 -- 172.21.15.16:0/143251371 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc02f990 con 0x7f97c4109bb0 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.572+0000 7f97c8bc6640 1 -- 172.21.15.16:0/143251371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c41a6bb0 con 0x7f97c4109bb0 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.574+0000 7f97cf13b740 1 Processor -- start 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97bb7fe640 1 -- 172.21.15.16:0/143251371 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f97bc02fb30 con 0x7f97c4109bb0 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97cf13b740 1 -- start start 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936e4350 0x563493a8a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936bb490 0x563493a8acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 0x563493a8b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97cf13b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563493a005f0 con 0x5634936c3670 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97cf13b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5634939efb80 con 0x5634936bb490 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97cf13b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5634939f3b30 con 0x5634936e4350 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 0x563493a8b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 0x563493a8b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44690/0 (socket says 172.21.15.16:44690) 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97bbfff640 1 -- 172.21.15.16:0/2849522395 learned_addr learned my addr 172.21.15.16:0/2849522395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.575+0000 7f97c259c640 1 --2- 172.21.15.16:0/2849522395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936bb490 0x563493a8acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97bbfff640 1 --2- 172.21.15.16:0/2849522395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936e4350 0x563493a8a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97bb7fe640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f97b00b4ca0 0x7f97b00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97c259c640 1 -- 172.21.15.16:0/2849522395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936e4350 msgr2=0x563493a8a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97c259c640 1 --2- 172.21.15.16:0/2849522395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936e4350 0x563493a8a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97c259c640 1 -- 172.21.15.16:0/2849522395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 msgr2=0x563493a8b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97c259c640 1 --2- 172.21.15.16:0/2849522395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 0x563493a8b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97c259c640 1 -- 172.21.15.16:0/2849522395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56349374f930 con 0x5634936bb490 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97bb7fe640 1 -- 172.21.15.16:0/143251371 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f97bc106af0 con 0x7f97c4109bb0 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97c2d9d640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f97b00b4ca0 0x7f97b00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97c259c640 1 --2- 172.21.15.16:0/2849522395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936bb490 0x563493a8acc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009c00 tx=0x7f97b4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=232b6221ddc2298e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97b8ff9640 1 -- 172.21.15.16:0/2849522395 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4004050 con 0x5634936bb490 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97b8ff9640 1 -- 172.21.15.16:0/2849522395 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97b40054d0 con 0x5634936bb490 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.576+0000 7f97b8ff9640 1 -- 172.21.15.16:0/2849522395 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4004ca0 con 0x5634936bb490 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.577+0000 7f97cf13b740 1 -- 172.21.15.16:0/2849522395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936bb490 msgr2=0x563493a8acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.577+0000 7f97c2d9d640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f97b00b4ca0 0x7f97b00b71b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f97bc004af0 tx=0x7f97bc02a8c0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.577+0000 7f97cf13b740 1 --2- 172.21.15.16:0/2849522395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936bb490 0x563493a8acc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009c00 tx=0x7f97b4011810 comp rx=0 tx=0).stop 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.577+0000 7f97cf13b740 1 -- 172.21.15.16:0/2849522395 shutdown_connections 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.577+0000 7f97cf13b740 1 --2- 172.21.15.16:0/2849522395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 0x563493a8b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.577+0000 7f97cf13b740 1 --2- 172.21.15.16:0/2849522395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936bb490 0x563493a8acc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.577+0000 7f97cf13b740 1 --2- 172.21.15.16:0/2849522395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936e4350 0x563493a8a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.577+0000 7f97cf13b740 1 -- 172.21.15.16:0/2849522395 >> 172.21.15.16:0/2849522395 conn(0x5634936d8740 msgr2=0x563493a88670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.577+0000 7f97cf13b740 1 -- 172.21.15.16:0/2849522395 shutdown_connections 2024-08-28T19:41:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.577+0000 7f97cf13b740 1 -- 172.21.15.16:0/2849522395 wait complete. 2024-08-28T19:41:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97cf13b740 1 -- start start 2024-08-28T19:41:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97cf13b740 1 Processor -- start 2024-08-28T19:41:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 0x563493a95eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 0x563493a963f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936e4350 0x563493aa2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97cf13b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563493852890 con 0x5634936c3670 2024-08-28T19:41:34.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97cf13b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563493976790 con 0x5634936e4350 2024-08-28T19:41:34.205 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:41:34.205 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:41:34.205 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "39fa9343-222a-493e-bb10-c8863c663453", 2024-08-28T19:41:34.205 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:41:34.205 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97c259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 0x563493a963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97cf13b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563493a98f60 con 0x5634936bb490 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97c259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 0x563493a963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44692/0 (socket says 172.21.15.16:44692) 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97c259c640 1 -- 172.21.15.16:0/3715052930 learned_addr learned my addr 172.21.15.16:0/3715052930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97bbfff640 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 0x563493a95eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.578+0000 7f97cf13b740 10 client.0 _ll_register_callbacks cb 0x5634936fbba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97bbfff640 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936e4350 0x563493aa2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97bbfff640 1 -- 172.21.15.16:0/3715052930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936e4350 msgr2=0x563493aa2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97bbfff640 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936e4350 0x563493aa2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97bbfff640 1 -- 172.21.15.16:0/3715052930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 msgr2=0x563493a963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97bbfff640 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 0x563493a963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97bbfff640 1 -- 172.21.15.16:0/3715052930 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634939adf30 con 0x5634936bb490 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97c259c640 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 0x563493a963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97bbfff640 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 0x563493a95eb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0058c0 tx=0x7f97ac004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3266b4b9205fefd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97a9ffb640 1 -- 172.21.15.16:0/3715052930 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac003e00 con 0x5634936bb490 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97cf13b740 1 -- 172.21.15.16:0/3715052930 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97b40054d0 con 0x5634936bb490 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97a9ffb640 1 -- 172.21.15.16:0/3715052930 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f97ac005020 con 0x5634936bb490 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97a9ffb640 1 -- 172.21.15.16:0/3715052930 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f97ac02c050 con 0x5634936bb490 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97cf13b740 10 client.29233 fetch_fsmap learned FSMap version 102 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.579+0000 7f97cf13b740 1 -- 172.21.15.16:0/3715052930 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x563493aa4d70 con 0x5634936bb490 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.580+0000 7f97a9ffb640 1 -- 172.21.15.16:0/3715052930 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97b40054d0 con 0x5634936bb490 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.580+0000 7f97a9ffb640 1 -- 172.21.15.16:0/3715052930 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 102) ==== 8398+0+0 (secure 0 0 0) 0x7f97ac021480 con 0x5634936bb490 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.580+0000 7f97cf13b740 10 client.29233 fetch_fsmap finished waiting for FSMap version 102 2024-08-28T19:41:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.580+0000 7f97cf13b740 10 client.29233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.580+0000 7f97cf13b740 20 client.29233 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.580+0000 7f97cf13b740 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5634936a2cd0 0x563493aa82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.580+0000 7f97cf13b740 4 client.29233 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.580+0000 7f97cf13b740 1 -- 172.21.15.16:0/3715052930 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563493a02540 con 0x5634936a2cd0 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.580+0000 7f97c259c640 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5634936a2cd0 0x563493aa82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.580+0000 7f97c259c640 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5634936a2cd0 0x563493aa82b0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85f86b8cd8da5113 server_cookie=8f5a174609f5ead2 in_seq=0 out_seq=0 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.580+0000 7f97a9ffb640 10 client.29233 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97a9ffb640 1 -- 172.21.15.16:0/3715052930 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563493a02540 con 0x5634936a2cd0 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97a9ffb640 10 client.29233 handle_command_reply: tid=0 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97cf13b740 1 client.29233 shutdown 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97cf13b740 1 -- 172.21.15.16:0/3715052930 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 msgr2=0x563493a95eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97cf13b740 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 0x563493a95eb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0058c0 tx=0x7f97ac004730 comp rx=0 tx=0).stop 2024-08-28T19:41:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97cf13b740 1 -- 172.21.15.16:0/3715052930 shutdown_connections 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97cf13b740 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936e4350 0x563493aa2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97cf13b740 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936c3670 0x563493a963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97cf13b740 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5634936a2cd0 0x563493aa82b0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97cf13b740 1 --2- 172.21.15.16:0/3715052930 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 0x563493a95eb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97a9ffb640 0 client.29233 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97cf13b740 1 -- 172.21.15.16:0/3715052930 >> 172.21.15.16:0/3715052930 conn(0x5634936d8740 msgr2=0x563493a8d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.581+0000 7f97cf13b740 1 -- 172.21.15.16:0/3715052930 shutdown_connections 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.582+0000 7f97cf13b740 1 -- 172.21.15.16:0/3715052930 wait complete. 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.582+0000 7f97cf13b740 20 client.29233 trim_cache size 0 max 16384 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97cf13b740 1 Processor -- start 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97cf13b740 1 -- start start 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 0x563493a8c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936c3670 0x563493a97d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936e4350 0x563493aa2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97cf13b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41b0c50 con 0x5634936e4350 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97cf13b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41b28e0 con 0x5634936c3670 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97cf13b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41b1380 con 0x5634936bb490 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936e4350 0x563493aa2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97bbfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 0x563493a8c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97bbfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 0x563493a8c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:53170/0 (socket says 172.21.15.16:53170) 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97bbfff640 1 -- 172.21.15.16:0/3983614322 learned_addr learned my addr 172.21.15.16:0/3983614322 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.602+0000 7f97c259c640 1 --2- 172.21.15.16:0/3983614322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936c3670 0x563493a97d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.603+0000 7f97bbfff640 1 -- 172.21.15.16:0/3983614322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936c3670 msgr2=0x563493a97d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.603+0000 7f97bbfff640 1 --2- 172.21.15.16:0/3983614322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936c3670 0x563493a97d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.603+0000 7f97bbfff640 1 -- 172.21.15.16:0/3983614322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936e4350 msgr2=0x563493aa2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.603+0000 7f97bbfff640 1 --2- 172.21.15.16:0/3983614322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936e4350 0x563493aa2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.603+0000 7f97bbfff640 1 -- 172.21.15.16:0/3983614322 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563493a83b80 con 0x5634936bb490 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.603+0000 7f97bbfff640 1 --2- 172.21.15.16:0/3983614322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 0x563493a8c280 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0045c0 tx=0x7f97ac000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbbf0aa247adee1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.603+0000 7f97abfff640 1 -- 172.21.15.16:0/3983614322 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac021480 con 0x5634936bb490 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.603+0000 7f97abfff640 1 -- 172.21.15.16:0/3983614322 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97ac021620 con 0x5634936bb490 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.603+0000 7f97abfff640 1 -- 172.21.15.16:0/3983614322 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0112d0 con 0x5634936bb490 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.604+0000 7f97cf13b740 1 -- 172.21.15.16:0/3983614322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 msgr2=0x563493a8c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.604+0000 7f97cf13b740 1 --2- 172.21.15.16:0/3983614322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 0x563493a8c280 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0045c0 tx=0x7f97ac000bc0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 -- 172.21.15.16:0/3983614322 shutdown_connections 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 --2- 172.21.15.16:0/3983614322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936e4350 0x563493aa2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 --2- 172.21.15.16:0/3983614322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936c3670 0x563493a97d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 --2- 172.21.15.16:0/3983614322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936bb490 0x563493a8c280 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 -- 172.21.15.16:0/3983614322 >> 172.21.15.16:0/3983614322 conn(0x5634936d8740 msgr2=0x563493a8f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 -- 172.21.15.16:0/3983614322 shutdown_connections 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 -- 172.21.15.16:0/3983614322 wait complete. 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 -- start start 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 Processor -- start 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936bb490 0x563493a9a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936c3670 0x563493a9a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936e4350 0x563493a9ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41b2150 con 0x5634936bb490 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41b1a50 con 0x5634936c3670 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41aec50 con 0x5634936e4350 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.605+0000 7f97cf13b740 10 client.0 _ll_register_callbacks cb 0x7f97ac004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936bb490 0x563493a9a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97c259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936c3670 0x563493a9a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936bb490 0x563493a9a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44704/0 (socket says 172.21.15.16:44704) 2024-08-28T19:41:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97bbfff640 1 -- 172.21.15.16:0/399160702 learned_addr learned my addr 172.21.15.16:0/399160702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97bbfff640 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936e4350 0x563493a9ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97bbfff640 1 -- 172.21.15.16:0/399160702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936e4350 msgr2=0x563493a9ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97bbfff640 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936e4350 0x563493a9ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97bbfff640 1 -- 172.21.15.16:0/399160702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936c3670 msgr2=0x563493a9a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97bbfff640 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936c3670 0x563493a9a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97bbfff640 1 -- 172.21.15.16:0/399160702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563493b0fed0 con 0x5634936bb490 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97bbfff640 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936bb490 0x563493a9a3d0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x563493a9b3b0 tx=0x7f97ac0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c620abe3fdccae server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97a9ffb640 1 -- 172.21.15.16:0/399160702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac021480 con 0x5634936bb490 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.606+0000 7f97a9ffb640 1 -- 172.21.15.16:0/399160702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f97ac011230 con 0x5634936bb490 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97cf13b740 1 -- 172.21.15.16:0/399160702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97ac021620 con 0x5634936bb490 2024-08-28T19:41:34.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97a9ffb640 1 -- 172.21.15.16:0/399160702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f97ac005040 con 0x5634936bb490 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97cf13b740 10 client.20223 fetch_fsmap learned FSMap version 102 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97cf13b740 1 -- 172.21.15.16:0/399160702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x563493b175b0 con 0x5634936bb490 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97a9ffb640 1 -- 172.21.15.16:0/399160702 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97ac021620 con 0x5634936bb490 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97a9ffb640 1 -- 172.21.15.16:0/399160702 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 102) ==== 8398+0+0 (secure 0 0 0) 0x7f97ac012020 con 0x5634936bb490 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97cf13b740 10 client.20223 fetch_fsmap finished waiting for FSMap version 102 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97cf13b740 10 client.20223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97cf13b740 20 client.20223 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97cf13b740 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563493678080 0x563493b19da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97cf13b740 4 client.20223 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.607+0000 7f97cf13b740 1 -- 172.21.15.16:0/399160702 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5634939fdfe0 con 0x563493678080 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.608+0000 7f97c259c640 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563493678080 0x563493b19da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.608+0000 7f97c259c640 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563493678080 0x563493b19da0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1e1d15b396b7d2 server_cookie=b9d6682e6f4cb4c1 in_seq=0 out_seq=0 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.608+0000 7f97a9ffb640 10 client.20223 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:34.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.611+0000 7f97a9ffb640 1 -- 172.21.15.16:0/399160702 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5634939fdfe0 con 0x563493678080 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.611+0000 7f97a9ffb640 10 client.20223 handle_command_reply: tid=0 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.611+0000 7f97cf13b740 1 client.20223 shutdown 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97cf13b740 1 -- 172.21.15.16:0/399160702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936bb490 msgr2=0x563493a9a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97cf13b740 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936bb490 0x563493a9a3d0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x563493a9b3b0 tx=0x7f97ac0096c0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97cf13b740 1 -- 172.21.15.16:0/399160702 shutdown_connections 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97cf13b740 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x563493678080 0x563493b19da0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97a9ffb640 0 client.20223 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97cf13b740 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5634936e4350 0x563493a9ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97cf13b740 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634936c3670 0x563493a9a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97cf13b740 1 --2- 172.21.15.16:0/399160702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5634936bb490 0x563493a9a3d0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97cf13b740 1 -- 172.21.15.16:0/399160702 >> 172.21.15.16:0/399160702 conn(0x5634936d8740 msgr2=0x563493a97aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97cf13b740 1 -- 172.21.15.16:0/399160702 shutdown_connections 2024-08-28T19:41:34.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97cf13b740 1 -- 172.21.15.16:0/399160702 wait complete. 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.612+0000 7f97cf13b740 20 client.20223 trim_cache size 0 max 16384 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.613+0000 7f97c8bc6640 1 -- 172.21.15.16:0/143251371 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f97b00b4ca0 msgr2=0x7f97b00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.613+0000 7f97c8bc6640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f97b00b4ca0 0x7f97b00b71b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f97bc004af0 tx=0x7f97bc02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.613+0000 7f97c8bc6640 1 -- 172.21.15.16:0/143251371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4109bb0 msgr2=0x7f97c419fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.613+0000 7f97c8bc6640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4109bb0 0x7f97c419fa30 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f97bc02f540 tx=0x7f97bc0316f0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.613+0000 7f97c8bc6640 1 -- 172.21.15.16:0/143251371 shutdown_connections 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.613+0000 7f97c8bc6640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4109bb0 0x7f97c419fa30 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.613+0000 7f97c8bc6640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f97b00b4ca0 0x7f97b00b71b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.613+0000 7f97c8bc6640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c4109260 0x7f97c419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.613+0000 7f97c8bc6640 1 --2- 172.21.15.16:0/143251371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97c4103920 0x7f97c419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.613+0000 7f97c8bc6640 1 -- 172.21.15.16:0/143251371 >> 172.21.15.16:0/143251371 conn(0x7f97c40ff350 msgr2=0x7f97c4106760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.616+0000 7f97c8bc6640 1 -- 172.21.15.16:0/143251371 shutdown_connections 2024-08-28T19:41:34.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:33.616+0000 7f97c8bc6640 1 -- 172.21.15.16:0/143251371 wait complete. 2024-08-28T19:41:34.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 166246 22.25 MB/sec execute 310 sec latency 304.462 ms 2024-08-28T19:41:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:34 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:41:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:34 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:41:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:34 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:41:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:34 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:41:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:34 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:34.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:34 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:34 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:41:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:34 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:41:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:34 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:41:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:34 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:41:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:34 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:34 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:34 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:41:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:34 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:41:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:34 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:41:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:34 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:41:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:34 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:34 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:34.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:41:35.330 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:35.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 166843 22.27 MB/sec execute 311 sec latency 150.748 ms 2024-08-28T19:41:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:35 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:41:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:35 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:35 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:41:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:35 smithi016 ceph-mon[18556]: pgmap v983: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 9.0 MiB/s wr, 352 op/s 2024-08-28T19:41:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:35 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:35 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:35.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:35 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:35 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:41:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:35 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:35 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:41:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:35 smithi028 ceph-mon[21769]: pgmap v983: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 9.0 MiB/s wr, 352 op/s 2024-08-28T19:41:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:35 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:35 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:35.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:35 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:35 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:41:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:35 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:35 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:41:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:35 smithi115 ceph-mon[22184]: pgmap v983: 129 pgs: 129 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 9.0 MiB/s wr, 352 op/s 2024-08-28T19:41:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:35 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:35 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:35 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:41:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.354+0000 7f4f326d6640 1 -- 172.21.15.16:0/1790706805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c108a90 msgr2=0x7f4f2c10f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.354+0000 7f4f326d6640 1 --2- 172.21.15.16:0/1790706805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c108a90 0x7f4f2c10f2c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009ca0 tx=0x7f4f2402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.355+0000 7f4f326d6640 1 -- 172.21.15.16:0/1790706805 shutdown_connections 2024-08-28T19:41:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.355+0000 7f4f326d6640 1 --2- 172.21.15.16:0/1790706805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c108a90 0x7f4f2c10f2c0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.355+0000 7f4f326d6640 1 --2- 172.21.15.16:0/1790706805 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4f2c108140 0x7f4f2c108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.355+0000 7f4f326d6640 1 --2- 172.21.15.16:0/1790706805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c102800 0x7f4f2c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.355+0000 7f4f326d6640 1 -- 172.21.15.16:0/1790706805 >> 172.21.15.16:0/1790706805 conn(0x7f4f2c0794d0 msgr2=0x7f4f2c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.355+0000 7f4f326d6640 1 -- 172.21.15.16:0/1790706805 shutdown_connections 2024-08-28T19:41:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.355+0000 7f4f326d6640 1 -- 172.21.15.16:0/1790706805 wait complete. 2024-08-28T19:41:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.356+0000 7f4f326d6640 1 Processor -- start 2024-08-28T19:41:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.356+0000 7f4f326d6640 1 -- start start 2024-08-28T19:41:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.357+0000 7f4f326d6640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4f2c102800 0x7f4f2c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.357+0000 7f4f326d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c108140 0x7f4f2c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.357+0000 7f4f326d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c108a90 0x7f4f2c19fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.357+0000 7f4f326d6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2c06af90 con 0x7f4f2c108140 2024-08-28T19:41:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.357+0000 7f4f2b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c108140 0x7f4f2c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.357+0000 7f4f326d6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2c06ae10 con 0x7f4f2c108a90 2024-08-28T19:41:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.357+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4f2c102800 0x7f4f2c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.357+0000 7f4f326d6640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2c06b110 con 0x7f4f2c102800 2024-08-28T19:41:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.358+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4f2c102800 0x7f4f2c19a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56220/0 (socket says 172.21.15.16:56220) 2024-08-28T19:41:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.358+0000 7f4f2bfff640 1 -- 172.21.15.16:0/1654397839 learned_addr learned my addr 172.21.15.16:0/1654397839 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:36.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.358+0000 7f4f30c73640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c108a90 0x7f4f2c19fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.358+0000 7f4f2bfff640 1 -- 172.21.15.16:0/1654397839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c108a90 msgr2=0x7f4f2c19fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:36.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.358+0000 7f4f2bfff640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c108a90 0x7f4f2c19fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.358+0000 7f4f2bfff640 1 -- 172.21.15.16:0/1654397839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c108140 msgr2=0x7f4f2c19a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:36.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.358+0000 7f4f2bfff640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c108140 0x7f4f2c19a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.358+0000 7f4f2bfff640 1 -- 172.21.15.16:0/1654397839 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2c1a0130 con 0x7f4f2c102800 2024-08-28T19:41:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.360+0000 7f4f2bfff640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4f2c102800 0x7f4f2c19a1c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00e9f0 tx=0x7f4f1c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.361+0000 7f4f297fa640 1 -- 172.21.15.16:0/1654397839 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c00cd80 con 0x7f4f2c102800 2024-08-28T19:41:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.361+0000 7f4f297fa640 1 -- 172.21.15.16:0/1654397839 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f1c0108e0 con 0x7f4f2c102800 2024-08-28T19:41:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.361+0000 7f4f297fa640 1 -- 172.21.15.16:0/1654397839 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c018980 con 0x7f4f2c102800 2024-08-28T19:41:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.361+0000 7f4f326d6640 1 -- 172.21.15.16:0/1654397839 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f2c1a0420 con 0x7f4f2c102800 2024-08-28T19:41:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.361+0000 7f4f326d6640 1 -- 172.21.15.16:0/1654397839 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f2c1a6b80 con 0x7f4f2c102800 2024-08-28T19:41:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.365+0000 7f4f38c4b740 1 Processor -- start 2024-08-28T19:41:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.365+0000 7f4f38c4b740 1 -- start start 2024-08-28T19:41:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 0x55d0655932c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f297fa640 1 -- 172.21.15.16:0/1654397839 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4f1c018b20 con 0x7f4f2c102800 2024-08-28T19:41:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d065593800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d065593d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d06550a630 con 0x55d0651ee070 2024-08-28T19:41:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d0654f9bc0 con 0x55d0651cd390 2024-08-28T19:41:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d0654fdb70 con 0x55d0651c51b0 2024-08-28T19:41:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d065593800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d065593800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:56228/0 (socket says 172.21.15.16:56228) 2024-08-28T19:41:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f2bfff640 1 -- 172.21.15.16:0/3764836967 learned_addr learned my addr 172.21.15.16:0/3764836967 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f30c73640 1 --2- 172.21.15.16:0/3764836967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 0x55d0655932c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.366+0000 7f4f2b7fe640 1 --2- 172.21.15.16:0/3764836967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d065593d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.367+0000 7f4f30c73640 1 -- 172.21.15.16:0/3764836967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 msgr2=0x55d065593800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.367+0000 7f4f30c73640 1 --2- 172.21.15.16:0/3764836967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d065593800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.367+0000 7f4f30c73640 1 -- 172.21.15.16:0/3764836967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 msgr2=0x55d065593d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.367+0000 7f4f30c73640 1 --2- 172.21.15.16:0/3764836967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d065593d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.367+0000 7f4f30c73640 1 -- 172.21.15.16:0/3764836967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d065259650 con 0x55d0651ee070 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.367+0000 7f4f297fa640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4f140b4df0 0x7f4f140b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.367+0000 7f4f30c73640 1 --2- 172.21.15.16:0/3764836967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 0x55d0655932c0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f4f24005cc0 tx=0x7f4f2402f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a267cdaa1053951 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.367+0000 7f4f2bfff640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4f140b4df0 0x7f4f140b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.367+0000 7f4f297fa640 1 -- 172.21.15.16:0/1654397839 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f4f1c0e5b00 con 0x7f4f2c102800 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.368+0000 7f4f127fc640 1 -- 172.21.15.16:0/3764836967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24031cf0 con 0x55d0651ee070 2024-08-28T19:41:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.368+0000 7f4f127fc640 1 -- 172.21.15.16:0/3764836967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f24036040 con 0x55d0651ee070 2024-08-28T19:41:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.368+0000 7f4f127fc640 1 -- 172.21.15.16:0/3764836967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24004040 con 0x55d0651ee070 2024-08-28T19:41:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.369+0000 7f4f38c4b740 1 -- 172.21.15.16:0/3764836967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 msgr2=0x55d0655932c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.369+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/3764836967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 0x55d0655932c0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f4f24005cc0 tx=0x7f4f2402f420 comp rx=0 tx=0).stop 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.369+0000 7f4f2bfff640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4f140b4df0 0x7f4f140b7300 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0a7ea0 tx=0x7f4f1c000bc0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.369+0000 7f4f38c4b740 1 -- 172.21.15.16:0/3764836967 shutdown_connections 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.369+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/3764836967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d065593d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.370+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/3764836967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d065593800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.370+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/3764836967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 0x55d0655932c0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.370+0000 7f4f38c4b740 1 -- 172.21.15.16:0/3764836967 >> 172.21.15.16:0/3764836967 conn(0x55d0651e2460 msgr2=0x55d0655911b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.370+0000 7f4f38c4b740 1 -- 172.21.15.16:0/3764836967 shutdown_connections 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.370+0000 7f4f38c4b740 1 -- 172.21.15.16:0/3764836967 wait complete. 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.370+0000 7f4f38c4b740 1 -- start start 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.371+0000 7f4f38c4b740 1 Processor -- start 2024-08-28T19:41:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.371+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d06559ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.371+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651cd390 0x55d06559ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.371+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651ee070 0x55d0655af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.371+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d06535c700 con 0x55d0651cd390 2024-08-28T19:41:36.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.371+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d065480700 con 0x55d0651ee070 2024-08-28T19:41:36.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.371+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d0655a5e50 con 0x55d0651c51b0 2024-08-28T19:41:36.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.371+0000 7f4f38c4b740 10 client.0 _ll_register_callbacks cb 0x55d0654afdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.371+0000 7f4f2b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651cd390 0x55d06559ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.371+0000 7f4f2b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651cd390 0x55d06559ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41138/0 (socket says 172.21.15.16:41138) 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.372+0000 7f4f2b7fe640 1 -- 172.21.15.16:0/2421774963 learned_addr learned my addr 172.21.15.16:0/2421774963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.372+0000 7f4f30c73640 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d06559ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.372+0000 7f4f2b7fe640 1 -- 172.21.15.16:0/2421774963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 msgr2=0x55d06559ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.372+0000 7f4f2b7fe640 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d06559ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.372+0000 7f4f2b7fe640 1 -- 172.21.15.16:0/2421774963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651ee070 msgr2=0x55d0655af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.372+0000 7f4f2b7fe640 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651ee070 0x55d0655af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.372+0000 7f4f2b7fe640 1 -- 172.21.15.16:0/2421774963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0654b98b0 con 0x55d0651cd390 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.373+0000 7f4f2b7fe640 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651cd390 0x55d06559ef60 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000beb0 tx=0x7f4f20012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b569b6d98922eea5 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.374+0000 7f4f0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.374+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/2421774963 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f20004230 con 0x55d0651cd390 2024-08-28T19:41:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.374+0000 7f4f38c4b740 1 -- 172.21.15.16:0/2421774963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f24036040 con 0x55d0651cd390 2024-08-28T19:41:36.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.374+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/2421774963 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f4f200043d0 con 0x55d0651cd390 2024-08-28T19:41:36.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.374+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/2421774963 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f4f200106a0 con 0x55d0651cd390 2024-08-28T19:41:36.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.375+0000 7f4f38c4b740 10 client.20235 fetch_fsmap learned FSMap version 102 2024-08-28T19:41:36.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.375+0000 7f4f38c4b740 1 -- 172.21.15.16:0/2421774963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55d0655a1730 con 0x55d0651cd390 2024-08-28T19:41:36.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.375+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/2421774963 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f24036040 con 0x55d0651cd390 2024-08-28T19:41:36.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.375+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/2421774963 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 102) ==== 8398+0+0 (secure 0 0 0) 0x7f4f20019070 con 0x55d0651cd390 2024-08-28T19:41:36.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.376+0000 7f4f38c4b740 10 client.20235 fetch_fsmap finished waiting for FSMap version 102 2024-08-28T19:41:36.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.376+0000 7f4f38c4b740 10 client.20235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:41:36.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.376+0000 7f4f38c4b740 20 client.20235 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:36.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.376+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0651ac9f0 0x55d0655a5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.376+0000 7f4f38c4b740 4 client.20235 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:41:36.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.376+0000 7f4f38c4b740 1 -- 172.21.15.16:0/2421774963 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d06550c580 con 0x55d0651ac9f0 2024-08-28T19:41:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.377+0000 7f4f30c73640 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0651ac9f0 0x55d0655a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.378+0000 7f4f30c73640 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0651ac9f0 0x55d0655a5120 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2328adcf7ab413d8 server_cookie=8d453a6d42eb06f5 in_seq=0 out_seq=0 2024-08-28T19:41:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.379+0000 7f4f0b7fe640 10 client.20235 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.380+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/2421774963 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d06550c580 con 0x55d0651ac9f0 2024-08-28T19:41:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.380+0000 7f4f0b7fe640 10 client.20235 handle_command_reply: tid=0 2024-08-28T19:41:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.380+0000 7f4f38c4b740 1 client.20235 shutdown 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.381+0000 7f4f38c4b740 1 -- 172.21.15.16:0/2421774963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651cd390 msgr2=0x55d06559ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.381+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651cd390 0x55d06559ef60 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000beb0 tx=0x7f4f20012820 comp rx=0 tx=0).stop 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.381+0000 7f4f38c4b740 1 -- 172.21.15.16:0/2421774963 shutdown_connections 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.381+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651ee070 0x55d0655af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.381+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651cd390 0x55d06559ef60 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.381+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d0651ac9f0 0x55d0655a5120 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.381+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/2421774963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d06559ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.381+0000 7f4f0b7fe640 0 client.20235 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.381+0000 7f4f38c4b740 1 -- 172.21.15.16:0/2421774963 >> 172.21.15.16:0/2421774963 conn(0x55d0651e2460 msgr2=0x55d065596450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.381+0000 7f4f38c4b740 1 -- 172.21.15.16:0/2421774963 shutdown_connections 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.382+0000 7f4f38c4b740 1 -- 172.21.15.16:0/2421774963 wait complete. 2024-08-28T19:41:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.382+0000 7f4f38c4b740 20 client.20235 trim_cache size 0 max 16384 2024-08-28T19:41:36.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f38c4b740 1 Processor -- start 2024-08-28T19:41:36.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f38c4b740 1 -- start start 2024-08-28T19:41:36.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d065591200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d0655af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 0x55d0655963c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f18010080 con 0x55d0651ee070 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f18011d10 con 0x55d0651cd390 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4f180107b0 con 0x55d0651c51b0 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f30c73640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 0x55d0655963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f30c73640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 0x55d0655963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41152/0 (socket says 172.21.15.16:41152) 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f30c73640 1 -- 172.21.15.16:0/3581675754 learned_addr learned my addr 172.21.15.16:0/3581675754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f2b7fe640 1 --2- 172.21.15.16:0/3581675754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d0655af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.412+0000 7f4f30c73640 1 --2- 172.21.15.16:0/3581675754 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d065591200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.413+0000 7f4f30c73640 1 -- 172.21.15.16:0/3581675754 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 msgr2=0x55d065591200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.413+0000 7f4f30c73640 1 --2- 172.21.15.16:0/3581675754 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d065591200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.413+0000 7f4f30c73640 1 -- 172.21.15.16:0/3581675754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 msgr2=0x55d0655af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.413+0000 7f4f30c73640 1 --2- 172.21.15.16:0/3581675754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d0655af140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.413+0000 7f4f30c73640 1 -- 172.21.15.16:0/3581675754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0655958a0 con 0x55d0651ee070 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.413+0000 7f4f30c73640 1 --2- 172.21.15.16:0/3581675754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 0x55d0655963c0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f4f24000f80 tx=0x7f4f24055040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5a66088a5b3909 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:36.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.413+0000 7f4f11ffb640 1 -- 172.21.15.16:0/3581675754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2404a040 con 0x55d0651ee070 2024-08-28T19:41:36.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.413+0000 7f4f11ffb640 1 -- 172.21.15.16:0/3581675754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f24036070 con 0x55d0651ee070 2024-08-28T19:41:36.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.413+0000 7f4f11ffb640 1 -- 172.21.15.16:0/3581675754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24031650 con 0x55d0651ee070 2024-08-28T19:41:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.415+0000 7f4f38c4b740 1 -- 172.21.15.16:0/3581675754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 msgr2=0x55d0655963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.415+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/3581675754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 0x55d0655963c0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f4f24000f80 tx=0x7f4f24055040 comp rx=0 tx=0).stop 2024-08-28T19:41:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.415+0000 7f4f38c4b740 1 -- 172.21.15.16:0/3581675754 shutdown_connections 2024-08-28T19:41:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.415+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/3581675754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651ee070 0x55d0655963c0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.415+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/3581675754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d0655af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.415+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/3581675754 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651c51b0 0x55d065591200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.415+0000 7f4f38c4b740 1 -- 172.21.15.16:0/3581675754 >> 172.21.15.16:0/3581675754 conn(0x55d0651e2460 msgr2=0x55d065590af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.415+0000 7f4f38c4b740 1 -- 172.21.15.16:0/3581675754 shutdown_connections 2024-08-28T19:41:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.415+0000 7f4f38c4b740 1 -- 172.21.15.16:0/3581675754 wait complete. 2024-08-28T19:41:36.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.416+0000 7f4f38c4b740 1 -- start start 2024-08-28T19:41:36.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.416+0000 7f4f38c4b740 1 Processor -- start 2024-08-28T19:41:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651c51b0 0x55d0655b7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d0655b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f38c4b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651ee070 0x55d0655b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f18011580 con 0x55d0651c51b0 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f18010e80 con 0x55d0651cd390 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f38c4b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1800e080 con 0x55d0651ee070 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f38c4b740 10 client.0 _ll_register_callbacks cb 0x7f4f2000b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f30c73640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651c51b0 0x55d0655b7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f30c73640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651c51b0 0x55d0655b7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41156/0 (socket says 172.21.15.16:41156) 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f30c73640 1 -- 172.21.15.16:0/116964836 learned_addr learned my addr 172.21.15.16:0/116964836 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f2b7fe640 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d0655b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f30c73640 1 -- 172.21.15.16:0/116964836 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651ee070 msgr2=0x55d0655b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:36.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f30c73640 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651ee070 0x55d0655b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f30c73640 1 -- 172.21.15.16:0/116964836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 msgr2=0x55d0655b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f30c73640 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d0655b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f30c73640 1 -- 172.21.15.16:0/116964836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0655b55a0 con 0x55d0651c51b0 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.417+0000 7f4f2b7fe640 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d0655b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.418+0000 7f4f30c73640 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651c51b0 0x55d0655b7400 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f4f240096c0 tx=0x7f4f24009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45ee3a891f0c498 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.418+0000 7f4f0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.418+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/116964836 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24053a00 con 0x55d0651c51b0 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.418+0000 7f4f38c4b740 1 -- 172.21.15.16:0/116964836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f24036070 con 0x55d0651c51b0 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.418+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/116964836 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f4f2404a040 con 0x55d0651c51b0 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.419+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/116964836 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f4f24031af0 con 0x55d0651c51b0 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.419+0000 7f4f38c4b740 10 client.20241 fetch_fsmap learned FSMap version 102 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.419+0000 7f4f38c4b740 1 -- 172.21.15.16:0/116964836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55d06561bd90 con 0x55d0651c51b0 2024-08-28T19:41:36.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.419+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/116964836 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f24036070 con 0x55d0651c51b0 2024-08-28T19:41:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.419+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/116964836 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 102) ==== 8398+0+0 (secure 0 0 0) 0x7f4f2402f8d0 con 0x55d0651c51b0 2024-08-28T19:41:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.420+0000 7f4f38c4b740 10 client.20241 fetch_fsmap finished waiting for FSMap version 102 2024-08-28T19:41:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.420+0000 7f4f38c4b740 10 client.20241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:41:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.420+0000 7f4f38c4b740 20 client.20241 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.420+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d065181da0 0x55d06561e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.420+0000 7f4f38c4b740 4 client.20241 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:41:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.420+0000 7f4f38c4b740 1 -- 172.21.15.16:0/116964836 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d065508020 con 0x55d065181da0 2024-08-28T19:41:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.420+0000 7f4f2b7fe640 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d065181da0 0x55d06561e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.421+0000 7f4f2b7fe640 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d065181da0 0x55d06561e000 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbb9b6fc8c558c5c server_cookie=a41c42eae66c0ccf in_seq=0 out_seq=0 2024-08-28T19:41:36.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.421+0000 7f4f0b7fe640 10 client.20241 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:36.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.426+0000 7f4f0b7fe640 1 -- 172.21.15.16:0/116964836 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7954 (crc 0 0 0) 0x55d065508020 con 0x55d065181da0 2024-08-28T19:41:36.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.426+0000 7f4f0b7fe640 10 client.20241 handle_command_reply: tid=0 2024-08-28T19:41:36.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.426+0000 7f4f38c4b740 1 client.20241 shutdown 2024-08-28T19:41:36.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.426+0000 7f4f38c4b740 1 -- 172.21.15.16:0/116964836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651c51b0 msgr2=0x55d0655b7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:36.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.426+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651c51b0 0x55d0655b7400 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f4f240096c0 tx=0x7f4f24009800 comp rx=0 tx=0).stop 2024-08-28T19:41:36.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.426+0000 7f4f38c4b740 1 -- 172.21.15.16:0/116964836 shutdown_connections 2024-08-28T19:41:36.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.426+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55d065181da0 0x55d06561e000 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.426+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d0651ee070 0x55d0655b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.426+0000 7f4f0b7fe640 0 client.20241 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:36.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.426+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0651cd390 0x55d0655b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.426+0000 7f4f38c4b740 1 --2- 172.21.15.16:0/116964836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0651c51b0 0x55d0655b7400 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.427+0000 7f4f38c4b740 1 -- 172.21.15.16:0/116964836 >> 172.21.15.16:0/116964836 conn(0x55d0651e2460 msgr2=0x55d0655963c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:36.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.427+0000 7f4f38c4b740 1 -- 172.21.15.16:0/116964836 shutdown_connections 2024-08-28T19:41:36.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.427+0000 7f4f38c4b740 1 -- 172.21.15.16:0/116964836 wait complete. 2024-08-28T19:41:36.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.427+0000 7f4f38c4b740 20 client.20241 trim_cache size 0 max 16384 2024-08-28T19:41:36.429 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:41:36.429 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:41:36.429 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:41:36.429 INFO:teuthology.orchestra.run.smithi016.stdout: "39fa9343-222a-493e-bb10-c8863c663453": { 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "39fa9343-222a-493e-bb10-c8863c663453", 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "39fa9343-222a-493e-bb10-c8863c663453": { 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:41:33.609756+0000", 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.430 INFO:teuthology.orchestra.run.smithi016.stdout: "a466ab70-970a-4b3e-927e-47c65e508132": { 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:40:38.307336+0000", 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:55.619275+0000", 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:41:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.434 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.435 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.436 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.437 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.438 INFO:teuthology.orchestra.run.smithi016.stdout: "8de90bf3-5743-4711-98c2-a56ca0757291": { 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:39:48.098772+0000", 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.440 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:41:36.441 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.442 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:41:36.444 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:36.446 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:41:36.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.428+0000 7f4f326d6640 1 -- 172.21.15.16:0/1654397839 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4f140b4df0 msgr2=0x7f4f140b7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:36.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.428+0000 7f4f326d6640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4f140b4df0 0x7f4f140b7300 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0a7ea0 tx=0x7f4f1c000bc0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.428+0000 7f4f326d6640 1 -- 172.21.15.16:0/1654397839 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4f2c102800 msgr2=0x7f4f2c19a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:36.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.428+0000 7f4f326d6640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4f2c102800 0x7f4f2c19a1c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00e9f0 tx=0x7f4f1c00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.428+0000 7f4f326d6640 1 -- 172.21.15.16:0/1654397839 shutdown_connections 2024-08-28T19:41:36.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.428+0000 7f4f326d6640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f4f140b4df0 0x7f4f140b7300 secure :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0a7ea0 tx=0x7f4f1c000bc0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.428+0000 7f4f326d6640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c108a90 0x7f4f2c19fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.428+0000 7f4f326d6640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c108140 0x7f4f2c19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.428+0000 7f4f326d6640 1 --2- 172.21.15.16:0/1654397839 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4f2c102800 0x7f4f2c19a1c0 secure :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00e9f0 tx=0x7f4f1c00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:41:36.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.428+0000 7f4f326d6640 1 -- 172.21.15.16:0/1654397839 >> 172.21.15.16:0/1654397839 conn(0x7f4f2c0794d0 msgr2=0x7f4f2c104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:36.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.429+0000 7f4f326d6640 1 -- 172.21.15.16:0/1654397839 shutdown_connections 2024-08-28T19:41:36.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:36.429+0000 7f4f326d6640 1 -- 172.21.15.16:0/1654397839 wait complete. 2024-08-28T19:41:36.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 167515 22.32 MB/sec execute 312 sec latency 122.123 ms 2024-08-28T19:41:36.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:36 smithi016 ceph-mon[18556]: pgmap v984: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 552 op/s 2024-08-28T19:41:36.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:36 smithi028 ceph-mon[21769]: pgmap v984: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 552 op/s 2024-08-28T19:41:36.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:36 smithi115 ceph-mon[22184]: pgmap v984: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 552 op/s 2024-08-28T19:41:37.242 INFO:tasks.cephfs.filesystem:scrub status for tag:39fa9343-222a-493e-bb10-c8863c663453 - {'path': '/', 'tag': '39fa9343-222a-493e-bb10-c8863c663453', 'options': 'recursive,force'} 2024-08-28T19:41:37.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 168139 22.37 MB/sec execute 313 sec latency 78.261 ms 2024-08-28T19:41:38.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 168674 22.37 MB/sec execute 314 sec latency 114.146 ms 2024-08-28T19:41:39.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:38 smithi016 ceph-mon[18556]: pgmap v985: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.7 MiB/s wr, 486 op/s 2024-08-28T19:41:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:38 smithi028 ceph-mon[21769]: pgmap v985: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.7 MiB/s wr, 486 op/s 2024-08-28T19:41:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:38 smithi115 ceph-mon[22184]: pgmap v985: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.7 MiB/s wr, 486 op/s 2024-08-28T19:41:39.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 169359 22.44 MB/sec execute 315 sec latency 83.557 ms 2024-08-28T19:41:40.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 169977 22.47 MB/sec execute 316 sec latency 96.123 ms 2024-08-28T19:41:41.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:40 smithi016 ceph-mon[18556]: pgmap v986: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 7.5 MiB/s wr, 519 op/s 2024-08-28T19:41:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:40 smithi028 ceph-mon[21769]: pgmap v986: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 7.5 MiB/s wr, 519 op/s 2024-08-28T19:41:41.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:40 smithi115 ceph-mon[22184]: pgmap v986: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 7.5 MiB/s wr, 519 op/s 2024-08-28T19:41:41.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 170572 22.49 MB/sec execute 317 sec latency 91.414 ms 2024-08-28T19:41:42.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 171242 22.53 MB/sec execute 318 sec latency 193.911 ms 2024-08-28T19:41:43.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:42 smithi016 ceph-mon[18556]: pgmap v987: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 8.4 MiB/s wr, 613 op/s 2024-08-28T19:41:43.277 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:42 smithi028 ceph-mon[21769]: pgmap v987: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 8.4 MiB/s wr, 613 op/s 2024-08-28T19:41:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:42 smithi115 ceph-mon[22184]: pgmap v987: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 8.4 MiB/s wr, 613 op/s 2024-08-28T19:41:43.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 172002 22.58 MB/sec execute 319 sec latency 131.311 ms 2024-08-28T19:41:44.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 172710 22.65 MB/sec execute 320 sec latency 80.676 ms 2024-08-28T19:41:45.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:44 smithi016 ceph-mon[18556]: pgmap v988: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 7.7 MiB/s wr, 554 op/s 2024-08-28T19:41:45.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:44 smithi028 ceph-mon[21769]: pgmap v988: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 7.7 MiB/s wr, 554 op/s 2024-08-28T19:41:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:44 smithi115 ceph-mon[22184]: pgmap v988: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 7.7 MiB/s wr, 554 op/s 2024-08-28T19:41:45.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 173363 22.70 MB/sec execute 321 sec latency 106.443 ms 2024-08-28T19:41:46.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 173886 22.70 MB/sec execute 322 sec latency 138.313 ms 2024-08-28T19:41:47.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:46 smithi016 ceph-mon[18556]: pgmap v989: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 11 MiB/s wr, 725 op/s 2024-08-28T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:47 smithi028 ceph-mon[21769]: pgmap v989: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 11 MiB/s wr, 725 op/s 2024-08-28T19:41:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:47 smithi115 ceph-mon[22184]: pgmap v989: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 11 MiB/s wr, 725 op/s 2024-08-28T19:41:47.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 174397 22.72 MB/sec execute 323 sec latency 250.439 ms 2024-08-28T19:41:48.472 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-08-28T19:41:48.473 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:41:48.490 INFO:tasks.workunit.client.0.smithi016.stdout: 10 175079 22.73 MB/sec execute 324 sec latency 96.864 ms 2024-08-28T19:41:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:49.181 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:49 smithi016 ceph-mon[18556]: pgmap v990: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 8.2 MiB/s wr, 544 op/s 2024-08-28T19:41:49.181 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:41:49.181 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:41:49.181 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:49 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:41:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.285+0000 7f16f9fb7640 1 -- 172.21.15.16:0/1899462469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4102880 msgr2=0x7f16f4102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.285+0000 7f16f9fb7640 1 --2- 172.21.15.16:0/1899462469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4102880 0x7f16f4102c90 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f16e4009900 tx=0x7f16e402a410 comp rx=0 tx=0).stop 2024-08-28T19:41:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.285+0000 7f16f9fb7640 1 -- 172.21.15.16:0/1899462469 shutdown_connections 2024-08-28T19:41:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.285+0000 7f16f9fb7640 1 --2- 172.21.15.16:0/1899462469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16f4108ac0 0x7f16f410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.285+0000 7f16f9fb7640 1 --2- 172.21.15.16:0/1899462469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f4108170 0x7f16f4108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.285+0000 7f16f9fb7640 1 --2- 172.21.15.16:0/1899462469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4102880 0x7f16f4102c90 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.285+0000 7f16f9fb7640 1 -- 172.21.15.16:0/1899462469 >> 172.21.15.16:0/1899462469 conn(0x7f16f4079550 msgr2=0x7f16f4079960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.285+0000 7f16f9fb7640 1 -- 172.21.15.16:0/1899462469 shutdown_connections 2024-08-28T19:41:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.285+0000 7f16f9fb7640 1 -- 172.21.15.16:0/1899462469 wait complete. 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f9fb7640 1 Processor -- start 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f9fb7640 1 -- start start 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f9fb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f4102880 0x7f16f41991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f9fb7640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16f4108170 0x7f16f4199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f9fb7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4108ac0 0x7f16f419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f9fb7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f16f4112790 con 0x7f16f4108ac0 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f9fb7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16f4112610 con 0x7f16f4102880 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f9fb7640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f16f4112910 con 0x7f16f4108170 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4108ac0 0x7f16f419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f4102880 0x7f16f41991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f4102880 0x7f16f41991e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:39190/0 (socket says 172.21.15.16:39190) 2024-08-28T19:41:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16ebfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16f4108170 0x7f16f4199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:49.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4108ac0 0x7f16f419ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35786/0 (socket says 172.21.15.16:35786) 2024-08-28T19:41:49.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.286+0000 7f16f37fe640 1 -- 172.21.15.16:0/660660036 learned_addr learned my addr 172.21.15.16:0/660660036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:49.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f37fe640 1 -- 172.21.15.16:0/660660036 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16f4108170 msgr2=0x7f16f4199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:49.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f37fe640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16f4108170 0x7f16f4199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:49.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f37fe640 1 -- 172.21.15.16:0/660660036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4108ac0 msgr2=0x7f16f419ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:49.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f37fe640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4108ac0 0x7f16f419ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:49.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f37fe640 1 -- 172.21.15.16:0/660660036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f419f150 con 0x7f16f4102880 2024-08-28T19:41:49.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f3fff640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4108ac0 0x7f16f419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:49.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16ebfff640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16f4108170 0x7f16f4199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:49.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f37fe640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f4102880 0x7f16f41991e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f16e4005940 tx=0x7f16e4004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f17fa640 1 -- 172.21.15.16:0/660660036 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e4046070 con 0x7f16f4102880 2024-08-28T19:41:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f9fb7640 1 -- 172.21.15.16:0/660660036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f419f3e0 con 0x7f16f4102880 2024-08-28T19:41:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f17fa640 1 -- 172.21.15.16:0/660660036 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16e4004440 con 0x7f16f4102880 2024-08-28T19:41:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f9fb7640 1 -- 172.21.15.16:0/660660036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f406b5c0 con 0x7f16f4102880 2024-08-28T19:41:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.287+0000 7f16f17fa640 1 -- 172.21.15.16:0/660660036 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e4035050 con 0x7f16f4102880 2024-08-28T19:41:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.288+0000 7f16f9fb7640 1 -- 172.21.15.16:0/660660036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f4103ab0 con 0x7f16f4102880 2024-08-28T19:41:49.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.291+0000 7f16f17fa640 1 -- 172.21.15.16:0/660660036 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f16e4049070 con 0x7f16f4102880 2024-08-28T19:41:49.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.292+0000 7f16f17fa640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f16e00b4ca0 0x7f16e00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:49.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.292+0000 7f16ebfff640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f16e00b4ca0 0x7f16e00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:49.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.292+0000 7f16f17fa640 1 -- 172.21.15.16:0/660660036 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f16e41075f0 con 0x7f16f4102880 2024-08-28T19:41:49.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.293+0000 7f16ebfff640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f16e00b4ca0 0x7f16e00b71b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f16f4104840 tx=0x7f16dc005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:49.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.298+0000 7f16f17fa640 1 -- 172.21.15.16:0/660660036 <== mon.2 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f16e40cd960 con 0x7f16f4102880 2024-08-28T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:49 smithi028 ceph-mon[21769]: pgmap v990: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 8.2 MiB/s wr, 544 op/s 2024-08-28T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:49 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:41:49.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:49 smithi115 ceph-mon[22184]: pgmap v990: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 8.2 MiB/s wr, 544 op/s 2024-08-28T19:41:49.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:41:49.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:41:49.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:49 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:41:49.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 175828 22.82 MB/sec execute 325 sec latency 110.569 ms 2024-08-28T19:41:49.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.610+0000 7f16f9fb7640 1 -- 172.21.15.16:0/660660036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16f4102c90 con 0x7f16f4102880 2024-08-28T19:41:49.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.611+0000 7f16f17fa640 1 -- 172.21.15.16:0/660660036 <== mon.2 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+11990 (secure 0 0 0) 0x7f16e40d2120 con 0x7f16f4102880 2024-08-28T19:41:49.621 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:41:49.621 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":102,"btime":"2024-08-28T19:41:32:464235+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":102,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:41:31.642633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:41:49.621 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 102 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.614+0000 7f16f9fb7640 1 -- 172.21.15.16:0/660660036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f16e00b4ca0 msgr2=0x7f16e00b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.614+0000 7f16f9fb7640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f16e00b4ca0 0x7f16e00b71b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f16f4104840 tx=0x7f16dc005f90 comp rx=0 tx=0).stop 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.614+0000 7f16f9fb7640 1 -- 172.21.15.16:0/660660036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f4102880 msgr2=0x7f16f41991e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.614+0000 7f16f9fb7640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f4102880 0x7f16f41991e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f16e4005940 tx=0x7f16e4004040 comp rx=0 tx=0).stop 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.614+0000 7f16f9fb7640 1 -- 172.21.15.16:0/660660036 shutdown_connections 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.615+0000 7f16f9fb7640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4108ac0 0x7f16f419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.615+0000 7f16f9fb7640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16f4108170 0x7f16f4199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.615+0000 7f16f9fb7640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f16e00b4ca0 0x7f16e00b71b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.615+0000 7f16f9fb7640 1 --2- 172.21.15.16:0/660660036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f4102880 0x7f16f41991e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.615+0000 7f16f9fb7640 1 -- 172.21.15.16:0/660660036 >> 172.21.15.16:0/660660036 conn(0x7f16f4079550 msgr2=0x7f16f4106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.615+0000 7f16f9fb7640 1 -- 172.21.15.16:0/660660036 shutdown_connections 2024-08-28T19:41:49.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:49.615+0000 7f16f9fb7640 1 -- 172.21.15.16:0/660660036 wait complete. 2024-08-28T19:41:50.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:50 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:41:50.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:50 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:41:50.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:50 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/660660036' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:50.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:50 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:41:50.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:50 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:41:50.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:50 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/660660036' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:50.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:50 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:41:50.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:50 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:41:50.396 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:50 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/660660036' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:41:50.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 176322 22.82 MB/sec execute 326 sec latency 159.951 ms 2024-08-28T19:41:50.822 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-08-28T19:41:50.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-08-28T19:41:51.205 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:51.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:51 smithi016 ceph-mon[18556]: pgmap v991: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 9.1 MiB/s wr, 579 op/s 2024-08-28T19:41:51.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:51 smithi028 ceph-mon[21769]: pgmap v991: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 9.1 MiB/s wr, 579 op/s 2024-08-28T19:41:51.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:51 smithi115 ceph-mon[22184]: pgmap v991: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 9.1 MiB/s wr, 579 op/s 2024-08-28T19:41:51.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 176952 22.84 MB/sec execute 327 sec latency 102.584 ms 2024-08-28T19:41:52.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.051+0000 7f485bd94640 1 -- 172.21.15.16:0/2883296246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4854109280 msgr2=0x7f4854109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:52.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.051+0000 7f485bd94640 1 --2- 172.21.15.16:0/2883296246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4854109280 0x7f4854109690 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f4844009900 tx=0x7f484402a410 comp rx=0 tx=0).stop 2024-08-28T19:41:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.052+0000 7f485bd94640 1 -- 172.21.15.16:0/2883296246 shutdown_connections 2024-08-28T19:41:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.052+0000 7f485bd94640 1 --2- 172.21.15.16:0/2883296246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4854109bd0 0x7f4854110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.052+0000 7f485bd94640 1 --2- 172.21.15.16:0/2883296246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4854109280 0x7f4854109690 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.052+0000 7f485bd94640 1 --2- 172.21.15.16:0/2883296246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4854103940 0x7f4854103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.052+0000 7f485bd94640 1 -- 172.21.15.16:0/2883296246 >> 172.21.15.16:0/2883296246 conn(0x7f48540ff390 msgr2=0x7f48541017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.053+0000 7f485bd94640 1 -- 172.21.15.16:0/2883296246 shutdown_connections 2024-08-28T19:41:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.054+0000 7f485bd94640 1 -- 172.21.15.16:0/2883296246 wait complete. 2024-08-28T19:41:52.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.055+0000 7f485bd94640 1 Processor -- start 2024-08-28T19:41:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.056+0000 7f485bd94640 1 -- start start 2024-08-28T19:41:52.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.057+0000 7f485bd94640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4854103940 0x7f485419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.057+0000 7f485bd94640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4854109280 0x7f485419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.057+0000 7f485bd94640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4854109bd0 0x7f485419fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.058+0000 7f485bd94640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4854113890 con 0x7f4854103940 2024-08-28T19:41:52.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.058+0000 7f485932f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4854109280 0x7f485419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.058+0000 7f485932f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4854109280 0x7f485419a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:39198/0 (socket says 172.21.15.16:39198) 2024-08-28T19:41:52.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.058+0000 7f485932f640 1 -- 172.21.15.16:0/1354850100 learned_addr learned my addr 172.21.15.16:0/1354850100 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:52.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.058+0000 7f4859b30640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4854103940 0x7f485419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.058+0000 7f485a331640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4854109bd0 0x7f485419fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.058+0000 7f485bd94640 1 -- 172.21.15.16:0/1354850100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4854113710 con 0x7f4854109280 2024-08-28T19:41:52.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.058+0000 7f485bd94640 1 -- 172.21.15.16:0/1354850100 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4854113a10 con 0x7f4854109bd0 2024-08-28T19:41:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.059+0000 7f485a331640 1 -- 172.21.15.16:0/1354850100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4854109280 msgr2=0x7f485419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.059+0000 7f485a331640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4854109280 0x7f485419a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.059+0000 7f485a331640 1 -- 172.21.15.16:0/1354850100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4854103940 msgr2=0x7f485419a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.059+0000 7f485a331640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4854103940 0x7f485419a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.059+0000 7f485a331640 1 -- 172.21.15.16:0/1354850100 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48541a01b0 con 0x7f4854109bd0 2024-08-28T19:41:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.059+0000 7f485a331640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4854109bd0 0x7f485419fa80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f484c00e940 tx=0x7f484c00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.060+0000 7f484affd640 1 -- 172.21.15.16:0/1354850100 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c004d20 con 0x7f4854109bd0 2024-08-28T19:41:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.060+0000 7f485bd94640 1 -- 172.21.15.16:0/1354850100 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48541a04a0 con 0x7f4854109bd0 2024-08-28T19:41:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.060+0000 7f484affd640 1 -- 172.21.15.16:0/1354850100 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f484c013770 con 0x7f4854109bd0 2024-08-28T19:41:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.060+0000 7f484affd640 1 -- 172.21.15.16:0/1354850100 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c0052f0 con 0x7f4854109bd0 2024-08-28T19:41:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.061+0000 7f485bd94640 1 -- 172.21.15.16:0/1354850100 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4854075390 con 0x7f4854109bd0 2024-08-28T19:41:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.065+0000 7f4862309740 1 Processor -- start 2024-08-28T19:41:52.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.066+0000 7f484affd640 1 -- 172.21.15.16:0/1354850100 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f484c00cb20 con 0x7f4854109bd0 2024-08-28T19:41:52.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.066+0000 7f4862309740 1 -- start start 2024-08-28T19:41:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f4862309740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 0x55c890e754a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f4862309740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890a9c980 0x55c890e759e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f4862309740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890aa4b60 0x55c890e75f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f4862309740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c890de1330 con 0x55c890aa4b60 2024-08-28T19:41:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f4862309740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c890dd08c0 con 0x55c890a9c980 2024-08-28T19:41:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f4862309740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c890dd4870 con 0x55c890ac5840 2024-08-28T19:41:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f485932f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 0x55c890e754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f485932f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 0x55c890e754a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:53520/0 (socket says 172.21.15.16:53520) 2024-08-28T19:41:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f485932f640 1 -- 172.21.15.16:0/1880740468 learned_addr learned my addr 172.21.15.16:0/1880740468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f4859b30640 1 --2- 172.21.15.16:0/1880740468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890a9c980 0x55c890e759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f485932f640 1 -- 172.21.15.16:0/1880740468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890a9c980 msgr2=0x55c890e759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f485932f640 1 --2- 172.21.15.16:0/1880740468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890a9c980 0x55c890e759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f485932f640 1 -- 172.21.15.16:0/1880740468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890aa4b60 msgr2=0x55c890e75f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f485932f640 1 --2- 172.21.15.16:0/1880740468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890aa4b60 0x55c890e75f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.067+0000 7f485932f640 1 -- 172.21.15.16:0/1880740468 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c890d88df0 con 0x55c890ac5840 2024-08-28T19:41:52.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.069+0000 7f484affd640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48400b4d20 0x7f48400b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.069+0000 7f485932f640 1 --2- 172.21.15.16:0/1880740468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 0x55c890e754a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4844006010 tx=0x7f4844032c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28ee79f55d588ec0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:52.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.069+0000 7f483ffff640 1 -- 172.21.15.16:0/1880740468 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844035030 con 0x55c890ac5840 2024-08-28T19:41:52.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.069+0000 7f484affd640 1 -- 172.21.15.16:0/1354850100 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f484c0e54c0 con 0x7f4854109bd0 2024-08-28T19:41:52.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.069+0000 7f4859b30640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48400b4d20 0x7f48400b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.070+0000 7f483ffff640 1 -- 172.21.15.16:0/1880740468 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4844041070 con 0x55c890ac5840 2024-08-28T19:41:52.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.070+0000 7f483ffff640 1 -- 172.21.15.16:0/1880740468 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844035030 con 0x55c890ac5840 2024-08-28T19:41:52.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.071+0000 7f4862309740 1 -- 172.21.15.16:0/1880740468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 msgr2=0x55c890e754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:52.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.071+0000 7f4862309740 1 --2- 172.21.15.16:0/1880740468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 0x55c890e754a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4844006010 tx=0x7f4844032c60 comp rx=0 tx=0).stop 2024-08-28T19:41:52.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.071+0000 7f4859b30640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48400b4d20 0x7f48400b7230 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x55c890e74120 tx=0x7f4850005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:52.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.071+0000 7f4862309740 1 -- 172.21.15.16:0/1880740468 shutdown_connections 2024-08-28T19:41:52.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.071+0000 7f4862309740 1 --2- 172.21.15.16:0/1880740468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890aa4b60 0x55c890e75f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.071+0000 7f4862309740 1 --2- 172.21.15.16:0/1880740468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890a9c980 0x55c890e759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.071+0000 7f4862309740 1 --2- 172.21.15.16:0/1880740468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 0x55c890e754a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.071+0000 7f4862309740 1 -- 172.21.15.16:0/1880740468 >> 172.21.15.16:0/1880740468 conn(0x55c890ab9990 msgr2=0x55c890e73390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:52.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.071+0000 7f4862309740 1 -- 172.21.15.16:0/1880740468 shutdown_connections 2024-08-28T19:41:52.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.071+0000 7f4862309740 1 -- 172.21.15.16:0/1880740468 wait complete. 2024-08-28T19:41:52.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.072+0000 7f4862309740 1 -- start start 2024-08-28T19:41:52.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.072+0000 7f4862309740 1 Processor -- start 2024-08-28T19:41:52.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f4862309740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a9c980 0x55c890e80b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f4862309740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e810d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f4862309740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 0x55c890e8cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f4859b30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a9c980 0x55c890e80b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f4862309740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c890c337a0 con 0x55c890a9c980 2024-08-28T19:41:52.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f4859b30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a9c980 0x55c890e80b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35816/0 (socket says 172.21.15.16:35816) 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f4859b30640 1 -- 172.21.15.16:0/3816436937 learned_addr learned my addr 172.21.15.16:0/3816436937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f4862309740 1 -- 172.21.15.16:0/3816436937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c890d57240 con 0x55c890aa4b60 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f4862309740 1 -- 172.21.15.16:0/3816436937 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c890e83c70 con 0x55c890ac5840 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f485932f640 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e810d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f485a331640 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 0x55c890e8cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.073+0000 7f4862309740 10 client.0 _ll_register_callbacks cb 0x55c890add090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.074+0000 7f4859b30640 1 -- 172.21.15.16:0/3816436937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 msgr2=0x55c890e8cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.074+0000 7f4859b30640 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 0x55c890e8cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.074+0000 7f4859b30640 1 -- 172.21.15.16:0/3816436937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 msgr2=0x55c890e810d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.074+0000 7f4859b30640 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e810d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.074+0000 7f4859b30640 1 -- 172.21.15.16:0/3816436937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c890d8d2b0 con 0x55c890a9c980 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.074+0000 7f485932f640 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e810d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:41:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.074+0000 7f4859b30640 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a9c980 0x55c890e80b90 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f48500045f0 tx=0x7f4850004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccbd7a4024577900 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.074+0000 7f483d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.074+0000 7f483d7fa640 1 -- 172.21.15.16:0/3816436937 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485001f640 con 0x55c890a9c980 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.074+0000 7f483d7fa640 1 -- 172.21.15.16:0/3816436937 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f485001f7e0 con 0x55c890a9c980 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.074+0000 7f4862309740 1 -- 172.21.15.16:0/3816436937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4844041070 con 0x55c890a9c980 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f483d7fa640 1 -- 172.21.15.16:0/3816436937 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f485001f9e0 con 0x55c890a9c980 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f4862309740 10 client.20256 fetch_fsmap learned FSMap version 103 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f4862309740 1 -- 172.21.15.16:0/3816436937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55c890b3fb90 con 0x55c890a9c980 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f483d7fa640 1 -- 172.21.15.16:0/3816436937 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f485001fb80 con 0x55c890a9c980 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f483d7fa640 1 -- 172.21.15.16:0/3816436937 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 103) ==== 8394+0+0 (secure 0 0 0) 0x7f4850027070 con 0x55c890a9c980 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f4862309740 10 client.20256 fetch_fsmap finished waiting for FSMap version 103 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f4862309740 10 client.20256 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f4862309740 20 client.20256 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f4862309740 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c890a841c0 0x55c890e92df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f4862309740 4 client.20256 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:41:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f4862309740 1 -- 172.21.15.16:0/3816436937 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4844041070 con 0x55c890a841c0 2024-08-28T19:41:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.075+0000 7f485932f640 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c890a841c0 0x55c890e92df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.076+0000 7f483d7fa640 10 client.20256 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.076+0000 7f485932f640 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c890a841c0 0x55c890e92df0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8e88d5ed393f9db server_cookie=5d6b2a5f57aa8487 in_seq=0 out_seq=0 2024-08-28T19:41:52.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.077+0000 7f483d7fa640 1 -- 172.21.15.16:0/3816436937 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f4844041070 con 0x55c890a841c0 2024-08-28T19:41:52.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.077+0000 7f483d7fa640 10 client.20256 handle_command_reply: tid=0 2024-08-28T19:41:52.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.077+0000 7f4862309740 1 client.20256 shutdown 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f4862309740 1 -- 172.21.15.16:0/3816436937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a9c980 msgr2=0x55c890e80b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f4862309740 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a9c980 0x55c890e80b90 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f48500045f0 tx=0x7f4850004730 comp rx=0 tx=0).stop 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f4862309740 1 -- 172.21.15.16:0/3816436937 shutdown_connections 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f4862309740 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c890a841c0 0x55c890e92df0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f483d7fa640 0 client.20256 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f4862309740 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890ac5840 0x55c890e8cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f4862309740 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e810d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f4862309740 1 --2- 172.21.15.16:0/3816436937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a9c980 0x55c890e80b90 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f4862309740 1 -- 172.21.15.16:0/3816436937 >> 172.21.15.16:0/3816436937 conn(0x55c890ab9990 msgr2=0x55c890e764d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f4862309740 1 -- 172.21.15.16:0/3816436937 shutdown_connections 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f4862309740 1 -- 172.21.15.16:0/3816436937 wait complete. 2024-08-28T19:41:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.078+0000 7f4862309740 20 client.20256 trim_cache size 0 max 16384 2024-08-28T19:41:52.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f4862309740 1 Processor -- start 2024-08-28T19:41:52.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f4862309740 1 -- start start 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f4862309740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 0x55c890e73250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f4862309740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890a9c980 0x55c890e827c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f4862309740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e8ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f4862309740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48541aea30 con 0x55c890a841c0 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f4862309740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48541b0690 con 0x55c890aa4b60 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f4862309740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f48541af130 con 0x55c890a9c980 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f4859b30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 0x55c890e73250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f4859b30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 0x55c890e73250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35824/0 (socket says 172.21.15.16:35824) 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f4859b30640 1 -- 172.21.15.16:0/278471664 learned_addr learned my addr 172.21.15.16:0/278471664 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.100+0000 7f485932f640 1 --2- 172.21.15.16:0/278471664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890a9c980 0x55c890e827c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.101+0000 7f485a331640 1 --2- 172.21.15.16:0/278471664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e8ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.101+0000 7f485932f640 1 -- 172.21.15.16:0/278471664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 msgr2=0x55c890e8ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:52.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.101+0000 7f485932f640 1 --2- 172.21.15.16:0/278471664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e8ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.101+0000 7f485932f640 1 -- 172.21.15.16:0/278471664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 msgr2=0x55c890e73250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:52.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.101+0000 7f485932f640 1 --2- 172.21.15.16:0/278471664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 0x55c890e73250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.101+0000 7f485932f640 1 -- 172.21.15.16:0/278471664 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c890da0d90 con 0x55c890a9c980 2024-08-28T19:41:52.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.101+0000 7f485932f640 1 --2- 172.21.15.16:0/278471664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890a9c980 0x55c890e827c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55c890e93590 tx=0x7f48440050c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b16cf6078b67b7da server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:52.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.101+0000 7f483f7fe640 1 -- 172.21.15.16:0/278471664 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844035030 con 0x55c890a9c980 2024-08-28T19:41:52.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.102+0000 7f483f7fe640 1 -- 172.21.15.16:0/278471664 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4844041070 con 0x55c890a9c980 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.102+0000 7f483f7fe640 1 -- 172.21.15.16:0/278471664 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48440049b0 con 0x55c890a9c980 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.103+0000 7f4862309740 1 -- 172.21.15.16:0/278471664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890a9c980 msgr2=0x55c890e827c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.103+0000 7f4862309740 1 --2- 172.21.15.16:0/278471664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890a9c980 0x55c890e827c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55c890e93590 tx=0x7f48440050c0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.103+0000 7f4862309740 1 -- 172.21.15.16:0/278471664 shutdown_connections 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.103+0000 7f4862309740 1 --2- 172.21.15.16:0/278471664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e8ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.103+0000 7f4862309740 1 --2- 172.21.15.16:0/278471664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890a9c980 0x55c890e827c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.103+0000 7f4862309740 1 --2- 172.21.15.16:0/278471664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 0x55c890e73250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.103+0000 7f4862309740 1 -- 172.21.15.16:0/278471664 >> 172.21.15.16:0/278471664 conn(0x55c890ab9990 msgr2=0x55c890e820b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.103+0000 7f4862309740 1 -- 172.21.15.16:0/278471664 shutdown_connections 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.103+0000 7f4862309740 1 -- 172.21.15.16:0/278471664 wait complete. 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.103+0000 7f4862309740 1 -- start start 2024-08-28T19:41:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.103+0000 7f4862309740 1 Processor -- start 2024-08-28T19:41:52.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4862309740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 0x55c890e890d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4862309740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890a9c980 0x55c890e896b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4862309740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e83e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4862309740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48541aff00 con 0x55c890a841c0 2024-08-28T19:41:52.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4862309740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48541af800 con 0x55c890aa4b60 2024-08-28T19:41:52.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4862309740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f48541aca30 con 0x55c890a9c980 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4862309740 10 client.0 _ll_register_callbacks cb 0x7f4850009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4859b30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 0x55c890e890d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4859b30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 0x55c890e890d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35832/0 (socket says 172.21.15.16:35832) 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4859b30640 1 -- 172.21.15.16:0/1997688929 learned_addr learned my addr 172.21.15.16:0/1997688929 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f485a331640 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e83e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4859b30640 1 -- 172.21.15.16:0/1997688929 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890a9c980 msgr2=0x55c890e896b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4859b30640 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890a9c980 0x55c890e896b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4859b30640 1 -- 172.21.15.16:0/1997688929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 msgr2=0x55c890e83e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4859b30640 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e83e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.104+0000 7f4859b30640 1 -- 172.21.15.16:0/1997688929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c890e84590 con 0x55c890a841c0 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.105+0000 7f4859b30640 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 0x55c890e890d0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f48500092e0 tx=0x7f485001d840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79cf480447ee61e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.105+0000 7f483d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.105+0000 7f483d7fa640 1 -- 172.21.15.16:0/1997688929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485001d9b0 con 0x55c890a841c0 2024-08-28T19:41:52.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.105+0000 7f4862309740 1 -- 172.21.15.16:0/1997688929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4844041070 con 0x55c890a841c0 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.105+0000 7f483d7fa640 1 -- 172.21.15.16:0/1997688929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f485001db50 con 0x55c890a841c0 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.105+0000 7f483d7fa640 1 -- 172.21.15.16:0/1997688929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f485001dd50 con 0x55c890a841c0 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.105+0000 7f4862309740 10 client.20262 fetch_fsmap learned FSMap version 103 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.105+0000 7f4862309740 1 -- 172.21.15.16:0/1997688929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55c890efe180 con 0x55c890a841c0 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.106+0000 7f483d7fa640 1 -- 172.21.15.16:0/1997688929 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4844041070 con 0x55c890a841c0 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.106+0000 7f483d7fa640 1 -- 172.21.15.16:0/1997688929 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 103) ==== 8394+0+0 (secure 0 0 0) 0x7f4850027050 con 0x55c890a841c0 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.106+0000 7f4862309740 10 client.20262 fetch_fsmap finished waiting for FSMap version 103 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.106+0000 7f4862309740 10 client.20262 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.106+0000 7f4862309740 20 client.20262 populate_metadata read hostname 'smithi016' 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.106+0000 7f4862309740 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c890ac5840 0x55c890f00af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.106+0000 7f4862309740 4 client.20262 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.106+0000 7f4862309740 1 -- 172.21.15.16:0/1997688929 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x55c890de3280 con 0x55c890ac5840 2024-08-28T19:41:52.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.107+0000 7f485932f640 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c890ac5840 0x55c890f00af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:52.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.107+0000 7f483d7fa640 10 client.20262 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:52.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.107+0000 7f485932f640 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c890ac5840 0x55c890f00af0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81f1771d9fc2d7dc server_cookie=12995bcb5a4e000e in_seq=0 out_seq=0 2024-08-28T19:41:52.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177360 22.86 MB/sec execute 328 sec latency 342.314 ms 2024-08-28T19:41:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f483d7fa640 1 -- 172.21.15.16:0/1997688929 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+592 (crc 0 0 0) 0x55c890de3280 con 0x55c890ac5840 2024-08-28T19:41:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f483d7fa640 10 client.20262 handle_command_reply: tid=0 2024-08-28T19:41:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f4862309740 1 client.20262 shutdown 2024-08-28T19:41:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f4862309740 1 -- 172.21.15.16:0/1997688929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 msgr2=0x55c890e890d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f4862309740 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 0x55c890e890d0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f48500092e0 tx=0x7f485001d840 comp rx=0 tx=0).stop 2024-08-28T19:41:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f4862309740 1 -- 172.21.15.16:0/1997688929 shutdown_connections 2024-08-28T19:41:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f4862309740 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c890aa4b60 0x55c890e83e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f4862309740 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c890ac5840 0x55c890f00af0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f483d7fa640 0 client.20262 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:41:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f4862309740 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c890a9c980 0x55c890e896b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f4862309740 1 --2- 172.21.15.16:0/1997688929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c890a841c0 0x55c890e890d0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.693+0000 7f4862309740 1 -- 172.21.15.16:0/1997688929 >> 172.21.15.16:0/1997688929 conn(0x55c890ab9990 msgr2=0x55c890e764d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:52.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.694+0000 7f4862309740 1 -- 172.21.15.16:0/1997688929 shutdown_connections 2024-08-28T19:41:52.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.694+0000 7f4862309740 1 -- 172.21.15.16:0/1997688929 wait complete. 2024-08-28T19:41:52.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.694+0000 7f4862309740 20 client.20262 trim_cache size 0 max 16384 2024-08-28T19:41:52.696 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:41:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 103, 2024-08-28T19:41:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:41:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 66, 2024-08-28T19:41:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:41:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: "5b45ee2f": { 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 66, 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 130.0, 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:41:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:41:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:41:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:41:52.698 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:41:52.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.694+0000 7f485bd94640 1 -- 172.21.15.16:0/1354850100 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48400b4d20 msgr2=0x7f48400b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.694+0000 7f485bd94640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48400b4d20 0x7f48400b7230 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x55c890e74120 tx=0x7f4850005f90 comp rx=0 tx=0).stop 2024-08-28T19:41:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.694+0000 7f485bd94640 1 -- 172.21.15.16:0/1354850100 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4854109bd0 msgr2=0x7f485419fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.694+0000 7f485bd94640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4854109bd0 0x7f485419fa80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f484c00e940 tx=0x7f484c00ee00 comp rx=0 tx=0).stop 2024-08-28T19:41:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.695+0000 7f485bd94640 1 -- 172.21.15.16:0/1354850100 shutdown_connections 2024-08-28T19:41:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.695+0000 7f485bd94640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4854109bd0 0x7f485419fa80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.695+0000 7f485bd94640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f48400b4d20 0x7f48400b7230 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.695+0000 7f485bd94640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4854109280 0x7f485419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.695+0000 7f485bd94640 1 --2- 172.21.15.16:0/1354850100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4854103940 0x7f485419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.695+0000 7f485bd94640 1 -- 172.21.15.16:0/1354850100 >> 172.21.15.16:0/1354850100 conn(0x7f48540ff390 msgr2=0x7f48541017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.695+0000 7f485bd94640 1 -- 172.21.15.16:0/1354850100 shutdown_connections 2024-08-28T19:41:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:52.695+0000 7f485bd94640 1 -- 172.21.15.16:0/1354850100 wait complete. 2024-08-28T19:41:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:53 smithi028 ceph-mon[21769]: pgmap v992: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 11 MiB/s wr, 686 op/s 2024-08-28T19:41:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:53 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:41:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:53 smithi115 ceph-mon[22184]: pgmap v992: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 11 MiB/s wr, 686 op/s 2024-08-28T19:41:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:53 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:41:53.418 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:53 smithi016 ceph-mon[18556]: pgmap v992: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 11 MiB/s wr, 686 op/s 2024-08-28T19:41:53.418 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:53 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:41:53.446 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5b45ee2f', quiesce took 5.0 seconds. Will release after: 5.0 2024-08-28T19:41:53.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177363 22.79 MB/sec execute 329 sec latency 1342.425 ms 2024-08-28T19:41:54.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177363 22.72 MB/sec execute 330 sec latency 2342.465 ms 2024-08-28T19:41:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:55 smithi028 ceph-mon[21769]: pgmap v993: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 7.9 MiB/s wr, 499 op/s 2024-08-28T19:41:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:55 smithi115 ceph-mon[22184]: pgmap v993: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 7.9 MiB/s wr, 499 op/s 2024-08-28T19:41:55.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177363 22.65 MB/sec execute 331 sec latency 3342.514 ms 2024-08-28T19:41:55.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:55 smithi016 ceph-mon[18556]: pgmap v993: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 7.9 MiB/s wr, 499 op/s 2024-08-28T19:41:56.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177363 22.58 MB/sec execute 332 sec latency 4342.557 ms 2024-08-28T19:41:57.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:57 smithi028 ceph-mon[21769]: pgmap v994: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 587 op/s 2024-08-28T19:41:57.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:57 smithi115 ceph-mon[22184]: pgmap v994: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 587 op/s 2024-08-28T19:41:57.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177363 22.52 MB/sec execute 333 sec latency 5342.598 ms 2024-08-28T19:41:57.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:57 smithi016 ceph-mon[18556]: pgmap v994: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 11 MiB/s wr, 587 op/s 2024-08-28T19:41:58.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:41:58.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177363 22.45 MB/sec execute 334 sec latency 6342.655 ms 2024-08-28T19:41:58.700 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:41:59.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:41:59 smithi016 ceph-mon[18556]: pgmap v995: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 11 MiB/s wr, 485 op/s 2024-08-28T19:41:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.261+0000 7f8673fff640 1 -- 172.21.15.16:0/3655068089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8674102620 msgr2=0x7f8674102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.261+0000 7f8673fff640 1 --2- 172.21.15.16:0/3655068089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8674102620 0x7f8674102a30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8664009900 tx=0x7f866402a440 comp rx=0 tx=0).stop 2024-08-28T19:41:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.261+0000 7f8673fff640 1 -- 172.21.15.16:0/3655068089 shutdown_connections 2024-08-28T19:41:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.261+0000 7f8673fff640 1 --2- 172.21.15.16:0/3655068089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86741088b0 0x7f867410f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.261+0000 7f8673fff640 1 --2- 172.21.15.16:0/3655068089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8674107f60 0x7f8674108370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.261+0000 7f8673fff640 1 --2- 172.21.15.16:0/3655068089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8674102620 0x7f8674102a30 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8664009900 tx=0x7f866402a440 comp rx=0 tx=0).stop 2024-08-28T19:41:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.261+0000 7f8673fff640 1 -- 172.21.15.16:0/3655068089 >> 172.21.15.16:0/3655068089 conn(0x7f86740fe050 msgr2=0x7f86741004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:59.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.264+0000 7f8673fff640 1 -- 172.21.15.16:0/3655068089 shutdown_connections 2024-08-28T19:41:59.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.264+0000 7f8673fff640 1 -- 172.21.15.16:0/3655068089 wait complete. 2024-08-28T19:41:59.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.265+0000 7f8673fff640 1 Processor -- start 2024-08-28T19:41:59.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.266+0000 7f8673fff640 1 -- start start 2024-08-28T19:41:59.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.266+0000 7f8673fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8674107f60 0x7f867410c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:59.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.266+0000 7f8673fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86741088b0 0x7f867410cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:59.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.266+0000 7f8673fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f867410d080 0x7f86741a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:59.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.266+0000 7f8673fff640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8674111610 con 0x7f86741088b0 2024-08-28T19:41:59.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.266+0000 7f8673fff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8674111490 con 0x7f867410d080 2024-08-28T19:41:59.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.266+0000 7f8673fff640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8674115c40 con 0x7f8674107f60 2024-08-28T19:41:59.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.267+0000 7f86727fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86741088b0 0x7f867410cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:59.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.267+0000 7f86727fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86741088b0 0x7f867410cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50872/0 (socket says 172.21.15.16:50872) 2024-08-28T19:41:59.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.267+0000 7f86727fc640 1 -- 172.21.15.16:0/1646719275 learned_addr learned my addr 172.21.15.16:0/1646719275 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:41:59.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.268+0000 7f86727fc640 1 -- 172.21.15.16:0/1646719275 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8674107f60 msgr2=0x7f867410c600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-28T19:41:59.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.268+0000 7f86727fc640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8674107f60 0x7f867410c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:59.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.268+0000 7f86727fc640 1 -- 172.21.15.16:0/1646719275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f867410d080 msgr2=0x7f86741a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:59.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.268+0000 7f86737fe640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f867410d080 0x7f86741a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:59.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.268+0000 7f86727fc640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f867410d080 0x7f86741a1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:59.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.268+0000 7f86727fc640 1 -- 172.21.15.16:0/1646719275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86741a2620 con 0x7f86741088b0 2024-08-28T19:41:59.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.268+0000 7f86737fe640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f867410d080 0x7f86741a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:41:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.269+0000 7f86727fc640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86741088b0 0x7f867410cb40 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f866800b450 tx=0x7f866800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:59.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.270+0000 7f865bfff640 1 -- 172.21.15.16:0/1646719275 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8668015020 con 0x7f86741088b0 2024-08-28T19:41:59.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.270+0000 7f8673fff640 1 -- 172.21.15.16:0/1646719275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86741a2910 con 0x7f86741088b0 2024-08-28T19:41:59.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.270+0000 7f8673fff640 1 -- 172.21.15.16:0/1646719275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86741a2f20 con 0x7f86741088b0 2024-08-28T19:41:59.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.271+0000 7f865bfff640 1 -- 172.21.15.16:0/1646719275 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8668011d40 con 0x7f86741088b0 2024-08-28T19:41:59.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.271+0000 7f865bfff640 1 -- 172.21.15.16:0/1646719275 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866800f340 con 0x7f86741088b0 2024-08-28T19:41:59.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.272+0000 7f8673fff640 1 -- 172.21.15.16:0/1646719275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86600052c0 con 0x7f86741088b0 2024-08-28T19:41:59.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.278+0000 7f865bfff640 1 -- 172.21.15.16:0/1646719275 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8668018020 con 0x7f86741088b0 2024-08-28T19:41:59.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.279+0000 7f865bfff640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f865c0b4dd0 0x7f865c0b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:41:59.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.279+0000 7f865bfff640 1 -- 172.21.15.16:0/1646719275 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f86680e5400 con 0x7f86741088b0 2024-08-28T19:41:59.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.280+0000 7f8672ffd640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f865c0b4dd0 0x7f865c0b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:41:59.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.280+0000 7f8672ffd640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f865c0b4dd0 0x7f865c0b72e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8664009230 tx=0x7f8664005fd0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:41:59.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.283+0000 7f865bfff640 1 -- 172.21.15.16:0/1646719275 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f866801f020 con 0x7f86741088b0 2024-08-28T19:41:59.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:41:59 smithi028 ceph-mon[21769]: pgmap v995: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 11 MiB/s wr, 485 op/s 2024-08-28T19:41:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:41:59 smithi115 ceph-mon[22184]: pgmap v995: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 11 MiB/s wr, 485 op/s 2024-08-28T19:41:59.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177363 22.38 MB/sec execute 335 sec latency 7342.700 ms 2024-08-28T19:41:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.565+0000 7f8673fff640 1 -- 172.21.15.16:0/1646719275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86600055b0 con 0x7f86741088b0 2024-08-28T19:41:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.566+0000 7f865bfff640 1 -- 172.21.15.16:0/1646719275 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+11988 (secure 0 0 0) 0x7f866800d040 con 0x7f86741088b0 2024-08-28T19:41:59.567 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:41:59.567 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":103,"btime":"2024-08-28T19:41:52:010016+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":103,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:41:51.643742+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:41:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 103 2024-08-28T19:41:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 -- 172.21.15.16:0/1646719275 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f865c0b4dd0 msgr2=0x7f865c0b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f865c0b4dd0 0x7f865c0b72e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8664009230 tx=0x7f8664005fd0 comp rx=0 tx=0).stop 2024-08-28T19:41:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 -- 172.21.15.16:0/1646719275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86741088b0 msgr2=0x7f867410cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:41:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86741088b0 0x7f867410cb40 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f866800b450 tx=0x7f866800b910 comp rx=0 tx=0).stop 2024-08-28T19:41:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 -- 172.21.15.16:0/1646719275 shutdown_connections 2024-08-28T19:41:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f865c0b4dd0 0x7f865c0b72e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f867410d080 0x7f86741a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86741088b0 0x7f867410cb40 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 --2- 172.21.15.16:0/1646719275 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8674107f60 0x7f867410c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:41:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 -- 172.21.15.16:0/1646719275 >> 172.21.15.16:0/1646719275 conn(0x7f86740fe050 msgr2=0x7f8674100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:41:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 -- 172.21.15.16:0/1646719275 shutdown_connections 2024-08-28T19:41:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:41:59.568+0000 7f8673fff640 1 -- 172.21.15.16:0/1646719275 wait complete. 2024-08-28T19:42:00.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:00 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1646719275' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:00.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:00 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1646719275' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:00.361 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id 5b45ee2f --release --await' 2024-08-28T19:42:00.361 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id 5b45ee2f --release --await 2024-08-28T19:42:00.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:00 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1646719275' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:00.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177363 22.32 MB/sec execute 336 sec latency 8342.737 ms 2024-08-28T19:42:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:01.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:01 smithi016 ceph-mon[18556]: pgmap v996: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 533 op/s 2024-08-28T19:42:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:01 smithi028 ceph-mon[21769]: pgmap v996: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 533 op/s 2024-08-28T19:42:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:01 smithi115 ceph-mon[22184]: pgmap v996: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 533 op/s 2024-08-28T19:42:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.372+0000 7f221b49b640 1 -- 172.21.15.16:0/4167026911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2214103900 msgr2=0x7f2214103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.372+0000 7f221b49b640 1 --2- 172.21.15.16:0/4167026911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2214103900 0x7f2214103d10 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f2210009900 tx=0x7f221002a410 comp rx=0 tx=0).stop 2024-08-28T19:42:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.372+0000 7f221b49b640 1 -- 172.21.15.16:0/4167026911 shutdown_connections 2024-08-28T19:42:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.372+0000 7f221b49b640 1 --2- 172.21.15.16:0/4167026911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2214109b90 0x7f22141103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.372+0000 7f221b49b640 1 --2- 172.21.15.16:0/4167026911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2214109240 0x7f2214109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.372+0000 7f221b49b640 1 --2- 172.21.15.16:0/4167026911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2214103900 0x7f2214103d10 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.372+0000 7f221b49b640 1 -- 172.21.15.16:0/4167026911 >> 172.21.15.16:0/4167026911 conn(0x7f221407b5f0 msgr2=0x7f221407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.372+0000 7f221b49b640 1 -- 172.21.15.16:0/4167026911 shutdown_connections 2024-08-28T19:42:01.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.372+0000 7f221b49b640 1 -- 172.21.15.16:0/4167026911 wait complete. 2024-08-28T19:42:01.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.373+0000 7f221b49b640 1 Processor -- start 2024-08-28T19:42:01.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.373+0000 7f221b49b640 1 -- start start 2024-08-28T19:42:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.373+0000 7f221b49b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2214103900 0x7f221419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.373+0000 7f221b49b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2214109240 0x7f221419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.373+0000 7f221b49b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2214109b90 0x7f221419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.373+0000 7f221b49b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2214113850 con 0x7f2214109240 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.373+0000 7f221b49b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22141136d0 con 0x7f2214103900 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.373+0000 7f221b49b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f22141139d0 con 0x7f2214109b90 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2218a36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2214109240 0x7f221419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2218a36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2214109240 0x7f221419a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50894/0 (socket says 172.21.15.16:50894) 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2218a36640 1 -- 172.21.15.16:0/3514757434 learned_addr learned my addr 172.21.15.16:0/3514757434 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2219237640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2214103900 0x7f221419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2219a38640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2214109b90 0x7f221419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2218a36640 1 -- 172.21.15.16:0/3514757434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2214109b90 msgr2=0x7f221419fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2218a36640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2214109b90 0x7f221419fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2218a36640 1 -- 172.21.15.16:0/3514757434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2214103900 msgr2=0x7f221419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2218a36640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2214103900 0x7f221419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2218a36640 1 -- 172.21.15.16:0/3514757434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22141a0170 con 0x7f2214109240 2024-08-28T19:42:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2219a38640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2214109b90 0x7f221419fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:42:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f2218a36640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2214109240 0x7f221419a7a0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f220400d740 tx=0x7f220400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f220a7fc640 1 -- 172.21.15.16:0/3514757434 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204017070 con 0x7f2214109240 2024-08-28T19:42:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f220a7fc640 1 -- 172.21.15.16:0/3514757434 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f220400de50 con 0x7f2214109240 2024-08-28T19:42:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f221b49b640 1 -- 172.21.15.16:0/3514757434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22141a0460 con 0x7f2214109240 2024-08-28T19:42:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f220a7fc640 1 -- 172.21.15.16:0/3514757434 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220400f620 con 0x7f2214109240 2024-08-28T19:42:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.374+0000 7f221b49b640 1 -- 172.21.15.16:0/3514757434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22141a6bc0 con 0x7f2214109240 2024-08-28T19:42:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.377+0000 7f2221a10740 1 Processor -- start 2024-08-28T19:42:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.377+0000 7f2221a10740 1 -- start start 2024-08-28T19:42:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.377+0000 7f220a7fc640 1 -- 172.21.15.16:0/3514757434 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f220401a070 con 0x7f2214109240 2024-08-28T19:42:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.377+0000 7f2221a10740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c632ad520 0x561c63652c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.377+0000 7f2221a10740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c63284660 0x561c636531c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.377+0000 7f2221a10740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 0x561c63653700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.377+0000 7f2221a10740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561c635c9a70 con 0x561c63284660 2024-08-28T19:42:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.377+0000 7f2221a10740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561c635b9000 con 0x561c6328c840 2024-08-28T19:42:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.377+0000 7f2221a10740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561c635bcfb0 con 0x561c632ad520 2024-08-28T19:42:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2218a36640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 0x561c63653700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2219237640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c63284660 0x561c636531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2219237640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c63284660 0x561c636531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50910/0 (socket says 172.21.15.16:50910) 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2219237640 1 -- 172.21.15.16:0/4076028287 learned_addr learned my addr 172.21.15.16:0/4076028287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2219a38640 1 --2- 172.21.15.16:0/4076028287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c632ad520 0x561c63652c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2218a36640 1 -- 172.21.15.16:0/4076028287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c632ad520 msgr2=0x561c63652c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2218a36640 1 --2- 172.21.15.16:0/4076028287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c632ad520 0x561c63652c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2218a36640 1 -- 172.21.15.16:0/4076028287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c63284660 msgr2=0x561c636531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2218a36640 1 --2- 172.21.15.16:0/4076028287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c63284660 0x561c636531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2218a36640 1 -- 172.21.15.16:0/4076028287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c63318b00 con 0x561c6328c840 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2219a38640 1 --2- 172.21.15.16:0/4076028287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c632ad520 0x561c63652c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2218a36640 1 --2- 172.21.15.16:0/4076028287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 0x561c63653700 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f22040e26f0 tx=0x7f22040eda00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70499c0ca0233c89 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f220a7fc640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22000b4ca0 0x7f22000b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f2219a38640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22000b4ca0 0x7f22000b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.378+0000 7f21ff7fe640 1 -- 172.21.15.16:0/4076028287 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22040f9070 con 0x561c6328c840 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.379+0000 7f220a7fc640 1 -- 172.21.15.16:0/3514757434 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f22040e5370 con 0x7f2214109240 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.379+0000 7f21ff7fe640 1 -- 172.21.15.16:0/4076028287 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22040ab1a0 con 0x561c6328c840 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.379+0000 7f21ff7fe640 1 -- 172.21.15.16:0/4076028287 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22040aae00 con 0x561c6328c840 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.379+0000 7f2221a10740 1 -- 172.21.15.16:0/4076028287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 msgr2=0x561c63653700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.379+0000 7f2221a10740 1 --2- 172.21.15.16:0/4076028287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 0x561c63653700 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f22040e26f0 tx=0x7f22040eda00 comp rx=0 tx=0).stop 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.379+0000 7f2219a38640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22000b4ca0 0x7f22000b71b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f220c005fd0 tx=0x7f220c009590 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.379+0000 7f2221a10740 1 -- 172.21.15.16:0/4076028287 shutdown_connections 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.379+0000 7f2221a10740 1 --2- 172.21.15.16:0/4076028287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 0x561c63653700 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.379+0000 7f2221a10740 1 --2- 172.21.15.16:0/4076028287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c63284660 0x561c636531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.379+0000 7f2221a10740 1 --2- 172.21.15.16:0/4076028287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c632ad520 0x561c63652c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.379+0000 7f2221a10740 1 -- 172.21.15.16:0/4076028287 >> 172.21.15.16:0/4076028287 conn(0x561c632a1910 msgr2=0x561c63650b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.380+0000 7f2221a10740 1 -- 172.21.15.16:0/4076028287 shutdown_connections 2024-08-28T19:42:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.380+0000 7f2221a10740 1 -- 172.21.15.16:0/4076028287 wait complete. 2024-08-28T19:42:01.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.380+0000 7f2221a10740 1 -- start start 2024-08-28T19:42:01.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.380+0000 7f2221a10740 1 Processor -- start 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.380+0000 7f2221a10740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c63662bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.380+0000 7f2221a10740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c63663110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2221a10740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c632ad520 0x561c6366ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2221a10740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561c63665790 con 0x561c6328c840 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2221a10740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561c6357a520 con 0x561c632ad520 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2221a10740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561c63665910 con 0x561c63284660 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2221a10740 10 client.0 _ll_register_callbacks cb 0x561c635b2c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2218a36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c63663110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2218a36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c63663110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50912/0 (socket says 172.21.15.16:50912) 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2218a36640 1 -- 172.21.15.16:0/3901351844 learned_addr learned my addr 172.21.15.16:0/3901351844 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2219237640 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c63662bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2219a38640 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c632ad520 0x561c6366ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2218a36640 1 -- 172.21.15.16:0/3901351844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 msgr2=0x561c63662bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2218a36640 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c63662bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2218a36640 1 -- 172.21.15.16:0/3901351844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c632ad520 msgr2=0x561c6366ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2218a36640 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c632ad520 0x561c6366ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2218a36640 1 -- 172.21.15.16:0/3901351844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c6338f960 con 0x561c6328c840 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2219a38640 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c632ad520 0x561c6366ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2218a36640 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c63663110 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f22040e27a0 tx=0x7f2204010b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88247dab06dd2b9f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f21fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f21fcff9640 1 -- 172.21.15.16:0/3901351844 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204021070 con 0x561c6328c840 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f2221a10740 1 -- 172.21.15.16:0/3901351844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561c635589a0 con 0x561c6328c840 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.381+0000 7f21fcff9640 1 -- 172.21.15.16:0/3901351844 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f22040a7c80 con 0x561c6328c840 2024-08-28T19:42:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f21fcff9640 1 -- 172.21.15.16:0/3901351844 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f22040aae00 con 0x561c6328c840 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f2221a10740 10 client.20277 fetch_fsmap learned FSMap version 103 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f2221a10740 1 -- 172.21.15.16:0/3901351844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x561c63665790 con 0x561c6328c840 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f21fcff9640 1 -- 172.21.15.16:0/3901351844 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561c635589a0 con 0x561c6328c840 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f21fcff9640 1 -- 172.21.15.16:0/3901351844 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 103) ==== 8394+0+0 (secure 0 0 0) 0x7f22040e2970 con 0x561c6328c840 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f2221a10740 10 client.20277 fetch_fsmap finished waiting for FSMap version 103 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f2221a10740 10 client.20277 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f2221a10740 20 client.20277 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f2221a10740 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561c6326bea0 0x561c63674a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f2221a10740 4 client.20277 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f2221a10740 1 -- 172.21.15.16:0/3901351844 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c635cb9c0 con 0x561c6326bea0 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.382+0000 7f2219237640 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561c6326bea0 0x561c63674a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.383+0000 7f21fcff9640 10 client.20277 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.383+0000 7f2219237640 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561c6326bea0 0x561c63674a70 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9a5a5320d6db5c8 server_cookie=3a8d9522603e82d5 in_seq=0 out_seq=0 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f21fcff9640 1 -- 172.21.15.16:0/3901351844 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561c635cb9c0 con 0x561c6326bea0 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f21fcff9640 10 client.20277 handle_command_reply: tid=0 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 1 client.20277 shutdown 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 1 -- 172.21.15.16:0/3901351844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 msgr2=0x561c63663110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c63663110 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f22040e27a0 tx=0x7f2204010b70 comp rx=0 tx=0).stop 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 1 -- 172.21.15.16:0/3901351844 shutdown_connections 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c632ad520 0x561c6366ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c63663110 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561c6326bea0 0x561c63674a70 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 1 --2- 172.21.15.16:0/3901351844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c63662bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 1 -- 172.21.15.16:0/3901351844 >> 172.21.15.16:0/3901351844 conn(0x561c632a1910 msgr2=0x561c63655dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f21fcff9640 0 client.20277 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 1 -- 172.21.15.16:0/3901351844 shutdown_connections 2024-08-28T19:42:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 1 -- 172.21.15.16:0/3901351844 wait complete. 2024-08-28T19:42:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.384+0000 7f2221a10740 20 client.20277 trim_cache size 0 max 16384 2024-08-28T19:42:01.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.401+0000 7f2221a10740 1 Processor -- start 2024-08-28T19:42:01.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.401+0000 7f2221a10740 1 -- start start 2024-08-28T19:42:01.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2221a10740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c63650490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2221a10740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c636509d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2221a10740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c632ad520 0x561c6365fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2221a10740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22141aeb40 con 0x561c6328c840 2024-08-28T19:42:01.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2221a10740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22141b07a0 con 0x561c632ad520 2024-08-28T19:42:01.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2221a10740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f22141af240 con 0x561c63284660 2024-08-28T19:42:01.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2218a36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c636509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2218a36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c636509d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50926/0 (socket says 172.21.15.16:50926) 2024-08-28T19:42:01.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2218a36640 1 -- 172.21.15.16:0/1225346367 learned_addr learned my addr 172.21.15.16:0/1225346367 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:01.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2219a38640 1 --2- 172.21.15.16:0/1225346367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c632ad520 0x561c6365fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2219237640 1 --2- 172.21.15.16:0/1225346367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c63650490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2218a36640 1 -- 172.21.15.16:0/1225346367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 msgr2=0x561c63650490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:01.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2218a36640 1 --2- 172.21.15.16:0/1225346367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c63650490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2218a36640 1 -- 172.21.15.16:0/1225346367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c632ad520 msgr2=0x561c6365fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:01.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2218a36640 1 --2- 172.21.15.16:0/1225346367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c632ad520 0x561c6365fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2218a36640 1 -- 172.21.15.16:0/1225346367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c63655930 con 0x561c6328c840 2024-08-28T19:42:01.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f2218a36640 1 --2- 172.21.15.16:0/1225346367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c636509d0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f22040ab620 tx=0x7f22040ab110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52475f824eb5c65 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:01.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.402+0000 7f21feffd640 1 -- 172.21.15.16:0/1225346367 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204021070 con 0x561c6328c840 2024-08-28T19:42:01.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.403+0000 7f21feffd640 1 -- 172.21.15.16:0/1225346367 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22040ab1a0 con 0x561c6328c840 2024-08-28T19:42:01.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.403+0000 7f21feffd640 1 -- 172.21.15.16:0/1225346367 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22040ab460 con 0x561c6328c840 2024-08-28T19:42:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.403+0000 7f2221a10740 1 -- 172.21.15.16:0/1225346367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 msgr2=0x561c636509d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.403+0000 7f2221a10740 1 --2- 172.21.15.16:0/1225346367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c636509d0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f22040ab620 tx=0x7f22040ab110 comp rx=0 tx=0).stop 2024-08-28T19:42:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.403+0000 7f2221a10740 1 -- 172.21.15.16:0/1225346367 shutdown_connections 2024-08-28T19:42:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.403+0000 7f2221a10740 1 --2- 172.21.15.16:0/1225346367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c632ad520 0x561c6365fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.403+0000 7f2221a10740 1 --2- 172.21.15.16:0/1225346367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c6328c840 0x561c636509d0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.403+0000 7f2221a10740 1 --2- 172.21.15.16:0/1225346367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c63650490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.403+0000 7f2221a10740 1 -- 172.21.15.16:0/1225346367 >> 172.21.15.16:0/1225346367 conn(0x561c632a1910 msgr2=0x561c63662bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.403+0000 7f2221a10740 1 -- 172.21.15.16:0/1225346367 shutdown_connections 2024-08-28T19:42:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.403+0000 7f2221a10740 1 -- 172.21.15.16:0/1225346367 wait complete. 2024-08-28T19:42:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2221a10740 1 -- start start 2024-08-28T19:42:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2221a10740 1 Processor -- start 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2221a10740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c6366adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2221a10740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 0x561c6366b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2221a10740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c632ad520 0x561c63665b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2221a10740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22141b0010 con 0x561c632ad520 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2221a10740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22141af910 con 0x561c6328c840 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2221a10740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f22141acb40 con 0x561c63284660 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2221a10740 10 client.0 _ll_register_callbacks cb 0x7f22040f7070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2218a36640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 0x561c6366b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2219a38640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c632ad520 0x561c63665b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2218a36640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 0x561c6366b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47648/0 (socket says 172.21.15.16:47648) 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.404+0000 7f2218a36640 1 -- 172.21.15.16:0/1941145906 learned_addr learned my addr 172.21.15.16:0/1941145906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f2219237640 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c6366adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f2218a36640 1 -- 172.21.15.16:0/1941145906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 msgr2=0x561c6366adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f2218a36640 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c6366adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f2218a36640 1 -- 172.21.15.16:0/1941145906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c632ad520 msgr2=0x561c63665b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f2218a36640 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c632ad520 0x561c63665b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f2218a36640 1 -- 172.21.15.16:0/1941145906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c63666120 con 0x561c6328c840 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f2218a36640 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 0x561c6366b2f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f22040ab390 tx=0x7f22040aafe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac0e281dfaa1b059 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f21fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f21fcff9640 1 -- 172.21.15.16:0/1941145906 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204021050 con 0x561c6328c840 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f2221a10740 1 -- 172.21.15.16:0/1941145906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22040ab1a0 con 0x561c6328c840 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f21fcff9640 1 -- 172.21.15.16:0/1941145906 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9490+0+0 (secure 0 0 0) 0x7f22040a7c80 con 0x561c6328c840 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f21fcff9640 1 -- 172.21.15.16:0/1941145906 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f22040a7550 con 0x561c6328c840 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f2221a10740 10 client.29741 fetch_fsmap learned FSMap version 103 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.405+0000 7f2221a10740 1 -- 172.21.15.16:0/1941145906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x561c63666120 con 0x561c6328c840 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.406+0000 7f21fcff9640 1 -- 172.21.15.16:0/1941145906 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22040ab1a0 con 0x561c6328c840 2024-08-28T19:42:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.406+0000 7f21fcff9640 1 -- 172.21.15.16:0/1941145906 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 103) ==== 8394+0+0 (secure 0 0 0) 0x7f2204059ce0 con 0x561c6328c840 2024-08-28T19:42:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.406+0000 7f2221a10740 10 client.29741 fetch_fsmap finished waiting for FSMap version 103 2024-08-28T19:42:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.406+0000 7f2221a10740 10 client.29741 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:42:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.406+0000 7f2221a10740 20 client.29741 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.406+0000 7f2221a10740 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561c63241250 0x561c636e1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.406+0000 7f2221a10740 4 client.29741 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5b45ee2f", "release": true, "await": true}] 2024-08-28T19:42:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.406+0000 7f2221a10740 1 -- 172.21.15.16:0/1941145906 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5b45ee2f", "release": true, "await": true}) -- 0x561c635c7460 con 0x561c63241250 2024-08-28T19:42:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.406+0000 7f2219237640 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561c63241250 0x561c636e1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.407+0000 7f21fcff9640 10 client.29741 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.407+0000 7f2219237640 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561c63241250 0x561c636e1e40 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=723ff8a7c2bd9600 server_cookie=9ad5fa6a41014ca7 in_seq=0 out_seq=0 2024-08-28T19:42:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.445+0000 7f21fcff9640 1 -- 172.21.15.16:0/1941145906 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+592 (crc 0 0 0) 0x561c635c7460 con 0x561c63241250 2024-08-28T19:42:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.445+0000 7f21fcff9640 10 client.29741 handle_command_reply: tid=0 2024-08-28T19:42:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.445+0000 7f2221a10740 1 client.29741 shutdown 2024-08-28T19:42:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f2221a10740 1 -- 172.21.15.16:0/1941145906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 msgr2=0x561c6366b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f2221a10740 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 0x561c6366b2f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f22040ab390 tx=0x7f22040aafe0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f2221a10740 1 -- 172.21.15.16:0/1941145906 shutdown_connections 2024-08-28T19:42:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f2221a10740 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c632ad520 0x561c63665b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f2221a10740 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c6328c840 0x561c6366b2f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f2221a10740 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561c63241250 0x561c636e1e40 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f2221a10740 1 --2- 172.21.15.16:0/1941145906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561c63284660 0x561c6366adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f21fcff9640 0 client.29741 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f2221a10740 1 -- 172.21.15.16:0/1941145906 >> 172.21.15.16:0/1941145906 conn(0x561c632a1910 msgr2=0x561c63650410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f2221a10740 1 -- 172.21.15.16:0/1941145906 shutdown_connections 2024-08-28T19:42:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f2221a10740 1 -- 172.21.15.16:0/1941145906 wait complete. 2024-08-28T19:42:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f2221a10740 20 client.29741 trim_cache size 0 max 16384 2024-08-28T19:42:01.448 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 103, 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 68, 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "5b45ee2f": { 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 68, 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:42:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 130.0, 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:01.450 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:42:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f221b49b640 1 -- 172.21.15.16:0/3514757434 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22000b4ca0 msgr2=0x7f22000b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f221b49b640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22000b4ca0 0x7f22000b71b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f220c005fd0 tx=0x7f220c009590 comp rx=0 tx=0).stop 2024-08-28T19:42:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f221b49b640 1 -- 172.21.15.16:0/3514757434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2214109240 msgr2=0x7f221419a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.446+0000 7f221b49b640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2214109240 0x7f221419a7a0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f220400d740 tx=0x7f220400dc00 comp rx=0 tx=0).stop 2024-08-28T19:42:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.447+0000 7f221b49b640 1 -- 172.21.15.16:0/3514757434 shutdown_connections 2024-08-28T19:42:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.447+0000 7f221b49b640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2214109b90 0x7f221419fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.447+0000 7f221b49b640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2214109240 0x7f221419a7a0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.447+0000 7f221b49b640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f22000b4ca0 0x7f22000b71b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.447+0000 7f221b49b640 1 --2- 172.21.15.16:0/3514757434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2214103900 0x7f221419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.447+0000 7f221b49b640 1 -- 172.21.15.16:0/3514757434 >> 172.21.15.16:0/3514757434 conn(0x7f221407b5f0 msgr2=0x7f22141072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.447+0000 7f221b49b640 1 -- 172.21.15.16:0/3514757434 shutdown_connections 2024-08-28T19:42:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:01.447+0000 7f221b49b640 1 -- 172.21.15.16:0/3514757434 wait complete. 2024-08-28T19:42:01.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177389 22.25 MB/sec execute 337 sec latency 9292.078 ms 2024-08-28T19:42:02.173 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5b45ee2f', total seconds elapsed: 13.7 2024-08-28T19:42:02.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177935 22.28 MB/sec execute 338 sec latency 125.468 ms 2024-08-28T19:42:03.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:03 smithi028 ceph-mon[21769]: pgmap v997: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 638 op/s 2024-08-28T19:42:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:03 smithi115 ceph-mon[22184]: pgmap v997: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 638 op/s 2024-08-28T19:42:03.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 178481 22.28 MB/sec execute 339 sec latency 115.750 ms 2024-08-28T19:42:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:03 smithi016 ceph-mon[18556]: pgmap v997: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 638 op/s 2024-08-28T19:42:04.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 178887 22.28 MB/sec execute 340 sec latency 120.454 ms 2024-08-28T19:42:04.870 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:42:04 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:42:04.608+0000 7f0923892640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 18 after trimming was started 2024-08-28T19:42:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:05 smithi115 ceph-mon[22184]: pgmap v998: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 13 MiB/s wr, 499 op/s 2024-08-28T19:42:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:05 smithi115 ceph-mon[22184]: osdmap e113: 12 total, 12 up, 12 in 2024-08-28T19:42:05.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 179517 22.33 MB/sec execute 341 sec latency 107.348 ms 2024-08-28T19:42:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:05 smithi016 ceph-mon[18556]: pgmap v998: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 13 MiB/s wr, 499 op/s 2024-08-28T19:42:05.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:05 smithi016 ceph-mon[18556]: osdmap e113: 12 total, 12 up, 12 in 2024-08-28T19:42:05.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:05 smithi028 ceph-mon[21769]: pgmap v998: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 13 MiB/s wr, 499 op/s 2024-08-28T19:42:05.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:05 smithi028 ceph-mon[21769]: osdmap e113: 12 total, 12 up, 12 in 2024-08-28T19:42:06.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:06 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:06.491 INFO:tasks.workunit.client.0.smithi016.stdout: 10 180241 22.37 MB/sec execute 342 sec latency 76.972 ms 2024-08-28T19:42:06.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:06 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:06.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:06 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:07.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:42:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:07 smithi115 ceph-mon[22184]: pgmap v1000: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 12 MiB/s wr, 537 op/s 2024-08-28T19:42:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:42:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:07 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.492 INFO:tasks.workunit.client.0.smithi016.stdout: 10 180775 22.41 MB/sec execute 343 sec latency 83.792 ms 2024-08-28T19:42:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:07 smithi016 ceph-mon[18556]: pgmap v1000: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 12 MiB/s wr, 537 op/s 2024-08-28T19:42:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:42:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:07 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:07 smithi028 ceph-mon[21769]: pgmap v1000: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 12 MiB/s wr, 537 op/s 2024-08-28T19:42:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:42:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:07 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:07.684 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.231+0000 7fcc7758d640 1 -- 172.21.15.16:0/3185244952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc701047d0 msgr2=0x7fcc70104be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.231+0000 7fcc7758d640 1 --2- 172.21.15.16:0/3185244952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc701047d0 0x7fcc70104be0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009be0 tx=0x7fcc6c02a6f0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.233+0000 7fcc7758d640 1 -- 172.21.15.16:0/3185244952 shutdown_connections 2024-08-28T19:42:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.233+0000 7fcc7758d640 1 --2- 172.21.15.16:0/3185244952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7010dc00 0x7fcc701100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.234+0000 7fcc7758d640 1 --2- 172.21.15.16:0/3185244952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc70105120 0x7fcc7010d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.234+0000 7fcc7758d640 1 --2- 172.21.15.16:0/3185244952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc701047d0 0x7fcc70104be0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.234+0000 7fcc7758d640 1 -- 172.21.15.16:0/3185244952 >> 172.21.15.16:0/3185244952 conn(0x7fcc70079550 msgr2=0x7fcc70079960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.234+0000 7fcc7758d640 1 -- 172.21.15.16:0/3185244952 shutdown_connections 2024-08-28T19:42:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.234+0000 7fcc7758d640 1 -- 172.21.15.16:0/3185244952 wait complete. 2024-08-28T19:42:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.235+0000 7fcc7758d640 1 Processor -- start 2024-08-28T19:42:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.235+0000 7fcc7758d640 1 -- start start 2024-08-28T19:42:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.235+0000 7fcc7758d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc701047d0 0x7fcc7019c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.235+0000 7fcc7758d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc70105120 0x7fcc7019ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.235+0000 7fcc7758d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7010dc00 0x7fcc701a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.235+0000 7fcc7758d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcc70115a90 con 0x7fcc701047d0 2024-08-28T19:42:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc7758d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc70115910 con 0x7fcc7010dc00 2024-08-28T19:42:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc7758d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcc70115c10 con 0x7fcc70105120 2024-08-28T19:42:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc75329640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc701047d0 0x7fcc7019c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc75329640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc701047d0 0x7fcc7019c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38798/0 (socket says 172.21.15.16:38798) 2024-08-28T19:42:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc75329640 1 -- 172.21.15.16:0/2536651869 learned_addr learned my addr 172.21.15.16:0/2536651869 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc74b28640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc70105120 0x7fcc7019ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7010dc00 0x7fcc701a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc75329640 1 -- 172.21.15.16:0/2536651869 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc70105120 msgr2=0x7fcc7019ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc75329640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc70105120 0x7fcc7019ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc75329640 1 -- 172.21.15.16:0/2536651869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7010dc00 msgr2=0x7fcc701a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc75329640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7010dc00 0x7fcc701a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc75329640 1 -- 172.21.15.16:0/2536651869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc701a2330 con 0x7fcc701047d0 2024-08-28T19:42:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.236+0000 7fcc75329640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc701047d0 0x7fcc7019c480 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c034040 tx=0x7fcc6c003c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.237+0000 7fcc667fc640 1 -- 172.21.15.16:0/2536651869 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c036040 con 0x7fcc701047d0 2024-08-28T19:42:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.237+0000 7fcc7758d640 1 -- 172.21.15.16:0/2536651869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc701a25c0 con 0x7fcc701047d0 2024-08-28T19:42:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.237+0000 7fcc7758d640 1 -- 172.21.15.16:0/2536651869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc701a2a30 con 0x7fcc701047d0 2024-08-28T19:42:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.238+0000 7fcc667fc640 1 -- 172.21.15.16:0/2536651869 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc6c042070 con 0x7fcc701047d0 2024-08-28T19:42:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.239+0000 7fcc667fc640 1 -- 172.21.15.16:0/2536651869 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c004040 con 0x7fcc701047d0 2024-08-28T19:42:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.242+0000 7fcc7db02740 1 Processor -- start 2024-08-28T19:42:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.242+0000 7fcc7db02740 1 -- start start 2024-08-28T19:42:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.242+0000 7fcc7db02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c4fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc7db02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688791b0 0x562768c50010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc7db02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562768881390 0x562768c50550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc7db02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562768bbe630 con 0x562768881390 2024-08-28T19:42:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc7db02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562768badbc0 con 0x5627688791b0 2024-08-28T19:42:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc7db02740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562768bb1b70 con 0x5627688a2070 2024-08-28T19:42:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc75b2a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c4fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc75b2a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c4fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55650/0 (socket says 172.21.15.16:55650) 2024-08-28T19:42:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc75b2a640 1 -- 172.21.15.16:0/2776198817 learned_addr learned my addr 172.21.15.16:0/2776198817 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc667fc640 1 -- 172.21.15.16:0/2536651869 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fcc6c0041e0 con 0x7fcc701047d0 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc74b28640 1 --2- 172.21.15.16:0/2776198817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562768881390 0x562768c50550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc75b2a640 1 -- 172.21.15.16:0/2776198817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688791b0 msgr2=0x562768c50010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/2776198817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688791b0 0x562768c50010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc75b2a640 1 -- 172.21.15.16:0/2776198817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562768881390 msgr2=0x562768c50550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/2776198817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562768881390 0x562768c50550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc75b2a640 1 -- 172.21.15.16:0/2776198817 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56276890d650 con 0x5627688a2070 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.243+0000 7fcc74b28640 1 --2- 172.21.15.16:0/2776198817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562768881390 0x562768c50550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.244+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/2776198817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c4fad0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009ca0 tx=0x7fcc68011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89acffbcd2e1bac0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.244+0000 7fcc5b7fe640 1 -- 172.21.15.16:0/2776198817 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6801e070 con 0x5627688a2070 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.244+0000 7fcc5b7fe640 1 -- 172.21.15.16:0/2776198817 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc68009e50 con 0x5627688a2070 2024-08-28T19:42:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.244+0000 7fcc5b7fe640 1 -- 172.21.15.16:0/2776198817 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6800ea50 con 0x5627688a2070 2024-08-28T19:42:08.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc667fc640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcc5c0b4e10 0x7fcc5c0b7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc75329640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcc5c0b4e10 0x7fcc5c0b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc667fc640 1 -- 172.21.15.16:0/2536651869 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9578+0+0 (secure 0 0 0) 0x7fcc6c106270 con 0x7fcc701047d0 2024-08-28T19:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc7db02740 1 -- 172.21.15.16:0/2776198817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 msgr2=0x562768c4fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc7db02740 1 --2- 172.21.15.16:0/2776198817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c4fad0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009ca0 tx=0x7fcc68011810 comp rx=0 tx=0).stop 2024-08-28T19:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc7db02740 1 -- 172.21.15.16:0/2776198817 shutdown_connections 2024-08-28T19:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc7db02740 1 --2- 172.21.15.16:0/2776198817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562768881390 0x562768c50550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc7db02740 1 --2- 172.21.15.16:0/2776198817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688791b0 0x562768c50010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc7db02740 1 --2- 172.21.15.16:0/2776198817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c4fad0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc7db02740 1 -- 172.21.15.16:0/2776198817 >> 172.21.15.16:0/2776198817 conn(0x562768896460 msgr2=0x562768c4d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc7db02740 1 -- 172.21.15.16:0/2776198817 shutdown_connections 2024-08-28T19:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.245+0000 7fcc7db02740 1 -- 172.21.15.16:0/2776198817 wait complete. 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.246+0000 7fcc7db02740 1 -- start start 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.246+0000 7fcc75329640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcc5c0b4e10 0x7fcc5c0b7320 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x562768c4e750 tx=0x7fcc6c02a6f0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.246+0000 7fcc7db02740 1 Processor -- start 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc7db02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688791b0 0x562768c5b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc7db02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 0x562768c5b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc7db02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688a2070 0x562768c67630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc7db02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562768a10700 con 0x5627688791b0 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc7db02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562768b34700 con 0x5627688a2070 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc7db02740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562768c5e2b0 con 0x562768881390 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc7db02740 10 client.0 _ll_register_callbacks cb 0x562768b63dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc74b28640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 0x562768c5b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc75b2a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688791b0 0x562768c5b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc74b28640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 0x562768c5b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55658/0 (socket says 172.21.15.16:55658) 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc74b28640 1 -- 172.21.15.16:0/3236408051 learned_addr learned my addr 172.21.15.16:0/3236408051 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.247+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688a2070 0x562768c67630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.248+0000 7fcc74b28640 1 -- 172.21.15.16:0/3236408051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688a2070 msgr2=0x562768c67630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.248+0000 7fcc74b28640 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688a2070 0x562768c67630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.248+0000 7fcc74b28640 1 -- 172.21.15.16:0/3236408051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688791b0 msgr2=0x562768c5b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.248+0000 7fcc74b28640 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688791b0 0x562768c5b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.248+0000 7fcc74b28640 1 -- 172.21.15.16:0/3236408051 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562768b6d8b0 con 0x562768881390 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.248+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688791b0 0x562768c5b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.248+0000 7fcc74b28640 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 0x562768c5b730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcc60011680 tx=0x7fcc60011b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c9bcc32de8874ec server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.248+0000 7fcc58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.249+0000 7fcc58ff9640 1 -- 172.21.15.16:0/3236408051 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6000fac0 con 0x562768881390 2024-08-28T19:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.249+0000 7fcc7db02740 1 -- 172.21.15.16:0/3236408051 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc68009e50 con 0x562768881390 2024-08-28T19:42:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.249+0000 7fcc58ff9640 1 -- 172.21.15.16:0/3236408051 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9578+0+0 (secure 0 0 0) 0x7fcc6000fc60 con 0x562768881390 2024-08-28T19:42:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.249+0000 7fcc58ff9640 1 -- 172.21.15.16:0/3236408051 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fcc60025050 con 0x562768881390 2024-08-28T19:42:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.249+0000 7fcc7db02740 10 client.29305 fetch_fsmap learned FSMap version 104 2024-08-28T19:42:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.249+0000 7fcc7db02740 1 -- 172.21.15.16:0/3236408051 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x562768c6a0a0 con 0x562768881390 2024-08-28T19:42:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.250+0000 7fcc58ff9640 1 -- 172.21.15.16:0/3236408051 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc68009e50 con 0x562768881390 2024-08-28T19:42:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.250+0000 7fcc58ff9640 1 -- 172.21.15.16:0/3236408051 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 104) ==== 8390+0+0 (secure 0 0 0) 0x7fcc6001a550 con 0x562768881390 2024-08-28T19:42:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.250+0000 7fcc7db02740 10 client.29305 fetch_fsmap finished waiting for FSMap version 104 2024-08-28T19:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.250+0000 7fcc7db02740 10 client.29305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.250+0000 7fcc7db02740 20 client.29305 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.250+0000 7fcc7db02740 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5627688609f0 0x562768c6dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.250+0000 7fcc7db02740 4 client.29305 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.250+0000 7fcc7db02740 1 -- 172.21.15.16:0/3236408051 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562768bc0580 con 0x5627688609f0 2024-08-28T19:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.250+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5627688609f0 0x562768c6dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.252+0000 7fcc58ff9640 10 client.29305 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.252+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5627688609f0 0x562768c6dad0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c00fc55b96a683e server_cookie=3d43da11ee350158 in_seq=0 out_seq=0 2024-08-28T19:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.253+0000 7fcc58ff9640 1 -- 172.21.15.16:0/3236408051 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562768bc0580 con 0x5627688609f0 2024-08-28T19:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.253+0000 7fcc58ff9640 10 client.29305 handle_command_reply: tid=0 2024-08-28T19:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.253+0000 7fcc7db02740 1 client.29305 shutdown 2024-08-28T19:42:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.254+0000 7fcc7db02740 1 -- 172.21.15.16:0/3236408051 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 msgr2=0x562768c5b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.254+0000 7fcc7db02740 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 0x562768c5b730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcc60011680 tx=0x7fcc60011b40 comp rx=0 tx=0).stop 2024-08-28T19:42:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.254+0000 7fcc7db02740 1 -- 172.21.15.16:0/3236408051 shutdown_connections 2024-08-28T19:42:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.254+0000 7fcc7db02740 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688a2070 0x562768c67630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.254+0000 7fcc7db02740 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5627688609f0 0x562768c6dad0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.254+0000 7fcc7db02740 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 0x562768c5b730 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.254+0000 7fcc7db02740 1 --2- 172.21.15.16:0/3236408051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688791b0 0x562768c5b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.254+0000 7fcc7db02740 1 -- 172.21.15.16:0/3236408051 >> 172.21.15.16:0/3236408051 conn(0x562768896460 msgr2=0x562768c52c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.254+0000 7fcc58ff9640 0 client.29305 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.254+0000 7fcc7db02740 1 -- 172.21.15.16:0/3236408051 shutdown_connections 2024-08-28T19:42:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.254+0000 7fcc7db02740 1 -- 172.21.15.16:0/3236408051 wait complete. 2024-08-28T19:42:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.255+0000 7fcc7db02740 20 client.29305 trim_cache size 0 max 16384 2024-08-28T19:42:08.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.289+0000 7fcc7db02740 1 Processor -- start 2024-08-28T19:42:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.293+0000 7fcc7db02740 1 -- start start 2024-08-28T19:42:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.293+0000 7fcc7db02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688791b0 0x562768c4d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.293+0000 7fcc7db02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562768881390 0x562768c67530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.293+0000 7fcc7db02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c583e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.293+0000 7fcc7db02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcc54010150 con 0x5627688791b0 2024-08-28T19:42:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.293+0000 7fcc7db02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc54011d80 con 0x562768881390 2024-08-28T19:42:08.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.293+0000 7fcc7db02740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcc54010850 con 0x5627688a2070 2024-08-28T19:42:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.293+0000 7fcc74b28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562768881390 0x562768c67530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.294+0000 7fcc75b2a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.294+0000 7fcc75b2a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c583e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55660/0 (socket says 172.21.15.16:55660) 2024-08-28T19:42:08.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.294+0000 7fcc75b2a640 1 -- 172.21.15.16:0/3332851715 learned_addr learned my addr 172.21.15.16:0/3332851715 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.294+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/3332851715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688791b0 0x562768c4d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.295+0000 7fcc75b2a640 1 -- 172.21.15.16:0/3332851715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562768881390 msgr2=0x562768c67530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.295+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/3332851715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562768881390 0x562768c67530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.295+0000 7fcc75b2a640 1 -- 172.21.15.16:0/3332851715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688791b0 msgr2=0x562768c4d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.295+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/3332851715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688791b0 0x562768c4d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.295+0000 7fcc75b2a640 1 -- 172.21.15.16:0/3332851715 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562768c596f0 con 0x5627688a2070 2024-08-28T19:42:08.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.296+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/3332851715 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c583e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcc68000c00 tx=0x7fcc68005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e363070f6a907f server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.297+0000 7fcc5affd640 1 -- 172.21.15.16:0/3332851715 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6801e040 con 0x5627688a2070 2024-08-28T19:42:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.297+0000 7fcc5affd640 1 -- 172.21.15.16:0/3332851715 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc680035f0 con 0x5627688a2070 2024-08-28T19:42:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.297+0000 7fcc5affd640 1 -- 172.21.15.16:0/3332851715 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc68019690 con 0x5627688a2070 2024-08-28T19:42:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.298+0000 7fcc7db02740 1 -- 172.21.15.16:0/3332851715 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 msgr2=0x562768c583e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.298+0000 7fcc7db02740 1 --2- 172.21.15.16:0/3332851715 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c583e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcc68000c00 tx=0x7fcc68005f90 comp rx=0 tx=0).stop 2024-08-28T19:42:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.299+0000 7fcc7db02740 1 -- 172.21.15.16:0/3332851715 shutdown_connections 2024-08-28T19:42:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.299+0000 7fcc7db02740 1 --2- 172.21.15.16:0/3332851715 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5627688a2070 0x562768c583e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.299+0000 7fcc7db02740 1 --2- 172.21.15.16:0/3332851715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562768881390 0x562768c67530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.299+0000 7fcc7db02740 1 --2- 172.21.15.16:0/3332851715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688791b0 0x562768c4d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.299+0000 7fcc7db02740 1 -- 172.21.15.16:0/3332851715 >> 172.21.15.16:0/3332851715 conn(0x562768896460 msgr2=0x562768c59a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.299+0000 7fcc7db02740 1 -- 172.21.15.16:0/3332851715 shutdown_connections 2024-08-28T19:42:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.299+0000 7fcc7db02740 1 -- 172.21.15.16:0/3332851715 wait complete. 2024-08-28T19:42:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.300+0000 7fcc7db02740 1 -- start start 2024-08-28T19:42:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.300+0000 7fcc7db02740 1 Processor -- start 2024-08-28T19:42:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.301+0000 7fcc7db02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688791b0 0x562768c5e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.301+0000 7fcc7db02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 0x562768c5eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.302+0000 7fcc75b2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688791b0 0x562768c5e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.302+0000 7fcc75b2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688791b0 0x562768c5e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:44334/0 (socket says 172.21.15.16:44334) 2024-08-28T19:42:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.302+0000 7fcc75b2a640 1 -- 172.21.15.16:0/874481388 learned_addr learned my addr 172.21.15.16:0/874481388 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.302+0000 7fcc74b28640 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 0x562768c5eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.302+0000 7fcc7db02740 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688a2070 0x562768cd6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.302+0000 7fcc7db02740 1 -- 172.21.15.16:0/874481388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcc540115f0 con 0x5627688a2070 2024-08-28T19:42:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.302+0000 7fcc7db02740 1 -- 172.21.15.16:0/874481388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc54010f20 con 0x5627688791b0 2024-08-28T19:42:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.302+0000 7fcc7db02740 1 -- 172.21.15.16:0/874481388 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5400e150 con 0x562768881390 2024-08-28T19:42:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.302+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688a2070 0x562768cd6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.303+0000 7fcc75b2a640 1 -- 172.21.15.16:0/874481388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 msgr2=0x562768c5eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.303+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 0x562768c5eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.303+0000 7fcc75b2a640 1 -- 172.21.15.16:0/874481388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688a2070 msgr2=0x562768cd6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.303+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688a2070 0x562768cd6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.303+0000 7fcc75b2a640 1 -- 172.21.15.16:0/874481388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562768cd69c0 con 0x5627688791b0 2024-08-28T19:42:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.303+0000 7fcc75b2a640 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688791b0 0x562768c5e960 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcc68005040 tx=0x7fcc680098f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9397f2aaeb1c095e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.303+0000 7fcc58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:42:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.303+0000 7fcc58ff9640 1 -- 172.21.15.16:0/874481388 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6801e030 con 0x5627688791b0 2024-08-28T19:42:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.303+0000 7fcc7db02740 10 client.0 _ll_register_callbacks cb 0x7fcc60005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.303+0000 7fcc7db02740 1 -- 172.21.15.16:0/874481388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc680035f0 con 0x5627688791b0 2024-08-28T19:42:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.304+0000 7fcc58ff9640 1 -- 172.21.15.16:0/874481388 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9578+0+0 (secure 0 0 0) 0x7fcc68005410 con 0x5627688791b0 2024-08-28T19:42:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.304+0000 7fcc58ff9640 1 -- 172.21.15.16:0/874481388 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fcc68045050 con 0x5627688791b0 2024-08-28T19:42:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.304+0000 7fcc7db02740 10 client.29753 fetch_fsmap learned FSMap version 104 2024-08-28T19:42:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.304+0000 7fcc7db02740 1 -- 172.21.15.16:0/874481388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=104}) -- 0x562768cd7910 con 0x5627688791b0 2024-08-28T19:42:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.306+0000 7fcc58ff9640 1 -- 172.21.15.16:0/874481388 <== mon.2 v2:172.21.15.28:3300/0 4 ==== fsmap(e 104) ==== 8390+0+0 (secure 0 0 0) 0x7fcc6803d020 con 0x5627688791b0 2024-08-28T19:42:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.306+0000 7fcc7db02740 10 client.29753 fetch_fsmap finished waiting for FSMap version 104 2024-08-28T19:42:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.306+0000 7fcc7db02740 10 client.29753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:42:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.306+0000 7fcc7db02740 20 client.29753 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.306+0000 7fcc7db02740 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562768835da0 0x562768cd9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.306+0000 7fcc7db02740 4 client.29753 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:42:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.306+0000 7fcc7db02740 1 -- 172.21.15.16:0/874481388 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562768bbc020 con 0x562768835da0 2024-08-28T19:42:08.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.308+0000 7fcc74b28640 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562768835da0 0x562768cd9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:08.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.309+0000 7fcc74b28640 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562768835da0 0x562768cd9d90 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a2d3b7d5685de49 server_cookie=76e86b62ce2475d5 in_seq=0 out_seq=0 2024-08-28T19:42:08.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.309+0000 7fcc58ff9640 10 client.29753 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:08.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.309+0000 7fcc58ff9640 1 -- 172.21.15.16:0/874481388 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7758 (crc 0 0 0) 0x562768bbc020 con 0x562768835da0 2024-08-28T19:42:08.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.309+0000 7fcc58ff9640 10 client.29753 handle_command_reply: tid=0 2024-08-28T19:42:08.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.310+0000 7fcc7db02740 1 client.29753 shutdown 2024-08-28T19:42:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.310+0000 7fcc7db02740 1 -- 172.21.15.16:0/874481388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688791b0 msgr2=0x562768c5e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.310+0000 7fcc7db02740 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688791b0 0x562768c5e960 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcc68005040 tx=0x7fcc680098f0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.310+0000 7fcc7db02740 1 -- 172.21.15.16:0/874481388 shutdown_connections 2024-08-28T19:42:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.310+0000 7fcc7db02740 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627688a2070 0x562768cd6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.310+0000 7fcc7db02740 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x562768835da0 0x562768cd9d90 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.310+0000 7fcc7db02740 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562768881390 0x562768c5eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.310+0000 7fcc7db02740 1 --2- 172.21.15.16:0/874481388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627688791b0 0x562768c5e960 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.310+0000 7fcc7db02740 1 -- 172.21.15.16:0/874481388 >> 172.21.15.16:0/874481388 conn(0x562768896460 msgr2=0x562768c50b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.311+0000 7fcc58ff9640 0 client.29753 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.311+0000 7fcc7db02740 1 -- 172.21.15.16:0/874481388 shutdown_connections 2024-08-28T19:42:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.311+0000 7fcc7db02740 1 -- 172.21.15.16:0/874481388 wait complete. 2024-08-28T19:42:08.313 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:42:08.313 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:42:08.313 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "39fa9343-222a-493e-bb10-c8863c663453": { 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:41:33.609756+0000", 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "a466ab70-970a-4b3e-927e-47c65e508132": { 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:40:38.307336+0000", 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.314 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:55.619275+0000", 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:42:08.315 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.316 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.319 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.320 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.321 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "8de90bf3-5743-4711-98c2-a56ca0757291": { 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:39:48.098772+0000", 2024-08-28T19:42:08.322 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:42:08.323 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.325 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.326 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:42:08.327 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:08.328 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:42:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:42:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:08.329 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:42:08.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.311+0000 7fcc7db02740 20 client.29753 trim_cache size 0 max 16384 2024-08-28T19:42:08.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.312+0000 7fcc7758d640 1 -- 172.21.15.16:0/2536651869 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcc5c0b4e10 msgr2=0x7fcc5c0b7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:08.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.312+0000 7fcc7758d640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcc5c0b4e10 0x7fcc5c0b7320 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x562768c4e750 tx=0x7fcc6c02a6f0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.312+0000 7fcc7758d640 1 -- 172.21.15.16:0/2536651869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc701047d0 msgr2=0x7fcc7019c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:08.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.312+0000 7fcc7758d640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc701047d0 0x7fcc7019c480 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c034040 tx=0x7fcc6c003c70 comp rx=0 tx=0).stop 2024-08-28T19:42:08.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.312+0000 7fcc7758d640 1 -- 172.21.15.16:0/2536651869 shutdown_connections 2024-08-28T19:42:08.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.312+0000 7fcc7758d640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcc5c0b4e10 0x7fcc5c0b7320 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.312+0000 7fcc7758d640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7010dc00 0x7fcc701a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.312+0000 7fcc7758d640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc70105120 0x7fcc7019ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.312+0000 7fcc7758d640 1 --2- 172.21.15.16:0/2536651869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc701047d0 0x7fcc7019c480 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.312+0000 7fcc7758d640 1 -- 172.21.15.16:0/2536651869 >> 172.21.15.16:0/2536651869 conn(0x7fcc70079550 msgr2=0x7fcc70107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.313+0000 7fcc7758d640 1 -- 172.21.15.16:0/2536651869 shutdown_connections 2024-08-28T19:42:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:08.313+0000 7fcc7758d640 1 -- 172.21.15.16:0/2536651869 wait complete. 2024-08-28T19:42:08.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:42:08.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:42:08.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:08.492 INFO:tasks.workunit.client.0.smithi016.stdout: 10 181224 22.42 MB/sec execute 344 sec latency 851.846 ms 2024-08-28T19:42:08.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:42:08.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:42:08.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:08.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:42:08.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:42:08.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:42:09.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:09 smithi016 ceph-mon[18556]: pgmap v1001: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 561 op/s 2024-08-28T19:42:09.157 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:42:09.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:42:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:09 smithi115 ceph-mon[22184]: pgmap v1001: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 561 op/s 2024-08-28T19:42:09.492 INFO:tasks.workunit.client.0.smithi016.stdout: 10 181664 22.40 MB/sec execute 345 sec latency 1851.949 ms 2024-08-28T19:42:09.515 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:09 smithi028 ceph-mon[21769]: pgmap v1001: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 561 op/s 2024-08-28T19:42:10.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.042+0000 7f6371a2b640 1 -- 172.21.15.16:0/2365911742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f636c109250 msgr2=0x7f636c109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:10.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.042+0000 7f6371a2b640 1 --2- 172.21.15.16:0/2365911742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f636c109250 0x7f636c109660 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f6360009900 tx=0x7f636002a410 comp rx=0 tx=0).stop 2024-08-28T19:42:10.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.042+0000 7f6371a2b640 1 -- 172.21.15.16:0/2365911742 shutdown_connections 2024-08-28T19:42:10.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.042+0000 7f6371a2b640 1 --2- 172.21.15.16:0/2365911742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c109ba0 0x7f636c1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:10.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.042+0000 7f6371a2b640 1 --2- 172.21.15.16:0/2365911742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f636c109250 0x7f636c109660 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:10.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.042+0000 7f6371a2b640 1 --2- 172.21.15.16:0/2365911742 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f636c103910 0x7f636c103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:10.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.042+0000 7f6371a2b640 1 -- 172.21.15.16:0/2365911742 >> 172.21.15.16:0/2365911742 conn(0x7f636c0ff380 msgr2=0x7f636c1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:10.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.042+0000 7f6371a2b640 1 -- 172.21.15.16:0/2365911742 shutdown_connections 2024-08-28T19:42:10.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.042+0000 7f6371a2b640 1 -- 172.21.15.16:0/2365911742 wait complete. 2024-08-28T19:42:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.043+0000 7f6371a2b640 1 Processor -- start 2024-08-28T19:42:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.043+0000 7f6371a2b640 1 -- start start 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f6371a2b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f636c103910 0x7f636c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f6371a2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f636c109250 0x7f636c19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f6371a2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c109ba0 0x7f636c19fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f6371a2b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f636c113860 con 0x7f636c109250 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f6371a2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f636c1136e0 con 0x7f636c109ba0 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f6371a2b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f636c1139e0 con 0x7f636c103910 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f636a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f636c109250 0x7f636c19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f636a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f636c109250 0x7f636c19a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38864/0 (socket says 172.21.15.16:38864) 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f636a7fc640 1 -- 172.21.15.16:0/1786934809 learned_addr learned my addr 172.21.15.16:0/1786934809 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f636affd640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f636c103910 0x7f636c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f636b7fe640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c109ba0 0x7f636c19fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f636affd640 1 -- 172.21.15.16:0/1786934809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c109ba0 msgr2=0x7f636c19fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f636affd640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c109ba0 0x7f636c19fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f636affd640 1 -- 172.21.15.16:0/1786934809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f636c109250 msgr2=0x7f636c19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f636affd640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f636c109250 0x7f636c19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f636affd640 1 -- 172.21.15.16:0/1786934809 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636c1a01c0 con 0x7f636c103910 2024-08-28T19:42:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.044+0000 7f636a7fc640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f636c109250 0x7f636c19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.045+0000 7f636affd640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f636c103910 0x7f636c19a250 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f635c00e9f0 tx=0x7f635c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.045+0000 7f6370a29640 1 -- 172.21.15.16:0/1786934809 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c00ccc0 con 0x7f636c103910 2024-08-28T19:42:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.045+0000 7f6371a2b640 1 -- 172.21.15.16:0/1786934809 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f636c1a04b0 con 0x7f636c103910 2024-08-28T19:42:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.045+0000 7f6371a2b640 1 -- 172.21.15.16:0/1786934809 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f636c1a6c10 con 0x7f636c103910 2024-08-28T19:42:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.045+0000 7f6370a29640 1 -- 172.21.15.16:0/1786934809 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f635c0108e0 con 0x7f636c103910 2024-08-28T19:42:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.045+0000 7f6370a29640 1 -- 172.21.15.16:0/1786934809 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c018880 con 0x7f636c103910 2024-08-28T19:42:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.046+0000 7f6371a2b640 1 -- 172.21.15.16:0/1786934809 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f636c103d20 con 0x7f636c103910 2024-08-28T19:42:10.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.049+0000 7f6370a29640 1 -- 172.21.15.16:0/1786934809 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f635c010430 con 0x7f636c103910 2024-08-28T19:42:10.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.049+0000 7f6370a29640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63540b4ca0 0x7f63540b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:10.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.050+0000 7f636a7fc640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63540b4ca0 0x7f63540b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:10.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.050+0000 7f6370a29640 1 -- 172.21.15.16:0/1786934809 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9578+0+0 (secure 0 0 0) 0x7f635c0e60e0 con 0x7f636c103910 2024-08-28T19:42:10.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.050+0000 7f636a7fc640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63540b4ca0 0x7f63540b71b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f636c106d40 tx=0x7f6360032ed0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.057+0000 7f6370a29640 1 -- 172.21.15.16:0/1786934809 <== mon.1 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f635c0ac460 con 0x7f636c103910 2024-08-28T19:42:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.346+0000 7f6371a2b640 1 -- 172.21.15.16:0/1786934809 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f636c10b9d0 con 0x7f636c103910 2024-08-28T19:42:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.348+0000 7f6370a29640 1 -- 172.21.15.16:0/1786934809 <== mon.1 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+11987 (secure 0 0 0) 0x7f635c0b0c20 con 0x7f636c103910 2024-08-28T19:42:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:42:10.350 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":104,"btime":"2024-08-28T19:42:05:059944+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":104,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:42:04.593729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24665,24649,24635,24625,24619]},"id":1}]} 2024-08-28T19:42:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 104 2024-08-28T19:42:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 -- 172.21.15.16:0/1786934809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63540b4ca0 msgr2=0x7f63540b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63540b4ca0 0x7f63540b71b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f636c106d40 tx=0x7f6360032ed0 comp rx=0 tx=0).stop 2024-08-28T19:42:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 -- 172.21.15.16:0/1786934809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f636c103910 msgr2=0x7f636c19a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f636c103910 0x7f636c19a250 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f635c00e9f0 tx=0x7f635c00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:42:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 -- 172.21.15.16:0/1786934809 shutdown_connections 2024-08-28T19:42:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f63540b4ca0 0x7f63540b71b0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c109ba0 0x7f636c19fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f636c109250 0x7f636c19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 --2- 172.21.15.16:0/1786934809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f636c103910 0x7f636c19a250 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 -- 172.21.15.16:0/1786934809 >> 172.21.15.16:0/1786934809 conn(0x7f636c0ff380 msgr2=0x7f636c1017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 -- 172.21.15.16:0/1786934809 shutdown_connections 2024-08-28T19:42:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:10.352+0000 7f6371a2b640 1 -- 172.21.15.16:0/1786934809 wait complete. 2024-08-28T19:42:10.492 INFO:tasks.workunit.client.0.smithi016.stdout: 10 182269 22.42 MB/sec execute 346 sec latency 2851.990 ms 2024-08-28T19:42:11.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:42:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:11 smithi115 ceph-mon[22184]: pgmap v1002: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.9 MiB/s wr, 501 op/s 2024-08-28T19:42:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:11 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1786934809' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:11 smithi115 ceph-mon[22184]: osdmap e114: 12 total, 12 up, 12 in 2024-08-28T19:42:11.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 183058 22.49 MB/sec execute 347 sec latency 3853.572 ms 2024-08-28T19:42:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:11 smithi016 ceph-mon[18556]: pgmap v1002: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.9 MiB/s wr, 501 op/s 2024-08-28T19:42:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:11 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1786934809' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:11.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:11 smithi016 ceph-mon[18556]: osdmap e114: 12 total, 12 up, 12 in 2024-08-28T19:42:11.522 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:11 smithi028 ceph-mon[21769]: pgmap v1002: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.9 MiB/s wr, 501 op/s 2024-08-28T19:42:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:11 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1786934809' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:11.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:11 smithi028 ceph-mon[21769]: osdmap e114: 12 total, 12 up, 12 in 2024-08-28T19:42:12.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.108+0000 7f1e06f25640 1 -- 172.21.15.16:0/2963496794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e00105120 msgr2=0x7f1e0010d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:12.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.108+0000 7f1e06f25640 1 --2- 172.21.15.16:0/2963496794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e00105120 0x7f1e0010d670 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009860 tx=0x7f1dec02a340 comp rx=0 tx=0).stop 2024-08-28T19:42:12.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.108+0000 7f1e06f25640 1 -- 172.21.15.16:0/2963496794 shutdown_connections 2024-08-28T19:42:12.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.108+0000 7f1e06f25640 1 --2- 172.21.15.16:0/2963496794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e0010dc00 0x7f1e001100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.108+0000 7f1e06f25640 1 --2- 172.21.15.16:0/2963496794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e00105120 0x7f1e0010d670 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.108+0000 7f1e06f25640 1 --2- 172.21.15.16:0/2963496794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e001047d0 0x7f1e00104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.108+0000 7f1e06f25640 1 -- 172.21.15.16:0/2963496794 >> 172.21.15.16:0/2963496794 conn(0x7f1e00079530 msgr2=0x7f1e00079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:12.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.109+0000 7f1e06f25640 1 -- 172.21.15.16:0/2963496794 shutdown_connections 2024-08-28T19:42:12.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.109+0000 7f1e06f25640 1 -- 172.21.15.16:0/2963496794 wait complete. 2024-08-28T19:42:12.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.110+0000 7f1e06f25640 1 Processor -- start 2024-08-28T19:42:12.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.110+0000 7f1e06f25640 1 -- start start 2024-08-28T19:42:12.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.110+0000 7f1e06f25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e001047d0 0x7f1e00102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.110+0000 7f1e06f25640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e00105120 0x7f1e00102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1e06f25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e0010dc00 0x7f1e00102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1e06f25640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e00113840 con 0x7f1e001047d0 2024-08-28T19:42:12.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1e06f25640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e001136c0 con 0x7f1e0010dc00 2024-08-28T19:42:12.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1e06f25640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1e001139c0 con 0x7f1e00105120 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1dfffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e00105120 0x7f1e00102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1dfffff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e00105120 0x7f1e00102a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55712/0 (socket says 172.21.15.16:55712) 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1dfffff640 1 -- 172.21.15.16:0/3264686643 learned_addr learned my addr 172.21.15.16:0/3264686643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1e04cc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e001047d0 0x7f1e00102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1e054c2640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e0010dc00 0x7f1e00102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1dfffff640 1 -- 172.21.15.16:0/3264686643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e0010dc00 msgr2=0x7f1e00102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1dfffff640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e0010dc00 0x7f1e00102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1dfffff640 1 -- 172.21.15.16:0/3264686643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e001047d0 msgr2=0x7f1e00102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1dfffff640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e001047d0 0x7f1e00102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1dfffff640 1 -- 172.21.15.16:0/3264686643 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e001a84d0 con 0x7f1e00105120 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1e054c2640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e0010dc00 0x7f1e00102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e001047d0 0x7f1e00102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.111+0000 7f1dfffff640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e00105120 0x7f1e00102a50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1dec035040 tx=0x7f1dec032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.112+0000 7f1dfdffb640 1 -- 172.21.15.16:0/3264686643 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec037040 con 0x7f1e00105120 2024-08-28T19:42:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.112+0000 7f1e06f25640 1 -- 172.21.15.16:0/3264686643 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e001a8760 con 0x7f1e00105120 2024-08-28T19:42:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.112+0000 7f1e06f25640 1 -- 172.21.15.16:0/3264686643 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e001a8c30 con 0x7f1e00105120 2024-08-28T19:42:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.112+0000 7f1dfdffb640 1 -- 172.21.15.16:0/3264686643 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dec0328a0 con 0x7f1e00105120 2024-08-28T19:42:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.112+0000 7f1dfdffb640 1 -- 172.21.15.16:0/3264686643 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0306b0 con 0x7f1e00105120 2024-08-28T19:42:12.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.115+0000 7f1e0d49a740 1 Processor -- start 2024-08-28T19:42:12.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.115+0000 7f1e0d49a740 1 -- start start 2024-08-28T19:42:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.115+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e25070 0x55e4551ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.115+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 0x55e4551ca850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.116+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 0x55e4551cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.116+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e455141680 con 0x55e454e25070 2024-08-28T19:42:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.116+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e455130c10 con 0x55e454e04390 2024-08-28T19:42:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.116+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e455134bc0 con 0x55e454dfc1b0 2024-08-28T19:42:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.116+0000 7f1e04cc1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 0x55e4551ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.116+0000 7f1e04cc1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 0x55e4551ca850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55720/0 (socket says 172.21.15.16:55720) 2024-08-28T19:42:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.116+0000 7f1e04cc1640 1 -- 172.21.15.16:0/3256237266 learned_addr learned my addr 172.21.15.16:0/3256237266 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.116+0000 7f1dfdffb640 1 -- 172.21.15.16:0/3264686643 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1dec043020 con 0x7f1e00105120 2024-08-28T19:42:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.117+0000 7f1dfffff640 1 --2- 172.21.15.16:0/3256237266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 0x55e4551cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.117+0000 7f1e04cc1640 1 -- 172.21.15.16:0/3256237266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 msgr2=0x55e4551cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.117+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/3256237266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 0x55e4551cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.117+0000 7f1e04cc1640 1 -- 172.21.15.16:0/3256237266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e25070 msgr2=0x55e4551ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.117+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/3256237266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e25070 0x55e4551ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.117+0000 7f1e04cc1640 1 -- 172.21.15.16:0/3256237266 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e454e90650 con 0x55e454dfc1b0 2024-08-28T19:42:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.117+0000 7f1dfffff640 1 --2- 172.21.15.16:0/3256237266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 0x55e4551cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:42:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.117+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/3256237266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 0x55e4551ca850 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1df40114f0 tx=0x7f1df40119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adb147bdecf7b647 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:12.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.118+0000 7f1deaffd640 1 -- 172.21.15.16:0/3256237266 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4005020 con 0x55e454dfc1b0 2024-08-28T19:42:12.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.118+0000 7f1deaffd640 1 -- 172.21.15.16:0/3256237266 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1df40047a0 con 0x55e454dfc1b0 2024-08-28T19:42:12.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.118+0000 7f1dfdffb640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1df00b4d20 0x7f1df00b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.119+0000 7f1dfdffb640 1 -- 172.21.15.16:0/3264686643 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f1dec104e10 con 0x7f1e00105120 2024-08-28T19:42:12.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.119+0000 7f1deaffd640 1 -- 172.21.15.16:0/3256237266 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df400f8a0 con 0x55e454dfc1b0 2024-08-28T19:42:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.119+0000 7f1e054c2640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1df00b4d20 0x7f1df00b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.120+0000 7f1e0d49a740 1 -- 172.21.15.16:0/3256237266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 msgr2=0x55e4551ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.120+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/3256237266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 0x55e4551ca850 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1df40114f0 tx=0x7f1df40119b0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.120+0000 7f1e0d49a740 1 -- 172.21.15.16:0/3256237266 shutdown_connections 2024-08-28T19:42:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.120+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/3256237266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 0x55e4551cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.120+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/3256237266 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 0x55e4551ca850 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.120+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/3256237266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e25070 0x55e4551ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.120+0000 7f1e0d49a740 1 -- 172.21.15.16:0/3256237266 >> 172.21.15.16:0/3256237266 conn(0x55e454e19460 msgr2=0x55e4551c8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.120+0000 7f1e0d49a740 1 -- 172.21.15.16:0/3256237266 shutdown_connections 2024-08-28T19:42:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.120+0000 7f1e0d49a740 1 -- 172.21.15.16:0/3256237266 wait complete. 2024-08-28T19:42:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.121+0000 7f1e054c2640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1df00b4d20 0x7f1df00b7230 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1df8005fd0 tx=0x7f1df8009f00 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.121+0000 7f1e0d49a740 1 -- start start 2024-08-28T19:42:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.121+0000 7f1e0d49a740 1 Processor -- start 2024-08-28T19:42:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.122+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 0x55e4551de170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.122+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e04390 0x55e4551de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.122+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e25070 0x55e4551ea6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.122+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e454f93700 con 0x55e454e04390 2024-08-28T19:42:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.122+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e4550b7750 con 0x55e454e25070 2024-08-28T19:42:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.122+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e4551e12c0 con 0x55e454dfc1b0 2024-08-28T19:42:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.122+0000 7f1e0d49a740 10 client.0 _ll_register_callbacks cb 0x55e4550e6e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.122+0000 7f1e04cc1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 0x55e4551de170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.122+0000 7f1e04cc1640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 0x55e4551de170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55726/0 (socket says 172.21.15.16:55726) 2024-08-28T19:42:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.122+0000 7f1e04cc1640 1 -- 172.21.15.16:0/2593613865 learned_addr learned my addr 172.21.15.16:0/2593613865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.122+0000 7f1dfffff640 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e04390 0x55e4551de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.123+0000 7f1dfffff640 1 -- 172.21.15.16:0/2593613865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 msgr2=0x55e4551de170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.123+0000 7f1dfffff640 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 0x55e4551de170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.123+0000 7f1dfffff640 1 -- 172.21.15.16:0/2593613865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e25070 msgr2=0x55e4551ea6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.123+0000 7f1dfffff640 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e25070 0x55e4551ea6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.123+0000 7f1dfffff640 1 -- 172.21.15.16:0/2593613865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4550f0900 con 0x55e454e04390 2024-08-28T19:42:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.123+0000 7f1dfffff640 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e04390 0x55e4551de6b0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f1dec102130 tx=0x7f1dec0caf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f344927de8cc9e17 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.123+0000 7f1ddffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:42:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.124+0000 7f1ddffff640 1 -- 172.21.15.16:0/2593613865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0cabe0 con 0x55e454e04390 2024-08-28T19:42:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.124+0000 7f1ddffff640 1 -- 172.21.15.16:0/2593613865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f1dec0cf590 con 0x55e454e04390 2024-08-28T19:42:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.124+0000 7f1e0d49a740 1 -- 172.21.15.16:0/2593613865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1df40047a0 con 0x55e454e04390 2024-08-28T19:42:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.124+0000 7f1ddffff640 1 -- 172.21.15.16:0/2593613865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f1dec0cb7d0 con 0x55e454e04390 2024-08-28T19:42:12.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.124+0000 7f1e0d49a740 10 client.20313 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:12.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.124+0000 7f1e0d49a740 1 -- 172.21.15.16:0/2593613865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x55e4551ed000 con 0x55e454e04390 2024-08-28T19:42:12.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.125+0000 7f1ddffff640 1 -- 172.21.15.16:0/2593613865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1df40047a0 con 0x55e454e04390 2024-08-28T19:42:12.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.125+0000 7f1ddffff640 1 -- 172.21.15.16:0/2593613865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7f1dec108030 con 0x55e454e04390 2024-08-28T19:42:12.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.125+0000 7f1e0d49a740 10 client.20313 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:12.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.125+0000 7f1e0d49a740 10 client.20313 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:42:12.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.125+0000 7f1e0d49a740 20 client.20313 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.125+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e454de39f0 0x55e4551f04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.125+0000 7f1e0d49a740 4 client.20313 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:42:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.125+0000 7f1e0d49a740 1 -- 172.21.15.16:0/2593613865 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4551435d0 con 0x55e454de39f0 2024-08-28T19:42:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.125+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e454de39f0 0x55e4551f04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.126+0000 7f1ddffff640 10 client.20313 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:42:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.126+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e454de39f0 0x55e4551f04d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a6685f4082ec4207 server_cookie=b4488be53c94fc7 in_seq=0 out_seq=0 2024-08-28T19:42:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1ddffff640 1 -- 172.21.15.16:0/2593613865 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e4551435d0 con 0x55e454de39f0 2024-08-28T19:42:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1ddffff640 10 client.20313 handle_command_reply: tid=0 2024-08-28T19:42:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1e0d49a740 1 client.20313 shutdown 2024-08-28T19:42:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1e0d49a740 1 -- 172.21.15.16:0/2593613865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e04390 msgr2=0x55e4551de6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e04390 0x55e4551de6b0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f1dec102130 tx=0x7f1dec0caf60 comp rx=0 tx=0).stop 2024-08-28T19:42:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1e0d49a740 1 -- 172.21.15.16:0/2593613865 shutdown_connections 2024-08-28T19:42:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e454de39f0 0x55e4551f04d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e25070 0x55e4551ea6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e04390 0x55e4551de6b0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/2593613865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454dfc1b0 0x55e4551de170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1e0d49a740 1 -- 172.21.15.16:0/2593613865 >> 172.21.15.16:0/2593613865 conn(0x55e454e19460 msgr2=0x55e4551c67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:12.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.128+0000 7f1ddffff640 0 client.20313 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:42:12.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.129+0000 7f1e0d49a740 1 -- 172.21.15.16:0/2593613865 shutdown_connections 2024-08-28T19:42:12.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.129+0000 7f1e0d49a740 1 -- 172.21.15.16:0/2593613865 wait complete. 2024-08-28T19:42:12.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.129+0000 7f1e0d49a740 20 client.20313 trim_cache size 0 max 16384 2024-08-28T19:42:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.146+0000 7f1e0d49a740 1 Processor -- start 2024-08-28T19:42:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.146+0000 7f1e0d49a740 1 -- start start 2024-08-28T19:42:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.146+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454dfc1b0 0x55e4551de630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.146+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 0x55e4551ea5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.146+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454e25070 0x55e4551db360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.146+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1de400ff90 con 0x55e454dfc1b0 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.146+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1de4011b90 con 0x55e454e04390 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.146+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1de4010660 con 0x55e454e25070 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1e04cc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454dfc1b0 0x55e4551de630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1dfffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 0x55e4551ea5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1e04cc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454dfc1b0 0x55e4551de630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38918/0 (socket says 172.21.15.16:38918) 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1e04cc1640 1 -- 172.21.15.16:0/1816802198 learned_addr learned my addr 172.21.15.16:0/1816802198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1e054c2640 1 --2- 172.21.15.16:0/1816802198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454e25070 0x55e4551db360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1e04cc1640 1 -- 172.21.15.16:0/1816802198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454e25070 msgr2=0x55e4551db360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/1816802198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454e25070 0x55e4551db360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1e04cc1640 1 -- 172.21.15.16:0/1816802198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 msgr2=0x55e4551ea5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/1816802198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 0x55e4551ea5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1e04cc1640 1 -- 172.21.15.16:0/1816802198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4551df0c0 con 0x55e454dfc1b0 2024-08-28T19:42:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1dfffff640 1 --2- 172.21.15.16:0/1816802198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 0x55e4551ea5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1e054c2640 1 --2- 172.21.15.16:0/1816802198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454e25070 0x55e4551db360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/1816802198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454dfc1b0 0x55e4551de630 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009690 tx=0x7f1df4005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b580a9154ffb6b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.147+0000 7f1dea7fc640 1 -- 172.21.15.16:0/1816802198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4004050 con 0x55e454dfc1b0 2024-08-28T19:42:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.148+0000 7f1dea7fc640 1 -- 172.21.15.16:0/1816802198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1df40046a0 con 0x55e454dfc1b0 2024-08-28T19:42:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.148+0000 7f1dea7fc640 1 -- 172.21.15.16:0/1816802198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df400fcf0 con 0x55e454dfc1b0 2024-08-28T19:42:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.149+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1816802198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454dfc1b0 msgr2=0x55e4551de630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.149+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/1816802198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454dfc1b0 0x55e4551de630 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009690 tx=0x7f1df4005e30 comp rx=0 tx=0).stop 2024-08-28T19:42:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.149+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1816802198 shutdown_connections 2024-08-28T19:42:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.149+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/1816802198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454e25070 0x55e4551db360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.149+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/1816802198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454e04390 0x55e4551ea5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.149+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/1816802198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454dfc1b0 0x55e4551de630 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.149+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1816802198 >> 172.21.15.16:0/1816802198 conn(0x55e454e19460 msgr2=0x55e4551ddf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:12.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.149+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1816802198 shutdown_connections 2024-08-28T19:42:12.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.151+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1816802198 wait complete. 2024-08-28T19:42:12.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.152+0000 7f1e0d49a740 1 -- start start 2024-08-28T19:42:12.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.152+0000 7f1e0d49a740 1 Processor -- start 2024-08-28T19:42:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.153+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454dfc1b0 0x55e4551e29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.153+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e04390 0x55e4551e2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.153+0000 7f1e0d49a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454e25070 0x55e455259ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.153+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1de4011400 con 0x55e454e04390 2024-08-28T19:42:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.153+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1de4010d30 con 0x55e454dfc1b0 2024-08-28T19:42:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.153+0000 7f1e0d49a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1de400df90 con 0x55e454e25070 2024-08-28T19:42:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.153+0000 7f1e0d49a740 10 client.0 _ll_register_callbacks cb 0x7f1dec0c6f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.153+0000 7f1e04cc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454dfc1b0 0x55e4551e29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.153+0000 7f1dfffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e04390 0x55e4551e2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.153+0000 7f1e04cc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454dfc1b0 0x55e4551e29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:44428/0 (socket says 172.21.15.16:44428) 2024-08-28T19:42:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.153+0000 7f1e04cc1640 1 -- 172.21.15.16:0/1874272206 learned_addr learned my addr 172.21.15.16:0/1874272206 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.154+0000 7f1e04cc1640 1 -- 172.21.15.16:0/1874272206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454e25070 msgr2=0x55e455259ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.154+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/1874272206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454e25070 0x55e455259ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.154+0000 7f1e04cc1640 1 -- 172.21.15.16:0/1874272206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e04390 msgr2=0x55e4551e2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.154+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/1874272206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e04390 0x55e4551e2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.154+0000 7f1e04cc1640 1 -- 172.21.15.16:0/1874272206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e45525a410 con 0x55e454dfc1b0 2024-08-28T19:42:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.155+0000 7f1e04cc1640 1 --2- 172.21.15.16:0/1874272206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454dfc1b0 0x55e4551e29c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009340 tx=0x7f1df4004280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b990467477da0f9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.155+0000 7f1ddffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:42:12.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.155+0000 7f1ddffff640 1 -- 172.21.15.16:0/1874272206 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4037060 con 0x55e454dfc1b0 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.155+0000 7f1ddffff640 1 -- 172.21.15.16:0/1874272206 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f1df4028040 con 0x55e454dfc1b0 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.155+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1874272206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1df40046a0 con 0x55e454dfc1b0 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.155+0000 7f1ddffff640 1 -- 172.21.15.16:0/1874272206 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f1df400f680 con 0x55e454dfc1b0 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.156+0000 7f1e0d49a740 10 client.29771 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.156+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1874272206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x55e45525b500 con 0x55e454dfc1b0 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.156+0000 7f1ddffff640 1 -- 172.21.15.16:0/1874272206 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1df400fd90 con 0x55e454dfc1b0 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.156+0000 7f1ddffff640 1 -- 172.21.15.16:0/1874272206 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7f1df4003c50 con 0x55e454dfc1b0 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.156+0000 7f1e0d49a740 10 client.29771 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.156+0000 7f1e0d49a740 10 client.29771 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.156+0000 7f1e0d49a740 20 client.29771 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.156+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/1874272206 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e454db8da0 0x55e45525d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:12.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.156+0000 7f1e0d49a740 4 client.29771 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:42:12.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.156+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1874272206 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f1df40046a0 con 0x55e454db8da0 2024-08-28T19:42:12.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.157+0000 7f1dfffff640 1 --2- 172.21.15.16:0/1874272206 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e454db8da0 0x55e45525d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:12.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.158+0000 7f1ddffff640 10 client.29771 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:42:12.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.158+0000 7f1dfffff640 1 --2- 172.21.15.16:0/1874272206 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e454db8da0 0x55e45525d9c0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e6eb3d55a0663a37 server_cookie=8204cfa48ba783f6 in_seq=0 out_seq=0 2024-08-28T19:42:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1ddffff640 1 -- 172.21.15.16:0/1874272206 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f1df40046a0 con 0x55e454db8da0 2024-08-28T19:42:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1ddffff640 10 client.29771 handle_command_reply: tid=0 2024-08-28T19:42:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1e0d49a740 1 client.29771 shutdown 2024-08-28T19:42:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1874272206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454dfc1b0 msgr2=0x55e4551e29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:12.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/1874272206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454dfc1b0 0x55e4551e29c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009340 tx=0x7f1df4004280 comp rx=0 tx=0).stop 2024-08-28T19:42:12.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1874272206 shutdown_connections 2024-08-28T19:42:12.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/1874272206 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x55e454db8da0 0x55e45525d9c0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/1874272206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e454e25070 0x55e455259ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/1874272206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e454e04390 0x55e4551e2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1e0d49a740 1 --2- 172.21.15.16:0/1874272206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e454dfc1b0 0x55e4551e29c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1874272206 >> 172.21.15.16:0/1874272206 conn(0x55e454e19460 msgr2=0x55e4551dc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:12.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.159+0000 7f1ddffff640 0 client.29771 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:42:12.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.160+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1874272206 shutdown_connections 2024-08-28T19:42:12.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.160+0000 7f1e0d49a740 1 -- 172.21.15.16:0/1874272206 wait complete. 2024-08-28T19:42:12.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.160+0000 7f1e0d49a740 20 client.29771 trim_cache size 0 max 16384 2024-08-28T19:42:12.162 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:42:12.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 -- 172.21.15.16:0/3264686643 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1df00b4d20 msgr2=0x7f1df00b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:12.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1df00b4d20 0x7f1df00b7230 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1df8005fd0 tx=0x7f1df8009f00 comp rx=0 tx=0).stop 2024-08-28T19:42:12.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 -- 172.21.15.16:0/3264686643 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e00105120 msgr2=0x7f1e00102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:12.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e00105120 0x7f1e00102a50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1dec035040 tx=0x7f1dec032d50 comp rx=0 tx=0).stop 2024-08-28T19:42:12.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 -- 172.21.15.16:0/3264686643 shutdown_connections 2024-08-28T19:42:12.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1df00b4d20 0x7f1df00b7230 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e0010dc00 0x7f1e00102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e00105120 0x7f1e00102a50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 --2- 172.21.15.16:0/3264686643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e001047d0 0x7f1e00102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:12.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 -- 172.21.15.16:0/3264686643 >> 172.21.15.16:0/3264686643 conn(0x7f1e00079530 msgr2=0x7f1e001005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:12.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 -- 172.21.15.16:0/3264686643 shutdown_connections 2024-08-28T19:42:12.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:12.162+0000 7f1e06f25640 1 -- 172.21.15.16:0/3264686643 wait complete. 2024-08-28T19:42:12.496 INFO:tasks.workunit.client.0.smithi016.stdout: 10 183591 22.54 MB/sec execute 348 sec latency 4853.612 ms 2024-08-28T19:42:13.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:42:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:13 smithi115 ceph-mon[22184]: pgmap v1004: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 9.6 MiB/s wr, 616 op/s 2024-08-28T19:42:13.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:13 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:13.449 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:13.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 183837 22.51 MB/sec execute 349 sec latency 5853.649 ms 2024-08-28T19:42:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:13 smithi016 ceph-mon[18556]: pgmap v1004: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 9.6 MiB/s wr, 616 op/s 2024-08-28T19:42:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:13 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:13 smithi028 ceph-mon[21769]: pgmap v1004: 129 pgs: 129 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 9.6 MiB/s wr, 616 op/s 2024-08-28T19:42:13.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:13 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:13.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.984+0000 7fe97a135640 1 -- 172.21.15.16:0/812235585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974108150 msgr2=0x7fe974108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:13.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.984+0000 7fe97a135640 1 --2- 172.21.15.16:0/812235585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974108150 0x7fe974108560 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fe968009900 tx=0x7fe96802a410 comp rx=0 tx=0).stop 2024-08-28T19:42:13.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.984+0000 7fe97a135640 1 -- 172.21.15.16:0/812235585 shutdown_connections 2024-08-28T19:42:13.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.984+0000 7fe97a135640 1 --2- 172.21.15.16:0/812235585 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe974108aa0 0x7fe97410f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:13.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.984+0000 7fe97a135640 1 --2- 172.21.15.16:0/812235585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974108150 0x7fe974108560 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:13.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.984+0000 7fe97a135640 1 --2- 172.21.15.16:0/812235585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974102ee0 0x7fe9741032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:13.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.984+0000 7fe97a135640 1 -- 172.21.15.16:0/812235585 >> 172.21.15.16:0/812235585 conn(0x7fe974077a00 msgr2=0x7fe974077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:13.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.984+0000 7fe97a135640 1 -- 172.21.15.16:0/812235585 shutdown_connections 2024-08-28T19:42:13.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.984+0000 7fe97a135640 1 -- 172.21.15.16:0/812235585 wait complete. 2024-08-28T19:42:13.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.985+0000 7fe97a135640 1 Processor -- start 2024-08-28T19:42:13.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.985+0000 7fe97a135640 1 -- start start 2024-08-28T19:42:13.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe97a135640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe974102ee0 0x7fe97419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:13.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe97a135640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974108150 0x7fe97419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe97a135640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974108aa0 0x7fe97419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe97a135640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe974112730 con 0x7fe974108150 2024-08-28T19:42:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe97a135640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9741125b0 con 0x7fe974108aa0 2024-08-28T19:42:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe972ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974108150 0x7fe97419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe97a135640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe9741128b0 con 0x7fe974102ee0 2024-08-28T19:42:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe972ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974108150 0x7fe97419a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38954/0 (socket says 172.21.15.16:38954) 2024-08-28T19:42:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe972ffd640 1 -- 172.21.15.16:0/314801728 learned_addr learned my addr 172.21.15.16:0/314801728 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe9737fe640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe974102ee0 0x7fe97419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe973fff640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974108aa0 0x7fe97419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe972ffd640 1 -- 172.21.15.16:0/314801728 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe974102ee0 msgr2=0x7fe97419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe972ffd640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe974102ee0 0x7fe97419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe972ffd640 1 -- 172.21.15.16:0/314801728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974108aa0 msgr2=0x7fe97419fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe972ffd640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974108aa0 0x7fe97419fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.986+0000 7fe972ffd640 1 -- 172.21.15.16:0/314801728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9741a01c0 con 0x7fe974108150 2024-08-28T19:42:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.987+0000 7fe972ffd640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974108150 0x7fe97419a7f0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fe968006010 tx=0x7fe968004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.987+0000 7fe970ff9640 1 -- 172.21.15.16:0/314801728 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96803f030 con 0x7fe974108150 2024-08-28T19:42:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.987+0000 7fe97a135640 1 -- 172.21.15.16:0/314801728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9741a0450 con 0x7fe974108150 2024-08-28T19:42:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.987+0000 7fe970ff9640 1 -- 172.21.15.16:0/314801728 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe968003aa0 con 0x7fe974108150 2024-08-28T19:42:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.987+0000 7fe97a135640 1 -- 172.21.15.16:0/314801728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9741a6c10 con 0x7fe974108150 2024-08-28T19:42:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.987+0000 7fe970ff9640 1 -- 172.21.15.16:0/314801728 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968035070 con 0x7fe974108150 2024-08-28T19:42:13.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.990+0000 7fe9806aa740 1 Processor -- start 2024-08-28T19:42:13.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe9806aa740 1 -- start start 2024-08-28T19:42:13.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdf1070 0x55ccbd196350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd196890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 0x55ccbd196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe9806aa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ccbd10d680 con 0x55ccbcdc81b0 2024-08-28T19:42:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe9806aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ccbd0fcc10 con 0x55ccbcdd0390 2024-08-28T19:42:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe9806aa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ccbd100bc0 con 0x55ccbcdf1070 2024-08-28T19:42:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe9737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe9737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38958/0 (socket says 172.21.15.16:38958) 2024-08-28T19:42:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe9737fe640 1 -- 172.21.15.16:0/3020435385 learned_addr learned my addr 172.21.15.16:0/3020435385 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe970ff9640 1 -- 172.21.15.16:0/314801728 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fe968042030 con 0x7fe974108150 2024-08-28T19:42:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.991+0000 7fe973fff640 1 --2- 172.21.15.16:0/3020435385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdf1070 0x55ccbd196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.992+0000 7fe972ffd640 1 --2- 172.21.15.16:0/3020435385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 0x55ccbd196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:13.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.992+0000 7fe9737fe640 1 -- 172.21.15.16:0/3020435385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdf1070 msgr2=0x55ccbd196350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:13.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.992+0000 7fe9737fe640 1 --2- 172.21.15.16:0/3020435385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdf1070 0x55ccbd196350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.992+0000 7fe9737fe640 1 -- 172.21.15.16:0/3020435385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 msgr2=0x55ccbd196dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.992+0000 7fe9737fe640 1 --2- 172.21.15.16:0/3020435385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 0x55ccbd196dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.992+0000 7fe9737fe640 1 -- 172.21.15.16:0/3020435385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccbce5c650 con 0x55ccbcdc81b0 2024-08-28T19:42:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.992+0000 7fe972ffd640 1 --2- 172.21.15.16:0/3020435385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 0x55ccbd196dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:42:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.992+0000 7fe9737fe640 1 --2- 172.21.15.16:0/3020435385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd196890 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fe964009cf0 tx=0x7fe964011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92249a9279b0074c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.993+0000 7fe959ffb640 1 -- 172.21.15.16:0/3020435385 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964004230 con 0x55ccbcdc81b0 2024-08-28T19:42:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.993+0000 7fe959ffb640 1 -- 172.21.15.16:0/3020435385 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9640043d0 con 0x55ccbcdc81b0 2024-08-28T19:42:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.993+0000 7fe959ffb640 1 -- 172.21.15.16:0/3020435385 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964019880 con 0x55ccbcdc81b0 2024-08-28T19:42:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.993+0000 7fe970ff9640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe95c0b4ca0 0x7fe95c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.993+0000 7fe970ff9640 1 -- 172.21.15.16:0/314801728 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7fe968105e80 con 0x7fe974108150 2024-08-28T19:42:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.993+0000 7fe973fff640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe95c0b4ca0 0x7fe95c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.994+0000 7fe9806aa740 1 -- 172.21.15.16:0/3020435385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 msgr2=0x55ccbd196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.994+0000 7fe9806aa740 1 --2- 172.21.15.16:0/3020435385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd196890 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fe964009cf0 tx=0x7fe964011810 comp rx=0 tx=0).stop 2024-08-28T19:42:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.994+0000 7fe9806aa740 1 -- 172.21.15.16:0/3020435385 shutdown_connections 2024-08-28T19:42:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.994+0000 7fe9806aa740 1 --2- 172.21.15.16:0/3020435385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 0x55ccbd196dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.994+0000 7fe9806aa740 1 --2- 172.21.15.16:0/3020435385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd196890 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.994+0000 7fe9806aa740 1 --2- 172.21.15.16:0/3020435385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdf1070 0x55ccbd196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.994+0000 7fe9806aa740 1 -- 172.21.15.16:0/3020435385 >> 172.21.15.16:0/3020435385 conn(0x55ccbcde5460 msgr2=0x55ccbd194240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.994+0000 7fe973fff640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe95c0b4ca0 0x7fe95c0b71b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe96c005fd0 tx=0x7fe96c009740 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.994+0000 7fe9806aa740 1 -- 172.21.15.16:0/3020435385 shutdown_connections 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.994+0000 7fe9806aa740 1 -- 172.21.15.16:0/3020435385 wait complete. 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.995+0000 7fe9806aa740 1 -- start start 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.995+0000 7fe9806aa740 1 Processor -- start 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.995+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdc81b0 0x55ccbd1a1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.995+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdd0390 0x55ccbd1a1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.995+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdf1070 0x55ccbd1ade80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.995+0000 7fe9806aa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ccbcf5f700 con 0x55ccbcdd0390 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.995+0000 7fe9806aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ccbd083750 con 0x55ccbcdf1070 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.995+0000 7fe9806aa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ccbd1a4b30 con 0x55ccbcdc81b0 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.995+0000 7fe9806aa740 10 client.0 _ll_register_callbacks cb 0x55ccbd0b2e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.996+0000 7fe9737fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdc81b0 0x55ccbd1a1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.996+0000 7fe973fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdf1070 0x55ccbd1ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.996+0000 7fe9737fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdc81b0 0x55ccbd1a1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55776/0 (socket says 172.21.15.16:55776) 2024-08-28T19:42:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.996+0000 7fe9737fe640 1 -- 172.21.15.16:0/3642456560 learned_addr learned my addr 172.21.15.16:0/3642456560 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.996+0000 7fe972ffd640 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdd0390 0x55ccbd1a1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.996+0000 7fe9737fe640 1 -- 172.21.15.16:0/3642456560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdf1070 msgr2=0x55ccbd1ade80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.996+0000 7fe9737fe640 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdf1070 0x55ccbd1ade80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.996+0000 7fe9737fe640 1 -- 172.21.15.16:0/3642456560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdd0390 msgr2=0x55ccbd1a1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.996+0000 7fe9737fe640 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdd0390 0x55ccbd1a1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.996+0000 7fe9737fe640 1 -- 172.21.15.16:0/3642456560 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccbd0bc900 con 0x55ccbcdc81b0 2024-08-28T19:42:14.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.997+0000 7fe9737fe640 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdc81b0 0x55ccbd1a1a30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe964009560 tx=0x7fe9640038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfc0567d59de0aa8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:14.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.997+0000 7fe952ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.997+0000 7fe952ffd640 1 -- 172.21.15.16:0/3642456560 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964027070 con 0x55ccbcdc81b0 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.997+0000 7fe9806aa740 1 -- 172.21.15.16:0/3642456560 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9640043d0 con 0x55ccbcdc81b0 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.997+0000 7fe952ffd640 1 -- 172.21.15.16:0/3642456560 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7fe964005240 con 0x55ccbcdc81b0 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.997+0000 7fe952ffd640 1 -- 172.21.15.16:0/3642456560 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7fe964033050 con 0x55ccbcdc81b0 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.997+0000 7fe9806aa740 10 client.29344 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.997+0000 7fe9806aa740 1 -- 172.21.15.16:0/3642456560 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x55ccbd1b4c70 con 0x55ccbcdc81b0 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.997+0000 7fe952ffd640 1 -- 172.21.15.16:0/3642456560 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9640043d0 con 0x55ccbcdc81b0 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.998+0000 7fe952ffd640 1 -- 172.21.15.16:0/3642456560 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7fe964004b70 con 0x55ccbcdc81b0 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.998+0000 7fe9806aa740 10 client.29344 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.998+0000 7fe9806aa740 10 client.29344 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.998+0000 7fe9806aa740 20 client.29344 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.998+0000 7fe9806aa740 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55ccbcdaf9f0 0x55ccbd1b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.998+0000 7fe9806aa740 4 client.29344 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.998+0000 7fe9806aa740 1 -- 172.21.15.16:0/3642456560 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccbd10f5d0 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.999+0000 7fe972ffd640 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55ccbcdaf9f0 0x55ccbd1b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.999+0000 7fe972ffd640 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55ccbcdaf9f0 0x55ccbd1b85b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2234101807499552 server_cookie=858052ad4168c17d in_seq=0 out_seq=0 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:13.999+0000 7fe952ffd640 10 client.29344 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.000+0000 7fe952ffd640 1 -- 172.21.15.16:0/3642456560 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ccbd10f5d0 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.000+0000 7fe952ffd640 10 client.29344 handle_command_reply: tid=0 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.000+0000 7fe9806aa740 1 client.29344 shutdown 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.001+0000 7fe9806aa740 1 -- 172.21.15.16:0/3642456560 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdc81b0 msgr2=0x55ccbd1a1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.001+0000 7fe9806aa740 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdc81b0 0x55ccbd1a1a30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe964009560 tx=0x7fe9640038a0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.001+0000 7fe9806aa740 1 -- 172.21.15.16:0/3642456560 shutdown_connections 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.001+0000 7fe9806aa740 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55ccbcdaf9f0 0x55ccbd1b85b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.001+0000 7fe9806aa740 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdf1070 0x55ccbd1ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.001+0000 7fe9806aa740 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdd0390 0x55ccbd1a1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.001+0000 7fe952ffd640 0 client.29344 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:42:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.001+0000 7fe9806aa740 1 --2- 172.21.15.16:0/3642456560 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdc81b0 0x55ccbd1a1a30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.001+0000 7fe9806aa740 1 -- 172.21.15.16:0/3642456560 >> 172.21.15.16:0/3642456560 conn(0x55ccbcde5460 msgr2=0x55ccbd1994e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.002+0000 7fe9806aa740 1 -- 172.21.15.16:0/3642456560 shutdown_connections 2024-08-28T19:42:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.002+0000 7fe9806aa740 1 -- 172.21.15.16:0/3642456560 wait complete. 2024-08-28T19:42:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.002+0000 7fe9806aa740 20 client.29344 trim_cache size 0 max 16384 2024-08-28T19:42:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.021+0000 7fe9806aa740 1 Processor -- start 2024-08-28T19:42:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.021+0000 7fe9806aa740 1 -- start start 2024-08-28T19:42:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.021+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd197a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.021+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd193810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.021+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 0x55ccbd193d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:14.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe9737fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:14.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe9806aa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9741aeaf0 con 0x55ccbcdc81b0 2024-08-28T19:42:14.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe9806aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9741b06f0 con 0x55ccbcdd0390 2024-08-28T19:42:14.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe9806aa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe9741af1c0 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe9737fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:55782/0 (socket says 172.21.15.16:55782) 2024-08-28T19:42:14.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe9737fe640 1 -- 172.21.15.16:0/2290464078 learned_addr learned my addr 172.21.15.16:0/2290464078 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:14.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe972ffd640 1 --2- 172.21.15.16:0/2290464078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd193810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:14.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe973fff640 1 --2- 172.21.15.16:0/2290464078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 0x55ccbd193d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:14.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe9737fe640 1 -- 172.21.15.16:0/2290464078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 msgr2=0x55ccbd193d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:14.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe9737fe640 1 --2- 172.21.15.16:0/2290464078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 0x55ccbd193d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe9737fe640 1 -- 172.21.15.16:0/2290464078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 msgr2=0x55ccbd193810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:14.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe9737fe640 1 --2- 172.21.15.16:0/2290464078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd193810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.022+0000 7fe9737fe640 1 -- 172.21.15.16:0/2290464078 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccbd19ff30 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.023+0000 7fe972ffd640 1 --2- 172.21.15.16:0/2290464078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd193810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:14.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.023+0000 7fe973fff640 1 --2- 172.21.15.16:0/2290464078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 0x55ccbd193d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:14.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.023+0000 7fe9737fe640 1 --2- 172.21.15.16:0/2290464078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd197a90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe964000c00 tx=0x7fe964005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bee4903ba3e33658 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:14.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.023+0000 7fe9597fa640 1 -- 172.21.15.16:0/2290464078 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96403c070 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.024+0000 7fe9597fa640 1 -- 172.21.15.16:0/2290464078 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe964030070 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.024+0000 7fe9597fa640 1 -- 172.21.15.16:0/2290464078 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964027070 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.025+0000 7fe9806aa740 1 -- 172.21.15.16:0/2290464078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdaf9f0 msgr2=0x55ccbd197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.025+0000 7fe9806aa740 1 --2- 172.21.15.16:0/2290464078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd197a90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe964000c00 tx=0x7fe964005db0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.025+0000 7fe9806aa740 1 -- 172.21.15.16:0/2290464078 shutdown_connections 2024-08-28T19:42:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.025+0000 7fe9806aa740 1 --2- 172.21.15.16:0/2290464078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdd0390 0x55ccbd193d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.025+0000 7fe9806aa740 1 --2- 172.21.15.16:0/2290464078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd193810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.025+0000 7fe9806aa740 1 --2- 172.21.15.16:0/2290464078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd197a90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.025+0000 7fe9806aa740 1 -- 172.21.15.16:0/2290464078 >> 172.21.15.16:0/2290464078 conn(0x55ccbcde5460 msgr2=0x55ccbd197380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.027+0000 7fe9806aa740 1 -- 172.21.15.16:0/2290464078 shutdown_connections 2024-08-28T19:42:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.027+0000 7fe9806aa740 1 -- 172.21.15.16:0/2290464078 wait complete. 2024-08-28T19:42:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.027+0000 7fe9806aa740 1 -- start start 2024-08-28T19:42:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.028+0000 7fe9806aa740 1 Processor -- start 2024-08-28T19:42:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.028+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd1a6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.028+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd1a6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.028+0000 7fe9806aa740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdd0390 0x55ccbd221f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.028+0000 7fe9806aa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9741aff60 con 0x55ccbcdc81b0 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.028+0000 7fe9806aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9741af890 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.028+0000 7fe9806aa740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe9741acaf0 con 0x55ccbcdd0390 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.028+0000 7fe9806aa740 10 client.0 _ll_register_callbacks cb 0x7fe96c0095e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.029+0000 7fe9737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd1a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.029+0000 7fe9737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd1a6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:44478/0 (socket says 172.21.15.16:44478) 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.029+0000 7fe9737fe640 1 -- 172.21.15.16:0/642655720 learned_addr learned my addr 172.21.15.16:0/642655720 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.029+0000 7fe972ffd640 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd1a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.029+0000 7fe9737fe640 1 -- 172.21.15.16:0/642655720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdd0390 msgr2=0x55ccbd221f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.029+0000 7fe9737fe640 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdd0390 0x55ccbd221f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.029+0000 7fe9737fe640 1 -- 172.21.15.16:0/642655720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 msgr2=0x55ccbd1a6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.029+0000 7fe9737fe640 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd1a6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.029+0000 7fe9737fe640 1 -- 172.21.15.16:0/642655720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccbd1a6f10 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.029+0000 7fe9737fe640 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd1a6100 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe96403a040 tx=0x7fe964009910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50b36a8a2362c344 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.030+0000 7fe952ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:42:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.030+0000 7fe952ffd640 1 -- 172.21.15.16:0/642655720 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96403c070 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.030+0000 7fe952ffd640 1 -- 172.21.15.16:0/642655720 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7fe964019a90 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.030+0000 7fe9806aa740 1 -- 172.21.15.16:0/642655720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe964030070 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.031+0000 7fe952ffd640 1 -- 172.21.15.16:0/642655720 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7fe964027040 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.032+0000 7fe9806aa740 10 client.29789 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.032+0000 7fe9806aa740 1 -- 172.21.15.16:0/642655720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x55ccbd223530 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.033+0000 7fe952ffd640 1 -- 172.21.15.16:0/642655720 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe964030070 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.034+0000 7fe952ffd640 1 -- 172.21.15.16:0/642655720 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7fe964004610 con 0x55ccbcdaf9f0 2024-08-28T19:42:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.035+0000 7fe9806aa740 10 client.29789 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.035+0000 7fe9806aa740 10 client.29789 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:42:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.036+0000 7fe9806aa740 20 client.29789 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.036+0000 7fe9806aa740 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55ccbcdf1070 0x55ccbd225c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.036+0000 7fe9806aa740 4 client.29789 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:42:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.036+0000 7fe9806aa740 1 -- 172.21.15.16:0/642655720 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ccbd10b070 con 0x55ccbcdf1070 2024-08-28T19:42:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.037+0000 7fe972ffd640 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55ccbcdf1070 0x55ccbd225c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.038+0000 7fe972ffd640 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55ccbcdf1070 0x55ccbd225c30 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b27515ef00146854 server_cookie=832eaf8fcf578015 in_seq=0 out_seq=0 2024-08-28T19:42:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.038+0000 7fe952ffd640 10 client.29789 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:42:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.038+0000 7fe952ffd640 1 -- 172.21.15.16:0/642655720 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ccbd10b070 con 0x55ccbcdf1070 2024-08-28T19:42:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.038+0000 7fe952ffd640 10 client.29789 handle_command_reply: tid=0 2024-08-28T19:42:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.038+0000 7fe9806aa740 1 client.29789 shutdown 2024-08-28T19:42:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe9806aa740 1 -- 172.21.15.16:0/642655720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdaf9f0 msgr2=0x55ccbd1a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe9806aa740 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd1a6100 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe96403a040 tx=0x7fe964009910 comp rx=0 tx=0).stop 2024-08-28T19:42:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe9806aa740 1 -- 172.21.15.16:0/642655720 shutdown_connections 2024-08-28T19:42:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe9806aa740 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x55ccbcdf1070 0x55ccbd225c30 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe9806aa740 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ccbcdd0390 0x55ccbd221f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe952ffd640 0 client.29789 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:42:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe9806aa740 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccbcdc81b0 0x55ccbd1a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe9806aa740 1 --2- 172.21.15.16:0/642655720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccbcdaf9f0 0x55ccbd1a6100 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe9806aa740 1 -- 172.21.15.16:0/642655720 >> 172.21.15.16:0/642655720 conn(0x55ccbcde5460 msgr2=0x55ccbd1a2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe9806aa740 1 -- 172.21.15.16:0/642655720 shutdown_connections 2024-08-28T19:42:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe9806aa740 1 -- 172.21.15.16:0/642655720 wait complete. 2024-08-28T19:42:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.039+0000 7fe9806aa740 20 client.29789 trim_cache size 0 max 16384 2024-08-28T19:42:14.043 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:42:14.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.040+0000 7fe97a135640 1 -- 172.21.15.16:0/314801728 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe95c0b4ca0 msgr2=0x7fe95c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:14.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.040+0000 7fe97a135640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe95c0b4ca0 0x7fe95c0b71b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe96c005fd0 tx=0x7fe96c009740 comp rx=0 tx=0).stop 2024-08-28T19:42:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.040+0000 7fe97a135640 1 -- 172.21.15.16:0/314801728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974108150 msgr2=0x7fe97419a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.040+0000 7fe97a135640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974108150 0x7fe97419a7f0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fe968006010 tx=0x7fe968004060 comp rx=0 tx=0).stop 2024-08-28T19:42:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.040+0000 7fe97a135640 1 -- 172.21.15.16:0/314801728 shutdown_connections 2024-08-28T19:42:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.040+0000 7fe97a135640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fe95c0b4ca0 0x7fe95c0b71b0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.040+0000 7fe97a135640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974108aa0 0x7fe97419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.040+0000 7fe97a135640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974108150 0x7fe97419a7f0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.040+0000 7fe97a135640 1 --2- 172.21.15.16:0/314801728 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe974102ee0 0x7fe97419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.040+0000 7fe97a135640 1 -- 172.21.15.16:0/314801728 >> 172.21.15.16:0/314801728 conn(0x7fe974077a00 msgr2=0x7fe974101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.041+0000 7fe97a135640 1 -- 172.21.15.16:0/314801728 shutdown_connections 2024-08-28T19:42:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:14.041+0000 7fe97a135640 1 -- 172.21.15.16:0/314801728 wait complete. 2024-08-28T19:42:14.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 184113 22.46 MB/sec execute 350 sec latency 6844.354 ms 2024-08-28T19:42:14.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:42:15.101 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:15 smithi115 ceph-mon[22184]: pgmap v1005: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 8.9 MiB/s wr, 558 op/s 2024-08-28T19:42:15.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:15 smithi016 ceph-mon[18556]: pgmap v1005: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 8.9 MiB/s wr, 558 op/s 2024-08-28T19:42:15.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 184670 22.49 MB/sec execute 351 sec latency 99.584 ms 2024-08-28T19:42:15.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:15 smithi028 ceph-mon[21769]: pgmap v1005: 129 pgs: 129 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 8.9 MiB/s wr, 558 op/s 2024-08-28T19:42:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.644+0000 7faa1aa86640 1 -- 172.21.15.16:0/2685739457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa14109f30 msgr2=0x7faa1410a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.644+0000 7faa1aa86640 1 --2- 172.21.15.16:0/2685739457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa14109f30 0x7faa1410a340 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7faa08009900 tx=0x7faa0802a410 comp rx=0 tx=0).stop 2024-08-28T19:42:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.644+0000 7faa1aa86640 1 -- 172.21.15.16:0/2685739457 shutdown_connections 2024-08-28T19:42:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.644+0000 7faa1aa86640 1 --2- 172.21.15.16:0/2685739457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa141101c0 0x7faa141169f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.644+0000 7faa1aa86640 1 --2- 172.21.15.16:0/2685739457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa1410f870 0x7faa1410fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.644+0000 7faa1aa86640 1 --2- 172.21.15.16:0/2685739457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa14109f30 0x7faa1410a340 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.644+0000 7faa1aa86640 1 -- 172.21.15.16:0/2685739457 >> 172.21.15.16:0/2685739457 conn(0x7faa14077f10 msgr2=0x7faa14107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:15.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.644+0000 7faa1aa86640 1 -- 172.21.15.16:0/2685739457 shutdown_connections 2024-08-28T19:42:15.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.644+0000 7faa1aa86640 1 -- 172.21.15.16:0/2685739457 wait complete. 2024-08-28T19:42:15.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.646+0000 7faa1aa86640 1 Processor -- start 2024-08-28T19:42:15.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.646+0000 7faa1aa86640 1 -- start start 2024-08-28T19:42:15.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.646+0000 7faa1aa86640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa14109f30 0x7faa141a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.646+0000 7faa1aa86640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1410f870 0x7faa141a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.647+0000 7faa1aa86640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa141101c0 0x7faa141a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.647+0000 7faa1aa86640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa14119e80 con 0x7faa1410f870 2024-08-28T19:42:15.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.647+0000 7faa1aa86640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa14119d00 con 0x7faa141101c0 2024-08-28T19:42:15.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.647+0000 7faa1aa86640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faa1411a000 con 0x7faa14109f30 2024-08-28T19:42:15.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.647+0000 7faa19023640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa141101c0 0x7faa141a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.647+0000 7faa18822640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa14109f30 0x7faa141a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.647+0000 7faa19023640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa141101c0 0x7faa141a6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:39864/0 (socket says 172.21.15.16:39864) 2024-08-28T19:42:15.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.647+0000 7faa18822640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa14109f30 0x7faa141a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:47160/0 (socket says 172.21.15.16:47160) 2024-08-28T19:42:15.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.647+0000 7faa19023640 1 -- 172.21.15.16:0/3063900927 learned_addr learned my addr 172.21.15.16:0/3063900927 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:15.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.647+0000 7faa13fff640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1410f870 0x7faa141a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.648+0000 7faa19023640 1 -- 172.21.15.16:0/3063900927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa14109f30 msgr2=0x7faa141a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:15.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.648+0000 7faa19023640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa14109f30 0x7faa141a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.648+0000 7faa19023640 1 -- 172.21.15.16:0/3063900927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1410f870 msgr2=0x7faa141a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:15.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.648+0000 7faa19023640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1410f870 0x7faa141a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.648+0000 7faa19023640 1 -- 172.21.15.16:0/3063900927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa141a6880 con 0x7faa141101c0 2024-08-28T19:42:15.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.648+0000 7faa13fff640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1410f870 0x7faa141a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:15.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.648+0000 7faa19023640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa141101c0 0x7faa141a6150 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00efd0 tx=0x7faa0c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:15.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.648+0000 7faa11ffb640 1 -- 172.21.15.16:0/3063900927 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c003b70 con 0x7faa141101c0 2024-08-28T19:42:15.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.649+0000 7faa11ffb640 1 -- 172.21.15.16:0/3063900927 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa0c003d10 con 0x7faa141101c0 2024-08-28T19:42:15.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.649+0000 7faa1aa86640 1 -- 172.21.15.16:0/3063900927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa141a6b70 con 0x7faa141101c0 2024-08-28T19:42:15.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.649+0000 7faa11ffb640 1 -- 172.21.15.16:0/3063900927 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0108a0 con 0x7faa141101c0 2024-08-28T19:42:15.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.649+0000 7faa1aa86640 1 -- 172.21.15.16:0/3063900927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa141ad2d0 con 0x7faa141101c0 2024-08-28T19:42:15.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.652+0000 7faa20ffb740 1 Processor -- start 2024-08-28T19:42:15.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.652+0000 7faa11ffb640 1 -- 172.21.15.16:0/3063900927 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7faa0c004020 con 0x7faa141101c0 2024-08-28T19:42:15.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa20ffb740 1 -- start start 2024-08-28T19:42:15.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b114310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad461b0 0x56408b114850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b114d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa20ffb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56408b08b680 con 0x56408ad4e390 2024-08-28T19:42:15.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa20ffb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56408b07ac10 con 0x56408ad6f070 2024-08-28T19:42:15.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa20ffb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56408b07ebc0 con 0x56408ad461b0 2024-08-28T19:42:15.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa11ffb640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa040b4da0 0x7faa040b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b114d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b114d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47752/0 (socket says 172.21.15.16:47752) 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa13fff640 1 -- 172.21.15.16:0/1398456495 learned_addr learned my addr 172.21.15.16:0/1398456495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa13fff640 1 --2- 172.21.15.16:0/1398456495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b114310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa18822640 1 --2- 172.21.15.16:0/1398456495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad461b0 0x56408b114850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.653+0000 7faa11ffb640 1 -- 172.21.15.16:0/3063900927 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7faa0c0e5c80 con 0x7faa141101c0 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.654+0000 7faa19023640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa040b4da0 0x7faa040b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.654+0000 7faa18822640 1 -- 172.21.15.16:0/1398456495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 msgr2=0x56408b114310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.654+0000 7faa18822640 1 --2- 172.21.15.16:0/1398456495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b114310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.654+0000 7faa18822640 1 -- 172.21.15.16:0/1398456495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 msgr2=0x56408b114d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.654+0000 7faa18822640 1 --2- 172.21.15.16:0/1398456495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b114d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.654+0000 7faa18822640 1 -- 172.21.15.16:0/1398456495 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56408adda650 con 0x56408ad461b0 2024-08-28T19:42:15.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.654+0000 7faa18822640 1 --2- 172.21.15.16:0/1398456495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad461b0 0x56408b114850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7faa08009230 tx=0x7faa08032b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32a6f248123d9811 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:15.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.655+0000 7faa19023640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa040b4da0 0x7faa040b72b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00cba0 tx=0x7faa0c009f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.655+0000 7fa9feffd640 1 -- 172.21.15.16:0/1398456495 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08040070 con 0x56408ad461b0 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.655+0000 7fa9feffd640 1 -- 172.21.15.16:0/1398456495 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa08004030 con 0x56408ad461b0 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.655+0000 7fa9feffd640 1 -- 172.21.15.16:0/1398456495 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08035040 con 0x56408ad461b0 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.656+0000 7faa20ffb740 1 -- 172.21.15.16:0/1398456495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad461b0 msgr2=0x56408b114850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.656+0000 7faa20ffb740 1 --2- 172.21.15.16:0/1398456495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad461b0 0x56408b114850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7faa08009230 tx=0x7faa08032b90 comp rx=0 tx=0).stop 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.656+0000 7faa20ffb740 1 -- 172.21.15.16:0/1398456495 shutdown_connections 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.656+0000 7faa20ffb740 1 --2- 172.21.15.16:0/1398456495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b114d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.656+0000 7faa20ffb740 1 --2- 172.21.15.16:0/1398456495 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad461b0 0x56408b114850 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.656+0000 7faa20ffb740 1 --2- 172.21.15.16:0/1398456495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b114310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.656+0000 7faa20ffb740 1 -- 172.21.15.16:0/1398456495 >> 172.21.15.16:0/1398456495 conn(0x56408ad63460 msgr2=0x56408b112200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.656+0000 7faa20ffb740 1 -- 172.21.15.16:0/1398456495 shutdown_connections 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.656+0000 7faa20ffb740 1 -- 172.21.15.16:0/1398456495 wait complete. 2024-08-28T19:42:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.657+0000 7faa20ffb740 1 -- start start 2024-08-28T19:42:15.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.657+0000 7faa20ffb740 1 Processor -- start 2024-08-28T19:42:15.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.658+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad461b0 0x56408b11fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.658+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b11ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.658+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b12bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.658+0000 7faa20ffb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56408aedd700 con 0x56408ad4e390 2024-08-28T19:42:15.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.658+0000 7faa20ffb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56408b001750 con 0x56408ad6f070 2024-08-28T19:42:15.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.658+0000 7faa20ffb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56408b122a90 con 0x56408ad461b0 2024-08-28T19:42:15.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.658+0000 7faa20ffb740 10 client.0 _ll_register_callbacks cb 0x56408b030e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:15.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.658+0000 7faa18822640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b11ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.658+0000 7faa18822640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b11ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47762/0 (socket says 172.21.15.16:47762) 2024-08-28T19:42:15.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.659+0000 7faa18822640 1 -- 172.21.15.16:0/1556080725 learned_addr learned my addr 172.21.15.16:0/1556080725 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:15.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.658+0000 7faa13fff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad461b0 0x56408b11fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.659+0000 7faa13fff640 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b12bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.659+0000 7faa18822640 1 -- 172.21.15.16:0/1556080725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad461b0 msgr2=0x56408b11fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:15.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.660+0000 7faa18822640 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad461b0 0x56408b11fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.660+0000 7faa18822640 1 -- 172.21.15.16:0/1556080725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 msgr2=0x56408b12bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:15.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.660+0000 7faa18822640 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b12bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.660+0000 7faa18822640 1 -- 172.21.15.16:0/1556080725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56408b03a900 con 0x56408ad4e390 2024-08-28T19:42:15.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.660+0000 7fa9f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:42:15.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.660+0000 7faa18822640 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b11ffb0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7faa08032eb0 tx=0x7faa08032ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18cae674fed2a53 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:15.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.660+0000 7fa9f3fff640 1 -- 172.21.15.16:0/1556080725 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08035070 con 0x56408ad4e390 2024-08-28T19:42:15.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.661+0000 7faa20ffb740 1 -- 172.21.15.16:0/1556080725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa08004030 con 0x56408ad4e390 2024-08-28T19:42:15.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.661+0000 7fa9f3fff640 1 -- 172.21.15.16:0/1556080725 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7faa080043f0 con 0x56408ad4e390 2024-08-28T19:42:15.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.661+0000 7fa9f3fff640 1 -- 172.21.15.16:0/1556080725 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7faa08035070 con 0x56408ad4e390 2024-08-28T19:42:15.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.661+0000 7faa20ffb740 10 client.20349 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:15.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.661+0000 7faa20ffb740 1 -- 172.21.15.16:0/1556080725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x56408b12e840 con 0x56408ad4e390 2024-08-28T19:42:15.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.662+0000 7fa9f3fff640 1 -- 172.21.15.16:0/1556080725 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa08004030 con 0x56408ad4e390 2024-08-28T19:42:15.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.662+0000 7fa9f3fff640 1 -- 172.21.15.16:0/1556080725 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7faa080047e0 con 0x56408ad4e390 2024-08-28T19:42:15.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.662+0000 7faa20ffb740 10 client.20349 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:15.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.662+0000 7faa20ffb740 10 client.20349 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:42:15.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.662+0000 7faa20ffb740 20 client.20349 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:15.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.662+0000 7faa20ffb740 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56408ad2d9f0 0x56408b131ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.662+0000 7faa20ffb740 4 client.20349 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:42:15.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.662+0000 7faa20ffb740 1 -- 172.21.15.16:0/1556080725 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56408b08d5d0 con 0x56408ad2d9f0 2024-08-28T19:42:15.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.663+0000 7faa13fff640 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56408ad2d9f0 0x56408b131ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.664+0000 7fa9f3fff640 10 client.20349 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:42:15.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.664+0000 7faa13fff640 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56408ad2d9f0 0x56408b131ba0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ce0349dd3bdd802 server_cookie=85792b921654a2a8 in_seq=0 out_seq=0 2024-08-28T19:42:15.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.664+0000 7fa9f3fff640 1 -- 172.21.15.16:0/1556080725 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56408b08d5d0 con 0x56408ad2d9f0 2024-08-28T19:42:15.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.664+0000 7fa9f3fff640 10 client.20349 handle_command_reply: tid=0 2024-08-28T19:42:15.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 1 client.20349 shutdown 2024-08-28T19:42:15.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 1 -- 172.21.15.16:0/1556080725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 msgr2=0x56408b11ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:15.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b11ffb0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7faa08032eb0 tx=0x7faa08032ee0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 1 -- 172.21.15.16:0/1556080725 shutdown_connections 2024-08-28T19:42:15.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56408ad2d9f0 0x56408b131ba0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b12bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b11ffb0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 1 --2- 172.21.15.16:0/1556080725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad461b0 0x56408b11fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 1 -- 172.21.15.16:0/1556080725 >> 172.21.15.16:0/1556080725 conn(0x56408ad63460 msgr2=0x56408b1174f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:15.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7fa9f3fff640 0 client.20349 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:42:15.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 1 -- 172.21.15.16:0/1556080725 shutdown_connections 2024-08-28T19:42:15.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 1 -- 172.21.15.16:0/1556080725 wait complete. 2024-08-28T19:42:15.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.665+0000 7faa20ffb740 20 client.20349 trim_cache size 0 max 16384 2024-08-28T19:42:15.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa20ffb740 1 Processor -- start 2024-08-28T19:42:15.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa20ffb740 1 -- start start 2024-08-28T19:42:15.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad461b0 0x56408b112230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad4e390 0x56408b12bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b1173d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa20ffb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa141b74f0 con 0x56408ad461b0 2024-08-28T19:42:15.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa20ffb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa141b90f0 con 0x56408ad6f070 2024-08-28T19:42:15.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa20ffb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faa141b7bc0 con 0x56408ad4e390 2024-08-28T19:42:15.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad461b0 0x56408b112230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad461b0 0x56408b112230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47764/0 (socket says 172.21.15.16:47764) 2024-08-28T19:42:15.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa13fff640 1 -- 172.21.15.16:0/2236548310 learned_addr learned my addr 172.21.15.16:0/2236548310 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:15.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa18822640 1 --2- 172.21.15.16:0/2236548310 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad4e390 0x56408b12bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.682+0000 7faa13fff640 1 --2- 172.21.15.16:0/2236548310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b1173d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.683+0000 7faa13fff640 1 -- 172.21.15.16:0/2236548310 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad4e390 msgr2=0x56408b12bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:15.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.683+0000 7faa13fff640 1 --2- 172.21.15.16:0/2236548310 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad4e390 0x56408b12bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.683+0000 7faa13fff640 1 -- 172.21.15.16:0/2236548310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 msgr2=0x56408b1173d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:15.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.683+0000 7faa13fff640 1 --2- 172.21.15.16:0/2236548310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b1173d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.683+0000 7faa13fff640 1 -- 172.21.15.16:0/2236548310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56408b120b10 con 0x56408ad461b0 2024-08-28T19:42:15.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.683+0000 7faa13fff640 1 --2- 172.21.15.16:0/2236548310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad461b0 0x56408b112230 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7faa00015d00 tx=0x7faa00012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c84a098fc8712e8 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:15.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.683+0000 7fa9fe7fc640 1 -- 172.21.15.16:0/2236548310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa00013a20 con 0x56408ad461b0 2024-08-28T19:42:15.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.683+0000 7fa9fe7fc640 1 -- 172.21.15.16:0/2236548310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa00013bc0 con 0x56408ad461b0 2024-08-28T19:42:15.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.683+0000 7fa9fe7fc640 1 -- 172.21.15.16:0/2236548310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa00015110 con 0x56408ad461b0 2024-08-28T19:42:15.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.684+0000 7faa20ffb740 1 -- 172.21.15.16:0/2236548310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad461b0 msgr2=0x56408b112230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:15.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.684+0000 7faa20ffb740 1 --2- 172.21.15.16:0/2236548310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad461b0 0x56408b112230 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7faa00015d00 tx=0x7faa00012040 comp rx=0 tx=0).stop 2024-08-28T19:42:15.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.684+0000 7faa20ffb740 1 -- 172.21.15.16:0/2236548310 shutdown_connections 2024-08-28T19:42:15.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.684+0000 7faa20ffb740 1 --2- 172.21.15.16:0/2236548310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad6f070 0x56408b1173d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.684+0000 7faa20ffb740 1 --2- 172.21.15.16:0/2236548310 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad4e390 0x56408b12bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.684+0000 7faa20ffb740 1 --2- 172.21.15.16:0/2236548310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad461b0 0x56408b112230 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.684+0000 7faa20ffb740 1 -- 172.21.15.16:0/2236548310 >> 172.21.15.16:0/2236548310 conn(0x56408ad63460 msgr2=0x56408b111b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:15.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.684+0000 7faa20ffb740 1 -- 172.21.15.16:0/2236548310 shutdown_connections 2024-08-28T19:42:15.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.685+0000 7faa20ffb740 1 -- 172.21.15.16:0/2236548310 wait complete. 2024-08-28T19:42:15.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.685+0000 7faa20ffb740 1 -- start start 2024-08-28T19:42:15.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.685+0000 7faa20ffb740 1 Processor -- start 2024-08-28T19:42:15.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad461b0 0x56408b132e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b133360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa20ffb740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad6f070 0x56408b19b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa20ffb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa141b8960 con 0x56408ad4e390 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa20ffb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa141b8290 con 0x56408ad461b0 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa20ffb740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faa141b5470 con 0x56408ad6f070 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa20ffb740 10 client.0 _ll_register_callbacks cb 0x7faa00013660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa18822640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa18822640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47778/0 (socket says 172.21.15.16:47778) 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa18822640 1 -- 172.21.15.16:0/199865882 learned_addr learned my addr 172.21.15.16:0/199865882 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa13fff640 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad6f070 0x56408b19b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa13fff640 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad461b0 0x56408b132e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa18822640 1 -- 172.21.15.16:0/199865882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad6f070 msgr2=0x56408b19b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa18822640 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad6f070 0x56408b19b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa18822640 1 -- 172.21.15.16:0/199865882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad461b0 msgr2=0x56408b132e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa18822640 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad461b0 0x56408b132e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa18822640 1 -- 172.21.15.16:0/199865882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56408b133b50 con 0x56408ad4e390 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.686+0000 7faa13fff640 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad461b0 0x56408b132e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.687+0000 7faa18822640 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b133360 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7faa08001eb0 tx=0x7faa08005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6418f7cbb88eba server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.687+0000 7fa9f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.687+0000 7fa9f3fff640 1 -- 172.21.15.16:0/199865882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08065060 con 0x56408ad4e390 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.687+0000 7fa9f3fff640 1 -- 172.21.15.16:0/199865882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7faa080043f0 con 0x56408ad4e390 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.687+0000 7faa20ffb740 1 -- 172.21.15.16:0/199865882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa00013bc0 con 0x56408ad4e390 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.687+0000 7fa9f3fff640 1 -- 172.21.15.16:0/199865882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7faa08035070 con 0x56408ad4e390 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.687+0000 7faa20ffb740 10 client.20355 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:15.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.687+0000 7faa20ffb740 1 -- 172.21.15.16:0/199865882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x56408b19c810 con 0x56408ad4e390 2024-08-28T19:42:15.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.688+0000 7fa9f3fff640 1 -- 172.21.15.16:0/199865882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa080054e0 con 0x56408ad4e390 2024-08-28T19:42:15.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.688+0000 7fa9f3fff640 1 -- 172.21.15.16:0/199865882 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7faa08031a70 con 0x56408ad4e390 2024-08-28T19:42:15.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.688+0000 7faa20ffb740 10 client.20355 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:15.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.688+0000 7faa20ffb740 10 client.20355 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:42:15.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.688+0000 7faa20ffb740 20 client.20355 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:15.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.688+0000 7faa20ffb740 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56408ad02da0 0x56408b19ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:15.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.688+0000 7faa20ffb740 4 client.20355 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:42:15.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.688+0000 7faa20ffb740 1 -- 172.21.15.16:0/199865882 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7faa00013bc0 con 0x56408ad02da0 2024-08-28T19:42:15.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.688+0000 7faa13fff640 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56408ad02da0 0x56408b19ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:15.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.689+0000 7faa13fff640 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56408ad02da0 0x56408b19ed70 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=38d248ca080925f9 server_cookie=3a69b74efb5d290d in_seq=0 out_seq=0 2024-08-28T19:42:15.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.689+0000 7fa9f3fff640 10 client.20355 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:42:15.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.691+0000 7fa9f3fff640 1 -- 172.21.15.16:0/199865882 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7faa00013bc0 con 0x56408ad02da0 2024-08-28T19:42:15.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.691+0000 7fa9f3fff640 10 client.20355 handle_command_reply: tid=0 2024-08-28T19:42:15.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 1 client.20355 shutdown 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 1 -- 172.21.15.16:0/199865882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 msgr2=0x56408b133360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b133360 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7faa08001eb0 tx=0x7faa08005900 comp rx=0 tx=0).stop 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 1 -- 172.21.15.16:0/199865882 shutdown_connections 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x56408ad02da0 0x56408b19ed70 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56408ad6f070 0x56408b19b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7fa9f3fff640 0 client.20355 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56408ad4e390 0x56408b133360 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 1 --2- 172.21.15.16:0/199865882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56408ad461b0 0x56408b132e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 1 -- 172.21.15.16:0/199865882 >> 172.21.15.16:0/199865882 conn(0x56408ad63460 msgr2=0x56408b115340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 1 -- 172.21.15.16:0/199865882 shutdown_connections 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 1 -- 172.21.15.16:0/199865882 wait complete. 2024-08-28T19:42:15.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.692+0000 7faa20ffb740 20 client.20355 trim_cache size 0 max 16384 2024-08-28T19:42:15.696 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:42:15.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 -- 172.21.15.16:0/3063900927 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa040b4da0 msgr2=0x7faa040b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:15.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa040b4da0 0x7faa040b72b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00cba0 tx=0x7faa0c009f90 comp rx=0 tx=0).stop 2024-08-28T19:42:15.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 -- 172.21.15.16:0/3063900927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa141101c0 msgr2=0x7faa141a6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:15.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa141101c0 0x7faa141a6150 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00efd0 tx=0x7faa0c00c6a0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 -- 172.21.15.16:0/3063900927 shutdown_connections 2024-08-28T19:42:15.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faa040b4da0 0x7faa040b72b0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa141101c0 0x7faa141a6150 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1410f870 0x7faa141a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 --2- 172.21.15.16:0/3063900927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa14109f30 0x7faa141a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:15.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 -- 172.21.15.16:0/3063900927 >> 172.21.15.16:0/3063900927 conn(0x7faa14077f10 msgr2=0x7faa14108450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:15.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 -- 172.21.15.16:0/3063900927 shutdown_connections 2024-08-28T19:42:15.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:15.693+0000 7faa1aa86640 1 -- 172.21.15.16:0/3063900927 wait complete. 2024-08-28T19:42:16.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 185264 22.49 MB/sec execute 352 sec latency 118.841 ms 2024-08-28T19:42:16.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:42:17.176 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:17.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:17 smithi016 ceph-mon[18556]: pgmap v1006: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.4 MiB/s wr, 556 op/s 2024-08-28T19:42:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:17 smithi115 ceph-mon[22184]: pgmap v1006: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.4 MiB/s wr, 556 op/s 2024-08-28T19:42:17.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 186024 22.54 MB/sec execute 353 sec latency 110.613 ms 2024-08-28T19:42:17.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:17 smithi028 ceph-mon[21769]: pgmap v1006: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.4 MiB/s wr, 556 op/s 2024-08-28T19:42:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.750+0000 7fec59eb2640 1 -- 172.21.15.16:0/3753690354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec5410aab0 msgr2=0x7fec54116d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.750+0000 7fec59eb2640 1 --2- 172.21.15.16:0/3753690354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec5410aab0 0x7fec54116d60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fec4c009be0 tx=0x7fec4c02a6c0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.750+0000 7fec59eb2640 1 -- 172.21.15.16:0/3753690354 shutdown_connections 2024-08-28T19:42:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.750+0000 7fec59eb2640 1 --2- 172.21.15.16:0/3753690354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec5410aab0 0x7fec54116d60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.750+0000 7fec59eb2640 1 --2- 172.21.15.16:0/3753690354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec5410a160 0x7fec5410a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.750+0000 7fec59eb2640 1 --2- 172.21.15.16:0/3753690354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec5410eef0 0x7fec5410f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.750+0000 7fec59eb2640 1 -- 172.21.15.16:0/3753690354 >> 172.21.15.16:0/3753690354 conn(0x7fec54071090 msgr2=0x7fec54073bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.750+0000 7fec59eb2640 1 -- 172.21.15.16:0/3753690354 shutdown_connections 2024-08-28T19:42:17.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.750+0000 7fec59eb2640 1 -- 172.21.15.16:0/3753690354 wait complete. 2024-08-28T19:42:17.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.751+0000 7fec59eb2640 1 Processor -- start 2024-08-28T19:42:17.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.751+0000 7fec59eb2640 1 -- start start 2024-08-28T19:42:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.751+0000 7fec59eb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec5410a160 0x7fec541a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.751+0000 7fec59eb2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec5410aab0 0x7fec541a2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.752+0000 7fec59eb2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec5410eef0 0x7fec541a8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.752+0000 7fec537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec5410a160 0x7fec541a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.752+0000 7fec59eb2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec54119e50 con 0x7fec5410a160 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.752+0000 7fec537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec5410a160 0x7fec541a29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47800/0 (socket says 172.21.15.16:47800) 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.752+0000 7fec537fe640 1 -- 172.21.15.16:0/2812593174 learned_addr learned my addr 172.21.15.16:0/2812593174 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.752+0000 7fec52ffd640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec5410aab0 0x7fec541a2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.752+0000 7fec53fff640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec5410eef0 0x7fec541a8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.752+0000 7fec59eb2640 1 -- 172.21.15.16:0/2812593174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec54119cd0 con 0x7fec5410aab0 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.752+0000 7fec59eb2640 1 -- 172.21.15.16:0/2812593174 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fec54119fd0 con 0x7fec5410eef0 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.753+0000 7fec52ffd640 1 -- 172.21.15.16:0/2812593174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec5410eef0 msgr2=0x7fec541a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.753+0000 7fec52ffd640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec5410eef0 0x7fec541a8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.753+0000 7fec52ffd640 1 -- 172.21.15.16:0/2812593174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec5410a160 msgr2=0x7fec541a29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.753+0000 7fec52ffd640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec5410a160 0x7fec541a29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.753+0000 7fec52ffd640 1 -- 172.21.15.16:0/2812593174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec541a8930 con 0x7fec5410aab0 2024-08-28T19:42:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.753+0000 7fec537fe640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec5410a160 0x7fec541a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.753+0000 7fec52ffd640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec5410aab0 0x7fec541a2f60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fec5406c480 tx=0x7fec48009a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.754+0000 7fec50ff9640 1 -- 172.21.15.16:0/2812593174 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec48018030 con 0x7fec5410aab0 2024-08-28T19:42:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.754+0000 7fec59eb2640 1 -- 172.21.15.16:0/2812593174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec541a8c20 con 0x7fec5410aab0 2024-08-28T19:42:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.754+0000 7fec50ff9640 1 -- 172.21.15.16:0/2812593174 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec48014620 con 0x7fec5410aab0 2024-08-28T19:42:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.754+0000 7fec50ff9640 1 -- 172.21.15.16:0/2812593174 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec48003850 con 0x7fec5410aab0 2024-08-28T19:42:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.754+0000 7fec59eb2640 1 -- 172.21.15.16:0/2812593174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec541af380 con 0x7fec5410aab0 2024-08-28T19:42:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.757+0000 7fec60427740 1 Processor -- start 2024-08-28T19:42:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.757+0000 7fec60427740 1 -- start start 2024-08-28T19:42:17.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.758+0000 7fec60427740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609affe9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.758+0000 7fec60427740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 0x5609affe9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.758+0000 7fec60427740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc23390 0x5609affe9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.758+0000 7fec60427740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5609aff60680 con 0x5609afc1b1b0 2024-08-28T19:42:17.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.758+0000 7fec60427740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5609aff4fc10 con 0x5609afc44070 2024-08-28T19:42:17.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.758+0000 7fec60427740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5609aff53bc0 con 0x5609afc23390 2024-08-28T19:42:17.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.758+0000 7fec53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609affe9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.758+0000 7fec53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609affe9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:39944/0 (socket says 172.21.15.16:39944) 2024-08-28T19:42:17.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.758+0000 7fec53fff640 1 -- 172.21.15.16:0/3180752389 learned_addr learned my addr 172.21.15.16:0/3180752389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:17.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.758+0000 7fec537fe640 1 --2- 172.21.15.16:0/3180752389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 0x5609affe9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.759+0000 7fec53fff640 1 -- 172.21.15.16:0/3180752389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc23390 msgr2=0x5609affe9dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-28T19:42:17.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.759+0000 7fec53fff640 1 --2- 172.21.15.16:0/3180752389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc23390 0x5609affe9dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.759+0000 7fec53fff640 1 -- 172.21.15.16:0/3180752389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 msgr2=0x5609affe9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:17.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.759+0000 7fec53fff640 1 --2- 172.21.15.16:0/3180752389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 0x5609affe9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.759+0000 7fec53fff640 1 -- 172.21.15.16:0/3180752389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609afcaf650 con 0x5609afc44070 2024-08-28T19:42:17.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.759+0000 7fec537fe640 1 --2- 172.21.15.16:0/3180752389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 0x5609affe9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:42:17.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.759+0000 7fec53fff640 1 --2- 172.21.15.16:0/3180752389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609affe9350 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fec4c009be0 tx=0x7fec4c030430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e9be83ac0e7c384 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.759+0000 7fec50ff9640 1 -- 172.21.15.16:0/2812593174 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fec4801b070 con 0x7fec5410aab0 2024-08-28T19:42:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.759+0000 7fec39ffb640 1 -- 172.21.15.16:0/3180752389 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c040030 con 0x5609afc44070 2024-08-28T19:42:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.760+0000 7fec39ffb640 1 -- 172.21.15.16:0/3180752389 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec4c003e20 con 0x5609afc44070 2024-08-28T19:42:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.760+0000 7fec39ffb640 1 -- 172.21.15.16:0/3180752389 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c036040 con 0x5609afc44070 2024-08-28T19:42:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.760+0000 7fec50ff9640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec3c0b4d20 0x7fec3c0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.760+0000 7fec60427740 1 -- 172.21.15.16:0/3180752389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 msgr2=0x5609affe9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.761+0000 7fec60427740 1 --2- 172.21.15.16:0/3180752389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609affe9350 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fec4c009be0 tx=0x7fec4c030430 comp rx=0 tx=0).stop 2024-08-28T19:42:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.761+0000 7fec60427740 1 -- 172.21.15.16:0/3180752389 shutdown_connections 2024-08-28T19:42:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.761+0000 7fec60427740 1 --2- 172.21.15.16:0/3180752389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc23390 0x5609affe9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.761+0000 7fec50ff9640 1 -- 172.21.15.16:0/2812593174 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7fec480e4e20 con 0x7fec5410aab0 2024-08-28T19:42:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.761+0000 7fec60427740 1 --2- 172.21.15.16:0/3180752389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 0x5609affe9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.761+0000 7fec60427740 1 --2- 172.21.15.16:0/3180752389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609affe9350 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.761+0000 7fec60427740 1 -- 172.21.15.16:0/3180752389 >> 172.21.15.16:0/3180752389 conn(0x5609afc38460 msgr2=0x5609affe7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.761+0000 7fec537fe640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec3c0b4d20 0x7fec3c0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.761+0000 7fec60427740 1 -- 172.21.15.16:0/3180752389 shutdown_connections 2024-08-28T19:42:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.761+0000 7fec60427740 1 -- 172.21.15.16:0/3180752389 wait complete. 2024-08-28T19:42:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.761+0000 7fec537fe640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec3c0b4d20 0x7fec3c0b7230 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x5609affe7fd0 tx=0x7fec44005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.762+0000 7fec60427740 1 -- start start 2024-08-28T19:42:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.762+0000 7fec60427740 1 Processor -- start 2024-08-28T19:42:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.763+0000 7fec60427740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc1b1b0 0x5609afff4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.763+0000 7fec60427740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 0x5609afff50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.763+0000 7fec60427740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609b0000dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.763+0000 7fec60427740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5609afdb2700 con 0x5609afc23390 2024-08-28T19:42:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.763+0000 7fec60427740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5609afed6750 con 0x5609afc44070 2024-08-28T19:42:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.763+0000 7fec60427740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5609afff7ad0 con 0x5609afc1b1b0 2024-08-28T19:42:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.763+0000 7fec60427740 10 client.0 _ll_register_callbacks cb 0x5609aff05e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.763+0000 7fec52ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 0x5609afff50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.763+0000 7fec52ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 0x5609afff50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47820/0 (socket says 172.21.15.16:47820) 2024-08-28T19:42:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.763+0000 7fec52ffd640 1 -- 172.21.15.16:0/1434785546 learned_addr learned my addr 172.21.15.16:0/1434785546 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.763+0000 7fec53fff640 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609b0000dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.764+0000 7fec537fe640 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc1b1b0 0x5609afff4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.764+0000 7fec52ffd640 1 -- 172.21.15.16:0/1434785546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc1b1b0 msgr2=0x5609afff4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.764+0000 7fec52ffd640 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc1b1b0 0x5609afff4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.764+0000 7fec52ffd640 1 -- 172.21.15.16:0/1434785546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 msgr2=0x5609b0000dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.764+0000 7fec52ffd640 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609b0000dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.764+0000 7fec52ffd640 1 -- 172.21.15.16:0/1434785546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609aff0f900 con 0x5609afc23390 2024-08-28T19:42:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.764+0000 7fec52ffd640 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 0x5609afff50c0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fec480aae40 tx=0x7fec48014d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51473a8af1c6552 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.764+0000 7fec32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:42:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.764+0000 7fec32ffd640 1 -- 172.21.15.16:0/1434785546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec480af9b0 con 0x5609afc23390 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.765+0000 7fec32ffd640 1 -- 172.21.15.16:0/1434785546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7fec480a73d0 con 0x5609afc23390 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.765+0000 7fec60427740 1 -- 172.21.15.16:0/1434785546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec4c003e20 con 0x5609afc23390 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.765+0000 7fec32ffd640 1 -- 172.21.15.16:0/1434785546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7fec480b4460 con 0x5609afc23390 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.765+0000 7fec60427740 10 client.20367 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.765+0000 7fec60427740 1 -- 172.21.15.16:0/1434785546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x5609b0007c90 con 0x5609afc23390 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.765+0000 7fec32ffd640 1 -- 172.21.15.16:0/1434785546 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec4c003e20 con 0x5609afc23390 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.765+0000 7fec32ffd640 1 -- 172.21.15.16:0/1434785546 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7fec480041a0 con 0x5609afc23390 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.765+0000 7fec60427740 10 client.20367 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.765+0000 7fec60427740 10 client.20367 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.766+0000 7fec60427740 20 client.20367 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.766+0000 7fec60427740 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5609afc029f0 0x5609b000b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.766+0000 7fec60427740 4 client.20367 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:42:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.766+0000 7fec60427740 1 -- 172.21.15.16:0/1434785546 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609aff625d0 con 0x5609afc029f0 2024-08-28T19:42:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.766+0000 7fec537fe640 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5609afc029f0 0x5609b000b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.767+0000 7fec32ffd640 10 client.20367 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:42:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.767+0000 7fec537fe640 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5609afc029f0 0x5609b000b1b0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c0c62abd212a67dc server_cookie=a1452abd82e005e4 in_seq=0 out_seq=0 2024-08-28T19:42:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.768+0000 7fec32ffd640 1 -- 172.21.15.16:0/1434785546 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5609aff625d0 con 0x5609afc029f0 2024-08-28T19:42:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.768+0000 7fec32ffd640 10 client.20367 handle_command_reply: tid=0 2024-08-28T19:42:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.768+0000 7fec60427740 1 client.20367 shutdown 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec60427740 1 -- 172.21.15.16:0/1434785546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 msgr2=0x5609afff50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec60427740 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 0x5609afff50c0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fec480aae40 tx=0x7fec48014d00 comp rx=0 tx=0).stop 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec60427740 1 -- 172.21.15.16:0/1434785546 shutdown_connections 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec60427740 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5609afc029f0 0x5609b000b1b0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec60427740 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609b0000dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec60427740 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 0x5609afff50c0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec32ffd640 0 client.20367 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec60427740 1 --2- 172.21.15.16:0/1434785546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc1b1b0 0x5609afff4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec60427740 1 -- 172.21.15.16:0/1434785546 >> 172.21.15.16:0/1434785546 conn(0x5609afc38460 msgr2=0x5609affec530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec60427740 1 -- 172.21.15.16:0/1434785546 shutdown_connections 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec60427740 1 -- 172.21.15.16:0/1434785546 wait complete. 2024-08-28T19:42:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.769+0000 7fec60427740 20 client.20367 trim_cache size 0 max 16384 2024-08-28T19:42:17.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.786+0000 7fec60427740 1 Processor -- start 2024-08-28T19:42:17.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.786+0000 7fec60427740 1 -- start start 2024-08-28T19:42:17.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.786+0000 7fec60427740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc1b1b0 0x5609b0000cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.786+0000 7fec60427740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 0x5609afff6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec60427740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc44070 0x5609affe5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec60427740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec541b72b0 con 0x5609afc23390 2024-08-28T19:42:17.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec60427740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec541b8eb0 con 0x5609afc1b1b0 2024-08-28T19:42:17.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec60427740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fec541b7980 con 0x5609afc44070 2024-08-28T19:42:17.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc1b1b0 0x5609b0000cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc1b1b0 0x5609b0000cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:39958/0 (socket says 172.21.15.16:39958) 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec537fe640 1 -- 172.21.15.16:0/2694868695 learned_addr learned my addr 172.21.15.16:0/2694868695 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec52ffd640 1 --2- 172.21.15.16:0/2694868695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 0x5609afff6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec52ffd640 1 -- 172.21.15.16:0/2694868695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc44070 msgr2=0x5609affe5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec53fff640 1 --2- 172.21.15.16:0/2694868695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc44070 0x5609affe5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec52ffd640 1 --2- 172.21.15.16:0/2694868695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc44070 0x5609affe5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec52ffd640 1 -- 172.21.15.16:0/2694868695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc1b1b0 msgr2=0x5609b0000cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec52ffd640 1 --2- 172.21.15.16:0/2694868695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc1b1b0 0x5609b0000cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec52ffd640 1 -- 172.21.15.16:0/2694868695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609afff3080 con 0x5609afc23390 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec53fff640 1 --2- 172.21.15.16:0/2694868695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc44070 0x5609affe5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.787+0000 7fec537fe640 1 --2- 172.21.15.16:0/2694868695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc1b1b0 0x5609b0000cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.788+0000 7fec52ffd640 1 --2- 172.21.15.16:0/2694868695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 0x5609afff6a60 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fec48000c00 tx=0x7fec480aafe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c858ad674a63400 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:17.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.788+0000 7fec397fa640 1 -- 172.21.15.16:0/2694868695 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec480af9b0 con 0x5609afc23390 2024-08-28T19:42:17.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.788+0000 7fec397fa640 1 -- 172.21.15.16:0/2694868695 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec480a6d60 con 0x5609afc23390 2024-08-28T19:42:17.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.788+0000 7fec397fa640 1 -- 172.21.15.16:0/2694868695 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec480039f0 con 0x5609afc23390 2024-08-28T19:42:17.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.789+0000 7fec60427740 1 -- 172.21.15.16:0/2694868695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 msgr2=0x5609afff6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:17.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.789+0000 7fec60427740 1 --2- 172.21.15.16:0/2694868695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 0x5609afff6a60 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fec48000c00 tx=0x7fec480aafe0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.789+0000 7fec60427740 1 -- 172.21.15.16:0/2694868695 shutdown_connections 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.789+0000 7fec60427740 1 --2- 172.21.15.16:0/2694868695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc44070 0x5609affe5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.789+0000 7fec60427740 1 --2- 172.21.15.16:0/2694868695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc23390 0x5609afff6a60 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.789+0000 7fec60427740 1 --2- 172.21.15.16:0/2694868695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc1b1b0 0x5609b0000cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.789+0000 7fec60427740 1 -- 172.21.15.16:0/2694868695 >> 172.21.15.16:0/2694868695 conn(0x5609afc38460 msgr2=0x5609affe5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.789+0000 7fec60427740 1 -- 172.21.15.16:0/2694868695 shutdown_connections 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.789+0000 7fec60427740 1 -- 172.21.15.16:0/2694868695 wait complete. 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.790+0000 7fec60427740 1 -- start start 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.790+0000 7fec60427740 1 Processor -- start 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec60427740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 0x5609afff9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec60427740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc23390 0x5609afff9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec60427740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609b006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec60427740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec541b8720 con 0x5609afc1b1b0 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec60427740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec541b8050 con 0x5609afc44070 2024-08-28T19:42:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec60427740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fec541b52b0 con 0x5609afc23390 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 0x5609afff9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 0x5609afff9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47846/0 (socket says 172.21.15.16:47846) 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec537fe640 1 -- 172.21.15.16:0/2726330301 learned_addr learned my addr 172.21.15.16:0/2726330301 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec52ffd640 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc23390 0x5609afff9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec60427740 10 client.0 _ll_register_callbacks cb 0x7fec480045a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.791+0000 7fec53fff640 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609b006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.792+0000 7fec537fe640 1 -- 172.21.15.16:0/2726330301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc23390 msgr2=0x5609afff9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.792+0000 7fec537fe640 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc23390 0x5609afff9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.792+0000 7fec537fe640 1 -- 172.21.15.16:0/2726330301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 msgr2=0x5609b006ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.792+0000 7fec537fe640 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609b006ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.792+0000 7fec537fe640 1 -- 172.21.15.16:0/2726330301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609b006f270 con 0x5609afc1b1b0 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.792+0000 7fec537fe640 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 0x5609afff9810 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fec4400faa0 tx=0x7fec440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad032dd4a397d77 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.792+0000 7fec32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:42:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.792+0000 7fec32ffd640 1 -- 172.21.15.16:0/2726330301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44035060 con 0x5609afc1b1b0 2024-08-28T19:42:17.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.793+0000 7fec32ffd640 1 -- 172.21.15.16:0/2726330301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7fec440054d0 con 0x5609afc1b1b0 2024-08-28T19:42:17.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.793+0000 7fec60427740 1 -- 172.21.15.16:0/2726330301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec480a6d60 con 0x5609afc1b1b0 2024-08-28T19:42:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.793+0000 7fec32ffd640 1 -- 172.21.15.16:0/2726330301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7fec4403c050 con 0x5609afc1b1b0 2024-08-28T19:42:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.793+0000 7fec60427740 10 client.20373 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.793+0000 7fec60427740 1 -- 172.21.15.16:0/2726330301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x5609b0076610 con 0x5609afc1b1b0 2024-08-28T19:42:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.793+0000 7fec32ffd640 1 -- 172.21.15.16:0/2726330301 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec480a6d60 con 0x5609afc1b1b0 2024-08-28T19:42:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.793+0000 7fec32ffd640 1 -- 172.21.15.16:0/2726330301 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7fec440307f0 con 0x5609afc1b1b0 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.793+0000 7fec60427740 10 client.20373 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.794+0000 7fec60427740 10 client.20373 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.794+0000 7fec60427740 20 client.20373 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.794+0000 7fec60427740 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5609afbd7da0 0x5609b0078de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.794+0000 7fec60427740 4 client.20373 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.794+0000 7fec60427740 1 -- 172.21.15.16:0/2726330301 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5609aff5e070 con 0x5609afbd7da0 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.794+0000 7fec52ffd640 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5609afbd7da0 0x5609b0078de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.794+0000 7fec32ffd640 10 client.20373 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.794+0000 7fec52ffd640 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5609afbd7da0 0x5609b0078de0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=85770002796aa35 server_cookie=c6890aba841de58c in_seq=0 out_seq=0 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec32ffd640 1 -- 172.21.15.16:0/2726330301 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5609aff5e070 con 0x5609afbd7da0 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec32ffd640 10 client.20373 handle_command_reply: tid=0 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec60427740 1 client.20373 shutdown 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec60427740 1 -- 172.21.15.16:0/2726330301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 msgr2=0x5609afff9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec60427740 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 0x5609afff9810 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fec4400faa0 tx=0x7fec440046c0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec60427740 1 -- 172.21.15.16:0/2726330301 shutdown_connections 2024-08-28T19:42:17.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec60427740 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x5609afbd7da0 0x5609b0078de0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec60427740 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609afc44070 0x5609b006ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec60427740 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5609afc23390 0x5609afff9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec60427740 1 --2- 172.21.15.16:0/2726330301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5609afc1b1b0 0x5609afff9810 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec32ffd640 0 client.20373 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:42:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec60427740 1 -- 172.21.15.16:0/2726330301 >> 172.21.15.16:0/2726330301 conn(0x5609afc38460 msgr2=0x5609affe5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.795+0000 7fec60427740 1 -- 172.21.15.16:0/2726330301 shutdown_connections 2024-08-28T19:42:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.796+0000 7fec60427740 1 -- 172.21.15.16:0/2726330301 wait complete. 2024-08-28T19:42:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.796+0000 7fec60427740 20 client.20373 trim_cache size 0 max 16384 2024-08-28T19:42:17.805 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.796+0000 7fec59eb2640 1 -- 172.21.15.16:0/2812593174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec3c0b4d20 msgr2=0x7fec3c0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.796+0000 7fec59eb2640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec3c0b4d20 0x7fec3c0b7230 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x5609affe7fd0 tx=0x7fec44005f90 comp rx=0 tx=0).stop 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.797+0000 7fec59eb2640 1 -- 172.21.15.16:0/2812593174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec5410aab0 msgr2=0x7fec541a2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.797+0000 7fec59eb2640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec5410aab0 0x7fec541a2f60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fec5406c480 tx=0x7fec48009a30 comp rx=0 tx=0).stop 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.797+0000 7fec59eb2640 1 -- 172.21.15.16:0/2812593174 shutdown_connections 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.797+0000 7fec59eb2640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec5410eef0 0x7fec541a8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.797+0000 7fec59eb2640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fec3c0b4d20 0x7fec3c0b7230 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.797+0000 7fec59eb2640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec5410aab0 0x7fec541a2f60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.797+0000 7fec59eb2640 1 --2- 172.21.15.16:0/2812593174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec5410a160 0x7fec541a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.797+0000 7fec59eb2640 1 -- 172.21.15.16:0/2812593174 >> 172.21.15.16:0/2812593174 conn(0x7fec54071090 msgr2=0x7fec5410d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.797+0000 7fec59eb2640 1 -- 172.21.15.16:0/2812593174 shutdown_connections 2024-08-28T19:42:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:17.797+0000 7fec59eb2640 1 -- 172.21.15.16:0/2812593174 wait complete. 2024-08-28T19:42:18.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 186721 22.62 MB/sec execute 354 sec latency 118.690 ms 2024-08-28T19:42:19.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:42:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:19 smithi115 ceph-mon[22184]: pgmap v1007: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.0 MiB/s wr, 472 op/s 2024-08-28T19:42:19.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 187166 22.61 MB/sec execute 355 sec latency 110.642 ms 2024-08-28T19:42:19.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:19 smithi016 ceph-mon[18556]: pgmap v1007: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.0 MiB/s wr, 472 op/s 2024-08-28T19:42:19.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:19 smithi028 ceph-mon[21769]: pgmap v1007: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.0 MiB/s wr, 472 op/s 2024-08-28T19:42:19.599 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:20.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.140+0000 7fa4cc700640 1 -- 172.21.15.16:0/1717149161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4c4108140 msgr2=0x7fa4c4108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.140+0000 7fa4cc700640 1 --2- 172.21.15.16:0/1717149161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4c4108140 0x7fa4c4108550 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009900 tx=0x7fa4b402a410 comp rx=0 tx=0).stop 2024-08-28T19:42:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.140+0000 7fa4cc700640 1 -- 172.21.15.16:0/1717149161 shutdown_connections 2024-08-28T19:42:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.140+0000 7fa4cc700640 1 --2- 172.21.15.16:0/1717149161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4c4108a90 0x7fa4c410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.140+0000 7fa4cc700640 1 --2- 172.21.15.16:0/1717149161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4c4108140 0x7fa4c4108550 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.140+0000 7fa4cc700640 1 --2- 172.21.15.16:0/1717149161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c4102800 0x7fa4c4102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.140+0000 7fa4cc700640 1 -- 172.21.15.16:0/1717149161 >> 172.21.15.16:0/1717149161 conn(0x7fa4c40794d0 msgr2=0x7fa4c40798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.140+0000 7fa4cc700640 1 -- 172.21.15.16:0/1717149161 shutdown_connections 2024-08-28T19:42:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.140+0000 7fa4cc700640 1 -- 172.21.15.16:0/1717149161 wait complete. 2024-08-28T19:42:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.141+0000 7fa4cc700640 1 Processor -- start 2024-08-28T19:42:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.141+0000 7fa4cc700640 1 -- start start 2024-08-28T19:42:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4cc700640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4c4102800 0x7fa4c41a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4cc700640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c4108140 0x7fa4c41a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4cc700640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4c4108a90 0x7fa4c41a6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4cc700640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c4112710 con 0x7fa4c4108a90 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4cc700640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c4112590 con 0x7fa4c4108140 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4ca49c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4c4102800 0x7fa4c41a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4c9c9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c4108140 0x7fa4c41a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4ca49c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4c4102800 0x7fa4c41a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:47276/0 (socket says 172.21.15.16:47276) 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4ca49c640 1 -- 172.21.15.16:0/1146736438 learned_addr learned my addr 172.21.15.16:0/1146736438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4c9c9b640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c4108140 0x7fa4c41a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:39992/0 (socket says 172.21.15.16:39992) 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4cac9d640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4c4108a90 0x7fa4c41a6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.142+0000 7fa4cc700640 1 -- 172.21.15.16:0/1146736438 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c4112890 con 0x7fa4c4102800 2024-08-28T19:42:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.143+0000 7fa4c9c9b640 1 -- 172.21.15.16:0/1146736438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4c4102800 msgr2=0x7fa4c41a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.143+0000 7fa4c9c9b640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4c4102800 0x7fa4c41a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.143+0000 7fa4c9c9b640 1 -- 172.21.15.16:0/1146736438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4c4108a90 msgr2=0x7fa4c41a6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.143+0000 7fa4c9c9b640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4c4108a90 0x7fa4c41a6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.143+0000 7fa4c9c9b640 1 -- 172.21.15.16:0/1146736438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c41a6740 con 0x7fa4c4108140 2024-08-28T19:42:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.143+0000 7fa4c9c9b640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c4108140 0x7fa4c41a0d80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa4b402a930 tx=0x7fa4b4003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.144+0000 7fa4bb7fe640 1 -- 172.21.15.16:0/1146736438 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b403f070 con 0x7fa4c4108140 2024-08-28T19:42:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.144+0000 7fa4bb7fe640 1 -- 172.21.15.16:0/1146736438 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4b4031b60 con 0x7fa4c4108140 2024-08-28T19:42:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.144+0000 7fa4bb7fe640 1 -- 172.21.15.16:0/1146736438 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4035030 con 0x7fa4c4108140 2024-08-28T19:42:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.144+0000 7fa4cc700640 1 -- 172.21.15.16:0/1146736438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c41a69d0 con 0x7fa4c4108140 2024-08-28T19:42:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.144+0000 7fa4cc700640 1 -- 172.21.15.16:0/1146736438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c41a6d70 con 0x7fa4c4108140 2024-08-28T19:42:20.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.147+0000 7fa4d2c75740 1 Processor -- start 2024-08-28T19:42:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.148+0000 7fa4d2c75740 1 -- start start 2024-08-28T19:42:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.148+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b88070 0x55f473f2d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.148+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 0x55f473f2d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.148+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f2dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.148+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f473ea4680 con 0x55f473b5f1b0 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.148+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f473e93c10 con 0x55f473b67390 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.148+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f473e97bc0 con 0x55f473b88070 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.148+0000 7fa4ca49c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 0x55f473f2d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.148+0000 7fa4ca49c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 0x55f473f2d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47874/0 (socket says 172.21.15.16:47874) 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.148+0000 7fa4ca49c640 1 -- 172.21.15.16:0/593923889 learned_addr learned my addr 172.21.15.16:0/593923889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.149+0000 7fa4cac9d640 1 --2- 172.21.15.16:0/593923889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b88070 0x55f473f2d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.149+0000 7fa4bb7fe640 1 -- 172.21.15.16:0/1146736438 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa4b4042070 con 0x7fa4c4108140 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.149+0000 7fa4ca49c640 1 -- 172.21.15.16:0/593923889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b88070 msgr2=0x55f473f2d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.149+0000 7fa4ca49c640 1 --2- 172.21.15.16:0/593923889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b88070 0x55f473f2d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.149+0000 7fa4ca49c640 1 -- 172.21.15.16:0/593923889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 msgr2=0x55f473f2dd90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-08-28T19:42:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.149+0000 7fa4ca49c640 1 --2- 172.21.15.16:0/593923889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f2dd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.149+0000 7fa4ca49c640 1 -- 172.21.15.16:0/593923889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f473bf3650 con 0x55f473b5f1b0 2024-08-28T19:42:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.149+0000 7fa4cac9d640 1 --2- 172.21.15.16:0/593923889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b88070 0x55f473f2d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:42:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.149+0000 7fa4ca49c640 1 --2- 172.21.15.16:0/593923889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 0x55f473f2d850 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0009d30 tx=0x7fa4c00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec34a94b39ddd38 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.149+0000 7fa4b8ff9640 1 -- 172.21.15.16:0/593923889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0012650 con 0x55f473b5f1b0 2024-08-28T19:42:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.149+0000 7fa4b8ff9640 1 -- 172.21.15.16:0/593923889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4c0009e20 con 0x55f473b5f1b0 2024-08-28T19:42:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.150+0000 7fa4b8ff9640 1 -- 172.21.15.16:0/593923889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c000fbd0 con 0x55f473b5f1b0 2024-08-28T19:42:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.150+0000 7fa4bb7fe640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4b00b4df0 0x7fa4b00b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.150+0000 7fa4bb7fe640 1 -- 172.21.15.16:0/1146736438 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7fa4b4105f60 con 0x7fa4c4108140 2024-08-28T19:42:20.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.150+0000 7fa4cac9d640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4b00b4df0 0x7fa4b00b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.150+0000 7fa4d2c75740 1 -- 172.21.15.16:0/593923889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 msgr2=0x55f473f2d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:20.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.150+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/593923889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 0x55f473f2d850 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0009d30 tx=0x7fa4c00117b0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.151+0000 7fa4d2c75740 1 -- 172.21.15.16:0/593923889 shutdown_connections 2024-08-28T19:42:20.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.151+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/593923889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f2dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.151+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/593923889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 0x55f473f2d850 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.151+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/593923889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b88070 0x55f473f2d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.151+0000 7fa4d2c75740 1 -- 172.21.15.16:0/593923889 >> 172.21.15.16:0/593923889 conn(0x55f473b7c460 msgr2=0x55f473f2b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.151+0000 7fa4d2c75740 1 -- 172.21.15.16:0/593923889 shutdown_connections 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.151+0000 7fa4cac9d640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4b00b4df0 0x7fa4b00b7300 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc005fd0 tx=0x7fa4bc009580 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.151+0000 7fa4d2c75740 1 -- 172.21.15.16:0/593923889 wait complete. 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.151+0000 7fa4d2c75740 1 -- start start 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4d2c75740 1 Processor -- start 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 0x55f473f389e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b67390 0x55f473f38f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b88070 0x55f473f44e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f473cf6700 con 0x55f473b5f1b0 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f473e1a750 con 0x55f473b88070 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f473f3bb40 con 0x55f473b67390 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4d2c75740 10 client.0 _ll_register_callbacks cb 0x55f473e49e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4cac9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b88070 0x55f473f44e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4cac9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b88070 0x55f473f44e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:40012/0 (socket says 172.21.15.16:40012) 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4cac9d640 1 -- 172.21.15.16:0/1613977489 learned_addr learned my addr 172.21.15.16:0/1613977489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4c9c9b640 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b67390 0x55f473f38f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.152+0000 7fa4ca49c640 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 0x55f473f389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4c9c9b640 1 -- 172.21.15.16:0/1613977489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b88070 msgr2=0x55f473f44e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4c9c9b640 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b88070 0x55f473f44e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4c9c9b640 1 -- 172.21.15.16:0/1613977489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 msgr2=0x55f473f389e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4c9c9b640 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 0x55f473f389e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4c9c9b640 1 -- 172.21.15.16:0/1613977489 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f473e53900 con 0x55f473b67390 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4ca49c640 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 0x55f473f389e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4cac9d640 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b88070 0x55f473f44e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4c9c9b640 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b67390 0x55f473f38f20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa4b41032e0 tx=0x7fa4b40cbbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c6b9cf808700f8e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4adffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:42:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4adffb640 1 -- 172.21.15.16:0/1613977489 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4049070 con 0x55f473b67390 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4d2c75740 1 -- 172.21.15.16:0/1613977489 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4c0009e20 con 0x55f473b67390 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.153+0000 7fa4adffb640 1 -- 172.21.15.16:0/1613977489 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7fa4b40c8510 con 0x55f473b67390 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.154+0000 7fa4adffb640 1 -- 172.21.15.16:0/1613977489 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7fa4b40cbd90 con 0x55f473b67390 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.154+0000 7fa4d2c75740 10 client.29389 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.154+0000 7fa4d2c75740 1 -- 172.21.15.16:0/1613977489 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x55f473f47860 con 0x55f473b67390 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.154+0000 7fa4adffb640 1 -- 172.21.15.16:0/1613977489 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4c0009e20 con 0x55f473b67390 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.154+0000 7fa4adffb640 1 -- 172.21.15.16:0/1613977489 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7fa4b403b4f0 con 0x55f473b67390 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.155+0000 7fa4d2c75740 10 client.29389 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.155+0000 7fa4d2c75740 10 client.29389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.155+0000 7fa4d2c75740 20 client.29389 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.155+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f473b469f0 0x55f473f4aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.155+0000 7fa4d2c75740 4 client.29389 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:42:20.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.155+0000 7fa4d2c75740 1 -- 172.21.15.16:0/1613977489 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f473ea65d0 con 0x55f473b469f0 2024-08-28T19:42:20.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.155+0000 7fa4ca49c640 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f473b469f0 0x55f473f4aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.156+0000 7fa4ca49c640 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f473b469f0 0x55f473f4aff0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96283cd97ec748f0 server_cookie=2292ac708ca410c8 in_seq=0 out_seq=0 2024-08-28T19:42:20.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.156+0000 7fa4adffb640 10 client.29389 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:20.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.158+0000 7fa4adffb640 1 -- 172.21.15.16:0/1613977489 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f473ea65d0 con 0x55f473b469f0 2024-08-28T19:42:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.158+0000 7fa4adffb640 10 client.29389 handle_command_reply: tid=0 2024-08-28T19:42:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.158+0000 7fa4d2c75740 1 client.29389 shutdown 2024-08-28T19:42:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.159+0000 7fa4d2c75740 1 -- 172.21.15.16:0/1613977489 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b67390 msgr2=0x55f473f38f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.159+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b67390 0x55f473f38f20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa4b41032e0 tx=0x7fa4b40cbbd0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.163+0000 7fa4d2c75740 1 -- 172.21.15.16:0/1613977489 shutdown_connections 2024-08-28T19:42:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.163+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b88070 0x55f473f44e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.163+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f473b469f0 0x55f473f4aff0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.163+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b67390 0x55f473f38f20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.163+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/1613977489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b5f1b0 0x55f473f389e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.163+0000 7fa4d2c75740 1 -- 172.21.15.16:0/1613977489 >> 172.21.15.16:0/1613977489 conn(0x55f473b7c460 msgr2=0x55f473f304a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.163+0000 7fa4adffb640 0 client.29389 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.163+0000 7fa4d2c75740 1 -- 172.21.15.16:0/1613977489 shutdown_connections 2024-08-28T19:42:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.164+0000 7fa4d2c75740 1 -- 172.21.15.16:0/1613977489 wait complete. 2024-08-28T19:42:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.164+0000 7fa4d2c75740 20 client.29389 trim_cache size 0 max 16384 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.183+0000 7fa4d2c75740 1 Processor -- start 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.183+0000 7fa4d2c75740 1 -- start start 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 0x55f473f30410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f30950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 0x55f473f35bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b71e0 con 0x55f473b88070 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b8de0 con 0x55f473b67390 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b78b0 con 0x55f473b5f1b0 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4c9c9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f30950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4ca49c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 0x55f473f30410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4cac9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 0x55f473f35bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4c9c9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f30950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:40024/0 (socket says 172.21.15.16:40024) 2024-08-28T19:42:20.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4ca49c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 0x55f473f30410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:47302/0 (socket says 172.21.15.16:47302) 2024-08-28T19:42:20.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4c9c9b640 1 -- 172.21.15.16:0/2700022313 learned_addr learned my addr 172.21.15.16:0/2700022313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:20.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4c9c9b640 1 -- 172.21.15.16:0/2700022313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 msgr2=0x55f473f30410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:20.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4c9c9b640 1 --2- 172.21.15.16:0/2700022313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 0x55f473f30410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4c9c9b640 1 -- 172.21.15.16:0/2700022313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 msgr2=0x55f473f35bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:20.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4c9c9b640 1 --2- 172.21.15.16:0/2700022313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 0x55f473f35bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4c9c9b640 1 -- 172.21.15.16:0/2700022313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f473f3b9d0 con 0x55f473b67390 2024-08-28T19:42:20.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4ca49c640 1 --2- 172.21.15.16:0/2700022313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 0x55f473f30410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:20.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.184+0000 7fa4cac9d640 1 --2- 172.21.15.16:0/2700022313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 0x55f473f35bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:42:20.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.185+0000 7fa4c9c9b640 1 --2- 172.21.15.16:0/2700022313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f30950 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40cc210 tx=0x7fa4b403b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2d5febc13388a90 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:20.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.185+0000 7fa4affff640 1 -- 172.21.15.16:0/2700022313 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4049030 con 0x55f473b67390 2024-08-28T19:42:20.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.185+0000 7fa4affff640 1 -- 172.21.15.16:0/2700022313 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4b40d0af0 con 0x55f473b67390 2024-08-28T19:42:20.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.185+0000 7fa4affff640 1 -- 172.21.15.16:0/2700022313 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b403b4f0 con 0x55f473b67390 2024-08-28T19:42:20.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.186+0000 7fa4d2c75740 1 -- 172.21.15.16:0/2700022313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 msgr2=0x55f473f30950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:20.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.186+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/2700022313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f30950 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40cc210 tx=0x7fa4b403b460 comp rx=0 tx=0).stop 2024-08-28T19:42:20.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.186+0000 7fa4d2c75740 1 -- 172.21.15.16:0/2700022313 shutdown_connections 2024-08-28T19:42:20.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.186+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/2700022313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 0x55f473f35bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.186+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/2700022313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f30950 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.186+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/2700022313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 0x55f473f30410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.186+0000 7fa4d2c75740 1 -- 172.21.15.16:0/2700022313 >> 172.21.15.16:0/2700022313 conn(0x55f473b7c460 msgr2=0x55f473f37260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:20.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.186+0000 7fa4d2c75740 1 -- 172.21.15.16:0/2700022313 shutdown_connections 2024-08-28T19:42:20.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.187+0000 7fa4d2c75740 1 -- 172.21.15.16:0/2700022313 wait complete. 2024-08-28T19:42:20.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.187+0000 7fa4d2c75740 1 -- start start 2024-08-28T19:42:20.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.187+0000 7fa4d2c75740 1 Processor -- start 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 0x55f473f41410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f3c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4d2c75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 0x55f473f3ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b8650 con 0x55f473b88070 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b7f80 con 0x55f473b67390 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4d2c75740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b51e0 con 0x55f473b5f1b0 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4d2c75740 10 client.0 _ll_register_callbacks cb 0x7fa4b4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4c9c9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f3c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4ca49c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 0x55f473f41410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4c9c9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f3c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:40032/0 (socket says 172.21.15.16:40032) 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4cac9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 0x55f473f3ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4c9c9b640 1 -- 172.21.15.16:0/985494419 learned_addr learned my addr 172.21.15.16:0/985494419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4cac9d640 1 -- 172.21.15.16:0/985494419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 msgr2=0x55f473f41410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4cac9d640 1 --2- 172.21.15.16:0/985494419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 0x55f473f41410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4cac9d640 1 -- 172.21.15.16:0/985494419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 msgr2=0x55f473f3c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4cac9d640 1 --2- 172.21.15.16:0/985494419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f3c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.188+0000 7fa4cac9d640 1 -- 172.21.15.16:0/985494419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f473f3d440 con 0x55f473b88070 2024-08-28T19:42:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.189+0000 7fa4cac9d640 1 --2- 172.21.15.16:0/985494419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 0x55f473f3ce00 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc011c40 tx=0x7fa4bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55871f685e22ec09 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.189+0000 7fa4adffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:42:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.189+0000 7fa4adffb640 1 -- 172.21.15.16:0/985494419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc020c50 con 0x55f473b88070 2024-08-28T19:42:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.189+0000 7fa4d2c75740 1 -- 172.21.15.16:0/985494419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4b40d0af0 con 0x55f473b88070 2024-08-28T19:42:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.189+0000 7fa4adffb640 1 -- 172.21.15.16:0/985494419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7fa4bc020df0 con 0x55f473b88070 2024-08-28T19:42:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.189+0000 7fa4adffb640 1 -- 172.21.15.16:0/985494419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7fa4bc02a470 con 0x55f473b88070 2024-08-28T19:42:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.189+0000 7fa4d2c75740 10 client.20391 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.189+0000 7fa4d2c75740 1 -- 172.21.15.16:0/985494419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x55f473fb57c0 con 0x55f473b88070 2024-08-28T19:42:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.189+0000 7fa4adffb640 1 -- 172.21.15.16:0/985494419 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4b40d0af0 con 0x55f473b88070 2024-08-28T19:42:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.190+0000 7fa4adffb640 1 -- 172.21.15.16:0/985494419 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7fa4bc00f040 con 0x55f473b88070 2024-08-28T19:42:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.190+0000 7fa4d2c75740 10 client.20391 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.190+0000 7fa4d2c75740 10 client.20391 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:42:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.190+0000 7fa4d2c75740 20 client.20391 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.190+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/985494419 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f473b1bda0 0x55f473fb80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.190+0000 7fa4d2c75740 4 client.20391 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:42:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.190+0000 7fa4d2c75740 1 -- 172.21.15.16:0/985494419 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f473ea2070 con 0x55f473b1bda0 2024-08-28T19:42:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.190+0000 7fa4ca49c640 1 --2- 172.21.15.16:0/985494419 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f473b1bda0 0x55f473fb80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.191+0000 7fa4ca49c640 1 --2- 172.21.15.16:0/985494419 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f473b1bda0 0x55f473fb80a0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d9b446ee2e7eb4a server_cookie=9728becd8b6c334 in_seq=0 out_seq=0 2024-08-28T19:42:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.191+0000 7fa4adffb640 10 client.20391 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:20.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.193+0000 7fa4adffb640 1 -- 172.21.15.16:0/985494419 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f473ea2070 con 0x55f473b1bda0 2024-08-28T19:42:20.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.193+0000 7fa4adffb640 10 client.20391 handle_command_reply: tid=0 2024-08-28T19:42:20.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 1 client.20391 shutdown 2024-08-28T19:42:20.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 1 -- 172.21.15.16:0/985494419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 msgr2=0x55f473f3ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:20.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/985494419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 0x55f473f3ce00 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc011c40 tx=0x7fa4bc004a40 comp rx=0 tx=0).stop 2024-08-28T19:42:20.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 1 -- 172.21.15.16:0/985494419 shutdown_connections 2024-08-28T19:42:20.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/985494419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f473b88070 0x55f473f3ce00 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/985494419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f473b67390 0x55f473f3c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/985494419 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55f473b1bda0 0x55f473fb80a0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 1 --2- 172.21.15.16:0/985494419 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f473b5f1b0 0x55f473f41410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 1 -- 172.21.15.16:0/985494419 >> 172.21.15.16:0/985494419 conn(0x55f473b7c460 msgr2=0x55f473f30410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:20.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4adffb640 0 client.20391 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:20.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 1 -- 172.21.15.16:0/985494419 shutdown_connections 2024-08-28T19:42:20.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 1 -- 172.21.15.16:0/985494419 wait complete. 2024-08-28T19:42:20.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.194+0000 7fa4d2c75740 20 client.20391 trim_cache size 0 max 16384 2024-08-28T19:42:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:42:20.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.195+0000 7fa4cc700640 1 -- 172.21.15.16:0/1146736438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4b00b4df0 msgr2=0x7fa4b00b7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:20.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.195+0000 7fa4cc700640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4b00b4df0 0x7fa4b00b7300 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc005fd0 tx=0x7fa4bc009580 comp rx=0 tx=0).stop 2024-08-28T19:42:20.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.195+0000 7fa4cc700640 1 -- 172.21.15.16:0/1146736438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c4108140 msgr2=0x7fa4c41a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:20.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.195+0000 7fa4cc700640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c4108140 0x7fa4c41a0d80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa4b402a930 tx=0x7fa4b4003c10 comp rx=0 tx=0).stop 2024-08-28T19:42:20.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.195+0000 7fa4cc700640 1 -- 172.21.15.16:0/1146736438 shutdown_connections 2024-08-28T19:42:20.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.195+0000 7fa4cc700640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4c4108a90 0x7fa4c41a6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.195+0000 7fa4cc700640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa4b00b4df0 0x7fa4b00b7300 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.195+0000 7fa4cc700640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c4108140 0x7fa4c41a0d80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.196+0000 7fa4cc700640 1 --2- 172.21.15.16:0/1146736438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa4c4102800 0x7fa4c41a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:20.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.196+0000 7fa4cc700640 1 -- 172.21.15.16:0/1146736438 >> 172.21.15.16:0/1146736438 conn(0x7fa4c40794d0 msgr2=0x7fa4c41051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:20.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.196+0000 7fa4cc700640 1 -- 172.21.15.16:0/1146736438 shutdown_connections 2024-08-28T19:42:20.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:20.196+0000 7fa4cc700640 1 -- 172.21.15.16:0/1146736438 wait complete. 2024-08-28T19:42:20.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 187701 22.63 MB/sec execute 356 sec latency 84.554 ms 2024-08-28T19:42:20.746 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 34.3 2024-08-28T19:42:20.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:42:21.097 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:21.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:21 smithi016 ceph-mon[18556]: pgmap v1008: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.2 MiB/s wr, 488 op/s 2024-08-28T19:42:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:21 smithi115 ceph-mon[22184]: pgmap v1008: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.2 MiB/s wr, 488 op/s 2024-08-28T19:42:21.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 188255 22.64 MB/sec execute 357 sec latency 97.770 ms 2024-08-28T19:42:21.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:21 smithi028 ceph-mon[21769]: pgmap v1008: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.2 MiB/s wr, 488 op/s 2024-08-28T19:42:21.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.648+0000 7f4619668640 1 -- 172.21.15.16:0/76923511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4614102800 msgr2=0x7f4614102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:21.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.648+0000 7f4619668640 1 --2- 172.21.15.16:0/76923511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4614102800 0x7f4614102c10 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f4604009900 tx=0x7f460402a410 comp rx=0 tx=0).stop 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.648+0000 7f4619668640 1 -- 172.21.15.16:0/76923511 shutdown_connections 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.648+0000 7f4619668640 1 --2- 172.21.15.16:0/76923511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4614108a90 0x7f461410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.648+0000 7f4619668640 1 --2- 172.21.15.16:0/76923511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4614108140 0x7f4614108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.648+0000 7f4619668640 1 --2- 172.21.15.16:0/76923511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4614102800 0x7f4614102c10 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.648+0000 7f4619668640 1 -- 172.21.15.16:0/76923511 >> 172.21.15.16:0/76923511 conn(0x7f46140794d0 msgr2=0x7f46140798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.649+0000 7f4619668640 1 -- 172.21.15.16:0/76923511 shutdown_connections 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.649+0000 7f4619668640 1 -- 172.21.15.16:0/76923511 wait complete. 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.650+0000 7f4619668640 1 Processor -- start 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.650+0000 7f4619668640 1 -- start start 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f4619668640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4614102800 0x7f46141a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f4619668640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4614108140 0x7f46141a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:21.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f4619668640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4614108a90 0x7f46141a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:21.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f4612ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4614102800 0x7f46141a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:21.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f4619668640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4614111600 con 0x7f4614102800 2024-08-28T19:42:21.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f4619668640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4614111480 con 0x7f4614108a90 2024-08-28T19:42:21.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f4612ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4614102800 0x7f46141a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47898/0 (socket says 172.21.15.16:47898) 2024-08-28T19:42:21.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f4619668640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4614111780 con 0x7f4614108140 2024-08-28T19:42:21.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f4612ffd640 1 -- 172.21.15.16:0/2404842892 learned_addr learned my addr 172.21.15.16:0/2404842892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:21.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f46127fc640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4614108140 0x7f46141a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:21.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f46137fe640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4614108a90 0x7f46141a60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:21.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.651+0000 7f4612ffd640 1 -- 172.21.15.16:0/2404842892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4614108140 msgr2=0x7f46141a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:42:21.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.652+0000 7f4612ffd640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4614108140 0x7f46141a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:21.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.652+0000 7f4612ffd640 1 -- 172.21.15.16:0/2404842892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4614108a90 msgr2=0x7f46141a60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:21.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.652+0000 7f4612ffd640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4614108a90 0x7f46141a60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:21.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.652+0000 7f4612ffd640 1 -- 172.21.15.16:0/2404842892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46141a67d0 con 0x7f4614102800 2024-08-28T19:42:21.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.652+0000 7f4612ffd640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4614102800 0x7f46141a0860 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f4604005940 tx=0x7f46040040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:21.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.652+0000 7f45fbfff640 1 -- 172.21.15.16:0/2404842892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460403f030 con 0x7f4614102800 2024-08-28T19:42:21.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.652+0000 7f45fbfff640 1 -- 172.21.15.16:0/2404842892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46040044c0 con 0x7f4614102800 2024-08-28T19:42:21.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.652+0000 7f45fbfff640 1 -- 172.21.15.16:0/2404842892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4604035070 con 0x7f4614102800 2024-08-28T19:42:21.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.652+0000 7f4619668640 1 -- 172.21.15.16:0/2404842892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46141a6a60 con 0x7f4614102800 2024-08-28T19:42:21.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.652+0000 7f4619668640 1 -- 172.21.15.16:0/2404842892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46141ad220 con 0x7f4614102800 2024-08-28T19:42:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.654+0000 7f4619668640 1 -- 172.21.15.16:0/2404842892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4614102c10 con 0x7f4614102800 2024-08-28T19:42:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.656+0000 7f45fbfff640 1 -- 172.21.15.16:0/2404842892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f4604042030 con 0x7f4614102800 2024-08-28T19:42:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.657+0000 7f45fbfff640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45fc0b4d20 0x7f45fc0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.657+0000 7f45fbfff640 1 -- 172.21.15.16:0/2404842892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f46041061e0 con 0x7f4614102800 2024-08-28T19:42:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.657+0000 7f46127fc640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45fc0b4d20 0x7f45fc0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.658+0000 7f46127fc640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45fc0b4d20 0x7f45fc0b7230 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f460800b3c0 tx=0x7f460800d040 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.660+0000 7f45fbfff640 1 -- 172.21.15.16:0/2404842892 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f4604049030 con 0x7f4614102800 2024-08-28T19:42:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.946+0000 7f4619668640 1 -- 172.21.15.16:0/2404842892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46141a1990 con 0x7f4614102800 2024-08-28T19:42:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.947+0000 7f45fbfff640 1 -- 172.21.15.16:0/2404842892 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+11986 (secure 0 0 0) 0x7f46141a1990 con 0x7f4614102800 2024-08-28T19:42:21.973 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":105,"btime":"2024-08-28T19:42:12:085855+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":105,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:42:11.646188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:42:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 105 2024-08-28T19:42:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 -- 172.21.15.16:0/2404842892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45fc0b4d20 msgr2=0x7f45fc0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45fc0b4d20 0x7f45fc0b7230 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f460800b3c0 tx=0x7f460800d040 comp rx=0 tx=0).stop 2024-08-28T19:42:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 -- 172.21.15.16:0/2404842892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4614102800 msgr2=0x7f46141a0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4614102800 0x7f46141a0860 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f4604005940 tx=0x7f46040040c0 comp rx=0 tx=0).stop 2024-08-28T19:42:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 -- 172.21.15.16:0/2404842892 shutdown_connections 2024-08-28T19:42:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f45fc0b4d20 0x7f45fc0b7230 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4614108a90 0x7f46141a60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4614108140 0x7f46141a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 --2- 172.21.15.16:0/2404842892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4614102800 0x7f46141a0860 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:21.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 -- 172.21.15.16:0/2404842892 >> 172.21.15.16:0/2404842892 conn(0x7f46140794d0 msgr2=0x7f46141011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:21.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 -- 172.21.15.16:0/2404842892 shutdown_connections 2024-08-28T19:42:21.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:21.950+0000 7f4619668640 1 -- 172.21.15.16:0/2404842892 wait complete. 2024-08-28T19:42:21.983 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-08-28T19:42:21.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-08-28T19:42:22.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:22 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2404842892' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:22.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 188767 22.66 MB/sec execute 358 sec latency 267.745 ms 2024-08-28T19:42:22.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:22 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2404842892' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:22 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2404842892' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:23.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 189290 22.66 MB/sec execute 359 sec latency 198.780 ms 2024-08-28T19:42:23.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:23 smithi016 ceph-mon[18556]: pgmap v1009: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 9.6 MiB/s wr, 581 op/s 2024-08-28T19:42:23.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:23 smithi028 ceph-mon[21769]: pgmap v1009: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 9.6 MiB/s wr, 581 op/s 2024-08-28T19:42:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:23 smithi115 ceph-mon[22184]: pgmap v1009: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 9.6 MiB/s wr, 581 op/s 2024-08-28T19:42:24.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 189951 22.70 MB/sec execute 360 sec latency 129.719 ms 2024-08-28T19:42:25.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 190572 22.75 MB/sec execute 361 sec latency 126.244 ms 2024-08-28T19:42:25.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:25 smithi016 ceph-mon[18556]: pgmap v1010: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 7.6 MiB/s wr, 460 op/s 2024-08-28T19:42:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:25 smithi028 ceph-mon[21769]: pgmap v1010: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 7.6 MiB/s wr, 460 op/s 2024-08-28T19:42:25.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:25 smithi115 ceph-mon[22184]: pgmap v1010: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 7.6 MiB/s wr, 460 op/s 2024-08-28T19:42:26.422 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db / --timeout 90 --expiration 154.3 --await-for 45.0' 2024-08-28T19:42:26.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db / --timeout 90 --expiration 154.3 --await-for 45.0 2024-08-28T19:42:26.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191176 22.76 MB/sec execute 362 sec latency 88.306 ms 2024-08-28T19:42:26.495 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.069+0000 7faed3928640 1 -- 172.21.15.16:0/1204753491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faecc102800 msgr2=0x7faecc102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.069+0000 7faed3928640 1 --2- 172.21.15.16:0/1204753491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faecc102800 0x7faecc102c10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7faec8009900 tx=0x7faec802a400 comp rx=0 tx=0).stop 2024-08-28T19:42:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.070+0000 7faed3928640 1 -- 172.21.15.16:0/1204753491 shutdown_connections 2024-08-28T19:42:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.070+0000 7faed3928640 1 --2- 172.21.15.16:0/1204753491 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faecc108a90 0x7faecc10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.070+0000 7faed3928640 1 --2- 172.21.15.16:0/1204753491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc108140 0x7faecc108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.070+0000 7faed3928640 1 --2- 172.21.15.16:0/1204753491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faecc102800 0x7faecc102c10 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.070+0000 7faed3928640 1 -- 172.21.15.16:0/1204753491 >> 172.21.15.16:0/1204753491 conn(0x7faecc0794d0 msgr2=0x7faecc0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.070+0000 7faed3928640 1 -- 172.21.15.16:0/1204753491 shutdown_connections 2024-08-28T19:42:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.070+0000 7faed3928640 1 -- 172.21.15.16:0/1204753491 wait complete. 2024-08-28T19:42:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.071+0000 7faed3928640 1 Processor -- start 2024-08-28T19:42:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.071+0000 7faed3928640 1 -- start start 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.071+0000 7faed3928640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faecc102800 0x7faecc197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.071+0000 7faed3928640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faecc108140 0x7faecc198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.071+0000 7faed16c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faecc102800 0x7faecc197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed3928640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc108a90 0x7faecc19d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed16c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faecc102800 0x7faecc197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:32964/0 (socket says 172.21.15.16:32964) 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed3928640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faecc111640 con 0x7faecc108a90 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed16c4640 1 -- 172.21.15.16:0/2574863240 learned_addr learned my addr 172.21.15.16:0/2574863240 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed3928640 1 -- 172.21.15.16:0/2574863240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faecc1114c0 con 0x7faecc102800 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed3928640 1 -- 172.21.15.16:0/2574863240 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faecc1117c0 con 0x7faecc108140 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed1ec5640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc108a90 0x7faecc19d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed0ec3640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faecc108140 0x7faecc198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed16c4640 1 -- 172.21.15.16:0/2574863240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faecc108140 msgr2=0x7faecc198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed16c4640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faecc108140 0x7faecc198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed16c4640 1 -- 172.21.15.16:0/2574863240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc108a90 msgr2=0x7faecc19d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed16c4640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc108a90 0x7faecc19d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.072+0000 7faed16c4640 1 -- 172.21.15.16:0/2574863240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faecc19de20 con 0x7faecc102800 2024-08-28T19:42:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.073+0000 7faed16c4640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faecc102800 0x7faecc197eb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7faec8009850 tx=0x7faec8030650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.073+0000 7faec27fc640 1 -- 172.21.15.16:0/2574863240 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec803f070 con 0x7faecc102800 2024-08-28T19:42:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.073+0000 7faed3928640 1 -- 172.21.15.16:0/2574863240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faecc19e0b0 con 0x7faecc102800 2024-08-28T19:42:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.074+0000 7faed3928640 1 -- 172.21.15.16:0/2574863240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faecc1a4870 con 0x7faecc102800 2024-08-28T19:42:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.074+0000 7faec27fc640 1 -- 172.21.15.16:0/2574863240 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec8004460 con 0x7faecc102800 2024-08-28T19:42:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.074+0000 7faec27fc640 1 -- 172.21.15.16:0/2574863240 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8035030 con 0x7faecc102800 2024-08-28T19:42:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.078+0000 7faec27fc640 1 -- 172.21.15.16:0/2574863240 <== mon.2 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7faec8042070 con 0x7faecc102800 2024-08-28T19:42:27.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.080+0000 7faec27fc640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faeb80b4da0 0x7faeb80b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.080+0000 7faed9e9d740 1 Processor -- start 2024-08-28T19:42:27.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.080+0000 7faed9e9d740 1 -- start start 2024-08-28T19:42:27.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.080+0000 7faec27fc640 1 -- 172.21.15.16:0/2574863240 <== mon.2 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7faec8106b70 con 0x7faecc102800 2024-08-28T19:42:27.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d341350 0x56519d6e7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed0ec3640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faeb80b4da0 0x7faeb80b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d318490 0x56519d6e7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d320670 0x56519d6e8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed9e9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56519d65d5f0 con 0x56519d318490 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed9e9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56519d64cb80 con 0x56519d320670 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed9e9d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56519d650b30 con 0x56519d341350 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed1ec5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d320670 0x56519d6e8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed16c4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d341350 0x56519d6e7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed1ec5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d320670 0x56519d6e8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:32966/0 (socket says 172.21.15.16:32966) 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed1ec5640 1 -- 172.21.15.16:0/2513114005 learned_addr learned my addr 172.21.15.16:0/2513114005 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed0ec3640 1 --2- 172.21.15.16:0/2513114005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d318490 0x56519d6e7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed16c4640 1 -- 172.21.15.16:0/2513114005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d320670 msgr2=0x56519d6e8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed16c4640 1 --2- 172.21.15.16:0/2513114005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d320670 0x56519d6e8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed16c4640 1 -- 172.21.15.16:0/2513114005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d318490 msgr2=0x56519d6e7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.081+0000 7faed16c4640 1 --2- 172.21.15.16:0/2513114005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d318490 0x56519d6e7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.082+0000 7faed16c4640 1 -- 172.21.15.16:0/2513114005 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56519d3ac930 con 0x56519d341350 2024-08-28T19:42:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.082+0000 7faed0ec3640 1 --2- 172.21.15.16:0/2513114005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d318490 0x56519d6e7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:42:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.082+0000 7faed0ec3640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faeb80b4da0 0x7faeb80b72b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7faebc00d5c0 tx=0x7faebc00b630 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.082+0000 7faed16c4640 1 --2- 172.21.15.16:0/2513114005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d341350 0x56519d6e7790 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faec8103e90 tx=0x7faec802a400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e36f9eb85c7fc49d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.082+0000 7faeb77fe640 1 -- 172.21.15.16:0/2513114005 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8049040 con 0x56519d341350 2024-08-28T19:42:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.083+0000 7faeb77fe640 1 -- 172.21.15.16:0/2513114005 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec80cce30 con 0x56519d341350 2024-08-28T19:42:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.083+0000 7faeb77fe640 1 -- 172.21.15.16:0/2513114005 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8003d00 con 0x56519d341350 2024-08-28T19:42:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.084+0000 7faed9e9d740 1 -- 172.21.15.16:0/2513114005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d341350 msgr2=0x56519d6e7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.084+0000 7faed9e9d740 1 --2- 172.21.15.16:0/2513114005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d341350 0x56519d6e7790 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faec8103e90 tx=0x7faec802a400 comp rx=0 tx=0).stop 2024-08-28T19:42:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.084+0000 7faed9e9d740 1 -- 172.21.15.16:0/2513114005 shutdown_connections 2024-08-28T19:42:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.084+0000 7faed9e9d740 1 --2- 172.21.15.16:0/2513114005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d320670 0x56519d6e8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.084+0000 7faed9e9d740 1 --2- 172.21.15.16:0/2513114005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d318490 0x56519d6e7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.084+0000 7faed9e9d740 1 --2- 172.21.15.16:0/2513114005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d341350 0x56519d6e7790 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.084+0000 7faed9e9d740 1 -- 172.21.15.16:0/2513114005 >> 172.21.15.16:0/2513114005 conn(0x56519d335740 msgr2=0x56519d6e5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.084+0000 7faed9e9d740 1 -- 172.21.15.16:0/2513114005 shutdown_connections 2024-08-28T19:42:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.084+0000 7faed9e9d740 1 -- 172.21.15.16:0/2513114005 wait complete. 2024-08-28T19:42:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.085+0000 7faed9e9d740 1 -- start start 2024-08-28T19:42:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.085+0000 7faed9e9d740 1 Processor -- start 2024-08-28T19:42:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.086+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d318490 0x56519d6fb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.086+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d320670 0x56519d6fbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.086+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d341350 0x56519d7076d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.086+0000 7faed16c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d318490 0x56519d6fb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.086+0000 7faed16c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d318490 0x56519d6fb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:32982/0 (socket says 172.21.15.16:32982) 2024-08-28T19:42:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.086+0000 7faed16c4640 1 -- 172.21.15.16:0/1943090719 learned_addr learned my addr 172.21.15.16:0/1943090719 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.086+0000 7faed9e9d740 1 -- 172.21.15.16:0/1943090719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56519d4af890 con 0x56519d341350 2024-08-28T19:42:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.086+0000 7faed9e9d740 1 -- 172.21.15.16:0/1943090719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56519d5d3790 con 0x56519d318490 2024-08-28T19:42:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.086+0000 7faed9e9d740 1 -- 172.21.15.16:0/1943090719 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56519d6fe460 con 0x56519d320670 2024-08-28T19:42:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.087+0000 7faed9e9d740 10 client.0 _ll_register_callbacks cb 0x56519d358ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.087+0000 7faed0ec3640 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d320670 0x56519d6fbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.087+0000 7faed16c4640 1 -- 172.21.15.16:0/1943090719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d320670 msgr2=0x56519d6fbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.087+0000 7faed16c4640 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d320670 0x56519d6fbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.087+0000 7faed16c4640 1 -- 172.21.15.16:0/1943090719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d341350 msgr2=0x56519d7076d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.087+0000 7faed16c4640 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d341350 0x56519d7076d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.087+0000 7faed16c4640 1 -- 172.21.15.16:0/1943090719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56519d60af30 con 0x56519d318490 2024-08-28T19:42:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.087+0000 7faed16c4640 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d318490 0x56519d6fb760 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7faec80ccb30 tx=0x7faec8004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=983d9f9c0a7f5ecc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.088+0000 7faeb4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:42:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.088+0000 7faeb4ff9640 1 -- 172.21.15.16:0/1943090719 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8049040 con 0x56519d318490 2024-08-28T19:42:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.088+0000 7faeb4ff9640 1 -- 172.21.15.16:0/1943090719 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7faec80d1300 con 0x56519d318490 2024-08-28T19:42:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.088+0000 7faed9e9d740 1 -- 172.21.15.16:0/1943090719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faec80cce30 con 0x56519d318490 2024-08-28T19:42:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.088+0000 7faeb4ff9640 1 -- 172.21.15.16:0/1943090719 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7faec80d6150 con 0x56519d318490 2024-08-28T19:42:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.088+0000 7faed9e9d740 10 client.29855 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.088+0000 7faed9e9d740 1 -- 172.21.15.16:0/1943090719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x56519d70a210 con 0x56519d318490 2024-08-28T19:42:27.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.088+0000 7faeb4ff9640 1 -- 172.21.15.16:0/1943090719 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec80cce30 con 0x56519d318490 2024-08-28T19:42:27.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.089+0000 7faeb4ff9640 1 -- 172.21.15.16:0/1943090719 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7faec8104110 con 0x56519d318490 2024-08-28T19:42:27.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.089+0000 7faed9e9d740 10 client.29855 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:27.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.089+0000 7faed9e9d740 10 client.29855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:42:27.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.089+0000 7faed9e9d740 20 client.29855 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:27.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.089+0000 7faed9e9d740 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56519d2ffcd0 0x56519d70d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.089+0000 7faed9e9d740 4 client.29855 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:42:27.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.089+0000 7faed9e9d740 1 -- 172.21.15.16:0/1943090719 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56519d65f540 con 0x56519d2ffcd0 2024-08-28T19:42:27.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.089+0000 7faed0ec3640 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56519d2ffcd0 0x56519d70d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.090+0000 7faed0ec3640 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56519d2ffcd0 0x56519d70d890 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15c42805ab2efbb7 server_cookie=a5aa4f5789d76cea in_seq=0 out_seq=0 2024-08-28T19:42:27.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.090+0000 7faeb4ff9640 10 client.29855 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:27.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faeb4ff9640 1 -- 172.21.15.16:0/1943090719 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56519d65f540 con 0x56519d2ffcd0 2024-08-28T19:42:27.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faeb4ff9640 10 client.29855 handle_command_reply: tid=0 2024-08-28T19:42:27.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faed9e9d740 1 client.29855 shutdown 2024-08-28T19:42:27.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faed9e9d740 1 -- 172.21.15.16:0/1943090719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d318490 msgr2=0x56519d6fb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:27.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faed9e9d740 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d318490 0x56519d6fb760 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7faec80ccb30 tx=0x7faec8004610 comp rx=0 tx=0).stop 2024-08-28T19:42:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faed9e9d740 1 -- 172.21.15.16:0/1943090719 shutdown_connections 2024-08-28T19:42:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faed9e9d740 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d341350 0x56519d7076d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faed9e9d740 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56519d2ffcd0 0x56519d70d890 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faed9e9d740 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d320670 0x56519d6fbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faed9e9d740 1 --2- 172.21.15.16:0/1943090719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d318490 0x56519d6fb760 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faeb4ff9640 0 client.29855 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faed9e9d740 1 -- 172.21.15.16:0/1943090719 >> 172.21.15.16:0/1943090719 conn(0x56519d335740 msgr2=0x56519d6e3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faed9e9d740 1 -- 172.21.15.16:0/1943090719 shutdown_connections 2024-08-28T19:42:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.093+0000 7faed9e9d740 1 -- 172.21.15.16:0/1943090719 wait complete. 2024-08-28T19:42:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.094+0000 7faed9e9d740 20 client.29855 trim_cache size 0 max 16384 2024-08-28T19:42:27.096 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:27.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed9e9d740 1 Processor -- start 2024-08-28T19:42:27.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed9e9d740 1 -- start start 2024-08-28T19:42:27.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d6fd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d318490 0x56519d6e4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d320670 0x56519d6e5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed9e9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faecc1aec90 con 0x56519d320670 2024-08-28T19:42:27.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed9e9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faecc1b0890 con 0x56519d2ffcd0 2024-08-28T19:42:27.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed9e9d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faecc1af360 con 0x56519d318490 2024-08-28T19:42:27.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed16c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d6fd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed16c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d6fd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:32996/0 (socket says 172.21.15.16:32996) 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed16c4640 1 -- 172.21.15.16:0/3210351733 learned_addr learned my addr 172.21.15.16:0/3210351733 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed0ec3640 1 --2- 172.21.15.16:0/3210351733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d318490 0x56519d6e4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.118+0000 7faed1ec5640 1 --2- 172.21.15.16:0/3210351733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d320670 0x56519d6e5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.119+0000 7faed16c4640 1 -- 172.21.15.16:0/3210351733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d318490 msgr2=0x56519d6e4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.119+0000 7faed16c4640 1 --2- 172.21.15.16:0/3210351733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d318490 0x56519d6e4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.119+0000 7faed16c4640 1 -- 172.21.15.16:0/3210351733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d320670 msgr2=0x56519d6e5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.119+0000 7faed16c4640 1 --2- 172.21.15.16:0/3210351733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d320670 0x56519d6e5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.119+0000 7faed16c4640 1 -- 172.21.15.16:0/3210351733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56519d6fca20 con 0x56519d2ffcd0 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.119+0000 7faed1ec5640 1 --2- 172.21.15.16:0/3210351733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d320670 0x56519d6e5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.119+0000 7faed16c4640 1 --2- 172.21.15.16:0/3210351733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d6fd640 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7faec802a9c0 tx=0x7faec802a400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b25299d9d0ba83c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:27.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.119+0000 7faeb6ffd640 1 -- 172.21.15.16:0/3210351733 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec80e7070 con 0x56519d2ffcd0 2024-08-28T19:42:27.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.119+0000 7faeb6ffd640 1 -- 172.21.15.16:0/3210351733 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec80b81b0 con 0x56519d2ffcd0 2024-08-28T19:42:27.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.119+0000 7faeb6ffd640 1 -- 172.21.15.16:0/3210351733 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec807c030 con 0x56519d2ffcd0 2024-08-28T19:42:27.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.120+0000 7faed9e9d740 1 -- 172.21.15.16:0/3210351733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 msgr2=0x56519d6fd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:27.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.120+0000 7faed9e9d740 1 --2- 172.21.15.16:0/3210351733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d6fd640 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7faec802a9c0 tx=0x7faec802a400 comp rx=0 tx=0).stop 2024-08-28T19:42:27.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.121+0000 7faed9e9d740 1 -- 172.21.15.16:0/3210351733 shutdown_connections 2024-08-28T19:42:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.121+0000 7faed9e9d740 1 --2- 172.21.15.16:0/3210351733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d320670 0x56519d6e5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.121+0000 7faed9e9d740 1 --2- 172.21.15.16:0/3210351733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d318490 0x56519d6e4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.121+0000 7faed9e9d740 1 --2- 172.21.15.16:0/3210351733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d6fd640 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.121+0000 7faed9e9d740 1 -- 172.21.15.16:0/3210351733 >> 172.21.15.16:0/3210351733 conn(0x56519d335740 msgr2=0x56519d6fbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.123+0000 7faed9e9d740 1 -- 172.21.15.16:0/3210351733 shutdown_connections 2024-08-28T19:42:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.123+0000 7faed9e9d740 1 -- 172.21.15.16:0/3210351733 wait complete. 2024-08-28T19:42:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.124+0000 7faed9e9d740 1 -- start start 2024-08-28T19:42:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.124+0000 7faed9e9d740 1 Processor -- start 2024-08-28T19:42:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.125+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d703fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.125+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d318490 0x56519d6ff4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.125+0000 7faed9e9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d320670 0x56519d6ffb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.125+0000 7faed9e9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faecc1b0100 con 0x56519d320670 2024-08-28T19:42:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.125+0000 7faed9e9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faecc1afa30 con 0x56519d2ffcd0 2024-08-28T19:42:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.125+0000 7faed9e9d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faecc1acc90 con 0x56519d318490 2024-08-28T19:42:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.126+0000 7faed9e9d740 10 client.0 _ll_register_callbacks cb 0x7faec400ba70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.126+0000 7faed16c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d703fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.126+0000 7faed16c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d703fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:33004/0 (socket says 172.21.15.16:33004) 2024-08-28T19:42:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.126+0000 7faed16c4640 1 -- 172.21.15.16:0/560900712 learned_addr learned my addr 172.21.15.16:0/560900712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:27.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.126+0000 7faed0ec3640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d318490 0x56519d6ff4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.126+0000 7faed1ec5640 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d320670 0x56519d6ffb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.126+0000 7faed16c4640 1 -- 172.21.15.16:0/560900712 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d318490 msgr2=0x56519d6ff4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.126+0000 7faed16c4640 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d318490 0x56519d6ff4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.126+0000 7faed16c4640 1 -- 172.21.15.16:0/560900712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d320670 msgr2=0x56519d6ffb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.126+0000 7faed16c4640 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d320670 0x56519d6ffb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.126+0000 7faed16c4640 1 -- 172.21.15.16:0/560900712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56519d700290 con 0x56519d2ffcd0 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.127+0000 7faed16c4640 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d703fb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7faec802a920 tx=0x7faec80cd260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d762f15d64200af3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.127+0000 7faeb4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.127+0000 7faeb4ff9640 1 -- 172.21.15.16:0/560900712 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec80e7070 con 0x56519d2ffcd0 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.127+0000 7faed9e9d740 1 -- 172.21.15.16:0/560900712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faec80b81b0 con 0x56519d2ffcd0 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.127+0000 7faeb4ff9640 1 -- 172.21.15.16:0/560900712 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7faec80d5cc0 con 0x56519d2ffcd0 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.127+0000 7faeb4ff9640 1 -- 172.21.15.16:0/560900712 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7faec8104320 con 0x56519d2ffcd0 2024-08-28T19:42:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.127+0000 7faed9e9d740 10 client.29861 fetch_fsmap learned FSMap version 105 2024-08-28T19:42:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.127+0000 7faed9e9d740 1 -- 172.21.15.16:0/560900712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x56519d700290 con 0x56519d2ffcd0 2024-08-28T19:42:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.128+0000 7faeb4ff9640 1 -- 172.21.15.16:0/560900712 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec80b81b0 con 0x56519d2ffcd0 2024-08-28T19:42:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.128+0000 7faeb4ff9640 1 -- 172.21.15.16:0/560900712 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 105) ==== 8386+0+0 (secure 0 0 0) 0x7faec80ee0c0 con 0x56519d2ffcd0 2024-08-28T19:42:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.128+0000 7faed9e9d740 10 client.29861 fetch_fsmap finished waiting for FSMap version 105 2024-08-28T19:42:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.128+0000 7faed9e9d740 10 client.29861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:42:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.128+0000 7faed9e9d740 20 client.29861 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.128+0000 7faed9e9d740 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56519d341350 0x56519d77a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.128+0000 7faed9e9d740 4 client.29861 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-08-28T19:42:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.128+0000 7faed9e9d740 1 -- 172.21.15.16:0/560900712 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56519d65afe0 con 0x56519d341350 2024-08-28T19:42:27.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.128+0000 7faed0ec3640 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56519d341350 0x56519d77a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:27.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.129+0000 7faeb4ff9640 10 client.29861 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:27.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.129+0000 7faed0ec3640 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56519d341350 0x56519d77a610 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ba38ed32f5feed4 server_cookie=e244b3410f99ce05 in_seq=0 out_seq=0 2024-08-28T19:42:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faeb4ff9640 1 -- 172.21.15.16:0/560900712 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56519d65afe0 con 0x56519d341350 2024-08-28T19:42:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faeb4ff9640 10 client.29861 handle_command_reply: tid=0 2024-08-28T19:42:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faed9e9d740 1 client.29861 shutdown 2024-08-28T19:42:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faed9e9d740 1 -- 172.21.15.16:0/560900712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 msgr2=0x56519d703fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faed9e9d740 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d703fb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7faec802a920 tx=0x7faec80cd260 comp rx=0 tx=0).stop 2024-08-28T19:42:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faed9e9d740 1 -- 172.21.15.16:0/560900712 shutdown_connections 2024-08-28T19:42:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faed9e9d740 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56519d320670 0x56519d6ffb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faed9e9d740 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56519d341350 0x56519d77a610 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faed9e9d740 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56519d318490 0x56519d6ff4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faed9e9d740 1 --2- 172.21.15.16:0/560900712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56519d2ffcd0 0x56519d703fb0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faed9e9d740 1 -- 172.21.15.16:0/560900712 >> 172.21.15.16:0/560900712 conn(0x56519d335740 msgr2=0x56519d6fbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.131+0000 7faeb4ff9640 0 client.29861 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.132+0000 7faed9e9d740 1 -- 172.21.15.16:0/560900712 shutdown_connections 2024-08-28T19:42:27.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.132+0000 7faed9e9d740 1 -- 172.21.15.16:0/560900712 wait complete. 2024-08-28T19:42:27.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.132+0000 7faed9e9d740 20 client.29861 trim_cache size 0 max 16384 2024-08-28T19:42:27.134 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:42:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-08-28T19:42:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "414082c7-7ecd-4292-9ec5-4c4504fedaf1", 2024-08-28T19:42:27.134 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-08-28T19:42:27.134 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:42:27.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.133+0000 7faed3928640 1 -- 172.21.15.16:0/2574863240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faeb80b4da0 msgr2=0x7faeb80b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:27.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.133+0000 7faed3928640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faeb80b4da0 0x7faeb80b72b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7faebc00d5c0 tx=0x7faebc00b630 comp rx=0 tx=0).stop 2024-08-28T19:42:27.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.133+0000 7faed3928640 1 -- 172.21.15.16:0/2574863240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faecc102800 msgr2=0x7faecc197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:27.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.133+0000 7faed3928640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faecc102800 0x7faecc197eb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7faec8009850 tx=0x7faec8030650 comp rx=0 tx=0).stop 2024-08-28T19:42:27.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.133+0000 7faed3928640 1 -- 172.21.15.16:0/2574863240 shutdown_connections 2024-08-28T19:42:27.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.133+0000 7faed3928640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc108a90 0x7faecc19d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.133+0000 7faed3928640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faecc108140 0x7faecc198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.133+0000 7faed3928640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faeb80b4da0 0x7faeb80b72b0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.133+0000 7faed3928640 1 --2- 172.21.15.16:0/2574863240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faecc102800 0x7faecc197eb0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:27.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.133+0000 7faed3928640 1 -- 172.21.15.16:0/2574863240 >> 172.21.15.16:0/2574863240 conn(0x7faecc0794d0 msgr2=0x7faecc109170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:27.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.134+0000 7faed3928640 1 -- 172.21.15.16:0/2574863240 shutdown_connections 2024-08-28T19:42:27.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:27.134+0000 7faed3928640 1 -- 172.21.15.16:0/2574863240 wait complete. 2024-08-28T19:42:27.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191603 22.78 MB/sec execute 363 sec latency 357.271 ms 2024-08-28T19:42:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:27 smithi016 ceph-mon[18556]: pgmap v1011: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 8.8 MiB/s wr, 534 op/s 2024-08-28T19:42:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:27 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:42:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:27 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:42:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:27 smithi028 ceph-mon[21769]: pgmap v1011: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 8.8 MiB/s wr, 534 op/s 2024-08-28T19:42:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:27 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:42:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:27 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:42:27.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:27 smithi115 ceph-mon[22184]: pgmap v1011: 129 pgs: 129 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 8.8 MiB/s wr, 534 op/s 2024-08-28T19:42:27.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:27 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:42:27.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:27 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:42:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.258+0000 7f2b84733640 1 -- 172.21.15.16:0/3901880530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c10aab0 msgr2=0x7f2b7c11c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.258+0000 7f2b84733640 1 --2- 172.21.15.16:0/3901880530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c10aab0 0x7f2b7c11c940 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c009900 tx=0x7f2b6c02a420 comp rx=0 tx=0).stop 2024-08-28T19:42:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.262+0000 7f2b84733640 1 -- 172.21.15.16:0/3901880530 shutdown_connections 2024-08-28T19:42:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.262+0000 7f2b84733640 1 --2- 172.21.15.16:0/3901880530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c11ce80 0x7f2b7c11f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.262+0000 7f2b84733640 1 --2- 172.21.15.16:0/3901880530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c10aab0 0x7f2b7c11c940 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.262+0000 7f2b84733640 1 --2- 172.21.15.16:0/3901880530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b7c10a160 0x7f2b7c10a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.262+0000 7f2b84733640 1 -- 172.21.15.16:0/3901880530 >> 172.21.15.16:0/3901880530 conn(0x7f2b7c070c20 msgr2=0x7f2b7c071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.263+0000 7f2b84733640 1 -- 172.21.15.16:0/3901880530 shutdown_connections 2024-08-28T19:42:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.263+0000 7f2b84733640 1 -- 172.21.15.16:0/3901880530 wait complete. 2024-08-28T19:42:28.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.266+0000 7f2b84733640 1 Processor -- start 2024-08-28T19:42:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.267+0000 7f2b84733640 1 -- start start 2024-08-28T19:42:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.267+0000 7f2b84733640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c10a160 0x7f2b7c1a9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.267+0000 7f2b84733640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c10aab0 0x7f2b7c1a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.267+0000 7f2b84733640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b7c11ce80 0x7f2b7c1ae870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.267+0000 7f2b84733640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c122660 con 0x7f2b7c10a160 2024-08-28T19:42:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.267+0000 7f2b84733640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c1224e0 con 0x7f2b7c10aab0 2024-08-28T19:42:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.267+0000 7f2b84733640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c1227e0 con 0x7f2b7c11ce80 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.267+0000 7f2b824cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c10a160 0x7f2b7c1a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.267+0000 7f2b81cce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c10aab0 0x7f2b7c1a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.267+0000 7f2b824cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c10a160 0x7f2b7c1a9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34166/0 (socket says 172.21.15.16:34166) 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.267+0000 7f2b824cf640 1 -- 172.21.15.16:0/2179171323 learned_addr learned my addr 172.21.15.16:0/2179171323 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.268+0000 7f2b82cd0640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b7c11ce80 0x7f2b7c1ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.268+0000 7f2b824cf640 1 -- 172.21.15.16:0/2179171323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b7c11ce80 msgr2=0x7f2b7c1ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.268+0000 7f2b824cf640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b7c11ce80 0x7f2b7c1ae870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.268+0000 7f2b824cf640 1 -- 172.21.15.16:0/2179171323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c10aab0 msgr2=0x7f2b7c1a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.268+0000 7f2b824cf640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c10aab0 0x7f2b7c1a95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.268+0000 7f2b824cf640 1 -- 172.21.15.16:0/2179171323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7c1aefa0 con 0x7f2b7c10a160 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.268+0000 7f2b81cce640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c10aab0 0x7f2b7c1a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:42:28.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.269+0000 7f2b824cf640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c10a160 0x7f2b7c1a9030 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400ebc0 tx=0x7f2b7400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.270+0000 7f2b737fe640 1 -- 172.21.15.16:0/2179171323 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74003b70 con 0x7f2b7c10a160 2024-08-28T19:42:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.270+0000 7f2b737fe640 1 -- 172.21.15.16:0/2179171323 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b74003d10 con 0x7f2b7c10a160 2024-08-28T19:42:28.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.270+0000 7f2b737fe640 1 -- 172.21.15.16:0/2179171323 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74019800 con 0x7f2b7c10a160 2024-08-28T19:42:28.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.272+0000 7f2b84733640 1 -- 172.21.15.16:0/2179171323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b7c1af290 con 0x7f2b7c10a160 2024-08-28T19:42:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.272+0000 7f2b84733640 1 -- 172.21.15.16:0/2179171323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b7c1b59f0 con 0x7f2b7c10a160 2024-08-28T19:42:28.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.277+0000 7f2b737fe640 1 -- 172.21.15.16:0/2179171323 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2b740199a0 con 0x7f2b7c10a160 2024-08-28T19:42:28.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.278+0000 7f2b737fe640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b680b4eb0 0x7f2b680b73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.278+0000 7f2b737fe640 1 -- 172.21.15.16:0/2179171323 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f2b740e70c0 con 0x7f2b7c10a160 2024-08-28T19:42:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.279+0000 7f2b8aca8740 1 Processor -- start 2024-08-28T19:42:28.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.279+0000 7f2b81cce640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b680b4eb0 0x7f2b680b73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.281+0000 7f2b8aca8740 1 -- start start 2024-08-28T19:42:28.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.282+0000 7f2b8aca8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320db840 0x558732482c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.282+0000 7f2b8aca8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587320b2980 0x5587324831d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.282+0000 7f2b8aca8740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320bab60 0x558732483710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.282+0000 7f2b8aca8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5587323f7330 con 0x5587320b2980 2024-08-28T19:42:28.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.282+0000 7f2b8aca8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5587323e68c0 con 0x5587320db840 2024-08-28T19:42:28.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.282+0000 7f2b8aca8740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5587323ea870 con 0x5587320bab60 2024-08-28T19:42:28.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.283+0000 7f2b82cd0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320bab60 0x558732483710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.283+0000 7f2b82cd0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320bab60 0x558732483710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:44832/0 (socket says 172.21.15.16:44832) 2024-08-28T19:42:28.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.283+0000 7f2b82cd0640 1 -- 172.21.15.16:0/3781722837 learned_addr learned my addr 172.21.15.16:0/3781722837 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.285+0000 7f2b824cf640 1 --2- 172.21.15.16:0/3781722837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320db840 0x558732482c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.286+0000 7f2b81cce640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b680b4eb0 0x7f2b680b73c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c009850 tx=0x7f2b6c000bc0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.288+0000 7f2b81cce640 1 --2- 172.21.15.16:0/3781722837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587320b2980 0x5587324831d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.288+0000 7f2b82cd0640 1 -- 172.21.15.16:0/3781722837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320db840 msgr2=0x558732482c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.288+0000 7f2b82cd0640 1 --2- 172.21.15.16:0/3781722837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320db840 0x558732482c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.288+0000 7f2b82cd0640 1 -- 172.21.15.16:0/3781722837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587320b2980 msgr2=0x5587324831d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.288+0000 7f2b82cd0640 1 --2- 172.21.15.16:0/3781722837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587320b2980 0x5587324831d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.288+0000 7f2b82cd0640 1 -- 172.21.15.16:0/3781722837 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55873239edf0 con 0x5587320bab60 2024-08-28T19:42:28.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.288+0000 7f2b81cce640 1 --2- 172.21.15.16:0/3781722837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587320b2980 0x5587324831d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:42:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.289+0000 7f2b82cd0640 1 --2- 172.21.15.16:0/3781722837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320bab60 0x558732483710 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009c10 tx=0x7f2b780117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a65962d6536ccbae server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.289+0000 7f2b70ff9640 1 -- 172.21.15.16:0/3781722837 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78012450 con 0x5587320bab60 2024-08-28T19:42:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.289+0000 7f2b70ff9640 1 -- 172.21.15.16:0/3781722837 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b78012aa0 con 0x5587320bab60 2024-08-28T19:42:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.290+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3781722837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320bab60 msgr2=0x558732483710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.290+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/3781722837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320bab60 0x558732483710 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009c10 tx=0x7f2b780117b0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.296+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3781722837 shutdown_connections 2024-08-28T19:42:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.296+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/3781722837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320bab60 0x558732483710 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.296+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/3781722837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587320b2980 0x5587324831d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.296+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/3781722837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320db840 0x558732482c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.296+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3781722837 >> 172.21.15.16:0/3781722837 conn(0x5587320cf990 msgr2=0x558732480b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:28.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.297+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3781722837 shutdown_connections 2024-08-28T19:42:28.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.297+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3781722837 wait complete. 2024-08-28T19:42:28.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.298+0000 7f2b8aca8740 1 -- start start 2024-08-28T19:42:28.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.298+0000 7f2b8aca8740 1 Processor -- start 2024-08-28T19:42:28.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.298+0000 7f2b8aca8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320b2980 0x55873248e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.298+0000 7f2b8aca8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587320bab60 0x55873248e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.298+0000 7f2b8aca8740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x55873248ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.298+0000 7f2b8aca8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5587322497a0 con 0x5587320bab60 2024-08-28T19:42:28.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.298+0000 7f2b8aca8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55873236d240 con 0x5587320b2980 2024-08-28T19:42:28.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.298+0000 7f2b8aca8740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5587324914c0 con 0x5587320db840 2024-08-28T19:42:28.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.299+0000 7f2b8aca8740 10 client.0 _ll_register_callbacks cb 0x5587320f3090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:28.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.299+0000 7f2b824cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320b2980 0x55873248e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.299+0000 7f2b824cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320b2980 0x55873248e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:33044/0 (socket says 172.21.15.16:33044) 2024-08-28T19:42:28.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.299+0000 7f2b824cf640 1 -- 172.21.15.16:0/2997741530 learned_addr learned my addr 172.21.15.16:0/2997741530 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:28.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.300+0000 7f2b81cce640 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587320bab60 0x55873248e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.300+0000 7f2b824cf640 1 -- 172.21.15.16:0/2997741530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 msgr2=0x55873248ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:28.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.300+0000 7f2b824cf640 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x55873248ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.300+0000 7f2b824cf640 1 -- 172.21.15.16:0/2997741530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587320bab60 msgr2=0x55873248e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:28.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.300+0000 7f2b824cf640 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587320bab60 0x55873248e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.300+0000 7f2b824cf640 1 -- 172.21.15.16:0/2997741530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587323a32b0 con 0x5587320b2980 2024-08-28T19:42:28.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.301+0000 7f2b824cf640 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320b2980 0x55873248e2d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2b740a9400 tx=0x7f2b740d02d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f43913d7cdcd2c6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:28.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.301+0000 7f2b65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:42:28.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.302+0000 7f2b65ffb640 1 -- 172.21.15.16:0/2997741530 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b740ad380 con 0x5587320b2980 2024-08-28T19:42:28.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.302+0000 7f2b65ffb640 1 -- 172.21.15.16:0/2997741530 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f2b740e4660 con 0x5587320b2980 2024-08-28T19:42:28.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.302+0000 7f2b8aca8740 1 -- 172.21.15.16:0/2997741530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b78012aa0 con 0x5587320b2980 2024-08-28T19:42:28.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.304+0000 7f2b65ffb640 1 -- 172.21.15.16:0/2997741530 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7f2b740e4860 con 0x5587320b2980 2024-08-28T19:42:28.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.305+0000 7f2b8aca8740 10 client.29873 fetch_fsmap learned FSMap version 106 2024-08-28T19:42:28.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.305+0000 7f2b8aca8740 1 -- 172.21.15.16:0/2997741530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x558732155b90 con 0x5587320b2980 2024-08-28T19:42:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.307+0000 7f2b65ffb640 1 -- 172.21.15.16:0/2997741530 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b78012aa0 con 0x5587320b2980 2024-08-28T19:42:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.307+0000 7f2b65ffb640 1 -- 172.21.15.16:0/2997741530 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 106) ==== 8390+0+0 (secure 0 0 0) 0x7f2b740cc020 con 0x5587320b2980 2024-08-28T19:42:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.307+0000 7f2b8aca8740 10 client.29873 fetch_fsmap finished waiting for FSMap version 106 2024-08-28T19:42:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.307+0000 7f2b8aca8740 10 client.29873 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:42:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.307+0000 7f2b8aca8740 20 client.29873 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.307+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55873209a1c0 0x5587324a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.307+0000 7f2b8aca8740 4 client.29873 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:42:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.307+0000 7f2b8aca8740 1 -- 172.21.15.16:0/2997741530 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587323f9280 con 0x55873209a1c0 2024-08-28T19:42:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.309+0000 7f2b81cce640 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55873209a1c0 0x5587324a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.315+0000 7f2b81cce640 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55873209a1c0 0x5587324a0af0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ca1b6b6a73ee555 server_cookie=b76dea0ec48e1eae in_seq=0 out_seq=0 2024-08-28T19:42:28.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.315+0000 7f2b65ffb640 10 client.29873 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:28.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.325+0000 7f2b65ffb640 1 -- 172.21.15.16:0/2997741530 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5587323f9280 con 0x55873209a1c0 2024-08-28T19:42:28.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.326+0000 7f2b65ffb640 10 client.29873 handle_command_reply: tid=0 2024-08-28T19:42:28.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.326+0000 7f2b8aca8740 1 client.29873 shutdown 2024-08-28T19:42:28.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.328+0000 7f2b8aca8740 1 -- 172.21.15.16:0/2997741530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320b2980 msgr2=0x55873248e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:28.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.328+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320b2980 0x55873248e2d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2b740a9400 tx=0x7f2b740d02d0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.329+0000 7f2b8aca8740 1 -- 172.21.15.16:0/2997741530 shutdown_connections 2024-08-28T19:42:28.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.329+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55873209a1c0 0x5587324a0af0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.329+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x55873248ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.329+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587320bab60 0x55873248e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.329+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/2997741530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320b2980 0x55873248e2d0 secure :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2b740a9400 tx=0x7f2b740d02d0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.329+0000 7f2b8aca8740 1 -- 172.21.15.16:0/2997741530 >> 172.21.15.16:0/2997741530 conn(0x5587320cf990 msgr2=0x558732485e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:28.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.329+0000 7f2b65ffb640 0 client.29873 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:28.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.334+0000 7f2b8aca8740 1 -- 172.21.15.16:0/2997741530 shutdown_connections 2024-08-28T19:42:28.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.335+0000 7f2b8aca8740 1 -- 172.21.15.16:0/2997741530 wait complete. 2024-08-28T19:42:28.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.339+0000 7f2b8aca8740 20 client.29873 trim_cache size 0 max 16384 2024-08-28T19:42:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.381+0000 7f2b8aca8740 1 Processor -- start 2024-08-28T19:42:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.381+0000 7f2b8aca8740 1 -- start start 2024-08-28T19:42:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.381+0000 7f2b8aca8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 0x5587324901b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.381+0000 7f2b8aca8740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x55873249a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.381+0000 7f2b8aca8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55873206f570 0x5587324864a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.381+0000 7f2b8aca8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b600100f0 con 0x55873206f570 2024-08-28T19:42:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.381+0000 7f2b8aca8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b60011d20 con 0x5587320bab60 2024-08-28T19:42:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.381+0000 7f2b8aca8740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2b600107c0 con 0x5587320db840 2024-08-28T19:42:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.382+0000 7f2b824cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 0x5587324901b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.383+0000 7f2b81cce640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x55873249a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.383+0000 7f2b824cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 0x5587324901b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:33056/0 (socket says 172.21.15.16:33056) 2024-08-28T19:42:28.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.383+0000 7f2b82cd0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55873206f570 0x5587324864a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.383+0000 7f2b824cf640 1 -- 172.21.15.16:0/906927102 learned_addr learned my addr 172.21.15.16:0/906927102 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:28.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.383+0000 7f2b824cf640 1 -- 172.21.15.16:0/906927102 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 msgr2=0x55873249a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:28.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.383+0000 7f2b824cf640 1 --2- 172.21.15.16:0/906927102 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x55873249a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.383+0000 7f2b824cf640 1 -- 172.21.15.16:0/906927102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55873206f570 msgr2=0x5587324864a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:28.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.383+0000 7f2b824cf640 1 --2- 172.21.15.16:0/906927102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55873206f570 0x5587324864a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.384+0000 7f2b824cf640 1 -- 172.21.15.16:0/906927102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587323b6d90 con 0x5587320bab60 2024-08-28T19:42:28.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.390+0000 7f2b824cf640 1 --2- 172.21.15.16:0/906927102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 0x5587324901b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2b740ad080 tx=0x7f2b74003f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f605a744024d98c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:28.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.391+0000 7f2b67fff640 1 -- 172.21.15.16:0/906927102 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b740b1bf0 con 0x5587320bab60 2024-08-28T19:42:28.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.391+0000 7f2b67fff640 1 -- 172.21.15.16:0/906927102 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b740b66a0 con 0x5587320bab60 2024-08-28T19:42:28.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.391+0000 7f2b67fff640 1 -- 172.21.15.16:0/906927102 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b740e4660 con 0x5587320bab60 2024-08-28T19:42:28.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.392+0000 7f2b8aca8740 1 -- 172.21.15.16:0/906927102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 msgr2=0x5587324901b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:28.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.392+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/906927102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 0x5587324901b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2b740ad080 tx=0x7f2b74003f70 comp rx=0 tx=0).stop 2024-08-28T19:42:28.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.393+0000 7f2b8aca8740 1 -- 172.21.15.16:0/906927102 shutdown_connections 2024-08-28T19:42:28.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.393+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/906927102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55873206f570 0x5587324864a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.393+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/906927102 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x55873249a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.393+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/906927102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 0x5587324901b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.393+0000 7f2b8aca8740 1 -- 172.21.15.16:0/906927102 >> 172.21.15.16:0/906927102 conn(0x5587320cf990 msgr2=0x558732485d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:28.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.394+0000 7f2b8aca8740 1 -- 172.21.15.16:0/906927102 shutdown_connections 2024-08-28T19:42:28.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.394+0000 7f2b8aca8740 1 -- 172.21.15.16:0/906927102 wait complete. 2024-08-28T19:42:28.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.394+0000 7f2b8aca8740 1 -- start start 2024-08-28T19:42:28.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.394+0000 7f2b8aca8740 1 Processor -- start 2024-08-28T19:42:28.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.395+0000 7f2b8aca8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55873206f570 0x558732496d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.395+0000 7f2b8aca8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 0x5587324931e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.396+0000 7f2b824cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55873206f570 0x558732496d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.396+0000 7f2b824cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55873206f570 0x558732496d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34188/0 (socket says 172.21.15.16:34188) 2024-08-28T19:42:28.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.396+0000 7f2b824cf640 1 -- 172.21.15.16:0/3918901052 learned_addr learned my addr 172.21.15.16:0/3918901052 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:28.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.396+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x558732493720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.396+0000 7f2b81cce640 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 0x5587324931e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.396+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3918901052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b60011590 con 0x55873206f570 2024-08-28T19:42:28.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.396+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3918901052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b60010ec0 con 0x5587320bab60 2024-08-28T19:42:28.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.396+0000 7f2b82cd0640 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x558732493720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.396+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3918901052 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6000e0f0 con 0x5587320db840 2024-08-28T19:42:28.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.397+0000 7f2b8aca8740 10 client.0 _ll_register_callbacks cb 0x7f2b7400eec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.398+0000 7f2b82cd0640 1 -- 172.21.15.16:0/3918901052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 msgr2=0x5587324931e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.398+0000 7f2b82cd0640 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 0x5587324931e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.398+0000 7f2b82cd0640 1 -- 172.21.15.16:0/3918901052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55873206f570 msgr2=0x558732496d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:28.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.398+0000 7f2b82cd0640 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55873206f570 0x558732496d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:28.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.398+0000 7f2b82cd0640 1 -- 172.21.15.16:0/3918901052 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558732493c60 con 0x5587320db840 2024-08-28T19:42:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.405+0000 7f2b82cd0640 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x558732493720 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55873248dc90 tx=0x7f2b78009f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=218b54737e7e2998 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.405+0000 7f2b65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:42:28.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.411+0000 7f2b65ffb640 1 -- 172.21.15.16:0/3918901052 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78028070 con 0x5587320db840 2024-08-28T19:42:28.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.411+0000 7f2b65ffb640 1 -- 172.21.15.16:0/3918901052 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f2b780050b0 con 0x5587320db840 2024-08-28T19:42:28.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.412+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3918901052 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b740b66a0 con 0x5587320db840 2024-08-28T19:42:28.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.421+0000 7f2b65ffb640 1 -- 172.21.15.16:0/3918901052 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7f2b78032050 con 0x5587320db840 2024-08-28T19:42:28.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.423+0000 7f2b8aca8740 10 client.29434 fetch_fsmap learned FSMap version 106 2024-08-28T19:42:28.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.423+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3918901052 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x55873250c150 con 0x5587320db840 2024-08-28T19:42:28.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.429+0000 7f2b65ffb640 1 -- 172.21.15.16:0/3918901052 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b740b66a0 con 0x5587320db840 2024-08-28T19:42:28.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.429+0000 7f2b65ffb640 1 -- 172.21.15.16:0/3918901052 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 106) ==== 8390+0+0 (secure 0 0 0) 0x7f2b78003ce0 con 0x5587320db840 2024-08-28T19:42:28.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.429+0000 7f2b8aca8740 10 client.29434 fetch_fsmap finished waiting for FSMap version 106 2024-08-28T19:42:28.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.429+0000 7f2b8aca8740 10 client.29434 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:42:28.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.429+0000 7f2b8aca8740 20 client.29434 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:28.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.429+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55873250c650 0x55873250ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:28.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.429+0000 7f2b8aca8740 4 client.29434 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 154.3, "await_for": 45.0}] 2024-08-28T19:42:28.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.429+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3918901052 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 154.3, "await_for": 45.0}) -- 0x5587323f4d20 con 0x55873250c650 2024-08-28T19:42:28.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.430+0000 7f2b824cf640 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55873250c650 0x55873250ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.432+0000 7f2b824cf640 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55873250c650 0x55873250ea70 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28747c294d358ba5 server_cookie=139ed480f2a7c611 in_seq=0 out_seq=0 2024-08-28T19:42:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:28.433+0000 7f2b65ffb640 10 client.29434 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191632 22.72 MB/sec execute 364 sec latency 1164.139 ms 2024-08-28T19:42:28.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:28 smithi016 ceph-mon[18556]: scrub queued for path: / 2024-08-28T19:42:28.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:28 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:28.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:28 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:42:28.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:28 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:42:28.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:28 smithi016 ceph-mon[18556]: scrub summary: active paths [/] 2024-08-28T19:42:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:28 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:28 smithi028 ceph-mon[21769]: scrub queued for path: / 2024-08-28T19:42:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:28 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:28 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:42:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:28 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:42:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:28 smithi028 ceph-mon[21769]: scrub summary: active paths [/] 2024-08-28T19:42:28.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:28 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:28 smithi115 ceph-mon[22184]: scrub queued for path: / 2024-08-28T19:42:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:28 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:28 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:42:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:28 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:42:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:28 smithi115 ceph-mon[22184]: scrub summary: active paths [/] 2024-08-28T19:42:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:28 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:29.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:42:29.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191635 22.65 MB/sec execute 365 sec latency 2165.101 ms 2024-08-28T19:42:29.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.744+0000 7f2b65ffb640 1 -- 172.21.15.16:0/3918901052 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+592 (crc 0 0 0) 0x5587323f4d20 con 0x55873250c650 2024-08-28T19:42:29.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.744+0000 7f2b65ffb640 10 client.29434 handle_command_reply: tid=0 2024-08-28T19:42:29.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.744+0000 7f2b8aca8740 1 client.29434 shutdown 2024-08-28T19:42:29.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:29 smithi016 ceph-mon[18556]: pgmap v1012: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 8.3 MiB/s wr, 513 op/s 2024-08-28T19:42:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.744+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3918901052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 msgr2=0x558732493720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.744+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x558732493720 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55873248dc90 tx=0x7f2b78009f10 comp rx=0 tx=0).stop 2024-08-28T19:42:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.744+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3918901052 shutdown_connections 2024-08-28T19:42:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.744+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55873250c650 0x55873250ea70 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.744+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587320db840 0x558732493720 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.744+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587320bab60 0x5587324931e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.744+0000 7f2b8aca8740 1 --2- 172.21.15.16:0/3918901052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55873206f570 0x558732496d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.744+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3918901052 >> 172.21.15.16:0/3918901052 conn(0x5587320cf990 msgr2=0x558732486750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.745+0000 7f2b65ffb640 0 client.29434 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.745+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3918901052 shutdown_connections 2024-08-28T19:42:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.745+0000 7f2b8aca8740 1 -- 172.21.15.16:0/3918901052 wait complete. 2024-08-28T19:42:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.745+0000 7f2b8aca8740 20 client.29434 trim_cache size 0 max 16384 2024-08-28T19:42:29.747 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 106, 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 70, 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "3685c114": { 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 70, 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 154.3, 2024-08-28T19:42:29.748 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:42:29.749 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:42:29.749 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:42:29.749 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:42:29.749 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-08-28T19:42:29.749 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:42:29.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:29.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:29.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:29.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:29.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:29.749 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:42:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.746+0000 7f2b84733640 1 -- 172.21.15.16:0/2179171323 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b680b4eb0 msgr2=0x7f2b680b73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.746+0000 7f2b84733640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b680b4eb0 0x7f2b680b73c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c009850 tx=0x7f2b6c000bc0 comp rx=0 tx=0).stop 2024-08-28T19:42:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.746+0000 7f2b84733640 1 -- 172.21.15.16:0/2179171323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c10a160 msgr2=0x7f2b7c1a9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.746+0000 7f2b84733640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c10a160 0x7f2b7c1a9030 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400ebc0 tx=0x7f2b7400c6a0 comp rx=0 tx=0).stop 2024-08-28T19:42:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.746+0000 7f2b84733640 1 -- 172.21.15.16:0/2179171323 shutdown_connections 2024-08-28T19:42:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.746+0000 7f2b84733640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b7c11ce80 0x7f2b7c1ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.746+0000 7f2b84733640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f2b680b4eb0 0x7f2b680b73c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.746+0000 7f2b84733640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c10aab0 0x7f2b7c1a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.747+0000 7f2b84733640 1 --2- 172.21.15.16:0/2179171323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c10a160 0x7f2b7c1a9030 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.747+0000 7f2b84733640 1 -- 172.21.15.16:0/2179171323 >> 172.21.15.16:0/2179171323 conn(0x7f2b7c070c20 msgr2=0x7f2b7c071b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.747+0000 7f2b84733640 1 -- 172.21.15.16:0/2179171323 shutdown_connections 2024-08-28T19:42:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:29.747+0000 7f2b84733640 1 -- 172.21.15.16:0/2179171323 wait complete. 2024-08-28T19:42:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:29 smithi028 ceph-mon[21769]: pgmap v1012: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 8.3 MiB/s wr, 513 op/s 2024-08-28T19:42:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:29 smithi115 ceph-mon[22184]: pgmap v1012: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 8.3 MiB/s wr, 513 op/s 2024-08-28T19:42:30.491 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '3685c114', quiesce took 9.7 seconds. Will release after: 24.599999999999998 2024-08-28T19:42:30.496 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 22.59 MB/sec execute 366 sec latency 3165.656 ms 2024-08-28T19:42:30.592 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:30.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:30.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:30 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:30 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:30 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:31.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.203+0000 7f85abfff640 1 -- 172.21.15.16:0/3504266209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ac1103e0 msgr2=0x7f85ac116c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:31.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.203+0000 7f85abfff640 1 --2- 172.21.15.16:0/3504266209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ac1103e0 0x7f85ac116c30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f859c009900 tx=0x7f859c02a440 comp rx=0 tx=0).stop 2024-08-28T19:42:31.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.210+0000 7f85abfff640 1 -- 172.21.15.16:0/3504266209 shutdown_connections 2024-08-28T19:42:31.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.210+0000 7f85abfff640 1 --2- 172.21.15.16:0/3504266209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ac1103e0 0x7f85ac116c30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.210+0000 7f85abfff640 1 --2- 172.21.15.16:0/3504266209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85ac10fa90 0x7f85ac10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.210+0000 7f85abfff640 1 --2- 172.21.15.16:0/3504266209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85ac10a3f0 0x7f85ac10a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.210+0000 7f85abfff640 1 -- 172.21.15.16:0/3504266209 >> 172.21.15.16:0/3504266209 conn(0x7f85ac06ebf0 msgr2=0x7f85ac06f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:31.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.214+0000 7f85abfff640 1 -- 172.21.15.16:0/3504266209 shutdown_connections 2024-08-28T19:42:31.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.215+0000 7f85abfff640 1 -- 172.21.15.16:0/3504266209 wait complete. 2024-08-28T19:42:31.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.216+0000 7f85abfff640 1 Processor -- start 2024-08-28T19:42:31.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.216+0000 7f85abfff640 1 -- start start 2024-08-28T19:42:31.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.216+0000 7f85abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ac10a3f0 0x7f85ac1a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.216+0000 7f85abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85ac10fa90 0x7f85ac1a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.216+0000 7f85abfff640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85ac1103e0 0x7f85ac1a6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.216+0000 7f85abfff640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac119d50 con 0x7f85ac10fa90 2024-08-28T19:42:31.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.216+0000 7f85abfff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac119bd0 con 0x7f85ac10a3f0 2024-08-28T19:42:31.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.216+0000 7f85abfff640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac119ed0 con 0x7f85ac1103e0 2024-08-28T19:42:31.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.217+0000 7f85ab7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85ac1103e0 0x7f85ac1a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.217+0000 7f85ab7fe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85ac1103e0 0x7f85ac1a6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:44864/0 (socket says 172.21.15.16:44864) 2024-08-28T19:42:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.217+0000 7f85ab7fe640 1 -- 172.21.15.16:0/2002610873 learned_addr learned my addr 172.21.15.16:0/2002610873 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.218+0000 7f85ab7fe640 1 -- 172.21.15.16:0/2002610873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ac10a3f0 msgr2=0x7f85ac1a07e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-28T19:42:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.218+0000 7f85ab7fe640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ac10a3f0 0x7f85ac1a07e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.218+0000 7f85ab7fe640 1 -- 172.21.15.16:0/2002610873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85ac10fa90 msgr2=0x7f85ac1a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.218+0000 7f85ab7fe640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85ac10fa90 0x7f85ac1a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.218+0000 7f85ab7fe640 1 -- 172.21.15.16:0/2002610873 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ac1a6750 con 0x7f85ac1103e0 2024-08-28T19:42:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.223+0000 7f85ab7fe640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85ac1103e0 0x7f85ac1a6020 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f859c02a9b0 tx=0x7f859c003d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.224+0000 7f85b0936640 1 -- 172.21.15.16:0/2002610873 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c03f070 con 0x7f85ac1103e0 2024-08-28T19:42:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.224+0000 7f85b0936640 1 -- 172.21.15.16:0/2002610873 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f859c031d60 con 0x7f85ac1103e0 2024-08-28T19:42:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.226+0000 7f85b0936640 1 -- 172.21.15.16:0/2002610873 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c035030 con 0x7f85ac1103e0 2024-08-28T19:42:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.227+0000 7f85abfff640 1 -- 172.21.15.16:0/2002610873 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ac1a69e0 con 0x7f85ac1103e0 2024-08-28T19:42:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.227+0000 7f85abfff640 1 -- 172.21.15.16:0/2002610873 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ac1ad1a0 con 0x7f85ac1103e0 2024-08-28T19:42:31.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.231+0000 7f85b7ead740 1 Processor -- start 2024-08-28T19:42:31.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.233+0000 7f85b7ead740 1 -- start start 2024-08-28T19:42:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.233+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x5610388adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.233+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.233+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388c1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.233+0000 7f85b7ead740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561038822630 con 0x5610384dd1b0 2024-08-28T19:42:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.233+0000 7f85b7ead740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561038811bc0 con 0x5610384e5390 2024-08-28T19:42:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.233+0000 7f85b7ead740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561038815b70 con 0x561038506070 2024-08-28T19:42:31.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.234+0000 7f85b0936640 1 -- 172.21.15.16:0/2002610873 <== mon.1 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f859c042070 con 0x7f85ac1103e0 2024-08-28T19:42:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.235+0000 7f85aa7fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x5610388adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.235+0000 7f85aa7fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x5610388adcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:44880/0 (socket says 172.21.15.16:44880) 2024-08-28T19:42:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.235+0000 7f85aa7fc640 1 -- 172.21.15.16:0/2501445349 learned_addr learned my addr 172.21.15.16:0/2501445349 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.235+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/2501445349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388c1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.237+0000 7f85b0936640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85900b4c50 0x7f85900b7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.237+0000 7f85ab7fe640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85900b4c50 0x7f85900b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.237+0000 7f85b0936640 1 -- 172.21.15.16:0/2002610873 <== mon.1 v2:172.21.15.115:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f859c1064a0 con 0x7f85ac1103e0 2024-08-28T19:42:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.237+0000 7f85aaffd640 1 --2- 172.21.15.16:0/2501445349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.241+0000 7f85ab7fe640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85900b4c50 0x7f85900b7160 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f859c045000 tx=0x7f859c0095e0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:31.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.241+0000 7f85aa7fc640 1 -- 172.21.15.16:0/2501445349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 msgr2=0x5610388c1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:31.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.241+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/2501445349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388c1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.241+0000 7f85aa7fc640 1 -- 172.21.15.16:0/2501445349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 msgr2=0x5610388ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:31.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.241+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/2501445349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388ae1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.241+0000 7f85aa7fc640 1 -- 172.21.15.16:0/2501445349 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561038571650 con 0x561038506070 2024-08-28T19:42:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.243+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/2501445349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x5610388adcb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f85980153a0 tx=0x7f8598009dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3268293fdf63fbc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.244+0000 7f85aaffd640 1 --2- 172.21.15.16:0/2501445349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:42:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.247+0000 7f85abfff640 1 -- 172.21.15.16:0/2501445349 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598012040 con 0x561038506070 2024-08-28T19:42:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.247+0000 7f85abfff640 1 -- 172.21.15.16:0/2501445349 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85980075a0 con 0x561038506070 2024-08-28T19:42:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.247+0000 7f85abfff640 1 -- 172.21.15.16:0/2501445349 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598004bc0 con 0x561038506070 2024-08-28T19:42:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.249+0000 7f85b7ead740 1 -- 172.21.15.16:0/2501445349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 msgr2=0x5610388adcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.249+0000 7f85b7ead740 1 --2- 172.21.15.16:0/2501445349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x5610388adcb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f85980153a0 tx=0x7f8598009dd0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.252+0000 7f85b7ead740 1 -- 172.21.15.16:0/2501445349 shutdown_connections 2024-08-28T19:42:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.253+0000 7f85b7ead740 1 --2- 172.21.15.16:0/2501445349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388c1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.253+0000 7f85b7ead740 1 --2- 172.21.15.16:0/2501445349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.253+0000 7f85b7ead740 1 --2- 172.21.15.16:0/2501445349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x5610388adcb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.253+0000 7f85b7ead740 1 -- 172.21.15.16:0/2501445349 >> 172.21.15.16:0/2501445349 conn(0x5610384fa460 msgr2=0x5610388ac090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:31.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.254+0000 7f85b7ead740 1 -- 172.21.15.16:0/2501445349 shutdown_connections 2024-08-28T19:42:31.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.254+0000 7f85b7ead740 1 -- 172.21.15.16:0/2501445349 wait complete. 2024-08-28T19:42:31.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.255+0000 7f85b7ead740 1 -- start start 2024-08-28T19:42:31.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.255+0000 7f85b7ead740 1 Processor -- start 2024-08-28T19:42:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.256+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5610384dd1b0 0x5610388bf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.256+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384e5390 0x5610388bfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.256+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561038506070 0x5610388cb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.256+0000 7f85b7ead740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561038674700 con 0x5610384e5390 2024-08-28T19:42:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.256+0000 7f85b7ead740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561038798700 con 0x561038506070 2024-08-28T19:42:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.256+0000 7f85b7ead740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5610388c2400 con 0x5610384dd1b0 2024-08-28T19:42:31.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.256+0000 7f85b7ead740 10 client.0 _ll_register_callbacks cb 0x5610387c7dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.256+0000 7f85aa7fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5610384dd1b0 0x5610388bf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.256+0000 7f85aa7fc640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5610384dd1b0 0x5610388bf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.16:44892/0 (socket says 172.21.15.16:44892) 2024-08-28T19:42:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.256+0000 7f85aa7fc640 1 -- 172.21.15.16:0/3377463055 learned_addr learned my addr 172.21.15.16:0/3377463055 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.256+0000 7f85aaffd640 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384e5390 0x5610388bfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.257+0000 7f85aa7fc640 1 -- 172.21.15.16:0/3377463055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561038506070 msgr2=0x5610388cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.257+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561038506070 0x5610388cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.257+0000 7f85aa7fc640 1 -- 172.21.15.16:0/3377463055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384e5390 msgr2=0x5610388bfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.257+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384e5390 0x5610388bfbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.257+0000 7f85aa7fc640 1 -- 172.21.15.16:0/3377463055 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610387d18b0 con 0x5610384dd1b0 2024-08-28T19:42:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.257+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5610384dd1b0 0x5610388bf690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f85980077d0 tx=0x7f8598004840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6635b69908337dd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.257+0000 7f858effd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:42:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.257+0000 7f858effd640 1 -- 172.21.15.16:0/3377463055 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598025060 con 0x5610384dd1b0 2024-08-28T19:42:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.258+0000 7f858effd640 1 -- 172.21.15.16:0/3377463055 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f8598008980 con 0x5610384dd1b0 2024-08-28T19:42:31.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.258+0000 7f85b7ead740 1 -- 172.21.15.16:0/3377463055 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85980075a0 con 0x5610384dd1b0 2024-08-28T19:42:31.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.259+0000 7f858effd640 1 -- 172.21.15.16:0/3377463055 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7f8598000ed0 con 0x5610384dd1b0 2024-08-28T19:42:31.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.259+0000 7f85b7ead740 10 client.29443 fetch_fsmap learned FSMap version 106 2024-08-28T19:42:31.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.259+0000 7f85b7ead740 1 -- 172.21.15.16:0/3377463055 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x5610388b2a10 con 0x5610384dd1b0 2024-08-28T19:42:31.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.263+0000 7f858effd640 1 -- 172.21.15.16:0/3377463055 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85980075a0 con 0x5610384dd1b0 2024-08-28T19:42:31.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.263+0000 7f858effd640 1 -- 172.21.15.16:0/3377463055 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 106) ==== 8390+0+0 (secure 0 0 0) 0x7f8598028050 con 0x5610384dd1b0 2024-08-28T19:42:31.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.263+0000 7f85b7ead740 10 client.29443 fetch_fsmap finished waiting for FSMap version 106 2024-08-28T19:42:31.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.263+0000 7f85b7ead740 10 client.29443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:42:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.263+0000 7f85b7ead740 20 client.29443 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.263+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5610384c49f0 0x5610388b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.263+0000 7f85b7ead740 4 client.29443 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:42:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.263+0000 7f85b7ead740 1 -- 172.21.15.16:0/3377463055 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561038824580 con 0x5610384c49f0 2024-08-28T19:42:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.264+0000 7f85aaffd640 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5610384c49f0 0x5610388b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.266+0000 7f858effd640 10 client.29443 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.266+0000 7f85aaffd640 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5610384c49f0 0x5610388b5e30 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7fa65f0308f0776 server_cookie=1dcdfc15858a3ef0 in_seq=0 out_seq=0 2024-08-28T19:42:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.270+0000 7f858effd640 1 -- 172.21.15.16:0/3377463055 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561038824580 con 0x5610384c49f0 2024-08-28T19:42:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.270+0000 7f858effd640 10 client.29443 handle_command_reply: tid=0 2024-08-28T19:42:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.270+0000 7f85b7ead740 1 client.29443 shutdown 2024-08-28T19:42:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.273+0000 7f85b7ead740 1 -- 172.21.15.16:0/3377463055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5610384dd1b0 msgr2=0x5610388bf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.273+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5610384dd1b0 0x5610388bf690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f85980077d0 tx=0x7f8598004840 comp rx=0 tx=0).stop 2024-08-28T19:42:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.273+0000 7f85b7ead740 1 -- 172.21.15.16:0/3377463055 shutdown_connections 2024-08-28T19:42:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.273+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561038506070 0x5610388cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.273+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384e5390 0x5610388bfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.273+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x5610384c49f0 0x5610388b5e30 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.273+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3377463055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5610384dd1b0 0x5610388bf690 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.273+0000 7f85b7ead740 1 -- 172.21.15.16:0/3377463055 >> 172.21.15.16:0/3377463055 conn(0x5610384fa460 msgr2=0x5610388aad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.273+0000 7f858effd640 0 client.29443 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.274+0000 7f85b7ead740 1 -- 172.21.15.16:0/3377463055 shutdown_connections 2024-08-28T19:42:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.274+0000 7f85b7ead740 1 -- 172.21.15.16:0/3377463055 wait complete. 2024-08-28T19:42:31.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.276+0000 7f85b7ead740 20 client.29443 trim_cache size 0 max 16384 2024-08-28T19:42:31.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.312+0000 7f85b7ead740 1 Processor -- start 2024-08-28T19:42:31.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.312+0000 7f85b7ead740 1 -- start start 2024-08-28T19:42:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.312+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.312+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388bc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.312+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x5610388bcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.312+0000 7f85b7ead740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b7060 con 0x5610384dd1b0 2024-08-28T19:42:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.312+0000 7f85b7ead740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b8c90 con 0x5610384e5390 2024-08-28T19:42:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.312+0000 7f85b7ead740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b7730 con 0x561038506070 2024-08-28T19:42:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.313+0000 7f85aaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388bc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.313+0000 7f85aaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388bc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:33104/0 (socket says 172.21.15.16:33104) 2024-08-28T19:42:31.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.313+0000 7f85aaffd640 1 -- 172.21.15.16:0/1831105493 learned_addr learned my addr 172.21.15.16:0/1831105493 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:31.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.313+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/1831105493 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x5610388bcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.314+0000 7f85aaffd640 1 -- 172.21.15.16:0/1831105493 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 msgr2=0x5610388bcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:31.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.314+0000 7f85aaffd640 1 --2- 172.21.15.16:0/1831105493 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x5610388bcdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.314+0000 7f85aaffd640 1 -- 172.21.15.16:0/1831105493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 msgr2=0x5610388ae0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:31.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.314+0000 7f85aaffd640 1 --2- 172.21.15.16:0/1831105493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.314+0000 7f85aaffd640 1 -- 172.21.15.16:0/1831105493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610388a4c40 con 0x5610384e5390 2024-08-28T19:42:31.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.315+0000 7f85aaffd640 1 --2- 172.21.15.16:0/1831105493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388bc880 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x5610388b6400 tx=0x7f85a4004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26fee956b71761cd server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:31.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.316+0000 7f85a17fa640 1 -- 172.21.15.16:0/1831105493 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a401b070 con 0x5610384e5390 2024-08-28T19:42:31.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.316+0000 7f85a17fa640 1 -- 172.21.15.16:0/1831105493 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561038824580 con 0x5610384e5390 2024-08-28T19:42:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.316+0000 7f85a17fa640 1 -- 172.21.15.16:0/1831105493 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a401ea00 con 0x5610384e5390 2024-08-28T19:42:31.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.318+0000 7f85b7ead740 1 -- 172.21.15.16:0/1831105493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 msgr2=0x5610388bc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:31.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.318+0000 7f85b7ead740 1 --2- 172.21.15.16:0/1831105493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388bc880 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x5610388b6400 tx=0x7f85a4004af0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.320+0000 7f85b7ead740 1 -- 172.21.15.16:0/1831105493 shutdown_connections 2024-08-28T19:42:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.320+0000 7f85b7ead740 1 --2- 172.21.15.16:0/1831105493 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x5610388bcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.320+0000 7f85b7ead740 1 --2- 172.21.15.16:0/1831105493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388bc880 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.320+0000 7f85b7ead740 1 --2- 172.21.15.16:0/1831105493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388ae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.320+0000 7f85b7ead740 1 -- 172.21.15.16:0/1831105493 >> 172.21.15.16:0/1831105493 conn(0x5610384fa460 msgr2=0x5610388ad990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.321+0000 7f85b7ead740 1 -- 172.21.15.16:0/1831105493 shutdown_connections 2024-08-28T19:42:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.321+0000 7f85b7ead740 1 -- 172.21.15.16:0/1831105493 wait complete. 2024-08-28T19:42:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.321+0000 7f85b7ead740 1 -- start start 2024-08-28T19:42:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.322+0000 7f85b7ead740 1 Processor -- start 2024-08-28T19:42:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.323+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.323+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.323+0000 7f85b7ead740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x561038943ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.323+0000 7f85aaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.323+0000 7f85aaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388c5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:33116/0 (socket says 172.21.15.16:33116) 2024-08-28T19:42:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.323+0000 7f85aaffd640 1 -- 172.21.15.16:0/3221546562 learned_addr learned my addr 172.21.15.16:0/3221546562 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.323+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x561038943ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.324+0000 7f85b7ead740 1 -- 172.21.15.16:0/3221546562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b8500 con 0x5610384dd1b0 2024-08-28T19:42:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.324+0000 7f85b7ead740 1 -- 172.21.15.16:0/3221546562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b7e30 con 0x5610384e5390 2024-08-28T19:42:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.324+0000 7f85b7ead740 1 -- 172.21.15.16:0/3221546562 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b5060 con 0x561038506070 2024-08-28T19:42:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.324+0000 7f85b7ead740 10 client.0 _ll_register_callbacks cb 0x7f8598001900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.325+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.325+0000 7f85aa7fc640 1 -- 172.21.15.16:0/3221546562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 msgr2=0x5610388c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.325+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.325+0000 7f85aa7fc640 1 -- 172.21.15.16:0/3221546562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 msgr2=0x5610388c5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.325+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388c5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.325+0000 7f85aa7fc640 1 -- 172.21.15.16:0/3221546562 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610388c6850 con 0x561038506070 2024-08-28T19:42:31.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.329+0000 7f85aa7fc640 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x561038943ed0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8598001350 tx=0x7f8598009dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34d43e4fe13dac9d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:31.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.330+0000 7f858effd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:42:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.333+0000 7f858effd640 1 -- 172.21.15.16:0/3221546562 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859802e070 con 0x561038506070 2024-08-28T19:42:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.334+0000 7f858effd640 1 -- 172.21.15.16:0/3221546562 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f8598004460 con 0x561038506070 2024-08-28T19:42:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.335+0000 7f85b7ead740 1 -- 172.21.15.16:0/3221546562 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561038824580 con 0x561038506070 2024-08-28T19:42:31.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.339+0000 7f858effd640 1 -- 172.21.15.16:0/3221546562 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7f859801d030 con 0x561038506070 2024-08-28T19:42:31.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.339+0000 7f85b7ead740 10 client.29449 fetch_fsmap learned FSMap version 106 2024-08-28T19:42:31.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.339+0000 7f85b7ead740 1 -- 172.21.15.16:0/3221546562 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x5610389454a0 con 0x561038506070 2024-08-28T19:42:31.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.340+0000 7f858effd640 1 -- 172.21.15.16:0/3221546562 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561038824580 con 0x561038506070 2024-08-28T19:42:31.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.341+0000 7f858effd640 1 -- 172.21.15.16:0/3221546562 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 106) ==== 8390+0+0 (secure 0 0 0) 0x7f8598028040 con 0x561038506070 2024-08-28T19:42:31.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.341+0000 7f85b7ead740 10 client.29449 fetch_fsmap finished waiting for FSMap version 106 2024-08-28T19:42:31.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.341+0000 7f85b7ead740 10 client.29449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:42:31.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.341+0000 7f85b7ead740 20 client.29449 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:31.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.341+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561038499da0 0x561038947c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:31.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.341+0000 7f85b7ead740 4 client.29449 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:42:31.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.341+0000 7f85b7ead740 1 -- 172.21.15.16:0/3221546562 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561038820020 con 0x561038499da0 2024-08-28T19:42:31.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.344+0000 7f85aaffd640 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561038499da0 0x561038947c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.351+0000 7f858effd640 10 client.29449 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.351+0000 7f85aaffd640 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561038499da0 0x561038947c80 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dde3947398c3366 server_cookie=7338b870d4c8e219 in_seq=0 out_seq=0 2024-08-28T19:42:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.352+0000 7f858effd640 1 -- 172.21.15.16:0/3221546562 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+8238 (crc 0 0 0) 0x561038820020 con 0x561038499da0 2024-08-28T19:42:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.352+0000 7f858effd640 10 client.29449 handle_command_reply: tid=0 2024-08-28T19:42:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.352+0000 7f85b7ead740 1 client.29449 shutdown 2024-08-28T19:42:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.353+0000 7f85b7ead740 1 -- 172.21.15.16:0/3221546562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 msgr2=0x561038943ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.353+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x561038943ed0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8598001350 tx=0x7f8598009dd0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.353+0000 7f85b7ead740 1 -- 172.21.15.16:0/3221546562 shutdown_connections 2024-08-28T19:42:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.353+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x561038499da0 0x561038947c80 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.353+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561038506070 0x561038943ed0 secure :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8598001350 tx=0x7f8598009dd0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.353+0000 7f858effd640 0 client.29449 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.353+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610384e5390 0x5610388c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.354+0000 7f85b7ead740 1 --2- 172.21.15.16:0/3221546562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610384dd1b0 0x5610388c5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.354+0000 7f85b7ead740 1 -- 172.21.15.16:0/3221546562 >> 172.21.15.16:0/3221546562 conn(0x5610384fa460 msgr2=0x5610388c0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.356+0000 7f85b7ead740 1 -- 172.21.15.16:0/3221546562 shutdown_connections 2024-08-28T19:42:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.357+0000 7f85b7ead740 1 -- 172.21.15.16:0/3221546562 wait complete. 2024-08-28T19:42:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.357+0000 7f85b7ead740 20 client.29449 trim_cache size 0 max 16384 2024-08-28T19:42:31.359 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:42:31.359 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:42:31.359 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:42:31.359 INFO:teuthology.orchestra.run.smithi016.stdout: "414082c7-7ecd-4292-9ec5-4c4504fedaf1": { 2024-08-28T19:42:31.359 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "414082c7-7ecd-4292-9ec5-4c4504fedaf1", 2024-08-28T19:42:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:42:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:42:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "414082c7-7ecd-4292-9ec5-4c4504fedaf1": { 2024-08-28T19:42:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:42:27.129235+0000", 2024-08-28T19:42:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "39fa9343-222a-493e-bb10-c8863c663453": { 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:41:33.609756+0000", 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "a466ab70-970a-4b3e-927e-47c65e508132": { 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:40:38.307336+0000", 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.361 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:55.619275+0000", 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:42:31.362 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.363 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.364 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:42:31.365 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.367 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.368 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "8de90bf3-5743-4711-98c2-a56ca0757291": { 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:39:48.098772+0000", 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.370 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.371 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.372 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.373 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:31.376 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:42:31.376 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:42:31.376 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:42:31.376 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:42:31.376 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:42:31.376 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:42:31.376 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:42:31.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:31.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:31.376 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:42:31.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.360+0000 7f85a27fc640 1 -- 172.21.15.16:0/2002610873 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85900b4c50 msgr2=0x7f85900b7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:31.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.360+0000 7f85a27fc640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85900b4c50 0x7f85900b7160 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f859c045000 tx=0x7f859c0095e0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.360+0000 7f85a27fc640 1 -- 172.21.15.16:0/2002610873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85ac1103e0 msgr2=0x7f85ac1a6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:31.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.360+0000 7f85a27fc640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85ac1103e0 0x7f85ac1a6020 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f859c02a9b0 tx=0x7f859c003d60 comp rx=0 tx=0).stop 2024-08-28T19:42:31.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.360+0000 7f85a27fc640 1 -- 172.21.15.16:0/2002610873 shutdown_connections 2024-08-28T19:42:31.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.360+0000 7f85a27fc640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85ac1103e0 0x7f85ac1a6020 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.360+0000 7f85a27fc640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85ac10fa90 0x7f85ac1a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.360+0000 7f85a27fc640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f85900b4c50 0x7f85900b7160 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.360+0000 7f85a27fc640 1 --2- 172.21.15.16:0/2002610873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ac10a3f0 0x7f85ac1a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:31.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.360+0000 7f85a27fc640 1 -- 172.21.15.16:0/2002610873 >> 172.21.15.16:0/2002610873 conn(0x7f85ac06ebf0 msgr2=0x7f85ac10dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:31.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.364+0000 7f85a27fc640 1 -- 172.21.15.16:0/2002610873 shutdown_connections 2024-08-28T19:42:31.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:31.364+0000 7f85a27fc640 1 -- 172.21.15.16:0/2002610873 wait complete. 2024-08-28T19:42:31.496 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 22.53 MB/sec execute 367 sec latency 4165.698 ms 2024-08-28T19:42:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:31 smithi016 ceph-mon[18556]: pgmap v1013: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.5 MiB/s wr, 516 op/s 2024-08-28T19:42:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:31 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:31 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:31 smithi016 ceph-mon[18556]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:31 smithi028 ceph-mon[21769]: pgmap v1013: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.5 MiB/s wr, 516 op/s 2024-08-28T19:42:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:31 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:31 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:31 smithi028 ceph-mon[21769]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:31 smithi115 ceph-mon[22184]: pgmap v1013: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.5 MiB/s wr, 516 op/s 2024-08-28T19:42:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:31 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:31 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:31.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:31 smithi115 ceph-mon[22184]: scrub summary: idle+waiting paths [/] 2024-08-28T19:42:32.496 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 22.47 MB/sec execute 368 sec latency 5165.741 ms 2024-08-28T19:42:32.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:32 smithi016 ceph-mon[18556]: pgmap v1014: 129 pgs: 129 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 18 MiB/s wr, 738 op/s 2024-08-28T19:42:32.702 INFO:tasks.cephfs.filesystem:scrub status for tag:414082c7-7ecd-4292-9ec5-4c4504fedaf1 - {'path': '/', 'tag': '414082c7-7ecd-4292-9ec5-4c4504fedaf1', 'options': 'recursive,force'} 2024-08-28T19:42:32.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:32 smithi028 ceph-mon[21769]: pgmap v1014: 129 pgs: 129 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 18 MiB/s wr, 738 op/s 2024-08-28T19:42:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:32 smithi115 ceph-mon[22184]: pgmap v1014: 129 pgs: 129 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 18 MiB/s wr, 738 op/s 2024-08-28T19:42:33.496 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 22.41 MB/sec execute 369 sec latency 6165.844 ms 2024-08-28T19:42:33.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:33 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:33 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:34.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:33 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:42:34.496 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 22.35 MB/sec execute 370 sec latency 7165.911 ms 2024-08-28T19:42:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:34 smithi028 ceph-mon[21769]: pgmap v1015: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 MiB/s wr, 617 op/s 2024-08-28T19:42:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:34 smithi115 ceph-mon[22184]: pgmap v1015: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 MiB/s wr, 617 op/s 2024-08-28T19:42:35.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:34 smithi016 ceph-mon[18556]: pgmap v1015: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 MiB/s wr, 617 op/s 2024-08-28T19:42:35.496 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 22.29 MB/sec execute 371 sec latency 8165.991 ms 2024-08-28T19:42:36.496 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 22.23 MB/sec execute 372 sec latency 9166.110 ms 2024-08-28T19:42:37.496 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 22.17 MB/sec execute 373 sec latency 10166.149 ms 2024-08-28T19:42:37.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:37 smithi016 ceph-mon[18556]: pgmap v1016: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 16 MiB/s wr, 640 op/s 2024-08-28T19:42:37.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:37 smithi028 ceph-mon[21769]: pgmap v1016: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 16 MiB/s wr, 640 op/s 2024-08-28T19:42:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:37 smithi115 ceph-mon[22184]: pgmap v1016: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 16 MiB/s wr, 640 op/s 2024-08-28T19:42:38.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 22.11 MB/sec execute 374 sec latency 11166.254 ms 2024-08-28T19:42:39.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 22.05 MB/sec execute 375 sec latency 12166.376 ms 2024-08-28T19:42:39.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:39 smithi016 ceph-mon[18556]: pgmap v1017: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 14 MiB/s wr, 545 op/s 2024-08-28T19:42:39.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:39 smithi028 ceph-mon[21769]: pgmap v1017: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 14 MiB/s wr, 545 op/s 2024-08-28T19:42:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:39 smithi115 ceph-mon[22184]: pgmap v1017: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 14 MiB/s wr, 545 op/s 2024-08-28T19:42:40.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.99 MB/sec execute 376 sec latency 13166.495 ms 2024-08-28T19:42:41.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.93 MB/sec execute 377 sec latency 14166.615 ms 2024-08-28T19:42:41.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:41 smithi016 ceph-mon[18556]: pgmap v1018: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 13 MiB/s wr, 479 op/s 2024-08-28T19:42:41.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:41 smithi028 ceph-mon[21769]: pgmap v1018: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 13 MiB/s wr, 479 op/s 2024-08-28T19:42:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:41 smithi115 ceph-mon[22184]: pgmap v1018: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 13 MiB/s wr, 479 op/s 2024-08-28T19:42:42.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.88 MB/sec execute 378 sec latency 15166.668 ms 2024-08-28T19:42:43.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.82 MB/sec execute 379 sec latency 16166.782 ms 2024-08-28T19:42:43.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:43 smithi016 ceph-mon[18556]: pgmap v1019: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 11 MiB/s wr, 443 op/s 2024-08-28T19:42:43.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:43 smithi028 ceph-mon[21769]: pgmap v1019: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 11 MiB/s wr, 443 op/s 2024-08-28T19:42:43.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:43 smithi115 ceph-mon[22184]: pgmap v1019: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 11 MiB/s wr, 443 op/s 2024-08-28T19:42:44.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.76 MB/sec execute 380 sec latency 17166.874 ms 2024-08-28T19:42:45.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.70 MB/sec execute 381 sec latency 18166.962 ms 2024-08-28T19:42:45.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:45 smithi016 ceph-mon[18556]: pgmap v1020: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 2.1 MiB/s wr, 178 op/s 2024-08-28T19:42:45.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:45 smithi028 ceph-mon[21769]: pgmap v1020: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 2.1 MiB/s wr, 178 op/s 2024-08-28T19:42:45.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:45 smithi115 ceph-mon[22184]: pgmap v1020: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 2.1 MiB/s wr, 178 op/s 2024-08-28T19:42:46.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.65 MB/sec execute 382 sec latency 19167.084 ms 2024-08-28T19:42:47.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.59 MB/sec execute 383 sec latency 20167.183 ms 2024-08-28T19:42:47.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:47 smithi016 ceph-mon[18556]: pgmap v1021: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 326 KiB/s wr, 110 op/s 2024-08-28T19:42:47.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:47 smithi028 ceph-mon[21769]: pgmap v1021: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 326 KiB/s wr, 110 op/s 2024-08-28T19:42:47.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:47 smithi115 ceph-mon[22184]: pgmap v1021: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 326 KiB/s wr, 110 op/s 2024-08-28T19:42:48.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.53 MB/sec execute 384 sec latency 21167.322 ms 2024-08-28T19:42:49.497 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.48 MB/sec execute 385 sec latency 22167.397 ms 2024-08-28T19:42:49.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:49 smithi016 ceph-mon[18556]: pgmap v1022: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 44 op/s 2024-08-28T19:42:49.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:49 smithi028 ceph-mon[21769]: pgmap v1022: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 44 op/s 2024-08-28T19:42:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:49 smithi115 ceph-mon[22184]: pgmap v1022: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 44 op/s 2024-08-28T19:42:50.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.42 MB/sec execute 386 sec latency 23167.455 ms 2024-08-28T19:42:51.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.37 MB/sec execute 387 sec latency 24167.495 ms 2024-08-28T19:42:51.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:51 smithi016 ceph-mon[18556]: pgmap v1023: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 17 op/s 2024-08-28T19:42:51.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:51 smithi028 ceph-mon[21769]: pgmap v1023: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 17 op/s 2024-08-28T19:42:51.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:51 smithi115 ceph-mon[22184]: pgmap v1023: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 17 op/s 2024-08-28T19:42:52.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.31 MB/sec execute 388 sec latency 25167.591 ms 2024-08-28T19:42:53.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.26 MB/sec execute 389 sec latency 26167.711 ms 2024-08-28T19:42:53.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:53 smithi016 ceph-mon[18556]: pgmap v1024: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 150 KiB/s wr, 20 op/s 2024-08-28T19:42:53.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:53 smithi028 ceph-mon[21769]: pgmap v1024: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 150 KiB/s wr, 20 op/s 2024-08-28T19:42:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:53 smithi115 ceph-mon[22184]: pgmap v1024: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 150 KiB/s wr, 20 op/s 2024-08-28T19:42:54.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.20 MB/sec execute 390 sec latency 27167.845 ms 2024-08-28T19:42:55.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:42:55.358 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:55.472 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:55 smithi016 ceph-mon[18556]: pgmap v1025: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 14 op/s 2024-08-28T19:42:55.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.15 MB/sec execute 391 sec latency 28167.940 ms 2024-08-28T19:42:55.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:55 smithi028 ceph-mon[21769]: pgmap v1025: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 14 op/s 2024-08-28T19:42:55.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.854+0000 7f0e1c718640 1 -- 172.21.15.16:0/3001651399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e14103960 msgr2=0x7f0e14103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:55.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.854+0000 7f0e1c718640 1 --2- 172.21.15.16:0/3001651399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e14103960 0x7f0e14103d70 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009900 tx=0x7f0e1002a3f0 comp rx=0 tx=0).stop 2024-08-28T19:42:55.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.854+0000 7f0e1c718640 1 -- 172.21.15.16:0/3001651399 shutdown_connections 2024-08-28T19:42:55.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.854+0000 7f0e1c718640 1 --2- 172.21.15.16:0/3001651399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e14109ba0 0x7f0e141103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:55.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.854+0000 7f0e1c718640 1 --2- 172.21.15.16:0/3001651399 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e14109250 0x7f0e14109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:55.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.854+0000 7f0e1c718640 1 --2- 172.21.15.16:0/3001651399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e14103960 0x7f0e14103d70 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:55.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.854+0000 7f0e1c718640 1 -- 172.21.15.16:0/3001651399 >> 172.21.15.16:0/3001651399 conn(0x7f0e1407b6b0 msgr2=0x7f0e1407bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:55.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.854+0000 7f0e1c718640 1 -- 172.21.15.16:0/3001651399 shutdown_connections 2024-08-28T19:42:55.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.854+0000 7f0e1c718640 1 -- 172.21.15.16:0/3001651399 wait complete. 2024-08-28T19:42:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.855+0000 7f0e1c718640 1 Processor -- start 2024-08-28T19:42:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.855+0000 7f0e1c718640 1 -- start start 2024-08-28T19:42:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.855+0000 7f0e1c718640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e14103960 0x7f0e1419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.855+0000 7f0e1c718640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e14109250 0x7f0e1419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.855+0000 7f0e1c718640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e14109ba0 0x7f0e1419f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.855+0000 7f0e1c718640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e14113860 con 0x7f0e14103960 2024-08-28T19:42:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.855+0000 7f0e1c718640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e141136e0 con 0x7f0e14109250 2024-08-28T19:42:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.855+0000 7f0e1c718640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0e141139e0 con 0x7f0e14109ba0 2024-08-28T19:42:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.855+0000 7f0e1a4b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e14103960 0x7f0e1419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e1a4b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e14103960 0x7f0e1419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41400/0 (socket says 172.21.15.16:41400) 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e1a4b4640 1 -- 172.21.15.16:0/315484025 learned_addr learned my addr 172.21.15.16:0/315484025 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e19cb3640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e14109250 0x7f0e1419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e1acb5640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e14109ba0 0x7f0e1419f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e1a4b4640 1 -- 172.21.15.16:0/315484025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e14109ba0 msgr2=0x7f0e1419f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e1a4b4640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e14109ba0 0x7f0e1419f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e1a4b4640 1 -- 172.21.15.16:0/315484025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e14109250 msgr2=0x7f0e1419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e1a4b4640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e14109250 0x7f0e1419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e1a4b4640 1 -- 172.21.15.16:0/315484025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e141a0060 con 0x7f0e14103960 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e1a4b4640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e14103960 0x7f0e1419a1b0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f0e10006010 tx=0x7f0e100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e0b7fe640 1 -- 172.21.15.16:0/315484025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1003f030 con 0x7f0e14103960 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e1c718640 1 -- 172.21.15.16:0/315484025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e141a02f0 con 0x7f0e14103960 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e0b7fe640 1 -- 172.21.15.16:0/315484025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e10003800 con 0x7f0e14103960 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e0b7fe640 1 -- 172.21.15.16:0/315484025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e10035070 con 0x7f0e14103960 2024-08-28T19:42:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.856+0000 7f0e1c718640 1 -- 172.21.15.16:0/315484025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e141a07d0 con 0x7f0e14103960 2024-08-28T19:42:55.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.857+0000 7f0e1c718640 1 -- 172.21.15.16:0/315484025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e14103d70 con 0x7f0e14103960 2024-08-28T19:42:55.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.859+0000 7f0e0b7fe640 1 -- 172.21.15.16:0/315484025 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0e10003cb0 con 0x7f0e14103960 2024-08-28T19:42:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.860+0000 7f0e0b7fe640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e000b4cd0 0x7f0e000b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.860+0000 7f0e19cb3640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e000b4cd0 0x7f0e000b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.860+0000 7f0e0b7fe640 1 -- 172.21.15.16:0/315484025 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f0e10106590 con 0x7f0e14103960 2024-08-28T19:42:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.861+0000 7f0e19cb3640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e000b4cd0 0x7f0e000b71e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0e04005e10 tx=0x7f0e04005da0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:55.863+0000 7f0e0b7fe640 1 -- 172.21.15.16:0/315484025 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f0e100ccfb0 con 0x7f0e14103960 2024-08-28T19:42:55.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:55 smithi115 ceph-mon[22184]: pgmap v1025: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 14 op/s 2024-08-28T19:42:56.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.090+0000 7f0e1c718640 1 -- 172.21.15.16:0/315484025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e1419b2e0 con 0x7f0e14103960 2024-08-28T19:42:56.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.091+0000 7f0e0b7fe640 1 -- 172.21.15.16:0/315484025 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+11991 (secure 0 0 0) 0x7f0e100cc8b0 con 0x7f0e14103960 2024-08-28T19:42:56.092 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:42:56.092 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":107,"btime":"2024-08-28T19:42:32:489400+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":107,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:42:32.040691+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:42:56.093 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 107 2024-08-28T19:42:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 -- 172.21.15.16:0/315484025 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e000b4cd0 msgr2=0x7f0e000b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e000b4cd0 0x7f0e000b71e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0e04005e10 tx=0x7f0e04005da0 comp rx=0 tx=0).stop 2024-08-28T19:42:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 -- 172.21.15.16:0/315484025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e14103960 msgr2=0x7f0e1419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e14103960 0x7f0e1419a1b0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f0e10006010 tx=0x7f0e100042f0 comp rx=0 tx=0).stop 2024-08-28T19:42:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 -- 172.21.15.16:0/315484025 shutdown_connections 2024-08-28T19:42:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0e14109ba0 0x7f0e1419f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0e000b4cd0 0x7f0e000b71e0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e14109250 0x7f0e1419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 --2- 172.21.15.16:0/315484025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e14103960 0x7f0e1419a1b0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 -- 172.21.15.16:0/315484025 >> 172.21.15.16:0/315484025 conn(0x7f0e1407b6b0 msgr2=0x7f0e141072a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 -- 172.21.15.16:0/315484025 shutdown_connections 2024-08-28T19:42:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:56.093+0000 7f0e1c718640 1 -- 172.21.15.16:0/315484025 wait complete. 2024-08-28T19:42:56.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.09 MB/sec execute 392 sec latency 29168.027 ms 2024-08-28T19:42:56.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:56 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/315484025' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:56.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:56 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/315484025' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:56 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/315484025' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:42:57.315 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --set-id 3685c114 --release --await' 2024-08-28T19:42:57.315 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --set-id 3685c114 --release --await 2024-08-28T19:42:57.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191637 21.04 MB/sec execute 393 sec latency 30168.071 ms 2024-08-28T19:42:57.578 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:42:57.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:57 smithi016 ceph-mon[18556]: pgmap v1026: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 117 KiB/s wr, 17 op/s 2024-08-28T19:42:57.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:57 smithi028 ceph-mon[21769]: pgmap v1026: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 117 KiB/s wr, 17 op/s 2024-08-28T19:42:57.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:57 smithi115 ceph-mon[22184]: pgmap v1026: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 117 KiB/s wr, 17 op/s 2024-08-28T19:42:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.256+0000 7f763b488640 1 -- 172.21.15.16:0/3940753606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634109b90 msgr2=0x7f76341103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.256+0000 7f763b488640 1 --2- 172.21.15.16:0/3940753606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634109b90 0x7f76341103c0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f762c009ca0 tx=0x7f762c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.256+0000 7f763b488640 1 -- 172.21.15.16:0/3940753606 shutdown_connections 2024-08-28T19:42:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.256+0000 7f763b488640 1 --2- 172.21.15.16:0/3940753606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634109b90 0x7f76341103c0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.256+0000 7f763b488640 1 --2- 172.21.15.16:0/3940753606 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7634109240 0x7f7634109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.256+0000 7f763b488640 1 --2- 172.21.15.16:0/3940753606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7634103900 0x7f7634103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.256+0000 7f763b488640 1 -- 172.21.15.16:0/3940753606 >> 172.21.15.16:0/3940753606 conn(0x7f763407b5f0 msgr2=0x7f763407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.256+0000 7f763b488640 1 -- 172.21.15.16:0/3940753606 shutdown_connections 2024-08-28T19:42:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.256+0000 7f763b488640 1 -- 172.21.15.16:0/3940753606 wait complete. 2024-08-28T19:42:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.257+0000 7f763b488640 1 Processor -- start 2024-08-28T19:42:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.257+0000 7f763b488640 1 -- start start 2024-08-28T19:42:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.257+0000 7f763b488640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634103900 0x7f76340754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.257+0000 7f763b488640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7634109240 0x7f76340759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.257+0000 7f763b488640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7634109b90 0x7f7634075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.257+0000 7f763b488640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7634113850 con 0x7f7634103900 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.257+0000 7f763b488640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76341136d0 con 0x7f7634109b90 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.257+0000 7f763b488640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f76341139d0 con 0x7f7634109240 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7639224640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634103900 0x7f76340754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7639224640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634103900 0x7f76340754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41410/0 (socket says 172.21.15.16:41410) 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7639224640 1 -- 172.21.15.16:0/1377950891 learned_addr learned my addr 172.21.15.16:0/1377950891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7639a25640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7634109b90 0x7f7634075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7638a23640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7634109240 0x7f76340759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7639224640 1 -- 172.21.15.16:0/1377950891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7634109240 msgr2=0x7f76340759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7639224640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7634109240 0x7f76340759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7639224640 1 -- 172.21.15.16:0/1377950891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7634109b90 msgr2=0x7f7634075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7639224640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7634109b90 0x7f7634075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7639224640 1 -- 172.21.15.16:0/1377950891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76341a8480 con 0x7f7634103900 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7638a23640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7634109240 0x7f76340759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f7639224640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634103900 0x7f76340754b0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f763000e9f0 tx=0x7f763000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f762a7fc640 1 -- 172.21.15.16:0/1377950891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763000cd80 con 0x7f7634103900 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f762a7fc640 1 -- 172.21.15.16:0/1377950891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76300108e0 con 0x7f7634103900 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f763b488640 1 -- 172.21.15.16:0/1377950891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76341a8770 con 0x7f7634103900 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.258+0000 7f762a7fc640 1 -- 172.21.15.16:0/1377950891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7630018980 con 0x7f7634103900 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.259+0000 7f763b488640 1 -- 172.21.15.16:0/1377950891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76341a8d00 con 0x7f7634103900 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.261+0000 7f76419fd740 1 Processor -- start 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.261+0000 7f762a7fc640 1 -- 172.21.15.16:0/1377950891 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f7630010430 con 0x7f7634103900 2024-08-28T19:42:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.261+0000 7f76419fd740 1 -- start start 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.261+0000 7f76419fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e3e520 0x55c1561e3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.261+0000 7f76419fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561e41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.261+0000 7f76419fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e1d840 0x55c1561e4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.261+0000 7f76419fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c15615aa70 con 0x55c155e1d840 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f76419fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c15614a000 con 0x55c155e3e520 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f76419fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c15614dfb0 con 0x55c155e15660 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7638a23640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e1d840 0x55c1561e4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7638a23640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e1d840 0x55c1561e4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41414/0 (socket says 172.21.15.16:41414) 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7639a25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e3e520 0x55c1561e3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7638a23640 1 -- 172.21.15.16:0/1957585303 learned_addr learned my addr 172.21.15.16:0/1957585303 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7639224640 1 --2- 172.21.15.16:0/1957585303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561e41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7639a25640 1 -- 172.21.15.16:0/1957585303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 msgr2=0x55c1561e41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7639a25640 1 --2- 172.21.15.16:0/1957585303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561e41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7639a25640 1 -- 172.21.15.16:0/1957585303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e1d840 msgr2=0x55c1561e4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7639a25640 1 --2- 172.21.15.16:0/1957585303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e1d840 0x55c1561e4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7639a25640 1 -- 172.21.15.16:0/1957585303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c155ea9b00 con 0x55c155e3e520 2024-08-28T19:42:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f762a7fc640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76200b4d20 0x7f76200b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7639a25640 1 --2- 172.21.15.16:0/1957585303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e3e520 0x55c1561e3c80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f762c005e30 tx=0x7f762c02f410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fef2a8605e51e50b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f7639224640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76200b4d20 0x7f76200b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.262+0000 7f762a7fc640 1 -- 172.21.15.16:0/1377950891 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f76300e5c50 con 0x7f7634103900 2024-08-28T19:42:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.263+0000 7f761f7fe640 1 -- 172.21.15.16:0/1957585303 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c031cf0 con 0x55c155e3e520 2024-08-28T19:42:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.263+0000 7f761f7fe640 1 -- 172.21.15.16:0/1957585303 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f762c036030 con 0x55c155e3e520 2024-08-28T19:42:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.263+0000 7f761f7fe640 1 -- 172.21.15.16:0/1957585303 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c004040 con 0x55c155e3e520 2024-08-28T19:42:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.263+0000 7f76419fd740 1 -- 172.21.15.16:0/1957585303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e3e520 msgr2=0x55c1561e3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.263+0000 7f76419fd740 1 --2- 172.21.15.16:0/1957585303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e3e520 0x55c1561e3c80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f762c005e30 tx=0x7f762c02f410 comp rx=0 tx=0).stop 2024-08-28T19:42:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.263+0000 7f7639224640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76200b4d20 0x7f76200b7230 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f76300e3080 tx=0x7f7630009340 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.264+0000 7f76419fd740 1 -- 172.21.15.16:0/1957585303 shutdown_connections 2024-08-28T19:42:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.264+0000 7f76419fd740 1 --2- 172.21.15.16:0/1957585303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e1d840 0x55c1561e4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.264+0000 7f76419fd740 1 --2- 172.21.15.16:0/1957585303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561e41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.264+0000 7f76419fd740 1 --2- 172.21.15.16:0/1957585303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e3e520 0x55c1561e3c80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.264+0000 7f76419fd740 1 -- 172.21.15.16:0/1957585303 >> 172.21.15.16:0/1957585303 conn(0x55c155e32910 msgr2=0x55c1561e1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.264+0000 7f76419fd740 1 -- 172.21.15.16:0/1957585303 shutdown_connections 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.264+0000 7f76419fd740 1 -- 172.21.15.16:0/1957585303 wait complete. 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.264+0000 7f76419fd740 1 -- start start 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.264+0000 7f76419fd740 1 Processor -- start 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.264+0000 7f76419fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e15660 0x55c1561ef800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f76419fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e1d840 0x55c1561efd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f76419fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e3e520 0x55c1561fb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f76419fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c1561f2280 con 0x55c155e15660 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f76419fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c15610b520 con 0x55c155e3e520 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f76419fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c1561f2400 con 0x55c155e1d840 2024-08-28T19:42:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f76419fd740 10 client.0 _ll_register_callbacks cb 0x55c156143c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f7639a25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e15660 0x55c1561ef800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f7639a25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e15660 0x55c1561ef800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41426/0 (socket says 172.21.15.16:41426) 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f7639a25640 1 -- 172.21.15.16:0/2857787018 learned_addr learned my addr 172.21.15.16:0/2857787018 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f7639a25640 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e3e520 0x55c1561fb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f7638a23640 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e1d840 0x55c1561efd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f7639a25640 1 -- 172.21.15.16:0/2857787018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e1d840 msgr2=0x55c1561efd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f7639a25640 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e1d840 0x55c1561efd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f7639a25640 1 -- 172.21.15.16:0/2857787018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e3e520 msgr2=0x55c1561fb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f7639a25640 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e3e520 0x55c1561fb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f7639a25640 1 -- 172.21.15.16:0/2857787018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c155f20960 con 0x55c155e15660 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f7639a25640 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e15660 0x55c1561ef800 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f762c003b90 tx=0x7f762c003bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506e34a3a4cc3adc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f761cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.265+0000 7f761cff9640 1 -- 172.21.15.16:0/2857787018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c058060 con 0x55c155e15660 2024-08-28T19:42:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f761cff9640 1 -- 172.21.15.16:0/2857787018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f762c0316f0 con 0x55c155e15660 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f76419fd740 1 -- 172.21.15.16:0/2857787018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c1560e99a0 con 0x55c155e15660 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f761cff9640 1 -- 172.21.15.16:0/2857787018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f762c0318f0 con 0x55c155e15660 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f76419fd740 10 client.20445 fetch_fsmap learned FSMap version 107 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f76419fd740 1 -- 172.21.15.16:0/2857787018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55c1561f2280 con 0x55c155e15660 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f761cff9640 1 -- 172.21.15.16:0/2857787018 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c1560e99a0 con 0x55c155e15660 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f761cff9640 1 -- 172.21.15.16:0/2857787018 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 107) ==== 8398+0+0 (secure 0 0 0) 0x7f762c04d070 con 0x55c155e15660 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f76419fd740 10 client.20445 fetch_fsmap finished waiting for FSMap version 107 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f76419fd740 10 client.20445 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f76419fd740 20 client.20445 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f76419fd740 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c155dfcea0 0x55c156201610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f76419fd740 4 client.20445 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.266+0000 7f76419fd740 1 -- 172.21.15.16:0/2857787018 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c15615c9c0 con 0x55c155dfcea0 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.267+0000 7f7638a23640 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c155dfcea0 0x55c156201610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.267+0000 7f761cff9640 10 client.20445 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.267+0000 7f7638a23640 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c155dfcea0 0x55c156201610 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9864b005935bb23 server_cookie=8a254ed90625fa9 in_seq=0 out_seq=0 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f761cff9640 1 -- 172.21.15.16:0/2857787018 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c15615c9c0 con 0x55c155dfcea0 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f761cff9640 10 client.20445 handle_command_reply: tid=0 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 1 client.20445 shutdown 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 1 -- 172.21.15.16:0/2857787018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e15660 msgr2=0x55c1561ef800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e15660 0x55c1561ef800 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f762c003b90 tx=0x7f762c003bf0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 1 -- 172.21.15.16:0/2857787018 shutdown_connections 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e3e520 0x55c1561fb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c155dfcea0 0x55c156201610 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e1d840 0x55c1561efd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f761cff9640 0 client.20445 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 1 --2- 172.21.15.16:0/2857787018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155e15660 0x55c1561ef800 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 1 -- 172.21.15.16:0/2857787018 >> 172.21.15.16:0/2857787018 conn(0x55c155e32910 msgr2=0x55c1561e6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 1 -- 172.21.15.16:0/2857787018 shutdown_connections 2024-08-28T19:42:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 1 -- 172.21.15.16:0/2857787018 wait complete. 2024-08-28T19:42:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.268+0000 7f76419fd740 20 client.20445 trim_cache size 0 max 16384 2024-08-28T19:42:58.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f76419fd740 1 Processor -- start 2024-08-28T19:42:58.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f76419fd740 1 -- start start 2024-08-28T19:42:58.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f76419fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561e5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f76419fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561e0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f76419fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e1d840 0x55c1561fb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f76419fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b2da0 con 0x55c155dfcea0 2024-08-28T19:42:58.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f76419fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b4a30 con 0x55c155e1d840 2024-08-28T19:42:58.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f76419fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b3470 con 0x55c155e15660 2024-08-28T19:42:58.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f7639a25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561e5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f7639a25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561e5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41442/0 (socket says 172.21.15.16:41442) 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f7639a25640 1 -- 172.21.15.16:0/3126372507 learned_addr learned my addr 172.21.15.16:0/3126372507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.286+0000 7f7639a25640 1 -- 172.21.15.16:0/3126372507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 msgr2=0x55c1561e0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.287+0000 7f7638a23640 1 --2- 172.21.15.16:0/3126372507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561e0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.287+0000 7f7639a25640 1 --2- 172.21.15.16:0/3126372507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561e0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.287+0000 7f7639a25640 1 -- 172.21.15.16:0/3126372507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e1d840 msgr2=0x55c1561fb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.287+0000 7f7639a25640 1 --2- 172.21.15.16:0/3126372507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e1d840 0x55c1561fb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.287+0000 7f7639a25640 1 -- 172.21.15.16:0/3126372507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1561ea280 con 0x55c155dfcea0 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.287+0000 7f7638a23640 1 --2- 172.21.15.16:0/3126372507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561e0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.287+0000 7f7639a25640 1 --2- 172.21.15.16:0/3126372507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561e5750 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f762c02ade0 tx=0x7f762c031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566b7a717abad88d server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.287+0000 7f761effd640 1 -- 172.21.15.16:0/3126372507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c058070 con 0x55c155dfcea0 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.287+0000 7f761effd640 1 -- 172.21.15.16:0/3126372507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f762c036030 con 0x55c155dfcea0 2024-08-28T19:42:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.287+0000 7f761effd640 1 -- 172.21.15.16:0/3126372507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c030cc0 con 0x55c155dfcea0 2024-08-28T19:42:58.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.288+0000 7f76419fd740 1 -- 172.21.15.16:0/3126372507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 msgr2=0x55c1561e5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.288+0000 7f76419fd740 1 --2- 172.21.15.16:0/3126372507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561e5750 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f762c02ade0 tx=0x7f762c031e20 comp rx=0 tx=0).stop 2024-08-28T19:42:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.288+0000 7f76419fd740 1 -- 172.21.15.16:0/3126372507 shutdown_connections 2024-08-28T19:42:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.288+0000 7f76419fd740 1 --2- 172.21.15.16:0/3126372507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e1d840 0x55c1561fb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.288+0000 7f76419fd740 1 --2- 172.21.15.16:0/3126372507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561e0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.288+0000 7f76419fd740 1 --2- 172.21.15.16:0/3126372507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561e5750 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.288+0000 7f76419fd740 1 -- 172.21.15.16:0/3126372507 >> 172.21.15.16:0/3126372507 conn(0x55c155e32910 msgr2=0x55c1561e1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.288+0000 7f76419fd740 1 -- 172.21.15.16:0/3126372507 shutdown_connections 2024-08-28T19:42:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.288+0000 7f76419fd740 1 -- 172.21.15.16:0/3126372507 wait complete. 2024-08-28T19:42:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.288+0000 7f76419fd740 1 -- start start 2024-08-28T19:42:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.288+0000 7f76419fd740 1 Processor -- start 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f76419fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561f7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f76419fd740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561f2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f76419fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e1d840 0x55c1561f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f76419fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b42a0 con 0x55c155dfcea0 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f76419fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b3bd0 con 0x55c155e1d840 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f76419fd740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b0da0 con 0x55c155e15660 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f76419fd740 10 client.0 _ll_register_callbacks cb 0x7f7624009630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7639a25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561f7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7639a25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561f7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41448/0 (socket says 172.21.15.16:41448) 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7639a25640 1 -- 172.21.15.16:0/877432179 learned_addr learned my addr 172.21.15.16:0/877432179 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7638a23640 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7639a25640 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e1d840 0x55c1561f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7639a25640 1 -- 172.21.15.16:0/877432179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 msgr2=0x55c1561f2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7639a25640 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561f2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7639a25640 1 -- 172.21.15.16:0/877432179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e1d840 msgr2=0x55c1561f2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7639a25640 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e1d840 0x55c1561f2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7639a25640 1 -- 172.21.15.16:0/877432179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1561f31c0 con 0x55c155dfcea0 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7638a23640 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f7639a25640 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561f7d60 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f762c009bf0 tx=0x7f762c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddf28c4545eb698 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.289+0000 7f761cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f761cff9640 1 -- 172.21.15.16:0/877432179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c06a060 con 0x55c155dfcea0 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f761cff9640 1 -- 172.21.15.16:0/877432179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f762c030a60 con 0x55c155dfcea0 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f76419fd740 1 -- 172.21.15.16:0/877432179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f762c036030 con 0x55c155dfcea0 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f761cff9640 1 -- 172.21.15.16:0/877432179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f762c058040 con 0x55c155dfcea0 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f76419fd740 10 client.20451 fetch_fsmap learned FSMap version 107 2024-08-28T19:42:58.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f76419fd740 1 -- 172.21.15.16:0/877432179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55c15626cd60 con 0x55c155dfcea0 2024-08-28T19:42:58.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f761cff9640 1 -- 172.21.15.16:0/877432179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f762c036030 con 0x55c155dfcea0 2024-08-28T19:42:58.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f761cff9640 1 -- 172.21.15.16:0/877432179 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 107) ==== 8398+0+0 (secure 0 0 0) 0x7f762c04d070 con 0x55c155dfcea0 2024-08-28T19:42:58.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f76419fd740 10 client.20451 fetch_fsmap finished waiting for FSMap version 107 2024-08-28T19:42:58.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f76419fd740 10 client.20451 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:42:58.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f76419fd740 20 client.20451 populate_metadata read hostname 'smithi016' 2024-08-28T19:42:58.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f76419fd740 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c155e3e520 0x55c15626f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:42:58.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f76419fd740 4 client.20451 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3685c114", "release": true, "await": true}] 2024-08-28T19:42:58.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.290+0000 7f76419fd740 1 -- 172.21.15.16:0/877432179 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3685c114", "release": true, "await": true}) -- 0x55c156158460 con 0x55c155e3e520 2024-08-28T19:42:58.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.291+0000 7f7638a23640 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c155e3e520 0x55c15626f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:42:58.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.291+0000 7f761cff9640 10 client.20451 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:58.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.291+0000 7f7638a23640 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c155e3e520 0x55c15626f320 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9a30d9459185a28 server_cookie=1f52092d88699314 in_seq=0 out_seq=0 2024-08-28T19:42:58.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.380+0000 7f761cff9640 1 -- 172.21.15.16:0/877432179 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+592 (crc 0 0 0) 0x55c156158460 con 0x55c155e3e520 2024-08-28T19:42:58.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.380+0000 7f761cff9640 10 client.20451 handle_command_reply: tid=0 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.380+0000 7f76419fd740 1 client.20451 shutdown 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f76419fd740 1 -- 172.21.15.16:0/877432179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 msgr2=0x55c1561f7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f76419fd740 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561f7d60 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f762c009bf0 tx=0x7f762c005fd0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f76419fd740 1 -- 172.21.15.16:0/877432179 shutdown_connections 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f76419fd740 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c155e1d840 0x55c1561f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f76419fd740 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55c155e3e520 0x55c15626f320 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f76419fd740 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c155e15660 0x55c1561f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f76419fd740 1 --2- 172.21.15.16:0/877432179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c155dfcea0 0x55c1561f7d60 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f761cff9640 0 client.20451 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f76419fd740 1 -- 172.21.15.16:0/877432179 >> 172.21.15.16:0/877432179 conn(0x55c155e32910 msgr2=0x55c1561e1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f76419fd740 1 -- 172.21.15.16:0/877432179 shutdown_connections 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f76419fd740 1 -- 172.21.15.16:0/877432179 wait complete. 2024-08-28T19:42:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.381+0000 7f76419fd740 20 client.20451 trim_cache size 0 max 16384 2024-08-28T19:42:58.384 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:42:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 107, 2024-08-28T19:42:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:42:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 72, 2024-08-28T19:42:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-08-28T19:42:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "3685c114": { 2024-08-28T19:42:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 72, 2024-08-28T19:42:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-08-28T19:42:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 154.3, 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:42:58.386 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:42:58.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 -- 172.21.15.16:0/1377950891 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76200b4d20 msgr2=0x7f76200b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76200b4d20 0x7f76200b7230 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f76300e3080 tx=0x7f7630009340 comp rx=0 tx=0).stop 2024-08-28T19:42:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 -- 172.21.15.16:0/1377950891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634103900 msgr2=0x7f76340754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:42:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634103900 0x7f76340754b0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f763000e9f0 tx=0x7f763000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 -- 172.21.15.16:0/1377950891 shutdown_connections 2024-08-28T19:42:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f76200b4d20 0x7f76200b7230 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7634109b90 0x7f7634075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7634109240 0x7f76340759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 --2- 172.21.15.16:0/1377950891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634103900 0x7f76340754b0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:42:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 -- 172.21.15.16:0/1377950891 >> 172.21.15.16:0/1377950891 conn(0x7f763407b5f0 msgr2=0x7f7634107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:42:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 -- 172.21.15.16:0/1377950891 shutdown_connections 2024-08-28T19:42:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:42:58.382+0000 7f763b488640 1 -- 172.21.15.16:0/1377950891 wait complete. 2024-08-28T19:42:58.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191664 20.99 MB/sec execute 394 sec latency 31063.397 ms 2024-08-28T19:42:59.295 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '3685c114', total seconds elapsed: 38.5 2024-08-28T19:42:59.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 192278 21.03 MB/sec execute 395 sec latency 74.405 ms 2024-08-28T19:42:59.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:42:59 smithi016 ceph-mon[18556]: pgmap v1027: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 13 op/s 2024-08-28T19:42:59.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:42:59 smithi028 ceph-mon[21769]: pgmap v1027: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 13 op/s 2024-08-28T19:42:59.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:42:59 smithi115 ceph-mon[22184]: pgmap v1027: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 13 op/s 2024-08-28T19:43:00.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 192571 21.01 MB/sec execute 396 sec latency 486.934 ms 2024-08-28T19:43:01.498 INFO:tasks.workunit.client.0.smithi016.stdout: 10 192571 20.95 MB/sec execute 397 sec latency 1486.995 ms 2024-08-28T19:43:01.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:01 smithi016 ceph-mon[18556]: pgmap v1028: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 66 KiB/s wr, 13 op/s 2024-08-28T19:43:01.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:01 smithi028 ceph-mon[21769]: pgmap v1028: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 66 KiB/s wr, 13 op/s 2024-08-28T19:43:01.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:01 smithi115 ceph-mon[22184]: pgmap v1028: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 66 KiB/s wr, 13 op/s 2024-08-28T19:43:01.970 INFO:tasks.workunit.client.0.smithi016.stdout:[229106] write failed on handle 13621 (Resource temporarily unavailable) 2024-08-28T19:43:01.970 INFO:tasks.workunit.client.0.smithi016.stdout:Child failed with status 1 2024-08-28T19:43:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T19:43:01.988 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-08-28T19:43:01.989 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-28T19:43:02.300 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d19e6118f7015f0047264006cb0c873528eb3ae6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-08-28T19:43:02.489 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=149d6c275a9d49ca9f865df730ba147a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cf5021f85c4b0bf435c74a1183036b3d19af44b5/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d19e6118f7015f0047264006cb0c873528eb3ae6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-08-28T19:43:02.492 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-08-28T19:43:02.503 INFO:tasks.quiescer:joining Quiescers 2024-08-28T19:43:02.503 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-08-28T19:43:02.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:43:02.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:43:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:03.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.297+0000 7f9c6dc91640 1 -- 172.21.15.16:0/3472909299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c68108ad0 msgr2=0x7f9c6810f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:03.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.297+0000 7f9c6dc91640 1 --2- 172.21.15.16:0/3472909299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c68108ad0 0x7f9c6810f300 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009ca0 tx=0x7f9c6002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.297+0000 7f9c6dc91640 1 -- 172.21.15.16:0/3472909299 shutdown_connections 2024-08-28T19:43:03.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.297+0000 7f9c6dc91640 1 --2- 172.21.15.16:0/3472909299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c68108ad0 0x7f9c6810f300 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.297+0000 7f9c6dc91640 1 --2- 172.21.15.16:0/3472909299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68108180 0x7f9c68108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.297+0000 7f9c6dc91640 1 --2- 172.21.15.16:0/3472909299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c68102840 0x7f9c68102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.297+0000 7f9c6dc91640 1 -- 172.21.15.16:0/3472909299 >> 172.21.15.16:0/3472909299 conn(0x7f9c680794f0 msgr2=0x7f9c68079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:03.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.297+0000 7f9c6dc91640 1 -- 172.21.15.16:0/3472909299 shutdown_connections 2024-08-28T19:43:03.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.297+0000 7f9c6dc91640 1 -- 172.21.15.16:0/3472909299 wait complete. 2024-08-28T19:43:03.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.298+0000 7f9c6dc91640 1 Processor -- start 2024-08-28T19:43:03.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.298+0000 7f9c6dc91640 1 -- start start 2024-08-28T19:43:03.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.298+0000 7f9c6dc91640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c68102840 0x7f9c6819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:03.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.298+0000 7f9c6dc91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c68108180 0x7f9c6819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:03.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.298+0000 7f9c6dc91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68108ad0 0x7f9c6819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:03.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.298+0000 7f9c6dc91640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9c681138b0 con 0x7f9c68108180 2024-08-28T19:43:03.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.298+0000 7f9c6dc91640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c68113730 con 0x7f9c68108ad0 2024-08-28T19:43:03.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.298+0000 7f9c6dc91640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9c68113a30 con 0x7f9c68102840 2024-08-28T19:43:03.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.298+0000 7f9c66ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c68108180 0x7f9c6819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:03.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.298+0000 7f9c66ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c68108180 0x7f9c6819a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41460/0 (socket says 172.21.15.16:41460) 2024-08-28T19:43:03.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.298+0000 7f9c66ffd640 1 -- 172.21.15.16:0/1054290252 learned_addr learned my addr 172.21.15.16:0/1054290252 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:03.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c67fff640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68108ad0 0x7f9c6819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:03.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c677fe640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c68102840 0x7f9c6819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:03.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c66ffd640 1 -- 172.21.15.16:0/1054290252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c68102840 msgr2=0x7f9c6819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:03.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c66ffd640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c68102840 0x7f9c6819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c66ffd640 1 -- 172.21.15.16:0/1054290252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68108ad0 msgr2=0x7f9c6819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:03.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c66ffd640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68108ad0 0x7f9c6819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c66ffd640 1 -- 172.21.15.16:0/1054290252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c681a0110 con 0x7f9c68108180 2024-08-28T19:43:03.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c67fff640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68108ad0 0x7f9c6819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:03.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c66ffd640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c68108180 0x7f9c6819a800 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00e9f0 tx=0x7f9c5c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:03.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c64ff9640 1 -- 172.21.15.16:0/1054290252 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c00ccc0 con 0x7f9c68108180 2024-08-28T19:43:03.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c64ff9640 1 -- 172.21.15.16:0/1054290252 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c5c0108e0 con 0x7f9c68108180 2024-08-28T19:43:03.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c6dc91640 1 -- 172.21.15.16:0/1054290252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c681a0400 con 0x7f9c68108180 2024-08-28T19:43:03.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c64ff9640 1 -- 172.21.15.16:0/1054290252 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c018860 con 0x7f9c68108180 2024-08-28T19:43:03.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.299+0000 7f9c6dc91640 1 -- 172.21.15.16:0/1054290252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c681a6c20 con 0x7f9c68108180 2024-08-28T19:43:03.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.300+0000 7f9c6dc91640 1 -- 172.21.15.16:0/1054290252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c68102c50 con 0x7f9c68108180 2024-08-28T19:43:03.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.303+0000 7f9c64ff9640 1 -- 172.21.15.16:0/1054290252 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f9c5c010430 con 0x7f9c68108180 2024-08-28T19:43:03.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.303+0000 7f9c64ff9640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9c500b4cd0 0x7f9c500b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:03.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.303+0000 7f9c677fe640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9c500b4cd0 0x7f9c500b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:03.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.304+0000 7f9c64ff9640 1 -- 172.21.15.16:0/1054290252 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9490+0+0 (secure 0 0 0) 0x7f9c5c0e6130 con 0x7f9c68108180 2024-08-28T19:43:03.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.304+0000 7f9c677fe640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9c500b4cd0 0x7f9c500b71e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9c58005e10 tx=0x7f9c58005da0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:03.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.307+0000 7f9c64ff9640 1 -- 172.21.15.16:0/1054290252 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f9c5c0acb50 con 0x7f9c68108180 2024-08-28T19:43:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:03.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.598+0000 7f9c6dc91640 1 -- 172.21.15.16:0/1054290252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c6810a8c0 con 0x7f9c68108180 2024-08-28T19:43:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.600+0000 7f9c64ff9640 1 -- 172.21.15.16:0/1054290252 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+11991 (secure 0 0 0) 0x7f9c5c0b5090 con 0x7f9c68108180 2024-08-28T19:43:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:43:03.601 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":107,"btime":"2024-08-28T19:42:32:489400+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":107,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:42:32.040691+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:43:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 107 2024-08-28T19:43:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.604+0000 7f9c6dc91640 1 -- 172.21.15.16:0/1054290252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9c500b4cd0 msgr2=0x7f9c500b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.604+0000 7f9c6dc91640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9c500b4cd0 0x7f9c500b71e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9c58005e10 tx=0x7f9c58005da0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.604+0000 7f9c6dc91640 1 -- 172.21.15.16:0/1054290252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c68108180 msgr2=0x7f9c6819a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.604+0000 7f9c6dc91640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c68108180 0x7f9c6819a800 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00e9f0 tx=0x7f9c5c00eeb0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.605+0000 7f9c6dc91640 1 -- 172.21.15.16:0/1054290252 shutdown_connections 2024-08-28T19:43:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.605+0000 7f9c6dc91640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f9c500b4cd0 0x7f9c500b71e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.605+0000 7f9c6dc91640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68108ad0 0x7f9c6819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.605+0000 7f9c6dc91640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c68108180 0x7f9c6819a800 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.605+0000 7f9c6dc91640 1 --2- 172.21.15.16:0/1054290252 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c68102840 0x7f9c6819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.605+0000 7f9c6dc91640 1 -- 172.21.15.16:0/1054290252 >> 172.21.15.16:0/1054290252 conn(0x7f9c680794f0 msgr2=0x7f9c68106210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.606+0000 7f9c6dc91640 1 -- 172.21.15.16:0/1054290252 shutdown_connections 2024-08-28T19:43:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:03.606+0000 7f9c6dc91640 1 -- 172.21.15.16:0/1054290252 wait complete. 2024-08-28T19:43:03.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:03 smithi016 ceph-mon[18556]: pgmap v1029: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.1 MiB/s wr, 83 op/s 2024-08-28T19:43:03.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:03 smithi028 ceph-mon[21769]: pgmap v1029: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.1 MiB/s wr, 83 op/s 2024-08-28T19:43:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:03 smithi115 ceph-mon[22184]: pgmap v1029: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.1 MiB/s wr, 83 op/s 2024-08-28T19:43:04.497 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24665 quiesce db --cancel --all' 2024-08-28T19:43:04.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- timeout 105 ceph tell mds.24665 quiesce db --cancel --all 2024-08-28T19:43:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.706+0000 7fc52988d640 1 -- 172.21.15.16:0/1708774167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc524109260 msgr2=0x7fc524109670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.706+0000 7fc52988d640 1 --2- 172.21.15.16:0/1708774167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc524109260 0x7fc524109670 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fc518009900 tx=0x7fc51802a410 comp rx=0 tx=0).stop 2024-08-28T19:43:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.706+0000 7fc52988d640 1 -- 172.21.15.16:0/1708774167 shutdown_connections 2024-08-28T19:43:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.706+0000 7fc52988d640 1 --2- 172.21.15.16:0/1708774167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc524109bb0 0x7fc524110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.706+0000 7fc52988d640 1 --2- 172.21.15.16:0/1708774167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc524109260 0x7fc524109670 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.706+0000 7fc52988d640 1 --2- 172.21.15.16:0/1708774167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc524101db0 0x7fc5241021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.706+0000 7fc52988d640 1 -- 172.21.15.16:0/1708774167 >> 172.21.15.16:0/1708774167 conn(0x7fc524077a00 msgr2=0x7fc524077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.706+0000 7fc52988d640 1 -- 172.21.15.16:0/1708774167 shutdown_connections 2024-08-28T19:43:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.706+0000 7fc52988d640 1 -- 172.21.15.16:0/1708774167 wait complete. 2024-08-28T19:43:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc52988d640 1 Processor -- start 2024-08-28T19:43:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc52988d640 1 -- start start 2024-08-28T19:43:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc52988d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc524101db0 0x7fc52419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc52988d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc524109260 0x7fc52419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc52988d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc524109bb0 0x7fc52419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc52988d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc524113860 con 0x7fc524101db0 2024-08-28T19:43:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc52988d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241136e0 con 0x7fc524109260 2024-08-28T19:43:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc52988d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241139e0 con 0x7fc524109bb0 2024-08-28T19:43:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc522ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc524101db0 0x7fc52419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc522ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc524101db0 0x7fc52419a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33196/0 (socket says 172.21.15.16:33196) 2024-08-28T19:43:04.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc522ffd640 1 -- 172.21.15.16:0/1509522037 learned_addr learned my addr 172.21.15.16:0/1509522037 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:04.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc5227fc640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc524109260 0x7fc52419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.707+0000 7fc5237fe640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc524109bb0 0x7fc52419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc522ffd640 1 -- 172.21.15.16:0/1509522037 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc524109bb0 msgr2=0x7fc52419fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:04.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc522ffd640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc524109bb0 0x7fc52419fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc522ffd640 1 -- 172.21.15.16:0/1509522037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc524109260 msgr2=0x7fc52419a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:04.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc522ffd640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc524109260 0x7fc52419a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc522ffd640 1 -- 172.21.15.16:0/1509522037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5241a01a0 con 0x7fc524101db0 2024-08-28T19:43:04.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc5227fc640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc524109260 0x7fc52419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:04.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc522ffd640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc524101db0 0x7fc52419a230 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fc51400c890 tx=0x7fc51400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:04.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc52888b640 1 -- 172.21.15.16:0/1509522037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc514004220 con 0x7fc524101db0 2024-08-28T19:43:04.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc52988d640 1 -- 172.21.15.16:0/1509522037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5241a0490 con 0x7fc524101db0 2024-08-28T19:43:04.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc52888b640 1 -- 172.21.15.16:0/1509522037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5140043c0 con 0x7fc524101db0 2024-08-28T19:43:04.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc52888b640 1 -- 172.21.15.16:0/1509522037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc514004a00 con 0x7fc524101db0 2024-08-28T19:43:04.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.708+0000 7fc52988d640 1 -- 172.21.15.16:0/1509522037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5241a6bf0 con 0x7fc524101db0 2024-08-28T19:43:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc52888b640 1 -- 172.21.15.16:0/1509522037 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fc5140079b0 con 0x7fc524101db0 2024-08-28T19:43:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc52fe02740 1 Processor -- start 2024-08-28T19:43:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc52fe02740 1 -- start start 2024-08-28T19:43:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d30022c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d3002800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 0x55a7d3002d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc52fe02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a7d2f79630 con 0x55a7d2c341b0 2024-08-28T19:43:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc52fe02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a7d2f68bc0 con 0x55a7d2c3c390 2024-08-28T19:43:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc52fe02740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a7d2f6cb70 con 0x55a7d2c5d070 2024-08-28T19:43:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc522ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d3002800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc522ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d3002800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33212/0 (socket says 172.21.15.16:33212) 2024-08-28T19:43:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc522ffd640 1 -- 172.21.15.16:0/2502370873 learned_addr learned my addr 172.21.15.16:0/2502370873 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc5237fe640 1 --2- 172.21.15.16:0/2502370873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d30022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.711+0000 7fc5227fc640 1 --2- 172.21.15.16:0/2502370873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 0x55a7d3002d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc52888b640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc50c0b4ca0 0x7fc50c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc5237fe640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc50c0b4ca0 0x7fc50c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc52888b640 1 -- 172.21.15.16:0/1509522037 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9578+0+0 (secure 0 0 0) 0x7fc5140e57a0 con 0x7fc524101db0 2024-08-28T19:43:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc522ffd640 1 -- 172.21.15.16:0/2502370873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 msgr2=0x55a7d30022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc522ffd640 1 --2- 172.21.15.16:0/2502370873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d30022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc522ffd640 1 -- 172.21.15.16:0/2502370873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 msgr2=0x55a7d3002d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc522ffd640 1 --2- 172.21.15.16:0/2502370873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 0x55a7d3002d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc522ffd640 1 -- 172.21.15.16:0/2502370873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7d2cc8650 con 0x55a7d2c341b0 2024-08-28T19:43:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc5227fc640 1 --2- 172.21.15.16:0/2502370873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 0x55a7d3002d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc522ffd640 1 --2- 172.21.15.16:0/2502370873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d3002800 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fc5140e2a40 tx=0x7fc514000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6edff9ec29a3bc31 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc509ffb640 1 -- 172.21.15.16:0/2502370873 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51401d070 con 0x55a7d2c341b0 2024-08-28T19:43:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc5237fe640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc50c0b4ca0 0x7fc50c0b71b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc51c012730 tx=0x7fc51c004ab0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc509ffb640 1 -- 172.21.15.16:0/2502370873 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc514005290 con 0x55a7d2c341b0 2024-08-28T19:43:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.712+0000 7fc509ffb640 1 -- 172.21.15.16:0/2502370873 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5140aba70 con 0x55a7d2c341b0 2024-08-28T19:43:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.713+0000 7fc52fe02740 1 -- 172.21.15.16:0/2502370873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 msgr2=0x55a7d3002800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.713+0000 7fc52fe02740 1 --2- 172.21.15.16:0/2502370873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d3002800 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fc5140e2a40 tx=0x7fc514000bc0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.713+0000 7fc52fe02740 1 -- 172.21.15.16:0/2502370873 shutdown_connections 2024-08-28T19:43:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.713+0000 7fc52fe02740 1 --2- 172.21.15.16:0/2502370873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 0x55a7d3002d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.713+0000 7fc52fe02740 1 --2- 172.21.15.16:0/2502370873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d3002800 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.713+0000 7fc52fe02740 1 --2- 172.21.15.16:0/2502370873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d30022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.713+0000 7fc52fe02740 1 -- 172.21.15.16:0/2502370873 >> 172.21.15.16:0/2502370873 conn(0x55a7d2c51460 msgr2=0x55a7d30001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.713+0000 7fc52fe02740 1 -- 172.21.15.16:0/2502370873 shutdown_connections 2024-08-28T19:43:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.714+0000 7fc52fe02740 1 -- 172.21.15.16:0/2502370873 wait complete. 2024-08-28T19:43:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.714+0000 7fc52fe02740 1 -- start start 2024-08-28T19:43:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.714+0000 7fc52fe02740 1 Processor -- start 2024-08-28T19:43:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.714+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 0x55a7d300daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.714+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 0x55a7d300dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d3019d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc52fe02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a7d2dcb700 con 0x55a7d2c3c390 2024-08-28T19:43:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc52fe02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a7d2eef700 con 0x55a7d2c341b0 2024-08-28T19:43:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc52fe02740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a7d3010a40 con 0x55a7d2c5d070 2024-08-28T19:43:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc52fe02740 10 client.0 _ll_register_callbacks cb 0x55a7d2f1edc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc5227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 0x55a7d300dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc5227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 0x55a7d300dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33222/0 (socket says 172.21.15.16:33222) 2024-08-28T19:43:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc5227fc640 1 -- 172.21.15.16:0/977951864 learned_addr learned my addr 172.21.15.16:0/977951864 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc522ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 0x55a7d300daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc5237fe640 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d3019d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc522ffd640 1 -- 172.21.15.16:0/977951864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 msgr2=0x55a7d3019d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc522ffd640 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d3019d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc522ffd640 1 -- 172.21.15.16:0/977951864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 msgr2=0x55a7d300dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc522ffd640 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 0x55a7d300dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc522ffd640 1 -- 172.21.15.16:0/977951864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7d2f288b0 con 0x55a7d2c341b0 2024-08-28T19:43:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc5227fc640 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 0x55a7d300dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc522ffd640 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 0x55a7d300daa0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc514004770 tx=0x7fc5140ab580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63ae49ff8040d0d9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.715+0000 7fc502ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:43:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc502ffd640 1 -- 172.21.15.16:0/977951864 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51401d030 con 0x55a7d2c341b0 2024-08-28T19:43:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc502ffd640 1 -- 172.21.15.16:0/977951864 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9578+0+0 (secure 0 0 0) 0x7fc5140a8060 con 0x55a7d2c341b0 2024-08-28T19:43:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc52fe02740 1 -- 172.21.15.16:0/977951864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a7d2f7b580 con 0x55a7d2c341b0 2024-08-28T19:43:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc502ffd640 1 -- 172.21.15.16:0/977951864 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fc5140ab1e0 con 0x55a7d2c341b0 2024-08-28T19:43:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc52fe02740 10 client.29924 fetch_fsmap learned FSMap version 107 2024-08-28T19:43:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc52fe02740 1 -- 172.21.15.16:0/977951864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55a7d301c850 con 0x55a7d2c341b0 2024-08-28T19:43:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc502ffd640 1 -- 172.21.15.16:0/977951864 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a7d2f7b580 con 0x55a7d2c341b0 2024-08-28T19:43:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc502ffd640 1 -- 172.21.15.16:0/977951864 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 107) ==== 8398+0+0 (secure 0 0 0) 0x7fc5140a7d00 con 0x55a7d2c341b0 2024-08-28T19:43:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc52fe02740 10 client.29924 fetch_fsmap finished waiting for FSMap version 107 2024-08-28T19:43:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc52fe02740 10 client.29924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:43:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc52fe02740 20 client.29924 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc52fe02740 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a7d2c1b9f0 0x55a7d301fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc52fe02740 4 client.29924 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:43:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.716+0000 7fc52fe02740 1 -- 172.21.15.16:0/977951864 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7d2f77020 con 0x55a7d2c1b9f0 2024-08-28T19:43:04.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:04 smithi016 ceph-mon[18556]: osdmap e115: 12 total, 12 up, 12 in 2024-08-28T19:43:04.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:04 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1054290252' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:43:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.717+0000 7fc5227fc640 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a7d2c1b9f0 0x55a7d301fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.717+0000 7fc5227fc640 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a7d2c1b9f0 0x55a7d301fbb0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e593063c98b4809c server_cookie=c29cfedb97cf5610 in_seq=0 out_seq=0 2024-08-28T19:43:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.717+0000 7fc502ffd640 10 client.29924 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc502ffd640 1 -- 172.21.15.16:0/977951864 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a7d2f77020 con 0x55a7d2c1b9f0 2024-08-28T19:43:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc502ffd640 10 client.29924 handle_command_reply: tid=0 2024-08-28T19:43:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc52fe02740 1 client.29924 shutdown 2024-08-28T19:43:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc52fe02740 1 -- 172.21.15.16:0/977951864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 msgr2=0x55a7d300daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc52fe02740 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 0x55a7d300daa0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc514004770 tx=0x7fc5140ab580 comp rx=0 tx=0).stop 2024-08-28T19:43:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc52fe02740 1 -- 172.21.15.16:0/977951864 shutdown_connections 2024-08-28T19:43:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc52fe02740 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a7d2c1b9f0 0x55a7d301fbb0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc52fe02740 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d3019d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc52fe02740 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 0x55a7d300dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc52fe02740 1 --2- 172.21.15.16:0/977951864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 0x55a7d300daa0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc52fe02740 1 -- 172.21.15.16:0/977951864 >> 172.21.15.16:0/977951864 conn(0x55a7d2c51460 msgr2=0x55a7d3005450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc502ffd640 0 client.29924 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.718+0000 7fc52fe02740 1 -- 172.21.15.16:0/977951864 shutdown_connections 2024-08-28T19:43:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.719+0000 7fc52fe02740 1 -- 172.21.15.16:0/977951864 wait complete. 2024-08-28T19:43:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.719+0000 7fc52fe02740 20 client.29924 trim_cache size 0 max 16384 2024-08-28T19:43:04.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.737+0000 7fc52fe02740 1 Processor -- start 2024-08-28T19:43:04.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.737+0000 7fc52fe02740 1 -- start start 2024-08-28T19:43:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d300e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 0x55a7d3019cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d2fffba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc52fe02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241aec70 con 0x55a7d2c341b0 2024-08-28T19:43:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc52fe02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241b0900 con 0x55a7d2c3c390 2024-08-28T19:43:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc52fe02740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241af340 con 0x55a7d2c5d070 2024-08-28T19:43:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc522ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d300e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc522ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d300e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33226/0 (socket says 172.21.15.16:33226) 2024-08-28T19:43:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc522ffd640 1 -- 172.21.15.16:0/3673225273 learned_addr learned my addr 172.21.15.16:0/3673225273 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc5237fe640 1 --2- 172.21.15.16:0/3673225273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d2fffba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc5227fc640 1 --2- 172.21.15.16:0/3673225273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 0x55a7d3019cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc5237fe640 1 -- 172.21.15.16:0/3673225273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 msgr2=0x55a7d3019cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc5237fe640 1 --2- 172.21.15.16:0/3673225273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 0x55a7d3019cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc5237fe640 1 -- 172.21.15.16:0/3673225273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 msgr2=0x55a7d300e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc5237fe640 1 --2- 172.21.15.16:0/3673225273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d300e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc5237fe640 1 -- 172.21.15.16:0/3673225273 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7d300ca00 con 0x55a7d2c5d070 2024-08-28T19:43:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc522ffd640 1 --2- 172.21.15.16:0/3673225273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d300e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc5227fc640 1 --2- 172.21.15.16:0/3673225273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 0x55a7d3019cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.738+0000 7fc5237fe640 1 --2- 172.21.15.16:0/3673225273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d2fffba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc51c008fd0 tx=0x7fc51c0086f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1393a07fb3ce3abc server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.739+0000 7fc5097fa640 1 -- 172.21.15.16:0/3673225273 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c018070 con 0x55a7d2c5d070 2024-08-28T19:43:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.739+0000 7fc5097fa640 1 -- 172.21.15.16:0/3673225273 <== mon.1 v2:172.21.15.115:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc51c003e20 con 0x55a7d2c5d070 2024-08-28T19:43:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.739+0000 7fc5097fa640 1 -- 172.21.15.16:0/3673225273 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c01fda0 con 0x55a7d2c5d070 2024-08-28T19:43:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.739+0000 7fc52fe02740 1 -- 172.21.15.16:0/3673225273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 msgr2=0x55a7d2fffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.739+0000 7fc52fe02740 1 --2- 172.21.15.16:0/3673225273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d2fffba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc51c008fd0 tx=0x7fc51c0086f0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.740+0000 7fc52fe02740 1 -- 172.21.15.16:0/3673225273 shutdown_connections 2024-08-28T19:43:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.740+0000 7fc52fe02740 1 --2- 172.21.15.16:0/3673225273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d2fffba0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.740+0000 7fc52fe02740 1 --2- 172.21.15.16:0/3673225273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c3c390 0x55a7d3019cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.740+0000 7fc52fe02740 1 --2- 172.21.15.16:0/3673225273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c341b0 0x55a7d300e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.740+0000 7fc52fe02740 1 -- 172.21.15.16:0/3673225273 >> 172.21.15.16:0/3673225273 conn(0x55a7d2c51460 msgr2=0x55a7d300da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.740+0000 7fc52fe02740 1 -- 172.21.15.16:0/3673225273 shutdown_connections 2024-08-28T19:43:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.740+0000 7fc52fe02740 1 -- 172.21.15.16:0/3673225273 wait complete. 2024-08-28T19:43:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.740+0000 7fc52fe02740 1 -- start start 2024-08-28T19:43:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.740+0000 7fc52fe02740 1 Processor -- start 2024-08-28T19:43:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 0x55a7d3023290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 0x55a7d30207b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc52fe02740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d3020cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc52fe02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241b0170 con 0x55a7d2c3c390 2024-08-28T19:43:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc52fe02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241afaa0 con 0x55a7d2c341b0 2024-08-28T19:43:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc52fe02740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241acc50 con 0x55a7d2c5d070 2024-08-28T19:43:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc522ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 0x55a7d3023290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc522ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 0x55a7d3023290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:43840/0 (socket says 172.21.15.16:43840) 2024-08-28T19:43:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc522ffd640 1 -- 172.21.15.16:0/738843187 learned_addr learned my addr 172.21.15.16:0/738843187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc5227fc640 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 0x55a7d30207b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc52fe02740 10 client.0 _ll_register_callbacks cb 0x7fc518009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc5237fe640 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d3020cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc5237fe640 1 -- 172.21.15.16:0/738843187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 msgr2=0x55a7d3023290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc5237fe640 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 0x55a7d3023290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.741+0000 7fc5237fe640 1 -- 172.21.15.16:0/738843187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 msgr2=0x55a7d30207b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.742+0000 7fc5237fe640 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 0x55a7d30207b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.742+0000 7fc5237fe640 1 -- 172.21.15.16:0/738843187 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7d3021360 con 0x55a7d2c5d070 2024-08-28T19:43:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.742+0000 7fc5227fc640 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 0x55a7d30207b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.742+0000 7fc5237fe640 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d3020cf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc51c008fd0 tx=0x7fc51c0125d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=974f28dcbecd4da2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.742+0000 7fc502ffd640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-08-28T19:43:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.742+0000 7fc502ffd640 1 -- 172.21.15.16:0/738843187 <== mon.1 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c018040 con 0x55a7d2c5d070 2024-08-28T19:43:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.742+0000 7fc52fe02740 1 -- 172.21.15.16:0/738843187 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc51c003e20 con 0x55a7d2c5d070 2024-08-28T19:43:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.742+0000 7fc502ffd640 1 -- 172.21.15.16:0/738843187 <== mon.1 v2:172.21.15.115:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9578+0+0 (secure 0 0 0) 0x7fc51c01fcf0 con 0x55a7d2c5d070 2024-08-28T19:43:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.742+0000 7fc502ffd640 1 -- 172.21.15.16:0/738843187 <== mon.1 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fc51c0376d0 con 0x55a7d2c5d070 2024-08-28T19:43:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.742+0000 7fc52fe02740 10 client.29482 fetch_fsmap learned FSMap version 107 2024-08-28T19:43:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.742+0000 7fc52fe02740 1 -- 172.21.15.16:0/738843187 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55a7d308a140 con 0x55a7d2c5d070 2024-08-28T19:43:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.743+0000 7fc502ffd640 1 -- 172.21.15.16:0/738843187 <== mon.1 v2:172.21.15.115:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc51c003e20 con 0x55a7d2c5d070 2024-08-28T19:43:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.743+0000 7fc502ffd640 1 -- 172.21.15.16:0/738843187 <== mon.1 v2:172.21.15.115:3300/0 5 ==== fsmap(e 107) ==== 8398+0+0 (secure 0 0 0) 0x7fc51c027030 con 0x55a7d2c5d070 2024-08-28T19:43:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.743+0000 7fc52fe02740 10 client.29482 fetch_fsmap finished waiting for FSMap version 107 2024-08-28T19:43:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.743+0000 7fc52fe02740 10 client.29482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:43:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.743+0000 7fc52fe02740 20 client.29482 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.743+0000 7fc52fe02740 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a7d2bf0da0 0x55a7d308c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.744+0000 7fc52fe02740 4 client.29482 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:43:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.744+0000 7fc52fe02740 1 -- 172.21.15.16:0/738843187 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a7d2f74760 con 0x55a7d2bf0da0 2024-08-28T19:43:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.744+0000 7fc522ffd640 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a7d2bf0da0 0x55a7d308c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.744+0000 7fc522ffd640 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a7d2bf0da0 0x55a7d308c8e0 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5d037e83acf3c75 server_cookie=d061e3a853ce6e72 in_seq=0 out_seq=0 2024-08-28T19:43:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.744+0000 7fc502ffd640 10 client.29482 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.744+0000 7fc502ffd640 1 -- 172.21.15.16:0/738843187 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+8238 (crc 0 0 0) 0x55a7d2f74760 con 0x55a7d2bf0da0 2024-08-28T19:43:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.744+0000 7fc502ffd640 10 client.29482 handle_command_reply: tid=0 2024-08-28T19:43:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.744+0000 7fc52fe02740 1 client.29482 shutdown 2024-08-28T19:43:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc52fe02740 1 -- 172.21.15.16:0/738843187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 msgr2=0x55a7d3020cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc52fe02740 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d3020cf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc51c008fd0 tx=0x7fc51c0125d0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc52fe02740 1 -- 172.21.15.16:0/738843187 shutdown_connections 2024-08-28T19:43:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc52fe02740 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a7d2bf0da0 0x55a7d308c8e0 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc52fe02740 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a7d2c5d070 0x55a7d3020cf0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc502ffd640 0 client.29482 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc52fe02740 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7d2c3c390 0x55a7d30207b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc52fe02740 1 --2- 172.21.15.16:0/738843187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7d2c341b0 0x55a7d3023290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc52fe02740 1 -- 172.21.15.16:0/738843187 >> 172.21.15.16:0/738843187 conn(0x55a7d2c51460 msgr2=0x55a7d30053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc52fe02740 1 -- 172.21.15.16:0/738843187 shutdown_connections 2024-08-28T19:43:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc52fe02740 1 -- 172.21.15.16:0/738843187 wait complete. 2024-08-28T19:43:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.745+0000 7fc52fe02740 20 client.29482 trim_cache size 0 max 16384 2024-08-28T19:43:04.755 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:43:04.755 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-28T19:43:04.755 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-08-28T19:43:04.755 INFO:teuthology.orchestra.run.smithi016.stdout: "414082c7-7ecd-4292-9ec5-4c4504fedaf1": { 2024-08-28T19:43:04.755 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "414082c7-7ecd-4292-9ec5-4c4504fedaf1", 2024-08-28T19:43:04.756 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-08-28T19:43:04.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:04.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.756 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:43:04.756 INFO:teuthology.orchestra.run.smithi016.stdout: "414082c7-7ecd-4292-9ec5-4c4504fedaf1": { 2024-08-28T19:43:04.756 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:42:27.129235+0000", 2024-08-28T19:43:04.756 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.757 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.757 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.757 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.757 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.757 INFO:teuthology.orchestra.run.smithi016.stdout: "39fa9343-222a-493e-bb10-c8863c663453": { 2024-08-28T19:43:04.757 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:41:33.609756+0000", 2024-08-28T19:43:04.757 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.758 INFO:teuthology.orchestra.run.smithi016.stdout: "a466ab70-970a-4b3e-927e-47c65e508132": { 2024-08-28T19:43:04.758 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:40:38.307336+0000", 2024-08-28T19:43:04.758 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.758 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.759 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.759 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.759 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.759 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:43:04.760 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:55.619275+0000", 2024-08-28T19:43:04.761 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.761 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.761 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.761 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.761 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.761 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:43:04.761 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:43:04.762 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.762 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.762 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.762 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.762 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.762 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:43:04.762 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:43:04.763 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.763 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.763 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.763 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.763 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.763 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:43:04.763 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:43:04.764 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.764 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.764 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.764 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.764 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.764 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:43:04.764 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:43:04.764 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.765 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.765 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.765 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.765 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.765 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:43:04.765 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:43:04.765 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.765 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.766 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.766 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.766 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.766 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:43:04.766 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:43:04.766 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.766 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.767 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.767 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.767 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.767 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:43:04.767 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:43:04.767 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.767 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.768 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.768 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.768 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.768 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:43:04.768 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:43:04.768 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.768 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.769 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.769 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.769 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.769 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:43:04.769 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:43:04.769 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.769 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.770 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.770 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.770 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.770 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:43:04.770 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:43:04.770 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.770 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.770 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.771 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.771 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.771 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:43:04.771 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:43:04.771 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.771 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.771 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.772 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.772 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.772 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:43:04.772 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:43:04.772 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.772 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.772 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.773 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.773 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.773 INFO:teuthology.orchestra.run.smithi016.stdout: "8de90bf3-5743-4711-98c2-a56ca0757291": { 2024-08-28T19:43:04.773 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:39:48.098772+0000", 2024-08-28T19:43:04.773 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.773 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.773 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.774 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.774 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.774 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:43:04.774 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:43:04.774 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.774 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.774 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.775 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.775 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.775 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:43:04.775 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:43:04.775 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.775 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.775 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.775 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.776 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.776 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:43:04.776 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:43:04.776 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.776 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.776 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.776 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.777 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.777 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:43:04.777 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:43:04.777 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.777 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.777 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.777 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.778 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.778 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:43:04.778 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:43:04.778 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.778 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.778 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.778 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.778 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.779 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:43:04.779 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:43:04.779 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.779 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.779 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.779 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.779 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.780 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:43:04.780 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:43:04.780 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.780 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.780 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.780 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.780 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.781 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:43:04.781 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:43:04.781 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.781 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.781 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.781 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.781 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.782 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:43:04.782 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:43:04.782 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.782 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.782 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.782 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.782 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.783 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:43:04.783 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:43:04.783 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.783 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.783 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.783 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.783 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:04.784 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:43:04.784 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:43:04.784 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:04.784 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:04.784 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:04.784 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:04.784 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:04.784 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:04.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:04.785 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:43:04.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 -- 172.21.15.16:0/1509522037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc50c0b4ca0 msgr2=0x7fc50c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc50c0b4ca0 0x7fc50c0b71b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc51c012730 tx=0x7fc51c004ab0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 -- 172.21.15.16:0/1509522037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc524101db0 msgr2=0x7fc52419a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc524101db0 0x7fc52419a230 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fc51400c890 tx=0x7fc51400cd50 comp rx=0 tx=0).stop 2024-08-28T19:43:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 -- 172.21.15.16:0/1509522037 shutdown_connections 2024-08-28T19:43:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc524109bb0 0x7fc52419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fc50c0b4ca0 0x7fc50c0b71b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc524109260 0x7fc52419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 --2- 172.21.15.16:0/1509522037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc524101db0 0x7fc52419a230 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 -- 172.21.15.16:0/1509522037 >> 172.21.15.16:0/1509522037 conn(0x7fc524077a00 msgr2=0x7fc5241052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 -- 172.21.15.16:0/1509522037 shutdown_connections 2024-08-28T19:43:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:04.747+0000 7fc52988d640 1 -- 172.21.15.16:0/1509522037 wait complete. 2024-08-28T19:43:04.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:04 smithi028 ceph-mon[21769]: osdmap e115: 12 total, 12 up, 12 in 2024-08-28T19:43:04.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:04 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1054290252' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:43:04.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:04 smithi115 ceph-mon[22184]: osdmap e115: 12 total, 12 up, 12 in 2024-08-28T19:43:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:04 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1054290252' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:43:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:05.460 INFO:tasks.cephfs.filesystem:scrub status for tag:414082c7-7ecd-4292-9ec5-4c4504fedaf1 - {'path': '/', 'tag': '414082c7-7ecd-4292-9ec5-4c4504fedaf1', 'options': 'recursive,force'} 2024-08-28T19:43:05.699 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:05 smithi016 ceph-mon[18556]: pgmap v1031: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1.9 MiB/s wr, 148 op/s 2024-08-28T19:43:05.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:05 smithi028 ceph-mon[21769]: pgmap v1031: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1.9 MiB/s wr, 148 op/s 2024-08-28T19:43:05.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:05 smithi115 ceph-mon[22184]: pgmap v1031: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1.9 MiB/s wr, 148 op/s 2024-08-28T19:43:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.966+0000 7f78190a2640 1 -- 172.21.15.16:0/133616467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7814107920 msgr2=0x7f7814107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.966+0000 7f78190a2640 1 --2- 172.21.15.16:0/133616467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7814107920 0x7f7814107d30 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f7808009900 tx=0x7f780802a410 comp rx=0 tx=0).stop 2024-08-28T19:43:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.966+0000 7f78190a2640 1 -- 172.21.15.16:0/133616467 shutdown_connections 2024-08-28T19:43:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.966+0000 7f78190a2640 1 --2- 172.21.15.16:0/133616467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7814108270 0x7f781410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.966+0000 7f78190a2640 1 --2- 172.21.15.16:0/133616467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7814107920 0x7f7814107d30 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.966+0000 7f78190a2640 1 --2- 172.21.15.16:0/133616467 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7814101fe0 0x7f78141023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.966+0000 7f78190a2640 1 -- 172.21.15.16:0/133616467 >> 172.21.15.16:0/133616467 conn(0x7f7814079d30 msgr2=0x7f781407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.966+0000 7f78190a2640 1 -- 172.21.15.16:0/133616467 shutdown_connections 2024-08-28T19:43:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.966+0000 7f78190a2640 1 -- 172.21.15.16:0/133616467 wait complete. 2024-08-28T19:43:05.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.967+0000 7f78190a2640 1 Processor -- start 2024-08-28T19:43:05.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.967+0000 7f78190a2640 1 -- start start 2024-08-28T19:43:05.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.967+0000 7f78190a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7814101fe0 0x7f78140feb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.967+0000 7f78190a2640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7814107920 0x7f78140ff0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.967+0000 7f78190a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7814108270 0x7f78140ff610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.967+0000 7f78190a2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78141116b0 con 0x7f7814108270 2024-08-28T19:43:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.967+0000 7f78190a2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7814111530 con 0x7f7814101fe0 2024-08-28T19:43:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.967+0000 7f78190a2640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7814111830 con 0x7f7814107920 2024-08-28T19:43:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f781359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7814108270 0x7f78140ff610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f781359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7814108270 0x7f78140ff610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33270/0 (socket says 172.21.15.16:33270) 2024-08-28T19:43:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f781359e640 1 -- 172.21.15.16:0/1998941779 learned_addr learned my addr 172.21.15.16:0/1998941779 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f781259c640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7814107920 0x7f78140ff0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f7812d9d640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7814101fe0 0x7f78140feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f781359e640 1 -- 172.21.15.16:0/1998941779 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7814107920 msgr2=0x7f78140ff0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f781359e640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7814107920 0x7f78140ff0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f781359e640 1 -- 172.21.15.16:0/1998941779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7814101fe0 msgr2=0x7f78140feb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f781359e640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7814101fe0 0x7f78140feb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f781359e640 1 -- 172.21.15.16:0/1998941779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78141a84b0 con 0x7f7814108270 2024-08-28T19:43:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f781359e640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7814108270 0x7f78140ff610 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f780c00cce0 tx=0x7f780c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f77fbfff640 1 -- 172.21.15.16:0/1998941779 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c00f450 con 0x7f7814108270 2024-08-28T19:43:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f77fbfff640 1 -- 172.21.15.16:0/1998941779 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f780c00faa0 con 0x7f7814108270 2024-08-28T19:43:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f78190a2640 1 -- 172.21.15.16:0/1998941779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78141a87a0 con 0x7f7814108270 2024-08-28T19:43:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f77fbfff640 1 -- 172.21.15.16:0/1998941779 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c018b70 con 0x7f7814108270 2024-08-28T19:43:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.968+0000 7f78190a2640 1 -- 172.21.15.16:0/1998941779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78141a8c60 con 0x7f7814108270 2024-08-28T19:43:05.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.971+0000 7f781f617740 1 Processor -- start 2024-08-28T19:43:05.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.971+0000 7f77fbfff640 1 -- 172.21.15.16:0/1998941779 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f780c018d10 con 0x7f7814108270 2024-08-28T19:43:05.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.971+0000 7f781f617740 1 -- start start 2024-08-28T19:43:05.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781f617740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca09b2d0 0x55a6ca440d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781f617740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca072410 0x55a6ca441240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781f617740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca07a5f0 0x55a6ca441780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781f617740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a6ca3b7610 con 0x55a6ca07a5f0 2024-08-28T19:43:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781f617740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a6ca3a6ba0 con 0x55a6ca09b2d0 2024-08-28T19:43:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781f617740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a6ca3aab50 con 0x55a6ca072410 2024-08-28T19:43:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca07a5f0 0x55a6ca441780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca07a5f0 0x55a6ca441780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33276/0 (socket says 172.21.15.16:33276) 2024-08-28T19:43:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781259c640 1 -- 172.21.15.16:0/3821020935 learned_addr learned my addr 172.21.15.16:0/3821020935 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781259c640 1 --2- 172.21.15.16:0/3821020935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca09b2d0 0x55a6ca440d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f7812d9d640 1 --2- 172.21.15.16:0/3821020935 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca072410 0x55a6ca441240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781259c640 1 -- 172.21.15.16:0/3821020935 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca072410 msgr2=0x55a6ca441240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781259c640 1 --2- 172.21.15.16:0/3821020935 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca072410 0x55a6ca441240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781259c640 1 -- 172.21.15.16:0/3821020935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca09b2d0 msgr2=0x55a6ca440d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f77fbfff640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f77fc0b4ca0 0x7f77fc0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781259c640 1 --2- 172.21.15.16:0/3821020935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca09b2d0 0x55a6ca440d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.972+0000 7f781259c640 1 -- 172.21.15.16:0/3821020935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6ca1068b0 con 0x55a6ca07a5f0 2024-08-28T19:43:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.973+0000 7f781359e640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f77fc0b4ca0 0x7f77fc0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.973+0000 7f781259c640 1 --2- 172.21.15.16:0/3821020935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca07a5f0 0x55a6ca441780 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f78080057c0 tx=0x7f7808000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b255d92efaf698b4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.973+0000 7f77fbfff640 1 -- 172.21.15.16:0/1998941779 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9578+0+0 (secure 0 0 0) 0x7f780c0e5d30 con 0x7f7814108270 2024-08-28T19:43:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.973+0000 7f77f97fa640 1 -- 172.21.15.16:0/3821020935 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7808035070 con 0x55a6ca07a5f0 2024-08-28T19:43:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.973+0000 7f77f97fa640 1 -- 172.21.15.16:0/3821020935 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7808005030 con 0x55a6ca07a5f0 2024-08-28T19:43:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.973+0000 7f77f97fa640 1 -- 172.21.15.16:0/3821020935 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78080044e0 con 0x55a6ca07a5f0 2024-08-28T19:43:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.974+0000 7f781f617740 1 -- 172.21.15.16:0/3821020935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca07a5f0 msgr2=0x55a6ca441780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.974+0000 7f781f617740 1 --2- 172.21.15.16:0/3821020935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca07a5f0 0x55a6ca441780 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f78080057c0 tx=0x7f7808000bc0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.974+0000 7f781359e640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f77fc0b4ca0 0x7f77fc0b71b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f780c0e3110 tx=0x7f780c00af30 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.974+0000 7f781f617740 1 -- 172.21.15.16:0/3821020935 shutdown_connections 2024-08-28T19:43:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.974+0000 7f781f617740 1 --2- 172.21.15.16:0/3821020935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca07a5f0 0x55a6ca441780 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.974+0000 7f781f617740 1 --2- 172.21.15.16:0/3821020935 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca072410 0x55a6ca441240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.974+0000 7f781f617740 1 --2- 172.21.15.16:0/3821020935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca09b2d0 0x55a6ca440d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.974+0000 7f781f617740 1 -- 172.21.15.16:0/3821020935 >> 172.21.15.16:0/3821020935 conn(0x55a6ca08f6c0 msgr2=0x55a6ca43ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.974+0000 7f781f617740 1 -- 172.21.15.16:0/3821020935 shutdown_connections 2024-08-28T19:43:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.974+0000 7f781f617740 1 -- 172.21.15.16:0/3821020935 wait complete. 2024-08-28T19:43:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.974+0000 7f781f617740 1 -- start start 2024-08-28T19:43:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781f617740 1 Processor -- start 2024-08-28T19:43:05.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781f617740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca44c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781f617740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca07a5f0 0x55a6ca44cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781f617740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca09b2d0 0x55a6ca4587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781f617740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a6ca209810 con 0x55a6ca072410 2024-08-28T19:43:05.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781f617740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a6ca32d960 con 0x55a6ca09b2d0 2024-08-28T19:43:05.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781f617740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a6ca44f480 con 0x55a6ca07a5f0 2024-08-28T19:43:05.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781f617740 10 client.0 _ll_register_callbacks cb 0x55a6ca0b2b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:05.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca44c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:05.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca44c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33284/0 (socket says 172.21.15.16:33284) 2024-08-28T19:43:05.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781259c640 1 -- 172.21.15.16:0/892075151 learned_addr learned my addr 172.21.15.16:0/892075151 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:05.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f7812d9d640 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca07a5f0 0x55a6ca44cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:05.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781259c640 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca09b2d0 0x55a6ca4587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:05.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781259c640 1 -- 172.21.15.16:0/892075151 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca07a5f0 msgr2=0x55a6ca44cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:05.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781259c640 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca07a5f0 0x55a6ca44cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781259c640 1 -- 172.21.15.16:0/892075151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca09b2d0 msgr2=0x55a6ca4587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:05.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781259c640 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca09b2d0 0x55a6ca4587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.975+0000 7f781259c640 1 -- 172.21.15.16:0/892075151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6ca458d10 con 0x55a6ca072410 2024-08-28T19:43:05.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f781259c640 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca44c850 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f780802a450 tx=0x7f7808032d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c045722c1d82da6e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:05.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f77f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:05.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f77f6ff5640 1 -- 172.21.15.16:0/892075151 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7808035030 con 0x55a6ca072410 2024-08-28T19:43:05.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f77f6ff5640 1 -- 172.21.15.16:0/892075151 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9578+0+0 (secure 0 0 0) 0x7f7808004040 con 0x55a6ca072410 2024-08-28T19:43:05.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f781f617740 1 -- 172.21.15.16:0/892075151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7808005030 con 0x55a6ca072410 2024-08-28T19:43:05.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f77f6ff5640 1 -- 172.21.15.16:0/892075151 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f7808059050 con 0x55a6ca072410 2024-08-28T19:43:05.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f781f617740 10 client.20487 fetch_fsmap learned FSMap version 107 2024-08-28T19:43:05.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f781f617740 1 -- 172.21.15.16:0/892075151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55a6ca45b640 con 0x55a6ca072410 2024-08-28T19:43:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f77f6ff5640 1 -- 172.21.15.16:0/892075151 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7808005030 con 0x55a6ca072410 2024-08-28T19:43:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f77f6ff5640 1 -- 172.21.15.16:0/892075151 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 107) ==== 8398+0+0 (secure 0 0 0) 0x7f78080035f0 con 0x55a6ca072410 2024-08-28T19:43:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f781f617740 10 client.20487 fetch_fsmap finished waiting for FSMap version 107 2024-08-28T19:43:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f781f617740 10 client.20487 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:43:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f781f617740 20 client.20487 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f781f617740 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6ca059c50 0x55a6ca45e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f781f617740 4 client.20487 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:43:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.976+0000 7f781f617740 1 -- 172.21.15.16:0/892075151 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6ca3b9560 con 0x55a6ca059c50 2024-08-28T19:43:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.977+0000 7f7812d9d640 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6ca059c50 0x55a6ca45e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.977+0000 7f77f6ff5640 10 client.20487 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.977+0000 7f7812d9d640 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6ca059c50 0x55a6ca45e9a0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e5f61b9769689b4 server_cookie=966d8028843e2b0d in_seq=0 out_seq=0 2024-08-28T19:43:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f77f6ff5640 1 -- 172.21.15.16:0/892075151 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a6ca3b9560 con 0x55a6ca059c50 2024-08-28T19:43:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f77f6ff5640 10 client.20487 handle_command_reply: tid=0 2024-08-28T19:43:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 1 client.20487 shutdown 2024-08-28T19:43:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 1 -- 172.21.15.16:0/892075151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 msgr2=0x55a6ca44c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca44c850 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f780802a450 tx=0x7f7808032d20 comp rx=0 tx=0).stop 2024-08-28T19:43:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 1 -- 172.21.15.16:0/892075151 shutdown_connections 2024-08-28T19:43:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca09b2d0 0x55a6ca4587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6ca059c50 0x55a6ca45e9a0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca07a5f0 0x55a6ca44cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 1 --2- 172.21.15.16:0/892075151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca44c850 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f77f6ff5640 0 client.20487 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 1 -- 172.21.15.16:0/892075151 >> 172.21.15.16:0/892075151 conn(0x55a6ca08f6c0 msgr2=0x55a6ca443e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 1 -- 172.21.15.16:0/892075151 shutdown_connections 2024-08-28T19:43:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 1 -- 172.21.15.16:0/892075151 wait complete. 2024-08-28T19:43:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.978+0000 7f781f617740 20 client.20487 trim_cache size 0 max 16384 2024-08-28T19:43:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.995+0000 7f781f617740 1 Processor -- start 2024-08-28T19:43:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f781f617740 1 -- start start 2024-08-28T19:43:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f781f617740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca059c50 0x55a6ca445d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f781f617740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca4462d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f781f617740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca07a5f0 0x55a6ca449840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f781f617740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78141b2eb0 con 0x55a6ca072410 2024-08-28T19:43:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f781f617740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78141b4b40 con 0x55a6ca059c50 2024-08-28T19:43:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f781f617740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f78141b3580 con 0x55a6ca07a5f0 2024-08-28T19:43:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f7812d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca4462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f7812d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca4462d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33298/0 (socket says 172.21.15.16:33298) 2024-08-28T19:43:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f7812d9d640 1 -- 172.21.15.16:0/933385168 learned_addr learned my addr 172.21.15.16:0/933385168 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f781259c640 1 --2- 172.21.15.16:0/933385168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca059c50 0x55a6ca445d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f7812d9d640 1 -- 172.21.15.16:0/933385168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca07a5f0 msgr2=0x55a6ca449840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f781259c640 1 --2- 172.21.15.16:0/933385168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca07a5f0 0x55a6ca449840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f7812d9d640 1 --2- 172.21.15.16:0/933385168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca07a5f0 0x55a6ca449840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f7812d9d640 1 -- 172.21.15.16:0/933385168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca059c50 msgr2=0x55a6ca445d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f7812d9d640 1 --2- 172.21.15.16:0/933385168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca059c50 0x55a6ca445d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f7812d9d640 1 -- 172.21.15.16:0/933385168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6ca44d910 con 0x55a6ca072410 2024-08-28T19:43:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f781259c640 1 --2- 172.21.15.16:0/933385168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca07a5f0 0x55a6ca449840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:43:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.996+0000 7f7812d9d640 1 --2- 172.21.15.16:0/933385168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca4462d0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f7804004390 tx=0x7f78040039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ca75eeaee1ddcb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.997+0000 7f77f8ff9640 1 -- 172.21.15.16:0/933385168 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780400f740 con 0x55a6ca072410 2024-08-28T19:43:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.997+0000 7f77f8ff9640 1 -- 172.21.15.16:0/933385168 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f780400fd90 con 0x55a6ca072410 2024-08-28T19:43:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.997+0000 7f77f8ff9640 1 -- 172.21.15.16:0/933385168 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780401bdc0 con 0x55a6ca072410 2024-08-28T19:43:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.997+0000 7f781f617740 1 -- 172.21.15.16:0/933385168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 msgr2=0x55a6ca4462d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.997+0000 7f781f617740 1 --2- 172.21.15.16:0/933385168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca4462d0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f7804004390 tx=0x7f78040039c0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.997+0000 7f781f617740 1 -- 172.21.15.16:0/933385168 shutdown_connections 2024-08-28T19:43:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.997+0000 7f781f617740 1 --2- 172.21.15.16:0/933385168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca07a5f0 0x55a6ca449840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.997+0000 7f781f617740 1 --2- 172.21.15.16:0/933385168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca072410 0x55a6ca4462d0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.997+0000 7f781f617740 1 --2- 172.21.15.16:0/933385168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca059c50 0x55a6ca445d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.997+0000 7f781f617740 1 -- 172.21.15.16:0/933385168 >> 172.21.15.16:0/933385168 conn(0x55a6ca08f6c0 msgr2=0x55a6ca447870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781f617740 1 -- 172.21.15.16:0/933385168 shutdown_connections 2024-08-28T19:43:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781f617740 1 -- 172.21.15.16:0/933385168 wait complete. 2024-08-28T19:43:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781f617740 1 -- start start 2024-08-28T19:43:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781f617740 1 Processor -- start 2024-08-28T19:43:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781f617740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca059c50 0x55a6ca4605a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781f617740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca072410 0x55a6ca460b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781f617740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca07a5f0 0x55a6ca4c2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781f617740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78141b43b0 con 0x55a6ca059c50 2024-08-28T19:43:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781f617740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78141b3ce0 con 0x55a6ca07a5f0 2024-08-28T19:43:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781f617740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f78141b0f00 con 0x55a6ca072410 2024-08-28T19:43:06.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781f617740 10 client.0 _ll_register_callbacks cb 0x7f7808032e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:06.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca059c50 0x55a6ca4605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:06.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca059c50 0x55a6ca4605a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33302/0 (socket says 172.21.15.16:33302) 2024-08-28T19:43:06.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.998+0000 7f781259c640 1 -- 172.21.15.16:0/2190928770 learned_addr learned my addr 172.21.15.16:0/2190928770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:06.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f781259c640 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca07a5f0 0x55a6ca4c2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:06.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f7812d9d640 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca072410 0x55a6ca460b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:06.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f781259c640 1 -- 172.21.15.16:0/2190928770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca072410 msgr2=0x55a6ca460b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:06.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f781259c640 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca072410 0x55a6ca460b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f781259c640 1 -- 172.21.15.16:0/2190928770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca07a5f0 msgr2=0x55a6ca4c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f781259c640 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca07a5f0 0x55a6ca4c2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f781259c640 1 -- 172.21.15.16:0/2190928770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6ca4c2990 con 0x55a6ca059c50 2024-08-28T19:43:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f781259c640 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca059c50 0x55a6ca4605a0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f7808009850 tx=0x7f7808032d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cbe6d24a0e68f5e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f77f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f77f6ff5640 1 -- 172.21.15.16:0/2190928770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7808035040 con 0x55a6ca059c50 2024-08-28T19:43:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f77f6ff5640 1 -- 172.21.15.16:0/2190928770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9578+0+0 (secure 0 0 0) 0x7f78080055f0 con 0x55a6ca059c50 2024-08-28T19:43:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f781f617740 1 -- 172.21.15.16:0/2190928770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f780400fd90 con 0x55a6ca059c50 2024-08-28T19:43:06.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f77f6ff5640 1 -- 172.21.15.16:0/2190928770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f780806b050 con 0x55a6ca059c50 2024-08-28T19:43:06.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f781f617740 10 client.20493 fetch_fsmap learned FSMap version 107 2024-08-28T19:43:06.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:05.999+0000 7f781f617740 1 -- 172.21.15.16:0/2190928770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55a6ca4c9cd0 con 0x55a6ca059c50 2024-08-28T19:43:06.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.000+0000 7f77f6ff5640 1 -- 172.21.15.16:0/2190928770 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f780400fd90 con 0x55a6ca059c50 2024-08-28T19:43:06.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.000+0000 7f77f6ff5640 1 -- 172.21.15.16:0/2190928770 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 107) ==== 8398+0+0 (secure 0 0 0) 0x7f78080046c0 con 0x55a6ca059c50 2024-08-28T19:43:06.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.000+0000 7f781f617740 10 client.20493 fetch_fsmap finished waiting for FSMap version 107 2024-08-28T19:43:06.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.000+0000 7f781f617740 10 client.20493 resolve_mds: validated gid 24665 aka daemon mds.b 2024-08-28T19:43:06.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.000+0000 7f781f617740 20 client.20493 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:06.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.000+0000 7f781f617740 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6ca09b2d0 0x55a6ca4cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:06.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.000+0000 7f781f617740 4 client.20493 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-08-28T19:43:06.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.000+0000 7f781f617740 1 -- 172.21.15.16:0/2190928770 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55a6ca3b5000 con 0x55a6ca09b2d0 2024-08-28T19:43:06.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.000+0000 7f7812d9d640 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6ca09b2d0 0x55a6ca4cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:06.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f77f6ff5640 10 client.20493 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:06.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f7812d9d640 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6ca09b2d0 0x55a6ca4cc4c0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7481e5bea61ea451 server_cookie=a60d31183ff23074 in_seq=0 out_seq=0 2024-08-28T19:43:06.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f77f6ff5640 1 -- 172.21.15.16:0/2190928770 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+81 (crc 0 0 0) 0x55a6ca3b5000 con 0x55a6ca09b2d0 2024-08-28T19:43:06.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f77f6ff5640 10 client.20493 handle_command_reply: tid=0 2024-08-28T19:43:06.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f781f617740 1 client.20493 shutdown 2024-08-28T19:43:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f781f617740 1 -- 172.21.15.16:0/2190928770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca059c50 msgr2=0x55a6ca4605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f781f617740 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca059c50 0x55a6ca4605a0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f7808009850 tx=0x7f7808032d20 comp rx=0 tx=0).stop 2024-08-28T19:43:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f781f617740 1 -- 172.21.15.16:0/2190928770 shutdown_connections 2024-08-28T19:43:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f781f617740 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6ca07a5f0 0x55a6ca4c2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f781f617740 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x55a6ca09b2d0 0x55a6ca4cc4c0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f781f617740 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a6ca072410 0x55a6ca460b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f781f617740 1 --2- 172.21.15.16:0/2190928770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6ca059c50 0x55a6ca4605a0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f781f617740 1 -- 172.21.15.16:0/2190928770 >> 172.21.15.16:0/2190928770 conn(0x55a6ca08f6c0 msgr2=0x55a6ca447870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:06.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f77f6ff5640 0 client.20493 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:06.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.001+0000 7f781f617740 1 -- 172.21.15.16:0/2190928770 shutdown_connections 2024-08-28T19:43:06.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f781f617740 1 -- 172.21.15.16:0/2190928770 wait complete. 2024-08-28T19:43:06.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f781f617740 20 client.20493 trim_cache size 0 max 16384 2024-08-28T19:43:06.014 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:43:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 107, 2024-08-28T19:43:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24665, 2024-08-28T19:43:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 72, 2024-08-28T19:43:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": {} 2024-08-28T19:43:06.015 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:43:06.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 -- 172.21.15.16:0/1998941779 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f77fc0b4ca0 msgr2=0x7f77fc0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:06.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f77fc0b4ca0 0x7f77fc0b71b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f780c0e3110 tx=0x7f780c00af30 comp rx=0 tx=0).stop 2024-08-28T19:43:06.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 -- 172.21.15.16:0/1998941779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7814108270 msgr2=0x7f78140ff610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:06.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7814108270 0x7f78140ff610 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f780c00cce0 tx=0x7f780c00a620 comp rx=0 tx=0).stop 2024-08-28T19:43:06.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 -- 172.21.15.16:0/1998941779 shutdown_connections 2024-08-28T19:43:06.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7814108270 0x7f78140ff610 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7814107920 0x7f78140ff0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f77fc0b4ca0 0x7f77fc0b71b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 --2- 172.21.15.16:0/1998941779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7814101fe0 0x7f78140feb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 -- 172.21.15.16:0/1998941779 >> 172.21.15.16:0/1998941779 conn(0x7f7814079d30 msgr2=0x7f7814101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:06.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 -- 172.21.15.16:0/1998941779 shutdown_connections 2024-08-28T19:43:06.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.002+0000 7f78190a2640 1 -- 172.21.15.16:0/1998941779 wait complete. 2024-08-28T19:43:06.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:06 smithi016 ceph-mon[18556]: pgmap v1032: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 2.0 MiB/s wr, 162 op/s 2024-08-28T19:43:06.704 INFO:tasks.quiescer:done joining Quiescers 2024-08-28T19:43:06.704 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-08-28T19:43:06.714 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-08-28T19:43:06.714 INFO:teuthology.task.exec:Executing custom commands... 2024-08-28T19:43:06.714 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-08-28T19:43:06.715 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-08-28T19:43:06.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-08-28T19:43:06.808 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-08-28T19:43:06.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:06 smithi028 ceph-mon[21769]: pgmap v1032: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 2.0 MiB/s wr, 162 op/s 2024-08-28T19:43:06.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:06 smithi115 ceph-mon[22184]: pgmap v1032: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 2.0 MiB/s wr, 162 op/s 2024-08-28T19:43:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.959+0000 7f5146f29640 1 -- 172.21.15.16:0/2510744088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5140108080 msgr2=0x7f5140108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.959+0000 7f5146f29640 1 --2- 172.21.15.16:0/2510744088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5140108080 0x7f5140108490 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f512c009900 tx=0x7f512c02a410 comp rx=0 tx=0).stop 2024-08-28T19:43:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.959+0000 7f5146f29640 1 -- 172.21.15.16:0/2510744088 shutdown_connections 2024-08-28T19:43:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.959+0000 7f5146f29640 1 --2- 172.21.15.16:0/2510744088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51401089d0 0x7f514010f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.959+0000 7f5146f29640 1 --2- 172.21.15.16:0/2510744088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5140108080 0x7f5140108490 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.959+0000 7f5146f29640 1 --2- 172.21.15.16:0/2510744088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5140102750 0x7f5140102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.959+0000 7f5146f29640 1 -- 172.21.15.16:0/2510744088 >> 172.21.15.16:0/2510744088 conn(0x7f514007a4e0 msgr2=0x7f514007a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.959+0000 7f5146f29640 1 -- 172.21.15.16:0/2510744088 shutdown_connections 2024-08-28T19:43:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.959+0000 7f5146f29640 1 -- 172.21.15.16:0/2510744088 wait complete. 2024-08-28T19:43:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.960+0000 7f5146f29640 1 Processor -- start 2024-08-28T19:43:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.960+0000 7f5146f29640 1 -- start start 2024-08-28T19:43:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.960+0000 7f5146f29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5140102750 0x7f514019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.960+0000 7f5146f29640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5140108080 0x7f514019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.960+0000 7f5146f29640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51401089d0 0x7f514019fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.960+0000 7f5146f29640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5140112660 con 0x7f5140108080 2024-08-28T19:43:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.960+0000 7f5146f29640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51401124e0 con 0x7f5140102750 2024-08-28T19:43:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.960+0000 7f5146f29640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f51401127e0 con 0x7f51401089d0 2024-08-28T19:43:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5140108080 0x7f514019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5140108080 0x7f514019a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33324/0 (socket says 172.21.15.16:33324) 2024-08-28T19:43:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513ffff640 1 -- 172.21.15.16:0/2209509263 learned_addr learned my addr 172.21.15.16:0/2209509263 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f5144cc5640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5140102750 0x7f514019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f51454c6640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51401089d0 0x7f514019fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513ffff640 1 -- 172.21.15.16:0/2209509263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51401089d0 msgr2=0x7f514019fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513ffff640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51401089d0 0x7f514019fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513ffff640 1 -- 172.21.15.16:0/2209509263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5140102750 msgr2=0x7f514019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513ffff640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5140102750 0x7f514019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513ffff640 1 -- 172.21.15.16:0/2209509263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51401a01e0 con 0x7f5140108080 2024-08-28T19:43:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513ffff640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5140108080 0x7f514019a850 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f512c02a930 tx=0x7f512c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513dffb640 1 -- 172.21.15.16:0/2209509263 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c03f030 con 0x7f5140108080 2024-08-28T19:43:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513dffb640 1 -- 172.21.15.16:0/2209509263 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f512c003aa0 con 0x7f5140108080 2024-08-28T19:43:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f5146f29640 1 -- 172.21.15.16:0/2209509263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51401a0470 con 0x7f5140108080 2024-08-28T19:43:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.961+0000 7f513dffb640 1 -- 172.21.15.16:0/2209509263 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c035070 con 0x7f5140108080 2024-08-28T19:43:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.962+0000 7f5146f29640 1 -- 172.21.15.16:0/2209509263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51401a6c60 con 0x7f5140108080 2024-08-28T19:43:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.962+0000 7f5146f29640 1 -- 172.21.15.16:0/2209509263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51401039c0 con 0x7f5140108080 2024-08-28T19:43:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.964+0000 7f513dffb640 1 -- 172.21.15.16:0/2209509263 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f512c042030 con 0x7f5140108080 2024-08-28T19:43:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.965+0000 7f513dffb640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51300b4ca0 0x7f51300b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.965+0000 7f5144cc5640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51300b4ca0 0x7f51300b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.965+0000 7f513dffb640 1 -- 172.21.15.16:0/2209509263 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9578+0+0 (secure 0 0 0) 0x7f512c1064d0 con 0x7f5140108080 2024-08-28T19:43:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.965+0000 7f5144cc5640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51300b4ca0 0x7f51300b71b0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5134005fd0 tx=0x7f513400b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:06.968+0000 7f513dffb640 1 -- 172.21.15.16:0/2209509263 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f512c0cce40 con 0x7f5140108080 2024-08-28T19:43:07.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.182+0000 7f5146f29640 1 -- 172.21.15.16:0/2209509263 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f51400635a0 con 0x7f51300b4ca0 2024-08-28T19:43:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.195+0000 7f513dffb640 1 -- 172.21.15.16:0/2209509263 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f51400635a0 con 0x7f51300b4ca0 2024-08-28T19:43:07.196 INFO:teuthology.orchestra.run.smithi016.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-08-28T00:00:00", "created": "2024-08-28T19:21:03", "first": "2024-08-28T19:22:00", "last": "2024-08-28T19:43:00", "last_pruned": "2024-08-28T19:43:01", "created_count": 22, "pruned_count": 16, "active": true} 2024-08-28T19:43:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 -- 172.21.15.16:0/2209509263 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51300b4ca0 msgr2=0x7f51300b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51300b4ca0 0x7f51300b71b0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5134005fd0 tx=0x7f513400b390 comp rx=0 tx=0).stop 2024-08-28T19:43:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 -- 172.21.15.16:0/2209509263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5140108080 msgr2=0x7f514019a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5140108080 0x7f514019a850 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f512c02a930 tx=0x7f512c004270 comp rx=0 tx=0).stop 2024-08-28T19:43:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 -- 172.21.15.16:0/2209509263 shutdown_connections 2024-08-28T19:43:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51401089d0 0x7f514019fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5140108080 0x7f514019a850 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f51300b4ca0 0x7f51300b71b0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 --2- 172.21.15.16:0/2209509263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5140102750 0x7f514019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 -- 172.21.15.16:0/2209509263 >> 172.21.15.16:0/2209509263 conn(0x7f514007a4e0 msgr2=0x7f5140101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 -- 172.21.15.16:0/2209509263 shutdown_connections 2024-08-28T19:43:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.197+0000 7f5146f29640 1 -- 172.21.15.16:0/2209509263 wait complete. 2024-08-28T19:43:07.209 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-08-28T19:43:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.322+0000 7fca5826e640 1 -- 172.21.15.16:0/597167097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca501068f0 msgr2=0x7fca50110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.322+0000 7fca5826e640 1 --2- 172.21.15.16:0/597167097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca501068f0 0x7fca50110330 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fca48009ca0 tx=0x7fca4802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.322+0000 7fca5826e640 1 -- 172.21.15.16:0/597167097 shutdown_connections 2024-08-28T19:43:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.322+0000 7fca5826e640 1 --2- 172.21.15.16:0/597167097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca501068f0 0x7fca50110330 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.322+0000 7fca5826e640 1 --2- 172.21.15.16:0/597167097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca50105fa0 0x7fca501063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.322+0000 7fca5826e640 1 --2- 172.21.15.16:0/597167097 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca50101e30 0x7fca50102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.322+0000 7fca5826e640 1 -- 172.21.15.16:0/597167097 >> 172.21.15.16:0/597167097 conn(0x7fca50079c60 msgr2=0x7fca5007a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.322+0000 7fca5826e640 1 -- 172.21.15.16:0/597167097 shutdown_connections 2024-08-28T19:43:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.322+0000 7fca5826e640 1 -- 172.21.15.16:0/597167097 wait complete. 2024-08-28T19:43:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.322+0000 7fca5826e640 1 Processor -- start 2024-08-28T19:43:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.322+0000 7fca5826e640 1 -- start start 2024-08-28T19:43:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca5826e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca50101e30 0x7fca5019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca5826e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca50105fa0 0x7fca5019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca5826e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca501068f0 0x7fca5019f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca5826e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fca501137d0 con 0x7fca50105fa0 2024-08-28T19:43:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca5826e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca50113650 con 0x7fca50101e30 2024-08-28T19:43:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca5826e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fca50113950 con 0x7fca501068f0 2024-08-28T19:43:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca55809640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca50105fa0 0x7fca5019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca55809640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca50105fa0 0x7fca5019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33350/0 (socket says 172.21.15.16:33350) 2024-08-28T19:43:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca55809640 1 -- 172.21.15.16:0/331750046 learned_addr learned my addr 172.21.15.16:0/331750046 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca5600a640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca50101e30 0x7fca5019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca5680b640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca501068f0 0x7fca5019f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca55809640 1 -- 172.21.15.16:0/331750046 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca501068f0 msgr2=0x7fca5019f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca55809640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca501068f0 0x7fca5019f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca55809640 1 -- 172.21.15.16:0/331750046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca50101e30 msgr2=0x7fca5019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca55809640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca50101e30 0x7fca5019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca55809640 1 -- 172.21.15.16:0/331750046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca501a00a0 con 0x7fca50105fa0 2024-08-28T19:43:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca55809640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca50105fa0 0x7fca5019a710 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fca4000e9f0 tx=0x7fca4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca477fe640 1 -- 172.21.15.16:0/331750046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4000ccc0 con 0x7fca50105fa0 2024-08-28T19:43:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca477fe640 1 -- 172.21.15.16:0/331750046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca400108e0 con 0x7fca50105fa0 2024-08-28T19:43:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.323+0000 7fca5826e640 1 -- 172.21.15.16:0/331750046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca501a0390 con 0x7fca50105fa0 2024-08-28T19:43:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.324+0000 7fca477fe640 1 -- 172.21.15.16:0/331750046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca40018860 con 0x7fca50105fa0 2024-08-28T19:43:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.324+0000 7fca5826e640 1 -- 172.21.15.16:0/331750046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca501a6b20 con 0x7fca50105fa0 2024-08-28T19:43:07.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.325+0000 7fca5826e640 1 -- 172.21.15.16:0/331750046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca50102240 con 0x7fca50105fa0 2024-08-28T19:43:07.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.326+0000 7fca477fe640 1 -- 172.21.15.16:0/331750046 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fca40010430 con 0x7fca50105fa0 2024-08-28T19:43:07.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.327+0000 7fca477fe640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca3c0b4ca0 0x7fca3c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.327+0000 7fca5600a640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca3c0b4ca0 0x7fca3c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.327+0000 7fca477fe640 1 -- 172.21.15.16:0/331750046 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9578+0+0 (secure 0 0 0) 0x7fca400e5c50 con 0x7fca50105fa0 2024-08-28T19:43:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.327+0000 7fca5600a640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca3c0b4ca0 0x7fca3c0b71b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009bd0 tx=0x7fca4c005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.331+0000 7fca477fe640 1 -- 172.21.15.16:0/331750046 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fca400abf70 con 0x7fca50105fa0 2024-08-28T19:43:07.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.541+0000 7fca5826e640 1 -- 172.21.15.16:0/331750046 --> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fca50109980 con 0x7fca3c0b4ca0 2024-08-28T19:43:07.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.554+0000 7fca477fe640 1 -- 172.21.15.16:0/331750046 <== mgr.14220 v2:172.21.15.16:6800/2594037529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fca50109980 con 0x7fca3c0b4ca0 2024-08-28T19:43:07.556 INFO:teuthology.orchestra.run.smithi016.stdout:/ 1m 6m3h 2024-08-28T19:43:07.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 -- 172.21.15.16:0/331750046 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca3c0b4ca0 msgr2=0x7fca3c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:07.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca3c0b4ca0 0x7fca3c0b71b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009bd0 tx=0x7fca4c005f90 comp rx=0 tx=0).stop 2024-08-28T19:43:07.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 -- 172.21.15.16:0/331750046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca50105fa0 msgr2=0x7fca5019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:07.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca50105fa0 0x7fca5019a710 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fca4000e9f0 tx=0x7fca4000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 -- 172.21.15.16:0/331750046 shutdown_connections 2024-08-28T19:43:07.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fca501068f0 0x7fca5019f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca50105fa0 0x7fca5019a710 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fca3c0b4ca0 0x7fca3c0b71b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 --2- 172.21.15.16:0/331750046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca50101e30 0x7fca5019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:07.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 -- 172.21.15.16:0/331750046 >> 172.21.15.16:0/331750046 conn(0x7fca50079c60 msgr2=0x7fca50109270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:07.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 -- 172.21.15.16:0/331750046 shutdown_connections 2024-08-28T19:43:07.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:07.557+0000 7fca5826e640 1 -- 172.21.15.16:0/331750046 wait complete. 2024-08-28T19:43:07.569 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-08-28T19:43:07.579 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-08-28T19:43:07.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:07 smithi028 ceph-mon[21769]: from='client.20499 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:43:07.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:07 smithi115 ceph-mon[22184]: from='client.20499 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:43:08.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:07 smithi016 ceph-mon[18556]: from='client.20499 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:43:08.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:08 smithi028 ceph-mon[21769]: from='client.20505 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:43:08.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:08 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:43:08.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:08 smithi028 ceph-mon[21769]: pgmap v1033: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-08-28T19:43:08.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:08 smithi115 ceph-mon[22184]: from='client.20505 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:43:08.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:08 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:43:08.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:08 smithi115 ceph-mon[22184]: pgmap v1033: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-08-28T19:43:09.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:08 smithi016 ceph-mon[18556]: from='client.20505 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-28T19:43:09.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:08 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:43:09.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:08 smithi016 ceph-mon[18556]: pgmap v1033: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-08-28T19:43:09.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:43:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:43:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:43:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:43:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: scrub summary: idle 2024-08-28T19:43:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:09.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:09.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:09.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:09.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:09.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:43:09.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:43:09.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:43:10.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:43:10.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:43:10.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:43:10.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: scrub summary: idle 2024-08-28T19:43:10.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:43:10.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:43:10.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:43:10.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:43:10.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:43:10.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:43:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: scrub summary: idle 2024-08-28T19:43:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.060 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:43:10.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:43:10.061 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:43:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:10 smithi115 ceph-mon[22184]: pgmap v1034: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 2.2 MiB/s wr, 178 op/s 2024-08-28T19:43:11.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:10 smithi016 ceph-mon[18556]: pgmap v1034: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 2.2 MiB/s wr, 178 op/s 2024-08-28T19:43:11.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:10 smithi028 ceph-mon[21769]: pgmap v1034: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 2.2 MiB/s wr, 178 op/s 2024-08-28T19:43:13.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:13 smithi028 ceph-mon[21769]: pgmap v1035: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 976 KiB/s wr, 103 op/s 2024-08-28T19:43:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:13 smithi115 ceph-mon[22184]: pgmap v1035: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 976 KiB/s wr, 103 op/s 2024-08-28T19:43:13.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:13 smithi016 ceph-mon[18556]: pgmap v1035: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 976 KiB/s wr, 103 op/s 2024-08-28T19:43:15.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:15 smithi028 ceph-mon[21769]: pgmap v1036: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 388 KiB/s wr, 49 op/s 2024-08-28T19:43:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:15 smithi115 ceph-mon[22184]: pgmap v1036: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 388 KiB/s wr, 49 op/s 2024-08-28T19:43:15.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:15 smithi016 ceph-mon[18556]: pgmap v1036: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 388 KiB/s wr, 49 op/s 2024-08-28T19:43:16.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:16 smithi016 ceph-mon[18556]: osdmap e116: 12 total, 12 up, 12 in 2024-08-28T19:43:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:16 smithi028 ceph-mon[21769]: osdmap e116: 12 total, 12 up, 12 in 2024-08-28T19:43:16.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:16 smithi115 ceph-mon[22184]: osdmap e116: 12 total, 12 up, 12 in 2024-08-28T19:43:17.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:17 smithi016 ceph-mon[18556]: pgmap v1038: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 322 KiB/s wr, 37 op/s 2024-08-28T19:43:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:17 smithi028 ceph-mon[21769]: pgmap v1038: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 322 KiB/s wr, 37 op/s 2024-08-28T19:43:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:17 smithi115 ceph-mon[22184]: pgmap v1038: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 322 KiB/s wr, 37 op/s 2024-08-28T19:43:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:19 smithi016 ceph-mon[18556]: pgmap v1039: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 182 KiB/s wr, 27 op/s 2024-08-28T19:43:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:19 smithi028 ceph-mon[21769]: pgmap v1039: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 182 KiB/s wr, 27 op/s 2024-08-28T19:43:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:19 smithi115 ceph-mon[22184]: pgmap v1039: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 182 KiB/s wr, 27 op/s 2024-08-28T19:43:21.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:21 smithi016 ceph-mon[18556]: pgmap v1040: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 162 KiB/s wr, 23 op/s 2024-08-28T19:43:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:21 smithi028 ceph-mon[21769]: pgmap v1040: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 162 KiB/s wr, 23 op/s 2024-08-28T19:43:21.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:21 smithi115 ceph-mon[22184]: pgmap v1040: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 162 KiB/s wr, 23 op/s 2024-08-28T19:43:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:23 smithi016 ceph-mon[18556]: pgmap v1041: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.4 MiB/s wr, 47 op/s 2024-08-28T19:43:23.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:23 smithi028 ceph-mon[21769]: pgmap v1041: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.4 MiB/s wr, 47 op/s 2024-08-28T19:43:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:23 smithi115 ceph-mon[22184]: pgmap v1041: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.4 MiB/s wr, 47 op/s 2024-08-28T19:43:25.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:25 smithi016 ceph-mon[18556]: pgmap v1042: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.8 MiB/s wr, 55 op/s 2024-08-28T19:43:25.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:25 smithi028 ceph-mon[21769]: pgmap v1042: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.8 MiB/s wr, 55 op/s 2024-08-28T19:43:25.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:25 smithi115 ceph-mon[22184]: pgmap v1042: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.8 MiB/s wr, 55 op/s 2024-08-28T19:43:27.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:27 smithi016 ceph-mon[18556]: pgmap v1043: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 1.7 MiB/s wr, 57 op/s 2024-08-28T19:43:27.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:27 smithi028 ceph-mon[21769]: pgmap v1043: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 1.7 MiB/s wr, 57 op/s 2024-08-28T19:43:27.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:27 smithi115 ceph-mon[22184]: pgmap v1043: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 1.7 MiB/s wr, 57 op/s 2024-08-28T19:43:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:29 smithi016 ceph-mon[18556]: pgmap v1044: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.5 MiB/s wr, 49 op/s 2024-08-28T19:43:29.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:29 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:43:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:29 smithi028 ceph-mon[21769]: pgmap v1044: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.5 MiB/s wr, 49 op/s 2024-08-28T19:43:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:29 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:43:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:29 smithi115 ceph-mon[22184]: pgmap v1044: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.5 MiB/s wr, 49 op/s 2024-08-28T19:43:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:29 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:43:31.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:31 smithi016 ceph-mon[18556]: pgmap v1045: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.5 MiB/s wr, 50 op/s 2024-08-28T19:43:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:31 smithi028 ceph-mon[21769]: pgmap v1045: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.5 MiB/s wr, 50 op/s 2024-08-28T19:43:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:31 smithi115 ceph-mon[22184]: pgmap v1045: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.5 MiB/s wr, 50 op/s 2024-08-28T19:43:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:33 smithi016 ceph-mon[18556]: pgmap v1046: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 1.5 MiB/s wr, 53 op/s 2024-08-28T19:43:33.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:33 smithi028 ceph-mon[21769]: pgmap v1046: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 1.5 MiB/s wr, 53 op/s 2024-08-28T19:43:33.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:33 smithi115 ceph-mon[22184]: pgmap v1046: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 1.5 MiB/s wr, 53 op/s 2024-08-28T19:43:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:34 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:43:34.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:34 smithi028 ceph-mon[21769]: pgmap v1047: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 447 KiB/s wr, 27 op/s 2024-08-28T19:43:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:34 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:43:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:34 smithi115 ceph-mon[22184]: pgmap v1047: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 447 KiB/s wr, 27 op/s 2024-08-28T19:43:35.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:34 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:43:35.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:34 smithi016 ceph-mon[18556]: pgmap v1047: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 447 KiB/s wr, 27 op/s 2024-08-28T19:43:35.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-08-28T19:43:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.215+0000 7f0a3bc16640 1 -- 172.21.15.16:0/949245319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a34109b90 msgr2=0x7f0a341103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.215+0000 7f0a3bc16640 1 --2- 172.21.15.16:0/949245319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a34109b90 0x7f0a341103c0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009ca0 tx=0x7f0a2c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.215+0000 7f0a3bc16640 1 -- 172.21.15.16:0/949245319 shutdown_connections 2024-08-28T19:43:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.215+0000 7f0a3bc16640 1 --2- 172.21.15.16:0/949245319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a34109b90 0x7f0a341103c0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.215+0000 7f0a3bc16640 1 --2- 172.21.15.16:0/949245319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a34109240 0x7f0a34109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.215+0000 7f0a3bc16640 1 --2- 172.21.15.16:0/949245319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a34103900 0x7f0a34103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.215+0000 7f0a3bc16640 1 -- 172.21.15.16:0/949245319 >> 172.21.15.16:0/949245319 conn(0x7f0a3407b5f0 msgr2=0x7f0a3407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.216+0000 7f0a3bc16640 1 -- 172.21.15.16:0/949245319 shutdown_connections 2024-08-28T19:43:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.216+0000 7f0a3bc16640 1 -- 172.21.15.16:0/949245319 wait complete. 2024-08-28T19:43:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.216+0000 7f0a3bc16640 1 Processor -- start 2024-08-28T19:43:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.216+0000 7f0a3bc16640 1 -- start start 2024-08-28T19:43:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.216+0000 7f0a3bc16640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a34103900 0x7f0a340754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.216+0000 7f0a3bc16640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a34109240 0x7f0a340759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.216+0000 7f0a3bc16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a34109b90 0x7f0a34075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a3bc16640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a34113850 con 0x7f0a34103900 2024-08-28T19:43:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a3bc16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341136d0 con 0x7f0a34109b90 2024-08-28T19:43:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a3bc16640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341139d0 con 0x7f0a34109240 2024-08-28T19:43:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a399b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a34103900 0x7f0a340754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a399b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a34103900 0x7f0a340754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50194/0 (socket says 172.21.15.16:50194) 2024-08-28T19:43:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a399b2640 1 -- 172.21.15.16:0/3374749438 learned_addr learned my addr 172.21.15.16:0/3374749438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a3a1b3640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a34109b90 0x7f0a34075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a391b1640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a34109240 0x7f0a340759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a399b2640 1 -- 172.21.15.16:0/3374749438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a34109240 msgr2=0x7f0a340759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a399b2640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a34109240 0x7f0a340759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a399b2640 1 -- 172.21.15.16:0/3374749438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a34109b90 msgr2=0x7f0a34075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a399b2640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a34109b90 0x7f0a34075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a399b2640 1 -- 172.21.15.16:0/3374749438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a341a84a0 con 0x7f0a34103900 2024-08-28T19:43:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a399b2640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a34103900 0x7f0a340754b0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000e9f0 tx=0x7f0a3000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a2affd640 1 -- 172.21.15.16:0/3374749438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3000cd80 con 0x7f0a34103900 2024-08-28T19:43:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a3bc16640 1 -- 172.21.15.16:0/3374749438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a341a8790 con 0x7f0a34103900 2024-08-28T19:43:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a2affd640 1 -- 172.21.15.16:0/3374749438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a300108e0 con 0x7f0a34103900 2024-08-28T19:43:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a2affd640 1 -- 172.21.15.16:0/3374749438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a30018980 con 0x7f0a34103900 2024-08-28T19:43:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.217+0000 7f0a3bc16640 1 -- 172.21.15.16:0/3374749438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a341a8cd0 con 0x7f0a34103900 2024-08-28T19:43:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.220+0000 7f0a2affd640 1 -- 172.21.15.16:0/3374749438 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f0a30010430 con 0x7f0a34103900 2024-08-28T19:43:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.220+0000 7f0a4218b740 1 Processor -- start 2024-08-28T19:43:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.220+0000 7f0a4218b740 1 -- start start 2024-08-28T19:43:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 0x56238a5ff300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a5ff840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a5ffd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a4218b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56238a576630 con 0x56238a2311b0 2024-08-28T19:43:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a4218b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56238a565bc0 con 0x56238a25a070 2024-08-28T19:43:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a4218b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56238a569b70 con 0x56238a239390 2024-08-28T19:43:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a399b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a5ff840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a3a1b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 0x56238a5ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a399b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a5ff840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50210/0 (socket says 172.21.15.16:50210) 2024-08-28T19:43:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a399b2640 1 -- 172.21.15.16:0/623203847 learned_addr learned my addr 172.21.15.16:0/623203847 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a3a1b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 0x56238a5ff300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:54716/0 (socket says 172.21.15.16:54716) 2024-08-28T19:43:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a391b1640 1 --2- 172.21.15.16:0/623203847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a5ffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a399b2640 1 -- 172.21.15.16:0/623203847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 msgr2=0x56238a5ffd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a399b2640 1 --2- 172.21.15.16:0/623203847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a5ffd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a399b2640 1 -- 172.21.15.16:0/623203847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 msgr2=0x56238a5ff300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a399b2640 1 --2- 172.21.15.16:0/623203847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 0x56238a5ff300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a399b2640 1 -- 172.21.15.16:0/623203847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56238a2c5650 con 0x56238a2311b0 2024-08-28T19:43:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a3a1b3640 1 --2- 172.21.15.16:0/623203847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 0x56238a5ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a2affd640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a200b4ca0 0x7f0a200b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a399b2640 1 --2- 172.21.15.16:0/623203847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a5ff840 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f0a300e2f10 tx=0x7f0a300ee760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa518f1e17d3cd38 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.221+0000 7f0a3a1b3640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a200b4ca0 0x7f0a200b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.222+0000 7f0a1ffff640 1 -- 172.21.15.16:0/623203847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a300a7eb0 con 0x56238a2311b0 2024-08-28T19:43:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.222+0000 7f0a2affd640 1 -- 172.21.15.16:0/3374749438 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f0a300e5c20 con 0x7f0a34103900 2024-08-28T19:43:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.222+0000 7f0a1ffff640 1 -- 172.21.15.16:0/623203847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a300f3070 con 0x56238a2311b0 2024-08-28T19:43:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.222+0000 7f0a1ffff640 1 -- 172.21.15.16:0/623203847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a300ab760 con 0x56238a2311b0 2024-08-28T19:43:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.222+0000 7f0a4218b740 1 -- 172.21.15.16:0/623203847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 msgr2=0x56238a5ff840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.223+0000 7f0a4218b740 1 --2- 172.21.15.16:0/623203847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a5ff840 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f0a300e2f10 tx=0x7f0a300ee760 comp rx=0 tx=0).stop 2024-08-28T19:43:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.222+0000 7f0a3a1b3640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a200b4ca0 0x7f0a200b71b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c02f500 tx=0x7f0a2c02f490 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.223+0000 7f0a4218b740 1 -- 172.21.15.16:0/623203847 shutdown_connections 2024-08-28T19:43:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.223+0000 7f0a4218b740 1 --2- 172.21.15.16:0/623203847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a5ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.223+0000 7f0a4218b740 1 --2- 172.21.15.16:0/623203847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a5ff840 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.223+0000 7f0a4218b740 1 --2- 172.21.15.16:0/623203847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 0x56238a5ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.223+0000 7f0a4218b740 1 -- 172.21.15.16:0/623203847 >> 172.21.15.16:0/623203847 conn(0x56238a24e460 msgr2=0x56238a5fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.223+0000 7f0a4218b740 1 -- 172.21.15.16:0/623203847 shutdown_connections 2024-08-28T19:43:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.223+0000 7f0a4218b740 1 -- 172.21.15.16:0/623203847 wait complete. 2024-08-28T19:43:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.223+0000 7f0a4218b740 1 -- start start 2024-08-28T19:43:36.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.223+0000 7f0a4218b740 1 Processor -- start 2024-08-28T19:43:36.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a60aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a60afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 0x56238a61f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a4218b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56238a3c8700 con 0x56238a2311b0 2024-08-28T19:43:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a4218b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56238a4ec700 con 0x56238a25a070 2024-08-28T19:43:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a4218b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56238a60dae0 con 0x56238a239390 2024-08-28T19:43:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a399b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a60aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a399b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a60aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50226/0 (socket says 172.21.15.16:50226) 2024-08-28T19:43:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a399b2640 1 -- 172.21.15.16:0/2778759345 learned_addr learned my addr 172.21.15.16:0/2778759345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a4218b740 10 client.0 _ll_register_callbacks cb 0x56238a51bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a3a1b3640 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 0x56238a61f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.224+0000 7f0a391b1640 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a60afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a399b2640 1 -- 172.21.15.16:0/2778759345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 msgr2=0x56238a60afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a399b2640 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a60afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a399b2640 1 -- 172.21.15.16:0/2778759345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 msgr2=0x56238a61f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a399b2640 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 0x56238a61f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a399b2640 1 -- 172.21.15.16:0/2778759345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56238a5258b0 con 0x56238a2311b0 2024-08-28T19:43:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a399b2640 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a60aa60 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f0a300a8070 tx=0x7f0a30009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8543bb18ae53f2d2 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/2778759345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a300ab940 con 0x56238a2311b0 2024-08-28T19:43:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/2778759345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f0a300691b0 con 0x56238a2311b0 2024-08-28T19:43:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a4218b740 1 -- 172.21.15.16:0/2778759345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a300f3070 con 0x56238a2311b0 2024-08-28T19:43:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/2778759345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f0a300abdc0 con 0x56238a2311b0 2024-08-28T19:43:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a4218b740 10 client.20517 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a4218b740 1 -- 172.21.15.16:0/2778759345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x56238a622040 con 0x56238a2311b0 2024-08-28T19:43:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.225+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/2778759345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a300f3070 con 0x56238a2311b0 2024-08-28T19:43:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.226+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/2778759345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7f0a300eb050 con 0x56238a2311b0 2024-08-28T19:43:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.226+0000 7f0a4218b740 10 client.20517 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.226+0000 7f0a4218b740 10 client.20517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:43:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.226+0000 7f0a4218b740 20 client.20517 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.226+0000 7f0a4218b740 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56238a2189f0 0x56238a6253a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.226+0000 7f0a4218b740 4 client.20517 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:43:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.226+0000 7f0a4218b740 1 -- 172.21.15.16:0/2778759345 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56238a578580 con 0x56238a2189f0 2024-08-28T19:43:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.226+0000 7f0a391b1640 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56238a2189f0 0x56238a6253a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.226+0000 7f0a1d7fa640 10 client.20517 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:36.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.226+0000 7f0a391b1640 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56238a2189f0 0x56238a6253a0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42b4c21c4795bb1d server_cookie=f6a99ca21831fbfa in_seq=0 out_seq=0 2024-08-28T19:43:36.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.227+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/2778759345 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56238a578580 con 0x56238a2189f0 2024-08-28T19:43:36.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.227+0000 7f0a1d7fa640 10 client.20517 handle_command_reply: tid=0 2024-08-28T19:43:36.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.227+0000 7f0a4218b740 1 client.20517 shutdown 2024-08-28T19:43:36.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.227+0000 7f0a4218b740 1 -- 172.21.15.16:0/2778759345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 msgr2=0x56238a60aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:36.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.227+0000 7f0a4218b740 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a60aa60 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f0a300a8070 tx=0x7f0a30009310 comp rx=0 tx=0).stop 2024-08-28T19:43:36.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.228+0000 7f0a4218b740 1 -- 172.21.15.16:0/2778759345 shutdown_connections 2024-08-28T19:43:36.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.228+0000 7f0a4218b740 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a25a070 0x56238a61f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.228+0000 7f0a4218b740 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56238a2189f0 0x56238a6253a0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.228+0000 7f0a4218b740 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a60afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.228+0000 7f0a4218b740 1 --2- 172.21.15.16:0/2778759345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a2311b0 0x56238a60aa60 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.228+0000 7f0a1d7fa640 0 client.20517 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:36.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.228+0000 7f0a4218b740 1 -- 172.21.15.16:0/2778759345 >> 172.21.15.16:0/2778759345 conn(0x56238a24e460 msgr2=0x56238a6024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:36.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.228+0000 7f0a4218b740 1 -- 172.21.15.16:0/2778759345 shutdown_connections 2024-08-28T19:43:36.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.228+0000 7f0a4218b740 1 -- 172.21.15.16:0/2778759345 wait complete. 2024-08-28T19:43:36.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.228+0000 7f0a4218b740 20 client.20517 trim_cache size 0 max 16384 2024-08-28T19:43:36.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a4218b740 1 Processor -- start 2024-08-28T19:43:36.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a4218b740 1 -- start start 2024-08-28T19:43:36.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 0x56238a602400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a602940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a25a070 0x56238a607c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a4218b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341aea30 con 0x56238a25a070 2024-08-28T19:43:36.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a4218b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341b0600 con 0x56238a2311b0 2024-08-28T19:43:36.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a4218b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341af100 con 0x56238a239390 2024-08-28T19:43:36.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a3a1b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a25a070 0x56238a607c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a3a1b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a25a070 0x56238a607c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50238/0 (socket says 172.21.15.16:50238) 2024-08-28T19:43:36.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a3a1b3640 1 -- 172.21.15.16:0/1819143406 learned_addr learned my addr 172.21.15.16:0/1819143406 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a399b2640 1 --2- 172.21.15.16:0/1819143406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 0x56238a602400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a391b1640 1 --2- 172.21.15.16:0/1819143406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a602940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a399b2640 1 -- 172.21.15.16:0/1819143406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 msgr2=0x56238a602940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a399b2640 1 --2- 172.21.15.16:0/1819143406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a602940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a399b2640 1 -- 172.21.15.16:0/1819143406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a25a070 msgr2=0x56238a607c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a399b2640 1 --2- 172.21.15.16:0/1819143406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a25a070 0x56238a607c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a399b2640 1 -- 172.21.15.16:0/1819143406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56238a5fdb60 con 0x56238a2311b0 2024-08-28T19:43:36.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.246+0000 7f0a3a1b3640 1 --2- 172.21.15.16:0/1819143406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a25a070 0x56238a607c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.247+0000 7f0a399b2640 1 --2- 172.21.15.16:0/1819143406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 0x56238a602400 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0a300abf80 tx=0x7f0a30018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a15f9bdc928d485 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.247+0000 7f0a1f7fe640 1 -- 172.21.15.16:0/1819143406 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a300ab760 con 0x56238a2311b0 2024-08-28T19:43:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.247+0000 7f0a1f7fe640 1 -- 172.21.15.16:0/1819143406 <== mon.2 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a300abb00 con 0x56238a2311b0 2024-08-28T19:43:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.247+0000 7f0a1f7fe640 1 -- 172.21.15.16:0/1819143406 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a300a7eb0 con 0x56238a2311b0 2024-08-28T19:43:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.248+0000 7f0a4218b740 1 -- 172.21.15.16:0/1819143406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 msgr2=0x56238a602400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.248+0000 7f0a4218b740 1 --2- 172.21.15.16:0/1819143406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 0x56238a602400 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0a300abf80 tx=0x7f0a30018420 comp rx=0 tx=0).stop 2024-08-28T19:43:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.248+0000 7f0a4218b740 1 -- 172.21.15.16:0/1819143406 shutdown_connections 2024-08-28T19:43:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.248+0000 7f0a4218b740 1 --2- 172.21.15.16:0/1819143406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a25a070 0x56238a607c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.248+0000 7f0a4218b740 1 --2- 172.21.15.16:0/1819143406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a239390 0x56238a602940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.248+0000 7f0a4218b740 1 --2- 172.21.15.16:0/1819143406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 0x56238a602400 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.248+0000 7f0a4218b740 1 -- 172.21.15.16:0/1819143406 >> 172.21.15.16:0/1819143406 conn(0x56238a24e460 msgr2=0x56238a6092e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.248+0000 7f0a4218b740 1 -- 172.21.15.16:0/1819143406 shutdown_connections 2024-08-28T19:43:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.248+0000 7f0a4218b740 1 -- 172.21.15.16:0/1819143406 wait complete. 2024-08-28T19:43:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.248+0000 7f0a4218b740 1 -- start start 2024-08-28T19:43:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.248+0000 7f0a4218b740 1 Processor -- start 2024-08-28T19:43:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 0x56238a6265d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a239390 0x56238a626b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a4218b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a25a070 0x56238a627050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a4218b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341aff30 con 0x56238a239390 2024-08-28T19:43:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a4218b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341af860 con 0x56238a2311b0 2024-08-28T19:43:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a4218b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341aca50 con 0x56238a25a070 2024-08-28T19:43:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a4218b740 10 client.0 _ll_register_callbacks cb 0x7f0a30005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a391b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a239390 0x56238a626b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a391b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a239390 0x56238a626b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50252/0 (socket says 172.21.15.16:50252) 2024-08-28T19:43:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a391b1640 1 -- 172.21.15.16:0/1929276759 learned_addr learned my addr 172.21.15.16:0/1929276759 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a3a1b3640 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a25a070 0x56238a627050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a399b2640 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 0x56238a6265d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a391b1640 1 -- 172.21.15.16:0/1929276759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a25a070 msgr2=0x56238a627050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a391b1640 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a25a070 0x56238a627050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a391b1640 1 -- 172.21.15.16:0/1929276759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 msgr2=0x56238a6265d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a391b1640 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 0x56238a6265d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a391b1640 1 -- 172.21.15.16:0/1929276759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56238a68ed20 con 0x56238a239390 2024-08-28T19:43:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.249+0000 7f0a399b2640 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 0x56238a6265d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a391b1640 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a239390 0x56238a626b10 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009e50 tx=0x7f0a24010970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8bc7e093afeca77 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/1929276759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24029060 con 0x56238a239390 2024-08-28T19:43:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/1929276759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f0a24004310 con 0x56238a239390 2024-08-28T19:43:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a4218b740 1 -- 172.21.15.16:0/1929276759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a300abb00 con 0x56238a239390 2024-08-28T19:43:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/1929276759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f0a24030050 con 0x56238a239390 2024-08-28T19:43:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a4218b740 10 client.20523 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a4218b740 1 -- 172.21.15.16:0/1929276759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x56238a690010 con 0x56238a239390 2024-08-28T19:43:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/1929276759 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a300abb00 con 0x56238a239390 2024-08-28T19:43:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/1929276759 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7f0a2401f800 con 0x56238a239390 2024-08-28T19:43:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a4218b740 10 client.20523 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a4218b740 10 client.20523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:43:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a4218b740 20 client.20523 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a4218b740 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56238a1edda0 0x56238a692770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a4218b740 4 client.20523 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-28T19:43:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.250+0000 7f0a4218b740 1 -- 172.21.15.16:0/1929276759 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56238a574020 con 0x56238a1edda0 2024-08-28T19:43:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.251+0000 7f0a399b2640 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56238a1edda0 0x56238a692770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.251+0000 7f0a399b2640 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56238a1edda0 0x56238a692770 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ad32cc290f3375c server_cookie=d123fd07dee5a354 in_seq=0 out_seq=0 2024-08-28T19:43:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.251+0000 7f0a1d7fa640 10 client.20523 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.251+0000 7f0a1d7fa640 1 -- 172.21.15.16:0/1929276759 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+8042 (crc 0 0 0) 0x56238a574020 con 0x56238a1edda0 2024-08-28T19:43:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.251+0000 7f0a1d7fa640 10 client.20523 handle_command_reply: tid=0 2024-08-28T19:43:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.251+0000 7f0a4218b740 1 client.20523 shutdown 2024-08-28T19:43:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a4218b740 1 -- 172.21.15.16:0/1929276759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a239390 msgr2=0x56238a626b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a4218b740 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a239390 0x56238a626b10 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009e50 tx=0x7f0a24010970 comp rx=0 tx=0).stop 2024-08-28T19:43:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a4218b740 1 -- 172.21.15.16:0/1929276759 shutdown_connections 2024-08-28T19:43:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a4218b740 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56238a1edda0 0x56238a692770 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a4218b740 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56238a25a070 0x56238a627050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a4218b740 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56238a239390 0x56238a626b10 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a1d7fa640 0 client.20523 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a4218b740 1 --2- 172.21.15.16:0/1929276759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238a2311b0 0x56238a6265d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a4218b740 1 -- 172.21.15.16:0/1929276759 >> 172.21.15.16:0/1929276759 conn(0x56238a24e460 msgr2=0x56238a602400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a4218b740 1 -- 172.21.15.16:0/1929276759 shutdown_connections 2024-08-28T19:43:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a4218b740 1 -- 172.21.15.16:0/1929276759 wait complete. 2024-08-28T19:43:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a4218b740 20 client.20523 trim_cache size 0 max 16384 2024-08-28T19:43:36.262 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:43:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-08-28T19:43:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {}, 2024-08-28T19:43:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_stats": { 2024-08-28T19:43:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: "414082c7-7ecd-4292-9ec5-4c4504fedaf1": { 2024-08-28T19:43:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:42:27.129235+0000", 2024-08-28T19:43:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: "39fa9343-222a-493e-bb10-c8863c663453": { 2024-08-28T19:43:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:41:33.609756+0000", 2024-08-28T19:43:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: "a466ab70-970a-4b3e-927e-47c65e508132": { 2024-08-28T19:43:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:40:38.307336+0000", 2024-08-28T19:43:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: "18e2c1b8-628f-4f38-b56c-dcde542d4ddb": { 2024-08-28T19:43:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:55.619275+0000", 2024-08-28T19:43:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: "44650602-6cd3-4083-8dac-11bd45aa6f43": { 2024-08-28T19:43:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:04.295349+0000", 2024-08-28T19:43:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: "3f4b095a-d877-4315-8c85-59f66049db8d": { 2024-08-28T19:43:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:32:18.491662+0000", 2024-08-28T19:43:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: "840047cc-208d-4758-921b-8f80ba8e251f": { 2024-08-28T19:43:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:35:26.532270+0000", 2024-08-28T19:43:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: "af6ece82-89e0-4ca5-a70b-36769e0fcc98": { 2024-08-28T19:43:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:31:32.137992+0000", 2024-08-28T19:43:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: "d8a48fd9-cf32-460f-8f4e-8dd4b3061fea": { 2024-08-28T19:43:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:34:39.314624+0000", 2024-08-28T19:43:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: "217eb860-c45a-4efe-97c5-379fc45c3873": { 2024-08-28T19:43:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:47.297278+0000", 2024-08-28T19:43:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: "f6d83c9f-bbbd-4891-ba71-cffd5e9780a4": { 2024-08-28T19:43:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:58.770740+0000", 2024-08-28T19:43:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: "b678eb41-8e9a-4031-8818-43d5f536898b": { 2024-08-28T19:43:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:06.486341+0000", 2024-08-28T19:43:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.275 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.275 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.275 INFO:teuthology.orchestra.run.smithi016.stdout: "f27568c6-5b0e-422a-bf5c-feccae126ec2": { 2024-08-28T19:43:36.275 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:38:02.233806+0000", 2024-08-28T19:43:36.275 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.275 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.276 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.276 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.276 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.276 INFO:teuthology.orchestra.run.smithi016.stdout: "fc26b76f-49a3-4ee7-aa70-a3bfe4ae111f": { 2024-08-28T19:43:36.276 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:55.506558+0000", 2024-08-28T19:43:36.276 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.276 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.277 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.277 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.277 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.277 INFO:teuthology.orchestra.run.smithi016.stdout: "fa70608a-b5e8-4395-998a-2f7edb54b221": { 2024-08-28T19:43:36.277 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:22:41.489236+0000", 2024-08-28T19:43:36.277 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.277 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.278 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.278 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.278 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.278 INFO:teuthology.orchestra.run.smithi016.stdout: "2e425dc0-868f-4ee6-89e5-7fe6d8d80a8d": { 2024-08-28T19:43:36.278 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:24:29.873118+0000", 2024-08-28T19:43:36.279 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.279 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.279 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.279 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.279 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.279 INFO:teuthology.orchestra.run.smithi016.stdout: "8de90bf3-5743-4711-98c2-a56ca0757291": { 2024-08-28T19:43:36.279 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:39:48.098772+0000", 2024-08-28T19:43:36.280 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.280 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.280 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.280 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.280 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.280 INFO:teuthology.orchestra.run.smithi016.stdout: "c36c0080-16dd-4e78-a556-732432a8ae9b": { 2024-08-28T19:43:36.280 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:37:09.160328+0000", 2024-08-28T19:43:36.281 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.281 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.281 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.281 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.281 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.281 INFO:teuthology.orchestra.run.smithi016.stdout: "1fc60423-efa1-4a92-91a1-0815560f5e52": { 2024-08-28T19:43:36.281 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:33:50.895013+0000", 2024-08-28T19:43:36.282 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.282 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.282 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.282 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.282 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.282 INFO:teuthology.orchestra.run.smithi016.stdout: "b842a7f8-9a3b-435e-88a7-49e794ad2bb8": { 2024-08-28T19:43:36.282 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:21:03.871040+0000", 2024-08-28T19:43:36.283 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.283 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.283 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.283 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.283 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.284 INFO:teuthology.orchestra.run.smithi016.stdout: "7af73992-6dbf-427c-84e6-e23aa2c95123": { 2024-08-28T19:43:36.284 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:25:18.232728+0000", 2024-08-28T19:43:36.284 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.284 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.284 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.284 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.284 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.285 INFO:teuthology.orchestra.run.smithi016.stdout: "c89de864-c285-403a-8de0-63c8123972fa": { 2024-08-28T19:43:36.285 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:26:54.897949+0000", 2024-08-28T19:43:36.285 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.285 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.285 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.285 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.285 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.286 INFO:teuthology.orchestra.run.smithi016.stdout: "38d6d63f-9bab-468d-8eac-25cd6efa2607": { 2024-08-28T19:43:36.286 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:27:41.055623+0000", 2024-08-28T19:43:36.286 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.287 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.287 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.287 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.288 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.288 INFO:teuthology.orchestra.run.smithi016.stdout: "4d705fea-6bef-4150-87f6-bc36721d0baa": { 2024-08-28T19:43:36.288 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:36:16.305843+0000", 2024-08-28T19:43:36.288 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.288 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.288 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.288 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.289 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.289 INFO:teuthology.orchestra.run.smithi016.stdout: "bcd1cb5e-342b-42b6-b8c5-504b6dc15ed2": { 2024-08-28T19:43:36.289 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:23:44.105435+0000", 2024-08-28T19:43:36.289 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "d4cf8d74-0ea8-4831-90ad-1492faca7486": { 2024-08-28T19:43:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:28:26.839308+0000", 2024-08-28T19:43:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.291 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.291 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.291 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.291 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.291 INFO:teuthology.orchestra.run.smithi016.stdout: "d863be82-a773-4056-b493-bb9d7d839e7e": { 2024-08-28T19:43:36.291 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:29:14.631936+0000", 2024-08-28T19:43:36.291 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.292 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.292 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.292 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.292 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:36.292 INFO:teuthology.orchestra.run.smithi016.stdout: "b7a2ba77-8b1b-4e1e-92b8-8a7de0999029": { 2024-08-28T19:43:36.292 INFO:teuthology.orchestra.run.smithi016.stdout: "start_time": "2024-08-28T19:30:00.247628+0000", 2024-08-28T19:43:36.292 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-08-28T19:43:36.293 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:36.293 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_passed": 0, 2024-08-28T19:43:36.293 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_failed": 0, 2024-08-28T19:43:36.293 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_skipped": 0 2024-08-28T19:43:36.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:36.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:36.293 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:43:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a3bc16640 1 -- 172.21.15.16:0/3374749438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a200b4ca0 msgr2=0x7f0a200b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a3bc16640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a200b4ca0 0x7f0a200b71b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c02f500 tx=0x7f0a2c02f490 comp rx=0 tx=0).stop 2024-08-28T19:43:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a3bc16640 1 -- 172.21.15.16:0/3374749438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a34103900 msgr2=0x7f0a340754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a3bc16640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a34103900 0x7f0a340754b0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000e9f0 tx=0x7f0a3000eeb0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a3bc16640 1 -- 172.21.15.16:0/3374749438 shutdown_connections 2024-08-28T19:43:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.252+0000 7f0a3bc16640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f0a200b4ca0 0x7f0a200b71b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.253+0000 7f0a3bc16640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a34109b90 0x7f0a34075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.253+0000 7f0a3bc16640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a34109240 0x7f0a340759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.253+0000 7f0a3bc16640 1 --2- 172.21.15.16:0/3374749438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a34103900 0x7f0a340754b0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.253+0000 7f0a3bc16640 1 -- 172.21.15.16:0/3374749438 >> 172.21.15.16:0/3374749438 conn(0x7f0a3407b5f0 msgr2=0x7f0a34107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.253+0000 7f0a3bc16640 1 -- 172.21.15.16:0/3374749438 shutdown_connections 2024-08-28T19:43:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:36.253+0000 7f0a3bc16640 1 -- 172.21.15.16:0/3374749438 wait complete. 2024-08-28T19:43:37.189 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:37 smithi016 ceph-mon[18556]: pgmap v1048: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 124 KiB/s wr, 20 op/s 2024-08-28T19:43:37.218 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-28T19:43:37.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:43:37.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:37 smithi028 ceph-mon[21769]: pgmap v1048: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 124 KiB/s wr, 20 op/s 2024-08-28T19:43:37.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:37 smithi115 ceph-mon[22184]: pgmap v1048: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 124 KiB/s wr, 20 op/s 2024-08-28T19:43:37.471 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.094+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2375999063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b4810dbe0 msgr2=0x7f8b481100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.094+0000 7f8b4fc88640 1 --2- 172.21.15.16:0/2375999063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b4810dbe0 0x7f8b481100b0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f8b40009ca0 tx=0x7f8b4002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:43:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.094+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2375999063 shutdown_connections 2024-08-28T19:43:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.094+0000 7f8b4fc88640 1 --2- 172.21.15.16:0/2375999063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b4810dbe0 0x7f8b481100b0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.094+0000 7f8b4fc88640 1 --2- 172.21.15.16:0/2375999063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b48105060 0x7f8b4810d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.094+0000 7f8b4fc88640 1 --2- 172.21.15.16:0/2375999063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b48104630 0x7f8b48104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.094+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2375999063 >> 172.21.15.16:0/2375999063 conn(0x7f8b480794b0 msgr2=0x7f8b480798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.094+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2375999063 shutdown_connections 2024-08-28T19:43:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.094+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2375999063 wait complete. 2024-08-28T19:43:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4fc88640 1 Processor -- start 2024-08-28T19:43:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4fc88640 1 -- start start 2024-08-28T19:43:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4fc88640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b48104630 0x7f8b4819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4fc88640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b48105060 0x7f8b4819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4fc88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b4810dbe0 0x7f8b4819fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4fc88640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b48113830 con 0x7f8b4810dbe0 2024-08-28T19:43:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4fc88640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b481136b0 con 0x7f8b48105060 2024-08-28T19:43:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4fc88640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8b481139b0 con 0x7f8b48104630 2024-08-28T19:43:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4e225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b4810dbe0 0x7f8b4819fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4e225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b4810dbe0 0x7f8b4819fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50262/0 (socket says 172.21.15.16:50262) 2024-08-28T19:43:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4e225640 1 -- 172.21.15.16:0/2146581963 learned_addr learned my addr 172.21.15.16:0/2146581963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.095+0000 7f8b4d223640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b48105060 0x7f8b4819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b4da24640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b48104630 0x7f8b4819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b4e225640 1 -- 172.21.15.16:0/2146581963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b48104630 msgr2=0x7f8b4819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b4e225640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b48104630 0x7f8b4819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b4e225640 1 -- 172.21.15.16:0/2146581963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b48105060 msgr2=0x7f8b4819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b4e225640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b48105060 0x7f8b4819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b4e225640 1 -- 172.21.15.16:0/2146581963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b481a0130 con 0x7f8b4810dbe0 2024-08-28T19:43:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b4da24640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b48104630 0x7f8b4819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b4e225640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b4810dbe0 0x7f8b4819fa00 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f8b40005cc0 tx=0x7f8b4002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b3effd640 1 -- 172.21.15.16:0/2146581963 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40003bf0 con 0x7f8b4810dbe0 2024-08-28T19:43:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2146581963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b481a03c0 con 0x7f8b4810dbe0 2024-08-28T19:43:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b3effd640 1 -- 172.21.15.16:0/2146581963 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b40003d90 con 0x7f8b4810dbe0 2024-08-28T19:43:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b3effd640 1 -- 172.21.15.16:0/2146581963 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4002fbc0 con 0x7f8b4810dbe0 2024-08-28T19:43:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.096+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2146581963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b481a6b80 con 0x7f8b4810dbe0 2024-08-28T19:43:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.097+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2146581963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b4806bf80 con 0x7f8b4810dbe0 2024-08-28T19:43:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.099+0000 7f8b3effd640 1 -- 172.21.15.16:0/2146581963 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8b4002fd60 con 0x7f8b4810dbe0 2024-08-28T19:43:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.099+0000 7f8b3effd640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b340b4ca0 0x7f8b340b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.099+0000 7f8b4da24640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b340b4ca0 0x7f8b340b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.099+0000 7f8b3effd640 1 -- 172.21.15.16:0/2146581963 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f8b40106ed0 con 0x7f8b4810dbe0 2024-08-28T19:43:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.100+0000 7f8b4da24640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b340b4ca0 0x7f8b340b71b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8b44005d00 tx=0x7f8b44005c70 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.104+0000 7f8b3effd640 1 -- 172.21.15.16:0/2146581963 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8b400cd8f0 con 0x7f8b4810dbe0 2024-08-28T19:43:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.326+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2146581963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b4819b2f0 con 0x7f8b4810dbe0 2024-08-28T19:43:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.327+0000 7f8b3effd640 1 -- 172.21.15.16:0/2146581963 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+11984 (secure 0 0 0) 0x7f8b400098b0 con 0x7f8b4810dbe0 2024-08-28T19:43:38.331 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:43:38.332 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":109,"btime":"2024-08-28T19:43:33:491728+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":109,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:43:33.371957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:43:38.333 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 109 2024-08-28T19:43:38.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.329+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2146581963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b340b4ca0 msgr2=0x7f8b340b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:38.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.329+0000 7f8b4fc88640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b340b4ca0 0x7f8b340b71b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8b44005d00 tx=0x7f8b44005c70 comp rx=0 tx=0).stop 2024-08-28T19:43:38.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.329+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2146581963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b4810dbe0 msgr2=0x7f8b4819fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:38.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.329+0000 7f8b4fc88640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b4810dbe0 0x7f8b4819fa00 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f8b40005cc0 tx=0x7f8b4002f380 comp rx=0 tx=0).stop 2024-08-28T19:43:38.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.329+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2146581963 shutdown_connections 2024-08-28T19:43:38.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.329+0000 7f8b4fc88640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b4810dbe0 0x7f8b4819fa00 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:38.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.329+0000 7f8b4fc88640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8b340b4ca0 0x7f8b340b71b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:38.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.329+0000 7f8b4fc88640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b48105060 0x7f8b4819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:38.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.329+0000 7f8b4fc88640 1 --2- 172.21.15.16:0/2146581963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8b48104630 0x7f8b4819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:38.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.329+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2146581963 >> 172.21.15.16:0/2146581963 conn(0x7f8b480794b0 msgr2=0x7f8b481076a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:38.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.330+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2146581963 shutdown_connections 2024-08-28T19:43:38.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:38.330+0000 7f8b4fc88640 1 -- 172.21.15.16:0/2146581963 wait complete. 2024-08-28T19:43:39.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-08-28T19:43:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:39 smithi028 ceph-mon[21769]: pgmap v1049: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-08-28T19:43:39.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:39 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2146581963' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:43:39.338 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:39 smithi115 ceph-mon[22184]: pgmap v1049: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-08-28T19:43:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:39 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2146581963' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:43:39.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:39 smithi016 ceph-mon[18556]: pgmap v1049: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-08-28T19:43:39.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:39 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2146581963' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:43:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.866+0000 7f31209d8640 1 -- 172.21.15.16:0/819070392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c075e20 msgr2=0x7f311c10f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.866+0000 7f31209d8640 1 --2- 172.21.15.16:0/819070392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c075e20 0x7f311c10f9a0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f3104009900 tx=0x7f310402a410 comp rx=0 tx=0).stop 2024-08-28T19:43:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.866+0000 7f31209d8640 1 -- 172.21.15.16:0/819070392 shutdown_connections 2024-08-28T19:43:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.866+0000 7f31209d8640 1 --2- 172.21.15.16:0/819070392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311c10fee0 0x7f311c112300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.866+0000 7f31209d8640 1 --2- 172.21.15.16:0/819070392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c075e20 0x7f311c10f9a0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.866+0000 7f31209d8640 1 --2- 172.21.15.16:0/819070392 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f311c0754d0 0x7f311c0758e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.866+0000 7f31209d8640 1 -- 172.21.15.16:0/819070392 >> 172.21.15.16:0/819070392 conn(0x7f311c07b5f0 msgr2=0x7f311c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.866+0000 7f31209d8640 1 -- 172.21.15.16:0/819070392 shutdown_connections 2024-08-28T19:43:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.866+0000 7f31209d8640 1 -- 172.21.15.16:0/819070392 wait complete. 2024-08-28T19:43:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.867+0000 7f31209d8640 1 Processor -- start 2024-08-28T19:43:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.867+0000 7f31209d8640 1 -- start start 2024-08-28T19:43:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.867+0000 7f31209d8640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f311c0754d0 0x7f311c19c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.867+0000 7f31209d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c075e20 0x7f311c19c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f31209d8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311c10fee0 0x7f311c1a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f31209d8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f311c115a20 con 0x7f311c075e20 2024-08-28T19:43:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f31209d8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1158a0 con 0x7f311c10fee0 2024-08-28T19:43:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f31209d8640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f311c115ba0 con 0x7f311c0754d0 2024-08-28T19:43:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f3119d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c075e20 0x7f311c19c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f3119d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c075e20 0x7f311c19c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50266/0 (socket says 172.21.15.16:50266) 2024-08-28T19:43:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f3119d9b640 1 -- 172.21.15.16:0/4062006008 learned_addr learned my addr 172.21.15.16:0/4062006008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f311ad9d640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311c10fee0 0x7f311c1a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f311a59c640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f311c0754d0 0x7f311c19c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f3119d9b640 1 -- 172.21.15.16:0/4062006008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f311c0754d0 msgr2=0x7f311c19c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f3119d9b640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f311c0754d0 0x7f311c19c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f3119d9b640 1 -- 172.21.15.16:0/4062006008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311c10fee0 msgr2=0x7f311c1a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f3119d9b640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311c10fee0 0x7f311c1a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f3119d9b640 1 -- 172.21.15.16:0/4062006008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311c1a2360 con 0x7f311c075e20 2024-08-28T19:43:39.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f3119d9b640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c075e20 0x7f311c19c990 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f310402a930 tx=0x7f3104004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:39.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.868+0000 7f31137fe640 1 -- 172.21.15.16:0/4062006008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310403f030 con 0x7f311c075e20 2024-08-28T19:43:39.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.869+0000 7f31209d8640 1 -- 172.21.15.16:0/4062006008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311c1a25f0 con 0x7f311c075e20 2024-08-28T19:43:39.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.869+0000 7f31137fe640 1 -- 172.21.15.16:0/4062006008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3104003aa0 con 0x7f311c075e20 2024-08-28T19:43:39.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.869+0000 7f31137fe640 1 -- 172.21.15.16:0/4062006008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104035070 con 0x7f311c075e20 2024-08-28T19:43:39.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.869+0000 7f31209d8640 1 -- 172.21.15.16:0/4062006008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f311c1a8db0 con 0x7f311c075e20 2024-08-28T19:43:39.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.871+0000 7f3126f4d740 1 Processor -- start 2024-08-28T19:43:39.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.871+0000 7f3126f4d740 1 -- start start 2024-08-28T19:43:39.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f31137fe640 1 -- 172.21.15.16:0/4062006008 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3104042030 con 0x7f311c075e20 2024-08-28T19:43:39.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905be070 0x564690963310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905951b0 0x564690963850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56469059d390 0x564690963d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f3126f4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5646908da680 con 0x5646905951b0 2024-08-28T19:43:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f3126f4d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5646908c9c10 con 0x5646905be070 2024-08-28T19:43:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f3126f4d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5646908cdbc0 con 0x56469059d390 2024-08-28T19:43:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f311a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905951b0 0x564690963850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f311a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905951b0 0x564690963850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50268/0 (socket says 172.21.15.16:50268) 2024-08-28T19:43:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f311a59c640 1 -- 172.21.15.16:0/1164865993 learned_addr learned my addr 172.21.15.16:0/1164865993 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f311ad9d640 1 --2- 172.21.15.16:0/1164865993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905be070 0x564690963310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f3119d9b640 1 --2- 172.21.15.16:0/1164865993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56469059d390 0x564690963d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f311a59c640 1 -- 172.21.15.16:0/1164865993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56469059d390 msgr2=0x564690963d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f311a59c640 1 --2- 172.21.15.16:0/1164865993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56469059d390 0x564690963d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f311a59c640 1 -- 172.21.15.16:0/1164865993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905be070 msgr2=0x564690963310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f311a59c640 1 --2- 172.21.15.16:0/1164865993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905be070 0x564690963310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f311a59c640 1 -- 172.21.15.16:0/1164865993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564690629650 con 0x5646905951b0 2024-08-28T19:43:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.872+0000 7f311ad9d640 1 --2- 172.21.15.16:0/1164865993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905be070 0x564690963310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.873+0000 7f311a59c640 1 --2- 172.21.15.16:0/1164865993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905951b0 0x564690963850 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f310c00b760 tx=0x7f310c00bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee4c19e3d35ddb9 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.873+0000 7f31137fe640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31080b4ca0 0x7f31080b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.873+0000 7f311ad9d640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31080b4ca0 0x7f31080b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.873+0000 7f3110ff9640 1 -- 172.21.15.16:0/1164865993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c003dd0 con 0x5646905951b0 2024-08-28T19:43:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.873+0000 7f31137fe640 1 -- 172.21.15.16:0/4062006008 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f3104105f80 con 0x7f311c075e20 2024-08-28T19:43:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.873+0000 7f3110ff9640 1 -- 172.21.15.16:0/1164865993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f310c015070 con 0x5646905951b0 2024-08-28T19:43:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.873+0000 7f3110ff9640 1 -- 172.21.15.16:0/1164865993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c004d90 con 0x5646905951b0 2024-08-28T19:43:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.874+0000 7f3126f4d740 1 -- 172.21.15.16:0/1164865993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905951b0 msgr2=0x564690963850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.874+0000 7f3126f4d740 1 --2- 172.21.15.16:0/1164865993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905951b0 0x564690963850 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f310c00b760 tx=0x7f310c00bc20 comp rx=0 tx=0).stop 2024-08-28T19:43:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.874+0000 7f311ad9d640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31080b4ca0 0x7f31080b71b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3114005fd0 tx=0x7f3114009600 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.874+0000 7f3126f4d740 1 -- 172.21.15.16:0/1164865993 shutdown_connections 2024-08-28T19:43:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.874+0000 7f3126f4d740 1 --2- 172.21.15.16:0/1164865993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56469059d390 0x564690963d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.874+0000 7f3126f4d740 1 --2- 172.21.15.16:0/1164865993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905951b0 0x564690963850 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.874+0000 7f3126f4d740 1 --2- 172.21.15.16:0/1164865993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905be070 0x564690963310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.874+0000 7f3126f4d740 1 -- 172.21.15.16:0/1164865993 >> 172.21.15.16:0/1164865993 conn(0x5646905b2460 msgr2=0x564690961200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.874+0000 7f3126f4d740 1 -- 172.21.15.16:0/1164865993 shutdown_connections 2024-08-28T19:43:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.874+0000 7f3126f4d740 1 -- 172.21.15.16:0/1164865993 wait complete. 2024-08-28T19:43:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.874+0000 7f3126f4d740 1 -- start start 2024-08-28T19:43:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f3126f4d740 1 Processor -- start 2024-08-28T19:43:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646905951b0 0x56469096ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56469059d390 0x56469096efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x56469097ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f3126f4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56469072c700 con 0x5646905be070 2024-08-28T19:43:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f3126f4d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564690850750 con 0x56469059d390 2024-08-28T19:43:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f3126f4d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564690971af0 con 0x5646905951b0 2024-08-28T19:43:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f3126f4d740 10 client.0 _ll_register_callbacks cb 0x56469087fe10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x56469097ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x56469097ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50276/0 (socket says 172.21.15.16:50276) 2024-08-28T19:43:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f311ad9d640 1 -- 172.21.15.16:0/1477439446 learned_addr learned my addr 172.21.15.16:0/1477439446 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f311a59c640 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646905951b0 0x56469096ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f3119d9b640 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56469059d390 0x56469096efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f311ad9d640 1 -- 172.21.15.16:0/1477439446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646905951b0 msgr2=0x56469096ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f311ad9d640 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646905951b0 0x56469096ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f311ad9d640 1 -- 172.21.15.16:0/1477439446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56469059d390 msgr2=0x56469096efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f311ad9d640 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56469059d390 0x56469096efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.875+0000 7f311ad9d640 1 -- 172.21.15.16:0/1477439446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564690889900 con 0x5646905be070 2024-08-28T19:43:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.876+0000 7f311ad9d640 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x56469097ae10 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f3114009b40 tx=0x7f3114004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99211c26b4fc80e4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.876+0000 7f3101ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.876+0000 7f3101ffb640 1 -- 172.21.15.16:0/1477439446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3114020620 con 0x5646905be070 2024-08-28T19:43:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.876+0000 7f3101ffb640 1 -- 172.21.15.16:0/1477439446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f31140207c0 con 0x5646905be070 2024-08-28T19:43:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.876+0000 7f3126f4d740 1 -- 172.21.15.16:0/1477439446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f310c015070 con 0x5646905be070 2024-08-28T19:43:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.876+0000 7f3101ffb640 1 -- 172.21.15.16:0/1477439446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f3114033050 con 0x5646905be070 2024-08-28T19:43:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.876+0000 7f3126f4d740 10 client.20541 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.876+0000 7f3126f4d740 1 -- 172.21.15.16:0/1477439446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x56469097d8f0 con 0x5646905be070 2024-08-28T19:43:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.876+0000 7f3101ffb640 1 -- 172.21.15.16:0/1477439446 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f310c015070 con 0x5646905be070 2024-08-28T19:43:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.877+0000 7f3101ffb640 1 -- 172.21.15.16:0/1477439446 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7f311400f040 con 0x5646905be070 2024-08-28T19:43:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.877+0000 7f3126f4d740 10 client.20541 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.877+0000 7f3126f4d740 10 client.20541 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:43:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.877+0000 7f3126f4d740 20 client.20541 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.877+0000 7f3126f4d740 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56469057c9f0 0x564690980ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.877+0000 7f3126f4d740 4 client.20541 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:43:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.877+0000 7f3126f4d740 1 -- 172.21.15.16:0/1477439446 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646908dc5d0 con 0x56469057c9f0 2024-08-28T19:43:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.877+0000 7f311a59c640 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56469057c9f0 0x564690980ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.877+0000 7f3101ffb640 10 client.20541 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:43:39.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.877+0000 7f311a59c640 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56469057c9f0 0x564690980ef0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=712935bccc6afbf4 server_cookie=84ef13964423bf42 in_seq=0 out_seq=0 2024-08-28T19:43:39.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3101ffb640 1 -- 172.21.15.16:0/1477439446 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5646908dc5d0 con 0x56469057c9f0 2024-08-28T19:43:39.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3101ffb640 10 client.20541 handle_command_reply: tid=0 2024-08-28T19:43:39.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3126f4d740 1 client.20541 shutdown 2024-08-28T19:43:39.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3126f4d740 1 -- 172.21.15.16:0/1477439446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 msgr2=0x56469097ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:39.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3126f4d740 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x56469097ae10 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f3114009b40 tx=0x7f3114004a40 comp rx=0 tx=0).stop 2024-08-28T19:43:39.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3126f4d740 1 -- 172.21.15.16:0/1477439446 shutdown_connections 2024-08-28T19:43:39.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3126f4d740 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x56469057c9f0 0x564690980ef0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3126f4d740 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x56469097ae10 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3126f4d740 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56469059d390 0x56469096efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3126f4d740 1 --2- 172.21.15.16:0/1477439446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646905951b0 0x56469096ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3126f4d740 1 -- 172.21.15.16:0/1477439446 >> 172.21.15.16:0/1477439446 conn(0x5646905b2460 msgr2=0x5646909664a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:39.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.878+0000 7f3101ffb640 0 client.20541 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:43:39.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.879+0000 7f3126f4d740 1 -- 172.21.15.16:0/1477439446 shutdown_connections 2024-08-28T19:43:39.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.879+0000 7f3126f4d740 1 -- 172.21.15.16:0/1477439446 wait complete. 2024-08-28T19:43:39.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.879+0000 7f3126f4d740 20 client.20541 trim_cache size 0 max 16384 2024-08-28T19:43:39.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.892+0000 7f3126f4d740 1 Processor -- start 2024-08-28T19:43:39.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.892+0000 7f3126f4d740 1 -- start start 2024-08-28T19:43:39.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.892+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646905951b0 0x564690966b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.892+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56469059d390 0x56469097ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.892+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905be070 0x564690960b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.892+0000 7f3126f4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1b2e10 con 0x56469059d390 2024-08-28T19:43:39.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.892+0000 7f3126f4d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1b49e0 con 0x5646905be070 2024-08-28T19:43:39.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.892+0000 7f3126f4d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1b34e0 con 0x5646905951b0 2024-08-28T19:43:39.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.892+0000 7f3119d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56469059d390 0x56469097ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.892+0000 7f3119d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56469059d390 0x56469097ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50280/0 (socket says 172.21.15.16:50280) 2024-08-28T19:43:39.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.892+0000 7f3119d9b640 1 -- 172.21.15.16:0/2902679407 learned_addr learned my addr 172.21.15.16:0/2902679407 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:39.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.893+0000 7f311a59c640 1 --2- 172.21.15.16:0/2902679407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646905951b0 0x564690966b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.893+0000 7f311ad9d640 1 --2- 172.21.15.16:0/2902679407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905be070 0x564690960b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.893+0000 7f3119d9b640 1 -- 172.21.15.16:0/2902679407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646905951b0 msgr2=0x564690966b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:39.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.893+0000 7f3119d9b640 1 --2- 172.21.15.16:0/2902679407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646905951b0 0x564690966b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.893+0000 7f3119d9b640 1 -- 172.21.15.16:0/2902679407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905be070 msgr2=0x564690960b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:39.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.893+0000 7f3119d9b640 1 --2- 172.21.15.16:0/2902679407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905be070 0x564690960b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.893+0000 7f3119d9b640 1 -- 172.21.15.16:0/2902679407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646909696c0 con 0x56469059d390 2024-08-28T19:43:39.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.893+0000 7f3119d9b640 1 --2- 172.21.15.16:0/2902679407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56469059d390 0x56469097ad10 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f31040c95c0 tx=0x7f3104031a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91f1365a9ec3cb0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:39.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.893+0000 7f3103fff640 1 -- 172.21.15.16:0/2902679407 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104049040 con 0x56469059d390 2024-08-28T19:43:39.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.893+0000 7f3103fff640 1 -- 172.21.15.16:0/2902679407 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31040cbe60 con 0x56469059d390 2024-08-28T19:43:39.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.893+0000 7f3103fff640 1 -- 172.21.15.16:0/2902679407 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104003c50 con 0x56469059d390 2024-08-28T19:43:39.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.894+0000 7f3126f4d740 1 -- 172.21.15.16:0/2902679407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56469059d390 msgr2=0x56469097ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:39.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.894+0000 7f3126f4d740 1 --2- 172.21.15.16:0/2902679407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56469059d390 0x56469097ad10 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f31040c95c0 tx=0x7f3104031a10 comp rx=0 tx=0).stop 2024-08-28T19:43:39.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.894+0000 7f3126f4d740 1 -- 172.21.15.16:0/2902679407 shutdown_connections 2024-08-28T19:43:39.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.894+0000 7f3126f4d740 1 --2- 172.21.15.16:0/2902679407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905be070 0x564690960b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.894+0000 7f3126f4d740 1 --2- 172.21.15.16:0/2902679407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56469059d390 0x56469097ad10 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.894+0000 7f3126f4d740 1 --2- 172.21.15.16:0/2902679407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646905951b0 0x564690966b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.894+0000 7f3126f4d740 1 -- 172.21.15.16:0/2902679407 >> 172.21.15.16:0/2902679407 conn(0x5646905b2460 msgr2=0x564690966410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:39.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.894+0000 7f3126f4d740 1 -- 172.21.15.16:0/2902679407 shutdown_connections 2024-08-28T19:43:39.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.894+0000 7f3126f4d740 1 -- 172.21.15.16:0/2902679407 wait complete. 2024-08-28T19:43:39.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.894+0000 7f3126f4d740 1 -- start start 2024-08-28T19:43:39.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.894+0000 7f3126f4d740 1 Processor -- start 2024-08-28T19:43:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905951b0 0x5646909737c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56469059d390 0x564690973da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f3126f4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x5646909e4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f3126f4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1b4310 con 0x5646905be070 2024-08-28T19:43:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f3126f4d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1b3c40 con 0x5646905951b0 2024-08-28T19:43:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f3126f4d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1b0e10 con 0x56469059d390 2024-08-28T19:43:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f3126f4d740 10 client.0 _ll_register_callbacks cb 0x7f3114009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x5646909e4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f311a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905951b0 0x5646909737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x5646909e4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50288/0 (socket says 172.21.15.16:50288) 2024-08-28T19:43:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f311ad9d640 1 -- 172.21.15.16:0/619134612 learned_addr learned my addr 172.21.15.16:0/619134612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f3119d9b640 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56469059d390 0x564690973da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f311ad9d640 1 -- 172.21.15.16:0/619134612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56469059d390 msgr2=0x564690973da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f311ad9d640 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56469059d390 0x564690973da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f311ad9d640 1 -- 172.21.15.16:0/619134612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905951b0 msgr2=0x5646909737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f311ad9d640 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905951b0 0x5646909737c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f311ad9d640 1 -- 172.21.15.16:0/619134612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646909e4fd0 con 0x5646905be070 2024-08-28T19:43:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f311a59c640 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905951b0 0x5646909737c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f311ad9d640 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x5646909e4960 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f3114018040 tx=0x7f3114011430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c460a3a037ad862c server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.895+0000 7f3101ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3101ffb640 1 -- 172.21.15.16:0/619134612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311401fb50 con 0x5646905be070 2024-08-28T19:43:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3101ffb640 1 -- 172.21.15.16:0/619134612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f311401d3e0 con 0x5646905be070 2024-08-28T19:43:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3126f4d740 1 -- 172.21.15.16:0/619134612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31040cbe60 con 0x5646905be070 2024-08-28T19:43:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3101ffb640 1 -- 172.21.15.16:0/619134612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f3114045050 con 0x5646905be070 2024-08-28T19:43:39.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3126f4d740 10 client.20547 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:39.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3126f4d740 1 -- 172.21.15.16:0/619134612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x5646909ec3d0 con 0x5646905be070 2024-08-28T19:43:39.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3101ffb640 1 -- 172.21.15.16:0/619134612 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31040cbe60 con 0x5646905be070 2024-08-28T19:43:39.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3101ffb640 1 -- 172.21.15.16:0/619134612 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7f311400f040 con 0x5646905be070 2024-08-28T19:43:39.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3126f4d740 10 client.20547 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:39.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3126f4d740 10 client.20547 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-08-28T19:43:39.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3126f4d740 20 client.20547 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:39.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3126f4d740 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x564690551da0 0x5646909eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:39.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3126f4d740 4 client.20547 mds_command: new command op to 24619 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:43:39.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.896+0000 7f3126f4d740 1 -- 172.21.15.16:0/619134612 --> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5646908d8070 con 0x564690551da0 2024-08-28T19:43:39.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f311a59c640 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x564690551da0 0x5646909eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:39.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3101ffb640 10 client.20547 ms_handle_connect on v2:172.21.15.115:6834/1297102593 2024-08-28T19:43:39.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f311a59c640 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x564690551da0 0x5646909eebb0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4d3b23b5165f4723 server_cookie=86ff4f1dab4b4d0b in_seq=0 out_seq=0 2024-08-28T19:43:39.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3101ffb640 1 -- 172.21.15.16:0/619134612 <== mds.4 v2:172.21.15.115:6834/1297102593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5646908d8070 con 0x564690551da0 2024-08-28T19:43:39.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3101ffb640 10 client.20547 handle_command_reply: tid=0 2024-08-28T19:43:39.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3126f4d740 1 client.20547 shutdown 2024-08-28T19:43:39.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3126f4d740 1 -- 172.21.15.16:0/619134612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 msgr2=0x5646909e4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:39.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3126f4d740 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x5646909e4960 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f3114018040 tx=0x7f3114011430 comp rx=0 tx=0).stop 2024-08-28T19:43:39.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3126f4d740 1 -- 172.21.15.16:0/619134612 shutdown_connections 2024-08-28T19:43:39.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3126f4d740 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.115:6834/1297102593,v1:172.21.15.115:6835/1297102593] conn(0x564690551da0 0x5646909eebb0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3126f4d740 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646905be070 0x5646909e4960 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3126f4d740 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56469059d390 0x564690973da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3126f4d740 1 --2- 172.21.15.16:0/619134612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646905951b0 0x5646909737c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3126f4d740 1 -- 172.21.15.16:0/619134612 >> 172.21.15.16:0/619134612 conn(0x5646905b2460 msgr2=0x5646909609e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:39.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.897+0000 7f3101ffb640 0 client.20547 ms_handle_reset on v2:172.21.15.115:6834/1297102593 2024-08-28T19:43:39.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f3126f4d740 1 -- 172.21.15.16:0/619134612 shutdown_connections 2024-08-28T19:43:39.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f3126f4d740 1 -- 172.21.15.16:0/619134612 wait complete. 2024-08-28T19:43:39.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f3126f4d740 20 client.20547 trim_cache size 0 max 16384 2024-08-28T19:43:39.911 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:43:39.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 -- 172.21.15.16:0/4062006008 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31080b4ca0 msgr2=0x7f31080b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:39.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31080b4ca0 0x7f31080b71b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3114005fd0 tx=0x7f3114009600 comp rx=0 tx=0).stop 2024-08-28T19:43:39.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 -- 172.21.15.16:0/4062006008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c075e20 msgr2=0x7f311c19c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:39.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c075e20 0x7f311c19c990 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f310402a930 tx=0x7f3104004270 comp rx=0 tx=0).stop 2024-08-28T19:43:39.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 -- 172.21.15.16:0/4062006008 shutdown_connections 2024-08-28T19:43:39.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f31080b4ca0 0x7f31080b71b0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311c10fee0 0x7f311c1a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c075e20 0x7f311c19c990 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 --2- 172.21.15.16:0/4062006008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f311c0754d0 0x7f311c19c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:39.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 -- 172.21.15.16:0/4062006008 >> 172.21.15.16:0/4062006008 conn(0x7f311c07b5f0 msgr2=0x7f311c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:39.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 -- 172.21.15.16:0/4062006008 shutdown_connections 2024-08-28T19:43:39.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:39.898+0000 7f31209d8640 1 -- 172.21.15.16:0/4062006008 wait complete. 2024-08-28T19:43:40.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-08-28T19:43:40.848 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:41.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:41 smithi016 ceph-mon[18556]: pgmap v1050: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 59 KiB/s wr, 16 op/s 2024-08-28T19:43:41.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:41 smithi028 ceph-mon[21769]: pgmap v1050: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 59 KiB/s wr, 16 op/s 2024-08-28T19:43:41.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.331+0000 7faf69092640 1 -- 172.21.15.16:0/3521212137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64109240 msgr2=0x7faf64109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:41.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.331+0000 7faf69092640 1 --2- 172.21.15.16:0/3521212137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64109240 0x7faf64109650 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009900 tx=0x7faf4c02a410 comp rx=0 tx=0).stop 2024-08-28T19:43:41.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.332+0000 7faf69092640 1 -- 172.21.15.16:0/3521212137 shutdown_connections 2024-08-28T19:43:41.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.332+0000 7faf69092640 1 --2- 172.21.15.16:0/3521212137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf64109b90 0x7faf641103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.332+0000 7faf69092640 1 --2- 172.21.15.16:0/3521212137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64109240 0x7faf64109650 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.332+0000 7faf69092640 1 --2- 172.21.15.16:0/3521212137 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf64103900 0x7faf64103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.332+0000 7faf69092640 1 -- 172.21.15.16:0/3521212137 >> 172.21.15.16:0/3521212137 conn(0x7faf6407b5f0 msgr2=0x7faf6407ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:41.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.332+0000 7faf69092640 1 -- 172.21.15.16:0/3521212137 shutdown_connections 2024-08-28T19:43:41.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.332+0000 7faf69092640 1 -- 172.21.15.16:0/3521212137 wait complete. 2024-08-28T19:43:41.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf69092640 1 Processor -- start 2024-08-28T19:43:41.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf69092640 1 -- start start 2024-08-28T19:43:41.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf69092640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf64103900 0x7faf6419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf69092640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64109240 0x7faf6419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf69092640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf64109b90 0x7faf6419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf69092640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faf64113850 con 0x7faf64109240 2024-08-28T19:43:41.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf69092640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf641136d0 con 0x7faf64109b90 2024-08-28T19:43:41.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf69092640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faf641139d0 con 0x7faf64103900 2024-08-28T19:43:41.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64109240 0x7faf6419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64109240 0x7faf6419a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50306/0 (socket says 172.21.15.16:50306) 2024-08-28T19:43:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf5bfff640 1 -- 172.21.15.16:0/2090337006 learned_addr learned my addr 172.21.15.16:0/2090337006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf6359e640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf64109b90 0x7faf6419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.333+0000 7faf62d9d640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf64103900 0x7faf6419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.334+0000 7faf5bfff640 1 -- 172.21.15.16:0/2090337006 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf64103900 msgr2=0x7faf6419a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.334+0000 7faf5bfff640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf64103900 0x7faf6419a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.334+0000 7faf5bfff640 1 -- 172.21.15.16:0/2090337006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf64109b90 msgr2=0x7faf6419fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.334+0000 7faf5bfff640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf64109b90 0x7faf6419fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.334+0000 7faf5bfff640 1 -- 172.21.15.16:0/2090337006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf641a0290 con 0x7faf64109240 2024-08-28T19:43:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.334+0000 7faf5bfff640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64109240 0x7faf6419a8c0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7faf4c006010 tx=0x7faf4c004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.334+0000 7faf60d99640 1 -- 172.21.15.16:0/2090337006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c03f030 con 0x7faf64109240 2024-08-28T19:43:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.334+0000 7faf69092640 1 -- 172.21.15.16:0/2090337006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf641a0520 con 0x7faf64109240 2024-08-28T19:43:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.334+0000 7faf60d99640 1 -- 172.21.15.16:0/2090337006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf4c003aa0 con 0x7faf64109240 2024-08-28T19:43:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.334+0000 7faf60d99640 1 -- 172.21.15.16:0/2090337006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c035070 con 0x7faf64109240 2024-08-28T19:43:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.334+0000 7faf69092640 1 -- 172.21.15.16:0/2090337006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf641a6ce0 con 0x7faf64109240 2024-08-28T19:43:41.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.336+0000 7faf6f607740 1 Processor -- start 2024-08-28T19:43:41.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf60d99640 1 -- 172.21.15.16:0/2090337006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7faf4c042030 con 0x7faf64109240 2024-08-28T19:43:41.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf6f607740 1 -- start start 2024-08-28T19:43:41.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf6f607740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c7c7070 0x555f0cb6c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf6f607740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c79e1b0 0x555f0cb6c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf6f607740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb6cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf6f607740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555f0cae3680 con 0x555f0c7a6390 2024-08-28T19:43:41.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf6f607740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555f0cad2c10 con 0x555f0c79e1b0 2024-08-28T19:43:41.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf6f607740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555f0cad6bc0 con 0x555f0c7c7070 2024-08-28T19:43:41.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb6cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb6cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50314/0 (socket says 172.21.15.16:50314) 2024-08-28T19:43:41.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf5bfff640 1 -- 172.21.15.16:0/3342011336 learned_addr learned my addr 172.21.15.16:0/3342011336 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:41.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf62d9d640 1 --2- 172.21.15.16:0/3342011336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c79e1b0 0x555f0cb6c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.337+0000 7faf6359e640 1 --2- 172.21.15.16:0/3342011336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c7c7070 0x555f0cb6c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf5bfff640 1 -- 172.21.15.16:0/3342011336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c7c7070 msgr2=0x555f0cb6c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:41.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf5bfff640 1 --2- 172.21.15.16:0/3342011336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c7c7070 0x555f0cb6c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf5bfff640 1 -- 172.21.15.16:0/3342011336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c79e1b0 msgr2=0x555f0cb6c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:41.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf5bfff640 1 --2- 172.21.15.16:0/3342011336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c79e1b0 0x555f0cb6c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf5bfff640 1 -- 172.21.15.16:0/3342011336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f0c832650 con 0x555f0c7a6390 2024-08-28T19:43:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf62d9d640 1 --2- 172.21.15.16:0/3342011336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c79e1b0 0x555f0cb6c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf6359e640 1 --2- 172.21.15.16:0/3342011336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c7c7070 0x555f0cb6c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf60d99640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faf500b4ca0 0x7faf500b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf5bfff640 1 --2- 172.21.15.16:0/3342011336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb6cd90 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7faf4c103060 tx=0x7faf4c10c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f899103faf9343a3 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf6359e640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faf500b4ca0 0x7faf500b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf60d99640 1 -- 172.21.15.16:0/2090337006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7faf4c105d70 con 0x7faf64109240 2024-08-28T19:43:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf597fa640 1 -- 172.21.15.16:0/3342011336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c0c8000 con 0x555f0c7a6390 2024-08-28T19:43:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf597fa640 1 -- 172.21.15.16:0/3342011336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf4c0cbc50 con 0x555f0c7a6390 2024-08-28T19:43:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.338+0000 7faf597fa640 1 -- 172.21.15.16:0/3342011336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c0cb8b0 con 0x555f0c7a6390 2024-08-28T19:43:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.339+0000 7faf6f607740 1 -- 172.21.15.16:0/3342011336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 msgr2=0x555f0cb6cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.339+0000 7faf6359e640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faf500b4ca0 0x7faf500b71b0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7faf5c005fd0 tx=0x7faf5c009500 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.339+0000 7faf6f607740 1 --2- 172.21.15.16:0/3342011336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb6cd90 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7faf4c103060 tx=0x7faf4c10c460 comp rx=0 tx=0).stop 2024-08-28T19:43:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.339+0000 7faf6f607740 1 -- 172.21.15.16:0/3342011336 shutdown_connections 2024-08-28T19:43:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.339+0000 7faf6f607740 1 --2- 172.21.15.16:0/3342011336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb6cd90 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.339+0000 7faf6f607740 1 --2- 172.21.15.16:0/3342011336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c79e1b0 0x555f0cb6c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.339+0000 7faf6f607740 1 --2- 172.21.15.16:0/3342011336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c7c7070 0x555f0cb6c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.339+0000 7faf6f607740 1 -- 172.21.15.16:0/3342011336 >> 172.21.15.16:0/3342011336 conn(0x555f0c7bb460 msgr2=0x555f0cb6a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.339+0000 7faf6f607740 1 -- 172.21.15.16:0/3342011336 shutdown_connections 2024-08-28T19:43:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.339+0000 7faf6f607740 1 -- 172.21.15.16:0/3342011336 wait complete. 2024-08-28T19:43:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf6f607740 1 -- start start 2024-08-28T19:43:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf6f607740 1 Processor -- start 2024-08-28T19:43:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf6f607740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 0x555f0cb77b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf6f607740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb78080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf6f607740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7c7070 0x555f0cb87f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf6f607740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555f0c935700 con 0x555f0c7a6390 2024-08-28T19:43:41.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf6f607740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555f0ca59750 con 0x555f0c7c7070 2024-08-28T19:43:41.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf6f607740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555f0cb7ecc0 con 0x555f0c79e1b0 2024-08-28T19:43:41.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf6f607740 10 client.0 _ll_register_callbacks cb 0x555f0ca88e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:41.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb78080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb78080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50318/0 (socket says 172.21.15.16:50318) 2024-08-28T19:43:41.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.340+0000 7faf5bfff640 1 -- 172.21.15.16:0/634178038 learned_addr learned my addr 172.21.15.16:0/634178038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:41.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf62d9d640 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 0x555f0cb77b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf6359e640 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7c7070 0x555f0cb87f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf5bfff640 1 -- 172.21.15.16:0/634178038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 msgr2=0x555f0cb77b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:41.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf5bfff640 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 0x555f0cb77b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf5bfff640 1 -- 172.21.15.16:0/634178038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7c7070 msgr2=0x555f0cb87f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:41.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf5bfff640 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7c7070 0x555f0cb87f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf5bfff640 1 -- 172.21.15.16:0/634178038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f0ca92900 con 0x555f0c7a6390 2024-08-28T19:43:41.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf62d9d640 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 0x555f0cb77b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf6359e640 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7c7070 0x555f0cb87f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf5bfff640 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb78080 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7faf4c1031a0 tx=0x7faf4c03b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fb0f14674bbda server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf4a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf4a7fc640 1 -- 172.21.15.16:0/634178038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c049070 con 0x555f0c7a6390 2024-08-28T19:43:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf4a7fc640 1 -- 172.21.15.16:0/634178038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7faf4c103370 con 0x555f0c7a6390 2024-08-28T19:43:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf6f607740 1 -- 172.21.15.16:0/634178038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf4c0cbc50 con 0x555f0c7a6390 2024-08-28T19:43:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf4a7fc640 1 -- 172.21.15.16:0/634178038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7faf4c0cba90 con 0x555f0c7a6390 2024-08-28T19:43:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf6f607740 10 client.20559 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf6f607740 1 -- 172.21.15.16:0/634178038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x555f0cb8aa90 con 0x555f0c7a6390 2024-08-28T19:43:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.341+0000 7faf4a7fc640 1 -- 172.21.15.16:0/634178038 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf4c0cbc50 con 0x555f0c7a6390 2024-08-28T19:43:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.342+0000 7faf4a7fc640 1 -- 172.21.15.16:0/634178038 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7faf4c0cc3a0 con 0x555f0c7a6390 2024-08-28T19:43:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.342+0000 7faf6f607740 10 client.20559 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.342+0000 7faf6f607740 10 client.20559 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:43:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.342+0000 7faf6f607740 20 client.20559 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.342+0000 7faf6f607740 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x555f0c7859f0 0x555f0cb8ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.342+0000 7faf6f607740 4 client.20559 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:43:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.342+0000 7faf6f607740 1 -- 172.21.15.16:0/634178038 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f0cae55d0 con 0x555f0c7859f0 2024-08-28T19:43:41.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.342+0000 7faf62d9d640 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x555f0c7859f0 0x555f0cb8ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.342+0000 7faf62d9d640 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x555f0c7859f0 0x555f0cb8ddf0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=54d2ed8c7488edb8 server_cookie=55bdd92b6f03679f in_seq=0 out_seq=0 2024-08-28T19:43:41.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.342+0000 7faf4a7fc640 10 client.20559 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:43:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf4a7fc640 1 -- 172.21.15.16:0/634178038 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555f0cae55d0 con 0x555f0c7859f0 2024-08-28T19:43:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf4a7fc640 10 client.20559 handle_command_reply: tid=0 2024-08-28T19:43:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf6f607740 1 client.20559 shutdown 2024-08-28T19:43:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf6f607740 1 -- 172.21.15.16:0/634178038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 msgr2=0x555f0cb78080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf6f607740 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb78080 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7faf4c1031a0 tx=0x7faf4c03b5d0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf6f607740 1 -- 172.21.15.16:0/634178038 shutdown_connections 2024-08-28T19:43:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf6f607740 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x555f0c7859f0 0x555f0cb8ddf0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf6f607740 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7c7070 0x555f0cb87f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf6f607740 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cb78080 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf4a7fc640 0 client.20559 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:43:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf6f607740 1 --2- 172.21.15.16:0/634178038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 0x555f0cb77b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.343+0000 7faf6f607740 1 -- 172.21.15.16:0/634178038 >> 172.21.15.16:0/634178038 conn(0x555f0c7bb460 msgr2=0x555f0cb6f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.344+0000 7faf6f607740 1 -- 172.21.15.16:0/634178038 shutdown_connections 2024-08-28T19:43:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.344+0000 7faf6f607740 1 -- 172.21.15.16:0/634178038 wait complete. 2024-08-28T19:43:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.344+0000 7faf6f607740 20 client.20559 trim_cache size 0 max 16384 2024-08-28T19:43:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.357+0000 7faf6f607740 1 Processor -- start 2024-08-28T19:43:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.357+0000 7faf6f607740 1 -- start start 2024-08-28T19:43:41.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.357+0000 7faf6f607740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7859f0 0x555f0cb6da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.357+0000 7faf6f607740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c79e1b0 0x555f0cb87e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.357+0000 7faf6f607740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c7a6390 0x555f0cb88360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.357+0000 7faf6f607740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faf641aeb70 con 0x555f0c79e1b0 2024-08-28T19:43:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.357+0000 7faf6f607740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf641b0740 con 0x555f0c7859f0 2024-08-28T19:43:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.357+0000 7faf6f607740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faf641af240 con 0x555f0c7a6390 2024-08-28T19:43:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.357+0000 7faf5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c79e1b0 0x555f0cb87e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c79e1b0 0x555f0cb87e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50334/0 (socket says 172.21.15.16:50334) 2024-08-28T19:43:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf5bfff640 1 -- 172.21.15.16:0/4196134551 learned_addr learned my addr 172.21.15.16:0/4196134551 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:41.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf62d9d640 1 --2- 172.21.15.16:0/4196134551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7859f0 0x555f0cb6da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf6359e640 1 --2- 172.21.15.16:0/4196134551 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c7a6390 0x555f0cb88360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf5bfff640 1 -- 172.21.15.16:0/4196134551 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c7a6390 msgr2=0x555f0cb88360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:41.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf5bfff640 1 --2- 172.21.15.16:0/4196134551 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c7a6390 0x555f0cb88360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf5bfff640 1 -- 172.21.15.16:0/4196134551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7859f0 msgr2=0x555f0cb6da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:41.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf5bfff640 1 --2- 172.21.15.16:0/4196134551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7859f0 0x555f0cb6da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf5bfff640 1 -- 172.21.15.16:0/4196134551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f0cb783f0 con 0x555f0c79e1b0 2024-08-28T19:43:41.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf5bfff640 1 --2- 172.21.15.16:0/4196134551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c79e1b0 0x555f0cb87e20 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0c81c0 tx=0x7faf4c109cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786ce6e989a5ffc4 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:41.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf58ff9640 1 -- 172.21.15.16:0/4196134551 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c049070 con 0x555f0c79e1b0 2024-08-28T19:43:41.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf58ff9640 1 -- 172.21.15.16:0/4196134551 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf4c109690 con 0x555f0c79e1b0 2024-08-28T19:43:41.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.358+0000 7faf58ff9640 1 -- 172.21.15.16:0/4196134551 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c0cbf10 con 0x555f0c79e1b0 2024-08-28T19:43:41.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.359+0000 7faf6f607740 1 -- 172.21.15.16:0/4196134551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c79e1b0 msgr2=0x555f0cb87e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:41.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.359+0000 7faf6f607740 1 --2- 172.21.15.16:0/4196134551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c79e1b0 0x555f0cb87e20 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0c81c0 tx=0x7faf4c109cb0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.359+0000 7faf6f607740 1 -- 172.21.15.16:0/4196134551 shutdown_connections 2024-08-28T19:43:41.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.359+0000 7faf6f607740 1 --2- 172.21.15.16:0/4196134551 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c7a6390 0x555f0cb88360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.359+0000 7faf6f607740 1 --2- 172.21.15.16:0/4196134551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c79e1b0 0x555f0cb87e20 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.359+0000 7faf6f607740 1 --2- 172.21.15.16:0/4196134551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7859f0 0x555f0cb6da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.359+0000 7faf6f607740 1 -- 172.21.15.16:0/4196134551 >> 172.21.15.16:0/4196134551 conn(0x555f0c7bb460 msgr2=0x555f0cb6d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:41.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.359+0000 7faf6f607740 1 -- 172.21.15.16:0/4196134551 shutdown_connections 2024-08-28T19:43:41.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.359+0000 7faf6f607740 1 -- 172.21.15.16:0/4196134551 wait complete. 2024-08-28T19:43:41.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6f607740 1 -- start start 2024-08-28T19:43:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6f607740 1 Processor -- start 2024-08-28T19:43:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6f607740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7859f0 0x555f0cb80270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6f607740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 0x555f0cb807f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6f607740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cbf7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6f607740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faf641b0070 con 0x555f0c7a6390 2024-08-28T19:43:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6f607740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf641af9a0 con 0x555f0c7859f0 2024-08-28T19:43:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6f607740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faf641acb70 con 0x555f0c79e1b0 2024-08-28T19:43:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6f607740 10 client.0 _ll_register_callbacks cb 0x7faf54004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:41.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cbf7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cbf7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50344/0 (socket says 172.21.15.16:50344) 2024-08-28T19:43:41.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf6359e640 1 -- 172.21.15.16:0/2346832055 learned_addr learned my addr 172.21.15.16:0/2346832055 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:41.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf62d9d640 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7859f0 0x555f0cb80270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.360+0000 7faf5bfff640 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 0x555f0cb807f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf6359e640 1 -- 172.21.15.16:0/2346832055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 msgr2=0x555f0cb807f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:41.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf6359e640 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 0x555f0cb807f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf6359e640 1 -- 172.21.15.16:0/2346832055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7859f0 msgr2=0x555f0cb80270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf6359e640 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7859f0 0x555f0cb80270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf6359e640 1 -- 172.21.15.16:0/2346832055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f0cb8e160 con 0x555f0c7a6390 2024-08-28T19:43:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf5bfff640 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 0x555f0cb807f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:43:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf6359e640 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cbf7590 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7faf5c0040c0 tx=0x7faf5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1d585fa2b83183 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf4a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf4a7fc640 1 -- 172.21.15.16:0/2346832055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c02f060 con 0x555f0c7a6390 2024-08-28T19:43:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf4a7fc640 1 -- 172.21.15.16:0/2346832055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7faf5c00f690 con 0x555f0c7a6390 2024-08-28T19:43:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf6f607740 1 -- 172.21.15.16:0/2346832055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf4c109690 con 0x555f0c7a6390 2024-08-28T19:43:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf4a7fc640 1 -- 172.21.15.16:0/2346832055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7faf5c036050 con 0x555f0c7a6390 2024-08-28T19:43:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf6f607740 10 client.20565 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf6f607740 1 -- 172.21.15.16:0/2346832055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x555f0cbf8ae0 con 0x555f0c7a6390 2024-08-28T19:43:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.361+0000 7faf4a7fc640 1 -- 172.21.15.16:0/2346832055 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf4c109690 con 0x555f0c7a6390 2024-08-28T19:43:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.362+0000 7faf4a7fc640 1 -- 172.21.15.16:0/2346832055 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7faf5c029070 con 0x555f0c7a6390 2024-08-28T19:43:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.362+0000 7faf6f607740 10 client.20565 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.362+0000 7faf6f607740 10 client.20565 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-08-28T19:43:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.362+0000 7faf6f607740 20 client.20565 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.362+0000 7faf6f607740 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x555f0c7c7070 0x555f0cbfb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.362+0000 7faf6f607740 4 client.20565 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:43:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.362+0000 7faf6f607740 1 -- 172.21.15.16:0/2346832055 --> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f0cae1070 con 0x555f0c7c7070 2024-08-28T19:43:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.362+0000 7faf62d9d640 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x555f0c7c7070 0x555f0cbfb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.362+0000 7faf62d9d640 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x555f0c7c7070 0x555f0cbfb300 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9cfe9f252d0c9dec server_cookie=cc2795a2c506ca5f in_seq=0 out_seq=0 2024-08-28T19:43:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.362+0000 7faf4a7fc640 10 client.20565 ms_handle_connect on v2:172.21.15.115:6836/4186796507 2024-08-28T19:43:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf4a7fc640 1 -- 172.21.15.16:0/2346832055 <== mds.3 v2:172.21.15.115:6836/4186796507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f0cae1070 con 0x555f0c7c7070 2024-08-28T19:43:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf4a7fc640 10 client.20565 handle_command_reply: tid=0 2024-08-28T19:43:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 1 client.20565 shutdown 2024-08-28T19:43:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 1 -- 172.21.15.16:0/2346832055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 msgr2=0x555f0cbf7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cbf7590 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7faf5c0040c0 tx=0x7faf5c004a40 comp rx=0 tx=0).stop 2024-08-28T19:43:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 1 -- 172.21.15.16:0/2346832055 shutdown_connections 2024-08-28T19:43:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.115:6836/4186796507,v1:172.21.15.115:6837/4186796507] conn(0x555f0c7c7070 0x555f0cbfb300 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f0c7a6390 0x555f0cbf7590 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555f0c79e1b0 0x555f0cb807f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf4a7fc640 0 client.20565 ms_handle_reset on v2:172.21.15.115:6836/4186796507 2024-08-28T19:43:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 1 --2- 172.21.15.16:0/2346832055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f0c7859f0 0x555f0cb80270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 1 -- 172.21.15.16:0/2346832055 >> 172.21.15.16:0/2346832055 conn(0x555f0c7bb460 msgr2=0x555f0cb6dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 1 -- 172.21.15.16:0/2346832055 shutdown_connections 2024-08-28T19:43:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 1 -- 172.21.15.16:0/2346832055 wait complete. 2024-08-28T19:43:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf6f607740 20 client.20565 trim_cache size 0 max 16384 2024-08-28T19:43:41.373 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:43:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.363+0000 7faf69092640 1 -- 172.21.15.16:0/2090337006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faf500b4ca0 msgr2=0x7faf500b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.364+0000 7faf69092640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faf500b4ca0 0x7faf500b71b0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7faf5c005fd0 tx=0x7faf5c009500 comp rx=0 tx=0).stop 2024-08-28T19:43:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.364+0000 7faf69092640 1 -- 172.21.15.16:0/2090337006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64109240 msgr2=0x7faf6419a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.364+0000 7faf69092640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64109240 0x7faf6419a8c0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7faf4c006010 tx=0x7faf4c004060 comp rx=0 tx=0).stop 2024-08-28T19:43:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.364+0000 7faf69092640 1 -- 172.21.15.16:0/2090337006 shutdown_connections 2024-08-28T19:43:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.364+0000 7faf69092640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7faf500b4ca0 0x7faf500b71b0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.364+0000 7faf69092640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf64109b90 0x7faf6419fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.364+0000 7faf69092640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64109240 0x7faf6419a8c0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.364+0000 7faf69092640 1 --2- 172.21.15.16:0/2090337006 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf64103900 0x7faf6419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.364+0000 7faf69092640 1 -- 172.21.15.16:0/2090337006 >> 172.21.15.16:0/2090337006 conn(0x7faf6407b5f0 msgr2=0x7faf64105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.364+0000 7faf69092640 1 -- 172.21.15.16:0/2090337006 shutdown_connections 2024-08-28T19:43:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:41.364+0000 7faf69092640 1 -- 172.21.15.16:0/2090337006 wait complete. 2024-08-28T19:43:41.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:41 smithi115 ceph-mon[22184]: pgmap v1050: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 59 KiB/s wr, 16 op/s 2024-08-28T19:43:42.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-08-28T19:43:42.535 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:43.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.000+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3148202571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91ec108160 msgr2=0x7f91ec108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:43.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.000+0000 7f91f3f4d640 1 --2- 172.21.15.16:0/3148202571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91ec108160 0x7f91ec108570 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009900 tx=0x7f91dc02a410 comp rx=0 tx=0).stop 2024-08-28T19:43:43.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.000+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3148202571 shutdown_connections 2024-08-28T19:43:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.000+0000 7f91f3f4d640 1 --2- 172.21.15.16:0/3148202571 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91ec108ab0 0x7f91ec10f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.000+0000 7f91f3f4d640 1 --2- 172.21.15.16:0/3148202571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91ec108160 0x7f91ec108570 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.000+0000 7f91f3f4d640 1 --2- 172.21.15.16:0/3148202571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec102820 0x7f91ec102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.000+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3148202571 >> 172.21.15.16:0/3148202571 conn(0x7f91ec0794e0 msgr2=0x7f91ec0798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.000+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3148202571 shutdown_connections 2024-08-28T19:43:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.000+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3148202571 wait complete. 2024-08-28T19:43:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f3f4d640 1 Processor -- start 2024-08-28T19:43:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f3f4d640 1 -- start start 2024-08-28T19:43:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f3f4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec102820 0x7f91ec19b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f3f4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91ec108160 0x7f91ec19b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f3f4d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91ec108ab0 0x7f91ec19bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f3f4d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec112700 con 0x7f91ec108160 2024-08-28T19:43:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f3f4d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec112580 con 0x7f91ec102820 2024-08-28T19:43:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f3f4d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec112880 con 0x7f91ec108ab0 2024-08-28T19:43:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f14e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91ec108160 0x7f91ec19b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f14e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91ec108160 0x7f91ec19b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50366/0 (socket says 172.21.15.16:50366) 2024-08-28T19:43:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f14e8640 1 -- 172.21.15.16:0/3187888648 learned_addr learned my addr 172.21.15.16:0/3187888648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.001+0000 7f91f1ce9640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec102820 0x7f91ec19b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91f24ea640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91ec108ab0 0x7f91ec19bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91f14e8640 1 -- 172.21.15.16:0/3187888648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91ec108ab0 msgr2=0x7f91ec19bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91f14e8640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91ec108ab0 0x7f91ec19bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91f14e8640 1 -- 172.21.15.16:0/3187888648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec102820 msgr2=0x7f91ec19b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91f14e8640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec102820 0x7f91ec19b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91f14e8640 1 -- 172.21.15.16:0/3187888648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ec1a64a0 con 0x7f91ec108160 2024-08-28T19:43:43.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91f24ea640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91ec108ab0 0x7f91ec19bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:43.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91f14e8640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91ec108160 0x7f91ec19b750 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f91dc032fd0 tx=0x7f91dc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:43.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91e2ffd640 1 -- 172.21.15.16:0/3187888648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc03f030 con 0x7f91ec108160 2024-08-28T19:43:43.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3187888648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ec1a6730 con 0x7f91ec108160 2024-08-28T19:43:43.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91e2ffd640 1 -- 172.21.15.16:0/3187888648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91dc004490 con 0x7f91ec108160 2024-08-28T19:43:43.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91e2ffd640 1 -- 172.21.15.16:0/3187888648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc035070 con 0x7f91ec108160 2024-08-28T19:43:43.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.002+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3187888648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ec1a6c10 con 0x7f91ec108160 2024-08-28T19:43:43.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.005+0000 7f91e2ffd640 1 -- 172.21.15.16:0/3187888648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f91dc042030 con 0x7f91ec108160 2024-08-28T19:43:43.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.005+0000 7f91fa4c2740 1 Processor -- start 2024-08-28T19:43:43.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.005+0000 7f91fa4c2740 1 -- start start 2024-08-28T19:43:43.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.005+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d457070 0x55923d7fc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.005+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d7fc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.005+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d436390 0x55923d7fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.005+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55923d773680 con 0x55923d42e1b0 2024-08-28T19:43:43.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.005+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55923d762c10 con 0x55923d436390 2024-08-28T19:43:43.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.005+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55923d766bc0 con 0x55923d457070 2024-08-28T19:43:43.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.005+0000 7f91f1ce9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d7fc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f1ce9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d7fc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50376/0 (socket says 172.21.15.16:50376) 2024-08-28T19:43:43.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f1ce9640 1 -- 172.21.15.16:0/4183471297 learned_addr learned my addr 172.21.15.16:0/4183471297 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:43.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91e2ffd640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91d80b4cd0 0x7f91d80b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f14e8640 1 --2- 172.21.15.16:0/4183471297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d436390 0x55923d7fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f24ea640 1 --2- 172.21.15.16:0/4183471297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d457070 0x55923d7fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f24ea640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91d80b4cd0 0x7f91d80b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f1ce9640 1 -- 172.21.15.16:0/4183471297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d457070 msgr2=0x55923d7fc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f1ce9640 1 --2- 172.21.15.16:0/4183471297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d457070 0x55923d7fc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f1ce9640 1 -- 172.21.15.16:0/4183471297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d436390 msgr2=0x55923d7fcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f1ce9640 1 --2- 172.21.15.16:0/4183471297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d436390 0x55923d7fcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f1ce9640 1 -- 172.21.15.16:0/4183471297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55923d4c2650 con 0x55923d42e1b0 2024-08-28T19:43:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f14e8640 1 --2- 172.21.15.16:0/4183471297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d436390 0x55923d7fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91e2ffd640 1 -- 172.21.15.16:0/3187888648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f91dc105de0 con 0x7f91ec108160 2024-08-28T19:43:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f1ce9640 1 --2- 172.21.15.16:0/4183471297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d7fc850 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f91e800bcc0 tx=0x7f91e80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4cd6e7642ec0cf0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91d7fff640 1 -- 172.21.15.16:0/4183471297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8005220 con 0x55923d42e1b0 2024-08-28T19:43:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91d7fff640 1 -- 172.21.15.16:0/4183471297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91e8004c50 con 0x55923d42e1b0 2024-08-28T19:43:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91f24ea640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91d80b4cd0 0x7f91d80b71e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f91e4005c40 tx=0x7f91e4009280 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.006+0000 7f91d7fff640 1 -- 172.21.15.16:0/4183471297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e80116a0 con 0x55923d42e1b0 2024-08-28T19:43:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.007+0000 7f91fa4c2740 1 -- 172.21.15.16:0/4183471297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 msgr2=0x55923d7fc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.007+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/4183471297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d7fc850 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f91e800bcc0 tx=0x7f91e80107a0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.007+0000 7f91fa4c2740 1 -- 172.21.15.16:0/4183471297 shutdown_connections 2024-08-28T19:43:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.007+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/4183471297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d436390 0x55923d7fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.007+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/4183471297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d7fc850 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.007+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/4183471297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d457070 0x55923d7fc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.007+0000 7f91fa4c2740 1 -- 172.21.15.16:0/4183471297 >> 172.21.15.16:0/4183471297 conn(0x55923d44b460 msgr2=0x55923d7fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.007+0000 7f91fa4c2740 1 -- 172.21.15.16:0/4183471297 shutdown_connections 2024-08-28T19:43:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.007+0000 7f91fa4c2740 1 -- 172.21.15.16:0/4183471297 wait complete. 2024-08-28T19:43:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.008+0000 7f91fa4c2740 1 -- start start 2024-08-28T19:43:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.008+0000 7f91fa4c2740 1 Processor -- start 2024-08-28T19:43:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.008+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d42e1b0 0x55923d80be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.008+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d436390 0x55923d80c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d457070 0x55923d818150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55923d5c5700 con 0x55923d436390 2024-08-28T19:43:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55923d6e9750 con 0x55923d42e1b0 2024-08-28T19:43:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55923d80ee20 con 0x55923d457070 2024-08-28T19:43:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91fa4c2740 10 client.0 _ll_register_callbacks cb 0x55923d718e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f14e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d436390 0x55923d80c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f14e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d436390 0x55923d80c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50378/0 (socket says 172.21.15.16:50378) 2024-08-28T19:43:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f14e8640 1 -- 172.21.15.16:0/2043760188 learned_addr learned my addr 172.21.15.16:0/2043760188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f24ea640 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d457070 0x55923d818150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f1ce9640 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d42e1b0 0x55923d80be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f14e8640 1 -- 172.21.15.16:0/2043760188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d457070 msgr2=0x55923d818150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f14e8640 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d457070 0x55923d818150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f14e8640 1 -- 172.21.15.16:0/2043760188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d42e1b0 msgr2=0x55923d80be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f14e8640 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d42e1b0 0x55923d80be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f14e8640 1 -- 172.21.15.16:0/2043760188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55923d722900 con 0x55923d436390 2024-08-28T19:43:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f24ea640 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d457070 0x55923d818150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91f14e8640 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d436390 0x55923d80c3b0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0c8180 tx=0x7f91dc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1952107e5029dbb1 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91d57fa640 1 -- 172.21.15.16:0/2043760188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc049070 con 0x55923d436390 2024-08-28T19:43:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91fa4c2740 1 -- 172.21.15.16:0/2043760188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91e8004c50 con 0x55923d436390 2024-08-28T19:43:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91d57fa640 1 -- 172.21.15.16:0/2043760188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f91dc0d0590 con 0x55923d436390 2024-08-28T19:43:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.009+0000 7f91d57fa640 1 -- 172.21.15.16:0/2043760188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f91dc0cba50 con 0x55923d436390 2024-08-28T19:43:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.010+0000 7f91fa4c2740 10 client.20577 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.010+0000 7f91fa4c2740 1 -- 172.21.15.16:0/2043760188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x55923d81ac10 con 0x55923d436390 2024-08-28T19:43:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.010+0000 7f91d57fa640 1 -- 172.21.15.16:0/2043760188 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91e8004c50 con 0x55923d436390 2024-08-28T19:43:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.010+0000 7f91d57fa640 1 -- 172.21.15.16:0/2043760188 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7f91dc0cc360 con 0x55923d436390 2024-08-28T19:43:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.010+0000 7f91fa4c2740 10 client.20577 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.010+0000 7f91fa4c2740 10 client.20577 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:43:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.010+0000 7f91fa4c2740 20 client.20577 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.010+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55923d4159f0 0x55923d81e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.010+0000 7f91fa4c2740 4 client.20577 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:43:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.010+0000 7f91fa4c2740 1 -- 172.21.15.16:0/2043760188 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55923d7755d0 con 0x55923d4159f0 2024-08-28T19:43:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.010+0000 7f91f1ce9640 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55923d4159f0 0x55923d81e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.011+0000 7f91f1ce9640 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55923d4159f0 0x55923d81e150 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=af8886341a262846 server_cookie=b1282033be16adf1 in_seq=0 out_seq=0 2024-08-28T19:43:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.011+0000 7f91d57fa640 10 client.20577 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:43:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.011+0000 7f91d57fa640 1 -- 172.21.15.16:0/2043760188 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55923d7755d0 con 0x55923d4159f0 2024-08-28T19:43:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.011+0000 7f91d57fa640 10 client.20577 handle_command_reply: tid=0 2024-08-28T19:43:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.011+0000 7f91fa4c2740 1 client.20577 shutdown 2024-08-28T19:43:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91fa4c2740 1 -- 172.21.15.16:0/2043760188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d436390 msgr2=0x55923d80c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d436390 0x55923d80c3b0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0c8180 tx=0x7f91dc009260 comp rx=0 tx=0).stop 2024-08-28T19:43:43.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91fa4c2740 1 -- 172.21.15.16:0/2043760188 shutdown_connections 2024-08-28T19:43:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55923d4159f0 0x55923d81e150 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d457070 0x55923d818150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d436390 0x55923d80c3b0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91d57fa640 0 client.20577 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:43:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/2043760188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d42e1b0 0x55923d80be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91fa4c2740 1 -- 172.21.15.16:0/2043760188 >> 172.21.15.16:0/2043760188 conn(0x55923d44b460 msgr2=0x55923d7ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91fa4c2740 1 -- 172.21.15.16:0/2043760188 shutdown_connections 2024-08-28T19:43:43.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91fa4c2740 1 -- 172.21.15.16:0/2043760188 wait complete. 2024-08-28T19:43:43.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.012+0000 7f91fa4c2740 20 client.20577 trim_cache size 0 max 16384 2024-08-28T19:43:43.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91fa4c2740 1 Processor -- start 2024-08-28T19:43:43.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91fa4c2740 1 -- start start 2024-08-28T19:43:43.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d4159f0 0x55923d7ffac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d818050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d436390 0x55923d7f9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec1aeb90 con 0x55923d42e1b0 2024-08-28T19:43:43.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec1b0760 con 0x55923d4159f0 2024-08-28T19:43:43.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec1af260 con 0x55923d436390 2024-08-28T19:43:43.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91f14e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d818050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91f14e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d818050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50386/0 (socket says 172.21.15.16:50386) 2024-08-28T19:43:43.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91f14e8640 1 -- 172.21.15.16:0/1738366002 learned_addr learned my addr 172.21.15.16:0/1738366002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:43.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91f24ea640 1 --2- 172.21.15.16:0/1738366002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d436390 0x55923d7f9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91f1ce9640 1 --2- 172.21.15.16:0/1738366002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d4159f0 0x55923d7ffac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91f14e8640 1 -- 172.21.15.16:0/1738366002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d436390 msgr2=0x55923d7f9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:43.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91f14e8640 1 --2- 172.21.15.16:0/1738366002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d436390 0x55923d7f9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91f14e8640 1 -- 172.21.15.16:0/1738366002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d4159f0 msgr2=0x55923d7ffac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91f14e8640 1 --2- 172.21.15.16:0/1738366002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d4159f0 0x55923d7ffac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.026+0000 7f91f14e8640 1 -- 172.21.15.16:0/1738366002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55923d80cda0 con 0x55923d42e1b0 2024-08-28T19:43:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.027+0000 7f91f14e8640 1 --2- 172.21.15.16:0/1738366002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d818050 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0c9370 tx=0x7f91dc032a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecc46db2f52a572 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.027+0000 7f91d77fe640 1 -- 172.21.15.16:0/1738366002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc049070 con 0x55923d42e1b0 2024-08-28T19:43:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.027+0000 7f91d77fe640 1 -- 172.21.15.16:0/1738366002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91dc0cc800 con 0x55923d42e1b0 2024-08-28T19:43:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.027+0000 7f91d77fe640 1 -- 172.21.15.16:0/1738366002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0d3160 con 0x55923d42e1b0 2024-08-28T19:43:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.028+0000 7f91fa4c2740 1 -- 172.21.15.16:0/1738366002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 msgr2=0x55923d818050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.028+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/1738366002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d818050 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0c9370 tx=0x7f91dc032a20 comp rx=0 tx=0).stop 2024-08-28T19:43:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.028+0000 7f91fa4c2740 1 -- 172.21.15.16:0/1738366002 shutdown_connections 2024-08-28T19:43:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.028+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/1738366002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d436390 0x55923d7f9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.028+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/1738366002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d818050 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.028+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/1738366002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d4159f0 0x55923d7ffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.028+0000 7f91fa4c2740 1 -- 172.21.15.16:0/1738366002 >> 172.21.15.16:0/1738366002 conn(0x55923d44b460 msgr2=0x55923d7ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.028+0000 7f91fa4c2740 1 -- 172.21.15.16:0/1738366002 shutdown_connections 2024-08-28T19:43:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.028+0000 7f91fa4c2740 1 -- 172.21.15.16:0/1738366002 wait complete. 2024-08-28T19:43:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.028+0000 7f91fa4c2740 1 -- start start 2024-08-28T19:43:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.028+0000 7f91fa4c2740 1 Processor -- start 2024-08-28T19:43:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d4159f0 0x55923d814730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d810bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91fa4c2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d436390 0x55923d811130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec1b0090 con 0x55923d42e1b0 2024-08-28T19:43:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec1af9c0 con 0x55923d4159f0 2024-08-28T19:43:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91fa4c2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec1acb90 con 0x55923d436390 2024-08-28T19:43:43.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91fa4c2740 10 client.0 _ll_register_callbacks cb 0x7f91dc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:43.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91f14e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d810bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91f14e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d810bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50390/0 (socket says 172.21.15.16:50390) 2024-08-28T19:43:43.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91f14e8640 1 -- 172.21.15.16:0/3222234464 learned_addr learned my addr 172.21.15.16:0/3222234464 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:43.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91f1ce9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d4159f0 0x55923d814730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91f24ea640 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d436390 0x55923d811130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91f14e8640 1 -- 172.21.15.16:0/3222234464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d436390 msgr2=0x55923d811130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:43.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91f14e8640 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d436390 0x55923d811130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91f14e8640 1 -- 172.21.15.16:0/3222234464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d4159f0 msgr2=0x55923d814730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:43.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91f14e8640 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d4159f0 0x55923d814730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91f14e8640 1 -- 172.21.15.16:0/3222234464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55923d811770 con 0x55923d42e1b0 2024-08-28T19:43:43.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.029+0000 7f91f1ce9640 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d4159f0 0x55923d814730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:43.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91f14e8640 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d810bf0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f91dc1032a0 tx=0x7f91dc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1034e5e7c0f4582 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:43.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:43.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91d57fa640 1 -- 172.21.15.16:0/3222234464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0e5020 con 0x55923d42e1b0 2024-08-28T19:43:43.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91d57fa640 1 -- 172.21.15.16:0/3222234464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f91dc0d4f90 con 0x55923d42e1b0 2024-08-28T19:43:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91fa4c2740 1 -- 172.21.15.16:0/3222234464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91dc0cc800 con 0x55923d42e1b0 2024-08-28T19:43:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91d57fa640 1 -- 172.21.15.16:0/3222234464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f91dc049070 con 0x55923d42e1b0 2024-08-28T19:43:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91fa4c2740 10 client.20583 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91fa4c2740 1 -- 172.21.15.16:0/3222234464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x55923d811770 con 0x55923d42e1b0 2024-08-28T19:43:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91d57fa640 1 -- 172.21.15.16:0/3222234464 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91dc0cc800 con 0x55923d42e1b0 2024-08-28T19:43:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91d57fa640 1 -- 172.21.15.16:0/3222234464 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7f91dc0c7a70 con 0x55923d42e1b0 2024-08-28T19:43:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91fa4c2740 10 client.20583 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:43.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91fa4c2740 10 client.20583 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-08-28T19:43:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91fa4c2740 20 client.20583 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55923d457070 0x55923d88bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91fa4c2740 4 client.20583 mds_command: new command op to 24635 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:43:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.030+0000 7f91fa4c2740 1 -- 172.21.15.16:0/3222234464 --> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55923d771070 con 0x55923d457070 2024-08-28T19:43:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.031+0000 7f91f1ce9640 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55923d457070 0x55923d88bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.031+0000 7f91f1ce9640 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55923d457070 0x55923d88bce0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4ce23158f8a42430 server_cookie=ae8d3e363bac5dd8 in_seq=0 out_seq=0 2024-08-28T19:43:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.031+0000 7f91d57fa640 10 client.20583 ms_handle_connect on v2:172.21.15.28:6836/2888152030 2024-08-28T19:43:43.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.031+0000 7f91d57fa640 1 -- 172.21.15.16:0/3222234464 <== mds.2 v2:172.21.15.28:6836/2888152030 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55923d771070 con 0x55923d457070 2024-08-28T19:43:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.031+0000 7f91d57fa640 10 client.20583 handle_command_reply: tid=0 2024-08-28T19:43:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.031+0000 7f91fa4c2740 1 client.20583 shutdown 2024-08-28T19:43:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.031+0000 7f91fa4c2740 1 -- 172.21.15.16:0/3222234464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 msgr2=0x55923d810bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.031+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d810bf0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f91dc1032a0 tx=0x7f91dc000bc0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91fa4c2740 1 -- 172.21.15.16:0/3222234464 shutdown_connections 2024-08-28T19:43:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.28:6836/2888152030,v1:172.21.15.28:6837/2888152030] conn(0x55923d457070 0x55923d88bce0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55923d436390 0x55923d811130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55923d42e1b0 0x55923d810bf0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91fa4c2740 1 --2- 172.21.15.16:0/3222234464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55923d4159f0 0x55923d814730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91d57fa640 0 client.20583 ms_handle_reset on v2:172.21.15.28:6836/2888152030 2024-08-28T19:43:43.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91fa4c2740 1 -- 172.21.15.16:0/3222234464 >> 172.21.15.16:0/3222234464 conn(0x55923d44b460 msgr2=0x55923d7f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:43.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91fa4c2740 1 -- 172.21.15.16:0/3222234464 shutdown_connections 2024-08-28T19:43:43.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91fa4c2740 1 -- 172.21.15.16:0/3222234464 wait complete. 2024-08-28T19:43:43.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91fa4c2740 20 client.20583 trim_cache size 0 max 16384 2024-08-28T19:43:43.049 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:43:43.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3187888648 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91d80b4cd0 msgr2=0x7f91d80b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:43.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91f3f4d640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91d80b4cd0 0x7f91d80b71e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f91e4005c40 tx=0x7f91e4009280 comp rx=0 tx=0).stop 2024-08-28T19:43:43.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3187888648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91ec108160 msgr2=0x7f91ec19b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:43.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91f3f4d640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91ec108160 0x7f91ec19b750 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f91dc032fd0 tx=0x7f91dc004270 comp rx=0 tx=0).stop 2024-08-28T19:43:43.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3187888648 shutdown_connections 2024-08-28T19:43:43.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91f3f4d640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91ec108ab0 0x7f91ec19bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91f3f4d640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91ec108160 0x7f91ec19b750 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91f3f4d640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f91d80b4cd0 0x7f91d80b71e0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91f3f4d640 1 --2- 172.21.15.16:0/3187888648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec102820 0x7f91ec19b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:43.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.032+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3187888648 >> 172.21.15.16:0/3187888648 conn(0x7f91ec0794e0 msgr2=0x7f91ec105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:43.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.033+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3187888648 shutdown_connections 2024-08-28T19:43:43.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:43.033+0000 7f91f3f4d640 1 -- 172.21.15.16:0/3187888648 wait complete. 2024-08-28T19:43:43.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:43 smithi016 ceph-mon[18556]: pgmap v1051: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 101 KiB/s wr, 21 op/s 2024-08-28T19:43:43.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:43 smithi028 ceph-mon[21769]: pgmap v1051: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 101 KiB/s wr, 21 op/s 2024-08-28T19:43:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:43 smithi115 ceph-mon[22184]: pgmap v1051: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 101 KiB/s wr, 21 op/s 2024-08-28T19:43:43.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-08-28T19:43:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:44.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.544+0000 7f182ef03640 1 -- 172.21.15.16:0/688545148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f182810dc10 msgr2=0x7f1828110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:44.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.544+0000 7f182ef03640 1 --2- 172.21.15.16:0/688545148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f182810dc10 0x7f1828110120 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f1820009ca0 tx=0x7f182002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.544+0000 7f182ef03640 1 -- 172.21.15.16:0/688545148 shutdown_connections 2024-08-28T19:43:44.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.544+0000 7f182ef03640 1 --2- 172.21.15.16:0/688545148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f182810dc10 0x7f1828110120 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.544+0000 7f182ef03640 1 --2- 172.21.15.16:0/688545148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1828105090 0x7f182810d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.544+0000 7f182ef03640 1 --2- 172.21.15.16:0/688545148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1828104660 0x7f1828104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.544+0000 7f182ef03640 1 -- 172.21.15.16:0/688545148 >> 172.21.15.16:0/688545148 conn(0x7f1828079510 msgr2=0x7f1828079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:44.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.544+0000 7f182ef03640 1 -- 172.21.15.16:0/688545148 shutdown_connections 2024-08-28T19:43:44.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.544+0000 7f182ef03640 1 -- 172.21.15.16:0/688545148 wait complete. 2024-08-28T19:43:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.545+0000 7f182ef03640 1 Processor -- start 2024-08-28T19:43:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.545+0000 7f182ef03640 1 -- start start 2024-08-28T19:43:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.545+0000 7f182ef03640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1828104660 0x7f18281981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.545+0000 7f182ef03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1828105090 0x7f1828198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.545+0000 7f182ef03640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f182810dc10 0x7f182819d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.545+0000 7f182ef03640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18281116a0 con 0x7f1828104660 2024-08-28T19:43:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.545+0000 7f182ef03640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1828111520 con 0x7f1828105090 2024-08-28T19:43:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.545+0000 7f182ef03640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1828111820 con 0x7f182810dc10 2024-08-28T19:43:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.545+0000 7f182cc9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1828104660 0x7f18281981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.545+0000 7f182cc9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1828104660 0x7f18281981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35912/0 (socket says 172.21.15.16:35912) 2024-08-28T19:43:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.545+0000 7f182cc9f640 1 -- 172.21.15.16:0/1384614823 learned_addr learned my addr 172.21.15.16:0/1384614823 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f182d4a0640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f182810dc10 0x7f182819d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f1827fff640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1828105090 0x7f1828198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f182cc9f640 1 -- 172.21.15.16:0/1384614823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f182810dc10 msgr2=0x7f182819d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f182cc9f640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f182810dc10 0x7f182819d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f182cc9f640 1 -- 172.21.15.16:0/1384614823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1828105090 msgr2=0x7f1828198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f182cc9f640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1828105090 0x7f1828198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f182cc9f640 1 -- 172.21.15.16:0/1384614823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182819e120 con 0x7f1828104660 2024-08-28T19:43:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f182d4a0640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f182810dc10 0x7f182819d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f1827fff640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1828105090 0x7f1828198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f182cc9f640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1828104660 0x7f18281981b0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f181800e9f0 tx=0x7f181800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f1825ffb640 1 -- 172.21.15.16:0/1384614823 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181800cd80 con 0x7f1828104660 2024-08-28T19:43:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f182ef03640 1 -- 172.21.15.16:0/1384614823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f182819e410 con 0x7f1828104660 2024-08-28T19:43:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f1825ffb640 1 -- 172.21.15.16:0/1384614823 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18180108e0 con 0x7f1828104660 2024-08-28T19:43:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f1825ffb640 1 -- 172.21.15.16:0/1384614823 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818018980 con 0x7f1828104660 2024-08-28T19:43:44.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.546+0000 7f182ef03640 1 -- 172.21.15.16:0/1384614823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18281a4b70 con 0x7f1828104660 2024-08-28T19:43:44.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.549+0000 7f1835478740 1 Processor -- start 2024-08-28T19:43:44.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f1835478740 1 -- start start 2024-08-28T19:43:44.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f1835478740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75daf070 0x564f76154310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f1835478740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d861b0 0x564f76154850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f1835478740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76154d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f1835478740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f760cb680 con 0x564f75d8e390 2024-08-28T19:43:44.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f1835478740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564f760bac10 con 0x564f75d861b0 2024-08-28T19:43:44.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f1835478740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564f760bebc0 con 0x564f75daf070 2024-08-28T19:43:44.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f1825ffb640 1 -- 172.21.15.16:0/1384614823 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1818010430 con 0x7f1828104660 2024-08-28T19:43:44.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f1827fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f182d4a0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75daf070 0x564f76154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f1827fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35914/0 (socket says 172.21.15.16:35914) 2024-08-28T19:43:44.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f1827fff640 1 -- 172.21.15.16:0/3952796423 learned_addr learned my addr 172.21.15.16:0/3952796423 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:44.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.550+0000 7f182cc9f640 1 --2- 172.21.15.16:0/3952796423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d861b0 0x564f76154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f1827fff640 1 -- 172.21.15.16:0/3952796423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75daf070 msgr2=0x564f76154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:44.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f1827fff640 1 --2- 172.21.15.16:0/3952796423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75daf070 0x564f76154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f1827fff640 1 -- 172.21.15.16:0/3952796423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d861b0 msgr2=0x564f76154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:44.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f1827fff640 1 --2- 172.21.15.16:0/3952796423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d861b0 0x564f76154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f1827fff640 1 -- 172.21.15.16:0/3952796423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f75e1a650 con 0x564f75d8e390 2024-08-28T19:43:44.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f1827fff640 1 --2- 172.21.15.16:0/3952796423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76154d90 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f181000f930 tx=0x7f1810005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28aaae2d98784eaa server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:44.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f181effd640 1 -- 172.21.15.16:0/3952796423 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1810003bd0 con 0x564f75d8e390 2024-08-28T19:43:44.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f181effd640 1 -- 172.21.15.16:0/3952796423 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f181000f4d0 con 0x564f75d8e390 2024-08-28T19:43:44.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f181effd640 1 -- 172.21.15.16:0/3952796423 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181001a410 con 0x564f75d8e390 2024-08-28T19:43:44.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f1825ffb640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18140b4ca0 0x7f18140b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f182d4a0640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18140b4ca0 0x7f18140b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.551+0000 7f1825ffb640 1 -- 172.21.15.16:0/1384614823 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f18180e5d80 con 0x7f1828104660 2024-08-28T19:43:44.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f1835478740 1 -- 172.21.15.16:0/3952796423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 msgr2=0x564f76154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:44.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f182d4a0640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18140b4ca0 0x7f18140b71b0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1820004af0 tx=0x7f1820005c30 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:44.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f1835478740 1 --2- 172.21.15.16:0/3952796423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76154d90 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f181000f930 tx=0x7f1810005f90 comp rx=0 tx=0).stop 2024-08-28T19:43:44.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f1835478740 1 -- 172.21.15.16:0/3952796423 shutdown_connections 2024-08-28T19:43:44.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f1835478740 1 --2- 172.21.15.16:0/3952796423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76154d90 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f1835478740 1 --2- 172.21.15.16:0/3952796423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d861b0 0x564f76154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f1835478740 1 --2- 172.21.15.16:0/3952796423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75daf070 0x564f76154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f1835478740 1 -- 172.21.15.16:0/3952796423 >> 172.21.15.16:0/3952796423 conn(0x564f75da3460 msgr2=0x564f76152200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:44.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f1835478740 1 -- 172.21.15.16:0/3952796423 shutdown_connections 2024-08-28T19:43:44.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f1835478740 1 -- 172.21.15.16:0/3952796423 wait complete. 2024-08-28T19:43:44.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f1835478740 1 -- start start 2024-08-28T19:43:44.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.552+0000 7f1835478740 1 Processor -- start 2024-08-28T19:43:44.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1835478740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d861b0 0x564f7615fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1835478740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f7615ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1835478740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75daf070 0x564f7616bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1835478740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f75f1d700 con 0x564f75d8e390 2024-08-28T19:43:44.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1835478740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564f76041750 con 0x564f75daf070 2024-08-28T19:43:44.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1835478740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564f76162af0 con 0x564f75d861b0 2024-08-28T19:43:44.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1835478740 10 client.0 _ll_register_callbacks cb 0x564f76070e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:44.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1827fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f7615ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1827fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f7615ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35918/0 (socket says 172.21.15.16:35918) 2024-08-28T19:43:44.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1827fff640 1 -- 172.21.15.16:0/2563698919 learned_addr learned my addr 172.21.15.16:0/2563698919 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:44.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f182d4a0640 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75daf070 0x564f7616bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f182cc9f640 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d861b0 0x564f7615fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1827fff640 1 -- 172.21.15.16:0/2563698919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d861b0 msgr2=0x564f7615fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:44.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1827fff640 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d861b0 0x564f7615fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1827fff640 1 -- 172.21.15.16:0/2563698919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75daf070 msgr2=0x564f7616bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:44.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1827fff640 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75daf070 0x564f7616bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.553+0000 7f1827fff640 1 -- 172.21.15.16:0/2563698919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f7607a900 con 0x564f75d8e390 2024-08-28T19:43:44.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.554+0000 7f1827fff640 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f7615ff60 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f1810003e80 tx=0x7f1810003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5be7f758457855 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:44.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.554+0000 7f1807fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:44.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.554+0000 7f1807fff640 1 -- 172.21.15.16:0/2563698919 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1810005350 con 0x564f75d8e390 2024-08-28T19:43:44.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.554+0000 7f1807fff640 1 -- 172.21.15.16:0/2563698919 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f18100054f0 con 0x564f75d8e390 2024-08-28T19:43:44.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.554+0000 7f1835478740 1 -- 172.21.15.16:0/2563698919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f181000f4d0 con 0x564f75d8e390 2024-08-28T19:43:44.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.554+0000 7f1807fff640 1 -- 172.21.15.16:0/2563698919 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f1810035050 con 0x564f75d8e390 2024-08-28T19:43:44.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.554+0000 7f1835478740 10 client.20595 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:44.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.554+0000 7f1835478740 1 -- 172.21.15.16:0/2563698919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x564f7616e860 con 0x564f75d8e390 2024-08-28T19:43:44.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.554+0000 7f1807fff640 1 -- 172.21.15.16:0/2563698919 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f181000f4d0 con 0x564f75d8e390 2024-08-28T19:43:44.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.554+0000 7f1807fff640 1 -- 172.21.15.16:0/2563698919 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7f1810004a60 con 0x564f75d8e390 2024-08-28T19:43:44.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.555+0000 7f1835478740 10 client.20595 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:44.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.555+0000 7f1835478740 10 client.20595 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:43:44.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.555+0000 7f1835478740 20 client.20595 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:44.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.555+0000 7f1835478740 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x564f75d6d9f0 0x564f76171c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.555+0000 7f1835478740 4 client.20595 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:43:44.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.555+0000 7f1835478740 1 -- 172.21.15.16:0/2563698919 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f760cd5d0 con 0x564f75d6d9f0 2024-08-28T19:43:44.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.555+0000 7f182cc9f640 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x564f75d6d9f0 0x564f76171c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.555+0000 7f182cc9f640 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x564f75d6d9f0 0x564f76171c20 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c5ca92cbd5b5d2a server_cookie=324af268e3b6843d in_seq=0 out_seq=0 2024-08-28T19:43:44.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.555+0000 7f1807fff640 10 client.20595 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:43:44.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1807fff640 1 -- 172.21.15.16:0/2563698919 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564f760cd5d0 con 0x564f75d6d9f0 2024-08-28T19:43:44.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1807fff640 10 client.20595 handle_command_reply: tid=0 2024-08-28T19:43:44.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1835478740 1 client.20595 shutdown 2024-08-28T19:43:44.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1835478740 1 -- 172.21.15.16:0/2563698919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 msgr2=0x564f7615ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:44.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1835478740 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f7615ff60 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f1810003e80 tx=0x7f1810003eb0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1835478740 1 -- 172.21.15.16:0/2563698919 shutdown_connections 2024-08-28T19:43:44.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1835478740 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x564f75d6d9f0 0x564f76171c20 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1835478740 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75daf070 0x564f7616bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1807fff640 0 client.20595 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:43:44.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1835478740 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f7615ff60 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1835478740 1 --2- 172.21.15.16:0/2563698919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d861b0 0x564f7615fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1835478740 1 -- 172.21.15.16:0/2563698919 >> 172.21.15.16:0/2563698919 conn(0x564f75da3460 msgr2=0x564f761574a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:44.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1835478740 1 -- 172.21.15.16:0/2563698919 shutdown_connections 2024-08-28T19:43:44.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.556+0000 7f1835478740 1 -- 172.21.15.16:0/2563698919 wait complete. 2024-08-28T19:43:44.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.557+0000 7f1835478740 20 client.20595 trim_cache size 0 max 16384 2024-08-28T19:43:44.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.570+0000 7f1835478740 1 Processor -- start 2024-08-28T19:43:44.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.570+0000 7f1835478740 1 -- start start 2024-08-28T19:43:44.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1835478740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d6d9f0 0x564f76155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1835478740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d861b0 0x564f76161650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1835478740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d8e390 0x564f761517d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1835478740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18281aeb70 con 0x564f75d861b0 2024-08-28T19:43:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1835478740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18281b0740 con 0x564f75d6d9f0 2024-08-28T19:43:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1835478740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f18281af240 con 0x564f75d8e390 2024-08-28T19:43:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1827fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d861b0 0x564f76161650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1827fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d861b0 0x564f76161650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35924/0 (socket says 172.21.15.16:35924) 2024-08-28T19:43:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1827fff640 1 -- 172.21.15.16:0/934665264 learned_addr learned my addr 172.21.15.16:0/934665264 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f182cc9f640 1 --2- 172.21.15.16:0/934665264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d6d9f0 0x564f76155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1827fff640 1 -- 172.21.15.16:0/934665264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d8e390 msgr2=0x564f761517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f182d4a0640 1 --2- 172.21.15.16:0/934665264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d8e390 0x564f761517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1827fff640 1 --2- 172.21.15.16:0/934665264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d8e390 0x564f761517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1827fff640 1 -- 172.21.15.16:0/934665264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d6d9f0 msgr2=0x564f76155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1827fff640 1 --2- 172.21.15.16:0/934665264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d6d9f0 0x564f76155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1827fff640 1 -- 172.21.15.16:0/934665264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f76160ce0 con 0x564f75d861b0 2024-08-28T19:43:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f182cc9f640 1 --2- 172.21.15.16:0/934665264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d6d9f0 0x564f76155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.571+0000 7f1827fff640 1 --2- 172.21.15.16:0/934665264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d861b0 0x564f76161650 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f1810001eb0 tx=0x7f1810005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0956755ff4e5cb server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.572+0000 7f181e7fc640 1 -- 172.21.15.16:0/934665264 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1810005350 con 0x564f75d861b0 2024-08-28T19:43:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.572+0000 7f181e7fc640 1 -- 172.21.15.16:0/934665264 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18100054f0 con 0x564f75d861b0 2024-08-28T19:43:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.572+0000 7f181e7fc640 1 -- 172.21.15.16:0/934665264 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181000ee90 con 0x564f75d861b0 2024-08-28T19:43:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.572+0000 7f1835478740 1 -- 172.21.15.16:0/934665264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d861b0 msgr2=0x564f76161650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.572+0000 7f1835478740 1 --2- 172.21.15.16:0/934665264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d861b0 0x564f76161650 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f1810001eb0 tx=0x7f1810005f70 comp rx=0 tx=0).stop 2024-08-28T19:43:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.572+0000 7f1835478740 1 -- 172.21.15.16:0/934665264 shutdown_connections 2024-08-28T19:43:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.572+0000 7f1835478740 1 --2- 172.21.15.16:0/934665264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d8e390 0x564f761517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.572+0000 7f1835478740 1 --2- 172.21.15.16:0/934665264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d861b0 0x564f76161650 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.572+0000 7f1835478740 1 --2- 172.21.15.16:0/934665264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d6d9f0 0x564f76155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.573+0000 7f1835478740 1 -- 172.21.15.16:0/934665264 >> 172.21.15.16:0/934665264 conn(0x564f75da3460 msgr2=0x564f76160f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.573+0000 7f1835478740 1 -- 172.21.15.16:0/934665264 shutdown_connections 2024-08-28T19:43:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.573+0000 7f1835478740 1 -- 172.21.15.16:0/934665264 wait complete. 2024-08-28T19:43:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.573+0000 7f1835478740 1 -- start start 2024-08-28T19:43:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.573+0000 7f1835478740 1 Processor -- start 2024-08-28T19:43:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1835478740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d6d9f0 0x564f761751a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1835478740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d861b0 0x564f76172670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1835478740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76172bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1835478740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18281b0070 con 0x564f75d8e390 2024-08-28T19:43:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1835478740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18281af9a0 con 0x564f75d861b0 2024-08-28T19:43:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1835478740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f18281acb70 con 0x564f75d6d9f0 2024-08-28T19:43:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1835478740 10 client.0 _ll_register_callbacks cb 0x7f1818005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f182d4a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76172bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f182d4a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76172bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35936/0 (socket says 172.21.15.16:35936) 2024-08-28T19:43:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f182d4a0640 1 -- 172.21.15.16:0/1369165243 learned_addr learned my addr 172.21.15.16:0/1369165243 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1827fff640 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d861b0 0x564f76172670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f182cc9f640 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d6d9f0 0x564f761751a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1827fff640 1 -- 172.21.15.16:0/1369165243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d6d9f0 msgr2=0x564f761751a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1827fff640 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d6d9f0 0x564f761751a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1827fff640 1 -- 172.21.15.16:0/1369165243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 msgr2=0x564f76172bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1827fff640 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76172bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1827fff640 1 -- 172.21.15.16:0/1369165243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f761730f0 con 0x564f75d861b0 2024-08-28T19:43:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f182d4a0640 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76172bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:43:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f182cc9f640 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d6d9f0 0x564f761751a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1827fff640 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d861b0 0x564f76172670 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f181001c040 tx=0x7f1810004800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e61e96222bca7db server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.574+0000 7f1807fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-08-28T19:43:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1807fff640 1 -- 172.21.15.16:0/1369165243 <== mon.2 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1810047060 con 0x564f75d861b0 2024-08-28T19:43:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1835478740 1 -- 172.21.15.16:0/1369165243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18100054f0 con 0x564f75d861b0 2024-08-28T19:43:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1807fff640 1 -- 172.21.15.16:0/1369165243 <== mon.2 v2:172.21.15.28:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f1810032040 con 0x564f75d861b0 2024-08-28T19:43:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1807fff640 1 -- 172.21.15.16:0/1369165243 <== mon.2 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f181004e050 con 0x564f75d861b0 2024-08-28T19:43:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1835478740 10 client.30053 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1835478740 1 -- 172.21.15.16:0/1369165243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x564f761dc040 con 0x564f75d861b0 2024-08-28T19:43:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1807fff640 1 -- 172.21.15.16:0/1369165243 <== mon.2 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18100054f0 con 0x564f75d861b0 2024-08-28T19:43:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1807fff640 1 -- 172.21.15.16:0/1369165243 <== mon.2 v2:172.21.15.28:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7f1810041030 con 0x564f75d861b0 2024-08-28T19:43:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1835478740 10 client.30053 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1835478740 10 client.30053 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-08-28T19:43:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1835478740 20 client.30053 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1835478740 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x564f75daf070 0x564f761de720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1835478740 4 client.30053 mds_command: new command op to 24649 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:43:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.575+0000 7f1835478740 1 -- 172.21.15.16:0/1369165243 --> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564f760c9070 con 0x564f75daf070 2024-08-28T19:43:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.576+0000 7f182cc9f640 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x564f75daf070 0x564f761de720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.576+0000 7f1807fff640 10 client.30053 ms_handle_connect on v2:172.21.15.115:6838/3027291414 2024-08-28T19:43:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.576+0000 7f182cc9f640 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x564f75daf070 0x564f761de720 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3d41a0d3ca45bb6a server_cookie=ac2d14b4d6a3591f in_seq=0 out_seq=0 2024-08-28T19:43:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.576+0000 7f1807fff640 1 -- 172.21.15.16:0/1369165243 <== mds.1 v2:172.21.15.115:6838/3027291414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564f760c9070 con 0x564f75daf070 2024-08-28T19:43:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.576+0000 7f1807fff640 10 client.30053 handle_command_reply: tid=0 2024-08-28T19:43:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.576+0000 7f1835478740 1 client.30053 shutdown 2024-08-28T19:43:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.576+0000 7f1835478740 1 -- 172.21.15.16:0/1369165243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d861b0 msgr2=0x564f76172670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.576+0000 7f1835478740 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d861b0 0x564f76172670 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f181001c040 tx=0x7f1810004800 comp rx=0 tx=0).stop 2024-08-28T19:43:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f1835478740 1 -- 172.21.15.16:0/1369165243 shutdown_connections 2024-08-28T19:43:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f1835478740 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.115:6838/3027291414,v1:172.21.15.115:6839/3027291414] conn(0x564f75daf070 0x564f761de720 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f1835478740 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f75d8e390 0x564f76172bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f1835478740 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f75d861b0 0x564f76172670 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f1835478740 1 --2- 172.21.15.16:0/1369165243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564f75d6d9f0 0x564f761751a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f1807fff640 0 client.30053 ms_handle_reset on v2:172.21.15.115:6838/3027291414 2024-08-28T19:43:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f1835478740 1 -- 172.21.15.16:0/1369165243 >> 172.21.15.16:0/1369165243 conn(0x564f75da3460 msgr2=0x564f76155ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f1835478740 1 -- 172.21.15.16:0/1369165243 shutdown_connections 2024-08-28T19:43:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f1835478740 1 -- 172.21.15.16:0/1369165243 wait complete. 2024-08-28T19:43:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f1835478740 20 client.30053 trim_cache size 0 max 16384 2024-08-28T19:43:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:43:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f182ef03640 1 -- 172.21.15.16:0/1384614823 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18140b4ca0 msgr2=0x7f18140b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f182ef03640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18140b4ca0 0x7f18140b71b0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1820004af0 tx=0x7f1820005c30 comp rx=0 tx=0).stop 2024-08-28T19:43:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f182ef03640 1 -- 172.21.15.16:0/1384614823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1828104660 msgr2=0x7f18281981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.577+0000 7f182ef03640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1828104660 0x7f18281981b0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f181800e9f0 tx=0x7f181800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.578+0000 7f182ef03640 1 -- 172.21.15.16:0/1384614823 shutdown_connections 2024-08-28T19:43:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.578+0000 7f182ef03640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f182810dc10 0x7f182819d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.578+0000 7f182ef03640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f18140b4ca0 0x7f18140b71b0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.578+0000 7f182ef03640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1828105090 0x7f1828198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.578+0000 7f182ef03640 1 --2- 172.21.15.16:0/1384614823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1828104660 0x7f18281981b0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.578+0000 7f182ef03640 1 -- 172.21.15.16:0/1384614823 >> 172.21.15.16:0/1384614823 conn(0x7f1828079510 msgr2=0x7f18281066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:44.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.578+0000 7f182ef03640 1 -- 172.21.15.16:0/1384614823 shutdown_connections 2024-08-28T19:43:44.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:44.578+0000 7f182ef03640 1 -- 172.21.15.16:0/1384614823 wait complete. 2024-08-28T19:43:45.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:45 smithi016 ceph-mon[18556]: pgmap v1052: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 79 KiB/s wr, 18 op/s 2024-08-28T19:43:45.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:45 smithi028 ceph-mon[21769]: pgmap v1052: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 79 KiB/s wr, 18 op/s 2024-08-28T19:43:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:45 smithi115 ceph-mon[22184]: pgmap v1052: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 79 KiB/s wr, 18 op/s 2024-08-28T19:43:45.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-08-28T19:43:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:46.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.088+0000 7fda58c97640 1 -- 172.21.15.16:0/1021529848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda54102800 msgr2=0x7fda54102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.088+0000 7fda58c97640 1 --2- 172.21.15.16:0/1021529848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda54102800 0x7fda54102c10 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fda44009900 tx=0x7fda4402a410 comp rx=0 tx=0).stop 2024-08-28T19:43:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.088+0000 7fda58c97640 1 -- 172.21.15.16:0/1021529848 shutdown_connections 2024-08-28T19:43:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.088+0000 7fda58c97640 1 --2- 172.21.15.16:0/1021529848 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda54108a90 0x7fda5410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.088+0000 7fda58c97640 1 --2- 172.21.15.16:0/1021529848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda54108140 0x7fda54108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.088+0000 7fda58c97640 1 --2- 172.21.15.16:0/1021529848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda54102800 0x7fda54102c10 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.088+0000 7fda58c97640 1 -- 172.21.15.16:0/1021529848 >> 172.21.15.16:0/1021529848 conn(0x7fda540794d0 msgr2=0x7fda540798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.088+0000 7fda58c97640 1 -- 172.21.15.16:0/1021529848 shutdown_connections 2024-08-28T19:43:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.088+0000 7fda58c97640 1 -- 172.21.15.16:0/1021529848 wait complete. 2024-08-28T19:43:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda58c97640 1 Processor -- start 2024-08-28T19:43:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda58c97640 1 -- start start 2024-08-28T19:43:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda58c97640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda54102800 0x7fda5419c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda58c97640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda54108140 0x7fda5419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda58c97640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda54108a90 0x7fda541a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda58c97640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda54112770 con 0x7fda54108a90 2024-08-28T19:43:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda58c97640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda541125f0 con 0x7fda54108140 2024-08-28T19:43:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda58c97640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fda541128f0 con 0x7fda54102800 2024-08-28T19:43:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda54108a90 0x7fda541a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda51d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda54108140 0x7fda5419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda54108a90 0x7fda541a1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35960/0 (socket says 172.21.15.16:35960) 2024-08-28T19:43:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.089+0000 7fda51d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda54108140 0x7fda5419ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.16:47884/0 (socket says 172.21.15.16:47884) 2024-08-28T19:43:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda52d9d640 1 -- 172.21.15.16:0/229811105 learned_addr learned my addr 172.21.15.16:0/229811105 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda5259c640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda54102800 0x7fda5419c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda52d9d640 1 -- 172.21.15.16:0/229811105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda54102800 msgr2=0x7fda5419c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda52d9d640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda54102800 0x7fda5419c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda52d9d640 1 -- 172.21.15.16:0/229811105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda54108140 msgr2=0x7fda5419ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda52d9d640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda54108140 0x7fda5419ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda52d9d640 1 -- 172.21.15.16:0/229811105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda541a2440 con 0x7fda54108a90 2024-08-28T19:43:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda5259c640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda54102800 0x7fda5419c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda52d9d640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda54108a90 0x7fda541a1d10 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00b970 tx=0x7fda4c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda437fe640 1 -- 172.21.15.16:0/229811105 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c019070 con 0x7fda54108a90 2024-08-28T19:43:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda437fe640 1 -- 172.21.15.16:0/229811105 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda4c014440 con 0x7fda54108a90 2024-08-28T19:43:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda58c97640 1 -- 172.21.15.16:0/229811105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda541a2730 con 0x7fda54108a90 2024-08-28T19:43:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda437fe640 1 -- 172.21.15.16:0/229811105 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c0075d0 con 0x7fda54108a90 2024-08-28T19:43:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.090+0000 7fda58c97640 1 -- 172.21.15.16:0/229811105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda541a8e90 con 0x7fda54108a90 2024-08-28T19:43:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.093+0000 7fda5f20c740 1 Processor -- start 2024-08-28T19:43:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.093+0000 7fda437fe640 1 -- 172.21.15.16:0/229811105 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fda4c01c070 con 0x7fda54108a90 2024-08-28T19:43:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda5f20c740 1 -- start start 2024-08-28T19:43:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222cf070 0x560722674350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x560722674890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222ae390 0x560722674dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda5f20c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5607225eb680 con 0x5607222a61b0 2024-08-28T19:43:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda5f20c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5607225dac10 con 0x5607222ae390 2024-08-28T19:43:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda5f20c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5607225debc0 con 0x5607222cf070 2024-08-28T19:43:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x560722674890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x560722674890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35970/0 (socket says 172.21.15.16:35970) 2024-08-28T19:43:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda5259c640 1 -- 172.21.15.16:0/166766349 learned_addr learned my addr 172.21.15.16:0/166766349 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda51d9b640 1 --2- 172.21.15.16:0/166766349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222cf070 0x560722674350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.094+0000 7fda51d9b640 1 --2- 172.21.15.16:0/166766349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222ae390 0x560722674dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda437fe640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda3c0b4ca0 0x7fda3c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda5259c640 1 -- 172.21.15.16:0/166766349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222cf070 msgr2=0x560722674350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda5259c640 1 --2- 172.21.15.16:0/166766349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222cf070 0x560722674350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda5259c640 1 -- 172.21.15.16:0/166766349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222ae390 msgr2=0x560722674dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda5259c640 1 --2- 172.21.15.16:0/166766349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222ae390 0x560722674dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda5259c640 1 -- 172.21.15.16:0/166766349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56072233a650 con 0x5607222a61b0 2024-08-28T19:43:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda52d9d640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda3c0b4ca0 0x7fda3c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda437fe640 1 -- 172.21.15.16:0/229811105 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7fda4c0e4d60 con 0x7fda54108a90 2024-08-28T19:43:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda5259c640 1 --2- 172.21.15.16:0/166766349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x560722674890 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fda440096c0 tx=0x7fda44031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed793fd4b0b9f6a6 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda40ff9640 1 -- 172.21.15.16:0/166766349 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4403f070 con 0x5607222a61b0 2024-08-28T19:43:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda40ff9640 1 -- 172.21.15.16:0/166766349 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda440316a0 con 0x5607222a61b0 2024-08-28T19:43:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda40ff9640 1 -- 172.21.15.16:0/166766349 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44035070 con 0x5607222a61b0 2024-08-28T19:43:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.095+0000 7fda52d9d640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda3c0b4ca0 0x7fda3c0b71b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0e26e0 tx=0x7fda4c077330 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.096+0000 7fda5f20c740 1 -- 172.21.15.16:0/166766349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 msgr2=0x560722674890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.096+0000 7fda5f20c740 1 --2- 172.21.15.16:0/166766349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x560722674890 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fda440096c0 tx=0x7fda44031e60 comp rx=0 tx=0).stop 2024-08-28T19:43:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.096+0000 7fda5f20c740 1 -- 172.21.15.16:0/166766349 shutdown_connections 2024-08-28T19:43:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.096+0000 7fda5f20c740 1 --2- 172.21.15.16:0/166766349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222ae390 0x560722674dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.096+0000 7fda5f20c740 1 --2- 172.21.15.16:0/166766349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x560722674890 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.096+0000 7fda5f20c740 1 --2- 172.21.15.16:0/166766349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222cf070 0x560722674350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.096+0000 7fda5f20c740 1 -- 172.21.15.16:0/166766349 >> 172.21.15.16:0/166766349 conn(0x5607222c3460 msgr2=0x560722672240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.096+0000 7fda5f20c740 1 -- 172.21.15.16:0/166766349 shutdown_connections 2024-08-28T19:43:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.096+0000 7fda5f20c740 1 -- 172.21.15.16:0/166766349 wait complete. 2024-08-28T19:43:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda5f20c740 1 -- start start 2024-08-28T19:43:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda5f20c740 1 Processor -- start 2024-08-28T19:43:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222a61b0 0x56072267faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222ae390 0x56072267ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222cf070 0x56072268be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda5f20c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56072243d700 con 0x5607222cf070 2024-08-28T19:43:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda5f20c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560722561750 con 0x5607222a61b0 2024-08-28T19:43:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda5f20c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560722682ad0 con 0x5607222ae390 2024-08-28T19:43:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda5f20c740 10 client.0 _ll_register_callbacks cb 0x560722590e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda51d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222cf070 0x56072268be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda51d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222cf070 0x56072268be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35974/0 (socket says 172.21.15.16:35974) 2024-08-28T19:43:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda51d9b640 1 -- 172.21.15.16:0/2339485322 learned_addr learned my addr 172.21.15.16:0/2339485322 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.097+0000 7fda5259c640 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222ae390 0x56072267ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda51d9b640 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222a61b0 0x56072267faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda51d9b640 1 -- 172.21.15.16:0/2339485322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222ae390 msgr2=0x56072267ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda51d9b640 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222ae390 0x56072267ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda51d9b640 1 -- 172.21.15.16:0/2339485322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222a61b0 msgr2=0x56072267faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda51d9b640 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222a61b0 0x56072267faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda51d9b640 1 -- 172.21.15.16:0/2339485322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56072259a900 con 0x5607222cf070 2024-08-28T19:43:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda51d9b640 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222cf070 0x56072268be20 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fda4800ad20 tx=0x7fda480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a7c6926b1d130e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda39ffb640 1 -- 172.21.15.16:0/2339485322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48004020 con 0x5607222cf070 2024-08-28T19:43:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda39ffb640 1 -- 172.21.15.16:0/2339485322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7fda4801a040 con 0x5607222cf070 2024-08-28T19:43:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda5f20c740 1 -- 172.21.15.16:0/2339485322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda440316a0 con 0x5607222cf070 2024-08-28T19:43:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda39ffb640 1 -- 172.21.15.16:0/2339485322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7fda4802b050 con 0x5607222cf070 2024-08-28T19:43:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda5f20c740 10 client.20613 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda5f20c740 1 -- 172.21.15.16:0/2339485322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x56072268e870 con 0x5607222cf070 2024-08-28T19:43:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.098+0000 7fda39ffb640 1 -- 172.21.15.16:0/2339485322 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda440316a0 con 0x5607222cf070 2024-08-28T19:43:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.099+0000 7fda39ffb640 1 -- 172.21.15.16:0/2339485322 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7fda480047a0 con 0x5607222cf070 2024-08-28T19:43:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.099+0000 7fda5f20c740 10 client.20613 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.099+0000 7fda5f20c740 10 client.20613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:43:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.099+0000 7fda5f20c740 20 client.20613 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.099+0000 7fda5f20c740 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56072228d9f0 0x560722691ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.099+0000 7fda5f20c740 4 client.20613 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-28T19:43:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.099+0000 7fda5f20c740 1 -- 172.21.15.16:0/2339485322 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607225ed5d0 con 0x56072228d9f0 2024-08-28T19:43:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.099+0000 7fda5259c640 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56072228d9f0 0x560722691ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.100+0000 7fda39ffb640 10 client.20613 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.100+0000 7fda5259c640 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56072228d9f0 0x560722691ea0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9234c3d7c37a85f4 server_cookie=75d27d9dc521eac6 in_seq=0 out_seq=0 2024-08-28T19:43:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.100+0000 7fda39ffb640 1 -- 172.21.15.16:0/2339485322 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5607225ed5d0 con 0x56072228d9f0 2024-08-28T19:43:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.100+0000 7fda39ffb640 10 client.20613 handle_command_reply: tid=0 2024-08-28T19:43:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.100+0000 7fda5f20c740 1 client.20613 shutdown 2024-08-28T19:43:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda5f20c740 1 -- 172.21.15.16:0/2339485322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222cf070 msgr2=0x56072268be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda5f20c740 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222cf070 0x56072268be20 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fda4800ad20 tx=0x7fda480046c0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda5f20c740 1 -- 172.21.15.16:0/2339485322 shutdown_connections 2024-08-28T19:43:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda5f20c740 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222cf070 0x56072268be20 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda5f20c740 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x56072228d9f0 0x560722691ea0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda5f20c740 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222ae390 0x56072267ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda5f20c740 1 --2- 172.21.15.16:0/2339485322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222a61b0 0x56072267faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda39ffb640 0 client.20613 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda5f20c740 1 -- 172.21.15.16:0/2339485322 >> 172.21.15.16:0/2339485322 conn(0x5607222c3460 msgr2=0x5607226774e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda5f20c740 1 -- 172.21.15.16:0/2339485322 shutdown_connections 2024-08-28T19:43:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda5f20c740 1 -- 172.21.15.16:0/2339485322 wait complete. 2024-08-28T19:43:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.101+0000 7fda5f20c740 20 client.20613 trim_cache size 0 max 16384 2024-08-28T19:43:46.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.114+0000 7fda5f20c740 1 Processor -- start 2024-08-28T19:43:46.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.114+0000 7fda5f20c740 1 -- start start 2024-08-28T19:43:46.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.114+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222a61b0 0x560722677b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.114+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222ae390 0x56072267cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.114+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222cf070 0x56072267d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.114+0000 7fda5f20c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda541b2f10 con 0x5607222ae390 2024-08-28T19:43:46.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.114+0000 7fda5f20c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda541b4ae0 con 0x5607222cf070 2024-08-28T19:43:46.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.114+0000 7fda5f20c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fda541b35e0 con 0x5607222a61b0 2024-08-28T19:43:46.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.114+0000 7fda5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222ae390 0x56072267cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222ae390 0x56072267cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35980/0 (socket says 172.21.15.16:35980) 2024-08-28T19:43:46.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda5259c640 1 -- 172.21.15.16:0/4086070148 learned_addr learned my addr 172.21.15.16:0/4086070148 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:46.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda51d9b640 1 --2- 172.21.15.16:0/4086070148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222a61b0 0x560722677b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda5259c640 1 -- 172.21.15.16:0/4086070148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222a61b0 msgr2=0x560722677b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:46.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda5259c640 1 --2- 172.21.15.16:0/4086070148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222a61b0 0x560722677b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda5259c640 1 -- 172.21.15.16:0/4086070148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222cf070 msgr2=0x56072267d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:46.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda5259c640 1 --2- 172.21.15.16:0/4086070148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222cf070 0x56072267d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda5259c640 1 -- 172.21.15.16:0/4086070148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607226821e0 con 0x5607222ae390 2024-08-28T19:43:46.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda51d9b640 1 --2- 172.21.15.16:0/4086070148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222a61b0 0x560722677b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:43:46.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda5259c640 1 --2- 172.21.15.16:0/4086070148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222ae390 0x56072267cc90 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x560722672fd0 tx=0x7fda44005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612a4db0bbfd726e server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:46.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda3bfff640 1 -- 172.21.15.16:0/4086070148 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44035050 con 0x5607222ae390 2024-08-28T19:43:46.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda3bfff640 1 -- 172.21.15.16:0/4086070148 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda44005020 con 0x5607222ae390 2024-08-28T19:43:46.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.115+0000 7fda3bfff640 1 -- 172.21.15.16:0/4086070148 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4403f040 con 0x5607222ae390 2024-08-28T19:43:46.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.116+0000 7fda5f20c740 1 -- 172.21.15.16:0/4086070148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222ae390 msgr2=0x56072267cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.116+0000 7fda5f20c740 1 --2- 172.21.15.16:0/4086070148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222ae390 0x56072267cc90 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x560722672fd0 tx=0x7fda44005a20 comp rx=0 tx=0).stop 2024-08-28T19:43:46.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.116+0000 7fda5f20c740 1 -- 172.21.15.16:0/4086070148 shutdown_connections 2024-08-28T19:43:46.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.116+0000 7fda5f20c740 1 --2- 172.21.15.16:0/4086070148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222cf070 0x56072267d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.116+0000 7fda5f20c740 1 --2- 172.21.15.16:0/4086070148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222ae390 0x56072267cc90 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.116+0000 7fda5f20c740 1 --2- 172.21.15.16:0/4086070148 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222a61b0 0x560722677b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.116+0000 7fda5f20c740 1 -- 172.21.15.16:0/4086070148 >> 172.21.15.16:0/4086070148 conn(0x5607222c3460 msgr2=0x560722677450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.116+0000 7fda5f20c740 1 -- 172.21.15.16:0/4086070148 shutdown_connections 2024-08-28T19:43:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.116+0000 7fda5f20c740 1 -- 172.21.15.16:0/4086070148 wait complete. 2024-08-28T19:43:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.116+0000 7fda5f20c740 1 -- start start 2024-08-28T19:43:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.116+0000 7fda5f20c740 1 Processor -- start 2024-08-28T19:43:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x5607226882d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222ae390 0x560722684790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda5f20c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222cf070 0x560722684cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda5f20c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda541b4410 con 0x5607222a61b0 2024-08-28T19:43:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda5f20c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda541b3d40 con 0x5607222cf070 2024-08-28T19:43:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda5f20c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fda541b0f10 con 0x5607222ae390 2024-08-28T19:43:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda5f20c740 10 client.0 _ll_register_callbacks cb 0x7fda4800ac30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-28T19:43:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda51d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x5607226882d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda51d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x5607226882d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35986/0 (socket says 172.21.15.16:35986) 2024-08-28T19:43:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda51d9b640 1 -- 172.21.15.16:0/814730741 learned_addr learned my addr 172.21.15.16:0/814730741 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda5259c640 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222ae390 0x560722684790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda51d9b640 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222cf070 0x560722684cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda51d9b640 1 -- 172.21.15.16:0/814730741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222ae390 msgr2=0x560722684790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda51d9b640 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222ae390 0x560722684790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda51d9b640 1 -- 172.21.15.16:0/814730741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222cf070 msgr2=0x560722684cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-28T19:43:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda51d9b640 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222cf070 0x560722684cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda51d9b640 1 -- 172.21.15.16:0/814730741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560722685210 con 0x5607222a61b0 2024-08-28T19:43:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.117+0000 7fda51d9b640 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x5607226882d0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fda48015010 tx=0x7fda4800a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6490cac129d7c7b server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-08-28T19:43:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda39ffb640 1 -- 172.21.15.16:0/814730741 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48035020 con 0x5607222a61b0 2024-08-28T19:43:46.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda39ffb640 1 -- 172.21.15.16:0/814730741 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7fda48024040 con 0x5607222a61b0 2024-08-28T19:43:46.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda5f20c740 1 -- 172.21.15.16:0/814730741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda44005020 con 0x5607222a61b0 2024-08-28T19:43:46.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda39ffb640 1 -- 172.21.15.16:0/814730741 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7fda4803d050 con 0x5607222a61b0 2024-08-28T19:43:46.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda5f20c740 10 client.20619 fetch_fsmap learned FSMap version 109 2024-08-28T19:43:46.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda5f20c740 1 -- 172.21.15.16:0/814730741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x5607226fd4f0 con 0x5607222a61b0 2024-08-28T19:43:46.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda39ffb640 1 -- 172.21.15.16:0/814730741 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda44005020 con 0x5607222a61b0 2024-08-28T19:43:46.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda39ffb640 1 -- 172.21.15.16:0/814730741 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 109) ==== 8382+0+0 (secure 0 0 0) 0x7fda4801cb00 con 0x5607222a61b0 2024-08-28T19:43:46.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda5f20c740 10 client.20619 fetch_fsmap finished waiting for FSMap version 109 2024-08-28T19:43:46.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda5f20c740 10 client.20619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-08-28T19:43:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda5f20c740 20 client.20619 populate_metadata read hostname 'smithi016' 2024-08-28T19:43:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda5f20c740 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560722262da0 0x5607226ffce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda5f20c740 4 client.20619 mds_command: new command op to 24665 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-08-28T19:43:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.118+0000 7fda5f20c740 1 -- 172.21.15.16:0/814730741 --> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5607225e9070 con 0x560722262da0 2024-08-28T19:43:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.119+0000 7fda5259c640 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560722262da0 0x5607226ffce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.119+0000 7fda5259c640 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560722262da0 0x5607226ffce0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=740da0a90528ab23 server_cookie=792f9d39e539e551 in_seq=0 out_seq=0 2024-08-28T19:43:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.119+0000 7fda39ffb640 10 client.20619 ms_handle_connect on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.119+0000 7fda39ffb640 1 -- 172.21.15.16:0/814730741 <== mds.0 v2:172.21.15.28:6838/2518320595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5607225e9070 con 0x560722262da0 2024-08-28T19:43:46.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.119+0000 7fda39ffb640 10 client.20619 handle_command_reply: tid=0 2024-08-28T19:43:46.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.119+0000 7fda5f20c740 1 client.20619 shutdown 2024-08-28T19:43:46.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda5f20c740 1 -- 172.21.15.16:0/814730741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 msgr2=0x5607226882d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda5f20c740 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x5607226882d0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fda48015010 tx=0x7fda4800a370 comp rx=0 tx=0).stop 2024-08-28T19:43:46.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda5f20c740 1 -- 172.21.15.16:0/814730741 shutdown_connections 2024-08-28T19:43:46.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda5f20c740 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607222cf070 0x560722684cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda5f20c740 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.28:6838/2518320595,v1:172.21.15.28:6839/2518320595] conn(0x560722262da0 0x5607226ffce0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda5f20c740 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607222ae390 0x560722684790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda5f20c740 1 --2- 172.21.15.16:0/814730741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607222a61b0 0x5607226882d0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda5f20c740 1 -- 172.21.15.16:0/814730741 >> 172.21.15.16:0/814730741 conn(0x5607222c3460 msgr2=0x560722675380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:46.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda39ffb640 0 client.20619 ms_handle_reset on v2:172.21.15.28:6838/2518320595 2024-08-28T19:43:46.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda5f20c740 1 -- 172.21.15.16:0/814730741 shutdown_connections 2024-08-28T19:43:46.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda5f20c740 1 -- 172.21.15.16:0/814730741 wait complete. 2024-08-28T19:43:46.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda5f20c740 20 client.20619 trim_cache size 0 max 16384 2024-08-28T19:43:46.130 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-08-28T19:43:46.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda58c97640 1 -- 172.21.15.16:0/229811105 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda3c0b4ca0 msgr2=0x7fda3c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda58c97640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda3c0b4ca0 0x7fda3c0b71b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0e26e0 tx=0x7fda4c077330 comp rx=0 tx=0).stop 2024-08-28T19:43:46.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda58c97640 1 -- 172.21.15.16:0/229811105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda54108a90 msgr2=0x7fda541a1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda58c97640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda54108a90 0x7fda541a1d10 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00b970 tx=0x7fda4c00be30 comp rx=0 tx=0).stop 2024-08-28T19:43:46.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda58c97640 1 -- 172.21.15.16:0/229811105 shutdown_connections 2024-08-28T19:43:46.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda58c97640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda54108a90 0x7fda541a1d10 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda58c97640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fda3c0b4ca0 0x7fda3c0b71b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda58c97640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda54108140 0x7fda5419ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda58c97640 1 --2- 172.21.15.16:0/229811105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda54102800 0x7fda5419c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.120+0000 7fda58c97640 1 -- 172.21.15.16:0/229811105 >> 172.21.15.16:0/229811105 conn(0x7fda540794d0 msgr2=0x7fda5406aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:46.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.121+0000 7fda58c97640 1 -- 172.21.15.16:0/229811105 shutdown_connections 2024-08-28T19:43:46.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.121+0000 7fda58c97640 1 -- 172.21.15.16:0/229811105 wait complete. 2024-08-28T19:43:46.842 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/thrasher.py", line 50, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-08-28T19:43:46.850 INFO:tasks.fwd_scrub:done joining 2024-08-28T19:43:46.850 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-08-28T19:43:46.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-28T19:43:46.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.977+0000 7f341259f640 1 -- 172.21.15.16:0/2536207052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c104550 msgr2=0x7f340c104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.977+0000 7f341259f640 1 --2- 172.21.15.16:0/2536207052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c104550 0x7f340c104960 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009900 tx=0x7f33fc02a410 comp rx=0 tx=0).stop 2024-08-28T19:43:46.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.977+0000 7f341259f640 1 -- 172.21.15.16:0/2536207052 shutdown_connections 2024-08-28T19:43:46.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.977+0000 7f341259f640 1 --2- 172.21.15.16:0/2536207052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f340c10db00 0x7f340c10ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.977+0000 7f341259f640 1 --2- 172.21.15.16:0/2536207052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f340c104f30 0x7f340c10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.977+0000 7f341259f640 1 --2- 172.21.15.16:0/2536207052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c104550 0x7f340c104960 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.977+0000 7f341259f640 1 -- 172.21.15.16:0/2536207052 >> 172.21.15.16:0/2536207052 conn(0x7f340c0fe120 msgr2=0x7f340c100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:46.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.977+0000 7f341259f640 1 -- 172.21.15.16:0/2536207052 shutdown_connections 2024-08-28T19:43:46.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.977+0000 7f341259f640 1 -- 172.21.15.16:0/2536207052 wait complete. 2024-08-28T19:43:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f341259f640 1 Processor -- start 2024-08-28T19:43:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f341259f640 1 -- start start 2024-08-28T19:43:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f341259f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c104550 0x7f340c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f341259f640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f340c104f30 0x7f340c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f341259f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f340c10db00 0x7f340c19fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f341259f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f340c1137c0 con 0x7f340c104550 2024-08-28T19:43:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f341259f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f340c113640 con 0x7f340c10db00 2024-08-28T19:43:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f341259f640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f340c113940 con 0x7f340c104f30 2024-08-28T19:43:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f340bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c104550 0x7f340c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f340bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c104550 0x7f340c19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36010/0 (socket says 172.21.15.16:36010) 2024-08-28T19:43:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f340bfff640 1 -- 172.21.15.16:0/2986157920 learned_addr learned my addr 172.21.15.16:0/2986157920 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f3410b3c640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f340c10db00 0x7f340c19fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f340b7fe640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f340c104f30 0x7f340c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f340bfff640 1 -- 172.21.15.16:0/2986157920 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f340c104f30 msgr2=0x7f340c19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f340bfff640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f340c104f30 0x7f340c19a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f340bfff640 1 -- 172.21.15.16:0/2986157920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f340c10db00 msgr2=0x7f340c19fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f340bfff640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f340c10db00 0x7f340c19fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f340bfff640 1 -- 172.21.15.16:0/2986157920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340c1a0130 con 0x7f340c104550 2024-08-28T19:43:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.978+0000 7f3410b3c640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f340c10db00 0x7f340c19fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:43:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.979+0000 7f340bfff640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c104550 0x7f340c19a200 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f33fc005ab0 tx=0x7f33fc003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.979+0000 7f34097fa640 1 -- 172.21.15.16:0/2986157920 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33fc03f070 con 0x7f340c104550 2024-08-28T19:43:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.979+0000 7f341259f640 1 -- 172.21.15.16:0/2986157920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f340c1a03c0 con 0x7f340c104550 2024-08-28T19:43:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.979+0000 7f34097fa640 1 -- 172.21.15.16:0/2986157920 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33fc031d60 con 0x7f340c104550 2024-08-28T19:43:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.979+0000 7f34097fa640 1 -- 172.21.15.16:0/2986157920 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33fc035030 con 0x7f340c104550 2024-08-28T19:43:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.979+0000 7f341259f640 1 -- 172.21.15.16:0/2986157920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f340c1a6bb0 con 0x7f340c104550 2024-08-28T19:43:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.980+0000 7f341259f640 1 -- 172.21.15.16:0/2986157920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f340c06ab00 con 0x7f340c104550 2024-08-28T19:43:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.981+0000 7f34097fa640 1 -- 172.21.15.16:0/2986157920 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f33fc042070 con 0x7f340c104550 2024-08-28T19:43:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.982+0000 7f34097fa640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f33f40b4ca0 0x7f33f40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.982+0000 7f34097fa640 1 -- 172.21.15.16:0/2986157920 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f33fc1068e0 con 0x7f340c104550 2024-08-28T19:43:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.982+0000 7f340b7fe640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f33f40b4ca0 0x7f33f40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.983+0000 7f340b7fe640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f33f40b4ca0 0x7f33f40b71b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f340c10fea0 tx=0x7f3400005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:46.986+0000 7f34097fa640 1 -- 172.21.15.16:0/2986157920 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f33fc0cd2a0 con 0x7f340c104550 2024-08-28T19:43:47.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.173+0000 7f341259f640 1 -- 172.21.15.16:0/2986157920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f340c1a6de0 con 0x7f340c104550 2024-08-28T19:43:47.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.175+0000 7f34097fa640 1 -- 172.21.15.16:0/2986157920 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+25368 (secure 0 0 0) 0x7f33fc0ccba0 con 0x7f340c104550 2024-08-28T19:43:47.176 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:43:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":116,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:43:15.450386+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":115,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":115,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]},{"pool":4,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:43:47.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 -- 172.21.15.16:0/2986157920 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f33f40b4ca0 msgr2=0x7f33f40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f33f40b4ca0 0x7f33f40b71b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f340c10fea0 tx=0x7f3400005f90 comp rx=0 tx=0).stop 2024-08-28T19:43:47.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 -- 172.21.15.16:0/2986157920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c104550 msgr2=0x7f340c19a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c104550 0x7f340c19a200 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f33fc005ab0 tx=0x7f33fc003c80 comp rx=0 tx=0).stop 2024-08-28T19:43:47.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 -- 172.21.15.16:0/2986157920 shutdown_connections 2024-08-28T19:43:47.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f33f40b4ca0 0x7f33f40b71b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f340c10db00 0x7f340c19fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f340c104f30 0x7f340c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 --2- 172.21.15.16:0/2986157920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c104550 0x7f340c19a200 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 -- 172.21.15.16:0/2986157920 >> 172.21.15.16:0/2986157920 conn(0x7f340c0fe120 msgr2=0x7f340c107650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:47.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 -- 172.21.15.16:0/2986157920 shutdown_connections 2024-08-28T19:43:47.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.177+0000 7f341259f640 1 -- 172.21.15.16:0/2986157920 wait complete. 2024-08-28T19:43:47.190 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-28T19:13:55.828283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-08-28T19:18:29.423422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-08-28T19:18:31.212294+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 39, 'snap_epoch': 115, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-08-28T19:18:37.500166+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 39, 'snap_epoch': 115, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-08-28T19:43:47.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-28T19:43:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.303+0000 7f263d02d640 1 -- 172.21.15.16:0/2273582929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26381089d0 msgr2=0x7f263810f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.303+0000 7f263d02d640 1 --2- 172.21.15.16:0/2273582929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26381089d0 0x7f263810f200 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f2630009ca0 tx=0x7f263002a8c0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.304+0000 7f263d02d640 1 -- 172.21.15.16:0/2273582929 shutdown_connections 2024-08-28T19:43:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.304+0000 7f263d02d640 1 --2- 172.21.15.16:0/2273582929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26381089d0 0x7f263810f200 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.304+0000 7f263d02d640 1 --2- 172.21.15.16:0/2273582929 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2638108080 0x7f2638108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.304+0000 7f263d02d640 1 --2- 172.21.15.16:0/2273582929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2638102750 0x7f2638102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.304+0000 7f263d02d640 1 -- 172.21.15.16:0/2273582929 >> 172.21.15.16:0/2273582929 conn(0x7f26380fe190 msgr2=0x7f26381005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.304+0000 7f263d02d640 1 -- 172.21.15.16:0/2273582929 shutdown_connections 2024-08-28T19:43:47.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.304+0000 7f263d02d640 1 -- 172.21.15.16:0/2273582929 wait complete. 2024-08-28T19:43:47.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.304+0000 7f263d02d640 1 Processor -- start 2024-08-28T19:43:47.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f263d02d640 1 -- start start 2024-08-28T19:43:47.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f263d02d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2638102750 0x7f26381990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f263d02d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2638108080 0x7f2638199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f263d02d640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26381089d0 0x7f263819e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f263d02d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26381126a0 con 0x7f2638102750 2024-08-28T19:43:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f263d02d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2638112520 con 0x7f2638108080 2024-08-28T19:43:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f263d02d640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2638112820 con 0x7f26381089d0 2024-08-28T19:43:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f2636d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2638102750 0x7f26381990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f2636d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2638102750 0x7f26381990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36026/0 (socket says 172.21.15.16:36026) 2024-08-28T19:43:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f2636d9d640 1 -- 172.21.15.16:0/1822483868 learned_addr learned my addr 172.21.15.16:0/1822483868 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f263759e640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26381089d0 0x7f263819e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f263659c640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2638108080 0x7f2638199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f2636d9d640 1 -- 172.21.15.16:0/1822483868 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26381089d0 msgr2=0x7f263819e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f2636d9d640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26381089d0 0x7f263819e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f2636d9d640 1 -- 172.21.15.16:0/1822483868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2638108080 msgr2=0x7f2638199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f2636d9d640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2638108080 0x7f2638199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f2636d9d640 1 -- 172.21.15.16:0/1822483868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263819efd0 con 0x7f2638102750 2024-08-28T19:43:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.305+0000 7f2636d9d640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2638102750 0x7f26381990a0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f262800e9f0 tx=0x7f262800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.306+0000 7f261ffff640 1 -- 172.21.15.16:0/1822483868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262800cd80 con 0x7f2638102750 2024-08-28T19:43:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.306+0000 7f261ffff640 1 -- 172.21.15.16:0/1822483868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26280108e0 con 0x7f2638102750 2024-08-28T19:43:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.306+0000 7f263d02d640 1 -- 172.21.15.16:0/1822483868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263819f2c0 con 0x7f2638102750 2024-08-28T19:43:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.306+0000 7f261ffff640 1 -- 172.21.15.16:0/1822483868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628018980 con 0x7f2638102750 2024-08-28T19:43:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.306+0000 7f263d02d640 1 -- 172.21.15.16:0/1822483868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26381a5a50 con 0x7f2638102750 2024-08-28T19:43:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.307+0000 7f263d02d640 1 -- 172.21.15.16:0/1822483868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26381039c0 con 0x7f2638102750 2024-08-28T19:43:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.308+0000 7f261ffff640 1 -- 172.21.15.16:0/1822483868 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f2628010430 con 0x7f2638102750 2024-08-28T19:43:47.313 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:47 smithi028 ceph-mon[21769]: pgmap v1053: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 81 KiB/s wr, 19 op/s 2024-08-28T19:43:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.309+0000 7f261ffff640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26200b4ca0 0x7f26200b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.309+0000 7f261ffff640 1 -- 172.21.15.16:0/1822483868 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f26280e6150 con 0x7f2638102750 2024-08-28T19:43:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.309+0000 7f263659c640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26200b4ca0 0x7f26200b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.310+0000 7f263659c640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26200b4ca0 0x7f26200b71b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2638101970 tx=0x7f262c005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.312+0000 7f261ffff640 1 -- 172.21.15.16:0/1822483868 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f26280ac4c0 con 0x7f2638102750 2024-08-28T19:43:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:47 smithi115 ceph-mon[22184]: pgmap v1053: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 81 KiB/s wr, 19 op/s 2024-08-28T19:43:47.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.498+0000 7f263d02d640 1 -- 172.21.15.16:0/1822483868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2638102b60 con 0x7f2638102750 2024-08-28T19:43:47.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.498+0000 7f261ffff640 1 -- 172.21.15.16:0/1822483868 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) ==== 93+0+10 (secure 0 0 0) 0x7f2638102b60 con 0x7f2638102750 2024-08-28T19:43:47.500 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-08-28T19:43:47.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:47 smithi016 ceph-mon[18556]: pgmap v1053: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 81 KiB/s wr, 19 op/s 2024-08-28T19:43:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.500+0000 7f263d02d640 1 -- 172.21.15.16:0/1822483868 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26200b4ca0 msgr2=0x7f26200b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.501+0000 7f263d02d640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26200b4ca0 0x7f26200b71b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2638101970 tx=0x7f262c005f90 comp rx=0 tx=0).stop 2024-08-28T19:43:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.501+0000 7f263d02d640 1 -- 172.21.15.16:0/1822483868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2638102750 msgr2=0x7f26381990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.501+0000 7f263d02d640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2638102750 0x7f26381990a0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f262800e9f0 tx=0x7f262800eeb0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.501+0000 7f263d02d640 1 -- 172.21.15.16:0/1822483868 shutdown_connections 2024-08-28T19:43:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.501+0000 7f263d02d640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26381089d0 0x7f263819e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.501+0000 7f263d02d640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f26200b4ca0 0x7f26200b71b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.501+0000 7f263d02d640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2638108080 0x7f2638199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.501+0000 7f263d02d640 1 --2- 172.21.15.16:0/1822483868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2638102750 0x7f26381990a0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.501+0000 7f263d02d640 1 -- 172.21.15.16:0/1822483868 >> 172.21.15.16:0/1822483868 conn(0x7f26380fe190 msgr2=0x7f26381076d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.501+0000 7f263d02d640 1 -- 172.21.15.16:0/1822483868 shutdown_connections 2024-08-28T19:43:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.501+0000 7f263d02d640 1 -- 172.21.15.16:0/1822483868 wait complete. 2024-08-28T19:43:47.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-08-28T19:43:47.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.627+0000 7f65f6de4640 1 -- 172.21.15.16:0/1872161447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f0109ad0 msgr2=0x7f65f0110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.627+0000 7f65f6de4640 1 --2- 172.21.15.16:0/1872161447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f0109ad0 0x7f65f0110300 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009900 tx=0x7f65e802a410 comp rx=0 tx=0).stop 2024-08-28T19:43:47.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.627+0000 7f65f6de4640 1 -- 172.21.15.16:0/1872161447 shutdown_connections 2024-08-28T19:43:47.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.627+0000 7f65f6de4640 1 --2- 172.21.15.16:0/1872161447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f0109ad0 0x7f65f0110300 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.627+0000 7f65f6de4640 1 --2- 172.21.15.16:0/1872161447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f0109180 0x7f65f0109590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.627+0000 7f65f6de4640 1 --2- 172.21.15.16:0/1872161447 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f65f0103850 0x7f65f0103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.627+0000 7f65f6de4640 1 -- 172.21.15.16:0/1872161447 >> 172.21.15.16:0/1872161447 conn(0x7f65f00ff270 msgr2=0x7f65f01016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:47.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.627+0000 7f65f6de4640 1 -- 172.21.15.16:0/1872161447 shutdown_connections 2024-08-28T19:43:47.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.627+0000 7f65f6de4640 1 -- 172.21.15.16:0/1872161447 wait complete. 2024-08-28T19:43:47.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.627+0000 7f65f6de4640 1 Processor -- start 2024-08-28T19:43:47.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.627+0000 7f65f6de4640 1 -- start start 2024-08-28T19:43:47.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65f6de4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f0103850 0x7f65f019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65f6de4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f0109180 0x7f65f019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65f6de4640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f65f0109ad0 0x7f65f019f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65f6de4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f65f0113780 con 0x7f65f0109180 2024-08-28T19:43:47.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65f6de4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65f0113600 con 0x7f65f0103850 2024-08-28T19:43:47.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65f6de4640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f65f0113900 con 0x7f65f0109ad0 2024-08-28T19:43:47.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f0109180 0x7f65f019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f0109180 0x7f65f019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36052/0 (socket says 172.21.15.16:36052) 2024-08-28T19:43:47.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65effff640 1 -- 172.21.15.16:0/1729801096 learned_addr learned my addr 172.21.15.16:0/1729801096 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:47.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65f4b80640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f0103850 0x7f65f019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65f5381640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f65f0109ad0 0x7f65f019f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65effff640 1 -- 172.21.15.16:0/1729801096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f65f0109ad0 msgr2=0x7f65f019f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65effff640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f65f0109ad0 0x7f65f019f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65effff640 1 -- 172.21.15.16:0/1729801096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f0103850 msgr2=0x7f65f019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65effff640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f0103850 0x7f65f019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65effff640 1 -- 172.21.15.16:0/1729801096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f019ffc0 con 0x7f65f0109180 2024-08-28T19:43:47.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65f5381640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f65f0109ad0 0x7f65f019f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:43:47.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65effff640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f0109180 0x7f65f019a6c0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f65e400ebc0 tx=0x7f65e400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:47.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65edffb640 1 -- 172.21.15.16:0/1729801096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4010430 con 0x7f65f0109180 2024-08-28T19:43:47.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65f6de4640 1 -- 172.21.15.16:0/1729801096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f01a02b0 con 0x7f65f0109180 2024-08-28T19:43:47.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65edffb640 1 -- 172.21.15.16:0/1729801096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65e4010a80 con 0x7f65f0109180 2024-08-28T19:43:47.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.628+0000 7f65edffb640 1 -- 172.21.15.16:0/1729801096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4017a00 con 0x7f65f0109180 2024-08-28T19:43:47.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.629+0000 7f65f6de4640 1 -- 172.21.15.16:0/1729801096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f01a6ad0 con 0x7f65f0109180 2024-08-28T19:43:47.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.630+0000 7f65f6de4640 1 -- 172.21.15.16:0/1729801096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f0103c60 con 0x7f65f0109180 2024-08-28T19:43:47.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.631+0000 7f65edffb640 1 -- 172.21.15.16:0/1729801096 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f65e4017ba0 con 0x7f65f0109180 2024-08-28T19:43:47.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.632+0000 7f65edffb640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65d80b4ca0 0x7f65d80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.632+0000 7f65f4b80640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65d80b4ca0 0x7f65d80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.632+0000 7f65edffb640 1 -- 172.21.15.16:0/1729801096 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f65e40e6350 con 0x7f65f0109180 2024-08-28T19:43:47.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.632+0000 7f65f4b80640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65d80b4ca0 0x7f65d80b71b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f65e000abe0 tx=0x7f65e000a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:47.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.636+0000 7f65edffb640 1 -- 172.21.15.16:0/1729801096 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f65e40acd10 con 0x7f65f0109180 2024-08-28T19:43:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.822+0000 7f65f6de4640 1 -- 172.21.15.16:0/1729801096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f65f002e430 con 0x7f65f0109180 2024-08-28T19:43:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.822+0000 7f65edffb640 1 -- 172.21.15.16:0/1729801096 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) ==== 104+0+11 (secure 0 0 0) 0x7f65e40b5250 con 0x7f65f0109180 2024-08-28T19:43:47.824 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-08-28T19:43:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.824+0000 7f65f6de4640 1 -- 172.21.15.16:0/1729801096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65d80b4ca0 msgr2=0x7f65d80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.824+0000 7f65f6de4640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65d80b4ca0 0x7f65d80b71b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f65e000abe0 tx=0x7f65e000a2d0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.824+0000 7f65f6de4640 1 -- 172.21.15.16:0/1729801096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f0109180 msgr2=0x7f65f019a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.824+0000 7f65f6de4640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f0109180 0x7f65f019a6c0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f65e400ebc0 tx=0x7f65e400c2d0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.825+0000 7f65f6de4640 1 -- 172.21.15.16:0/1729801096 shutdown_connections 2024-08-28T19:43:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.825+0000 7f65f6de4640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f65f0109ad0 0x7f65f019f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.825+0000 7f65f6de4640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f0109180 0x7f65f019a6c0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.825+0000 7f65f6de4640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f65d80b4ca0 0x7f65d80b71b0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.825+0000 7f65f6de4640 1 --2- 172.21.15.16:0/1729801096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f0103850 0x7f65f019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.825+0000 7f65f6de4640 1 -- 172.21.15.16:0/1729801096 >> 172.21.15.16:0/1729801096 conn(0x7f65f00ff270 msgr2=0x7f65f0101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.825+0000 7f65f6de4640 1 -- 172.21.15.16:0/1729801096 shutdown_connections 2024-08-28T19:43:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.825+0000 7f65f6de4640 1 -- 172.21.15.16:0/1729801096 wait complete. 2024-08-28T19:43:47.837 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-08-28T19:43:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.950+0000 7f3228099640 1 -- 172.21.15.16:0/3301010680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3220104e70 msgr2=0x7f3220105280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.950+0000 7f3228099640 1 --2- 172.21.15.16:0/3301010680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3220104e70 0x7f3220105280 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f3210009930 tx=0x7f321002a420 comp rx=0 tx=0).stop 2024-08-28T19:43:47.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.950+0000 7f3228099640 1 -- 172.21.15.16:0/3301010680 shutdown_connections 2024-08-28T19:43:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.950+0000 7f3228099640 1 --2- 172.21.15.16:0/3301010680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32201057c0 0x7f322010f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.950+0000 7f3228099640 1 --2- 172.21.15.16:0/3301010680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3220104e70 0x7f3220105280 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.950+0000 7f3228099640 1 --2- 172.21.15.16:0/3301010680 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3220102eb0 0x7f32201032c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.950+0000 7f3228099640 1 -- 172.21.15.16:0/3301010680 >> 172.21.15.16:0/3301010680 conn(0x7f3220079bb0 msgr2=0x7f3220079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.950+0000 7f3228099640 1 -- 172.21.15.16:0/3301010680 shutdown_connections 2024-08-28T19:43:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.950+0000 7f3228099640 1 -- 172.21.15.16:0/3301010680 wait complete. 2024-08-28T19:43:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3228099640 1 Processor -- start 2024-08-28T19:43:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3228099640 1 -- start start 2024-08-28T19:43:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3228099640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3220102eb0 0x7f322019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3228099640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3220104e70 0x7f322019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3228099640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32201057c0 0x7f322019fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3228099640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f32201137c0 con 0x7f3220102eb0 2024-08-28T19:43:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3228099640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3220113640 con 0x7f32201057c0 2024-08-28T19:43:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3228099640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3220113940 con 0x7f3220104e70 2024-08-28T19:43:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3225e35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3220102eb0 0x7f322019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3225e35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3220102eb0 0x7f322019a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36064/0 (socket says 172.21.15.16:36064) 2024-08-28T19:43:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3225e35640 1 -- 172.21.15.16:0/114599279 learned_addr learned my addr 172.21.15.16:0/114599279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3225634640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3220104e70 0x7f322019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3226636640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32201057c0 0x7f322019fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3225e35640 1 -- 172.21.15.16:0/114599279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3220104e70 msgr2=0x7f322019a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3225e35640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3220104e70 0x7f322019a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3225e35640 1 -- 172.21.15.16:0/114599279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32201057c0 msgr2=0x7f322019fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3225e35640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32201057c0 0x7f322019fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3225e35640 1 -- 172.21.15.16:0/114599279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32201a00c0 con 0x7f3220102eb0 2024-08-28T19:43:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.951+0000 7f3226636640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32201057c0 0x7f322019fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:43:47.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.952+0000 7f3225e35640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3220102eb0 0x7f322019a220 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f321c00d770 tx=0x7f321c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:47.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.952+0000 7f3216ffd640 1 -- 172.21.15.16:0/114599279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c017070 con 0x7f3220102eb0 2024-08-28T19:43:47.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.952+0000 7f3216ffd640 1 -- 172.21.15.16:0/114599279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f321c0054d0 con 0x7f3220102eb0 2024-08-28T19:43:47.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.952+0000 7f3228099640 1 -- 172.21.15.16:0/114599279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32201a03b0 con 0x7f3220102eb0 2024-08-28T19:43:47.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.952+0000 7f3216ffd640 1 -- 172.21.15.16:0/114599279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c004c30 con 0x7f3220102eb0 2024-08-28T19:43:47.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.952+0000 7f3228099640 1 -- 172.21.15.16:0/114599279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32201a0880 con 0x7f3220102eb0 2024-08-28T19:43:47.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.953+0000 7f3228099640 1 -- 172.21.15.16:0/114599279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32201032c0 con 0x7f3220102eb0 2024-08-28T19:43:47.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.954+0000 7f3216ffd640 1 -- 172.21.15.16:0/114599279 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f321c004dd0 con 0x7f3220102eb0 2024-08-28T19:43:47.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.955+0000 7f3216ffd640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f320c0b4ca0 0x7f320c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:47.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.955+0000 7f3225634640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f320c0b4ca0 0x7f320c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:47.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.955+0000 7f3216ffd640 1 -- 172.21.15.16:0/114599279 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f321c0e5880 con 0x7f3220102eb0 2024-08-28T19:43:47.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.956+0000 7f3225634640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f320c0b4ca0 0x7f320c0b71b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3210004770 tx=0x7f3210000bc0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:47.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:47.958+0000 7f3216ffd640 1 -- 172.21.15.16:0/114599279 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f32201a0880 con 0x7f3220102eb0 2024-08-28T19:43:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.142+0000 7f3228099640 1 -- 172.21.15.16:0/114599279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f322002e430 con 0x7f3220102eb0 2024-08-28T19:43:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.143+0000 7f3216ffd640 1 -- 172.21.15.16:0/114599279 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) ==== 100+0+11 (secure 0 0 0) 0x7f321c0ac240 con 0x7f3220102eb0 2024-08-28T19:43:48.144 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-08-28T19:43:48.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 -- 172.21.15.16:0/114599279 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f320c0b4ca0 msgr2=0x7f320c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:48.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f320c0b4ca0 0x7f320c0b71b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3210004770 tx=0x7f3210000bc0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 -- 172.21.15.16:0/114599279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3220102eb0 msgr2=0x7f322019a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:48.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3220102eb0 0x7f322019a220 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f321c00d770 tx=0x7f321c00dc30 comp rx=0 tx=0).stop 2024-08-28T19:43:48.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 -- 172.21.15.16:0/114599279 shutdown_connections 2024-08-28T19:43:48.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f320c0b4ca0 0x7f320c0b71b0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32201057c0 0x7f322019fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3220104e70 0x7f322019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 --2- 172.21.15.16:0/114599279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3220102eb0 0x7f322019a220 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 -- 172.21.15.16:0/114599279 >> 172.21.15.16:0/114599279 conn(0x7f3220079bb0 msgr2=0x7f322006a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:48.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 -- 172.21.15.16:0/114599279 shutdown_connections 2024-08-28T19:43:48.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.145+0000 7f3228099640 1 -- 172.21.15.16:0/114599279 wait complete. 2024-08-28T19:43:48.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:48 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2986157920' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:43:48.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:48 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1822483868' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-28T19:43:48.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:48 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1729801096' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-08-28T19:43:48.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-08-28T19:43:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.270+0000 7f858905a640 1 -- 172.21.15.16:0/1248837775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584101d80 msgr2=0x7f8584102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.270+0000 7f858905a640 1 --2- 172.21.15.16:0/1248837775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584101d80 0x7f8584102190 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f8574009900 tx=0x7f857402a410 comp rx=0 tx=0).stop 2024-08-28T19:43:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.270+0000 7f858905a640 1 -- 172.21.15.16:0/1248837775 shutdown_connections 2024-08-28T19:43:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.270+0000 7f858905a640 1 --2- 172.21.15.16:0/1248837775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85841068b0 0x7f8584110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.270+0000 7f858905a640 1 --2- 172.21.15.16:0/1248837775 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8584105f60 0x7f8584106370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.270+0000 7f858905a640 1 --2- 172.21.15.16:0/1248837775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584101d80 0x7f8584102190 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.270+0000 7f858905a640 1 -- 172.21.15.16:0/1248837775 >> 172.21.15.16:0/1248837775 conn(0x7f8584079bb0 msgr2=0x7f8584079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.270+0000 7f858905a640 1 -- 172.21.15.16:0/1248837775 shutdown_connections 2024-08-28T19:43:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.270+0000 7f858905a640 1 -- 172.21.15.16:0/1248837775 wait complete. 2024-08-28T19:43:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.271+0000 7f858905a640 1 Processor -- start 2024-08-28T19:43:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.271+0000 7f858905a640 1 -- start start 2024-08-28T19:43:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f858905a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584101d80 0x7f858419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f858905a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8584105f60 0x7f858419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f858905a640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85841068b0 0x7f858419fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f858905a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8584113790 con 0x7f8584101d80 2024-08-28T19:43:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f858905a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8584113610 con 0x7f8584105f60 2024-08-28T19:43:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f858905a640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8584113910 con 0x7f85841068b0 2024-08-28T19:43:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f8582d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584101d80 0x7f858419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f8582d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584101d80 0x7f858419a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36084/0 (socket says 172.21.15.16:36084) 2024-08-28T19:43:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f8582d9d640 1 -- 172.21.15.16:0/1701129580 learned_addr learned my addr 172.21.15.16:0/1701129580 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f858259c640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8584105f60 0x7f858419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f858359e640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85841068b0 0x7f858419fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f8582d9d640 1 -- 172.21.15.16:0/1701129580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85841068b0 msgr2=0x7f858419fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f8582d9d640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85841068b0 0x7f858419fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f8582d9d640 1 -- 172.21.15.16:0/1701129580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8584105f60 msgr2=0x7f858419a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f8582d9d640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8584105f60 0x7f858419a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f8582d9d640 1 -- 172.21.15.16:0/1701129580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85841a00d0 con 0x7f8584101d80 2024-08-28T19:43:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f8582d9d640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584101d80 0x7f858419a230 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f8574006010 tx=0x7f8574004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f856bfff640 1 -- 172.21.15.16:0/1701129580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857403f030 con 0x7f8584101d80 2024-08-28T19:43:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.272+0000 7f858905a640 1 -- 172.21.15.16:0/1701129580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85841a0360 con 0x7f8584101d80 2024-08-28T19:43:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.273+0000 7f856bfff640 1 -- 172.21.15.16:0/1701129580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8574004490 con 0x7f8584101d80 2024-08-28T19:43:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.273+0000 7f856bfff640 1 -- 172.21.15.16:0/1701129580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8574035070 con 0x7f8584101d80 2024-08-28T19:43:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.273+0000 7f858905a640 1 -- 172.21.15.16:0/1701129580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85841a08a0 con 0x7f8584101d80 2024-08-28T19:43:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.274+0000 7f858905a640 1 -- 172.21.15.16:0/1701129580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8584102190 con 0x7f8584101d80 2024-08-28T19:43:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.275+0000 7f856bfff640 1 -- 172.21.15.16:0/1701129580 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8574042030 con 0x7f8584101d80 2024-08-28T19:43:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.276+0000 7f856bfff640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f856c0b4ca0 0x7f856c0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.276+0000 7f858259c640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f856c0b4ca0 0x7f856c0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.276+0000 7f856bfff640 1 -- 172.21.15.16:0/1701129580 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f85741061c0 con 0x7f8584101d80 2024-08-28T19:43:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.276+0000 7f858259c640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f856c0b4ca0 0x7f856c0b71b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8584108ed0 tx=0x7f857800a2d0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.279+0000 7f856bfff640 1 -- 172.21.15.16:0/1701129580 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f85740ccb80 con 0x7f8584101d80 2024-08-28T19:43:48.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:48 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2986157920' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:43:48.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:48 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1822483868' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-28T19:43:48.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:48 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1729801096' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-08-28T19:43:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:48 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2986157920' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:43:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:48 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1822483868' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-28T19:43:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:48 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1729801096' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-08-28T19:43:48.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.463+0000 7f858905a640 1 -- 172.21.15.16:0/1701129580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f858410b9d0 con 0x7f8584101d80 2024-08-28T19:43:48.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.464+0000 7f856bfff640 1 -- 172.21.15.16:0/1701129580 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v116) ==== 103+0+11 (secure 0 0 0) 0x7f85740cc480 con 0x7f8584101d80 2024-08-28T19:43:48.465 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-08-28T19:43:48.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 -- 172.21.15.16:0/1701129580 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f856c0b4ca0 msgr2=0x7f856c0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:48.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f856c0b4ca0 0x7f856c0b71b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8584108ed0 tx=0x7f857800a2d0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 -- 172.21.15.16:0/1701129580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584101d80 msgr2=0x7f858419a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:48.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584101d80 0x7f858419a230 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f8574006010 tx=0x7f8574004060 comp rx=0 tx=0).stop 2024-08-28T19:43:48.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 -- 172.21.15.16:0/1701129580 shutdown_connections 2024-08-28T19:43:48.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f85841068b0 0x7f858419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f856c0b4ca0 0x7f856c0b71b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8584105f60 0x7f858419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 --2- 172.21.15.16:0/1701129580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584101d80 0x7f858419a230 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 -- 172.21.15.16:0/1701129580 >> 172.21.15.16:0/1701129580 conn(0x7f8584079bb0 msgr2=0x7f8584107230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:48.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 -- 172.21.15.16:0/1701129580 shutdown_connections 2024-08-28T19:43:48.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.466+0000 7f858905a640 1 -- 172.21.15.16:0/1701129580 wait complete. 2024-08-28T19:43:48.479 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-08-28T19:43:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.589+0000 7f8f3e051640 1 -- 172.21.15.16:0/2820502740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f38103830 msgr2=0x7f8f38103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.589+0000 7f8f3e051640 1 --2- 172.21.15.16:0/2820502740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f38103830 0x7f8f38103c40 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009900 tx=0x7f8f2802a410 comp rx=0 tx=0).stop 2024-08-28T19:43:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.589+0000 7f8f3e051640 1 -- 172.21.15.16:0/2820502740 shutdown_connections 2024-08-28T19:43:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.589+0000 7f8f3e051640 1 --2- 172.21.15.16:0/2820502740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f38109ab0 0x7f8f381102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.589+0000 7f8f3e051640 1 --2- 172.21.15.16:0/2820502740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f38109160 0x7f8f38109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.589+0000 7f8f3e051640 1 --2- 172.21.15.16:0/2820502740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f38103830 0x7f8f38103c40 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.589+0000 7f8f3e051640 1 -- 172.21.15.16:0/2820502740 >> 172.21.15.16:0/2820502740 conn(0x7f8f380ff270 msgr2=0x7f8f381016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.589+0000 7f8f3e051640 1 -- 172.21.15.16:0/2820502740 shutdown_connections 2024-08-28T19:43:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.589+0000 7f8f3e051640 1 -- 172.21.15.16:0/2820502740 wait complete. 2024-08-28T19:43:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.590+0000 7f8f3e051640 1 Processor -- start 2024-08-28T19:43:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.590+0000 7f8f3e051640 1 -- start start 2024-08-28T19:43:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.590+0000 7f8f3e051640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f38103830 0x7f8f3819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.590+0000 7f8f3e051640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f38109160 0x7f8f3819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.590+0000 7f8f3e051640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f38109ab0 0x7f8f3819f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.590+0000 7f8f3e051640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f38113760 con 0x7f8f38109ab0 2024-08-28T19:43:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.590+0000 7f8f3e051640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f381135e0 con 0x7f8f38109160 2024-08-28T19:43:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.590+0000 7f8f3e051640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8f381138e0 con 0x7f8f38103830 2024-08-28T19:43:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.590+0000 7f8f37fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f38109ab0 0x7f8f3819f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.590+0000 7f8f37fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f38109ab0 0x7f8f3819f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36096/0 (socket says 172.21.15.16:36096) 2024-08-28T19:43:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.590+0000 7f8f37fff640 1 -- 172.21.15.16:0/1164816975 learned_addr learned my addr 172.21.15.16:0/1164816975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:43:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f2effd640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f38109160 0x7f8f3819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f377fe640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f38103830 0x7f8f3819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f37fff640 1 -- 172.21.15.16:0/1164816975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f38103830 msgr2=0x7f8f3819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f37fff640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f38103830 0x7f8f3819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f37fff640 1 -- 172.21.15.16:0/1164816975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f38109160 msgr2=0x7f8f3819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f37fff640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f38109160 0x7f8f3819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f37fff640 1 -- 172.21.15.16:0/1164816975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f381a0040 con 0x7f8f38109ab0 2024-08-28T19:43:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f2effd640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f38109160 0x7f8f3819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:43:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f37fff640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f38109ab0 0x7f8f3819f9d0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f8f3000cce0 tx=0x7f8f3000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f357fa640 1 -- 172.21.15.16:0/1164816975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3000f450 con 0x7f8f38109ab0 2024-08-28T19:43:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f3e051640 1 -- 172.21.15.16:0/1164816975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f381a0330 con 0x7f8f38109ab0 2024-08-28T19:43:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f357fa640 1 -- 172.21.15.16:0/1164816975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f3000faa0 con 0x7f8f38109ab0 2024-08-28T19:43:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f357fa640 1 -- 172.21.15.16:0/1164816975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f30018b70 con 0x7f8f38109ab0 2024-08-28T19:43:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.591+0000 7f8f3e051640 1 -- 172.21.15.16:0/1164816975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f381a6b50 con 0x7f8f38109ab0 2024-08-28T19:43:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.592+0000 7f8f3e051640 1 -- 172.21.15.16:0/1164816975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f38103c40 con 0x7f8f38109ab0 2024-08-28T19:43:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.594+0000 7f8f357fa640 1 -- 172.21.15.16:0/1164816975 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8f30018d10 con 0x7f8f38109ab0 2024-08-28T19:43:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.594+0000 7f8f357fa640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8f240b4ca0 0x7f8f240b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:43:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.595+0000 7f8f377fe640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8f240b4ca0 0x7f8f240b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:43:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.595+0000 7f8f357fa640 1 -- 172.21.15.16:0/1164816975 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9490+0+0 (secure 0 0 0) 0x7f8f300e5ef0 con 0x7f8f38109ab0 2024-08-28T19:43:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.595+0000 7f8f377fe640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8f240b4ca0 0x7f8f240b71b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f8f28005f70 tx=0x7f8f28000bc0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:43:48.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.598+0000 7f8f357fa640 1 -- 172.21.15.16:0/1164816975 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8f30014030 con 0x7f8f38109ab0 2024-08-28T19:43:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.836+0000 7f8f3e051640 1 -- 172.21.15.16:0/1164816975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f8f3819b2c0 con 0x7f8f38109ab0 2024-08-28T19:43:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.844+0000 7f8f357fa640 1 -- 172.21.15.16:0/1164816975 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v24) ==== 81+0+826323 (secure 0 0 0) 0x7f8f3819b2c0 con 0x7f8f38109ab0 2024-08-28T19:43:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:43:48.856 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:43:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 24, 2024-08-28T19:43:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "active_gid": 14220, 2024-08-28T19:43:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "active_name": "x", 2024-08-28T19:43:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addrs": { 2024-08-28T19:43:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-08-28T19:43:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-08-28T19:43:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-08-28T19:43:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2594037529 2024-08-28T19:43:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-08-28T19:43:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-08-28T19:43:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2594037529 2024-08-28T19:43:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:43:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addr": "172.21.15.16:6801/2594037529", 2024-08-28T19:43:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "active_change": "2024-08-28T19:12:07.368108+0000", 2024-08-28T19:43:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "active_mgr_features": 4540701547738038271, 2024-08-28T19:43:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "available": true, 2024-08-28T19:43:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "standbys": [ 2024-08-28T19:43:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 24160, 2024-08-28T19:43:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "z", 2024-08-28T19:43:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540701547738038271, 2024-08-28T19:43:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-08-28T19:43:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-08-28T19:43:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-08-28T19:43:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-08-28T19:43:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-08-28T19:43:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-08-28T19:43:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-08-28T19:43:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-08-28T19:43:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-08-28T19:43:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-08-28T19:43:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-08-28T19:43:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-08-28T19:43:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-08-28T19:43:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-08-28T19:43:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-08-28T19:43:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-08-28T19:43:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-08-28T19:43:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-08-28T19:43:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-08-28T19:43:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-08-28T19:43:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-08-28T19:43:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-08-28T19:43:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-08-28T19:43:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-08-28T19:43:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-08-28T19:43:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-08-28T19:43:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-08-28T19:43:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-08-28T19:43:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-08-28T19:43:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-08-28T19:43:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-08-28T19:43:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-08-28T19:43:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-08-28T19:43:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-08-28T19:43:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-08-28T19:43:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-08-28T19:43:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-08-28T19:43:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-28T19:43:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-08-28T19:43:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-08-28T19:43:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-08-28T19:43:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-08-28T19:43:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-08-28T19:43:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-28T19:43:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-08-28T19:43:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-08-28T19:43:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-08-28T19:43:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-08-28T19:43:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-08-28T19:43:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-08-28T19:43:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-08-28T19:43:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-08-28T19:43:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-08-28T19:43:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-08-28T19:43:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-08-28T19:43:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-08-28T19:43:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-08-28T19:43:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-08-28T19:43:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-08-28T19:43:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-08-28T19:43:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-08-28T19:43:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-08-28T19:43:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-08-28T19:43:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-08-28T19:43:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-08-28T19:43:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-28T19:43:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-08-28T19:43:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-08-28T19:43:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-08-28T19:43:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-08-28T19:43:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-08-28T19:43:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-28T19:43:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-08-28T19:43:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-08-28T19:43:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-08-28T19:43:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-08-28T19:43:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-08-28T19:43:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-08-28T19:43:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-08-28T19:43:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-08-28T19:43:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-08-28T19:43:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-08-28T19:43:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-08-28T19:43:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-08-28T19:43:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-08-28T19:43:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-08-28T19:43:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-08-28T19:43:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-08-28T19:43:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-08-28T19:43:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-08-28T19:43:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-08-28T19:43:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-08-28T19:43:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-08-28T19:43:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-08-28T19:43:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-08-28T19:43:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-08-28T19:43:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-08-28T19:43:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-08-28T19:43:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-08-28T19:43:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-08-28T19:43:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-08-28T19:43:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-08-28T19:43:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-08-28T19:43:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-08-28T19:43:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-08-28T19:43:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-08-28T19:43:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-08-28T19:43:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-08-28T19:43:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-08-28T19:43:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-08-28T19:43:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-08-28T19:43:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-08-28T19:43:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-08-28T19:43:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-08-28T19:43:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-08-28T19:43:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-08-28T19:43:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-08-28T19:43:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-08-28T19:43:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-08-28T19:43:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-08-28T19:43:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-08-28T19:43:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-08-28T19:43:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-08-28T19:43:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-08-28T19:43:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-08-28T19:43:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-08-28T19:43:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-08-28T19:43:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-08-28T19:43:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-08-28T19:43:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-08-28T19:43:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-08-28T19:43:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-08-28T19:43:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-08-28T19:43:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-08-28T19:43:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-08-28T19:43:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-08-28T19:43:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-08-28T19:43:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-08-28T19:43:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-08-28T19:43:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-08-28T19:43:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-08-28T19:43:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-08-28T19:43:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-08-28T19:43:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-08-28T19:43:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-08-28T19:43:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-08-28T19:43:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-08-28T19:43:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:10.4.0", 2024-08-28T19:43:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-08-28T19:43:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-08-28T19:43:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-08-28T19:43:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-08-28T19:43:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-08-28T19:43:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-08-28T19:43:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-08-28T19:43:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-08-28T19:43:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-08-28T19:43:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-08-28T19:43:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-08-28T19:43:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-08-28T19:43:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-08-28T19:43:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-08-28T19:43:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-08-28T19:43:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-08-28T19:43:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-08-28T19:43:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-08-28T19:43:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-08-28T19:43:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-08-28T19:43:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-08-28T19:43:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-08-28T19:43:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-08-28T19:43:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-08-28T19:43:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nginx": { 2024-08-28T19:43:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nginx", 2024-08-28T19:43:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nginx:1.26.1", 2024-08-28T19:43:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nginx container image", 2024-08-28T19:43:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-08-28T19:43:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-08-28T19:43:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-08-28T19:43:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-08-28T19:43:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-08-28T19:43:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-08-28T19:43:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-08-28T19:43:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_oauth2_proxy": { 2024-08-28T19:43:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_oauth2_proxy", 2024-08-28T19:43:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-08-28T19:43:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "oauth2-proxy container image", 2024-08-28T19:43:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-08-28T19:43:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-08-28T19:43:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-08-28T19:43:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-08-28T19:43:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-08-28T19:43:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-08-28T19:43:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-08-28T19:43:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-08-28T19:43:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-08-28T19:43:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-08-28T19:43:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-08-28T19:43:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-08-28T19:43:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-08-28T19:43:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-08-28T19:43:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-08-28T19:43:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-08-28T19:43:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-08-28T19:43:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-08-28T19:43:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-08-28T19:43:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-08-28T19:43:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-08-28T19:43:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-08-28T19:43:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-08-28T19:43:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-08-28T19:43:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-08-28T19:43:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-08-28T19:43:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-08-28T19:43:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-08-28T19:43:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-08-28T19:43:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-08-28T19:43:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-08-28T19:43:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-08-28T19:43:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-08-28T19:43:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-08-28T19:43:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-08-28T19:43:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-08-28T19:43:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-08-28T19:43:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-08-28T19:43:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-08-28T19:43:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.993 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.993 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-08-28T19:43:48.993 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-08-28T19:43:48.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-08-28T19:43:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-08-28T19:43:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.996 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-08-28T19:43:48.996 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-08-28T19:43:48.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-08-28T19:43:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-08-28T19:43:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-08-28T19:43:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-08-28T19:43:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-08-28T19:43:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-08-28T19:43:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-08-28T19:43:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-08-28T19:43:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-08-28T19:43:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-08-28T19:43:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-08-28T19:43:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-08-28T19:43:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-08-28T19:43:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-08-28T19:43:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-08-28T19:43:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-08-28T19:43:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-08-28T19:43:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-08-28T19:43:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-08-28T19:43:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-08-28T19:43:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-08-28T19:43:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-08-28T19:43:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-08-28T19:43:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.019 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.019 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.019 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:49.019 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.019 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.019 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.019 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-08-28T19:43:49.019 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.019 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.020 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.020 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-08-28T19:43:49.020 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-08-28T19:43:49.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.020 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.021 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.021 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.021 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.021 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-08-28T19:43:49.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.021 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.022 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-08-28T19:43:49.022 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-08-28T19:43:49.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.022 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.022 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-08-28T19:43:49.022 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.023 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.023 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.023 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-08-28T19:43:49.023 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-08-28T19:43:49.023 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.023 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-08-28T19:43:49.023 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.024 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.024 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.024 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-08-28T19:43:49.024 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-08-28T19:43:49.024 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.024 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.025 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.025 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-08-28T19:43:49.025 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.025 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.025 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.025 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-08-28T19:43:49.025 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-08-28T19:43:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-08-28T19:43:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-08-28T19:43:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-08-28T19:43:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-08-28T19:43:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-08-28T19:43:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-08-28T19:43:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-08-28T19:43:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-08-28T19:43:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-08-28T19:43:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-08-28T19:43:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-08-28T19:43:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-08-28T19:43:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-08-28T19:43:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-08-28T19:43:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-08-28T19:43:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-08-28T19:43:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-08-28T19:43:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-08-28T19:43:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-08-28T19:43:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-08-28T19:43:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-08-28T19:43:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-08-28T19:43:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-08-28T19:43:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-08-28T19:43:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-08-28T19:43:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-08-28T19:43:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-08-28T19:43:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-08-28T19:43:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-08-28T19:43:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-08-28T19:43:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-08-28T19:43:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-08-28T19:43:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-08-28T19:43:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-08-28T19:43:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-08-28T19:43:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-08-28T19:43:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-08-28T19:43:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-08-28T19:43:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-08-28T19:43:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-08-28T19:43:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-08-28T19:43:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.063 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.064 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.064 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.064 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.064 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-08-28T19:43:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-08-28T19:43:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-08-28T19:43:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-08-28T19:43:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.075 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.075 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.075 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-08-28T19:43:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-08-28T19:43:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-08-28T19:43:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-08-28T19:43:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-08-28T19:43:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-08-28T19:43:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-08-28T19:43:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-08-28T19:43:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-08-28T19:43:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-08-28T19:43:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-08-28T19:43:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-08-28T19:43:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-08-28T19:43:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-08-28T19:43:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-08-28T19:43:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-08-28T19:43:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-08-28T19:43:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-08-28T19:43:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-08-28T19:43:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-08-28T19:43:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-08-28T19:43:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-08-28T19:43:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-08-28T19:43:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-08-28T19:43:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-08-28T19:43:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-08-28T19:43:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-08-28T19:43:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-08-28T19:43:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-08-28T19:43:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-08-28T19:43:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-08-28T19:43:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-08-28T19:43:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-08-28T19:43:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-08-28T19:43:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-08-28T19:43:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-08-28T19:43:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-08-28T19:43:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-08-28T19:43:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-08-28T19:43:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-08-28T19:43:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-08-28T19:43:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-08-28T19:43:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-08-28T19:43:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-08-28T19:43:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-08-28T19:43:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-08-28T19:43:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-08-28T19:43:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-08-28T19:43:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-08-28T19:43:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-08-28T19:43:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-08-28T19:43:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: "MANAGED_BY_CLUSTERS": { 2024-08-28T19:43:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-08-28T19:43:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "[]", 2024-08-28T19:43:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "MULTICLUSTER_CONFIG": { 2024-08-28T19:43:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MULTICLUSTER_CONFIG", 2024-08-28T19:43:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "{}", 2024-08-28T19:43:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-08-28T19:43:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-08-28T19:43:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-08-28T19:43:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-08-28T19:43:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-08-28T19:43:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-08-28T19:43:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-08-28T19:43:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-08-28T19:43:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-08-28T19:43:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-08-28T19:43:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-08-28T19:43:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-08-28T19:43:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-08-28T19:43:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-08-28T19:43:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-08-28T19:43:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-08-28T19:43:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.142 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.142 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-08-28T19:43:49.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-08-28T19:43:49.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.142 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.142 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.142 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.142 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.143 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.143 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.143 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.143 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.143 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.143 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.143 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-08-28T19:43:49.144 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-08-28T19:43:49.144 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.144 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.144 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.144 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.144 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.144 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.144 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.145 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.145 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.145 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.145 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.145 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-08-28T19:43:49.145 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-08-28T19:43:49.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.146 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.146 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.146 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-08-28T19:43:49.146 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.146 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.146 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.146 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.146 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.147 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.147 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.147 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-08-28T19:43:49.147 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-08-28T19:43:49.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.147 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.147 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.148 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:49.148 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.148 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.148 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.148 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.148 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.148 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.148 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.149 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-08-28T19:43:49.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-08-28T19:43:49.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.149 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.149 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.149 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-08-28T19:43:49.149 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.150 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.150 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.150 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.150 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.150 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.150 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.150 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-08-28T19:43:49.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-08-28T19:43:49.151 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.151 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.151 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.151 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-08-28T19:43:49.151 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.151 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.151 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.152 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.152 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.152 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.152 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.152 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-08-28T19:43:49.152 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-08-28T19:43:49.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.153 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.153 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.153 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.153 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.153 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.153 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.154 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.154 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.154 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.154 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.154 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-08-28T19:43:49.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-08-28T19:43:49.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.155 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.155 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-08-28T19:43:49.155 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.155 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.155 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.155 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.155 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.156 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.156 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-08-28T19:43:49.156 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-08-28T19:43:49.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.156 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.157 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.157 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.157 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.157 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.157 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.157 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.158 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.158 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.158 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-08-28T19:43:49.158 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-08-28T19:43:49.158 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.158 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.158 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.159 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.159 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.159 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.159 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.159 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.159 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.159 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.160 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-08-28T19:43:49.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-08-28T19:43:49.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.160 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.160 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.160 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.160 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.161 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.161 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.161 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.161 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.161 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.161 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-08-28T19:43:49.162 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-08-28T19:43:49.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.162 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.162 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.162 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.162 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.162 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.163 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.163 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.163 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.163 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.163 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-08-28T19:43:49.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-08-28T19:43:49.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.164 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.164 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:49.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.164 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.165 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-08-28T19:43:49.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-08-28T19:43:49.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:49.166 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.166 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.166 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.166 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.167 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.167 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-08-28T19:43:49.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-08-28T19:43:49.167 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.167 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.168 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.168 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.168 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.168 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.168 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.168 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.168 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.169 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-08-28T19:43:49.169 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-08-28T19:43:49.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.169 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.170 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.170 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.170 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.170 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.170 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.170 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.171 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.171 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-08-28T19:43:49.171 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-08-28T19:43:49.171 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.171 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.172 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.172 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.172 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.172 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-08-28T19:43:49.172 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.173 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-08-28T19:43:49.173 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-08-28T19:43:49.173 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.173 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.173 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.173 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-08-28T19:43:49.173 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.174 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.174 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.174 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.174 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.174 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.174 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-08-28T19:43:49.175 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-08-28T19:43:49.175 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.175 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.175 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.175 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.175 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.175 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.175 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.176 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.176 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.176 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.176 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.176 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.177 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.177 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.177 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.177 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.177 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.177 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.177 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.178 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.178 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.178 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.178 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.178 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.178 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.178 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.179 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.179 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.180 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.180 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.180 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.180 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.180 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.181 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.181 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.181 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.181 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.181 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.182 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.182 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.182 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.183 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.183 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.183 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.184 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.184 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.184 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.184 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.184 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.185 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.185 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.185 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.185 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.185 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.186 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.186 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.186 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.186 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.189 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.189 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-08-28T19:43:49.189 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-08-28T19:43:49.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.190 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.190 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.190 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.190 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.191 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.191 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-08-28T19:43:49.191 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.191 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.192 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-08-28T19:43:49.192 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-08-28T19:43:49.192 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.192 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.192 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.193 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.193 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.193 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.193 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.193 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.194 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.194 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.194 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.194 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-08-28T19:43:49.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-08-28T19:43:49.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.194 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.195 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.195 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-08-28T19:43:49.195 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.195 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.195 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.195 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.196 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.196 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-08-28T19:43:49.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-08-28T19:43:49.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.196 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.196 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-08-28T19:43:49.197 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.197 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.197 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.197 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.197 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.198 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.198 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.198 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.198 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.198 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.198 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.198 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.199 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.199 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.199 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.199 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-08-28T19:43:49.199 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-08-28T19:43:49.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.200 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.200 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.200 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.200 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.200 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.200 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.200 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.201 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.201 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.201 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-08-28T19:43:49.201 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-08-28T19:43:49.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.201 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.202 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-08-28T19:43:49.202 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.202 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.202 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.202 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.202 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.202 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.203 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-08-28T19:43:49.203 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-08-28T19:43:49.203 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.203 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.203 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-08-28T19:43:49.203 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.204 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.204 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.204 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.204 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-08-28T19:43:49.204 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.204 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.204 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.204 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.205 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-08-28T19:43:49.205 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-08-28T19:43:49.205 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.205 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.205 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.205 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-08-28T19:43:49.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-08-28T19:43:49.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-08-28T19:43:49.206 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.206 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.206 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.206 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.206 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.207 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-08-28T19:43:49.207 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-08-28T19:43:49.207 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.207 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.207 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.207 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.207 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.207 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.208 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.208 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.208 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.208 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-08-28T19:43:49.209 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.209 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.209 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.209 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-08-28T19:43:49.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-08-28T19:43:49.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.210 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.210 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.210 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.210 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-08-28T19:43:49.211 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.211 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.211 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.212 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.212 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.212 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.212 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.212 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.212 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.212 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.213 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.213 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.213 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.213 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.213 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.213 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.213 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.214 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.214 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.214 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.215 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.215 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.215 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.215 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.215 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.215 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.216 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.216 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.216 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.216 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.216 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.216 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.217 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.217 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.217 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.217 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.217 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.217 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.218 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.218 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.218 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.218 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.218 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.218 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.218 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.218 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.219 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.219 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.219 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.219 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.219 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.220 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.220 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.220 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.220 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.220 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.220 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.220 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.221 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-08-28T19:43:49.221 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-08-28T19:43:49.221 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.221 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.221 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.221 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-08-28T19:43:49.222 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.222 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.222 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.222 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-08-28T19:43:49.222 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.222 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.222 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.223 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-08-28T19:43:49.223 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-08-28T19:43:49.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.223 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.223 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-08-28T19:43:49.223 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.224 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.224 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.224 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-08-28T19:43:49.224 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.224 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.225 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-08-28T19:43:49.225 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-08-28T19:43:49.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.225 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.225 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-08-28T19:43:49.225 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.225 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.226 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.226 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-08-28T19:43:49.226 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.226 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.226 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-08-28T19:43:49.226 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-08-28T19:43:49.226 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.227 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.227 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.227 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-08-28T19:43:49.227 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.227 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.227 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.227 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-08-28T19:43:49.227 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.228 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.228 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-08-28T19:43:49.228 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-08-28T19:43:49.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.228 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.229 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.229 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.229 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.229 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.229 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-08-28T19:43:49.229 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.229 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.229 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.230 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:49.230 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:49.230 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.230 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:49.230 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.231 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.231 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.231 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-08-28T19:43:49.231 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.231 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.231 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.232 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.232 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.232 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.232 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.232 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.232 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.233 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.233 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.233 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.233 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-08-28T19:43:49.233 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-08-28T19:43:49.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.234 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.234 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-08-28T19:43:49.234 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.234 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.234 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.234 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-08-28T19:43:49.235 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.235 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.235 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.235 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.235 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-08-28T19:43:49.236 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.236 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.236 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.236 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.236 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.236 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.236 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.237 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.237 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.237 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.237 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.237 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.238 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.238 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.238 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.238 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.238 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.238 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.239 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.239 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.239 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.239 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.239 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.239 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.239 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.240 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.240 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.240 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.240 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.241 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.241 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.241 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.241 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.241 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.241 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.242 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.242 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.242 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.242 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.242 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.242 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.242 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.242 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.242 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.243 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.243 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.243 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.243 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.244 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.244 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.244 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.244 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.245 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.245 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.245 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-08-28T19:43:49.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-08-28T19:43:49.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.245 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.246 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-08-28T19:43:49.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.246 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.246 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.246 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.246 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.246 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.247 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.247 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-08-28T19:43:49.247 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-08-28T19:43:49.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.247 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.247 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-08-28T19:43:49.248 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.248 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.248 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.248 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.248 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.249 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:49.249 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:49.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.249 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.249 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:49.249 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.250 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.250 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.250 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.250 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.250 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.250 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.250 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.251 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.251 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.251 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.251 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.251 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.251 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.252 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.252 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.252 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.252 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.252 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.253 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-08-28T19:43:49.253 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-08-28T19:43:49.253 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-08-28T19:43:49.253 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.253 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-08-28T19:43:49.253 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-08-28T19:43:49.254 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.254 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.254 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.254 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-08-28T19:43:49.254 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.254 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.254 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.254 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-08-28T19:43:49.255 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.255 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.255 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.255 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-08-28T19:43:49.255 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-08-28T19:43:49.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.255 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.256 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-08-28T19:43:49.256 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.256 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.256 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.256 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-08-28T19:43:49.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.257 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.257 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-08-28T19:43:49.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-08-28T19:43:49.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.257 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.258 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.258 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.258 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.258 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-08-28T19:43:49.258 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.258 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.259 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:49.259 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:49.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-08-28T19:43:49.259 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-08-28T19:43:49.259 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.260 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-08-28T19:43:49.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.260 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.260 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.260 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.261 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.261 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.261 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.261 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.261 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.262 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.262 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.262 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.262 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.262 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.262 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.263 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.263 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.263 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.263 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.264 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.264 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.264 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.264 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.264 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.264 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.264 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.265 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.265 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.265 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.265 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.265 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.265 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.265 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.265 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.266 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.266 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.266 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.266 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.266 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.266 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.266 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.266 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.267 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.267 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.267 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.267 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.267 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.267 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.268 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.268 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.268 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.268 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.268 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.268 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.268 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.268 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.269 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.269 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.269 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-08-28T19:43:49.269 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-08-28T19:43:49.269 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.269 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.269 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.270 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.270 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.270 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.270 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-08-28T19:43:49.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.270 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.271 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-08-28T19:43:49.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-08-28T19:43:49.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.271 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.271 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-08-28T19:43:49.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.271 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.272 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.272 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-08-28T19:43:49.272 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.272 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.272 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.272 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.273 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.273 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.273 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.273 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.273 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.273 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.274 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.274 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.274 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-08-28T19:43:49.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-08-28T19:43:49.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.274 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.275 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.275 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-08-28T19:43:49.275 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.275 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.275 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.275 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-08-28T19:43:49.275 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.275 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.275 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.276 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-08-28T19:43:49.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-08-28T19:43:49.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.276 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.276 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.276 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:49.276 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:49.277 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-08-28T19:43:49.277 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-08-28T19:43:49.277 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.277 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.277 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.278 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-08-28T19:43:49.278 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-08-28T19:43:49.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.278 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.279 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.279 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.279 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-08-28T19:43:49.279 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.279 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.280 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-08-28T19:43:49.280 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-08-28T19:43:49.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.280 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.280 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-08-28T19:43:49.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.281 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.281 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.281 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-08-28T19:43:49.281 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.281 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.281 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.281 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.282 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.282 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-08-28T19:43:49.282 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.282 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.282 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.283 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.283 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.283 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.283 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.283 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.283 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.284 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.284 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.284 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.284 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.284 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.284 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.284 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-08-28T19:43:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-08-28T19:43:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-08-28T19:43:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-08-28T19:43:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-08-28T19:43:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-08-28T19:43:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-08-28T19:43:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-08-28T19:43:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-08-28T19:43:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-08-28T19:43:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-08-28T19:43:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-08-28T19:43:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-08-28T19:43:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-08-28T19:43:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-08-28T19:43:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-08-28T19:43:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-08-28T19:43:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-08-28T19:43:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-08-28T19:43:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-08-28T19:43:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-08-28T19:43:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-08-28T19:43:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-08-28T19:43:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-08-28T19:43:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-08-28T19:43:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-08-28T19:43:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-08-28T19:43:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-08-28T19:43:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-08-28T19:43:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-08-28T19:43:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-08-28T19:43:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-08-28T19:43:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-08-28T19:43:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-08-28T19:43:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-08-28T19:43:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-08-28T19:43:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-08-28T19:43:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-08-28T19:43:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-08-28T19:43:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-08-28T19:43:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-08-28T19:43:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-08-28T19:43:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-08-28T19:43:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.395 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-08-28T19:43:49.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-08-28T19:43:49.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.395 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.395 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.395 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.395 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.395 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.395 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.396 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-08-28T19:43:49.396 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-08-28T19:43:49.396 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-08-28T19:43:49.396 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.396 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-08-28T19:43:49.396 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.396 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.396 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.397 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.397 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.397 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.397 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.397 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.397 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.398 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.398 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.398 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.398 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.398 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.398 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.398 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.399 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-08-28T19:43:49.399 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.399 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.399 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.399 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.399 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.400 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.400 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.400 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.400 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.400 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.401 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.401 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.401 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.401 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.401 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.401 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.401 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.401 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.402 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.402 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.402 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.402 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.402 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.403 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.403 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.403 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.403 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.403 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.404 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.404 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.404 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.404 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.404 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.405 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.405 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.405 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.405 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.405 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.405 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.406 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.406 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.406 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.406 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.406 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.406 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.406 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.407 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.407 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.407 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.407 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.407 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.407 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.407 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.407 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.408 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.408 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.408 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.408 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.408 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.408 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.409 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.409 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.409 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.409 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.409 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.410 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.410 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.410 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.410 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.410 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-08-28T19:43:49.411 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.411 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.411 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.411 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.411 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.412 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.412 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.412 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.412 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.413 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.413 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.413 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.413 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.413 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.413 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.413 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.413 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.414 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.414 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.414 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.414 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.414 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.415 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.415 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.415 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.415 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.415 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.415 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.415 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.415 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-08-28T19:43:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-08-28T19:43:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-08-28T19:43:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-08-28T19:43:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-08-28T19:43:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-08-28T19:43:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-08-28T19:43:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-08-28T19:43:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-08-28T19:43:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-08-28T19:43:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-08-28T19:43:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-08-28T19:43:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-08-28T19:43:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-08-28T19:43:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-08-28T19:43:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-08-28T19:43:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-08-28T19:43:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-08-28T19:43:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-08-28T19:43:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-08-28T19:43:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-08-28T19:43:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-08-28T19:43:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-08-28T19:43:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.461 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.461 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.461 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-08-28T19:43:49.461 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-08-28T19:43:49.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.461 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.461 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.462 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.462 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.462 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.462 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.462 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.462 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.462 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.463 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.463 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.463 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.463 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.463 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.463 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.464 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.464 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.464 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.464 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.464 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.464 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.465 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.465 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.465 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.465 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.465 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.465 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.466 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.466 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.466 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.467 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.467 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.468 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.468 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.468 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.468 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.469 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.469 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.469 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.469 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.470 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.470 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.470 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.470 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.470 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.471 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.471 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.471 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.471 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.472 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.472 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.472 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.473 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.473 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.473 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.474 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.474 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.474 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.474 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.474 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.474 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.475 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-08-28T19:43:49.475 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-08-28T19:43:49.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.475 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.475 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.475 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.475 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.476 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.476 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.476 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.476 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.476 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.476 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.476 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-08-28T19:43:49.477 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-08-28T19:43:49.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.477 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.477 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.477 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-08-28T19:43:49.477 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.477 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.477 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.478 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.478 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.478 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.478 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-08-28T19:43:49.478 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-08-28T19:43:49.478 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:49.479 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.479 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-08-28T19:43:49.479 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.479 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.479 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.480 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.480 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-08-28T19:43:49.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-08-28T19:43:49.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.480 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.480 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.481 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-08-28T19:43:49.481 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.481 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.481 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.481 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-08-28T19:43:49.481 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.481 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.481 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.482 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-08-28T19:43:49.482 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-08-28T19:43:49.482 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.482 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.482 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.482 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-08-28T19:43:49.482 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.483 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-08-28T19:43:49.483 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.483 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.483 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.483 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.484 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.484 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.484 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.484 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.484 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-08-28T19:43:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-08-28T19:43:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-08-28T19:43:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-08-28T19:43:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-08-28T19:43:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-08-28T19:43:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-08-28T19:43:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-08-28T19:43:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-08-28T19:43:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-08-28T19:43:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-08-28T19:43:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-08-28T19:43:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-08-28T19:43:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-08-28T19:43:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.497 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.497 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.497 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.497 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.498 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.498 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.498 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.498 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.498 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.498 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.498 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.498 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.498 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.499 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.499 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.499 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.499 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.499 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.499 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.500 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.500 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.500 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.500 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.500 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.500 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.501 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.501 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.501 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-08-28T19:43:49.501 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-08-28T19:43:49.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.502 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.502 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.502 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:49.502 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.502 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.502 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.502 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.502 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.503 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.503 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.503 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-08-28T19:43:49.503 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-08-28T19:43:49.503 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.503 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.503 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.504 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.504 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.504 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.504 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.504 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.504 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.504 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.505 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.505 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.505 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.505 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.505 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.505 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.505 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.506 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.506 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.506 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.506 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.506 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.506 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.506 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-08-28T19:43:49.506 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-08-28T19:43:49.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.507 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.507 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.507 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.507 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.507 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.507 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.507 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.508 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.508 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-08-28T19:43:49.509 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.509 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.509 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.509 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-08-28T19:43:49.509 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-08-28T19:43:49.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.509 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.510 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.510 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.510 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.510 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.510 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.510 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.510 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.511 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.511 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-08-28T19:43:49.511 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-08-28T19:43:49.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.511 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.512 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.512 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.512 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.512 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.512 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.512 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.512 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.513 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.513 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.513 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.513 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.514 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.514 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.514 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.514 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.514 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.514 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.514 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.515 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.515 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.515 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.515 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.515 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.515 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.515 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.515 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.516 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.516 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.516 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.516 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.516 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.517 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.517 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.517 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.517 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.517 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.517 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.518 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.518 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.518 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.518 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.518 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.518 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.518 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.518 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.519 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.519 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.519 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.519 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.519 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.519 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.519 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.520 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.520 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.520 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.520 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.520 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.520 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.521 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.521 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.521 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.521 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.521 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.521 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.521 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.521 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.522 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.522 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.522 INFO:teuthology.orchestra.run.smithi016.stdout: "max_requests": { 2024-08-28T19:43:49.522 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_requests", 2024-08-28T19:43:49.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.522 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.522 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-08-28T19:43:49.523 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.523 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.523 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.523 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-08-28T19:43:49.523 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.523 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.523 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-08-28T19:43:49.524 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-08-28T19:43:49.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.524 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.524 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.524 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.524 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.524 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.524 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.525 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.525 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.525 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.525 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.525 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-08-28T19:43:49.525 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-08-28T19:43:49.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.526 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.526 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.526 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.526 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.526 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.526 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.526 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.526 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.527 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.527 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.527 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.527 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.527 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.527 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.528 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.528 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.528 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.528 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.528 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.528 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.529 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.529 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.529 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-08-28T19:43:49.529 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.529 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.529 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.530 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.530 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.530 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.530 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.530 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.530 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.530 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.530 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.531 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.531 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.531 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.531 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.531 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.531 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.531 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.531 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.532 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.532 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.532 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.532 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.532 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.533 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.533 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.533 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.533 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.533 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.533 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.533 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.534 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.534 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.534 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.534 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.534 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.535 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.535 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.535 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.535 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.535 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.535 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.535 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.535 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.536 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.536 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.536 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.536 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.536 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.536 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.536 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.537 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.537 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.537 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.537 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.537 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.537 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.537 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.538 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.538 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.538 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.538 INFO:teuthology.orchestra.run.smithi016.stdout: "secondary_zone_period_retry_limit": { 2024-08-28T19:43:49.538 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secondary_zone_period_retry_limit", 2024-08-28T19:43:49.538 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.538 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.539 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.539 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:49.539 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.539 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.539 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.539 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-08-28T19:43:49.539 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.539 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.540 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.540 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.540 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.540 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.540 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.540 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.541 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.541 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.541 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.541 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.541 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.541 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-08-28T19:43:49.542 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.542 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.543 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.543 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.543 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.543 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.543 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.544 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.544 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.544 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.544 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.544 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.544 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.544 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.545 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.545 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.545 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.545 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.545 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.545 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.546 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.546 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.546 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.546 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.546 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.546 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.546 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.547 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.547 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.547 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.547 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.547 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.547 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.548 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.548 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.548 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.548 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.548 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.548 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.548 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.549 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.549 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.549 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.549 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.549 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.549 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.549 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.549 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.550 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.550 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.550 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.550 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.550 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.550 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.551 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.551 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.551 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.551 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.551 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.551 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.552 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.552 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.552 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.552 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.552 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.552 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.553 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.553 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.553 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.553 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.553 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.553 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.553 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.554 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-08-28T19:43:49.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-08-28T19:43:49.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.554 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-08-28T19:43:49.554 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.555 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.555 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.555 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-08-28T19:43:49.555 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.556 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.556 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-08-28T19:43:49.556 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.556 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.557 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.557 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.557 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.557 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.557 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.558 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.558 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.558 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.558 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.558 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.558 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.559 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.559 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.559 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.559 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.559 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.559 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.559 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.559 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.560 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.560 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.560 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.560 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.560 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.560 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.560 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.561 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.561 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.561 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.561 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.561 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.561 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.561 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.562 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.562 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.562 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.562 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.562 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.565 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.565 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.565 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.565 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.565 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.565 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.566 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.566 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.566 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.566 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.566 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-08-28T19:43:49.566 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-08-28T19:43:49.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.567 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.567 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.567 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.567 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.567 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.567 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.568 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.568 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.568 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.568 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-08-28T19:43:49.568 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-08-28T19:43:49.569 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.569 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.569 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.569 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-08-28T19:43:49.569 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.570 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.570 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.570 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.570 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.570 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.571 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.571 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-08-28T19:43:49.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-08-28T19:43:49.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.572 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.572 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.572 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.572 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.573 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.573 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.573 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.573 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.573 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.574 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-08-28T19:43:49.574 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-08-28T19:43:49.574 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.574 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.575 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.575 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.575 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.575 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.575 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.576 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.576 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.576 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.576 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-08-28T19:43:49.577 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-08-28T19:43:49.577 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:49.577 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.577 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.577 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.578 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.578 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.578 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.578 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.578 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.579 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.579 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.579 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-08-28T19:43:49.579 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-08-28T19:43:49.580 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.580 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.580 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.580 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.580 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.581 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.581 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.581 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.581 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.582 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.582 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.582 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-08-28T19:43:49.582 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-08-28T19:43:49.582 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.582 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.583 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.583 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.583 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.583 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.583 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.583 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.583 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.584 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.584 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-08-28T19:43:49.584 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-08-28T19:43:49.584 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.584 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.584 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.584 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.585 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.585 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.585 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.585 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.585 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.585 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.585 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.586 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-08-28T19:43:49.586 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-08-28T19:43:49.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.586 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.586 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.586 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:49.586 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-08-28T19:43:49.587 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.587 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.587 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.587 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.587 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.587 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.588 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.588 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.588 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.588 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.588 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.589 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.589 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.589 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.589 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.589 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-08-28T19:43:49.589 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-08-28T19:43:49.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.590 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.590 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.590 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.590 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.590 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.590 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.590 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.590 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.591 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.591 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.591 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-08-28T19:43:49.591 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-08-28T19:43:49.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.591 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.592 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.592 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.592 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.592 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.592 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.592 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.592 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.592 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.593 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-08-28T19:43:49.593 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-08-28T19:43:49.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.593 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.593 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.593 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.593 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.594 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.594 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.594 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.594 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.594 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smb", 2024-08-28T19:43:49.595 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.595 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.595 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.595 INFO:teuthology.orchestra.run.smithi016.stdout: "internal_store_backend": { 2024-08-28T19:43:49.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "internal_store_backend", 2024-08-28T19:43:49.596 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.596 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.596 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.596 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.596 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.596 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.596 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.597 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-08-28T19:43:49.597 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.597 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.597 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.597 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.597 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.597 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.598 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.598 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.598 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.598 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.598 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.598 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.598 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.598 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.599 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.599 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.599 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.599 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.599 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.599 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.599 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.600 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.600 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.600 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.600 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.600 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.600 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.601 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.601 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.601 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.601 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.601 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.601 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.601 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.602 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.602 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.602 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.602 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.602 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.602 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.602 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.602 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.603 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.603 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.603 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.603 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.603 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.603 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.603 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.603 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.604 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.604 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.604 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.604 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.604 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.605 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.605 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.605 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.605 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.605 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.605 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.605 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.606 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.606 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.606 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.606 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.606 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.606 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.606 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.607 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.607 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.607 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.607 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.607 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.607 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.608 INFO:teuthology.orchestra.run.smithi016.stdout: "update_orchestration": { 2024-08-28T19:43:49.608 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "update_orchestration", 2024-08-28T19:43:49.608 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.608 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.608 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.608 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.608 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.609 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.609 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.609 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-08-28T19:43:49.609 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.609 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.609 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.609 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.609 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-08-28T19:43:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-08-28T19:43:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-08-28T19:43:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.612 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-08-28T19:43:49.612 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.612 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.612 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.612 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-08-28T19:43:49.612 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-08-28T19:43:49.613 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.613 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.613 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.613 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.613 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.613 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.613 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.613 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-08-28T19:43:49.614 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.614 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.614 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.614 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.614 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.614 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.615 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.615 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.615 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.615 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.615 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.615 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.615 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.616 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.616 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.616 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.616 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.616 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.616 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.616 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.617 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.617 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.617 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.617 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.618 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.618 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.618 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.618 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.618 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.618 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.619 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.619 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.619 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.619 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.620 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.620 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.620 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.620 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.620 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.620 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.620 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.621 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.621 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.621 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.621 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.621 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.621 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.621 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.621 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.622 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.622 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.622 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.622 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.622 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.623 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.623 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.623 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.623 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.623 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.623 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.623 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.624 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.624 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.624 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.624 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.624 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.624 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.624 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.625 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.625 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.625 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.625 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.625 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.625 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.625 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.626 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-08-28T19:43:49.626 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.626 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.626 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.626 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.626 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.626 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.627 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.627 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.627 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.627 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.627 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.627 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.627 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.628 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.628 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.628 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.628 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.628 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.628 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.628 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.629 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.629 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.629 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.629 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.629 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.629 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.630 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.630 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.630 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.630 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.630 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.630 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.631 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.631 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.631 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.631 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.631 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.631 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.631 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.631 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.632 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.632 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.632 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.632 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.632 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.632 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.632 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.633 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.633 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.633 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.633 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.633 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.633 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.634 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.634 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.634 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.634 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.634 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.634 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.635 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.635 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.635 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.636 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.636 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.636 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.636 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.636 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.636 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.637 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.637 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.637 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.637 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.637 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.638 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-08-28T19:43:49.638 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.638 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.638 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.638 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.638 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.638 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.639 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.639 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.639 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.639 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.639 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.639 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.639 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.639 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.641 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.641 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.641 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.641 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.641 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.641 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.642 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.642 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.642 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.642 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.642 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.642 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.643 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.643 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.643 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.643 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.643 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.644 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.644 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.644 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.644 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.644 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.644 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.644 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.644 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.645 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.645 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.645 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.645 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.645 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.646 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.646 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.646 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.646 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.646 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.646 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.646 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.646 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.647 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.647 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.647 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.647 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.647 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.647 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.648 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.648 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.648 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.648 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.648 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.648 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.648 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.649 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.649 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-08-28T19:43:49.649 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.649 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.650 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.650 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-08-28T19:43:49.650 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-08-28T19:43:49.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.650 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.650 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.650 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-08-28T19:43:49.650 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.651 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.651 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.651 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.651 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.651 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.651 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:49.652 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:49.652 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.652 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.652 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-08-28T19:43:49.652 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.652 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.652 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.653 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.653 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.653 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.653 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.653 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.653 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.654 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.654 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.654 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.654 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.654 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.654 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.654 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.654 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.655 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.655 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.655 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.655 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.655 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.655 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.655 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.656 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.656 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.656 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.656 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.657 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.657 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.657 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.657 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.657 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.657 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.658 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.658 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.658 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.658 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.658 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.659 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.659 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.659 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.659 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.659 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.659 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.659 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.659 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.660 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.660 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.660 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.660 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.660 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.660 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.660 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.661 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.661 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.661 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.661 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.661 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.661 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.661 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.661 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.662 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.662 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.662 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.662 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.662 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.662 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.662 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.662 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.663 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.663 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.663 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.663 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.663 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.663 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.663 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.664 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.664 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.664 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.664 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-08-28T19:43:49.665 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.665 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.665 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.665 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-08-28T19:43:49.665 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-08-28T19:43:49.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.665 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.665 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.665 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.666 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.666 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.666 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.666 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-08-28T19:43:49.666 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.666 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.667 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-08-28T19:43:49.667 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-08-28T19:43:49.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.667 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.667 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.667 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.668 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.668 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.668 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-08-28T19:43:49.668 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.668 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.668 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.668 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-08-28T19:43:49.669 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-08-28T19:43:49.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.669 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.669 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.669 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.669 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.669 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.669 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.670 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-08-28T19:43:49.670 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.670 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.670 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.670 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-08-28T19:43:49.670 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-08-28T19:43:49.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.671 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.671 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.671 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.671 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.671 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.671 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.671 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-08-28T19:43:49.672 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.672 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.672 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.672 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-08-28T19:43:49.672 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-08-28T19:43:49.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.672 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.673 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.673 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.673 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.673 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.673 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.673 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-08-28T19:43:49.673 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.673 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.674 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.674 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-08-28T19:43:49.674 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-08-28T19:43:49.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.674 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.674 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.674 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.675 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.675 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.675 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.675 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.675 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.675 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-08-28T19:43:49.676 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-08-28T19:43:49.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.676 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.676 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.676 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.677 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.677 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.677 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.677 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.677 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.677 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.677 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-08-28T19:43:49.677 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-08-28T19:43:49.678 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.678 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.678 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.678 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-08-28T19:43:49.678 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.678 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.678 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.678 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.679 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.679 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.679 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-08-28T19:43:49.679 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-08-28T19:43:49.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.679 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.680 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.680 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.680 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.680 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.680 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.680 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.680 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.680 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.681 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:49.681 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:49.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.681 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.681 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-08-28T19:43:49.682 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-08-28T19:43:49.682 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.682 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.682 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.682 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.682 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.683 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-08-28T19:43:49.683 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-08-28T19:43:49.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.683 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.683 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-08-28T19:43:49.683 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.684 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.684 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.684 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.684 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.684 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.685 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-08-28T19:43:49.685 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-08-28T19:43:49.685 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.685 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.685 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.686 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.686 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.686 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.686 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.686 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.686 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-08-28T19:43:49.687 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-08-28T19:43:49.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.687 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.687 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.687 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.687 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.687 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.688 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.688 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.688 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.688 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.688 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.689 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.689 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.689 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.689 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.689 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.689 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.689 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.689 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.690 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.690 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.690 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.690 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.690 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.690 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.690 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-08-28T19:43:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-08-28T19:43:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-08-28T19:43:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-08-28T19:43:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.702 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.702 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.702 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.702 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-08-28T19:43:49.702 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-08-28T19:43:49.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-08-28T19:43:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-08-28T19:43:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.710 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.710 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.710 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.710 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.712 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.712 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.712 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.712 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.712 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.712 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.712 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.713 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.713 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.713 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.713 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.714 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.716 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.716 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.716 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.716 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.717 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.717 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.717 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.717 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.718 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.718 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.718 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.718 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.718 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.718 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.718 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.719 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.719 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.719 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-08-28T19:43:49.720 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.720 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.720 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.720 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.720 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.720 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.720 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.721 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.721 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.721 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.721 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.721 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.721 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.721 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.721 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.722 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.722 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.722 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.722 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.722 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.722 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.722 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.723 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.723 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.723 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.723 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.723 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.723 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.723 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.724 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.724 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.724 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.724 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.724 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.724 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.724 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.725 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.725 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.725 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.725 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.725 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.725 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.726 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.726 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.726 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.726 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.726 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.726 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.726 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.726 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.727 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.727 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.727 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.727 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.727 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.728 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.728 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.728 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.728 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.728 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.728 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.729 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.729 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-08-28T19:43:49.729 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-08-28T19:43:49.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.729 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.729 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.729 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-08-28T19:43:49.730 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.730 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.730 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.730 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-08-28T19:43:49.730 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.730 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.731 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-08-28T19:43:49.731 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-08-28T19:43:49.731 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.731 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.731 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.731 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.731 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.732 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.732 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-08-28T19:43:49.732 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.732 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.732 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.732 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-08-28T19:43:49.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-08-28T19:43:49.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.733 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.733 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.733 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.733 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.733 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.733 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.734 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-08-28T19:43:49.734 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.734 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.734 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-08-28T19:43:49.734 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-08-28T19:43:49.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.734 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.735 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.735 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.735 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.735 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-08-28T19:43:49.735 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.735 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.736 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.736 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.736 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.736 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.737 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.737 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.737 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.737 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.738 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.738 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-08-28T19:43:49.738 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.739 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.739 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.739 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-08-28T19:43:49.739 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-08-28T19:43:49.739 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:49.739 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.739 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-08-28T19:43:49.740 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.740 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.740 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.740 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.740 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.740 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.741 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.741 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-08-28T19:43:49.741 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-08-28T19:43:49.741 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.741 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.741 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.742 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.742 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.742 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.742 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.742 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.743 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.743 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:49.743 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:49.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.743 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.744 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:49.744 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.744 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.744 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.744 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.744 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.745 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.745 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.745 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.745 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.746 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.746 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.746 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.746 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.746 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.746 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.746 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.747 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.747 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.747 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.747 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.747 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.747 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.748 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.748 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.748 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.749 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.749 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.749 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.749 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.750 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.750 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.750 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.750 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.751 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.751 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.751 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.751 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.751 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.751 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.751 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.754 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.754 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.754 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.754 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.754 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.754 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.755 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.755 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.755 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.755 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.756 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.756 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.756 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.756 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.756 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.756 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.757 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.757 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.757 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-08-28T19:43:49.757 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-08-28T19:43:49.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.758 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.758 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.758 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.758 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.758 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.758 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.759 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.759 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.759 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.759 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.759 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-08-28T19:43:49.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-08-28T19:43:49.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.760 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.760 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-08-28T19:43:49.760 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.760 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.760 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.761 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.761 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.761 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.761 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-08-28T19:43:49.761 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-08-28T19:43:49.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.762 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.762 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-08-28T19:43:49.762 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.762 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.762 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.762 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.762 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.763 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.763 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.763 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:43:49.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.764 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 24170, 2024-08-28T19:43:49.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "y", 2024-08-28T19:43:49.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540701547738038271, 2024-08-28T19:43:49.764 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-08-28T19:43:49.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-08-28T19:43:49.764 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.765 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.765 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.765 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:49.765 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:49.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.765 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.766 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:49.766 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.766 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.766 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.766 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-08-28T19:43:49.766 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.767 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.767 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.767 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.767 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.768 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.768 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.768 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.768 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.768 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.768 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.768 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.769 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.769 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.769 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.769 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.769 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.770 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.770 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.770 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.770 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.771 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.771 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.771 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.771 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.771 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.771 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.772 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.772 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.772 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.772 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.773 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.773 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.773 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.773 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.775 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.775 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.775 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.775 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.776 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.776 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.776 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.776 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.776 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.776 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.777 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.777 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-08-28T19:43:49.777 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-08-28T19:43:49.777 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.777 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.777 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.778 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.778 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.778 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.778 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.778 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-08-28T19:43:49.778 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.778 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.779 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.779 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-08-28T19:43:49.779 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-08-28T19:43:49.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.779 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.779 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.780 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-08-28T19:43:49.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.780 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.780 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.780 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-08-28T19:43:49.780 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.781 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.781 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-08-28T19:43:49.781 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-08-28T19:43:49.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.781 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.782 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.782 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.782 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.782 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.782 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-08-28T19:43:49.782 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.783 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.783 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-08-28T19:43:49.783 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-08-28T19:43:49.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.783 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.784 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.784 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.784 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.784 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.784 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-08-28T19:43:49.784 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.785 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.785 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.785 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-08-28T19:43:49.785 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-08-28T19:43:49.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.785 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.786 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-08-28T19:43:49.786 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.786 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.786 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.786 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-08-28T19:43:49.787 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.787 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-08-28T19:43:49.787 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-08-28T19:43:49.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.787 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.788 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.788 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.788 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.788 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.788 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-08-28T19:43:49.788 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.789 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.789 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-08-28T19:43:49.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-08-28T19:43:49.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.789 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.789 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.790 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.790 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.790 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.790 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.790 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-08-28T19:43:49.790 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.790 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.791 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.791 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-08-28T19:43:49.791 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-08-28T19:43:49.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.791 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.792 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.792 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.792 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.792 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.792 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-08-28T19:43:49.792 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.793 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.793 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.794 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.794 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.794 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.794 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.794 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.794 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-08-28T19:43:49.795 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.796 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.796 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.796 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-08-28T19:43:49.796 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-08-28T19:43:49.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.796 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.797 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.797 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.797 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.797 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.797 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-08-28T19:43:49.797 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.797 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.798 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.798 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-08-28T19:43:49.798 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-08-28T19:43:49.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.798 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.799 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-08-28T19:43:49.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.799 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.799 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-08-28T19:43:49.799 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-28T19:43:49.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.800 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.800 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-08-28T19:43:49.800 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-08-28T19:43:49.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:49.800 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.801 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.801 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.801 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-08-28T19:43:49.801 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-08-28T19:43:49.801 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.801 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-08-28T19:43:49.801 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-28T19:43:49.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.802 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.802 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-08-28T19:43:49.802 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-08-28T19:43:49.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:49.802 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-08-28T19:43:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-08-28T19:43:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-08-28T19:43:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.804 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.804 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-08-28T19:43:49.804 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-08-28T19:43:49.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.804 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.804 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.805 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-08-28T19:43:49.805 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.805 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.805 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.805 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-08-28T19:43:49.805 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-08-28T19:43:49.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.806 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-08-28T19:43:49.806 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-08-28T19:43:49.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:49.806 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.806 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-08-28T19:43:49.807 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-08-28T19:43:49.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-08-28T19:43:49.807 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.807 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-08-28T19:43:49.807 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-08-28T19:43:49.807 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.808 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.808 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-08-28T19:43:49.808 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-08-28T19:43:49.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.808 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.808 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.808 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-08-28T19:43:49.809 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.809 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.809 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.809 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-08-28T19:43:49.809 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-28T19:43:49.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.809 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.810 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-08-28T19:43:49.810 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-08-28T19:43:49.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:49.810 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.810 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.810 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-08-28T19:43:49.811 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-08-28T19:43:49.811 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.811 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-08-28T19:43:49.811 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-28T19:43:49.811 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.811 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.812 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.812 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.812 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.812 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.812 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.812 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.813 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.813 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.813 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.813 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.813 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.813 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.813 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.814 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.814 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.814 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.814 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.814 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.815 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.815 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.815 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.815 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.816 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.816 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.816 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.816 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.816 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.817 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.817 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.817 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.817 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.817 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.818 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.818 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.818 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.818 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.818 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.818 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.818 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.819 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.819 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.819 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.819 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.819 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.819 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.820 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.820 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.820 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.820 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.821 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.821 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.821 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.821 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.821 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.821 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.822 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.822 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.822 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.822 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-08-28T19:43:49.822 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-08-28T19:43:49.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:49.823 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.823 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.823 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.823 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.824 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-08-28T19:43:49.824 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.824 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.824 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.824 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-08-28T19:43:49.825 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-08-28T19:43:49.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.825 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.825 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.825 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-08-28T19:43:49.825 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.825 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.825 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.826 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-08-28T19:43:49.826 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-08-28T19:43:49.826 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-08-28T19:43:49.826 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-08-28T19:43:49.826 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-08-28T19:43:49.826 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.826 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-08-28T19:43:49.827 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.827 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.827 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-08-28T19:43:49.827 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-08-28T19:43:49.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.828 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.828 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.828 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.828 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.828 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.828 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.828 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-08-28T19:43:49.828 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.829 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.829 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.829 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:49.829 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:49.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.830 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.830 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.830 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:49.830 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.830 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.830 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.830 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-08-28T19:43:49.831 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.831 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.831 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.831 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.831 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.832 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.832 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.832 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.832 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.832 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.833 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.833 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.833 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.833 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-08-28T19:43:49.833 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-08-28T19:43:49.833 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.834 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.834 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.834 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:49.834 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:49.834 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.834 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.834 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-08-28T19:43:49.835 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-08-28T19:43:49.835 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.835 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.835 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-08-28T19:43:49.835 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-08-28T19:43:49.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:49.836 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.836 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.836 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:49.836 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.836 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.836 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.836 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-08-28T19:43:49.837 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.837 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.837 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.838 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-08-28T19:43:49.838 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.838 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.838 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.838 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-08-28T19:43:49.838 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-08-28T19:43:49.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:49.839 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.839 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-08-28T19:43:49.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.839 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.839 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.839 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-08-28T19:43:49.840 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.840 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.840 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.840 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-08-28T19:43:49.840 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-08-28T19:43:49.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.841 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.841 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.841 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-08-28T19:43:49.841 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.841 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.841 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.841 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-08-28T19:43:49.842 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.842 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.842 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.842 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-08-28T19:43:49.842 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-08-28T19:43:49.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.843 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.843 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.843 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-08-28T19:43:49.843 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.843 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.843 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.843 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-08-28T19:43:49.844 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.844 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.844 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.844 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-08-28T19:43:49.844 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-08-28T19:43:49.844 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.845 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.845 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.845 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.845 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.845 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-08-28T19:43:49.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-08-28T19:43:49.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.846 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-08-28T19:43:49.846 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-08-28T19:43:49.846 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.846 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:49.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.847 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.847 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.847 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-08-28T19:43:49.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.848 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-08-28T19:43:49.848 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-08-28T19:43:49.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:49.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-08-28T19:43:49.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.850 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-08-28T19:43:49.850 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.850 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-08-28T19:43:49.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-08-28T19:43:49.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.851 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.851 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.851 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-08-28T19:43:49.852 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-08-28T19:43:49.852 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-08-28T19:43:49.852 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.852 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-08-28T19:43:49.852 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.852 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.853 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-08-28T19:43:49.853 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-08-28T19:43:49.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.853 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.853 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.854 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.854 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.854 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.854 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-08-28T19:43:49.854 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.854 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.854 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.855 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-08-28T19:43:49.855 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-08-28T19:43:49.855 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.855 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.855 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.856 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.856 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.856 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.856 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-08-28T19:43:49.856 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.856 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.856 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.857 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-08-28T19:43:49.857 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-08-28T19:43:49.857 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.857 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.857 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.857 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.858 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.858 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.858 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.858 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-08-28T19:43:49.858 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.858 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.859 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-08-28T19:43:49.859 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-08-28T19:43:49.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.859 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.859 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-08-28T19:43:49.859 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.860 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.860 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.860 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:49.860 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.860 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.861 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-08-28T19:43:49.861 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-08-28T19:43:49.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.861 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-08-28T19:43:49.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.862 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.862 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.862 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-08-28T19:43:49.862 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.862 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.862 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-08-28T19:43:49.863 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-08-28T19:43:49.863 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.863 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.863 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.863 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-08-28T19:43:49.863 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.863 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.863 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.864 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-08-28T19:43:49.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.864 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-08-28T19:43:49.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-08-28T19:43:49.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.865 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:10.4.0", 2024-08-28T19:43:49.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:49.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.866 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-08-28T19:43:49.866 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-08-28T19:43:49.866 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-08-28T19:43:49.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.867 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-08-28T19:43:49.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.868 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-08-28T19:43:49.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-08-28T19:43:49.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.868 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-08-28T19:43:49.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.869 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-08-28T19:43:49.869 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.869 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.870 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.870 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-08-28T19:43:49.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-08-28T19:43:49.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-08-28T19:43:49.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.871 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.871 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-08-28T19:43:49.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.872 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.872 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-08-28T19:43:49.872 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-08-28T19:43:49.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.872 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.872 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-08-28T19:43:49.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.873 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-08-28T19:43:49.873 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.873 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.874 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-08-28T19:43:49.874 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-08-28T19:43:49.874 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.874 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.874 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-08-28T19:43:49.875 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.875 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.875 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.875 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-08-28T19:43:49.875 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.875 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.876 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-08-28T19:43:49.876 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-08-28T19:43:49.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.876 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.877 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-08-28T19:43:49.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.877 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-08-28T19:43:49.877 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.878 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.878 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nginx": { 2024-08-28T19:43:49.878 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nginx", 2024-08-28T19:43:49.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.878 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.878 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.879 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nginx:1.26.1", 2024-08-28T19:43:49.879 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.879 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.879 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.879 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nginx container image", 2024-08-28T19:43:49.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.880 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.880 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-08-28T19:43:49.880 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-08-28T19:43:49.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.880 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.881 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.881 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-08-28T19:43:49.881 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.881 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.881 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:49.881 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.882 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.882 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.882 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-08-28T19:43:49.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-08-28T19:43:49.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.883 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.883 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.883 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-08-28T19:43:49.883 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.883 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.883 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-08-28T19:43:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_oauth2_proxy": { 2024-08-28T19:43:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_oauth2_proxy", 2024-08-28T19:43:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-08-28T19:43:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "oauth2-proxy container image", 2024-08-28T19:43:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-08-28T19:43:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-08-28T19:43:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-08-28T19:43:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-08-28T19:43:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-08-28T19:43:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-08-28T19:43:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-08-28T19:43:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-08-28T19:43:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-08-28T19:43:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-08-28T19:43:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-08-28T19:43:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-08-28T19:43:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-08-28T19:43:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-08-28T19:43:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-08-28T19:43:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-08-28T19:43:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-08-28T19:43:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-08-28T19:43:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-08-28T19:43:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-08-28T19:43:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-08-28T19:43:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-08-28T19:43:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-08-28T19:43:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-08-28T19:43:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-08-28T19:43:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-08-28T19:43:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-08-28T19:43:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-08-28T19:43:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-08-28T19:43:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-08-28T19:43:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-08-28T19:43:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-08-28T19:43:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-08-28T19:43:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-08-28T19:43:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-08-28T19:43:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-08-28T19:43:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-08-28T19:43:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-08-28T19:43:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-08-28T19:43:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-08-28T19:43:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-08-28T19:43:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-08-28T19:43:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-08-28T19:43:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-08-28T19:43:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-08-28T19:43:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-08-28T19:43:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-08-28T19:43:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-08-28T19:43:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-08-28T19:43:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-08-28T19:43:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-08-28T19:43:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-08-28T19:43:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-08-28T19:43:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-08-28T19:43:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-08-28T19:43:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-08-28T19:43:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-08-28T19:43:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-08-28T19:43:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-08-28T19:43:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-08-28T19:43:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-08-28T19:43:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-08-28T19:43:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-08-28T19:43:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-08-28T19:43:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-08-28T19:43:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-08-28T19:43:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-08-28T19:43:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-08-28T19:43:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-08-28T19:43:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-08-28T19:43:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-08-28T19:43:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-08-28T19:43:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-08-28T19:43:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-08-28T19:43:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-08-28T19:43:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-08-28T19:43:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-08-28T19:43:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-08-28T19:43:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-08-28T19:43:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-08-28T19:43:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-08-28T19:43:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-08-28T19:43:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-08-28T19:43:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-08-28T19:43:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-08-28T19:43:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-08-28T19:43:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-08-28T19:43:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-08-28T19:43:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-08-28T19:43:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-08-28T19:43:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-08-28T19:43:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-08-28T19:43:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-08-28T19:43:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-08-28T19:43:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-08-28T19:43:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-08-28T19:43:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-08-28T19:43:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-08-28T19:43:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-08-28T19:43:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-08-28T19:43:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-08-28T19:43:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-08-28T19:43:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-08-28T19:43:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-08-28T19:43:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-08-28T19:43:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-08-28T19:43:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-08-28T19:43:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-08-28T19:43:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-08-28T19:43:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-08-28T19:43:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-08-28T19:43:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-08-28T19:43:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-08-28T19:43:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-08-28T19:43:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-08-28T19:43:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-08-28T19:43:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-08-28T19:43:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-08-28T19:43:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-08-28T19:43:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-08-28T19:43:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-08-28T19:43:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-08-28T19:43:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-08-28T19:43:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-08-28T19:43:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.970 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.970 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.970 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.970 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.970 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.971 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.971 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.971 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.971 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.971 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.971 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.972 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:49.972 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:49.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.972 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.972 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.972 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.973 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.973 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.973 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.973 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.973 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.973 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.974 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:49.974 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:49.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.974 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.974 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:49.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.975 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.975 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:49.975 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:49.975 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:49.975 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:49.975 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:49.975 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:49.976 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:49.976 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:49.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.976 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.976 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.977 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:49.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:49.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.977 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.977 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.977 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.977 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.978 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.978 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.978 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.978 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.979 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-08-28T19:43:49.979 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-08-28T19:43:49.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.979 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.979 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.979 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-08-28T19:43:49.979 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.979 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.980 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.980 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-08-28T19:43:49.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.981 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:49.981 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:49.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.981 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:49.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:49.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.982 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.982 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.982 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.982 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.982 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.982 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.983 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-08-28T19:43:49.983 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-08-28T19:43:49.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:49.983 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.983 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:49.983 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-08-28T19:43:49.983 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.984 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.984 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.984 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-08-28T19:43:49.984 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.984 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.984 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.984 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.985 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:49.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:49.985 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-08-28T19:43:49.985 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:49.985 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:49.986 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:49.986 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-08-28T19:43:49.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-08-28T19:43:49.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:49.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:49.987 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.987 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.987 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.987 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.988 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-08-28T19:43:49.988 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-08-28T19:43:49.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:49.988 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.988 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.988 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:49.989 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.989 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.989 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.989 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.989 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.989 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.990 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-08-28T19:43:49.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-08-28T19:43:49.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.990 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.990 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.990 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.991 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.991 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.991 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.991 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.991 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.991 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.992 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-08-28T19:43:49.992 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-08-28T19:43:49.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.992 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.992 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.992 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:49.993 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.993 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.993 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.993 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.993 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.993 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.993 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.994 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-08-28T19:43:49.994 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-08-28T19:43:49.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.994 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.994 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.995 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.995 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.995 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.995 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.995 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.996 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.996 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-08-28T19:43:49.996 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-08-28T19:43:49.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.996 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.997 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.997 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.997 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.997 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.997 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:49.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:49.998 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:49.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:49.998 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-08-28T19:43:49.998 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-08-28T19:43:49.998 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:49.998 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:49.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:49.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:49.999 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:49.999 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:49.999 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:49.999 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:49.999 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.000 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.000 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-08-28T19:43:50.000 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-08-28T19:43:50.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.001 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.001 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.002 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.002 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.002 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-08-28T19:43:50.002 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-08-28T19:43:50.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.003 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.003 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.003 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.003 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.003 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.003 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.004 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.004 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.004 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.004 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-08-28T19:43:50.004 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-08-28T19:43:50.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.004 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.005 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.005 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.005 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.005 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.005 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.005 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.006 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.006 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-08-28T19:43:50.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-08-28T19:43:50.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.007 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.007 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.007 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.007 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.007 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.007 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.008 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-08-28T19:43:50.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-08-28T19:43:50.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.008 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.009 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.009 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.009 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.009 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.009 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.009 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.010 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.010 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.010 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-08-28T19:43:50.010 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-08-28T19:43:50.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.011 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.011 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.011 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.011 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.011 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.011 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.012 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.012 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-08-28T19:43:50.012 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-08-28T19:43:50.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.013 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.013 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.013 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.013 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.014 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.014 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.014 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.014 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.014 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-08-28T19:43:50.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-08-28T19:43:50.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.015 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-08-28T19:43:50.015 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.015 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.015 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.016 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.016 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.016 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.016 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.016 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-08-28T19:43:50.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-08-28T19:43:50.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.017 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.017 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.017 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.018 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.018 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.018 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.018 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.018 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-08-28T19:43:50.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-08-28T19:43:50.019 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.019 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.019 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.019 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.019 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.019 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.019 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.020 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.020 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.020 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.020 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.020 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-08-28T19:43:50.021 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-08-28T19:43:50.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.021 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.021 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.021 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-08-28T19:43:50.021 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.021 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.022 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.022 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.022 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.022 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.022 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-08-28T19:43:50.023 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-08-28T19:43:50.023 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.023 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.023 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.023 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.023 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.023 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.024 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.024 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.024 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.024 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.024 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.024 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-08-28T19:43:50.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-08-28T19:43:50.025 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.025 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.025 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.025 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.025 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.025 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.026 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.026 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.026 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.026 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.026 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.026 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-08-28T19:43:50.026 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-08-28T19:43:50.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.027 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.027 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.027 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.027 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.027 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.028 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.028 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.028 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.028 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.028 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-08-28T19:43:50.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-08-28T19:43:50.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.029 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.029 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.029 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.029 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.029 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.029 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.029 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.030 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.030 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.030 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.030 INFO:teuthology.orchestra.run.smithi016.stdout: "MANAGED_BY_CLUSTERS": { 2024-08-28T19:43:50.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-08-28T19:43:50.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.031 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.031 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "[]", 2024-08-28T19:43:50.031 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.031 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.031 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.032 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.032 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.032 INFO:teuthology.orchestra.run.smithi016.stdout: "MULTICLUSTER_CONFIG": { 2024-08-28T19:43:50.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MULTICLUSTER_CONFIG", 2024-08-28T19:43:50.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.033 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.033 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "{}", 2024-08-28T19:43:50.033 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.033 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.033 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.033 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.034 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.034 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.034 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.034 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-08-28T19:43:50.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-08-28T19:43:50.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.035 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.035 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.035 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.035 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.035 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.035 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.035 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.036 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.036 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.036 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.036 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-08-28T19:43:50.036 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-08-28T19:43:50.036 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.036 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.037 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.037 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.037 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.037 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.037 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.037 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.038 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.038 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-08-28T19:43:50.038 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-08-28T19:43:50.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.038 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.039 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.039 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.039 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.039 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.039 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.039 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.039 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.040 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.040 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-08-28T19:43:50.040 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-08-28T19:43:50.040 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.040 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.040 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.040 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.041 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.041 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.041 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.041 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.041 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.041 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.041 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.042 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-08-28T19:43:50.042 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-08-28T19:43:50.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.042 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.042 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.042 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.042 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.043 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.043 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.043 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.043 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.043 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.044 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-08-28T19:43:50.044 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-08-28T19:43:50.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.044 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.044 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.044 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.045 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.045 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.045 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.045 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.045 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.045 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-08-28T19:43:50.046 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-08-28T19:43:50.046 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.046 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.046 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.046 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.046 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.047 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.047 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.047 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.047 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.047 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.047 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.048 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-08-28T19:43:50.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-08-28T19:43:50.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.048 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.048 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.048 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.048 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.049 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.049 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.049 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.049 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.049 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.050 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-08-28T19:43:50.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-08-28T19:43:50.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.050 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.050 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.050 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.051 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.051 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.051 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.051 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.051 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.052 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-08-28T19:43:50.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-08-28T19:43:50.052 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.052 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.052 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.052 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.053 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.053 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.053 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.053 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.054 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-08-28T19:43:50.054 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-08-28T19:43:50.054 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.054 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.054 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.054 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-08-28T19:43:50.054 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.055 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.055 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.055 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.055 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.055 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-08-28T19:43:50.056 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-08-28T19:43:50.056 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.056 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.056 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.056 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:50.056 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.056 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.057 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.057 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.057 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.057 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.057 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.058 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-08-28T19:43:50.058 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-08-28T19:43:50.058 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.058 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.058 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-08-28T19:43:50.058 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.059 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.059 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.059 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.059 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.059 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.059 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.060 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-08-28T19:43:50.060 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-08-28T19:43:50.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.060 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.060 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.060 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-08-28T19:43:50.061 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.061 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.061 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.061 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.061 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.061 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.062 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-08-28T19:43:50.062 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-08-28T19:43:50.062 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.062 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.062 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.062 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.062 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.063 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.063 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.063 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.063 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.063 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.064 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-08-28T19:43:50.064 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-08-28T19:43:50.064 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.064 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.064 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.064 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-08-28T19:43:50.064 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.065 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.065 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.065 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.065 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.065 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.065 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.066 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-08-28T19:43:50.066 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-08-28T19:43:50.066 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.066 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.067 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.067 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.067 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.067 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.067 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.067 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.067 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.068 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-08-28T19:43:50.068 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-08-28T19:43:50.068 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.068 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.068 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.069 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.069 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.069 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.069 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.069 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.069 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.070 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.070 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-08-28T19:43:50.070 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-08-28T19:43:50.070 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.071 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.071 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.071 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.071 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.071 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.071 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.071 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.071 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.072 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.072 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.072 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-08-28T19:43:50.072 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-08-28T19:43:50.072 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.072 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.073 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.073 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.073 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.073 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.073 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.073 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.073 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.074 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.074 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-08-28T19:43:50.074 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-08-28T19:43:50.074 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.074 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.075 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.075 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:50.075 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.075 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.075 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.075 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.075 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.076 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.076 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.076 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-08-28T19:43:50.076 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-08-28T19:43:50.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.076 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.077 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.077 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:50.077 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.077 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.077 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.077 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.077 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.078 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.078 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-08-28T19:43:50.078 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-08-28T19:43:50.078 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.078 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.079 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.079 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.079 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.079 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.079 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.079 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.080 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.080 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.080 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-08-28T19:43:50.080 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-08-28T19:43:50.080 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.080 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.081 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.081 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.081 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.081 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.081 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.081 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.081 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.082 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.082 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.082 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-08-28T19:43:50.082 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-08-28T19:43:50.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.082 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.083 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.083 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.083 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.083 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-08-28T19:43:50.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.084 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.084 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.084 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-08-28T19:43:50.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-08-28T19:43:50.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.084 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.084 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.085 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-08-28T19:43:50.085 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.085 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.085 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.085 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.085 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.086 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.086 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-08-28T19:43:50.086 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-08-28T19:43:50.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.086 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.086 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.087 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.087 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.087 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.087 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.087 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.087 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.088 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.088 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.088 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.088 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.088 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.088 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.088 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.089 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.089 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.089 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.089 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.089 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.089 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.092 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.092 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.092 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.092 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.092 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.092 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.093 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.093 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.093 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.093 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.093 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.094 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.094 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.094 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.094 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.094 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.094 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.094 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.094 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.095 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.095 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.095 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.095 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.095 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.096 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.096 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.096 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.096 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.096 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.097 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.097 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.097 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.097 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-08-28T19:43:50.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-08-28T19:43:50.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.098 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.098 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.098 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.098 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.098 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.098 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-08-28T19:43:50.098 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.099 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.099 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.099 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-08-28T19:43:50.099 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-08-28T19:43:50.099 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.099 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.099 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.100 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.100 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.100 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.100 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.100 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.100 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.101 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.101 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.101 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-08-28T19:43:50.101 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-08-28T19:43:50.101 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.101 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-08-28T19:43:50.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.102 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.102 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.102 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.102 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.102 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.103 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-08-28T19:43:50.103 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-08-28T19:43:50.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.103 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.104 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-08-28T19:43:50.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.104 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.104 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.104 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.104 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.105 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.105 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.105 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.106 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.106 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.106 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.106 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.106 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.107 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.107 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-08-28T19:43:50.107 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-08-28T19:43:50.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.107 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.107 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.108 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.108 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.108 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.108 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.108 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.108 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.109 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-08-28T19:43:50.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-08-28T19:43:50.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.109 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.109 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-08-28T19:43:50.110 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.110 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.110 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.110 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.110 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.111 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.111 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.111 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-08-28T19:43:50.111 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-08-28T19:43:50.111 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.111 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.112 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.112 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-08-28T19:43:50.112 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.112 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.112 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.112 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.113 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-08-28T19:43:50.113 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.113 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.113 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.114 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-08-28T19:43:50.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-08-28T19:43:50.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.115 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-08-28T19:43:50.115 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-08-28T19:43:50.115 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-08-28T19:43:50.115 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.115 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.115 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.116 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.116 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-08-28T19:43:50.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-08-28T19:43:50.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.116 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.117 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.117 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.117 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.118 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.118 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.118 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.118 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-08-28T19:43:50.118 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.118 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.119 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.119 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-08-28T19:43:50.119 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-08-28T19:43:50.119 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.119 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.119 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.120 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.120 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.120 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.120 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-08-28T19:43:50.120 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.120 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.120 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.121 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.121 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.121 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.122 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.122 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.122 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.122 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.122 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.122 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.122 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.122 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.123 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.123 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.123 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.123 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.123 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.123 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.124 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.124 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.124 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.124 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.124 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.125 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.125 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.125 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.125 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.125 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.126 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.126 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.126 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.126 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.127 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.127 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.127 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.127 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.127 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.128 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.128 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.128 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.128 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.129 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.129 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.129 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.129 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.129 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.130 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.130 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.130 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.130 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.130 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.130 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.131 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-08-28T19:43:50.131 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-08-28T19:43:50.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.131 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-08-28T19:43:50.131 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.132 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.132 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.132 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-08-28T19:43:50.132 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.132 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.132 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.133 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-08-28T19:43:50.133 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-08-28T19:43:50.133 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.133 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.133 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.133 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-08-28T19:43:50.133 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.134 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-08-28T19:43:50.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.134 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.135 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-08-28T19:43:50.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-08-28T19:43:50.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.135 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.135 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.135 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-08-28T19:43:50.135 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.136 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.136 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.136 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-08-28T19:43:50.136 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.136 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.137 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-08-28T19:43:50.137 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-08-28T19:43:50.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.137 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-08-28T19:43:50.138 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.138 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.138 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.138 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-08-28T19:43:50.138 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.138 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.139 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-08-28T19:43:50.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-08-28T19:43:50.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.139 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.139 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.139 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.139 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.140 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.140 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-08-28T19:43:50.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.140 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.140 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.142 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.142 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.142 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-08-28T19:43:50.142 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.142 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.142 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.143 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.143 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.143 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.143 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.143 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.143 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.143 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.144 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.144 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.144 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.144 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.145 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-08-28T19:43:50.145 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-08-28T19:43:50.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.145 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.145 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-08-28T19:43:50.145 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.146 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.146 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.146 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-08-28T19:43:50.146 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.147 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.147 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.147 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.147 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-08-28T19:43:50.147 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.148 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.148 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.148 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.148 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.148 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.149 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.149 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.149 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.149 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.149 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.149 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.150 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.150 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.150 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.150 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.150 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.150 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.150 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.151 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.151 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.151 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.151 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.151 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.152 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.152 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.152 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.152 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.153 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.153 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.153 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.153 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.153 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.153 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.154 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.154 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.154 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.154 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.154 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.155 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.155 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.155 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.155 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.155 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.156 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.156 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.156 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.156 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.156 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.156 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.157 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.157 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.157 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.157 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.158 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.158 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.158 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.158 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.158 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.158 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.158 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-08-28T19:43:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-08-28T19:43:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-08-28T19:43:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.161 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.161 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-08-28T19:43:50.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-08-28T19:43:50.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.161 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.162 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-08-28T19:43:50.162 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.162 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.162 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.163 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.163 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:50.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:50.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.163 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.164 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:50.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.164 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.165 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.166 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.166 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.166 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.166 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.167 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.167 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.167 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-08-28T19:43:50.167 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-08-28T19:43:50.168 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-08-28T19:43:50.168 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.168 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-08-28T19:43:50.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-08-28T19:43:50.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.168 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.168 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-08-28T19:43:50.169 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.169 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.169 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.169 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-08-28T19:43:50.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.169 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.170 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-08-28T19:43:50.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-08-28T19:43:50.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.170 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-08-28T19:43:50.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.171 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.171 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-08-28T19:43:50.171 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.172 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-08-28T19:43:50.172 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-08-28T19:43:50.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.172 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.172 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.173 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.173 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.173 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.173 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.173 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-08-28T19:43:50.173 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.174 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.174 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:50.174 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:50.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.174 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.175 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-08-28T19:43:50.175 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-08-28T19:43:50.175 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.175 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.175 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-08-28T19:43:50.175 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.175 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.175 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.176 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.176 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.177 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.177 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.177 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.177 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.177 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.177 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.177 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.178 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.178 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.178 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.178 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.178 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.178 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.178 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.179 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.179 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.179 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.180 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.180 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.180 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.180 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.181 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.181 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.182 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.182 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.182 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.182 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.182 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.182 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.182 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.183 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.183 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.183 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.183 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.183 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.183 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.183 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.184 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.184 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.184 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.184 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.184 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.185 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.185 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.185 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.186 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.186 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-08-28T19:43:50.186 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-08-28T19:43:50.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.186 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.187 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.187 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.187 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.187 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.187 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-08-28T19:43:50.187 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.188 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.188 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-08-28T19:43:50.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-08-28T19:43:50.188 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.188 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.188 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.189 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-08-28T19:43:50.189 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.189 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.189 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.189 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-08-28T19:43:50.189 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.190 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.190 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.190 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.190 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.191 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.191 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.191 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.191 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.191 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.191 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.192 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.192 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-08-28T19:43:50.192 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-08-28T19:43:50.192 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.192 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.192 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-08-28T19:43:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-08-28T19:43:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.193 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.194 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.194 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-08-28T19:43:50.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-08-28T19:43:50.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.194 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.195 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:50.195 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:50.195 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-08-28T19:43:50.195 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-08-28T19:43:50.195 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.196 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.196 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-08-28T19:43:50.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-08-28T19:43:50.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.196 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.197 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.197 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.197 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.197 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.197 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-08-28T19:43:50.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-08-28T19:43:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-08-28T19:43:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-08-28T19:43:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-08-28T19:43:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-08-28T19:43:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.211 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.211 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.212 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.212 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.212 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.212 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.212 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.212 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.212 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.213 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.213 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-08-28T19:43:50.214 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.214 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.214 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.214 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.214 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.215 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.215 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.215 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.215 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.215 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.215 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.215 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.216 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.216 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.216 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.216 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.216 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.217 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.217 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.217 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.217 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.217 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.218 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.218 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.218 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.218 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.218 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.219 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.219 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.219 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.219 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.220 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.220 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.220 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.220 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.220 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.221 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.221 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.221 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.221 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.221 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.222 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.222 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.222 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.222 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.222 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.222 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.223 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.223 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.223 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.223 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.223 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.223 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.224 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.224 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.224 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.224 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.225 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.225 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.225 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.225 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.225 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.225 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.226 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.226 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.226 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.226 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.226 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.226 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-08-28T19:43:50.227 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.227 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.227 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.227 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-08-28T19:43:50.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-08-28T19:43:50.227 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.228 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.228 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-08-28T19:43:50.228 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.228 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.228 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.228 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-08-28T19:43:50.228 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.229 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.229 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.229 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-08-28T19:43:50.229 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-08-28T19:43:50.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.229 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:50.230 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-08-28T19:43:50.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.230 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.230 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-08-28T19:43:50.230 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.230 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.231 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.231 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.231 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.231 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.231 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.231 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.232 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.232 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.232 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.232 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.232 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.232 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.232 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.233 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.233 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.233 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.233 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.233 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.233 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.234 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.234 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.234 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.234 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.234 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.235 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.235 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.235 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.235 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.235 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.236 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.236 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.236 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.236 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.236 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.236 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.237 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.237 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.237 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.237 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.237 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.238 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.238 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.238 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.238 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.238 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.238 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.239 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.239 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.239 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.239 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.239 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.240 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.240 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.240 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.240 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.240 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.240 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.241 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.241 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.241 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.241 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.241 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.242 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.242 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.242 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.242 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.242 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.242 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-08-28T19:43:50.243 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.243 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.243 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.243 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-08-28T19:43:50.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-08-28T19:43:50.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.244 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.244 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-08-28T19:43:50.244 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.244 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.244 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-08-28T19:43:50.245 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-08-28T19:43:50.245 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.245 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.245 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.245 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.246 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.246 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.246 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.246 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.246 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.246 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.246 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.247 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.247 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.247 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.247 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.247 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.247 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.247 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.248 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.248 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.248 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.248 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.249 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.249 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.249 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.249 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.249 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.249 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.250 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.250 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.250 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.250 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.251 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.251 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.251 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.251 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.251 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.251 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.251 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.252 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.252 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.252 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.252 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.252 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.252 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.253 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.253 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.253 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.253 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.254 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.254 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.254 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.254 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.254 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.254 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.255 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.255 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-08-28T19:43:50.255 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-08-28T19:43:50.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.255 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.256 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.256 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.256 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.256 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.256 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-08-28T19:43:50.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-08-28T19:43:50.257 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.257 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.257 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-08-28T19:43:50.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-08-28T19:43:50.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.258 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-08-28T19:43:50.258 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.258 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.258 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.258 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-08-28T19:43:50.258 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.259 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.259 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-08-28T19:43:50.259 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-08-28T19:43:50.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-08-28T19:43:50.260 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.260 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.260 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-08-28T19:43:50.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.261 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.261 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-08-28T19:43:50.261 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-08-28T19:43:50.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.261 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.262 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.262 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.262 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.262 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-08-28T19:43:50.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.262 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.263 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.263 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.263 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.264 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.264 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.264 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.264 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.264 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.264 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.264 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.265 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-08-28T19:43:50.265 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-08-28T19:43:50.265 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.265 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.265 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.265 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-08-28T19:43:50.265 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.266 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.266 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.266 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-08-28T19:43:50.266 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-08-28T19:43:50.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.266 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.267 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.267 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-08-28T19:43:50.267 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.267 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.267 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.267 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.268 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.268 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.268 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.268 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.269 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.269 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.269 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.269 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.269 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.269 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.269 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.270 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.270 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.270 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.271 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.271 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.271 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.271 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.272 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.272 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.272 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.272 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.272 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.278 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.278 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.278 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.279 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.279 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.279 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.280 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-08-28T19:43:50.280 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.280 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.280 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.280 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.280 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.281 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.281 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.281 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.281 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.281 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.281 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.282 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.282 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.282 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.282 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.282 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.282 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.282 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.282 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.283 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.283 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.283 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.283 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.284 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.284 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.284 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.284 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.284 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.284 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.284 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.285 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.285 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.285 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.286 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.286 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.286 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.286 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.286 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.286 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.286 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.286 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.286 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.287 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.287 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.287 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.287 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.287 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.287 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.288 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.288 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.288 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.288 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.288 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.289 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.289 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.289 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.289 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.289 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.290 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.290 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.290 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.290 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.290 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.291 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.291 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.291 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.291 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.292 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-08-28T19:43:50.292 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.292 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.292 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.292 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.293 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.293 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.293 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.293 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.293 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.294 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.294 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.294 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.294 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.294 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.294 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.294 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.295 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.295 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.295 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.295 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.295 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.296 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.296 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.296 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.296 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.296 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.296 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.296 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.297 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.297 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.297 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.297 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.297 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.298 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.298 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.298 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.298 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.298 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.298 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.298 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.299 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.299 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.299 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.299 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.299 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.299 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.299 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.300 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.300 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.300 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.300 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.301 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.301 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.301 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.301 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.302 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.302 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.302 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.302 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.302 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.304 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.304 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.304 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.304 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.304 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.305 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-08-28T19:43:50.305 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.305 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.305 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.305 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-08-28T19:43:50.305 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-08-28T19:43:50.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.306 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.306 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.306 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.306 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.306 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-08-28T19:43:50.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.307 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.307 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.307 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.308 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.308 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.308 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.308 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.309 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.309 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.309 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.309 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.309 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.309 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.310 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.310 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.310 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.310 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.310 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.311 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.311 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.311 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.311 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.312 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.312 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.312 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.312 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.312 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.312 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.313 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.313 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.313 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.313 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.313 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.314 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.314 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.314 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.314 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.314 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.314 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.314 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.315 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.315 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.315 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.315 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.315 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.315 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.316 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.316 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.316 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.316 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.317 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.317 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.317 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.317 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.317 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.318 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-08-28T19:43:50.318 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-08-28T19:43:50.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.318 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.318 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.318 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.318 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.319 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.319 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-08-28T19:43:50.319 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-08-28T19:43:50.319 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-08-28T19:43:50.319 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.319 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-08-28T19:43:50.319 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.320 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.320 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.320 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.320 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.320 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.321 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.321 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.321 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.321 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.321 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.321 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.321 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.321 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.322 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.322 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.322 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.322 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-08-28T19:43:50.322 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.323 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.323 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.323 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.323 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.324 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.324 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.324 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.324 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.324 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.325 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.325 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.325 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.325 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.325 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.325 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.325 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.326 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.326 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.326 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.326 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.326 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.327 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.327 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.327 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.327 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.327 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.327 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.328 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.328 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.328 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.328 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.328 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.329 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.329 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.329 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.329 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.329 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.329 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.329 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.332 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.332 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.332 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.332 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.332 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.333 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.333 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.333 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.333 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.333 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.333 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.334 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.334 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.334 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.334 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.334 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.334 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.335 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.335 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.335 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.335 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.336 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.336 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.336 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.336 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.336 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.336 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.337 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-08-28T19:43:50.337 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.338 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.338 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.338 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.338 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.338 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.338 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.338 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.339 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.339 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.339 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.339 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.339 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.340 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.340 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.340 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.340 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.340 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.340 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.341 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.341 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.341 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.341 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.341 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.342 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.342 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.342 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.342 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.342 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.342 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.342 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.343 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.343 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.343 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.343 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.343 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.343 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.344 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.344 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.344 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.344 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.344 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.344 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.344 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.345 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.345 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.345 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.345 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.345 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.345 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.345 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.346 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.346 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.346 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.346 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.346 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.346 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.346 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.347 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.347 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.347 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.347 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.347 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.348 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.348 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.348 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.348 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.348 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.349 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.349 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.349 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.349 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.349 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.349 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.349 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-08-28T19:43:50.350 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.350 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.350 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.351 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.351 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.351 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.351 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.351 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.351 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.352 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.352 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.352 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.352 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.352 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.352 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.352 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.353 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.353 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.353 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.353 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.354 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.354 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.354 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.354 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.354 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.354 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.354 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.354 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.355 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.355 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.355 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.355 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.355 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.356 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.356 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.356 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.356 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.356 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.356 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.356 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.356 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.357 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.357 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.357 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.357 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.357 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.357 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.357 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.358 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.358 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.358 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.358 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.358 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.358 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.359 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.359 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.359 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.359 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.359 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.359 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.362 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.362 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.362 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.362 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.362 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-08-28T19:43:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-08-28T19:43:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-08-28T19:43:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-08-28T19:43:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-08-28T19:43:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-08-28T19:43:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-08-28T19:43:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-08-28T19:43:50.368 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-08-28T19:43:50.368 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.368 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.368 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.368 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-08-28T19:43:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-08-28T19:43:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-08-28T19:43:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.378 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.378 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.378 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.378 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.378 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.378 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.378 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-08-28T19:43:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-08-28T19:43:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-08-28T19:43:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-08-28T19:43:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:50.384 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.384 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.384 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.384 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:50.384 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.384 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.385 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.385 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-08-28T19:43:50.385 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.385 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.385 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.385 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.385 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.388 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-08-28T19:43:50.388 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.388 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.388 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.388 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-08-28T19:43:50.388 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-08-28T19:43:50.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.389 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.389 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.389 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.389 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.390 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.390 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.390 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.390 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.390 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.390 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-08-28T19:43:50.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-08-28T19:43:50.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.391 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.391 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.391 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.392 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.392 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.392 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-08-28T19:43:50.392 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-08-28T19:43:50.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.392 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.392 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.393 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.393 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.393 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.393 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.393 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.393 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.394 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.394 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.394 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.394 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.394 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.394 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.394 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.394 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.395 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.395 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.395 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.395 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.396 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.396 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.396 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.396 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.396 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.396 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.396 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.397 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.397 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.397 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.397 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.397 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.397 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.398 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.398 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.398 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.398 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.398 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.398 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.398 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.399 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.399 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.399 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.399 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.399 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.399 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.399 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.400 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.400 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.400 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.400 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.401 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.401 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.401 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.401 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.401 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.401 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.402 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-08-28T19:43:50.402 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-08-28T19:43:50.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.402 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.402 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.402 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.403 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.403 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.403 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.403 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.403 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.404 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-08-28T19:43:50.404 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-08-28T19:43:50.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.404 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.404 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-08-28T19:43:50.404 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.405 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.405 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.405 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.405 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.406 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-08-28T19:43:50.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-08-28T19:43:50.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:50.406 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.406 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-08-28T19:43:50.406 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.406 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.407 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.407 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.407 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.407 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.407 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.407 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-08-28T19:43:50.407 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-08-28T19:43:50.407 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.408 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.408 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-08-28T19:43:50.408 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.408 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.408 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.408 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-08-28T19:43:50.409 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.409 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.409 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-08-28T19:43:50.409 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-08-28T19:43:50.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.409 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.410 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.410 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-08-28T19:43:50.410 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.410 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.410 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.410 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-08-28T19:43:50.411 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.411 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.411 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.411 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.412 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.412 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.412 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.412 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.412 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.412 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.413 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.413 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.413 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.413 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-08-28T19:43:50.413 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-08-28T19:43:50.413 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.414 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.414 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.414 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-08-28T19:43:50.414 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.414 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.414 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.414 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.415 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.415 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.415 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.415 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-08-28T19:43:50.415 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-08-28T19:43:50.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.416 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.416 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-08-28T19:43:50.416 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.416 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.416 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.417 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-08-28T19:43:50.417 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-08-28T19:43:50.417 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.417 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.417 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.417 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.418 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-08-28T19:43:50.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-08-28T19:43:50.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.418 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.418 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-08-28T19:43:50.419 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-08-28T19:43:50.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-08-28T19:43:50.419 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.419 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.419 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.419 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.420 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.420 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.420 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-08-28T19:43:50.420 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.421 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.421 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.421 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.421 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.421 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.422 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.422 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.422 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.422 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.422 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.422 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.422 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.423 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.423 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.423 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.423 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.423 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.423 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.424 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.424 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.424 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.425 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.425 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.425 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.425 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.425 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.425 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.426 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.426 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.426 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.426 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.427 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.427 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.427 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.427 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.427 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.428 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.428 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.428 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.428 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.428 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.428 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.428 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.431 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.431 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.431 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.431 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.431 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.432 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.432 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-08-28T19:43:50.432 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-08-28T19:43:50.432 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.432 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.433 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:50.433 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.433 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.433 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.433 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.433 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.434 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.434 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.434 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-08-28T19:43:50.434 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-08-28T19:43:50.434 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.434 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.435 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.435 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.435 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.435 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.435 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.435 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.436 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.436 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.436 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.436 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.437 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.437 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.437 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.437 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.437 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.437 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.438 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.438 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.438 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.438 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-08-28T19:43:50.438 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-08-28T19:43:50.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.439 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.439 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.439 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.439 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.439 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.439 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.439 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.440 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.440 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.440 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.440 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.440 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.441 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-08-28T19:43:50.441 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.441 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.441 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.441 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-08-28T19:43:50.441 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-08-28T19:43:50.441 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.442 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.442 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.442 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.442 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.442 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.442 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.442 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.443 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.443 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.443 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-08-28T19:43:50.443 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-08-28T19:43:50.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.443 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.444 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.444 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.444 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.444 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.444 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.444 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.444 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.444 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.445 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.445 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.445 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.445 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.445 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.445 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.446 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.446 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.446 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.446 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.446 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.446 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.446 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.446 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.447 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.447 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.447 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.447 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.447 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.447 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.448 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.448 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.448 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.448 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.448 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.448 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.448 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.449 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.449 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.449 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.449 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.449 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.450 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.450 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.450 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.450 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.450 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.450 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.450 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.451 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.451 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.451 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.451 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.451 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.451 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.451 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.452 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.452 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.452 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.452 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.452 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.453 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.453 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.453 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.453 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.453 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.453 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.453 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.454 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.454 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.454 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.454 INFO:teuthology.orchestra.run.smithi016.stdout: "max_requests": { 2024-08-28T19:43:50.455 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_requests", 2024-08-28T19:43:50.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.455 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.455 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-08-28T19:43:50.455 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.455 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.455 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-08-28T19:43:50.456 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.456 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.456 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-08-28T19:43:50.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-08-28T19:43:50.457 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.457 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.457 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.457 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.457 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.457 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-08-28T19:43:50.458 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-08-28T19:43:50.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.459 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.459 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.459 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.459 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.459 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.460 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.460 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.460 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.460 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.460 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.460 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.461 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.461 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.461 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.461 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.461 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.462 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.462 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.462 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.462 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.462 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.462 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-08-28T19:43:50.463 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.463 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.463 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.463 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.464 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.464 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.464 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.464 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.464 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.464 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.464 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.465 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.465 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.465 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.465 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.465 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.465 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.465 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.466 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.466 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.466 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.466 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.466 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.466 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.467 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.467 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.467 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.467 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.467 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.467 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.467 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.467 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.468 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.468 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.468 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.468 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.468 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.469 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.469 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.469 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.469 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.469 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.469 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.469 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.469 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.470 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.470 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.470 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.470 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.470 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.470 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.471 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.471 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.471 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.471 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.472 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.472 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.472 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.472 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.472 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.472 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.473 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.473 INFO:teuthology.orchestra.run.smithi016.stdout: "secondary_zone_period_retry_limit": { 2024-08-28T19:43:50.473 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secondary_zone_period_retry_limit", 2024-08-28T19:43:50.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.473 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.474 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:50.474 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.474 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.474 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.474 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-08-28T19:43:50.474 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.474 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.474 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.475 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.475 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.475 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.475 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.475 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.476 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.476 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.476 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.476 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.476 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.476 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.476 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.477 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.477 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.477 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-08-28T19:43:50.477 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.477 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.477 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.478 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.478 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.478 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.478 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.478 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.478 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.478 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.479 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.479 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.479 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.479 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.479 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.480 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.480 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.480 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.480 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.480 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.481 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.481 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.481 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.481 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.481 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.481 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.482 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.482 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.482 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.482 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.482 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.482 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.483 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.483 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.483 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.483 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.483 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.484 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.484 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.484 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.484 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.484 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.484 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.484 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.484 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.485 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.485 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.485 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.485 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.485 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.486 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.486 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.486 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.486 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.486 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.486 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.487 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.487 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.487 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.487 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.488 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.488 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.488 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.488 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.489 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.489 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.489 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.489 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-08-28T19:43:50.489 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-08-28T19:43:50.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.490 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.490 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-08-28T19:43:50.490 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.490 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.490 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.490 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-08-28T19:43:50.491 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.491 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-08-28T19:43:50.492 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.492 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.492 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.492 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.493 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.493 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.493 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.493 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.493 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.493 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.494 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.494 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.494 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.494 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.494 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.494 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.494 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.495 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.495 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.495 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.495 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.495 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.496 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.496 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.496 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.496 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.496 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.496 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.496 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.497 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.497 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.497 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.497 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.497 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.497 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.498 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.498 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.498 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.498 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.498 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.499 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.499 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.499 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.499 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.499 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.499 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.499 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.499 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.500 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.500 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.500 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.500 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.500 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.501 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.501 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.501 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.501 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.501 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.502 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.502 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.502 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.502 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-08-28T19:43:50.502 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-08-28T19:43:50.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.503 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.503 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.503 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.503 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.503 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.504 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.504 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.504 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.504 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-08-28T19:43:50.504 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-08-28T19:43:50.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.504 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.505 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.505 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-08-28T19:43:50.505 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.505 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.505 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.505 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.505 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.506 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.506 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.506 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-08-28T19:43:50.506 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-08-28T19:43:50.506 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.506 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.507 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.507 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.507 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.507 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.507 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.507 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.507 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.508 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-08-28T19:43:50.508 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-08-28T19:43:50.508 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.508 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.508 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.509 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.509 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.509 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.509 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.509 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.509 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.509 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.510 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.510 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-08-28T19:43:50.510 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-08-28T19:43:50.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:50.510 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.510 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.511 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.511 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.511 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.511 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.511 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.511 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.511 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.512 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-08-28T19:43:50.512 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-08-28T19:43:50.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.512 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.512 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.513 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.513 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.513 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.513 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.513 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.513 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.513 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.514 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-08-28T19:43:50.514 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-08-28T19:43:50.514 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.514 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.514 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.515 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.515 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.515 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.515 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.515 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.515 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.515 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.516 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-08-28T19:43:50.516 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-08-28T19:43:50.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.516 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.516 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.516 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.517 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.517 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.517 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.517 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.517 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.518 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-08-28T19:43:50.518 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-08-28T19:43:50.518 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.518 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.518 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.518 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.518 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:50.518 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-08-28T19:43:50.519 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.519 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.519 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.519 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.519 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.519 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.520 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.520 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.520 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.520 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.520 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.520 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.521 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.521 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.521 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.521 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.521 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.521 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-08-28T19:43:50.521 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-08-28T19:43:50.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.522 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.522 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.522 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.522 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.522 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.523 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.523 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.523 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.523 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-08-28T19:43:50.523 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-08-28T19:43:50.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.524 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.524 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.524 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.524 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.524 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.524 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.525 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.525 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.525 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.525 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.525 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-08-28T19:43:50.525 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-08-28T19:43:50.526 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.526 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.526 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.526 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.526 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.526 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.526 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.527 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.527 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.527 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.527 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.527 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.527 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.528 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smb", 2024-08-28T19:43:50.528 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.528 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.528 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.528 INFO:teuthology.orchestra.run.smithi016.stdout: "internal_store_backend": { 2024-08-28T19:43:50.528 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "internal_store_backend", 2024-08-28T19:43:50.529 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.529 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.529 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.529 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.529 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.529 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.529 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.529 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-08-28T19:43:50.530 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.530 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.530 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.530 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.530 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.531 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.531 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.531 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.531 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.531 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.531 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.531 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.532 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.532 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.532 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.532 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.532 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.532 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.532 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.533 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.533 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.533 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.533 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.533 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.533 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.534 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.534 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.534 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.534 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.534 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.534 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.535 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.535 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.535 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.535 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.535 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.536 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.536 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.536 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.536 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.536 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.536 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.537 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.537 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.537 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.537 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.537 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.537 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.537 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.537 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.538 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.538 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.538 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.538 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.538 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.539 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.539 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.539 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.539 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.539 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.539 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.539 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.539 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.540 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.540 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.540 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.540 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.541 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.541 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.541 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.541 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.541 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.541 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.541 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.542 INFO:teuthology.orchestra.run.smithi016.stdout: "update_orchestration": { 2024-08-28T19:43:50.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "update_orchestration", 2024-08-28T19:43:50.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.542 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.542 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.542 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.543 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.543 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.543 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-08-28T19:43:50.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.543 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.543 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-08-28T19:43:50.544 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.544 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.545 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.545 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-08-28T19:43:50.545 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-08-28T19:43:50.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.545 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.545 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.545 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.546 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.546 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.546 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.546 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-08-28T19:43:50.546 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.547 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-08-28T19:43:50.547 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-08-28T19:43:50.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.547 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.547 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.547 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.548 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.548 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.548 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.548 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-08-28T19:43:50.548 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.548 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.548 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.549 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.549 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.549 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.549 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.549 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.549 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.550 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.550 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.550 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.550 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.550 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.550 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.551 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.551 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.551 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.551 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.551 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.551 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.551 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.551 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.552 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.552 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.552 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.552 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.552 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.552 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.553 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.553 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.553 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.553 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.553 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.553 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.554 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.554 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.554 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.555 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.555 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.555 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.555 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.555 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.555 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.555 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.556 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.556 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.556 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.556 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.556 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.556 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.557 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.557 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.557 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.557 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.557 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.558 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.558 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.558 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.558 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.558 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.558 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.559 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.559 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.559 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.559 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.559 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.559 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.560 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.560 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.560 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.560 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.560 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.560 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.560 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.561 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-08-28T19:43:50.561 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.561 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.561 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.562 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.562 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.562 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.562 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.562 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.562 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.562 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.563 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.563 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.563 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.563 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.563 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.563 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.564 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.564 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.564 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.564 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.564 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.564 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.565 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.565 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.565 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.565 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.566 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.566 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.566 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.566 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.566 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.566 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.567 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.567 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.567 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.567 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.567 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.568 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.568 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.568 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.568 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.568 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.568 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.568 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.569 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.569 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.569 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.569 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.569 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.570 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.570 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.570 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.570 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.570 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.570 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.571 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.571 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.571 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.571 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.571 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.571 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.571 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.572 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.572 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.572 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.572 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.573 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.573 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.573 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.573 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.573 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.573 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.573 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.574 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-08-28T19:43:50.574 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.574 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.575 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.575 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.575 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.575 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.575 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.576 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.576 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.576 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.576 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.576 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.576 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.577 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.577 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.577 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.577 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.577 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.577 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.578 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.578 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.578 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.578 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.578 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.579 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.579 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.579 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.579 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.579 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.579 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.579 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.580 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.580 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.580 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.580 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.580 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.581 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.581 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.581 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.581 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.581 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.581 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.581 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.582 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.582 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.582 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.582 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.582 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.582 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.582 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.582 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.583 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.583 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.583 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.583 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.584 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.584 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.584 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.584 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.584 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.584 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.584 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.585 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.585 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.585 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.585 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.585 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.585 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.586 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.586 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.586 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.586 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.586 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.586 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.586 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.587 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.587 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.587 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.587 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-08-28T19:43:50.587 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.587 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.588 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.588 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-08-28T19:43:50.588 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-08-28T19:43:50.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.588 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.588 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-08-28T19:43:50.589 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.589 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.589 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.589 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.589 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.589 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.590 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.590 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:50.590 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:50.590 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.590 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.590 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.591 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-08-28T19:43:50.591 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.591 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.591 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.591 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.591 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.591 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.592 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.592 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.592 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.592 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.592 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.592 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.593 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.593 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.593 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.593 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.593 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.593 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.593 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.594 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.594 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.594 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.594 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.594 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.594 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.595 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.595 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.595 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.596 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.596 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.596 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.596 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.596 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.596 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.597 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.597 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.597 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.597 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.598 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.598 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.598 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.598 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.598 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.598 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.598 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.599 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.599 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.599 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.599 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.599 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.599 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.599 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.600 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.600 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.600 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.600 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.600 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.600 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.601 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.601 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.601 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.601 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.601 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.601 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.601 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.602 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.602 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.602 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.602 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.602 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.602 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.603 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.603 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.603 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.603 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.603 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.604 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-08-28T19:43:50.604 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.604 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.604 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.604 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-08-28T19:43:50.604 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-08-28T19:43:50.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.605 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.605 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.605 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.605 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.605 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.605 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-08-28T19:43:50.606 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.606 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.606 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.606 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-08-28T19:43:50.606 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-08-28T19:43:50.606 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.606 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.607 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.607 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.607 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.607 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.607 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-08-28T19:43:50.607 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.608 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.608 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.608 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-08-28T19:43:50.608 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-08-28T19:43:50.608 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.608 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.609 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.609 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.609 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.609 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.609 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.609 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-08-28T19:43:50.609 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.609 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.610 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.610 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-08-28T19:43:50.610 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-08-28T19:43:50.610 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.610 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.611 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.611 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.611 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.611 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.611 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-08-28T19:43:50.611 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.611 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.612 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.612 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-08-28T19:43:50.612 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-08-28T19:43:50.612 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.612 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.613 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.613 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.613 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.613 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.613 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.613 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-08-28T19:43:50.613 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.614 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.614 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.614 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-08-28T19:43:50.614 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-08-28T19:43:50.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.614 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.615 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.615 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.615 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.615 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.615 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.615 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.616 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.616 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.616 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-08-28T19:43:50.616 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-08-28T19:43:50.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.617 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.617 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.617 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.617 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.617 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.617 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.617 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.617 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.618 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.618 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-08-28T19:43:50.618 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-08-28T19:43:50.618 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.618 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.619 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-08-28T19:43:50.619 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.619 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.619 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.619 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.619 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.620 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.620 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-08-28T19:43:50.620 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-08-28T19:43:50.620 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.620 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.620 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.621 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.621 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.621 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.621 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.621 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.621 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.621 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.621 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.622 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:50.622 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:50.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.622 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.622 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-08-28T19:43:50.622 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-08-28T19:43:50.623 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.623 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.623 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.623 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.623 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.623 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.624 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-08-28T19:43:50.624 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-08-28T19:43:50.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.624 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.624 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.624 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-08-28T19:43:50.624 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.625 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.625 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.625 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.625 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.625 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.625 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.625 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-08-28T19:43:50.626 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-08-28T19:43:50.626 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.626 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.626 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.626 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.626 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.626 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.627 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.627 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.627 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.627 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.627 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.627 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-08-28T19:43:50.627 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-08-28T19:43:50.628 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.628 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.628 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.628 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.628 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.628 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.628 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.628 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.629 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.629 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.629 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.629 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.629 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.630 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.630 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.630 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.630 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.630 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.630 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.630 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.631 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.631 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.631 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.631 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.631 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.631 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.631 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.631 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.632 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.632 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.632 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.632 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.632 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.633 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.633 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.633 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.633 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.633 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.633 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.634 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.634 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.634 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.634 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.635 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.635 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.635 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.635 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.635 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.635 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.635 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.636 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.636 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.636 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.636 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.636 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.636 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.636 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.636 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.637 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.637 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.637 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.637 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.637 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.638 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.638 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.638 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.638 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.638 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.638 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.638 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.639 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-08-28T19:43:50.639 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-08-28T19:43:50.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.639 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.639 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.640 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.640 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.640 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.640 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.640 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.640 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.641 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-08-28T19:43:50.641 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-08-28T19:43:50.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.641 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.641 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.641 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.642 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.642 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.642 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.642 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.642 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.642 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.642 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.643 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.643 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.643 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.643 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.644 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.644 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.644 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.644 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.644 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.644 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.644 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.645 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-08-28T19:43:50.645 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-08-28T19:43:50.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.645 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.645 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-08-28T19:43:50.646 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.646 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.646 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.646 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.646 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.646 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.646 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.647 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-08-28T19:43:50.647 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.648 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.648 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.648 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.648 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.648 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.649 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.649 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.649 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.649 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.649 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.649 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.649 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.650 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.650 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.650 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.650 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.650 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.650 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.650 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.651 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.651 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.651 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.651 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.652 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.652 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.652 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.652 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.652 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.652 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.652 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.653 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.653 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.653 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.653 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.653 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.654 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.654 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.654 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.654 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.654 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.654 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.654 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.654 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.656 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.658 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-08-28T19:43:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.660 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.661 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.662 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.663 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.664 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.668 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.670 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.670 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.670 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.670 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.670 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.671 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-08-28T19:43:50.671 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-08-28T19:43:50.671 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.671 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.671 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.671 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-08-28T19:43:50.671 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.671 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.672 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.672 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-08-28T19:43:50.672 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.672 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.672 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.672 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-08-28T19:43:50.672 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-08-28T19:43:50.673 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.673 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.673 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.673 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.673 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.673 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.673 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.674 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-08-28T19:43:50.674 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.674 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.674 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.674 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-08-28T19:43:50.674 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-08-28T19:43:50.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.675 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.675 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.675 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.675 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.675 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.675 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-08-28T19:43:50.675 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.676 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.676 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-08-28T19:43:50.676 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-08-28T19:43:50.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.676 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.677 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.677 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.677 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.677 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.677 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.677 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-08-28T19:43:50.677 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.677 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.678 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.678 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.678 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.678 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.678 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.678 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.678 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.679 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.679 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.679 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.679 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.679 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.679 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-08-28T19:43:50.680 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.680 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.681 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.681 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-08-28T19:43:50.681 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-08-28T19:43:50.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:50.681 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.681 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-08-28T19:43:50.682 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.682 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.682 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.682 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.682 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.682 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.682 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-08-28T19:43:50.683 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-08-28T19:43:50.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.683 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.683 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.683 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.683 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.686 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.686 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.686 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.686 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.686 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.688 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.688 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.688 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.688 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.688 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.688 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.690 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.690 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.690 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.690 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.690 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.690 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.690 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.692 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.692 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.692 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.692 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.692 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.692 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.692 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.692 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.693 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.693 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.693 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.693 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.693 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-08-28T19:43:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-08-28T19:43:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-08-28T19:43:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-08-28T19:43:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-08-28T19:43:50.700 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.700 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.700 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.700 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.700 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.700 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.701 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-08-28T19:43:50.701 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-08-28T19:43:50.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.701 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.701 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-08-28T19:43:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:43:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: "modules": [ 2024-08-28T19:43:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-08-28T19:43:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard", 2024-08-28T19:43:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: "iostat", 2024-08-28T19:43:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: "nfs", 2024-08-28T19:43:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: "restful", 2024-08-28T19:43:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_schedule" 2024-08-28T19:43:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-08-28T19:43:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-08-28T19:43:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-08-28T19:43:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.708 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.708 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.708 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.708 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.708 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.708 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.710 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.710 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.710 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.710 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.710 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.710 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.712 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.712 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.712 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.712 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.713 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.713 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.713 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.713 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.713 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.713 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.713 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.714 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.714 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.714 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.714 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.715 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.715 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.715 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.715 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.715 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.715 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.716 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.716 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.716 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.716 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.716 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-08-28T19:43:50.717 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-08-28T19:43:50.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.717 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.717 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.717 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.717 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.717 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.718 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-08-28T19:43:50.718 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.718 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.718 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.718 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-08-28T19:43:50.718 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-08-28T19:43:50.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.719 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.719 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-08-28T19:43:50.719 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.719 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.719 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.719 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-08-28T19:43:50.720 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.720 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.720 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.720 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-08-28T19:43:50.720 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-08-28T19:43:50.720 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.721 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.721 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.721 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.721 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.721 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.721 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.721 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-08-28T19:43:50.721 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.722 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.722 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-08-28T19:43:50.722 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-08-28T19:43:50.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.722 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.723 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.723 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.723 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.723 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.723 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.723 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-08-28T19:43:50.723 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.724 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.724 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-08-28T19:43:50.724 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-08-28T19:43:50.724 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.724 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.724 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-08-28T19:43:50.725 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.725 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.725 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.725 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-08-28T19:43:50.725 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.725 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.726 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-08-28T19:43:50.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-08-28T19:43:50.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.726 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.726 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.726 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.727 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.727 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.727 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-08-28T19:43:50.727 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.727 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.727 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-08-28T19:43:50.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-08-28T19:43:50.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.728 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.728 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.728 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.728 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.728 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.729 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-08-28T19:43:50.729 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.729 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.729 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-08-28T19:43:50.729 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-08-28T19:43:50.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.730 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.730 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.730 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.730 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.730 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.730 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-08-28T19:43:50.731 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.731 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.731 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.731 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.731 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.731 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.732 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.732 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.732 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.732 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.732 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.732 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.732 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.733 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.733 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.733 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.733 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-08-28T19:43:50.734 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.734 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.734 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.734 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-08-28T19:43:50.734 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-08-28T19:43:50.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.734 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.735 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.735 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.735 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.735 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-08-28T19:43:50.735 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.735 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.735 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.736 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-08-28T19:43:50.736 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-08-28T19:43:50.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.736 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.736 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-08-28T19:43:50.737 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.737 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.737 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-08-28T19:43:50.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-28T19:43:50.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.737 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.738 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-08-28T19:43:50.738 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-08-28T19:43:50.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:50.738 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.738 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-08-28T19:43:50.739 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-08-28T19:43:50.739 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.739 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-08-28T19:43:50.739 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-28T19:43:50.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.739 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.740 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-08-28T19:43:50.740 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-08-28T19:43:50.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:50.740 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.740 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-08-28T19:43:50.741 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:50.741 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-08-28T19:43:50.741 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.741 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-08-28T19:43:50.741 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.741 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.742 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-08-28T19:43:50.742 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-08-28T19:43:50.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.742 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-08-28T19:43:50.743 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.743 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.743 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.743 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-08-28T19:43:50.743 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-08-28T19:43:50.743 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.743 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.744 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-08-28T19:43:50.744 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-08-28T19:43:50.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:50.744 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.744 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.744 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-08-28T19:43:50.745 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-08-28T19:43:50.745 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-08-28T19:43:50.745 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.745 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-08-28T19:43:50.745 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-08-28T19:43:50.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.745 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.746 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-08-28T19:43:50.746 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-08-28T19:43:50.746 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.746 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.746 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-08-28T19:43:50.746 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.747 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.747 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.747 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-08-28T19:43:50.747 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-28T19:43:50.747 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.747 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.747 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-08-28T19:43:50.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-08-28T19:43:50.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:50.748 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.748 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.748 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.748 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-08-28T19:43:50.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-08-28T19:43:50.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.749 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-08-28T19:43:50.749 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-28T19:43:50.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.749 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.749 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.750 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.750 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.750 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.750 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.751 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.751 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.751 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.751 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.751 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.751 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.751 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.751 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.752 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.752 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.752 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.752 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.752 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.752 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.753 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.753 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.753 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.753 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.753 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.754 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.754 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.754 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.754 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.755 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.755 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.755 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.756 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.756 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.756 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.756 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.756 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.756 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.756 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.757 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.757 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.757 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.757 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.758 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.758 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.758 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.758 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.758 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.758 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.758 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.759 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.759 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.759 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.759 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-08-28T19:43:50.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-08-28T19:43:50.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:50.760 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.760 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.760 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.760 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.760 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.761 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-08-28T19:43:50.761 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.761 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-08-28T19:43:50.761 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-08-28T19:43:50.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.762 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.762 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-08-28T19:43:50.762 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.762 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.762 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.762 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-08-28T19:43:50.763 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-08-28T19:43:50.763 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-08-28T19:43:50.763 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-08-28T19:43:50.763 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-08-28T19:43:50.763 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.763 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-08-28T19:43:50.763 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.763 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.764 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.764 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-08-28T19:43:50.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-08-28T19:43:50.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.764 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.764 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.764 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.765 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.765 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.765 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.765 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-08-28T19:43:50.765 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.765 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.765 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:50.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.767 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.767 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.767 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-08-28T19:43:50.767 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.767 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.768 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.768 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.768 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.768 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.768 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.768 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.769 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.769 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.769 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.769 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.770 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.770 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-08-28T19:43:50.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-08-28T19:43:50.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.770 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.770 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.770 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:50.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:50.771 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.771 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.771 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-08-28T19:43:50.771 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-08-28T19:43:50.771 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.771 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.772 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-08-28T19:43:50.772 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-08-28T19:43:50.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:50.772 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.772 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.772 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:50.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.773 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.773 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-08-28T19:43:50.773 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.773 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.773 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.774 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-08-28T19:43:50.774 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.774 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.774 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.775 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-08-28T19:43:50.775 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-08-28T19:43:50.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:50.775 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.775 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-08-28T19:43:50.775 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.775 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.776 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.776 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-08-28T19:43:50.776 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.776 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.776 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-08-28T19:43:50.777 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-08-28T19:43:50.777 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.777 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.777 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.777 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-08-28T19:43:50.777 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.778 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.778 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.778 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-08-28T19:43:50.778 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.778 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.778 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.778 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-08-28T19:43:50.779 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-08-28T19:43:50.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.779 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.779 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.779 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-08-28T19:43:50.779 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.780 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.780 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.780 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-08-28T19:43:50.780 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.780 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.781 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-08-28T19:43:50.781 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-08-28T19:43:50.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.781 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.781 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.781 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.782 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.782 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.782 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-08-28T19:43:50.782 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-08-28T19:43:50.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.782 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.783 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-08-28T19:43:50.783 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-08-28T19:43:50.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.783 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.783 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:50.783 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.784 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.784 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.784 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-08-28T19:43:50.784 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.784 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.784 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.785 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-08-28T19:43:50.785 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-08-28T19:43:50.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:50.785 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.785 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.785 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-08-28T19:43:50.786 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.786 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.786 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.786 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-08-28T19:43:50.786 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.786 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.786 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.787 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-08-28T19:43:50.787 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-08-28T19:43:50.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.787 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.787 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.787 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.787 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.788 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.788 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-08-28T19:43:50.788 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-08-28T19:43:50.788 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-08-28T19:43:50.788 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.788 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-08-28T19:43:50.788 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.789 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.789 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.789 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-08-28T19:43:50.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-08-28T19:43:50.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.789 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.790 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.790 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.790 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.790 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.790 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.790 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-08-28T19:43:50.790 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.791 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.791 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.791 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-08-28T19:43:50.791 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-08-28T19:43:50.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.791 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.792 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.792 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.792 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.792 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.792 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-08-28T19:43:50.792 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.793 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.793 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-08-28T19:43:50.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-08-28T19:43:50.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.794 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.794 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.794 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.794 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.794 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-08-28T19:43:50.794 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.795 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.795 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-08-28T19:43:50.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-08-28T19:43:50.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.795 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.795 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.796 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-08-28T19:43:50.796 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.796 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.796 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.796 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:50.796 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.796 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.797 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-08-28T19:43:50.797 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-08-28T19:43:50.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.797 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.798 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-08-28T19:43:50.798 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.798 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.798 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.798 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-08-28T19:43:50.798 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.798 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.799 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-08-28T19:43:50.799 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-08-28T19:43:50.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.799 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.799 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-08-28T19:43:50.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.800 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.800 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.800 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-08-28T19:43:50.800 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.800 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.801 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-08-28T19:43:50.801 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-08-28T19:43:50.801 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.801 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.801 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.801 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:10.4.0", 2024-08-28T19:43:50.801 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.802 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.802 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.802 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:50.802 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.802 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.803 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-08-28T19:43:50.803 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-08-28T19:43:50.803 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.803 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.803 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.803 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-08-28T19:43:50.803 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.804 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.804 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.804 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-08-28T19:43:50.804 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.804 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.804 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.804 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-08-28T19:43:50.805 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-08-28T19:43:50.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.805 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.805 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-08-28T19:43:50.805 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.806 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.806 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.806 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-08-28T19:43:50.806 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.806 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-08-28T19:43:50.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-08-28T19:43:50.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.807 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-08-28T19:43:50.807 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.808 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.808 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-08-28T19:43:50.808 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.808 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.808 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.808 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-08-28T19:43:50.809 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-08-28T19:43:50.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.809 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-08-28T19:43:50.809 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.809 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.810 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.810 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-08-28T19:43:50.810 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.810 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.810 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-08-28T19:43:50.811 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-08-28T19:43:50.811 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.811 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.811 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-08-28T19:43:50.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.811 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.812 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.812 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-08-28T19:43:50.812 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.812 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.812 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.812 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-08-28T19:43:50.813 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-08-28T19:43:50.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.813 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.813 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-08-28T19:43:50.813 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.813 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.814 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.814 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-08-28T19:43:50.814 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.815 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nginx": { 2024-08-28T19:43:50.815 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nginx", 2024-08-28T19:43:50.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.815 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.815 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nginx:1.26.1", 2024-08-28T19:43:50.815 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.816 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.816 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.816 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nginx container image", 2024-08-28T19:43:50.816 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.816 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-08-28T19:43:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-08-28T19:43:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-08-28T19:43:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-08-28T19:43:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-08-28T19:43:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-08-28T19:43:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-08-28T19:43:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_oauth2_proxy": { 2024-08-28T19:43:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_oauth2_proxy", 2024-08-28T19:43:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-08-28T19:43:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "oauth2-proxy container image", 2024-08-28T19:43:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-08-28T19:43:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-08-28T19:43:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-08-28T19:43:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-08-28T19:43:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-08-28T19:43:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-08-28T19:43:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-08-28T19:43:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-08-28T19:43:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-08-28T19:43:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-08-28T19:43:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-08-28T19:43:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-08-28T19:43:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-08-28T19:43:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-08-28T19:43:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-08-28T19:43:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-08-28T19:43:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-08-28T19:43:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-08-28T19:43:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-08-28T19:43:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-08-28T19:43:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-08-28T19:43:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-08-28T19:43:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-08-28T19:43:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-08-28T19:43:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-08-28T19:43:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-08-28T19:43:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-08-28T19:43:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-08-28T19:43:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-08-28T19:43:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-08-28T19:43:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-08-28T19:43:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-08-28T19:43:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-08-28T19:43:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-08-28T19:43:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-08-28T19:43:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-08-28T19:43:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-08-28T19:43:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-08-28T19:43:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-08-28T19:43:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-08-28T19:43:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-08-28T19:43:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-08-28T19:43:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-08-28T19:43:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-08-28T19:43:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-08-28T19:43:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-08-28T19:43:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-08-28T19:43:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-08-28T19:43:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-08-28T19:43:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.850 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.850 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-08-28T19:43:50.850 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.850 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-08-28T19:43:50.851 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-08-28T19:43:50.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.851 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.852 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.852 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-08-28T19:43:50.852 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.852 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.852 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.853 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.853 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.853 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.853 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.854 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.854 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.854 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.854 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.854 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.854 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.854 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.855 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.855 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.855 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.855 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.855 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-08-28T19:43:50.856 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-08-28T19:43:50.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.856 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.856 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.856 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.856 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.857 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.857 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.857 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-08-28T19:43:50.857 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.857 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.857 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.858 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.858 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.858 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.858 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.859 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.859 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.859 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.859 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-08-28T19:43:50.859 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.859 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.860 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.860 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.861 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.861 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.861 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.861 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.861 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.862 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.862 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.862 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.862 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.862 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.862 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.863 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.863 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.863 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.863 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.863 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.863 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.863 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.864 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.864 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.864 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.864 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.865 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-08-28T19:43:50.865 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-08-28T19:43:50.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.865 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.866 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.866 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-08-28T19:43:50.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.866 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-08-28T19:43:50.867 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-08-28T19:43:50.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-08-28T19:43:50.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.868 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-08-28T19:43:50.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.868 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-08-28T19:43:50.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-08-28T19:43:50.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.869 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:50.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.869 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-08-28T19:43:50.870 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.870 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.870 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.870 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-08-28T19:43:50.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-08-28T19:43:50.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.871 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:50.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.871 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.871 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-08-28T19:43:50.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.872 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.872 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-08-28T19:43:50.872 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-08-28T19:43:50.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.872 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.872 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.873 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-08-28T19:43:50.873 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.873 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.874 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-08-28T19:43:50.874 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-08-28T19:43:50.874 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.874 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.874 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-08-28T19:43:50.874 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.874 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.875 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.875 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-08-28T19:43:50.875 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-08-28T19:43:50.875 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.875 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-08-28T19:43:50.875 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.875 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.876 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-08-28T19:43:50.876 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-08-28T19:43:50.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.876 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.876 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-08-28T19:43:50.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.877 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-08-28T19:43:50.877 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.877 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.878 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-08-28T19:43:50.878 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-08-28T19:43:50.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.878 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.878 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.878 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-08-28T19:43:50.878 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.878 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.879 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.879 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-08-28T19:43:50.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.879 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-08-28T19:43:50.879 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-08-28T19:43:50.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.880 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.880 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-08-28T19:43:50.881 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.881 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.881 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-08-28T19:43:50.881 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-08-28T19:43:50.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.882 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.882 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.882 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.882 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.882 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.882 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-08-28T19:43:50.882 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.883 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.883 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-08-28T19:43:50.883 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-08-28T19:43:50.883 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.883 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.883 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.884 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.884 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.884 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-08-28T19:43:50.884 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.884 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.884 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.885 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-08-28T19:43:50.885 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-08-28T19:43:50.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.885 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.885 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.885 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.886 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.886 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.886 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-08-28T19:43:50.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.886 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.887 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-08-28T19:43:50.887 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-08-28T19:43:50.887 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.887 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.887 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.887 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.887 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.888 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.888 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-08-28T19:43:50.888 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.888 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.888 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.888 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-08-28T19:43:50.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-08-28T19:43:50.889 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.889 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.889 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.889 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-08-28T19:43:50.889 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.889 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-08-28T19:43:50.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.890 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.890 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.890 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.890 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.891 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.891 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.891 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.891 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.892 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.892 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-08-28T19:43:50.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-08-28T19:43:50.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.892 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.892 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.892 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.893 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-08-28T19:43:50.893 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.893 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.894 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-08-28T19:43:50.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-08-28T19:43:50.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.894 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.894 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.894 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.894 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-08-28T19:43:50.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.895 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-08-28T19:43:50.895 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-08-28T19:43:50.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.896 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-08-28T19:43:50.897 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.897 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.897 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.897 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-08-28T19:43:50.897 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-08-28T19:43:50.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.898 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.898 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.898 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.898 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.898 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-08-28T19:43:50.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.899 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.899 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-08-28T19:43:50.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-08-28T19:43:50.899 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.899 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.900 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.900 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.900 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.900 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.900 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-08-28T19:43:50.900 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.900 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.901 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-08-28T19:43:50.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-08-28T19:43:50.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.901 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.901 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.901 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-08-28T19:43:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-08-28T19:43:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:50.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:50.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.904 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.904 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.904 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.905 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.905 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.905 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.905 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.905 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.905 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.905 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.905 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.906 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.906 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.906 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.906 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:50.906 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:50.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.906 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.907 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.907 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.907 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.907 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.907 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.907 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.907 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.908 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.908 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:50.908 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:50.908 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.908 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.908 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.908 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:50.909 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.909 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.909 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:50.909 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:50.909 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:50.909 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:50.909 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:50.909 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:50.910 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:50.910 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:50.910 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.910 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.910 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.910 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.910 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-08-28T19:43:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-08-28T19:43:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.913 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-08-28T19:43:50.913 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.913 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.913 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.914 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-08-28T19:43:50.914 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.914 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.914 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.914 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:50.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:50.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.915 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:50.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.915 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:50.915 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.915 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.915 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.915 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.915 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.916 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.916 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.916 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-08-28T19:43:50.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-08-28T19:43:50.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:50.916 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:50.917 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-08-28T19:43:50.917 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.917 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.917 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.917 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-08-28T19:43:50.917 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.918 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:50.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:50.918 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-08-28T19:43:50.918 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:50.918 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:50.919 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:50.919 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-08-28T19:43:50.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-08-28T19:43:50.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.919 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.919 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:50.919 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.920 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.920 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.920 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.921 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-08-28T19:43:50.921 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-08-28T19:43:50.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.921 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.921 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.921 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.922 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.922 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.922 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.922 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.922 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.922 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-08-28T19:43:50.923 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-08-28T19:43:50.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.923 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.923 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.923 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.924 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.924 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.924 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.924 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.924 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-08-28T19:43:50.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-08-28T19:43:50.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.925 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.925 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.925 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.925 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.925 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.925 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.926 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.926 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.926 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-08-28T19:43:50.926 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-08-28T19:43:50.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.927 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.927 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.927 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.927 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.927 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.927 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.928 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.928 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-08-28T19:43:50.928 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-08-28T19:43:50.928 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.928 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.929 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.929 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.929 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.929 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.929 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.929 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.929 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.930 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.930 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-08-28T19:43:50.930 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-08-28T19:43:50.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.930 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.930 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.930 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-08-28T19:43:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-08-28T19:43:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.933 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.933 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.933 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.933 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.933 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.933 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.933 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-08-28T19:43:50.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-08-28T19:43:50.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.934 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.934 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.934 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.934 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.934 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-08-28T19:43:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-08-28T19:43:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.936 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.936 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.936 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.936 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.936 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.937 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.937 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.937 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-08-28T19:43:50.937 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-08-28T19:43:50.937 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.938 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.938 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.938 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.938 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.938 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.938 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.939 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.939 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.939 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.939 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-08-28T19:43:50.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-08-28T19:43:50.939 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.940 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.940 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.940 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.940 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.940 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.940 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.941 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.941 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-08-28T19:43:50.941 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-08-28T19:43:50.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.942 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.942 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.942 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.942 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.942 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.942 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.942 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.943 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.943 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.943 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-08-28T19:43:50.943 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-08-28T19:43:50.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.943 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.944 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.944 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.944 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.944 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.944 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.944 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.944 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.945 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.945 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-08-28T19:43:50.945 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-08-28T19:43:50.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.945 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.945 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-08-28T19:43:50.946 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.946 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.946 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.946 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.946 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.946 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.946 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.947 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-08-28T19:43:50.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-08-28T19:43:50.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.947 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.947 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.947 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.948 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.948 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.948 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.948 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.948 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.949 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-08-28T19:43:50.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-08-28T19:43:50.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.949 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.949 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.949 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.949 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.950 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.950 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.950 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.950 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.950 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.950 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.951 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-08-28T19:43:50.951 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-08-28T19:43:50.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.951 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.951 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-08-28T19:43:50.952 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.952 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.952 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.952 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.952 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.952 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.953 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-08-28T19:43:50.953 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-08-28T19:43:50.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.953 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.953 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.953 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.954 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.954 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.954 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.954 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.954 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.955 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-08-28T19:43:50.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-08-28T19:43:50.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.955 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.955 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.955 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.956 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.956 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.956 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.956 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.957 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-08-28T19:43:50.957 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-08-28T19:43:50.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.957 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.958 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.958 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.958 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.958 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.958 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-08-28T19:43:50.959 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-08-28T19:43:50.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.959 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.959 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.959 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.959 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.959 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.960 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.960 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.960 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.960 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.960 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.960 INFO:teuthology.orchestra.run.smithi016.stdout: "MANAGED_BY_CLUSTERS": { 2024-08-28T19:43:50.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-08-28T19:43:50.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.961 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.961 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.961 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "[]", 2024-08-28T19:43:50.961 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.961 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.961 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.962 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.962 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.962 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.962 INFO:teuthology.orchestra.run.smithi016.stdout: "MULTICLUSTER_CONFIG": { 2024-08-28T19:43:50.963 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MULTICLUSTER_CONFIG", 2024-08-28T19:43:50.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.963 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.963 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.963 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "{}", 2024-08-28T19:43:50.963 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.963 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.964 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.964 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.964 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.964 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.964 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.965 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-08-28T19:43:50.965 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-08-28T19:43:50.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.965 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.965 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.965 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.965 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.966 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.966 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.966 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.966 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.967 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-08-28T19:43:50.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-08-28T19:43:50.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.967 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.967 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.967 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.968 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.968 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.968 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.968 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.968 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.969 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-08-28T19:43:50.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-08-28T19:43:50.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.969 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.969 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.970 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.970 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.970 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.970 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.970 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.971 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-08-28T19:43:50.971 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-08-28T19:43:50.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.971 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.971 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.971 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.972 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.972 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.972 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.972 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.972 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.972 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.973 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-08-28T19:43:50.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-08-28T19:43:50.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.973 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.973 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.973 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.974 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.974 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.974 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.974 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-08-28T19:43:50.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-08-28T19:43:50.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.975 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.975 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.975 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.975 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.976 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.976 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.976 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-08-28T19:43:50.976 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-08-28T19:43:50.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.977 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.977 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.977 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.977 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.977 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.977 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.978 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.978 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.978 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-08-28T19:43:50.978 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-08-28T19:43:50.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.978 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.979 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.979 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.979 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.979 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.979 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.979 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.979 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.980 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-08-28T19:43:50.980 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-08-28T19:43:50.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.980 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:50.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.981 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.981 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.981 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.981 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.982 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.982 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.982 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-08-28T19:43:50.982 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-08-28T19:43:50.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.982 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.983 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.983 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.983 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.983 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.983 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.983 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.984 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-08-28T19:43:50.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-08-28T19:43:50.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.984 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.984 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-08-28T19:43:50.985 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.985 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.985 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.985 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.985 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.985 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.986 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-08-28T19:43:50.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-08-28T19:43:50.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:50.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.987 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.987 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.987 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.987 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-08-28T19:43:50.988 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-08-28T19:43:50.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.988 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.988 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.988 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-08-28T19:43:50.988 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.988 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.989 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.989 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.989 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.989 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.989 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-08-28T19:43:50.989 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-08-28T19:43:50.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:50.990 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.990 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.990 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-08-28T19:43:50.990 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.990 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.990 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.991 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.991 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.991 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.991 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.991 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-08-28T19:43:50.991 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-08-28T19:43:50.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.992 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.992 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.992 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.992 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.992 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.992 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.993 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.993 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.993 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.993 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.993 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-08-28T19:43:50.993 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-08-28T19:43:50.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.994 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.994 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-08-28T19:43:50.994 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.994 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.994 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.995 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.995 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.995 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.995 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-08-28T19:43:50.995 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-08-28T19:43:50.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:50.996 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.996 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:50.996 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.996 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.997 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.997 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.997 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.997 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-08-28T19:43:50.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-08-28T19:43:50.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.998 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.998 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:50.998 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:50.998 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:50.998 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:50.998 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:50.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:50.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:50.999 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:50.999 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:50.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:50.999 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-08-28T19:43:50.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-08-28T19:43:50.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:50.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:50.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.000 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.000 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.000 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.000 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.000 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.001 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.001 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-08-28T19:43:51.001 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-08-28T19:43:51.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.002 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.002 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.002 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.002 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.002 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.002 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.002 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.003 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-08-28T19:43:51.003 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-08-28T19:43:51.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.003 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.003 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.003 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:51.003 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.004 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.004 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.004 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.004 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.004 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.004 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.005 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-08-28T19:43:51.005 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-08-28T19:43:51.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.005 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.005 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:51.005 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.006 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.006 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.006 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.006 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.006 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.006 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-08-28T19:43:51.007 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-08-28T19:43:51.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.007 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.007 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.007 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.007 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.008 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.008 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.008 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.008 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.009 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-08-28T19:43:51.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-08-28T19:43:51.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.009 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.009 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.009 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.010 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.010 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.010 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.010 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.010 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.010 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.011 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-08-28T19:43:51.011 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-08-28T19:43:51.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.011 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.011 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.011 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.011 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.012 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-08-28T19:43:51.012 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.013 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-08-28T19:43:51.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-08-28T19:43:51.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.013 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-08-28T19:43:51.013 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.014 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.014 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.014 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.014 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.014 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.015 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-08-28T19:43:51.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-08-28T19:43:51.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.015 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.016 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.016 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.016 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.016 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.016 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.016 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.016 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.017 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.017 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.017 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.017 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.018 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.018 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.018 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.018 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.018 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.018 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.018 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.019 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.019 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.019 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.019 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.019 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.019 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.019 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.020 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.020 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.020 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.020 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.021 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.021 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.021 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.021 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.022 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.022 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.022 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.022 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.023 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.023 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.023 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.023 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.023 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.023 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.023 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.024 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.024 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.024 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.024 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.024 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.024 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.024 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.025 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.025 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.025 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.025 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.025 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.026 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.026 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.026 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.026 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.026 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.026 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.026 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.027 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-08-28T19:43:51.027 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-08-28T19:43:51.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.027 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.027 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.027 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.028 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.028 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.028 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-08-28T19:43:51.028 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.028 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.028 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.029 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-08-28T19:43:51.029 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-08-28T19:43:51.029 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.029 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.029 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.029 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.030 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.030 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.030 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.030 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.030 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.030 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.030 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.031 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-08-28T19:43:51.031 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-08-28T19:43:51.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.031 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.031 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-08-28T19:43:51.032 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.032 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.032 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.032 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.032 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.032 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.033 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.033 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-08-28T19:43:51.033 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-08-28T19:43:51.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.033 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-08-28T19:43:51.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.034 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.034 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.034 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.034 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.035 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.035 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.035 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.035 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.035 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.036 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.036 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.036 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.036 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.036 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.036 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.036 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.037 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-08-28T19:43:51.037 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-08-28T19:43:51.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.037 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.038 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.038 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.038 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.038 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.038 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.039 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.039 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-08-28T19:43:51.039 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-08-28T19:43:51.039 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.039 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.039 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.040 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-08-28T19:43:51.040 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.040 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.040 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.040 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.040 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.040 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.041 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.041 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-08-28T19:43:51.041 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-08-28T19:43:51.041 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.041 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.041 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.042 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-08-28T19:43:51.042 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.042 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.042 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.042 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.042 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-08-28T19:43:51.042 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.043 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.043 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.043 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.043 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-08-28T19:43:51.043 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-08-28T19:43:51.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.044 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.044 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-08-28T19:43:51.044 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-08-28T19:43:51.044 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-08-28T19:43:51.044 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.044 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.045 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.045 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.045 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-08-28T19:43:51.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-08-28T19:43:51.045 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.045 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.046 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.046 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.046 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.046 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.046 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.046 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.046 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.047 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.047 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.047 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.047 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.047 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-08-28T19:43:51.047 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.048 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.048 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.048 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-08-28T19:43:51.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-08-28T19:43:51.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.048 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.048 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.048 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.049 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.049 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.049 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.049 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-08-28T19:43:51.049 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.049 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.050 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.050 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.050 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.051 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.051 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.051 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.051 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.051 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.051 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.052 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.052 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.052 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.053 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.053 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.053 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.053 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.054 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.054 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.054 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.054 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.054 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.055 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.055 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.055 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.056 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.056 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.056 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.056 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.056 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.056 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.056 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-08-28T19:43:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-08-28T19:43:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-08-28T19:43:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-08-28T19:43:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-08-28T19:43:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-08-28T19:43:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-08-28T19:43:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-08-28T19:43:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-08-28T19:43:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-08-28T19:43:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-08-28T19:43:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-08-28T19:43:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-08-28T19:43:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-08-28T19:43:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-08-28T19:43:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-08-28T19:43:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-08-28T19:43:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-08-28T19:43:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-08-28T19:43:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.069 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:51.069 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:51.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:51.069 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.072 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.072 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:51.072 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.072 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.073 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.073 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-08-28T19:43:51.073 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.073 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.073 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.073 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.074 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.074 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.074 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.074 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.074 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.074 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.075 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.075 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.075 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.075 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.075 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-08-28T19:43:51.076 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-08-28T19:43:51.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:51.076 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-08-28T19:43:51.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.077 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.077 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-08-28T19:43:51.077 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.077 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.077 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.078 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-08-28T19:43:51.078 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.078 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.078 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.079 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.079 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.079 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.079 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.079 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.079 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.080 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.080 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.080 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.080 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.080 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.080 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.080 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.081 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.081 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.081 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.081 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.081 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.081 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.081 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.082 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.082 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.082 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.082 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.082 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.083 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.083 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.084 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.084 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.084 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.084 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.084 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.085 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.085 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.085 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.085 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.085 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.085 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.086 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.086 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.086 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.087 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.087 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.087 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.088 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.088 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.088 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.088 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.088 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-08-28T19:43:51.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-08-28T19:43:51.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.089 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.089 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-08-28T19:43:51.089 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.089 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.089 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.090 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.090 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-08-28T19:43:51.090 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-08-28T19:43:51.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.091 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-08-28T19:43:51.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.091 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.091 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.092 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.092 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.092 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:51.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:51.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.092 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.093 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-08-28T19:43:51.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.093 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.094 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.094 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.094 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.095 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.095 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.095 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.095 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.095 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.095 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.096 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.096 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.096 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-08-28T19:43:51.097 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-08-28T19:43:51.097 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-08-28T19:43:51.097 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.097 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-08-28T19:43:51.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-08-28T19:43:51.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.097 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.098 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-08-28T19:43:51.098 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.098 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.098 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.098 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-08-28T19:43:51.098 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.099 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.099 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.099 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-08-28T19:43:51.099 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-08-28T19:43:51.099 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.099 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.099 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.100 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-08-28T19:43:51.100 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.100 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.100 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.100 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-08-28T19:43:51.100 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.101 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.101 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.101 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-08-28T19:43:51.101 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-08-28T19:43:51.101 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.101 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.102 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.102 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.102 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.102 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-08-28T19:43:51.102 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.102 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.103 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:51.103 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:51.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:51.103 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.103 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-08-28T19:43:51.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-08-28T19:43:51.104 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.104 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.104 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-08-28T19:43:51.104 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.104 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.105 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.105 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.105 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.105 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.106 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.106 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.106 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.106 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.106 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.106 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.106 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.107 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.107 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.107 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.107 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.107 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.110 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.110 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.110 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.110 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.110 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.110 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.110 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.111 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.111 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.111 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.111 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.111 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.111 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.111 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.111 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.112 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.112 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.112 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.112 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.112 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.112 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.113 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.113 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.113 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.114 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.114 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-08-28T19:43:51.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-08-28T19:43:51.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.115 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.115 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.115 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.115 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.115 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.115 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-08-28T19:43:51.115 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.116 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.116 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-08-28T19:43:51.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-08-28T19:43:51.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.116 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.117 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-08-28T19:43:51.117 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.117 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-08-28T19:43:51.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.118 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.118 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.118 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.118 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.118 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.119 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.119 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.119 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.119 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.119 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.119 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.119 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.120 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-08-28T19:43:51.120 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-08-28T19:43:51.120 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.120 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.120 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.120 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-08-28T19:43:51.121 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.121 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.121 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.121 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-08-28T19:43:51.121 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.122 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-08-28T19:43:51.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-08-28T19:43:51.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.122 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.122 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.122 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:51.122 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:51.123 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-08-28T19:43:51.123 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.123 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-08-28T19:43:51.123 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.123 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.124 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-08-28T19:43:51.124 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-08-28T19:43:51.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.124 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.124 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.125 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.125 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.125 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.125 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-08-28T19:43:51.125 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.125 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.126 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-08-28T19:43:51.126 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-08-28T19:43:51.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.126 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.127 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-08-28T19:43:51.127 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.127 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.127 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.127 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-08-28T19:43:51.127 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.128 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.128 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.128 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-08-28T19:43:51.128 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.129 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.129 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.129 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.129 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.129 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.129 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.130 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.130 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.130 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.130 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.130 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.130 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.130 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.131 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.131 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.131 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.131 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.131 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.131 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.132 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.132 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.132 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.132 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.133 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.133 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.133 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.133 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.133 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.133 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.134 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.134 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.134 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.134 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.135 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.135 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.135 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.135 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.135 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.135 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.135 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.135 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.136 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.136 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.136 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.136 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.136 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.136 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.137 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.137 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.138 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.138 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.138 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.138 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.138 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.139 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.139 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.139 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.139 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.139 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.139 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.140 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.140 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.140 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.141 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-08-28T19:43:51.141 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.141 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.141 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.141 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.142 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.142 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.142 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.142 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.142 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.142 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.143 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.143 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.143 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.143 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.143 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.143 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.143 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.144 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.144 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.144 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.144 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.145 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.145 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.145 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.145 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.145 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.145 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.146 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.146 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.146 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.146 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.147 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.147 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.147 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.147 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.147 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.147 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.148 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.148 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.148 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.148 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.148 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.148 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.148 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.149 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.149 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.149 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.149 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.149 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.150 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.150 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.150 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.150 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.150 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.150 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.150 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.150 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.151 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.151 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.151 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.151 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.152 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.152 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.152 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.152 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.153 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.153 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.153 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.153 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.153 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.153 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-08-28T19:43:51.154 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.154 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.154 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.154 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-08-28T19:43:51.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-08-28T19:43:51.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.155 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.155 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-08-28T19:43:51.155 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.155 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.155 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.155 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-08-28T19:43:51.156 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.156 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.156 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-08-28T19:43:51.156 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-08-28T19:43:51.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.157 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.157 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:51.157 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-08-28T19:43:51.157 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.157 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.157 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-08-28T19:43:51.158 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.158 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.158 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.158 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.158 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.158 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.158 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.158 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.159 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.159 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.159 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.159 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.159 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.159 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.160 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.160 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.160 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.160 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.160 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.160 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.161 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.161 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.161 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.162 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.162 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.162 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.162 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.162 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.163 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.163 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.163 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.164 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.164 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.164 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.164 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.164 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.164 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.165 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.165 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.165 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.165 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.166 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.166 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.166 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.166 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.167 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.167 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.167 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.167 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.168 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.168 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.168 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.168 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.168 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.169 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.169 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-08-28T19:43:51.170 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.170 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.170 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.170 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-08-28T19:43:51.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-08-28T19:43:51.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.171 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-08-28T19:43:51.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.171 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.171 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-08-28T19:43:51.172 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-08-28T19:43:51.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.172 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.172 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.173 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.173 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.173 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.173 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.173 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.173 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.173 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.174 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.174 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.174 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.174 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.174 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.174 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.174 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.174 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.175 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.175 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.175 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.175 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.175 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.176 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.176 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.176 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.176 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.176 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.176 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.176 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.177 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.177 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.177 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.177 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.178 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.178 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.178 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.178 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.178 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.178 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.178 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.178 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.179 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.179 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.179 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.179 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.179 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.180 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.180 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.180 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.180 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.180 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.180 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.181 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.181 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.181 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.181 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.181 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.181 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.181 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.182 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-08-28T19:43:51.182 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-08-28T19:43:51.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.182 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.182 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.182 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.182 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.183 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-08-28T19:43:51.183 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-08-28T19:43:51.183 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.183 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.184 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-08-28T19:43:51.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-08-28T19:43:51.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.184 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.184 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.184 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-08-28T19:43:51.184 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.185 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.185 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-08-28T19:43:51.185 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.185 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.186 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-08-28T19:43:51.186 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-08-28T19:43:51.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.186 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.186 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-08-28T19:43:51.186 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.187 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.187 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.187 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-08-28T19:43:51.187 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.187 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.188 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-08-28T19:43:51.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-08-28T19:43:51.188 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.188 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.188 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.188 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.188 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.189 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.189 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.189 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-08-28T19:43:51.189 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.189 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.190 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.190 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.190 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.190 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.190 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.191 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.191 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.191 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.191 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.191 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.191 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-08-28T19:43:51.192 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-08-28T19:43:51.192 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.192 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.192 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.192 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-08-28T19:43:51.192 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.193 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.193 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.193 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-08-28T19:43:51.193 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-08-28T19:43:51.193 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.193 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-08-28T19:43:51.194 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.194 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.194 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.194 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.195 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.195 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.195 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.195 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.195 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.195 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.196 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.196 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.196 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.196 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.196 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.196 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.196 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.197 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.197 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.197 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.197 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.198 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.198 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.198 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.198 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.198 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.198 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.198 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.199 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.199 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.199 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.199 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.199 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.200 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.200 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.200 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.200 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.200 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.200 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.200 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.201 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.201 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.201 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.201 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.201 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.201 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.201 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.202 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.202 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.202 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.203 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.203 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.203 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.203 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.203 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.203 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.203 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.204 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.204 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.204 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.204 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.204 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.204 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.204 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.205 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.205 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.205 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.205 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.205 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-08-28T19:43:51.206 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.206 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.206 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.206 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.208 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.208 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.208 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.208 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.208 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.208 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.208 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.209 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.209 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.209 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.209 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.209 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.210 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.210 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.210 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.210 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.210 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.211 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.211 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.211 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.211 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.211 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.212 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.212 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.212 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.212 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.212 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.212 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.212 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.212 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.213 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.213 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.213 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.213 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.213 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.213 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.214 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.214 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.214 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.214 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.214 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.214 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.214 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.215 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.215 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.215 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.215 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.216 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.216 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.216 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.216 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.216 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.216 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.216 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.216 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.216 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.217 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-08-28T19:43:51.217 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.217 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.217 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.217 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.217 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.218 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.218 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.218 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.218 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.218 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.218 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.219 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.219 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.219 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.219 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.219 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.219 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.219 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.219 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.219 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.220 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.220 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.220 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.220 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.220 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.221 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.221 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.221 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.221 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.221 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.221 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.222 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.222 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.222 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.222 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.222 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.222 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.222 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.222 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.222 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.223 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.223 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.223 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.223 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.223 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.223 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.223 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.223 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.223 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.224 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.224 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.224 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.224 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.224 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.225 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.225 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.225 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.225 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.225 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.225 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.225 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.225 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.226 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.226 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.226 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.226 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.226 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.226 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.226 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.226 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.226 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.227 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.227 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.227 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.227 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.227 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-08-28T19:43:51.228 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.228 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.228 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.228 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-08-28T19:43:51.228 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-08-28T19:43:51.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.228 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.229 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.229 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.229 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.229 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.229 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-08-28T19:43:51.229 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.229 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.229 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.230 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.230 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.230 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.230 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.230 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.231 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.231 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.231 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.231 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.231 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.231 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.231 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.231 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.232 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.232 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.232 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.232 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.232 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.233 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.233 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.233 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.233 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.233 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.233 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.234 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.234 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.234 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.234 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.234 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.235 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.235 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.235 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.235 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.235 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.235 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.235 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.235 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.235 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.235 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.236 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.236 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.236 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.236 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.236 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.236 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.237 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.237 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.237 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.237 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.237 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.238 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-08-28T19:43:51.238 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-08-28T19:43:51.238 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.238 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.238 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.238 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.238 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.238 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.239 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.239 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-08-28T19:43:51.239 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-08-28T19:43:51.239 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-08-28T19:43:51.239 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.239 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-08-28T19:43:51.239 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.239 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.240 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.240 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.240 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.240 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.240 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.240 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.240 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.241 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.241 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.241 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.241 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.241 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.241 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.241 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.242 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-08-28T19:43:51.242 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.242 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.242 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.242 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.242 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.243 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.243 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.243 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.243 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.243 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.243 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.243 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.244 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.244 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.244 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.244 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.244 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.244 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.245 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.245 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.245 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.245 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.245 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.246 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.246 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.246 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.246 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.246 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.246 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.246 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.246 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.247 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.247 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.247 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.247 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.247 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.247 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.247 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.247 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.249 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.250 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.250 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.250 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.250 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.250 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.250 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.251 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.251 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.251 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.251 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.251 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.251 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.252 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.252 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.252 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-08-28T19:43:51.252 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.252 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.253 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.253 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.253 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.253 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.253 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.253 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.254 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.254 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.254 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.254 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.254 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.254 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.254 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.254 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.254 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.255 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.255 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.255 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.255 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.255 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.255 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.257 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.257 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.257 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.258 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.258 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.258 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.258 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.258 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.258 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.258 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.258 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.258 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.258 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.259 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.259 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.259 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.259 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.259 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.259 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.260 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.260 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.260 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.260 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.260 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.261 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.261 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.261 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.261 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.262 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.262 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.262 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.262 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.262 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.262 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-08-28T19:43:51.263 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.263 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.263 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.263 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.263 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.264 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.264 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.264 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.264 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.264 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.264 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.264 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.264 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.264 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.264 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.265 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.265 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.265 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.265 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.265 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.265 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.265 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.265 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.266 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.266 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.266 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.266 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.266 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.266 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.266 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.266 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.266 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.267 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.267 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.267 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.267 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.267 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.267 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.267 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.267 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.268 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.268 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.268 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.268 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.268 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.268 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.268 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.268 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.269 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.269 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.269 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.269 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.269 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.269 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.269 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.269 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.270 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.270 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.270 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.270 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.270 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.270 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.271 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:51.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:51.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:51.271 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.271 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:51.272 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.272 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.272 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.272 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.272 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.272 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.272 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.273 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.273 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.273 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.273 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.273 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.273 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.274 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.274 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.274 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-08-28T19:43:51.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-08-28T19:43:51.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:51.274 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.274 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-08-28T19:43:51.275 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-08-28T19:43:51.275 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.275 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.275 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-08-28T19:43:51.275 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-08-28T19:43:51.275 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.275 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.275 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.275 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-08-28T19:43:51.276 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.276 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.276 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.276 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-08-28T19:43:51.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-08-28T19:43:51.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.277 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.277 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.277 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.277 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.277 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-08-28T19:43:51.277 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.278 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.278 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.278 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-08-28T19:43:51.278 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-08-28T19:43:51.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.279 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.279 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.279 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.279 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.279 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.279 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.280 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.280 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.280 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.280 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.281 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.281 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.281 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.281 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.281 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.281 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.281 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.281 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.281 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.282 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.282 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.282 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.282 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.282 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.282 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.283 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.283 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.283 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.283 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.283 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.283 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.283 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.284 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.284 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.284 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.284 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.284 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.284 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.284 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.284 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.285 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.285 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.285 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.285 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.285 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.285 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.285 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.285 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.285 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.286 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.286 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.286 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.286 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.286 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.286 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.286 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.287 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.287 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.287 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.287 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.287 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.287 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-08-28T19:43:51.288 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-08-28T19:43:51.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.288 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.288 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-08-28T19:43:51.288 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.288 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.288 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.288 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-08-28T19:43:51.289 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.289 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-08-28T19:43:51.289 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-08-28T19:43:51.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:51.289 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.289 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.290 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:51.290 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.290 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.290 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.290 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-08-28T19:43:51.290 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.290 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.290 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.291 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.291 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.291 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.291 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.291 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.291 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.292 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.292 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.292 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.292 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.293 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-08-28T19:43:51.293 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.293 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.293 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.293 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-08-28T19:43:51.293 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-08-28T19:43:51.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.293 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.294 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.294 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.294 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.294 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.294 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.294 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.294 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.294 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.294 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.295 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-08-28T19:43:51.295 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-08-28T19:43:51.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.295 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.295 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.295 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.295 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.296 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.296 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-08-28T19:43:51.296 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-08-28T19:43:51.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.296 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.296 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.296 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.297 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.297 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.297 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.297 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.297 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.297 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.298 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.298 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.298 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.298 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.298 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.298 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.298 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.298 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.299 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.299 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.299 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.299 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.299 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.300 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.300 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.300 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.300 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.300 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.300 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.301 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.301 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.301 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.301 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.302 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.302 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.302 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.302 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.302 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.302 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.302 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.302 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.303 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.303 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.303 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.303 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.304 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.304 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.304 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.304 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.304 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.304 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.304 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.305 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.305 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-08-28T19:43:51.305 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-08-28T19:43:51.305 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.305 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.305 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.305 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.306 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.306 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.306 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.306 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.306 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.306 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.306 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-08-28T19:43:51.306 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-08-28T19:43:51.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.307 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.307 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.307 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-08-28T19:43:51.307 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.307 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.308 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.308 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.308 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-08-28T19:43:51.308 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-08-28T19:43:51.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:51.308 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.309 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-08-28T19:43:51.309 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.309 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.309 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.309 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.309 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.309 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.310 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-08-28T19:43:51.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-08-28T19:43:51.310 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.310 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.310 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.310 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-08-28T19:43:51.310 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.310 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.311 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.311 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-08-28T19:43:51.311 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.311 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.311 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.311 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-08-28T19:43:51.311 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-08-28T19:43:51.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.312 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.312 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.312 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-08-28T19:43:51.312 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.312 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.312 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.312 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-08-28T19:43:51.312 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.313 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.313 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.313 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.313 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.313 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.314 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.314 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.314 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.314 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.314 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.314 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.314 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-08-28T19:43:51.315 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-08-28T19:43:51.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.315 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.315 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-08-28T19:43:51.315 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.315 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.315 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.315 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.316 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.316 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.316 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.316 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-08-28T19:43:51.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-08-28T19:43:51.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.316 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-08-28T19:43:51.317 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.317 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.317 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-08-28T19:43:51.317 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-08-28T19:43:51.317 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.317 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.318 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.318 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.318 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.318 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-08-28T19:43:51.318 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-08-28T19:43:51.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.318 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.319 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-08-28T19:43:51.319 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-08-28T19:43:51.319 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-08-28T19:43:51.319 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.319 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.319 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.319 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.320 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.320 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-08-28T19:43:51.320 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.320 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.321 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.321 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.321 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.321 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.321 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.321 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.322 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.322 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.322 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.322 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.322 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.322 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.322 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.322 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.323 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.323 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.323 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.323 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.323 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.323 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.324 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.324 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.324 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.324 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.324 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.324 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.325 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.325 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.325 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.325 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.325 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.325 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.325 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.326 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.326 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.326 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.326 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.326 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.326 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.326 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.326 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.326 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.327 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.327 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.327 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.327 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.327 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.328 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.328 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.328 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.328 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.328 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.329 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.329 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.329 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.329 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.329 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-08-28T19:43:51.329 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-08-28T19:43:51.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.329 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.330 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-08-28T19:43:51.330 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.330 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.330 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.330 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.330 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.331 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.331 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-08-28T19:43:51.331 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-08-28T19:43:51.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.331 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.331 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.331 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.332 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.332 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.332 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.332 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.332 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.332 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.333 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.333 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.333 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.333 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.333 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.333 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.333 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.333 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.334 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.334 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.334 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-08-28T19:43:51.334 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-08-28T19:43:51.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.335 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.335 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.335 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.335 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.335 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.335 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.335 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.336 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.336 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.336 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.336 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-08-28T19:43:51.336 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.336 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.337 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.337 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-08-28T19:43:51.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-08-28T19:43:51.337 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.337 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.337 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.337 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.337 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.337 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.338 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.338 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.338 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.338 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.338 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-08-28T19:43:51.338 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-08-28T19:43:51.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.339 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.339 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.339 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.339 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.339 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.339 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.340 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.340 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.340 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.341 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.341 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.341 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.341 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.341 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.341 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.341 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.341 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.342 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.342 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.342 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.342 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.342 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.342 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.342 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.342 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.343 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.343 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.343 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.343 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.343 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.343 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.343 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.344 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.344 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.344 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.344 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.344 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.344 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.344 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.345 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.345 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.345 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.345 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.345 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.345 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.345 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.346 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.346 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.346 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.346 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.346 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.346 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.346 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.346 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.347 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.347 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.347 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.347 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.347 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.347 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.347 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.347 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.347 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.348 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.348 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.348 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.348 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.348 INFO:teuthology.orchestra.run.smithi016.stdout: "max_requests": { 2024-08-28T19:43:51.348 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_requests", 2024-08-28T19:43:51.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.349 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.349 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.349 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-08-28T19:43:51.349 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.349 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.349 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.349 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-08-28T19:43:51.349 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.350 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.350 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-08-28T19:43:51.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-08-28T19:43:51.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.350 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.350 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.351 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.351 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.351 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.351 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.351 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.351 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.351 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.352 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-08-28T19:43:51.352 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-08-28T19:43:51.352 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.352 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.352 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.352 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.352 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.353 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.353 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.353 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.353 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.353 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.353 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.353 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.354 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.354 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.354 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.354 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.354 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.354 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.354 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.354 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.354 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-08-28T19:43:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.356 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.356 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.356 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.356 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.356 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.356 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.356 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.356 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.357 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.357 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.357 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.357 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.357 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.357 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.357 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.357 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.358 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.358 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.358 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.358 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.358 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.358 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.358 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.358 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.359 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.359 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.359 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.359 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.359 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.359 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.359 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.359 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.360 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.360 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.360 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.360 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.360 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.360 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.360 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.361 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.361 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.361 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.361 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.361 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.361 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.361 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.361 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.361 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.362 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.362 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.362 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.362 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.362 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.362 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.362 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.362 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.363 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.363 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.363 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.363 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.363 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.363 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.363 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.364 INFO:teuthology.orchestra.run.smithi016.stdout: "secondary_zone_period_retry_limit": { 2024-08-28T19:43:51.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secondary_zone_period_retry_limit", 2024-08-28T19:43:51.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.364 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.364 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.364 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-08-28T19:43:51.364 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.364 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.365 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.365 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-08-28T19:43:51.365 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.365 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.365 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.365 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.365 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.366 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.366 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.366 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.366 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.366 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.366 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.366 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.367 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.367 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.367 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.367 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-08-28T19:43:51.367 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.367 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.368 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.368 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.368 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.368 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.368 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.368 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.368 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.369 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.369 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.369 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.369 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.369 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.369 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.369 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.369 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.369 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.370 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.370 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.370 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.370 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.370 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.370 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.371 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.371 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.371 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.371 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.371 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.371 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.371 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.371 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.371 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.372 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.372 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.372 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.372 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.372 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.372 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.372 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.372 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.373 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.373 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.373 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.373 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.373 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.373 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.373 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.373 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.374 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.374 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.374 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.374 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.374 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.374 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.374 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.375 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.375 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.375 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.375 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.375 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.375 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.375 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.375 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.376 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.376 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.376 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.376 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.376 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.376 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.377 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.377 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.377 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.377 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.377 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.377 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.377 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.377 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.378 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-08-28T19:43:51.378 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-08-28T19:43:51.378 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.378 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.378 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.378 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-08-28T19:43:51.378 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.378 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.379 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.379 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-08-28T19:43:51.379 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.379 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.379 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.379 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.379 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.380 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-08-28T19:43:51.380 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.380 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.380 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.380 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.380 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.380 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.380 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.381 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.381 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.381 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.381 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.381 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.381 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.381 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.381 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.382 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.382 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.382 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.382 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.382 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.382 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.382 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.383 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.383 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.383 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.383 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.383 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.383 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.383 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.383 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.384 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.384 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.384 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.384 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.384 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.384 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.385 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.385 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.385 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.385 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.385 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.385 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.385 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.385 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.386 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.386 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.386 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.386 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.386 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.386 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.386 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.386 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.386 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.386 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.387 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.387 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.387 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.387 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.387 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.387 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.387 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.388 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.388 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.388 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.388 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.388 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.388 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-08-28T19:43:51.388 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-08-28T19:43:51.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.389 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.389 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.389 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.389 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.389 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.389 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.390 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.390 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.390 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.390 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-08-28T19:43:51.390 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-08-28T19:43:51.390 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.391 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.391 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-08-28T19:43:51.391 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.391 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.391 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.391 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.391 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.392 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.392 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-08-28T19:43:51.392 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-08-28T19:43:51.392 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.392 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.392 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.393 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.393 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.393 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.393 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.393 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.393 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.393 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.393 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.394 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-08-28T19:43:51.394 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-08-28T19:43:51.394 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.394 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.394 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.394 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.394 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.394 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.395 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.395 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.395 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.395 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.395 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-08-28T19:43:51.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-08-28T19:43:51.396 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-08-28T19:43:51.396 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.396 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.396 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.396 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.396 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.396 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.396 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.397 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.397 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.397 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-08-28T19:43:51.397 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-08-28T19:43:51.397 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.397 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.398 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.398 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.398 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.398 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.398 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.398 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.398 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.399 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-08-28T19:43:51.399 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-08-28T19:43:51.399 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.399 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.399 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.399 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.399 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.400 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.400 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.400 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.400 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.400 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.400 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.400 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-08-28T19:43:51.401 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-08-28T19:43:51.401 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.401 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.401 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.401 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.401 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.401 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.402 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.402 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.402 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.402 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-08-28T19:43:51.402 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-08-28T19:43:51.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.402 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.403 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-08-28T19:43:51.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-08-28T19:43:51.403 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.403 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.403 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.403 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.403 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.404 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.404 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.404 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.404 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.404 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.405 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.405 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.405 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.405 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.405 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-08-28T19:43:51.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-08-28T19:43:51.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.406 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.406 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.406 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.406 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.406 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.406 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.407 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.407 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.407 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.407 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-08-28T19:43:51.407 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-08-28T19:43:51.407 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.407 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.408 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.408 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.408 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.408 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.408 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.408 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-08-28T19:43:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-08-28T19:43:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.410 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.410 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.410 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.410 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.410 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.410 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.410 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smb", 2024-08-28T19:43:51.411 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.411 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.411 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.412 INFO:teuthology.orchestra.run.smithi016.stdout: "internal_store_backend": { 2024-08-28T19:43:51.412 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "internal_store_backend", 2024-08-28T19:43:51.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.412 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.412 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.412 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.412 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.413 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.413 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-08-28T19:43:51.413 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.413 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.413 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.413 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.413 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.414 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.414 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.414 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.414 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.414 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.414 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.414 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.414 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.414 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.415 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.415 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.415 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.415 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.415 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.415 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.415 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.415 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.416 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.416 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.416 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.416 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.417 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.417 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.417 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.417 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.417 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.418 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.418 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.418 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.418 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.419 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.419 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.419 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.419 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.419 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.419 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.419 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.419 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.420 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.420 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.420 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.420 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.420 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.421 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.421 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.421 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.421 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.421 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.421 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.421 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.422 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.422 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.422 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.422 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.422 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.422 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.423 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.423 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.423 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.423 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.423 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.423 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.423 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.423 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.424 INFO:teuthology.orchestra.run.smithi016.stdout: "update_orchestration": { 2024-08-28T19:43:51.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "update_orchestration", 2024-08-28T19:43:51.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.424 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.424 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.424 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.424 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.425 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.425 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-08-28T19:43:51.425 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.425 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.425 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.426 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-08-28T19:43:51.426 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.426 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.426 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.426 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-08-28T19:43:51.426 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-08-28T19:43:51.426 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.426 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.426 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.427 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.427 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.427 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.427 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.427 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-08-28T19:43:51.427 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.427 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.427 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.428 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-08-28T19:43:51.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-08-28T19:43:51.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.428 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.428 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.428 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.428 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.428 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.429 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-08-28T19:43:51.429 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.429 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.429 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.429 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.429 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.429 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.429 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.430 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.430 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.430 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.430 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.430 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.430 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.430 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.430 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.431 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.431 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.431 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.431 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.431 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.431 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.431 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.432 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.432 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.432 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.432 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.432 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.432 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.432 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.432 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.432 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.433 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.433 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.433 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.433 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.433 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.433 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.433 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.434 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.434 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.434 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.434 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.434 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.434 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.434 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.434 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.434 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.435 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.435 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.435 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.435 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.435 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.435 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.435 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.436 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.436 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.436 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.436 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.436 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.436 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.436 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.436 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.437 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.437 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.437 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.437 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.437 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.437 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.438 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.438 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.438 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.438 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.438 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.438 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.438 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.439 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-08-28T19:43:51.439 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.439 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.439 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.439 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.439 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.440 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.440 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.440 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.440 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.440 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.440 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.440 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.441 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.441 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.441 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.441 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.441 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.441 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.441 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.441 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.441 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.442 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.442 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.442 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.442 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.442 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.442 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.443 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.443 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.443 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.443 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.443 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.443 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.443 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.443 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.444 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.444 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.444 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.444 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.444 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.444 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.444 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.444 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.444 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.445 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.445 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.445 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.445 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.445 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.445 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.445 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.445 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.446 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.446 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.446 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.446 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.446 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.446 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.446 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.446 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.446 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.447 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.447 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.447 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.447 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.447 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.447 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.448 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.448 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.448 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.448 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.448 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.448 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.448 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.448 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.449 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.449 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.449 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-08-28T19:43:51.449 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.449 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.449 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.449 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.450 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.450 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.450 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.450 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.450 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.450 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.450 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.450 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.451 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.451 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.451 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.451 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.451 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.451 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.451 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.451 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.451 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.452 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.452 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.452 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.452 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.452 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.452 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.452 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.453 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.453 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.453 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.453 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.453 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.453 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.453 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.453 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.454 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.454 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.454 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.454 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.454 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.454 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.454 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.454 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.455 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.455 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.455 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.455 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.455 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.455 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.455 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.455 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.455 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.455 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.456 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.456 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.456 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.456 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.456 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.457 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.457 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.457 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.457 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.457 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.457 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.457 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.458 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.458 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.458 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.458 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.458 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.458 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.458 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.458 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.459 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.459 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-08-28T19:43:51.459 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.459 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.459 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.460 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-08-28T19:43:51.460 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-08-28T19:43:51.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.460 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.460 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-08-28T19:43:51.460 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.460 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.461 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.461 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.461 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.461 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.461 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:51.461 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:51.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:51.462 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.462 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.462 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-08-28T19:43:51.462 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.462 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.462 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.462 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.462 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.463 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.463 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.463 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.463 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.463 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.463 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.464 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.464 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.464 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.464 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.464 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.464 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.464 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.464 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.464 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.465 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.465 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.465 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.465 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.465 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.465 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.465 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.466 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.466 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.466 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.466 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.466 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.466 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.466 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.466 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.466 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.467 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.467 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.467 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.467 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.467 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.467 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.467 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.468 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.468 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.468 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.468 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.468 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.468 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.468 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.468 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.469 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.469 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.469 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.469 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.469 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.469 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.469 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.470 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.470 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.470 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.470 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.470 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.470 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.470 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.470 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.471 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.471 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.471 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.471 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.471 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.472 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.472 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.472 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.472 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.472 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.473 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-08-28T19:43:51.473 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.473 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.473 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.473 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-08-28T19:43:51.473 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-08-28T19:43:51.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.474 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.474 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.474 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.474 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.474 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.474 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-08-28T19:43:51.474 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.474 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.475 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.475 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-08-28T19:43:51.475 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-08-28T19:43:51.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.475 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.475 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.475 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.475 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.476 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.476 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.476 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-08-28T19:43:51.476 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.476 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.476 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.476 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-08-28T19:43:51.476 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-08-28T19:43:51.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.477 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.477 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.477 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.477 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.477 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.477 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.477 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-08-28T19:43:51.477 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.478 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.478 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-08-28T19:43:51.478 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-08-28T19:43:51.478 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.478 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.478 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.478 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.478 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.479 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-08-28T19:43:51.479 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.479 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.479 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.479 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-08-28T19:43:51.479 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-08-28T19:43:51.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.480 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.480 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.480 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.480 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.480 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.480 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.480 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-08-28T19:43:51.481 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.481 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.481 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.481 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-08-28T19:43:51.481 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-08-28T19:43:51.481 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.481 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.481 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.482 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.482 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.482 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.482 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.482 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.482 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.482 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.483 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-08-28T19:43:51.483 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-08-28T19:43:51.483 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.483 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.483 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.483 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.484 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.484 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.484 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.484 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-08-28T19:43:51.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-08-28T19:43:51.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.484 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.484 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.485 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-08-28T19:43:51.485 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.485 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.485 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.485 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.485 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.485 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.486 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-08-28T19:43:51.486 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-08-28T19:43:51.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.486 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.486 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.486 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.486 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.486 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.487 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.487 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.487 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.487 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:51.487 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:51.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.487 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.487 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.488 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-08-28T19:43:51.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-08-28T19:43:51.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.488 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.488 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.488 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.488 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.488 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.489 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-08-28T19:43:51.489 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-08-28T19:43:51.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.489 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.489 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.489 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-08-28T19:43:51.489 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.489 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-08-28T19:43:51.491 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-08-28T19:43:51.491 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.491 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.491 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.491 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.491 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.491 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.491 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.492 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.492 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.492 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.492 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.492 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-08-28T19:43:51.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-08-28T19:43:51.492 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.493 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.493 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.493 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.493 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.493 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.493 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.493 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.494 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.494 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.494 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.494 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.494 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.494 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.494 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.494 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.495 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.495 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.495 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.495 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.495 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.495 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.495 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.495 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.495 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.496 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.496 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.496 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.496 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.496 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.496 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.497 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.497 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.497 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.497 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.497 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.497 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.498 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.498 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.498 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.498 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.498 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.498 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.499 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.499 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.499 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.499 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.499 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.499 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.499 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.499 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.499 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.500 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.500 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.500 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.500 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.500 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.500 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.500 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.501 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.501 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.501 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.501 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.501 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.501 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.501 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.501 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.502 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.502 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.502 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-08-28T19:43:51.502 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-08-28T19:43:51.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.502 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.502 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.503 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.503 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.503 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.503 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.503 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.503 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.503 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.504 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-08-28T19:43:51.504 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-08-28T19:43:51.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.504 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.504 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.504 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.504 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.504 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.504 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.505 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.505 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.505 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.505 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.505 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.505 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.505 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.505 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.506 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.506 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.506 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.506 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.506 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.506 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.506 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.506 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.507 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-08-28T19:43:51.507 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-08-28T19:43:51.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.507 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.507 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.507 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-08-28T19:43:51.507 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.507 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.508 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.508 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.508 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.508 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-08-28T19:43:51.509 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.509 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.509 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.509 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.509 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.509 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.510 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.510 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.510 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.510 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.510 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.510 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.510 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.510 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.511 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.511 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.511 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.511 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.511 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.512 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.512 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.512 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.513 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.513 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.513 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.513 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.513 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.514 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.514 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.514 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.514 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.514 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.515 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.515 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.515 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.515 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.515 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.515 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.515 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.515 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.515 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.516 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.516 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.516 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.516 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.516 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.516 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.516 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.516 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.517 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.517 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.517 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.517 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.517 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.518 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.518 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.518 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.518 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.518 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.518 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.518 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.518 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.519 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.519 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.519 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.519 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.519 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.519 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.519 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.520 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.520 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.520 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.520 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.520 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.520 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.520 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.520 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.521 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-08-28T19:43:51.521 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.521 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.521 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.521 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.521 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.521 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.522 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.522 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.522 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.522 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.522 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.522 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.522 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.523 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.523 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.523 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.523 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.523 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.523 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.523 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.524 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.524 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.524 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.524 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.524 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.524 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.525 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.525 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.525 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.525 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.525 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.525 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.525 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.525 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.526 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.526 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.526 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.526 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.526 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.526 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.526 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.526 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.526 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.527 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.527 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.527 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.527 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.527 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.527 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.527 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.527 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.528 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.528 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.528 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.528 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.528 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.528 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.528 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.529 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.529 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.529 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.529 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.529 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.529 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.529 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.530 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-08-28T19:43:51.530 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-08-28T19:43:51.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.530 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.530 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.530 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-08-28T19:43:51.530 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.530 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.530 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.531 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-08-28T19:43:51.531 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.531 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.531 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.531 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-08-28T19:43:51.531 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-08-28T19:43:51.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-08-28T19:43:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-08-28T19:43:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-08-28T19:43:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-08-28T19:43:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-08-28T19:43:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-08-28T19:43:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-08-28T19:43:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-08-28T19:43:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-08-28T19:43:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-08-28T19:43:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-08-28T19:43:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-08-28T19:43:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-08-28T19:43:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-08-28T19:43:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-08-28T19:43:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-08-28T19:43:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-08-28T19:43:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-08-28T19:43:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-08-28T19:43:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-08-28T19:43:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-08-28T19:43:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-08-28T19:43:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-08-28T19:43:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-08-28T19:43:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-08-28T19:43:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-08-28T19:43:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-08-28T19:43:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-08-28T19:43:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-08-28T19:43:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-08-28T19:43:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-08-28T19:43:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-08-28T19:43:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-08-28T19:43:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-08-28T19:43:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-08-28T19:43:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-08-28T19:43:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-08-28T19:43:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-08-28T19:43:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-08-28T19:43:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-08-28T19:43:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-08-28T19:43:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-08-28T19:43:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-08-28T19:43:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-08-28T19:43:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-08-28T19:43:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-08-28T19:43:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-08-28T19:43:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-08-28T19:43:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-08-28T19:43:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-08-28T19:43:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-08-28T19:43:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-08-28T19:43:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-08-28T19:43:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-08-28T19:43:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-08-28T19:43:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-08-28T19:43:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-08-28T19:43:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-08-28T19:43:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-08-28T19:43:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-08-28T19:43:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-08-28T19:43:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-08-28T19:43:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-08-28T19:43:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-08-28T19:43:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-08-28T19:43:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: "services": { 2024-08-28T19:43:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard": "https://172.21.15.16:8443/" 2024-08-28T19:43:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: "always_on_modules": { 2024-08-28T19:43:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: "octopus": [ 2024-08-28T19:43:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-08-28T19:43:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-08-28T19:43:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-08-28T19:43:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-08-28T19:43:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-08-28T19:43:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-08-28T19:43:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-08-28T19:43:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-08-28T19:43:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-08-28T19:43:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-08-28T19:43:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "pacific": [ 2024-08-28T19:43:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-08-28T19:43:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-08-28T19:43:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-08-28T19:43:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-08-28T19:43:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-08-28T19:43:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-08-28T19:43:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-08-28T19:43:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-08-28T19:43:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-08-28T19:43:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-08-28T19:43:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: "quincy": [ 2024-08-28T19:43:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-08-28T19:43:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-08-28T19:43:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-08-28T19:43:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-08-28T19:43:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-08-28T19:43:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-08-28T19:43:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-08-28T19:43:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-08-28T19:43:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-08-28T19:43:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-08-28T19:43:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "reef": [ 2024-08-28T19:43:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-08-28T19:43:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-08-28T19:43:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-08-28T19:43:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-08-28T19:43:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-08-28T19:43:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-08-28T19:43:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-08-28T19:43:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-08-28T19:43:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-08-28T19:43:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-08-28T19:43:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-08-28T19:43:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "squid": [ 2024-08-28T19:43:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-08-28T19:43:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-08-28T19:43:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-08-28T19:43:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-08-28T19:43:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-08-28T19:43:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-08-28T19:43:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-08-28T19:43:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-08-28T19:43:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-08-28T19:43:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-08-28T19:43:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:43:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "last_failure_osd_epoch": 5, 2024-08-28T19:43:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "active_clients": [ 2024-08-28T19:43:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-08-28T19:43:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-08-28T19:43:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-08-28T19:43:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-08-28T19:43:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3997760726 2024-08-28T19:43:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:43:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "libcephsqlite", 2024-08-28T19:43:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-08-28T19:43:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-08-28T19:43:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-08-28T19:43:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2612310139 2024-08-28T19:43:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:43:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-08-28T19:43:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-08-28T19:43:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-08-28T19:43:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-08-28T19:43:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2754080649 2024-08-28T19:43:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:43:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-08-28T19:43:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-08-28T19:43:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-08-28T19:43:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-08-28T19:43:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1386478247 2024-08-28T19:43:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:43:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-08-28T19:43:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-08-28T19:43:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-08-28T19:43:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-08-28T19:43:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 126066066 2024-08-28T19:43:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:43:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-08-28T19:43:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-08-28T19:43:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-08-28T19:43:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-08-28T19:43:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-08-28T19:43:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3246555201 2024-08-28T19:43:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:43:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-08-28T19:43:51.574 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:43:51.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 -- 172.21.15.16:0/1164816975 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8f240b4ca0 msgr2=0x7f8f240b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:51.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8f240b4ca0 0x7f8f240b71b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f8f28005f70 tx=0x7f8f28000bc0 comp rx=0 tx=0).stop 2024-08-28T19:43:51.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 -- 172.21.15.16:0/1164816975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f38109ab0 msgr2=0x7f8f3819f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:43:51.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f38109ab0 0x7f8f3819f9d0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f8f3000cce0 tx=0x7f8f3000a620 comp rx=0 tx=0).stop 2024-08-28T19:43:51.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 -- 172.21.15.16:0/1164816975 shutdown_connections 2024-08-28T19:43:51.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f38109ab0 0x7f8f3819f9d0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:51.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8f240b4ca0 0x7f8f240b71b0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:51.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f38109160 0x7f8f3819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:51.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 --2- 172.21.15.16:0/1164816975 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f38103830 0x7f8f3819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:43:51.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 -- 172.21.15.16:0/1164816975 >> 172.21.15.16:0/1164816975 conn(0x7f8f380ff270 msgr2=0x7f8f38101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:43:51.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 -- 172.21.15.16:0/1164816975 shutdown_connections 2024-08-28T19:43:51.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:43:48.851+0000 7f8f3e051640 1 -- 172.21.15.16:0/1164816975 wait complete. 2024-08-28T19:43:51.583 DEBUG:tasks.check_counter:Getting stats from a 2024-08-28T19:43:51.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.a perf dump 2024-08-28T19:43:51.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:49 smithi016 ceph-mon[18556]: pgmap v1054: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 88 KiB/s wr, 17 op/s 2024-08-28T19:43:51.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:49 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/114599279' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-08-28T19:43:51.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:49 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1701129580' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-08-28T19:43:51.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:49 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1164816975' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-08-28T19:43:51.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:51 smithi016 ceph-mon[18556]: pgmap v1055: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 81 KiB/s wr, 17 op/s 2024-08-28T19:43:51.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:49 smithi115 ceph-mon[22184]: pgmap v1054: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 88 KiB/s wr, 17 op/s 2024-08-28T19:43:51.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:49 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/114599279' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-08-28T19:43:51.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:49 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1701129580' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-08-28T19:43:51.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:49 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1164816975' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-08-28T19:43:51.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:51 smithi115 ceph-mon[22184]: pgmap v1055: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 81 KiB/s wr, 17 op/s 2024-08-28T19:43:51.587 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:49 smithi028 ceph-mon[21769]: pgmap v1054: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 88 KiB/s wr, 17 op/s 2024-08-28T19:43:51.587 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:49 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/114599279' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-08-28T19:43:51.587 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:49 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1701129580' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-08-28T19:43:51.587 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:49 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1164816975' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-08-28T19:43:51.587 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:51 smithi028 ceph-mon[21769]: pgmap v1055: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 81 KiB/s wr, 17 op/s 2024-08-28T19:43:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:52.347 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:43:52.347 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:43:52.348 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 3147, 2024-08-28T19:43:52.348 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 3149, 2024-08-28T19:43:52.348 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 142002944, 2024-08-28T19:43:52.348 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1000747, 2024-08-28T19:43:52.348 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 8, 2024-08-28T19:43:52.348 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 4, 2024-08-28T19:43:52.348 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.441418846, 2024-08-28T19:43:52.348 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.146844937, 2024-08-28T19:43:52.349 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.174564005, 2024-08-28T19:43:52.349 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.095363367, 2024-08-28T19:43:52.349 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:52.349 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3147, 2024-08-28T19:43:52.349 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.115258000, 2024-08-28T19:43:52.349 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000036624 2024-08-28T19:43:52.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.349 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:52.350 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.350 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.350 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.350 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-28T19:43:52.350 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-08-28T19:43:52.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.350 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:43:52.351 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1829, 2024-08-28T19:43:52.351 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 1743, 2024-08-28T19:43:52.351 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 127636706, 2024-08-28T19:43:52.351 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 670786, 2024-08-28T19:43:52.351 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 6, 2024-08-28T19:43:52.351 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 5, 2024-08-28T19:43:52.351 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.283543566, 2024-08-28T19:43:52.351 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.087275014, 2024-08-28T19:43:52.352 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.159720133, 2024-08-28T19:43:52.352 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.036935271, 2024-08-28T19:43:52.352 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:52.352 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1742, 2024-08-28T19:43:52.352 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.065279523, 2024-08-28T19:43:52.352 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000037473 2024-08-28T19:43:52.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.352 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:52.353 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.353 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.353 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.353 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1478448, 2024-08-28T19:43:52.353 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 258272 2024-08-28T19:43:52.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.354 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:43:52.354 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 845, 2024-08-28T19:43:52.354 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 1152, 2024-08-28T19:43:52.354 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 100885612, 2024-08-28T19:43:52.354 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 989218, 2024-08-28T19:43:52.354 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 8, 2024-08-28T19:43:52.354 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 5, 2024-08-28T19:43:52.354 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.178366972, 2024-08-28T19:43:52.355 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.059460500, 2024-08-28T19:43:52.355 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.085797909, 2024-08-28T19:43:52.355 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.024341581, 2024-08-28T19:43:52.355 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:52.355 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1151, 2024-08-28T19:43:52.355 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.054208085, 2024-08-28T19:43:52.355 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000047096 2024-08-28T19:43:52.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.355 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:52.356 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.356 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.356 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.356 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-08-28T19:43:52.356 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 722416 2024-08-28T19:43:52.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.357 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-08-28T19:43:52.357 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 1, 2024-08-28T19:43:52.357 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-08-28T19:43:52.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.357 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-MDSRank": { 2024-08-28T19:43:52.357 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-08-28T19:43:52.357 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-08-28T19:43:52.357 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 8776, 2024-08-28T19:43:52.357 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.235399888, 2024-08-28T19:43:52.358 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000026823 2024-08-28T19:43:52.358 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:52.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.358 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-PurgeQueue": { 2024-08-28T19:43:52.358 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-08-28T19:43:52.358 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-08-28T19:43:52.358 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.358 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.359 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:52.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.359 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-08-28T19:43:52.359 INFO:teuthology.orchestra.run.smithi016.stdout: "request": 0, 2024-08-28T19:43:52.359 INFO:teuthology.orchestra.run.smithi016.stdout: "reply": 0, 2024-08-28T19:43:52.359 INFO:teuthology.orchestra.run.smithi016.stdout: "reply_latency": { 2024-08-28T19:43:52.359 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.360 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.360 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.360 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_reply": 0, 2024-08-28T19:43:52.360 INFO:teuthology.orchestra.run.smithi016.stdout: "forward": 0, 2024-08-28T19:43:52.360 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_complete": 0, 2024-08-28T19:43:52.360 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_keys": 0, 2024-08-28T19:43:52.361 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_commit": 0, 2024-08-28T19:43:52.361 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_split": 0, 2024-08-28T19:43:52.361 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_merge": 0, 2024-08-28T19:43:52.361 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes": 16319, 2024-08-28T19:43:52.361 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_top": 46, 2024-08-28T19:43:52.361 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_bottom": 16273, 2024-08-28T19:43:52.361 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pin_tail": 0, 2024-08-28T19:43:52.361 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pinned": 16254, 2024-08-28T19:43:52.361 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_expired": 0, 2024-08-28T19:43:52.362 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_with_caps": 0, 2024-08-28T19:43:52.362 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": 0, 2024-08-28T19:43:52.362 INFO:teuthology.orchestra.run.smithi016.stdout: "subtrees": 12, 2024-08-28T19:43:52.362 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse": 0, 2024-08-28T19:43:52.362 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_hit": 0, 2024-08-28T19:43:52.362 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_forward": 0, 2024-08-28T19:43:52.362 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_discover": 0, 2024-08-28T19:43:52.362 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_dir_fetch": 0, 2024-08-28T19:43:52.362 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_remote_ino": 0, 2024-08-28T19:43:52.363 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_lock": 0, 2024-08-28T19:43:52.363 INFO:teuthology.orchestra.run.smithi016.stdout: "load_cent": 0, 2024-08-28T19:43:52.364 INFO:teuthology.orchestra.run.smithi016.stdout: "q": 0, 2024-08-28T19:43:52.364 INFO:teuthology.orchestra.run.smithi016.stdout: "exported": 0, 2024-08-28T19:43:52.364 INFO:teuthology.orchestra.run.smithi016.stdout: "exported_inodes": 0, 2024-08-28T19:43:52.364 INFO:teuthology.orchestra.run.smithi016.stdout: "imported": 0, 2024-08-28T19:43:52.364 INFO:teuthology.orchestra.run.smithi016.stdout: "imported_inodes": 0, 2024-08-28T19:43:52.364 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_dir_fetch": 0, 2024-08-28T19:43:52.364 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_backtrace_fetch": 0, 2024-08-28T19:43:52.364 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_peer_discover": 0, 2024-08-28T19:43:52.365 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rfiles": 0, 2024-08-28T19:43:52.365 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rbytes": 0, 2024-08-28T19:43:52.365 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rsnaps": 0, 2024-08-28T19:43:52.365 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_fetch": 0, 2024-08-28T19:43:52.365 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_set_tag": 0, 2024-08-28T19:43:52.365 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_repaired": 0, 2024-08-28T19:43:52.365 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_inotable_repaired": 0, 2024-08-28T19:43:52.365 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_inodes": 0, 2024-08-28T19:43:52.366 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_base_inodes": 0, 2024-08-28T19:43:52.366 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dirfrag_rstats": 0, 2024-08-28T19:43:52.366 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_file_inodes": 0, 2024-08-28T19:43:52.366 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_inode_file_caps": 0, 2024-08-28T19:43:52.366 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_revoke": 0, 2024-08-28T19:43:52.366 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_grant": 0, 2024-08-28T19:43:52.366 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_trunc": 0, 2024-08-28T19:43:52.366 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-28T19:43:52.366 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-28T19:43:52.367 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps": 0, 2024-08-28T19:43:52.367 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps_dirty": 0, 2024-08-28T19:43:52.367 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_cap_release": 0, 2024-08-28T19:43:52.367 INFO:teuthology.orchestra.run.smithi016.stdout: "process_request_cap_release": 0 2024-08-28T19:43:52.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.367 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache": { 2024-08-28T19:43:52.367 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update": 0, 2024-08-28T19:43:52.367 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update_receipt": 0, 2024-08-28T19:43:52.367 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_try_discover": 0, 2024-08-28T19:43:52.368 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_send_discover": 0, 2024-08-28T19:43:52.368 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_handle_discover": 0, 2024-08-28T19:43:52.368 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays": 0, 2024-08-28T19:43:52.368 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_delayed": 0, 2024-08-28T19:43:52.368 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_enqueuing": 0, 2024-08-28T19:43:52.368 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_created": 0, 2024-08-28T19:43:52.368 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_enqueued": 0, 2024-08-28T19:43:52.369 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_reintegrated": 0, 2024-08-28T19:43:52.369 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_migrated": 0, 2024-08-28T19:43:52.369 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_processing": 0, 2024-08-28T19:43:52.369 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_enqueued": 0, 2024-08-28T19:43:52.369 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_prioritized": 0, 2024-08-28T19:43:52.369 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_started": 0, 2024-08-28T19:43:52.369 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_completed": 0, 2024-08-28T19:43:52.369 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_path": 0, 2024-08-28T19:43:52.369 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_inode": 0, 2024-08-28T19:43:52.370 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_enqueue_scrub": 0, 2024-08-28T19:43:52.370 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_exportdir": 0, 2024-08-28T19:43:52.370 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_flush": 0, 2024-08-28T19:43:52.370 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragmentdir": 0, 2024-08-28T19:43:52.370 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragstats": 0, 2024-08-28T19:43:52.370 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_inodestats": 0, 2024-08-28T19:43:52.370 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:52.370 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_succeeded": 0, 2024-08-28T19:43:52.371 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_write_failed": 0 2024-08-28T19:43:52.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.371 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log": { 2024-08-28T19:43:52.371 INFO:teuthology.orchestra.run.smithi016.stdout: "evlrg": 0, 2024-08-28T19:43:52.371 INFO:teuthology.orchestra.run.smithi016.stdout: "evadd": 0, 2024-08-28T19:43:52.371 INFO:teuthology.orchestra.run.smithi016.stdout: "evex": 0, 2024-08-28T19:43:52.371 INFO:teuthology.orchestra.run.smithi016.stdout: "evtrm": 0, 2024-08-28T19:43:52.372 INFO:teuthology.orchestra.run.smithi016.stdout: "ev": 79384, 2024-08-28T19:43:52.372 INFO:teuthology.orchestra.run.smithi016.stdout: "evexg": 0, 2024-08-28T19:43:52.372 INFO:teuthology.orchestra.run.smithi016.stdout: "evexd": 0, 2024-08-28T19:43:52.372 INFO:teuthology.orchestra.run.smithi016.stdout: "segadd": 0, 2024-08-28T19:43:52.372 INFO:teuthology.orchestra.run.smithi016.stdout: "segex": 0, 2024-08-28T19:43:52.372 INFO:teuthology.orchestra.run.smithi016.stdout: "segtrm": 0, 2024-08-28T19:43:52.372 INFO:teuthology.orchestra.run.smithi016.stdout: "seg": 115, 2024-08-28T19:43:52.372 INFO:teuthology.orchestra.run.smithi016.stdout: "segmjr": 8, 2024-08-28T19:43:52.372 INFO:teuthology.orchestra.run.smithi016.stdout: "segexg": 0, 2024-08-28T19:43:52.373 INFO:teuthology.orchestra.run.smithi016.stdout: "segexd": 0, 2024-08-28T19:43:52.373 INFO:teuthology.orchestra.run.smithi016.stdout: "expos": 4194304, 2024-08-28T19:43:52.373 INFO:teuthology.orchestra.run.smithi016.stdout: "wrpos": 371732242, 2024-08-28T19:43:52.373 INFO:teuthology.orchestra.run.smithi016.stdout: "rdpos": 371724367, 2024-08-28T19:43:52.373 INFO:teuthology.orchestra.run.smithi016.stdout: "jlat": { 2024-08-28T19:43:52.373 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.373 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.373 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.374 INFO:teuthology.orchestra.run.smithi016.stdout: "replayed": 79384 2024-08-28T19:43:52.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.374 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mem": { 2024-08-28T19:43:52.374 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1178, 2024-08-28T19:43:52.374 INFO:teuthology.orchestra.run.smithi016.stdout: "ino+": 16434, 2024-08-28T19:43:52.374 INFO:teuthology.orchestra.run.smithi016.stdout: "ino-": 15256, 2024-08-28T19:43:52.374 INFO:teuthology.orchestra.run.smithi016.stdout: "dir": 75, 2024-08-28T19:43:52.375 INFO:teuthology.orchestra.run.smithi016.stdout: "dir+": 92, 2024-08-28T19:43:52.375 INFO:teuthology.orchestra.run.smithi016.stdout: "dir-": 17, 2024-08-28T19:43:52.375 INFO:teuthology.orchestra.run.smithi016.stdout: "dn": 16319, 2024-08-28T19:43:52.375 INFO:teuthology.orchestra.run.smithi016.stdout: "dn+": 16637, 2024-08-28T19:43:52.375 INFO:teuthology.orchestra.run.smithi016.stdout: "dn-": 318, 2024-08-28T19:43:52.375 INFO:teuthology.orchestra.run.smithi016.stdout: "cap": 0, 2024-08-28T19:43:52.375 INFO:teuthology.orchestra.run.smithi016.stdout: "cap+": 0, 2024-08-28T19:43:52.375 INFO:teuthology.orchestra.run.smithi016.stdout: "cap-": 0, 2024-08-28T19:43:52.376 INFO:teuthology.orchestra.run.smithi016.stdout: "rss": 139068, 2024-08-28T19:43:52.376 INFO:teuthology.orchestra.run.smithi016.stdout: "heap": 349241344 2024-08-28T19:43:52.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.376 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_server": { 2024-08-28T19:43:52.376 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_client_request": 0, 2024-08-28T19:43:52.376 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_server_request": 0, 2024-08-28T19:43:52.376 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_request": 0, 2024-08-28T19:43:52.376 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_session": 0, 2024-08-28T19:43:52.377 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_peer_request": 0, 2024-08-28T19:43:52.377 INFO:teuthology.orchestra.run.smithi016.stdout: "req_create_latency": { 2024-08-28T19:43:52.377 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.377 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.377 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.377 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getattr_latency": { 2024-08-28T19:43:52.377 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.378 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.378 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.378 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getfilelock_latency": { 2024-08-28T19:43:52.378 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.378 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.378 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.379 INFO:teuthology.orchestra.run.smithi016.stdout: "req_link_latency": { 2024-08-28T19:43:52.379 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.379 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.379 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.379 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookup_latency": { 2024-08-28T19:43:52.379 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.380 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.380 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.380 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookuphash_latency": { 2024-08-28T19:43:52.380 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.380 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.380 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.381 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupino_latency": { 2024-08-28T19:43:52.381 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.381 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.381 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.381 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupname_latency": { 2024-08-28T19:43:52.381 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.381 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.382 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.382 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupparent_latency": { 2024-08-28T19:43:52.382 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.382 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.382 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.382 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupsnap_latency": { 2024-08-28T19:43:52.382 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.383 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.383 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.383 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lssnap_latency": { 2024-08-28T19:43:52.383 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.384 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.384 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.384 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mkdir_latency": { 2024-08-28T19:43:52.384 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.384 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.384 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.385 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mknod_latency": { 2024-08-28T19:43:52.385 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.385 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.385 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.385 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mksnap_latency": { 2024-08-28T19:43:52.385 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.385 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.386 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.386 INFO:teuthology.orchestra.run.smithi016.stdout: "req_open_latency": { 2024-08-28T19:43:52.386 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.386 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.386 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.386 INFO:teuthology.orchestra.run.smithi016.stdout: "req_readdir_latency": { 2024-08-28T19:43:52.387 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.387 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.387 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.387 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rename_latency": { 2024-08-28T19:43:52.387 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.387 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.387 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.388 INFO:teuthology.orchestra.run.smithi016.stdout: "req_renamesnap_latency": { 2024-08-28T19:43:52.388 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.388 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.388 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.388 INFO:teuthology.orchestra.run.smithi016.stdout: "req_snapdiff_latency": { 2024-08-28T19:43:52.388 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.389 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.389 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.389 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmdir_latency": { 2024-08-28T19:43:52.389 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.389 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.389 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.390 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmsnap_latency": { 2024-08-28T19:43:52.390 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.390 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.390 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.390 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmxattr_latency": { 2024-08-28T19:43:52.390 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.390 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setattr_latency": { 2024-08-28T19:43:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setdirlayout_latency": { 2024-08-28T19:43:52.392 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.392 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.392 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.392 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setfilelock_latency": { 2024-08-28T19:43:52.392 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.392 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.393 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.393 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setlayout_latency": { 2024-08-28T19:43:52.393 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.393 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.393 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.393 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setxattr_latency": { 2024-08-28T19:43:52.394 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.394 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.394 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.394 INFO:teuthology.orchestra.run.smithi016.stdout: "req_symlink_latency": { 2024-08-28T19:43:52.394 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.394 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.394 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.395 INFO:teuthology.orchestra.run.smithi016.stdout: "req_unlink_latency": { 2024-08-28T19:43:52.395 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.395 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.395 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.395 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_revoke_eviction": 0, 2024-08-28T19:43:52.395 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_acquisition_throttle": 0, 2024-08-28T19:43:52.395 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getvxattr_latency": { 2024-08-28T19:43:52.396 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.396 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.396 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:52.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.396 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessions": { 2024-08-28T19:43:52.396 INFO:teuthology.orchestra.run.smithi016.stdout: "session_count": 2, 2024-08-28T19:43:52.397 INFO:teuthology.orchestra.run.smithi016.stdout: "session_add": 2, 2024-08-28T19:43:52.397 INFO:teuthology.orchestra.run.smithi016.stdout: "session_remove": 0, 2024-08-28T19:43:52.397 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_open": 2, 2024-08-28T19:43:52.397 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_stale": 0, 2024-08-28T19:43:52.397 INFO:teuthology.orchestra.run.smithi016.stdout: "total_load": 0, 2024-08-28T19:43:52.397 INFO:teuthology.orchestra.run.smithi016.stdout: "average_load": 0, 2024-08-28T19:43:52.397 INFO:teuthology.orchestra.run.smithi016.stdout: "avg_session_uptime": 2459, 2024-08-28T19:43:52.397 INFO:teuthology.orchestra.run.smithi016.stdout: "mdthresh_evicted": 0 2024-08-28T19:43:52.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.398 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-08-28T19:43:52.398 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:43:52.398 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-08-28T19:43:52.398 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:43:52.398 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:43:52.398 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:43:52.398 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:43:52.398 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:43:52.399 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:43:52.399 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:43:52.399 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:43:52.399 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:43:52.399 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:43:52.399 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:43:52.399 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:43:52.399 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:43:52.400 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-08-28T19:43:52.400 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:43:52.400 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-08-28T19:43:52.400 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:43:52.400 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:43:52.400 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:43:52.400 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:43:52.401 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:43:52.401 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:43:52.401 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:43:52.401 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-08-28T19:43:52.401 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:43:52.401 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:43:52.401 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:43:52.401 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-08-28T19:43:52.402 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-08-28T19:43:52.402 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-08-28T19:43:52.402 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:43:52.402 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:43:52.402 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:43:52.402 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:43:52.402 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 291726, 2024-08-28T19:43:52.402 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 1453, 2024-08-28T19:43:52.402 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-08-28T19:43:52.403 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-08-28T19:43:52.403 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-08-28T19:43:52.403 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-08-28T19:43:52.403 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:43:52.403 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-08-28T19:43:52.403 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:43:52.403 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-08-28T19:43:52.403 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 9648, 2024-08-28T19:43:52.403 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 196, 2024-08-28T19:43:52.404 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:43:52.404 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:43:52.404 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-08-28T19:43:52.404 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-08-28T19:43:52.404 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 14257795, 2024-08-28T19:43:52.404 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 97443, 2024-08-28T19:43:52.404 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-08-28T19:43:52.404 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-08-28T19:43:52.404 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-08-28T19:43:52.405 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-08-28T19:43:52.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.405 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter": { 2024-08-28T19:43:52.405 INFO:teuthology.orchestra.run.smithi016.stdout: "op_active": 0, 2024-08-28T19:43:52.405 INFO:teuthology.orchestra.run.smithi016.stdout: "op_laggy": 0, 2024-08-28T19:43:52.405 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send": 5283, 2024-08-28T19:43:52.405 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send_bytes": 21, 2024-08-28T19:43:52.405 INFO:teuthology.orchestra.run.smithi016.stdout: "op_resend": 3, 2024-08-28T19:43:52.406 INFO:teuthology.orchestra.run.smithi016.stdout: "op_reply": 5283, 2024-08-28T19:43:52.406 INFO:teuthology.orchestra.run.smithi016.stdout: "op_latency": { 2024-08-28T19:43:52.406 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 5283, 2024-08-28T19:43:52.406 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 14.014770840, 2024-08-28T19:43:52.406 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.002652805 2024-08-28T19:43:52.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.406 INFO:teuthology.orchestra.run.smithi016.stdout: "op_inflight": 0, 2024-08-28T19:43:52.406 INFO:teuthology.orchestra.run.smithi016.stdout: "oplen_avg": { 2024-08-28T19:43:52.406 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 5283, 2024-08-28T19:43:52.407 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 5284 2024-08-28T19:43:52.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.407 INFO:teuthology.orchestra.run.smithi016.stdout: "op": 5283, 2024-08-28T19:43:52.407 INFO:teuthology.orchestra.run.smithi016.stdout: "op_r": 5283, 2024-08-28T19:43:52.407 INFO:teuthology.orchestra.run.smithi016.stdout: "op_w": 0, 2024-08-28T19:43:52.407 INFO:teuthology.orchestra.run.smithi016.stdout: "op_rmw": 0, 2024-08-28T19:43:52.407 INFO:teuthology.orchestra.run.smithi016.stdout: "op_pg": 0, 2024-08-28T19:43:52.407 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_stat": 3043, 2024-08-28T19:43:52.407 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_create": 0, 2024-08-28T19:43:52.407 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_read": 2238, 2024-08-28T19:43:52.408 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_write": 0, 2024-08-28T19:43:52.408 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writefull": 0, 2024-08-28T19:43:52.408 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writesame": 0, 2024-08-28T19:43:52.408 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_append": 0, 2024-08-28T19:43:52.408 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_zero": 0, 2024-08-28T19:43:52.408 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_truncate": 0, 2024-08-28T19:43:52.408 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_delete": 0, 2024-08-28T19:43:52.408 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_mapext": 0, 2024-08-28T19:43:52.408 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_sparse_read": 0, 2024-08-28T19:43:52.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_clonerange": 0, 2024-08-28T19:43:52.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_getxattr": 1, 2024-08-28T19:43:52.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_setxattr": 0, 2024-08-28T19:43:52.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_cmpxattr": 0, 2024-08-28T19:43:52.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_rmxattr": 0, 2024-08-28T19:43:52.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_resetxattrs": 0, 2024-08-28T19:43:52.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_call": 0, 2024-08-28T19:43:52.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_watch": 0, 2024-08-28T19:43:52.410 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_notify": 0, 2024-08-28T19:43:52.410 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_src_cmpxattr": 0, 2024-08-28T19:43:52.410 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls": 0, 2024-08-28T19:43:52.410 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls_filter": 0, 2024-08-28T19:43:52.410 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_other": 0, 2024-08-28T19:43:52.410 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_active": 0, 2024-08-28T19:43:52.410 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_send": 0, 2024-08-28T19:43:52.410 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_resend": 0, 2024-08-28T19:43:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_ping": 0, 2024-08-28T19:43:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_active": 0, 2024-08-28T19:43:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_send": 0, 2024-08-28T19:43:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_resend": 0, 2024-08-28T19:43:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_active": 0, 2024-08-28T19:43:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_send": 0, 2024-08-28T19:43:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_resend": 0, 2024-08-28T19:43:52.411 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_active": 0, 2024-08-28T19:43:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_send": 0, 2024-08-28T19:43:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_resend": 0, 2024-08-28T19:43:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "command_active": 0, 2024-08-28T19:43:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "command_send": 0, 2024-08-28T19:43:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "command_resend": 0, 2024-08-28T19:43:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "map_epoch": 116, 2024-08-28T19:43:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "map_full": 0, 2024-08-28T19:43:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: "map_inc": 34, 2024-08-28T19:43:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_sessions": 11, 2024-08-28T19:43:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_open": 12, 2024-08-28T19:43:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_close": 1, 2024-08-28T19:43:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_laggy": 0, 2024-08-28T19:43:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_wr": 0, 2024-08-28T19:43:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_rd": 2, 2024-08-28T19:43:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_del": 0 2024-08-28T19:43:52.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "oft": { 2024-08-28T19:43:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_objs": 0, 2024-08-28T19:43:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_kv_pairs": 0, 2024-08-28T19:43:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_updates": 0, 2024-08-28T19:43:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_removes": 0 2024-08-28T19:43:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "purge_queue": { 2024-08-28T19:43:52.414 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops": 0, 2024-08-28T19:43:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops_high_water": 0, 2024-08-28T19:43:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing": 0, 2024-08-28T19:43:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_high_water": 0, 2024-08-28T19:43:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed_ops": 0, 2024-08-28T19:43:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed": 0, 2024-08-28T19:43:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_item_in_journal": 0 2024-08-28T19:43:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.415 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:43:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-08-28T19:43:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 5817, 2024-08-28T19:43:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 369971794, 2024-08-28T19:43:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 5817, 2024-08-28T19:43:52.416 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-08-28T19:43:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-08-28T19:43:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 5817, 2024-08-28T19:43:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 369971794, 2024-08-28T19:43:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_bytes": { 2024-08-28T19:43:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-08-28T19:43:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-08-28T19:43:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-08-28T19:43:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:52.418 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-08-28T19:43:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 5283, 2024-08-28T19:43:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 367578626, 2024-08-28T19:43:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 5283, 2024-08-28T19:43:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 367578626, 2024-08-28T19:43:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.419 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_ops": { 2024-08-28T19:43:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 1024, 2024-08-28T19:43:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-08-28T19:43:52.420 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-08-28T19:43:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-08-28T19:43:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 5283, 2024-08-28T19:43:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 5283, 2024-08-28T19:43:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 5283, 2024-08-28T19:43:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 5283, 2024-08-28T19:43:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.421 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.422 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:52.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.422 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle": { 2024-08-28T19:43:52.422 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:52.422 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-08-28T19:43:52.422 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:52.422 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-08-28T19:43:52.423 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-08-28T19:43:52.423 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:52.423 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-08-28T19:43:52.423 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-08-28T19:43:52.423 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-08-28T19:43:52.423 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-08-28T19:43:52.423 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-08-28T19:43:52.423 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:52.424 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.424 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.424 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.424 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:52.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.424 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle-0x55bb7372ac30": { 2024-08-28T19:43:52.424 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:52.424 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-08-28T19:43:52.425 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:52.425 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-08-28T19:43:52.425 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-08-28T19:43:52.425 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:52.425 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-08-28T19:43:52.425 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-08-28T19:43:52.425 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-08-28T19:43:52.425 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-08-28T19:43:52.425 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-08-28T19:43:52.426 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:52.426 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:52.426 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:52.426 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:52.426 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:52.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:52.426 INFO:teuthology.orchestra.run.smithi016.stdout: "trackedop": { 2024-08-28T19:43:52.426 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_ops_count": 0 2024-08-28T19:43:52.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:52.427 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:43:53.145 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-08-28T19:43:53.145 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-08-28T19:43:53.145 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-08-28T19:43:53.145 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-08-28T19:43:53.145 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-08-28T19:43:53.146 DEBUG:tasks.check_counter:Getting stats from d 2024-08-28T19:43:53.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.d perf dump 2024-08-28T19:43:53.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:53 smithi016 ceph-mon[18556]: pgmap v1056: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 78 KiB/s wr, 19 op/s 2024-08-28T19:43:53.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:53 smithi028 ceph-mon[21769]: pgmap v1056: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 78 KiB/s wr, 19 op/s 2024-08-28T19:43:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:53 smithi115 ceph-mon[22184]: pgmap v1056: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 78 KiB/s wr, 19 op/s 2024-08-28T19:43:53.401 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:53.908 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:43:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:43:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-08-28T19:43:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-08-28T19:43:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-08-28T19:43:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-08-28T19:43:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-08-28T19:43:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-08-28T19:43:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001190030, 2024-08-28T19:43:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-08-28T19:43:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000248203, 2024-08-28T19:43:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-28T19:43:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-28T19:43:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-08-28T19:43:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:43:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 6, 2024-08-28T19:43:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 317, 2024-08-28T19:43:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3264, 2024-08-28T19:43:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 216016, 2024-08-28T19:43:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-08-28T19:43:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-08-28T19:43:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.031755341, 2024-08-28T19:43:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.024497062, 2024-08-28T19:43:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.004168907, 2024-08-28T19:43:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-28T19:43:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 316, 2024-08-28T19:43:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.022121884, 2024-08-28T19:43:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000070005 2024-08-28T19:43:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-08-28T19:43:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 216016 2024-08-28T19:43:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:43:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 418, 2024-08-28T19:43:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 413, 2024-08-28T19:43:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 884592, 2024-08-28T19:43:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 255296, 2024-08-28T19:43:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-08-28T19:43:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-08-28T19:43:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.066119739, 2024-08-28T19:43:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.034968689, 2024-08-28T19:43:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.038166961, 2024-08-28T19:43:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-28T19:43:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 412, 2024-08-28T19:43:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.021031627, 2024-08-28T19:43:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000051047 2024-08-28T19:43:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 884592, 2024-08-28T19:43:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 255296 2024-08-28T19:43:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-08-28T19:43:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-08-28T19:43:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-08-28T19:43:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-08-28T19:43:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:43:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-08-28T19:43:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:43:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:43:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:43:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:43:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:43:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:43:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:43:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:43:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:43:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:43:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:43:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:43:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:43:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-08-28T19:43:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:43:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-08-28T19:43:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:43:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:43:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:43:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:43:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:43:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:43:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:43:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-08-28T19:43:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:43:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:43:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:43:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-08-28T19:43:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-08-28T19:43:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-08-28T19:43:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:43:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:43:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:43:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:43:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148744, 2024-08-28T19:43:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2024-08-28T19:43:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-08-28T19:43:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-08-28T19:43:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-08-28T19:43:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-08-28T19:43:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:43:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-08-28T19:43:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:43:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-08-28T19:43:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-08-28T19:43:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-08-28T19:43:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:43:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:43:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-08-28T19:43:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-08-28T19:43:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-08-28T19:43:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-08-28T19:43:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-08-28T19:43:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-08-28T19:43:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-08-28T19:43:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-08-28T19:43:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:43:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-08-28T19:43:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 420, 2024-08-28T19:43:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 825474, 2024-08-28T19:43:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 420, 2024-08-28T19:43:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-08-28T19:43:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-08-28T19:43:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 420, 2024-08-28T19:43:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 825474, 2024-08-28T19:43:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:53.927 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:43:54.656 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-08-28T19:43:54.656 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-08-28T19:43:54.656 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-08-28T19:43:54.656 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-08-28T19:43:54.656 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-08-28T19:43:54.657 DEBUG:tasks.check_counter:Getting stats from g 2024-08-28T19:43:54.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.g perf dump 2024-08-28T19:43:54.910 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:55.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:55 smithi028 ceph-mon[21769]: pgmap v1057: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 32 KiB/s wr, 14 op/s 2024-08-28T19:43:55.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:55 smithi016 ceph-mon[18556]: pgmap v1057: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 32 KiB/s wr, 14 op/s 2024-08-28T19:43:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:55 smithi115 ceph-mon[22184]: pgmap v1057: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 32 KiB/s wr, 14 op/s 2024-08-28T19:43:55.385 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:43:55.386 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:43:55.386 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2861, 2024-08-28T19:43:55.386 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2775, 2024-08-28T19:43:55.386 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 107031646, 2024-08-28T19:43:55.386 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 999538, 2024-08-28T19:43:55.386 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 7, 2024-08-28T19:43:55.386 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-08-28T19:43:55.386 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.394799185, 2024-08-28T19:43:55.387 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.164840225, 2024-08-28T19:43:55.387 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.155600556, 2024-08-28T19:43:55.387 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.066486202, 2024-08-28T19:43:55.387 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:55.387 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2774, 2024-08-28T19:43:55.387 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.115790769, 2024-08-28T19:43:55.387 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000041741 2024-08-28T19:43:55.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.387 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:55.388 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.388 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.388 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.388 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1478992, 2024-08-28T19:43:55.388 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 259552 2024-08-28T19:43:55.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.388 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:43:55.388 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2011, 2024-08-28T19:43:55.389 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2319, 2024-08-28T19:43:55.389 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 105374142, 2024-08-28T19:43:55.389 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1361871, 2024-08-28T19:43:55.389 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 8, 2024-08-28T19:43:55.389 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 6, 2024-08-28T19:43:55.389 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.273256744, 2024-08-28T19:43:55.389 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.088860668, 2024-08-28T19:43:55.389 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.120555556, 2024-08-28T19:43:55.390 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.049205154, 2024-08-28T19:43:55.390 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:55.390 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2318, 2024-08-28T19:43:55.390 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.079857002, 2024-08-28T19:43:55.390 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000034450 2024-08-28T19:43:55.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.390 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:55.390 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.391 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.391 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.391 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-08-28T19:43:55.391 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 724224 2024-08-28T19:43:55.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.391 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:43:55.391 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1002, 2024-08-28T19:43:55.392 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 1004, 2024-08-28T19:43:55.392 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 134485109, 2024-08-28T19:43:55.392 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 318648, 2024-08-28T19:43:55.392 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 7, 2024-08-28T19:43:55.392 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 5, 2024-08-28T19:43:55.392 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.185598129, 2024-08-28T19:43:55.392 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.038201455, 2024-08-28T19:43:55.392 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.111015064, 2024-08-28T19:43:55.393 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.028490578, 2024-08-28T19:43:55.393 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:55.393 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1002, 2024-08-28T19:43:55.393 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.045270639, 2024-08-28T19:43:55.393 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000045180 2024-08-28T19:43:55.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.393 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:55.393 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.394 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.394 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.394 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-28T19:43:55.394 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-08-28T19:43:55.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.394 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-08-28T19:43:55.395 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 1, 2024-08-28T19:43:55.395 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-08-28T19:43:55.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.395 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-MDSRank": { 2024-08-28T19:43:55.395 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-08-28T19:43:55.395 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-08-28T19:43:55.395 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 8897, 2024-08-28T19:43:55.395 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.210581811, 2024-08-28T19:43:55.396 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000023668 2024-08-28T19:43:55.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:55.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.396 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-PurgeQueue": { 2024-08-28T19:43:55.396 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-08-28T19:43:55.396 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-08-28T19:43:55.396 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.396 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.397 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:55.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.397 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-08-28T19:43:55.397 INFO:teuthology.orchestra.run.smithi016.stdout: "request": 0, 2024-08-28T19:43:55.397 INFO:teuthology.orchestra.run.smithi016.stdout: "reply": 0, 2024-08-28T19:43:55.397 INFO:teuthology.orchestra.run.smithi016.stdout: "reply_latency": { 2024-08-28T19:43:55.397 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.398 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.398 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.398 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_reply": 0, 2024-08-28T19:43:55.398 INFO:teuthology.orchestra.run.smithi016.stdout: "forward": 0, 2024-08-28T19:43:55.398 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_complete": 0, 2024-08-28T19:43:55.398 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_keys": 0, 2024-08-28T19:43:55.398 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_commit": 0, 2024-08-28T19:43:55.399 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_split": 0, 2024-08-28T19:43:55.399 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_merge": 0, 2024-08-28T19:43:55.399 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes": 14970, 2024-08-28T19:43:55.399 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_top": 42, 2024-08-28T19:43:55.399 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_bottom": 14928, 2024-08-28T19:43:55.399 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pin_tail": 0, 2024-08-28T19:43:55.399 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pinned": 14911, 2024-08-28T19:43:55.399 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_expired": 0, 2024-08-28T19:43:55.400 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_with_caps": 0, 2024-08-28T19:43:55.400 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": 0, 2024-08-28T19:43:55.400 INFO:teuthology.orchestra.run.smithi016.stdout: "subtrees": 9, 2024-08-28T19:43:55.400 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse": 0, 2024-08-28T19:43:55.400 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_hit": 0, 2024-08-28T19:43:55.400 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_forward": 0, 2024-08-28T19:43:55.400 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_discover": 0, 2024-08-28T19:43:55.400 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_dir_fetch": 0, 2024-08-28T19:43:55.400 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_remote_ino": 0, 2024-08-28T19:43:55.401 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_lock": 0, 2024-08-28T19:43:55.401 INFO:teuthology.orchestra.run.smithi016.stdout: "load_cent": 0, 2024-08-28T19:43:55.401 INFO:teuthology.orchestra.run.smithi016.stdout: "q": 0, 2024-08-28T19:43:55.401 INFO:teuthology.orchestra.run.smithi016.stdout: "exported": 0, 2024-08-28T19:43:55.401 INFO:teuthology.orchestra.run.smithi016.stdout: "exported_inodes": 0, 2024-08-28T19:43:55.401 INFO:teuthology.orchestra.run.smithi016.stdout: "imported": 0, 2024-08-28T19:43:55.401 INFO:teuthology.orchestra.run.smithi016.stdout: "imported_inodes": 0, 2024-08-28T19:43:55.401 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_dir_fetch": 0, 2024-08-28T19:43:55.402 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_backtrace_fetch": 0, 2024-08-28T19:43:55.402 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_peer_discover": 0, 2024-08-28T19:43:55.402 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rfiles": 0, 2024-08-28T19:43:55.402 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rbytes": 0, 2024-08-28T19:43:55.402 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rsnaps": 0, 2024-08-28T19:43:55.402 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_fetch": 0, 2024-08-28T19:43:55.402 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_set_tag": 0, 2024-08-28T19:43:55.402 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_repaired": 0, 2024-08-28T19:43:55.403 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_inotable_repaired": 0, 2024-08-28T19:43:55.403 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_inodes": 0, 2024-08-28T19:43:55.403 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_base_inodes": 0, 2024-08-28T19:43:55.403 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dirfrag_rstats": 0, 2024-08-28T19:43:55.403 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_file_inodes": 0, 2024-08-28T19:43:55.403 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_inode_file_caps": 0, 2024-08-28T19:43:55.403 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_revoke": 0, 2024-08-28T19:43:55.403 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_grant": 0, 2024-08-28T19:43:55.404 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_trunc": 0, 2024-08-28T19:43:55.404 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-28T19:43:55.404 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-28T19:43:55.404 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps": 0, 2024-08-28T19:43:55.404 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps_dirty": 0, 2024-08-28T19:43:55.404 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_cap_release": 0, 2024-08-28T19:43:55.404 INFO:teuthology.orchestra.run.smithi016.stdout: "process_request_cap_release": 0 2024-08-28T19:43:55.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.405 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache": { 2024-08-28T19:43:55.405 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update": 0, 2024-08-28T19:43:55.405 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update_receipt": 0, 2024-08-28T19:43:55.405 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_try_discover": 0, 2024-08-28T19:43:55.405 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_send_discover": 0, 2024-08-28T19:43:55.405 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_handle_discover": 0, 2024-08-28T19:43:55.405 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays": 0, 2024-08-28T19:43:55.405 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_delayed": 0, 2024-08-28T19:43:55.406 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_enqueuing": 0, 2024-08-28T19:43:55.406 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_created": 0, 2024-08-28T19:43:55.406 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_enqueued": 0, 2024-08-28T19:43:55.406 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_reintegrated": 0, 2024-08-28T19:43:55.406 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_migrated": 0, 2024-08-28T19:43:55.406 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_processing": 0, 2024-08-28T19:43:55.406 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_enqueued": 0, 2024-08-28T19:43:55.406 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_prioritized": 0, 2024-08-28T19:43:55.406 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_started": 0, 2024-08-28T19:43:55.407 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_completed": 0, 2024-08-28T19:43:55.407 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_path": 0, 2024-08-28T19:43:55.407 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_inode": 0, 2024-08-28T19:43:55.407 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_enqueue_scrub": 0, 2024-08-28T19:43:55.407 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_exportdir": 0, 2024-08-28T19:43:55.407 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_flush": 0, 2024-08-28T19:43:55.407 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragmentdir": 0, 2024-08-28T19:43:55.407 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragstats": 0, 2024-08-28T19:43:55.407 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_inodestats": 0, 2024-08-28T19:43:55.408 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-08-28T19:43:55.408 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_succeeded": 0, 2024-08-28T19:43:55.408 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_write_failed": 0 2024-08-28T19:43:55.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.408 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log": { 2024-08-28T19:43:55.408 INFO:teuthology.orchestra.run.smithi016.stdout: "evlrg": 0, 2024-08-28T19:43:55.408 INFO:teuthology.orchestra.run.smithi016.stdout: "evadd": 0, 2024-08-28T19:43:55.408 INFO:teuthology.orchestra.run.smithi016.stdout: "evex": 0, 2024-08-28T19:43:55.409 INFO:teuthology.orchestra.run.smithi016.stdout: "evtrm": 0, 2024-08-28T19:43:55.409 INFO:teuthology.orchestra.run.smithi016.stdout: "ev": 80015, 2024-08-28T19:43:55.409 INFO:teuthology.orchestra.run.smithi016.stdout: "evexg": 0, 2024-08-28T19:43:55.409 INFO:teuthology.orchestra.run.smithi016.stdout: "evexd": 0, 2024-08-28T19:43:55.409 INFO:teuthology.orchestra.run.smithi016.stdout: "segadd": 0, 2024-08-28T19:43:55.409 INFO:teuthology.orchestra.run.smithi016.stdout: "segex": 0, 2024-08-28T19:43:55.409 INFO:teuthology.orchestra.run.smithi016.stdout: "segtrm": 0, 2024-08-28T19:43:55.410 INFO:teuthology.orchestra.run.smithi016.stdout: "seg": 112, 2024-08-28T19:43:55.410 INFO:teuthology.orchestra.run.smithi016.stdout: "segmjr": 8, 2024-08-28T19:43:55.410 INFO:teuthology.orchestra.run.smithi016.stdout: "segexg": 0, 2024-08-28T19:43:55.410 INFO:teuthology.orchestra.run.smithi016.stdout: "segexd": 0, 2024-08-28T19:43:55.410 INFO:teuthology.orchestra.run.smithi016.stdout: "expos": 4194304, 2024-08-28T19:43:55.410 INFO:teuthology.orchestra.run.smithi016.stdout: "wrpos": 348111694, 2024-08-28T19:43:55.410 INFO:teuthology.orchestra.run.smithi016.stdout: "rdpos": 348111120, 2024-08-28T19:43:55.410 INFO:teuthology.orchestra.run.smithi016.stdout: "jlat": { 2024-08-28T19:43:55.410 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.411 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.411 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.411 INFO:teuthology.orchestra.run.smithi016.stdout: "replayed": 80015 2024-08-28T19:43:55.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.411 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mem": { 2024-08-28T19:43:55.411 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1188, 2024-08-28T19:43:55.411 INFO:teuthology.orchestra.run.smithi016.stdout: "ino+": 16044, 2024-08-28T19:43:55.412 INFO:teuthology.orchestra.run.smithi016.stdout: "ino-": 14856, 2024-08-28T19:43:55.412 INFO:teuthology.orchestra.run.smithi016.stdout: "dir": 70, 2024-08-28T19:43:55.412 INFO:teuthology.orchestra.run.smithi016.stdout: "dir+": 135, 2024-08-28T19:43:55.412 INFO:teuthology.orchestra.run.smithi016.stdout: "dir-": 65, 2024-08-28T19:43:55.412 INFO:teuthology.orchestra.run.smithi016.stdout: "dn": 14970, 2024-08-28T19:43:55.412 INFO:teuthology.orchestra.run.smithi016.stdout: "dn+": 16437, 2024-08-28T19:43:55.412 INFO:teuthology.orchestra.run.smithi016.stdout: "dn-": 1467, 2024-08-28T19:43:55.412 INFO:teuthology.orchestra.run.smithi016.stdout: "cap": 0, 2024-08-28T19:43:55.413 INFO:teuthology.orchestra.run.smithi016.stdout: "cap+": 0, 2024-08-28T19:43:55.413 INFO:teuthology.orchestra.run.smithi016.stdout: "cap-": 0, 2024-08-28T19:43:55.413 INFO:teuthology.orchestra.run.smithi016.stdout: "rss": 137416, 2024-08-28T19:43:55.413 INFO:teuthology.orchestra.run.smithi016.stdout: "heap": 346624000 2024-08-28T19:43:55.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.413 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_server": { 2024-08-28T19:43:55.413 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_client_request": 0, 2024-08-28T19:43:55.413 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_server_request": 0, 2024-08-28T19:43:55.413 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_request": 0, 2024-08-28T19:43:55.414 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_session": 0, 2024-08-28T19:43:55.414 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_peer_request": 0, 2024-08-28T19:43:55.414 INFO:teuthology.orchestra.run.smithi016.stdout: "req_create_latency": { 2024-08-28T19:43:55.414 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.414 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.414 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.415 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getattr_latency": { 2024-08-28T19:43:55.415 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.415 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.415 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.415 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getfilelock_latency": { 2024-08-28T19:43:55.415 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.415 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.416 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.416 INFO:teuthology.orchestra.run.smithi016.stdout: "req_link_latency": { 2024-08-28T19:43:55.416 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.416 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.416 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.416 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookup_latency": { 2024-08-28T19:43:55.416 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.417 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.417 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookuphash_latency": { 2024-08-28T19:43:55.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.417 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.417 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupino_latency": { 2024-08-28T19:43:55.418 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.418 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.418 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.418 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupname_latency": { 2024-08-28T19:43:55.418 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.418 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.418 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.419 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupparent_latency": { 2024-08-28T19:43:55.419 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.419 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.419 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.419 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupsnap_latency": { 2024-08-28T19:43:55.419 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.419 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lssnap_latency": { 2024-08-28T19:43:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mkdir_latency": { 2024-08-28T19:43:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.422 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.422 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mknod_latency": { 2024-08-28T19:43:55.422 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.422 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.422 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.422 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mksnap_latency": { 2024-08-28T19:43:55.422 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "req_open_latency": { 2024-08-28T19:43:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.424 INFO:teuthology.orchestra.run.smithi016.stdout: "req_readdir_latency": { 2024-08-28T19:43:55.424 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.424 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.424 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.424 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rename_latency": { 2024-08-28T19:43:55.424 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.424 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.425 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.425 INFO:teuthology.orchestra.run.smithi016.stdout: "req_renamesnap_latency": { 2024-08-28T19:43:55.425 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.425 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.425 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.425 INFO:teuthology.orchestra.run.smithi016.stdout: "req_snapdiff_latency": { 2024-08-28T19:43:55.426 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.426 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.426 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.426 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmdir_latency": { 2024-08-28T19:43:55.426 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.426 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.426 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.427 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmsnap_latency": { 2024-08-28T19:43:55.427 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.427 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.427 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.427 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmxattr_latency": { 2024-08-28T19:43:55.427 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.427 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.428 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.428 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setattr_latency": { 2024-08-28T19:43:55.428 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.428 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.428 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.429 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setdirlayout_latency": { 2024-08-28T19:43:55.429 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.429 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.429 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.429 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setfilelock_latency": { 2024-08-28T19:43:55.429 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.429 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.429 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.430 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setlayout_latency": { 2024-08-28T19:43:55.430 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.430 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.430 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.430 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setxattr_latency": { 2024-08-28T19:43:55.430 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.431 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.431 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.431 INFO:teuthology.orchestra.run.smithi016.stdout: "req_symlink_latency": { 2024-08-28T19:43:55.431 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.431 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.431 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.431 INFO:teuthology.orchestra.run.smithi016.stdout: "req_unlink_latency": { 2024-08-28T19:43:55.432 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.432 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.432 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.432 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_revoke_eviction": 0, 2024-08-28T19:43:55.432 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_acquisition_throttle": 0, 2024-08-28T19:43:55.432 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getvxattr_latency": { 2024-08-28T19:43:55.432 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessions": { 2024-08-28T19:43:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: "session_count": 1, 2024-08-28T19:43:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: "session_add": 2, 2024-08-28T19:43:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: "session_remove": 1, 2024-08-28T19:43:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_open": 1, 2024-08-28T19:43:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_stale": 0, 2024-08-28T19:43:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "total_load": 0, 2024-08-28T19:43:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "average_load": 0, 2024-08-28T19:43:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "avg_session_uptime": 2461, 2024-08-28T19:43:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "mdthresh_evicted": 0 2024-08-28T19:43:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-08-28T19:43:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:43:55.435 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-08-28T19:43:55.435 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:43:55.435 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:43:55.435 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:43:55.435 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:43:55.435 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:43:55.435 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:43:55.435 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:43:55.436 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:43:55.436 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:43:55.436 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:43:55.436 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:43:55.436 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:43:55.436 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:43:55.436 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-08-28T19:43:55.436 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:43:55.437 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-08-28T19:43:55.437 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:43:55.437 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:43:55.437 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:43:55.437 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:43:55.437 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:43:55.437 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:43:55.437 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:43:55.437 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-08-28T19:43:55.438 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:43:55.438 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:43:55.438 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:43:55.438 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-08-28T19:43:55.438 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-08-28T19:43:55.438 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-08-28T19:43:55.438 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:43:55.438 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:43:55.439 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:43:55.439 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:43:55.439 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 290935, 2024-08-28T19:43:55.439 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 1470, 2024-08-28T19:43:55.439 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-08-28T19:43:55.439 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-08-28T19:43:55.439 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-08-28T19:43:55.439 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-08-28T19:43:55.439 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:43:55.440 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-08-28T19:43:55.440 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:43:55.440 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-08-28T19:43:55.440 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 9648, 2024-08-28T19:43:55.440 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 196, 2024-08-28T19:43:55.440 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:43:55.440 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:43:55.440 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-08-28T19:43:55.441 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-08-28T19:43:55.441 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 13358277, 2024-08-28T19:43:55.441 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 92270, 2024-08-28T19:43:55.441 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-08-28T19:43:55.441 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-08-28T19:43:55.441 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-08-28T19:43:55.441 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-08-28T19:43:55.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.441 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter": { 2024-08-28T19:43:55.442 INFO:teuthology.orchestra.run.smithi016.stdout: "op_active": 0, 2024-08-28T19:43:55.442 INFO:teuthology.orchestra.run.smithi016.stdout: "op_laggy": 0, 2024-08-28T19:43:55.442 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send": 5334, 2024-08-28T19:43:55.442 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send_bytes": 21, 2024-08-28T19:43:55.442 INFO:teuthology.orchestra.run.smithi016.stdout: "op_resend": 3, 2024-08-28T19:43:55.442 INFO:teuthology.orchestra.run.smithi016.stdout: "op_reply": 5334, 2024-08-28T19:43:55.442 INFO:teuthology.orchestra.run.smithi016.stdout: "op_latency": { 2024-08-28T19:43:55.442 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 5334, 2024-08-28T19:43:55.443 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 19.171686553, 2024-08-28T19:43:55.443 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.003594241 2024-08-28T19:43:55.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.443 INFO:teuthology.orchestra.run.smithi016.stdout: "op_inflight": 0, 2024-08-28T19:43:55.443 INFO:teuthology.orchestra.run.smithi016.stdout: "oplen_avg": { 2024-08-28T19:43:55.443 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 5334, 2024-08-28T19:43:55.443 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 5335 2024-08-28T19:43:55.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.443 INFO:teuthology.orchestra.run.smithi016.stdout: "op": 5334, 2024-08-28T19:43:55.444 INFO:teuthology.orchestra.run.smithi016.stdout: "op_r": 5334, 2024-08-28T19:43:55.444 INFO:teuthology.orchestra.run.smithi016.stdout: "op_w": 0, 2024-08-28T19:43:55.444 INFO:teuthology.orchestra.run.smithi016.stdout: "op_rmw": 0, 2024-08-28T19:43:55.444 INFO:teuthology.orchestra.run.smithi016.stdout: "op_pg": 0, 2024-08-28T19:43:55.444 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_stat": 3084, 2024-08-28T19:43:55.444 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_create": 0, 2024-08-28T19:43:55.444 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_read": 2248, 2024-08-28T19:43:55.444 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_write": 0, 2024-08-28T19:43:55.445 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writefull": 0, 2024-08-28T19:43:55.445 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writesame": 0, 2024-08-28T19:43:55.445 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_append": 0, 2024-08-28T19:43:55.445 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_zero": 0, 2024-08-28T19:43:55.445 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_truncate": 0, 2024-08-28T19:43:55.445 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_delete": 0, 2024-08-28T19:43:55.445 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_mapext": 0, 2024-08-28T19:43:55.445 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_sparse_read": 0, 2024-08-28T19:43:55.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_clonerange": 0, 2024-08-28T19:43:55.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_getxattr": 1, 2024-08-28T19:43:55.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_setxattr": 0, 2024-08-28T19:43:55.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_cmpxattr": 0, 2024-08-28T19:43:55.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_rmxattr": 0, 2024-08-28T19:43:55.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_resetxattrs": 0, 2024-08-28T19:43:55.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_call": 0, 2024-08-28T19:43:55.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_watch": 0, 2024-08-28T19:43:55.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_notify": 0, 2024-08-28T19:43:55.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_src_cmpxattr": 0, 2024-08-28T19:43:55.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls": 0, 2024-08-28T19:43:55.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls_filter": 0, 2024-08-28T19:43:55.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_other": 0, 2024-08-28T19:43:55.447 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_active": 0, 2024-08-28T19:43:55.447 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_send": 0, 2024-08-28T19:43:55.447 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_resend": 0, 2024-08-28T19:43:55.447 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_ping": 0, 2024-08-28T19:43:55.448 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_active": 0, 2024-08-28T19:43:55.448 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_send": 0, 2024-08-28T19:43:55.448 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_resend": 0, 2024-08-28T19:43:55.448 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_active": 0, 2024-08-28T19:43:55.448 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_send": 0, 2024-08-28T19:43:55.448 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_resend": 0, 2024-08-28T19:43:55.448 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_active": 0, 2024-08-28T19:43:55.448 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_send": 0, 2024-08-28T19:43:55.448 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_resend": 0, 2024-08-28T19:43:55.449 INFO:teuthology.orchestra.run.smithi016.stdout: "command_active": 0, 2024-08-28T19:43:55.449 INFO:teuthology.orchestra.run.smithi016.stdout: "command_send": 0, 2024-08-28T19:43:55.449 INFO:teuthology.orchestra.run.smithi016.stdout: "command_resend": 0, 2024-08-28T19:43:55.449 INFO:teuthology.orchestra.run.smithi016.stdout: "map_epoch": 116, 2024-08-28T19:43:55.449 INFO:teuthology.orchestra.run.smithi016.stdout: "map_full": 0, 2024-08-28T19:43:55.449 INFO:teuthology.orchestra.run.smithi016.stdout: "map_inc": 34, 2024-08-28T19:43:55.449 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_sessions": 11, 2024-08-28T19:43:55.449 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_open": 12, 2024-08-28T19:43:55.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_close": 1, 2024-08-28T19:43:55.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_laggy": 0, 2024-08-28T19:43:55.450 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_wr": 0, 2024-08-28T19:43:55.450 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_rd": 2, 2024-08-28T19:43:55.450 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_del": 0 2024-08-28T19:43:55.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.450 INFO:teuthology.orchestra.run.smithi016.stdout: "oft": { 2024-08-28T19:43:55.450 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_objs": 0, 2024-08-28T19:43:55.450 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_kv_pairs": 0, 2024-08-28T19:43:55.451 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_updates": 0, 2024-08-28T19:43:55.451 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_removes": 0 2024-08-28T19:43:55.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.451 INFO:teuthology.orchestra.run.smithi016.stdout: "purge_queue": { 2024-08-28T19:43:55.451 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops": 0, 2024-08-28T19:43:55.451 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops_high_water": 0, 2024-08-28T19:43:55.451 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing": 0, 2024-08-28T19:43:55.451 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_high_water": 0, 2024-08-28T19:43:55.452 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed_ops": 0, 2024-08-28T19:43:55.452 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed": 0, 2024-08-28T19:43:55.452 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_item_in_journal": 0 2024-08-28T19:43:55.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.452 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:43:55.452 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:55.452 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-08-28T19:43:55.452 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:55.452 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 5870, 2024-08-28T19:43:55.453 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 346332555, 2024-08-28T19:43:55.453 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:55.453 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 5870, 2024-08-28T19:43:55.453 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-08-28T19:43:55.453 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-08-28T19:43:55.453 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 5870, 2024-08-28T19:43:55.453 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 346332555, 2024-08-28T19:43:55.453 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:55.454 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.454 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.454 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:55.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.454 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_bytes": { 2024-08-28T19:43:55.454 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:55.454 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-08-28T19:43:55.454 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:55.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-08-28T19:43:55.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-08-28T19:43:55.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:55.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-08-28T19:43:55.455 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 5334, 2024-08-28T19:43:55.455 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 343932231, 2024-08-28T19:43:55.455 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 5334, 2024-08-28T19:43:55.455 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 343932231, 2024-08-28T19:43:55.456 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:55.456 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.456 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.456 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.456 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:55.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.456 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_ops": { 2024-08-28T19:43:55.456 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:55.457 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 1024, 2024-08-28T19:43:55.457 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:55.457 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-08-28T19:43:55.457 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-08-28T19:43:55.457 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:55.457 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-08-28T19:43:55.457 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 5334, 2024-08-28T19:43:55.457 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 5334, 2024-08-28T19:43:55.458 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 5334, 2024-08-28T19:43:55.458 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 5334, 2024-08-28T19:43:55.458 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:55.458 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.458 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.458 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:55.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.458 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle": { 2024-08-28T19:43:55.459 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:55.459 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-08-28T19:43:55.459 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:55.459 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-08-28T19:43:55.459 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-08-28T19:43:55.459 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:55.459 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-08-28T19:43:55.459 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-08-28T19:43:55.460 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-08-28T19:43:55.460 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-08-28T19:43:55.460 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-08-28T19:43:55.460 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:55.460 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.460 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.460 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:55.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.461 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle-0x55c8f30f0c30": { 2024-08-28T19:43:55.461 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:55.461 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-08-28T19:43:55.461 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:55.461 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-08-28T19:43:55.461 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-08-28T19:43:55.461 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:55.461 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-08-28T19:43:55.462 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-08-28T19:43:55.462 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-08-28T19:43:55.462 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-08-28T19:43:55.462 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-08-28T19:43:55.462 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:55.462 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:55.462 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:55.462 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:55.462 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:55.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:55.463 INFO:teuthology.orchestra.run.smithi016.stdout: "trackedop": { 2024-08-28T19:43:55.463 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_ops_count": 0 2024-08-28T19:43:55.463 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:55.463 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:43:56.334 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-08-28T19:43:56.334 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-08-28T19:43:56.334 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-08-28T19:43:56.334 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-08-28T19:43:56.334 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-08-28T19:43:56.334 DEBUG:tasks.check_counter:Getting stats from j 2024-08-28T19:43:56.335 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.j perf dump 2024-08-28T19:43:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:43:57.107 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-08-28T19:43:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:43:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-08-28T19:43:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-08-28T19:43:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-08-28T19:43:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-08-28T19:43:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-08-28T19:43:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-08-28T19:43:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001177247, 2024-08-28T19:43:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-08-28T19:43:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000279653, 2024-08-28T19:43:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-28T19:43:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-28T19:43:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-08-28T19:43:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:43:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1, 2024-08-28T19:43:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 315, 2024-08-28T19:43:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 160, 2024-08-28T19:43:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 135328, 2024-08-28T19:43:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 3, 2024-08-28T19:43:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-08-28T19:43:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.028437627, 2024-08-28T19:43:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.023965752, 2024-08-28T19:43:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001481285, 2024-08-28T19:43:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-28T19:43:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 313, 2024-08-28T19:43:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.014579936, 2024-08-28T19:43:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000046581 2024-08-28T19:43:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 160, 2024-08-28T19:43:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 135328 2024-08-28T19:43:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:43:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 423, 2024-08-28T19:43:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 416, 2024-08-28T19:43:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 887808, 2024-08-28T19:43:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 256384, 2024-08-28T19:43:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-08-28T19:43:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-08-28T19:43:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.069776264, 2024-08-28T19:43:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.034020404, 2024-08-28T19:43:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.043703401, 2024-08-28T19:43:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-28T19:43:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 416, 2024-08-28T19:43:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.024086948, 2024-08-28T19:43:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000057901 2024-08-28T19:43:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 887808, 2024-08-28T19:43:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 256384 2024-08-28T19:43:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-08-28T19:43:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-08-28T19:43:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-08-28T19:43:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-08-28T19:43:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:43:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-08-28T19:43:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:43:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:43:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:43:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:43:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:43:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:43:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:43:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:43:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:43:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:43:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:43:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:43:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:43:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-08-28T19:43:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:43:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-08-28T19:43:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:43:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:43:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:43:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:43:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:43:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:43:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:43:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-08-28T19:43:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:43:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:43:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:43:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-08-28T19:43:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-08-28T19:43:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-08-28T19:43:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:43:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:43:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:43:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:43:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148967, 2024-08-28T19:43:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 29, 2024-08-28T19:43:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-08-28T19:43:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-08-28T19:43:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-08-28T19:43:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-08-28T19:43:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:43:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-08-28T19:43:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:43:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-08-28T19:43:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-08-28T19:43:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-08-28T19:43:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:43:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:43:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-08-28T19:43:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-08-28T19:43:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-08-28T19:43:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-08-28T19:43:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-08-28T19:43:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-08-28T19:43:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-08-28T19:43:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-08-28T19:43:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-08-28T19:43:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:43:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-08-28T19:43:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-08-28T19:43:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-08-28T19:43:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 420, 2024-08-28T19:43:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 825577, 2024-08-28T19:43:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 420, 2024-08-28T19:43:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-08-28T19:43:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-08-28T19:43:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 420, 2024-08-28T19:43:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 825577, 2024-08-28T19:43:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-08-28T19:43:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-08-28T19:43:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-08-28T19:43:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-08-28T19:43:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:57.128 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-08-28T19:43:57.128 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-08-28T19:43:57.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:57 smithi016 ceph-mon[18556]: pgmap v1058: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-08-28T19:43:57.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:57 smithi028 ceph-mon[21769]: pgmap v1058: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-08-28T19:43:57.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:57 smithi115 ceph-mon[22184]: pgmap v1058: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-08-28T19:43:57.898 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-08-28T19:43:57.898 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-08-28T19:43:57.898 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-08-28T19:43:57.898 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-08-28T19:43:57.899 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-08-28T19:43:57.899 DEBUG:tasks.check_counter:Getting stats from b 2024-08-28T19:43:57.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.b perf dump 2024-08-28T19:43:58.230 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:43:59.256 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:43:59 smithi028 ceph-mon[21769]: pgmap v1059: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-08-28T19:43:59.257 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-08-28T19:43:59.257 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:43:59.257 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 230080, 2024-08-28T19:43:59.257 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 264870, 2024-08-28T19:43:59.258 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 69622483, 2024-08-28T19:43:59.258 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 407881328, 2024-08-28T19:43:59.258 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 292, 2024-08-28T19:43:59.258 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 7, 2024-08-28T19:43:59.258 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 18.687677013, 2024-08-28T19:43:59.258 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 8.877322368, 2024-08-28T19:43:59.258 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 9.274004768, 2024-08-28T19:43:59.258 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 1.817230999, 2024-08-28T19:43:59.259 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:59.259 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 264874, 2024-08-28T19:43:59.259 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 17.248513736, 2024-08-28T19:43:59.259 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000065119 2024-08-28T19:43:59.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.259 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:59.259 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 415714, 2024-08-28T19:43:59.259 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.083225983, 2024-08-28T19:43:59.260 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000200 2024-08-28T19:43:59.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.260 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-28T19:43:59.260 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-08-28T19:43:59.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.260 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:43:59.260 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 108403, 2024-08-28T19:43:59.260 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 140511, 2024-08-28T19:43:59.260 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 30148166, 2024-08-28T19:43:59.261 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 325911405, 2024-08-28T19:43:59.261 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 14, 2024-08-28T19:43:59.261 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 7, 2024-08-28T19:43:59.261 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 10.757151273, 2024-08-28T19:43:59.261 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 5.254366670, 2024-08-28T19:43:59.261 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 4.214074169, 2024-08-28T19:43:59.261 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 1.516000184, 2024-08-28T19:43:59.261 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:59.262 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 140509, 2024-08-28T19:43:59.262 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 9.466263186, 2024-08-28T19:43:59.262 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000067371 2024-08-28T19:43:59.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.262 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:59.262 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 172104, 2024-08-28T19:43:59.262 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.038886107, 2024-08-28T19:43:59.262 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000225 2024-08-28T19:43:59.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.263 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 160, 2024-08-28T19:43:59.263 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 657408 2024-08-28T19:43:59.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.263 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:43:59.263 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 407028, 2024-08-28T19:43:59.263 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 547927, 2024-08-28T19:43:59.263 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 123151771, 2024-08-28T19:43:59.264 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 595533418, 2024-08-28T19:43:59.264 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 33, 2024-08-28T19:43:59.264 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 9, 2024-08-28T19:43:59.264 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 37.864981475, 2024-08-28T19:43:59.264 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 20.589956233, 2024-08-28T19:43:59.264 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 18.415524126, 2024-08-28T19:43:59.264 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 1.911056628, 2024-08-28T19:43:59.264 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:43:59.265 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 547921, 2024-08-28T19:43:59.265 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 35.618171203, 2024-08-28T19:43:59.265 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000065006 2024-08-28T19:43:59.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.265 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:43:59.265 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 824628, 2024-08-28T19:43:59.265 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.184079968, 2024-08-28T19:43:59.265 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000223 2024-08-28T19:43:59.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.266 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 2298976, 2024-08-28T19:43:59.266 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 571072 2024-08-28T19:43:59.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.266 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-08-28T19:43:59.266 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-08-28T19:43:59.266 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-08-28T19:43:59.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.266 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-08-28T19:43:59.267 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-08-28T19:43:59.267 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-08-28T19:43:59.267 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 95619, 2024-08-28T19:43:59.267 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 103.437801494, 2024-08-28T19:43:59.267 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001081770 2024-08-28T19:43:59.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:43:59.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.267 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-08-28T19:43:59.267 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-08-28T19:43:59.268 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-08-28T19:43:59.268 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 35747, 2024-08-28T19:43:59.268 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6.905529094, 2024-08-28T19:43:59.268 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000193177 2024-08-28T19:43:59.268 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:43:59.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.268 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-08-28T19:43:59.268 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 187103, 2024-08-28T19:43:59.269 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 171154, 2024-08-28T19:43:59.269 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-08-28T19:43:59.269 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 171154, 2024-08-28T19:43:59.269 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1485.340607921, 2024-08-28T19:43:59.269 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.008678386 2024-08-28T19:43:59.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.269 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-08-28T19:43:59.269 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 15949, 2024-08-28T19:43:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 40, 2024-08-28T19:43:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 2, 2024-08-28T19:43:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 123, 2024-08-28T19:43:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 11, 2024-08-28T19:43:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 8, 2024-08-28T19:43:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 2370, 2024-08-28T19:43:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 512, 2024-08-28T19:43:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 1843, 2024-08-28T19:43:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 15, 2024-08-28T19:43:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 1639, 2024-08-28T19:43:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 4155, 2024-08-28T19:43:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 125, 2024-08-28T19:43:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 125, 2024-08-28T19:43:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 29, 2024-08-28T19:43:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 283674, 2024-08-28T19:43:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 185249, 2024-08-28T19:43:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 15900, 2024-08-28T19:43:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 146, 2024-08-28T19:43:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 2, 2024-08-28T19:43:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-08-28T19:43:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 1888, 2024-08-28T19:43:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 972, 2024-08-28T19:43:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-08-28T19:43:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 45, 2024-08-28T19:43:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 3105, 2024-08-28T19:43:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 38, 2024-08-28T19:43:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 2107, 2024-08-28T19:43:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-08-28T19:43:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 3435, 2024-08-28T19:43:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-08-28T19:43:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 662, 2024-08-28T19:43:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 216702462, 2024-08-28T19:43:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 6, 2024-08-28T19:43:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 3435, 2024-08-28T19:43:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-08-28T19:43:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-08-28T19:43:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-08-28T19:43:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 821, 2024-08-28T19:43:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 28, 2024-08-28T19:43:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 793, 2024-08-28T19:43:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 2614, 2024-08-28T19:43:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 61, 2024-08-28T19:43:59.275 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 43183, 2024-08-28T19:43:59.275 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 92770, 2024-08-28T19:43:59.275 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-08-28T19:43:59.275 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-28T19:43:59.275 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-28T19:43:59.275 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 77223, 2024-08-28T19:43:59.275 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 17929, 2024-08-28T19:43:59.275 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 7979, 2024-08-28T19:43:59.276 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 114794 2024-08-28T19:43:59.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.276 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-08-28T19:43:59.276 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 33, 2024-08-28T19:43:59.276 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 67, 2024-08-28T19:43:59.276 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 42, 2024-08-28T19:43:59.276 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 184, 2024-08-28T19:43:59.276 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 359, 2024-08-28T19:43:59.277 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-08-28T19:43:59.277 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-08-28T19:43:59.277 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-08-28T19:43:59.277 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 18756, 2024-08-28T19:43:59.277 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 18756, 2024-08-28T19:43:59.277 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-08-28T19:43:59.277 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-08-28T19:43:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-08-28T19:43:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-08-28T19:43:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-08-28T19:43:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-08-28T19:43:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-08-28T19:43:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 18, 2024-08-28T19:43:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 3149, 2024-08-28T19:43:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 28, 2024-08-28T19:43:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 48, 2024-08-28T19:43:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-08-28T19:43:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 23, 2024-08-28T19:43:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-08-28T19:43:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0, 2024-08-28T19:43:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-08-28T19:43:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_succeeded": 0, 2024-08-28T19:43:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_write_failed": 0 2024-08-28T19:43:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-08-28T19:43:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-08-28T19:43:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 127684, 2024-08-28T19:43:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 59871, 2024-08-28T19:43:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 49161, 2024-08-28T19:43:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 78523, 2024-08-28T19:43:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-08-28T19:43:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 10710, 2024-08-28T19:43:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 230, 2024-08-28T19:43:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 102, 2024-08-28T19:43:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 87, 2024-08-28T19:43:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 143, 2024-08-28T19:43:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 7, 2024-08-28T19:43:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-08-28T19:43:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 15, 2024-08-28T19:43:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 279254819, 2024-08-28T19:43:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 837013794, 2024-08-28T19:43:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 0, 2024-08-28T19:43:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-08-28T19:43:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 51993, 2024-08-28T19:43:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 530.710223473, 2024-08-28T19:43:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.010207339 2024-08-28T19:43:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 0 2024-08-28T19:43:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-08-28T19:43:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1968, 2024-08-28T19:43:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 23377, 2024-08-28T19:43:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 21409, 2024-08-28T19:43:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 226, 2024-08-28T19:43:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 424, 2024-08-28T19:43:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 198, 2024-08-28T19:43:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 2370, 2024-08-28T19:43:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 44304, 2024-08-28T19:43:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 41934, 2024-08-28T19:43:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 125, 2024-08-28T19:43:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 22856, 2024-08-28T19:43:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 22731, 2024-08-28T19:43:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 162968, 2024-08-28T19:43:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 385761280 2024-08-28T19:43:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-08-28T19:43:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 262369, 2024-08-28T19:43:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 18858, 2024-08-28T19:43:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 187103, 2024-08-28T19:43:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 2685, 2024-08-28T19:43:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 40870, 2024-08-28T19:43:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-08-28T19:43:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 19459, 2024-08-28T19:43:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 313.296133410, 2024-08-28T19:43:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.016100320 2024-08-28T19:43:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-08-28T19:43:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 24869, 2024-08-28T19:43:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 188.610755762, 2024-08-28T19:43:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.007584171 2024-08-28T19:43:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-08-28T19:43:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-08-28T19:43:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-08-28T19:43:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 38800, 2024-08-28T19:43:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 204.624597502, 2024-08-28T19:43:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.005273829 2024-08-28T19:43:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-08-28T19:43:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-08-28T19:43:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-08-28T19:43:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000166857, 2024-08-28T19:43:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000166857 2024-08-28T19:43:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-08-28T19:43:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-08-28T19:43:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.291 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.291 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.292 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-08-28T19:43:59.292 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.292 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.292 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.292 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-08-28T19:43:59.292 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 22, 2024-08-28T19:43:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.005103555, 2024-08-28T19:43:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000231979 2024-08-28T19:43:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-08-28T19:43:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 119, 2024-08-28T19:43:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.661356382, 2024-08-28T19:43:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.005557616 2024-08-28T19:43:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-08-28T19:43:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-08-28T19:43:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 22, 2024-08-28T19:43:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 12.337727581, 2024-08-28T19:43:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.560805799 2024-08-28T19:43:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-08-28T19:43:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 16, 2024-08-28T19:43:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.034248010, 2024-08-28T19:43:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002140500 2024-08-28T19:43:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-08-28T19:43:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 64634, 2024-08-28T19:43:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 258.217800742, 2024-08-28T19:43:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.003995076 2024-08-28T19:43:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-08-28T19:43:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3724, 2024-08-28T19:43:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 165.322177214, 2024-08-28T19:43:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.044393710 2024-08-28T19:43:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-08-28T19:43:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-08-28T19:43:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-08-28T19:43:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-08-28T19:43:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.002774576, 2024-08-28T19:43:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002774576 2024-08-28T19:43:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-08-28T19:43:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 16, 2024-08-28T19:43:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 5.142710076, 2024-08-28T19:43:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.321419379 2024-08-28T19:43:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-08-28T19:43:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-08-28T19:43:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 167, 2024-08-28T19:43:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 2.703140532, 2024-08-28T19:43:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.016186470 2024-08-28T19:43:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-08-28T19:43:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-08-28T19:43:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 540, 2024-08-28T19:43:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.300155449, 2024-08-28T19:43:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002407695 2024-08-28T19:43:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-08-28T19:43:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-08-28T19:43:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8, 2024-08-28T19:43:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.022073940, 2024-08-28T19:43:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002759242 2024-08-28T19:43:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-08-28T19:43:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-08-28T19:43:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 18756, 2024-08-28T19:43:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 333.059686333, 2024-08-28T19:43:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.017757500 2024-08-28T19:43:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-08-28T19:43:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-08-28T19:43:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-08-28T19:43:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:43:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-08-28T19:43:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 2, 2024-08-28T19:43:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 4, 2024-08-28T19:43:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 2, 2024-08-28T19:43:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 2, 2024-08-28T19:43:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-08-28T19:43:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 16072, 2024-08-28T19:43:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 8036, 2024-08-28T19:43:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 1350, 2024-08-28T19:43:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-08-28T19:43:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-08-28T19:43:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:43:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-08-28T19:43:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:43:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:43:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:43:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:43:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:43:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:43:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:43:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:43:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:43:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:43:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:43:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:43:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:43:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-08-28T19:43:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:43:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-08-28T19:43:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:43:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:43:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:43:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:43:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:43:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:43:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:43:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-08-28T19:43:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:43:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:43:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:43:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-08-28T19:43:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-08-28T19:43:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-08-28T19:43:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:43:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:43:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:43:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:43:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 363493, 2024-08-28T19:43:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 1527, 2024-08-28T19:43:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-08-28T19:43:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-08-28T19:43:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-08-28T19:43:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-08-28T19:43:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:43:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-08-28T19:43:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:43:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-08-28T19:43:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 9648, 2024-08-28T19:43:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 196, 2024-08-28T19:43:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:43:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:43:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-08-28T19:43:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-08-28T19:43:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 7438930, 2024-08-28T19:43:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 66862, 2024-08-28T19:43:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-08-28T19:43:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-08-28T19:43:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-08-28T19:43:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-08-28T19:43:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-08-28T19:43:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-08-28T19:43:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-08-28T19:43:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 96786, 2024-08-28T19:43:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 838473327, 2024-08-28T19:43:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 32, 2024-08-28T19:43:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 96786, 2024-08-28T19:43:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-08-28T19:43:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 96786, 2024-08-28T19:43:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 961.053495890, 2024-08-28T19:43:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.009929674 2024-08-28T19:43:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-08-28T19:43:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-08-28T19:43:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 96786, 2024-08-28T19:43:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 98205 2024-08-28T19:43:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 96786, 2024-08-28T19:43:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 4272, 2024-08-28T19:43:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 92514, 2024-08-28T19:43:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-08-28T19:43:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-08-28T19:43:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 0, 2024-08-28T19:43:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 50, 2024-08-28T19:43:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 767, 2024-08-28T19:43:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 52732, 2024-08-28T19:43:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 454, 2024-08-28T19:43:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-08-28T19:43:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-08-28T19:43:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-08-28T19:43:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 11, 2024-08-28T19:43:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 38333, 2024-08-28T19:43:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-08-28T19:43:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-08-28T19:43:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-08-28T19:43:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 3470, 2024-08-28T19:43:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 96, 2024-08-28T19:43:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-08-28T19:43:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-08-28T19:43:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-08-28T19:43:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-08-28T19:43:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-08-28T19:43:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-08-28T19:43:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-08-28T19:43:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-08-28T19:43:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-08-28T19:43:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-08-28T19:43:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-08-28T19:43:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-08-28T19:43:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-08-28T19:43:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-08-28T19:43:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-08-28T19:43:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-08-28T19:43:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-08-28T19:43:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-08-28T19:43:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-08-28T19:43:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-08-28T19:43:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-08-28T19:43:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-08-28T19:43:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-08-28T19:43:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-08-28T19:43:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-08-28T19:43:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-08-28T19:43:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 116, 2024-08-28T19:43:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-08-28T19:43:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 42, 2024-08-28T19:43:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 12, 2024-08-28T19:43:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 12, 2024-08-28T19:43:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-08-28T19:43:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-08-28T19:43:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 1731, 2024-08-28T19:43:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 84, 2024-08-28T19:43:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 477 2024-08-28T19:43:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-08-28T19:43:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 1, 2024-08-28T19:43:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-08-28T19:43:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-08-28T19:43:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-08-28T19:43:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-08-28T19:43:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-08-28T19:43:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 10, 2024-08-28T19:43:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-08-28T19:43:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 5, 2024-08-28T19:43:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 37512, 2024-08-28T19:43:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 18756, 2024-08-28T19:43:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-08-28T19:43:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:43:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:43:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-08-28T19:43:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:43:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 745507, 2024-08-28T19:43:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 155710540, 2024-08-28T19:43:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 745507, 2024-08-28T19:43:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:43:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:43:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 745507, 2024-08-28T19:43:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 155710540, 2024-08-28T19:43:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:43:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:43:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-08-28T19:43:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:43:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-08-28T19:43:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:43:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:43:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:43:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:43:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 96786, 2024-08-28T19:43:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 840520515, 2024-08-28T19:43:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 96786, 2024-08-28T19:43:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 840520515, 2024-08-28T19:43:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:43:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:43:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-08-28T19:43:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:43:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-08-28T19:43:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:43:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:43:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:43:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:43:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 96786, 2024-08-28T19:43:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 96786, 2024-08-28T19:43:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 96786, 2024-08-28T19:43:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 96786, 2024-08-28T19:43:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:43:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:43:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-08-28T19:43:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:43:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-08-28T19:43:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:43:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 18756, 2024-08-28T19:43:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 2047836, 2024-08-28T19:43:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 18756, 2024-08-28T19:43:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:43:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:43:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 739, 2024-08-28T19:43:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 2047836, 2024-08-28T19:43:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:43:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:43:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x55c5e2df05b0": { 2024-08-28T19:43:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:43:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-08-28T19:43:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:43:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 127684, 2024-08-28T19:43:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 832819490, 2024-08-28T19:43:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:43:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 127684, 2024-08-28T19:43:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:43:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:43:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 51993, 2024-08-28T19:43:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 832819490, 2024-08-28T19:43:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:43:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:43:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:43:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:43:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:43:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:43:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "trackedop": { 2024-08-28T19:43:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 3782 2024-08-28T19:43:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:43:59.339 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-08-28T19:43:59.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:43:59 smithi115 ceph-mon[22184]: pgmap v1059: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-08-28T19:43:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:43:59 smithi016 ceph-mon[18556]: pgmap v1059: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-08-28T19:44:00.475 INFO:tasks.check_counter:Daemon mds.b mds.exported=45 2024-08-28T19:44:00.476 INFO:tasks.check_counter:Daemon mds.b mds.imported=38 2024-08-28T19:44:00.476 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-08-28T19:44:00.476 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=22 2024-08-28T19:44:00.476 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=16 2024-08-28T19:44:00.476 DEBUG:tasks.check_counter:Getting stats from e 2024-08-28T19:44:00.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.e perf dump 2024-08-28T19:44:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:44:01.309 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:01 smithi028 ceph-mon[21769]: pgmap v1060: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 12 op/s 2024-08-28T19:44:01.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:01 smithi115 ceph-mon[22184]: pgmap v1060: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 12 op/s 2024-08-28T19:44:01.457 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-08-28T19:44:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:44:01.458 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1816, 2024-08-28T19:44:01.458 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1818, 2024-08-28T19:44:01.458 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 315039207, 2024-08-28T19:44:01.458 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 577494, 2024-08-28T19:44:01.458 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 8, 2024-08-28T19:44:01.458 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-08-28T19:44:01.458 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.357000252, 2024-08-28T19:44:01.458 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.060848765, 2024-08-28T19:44:01.459 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.234803712, 2024-08-28T19:44:01.459 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.047476672, 2024-08-28T19:44:01.459 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:01.459 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1816, 2024-08-28T19:44:01.459 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.064345214, 2024-08-28T19:44:01.459 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000035432 2024-08-28T19:44:01.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.459 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:01.460 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.460 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.460 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.460 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-28T19:44:01.460 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-08-28T19:44:01.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.460 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:44:01.460 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 3186, 2024-08-28T19:44:01.461 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 3498, 2024-08-28T19:44:01.461 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 307139040, 2024-08-28T19:44:01.461 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 1648131, 2024-08-28T19:44:01.461 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-08-28T19:44:01.461 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 6, 2024-08-28T19:44:01.461 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.497297818, 2024-08-28T19:44:01.461 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.171610582, 2024-08-28T19:44:01.461 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.214638651, 2024-08-28T19:44:01.462 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.086048385, 2024-08-28T19:44:01.462 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:01.462 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3497, 2024-08-28T19:44:01.462 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.141822537, 2024-08-28T19:44:01.462 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000040555 2024-08-28T19:44:01.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.462 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:01.462 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.462 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.463 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.463 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-08-28T19:44:01.463 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 636512 2024-08-28T19:44:01.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.463 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:44:01.463 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1784, 2024-08-28T19:44:01.463 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1698, 2024-08-28T19:44:01.464 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 213947301, 2024-08-28T19:44:01.464 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 658514, 2024-08-28T19:44:01.464 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-08-28T19:44:01.464 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-08-28T19:44:01.464 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.320575640, 2024-08-28T19:44:01.464 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.080188930, 2024-08-28T19:44:01.464 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.209990318, 2024-08-28T19:44:01.464 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.034627613, 2024-08-28T19:44:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1697, 2024-08-28T19:44:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.071033164, 2024-08-28T19:44:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000041858 2024-08-28T19:44:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.466 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1480512, 2024-08-28T19:44:01.466 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 262912 2024-08-28T19:44:01.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.466 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-08-28T19:44:01.466 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-08-28T19:44:01.466 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-08-28T19:44:01.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.467 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-08-28T19:44:01.467 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-08-28T19:44:01.467 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-08-28T19:44:01.467 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 10022, 2024-08-28T19:44:01.467 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.255826077, 2024-08-28T19:44:01.467 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000025526 2024-08-28T19:44:01.467 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:01.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.467 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-08-28T19:44:01.468 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-08-28T19:44:01.468 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-08-28T19:44:01.468 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.468 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.468 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:01.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.468 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-08-28T19:44:01.468 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-08-28T19:44:01.469 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-08-28T19:44:01.469 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-08-28T19:44:01.469 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.469 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.469 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.469 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-08-28T19:44:01.469 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-08-28T19:44:01.470 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-08-28T19:44:01.470 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-08-28T19:44:01.470 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-08-28T19:44:01.470 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-08-28T19:44:01.470 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-08-28T19:44:01.470 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 20869, 2024-08-28T19:44:01.470 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 3718, 2024-08-28T19:44:01.470 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 17151, 2024-08-28T19:44:01.470 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-08-28T19:44:01.471 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 15558, 2024-08-28T19:44:01.471 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-08-28T19:44:01.471 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-08-28T19:44:01.471 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-08-28T19:44:01.471 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 15, 2024-08-28T19:44:01.471 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-08-28T19:44:01.471 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-08-28T19:44:01.471 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-08-28T19:44:01.472 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-08-28T19:44:01.472 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-08-28T19:44:01.472 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-08-28T19:44:01.472 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-08-28T19:44:01.472 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-08-28T19:44:01.472 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-08-28T19:44:01.472 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-08-28T19:44:01.472 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-08-28T19:44:01.472 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-08-28T19:44:01.473 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-08-28T19:44:01.473 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-08-28T19:44:01.473 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-08-28T19:44:01.473 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-08-28T19:44:01.473 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 662, 2024-08-28T19:44:01.473 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 216702462, 2024-08-28T19:44:01.474 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 6, 2024-08-28T19:44:01.474 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-08-28T19:44:01.474 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-08-28T19:44:01.474 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-08-28T19:44:01.474 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-08-28T19:44:01.474 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-08-28T19:44:01.474 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-08-28T19:44:01.474 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-08-28T19:44:01.475 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-08-28T19:44:01.475 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-08-28T19:44:01.475 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-08-28T19:44:01.475 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-08-28T19:44:01.475 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-08-28T19:44:01.475 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-28T19:44:01.475 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-28T19:44:01.475 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-08-28T19:44:01.476 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-08-28T19:44:01.476 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-08-28T19:44:01.476 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-08-28T19:44:01.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.476 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-08-28T19:44:01.476 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-08-28T19:44:01.476 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-08-28T19:44:01.476 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-08-28T19:44:01.476 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-08-28T19:44:01.477 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-08-28T19:44:01.477 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-08-28T19:44:01.477 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-08-28T19:44:01.477 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-08-28T19:44:01.477 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-08-28T19:44:01.477 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-08-28T19:44:01.477 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-08-28T19:44:01.477 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-08-28T19:44:01.478 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-08-28T19:44:01.478 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-08-28T19:44:01.478 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-08-28T19:44:01.478 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-08-28T19:44:01.478 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-08-28T19:44:01.478 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-08-28T19:44:01.478 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-08-28T19:44:01.478 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-08-28T19:44:01.478 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-08-28T19:44:01.479 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-08-28T19:44:01.479 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-08-28T19:44:01.479 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-08-28T19:44:01.479 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0, 2024-08-28T19:44:01.479 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-08-28T19:44:01.479 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_succeeded": 0, 2024-08-28T19:44:01.479 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_write_failed": 0 2024-08-28T19:44:01.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-08-28T19:44:01.480 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-08-28T19:44:01.480 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-08-28T19:44:01.480 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-08-28T19:44:01.480 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-08-28T19:44:01.480 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 127691, 2024-08-28T19:44:01.480 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-08-28T19:44:01.480 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-08-28T19:44:01.480 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-08-28T19:44:01.481 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-08-28T19:44:01.481 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-08-28T19:44:01.481 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 143, 2024-08-28T19:44:01.481 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 12, 2024-08-28T19:44:01.481 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-08-28T19:44:01.481 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-08-28T19:44:01.482 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 279254819, 2024-08-28T19:44:01.482 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 837076227, 2024-08-28T19:44:01.482 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 837072064, 2024-08-28T19:44:01.482 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-08-28T19:44:01.482 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.482 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.482 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.483 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 127691 2024-08-28T19:44:01.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-08-28T19:44:01.483 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1946, 2024-08-28T19:44:01.483 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 23019, 2024-08-28T19:44:01.483 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 21073, 2024-08-28T19:44:01.483 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 95, 2024-08-28T19:44:01.483 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 287, 2024-08-28T19:44:01.484 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 192, 2024-08-28T19:44:01.484 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 20869, 2024-08-28T19:44:01.484 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 23965, 2024-08-28T19:44:01.484 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 3096, 2024-08-28T19:44:01.484 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-08-28T19:44:01.484 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-08-28T19:44:01.484 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-08-28T19:44:01.485 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 151048, 2024-08-28T19:44:01.485 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 367357952 2024-08-28T19:44:01.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-08-28T19:44:01.485 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-08-28T19:44:01.485 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-08-28T19:44:01.486 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-08-28T19:44:01.486 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-08-28T19:44:01.486 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-08-28T19:44:01.486 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-08-28T19:44:01.486 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.486 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.486 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.487 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-08-28T19:44:01.487 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.487 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.487 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.487 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-08-28T19:44:01.487 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.487 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.487 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.488 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-08-28T19:44:01.488 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.488 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.488 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.488 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-08-28T19:44:01.488 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.488 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.489 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.489 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-08-28T19:44:01.489 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.489 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.489 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.490 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-08-28T19:44:01.490 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.490 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.490 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.490 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-08-28T19:44:01.490 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.491 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.491 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.491 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-08-28T19:44:01.491 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.491 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.491 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.491 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-08-28T19:44:01.492 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.492 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.492 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.492 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-08-28T19:44:01.492 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.492 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.493 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.493 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-08-28T19:44:01.493 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.493 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.493 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.494 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-08-28T19:44:01.494 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.494 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.494 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.494 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-08-28T19:44:01.494 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.495 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.495 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.495 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-08-28T19:44:01.495 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.495 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.495 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.496 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-08-28T19:44:01.496 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.496 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.496 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.496 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-08-28T19:44:01.496 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.496 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.497 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.497 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-08-28T19:44:01.497 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.497 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.497 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.497 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-08-28T19:44:01.498 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.498 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.498 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.498 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-08-28T19:44:01.498 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.498 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.498 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.499 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-08-28T19:44:01.499 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.499 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.499 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.499 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-08-28T19:44:01.499 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.500 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.500 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.500 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-08-28T19:44:01.500 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.500 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.501 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.501 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-08-28T19:44:01.501 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.501 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.501 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.501 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-08-28T19:44:01.502 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.502 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.502 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.502 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-08-28T19:44:01.502 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.502 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.502 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.503 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-08-28T19:44:01.503 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.503 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.503 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.503 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-08-28T19:44:01.503 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.504 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.504 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.504 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-08-28T19:44:01.504 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.504 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.504 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.505 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-08-28T19:44:01.505 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-08-28T19:44:01.505 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-08-28T19:44:01.505 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.505 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.505 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:01.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-08-28T19:44:01.506 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 2, 2024-08-28T19:44:01.506 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 4, 2024-08-28T19:44:01.506 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 2, 2024-08-28T19:44:01.506 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 2, 2024-08-28T19:44:01.506 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-08-28T19:44:01.506 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-08-28T19:44:01.506 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-08-28T19:44:01.507 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 2482, 2024-08-28T19:44:01.507 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-08-28T19:44:01.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.507 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-08-28T19:44:01.507 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:44:01.507 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-08-28T19:44:01.507 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:44:01.507 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:44:01.508 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:44:01.508 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:44:01.508 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:44:01.508 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:44:01.508 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:44:01.508 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:44:01.508 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:44:01.508 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:44:01.508 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:44:01.509 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:44:01.509 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:44:01.509 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-08-28T19:44:01.509 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:44:01.509 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-08-28T19:44:01.509 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:44:01.509 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:44:01.509 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:44:01.510 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:44:01.510 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:44:01.510 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:44:01.510 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:44:01.510 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-08-28T19:44:01.510 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:44:01.510 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:44:01.511 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:44:01.511 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-08-28T19:44:01.511 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-08-28T19:44:01.511 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-08-28T19:44:01.511 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:44:01.511 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:44:01.511 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:44:01.511 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:44:01.511 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 329030, 2024-08-28T19:44:01.512 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 2481, 2024-08-28T19:44:01.512 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-08-28T19:44:01.512 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-08-28T19:44:01.512 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-08-28T19:44:01.512 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-08-28T19:44:01.512 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:44:01.512 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-08-28T19:44:01.513 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:44:01.513 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-08-28T19:44:01.513 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 9648, 2024-08-28T19:44:01.513 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 196, 2024-08-28T19:44:01.513 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:44:01.513 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:44:01.513 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-08-28T19:44:01.513 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-08-28T19:44:01.514 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 19508235, 2024-08-28T19:44:01.514 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 138713, 2024-08-28T19:44:01.514 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-08-28T19:44:01.514 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-08-28T19:44:01.514 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-08-28T19:44:01.514 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-08-28T19:44:01.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.515 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-08-28T19:44:01.515 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-08-28T19:44:01.515 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-08-28T19:44:01.515 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 6241, 2024-08-28T19:44:01.515 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 26, 2024-08-28T19:44:01.515 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 4, 2024-08-28T19:44:01.515 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 6241, 2024-08-28T19:44:01.515 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-08-28T19:44:01.516 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6241, 2024-08-28T19:44:01.516 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 15.512556162, 2024-08-28T19:44:01.516 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002485588 2024-08-28T19:44:01.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.516 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-08-28T19:44:01.516 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-08-28T19:44:01.516 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6241, 2024-08-28T19:44:01.516 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6242 2024-08-28T19:44:01.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.517 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 6241, 2024-08-28T19:44:01.517 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 6241, 2024-08-28T19:44:01.517 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-08-28T19:44:01.517 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-08-28T19:44:01.517 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-08-28T19:44:01.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 3615, 2024-08-28T19:44:01.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-08-28T19:44:01.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 2623, 2024-08-28T19:44:01.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-08-28T19:44:01.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-08-28T19:44:01.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-08-28T19:44:01.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-08-28T19:44:01.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-08-28T19:44:01.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-08-28T19:44:01.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-08-28T19:44:01.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-08-28T19:44:01.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-08-28T19:44:01.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-08-28T19:44:01.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 2, 2024-08-28T19:44:01.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-08-28T19:44:01.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-08-28T19:44:01.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-08-28T19:44:01.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-08-28T19:44:01.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-08-28T19:44:01.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-08-28T19:44:01.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-08-28T19:44:01.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-08-28T19:44:01.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-08-28T19:44:01.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-08-28T19:44:01.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-08-28T19:44:01.521 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-08-28T19:44:01.521 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-08-28T19:44:01.521 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-08-28T19:44:01.521 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-08-28T19:44:01.521 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-08-28T19:44:01.521 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-08-28T19:44:01.521 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-08-28T19:44:01.521 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-08-28T19:44:01.522 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-08-28T19:44:01.522 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-08-28T19:44:01.522 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-08-28T19:44:01.522 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-08-28T19:44:01.522 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-08-28T19:44:01.522 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-08-28T19:44:01.522 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-08-28T19:44:01.522 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-08-28T19:44:01.523 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 116, 2024-08-28T19:44:01.523 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-08-28T19:44:01.523 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 34, 2024-08-28T19:44:01.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 11, 2024-08-28T19:44:01.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 11, 2024-08-28T19:44:01.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-08-28T19:44:01.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-08-28T19:44:01.523 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-08-28T19:44:01.524 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-08-28T19:44:01.524 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-08-28T19:44:01.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.524 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-08-28T19:44:01.524 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-08-28T19:44:01.524 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-08-28T19:44:01.524 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-08-28T19:44:01.524 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-08-28T19:44:01.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.525 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-08-28T19:44:01.525 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-08-28T19:44:01.525 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-08-28T19:44:01.525 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-08-28T19:44:01.525 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-08-28T19:44:01.525 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-08-28T19:44:01.526 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-08-28T19:44:01.526 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-08-28T19:44:01.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.526 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:44:01.526 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:01.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-08-28T19:44:01.527 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:01.527 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 6782, 2024-08-28T19:44:01.527 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 835484863, 2024-08-28T19:44:01.527 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:01.528 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 6782, 2024-08-28T19:44:01.528 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:44:01.528 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:44:01.528 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 6782, 2024-08-28T19:44:01.528 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 835484863, 2024-08-28T19:44:01.528 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:01.529 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.529 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.529 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:01.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.530 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-08-28T19:44:01.530 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:01.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-08-28T19:44:01.530 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:01.530 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:44:01.531 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:44:01.531 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:01.531 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:01.531 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 6241, 2024-08-28T19:44:01.531 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 832927176, 2024-08-28T19:44:01.532 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 6241, 2024-08-28T19:44:01.532 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 832927176, 2024-08-28T19:44:01.532 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:01.532 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.532 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.533 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.533 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:01.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.533 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-08-28T19:44:01.533 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:01.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-08-28T19:44:01.534 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:01.534 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:44:01.534 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:44:01.534 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:01.535 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:01.535 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 6241, 2024-08-28T19:44:01.535 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 6241, 2024-08-28T19:44:01.535 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 6241, 2024-08-28T19:44:01.535 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 6241, 2024-08-28T19:44:01.536 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:01.536 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.536 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.536 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.536 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:01.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.537 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-08-28T19:44:01.537 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:01.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-08-28T19:44:01.537 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:01.538 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:44:01.538 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:44:01.538 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:01.538 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:01.538 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:44:01.539 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:44:01.539 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-08-28T19:44:01.539 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-08-28T19:44:01.539 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:01.539 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.539 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.540 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.540 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:01.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.540 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x55e1f0a048f0": { 2024-08-28T19:44:01.540 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:01.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-08-28T19:44:01.540 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:01.540 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:44:01.541 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:44:01.541 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:01.541 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:01.541 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:44:01.541 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:44:01.541 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-08-28T19:44:01.541 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-08-28T19:44:01.541 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:01.541 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:01.542 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:01.542 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:01.542 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:01.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:01.542 INFO:teuthology.orchestra.run.smithi028.stdout: "trackedop": { 2024-08-28T19:44:01.542 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-08-28T19:44:01.542 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:01.542 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-08-28T19:44:01.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:01 smithi016 ceph-mon[18556]: pgmap v1060: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 12 op/s 2024-08-28T19:44:02.577 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-08-28T19:44:02.578 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-08-28T19:44:02.578 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-08-28T19:44:02.578 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-08-28T19:44:02.578 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-08-28T19:44:02.578 DEBUG:tasks.check_counter:Getting stats from h 2024-08-28T19:44:02.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.h perf dump 2024-08-28T19:44:02.839 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:44:03.227 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:03 smithi028 ceph-mon[21769]: pgmap v1061: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 15 op/s 2024-08-28T19:44:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:03 smithi115 ceph-mon[22184]: pgmap v1061: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 15 op/s 2024-08-28T19:44:03.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:03 smithi016 ceph-mon[18556]: pgmap v1061: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 15 op/s 2024-08-28T19:44:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-08-28T19:44:03.504 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:44:03.505 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 194780, 2024-08-28T19:44:03.505 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 155279, 2024-08-28T19:44:03.505 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 69471273, 2024-08-28T19:44:03.505 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 203354954, 2024-08-28T19:44:03.505 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 60, 2024-08-28T19:44:03.505 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 7, 2024-08-28T19:44:03.505 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 15.280256484, 2024-08-28T19:44:03.505 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 7.124553738, 2024-08-28T19:44:03.506 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 7.966427865, 2024-08-28T19:44:03.506 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 1.747020677, 2024-08-28T19:44:03.506 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:03.506 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 155281, 2024-08-28T19:44:03.506 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 8.149623329, 2024-08-28T19:44:03.506 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000052483 2024-08-28T19:44:03.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.506 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:03.506 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 289791, 2024-08-28T19:44:03.507 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.053169190, 2024-08-28T19:44:03.507 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000183 2024-08-28T19:44:03.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.507 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-28T19:44:03.507 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-08-28T19:44:03.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.507 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:44:03.508 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 236154, 2024-08-28T19:44:03.508 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 341271, 2024-08-28T19:44:03.508 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 75383601, 2024-08-28T19:44:03.508 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 385824040, 2024-08-28T19:44:03.508 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 9, 2024-08-28T19:44:03.508 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 7, 2024-08-28T19:44:03.508 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 22.923065131, 2024-08-28T19:44:03.508 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 12.188422237, 2024-08-28T19:44:03.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 10.409355333, 2024-08-28T19:44:03.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 1.422384947, 2024-08-28T19:44:03.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:03.509 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 341270, 2024-08-28T19:44:03.509 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 26.281294496, 2024-08-28T19:44:03.509 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000077010 2024-08-28T19:44:03.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:03.509 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 480770, 2024-08-28T19:44:03.510 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.124668723, 2024-08-28T19:44:03.510 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000259 2024-08-28T19:44:03.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.510 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1511872, 2024-08-28T19:44:03.510 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 297456 2024-08-28T19:44:03.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.510 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:44:03.510 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 48967, 2024-08-28T19:44:03.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 48244, 2024-08-28T19:44:03.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 20063994, 2024-08-28T19:44:03.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 165461986, 2024-08-28T19:44:03.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 25, 2024-08-28T19:44:03.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 9, 2024-08-28T19:44:03.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 5.061871680, 2024-08-28T19:44:03.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 2.281572622, 2024-08-28T19:44:03.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 1.703953379, 2024-08-28T19:44:03.512 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 1.139262228, 2024-08-28T19:44:03.512 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:03.512 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 48239, 2024-08-28T19:44:03.512 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 4.503010351, 2024-08-28T19:44:03.512 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000093347 2024-08-28T19:44:03.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.512 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:03.512 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 50639, 2024-08-28T19:44:03.513 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.011184664, 2024-08-28T19:44:03.513 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000220 2024-08-28T19:44:03.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.513 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 6112, 2024-08-28T19:44:03.513 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 744176 2024-08-28T19:44:03.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.513 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-08-28T19:44:03.513 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-08-28T19:44:03.514 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-08-28T19:44:03.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.514 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-08-28T19:44:03.514 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-08-28T19:44:03.514 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-08-28T19:44:03.514 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 63141, 2024-08-28T19:44:03.514 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 75.988355794, 2024-08-28T19:44:03.514 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001203470 2024-08-28T19:44:03.515 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:03.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.515 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-08-28T19:44:03.515 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-08-28T19:44:03.515 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-08-28T19:44:03.515 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 36502, 2024-08-28T19:44:03.515 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 4.833834072, 2024-08-28T19:44:03.515 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000132426 2024-08-28T19:44:03.516 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:03.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.516 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-08-28T19:44:03.516 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 131674, 2024-08-28T19:44:03.516 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 123700, 2024-08-28T19:44:03.516 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-08-28T19:44:03.517 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 123700, 2024-08-28T19:44:03.517 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 819.366668339, 2024-08-28T19:44:03.517 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.006623821 2024-08-28T19:44:03.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.517 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-08-28T19:44:03.517 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 7974, 2024-08-28T19:44:03.517 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 3, 2024-08-28T19:44:03.517 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-08-28T19:44:03.518 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 56, 2024-08-28T19:44:03.518 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 5, 2024-08-28T19:44:03.518 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 3, 2024-08-28T19:44:03.518 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 1539, 2024-08-28T19:44:03.518 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 364, 2024-08-28T19:44:03.518 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 1064, 2024-08-28T19:44:03.518 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 111, 2024-08-28T19:44:03.518 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 1019, 2024-08-28T19:44:03.519 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 585, 2024-08-28T19:44:03.519 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 83, 2024-08-28T19:44:03.519 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 83, 2024-08-28T19:44:03.519 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 21, 2024-08-28T19:44:03.519 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 210544, 2024-08-28T19:44:03.519 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 122204, 2024-08-28T19:44:03.519 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 7960, 2024-08-28T19:44:03.519 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 101, 2024-08-28T19:44:03.520 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-08-28T19:44:03.520 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-08-28T19:44:03.520 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 2540, 2024-08-28T19:44:03.520 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 464, 2024-08-28T19:44:03.520 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-08-28T19:44:03.520 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 9, 2024-08-28T19:44:03.520 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 360, 2024-08-28T19:44:03.521 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 12, 2024-08-28T19:44:03.521 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 622, 2024-08-28T19:44:03.521 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-08-28T19:44:03.521 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 1422, 2024-08-28T19:44:03.521 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-08-28T19:44:03.521 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 730, 2024-08-28T19:44:03.521 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 212039648, 2024-08-28T19:44:03.521 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 6, 2024-08-28T19:44:03.522 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 1422, 2024-08-28T19:44:03.522 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-08-28T19:44:03.522 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-08-28T19:44:03.522 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-08-28T19:44:03.522 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 180, 2024-08-28T19:44:03.522 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-08-28T19:44:03.522 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 180, 2024-08-28T19:44:03.522 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 1242, 2024-08-28T19:44:03.522 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 35, 2024-08-28T19:44:03.523 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 16707, 2024-08-28T19:44:03.523 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 84271, 2024-08-28T19:44:03.523 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-08-28T19:44:03.523 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-28T19:44:03.523 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-28T19:44:03.523 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 49594, 2024-08-28T19:44:03.523 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 18673, 2024-08-28T19:44:03.523 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 7074, 2024-08-28T19:44:03.524 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 90104 2024-08-28T19:44:03.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.524 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-08-28T19:44:03.524 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 25, 2024-08-28T19:44:03.524 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 156, 2024-08-28T19:44:03.524 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 100, 2024-08-28T19:44:03.524 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 213, 2024-08-28T19:44:03.524 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 140, 2024-08-28T19:44:03.525 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-08-28T19:44:03.525 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-08-28T19:44:03.525 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-08-28T19:44:03.525 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 18681, 2024-08-28T19:44:03.525 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 18681, 2024-08-28T19:44:03.525 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-08-28T19:44:03.525 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-08-28T19:44:03.525 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-08-28T19:44:03.526 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-08-28T19:44:03.526 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-08-28T19:44:03.526 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-08-28T19:44:03.526 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-08-28T19:44:03.526 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 18, 2024-08-28T19:44:03.526 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 1506, 2024-08-28T19:44:03.526 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-08-28T19:44:03.526 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 11, 2024-08-28T19:44:03.527 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-08-28T19:44:03.527 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 8, 2024-08-28T19:44:03.527 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-08-28T19:44:03.527 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0, 2024-08-28T19:44:03.527 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-08-28T19:44:03.527 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_succeeded": 0, 2024-08-28T19:44:03.527 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_write_failed": 0 2024-08-28T19:44:03.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.528 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-08-28T19:44:03.528 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-08-28T19:44:03.528 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 104192, 2024-08-28T19:44:03.528 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 23887, 2024-08-28T19:44:03.528 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 12291, 2024-08-28T19:44:03.528 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 91901, 2024-08-28T19:44:03.528 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-08-28T19:44:03.528 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 11596, 2024-08-28T19:44:03.529 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 155, 2024-08-28T19:44:03.529 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 27, 2024-08-28T19:44:03.529 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 14, 2024-08-28T19:44:03.529 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 141, 2024-08-28T19:44:03.529 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 8, 2024-08-28T19:44:03.529 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-08-28T19:44:03.529 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 13, 2024-08-28T19:44:03.529 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 22295538, 2024-08-28T19:44:03.530 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 466023648, 2024-08-28T19:44:03.530 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 0, 2024-08-28T19:44:03.530 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-08-28T19:44:03.530 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 34116, 2024-08-28T19:44:03.530 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 319.076615701, 2024-08-28T19:44:03.530 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.009352697 2024-08-28T19:44:03.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.530 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 0 2024-08-28T19:44:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-08-28T19:44:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1087, 2024-08-28T19:44:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 20184, 2024-08-28T19:44:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 19097, 2024-08-28T19:44:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 179, 2024-08-28T19:44:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 330, 2024-08-28T19:44:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 151, 2024-08-28T19:44:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 1540, 2024-08-28T19:44:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 35941, 2024-08-28T19:44:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 34401, 2024-08-28T19:44:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 83, 2024-08-28T19:44:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 20884, 2024-08-28T19:44:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 20801, 2024-08-28T19:44:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 147172, 2024-08-28T19:44:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 358490112 2024-08-28T19:44:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-08-28T19:44:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 179194, 2024-08-28T19:44:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 2911, 2024-08-28T19:44:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 131674, 2024-08-28T19:44:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 3488, 2024-08-28T19:44:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 11026, 2024-08-28T19:44:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-08-28T19:44:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 18669, 2024-08-28T19:44:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 315.647142524, 2024-08-28T19:44:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.016907554 2024-08-28T19:44:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-08-28T19:44:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3924, 2024-08-28T19:44:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 67.103859478, 2024-08-28T19:44:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.017100881 2024-08-28T19:44:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-08-28T19:44:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-08-28T19:44:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-08-28T19:44:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 36847, 2024-08-28T19:44:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 88.528716520, 2024-08-28T19:44:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002402603 2024-08-28T19:44:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-08-28T19:44:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-08-28T19:44:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-08-28T19:44:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-08-28T19:44:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-08-28T19:44:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-08-28T19:44:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-08-28T19:44:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-08-28T19:44:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-08-28T19:44:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-08-28T19:44:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-08-28T19:44:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.001134609, 2024-08-28T19:44:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001134609 2024-08-28T19:44:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-08-28T19:44:03.544 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 40229, 2024-08-28T19:44:03.544 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 197.184251991, 2024-08-28T19:44:03.544 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.004901544 2024-08-28T19:44:03.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.544 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-08-28T19:44:03.544 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4483, 2024-08-28T19:44:03.544 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 91.413276549, 2024-08-28T19:44:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.020391094 2024-08-28T19:44:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-08-28T19:44:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-08-28T19:44:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-08-28T19:44:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-08-28T19:44:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-08-28T19:44:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-08-28T19:44:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 302, 2024-08-28T19:44:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 10.508202507, 2024-08-28T19:44:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.034795372 2024-08-28T19:44:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-08-28T19:44:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-08-28T19:44:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 564, 2024-08-28T19:44:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.782727652, 2024-08-28T19:44:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001387814 2024-08-28T19:44:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-08-28T19:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-08-28T19:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-08-28T19:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-08-28T19:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 18681, 2024-08-28T19:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 48.197356509, 2024-08-28T19:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002580020 2024-08-28T19:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-08-28T19:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-08-28T19:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-08-28T19:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-08-28T19:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 2, 2024-08-28T19:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 2, 2024-08-28T19:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 0, 2024-08-28T19:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 2, 2024-08-28T19:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-08-28T19:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 14466, 2024-08-28T19:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 7233, 2024-08-28T19:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 1309, 2024-08-28T19:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-08-28T19:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-08-28T19:44:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:44:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-08-28T19:44:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:44:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:44:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:44:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:44:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:44:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:44:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:44:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:44:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:44:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:44:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:44:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:44:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:44:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-08-28T19:44:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:44:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-08-28T19:44:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:44:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:44:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:44:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:44:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:44:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:44:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:44:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-08-28T19:44:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:44:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:44:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:44:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-08-28T19:44:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-08-28T19:44:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-08-28T19:44:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:44:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:44:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:44:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:44:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 321879, 2024-08-28T19:44:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 677, 2024-08-28T19:44:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-08-28T19:44:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-08-28T19:44:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-08-28T19:44:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-08-28T19:44:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:44:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-08-28T19:44:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:44:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-08-28T19:44:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 9648, 2024-08-28T19:44:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 196, 2024-08-28T19:44:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:44:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:44:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-08-28T19:44:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-08-28T19:44:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 4237126, 2024-08-28T19:44:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 35424, 2024-08-28T19:44:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-08-28T19:44:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-08-28T19:44:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-08-28T19:44:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-08-28T19:44:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-08-28T19:44:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-08-28T19:44:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-08-28T19:44:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 76233, 2024-08-28T19:44:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 466244411, 2024-08-28T19:44:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 28, 2024-08-28T19:44:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 76233, 2024-08-28T19:44:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-08-28T19:44:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 76233, 2024-08-28T19:44:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 842.682899868, 2024-08-28T19:44:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.011054043 2024-08-28T19:44:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-08-28T19:44:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-08-28T19:44:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 76233, 2024-08-28T19:44:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 77552 2024-08-28T19:44:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 76233, 2024-08-28T19:44:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 2325, 2024-08-28T19:44:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 73908, 2024-08-28T19:44:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-08-28T19:44:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-08-28T19:44:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 0, 2024-08-28T19:44:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 21, 2024-08-28T19:44:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 900, 2024-08-28T19:44:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 35016, 2024-08-28T19:44:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 525, 2024-08-28T19:44:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-08-28T19:44:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-08-28T19:44:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-08-28T19:44:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 5, 2024-08-28T19:44:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 37512, 2024-08-28T19:44:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-08-28T19:44:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-08-28T19:44:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-08-28T19:44:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1422, 2024-08-28T19:44:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 42, 2024-08-28T19:44:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-08-28T19:44:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-08-28T19:44:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-08-28T19:44:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-08-28T19:44:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-08-28T19:44:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-08-28T19:44:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-08-28T19:44:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-08-28T19:44:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-08-28T19:44:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-08-28T19:44:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-08-28T19:44:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-08-28T19:44:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-08-28T19:44:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-08-28T19:44:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-08-28T19:44:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-08-28T19:44:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-08-28T19:44:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-08-28T19:44:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-08-28T19:44:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-08-28T19:44:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-08-28T19:44:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-08-28T19:44:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-08-28T19:44:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-08-28T19:44:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-08-28T19:44:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-08-28T19:44:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 116, 2024-08-28T19:44:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-08-28T19:44:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 34, 2024-08-28T19:44:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 12, 2024-08-28T19:44:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 12, 2024-08-28T19:44:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-08-28T19:44:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-08-28T19:44:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 1534, 2024-08-28T19:44:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 6, 2024-08-28T19:44:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 569 2024-08-28T19:44:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-08-28T19:44:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 1, 2024-08-28T19:44:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-08-28T19:44:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-08-28T19:44:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-08-28T19:44:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-08-28T19:44:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-08-28T19:44:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 10, 2024-08-28T19:44:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-08-28T19:44:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 5, 2024-08-28T19:44:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 37362, 2024-08-28T19:44:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 18681, 2024-08-28T19:44:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-08-28T19:44:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:44:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-08-28T19:44:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 479897, 2024-08-28T19:44:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 121693562, 2024-08-28T19:44:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 479897, 2024-08-28T19:44:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:44:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:44:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 479897, 2024-08-28T19:44:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 121693562, 2024-08-28T19:44:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-08-28T19:44:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-08-28T19:44:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:44:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:44:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 76233, 2024-08-28T19:44:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 468283000, 2024-08-28T19:44:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 76233, 2024-08-28T19:44:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 468283000, 2024-08-28T19:44:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-08-28T19:44:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-08-28T19:44:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:44:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:44:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 76233, 2024-08-28T19:44:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 76233, 2024-08-28T19:44:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 76233, 2024-08-28T19:44:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 76233, 2024-08-28T19:44:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-08-28T19:44:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-08-28T19:44:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 18681, 2024-08-28T19:44:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 2038637, 2024-08-28T19:44:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 18681, 2024-08-28T19:44:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:44:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:44:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 900, 2024-08-28T19:44:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 2038637, 2024-08-28T19:44:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x55cef56f0820": { 2024-08-28T19:44:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 62111, 2024-08-28T19:44:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-08-28T19:44:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 104192, 2024-08-28T19:44:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 461829344, 2024-08-28T19:44:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 104192, 2024-08-28T19:44:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:44:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:44:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 34116, 2024-08-28T19:44:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 461767233, 2024-08-28T19:44:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "trackedop": { 2024-08-28T19:44:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 1560 2024-08-28T19:44:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:03.586 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-08-28T19:44:04.433 INFO:tasks.check_counter:Daemon mds.h mds.exported=9 2024-08-28T19:44:04.433 INFO:tasks.check_counter:Daemon mds.h mds.imported=12 2024-08-28T19:44:04.433 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-08-28T19:44:04.434 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-08-28T19:44:04.434 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-08-28T19:44:04.434 DEBUG:tasks.check_counter:Getting stats from k 2024-08-28T19:44:04.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.k perf dump 2024-08-28T19:44:04.692 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.b/config 2024-08-28T19:44:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:05 smithi115 ceph-mon[22184]: pgmap v1062: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 57 KiB/s wr, 27 op/s 2024-08-28T19:44:05.371 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:05 smithi028 ceph-mon[21769]: pgmap v1062: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 57 KiB/s wr, 27 op/s 2024-08-28T19:44:05.474 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-08-28T19:44:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:44:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 3581, 2024-08-28T19:44:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 3583, 2024-08-28T19:44:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 168805365, 2024-08-28T19:44:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 1138765, 2024-08-28T19:44:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 8, 2024-08-28T19:44:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-08-28T19:44:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.455279315, 2024-08-28T19:44:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.158446390, 2024-08-28T19:44:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.173672045, 2024-08-28T19:44:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.097972047, 2024-08-28T19:44:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3581, 2024-08-28T19:44:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.131565043, 2024-08-28T19:44:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000036739 2024-08-28T19:44:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-28T19:44:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-08-28T19:44:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:44:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1369, 2024-08-28T19:44:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1283, 2024-08-28T19:44:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 93965362, 2024-08-28T19:44:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 527720, 2024-08-28T19:44:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-08-28T19:44:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 5, 2024-08-28T19:44:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.205711930, 2024-08-28T19:44:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.069396294, 2024-08-28T19:44:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.122132375, 2024-08-28T19:44:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.023266243, 2024-08-28T19:44:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1282, 2024-08-28T19:44:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.056985982, 2024-08-28T19:44:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000044450 2024-08-28T19:44:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1481168, 2024-08-28T19:44:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 264672 2024-08-28T19:44:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:44:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1048, 2024-08-28T19:44:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1363, 2024-08-28T19:44:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 134883142, 2024-08-28T19:44:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 1060124, 2024-08-28T19:44:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 8, 2024-08-28T19:44:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 5, 2024-08-28T19:44:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.198937101, 2024-08-28T19:44:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.054680637, 2024-08-28T19:44:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.105590003, 2024-08-28T19:44:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.029029729, 2024-08-28T19:44:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1362, 2024-08-28T19:44:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.115298193, 2024-08-28T19:44:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000084653 2024-08-28T19:44:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-08-28T19:44:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 728768 2024-08-28T19:44:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-08-28T19:44:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-08-28T19:44:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-08-28T19:44:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-08-28T19:44:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-08-28T19:44:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-08-28T19:44:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 9000, 2024-08-28T19:44:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.219396872, 2024-08-28T19:44:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000024377 2024-08-28T19:44:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-08-28T19:44:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-08-28T19:44:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-08-28T19:44:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-08-28T19:44:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-08-28T19:44:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-08-28T19:44:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-08-28T19:44:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-08-28T19:44:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-08-28T19:44:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-08-28T19:44:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-08-28T19:44:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-08-28T19:44:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-08-28T19:44:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-08-28T19:44:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 20399, 2024-08-28T19:44:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 88, 2024-08-28T19:44:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 20311, 2024-08-28T19:44:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-08-28T19:44:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 20274, 2024-08-28T19:44:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-08-28T19:44:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-08-28T19:44:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-08-28T19:44:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 14, 2024-08-28T19:44:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-08-28T19:44:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-08-28T19:44:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-08-28T19:44:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-08-28T19:44:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-08-28T19:44:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-08-28T19:44:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-08-28T19:44:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-08-28T19:44:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-08-28T19:44:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-08-28T19:44:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-08-28T19:44:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-08-28T19:44:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-08-28T19:44:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-08-28T19:44:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-08-28T19:44:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-08-28T19:44:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-08-28T19:44:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-08-28T19:44:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-08-28T19:44:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-08-28T19:44:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-08-28T19:44:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-08-28T19:44:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-08-28T19:44:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-08-28T19:44:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-08-28T19:44:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-08-28T19:44:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-08-28T19:44:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-08-28T19:44:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-08-28T19:44:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-08-28T19:44:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-08-28T19:44:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-28T19:44:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-28T19:44:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-08-28T19:44:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-08-28T19:44:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-08-28T19:44:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-08-28T19:44:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-08-28T19:44:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-08-28T19:44:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-08-28T19:44:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-08-28T19:44:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-08-28T19:44:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-08-28T19:44:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-08-28T19:44:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-08-28T19:44:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-08-28T19:44:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-08-28T19:44:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-08-28T19:44:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-08-28T19:44:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-08-28T19:44:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-08-28T19:44:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-08-28T19:44:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-08-28T19:44:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-08-28T19:44:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-08-28T19:44:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-08-28T19:44:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-08-28T19:44:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-08-28T19:44:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-08-28T19:44:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-08-28T19:44:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-08-28T19:44:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-08-28T19:44:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0, 2024-08-28T19:44:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-08-28T19:44:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_succeeded": 0, 2024-08-28T19:44:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_write_failed": 0 2024-08-28T19:44:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-08-28T19:44:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-08-28T19:44:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-08-28T19:44:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-08-28T19:44:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-08-28T19:44:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 96824, 2024-08-28T19:44:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-08-28T19:44:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-08-28T19:44:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-08-28T19:44:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-08-28T19:44:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-08-28T19:44:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 134, 2024-08-28T19:44:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 9, 2024-08-28T19:44:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-08-28T19:44:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-08-28T19:44:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194352, 2024-08-28T19:44:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 398789408, 2024-08-28T19:44:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 398780951, 2024-08-28T19:44:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-08-28T19:44:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 96824 2024-08-28T19:44:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-08-28T19:44:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1296, 2024-08-28T19:44:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 20818, 2024-08-28T19:44:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 19522, 2024-08-28T19:44:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 83, 2024-08-28T19:44:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 150, 2024-08-28T19:44:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 67, 2024-08-28T19:44:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 20399, 2024-08-28T19:44:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 21190, 2024-08-28T19:44:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 791, 2024-08-28T19:44:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-08-28T19:44:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-08-28T19:44:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-08-28T19:44:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 145260, 2024-08-28T19:44:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 353083392 2024-08-28T19:44:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-08-28T19:44:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-08-28T19:44:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-08-28T19:44:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-08-28T19:44:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-08-28T19:44:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-08-28T19:44:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-08-28T19:44:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-08-28T19:44:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-08-28T19:44:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-08-28T19:44:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-08-28T19:44:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-08-28T19:44:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-08-28T19:44:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-08-28T19:44:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-08-28T19:44:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-08-28T19:44:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-08-28T19:44:05.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:05 smithi016 ceph-mon[18556]: pgmap v1062: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 57 KiB/s wr, 27 op/s 2024-08-28T19:44:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-08-28T19:44:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-08-28T19:44:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-08-28T19:44:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-08-28T19:44:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-08-28T19:44:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-08-28T19:44:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-08-28T19:44:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-08-28T19:44:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-08-28T19:44:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-08-28T19:44:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-08-28T19:44:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-08-28T19:44:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-08-28T19:44:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-08-28T19:44:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-08-28T19:44:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-08-28T19:44:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-08-28T19:44:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-08-28T19:44:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-08-28T19:44:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-08-28T19:44:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-08-28T19:44:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-08-28T19:44:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-08-28T19:44:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 2, 2024-08-28T19:44:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 1, 2024-08-28T19:44:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 1, 2024-08-28T19:44:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-08-28T19:44:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-08-28T19:44:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-08-28T19:44:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 2488, 2024-08-28T19:44:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-08-28T19:44:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-08-28T19:44:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:44:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-08-28T19:44:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:44:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:44:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:44:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:44:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:44:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:44:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:44:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:44:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:44:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:44:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:44:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:44:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:44:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-08-28T19:44:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:44:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-08-28T19:44:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:44:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:44:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:44:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:44:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:44:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:44:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:44:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-08-28T19:44:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:44:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:44:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:44:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-08-28T19:44:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-08-28T19:44:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-08-28T19:44:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:44:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:44:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:44:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:44:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 296721, 2024-08-28T19:44:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 1588, 2024-08-28T19:44:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-08-28T19:44:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-08-28T19:44:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-08-28T19:44:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-08-28T19:44:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:44:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-08-28T19:44:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:44:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-08-28T19:44:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 9648, 2024-08-28T19:44:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 196, 2024-08-28T19:44:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:44:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:44:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-08-28T19:44:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-08-28T19:44:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 17336863, 2024-08-28T19:44:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 116547, 2024-08-28T19:44:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-08-28T19:44:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-08-28T19:44:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-08-28T19:44:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-08-28T19:44:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-08-28T19:44:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-08-28T19:44:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-08-28T19:44:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 5450, 2024-08-28T19:44:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 21, 2024-08-28T19:44:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 3, 2024-08-28T19:44:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 5450, 2024-08-28T19:44:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-08-28T19:44:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 5450, 2024-08-28T19:44:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 13.229621734, 2024-08-28T19:44:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002427453 2024-08-28T19:44:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-08-28T19:44:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-08-28T19:44:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 5450, 2024-08-28T19:44:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 5451 2024-08-28T19:44:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 5450, 2024-08-28T19:44:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 5450, 2024-08-28T19:44:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-08-28T19:44:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-08-28T19:44:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-08-28T19:44:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 3194, 2024-08-28T19:44:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-08-28T19:44:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 2254, 2024-08-28T19:44:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-08-28T19:44:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-08-28T19:44:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-08-28T19:44:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-08-28T19:44:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-08-28T19:44:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-08-28T19:44:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-08-28T19:44:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-08-28T19:44:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-08-28T19:44:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-08-28T19:44:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1, 2024-08-28T19:44:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-08-28T19:44:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-08-28T19:44:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-08-28T19:44:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-08-28T19:44:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-08-28T19:44:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-08-28T19:44:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-08-28T19:44:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-08-28T19:44:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-08-28T19:44:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-08-28T19:44:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-08-28T19:44:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-08-28T19:44:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-08-28T19:44:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-08-28T19:44:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-08-28T19:44:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-08-28T19:44:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-08-28T19:44:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-08-28T19:44:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-08-28T19:44:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-08-28T19:44:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-08-28T19:44:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-08-28T19:44:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-08-28T19:44:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-08-28T19:44:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-08-28T19:44:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-08-28T19:44:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-08-28T19:44:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 116, 2024-08-28T19:44:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-08-28T19:44:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 34, 2024-08-28T19:44:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 11, 2024-08-28T19:44:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 12, 2024-08-28T19:44:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 1, 2024-08-28T19:44:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-08-28T19:44:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-08-28T19:44:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-08-28T19:44:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-08-28T19:44:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-08-28T19:44:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-08-28T19:44:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-08-28T19:44:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-08-28T19:44:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-08-28T19:44:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-08-28T19:44:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-08-28T19:44:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-08-28T19:44:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-08-28T19:44:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-08-28T19:44:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-08-28T19:44:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-08-28T19:44:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-08-28T19:44:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:44:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-08-28T19:44:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 5994, 2024-08-28T19:44:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 397083951, 2024-08-28T19:44:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 5994, 2024-08-28T19:44:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:44:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:44:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 5994, 2024-08-28T19:44:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 397083951, 2024-08-28T19:44:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-08-28T19:44:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-08-28T19:44:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:44:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:44:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 5450, 2024-08-28T19:44:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 394660393, 2024-08-28T19:44:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 5450, 2024-08-28T19:44:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 394660393, 2024-08-28T19:44:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-08-28T19:44:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-08-28T19:44:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:44:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:44:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 5450, 2024-08-28T19:44:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 5450, 2024-08-28T19:44:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 5450, 2024-08-28T19:44:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 5450, 2024-08-28T19:44:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-08-28T19:44:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-08-28T19:44:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:44:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:44:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:44:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:44:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-08-28T19:44:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-08-28T19:44:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x5586ae61cc30": { 2024-08-28T19:44:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-08-28T19:44:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-08-28T19:44:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-08-28T19:44:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-08-28T19:44:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-08-28T19:44:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-08-28T19:44:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-08-28T19:44:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-08-28T19:44:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-08-28T19:44:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-08-28T19:44:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-08-28T19:44:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-08-28T19:44:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-08-28T19:44:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-08-28T19:44:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "trackedop": { 2024-08-28T19:44:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-08-28T19:44:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-08-28T19:44:05.591 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-08-28T19:44:06.422 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-08-28T19:44:06.422 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-08-28T19:44:06.422 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-08-28T19:44:06.422 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-08-28T19:44:06.422 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-08-28T19:44:06.423 DEBUG:tasks.check_counter:Getting stats from c 2024-08-28T19:44:06.423 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.c perf dump 2024-08-28T19:44:06.755 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:44:07.129 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:07 smithi115 ceph-mon[22184]: pgmap v1063: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 83 KiB/s wr, 31 op/s 2024-08-28T19:44:07.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:07 smithi016 ceph-mon[18556]: pgmap v1063: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 83 KiB/s wr, 31 op/s 2024-08-28T19:44:07.525 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-08-28T19:44:07.525 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:44:07.526 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 176547, 2024-08-28T19:44:07.526 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 262492, 2024-08-28T19:44:07.526 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 56304174, 2024-08-28T19:44:07.526 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 345209639, 2024-08-28T19:44:07.526 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 71, 2024-08-28T19:44:07.526 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 11, 2024-08-28T19:44:07.526 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 19.934098468, 2024-08-28T19:44:07.526 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 9.860430459, 2024-08-28T19:44:07.527 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 8.444441617, 2024-08-28T19:44:07.527 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 1.551939550, 2024-08-28T19:44:07.527 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:07.527 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 262487, 2024-08-28T19:44:07.527 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 38.112702954, 2024-08-28T19:44:07.527 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000145198 2024-08-28T19:44:07.527 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.527 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:07.528 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 350837, 2024-08-28T19:44:07.528 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.112818785, 2024-08-28T19:44:07.528 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000321 2024-08-28T19:44:07.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.528 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 7760, 2024-08-28T19:44:07.528 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 27600 2024-08-28T19:44:07.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.528 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:44:07.528 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 52782, 2024-08-28T19:44:07.529 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 45115, 2024-08-28T19:44:07.529 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 20984622, 2024-08-28T19:44:07.529 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 137903707, 2024-08-28T19:44:07.529 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 13, 2024-08-28T19:44:07.529 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 7, 2024-08-28T19:44:07.529 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 6.075899505, 2024-08-28T19:44:07.529 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 2.694624220, 2024-08-28T19:44:07.529 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 2.033003174, 2024-08-28T19:44:07.530 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 1.482296313, 2024-08-28T19:44:07.530 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:07.530 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 45114, 2024-08-28T19:44:07.530 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 6.420507560, 2024-08-28T19:44:07.530 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000142317 2024-08-28T19:44:07.530 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.530 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:07.530 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 52167, 2024-08-28T19:44:07.531 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.010952803, 2024-08-28T19:44:07.531 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000209 2024-08-28T19:44:07.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.531 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 1508480, 2024-08-28T19:44:07.531 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 292304 2024-08-28T19:44:07.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.531 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:44:07.531 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 165243, 2024-08-28T19:44:07.532 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 123945, 2024-08-28T19:44:07.532 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 63237862, 2024-08-28T19:44:07.532 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 123973899, 2024-08-28T19:44:07.532 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 10, 2024-08-28T19:44:07.532 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 5, 2024-08-28T19:44:07.532 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 11.879407951, 2024-08-28T19:44:07.532 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 5.226279758, 2024-08-28T19:44:07.532 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 6.758031156, 2024-08-28T19:44:07.532 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 1.125604605, 2024-08-28T19:44:07.533 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:07.533 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 123947, 2024-08-28T19:44:07.533 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 12.997309847, 2024-08-28T19:44:07.533 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000104861 2024-08-28T19:44:07.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.533 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:07.533 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 256571, 2024-08-28T19:44:07.533 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.048053209, 2024-08-28T19:44:07.534 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000187 2024-08-28T19:44:07.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.534 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 320, 2024-08-28T19:44:07.534 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 730064 2024-08-28T19:44:07.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.534 INFO:teuthology.orchestra.run.smithi115.stdout: "cct": { 2024-08-28T19:44:07.534 INFO:teuthology.orchestra.run.smithi115.stdout: "total_workers": 1, 2024-08-28T19:44:07.534 INFO:teuthology.orchestra.run.smithi115.stdout: "unhealthy_workers": 0 2024-08-28T19:44:07.535 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.535 INFO:teuthology.orchestra.run.smithi115.stdout: "finisher-MDSRank": { 2024-08-28T19:44:07.535 INFO:teuthology.orchestra.run.smithi115.stdout: "queue_len": 0, 2024-08-28T19:44:07.535 INFO:teuthology.orchestra.run.smithi115.stdout: "complete_latency": { 2024-08-28T19:44:07.535 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 44789, 2024-08-28T19:44:07.535 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 69.013707269, 2024-08-28T19:44:07.535 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.001540862 2024-08-28T19:44:07.535 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:07.535 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.536 INFO:teuthology.orchestra.run.smithi115.stdout: "finisher-PurgeQueue": { 2024-08-28T19:44:07.536 INFO:teuthology.orchestra.run.smithi115.stdout: "queue_len": 0, 2024-08-28T19:44:07.536 INFO:teuthology.orchestra.run.smithi115.stdout: "complete_latency": { 2024-08-28T19:44:07.536 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 28885, 2024-08-28T19:44:07.536 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 4.152263574, 2024-08-28T19:44:07.536 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000143751 2024-08-28T19:44:07.536 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:07.536 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.537 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2024-08-28T19:44:07.537 INFO:teuthology.orchestra.run.smithi115.stdout: "request": 106028, 2024-08-28T19:44:07.537 INFO:teuthology.orchestra.run.smithi115.stdout: "reply": 99468, 2024-08-28T19:44:07.537 INFO:teuthology.orchestra.run.smithi115.stdout: "reply_latency": { 2024-08-28T19:44:07.537 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 99468, 2024-08-28T19:44:07.537 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 671.651531026, 2024-08-28T19:44:07.537 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.006752438 2024-08-28T19:44:07.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.538 INFO:teuthology.orchestra.run.smithi115.stdout: "slow_reply": 0, 2024-08-28T19:44:07.538 INFO:teuthology.orchestra.run.smithi115.stdout: "forward": 6560, 2024-08-28T19:44:07.538 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_fetch_complete": 6, 2024-08-28T19:44:07.538 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_fetch_keys": 0, 2024-08-28T19:44:07.538 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_commit": 47, 2024-08-28T19:44:07.538 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_split": 4, 2024-08-28T19:44:07.538 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_merge": 0, 2024-08-28T19:44:07.538 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes": 1433, 2024-08-28T19:44:07.539 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_top": 267, 2024-08-28T19:44:07.539 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_bottom": 1063, 2024-08-28T19:44:07.539 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_pin_tail": 103, 2024-08-28T19:44:07.539 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_pinned": 1052, 2024-08-28T19:44:07.539 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_expired": 425, 2024-08-28T19:44:07.539 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_with_caps": 92, 2024-08-28T19:44:07.539 INFO:teuthology.orchestra.run.smithi115.stdout: "caps": 92, 2024-08-28T19:44:07.539 INFO:teuthology.orchestra.run.smithi115.stdout: "subtrees": 20, 2024-08-28T19:44:07.539 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse": 165718, 2024-08-28T19:44:07.540 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_hit": 99817, 2024-08-28T19:44:07.540 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_forward": 6544, 2024-08-28T19:44:07.540 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_discover": 78, 2024-08-28T19:44:07.540 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_dir_fetch": 0, 2024-08-28T19:44:07.540 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_remote_ino": 0, 2024-08-28T19:44:07.540 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_lock": 2113, 2024-08-28T19:44:07.540 INFO:teuthology.orchestra.run.smithi115.stdout: "load_cent": 1196, 2024-08-28T19:44:07.540 INFO:teuthology.orchestra.run.smithi115.stdout: "q": 0, 2024-08-28T19:44:07.541 INFO:teuthology.orchestra.run.smithi115.stdout: "exported": 11, 2024-08-28T19:44:07.541 INFO:teuthology.orchestra.run.smithi115.stdout: "exported_inodes": 296, 2024-08-28T19:44:07.541 INFO:teuthology.orchestra.run.smithi115.stdout: "imported": 9, 2024-08-28T19:44:07.541 INFO:teuthology.orchestra.run.smithi115.stdout: "imported_inodes": 613, 2024-08-28T19:44:07.541 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_dir_fetch": 0, 2024-08-28T19:44:07.541 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_backtrace_fetch": 1563, 2024-08-28T19:44:07.541 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_peer_discover": 0, 2024-08-28T19:44:07.541 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rfiles": 730, 2024-08-28T19:44:07.542 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rbytes": 212039648, 2024-08-28T19:44:07.542 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rsnaps": 6, 2024-08-28T19:44:07.542 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_backtrace_fetch": 1563, 2024-08-28T19:44:07.542 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_set_tag": 0, 2024-08-28T19:44:07.542 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_backtrace_repaired": 0, 2024-08-28T19:44:07.542 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_inotable_repaired": 0, 2024-08-28T19:44:07.542 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dir_inodes": 180, 2024-08-28T19:44:07.542 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dir_base_inodes": 0, 2024-08-28T19:44:07.543 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dirfrag_rstats": 180, 2024-08-28T19:44:07.543 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_file_inodes": 1383, 2024-08-28T19:44:07.543 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_inode_file_caps": 9, 2024-08-28T19:44:07.543 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_revoke": 14227, 2024-08-28T19:44:07.543 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_grant": 70626, 2024-08-28T19:44:07.543 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_trunc": 0, 2024-08-28T19:44:07.543 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-28T19:44:07.543 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-28T19:44:07.543 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_caps": 37409, 2024-08-28T19:44:07.544 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_caps_dirty": 11589, 2024-08-28T19:44:07.544 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_cap_release": 5655, 2024-08-28T19:44:07.544 INFO:teuthology.orchestra.run.smithi115.stdout: "process_request_cap_release": 72554 2024-08-28T19:44:07.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.544 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_cache": { 2024-08-28T19:44:07.544 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_update": 15, 2024-08-28T19:44:07.544 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_update_receipt": 141, 2024-08-28T19:44:07.544 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_try_discover": 89, 2024-08-28T19:44:07.545 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_send_discover": 176, 2024-08-28T19:44:07.545 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_handle_discover": 118, 2024-08-28T19:44:07.545 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays": 1, 2024-08-28T19:44:07.545 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays_delayed": 0, 2024-08-28T19:44:07.545 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays_enqueuing": 0, 2024-08-28T19:44:07.545 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_created": 14954, 2024-08-28T19:44:07.545 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_enqueued": 14954, 2024-08-28T19:44:07.545 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_reintegrated": 0, 2024-08-28T19:44:07.546 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_migrated": 0, 2024-08-28T19:44:07.546 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_processing": 0, 2024-08-28T19:44:07.546 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_enqueued": 0, 2024-08-28T19:44:07.546 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_prioritized": 0, 2024-08-28T19:44:07.546 INFO:teuthology.orchestra.run.smithi115.stdout: "recovery_started": 0, 2024-08-28T19:44:07.546 INFO:teuthology.orchestra.run.smithi115.stdout: "recovery_completed": 0, 2024-08-28T19:44:07.546 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_quiesce_path": 18, 2024-08-28T19:44:07.546 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_quiesce_inode": 1623, 2024-08-28T19:44:07.546 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_enqueue_scrub": 0, 2024-08-28T19:44:07.547 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_exportdir": 22, 2024-08-28T19:44:07.547 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_flush": 0, 2024-08-28T19:44:07.547 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_fragmentdir": 4, 2024-08-28T19:44:07.547 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_fragstats": 0, 2024-08-28T19:44:07.547 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_inodestats": 0, 2024-08-28T19:44:07.547 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_started": 0, 2024-08-28T19:44:07.547 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_succeeded": 0, 2024-08-28T19:44:07.547 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_write_failed": 0 2024-08-28T19:44:07.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.548 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_log": { 2024-08-28T19:44:07.548 INFO:teuthology.orchestra.run.smithi115.stdout: "evlrg": 0, 2024-08-28T19:44:07.548 INFO:teuthology.orchestra.run.smithi115.stdout: "evadd": 79405, 2024-08-28T19:44:07.548 INFO:teuthology.orchestra.run.smithi115.stdout: "evex": 0, 2024-08-28T19:44:07.548 INFO:teuthology.orchestra.run.smithi115.stdout: "evtrm": 0, 2024-08-28T19:44:07.548 INFO:teuthology.orchestra.run.smithi115.stdout: "ev": 79405, 2024-08-28T19:44:07.548 INFO:teuthology.orchestra.run.smithi115.stdout: "evexg": 0, 2024-08-28T19:44:07.549 INFO:teuthology.orchestra.run.smithi115.stdout: "evexd": 0, 2024-08-28T19:44:07.549 INFO:teuthology.orchestra.run.smithi115.stdout: "segadd": 115, 2024-08-28T19:44:07.549 INFO:teuthology.orchestra.run.smithi115.stdout: "segex": 0, 2024-08-28T19:44:07.549 INFO:teuthology.orchestra.run.smithi115.stdout: "segtrm": 0, 2024-08-28T19:44:07.549 INFO:teuthology.orchestra.run.smithi115.stdout: "seg": 115, 2024-08-28T19:44:07.549 INFO:teuthology.orchestra.run.smithi115.stdout: "segmjr": 8, 2024-08-28T19:44:07.549 INFO:teuthology.orchestra.run.smithi115.stdout: "segexg": 0, 2024-08-28T19:44:07.549 INFO:teuthology.orchestra.run.smithi115.stdout: "segexd": 0, 2024-08-28T19:44:07.549 INFO:teuthology.orchestra.run.smithi115.stdout: "expos": 4194304, 2024-08-28T19:44:07.550 INFO:teuthology.orchestra.run.smithi115.stdout: "wrpos": 371936717, 2024-08-28T19:44:07.550 INFO:teuthology.orchestra.run.smithi115.stdout: "rdpos": 0, 2024-08-28T19:44:07.550 INFO:teuthology.orchestra.run.smithi115.stdout: "jlat": { 2024-08-28T19:44:07.550 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 24579, 2024-08-28T19:44:07.550 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 280.966471954, 2024-08-28T19:44:07.550 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.011431159 2024-08-28T19:44:07.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.551 INFO:teuthology.orchestra.run.smithi115.stdout: "replayed": 0 2024-08-28T19:44:07.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.551 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_mem": { 2024-08-28T19:44:07.551 INFO:teuthology.orchestra.run.smithi115.stdout: "ino": 1234, 2024-08-28T19:44:07.551 INFO:teuthology.orchestra.run.smithi115.stdout: "ino+": 16506, 2024-08-28T19:44:07.551 INFO:teuthology.orchestra.run.smithi115.stdout: "ino-": 15272, 2024-08-28T19:44:07.551 INFO:teuthology.orchestra.run.smithi115.stdout: "dir": 188, 2024-08-28T19:44:07.552 INFO:teuthology.orchestra.run.smithi115.stdout: "dir+": 265, 2024-08-28T19:44:07.552 INFO:teuthology.orchestra.run.smithi115.stdout: "dir-": 77, 2024-08-28T19:44:07.552 INFO:teuthology.orchestra.run.smithi115.stdout: "dn": 1433, 2024-08-28T19:44:07.552 INFO:teuthology.orchestra.run.smithi115.stdout: "dn+": 29089, 2024-08-28T19:44:07.552 INFO:teuthology.orchestra.run.smithi115.stdout: "dn-": 27656, 2024-08-28T19:44:07.552 INFO:teuthology.orchestra.run.smithi115.stdout: "cap": 92, 2024-08-28T19:44:07.552 INFO:teuthology.orchestra.run.smithi115.stdout: "cap+": 17025, 2024-08-28T19:44:07.552 INFO:teuthology.orchestra.run.smithi115.stdout: "cap-": 16933, 2024-08-28T19:44:07.553 INFO:teuthology.orchestra.run.smithi115.stdout: "rss": 148732, 2024-08-28T19:44:07.553 INFO:teuthology.orchestra.run.smithi115.stdout: "heap": 354299904 2024-08-28T19:44:07.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.553 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_server": { 2024-08-28T19:44:07.553 INFO:teuthology.orchestra.run.smithi115.stdout: "dispatch_client_request": 142873, 2024-08-28T19:44:07.553 INFO:teuthology.orchestra.run.smithi115.stdout: "dispatch_server_request": 1655, 2024-08-28T19:44:07.553 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_request": 106028, 2024-08-28T19:44:07.553 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_session": 1750, 2024-08-28T19:44:07.554 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_peer_request": 6677, 2024-08-28T19:44:07.554 INFO:teuthology.orchestra.run.smithi115.stdout: "req_create_latency": { 2024-08-28T19:44:07.554 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 14978, 2024-08-28T19:44:07.554 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 139.913114518, 2024-08-28T19:44:07.554 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.009341241 2024-08-28T19:44:07.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.554 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getattr_latency": { 2024-08-28T19:44:07.554 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 2808, 2024-08-28T19:44:07.555 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 58.283416313, 2024-08-28T19:44:07.555 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.020756202 2024-08-28T19:44:07.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.555 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getfilelock_latency": { 2024-08-28T19:44:07.555 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.555 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.555 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.556 INFO:teuthology.orchestra.run.smithi115.stdout: "req_link_latency": { 2024-08-28T19:44:07.556 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.556 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.556 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.556 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookup_latency": { 2024-08-28T19:44:07.556 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 28523, 2024-08-28T19:44:07.556 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 72.173522951, 2024-08-28T19:44:07.557 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.002530362 2024-08-28T19:44:07.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.557 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookuphash_latency": { 2024-08-28T19:44:07.557 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.557 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.557 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.557 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupino_latency": { 2024-08-28T19:44:07.558 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.558 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.558 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.558 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupname_latency": { 2024-08-28T19:44:07.558 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.558 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.558 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.559 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupparent_latency": { 2024-08-28T19:44:07.559 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.559 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:07 smithi028 ceph-mon[21769]: pgmap v1063: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 83 KiB/s wr, 31 op/s 2024-08-28T19:44:07.560 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.560 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupsnap_latency": { 2024-08-28T19:44:07.561 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.561 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.561 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.561 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lssnap_latency": { 2024-08-28T19:44:07.561 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.561 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.561 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.562 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mkdir_latency": { 2024-08-28T19:44:07.562 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.562 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.562 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.562 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mknod_latency": { 2024-08-28T19:44:07.562 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.562 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.563 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.563 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mksnap_latency": { 2024-08-28T19:44:07.563 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.563 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.563 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.563 INFO:teuthology.orchestra.run.smithi115.stdout: "req_open_latency": { 2024-08-28T19:44:07.563 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.564 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.564 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.564 INFO:teuthology.orchestra.run.smithi115.stdout: "req_readdir_latency": { 2024-08-28T19:44:07.564 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 33858, 2024-08-28T19:44:07.564 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 179.242676822, 2024-08-28T19:44:07.564 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.005293953 2024-08-28T19:44:07.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.565 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rename_latency": { 2024-08-28T19:44:07.565 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 3863, 2024-08-28T19:44:07.565 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 83.322116393, 2024-08-28T19:44:07.565 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.021569276 2024-08-28T19:44:07.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.565 INFO:teuthology.orchestra.run.smithi115.stdout: "req_renamesnap_latency": { 2024-08-28T19:44:07.565 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.565 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.566 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.566 INFO:teuthology.orchestra.run.smithi115.stdout: "req_snapdiff_latency": { 2024-08-28T19:44:07.566 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.566 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.566 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.566 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmdir_latency": { 2024-08-28T19:44:07.567 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.567 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.567 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.567 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmsnap_latency": { 2024-08-28T19:44:07.567 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.567 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.567 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.568 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmxattr_latency": { 2024-08-28T19:44:07.568 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.568 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.568 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.568 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setattr_latency": { 2024-08-28T19:44:07.568 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 56, 2024-08-28T19:44:07.569 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 21.101376978, 2024-08-28T19:44:07.569 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.376810303 2024-08-28T19:44:07.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.569 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setdirlayout_latency": { 2024-08-28T19:44:07.569 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.569 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.569 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.570 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setfilelock_latency": { 2024-08-28T19:44:07.570 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 428, 2024-08-28T19:44:07.570 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.675575129, 2024-08-28T19:44:07.570 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.001578446 2024-08-28T19:44:07.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.570 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setlayout_latency": { 2024-08-28T19:44:07.570 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.570 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.571 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.571 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setxattr_latency": { 2024-08-28T19:44:07.571 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.571 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.571 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.571 INFO:teuthology.orchestra.run.smithi115.stdout: "req_symlink_latency": { 2024-08-28T19:44:07.572 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.572 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.572 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.572 INFO:teuthology.orchestra.run.smithi115.stdout: "req_unlink_latency": { 2024-08-28T19:44:07.572 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 14954, 2024-08-28T19:44:07.572 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 116.939731922, 2024-08-28T19:44:07.572 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.007819963 2024-08-28T19:44:07.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.573 INFO:teuthology.orchestra.run.smithi115.stdout: "cap_revoke_eviction": 0, 2024-08-28T19:44:07.573 INFO:teuthology.orchestra.run.smithi115.stdout: "cap_acquisition_throttle": 0, 2024-08-28T19:44:07.573 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getvxattr_latency": { 2024-08-28T19:44:07.573 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.573 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.573 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.573 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:07.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.574 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_sessions": { 2024-08-28T19:44:07.574 INFO:teuthology.orchestra.run.smithi115.stdout: "session_count": 2, 2024-08-28T19:44:07.574 INFO:teuthology.orchestra.run.smithi115.stdout: "session_add": 2, 2024-08-28T19:44:07.574 INFO:teuthology.orchestra.run.smithi115.stdout: "session_remove": 0, 2024-08-28T19:44:07.574 INFO:teuthology.orchestra.run.smithi115.stdout: "sessions_open": 2, 2024-08-28T19:44:07.574 INFO:teuthology.orchestra.run.smithi115.stdout: "sessions_stale": 0, 2024-08-28T19:44:07.574 INFO:teuthology.orchestra.run.smithi115.stdout: "total_load": 13240, 2024-08-28T19:44:07.574 INFO:teuthology.orchestra.run.smithi115.stdout: "average_load": 6620, 2024-08-28T19:44:07.575 INFO:teuthology.orchestra.run.smithi115.stdout: "avg_session_uptime": 1312, 2024-08-28T19:44:07.575 INFO:teuthology.orchestra.run.smithi115.stdout: "mdthresh_evicted": 0 2024-08-28T19:44:07.575 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.575 INFO:teuthology.orchestra.run.smithi115.stdout: "mempool": { 2024-08-28T19:44:07.575 INFO:teuthology.orchestra.run.smithi115.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:44:07.575 INFO:teuthology.orchestra.run.smithi115.stdout: "bloom_filter_items": 0, 2024-08-28T19:44:07.575 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:44:07.575 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:44:07.575 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:44:07.576 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:44:07.576 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:44:07.576 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:44:07.576 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:44:07.576 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:44:07.576 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:44:07.576 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:44:07.576 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:44:07.577 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:44:07.577 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:44:07.577 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_extent_items": 0, 2024-08-28T19:44:07.577 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:44:07.577 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_blob_items": 0, 2024-08-28T19:44:07.577 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:44:07.577 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:44:07.577 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:44:07.578 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:44:07.578 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:44:07.578 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:44:07.578 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:44:07.578 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_txc_items": 0, 2024-08-28T19:44:07.578 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:44:07.578 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:44:07.578 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:44:07.578 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_items": 0, 2024-08-28T19:44:07.579 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_bytes": 0, 2024-08-28T19:44:07.579 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_items": 0, 2024-08-28T19:44:07.579 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:44:07.579 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:44:07.579 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:44:07.579 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:44:07.579 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_anon_bytes": 586377, 2024-08-28T19:44:07.579 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_anon_items": 794, 2024-08-28T19:44:07.579 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_meta_bytes": 0, 2024-08-28T19:44:07.580 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_meta_items": 0, 2024-08-28T19:44:07.580 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_bytes": 0, 2024-08-28T19:44:07.580 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_items": 0, 2024-08-28T19:44:07.580 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:44:07.580 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_mapbl_items": 0, 2024-08-28T19:44:07.580 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:44:07.580 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_pglog_items": 0, 2024-08-28T19:44:07.580 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_bytes": 9648, 2024-08-28T19:44:07.580 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_items": 196, 2024-08-28T19:44:07.581 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:44:07.581 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:44:07.581 INFO:teuthology.orchestra.run.smithi115.stdout: "pgmap_bytes": 0, 2024-08-28T19:44:07.581 INFO:teuthology.orchestra.run.smithi115.stdout: "pgmap_items": 0, 2024-08-28T19:44:07.581 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_co_bytes": 4617439, 2024-08-28T19:44:07.581 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_co_items": 36340, 2024-08-28T19:44:07.581 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_1_bytes": 0, 2024-08-28T19:44:07.582 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_1_items": 0, 2024-08-28T19:44:07.582 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_2_bytes": 0, 2024-08-28T19:44:07.582 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_2_items": 0 2024-08-28T19:44:07.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.582 INFO:teuthology.orchestra.run.smithi115.stdout: "objecter": { 2024-08-28T19:44:07.582 INFO:teuthology.orchestra.run.smithi115.stdout: "op_active": 0, 2024-08-28T19:44:07.582 INFO:teuthology.orchestra.run.smithi115.stdout: "op_laggy": 0, 2024-08-28T19:44:07.582 INFO:teuthology.orchestra.run.smithi115.stdout: "op_send": 58943, 2024-08-28T19:44:07.582 INFO:teuthology.orchestra.run.smithi115.stdout: "op_send_bytes": 371578271, 2024-08-28T19:44:07.583 INFO:teuthology.orchestra.run.smithi115.stdout: "op_resend": 33, 2024-08-28T19:44:07.583 INFO:teuthology.orchestra.run.smithi115.stdout: "op_reply": 58943, 2024-08-28T19:44:07.583 INFO:teuthology.orchestra.run.smithi115.stdout: "op_latency": { 2024-08-28T19:44:07.583 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 58943, 2024-08-28T19:44:07.583 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 784.235568967, 2024-08-28T19:44:07.583 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.013304982 2024-08-28T19:44:07.583 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.584 INFO:teuthology.orchestra.run.smithi115.stdout: "op_inflight": 0, 2024-08-28T19:44:07.584 INFO:teuthology.orchestra.run.smithi115.stdout: "oplen_avg": { 2024-08-28T19:44:07.584 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 58943, 2024-08-28T19:44:07.584 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 60098 2024-08-28T19:44:07.584 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.584 INFO:teuthology.orchestra.run.smithi115.stdout: "op": 58943, 2024-08-28T19:44:07.584 INFO:teuthology.orchestra.run.smithi115.stdout: "op_r": 2350, 2024-08-28T19:44:07.584 INFO:teuthology.orchestra.run.smithi115.stdout: "op_w": 56593, 2024-08-28T19:44:07.584 INFO:teuthology.orchestra.run.smithi115.stdout: "op_rmw": 0, 2024-08-28T19:44:07.585 INFO:teuthology.orchestra.run.smithi115.stdout: "op_pg": 0, 2024-08-28T19:44:07.585 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_stat": 0, 2024-08-28T19:44:07.585 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_create": 10, 2024-08-28T19:44:07.585 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_read": 781, 2024-08-28T19:44:07.585 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_write": 25360, 2024-08-28T19:44:07.585 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_writefull": 475, 2024-08-28T19:44:07.585 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_writesame": 0, 2024-08-28T19:44:07.585 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_append": 0, 2024-08-28T19:44:07.586 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_zero": 1, 2024-08-28T19:44:07.586 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_truncate": 4, 2024-08-28T19:44:07.586 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_delete": 30005, 2024-08-28T19:44:07.586 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_mapext": 0, 2024-08-28T19:44:07.586 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_sparse_read": 0, 2024-08-28T19:44:07.586 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_clonerange": 0, 2024-08-28T19:44:07.586 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_getxattr": 1563, 2024-08-28T19:44:07.586 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_setxattr": 20, 2024-08-28T19:44:07.586 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_cmpxattr": 0, 2024-08-28T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_rmxattr": 0, 2024-08-28T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_resetxattrs": 0, 2024-08-28T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_call": 0, 2024-08-28T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_watch": 0, 2024-08-28T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_notify": 0, 2024-08-28T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_src_cmpxattr": 0, 2024-08-28T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_pgls": 0, 2024-08-28T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_pgls_filter": 0, 2024-08-28T19:44:07.587 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_other": 0, 2024-08-28T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_active": 0, 2024-08-28T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_send": 0, 2024-08-28T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_resend": 0, 2024-08-28T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_ping": 0, 2024-08-28T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_active": 0, 2024-08-28T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_send": 0, 2024-08-28T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_resend": 0, 2024-08-28T19:44:07.588 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_active": 0, 2024-08-28T19:44:07.589 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_send": 0, 2024-08-28T19:44:07.589 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_resend": 0, 2024-08-28T19:44:07.589 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_active": 0, 2024-08-28T19:44:07.589 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_send": 0, 2024-08-28T19:44:07.589 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_resend": 0, 2024-08-28T19:44:07.589 INFO:teuthology.orchestra.run.smithi115.stdout: "command_active": 0, 2024-08-28T19:44:07.589 INFO:teuthology.orchestra.run.smithi115.stdout: "command_send": 0, 2024-08-28T19:44:07.589 INFO:teuthology.orchestra.run.smithi115.stdout: "command_resend": 0, 2024-08-28T19:44:07.590 INFO:teuthology.orchestra.run.smithi115.stdout: "map_epoch": 116, 2024-08-28T19:44:07.590 INFO:teuthology.orchestra.run.smithi115.stdout: "map_full": 0, 2024-08-28T19:44:07.590 INFO:teuthology.orchestra.run.smithi115.stdout: "map_inc": 34, 2024-08-28T19:44:07.590 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_sessions": 12, 2024-08-28T19:44:07.590 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_session_open": 12, 2024-08-28T19:44:07.590 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_session_close": 0, 2024-08-28T19:44:07.590 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_laggy": 0, 2024-08-28T19:44:07.590 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_wr": 1367, 2024-08-28T19:44:07.590 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_rd": 12, 2024-08-28T19:44:07.591 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_del": 500 2024-08-28T19:44:07.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.591 INFO:teuthology.orchestra.run.smithi115.stdout: "oft": { 2024-08-28T19:44:07.591 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_objs": 1, 2024-08-28T19:44:07.591 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_kv_pairs": 0, 2024-08-28T19:44:07.591 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_updates": 0, 2024-08-28T19:44:07.591 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_removes": 0 2024-08-28T19:44:07.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.592 INFO:teuthology.orchestra.run.smithi115.stdout: "purge_queue": { 2024-08-28T19:44:07.592 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_ops": 0, 2024-08-28T19:44:07.592 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_ops_high_water": 10, 2024-08-28T19:44:07.592 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing": 0, 2024-08-28T19:44:07.592 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_high_water": 5, 2024-08-28T19:44:07.592 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executed_ops": 29907, 2024-08-28T19:44:07.592 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executed": 14954, 2024-08-28T19:44:07.592 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_item_in_journal": 0 2024-08-28T19:44:07.593 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.593 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:44:07.593 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:07.593 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 104857600, 2024-08-28T19:44:07.593 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:07.593 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 394568, 2024-08-28T19:44:07.593 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 104980314, 2024-08-28T19:44:07.593 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:07.594 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 394568, 2024-08-28T19:44:07.594 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:07.594 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:07.594 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 394568, 2024-08-28T19:44:07.594 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 104980314, 2024-08-28T19:44:07.594 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:07.594 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.594 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.595 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.595 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:07.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.595 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-objecter_bytes": { 2024-08-28T19:44:07.595 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:07.595 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 104857600, 2024-08-28T19:44:07.595 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:07.595 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 0, 2024-08-28T19:44:07.595 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 0, 2024-08-28T19:44:07.596 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:07.596 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:07.596 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 58943, 2024-08-28T19:44:07.596 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 373211153, 2024-08-28T19:44:07.596 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 58943, 2024-08-28T19:44:07.596 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 373211153, 2024-08-28T19:44:07.596 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:07.596 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.597 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.597 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.597 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:07.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.597 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-objecter_ops": { 2024-08-28T19:44:07.597 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:07.597 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 1024, 2024-08-28T19:44:07.597 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 0, 2024-08-28T19:44:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 0, 2024-08-28T19:44:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 58943, 2024-08-28T19:44:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 58943, 2024-08-28T19:44:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 58943, 2024-08-28T19:44:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 58943, 2024-08-28T19:44:07.598 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-write_buf_throttle": { 2024-08-28T19:44:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:07.599 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 3758096384, 2024-08-28T19:44:07.600 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:07.600 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 14954, 2024-08-28T19:44:07.600 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 1632978, 2024-08-28T19:44:07.600 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:07.600 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 14954, 2024-08-28T19:44:07.600 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:07.600 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:07.600 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 781, 2024-08-28T19:44:07.600 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 1632978, 2024-08-28T19:44:07.601 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:07.601 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.601 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.601 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.601 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:07.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.601 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-write_buf_throttle-0x557a671ca820": { 2024-08-28T19:44:07.601 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:07.602 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 3758096384, 2024-08-28T19:44:07.602 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:07.602 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 79405, 2024-08-28T19:44:07.602 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 367742413, 2024-08-28T19:44:07.602 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:07.602 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 79405, 2024-08-28T19:44:07.602 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:07.602 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:07.602 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 24579, 2024-08-28T19:44:07.603 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 367742413, 2024-08-28T19:44:07.603 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:07.603 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:07.603 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:07.603 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:07.603 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:07.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:07.603 INFO:teuthology.orchestra.run.smithi115.stdout: "trackedop": { 2024-08-28T19:44:07.604 INFO:teuthology.orchestra.run.smithi115.stdout: "slow_ops_count": 1694 2024-08-28T19:44:07.604 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:07.604 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-08-28T19:44:08.598 INFO:tasks.check_counter:Daemon mds.c mds.exported=11 2024-08-28T19:44:08.598 INFO:tasks.check_counter:Daemon mds.c mds.imported=9 2024-08-28T19:44:08.598 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-08-28T19:44:08.599 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-08-28T19:44:08.599 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-08-28T19:44:08.599 DEBUG:tasks.check_counter:Getting stats from f 2024-08-28T19:44:08.599 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.f perf dump 2024-08-28T19:44:08.866 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:44:09.242 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:09 smithi115 ceph-mon[22184]: pgmap v1064: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 118 KiB/s wr, 30 op/s 2024-08-28T19:44:09.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:09 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:44:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:09 smithi016 ceph-mon[18556]: pgmap v1064: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 118 KiB/s wr, 30 op/s 2024-08-28T19:44:09.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:09 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:44:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:09 smithi028 ceph-mon[21769]: pgmap v1064: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 118 KiB/s wr, 30 op/s 2024-08-28T19:44:09.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:09 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-28T19:44:09.611 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-08-28T19:44:09.611 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:44:09.611 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 300095, 2024-08-28T19:44:09.611 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 353637, 2024-08-28T19:44:09.612 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 103438989, 2024-08-28T19:44:09.612 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 307471780, 2024-08-28T19:44:09.612 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 11, 2024-08-28T19:44:09.612 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 5, 2024-08-28T19:44:09.612 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 26.721822109, 2024-08-28T19:44:09.612 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 13.103569023, 2024-08-28T19:44:09.612 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 13.691571058, 2024-08-28T19:44:09.613 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 1.283143586, 2024-08-28T19:44:09.613 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:09.613 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 353637, 2024-08-28T19:44:09.613 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 49.408274180, 2024-08-28T19:44:09.613 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000139714 2024-08-28T19:44:09.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.613 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:09.613 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 571637, 2024-08-28T19:44:09.614 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.129324786, 2024-08-28T19:44:09.614 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000226 2024-08-28T19:44:09.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.614 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 1505456, 2024-08-28T19:44:09.614 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 299344 2024-08-28T19:44:09.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.614 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:44:09.614 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 75343, 2024-08-28T19:44:09.614 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 82002, 2024-08-28T19:44:09.615 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 26310890, 2024-08-28T19:44:09.615 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 117941154, 2024-08-28T19:44:09.615 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 13, 2024-08-28T19:44:09.615 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 7, 2024-08-28T19:44:09.615 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 9.662495658, 2024-08-28T19:44:09.615 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 4.537556753, 2024-08-28T19:44:09.615 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 3.537443874, 2024-08-28T19:44:09.615 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 1.922708397, 2024-08-28T19:44:09.616 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:09.616 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 82000, 2024-08-28T19:44:09.616 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 11.987730043, 2024-08-28T19:44:09.616 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000146191 2024-08-28T19:44:09.616 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.616 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:09.616 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 118367, 2024-08-28T19:44:09.616 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.025989658, 2024-08-28T19:44:09.617 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000219 2024-08-28T19:44:09.617 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.617 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-08-28T19:44:09.617 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 642480 2024-08-28T19:44:09.617 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.617 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:44:09.617 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 50422, 2024-08-28T19:44:09.617 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 51505, 2024-08-28T19:44:09.618 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 14822819, 2024-08-28T19:44:09.618 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 193291430, 2024-08-28T19:44:09.618 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 69, 2024-08-28T19:44:09.618 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 10, 2024-08-28T19:44:09.618 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 6.917110693, 2024-08-28T19:44:09.618 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 2.723317363, 2024-08-28T19:44:09.618 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 1.829323887, 2024-08-28T19:44:09.618 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 2.359815282, 2024-08-28T19:44:09.618 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:09.619 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 51503, 2024-08-28T19:44:09.619 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 8.740901355, 2024-08-28T19:44:09.619 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000169716 2024-08-28T19:44:09.619 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.619 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:09.619 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 50171, 2024-08-28T19:44:09.619 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.013616140, 2024-08-28T19:44:09.619 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000271 2024-08-28T19:44:09.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.620 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-28T19:44:09.620 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 0 2024-08-28T19:44:09.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.620 INFO:teuthology.orchestra.run.smithi115.stdout: "cct": { 2024-08-28T19:44:09.620 INFO:teuthology.orchestra.run.smithi115.stdout: "total_workers": 1, 2024-08-28T19:44:09.620 INFO:teuthology.orchestra.run.smithi115.stdout: "unhealthy_workers": 0 2024-08-28T19:44:09.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.621 INFO:teuthology.orchestra.run.smithi115.stdout: "finisher-MDSRank": { 2024-08-28T19:44:09.621 INFO:teuthology.orchestra.run.smithi115.stdout: "queue_len": 0, 2024-08-28T19:44:09.621 INFO:teuthology.orchestra.run.smithi115.stdout: "complete_latency": { 2024-08-28T19:44:09.621 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 43637, 2024-08-28T19:44:09.621 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 62.449568696, 2024-08-28T19:44:09.621 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.001431115 2024-08-28T19:44:09.621 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:09.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.621 INFO:teuthology.orchestra.run.smithi115.stdout: "finisher-PurgeQueue": { 2024-08-28T19:44:09.622 INFO:teuthology.orchestra.run.smithi115.stdout: "queue_len": 0, 2024-08-28T19:44:09.622 INFO:teuthology.orchestra.run.smithi115.stdout: "complete_latency": { 2024-08-28T19:44:09.622 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 26529, 2024-08-28T19:44:09.622 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 3.802949955, 2024-08-28T19:44:09.622 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000143350 2024-08-28T19:44:09.622 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:09.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.622 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2024-08-28T19:44:09.623 INFO:teuthology.orchestra.run.smithi115.stdout: "request": 102509, 2024-08-28T19:44:09.623 INFO:teuthology.orchestra.run.smithi115.stdout: "reply": 98941, 2024-08-28T19:44:09.623 INFO:teuthology.orchestra.run.smithi115.stdout: "reply_latency": { 2024-08-28T19:44:09.623 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 98941, 2024-08-28T19:44:09.623 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 888.944417111, 2024-08-28T19:44:09.623 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.008984590 2024-08-28T19:44:09.623 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.623 INFO:teuthology.orchestra.run.smithi115.stdout: "slow_reply": 0, 2024-08-28T19:44:09.624 INFO:teuthology.orchestra.run.smithi115.stdout: "forward": 3567, 2024-08-28T19:44:09.624 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_fetch_complete": 14, 2024-08-28T19:44:09.624 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_fetch_keys": 0, 2024-08-28T19:44:09.624 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_commit": 47, 2024-08-28T19:44:09.624 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_split": 4, 2024-08-28T19:44:09.624 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_merge": 0, 2024-08-28T19:44:09.624 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes": 1461, 2024-08-28T19:44:09.624 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_top": 253, 2024-08-28T19:44:09.625 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_bottom": 1005, 2024-08-28T19:44:09.625 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_pin_tail": 203, 2024-08-28T19:44:09.625 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_pinned": 1100, 2024-08-28T19:44:09.625 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_expired": 1711, 2024-08-28T19:44:09.625 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_with_caps": 116, 2024-08-28T19:44:09.625 INFO:teuthology.orchestra.run.smithi115.stdout: "caps": 116, 2024-08-28T19:44:09.625 INFO:teuthology.orchestra.run.smithi115.stdout: "subtrees": 20, 2024-08-28T19:44:09.625 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse": 165193, 2024-08-28T19:44:09.626 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_hit": 100154, 2024-08-28T19:44:09.626 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_forward": 3537, 2024-08-28T19:44:09.626 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_discover": 83, 2024-08-28T19:44:09.626 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_dir_fetch": 2, 2024-08-28T19:44:09.626 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_remote_ino": 0, 2024-08-28T19:44:09.626 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_lock": 1633, 2024-08-28T19:44:09.626 INFO:teuthology.orchestra.run.smithi115.stdout: "load_cent": 2101, 2024-08-28T19:44:09.627 INFO:teuthology.orchestra.run.smithi115.stdout: "q": 0, 2024-08-28T19:44:09.627 INFO:teuthology.orchestra.run.smithi115.stdout: "exported": 23, 2024-08-28T19:44:09.627 INFO:teuthology.orchestra.run.smithi115.stdout: "exported_inodes": 1443, 2024-08-28T19:44:09.627 INFO:teuthology.orchestra.run.smithi115.stdout: "imported": 23, 2024-08-28T19:44:09.627 INFO:teuthology.orchestra.run.smithi115.stdout: "imported_inodes": 1584, 2024-08-28T19:44:09.627 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_dir_fetch": 0, 2024-08-28T19:44:09.627 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_backtrace_fetch": 1650, 2024-08-28T19:44:09.628 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_peer_discover": 1, 2024-08-28T19:44:09.628 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rfiles": 730, 2024-08-28T19:44:09.628 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rbytes": 212039648, 2024-08-28T19:44:09.628 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rsnaps": 6, 2024-08-28T19:44:09.628 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_backtrace_fetch": 1650, 2024-08-28T19:44:09.628 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_set_tag": 0, 2024-08-28T19:44:09.628 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_backtrace_repaired": 0, 2024-08-28T19:44:09.628 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_inotable_repaired": 0, 2024-08-28T19:44:09.629 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dir_inodes": 170, 2024-08-28T19:44:09.629 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dir_base_inodes": 0, 2024-08-28T19:44:09.629 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dirfrag_rstats": 170, 2024-08-28T19:44:09.629 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_file_inodes": 1480, 2024-08-28T19:44:09.629 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_inode_file_caps": 38, 2024-08-28T19:44:09.629 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_revoke": 18863, 2024-08-28T19:44:09.629 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_grant": 67262, 2024-08-28T19:44:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_trunc": 0, 2024-08-28T19:44:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-28T19:44:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-28T19:44:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_caps": 44098, 2024-08-28T19:44:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_caps_dirty": 12508, 2024-08-28T19:44:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_cap_release": 5376, 2024-08-28T19:44:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: "process_request_cap_release": 76269 2024-08-28T19:44:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.630 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_cache": { 2024-08-28T19:44:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_update": 34, 2024-08-28T19:44:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_update_receipt": 165, 2024-08-28T19:44:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_try_discover": 107, 2024-08-28T19:44:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_send_discover": 195, 2024-08-28T19:44:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_handle_discover": 77, 2024-08-28T19:44:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays": 0, 2024-08-28T19:44:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays_delayed": 0, 2024-08-28T19:44:09.631 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays_enqueuing": 0, 2024-08-28T19:44:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_created": 13595, 2024-08-28T19:44:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_enqueued": 13595, 2024-08-28T19:44:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_reintegrated": 0, 2024-08-28T19:44:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_migrated": 0, 2024-08-28T19:44:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_processing": 0, 2024-08-28T19:44:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_enqueued": 0, 2024-08-28T19:44:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_prioritized": 0, 2024-08-28T19:44:09.632 INFO:teuthology.orchestra.run.smithi115.stdout: "recovery_started": 0, 2024-08-28T19:44:09.633 INFO:teuthology.orchestra.run.smithi115.stdout: "recovery_completed": 0, 2024-08-28T19:44:09.633 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_quiesce_path": 18, 2024-08-28T19:44:09.633 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_quiesce_inode": 1946, 2024-08-28T19:44:09.633 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_enqueue_scrub": 0, 2024-08-28T19:44:09.633 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_exportdir": 26, 2024-08-28T19:44:09.633 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_flush": 0, 2024-08-28T19:44:09.633 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_fragmentdir": 4, 2024-08-28T19:44:09.633 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_fragstats": 0, 2024-08-28T19:44:09.634 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_inodestats": 0, 2024-08-28T19:44:09.634 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_started": 0, 2024-08-28T19:44:09.634 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_succeeded": 0, 2024-08-28T19:44:09.634 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_write_failed": 0 2024-08-28T19:44:09.634 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.634 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_log": { 2024-08-28T19:44:09.634 INFO:teuthology.orchestra.run.smithi115.stdout: "evlrg": 0, 2024-08-28T19:44:09.634 INFO:teuthology.orchestra.run.smithi115.stdout: "evadd": 80039, 2024-08-28T19:44:09.635 INFO:teuthology.orchestra.run.smithi115.stdout: "evex": 0, 2024-08-28T19:44:09.635 INFO:teuthology.orchestra.run.smithi115.stdout: "evtrm": 0, 2024-08-28T19:44:09.635 INFO:teuthology.orchestra.run.smithi115.stdout: "ev": 80039, 2024-08-28T19:44:09.635 INFO:teuthology.orchestra.run.smithi115.stdout: "evexg": 0, 2024-08-28T19:44:09.635 INFO:teuthology.orchestra.run.smithi115.stdout: "evexd": 0, 2024-08-28T19:44:09.635 INFO:teuthology.orchestra.run.smithi115.stdout: "segadd": 113, 2024-08-28T19:44:09.635 INFO:teuthology.orchestra.run.smithi115.stdout: "segex": 0, 2024-08-28T19:44:09.635 INFO:teuthology.orchestra.run.smithi115.stdout: "segtrm": 0, 2024-08-28T19:44:09.635 INFO:teuthology.orchestra.run.smithi115.stdout: "seg": 113, 2024-08-28T19:44:09.636 INFO:teuthology.orchestra.run.smithi115.stdout: "segmjr": 8, 2024-08-28T19:44:09.636 INFO:teuthology.orchestra.run.smithi115.stdout: "segexg": 0, 2024-08-28T19:44:09.636 INFO:teuthology.orchestra.run.smithi115.stdout: "segexd": 0, 2024-08-28T19:44:09.636 INFO:teuthology.orchestra.run.smithi115.stdout: "expos": 4194304, 2024-08-28T19:44:09.636 INFO:teuthology.orchestra.run.smithi115.stdout: "wrpos": 348310330, 2024-08-28T19:44:09.636 INFO:teuthology.orchestra.run.smithi115.stdout: "rdpos": 0, 2024-08-28T19:44:09.636 INFO:teuthology.orchestra.run.smithi115.stdout: "jlat": { 2024-08-28T19:44:09.636 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 24084, 2024-08-28T19:44:09.637 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 309.698986890, 2024-08-28T19:44:09.637 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.012859117 2024-08-28T19:44:09.637 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.637 INFO:teuthology.orchestra.run.smithi115.stdout: "replayed": 0 2024-08-28T19:44:09.637 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.637 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_mem": { 2024-08-28T19:44:09.637 INFO:teuthology.orchestra.run.smithi115.stdout: "ino": 1252, 2024-08-28T19:44:09.637 INFO:teuthology.orchestra.run.smithi115.stdout: "ino+": 16269, 2024-08-28T19:44:09.638 INFO:teuthology.orchestra.run.smithi115.stdout: "ino-": 15017, 2024-08-28T19:44:09.638 INFO:teuthology.orchestra.run.smithi115.stdout: "dir": 178, 2024-08-28T19:44:09.638 INFO:teuthology.orchestra.run.smithi115.stdout: "dir+": 276, 2024-08-28T19:44:09.638 INFO:teuthology.orchestra.run.smithi115.stdout: "dir-": 98, 2024-08-28T19:44:09.638 INFO:teuthology.orchestra.run.smithi115.stdout: "dn": 1461, 2024-08-28T19:44:09.638 INFO:teuthology.orchestra.run.smithi115.stdout: "dn+": 27454, 2024-08-28T19:44:09.638 INFO:teuthology.orchestra.run.smithi115.stdout: "dn-": 25993, 2024-08-28T19:44:09.638 INFO:teuthology.orchestra.run.smithi115.stdout: "cap": 71, 2024-08-28T19:44:09.639 INFO:teuthology.orchestra.run.smithi115.stdout: "cap+": 16386, 2024-08-28T19:44:09.639 INFO:teuthology.orchestra.run.smithi115.stdout: "cap-": 16315, 2024-08-28T19:44:09.639 INFO:teuthology.orchestra.run.smithi115.stdout: "rss": 148868, 2024-08-28T19:44:09.639 INFO:teuthology.orchestra.run.smithi115.stdout: "heap": 359538688 2024-08-28T19:44:09.639 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.639 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_server": { 2024-08-28T19:44:09.639 INFO:teuthology.orchestra.run.smithi115.stdout: "dispatch_client_request": 150290, 2024-08-28T19:44:09.639 INFO:teuthology.orchestra.run.smithi115.stdout: "dispatch_server_request": 1325, 2024-08-28T19:44:09.640 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_request": 102509, 2024-08-28T19:44:09.640 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_session": 2108, 2024-08-28T19:44:09.640 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_peer_request": 7689, 2024-08-28T19:44:09.640 INFO:teuthology.orchestra.run.smithi115.stdout: "req_create_latency": { 2024-08-28T19:44:09.640 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 13562, 2024-08-28T19:44:09.640 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 257.521298454, 2024-08-28T19:44:09.640 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.018988445 2024-08-28T19:44:09.640 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.641 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getattr_latency": { 2024-08-28T19:44:09.641 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 4785, 2024-08-28T19:44:09.641 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 81.512638669, 2024-08-28T19:44:09.641 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.017035034 2024-08-28T19:44:09.641 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.641 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getfilelock_latency": { 2024-08-28T19:44:09.641 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.641 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.641 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.642 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.642 INFO:teuthology.orchestra.run.smithi115.stdout: "req_link_latency": { 2024-08-28T19:44:09.642 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.642 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.642 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.642 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.642 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookup_latency": { 2024-08-28T19:44:09.643 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 25044, 2024-08-28T19:44:09.643 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 89.887754172, 2024-08-28T19:44:09.643 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.003589193 2024-08-28T19:44:09.643 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.643 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookuphash_latency": { 2024-08-28T19:44:09.643 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.643 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.644 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.644 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.644 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupino_latency": { 2024-08-28T19:44:09.644 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 1, 2024-08-28T19:44:09.644 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000122865, 2024-08-28T19:44:09.644 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000122865 2024-08-28T19:44:09.644 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.644 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupname_latency": { 2024-08-28T19:44:09.645 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.645 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.645 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.645 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.645 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupparent_latency": { 2024-08-28T19:44:09.645 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.645 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.646 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.646 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.646 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupsnap_latency": { 2024-08-28T19:44:09.646 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.647 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.647 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.647 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.647 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lssnap_latency": { 2024-08-28T19:44:09.647 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.647 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.647 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.647 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.647 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mkdir_latency": { 2024-08-28T19:44:09.648 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.648 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.648 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.648 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.648 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mknod_latency": { 2024-08-28T19:44:09.648 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.648 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.648 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.649 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.649 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mksnap_latency": { 2024-08-28T19:44:09.649 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.649 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.649 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.649 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.649 INFO:teuthology.orchestra.run.smithi115.stdout: "req_open_latency": { 2024-08-28T19:44:09.649 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 6, 2024-08-28T19:44:09.650 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.011179965, 2024-08-28T19:44:09.650 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.001863327 2024-08-28T19:44:09.650 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.650 INFO:teuthology.orchestra.run.smithi115.stdout: "req_readdir_latency": { 2024-08-28T19:44:09.650 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 38374, 2024-08-28T19:44:09.650 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 171.677196810, 2024-08-28T19:44:09.650 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.004473789 2024-08-28T19:44:09.650 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.651 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rename_latency": { 2024-08-28T19:44:09.651 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 2754, 2024-08-28T19:44:09.651 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 143.448768854, 2024-08-28T19:44:09.651 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.052087425 2024-08-28T19:44:09.651 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.651 INFO:teuthology.orchestra.run.smithi115.stdout: "req_renamesnap_latency": { 2024-08-28T19:44:09.651 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.651 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.651 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.652 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.652 INFO:teuthology.orchestra.run.smithi115.stdout: "req_snapdiff_latency": { 2024-08-28T19:44:09.652 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.652 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.652 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.652 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.652 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmdir_latency": { 2024-08-28T19:44:09.652 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.653 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.653 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.653 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.653 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmsnap_latency": { 2024-08-28T19:44:09.653 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.653 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.653 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.653 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.654 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmxattr_latency": { 2024-08-28T19:44:09.654 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.654 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.654 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.654 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.654 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setattr_latency": { 2024-08-28T19:44:09.654 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 84, 2024-08-28T19:44:09.655 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 41.412129732, 2024-08-28T19:44:09.655 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.493001544 2024-08-28T19:44:09.655 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.655 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setdirlayout_latency": { 2024-08-28T19:44:09.655 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.655 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.655 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.655 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.656 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setfilelock_latency": { 2024-08-28T19:44:09.656 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 736, 2024-08-28T19:44:09.656 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.661068943, 2024-08-28T19:44:09.656 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000898191 2024-08-28T19:44:09.656 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.656 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setlayout_latency": { 2024-08-28T19:44:09.657 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.657 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.657 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.657 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.657 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setxattr_latency": { 2024-08-28T19:44:09.657 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.657 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.657 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.658 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.658 INFO:teuthology.orchestra.run.smithi115.stdout: "req_symlink_latency": { 2024-08-28T19:44:09.658 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.658 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.658 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.658 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.658 INFO:teuthology.orchestra.run.smithi115.stdout: "req_unlink_latency": { 2024-08-28T19:44:09.659 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 13595, 2024-08-28T19:44:09.659 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 102.812258647, 2024-08-28T19:44:09.659 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.007562505 2024-08-28T19:44:09.659 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.659 INFO:teuthology.orchestra.run.smithi115.stdout: "cap_revoke_eviction": 0, 2024-08-28T19:44:09.659 INFO:teuthology.orchestra.run.smithi115.stdout: "cap_acquisition_throttle": 0, 2024-08-28T19:44:09.659 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getvxattr_latency": { 2024-08-28T19:44:09.659 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.660 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.660 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.660 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:09.660 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.660 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_sessions": { 2024-08-28T19:44:09.660 INFO:teuthology.orchestra.run.smithi115.stdout: "session_count": 1, 2024-08-28T19:44:09.661 INFO:teuthology.orchestra.run.smithi115.stdout: "session_add": 2, 2024-08-28T19:44:09.661 INFO:teuthology.orchestra.run.smithi115.stdout: "session_remove": 1, 2024-08-28T19:44:09.661 INFO:teuthology.orchestra.run.smithi115.stdout: "sessions_open": 1, 2024-08-28T19:44:09.661 INFO:teuthology.orchestra.run.smithi115.stdout: "sessions_stale": 0, 2024-08-28T19:44:09.661 INFO:teuthology.orchestra.run.smithi115.stdout: "total_load": 13726, 2024-08-28T19:44:09.661 INFO:teuthology.orchestra.run.smithi115.stdout: "average_load": 13726, 2024-08-28T19:44:09.661 INFO:teuthology.orchestra.run.smithi115.stdout: "avg_session_uptime": 1369, 2024-08-28T19:44:09.661 INFO:teuthology.orchestra.run.smithi115.stdout: "mdthresh_evicted": 0 2024-08-28T19:44:09.662 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.662 INFO:teuthology.orchestra.run.smithi115.stdout: "mempool": { 2024-08-28T19:44:09.662 INFO:teuthology.orchestra.run.smithi115.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:44:09.662 INFO:teuthology.orchestra.run.smithi115.stdout: "bloom_filter_items": 0, 2024-08-28T19:44:09.662 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:44:09.662 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:44:09.662 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:44:09.662 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:44:09.663 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:44:09.663 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:44:09.663 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:44:09.663 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:44:09.663 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:44:09.663 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:44:09.664 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:44:09.664 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:44:09.664 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:44:09.664 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_extent_items": 0, 2024-08-28T19:44:09.664 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:44:09.664 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_blob_items": 0, 2024-08-28T19:44:09.664 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:44:09.664 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:44:09.664 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:44:09.665 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:44:09.665 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:44:09.665 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:44:09.665 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:44:09.665 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_txc_items": 0, 2024-08-28T19:44:09.665 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:44:09.665 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:44:09.665 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:44:09.666 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_items": 0, 2024-08-28T19:44:09.666 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_bytes": 0, 2024-08-28T19:44:09.666 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_items": 0, 2024-08-28T19:44:09.666 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:44:09.666 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:44:09.666 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:44:09.666 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:44:09.666 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_anon_bytes": 455526, 2024-08-28T19:44:09.667 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_anon_items": 957, 2024-08-28T19:44:09.667 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_meta_bytes": 0, 2024-08-28T19:44:09.667 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_meta_items": 0, 2024-08-28T19:44:09.667 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_bytes": 0, 2024-08-28T19:44:09.667 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_items": 0, 2024-08-28T19:44:09.667 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:44:09.667 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_mapbl_items": 0, 2024-08-28T19:44:09.667 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:44:09.668 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_pglog_items": 0, 2024-08-28T19:44:09.668 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_bytes": 9648, 2024-08-28T19:44:09.668 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_items": 196, 2024-08-28T19:44:09.668 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:44:09.668 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:44:09.668 INFO:teuthology.orchestra.run.smithi115.stdout: "pgmap_bytes": 0, 2024-08-28T19:44:09.668 INFO:teuthology.orchestra.run.smithi115.stdout: "pgmap_items": 0, 2024-08-28T19:44:09.668 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_co_bytes": 4718740, 2024-08-28T19:44:09.669 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_co_items": 40003, 2024-08-28T19:44:09.669 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_1_bytes": 0, 2024-08-28T19:44:09.669 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_1_items": 0, 2024-08-28T19:44:09.669 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_2_bytes": 0, 2024-08-28T19:44:09.669 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_2_items": 0 2024-08-28T19:44:09.669 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.669 INFO:teuthology.orchestra.run.smithi115.stdout: "objecter": { 2024-08-28T19:44:09.669 INFO:teuthology.orchestra.run.smithi115.stdout: "op_active": 0, 2024-08-28T19:44:09.669 INFO:teuthology.orchestra.run.smithi115.stdout: "op_laggy": 0, 2024-08-28T19:44:09.670 INFO:teuthology.orchestra.run.smithi115.stdout: "op_send": 55832, 2024-08-28T19:44:09.670 INFO:teuthology.orchestra.run.smithi115.stdout: "op_send_bytes": 347714690, 2024-08-28T19:44:09.670 INFO:teuthology.orchestra.run.smithi115.stdout: "op_resend": 33, 2024-08-28T19:44:09.670 INFO:teuthology.orchestra.run.smithi115.stdout: "op_reply": 55832, 2024-08-28T19:44:09.670 INFO:teuthology.orchestra.run.smithi115.stdout: "op_latency": { 2024-08-28T19:44:09.670 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 55832, 2024-08-28T19:44:09.670 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 766.434928310, 2024-08-28T19:44:09.670 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.013727520 2024-08-28T19:44:09.671 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.671 INFO:teuthology.orchestra.run.smithi115.stdout: "op_inflight": 0, 2024-08-28T19:44:09.671 INFO:teuthology.orchestra.run.smithi115.stdout: "oplen_avg": { 2024-08-28T19:44:09.671 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 55832, 2024-08-28T19:44:09.671 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 57052 2024-08-28T19:44:09.671 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.671 INFO:teuthology.orchestra.run.smithi115.stdout: "op": 55832, 2024-08-28T19:44:09.671 INFO:teuthology.orchestra.run.smithi115.stdout: "op_r": 2430, 2024-08-28T19:44:09.671 INFO:teuthology.orchestra.run.smithi115.stdout: "op_w": 53402, 2024-08-28T19:44:09.672 INFO:teuthology.orchestra.run.smithi115.stdout: "op_rmw": 0, 2024-08-28T19:44:09.672 INFO:teuthology.orchestra.run.smithi115.stdout: "op_pg": 0, 2024-08-28T19:44:09.672 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_stat": 0, 2024-08-28T19:44:09.672 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_create": 10, 2024-08-28T19:44:09.672 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_read": 766, 2024-08-28T19:44:09.672 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_write": 24850, 2024-08-28T19:44:09.672 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_writefull": 465, 2024-08-28T19:44:09.672 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_writesame": 0, 2024-08-28T19:44:09.673 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_append": 0, 2024-08-28T19:44:09.673 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_zero": 0, 2024-08-28T19:44:09.673 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_truncate": 4, 2024-08-28T19:44:09.673 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_delete": 27284, 2024-08-28T19:44:09.673 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_mapext": 0, 2024-08-28T19:44:09.673 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_sparse_read": 0, 2024-08-28T19:44:09.673 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_clonerange": 0, 2024-08-28T19:44:09.673 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_getxattr": 1650, 2024-08-28T19:44:09.674 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_setxattr": 20, 2024-08-28T19:44:09.674 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_cmpxattr": 0, 2024-08-28T19:44:09.674 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_rmxattr": 0, 2024-08-28T19:44:09.674 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_resetxattrs": 0, 2024-08-28T19:44:09.674 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_call": 0, 2024-08-28T19:44:09.674 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_watch": 0, 2024-08-28T19:44:09.674 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_notify": 0, 2024-08-28T19:44:09.674 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_src_cmpxattr": 0, 2024-08-28T19:44:09.675 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_pgls": 0, 2024-08-28T19:44:09.675 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_pgls_filter": 0, 2024-08-28T19:44:09.675 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_other": 0, 2024-08-28T19:44:09.675 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_active": 0, 2024-08-28T19:44:09.675 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_send": 0, 2024-08-28T19:44:09.675 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_resend": 0, 2024-08-28T19:44:09.675 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_ping": 0, 2024-08-28T19:44:09.675 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_active": 0, 2024-08-28T19:44:09.676 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_send": 0, 2024-08-28T19:44:09.676 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_resend": 0, 2024-08-28T19:44:09.676 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_active": 0, 2024-08-28T19:44:09.676 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_send": 0, 2024-08-28T19:44:09.676 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_resend": 0, 2024-08-28T19:44:09.676 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_active": 0, 2024-08-28T19:44:09.676 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_send": 0, 2024-08-28T19:44:09.676 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_resend": 0, 2024-08-28T19:44:09.676 INFO:teuthology.orchestra.run.smithi115.stdout: "command_active": 0, 2024-08-28T19:44:09.677 INFO:teuthology.orchestra.run.smithi115.stdout: "command_send": 0, 2024-08-28T19:44:09.677 INFO:teuthology.orchestra.run.smithi115.stdout: "command_resend": 0, 2024-08-28T19:44:09.677 INFO:teuthology.orchestra.run.smithi115.stdout: "map_epoch": 117, 2024-08-28T19:44:09.677 INFO:teuthology.orchestra.run.smithi115.stdout: "map_full": 0, 2024-08-28T19:44:09.677 INFO:teuthology.orchestra.run.smithi115.stdout: "map_inc": 35, 2024-08-28T19:44:09.677 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_sessions": 12, 2024-08-28T19:44:09.677 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_session_open": 12, 2024-08-28T19:44:09.677 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_session_close": 0, 2024-08-28T19:44:09.678 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_laggy": 0, 2024-08-28T19:44:09.678 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_wr": 1464, 2024-08-28T19:44:09.678 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_rd": 28, 2024-08-28T19:44:09.678 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_del": 511 2024-08-28T19:44:09.678 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.678 INFO:teuthology.orchestra.run.smithi115.stdout: "oft": { 2024-08-28T19:44:09.678 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_objs": 1, 2024-08-28T19:44:09.678 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_kv_pairs": 0, 2024-08-28T19:44:09.679 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_updates": 0, 2024-08-28T19:44:09.679 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_removes": 0 2024-08-28T19:44:09.679 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.679 INFO:teuthology.orchestra.run.smithi115.stdout: "purge_queue": { 2024-08-28T19:44:09.679 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_ops": 0, 2024-08-28T19:44:09.679 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_ops_high_water": 10, 2024-08-28T19:44:09.679 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing": 0, 2024-08-28T19:44:09.679 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_high_water": 5, 2024-08-28T19:44:09.680 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executed_ops": 27190, 2024-08-28T19:44:09.680 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executed": 13595, 2024-08-28T19:44:09.680 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_item_in_journal": 0 2024-08-28T19:44:09.680 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.680 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:44:09.680 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:09.680 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 104857600, 2024-08-28T19:44:09.680 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:09.681 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 425859, 2024-08-28T19:44:09.681 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 106212234, 2024-08-28T19:44:09.681 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:09.681 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 425859, 2024-08-28T19:44:09.681 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:09.681 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:09.681 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 425859, 2024-08-28T19:44:09.681 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 106212234, 2024-08-28T19:44:09.682 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:09.682 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.682 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.682 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.682 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:09.682 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.682 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-objecter_bytes": { 2024-08-28T19:44:09.682 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:09.682 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 104857600, 2024-08-28T19:44:09.683 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:09.683 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 0, 2024-08-28T19:44:09.683 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 0, 2024-08-28T19:44:09.683 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:09.683 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:09.683 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 55832, 2024-08-28T19:44:09.683 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 349198817, 2024-08-28T19:44:09.683 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 55832, 2024-08-28T19:44:09.684 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 349198817, 2024-08-28T19:44:09.684 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:09.684 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.684 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.684 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.684 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:09.684 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.684 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-objecter_ops": { 2024-08-28T19:44:09.685 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:09.685 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 1024, 2024-08-28T19:44:09.685 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:09.685 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 0, 2024-08-28T19:44:09.685 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 0, 2024-08-28T19:44:09.685 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:09.685 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:09.685 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 55832, 2024-08-28T19:44:09.686 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 55832, 2024-08-28T19:44:09.686 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 55832, 2024-08-28T19:44:09.686 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 55832, 2024-08-28T19:44:09.686 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:09.686 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.686 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.686 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.686 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:09.687 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.687 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-write_buf_throttle": { 2024-08-28T19:44:09.687 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:09.687 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 3758096384, 2024-08-28T19:44:09.687 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:09.687 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 13595, 2024-08-28T19:44:09.687 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 1484351, 2024-08-28T19:44:09.687 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:09.687 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 13595, 2024-08-28T19:44:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 766, 2024-08-28T19:44:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 1484351, 2024-08-28T19:44:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.688 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-write_buf_throttle-0x55e6926f6820": { 2024-08-28T19:44:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 16762, 2024-08-28T19:44:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 3758096384, 2024-08-28T19:44:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 80039, 2024-08-28T19:44:09.689 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 344116026, 2024-08-28T19:44:09.690 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:09.690 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 80039, 2024-08-28T19:44:09.690 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:09.690 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:09.690 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 24084, 2024-08-28T19:44:09.690 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 344099264, 2024-08-28T19:44:09.690 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:09.690 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:09.691 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:09.691 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:09.691 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:09.691 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:09.691 INFO:teuthology.orchestra.run.smithi115.stdout: "trackedop": { 2024-08-28T19:44:09.691 INFO:teuthology.orchestra.run.smithi115.stdout: "slow_ops_count": 2075 2024-08-28T19:44:09.691 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:09.691 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-08-28T19:44:10.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:10 smithi115 ceph-mon[22184]: osdmap e117: 12 total, 12 up, 12 in 2024-08-28T19:44:10.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:44:10.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:44:10.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:10 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:44:10.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:10 smithi016 ceph-mon[18556]: osdmap e117: 12 total, 12 up, 12 in 2024-08-28T19:44:10.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:44:10.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:44:10.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:10 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:44:10.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:10 smithi028 ceph-mon[21769]: osdmap e117: 12 total, 12 up, 12 in 2024-08-28T19:44:10.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-28T19:44:10.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-28T19:44:10.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:10 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' 2024-08-28T19:44:10.613 INFO:tasks.check_counter:Daemon mds.f mds.exported=23 2024-08-28T19:44:10.613 INFO:tasks.check_counter:Daemon mds.f mds.imported=23 2024-08-28T19:44:10.613 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-08-28T19:44:10.613 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-08-28T19:44:10.613 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-08-28T19:44:10.613 DEBUG:tasks.check_counter:Getting stats from i 2024-08-28T19:44:10.614 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.i perf dump 2024-08-28T19:44:10.868 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:44:11.414 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-08-28T19:44:11.414 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:44:11.414 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 82693, 2024-08-28T19:44:11.414 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 89275, 2024-08-28T19:44:11.415 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 31370920, 2024-08-28T19:44:11.415 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 110603184, 2024-08-28T19:44:11.415 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 11, 2024-08-28T19:44:11.415 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 4, 2024-08-28T19:44:11.415 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 8.716158428, 2024-08-28T19:44:11.415 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 4.540986163, 2024-08-28T19:44:11.415 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 3.752158572, 2024-08-28T19:44:11.416 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 0.860413846, 2024-08-28T19:44:11.416 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:11.416 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 89276, 2024-08-28T19:44:11.416 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 14.675428353, 2024-08-28T19:44:11.416 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000164382 2024-08-28T19:44:11.416 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.416 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:11.416 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 129465, 2024-08-28T19:44:11.417 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.028387133, 2024-08-28T19:44:11.417 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000219 2024-08-28T19:44:11.417 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.417 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 1517920, 2024-08-28T19:44:11.417 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 295312 2024-08-28T19:44:11.417 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.417 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:44:11.418 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 148379, 2024-08-28T19:44:11.418 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 144604, 2024-08-28T19:44:11.418 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 49261400, 2024-08-28T19:44:11.418 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 183307108, 2024-08-28T19:44:11.418 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 11, 2024-08-28T19:44:11.418 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 7, 2024-08-28T19:44:11.418 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 12.761971827, 2024-08-28T19:44:11.418 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 6.058709933, 2024-08-28T19:44:11.419 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 6.091639071, 2024-08-28T19:44:11.419 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 1.392929850, 2024-08-28T19:44:11.419 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:11.419 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 144603, 2024-08-28T19:44:11.419 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 16.911540904, 2024-08-28T19:44:11.419 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000116951 2024-08-28T19:44:11.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.419 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:11.420 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 239281, 2024-08-28T19:44:11.420 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.048912387, 2024-08-28T19:44:11.420 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000204 2024-08-28T19:44:11.420 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.420 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-08-28T19:44:11.420 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 733472 2024-08-28T19:44:11.421 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.421 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:44:11.421 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 219660, 2024-08-28T19:44:11.421 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 325898, 2024-08-28T19:44:11.421 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 68240953, 2024-08-28T19:44:11.421 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 430202444, 2024-08-28T19:44:11.424 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 72, 2024-08-28T19:44:11.424 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 11, 2024-08-28T19:44:11.425 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 23.350980659, 2024-08-28T19:44:11.425 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 11.572472667, 2024-08-28T19:44:11.425 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 9.975594269, 2024-08-28T19:44:11.425 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 2.141775527, 2024-08-28T19:44:11.425 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:11.425 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 325894, 2024-08-28T19:44:11.425 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 44.617608827, 2024-08-28T19:44:11.425 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000136908 2024-08-28T19:44:11.426 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.426 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:11.426 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 433646, 2024-08-28T19:44:11.426 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.108393661, 2024-08-28T19:44:11.426 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000249 2024-08-28T19:44:11.426 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.426 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 928, 2024-08-28T19:44:11.426 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 736 2024-08-28T19:44:11.426 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.427 INFO:teuthology.orchestra.run.smithi115.stdout: "cct": { 2024-08-28T19:44:11.427 INFO:teuthology.orchestra.run.smithi115.stdout: "total_workers": 1, 2024-08-28T19:44:11.427 INFO:teuthology.orchestra.run.smithi115.stdout: "unhealthy_workers": 0 2024-08-28T19:44:11.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.427 INFO:teuthology.orchestra.run.smithi115.stdout: "finisher-MDSRank": { 2024-08-28T19:44:11.427 INFO:teuthology.orchestra.run.smithi115.stdout: "queue_len": 0, 2024-08-28T19:44:11.427 INFO:teuthology.orchestra.run.smithi115.stdout: "complete_latency": { 2024-08-28T19:44:11.427 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 49189, 2024-08-28T19:44:11.428 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 75.948995197, 2024-08-28T19:44:11.428 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.001544023 2024-08-28T19:44:11.428 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:11.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.428 INFO:teuthology.orchestra.run.smithi115.stdout: "finisher-PurgeQueue": { 2024-08-28T19:44:11.428 INFO:teuthology.orchestra.run.smithi115.stdout: "queue_len": 0, 2024-08-28T19:44:11.428 INFO:teuthology.orchestra.run.smithi115.stdout: "complete_latency": { 2024-08-28T19:44:11.428 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 36118, 2024-08-28T19:44:11.429 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 4.860493800, 2024-08-28T19:44:11.429 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000134572 2024-08-28T19:44:11.429 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:11.429 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.429 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2024-08-28T19:44:11.429 INFO:teuthology.orchestra.run.smithi115.stdout: "request": 136904, 2024-08-28T19:44:11.429 INFO:teuthology.orchestra.run.smithi115.stdout: "reply": 130543, 2024-08-28T19:44:11.429 INFO:teuthology.orchestra.run.smithi115.stdout: "reply_latency": { 2024-08-28T19:44:11.430 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 130543, 2024-08-28T19:44:11.430 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 884.668463518, 2024-08-28T19:44:11.430 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.006776835 2024-08-28T19:44:11.430 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.430 INFO:teuthology.orchestra.run.smithi115.stdout: "slow_reply": 0, 2024-08-28T19:44:11.430 INFO:teuthology.orchestra.run.smithi115.stdout: "forward": 6361, 2024-08-28T19:44:11.430 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_fetch_complete": 3, 2024-08-28T19:44:11.430 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_fetch_keys": 0, 2024-08-28T19:44:11.431 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_commit": 74, 2024-08-28T19:44:11.431 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_split": 7, 2024-08-28T19:44:11.431 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_merge": 3, 2024-08-28T19:44:11.431 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes": 1868, 2024-08-28T19:44:11.431 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_top": 543, 2024-08-28T19:44:11.431 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_bottom": 1232, 2024-08-28T19:44:11.431 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_pin_tail": 93, 2024-08-28T19:44:11.431 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_pinned": 1012, 2024-08-28T19:44:11.431 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_expired": 929, 2024-08-28T19:44:11.432 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_with_caps": 98, 2024-08-28T19:44:11.432 INFO:teuthology.orchestra.run.smithi115.stdout: "caps": 98, 2024-08-28T19:44:11.432 INFO:teuthology.orchestra.run.smithi115.stdout: "subtrees": 21, 2024-08-28T19:44:11.432 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse": 213586, 2024-08-28T19:44:11.432 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_hit": 119889, 2024-08-28T19:44:11.432 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_forward": 6355, 2024-08-28T19:44:11.432 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_discover": 63, 2024-08-28T19:44:11.432 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_dir_fetch": 0, 2024-08-28T19:44:11.432 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_remote_ino": 0, 2024-08-28T19:44:11.433 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_lock": 1590, 2024-08-28T19:44:11.433 INFO:teuthology.orchestra.run.smithi115.stdout: "load_cent": 202, 2024-08-28T19:44:11.433 INFO:teuthology.orchestra.run.smithi115.stdout: "q": 0, 2024-08-28T19:44:11.433 INFO:teuthology.orchestra.run.smithi115.stdout: "exported": 11, 2024-08-28T19:44:11.433 INFO:teuthology.orchestra.run.smithi115.stdout: "exported_inodes": 783, 2024-08-28T19:44:11.433 INFO:teuthology.orchestra.run.smithi115.stdout: "imported": 17, 2024-08-28T19:44:11.433 INFO:teuthology.orchestra.run.smithi115.stdout: "imported_inodes": 1061, 2024-08-28T19:44:11.433 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_dir_fetch": 0, 2024-08-28T19:44:11.434 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_backtrace_fetch": 1804, 2024-08-28T19:44:11.434 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_peer_discover": 0, 2024-08-28T19:44:11.434 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rfiles": 730, 2024-08-28T19:44:11.434 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rbytes": 212039648, 2024-08-28T19:44:11.434 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rsnaps": 6, 2024-08-28T19:44:11.434 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_backtrace_fetch": 1804, 2024-08-28T19:44:11.434 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_set_tag": 0, 2024-08-28T19:44:11.434 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_backtrace_repaired": 0, 2024-08-28T19:44:11.434 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_inotable_repaired": 0, 2024-08-28T19:44:11.435 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dir_inodes": 200, 2024-08-28T19:44:11.435 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dir_base_inodes": 0, 2024-08-28T19:44:11.435 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dirfrag_rstats": 200, 2024-08-28T19:44:11.435 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_file_inodes": 1604, 2024-08-28T19:44:11.435 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_inode_file_caps": 12, 2024-08-28T19:44:11.435 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_revoke": 18319, 2024-08-28T19:44:11.435 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_grant": 84760, 2024-08-28T19:44:11.435 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_trunc": 0, 2024-08-28T19:44:11.436 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-28T19:44:11.436 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-28T19:44:11.436 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_caps": 42237, 2024-08-28T19:44:11.436 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_caps_dirty": 11114, 2024-08-28T19:44:11.436 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_cap_release": 5584, 2024-08-28T19:44:11.436 INFO:teuthology.orchestra.run.smithi115.stdout: "process_request_cap_release": 99136 2024-08-28T19:44:11.436 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.436 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_cache": { 2024-08-28T19:44:11.437 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_update": 114, 2024-08-28T19:44:11.437 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_update_receipt": 95, 2024-08-28T19:44:11.437 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_try_discover": 65, 2024-08-28T19:44:11.437 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_send_discover": 120, 2024-08-28T19:44:11.437 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_handle_discover": 194, 2024-08-28T19:44:11.437 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays": 0, 2024-08-28T19:44:11.437 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays_delayed": 0, 2024-08-28T19:44:11.437 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays_enqueuing": 0, 2024-08-28T19:44:11.438 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_created": 18925, 2024-08-28T19:44:11.438 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_enqueued": 18925, 2024-08-28T19:44:11.438 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_reintegrated": 0, 2024-08-28T19:44:11.438 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_migrated": 0, 2024-08-28T19:44:11.438 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_processing": 0, 2024-08-28T19:44:11.438 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_enqueued": 0, 2024-08-28T19:44:11.438 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_prioritized": 0, 2024-08-28T19:44:11.438 INFO:teuthology.orchestra.run.smithi115.stdout: "recovery_started": 0, 2024-08-28T19:44:11.439 INFO:teuthology.orchestra.run.smithi115.stdout: "recovery_completed": 0, 2024-08-28T19:44:11.439 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_quiesce_path": 18, 2024-08-28T19:44:11.439 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_quiesce_inode": 1742, 2024-08-28T19:44:11.439 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_enqueue_scrub": 0, 2024-08-28T19:44:11.439 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_exportdir": 18, 2024-08-28T19:44:11.439 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_flush": 0, 2024-08-28T19:44:11.439 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_fragmentdir": 10, 2024-08-28T19:44:11.439 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_fragstats": 0, 2024-08-28T19:44:11.440 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_inodestats": 0, 2024-08-28T19:44:11.440 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_started": 0, 2024-08-28T19:44:11.440 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_succeeded": 0, 2024-08-28T19:44:11.440 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_write_failed": 0 2024-08-28T19:44:11.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.440 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_log": { 2024-08-28T19:44:11.440 INFO:teuthology.orchestra.run.smithi115.stdout: "evlrg": 0, 2024-08-28T19:44:11.440 INFO:teuthology.orchestra.run.smithi115.stdout: "evadd": 96831, 2024-08-28T19:44:11.441 INFO:teuthology.orchestra.run.smithi115.stdout: "evex": 6038, 2024-08-28T19:44:11.441 INFO:teuthology.orchestra.run.smithi115.stdout: "evtrm": 1, 2024-08-28T19:44:11.441 INFO:teuthology.orchestra.run.smithi115.stdout: "ev": 96830, 2024-08-28T19:44:11.441 INFO:teuthology.orchestra.run.smithi115.stdout: "evexg": 0, 2024-08-28T19:44:11.441 INFO:teuthology.orchestra.run.smithi115.stdout: "evexd": 6037, 2024-08-28T19:44:11.441 INFO:teuthology.orchestra.run.smithi115.stdout: "segadd": 135, 2024-08-28T19:44:11.441 INFO:teuthology.orchestra.run.smithi115.stdout: "segex": 7, 2024-08-28T19:44:11.441 INFO:teuthology.orchestra.run.smithi115.stdout: "segtrm": 1, 2024-08-28T19:44:11.442 INFO:teuthology.orchestra.run.smithi115.stdout: "seg": 134, 2024-08-28T19:44:11.442 INFO:teuthology.orchestra.run.smithi115.stdout: "segmjr": 8, 2024-08-28T19:44:11.442 INFO:teuthology.orchestra.run.smithi115.stdout: "segexg": 0, 2024-08-28T19:44:11.442 INFO:teuthology.orchestra.run.smithi115.stdout: "segexd": 6, 2024-08-28T19:44:11.442 INFO:teuthology.orchestra.run.smithi115.stdout: "expos": 4194352, 2024-08-28T19:44:11.442 INFO:teuthology.orchestra.run.smithi115.stdout: "wrpos": 398855380, 2024-08-28T19:44:11.442 INFO:teuthology.orchestra.run.smithi115.stdout: "rdpos": 0, 2024-08-28T19:44:11.442 INFO:teuthology.orchestra.run.smithi115.stdout: "jlat": { 2024-08-28T19:44:11.442 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 27083, 2024-08-28T19:44:11.443 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 317.296609680, 2024-08-28T19:44:11.443 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.011715711 2024-08-28T19:44:11.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.443 INFO:teuthology.orchestra.run.smithi115.stdout: "replayed": 0 2024-08-28T19:44:11.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.443 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_mem": { 2024-08-28T19:44:11.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ino": 1339, 2024-08-28T19:44:11.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ino+": 20890, 2024-08-28T19:44:11.443 INFO:teuthology.orchestra.run.smithi115.stdout: "ino-": 19551, 2024-08-28T19:44:11.444 INFO:teuthology.orchestra.run.smithi115.stdout: "dir": 166, 2024-08-28T19:44:11.444 INFO:teuthology.orchestra.run.smithi115.stdout: "dir+": 277, 2024-08-28T19:44:11.444 INFO:teuthology.orchestra.run.smithi115.stdout: "dir-": 111, 2024-08-28T19:44:11.444 INFO:teuthology.orchestra.run.smithi115.stdout: "dn": 1868, 2024-08-28T19:44:11.444 INFO:teuthology.orchestra.run.smithi115.stdout: "dn+": 38300, 2024-08-28T19:44:11.444 INFO:teuthology.orchestra.run.smithi115.stdout: "dn-": 36432, 2024-08-28T19:44:11.444 INFO:teuthology.orchestra.run.smithi115.stdout: "cap": 98, 2024-08-28T19:44:11.444 INFO:teuthology.orchestra.run.smithi115.stdout: "cap+": 21736, 2024-08-28T19:44:11.444 INFO:teuthology.orchestra.run.smithi115.stdout: "cap-": 21638, 2024-08-28T19:44:11.445 INFO:teuthology.orchestra.run.smithi115.stdout: "rss": 151284, 2024-08-28T19:44:11.445 INFO:teuthology.orchestra.run.smithi115.stdout: "heap": 357445632 2024-08-28T19:44:11.445 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.445 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_server": { 2024-08-28T19:44:11.445 INFO:teuthology.orchestra.run.smithi115.stdout: "dispatch_client_request": 192540, 2024-08-28T19:44:11.445 INFO:teuthology.orchestra.run.smithi115.stdout: "dispatch_server_request": 1224, 2024-08-28T19:44:11.445 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_request": 136904, 2024-08-28T19:44:11.445 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_session": 1701, 2024-08-28T19:44:11.446 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_peer_request": 8710, 2024-08-28T19:44:11.446 INFO:teuthology.orchestra.run.smithi115.stdout: "req_create_latency": { 2024-08-28T19:44:11.446 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 18905, 2024-08-28T19:44:11.446 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 331.022664135, 2024-08-28T19:44:11.446 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.017509794 2024-08-28T19:44:11.446 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.446 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getattr_latency": { 2024-08-28T19:44:11.446 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 3194, 2024-08-28T19:44:11.446 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 54.587056738, 2024-08-28T19:44:11.447 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.017090499 2024-08-28T19:44:11.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.447 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getfilelock_latency": { 2024-08-28T19:44:11.447 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.447 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.447 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.447 INFO:teuthology.orchestra.run.smithi115.stdout: "req_link_latency": { 2024-08-28T19:44:11.447 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.448 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.448 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.448 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookup_latency": { 2024-08-28T19:44:11.448 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 35550, 2024-08-28T19:44:11.448 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 79.719563230, 2024-08-28T19:44:11.448 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.002242463 2024-08-28T19:44:11.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.448 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookuphash_latency": { 2024-08-28T19:44:11.449 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.449 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.449 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.449 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.449 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupino_latency": { 2024-08-28T19:44:11.449 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.449 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.449 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.450 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.450 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupname_latency": { 2024-08-28T19:44:11.450 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.450 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.450 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.450 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.450 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupparent_latency": { 2024-08-28T19:44:11.451 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.451 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.452 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.452 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.452 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupsnap_latency": { 2024-08-28T19:44:11.452 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.452 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.452 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.452 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.453 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lssnap_latency": { 2024-08-28T19:44:11.453 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.453 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.453 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.453 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.453 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mkdir_latency": { 2024-08-28T19:44:11.453 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.454 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.454 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.454 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.454 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mknod_latency": { 2024-08-28T19:44:11.454 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.454 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.454 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.454 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.455 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mksnap_latency": { 2024-08-28T19:44:11.455 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.455 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.455 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.455 INFO:teuthology.orchestra.run.smithi115.stdout: "req_open_latency": { 2024-08-28T19:44:11.455 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.456 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.456 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.456 INFO:teuthology.orchestra.run.smithi115.stdout: "req_readdir_latency": { 2024-08-28T19:44:11.456 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 50446, 2024-08-28T19:44:11.456 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 228.814898768, 2024-08-28T19:44:11.457 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.004535838 2024-08-28T19:44:11.457 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.457 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rename_latency": { 2024-08-28T19:44:11.457 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 3016, 2024-08-28T19:44:11.457 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 43.360014496, 2024-08-28T19:44:11.457 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.014376662 2024-08-28T19:44:11.458 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.458 INFO:teuthology.orchestra.run.smithi115.stdout: "req_renamesnap_latency": { 2024-08-28T19:44:11.458 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.458 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.458 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.458 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.458 INFO:teuthology.orchestra.run.smithi115.stdout: "req_snapdiff_latency": { 2024-08-28T19:44:11.458 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.459 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.459 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.459 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmdir_latency": { 2024-08-28T19:44:11.459 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.459 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.459 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.460 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmsnap_latency": { 2024-08-28T19:44:11.460 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.460 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.460 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.460 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmxattr_latency": { 2024-08-28T19:44:11.460 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.460 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.461 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.461 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.461 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setattr_latency": { 2024-08-28T19:44:11.461 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 39, 2024-08-28T19:44:11.461 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.773355016, 2024-08-28T19:44:11.461 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.019829615 2024-08-28T19:44:11.461 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.461 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setdirlayout_latency": { 2024-08-28T19:44:11.462 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.462 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.462 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.462 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.462 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setfilelock_latency": { 2024-08-28T19:44:11.462 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 468, 2024-08-28T19:44:11.462 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.826128002, 2024-08-28T19:44:11.462 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.001765230 2024-08-28T19:44:11.463 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.463 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setlayout_latency": { 2024-08-28T19:44:11.463 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.463 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.463 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.463 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.463 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setxattr_latency": { 2024-08-28T19:44:11.464 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.464 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.464 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.464 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.464 INFO:teuthology.orchestra.run.smithi115.stdout: "req_symlink_latency": { 2024-08-28T19:44:11.464 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.464 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.464 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.465 INFO:teuthology.orchestra.run.smithi115.stdout: "req_unlink_latency": { 2024-08-28T19:44:11.465 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 18925, 2024-08-28T19:44:11.465 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 145.564783133, 2024-08-28T19:44:11.465 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.007691666 2024-08-28T19:44:11.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.465 INFO:teuthology.orchestra.run.smithi115.stdout: "cap_revoke_eviction": 0, 2024-08-28T19:44:11.465 INFO:teuthology.orchestra.run.smithi115.stdout: "cap_acquisition_throttle": 0, 2024-08-28T19:44:11.465 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getvxattr_latency": { 2024-08-28T19:44:11.466 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.466 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.466 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.466 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:11.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.466 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_sessions": { 2024-08-28T19:44:11.466 INFO:teuthology.orchestra.run.smithi115.stdout: "session_count": 1, 2024-08-28T19:44:11.466 INFO:teuthology.orchestra.run.smithi115.stdout: "session_add": 2, 2024-08-28T19:44:11.467 INFO:teuthology.orchestra.run.smithi115.stdout: "session_remove": 1, 2024-08-28T19:44:11.467 INFO:teuthology.orchestra.run.smithi115.stdout: "sessions_open": 1, 2024-08-28T19:44:11.467 INFO:teuthology.orchestra.run.smithi115.stdout: "sessions_stale": 0, 2024-08-28T19:44:11.467 INFO:teuthology.orchestra.run.smithi115.stdout: "total_load": 15050, 2024-08-28T19:44:11.467 INFO:teuthology.orchestra.run.smithi115.stdout: "average_load": 15050, 2024-08-28T19:44:11.467 INFO:teuthology.orchestra.run.smithi115.stdout: "avg_session_uptime": 1306, 2024-08-28T19:44:11.467 INFO:teuthology.orchestra.run.smithi115.stdout: "mdthresh_evicted": 0 2024-08-28T19:44:11.467 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.467 INFO:teuthology.orchestra.run.smithi115.stdout: "mempool": { 2024-08-28T19:44:11.468 INFO:teuthology.orchestra.run.smithi115.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:44:11.468 INFO:teuthology.orchestra.run.smithi115.stdout: "bloom_filter_items": 0, 2024-08-28T19:44:11.468 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:44:11.468 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:44:11.468 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:44:11.468 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:44:11.468 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:44:11.469 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:44:11.469 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:44:11.470 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:44:11.470 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:44:11.470 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:44:11.470 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:44:11.470 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:44:11.470 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:44:11.470 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_extent_items": 0, 2024-08-28T19:44:11.470 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:44:11.471 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_blob_items": 0, 2024-08-28T19:44:11.471 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:44:11.471 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:44:11.471 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:44:11.471 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:44:11.471 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:44:11.471 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:44:11.471 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:44:11.472 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_txc_items": 0, 2024-08-28T19:44:11.472 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:44:11.472 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:44:11.472 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:44:11.472 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_items": 0, 2024-08-28T19:44:11.472 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_bytes": 0, 2024-08-28T19:44:11.472 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_items": 0, 2024-08-28T19:44:11.472 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:44:11.473 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:44:11.473 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:44:11.473 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:44:11.473 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_anon_bytes": 330691, 2024-08-28T19:44:11.473 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_anon_items": 803, 2024-08-28T19:44:11.473 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_meta_bytes": 176, 2024-08-28T19:44:11.473 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_meta_items": 2, 2024-08-28T19:44:11.473 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_bytes": 0, 2024-08-28T19:44:11.474 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_items": 0, 2024-08-28T19:44:11.474 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:44:11.474 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_mapbl_items": 0, 2024-08-28T19:44:11.474 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:44:11.474 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_pglog_items": 0, 2024-08-28T19:44:11.474 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_bytes": 9888, 2024-08-28T19:44:11.474 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_items": 202, 2024-08-28T19:44:11.474 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:44:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:44:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "pgmap_bytes": 0, 2024-08-28T19:44:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "pgmap_items": 0, 2024-08-28T19:44:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_co_bytes": 5146118, 2024-08-28T19:44:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_co_items": 42591, 2024-08-28T19:44:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_1_bytes": 0, 2024-08-28T19:44:11.475 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_1_items": 0, 2024-08-28T19:44:11.476 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_2_bytes": 0, 2024-08-28T19:44:11.476 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_2_items": 0 2024-08-28T19:44:11.476 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.476 INFO:teuthology.orchestra.run.smithi115.stdout: "objecter": { 2024-08-28T19:44:11.476 INFO:teuthology.orchestra.run.smithi115.stdout: "op_active": 0, 2024-08-28T19:44:11.476 INFO:teuthology.orchestra.run.smithi115.stdout: "op_laggy": 0, 2024-08-28T19:44:11.476 INFO:teuthology.orchestra.run.smithi115.stdout: "op_send": 69713, 2024-08-28T19:44:11.476 INFO:teuthology.orchestra.run.smithi115.stdout: "op_send_bytes": 399514552, 2024-08-28T19:44:11.477 INFO:teuthology.orchestra.run.smithi115.stdout: "op_resend": 33, 2024-08-28T19:44:11.477 INFO:teuthology.orchestra.run.smithi115.stdout: "op_reply": 69713, 2024-08-28T19:44:11.477 INFO:teuthology.orchestra.run.smithi115.stdout: "op_latency": { 2024-08-28T19:44:11.477 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 69713, 2024-08-28T19:44:11.477 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 825.868722485, 2024-08-28T19:44:11.477 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.011846696 2024-08-28T19:44:11.477 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.477 INFO:teuthology.orchestra.run.smithi115.stdout: "op_inflight": 0, 2024-08-28T19:44:11.478 INFO:teuthology.orchestra.run.smithi115.stdout: "oplen_avg": { 2024-08-28T19:44:11.478 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 69713, 2024-08-28T19:44:11.478 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 70909 2024-08-28T19:44:11.478 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.478 INFO:teuthology.orchestra.run.smithi115.stdout: "op": 69713, 2024-08-28T19:44:11.478 INFO:teuthology.orchestra.run.smithi115.stdout: "op_r": 2597, 2024-08-28T19:44:11.478 INFO:teuthology.orchestra.run.smithi115.stdout: "op_w": 67116, 2024-08-28T19:44:11.478 INFO:teuthology.orchestra.run.smithi115.stdout: "op_rmw": 0, 2024-08-28T19:44:11.478 INFO:teuthology.orchestra.run.smithi115.stdout: "op_pg": 0, 2024-08-28T19:44:11.479 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_stat": 0, 2024-08-28T19:44:11.479 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_create": 17, 2024-08-28T19:44:11.479 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_read": 790, 2024-08-28T19:44:11.479 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_write": 27873, 2024-08-28T19:44:11.479 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_writefull": 468, 2024-08-28T19:44:11.479 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_writesame": 0, 2024-08-28T19:44:11.479 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_append": 0, 2024-08-28T19:44:11.479 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_zero": 0, 2024-08-28T19:44:11.480 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_truncate": 7, 2024-08-28T19:44:11.480 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_delete": 37980, 2024-08-28T19:44:11.480 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_mapext": 0, 2024-08-28T19:44:11.480 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_sparse_read": 0, 2024-08-28T19:44:11.480 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_clonerange": 0, 2024-08-28T19:44:11.480 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_getxattr": 1804, 2024-08-28T19:44:11.480 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_setxattr": 34, 2024-08-28T19:44:11.480 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_cmpxattr": 0, 2024-08-28T19:44:11.481 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_rmxattr": 0, 2024-08-28T19:44:11.481 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_resetxattrs": 0, 2024-08-28T19:44:11.481 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_call": 0, 2024-08-28T19:44:11.481 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_watch": 0, 2024-08-28T19:44:11.481 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_notify": 0, 2024-08-28T19:44:11.481 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_src_cmpxattr": 0, 2024-08-28T19:44:11.481 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_pgls": 0, 2024-08-28T19:44:11.481 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_pgls_filter": 0, 2024-08-28T19:44:11.482 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_other": 0, 2024-08-28T19:44:11.482 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_active": 0, 2024-08-28T19:44:11.482 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_send": 0, 2024-08-28T19:44:11.482 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_resend": 0, 2024-08-28T19:44:11.482 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_ping": 0, 2024-08-28T19:44:11.482 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_active": 0, 2024-08-28T19:44:11.482 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_send": 0, 2024-08-28T19:44:11.482 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_resend": 0, 2024-08-28T19:44:11.483 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_active": 0, 2024-08-28T19:44:11.483 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_send": 0, 2024-08-28T19:44:11.483 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_resend": 0, 2024-08-28T19:44:11.483 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_active": 0, 2024-08-28T19:44:11.483 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_send": 0, 2024-08-28T19:44:11.483 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_resend": 0, 2024-08-28T19:44:11.483 INFO:teuthology.orchestra.run.smithi115.stdout: "command_active": 0, 2024-08-28T19:44:11.484 INFO:teuthology.orchestra.run.smithi115.stdout: "command_send": 0, 2024-08-28T19:44:11.484 INFO:teuthology.orchestra.run.smithi115.stdout: "command_resend": 0, 2024-08-28T19:44:11.484 INFO:teuthology.orchestra.run.smithi115.stdout: "map_epoch": 117, 2024-08-28T19:44:11.484 INFO:teuthology.orchestra.run.smithi115.stdout: "map_full": 0, 2024-08-28T19:44:11.484 INFO:teuthology.orchestra.run.smithi115.stdout: "map_inc": 35, 2024-08-28T19:44:11.484 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_sessions": 12, 2024-08-28T19:44:11.484 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_session_open": 12, 2024-08-28T19:44:11.484 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_session_close": 0, 2024-08-28T19:44:11.485 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_laggy": 0, 2024-08-28T19:44:11.485 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_wr": 1425, 2024-08-28T19:44:11.485 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_rd": 6, 2024-08-28T19:44:11.485 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_del": 505 2024-08-28T19:44:11.485 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.485 INFO:teuthology.orchestra.run.smithi115.stdout: "oft": { 2024-08-28T19:44:11.485 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_objs": 1, 2024-08-28T19:44:11.485 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_kv_pairs": 0, 2024-08-28T19:44:11.486 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_updates": 0, 2024-08-28T19:44:11.486 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_removes": 0 2024-08-28T19:44:11.486 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.486 INFO:teuthology.orchestra.run.smithi115.stdout: "purge_queue": { 2024-08-28T19:44:11.486 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_ops": 0, 2024-08-28T19:44:11.486 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_ops_high_water": 10, 2024-08-28T19:44:11.486 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing": 0, 2024-08-28T19:44:11.486 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_high_water": 5, 2024-08-28T19:44:11.487 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executed_ops": 37850, 2024-08-28T19:44:11.487 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executed": 18925, 2024-08-28T19:44:11.487 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_item_in_journal": 0 2024-08-28T19:44:11.487 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.487 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:44:11.487 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:11.487 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 104857600, 2024-08-28T19:44:11.488 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:11.488 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 450728, 2024-08-28T19:44:11.488 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 108273343, 2024-08-28T19:44:11.488 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:11.488 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 450728, 2024-08-28T19:44:11.488 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:11.488 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:11.488 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 450728, 2024-08-28T19:44:11.488 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 108273343, 2024-08-28T19:44:11.489 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:11.489 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.489 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.489 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.489 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:11.489 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.489 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-objecter_bytes": { 2024-08-28T19:44:11.489 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:11.490 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 104857600, 2024-08-28T19:44:11.490 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:11.490 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 0, 2024-08-28T19:44:11.490 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 0, 2024-08-28T19:44:11.490 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:11.490 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:11.490 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 69713, 2024-08-28T19:44:11.490 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 401579889, 2024-08-28T19:44:11.490 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 69713, 2024-08-28T19:44:11.491 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 401579889, 2024-08-28T19:44:11.491 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:11.491 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.491 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.491 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.491 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:11.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.491 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-objecter_ops": { 2024-08-28T19:44:11.491 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:11.492 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 1024, 2024-08-28T19:44:11.492 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:11.492 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 0, 2024-08-28T19:44:11.492 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 0, 2024-08-28T19:44:11.492 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:11.492 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:11.492 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 69713, 2024-08-28T19:44:11.492 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 69713, 2024-08-28T19:44:11.493 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 69713, 2024-08-28T19:44:11.493 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 69713, 2024-08-28T19:44:11.493 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:11.493 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.493 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.493 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.493 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:11.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.494 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-write_buf_throttle": { 2024-08-28T19:44:11.494 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:11.494 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 3758096384, 2024-08-28T19:44:11.494 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:11.494 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 18925, 2024-08-28T19:44:11.494 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 2065385, 2024-08-28T19:44:11.494 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:11.494 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 18925, 2024-08-28T19:44:11.494 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:11.495 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:11.495 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 790, 2024-08-28T19:44:11.495 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 2065385, 2024-08-28T19:44:11.495 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:11.495 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.495 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.495 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.495 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:11.495 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.496 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-write_buf_throttle-0x558cf27d0750": { 2024-08-28T19:44:11.496 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 16762, 2024-08-28T19:44:11.496 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 3758096384, 2024-08-28T19:44:11.496 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:11.496 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 96831, 2024-08-28T19:44:11.496 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 394661076, 2024-08-28T19:44:11.496 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:11.496 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 96831, 2024-08-28T19:44:11.497 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:11.497 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:11.497 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 27083, 2024-08-28T19:44:11.497 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 394644314, 2024-08-28T19:44:11.497 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:11.497 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:11.497 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:11.497 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:11.498 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:11.498 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:11.498 INFO:teuthology.orchestra.run.smithi115.stdout: "trackedop": { 2024-08-28T19:44:11.498 INFO:teuthology.orchestra.run.smithi115.stdout: "slow_ops_count": 1742 2024-08-28T19:44:11.498 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:11.498 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-08-28T19:44:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:11 smithi115 ceph-mon[22184]: pgmap v1066: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 160 KiB/s wr, 35 op/s 2024-08-28T19:44:11.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:11 smithi016 ceph-mon[18556]: pgmap v1066: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 160 KiB/s wr, 35 op/s 2024-08-28T19:44:11.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:11 smithi028 ceph-mon[21769]: pgmap v1066: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 160 KiB/s wr, 35 op/s 2024-08-28T19:44:12.687 INFO:tasks.check_counter:Daemon mds.i mds.exported=11 2024-08-28T19:44:12.687 INFO:tasks.check_counter:Daemon mds.i mds.imported=17 2024-08-28T19:44:12.687 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-08-28T19:44:12.687 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-08-28T19:44:12.687 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-08-28T19:44:12.687 DEBUG:tasks.check_counter:Getting stats from l 2024-08-28T19:44:12.688 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph daemon mds.l perf dump 2024-08-28T19:44:12.935 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.c/config 2024-08-28T19:44:13.512 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-08-28T19:44:13.513 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-0": { 2024-08-28T19:44:13.513 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 1518, 2024-08-28T19:44:13.513 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 1432, 2024-08-28T19:44:13.513 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 161631985, 2024-08-28T19:44:13.513 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 575870, 2024-08-28T19:44:13.513 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 7, 2024-08-28T19:44:13.513 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 3, 2024-08-28T19:44:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 0.269230994, 2024-08-28T19:44:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 0.074642877, 2024-08-28T19:44:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 0.172459674, 2024-08-28T19:44:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 0.027606612, 2024-08-28T19:44:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 1431, 2024-08-28T19:44:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.072061640, 2024-08-28T19:44:13.514 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000050357 2024-08-28T19:44:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 1483296, 2024-08-28T19:44:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 266768 2024-08-28T19:44:13.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.516 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-1": { 2024-08-28T19:44:13.516 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 1864, 2024-08-28T19:44:13.516 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 2181, 2024-08-28T19:44:13.516 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 176684416, 2024-08-28T19:44:13.516 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 1231871, 2024-08-28T19:44:13.517 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 7, 2024-08-28T19:44:13.517 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 6, 2024-08-28T19:44:13.517 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 0.282880311, 2024-08-28T19:44:13.517 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 0.093383427, 2024-08-28T19:44:13.517 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 0.125617724, 2024-08-28T19:44:13.517 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 0.049452733, 2024-08-28T19:44:13.517 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:13.517 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 2180, 2024-08-28T19:44:13.518 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.105866713, 2024-08-28T19:44:13.518 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000048562 2024-08-28T19:44:13.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.518 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:13.518 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.518 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.518 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.518 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-08-28T19:44:13.519 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 640704 2024-08-28T19:44:13.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.519 INFO:teuthology.orchestra.run.smithi115.stdout: "AsyncMessenger::Worker-2": { 2024-08-28T19:44:13.519 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_messages": 2688, 2024-08-28T19:44:13.519 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages": 2690, 2024-08-28T19:44:13.519 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_bytes": 126634567, 2024-08-28T19:44:13.519 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_bytes": 854842, 2024-08-28T19:44:13.519 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_created_connections": 6, 2024-08-28T19:44:13.520 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_active_connections": 5, 2024-08-28T19:44:13.520 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_total_time": 0.382444069, 2024-08-28T19:44:13.520 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_send_time": 0.123997159, 2024-08-28T19:44:13.520 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_recv_time": 0.147245982, 2024-08-28T19:44:13.520 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_running_fast_dispatch_time": 0.087792136, 2024-08-28T19:44:13.520 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_messages_queue_lat": { 2024-08-28T19:44:13.520 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 2688, 2024-08-28T19:44:13.520 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.130177804, 2024-08-28T19:44:13.521 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000048429 2024-08-28T19:44:13.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.521 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_handle_ack_lat": { 2024-08-28T19:44:13.521 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.521 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.521 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.521 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-28T19:44:13.522 INFO:teuthology.orchestra.run.smithi115.stdout: "msgr_send_encrypted_bytes": 0 2024-08-28T19:44:13.522 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.522 INFO:teuthology.orchestra.run.smithi115.stdout: "cct": { 2024-08-28T19:44:13.522 INFO:teuthology.orchestra.run.smithi115.stdout: "total_workers": 1, 2024-08-28T19:44:13.522 INFO:teuthology.orchestra.run.smithi115.stdout: "unhealthy_workers": 0 2024-08-28T19:44:13.522 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.522 INFO:teuthology.orchestra.run.smithi115.stdout: "finisher-MDSRank": { 2024-08-28T19:44:13.522 INFO:teuthology.orchestra.run.smithi115.stdout: "queue_len": 0, 2024-08-28T19:44:13.523 INFO:teuthology.orchestra.run.smithi115.stdout: "complete_latency": { 2024-08-28T19:44:13.523 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 9123, 2024-08-28T19:44:13.523 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.244853616, 2024-08-28T19:44:13.523 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000026839 2024-08-28T19:44:13.523 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:13.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.523 INFO:teuthology.orchestra.run.smithi115.stdout: "finisher-PurgeQueue": { 2024-08-28T19:44:13.523 INFO:teuthology.orchestra.run.smithi115.stdout: "queue_len": 0, 2024-08-28T19:44:13.524 INFO:teuthology.orchestra.run.smithi115.stdout: "complete_latency": { 2024-08-28T19:44:13.524 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.524 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.524 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.524 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:13.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.524 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": { 2024-08-28T19:44:13.524 INFO:teuthology.orchestra.run.smithi115.stdout: "request": 0, 2024-08-28T19:44:13.525 INFO:teuthology.orchestra.run.smithi115.stdout: "reply": 0, 2024-08-28T19:44:13.525 INFO:teuthology.orchestra.run.smithi115.stdout: "reply_latency": { 2024-08-28T19:44:13.525 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.525 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.525 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.525 INFO:teuthology.orchestra.run.smithi115.stdout: "slow_reply": 0, 2024-08-28T19:44:13.525 INFO:teuthology.orchestra.run.smithi115.stdout: "forward": 0, 2024-08-28T19:44:13.526 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_fetch_complete": 0, 2024-08-28T19:44:13.526 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_fetch_keys": 0, 2024-08-28T19:44:13.526 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_commit": 0, 2024-08-28T19:44:13.526 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_split": 0, 2024-08-28T19:44:13.526 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_merge": 0, 2024-08-28T19:44:13.526 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes": 19910, 2024-08-28T19:44:13.527 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_top": 1246, 2024-08-28T19:44:13.527 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_bottom": 18664, 2024-08-28T19:44:13.527 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_pin_tail": 0, 2024-08-28T19:44:13.527 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_pinned": 18130, 2024-08-28T19:44:13.527 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_expired": 0, 2024-08-28T19:44:13.527 INFO:teuthology.orchestra.run.smithi115.stdout: "inodes_with_caps": 0, 2024-08-28T19:44:13.527 INFO:teuthology.orchestra.run.smithi115.stdout: "caps": 0, 2024-08-28T19:44:13.527 INFO:teuthology.orchestra.run.smithi115.stdout: "subtrees": 12, 2024-08-28T19:44:13.528 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse": 0, 2024-08-28T19:44:13.528 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_hit": 0, 2024-08-28T19:44:13.528 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_forward": 0, 2024-08-28T19:44:13.528 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_discover": 0, 2024-08-28T19:44:13.528 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_dir_fetch": 0, 2024-08-28T19:44:13.528 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_remote_ino": 0, 2024-08-28T19:44:13.528 INFO:teuthology.orchestra.run.smithi115.stdout: "traverse_lock": 0, 2024-08-28T19:44:13.528 INFO:teuthology.orchestra.run.smithi115.stdout: "load_cent": 0, 2024-08-28T19:44:13.529 INFO:teuthology.orchestra.run.smithi115.stdout: "q": 0, 2024-08-28T19:44:13.529 INFO:teuthology.orchestra.run.smithi115.stdout: "exported": 0, 2024-08-28T19:44:13.529 INFO:teuthology.orchestra.run.smithi115.stdout: "exported_inodes": 0, 2024-08-28T19:44:13.529 INFO:teuthology.orchestra.run.smithi115.stdout: "imported": 0, 2024-08-28T19:44:13.529 INFO:teuthology.orchestra.run.smithi115.stdout: "imported_inodes": 0, 2024-08-28T19:44:13.529 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_dir_fetch": 0, 2024-08-28T19:44:13.529 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_backtrace_fetch": 0, 2024-08-28T19:44:13.529 INFO:teuthology.orchestra.run.smithi115.stdout: "openino_peer_discover": 0, 2024-08-28T19:44:13.530 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rfiles": 0, 2024-08-28T19:44:13.530 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rbytes": 0, 2024-08-28T19:44:13.530 INFO:teuthology.orchestra.run.smithi115.stdout: "root_rsnaps": 0, 2024-08-28T19:44:13.530 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_backtrace_fetch": 0, 2024-08-28T19:44:13.530 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_set_tag": 0, 2024-08-28T19:44:13.530 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_backtrace_repaired": 0, 2024-08-28T19:44:13.530 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_inotable_repaired": 0, 2024-08-28T19:44:13.530 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dir_inodes": 0, 2024-08-28T19:44:13.531 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dir_base_inodes": 0, 2024-08-28T19:44:13.531 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_dirfrag_rstats": 0, 2024-08-28T19:44:13.531 INFO:teuthology.orchestra.run.smithi115.stdout: "scrub_file_inodes": 0, 2024-08-28T19:44:13.531 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_inode_file_caps": 0, 2024-08-28T19:44:13.531 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_revoke": 0, 2024-08-28T19:44:13.531 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_grant": 0, 2024-08-28T19:44:13.531 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_trunc": 0, 2024-08-28T19:44:13.531 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-28T19:44:13.532 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-28T19:44:13.532 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_caps": 0, 2024-08-28T19:44:13.532 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_caps_dirty": 0, 2024-08-28T19:44:13.532 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_cap_release": 0, 2024-08-28T19:44:13.532 INFO:teuthology.orchestra.run.smithi115.stdout: "process_request_cap_release": 0 2024-08-28T19:44:13.532 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.532 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_cache": { 2024-08-28T19:44:13.532 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_update": 0, 2024-08-28T19:44:13.533 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_update_receipt": 0, 2024-08-28T19:44:13.533 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_try_discover": 0, 2024-08-28T19:44:13.533 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_send_discover": 0, 2024-08-28T19:44:13.533 INFO:teuthology.orchestra.run.smithi115.stdout: "dir_handle_discover": 0, 2024-08-28T19:44:13.533 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays": 0, 2024-08-28T19:44:13.533 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays_delayed": 0, 2024-08-28T19:44:13.533 INFO:teuthology.orchestra.run.smithi115.stdout: "num_strays_enqueuing": 0, 2024-08-28T19:44:13.533 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_created": 0, 2024-08-28T19:44:13.534 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_enqueued": 0, 2024-08-28T19:44:13.534 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_reintegrated": 0, 2024-08-28T19:44:13.534 INFO:teuthology.orchestra.run.smithi115.stdout: "strays_migrated": 0, 2024-08-28T19:44:13.534 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_processing": 0, 2024-08-28T19:44:13.534 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_enqueued": 0, 2024-08-28T19:44:13.534 INFO:teuthology.orchestra.run.smithi115.stdout: "num_recovering_prioritized": 0, 2024-08-28T19:44:13.534 INFO:teuthology.orchestra.run.smithi115.stdout: "recovery_started": 0, 2024-08-28T19:44:13.534 INFO:teuthology.orchestra.run.smithi115.stdout: "recovery_completed": 0, 2024-08-28T19:44:13.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_quiesce_path": 0, 2024-08-28T19:44:13.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_quiesce_inode": 0, 2024-08-28T19:44:13.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_enqueue_scrub": 0, 2024-08-28T19:44:13.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_exportdir": 0, 2024-08-28T19:44:13.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_flush": 0, 2024-08-28T19:44:13.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_fragmentdir": 0, 2024-08-28T19:44:13.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_fragstats": 0, 2024-08-28T19:44:13.535 INFO:teuthology.orchestra.run.smithi115.stdout: "ireq_inodestats": 0, 2024-08-28T19:44:13.535 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_started": 0, 2024-08-28T19:44:13.536 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_succeeded": 0, 2024-08-28T19:44:13.536 INFO:teuthology.orchestra.run.smithi115.stdout: "uninline_write_failed": 0 2024-08-28T19:44:13.536 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.536 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_log": { 2024-08-28T19:44:13.536 INFO:teuthology.orchestra.run.smithi115.stdout: "evlrg": 0, 2024-08-28T19:44:13.537 INFO:teuthology.orchestra.run.smithi115.stdout: "evadd": 0, 2024-08-28T19:44:13.537 INFO:teuthology.orchestra.run.smithi115.stdout: "evex": 0, 2024-08-28T19:44:13.537 INFO:teuthology.orchestra.run.smithi115.stdout: "evtrm": 0, 2024-08-28T19:44:13.537 INFO:teuthology.orchestra.run.smithi115.stdout: "ev": 104195, 2024-08-28T19:44:13.537 INFO:teuthology.orchestra.run.smithi115.stdout: "evexg": 0, 2024-08-28T19:44:13.537 INFO:teuthology.orchestra.run.smithi115.stdout: "evexd": 0, 2024-08-28T19:44:13.537 INFO:teuthology.orchestra.run.smithi115.stdout: "segadd": 0, 2024-08-28T19:44:13.537 INFO:teuthology.orchestra.run.smithi115.stdout: "segex": 0, 2024-08-28T19:44:13.538 INFO:teuthology.orchestra.run.smithi115.stdout: "segtrm": 0, 2024-08-28T19:44:13.538 INFO:teuthology.orchestra.run.smithi115.stdout: "seg": 141, 2024-08-28T19:44:13.538 INFO:teuthology.orchestra.run.smithi115.stdout: "segmjr": 10, 2024-08-28T19:44:13.538 INFO:teuthology.orchestra.run.smithi115.stdout: "segexg": 0, 2024-08-28T19:44:13.538 INFO:teuthology.orchestra.run.smithi115.stdout: "segexd": 0, 2024-08-28T19:44:13.538 INFO:teuthology.orchestra.run.smithi115.stdout: "expos": 22295538, 2024-08-28T19:44:13.538 INFO:teuthology.orchestra.run.smithi115.stdout: "wrpos": 466041040, 2024-08-28T19:44:13.538 INFO:teuthology.orchestra.run.smithi115.stdout: "rdpos": 466032097, 2024-08-28T19:44:13.538 INFO:teuthology.orchestra.run.smithi115.stdout: "jlat": { 2024-08-28T19:44:13.539 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.539 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.539 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.539 INFO:teuthology.orchestra.run.smithi115.stdout: "replayed": 104195 2024-08-28T19:44:13.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.539 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_mem": { 2024-08-28T19:44:13.539 INFO:teuthology.orchestra.run.smithi115.stdout: "ino": 1054, 2024-08-28T19:44:13.539 INFO:teuthology.orchestra.run.smithi115.stdout: "ino+": 20071, 2024-08-28T19:44:13.540 INFO:teuthology.orchestra.run.smithi115.stdout: "ino-": 19017, 2024-08-28T19:44:13.540 INFO:teuthology.orchestra.run.smithi115.stdout: "dir": 65, 2024-08-28T19:44:13.540 INFO:teuthology.orchestra.run.smithi115.stdout: "dir+": 106, 2024-08-28T19:44:13.540 INFO:teuthology.orchestra.run.smithi115.stdout: "dir-": 41, 2024-08-28T19:44:13.540 INFO:teuthology.orchestra.run.smithi115.stdout: "dn": 19910, 2024-08-28T19:44:13.540 INFO:teuthology.orchestra.run.smithi115.stdout: "dn+": 20262, 2024-08-28T19:44:13.540 INFO:teuthology.orchestra.run.smithi115.stdout: "dn-": 352, 2024-08-28T19:44:13.540 INFO:teuthology.orchestra.run.smithi115.stdout: "cap": 0, 2024-08-28T19:44:13.541 INFO:teuthology.orchestra.run.smithi115.stdout: "cap+": 0, 2024-08-28T19:44:13.541 INFO:teuthology.orchestra.run.smithi115.stdout: "cap-": 0, 2024-08-28T19:44:13.541 INFO:teuthology.orchestra.run.smithi115.stdout: "rss": 143556, 2024-08-28T19:44:13.541 INFO:teuthology.orchestra.run.smithi115.stdout: "heap": 348651520 2024-08-28T19:44:13.541 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.541 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_server": { 2024-08-28T19:44:13.541 INFO:teuthology.orchestra.run.smithi115.stdout: "dispatch_client_request": 0, 2024-08-28T19:44:13.541 INFO:teuthology.orchestra.run.smithi115.stdout: "dispatch_server_request": 0, 2024-08-28T19:44:13.542 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_request": 0, 2024-08-28T19:44:13.542 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_client_session": 0, 2024-08-28T19:44:13.542 INFO:teuthology.orchestra.run.smithi115.stdout: "handle_peer_request": 0, 2024-08-28T19:44:13.542 INFO:teuthology.orchestra.run.smithi115.stdout: "req_create_latency": { 2024-08-28T19:44:13.542 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.542 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.542 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.542 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.542 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getattr_latency": { 2024-08-28T19:44:13.543 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.543 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.543 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.543 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.543 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getfilelock_latency": { 2024-08-28T19:44:13.543 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.543 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.543 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.544 INFO:teuthology.orchestra.run.smithi115.stdout: "req_link_latency": { 2024-08-28T19:44:13.544 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.544 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.544 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.544 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookup_latency": { 2024-08-28T19:44:13.544 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.544 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.545 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.545 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.545 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookuphash_latency": { 2024-08-28T19:44:13.545 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.545 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.545 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.545 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.545 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupino_latency": { 2024-08-28T19:44:13.546 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.546 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.546 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.546 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupname_latency": { 2024-08-28T19:44:13.546 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.546 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.546 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.547 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupparent_latency": { 2024-08-28T19:44:13.547 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.547 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.547 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.547 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lookupsnap_latency": { 2024-08-28T19:44:13.547 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.547 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.548 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.548 INFO:teuthology.orchestra.run.smithi115.stdout: "req_lssnap_latency": { 2024-08-28T19:44:13.549 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.549 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.549 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.549 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mkdir_latency": { 2024-08-28T19:44:13.549 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.549 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.550 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.550 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mknod_latency": { 2024-08-28T19:44:13.550 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.550 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.550 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.550 INFO:teuthology.orchestra.run.smithi115.stdout: "req_mksnap_latency": { 2024-08-28T19:44:13.550 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.551 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.551 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.551 INFO:teuthology.orchestra.run.smithi115.stdout: "req_open_latency": { 2024-08-28T19:44:13.551 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.551 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.551 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.552 INFO:teuthology.orchestra.run.smithi115.stdout: "req_readdir_latency": { 2024-08-28T19:44:13.552 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.552 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.552 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.552 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rename_latency": { 2024-08-28T19:44:13.552 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.552 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.553 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.553 INFO:teuthology.orchestra.run.smithi115.stdout: "req_renamesnap_latency": { 2024-08-28T19:44:13.553 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.553 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.553 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.553 INFO:teuthology.orchestra.run.smithi115.stdout: "req_snapdiff_latency": { 2024-08-28T19:44:13.553 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.554 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.554 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.554 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmdir_latency": { 2024-08-28T19:44:13.554 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.554 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.554 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.555 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmsnap_latency": { 2024-08-28T19:44:13.555 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.555 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.555 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.555 INFO:teuthology.orchestra.run.smithi115.stdout: "req_rmxattr_latency": { 2024-08-28T19:44:13.555 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.555 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.555 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.556 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setattr_latency": { 2024-08-28T19:44:13.556 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.556 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.556 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.556 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setdirlayout_latency": { 2024-08-28T19:44:13.556 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setfilelock_latency": { 2024-08-28T19:44:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.558 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setlayout_latency": { 2024-08-28T19:44:13.558 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.558 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.558 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.558 INFO:teuthology.orchestra.run.smithi115.stdout: "req_setxattr_latency": { 2024-08-28T19:44:13.558 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.558 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.559 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.559 INFO:teuthology.orchestra.run.smithi115.stdout: "req_symlink_latency": { 2024-08-28T19:44:13.559 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.559 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.559 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.559 INFO:teuthology.orchestra.run.smithi115.stdout: "req_unlink_latency": { 2024-08-28T19:44:13.559 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.560 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.560 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.560 INFO:teuthology.orchestra.run.smithi115.stdout: "cap_revoke_eviction": 0, 2024-08-28T19:44:13.560 INFO:teuthology.orchestra.run.smithi115.stdout: "cap_acquisition_throttle": 0, 2024-08-28T19:44:13.560 INFO:teuthology.orchestra.run.smithi115.stdout: "req_getvxattr_latency": { 2024-08-28T19:44:13.560 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.560 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.561 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.561 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:13.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.561 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_sessions": { 2024-08-28T19:44:13.561 INFO:teuthology.orchestra.run.smithi115.stdout: "session_count": 2, 2024-08-28T19:44:13.561 INFO:teuthology.orchestra.run.smithi115.stdout: "session_add": 2, 2024-08-28T19:44:13.561 INFO:teuthology.orchestra.run.smithi115.stdout: "session_remove": 0, 2024-08-28T19:44:13.561 INFO:teuthology.orchestra.run.smithi115.stdout: "sessions_open": 2, 2024-08-28T19:44:13.562 INFO:teuthology.orchestra.run.smithi115.stdout: "sessions_stale": 0, 2024-08-28T19:44:13.562 INFO:teuthology.orchestra.run.smithi115.stdout: "total_load": 0, 2024-08-28T19:44:13.562 INFO:teuthology.orchestra.run.smithi115.stdout: "average_load": 0, 2024-08-28T19:44:13.562 INFO:teuthology.orchestra.run.smithi115.stdout: "avg_session_uptime": 2463, 2024-08-28T19:44:13.562 INFO:teuthology.orchestra.run.smithi115.stdout: "mdthresh_evicted": 0 2024-08-28T19:44:13.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.562 INFO:teuthology.orchestra.run.smithi115.stdout: "mempool": { 2024-08-28T19:44:13.562 INFO:teuthology.orchestra.run.smithi115.stdout: "bloom_filter_bytes": 0, 2024-08-28T19:44:13.563 INFO:teuthology.orchestra.run.smithi115.stdout: "bloom_filter_items": 0, 2024-08-28T19:44:13.563 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_alloc_bytes": 0, 2024-08-28T19:44:13.563 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_alloc_items": 0, 2024-08-28T19:44:13.563 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_data_bytes": 0, 2024-08-28T19:44:13.563 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_data_items": 0, 2024-08-28T19:44:13.563 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-28T19:44:13.563 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_onode_items": 0, 2024-08-28T19:44:13.563 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-28T19:44:13.563 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_meta_items": 0, 2024-08-28T19:44:13.564 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_other_bytes": 0, 2024-08-28T19:44:13.564 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_other_items": 0, 2024-08-28T19:44:13.564 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-28T19:44:13.564 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_cache_buffer_items": 0, 2024-08-28T19:44:13.564 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_extent_bytes": 0, 2024-08-28T19:44:13.564 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_extent_items": 0, 2024-08-28T19:44:13.564 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_blob_bytes": 0, 2024-08-28T19:44:13.564 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_blob_items": 0, 2024-08-28T19:44:13.565 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-28T19:44:13.565 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_shared_blob_items": 0, 2024-08-28T19:44:13.565 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-28T19:44:13.565 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_inline_bl_items": 0, 2024-08-28T19:44:13.565 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_fsck_bytes": 0, 2024-08-28T19:44:13.565 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_fsck_items": 0, 2024-08-28T19:44:13.565 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_txc_bytes": 0, 2024-08-28T19:44:13.565 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_txc_items": 0, 2024-08-28T19:44:13.566 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-28T19:44:13.566 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_deferred_items": 0, 2024-08-28T19:44:13.566 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_bytes": 0, 2024-08-28T19:44:13.566 INFO:teuthology.orchestra.run.smithi115.stdout: "bluestore_writing_items": 0, 2024-08-28T19:44:13.566 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_bytes": 0, 2024-08-28T19:44:13.566 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_items": 0, 2024-08-28T19:44:13.566 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_reader_bytes": 0, 2024-08-28T19:44:13.566 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_reader_items": 0, 2024-08-28T19:44:13.567 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_writer_bytes": 0, 2024-08-28T19:44:13.567 INFO:teuthology.orchestra.run.smithi115.stdout: "bluefs_file_writer_items": 0, 2024-08-28T19:44:13.567 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_anon_bytes": 289204, 2024-08-28T19:44:13.567 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_anon_items": 1347, 2024-08-28T19:44:13.567 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_meta_bytes": 0, 2024-08-28T19:44:13.567 INFO:teuthology.orchestra.run.smithi115.stdout: "buffer_meta_items": 0, 2024-08-28T19:44:13.567 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_bytes": 0, 2024-08-28T19:44:13.567 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_items": 0, 2024-08-28T19:44:13.568 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_mapbl_bytes": 0, 2024-08-28T19:44:13.568 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_mapbl_items": 0, 2024-08-28T19:44:13.568 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_pglog_bytes": 0, 2024-08-28T19:44:13.568 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_pglog_items": 0, 2024-08-28T19:44:13.568 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_bytes": 9888, 2024-08-28T19:44:13.568 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_items": 202, 2024-08-28T19:44:13.568 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_mapping_bytes": 0, 2024-08-28T19:44:13.568 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_mapping_items": 0, 2024-08-28T19:44:13.568 INFO:teuthology.orchestra.run.smithi115.stdout: "pgmap_bytes": 0, 2024-08-28T19:44:13.569 INFO:teuthology.orchestra.run.smithi115.stdout: "pgmap_items": 0, 2024-08-28T19:44:13.569 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_co_bytes": 16418817, 2024-08-28T19:44:13.569 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_co_items": 108971, 2024-08-28T19:44:13.569 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_1_bytes": 0, 2024-08-28T19:44:13.569 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_1_items": 0, 2024-08-28T19:44:13.569 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_2_bytes": 0, 2024-08-28T19:44:13.569 INFO:teuthology.orchestra.run.smithi115.stdout: "unittest_2_items": 0 2024-08-28T19:44:13.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.570 INFO:teuthology.orchestra.run.smithi115.stdout: "objecter": { 2024-08-28T19:44:13.570 INFO:teuthology.orchestra.run.smithi115.stdout: "op_active": 0, 2024-08-28T19:44:13.570 INFO:teuthology.orchestra.run.smithi115.stdout: "op_laggy": 0, 2024-08-28T19:44:13.570 INFO:teuthology.orchestra.run.smithi115.stdout: "op_send": 5519, 2024-08-28T19:44:13.570 INFO:teuthology.orchestra.run.smithi115.stdout: "op_send_bytes": 21, 2024-08-28T19:44:13.570 INFO:teuthology.orchestra.run.smithi115.stdout: "op_resend": 3, 2024-08-28T19:44:13.570 INFO:teuthology.orchestra.run.smithi115.stdout: "op_reply": 5519, 2024-08-28T19:44:13.570 INFO:teuthology.orchestra.run.smithi115.stdout: "op_latency": { 2024-08-28T19:44:13.571 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 5519, 2024-08-28T19:44:13.571 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 14.290718369, 2024-08-28T19:44:13.571 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.002589367 2024-08-28T19:44:13.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.571 INFO:teuthology.orchestra.run.smithi115.stdout: "op_inflight": 0, 2024-08-28T19:44:13.571 INFO:teuthology.orchestra.run.smithi115.stdout: "oplen_avg": { 2024-08-28T19:44:13.571 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 5519, 2024-08-28T19:44:13.571 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 5520 2024-08-28T19:44:13.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.572 INFO:teuthology.orchestra.run.smithi115.stdout: "op": 5519, 2024-08-28T19:44:13.572 INFO:teuthology.orchestra.run.smithi115.stdout: "op_r": 5519, 2024-08-28T19:44:13.572 INFO:teuthology.orchestra.run.smithi115.stdout: "op_w": 0, 2024-08-28T19:44:13.572 INFO:teuthology.orchestra.run.smithi115.stdout: "op_rmw": 0, 2024-08-28T19:44:13.572 INFO:teuthology.orchestra.run.smithi115.stdout: "op_pg": 0, 2024-08-28T19:44:13.572 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_stat": 3166, 2024-08-28T19:44:13.572 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_create": 0, 2024-08-28T19:44:13.572 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_read": 2351, 2024-08-28T19:44:13.573 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_write": 0, 2024-08-28T19:44:13.573 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_writefull": 0, 2024-08-28T19:44:13.573 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_writesame": 0, 2024-08-28T19:44:13.573 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_append": 0, 2024-08-28T19:44:13.573 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_zero": 0, 2024-08-28T19:44:13.573 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_truncate": 0, 2024-08-28T19:44:13.573 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_delete": 0, 2024-08-28T19:44:13.573 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_mapext": 0, 2024-08-28T19:44:13.574 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_sparse_read": 0, 2024-08-28T19:44:13.574 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_clonerange": 0, 2024-08-28T19:44:13.574 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_getxattr": 1, 2024-08-28T19:44:13.574 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_setxattr": 0, 2024-08-28T19:44:13.574 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_cmpxattr": 0, 2024-08-28T19:44:13.574 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_rmxattr": 0, 2024-08-28T19:44:13.574 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_resetxattrs": 0, 2024-08-28T19:44:13.574 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_call": 0, 2024-08-28T19:44:13.574 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_watch": 0, 2024-08-28T19:44:13.575 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_notify": 0, 2024-08-28T19:44:13.575 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_src_cmpxattr": 0, 2024-08-28T19:44:13.575 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_pgls": 0, 2024-08-28T19:44:13.575 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_pgls_filter": 0, 2024-08-28T19:44:13.575 INFO:teuthology.orchestra.run.smithi115.stdout: "osdop_other": 0, 2024-08-28T19:44:13.575 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_active": 0, 2024-08-28T19:44:13.575 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_send": 0, 2024-08-28T19:44:13.575 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_resend": 0, 2024-08-28T19:44:13.576 INFO:teuthology.orchestra.run.smithi115.stdout: "linger_ping": 0, 2024-08-28T19:44:13.576 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_active": 0, 2024-08-28T19:44:13.576 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_send": 0, 2024-08-28T19:44:13.576 INFO:teuthology.orchestra.run.smithi115.stdout: "poolop_resend": 0, 2024-08-28T19:44:13.576 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_active": 0, 2024-08-28T19:44:13.576 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_send": 0, 2024-08-28T19:44:13.576 INFO:teuthology.orchestra.run.smithi115.stdout: "poolstat_resend": 0, 2024-08-28T19:44:13.576 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_active": 0, 2024-08-28T19:44:13.577 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_send": 0, 2024-08-28T19:44:13.577 INFO:teuthology.orchestra.run.smithi115.stdout: "statfs_resend": 0, 2024-08-28T19:44:13.577 INFO:teuthology.orchestra.run.smithi115.stdout: "command_active": 0, 2024-08-28T19:44:13.577 INFO:teuthology.orchestra.run.smithi115.stdout: "command_send": 0, 2024-08-28T19:44:13.577 INFO:teuthology.orchestra.run.smithi115.stdout: "command_resend": 0, 2024-08-28T19:44:13.577 INFO:teuthology.orchestra.run.smithi115.stdout: "map_epoch": 117, 2024-08-28T19:44:13.577 INFO:teuthology.orchestra.run.smithi115.stdout: "map_full": 0, 2024-08-28T19:44:13.577 INFO:teuthology.orchestra.run.smithi115.stdout: "map_inc": 35, 2024-08-28T19:44:13.578 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_sessions": 11, 2024-08-28T19:44:13.578 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_session_open": 11, 2024-08-28T19:44:13.578 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_session_close": 0, 2024-08-28T19:44:13.578 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_laggy": 0, 2024-08-28T19:44:13.578 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_wr": 0, 2024-08-28T19:44:13.578 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_rd": 2, 2024-08-28T19:44:13.578 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_del": 0 2024-08-28T19:44:13.578 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.578 INFO:teuthology.orchestra.run.smithi115.stdout: "oft": { 2024-08-28T19:44:13.579 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_objs": 0, 2024-08-28T19:44:13.579 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_kv_pairs": 0, 2024-08-28T19:44:13.579 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_updates": 0, 2024-08-28T19:44:13.579 INFO:teuthology.orchestra.run.smithi115.stdout: "omap_total_removes": 0 2024-08-28T19:44:13.579 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.579 INFO:teuthology.orchestra.run.smithi115.stdout: "purge_queue": { 2024-08-28T19:44:13.579 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_ops": 0, 2024-08-28T19:44:13.579 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_ops_high_water": 0, 2024-08-28T19:44:13.580 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing": 0, 2024-08-28T19:44:13.580 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executing_high_water": 0, 2024-08-28T19:44:13.580 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executed_ops": 0, 2024-08-28T19:44:13.580 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_executed": 0, 2024-08-28T19:44:13.580 INFO:teuthology.orchestra.run.smithi115.stdout: "pq_item_in_journal": 0 2024-08-28T19:44:13.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.580 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-28T19:44:13.580 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:13.581 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 104857600, 2024-08-28T19:44:13.581 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:13.581 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 6066, 2024-08-28T19:44:13.581 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 464374407, 2024-08-28T19:44:13.581 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:13.581 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 6066, 2024-08-28T19:44:13.581 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:13.581 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:13.581 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 6066, 2024-08-28T19:44:13.582 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 464374407, 2024-08-28T19:44:13.582 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:13.582 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.582 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.582 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.582 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:13.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.582 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-objecter_bytes": { 2024-08-28T19:44:13.583 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:13.583 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 104857600, 2024-08-28T19:44:13.583 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:13.583 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 0, 2024-08-28T19:44:13.583 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 0, 2024-08-28T19:44:13.583 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:13.583 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:13.583 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 5519, 2024-08-28T19:44:13.583 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 461936762, 2024-08-28T19:44:13.584 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 5519, 2024-08-28T19:44:13.584 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 461936762, 2024-08-28T19:44:13.584 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:13.584 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.584 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.584 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.584 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:13.584 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.584 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-objecter_ops": { 2024-08-28T19:44:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 1024, 2024-08-28T19:44:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 0, 2024-08-28T19:44:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 0, 2024-08-28T19:44:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 5519, 2024-08-28T19:44:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 5519, 2024-08-28T19:44:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 5519, 2024-08-28T19:44:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 5519, 2024-08-28T19:44:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-write_buf_throttle": { 2024-08-28T19:44:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 3758096384, 2024-08-28T19:44:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 0, 2024-08-28T19:44:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 0, 2024-08-28T19:44:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 0, 2024-08-28T19:44:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 0, 2024-08-28T19:44:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.589 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:13.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.589 INFO:teuthology.orchestra.run.smithi115.stdout: "throttle-write_buf_throttle-0x55844521ec30": { 2024-08-28T19:44:13.589 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 0, 2024-08-28T19:44:13.589 INFO:teuthology.orchestra.run.smithi115.stdout: "max": 3758096384, 2024-08-28T19:44:13.589 INFO:teuthology.orchestra.run.smithi115.stdout: "get_started": 0, 2024-08-28T19:44:13.589 INFO:teuthology.orchestra.run.smithi115.stdout: "get": 0, 2024-08-28T19:44:13.589 INFO:teuthology.orchestra.run.smithi115.stdout: "get_sum": 0, 2024-08-28T19:44:13.590 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_fail": 0, 2024-08-28T19:44:13.590 INFO:teuthology.orchestra.run.smithi115.stdout: "get_or_fail_success": 0, 2024-08-28T19:44:13.590 INFO:teuthology.orchestra.run.smithi115.stdout: "take": 0, 2024-08-28T19:44:13.590 INFO:teuthology.orchestra.run.smithi115.stdout: "take_sum": 0, 2024-08-28T19:44:13.590 INFO:teuthology.orchestra.run.smithi115.stdout: "put": 0, 2024-08-28T19:44:13.590 INFO:teuthology.orchestra.run.smithi115.stdout: "put_sum": 0, 2024-08-28T19:44:13.590 INFO:teuthology.orchestra.run.smithi115.stdout: "wait": { 2024-08-28T19:44:13.590 INFO:teuthology.orchestra.run.smithi115.stdout: "avgcount": 0, 2024-08-28T19:44:13.591 INFO:teuthology.orchestra.run.smithi115.stdout: "sum": 0.000000000, 2024-08-28T19:44:13.591 INFO:teuthology.orchestra.run.smithi115.stdout: "avgtime": 0.000000000 2024-08-28T19:44:13.591 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:13.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-08-28T19:44:13.591 INFO:teuthology.orchestra.run.smithi115.stdout: "trackedop": { 2024-08-28T19:44:13.591 INFO:teuthology.orchestra.run.smithi115.stdout: "slow_ops_count": 0 2024-08-28T19:44:13.591 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-08-28T19:44:13.591 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-08-28T19:44:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:13 smithi115 ceph-mon[22184]: pgmap v1067: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 183 KiB/s wr, 37 op/s 2024-08-28T19:44:13.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:13 smithi016 ceph-mon[18556]: pgmap v1067: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 183 KiB/s wr, 37 op/s 2024-08-28T19:44:13.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:13 smithi028 ceph-mon[21769]: pgmap v1067: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 183 KiB/s wr, 37 op/s 2024-08-28T19:44:14.229 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-08-28T19:44:14.229 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-08-28T19:44:14.229 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-08-28T19:44:14.229 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-08-28T19:44:14.230 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-08-28T19:44:14.230 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-08-28T19:44:14.365 INFO:tasks.kclient:Unmounting kernel clients... 2024-08-28T19:44:14.366 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:44:14.366 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-28T19:44:14.386 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-08-28T19:44:14.386 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-28T19:44:14.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd blocklist ls 2024-08-28T19:44:14.748 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:44:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.202+0000 7f3ba1072640 1 -- 172.21.15.16:0/1175102380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c109b90 msgr2=0x7f3b9c1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.202+0000 7f3ba1072640 1 --2- 172.21.15.16:0/1175102380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c109b90 0x7f3b9c1103c0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009ca0 tx=0x7f3b9402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:44:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.202+0000 7f3ba1072640 1 -- 172.21.15.16:0/1175102380 shutdown_connections 2024-08-28T19:44:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.202+0000 7f3ba1072640 1 --2- 172.21.15.16:0/1175102380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c109b90 0x7f3b9c1103c0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.202+0000 7f3ba1072640 1 --2- 172.21.15.16:0/1175102380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b9c109240 0x7f3b9c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.202+0000 7f3ba1072640 1 --2- 172.21.15.16:0/1175102380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9c103900 0x7f3b9c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:15.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.202+0000 7f3ba1072640 1 -- 172.21.15.16:0/1175102380 >> 172.21.15.16:0/1175102380 conn(0x7f3b9c07b5f0 msgr2=0x7f3b9c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:15.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.202+0000 7f3ba1072640 1 -- 172.21.15.16:0/1175102380 shutdown_connections 2024-08-28T19:44:15.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.202+0000 7f3ba1072640 1 -- 172.21.15.16:0/1175102380 wait complete. 2024-08-28T19:44:15.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.203+0000 7f3ba1072640 1 Processor -- start 2024-08-28T19:44:15.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.203+0000 7f3ba1072640 1 -- start start 2024-08-28T19:44:15.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3ba1072640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c103900 0x7f3b9c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:15.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3ba1072640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9c109240 0x7f3b9c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:15.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3ba1072640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b9c109b90 0x7f3b9c19fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:15.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3ba1072640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1138a0 con 0x7f3b9c103900 2024-08-28T19:44:15.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3ba1072640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c113720 con 0x7f3b9c109240 2024-08-28T19:44:15.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3ba1072640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c113a20 con 0x7f3b9c109b90 2024-08-28T19:44:15.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3b9ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c103900 0x7f3b9c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:15.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3b9ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c103900 0x7f3b9c19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33058/0 (socket says 172.21.15.16:33058) 2024-08-28T19:44:15.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3b9ad9d640 1 -- 172.21.15.16:0/1399305651 learned_addr learned my addr 172.21.15.16:0/1399305651 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:44:15.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3b9b59e640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b9c109b90 0x7f3b9c19fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:15.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3b9a59c640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9c109240 0x7f3b9c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:15.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3b9ad9d640 1 -- 172.21.15.16:0/1399305651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b9c109b90 msgr2=0x7f3b9c19fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:15.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3b9ad9d640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b9c109b90 0x7f3b9c19fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:15.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3b9ad9d640 1 -- 172.21.15.16:0/1399305651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9c109240 msgr2=0x7f3b9c19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3b9ad9d640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9c109240 0x7f3b9c19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.204+0000 7f3b9ad9d640 1 -- 172.21.15.16:0/1399305651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9c1a0160 con 0x7f3b9c103900 2024-08-28T19:44:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.205+0000 7f3b9ad9d640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c103900 0x7f3b9c19a1f0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00d950 tx=0x7f3b8c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.205+0000 7f3b83fff640 1 -- 172.21.15.16:0/1399305651 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c017030 con 0x7f3b9c103900 2024-08-28T19:44:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.205+0000 7f3b83fff640 1 -- 172.21.15.16:0/1399305651 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b8c004020 con 0x7f3b9c103900 2024-08-28T19:44:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.205+0000 7f3ba1072640 1 -- 172.21.15.16:0/1399305651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b9c1a0450 con 0x7f3b9c103900 2024-08-28T19:44:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.205+0000 7f3b83fff640 1 -- 172.21.15.16:0/1399305651 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c0047a0 con 0x7f3b9c103900 2024-08-28T19:44:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.205+0000 7f3ba1072640 1 -- 172.21.15.16:0/1399305651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b9c1a6bb0 con 0x7f3b9c103900 2024-08-28T19:44:15.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.206+0000 7f3ba1072640 1 -- 172.21.15.16:0/1399305651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b9c103d10 con 0x7f3b9c103900 2024-08-28T19:44:15.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.207+0000 7f3b83fff640 1 -- 172.21.15.16:0/1399305651 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f3b8c009a30 con 0x7f3b9c103900 2024-08-28T19:44:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.208+0000 7f3b83fff640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3b840b4ca0 0x7f3b840b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.208+0000 7f3b9a59c640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3b840b4ca0 0x7f3b840b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.208+0000 7f3b83fff640 1 -- 172.21.15.16:0/1399305651 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9578+0+0 (secure 0 0 0) 0x7f3b8c0e5a20 con 0x7f3b9c103900 2024-08-28T19:44:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.209+0000 7f3b9a59c640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3b840b4ca0 0x7f3b840b71b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c108f10 tx=0x7f3b9000b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.211+0000 7f3b83fff640 1 -- 172.21.15.16:0/1399305651 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f3b8c0ac3f0 con 0x7f3b9c103900 2024-08-28T19:44:15.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.412+0000 7f3ba1072640 1 -- 172.21.15.16:0/1399305651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3b9c19b320 con 0x7f3b9c103900 2024-08-28T19:44:15.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.412+0000 7f3b83fff640 1 -- 172.21.15.16:0/1399305651 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v117) ==== 81+0+887 (secure 0 0 0) 0x7f3b8c0abcf0 con 0x7f3b9c103900 2024-08-28T19:44:15.415 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4224232323 2024-08-29T19:12:07.367977+0000 2024-08-28T19:44:15.415 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4022283606 2024-08-29T19:12:07.367977+0000 2024-08-28T19:44:15.415 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3264341019 2024-08-29T19:12:07.367977+0000 2024-08-28T19:44:15.415 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3057984026 2024-08-29T19:10:41.419191+0000 2024-08-28T19:44:15.415 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3042282523 2024-08-29T19:10:41.419191+0000 2024-08-28T19:44:15.415 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3243085606 2024-08-29T19:10:41.419191+0000 2024-08-28T19:44:15.416 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1768458721 2024-08-29T19:10:41.419191+0000 2024-08-28T19:44:15.416 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/949818237 2024-08-29T19:11:05.284513+0000 2024-08-28T19:44:15.416 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3057984026 2024-08-29T19:10:41.419191+0000 2024-08-28T19:44:15.416 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1317220879 2024-08-29T19:11:05.284513+0000 2024-08-28T19:44:15.416 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2455667776 2024-08-29T19:11:05.284513+0000 2024-08-28T19:44:15.416 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3371171272 2024-08-29T19:11:05.284513+0000 2024-08-28T19:44:15.416 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3264341019 2024-08-29T19:12:07.367977+0000 2024-08-28T19:44:15.417 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3371171272 2024-08-29T19:11:05.284513+0000 2024-08-28T19:44:15.417 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3868803359 2024-08-29T19:12:07.367977+0000 2024-08-28T19:44:15.417 INFO:teuthology.orchestra.run.smithi016.stderr:listed 15 entries 2024-08-28T19:44:15.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 -- 172.21.15.16:0/1399305651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3b840b4ca0 msgr2=0x7f3b840b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:15.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3b840b4ca0 0x7f3b840b71b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c108f10 tx=0x7f3b9000b390 comp rx=0 tx=0).stop 2024-08-28T19:44:15.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 -- 172.21.15.16:0/1399305651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c103900 msgr2=0x7f3b9c19a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:15.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c103900 0x7f3b9c19a1f0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00d950 tx=0x7f3b8c00de10 comp rx=0 tx=0).stop 2024-08-28T19:44:15.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 -- 172.21.15.16:0/1399305651 shutdown_connections 2024-08-28T19:44:15.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3b9c109b90 0x7f3b9c19fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:15.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f3b840b4ca0 0x7f3b840b71b0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:15.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9c109240 0x7f3b9c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:15.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 --2- 172.21.15.16:0/1399305651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c103900 0x7f3b9c19a1f0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:15.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 -- 172.21.15.16:0/1399305651 >> 172.21.15.16:0/1399305651 conn(0x7f3b9c07b5f0 msgr2=0x7f3b9c107230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:15.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 -- 172.21.15.16:0/1399305651 shutdown_connections 2024-08-28T19:44:15.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:15.415+0000 7f3ba1072640 1 -- 172.21.15.16:0/1399305651 wait complete. 2024-08-28T19:44:15.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:15 smithi016 ceph-mon[18556]: pgmap v1068: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 160 KiB/s wr, 20 op/s 2024-08-28T19:44:15.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:15 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1399305651' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-08-28T19:44:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:15 smithi028 ceph-mon[21769]: pgmap v1068: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 160 KiB/s wr, 20 op/s 2024-08-28T19:44:15.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:15 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1399305651' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-08-28T19:44:15.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:15 smithi115 ceph-mon[22184]: pgmap v1068: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 160 KiB/s wr, 20 op/s 2024-08-28T19:44:15.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:15 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1399305651' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-08-28T19:44:16.199 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-28T19:44:16.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:44:16.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-28T19:44:16.399 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-08-28T19:44:16.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:44:16.400 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-28T19:44:16.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:44:16.446 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-08-28T19:44:16.490 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-28T19:44:16.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:44:16.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-28T19:44:16.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-28T19:44:16.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-08-28T19:44:16.675 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-08-28T19:44:16.687 INFO:tasks.cephadm:Teardown begin 2024-08-28T19:44:16.687 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-08-28T19:44:16.687 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:44:16.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:16 smithi016 ceph-mon[18556]: osdmap e118: 12 total, 12 up, 12 in 2024-08-28T19:44:16.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:16 smithi028 ceph-mon[21769]: osdmap e118: 12 total, 12 up, 12 in 2024-08-28T19:44:16.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:16 smithi115 ceph-mon[22184]: osdmap e118: 12 total, 12 up, 12 in 2024-08-28T19:44:16.947 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:44:17.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.413+0000 7fa64e690640 1 -- 172.21.15.16:0/2869932055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa648109240 msgr2=0x7fa648109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:17.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.413+0000 7fa64e690640 1 --2- 172.21.15.16:0/2869932055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa648109240 0x7fa648109650 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fa63c009900 tx=0x7fa63c02a410 comp rx=0 tx=0).stop 2024-08-28T19:44:17.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.413+0000 7fa64e690640 1 -- 172.21.15.16:0/2869932055 shutdown_connections 2024-08-28T19:44:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.413+0000 7fa64e690640 1 --2- 172.21.15.16:0/2869932055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa648109b90 0x7fa6481103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.413+0000 7fa64e690640 1 --2- 172.21.15.16:0/2869932055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa648109240 0x7fa648109650 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.413+0000 7fa64e690640 1 --2- 172.21.15.16:0/2869932055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa648103900 0x7fa648103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.413+0000 7fa64e690640 1 -- 172.21.15.16:0/2869932055 >> 172.21.15.16:0/2869932055 conn(0x7fa64807b5f0 msgr2=0x7fa64807ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.413+0000 7fa64e690640 1 -- 172.21.15.16:0/2869932055 shutdown_connections 2024-08-28T19:44:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.413+0000 7fa64e690640 1 -- 172.21.15.16:0/2869932055 wait complete. 2024-08-28T19:44:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.414+0000 7fa64e690640 1 Processor -- start 2024-08-28T19:44:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.414+0000 7fa64e690640 1 -- start start 2024-08-28T19:44:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.414+0000 7fa64e690640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa648103900 0x7fa64819a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.414+0000 7fa64e690640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa648109240 0x7fa64819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.414+0000 7fa64e690640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa648109b90 0x7fa64819fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.414+0000 7fa64e690640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa648113850 con 0x7fa648103900 2024-08-28T19:44:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.414+0000 7fa64e690640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481136d0 con 0x7fa648109240 2024-08-28T19:44:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.414+0000 7fa64e690640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481139d0 con 0x7fa648109b90 2024-08-28T19:44:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.414+0000 7fa647fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa648103900 0x7fa64819a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.414+0000 7fa647fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa648103900 0x7fa64819a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33062/0 (socket says 172.21.15.16:33062) 2024-08-28T19:44:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.414+0000 7fa647fff640 1 -- 172.21.15.16:0/1684637104 learned_addr learned my addr 172.21.15.16:0/1684637104 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:44:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa64cc2d640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa648109b90 0x7fa64819fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa6477fe640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa648109240 0x7fa64819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa647fff640 1 -- 172.21.15.16:0/1684637104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa648109b90 msgr2=0x7fa64819fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa647fff640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa648109b90 0x7fa64819fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa647fff640 1 -- 172.21.15.16:0/1684637104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa648109240 msgr2=0x7fa64819a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa647fff640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa648109240 0x7fa64819a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa647fff640 1 -- 172.21.15.16:0/1684637104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6481a0290 con 0x7fa648103900 2024-08-28T19:44:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa64cc2d640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa648109b90 0x7fa64819fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:44:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa647fff640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa648103900 0x7fa64819a320 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fa63800d770 tx=0x7fa63800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa6457fa640 1 -- 172.21.15.16:0/1684637104 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638017070 con 0x7fa648103900 2024-08-28T19:44:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa6457fa640 1 -- 172.21.15.16:0/1684637104 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6380054d0 con 0x7fa648103900 2024-08-28T19:44:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa64e690640 1 -- 172.21.15.16:0/1684637104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6481a0580 con 0x7fa648103900 2024-08-28T19:44:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa6457fa640 1 -- 172.21.15.16:0/1684637104 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638004c30 con 0x7fa648103900 2024-08-28T19:44:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.415+0000 7fa64e690640 1 -- 172.21.15.16:0/1684637104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6481a6ce0 con 0x7fa648103900 2024-08-28T19:44:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.416+0000 7fa64e690640 1 -- 172.21.15.16:0/1684637104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa648103d10 con 0x7fa648103900 2024-08-28T19:44:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.418+0000 7fa6457fa640 1 -- 172.21.15.16:0/1684637104 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa638004dd0 con 0x7fa648103900 2024-08-28T19:44:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.418+0000 7fa6457fa640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6300b4ca0 0x7fa6300b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.418+0000 7fa6457fa640 1 -- 172.21.15.16:0/1684637104 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9490+0+0 (secure 0 0 0) 0x7fa6380e5870 con 0x7fa648103900 2024-08-28T19:44:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.418+0000 7fa6477fe640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6300b4ca0 0x7fa6300b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.419+0000 7fa6477fe640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6300b4ca0 0x7fa6300b71b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa63c009230 tx=0x7fa63c031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.422+0000 7fa6457fa640 1 -- 172.21.15.16:0/1684637104 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fa6380ac290 con 0x7fa648103900 2024-08-28T19:44:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.645+0000 7fa64e690640 1 -- 172.21.15.16:0/1684637104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa64810b960 con 0x7fa648103900 2024-08-28T19:44:17.647 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:17 smithi016 ceph-mon[18556]: pgmap v1070: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 20 op/s 2024-08-28T19:44:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.646+0000 7fa6457fa640 1 -- 172.21.15.16:0/1684637104 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+11984 (secure 0 0 0) 0x7fa6380abb90 con 0x7fa648103900 2024-08-28T19:44:17.648 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:44:17.648 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":109,"btime":"2024-08-28T19:43:33:491728+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":109,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:43:33.371957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:44:17.649 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 109 2024-08-28T19:44:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 -- 172.21.15.16:0/1684637104 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6300b4ca0 msgr2=0x7fa6300b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6300b4ca0 0x7fa6300b71b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa63c009230 tx=0x7fa63c031400 comp rx=0 tx=0).stop 2024-08-28T19:44:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 -- 172.21.15.16:0/1684637104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa648103900 msgr2=0x7fa64819a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa648103900 0x7fa64819a320 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fa63800d770 tx=0x7fa63800dc30 comp rx=0 tx=0).stop 2024-08-28T19:44:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 -- 172.21.15.16:0/1684637104 shutdown_connections 2024-08-28T19:44:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa648109b90 0x7fa64819fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa6300b4ca0 0x7fa6300b71b0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa648109240 0x7fa64819a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 --2- 172.21.15.16:0/1684637104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa648103900 0x7fa64819a320 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 -- 172.21.15.16:0/1684637104 >> 172.21.15.16:0/1684637104 conn(0x7fa64807b5f0 msgr2=0x7fa648105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 -- 172.21.15.16:0/1684637104 shutdown_connections 2024-08-28T19:44:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:17.648+0000 7fa64e690640 1 -- 172.21.15.16:0/1684637104 wait complete. 2024-08-28T19:44:17.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:17 smithi028 ceph-mon[21769]: pgmap v1070: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 20 op/s 2024-08-28T19:44:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:17 smithi115 ceph-mon[22184]: pgmap v1070: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 20 op/s 2024-08-28T19:44:18.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs dump --format=json 2024-08-28T19:44:18.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:18 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1684637104' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:44:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:44:18.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:18 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1684637104' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:44:18.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:18 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1684637104' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:44:19.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.443+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1859261461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e4108ab0 msgr2=0x7fa7e410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:19.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.443+0000 7fa7e9c1b640 1 --2- 172.21.15.16:0/1859261461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e4108ab0 0x7fa7e410f2e0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc009ca0 tx=0x7fa7dc02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:44:19.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.443+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1859261461 shutdown_connections 2024-08-28T19:44:19.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.443+0000 7fa7e9c1b640 1 --2- 172.21.15.16:0/1859261461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e4108ab0 0x7fa7e410f2e0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:19.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.443+0000 7fa7e9c1b640 1 --2- 172.21.15.16:0/1859261461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7e4108160 0x7fa7e4108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:19.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.443+0000 7fa7e9c1b640 1 --2- 172.21.15.16:0/1859261461 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7e4102820 0x7fa7e4102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:19.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.443+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1859261461 >> 172.21.15.16:0/1859261461 conn(0x7fa7e40794e0 msgr2=0x7fa7e40798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:19.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.443+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1859261461 shutdown_connections 2024-08-28T19:44:19.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.444+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1859261461 wait complete. 2024-08-28T19:44:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.444+0000 7fa7e9c1b640 1 Processor -- start 2024-08-28T19:44:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.444+0000 7fa7e9c1b640 1 -- start start 2024-08-28T19:44:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e9c1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7e4102820 0x7fa7e419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e9c1b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e4108160 0x7fa7e419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e9c1b640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7e4108ab0 0x7fa7e419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e9c1b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4113830 con 0x7fa7e4108160 2024-08-28T19:44:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e9c1b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e41136b0 con 0x7fa7e4102820 2024-08-28T19:44:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e9c1b640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e41139b0 con 0x7fa7e4108ab0 2024-08-28T19:44:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e4108160 0x7fa7e419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e4108160 0x7fa7e419a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33092/0 (socket says 172.21.15.16:33092) 2024-08-28T19:44:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e2ffd640 1 -- 172.21.15.16:0/1514488573 learned_addr learned my addr 172.21.15.16:0/1514488573 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:44:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e37fe640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7e4102820 0x7fa7e419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e3fff640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7e4108ab0 0x7fa7e419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e2ffd640 1 -- 172.21.15.16:0/1514488573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7e4108ab0 msgr2=0x7fa7e419faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e2ffd640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7e4108ab0 0x7fa7e419faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e2ffd640 1 -- 172.21.15.16:0/1514488573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7e4102820 msgr2=0x7fa7e419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e2ffd640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7e4102820 0x7fa7e419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e2ffd640 1 -- 172.21.15.16:0/1514488573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e41a01d0 con 0x7fa7e4108160 2024-08-28T19:44:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.445+0000 7fa7e2ffd640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e4108160 0x7fa7e419a800 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800c970 tx=0x7fa7d800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.446+0000 7fa7e0ff9640 1 -- 172.21.15.16:0/1514488573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8007bb0 con 0x7fa7e4108160 2024-08-28T19:44:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.446+0000 7fa7e0ff9640 1 -- 172.21.15.16:0/1514488573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7d8007d50 con 0x7fa7e4108160 2024-08-28T19:44:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.446+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1514488573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e41a04c0 con 0x7fa7e4108160 2024-08-28T19:44:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.446+0000 7fa7e0ff9640 1 -- 172.21.15.16:0/1514488573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d800f450 con 0x7fa7e4108160 2024-08-28T19:44:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.446+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1514488573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e41a6c20 con 0x7fa7e4108160 2024-08-28T19:44:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.447+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1514488573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e4103aa0 con 0x7fa7e4108160 2024-08-28T19:44:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.448+0000 7fa7e0ff9640 1 -- 172.21.15.16:0/1514488573 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fa7d8004f20 con 0x7fa7e4108160 2024-08-28T19:44:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.449+0000 7fa7e0ff9640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa7cc0b4ca0 0x7fa7cc0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.449+0000 7fa7e37fe640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa7cc0b4ca0 0x7fa7cc0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.449+0000 7fa7e0ff9640 1 -- 172.21.15.16:0/1514488573 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9490+0+0 (secure 0 0 0) 0x7fa7d80e5820 con 0x7fa7e4108160 2024-08-28T19:44:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.449+0000 7fa7e37fe640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa7cc0b4ca0 0x7fa7cc0b71b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4005fd0 tx=0x7fa7d400b390 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.452+0000 7fa7e0ff9640 1 -- 172.21.15.16:0/1514488573 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fa7d8014030 con 0x7fa7e4108160 2024-08-28T19:44:19.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.678+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1514488573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7e4102c30 con 0x7fa7e4108160 2024-08-28T19:44:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.679+0000 7fa7e0ff9640 1 -- 172.21.15.16:0/1514488573 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+11984 (secure 0 0 0) 0x7fa7d80abbf0 con 0x7fa7e4108160 2024-08-28T19:44:19.682 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:44:19.683 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":109,"btime":"2024-08-28T19:43:33:491728+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14751,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2536532891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2536532891},{"type":"v1","addr":"172.21.15.16:6835","nonce":2536532891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2448628159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2448628159},{"type":"v1","addr":"172.21.15.16:6837","nonce":2448628159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":109,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-28T19:18:33.178088+0000","modified":"2024-08-28T19:43:33.371957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24665,"mds_1":24649,"mds_2":24635,"mds_3":24625,"mds_4":24619},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14817":{"gid":14817,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/3069459721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3069459721},{"type":"v1","addr":"172.21.15.16:6839","nonce":3069459721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14823":{"gid":14823,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2698492790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2698492790},{"type":"v1","addr":"172.21.15.16:6841","nonce":2698492790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.115:6833/2581346583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":2581346583},{"type":"v1","addr":"172.21.15.115:6833","nonce":2581346583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/798033587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":798033587},{"type":"v1","addr":"172.21.15.28:6833","nonce":798033587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24605":{"gid":24605,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/2983279759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2983279759},{"type":"v1","addr":"172.21.15.28:6835","nonce":2983279759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24619":{"gid":24619,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/1297102593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":1297102593},{"type":"v1","addr":"172.21.15.115:6835","nonce":1297102593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.115:6837/4186796507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":4186796507},{"type":"v1","addr":"172.21.15.115:6837","nonce":4186796507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24635":{"gid":24635,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.28:6837/2888152030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2888152030},{"type":"v1","addr":"172.21.15.28:6837","nonce":2888152030}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24649":{"gid":24649,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.115:6839/3027291414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":3027291414},{"type":"v1","addr":"172.21.15.115:6839","nonce":3027291414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6839/2518320595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2518320595},{"type":"v1","addr":"172.21.15.28:6839","nonce":2518320595}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24665,"qdb_cluster":[24619,24625,24635,24649,24665]},"id":1}]} 2024-08-28T19:44:19.685 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 109 2024-08-28T19:44:19.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1514488573 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa7cc0b4ca0 msgr2=0x7fa7cc0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:19.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa7cc0b4ca0 0x7fa7cc0b71b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4005fd0 tx=0x7fa7d400b390 comp rx=0 tx=0).stop 2024-08-28T19:44:19.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1514488573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e4108160 msgr2=0x7fa7e419a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:19.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e4108160 0x7fa7e419a800 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800c970 tx=0x7fa7d800ce30 comp rx=0 tx=0).stop 2024-08-28T19:44:19.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1514488573 shutdown_connections 2024-08-28T19:44:19.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7e4108ab0 0x7fa7e419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:19.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e4108160 0x7fa7e419a800 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:19.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fa7cc0b4ca0 0x7fa7cc0b71b0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:19.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 --2- 172.21.15.16:0/1514488573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7e4102820 0x7fa7e419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:19.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1514488573 >> 172.21.15.16:0/1514488573 conn(0x7fa7e40794e0 msgr2=0x7fa7e41041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:19.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1514488573 shutdown_connections 2024-08-28T19:44:19.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:19.681+0000 7fa7e9c1b640 1 -- 172.21.15.16:0/1514488573 wait complete. 2024-08-28T19:44:19.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:19 smithi016 ceph-mon[18556]: pgmap v1071: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 16 op/s 2024-08-28T19:44:19.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:19 smithi028 ceph-mon[21769]: pgmap v1071: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 16 op/s 2024-08-28T19:44:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:19 smithi115 ceph-mon[22184]: pgmap v1071: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 16 op/s 2024-08-28T19:44:20.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-28T19:44:20.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:20 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1514488573' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:44:20.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:20 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1514488573' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:44:20.911 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:44:21.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:20 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1514488573' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-08-28T19:44:21.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:21 smithi016 ceph-mon[18556]: pgmap v1072: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 17 op/s 2024-08-28T19:44:21.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.453+0000 7f687e311640 1 -- 172.21.15.16:0/523499463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6878102f60 msgr2=0x7f6878103370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:21.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.453+0000 7f687e311640 1 --2- 172.21.15.16:0/523499463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6878102f60 0x7f6878103370 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f6868009900 tx=0x7f686802a410 comp rx=0 tx=0).stop 2024-08-28T19:44:21.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.453+0000 7f687e311640 1 -- 172.21.15.16:0/523499463 shutdown_connections 2024-08-28T19:44:21.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.453+0000 7f687e311640 1 --2- 172.21.15.16:0/523499463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6878108cf0 0x7f687810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.453+0000 7f687e311640 1 --2- 172.21.15.16:0/523499463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68781083a0 0x7f68781087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.453+0000 7f687e311640 1 --2- 172.21.15.16:0/523499463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6878102f60 0x7f6878103370 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.453+0000 7f687e311640 1 -- 172.21.15.16:0/523499463 >> 172.21.15.16:0/523499463 conn(0x7f6878077a60 msgr2=0x7f6878077ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.453+0000 7f687e311640 1 -- 172.21.15.16:0/523499463 shutdown_connections 2024-08-28T19:44:21.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.453+0000 7f687e311640 1 -- 172.21.15.16:0/523499463 wait complete. 2024-08-28T19:44:21.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.454+0000 7f687e311640 1 Processor -- start 2024-08-28T19:44:21.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.454+0000 7f687e311640 1 -- start start 2024-08-28T19:44:21.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.454+0000 7f687e311640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6878102f60 0x7f687819b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:21.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.454+0000 7f687e311640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68781083a0 0x7f687819b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:21.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.454+0000 7f687e311640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6878108cf0 0x7f687819bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:21.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.454+0000 7f687e311640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68781129b0 con 0x7f6878108cf0 2024-08-28T19:44:21.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.454+0000 7f687e311640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6878112830 con 0x7f68781083a0 2024-08-28T19:44:21.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.454+0000 7f687e311640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6878112b30 con 0x7f6878102f60 2024-08-28T19:44:21.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.454+0000 7f687c8ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6878108cf0 0x7f687819bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:21.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.454+0000 7f687c8ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6878108cf0 0x7f687819bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33110/0 (socket says 172.21.15.16:33110) 2024-08-28T19:44:21.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.454+0000 7f687c8ae640 1 -- 172.21.15.16:0/1879928549 learned_addr learned my addr 172.21.15.16:0/1879928549 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:44:21.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f686ffff640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68781083a0 0x7f687819b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:21.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f6877fff640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6878102f60 0x7f687819b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:21.678 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-08-28T19:44:21.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f687c8ae640 1 -- 172.21.15.16:0/1879928549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6878102f60 msgr2=0x7f687819b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:21.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f687c8ae640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6878102f60 0x7f687819b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:21.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f687c8ae640 1 -- 172.21.15.16:0/1879928549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68781083a0 msgr2=0x7f687819b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f687c8ae640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68781083a0 0x7f687819b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f687c8ae640 1 -- 172.21.15.16:0/1879928549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68781a6530 con 0x7f6878108cf0 2024-08-28T19:44:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f686ffff640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68781083a0 0x7f687819b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:44:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f687c8ae640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6878108cf0 0x7f687819bca0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f687000eaa0 tx=0x7f687000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f6875ffb640 1 -- 172.21.15.16:0/1879928549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687000cdb0 con 0x7f6878108cf0 2024-08-28T19:44:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f687e311640 1 -- 172.21.15.16:0/1879928549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68781a6820 con 0x7f6878108cf0 2024-08-28T19:44:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f6875ffb640 1 -- 172.21.15.16:0/1879928549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6870005020 con 0x7f6878108cf0 2024-08-28T19:44:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f6875ffb640 1 -- 172.21.15.16:0/1879928549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68700116d0 con 0x7f6878108cf0 2024-08-28T19:44:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.455+0000 7f687e311640 1 -- 172.21.15.16:0/1879928549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68781a6d60 con 0x7f6878108cf0 2024-08-28T19:44:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.456+0000 7f687e311640 1 -- 172.21.15.16:0/1879928549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6878103370 con 0x7f6878108cf0 2024-08-28T19:44:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.458+0000 7f6875ffb640 1 -- 172.21.15.16:0/1879928549 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f68700054a0 con 0x7f6878108cf0 2024-08-28T19:44:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.459+0000 7f6875ffb640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f68640b4ca0 0x7f68640b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.459+0000 7f6877fff640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f68640b4ca0 0x7f68640b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.459+0000 7f6875ffb640 1 -- 172.21.15.16:0/1879928549 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9490+0+0 (secure 0 0 0) 0x7f68700e5ef0 con 0x7f6878108cf0 2024-08-28T19:44:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.459+0000 7f6877fff640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f68640b4ca0 0x7f68640b71b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6868000c00 tx=0x7f6868031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.462+0000 7f6875ffb640 1 -- 172.21.15.16:0/1879928549 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f6870016030 con 0x7f6878108cf0 2024-08-28T19:44:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.649+0000 7f687e311640 1 -- 172.21.15.16:0/1879928549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6878102030 con 0x7f6878108cf0 2024-08-28T19:44:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.651+0000 7f6875ffb640 1 -- 172.21.15.16:0/1879928549 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+25368 (secure 0 0 0) 0x7f68700ac910 con 0x7f6878108cf0 2024-08-28T19:44:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 -- 172.21.15.16:0/1879928549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f68640b4ca0 msgr2=0x7f68640b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f68640b4ca0 0x7f68640b71b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6868000c00 tx=0x7f6868031400 comp rx=0 tx=0).stop 2024-08-28T19:44:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 -- 172.21.15.16:0/1879928549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6878108cf0 msgr2=0x7f687819bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6878108cf0 0x7f687819bca0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f687000eaa0 tx=0x7f687000ef60 comp rx=0 tx=0).stop 2024-08-28T19:44:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 -- 172.21.15.16:0/1879928549 shutdown_connections 2024-08-28T19:44:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6878108cf0 0x7f687819bca0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f68640b4ca0 0x7f68640b71b0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68781083a0 0x7f687819b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 --2- 172.21.15.16:0/1879928549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6878102f60 0x7f687819b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 -- 172.21.15.16:0/1879928549 >> 172.21.15.16:0/1879928549 conn(0x7f6878077a60 msgr2=0x7f68781063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 -- 172.21.15.16:0/1879928549 shutdown_connections 2024-08-28T19:44:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:21.653+0000 7f687e311640 1 -- 172.21.15.16:0/1879928549 wait complete. 2024-08-28T19:44:21.683 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":118,"fsid":"eabc438a-6570-11ef-bcd4-c7b262605968","created":"2024-08-28T19:10:20.239380+0000","modified":"2024-08-28T19:44:15.468163+0000","last_up_change":"2024-08-28T19:17:41.649492+0000","last_in_change":"2024-08-28T19:17:18.182185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-28T19:13:55.828283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-28T19:18:29.423422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-28T19:18:31.212294+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":41,"snap_epoch":117,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-28T19:18:37.500166+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":41,"snap_epoch":117,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7a5e8a2d-958b-4c1f-b2be-90743c629505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6803","nonce":1371554327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6805","nonce":1371554327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6809","nonce":1371554327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1371554327},{"type":"v1","addr":"172.21.15.16:6807","nonce":1371554327}]},"public_addr":"172.21.15.16:6803/1371554327","cluster_addr":"172.21.15.16:6805/1371554327","heartbeat_back_addr":"172.21.15.16:6809/1371554327","heartbeat_front_addr":"172.21.15.16:6807/1371554327","state":["exists","up"]},{"osd":1,"uuid":"d4767044-e5ca-447f-9666-0840aa1028dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6801","nonce":666617235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6803","nonce":666617235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6807","nonce":666617235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":666617235},{"type":"v1","addr":"172.21.15.28:6805","nonce":666617235}]},"public_addr":"172.21.15.28:6801/666617235","cluster_addr":"172.21.15.28:6803/666617235","heartbeat_back_addr":"172.21.15.28:6807/666617235","heartbeat_front_addr":"172.21.15.28:6805/666617235","state":["exists","up"]},{"osd":2,"uuid":"ddf6f747-c099-48b5-a9ee-6b4f1ec6ae22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6801","nonce":1599247648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6803","nonce":1599247648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6807","nonce":1599247648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1599247648},{"type":"v1","addr":"172.21.15.115:6805","nonce":1599247648}]},"public_addr":"172.21.15.115:6801/1599247648","cluster_addr":"172.21.15.115:6803/1599247648","heartbeat_back_addr":"172.21.15.115:6807/1599247648","heartbeat_front_addr":"172.21.15.115:6805/1599247648","state":["exists","up"]},{"osd":3,"uuid":"0edc63dd-6f28-4c2a-9bde-d6be083fc6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6811","nonce":3791831569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6813","nonce":3791831569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6817","nonce":3791831569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3791831569},{"type":"v1","addr":"172.21.15.16:6815","nonce":3791831569}]},"public_addr":"172.21.15.16:6811/3791831569","cluster_addr":"172.21.15.16:6813/3791831569","heartbeat_back_addr":"172.21.15.16:6817/3791831569","heartbeat_front_addr":"172.21.15.16:6815/3791831569","state":["exists","up"]},{"osd":4,"uuid":"fcaeb7d4-aa38-40a7-abdf-3f16cf9047df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6809","nonce":302267969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6811","nonce":302267969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6815","nonce":302267969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":302267969},{"type":"v1","addr":"172.21.15.28:6813","nonce":302267969}]},"public_addr":"172.21.15.28:6809/302267969","cluster_addr":"172.21.15.28:6811/302267969","heartbeat_back_addr":"172.21.15.28:6815/302267969","heartbeat_front_addr":"172.21.15.28:6813/302267969","state":["exists","up"]},{"osd":5,"uuid":"762a439b-bec1-471e-ab08-aa233ee83d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6809","nonce":1119597596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6811","nonce":1119597596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6815","nonce":1119597596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1119597596},{"type":"v1","addr":"172.21.15.115:6813","nonce":1119597596}]},"public_addr":"172.21.15.115:6809/1119597596","cluster_addr":"172.21.15.115:6811/1119597596","heartbeat_back_addr":"172.21.15.115:6815/1119597596","heartbeat_front_addr":"172.21.15.115:6813/1119597596","state":["exists","up"]},{"osd":6,"uuid":"33c56c57-ce44-4000-94b3-518d7ce0afd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6819","nonce":1671078622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6821","nonce":1671078622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6825","nonce":1671078622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1671078622},{"type":"v1","addr":"172.21.15.16:6823","nonce":1671078622}]},"public_addr":"172.21.15.16:6819/1671078622","cluster_addr":"172.21.15.16:6821/1671078622","heartbeat_back_addr":"172.21.15.16:6825/1671078622","heartbeat_front_addr":"172.21.15.16:6823/1671078622","state":["exists","up"]},{"osd":7,"uuid":"8ef482ae-33bd-4121-8b16-c80ad1b9178b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6817","nonce":2650596315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6819","nonce":2650596315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6823","nonce":2650596315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2650596315},{"type":"v1","addr":"172.21.15.28:6821","nonce":2650596315}]},"public_addr":"172.21.15.28:6817/2650596315","cluster_addr":"172.21.15.28:6819/2650596315","heartbeat_back_addr":"172.21.15.28:6823/2650596315","heartbeat_front_addr":"172.21.15.28:6821/2650596315","state":["exists","up"]},{"osd":8,"uuid":"f814cd8e-30c2-4e74-aa5c-eb2d40e27586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6817","nonce":557022149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6819","nonce":557022149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6823","nonce":557022149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":557022149},{"type":"v1","addr":"172.21.15.115:6821","nonce":557022149}]},"public_addr":"172.21.15.115:6817/557022149","cluster_addr":"172.21.15.115:6819/557022149","heartbeat_back_addr":"172.21.15.115:6823/557022149","heartbeat_front_addr":"172.21.15.115:6821/557022149","state":["exists","up"]},{"osd":9,"uuid":"9617dc8d-0eb5-4fe1-bc88-5d37d40c065f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6827","nonce":1827017769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6829","nonce":1827017769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6833","nonce":1827017769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1827017769},{"type":"v1","addr":"172.21.15.16:6831","nonce":1827017769}]},"public_addr":"172.21.15.16:6827/1827017769","cluster_addr":"172.21.15.16:6829/1827017769","heartbeat_back_addr":"172.21.15.16:6833/1827017769","heartbeat_front_addr":"172.21.15.16:6831/1827017769","state":["exists","up"]},{"osd":10,"uuid":"dbed2c84-11b5-409d-a8e5-33c9fddc5698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6825","nonce":4224878207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6827","nonce":4224878207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6831","nonce":4224878207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":4224878207},{"type":"v1","addr":"172.21.15.28:6829","nonce":4224878207}]},"public_addr":"172.21.15.28:6825/4224878207","cluster_addr":"172.21.15.28:6827/4224878207","heartbeat_back_addr":"172.21.15.28:6831/4224878207","heartbeat_front_addr":"172.21.15.28:6829/4224878207","state":["exists","up"]},{"osd":11,"uuid":"3ce535d6-4278-46a0-af6d-a7c90a74aaee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6825","nonce":3906373111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6827","nonce":3906373111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6831","nonce":3906373111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3906373111},{"type":"v1","addr":"172.21.15.115:6829","nonce":3906373111}]},"public_addr":"172.21.15.115:6825/3906373111","cluster_addr":"172.21.15.115:6827/3906373111","heartbeat_back_addr":"172.21.15.115:6831/3906373111","heartbeat_front_addr":"172.21.15.115:6829/3906373111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:04.852883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:28.692848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:13:53.154219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:17.521411+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:14:41.880119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:07.121399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:32.530594+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:15:58.294537+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:23.153669+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:16:48.258043+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:14.175444+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-28T19:17:39.582207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4224232323":"2024-08-29T19:12:07.367977+0000","172.21.15.16:0/4022283606":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6800/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3042282523":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/3243085606":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1768458721":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/949818237":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3057984026":"2024-08-29T19:10:41.419191+0000","172.21.15.16:0/1317220879":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/2455667776":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6800/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:6801/3264341019":"2024-08-29T19:12:07.367977+0000","172.21.15.16:6801/3371171272":"2024-08-29T19:11:05.284513+0000","172.21.15.16:0/3868803359":"2024-08-29T19:12:07.367977+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":28,"length":1},{"begin":41,"length":1}]},{"pool":4,"snaps":[{"begin":28,"length":1},{"begin":41,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-28T19:44:21.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:21 smithi028 ceph-mon[21769]: pgmap v1072: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 17 op/s 2024-08-28T19:44:21.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:21 smithi115 ceph-mon[22184]: pgmap v1072: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 17 op/s 2024-08-28T19:44:22.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs fail cephfs 2024-08-28T19:44:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:44:22.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:22 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/1879928549' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:44:22.810 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:22 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/1879928549' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:44:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:22 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/1879928549' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-28T19:44:23.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.053+0000 7fcc1450e640 1 -- 172.21.15.16:0/4130218279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c109b90 msgr2=0x7fcc0c1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.053+0000 7fcc1450e640 1 --2- 172.21.15.16:0/4130218279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c109b90 0x7fcc0c1103c0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009ca0 tx=0x7fcc0402a8c0 comp rx=0 tx=0).stop 2024-08-28T19:44:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.054+0000 7fcc1450e640 1 -- 172.21.15.16:0/4130218279 shutdown_connections 2024-08-28T19:44:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.054+0000 7fcc1450e640 1 --2- 172.21.15.16:0/4130218279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c109b90 0x7fcc0c1103c0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.054+0000 7fcc1450e640 1 --2- 172.21.15.16:0/4130218279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c109240 0x7fcc0c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.054+0000 7fcc1450e640 1 --2- 172.21.15.16:0/4130218279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc0c103900 0x7fcc0c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.054+0000 7fcc1450e640 1 -- 172.21.15.16:0/4130218279 >> 172.21.15.16:0/4130218279 conn(0x7fcc0c07b5f0 msgr2=0x7fcc0c07ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.054+0000 7fcc1450e640 1 -- 172.21.15.16:0/4130218279 shutdown_connections 2024-08-28T19:44:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.054+0000 7fcc1450e640 1 -- 172.21.15.16:0/4130218279 wait complete. 2024-08-28T19:44:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.054+0000 7fcc1450e640 1 Processor -- start 2024-08-28T19:44:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.054+0000 7fcc1450e640 1 -- start start 2024-08-28T19:44:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc1450e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c103900 0x7fcc0c0754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc1450e640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc0c109240 0x7fcc0c0759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc1450e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c109b90 0x7fcc0c075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc1450e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c113850 con 0x7fcc0c103900 2024-08-28T19:44:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc1450e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c1136d0 con 0x7fcc0c109b90 2024-08-28T19:44:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc1450e640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c1139d0 con 0x7fcc0c109240 2024-08-28T19:44:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc122aa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c103900 0x7fcc0c0754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc122aa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c103900 0x7fcc0c0754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33130/0 (socket says 172.21.15.16:33130) 2024-08-28T19:44:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc122aa640 1 -- 172.21.15.16:0/226625692 learned_addr learned my addr 172.21.15.16:0/226625692 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:44:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc11aa9640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc0c109240 0x7fcc0c0759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc12aab640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c109b90 0x7fcc0c075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc122aa640 1 -- 172.21.15.16:0/226625692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc0c109240 msgr2=0x7fcc0c0759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc122aa640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc0c109240 0x7fcc0c0759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc122aa640 1 -- 172.21.15.16:0/226625692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c109b90 msgr2=0x7fcc0c075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc122aa640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c109b90 0x7fcc0c075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc122aa640 1 -- 172.21.15.16:0/226625692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c1a8480 con 0x7fcc0c103900 2024-08-28T19:44:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc11aa9640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc0c109240 0x7fcc0c0759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:44:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc12aab640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c109b90 0x7fcc0c075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-28T19:44:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc122aa640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c103900 0x7fcc0c0754b0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800d770 tx=0x7fcc0800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc037fe640 1 -- 172.21.15.16:0/226625692 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc08017070 con 0x7fcc0c103900 2024-08-28T19:44:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc037fe640 1 -- 172.21.15.16:0/226625692 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc080054d0 con 0x7fcc0c103900 2024-08-28T19:44:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc1450e640 1 -- 172.21.15.16:0/226625692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0c1a8770 con 0x7fcc0c103900 2024-08-28T19:44:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc037fe640 1 -- 172.21.15.16:0/226625692 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc08004c30 con 0x7fcc0c103900 2024-08-28T19:44:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.055+0000 7fcc1450e640 1 -- 172.21.15.16:0/226625692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc0c1a8c60 con 0x7fcc0c103900 2024-08-28T19:44:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.056+0000 7fcc1450e640 1 -- 172.21.15.16:0/226625692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc0c103d10 con 0x7fcc0c103900 2024-08-28T19:44:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.058+0000 7fcc037fe640 1 -- 172.21.15.16:0/226625692 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fcc08004dd0 con 0x7fcc0c103900 2024-08-28T19:44:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.059+0000 7fcc037fe640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcbf80b4ca0 0x7fcbf80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.059+0000 7fcc11aa9640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcbf80b4ca0 0x7fcbf80b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.059+0000 7fcc037fe640 1 -- 172.21.15.16:0/226625692 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9490+0+0 (secure 0 0 0) 0x7fcc080e5920 con 0x7fcc0c103900 2024-08-28T19:44:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.059+0000 7fcc11aa9640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcbf80b4ca0 0x7fcbf80b71b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c106e90 tx=0x7fcbfc005ec0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.062+0000 7fcc037fe640 1 -- 172.21.15.16:0/226625692 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fcc080ac2e0 con 0x7fcc0c103900 2024-08-28T19:44:23.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.288+0000 7fcc1450e640 1 -- 172.21.15.16:0/226625692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fcc0c02e430 con 0x7fcc0c103900 2024-08-28T19:44:23.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.291+0000 7fcc037fe640 1 -- 172.21.15.16:0/226625692 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v111) ==== 161+0+0 (secure 0 0 0) 0x7fcc080abbe0 con 0x7fcc0c103900 2024-08-28T19:44:23.293 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-28T19:44:23.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.296+0000 7fcc1450e640 1 -- 172.21.15.16:0/226625692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcbf80b4ca0 msgr2=0x7fcbf80b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:23.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.296+0000 7fcc1450e640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcbf80b4ca0 0x7fcbf80b71b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c106e90 tx=0x7fcbfc005ec0 comp rx=0 tx=0).stop 2024-08-28T19:44:23.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.296+0000 7fcc1450e640 1 -- 172.21.15.16:0/226625692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c103900 msgr2=0x7fcc0c0754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:23.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.296+0000 7fcc1450e640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c103900 0x7fcc0c0754b0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800d770 tx=0x7fcc0800dc30 comp rx=0 tx=0).stop 2024-08-28T19:44:23.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.296+0000 7fcc1450e640 1 -- 172.21.15.16:0/226625692 shutdown_connections 2024-08-28T19:44:23.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.296+0000 7fcc1450e640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fcbf80b4ca0 0x7fcbf80b71b0 secure :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c106e90 tx=0x7fcbfc005ec0 comp rx=0 tx=0).stop 2024-08-28T19:44:23.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.296+0000 7fcc1450e640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c109b90 0x7fcc0c075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:23.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.296+0000 7fcc1450e640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc0c109240 0x7fcc0c0759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:23.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.296+0000 7fcc1450e640 1 --2- 172.21.15.16:0/226625692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c103900 0x7fcc0c0754b0 secure :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800d770 tx=0x7fcc0800dc30 comp rx=0 tx=0).stop 2024-08-28T19:44:23.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.296+0000 7fcc1450e640 1 -- 172.21.15.16:0/226625692 >> 172.21.15.16:0/226625692 conn(0x7fcc0c07b5f0 msgr2=0x7fcc0c105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:23.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.297+0000 7fcc1450e640 1 -- 172.21.15.16:0/226625692 shutdown_connections 2024-08-28T19:44:23.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:23.297+0000 7fcc1450e640 1 -- 172.21.15.16:0/226625692 wait complete. 2024-08-28T19:44:23.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a[18552]: 2024-08-28T19:44:23.289+0000 7f46600be640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-28T19:44:23.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-mon[18556]: pgmap v1073: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 52 KiB/s wr, 16 op/s 2024-08-28T19:44:23.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-mon[18556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:44:23.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/226625692' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-08-28T19:44:23.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-mon[18556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-08-28T19:44:23.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-mon[18556]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-28T19:44:23.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/226625692' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-08-28T19:44:23.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-mon[18556]: osdmap e119: 12 total, 12 up, 12 in 2024-08-28T19:44:23.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-mon[18556]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-08-28T19:44:23.615 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a[63654]: ignoring --setuser ceph since I am not root 2024-08-28T19:44:23.615 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a[63654]: ignoring --setgroup ceph since I am not root 2024-08-28T19:44:23.616 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-mds[63665]: -- 172.21.15.16:0/2600070152 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556610c09860 con 0x556614dafa80 2024-08-28T19:44:23.616 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a[63654]: starting mds.a at 2024-08-28T19:44:23.616 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g[63297]: ignoring --setuser ceph since I am not root 2024-08-28T19:44:23.616 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g[63297]: ignoring --setgroup ceph since I am not root 2024-08-28T19:44:23.617 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-mds[63313]: -- 172.21.15.16:0/1061586700 <== mon.1 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562cf7d51c20 con 0x562cfbef7180 2024-08-28T19:44:23.617 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:44:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g[63297]: starting mds.g at 2024-08-28T19:44:23.809 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e[52312]: ignoring --setuser ceph since I am not root 2024-08-28T19:44:23.809 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e[52312]: ignoring --setgroup ceph since I am not root 2024-08-28T19:44:23.809 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mds[52316]: -- 172.21.15.28:0/4011052794 <== mon.1 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556cad78b860 con 0x556cb1931a80 2024-08-28T19:44:23.810 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e[52312]: starting mds.e at 2024-08-28T19:44:23.810 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b[52931]: ignoring --setuser ceph since I am not root 2024-08-28T19:44:23.810 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b[52931]: ignoring --setgroup ceph since I am not root 2024-08-28T19:44:23.810 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mds[52935]: -- 172.21.15.28:0/81977360 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5570297a9520 con 0x55702d945180 2024-08-28T19:44:23.810 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mds[52935]: -- 172.21.15.28:0/81977360 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55702979f860 con 0x55702d945180 2024-08-28T19:44:23.810 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b[52931]: starting mds.b at 2024-08-28T19:44:23.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mon[21769]: pgmap v1073: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 52 KiB/s wr, 16 op/s 2024-08-28T19:44:23.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mon[21769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:44:23.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/226625692' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-08-28T19:44:23.811 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mon[21769]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-08-28T19:44:23.812 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mon[21769]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-28T19:44:23.812 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/226625692' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-08-28T19:44:23.812 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mon[21769]: osdmap e119: 12 total, 12 up, 12 in 2024-08-28T19:44:23.812 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mon[21769]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-08-28T19:44:23.812 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k[52007]: ignoring --setuser ceph since I am not root 2024-08-28T19:44:23.812 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k[52007]: ignoring --setgroup ceph since I am not root 2024-08-28T19:44:23.813 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mds[52011]: -- 172.21.15.28:0/763981533 <== mon.2 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c22c57f860 con 0x55c230725180 2024-08-28T19:44:23.813 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k[52007]: starting mds.k at 2024-08-28T19:44:23.813 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-h[52614]: ignoring --setuser ceph since I am not root 2024-08-28T19:44:23.813 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-h[52614]: ignoring --setgroup ceph since I am not root 2024-08-28T19:44:23.813 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-mds[52618]: -- 172.21.15.28:0/1969845498 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c214049860 con 0x55c2181ef600 2024-08-28T19:44:23.813 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:44:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-h[52614]: starting mds.h at 2024-08-28T19:44:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mon[22184]: pgmap v1073: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 52 KiB/s wr, 16 op/s 2024-08-28T19:44:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mon[22184]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-08-28T19:44:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/226625692' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-08-28T19:44:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mon[22184]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-08-28T19:44:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mon[22184]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-28T19:44:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/226625692' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-08-28T19:44:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mon[22184]: osdmap e119: 12 total, 12 up, 12 in 2024-08-28T19:44:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mon[22184]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-08-28T19:44:23.873 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f[53356]: ignoring --setuser ceph since I am not root 2024-08-28T19:44:23.873 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f[53356]: ignoring --setgroup ceph since I am not root 2024-08-28T19:44:23.873 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mds[53360]: -- 172.21.15.115:0/3788774110 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f188495860 con 0x55f18c63b600 2024-08-28T19:44:23.873 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f[53356]: starting mds.f at 2024-08-28T19:44:23.873 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i[53958]: ignoring --setuser ceph since I am not root 2024-08-28T19:44:23.874 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i[53958]: ignoring --setgroup ceph since I am not root 2024-08-28T19:44:23.874 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mds[53962]: -- 172.21.15.115:0/590847328 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563eabdfb860 con 0x563eaffa1180 2024-08-28T19:44:23.874 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i[53958]: starting mds.i at 2024-08-28T19:44:23.874 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l[53055]: ignoring --setuser ceph since I am not root 2024-08-28T19:44:23.874 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l[53055]: ignoring --setgroup ceph since I am not root 2024-08-28T19:44:23.874 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mds[53059]: -- 172.21.15.115:0/3916631435 <== mon.1 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cfe4363860 con 0x55cfe8509180 2024-08-28T19:44:23.875 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l[53055]: starting mds.l at 2024-08-28T19:44:23.875 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-c[53655]: ignoring --setuser ceph since I am not root 2024-08-28T19:44:23.875 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-c[53655]: ignoring --setgroup ceph since I am not root 2024-08-28T19:44:23.875 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-mds[53659]: -- 172.21.15.115:0/1237865042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5591ce775860 con 0x5591d291b180 2024-08-28T19:44:23.875 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:44:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-c[53655]: starting mds.c at 2024-08-28T19:44:23.958 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-08-28T19:44:24.210 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:44:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.701+0000 7f1c9683c640 1 -- 172.21.15.16:0/1504672535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c90104660 msgr2=0x7f1c90104a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.701+0000 7f1c9683c640 1 --2- 172.21.15.16:0/1504672535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c90104660 0x7f1c90104a70 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009900 tx=0x7f1c8002a410 comp rx=0 tx=0).stop 2024-08-28T19:44:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.701+0000 7f1c9683c640 1 -- 172.21.15.16:0/1504672535 shutdown_connections 2024-08-28T19:44:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.701+0000 7f1c9683c640 1 --2- 172.21.15.16:0/1504672535 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c9010dc10 0x7f1c901100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.701+0000 7f1c9683c640 1 --2- 172.21.15.16:0/1504672535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c90105040 0x7f1c9010d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.701+0000 7f1c9683c640 1 --2- 172.21.15.16:0/1504672535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c90104660 0x7f1c90104a70 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.701+0000 7f1c9683c640 1 -- 172.21.15.16:0/1504672535 >> 172.21.15.16:0/1504672535 conn(0x7f1c90079500 msgr2=0x7f1c90079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.701+0000 7f1c9683c640 1 -- 172.21.15.16:0/1504672535 shutdown_connections 2024-08-28T19:44:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.701+0000 7f1c9683c640 1 -- 172.21.15.16:0/1504672535 wait complete. 2024-08-28T19:44:24.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c9683c640 1 Processor -- start 2024-08-28T19:44:24.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c9683c640 1 -- start start 2024-08-28T19:44:24.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c9683c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c90104660 0x7f1c9019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:24.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c9683c640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c90105040 0x7f1c9019a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:24.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c9683c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9010dc10 0x7f1c9019fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c9683c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c901138d0 con 0x7f1c9010dc10 2024-08-28T19:44:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c9683c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c90113750 con 0x7f1c90104660 2024-08-28T19:44:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c9683c640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1c90113a50 con 0x7f1c90105040 2024-08-28T19:44:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c94dd9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9010dc10 0x7f1c9019fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c94dd9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9010dc10 0x7f1c9019fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48408/0 (socket says 172.21.15.16:48408) 2024-08-28T19:44:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c94dd9640 1 -- 172.21.15.16:0/633580409 learned_addr learned my addr 172.21.15.16:0/633580409 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:44:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c87fff640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c90105040 0x7f1c9019a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c8ffff640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c90104660 0x7f1c9019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c94dd9640 1 -- 172.21.15.16:0/633580409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c90105040 msgr2=0x7f1c9019a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c94dd9640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c90105040 0x7f1c9019a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c94dd9640 1 -- 172.21.15.16:0/633580409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c90104660 msgr2=0x7f1c9019a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c94dd9640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c90104660 0x7f1c9019a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.702+0000 7f1c94dd9640 1 -- 172.21.15.16:0/633580409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c901a0150 con 0x7f1c9010dc10 2024-08-28T19:44:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.703+0000 7f1c94dd9640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9010dc10 0x7f1c9019fae0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800b970 tx=0x7f1c8800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.703+0000 7f1c8dffb640 1 -- 172.21.15.16:0/633580409 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c88019070 con 0x7f1c9010dc10 2024-08-28T19:44:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.703+0000 7f1c8dffb640 1 -- 172.21.15.16:0/633580409 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c88014440 con 0x7f1c9010dc10 2024-08-28T19:44:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.703+0000 7f1c9683c640 1 -- 172.21.15.16:0/633580409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c901a0440 con 0x7f1c9010dc10 2024-08-28T19:44:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.703+0000 7f1c8dffb640 1 -- 172.21.15.16:0/633580409 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c880075d0 con 0x7f1c9010dc10 2024-08-28T19:44:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.703+0000 7f1c9683c640 1 -- 172.21.15.16:0/633580409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c901a6c60 con 0x7f1c9010dc10 2024-08-28T19:44:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.704+0000 7f1c9683c640 1 -- 172.21.15.16:0/633580409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c9006bf30 con 0x7f1c9010dc10 2024-08-28T19:44:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.706+0000 7f1c8dffb640 1 -- 172.21.15.16:0/633580409 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f1c8801c070 con 0x7f1c9010dc10 2024-08-28T19:44:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.707+0000 7f1c8dffb640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c7c0b4cd0 0x7f1c7c0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.707+0000 7f1c8ffff640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c7c0b4cd0 0x7f1c7c0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.707+0000 7f1c8dffb640 1 -- 172.21.15.16:0/633580409 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9832+0+0 (secure 0 0 0) 0x7f1c880e53e0 con 0x7f1c9010dc10 2024-08-28T19:44:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.708+0000 7f1c8ffff640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c7c0b4cd0 0x7f1c7c0b71e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1c80000c00 tx=0x7f1c80031400 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:24.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.710+0000 7f1c8dffb640 1 -- 172.21.15.16:0/633580409 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f1c880abcb0 con 0x7f1c9010dc10 2024-08-28T19:44:24.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:24.933+0000 7f1c9683c640 1 -- 172.21.15.16:0/633580409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1c9019b3d0 con 0x7f1c9010dc10 2024-08-28T19:44:25.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.510+0000 7f1c8dffb640 1 -- 172.21.15.16:0/633580409 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v113) ==== 104+0+0 (secure 0 0 0) 0x7f1c880ab5b0 con 0x7f1c9010dc10 2024-08-28T19:44:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.512+0000 7f1c9683c640 1 -- 172.21.15.16:0/633580409 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c7c0b4cd0 msgr2=0x7f1c7c0b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.512+0000 7f1c9683c640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c7c0b4cd0 0x7f1c7c0b71e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1c80000c00 tx=0x7f1c80031400 comp rx=0 tx=0).stop 2024-08-28T19:44:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.512+0000 7f1c9683c640 1 -- 172.21.15.16:0/633580409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9010dc10 msgr2=0x7f1c9019fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.512+0000 7f1c9683c640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9010dc10 0x7f1c9019fae0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800b970 tx=0x7f1c8800be30 comp rx=0 tx=0).stop 2024-08-28T19:44:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.512+0000 7f1c9683c640 1 -- 172.21.15.16:0/633580409 shutdown_connections 2024-08-28T19:44:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.512+0000 7f1c9683c640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9010dc10 0x7f1c9019fae0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.512+0000 7f1c9683c640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1c90105040 0x7f1c9019a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.512+0000 7f1c9683c640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f1c7c0b4cd0 0x7f1c7c0b71e0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.512+0000 7f1c9683c640 1 --2- 172.21.15.16:0/633580409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c90104660 0x7f1c9019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:25.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.512+0000 7f1c9683c640 1 -- 172.21.15.16:0/633580409 >> 172.21.15.16:0/633580409 conn(0x7f1c90079500 msgr2=0x7f1c901006a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:25.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.512+0000 7f1c9683c640 1 -- 172.21.15.16:0/633580409 shutdown_connections 2024-08-28T19:44:25.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:25.513+0000 7f1c9683c640 1 -- 172.21.15.16:0/633580409 wait complete. 2024-08-28T19:44:25.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: pgmap v1075: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-08-28T19:44:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.16:6839/4156258876,v1:172.21.15.16:6841/4156258876] up:boot 2024-08-28T19:44:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.16:6838/3002218840,v1:172.21.15.16:6840/3002218840] up:boot 2024-08-28T19:44:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.115:6832/60753407,v1:172.21.15.115:6835/60753407] up:boot 2024-08-28T19:44:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.115:6833/231676009,v1:172.21.15.115:6834/231676009] up:boot 2024-08-28T19:44:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.115:6836/3398864882,v1:172.21.15.115:6838/3398864882] up:boot 2024-08-28T19:44:25.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.115:6837/2289019328,v1:172.21.15.115:6839/2289019328] up:boot 2024-08-28T19:44:25.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.28:6833/534041106,v1:172.21.15.28:6836/534041106] up:boot 2024-08-28T19:44:25.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.28:6832/3188674501,v1:172.21.15.28:6837/3188674501] up:boot 2024-08-28T19:44:25.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.28:6834/1935931657,v1:172.21.15.28:6838/1935931657] up:boot 2024-08-28T19:44:25.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: mds.? [v2:172.21.15.28:6835/1369912233,v1:172.21.15.28:6839/1369912233] up:boot 2024-08-28T19:44:25.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-08-28T19:44:25.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-08-28T19:44:25.561 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-08-28T19:44:25.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-08-28T19:44:25.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-08-28T19:44:25.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-08-28T19:44:25.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-08-28T19:44:25.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-08-28T19:44:25.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-08-28T19:44:25.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-08-28T19:44:25.562 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-08-28T19:44:25.563 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:25 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/633580409' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-08-28T19:44:25.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: pgmap v1075: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-08-28T19:44:25.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.16:6839/4156258876,v1:172.21.15.16:6841/4156258876] up:boot 2024-08-28T19:44:25.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.16:6838/3002218840,v1:172.21.15.16:6840/3002218840] up:boot 2024-08-28T19:44:25.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.115:6832/60753407,v1:172.21.15.115:6835/60753407] up:boot 2024-08-28T19:44:25.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.115:6833/231676009,v1:172.21.15.115:6834/231676009] up:boot 2024-08-28T19:44:25.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.115:6836/3398864882,v1:172.21.15.115:6838/3398864882] up:boot 2024-08-28T19:44:25.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.115:6837/2289019328,v1:172.21.15.115:6839/2289019328] up:boot 2024-08-28T19:44:25.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.28:6833/534041106,v1:172.21.15.28:6836/534041106] up:boot 2024-08-28T19:44:25.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.28:6832/3188674501,v1:172.21.15.28:6837/3188674501] up:boot 2024-08-28T19:44:25.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.28:6834/1935931657,v1:172.21.15.28:6838/1935931657] up:boot 2024-08-28T19:44:25.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: mds.? [v2:172.21.15.28:6835/1369912233,v1:172.21.15.28:6839/1369912233] up:boot 2024-08-28T19:44:25.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-08-28T19:44:25.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-08-28T19:44:25.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-08-28T19:44:25.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-08-28T19:44:25.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-08-28T19:44:25.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-08-28T19:44:25.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-08-28T19:44:25.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-08-28T19:44:25.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-08-28T19:44:25.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-08-28T19:44:25.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-08-28T19:44:25.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:25 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/633580409' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-08-28T19:44:25.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: pgmap v1075: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-08-28T19:44:25.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.16:6839/4156258876,v1:172.21.15.16:6841/4156258876] up:boot 2024-08-28T19:44:25.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.16:6838/3002218840,v1:172.21.15.16:6840/3002218840] up:boot 2024-08-28T19:44:25.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.115:6832/60753407,v1:172.21.15.115:6835/60753407] up:boot 2024-08-28T19:44:25.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.115:6833/231676009,v1:172.21.15.115:6834/231676009] up:boot 2024-08-28T19:44:25.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.115:6836/3398864882,v1:172.21.15.115:6838/3398864882] up:boot 2024-08-28T19:44:25.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.115:6837/2289019328,v1:172.21.15.115:6839/2289019328] up:boot 2024-08-28T19:44:25.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.28:6833/534041106,v1:172.21.15.28:6836/534041106] up:boot 2024-08-28T19:44:25.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.28:6832/3188674501,v1:172.21.15.28:6837/3188674501] up:boot 2024-08-28T19:44:25.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.28:6834/1935931657,v1:172.21.15.28:6838/1935931657] up:boot 2024-08-28T19:44:25.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: mds.? [v2:172.21.15.28:6835/1369912233,v1:172.21.15.28:6839/1369912233] up:boot 2024-08-28T19:44:25.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-08-28T19:44:25.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-08-28T19:44:25.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-08-28T19:44:25.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-08-28T19:44:25.752 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-08-28T19:44:25.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-08-28T19:44:25.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-08-28T19:44:25.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-08-28T19:44:25.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-08-28T19:44:25.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-08-28T19:44:25.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: from='mgr.14220 172.21.15.16:0/2045126432' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-08-28T19:44:25.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:25 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/633580409' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-08-28T19:44:26.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-08-28T19:44:26.488 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:44:26.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:26 smithi016 ceph-mon[18556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-08-28T19:44:26.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:26 smithi016 ceph-mon[18556]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-28T19:44:26.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:26 smithi016 ceph-mon[18556]: Cluster is now healthy 2024-08-28T19:44:26.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:26 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/633580409' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-08-28T19:44:26.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:26 smithi016 ceph-mon[18556]: fsmap 12 up:standby 2024-08-28T19:44:26.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:26 smithi028 ceph-mon[21769]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-08-28T19:44:26.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:26 smithi028 ceph-mon[21769]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-28T19:44:26.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:26 smithi028 ceph-mon[21769]: Cluster is now healthy 2024-08-28T19:44:26.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:26 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/633580409' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-08-28T19:44:26.560 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:26 smithi028 ceph-mon[21769]: fsmap 12 up:standby 2024-08-28T19:44:26.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:26 smithi115 ceph-mon[22184]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-08-28T19:44:26.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:26 smithi115 ceph-mon[22184]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-28T19:44:26.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:26 smithi115 ceph-mon[22184]: Cluster is now healthy 2024-08-28T19:44:26.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:26 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/633580409' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-08-28T19:44:26.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:26 smithi115 ceph-mon[22184]: fsmap 12 up:standby 2024-08-28T19:44:27.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.037+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/2263311083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea410dbb0 msgr2=0x7f8ea41100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.037+0000 7f8ea9ca9640 1 --2- 172.21.15.16:0/2263311083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea410dbb0 0x7f8ea41100c0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c009ca0 tx=0x7f8e9c02a8c0 comp rx=0 tx=0).stop 2024-08-28T19:44:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.037+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/2263311083 shutdown_connections 2024-08-28T19:44:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.037+0000 7f8ea9ca9640 1 --2- 172.21.15.16:0/2263311083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea410dbb0 0x7f8ea41100c0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.037+0000 7f8ea9ca9640 1 --2- 172.21.15.16:0/2263311083 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ea4105030 0x7f8ea410d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.037+0000 7f8ea9ca9640 1 --2- 172.21.15.16:0/2263311083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4104600 0x7f8ea4104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.038+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/2263311083 >> 172.21.15.16:0/2263311083 conn(0x7f8ea40794e0 msgr2=0x7f8ea40798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.038+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/2263311083 shutdown_connections 2024-08-28T19:44:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.038+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/2263311083 wait complete. 2024-08-28T19:44:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.038+0000 7f8ea9ca9640 1 Processor -- start 2024-08-28T19:44:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.038+0000 7f8ea9ca9640 1 -- start start 2024-08-28T19:44:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.039+0000 7f8ea9ca9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4104600 0x7f8ea419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.039+0000 7f8ea9ca9640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ea4105030 0x7f8ea419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.039+0000 7f8ea9ca9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea410dbb0 0x7f8ea419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.039+0000 7f8ea9ca9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea4113830 con 0x7f8ea410dbb0 2024-08-28T19:44:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.039+0000 7f8ea9ca9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea41136b0 con 0x7f8ea4104600 2024-08-28T19:44:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.039+0000 7f8ea9ca9640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea41139b0 con 0x7f8ea4105030 2024-08-28T19:44:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.039+0000 7f8ea3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea410dbb0 0x7f8ea419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.039+0000 7f8ea3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea410dbb0 0x7f8ea419faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48424/0 (socket says 172.21.15.16:48424) 2024-08-28T19:44:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.039+0000 7f8ea3fff640 1 -- 172.21.15.16:0/3835719441 learned_addr learned my addr 172.21.15.16:0/3835719441 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:44:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.039+0000 7f8e9affd640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ea4105030 0x7f8ea419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.039+0000 7f8ea37fe640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4104600 0x7f8ea419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8ea3fff640 1 -- 172.21.15.16:0/3835719441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ea4105030 msgr2=0x7f8ea419a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8ea3fff640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ea4105030 0x7f8ea419a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8ea3fff640 1 -- 172.21.15.16:0/3835719441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4104600 msgr2=0x7f8ea419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8ea3fff640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4104600 0x7f8ea419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8ea3fff640 1 -- 172.21.15.16:0/3835719441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea41a0110 con 0x7f8ea410dbb0 2024-08-28T19:44:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8e9affd640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ea4105030 0x7f8ea419a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:44:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8ea3fff640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea410dbb0 0x7f8ea419faa0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c005cc0 tx=0x7f8e9c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8ea17fa640 1 -- 172.21.15.16:0/3835719441 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c003bf0 con 0x7f8ea410dbb0 2024-08-28T19:44:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/3835719441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea41a03a0 con 0x7f8ea410dbb0 2024-08-28T19:44:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8ea17fa640 1 -- 172.21.15.16:0/3835719441 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e9c003d90 con 0x7f8ea410dbb0 2024-08-28T19:44:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8ea17fa640 1 -- 172.21.15.16:0/3835719441 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c02fbc0 con 0x7f8ea410dbb0 2024-08-28T19:44:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.040+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/3835719441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea41a6c20 con 0x7f8ea410dbb0 2024-08-28T19:44:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.041+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/3835719441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ea406bf80 con 0x7f8ea410dbb0 2024-08-28T19:44:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.042+0000 7f8ea17fa640 1 -- 172.21.15.16:0/3835719441 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f8e9c02fd60 con 0x7f8ea410dbb0 2024-08-28T19:44:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.043+0000 7f8ea17fa640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e900b4cd0 0x7f8e900b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:27.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.043+0000 7f8ea37fe640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e900b4cd0 0x7f8e900b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:27.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.043+0000 7f8ea17fa640 1 -- 172.21.15.16:0/3835719441 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9832+0+0 (secure 0 0 0) 0x7f8e9c106f00 con 0x7f8ea410dbb0 2024-08-28T19:44:27.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.044+0000 7f8ea37fe640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e900b4cd0 0x7f8e900b71e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f8e94005fd0 tx=0x7f8e94005850 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:27.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.048+0000 7f8ea17fa640 1 -- 172.21.15.16:0/3835719441 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f8e9c0cd770 con 0x7f8ea410dbb0 2024-08-28T19:44:27.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.233+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/3835719441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8ea41a6df0 con 0x7f8ea410dbb0 2024-08-28T19:44:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:27 smithi016 ceph-mon[18556]: pgmap v1076: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 13 op/s 2024-08-28T19:44:27.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:27 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3835719441' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-08-28T19:44:27.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.514+0000 7f8ea17fa640 1 -- 172.21.15.16:0/3835719441 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v120) ==== 181+0+0 (secure 0 0 0) 0x7f8e9c0cd070 con 0x7f8ea410dbb0 2024-08-28T19:44:27.515 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-08-28T19:44:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.518+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/3835719441 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e900b4cd0 msgr2=0x7f8e900b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.518+0000 7f8ea9ca9640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e900b4cd0 0x7f8e900b71e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f8e94005fd0 tx=0x7f8e94005850 comp rx=0 tx=0).stop 2024-08-28T19:44:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.518+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/3835719441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea410dbb0 msgr2=0x7f8ea419faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.518+0000 7f8ea9ca9640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea410dbb0 0x7f8ea419faa0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c005cc0 tx=0x7f8e9c02f380 comp rx=0 tx=0).stop 2024-08-28T19:44:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.519+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/3835719441 shutdown_connections 2024-08-28T19:44:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.519+0000 7f8ea9ca9640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea410dbb0 0x7f8ea419faa0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.519+0000 7f8ea9ca9640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ea4105030 0x7f8ea419a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.519+0000 7f8ea9ca9640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f8e900b4cd0 0x7f8e900b71e0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.519+0000 7f8ea9ca9640 1 --2- 172.21.15.16:0/3835719441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4104600 0x7f8ea419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.519+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/3835719441 >> 172.21.15.16:0/3835719441 conn(0x7f8ea40794e0 msgr2=0x7f8ea40ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.520+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/3835719441 shutdown_connections 2024-08-28T19:44:27.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:27.520+0000 7f8ea9ca9640 1 -- 172.21.15.16:0/3835719441 wait complete. 2024-08-28T19:44:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:27 smithi028 ceph-mon[21769]: pgmap v1076: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 13 op/s 2024-08-28T19:44:27.559 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:27 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3835719441' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-08-28T19:44:27.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:27 smithi115 ceph-mon[22184]: pgmap v1076: 129 pgs: 129 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 13 op/s 2024-08-28T19:44:27.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:27 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3835719441' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-08-28T19:44:28.212 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-08-28T19:44:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:44:28.732 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:28 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/3835719441' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-08-28T19:44:28.732 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:28 smithi028 ceph-mon[21769]: osdmap e120: 12 total, 12 up, 12 in 2024-08-28T19:44:28.733 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:28 smithi028 ceph-mon[21769]: pgmap v1078: 97 pgs: 97 active+clean; 291 MiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:44:28.828 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:28 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/3835719441' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-08-28T19:44:28.828 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:28 smithi016 ceph-mon[18556]: osdmap e120: 12 total, 12 up, 12 in 2024-08-28T19:44:28.828 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:28 smithi016 ceph-mon[18556]: pgmap v1078: 97 pgs: 97 active+clean; 291 MiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:44:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:28 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/3835719441' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-08-28T19:44:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:28 smithi115 ceph-mon[22184]: osdmap e120: 12 total, 12 up, 12 in 2024-08-28T19:44:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:28 smithi115 ceph-mon[22184]: pgmap v1078: 97 pgs: 97 active+clean; 291 MiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:44:29.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.063+0000 7fae765b0640 1 -- 172.21.15.16:0/1360160762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae70109be0 msgr2=0x7fae70110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:29.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.064+0000 7fae765b0640 1 --2- 172.21.15.16:0/1360160762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae70109be0 0x7fae70110410 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fae68009ca0 tx=0x7fae6802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:44:29.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.064+0000 7fae765b0640 1 -- 172.21.15.16:0/1360160762 shutdown_connections 2024-08-28T19:44:29.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.064+0000 7fae765b0640 1 --2- 172.21.15.16:0/1360160762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae70109be0 0x7fae70110410 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:29.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.064+0000 7fae765b0640 1 --2- 172.21.15.16:0/1360160762 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae70109290 0x7fae701096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:29.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.064+0000 7fae765b0640 1 --2- 172.21.15.16:0/1360160762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae70103950 0x7fae70103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:29.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.064+0000 7fae765b0640 1 -- 172.21.15.16:0/1360160762 >> 172.21.15.16:0/1360160762 conn(0x7fae700ff3a0 msgr2=0x7fae701017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:29.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.064+0000 7fae765b0640 1 -- 172.21.15.16:0/1360160762 shutdown_connections 2024-08-28T19:44:29.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.064+0000 7fae765b0640 1 -- 172.21.15.16:0/1360160762 wait complete. 2024-08-28T19:44:29.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.064+0000 7fae765b0640 1 Processor -- start 2024-08-28T19:44:29.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.064+0000 7fae765b0640 1 -- start start 2024-08-28T19:44:29.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae765b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae70103950 0x7fae700754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:29.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae765b0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae70109290 0x7fae700759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:29.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae765b0640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae70109be0 0x7fae70075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:29.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae765b0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae701138a0 con 0x7fae70109290 2024-08-28T19:44:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae765b0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae70113720 con 0x7fae70103950 2024-08-28T19:44:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae765b0640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fae70113a20 con 0x7fae70109be0 2024-08-28T19:44:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae6f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae70109290 0x7fae700759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae6f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae70109290 0x7fae700759f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48436/0 (socket says 172.21.15.16:48436) 2024-08-28T19:44:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae6f7fe640 1 -- 172.21.15.16:0/493013710 learned_addr learned my addr 172.21.15.16:0/493013710 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:44:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae6ffff640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae70103950 0x7fae700754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae74b4d640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae70109be0 0x7fae70075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae6f7fe640 1 -- 172.21.15.16:0/493013710 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae70109be0 msgr2=0x7fae70075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae6f7fe640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae70109be0 0x7fae70075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae6f7fe640 1 -- 172.21.15.16:0/493013710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae70103950 msgr2=0x7fae700754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae6f7fe640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae70103950 0x7fae700754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae6f7fe640 1 -- 172.21.15.16:0/493013710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae701a84d0 con 0x7fae70109290 2024-08-28T19:44:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.065+0000 7fae6ffff640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae70103950 0x7fae700754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-28T19:44:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.066+0000 7fae6f7fe640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae70109290 0x7fae700759f0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fae6400d950 tx=0x7fae6400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.066+0000 7fae6d7fa640 1 -- 172.21.15.16:0/493013710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae64017070 con 0x7fae70109290 2024-08-28T19:44:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.066+0000 7fae765b0640 1 -- 172.21.15.16:0/493013710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae701a87c0 con 0x7fae70109290 2024-08-28T19:44:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.066+0000 7fae6d7fa640 1 -- 172.21.15.16:0/493013710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae64009c30 con 0x7fae70109290 2024-08-28T19:44:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.066+0000 7fae6d7fa640 1 -- 172.21.15.16:0/493013710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae640047a0 con 0x7fae70109290 2024-08-28T19:44:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.066+0000 7fae765b0640 1 -- 172.21.15.16:0/493013710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae701a8d00 con 0x7fae70109290 2024-08-28T19:44:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.067+0000 7fae765b0640 1 -- 172.21.15.16:0/493013710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae70103d60 con 0x7fae70109290 2024-08-28T19:44:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.069+0000 7fae6d7fa640 1 -- 172.21.15.16:0/493013710 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7fae64004020 con 0x7fae70109290 2024-08-28T19:44:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.069+0000 7fae6d7fa640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae580b4ca0 0x7fae580b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.069+0000 7fae6ffff640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae580b4ca0 0x7fae580b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.069+0000 7fae6d7fa640 1 -- 172.21.15.16:0/493013710 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9270+0+0 (secure 0 0 0) 0x7fae640e5860 con 0x7fae70109290 2024-08-28T19:44:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.070+0000 7fae6ffff640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae580b4ca0 0x7fae580b71b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fae70076f00 tx=0x7fae60005f90 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.072+0000 7fae6d7fa640 1 -- 172.21.15.16:0/493013710 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7fae64020020 con 0x7fae70109290 2024-08-28T19:44:29.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.260+0000 7fae765b0640 1 -- 172.21.15.16:0/493013710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fae7002e430 con 0x7fae70109290 2024-08-28T19:44:29.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.531+0000 7fae6d7fa640 1 -- 172.21.15.16:0/493013710 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v121) ==== 169+0+0 (secure 0 0 0) 0x7fae640ac360 con 0x7fae70109290 2024-08-28T19:44:29.533 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-08-28T19:44:29.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.535+0000 7fae765b0640 1 -- 172.21.15.16:0/493013710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae580b4ca0 msgr2=0x7fae580b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:29.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.535+0000 7fae765b0640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae580b4ca0 0x7fae580b71b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fae70076f00 tx=0x7fae60005f90 comp rx=0 tx=0).stop 2024-08-28T19:44:29.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.535+0000 7fae765b0640 1 -- 172.21.15.16:0/493013710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae70109290 msgr2=0x7fae700759f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:29.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.535+0000 7fae765b0640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae70109290 0x7fae700759f0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fae6400d950 tx=0x7fae6400de10 comp rx=0 tx=0).stop 2024-08-28T19:44:29.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.536+0000 7fae765b0640 1 -- 172.21.15.16:0/493013710 shutdown_connections 2024-08-28T19:44:29.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.536+0000 7fae765b0640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae70109be0 0x7fae70075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:29.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.536+0000 7fae765b0640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae70109290 0x7fae700759f0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:29.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.536+0000 7fae765b0640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7fae580b4ca0 0x7fae580b71b0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:29.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.536+0000 7fae765b0640 1 --2- 172.21.15.16:0/493013710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae70103950 0x7fae700754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:29.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.536+0000 7fae765b0640 1 -- 172.21.15.16:0/493013710 >> 172.21.15.16:0/493013710 conn(0x7fae700ff3a0 msgr2=0x7fae701017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:29.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.536+0000 7fae765b0640 1 -- 172.21.15.16:0/493013710 shutdown_connections 2024-08-28T19:44:29.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:29.537+0000 7fae765b0640 1 -- 172.21.15.16:0/493013710 wait complete. 2024-08-28T19:44:29.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:29 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/493013710' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-08-28T19:44:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:29 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/493013710' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-08-28T19:44:29.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:29 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/493013710' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-08-28T19:44:30.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6 shell --fsid eabc438a-6570-11ef-bcd4-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-08-28T19:44:30.585 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/mon.a/config 2024-08-28T19:44:30.705 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:30 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/493013710' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-08-28T19:44:30.705 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:30 smithi016 ceph-mon[18556]: osdmap e121: 12 total, 12 up, 12 in 2024-08-28T19:44:30.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:30 smithi016 ceph-mon[18556]: pgmap v1080: 33 pgs: 33 active+clean; 291 MiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:44:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:30 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/493013710' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-08-28T19:44:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:30 smithi028 ceph-mon[21769]: osdmap e121: 12 total, 12 up, 12 in 2024-08-28T19:44:30.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:30 smithi028 ceph-mon[21769]: pgmap v1080: 33 pgs: 33 active+clean; 291 MiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:44:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:30 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/493013710' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-08-28T19:44:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:30 smithi115 ceph-mon[22184]: osdmap e121: 12 total, 12 up, 12 in 2024-08-28T19:44:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:30 smithi115 ceph-mon[22184]: pgmap v1080: 33 pgs: 33 active+clean; 291 MiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:44:31.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.065+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2121539307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d00109b90 msgr2=0x7f5d001103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:31.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.065+0000 7f5d07dfe640 1 --2- 172.21.15.16:0/2121539307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d00109b90 0x7f5d001103c0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009ca0 tx=0x7f5cf802a8c0 comp rx=0 tx=0).stop 2024-08-28T19:44:31.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.065+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2121539307 shutdown_connections 2024-08-28T19:44:31.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.065+0000 7f5d07dfe640 1 --2- 172.21.15.16:0/2121539307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d00109b90 0x7f5d001103c0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:31.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.065+0000 7f5d07dfe640 1 --2- 172.21.15.16:0/2121539307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d00109240 0x7f5d00109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:31.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.065+0000 7f5d07dfe640 1 --2- 172.21.15.16:0/2121539307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d00103900 0x7f5d00103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:31.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.065+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2121539307 >> 172.21.15.16:0/2121539307 conn(0x7f5d0007b5f0 msgr2=0x7f5d0007ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:31.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.065+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2121539307 shutdown_connections 2024-08-28T19:44:31.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.065+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2121539307 wait complete. 2024-08-28T19:44:31.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.066+0000 7f5d07dfe640 1 Processor -- start 2024-08-28T19:44:31.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.066+0000 7f5d07dfe640 1 -- start start 2024-08-28T19:44:31.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.066+0000 7f5d07dfe640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d00103900 0x7f5d0019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:31.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.066+0000 7f5d07dfe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d00109240 0x7f5d0019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:31.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.066+0000 7f5d07dfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d00109b90 0x7f5d0019f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:31.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.066+0000 7f5d07dfe640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d00113850 con 0x7f5d00109240 2024-08-28T19:44:31.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.066+0000 7f5d07dfe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d001136d0 con 0x7f5d00109b90 2024-08-28T19:44:31.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.066+0000 7f5d07dfe640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5d001139d0 con 0x7f5d00103900 2024-08-28T19:44:31.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d05399640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d00109240 0x7f5d0019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:31.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d05399640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d00109240 0x7f5d0019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48460/0 (socket says 172.21.15.16:48460) 2024-08-28T19:44:31.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d05399640 1 -- 172.21.15.16:0/2370718593 learned_addr learned my addr 172.21.15.16:0/2370718593 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-08-28T19:44:31.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d05b9a640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d00103900 0x7f5d0019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:31.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d0639b640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d00109b90 0x7f5d0019f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d05399640 1 -- 172.21.15.16:0/2370718593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d00103900 msgr2=0x7f5d0019a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d05399640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d00103900 0x7f5d0019a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d05399640 1 -- 172.21.15.16:0/2370718593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d00109b90 msgr2=0x7f5d0019f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d05399640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d00109b90 0x7f5d0019f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d05399640 1 -- 172.21.15.16:0/2370718593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d001a0050 con 0x7f5d00109240 2024-08-28T19:44:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d05b9a640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d00103900 0x7f5d0019a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-28T19:44:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d05399640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d00109240 0x7f5d0019a740 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000c8c0 tx=0x7f5cf000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5cf6ffd640 1 -- 172.21.15.16:0/2370718593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0007bb0 con 0x7f5d00109240 2024-08-28T19:44:31.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2370718593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d001a0340 con 0x7f5d00109240 2024-08-28T19:44:31.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5cf6ffd640 1 -- 172.21.15.16:0/2370718593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cf0007d50 con 0x7f5d00109240 2024-08-28T19:44:31.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5cf6ffd640 1 -- 172.21.15.16:0/2370718593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf00187c0 con 0x7f5d00109240 2024-08-28T19:44:31.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.067+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2370718593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d00075390 con 0x7f5d00109240 2024-08-28T19:44:31.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.068+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2370718593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d00104b80 con 0x7f5d00109240 2024-08-28T19:44:31.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.070+0000 7f5cf6ffd640 1 -- 172.21.15.16:0/2370718593 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 152708+0+0 (secure 0 0 0) 0x7f5cf000fa20 con 0x7f5d00109240 2024-08-28T19:44:31.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.071+0000 7f5cf6ffd640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cec0b90e0 0x7f5cec0bb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-28T19:44:31.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.071+0000 7f5d05b9a640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cec0b90e0 0x7f5cec0bb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-28T19:44:31.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.071+0000 7f5cf6ffd640 1 -- 172.21.15.16:0/2370718593 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8711+0+0 (secure 0 0 0) 0x7f5cf00e5a40 con 0x7f5d00109240 2024-08-28T19:44:31.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.071+0000 7f5d05b9a640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cec0b90e0 0x7f5cec0bb5f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0058c0 tx=0x7f5cfc005ec0 comp rx=0 tx=0).ready entity=mgr.14220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-28T19:44:31.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.074+0000 7f5cf6ffd640 1 -- 172.21.15.16:0/2370718593 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196895 (secure 0 0 0) 0x7f5cf0014030 con 0x7f5d00109240 2024-08-28T19:44:31.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.259+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2370718593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5d00103d10 con 0x7f5d00109240 2024-08-28T19:44:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.543+0000 7f5cf6ffd640 1 -- 172.21.15.16:0/2370718593 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v122) ==== 178+0+0 (secure 0 0 0) 0x7f5d00103d10 con 0x7f5d00109240 2024-08-28T19:44:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' removed 2024-08-28T19:44:31.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.546+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2370718593 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cec0b90e0 msgr2=0x7f5cec0bb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:31.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.546+0000 7f5d07dfe640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cec0b90e0 0x7f5cec0bb5f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0058c0 tx=0x7f5cfc005ec0 comp rx=0 tx=0).stop 2024-08-28T19:44:31.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.546+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2370718593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d00109240 msgr2=0x7f5d0019a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-28T19:44:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.546+0000 7f5d07dfe640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d00109240 0x7f5d0019a740 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000c8c0 tx=0x7f5cf000cd80 comp rx=0 tx=0).stop 2024-08-28T19:44:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.546+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2370718593 shutdown_connections 2024-08-28T19:44:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.546+0000 7f5d07dfe640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.16:6800/2594037529,v1:172.21.15.16:6801/2594037529] conn(0x7f5cec0b90e0 0x7f5cec0bb5f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.546+0000 7f5d07dfe640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d00109b90 0x7f5d0019f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.546+0000 7f5d07dfe640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d00109240 0x7f5d0019a740 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.546+0000 7f5d07dfe640 1 --2- 172.21.15.16:0/2370718593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d00103900 0x7f5d0019a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-28T19:44:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.546+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2370718593 >> 172.21.15.16:0/2370718593 conn(0x7f5d0007b5f0 msgr2=0x7f5d001051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-28T19:44:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.547+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2370718593 shutdown_connections 2024-08-28T19:44:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-08-28T19:44:31.547+0000 7f5d07dfe640 1 -- 172.21.15.16:0/2370718593 wait complete. 2024-08-28T19:44:31.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:31 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2370718593' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-08-28T19:44:31.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:31 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2370718593' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-08-28T19:44:31.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:31 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2370718593' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-08-28T19:44:32.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-28T19:44:32.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-28T19:44:32.350 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-28T19:44:32.426 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-08-28T19:44:32.427 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-28T19:44:32.443 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-28T19:44:32.461 DEBUG:teuthology.orchestra.run.smithi115:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-28T19:44:32.477 INFO:tasks.cephadm:Stopping all daemons... 2024-08-28T19:44:32.478 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-08-28T19:44:32.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.a 2024-08-28T19:44:32.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:32 smithi016 systemd[1]: Stopping Ceph mon.a for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:44:32.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:32 smithi016 ceph-mon[18556]: from='client.? 172.21.15.16:0/2370718593' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-08-28T19:44:32.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:32 smithi016 ceph-mon[18556]: osdmap e122: 12 total, 12 up, 12 in 2024-08-28T19:44:32.808 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:32 smithi016 ceph-mon[18556]: pgmap v1082: 1 pgs: 1 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:44:32.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:32 smithi028 ceph-mon[21769]: from='client.? 172.21.15.16:0/2370718593' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-08-28T19:44:32.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:32 smithi028 ceph-mon[21769]: osdmap e122: 12 total, 12 up, 12 in 2024-08-28T19:44:32.809 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:32 smithi028 ceph-mon[21769]: pgmap v1082: 1 pgs: 1 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:44:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:32 smithi115 ceph-mon[22184]: from='client.? 172.21.15.16:0/2370718593' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-08-28T19:44:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:32 smithi115 ceph-mon[22184]: osdmap e122: 12 total, 12 up, 12 in 2024-08-28T19:44:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:32 smithi115 ceph-mon[22184]: pgmap v1082: 1 pgs: 1 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-28T19:44:33.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:32 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a[18552]: 2024-08-28T19:44:32.805+0000 7f46658c9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:44:33.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:32 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a[18552]: 2024-08-28T19:44:32.805+0000 7f46658c9640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-08-28T19:44:33.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:32 smithi016 podman[130674]: 2024-08-28 19:44:32.836787364 +0000 UTC m=+0.150019045 container died 1408e30d070d47f0671111208cd536ed1611b528dc95b5824ce4b923cdb1f80a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.1, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-08-28T19:44:33.475 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:33 smithi016 podman[130674]: 2024-08-28 19:44:33.181671576 +0000 UTC m=+0.494903247 container cleanup 1408e30d070d47f0671111208cd536ed1611b528dc95b5824ce4b923cdb1f80a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, org.label-schema.build-date=20240731) 2024-08-28T19:44:33.475 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:33 smithi016 bash[130674]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a 2024-08-28T19:44:33.651 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.a.service' 2024-08-28T19:44:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:33 smithi016 podman[130686]: 2024-08-28 19:44:33.475050072 +0000 UTC m=+0.640367508 container remove 1408e30d070d47f0671111208cd536ed1611b528dc95b5824ce4b923cdb1f80a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-08-28T19:44:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:33 smithi016 systemd[1]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.a.service: Deactivated successfully. 2024-08-28T19:44:33.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:33 smithi016 systemd[1]: Stopped Ceph mon.a for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:44:33.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Aug 28 19:44:33 smithi016 systemd[1]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.a.service: Consumed 32.277s CPU time. 2024-08-28T19:44:33.922 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:44:33.923 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-08-28T19:44:33.923 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-08-28T19:44:33.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.b 2024-08-28T19:44:34.311 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:33 smithi028 systemd[1]: Stopping Ceph mon.b for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:44:34.619 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:34 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-b[21765]: 2024-08-28T19:44:34.311+0000 7f4480ac9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:44:34.619 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:34 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-b[21765]: 2024-08-28T19:44:34.311+0000 7f4480ac9640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-08-28T19:44:34.619 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:34 smithi028 podman[61065]: 2024-08-28 19:44:34.332036866 +0000 UTC m=+0.153654438 container died 684e32f5e71d61a34699519cc1f5717518e193d85d1148677366366feb218de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-28T19:44:35.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:34 smithi028 podman[61065]: 2024-08-28 19:44:34.738558879 +0000 UTC m=+0.560176431 container cleanup 684e32f5e71d61a34699519cc1f5717518e193d85d1148677366366feb218de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-08-28T19:44:35.059 INFO:journalctl@ceph.mon.b.smithi028.stdout:Aug 28 19:44:34 smithi028 bash[61065]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-b 2024-08-28T19:44:35.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.b.service' 2024-08-28T19:44:35.370 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:44:35.370 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-08-28T19:44:35.370 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-08-28T19:44:35.370 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.c 2024-08-28T19:44:35.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:35 smithi115 systemd[1]: Stopping Ceph mon.c for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:44:36.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:35 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-c[22180]: 2024-08-28T19:44:35.768+0000 7f9e086f7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:44:36.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:35 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-c[22180]: 2024-08-28T19:44:35.768+0000 7f9e086f7640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-08-28T19:44:36.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:35 smithi115 podman[61139]: 2024-08-28 19:44:35.912829685 +0000 UTC m=+0.252643851 container died 703bcb8f9d936ede9320098d38098fe9d864a2437a5465cebaca12c164d19f92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-c, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux ) 2024-08-28T19:44:36.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:36 smithi115 podman[61139]: 2024-08-28 19:44:36.344312879 +0000 UTC m=+0.684127032 container cleanup 703bcb8f9d936ede9320098d38098fe9d864a2437a5465cebaca12c164d19f92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux ) 2024-08-28T19:44:36.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:Aug 28 19:44:36 smithi115 bash[61139]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mon-c 2024-08-28T19:44:36.791 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mon.c.service' 2024-08-28T19:44:36.838 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:44:36.838 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-08-28T19:44:36.839 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-08-28T19:44:36.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.x 2024-08-28T19:44:37.169 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:44:36 smithi016 systemd[1]: Stopping Ceph mgr.x for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:44:37.500 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:44:37 smithi016 podman[130797]: 2024-08-28 19:44:37.177926702 +0000 UTC m=+0.145384655 container died 63620dafa962f73710b89df2bde3866d6dc59c690bec0ec7673a72ac9c492d45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.1) 2024-08-28T19:44:38.000 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:44:37 smithi016 podman[130797]: 2024-08-28 19:44:37.863659209 +0000 UTC m=+0.831117163 container cleanup 63620dafa962f73710b89df2bde3866d6dc59c690bec0ec7673a72ac9c492d45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d19e6118f7015f0047264006cb0c873528eb3ae6, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-08-28T19:44:38.000 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Aug 28 19:44:37 smithi016 bash[130797]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-x 2024-08-28T19:44:38.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.x.service' 2024-08-28T19:44:38.327 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:44:38.328 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-08-28T19:44:38.328 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-08-28T19:44:38.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.y 2024-08-28T19:44:38.661 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:44:38 smithi028 systemd[1]: Stopping Ceph mgr.y for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:44:38.959 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:44:38 smithi028 podman[61183]: 2024-08-28 19:44:38.660914142 +0000 UTC m=+0.139523879 container died fed856f47993fabfbb1f2a3a9a9e6339a9a463b91b4b51ec508dd2be09c557c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-y, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.1) 2024-08-28T19:44:39.309 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:44:39 smithi028 podman[61183]: 2024-08-28 19:44:39.065948392 +0000 UTC m=+0.544558126 container cleanup fed856f47993fabfbb1f2a3a9a9e6339a9a463b91b4b51ec508dd2be09c557c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-y, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-28T19:44:39.309 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:44:39 smithi028 bash[61183]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-y 2024-08-28T19:44:39.730 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:44:39 smithi028 podman[61195]: 2024-08-28 19:44:39.450988179 +0000 UTC m=+0.784689787 container remove fed856f47993fabfbb1f2a3a9a9e6339a9a463b91b4b51ec508dd2be09c557c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-y, io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-28T19:44:39.730 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Aug 28 19:44:39 smithi028 systemd[1]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-08-28T19:44:39.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.y.service' 2024-08-28T19:44:39.846 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:44:39.846 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-08-28T19:44:39.847 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-08-28T19:44:39.847 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.z 2024-08-28T19:44:40.196 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:44:39 smithi115 systemd[1]: Stopping Ceph mgr.z for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:44:40.460 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:44:40 smithi115 podman[61271]: 2024-08-28 19:44:40.195319904 +0000 UTC m=+0.142007243 container died b6a0a772a01ca7d7a2d0999f76cd7bced3b48b1f0c16088276b3585b614eddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-08-28T19:44:40.835 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:44:40 smithi115 podman[61271]: 2024-08-28 19:44:40.54980256 +0000 UTC m=+0.496489895 container cleanup b6a0a772a01ca7d7a2d0999f76cd7bced3b48b1f0c16088276b3585b614eddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-z, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-08-28T19:44:40.835 INFO:journalctl@ceph.mgr.z.smithi115.stdout:Aug 28 19:44:40 smithi115 bash[61271]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mgr-z 2024-08-28T19:44:40.967 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mgr.z.service' 2024-08-28T19:44:41.003 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:44:41.003 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-08-28T19:44:41.003 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-08-28T19:44:41.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.0 2024-08-28T19:44:41.311 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:41 smithi016 systemd[1]: Stopping Ceph osd.0 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:44:41.750 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:41 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0[36367]: 2024-08-28T19:44:41.309+0000 7fd053e69640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:44:41.750 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:41 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0[36367]: 2024-08-28T19:44:41.309+0000 7fd053e69640 -1 osd.0 122 *** Got signal Terminated *** 2024-08-28T19:44:41.750 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:41 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0[36367]: 2024-08-28T19:44:41.309+0000 7fd053e69640 -1 osd.0 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:44:47.154 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:46 smithi016 podman[130918]: 2024-08-28 19:44:46.859332074 +0000 UTC m=+5.659106756 container died e03638c66c6898454f775a2dcd44714e44280d870d283bbf9b59b27ae8baff97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-08-28T19:44:47.500 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:47 smithi016 podman[130918]: 2024-08-28 19:44:47.352681793 +0000 UTC m=+6.152456465 container cleanup e03638c66c6898454f775a2dcd44714e44280d870d283bbf9b59b27ae8baff97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-28T19:44:47.500 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:47 smithi016 bash[130918]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0 2024-08-28T19:44:47.952 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:47 smithi016 podman[130934]: 2024-08-28 19:44:47.638236844 +0000 UTC m=+0.774291056 container remove e03638c66c6898454f775a2dcd44714e44280d870d283bbf9b59b27ae8baff97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-08-28T19:44:48.250 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:47 smithi016 podman[131022]: 2024-08-28 19:44:47.855298201 +0000 UTC m=+0.015791453 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:44:48.250 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:47 smithi016 podman[131022]: 2024-08-28 19:44:47.955948604 +0000 UTC m=+0.116441889 container create ac496be21735dc39a5fcc530841a00c177daa358b6117df55a08ea443c156f56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-28T19:44:48.250 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:48 smithi016 podman[131022]: 2024-08-28 19:44:48.191841817 +0000 UTC m=+0.352335058 container init ac496be21735dc39a5fcc530841a00c177daa358b6117df55a08ea443c156f56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, io.buildah.version=1.37.1) 2024-08-28T19:44:48.251 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:48 smithi016 podman[131022]: 2024-08-28 19:44:48.195107856 +0000 UTC m=+0.355601101 container start ac496be21735dc39a5fcc530841a00c177daa358b6117df55a08ea443c156f56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-08-28T19:44:48.746 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:48 smithi016 podman[131022]: 2024-08-28 19:44:48.308840441 +0000 UTC m=+0.469333683 container attach ac496be21735dc39a5fcc530841a00c177daa358b6117df55a08ea443c156f56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-08-28T19:44:48.746 INFO:journalctl@ceph.osd.0.smithi016.stdout:Aug 28 19:44:48 smithi016 podman[131022]: 2024-08-28 19:44:48.389398148 +0000 UTC m=+0.549891395 container died ac496be21735dc39a5fcc530841a00c177daa358b6117df55a08ea443c156f56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-28T19:44:49.138 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.0.service' 2024-08-28T19:44:49.182 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:44:49.182 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-08-28T19:44:49.182 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-08-28T19:44:49.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.1 2024-08-28T19:44:49.530 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:49 smithi028 systemd[1]: Stopping Ceph osd.1 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:44:49.809 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:49 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1[27359]: 2024-08-28T19:44:49.528+0000 7f416e3c7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:44:49.809 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:49 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1[27359]: 2024-08-28T19:44:49.528+0000 7f416e3c7640 -1 osd.1 122 *** Got signal Terminated *** 2024-08-28T19:44:49.809 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:49 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1[27359]: 2024-08-28T19:44:49.529+0000 7f416e3c7640 -1 osd.1 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:44:55.281 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:54 smithi028 podman[61310]: 2024-08-28 19:44:54.992733044 +0000 UTC m=+5.611102688 container died 74e0fe42a0dd8233b85c7f87d28c18719b007ebf7c2811b56ce407d8bb93a218 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1) 2024-08-28T19:44:55.559 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:55 smithi028 podman[61310]: 2024-08-28 19:44:55.379842156 +0000 UTC m=+5.998211798 container cleanup 74e0fe42a0dd8233b85c7f87d28c18719b007ebf7c2811b56ce407d8bb93a218 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-08-28T19:44:55.559 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:55 smithi028 bash[61310]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1 2024-08-28T19:44:55.992 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:55 smithi028 podman[61325]: 2024-08-28 19:44:55.648257249 +0000 UTC m=+0.647355030 container remove 74e0fe42a0dd8233b85c7f87d28c18719b007ebf7c2811b56ce407d8bb93a218 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, org.label-schema.license=GPLv2) 2024-08-28T19:44:56.253 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:55 smithi028 podman[61419]: 2024-08-28 19:44:55.894777851 +0000 UTC m=+0.016018727 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:44:56.253 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:56 smithi028 podman[61419]: 2024-08-28 19:44:56.015742785 +0000 UTC m=+0.136983658 container create 506de53c8cdf52074cf0f926c535ce2dc3a69d68a16dffcb223431f2a10897d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-28T19:44:56.559 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:56 smithi028 podman[61419]: 2024-08-28 19:44:56.252394294 +0000 UTC m=+0.373635169 container init 506de53c8cdf52074cf0f926c535ce2dc3a69d68a16dffcb223431f2a10897d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-28T19:44:56.560 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:56 smithi028 podman[61419]: 2024-08-28 19:44:56.256039832 +0000 UTC m=+0.377280707 container start 506de53c8cdf52074cf0f926c535ce2dc3a69d68a16dffcb223431f2a10897d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.1) 2024-08-28T19:44:56.560 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:56 smithi028 podman[61419]: 2024-08-28 19:44:56.360881415 +0000 UTC m=+0.482122297 container attach 506de53c8cdf52074cf0f926c535ce2dc3a69d68a16dffcb223431f2a10897d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.1, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-08-28T19:44:56.989 INFO:journalctl@ceph.osd.1.smithi028.stdout:Aug 28 19:44:56 smithi028 podman[61419]: 2024-08-28 19:44:56.631732117 +0000 UTC m=+0.752972994 container died 506de53c8cdf52074cf0f926c535ce2dc3a69d68a16dffcb223431f2a10897d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-08-28T19:44:57.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.1.service' 2024-08-28T19:44:57.376 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:44:57.376 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-08-28T19:44:57.377 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-08-28T19:44:57.377 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.2 2024-08-28T19:44:57.688 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:44:57 smithi115 systemd[1]: Stopping Ceph osd.2 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:44:58.120 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:44:57 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2[28467]: 2024-08-28T19:44:57.686+0000 7f49edb3f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:44:58.121 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:44:57 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2[28467]: 2024-08-28T19:44:57.686+0000 7f49edb3f640 -1 osd.2 122 *** Got signal Terminated *** 2024-08-28T19:44:58.121 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:44:57 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2[28467]: 2024-08-28T19:44:57.686+0000 7f49edb3f640 -1 osd.2 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:45:03.472 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:45:03 smithi115 podman[61395]: 2024-08-28 19:45:03.220068344 +0000 UTC m=+5.653036767 container died 782f2e07fbd8e3340e5ef0cc843bb7a6eaddf87b0143286606299ae38b113d0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-08-28T19:45:03.789 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:45:03 smithi115 podman[61395]: 2024-08-28 19:45:03.553064882 +0000 UTC m=+5.986033303 container cleanup 782f2e07fbd8e3340e5ef0cc843bb7a6eaddf87b0143286606299ae38b113d0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, CEPH_POINT_RELEASE=) 2024-08-28T19:45:03.789 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:45:03 smithi115 bash[61395]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2 2024-08-28T19:45:04.120 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:45:03 smithi115 podman[61413]: 2024-08-28 19:45:03.788082537 +0000 UTC m=+0.570313654 container remove 782f2e07fbd8e3340e5ef0cc843bb7a6eaddf87b0143286606299ae38b113d0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-08-28T19:45:04.467 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:45:04 smithi115 podman[61504]: 2024-08-28 19:45:04.092369356 +0000 UTC m=+0.015249878 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:45:04.468 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:45:04 smithi115 podman[61504]: 2024-08-28 19:45:04.230697685 +0000 UTC m=+0.153578199 container create 419779eb5bb3f575552f55c3e01faee5d76f55fb80b8d46c928fa98666350374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-08-28T19:45:04.468 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:45:04 smithi115 podman[61504]: 2024-08-28 19:45:04.46717057 +0000 UTC m=+0.390051083 container init 419779eb5bb3f575552f55c3e01faee5d76f55fb80b8d46c928fa98666350374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-08-28T19:45:04.785 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:45:04 smithi115 podman[61504]: 2024-08-28 19:45:04.470667987 +0000 UTC m=+0.393548502 container start 419779eb5bb3f575552f55c3e01faee5d76f55fb80b8d46c928fa98666350374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-08-28T19:45:04.786 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:45:04 smithi115 podman[61504]: 2024-08-28 19:45:04.600495482 +0000 UTC m=+0.523376017 container attach 419779eb5bb3f575552f55c3e01faee5d76f55fb80b8d46c928fa98666350374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.1) 2024-08-28T19:45:05.087 INFO:journalctl@ceph.osd.2.smithi115.stdout:Aug 28 19:45:04 smithi115 podman[61504]: 2024-08-28 19:45:04.785419071 +0000 UTC m=+0.708299585 container died 419779eb5bb3f575552f55c3e01faee5d76f55fb80b8d46c928fa98666350374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-08-28T19:45:05.417 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.2.service' 2024-08-28T19:45:05.457 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:45:05.457 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-08-28T19:45:05.457 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-08-28T19:45:05.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.3 2024-08-28T19:45:05.750 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:05 smithi016 systemd[1]: Stopping Ceph osd.3 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:45:06.250 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:05 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3[42974]: 2024-08-28T19:45:05.766+0000 7fca132d9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:45:06.251 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:05 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3[42974]: 2024-08-28T19:45:05.766+0000 7fca132d9640 -1 osd.3 122 *** Got signal Terminated *** 2024-08-28T19:45:06.251 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:05 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3[42974]: 2024-08-28T19:45:05.766+0000 7fca132d9640 -1 osd.3 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:45:08.042 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:07 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:07.714+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:08.500 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:08 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3[42974]: 2024-08-28T19:45:08.152+0000 7fca0b8bc640 -1 osd.3 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:46.071231+0000 front 2024-08-28T19:44:46.071182+0000 (oldest deadline 2024-08-28T19:45:07.171375+0000) 2024-08-28T19:45:09.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:08 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:08.682+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:09.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:08 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:08.722+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:09.500 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3[42974]: 2024-08-28T19:45:09.177+0000 7fca0b8bc640 -1 osd.3 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:46.071231+0000 front 2024-08-28T19:44:46.071182+0000 (oldest deadline 2024-08-28T19:45:07.171375+0000) 2024-08-28T19:45:10.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:09 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:09.643+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:10.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:09 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:09.725+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:10.121 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:09.838+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:10.500 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:10 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3[42974]: 2024-08-28T19:45:10.204+0000 7fca0b8bc640 -1 osd.3 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:46.071231+0000 front 2024-08-28T19:44:46.071182+0000 (oldest deadline 2024-08-28T19:45:07.171375+0000) 2024-08-28T19:45:10.870 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:10 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:10.443+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:11.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:10 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:10.611+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:11.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:10 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:10.736+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:11.370 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:10 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:10.873+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:11.659 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:11 smithi016 podman[131135]: 2024-08-28 19:45:11.342247925 +0000 UTC m=+5.680750640 container died 9c8828eb6d76f9eba6dde3486713473b78070ad351fc9128399d44bcb2dad313 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-08-28T19:45:11.660 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:11 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:11.568+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:11.870 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:11 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:11.431+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:12.000 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:11 smithi016 podman[131135]: 2024-08-28 19:45:11.835918813 +0000 UTC m=+6.174421527 container cleanup 9c8828eb6d76f9eba6dde3486713473b78070ad351fc9128399d44bcb2dad313 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-08-28T19:45:12.000 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:11 smithi016 bash[131135]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3 2024-08-28T19:45:12.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:11 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:11.711+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:12.371 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:11 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:11.878+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:12.447 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:12 smithi016 podman[131152]: 2024-08-28 19:45:12.112778871 +0000 UTC m=+0.766304427 container remove 9c8828eb6d76f9eba6dde3486713473b78070ad351fc9128399d44bcb2dad313 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-28T19:45:12.700 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:12 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:12.610+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:12.700 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:12 smithi016 podman[131236]: 2024-08-28 19:45:12.447068669 +0000 UTC m=+0.106995647 container create ec16b5df4438d91a926aee95d59a2c116129c03155a97f2475f45ed2f94ee7ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.1) 2024-08-28T19:45:12.701 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:12 smithi016 podman[131236]: 2024-08-28 19:45:12.35621338 +0000 UTC m=+0.016140371 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:45:12.701 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:12 smithi016 podman[131236]: 2024-08-28 19:45:12.699770485 +0000 UTC m=+0.359697465 container init ec16b5df4438d91a926aee95d59a2c116129c03155a97f2475f45ed2f94ee7ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, org.label-schema.build-date=20240731, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-08-28T19:45:12.844 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:12 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:12.437+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:13.000 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:12 smithi016 podman[131236]: 2024-08-28 19:45:12.703288371 +0000 UTC m=+0.363215352 container start ec16b5df4438d91a926aee95d59a2c116129c03155a97f2475f45ed2f94ee7ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-08-28T19:45:13.001 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:12 smithi016 podman[131236]: 2024-08-28 19:45:12.799842253 +0000 UTC m=+0.459769240 container attach ec16b5df4438d91a926aee95d59a2c116129c03155a97f2475f45ed2f94ee7ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3-deactivate, ceph=True, io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_CLEAN=True) 2024-08-28T19:45:13.001 INFO:journalctl@ceph.osd.3.smithi016.stdout:Aug 28 19:45:12 smithi016 podman[131236]: 2024-08-28 19:45:12.896409815 +0000 UTC m=+0.556336814 container died ec16b5df4438d91a926aee95d59a2c116129c03155a97f2475f45ed2f94ee7ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-08-28T19:45:13.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:12 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:12.737+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:13.120 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:12 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:12.843+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:13.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.3.service' 2024-08-28T19:45:13.539 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:45:13.539 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-08-28T19:45:13.539 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-08-28T19:45:13.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.4 2024-08-28T19:45:13.750 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:13 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:13.600+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:13.835 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:13 smithi028 systemd[1]: Stopping Ceph osd.4 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:45:13.836 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:13 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:13.717+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:13.855 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:13 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:13.482+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:14.120 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:13 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:13.855+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:14.309 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:13 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4[34221]: 2024-08-28T19:45:13.834+0000 7fd91f9bb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:45:14.309 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:13 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4[34221]: 2024-08-28T19:45:13.834+0000 7fd91f9bb640 -1 osd.4 122 *** Got signal Terminated *** 2024-08-28T19:45:14.309 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:13 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4[34221]: 2024-08-28T19:45:13.834+0000 7fd91f9bb640 -1 osd.4 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:45:14.810 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:14 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:14.475+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:15.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:14 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:14.556+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:15.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:14 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:14.727+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:15.121 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:14 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:14.809+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:15.817 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:15 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:15.474+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:16.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:15 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:15.517+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:16.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:15 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:15.517+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:16.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:15 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:15.756+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:16.095 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:15 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:15.816+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:16.560 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:16 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:16.202+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:16.788 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:16 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:16.442+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:16.944 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:16 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:16.553+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:16.944 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:16 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:16.553+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:17.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:16 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:16.764+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:17.120 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:16 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:16.787+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:17.250 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:16 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:16.943+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:17.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:17 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:17.249+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:17.764 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:17 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:17.407+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:17.966 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:17 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:17.565+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:17.967 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:17 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:17.565+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:18.120 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:17 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:17.763+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:18.121 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:17 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:17.763+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.437204+0000 front 2024-08-28T19:44:54.437291+0000 (oldest deadline 2024-08-28T19:45:17.337174+0000) 2024-08-28T19:45:18.222 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:17 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:17.812+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:18.250 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:17 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:17.964+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:18.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:18 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:18.221+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:18.739 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:18 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:18.454+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:18.750 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:18 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:18.606+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:18.750 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:18 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:18.606+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:19.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:18 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:18.801+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:19.120 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:18 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:18.738+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:19.121 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:18 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:18.738+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.437204+0000 front 2024-08-28T19:44:54.437291+0000 (oldest deadline 2024-08-28T19:45:17.337174+0000) 2024-08-28T19:45:19.250 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:18 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:18.946+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:19.546 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:19 smithi028 podman[61533]: 2024-08-28 19:45:19.244307446 +0000 UTC m=+5.506819752 container died 590ecde2c25ba12b51c8b87112fc7a1ea1ac42acba7ecd7deb19d34c51b06544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-08-28T19:45:19.546 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:19 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:19.266+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:19.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:19 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:19.459+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:19.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:19 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:19.459+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:19.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:19 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:19.223+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:19.809 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:19 smithi028 podman[61533]: 2024-08-28 19:45:19.635160891 +0000 UTC m=+5.897673198 container cleanup 590ecde2c25ba12b51c8b87112fc7a1ea1ac42acba7ecd7deb19d34c51b06544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-08-28T19:45:19.809 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:19 smithi028 bash[61533]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4 2024-08-28T19:45:19.810 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:19 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:19.785+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:19.810 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:19 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:19.785+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:19.927 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:19 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:19.617+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:19.927 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:19 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:19.617+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:20.121 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:19 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:19.697+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:20.121 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:19 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:19.697+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.437204+0000 front 2024-08-28T19:44:54.437291+0000 (oldest deadline 2024-08-28T19:45:17.337174+0000) 2024-08-28T19:45:20.196 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:19 smithi028 podman[61548]: 2024-08-28 19:45:19.886825923 +0000 UTC m=+0.644025957 container remove 590ecde2c25ba12b51c8b87112fc7a1ea1ac42acba7ecd7deb19d34c51b06544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-28T19:45:20.252 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:19 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:19.925+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:20.507 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:20 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:20.267+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:20.507 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:20 smithi028 podman[61638]: 2024-08-28 19:45:20.196096516 +0000 UTC m=+0.112470803 container create ff8387991bfd329208d0e97caf5161fcbe92270def15e57a74efdd3ee830bdb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.1, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-08-28T19:45:20.507 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:20 smithi028 podman[61638]: 2024-08-28 19:45:20.099516091 +0000 UTC m=+0.015890381 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:45:20.508 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:20 smithi028 podman[61638]: 2024-08-28 19:45:20.424013234 +0000 UTC m=+0.340387521 container init ff8387991bfd329208d0e97caf5161fcbe92270def15e57a74efdd3ee830bdb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4-deactivate, ceph=True, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-08-28T19:45:20.508 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:20 smithi028 podman[61638]: 2024-08-28 19:45:20.427473672 +0000 UTC m=+0.343847960 container start ff8387991bfd329208d0e97caf5161fcbe92270def15e57a74efdd3ee830bdb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.1, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731) 2024-08-28T19:45:20.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:20 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:20.227+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:20.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:20 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:20.452+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:20.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:20 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:20.452+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:20.799 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:20 smithi028 podman[61638]: 2024-08-28 19:45:20.506831242 +0000 UTC m=+0.423205530 container attach ff8387991bfd329208d0e97caf5161fcbe92270def15e57a74efdd3ee830bdb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-28T19:45:20.799 INFO:journalctl@ceph.osd.4.smithi028.stdout:Aug 28 19:45:20 smithi028 podman[61638]: 2024-08-28 19:45:20.617506031 +0000 UTC m=+0.533880319 container died ff8387991bfd329208d0e97caf5161fcbe92270def15e57a74efdd3ee830bdb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-28T19:45:21.000 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:20 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:20.905+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:21.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:20 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:20.665+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:21.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:20 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:20.665+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:21.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:20 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:20.797+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:21.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:20 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:20.797+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:21.120 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:20 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:20.688+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:21.121 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:20 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:20.688+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.437204+0000 front 2024-08-28T19:44:54.437291+0000 (oldest deadline 2024-08-28T19:45:17.337174+0000) 2024-08-28T19:45:21.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.4.service' 2024-08-28T19:45:21.271 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:45:21.271 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-08-28T19:45:21.272 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-08-28T19:45:21.272 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.5 2024-08-28T19:45:21.466 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:21 smithi115 systemd[1]: Stopping Ceph osd.5 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:45:21.467 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:21 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:21.214+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:21.467 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:21 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:21.463+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:21.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:21 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:21.225+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:21.870 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:21 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:21.463+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:21.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:21 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:21.589+0000 7f093eaf0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:45:21.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:21 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:21.589+0000 7f093eaf0640 -1 osd.5 122 *** Got signal Terminated *** 2024-08-28T19:45:21.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:21 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:21.589+0000 7f093eaf0640 -1 osd.5 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:45:21.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:21 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:21.700+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:21.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:21 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:21.700+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.437204+0000 front 2024-08-28T19:44:54.437291+0000 (oldest deadline 2024-08-28T19:45:17.337174+0000) 2024-08-28T19:45:21.941 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:21 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:21.686+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:21.942 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:21 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:21.686+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:22.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:21 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:21.771+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:22.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:21 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:21.771+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:22.250 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:21 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:21.940+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:22.431 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:22 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:22.169+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:22.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:22 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:22.235+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:22.683 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:22 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:22.429+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:22.683 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:22 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:22.429+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:22.967 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:22 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:22.715+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:22.967 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:22 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:22.715+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:23.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:22 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:22.793+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:23.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:22 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:22.793+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:23.121 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:22 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:22.682+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:23.121 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:22 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:22.682+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.437204+0000 front 2024-08-28T19:44:54.437291+0000 (oldest deadline 2024-08-28T19:45:17.337174+0000) 2024-08-28T19:45:23.250 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:22 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:22.965+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:23.479 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:23.171+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:23.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:23.229+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:23.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:23.682+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:23.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:23.682+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.437204+0000 front 2024-08-28T19:44:54.437291+0000 (oldest deadline 2024-08-28T19:45:17.337174+0000) 2024-08-28T19:45:23.871 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:23.478+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:23.871 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:23 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:23.478+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:24.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:23.730+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:24.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:23.730+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:24.001 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:23.940+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:24.001 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:23 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:23.940+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:58.271344+0000 front 2024-08-28T19:44:58.271402+0000 (oldest deadline 2024-08-28T19:45:23.571352+0000) 2024-08-28T19:45:24.182 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:23.817+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:24.183 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:23 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:23.817+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:24.488 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:24 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:24.176+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:24.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:24 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:24.182+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:24.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:24 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:24.646+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:24.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:24 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:24.646+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.437204+0000 front 2024-08-28T19:44:54.437291+0000 (oldest deadline 2024-08-28T19:45:17.337174+0000) 2024-08-28T19:45:24.872 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:24 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:24.486+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:24.872 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:24 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:24.486+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:24.872 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:24 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:24.486+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:24.985 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:24 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:24.691+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:24.986 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:24 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:24.691+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:25.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:24 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:24.776+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:25.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:24 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:24.776+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:25.250 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:24 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:24.984+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:25.250 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:24 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:24.984+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:58.271344+0000 front 2024-08-28T19:44:58.271402+0000 (oldest deadline 2024-08-28T19:45:23.571352+0000) 2024-08-28T19:45:25.528 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:25 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:25.207+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:25.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:25 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:25.204+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:25.872 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:25 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:25.633+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.735445+0000 front 2024-08-28T19:44:45.735343+0000 (oldest deadline 2024-08-28T19:45:09.835445+0000) 2024-08-28T19:45:25.872 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:25 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:25.633+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.437204+0000 front 2024-08-28T19:44:54.437291+0000 (oldest deadline 2024-08-28T19:45:17.337174+0000) 2024-08-28T19:45:25.873 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:25 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5[35352]: 2024-08-28T19:45:25.633+0000 7f093b0db640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.638193+0000 front 2024-08-28T19:45:02.638220+0000 (oldest deadline 2024-08-28T19:45:24.938426+0000) 2024-08-28T19:45:25.873 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:25 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:25.527+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:25.873 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:25 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:25.527+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:25.873 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:25 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:25.527+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:26.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:25 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:25.686+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:26.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:25 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:25.686+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:26.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:25 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:25.797+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:26.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:25 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:25.797+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:26.499 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:26 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:26.193+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:26.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:26 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:26.032+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:26.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:26 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:26.032+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:58.271344+0000 front 2024-08-28T19:44:58.271402+0000 (oldest deadline 2024-08-28T19:45:23.571352+0000) 2024-08-28T19:45:26.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:26 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:26.196+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:26.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:26 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:26.196+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:26.870 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:26 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:26.498+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:26.871 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:26 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:26.498+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:26.871 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:26 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:26.498+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:27.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:26 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:26.713+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:27.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:26 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:26.713+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:27.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:26 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:26.713+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:27.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:26 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:26.752+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:27.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:26 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:26.752+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:27.338 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:27 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:27.180+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:27.338 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:27 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:27.180+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:27.465 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:27 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:27.210+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:27.465 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:27 smithi115 podman[61620]: 2024-08-28 19:45:27.124453664 +0000 UTC m=+5.653050413 container died 89f59934a2126de2a99458d7c28908da178cd65d950a841d571eab32b005f18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-28T19:45:27.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:27 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:27.032+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:27.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:27 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:27.032+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:58.271344+0000 front 2024-08-28T19:44:58.271402+0000 (oldest deadline 2024-08-28T19:45:23.571352+0000) 2024-08-28T19:45:27.870 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:27 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:27.504+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:27.871 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:27 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:27.504+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:27.871 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:27 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:27.504+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:27.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:27 smithi115 podman[61620]: 2024-08-28 19:45:27.464905633 +0000 UTC m=+5.993502375 container cleanup 89f59934a2126de2a99458d7c28908da178cd65d950a841d571eab32b005f18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-08-28T19:45:27.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:27 smithi115 bash[61620]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5 2024-08-28T19:45:27.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:27 smithi115 podman[61634]: 2024-08-28 19:45:27.683144265 +0000 UTC m=+0.558191594 container remove 89f59934a2126de2a99458d7c28908da178cd65d950a841d571eab32b005f18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-28T19:45:27.966 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:27 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:27.700+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:27.967 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:27 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:27.700+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:27.967 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:27 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:27.700+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:28.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:27 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:27.736+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:28.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:27 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:27.736+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:28.250 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:28 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:28.081+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:28.250 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:28 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:28.081+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:58.271344+0000 front 2024-08-28T19:44:58.271402+0000 (oldest deadline 2024-08-28T19:45:23.571352+0000) 2024-08-28T19:45:28.362 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:28 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:28.224+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:28.362 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:28 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:28.224+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:28.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:28 smithi115 podman[61724]: 2024-08-28 19:45:27.976102049 +0000 UTC m=+0.015598635 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:45:28.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:28 smithi115 podman[61724]: 2024-08-28 19:45:28.125710018 +0000 UTC m=+0.165206630 container create 9a0be6df1d2808c0904a77a3af190a18aa7bfe927439e00582fe9bcbe5d8b01c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , ceph=True) 2024-08-28T19:45:28.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:28 smithi115 podman[61724]: 2024-08-28 19:45:28.361588878 +0000 UTC m=+0.401085448 container init 9a0be6df1d2808c0904a77a3af190a18aa7bfe927439e00582fe9bcbe5d8b01c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-08-28T19:45:28.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:28 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:28.181+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:28.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:28 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:28.181+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:28.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:28 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:28.471+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:28.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:28 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:28.471+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:28.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:28 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:28.471+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:28.621 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:28 smithi115 podman[61724]: 2024-08-28 19:45:28.365104819 +0000 UTC m=+0.404601389 container start 9a0be6df1d2808c0904a77a3af190a18aa7bfe927439e00582fe9bcbe5d8b01c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-08-28T19:45:28.622 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:28 smithi115 podman[61724]: 2024-08-28 19:45:28.453500117 +0000 UTC m=+0.492996691 container attach 9a0be6df1d2808c0904a77a3af190a18aa7bfe927439e00582fe9bcbe5d8b01c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, ceph=True) 2024-08-28T19:45:28.622 INFO:journalctl@ceph.osd.5.smithi115.stdout:Aug 28 19:45:28 smithi115 podman[61724]: 2024-08-28 19:45:28.552197551 +0000 UTC m=+0.591694130 container died 9a0be6df1d2808c0904a77a3af190a18aa7bfe927439e00582fe9bcbe5d8b01c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-5-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-28T19:45:29.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:28 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:28.725+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:29.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:28 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:28.725+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:29.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:28 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:28.725+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:29.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:28 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:28.784+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:29.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:28 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:28.784+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:29.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:28 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:28.784+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:29.228 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.5.service' 2024-08-28T19:45:29.269 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:45:29.270 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-08-28T19:45:29.270 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-08-28T19:45:29.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.6 2024-08-28T19:45:29.322 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:29 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:29.052+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:29.322 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:29 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:29.052+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:58.271344+0000 front 2024-08-28T19:44:58.271402+0000 (oldest deadline 2024-08-28T19:45:23.571352+0000) 2024-08-28T19:45:29.468 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:29 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:29.269+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:29.469 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:29 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:29.269+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:29.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:29 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:29.162+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:29.560 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:29 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:29.162+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:29.583 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:29 smithi016 systemd[1]: Stopping Ceph osd.6 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:45:29.871 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:29 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:29.467+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:29.871 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:29 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:29.467+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:29.871 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:29 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:29.467+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:30.000 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:29 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:29.581+0000 7fb7dbe5f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:45:30.001 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:29 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:29.581+0000 7fb7dbe5f640 -1 osd.6 122 *** Got signal Terminated *** 2024-08-28T19:45:30.001 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:29 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:29.581+0000 7fb7dbe5f640 -1 osd.6 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:45:30.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:29 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:29.764+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:30.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:29 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:29.764+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:30.002 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:29 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:29.764+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:30.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:29 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:29.795+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:30.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:29 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:29.795+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:30.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:29 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:29.795+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:30.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:30 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:30.051+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:30.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:30 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:30.051+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:58.271344+0000 front 2024-08-28T19:44:58.271402+0000 (oldest deadline 2024-08-28T19:45:23.571352+0000) 2024-08-28T19:45:30.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:30 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:30.161+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:30.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:30 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:30.161+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:30.620 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:30 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:30.227+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:30.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:30 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:30.227+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:30.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:30 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:30.450+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:30.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:30 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:30.450+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:30.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:30 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:30.450+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:31.002 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:30 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:30.726+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:31.003 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:30 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:30.726+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:31.003 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:30 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:30.726+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:31.156 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:30 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:30.842+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:31.157 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:30 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:30.842+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:31.157 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:30 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:30.842+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:31.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:31 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:31.057+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:31.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:31 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:31.057+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:58.271344+0000 front 2024-08-28T19:44:58.271402+0000 (oldest deadline 2024-08-28T19:45:23.571352+0000) 2024-08-28T19:45:31.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:31 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:31.155+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:31.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:31 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:31.155+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:31.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:31 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:31.191+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:31.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:31 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:31.191+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:31.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:31 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:31.403+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:31.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:31 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:31.403+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:31.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:31 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:31.403+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:32.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:31 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:31.692+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:32.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:31 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:31.692+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:32.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:31 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:31.692+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:32.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:31 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:31.805+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:32.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:31 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:31.805+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:32.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:31 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:31.805+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:32.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:32 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:32.078+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:32.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:32 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:32.078+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:58.271344+0000 front 2024-08-28T19:44:58.271402+0000 (oldest deadline 2024-08-28T19:45:23.571352+0000) 2024-08-28T19:45:32.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:32 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:32.183+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:32.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:32 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:32.183+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:32.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:32 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:32.215+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:32.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:32 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:32.215+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:32.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:32 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:32.412+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:32.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:32 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:32.412+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:32.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:32 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:32.412+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:33.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:32 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:32.693+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:33.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:32 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:32.693+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:33.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:32 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:32.693+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:33.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:32 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:32.693+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:33.233 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:32 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:32.834+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:33.233 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:32 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:32.834+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:33.233 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:32 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:32.834+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:33.233 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:32 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:32.834+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:33.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:33 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:33.107+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:33.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:33 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:33.107+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:58.271344+0000 front 2024-08-28T19:44:58.271402+0000 (oldest deadline 2024-08-28T19:45:23.571352+0000) 2024-08-28T19:45:33.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:33 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:33.231+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:33.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:33 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:33.231+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:33.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:33 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:33.231+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.721944+0000 front 2024-08-28T19:45:09.721963+0000 (oldest deadline 2024-08-28T19:45:33.222000+0000) 2024-08-28T19:45:33.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:33 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:33.255+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:33.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:33 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:33.255+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:33.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:33 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:33.255+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:33.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:33 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:33.425+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:33.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:33 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:33.425+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:33.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:33 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:33.425+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:34.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:33 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:33.664+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:34.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:33 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:33.664+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:34.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:33 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:33.664+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:34.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:33 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:33.664+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:34.226 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:33 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:33.880+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:34.227 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:33 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:33.880+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:34.227 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:33 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:33.880+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:34.227 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:33 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:33.880+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:34.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:34 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:34.098+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:51.270022+0000 front 2024-08-28T19:44:51.269925+0000 (oldest deadline 2024-08-28T19:45:16.569747+0000) 2024-08-28T19:45:34.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:34 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:34.098+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:58.271344+0000 front 2024-08-28T19:44:58.271402+0000 (oldest deadline 2024-08-28T19:45:23.571352+0000) 2024-08-28T19:45:34.500 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:34 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6[49905]: 2024-08-28T19:45:34.098+0000 7fb7d844a640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.573054+0000 front 2024-08-28T19:45:10.573012+0000 (oldest deadline 2024-08-28T19:45:34.073162+0000) 2024-08-28T19:45:34.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:34 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:34.225+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:34.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:34 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:34.225+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:34.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:34 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:34.225+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.721944+0000 front 2024-08-28T19:45:09.721963+0000 (oldest deadline 2024-08-28T19:45:33.222000+0000) 2024-08-28T19:45:34.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:34 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:34.428+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:34.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:34 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:34.428+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:34.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:34 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:34.428+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:34.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:34 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:34.294+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:34.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:34 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:34.294+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:34.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:34 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:34.294+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:34.989 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:34 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:34.627+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:34.989 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:34 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:34.627+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:34.989 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:34 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:34.627+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:34.990 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:34 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:34.627+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:35.178 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:34 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:34.857+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:35.178 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:34 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:34.857+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:35.178 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:34 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:34.857+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:35.179 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:34 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:34.857+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:35.250 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:34 smithi016 podman[131347]: 2024-08-28 19:45:34.988873583 +0000 UTC m=+5.513779614 container died d467e691f6bda37acefb468ffb0cfc4375468c2f1b85dd2f854f9dbcd029a9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-08-28T19:45:35.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:35 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:35.177+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:35.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:35 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:35.177+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:35.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:35 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:35.177+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.721944+0000 front 2024-08-28T19:45:09.721963+0000 (oldest deadline 2024-08-28T19:45:33.222000+0000) 2024-08-28T19:45:35.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:35 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:35.392+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:35.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:35 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:35.392+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:35.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:35 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:35.392+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:35.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:35 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:35.303+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:35.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:35 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:35.303+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:35.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:35 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:35.303+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:35.701 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:35 smithi016 podman[131347]: 2024-08-28 19:45:35.523985688 +0000 UTC m=+6.048891727 container cleanup d467e691f6bda37acefb468ffb0cfc4375468c2f1b85dd2f854f9dbcd029a9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6, org.label-schema.build-date=20240731, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-08-28T19:45:35.701 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:35 smithi016 bash[131347]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6 2024-08-28T19:45:35.702 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:35 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:35.588+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:35.702 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:35 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:35.588+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:35.702 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:35 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:35.588+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:35.702 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:35 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:35.588+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:36.000 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:35 smithi016 podman[131363]: 2024-08-28 19:45:35.700764008 +0000 UTC m=+0.709813038 container remove d467e691f6bda37acefb468ffb0cfc4375468c2f1b85dd2f854f9dbcd029a9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, ceph=True) 2024-08-28T19:45:36.227 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:35 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:35.809+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:36.227 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:35 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:35.809+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:36.227 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:35 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:35.809+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:36.227 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:35 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:35.809+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:36.371 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:36 smithi016 podman[131452]: 2024-08-28 19:45:36.118356245 +0000 UTC m=+0.087825912 container create ff64c01760d11523287998c88d3c316b22188cc5c462a8c8a451c6db5cfda1b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.1) 2024-08-28T19:45:36.371 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:36 smithi016 podman[131452]: 2024-08-28 19:45:36.046259615 +0000 UTC m=+0.015729298 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:45:36.371 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:36 smithi016 podman[131452]: 2024-08-28 19:45:36.370864067 +0000 UTC m=+0.340333734 container init ff64c01760d11523287998c88d3c316b22188cc5c462a8c8a451c6db5cfda1b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.1, org.label-schema.vendor=CentOS) 2024-08-28T19:45:36.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:36 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:36.225+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:36.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:36 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:36.225+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:36.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:36 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:36.225+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.721944+0000 front 2024-08-28T19:45:09.721963+0000 (oldest deadline 2024-08-28T19:45:33.222000+0000) 2024-08-28T19:45:36.620 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:36 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:36.264+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:36.620 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:36 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:36.264+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:36.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:36 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:36.264+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:36.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:36 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:36.420+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:36.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:36 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:36.420+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:36.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:36 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:36.420+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:36.750 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:36 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:36.594+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:36.750 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:36 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:36.594+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:36.750 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:36 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:36.594+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:36.750 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:36 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:36.594+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:36.751 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:36 smithi016 podman[131452]: 2024-08-28 19:45:36.374302673 +0000 UTC m=+0.343772342 container start ff64c01760d11523287998c88d3c316b22188cc5c462a8c8a451c6db5cfda1b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.1) 2024-08-28T19:45:36.751 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:36 smithi016 podman[131452]: 2024-08-28 19:45:36.470986333 +0000 UTC m=+0.440456014 container attach ff64c01760d11523287998c88d3c316b22188cc5c462a8c8a451c6db5cfda1b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-08-28T19:45:36.751 INFO:journalctl@ceph.osd.6.smithi016.stdout:Aug 28 19:45:36 smithi016 podman[131452]: 2024-08-28 19:45:36.561823958 +0000 UTC m=+0.531293640 container died ff64c01760d11523287998c88d3c316b22188cc5c462a8c8a451c6db5cfda1b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-08-28T19:45:37.215 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:36 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:36.839+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:37.215 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:36 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:36.839+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:37.215 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:36 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:36.839+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:37.215 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:36 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:36.839+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:37.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.6.service' 2024-08-28T19:45:37.329 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:45:37.329 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-08-28T19:45:37.329 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-08-28T19:45:37.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.7 2024-08-28T19:45:37.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:37 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:37.213+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:37.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:37 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:37.213+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:37.560 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:37 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:37.213+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.721944+0000 front 2024-08-28T19:45:09.721963+0000 (oldest deadline 2024-08-28T19:45:33.222000+0000) 2024-08-28T19:45:37.560 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:37 smithi028 systemd[1]: Stopping Ceph osd.7 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:45:37.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:37 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:37.378+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:37.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:37 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:37.378+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:37.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:37 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:37.378+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:37.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:37 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:37.230+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:37.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:37 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:37.230+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:37.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:37 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:37.230+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:38.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:37 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:37.624+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:38.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:37 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:37.624+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:38.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:37 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:37.624+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:38.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:37 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:37.624+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:38.059 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:37 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:37.654+0000 7faa1f315640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:45:38.059 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:37 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:37.654+0000 7faa1f315640 -1 osd.7 122 *** Got signal Terminated *** 2024-08-28T19:45:38.060 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:37 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:37.654+0000 7faa1f315640 -1 osd.7 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:45:38.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:37 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:37.882+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:38.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:37 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:37.882+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:38.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:37 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:37.882+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:38.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:37 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:37.882+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:38.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:38 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:38.173+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:38.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:38 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:38.173+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:38.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:38 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:38.173+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.721944+0000 front 2024-08-28T19:45:09.721963+0000 (oldest deadline 2024-08-28T19:45:33.222000+0000) 2024-08-28T19:45:38.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:38 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:38.244+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:38.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:38 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:38.244+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:38.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:38 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:38.244+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:38.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:38 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:38.335+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:38.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:38 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:38.335+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:38.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:38 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:38.335+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:38.750 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:38 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:38.669+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:38.750 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:38 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:38.669+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:38.750 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:38 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:38.669+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:38.751 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:38 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:38.669+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:39.194 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:38 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:38.845+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:39.194 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:38 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:38.845+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:39.194 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:38 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:38.845+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:39.195 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:38 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:38.845+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:39.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:39 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:39.193+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:39.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:39 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:39.193+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:39.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:39 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:39.193+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.721944+0000 front 2024-08-28T19:45:09.721963+0000 (oldest deadline 2024-08-28T19:45:33.222000+0000) 2024-08-28T19:45:39.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:39 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:39.273+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:39.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:39 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:39.273+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:39.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:39 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:39.273+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:39.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:39 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:39.297+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:39.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:39 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:39.297+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:39.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:39 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:39.297+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:40.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:39 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:39.710+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:40.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:39 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:39.710+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:40.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:39 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:39.710+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:40.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:39 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:39.710+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:40.211 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:39 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:39.867+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:40.211 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:39 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:39.867+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:40.211 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:39 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:39.867+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:40.211 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:39 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:39.867+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:40.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:40 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:40.209+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:40.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:40 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:40.209+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:40.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:40 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:40.209+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.721944+0000 front 2024-08-28T19:45:09.721963+0000 (oldest deadline 2024-08-28T19:45:33.222000+0000) 2024-08-28T19:45:40.560 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:40 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:40.209+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.523014+0000 front 2024-08-28T19:45:18.523039+0000 (oldest deadline 2024-08-28T19:45:39.523831+0000) 2024-08-28T19:45:40.620 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:40 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:40.290+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:40.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:40 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:40.290+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:40.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:40 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:40.290+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:40.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:40 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:40.298+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:40.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:40 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:40.298+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:40.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:40 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:40.298+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:41.234 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:40 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:40.843+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:41.234 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:40 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:40.843+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:41.234 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:40 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:40.843+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:41.234 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:40 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:40.843+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:41.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:40 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:40.754+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:41.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:40 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:40.754+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:41.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:40 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:40.754+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:41.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:40 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:40.754+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:41.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:41 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:41.232+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:41.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:41 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:41.232+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:41.560 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:41 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:41.232+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.721944+0000 front 2024-08-28T19:45:09.721963+0000 (oldest deadline 2024-08-28T19:45:33.222000+0000) 2024-08-28T19:45:41.560 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:41 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:41.232+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.523014+0000 front 2024-08-28T19:45:18.523039+0000 (oldest deadline 2024-08-28T19:45:39.523831+0000) 2024-08-28T19:45:41.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:41 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:41.256+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:41.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:41 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:41.256+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:41.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:41 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:41.256+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:41.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:41 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:41.243+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:41.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:41 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:41.243+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:41.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:41 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:41.243+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:41.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:41 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:41.243+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.435316+0000 front 2024-08-28T19:45:17.435242+0000 (oldest deadline 2024-08-28T19:45:40.935232+0000) 2024-08-28T19:45:42.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:41 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:41.802+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:42.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:41 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:41.802+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:42.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:41 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:41.802+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:42.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:41 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:41.802+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:42.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:41 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:41.756+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:42.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:41 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:41.756+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:42.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:41 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:41.756+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:42.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:41 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:41.756+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:42.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:41 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:41.756+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:42.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:42 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:42.249+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:53.816327+0000 front 2024-08-28T19:44:53.816657+0000 (oldest deadline 2024-08-28T19:45:15.516505+0000) 2024-08-28T19:45:42.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:42 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:42.249+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.218885+0000 front 2024-08-28T19:45:01.218921+0000 (oldest deadline 2024-08-28T19:45:25.819550+0000) 2024-08-28T19:45:42.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:42 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:42.249+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.721944+0000 front 2024-08-28T19:45:09.721963+0000 (oldest deadline 2024-08-28T19:45:33.222000+0000) 2024-08-28T19:45:42.559 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:42 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7[41154]: 2024-08-28T19:45:42.249+0000 7faa178f8640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.523014+0000 front 2024-08-28T19:45:18.523039+0000 (oldest deadline 2024-08-28T19:45:39.523831+0000) 2024-08-28T19:45:42.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:42 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:42.297+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:42.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:42 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:42.297+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:42.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:42 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:42.297+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:42.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:42 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:42.291+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:42.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:42 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:42.291+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:42.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:42 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:42.291+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:42.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:42 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:42.291+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.435316+0000 front 2024-08-28T19:45:17.435242+0000 (oldest deadline 2024-08-28T19:45:40.935232+0000) 2024-08-28T19:45:43.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:42 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:42.754+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:43.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:42 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:42.754+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:43.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:42 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:42.754+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:43.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:42 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:42.754+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:43.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:42 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:42.779+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:43.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:42 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:42.779+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:43.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:42 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:42.779+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:43.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:42 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:42.779+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:43.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:42 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:42.779+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:43.491 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:43 smithi028 podman[61753]: 2024-08-28 19:45:43.211944017 +0000 UTC m=+5.681046859 container died 691b97276c766a5e8d5943715eb6c5c0f3c4ba2f170fc90083436c8df72b770b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-28T19:45:43.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:43 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:43.289+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:43.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:43 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:43.289+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:43.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:43 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:43.289+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:43.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:43 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:43.277+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:43.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:43 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:43.277+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:43.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:43 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:43.277+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:43.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:43 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:43.277+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.435316+0000 front 2024-08-28T19:45:17.435242+0000 (oldest deadline 2024-08-28T19:45:40.935232+0000) 2024-08-28T19:45:43.776 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:43 smithi028 podman[61753]: 2024-08-28 19:45:43.590021264 +0000 UTC m=+6.059124165 container cleanup 691b97276c766a5e8d5943715eb6c5c0f3c4ba2f170fc90083436c8df72b770b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.1, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-08-28T19:45:43.776 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:43 smithi028 bash[61753]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7 2024-08-28T19:45:44.059 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:43 smithi028 podman[61767]: 2024-08-28 19:45:43.866322369 +0000 UTC m=+0.653171165 container remove 691b97276c766a5e8d5943715eb6c5c0f3c4ba2f170fc90083436c8df72b770b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-08-28T19:45:44.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:43 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:43.774+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:44.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:43 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:43.774+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:44.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:43 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:43.774+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:44.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:43 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:43.774+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:44.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:43 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:43.774+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:44.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:43 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:43.794+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:44.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:43 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:43.794+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:44.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:43 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:43.794+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:44.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:43 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:43.794+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:44.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:43 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:43.794+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:44.428 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:44 smithi028 podman[61855]: 2024-08-28 19:45:44.080528313 +0000 UTC m=+0.016299472 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:45:44.428 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:44 smithi028 podman[61855]: 2024-08-28 19:45:44.20062735 +0000 UTC m=+0.136398554 container create f01e8e2cfc5bb334102e7eb7ac4ecfd8b7f37bb0575045670e3c5b4534b36f34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611) 2024-08-28T19:45:44.429 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:44 smithi028 podman[61855]: 2024-08-28 19:45:44.428028084 +0000 UTC m=+0.363799246 container init f01e8e2cfc5bb334102e7eb7ac4ecfd8b7f37bb0575045670e3c5b4534b36f34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-08-28T19:45:44.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:44 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:44.265+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:44.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:44 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:44.265+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:44.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:44 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:44.265+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:44.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:44 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:44.265+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.435316+0000 front 2024-08-28T19:45:17.435242+0000 (oldest deadline 2024-08-28T19:45:40.935232+0000) 2024-08-28T19:45:44.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:44 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:44.304+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:44.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:44 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:44.304+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:44.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:44 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:44.304+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:44.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:44 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:44.304+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:44.756 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:44 smithi028 podman[61855]: 2024-08-28 19:45:44.431495367 +0000 UTC m=+0.367266535 container start f01e8e2cfc5bb334102e7eb7ac4ecfd8b7f37bb0575045670e3c5b4534b36f34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1) 2024-08-28T19:45:44.757 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:44 smithi028 podman[61855]: 2024-08-28 19:45:44.502950927 +0000 UTC m=+0.438722093 container attach f01e8e2cfc5bb334102e7eb7ac4ecfd8b7f37bb0575045670e3c5b4534b36f34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-08-28T19:45:44.757 INFO:journalctl@ceph.osd.7.smithi028.stdout:Aug 28 19:45:44 smithi028 podman[61855]: 2024-08-28 19:45:44.624922473 +0000 UTC m=+0.560693635 container died f01e8e2cfc5bb334102e7eb7ac4ecfd8b7f37bb0575045670e3c5b4534b36f34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.1, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-08-28T19:45:45.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:44 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:44.755+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:45.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:44 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:44.755+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:45.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:44 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:44.755+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:45.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:44 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:44.755+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:45.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:44 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:44.755+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:45.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:44.752+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:45.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:44.752+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:45.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:44.752+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:45.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:44.752+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:45.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:44 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:44.752+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:45.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.7.service' 2024-08-28T19:45:45.434 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:45:45.434 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-08-28T19:45:45.434 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-08-28T19:45:45.434 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.8 2024-08-28T19:45:45.620 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:45 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:45.279+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:45.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:45 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:45.279+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:45.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:45 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:45.279+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:45.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:45 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:45.279+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.435316+0000 front 2024-08-28T19:45:17.435242+0000 (oldest deadline 2024-08-28T19:45:40.935232+0000) 2024-08-28T19:45:45.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:45 smithi115 systemd[1]: Stopping Ceph osd.8 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:45:45.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:45 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:45.333+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:45.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:45 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:45.333+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:45.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:45 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:45.333+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:45.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:45 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:45.333+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:46.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:45.725+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:46.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:45.725+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:46.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:45.725+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:46.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:45.725+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:46.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:45 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:45.725+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:46.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:45 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:45.724+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:46.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:45 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:45.724+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:46.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:45 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:45.724+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:46.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:45 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:45.724+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:46.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:45 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:45.724+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:46.121 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:45 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:45.753+0000 7f5c32101640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:45:46.121 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:45 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:45.753+0000 7f5c32101640 -1 osd.8 122 *** Got signal Terminated *** 2024-08-28T19:45:46.121 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:45 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:45.753+0000 7f5c32101640 -1 osd.8 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:45:46.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:46 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:46.373+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:46.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:46 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:46.373+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:46.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:46 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:46.373+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:46.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:46 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:46.373+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:46.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:46 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:46.311+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:46.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:46 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:46.311+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:46.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:46 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:46.311+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:46.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:46 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:46.311+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.435316+0000 front 2024-08-28T19:45:17.435242+0000 (oldest deadline 2024-08-28T19:45:40.935232+0000) 2024-08-28T19:45:47.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:46 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:46.712+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:47.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:46 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:46.712+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:47.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:46 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:46.712+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:47.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:46 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:46.712+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:47.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:46 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:46.712+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:47.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:46 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:46.759+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:47.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:46 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:46.759+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:47.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:46 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:46.759+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:47.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:46 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:46.759+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:47.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:46 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:46.759+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:47.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:47 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:47.329+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:47.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:47 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:47.329+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:47.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:47 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:47.329+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:47.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:47 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:47.329+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.435316+0000 front 2024-08-28T19:45:17.435242+0000 (oldest deadline 2024-08-28T19:45:40.935232+0000) 2024-08-28T19:45:47.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:47 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:47.406+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:47.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:47 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:47.406+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:47.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:47 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:47.406+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:47.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:47 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:47.406+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:48.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:47 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:47.739+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:48.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:47 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:47.739+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:48.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:47 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:47.739+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:48.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:47 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:47.739+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:48.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:47 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:47.739+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:48.151 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:47 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:47.780+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:48.152 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:47 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:47.780+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:48.152 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:47 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:47.780+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:48.152 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:47 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:47.780+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:48.152 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:47 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:47.780+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:48.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:48 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:48.317+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:48.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:48 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:48.317+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:48.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:48 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:48.317+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:48.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:48 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:48.317+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.435316+0000 front 2024-08-28T19:45:17.435242+0000 (oldest deadline 2024-08-28T19:45:40.935232+0000) 2024-08-28T19:45:48.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:48 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:48.394+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:48.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:48 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:48.394+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:48.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:48 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:48.394+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:48.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:48 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:48.394+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:49.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:48 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:48.735+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:49.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:48 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:48.735+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:49.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:48 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:48.735+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:49.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:48 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:48.735+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:49.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:48 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:48.735+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:49.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:48 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:48.783+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:49.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:48 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:48.783+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:49.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:48 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:48.783+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:49.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:48 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:48.783+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:49.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:48 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:48.783+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:49.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:48 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:48.783+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.327069+0000 front 2024-08-28T19:45:23.326959+0000 (oldest deadline 2024-08-28T19:45:48.626963+0000) 2024-08-28T19:45:49.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:49 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:49.368+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:49.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:49 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:49.368+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:49.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:49 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:49.368+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:49.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:49 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:49.368+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:49.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:49 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:49.359+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:49.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:49 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:49.359+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:49.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:49 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:49.359+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:49.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:49 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:49.359+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.435316+0000 front 2024-08-28T19:45:17.435242+0000 (oldest deadline 2024-08-28T19:45:40.935232+0000) 2024-08-28T19:45:49.622 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:49 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:49.359+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:25.636287+0000 front 2024-08-28T19:45:25.636319+0000 (oldest deadline 2024-08-28T19:45:48.536413+0000) 2024-08-28T19:45:50.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:49 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:49.764+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:50.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:49 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:49.764+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:50.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:49 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:49.764+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:50.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:49 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:49.764+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:50.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:49 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:49.764+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:50.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:49 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:49.764+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:45:50.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:49 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:49.817+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:50.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:49 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:49.817+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:50.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:49 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:49.817+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:50.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:49 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:49.817+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:50.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:49 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:49.817+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:50.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:49 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:49.817+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.327069+0000 front 2024-08-28T19:45:23.326959+0000 (oldest deadline 2024-08-28T19:45:48.626963+0000) 2024-08-28T19:45:50.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:50 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:50.385+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:52.931056+0000 front 2024-08-28T19:44:52.930883+0000 (oldest deadline 2024-08-28T19:45:18.230805+0000) 2024-08-28T19:45:50.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:50 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:50.385+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.331818+0000 front 2024-08-28T19:45:02.331974+0000 (oldest deadline 2024-08-28T19:45:27.632025+0000) 2024-08-28T19:45:50.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:50 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:50.385+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:09.333264+0000 front 2024-08-28T19:45:09.333145+0000 (oldest deadline 2024-08-28T19:45:32.833311+0000) 2024-08-28T19:45:50.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:50 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:50.385+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.435316+0000 front 2024-08-28T19:45:17.435242+0000 (oldest deadline 2024-08-28T19:45:40.935232+0000) 2024-08-28T19:45:50.621 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:50 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8[41849]: 2024-08-28T19:45:50.385+0000 7f5c2a6e4640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:25.636287+0000 front 2024-08-28T19:45:25.636319+0000 (oldest deadline 2024-08-28T19:45:48.536413+0000) 2024-08-28T19:45:50.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:50 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:50.331+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:50.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:50 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:50.331+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:50.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:50 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:50.331+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:50.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:50 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:50.331+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:50.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:50 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:50.331+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:45:51.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:50 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:50.762+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:51.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:50 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:50.762+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:51.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:50 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:50.762+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:51.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:50 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:50.762+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:51.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:50 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:50.762+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:51.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:50 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:50.762+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:45:51.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:50 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:50.845+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:51.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:50 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:50.845+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:51.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:50 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:50.845+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:51.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:50 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:50.845+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:51.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:50 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:50.845+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:51.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:50 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:50.845+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.327069+0000 front 2024-08-28T19:45:23.326959+0000 (oldest deadline 2024-08-28T19:45:48.626963+0000) 2024-08-28T19:45:51.503 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:51 smithi115 podman[61838]: 2024-08-28 19:45:51.181359392 +0000 UTC m=+5.546907448 container died 968bb9423fcaa3108ea088783ff4dbb749872db01c2113f81c9081304fd0fa19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611) 2024-08-28T19:45:51.504 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:51 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:51.359+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:51.504 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:51 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:51.359+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:51.504 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:51 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:51.359+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:51.504 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:51 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:51.359+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:51.504 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:51 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:51.359+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:45:51.870 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:51 smithi115 podman[61838]: 2024-08-28 19:45:51.503155115 +0000 UTC m=+5.868703164 container cleanup 968bb9423fcaa3108ea088783ff4dbb749872db01c2113f81c9081304fd0fa19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611) 2024-08-28T19:45:51.871 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:51 smithi115 bash[61838]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8 2024-08-28T19:45:51.871 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:51 smithi115 podman[61853]: 2024-08-28 19:45:51.721051819 +0000 UTC m=+0.536420422 container remove 968bb9423fcaa3108ea088783ff4dbb749872db01c2113f81c9081304fd0fa19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.1, CEPH_POINT_RELEASE=) 2024-08-28T19:45:52.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:51 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:51.725+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:52.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:51 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:51.725+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:52.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:51 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:51.725+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:52.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:51 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:51.725+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:52.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:51 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:51.725+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:52.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:51 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:51.725+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:45:52.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:51 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:51.822+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:52.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:51 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:51.822+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:52.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:51 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:51.822+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:52.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:51 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:51.822+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:52.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:51 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:51.822+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:52.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:51 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:51.822+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.327069+0000 front 2024-08-28T19:45:23.326959+0000 (oldest deadline 2024-08-28T19:45:48.626963+0000) 2024-08-28T19:45:52.371 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:52 smithi115 podman[61939]: 2024-08-28 19:45:52.014848542 +0000 UTC m=+0.015684823 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:45:52.371 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:52 smithi115 podman[61939]: 2024-08-28 19:45:52.121805028 +0000 UTC m=+0.122641308 container create 20c9da4f3e5b1567a852bdb471478a8597cb395beb45b0a2ea265f218e54bd13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-08-28T19:45:52.372 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:52 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:52.318+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:52.372 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:52 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:52.318+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:52.372 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:52 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:52.318+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:52.372 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:52 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:52.318+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:52.372 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:52 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:52.318+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:45:52.871 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:52 smithi115 podman[61939]: 2024-08-28 19:45:52.410608968 +0000 UTC m=+0.411445248 container init 20c9da4f3e5b1567a852bdb471478a8597cb395beb45b0a2ea265f218e54bd13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8-deactivate, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-08-28T19:45:52.871 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:52 smithi115 podman[61939]: 2024-08-28 19:45:52.413882071 +0000 UTC m=+0.414718352 container start 20c9da4f3e5b1567a852bdb471478a8597cb395beb45b0a2ea265f218e54bd13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-28T19:45:52.871 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:52 smithi115 podman[61939]: 2024-08-28 19:45:52.516553748 +0000 UTC m=+0.517390030 container attach 20c9da4f3e5b1567a852bdb471478a8597cb395beb45b0a2ea265f218e54bd13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-28T19:45:52.872 INFO:journalctl@ceph.osd.8.smithi115.stdout:Aug 28 19:45:52 smithi115 podman[61939]: 2024-08-28 19:45:52.602496425 +0000 UTC m=+0.603332708 container died 20c9da4f3e5b1567a852bdb471478a8597cb395beb45b0a2ea265f218e54bd13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611) 2024-08-28T19:45:53.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:52 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:52.693+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:53.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:52 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:52.693+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:53.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:52 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:52.693+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:53.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:52 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:52.693+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:53.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:52 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:52.693+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:53.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:52 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:52.693+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:45:53.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:52 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:52.802+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:53.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:52 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:52.802+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:53.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:52 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:52.802+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:53.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:52 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:52.802+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:53.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:52 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:52.802+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:53.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:52 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:52.802+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.327069+0000 front 2024-08-28T19:45:23.326959+0000 (oldest deadline 2024-08-28T19:45:48.626963+0000) 2024-08-28T19:45:53.463 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:53 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:53.300+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:53.463 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:53 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:53.300+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:53.463 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:53 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:53.300+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:53.463 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:53 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:53.300+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:53.464 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:53 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:53.300+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:45:53.491 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.8.service' 2024-08-28T19:45:53.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:45:53.532 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-08-28T19:45:53.532 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-08-28T19:45:53.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.9 2024-08-28T19:45:53.845 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:53 smithi016 systemd[1]: Stopping Ceph osd.9 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:45:53.845 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:53 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:53.781+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:53.845 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:53 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:53.781+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:53.845 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:53 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:53.781+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:53.845 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:53 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:53.781+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:53.846 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:53 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:53.781+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:53.846 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:53 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:53.781+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.327069+0000 front 2024-08-28T19:45:23.326959+0000 (oldest deadline 2024-08-28T19:45:48.626963+0000) 2024-08-28T19:45:53.846 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:53 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:53.843+0000 7f1ab00a2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:45:53.846 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:53 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:53.843+0000 7f1ab00a2640 -1 osd.9 122 *** Got signal Terminated *** 2024-08-28T19:45:54.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:53 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:53.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:54.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:53 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:53.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:54.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:53 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:53.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:54.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:53 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:53.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:54.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:53 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:53.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:54.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:53 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:53.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:45:54.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:53 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:53.843+0000 7f1ab00a2640 -1 osd.9 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:45:54.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:54 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:54.293+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:54.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:54 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:54.293+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:54.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:54 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:54.293+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:54.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:54 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:54.293+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:54.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:54 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:54.293+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:45:55.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:54 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:54.627+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:55.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:54 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:54.627+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:55.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:54 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:54.627+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:55.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:54 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:54.627+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:55.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:54 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:54.627+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:55.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:54 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:54.627+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:45:55.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:54 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:54.749+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:55.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:54 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:54.749+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:55.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:54 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:54.749+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:55.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:54 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:54.749+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:55.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:54 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:54.749+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:55.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:54 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:54.749+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.327069+0000 front 2024-08-28T19:45:23.326959+0000 (oldest deadline 2024-08-28T19:45:48.626963+0000) 2024-08-28T19:45:55.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:55 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:55.327+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:55.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:55 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:55.327+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:55.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:55 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:55.327+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:55.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:55 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:55.327+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:55.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:55 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:55.327+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:45:56.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:55 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:55.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:56.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:55 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:55.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:56.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:55 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:55.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:56.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:55 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:55.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:56.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:55 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:55.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:56.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:55 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:55.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:45:56.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:55 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:55.661+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:45:56.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:55 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:55.753+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:56.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:55 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:55.753+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:56.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:55 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:55.753+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:56.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:55 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:55.753+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:56.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:55 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:55.753+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:56.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:55 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:55.753+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.327069+0000 front 2024-08-28T19:45:23.326959+0000 (oldest deadline 2024-08-28T19:45:48.626963+0000) 2024-08-28T19:45:56.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:56 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:56.289+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:56.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:56 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:56.289+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:56.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:56 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:56.289+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:56.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:56 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:56.289+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:56.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:56 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:56.289+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:45:56.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:56 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:56.289+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:45:57.000 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:56 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:56.739+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:57.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:56 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:56.739+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:57.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:56 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:56.739+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:57.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:56 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:56.739+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:57.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:56 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:56.739+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:57.001 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:56 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:56.739+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.327069+0000 front 2024-08-28T19:45:23.326959+0000 (oldest deadline 2024-08-28T19:45:48.626963+0000) 2024-08-28T19:45:57.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:56 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:56.640+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:57.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:56 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:56.640+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:57.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:56 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:56.640+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:57.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:56 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:56.640+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:57.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:56 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:56.640+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:57.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:56 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:56.640+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:45:57.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:56 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:56.640+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:45:57.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:57 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:57.309+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:57.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:57 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:57.309+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:57.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:57 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:57.309+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:57.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:57 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:57.309+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:57.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:57 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:57.309+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:45:57.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:57 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:57.309+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:45:58.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:57 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:57.650+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:58.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:57 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:57.650+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:58.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:57 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:57.650+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:58.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:57 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:57.650+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:58.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:57 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:57.650+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:58.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:57 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:57.650+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:45:58.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:57 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:57.650+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:45:58.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:57 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:57.777+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:58.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:57 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:57.777+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:58.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:57 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:57.777+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:58.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:57 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:57.777+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:58.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:57 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:57.777+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:58.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:57 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:57.777+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.327069+0000 front 2024-08-28T19:45:23.326959+0000 (oldest deadline 2024-08-28T19:45:48.626963+0000) 2024-08-28T19:45:58.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:58 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:58.262+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:58.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:58 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:58.262+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:58.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:58 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:58.262+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:58.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:58 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:58.262+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:58.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:58 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:58.262+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:45:58.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:58 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:58.262+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:45:59.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:58 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:58.628+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:45:59.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:58 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:58.628+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:45:59.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:58 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:58.628+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:45:59.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:58 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:58.628+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:45:59.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:58 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:58.628+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:45:59.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:58 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:58.628+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:45:59.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:58 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:58.628+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:45:59.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:58 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:58.791+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:40.517236+0000 front 2024-08-28T19:44:40.517260+0000 (oldest deadline 2024-08-28T19:45:08.118188+0000) 2024-08-28T19:45:59.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:58 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:58.791+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.519787+0000 front 2024-08-28T19:44:54.519645+0000 (oldest deadline 2024-08-28T19:45:15.019879+0000) 2024-08-28T19:45:59.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:58 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:58.791+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:01.421814+0000 front 2024-08-28T19:45:01.421783+0000 (oldest deadline 2024-08-28T19:45:26.121917+0000) 2024-08-28T19:45:59.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:58 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:58.791+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.624366+0000 front 2024-08-28T19:45:10.624339+0000 (oldest deadline 2024-08-28T19:45:31.724484+0000) 2024-08-28T19:45:59.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:58 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:58.791+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.725781+0000 front 2024-08-28T19:45:18.725854+0000 (oldest deadline 2024-08-28T19:45:41.025807+0000) 2024-08-28T19:45:59.251 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:58 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9[56865]: 2024-08-28T19:45:58.791+0000 7f1aa8685640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.327069+0000 front 2024-08-28T19:45:23.326959+0000 (oldest deadline 2024-08-28T19:45:48.626963+0000) 2024-08-28T19:45:59.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:59 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:59.220+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:45:59.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:59 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:59.220+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:45:59.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:59 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:59.220+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:45:59.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:59 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:59.220+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:45:59.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:59 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:59.220+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:45:59.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:45:59 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:45:59.220+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:45:59.750 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:59 smithi016 podman[131571]: 2024-08-28 19:45:59.360090423 +0000 UTC m=+5.627696072 container died eb84bbff517997de8ca6236e0843441a0f07945d5d4333794f553f05d88a806f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-08-28T19:46:00.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:59 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:59.598+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:46:00.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:59 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:59.598+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:46:00.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:59 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:59.598+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:46:00.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:59 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:59.598+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:46:00.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:59 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:59.598+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:46:00.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:59 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:59.598+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:46:00.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:45:59 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:45:59.598+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:46:00.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:59 smithi016 podman[131571]: 2024-08-28 19:45:59.75235834 +0000 UTC m=+6.019963989 container cleanup eb84bbff517997de8ca6236e0843441a0f07945d5d4333794f553f05d88a806f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-08-28T19:46:00.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:59 smithi016 bash[131571]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9 2024-08-28T19:46:00.250 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:45:59 smithi016 podman[131586]: 2024-08-28 19:45:59.970445226 +0000 UTC m=+0.605286604 container remove eb84bbff517997de8ca6236e0843441a0f07945d5d4333794f553f05d88a806f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True) 2024-08-28T19:46:00.557 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:46:00 smithi016 podman[131674]: 2024-08-28 19:46:00.304697769 +0000 UTC m=+0.110694999 container create e49b6eafaa00209ea772825b9367e771f8816c67c65a992b5665e944f73e31cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-28T19:46:00.557 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:46:00 smithi016 podman[131674]: 2024-08-28 19:46:00.209760553 +0000 UTC m=+0.015757754 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:46:00.557 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:46:00 smithi016 podman[131674]: 2024-08-28 19:46:00.483069296 +0000 UTC m=+0.289066489 container init e49b6eafaa00209ea772825b9367e771f8816c67c65a992b5665e944f73e31cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-08-28T19:46:00.558 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:46:00 smithi016 podman[131674]: 2024-08-28 19:46:00.48660688 +0000 UTC m=+0.292604065 container start e49b6eafaa00209ea772825b9367e771f8816c67c65a992b5665e944f73e31cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-08-28T19:46:00.558 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:46:00 smithi016 podman[131674]: 2024-08-28 19:46:00.557089658 +0000 UTC m=+0.363086842 container attach e49b6eafaa00209ea772825b9367e771f8816c67c65a992b5665e944f73e31cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0) 2024-08-28T19:46:00.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:00 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:00.211+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:00.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:00 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:00.211+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:00.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:00 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:00.211+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:00.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:00 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:00.211+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:00.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:00 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:00.211+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:00.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:00 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:00.211+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:00.969 INFO:journalctl@ceph.osd.9.smithi016.stdout:Aug 28 19:46:00 smithi016 podman[131674]: 2024-08-28 19:46:00.6835077 +0000 UTC m=+0.489504908 container died e49b6eafaa00209ea772825b9367e771f8816c67c65a992b5665e944f73e31cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-08-28T19:46:01.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:00 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:00.594+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:46:01.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:00 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:00.594+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:46:01.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:00 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:00.594+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:46:01.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:00 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:00.594+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:46:01.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:00 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:00.594+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:46:01.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:00 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:00.594+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:46:01.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:00 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:00.594+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:46:01.286 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.9.service' 2024-08-28T19:46:01.330 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:01.330 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-08-28T19:46:01.331 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-08-28T19:46:01.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.10 2024-08-28T19:46:01.616 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:01 smithi028 systemd[1]: Stopping Ceph osd.10 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:01.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:01 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:01.236+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:01.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:01 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:01.236+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:01.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:01 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:01.236+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:01.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:01 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:01.236+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:01.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:01 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:01.236+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:01.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:01 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:01.236+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:02.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:01 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:01.615+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:46:02.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:01 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:01.615+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:46:02.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:01 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:01.615+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:46:02.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:01 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:01.615+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:46:02.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:01 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:01.615+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:46:02.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:01 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:01.615+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:46:02.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:01 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:01.615+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:46:02.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:01 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:01.668+0000 7fdb9ad62640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:02.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:01 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:01.668+0000 7fdb9ad62640 -1 osd.10 122 *** Got signal Terminated *** 2024-08-28T19:46:02.061 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:01 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:01.668+0000 7fdb9ad62640 -1 osd.10 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:46:02.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:02 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:02.245+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:02.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:02 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:02.245+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:02.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:02 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:02.245+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:02.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:02 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:02.245+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:02.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:02 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:02.245+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:02.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:02 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:02.245+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:03.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:02 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:02.583+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:46:03.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:02 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:02.583+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:46:03.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:02 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:02.583+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:46:03.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:02 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:02.583+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:46:03.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:02 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:02.583+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:46:03.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:02 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:02.583+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:46:03.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:02 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:02.583+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:46:03.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:03 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:03.219+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:03.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:03 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:03.219+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:03.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:03 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:03.219+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:03.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:03 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:03.219+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:03.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:03 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:03.219+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:03.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:03 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:03.219+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:04.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:03 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:03.622+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:46:04.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:03 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:03.622+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:46:04.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:03 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:03.622+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:46:04.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:03 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:03.622+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:46:04.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:03 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:03.622+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:46:04.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:03 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:03.622+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:46:04.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:03 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:03.622+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:46:04.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:04 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:04.198+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:04.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:04 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:04.198+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:04.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:04 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:04.198+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:04.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:04 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:04.198+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:04.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:04 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:04.198+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:04.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:04 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:04.198+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:04.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:04 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:04.198+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6820 osd.7 since back 2024-08-28T19:45:38.768668+0000 front 2024-08-28T19:45:38.768833+0000 (oldest deadline 2024-08-28T19:46:04.068538+0000) 2024-08-28T19:46:05.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:04 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:04.621+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:46:05.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:04 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:04.621+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:46:05.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:04 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:04.621+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:46:05.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:04 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:04.621+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:46:05.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:04 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:04.621+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:46:05.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:04 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:04.621+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:46:05.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:04 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:04.621+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:46:05.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:05 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:05.154+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:05.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:05 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:05.154+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:05.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:05 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:05.154+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:05.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:05 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:05.154+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:05.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:05 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:05.154+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:05.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:05 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:05.154+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:05.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:05 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:05.154+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6820 osd.7 since back 2024-08-28T19:45:38.768668+0000 front 2024-08-28T19:45:38.768833+0000 (oldest deadline 2024-08-28T19:46:04.068538+0000) 2024-08-28T19:46:06.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:05 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:05.606+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:46:06.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:05 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:05.606+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:46:06.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:05 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:05.606+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:46:06.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:05 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:05.606+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:46:06.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:05 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:05.606+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:46:06.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:05 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:05.606+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:46:06.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:05 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:05.606+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:46:06.371 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:06 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:06.117+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:06.371 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:06 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:06.117+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:06.371 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:06 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:06.117+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:06.371 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:06 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:06.117+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:06.371 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:06 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:06.117+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:06.372 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:06 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:06.117+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:06.372 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:06 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:06.117+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6820 osd.7 since back 2024-08-28T19:45:38.768668+0000 front 2024-08-28T19:45:38.768833+0000 (oldest deadline 2024-08-28T19:46:04.068538+0000) 2024-08-28T19:46:07.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:06 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:06.653+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:42.535716+0000 front 2024-08-28T19:44:42.535568+0000 (oldest deadline 2024-08-28T19:45:07.235471+0000) 2024-08-28T19:46:07.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:06 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:06.653+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.237183+0000 front 2024-08-28T19:44:54.237271+0000 (oldest deadline 2024-08-28T19:45:19.537373+0000) 2024-08-28T19:46:07.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:06 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:06.653+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:45:02.339015+0000 front 2024-08-28T19:45:02.338978+0000 (oldest deadline 2024-08-28T19:45:28.239091+0000) 2024-08-28T19:46:07.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:06 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:06.653+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-08-28T19:45:10.540328+0000 front 2024-08-28T19:45:10.540301+0000 (oldest deadline 2024-08-28T19:45:32.240350+0000) 2024-08-28T19:46:07.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:06 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:06.653+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:18.542048+0000 front 2024-08-28T19:45:18.542024+0000 (oldest deadline 2024-08-28T19:45:43.242249+0000) 2024-08-28T19:46:07.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:06 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:06.653+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.242917+0000 front 2024-08-28T19:45:23.243025+0000 (oldest deadline 2024-08-28T19:45:49.142786+0000) 2024-08-28T19:46:07.060 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:06 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10[47718]: 2024-08-28T19:46:06.653+0000 7fdb9734d640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:29.644056+0000 front 2024-08-28T19:45:29.643971+0000 (oldest deadline 2024-08-28T19:45:55.543974+0000) 2024-08-28T19:46:07.336 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:07 smithi028 podman[61982]: 2024-08-28 19:46:07.065195905 +0000 UTC m=+5.526217231 container died 07b52226f45d15c83303895080ddba5ffe91e2dfde3bb406dfd5267ad1f73c47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.1, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-28T19:46:07.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:07 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:07.155+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:07.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:07 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:07.155+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:07.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:07 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:07.155+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:07.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:07 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:07.155+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:07.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:07 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:07.155+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:07.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:07 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:07.155+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:07.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:07 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:07.155+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6820 osd.7 since back 2024-08-28T19:45:38.768668+0000 front 2024-08-28T19:45:38.768833+0000 (oldest deadline 2024-08-28T19:46:04.068538+0000) 2024-08-28T19:46:07.762 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:07 smithi028 podman[61982]: 2024-08-28 19:46:07.434957799 +0000 UTC m=+5.895979115 container cleanup 07b52226f45d15c83303895080ddba5ffe91e2dfde3bb406dfd5267ad1f73c47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True) 2024-08-28T19:46:07.762 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:07 smithi028 bash[61982]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10 2024-08-28T19:46:08.059 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:07 smithi028 podman[61997]: 2024-08-28 19:46:07.76176637 +0000 UTC m=+0.694922221 container remove 07b52226f45d15c83303895080ddba5ffe91e2dfde3bb406dfd5267ad1f73c47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-28T19:46:08.326 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:08 smithi028 podman[62086]: 2024-08-28 19:46:07.977241944 +0000 UTC m=+0.016077241 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:46:08.326 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:08 smithi028 podman[62086]: 2024-08-28 19:46:08.212677028 +0000 UTC m=+0.251512364 container create 497ccc7e89cadff799e07fbb4479a6c9bea86eabdfa98b34e0806daf657d6e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10-deactivate, io.buildah.version=1.37.1, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-28T19:46:08.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:08 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:08.166+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:08.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:08 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:08.166+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:08.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:08 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:08.166+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:08.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:08 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:08.166+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:08.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:08 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:08.166+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:08.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:08 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:08.166+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:08.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:08 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:08.166+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6820 osd.7 since back 2024-08-28T19:45:38.768668+0000 front 2024-08-28T19:45:38.768833+0000 (oldest deadline 2024-08-28T19:46:04.068538+0000) 2024-08-28T19:46:08.660 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:08 smithi028 podman[62086]: 2024-08-28 19:46:08.465128972 +0000 UTC m=+0.503964255 container init 497ccc7e89cadff799e07fbb4479a6c9bea86eabdfa98b34e0806daf657d6e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-08-28T19:46:08.661 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:08 smithi028 podman[62086]: 2024-08-28 19:46:08.468542365 +0000 UTC m=+0.507377648 container start 497ccc7e89cadff799e07fbb4479a6c9bea86eabdfa98b34e0806daf657d6e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, CEPH_POINT_RELEASE=) 2024-08-28T19:46:08.661 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:08 smithi028 podman[62086]: 2024-08-28 19:46:08.540007934 +0000 UTC m=+0.578843224 container attach 497ccc7e89cadff799e07fbb4479a6c9bea86eabdfa98b34e0806daf657d6e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10-deactivate, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-28T19:46:08.661 INFO:journalctl@ceph.osd.10.smithi028.stdout:Aug 28 19:46:08 smithi028 podman[62086]: 2024-08-28 19:46:08.659877868 +0000 UTC m=+0.698713161 container died 497ccc7e89cadff799e07fbb4479a6c9bea86eabdfa98b34e0806daf657d6e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-10-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611) 2024-08-28T19:46:09.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.10.service' 2024-08-28T19:46:09.273 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:09.273 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-08-28T19:46:09.274 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-08-28T19:46:09.274 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.11 2024-08-28T19:46:09.565 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:09.203+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:09.565 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:09.203+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:09.565 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:09.203+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:09.565 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:09.203+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:09.565 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:09.203+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:09.566 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:09.203+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:09.566 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:09.203+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6820 osd.7 since back 2024-08-28T19:45:38.768668+0000 front 2024-08-28T19:45:38.768833+0000 (oldest deadline 2024-08-28T19:46:04.068538+0000) 2024-08-28T19:46:09.566 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:09 smithi115 systemd[1]: Stopping Ceph osd.11 for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:09.566 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:09.563+0000 7f0e4a5bf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:09.566 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:09.563+0000 7f0e4a5bf640 -1 osd.11 122 *** Got signal Terminated *** 2024-08-28T19:46:09.566 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:09 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:09.563+0000 7f0e4a5bf640 -1 osd.11 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-28T19:46:10.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:10 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:10.218+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:10.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:10 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:10.218+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:10.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:10 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:10.218+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:10.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:10 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:10.218+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:10.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:10 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:10.218+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:10.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:10 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:10.218+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:10.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:10 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:10.218+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6820 osd.7 since back 2024-08-28T19:45:38.768668+0000 front 2024-08-28T19:45:38.768833+0000 (oldest deadline 2024-08-28T19:46:04.068538+0000) 2024-08-28T19:46:11.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:11 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:11.177+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:11.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:11 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:11.177+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:11.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:11 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:11.177+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:11.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:11 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:11.177+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:11.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:11 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:11.177+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:11.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:11 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:11.177+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:11.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:11 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:11.177+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6820 osd.7 since back 2024-08-28T19:45:38.768668+0000 front 2024-08-28T19:45:38.768833+0000 (oldest deadline 2024-08-28T19:46:04.068538+0000) 2024-08-28T19:46:12.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:12 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:12.148+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:12.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:12 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:12.148+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:12.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:12 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:12.148+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:12.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:12 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:12.148+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:12.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:12 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:12.148+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:12.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:12 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:12.148+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:12.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:12 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:12.148+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6820 osd.7 since back 2024-08-28T19:45:38.768668+0000 front 2024-08-28T19:45:38.768833+0000 (oldest deadline 2024-08-28T19:46:04.068538+0000) 2024-08-28T19:46:13.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:13 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:13.149+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:13.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:13 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:13.149+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:13.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:13 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:13.149+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:13.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:13 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:13.149+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:13.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:13 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:13.149+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:13.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:13 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:13.149+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:13.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:13 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:13.149+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6820 osd.7 since back 2024-08-28T19:45:38.768668+0000 front 2024-08-28T19:45:38.768833+0000 (oldest deadline 2024-08-28T19:46:04.068538+0000) 2024-08-28T19:46:14.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:14 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:14.163+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-08-28T19:44:45.060138+0000 front 2024-08-28T19:44:45.060026+0000 (oldest deadline 2024-08-28T19:45:10.359994+0000) 2024-08-28T19:46:14.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:14 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:14.163+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6804 osd.1 since back 2024-08-28T19:44:54.361768+0000 front 2024-08-28T19:44:54.361701+0000 (oldest deadline 2024-08-28T19:45:19.061659+0000) 2024-08-28T19:46:14.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:14 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:14.163+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6804 osd.2 since back 2024-08-28T19:44:59.062119+0000 front 2024-08-28T19:44:59.062083+0000 (oldest deadline 2024-08-28T19:45:24.362117+0000) 2024-08-28T19:46:14.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:14 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:14.163+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6812 osd.4 since back 2024-08-28T19:45:17.765054+0000 front 2024-08-28T19:45:17.765086+0000 (oldest deadline 2024-08-28T19:45:43.665206+0000) 2024-08-28T19:46:14.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:14 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:14.163+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-08-28T19:45:23.665575+0000 front 2024-08-28T19:45:23.665716+0000 (oldest deadline 2024-08-28T19:45:49.565698+0000) 2024-08-28T19:46:14.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:14 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:14.163+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-08-28T19:45:31.266752+0000 front 2024-08-28T19:45:31.266815+0000 (oldest deadline 2024-08-28T19:45:55.366828+0000) 2024-08-28T19:46:14.622 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:14 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11[48541]: 2024-08-28T19:46:14.163+0000 7f0e42ba2640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.28:6820 osd.7 since back 2024-08-28T19:45:38.768668+0000 front 2024-08-28T19:45:38.768833+0000 (oldest deadline 2024-08-28T19:46:04.068538+0000) 2024-08-28T19:46:15.321 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:15 smithi115 podman[62054]: 2024-08-28 19:46:15.003468275 +0000 UTC m=+5.536327317 container died 9965a6b9afe9d78211fb28fa41cc4f89d01d9694d2aff538a5336aff41d2cf55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-08-28T19:46:15.321 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:15 smithi115 podman[62054]: 2024-08-28 19:46:15.321005418 +0000 UTC m=+5.853864450 container cleanup 9965a6b9afe9d78211fb28fa41cc4f89d01d9694d2aff538a5336aff41d2cf55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-28T19:46:15.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:15 smithi115 bash[62054]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11 2024-08-28T19:46:15.621 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:15 smithi115 podman[62069]: 2024-08-28 19:46:15.522332147 +0000 UTC m=+0.522178966 container remove 9965a6b9afe9d78211fb28fa41cc4f89d01d9694d2aff538a5336aff41d2cf55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11, GIT_CLEAN=True, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-08-28T19:46:16.217 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:15 smithi115 podman[62156]: 2024-08-28 19:46:15.824378405 +0000 UTC m=+0.015926205 image pull 79f4cde5e36911a5dca995eeb98098c2c8aae81dd52a088ef2c867bc3363bad0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61 2024-08-28T19:46:16.217 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:15 smithi115 podman[62156]: 2024-08-28 19:46:15.922819046 +0000 UTC m=+0.114366826 container create 34a10bbf56c002f8a75947b6976ea686ea6502f0098779fa9bc3858da9a28851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, io.buildah.version=1.37.1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-08-28T19:46:16.217 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:16 smithi115 podman[62156]: 2024-08-28 19:46:16.133568342 +0000 UTC m=+0.325116135 container init 34a10bbf56c002f8a75947b6976ea686ea6502f0098779fa9bc3858da9a28851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-28T19:46:16.218 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:16 smithi115 podman[62156]: 2024-08-28 19:46:16.136837367 +0000 UTC m=+0.328385152 container start 34a10bbf56c002f8a75947b6976ea686ea6502f0098779fa9bc3858da9a28851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-28T19:46:16.218 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:16 smithi115 podman[62156]: 2024-08-28 19:46:16.216850346 +0000 UTC m=+0.408398138 container attach 34a10bbf56c002f8a75947b6976ea686ea6502f0098779fa9bc3858da9a28851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.1, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-08-28T19:46:16.620 INFO:journalctl@ceph.osd.11.smithi115.stdout:Aug 28 19:46:16 smithi115 podman[62156]: 2024-08-28 19:46:16.324994528 +0000 UTC m=+0.516542309 container died 34a10bbf56c002f8a75947b6976ea686ea6502f0098779fa9bc3858da9a28851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1, org.label-schema.build-date=20240731) 2024-08-28T19:46:16.909 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@osd.11.service' 2024-08-28T19:46:16.946 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:16.947 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-08-28T19:46:16.947 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-08-28T19:46:16.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.a 2024-08-28T19:46:17.238 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:46:16 smithi016 systemd[1]: Stopping Ceph mds.a for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:17.500 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:46:17 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a[63654]: 2024-08-28T19:46:17.236+0000 7f2e1555a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:17.500 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:46:17 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a[63654]: 2024-08-28T19:46:17.236+0000 7f2e1555a640 -1 mds.a *** got signal Terminated *** 2024-08-28T19:46:19.655 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:46:19 smithi016 podman[131777]: 2024-08-28 19:46:19.400455833 +0000 UTC m=+2.261049745 container died d0926d7b4badab2efed1b27b0d76be84f4c50ca225f8c3567334802dad076301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-08-28T19:46:20.250 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:46:19 smithi016 podman[131777]: 2024-08-28 19:46:19.937622516 +0000 UTC m=+2.798216494 container cleanup d0926d7b4badab2efed1b27b0d76be84f4c50ca225f8c3567334802dad076301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, org.label-schema.build-date=20240731, ceph=True) 2024-08-28T19:46:20.251 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:46:19 smithi016 bash[131777]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a 2024-08-28T19:46:20.251 INFO:journalctl@ceph.mds.a.smithi016.stdout:Aug 28 19:46:20 smithi016 podman[131791]: 2024-08-28 19:46:20.155888778 +0000 UTC m=+0.754018721 container remove d0926d7b4badab2efed1b27b0d76be84f4c50ca225f8c3567334802dad076301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-08-28T19:46:20.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.a.service' 2024-08-28T19:46:20.448 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:20.449 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-08-28T19:46:20.449 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-08-28T19:46:20.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.d 2024-08-28T19:46:20.750 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:46:20 smithi016 systemd[1]: Stopping Ceph mds.d for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:21.250 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:46:20 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-d[62429]: 2024-08-28T19:46:20.782+0000 7f63318cc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:21.250 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:46:20 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-d[62429]: 2024-08-28T19:46:20.782+0000 7f63318cc640 -1 mds.d *** got signal Terminated *** 2024-08-28T19:46:25.167 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:46:24 smithi016 podman[131896]: 2024-08-28 19:46:24.792052526 +0000 UTC m=+4.126410509 container died 50e1601715d47d71f154ca703f86ae4cc9f0ad0585de3f991466c96644d63f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-d, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS) 2024-08-28T19:46:25.500 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:46:25 smithi016 podman[131896]: 2024-08-28 19:46:25.316787234 +0000 UTC m=+4.651145216 container cleanup 50e1601715d47d71f154ca703f86ae4cc9f0ad0585de3f991466c96644d63f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.1) 2024-08-28T19:46:25.500 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:46:25 smithi016 bash[131896]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-d 2024-08-28T19:46:25.758 INFO:journalctl@ceph.mds.d.smithi016.stdout:Aug 28 19:46:25 smithi016 podman[131920]: 2024-08-28 19:46:25.501475975 +0000 UTC m=+0.703512872 container remove 50e1601715d47d71f154ca703f86ae4cc9f0ad0585de3f991466c96644d63f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, org.label-schema.build-date=20240731) 2024-08-28T19:46:25.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.d.service' 2024-08-28T19:46:25.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:25.797 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-08-28T19:46:25.797 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-08-28T19:46:25.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.g 2024-08-28T19:46:26.139 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:46:25 smithi016 systemd[1]: Stopping Ceph mds.g for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:26.500 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:46:26 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g[63297]: 2024-08-28T19:46:26.136+0000 7efe72026640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:26.500 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:46:26 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g[63297]: 2024-08-28T19:46:26.136+0000 7efe72026640 -1 mds.g *** got signal Terminated *** 2024-08-28T19:46:27.750 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:46:27 smithi016 podman[132028]: 2024-08-28 19:46:27.397873869 +0000 UTC m=+1.371857902 container died 100d766a4ef23e040a2e73c3432b0f155b497a98c646865ce6a108aa6c8d6252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-28T19:46:28.106 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:46:27 smithi016 podman[132028]: 2024-08-28 19:46:27.937969148 +0000 UTC m=+1.911953180 container cleanup 100d766a4ef23e040a2e73c3432b0f155b497a98c646865ce6a108aa6c8d6252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-08-28T19:46:28.107 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:46:27 smithi016 bash[132028]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g 2024-08-28T19:46:28.368 INFO:journalctl@ceph.mds.g.smithi016.stdout:Aug 28 19:46:28 smithi016 podman[132041]: 2024-08-28 19:46:28.106018954 +0000 UTC m=+0.701208715 container remove 100d766a4ef23e040a2e73c3432b0f155b497a98c646865ce6a108aa6c8d6252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-g, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731) 2024-08-28T19:46:28.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.g.service' 2024-08-28T19:46:28.404 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:28.405 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-08-28T19:46:28.405 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-08-28T19:46:28.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.j 2024-08-28T19:46:28.726 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:46:28 smithi016 systemd[1]: Stopping Ceph mds.j for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:29.000 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:46:28 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-j[62011]: 2024-08-28T19:46:28.725+0000 7f8feed7c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:29.000 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:46:28 smithi016 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-j[62011]: 2024-08-28T19:46:28.725+0000 7f8feed7c640 -1 mds.j *** got signal Terminated *** 2024-08-28T19:46:30.159 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:46:29 smithi016 podman[132148]: 2024-08-28 19:46:29.791215841 +0000 UTC m=+1.168685363 container died 2fdb9fc7cc45bf8fab94aa1bedb1d1b8716e6d121aeb3d123fc136e4a1b28c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-j, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-08-28T19:46:30.500 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:46:30 smithi016 podman[132148]: 2024-08-28 19:46:30.325285695 +0000 UTC m=+1.702755249 container cleanup 2fdb9fc7cc45bf8fab94aa1bedb1d1b8716e6d121aeb3d123fc136e4a1b28c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, org.label-schema.build-date=20240731) 2024-08-28T19:46:30.501 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:46:30 smithi016 bash[132148]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-j 2024-08-28T19:46:30.757 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.j.service' 2024-08-28T19:46:30.778 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:46:30 smithi016 podman[132161]: 2024-08-28 19:46:30.526585688 +0000 UTC m=+0.730718287 container remove 2fdb9fc7cc45bf8fab94aa1bedb1d1b8716e6d121aeb3d123fc136e4a1b28c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-j, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1) 2024-08-28T19:46:30.778 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:46:30 smithi016 systemd[1]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.j.service: Deactivated successfully. 2024-08-28T19:46:30.778 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:46:30 smithi016 systemd[1]: Stopped Ceph mds.j for eabc438a-6570-11ef-bcd4-c7b262605968. 2024-08-28T19:46:30.778 INFO:journalctl@ceph.mds.j.smithi016.stdout:Aug 28 19:46:30 smithi016 systemd[1]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.j.service: Consumed 1.200s CPU time. 2024-08-28T19:46:30.788 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:30.789 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-08-28T19:46:30.789 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-08-28T19:46:30.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.b 2024-08-28T19:46:31.079 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:46:30 smithi028 systemd[1]: Stopping Ceph mds.b for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:31.353 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:46:31 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b[52931]: 2024-08-28T19:46:31.078+0000 7f21d22de640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:31.353 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:46:31 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b[52931]: 2024-08-28T19:46:31.078+0000 7f21d22de640 -1 mds.b *** got signal Terminated *** 2024-08-28T19:46:35.370 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:46:35 smithi028 podman[62188]: 2024-08-28 19:46:35.096870609 +0000 UTC m=+4.111242179 container died 3427fd35e589418f9cbec98ea6f80c0cb22668af9b4257285b94e11d820126e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-08-28T19:46:35.712 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:46:35 smithi028 podman[62188]: 2024-08-28 19:46:35.469310678 +0000 UTC m=+4.483682248 container cleanup 3427fd35e589418f9cbec98ea6f80c0cb22668af9b4257285b94e11d820126e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-08-28T19:46:35.713 INFO:journalctl@ceph.mds.b.smithi028.stdout:Aug 28 19:46:35 smithi028 bash[62188]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-b 2024-08-28T19:46:35.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.b.service' 2024-08-28T19:46:35.884 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:35.885 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-08-28T19:46:35.885 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-08-28T19:46:35.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.e 2024-08-28T19:46:36.309 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:46:35 smithi028 systemd[1]: Stopping Ceph mds.e for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:36.309 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:46:36 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e[52312]: 2024-08-28T19:46:36.206+0000 7f8c79275640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:36.310 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:46:36 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e[52312]: 2024-08-28T19:46:36.206+0000 7f8c79275640 -1 mds.e *** got signal Terminated *** 2024-08-28T19:46:39.711 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:46:39 smithi028 podman[62320]: 2024-08-28 19:46:39.444895335 +0000 UTC m=+3.336541950 container died 145f4b253106d50dc672c485bc736ebc7d660f3ce0c5178e43eb8186edecdf89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.1, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-08-28T19:46:40.059 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:46:39 smithi028 podman[62320]: 2024-08-28 19:46:39.809601471 +0000 UTC m=+3.701248074 container cleanup 145f4b253106d50dc672c485bc736ebc7d660f3ce0c5178e43eb8186edecdf89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611) 2024-08-28T19:46:40.059 INFO:journalctl@ceph.mds.e.smithi028.stdout:Aug 28 19:46:39 smithi028 bash[62320]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-e 2024-08-28T19:46:40.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.e.service' 2024-08-28T19:46:40.233 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:40.233 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-08-28T19:46:40.233 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-08-28T19:46:40.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.h 2024-08-28T19:46:40.681 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:46:40 smithi028 systemd[1]: Stopping Ceph mds.h for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:41.059 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:46:40 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-h[52614]: 2024-08-28T19:46:40.680+0000 7f7455a31640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:41.059 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:46:40 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-h[52614]: 2024-08-28T19:46:40.680+0000 7f7455a31640 -1 mds.h *** got signal Terminated *** 2024-08-28T19:46:43.757 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:46:43 smithi028 podman[62439]: 2024-08-28 19:46:43.443172478 +0000 UTC m=+2.987715897 container died 777be3f361a35ed3daa20cd2745385ce01584d8558e2624806a5e2a086696212 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-08-28T19:46:44.059 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:46:43 smithi028 podman[62439]: 2024-08-28 19:46:43.838259257 +0000 UTC m=+3.382802671 container cleanup 777be3f361a35ed3daa20cd2745385ce01584d8558e2624806a5e2a086696212 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.1) 2024-08-28T19:46:44.059 INFO:journalctl@ceph.mds.h.smithi028.stdout:Aug 28 19:46:43 smithi028 bash[62439]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-h 2024-08-28T19:46:44.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.h.service' 2024-08-28T19:46:44.258 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:44.258 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-08-28T19:46:44.259 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-08-28T19:46:44.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.k 2024-08-28T19:46:44.559 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:46:44 smithi028 systemd[1]: Stopping Ceph mds.k for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:45.059 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:46:44 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k[52007]: 2024-08-28T19:46:44.580+0000 7f3323517640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:45.059 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:46:44 smithi028 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k[52007]: 2024-08-28T19:46:44.580+0000 7f3323517640 -1 mds.k *** got signal Terminated *** 2024-08-28T19:46:47.701 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:46:47 smithi028 podman[62562]: 2024-08-28 19:46:47.440979678 +0000 UTC m=+2.967596176 container died 6f3c59426cde30f5529da075a4cfcaed596f8239509b83f6281da4855bb8095c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.1, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-08-28T19:46:48.044 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:46:47 smithi028 podman[62562]: 2024-08-28 19:46:47.80081975 +0000 UTC m=+3.327436240 container cleanup 6f3c59426cde30f5529da075a4cfcaed596f8239509b83f6281da4855bb8095c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-28T19:46:48.045 INFO:journalctl@ceph.mds.k.smithi028.stdout:Aug 28 19:46:47 smithi028 bash[62562]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-k 2024-08-28T19:46:48.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.k.service' 2024-08-28T19:46:48.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:48.197 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-08-28T19:46:48.198 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-08-28T19:46:48.198 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.c 2024-08-28T19:46:48.502 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:46:48 smithi115 systemd[1]: Stopping Ceph mds.c for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:48.764 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:46:48 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-c[53655]: 2024-08-28T19:46:48.500+0000 7fae5fe19640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:48.764 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:46:48 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-c[53655]: 2024-08-28T19:46:48.500+0000 7fae5fe19640 -1 mds.c *** got signal Terminated *** 2024-08-28T19:46:51.711 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:46:51 smithi115 podman[62260]: 2024-08-28 19:46:51.432966438 +0000 UTC m=+3.045054812 container died 81c2473c43d4d655cf77cca177df148de41958ff5f4c896f48296cc61a71334f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-c, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-28T19:46:52.046 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:46:51 smithi115 podman[62260]: 2024-08-28 19:46:51.786754356 +0000 UTC m=+3.398842720 container cleanup 81c2473c43d4d655cf77cca177df148de41958ff5f4c896f48296cc61a71334f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-08-28T19:46:52.047 INFO:journalctl@ceph.mds.c.smithi115.stdout:Aug 28 19:46:51 smithi115 bash[62260]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-c 2024-08-28T19:46:52.217 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.c.service' 2024-08-28T19:46:52.254 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:52.254 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-08-28T19:46:52.254 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-08-28T19:46:52.254 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.f 2024-08-28T19:46:52.591 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:46:52 smithi115 systemd[1]: Stopping Ceph mds.f for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:52.870 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:46:52 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f[53356]: 2024-08-28T19:46:52.590+0000 7f554506c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:52.871 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:46:52 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f[53356]: 2024-08-28T19:46:52.590+0000 7f554506c640 -1 mds.f *** got signal Terminated *** 2024-08-28T19:46:55.741 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:46:55 smithi115 podman[62392]: 2024-08-28 19:46:55.428346372 +0000 UTC m=+2.962986114 container died c6e9556c803c554166000fe860517f2ab58fb2568c7cfa3f1a3a63dd42daff8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-28T19:46:56.050 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:46:55 smithi115 podman[62392]: 2024-08-28 19:46:55.823455432 +0000 UTC m=+3.358095176 container cleanup c6e9556c803c554166000fe860517f2ab58fb2568c7cfa3f1a3a63dd42daff8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-08-28T19:46:56.050 INFO:journalctl@ceph.mds.f.smithi115.stdout:Aug 28 19:46:55 smithi115 bash[62392]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-f 2024-08-28T19:46:56.191 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.f.service' 2024-08-28T19:46:56.225 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:46:56.225 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-08-28T19:46:56.225 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-08-28T19:46:56.225 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.i 2024-08-28T19:46:56.620 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:46:56 smithi115 systemd[1]: Stopping Ceph mds.i for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:46:56.621 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:46:56 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i[53958]: 2024-08-28T19:46:56.535+0000 7fc79d31b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:46:56.621 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:46:56 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i[53958]: 2024-08-28T19:46:56.535+0000 7fc79d31b640 -1 mds.i *** got signal Terminated *** 2024-08-28T19:46:59.620 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:46:59 smithi115 podman[62513]: 2024-08-28 19:46:59.424039482 +0000 UTC m=+2.988521746 container died 65c78c0373ffee1c9258395ad882aae0a94f3bbbaa490f85619fc5a4f48b5523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-08-28T19:46:59.937 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:46:59 smithi115 podman[62513]: 2024-08-28 19:46:59.735609984 +0000 UTC m=+3.300092244 container cleanup 65c78c0373ffee1c9258395ad882aae0a94f3bbbaa490f85619fc5a4f48b5523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.1, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-08-28T19:46:59.937 INFO:journalctl@ceph.mds.i.smithi115.stdout:Aug 28 19:46:59 smithi115 bash[62513]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-i 2024-08-28T19:47:00.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.i.service' 2024-08-28T19:47:00.181 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:47:00.182 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-08-28T19:47:00.182 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-08-28T19:47:00.182 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.l 2024-08-28T19:47:00.620 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:47:00 smithi115 systemd[1]: Stopping Ceph mds.l for eabc438a-6570-11ef-bcd4-c7b262605968... 2024-08-28T19:47:00.621 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:47:00 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l[53055]: 2024-08-28T19:47:00.488+0000 7fb23a64c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-28T19:47:00.621 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:47:00 smithi115 ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l[53055]: 2024-08-28T19:47:00.488+0000 7fb23a64c640 -1 mds.l *** got signal Terminated *** 2024-08-28T19:47:03.757 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:47:03 smithi115 podman[62635]: 2024-08-28 19:47:03.422463973 +0000 UTC m=+3.026053676 container died b1f104dd337314daa8f87a29d0512d4b9ed97f686e3bd2ed5a4a3b37f7be6d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, io.buildah.version=1.37.1, org.label-schema.build-date=20240731, ceph=True) 2024-08-28T19:47:04.032 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:47:03 smithi115 podman[62635]: 2024-08-28 19:47:03.855531095 +0000 UTC m=+3.459120796 container cleanup b1f104dd337314daa8f87a29d0512d4b9ed97f686e3bd2ed5a4a3b37f7be6d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e18f78946a0f2b5a73edfdd8c722a31841f880199baf0b88caf17994ffbc1a61, name=ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-20240828.085847-main-debug-d19e611, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-08-28T19:47:04.032 INFO:journalctl@ceph.mds.l.smithi115.stdout:Aug 28 19:47:03 smithi115 bash[62635]: ceph-eabc438a-6570-11ef-bcd4-c7b262605968-mds-l 2024-08-28T19:47:04.231 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eabc438a-6570-11ef-bcd4-c7b262605968@mds.l.service' 2024-08-28T19:47:04.262 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-28T19:47:04.263 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-08-28T19:47:04.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eabc438a-6570-11ef-bcd4-c7b262605968 --force --keep-logs 2024-08-28T19:47:04.473 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:47:06.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eabc438a-6570-11ef-bcd4-c7b262605968 --force --keep-logs 2024-08-28T19:47:07.090 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:47:09.663 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eabc438a-6570-11ef-bcd4-c7b262605968 --force --keep-logs 2024-08-28T19:47:09.864 INFO:teuthology.orchestra.run.smithi115.stdout:Deleting cluster with fsid: eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T19:47:12.279 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-28T19:47:12.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-28T19:47:12.345 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-28T19:47:12.373 INFO:tasks.cephadm:Archiving crash dumps... 2024-08-28T19:47:12.375 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752/remote/smithi016/crash 2024-08-28T19:47:12.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/crash -- . 2024-08-28T19:47:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-08-28T19:47:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-08-28T19:47:12.414 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752/remote/smithi028/crash 2024-08-28T19:47:12.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/crash -- . 2024-08-28T19:47:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-08-28T19:47:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-08-28T19:47:12.500 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752/remote/smithi115/crash 2024-08-28T19:47:12.501 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/crash -- . 2024-08-28T19:47:12.548 INFO:teuthology.orchestra.run.smithi115.stderr:tar: /var/lib/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-08-28T19:47:12.549 INFO:teuthology.orchestra.run.smithi115.stderr:tar: Error is not recoverable: exiting now 2024-08-28T19:47:12.550 INFO:tasks.cephadm:Checking cluster log for badness... 2024-08-28T19:47:12.550 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-08-28T19:47:12.592 INFO:tasks.cephadm:Compressing logs... 2024-08-28T19:47:12.593 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-28T19:47:12.635 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-28T19:47:12.638 DEBUG:teuthology.orchestra.run.smithi115:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-28T19:47:12.661 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’: No such file or directorygzip 2024-08-28T19:47:12.661 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-28T19:47:12.661 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.d.log 2024-08-28T19:47:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.a.log 2024-08-28T19:47:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.log 2024-08-28T19:47:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-08-28T19:47:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.0.log 2024-08-28T19:47:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.3.log 2024-08-28T19:47:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.g.log 2024-08-28T19:47:12.664 INFO:teuthology.orchestra.run.smithi016.stderr: 84.5% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-08-28T19:47:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.j.log 2024-08-28T19:47:12.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mgr.x.log 2024-08-28T19:47:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.j.log: 87.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.log.gz 2024-08-28T19:47:12.672 INFO:teuthology.orchestra.run.smithi016.stderr: 95.1% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.d.log.gz 2024-08-28T19:47:12.673 INFO:teuthology.orchestra.run.smithi016.stderr: 95.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-28T19:47:12.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-volume.log 2024-08-28T19:47:12.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.6.log 2024-08-28T19:47:12.675 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’gzip: No such file or directory -5 2024-08-28T19:47:12.675 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/cephadm.log 2024-08-28T19:47:12.675 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.e.log 2024-08-28T19:47:12.677 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mon.b.log 2024-08-28T19:47:12.677 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.k.log 2024-08-28T19:47:12.677 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.e.log: /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.10.log 2024-08-28T19:47:12.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.audit.log 2024-08-28T19:47:12.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.6.log: 95.1% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.j.log.gz 2024-08-28T19:47:12.683 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mon.a.log 2024-08-28T19:47:12.684 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.audit.log: 90.1% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-08-28T19:47:12.684 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mgr.y.log 2024-08-28T19:47:12.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.log 2024-08-28T19:47:12.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-08-28T19:47:12.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.h.log 2024-08-28T19:47:12.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-volume.log 2024-08-28T19:47:12.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.1.log 2024-08-28T19:47:12.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.audit.log 2024-08-28T19:47:12.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.b.log 2024-08-28T19:47:12.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.7.log 2024-08-28T19:47:12.686 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.4.log 2024-08-28T19:47:12.687 INFO:teuthology.orchestra.run.smithi115.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-28T19:47:12.687 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-28T19:47:12.687 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mon.c.log 2024-08-28T19:47:12.687 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.l.log 2024-08-28T19:47:12.687 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.2.log 2024-08-28T19:47:12.687 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.log 2024-08-28T19:47:12.687 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mgr.z.log 2024-08-28T19:47:12.688 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-08-28T19:47:12.688 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.i.log 2024-08-28T19:47:12.688 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.f.log 2024-08-28T19:47:12.688 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.c.log 2024-08-28T19:47:12.688 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.8.log 2024-08-28T19:47:12.688 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-volume.log 2024-08-28T19:47:12.689 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.audit.log 2024-08-28T19:47:12.689 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.11.log 2024-08-28T19:47:12.689 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.5.log 2024-08-28T19:47:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.9.log 2024-08-28T19:47:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mon.a.log: 90.9% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-08-28T19:47:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-08-28T19:47:12.711 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-08-28T19:47:12.719 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.7.log: /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.4.log: 82.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-08-28T19:47:12.818 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.11.log: /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.5.log: 93.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mgr.z.log.gz 2024-08-28T19:47:12.821 INFO:teuthology.orchestra.run.smithi115.stderr: 90.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-08-28T19:47:12.842 INFO:teuthology.orchestra.run.smithi115.stderr: 82.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-08-28T19:47:12.851 INFO:teuthology.orchestra.run.smithi115.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-28T19:47:12.943 INFO:teuthology.orchestra.run.smithi028.stderr: 93.1% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mgr.y.log.gz 2024-08-28T19:47:12.958 INFO:teuthology.orchestra.run.smithi115.stderr: 87.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.log.gz 2024-08-28T19:47:12.989 INFO:teuthology.orchestra.run.smithi115.stderr: 91.0% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-08-28T19:47:12.993 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.log.gz 2024-08-28T19:47:13.016 INFO:teuthology.orchestra.run.smithi028.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-28T19:47:13.044 INFO:teuthology.orchestra.run.smithi028.stderr: 90.8% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-08-28T19:47:13.048 INFO:teuthology.orchestra.run.smithi028.stderr: 90.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-08-28T19:47:13.243 INFO:teuthology.orchestra.run.smithi016.stderr: 90.5% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mgr.x.log.gz 2024-08-28T19:47:14.269 INFO:teuthology.orchestra.run.smithi016.stderr: 90.3% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mon.a.log.gz 2024-08-28T19:47:16.908 INFO:teuthology.orchestra.run.smithi016.stderr: 95.0% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.g.log.gz 2024-08-28T19:47:18.404 INFO:teuthology.orchestra.run.smithi016.stderr: 95.1% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.a.log.gz 2024-08-28T19:47:24.387 INFO:teuthology.orchestra.run.smithi115.stderr: 91.6% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mon.c.log.gz 2024-08-28T19:47:49.564 INFO:teuthology.orchestra.run.smithi028.stderr: 95.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.k.log.gz 2024-08-28T19:47:53.484 INFO:teuthology.orchestra.run.smithi016.stderr: 91.4% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.0.log.gz 2024-08-28T19:47:57.665 INFO:teuthology.orchestra.run.smithi028.stderr: 93.3% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mon.b.log.gz 2024-08-28T19:48:00.667 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.6.log.gz 2024-08-28T19:48:05.226 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.3.log.gz 2024-08-28T19:48:09.440 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.9.log.gz 2024-08-28T19:48:09.443 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-08-28T19:48:09.443 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m56.793s 2024-08-28T19:48:09.443 INFO:teuthology.orchestra.run.smithi016.stderr:user 3m23.779s 2024-08-28T19:48:09.443 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m5.955s 2024-08-28T19:48:53.778 INFO:teuthology.orchestra.run.smithi028.stderr: 95.1% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.e.log.gz 2024-08-28T19:49:40.372 INFO:teuthology.orchestra.run.smithi028.stderr: 91.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.10.log.gz 2024-08-28T19:52:01.144 INFO:teuthology.orchestra.run.smithi028.stderr: 91.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.4.log.gz 2024-08-28T19:52:03.576 INFO:teuthology.orchestra.run.smithi115.stderr: 91.0% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.8.log.gz 2024-08-28T19:52:05.427 INFO:teuthology.orchestra.run.smithi115.stderr: 95.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.l.log.gz 2024-08-28T19:52:21.471 INFO:teuthology.orchestra.run.smithi028.stderr: 91.4% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.1.log.gz 2024-08-28T19:54:12.413 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.7.log.gz 2024-08-28T19:56:24.534 INFO:teuthology.orchestra.run.smithi115.stderr: 93.0% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.c.log.gz 2024-08-28T19:57:17.650 INFO:teuthology.orchestra.run.smithi115.stderr: 91.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.2.log.gz 2024-08-28T19:57:41.620 INFO:teuthology.orchestra.run.smithi115.stderr: 91.4% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.11.log.gz 2024-08-28T19:58:26.170 INFO:teuthology.orchestra.run.smithi028.stderr: 93.1% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.h.log.gz 2024-08-28T19:59:33.042 INFO:teuthology.orchestra.run.smithi028.stderr: 93.2% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.b.log.gz 2024-08-28T19:59:33.044 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-08-28T19:59:33.045 INFO:teuthology.orchestra.run.smithi028.stderr:real 12m20.391s 2024-08-28T19:59:33.048 INFO:teuthology.orchestra.run.smithi028.stderr:user 7m48.438s 2024-08-28T19:59:33.048 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m18.044s 2024-08-28T20:01:50.434 INFO:teuthology.orchestra.run.smithi115.stderr: 90.9% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-osd.5.log.gz 2024-08-28T20:02:22.953 INFO:teuthology.orchestra.run.smithi115.stderr: 93.0% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.f.log.gz 2024-08-28T20:03:16.744 INFO:teuthology.orchestra.run.smithi115.stderr: 93.0% -- replaced with /var/log/ceph/eabc438a-6570-11ef-bcd4-c7b262605968/ceph-mds.i.log.gz 2024-08-28T20:03:16.746 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-08-28T20:03:16.746 INFO:teuthology.orchestra.run.smithi115.stderr:real 16m4.092s 2024-08-28T20:03:16.747 INFO:teuthology.orchestra.run.smithi115.stderr:user 7m56.424s 2024-08-28T20:03:16.747 INFO:teuthology.orchestra.run.smithi115.stderr:sys 0m21.852s 2024-08-28T20:03:16.747 INFO:tasks.cephadm:Archiving logs... 2024-08-28T20:03:16.748 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752/remote/smithi016/log 2024-08-28T20:03:16.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-28T20:03:49.606 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752/remote/smithi028/log 2024-08-28T20:03:49.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-28T20:05:00.640 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752/remote/smithi115/log 2024-08-28T20:05:00.642 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-28T20:06:16.671 INFO:tasks.cephadm:Removing cluster... 2024-08-28T20:06:16.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eabc438a-6570-11ef-bcd4-c7b262605968 --force 2024-08-28T20:06:16.884 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T20:06:17.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eabc438a-6570-11ef-bcd4-c7b262605968 --force 2024-08-28T20:06:17.662 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T20:06:18.438 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eabc438a-6570-11ef-bcd4-c7b262605968 --force 2024-08-28T20:06:18.641 INFO:teuthology.orchestra.run.smithi115.stdout:Deleting cluster with fsid: eabc438a-6570-11ef-bcd4-c7b262605968 2024-08-28T20:06:19.458 INFO:tasks.cephadm:Removing cephadm ... 2024-08-28T20:06:19.458 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-28T20:06:19.478 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-28T20:06:19.506 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-28T20:06:19.523 INFO:tasks.cephadm:Teardown complete 2024-08-28T20:06:19.523 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-28T20:06:19.535 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-28T20:06:19.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-28T20:06:19.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-28T20:06:19.549 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-28T20:06:19.700 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-28T20:06:19.710 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-28T20:06:19.711 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-28T20:06:19.714 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-28T20:06:19.716 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-28T20:06:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-08-28T20:06:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-08-28T20:06:19.731 INFO:teuthology.orchestra.run.smithi115.stderr:bash: line 1: ntpq: command not found 2024-08-28T20:06:19.734 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-28T20:06:19.734 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-08-28T20:06:19.734 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv01.front.sepia.ceph.com 3 7 377 72 +285us[ +277us] +/- 61ms 2024-08-28T20:06:19.734 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 16 -720us[ -720us] +/- 64ms 2024-08-28T20:06:19.734 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +656us[ +656us] +/- 82ms 2024-08-28T20:06:19.734 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T20:06:19.758 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-28T20:06:19.758 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-08-28T20:06:19.758 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv01.front.sepia.ceph.com 3 6 377 10 +248us[ +256us] +/- 62ms 2024-08-28T20:06:19.758 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 19 -706us[ -698us] +/- 64ms 2024-08-28T20:06:19.759 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +702us[ +710us] +/- 82ms 2024-08-28T20:06:19.759 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T20:06:19.766 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-28T20:06:19.766 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-08-28T20:06:19.766 INFO:teuthology.orchestra.run.smithi115.stdout:^* hv01.front.sepia.ceph.com 3 6 377 11 +262us[ +261us] +/- 62ms 2024-08-28T20:06:19.767 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 -732us[ -732us] +/- 64ms 2024-08-28T20:06:19.767 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +702us[ +702us] +/- 82ms 2024-08-28T20:06:19.767 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-28T20:06:19.768 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-28T20:06:19.801 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-28T20:06:19.802 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-28T20:06:19.867 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-28T20:06:19.955 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-28T20:06:20.033 DEBUG:teuthology.orchestra.run.smithi115:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-28T20:06:20.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-28T20:06:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T20:06:20.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-28T20:06:20.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T20:06:20.172 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-28T20:06:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T20:06:20.199 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-28T20:06:20.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-28T20:06:20.260 INFO:teuthology.task.internal:Duration was 3698.849548 seconds 2024-08-28T20:06:20.260 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-28T20:06:20.282 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-28T20:06:20.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-28T20:06:20.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-28T20:06:20.287 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-28T20:06:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-28T20:06:20.347 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-28T20:06:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-28T20:06:20.794 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-28T20:06:20.794 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-08-28T20:06:20.794 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-28T20:06:20.858 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-08-28T20:06:20.858 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-28T20:06:20.888 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2024-08-28T20:06:20.888 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-28T20:06:20.919 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-28T20:06:20.920 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-28T20:06:20.922 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-28T20:06:20.930 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-28T20:06:20.988 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-28T20:06:20.989 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-28T20:06:20.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-28T20:06:21.004 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-28T20:06:23.140 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-28T20:06:23.152 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-28T20:06:23.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-28T20:06:23.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-28T20:06:23.212 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-28T20:06:23.256 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-28T20:06:23.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-28T20:06:23.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-28T20:06:23.272 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-28T20:06:23.290 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-08-28T20:06:23.304 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-08-28T20:06:23.329 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2024-08-28T20:06:23.353 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-28T20:06:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T20:06:23.368 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-28T20:06:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T20:06:23.398 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-28T20:06:23.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-28T20:06:23.414 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-28T20:06:23.425 INFO:teuthology.task.internal:Transferring archived files... 2024-08-28T20:06:23.425 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752/remote/smithi016 2024-08-28T20:06:23.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-28T20:06:23.504 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752/remote/smithi028 2024-08-28T20:06:23.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-28T20:06:23.542 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-08-28_12:36:35-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7877752/remote/smithi115 2024-08-28T20:06:23.542 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-28T20:06:23.579 INFO:teuthology.task.internal:Removing archive directory... 2024-08-28T20:06:23.579 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-28T20:06:23.582 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-28T20:06:23.584 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-28T20:06:23.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-28T20:06:23.643 INFO:teuthology.task.internal:Not uploading archives. 2024-08-28T20:06:23.643 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-28T20:06:23.694 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-28T20:06:23.694 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-28T20:06:23.697 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-28T20:06:23.699 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-28T20:06:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 28 20:06 /home/ubuntu/cephtest 2024-08-28T20:06:23.713 INFO:teuthology.orchestra.run.smithi028.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 28 20:06 /home/ubuntu/cephtest 2024-08-28T20:06:23.715 INFO:teuthology.orchestra.run.smithi115.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 28 20:06 /home/ubuntu/cephtest 2024-08-28T20:06:23.734 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-28T20:06:23.746 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-28T20:06:23.778 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 3698.849548101425 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d19e6118f7015f0047264006cb0c873528eb3ae6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_mchangir@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=149d6c275a9d49ca9f865df730ba147a status: fail success: false 2024-08-28T20:06:23.779 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-28T20:06:23.912 INFO:teuthology.run:FAIL